2022-09-07T21:08:20.784 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-07T21:08:20.785 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2022-09-07_15:12:08-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi --verbose --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2022-09-07_15:12:08-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7018878 --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} -- /home/teuthworker/archive/pdonnell-2022-09-07_15:12:08-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7018878/orig.config.yaml 2022-09-07T21:08:20.855 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-09-07T21:08:20.963 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2022-09-07_15:12:08-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7018878 branch: wip-pdonnell-testing-20220907.015905 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7018878' kernel: client: branch: testing kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2022-09-07_15:12:08-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20220907.015905 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - SLOW_OPS - slow request sha1: 152194198413033811b31149de909a7ec5852a7f subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 152194198413033811b31149de909a7ec5852a7f kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20220907.015905 sha1: 75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d owner: scheduled_pdonnell@teuthology priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 sha1: 152194198413033811b31149de909a7ec5852a7f sleep_before_teardown: 0 suite: fs:workload suite_branch: wip-pdonnell-testing-20220907.015905 suite_path: /home/teuthworker/src/github.com_batrick_ceph_75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: 75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi047.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/KRF2hnjF9IYYhMQ/CIy1KEJjbQICUDvtsBW6PjMlMdhLmiH/JVuZ71AC31MJ7u+/KnViPBJ7fqk3sDVcx3FVwqzqvZ2amm89Gvsa7RiSSvvco+4NHCXw2VFwboKxD2gz1qlpgmUP+mvgAKVp97L0g7lvjmlY4Pq8o4il4BV3Kn8WrWWiOycXPooNV1ObCgL5upGE0BheVOChPoEz08CMVpWbZei8nT0zrzLPV8g0+fvTJzx9sGmUSaoZfMuzEr7KE/VbMkZU+VP1f8CrqxakttrHUnicnKl+8di6iXfsILXEih3vKo6wCvowAJvy9eN7wWRBpqCqcebAAYLBtpnmZByDU52Q4jVbz6crtKrqr9oJLPYyuCu13gEPGV4B0IXmzEF7JLtqTru36gGiIvQkI7IKlU0Ke/2cBkDKaxfGFGklnHmh3i3Y08SnZKY9v5oh505GOl3edp+JIpFMGN0lcAOoEG7RrKmx6FjAP/zPp2URf0xR+PiCRqruBU112qE= smithi055.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCvwiaGnoNTB9RcOPAPuy02ImM6fw2yQ4TARMLPyc3G0ssTILFGgLcqevZmkmU9h3j3Sy5uHaKERDryd4ahNSCWQLiqbSFZs/dApEg2GX87p+sOyGr9d5Y7UPU+GagnxVVWYr90CccdLEU55BT3tKAHie0XP87imzU5mKlLmGj7XpfTkdRqHh06c8D3FQNoIbqHxvRE+SWT8YAJ/QxChilVWFFLGyoqM8kTEYGx5V4qn8NH6Z4COocLcdVPzFUSeexrBCxPyQduej0bmfUk+Poi6Mo89+E6aCeaUEdwSt+Q2iUz4Yyu7ipVkAcg+KGMprY9jdq3l7B/EPnVg/FAf14rQPOMJVe9cr8WMAYxFJaTsFVud5Hiq4Z1irF+6OH0V7Ah6WLhyPE7oebxUnyxYScNGfxjbT+UQsf898Sw2m1TpBdPHvii5Gb4SstMhJ7QrT8OQYvFB/Xp98QDx3Yp1ylAFjqn7sivQRw4nmVZCNXu3OGVMbsJAi2NEVnqSe64Dk= smithi059.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuNZh6KmNTqXxG/b5T5G9en3OofMZJR3ffhnzzaggdEULkISevCOhKGyodRQmAqH6ZjRJ97W7/dpGUzs4gSRVsxmwtBjScQGCe2rGALxU2C49y2vCJK3eacS2e8u4uBUlzIBM1deK1tqs9TynQjtsHXK0YO2y9y/20PXi192RG9oG3Fvf38Vs7lw8TG6uhVICFdkSW9Ttxtbi74TRE7gV3Gmej4J/69cxMmH4O79JMouEnrtPIaMJJvEEjr1vwkggBDkVKQ08Pk/FwuyYrsGYtPSsHLm4EeI1u4JU2Hgj35U7Yv2tlcSW0egLU9xhp63DpNk/pqMJRlzDflSuUR6e5gb0cxtAOgcZjUV3cIc+MAalsPlWqA40sSerB08oAmQsevRI/jgqJCGk2yK1/6aDNBWw1PpsUAsz3l+Jma7px8OQ+l8fMlWxrxNVTEHqMBHM3l94kLJGi+hCUXSfMKsINYRdacietVCDzeHjV+S+vcsgNz/60HA8ow3/sj+V02tk= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b58f2c18636eb10faa77ed3614abd00cb85dfc2c timestamp: 2022-09-07_15:12:08 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1713 2022-09-07T21:08:20.964 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d/qa; will attempt to use it 2022-09-07T21:08:20.965 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d/qa/tasks 2022-09-07T21:08:20.965 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-09-07T21:08:20.967 INFO:teuthology.task.internal:Checking packages... 2022-09-07T21:08:21.000 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '152194198413033811b31149de909a7ec5852a7f' 2022-09-07T21:08:21.001 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-09-07T21:08:21.001 INFO:teuthology.packaging:ref: None 2022-09-07T21:08:21.001 INFO:teuthology.packaging:tag: None 2022-09-07T21:08:21.002 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20220907.015905 2022-09-07T21:08:21.002 INFO:teuthology.packaging:sha1: 152194198413033811b31149de909a7ec5852a7f 2022-09-07T21:08:21.003 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20220907.015905 2022-09-07T21:08:21.165 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-14857.g15219419 2022-09-07T21:08:21.166 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-09-07T21:08:21.173 INFO:teuthology.task.internal:no buildpackages task found 2022-09-07T21:08:21.174 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-09-07T21:08:21.184 INFO:teuthology.task.internal:Saving configuration 2022-09-07T21:08:21.211 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-09-07T21:08:21.218 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-07T21:08:21.243 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2022-09-07_15:12:08-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7018878', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-07 21:03:50.031549', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGJZl1ARPEc5f7wPPQn9lkj6/3PgRNMpAuvhYIsb8J14hb2P+a+Zs3GiayyCVBb33PHHzWN7BD2DLIA3Gi01ejY='} 2022-09-07T21:08:21.267 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2022-09-07_15:12:08-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7018878', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-07 21:03:50.030351', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN7bw6Ktvr09p4qOZM5LHst631WrjHsi8Q5h98AyAB0hCNLOoE8q8MaxCbvxyN+o/OsSg3gqXsWiLOWaAIihvAQ='} 2022-09-07T21:08:21.296 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2022-09-07_15:12:08-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7018878', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-07 21:03:50.028945', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAcQPgLq/NfZwGHAl5MII/g2xmU/GrV5gWMffHoq9RYi1cUazJhS4a8HOHXa9fz0n0b+UwP7YR2L38qdGn/ws3Q='} 2022-09-07T21:08:21.296 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-09-07T21:08:21.304 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2022-09-07T21:08:21.304 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2022-09-07T21:08:21.304 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2022-09-07T21:08:21.304 INFO:teuthology.run_tasks:Running task console_log... 2022-09-07T21:08:21.419 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7bb3a977b8>, signals=[15]) 2022-09-07T21:08:21.420 INFO:teuthology.run_tasks:Running task internal.connect... 2022-09-07T21:08:21.427 INFO:teuthology.task.internal:Opening connections... 2022-09-07T21:08:21.427 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi047.front.sepia.ceph.com 2022-09-07T21:08:21.429 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-07T21:08:21.527 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2022-09-07T21:08:21.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-07T21:08:21.621 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2022-09-07T21:08:21.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-07T21:08:21.719 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-09-07T21:08:21.726 DEBUG:teuthology.orchestra.run.smithi047:> uname -m 2022-09-07T21:08:21.748 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2022-09-07T21:08:21.749 DEBUG:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2022-09-07T21:08:21.809 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="Red Hat Enterprise Linux" 2022-09-07T21:08:21.810 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="8.6 (Ootpa)" 2022-09-07T21:08:21.810 INFO:teuthology.orchestra.run.smithi047.stdout:ID="rhel" 2022-09-07T21:08:21.810 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE="fedora" 2022-09-07T21:08:21.810 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="8.6" 2022-09-07T21:08:21.810 INFO:teuthology.orchestra.run.smithi047.stdout:PLATFORM_ID="platform:el8" 2022-09-07T21:08:21.811 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-07T21:08:21.811 INFO:teuthology.orchestra.run.smithi047.stdout:ANSI_COLOR="0;31" 2022-09-07T21:08:21.811 INFO:teuthology.orchestra.run.smithi047.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-07T21:08:21.811 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://www.redhat.com/" 2022-09-07T21:08:21.812 INFO:teuthology.orchestra.run.smithi047.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-07T21:08:21.812 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-07T21:08:21.812 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:08:21.812 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-07T21:08:21.813 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-07T21:08:21.813 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-07T21:08:21.813 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-07T21:08:21.815 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2022-09-07T21:08:21.837 DEBUG:teuthology.orchestra.run.smithi055:> uname -m 2022-09-07T21:08:21.859 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2022-09-07T21:08:21.860 DEBUG:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2022-09-07T21:08:21.920 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="Red Hat Enterprise Linux" 2022-09-07T21:08:21.920 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="8.6 (Ootpa)" 2022-09-07T21:08:21.921 INFO:teuthology.orchestra.run.smithi055.stdout:ID="rhel" 2022-09-07T21:08:21.921 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="fedora" 2022-09-07T21:08:21.921 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="8.6" 2022-09-07T21:08:21.922 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el8" 2022-09-07T21:08:21.922 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-07T21:08:21.923 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31" 2022-09-07T21:08:21.923 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-07T21:08:21.923 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://www.redhat.com/" 2022-09-07T21:08:21.923 INFO:teuthology.orchestra.run.smithi055.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-07T21:08:21.924 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-07T21:08:21.924 INFO:teuthology.orchestra.run.smithi055.stdout: 2022-09-07T21:08:21.925 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-07T21:08:21.925 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-07T21:08:21.925 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-07T21:08:21.925 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-07T21:08:21.926 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2022-09-07T21:08:21.947 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2022-09-07T21:08:21.969 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2022-09-07T21:08:21.970 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2022-09-07T21:08:22.030 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="Red Hat Enterprise Linux" 2022-09-07T21:08:22.030 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="8.6 (Ootpa)" 2022-09-07T21:08:22.030 INFO:teuthology.orchestra.run.smithi059.stdout:ID="rhel" 2022-09-07T21:08:22.031 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="fedora" 2022-09-07T21:08:22.031 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="8.6" 2022-09-07T21:08:22.031 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el8" 2022-09-07T21:08:22.031 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-07T21:08:22.032 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2022-09-07T21:08:22.032 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-07T21:08:22.032 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://www.redhat.com/" 2022-09-07T21:08:22.032 INFO:teuthology.orchestra.run.smithi059.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-07T21:08:22.032 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-07T21:08:22.033 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:08:22.033 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-07T21:08:22.033 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-07T21:08:22.033 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-07T21:08:22.034 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-07T21:08:22.034 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2022-09-07T21:08:22.053 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-09-07T21:08:22.064 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-09-07T21:08:22.071 INFO:teuthology.task.internal:Checking for old test directory... 2022-09-07T21:08:22.072 DEBUG:teuthology.orchestra.run.smithi047:> test '!' -e /home/ubuntu/cephtest 2022-09-07T21:08:22.075 DEBUG:teuthology.orchestra.run.smithi055:> test '!' -e /home/ubuntu/cephtest 2022-09-07T21:08:22.077 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2022-09-07T21:08:22.097 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-09-07T21:08:22.104 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-09-07T21:08:22.105 DEBUG:teuthology.orchestra.run.smithi047:> test -z $(ls -A /var/lib/ceph) 2022-09-07T21:08:22.135 DEBUG:teuthology.orchestra.run.smithi055:> test -z $(ls -A /var/lib/ceph) 2022-09-07T21:08:22.138 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2022-09-07T21:08:22.175 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-09-07T21:08:22.262 INFO:teuthology.run_tasks:Running task kernel... 2022-09-07T21:08:22.287 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2022-09-07T21:08:22.288 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2022-09-07T21:08:22.289 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2022-09-07T21:08:22.290 DEBUG:teuthology.orchestra.run.smithi047:> test -f /run/.containerenv -o -f /.dockerenv 2022-09-07T21:08:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-07T21:08:22.312 DEBUG:teuthology.orchestra.run.smithi047:> sudo lsb_release -is 2022-09-07T21:08:22.407 INFO:teuthology.orchestra.run.smithi047.stdout:RedHatEnterprise 2022-09-07T21:08:22.408 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-07T21:08:22.409 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2022-09-07T21:08:22.616 DEBUG:teuthology.task.kernel:sha1 for client.0 is a127ffb7df2a46d9c720771310f2b324a2af81f2 2022-09-07T21:08:22.617 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "a127ffb7df2a46d9c720771310f2b324a2af81f2"... 2022-09-07T21:08:22.617 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2022-09-07T21:08:22.640 INFO:teuthology.orchestra.run.smithi047.stdout:4.18.0-372.9.1.el8.x86_64 2022-09-07T21:08:22.641 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-372.9.1.el8.x86_64 vs a127ffb7df2a46d9c720771310f2b324a2af81f2 2022-09-07T21:08:22.641 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2022-09-07T21:08:22.642 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum upgrade -y linux-firmware 2022-09-07T21:08:23.781 INFO:teuthology.orchestra.run.smithi047.stdout:Updating Subscription Management repositories. 2022-09-07T21:08:23.782 INFO:teuthology.orchestra.run.smithi047.stdout:Unable to read consumer identity 2022-09-07T21:08:23.856 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:08:23.857 INFO:teuthology.orchestra.run.smithi047.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2022-09-07T21:08:23.857 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:08:25.288 INFO:teuthology.orchestra.run.smithi047.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2022-09-07T21:08:26.430 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 11 MB/s | 13 MB 00:01 2022-09-07T21:08:29.447 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 426 kB/s | 24 kB 00:00 2022-09-07T21:08:30.179 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2022-09-07T21:08:30.180 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2022-09-07T21:08:30.180 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2022-09-07T21:08:30.309 INFO:teuthology.task.kernel:Downloading kernel a127ffb7df2a46d9c720771310f2b324a2af81f2 on client.0... 2022-09-07T21:08:30.310 DEBUG:teuthology.orchestra.run.smithi047:> sudo lsb_release -is 2022-09-07T21:08:30.352 INFO:teuthology.orchestra.run.smithi047.stdout:RedHatEnterprise 2022-09-07T21:08:30.354 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-07T21:08:30.354 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a127ffb7df2a46d9c720771310f2b324a2af81f2 2022-09-07T21:08:30.500 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/a127ffb7df2a46d9c720771310f2b324a2af81f2/centos/8/flavors/default/x86_64/ 2022-09-07T21:08:30.501 DEBUG:teuthology.orchestra.run.smithi047:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.0.0_rc4_ceph_ga127ffb7df2a-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/a127ffb7df2a46d9c720771310f2b324a2af81f2/centos/8/flavors/default/x86_64/ --input-file=- 2022-09-07T21:08:30.503 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi047.front.sepia.ceph.com to complete... 2022-09-07T21:08:49.413 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07 21:08:49 URL:https://4.chacra.ceph.com/r/kernel/testing/a127ffb7df2a46d9c720771310f2b324a2af81f2/centos/8/flavors/default/x86_64/kernel-6.0.0_rc4_ceph_ga127ffb7df2a-1.x86_64.rpm [528623720/528623720] -> "/tmp/kernel.x86_64.rpm" [1] 2022-09-07T21:08:49.414 INFO:teuthology.orchestra.run.smithi047.stderr:FINISHED --2022-09-07 21:08:49-- 2022-09-07T21:08:49.414 INFO:teuthology.orchestra.run.smithi047.stderr:Total wall clock time: 19s 2022-09-07T21:08:49.415 INFO:teuthology.orchestra.run.smithi047.stderr:Downloaded: 1 files, 504M in 18s (27.3 MB/s) 2022-09-07T21:08:49.416 INFO:teuthology.task.kernel:Installing kernel a127ffb7df2a46d9c720771310f2b324a2af81f2 on client.0... 2022-09-07T21:08:49.416 DEBUG:teuthology.orchestra.run.smithi047:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2022-09-07T21:08:51.624 INFO:teuthology.orchestra.run.smithi047.stdout:Verifying... ######################################## 2022-09-07T21:08:51.738 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing... ######################################## 2022-09-07T21:08:51.828 INFO:teuthology.orchestra.run.smithi047.stdout:Updating / installing... 2022-09-07T21:09:34.148 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-6.0.0_rc4_ceph_ga127ffb7df2a-1 ######################################## 2022-09-07T21:09:58.767 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi047.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2022-09-07T21:09:58.769 DEBUG:teuthology.orchestra.run.smithi047:> rpm -qlp /tmp/kernel.x86_64.rpm 2022-09-07T21:09:58.847 INFO:teuthology.orchestra.run.smithi047.stdout:/boot/System.map-6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-07T21:09:58.847 INFO:teuthology.orchestra.run.smithi047.stdout:/boot/config-6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-07T21:09:58.847 INFO:teuthology.orchestra.run.smithi047.stdout:/boot/vmlinux-6.0.0-rc4-ceph-ga127ffb7df2a.bz2 2022-09-07T21:09:58.848 INFO:teuthology.orchestra.run.smithi047.stdout:/boot/vmlinuz-6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-07T21:09:58.848 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-07T21:09:58.848 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel 2022-09-07T21:09:58.848 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch 2022-09-07T21:09:58.849 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86 2022-09-07T21:09:58.849 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto 2022-09-07T21:09:58.849 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/blowfish-x86_64.ko 2022-09-07T21:09:58.850 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2022-09-07T21:09:58.850 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2022-09-07T21:09:58.850 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/camellia-x86_64.ko 2022-09-07T21:09:58.851 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2022-09-07T21:09:58.851 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2022-09-07T21:09:58.851 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/crc32-pclmul.ko 2022-09-07T21:09:58.852 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/crc32c-intel.ko 2022-09-07T21:09:58.852 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2022-09-07T21:09:58.852 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2022-09-07T21:09:58.853 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2022-09-07T21:09:58.853 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2022-09-07T21:09:58.854 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/serpent-avx2.ko 2022-09-07T21:09:58.854 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2022-09-07T21:09:58.855 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/sha1-ssse3.ko 2022-09-07T21:09:58.855 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/sha256-ssse3.ko 2022-09-07T21:09:58.856 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/sha512-ssse3.ko 2022-09-07T21:09:58.857 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2022-09-07T21:09:58.857 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2022-09-07T21:09:58.857 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/twofish-x86_64.ko 2022-09-07T21:09:58.858 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kernel 2022-09-07T21:09:58.859 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kernel/cpu 2022-09-07T21:09:58.859 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kernel/cpu/mce 2022-09-07T21:09:58.860 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2022-09-07T21:09:58.860 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kvm 2022-09-07T21:09:58.861 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kvm/kvm-amd.ko 2022-09-07T21:09:58.861 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kvm/kvm-intel.ko 2022-09-07T21:09:58.862 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kvm/kvm.ko 2022-09-07T21:09:58.862 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto 2022-09-07T21:09:58.862 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/ansi_cprng.ko 2022-09-07T21:09:58.863 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/anubis.ko 2022-09-07T21:09:58.863 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/arc4.ko 2022-09-07T21:09:58.864 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/async_tx 2022-09-07T21:09:58.864 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/async_tx/async_memcpy.ko 2022-09-07T21:09:58.865 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/async_tx/async_pq.ko 2022-09-07T21:09:58.865 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/async_tx/async_raid6_recov.ko 2022-09-07T21:09:58.866 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/async_tx/async_tx.ko 2022-09-07T21:09:58.866 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/async_tx/async_xor.ko 2022-09-07T21:09:58.866 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/async_tx/raid6test.ko 2022-09-07T21:09:58.867 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/authenc.ko 2022-09-07T21:09:58.867 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/authencesn.ko 2022-09-07T21:09:58.868 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/blake2b_generic.ko 2022-09-07T21:09:58.868 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/blowfish_common.ko 2022-09-07T21:09:58.868 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/blowfish_generic.ko 2022-09-07T21:09:58.869 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/camellia_generic.ko 2022-09-07T21:09:58.869 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/cast5_generic.ko 2022-09-07T21:09:58.869 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/cast6_generic.ko 2022-09-07T21:09:58.870 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/cast_common.ko 2022-09-07T21:09:58.870 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/ccm.ko 2022-09-07T21:09:58.871 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/cmac.ko 2022-09-07T21:09:58.871 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/crc32_generic.ko 2022-09-07T21:09:58.872 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/crypto_engine.ko 2022-09-07T21:09:58.872 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/crypto_user.ko 2022-09-07T21:09:58.872 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/cts.ko 2022-09-07T21:09:58.872 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/des_generic.ko 2022-09-07T21:09:58.873 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/dh_generic.ko 2022-09-07T21:09:58.873 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/echainiv.ko 2022-09-07T21:09:58.873 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/essiv.ko 2022-09-07T21:09:58.873 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/fcrypt.ko 2022-09-07T21:09:58.874 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/khazad.ko 2022-09-07T21:09:58.874 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/lz4.ko 2022-09-07T21:09:58.874 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/lz4hc.ko 2022-09-07T21:09:58.874 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/md4.ko 2022-09-07T21:09:58.875 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/michael_mic.ko 2022-09-07T21:09:58.875 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/pcbc.ko 2022-09-07T21:09:58.875 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/pcrypt.ko 2022-09-07T21:09:58.875 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/rmd160.ko 2022-09-07T21:09:58.876 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/seed.ko 2022-09-07T21:09:58.876 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/serpent_generic.ko 2022-09-07T21:09:58.876 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/tcrypt.ko 2022-09-07T21:09:58.876 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/tea.ko 2022-09-07T21:09:58.876 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/twofish_common.ko 2022-09-07T21:09:58.877 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/twofish_generic.ko 2022-09-07T21:09:58.877 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/vmac.ko 2022-09-07T21:09:58.877 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/wp512.ko 2022-09-07T21:09:58.877 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/xcbc.ko 2022-09-07T21:09:58.878 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/xor.ko 2022-09-07T21:09:58.878 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/xxhash_generic.ko 2022-09-07T21:09:58.878 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers 2022-09-07T21:09:58.878 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi 2022-09-07T21:09:58.879 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/acpi_ipmi.ko 2022-09-07T21:09:58.879 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/acpi_pad.ko 2022-09-07T21:09:58.879 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/custom_method.ko 2022-09-07T21:09:58.880 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/ec_sys.ko 2022-09-07T21:09:58.880 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/platform_profile.ko 2022-09-07T21:09:58.880 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/sbs.ko 2022-09-07T21:09:58.881 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/sbshc.ko 2022-09-07T21:09:58.881 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/video.ko 2022-09-07T21:09:58.882 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata 2022-09-07T21:09:58.882 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/acard-ahci.ko 2022-09-07T21:09:58.883 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/ahci_platform.ko 2022-09-07T21:09:58.883 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/ata_generic.ko 2022-09-07T21:09:58.883 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/libahci_platform.ko 2022-09-07T21:09:58.883 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_acpi.ko 2022-09-07T21:09:58.884 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_ali.ko 2022-09-07T21:09:58.884 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_amd.ko 2022-09-07T21:09:58.884 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_artop.ko 2022-09-07T21:09:58.885 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_atiixp.ko 2022-09-07T21:09:58.885 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_atp867x.ko 2022-09-07T21:09:58.885 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_cmd640.ko 2022-09-07T21:09:58.886 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_cmd64x.ko 2022-09-07T21:09:58.886 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_cypress.ko 2022-09-07T21:09:58.886 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_efar.ko 2022-09-07T21:09:58.886 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_hpt366.ko 2022-09-07T21:09:58.887 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_hpt37x.ko 2022-09-07T21:09:58.887 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_hpt3x2n.ko 2022-09-07T21:09:58.887 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_hpt3x3.ko 2022-09-07T21:09:58.888 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_it8213.ko 2022-09-07T21:09:58.888 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_it821x.ko 2022-09-07T21:09:58.888 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_jmicron.ko 2022-09-07T21:09:58.888 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_marvell.ko 2022-09-07T21:09:58.889 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_mpiix.ko 2022-09-07T21:09:58.889 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_netcell.ko 2022-09-07T21:09:58.889 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_ninja32.ko 2022-09-07T21:09:58.889 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_ns87410.ko 2022-09-07T21:09:58.890 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_ns87415.ko 2022-09-07T21:09:58.890 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_oldpiix.ko 2022-09-07T21:09:58.890 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_opti.ko 2022-09-07T21:09:58.891 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_optidma.ko 2022-09-07T21:09:58.891 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_pcmcia.ko 2022-09-07T21:09:58.891 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_pdc2027x.ko 2022-09-07T21:09:58.891 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_pdc202xx_old.ko 2022-09-07T21:09:58.892 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_piccolo.ko 2022-09-07T21:09:58.892 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_rdc.ko 2022-09-07T21:09:58.892 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_sch.ko 2022-09-07T21:09:58.892 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_serverworks.ko 2022-09-07T21:09:58.893 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_sil680.ko 2022-09-07T21:09:58.893 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_sis.ko 2022-09-07T21:09:58.894 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_sl82c105.ko 2022-09-07T21:09:58.894 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_triflex.ko 2022-09-07T21:09:58.895 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_via.ko 2022-09-07T21:09:58.895 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pdc_adma.ko 2022-09-07T21:09:58.896 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_inic162x.ko 2022-09-07T21:09:58.896 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_mv.ko 2022-09-07T21:09:58.897 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_nv.ko 2022-09-07T21:09:58.897 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_promise.ko 2022-09-07T21:09:58.898 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_qstor.ko 2022-09-07T21:09:58.898 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_sil.ko 2022-09-07T21:09:58.898 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_sil24.ko 2022-09-07T21:09:58.899 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_sis.ko 2022-09-07T21:09:58.899 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_svw.ko 2022-09-07T21:09:58.900 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_sx4.ko 2022-09-07T21:09:58.900 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_uli.ko 2022-09-07T21:09:58.901 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_via.ko 2022-09-07T21:09:58.901 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_vsc.ko 2022-09-07T21:09:58.902 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/auxdisplay 2022-09-07T21:09:58.902 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/auxdisplay/ks0108.ko 2022-09-07T21:09:58.903 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/base 2022-09-07T21:09:58.903 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/base/regmap 2022-09-07T21:09:58.903 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/base/regmap/regmap-i2c.ko 2022-09-07T21:09:58.904 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/bcma 2022-09-07T21:09:58.904 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/bcma/bcma.ko 2022-09-07T21:09:58.905 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block 2022-09-07T21:09:58.905 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/aoe 2022-09-07T21:09:58.906 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/aoe/aoe.ko 2022-09-07T21:09:58.906 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/brd.ko 2022-09-07T21:09:58.906 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/drbd 2022-09-07T21:09:58.907 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/drbd/drbd.ko 2022-09-07T21:09:58.907 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/floppy.ko 2022-09-07T21:09:58.908 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/loop.ko 2022-09-07T21:09:58.908 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/mtip32xx 2022-09-07T21:09:58.909 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/mtip32xx/mtip32xx.ko 2022-09-07T21:09:58.909 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/nbd.ko 2022-09-07T21:09:58.909 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/null_blk 2022-09-07T21:09:58.909 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/null_blk/null_blk.ko 2022-09-07T21:09:58.910 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/pktcdvd.ko 2022-09-07T21:09:58.910 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/rbd.ko 2022-09-07T21:09:58.910 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/virtio_blk.ko 2022-09-07T21:09:58.911 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/xen-blkback 2022-09-07T21:09:58.911 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/xen-blkback/xen-blkback.ko 2022-09-07T21:09:58.911 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/xen-blkfront.ko 2022-09-07T21:09:58.912 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/zram 2022-09-07T21:09:58.912 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/zram/zram.ko 2022-09-07T21:09:58.912 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char 2022-09-07T21:09:58.912 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/hangcheck-timer.ko 2022-09-07T21:09:58.913 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/hw_random 2022-09-07T21:09:58.913 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/hw_random/amd-rng.ko 2022-09-07T21:09:58.913 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/hw_random/intel-rng.ko 2022-09-07T21:09:58.913 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/hw_random/timeriomem-rng.ko 2022-09-07T21:09:58.914 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/hw_random/via-rng.ko 2022-09-07T21:09:58.914 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/hw_random/virtio-rng.ko 2022-09-07T21:09:58.914 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ipmi 2022-09-07T21:09:58.914 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ipmi/ipmi_devintf.ko 2022-09-07T21:09:58.915 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2022-09-07T21:09:58.915 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2022-09-07T21:09:58.915 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ipmi/ipmi_si.ko 2022-09-07T21:09:58.915 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ipmi/ipmi_ssif.ko 2022-09-07T21:09:58.916 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2022-09-07T21:09:58.916 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/lp.ko 2022-09-07T21:09:58.916 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/mwave 2022-09-07T21:09:58.916 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/mwave/mwave.ko 2022-09-07T21:09:58.917 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/pcmcia 2022-09-07T21:09:58.917 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/pcmcia/cm4000_cs.ko 2022-09-07T21:09:58.917 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/pcmcia/cm4040_cs.ko 2022-09-07T21:09:58.917 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ppdev.ko 2022-09-07T21:09:58.917 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tlclk.ko 2022-09-07T21:09:58.918 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tpm 2022-09-07T21:09:58.918 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tpm/tpm.ko 2022-09-07T21:09:58.918 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tpm/tpm_atmel.ko 2022-09-07T21:09:58.919 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tpm/tpm_infineon.ko 2022-09-07T21:09:58.919 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tpm/tpm_nsc.ko 2022-09-07T21:09:58.919 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tpm/tpm_tis.ko 2022-09-07T21:09:58.920 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tpm/tpm_tis_core.ko 2022-09-07T21:09:58.920 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/uv_mmtimer.ko 2022-09-07T21:09:58.920 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/virtio_console.ko 2022-09-07T21:09:58.920 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/cpufreq 2022-09-07T21:09:58.921 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/cpufreq/acpi-cpufreq.ko 2022-09-07T21:09:58.921 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2022-09-07T21:09:58.921 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/cpufreq/p4-clockmod.ko 2022-09-07T21:09:58.922 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/cpufreq/pcc-cpufreq.ko 2022-09-07T21:09:58.922 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/cpufreq/powernow-k8.ko 2022-09-07T21:09:58.922 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/cpufreq/speedstep-lib.ko 2022-09-07T21:09:58.923 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto 2022-09-07T21:09:58.923 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/ccp 2022-09-07T21:09:58.923 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/ccp/ccp-crypto.ko 2022-09-07T21:09:58.923 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/ccp/ccp.ko 2022-09-07T21:09:58.924 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/padlock-aes.ko 2022-09-07T21:09:58.924 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/padlock-sha.ko 2022-09-07T21:09:58.924 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/qat 2022-09-07T21:09:58.924 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/qat/qat_common 2022-09-07T21:09:58.925 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/qat/qat_common/intel_qat.ko 2022-09-07T21:09:58.925 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/qat/qat_dh895xcc 2022-09-07T21:09:58.925 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko 2022-09-07T21:09:58.925 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/virtio 2022-09-07T21:09:58.925 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/virtio/virtio_crypto.ko 2022-09-07T21:09:58.926 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/dca 2022-09-07T21:09:58.926 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/dca/dca.ko 2022-09-07T21:09:58.926 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/devfreq 2022-09-07T21:09:58.926 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/devfreq/governor_simpleondemand.ko 2022-09-07T21:09:58.927 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/dma 2022-09-07T21:09:58.927 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/dma/dw 2022-09-07T21:09:58.927 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/dma/dw/dw_dmac.ko 2022-09-07T21:09:58.927 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/dma/ioat 2022-09-07T21:09:58.928 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/dma/ioat/ioatdma.ko 2022-09-07T21:09:58.928 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac 2022-09-07T21:09:58.928 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/amd64_edac.ko 2022-09-07T21:09:58.928 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/e752x_edac.ko 2022-09-07T21:09:58.928 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/edac_mce_amd.ko 2022-09-07T21:09:58.929 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i3000_edac.ko 2022-09-07T21:09:58.929 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i3200_edac.ko 2022-09-07T21:09:58.929 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i5000_edac.ko 2022-09-07T21:09:58.929 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i5100_edac.ko 2022-09-07T21:09:58.929 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i5400_edac.ko 2022-09-07T21:09:58.930 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i7300_edac.ko 2022-09-07T21:09:58.930 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i7core_edac.ko 2022-09-07T21:09:58.931 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i82975x_edac.ko 2022-09-07T21:09:58.931 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/ie31200_edac.ko 2022-09-07T21:09:58.931 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/sb_edac.ko 2022-09-07T21:09:58.932 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/x38_edac.ko 2022-09-07T21:09:58.932 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/firmware 2022-09-07T21:09:58.932 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/firmware/edd.ko 2022-09-07T21:09:58.933 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/firmware/iscsi_ibft.ko 2022-09-07T21:09:58.933 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/gpio 2022-09-07T21:09:58.934 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/gpio/gpio-ich.ko 2022-09-07T21:09:58.934 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/gpio/gpio-viperboard.ko 2022-09-07T21:09:58.935 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid 2022-09-07T21:09:58.935 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-appleir.ko 2022-09-07T21:09:58.935 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-aureal.ko 2022-09-07T21:09:58.936 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-axff.ko 2022-09-07T21:09:58.936 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-dr.ko 2022-09-07T21:09:58.936 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-elecom.ko 2022-09-07T21:09:58.937 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-elo.ko 2022-09-07T21:09:58.937 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-emsff.ko 2022-09-07T21:09:58.937 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-gaff.ko 2022-09-07T21:09:58.937 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-gt683r.ko 2022-09-07T21:09:58.938 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-gyration.ko 2022-09-07T21:09:58.938 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-holtek-kbd.ko 2022-09-07T21:09:58.938 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-holtek-mouse.ko 2022-09-07T21:09:58.938 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-holtekff.ko 2022-09-07T21:09:58.939 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-hyperv.ko 2022-09-07T21:09:58.939 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-icade.ko 2022-09-07T21:09:58.946 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-keytouch.ko 2022-09-07T21:09:58.947 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-kye.ko 2022-09-07T21:09:58.947 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-lcpower.ko 2022-09-07T21:09:58.947 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-led.ko 2022-09-07T21:09:58.947 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-lenovo.ko 2022-09-07T21:09:58.948 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-logitech-dj.ko 2022-09-07T21:09:58.948 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-logitech-hidpp.ko 2022-09-07T21:09:58.948 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-multitouch.ko 2022-09-07T21:09:58.948 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-ortek.ko 2022-09-07T21:09:58.949 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-penmount.ko 2022-09-07T21:09:58.949 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-petalynx.ko 2022-09-07T21:09:58.949 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-picolcd.ko 2022-09-07T21:09:58.949 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-pl.ko 2022-09-07T21:09:58.950 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-plantronics.ko 2022-09-07T21:09:58.950 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-primax.ko 2022-09-07T21:09:58.950 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-rmi.ko 2022-09-07T21:09:58.950 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-arvo.ko 2022-09-07T21:09:58.950 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-common.ko 2022-09-07T21:09:58.951 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-isku.ko 2022-09-07T21:09:58.951 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-kone.ko 2022-09-07T21:09:58.951 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-koneplus.ko 2022-09-07T21:09:58.951 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-konepure.ko 2022-09-07T21:09:58.952 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-kovaplus.ko 2022-09-07T21:09:58.952 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-lua.ko 2022-09-07T21:09:58.952 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-pyra.ko 2022-09-07T21:09:58.952 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-ryos.ko 2022-09-07T21:09:58.952 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-savu.ko 2022-09-07T21:09:58.953 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat.ko 2022-09-07T21:09:58.953 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-saitek.ko 2022-09-07T21:09:58.953 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-samsung.ko 2022-09-07T21:09:58.953 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-sensor-hub.ko 2022-09-07T21:09:58.954 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-sjoy.ko 2022-09-07T21:09:58.954 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-sony.ko 2022-09-07T21:09:58.954 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-speedlink.ko 2022-09-07T21:09:58.954 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-steelseries.ko 2022-09-07T21:09:58.955 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-sunplus.ko 2022-09-07T21:09:58.955 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-thrustmaster.ko 2022-09-07T21:09:58.955 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-tivo.ko 2022-09-07T21:09:58.955 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-tmff.ko 2022-09-07T21:09:58.955 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-topseed.ko 2022-09-07T21:09:58.956 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-twinhan.ko 2022-09-07T21:09:58.956 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-uclogic.ko 2022-09-07T21:09:58.956 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-waltop.ko 2022-09-07T21:09:58.956 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-wiimote.ko 2022-09-07T21:09:58.956 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-xinmo.ko 2022-09-07T21:09:58.957 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-zpff.ko 2022-09-07T21:09:58.957 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-zydacron.ko 2022-09-07T21:09:58.957 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/uhid.ko 2022-09-07T21:09:58.957 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/wacom.ko 2022-09-07T21:09:58.958 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hv 2022-09-07T21:09:58.958 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hv/hv_balloon.ko 2022-09-07T21:09:58.958 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hv/hv_utils.ko 2022-09-07T21:09:58.958 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hv/hv_vmbus.ko 2022-09-07T21:09:58.959 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon 2022-09-07T21:09:58.959 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/abituguru.ko 2022-09-07T21:09:58.959 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/abituguru3.ko 2022-09-07T21:09:58.959 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/acpi_power_meter.ko 2022-09-07T21:09:58.959 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ad7414.ko 2022-09-07T21:09:58.960 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ad7418.ko 2022-09-07T21:09:58.960 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adc128d818.ko 2022-09-07T21:09:58.960 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adm1025.ko 2022-09-07T21:09:58.960 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adm1026.ko 2022-09-07T21:09:58.960 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adm1029.ko 2022-09-07T21:09:58.961 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adm1031.ko 2022-09-07T21:09:58.961 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adm9240.ko 2022-09-07T21:09:58.961 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ads7828.ko 2022-09-07T21:09:58.961 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adt7410.ko 2022-09-07T21:09:58.962 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adt7411.ko 2022-09-07T21:09:58.962 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adt7462.ko 2022-09-07T21:09:58.962 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adt7470.ko 2022-09-07T21:09:58.962 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adt7475.ko 2022-09-07T21:09:58.962 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adt7x10.ko 2022-09-07T21:09:58.963 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/amc6821.ko 2022-09-07T21:09:58.963 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/applesmc.ko 2022-09-07T21:09:58.963 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/asb100.ko 2022-09-07T21:09:58.963 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/asc7621.ko 2022-09-07T21:09:58.964 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/asus_atk0110.ko 2022-09-07T21:09:58.964 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/atxp1.ko 2022-09-07T21:09:58.964 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/coretemp.ko 2022-09-07T21:09:58.964 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/dell-smm-hwmon.ko 2022-09-07T21:09:58.964 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/dme1737.ko 2022-09-07T21:09:58.965 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ds1621.ko 2022-09-07T21:09:58.965 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ds620.ko 2022-09-07T21:09:58.965 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/emc1403.ko 2022-09-07T21:09:58.965 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/emc6w201.ko 2022-09-07T21:09:58.966 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/f71805f.ko 2022-09-07T21:09:58.966 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/f71882fg.ko 2022-09-07T21:09:58.966 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/f75375s.ko 2022-09-07T21:09:58.966 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/fam15h_power.ko 2022-09-07T21:09:58.966 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/fschmd.ko 2022-09-07T21:09:58.967 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/g760a.ko 2022-09-07T21:09:58.967 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/g762.ko 2022-09-07T21:09:58.967 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/gl518sm.ko 2022-09-07T21:09:58.967 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/gl520sm.ko 2022-09-07T21:09:58.968 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/hwmon-vid.ko 2022-09-07T21:09:58.968 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/i5500_temp.ko 2022-09-07T21:09:58.968 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/i5k_amb.ko 2022-09-07T21:09:58.968 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ibmaem.ko 2022-09-07T21:09:58.968 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ibmpex.ko 2022-09-07T21:09:58.969 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ina209.ko 2022-09-07T21:09:58.969 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ina2xx.ko 2022-09-07T21:09:58.969 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/it87.ko 2022-09-07T21:09:58.969 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/k10temp.ko 2022-09-07T21:09:58.969 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/k8temp.ko 2022-09-07T21:09:58.970 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lineage-pem.ko 2022-09-07T21:09:58.970 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm63.ko 2022-09-07T21:09:58.970 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm73.ko 2022-09-07T21:09:58.970 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm75.ko 2022-09-07T21:09:58.971 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm77.ko 2022-09-07T21:09:58.971 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm78.ko 2022-09-07T21:09:58.971 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm80.ko 2022-09-07T21:09:58.971 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm83.ko 2022-09-07T21:09:58.971 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm85.ko 2022-09-07T21:09:58.972 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm87.ko 2022-09-07T21:09:58.972 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm90.ko 2022-09-07T21:09:58.972 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm92.ko 2022-09-07T21:09:58.972 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm93.ko 2022-09-07T21:09:58.973 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm95234.ko 2022-09-07T21:09:58.973 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm95241.ko 2022-09-07T21:09:58.973 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm95245.ko 2022-09-07T21:09:58.973 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ltc2945.ko 2022-09-07T21:09:58.974 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ltc4151.ko 2022-09-07T21:09:58.974 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ltc4215.ko 2022-09-07T21:09:58.974 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ltc4222.ko 2022-09-07T21:09:58.974 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ltc4245.ko 2022-09-07T21:09:58.974 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ltc4260.ko 2022-09-07T21:09:58.975 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ltc4261.ko 2022-09-07T21:09:58.975 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/max16065.ko 2022-09-07T21:09:58.975 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/max1619.ko 2022-09-07T21:09:58.975 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/max1668.ko 2022-09-07T21:09:58.976 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/max197.ko 2022-09-07T21:09:58.976 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/max6639.ko 2022-09-07T21:09:58.976 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/max6650.ko 2022-09-07T21:09:58.976 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/max6697.ko 2022-09-07T21:09:58.977 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/mcp3021.ko 2022-09-07T21:09:58.977 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/nct6683.ko 2022-09-07T21:09:58.977 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/nct6775-core.ko 2022-09-07T21:09:58.977 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/nct6775.ko 2022-09-07T21:09:58.977 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/nct7802.ko 2022-09-07T21:09:58.978 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pc87360.ko 2022-09-07T21:09:58.978 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pc87427.ko 2022-09-07T21:09:58.978 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pcf8591.ko 2022-09-07T21:09:58.978 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus 2022-09-07T21:09:58.979 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/adm1275.ko 2022-09-07T21:09:58.979 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/lm25066.ko 2022-09-07T21:09:58.979 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/ltc2978.ko 2022-09-07T21:09:58.979 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/max16064.ko 2022-09-07T21:09:58.979 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/max34440.ko 2022-09-07T21:09:58.980 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/max8688.ko 2022-09-07T21:09:58.980 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/pmbus.ko 2022-09-07T21:09:58.980 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2022-09-07T21:09:58.981 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/tps40422.ko 2022-09-07T21:09:58.981 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/ucd9000.ko 2022-09-07T21:09:58.981 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/ucd9200.ko 2022-09-07T21:09:58.981 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/zl6100.ko 2022-09-07T21:09:58.981 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/powr1220.ko 2022-09-07T21:09:58.982 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/sch5627.ko 2022-09-07T21:09:58.982 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/sch5636.ko 2022-09-07T21:09:58.982 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/sch56xx-common.ko 2022-09-07T21:09:58.982 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/sht15.ko 2022-09-07T21:09:58.983 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/sht21.ko 2022-09-07T21:09:58.983 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/shtc1.ko 2022-09-07T21:09:58.983 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/sis5595.ko 2022-09-07T21:09:58.983 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/smsc47b397.ko 2022-09-07T21:09:58.983 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/smsc47m1.ko 2022-09-07T21:09:58.984 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/smsc47m192.ko 2022-09-07T21:09:58.984 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/thmc50.ko 2022-09-07T21:09:58.984 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/tmp102.ko 2022-09-07T21:09:58.984 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/tmp103.ko 2022-09-07T21:09:58.985 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/tmp401.ko 2022-09-07T21:09:58.985 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/tmp421.ko 2022-09-07T21:09:58.985 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/via-cputemp.ko 2022-09-07T21:09:58.985 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/via686a.ko 2022-09-07T21:09:58.986 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/vt1211.ko 2022-09-07T21:09:58.986 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/vt8231.ko 2022-09-07T21:09:58.986 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83627ehf.ko 2022-09-07T21:09:58.986 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83627hf.ko 2022-09-07T21:09:58.986 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83781d.ko 2022-09-07T21:09:58.987 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83791d.ko 2022-09-07T21:09:58.987 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83792d.ko 2022-09-07T21:09:58.987 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83793.ko 2022-09-07T21:09:58.987 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83795.ko 2022-09-07T21:09:58.988 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83l785ts.ko 2022-09-07T21:09:58.988 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83l786ng.ko 2022-09-07T21:09:58.988 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c 2022-09-07T21:09:58.988 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/algos 2022-09-07T21:09:58.988 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2022-09-07T21:09:58.989 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2022-09-07T21:09:58.989 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses 2022-09-07T21:09:58.989 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2022-09-07T21:09:58.989 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-amd756.ko 2022-09-07T21:09:58.990 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-amd8111.ko 2022-09-07T21:09:58.990 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2022-09-07T21:09:58.990 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-designware-core.ko 2022-09-07T21:09:58.990 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2022-09-07T21:09:58.990 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2022-09-07T21:09:58.991 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2022-09-07T21:09:58.991 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-i801.ko 2022-09-07T21:09:58.991 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-isch.ko 2022-09-07T21:09:58.991 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-ismt.ko 2022-09-07T21:09:58.992 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2022-09-07T21:09:58.992 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-nforce2.ko 2022-09-07T21:09:58.992 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-parport.ko 2022-09-07T21:09:58.992 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2022-09-07T21:09:58.993 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-piix4.ko 2022-09-07T21:09:58.993 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-scmi.ko 2022-09-07T21:09:58.993 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-simtec.ko 2022-09-07T21:09:58.993 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-sis96x.ko 2022-09-07T21:09:58.993 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2022-09-07T21:09:58.994 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-via.ko 2022-09-07T21:09:58.994 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-viapro.ko 2022-09-07T21:09:58.994 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-viperboard.ko 2022-09-07T21:09:58.994 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/i2c-dev.ko 2022-09-07T21:09:58.995 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/i2c-mux.ko 2022-09-07T21:09:58.995 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/i2c-slave-eeprom.ko 2022-09-07T21:09:58.995 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/i2c-smbus.ko 2022-09-07T21:09:58.995 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/i2c-stub.ko 2022-09-07T21:09:58.995 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband 2022-09-07T21:09:58.996 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core 2022-09-07T21:09:58.996 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core/ib_cm.ko 2022-09-07T21:09:58.996 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core/ib_core.ko 2022-09-07T21:09:58.996 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core/ib_umad.ko 2022-09-07T21:09:58.996 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core/ib_uverbs.ko 2022-09-07T21:09:58.997 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core/iw_cm.ko 2022-09-07T21:09:58.997 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core/rdma_cm.ko 2022-09-07T21:09:58.997 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core/rdma_ucm.ko 2022-09-07T21:09:58.997 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw 2022-09-07T21:09:58.998 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/cxgb4 2022-09-07T21:09:58.998 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2022-09-07T21:09:58.998 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/mlx4 2022-09-07T21:09:58.998 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2022-09-07T21:09:58.998 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/mlx5 2022-09-07T21:09:58.999 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2022-09-07T21:09:58.999 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/mthca 2022-09-07T21:09:58.999 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2022-09-07T21:09:58.999 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp 2022-09-07T21:09:59.000 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/ipoib 2022-09-07T21:09:59.000 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2022-09-07T21:09:59.000 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/iser 2022-09-07T21:09:59.000 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2022-09-07T21:09:59.000 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/isert 2022-09-07T21:09:59.001 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2022-09-07T21:09:59.001 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/srp 2022-09-07T21:09:59.001 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2022-09-07T21:09:59.001 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/srpt 2022-09-07T21:09:59.002 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2022-09-07T21:09:59.002 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input 2022-09-07T21:09:59.002 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/gameport 2022-09-07T21:09:59.002 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/gameport/emu10k1-gp.ko 2022-09-07T21:09:59.002 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/gameport/fm801-gp.ko 2022-09-07T21:09:59.003 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/gameport/gameport.ko 2022-09-07T21:09:59.003 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/gameport/lightning.ko 2022-09-07T21:09:59.003 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/gameport/ns558.ko 2022-09-07T21:09:59.003 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/joydev.ko 2022-09-07T21:09:59.004 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/keyboard 2022-09-07T21:09:59.004 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/keyboard/gpio_keys.ko 2022-09-07T21:09:59.004 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse 2022-09-07T21:09:59.004 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/appletouch.ko 2022-09-07T21:09:59.004 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/bcm5974.ko 2022-09-07T21:09:59.005 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/cyapatp.ko 2022-09-07T21:09:59.005 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/elan_i2c.ko 2022-09-07T21:09:59.005 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/sermouse.ko 2022-09-07T21:09:59.005 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/synaptics_i2c.ko 2022-09-07T21:09:59.006 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/synaptics_usb.ko 2022-09-07T21:09:59.006 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/vsxxxaa.ko 2022-09-07T21:09:59.006 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/rmi4 2022-09-07T21:09:59.006 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/rmi4/rmi_core.ko 2022-09-07T21:09:59.006 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/serio 2022-09-07T21:09:59.007 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/serio/altera_ps2.ko 2022-09-07T21:09:59.007 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/serio/arc_ps2.ko 2022-09-07T21:09:59.007 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/serio/hyperv-keyboard.ko 2022-09-07T21:09:59.007 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/serio/serio_raw.ko 2022-09-07T21:09:59.008 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/sparse-keymap.ko 2022-09-07T21:09:59.008 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/iommu 2022-09-07T21:09:59.008 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/iommu/amd 2022-09-07T21:09:59.008 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/iommu/amd/iommu_v2.ko 2022-09-07T21:09:59.008 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds 2022-09-07T21:09:59.009 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/leds-blinkm.ko 2022-09-07T21:09:59.009 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/leds-lm3530.ko 2022-09-07T21:09:59.009 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/leds-lp3944.ko 2022-09-07T21:09:59.009 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/leds-ss4200.ko 2022-09-07T21:09:59.010 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger 2022-09-07T21:09:59.010 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-audio.ko 2022-09-07T21:09:59.010 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2022-09-07T21:09:59.010 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-camera.ko 2022-09-07T21:09:59.010 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2022-09-07T21:09:59.011 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2022-09-07T21:09:59.011 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2022-09-07T21:09:59.011 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2022-09-07T21:09:59.011 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-timer.ko 2022-09-07T21:09:59.012 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-transient.ko 2022-09-07T21:09:59.012 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md 2022-09-07T21:09:59.012 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/bcache 2022-09-07T21:09:59.012 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/bcache/bcache.ko 2022-09-07T21:09:59.012 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-bio-prison.ko 2022-09-07T21:09:59.013 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-cache-smq.ko 2022-09-07T21:09:59.013 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-cache.ko 2022-09-07T21:09:59.013 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-crypt.ko 2022-09-07T21:09:59.013 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-delay.ko 2022-09-07T21:09:59.013 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-flakey.ko 2022-09-07T21:09:59.014 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-log-userspace.ko 2022-09-07T21:09:59.014 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-multipath.ko 2022-09-07T21:09:59.014 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-queue-length.ko 2022-09-07T21:09:59.014 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-raid.ko 2022-09-07T21:09:59.015 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-round-robin.ko 2022-09-07T21:09:59.015 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-service-time.ko 2022-09-07T21:09:59.015 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-switch.ko 2022-09-07T21:09:59.015 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-thin-pool.ko 2022-09-07T21:09:59.015 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-verity.ko 2022-09-07T21:09:59.016 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/faulty.ko 2022-09-07T21:09:59.016 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/linear.ko 2022-09-07T21:09:59.016 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/multipath.ko 2022-09-07T21:09:59.016 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/persistent-data 2022-09-07T21:09:59.017 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2022-09-07T21:09:59.017 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/raid0.ko 2022-09-07T21:09:59.017 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/raid1.ko 2022-09-07T21:09:59.017 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/raid10.ko 2022-09-07T21:09:59.017 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/raid456.ko 2022-09-07T21:09:59.018 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message 2022-09-07T21:09:59.018 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion 2022-09-07T21:09:59.018 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion/mptbase.ko 2022-09-07T21:09:59.018 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion/mptctl.ko 2022-09-07T21:09:59.019 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion/mptfc.ko 2022-09-07T21:09:59.019 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion/mptlan.ko 2022-09-07T21:09:59.019 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion/mptsas.ko 2022-09-07T21:09:59.019 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion/mptscsih.ko 2022-09-07T21:09:59.019 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion/mptspi.ko 2022-09-07T21:09:59.020 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd 2022-09-07T21:09:59.020 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd/lpc_ich.ko 2022-09-07T21:09:59.020 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd/lpc_sch.ko 2022-09-07T21:09:59.020 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd/mfd-core.ko 2022-09-07T21:09:59.021 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd/sm501.ko 2022-09-07T21:09:59.021 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd/viperboard.ko 2022-09-07T21:09:59.021 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd/vx855.ko 2022-09-07T21:09:59.021 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd/wl1273-core.ko 2022-09-07T21:09:59.021 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc 2022-09-07T21:09:59.022 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/altera-stapl 2022-09-07T21:09:59.022 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2022-09-07T21:09:59.022 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/apds9802als.ko 2022-09-07T21:09:59.022 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/apds990x.ko 2022-09-07T21:09:59.022 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/bh1770glc.ko 2022-09-07T21:09:59.023 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/cb710 2022-09-07T21:09:59.023 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/cb710/cb710.ko 2022-09-07T21:09:59.023 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/eeprom 2022-09-07T21:09:59.023 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/eeprom/at24.ko 2022-09-07T21:09:59.024 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/eeprom/eeprom.ko 2022-09-07T21:09:59.024 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2022-09-07T21:09:59.024 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/eeprom/max6875.ko 2022-09-07T21:09:59.024 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/enclosure.ko 2022-09-07T21:09:59.024 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/hpilo.ko 2022-09-07T21:09:59.025 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/isl29003.ko 2022-09-07T21:09:59.025 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/isl29020.ko 2022-09-07T21:09:59.025 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/lis3lv02d 2022-09-07T21:09:59.025 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2022-09-07T21:09:59.026 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2022-09-07T21:09:59.026 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/mei 2022-09-07T21:09:59.026 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/mei/mei-me.ko 2022-09-07T21:09:59.026 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/mei/mei-txe.ko 2022-09-07T21:09:59.026 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/mei/mei.ko 2022-09-07T21:09:59.027 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/sgi-gru 2022-09-07T21:09:59.027 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/sgi-gru/gru.ko 2022-09-07T21:09:59.027 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/sgi-xp 2022-09-07T21:09:59.027 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/sgi-xp/xp.ko 2022-09-07T21:09:59.028 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/sgi-xp/xpc.ko 2022-09-07T21:09:59.028 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/sgi-xp/xpnet.ko 2022-09-07T21:09:59.028 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/tifm_7xx1.ko 2022-09-07T21:09:59.028 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/tifm_core.ko 2022-09-07T21:09:59.028 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/tsl2550.ko 2022-09-07T21:09:59.029 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/vmw_balloon.ko 2022-09-07T21:09:59.029 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/vmw_vmci 2022-09-07T21:09:59.029 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2022-09-07T21:09:59.029 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net 2022-09-07T21:09:59.030 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/bonding 2022-09-07T21:09:59.030 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/bonding/bonding.ko 2022-09-07T21:09:59.031 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dsa 2022-09-07T21:09:59.031 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dsa/b53 2022-09-07T21:09:59.031 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dsa/b53/b53_common.ko 2022-09-07T21:09:59.032 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dsa/bcm-sf2.ko 2022-09-07T21:09:59.032 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dsa/mv88e6060.ko 2022-09-07T21:09:59.032 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dsa/mv88e6xxx 2022-09-07T21:09:59.032 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2022-09-07T21:09:59.032 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dummy.ko 2022-09-07T21:09:59.033 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/eql.ko 2022-09-07T21:09:59.033 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet 2022-09-07T21:09:59.033 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/3com 2022-09-07T21:09:59.033 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2022-09-07T21:09:59.033 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2022-09-07T21:09:59.034 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/3com/3c59x.ko 2022-09-07T21:09:59.034 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/3com/typhoon.ko 2022-09-07T21:09:59.034 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/8390 2022-09-07T21:09:59.034 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/8390/8390.ko 2022-09-07T21:09:59.035 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2022-09-07T21:09:59.035 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2022-09-07T21:09:59.035 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2022-09-07T21:09:59.035 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/adaptec 2022-09-07T21:09:59.036 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/adaptec/starfire.ko 2022-09-07T21:09:59.036 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/agere 2022-09-07T21:09:59.036 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/agere/et131x.ko 2022-09-07T21:09:59.036 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/alteon 2022-09-07T21:09:59.036 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/alteon/acenic.ko 2022-09-07T21:09:59.037 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/altera 2022-09-07T21:09:59.037 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/altera/altera_tse.ko 2022-09-07T21:09:59.037 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/amd 2022-09-07T21:09:59.037 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/amd/amd8111e.ko 2022-09-07T21:09:59.038 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2022-09-07T21:09:59.038 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/amd/pcnet32.ko 2022-09-07T21:09:59.038 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros 2022-09-07T21:09:59.038 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/alx 2022-09-07T21:09:59.038 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2022-09-07T21:09:59.039 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/atl1c 2022-09-07T21:09:59.039 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2022-09-07T21:09:59.039 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/atl1e 2022-09-07T21:09:59.039 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2022-09-07T21:09:59.039 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/atlx 2022-09-07T21:09:59.040 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2022-09-07T21:09:59.040 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2022-09-07T21:09:59.040 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom 2022-09-07T21:09:59.040 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/b44.ko 2022-09-07T21:09:59.041 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2022-09-07T21:09:59.041 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/bnx2x 2022-09-07T21:09:59.041 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2022-09-07T21:09:59.041 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/cnic.ko 2022-09-07T21:09:59.041 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/genet 2022-09-07T21:09:59.042 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2022-09-07T21:09:59.042 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/tg3.ko 2022-09-07T21:09:59.042 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/brocade 2022-09-07T21:09:59.042 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/brocade/bna 2022-09-07T21:09:59.043 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2022-09-07T21:09:59.043 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio 2022-09-07T21:09:59.043 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb 2022-09-07T21:09:59.043 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2022-09-07T21:09:59.044 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb3 2022-09-07T21:09:59.044 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2022-09-07T21:09:59.044 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb4 2022-09-07T21:09:59.044 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2022-09-07T21:09:59.044 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2022-09-07T21:09:59.045 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2022-09-07T21:09:59.045 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/libcxgb 2022-09-07T21:09:59.045 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2022-09-07T21:09:59.045 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/cisco 2022-09-07T21:09:59.046 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/cisco/enic 2022-09-07T21:09:59.046 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2022-09-07T21:09:59.046 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec 2022-09-07T21:09:59.046 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec/tulip 2022-09-07T21:09:59.046 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2022-09-07T21:09:59.047 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2022-09-07T21:09:59.047 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2022-09-07T21:09:59.047 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2022-09-07T21:09:59.047 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2022-09-07T21:09:59.048 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2022-09-07T21:09:59.048 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dlink 2022-09-07T21:09:59.048 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dlink/dl2k.ko 2022-09-07T21:09:59.048 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dlink/sundance.ko 2022-09-07T21:09:59.048 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dnet.ko 2022-09-07T21:09:59.049 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/emulex 2022-09-07T21:09:59.049 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/emulex/benet 2022-09-07T21:09:59.049 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2022-09-07T21:09:59.049 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/ethoc.ko 2022-09-07T21:09:59.050 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/fealnx.ko 2022-09-07T21:09:59.050 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel 2022-09-07T21:09:59.050 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/e100.ko 2022-09-07T21:09:59.050 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/e1000 2022-09-07T21:09:59.050 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2022-09-07T21:09:59.051 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/e1000e 2022-09-07T21:09:59.051 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2022-09-07T21:09:59.051 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/fm10k 2022-09-07T21:09:59.051 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2022-09-07T21:09:59.052 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/i40e 2022-09-07T21:09:59.052 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2022-09-07T21:09:59.052 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/iavf 2022-09-07T21:09:59.052 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2022-09-07T21:09:59.052 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/igb 2022-09-07T21:09:59.053 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/igb/igb.ko 2022-09-07T21:09:59.053 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/igbvf 2022-09-07T21:09:59.053 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2022-09-07T21:09:59.053 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/ixgb 2022-09-07T21:09:59.054 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/ixgb/ixgb.ko 2022-09-07T21:09:59.054 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/ixgbe 2022-09-07T21:09:59.054 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2022-09-07T21:09:59.054 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/ixgbevf 2022-09-07T21:09:59.055 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2022-09-07T21:09:59.055 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/jme.ko 2022-09-07T21:09:59.055 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/marvell 2022-09-07T21:09:59.055 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2022-09-07T21:09:59.056 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/marvell/skge.ko 2022-09-07T21:09:59.056 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/marvell/sky2.ko 2022-09-07T21:09:59.056 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/mellanox 2022-09-07T21:09:59.056 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/mellanox/mlx4 2022-09-07T21:09:59.056 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2022-09-07T21:09:59.057 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2022-09-07T21:09:59.057 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/mellanox/mlx5 2022-09-07T21:09:59.057 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/mellanox/mlx5/core 2022-09-07T21:09:59.057 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2022-09-07T21:09:59.058 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/micrel 2022-09-07T21:09:59.058 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2022-09-07T21:09:59.058 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/myricom 2022-09-07T21:09:59.058 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/myricom/myri10ge 2022-09-07T21:09:59.059 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2022-09-07T21:09:59.059 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/natsemi 2022-09-07T21:09:59.059 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2022-09-07T21:09:59.059 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2022-09-07T21:09:59.059 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/neterion 2022-09-07T21:09:59.060 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/neterion/s2io.ko 2022-09-07T21:09:59.060 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/nvidia 2022-09-07T21:09:59.060 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2022-09-07T21:09:59.060 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/packetengines 2022-09-07T21:09:59.061 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2022-09-07T21:09:59.061 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2022-09-07T21:09:59.061 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/qlogic 2022-09-07T21:09:59.061 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/qlogic/netxen 2022-09-07T21:09:59.061 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2022-09-07T21:09:59.062 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2022-09-07T21:09:59.062 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/qlogic/qlcnic 2022-09-07T21:09:59.062 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2022-09-07T21:09:59.062 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/rdc 2022-09-07T21:09:59.063 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/rdc/r6040.ko 2022-09-07T21:09:59.063 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/realtek 2022-09-07T21:09:59.063 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/realtek/8139cp.ko 2022-09-07T21:09:59.063 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/realtek/8139too.ko 2022-09-07T21:09:59.063 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/realtek/atp.ko 2022-09-07T21:09:59.064 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/realtek/r8169.ko 2022-09-07T21:09:59.064 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/rocker 2022-09-07T21:09:59.064 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/rocker/rocker.ko 2022-09-07T21:09:59.064 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sfc 2022-09-07T21:09:59.065 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sfc/sfc.ko 2022-09-07T21:09:59.065 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/silan 2022-09-07T21:09:59.065 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/silan/sc92031.ko 2022-09-07T21:09:59.065 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sis 2022-09-07T21:09:59.065 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sis/sis190.ko 2022-09-07T21:09:59.066 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sis/sis900.ko 2022-09-07T21:09:59.066 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/smsc 2022-09-07T21:09:59.066 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/smsc/epic100.ko 2022-09-07T21:09:59.066 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2022-09-07T21:09:59.067 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2022-09-07T21:09:59.067 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2022-09-07T21:09:59.067 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/stmicro 2022-09-07T21:09:59.067 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/stmicro/stmmac 2022-09-07T21:09:59.067 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2022-09-07T21:09:59.068 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2022-09-07T21:09:59.068 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sun 2022-09-07T21:09:59.068 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sun/cassini.ko 2022-09-07T21:09:59.068 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sun/niu.ko 2022-09-07T21:09:59.069 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sun/sungem.ko 2022-09-07T21:09:59.069 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sun/sunhme.ko 2022-09-07T21:09:59.069 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/tehuti 2022-09-07T21:09:59.069 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2022-09-07T21:09:59.069 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/ti 2022-09-07T21:09:59.070 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/ti/tlan.ko 2022-09-07T21:09:59.070 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/via 2022-09-07T21:09:59.070 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/via/via-rhine.ko 2022-09-07T21:09:59.070 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/via/via-velocity.ko 2022-09-07T21:09:59.070 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/wiznet 2022-09-07T21:09:59.071 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/wiznet/w5100.ko 2022-09-07T21:09:59.071 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/wiznet/w5300.ko 2022-09-07T21:09:59.071 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/xircom 2022-09-07T21:09:59.071 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2022-09-07T21:09:59.072 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/geneve.ko 2022-09-07T21:09:59.072 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/hyperv 2022-09-07T21:09:59.072 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/hyperv/hv_netvsc.ko 2022-09-07T21:09:59.072 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ieee802154 2022-09-07T21:09:59.072 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ieee802154/fakelb.ko 2022-09-07T21:09:59.073 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ifb.ko 2022-09-07T21:09:59.073 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ipvlan 2022-09-07T21:09:59.073 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ipvlan/ipvlan.ko 2022-09-07T21:09:59.073 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/macvlan.ko 2022-09-07T21:09:59.074 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/macvtap.ko 2022-09-07T21:09:59.074 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/mdio 2022-09-07T21:09:59.074 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/mdio.ko 2022-09-07T21:09:59.074 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2022-09-07T21:09:59.074 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/mdio/mdio-bitbang.ko 2022-09-07T21:09:59.075 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/mii.ko 2022-09-07T21:09:59.075 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/net_failover.ko 2022-09-07T21:09:59.075 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/netconsole.ko 2022-09-07T21:09:59.075 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/nlmon.ko 2022-09-07T21:09:59.076 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/pcs 2022-09-07T21:09:59.076 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/pcs/pcs_xpcs.ko 2022-09-07T21:09:59.076 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy 2022-09-07T21:09:59.076 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/amd.ko 2022-09-07T21:09:59.076 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/bcm-phy-lib.ko 2022-09-07T21:09:59.077 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/bcm-phy-ptp.ko 2022-09-07T21:09:59.077 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/bcm7xxx.ko 2022-09-07T21:09:59.077 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/bcm87xx.ko 2022-09-07T21:09:59.077 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/broadcom.ko 2022-09-07T21:09:59.078 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/cicada.ko 2022-09-07T21:09:59.078 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/davicom.ko 2022-09-07T21:09:59.078 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/dp83640.ko 2022-09-07T21:09:59.078 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/et1011c.ko 2022-09-07T21:09:59.078 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/icplus.ko 2022-09-07T21:09:59.079 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/lxt.ko 2022-09-07T21:09:59.079 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/marvell.ko 2022-09-07T21:09:59.079 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/micrel.ko 2022-09-07T21:09:59.079 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/national.ko 2022-09-07T21:09:59.080 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/phylink.ko 2022-09-07T21:09:59.080 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/qsemi.ko 2022-09-07T21:09:59.080 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/realtek.ko 2022-09-07T21:09:59.080 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/smsc.ko 2022-09-07T21:09:59.081 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/ste10Xp.ko 2022-09-07T21:09:59.081 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/vitesse.ko 2022-09-07T21:09:59.081 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp 2022-09-07T21:09:59.081 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/bsd_comp.ko 2022-09-07T21:09:59.081 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/ppp_async.ko 2022-09-07T21:09:59.082 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/ppp_deflate.ko 2022-09-07T21:09:59.082 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/ppp_generic.ko 2022-09-07T21:09:59.082 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/ppp_mppe.ko 2022-09-07T21:09:59.082 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/ppp_synctty.ko 2022-09-07T21:09:59.083 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/pppoe.ko 2022-09-07T21:09:59.083 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/pppox.ko 2022-09-07T21:09:59.083 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/pptp.ko 2022-09-07T21:09:59.083 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/slip 2022-09-07T21:09:59.083 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/slip/slhc.ko 2022-09-07T21:09:59.084 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/slip/slip.ko 2022-09-07T21:09:59.084 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/sungem_phy.ko 2022-09-07T21:09:59.084 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/tap.ko 2022-09-07T21:09:59.084 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/team 2022-09-07T21:09:59.085 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/team/team.ko 2022-09-07T21:09:59.085 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/team/team_mode_activebackup.ko 2022-09-07T21:09:59.085 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/team/team_mode_broadcast.ko 2022-09-07T21:09:59.085 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/team/team_mode_loadbalance.ko 2022-09-07T21:09:59.085 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/team/team_mode_random.ko 2022-09-07T21:09:59.086 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/team/team_mode_roundrobin.ko 2022-09-07T21:09:59.086 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/tun.ko 2022-09-07T21:09:59.086 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/veth.ko 2022-09-07T21:09:59.086 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/virtio_net.ko 2022-09-07T21:09:59.087 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/vmxnet3 2022-09-07T21:09:59.087 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/vmxnet3/vmxnet3.ko 2022-09-07T21:09:59.087 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/vxlan 2022-09-07T21:09:59.087 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/vxlan/vxlan.ko 2022-09-07T21:09:59.087 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/xen-netback 2022-09-07T21:09:59.088 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/xen-netback/xen-netback.ko 2022-09-07T21:09:59.088 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/xen-netfront.ko 2022-09-07T21:09:59.088 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ntb 2022-09-07T21:09:59.088 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ntb/ntb.ko 2022-09-07T21:09:59.089 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/nvme 2022-09-07T21:09:59.089 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/nvme/host 2022-09-07T21:09:59.089 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/nvme/host/nvme-core.ko 2022-09-07T21:09:59.089 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/nvme/host/nvme.ko 2022-09-07T21:09:59.089 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/parport 2022-09-07T21:09:59.090 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/parport/parport.ko 2022-09-07T21:09:59.090 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/parport/parport_cs.ko 2022-09-07T21:09:59.090 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/parport/parport_pc.ko 2022-09-07T21:09:59.090 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/parport/parport_serial.ko 2022-09-07T21:09:59.091 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pci 2022-09-07T21:09:59.091 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pci/hotplug 2022-09-07T21:09:59.091 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2022-09-07T21:09:59.091 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pci/pcie 2022-09-07T21:09:59.091 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pci/pcie/aer_inject.ko 2022-09-07T21:09:59.092 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pci/xen-pcifront.ko 2022-09-07T21:09:59.092 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pcmcia 2022-09-07T21:09:59.092 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pcmcia/i82092.ko 2022-09-07T21:09:59.092 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pcmcia/pd6729.ko 2022-09-07T21:09:59.093 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pcmcia/yenta_socket.ko 2022-09-07T21:09:59.093 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pinctrl 2022-09-07T21:09:59.093 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pinctrl/intel 2022-09-07T21:09:59.093 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2022-09-07T21:09:59.093 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform 2022-09-07T21:09:59.094 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86 2022-09-07T21:09:59.094 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/acer-wmi.ko 2022-09-07T21:09:59.094 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/acerhdf.ko 2022-09-07T21:09:59.094 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/apple-gmux.ko 2022-09-07T21:09:59.095 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/asus-laptop.ko 2022-09-07T21:09:59.095 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/asus-nb-wmi.ko 2022-09-07T21:09:59.095 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/asus-wmi.ko 2022-09-07T21:09:59.095 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/classmate-laptop.ko 2022-09-07T21:09:59.095 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/eeepc-laptop.ko 2022-09-07T21:09:59.096 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/eeepc-wmi.ko 2022-09-07T21:09:59.096 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/fujitsu-laptop.ko 2022-09-07T21:09:59.096 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/fujitsu-tablet.ko 2022-09-07T21:09:59.096 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/hdaps.ko 2022-09-07T21:09:59.097 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/hp-wmi.ko 2022-09-07T21:09:59.097 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/hp_accel.ko 2022-09-07T21:09:59.097 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/intel 2022-09-07T21:09:59.097 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/intel/intel-rst.ko 2022-09-07T21:09:59.097 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/intel_ips.ko 2022-09-07T21:09:59.098 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/msi-wmi.ko 2022-09-07T21:09:59.098 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/mxm-wmi.ko 2022-09-07T21:09:59.098 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/panasonic-laptop.ko 2022-09-07T21:09:59.098 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/samsung-laptop.ko 2022-09-07T21:09:59.099 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/samsung-q10.ko 2022-09-07T21:09:59.099 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/topstar-laptop.ko 2022-09-07T21:09:59.099 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2022-09-07T21:09:59.099 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/toshiba_haps.ko 2022-09-07T21:09:59.099 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/wmi-bmof.ko 2022-09-07T21:09:59.100 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/wmi.ko 2022-09-07T21:09:59.100 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/powercap 2022-09-07T21:09:59.100 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/powercap/intel_rapl_common.ko 2022-09-07T21:09:59.100 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/powercap/intel_rapl_msr.ko 2022-09-07T21:09:59.101 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pps 2022-09-07T21:09:59.101 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pps/clients 2022-09-07T21:09:59.101 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pps/clients/pps-gpio.ko 2022-09-07T21:09:59.101 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pps/clients/pps-ldisc.ko 2022-09-07T21:09:59.101 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pps/clients/pps_parport.ko 2022-09-07T21:09:59.102 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pps/pps_core.ko 2022-09-07T21:09:59.102 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ptp 2022-09-07T21:09:59.102 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ptp/ptp.ko 2022-09-07T21:09:59.102 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ptp/ptp_kvm.ko 2022-09-07T21:09:59.103 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc 2022-09-07T21:09:59.103 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-bq32k.ko 2022-09-07T21:09:59.103 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-bq4802.ko 2022-09-07T21:09:59.103 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds1286.ko 2022-09-07T21:09:59.103 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds1307.ko 2022-09-07T21:09:59.104 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds1374.ko 2022-09-07T21:09:59.104 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds1511.ko 2022-09-07T21:09:59.104 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds1553.ko 2022-09-07T21:09:59.104 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds1672.ko 2022-09-07T21:09:59.105 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds1742.ko 2022-09-07T21:09:59.105 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds2404.ko 2022-09-07T21:09:59.105 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds3232.ko 2022-09-07T21:09:59.105 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-em3027.ko 2022-09-07T21:09:59.105 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-fm3130.ko 2022-09-07T21:09:59.106 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-isl12022.ko 2022-09-07T21:09:59.106 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-isl1208.ko 2022-09-07T21:09:59.106 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-m41t80.ko 2022-09-07T21:09:59.106 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-m48t35.ko 2022-09-07T21:09:59.107 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-m48t59.ko 2022-09-07T21:09:59.107 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-max6900.ko 2022-09-07T21:09:59.107 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-msm6242.ko 2022-09-07T21:09:59.107 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-pcf2127.ko 2022-09-07T21:09:59.107 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-pcf85063.ko 2022-09-07T21:09:59.108 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-pcf8523.ko 2022-09-07T21:09:59.108 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-pcf8563.ko 2022-09-07T21:09:59.108 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-pcf8583.ko 2022-09-07T21:09:59.108 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-rp5c01.ko 2022-09-07T21:09:59.109 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-rs5c372.ko 2022-09-07T21:09:59.109 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-rv3029c2.ko 2022-09-07T21:09:59.109 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-rx8025.ko 2022-09-07T21:09:59.109 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-rx8581.ko 2022-09-07T21:09:59.109 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-stk17ta8.ko 2022-09-07T21:09:59.110 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-v3020.ko 2022-09-07T21:09:59.110 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-x1205.ko 2022-09-07T21:09:59.110 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi 2022-09-07T21:09:59.110 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/3w-9xxx.ko 2022-09-07T21:09:59.111 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/3w-sas.ko 2022-09-07T21:09:59.111 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/3w-xxxx.ko 2022-09-07T21:09:59.111 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/BusLogic.ko 2022-09-07T21:09:59.111 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/a100u2w.ko 2022-09-07T21:09:59.111 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/aacraid 2022-09-07T21:09:59.112 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/aacraid/aacraid.ko 2022-09-07T21:09:59.112 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/advansys.ko 2022-09-07T21:09:59.112 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/aic7xxx 2022-09-07T21:09:59.112 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2022-09-07T21:09:59.113 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2022-09-07T21:09:59.113 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/am53c974.ko 2022-09-07T21:09:59.113 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/arcmsr 2022-09-07T21:09:59.113 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/arcmsr/arcmsr.ko 2022-09-07T21:09:59.114 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/atp870u.ko 2022-09-07T21:09:59.114 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/be2iscsi 2022-09-07T21:09:59.114 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2022-09-07T21:09:59.114 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/bfa 2022-09-07T21:09:59.114 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/bfa/bfa.ko 2022-09-07T21:09:59.115 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/bnx2fc 2022-09-07T21:09:59.115 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2022-09-07T21:09:59.115 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/bnx2i 2022-09-07T21:09:59.115 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/bnx2i/bnx2i.ko 2022-09-07T21:09:59.115 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/ch.ko 2022-09-07T21:09:59.116 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/csiostor 2022-09-07T21:09:59.116 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/csiostor/csiostor.ko 2022-09-07T21:09:59.116 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/cxgbi 2022-09-07T21:09:59.116 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/cxgbi/cxgb3i 2022-09-07T21:09:59.117 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2022-09-07T21:09:59.117 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/cxgbi/cxgb4i 2022-09-07T21:09:59.117 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2022-09-07T21:09:59.117 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2022-09-07T21:09:59.117 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/dc395x.ko 2022-09-07T21:09:59.118 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/device_handler 2022-09-07T21:09:59.118 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2022-09-07T21:09:59.118 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2022-09-07T21:09:59.118 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2022-09-07T21:09:59.119 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2022-09-07T21:09:59.119 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/dmx3191d.ko 2022-09-07T21:09:59.119 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/esas2r 2022-09-07T21:09:59.119 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/esas2r/esas2r.ko 2022-09-07T21:09:59.120 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/esp_scsi.ko 2022-09-07T21:09:59.120 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/fcoe 2022-09-07T21:09:59.120 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/fcoe/fcoe.ko 2022-09-07T21:09:59.120 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/fcoe/libfcoe.ko 2022-09-07T21:09:59.120 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/fnic 2022-09-07T21:09:59.121 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/fnic/fnic.ko 2022-09-07T21:09:59.121 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/hpsa.ko 2022-09-07T21:09:59.121 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/hptiop.ko 2022-09-07T21:09:59.121 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/hv_storvsc.ko 2022-09-07T21:09:59.121 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/initio.ko 2022-09-07T21:09:59.122 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/ipr.ko 2022-09-07T21:09:59.122 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/ips.ko 2022-09-07T21:09:59.122 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/isci 2022-09-07T21:09:59.122 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/isci/isci.ko 2022-09-07T21:09:59.123 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2022-09-07T21:09:59.123 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/iscsi_tcp.ko 2022-09-07T21:09:59.123 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/libfc 2022-09-07T21:09:59.123 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/libfc/libfc.ko 2022-09-07T21:09:59.123 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/libiscsi.ko 2022-09-07T21:09:59.124 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/libiscsi_tcp.ko 2022-09-07T21:09:59.124 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/libsas 2022-09-07T21:09:59.124 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/libsas/libsas.ko 2022-09-07T21:09:59.124 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/lpfc 2022-09-07T21:09:59.125 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/lpfc/lpfc.ko 2022-09-07T21:09:59.125 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/megaraid 2022-09-07T21:09:59.125 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/megaraid.ko 2022-09-07T21:09:59.125 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2022-09-07T21:09:59.126 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2022-09-07T21:09:59.126 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2022-09-07T21:09:59.126 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/mpt3sas 2022-09-07T21:09:59.126 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2022-09-07T21:09:59.126 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/mvsas 2022-09-07T21:09:59.127 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/mvsas/mvsas.ko 2022-09-07T21:09:59.127 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/mvumi.ko 2022-09-07T21:09:59.127 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/pm8001 2022-09-07T21:09:59.127 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/pm8001/pm80xx.ko 2022-09-07T21:09:59.128 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/pmcraid.ko 2022-09-07T21:09:59.128 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/qla1280.ko 2022-09-07T21:09:59.128 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/qla2xxx 2022-09-07T21:09:59.128 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2022-09-07T21:09:59.128 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2022-09-07T21:09:59.129 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/qla4xxx 2022-09-07T21:09:59.129 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2022-09-07T21:09:59.129 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/raid_class.ko 2022-09-07T21:09:59.129 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/scsi_debug.ko 2022-09-07T21:09:59.129 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/scsi_transport_fc.ko 2022-09-07T21:09:59.130 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/scsi_transport_iscsi.ko 2022-09-07T21:09:59.130 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/scsi_transport_sas.ko 2022-09-07T21:09:59.130 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/scsi_transport_spi.ko 2022-09-07T21:09:59.130 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/scsi_transport_srp.ko 2022-09-07T21:09:59.131 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/ses.ko 2022-09-07T21:09:59.131 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/st.ko 2022-09-07T21:09:59.131 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/stex.ko 2022-09-07T21:09:59.131 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/sym53c8xx_2 2022-09-07T21:09:59.131 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2022-09-07T21:09:59.132 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/virtio_scsi.ko 2022-09-07T21:09:59.132 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/vmw_pvscsi.ko 2022-09-07T21:09:59.132 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/wd719x.ko 2022-09-07T21:09:59.132 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ssb 2022-09-07T21:09:59.133 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ssb/ssb.ko 2022-09-07T21:09:59.133 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target 2022-09-07T21:09:59.133 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/iscsi 2022-09-07T21:09:59.133 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2022-09-07T21:09:59.133 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/loopback 2022-09-07T21:09:59.134 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/loopback/tcm_loop.ko 2022-09-07T21:09:59.134 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/target_core_file.ko 2022-09-07T21:09:59.134 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/target_core_iblock.ko 2022-09-07T21:09:59.134 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/target_core_mod.ko 2022-09-07T21:09:59.135 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/target_core_pscsi.ko 2022-09-07T21:09:59.135 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/target_core_user.ko 2022-09-07T21:09:59.135 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/tcm_fc 2022-09-07T21:09:59.135 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/tcm_fc/tcm_fc.ko 2022-09-07T21:09:59.135 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal 2022-09-07T21:09:59.136 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel 2022-09-07T21:09:59.136 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal 2022-09-07T21:09:59.136 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2022-09-07T21:09:59.136 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2022-09-07T21:09:59.137 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2022-09-07T21:09:59.137 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2022-09-07T21:09:59.137 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2022-09-07T21:09:59.137 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2022-09-07T21:09:59.137 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2022-09-07T21:09:59.138 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2022-09-07T21:09:59.138 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2022-09-07T21:09:59.138 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2022-09-07T21:09:59.138 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2022-09-07T21:09:59.139 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2022-09-07T21:09:59.139 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2022-09-07T21:09:59.139 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2022-09-07T21:09:59.139 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2022-09-07T21:09:59.139 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty 2022-09-07T21:09:59.140 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/ipwireless 2022-09-07T21:09:59.140 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/ipwireless/ipwireless.ko 2022-09-07T21:09:59.140 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/n_gsm.ko 2022-09-07T21:09:59.140 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/n_hdlc.ko 2022-09-07T21:09:59.141 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/nozomi.ko 2022-09-07T21:09:59.141 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/serial 2022-09-07T21:09:59.141 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/serial/8250 2022-09-07T21:09:59.141 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/serial/8250/serial_cs.ko 2022-09-07T21:09:59.141 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/serial/arc_uart.ko 2022-09-07T21:09:59.142 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/serial/jsm 2022-09-07T21:09:59.142 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/serial/jsm/jsm.ko 2022-09-07T21:09:59.142 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/synclink_gt.ko 2022-09-07T21:09:59.142 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ufs 2022-09-07T21:09:59.143 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ufs/core 2022-09-07T21:09:59.143 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ufs/core/ufshcd-core.ko 2022-09-07T21:09:59.143 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ufs/host 2022-09-07T21:09:59.143 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ufs/host/ufshcd-pci.ko 2022-09-07T21:09:59.144 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/uio 2022-09-07T21:09:59.144 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/uio/uio.ko 2022-09-07T21:09:59.144 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/uio/uio_aec.ko 2022-09-07T21:09:59.144 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/uio/uio_cif.ko 2022-09-07T21:09:59.144 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/uio/uio_pci_generic.ko 2022-09-07T21:09:59.145 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/uio/uio_sercos3.ko 2022-09-07T21:09:59.145 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb 2022-09-07T21:09:59.145 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/class 2022-09-07T21:09:59.145 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/class/cdc-acm.ko 2022-09-07T21:09:59.146 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/class/cdc-wdm.ko 2022-09-07T21:09:59.146 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/class/usblp.ko 2022-09-07T21:09:59.146 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/class/usbtmc.ko 2022-09-07T21:09:59.146 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/host 2022-09-07T21:09:59.146 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/host/sl811-hcd.ko 2022-09-07T21:09:59.147 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/host/u132-hcd.ko 2022-09-07T21:09:59.147 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/image 2022-09-07T21:09:59.147 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/image/mdc800.ko 2022-09-07T21:09:59.147 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/image/microtek.ko 2022-09-07T21:09:59.147 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc 2022-09-07T21:09:59.148 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/adutux.ko 2022-09-07T21:09:59.148 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/appledisplay.ko 2022-09-07T21:09:59.148 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/emi26.ko 2022-09-07T21:09:59.148 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/emi62.ko 2022-09-07T21:09:59.149 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/ezusb.ko 2022-09-07T21:09:59.149 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/ftdi-elan.ko 2022-09-07T21:09:59.149 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/idmouse.ko 2022-09-07T21:09:59.149 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/iowarrior.ko 2022-09-07T21:09:59.149 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/isight_firmware.ko 2022-09-07T21:09:59.150 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/ldusb.ko 2022-09-07T21:09:59.150 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/legousbtower.ko 2022-09-07T21:09:59.150 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/sisusbvga 2022-09-07T21:09:59.150 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2022-09-07T21:09:59.151 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/trancevibrator.ko 2022-09-07T21:09:59.151 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/usb3503.ko 2022-09-07T21:09:59.151 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/usblcd.ko 2022-09-07T21:09:59.151 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/usbsevseg.ko 2022-09-07T21:09:59.151 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/uss720.ko 2022-09-07T21:09:59.152 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/yurex.ko 2022-09-07T21:09:59.152 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/phy 2022-09-07T21:09:59.152 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/phy/phy-generic.ko 2022-09-07T21:09:59.152 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial 2022-09-07T21:09:59.153 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/aircable.ko 2022-09-07T21:09:59.153 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ark3116.ko 2022-09-07T21:09:59.153 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/belkin_sa.ko 2022-09-07T21:09:59.153 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ch341.ko 2022-09-07T21:09:59.153 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/cp210x.ko 2022-09-07T21:09:59.154 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/cyberjack.ko 2022-09-07T21:09:59.154 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/cypress_m8.ko 2022-09-07T21:09:59.154 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/digi_acceleport.ko 2022-09-07T21:09:59.154 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/empeg.ko 2022-09-07T21:09:59.155 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ftdi_sio.ko 2022-09-07T21:09:59.155 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/garmin_gps.ko 2022-09-07T21:09:59.155 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/io_edgeport.ko 2022-09-07T21:09:59.155 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/io_ti.ko 2022-09-07T21:09:59.155 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ipaq.ko 2022-09-07T21:09:59.156 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ipw.ko 2022-09-07T21:09:59.156 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ir-usb.ko 2022-09-07T21:09:59.156 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/iuu_phoenix.ko 2022-09-07T21:09:59.156 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/keyspan.ko 2022-09-07T21:09:59.156 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/keyspan_pda.ko 2022-09-07T21:09:59.157 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/kl5kusb105.ko 2022-09-07T21:09:59.157 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/kobil_sct.ko 2022-09-07T21:09:59.157 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/mct_u232.ko 2022-09-07T21:09:59.157 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/mos7720.ko 2022-09-07T21:09:59.158 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/mos7840.ko 2022-09-07T21:09:59.158 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/navman.ko 2022-09-07T21:09:59.158 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/omninet.ko 2022-09-07T21:09:59.158 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/opticon.ko 2022-09-07T21:09:59.158 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/option.ko 2022-09-07T21:09:59.159 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/oti6858.ko 2022-09-07T21:09:59.159 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/pl2303.ko 2022-09-07T21:09:59.159 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/qcaux.ko 2022-09-07T21:09:59.159 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/qcserial.ko 2022-09-07T21:09:59.160 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/quatech2.ko 2022-09-07T21:09:59.160 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/safe_serial.ko 2022-09-07T21:09:59.160 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/sierra.ko 2022-09-07T21:09:59.160 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/spcp8x5.ko 2022-09-07T21:09:59.160 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ssu100.ko 2022-09-07T21:09:59.161 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/symbolserial.ko 2022-09-07T21:09:59.161 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2022-09-07T21:09:59.161 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/usb-serial-simple.ko 2022-09-07T21:09:59.161 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/usb_debug.ko 2022-09-07T21:09:59.162 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/usb_wwan.ko 2022-09-07T21:09:59.162 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/visor.ko 2022-09-07T21:09:59.162 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/whiteheat.ko 2022-09-07T21:09:59.162 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/xsens_mt.ko 2022-09-07T21:09:59.162 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage 2022-09-07T21:09:59.163 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/uas.ko 2022-09-07T21:09:59.163 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-alauda.ko 2022-09-07T21:09:59.163 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-cypress.ko 2022-09-07T21:09:59.163 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-datafab.ko 2022-09-07T21:09:59.163 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-eneub6250.ko 2022-09-07T21:09:59.164 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-freecom.ko 2022-09-07T21:09:59.164 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-isd200.ko 2022-09-07T21:09:59.164 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-jumpshot.ko 2022-09-07T21:09:59.164 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-karma.ko 2022-09-07T21:09:59.165 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-onetouch.ko 2022-09-07T21:09:59.165 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-realtek.ko 2022-09-07T21:09:59.165 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-sddr09.ko 2022-09-07T21:09:59.165 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-sddr55.ko 2022-09-07T21:09:59.165 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-usbat.ko 2022-09-07T21:09:59.166 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/usb-storage.ko 2022-09-07T21:09:59.166 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/usbip 2022-09-07T21:09:59.166 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/usbip/usbip-core.ko 2022-09-07T21:09:59.166 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/usbip/usbip-host.ko 2022-09-07T21:09:59.167 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/usbip/vhci-hcd.ko 2022-09-07T21:09:59.167 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vfio 2022-09-07T21:09:59.167 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vfio/pci 2022-09-07T21:09:59.167 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vfio/pci/vfio-pci-core.ko 2022-09-07T21:09:59.168 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vfio/pci/vfio-pci.ko 2022-09-07T21:09:59.168 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vfio/vfio.ko 2022-09-07T21:09:59.168 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vfio/vfio_iommu_type1.ko 2022-09-07T21:09:59.168 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vfio/vfio_virqfd.ko 2022-09-07T21:09:59.169 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vhost 2022-09-07T21:09:59.169 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vhost/vhost.ko 2022-09-07T21:09:59.169 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vhost/vhost_iotlb.ko 2022-09-07T21:09:59.169 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vhost/vhost_net.ko 2022-09-07T21:09:59.169 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vhost/vhost_scsi.ko 2022-09-07T21:09:59.170 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/video 2022-09-07T21:09:59.170 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/video/backlight 2022-09-07T21:09:59.170 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/video/backlight/apple_bl.ko 2022-09-07T21:09:59.170 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/video/backlight/lcd.ko 2022-09-07T21:09:59.171 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/video/backlight/platform_lcd.ko 2022-09-07T21:09:59.171 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio 2022-09-07T21:09:59.171 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio/virtio.ko 2022-09-07T21:09:59.171 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio/virtio_balloon.ko 2022-09-07T21:09:59.171 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio/virtio_mmio.ko 2022-09-07T21:09:59.172 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio/virtio_pci.ko 2022-09-07T21:09:59.172 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2022-09-07T21:09:59.172 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2022-09-07T21:09:59.172 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio/virtio_ring.ko 2022-09-07T21:09:59.173 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1 2022-09-07T21:09:59.173 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/masters 2022-09-07T21:09:59.173 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/masters/ds1wm.ko 2022-09-07T21:09:59.173 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/masters/ds2482.ko 2022-09-07T21:09:59.173 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/masters/ds2490.ko 2022-09-07T21:09:59.174 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves 2022-09-07T21:09:59.174 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2406.ko 2022-09-07T21:09:59.174 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2408.ko 2022-09-07T21:09:59.174 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2413.ko 2022-09-07T21:09:59.175 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2423.ko 2022-09-07T21:09:59.175 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2431.ko 2022-09-07T21:09:59.175 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2433.ko 2022-09-07T21:09:59.175 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2780.ko 2022-09-07T21:09:59.175 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2781.ko 2022-09-07T21:09:59.176 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds28e04.ko 2022-09-07T21:09:59.176 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_smem.ko 2022-09-07T21:09:59.176 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_therm.ko 2022-09-07T21:09:59.176 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/wire.ko 2022-09-07T21:09:59.177 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog 2022-09-07T21:09:59.177 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/alim1535_wdt.ko 2022-09-07T21:09:59.177 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/alim7101_wdt.ko 2022-09-07T21:09:59.177 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/f71808e_wdt.ko 2022-09-07T21:09:59.177 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/hpwdt.ko 2022-09-07T21:09:59.178 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/i6300esb.ko 2022-09-07T21:09:59.178 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/iTCO_vendor_support.ko 2022-09-07T21:09:59.178 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/iTCO_wdt.ko 2022-09-07T21:09:59.178 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/ib700wdt.ko 2022-09-07T21:09:59.179 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/ibmasr.ko 2022-09-07T21:09:59.179 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/ie6xx_wdt.ko 2022-09-07T21:09:59.179 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/it8712f_wdt.ko 2022-09-07T21:09:59.179 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/it87_wdt.ko 2022-09-07T21:09:59.179 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/machzwd.ko 2022-09-07T21:09:59.180 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/nv_tco.ko 2022-09-07T21:09:59.180 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/pcwd_pci.ko 2022-09-07T21:09:59.180 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/pcwd_usb.ko 2022-09-07T21:09:59.180 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2022-09-07T21:09:59.181 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/sch311x_wdt.ko 2022-09-07T21:09:59.181 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/softdog.ko 2022-09-07T21:09:59.181 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/sp5100_tco.ko 2022-09-07T21:09:59.181 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/via_wdt.ko 2022-09-07T21:09:59.181 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/w83627hf_wdt.ko 2022-09-07T21:09:59.182 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/w83877f_wdt.ko 2022-09-07T21:09:59.182 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/w83977f_wdt.ko 2022-09-07T21:09:59.182 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/wdt_pci.ko 2022-09-07T21:09:59.182 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/xen_wdt.ko 2022-09-07T21:09:59.183 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen 2022-09-07T21:09:59.183 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xen-acpi-processor.ko 2022-09-07T21:09:59.183 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xen-evtchn.ko 2022-09-07T21:09:59.183 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xen-gntalloc.ko 2022-09-07T21:09:59.183 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xen-gntdev.ko 2022-09-07T21:09:59.184 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xen-pciback 2022-09-07T21:09:59.184 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2022-09-07T21:09:59.184 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xen-privcmd.ko 2022-09-07T21:09:59.184 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xenfs 2022-09-07T21:09:59.185 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xenfs/xenfs.ko 2022-09-07T21:09:59.185 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs 2022-09-07T21:09:59.185 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/9p 2022-09-07T21:09:59.185 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/9p/9p.ko 2022-09-07T21:09:59.185 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/affs 2022-09-07T21:09:59.186 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/affs/affs.ko 2022-09-07T21:09:59.186 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/befs 2022-09-07T21:09:59.186 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/befs/befs.ko 2022-09-07T21:09:59.186 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/binfmt_misc.ko 2022-09-07T21:09:59.187 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/btrfs 2022-09-07T21:09:59.187 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/btrfs/btrfs.ko 2022-09-07T21:09:59.187 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/cachefiles 2022-09-07T21:09:59.187 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/cachefiles/cachefiles.ko 2022-09-07T21:09:59.188 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ceph 2022-09-07T21:09:59.188 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ceph/ceph.ko 2022-09-07T21:09:59.188 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/cifs 2022-09-07T21:09:59.188 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/cifs/cifs.ko 2022-09-07T21:09:59.188 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/coda 2022-09-07T21:09:59.189 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/coda/coda.ko 2022-09-07T21:09:59.189 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/cramfs 2022-09-07T21:09:59.189 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/cramfs/cramfs.ko 2022-09-07T21:09:59.189 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/dlm 2022-09-07T21:09:59.190 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/dlm/dlm.ko 2022-09-07T21:09:59.190 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ecryptfs 2022-09-07T21:09:59.190 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ecryptfs/ecryptfs.ko 2022-09-07T21:09:59.190 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/f2fs 2022-09-07T21:09:59.190 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/f2fs/f2fs.ko 2022-09-07T21:09:59.191 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fat 2022-09-07T21:09:59.191 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fat/fat.ko 2022-09-07T21:09:59.191 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fat/msdos.ko 2022-09-07T21:09:59.191 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fat/vfat.ko 2022-09-07T21:09:59.192 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fscache 2022-09-07T21:09:59.192 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fscache/fscache.ko 2022-09-07T21:09:59.192 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fuse 2022-09-07T21:09:59.192 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fuse/cuse.ko 2022-09-07T21:09:59.192 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fuse/fuse.ko 2022-09-07T21:09:59.193 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/gfs2 2022-09-07T21:09:59.193 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/gfs2/gfs2.ko 2022-09-07T21:09:59.193 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/hfs 2022-09-07T21:09:59.193 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/hfs/hfs.ko 2022-09-07T21:09:59.194 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/hfsplus 2022-09-07T21:09:59.194 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/hfsplus/hfsplus.ko 2022-09-07T21:09:59.194 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/isofs 2022-09-07T21:09:59.194 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/isofs/isofs.ko 2022-09-07T21:09:59.195 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/jfs 2022-09-07T21:09:59.195 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/jfs/jfs.ko 2022-09-07T21:09:59.195 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/lockd 2022-09-07T21:09:59.195 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/lockd/lockd.ko 2022-09-07T21:09:59.195 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/minix 2022-09-07T21:09:59.196 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/minix/minix.ko 2022-09-07T21:09:59.196 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/netfs 2022-09-07T21:09:59.196 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/netfs/netfs.ko 2022-09-07T21:09:59.196 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs 2022-09-07T21:09:59.197 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/blocklayout 2022-09-07T21:09:59.197 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2022-09-07T21:09:59.197 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/filelayout 2022-09-07T21:09:59.197 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2022-09-07T21:09:59.197 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/flexfilelayout 2022-09-07T21:09:59.198 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2022-09-07T21:09:59.198 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/nfs.ko 2022-09-07T21:09:59.198 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/nfsv3.ko 2022-09-07T21:09:59.198 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/nfsv4.ko 2022-09-07T21:09:59.199 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs_common 2022-09-07T21:09:59.199 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs_common/grace.ko 2022-09-07T21:09:59.199 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs_common/nfs_acl.ko 2022-09-07T21:09:59.199 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfsd 2022-09-07T21:09:59.199 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfsd/nfsd.ko 2022-09-07T21:09:59.200 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nilfs2 2022-09-07T21:09:59.200 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nilfs2/nilfs2.ko 2022-09-07T21:09:59.200 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls 2022-09-07T21:09:59.200 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-celtic.ko 2022-09-07T21:09:59.201 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-centeuro.ko 2022-09-07T21:09:59.201 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-croatian.ko 2022-09-07T21:09:59.201 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-cyrillic.ko 2022-09-07T21:09:59.201 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-gaelic.ko 2022-09-07T21:09:59.201 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-greek.ko 2022-09-07T21:09:59.202 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-iceland.ko 2022-09-07T21:09:59.202 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-inuit.ko 2022-09-07T21:09:59.202 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-roman.ko 2022-09-07T21:09:59.202 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-romanian.ko 2022-09-07T21:09:59.203 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-turkish.ko 2022-09-07T21:09:59.203 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp1250.ko 2022-09-07T21:09:59.203 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp1251.ko 2022-09-07T21:09:59.203 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp1255.ko 2022-09-07T21:09:59.203 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp737.ko 2022-09-07T21:09:59.204 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp775.ko 2022-09-07T21:09:59.204 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp850.ko 2022-09-07T21:09:59.204 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp852.ko 2022-09-07T21:09:59.204 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp855.ko 2022-09-07T21:09:59.205 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp857.ko 2022-09-07T21:09:59.205 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp860.ko 2022-09-07T21:09:59.205 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp861.ko 2022-09-07T21:09:59.205 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp862.ko 2022-09-07T21:09:59.205 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp863.ko 2022-09-07T21:09:59.206 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp864.ko 2022-09-07T21:09:59.206 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp865.ko 2022-09-07T21:09:59.206 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp866.ko 2022-09-07T21:09:59.206 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp869.ko 2022-09-07T21:09:59.207 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp874.ko 2022-09-07T21:09:59.207 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp932.ko 2022-09-07T21:09:59.207 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp936.ko 2022-09-07T21:09:59.207 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp949.ko 2022-09-07T21:09:59.207 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp950.ko 2022-09-07T21:09:59.208 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_euc-jp.ko 2022-09-07T21:09:59.208 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-1.ko 2022-09-07T21:09:59.208 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-13.ko 2022-09-07T21:09:59.208 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-14.ko 2022-09-07T21:09:59.209 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-15.ko 2022-09-07T21:09:59.209 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-2.ko 2022-09-07T21:09:59.209 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-3.ko 2022-09-07T21:09:59.209 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-4.ko 2022-09-07T21:09:59.209 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-5.ko 2022-09-07T21:09:59.210 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-6.ko 2022-09-07T21:09:59.210 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-7.ko 2022-09-07T21:09:59.210 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-9.ko 2022-09-07T21:09:59.210 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_koi8-r.ko 2022-09-07T21:09:59.211 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_koi8-ru.ko 2022-09-07T21:09:59.211 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_koi8-u.ko 2022-09-07T21:09:59.211 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_utf8.ko 2022-09-07T21:09:59.211 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2 2022-09-07T21:09:59.211 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/cluster 2022-09-07T21:09:59.212 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2022-09-07T21:09:59.212 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/dlm 2022-09-07T21:09:59.212 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2022-09-07T21:09:59.212 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/dlmfs 2022-09-07T21:09:59.213 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2022-09-07T21:09:59.213 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/ocfs2.ko 2022-09-07T21:09:59.213 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2022-09-07T21:09:59.213 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/ocfs2_stack_user.ko 2022-09-07T21:09:59.213 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/ocfs2_stackglue.ko 2022-09-07T21:09:59.214 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/overlayfs 2022-09-07T21:09:59.214 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/overlayfs/overlay.ko 2022-09-07T21:09:59.214 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/pstore 2022-09-07T21:09:59.214 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/pstore/ramoops.ko 2022-09-07T21:09:59.215 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/reiserfs 2022-09-07T21:09:59.215 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/reiserfs/reiserfs.ko 2022-09-07T21:09:59.215 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/romfs 2022-09-07T21:09:59.215 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/romfs/romfs.ko 2022-09-07T21:09:59.216 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/smbfs_common 2022-09-07T21:09:59.216 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/smbfs_common/cifs_arc4.ko 2022-09-07T21:09:59.216 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/smbfs_common/cifs_md4.ko 2022-09-07T21:09:59.216 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/squashfs 2022-09-07T21:09:59.216 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/squashfs/squashfs.ko 2022-09-07T21:09:59.217 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/sysv 2022-09-07T21:09:59.217 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/sysv/sysv.ko 2022-09-07T21:09:59.217 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/udf 2022-09-07T21:09:59.217 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/udf/udf.ko 2022-09-07T21:09:59.217 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ufs 2022-09-07T21:09:59.218 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ufs/ufs.ko 2022-09-07T21:09:59.218 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/xfs 2022-09-07T21:09:59.218 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/xfs/xfs.ko 2022-09-07T21:09:59.218 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/kernel 2022-09-07T21:09:59.219 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/kernel/rcu 2022-09-07T21:09:59.219 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/kernel/rcu/rcutorture.ko 2022-09-07T21:09:59.219 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/kernel/torture.ko 2022-09-07T21:09:59.219 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/kernel/trace 2022-09-07T21:09:59.220 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/kernel/trace/ring_buffer_benchmark.ko 2022-09-07T21:09:59.220 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib 2022-09-07T21:09:59.220 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/asn1_encoder.ko 2022-09-07T21:09:59.220 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/crc-itu-t.ko 2022-09-07T21:09:59.220 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/crc8.ko 2022-09-07T21:09:59.221 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/crypto 2022-09-07T21:09:59.221 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/crypto/libarc4.ko 2022-09-07T21:09:59.221 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/crypto/libdes.ko 2022-09-07T21:09:59.221 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/libcrc32c.ko 2022-09-07T21:09:59.222 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/lru_cache.ko 2022-09-07T21:09:59.222 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/lz4 2022-09-07T21:09:59.222 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/lz4/lz4_compress.ko 2022-09-07T21:09:59.222 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/lz4/lz4hc_compress.ko 2022-09-07T21:09:59.222 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/math 2022-09-07T21:09:59.223 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/math/cordic.ko 2022-09-07T21:09:59.223 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/raid6 2022-09-07T21:09:59.223 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/raid6/raid6_pq.ko 2022-09-07T21:09:59.223 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/reed_solomon 2022-09-07T21:09:59.224 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/reed_solomon/reed_solomon.ko 2022-09-07T21:09:59.224 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/ts_bm.ko 2022-09-07T21:09:59.224 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/ts_fsm.ko 2022-09-07T21:09:59.224 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/ts_kmp.ko 2022-09-07T21:09:59.224 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/zstd 2022-09-07T21:09:59.225 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/zstd/zstd_compress.ko 2022-09-07T21:09:59.225 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/mm 2022-09-07T21:09:59.225 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/mm/hwpoison-inject.ko 2022-09-07T21:09:59.225 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net 2022-09-07T21:09:59.226 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan 2022-09-07T21:09:59.226 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/6lowpan.ko 2022-09-07T21:09:59.226 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/nhc_dest.ko 2022-09-07T21:09:59.226 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/nhc_fragment.ko 2022-09-07T21:09:59.226 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/nhc_hop.ko 2022-09-07T21:09:59.227 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/nhc_ipv6.ko 2022-09-07T21:09:59.227 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/nhc_mobility.ko 2022-09-07T21:09:59.227 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/nhc_routing.ko 2022-09-07T21:09:59.227 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/nhc_udp.ko 2022-09-07T21:09:59.228 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/802 2022-09-07T21:09:59.228 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/802/garp.ko 2022-09-07T21:09:59.228 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/802/mrp.ko 2022-09-07T21:09:59.228 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/802/p8022.ko 2022-09-07T21:09:59.228 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/802/psnap.ko 2022-09-07T21:09:59.229 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/802/stp.ko 2022-09-07T21:09:59.229 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/8021q 2022-09-07T21:09:59.229 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/8021q/8021q.ko 2022-09-07T21:09:59.229 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/9p 2022-09-07T21:09:59.230 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/9p/9pnet.ko 2022-09-07T21:09:59.230 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/9p/9pnet_fd.ko 2022-09-07T21:09:59.230 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/9p/9pnet_rdma.ko 2022-09-07T21:09:59.230 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/9p/9pnet_virtio.ko 2022-09-07T21:09:59.231 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/batman-adv 2022-09-07T21:09:59.231 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/batman-adv/batman-adv.ko 2022-09-07T21:09:59.231 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge 2022-09-07T21:09:59.231 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/br_netfilter.ko 2022-09-07T21:09:59.232 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/bridge.ko 2022-09-07T21:09:59.232 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter 2022-09-07T21:09:59.232 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_802_3.ko 2022-09-07T21:09:59.232 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_among.ko 2022-09-07T21:09:59.232 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_arp.ko 2022-09-07T21:09:59.233 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_arpreply.ko 2022-09-07T21:09:59.233 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_dnat.ko 2022-09-07T21:09:59.233 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_ip.ko 2022-09-07T21:09:59.233 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_ip6.ko 2022-09-07T21:09:59.234 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_limit.ko 2022-09-07T21:09:59.234 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_log.ko 2022-09-07T21:09:59.234 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_mark.ko 2022-09-07T21:09:59.234 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_mark_m.ko 2022-09-07T21:09:59.235 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_nflog.ko 2022-09-07T21:09:59.235 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_pkttype.ko 2022-09-07T21:09:59.235 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_redirect.ko 2022-09-07T21:09:59.235 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_snat.ko 2022-09-07T21:09:59.235 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_stp.ko 2022-09-07T21:09:59.236 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_vlan.ko 2022-09-07T21:09:59.236 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebtable_broute.ko 2022-09-07T21:09:59.236 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebtable_filter.ko 2022-09-07T21:09:59.236 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebtable_nat.ko 2022-09-07T21:09:59.237 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebtables.ko 2022-09-07T21:09:59.237 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2022-09-07T21:09:59.237 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2022-09-07T21:09:59.237 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2022-09-07T21:09:59.238 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ceph 2022-09-07T21:09:59.238 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ceph/libceph.ko 2022-09-07T21:09:59.238 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/core 2022-09-07T21:09:59.238 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/core/failover.ko 2022-09-07T21:09:59.238 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/core/pktgen.ko 2022-09-07T21:09:59.239 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dccp 2022-09-07T21:09:59.239 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dccp/dccp.ko 2022-09-07T21:09:59.239 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dccp/dccp_diag.ko 2022-09-07T21:09:59.239 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dccp/dccp_ipv4.ko 2022-09-07T21:09:59.240 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dccp/dccp_ipv6.ko 2022-09-07T21:09:59.240 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dns_resolver 2022-09-07T21:09:59.240 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dns_resolver/dns_resolver.ko 2022-09-07T21:09:59.240 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dsa 2022-09-07T21:09:59.240 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dsa/dsa_core.ko 2022-09-07T21:09:59.241 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dsa/tag_brcm.ko 2022-09-07T21:09:59.241 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dsa/tag_dsa.ko 2022-09-07T21:09:59.241 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dsa/tag_trailer.ko 2022-09-07T21:09:59.241 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ieee802154 2022-09-07T21:09:59.242 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ieee802154/6lowpan 2022-09-07T21:09:59.242 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2022-09-07T21:09:59.242 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ieee802154/ieee802154.ko 2022-09-07T21:09:59.242 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ieee802154/ieee802154_socket.ko 2022-09-07T21:09:59.242 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4 2022-09-07T21:09:59.243 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/ah4.ko 2022-09-07T21:09:59.243 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/esp4.ko 2022-09-07T21:09:59.243 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/fou.ko 2022-09-07T21:09:59.243 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/gre.ko 2022-09-07T21:09:59.244 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/inet_diag.ko 2022-09-07T21:09:59.244 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/ip_gre.ko 2022-09-07T21:09:59.244 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/ip_tunnel.ko 2022-09-07T21:09:59.244 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/ip_vti.ko 2022-09-07T21:09:59.245 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/ipcomp.ko 2022-09-07T21:09:59.245 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/ipip.ko 2022-09-07T21:09:59.245 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter 2022-09-07T21:09:59.245 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/arp_tables.ko 2022-09-07T21:09:59.245 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/arpt_mangle.ko 2022-09-07T21:09:59.246 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/arptable_filter.ko 2022-09-07T21:09:59.246 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/ipt_CLUSTERIP.ko 2022-09-07T21:09:59.246 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/ipt_ECN.ko 2022-09-07T21:09:59.246 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2022-09-07T21:09:59.247 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/ipt_ah.ko 2022-09-07T21:09:59.247 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2022-09-07T21:09:59.247 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/iptable_mangle.ko 2022-09-07T21:09:59.247 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/iptable_nat.ko 2022-09-07T21:09:59.248 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/iptable_raw.ko 2022-09-07T21:09:59.248 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/iptable_security.ko 2022-09-07T21:09:59.248 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2022-09-07T21:09:59.248 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2022-09-07T21:09:59.248 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2022-09-07T21:09:59.249 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2022-09-07T21:09:59.249 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2022-09-07T21:09:59.249 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2022-09-07T21:09:59.249 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2022-09-07T21:09:59.249 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2022-09-07T21:09:59.250 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2022-09-07T21:09:59.250 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2022-09-07T21:09:59.250 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_bic.ko 2022-09-07T21:09:59.250 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_dctcp.ko 2022-09-07T21:09:59.251 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_diag.ko 2022-09-07T21:09:59.251 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_highspeed.ko 2022-09-07T21:09:59.251 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_htcp.ko 2022-09-07T21:09:59.251 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_hybla.ko 2022-09-07T21:09:59.252 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_illinois.ko 2022-09-07T21:09:59.252 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_lp.ko 2022-09-07T21:09:59.252 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_scalable.ko 2022-09-07T21:09:59.252 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_vegas.ko 2022-09-07T21:09:59.252 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_veno.ko 2022-09-07T21:09:59.253 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_westwood.ko 2022-09-07T21:09:59.253 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_yeah.ko 2022-09-07T21:09:59.253 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tunnel4.ko 2022-09-07T21:09:59.253 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/udp_diag.ko 2022-09-07T21:09:59.254 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/udp_tunnel.ko 2022-09-07T21:09:59.254 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/xfrm4_tunnel.ko 2022-09-07T21:09:59.254 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6 2022-09-07T21:09:59.254 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/ah6.ko 2022-09-07T21:09:59.255 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/esp6.ko 2022-09-07T21:09:59.255 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/fou6.ko 2022-09-07T21:09:59.255 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/ip6_tunnel.ko 2022-09-07T21:09:59.255 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/ip6_udp_tunnel.ko 2022-09-07T21:09:59.255 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/ip6_vti.ko 2022-09-07T21:09:59.256 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/ipcomp6.ko 2022-09-07T21:09:59.256 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter 2022-09-07T21:09:59.256 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6_tables.ko 2022-09-07T21:09:59.256 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2022-09-07T21:09:59.257 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2022-09-07T21:09:59.257 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_ah.ko 2022-09-07T21:09:59.257 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2022-09-07T21:09:59.257 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_frag.ko 2022-09-07T21:09:59.257 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2022-09-07T21:09:59.258 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2022-09-07T21:09:59.258 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_mh.ko 2022-09-07T21:09:59.258 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2022-09-07T21:09:59.258 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_rt.ko 2022-09-07T21:09:59.259 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6table_filter.ko 2022-09-07T21:09:59.259 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2022-09-07T21:09:59.259 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6table_nat.ko 2022-09-07T21:09:59.259 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6table_raw.ko 2022-09-07T21:09:59.259 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6table_security.ko 2022-09-07T21:09:59.260 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2022-09-07T21:09:59.260 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2022-09-07T21:09:59.260 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2022-09-07T21:09:59.260 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2022-09-07T21:09:59.261 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2022-09-07T21:09:59.261 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2022-09-07T21:09:59.261 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2022-09-07T21:09:59.261 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2022-09-07T21:09:59.261 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/sit.ko 2022-09-07T21:09:59.262 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/tunnel6.ko 2022-09-07T21:09:59.262 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/xfrm6_tunnel.ko 2022-09-07T21:09:59.262 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/key 2022-09-07T21:09:59.262 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/key/af_key.ko 2022-09-07T21:09:59.263 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp 2022-09-07T21:09:59.263 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp/l2tp_core.ko 2022-09-07T21:09:59.263 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp/l2tp_debugfs.ko 2022-09-07T21:09:59.263 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp/l2tp_eth.ko 2022-09-07T21:09:59.263 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp/l2tp_ip.ko 2022-09-07T21:09:59.264 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp/l2tp_ip6.ko 2022-09-07T21:09:59.264 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp/l2tp_netlink.ko 2022-09-07T21:09:59.264 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp/l2tp_ppp.ko 2022-09-07T21:09:59.264 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/llc 2022-09-07T21:09:59.265 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/llc/llc.ko 2022-09-07T21:09:59.265 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/mac802154 2022-09-07T21:09:59.265 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/mac802154/mac802154.ko 2022-09-07T21:09:59.265 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/mpls 2022-09-07T21:09:59.265 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/mpls/mpls_gso.ko 2022-09-07T21:09:59.266 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter 2022-09-07T21:09:59.266 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset 2022-09-07T21:09:59.266 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set.ko 2022-09-07T21:09:59.266 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2022-09-07T21:09:59.267 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2022-09-07T21:09:59.267 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2022-09-07T21:09:59.267 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2022-09-07T21:09:59.267 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2022-09-07T21:09:59.268 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2022-09-07T21:09:59.268 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2022-09-07T21:09:59.268 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2022-09-07T21:09:59.268 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2022-09-07T21:09:59.268 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2022-09-07T21:09:59.269 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2022-09-07T21:09:59.269 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2022-09-07T21:09:59.269 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2022-09-07T21:09:59.269 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2022-09-07T21:09:59.269 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_list_set.ko 2022-09-07T21:09:59.270 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs 2022-09-07T21:09:59.270 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs.ko 2022-09-07T21:09:59.271 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2022-09-07T21:09:59.271 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2022-09-07T21:09:59.272 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2022-09-07T21:09:59.272 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2022-09-07T21:09:59.272 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2022-09-07T21:09:59.273 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2022-09-07T21:09:59.273 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2022-09-07T21:09:59.273 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2022-09-07T21:09:59.273 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2022-09-07T21:09:59.274 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2022-09-07T21:09:59.274 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2022-09-07T21:09:59.274 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2022-09-07T21:09:59.275 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2022-09-07T21:09:59.275 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conncount.ko 2022-09-07T21:09:59.275 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack.ko 2022-09-07T21:09:59.275 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_amanda.ko 2022-09-07T21:09:59.275 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_broadcast.ko 2022-09-07T21:09:59.276 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_ftp.ko 2022-09-07T21:09:59.276 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_h323.ko 2022-09-07T21:09:59.276 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_irc.ko 2022-09-07T21:09:59.276 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2022-09-07T21:09:59.277 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_netlink.ko 2022-09-07T21:09:59.277 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_pptp.ko 2022-09-07T21:09:59.277 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_sane.ko 2022-09-07T21:09:59.277 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_sip.ko 2022-09-07T21:09:59.277 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_snmp.ko 2022-09-07T21:09:59.278 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_tftp.ko 2022-09-07T21:09:59.278 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_dup_netdev.ko 2022-09-07T21:09:59.278 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_flow_table.ko 2022-09-07T21:09:59.279 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_flow_table_inet.ko 2022-09-07T21:09:59.279 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_log_syslog.ko 2022-09-07T21:09:59.279 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_nat.ko 2022-09-07T21:09:59.279 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_nat_amanda.ko 2022-09-07T21:09:59.280 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_nat_ftp.ko 2022-09-07T21:09:59.280 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_nat_irc.ko 2022-09-07T21:09:59.280 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_nat_sip.ko 2022-09-07T21:09:59.281 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_nat_tftp.ko 2022-09-07T21:09:59.281 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_synproxy_core.ko 2022-09-07T21:09:59.282 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_tables.ko 2022-09-07T21:09:59.282 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nfnetlink.ko 2022-09-07T21:09:59.282 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nfnetlink_acct.ko 2022-09-07T21:09:59.283 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nfnetlink_log.ko 2022-09-07T21:09:59.283 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nfnetlink_osf.ko 2022-09-07T21:09:59.283 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nfnetlink_queue.ko 2022-09-07T21:09:59.283 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_chain_nat.ko 2022-09-07T21:09:59.283 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_compat.ko 2022-09-07T21:09:59.284 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_ct.ko 2022-09-07T21:09:59.284 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_dup_netdev.ko 2022-09-07T21:09:59.284 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_fib.ko 2022-09-07T21:09:59.284 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_fwd_netdev.ko 2022-09-07T21:09:59.285 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_hash.ko 2022-09-07T21:09:59.285 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_limit.ko 2022-09-07T21:09:59.285 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_log.ko 2022-09-07T21:09:59.285 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_masq.ko 2022-09-07T21:09:59.286 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_nat.ko 2022-09-07T21:09:59.286 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_numgen.ko 2022-09-07T21:09:59.287 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_objref.ko 2022-09-07T21:09:59.287 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_queue.ko 2022-09-07T21:09:59.287 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_quota.ko 2022-09-07T21:09:59.288 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_redir.ko 2022-09-07T21:09:59.288 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_reject.ko 2022-09-07T21:09:59.289 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_reject_inet.ko 2022-09-07T21:09:59.289 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_socket.ko 2022-09-07T21:09:59.290 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_tunnel.ko 2022-09-07T21:09:59.290 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_AUDIT.ko 2022-09-07T21:09:59.291 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_CHECKSUM.ko 2022-09-07T21:09:59.291 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_CLASSIFY.ko 2022-09-07T21:09:59.291 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_CONNSECMARK.ko 2022-09-07T21:09:59.292 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_CT.ko 2022-09-07T21:09:59.292 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_DSCP.ko 2022-09-07T21:09:59.293 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_HL.ko 2022-09-07T21:09:59.293 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_HMARK.ko 2022-09-07T21:09:59.294 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_IDLETIMER.ko 2022-09-07T21:09:59.295 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_LED.ko 2022-09-07T21:09:59.295 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_LOG.ko 2022-09-07T21:09:59.296 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_MASQUERADE.ko 2022-09-07T21:09:59.296 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_NETMAP.ko 2022-09-07T21:09:59.296 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_NFLOG.ko 2022-09-07T21:09:59.296 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_NFQUEUE.ko 2022-09-07T21:09:59.296 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_RATEEST.ko 2022-09-07T21:09:59.297 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_REDIRECT.ko 2022-09-07T21:09:59.297 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_SECMARK.ko 2022-09-07T21:09:59.297 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_TCPMSS.ko 2022-09-07T21:09:59.297 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2022-09-07T21:09:59.298 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_TEE.ko 2022-09-07T21:09:59.298 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_TPROXY.ko 2022-09-07T21:09:59.298 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_TRACE.ko 2022-09-07T21:09:59.298 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_addrtype.ko 2022-09-07T21:09:59.299 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_bpf.ko 2022-09-07T21:09:59.299 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_cgroup.ko 2022-09-07T21:09:59.299 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_cluster.ko 2022-09-07T21:09:59.299 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_comment.ko 2022-09-07T21:09:59.299 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_connbytes.ko 2022-09-07T21:09:59.300 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_connlabel.ko 2022-09-07T21:09:59.300 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_connlimit.ko 2022-09-07T21:09:59.300 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_connmark.ko 2022-09-07T21:09:59.300 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_conntrack.ko 2022-09-07T21:09:59.301 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_cpu.ko 2022-09-07T21:09:59.301 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_dccp.ko 2022-09-07T21:09:59.301 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_devgroup.ko 2022-09-07T21:09:59.301 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_dscp.ko 2022-09-07T21:09:59.301 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_ecn.ko 2022-09-07T21:09:59.302 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_esp.ko 2022-09-07T21:09:59.302 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_hashlimit.ko 2022-09-07T21:09:59.302 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_helper.ko 2022-09-07T21:09:59.302 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_hl.ko 2022-09-07T21:09:59.303 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_ipcomp.ko 2022-09-07T21:09:59.303 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_iprange.ko 2022-09-07T21:09:59.304 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_ipvs.ko 2022-09-07T21:09:59.304 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_l2tp.ko 2022-09-07T21:09:59.304 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_length.ko 2022-09-07T21:09:59.304 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_limit.ko 2022-09-07T21:09:59.305 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_mac.ko 2022-09-07T21:09:59.305 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_mark.ko 2022-09-07T21:09:59.305 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_multiport.ko 2022-09-07T21:09:59.305 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_nat.ko 2022-09-07T21:09:59.306 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_nfacct.ko 2022-09-07T21:09:59.306 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_osf.ko 2022-09-07T21:09:59.306 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_owner.ko 2022-09-07T21:09:59.307 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_physdev.ko 2022-09-07T21:09:59.307 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_pkttype.ko 2022-09-07T21:09:59.307 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_policy.ko 2022-09-07T21:09:59.307 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_quota.ko 2022-09-07T21:09:59.308 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_rateest.ko 2022-09-07T21:09:59.308 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_realm.ko 2022-09-07T21:09:59.308 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_recent.ko 2022-09-07T21:09:59.308 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_sctp.ko 2022-09-07T21:09:59.308 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_set.ko 2022-09-07T21:09:59.309 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_socket.ko 2022-09-07T21:09:59.309 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_state.ko 2022-09-07T21:09:59.309 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_statistic.ko 2022-09-07T21:09:59.309 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_string.ko 2022-09-07T21:09:59.309 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_tcpmss.ko 2022-09-07T21:09:59.310 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_time.ko 2022-09-07T21:09:59.310 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_u32.ko 2022-09-07T21:09:59.310 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netlink 2022-09-07T21:09:59.310 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netlink/netlink_diag.ko 2022-09-07T21:09:59.311 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/nsh 2022-09-07T21:09:59.311 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/nsh/nsh.ko 2022-09-07T21:09:59.311 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/openvswitch 2022-09-07T21:09:59.311 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/openvswitch/openvswitch.ko 2022-09-07T21:09:59.312 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/openvswitch/vport-geneve.ko 2022-09-07T21:09:59.312 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/openvswitch/vport-gre.ko 2022-09-07T21:09:59.312 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/openvswitch/vport-vxlan.ko 2022-09-07T21:09:59.312 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/packet 2022-09-07T21:09:59.312 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/packet/af_packet_diag.ko 2022-09-07T21:09:59.313 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/rds 2022-09-07T21:09:59.313 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/rds/rds.ko 2022-09-07T21:09:59.313 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/rds/rds_rdma.ko 2022-09-07T21:09:59.313 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/rds/rds_tcp.ko 2022-09-07T21:09:59.314 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched 2022-09-07T21:09:59.314 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_csum.ko 2022-09-07T21:09:59.314 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_gact.ko 2022-09-07T21:09:59.315 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_ipt.ko 2022-09-07T21:09:59.315 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_mirred.ko 2022-09-07T21:09:59.315 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_nat.ko 2022-09-07T21:09:59.316 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_pedit.ko 2022-09-07T21:09:59.316 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_police.ko 2022-09-07T21:09:59.316 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_simple.ko 2022-09-07T21:09:59.316 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_skbedit.ko 2022-09-07T21:09:59.317 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_vlan.ko 2022-09-07T21:09:59.317 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_basic.ko 2022-09-07T21:09:59.317 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_bpf.ko 2022-09-07T21:09:59.317 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_flow.ko 2022-09-07T21:09:59.318 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_fw.ko 2022-09-07T21:09:59.318 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_route.ko 2022-09-07T21:09:59.318 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_rsvp.ko 2022-09-07T21:09:59.318 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_rsvp6.ko 2022-09-07T21:09:59.319 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_tcindex.ko 2022-09-07T21:09:59.319 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_u32.ko 2022-09-07T21:09:59.319 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/em_cmp.ko 2022-09-07T21:09:59.319 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/em_ipset.ko 2022-09-07T21:09:59.319 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/em_meta.ko 2022-09-07T21:09:59.320 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/em_nbyte.ko 2022-09-07T21:09:59.320 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/em_text.ko 2022-09-07T21:09:59.320 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/em_u32.ko 2022-09-07T21:09:59.320 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_cbq.ko 2022-09-07T21:09:59.321 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_choke.ko 2022-09-07T21:09:59.321 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_codel.ko 2022-09-07T21:09:59.321 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_drr.ko 2022-09-07T21:09:59.321 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_dsmark.ko 2022-09-07T21:09:59.322 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_fq.ko 2022-09-07T21:09:59.322 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_gred.ko 2022-09-07T21:09:59.322 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_hfsc.ko 2022-09-07T21:09:59.322 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_hhf.ko 2022-09-07T21:09:59.322 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_htb.ko 2022-09-07T21:09:59.323 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_ingress.ko 2022-09-07T21:09:59.323 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_mqprio.ko 2022-09-07T21:09:59.323 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_multiq.ko 2022-09-07T21:09:59.323 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_netem.ko 2022-09-07T21:09:59.324 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_pie.ko 2022-09-07T21:09:59.324 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_plug.ko 2022-09-07T21:09:59.324 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_prio.ko 2022-09-07T21:09:59.324 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_qfq.ko 2022-09-07T21:09:59.324 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_red.ko 2022-09-07T21:09:59.325 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_sfb.ko 2022-09-07T21:09:59.325 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_sfq.ko 2022-09-07T21:09:59.326 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_tbf.ko 2022-09-07T21:09:59.326 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_teql.ko 2022-09-07T21:09:59.326 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sctp 2022-09-07T21:09:59.326 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sctp/sctp.ko 2022-09-07T21:09:59.327 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sctp/sctp_diag.ko 2022-09-07T21:09:59.327 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sunrpc 2022-09-07T21:09:59.327 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sunrpc/auth_gss 2022-09-07T21:09:59.328 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2022-09-07T21:09:59.328 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2022-09-07T21:09:59.328 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sunrpc/sunrpc.ko 2022-09-07T21:09:59.328 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sunrpc/xprtrdma 2022-09-07T21:09:59.329 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2022-09-07T21:09:59.329 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/tipc 2022-09-07T21:09:59.329 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/tipc/diag.ko 2022-09-07T21:09:59.329 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/tipc/tipc.ko 2022-09-07T21:09:59.330 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/unix 2022-09-07T21:09:59.330 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/unix/unix_diag.ko 2022-09-07T21:09:59.330 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/vmw_vsock 2022-09-07T21:09:59.330 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2022-09-07T21:09:59.330 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2022-09-07T21:09:59.331 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/vmw_vsock/vsock.ko 2022-09-07T21:09:59.331 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/vmw_vsock/vsock_diag.ko 2022-09-07T21:09:59.331 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/vmw_vsock/vsock_loopback.ko 2022-09-07T21:09:59.331 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/xfrm 2022-09-07T21:09:59.332 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/xfrm/xfrm_ipcomp.ko 2022-09-07T21:09:59.332 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/security 2022-09-07T21:09:59.332 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/security/keys 2022-09-07T21:09:59.332 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/security/keys/encrypted-keys 2022-09-07T21:09:59.333 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2022-09-07T21:09:59.333 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/security/keys/trusted-keys 2022-09-07T21:09:59.333 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/security/keys/trusted-keys/trusted.ko 2022-09-07T21:09:59.333 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/virt 2022-09-07T21:09:59.333 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/virt/lib 2022-09-07T21:09:59.334 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/virt/lib/irqbypass.ko 2022-09-07T21:09:59.334 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.alias 2022-09-07T21:09:59.334 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.alias.bin 2022-09-07T21:09:59.334 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.builtin 2022-09-07T21:09:59.335 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.builtin.bin 2022-09-07T21:09:59.335 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.builtin.modinfo 2022-09-07T21:09:59.335 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.dep 2022-09-07T21:09:59.335 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.dep.bin 2022-09-07T21:09:59.336 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.devname 2022-09-07T21:09:59.336 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.order 2022-09-07T21:09:59.336 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.softdep 2022-09-07T21:09:59.337 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.symbols 2022-09-07T21:09:59.337 INFO:teuthology.orchestra.run.smithi047.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.symbols.bin 2022-09-07T21:09:59.337 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/lib/.build-id 2022-09-07T21:09:59.337 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/lib/.build-id/26 2022-09-07T21:09:59.337 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/lib/.build-id/26/f3c15f7f2668ff3dc963ef43e786eb787aa4cb 2022-09-07T21:09:59.339 DEBUG:teuthology.task.kernel:get_image_version: 6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-07T21:09:59.339 DEBUG:teuthology.orchestra.run.smithi047:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2022-09-07T21:09:59.368 INFO:teuthology.orchestra.run.smithi047.stdout:postinstall scriptlet (using /bin/sh): 2022-09-07T21:09:59.369 INFO:teuthology.orchestra.run.smithi047.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.0.0-rc4-ceph-ga127ffb7df2a -a -r /boot/System.map-6.0.0-rc4-ceph-ga127ffb7df2a ]; then 2022-09-07T21:09:59.369 INFO:teuthology.orchestra.run.smithi047.stdout:cp /boot/vmlinuz-6.0.0-rc4-ceph-ga127ffb7df2a /boot/.vmlinuz-6.0.0-rc4-ceph-ga127ffb7df2a-rpm 2022-09-07T21:09:59.370 INFO:teuthology.orchestra.run.smithi047.stdout:cp /boot/System.map-6.0.0-rc4-ceph-ga127ffb7df2a /boot/.System.map-6.0.0-rc4-ceph-ga127ffb7df2a-rpm 2022-09-07T21:09:59.370 INFO:teuthology.orchestra.run.smithi047.stdout:rm -f /boot/vmlinuz-6.0.0-rc4-ceph-ga127ffb7df2a /boot/System.map-6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-07T21:09:59.370 INFO:teuthology.orchestra.run.smithi047.stdout:/sbin/installkernel 6.0.0-rc4-ceph-ga127ffb7df2a /boot/.vmlinuz-6.0.0-rc4-ceph-ga127ffb7df2a-rpm /boot/.System.map-6.0.0-rc4-ceph-ga127ffb7df2a-rpm 2022-09-07T21:09:59.370 INFO:teuthology.orchestra.run.smithi047.stdout:rm -f /boot/.vmlinuz-6.0.0-rc4-ceph-ga127ffb7df2a-rpm /boot/.System.map-6.0.0-rc4-ceph-ga127ffb7df2a-rpm 2022-09-07T21:09:59.371 INFO:teuthology.orchestra.run.smithi047.stdout:fi 2022-09-07T21:09:59.371 INFO:teuthology.orchestra.run.smithi047.stdout:preuninstall scriptlet (using /bin/sh): 2022-09-07T21:09:59.371 INFO:teuthology.orchestra.run.smithi047.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2022-09-07T21:09:59.371 INFO:teuthology.orchestra.run.smithi047.stdout:new-kernel-pkg --remove 6.0.0-rc4-ceph-ga127ffb7df2a --rminitrd --initrdfile=/boot/initramfs-6.0.0-rc4-ceph-ga127ffb7df2a.img 2022-09-07T21:09:59.372 INFO:teuthology.orchestra.run.smithi047.stdout:elif [ -x /usr/bin/kernel-install ]; then 2022-09-07T21:09:59.372 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-install remove 6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-07T21:09:59.372 INFO:teuthology.orchestra.run.smithi047.stdout:fi 2022-09-07T21:09:59.372 INFO:teuthology.orchestra.run.smithi047.stdout:postuninstall scriptlet (using /bin/sh): 2022-09-07T21:09:59.372 INFO:teuthology.orchestra.run.smithi047.stdout:if [ -x /sbin/update-bootloader ]; then 2022-09-07T21:09:59.373 INFO:teuthology.orchestra.run.smithi047.stdout:/sbin/update-bootloader --remove 6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-07T21:09:59.373 INFO:teuthology.orchestra.run.smithi047.stdout:fi 2022-09-07T21:09:59.374 DEBUG:teuthology.orchestra.run.smithi047:> sudo rpm -qi grub2-tools 2022-09-07T21:09:59.455 INFO:teuthology.orchestra.run.smithi047.stdout:Name : grub2-tools 2022-09-07T21:09:59.456 INFO:teuthology.orchestra.run.smithi047.stdout:Epoch : 1 2022-09-07T21:09:59.456 INFO:teuthology.orchestra.run.smithi047.stdout:Version : 2.02 2022-09-07T21:09:59.457 INFO:teuthology.orchestra.run.smithi047.stdout:Release : 123.el8 2022-09-07T21:09:59.457 INFO:teuthology.orchestra.run.smithi047.stdout:Architecture: x86_64 2022-09-07T21:09:59.457 INFO:teuthology.orchestra.run.smithi047.stdout:Install Date: Thu 14 Jul 2022 06:22:19 PM UTC 2022-09-07T21:09:59.457 INFO:teuthology.orchestra.run.smithi047.stdout:Group : System Environment/Base 2022-09-07T21:09:59.458 INFO:teuthology.orchestra.run.smithi047.stdout:Size : 9265450 2022-09-07T21:09:59.458 INFO:teuthology.orchestra.run.smithi047.stdout:License : GPLv3+ 2022-09-07T21:09:59.458 INFO:teuthology.orchestra.run.smithi047.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2022-09-07T21:09:59.459 INFO:teuthology.orchestra.run.smithi047.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2022-09-07T21:09:59.459 INFO:teuthology.orchestra.run.smithi047.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2022-09-07T21:09:59.459 INFO:teuthology.orchestra.run.smithi047.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2022-09-07T21:09:59.459 INFO:teuthology.orchestra.run.smithi047.stdout:Relocations : (not relocatable) 2022-09-07T21:09:59.460 INFO:teuthology.orchestra.run.smithi047.stdout:Packager : Red Hat, Inc. 2022-09-07T21:09:59.460 INFO:teuthology.orchestra.run.smithi047.stdout:Vendor : Red Hat, Inc. 2022-09-07T21:09:59.460 INFO:teuthology.orchestra.run.smithi047.stdout:URL : http://www.gnu.org/software/grub/ 2022-09-07T21:09:59.461 INFO:teuthology.orchestra.run.smithi047.stdout:Summary : Support tools for GRUB. 2022-09-07T21:09:59.461 INFO:teuthology.orchestra.run.smithi047.stdout:Description : 2022-09-07T21:09:59.461 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:09:59.461 INFO:teuthology.orchestra.run.smithi047.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2022-09-07T21:09:59.462 INFO:teuthology.orchestra.run.smithi047.stdout:customizable bootloader with modular architecture. It supports a rich 2022-09-07T21:09:59.462 INFO:teuthology.orchestra.run.smithi047.stdout:variety of kernel formats, file systems, computer architectures and 2022-09-07T21:09:59.462 INFO:teuthology.orchestra.run.smithi047.stdout:hardware devices. 2022-09-07T21:09:59.462 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:09:59.462 INFO:teuthology.orchestra.run.smithi047.stdout:This subpackage provides tools for support of all platforms. 2022-09-07T21:09:59.464 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2022-09-07T21:09:59.464 INFO:teuthology.task.kernel:Updating grub on smithi047 to boot 6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-07T21:09:59.464 DEBUG:teuthology.orchestra.run.smithi047:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2022-09-07T21:10:02.446 INFO:teuthology.orchestra.run.smithi047.stderr:Generating grub configuration file ... 2022-09-07T21:10:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:done 2022-09-07T21:10:04.252 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2022-09-07T21:10:04.269 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.uLvgjDpPac 2022-09-07T21:10:04.270 DEBUG:teuthology.orchestra.run.smithi047:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uLvgjDpPac 2022-09-07T21:10:04.346 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.uLvgjDpPac 2022-09-07T21:10:04.502 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.uLvgjDpPac is 6KB 2022-09-07T21:10:04.553 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.uLvgjDpPac 2022-09-07T21:10:04.571 DEBUG:teuthology.orchestra.run.smithi047:> sudo /bin/ls /boot/loader/entries || true 2022-09-07T21:10:04.636 INFO:teuthology.orchestra.run.smithi047.stdout:5a84457b7b52416e95d47d02e2e4a60f-0-rescue.conf 2022-09-07T21:10:04.637 INFO:teuthology.orchestra.run.smithi047.stdout:5a84457b7b52416e95d47d02e2e4a60f-4.18.0-372.9.1.el8.x86_64.conf 2022-09-07T21:10:04.637 INFO:teuthology.orchestra.run.smithi047.stdout:5a84457b7b52416e95d47d02e2e4a60f-6.0.0-rc4-ceph-ga127ffb7df2a.conf 2022-09-07T21:10:04.638 DEBUG:teuthology.orchestra.run.smithi047:> sudo grub2-set-default 5a84457b7b52416e95d47d02e2e4a60f-6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-07T21:10:04.806 DEBUG:teuthology.orchestra.run.smithi047:> sudo shutdown -r now 2022-09-07T21:10:34.811 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2022-09-07T21:10:34.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2022-09-07T21:10:34.813 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-07T21:11:34.815 DEBUG:teuthology.orchestra.remote:timed out 2022-09-07T21:12:04.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2022-09-07T21:12:04.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-07T21:12:05.188 DEBUG:teuthology.orchestra.run.smithi047:> true 2022-09-07T21:12:05.847 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2022-09-07T21:12:05.847 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.0.0-rc4-ceph-ga127ffb7df2a"... 2022-09-07T21:12:05.848 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2022-09-07T21:12:05.905 INFO:teuthology.orchestra.run.smithi047.stdout:6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-07T21:12:05.906 DEBUG:teuthology.task.kernel:current kernel version is 6.0.0-rc4-ceph-ga127ffb7df2a vs 6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-07T21:12:05.906 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-09-07T21:12:05.907 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2022-09-07T21:12:06.908 DEBUG:teuthology.parallel:result is None 2022-09-07T21:12:06.909 INFO:teuthology.run_tasks:Running task internal.base... 2022-09-07T21:12:06.921 INFO:teuthology.task.internal:Creating test directory... 2022-09-07T21:12:06.922 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-09-07T21:12:06.928 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-09-07T21:12:06.932 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-09-07T21:12:06.984 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-09-07T21:12:06.992 INFO:teuthology.run_tasks:Running task internal.archive... 2022-09-07T21:12:06.999 INFO:teuthology.task.internal:Creating archive directory... 2022-09-07T21:12:06.999 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-09-07T21:12:07.004 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-09-07T21:12:07.007 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-09-07T21:12:07.076 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-09-07T21:12:07.083 INFO:teuthology.task.internal:Enabling coredump saving... 2022-09-07T21:12:07.084 DEBUG:teuthology.orchestra.run.smithi055:> test -f /run/.containerenv -o -f /.dockerenv 2022-09-07T21:12:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-07T21:12:07.104 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2022-09-07T21:12:07.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-07T21:12:07.125 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-09-07T21:12:07.131 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-09-07T21:12:07.147 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-09-07T21:12:07.195 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-07T21:12:07.215 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-07T21:12:07.218 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-07T21:12:07.235 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-07T21:12:07.280 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-07T21:12:07.320 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-07T21:12:07.328 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-09-07T21:12:07.336 INFO:teuthology.task.internal:Configuring sudo... 2022-09-07T21:12:07.338 DEBUG:teuthology.orchestra.run.smithi047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-09-07T21:12:07.343 DEBUG:teuthology.orchestra.run.smithi055:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-09-07T21:12:07.346 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-09-07T21:12:07.427 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-09-07T21:12:07.442 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-09-07T21:12:07.443 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-09-07T21:12:07.448 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-09-07T21:12:07.451 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-09-07T21:12:07.500 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-07T21:12:07.589 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-07T21:12:07.702 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-07T21:12:07.794 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-07T21:12:07.882 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:12:07.883 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-09-07T21:12:07.972 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-07T21:12:08.026 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-07T21:12:08.080 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-07T21:12:08.168 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-07T21:12:08.214 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2022-09-07T21:12:08.214 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-09-07T21:12:08.284 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-07T21:12:08.333 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-07T21:12:08.396 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-07T21:12:08.442 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-07T21:12:08.488 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-09-07T21:12:08.489 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-09-07T21:12:08.558 DEBUG:teuthology.orchestra.run.smithi047:> sudo service rsyslog restart 2022-09-07T21:12:08.563 DEBUG:teuthology.orchestra.run.smithi055:> sudo service rsyslog restart 2022-09-07T21:12:08.566 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2022-09-07T21:12:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-09-07T21:12:08.716 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-09-07T21:12:08.747 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-09-07T21:12:08.994 INFO:teuthology.run_tasks:Running task internal.timer... 2022-09-07T21:12:09.003 INFO:teuthology.task.internal:Starting timer... 2022-09-07T21:12:09.003 INFO:teuthology.run_tasks:Running task pcp... 2022-09-07T21:12:09.015 INFO:teuthology.run_tasks:Running task selinux... 2022-09-07T21:12:09.024 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-09-07T21:12:09.025 DEBUG:teuthology.orchestra.run.smithi047:> sudo service auditd rotate 2022-09-07T21:12:09.178 INFO:teuthology.orchestra.run.smithi047.stdout:Rotating logs: 2022-09-07T21:12:09.184 DEBUG:teuthology.orchestra.run.smithi055:> sudo service auditd rotate 2022-09-07T21:12:09.239 INFO:teuthology.orchestra.run.smithi055.stdout:Rotating logs: 2022-09-07T21:12:09.241 DEBUG:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2022-09-07T21:12:09.308 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: 2022-09-07T21:12:09.310 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-09-07T21:12:09.311 DEBUG:teuthology.orchestra.run.smithi047:> /usr/sbin/getenforce 2022-09-07T21:12:09.377 INFO:teuthology.orchestra.run.smithi047.stdout:Permissive 2022-09-07T21:12:09.379 DEBUG:teuthology.orchestra.run.smithi055:> /usr/sbin/getenforce 2022-09-07T21:12:09.404 INFO:teuthology.orchestra.run.smithi055.stdout:Permissive 2022-09-07T21:12:09.404 DEBUG:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2022-09-07T21:12:09.433 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2022-09-07T21:12:09.434 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi047.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi055.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi059.front.sepia.ceph.com': 'permissive'} 2022-09-07T21:12:09.434 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-09-07T21:12:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-07T21:12:09.531 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-09-07T21:12:09.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-07T21:12:09.562 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-09-07T21:12:09.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-07T21:12:09.593 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-09-07T21:12:09.593 DEBUG:teuthology.orchestra.run.smithi047:> sudo /usr/sbin/setenforce permissive 2022-09-07T21:12:09.684 DEBUG:teuthology.orchestra.run.smithi055:> sudo /usr/sbin/setenforce permissive 2022-09-07T21:12:09.713 DEBUG:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2022-09-07T21:12:09.743 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-09-07T21:12:09.755 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-09-07T21:12:09.768 INFO:teuthology.repo_utils:Fetching main from origin 2022-09-07T21:12:09.813 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-09-07T21:12:09.848 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-09-07T21:12:09.851 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi047.front.sepia.ceph.com,smithi055.front.sepia.ceph.com,smithi059.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-09-07T21:24:21.019 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi047.front.sepia.ceph.com'), Remote(name='ubuntu@smithi055.front.sepia.ceph.com'), Remote(name='ubuntu@smithi059.front.sepia.ceph.com')] 2022-09-07T21:24:21.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2022-09-07T21:24:21.050 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-07T21:24:21.166 DEBUG:teuthology.orchestra.run.smithi047:> true 2022-09-07T21:24:21.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2022-09-07T21:24:21.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-07T21:24:21.392 DEBUG:teuthology.orchestra.run.smithi055:> true 2022-09-07T21:24:21.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2022-09-07T21:24:21.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-07T21:24:21.547 DEBUG:teuthology.orchestra.run.smithi059:> true 2022-09-07T21:24:21.623 INFO:teuthology.run_tasks:Running task clock... 2022-09-07T21:24:21.639 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-09-07T21:24:21.640 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-09-07T21:24:21.643 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-09-07T21:24:21.649 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-09-07T21:24:21.650 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-09-07T21:24:21.654 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-09-07T21:24:21.655 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-09-07T21:24:21.690 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-09-07T21:24:21.702 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-09-07T21:24:21.707 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-09-07T21:24:21.720 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-09-07T21:24:21.738 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-09-07T21:24:21.779 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-09-07T21:24:21.783 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: ntpd: command not found 2022-09-07T21:24:21.796 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2022-09-07T21:24:21.798 INFO:teuthology.orchestra.run.smithi055.stdout:506 Cannot talk to daemon 2022-09-07T21:24:21.809 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2022-09-07T21:24:21.813 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-09-07T21:24:21.824 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-09-07T21:24:21.828 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-09-07T21:24:21.839 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-09-07T21:24:21.893 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: ntpd: command not found 2022-09-07T21:24:21.908 INFO:teuthology.orchestra.run.smithi055.stderr:bash: ntpq: command not found 2022-09-07T21:24:21.914 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-09-07T21:24:21.914 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2022-09-07T21:24:21.914 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-07T21:24:21.915 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-07T21:24:21.915 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-07T21:24:21.916 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-07T21:24:21.917 INFO:teuthology.orchestra.run.smithi059.stderr:bash: ntpq: command not found 2022-09-07T21:24:21.923 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-09-07T21:24:21.923 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2022-09-07T21:24:21.923 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-07T21:24:21.923 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-07T21:24:21.924 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-07T21:24:21.924 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-07T21:24:21.928 INFO:teuthology.orchestra.run.smithi047.stdout:506 Cannot talk to daemon 2022-09-07T21:24:21.969 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-09-07T21:24:22.011 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-09-07T21:24:22.135 INFO:teuthology.orchestra.run.smithi047.stderr:bash: ntpq: command not found 2022-09-07T21:24:22.144 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-09-07T21:24:22.145 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2022-09-07T21:24:22.145 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-07T21:24:22.145 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-07T21:24:22.146 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-07T21:24:22.146 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-07T21:24:22.147 INFO:teuthology.run_tasks:Running task pexec... 2022-09-07T21:24:22.163 INFO:teuthology.task.pexec:Executing custom commands... 2022-09-07T21:24:22.164 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi047.front.sepia.ceph.com 2022-09-07T21:24:22.165 DEBUG:teuthology.orchestra.run.smithi047:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-09-07T21:24:22.165 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi055.front.sepia.ceph.com 2022-09-07T21:24:22.166 DEBUG:teuthology.orchestra.run.smithi055:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-09-07T21:24:22.166 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi059.front.sepia.ceph.com 2022-09-07T21:24:22.167 DEBUG:teuthology.orchestra.run.smithi059:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-09-07T21:24:22.563 INFO:teuthology.orchestra.run.smithi059.stdout:Updating Subscription Management repositories. 2022-09-07T21:24:22.563 INFO:teuthology.orchestra.run.smithi055.stdout:Updating Subscription Management repositories. 2022-09-07T21:24:22.726 INFO:teuthology.orchestra.run.smithi047.stdout:Updating Subscription Management repositories. 2022-09-07T21:24:23.076 INFO:teuthology.orchestra.run.smithi059.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 158 kB/s | 2.8 kB 00:00 2022-09-07T21:24:23.134 INFO:teuthology.orchestra.run.smithi055.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 131 kB/s | 2.4 kB 00:00 2022-09-07T21:24:23.326 INFO:teuthology.orchestra.run.smithi055.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 152 kB/s | 2.8 kB 00:00 2022-09-07T21:24:23.369 INFO:teuthology.orchestra.run.smithi059.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 130 kB/s | 2.4 kB 00:00 2022-09-07T21:24:23.445 INFO:teuthology.orchestra.run.smithi047.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 127 kB/s | 2.4 kB 00:00 2022-09-07T21:24:23.701 INFO:teuthology.orchestra.run.smithi047.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 111 kB/s | 2.8 kB 00:00 2022-09-07T21:24:25.428 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2022-09-07T21:24:25.429 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2022-09-07T21:24:25.429 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2022-09-07T21:24:25.429 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2022-09-07T21:24:25.430 INFO:teuthology.orchestra.run.smithi059.stdout:Resetting modules: 2022-09-07T21:24:25.430 INFO:teuthology.orchestra.run.smithi059.stdout: container-tools 2022-09-07T21:24:25.430 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:24:25.430 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2022-09-07T21:24:25.431 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2022-09-07T21:24:25.431 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:24:25.465 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2022-09-07T21:24:25.466 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2022-09-07T21:24:25.466 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2022-09-07T21:24:25.466 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2022-09-07T21:24:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:Resetting modules: 2022-09-07T21:24:25.467 INFO:teuthology.orchestra.run.smithi055.stdout: container-tools 2022-09-07T21:24:25.467 INFO:teuthology.orchestra.run.smithi055.stdout: 2022-09-07T21:24:25.467 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2022-09-07T21:24:25.468 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2022-09-07T21:24:25.468 INFO:teuthology.orchestra.run.smithi055.stdout: 2022-09-07T21:24:25.667 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2022-09-07T21:24:25.681 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2022-09-07T21:24:25.994 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2022-09-07T21:24:25.996 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2022-09-07T21:24:25.996 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2022-09-07T21:24:25.996 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2022-09-07T21:24:25.997 INFO:teuthology.orchestra.run.smithi047.stdout:Resetting modules: 2022-09-07T21:24:25.997 INFO:teuthology.orchestra.run.smithi047.stdout: container-tools 2022-09-07T21:24:25.997 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:24:25.997 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2022-09-07T21:24:25.998 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2022-09-07T21:24:25.998 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:24:26.120 INFO:teuthology.orchestra.run.smithi059.stdout:Updating Subscription Management repositories. 2022-09-07T21:24:26.144 INFO:teuthology.orchestra.run.smithi055.stdout:Updating Subscription Management repositories. 2022-09-07T21:24:26.296 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2022-09-07T21:24:26.630 INFO:teuthology.orchestra.run.smithi059.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 140 kB/s | 2.8 kB 00:00 2022-09-07T21:24:26.677 INFO:teuthology.orchestra.run.smithi055.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 129 kB/s | 2.4 kB 00:00 2022-09-07T21:24:26.878 INFO:teuthology.orchestra.run.smithi055.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 146 kB/s | 2.8 kB 00:00 2022-09-07T21:24:26.936 INFO:teuthology.orchestra.run.smithi059.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 134 kB/s | 2.4 kB 00:00 2022-09-07T21:24:26.943 INFO:teuthology.orchestra.run.smithi047.stdout:Updating Subscription Management repositories. 2022-09-07T21:24:27.689 INFO:teuthology.orchestra.run.smithi047.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 117 kB/s | 2.4 kB 00:00 2022-09-07T21:24:27.937 INFO:teuthology.orchestra.run.smithi047.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 150 kB/s | 2.8 kB 00:00 2022-09-07T21:24:28.928 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2022-09-07T21:24:28.931 INFO:teuthology.orchestra.run.smithi059.stdout:========================================================================================================================= 2022-09-07T21:24:28.931 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2022-09-07T21:24:28.931 INFO:teuthology.orchestra.run.smithi059.stdout:========================================================================================================================= 2022-09-07T21:24:28.931 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2022-09-07T21:24:28.932 INFO:teuthology.orchestra.run.smithi059.stdout: conmon x86_64 2:2.1.2-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 55 k 2022-09-07T21:24:28.932 INFO:teuthology.orchestra.run.smithi059.stdout: container-selinux noarch 2:2.188.0-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 59 k 2022-09-07T21:24:28.932 INFO:teuthology.orchestra.run.smithi059.stdout: containernetworking-plugins x86_64 1:1.1.1-3.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 18 M 2022-09-07T21:24:28.932 INFO:teuthology.orchestra.run.smithi059.stdout: containers-common x86_64 2:1-35.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 109 k 2022-09-07T21:24:28.932 INFO:teuthology.orchestra.run.smithi059.stdout: criu x86_64 3.15-3.module+el8.6.0+15875+dc9a2b96 rhel-8-for-x86_64-appstream-rpms 518 k 2022-09-07T21:24:28.933 INFO:teuthology.orchestra.run.smithi059.stdout: fuse-overlayfs x86_64 1.9-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 73 k 2022-09-07T21:24:28.933 INFO:teuthology.orchestra.run.smithi059.stdout: libslirp x86_64 4.4.0-1.module+el8.6.0+15875+dc9a2b96 rhel-8-for-x86_64-appstream-rpms 70 k 2022-09-07T21:24:28.933 INFO:teuthology.orchestra.run.smithi059.stdout: runc x86_64 1:1.1.3-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 3.0 M 2022-09-07T21:24:28.933 INFO:teuthology.orchestra.run.smithi059.stdout: slirp4netns x86_64 1.2.0-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-09-07T21:24:28.933 INFO:teuthology.orchestra.run.smithi059.stdout:Installing group/module packages: 2022-09-07T21:24:28.934 INFO:teuthology.orchestra.run.smithi059.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 8.0 M 2022-09-07T21:24:28.934 INFO:teuthology.orchestra.run.smithi059.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 524 k 2022-09-07T21:24:28.934 INFO:teuthology.orchestra.run.smithi059.stdout: crun x86_64 1.4.5-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 209 k 2022-09-07T21:24:28.934 INFO:teuthology.orchestra.run.smithi059.stdout: python3-podman noarch 4.0.0-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 149 k 2022-09-07T21:24:28.935 INFO:teuthology.orchestra.run.smithi059.stdout: skopeo x86_64 2:1.8.0-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 6.9 M 2022-09-07T21:24:28.935 INFO:teuthology.orchestra.run.smithi059.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 2.2 M 2022-09-07T21:24:28.935 INFO:teuthology.orchestra.run.smithi059.stdout: udica noarch 0.2.6-4.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 49 k 2022-09-07T21:24:28.935 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2022-09-07T21:24:28.935 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2022-09-07T21:24:28.936 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2022-09-07T21:24:28.936 INFO:teuthology.orchestra.run.smithi059.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-09-07T21:24:28.936 INFO:teuthology.orchestra.run.smithi059.stdout:Installing module profiles: 2022-09-07T21:24:28.936 INFO:teuthology.orchestra.run.smithi059.stdout: container-tools/common 2022-09-07T21:24:28.937 INFO:teuthology.orchestra.run.smithi059.stdout:Enabling module streams: 2022-09-07T21:24:28.937 INFO:teuthology.orchestra.run.smithi059.stdout: container-tools rhel8 2022-09-07T21:24:28.937 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:24:28.937 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2022-09-07T21:24:28.937 INFO:teuthology.orchestra.run.smithi059.stdout:========================================================================================================================= 2022-09-07T21:24:28.938 INFO:teuthology.orchestra.run.smithi059.stdout:Install 10 Packages 2022-09-07T21:24:28.938 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 9 Packages 2022-09-07T21:24:28.938 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:24:28.939 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 40 M 2022-09-07T21:24:28.939 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2022-09-07T21:24:28.981 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2022-09-07T21:24:28.984 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================================================= 2022-09-07T21:24:28.984 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2022-09-07T21:24:28.985 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================================================= 2022-09-07T21:24:28.985 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrading: 2022-09-07T21:24:28.985 INFO:teuthology.orchestra.run.smithi055.stdout: conmon x86_64 2:2.1.2-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 55 k 2022-09-07T21:24:28.985 INFO:teuthology.orchestra.run.smithi055.stdout: container-selinux noarch 2:2.188.0-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 59 k 2022-09-07T21:24:28.986 INFO:teuthology.orchestra.run.smithi055.stdout: containernetworking-plugins x86_64 1:1.1.1-3.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 18 M 2022-09-07T21:24:28.986 INFO:teuthology.orchestra.run.smithi055.stdout: containers-common x86_64 2:1-35.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 109 k 2022-09-07T21:24:28.986 INFO:teuthology.orchestra.run.smithi055.stdout: criu x86_64 3.15-3.module+el8.6.0+15875+dc9a2b96 rhel-8-for-x86_64-appstream-rpms 518 k 2022-09-07T21:24:28.987 INFO:teuthology.orchestra.run.smithi055.stdout: fuse-overlayfs x86_64 1.9-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 73 k 2022-09-07T21:24:28.987 INFO:teuthology.orchestra.run.smithi055.stdout: libslirp x86_64 4.4.0-1.module+el8.6.0+15875+dc9a2b96 rhel-8-for-x86_64-appstream-rpms 70 k 2022-09-07T21:24:28.987 INFO:teuthology.orchestra.run.smithi055.stdout: runc x86_64 1:1.1.3-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 3.0 M 2022-09-07T21:24:28.987 INFO:teuthology.orchestra.run.smithi055.stdout: slirp4netns x86_64 1.2.0-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-09-07T21:24:28.987 INFO:teuthology.orchestra.run.smithi055.stdout:Installing group/module packages: 2022-09-07T21:24:28.988 INFO:teuthology.orchestra.run.smithi055.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 8.0 M 2022-09-07T21:24:28.988 INFO:teuthology.orchestra.run.smithi055.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 524 k 2022-09-07T21:24:28.988 INFO:teuthology.orchestra.run.smithi055.stdout: crun x86_64 1.4.5-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 209 k 2022-09-07T21:24:28.988 INFO:teuthology.orchestra.run.smithi055.stdout: python3-podman noarch 4.0.0-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 149 k 2022-09-07T21:24:28.988 INFO:teuthology.orchestra.run.smithi055.stdout: skopeo x86_64 2:1.8.0-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 6.9 M 2022-09-07T21:24:28.989 INFO:teuthology.orchestra.run.smithi055.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 2.2 M 2022-09-07T21:24:28.989 INFO:teuthology.orchestra.run.smithi055.stdout: udica noarch 0.2.6-4.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 49 k 2022-09-07T21:24:28.989 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2022-09-07T21:24:28.989 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2022-09-07T21:24:28.990 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2022-09-07T21:24:28.990 INFO:teuthology.orchestra.run.smithi055.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-09-07T21:24:28.990 INFO:teuthology.orchestra.run.smithi055.stdout:Installing module profiles: 2022-09-07T21:24:28.990 INFO:teuthology.orchestra.run.smithi055.stdout: container-tools/common 2022-09-07T21:24:28.990 INFO:teuthology.orchestra.run.smithi055.stdout:Enabling module streams: 2022-09-07T21:24:28.991 INFO:teuthology.orchestra.run.smithi055.stdout: container-tools rhel8 2022-09-07T21:24:28.991 INFO:teuthology.orchestra.run.smithi055.stdout: 2022-09-07T21:24:28.991 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2022-09-07T21:24:28.991 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================================================= 2022-09-07T21:24:28.991 INFO:teuthology.orchestra.run.smithi055.stdout:Install 10 Packages 2022-09-07T21:24:28.992 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrade 9 Packages 2022-09-07T21:24:28.992 INFO:teuthology.orchestra.run.smithi055.stdout: 2022-09-07T21:24:28.993 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 40 M 2022-09-07T21:24:28.993 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2022-09-07T21:24:29.002 INFO:teuthology.orchestra.run.smithi059.stdout:(1/19): python3-pytoml-0.1.14-5.git7dea353.el8. 372 kB/s | 25 kB 00:00 2022-09-07T21:24:29.027 INFO:teuthology.orchestra.run.smithi059.stdout:(2/19): yajl-2.1.0-10.el8.x86_64.rpm 443 kB/s | 41 kB 00:00 2022-09-07T21:24:29.050 INFO:teuthology.orchestra.run.smithi055.stdout:(1/19): yajl-2.1.0-10.el8.x86_64.rpm 662 kB/s | 41 kB 00:00 2022-09-07T21:24:29.052 INFO:teuthology.orchestra.run.smithi059.stdout:(3/19): python3-pyxdg-0.25-16.el8.noarch.rpm 799 kB/s | 94 kB 00:00 2022-09-07T21:24:29.076 INFO:teuthology.orchestra.run.smithi055.stdout:(2/19): python3-pyxdg-0.25-16.el8.noarch.rpm 1.1 MB/s | 94 kB 00:00 2022-09-07T21:24:29.086 INFO:teuthology.orchestra.run.smithi059.stdout:(4/19): udica-0.2.6-4.module+el8.6.0+15917+093c 1.5 MB/s | 49 kB 00:00 2022-09-07T21:24:29.101 INFO:teuthology.orchestra.run.smithi055.stdout:(3/19): python3-pytoml-0.1.14-5.git7dea353.el8. 221 kB/s | 25 kB 00:00 2022-09-07T21:24:29.111 INFO:teuthology.orchestra.run.smithi059.stdout:(5/19): cockpit-podman-49.1-1.module+el8.6.0+15 6.1 MB/s | 524 kB 00:00 2022-09-07T21:24:29.143 INFO:teuthology.orchestra.run.smithi055.stdout:(4/19): udica-0.2.6-4.module+el8.6.0+15917+093c 1.2 MB/s | 49 kB 00:00 2022-09-07T21:24:29.177 INFO:teuthology.orchestra.run.smithi055.stdout:(5/19): cockpit-podman-49.1-1.module+el8.6.0+15 5.1 MB/s | 524 kB 00:00 2022-09-07T21:24:29.212 INFO:teuthology.orchestra.run.smithi059.stdout:(6/19): toolbox-0.0.99.3-0.6.module+el8.6.0+159 10 MB/s | 2.2 MB 00:00 2022-09-07T21:24:29.227 INFO:teuthology.orchestra.run.smithi055.stdout:(6/19): toolbox-0.0.99.3-0.6.module+el8.6.0+159 12 MB/s | 2.2 MB 00:00 2022-09-07T21:24:29.253 INFO:teuthology.orchestra.run.smithi059.stdout:(7/19): crun-1.4.5-2.module+el8.6.0+15917+093ca 1.5 MB/s | 209 kB 00:00 2022-09-07T21:24:29.261 INFO:teuthology.orchestra.run.smithi055.stdout:(7/19): crun-1.4.5-2.module+el8.6.0+15917+093ca 2.5 MB/s | 209 kB 00:00 2022-09-07T21:24:29.286 INFO:teuthology.orchestra.run.smithi055.stdout:(8/19): python3-podman-4.0.0-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2022-09-07T21:24:29.287 INFO:teuthology.orchestra.run.smithi059.stdout:(8/19): python3-podman-4.0.0-1.module+el8.6.0+1 1.9 MB/s | 149 kB 00:00 2022-09-07T21:24:29.312 INFO:teuthology.orchestra.run.smithi059.stdout:(9/19): container-selinux-2.188.0-1.module+el8. 2.3 MB/s | 59 kB 00:00 2022-09-07T21:24:29.349 INFO:teuthology.orchestra.run.smithi055.stdout:(9/19): container-selinux-2.188.0-1.module+el8. 959 kB/s | 59 kB 00:00 2022-09-07T21:24:29.471 INFO:teuthology.orchestra.run.smithi059.stdout:(10/19): buildah-1.26.2-1.module+el8.6.0+15917+ 21 MB/s | 8.0 MB 00:00 2022-09-07T21:24:29.486 INFO:teuthology.orchestra.run.smithi055.stdout:(10/19): buildah-1.26.2-1.module+el8.6.0+15917+ 23 MB/s | 8.0 MB 00:00 2022-09-07T21:24:29.530 INFO:teuthology.orchestra.run.smithi059.stdout:(11/19): runc-1.1.3-2.module+el8.6.0+15917+093c 14 MB/s | 3.0 MB 00:00 2022-09-07T21:24:29.545 INFO:teuthology.orchestra.run.smithi055.stdout:(11/19): runc-1.1.3-2.module+el8.6.0+15917+093c 15 MB/s | 3.0 MB 00:00 2022-09-07T21:24:29.637 INFO:teuthology.orchestra.run.smithi055.stdout:(12/19): skopeo-1.8.0-2.module+el8.6.0+15917+09 18 MB/s | 6.9 MB 00:00 2022-09-07T21:24:29.655 INFO:teuthology.orchestra.run.smithi059.stdout:(12/19): skopeo-1.8.0-2.module+el8.6.0+15917+09 17 MB/s | 6.9 MB 00:00 2022-09-07T21:24:29.680 INFO:teuthology.orchestra.run.smithi059.stdout:(13/19): conmon-2.1.2-2.module+el8.6.0+15917+09 266 kB/s | 55 kB 00:00 2022-09-07T21:24:29.685 INFO:teuthology.orchestra.run.smithi055.stdout:(13/19): conmon-2.1.2-2.module+el8.6.0+15917+09 279 kB/s | 55 kB 00:00 2022-09-07T21:24:29.706 INFO:teuthology.orchestra.run.smithi059.stdout:(14/19): fuse-overlayfs-1.9-1.module+el8.6.0+15 419 kB/s | 73 kB 00:00 2022-09-07T21:24:29.721 INFO:teuthology.orchestra.run.smithi055.stdout:(14/19): fuse-overlayfs-1.9-1.module+el8.6.0+15 420 kB/s | 73 kB 00:00 2022-09-07T21:24:29.747 INFO:teuthology.orchestra.run.smithi059.stdout:(15/19): slirp4netns-1.2.0-2.module+el8.6.0+159 593 kB/s | 54 kB 00:00 2022-09-07T21:24:29.754 INFO:teuthology.orchestra.run.smithi055.stdout:(15/19): slirp4netns-1.2.0-2.module+el8.6.0+159 463 kB/s | 54 kB 00:00 2022-09-07T21:24:29.773 INFO:teuthology.orchestra.run.smithi059.stdout:(16/19): libslirp-4.4.0-1.module+el8.6.0+15875+ 763 kB/s | 70 kB 00:00 2022-09-07T21:24:29.780 INFO:teuthology.orchestra.run.smithi055.stdout:(16/19): libslirp-4.4.0-1.module+el8.6.0+15875+ 752 kB/s | 70 kB 00:00 2022-09-07T21:24:29.813 INFO:teuthology.orchestra.run.smithi055.stdout:(17/19): containers-common-1-35.module+el8.6.0+ 1.8 MB/s | 109 kB 00:00 2022-09-07T21:24:29.814 INFO:teuthology.orchestra.run.smithi059.stdout:(17/19): containers-common-1-35.module+el8.6.0+ 1.6 MB/s | 109 kB 00:00 2022-09-07T21:24:29.839 INFO:teuthology.orchestra.run.smithi055.stdout:(18/19): criu-3.15-3.module+el8.6.0+15875+dc9a2 8.6 MB/s | 518 kB 00:00 2022-09-07T21:24:29.871 INFO:teuthology.orchestra.run.smithi059.stdout:(18/19): criu-3.15-3.module+el8.6.0+15875+dc9a2 5.2 MB/s | 518 kB 00:00 2022-09-07T21:24:30.132 INFO:teuthology.orchestra.run.smithi059.stdout:(19/19): containernetworking-plugins-1.1.1-3.mo 43 MB/s | 18 MB 00:00 2022-09-07T21:24:30.133 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2022-09-07T21:24:30.134 INFO:teuthology.orchestra.run.smithi059.stdout:Total 34 MB/s | 40 MB 00:01 2022-09-07T21:24:30.139 INFO:teuthology.orchestra.run.smithi055.stdout:(19/19): containernetworking-plugins-1.1.1-3.mo 44 MB/s | 18 MB 00:00 2022-09-07T21:24:30.140 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2022-09-07T21:24:30.140 INFO:teuthology.orchestra.run.smithi055.stdout:Total 35 MB/s | 40 MB 00:01 2022-09-07T21:24:30.171 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2022-09-07T21:24:30.174 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================================================= 2022-09-07T21:24:30.175 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2022-09-07T21:24:30.175 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================================================= 2022-09-07T21:24:30.175 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrading: 2022-09-07T21:24:30.175 INFO:teuthology.orchestra.run.smithi047.stdout: conmon x86_64 2:2.1.2-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 55 k 2022-09-07T21:24:30.175 INFO:teuthology.orchestra.run.smithi047.stdout: container-selinux noarch 2:2.188.0-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 59 k 2022-09-07T21:24:30.176 INFO:teuthology.orchestra.run.smithi047.stdout: containernetworking-plugins x86_64 1:1.1.1-3.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 18 M 2022-09-07T21:24:30.176 INFO:teuthology.orchestra.run.smithi047.stdout: containers-common x86_64 2:1-35.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 109 k 2022-09-07T21:24:30.176 INFO:teuthology.orchestra.run.smithi047.stdout: criu x86_64 3.15-3.module+el8.6.0+15875+dc9a2b96 rhel-8-for-x86_64-appstream-rpms 518 k 2022-09-07T21:24:30.176 INFO:teuthology.orchestra.run.smithi047.stdout: fuse-overlayfs x86_64 1.9-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 73 k 2022-09-07T21:24:30.177 INFO:teuthology.orchestra.run.smithi047.stdout: libslirp x86_64 4.4.0-1.module+el8.6.0+15875+dc9a2b96 rhel-8-for-x86_64-appstream-rpms 70 k 2022-09-07T21:24:30.177 INFO:teuthology.orchestra.run.smithi047.stdout: runc x86_64 1:1.1.3-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 3.0 M 2022-09-07T21:24:30.177 INFO:teuthology.orchestra.run.smithi047.stdout: slirp4netns x86_64 1.2.0-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-09-07T21:24:30.177 INFO:teuthology.orchestra.run.smithi047.stdout:Installing group/module packages: 2022-09-07T21:24:30.177 INFO:teuthology.orchestra.run.smithi047.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 8.0 M 2022-09-07T21:24:30.178 INFO:teuthology.orchestra.run.smithi047.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 524 k 2022-09-07T21:24:30.178 INFO:teuthology.orchestra.run.smithi047.stdout: crun x86_64 1.4.5-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 209 k 2022-09-07T21:24:30.178 INFO:teuthology.orchestra.run.smithi047.stdout: python3-podman noarch 4.0.0-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 149 k 2022-09-07T21:24:30.178 INFO:teuthology.orchestra.run.smithi047.stdout: skopeo x86_64 2:1.8.0-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 6.9 M 2022-09-07T21:24:30.179 INFO:teuthology.orchestra.run.smithi047.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 2.2 M 2022-09-07T21:24:30.179 INFO:teuthology.orchestra.run.smithi047.stdout: udica noarch 0.2.6-4.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 49 k 2022-09-07T21:24:30.179 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2022-09-07T21:24:30.179 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2022-09-07T21:24:30.179 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2022-09-07T21:24:30.180 INFO:teuthology.orchestra.run.smithi047.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-09-07T21:24:30.180 INFO:teuthology.orchestra.run.smithi047.stdout:Installing module profiles: 2022-09-07T21:24:30.180 INFO:teuthology.orchestra.run.smithi047.stdout: container-tools/common 2022-09-07T21:24:30.180 INFO:teuthology.orchestra.run.smithi047.stdout:Enabling module streams: 2022-09-07T21:24:30.180 INFO:teuthology.orchestra.run.smithi047.stdout: container-tools rhel8 2022-09-07T21:24:30.181 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:24:30.181 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2022-09-07T21:24:30.181 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================================================= 2022-09-07T21:24:30.181 INFO:teuthology.orchestra.run.smithi047.stdout:Install 10 Packages 2022-09-07T21:24:30.181 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrade 9 Packages 2022-09-07T21:24:30.182 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:24:30.183 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 40 M 2022-09-07T21:24:30.183 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2022-09-07T21:24:30.281 INFO:teuthology.orchestra.run.smithi047.stdout:(1/19): python3-pyxdg-0.25-16.el8.noarch.rpm 932 kB/s | 94 kB 00:00 2022-09-07T21:24:30.302 INFO:teuthology.orchestra.run.smithi047.stdout:(2/19): python3-pytoml-0.1.14-5.git7dea353.el8. 204 kB/s | 25 kB 00:00 2022-09-07T21:24:30.319 INFO:teuthology.orchestra.run.smithi047.stdout:(3/19): yajl-2.1.0-10.el8.x86_64.rpm 295 kB/s | 41 kB 00:00 2022-09-07T21:24:30.353 INFO:teuthology.orchestra.run.smithi047.stdout:(4/19): udica-0.2.6-4.module+el8.6.0+15917+093c 1.5 MB/s | 49 kB 00:00 2022-09-07T21:24:30.403 INFO:teuthology.orchestra.run.smithi047.stdout:(5/19): cockpit-podman-49.1-1.module+el8.6.0+15 5.1 MB/s | 524 kB 00:00 2022-09-07T21:24:30.453 INFO:teuthology.orchestra.run.smithi047.stdout:(6/19): crun-1.4.5-2.module+el8.6.0+15917+093ca 4.1 MB/s | 209 kB 00:00 2022-09-07T21:24:30.520 INFO:teuthology.orchestra.run.smithi047.stdout:(7/19): toolbox-0.0.99.3-0.6.module+el8.6.0+159 9.2 MB/s | 2.2 MB 00:00 2022-09-07T21:24:30.558 INFO:teuthology.orchestra.run.smithi047.stdout:(8/19): python3-podman-4.0.0-1.module+el8.6.0+1 1.5 MB/s | 149 kB 00:00 2022-09-07T21:24:30.562 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2022-09-07T21:24:30.572 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2022-09-07T21:24:30.596 INFO:teuthology.orchestra.run.smithi047.stdout:(9/19): container-selinux-2.188.0-1.module+el8. 1.4 MB/s | 59 kB 00:00 2022-09-07T21:24:30.610 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2022-09-07T21:24:30.611 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2022-09-07T21:24:30.619 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2022-09-07T21:24:30.619 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2022-09-07T21:24:30.738 INFO:teuthology.orchestra.run.smithi047.stdout:(10/19): runc-1.1.3-2.module+el8.6.0+15917+093c 21 MB/s | 3.0 MB 00:00 2022-09-07T21:24:30.780 INFO:teuthology.orchestra.run.smithi047.stdout:(11/19): conmon-2.1.2-2.module+el8.6.0+15917+09 1.3 MB/s | 55 kB 00:00 2022-09-07T21:24:30.831 INFO:teuthology.orchestra.run.smithi047.stdout:(12/19): fuse-overlayfs-1.9-1.module+el8.6.0+15 1.4 MB/s | 73 kB 00:00 2022-09-07T21:24:30.971 INFO:teuthology.orchestra.run.smithi047.stdout:(13/19): buildah-1.26.2-1.module+el8.6.0+15917+ 13 MB/s | 8.0 MB 00:00 2022-09-07T21:24:30.983 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2022-09-07T21:24:30.987 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2022-09-07T21:24:30.993 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2022-09-07T21:24:30.997 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2022-09-07T21:24:31.006 INFO:teuthology.orchestra.run.smithi047.stdout:(14/19): slirp4netns-1.2.0-2.module+el8.6.0+159 309 kB/s | 54 kB 00:00 2022-09-07T21:24:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:(15/19): libslirp-4.4.0-1.module+el8.6.0+15875+ 924 kB/s | 70 kB 00:00 2022-09-07T21:24:31.148 INFO:teuthology.orchestra.run.smithi047.stdout:(16/19): skopeo-1.8.0-2.module+el8.6.0+15917+09 11 MB/s | 6.9 MB 00:00 2022-09-07T21:24:31.190 INFO:teuthology.orchestra.run.smithi047.stdout:(17/19): containers-common-1-35.module+el8.6.0+ 767 kB/s | 109 kB 00:00 2022-09-07T21:24:31.232 INFO:teuthology.orchestra.run.smithi047.stdout:(18/19): criu-3.15-3.module+el8.6.0+15875+dc9a2 6.1 MB/s | 518 kB 00:00 2022-09-07T21:24:31.574 INFO:teuthology.orchestra.run.smithi047.stdout:(19/19): containernetworking-plugins-1.1.1-3.mo 32 MB/s | 18 MB 00:00 2022-09-07T21:24:31.575 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2022-09-07T21:24:31.575 INFO:teuthology.orchestra.run.smithi047.stdout:Total 29 MB/s | 40 MB 00:01 2022-09-07T21:24:32.086 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2022-09-07T21:24:32.100 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917+0 1/1 2022-09-07T21:24:32.120 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-07T21:24:32.148 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2022-09-07T21:24:32.162 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917+0 1/1 2022-09-07T21:24:32.182 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-07T21:24:32.354 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-07T21:24:32.372 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-07T21:24:32.636 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2022-09-07T21:24:32.691 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2022-09-07T21:24:32.692 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2022-09-07T21:24:33.338 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2022-09-07T21:24:33.343 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2022-09-07T21:24:34.481 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2022-09-07T21:24:34.530 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917+0 1/1 2022-09-07T21:24:34.594 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-07T21:24:34.844 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-07T21:24:50.138 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-07T21:24:50.357 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-07T21:24:50.744 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 2/28 2022-09-07T21:24:50.917 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 2/28 2022-09-07T21:24:52.579 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_6 3/28 2022-09-07T21:24:52.740 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_6 3/28 2022-09-07T21:24:52.793 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : containernetworking-plugins-1:1.1.1-3.module+el8.6 4/28 2022-09-07T21:24:53.007 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86 5/28 2022-09-07T21:24:53.111 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : containernetworking-plugins-1:1.1.1-3.module+el8.6 4/28 2022-09-07T21:24:53.224 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8. 6/28 2022-09-07T21:24:53.326 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86 5/28 2022-09-07T21:24:53.360 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 7/28 2022-09-07T21:24:53.382 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 7/28 2022-09-07T21:24:53.569 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8. 6/28 2022-09-07T21:24:53.579 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : yajl-2.1.0-10.el8.x86_64 8/28 2022-09-07T21:24:53.758 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 9/28 2022-09-07T21:24:53.758 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : containers-common-2:1-35.module+el8.6.0+15917+093c 10/28 2022-09-07T21:24:53.759 INFO:teuthology.orchestra.run.smithi055.stdout:warning: /etc/containers/registries.conf created as /etc/containers/registries.conf.rpmnew 2022-09-07T21:24:53.759 INFO:teuthology.orchestra.run.smithi055.stdout: 2022-09-07T21:24:53.778 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 7/28 2022-09-07T21:24:53.800 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 7/28 2022-09-07T21:24:53.947 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : yajl-2.1.0-10.el8.x86_64 8/28 2022-09-07T21:24:54.133 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 9/28 2022-09-07T21:24:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : containers-common-2:1-35.module+el8.6.0+15917+093c 10/28 2022-09-07T21:24:54.134 INFO:teuthology.orchestra.run.smithi059.stdout:warning: /etc/containers/registries.conf created as /etc/containers/registries.conf.rpmnew 2022-09-07T21:24:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:24:54.226 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 11/28 2022-09-07T21:24:54.408 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 12/28 2022-09-07T21:24:54.585 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 11/28 2022-09-07T21:24:54.793 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 12/28 2022-09-07T21:24:54.832 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6 13/28 2022-09-07T21:24:55.200 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6 13/28 2022-09-07T21:24:55.734 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 14/28 2022-09-07T21:24:55.781 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-07T21:24:56.070 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 14/28 2022-09-07T21:24:56.467 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 2/28 2022-09-07T21:24:56.620 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x 15/28 2022-09-07T21:24:56.822 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86 16/28 2022-09-07T21:24:56.982 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x 15/28 2022-09-07T21:24:57.021 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 17/28 2022-09-07T21:24:57.198 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86 16/28 2022-09-07T21:24:57.253 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86 18/28 2022-09-07T21:24:57.414 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 17/28 2022-09-07T21:24:57.436 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f 19/28 2022-09-07T21:24:57.646 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : containers-common-2:1-27.module+el8.6.0+14877+f643 20/28 2022-09-07T21:24:57.654 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86 18/28 2022-09-07T21:24:57.837 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f 19/28 2022-09-07T21:24:57.877 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 21/28 2022-09-07T21:24:58.039 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : containers-common-2:1-27.module+el8.6.0+14877+f643 20/28 2022-09-07T21:24:58.060 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 22/28 2022-09-07T21:24:58.061 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/28 2022-09-07T21:24:58.237 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 21/28 2022-09-07T21:24:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/28 2022-09-07T21:24:58.378 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 22/28 2022-09-07T21:24:58.379 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/28 2022-09-07T21:24:58.530 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 24/28 2022-09-07T21:24:58.578 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/28 2022-09-07T21:24:58.608 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_6 3/28 2022-09-07T21:24:58.718 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 25/28 2022-09-07T21:24:58.773 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 24/28 2022-09-07T21:24:58.860 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : containernetworking-plugins-1:1.1.1-3.module+el8.6 4/28 2022-09-07T21:24:58.862 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 26/28 2022-09-07T21:24:58.911 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 25/28 2022-09-07T21:24:59.039 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 26/28 2022-09-07T21:24:59.049 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86 5/28 2022-09-07T21:24:59.076 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : containernetworking-plugins-1:1.0.1-2.module+el8.6 27/28 2022-09-07T21:24:59.243 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 28/28 2022-09-07T21:24:59.245 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : containernetworking-plugins-1:1.0.1-2.module+el8.6 27/28 2022-09-07T21:24:59.443 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8. 6/28 2022-09-07T21:24:59.453 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 28/28 2022-09-07T21:24:59.665 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 7/28 2022-09-07T21:24:59.725 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 7/28 2022-09-07T21:24:59.954 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : yajl-2.1.0-10.el8.x86_64 8/28 2022-09-07T21:25:00.151 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 9/28 2022-09-07T21:25:00.151 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : containers-common-2:1-35.module+el8.6.0+15917+093c 10/28 2022-09-07T21:25:00.151 INFO:teuthology.orchestra.run.smithi047.stdout:warning: /etc/containers/registries.conf created as /etc/containers/registries.conf.rpmnew 2022-09-07T21:25:00.151 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:25:00.748 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 11/28 2022-09-07T21:25:01.034 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 12/28 2022-09-07T21:25:01.509 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6 13/28 2022-09-07T21:25:02.576 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 14/28 2022-09-07T21:25:03.609 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x 15/28 2022-09-07T21:25:03.849 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86 16/28 2022-09-07T21:25:04.063 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 17/28 2022-09-07T21:25:04.348 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86 18/28 2022-09-07T21:25:04.534 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f 19/28 2022-09-07T21:25:04.816 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : containers-common-2:1-27.module+el8.6.0+14877+f643 20/28 2022-09-07T21:25:05.009 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 21/28 2022-09-07T21:25:05.232 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 22/28 2022-09-07T21:25:05.233 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/28 2022-09-07T21:25:05.467 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/28 2022-09-07T21:25:05.733 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 24/28 2022-09-07T21:25:05.932 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 25/28 2022-09-07T21:25:06.103 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 26/28 2022-09-07T21:25:06.349 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : containernetworking-plugins-1:1.0.1-2.module+el8.6 27/28 2022-09-07T21:25:06.548 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 28/28 2022-09-07T21:25:16.245 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 28/28 2022-09-07T21:25:16.434 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 28/28 2022-09-07T21:25:16.681 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 28/28 2022-09-07T21:25:16.681 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/28 2022-09-07T21:25:16.682 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2/28 2022-09-07T21:25:16.682 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 3/28 2022-09-07T21:25:16.682 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 4/28 2022-09-07T21:25:16.682 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f 5/28 2022-09-07T21:25:16.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 6/28 2022-09-07T21:25:16.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x 7/28 2022-09-07T21:25:16.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 8/28 2022-09-07T21:25:16.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6 9/28 2022-09-07T21:25:16.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86 10/28 2022-09-07T21:25:16.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : container-selinux-2:2.188.0-1.module+el8.6.0+15917 11/28 2022-09-07T21:25:16.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : container-selinux-2:2.179.1-1.module+el8.6.0+14877 12/28 2022-09-07T21:25:16.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_6 13/28 2022-09-07T21:25:16.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 14/28 2022-09-07T21:25:16.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86 15/28 2022-09-07T21:25:16.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 16/28 2022-09-07T21:25:16.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 17/28 2022-09-07T21:25:16.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 18/28 2022-09-07T21:25:16.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8. 19/28 2022-09-07T21:25:16.686 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 20/28 2022-09-07T21:25:16.686 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86 21/28 2022-09-07T21:25:16.686 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 22/28 2022-09-07T21:25:16.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : containernetworking-plugins-1:1.1.1-3.module+el8.6 23/28 2022-09-07T21:25:16.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : containernetworking-plugins-1:1.0.1-2.module+el8.6 24/28 2022-09-07T21:25:16.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : containers-common-2:1-35.module+el8.6.0+15917+093c 25/28 2022-09-07T21:25:16.688 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : containers-common-2:1-27.module+el8.6.0+14877+f643 26/28 2022-09-07T21:25:16.688 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 27/28 2022-09-07T21:25:16.863 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 28/28 2022-09-07T21:25:16.864 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/28 2022-09-07T21:25:16.864 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2/28 2022-09-07T21:25:16.864 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 3/28 2022-09-07T21:25:16.865 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 4/28 2022-09-07T21:25:16.865 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f 5/28 2022-09-07T21:25:16.865 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 6/28 2022-09-07T21:25:16.865 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x 7/28 2022-09-07T21:25:16.866 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 8/28 2022-09-07T21:25:16.866 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6 9/28 2022-09-07T21:25:16.866 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86 10/28 2022-09-07T21:25:16.866 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : container-selinux-2:2.188.0-1.module+el8.6.0+15917 11/28 2022-09-07T21:25:16.866 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : container-selinux-2:2.179.1-1.module+el8.6.0+14877 12/28 2022-09-07T21:25:16.867 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_6 13/28 2022-09-07T21:25:16.867 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 14/28 2022-09-07T21:25:16.867 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86 15/28 2022-09-07T21:25:16.867 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 16/28 2022-09-07T21:25:16.868 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 17/28 2022-09-07T21:25:16.868 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 18/28 2022-09-07T21:25:16.868 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8. 19/28 2022-09-07T21:25:16.868 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 20/28 2022-09-07T21:25:16.868 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86 21/28 2022-09-07T21:25:16.869 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 22/28 2022-09-07T21:25:16.869 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : containernetworking-plugins-1:1.1.1-3.module+el8.6 23/28 2022-09-07T21:25:16.869 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : containernetworking-plugins-1:1.0.1-2.module+el8.6 24/28 2022-09-07T21:25:16.869 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : containers-common-2:1-35.module+el8.6.0+15917+093c 25/28 2022-09-07T21:25:16.870 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : containers-common-2:1-27.module+el8.6.0+14877+f643 26/28 2022-09-07T21:25:16.870 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 27/28 2022-09-07T21:25:17.447 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 28/28 2022-09-07T21:25:17.448 INFO:teuthology.orchestra.run.smithi055.stdout:Installed products updated. 2022-09-07T21:25:17.678 INFO:teuthology.orchestra.run.smithi055.stdout: 2022-09-07T21:25:17.678 INFO:teuthology.orchestra.run.smithi055.stdout:Upgraded: 2022-09-07T21:25:17.679 INFO:teuthology.orchestra.run.smithi055.stdout: conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.679 INFO:teuthology.orchestra.run.smithi055.stdout: container-selinux-2:2.188.0-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-07T21:25:17.679 INFO:teuthology.orchestra.run.smithi055.stdout: containernetworking-plugins-1:1.1.1-3.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.679 INFO:teuthology.orchestra.run.smithi055.stdout: containers-common-2:1-35.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.680 INFO:teuthology.orchestra.run.smithi055.stdout: criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 2022-09-07T21:25:17.680 INFO:teuthology.orchestra.run.smithi055.stdout: fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.680 INFO:teuthology.orchestra.run.smithi055.stdout: libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86_64 2022-09-07T21:25:17.680 INFO:teuthology.orchestra.run.smithi055.stdout: runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.680 INFO:teuthology.orchestra.run.smithi055.stdout: slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.681 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2022-09-07T21:25:17.681 INFO:teuthology.orchestra.run.smithi055.stdout: buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.681 INFO:teuthology.orchestra.run.smithi055.stdout: cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-07T21:25:17.682 INFO:teuthology.orchestra.run.smithi055.stdout: crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.682 INFO:teuthology.orchestra.run.smithi055.stdout: python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-07T21:25:17.683 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2022-09-07T21:25:17.683 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyxdg-0.25-16.el8.noarch 2022-09-07T21:25:17.683 INFO:teuthology.orchestra.run.smithi055.stdout: skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.684 INFO:teuthology.orchestra.run.smithi055.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.684 INFO:teuthology.orchestra.run.smithi055.stdout: udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 2022-09-07T21:25:17.685 INFO:teuthology.orchestra.run.smithi055.stdout: yajl-2.1.0-10.el8.x86_64 2022-09-07T21:25:17.685 INFO:teuthology.orchestra.run.smithi055.stdout: 2022-09-07T21:25:17.686 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2022-09-07T21:25:17.687 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 28/28 2022-09-07T21:25:17.687 INFO:teuthology.orchestra.run.smithi059.stdout:Installed products updated. 2022-09-07T21:25:17.912 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:25:17.913 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2022-09-07T21:25:17.913 INFO:teuthology.orchestra.run.smithi059.stdout: conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.914 INFO:teuthology.orchestra.run.smithi059.stdout: container-selinux-2:2.188.0-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-07T21:25:17.914 INFO:teuthology.orchestra.run.smithi059.stdout: containernetworking-plugins-1:1.1.1-3.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.915 INFO:teuthology.orchestra.run.smithi059.stdout: containers-common-2:1-35.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.915 INFO:teuthology.orchestra.run.smithi059.stdout: criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 2022-09-07T21:25:17.916 INFO:teuthology.orchestra.run.smithi059.stdout: fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.916 INFO:teuthology.orchestra.run.smithi059.stdout: libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86_64 2022-09-07T21:25:17.916 INFO:teuthology.orchestra.run.smithi059.stdout: runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.916 INFO:teuthology.orchestra.run.smithi059.stdout: slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.916 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2022-09-07T21:25:17.917 INFO:teuthology.orchestra.run.smithi059.stdout: buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.917 INFO:teuthology.orchestra.run.smithi059.stdout: cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-07T21:25:17.918 INFO:teuthology.orchestra.run.smithi059.stdout: crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.918 INFO:teuthology.orchestra.run.smithi059.stdout: python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-07T21:25:17.918 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2022-09-07T21:25:17.918 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyxdg-0.25-16.el8.noarch 2022-09-07T21:25:17.919 INFO:teuthology.orchestra.run.smithi059.stdout: skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.919 INFO:teuthology.orchestra.run.smithi059.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:17.919 INFO:teuthology.orchestra.run.smithi059.stdout: udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 2022-09-07T21:25:17.919 INFO:teuthology.orchestra.run.smithi059.stdout: yajl-2.1.0-10.el8.x86_64 2022-09-07T21:25:17.920 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:25:17.920 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2022-09-07T21:25:17.977 DEBUG:teuthology.parallel:result is None 2022-09-07T21:25:18.212 DEBUG:teuthology.parallel:result is None 2022-09-07T21:25:30.058 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 28/28 2022-09-07T21:25:31.295 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 28/28 2022-09-07T21:25:31.295 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/28 2022-09-07T21:25:31.296 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2/28 2022-09-07T21:25:31.296 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 3/28 2022-09-07T21:25:31.296 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 4/28 2022-09-07T21:25:31.296 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f 5/28 2022-09-07T21:25:31.297 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 6/28 2022-09-07T21:25:31.297 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x 7/28 2022-09-07T21:25:31.297 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 8/28 2022-09-07T21:25:31.297 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6 9/28 2022-09-07T21:25:31.298 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86 10/28 2022-09-07T21:25:31.298 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : container-selinux-2:2.188.0-1.module+el8.6.0+15917 11/28 2022-09-07T21:25:31.298 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : container-selinux-2:2.179.1-1.module+el8.6.0+14877 12/28 2022-09-07T21:25:31.298 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_6 13/28 2022-09-07T21:25:31.298 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 14/28 2022-09-07T21:25:31.299 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86 15/28 2022-09-07T21:25:31.299 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 16/28 2022-09-07T21:25:31.299 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 17/28 2022-09-07T21:25:31.299 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 18/28 2022-09-07T21:25:31.300 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8. 19/28 2022-09-07T21:25:31.300 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 20/28 2022-09-07T21:25:31.300 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86 21/28 2022-09-07T21:25:31.300 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 22/28 2022-09-07T21:25:31.300 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : containernetworking-plugins-1:1.1.1-3.module+el8.6 23/28 2022-09-07T21:25:31.301 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : containernetworking-plugins-1:1.0.1-2.module+el8.6 24/28 2022-09-07T21:25:31.301 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : containers-common-2:1-35.module+el8.6.0+15917+093c 25/28 2022-09-07T21:25:31.301 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : containers-common-2:1-27.module+el8.6.0+14877+f643 26/28 2022-09-07T21:25:31.301 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 27/28 2022-09-07T21:25:32.223 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 28/28 2022-09-07T21:25:32.223 INFO:teuthology.orchestra.run.smithi047.stdout:Installed products updated. 2022-09-07T21:25:32.431 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:25:32.432 INFO:teuthology.orchestra.run.smithi047.stdout:Upgraded: 2022-09-07T21:25:32.432 INFO:teuthology.orchestra.run.smithi047.stdout: conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:32.433 INFO:teuthology.orchestra.run.smithi047.stdout: container-selinux-2:2.188.0-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-07T21:25:32.433 INFO:teuthology.orchestra.run.smithi047.stdout: containernetworking-plugins-1:1.1.1-3.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:32.433 INFO:teuthology.orchestra.run.smithi047.stdout: containers-common-2:1-35.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:32.434 INFO:teuthology.orchestra.run.smithi047.stdout: criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 2022-09-07T21:25:32.434 INFO:teuthology.orchestra.run.smithi047.stdout: fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:32.435 INFO:teuthology.orchestra.run.smithi047.stdout: libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86_64 2022-09-07T21:25:32.435 INFO:teuthology.orchestra.run.smithi047.stdout: runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:32.436 INFO:teuthology.orchestra.run.smithi047.stdout: slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:32.436 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2022-09-07T21:25:32.436 INFO:teuthology.orchestra.run.smithi047.stdout: buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:32.437 INFO:teuthology.orchestra.run.smithi047.stdout: cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-07T21:25:32.437 INFO:teuthology.orchestra.run.smithi047.stdout: crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:32.438 INFO:teuthology.orchestra.run.smithi047.stdout: python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-07T21:25:32.438 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2022-09-07T21:25:32.439 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyxdg-0.25-16.el8.noarch 2022-09-07T21:25:32.439 INFO:teuthology.orchestra.run.smithi047.stdout: skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:32.439 INFO:teuthology.orchestra.run.smithi047.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-07T21:25:32.440 INFO:teuthology.orchestra.run.smithi047.stdout: udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 2022-09-07T21:25:32.440 INFO:teuthology.orchestra.run.smithi047.stdout: yajl-2.1.0-10.el8.x86_64 2022-09-07T21:25:32.441 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:25:32.441 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2022-09-07T21:25:32.823 DEBUG:teuthology.parallel:result is None 2022-09-07T21:25:32.825 INFO:teuthology.run_tasks:Running task install... 2022-09-07T21:25:32.843 DEBUG:teuthology.task.install:project ceph 2022-09-07T21:25:32.845 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '152194198413033811b31149de909a7ec5852a7f'}} 2022-09-07T21:25:32.845 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '152194198413033811b31149de909a7ec5852a7f'} 2022-09-07T21:25:32.846 INFO:teuthology.task.install:Using flavor: default 2022-09-07T21:25:32.861 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-09-07T21:25:32.861 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2022-09-07T21:25:32.862 DEBUG:teuthology.orchestra.run.smithi047:> sudo lsb_release -is 2022-09-07T21:25:33.009 INFO:teuthology.orchestra.run.smithi047.stdout:RedHatEnterprise 2022-09-07T21:25:33.014 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-07T21:25:33.015 DEBUG:teuthology.orchestra.run.smithi055:> sudo lsb_release -is 2022-09-07T21:25:33.016 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '152194198413033811b31149de909a7ec5852a7f', 'tag': None, 'wait_for_package': False} 2022-09-07T21:25:33.016 DEBUG:teuthology.orchestra.run.smithi047:> sudo lsb_release -is 2022-09-07T21:25:33.070 INFO:teuthology.orchestra.run.smithi055.stdout:RedHatEnterprise 2022-09-07T21:25:33.071 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-07T21:25:33.072 DEBUG:teuthology.orchestra.run.smithi059:> sudo lsb_release -is 2022-09-07T21:25:33.073 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '152194198413033811b31149de909a7ec5852a7f', 'tag': None, 'wait_for_package': False} 2022-09-07T21:25:33.073 DEBUG:teuthology.orchestra.run.smithi055:> sudo lsb_release -is 2022-09-07T21:25:33.120 INFO:teuthology.orchestra.run.smithi055.stdout:RedHatEnterprise 2022-09-07T21:25:33.121 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-07T21:25:33.122 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=152194198413033811b31149de909a7ec5852a7f 2022-09-07T21:25:33.131 INFO:teuthology.orchestra.run.smithi059.stdout:RedHatEnterprise 2022-09-07T21:25:33.131 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-07T21:25:33.132 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '152194198413033811b31149de909a7ec5852a7f', 'tag': None, 'wait_for_package': False} 2022-09-07T21:25:33.133 DEBUG:teuthology.orchestra.run.smithi059:> sudo lsb_release -is 2022-09-07T21:25:33.161 INFO:teuthology.orchestra.run.smithi047.stdout:RedHatEnterprise 2022-09-07T21:25:33.166 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-07T21:25:33.167 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=152194198413033811b31149de909a7ec5852a7f 2022-09-07T21:25:33.184 INFO:teuthology.orchestra.run.smithi059.stdout:RedHatEnterprise 2022-09-07T21:25:33.186 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-07T21:25:33.187 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=152194198413033811b31149de909a7ec5852a7f 2022-09-07T21:25:33.282 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220907.015905/152194198413033811b31149de909a7ec5852a7f/centos/8/flavors/default/ 2022-09-07T21:25:33.282 INFO:teuthology.task.install.rpm:Package version is 17.0.0-14857.g15219419 2022-09-07T21:25:33.327 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220907.015905/152194198413033811b31149de909a7ec5852a7f/centos/8/flavors/default/ 2022-09-07T21:25:33.328 INFO:teuthology.task.install.rpm:Package version is 17.0.0-14857.g15219419 2022-09-07T21:25:33.335 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220907.015905/152194198413033811b31149de909a7ec5852a7f/centos/8/flavors/default/ 2022-09-07T21:25:33.336 INFO:teuthology.task.install.rpm:Package version is 17.0.0-14857.g15219419 2022-09-07T21:25:33.405 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220907.015905/152194198413033811b31149de909a7ec5852a7f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220907.015905/152194198413033811b31149de909a7ec5852a7f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220907.015905/152194198413033811b31149de909a7ec5852a7f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-09-07T21:25:33.406 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2022-09-07T21:25:33.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-09-07T21:25:33.445 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2022-09-07T21:25:33.446 DEBUG:teuthology.orchestra.run.smithi055:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/152194198413033811b31149de909a7ec5852a7f/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-09-07T21:25:33.447 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220907.015905/152194198413033811b31149de909a7ec5852a7f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220907.015905/152194198413033811b31149de909a7ec5852a7f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220907.015905/152194198413033811b31149de909a7ec5852a7f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-09-07T21:25:33.448 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:25:33.449 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-09-07T21:25:33.453 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220907.015905/152194198413033811b31149de909a7ec5852a7f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220907.015905/152194198413033811b31149de909a7ec5852a7f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220907.015905/152194198413033811b31149de909a7ec5852a7f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-09-07T21:25:33.453 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-09-07T21:25:33.454 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-09-07T21:25:33.486 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2022-09-07T21:25:33.487 DEBUG:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/152194198413033811b31149de909a7ec5852a7f/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-09-07T21:25:33.518 DEBUG:teuthology.orchestra.run.smithi055:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-09-07T21:25:33.542 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2022-09-07T21:25:33.543 DEBUG:teuthology.orchestra.run.smithi047:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/152194198413033811b31149de909a7ec5852a7f/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-09-07T21:25:33.559 DEBUG:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-09-07T21:25:33.607 DEBUG:teuthology.orchestra.run.smithi055:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-09-07T21:25:33.648 DEBUG:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-09-07T21:25:33.672 DEBUG:teuthology.orchestra.run.smithi047:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-09-07T21:25:33.678 INFO:teuthology.orchestra.run.smithi055.stdout:check_obsoletes = 1 2022-09-07T21:25:33.680 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum clean all 2022-09-07T21:25:33.715 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2022-09-07T21:25:33.717 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum clean all 2022-09-07T21:25:33.795 DEBUG:teuthology.orchestra.run.smithi047:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-09-07T21:25:33.883 INFO:teuthology.orchestra.run.smithi047.stdout:check_obsoletes = 1 2022-09-07T21:25:33.889 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum clean all 2022-09-07T21:25:34.077 INFO:teuthology.orchestra.run.smithi055.stdout:Updating Subscription Management repositories. 2022-09-07T21:25:34.112 INFO:teuthology.orchestra.run.smithi059.stdout:Updating Subscription Management repositories. 2022-09-07T21:25:34.280 INFO:teuthology.orchestra.run.smithi055.stdout:52 files removed 2022-09-07T21:25:34.319 INFO:teuthology.orchestra.run.smithi059.stdout:52 files removed 2022-09-07T21:25:34.331 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2022-09-07T21:25:34.371 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2022-09-07T21:25:34.422 INFO:teuthology.orchestra.run.smithi047.stdout:Updating Subscription Management repositories. 2022-09-07T21:25:34.684 INFO:teuthology.orchestra.run.smithi055.stdout:Updating Subscription Management repositories. 2022-09-07T21:25:34.724 INFO:teuthology.orchestra.run.smithi059.stdout:Updating Subscription Management repositories. 2022-09-07T21:25:34.738 INFO:teuthology.orchestra.run.smithi047.stdout:52 files removed 2022-09-07T21:25:34.806 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2022-09-07T21:25:35.317 INFO:teuthology.orchestra.run.smithi055.stdout:ceph packages for x86_64 262 kB/s | 81 kB 00:00 2022-09-07T21:25:35.328 INFO:teuthology.orchestra.run.smithi047.stdout:Updating Subscription Management repositories. 2022-09-07T21:25:35.361 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 260 kB/s | 81 kB 00:00 2022-09-07T21:25:35.597 INFO:teuthology.orchestra.run.smithi055.stdout:ceph noarch packages 38 kB/s | 9.4 kB 00:00 2022-09-07T21:25:35.645 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 38 kB/s | 9.4 kB 00:00 2022-09-07T21:25:35.873 INFO:teuthology.orchestra.run.smithi055.stdout:ceph source packages 6.0 kB/s | 1.5 kB 00:00 2022-09-07T21:25:35.912 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 6.2 kB/s | 1.5 kB 00:00 2022-09-07T21:25:36.037 INFO:teuthology.orchestra.run.smithi047.stdout:ceph packages for x86_64 293 kB/s | 81 kB 00:00 2022-09-07T21:25:36.065 INFO:teuthology.orchestra.run.smithi055.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2022-09-07T21:25:36.113 INFO:teuthology.orchestra.run.smithi059.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2022-09-07T21:25:36.321 INFO:teuthology.orchestra.run.smithi047.stdout:ceph noarch packages 38 kB/s | 9.4 kB 00:00 2022-09-07T21:25:36.605 INFO:teuthology.orchestra.run.smithi047.stdout:ceph source packages 5.8 kB/s | 1.5 kB 00:00 2022-09-07T21:25:36.848 INFO:teuthology.orchestra.run.smithi047.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2022-09-07T21:25:37.039 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 14 MB/s | 13 MB 00:00 2022-09-07T21:25:37.090 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 13 MB/s | 13 MB 00:01 2022-09-07T21:25:37.882 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 13 MB/s | 13 MB 00:01 2022-09-07T21:25:40.032 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 411 kB/s | 24 kB 00:00 2022-09-07T21:25:40.080 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 458 kB/s | 24 kB 00:00 2022-09-07T21:25:40.882 INFO:teuthology.orchestra.run.smithi055.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 MB/s | 51 MB 00:00 2022-09-07T21:25:40.891 INFO:teuthology.orchestra.run.smithi059.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 62 MB/s | 46 MB 00:00 2022-09-07T21:25:41.441 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 546 kB/s | 24 kB 00:00 2022-09-07T21:25:42.576 INFO:teuthology.orchestra.run.smithi047.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 48 MB/s | 51 MB 00:01 2022-09-07T21:25:49.110 INFO:teuthology.orchestra.run.smithi055.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 65 MB/s | 46 MB 00:00 2022-09-07T21:25:49.550 INFO:teuthology.orchestra.run.smithi059.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 60 MB/s | 51 MB 00:00 2022-09-07T21:25:51.539 INFO:teuthology.orchestra.run.smithi047.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 MB/s | 46 MB 00:01 2022-09-07T21:26:00.568 INFO:teuthology.orchestra.run.smithi055.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-09-07T21:26:00.569 INFO:teuthology.orchestra.run.smithi055.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-09-07T21:26:00.584 INFO:teuthology.orchestra.run.smithi055.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2022-09-07T21:26:00.585 INFO:teuthology.orchestra.run.smithi055.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2022-09-07T21:26:00.586 INFO:teuthology.orchestra.run.smithi055.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2022-09-07T21:26:00.587 INFO:teuthology.orchestra.run.smithi055.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-09-07T21:26:00.589 INFO:teuthology.orchestra.run.smithi055.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-09-07T21:26:00.594 INFO:teuthology.orchestra.run.smithi055.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2022-09-07T21:26:00.595 INFO:teuthology.orchestra.run.smithi055.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2022-09-07T21:26:00.735 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2022-09-07T21:26:00.749 INFO:teuthology.orchestra.run.smithi055.stdout:=========================================================================================================================== 2022-09-07T21:26:00.749 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2022-09-07T21:26:00.749 INFO:teuthology.orchestra.run.smithi055.stdout:=========================================================================================================================== 2022-09-07T21:26:00.750 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2022-09-07T21:26:00.750 INFO:teuthology.orchestra.run.smithi055.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-7.el8_6 rhel-8-for-x86_64-baseos-rpms 137 k 2022-09-07T21:26:00.750 INFO:teuthology.orchestra.run.smithi055.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-09-07T21:26:00.750 INFO:teuthology.orchestra.run.smithi055.stdout: ceph x86_64 2:17.0.0-14857.g15219419.el8 ceph 6.4 k 2022-09-07T21:26:00.751 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base x86_64 2:17.0.0-14857.g15219419.el8 ceph 5.2 M 2022-09-07T21:26:00.751 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse x86_64 2:17.0.0-14857.g15219419.el8 ceph 844 k 2022-09-07T21:26:00.751 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-14857.g15219419.el8 ceph 165 k 2022-09-07T21:26:00.751 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr x86_64 2:17.0.0-14857.g15219419.el8 ceph 1.5 M 2022-09-07T21:26:00.751 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 115 k 2022-09-07T21:26:00.752 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 1.6 M 2022-09-07T21:26:00.752 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 7.4 M 2022-09-07T21:26:00.752 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 47 k 2022-09-07T21:26:00.752 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw x86_64 2:17.0.0-14857.g15219419.el8 ceph 18 M 2022-09-07T21:26:00.753 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test x86_64 2:17.0.0-14857.g15219419.el8 ceph 54 M 2022-09-07T21:26:00.753 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 263 k 2022-09-07T21:26:00.753 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 81 k 2022-09-07T21:26:00.754 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-mirror x86_64 2:17.0.0-14857.g15219419.el8 ceph 257 k 2022-09-07T21:26:00.754 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-top noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 15 k 2022-09-07T21:26:00.754 INFO:teuthology.orchestra.run.smithi055.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2022-09-07T21:26:00.755 INFO:teuthology.orchestra.run.smithi055.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-09-07T21:26:00.755 INFO:teuthology.orchestra.run.smithi055.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-09-07T21:26:00.755 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel x86_64 2:17.0.0-14857.g15219419.el8 ceph 30 k 2022-09-07T21:26:00.755 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2 x86_64 2:17.0.0-14857.g15219419.el8 ceph 766 k 2022-09-07T21:26:00.756 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel x86_64 2:17.0.0-14857.g15219419.el8 ceph 139 k 2022-09-07T21:26:00.756 INFO:teuthology.orchestra.run.smithi055.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-09-07T21:26:00.756 INFO:teuthology.orchestra.run.smithi055.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2022-09-07T21:26:00.758 INFO:teuthology.orchestra.run.smithi055.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-09-07T21:26:00.758 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2022-09-07T21:26:00.759 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2022-09-07T21:26:00.759 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2022-09-07T21:26:00.760 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs x86_64 2:17.0.0-14857.g15219419.el8 ceph 190 k 2022-09-07T21:26:00.760 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados x86_64 2:17.0.0-14857.g15219419.el8 ceph 372 k 2022-09-07T21:26:00.761 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd x86_64 2:17.0.0-14857.g15219419.el8 ceph 358 k 2022-09-07T21:26:00.761 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw x86_64 2:17.0.0-14857.g15219419.el8 ceph 107 k 2022-09-07T21:26:00.761 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse x86_64 2:17.0.0-14857.g15219419.el8 ceph 94 k 2022-09-07T21:26:00.762 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror x86_64 2:17.0.0-14857.g15219419.el8 ceph 3.5 M 2022-09-07T21:26:00.762 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd x86_64 2:17.0.0-14857.g15219419.el8 ceph 175 k 2022-09-07T21:26:00.762 INFO:teuthology.orchestra.run.smithi055.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2022-09-07T21:26:00.763 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrading: 2022-09-07T21:26:00.763 INFO:teuthology.orchestra.run.smithi055.stdout: librados2 x86_64 2:17.0.0-14857.g15219419.el8 ceph 3.6 M 2022-09-07T21:26:00.764 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1 x86_64 2:17.0.0-14857.g15219419.el8 ceph 3.6 M 2022-09-07T21:26:00.764 INFO:teuthology.orchestra.run.smithi055.stdout: openssl x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 709 k 2022-09-07T21:26:00.764 INFO:teuthology.orchestra.run.smithi055.stdout: openssl-libs x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 1.5 M 2022-09-07T21:26:00.765 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2022-09-07T21:26:00.765 INFO:teuthology.orchestra.run.smithi055.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-09-07T21:26:00.766 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common x86_64 2:17.0.0-14857.g15219419.el8 ceph 24 M 2022-09-07T21:26:00.766 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 23 k 2022-09-07T21:26:00.766 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds x86_64 2:17.0.0-14857.g15219419.el8 ceph 2.3 M 2022-09-07T21:26:00.766 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 233 k 2022-09-07T21:26:00.767 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon x86_64 2:17.0.0-14857.g15219419.el8 ceph 3.9 M 2022-09-07T21:26:00.767 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd x86_64 2:17.0.0-14857.g15219419.el8 ceph 17 M 2022-09-07T21:26:00.767 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 14 k 2022-09-07T21:26:00.767 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux x86_64 2:17.0.0-14857.g15219419.el8 ceph 24 k 2022-09-07T21:26:00.767 INFO:teuthology.orchestra.run.smithi055.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-09-07T21:26:00.768 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-09-07T21:26:00.768 INFO:teuthology.orchestra.run.smithi055.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2022-09-07T21:26:00.768 INFO:teuthology.orchestra.run.smithi055.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-09-07T21:26:00.769 INFO:teuthology.orchestra.run.smithi055.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2022-09-07T21:26:00.769 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite x86_64 2:17.0.0-14857.g15219419.el8 ceph 137 k 2022-09-07T21:26:00.769 INFO:teuthology.orchestra.run.smithi055.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2022-09-07T21:26:00.770 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2022-09-07T21:26:00.770 INFO:teuthology.orchestra.run.smithi055.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-09-07T21:26:00.770 INFO:teuthology.orchestra.run.smithi055.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2022-09-07T21:26:00.771 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2022-09-07T21:26:00.771 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-09-07T21:26:00.771 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1 x86_64 2:17.0.0-14857.g15219419.el8 ceph 473 k 2022-09-07T21:26:00.771 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2022-09-07T21:26:00.772 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2 x86_64 2:17.0.0-14857.g15219419.el8 ceph 5.8 M 2022-09-07T21:26:00.772 INFO:teuthology.orchestra.run.smithi055.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-09-07T21:26:00.772 INFO:teuthology.orchestra.run.smithi055.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2022-09-07T21:26:00.772 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-09-07T21:26:00.772 INFO:teuthology.orchestra.run.smithi055.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-09-07T21:26:00.773 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-09-07T21:26:00.773 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-09-07T21:26:00.773 INFO:teuthology.orchestra.run.smithi055.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-09-07T21:26:00.773 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2022-09-07T21:26:00.773 INFO:teuthology.orchestra.run.smithi055.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-09-07T21:26:00.774 INFO:teuthology.orchestra.run.smithi055.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-09-07T21:26:00.774 INFO:teuthology.orchestra.run.smithi055.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-09-07T21:26:00.774 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-09-07T21:26:00.775 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-09-07T21:26:00.775 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-09-07T21:26:00.775 INFO:teuthology.orchestra.run.smithi055.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-09-07T21:26:00.775 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-09-07T21:26:00.776 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse x86_64 2:17.0.0-14857.g15219419.el8 ceph 45 k 2022-09-07T21:26:00.776 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common x86_64 2:17.0.0-14857.g15219419.el8 ceph 109 k 2022-09-07T21:26:00.776 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-09-07T21:26:00.777 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-09-07T21:26:00.777 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-09-07T21:26:00.777 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-09-07T21:26:00.777 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-09-07T21:26:00.778 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-09-07T21:26:00.778 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-09-07T21:26:00.778 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-09-07T21:26:00.779 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2022-09-07T21:26:00.779 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-09-07T21:26:00.780 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-09-07T21:26:00.780 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-09-07T21:26:00.780 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-09-07T21:26:00.781 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-09-07T21:26:00.781 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-09-07T21:26:00.782 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-09-07T21:26:00.782 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-09-07T21:26:00.782 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-09-07T21:26:00.783 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-09-07T21:26:00.783 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-09-07T21:26:00.783 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-09-07T21:26:00.783 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-09-07T21:26:00.784 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-09-07T21:26:00.784 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-09-07T21:26:00.784 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-09-07T21:26:00.784 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-09-07T21:26:00.785 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-09-07T21:26:00.785 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-09-07T21:26:00.785 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-09-07T21:26:00.785 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-09-07T21:26:00.785 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-09-07T21:26:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-09-07T21:26:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-09-07T21:26:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2022-09-07T21:26:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-09-07T21:26:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-09-07T21:26:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-09-07T21:26:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-09-07T21:26:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-09-07T21:26:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-09-07T21:26:00.788 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-09-07T21:26:00.788 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-09-07T21:26:00.788 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-09-07T21:26:00.788 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-09-07T21:26:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2022-09-07T21:26:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-09-07T21:26:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2022-09-07T21:26:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2022-09-07T21:26:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2022-09-07T21:26:00.791 INFO:teuthology.orchestra.run.smithi055.stdout:Enabling module streams: 2022-09-07T21:26:00.791 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql 10 2022-09-07T21:26:00.791 INFO:teuthology.orchestra.run.smithi055.stdout: 2022-09-07T21:26:00.792 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2022-09-07T21:26:00.792 INFO:teuthology.orchestra.run.smithi055.stdout:=========================================================================================================================== 2022-09-07T21:26:00.792 INFO:teuthology.orchestra.run.smithi055.stdout:Install 128 Packages 2022-09-07T21:26:00.792 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrade 4 Packages 2022-09-07T21:26:00.793 INFO:teuthology.orchestra.run.smithi055.stdout: 2022-09-07T21:26:00.793 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 216 M 2022-09-07T21:26:00.794 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2022-09-07T21:26:00.944 INFO:teuthology.orchestra.run.smithi055.stdout:(1/132): ceph-17.0.0-14857.g15219419.el8.x86_64 36 kB/s | 6.4 kB 00:00 2022-09-07T21:26:01.232 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-09-07T21:26:01.233 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-09-07T21:26:01.249 INFO:teuthology.orchestra.run.smithi059.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2022-09-07T21:26:01.250 INFO:teuthology.orchestra.run.smithi059.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2022-09-07T21:26:01.251 INFO:teuthology.orchestra.run.smithi059.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2022-09-07T21:26:01.253 INFO:teuthology.orchestra.run.smithi059.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-09-07T21:26:01.255 INFO:teuthology.orchestra.run.smithi059.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-09-07T21:26:01.259 INFO:teuthology.orchestra.run.smithi059.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2022-09-07T21:26:01.260 INFO:teuthology.orchestra.run.smithi059.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2022-09-07T21:26:01.361 INFO:teuthology.orchestra.run.smithi055.stdout:(2/132): ceph-fuse-17.0.0-14857.g15219419.el8.x 2.0 MB/s | 844 kB 00:00 2022-09-07T21:26:01.399 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2022-09-07T21:26:01.412 INFO:teuthology.orchestra.run.smithi059.stdout:=========================================================================================================================== 2022-09-07T21:26:01.413 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2022-09-07T21:26:01.413 INFO:teuthology.orchestra.run.smithi059.stdout:=========================================================================================================================== 2022-09-07T21:26:01.413 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2022-09-07T21:26:01.413 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-7.el8_6 rhel-8-for-x86_64-baseos-rpms 137 k 2022-09-07T21:26:01.414 INFO:teuthology.orchestra.run.smithi059.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-09-07T21:26:01.414 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_64 2:17.0.0-14857.g15219419.el8 ceph 6.4 k 2022-09-07T21:26:01.415 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:17.0.0-14857.g15219419.el8 ceph 5.2 M 2022-09-07T21:26:01.415 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:17.0.0-14857.g15219419.el8 ceph 844 k 2022-09-07T21:26:01.416 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-14857.g15219419.el8 ceph 165 k 2022-09-07T21:26:01.416 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_64 2:17.0.0-14857.g15219419.el8 ceph 1.5 M 2022-09-07T21:26:01.416 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 115 k 2022-09-07T21:26:01.417 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 1.6 M 2022-09-07T21:26:01.417 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 7.4 M 2022-09-07T21:26:01.418 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 47 k 2022-09-07T21:26:01.418 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:17.0.0-14857.g15219419.el8 ceph 18 M 2022-09-07T21:26:01.419 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:17.0.0-14857.g15219419.el8 ceph 54 M 2022-09-07T21:26:01.419 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 263 k 2022-09-07T21:26:01.419 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 81 k 2022-09-07T21:26:01.420 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-mirror x86_64 2:17.0.0-14857.g15219419.el8 ceph 257 k 2022-09-07T21:26:01.422 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-top noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 15 k 2022-09-07T21:26:01.422 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2022-09-07T21:26:01.422 INFO:teuthology.orchestra.run.smithi059.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-09-07T21:26:01.423 INFO:teuthology.orchestra.run.smithi059.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-09-07T21:26:01.423 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:17.0.0-14857.g15219419.el8 ceph 30 k 2022-09-07T21:26:01.423 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:17.0.0-14857.g15219419.el8 ceph 766 k 2022-09-07T21:26:01.424 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:17.0.0-14857.g15219419.el8 ceph 139 k 2022-09-07T21:26:01.424 INFO:teuthology.orchestra.run.smithi059.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-09-07T21:26:01.424 INFO:teuthology.orchestra.run.smithi059.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2022-09-07T21:26:01.425 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-09-07T21:26:01.425 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2022-09-07T21:26:01.425 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2022-09-07T21:26:01.426 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2022-09-07T21:26:01.426 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:17.0.0-14857.g15219419.el8 ceph 190 k 2022-09-07T21:26:01.426 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:17.0.0-14857.g15219419.el8 ceph 372 k 2022-09-07T21:26:01.427 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:17.0.0-14857.g15219419.el8 ceph 358 k 2022-09-07T21:26:01.427 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:17.0.0-14857.g15219419.el8 ceph 107 k 2022-09-07T21:26:01.428 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:17.0.0-14857.g15219419.el8 ceph 94 k 2022-09-07T21:26:01.428 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:17.0.0-14857.g15219419.el8 ceph 3.5 M 2022-09-07T21:26:01.428 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:17.0.0-14857.g15219419.el8 ceph 175 k 2022-09-07T21:26:01.429 INFO:teuthology.orchestra.run.smithi059.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2022-09-07T21:26:01.429 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2022-09-07T21:26:01.429 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:17.0.0-14857.g15219419.el8 ceph 3.6 M 2022-09-07T21:26:01.429 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:17.0.0-14857.g15219419.el8 ceph 3.6 M 2022-09-07T21:26:01.430 INFO:teuthology.orchestra.run.smithi059.stdout: openssl x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 709 k 2022-09-07T21:26:01.430 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-libs x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 1.5 M 2022-09-07T21:26:01.430 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2022-09-07T21:26:01.431 INFO:teuthology.orchestra.run.smithi059.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-09-07T21:26:01.431 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:17.0.0-14857.g15219419.el8 ceph 24 M 2022-09-07T21:26:01.431 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 23 k 2022-09-07T21:26:01.431 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_64 2:17.0.0-14857.g15219419.el8 ceph 2.3 M 2022-09-07T21:26:01.432 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 233 k 2022-09-07T21:26:01.432 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_64 2:17.0.0-14857.g15219419.el8 ceph 3.9 M 2022-09-07T21:26:01.432 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_64 2:17.0.0-14857.g15219419.el8 ceph 17 M 2022-09-07T21:26:01.432 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 14 k 2022-09-07T21:26:01.433 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:17.0.0-14857.g15219419.el8 ceph 24 k 2022-09-07T21:26:01.433 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-09-07T21:26:01.433 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-09-07T21:26:01.433 INFO:teuthology.orchestra.run.smithi059.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2022-09-07T21:26:01.434 INFO:teuthology.orchestra.run.smithi059.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-09-07T21:26:01.434 INFO:teuthology.orchestra.run.smithi059.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2022-09-07T21:26:01.434 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite x86_64 2:17.0.0-14857.g15219419.el8 ceph 137 k 2022-09-07T21:26:01.434 INFO:teuthology.orchestra.run.smithi059.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2022-09-07T21:26:01.435 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2022-09-07T21:26:01.435 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-09-07T21:26:01.435 INFO:teuthology.orchestra.run.smithi059.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2022-09-07T21:26:01.435 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2022-09-07T21:26:01.436 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-09-07T21:26:01.436 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:17.0.0-14857.g15219419.el8 ceph 473 k 2022-09-07T21:26:01.436 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2022-09-07T21:26:01.436 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:17.0.0-14857.g15219419.el8 ceph 5.8 M 2022-09-07T21:26:01.437 INFO:teuthology.orchestra.run.smithi059.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-09-07T21:26:01.437 INFO:teuthology.orchestra.run.smithi059.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2022-09-07T21:26:01.437 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-09-07T21:26:01.437 INFO:teuthology.orchestra.run.smithi059.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-09-07T21:26:01.438 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-09-07T21:26:01.438 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-09-07T21:26:01.439 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-09-07T21:26:01.439 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2022-09-07T21:26:01.439 INFO:teuthology.orchestra.run.smithi059.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-09-07T21:26:01.439 INFO:teuthology.orchestra.run.smithi059.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-09-07T21:26:01.440 INFO:teuthology.orchestra.run.smithi059.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-09-07T21:26:01.440 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-09-07T21:26:01.440 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-09-07T21:26:01.440 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-09-07T21:26:01.441 INFO:teuthology.orchestra.run.smithi059.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-09-07T21:26:01.441 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-09-07T21:26:01.441 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse x86_64 2:17.0.0-14857.g15219419.el8 ceph 45 k 2022-09-07T21:26:01.441 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:17.0.0-14857.g15219419.el8 ceph 109 k 2022-09-07T21:26:01.442 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-09-07T21:26:01.442 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-09-07T21:26:01.442 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-09-07T21:26:01.442 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-09-07T21:26:01.443 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-09-07T21:26:01.443 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-09-07T21:26:01.443 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-09-07T21:26:01.443 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-09-07T21:26:01.443 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2022-09-07T21:26:01.444 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-09-07T21:26:01.444 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-09-07T21:26:01.444 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-09-07T21:26:01.445 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-09-07T21:26:01.445 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-09-07T21:26:01.445 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-09-07T21:26:01.446 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-09-07T21:26:01.446 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-09-07T21:26:01.446 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-09-07T21:26:01.446 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-09-07T21:26:01.447 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-09-07T21:26:01.447 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-09-07T21:26:01.447 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-09-07T21:26:01.447 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-09-07T21:26:01.448 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-09-07T21:26:01.448 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-09-07T21:26:01.448 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-09-07T21:26:01.448 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-09-07T21:26:01.449 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-09-07T21:26:01.449 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-09-07T21:26:01.449 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-09-07T21:26:01.449 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-09-07T21:26:01.450 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-09-07T21:26:01.450 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-09-07T21:26:01.450 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2022-09-07T21:26:01.450 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-09-07T21:26:01.451 INFO:teuthology.orchestra.run.smithi059.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-09-07T21:26:01.451 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-09-07T21:26:01.451 INFO:teuthology.orchestra.run.smithi059.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-09-07T21:26:01.451 INFO:teuthology.orchestra.run.smithi059.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-09-07T21:26:01.452 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-09-07T21:26:01.452 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-09-07T21:26:01.452 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-09-07T21:26:01.452 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-09-07T21:26:01.452 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-09-07T21:26:01.453 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2022-09-07T21:26:01.454 INFO:teuthology.orchestra.run.smithi059.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-09-07T21:26:01.454 INFO:teuthology.orchestra.run.smithi059.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2022-09-07T21:26:01.454 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2022-09-07T21:26:01.454 INFO:teuthology.orchestra.run.smithi059.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2022-09-07T21:26:01.454 INFO:teuthology.orchestra.run.smithi059.stdout:Enabling module streams: 2022-09-07T21:26:01.455 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql 10 2022-09-07T21:26:01.455 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:26:01.455 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2022-09-07T21:26:01.455 INFO:teuthology.orchestra.run.smithi059.stdout:=========================================================================================================================== 2022-09-07T21:26:01.456 INFO:teuthology.orchestra.run.smithi059.stdout:Install 128 Packages 2022-09-07T21:26:01.456 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 4 Packages 2022-09-07T21:26:01.456 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:26:01.457 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 216 M 2022-09-07T21:26:01.457 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2022-09-07T21:26:01.477 INFO:teuthology.orchestra.run.smithi055.stdout:(3/132): ceph-immutable-object-cache-17.0.0-148 1.4 MB/s | 165 kB 00:00 2022-09-07T21:26:01.601 INFO:teuthology.orchestra.run.smithi059.stdout:(1/132): ceph-17.0.0-14857.g15219419.el8.x86_64 38 kB/s | 6.4 kB 00:00 2022-09-07T21:26:01.811 INFO:teuthology.orchestra.run.smithi055.stdout:(4/132): ceph-base-17.0.0-14857.g15219419.el8.x 4.9 MB/s | 5.2 MB 00:01 2022-09-07T21:26:02.103 INFO:teuthology.orchestra.run.smithi055.stdout:(5/132): ceph-mgr-17.0.0-14857.g15219419.el8.x8 5.2 MB/s | 1.5 MB 00:00 2022-09-07T21:26:02.286 INFO:teuthology.orchestra.run.smithi055.stdout:(6/132): ceph-mds-17.0.0-14857.g15219419.el8.x8 2.8 MB/s | 2.3 MB 00:00 2022-09-07T21:26:02.310 INFO:teuthology.orchestra.run.smithi059.stdout:(2/132): ceph-fuse-17.0.0-14857.g15219419.el8.x 1.2 MB/s | 844 kB 00:00 2022-09-07T21:26:02.701 INFO:teuthology.orchestra.run.smithi059.stdout:(3/132): ceph-immutable-object-cache-17.0.0-148 422 kB/s | 165 kB 00:00 2022-09-07T21:26:02.770 INFO:teuthology.orchestra.run.smithi055.stdout:(7/132): ceph-mon-17.0.0-14857.g15219419.el8.x8 5.8 MB/s | 3.9 MB 00:00 2022-09-07T21:26:03.521 INFO:teuthology.orchestra.run.smithi055.stdout:(8/132): ceph-common-17.0.0-14857.g15219419.el8 8.7 MB/s | 24 MB 00:02 2022-09-07T21:26:03.686 INFO:teuthology.orchestra.run.smithi059.stdout:(4/132): ceph-base-17.0.0-14857.g15219419.el8.x 2.3 MB/s | 5.2 MB 00:02 2022-09-07T21:26:03.746 INFO:teuthology.orchestra.run.smithi055.stdout:(9/132): ceph-selinux-17.0.0-14857.g15219419.el 108 kB/s | 24 kB 00:00 2022-09-07T21:26:03.935 INFO:teuthology.orchestra.run.smithi059.stdout:(5/132): ceph-mgr-17.0.0-14857.g15219419.el8.x8 6.1 MB/s | 1.5 MB 00:00 2022-09-07T21:26:04.074 INFO:teuthology.orchestra.run.smithi047.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-09-07T21:26:04.074 INFO:teuthology.orchestra.run.smithi047.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-09-07T21:26:04.090 INFO:teuthology.orchestra.run.smithi047.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2022-09-07T21:26:04.091 INFO:teuthology.orchestra.run.smithi047.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2022-09-07T21:26:04.093 INFO:teuthology.orchestra.run.smithi047.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2022-09-07T21:26:04.094 INFO:teuthology.orchestra.run.smithi047.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-09-07T21:26:04.100 INFO:teuthology.orchestra.run.smithi047.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-09-07T21:26:04.101 INFO:teuthology.orchestra.run.smithi047.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2022-09-07T21:26:04.102 INFO:teuthology.orchestra.run.smithi047.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2022-09-07T21:26:04.250 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2022-09-07T21:26:04.260 INFO:teuthology.orchestra.run.smithi059.stdout:(6/132): ceph-mds-17.0.0-14857.g15219419.el8.x8 1.5 MB/s | 2.3 MB 00:01 2022-09-07T21:26:04.264 INFO:teuthology.orchestra.run.smithi047.stdout:=========================================================================================================================== 2022-09-07T21:26:04.264 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2022-09-07T21:26:04.265 INFO:teuthology.orchestra.run.smithi047.stdout:=========================================================================================================================== 2022-09-07T21:26:04.265 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2022-09-07T21:26:04.265 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-7.el8_6 rhel-8-for-x86_64-baseos-rpms 137 k 2022-09-07T21:26:04.265 INFO:teuthology.orchestra.run.smithi047.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-09-07T21:26:04.266 INFO:teuthology.orchestra.run.smithi047.stdout: ceph x86_64 2:17.0.0-14857.g15219419.el8 ceph 6.4 k 2022-09-07T21:26:04.266 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base x86_64 2:17.0.0-14857.g15219419.el8 ceph 5.2 M 2022-09-07T21:26:04.266 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse x86_64 2:17.0.0-14857.g15219419.el8 ceph 844 k 2022-09-07T21:26:04.266 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-14857.g15219419.el8 ceph 165 k 2022-09-07T21:26:04.267 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr x86_64 2:17.0.0-14857.g15219419.el8 ceph 1.5 M 2022-09-07T21:26:04.267 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 115 k 2022-09-07T21:26:04.268 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 1.6 M 2022-09-07T21:26:04.268 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 7.4 M 2022-09-07T21:26:04.268 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 47 k 2022-09-07T21:26:04.269 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw x86_64 2:17.0.0-14857.g15219419.el8 ceph 18 M 2022-09-07T21:26:04.269 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test x86_64 2:17.0.0-14857.g15219419.el8 ceph 54 M 2022-09-07T21:26:04.270 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-volume noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 263 k 2022-09-07T21:26:04.270 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 81 k 2022-09-07T21:26:04.271 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-mirror x86_64 2:17.0.0-14857.g15219419.el8 ceph 257 k 2022-09-07T21:26:04.271 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-top noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 15 k 2022-09-07T21:26:04.271 INFO:teuthology.orchestra.run.smithi047.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2022-09-07T21:26:04.272 INFO:teuthology.orchestra.run.smithi047.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-09-07T21:26:04.272 INFO:teuthology.orchestra.run.smithi047.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-09-07T21:26:04.273 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel x86_64 2:17.0.0-14857.g15219419.el8 ceph 30 k 2022-09-07T21:26:04.273 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2 x86_64 2:17.0.0-14857.g15219419.el8 ceph 766 k 2022-09-07T21:26:04.274 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel x86_64 2:17.0.0-14857.g15219419.el8 ceph 139 k 2022-09-07T21:26:04.274 INFO:teuthology.orchestra.run.smithi047.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-09-07T21:26:04.275 INFO:teuthology.orchestra.run.smithi047.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2022-09-07T21:26:04.275 INFO:teuthology.orchestra.run.smithi047.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-09-07T21:26:04.276 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2022-09-07T21:26:04.276 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2022-09-07T21:26:04.277 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2022-09-07T21:26:04.277 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs x86_64 2:17.0.0-14857.g15219419.el8 ceph 190 k 2022-09-07T21:26:04.277 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados x86_64 2:17.0.0-14857.g15219419.el8 ceph 372 k 2022-09-07T21:26:04.278 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd x86_64 2:17.0.0-14857.g15219419.el8 ceph 358 k 2022-09-07T21:26:04.278 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw x86_64 2:17.0.0-14857.g15219419.el8 ceph 107 k 2022-09-07T21:26:04.279 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse x86_64 2:17.0.0-14857.g15219419.el8 ceph 94 k 2022-09-07T21:26:04.279 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror x86_64 2:17.0.0-14857.g15219419.el8 ceph 3.5 M 2022-09-07T21:26:04.280 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd x86_64 2:17.0.0-14857.g15219419.el8 ceph 175 k 2022-09-07T21:26:04.280 INFO:teuthology.orchestra.run.smithi047.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2022-09-07T21:26:04.280 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrading: 2022-09-07T21:26:04.281 INFO:teuthology.orchestra.run.smithi047.stdout: librados2 x86_64 2:17.0.0-14857.g15219419.el8 ceph 3.6 M 2022-09-07T21:26:04.281 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1 x86_64 2:17.0.0-14857.g15219419.el8 ceph 3.6 M 2022-09-07T21:26:04.282 INFO:teuthology.orchestra.run.smithi047.stdout: openssl x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 709 k 2022-09-07T21:26:04.282 INFO:teuthology.orchestra.run.smithi047.stdout: openssl-libs x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 1.5 M 2022-09-07T21:26:04.282 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2022-09-07T21:26:04.283 INFO:teuthology.orchestra.run.smithi047.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-09-07T21:26:04.283 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common x86_64 2:17.0.0-14857.g15219419.el8 ceph 24 M 2022-09-07T21:26:04.284 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 23 k 2022-09-07T21:26:04.284 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds x86_64 2:17.0.0-14857.g15219419.el8 ceph 2.3 M 2022-09-07T21:26:04.284 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 233 k 2022-09-07T21:26:04.285 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon x86_64 2:17.0.0-14857.g15219419.el8 ceph 3.9 M 2022-09-07T21:26:04.285 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd x86_64 2:17.0.0-14857.g15219419.el8 ceph 17 M 2022-09-07T21:26:04.285 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts noarch 2:17.0.0-14857.g15219419.el8 ceph-noarch 14 k 2022-09-07T21:26:04.286 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux x86_64 2:17.0.0-14857.g15219419.el8 ceph 24 k 2022-09-07T21:26:04.286 INFO:teuthology.orchestra.run.smithi047.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-09-07T21:26:04.286 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-09-07T21:26:04.287 INFO:teuthology.orchestra.run.smithi047.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2022-09-07T21:26:04.287 INFO:teuthology.orchestra.run.smithi047.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-09-07T21:26:04.288 INFO:teuthology.orchestra.run.smithi047.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2022-09-07T21:26:04.288 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite x86_64 2:17.0.0-14857.g15219419.el8 ceph 137 k 2022-09-07T21:26:04.288 INFO:teuthology.orchestra.run.smithi047.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2022-09-07T21:26:04.289 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2022-09-07T21:26:04.289 INFO:teuthology.orchestra.run.smithi047.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-09-07T21:26:04.289 INFO:teuthology.orchestra.run.smithi047.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2022-09-07T21:26:04.290 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2022-09-07T21:26:04.290 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-09-07T21:26:04.293 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1 x86_64 2:17.0.0-14857.g15219419.el8 ceph 473 k 2022-09-07T21:26:04.293 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2022-09-07T21:26:04.293 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2 x86_64 2:17.0.0-14857.g15219419.el8 ceph 5.8 M 2022-09-07T21:26:04.294 INFO:teuthology.orchestra.run.smithi047.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-09-07T21:26:04.294 INFO:teuthology.orchestra.run.smithi047.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2022-09-07T21:26:04.295 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-09-07T21:26:04.295 INFO:teuthology.orchestra.run.smithi047.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-09-07T21:26:04.295 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-09-07T21:26:04.295 INFO:teuthology.orchestra.run.smithi047.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-09-07T21:26:04.296 INFO:teuthology.orchestra.run.smithi047.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-09-07T21:26:04.296 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2022-09-07T21:26:04.296 INFO:teuthology.orchestra.run.smithi047.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-09-07T21:26:04.296 INFO:teuthology.orchestra.run.smithi047.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-09-07T21:26:04.296 INFO:teuthology.orchestra.run.smithi047.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-09-07T21:26:04.297 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-09-07T21:26:04.297 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-09-07T21:26:04.297 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-09-07T21:26:04.297 INFO:teuthology.orchestra.run.smithi047.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-09-07T21:26:04.298 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-09-07T21:26:04.298 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse x86_64 2:17.0.0-14857.g15219419.el8 ceph 45 k 2022-09-07T21:26:04.298 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common x86_64 2:17.0.0-14857.g15219419.el8 ceph 109 k 2022-09-07T21:26:04.298 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-09-07T21:26:04.299 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-09-07T21:26:04.299 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-09-07T21:26:04.299 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-09-07T21:26:04.300 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-09-07T21:26:04.300 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-09-07T21:26:04.300 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-09-07T21:26:04.300 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-09-07T21:26:04.300 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2022-09-07T21:26:04.301 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-09-07T21:26:04.301 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-09-07T21:26:04.301 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-09-07T21:26:04.302 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-09-07T21:26:04.302 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-09-07T21:26:04.302 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-09-07T21:26:04.302 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-09-07T21:26:04.303 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-09-07T21:26:04.303 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-09-07T21:26:04.303 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-09-07T21:26:04.303 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-09-07T21:26:04.303 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-09-07T21:26:04.304 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-09-07T21:26:04.304 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-09-07T21:26:04.304 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-09-07T21:26:04.305 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-09-07T21:26:04.305 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-09-07T21:26:04.305 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-09-07T21:26:04.306 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-09-07T21:26:04.306 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-09-07T21:26:04.307 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-09-07T21:26:04.307 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-09-07T21:26:04.307 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-09-07T21:26:04.307 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-09-07T21:26:04.308 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2022-09-07T21:26:04.308 INFO:teuthology.orchestra.run.smithi047.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-09-07T21:26:04.308 INFO:teuthology.orchestra.run.smithi047.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-09-07T21:26:04.308 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-09-07T21:26:04.309 INFO:teuthology.orchestra.run.smithi047.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-09-07T21:26:04.309 INFO:teuthology.orchestra.run.smithi047.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-09-07T21:26:04.309 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-09-07T21:26:04.310 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-09-07T21:26:04.310 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-09-07T21:26:04.310 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-09-07T21:26:04.310 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-09-07T21:26:04.311 INFO:teuthology.orchestra.run.smithi047.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2022-09-07T21:26:04.311 INFO:teuthology.orchestra.run.smithi047.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-09-07T21:26:04.311 INFO:teuthology.orchestra.run.smithi047.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2022-09-07T21:26:04.311 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2022-09-07T21:26:04.312 INFO:teuthology.orchestra.run.smithi047.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2022-09-07T21:26:04.312 INFO:teuthology.orchestra.run.smithi047.stdout:Enabling module streams: 2022-09-07T21:26:04.313 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql 10 2022-09-07T21:26:04.313 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:26:04.313 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2022-09-07T21:26:04.314 INFO:teuthology.orchestra.run.smithi047.stdout:=========================================================================================================================== 2022-09-07T21:26:04.314 INFO:teuthology.orchestra.run.smithi047.stdout:Install 128 Packages 2022-09-07T21:26:04.314 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrade 4 Packages 2022-09-07T21:26:04.315 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:26:04.316 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 216 M 2022-09-07T21:26:04.316 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2022-09-07T21:26:04.677 INFO:teuthology.orchestra.run.smithi047.stdout:(1/132): ceph-17.0.0-14857.g15219419.el8.x86_64 17 kB/s | 6.4 kB 00:00 2022-09-07T21:26:04.719 INFO:teuthology.orchestra.run.smithi059.stdout:(7/132): ceph-mon-17.0.0-14857.g15219419.el8.x8 4.9 MB/s | 3.9 MB 00:00 2022-09-07T21:26:05.555 INFO:teuthology.orchestra.run.smithi055.stdout:(10/132): ceph-radosgw-17.0.0-14857.g15219419.e 6.3 MB/s | 18 MB 00:02 2022-09-07T21:26:05.790 INFO:teuthology.orchestra.run.smithi055.stdout:(11/132): cephfs-mirror-17.0.0-14857.g15219419. 1.1 MB/s | 257 kB 00:00 2022-09-07T21:26:05.927 INFO:teuthology.orchestra.run.smithi047.stdout:(2/132): ceph-fuse-17.0.0-14857.g15219419.el8.x 676 kB/s | 844 kB 00:01 2022-09-07T21:26:06.177 INFO:teuthology.orchestra.run.smithi047.stdout:(3/132): ceph-immutable-object-cache-17.0.0-148 661 kB/s | 165 kB 00:00 2022-09-07T21:26:06.448 INFO:teuthology.orchestra.run.smithi055.stdout:(12/132): libcephfs-devel-17.0.0-14857.g1521941 46 kB/s | 30 kB 00:00 2022-09-07T21:26:06.672 INFO:teuthology.orchestra.run.smithi055.stdout:(13/132): libcephfs2-17.0.0-14857.g15219419.el8 3.3 MB/s | 766 kB 00:00 2022-09-07T21:26:06.788 INFO:teuthology.orchestra.run.smithi055.stdout:(14/132): libcephsqlite-17.0.0-14857.g15219419. 1.1 MB/s | 137 kB 00:00 2022-09-07T21:26:06.872 INFO:teuthology.orchestra.run.smithi055.stdout:(15/132): librados-devel-17.0.0-14857.g15219419 1.6 MB/s | 139 kB 00:00 2022-09-07T21:26:07.022 INFO:teuthology.orchestra.run.smithi055.stdout:(16/132): libradosstriper1-17.0.0-14857.g152194 3.1 MB/s | 473 kB 00:00 2022-09-07T21:26:07.815 INFO:teuthology.orchestra.run.smithi055.stdout:(17/132): ceph-osd-17.0.0-14857.g15219419.el8.x 3.0 MB/s | 17 MB 00:05 2022-09-07T21:26:07.877 INFO:teuthology.orchestra.run.smithi059.stdout:(8/132): ceph-radosgw-17.0.0-14857.g15219419.el 5.6 MB/s | 18 MB 00:03 2022-09-07T21:26:07.881 INFO:teuthology.orchestra.run.smithi055.stdout:(18/132): python3-ceph-argparse-17.0.0-14857.g1 673 kB/s | 45 kB 00:00 2022-09-07T21:26:07.957 INFO:teuthology.orchestra.run.smithi055.stdout:(19/132): python3-ceph-common-17.0.0-14857.g152 1.4 MB/s | 109 kB 00:00 2022-09-07T21:26:07.961 INFO:teuthology.orchestra.run.smithi059.stdout:(9/132): ceph-selinux-17.0.0-14857.g15219419.el 292 kB/s | 24 kB 00:00 2022-09-07T21:26:08.032 INFO:teuthology.orchestra.run.smithi055.stdout:(20/132): python3-cephfs-17.0.0-14857.g15219419 2.5 MB/s | 190 kB 00:00 2022-09-07T21:26:08.053 INFO:teuthology.orchestra.run.smithi047.stdout:(4/132): ceph-mds-17.0.0-14857.g15219419.el8.x8 1.2 MB/s | 2.3 MB 00:01 2022-09-07T21:26:08.190 INFO:teuthology.orchestra.run.smithi055.stdout:(21/132): python3-rados-17.0.0-14857.g15219419. 2.3 MB/s | 372 kB 00:00 2022-09-07T21:26:08.324 INFO:teuthology.orchestra.run.smithi055.stdout:(22/132): librgw2-17.0.0-14857.g15219419.el8.x8 4.4 MB/s | 5.8 MB 00:01 2022-09-07T21:26:08.358 INFO:teuthology.orchestra.run.smithi055.stdout:(23/132): python3-rbd-17.0.0-14857.g15219419.el 2.1 MB/s | 358 kB 00:00 2022-09-07T21:26:08.433 INFO:teuthology.orchestra.run.smithi055.stdout:(24/132): python3-rgw-17.0.0-14857.g15219419.el 983 kB/s | 107 kB 00:00 2022-09-07T21:26:08.458 INFO:teuthology.orchestra.run.smithi055.stdout:(25/132): rbd-fuse-17.0.0-14857.g15219419.el8.x 941 kB/s | 94 kB 00:00 2022-09-07T21:26:08.550 INFO:teuthology.orchestra.run.smithi055.stdout:(26/132): rbd-nbd-17.0.0-14857.g15219419.el8.x8 1.9 MB/s | 175 kB 00:00 2022-09-07T21:26:08.634 INFO:teuthology.orchestra.run.smithi055.stdout:(27/132): ceph-grafana-dashboards-17.0.0-14857. 274 kB/s | 23 kB 00:00 2022-09-07T21:26:08.768 INFO:teuthology.orchestra.run.smithi055.stdout:(28/132): ceph-mgr-cephadm-17.0.0-14857.g152194 863 kB/s | 115 kB 00:00 2022-09-07T21:26:09.261 INFO:teuthology.orchestra.run.smithi047.stdout:(5/132): ceph-mgr-17.0.0-14857.g15219419.el8.x8 1.3 MB/s | 1.5 MB 00:01 2022-09-07T21:26:09.293 INFO:teuthology.orchestra.run.smithi055.stdout:(29/132): rbd-mirror-17.0.0-14857.g15219419.el8 4.0 MB/s | 3.5 MB 00:00 2022-09-07T21:26:09.386 INFO:teuthology.orchestra.run.smithi055.stdout:(30/132): ceph-mgr-dashboard-17.0.0-14857.g1521 2.7 MB/s | 1.6 MB 00:00 2022-09-07T21:26:09.445 INFO:teuthology.orchestra.run.smithi047.stdout:(6/132): ceph-base-17.0.0-14857.g15219419.el8.x 1.0 MB/s | 5.2 MB 00:05 2022-09-07T21:26:09.527 INFO:teuthology.orchestra.run.smithi055.stdout:(31/132): ceph-mgr-modules-core-17.0.0-14857.g1 1.6 MB/s | 233 kB 00:00 2022-09-07T21:26:09.619 INFO:teuthology.orchestra.run.smithi055.stdout:(32/132): ceph-mgr-rook-17.0.0-14857.g15219419. 512 kB/s | 47 kB 00:00 2022-09-07T21:26:09.736 INFO:teuthology.orchestra.run.smithi055.stdout:(33/132): ceph-prometheus-alerts-17.0.0-14857.g 124 kB/s | 14 kB 00:00 2022-09-07T21:26:09.894 INFO:teuthology.orchestra.run.smithi055.stdout:(34/132): ceph-volume-17.0.0-14857.g15219419.el 1.6 MB/s | 263 kB 00:00 2022-09-07T21:26:10.428 INFO:teuthology.orchestra.run.smithi055.stdout:(35/132): ceph-test-17.0.0-14857.g15219419.el8. 8.1 MB/s | 54 MB 00:06 2022-09-07T21:26:10.445 INFO:teuthology.orchestra.run.smithi055.stdout:(36/132): cephadm-17.0.0-14857.g15219419.el8.no 147 kB/s | 81 kB 00:00 2022-09-07T21:26:10.496 INFO:teuthology.orchestra.run.smithi055.stdout:(37/132): cephfs-top-17.0.0-14857.g15219419.el8 231 kB/s | 15 kB 00:00 2022-09-07T21:26:10.637 INFO:teuthology.orchestra.run.smithi055.stdout:(38/132): liboath-2.6.2-3.el8.x86_64.rpm 416 kB/s | 59 kB 00:00 2022-09-07T21:26:10.663 INFO:teuthology.orchestra.run.smithi055.stdout:(39/132): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2022-09-07T21:26:10.755 INFO:teuthology.orchestra.run.smithi055.stdout:(40/132): libunwind-1.3.1-3.el8.x86_64.rpm 641 kB/s | 75 kB 00:00 2022-09-07T21:26:10.781 INFO:teuthology.orchestra.run.smithi055.stdout:(41/132): python3-asyncssh-2.7.0-2.el8.noarch.r 4.0 MB/s | 476 kB 00:00 2022-09-07T21:26:10.881 INFO:teuthology.orchestra.run.smithi055.stdout:(42/132): ceph-mgr-diskprediction-local-17.0.0- 4.7 MB/s | 7.4 MB 00:01 2022-09-07T21:26:10.897 INFO:teuthology.orchestra.run.smithi055.stdout:(43/132): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 307 kB/s | 44 kB 00:00 2022-09-07T21:26:10.923 INFO:teuthology.orchestra.run.smithi055.stdout:(44/132): python3-beautifulsoup4-4.6.3-2.el8.1. 1.3 MB/s | 185 kB 00:00 2022-09-07T21:26:10.940 INFO:teuthology.orchestra.run.smithi055.stdout:(45/132): python3-certifi-2018.10.15-7.el8.noar 388 kB/s | 16 kB 00:00 2022-09-07T21:26:10.974 INFO:teuthology.orchestra.run.smithi055.stdout:(46/132): python3-cheroot-8.5.2-1.el8.noarch.rp 3.4 MB/s | 173 kB 00:00 2022-09-07T21:26:10.991 INFO:teuthology.orchestra.run.smithi055.stdout:(47/132): python3-cachetools-3.1.1-4.el8.noarch 302 kB/s | 33 kB 00:00 2022-09-07T21:26:11.013 INFO:teuthology.orchestra.run.smithi047.stdout:(7/132): ceph-mon-17.0.0-14857.g15219419.el8.x8 2.2 MB/s | 3.9 MB 00:01 2022-09-07T21:26:11.050 INFO:teuthology.orchestra.run.smithi055.stdout:(48/132): python3-google-auth-1.1.1-10.el8.noar 1.0 MB/s | 81 kB 00:00 2022-09-07T21:26:11.084 INFO:teuthology.orchestra.run.smithi055.stdout:(49/132): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2022-09-07T21:26:11.108 INFO:teuthology.orchestra.run.smithi055.stdout:(50/132): python3-jaraco-functools-2.0-4.el8.no 306 kB/s | 18 kB 00:00 2022-09-07T21:26:11.133 INFO:teuthology.orchestra.run.smithi055.stdout:(51/132): python3-cherrypy-18.4.0-1.el8.noarch. 1.9 MB/s | 384 kB 00:00 2022-09-07T21:26:11.159 INFO:teuthology.orchestra.run.smithi055.stdout:(52/132): python3-logutils-0.3.5-11.el8.noarch. 971 kB/s | 49 kB 00:00 2022-09-07T21:26:11.183 INFO:teuthology.orchestra.run.smithi055.stdout:(53/132): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2022-09-07T21:26:11.209 INFO:teuthology.orchestra.run.smithi055.stdout:(54/132): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2022-09-07T21:26:11.242 INFO:teuthology.orchestra.run.smithi055.stdout:(55/132): python3-pecan-1.3.2-9.el8.noarch.rpm 4.7 MB/s | 283 kB 00:00 2022-09-07T21:26:11.268 INFO:teuthology.orchestra.run.smithi055.stdout:(56/132): python3-portend-2.6-1.el8.noarch.rpm 267 kB/s | 16 kB 00:00 2022-09-07T21:26:11.293 INFO:teuthology.orchestra.run.smithi055.stdout:(57/132): python3-repoze-lru-0.7-6.el8.noarch.r 670 kB/s | 34 kB 00:00 2022-09-07T21:26:11.328 INFO:teuthology.orchestra.run.smithi055.stdout:(58/132): python3-kubernetes-11.0.0-6.el8.noarc 4.8 MB/s | 1.2 MB 00:00 2022-09-07T21:26:11.381 INFO:teuthology.orchestra.run.smithi055.stdout:(59/132): python3-rsa-4.8-1.el8.noarch.rpm 703 kB/s | 61 kB 00:00 2022-09-07T21:26:11.410 INFO:teuthology.orchestra.run.smithi055.stdout:(60/132): python3-simplegeneric-0.8.1-17.el8.no 224 kB/s | 19 kB 00:00 2022-09-07T21:26:11.436 INFO:teuthology.orchestra.run.smithi055.stdout:(61/132): python3-singledispatch-3.4.0.3-18.el8 438 kB/s | 24 kB 00:00 2022-09-07T21:26:11.461 INFO:teuthology.orchestra.run.smithi055.stdout:(62/132): python3-routes-2.4.1-12.el8.noarch.rp 1.0 MB/s | 196 kB 00:00 2022-09-07T21:26:11.478 INFO:teuthology.orchestra.run.smithi055.stdout:(63/132): python3-tempora-1.14.1-5.el8.noarch.r 435 kB/s | 29 kB 00:00 2022-09-07T21:26:11.504 INFO:teuthology.orchestra.run.smithi055.stdout:(64/132): python3-trustme-0.6.0-4.el8.noarch.rp 409 kB/s | 27 kB 00:00 2022-09-07T21:26:11.530 INFO:teuthology.orchestra.run.smithi055.stdout:(65/132): python3-waitress-1.2.1-2.el8.1.noarch 3.5 MB/s | 241 kB 00:00 2022-09-07T21:26:11.557 INFO:teuthology.orchestra.run.smithi055.stdout:(66/132): python3-webob-1.8.5-1.el8.1.noarch.rp 3.2 MB/s | 251 kB 00:00 2022-09-07T21:26:11.571 INFO:teuthology.orchestra.run.smithi055.stdout:(67/132): python3-websocket-client-0.56.0-5.el8 912 kB/s | 61 kB 00:00 2022-09-07T21:26:11.602 INFO:teuthology.orchestra.run.smithi055.stdout:(68/132): python3-webtest-2.0.33-1.el8.noarch.r 1.2 MB/s | 85 kB 00:00 2022-09-07T21:26:11.623 INFO:teuthology.orchestra.run.smithi055.stdout:(69/132): python3-zc-lockfile-2.0-2.el8.noarch. 390 kB/s | 23 kB 00:00 2022-09-07T21:26:11.647 INFO:teuthology.orchestra.run.smithi055.stdout:(70/132): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2022-09-07T21:26:11.731 INFO:teuthology.orchestra.run.smithi055.stdout:(71/132): pcre2-devel-10.32-2.el8.x86_64.rpm 5.1 MB/s | 605 kB 00:00 2022-09-07T21:26:11.757 INFO:teuthology.orchestra.run.smithi055.stdout:(72/132): fuse-2.9.7-15.el8.x86_64.rpm 763 kB/s | 83 kB 00:00 2022-09-07T21:26:11.806 INFO:teuthology.orchestra.run.smithi055.stdout:(73/132): thrift-0.13.0-2.el8.x86_64.rpm 7.4 MB/s | 1.7 MB 00:00 2022-09-07T21:26:11.831 INFO:teuthology.orchestra.run.smithi055.stdout:(74/132): libsepol-devel-2.9-3.el8.x86_64.rpm 869 kB/s | 87 kB 00:00 2022-09-07T21:26:11.849 INFO:teuthology.orchestra.run.smithi055.stdout:(75/132): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.4 MB/s | 220 kB 00:00 2022-09-07T21:26:11.890 INFO:teuthology.orchestra.run.smithi055.stdout:(76/132): libverto-devel-0.3.0-5.el8.x86_64.rpm 314 kB/s | 18 kB 00:00 2022-09-07T21:26:11.924 INFO:teuthology.orchestra.run.smithi055.stdout:(77/132): libselinux-devel-2.9-5.el8.x86_64.rpm 1.7 MB/s | 200 kB 00:00 2022-09-07T21:26:11.949 INFO:teuthology.orchestra.run.smithi055.stdout:(78/132): librabbitmq-0.9.0-3.el8.x86_64.rpm 472 kB/s | 47 kB 00:00 2022-09-07T21:26:11.991 INFO:teuthology.orchestra.run.smithi055.stdout:(79/132): python3-cryptography-3.2.1-5.el8.x86_ 5.4 MB/s | 559 kB 00:00 2022-09-07T21:26:12.041 INFO:teuthology.orchestra.run.smithi055.stdout:(80/132): python3-pycparser-2.14-14.el8.noarch. 1.2 MB/s | 109 kB 00:00 2022-09-07T21:26:12.068 INFO:teuthology.orchestra.run.smithi055.stdout:(81/132): python3-oauthlib-2.1.0-1.el8.noarch.r 1.1 MB/s | 155 kB 00:00 2022-09-07T21:26:12.109 INFO:teuthology.orchestra.run.smithi055.stdout:(82/132): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.7 MB/s | 560 kB 00:00 2022-09-07T21:26:12.135 INFO:teuthology.orchestra.run.smithi055.stdout:(83/132): python3-jwt-1.6.1-2.el8.noarch.rpm 639 kB/s | 43 kB 00:00 2022-09-07T21:26:12.159 INFO:teuthology.orchestra.run.smithi055.stdout:(84/132): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.4 MB/s | 171 kB 00:00 2022-09-07T21:26:12.193 INFO:teuthology.orchestra.run.smithi055.stdout:(85/132): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 687 kB/s | 58 kB 00:00 2022-09-07T21:26:12.219 INFO:teuthology.orchestra.run.smithi055.stdout:(86/132): elfutils-libelf-devel-0.186-1.el8.x86 715 kB/s | 60 kB 00:00 2022-09-07T21:26:12.243 INFO:teuthology.orchestra.run.smithi055.stdout:(87/132): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.8 MB/s | 238 kB 00:00 2022-09-07T21:26:12.278 INFO:teuthology.orchestra.run.smithi055.stdout:(88/132): libaio-devel-0.3.112-1.el8.x86_64.rpm 223 kB/s | 19 kB 00:00 2022-09-07T21:26:12.302 INFO:teuthology.orchestra.run.smithi055.stdout:(89/132): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.1 MB/s | 98 kB 00:00 2022-09-07T21:26:12.356 INFO:teuthology.orchestra.run.smithi055.stdout:(90/132): NetworkManager-initscripts-updown-1.3 1.2 MB/s | 137 kB 00:00 2022-09-07T21:26:12.405 INFO:teuthology.orchestra.run.smithi055.stdout:(91/132): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.8 MB/s | 229 kB 00:00 2022-09-07T21:26:12.428 INFO:teuthology.orchestra.run.smithi055.stdout:(92/132): python3-requests-oauthlib-1.0.0-1.el8 343 kB/s | 43 kB 00:00 2022-09-07T21:26:12.454 INFO:teuthology.orchestra.run.smithi055.stdout:(93/132): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 744 kB/s | 72 kB 00:00 2022-09-07T21:26:12.487 INFO:teuthology.orchestra.run.smithi055.stdout:(94/132): libcom_err-devel-1.45.6-4.el8.x86_64. 459 kB/s | 39 kB 00:00 2022-09-07T21:26:12.513 INFO:teuthology.orchestra.run.smithi055.stdout:(95/132): keyutils-libs-devel-1.5.10-9.el8.x86_ 840 kB/s | 48 kB 00:00 2022-09-07T21:26:12.555 INFO:teuthology.orchestra.run.smithi055.stdout:(96/132): libgfortran-8.5.0-10.1.el8_6.x86_64.r 9.4 MB/s | 644 kB 00:00 2022-09-07T21:26:12.597 INFO:teuthology.orchestra.run.smithi055.stdout:(97/132): postgresql-contrib-10.21-2.module+el8 9.6 MB/s | 808 kB 00:00 2022-09-07T21:26:12.670 INFO:teuthology.orchestra.run.smithi055.stdout:(98/132): openssl-devel-1.1.1k-7.el8_6.x86_64.r 9.6 MB/s | 2.3 MB 00:00 2022-09-07T21:26:12.831 INFO:teuthology.orchestra.run.smithi055.stdout:(99/132): bison-3.0.4-10.el8.x86_64.rpm 2.4 MB/s | 688 kB 00:00 2022-09-07T21:26:12.890 INFO:teuthology.orchestra.run.smithi055.stdout:(100/132): postgresql-10.21-2.module+el8.6.0+15 5.2 MB/s | 1.5 MB 00:00 2022-09-07T21:26:12.920 INFO:teuthology.orchestra.run.smithi059.stdout:(10/132): ceph-common-17.0.0-14857.g15219419.el 2.1 MB/s | 24 MB 00:11 2022-09-07T21:26:12.931 INFO:teuthology.orchestra.run.smithi055.stdout:(101/132): python3-werkzeug-0.12.2-4.el8.noarch 11 MB/s | 457 kB 00:00 2022-09-07T21:26:13.031 INFO:teuthology.orchestra.run.smithi055.stdout:(102/132): python3-numpy-1.14.3-10.el8.x86_64.r 10 MB/s | 3.7 MB 00:00 2022-09-07T21:26:13.048 INFO:teuthology.orchestra.run.smithi055.stdout:(103/132): python3-markupsafe-0.23-19.el8.x86_6 332 kB/s | 39 kB 00:00 2022-09-07T21:26:13.082 INFO:teuthology.orchestra.run.smithi055.stdout:(104/132): python3-prettytable-0.7.2-14.el8.noa 880 kB/s | 44 kB 00:00 2022-09-07T21:26:13.115 INFO:teuthology.orchestra.run.smithi055.stdout:(105/132): librdkafka-0.11.4-3.el8.x86_64.rpm 5.2 MB/s | 354 kB 00:00 2022-09-07T21:26:13.182 INFO:teuthology.orchestra.run.smithi055.stdout:(106/132): postgresql-server-10.21-2.module+el8 14 MB/s | 5.1 MB 00:00 2022-09-07T21:26:13.226 INFO:teuthology.orchestra.run.smithi055.stdout:(107/132): python3-jinja2-2.10.1-3.el8.noarch.r 3.7 MB/s | 538 kB 00:00 2022-09-07T21:26:13.237 INFO:teuthology.orchestra.run.smithi059.stdout:(11/132): ceph-osd-17.0.0-14857.g15219419.el8.x 1.9 MB/s | 17 MB 00:08 2022-09-07T21:26:13.258 INFO:teuthology.orchestra.run.smithi055.stdout:(108/132): libtool-2.4.6-25.el8.x86_64.rpm 4.9 MB/s | 709 kB 00:00 2022-09-07T21:26:13.263 INFO:teuthology.orchestra.run.smithi059.stdout:(12/132): cephfs-mirror-17.0.0-14857.g15219419. 753 kB/s | 257 kB 00:00 2022-09-07T21:26:13.288 INFO:teuthology.orchestra.run.smithi055.stdout:(109/132): python3-numpy-f2py-1.14.3-10.el8.x86 2.2 MB/s | 225 kB 00:00 2022-09-07T21:26:13.299 INFO:teuthology.orchestra.run.smithi059.stdout:(13/132): libcephfs-devel-17.0.0-14857.g1521941 523 kB/s | 30 kB 00:00 2022-09-07T21:26:13.343 INFO:teuthology.orchestra.run.smithi055.stdout:(110/132): python3-mako-1.0.6-13.el8.noarch.rpm 1.9 MB/s | 157 kB 00:00 2022-09-07T21:26:13.387 INFO:teuthology.orchestra.run.smithi055.stdout:(111/132): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.4 MB/s | 103 kB 00:00 2022-09-07T21:26:13.423 INFO:teuthology.orchestra.run.smithi059.stdout:(14/132): libcephsqlite-17.0.0-14857.g15219419. 1.1 MB/s | 137 kB 00:00 2022-09-07T21:26:13.425 INFO:teuthology.orchestra.run.smithi055.stdout:(112/132): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2022-09-07T21:26:13.501 INFO:teuthology.orchestra.run.smithi055.stdout:(113/132): openblas-threads-0.3.15-3.el8.x86_64 18 MB/s | 4.9 MB 00:00 2022-09-07T21:26:13.535 INFO:teuthology.orchestra.run.smithi055.stdout:(114/132): flex-2.6.1-9.el8.x86_64.rpm 2.9 MB/s | 320 kB 00:00 2022-09-07T21:26:13.596 INFO:teuthology.orchestra.run.smithi059.stdout:(15/132): librados-devel-17.0.0-14857.g15219419 792 kB/s | 139 kB 00:00 2022-09-07T21:26:13.638 INFO:teuthology.orchestra.run.smithi059.stdout:(16/132): libcephfs2-17.0.0-14857.g15219419.el8 2.0 MB/s | 766 kB 00:00 2022-09-07T21:26:13.639 INFO:teuthology.orchestra.run.smithi055.stdout:(115/132): python3-babel-2.5.1-7.el8.noarch.rpm 14 MB/s | 4.8 MB 00:00 2022-09-07T21:26:13.668 INFO:teuthology.orchestra.run.smithi055.stdout:(116/132): jq-1.6-3.el8.x86_64.rpm 1.5 MB/s | 202 kB 00:00 2022-09-07T21:26:13.769 INFO:teuthology.orchestra.run.smithi055.stdout:(117/132): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2022-09-07T21:26:13.788 INFO:teuthology.orchestra.run.smithi059.stdout:(17/132): libradosstriper1-17.0.0-14857.g152194 2.4 MB/s | 473 kB 00:00 2022-09-07T21:26:13.812 INFO:teuthology.orchestra.run.smithi055.stdout:(118/132): automake-1.16.1-7.el8.noarch.rpm 5.0 MB/s | 713 kB 00:00 2022-09-07T21:26:13.836 INFO:teuthology.orchestra.run.smithi055.stdout:(119/132): python3-pytz-2017.2-9.el8.noarch.rpm 812 kB/s | 54 kB 00:00 2022-09-07T21:26:13.872 INFO:teuthology.orchestra.run.smithi059.stdout:(18/132): python3-ceph-argparse-17.0.0-14857.g1 535 kB/s | 45 kB 00:00 2022-09-07T21:26:13.878 INFO:teuthology.orchestra.run.smithi055.stdout:(120/132): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.8 MB/s | 126 kB 00:00 2022-09-07T21:26:13.903 INFO:teuthology.orchestra.run.smithi055.stdout:(121/132): python3-jsonpointer-1.10-11.el8.noar 299 kB/s | 20 kB 00:00 2022-09-07T21:26:13.955 INFO:teuthology.orchestra.run.smithi059.stdout:(19/132): python3-ceph-common-17.0.0-14857.g152 1.3 MB/s | 109 kB 00:00 2022-09-07T21:26:13.978 INFO:teuthology.orchestra.run.smithi055.stdout:(122/132): libpq-13.5-1.el8.x86_64.rpm 1.9 MB/s | 198 kB 00:00 2022-09-07T21:26:14.004 INFO:teuthology.orchestra.run.smithi055.stdout:(123/132): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2022-09-07T21:26:14.037 INFO:teuthology.orchestra.run.smithi055.stdout:(124/132): oniguruma-6.8.2-2.el8.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2022-09-07T21:26:14.063 INFO:teuthology.orchestra.run.smithi055.stdout:(125/132): python3-jsonpatch-1.21-2.el8.noarch. 456 kB/s | 27 kB 00:00 2022-09-07T21:26:14.105 INFO:teuthology.orchestra.run.smithi055.stdout:(126/132): uuid-1.6.2-43.el8.x86_64.rpm 956 kB/s | 64 kB 00:00 2022-09-07T21:26:14.106 INFO:teuthology.orchestra.run.smithi059.stdout:(20/132): python3-cephfs-17.0.0-14857.g15219419 1.2 MB/s | 190 kB 00:00 2022-09-07T21:26:14.130 INFO:teuthology.orchestra.run.smithi055.stdout:(127/132): python3-pyasn1-modules-0.3.7-6.el8.n 1.6 MB/s | 110 kB 00:00 2022-09-07T21:26:14.314 INFO:teuthology.orchestra.run.smithi055.stdout:(128/132): python3-scipy-1.0.0-21.module+el8.5. 20 MB/s | 14 MB 00:00 2022-09-07T21:26:14.314 INFO:teuthology.orchestra.run.smithi059.stdout:(21/132): python3-rados-17.0.0-14857.g15219419. 1.7 MB/s | 372 kB 00:00 2022-09-07T21:26:14.364 INFO:teuthology.orchestra.run.smithi055.stdout:(129/132): openssl-1.1.1k-7.el8_6.x86_64.rpm 14 MB/s | 709 kB 00:00 2022-09-07T21:26:14.422 INFO:teuthology.orchestra.run.smithi055.stdout:(130/132): openssl-libs-1.1.1k-7.el8_6.x86_64.r 25 MB/s | 1.5 MB 00:00 2022-09-07T21:26:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:(22/132): python3-rbd-17.0.0-14857.g15219419.el 1.6 MB/s | 358 kB 00:00 2022-09-07T21:26:14.623 INFO:teuthology.orchestra.run.smithi059.stdout:(23/132): python3-rgw-17.0.0-14857.g15219419.el 1.3 MB/s | 107 kB 00:00 2022-09-07T21:26:14.706 INFO:teuthology.orchestra.run.smithi059.stdout:(24/132): rbd-fuse-17.0.0-14857.g15219419.el8.x 1.1 MB/s | 94 kB 00:00 2022-09-07T21:26:15.032 INFO:teuthology.orchestra.run.smithi055.stdout:(131/132): librbd1-17.0.0-14857.g15219419.el8.x 4.0 MB/s | 3.6 MB 00:00 2022-09-07T21:26:15.306 INFO:teuthology.orchestra.run.smithi055.stdout:(132/132): librados2-17.0.0-14857.g15219419.el8 3.0 MB/s | 3.6 MB 00:01 2022-09-07T21:26:15.309 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2022-09-07T21:26:15.310 INFO:teuthology.orchestra.run.smithi055.stdout:Total 15 MB/s | 216 MB 00:14 2022-09-07T21:26:15.698 INFO:teuthology.orchestra.run.smithi059.stdout:(25/132): ceph-test-17.0.0-14857.g15219419.el8. 7.0 MB/s | 54 MB 00:07 2022-09-07T21:26:15.851 INFO:teuthology.orchestra.run.smithi059.stdout:(26/132): rbd-nbd-17.0.0-14857.g15219419.el8.x8 1.1 MB/s | 175 kB 00:00 2022-09-07T21:26:15.932 INFO:teuthology.orchestra.run.smithi059.stdout:(27/132): ceph-grafana-dashboards-17.0.0-14857. 284 kB/s | 23 kB 00:00 2022-09-07T21:26:15.999 INFO:teuthology.orchestra.run.smithi059.stdout:(28/132): librgw2-17.0.0-14857.g15219419.el8.x8 2.4 MB/s | 5.8 MB 00:02 2022-09-07T21:26:16.025 INFO:teuthology.orchestra.run.smithi059.stdout:(29/132): ceph-mgr-cephadm-17.0.0-14857.g152194 1.2 MB/s | 115 kB 00:00 2022-09-07T21:26:16.131 INFO:teuthology.orchestra.run.smithi059.stdout:(30/132): rbd-mirror-17.0.0-14857.g15219419.el8 2.4 MB/s | 3.5 MB 00:01 2022-09-07T21:26:16.187 INFO:teuthology.orchestra.run.smithi047.stdout:(8/132): ceph-osd-17.0.0-14857.g15219419.el8.x8 2.5 MB/s | 17 MB 00:06 2022-09-07T21:26:16.215 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2022-09-07T21:26:16.237 INFO:teuthology.orchestra.run.smithi047.stdout:(9/132): ceph-selinux-17.0.0-14857.g15219419.el 491 kB/s | 24 kB 00:00 2022-09-07T21:26:16.246 INFO:teuthology.orchestra.run.smithi059.stdout:(31/132): ceph-mgr-modules-core-17.0.0-14857.g1 2.0 MB/s | 233 kB 00:00 2022-09-07T21:26:16.350 INFO:teuthology.orchestra.run.smithi059.stdout:(32/132): ceph-mgr-rook-17.0.0-14857.g15219419. 433 kB/s | 47 kB 00:00 2022-09-07T21:26:16.389 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2022-09-07T21:26:16.390 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2022-09-07T21:26:16.409 INFO:teuthology.orchestra.run.smithi059.stdout:(33/132): ceph-prometheus-alerts-17.0.0-14857.g 247 kB/s | 14 kB 00:00 2022-09-07T21:26:16.476 INFO:teuthology.orchestra.run.smithi059.stdout:(34/132): ceph-mgr-dashboard-17.0.0-14857.g1521 3.5 MB/s | 1.6 MB 00:00 2022-09-07T21:26:16.550 INFO:teuthology.orchestra.run.smithi059.stdout:(35/132): ceph-volume-17.0.0-14857.g15219419.el 1.8 MB/s | 263 kB 00:00 2022-09-07T21:26:16.585 INFO:teuthology.orchestra.run.smithi059.stdout:(36/132): cephadm-17.0.0-14857.g15219419.el8.no 746 kB/s | 81 kB 00:00 2022-09-07T21:26:16.610 INFO:teuthology.orchestra.run.smithi059.stdout:(37/132): cephfs-top-17.0.0-14857.g15219419.el8 259 kB/s | 15 kB 00:00 2022-09-07T21:26:16.788 INFO:teuthology.orchestra.run.smithi059.stdout:(38/132): liboath-2.6.2-3.el8.x86_64.rpm 331 kB/s | 59 kB 00:00 2022-09-07T21:26:16.852 INFO:teuthology.orchestra.run.smithi059.stdout:(39/132): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2022-09-07T21:26:16.855 INFO:teuthology.orchestra.run.smithi047.stdout:(10/132): ceph-radosgw-17.0.0-14857.g15219419.e 3.0 MB/s | 18 MB 00:05 2022-09-07T21:26:16.877 INFO:teuthology.orchestra.run.smithi059.stdout:(40/132): libunwind-1.3.1-3.el8.x86_64.rpm 843 kB/s | 75 kB 00:00 2022-09-07T21:26:16.962 INFO:teuthology.orchestra.run.smithi059.stdout:(41/132): ceph-mgr-diskprediction-local-17.0.0- 7.9 MB/s | 7.4 MB 00:00 2022-09-07T21:26:16.963 INFO:teuthology.orchestra.run.smithi047.stdout:(11/132): cephfs-mirror-17.0.0-14857.g15219419. 2.3 MB/s | 257 kB 00:00 2022-09-07T21:26:16.978 INFO:teuthology.orchestra.run.smithi059.stdout:(42/132): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 435 kB/s | 44 kB 00:00 2022-09-07T21:26:17.014 INFO:teuthology.orchestra.run.smithi047.stdout:(12/132): libcephfs-devel-17.0.0-14857.g1521941 614 kB/s | 30 kB 00:00 2022-09-07T21:26:17.047 INFO:teuthology.orchestra.run.smithi059.stdout:(43/132): python3-asyncssh-2.7.0-2.el8.noarch.r 2.4 MB/s | 476 kB 00:00 2022-09-07T21:26:17.073 INFO:teuthology.orchestra.run.smithi059.stdout:(44/132): python3-cachetools-3.1.1-4.el8.noarch 351 kB/s | 33 kB 00:00 2022-09-07T21:26:17.098 INFO:teuthology.orchestra.run.smithi059.stdout:(45/132): python3-certifi-2018.10.15-7.el8.noar 325 kB/s | 16 kB 00:00 2022-09-07T21:26:17.156 INFO:teuthology.orchestra.run.smithi059.stdout:(46/132): python3-cheroot-8.5.2-1.el8.noarch.rp 2.0 MB/s | 173 kB 00:00 2022-09-07T21:26:17.182 INFO:teuthology.orchestra.run.smithi059.stdout:(47/132): python3-cherrypy-18.4.0-1.el8.noarch. 4.5 MB/s | 384 kB 00:00 2022-09-07T21:26:17.207 INFO:teuthology.orchestra.run.smithi059.stdout:(48/132): python3-beautifulsoup4-4.6.3-2.el8.1. 755 kB/s | 185 kB 00:00 2022-09-07T21:26:17.221 INFO:teuthology.orchestra.run.smithi047.stdout:(13/132): libcephfs2-17.0.0-14857.g15219419.el8 3.6 MB/s | 766 kB 00:00 2022-09-07T21:26:17.262 INFO:teuthology.orchestra.run.smithi059.stdout:(49/132): python3-google-auth-1.1.1-10.el8.noar 773 kB/s | 81 kB 00:00 2022-09-07T21:26:17.291 INFO:teuthology.orchestra.run.smithi059.stdout:(50/132): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2022-09-07T21:26:17.306 INFO:teuthology.orchestra.run.smithi047.stdout:(14/132): libcephsqlite-17.0.0-14857.g15219419. 1.6 MB/s | 137 kB 00:00 2022-09-07T21:26:17.316 INFO:teuthology.orchestra.run.smithi059.stdout:(51/132): python3-jaraco-functools-2.0-4.el8.no 161 kB/s | 18 kB 00:00 2022-09-07T21:26:17.342 INFO:teuthology.orchestra.run.smithi059.stdout:(52/132): python3-logutils-0.3.5-11.el8.noarch. 974 kB/s | 49 kB 00:00 2022-09-07T21:26:17.368 INFO:teuthology.orchestra.run.smithi059.stdout:(53/132): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2022-09-07T21:26:17.389 INFO:teuthology.orchestra.run.smithi047.stdout:(15/132): librados-devel-17.0.0-14857.g15219419 1.6 MB/s | 139 kB 00:00 2022-09-07T21:26:17.392 INFO:teuthology.orchestra.run.smithi059.stdout:(54/132): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2022-09-07T21:26:17.434 INFO:teuthology.orchestra.run.smithi059.stdout:(55/132): python3-kubernetes-11.0.0-6.el8.noarc 6.8 MB/s | 1.2 MB 00:00 2022-09-07T21:26:17.459 INFO:teuthology.orchestra.run.smithi059.stdout:(56/132): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2022-09-07T21:26:17.518 INFO:teuthology.orchestra.run.smithi059.stdout:(57/132): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2022-09-07T21:26:17.531 INFO:teuthology.orchestra.run.smithi047.stdout:(16/132): libradosstriper1-17.0.0-14857.g152194 3.3 MB/s | 473 kB 00:00 2022-09-07T21:26:17.543 INFO:teuthology.orchestra.run.smithi059.stdout:(58/132): python3-repoze-lru-0.7-6.el8.noarch.r 309 kB/s | 34 kB 00:00 2022-09-07T21:26:17.568 INFO:teuthology.orchestra.run.smithi059.stdout:(59/132): python3-routes-2.4.1-12.el8.noarch.rp 1.8 MB/s | 196 kB 00:00 2022-09-07T21:26:17.594 INFO:teuthology.orchestra.run.smithi059.stdout:(60/132): python3-rsa-4.8-1.el8.noarch.rpm 814 kB/s | 61 kB 00:00 2022-09-07T21:26:17.603 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2022-09-07T21:26:17.608 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2022-09-07T21:26:17.619 INFO:teuthology.orchestra.run.smithi059.stdout:(61/132): python3-simplegeneric-0.8.1-17.el8.no 248 kB/s | 19 kB 00:00 2022-09-07T21:26:17.644 INFO:teuthology.orchestra.run.smithi059.stdout:(62/132): python3-singledispatch-3.4.0.3-18.el8 316 kB/s | 24 kB 00:00 2022-09-07T21:26:17.669 INFO:teuthology.orchestra.run.smithi059.stdout:(63/132): python3-tempora-1.14.1-5.el8.noarch.r 388 kB/s | 29 kB 00:00 2022-09-07T21:26:17.721 INFO:teuthology.orchestra.run.smithi059.stdout:(64/132): python3-trustme-0.6.0-4.el8.noarch.rp 271 kB/s | 27 kB 00:00 2022-09-07T21:26:17.745 INFO:teuthology.orchestra.run.smithi059.stdout:(65/132): python3-waitress-1.2.1-2.el8.1.noarch 2.3 MB/s | 241 kB 00:00 2022-09-07T21:26:17.770 INFO:teuthology.orchestra.run.smithi059.stdout:(66/132): python3-webob-1.8.5-1.el8.1.noarch.rp 2.4 MB/s | 251 kB 00:00 2022-09-07T21:26:17.795 INFO:teuthology.orchestra.run.smithi059.stdout:(67/132): python3-websocket-client-0.56.0-5.el8 825 kB/s | 61 kB 00:00 2022-09-07T21:26:17.831 INFO:teuthology.orchestra.run.smithi059.stdout:(68/132): python3-webtest-2.0.33-1.el8.noarch.r 1.4 MB/s | 85 kB 00:00 2022-09-07T21:26:17.854 INFO:teuthology.orchestra.run.smithi059.stdout:(69/132): python3-zc-lockfile-2.0-2.el8.noarch. 275 kB/s | 23 kB 00:00 2022-09-07T21:26:17.874 INFO:teuthology.orchestra.run.smithi047.stdout:(17/132): ceph-common-17.0.0-14857.g15219419.el 1.8 MB/s | 24 MB 00:13 2022-09-07T21:26:17.882 INFO:teuthology.orchestra.run.smithi059.stdout:(70/132): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2022-09-07T21:26:17.959 INFO:teuthology.orchestra.run.smithi059.stdout:(71/132): postgresql-contrib-10.21-2.module+el8 7.6 MB/s | 808 kB 00:00 2022-09-07T21:26:17.965 INFO:teuthology.orchestra.run.smithi047.stdout:(18/132): python3-ceph-argparse-17.0.0-14857.g1 490 kB/s | 45 kB 00:00 2022-09-07T21:26:18.005 INFO:teuthology.orchestra.run.smithi059.stdout:(72/132): bison-3.0.4-10.el8.x86_64.rpm 5.4 MB/s | 688 kB 00:00 2022-09-07T21:26:18.048 INFO:teuthology.orchestra.run.smithi047.stdout:(19/132): python3-ceph-common-17.0.0-14857.g152 1.3 MB/s | 109 kB 00:00 2022-09-07T21:26:18.055 INFO:teuthology.orchestra.run.smithi059.stdout:(73/132): postgresql-10.21-2.module+el8.6.0+153 16 MB/s | 1.5 MB 00:00 2022-09-07T21:26:18.149 INFO:teuthology.orchestra.run.smithi059.stdout:(74/132): python3-numpy-1.14.3-10.el8.x86_64.rp 28 MB/s | 3.7 MB 00:00 2022-09-07T21:26:18.157 INFO:teuthology.orchestra.run.smithi047.stdout:(20/132): python3-cephfs-17.0.0-14857.g15219419 1.7 MB/s | 190 kB 00:00 2022-09-07T21:26:18.207 INFO:teuthology.orchestra.run.smithi059.stdout:(75/132): thrift-0.13.0-2.el8.x86_64.rpm 4.2 MB/s | 1.7 MB 00:00 2022-09-07T21:26:18.248 INFO:teuthology.orchestra.run.smithi059.stdout:(76/132): python3-werkzeug-0.12.2-4.el8.noarch. 4.1 MB/s | 457 kB 00:00 2022-09-07T21:26:18.316 INFO:teuthology.orchestra.run.smithi047.stdout:(21/132): python3-rados-17.0.0-14857.g15219419. 2.3 MB/s | 372 kB 00:00 2022-09-07T21:26:18.342 INFO:teuthology.orchestra.run.smithi059.stdout:(77/132): python3-markupsafe-0.23-19.el8.x86_64 289 kB/s | 39 kB 00:00 2022-09-07T21:26:18.466 INFO:teuthology.orchestra.run.smithi047.stdout:(22/132): python3-rbd-17.0.0-14857.g15219419.el 2.3 MB/s | 358 kB 00:00 2022-09-07T21:26:18.574 INFO:teuthology.orchestra.run.smithi047.stdout:(23/132): python3-rgw-17.0.0-14857.g15219419.el 990 kB/s | 107 kB 00:00 2022-09-07T21:26:18.582 INFO:teuthology.orchestra.run.smithi059.stdout:(78/132): postgresql-server-10.21-2.module+el8. 9.6 MB/s | 5.1 MB 00:00 2022-09-07T21:26:18.599 INFO:teuthology.orchestra.run.smithi059.stdout:(79/132): python3-prettytable-0.7.2-14.el8.noar 125 kB/s | 44 kB 00:00 2022-09-07T21:26:18.633 INFO:teuthology.orchestra.run.smithi059.stdout:(80/132): librdkafka-0.11.4-3.el8.x86_64.rpm 1.2 MB/s | 354 kB 00:00 2022-09-07T21:26:18.635 INFO:teuthology.orchestra.run.smithi047.stdout:(24/132): rbd-fuse-17.0.0-14857.g15219419.el8.x 1.6 MB/s | 94 kB 00:00 2022-09-07T21:26:18.675 INFO:teuthology.orchestra.run.smithi059.stdout:(81/132): python3-jinja2-2.10.1-3.el8.noarch.rp 5.7 MB/s | 538 kB 00:00 2022-09-07T21:26:18.734 INFO:teuthology.orchestra.run.smithi059.stdout:(82/132): python3-numpy-f2py-1.14.3-10.el8.x86_ 2.2 MB/s | 225 kB 00:00 2022-09-07T21:26:18.767 INFO:teuthology.orchestra.run.smithi059.stdout:(83/132): libtool-2.4.6-25.el8.x86_64.rpm 4.1 MB/s | 709 kB 00:00 2022-09-07T21:26:18.800 INFO:teuthology.orchestra.run.smithi059.stdout:(84/132): python3-mako-1.0.6-13.el8.noarch.rpm 2.3 MB/s | 157 kB 00:00 2022-09-07T21:26:18.843 INFO:teuthology.orchestra.run.smithi059.stdout:(85/132): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.4 MB/s | 103 kB 00:00 2022-09-07T21:26:18.886 INFO:teuthology.orchestra.run.smithi059.stdout:(86/132): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2022-09-07T21:26:18.919 INFO:teuthology.orchestra.run.smithi047.stdout:(25/132): librgw2-17.0.0-14857.g15219419.el8.x8 4.2 MB/s | 5.8 MB 00:01 2022-09-07T21:26:18.992 INFO:teuthology.orchestra.run.smithi047.stdout:(26/132): rbd-nbd-17.0.0-14857.g15219419.el8.x8 2.3 MB/s | 175 kB 00:00 2022-09-07T21:26:18.994 INFO:teuthology.orchestra.run.smithi059.stdout:(87/132): openblas-threads-0.3.15-3.el8.x86_64. 15 MB/s | 4.9 MB 00:00 2022-09-07T21:26:19.028 INFO:teuthology.orchestra.run.smithi059.stdout:(88/132): flex-2.6.1-9.el8.x86_64.rpm 2.2 MB/s | 320 kB 00:00 2022-09-07T21:26:19.043 INFO:teuthology.orchestra.run.smithi047.stdout:(27/132): ceph-grafana-dashboards-17.0.0-14857. 460 kB/s | 23 kB 00:00 2022-09-07T21:26:19.092 INFO:teuthology.orchestra.run.smithi047.stdout:(28/132): ceph-mgr-cephadm-17.0.0-14857.g152194 2.3 MB/s | 115 kB 00:00 2022-09-07T21:26:19.102 INFO:teuthology.orchestra.run.smithi059.stdout:(89/132): python3-babel-2.5.1-7.el8.noarch.rpm 14 MB/s | 4.8 MB 00:00 2022-09-07T21:26:19.128 INFO:teuthology.orchestra.run.smithi059.stdout:(90/132): jq-1.6-3.el8.x86_64.rpm 2.0 MB/s | 202 kB 00:00 2022-09-07T21:26:19.197 INFO:teuthology.orchestra.run.smithi059.stdout:(91/132): automake-1.16.1-7.el8.noarch.rpm 11 MB/s | 713 kB 00:00 2022-09-07T21:26:19.236 INFO:teuthology.orchestra.run.smithi059.stdout:(92/132): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2022-09-07T21:26:19.311 INFO:teuthology.orchestra.run.smithi059.stdout:(93/132): openblas-0.3.15-3.el8.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2022-09-07T21:26:19.337 INFO:teuthology.orchestra.run.smithi059.stdout:(94/132): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2022-09-07T21:26:19.370 INFO:teuthology.orchestra.run.smithi059.stdout:(95/132): python3-jsonpointer-1.10-11.el8.noarc 343 kB/s | 20 kB 00:00 2022-09-07T21:26:19.408 INFO:teuthology.orchestra.run.smithi047.stdout:(29/132): ceph-mgr-dashboard-17.0.0-14857.g1521 5.3 MB/s | 1.6 MB 00:00 2022-09-07T21:26:19.430 INFO:teuthology.orchestra.run.smithi059.stdout:(96/132): libpq-13.5-1.el8.x86_64.rpm 2.1 MB/s | 198 kB 00:00 2022-09-07T21:26:19.472 INFO:teuthology.orchestra.run.smithi059.stdout:(97/132): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2022-09-07T21:26:19.496 INFO:teuthology.orchestra.run.smithi059.stdout:(98/132): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.0 MB/s | 259 kB 00:00 2022-09-07T21:26:19.530 INFO:teuthology.orchestra.run.smithi059.stdout:(99/132): python3-jsonpatch-1.21-2.el8.noarch.r 453 kB/s | 27 kB 00:00 2022-09-07T21:26:19.555 INFO:teuthology.orchestra.run.smithi059.stdout:(100/132): uuid-1.6.2-43.el8.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2022-09-07T21:26:19.560 INFO:teuthology.orchestra.run.smithi047.stdout:(30/132): rbd-mirror-17.0.0-14857.g15219419.el8 3.7 MB/s | 3.5 MB 00:00 2022-09-07T21:26:19.606 INFO:teuthology.orchestra.run.smithi059.stdout:(101/132): python3-pyasn1-modules-0.3.7-6.el8.n 1.4 MB/s | 110 kB 00:00 2022-09-07T21:26:19.645 INFO:teuthology.orchestra.run.smithi047.stdout:(31/132): ceph-mgr-modules-core-17.0.0-14857.g1 2.7 MB/s | 233 kB 00:00 2022-09-07T21:26:19.673 INFO:teuthology.orchestra.run.smithi059.stdout:(102/132): pcre2-devel-10.32-2.el8.x86_64.rpm 5.0 MB/s | 605 kB 00:00 2022-09-07T21:26:19.693 INFO:teuthology.orchestra.run.smithi047.stdout:(32/132): ceph-mgr-rook-17.0.0-14857.g15219419. 941 kB/s | 47 kB 00:00 2022-09-07T21:26:19.706 INFO:teuthology.orchestra.run.smithi059.stdout:(103/132): fuse-2.9.7-15.el8.x86_64.rpm 826 kB/s | 83 kB 00:00 2022-09-07T21:26:19.723 INFO:teuthology.orchestra.run.smithi059.stdout:(104/132): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2022-09-07T21:26:19.746 INFO:teuthology.orchestra.run.smithi047.stdout:(33/132): ceph-prometheus-alerts-17.0.0-14857.g 288 kB/s | 14 kB 00:00 2022-09-07T21:26:19.765 INFO:teuthology.orchestra.run.smithi059.stdout:(105/132): pcre2-utf32-10.32-2.el8.x86_64.rpm 3.7 MB/s | 220 kB 00:00 2022-09-07T21:26:19.853 INFO:teuthology.orchestra.run.smithi047.stdout:(34/132): ceph-volume-17.0.0-14857.g15219419.el 2.4 MB/s | 263 kB 00:00 2022-09-07T21:26:19.945 INFO:teuthology.orchestra.run.smithi047.stdout:(35/132): cephadm-17.0.0-14857.g15219419.el8.no 886 kB/s | 81 kB 00:00 2022-09-07T21:26:19.949 INFO:teuthology.orchestra.run.smithi059.stdout:(106/132): python3-scipy-1.0.0-21.module+el8.5. 16 MB/s | 14 MB 00:00 2022-09-07T21:26:19.974 INFO:teuthology.orchestra.run.smithi059.stdout:(107/132): libselinux-devel-2.9-5.el8.x86_64.rp 802 kB/s | 200 kB 00:00 2022-09-07T21:26:19.987 INFO:teuthology.orchestra.run.smithi047.stdout:(36/132): cephfs-top-17.0.0-14857.g15219419.el8 373 kB/s | 15 kB 00:00 2022-09-07T21:26:20.008 INFO:teuthology.orchestra.run.smithi059.stdout:(108/132): libverto-devel-0.3.0-5.el8.x86_64.rp 74 kB/s | 18 kB 00:00 2022-09-07T21:26:20.025 INFO:teuthology.orchestra.run.smithi059.stdout:(109/132): librabbitmq-0.9.0-3.el8.x86_64.rpm 622 kB/s | 47 kB 00:00 2022-09-07T21:26:20.060 INFO:teuthology.orchestra.run.smithi059.stdout:(110/132): python3-cryptography-3.2.1-5.el8.x86 6.6 MB/s | 559 kB 00:00 2022-09-07T21:26:20.091 INFO:teuthology.orchestra.run.smithi059.stdout:(111/132): python3-oauthlib-2.1.0-1.el8.noarch. 1.8 MB/s | 155 kB 00:00 2022-09-07T21:26:20.117 INFO:teuthology.orchestra.run.smithi059.stdout:(112/132): python3-pycparser-2.14-14.el8.noarch 1.2 MB/s | 109 kB 00:00 2022-09-07T21:26:20.184 INFO:teuthology.orchestra.run.smithi059.stdout:(113/132): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.4 MB/s | 560 kB 00:00 2022-09-07T21:26:20.195 INFO:teuthology.orchestra.run.smithi047.stdout:(37/132): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2022-09-07T21:26:20.209 INFO:teuthology.orchestra.run.smithi059.stdout:(114/132): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.4 MB/s | 171 kB 00:00 2022-09-07T21:26:20.235 INFO:teuthology.orchestra.run.smithi059.stdout:(115/132): python3-jwt-1.6.1-2.el8.noarch.rpm 367 kB/s | 43 kB 00:00 2022-09-07T21:26:20.237 INFO:teuthology.orchestra.run.smithi047.stdout:(38/132): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2022-09-07T21:26:20.271 INFO:teuthology.orchestra.run.smithi059.stdout:(116/132): zlib-devel-1.2.11-18.el8_5.x86_64.rp 699 kB/s | 58 kB 00:00 2022-09-07T21:26:20.293 INFO:teuthology.orchestra.run.smithi059.stdout:(117/132): elfutils-libelf-devel-0.186-1.el8.x8 725 kB/s | 60 kB 00:00 2022-09-07T21:26:20.304 INFO:teuthology.orchestra.run.smithi047.stdout:(39/132): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2022-09-07T21:26:20.310 INFO:teuthology.orchestra.run.smithi059.stdout:(118/132): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.1 MB/s | 238 kB 00:00 2022-09-07T21:26:20.344 INFO:teuthology.orchestra.run.smithi059.stdout:(119/132): libaio-devel-0.3.112-1.el8.x86_64.rp 248 kB/s | 19 kB 00:00 2022-09-07T21:26:20.405 INFO:teuthology.orchestra.run.smithi047.stdout:(40/132): python3-asyncssh-2.7.0-2.el8.noarch.r 4.7 MB/s | 476 kB 00:00 2022-09-07T21:26:20.406 INFO:teuthology.orchestra.run.smithi059.stdout:(120/132): libuuid-devel-2.32.1-35.el8.x86_64.r 892 kB/s | 98 kB 00:00 2022-09-07T21:26:20.436 INFO:teuthology.orchestra.run.smithi059.stdout:(121/132): NetworkManager-initscripts-updown-1. 1.1 MB/s | 137 kB 00:00 2022-09-07T21:26:20.446 INFO:teuthology.orchestra.run.smithi047.stdout:(41/132): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.0 MB/s | 44 kB 00:00 2022-09-07T21:26:20.470 INFO:teuthology.orchestra.run.smithi059.stdout:(122/132): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.8 MB/s | 229 kB 00:00 2022-09-07T21:26:20.495 INFO:teuthology.orchestra.run.smithi059.stdout:(123/132): python3-requests-oauthlib-1.0.0-1.el 483 kB/s | 43 kB 00:00 2022-09-07T21:26:20.513 INFO:teuthology.orchestra.run.smithi047.stdout:(42/132): python3-beautifulsoup4-4.6.3-2.el8.1. 2.7 MB/s | 185 kB 00:00 2022-09-07T21:26:20.521 INFO:teuthology.orchestra.run.smithi059.stdout:(124/132): xfsprogs-devel-5.0.0-10.el8.x86_64.r 886 kB/s | 72 kB 00:00 2022-09-07T21:26:20.554 INFO:teuthology.orchestra.run.smithi059.stdout:(125/132): libcom_err-devel-1.45.6-4.el8.x86_64 467 kB/s | 39 kB 00:00 2022-09-07T21:26:20.564 INFO:teuthology.orchestra.run.smithi047.stdout:(43/132): python3-cachetools-3.1.1-4.el8.noarch 660 kB/s | 33 kB 00:00 2022-09-07T21:26:20.579 INFO:teuthology.orchestra.run.smithi059.stdout:(126/132): keyutils-libs-devel-1.5.10-9.el8.x86 842 kB/s | 48 kB 00:00 2022-09-07T21:26:20.690 INFO:teuthology.orchestra.run.smithi059.stdout:(127/132): openssl-devel-1.1.1k-7.el8_6.x86_64. 12 MB/s | 2.3 MB 00:00 2022-09-07T21:26:20.692 INFO:teuthology.orchestra.run.smithi047.stdout:(44/132): ceph-mgr-diskprediction-local-17.0.0- 5.7 MB/s | 7.4 MB 00:01 2022-09-07T21:26:20.713 INFO:teuthology.orchestra.run.smithi059.stdout:(128/132): libgfortran-8.5.0-10.1.el8_6.x86_64. 4.0 MB/s | 644 kB 00:00 2022-09-07T21:26:20.714 INFO:teuthology.orchestra.run.smithi047.stdout:(45/132): python3-certifi-2018.10.15-7.el8.noar 108 kB/s | 16 kB 00:00 2022-09-07T21:26:20.772 INFO:teuthology.orchestra.run.smithi059.stdout:(129/132): openssl-1.1.1k-7.el8_6.x86_64.rpm 12 MB/s | 709 kB 00:00 2022-09-07T21:26:20.807 INFO:teuthology.orchestra.run.smithi047.stdout:(46/132): python3-cherrypy-18.4.0-1.el8.noarch. 4.1 MB/s | 384 kB 00:00 2022-09-07T21:26:20.854 INFO:teuthology.orchestra.run.smithi059.stdout:(130/132): openssl-libs-1.1.1k-7.el8_6.x86_64.r 18 MB/s | 1.5 MB 00:00 2022-09-07T21:26:20.856 INFO:teuthology.orchestra.run.smithi047.stdout:(47/132): python3-google-auth-1.1.1-10.el8.noar 1.6 MB/s | 81 kB 00:00 2022-09-07T21:26:20.899 INFO:teuthology.orchestra.run.smithi047.stdout:(48/132): python3-cheroot-8.5.2-1.el8.noarch.rp 826 kB/s | 173 kB 00:00 2022-09-07T21:26:20.925 INFO:teuthology.orchestra.run.smithi047.stdout:(49/132): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2022-09-07T21:26:20.978 INFO:teuthology.orchestra.run.smithi047.stdout:(50/132): python3-jaraco-functools-2.0-4.el8.no 223 kB/s | 18 kB 00:00 2022-09-07T21:26:21.041 INFO:teuthology.orchestra.run.smithi047.stdout:(51/132): python3-logutils-0.3.5-11.el8.noarch. 778 kB/s | 49 kB 00:00 2022-09-07T21:26:21.091 INFO:teuthology.orchestra.run.smithi047.stdout:(52/132): python3-more-itertools-7.2.0-3.el8.no 1.2 MB/s | 59 kB 00:00 2022-09-07T21:26:21.133 INFO:teuthology.orchestra.run.smithi047.stdout:(53/132): python3-kubernetes-11.0.0-6.el8.noarc 5.6 MB/s | 1.2 MB 00:00 2022-09-07T21:26:21.157 INFO:teuthology.orchestra.run.smithi059.stdout:(131/132): librbd1-17.0.0-14857.g15219419.el8.x 7.6 MB/s | 3.6 MB 00:00 2022-09-07T21:26:21.159 INFO:teuthology.orchestra.run.smithi047.stdout:(54/132): python3-natsort-7.1.1-2.el8.noarch.rp 899 kB/s | 60 kB 00:00 2022-09-07T21:26:21.204 INFO:teuthology.orchestra.run.smithi047.stdout:(55/132): python3-pecan-1.3.2-9.el8.noarch.rpm 4.1 MB/s | 283 kB 00:00 2022-09-07T21:26:21.226 INFO:teuthology.orchestra.run.smithi047.stdout:(56/132): python3-portend-2.6-1.el8.noarch.rpm 234 kB/s | 16 kB 00:00 2022-09-07T21:26:21.251 INFO:teuthology.orchestra.run.smithi047.stdout:(57/132): python3-repoze-lru-0.7-6.el8.noarch.r 673 kB/s | 34 kB 00:00 2022-09-07T21:26:21.287 INFO:teuthology.orchestra.run.smithi059.stdout:(132/132): librados2-17.0.0-14857.g15219419.el8 5.1 MB/s | 3.6 MB 00:00 2022-09-07T21:26:21.287 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2022-09-07T21:26:21.288 INFO:teuthology.orchestra.run.smithi059.stdout:Total 11 MB/s | 216 MB 00:19 2022-09-07T21:26:21.294 INFO:teuthology.orchestra.run.smithi047.stdout:(58/132): python3-routes-2.4.1-12.el8.noarch.rp 2.9 MB/s | 196 kB 00:00 2022-09-07T21:26:21.344 INFO:teuthology.orchestra.run.smithi047.stdout:(59/132): python3-rsa-4.8-1.el8.noarch.rpm 671 kB/s | 61 kB 00:00 2022-09-07T21:26:21.369 INFO:teuthology.orchestra.run.smithi047.stdout:(60/132): python3-simplegeneric-0.8.1-17.el8.no 248 kB/s | 19 kB 00:00 2022-09-07T21:26:21.395 INFO:teuthology.orchestra.run.smithi047.stdout:(61/132): python3-singledispatch-3.4.0.3-18.el8 477 kB/s | 24 kB 00:00 2022-09-07T21:26:21.420 INFO:teuthology.orchestra.run.smithi047.stdout:(62/132): python3-tempora-1.14.1-5.el8.noarch.r 585 kB/s | 29 kB 00:00 2022-09-07T21:26:21.445 INFO:teuthology.orchestra.run.smithi047.stdout:(63/132): python3-trustme-0.6.0-4.el8.noarch.rp 550 kB/s | 27 kB 00:00 2022-09-07T21:26:21.487 INFO:teuthology.orchestra.run.smithi047.stdout:(64/132): python3-waitress-1.2.1-2.el8.1.noarch 3.5 MB/s | 241 kB 00:00 2022-09-07T21:26:21.530 INFO:teuthology.orchestra.run.smithi047.stdout:(65/132): python3-webob-1.8.5-1.el8.1.noarch.rp 2.9 MB/s | 251 kB 00:00 2022-09-07T21:26:21.554 INFO:teuthology.orchestra.run.smithi047.stdout:(66/132): python3-websocket-client-0.56.0-5.el8 918 kB/s | 61 kB 00:00 2022-09-07T21:26:21.589 INFO:teuthology.orchestra.run.smithi047.stdout:(67/132): python3-webtest-2.0.33-1.el8.noarch.r 1.4 MB/s | 85 kB 00:00 2022-09-07T21:26:21.613 INFO:teuthology.orchestra.run.smithi047.stdout:(68/132): python3-zc-lockfile-2.0-2.el8.noarch. 394 kB/s | 23 kB 00:00 2022-09-07T21:26:21.647 INFO:teuthology.orchestra.run.smithi047.stdout:(69/132): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.1 MB/s | 70 kB 00:00 2022-09-07T21:26:21.730 INFO:teuthology.orchestra.run.smithi047.stdout:(70/132): pcre2-devel-10.32-2.el8.x86_64.rpm 7.1 MB/s | 605 kB 00:00 2022-09-07T21:26:21.781 INFO:teuthology.orchestra.run.smithi047.stdout:(71/132): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2022-09-07T21:26:21.834 INFO:teuthology.orchestra.run.smithi047.stdout:(72/132): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2022-09-07T21:26:21.882 INFO:teuthology.orchestra.run.smithi047.stdout:(73/132): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2022-09-07T21:26:21.932 INFO:teuthology.orchestra.run.smithi047.stdout:(74/132): thrift-0.13.0-2.el8.x86_64.rpm 5.1 MB/s | 1.7 MB 00:00 2022-09-07T21:26:21.974 INFO:teuthology.orchestra.run.smithi047.stdout:(75/132): libselinux-devel-2.9-5.el8.x86_64.rpm 2.1 MB/s | 200 kB 00:00 2022-09-07T21:26:22.031 INFO:teuthology.orchestra.run.smithi047.stdout:(76/132): libverto-devel-0.3.0-5.el8.x86_64.rpm 185 kB/s | 18 kB 00:00 2022-09-07T21:26:22.066 INFO:teuthology.orchestra.run.smithi047.stdout:(77/132): librabbitmq-0.9.0-3.el8.x86_64.rpm 511 kB/s | 47 kB 00:00 2022-09-07T21:26:22.117 INFO:teuthology.orchestra.run.smithi047.stdout:(78/132): python3-oauthlib-2.1.0-1.el8.noarch.r 3.1 MB/s | 155 kB 00:00 2022-09-07T21:26:22.158 INFO:teuthology.orchestra.run.smithi047.stdout:(79/132): python3-cryptography-3.2.1-5.el8.x86_ 4.3 MB/s | 559 kB 00:00 2022-09-07T21:26:22.168 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2022-09-07T21:26:22.214 INFO:teuthology.orchestra.run.smithi047.stdout:(80/132): python3-pycparser-2.14-14.el8.noarch. 1.2 MB/s | 109 kB 00:00 2022-09-07T21:26:22.259 INFO:teuthology.orchestra.run.smithi047.stdout:(81/132): krb5-devel-1.18.2-14.el8.x86_64.rpm 5.5 MB/s | 560 kB 00:00 2022-09-07T21:26:22.301 INFO:teuthology.orchestra.run.smithi047.stdout:(82/132): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.8 MB/s | 171 kB 00:00 2022-09-07T21:26:22.334 INFO:teuthology.orchestra.run.smithi047.stdout:(83/132): python3-jwt-1.6.1-2.el8.noarch.rpm 568 kB/s | 43 kB 00:00 2022-09-07T21:26:22.341 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2022-09-07T21:26:22.342 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2022-09-07T21:26:22.384 INFO:teuthology.orchestra.run.smithi047.stdout:(84/132): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 710 kB/s | 58 kB 00:00 2022-09-07T21:26:22.429 INFO:teuthology.orchestra.run.smithi047.stdout:(85/132): elfutils-libelf-devel-0.186-1.el8.x86 653 kB/s | 60 kB 00:00 2022-09-07T21:26:22.470 INFO:teuthology.orchestra.run.smithi047.stdout:(86/132): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.8 MB/s | 238 kB 00:00 2022-09-07T21:26:22.495 INFO:teuthology.orchestra.run.smithi047.stdout:(87/132): libaio-devel-0.3.112-1.el8.x86_64.rpm 278 kB/s | 19 kB 00:00 2022-09-07T21:26:22.528 INFO:teuthology.orchestra.run.smithi047.stdout:(88/132): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.7 MB/s | 98 kB 00:00 2022-09-07T21:26:22.570 INFO:teuthology.orchestra.run.smithi047.stdout:(89/132): NetworkManager-initscripts-updown-1.3 1.8 MB/s | 137 kB 00:00 2022-09-07T21:26:22.613 INFO:teuthology.orchestra.run.smithi047.stdout:(90/132): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.7 MB/s | 229 kB 00:00 2022-09-07T21:26:22.637 INFO:teuthology.orchestra.run.smithi047.stdout:(91/132): python3-requests-oauthlib-1.0.0-1.el8 641 kB/s | 43 kB 00:00 2022-09-07T21:26:22.671 INFO:teuthology.orchestra.run.smithi047.stdout:(92/132): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.2 MB/s | 72 kB 00:00 2022-09-07T21:26:22.922 INFO:teuthology.orchestra.run.smithi047.stdout:(93/132): libcom_err-devel-1.45.6-4.el8.x86_64. 136 kB/s | 39 kB 00:00 2022-09-07T21:26:22.965 INFO:teuthology.orchestra.run.smithi047.stdout:(94/132): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.1 MB/s | 48 kB 00:00 2022-09-07T21:26:23.039 INFO:teuthology.orchestra.run.smithi047.stdout:(95/132): libgfortran-8.5.0-10.1.el8_6.x86_64.r 8.4 MB/s | 644 kB 00:00 2022-09-07T21:26:23.129 INFO:teuthology.orchestra.run.smithi047.stdout:(96/132): openssl-devel-1.1.1k-7.el8_6.x86_64.r 5.1 MB/s | 2.3 MB 00:00 2022-09-07T21:26:23.189 INFO:teuthology.orchestra.run.smithi047.stdout:(97/132): postgresql-contrib-10.21-2.module+el8 5.3 MB/s | 808 kB 00:00 2022-09-07T21:26:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2022-09-07T21:26:23.231 INFO:teuthology.orchestra.run.smithi047.stdout:(98/132): bison-3.0.4-10.el8.x86_64.rpm 6.5 MB/s | 688 kB 00:00 2022-09-07T21:26:23.312 INFO:teuthology.orchestra.run.smithi047.stdout:(99/132): postgresql-10.21-2.module+el8.6.0+153 13 MB/s | 1.5 MB 00:00 2022-09-07T21:26:23.433 INFO:teuthology.orchestra.run.smithi047.stdout:(100/132): python3-numpy-1.14.3-10.el8.x86_64.r 19 MB/s | 3.7 MB 00:00 2022-09-07T21:26:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/136 2022-09-07T21:26:23.508 INFO:teuthology.orchestra.run.smithi047.stdout:(101/132): python3-werkzeug-0.12.2-4.el8.noarch 6.0 MB/s | 457 kB 00:00 2022-09-07T21:26:23.540 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2022-09-07T21:26:23.545 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2022-09-07T21:26:23.549 INFO:teuthology.orchestra.run.smithi047.stdout:(102/132): python3-markupsafe-0.23-19.el8.x86_6 943 kB/s | 39 kB 00:00 2022-09-07T21:26:23.634 INFO:teuthology.orchestra.run.smithi047.stdout:(103/132): postgresql-server-10.21-2.module+el8 16 MB/s | 5.1 MB 00:00 2022-09-07T21:26:23.658 INFO:teuthology.orchestra.run.smithi047.stdout:(104/132): python3-prettytable-0.7.2-14.el8.noa 405 kB/s | 44 kB 00:00 2022-09-07T21:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout:(105/132): librdkafka-0.11.4-3.el8.x86_64.rpm 4.6 MB/s | 354 kB 00:00 2022-09-07T21:26:23.716 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/136 2022-09-07T21:26:23.755 INFO:teuthology.orchestra.run.smithi047.stdout:(106/132): python3-jinja2-2.10.1-3.el8.noarch.r 5.8 MB/s | 538 kB 00:00 2022-09-07T21:26:23.834 INFO:teuthology.orchestra.run.smithi047.stdout:(107/132): libtool-2.4.6-25.el8.x86_64.rpm 5.5 MB/s | 709 kB 00:00 2022-09-07T21:26:23.876 INFO:teuthology.orchestra.run.smithi047.stdout:(108/132): python3-numpy-f2py-1.14.3-10.el8.x86 1.8 MB/s | 225 kB 00:00 2022-09-07T21:26:23.888 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/136 2022-09-07T21:26:23.926 INFO:teuthology.orchestra.run.smithi047.stdout:(109/132): python3-mako-1.0.6-13.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2022-09-07T21:26:24.090 INFO:teuthology.orchestra.run.smithi047.stdout:(110/132): openblas-threads-0.3.15-3.el8.x86_64 19 MB/s | 4.9 MB 00:00 2022-09-07T21:26:24.135 INFO:teuthology.orchestra.run.smithi047.stdout:(111/132): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2022-09-07T21:26:24.135 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : liboath-2.6.2-3.el8.x86_64 3/136 2022-09-07T21:26:24.229 INFO:teuthology.orchestra.run.smithi047.stdout:(112/132): python3-babel-2.5.1-7.el8.noarch.rpm 16 MB/s | 4.8 MB 00:00 2022-09-07T21:26:24.269 INFO:teuthology.orchestra.run.smithi047.stdout:(113/132): socat-1.7.4.1-1.el8.x86_64.rpm 2.4 MB/s | 323 kB 00:00 2022-09-07T21:26:24.311 INFO:teuthology.orchestra.run.smithi047.stdout:(114/132): flex-2.6.1-9.el8.x86_64.rpm 3.7 MB/s | 320 kB 00:00 2022-09-07T21:26:24.324 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 4/136 2022-09-07T21:26:24.361 INFO:teuthology.orchestra.run.smithi047.stdout:(115/132): jq-1.6-3.el8.x86_64.rpm 4.0 MB/s | 202 kB 00:00 2022-09-07T21:26:24.506 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 5/136 2022-09-07T21:26:24.525 INFO:teuthology.orchestra.run.smithi047.stdout:(116/132): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2022-09-07T21:26:24.595 INFO:teuthology.orchestra.run.smithi047.stdout:(117/132): automake-1.16.1-7.el8.noarch.rpm 10 MB/s | 713 kB 00:00 2022-09-07T21:26:24.609 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 5/136 2022-09-07T21:26:24.629 INFO:teuthology.orchestra.run.smithi047.stdout:(118/132): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2022-09-07T21:26:24.679 INFO:teuthology.orchestra.run.smithi047.stdout:(119/132): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2022-09-07T21:26:24.704 INFO:teuthology.orchestra.run.smithi047.stdout:(120/132): python3-jsonpointer-1.10-11.el8.noar 815 kB/s | 20 kB 00:00 2022-09-07T21:26:24.755 INFO:teuthology.orchestra.run.smithi047.stdout:(121/132): libpq-13.5-1.el8.x86_64.rpm 3.9 MB/s | 198 kB 00:00 2022-09-07T21:26:24.805 INFO:teuthology.orchestra.run.smithi047.stdout:(122/132): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2022-09-07T21:26:24.848 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 6/136 2022-09-07T21:26:24.882 INFO:teuthology.orchestra.run.smithi047.stdout:(123/132): oniguruma-6.8.2-2.el8.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2022-09-07T21:26:24.922 INFO:teuthology.orchestra.run.smithi047.stdout:(124/132): python3-jsonpatch-1.21-2.el8.noarch. 642 kB/s | 27 kB 00:00 2022-09-07T21:26:24.956 INFO:teuthology.orchestra.run.smithi047.stdout:(125/132): uuid-1.6.2-43.el8.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2022-09-07T21:26:25.006 INFO:teuthology.orchestra.run.smithi047.stdout:(126/132): python3-pyasn1-modules-0.3.7-6.el8.n 2.2 MB/s | 110 kB 00:00 2022-09-07T21:26:25.182 INFO:teuthology.orchestra.run.smithi047.stdout:(127/132): python3-scipy-1.0.0-21.module+el8.5. 17 MB/s | 14 MB 00:00 2022-09-07T21:26:25.263 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 6/136 2022-09-07T21:26:25.517 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librados2-2:17.0.0-14857.g15219419.el8.x86_64 7/136 2022-09-07T21:26:25.915 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:17.0.0-14857.g15219419.el8.x86_64 7/136 2022-09-07T21:26:26.040 INFO:teuthology.orchestra.run.smithi047.stdout:(128/132): ceph-test-17.0.0-14857.g15219419.el8 5.5 MB/s | 54 MB 00:09 2022-09-07T21:26:26.117 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librbd1-2:17.0.0-14857.g15219419.el8.x86_64 8/136 2022-09-07T21:26:26.141 INFO:teuthology.orchestra.run.smithi047.stdout:(129/132): librados2-17.0.0-14857.g15219419.el8 3.2 MB/s | 3.6 MB 00:01 2022-09-07T21:26:26.191 INFO:teuthology.orchestra.run.smithi047.stdout:(130/132): openssl-1.1.1k-7.el8_6.x86_64.rpm 4.6 MB/s | 709 kB 00:00 2022-09-07T21:26:26.245 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:17.0.0-14857.g15219419.el8.x86_64 8/136 2022-09-07T21:26:26.258 INFO:teuthology.orchestra.run.smithi047.stdout:(131/132): openssl-libs-1.1.1k-7.el8_6.x86_64.r 13 MB/s | 1.5 MB 00:00 2022-09-07T21:26:26.376 INFO:teuthology.orchestra.run.smithi047.stdout:(132/132): librbd1-17.0.0-14857.g15219419.el8.x 3.0 MB/s | 3.6 MB 00:01 2022-09-07T21:26:26.379 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2022-09-07T21:26:26.380 INFO:teuthology.orchestra.run.smithi047.stdout:Total 9.8 MB/s | 216 MB 00:22 2022-09-07T21:26:26.447 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs2-2:17.0.0-14857.g15219419.el8.x86_64 9/136 2022-09-07T21:26:26.548 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs2-2:17.0.0-14857.g15219419.el8.x86_64 9/136 2022-09-07T21:26:26.771 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rados-2:17.0.0-14857.g15219419.el8.x86_6 10/136 2022-09-07T21:26:26.988 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libpq-13.5-1.el8.x86_64 11/136 2022-09-07T21:26:27.193 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 12/136 2022-09-07T21:26:27.592 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 13/136 2022-09-07T21:26:27.838 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 14/136 2022-09-07T21:26:28.022 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephsqlite-2:17.0.0-14857.g15219419.el8.x86_6 15/136 2022-09-07T21:26:28.121 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephsqlite-2:17.0.0-14857.g15219419.el8.x86_6 15/136 2022-09-07T21:26:28.255 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libradosstriper1-2:17.0.0-14857.g15219419.el8.x8 16/136 2022-09-07T21:26:28.366 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libradosstriper1-2:17.0.0-14857.g15219419.el8.x8 16/136 2022-09-07T21:26:28.527 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 17/136 2022-09-07T21:26:28.833 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 18/136 2022-09-07T21:26:29.300 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2022-09-07T21:26:29.438 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2022-09-07T21:26:29.518 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/136 2022-09-07T21:26:29.636 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2022-09-07T21:26:29.637 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2022-09-07T21:26:29.679 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/136 2022-09-07T21:26:29.731 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/136 2022-09-07T21:26:29.862 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 20/136 2022-09-07T21:26:29.880 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/136 2022-09-07T21:26:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 21/136 2022-09-07T21:26:30.073 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.2-3.el8.x86_64 3/136 2022-09-07T21:26:30.148 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 21/136 2022-09-07T21:26:30.240 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/136 2022-09-07T21:26:30.297 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 22/136 2022-09-07T21:26:30.312 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/136 2022-09-07T21:26:30.543 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 5/136 2022-09-07T21:26:30.773 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 6/136 2022-09-07T21:26:30.961 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2022-09-07T21:26:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2022-09-07T21:26:30.967 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 22/136 2022-09-07T21:26:31.206 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 6/136 2022-09-07T21:26:31.223 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 23/136 2022-09-07T21:26:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:17.0.0-14857.g15219419.el8.x86_64 7/136 2022-09-07T21:26:31.846 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 23/136 2022-09-07T21:26:31.860 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:17.0.0-14857.g15219419.el8.x86_64 7/136 2022-09-07T21:26:32.015 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:17.0.0-14857.g15219419.el8.x86_64 8/136 2022-09-07T21:26:32.072 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-0.3.15-3.el8.x86_64 24/136 2022-09-07T21:26:32.177 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:17.0.0-14857.g15219419.el8.x86_64 8/136 2022-09-07T21:26:32.373 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:17.0.0-14857.g15219419.el8.x86_64 9/136 2022-09-07T21:26:32.472 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:17.0.0-14857.g15219419.el8.x86_64 9/136 2022-09-07T21:26:32.647 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:17.0.0-14857.g15219419.el8.x86_6 10/136 2022-09-07T21:26:32.784 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 24/136 2022-09-07T21:26:32.888 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libpq-13.5-1.el8.x86_64 11/136 2022-09-07T21:26:33.102 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 12/136 2022-09-07T21:26:33.131 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/136 2022-09-07T21:26:33.399 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 26/136 2022-09-07T21:26:33.507 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 13/136 2022-09-07T21:26:33.579 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-common-2:17.0.0-14857.g15219419.el8 27/136 2022-09-07T21:26:33.741 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-argparse-2:17.0.0-14857.g15219419.e 28/136 2022-09-07T21:26:33.743 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 14/136 2022-09-07T21:26:33.898 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephsqlite-2:17.0.0-14857.g15219419.el8.x86_6 15/136 2022-09-07T21:26:34.006 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cephfs-2:17.0.0-14857.g15219419.el8.x86_ 29/136 2022-09-07T21:26:34.036 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephsqlite-2:17.0.0-14857.g15219419.el8.x86_6 15/136 2022-09-07T21:26:34.188 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 30/136 2022-09-07T21:26:34.231 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:17.0.0-14857.g15219419.el8.x8 16/136 2022-09-07T21:26:34.298 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:17.0.0-14857.g15219419.el8.x8 16/136 2022-09-07T21:26:34.487 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 17/136 2022-09-07T21:26:34.656 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 18/136 2022-09-07T21:26:35.080 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 18/136 2022-09-07T21:26:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 19/136 2022-09-07T21:26:35.922 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 19/136 2022-09-07T21:26:36.050 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 30/136 2022-09-07T21:26:36.153 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 20/136 2022-09-07T21:26:36.410 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 31/136 2022-09-07T21:26:36.593 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2022-09-07T21:26:36.785 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 20/136 2022-09-07T21:26:36.944 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/136 2022-09-07T21:26:36.981 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.15-3.el8.x86_64 21/136 2022-09-07T21:26:37.260 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/136 2022-09-07T21:26:37.450 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 32/136 2022-09-07T21:26:37.502 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/136 2022-09-07T21:26:37.667 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : liboath-2.6.2-3.el8.x86_64 3/136 2022-09-07T21:26:37.747 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 21/136 2022-09-07T21:26:37.827 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 33/136 2022-09-07T21:26:37.917 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 4/136 2022-09-07T21:26:38.045 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 22/136 2022-09-07T21:26:38.082 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 34/136 2022-09-07T21:26:38.144 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 5/136 2022-09-07T21:26:38.209 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 23/136 2022-09-07T21:26:38.307 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 5/136 2022-09-07T21:26:38.378 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/136 2022-09-07T21:26:38.503 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/136 2022-09-07T21:26:38.602 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 6/136 2022-09-07T21:26:38.609 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/136 2022-09-07T21:26:38.782 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 25/136 2022-09-07T21:26:38.828 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rbd-2:17.0.0-14857.g15219419.el8.x86_64 37/136 2022-09-07T21:26:39.032 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 26/136 2022-09-07T21:26:39.062 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librados-devel-2:17.0.0-14857.g15219419.el8.x86_ 38/136 2022-09-07T21:26:39.158 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 6/136 2022-09-07T21:26:39.229 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:17.0.0-14857.g15219419.el8 27/136 2022-09-07T21:26:39.308 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 39/136 2022-09-07T21:26:39.342 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:17.0.0-14857.g15219419.e 28/136 2022-09-07T21:26:39.427 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librados2-2:17.0.0-14857.g15219419.el8.x86_64 7/136 2022-09-07T21:26:39.479 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/136 2022-09-07T21:26:39.592 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/136 2022-09-07T21:26:39.613 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:17.0.0-14857.g15219419.el8.x86_ 29/136 2022-09-07T21:26:39.763 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 41/136 2022-09-07T21:26:39.908 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 41/136 2022-09-07T21:26:39.984 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-2:17.0.0-14857.g15219419.el8.x86_64 7/136 2022-09-07T21:26:40.070 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : jq-1.6-3.el8.x86_64 42/136 2022-09-07T21:26:40.243 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librbd1-2:17.0.0-14857.g15219419.el8.x86_64 8/136 2022-09-07T21:26:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 43/136 2022-09-07T21:26:40.433 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-2:17.0.0-14857.g15219419.el8.x86_64 8/136 2022-09-07T21:26:40.511 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 44/136 2022-09-07T21:26:40.603 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 30/136 2022-09-07T21:26:40.702 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs2-2:17.0.0-14857.g15219419.el8.x86_64 9/136 2022-09-07T21:26:40.738 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : automake-1.16.1-7.el8.noarch 45/136 2022-09-07T21:26:40.858 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs2-2:17.0.0-14857.g15219419.el8.x86_64 9/136 2022-09-07T21:26:40.936 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 46/136 2022-09-07T21:26:40.967 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/136 2022-09-07T21:26:41.076 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 47/136 2022-09-07T21:26:41.124 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rados-2:17.0.0-14857.g15219419.el8.x86_6 10/136 2022-09-07T21:26:41.263 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/136 2022-09-07T21:26:41.293 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 48/136 2022-09-07T21:26:41.428 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libpq-13.5-1.el8.x86_64 11/136 2022-09-07T21:26:41.472 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 33/136 2022-09-07T21:26:41.550 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 49/136 2022-09-07T21:26:41.701 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 12/136 2022-09-07T21:26:41.730 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 50/136 2022-09-07T21:26:41.988 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/136 2022-09-07T21:26:42.144 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 13/136 2022-09-07T21:26:42.449 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/136 2022-09-07T21:26:42.450 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 14/136 2022-09-07T21:26:42.627 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephsqlite-2:17.0.0-14857.g15219419.el8.x86_6 15/136 2022-09-07T21:26:42.739 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/136 2022-09-07T21:26:42.797 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephsqlite-2:17.0.0-14857.g15219419.el8.x86_6 15/136 2022-09-07T21:26:42.993 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libradosstriper1-2:17.0.0-14857.g15219419.el8.x8 16/136 2022-09-07T21:26:43.000 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/136 2022-09-07T21:26:43.133 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libradosstriper1-2:17.0.0-14857.g15219419.el8.x8 16/136 2022-09-07T21:26:43.289 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/136 2022-09-07T21:26:43.301 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 33/136 2022-09-07T21:26:43.352 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 17/136 2022-09-07T21:26:43.578 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/136 2022-09-07T21:26:43.684 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 34/136 2022-09-07T21:26:43.709 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 18/136 2022-09-07T21:26:43.935 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/136 2022-09-07T21:26:43.943 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/136 2022-09-07T21:26:43.945 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/136 2022-09-07T21:26:44.151 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 20/136 2022-09-07T21:26:44.185 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/136 2022-09-07T21:26:44.196 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/136 2022-09-07T21:26:44.318 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 21/136 2022-09-07T21:26:44.356 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/136 2022-09-07T21:26:44.387 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:17.0.0-14857.g15219419.el8.x86_64 37/136 2022-09-07T21:26:44.555 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 21/136 2022-09-07T21:26:44.630 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:17.0.0-14857.g15219419.el8.x86_ 38/136 2022-09-07T21:26:44.631 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/136 2022-09-07T21:26:44.777 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 22/136 2022-09-07T21:26:44.834 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 61/136 2022-09-07T21:26:44.861 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 39/136 2022-09-07T21:26:45.018 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 40/136 2022-09-07T21:26:45.157 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 62/136 2022-09-07T21:26:45.168 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 41/136 2022-09-07T21:26:45.359 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 63/136 2022-09-07T21:26:45.397 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/136 2022-09-07T21:26:45.608 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 22/136 2022-09-07T21:26:45.622 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 43/136 2022-09-07T21:26:45.633 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 64/136 2022-09-07T21:26:45.801 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 23/136 2022-09-07T21:26:45.897 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 44/136 2022-09-07T21:26:45.904 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/136 2022-09-07T21:26:46.172 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/136 2022-09-07T21:26:46.389 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 45/136 2022-09-07T21:26:46.413 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : fuse-2.9.7-15.el8.x86_64 67/136 2022-09-07T21:26:46.589 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 68/136 2022-09-07T21:26:46.613 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 23/136 2022-09-07T21:26:46.684 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 46/136 2022-09-07T21:26:46.818 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 69/136 2022-09-07T21:26:46.841 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 47/136 2022-09-07T21:26:46.868 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-0.3.15-3.el8.x86_64 24/136 2022-09-07T21:26:47.125 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 48/136 2022-09-07T21:26:47.126 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 70/136 2022-09-07T21:26:47.319 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 71/136 2022-09-07T21:26:47.395 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 49/136 2022-09-07T21:26:47.535 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 72/136 2022-09-07T21:26:47.732 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 73/136 2022-09-07T21:26:47.736 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 50/136 2022-09-07T21:26:47.861 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 24/136 2022-09-07T21:26:47.932 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 74/136 2022-09-07T21:26:47.989 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 51/136 2022-09-07T21:26:48.193 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 75/136 2022-09-07T21:26:48.199 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 52/136 2022-09-07T21:26:48.213 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/136 2022-09-07T21:26:48.358 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 76/136 2022-09-07T21:26:48.449 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 53/136 2022-09-07T21:26:48.511 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 26/136 2022-09-07T21:26:48.581 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 77/136 2022-09-07T21:26:48.625 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 54/136 2022-09-07T21:26:48.752 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 78/136 2022-09-07T21:26:48.776 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-common-2:17.0.0-14857.g15219419.el8 27/136 2022-09-07T21:26:48.957 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-argparse-2:17.0.0-14857.g15219419.e 28/136 2022-09-07T21:26:48.968 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 79/136 2022-09-07T21:26:48.999 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/136 2022-09-07T21:26:49.202 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/136 2022-09-07T21:26:49.208 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/136 2022-09-07T21:26:49.283 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cephfs-2:17.0.0-14857.g15219419.el8.x86_ 29/136 2022-09-07T21:26:49.401 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 81/136 2022-09-07T21:26:49.475 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 57/136 2022-09-07T21:26:49.534 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 30/136 2022-09-07T21:26:49.637 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-portend-2.6-1.el8.noarch 82/136 2022-09-07T21:26:49.703 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 58/136 2022-09-07T21:26:49.894 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 83/136 2022-09-07T21:26:49.989 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 59/136 2022-09-07T21:26:50.186 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 84/136 2022-09-07T21:26:50.248 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.7-15.el8.x86_64 60/136 2022-09-07T21:26:50.454 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/136 2022-09-07T21:26:50.463 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 85/136 2022-09-07T21:26:50.529 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/136 2022-09-07T21:26:50.720 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 62/136 2022-09-07T21:26:50.786 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 62/136 2022-09-07T21:26:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : jq-1.6-3.el8.x86_64 63/136 2022-09-07T21:26:51.171 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 64/136 2022-09-07T21:26:51.392 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 65/136 2022-09-07T21:26:51.618 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : automake-1.16.1-7.el8.noarch 66/136 2022-09-07T21:26:51.651 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 86/136 2022-09-07T21:26:51.790 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 67/136 2022-09-07T21:26:51.910 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 30/136 2022-09-07T21:26:51.917 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 87/136 2022-09-07T21:26:52.013 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 68/136 2022-09-07T21:26:52.187 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 88/136 2022-09-07T21:26:52.244 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 69/136 2022-09-07T21:26:52.344 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 31/136 2022-09-07T21:26:52.515 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 89/136 2022-09-07T21:26:52.551 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 70/136 2022-09-07T21:26:52.745 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 71/136 2022-09-07T21:26:52.812 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 90/136 2022-09-07T21:26:52.952 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 72/136 2022-09-07T21:26:53.063 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 91/136 2022-09-07T21:26:53.141 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 73/136 2022-09-07T21:26:53.374 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 74/136 2022-09-07T21:26:53.575 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 32/136 2022-09-07T21:26:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 75/136 2022-09-07T21:26:53.807 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 92/136 2022-09-07T21:26:53.842 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 76/136 2022-09-07T21:26:54.001 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 33/136 2022-09-07T21:26:54.002 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librgw2-2:17.0.0-14857.g15219419.el8.x86_64 93/136 2022-09-07T21:26:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 77/136 2022-09-07T21:26:54.058 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librgw2-2:17.0.0-14857.g15219419.el8.x86_64 93/136 2022-09-07T21:26:54.220 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rgw-2:17.0.0-14857.g15219419.el8.x86_64 94/136 2022-09-07T21:26:54.243 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 78/136 2022-09-07T21:26:54.348 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 34/136 2022-09-07T21:26:54.444 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 79/136 2022-09-07T21:26:54.616 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/136 2022-09-07T21:26:54.629 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/136 2022-09-07T21:26:54.810 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 81/136 2022-09-07T21:26:54.910 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/136 2022-09-07T21:26:55.087 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-2.6-1.el8.noarch 82/136 2022-09-07T21:26:55.152 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rbd-2:17.0.0-14857.g15219419.el8.x86_64 37/136 2022-09-07T21:26:55.337 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 83/136 2022-09-07T21:26:55.450 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librados-devel-2:17.0.0-14857.g15219419.el8.x86_ 38/136 2022-09-07T21:26:55.563 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 84/136 2022-09-07T21:26:55.705 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 39/136 2022-09-07T21:26:55.922 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 85/136 2022-09-07T21:26:55.983 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/136 2022-09-07T21:26:56.115 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/136 2022-09-07T21:26:56.350 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 41/136 2022-09-07T21:26:56.487 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 41/136 2022-09-07T21:26:56.718 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : jq-1.6-3.el8.x86_64 42/136 2022-09-07T21:26:56.911 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 43/136 2022-09-07T21:26:57.077 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 86/136 2022-09-07T21:26:57.252 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 44/136 2022-09-07T21:26:57.358 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 87/136 2022-09-07T21:26:57.553 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : automake-1.16.1-7.el8.noarch 45/136 2022-09-07T21:26:57.604 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 88/136 2022-09-07T21:26:57.795 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 46/136 2022-09-07T21:26:57.910 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 89/136 2022-09-07T21:26:57.940 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 47/136 2022-09-07T21:26:58.167 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 48/136 2022-09-07T21:26:58.218 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 90/136 2022-09-07T21:26:58.413 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 91/136 2022-09-07T21:26:58.454 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 49/136 2022-09-07T21:26:58.562 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 95/136 2022-09-07T21:26:58.720 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 50/136 2022-09-07T21:26:58.802 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 95/136 2022-09-07T21:26:58.971 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/136 2022-09-07T21:26:59.149 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 92/136 2022-09-07T21:26:59.303 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:17.0.0-14857.g15219419.el8.x86_64 93/136 2022-09-07T21:26:59.397 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 95/136 2022-09-07T21:26:59.423 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:17.0.0-14857.g15219419.el8.x86_64 93/136 2022-09-07T21:26:59.477 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/136 2022-09-07T21:26:59.616 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:17.0.0-14857.g15219419.el8.x86_64 94/136 2022-09-07T21:26:59.697 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-base-2:17.0.0-14857.g15219419.el8.x86_64 96/136 2022-09-07T21:26:59.768 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-base-2:17.0.0-14857.g15219419.el8.x86_64 96/136 2022-09-07T21:26:59.860 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/136 2022-09-07T21:26:59.943 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-selinux-2:17.0.0-14857.g15219419.el8.x86_64 97/136 2022-09-07T21:27:00.136 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/136 2022-09-07T21:27:00.445 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/136 2022-09-07T21:27:00.772 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/136 2022-09-07T21:27:01.184 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/136 2022-09-07T21:27:01.419 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/136 2022-09-07T21:27:01.629 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/136 2022-09-07T21:27:01.927 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/136 2022-09-07T21:27:02.099 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 61/136 2022-09-07T21:27:02.531 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 62/136 2022-09-07T21:27:02.788 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 63/136 2022-09-07T21:27:03.141 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 64/136 2022-09-07T21:27:03.474 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/136 2022-09-07T21:27:03.804 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/136 2022-09-07T21:27:03.942 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 95/136 2022-09-07T21:27:04.062 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : fuse-2.9.7-15.el8.x86_64 67/136 2022-09-07T21:27:04.144 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 95/136 2022-09-07T21:27:04.244 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 68/136 2022-09-07T21:27:04.489 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 69/136 2022-09-07T21:27:04.732 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 95/136 2022-09-07T21:27:04.887 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 70/136 2022-09-07T21:27:05.048 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:17.0.0-14857.g15219419.el8.x86_64 96/136 2022-09-07T21:27:05.133 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:17.0.0-14857.g15219419.el8.x86_64 96/136 2022-09-07T21:27:05.142 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 71/136 2022-09-07T21:27:05.285 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:17.0.0-14857.g15219419.el8.x86_64 97/136 2022-09-07T21:27:05.339 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 72/136 2022-09-07T21:27:05.549 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 73/136 2022-09-07T21:27:05.788 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 74/136 2022-09-07T21:27:06.030 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 75/136 2022-09-07T21:27:06.286 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 76/136 2022-09-07T21:27:06.537 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 77/136 2022-09-07T21:27:06.715 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 78/136 2022-09-07T21:27:07.008 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 79/136 2022-09-07T21:27:07.249 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/136 2022-09-07T21:27:07.449 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 81/136 2022-09-07T21:27:07.795 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-portend-2.6-1.el8.noarch 82/136 2022-09-07T21:27:08.092 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 83/136 2022-09-07T21:27:08.321 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 84/136 2022-09-07T21:27:08.617 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 85/136 2022-09-07T21:27:10.817 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 86/136 2022-09-07T21:27:11.125 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 87/136 2022-09-07T21:27:11.417 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 88/136 2022-09-07T21:27:11.814 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 89/136 2022-09-07T21:27:12.110 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 90/136 2022-09-07T21:27:12.366 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 91/136 2022-09-07T21:27:13.259 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 92/136 2022-09-07T21:27:13.439 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librgw2-2:17.0.0-14857.g15219419.el8.x86_64 93/136 2022-09-07T21:27:13.568 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librgw2-2:17.0.0-14857.g15219419.el8.x86_64 93/136 2022-09-07T21:27:13.798 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rgw-2:17.0.0-14857.g15219419.el8.x86_64 94/136 2022-09-07T21:27:19.497 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-selinux-2:17.0.0-14857.g15219419.el8.x86_64 97/136 2022-09-07T21:27:19.498 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /sys 2022-09-07T21:27:19.498 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /proc 2022-09-07T21:27:19.499 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /dev 2022-09-07T21:27:19.499 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /run 2022-09-07T21:27:19.499 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /mnt 2022-09-07T21:27:19.499 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /var/tmp 2022-09-07T21:27:19.499 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /home 2022-09-07T21:27:19.500 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /tmp 2022-09-07T21:27:19.500 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /dev 2022-09-07T21:27:19.500 INFO:teuthology.orchestra.run.smithi055.stdout: 2022-09-07T21:27:19.879 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 95/136 2022-09-07T21:27:19.882 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-volume-2:17.0.0-14857.g15219419.el8.noarch 98/136 2022-09-07T21:27:20.715 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 95/136 2022-09-07T21:27:21.481 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 95/136 2022-09-07T21:27:21.482 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-volume-2:17.0.0-14857.g15219419.el8.noarch 98/136 2022-09-07T21:27:21.660 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-osd-2:17.0.0-14857.g15219419.el8.x86_64 99/136 2022-09-07T21:27:21.839 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-base-2:17.0.0-14857.g15219419.el8.x86_64 96/136 2022-09-07T21:27:21.894 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-osd-2:17.0.0-14857.g15219419.el8.x86_64 99/136 2022-09-07T21:27:22.024 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-base-2:17.0.0-14857.g15219419.el8.x86_64 96/136 2022-09-07T21:27:22.082 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mds-2:17.0.0-14857.g15219419.el8.x86_64 100/136 2022-09-07T21:27:22.196 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-selinux-2:17.0.0-14857.g15219419.el8.x86_64 97/136 2022-09-07T21:27:22.491 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mds-2:17.0.0-14857.g15219419.el8.x86_64 100/136 2022-09-07T21:27:22.682 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mon-2:17.0.0-14857.g15219419.el8.x86_64 101/136 2022-09-07T21:27:22.701 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mon-2:17.0.0-14857.g15219419.el8.x86_64 101/136 2022-09-07T21:27:24.369 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:17.0.0-14857.g15219419.el8.noarch 102/136 2022-09-07T21:27:24.530 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephadm-2:17.0.0-14857.g15219419.el8.noarch 102/136 2022-09-07T21:27:24.665 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:17.0.0-14857.g15219419.el8.x86_64 97/136 2022-09-07T21:27:24.665 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2022-09-07T21:27:24.666 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2022-09-07T21:27:24.666 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /dev 2022-09-07T21:27:24.666 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /run 2022-09-07T21:27:24.667 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2022-09-07T21:27:24.667 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2022-09-07T21:27:24.667 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2022-09-07T21:27:24.667 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2022-09-07T21:27:24.667 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /dev 2022-09-07T21:27:24.668 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:27:24.688 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-14857.g15219419. 103/136 2022-09-07T21:27:24.845 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-14857.g15219419 104/136 2022-09-07T21:27:25.033 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-volume-2:17.0.0-14857.g15219419.el8.noarch 98/136 2022-09-07T21:27:25.057 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-14857.g15219419.el8.no 105/136 2022-09-07T21:27:25.268 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-14857.g15219419.el8.no 105/136 2022-09-07T21:27:25.526 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-14857.g15219419.el8. 106/136 2022-09-07T21:27:26.557 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-14857.g15219419.el8. 106/136 2022-09-07T21:27:26.608 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-volume-2:17.0.0-14857.g15219419.el8.noarch 98/136 2022-09-07T21:27:26.723 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-14857.g15 107/136 2022-09-07T21:27:26.818 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-14857.g15 107/136 2022-09-07T21:27:26.820 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:17.0.0-14857.g15219419.el8.x86_64 99/136 2022-09-07T21:27:27.049 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:17.0.0-14857.g15219419.el8.x86_64 99/136 2022-09-07T21:27:27.238 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-14857.g15219419.e 108/136 2022-09-07T21:27:27.266 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:17.0.0-14857.g15219419.el8.x86_64 100/136 2022-09-07T21:27:27.413 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-2:17.0.0-14857.g15219419.el8.x86_64 109/136 2022-09-07T21:27:27.450 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-2:17.0.0-14857.g15219419.el8.x86_64 109/136 2022-09-07T21:27:27.618 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-rook-2:17.0.0-14857.g15219419.el8.noarc 110/136 2022-09-07T21:27:27.626 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-14857.g15219419.el8.noarc 110/136 2022-09-07T21:27:27.670 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:17.0.0-14857.g15219419.el8.x86_64 100/136 2022-09-07T21:27:27.816 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:17.0.0-14857.g15219419.el8.x86_64 101/136 2022-09-07T21:27:27.834 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:17.0.0-14857.g15219419.el8.x86_64 101/136 2022-09-07T21:27:29.384 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-2:17.0.0-14857.g15219419.el8.x86_64 111/136 2022-09-07T21:27:29.602 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-radosgw-2:17.0.0-14857.g15219419.el8.x86_64 112/136 2022-09-07T21:27:29.638 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-14857.g15219419.el8.x86_64 112/136 2022-09-07T21:27:29.680 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:17.0.0-14857.g15219419.el8.noarch 102/136 2022-09-07T21:27:29.831 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-14857.g1521 113/136 2022-09-07T21:27:29.864 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:17.0.0-14857.g15219419.el8.noarch 102/136 2022-09-07T21:27:29.877 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-14857.g1521 113/136 2022-09-07T21:27:29.964 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-14857.g15219419. 103/136 2022-09-07T21:27:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephfs-mirror-2:17.0.0-14857.g15219419.el8.x86_6 114/136 2022-09-07T21:27:30.162 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-14857.g15219419 104/136 2022-09-07T21:27:30.374 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-14857.g15219419.el8.no 105/136 2022-09-07T21:27:30.396 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-14857.g15219419.el8.x86_6 114/136 2022-09-07T21:27:30.531 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-mirror-2:17.0.0-14857.g15219419.el8.x86_64 115/136 2022-09-07T21:27:30.584 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-14857.g15219419.el8.no 105/136 2022-09-07T21:27:30.952 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-14857.g15219419.el8. 106/136 2022-09-07T21:27:31.979 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-14857.g15219419.el8. 106/136 2022-09-07T21:27:32.115 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-14857.g15 107/136 2022-09-07T21:27:32.210 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-14857.g15 107/136 2022-09-07T21:27:32.595 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-14857.g15219419.e 108/136 2022-09-07T21:27:32.859 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:17.0.0-14857.g15219419.el8.x86_64 109/136 2022-09-07T21:27:32.897 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:17.0.0-14857.g15219419.el8.x86_64 109/136 2022-09-07T21:27:33.057 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:17.0.0-14857.g15219419.el8.noarc 110/136 2022-09-07T21:27:33.068 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-14857.g15219419.el8.noarc 110/136 2022-09-07T21:27:34.841 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:17.0.0-14857.g15219419.el8.x86_64 111/136 2022-09-07T21:27:35.080 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:17.0.0-14857.g15219419.el8.x86_64 112/136 2022-09-07T21:27:35.115 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-14857.g15219419.el8.x86_64 112/136 2022-09-07T21:27:35.307 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-14857.g1521 113/136 2022-09-07T21:27:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-14857.g1521 113/136 2022-09-07T21:27:35.541 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephfs-mirror-2:17.0.0-14857.g15219419.el8.x86_6 114/136 2022-09-07T21:27:35.909 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-14857.g15219419.el8.x86_6 114/136 2022-09-07T21:27:35.947 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-mirror-2:17.0.0-14857.g15219419.el8.x86_64 115/136 2022-09-07T21:27:36.100 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:17.0.0-14857.g15219419.el8.x86_64 115/136 2022-09-07T21:27:36.234 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-test-2:17.0.0-14857.g15219419.el8.x86_64 116/136 2022-09-07T21:27:37.123 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-fuse-2:17.0.0-14857.g15219419.el8.x86_64 117/136 2022-09-07T21:27:37.400 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/136 2022-09-07T21:27:37.670 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 119/136 2022-09-07T21:27:37.897 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libtool-2.4.6-25.el8.x86_64 120/136 2022-09-07T21:27:38.101 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 120/136 2022-09-07T21:27:38.351 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 121/136 2022-09-07T21:27:38.551 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs-devel-2:17.0.0-14857.g15219419.el8.x86 122/136 2022-09-07T21:27:38.706 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 123/136 2022-09-07T21:27:40.852 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-07T21:27:41.130 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-07T21:27:41.149 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-07T21:27:41.405 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephfs-top-2:17.0.0-14857.g15219419.el8.noarch 125/136 2022-09-07T21:27:41.453 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:17.0.0-14857.g15219419.el8.x86_64 115/136 2022-09-07T21:27:41.647 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-fuse-2:17.0.0-14857.g15219419.el8.x86_64 126/136 2022-09-07T21:27:41.769 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:17.0.0-14857.g15219419.el8.x86_64 116/136 2022-09-07T21:27:41.964 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libtool-2.4.6-25.el8.x86_64 117/136 2022-09-07T21:27:41.980 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-nbd-2:17.0.0-14857.g15219419.el8.x86_64 127/136 2022-09-07T21:27:42.167 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 117/136 2022-09-07T21:27:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : openssl-1:1.1.1k-7.el8_6.x86_64 128/136 2022-09-07T21:27:42.492 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 118/136 2022-09-07T21:27:42.579 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flex-2.6.1-9.el8.x86_64 129/136 2022-09-07T21:27:42.660 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 129/136 2022-09-07T21:27:42.863 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : bison-3.0.4-10.el8.x86_64 130/136 2022-09-07T21:27:42.876 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 130/136 2022-09-07T21:27:43.071 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-7.el8 131/136 2022-09-07T21:27:43.080 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-7.el8 131/136 2022-09-07T21:27:43.243 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 132/136 2022-09-07T21:27:43.399 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:17.0.0-14857.g15219419.el8.x86_64 119/136 2022-09-07T21:27:43.457 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 133/136 2022-09-07T21:27:43.457 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 134/136 2022-09-07T21:27:43.666 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 120/136 2022-09-07T21:27:43.667 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 134/136 2022-09-07T21:27:43.668 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 135/136 2022-09-07T21:27:43.860 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 121/136 2022-09-07T21:27:43.882 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 135/136 2022-09-07T21:27:43.883 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-07T21:27:44.043 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:17.0.0-14857.g15219419.el8.x86 122/136 2022-09-07T21:27:44.206 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 123/136 2022-09-07T21:27:44.795 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-07T21:27:44.795 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-2:17.0.0-14857.g15219419.el8.x86_64 1/136 2022-09-07T21:27:44.795 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-base-2:17.0.0-14857.g15219419.el8.x86_64 2/136 2022-09-07T21:27:44.795 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 3/136 2022-09-07T21:27:44.796 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-fuse-2:17.0.0-14857.g15219419.el8.x86_64 4/136 2022-09-07T21:27:44.796 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-14857.g1521 5/136 2022-09-07T21:27:44.796 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mds-2:17.0.0-14857.g15219419.el8.x86_64 6/136 2022-09-07T21:27:44.796 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-2:17.0.0-14857.g15219419.el8.x86_64 7/136 2022-09-07T21:27:44.797 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mon-2:17.0.0-14857.g15219419.el8.x86_64 8/136 2022-09-07T21:27:44.797 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-osd-2:17.0.0-14857.g15219419.el8.x86_64 9/136 2022-09-07T21:27:44.797 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-radosgw-2:17.0.0-14857.g15219419.el8.x86_64 10/136 2022-09-07T21:27:44.797 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-selinux-2:17.0.0-14857.g15219419.el8.x86_64 11/136 2022-09-07T21:27:44.797 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-test-2:17.0.0-14857.g15219419.el8.x86_64 12/136 2022-09-07T21:27:44.798 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephfs-mirror-2:17.0.0-14857.g15219419.el8.x86_6 13/136 2022-09-07T21:27:44.798 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs-devel-2:17.0.0-14857.g15219419.el8.x86 14/136 2022-09-07T21:27:44.798 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs2-2:17.0.0-14857.g15219419.el8.x86_64 15/136 2022-09-07T21:27:44.798 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephsqlite-2:17.0.0-14857.g15219419.el8.x86_6 16/136 2022-09-07T21:27:44.799 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados-devel-2:17.0.0-14857.g15219419.el8.x86_ 17/136 2022-09-07T21:27:44.799 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libradosstriper1-2:17.0.0-14857.g15219419.el8.x8 18/136 2022-09-07T21:27:44.799 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librgw2-2:17.0.0-14857.g15219419.el8.x86_64 19/136 2022-09-07T21:27:44.799 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-argparse-2:17.0.0-14857.g15219419.e 20/136 2022-09-07T21:27:44.800 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-common-2:17.0.0-14857.g15219419.el8 21/136 2022-09-07T21:27:44.800 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cephfs-2:17.0.0-14857.g15219419.el8.x86_ 22/136 2022-09-07T21:27:44.800 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rados-2:17.0.0-14857.g15219419.el8.x86_6 23/136 2022-09-07T21:27:44.800 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rbd-2:17.0.0-14857.g15219419.el8.x86_64 24/136 2022-09-07T21:27:44.800 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rgw-2:17.0.0-14857.g15219419.el8.x86_64 25/136 2022-09-07T21:27:44.801 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-fuse-2:17.0.0-14857.g15219419.el8.x86_64 26/136 2022-09-07T21:27:44.801 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-mirror-2:17.0.0-14857.g15219419.el8.x86_64 27/136 2022-09-07T21:27:44.801 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-nbd-2:17.0.0-14857.g15219419.el8.x86_64 28/136 2022-09-07T21:27:44.801 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-14857.g15219419 29/136 2022-09-07T21:27:44.815 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-14857.g15219419.el8.no 30/136 2022-09-07T21:27:44.816 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-14857.g15219419.el8. 31/136 2022-09-07T21:27:44.816 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-14857.g15 32/136 2022-09-07T21:27:44.816 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-14857.g15219419.e 33/136 2022-09-07T21:27:44.817 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-rook-2:17.0.0-14857.g15219419.el8.noarc 34/136 2022-09-07T21:27:44.817 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-14857.g15219419. 35/136 2022-09-07T21:27:44.817 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-volume-2:17.0.0-14857.g15219419.el8.noarch 36/136 2022-09-07T21:27:44.817 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephadm-2:17.0.0-14857.g15219419.el8.noarch 37/136 2022-09-07T21:27:44.818 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephfs-top-2:17.0.0-14857.g15219419.el8.noarch 38/136 2022-09-07T21:27:44.818 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/136 2022-09-07T21:27:44.818 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/136 2022-09-07T21:27:44.819 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/136 2022-09-07T21:27:44.819 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/136 2022-09-07T21:27:44.819 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/136 2022-09-07T21:27:44.819 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/136 2022-09-07T21:27:44.820 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/136 2022-09-07T21:27:44.820 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/136 2022-09-07T21:27:44.820 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/136 2022-09-07T21:27:44.821 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/136 2022-09-07T21:27:44.821 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/136 2022-09-07T21:27:44.822 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/136 2022-09-07T21:27:44.822 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/136 2022-09-07T21:27:44.822 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/136 2022-09-07T21:27:44.823 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/136 2022-09-07T21:27:44.823 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/136 2022-09-07T21:27:44.824 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/136 2022-09-07T21:27:44.825 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/136 2022-09-07T21:27:44.825 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/136 2022-09-07T21:27:44.826 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/136 2022-09-07T21:27:44.826 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/136 2022-09-07T21:27:44.827 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/136 2022-09-07T21:27:44.827 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/136 2022-09-07T21:27:44.828 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/136 2022-09-07T21:27:44.828 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/136 2022-09-07T21:27:44.828 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/136 2022-09-07T21:27:44.829 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/136 2022-09-07T21:27:44.829 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/136 2022-09-07T21:27:44.829 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/136 2022-09-07T21:27:44.829 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/136 2022-09-07T21:27:44.830 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/136 2022-09-07T21:27:44.830 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/136 2022-09-07T21:27:44.830 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/136 2022-09-07T21:27:44.830 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 72/136 2022-09-07T21:27:44.831 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 73/136 2022-09-07T21:27:44.831 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 74/136 2022-09-07T21:27:44.831 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/136 2022-09-07T21:27:44.832 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 76/136 2022-09-07T21:27:44.832 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 77/136 2022-09-07T21:27:44.832 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 78/136 2022-09-07T21:27:44.832 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 79/136 2022-09-07T21:27:44.833 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 80/136 2022-09-07T21:27:44.833 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 81/136 2022-09-07T21:27:44.833 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 82/136 2022-09-07T21:27:44.833 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 83/136 2022-09-07T21:27:44.833 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 84/136 2022-09-07T21:27:44.834 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/136 2022-09-07T21:27:44.834 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 86/136 2022-09-07T21:27:44.834 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/136 2022-09-07T21:27:44.835 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 88/136 2022-09-07T21:27:44.835 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 89/136 2022-09-07T21:27:44.835 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-7.el8 90/136 2022-09-07T21:27:44.835 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 91/136 2022-09-07T21:27:44.836 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 92/136 2022-09-07T21:27:44.836 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 93/136 2022-09-07T21:27:44.836 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/136 2022-09-07T21:27:44.836 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 95/136 2022-09-07T21:27:44.837 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 96/136 2022-09-07T21:27:44.837 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 97/136 2022-09-07T21:27:44.837 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 98/136 2022-09-07T21:27:44.837 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : bison-3.0.4-10.el8.x86_64 99/136 2022-09-07T21:27:44.838 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 100/136 2022-09-07T21:27:44.838 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 101/136 2022-09-07T21:27:44.838 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 102/136 2022-09-07T21:27:44.838 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 103/136 2022-09-07T21:27:44.839 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 104/136 2022-09-07T21:27:44.839 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 105/136 2022-09-07T21:27:44.839 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 106/136 2022-09-07T21:27:44.839 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 107/136 2022-09-07T21:27:44.839 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 108/136 2022-09-07T21:27:44.840 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 109/136 2022-09-07T21:27:44.840 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 110/136 2022-09-07T21:27:44.840 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 111/136 2022-09-07T21:27:44.840 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 112/136 2022-09-07T21:27:44.841 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 113/136 2022-09-07T21:27:44.841 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 114/136 2022-09-07T21:27:44.841 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flex-2.6.1-9.el8.x86_64 115/136 2022-09-07T21:27:44.841 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 116/136 2022-09-07T21:27:44.842 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : jq-1.6-3.el8.x86_64 117/136 2022-09-07T21:27:44.842 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 118/136 2022-09-07T21:27:44.842 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : automake-1.16.1-7.el8.noarch 119/136 2022-09-07T21:27:44.842 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 120/136 2022-09-07T21:27:44.843 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 121/136 2022-09-07T21:27:44.843 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 122/136 2022-09-07T21:27:44.843 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libpq-13.5-1.el8.x86_64 123/136 2022-09-07T21:27:44.843 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 124/136 2022-09-07T21:27:44.843 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 125/136 2022-09-07T21:27:44.844 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 126/136 2022-09-07T21:27:44.844 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 127/136 2022-09-07T21:27:44.844 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 128/136 2022-09-07T21:27:44.844 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:17.0.0-14857.g15219419.el8.x86_64 129/136 2022-09-07T21:27:44.845 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/136 2022-09-07T21:27:44.845 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:17.0.0-14857.g15219419.el8.x86_64 131/136 2022-09-07T21:27:44.845 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/136 2022-09-07T21:27:44.845 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openssl-1:1.1.1k-7.el8_6.x86_64 133/136 2022-09-07T21:27:44.846 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 134/136 2022-09-07T21:27:44.846 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openssl-libs-1:1.1.1k-7.el8_6.x86_64 135/136 2022-09-07T21:27:44.870 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-selinux-2:17.0.0-14857.g15219419.el8.x86_64 97/136 2022-09-07T21:27:44.870 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /sys 2022-09-07T21:27:44.871 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /proc 2022-09-07T21:27:44.871 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /dev 2022-09-07T21:27:44.871 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /run 2022-09-07T21:27:44.871 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /mnt 2022-09-07T21:27:44.872 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /var/tmp 2022-09-07T21:27:44.872 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /home 2022-09-07T21:27:44.872 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /tmp 2022-09-07T21:27:44.873 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /dev 2022-09-07T21:27:44.873 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:27:45.294 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-volume-2:17.0.0-14857.g15219419.el8.noarch 98/136 2022-09-07T21:27:46.493 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-07T21:27:46.780 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-07T21:27:46.799 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-07T21:27:46.809 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-07T21:27:46.809 INFO:teuthology.orchestra.run.smithi055.stdout:Installed products updated. 2022-09-07T21:27:47.045 INFO:teuthology.orchestra.run.smithi055.stdout: 2022-09-07T21:27:47.045 INFO:teuthology.orchestra.run.smithi055.stdout:Upgraded: 2022-09-07T21:27:47.045 INFO:teuthology.orchestra.run.smithi055.stdout: librados2-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.046 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.046 INFO:teuthology.orchestra.run.smithi055.stdout: openssl-1:1.1.1k-7.el8_6.x86_64 2022-09-07T21:27:47.046 INFO:teuthology.orchestra.run.smithi055.stdout: openssl-libs-1:1.1.1k-7.el8_6.x86_64 2022-09-07T21:27:47.046 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2022-09-07T21:27:47.047 INFO:teuthology.orchestra.run.smithi055.stdout: NetworkManager-initscripts-updown-1:1.36.0-7.el8_6.noarch 2022-09-07T21:27:47.047 INFO:teuthology.orchestra.run.smithi055.stdout: automake-1.16.1-7.el8.noarch 2022-09-07T21:27:47.047 INFO:teuthology.orchestra.run.smithi055.stdout: bison-3.0.4-10.el8.x86_64 2022-09-07T21:27:47.047 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.048 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.048 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.048 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.048 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:47.048 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.049 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.049 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.049 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:47.049 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:47.050 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:47.050 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:47.050 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:47.050 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.051 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.051 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:47.051 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.052 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.052 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.052 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:47.052 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:47.053 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-mirror-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.053 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-top-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:47.053 INFO:teuthology.orchestra.run.smithi055.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2022-09-07T21:27:47.053 INFO:teuthology.orchestra.run.smithi055.stdout: flex-2.6.1-9.el8.x86_64 2022-09-07T21:27:47.054 INFO:teuthology.orchestra.run.smithi055.stdout: fuse-2.9.7-15.el8.x86_64 2022-09-07T21:27:47.054 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-09-07T21:27:47.054 INFO:teuthology.orchestra.run.smithi055.stdout: jq-1.6-3.el8.x86_64 2022-09-07T21:27:47.054 INFO:teuthology.orchestra.run.smithi055.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2022-09-07T21:27:47.054 INFO:teuthology.orchestra.run.smithi055.stdout: krb5-devel-1.18.2-14.el8.x86_64 2022-09-07T21:27:47.055 INFO:teuthology.orchestra.run.smithi055.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-09-07T21:27:47.055 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.055 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.055 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.056 INFO:teuthology.orchestra.run.smithi055.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2022-09-07T21:27:47.056 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2022-09-07T21:27:47.056 INFO:teuthology.orchestra.run.smithi055.stdout: liboath-2.6.2-3.el8.x86_64 2022-09-07T21:27:47.056 INFO:teuthology.orchestra.run.smithi055.stdout: libpq-13.5-1.el8.x86_64 2022-09-07T21:27:47.056 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2022-09-07T21:27:47.057 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-09-07T21:27:47.057 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.057 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.057 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-09-07T21:27:47.057 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.058 INFO:teuthology.orchestra.run.smithi055.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-09-07T21:27:47.058 INFO:teuthology.orchestra.run.smithi055.stdout: libsepol-devel-2.9-3.el8.x86_64 2022-09-07T21:27:47.058 INFO:teuthology.orchestra.run.smithi055.stdout: libtool-2.4.6-25.el8.x86_64 2022-09-07T21:27:47.059 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind-1.3.1-3.el8.x86_64 2022-09-07T21:27:47.059 INFO:teuthology.orchestra.run.smithi055.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2022-09-07T21:27:47.059 INFO:teuthology.orchestra.run.smithi055.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-09-07T21:27:47.059 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-09-07T21:27:47.060 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-09-07T21:27:47.060 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-0.3.15-3.el8.x86_64 2022-09-07T21:27:47.060 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-09-07T21:27:47.061 INFO:teuthology.orchestra.run.smithi055.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2022-09-07T21:27:47.061 INFO:teuthology.orchestra.run.smithi055.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-09-07T21:27:47.061 INFO:teuthology.orchestra.run.smithi055.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-09-07T21:27:47.061 INFO:teuthology.orchestra.run.smithi055.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-09-07T21:27:47.062 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-07T21:27:47.062 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-07T21:27:47.062 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-07T21:27:47.062 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-09-07T21:27:47.062 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel-2.5.1-7.el8.noarch 2022-09-07T21:27:47.063 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-09-07T21:27:47.063 INFO:teuthology.orchestra.run.smithi055.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-09-07T21:27:47.063 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-09-07T21:27:47.064 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.064 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.064 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.064 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-09-07T21:27:47.065 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-09-07T21:27:47.065 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-09-07T21:27:47.065 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-09-07T21:27:47.065 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-09-07T21:27:47.065 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-09-07T21:27:47.066 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-6.2-6.el8.noarch 2022-09-07T21:27:47.066 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-09-07T21:27:47.066 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-09-07T21:27:47.066 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-09-07T21:27:47.067 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-09-07T21:27:47.067 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-09-07T21:27:47.067 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-09-07T21:27:47.067 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-09-07T21:27:47.067 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako-1.0.6-13.el8.noarch 2022-09-07T21:27:47.068 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-09-07T21:27:47.068 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-09-07T21:27:47.068 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-09-07T21:27:47.068 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-09-07T21:27:47.069 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-09-07T21:27:47.069 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-09-07T21:27:47.069 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-09-07T21:27:47.070 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend-2.6-1.el8.noarch 2022-09-07T21:27:47.070 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-09-07T21:27:47.070 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-09-07T21:27:47.070 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-09-07T21:27:47.071 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-09-07T21:27:47.071 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser-2.14-14.el8.noarch 2022-09-07T21:27:47.071 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz-2017.2-9.el8.noarch 2022-09-07T21:27:47.071 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.071 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.072 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-09-07T21:27:47.072 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-09-07T21:27:47.072 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.072 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes-2.4.1-12.el8.noarch 2022-09-07T21:27:47.073 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa-4.8-1.el8.noarch 2022-09-07T21:27:47.073 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2022-09-07T21:27:47.073 INFO:teuthology.orchestra.run.smithi055.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-09-07T21:27:47.073 INFO:teuthology.orchestra.run.smithi055.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-09-07T21:27:47.073 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-09-07T21:27:47.074 INFO:teuthology.orchestra.run.smithi055.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-09-07T21:27:47.074 INFO:teuthology.orchestra.run.smithi055.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-09-07T21:27:47.074 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-09-07T21:27:47.074 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-09-07T21:27:47.075 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-09-07T21:27:47.075 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-09-07T21:27:47.075 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-09-07T21:27:47.075 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.076 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.076 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:47.076 INFO:teuthology.orchestra.run.smithi055.stdout: socat-1.7.4.1-1.el8.x86_64 2022-09-07T21:27:47.076 INFO:teuthology.orchestra.run.smithi055.stdout: thrift-0.13.0-2.el8.x86_64 2022-09-07T21:27:47.076 INFO:teuthology.orchestra.run.smithi055.stdout: uuid-1.6.2-43.el8.x86_64 2022-09-07T21:27:47.077 INFO:teuthology.orchestra.run.smithi055.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2022-09-07T21:27:47.077 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-09-07T21:27:47.078 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephfs-top-2:17.0.0-14857.g15219419.el8.noarch 125/136 2022-09-07T21:27:47.078 INFO:teuthology.orchestra.run.smithi055.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2022-09-07T21:27:47.079 INFO:teuthology.orchestra.run.smithi055.stdout: 2022-09-07T21:27:47.079 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2022-09-07T21:27:47.156 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-volume-2:17.0.0-14857.g15219419.el8.noarch 98/136 2022-09-07T21:27:47.245 DEBUG:teuthology.parallel:result is None 2022-09-07T21:27:47.322 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:17.0.0-14857.g15219419.el8.x86_64 126/136 2022-09-07T21:27:47.427 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-osd-2:17.0.0-14857.g15219419.el8.x86_64 99/136 2022-09-07T21:27:47.615 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:17.0.0-14857.g15219419.el8.x86_64 127/136 2022-09-07T21:27:47.754 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-osd-2:17.0.0-14857.g15219419.el8.x86_64 99/136 2022-09-07T21:27:47.882 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : openssl-1:1.1.1k-7.el8_6.x86_64 128/136 2022-09-07T21:27:47.935 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mds-2:17.0.0-14857.g15219419.el8.x86_64 100/136 2022-09-07T21:27:48.072 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-7.el8 129/136 2022-09-07T21:27:48.083 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-7.el8 129/136 2022-09-07T21:27:48.278 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 130/136 2022-09-07T21:27:48.464 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mds-2:17.0.0-14857.g15219419.el8.x86_64 100/136 2022-09-07T21:27:48.540 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flex-2.6.1-9.el8.x86_64 131/136 2022-09-07T21:27:48.622 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 131/136 2022-09-07T21:27:48.644 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mon-2:17.0.0-14857.g15219419.el8.x86_64 101/136 2022-09-07T21:27:48.717 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mon-2:17.0.0-14857.g15219419.el8.x86_64 101/136 2022-09-07T21:27:48.898 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : bison-3.0.4-10.el8.x86_64 132/136 2022-09-07T21:27:48.914 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 132/136 2022-09-07T21:27:49.134 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 133/136 2022-09-07T21:27:49.134 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 134/136 2022-09-07T21:27:49.344 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 134/136 2022-09-07T21:27:49.345 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 135/136 2022-09-07T21:27:49.599 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 135/136 2022-09-07T21:27:49.599 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-07T21:27:50.486 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-07T21:27:50.486 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:17.0.0-14857.g15219419.el8.x86_64 1/136 2022-09-07T21:27:50.487 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:17.0.0-14857.g15219419.el8.x86_64 2/136 2022-09-07T21:27:50.487 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 3/136 2022-09-07T21:27:50.487 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:17.0.0-14857.g15219419.el8.x86_64 4/136 2022-09-07T21:27:50.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-14857.g1521 5/136 2022-09-07T21:27:50.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:17.0.0-14857.g15219419.el8.x86_64 6/136 2022-09-07T21:27:50.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:17.0.0-14857.g15219419.el8.x86_64 7/136 2022-09-07T21:27:50.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:17.0.0-14857.g15219419.el8.x86_64 8/136 2022-09-07T21:27:50.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:17.0.0-14857.g15219419.el8.x86_64 9/136 2022-09-07T21:27:50.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:17.0.0-14857.g15219419.el8.x86_64 10/136 2022-09-07T21:27:50.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:17.0.0-14857.g15219419.el8.x86_64 11/136 2022-09-07T21:27:50.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:17.0.0-14857.g15219419.el8.x86_64 12/136 2022-09-07T21:27:50.490 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephfs-mirror-2:17.0.0-14857.g15219419.el8.x86_6 13/136 2022-09-07T21:27:50.490 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:17.0.0-14857.g15219419.el8.x86 14/136 2022-09-07T21:27:50.490 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:17.0.0-14857.g15219419.el8.x86_64 15/136 2022-09-07T21:27:50.490 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephsqlite-2:17.0.0-14857.g15219419.el8.x86_6 16/136 2022-09-07T21:27:50.491 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:17.0.0-14857.g15219419.el8.x86_ 17/136 2022-09-07T21:27:50.491 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:17.0.0-14857.g15219419.el8.x8 18/136 2022-09-07T21:27:50.491 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:17.0.0-14857.g15219419.el8.x86_64 19/136 2022-09-07T21:27:50.491 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:17.0.0-14857.g15219419.e 20/136 2022-09-07T21:27:50.492 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:17.0.0-14857.g15219419.el8 21/136 2022-09-07T21:27:50.492 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:17.0.0-14857.g15219419.el8.x86_ 22/136 2022-09-07T21:27:50.492 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:17.0.0-14857.g15219419.el8.x86_6 23/136 2022-09-07T21:27:50.492 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:17.0.0-14857.g15219419.el8.x86_64 24/136 2022-09-07T21:27:50.492 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:17.0.0-14857.g15219419.el8.x86_64 25/136 2022-09-07T21:27:50.493 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:17.0.0-14857.g15219419.el8.x86_64 26/136 2022-09-07T21:27:50.493 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:17.0.0-14857.g15219419.el8.x86_64 27/136 2022-09-07T21:27:50.494 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:17.0.0-14857.g15219419.el8.x86_64 28/136 2022-09-07T21:27:50.494 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-14857.g15219419 29/136 2022-09-07T21:27:50.494 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-14857.g15219419.el8.no 30/136 2022-09-07T21:27:50.494 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-14857.g15219419.el8. 31/136 2022-09-07T21:27:50.495 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-14857.g15 32/136 2022-09-07T21:27:50.495 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-14857.g15219419.e 33/136 2022-09-07T21:27:50.495 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:17.0.0-14857.g15219419.el8.noarc 34/136 2022-09-07T21:27:50.496 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-14857.g15219419. 35/136 2022-09-07T21:27:50.496 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-volume-2:17.0.0-14857.g15219419.el8.noarch 36/136 2022-09-07T21:27:50.496 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:17.0.0-14857.g15219419.el8.noarch 37/136 2022-09-07T21:27:50.496 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephfs-top-2:17.0.0-14857.g15219419.el8.noarch 38/136 2022-09-07T21:27:50.497 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/136 2022-09-07T21:27:50.497 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/136 2022-09-07T21:27:50.497 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/136 2022-09-07T21:27:50.497 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/136 2022-09-07T21:27:50.497 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/136 2022-09-07T21:27:50.498 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/136 2022-09-07T21:27:50.498 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/136 2022-09-07T21:27:50.498 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/136 2022-09-07T21:27:50.498 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/136 2022-09-07T21:27:50.499 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/136 2022-09-07T21:27:50.499 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/136 2022-09-07T21:27:50.499 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/136 2022-09-07T21:27:50.499 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/136 2022-09-07T21:27:50.500 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/136 2022-09-07T21:27:50.500 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/136 2022-09-07T21:27:50.500 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/136 2022-09-07T21:27:50.500 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/136 2022-09-07T21:27:50.500 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/136 2022-09-07T21:27:50.501 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/136 2022-09-07T21:27:50.501 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/136 2022-09-07T21:27:50.501 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/136 2022-09-07T21:27:50.502 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/136 2022-09-07T21:27:50.502 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/136 2022-09-07T21:27:50.502 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/136 2022-09-07T21:27:50.502 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/136 2022-09-07T21:27:50.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/136 2022-09-07T21:27:50.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/136 2022-09-07T21:27:50.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/136 2022-09-07T21:27:50.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/136 2022-09-07T21:27:50.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/136 2022-09-07T21:27:50.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/136 2022-09-07T21:27:50.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/136 2022-09-07T21:27:50.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/136 2022-09-07T21:27:50.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 72/136 2022-09-07T21:27:50.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : bison-3.0.4-10.el8.x86_64 73/136 2022-09-07T21:27:50.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 74/136 2022-09-07T21:27:50.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 75/136 2022-09-07T21:27:50.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 76/136 2022-09-07T21:27:50.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/136 2022-09-07T21:27:50.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 78/136 2022-09-07T21:27:50.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 79/136 2022-09-07T21:27:50.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 80/136 2022-09-07T21:27:50.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 81/136 2022-09-07T21:27:50.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 82/136 2022-09-07T21:27:50.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 83/136 2022-09-07T21:27:50.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 84/136 2022-09-07T21:27:50.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 85/136 2022-09-07T21:27:50.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 86/136 2022-09-07T21:27:50.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 87/136 2022-09-07T21:27:50.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 88/136 2022-09-07T21:27:50.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flex-2.6.1-9.el8.x86_64 89/136 2022-09-07T21:27:50.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 90/136 2022-09-07T21:27:50.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : jq-1.6-3.el8.x86_64 91/136 2022-09-07T21:27:50.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 92/136 2022-09-07T21:27:50.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : automake-1.16.1-7.el8.noarch 93/136 2022-09-07T21:27:50.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 94/136 2022-09-07T21:27:50.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 95/136 2022-09-07T21:27:50.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 96/136 2022-09-07T21:27:50.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libpq-13.5-1.el8.x86_64 97/136 2022-09-07T21:27:50.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 98/136 2022-09-07T21:27:50.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 99/136 2022-09-07T21:27:50.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 100/136 2022-09-07T21:27:50.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 101/136 2022-09-07T21:27:50.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 102/136 2022-09-07T21:27:50.513 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 103/136 2022-09-07T21:27:50.513 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 104/136 2022-09-07T21:27:50.513 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 105/136 2022-09-07T21:27:50.514 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/136 2022-09-07T21:27:50.514 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 107/136 2022-09-07T21:27:50.514 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 108/136 2022-09-07T21:27:50.514 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 109/136 2022-09-07T21:27:50.515 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 110/136 2022-09-07T21:27:50.515 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 111/136 2022-09-07T21:27:50.515 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 112/136 2022-09-07T21:27:50.516 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 113/136 2022-09-07T21:27:50.516 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 114/136 2022-09-07T21:27:50.516 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 115/136 2022-09-07T21:27:50.516 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 116/136 2022-09-07T21:27:50.517 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 117/136 2022-09-07T21:27:50.517 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/136 2022-09-07T21:27:50.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 119/136 2022-09-07T21:27:50.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 120/136 2022-09-07T21:27:50.518 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-7.el8 121/136 2022-09-07T21:27:50.519 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 122/136 2022-09-07T21:27:50.519 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 123/136 2022-09-07T21:27:50.519 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 124/136 2022-09-07T21:27:50.519 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/136 2022-09-07T21:27:50.520 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 126/136 2022-09-07T21:27:50.520 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 127/136 2022-09-07T21:27:50.520 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 128/136 2022-09-07T21:27:50.520 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:17.0.0-14857.g15219419.el8.x86_64 129/136 2022-09-07T21:27:50.521 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/136 2022-09-07T21:27:50.521 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:17.0.0-14857.g15219419.el8.x86_64 131/136 2022-09-07T21:27:50.521 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/136 2022-09-07T21:27:50.521 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-1:1.1.1k-7.el8_6.x86_64 133/136 2022-09-07T21:27:50.522 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 134/136 2022-09-07T21:27:50.522 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-libs-1:1.1.1k-7.el8_6.x86_64 135/136 2022-09-07T21:27:51.957 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:17.0.0-14857.g15219419.el8.noarch 102/136 2022-09-07T21:27:52.172 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephadm-2:17.0.0-14857.g15219419.el8.noarch 102/136 2022-09-07T21:27:52.369 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-14857.g15219419. 103/136 2022-09-07T21:27:52.583 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-14857.g15219419 104/136 2022-09-07T21:27:52.619 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-07T21:27:52.620 INFO:teuthology.orchestra.run.smithi059.stdout:Installed products updated. 2022-09-07T21:27:52.785 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-14857.g15219419.el8.no 105/136 2022-09-07T21:27:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:27:52.847 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2022-09-07T21:27:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-1:1.1.1k-7.el8_6.x86_64 2022-09-07T21:27:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-libs-1:1.1.1k-7.el8_6.x86_64 2022-09-07T21:27:52.848 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2022-09-07T21:27:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-initscripts-updown-1:1.36.0-7.el8_6.noarch 2022-09-07T21:27:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: automake-1.16.1-7.el8.noarch 2022-09-07T21:27:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: bison-3.0.4-10.el8.x86_64 2022-09-07T21:27:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:52.853 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.853 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.853 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.854 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:52.854 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:52.854 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-mirror-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.854 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-top-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:27:52.855 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2022-09-07T21:27:52.855 INFO:teuthology.orchestra.run.smithi059.stdout: flex-2.6.1-9.el8.x86_64 2022-09-07T21:27:52.855 INFO:teuthology.orchestra.run.smithi059.stdout: fuse-2.9.7-15.el8.x86_64 2022-09-07T21:27:52.855 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-09-07T21:27:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: jq-1.6-3.el8.x86_64 2022-09-07T21:27:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2022-09-07T21:27:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: krb5-devel-1.18.2-14.el8.x86_64 2022-09-07T21:27:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-09-07T21:27:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2022-09-07T21:27:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2022-09-07T21:27:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.2-3.el8.x86_64 2022-09-07T21:27:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: libpq-13.5-1.el8.x86_64 2022-09-07T21:27:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2022-09-07T21:27:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-09-07T21:27:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-09-07T21:27:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-09-07T21:27:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: libsepol-devel-2.9-3.el8.x86_64 2022-09-07T21:27:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: libtool-2.4.6-25.el8.x86_64 2022-09-07T21:27:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.3.1-3.el8.x86_64 2022-09-07T21:27:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2022-09-07T21:27:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-09-07T21:27:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-09-07T21:27:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-09-07T21:27:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.15-3.el8.x86_64 2022-09-07T21:27:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-09-07T21:27:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2022-09-07T21:27:52.862 INFO:teuthology.orchestra.run.smithi059.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-09-07T21:27:52.862 INFO:teuthology.orchestra.run.smithi059.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-09-07T21:27:52.862 INFO:teuthology.orchestra.run.smithi059.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-09-07T21:27:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-07T21:27:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-07T21:27:52.864 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-07T21:27:52.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-09-07T21:27:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel-2.5.1-7.el8.noarch 2022-09-07T21:27:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-09-07T21:27:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-09-07T21:27:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-09-07T21:27:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-09-07T21:27:52.868 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-09-07T21:27:52.868 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-09-07T21:27:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-09-07T21:27:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-09-07T21:27:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-09-07T21:27:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-6.2-6.el8.noarch 2022-09-07T21:27:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-09-07T21:27:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-09-07T21:27:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-09-07T21:27:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-09-07T21:27:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-09-07T21:27:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-09-07T21:27:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-09-07T21:27:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.0.6-13.el8.noarch 2022-09-07T21:27:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-09-07T21:27:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-09-07T21:27:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-09-07T21:27:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-09-07T21:27:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-09-07T21:27:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-09-07T21:27:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-09-07T21:27:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-2.6-1.el8.noarch 2022-09-07T21:27:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-09-07T21:27:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-09-07T21:27:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-09-07T21:27:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-09-07T21:27:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser-2.14-14.el8.noarch 2022-09-07T21:27:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2017.2-9.el8.noarch 2022-09-07T21:27:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-09-07T21:27:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-09-07T21:27:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.4.1-12.el8.noarch 2022-09-07T21:27:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-4.8-1.el8.noarch 2022-09-07T21:27:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2022-09-07T21:27:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-09-07T21:27:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-09-07T21:27:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-09-07T21:27:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-09-07T21:27:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-09-07T21:27:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-09-07T21:27:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-09-07T21:27:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-09-07T21:27:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-09-07T21:27:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-09-07T21:27:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:27:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: socat-1.7.4.1-1.el8.x86_64 2022-09-07T21:27:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: thrift-0.13.0-2.el8.x86_64 2022-09-07T21:27:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: uuid-1.6.2-43.el8.x86_64 2022-09-07T21:27:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2022-09-07T21:27:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-09-07T21:27:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2022-09-07T21:27:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:27:52.882 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2022-09-07T21:27:53.040 DEBUG:teuthology.parallel:result is None 2022-09-07T21:27:53.089 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-14857.g15219419.el8.no 105/136 2022-09-07T21:27:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-14857.g15219419.el8. 106/136 2022-09-07T21:27:54.698 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-14857.g15219419.el8. 106/136 2022-09-07T21:27:54.909 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-14857.g15 107/136 2022-09-07T21:27:55.081 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-14857.g15 107/136 2022-09-07T21:27:55.560 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-14857.g15219419.e 108/136 2022-09-07T21:27:55.844 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-2:17.0.0-14857.g15219419.el8.x86_64 109/136 2022-09-07T21:27:55.945 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-2:17.0.0-14857.g15219419.el8.x86_64 109/136 2022-09-07T21:27:56.192 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-rook-2:17.0.0-14857.g15219419.el8.noarc 110/136 2022-09-07T21:27:56.248 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-14857.g15219419.el8.noarc 110/136 2022-09-07T21:27:58.258 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-2:17.0.0-14857.g15219419.el8.x86_64 111/136 2022-09-07T21:27:58.448 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-radosgw-2:17.0.0-14857.g15219419.el8.x86_64 112/136 2022-09-07T21:27:58.543 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-14857.g15219419.el8.x86_64 112/136 2022-09-07T21:27:58.701 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-14857.g1521 113/136 2022-09-07T21:27:58.807 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-14857.g1521 113/136 2022-09-07T21:27:59.034 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephfs-mirror-2:17.0.0-14857.g15219419.el8.x86_6 114/136 2022-09-07T21:27:59.525 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-14857.g15219419.el8.x86_6 114/136 2022-09-07T21:27:59.734 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-mirror-2:17.0.0-14857.g15219419.el8.x86_64 115/136 2022-09-07T21:28:06.017 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-mirror-2:17.0.0-14857.g15219419.el8.x86_64 115/136 2022-09-07T21:28:06.421 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-test-2:17.0.0-14857.g15219419.el8.x86_64 116/136 2022-09-07T21:28:07.949 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-fuse-2:17.0.0-14857.g15219419.el8.x86_64 117/136 2022-09-07T21:28:08.290 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/136 2022-09-07T21:28:08.586 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 119/136 2022-09-07T21:28:08.807 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libtool-2.4.6-25.el8.x86_64 120/136 2022-09-07T21:28:09.127 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 120/136 2022-09-07T21:28:09.363 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 121/136 2022-09-07T21:28:09.605 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs-devel-2:17.0.0-14857.g15219419.el8.x86 122/136 2022-09-07T21:28:09.816 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 123/136 2022-09-07T21:28:13.755 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-07T21:28:14.191 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-07T21:28:14.256 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-07T21:28:14.475 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephfs-top-2:17.0.0-14857.g15219419.el8.noarch 125/136 2022-09-07T21:28:14.778 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-fuse-2:17.0.0-14857.g15219419.el8.x86_64 126/136 2022-09-07T21:28:15.116 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-nbd-2:17.0.0-14857.g15219419.el8.x86_64 127/136 2022-09-07T21:28:15.463 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : openssl-1:1.1.1k-7.el8_6.x86_64 128/136 2022-09-07T21:28:15.765 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flex-2.6.1-9.el8.x86_64 129/136 2022-09-07T21:28:15.903 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 129/136 2022-09-07T21:28:16.206 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : bison-3.0.4-10.el8.x86_64 130/136 2022-09-07T21:28:16.260 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 130/136 2022-09-07T21:28:16.489 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-7.el8 131/136 2022-09-07T21:28:16.538 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-7.el8 131/136 2022-09-07T21:28:16.712 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 132/136 2022-09-07T21:28:16.935 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 133/136 2022-09-07T21:28:16.936 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 134/136 2022-09-07T21:28:17.232 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 134/136 2022-09-07T21:28:17.232 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 135/136 2022-09-07T21:28:18.878 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 135/136 2022-09-07T21:28:18.879 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-07T21:28:19.690 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-07T21:28:19.691 INFO:teuthology.orchestra.run.smithi047.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2022-09-07T21:28:19.691 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:28:20.544 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-2:17.0.0-14857.g15219419.el8.x86_64 1/136 2022-09-07T21:28:20.544 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-base-2:17.0.0-14857.g15219419.el8.x86_64 2/136 2022-09-07T21:28:20.545 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 3/136 2022-09-07T21:28:20.545 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-fuse-2:17.0.0-14857.g15219419.el8.x86_64 4/136 2022-09-07T21:28:20.545 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-14857.g1521 5/136 2022-09-07T21:28:20.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mds-2:17.0.0-14857.g15219419.el8.x86_64 6/136 2022-09-07T21:28:20.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-2:17.0.0-14857.g15219419.el8.x86_64 7/136 2022-09-07T21:28:20.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mon-2:17.0.0-14857.g15219419.el8.x86_64 8/136 2022-09-07T21:28:20.547 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-osd-2:17.0.0-14857.g15219419.el8.x86_64 9/136 2022-09-07T21:28:20.547 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-radosgw-2:17.0.0-14857.g15219419.el8.x86_64 10/136 2022-09-07T21:28:20.547 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-selinux-2:17.0.0-14857.g15219419.el8.x86_64 11/136 2022-09-07T21:28:20.547 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-test-2:17.0.0-14857.g15219419.el8.x86_64 12/136 2022-09-07T21:28:20.548 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephfs-mirror-2:17.0.0-14857.g15219419.el8.x86_6 13/136 2022-09-07T21:28:20.548 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs-devel-2:17.0.0-14857.g15219419.el8.x86 14/136 2022-09-07T21:28:20.548 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs2-2:17.0.0-14857.g15219419.el8.x86_64 15/136 2022-09-07T21:28:20.548 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephsqlite-2:17.0.0-14857.g15219419.el8.x86_6 16/136 2022-09-07T21:28:20.548 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados-devel-2:17.0.0-14857.g15219419.el8.x86_ 17/136 2022-09-07T21:28:20.549 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libradosstriper1-2:17.0.0-14857.g15219419.el8.x8 18/136 2022-09-07T21:28:20.549 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librgw2-2:17.0.0-14857.g15219419.el8.x86_64 19/136 2022-09-07T21:28:20.549 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-argparse-2:17.0.0-14857.g15219419.e 20/136 2022-09-07T21:28:20.549 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-common-2:17.0.0-14857.g15219419.el8 21/136 2022-09-07T21:28:20.550 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cephfs-2:17.0.0-14857.g15219419.el8.x86_ 22/136 2022-09-07T21:28:20.550 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rados-2:17.0.0-14857.g15219419.el8.x86_6 23/136 2022-09-07T21:28:20.550 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rbd-2:17.0.0-14857.g15219419.el8.x86_64 24/136 2022-09-07T21:28:20.550 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rgw-2:17.0.0-14857.g15219419.el8.x86_64 25/136 2022-09-07T21:28:20.550 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-fuse-2:17.0.0-14857.g15219419.el8.x86_64 26/136 2022-09-07T21:28:20.551 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-mirror-2:17.0.0-14857.g15219419.el8.x86_64 27/136 2022-09-07T21:28:20.551 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-nbd-2:17.0.0-14857.g15219419.el8.x86_64 28/136 2022-09-07T21:28:20.551 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-14857.g15219419 29/136 2022-09-07T21:28:20.551 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-14857.g15219419.el8.no 30/136 2022-09-07T21:28:20.552 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-14857.g15219419.el8. 31/136 2022-09-07T21:28:20.552 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-14857.g15 32/136 2022-09-07T21:28:20.565 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-14857.g15219419.e 33/136 2022-09-07T21:28:20.566 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-rook-2:17.0.0-14857.g15219419.el8.noarc 34/136 2022-09-07T21:28:20.566 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-14857.g15219419. 35/136 2022-09-07T21:28:20.566 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-volume-2:17.0.0-14857.g15219419.el8.noarch 36/136 2022-09-07T21:28:20.566 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephadm-2:17.0.0-14857.g15219419.el8.noarch 37/136 2022-09-07T21:28:20.567 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephfs-top-2:17.0.0-14857.g15219419.el8.noarch 38/136 2022-09-07T21:28:20.567 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/136 2022-09-07T21:28:20.567 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/136 2022-09-07T21:28:20.567 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/136 2022-09-07T21:28:20.568 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/136 2022-09-07T21:28:20.568 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/136 2022-09-07T21:28:20.568 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/136 2022-09-07T21:28:20.568 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/136 2022-09-07T21:28:20.568 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/136 2022-09-07T21:28:20.569 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/136 2022-09-07T21:28:20.569 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/136 2022-09-07T21:28:20.569 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/136 2022-09-07T21:28:20.569 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/136 2022-09-07T21:28:20.570 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/136 2022-09-07T21:28:20.570 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/136 2022-09-07T21:28:20.570 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/136 2022-09-07T21:28:20.570 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/136 2022-09-07T21:28:20.571 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/136 2022-09-07T21:28:20.571 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/136 2022-09-07T21:28:20.571 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/136 2022-09-07T21:28:20.571 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/136 2022-09-07T21:28:20.571 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/136 2022-09-07T21:28:20.572 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/136 2022-09-07T21:28:20.572 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/136 2022-09-07T21:28:20.572 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/136 2022-09-07T21:28:20.572 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/136 2022-09-07T21:28:20.573 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/136 2022-09-07T21:28:20.573 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/136 2022-09-07T21:28:20.573 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/136 2022-09-07T21:28:20.573 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/136 2022-09-07T21:28:20.574 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/136 2022-09-07T21:28:20.574 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/136 2022-09-07T21:28:20.574 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/136 2022-09-07T21:28:20.574 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/136 2022-09-07T21:28:20.575 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 72/136 2022-09-07T21:28:20.575 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 73/136 2022-09-07T21:28:20.575 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 74/136 2022-09-07T21:28:20.575 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/136 2022-09-07T21:28:20.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 76/136 2022-09-07T21:28:20.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 77/136 2022-09-07T21:28:20.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 78/136 2022-09-07T21:28:20.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 79/136 2022-09-07T21:28:20.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 80/136 2022-09-07T21:28:20.577 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 81/136 2022-09-07T21:28:20.577 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 82/136 2022-09-07T21:28:20.577 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 83/136 2022-09-07T21:28:20.577 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 84/136 2022-09-07T21:28:20.578 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/136 2022-09-07T21:28:20.578 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 86/136 2022-09-07T21:28:20.578 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/136 2022-09-07T21:28:20.578 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 88/136 2022-09-07T21:28:20.579 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 89/136 2022-09-07T21:28:20.579 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-7.el8 90/136 2022-09-07T21:28:20.579 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 91/136 2022-09-07T21:28:20.579 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 92/136 2022-09-07T21:28:20.580 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 93/136 2022-09-07T21:28:20.580 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/136 2022-09-07T21:28:20.580 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 95/136 2022-09-07T21:28:20.580 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 96/136 2022-09-07T21:28:20.581 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 97/136 2022-09-07T21:28:20.581 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 98/136 2022-09-07T21:28:20.581 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : bison-3.0.4-10.el8.x86_64 99/136 2022-09-07T21:28:20.581 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 100/136 2022-09-07T21:28:20.582 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 101/136 2022-09-07T21:28:20.582 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 102/136 2022-09-07T21:28:20.582 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 103/136 2022-09-07T21:28:20.582 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 104/136 2022-09-07T21:28:20.583 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 105/136 2022-09-07T21:28:20.583 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 106/136 2022-09-07T21:28:20.583 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 107/136 2022-09-07T21:28:20.583 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 108/136 2022-09-07T21:28:20.583 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 109/136 2022-09-07T21:28:20.584 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 110/136 2022-09-07T21:28:20.584 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 111/136 2022-09-07T21:28:20.584 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 112/136 2022-09-07T21:28:20.584 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 113/136 2022-09-07T21:28:20.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 114/136 2022-09-07T21:28:20.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flex-2.6.1-9.el8.x86_64 115/136 2022-09-07T21:28:20.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 116/136 2022-09-07T21:28:20.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : jq-1.6-3.el8.x86_64 117/136 2022-09-07T21:28:20.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 118/136 2022-09-07T21:28:20.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : automake-1.16.1-7.el8.noarch 119/136 2022-09-07T21:28:20.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 120/136 2022-09-07T21:28:20.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 121/136 2022-09-07T21:28:20.587 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 122/136 2022-09-07T21:28:20.587 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libpq-13.5-1.el8.x86_64 123/136 2022-09-07T21:28:20.587 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 124/136 2022-09-07T21:28:20.587 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 125/136 2022-09-07T21:28:20.588 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 126/136 2022-09-07T21:28:20.588 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 127/136 2022-09-07T21:28:20.588 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 128/136 2022-09-07T21:28:20.588 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-2:17.0.0-14857.g15219419.el8.x86_64 129/136 2022-09-07T21:28:20.588 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/136 2022-09-07T21:28:20.589 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-2:17.0.0-14857.g15219419.el8.x86_64 131/136 2022-09-07T21:28:20.589 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/136 2022-09-07T21:28:20.589 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openssl-1:1.1.1k-7.el8_6.x86_64 133/136 2022-09-07T21:28:20.589 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 134/136 2022-09-07T21:28:20.590 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openssl-libs-1:1.1.1k-7.el8_6.x86_64 135/136 2022-09-07T21:28:22.670 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-07T21:28:22.671 INFO:teuthology.orchestra.run.smithi047.stdout:Installed products updated. 2022-09-07T21:28:22.911 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:28:22.912 INFO:teuthology.orchestra.run.smithi047.stdout:Upgraded: 2022-09-07T21:28:22.912 INFO:teuthology.orchestra.run.smithi047.stdout: librados2-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.912 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.912 INFO:teuthology.orchestra.run.smithi047.stdout: openssl-1:1.1.1k-7.el8_6.x86_64 2022-09-07T21:28:22.913 INFO:teuthology.orchestra.run.smithi047.stdout: openssl-libs-1:1.1.1k-7.el8_6.x86_64 2022-09-07T21:28:22.913 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2022-09-07T21:28:22.913 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-initscripts-updown-1:1.36.0-7.el8_6.noarch 2022-09-07T21:28:22.913 INFO:teuthology.orchestra.run.smithi047.stdout: automake-1.16.1-7.el8.noarch 2022-09-07T21:28:22.914 INFO:teuthology.orchestra.run.smithi047.stdout: bison-3.0.4-10.el8.x86_64 2022-09-07T21:28:22.914 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.914 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.914 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.915 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.915 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:28:22.915 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.915 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.916 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.916 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:28:22.916 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:28:22.916 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:28:22.916 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:28:22.917 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:28:22.917 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.917 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.917 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:28:22.918 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.918 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.918 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.918 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-volume-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:28:22.919 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:28:22.919 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-mirror-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.919 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-top-2:17.0.0-14857.g15219419.el8.noarch 2022-09-07T21:28:22.919 INFO:teuthology.orchestra.run.smithi047.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2022-09-07T21:28:22.920 INFO:teuthology.orchestra.run.smithi047.stdout: flex-2.6.1-9.el8.x86_64 2022-09-07T21:28:22.920 INFO:teuthology.orchestra.run.smithi047.stdout: fuse-2.9.7-15.el8.x86_64 2022-09-07T21:28:22.920 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-09-07T21:28:22.920 INFO:teuthology.orchestra.run.smithi047.stdout: jq-1.6-3.el8.x86_64 2022-09-07T21:28:22.920 INFO:teuthology.orchestra.run.smithi047.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2022-09-07T21:28:22.921 INFO:teuthology.orchestra.run.smithi047.stdout: krb5-devel-1.18.2-14.el8.x86_64 2022-09-07T21:28:22.921 INFO:teuthology.orchestra.run.smithi047.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-09-07T21:28:22.921 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.922 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.922 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.922 INFO:teuthology.orchestra.run.smithi047.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2022-09-07T21:28:22.922 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2022-09-07T21:28:22.923 INFO:teuthology.orchestra.run.smithi047.stdout: liboath-2.6.2-3.el8.x86_64 2022-09-07T21:28:22.923 INFO:teuthology.orchestra.run.smithi047.stdout: libpq-13.5-1.el8.x86_64 2022-09-07T21:28:22.923 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2022-09-07T21:28:22.923 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-09-07T21:28:22.924 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.924 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.924 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-09-07T21:28:22.924 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.924 INFO:teuthology.orchestra.run.smithi047.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-09-07T21:28:22.925 INFO:teuthology.orchestra.run.smithi047.stdout: libsepol-devel-2.9-3.el8.x86_64 2022-09-07T21:28:22.925 INFO:teuthology.orchestra.run.smithi047.stdout: libtool-2.4.6-25.el8.x86_64 2022-09-07T21:28:22.925 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind-1.3.1-3.el8.x86_64 2022-09-07T21:28:22.925 INFO:teuthology.orchestra.run.smithi047.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2022-09-07T21:28:22.926 INFO:teuthology.orchestra.run.smithi047.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-09-07T21:28:22.926 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-09-07T21:28:22.926 INFO:teuthology.orchestra.run.smithi047.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-09-07T21:28:22.926 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-0.3.15-3.el8.x86_64 2022-09-07T21:28:22.927 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-09-07T21:28:22.927 INFO:teuthology.orchestra.run.smithi047.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2022-09-07T21:28:22.927 INFO:teuthology.orchestra.run.smithi047.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-09-07T21:28:22.927 INFO:teuthology.orchestra.run.smithi047.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-09-07T21:28:22.927 INFO:teuthology.orchestra.run.smithi047.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-09-07T21:28:22.928 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-07T21:28:22.928 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-07T21:28:22.928 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-07T21:28:22.928 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-09-07T21:28:22.929 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel-2.5.1-7.el8.noarch 2022-09-07T21:28:22.929 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-09-07T21:28:22.929 INFO:teuthology.orchestra.run.smithi047.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-09-07T21:28:22.929 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-09-07T21:28:22.929 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.930 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.930 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.930 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-09-07T21:28:22.930 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-09-07T21:28:22.931 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-09-07T21:28:22.931 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-09-07T21:28:22.931 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-09-07T21:28:22.931 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-09-07T21:28:22.931 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-6.2-6.el8.noarch 2022-09-07T21:28:22.932 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-09-07T21:28:22.932 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-09-07T21:28:22.932 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-09-07T21:28:22.932 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-09-07T21:28:22.933 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-09-07T21:28:22.933 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-09-07T21:28:22.933 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-09-07T21:28:22.933 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako-1.0.6-13.el8.noarch 2022-09-07T21:28:22.934 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-09-07T21:28:22.934 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-09-07T21:28:22.934 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-09-07T21:28:22.934 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-09-07T21:28:22.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-09-07T21:28:22.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-09-07T21:28:22.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-09-07T21:28:22.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend-2.6-1.el8.noarch 2022-09-07T21:28:22.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-09-07T21:28:22.936 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-09-07T21:28:22.936 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-09-07T21:28:22.936 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-09-07T21:28:22.936 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser-2.14-14.el8.noarch 2022-09-07T21:28:22.937 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz-2017.2-9.el8.noarch 2022-09-07T21:28:22.937 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.937 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.937 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-09-07T21:28:22.937 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-09-07T21:28:22.938 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.938 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes-2.4.1-12.el8.noarch 2022-09-07T21:28:22.938 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa-4.8-1.el8.noarch 2022-09-07T21:28:22.938 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2022-09-07T21:28:22.939 INFO:teuthology.orchestra.run.smithi047.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-09-07T21:28:22.939 INFO:teuthology.orchestra.run.smithi047.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-09-07T21:28:22.939 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-09-07T21:28:22.939 INFO:teuthology.orchestra.run.smithi047.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-09-07T21:28:22.940 INFO:teuthology.orchestra.run.smithi047.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-09-07T21:28:22.940 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-09-07T21:28:22.940 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-09-07T21:28:22.940 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-09-07T21:28:22.940 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-09-07T21:28:22.941 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-09-07T21:28:22.941 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.941 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.941 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd-2:17.0.0-14857.g15219419.el8.x86_64 2022-09-07T21:28:22.942 INFO:teuthology.orchestra.run.smithi047.stdout: socat-1.7.4.1-1.el8.x86_64 2022-09-07T21:28:22.942 INFO:teuthology.orchestra.run.smithi047.stdout: thrift-0.13.0-2.el8.x86_64 2022-09-07T21:28:22.942 INFO:teuthology.orchestra.run.smithi047.stdout: uuid-1.6.2-43.el8.x86_64 2022-09-07T21:28:22.942 INFO:teuthology.orchestra.run.smithi047.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2022-09-07T21:28:22.943 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-09-07T21:28:22.943 INFO:teuthology.orchestra.run.smithi047.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2022-09-07T21:28:22.943 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:28:22.944 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2022-09-07T21:28:23.338 DEBUG:teuthology.parallel:result is None 2022-09-07T21:28:23.339 DEBUG:teuthology.orchestra.run.smithi047:> sudo lsb_release -is 2022-09-07T21:28:23.519 INFO:teuthology.orchestra.run.smithi047.stdout:RedHatEnterprise 2022-09-07T21:28:23.524 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-07T21:28:23.525 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=152194198413033811b31149de909a7ec5852a7f 2022-09-07T21:28:23.686 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-09-07T21:28:23.758 INFO:teuthology.orchestra.run.smithi047.stdout:17.0.0-14857.g15219419.el8 2022-09-07T21:28:23.759 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-14857.g15219419.el8 2022-09-07T21:28:23.760 INFO:teuthology.task.install:The correct ceph version 17.0.0-14857.g15219419 is installed. 2022-09-07T21:28:23.760 DEBUG:teuthology.orchestra.run.smithi055:> sudo lsb_release -is 2022-09-07T21:28:23.827 INFO:teuthology.orchestra.run.smithi055.stdout:RedHatEnterprise 2022-09-07T21:28:23.829 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-07T21:28:23.829 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=152194198413033811b31149de909a7ec5852a7f 2022-09-07T21:28:23.991 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-09-07T21:28:24.019 INFO:teuthology.orchestra.run.smithi055.stdout:17.0.0-14857.g15219419.el8 2022-09-07T21:28:24.020 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-14857.g15219419.el8 2022-09-07T21:28:24.020 INFO:teuthology.task.install:The correct ceph version 17.0.0-14857.g15219419 is installed. 2022-09-07T21:28:24.021 DEBUG:teuthology.orchestra.run.smithi059:> sudo lsb_release -is 2022-09-07T21:28:24.089 INFO:teuthology.orchestra.run.smithi059.stdout:RedHatEnterprise 2022-09-07T21:28:24.091 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-07T21:28:24.092 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=152194198413033811b31149de909a7ec5852a7f 2022-09-07T21:28:24.240 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-09-07T21:28:24.267 INFO:teuthology.orchestra.run.smithi059.stdout:17.0.0-14857.g15219419.el8 2022-09-07T21:28:24.267 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-14857.g15219419.el8 2022-09-07T21:28:24.268 INFO:teuthology.task.install:The correct ceph version 17.0.0-14857.g15219419 is installed. 2022-09-07T21:28:24.268 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-09-07T21:28:24.269 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:28:24.269 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-09-07T21:28:24.363 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2022-09-07T21:28:24.364 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-09-07T21:28:24.398 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-09-07T21:28:24.399 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-09-07T21:28:24.432 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-09-07T21:28:24.432 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:28:24.433 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/daemon-helper 2022-09-07T21:28:24.524 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-09-07T21:28:24.656 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2022-09-07T21:28:24.657 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/daemon-helper 2022-09-07T21:28:24.691 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-09-07T21:28:24.763 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-09-07T21:28:24.764 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2022-09-07T21:28:24.798 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-09-07T21:28:24.869 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-09-07T21:28:24.869 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:28:24.870 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/adjust-ulimits 2022-09-07T21:28:24.967 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-09-07T21:28:25.100 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2022-09-07T21:28:25.100 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/adjust-ulimits 2022-09-07T21:28:25.134 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-09-07T21:28:25.207 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-09-07T21:28:25.208 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2022-09-07T21:28:25.242 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-09-07T21:28:25.315 INFO:teuthology.run_tasks:Running task cephadm... 2022-09-07T21:28:25.408 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'SLOW_OPS', 'slow request'], 'sha1': '152194198413033811b31149de909a7ec5852a7f', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2022-09-07T21:28:25.409 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f 2022-09-07T21:28:25.410 INFO:tasks.cephadm:Cluster fsid is 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:28:25.410 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2022-09-07T21:28:25.411 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.47', 'mon.b': '172.21.15.55', 'mon.c': '172.21.15.59'} 2022-09-07T21:28:25.411 INFO:tasks.cephadm:First mon is mon.a on smithi047 2022-09-07T21:28:25.411 INFO:tasks.cephadm:First mgr is x 2022-09-07T21:28:25.412 INFO:tasks.cephadm:Normalizing hostnames... 2022-09-07T21:28:25.413 DEBUG:teuthology.orchestra.run.smithi047:> sudo hostname $(hostname -s) 2022-09-07T21:28:25.511 DEBUG:teuthology.orchestra.run.smithi055:> sudo hostname $(hostname -s) 2022-09-07T21:28:25.545 DEBUG:teuthology.orchestra.run.smithi059:> sudo hostname $(hostname -s) 2022-09-07T21:28:25.579 INFO:tasks.cephadm:Downloading cephadm (repo git://git.ceph.com/ceph-ci.git ref 152194198413033811b31149de909a7ec5852a7f)... 2022-09-07T21:28:25.580 DEBUG:teuthology.orchestra.run.smithi047:> git archive --remote=git://git.ceph.com/ceph-ci.git 152194198413033811b31149de909a7ec5852a7f src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2022-09-07T21:28:25.694 DEBUG:teuthology.orchestra.run.smithi055:> git archive --remote=git://git.ceph.com/ceph-ci.git 152194198413033811b31149de909a7ec5852a7f src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2022-09-07T21:28:25.759 DEBUG:teuthology.orchestra.run.smithi059:> git archive --remote=git://git.ceph.com/ceph-ci.git 152194198413033811b31149de909a7ec5852a7f src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2022-09-07T21:28:25.823 DEBUG:teuthology.orchestra.run.smithi047:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-09-07T21:28:25.889 DEBUG:teuthology.orchestra.run.smithi055:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-09-07T21:28:25.911 DEBUG:teuthology.orchestra.run.smithi059:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-09-07T21:28:25.955 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f on all hosts... 2022-09-07T21:28:25.956 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f pull 2022-09-07T21:28:25.963 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f pull 2022-09-07T21:28:25.966 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f pull 2022-09-07T21:28:26.871 INFO:teuthology.orchestra.run.smithi055.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f... 2022-09-07T21:28:26.966 INFO:teuthology.orchestra.run.smithi059.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f... 2022-09-07T21:28:31.466 INFO:teuthology.orchestra.run.smithi047.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f... 2022-09-07T21:29:15.046 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2022-09-07T21:29:15.047 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph_version": "ceph version 17.0.0-14857-g15219419 (152194198413033811b31149de909a7ec5852a7f) quincy (dev)", 2022-09-07T21:29:15.047 INFO:teuthology.orchestra.run.smithi047.stdout: "image_id": "6405dbf5acb7a3494d2b44504da4bc6ac177ba034fa1511424e72538f0aa6923", 2022-09-07T21:29:15.047 INFO:teuthology.orchestra.run.smithi047.stdout: "repo_digests": [ 2022-09-07T21:29:15.048 INFO:teuthology.orchestra.run.smithi047.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:890f81b462827c9c5f6424b47803267c6c59d344d092a29bdfcf3547a2647a6f" 2022-09-07T21:29:15.048 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2022-09-07T21:29:15.049 INFO:teuthology.orchestra.run.smithi047.stdout:} 2022-09-07T21:29:20.940 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2022-09-07T21:29:20.941 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 17.0.0-14857-g15219419 (152194198413033811b31149de909a7ec5852a7f) quincy (dev)", 2022-09-07T21:29:20.941 INFO:teuthology.orchestra.run.smithi059.stdout: "image_id": "6405dbf5acb7a3494d2b44504da4bc6ac177ba034fa1511424e72538f0aa6923", 2022-09-07T21:29:20.941 INFO:teuthology.orchestra.run.smithi059.stdout: "repo_digests": [ 2022-09-07T21:29:20.941 INFO:teuthology.orchestra.run.smithi059.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:890f81b462827c9c5f6424b47803267c6c59d344d092a29bdfcf3547a2647a6f" 2022-09-07T21:29:20.942 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2022-09-07T21:29:20.942 INFO:teuthology.orchestra.run.smithi059.stdout:} 2022-09-07T21:29:21.247 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2022-09-07T21:29:21.247 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_version": "ceph version 17.0.0-14857-g15219419 (152194198413033811b31149de909a7ec5852a7f) quincy (dev)", 2022-09-07T21:29:21.247 INFO:teuthology.orchestra.run.smithi055.stdout: "image_id": "6405dbf5acb7a3494d2b44504da4bc6ac177ba034fa1511424e72538f0aa6923", 2022-09-07T21:29:21.247 INFO:teuthology.orchestra.run.smithi055.stdout: "repo_digests": [ 2022-09-07T21:29:21.248 INFO:teuthology.orchestra.run.smithi055.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:890f81b462827c9c5f6424b47803267c6c59d344d092a29bdfcf3547a2647a6f" 2022-09-07T21:29:21.248 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2022-09-07T21:29:21.248 INFO:teuthology.orchestra.run.smithi055.stdout:} 2022-09-07T21:29:21.267 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph 2022-09-07T21:29:21.368 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /etc/ceph 2022-09-07T21:29:21.401 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph 2022-09-07T21:29:21.442 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 777 /etc/ceph 2022-09-07T21:29:21.534 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 777 /etc/ceph 2022-09-07T21:29:21.565 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /etc/ceph 2022-09-07T21:29:21.600 INFO:tasks.cephadm:Writing seed config... 2022-09-07T21:29:21.601 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2022-09-07T21:29:21.602 INFO:tasks.cephadm: override: [client] debug client = 20 2022-09-07T21:29:21.602 INFO:tasks.cephadm: override: [client] debug ms = 1 2022-09-07T21:29:21.602 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2022-09-07T21:29:21.603 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2022-09-07T21:29:21.603 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2022-09-07T21:29:21.603 INFO:tasks.cephadm: override: [mds] debug mds = 20 2022-09-07T21:29:21.604 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2022-09-07T21:29:21.604 INFO:tasks.cephadm: override: [mds] debug ms = 1 2022-09-07T21:29:21.604 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2022-09-07T21:29:21.605 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2022-09-07T21:29:21.605 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2022-09-07T21:29:21.605 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2022-09-07T21:29:21.606 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2022-09-07T21:29:21.606 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2022-09-07T21:29:21.607 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2022-09-07T21:29:21.607 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2022-09-07T21:29:21.607 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2022-09-07T21:29:21.608 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2022-09-07T21:29:21.608 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2022-09-07T21:29:21.608 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2022-09-07T21:29:21.609 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2022-09-07T21:29:21.609 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2022-09-07T21:29:21.610 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2022-09-07T21:29:21.610 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2022-09-07T21:29:21.611 INFO:tasks.cephadm: override: [mon] debug mon = 20 2022-09-07T21:29:21.612 INFO:tasks.cephadm: override: [mon] debug ms = 1 2022-09-07T21:29:21.612 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2022-09-07T21:29:21.613 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2022-09-07T21:29:21.613 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2022-09-07T21:29:21.614 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2022-09-07T21:29:21.615 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2022-09-07T21:29:21.615 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2022-09-07T21:29:21.616 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2022-09-07T21:29:21.616 INFO:tasks.cephadm: override: [osd] debug ms = 1 2022-09-07T21:29:21.617 INFO:tasks.cephadm: override: [osd] debug osd = 20 2022-09-07T21:29:21.617 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2022-09-07T21:29:21.617 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2022-09-07T21:29:21.618 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2022-09-07T21:29:21.618 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2022-09-07T21:29:21.618 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2022-09-07T21:29:21.619 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2022-09-07T21:29:21.619 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2022-09-07T21:29:21.619 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2022-09-07T21:29:21.620 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2022-09-07T21:29:21.620 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2022-09-07T21:29:21.620 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2022-09-07T21:29:21.621 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:29:21.622 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2022-09-07T21:29:21.680 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 01b8396e-2ef4-11ed-8431-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2022-09-07T21:29:21.680 DEBUG:teuthology.orchestra.run.smithi047:mon.a> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mon.a.service 2022-09-07T21:29:21.725 DEBUG:teuthology.orchestra.run.smithi047:mgr.x> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mgr.x.service 2022-09-07T21:29:21.771 INFO:tasks.cephadm:Bootstrapping... 2022-09-07T21:29:21.772 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f -v bootstrap --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.47 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2022-09-07T21:29:21.811 INFO:journalctl@ceph.mon.a.smithi047.stdout:-- Logs begin at Wed 2022-09-07 21:11:20 UTC. -- 2022-09-07T21:29:21.858 INFO:journalctl@ceph.mgr.x.smithi047.stdout:-- Logs begin at Wed 2022-09-07 21:11:20 UTC. -- 2022-09-07T21:29:22.106 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2022-09-07T21:29:22.107 INFO:teuthology.orchestra.run.smithi047.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f', '-v', 'bootstrap', '--fsid', '01b8396e-2ef4-11ed-8431-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.47', '--skip-admin-label'] 2022-09-07T21:29:22.301 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stdout 4.1.1 2022-09-07T21:29:22.302 INFO:teuthology.orchestra.run.smithi047.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2022-09-07T21:29:22.302 INFO:teuthology.orchestra.run.smithi047.stdout:Verifying podman|docker is present... 2022-09-07T21:29:22.494 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stdout 4.1.1 2022-09-07T21:29:22.494 INFO:teuthology.orchestra.run.smithi047.stdout:Verifying lvm2 is present... 2022-09-07T21:29:22.495 INFO:teuthology.orchestra.run.smithi047.stdout:Verifying time synchronization is in place... 2022-09-07T21:29:22.513 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2022-09-07T21:29:22.513 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2022-09-07T21:29:22.530 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2022-09-07T21:29:22.531 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stdout inactive 2022-09-07T21:29:22.548 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stdout enabled 2022-09-07T21:29:22.566 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stdout active 2022-09-07T21:29:22.566 INFO:teuthology.orchestra.run.smithi047.stdout:Unit chronyd.service is enabled and running 2022-09-07T21:29:22.567 INFO:teuthology.orchestra.run.smithi047.stdout:Repeating the final host check... 2022-09-07T21:29:22.758 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stdout 4.1.1 2022-09-07T21:29:22.758 INFO:teuthology.orchestra.run.smithi047.stdout:podman (/bin/podman) version 4.1.1 is present 2022-09-07T21:29:22.759 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl is present 2022-09-07T21:29:22.759 INFO:teuthology.orchestra.run.smithi047.stdout:lvcreate is present 2022-09-07T21:29:22.776 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2022-09-07T21:29:22.777 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2022-09-07T21:29:22.793 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2022-09-07T21:29:22.794 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stdout inactive 2022-09-07T21:29:22.811 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stdout enabled 2022-09-07T21:29:22.830 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stdout active 2022-09-07T21:29:22.830 INFO:teuthology.orchestra.run.smithi047.stdout:Unit chronyd.service is enabled and running 2022-09-07T21:29:22.831 INFO:teuthology.orchestra.run.smithi047.stdout:Host looks OK 2022-09-07T21:29:22.831 INFO:teuthology.orchestra.run.smithi047.stdout:Cluster fsid: 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:29:22.831 INFO:teuthology.orchestra.run.smithi047.stdout:Acquiring lock 140158608354496 on /run/cephadm/01b8396e-2ef4-11ed-8431-001a4aab830c.lock 2022-09-07T21:29:22.832 INFO:teuthology.orchestra.run.smithi047.stdout:Lock 140158608354496 acquired on /run/cephadm/01b8396e-2ef4-11ed-8431-001a4aab830c.lock 2022-09-07T21:29:22.832 INFO:teuthology.orchestra.run.smithi047.stdout:Verifying IP 172.21.15.47 port 3300 ... 2022-09-07T21:29:22.833 INFO:teuthology.orchestra.run.smithi047.stdout:Verifying IP 172.21.15.47 port 6789 ... 2022-09-07T21:29:22.833 INFO:teuthology.orchestra.run.smithi047.stdout:Base mon IP(s) is [172.21.15.47:3300, 172.21.15.47:6789], mon addrv is [v2:172.21.15.47:3300,v1:172.21.15.47:6789] 2022-09-07T21:29:22.841 INFO:teuthology.orchestra.run.smithi047.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2022-09-07T21:29:22.841 INFO:teuthology.orchestra.run.smithi047.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.47 metric 100 2022-09-07T21:29:22.851 INFO:teuthology.orchestra.run.smithi047.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2022-09-07T21:29:22.851 INFO:teuthology.orchestra.run.smithi047.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2022-09-07T21:29:22.851 INFO:teuthology.orchestra.run.smithi047.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1778sec hoplimit 64 pref medium 2022-09-07T21:29:22.860 INFO:teuthology.orchestra.run.smithi047.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2022-09-07T21:29:22.861 INFO:teuthology.orchestra.run.smithi047.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2022-09-07T21:29:22.861 INFO:teuthology.orchestra.run.smithi047.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2022-09-07T21:29:22.861 INFO:teuthology.orchestra.run.smithi047.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2022-09-07T21:29:22.862 INFO:teuthology.orchestra.run.smithi047.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:148e/64 scope link 2022-09-07T21:29:22.862 INFO:teuthology.orchestra.run.smithi047.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2022-09-07T21:29:22.863 INFO:teuthology.orchestra.run.smithi047.stdout:Mon IP `172.21.15.47` is in CIDR network `172.21.0.0/20` 2022-09-07T21:29:22.863 INFO:teuthology.orchestra.run.smithi047.stdout:Mon IP `172.21.15.47` is in CIDR network `172.21.0.0/20` 2022-09-07T21:29:22.864 INFO:teuthology.orchestra.run.smithi047.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2022-09-07T21:29:22.864 INFO:teuthology.orchestra.run.smithi047.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2022-09-07T21:29:22.864 INFO:teuthology.orchestra.run.smithi047.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f... 2022-09-07T21:29:23.360 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stdout 6405dbf5acb7a3494d2b44504da4bc6ac177ba034fa1511424e72538f0aa6923 2022-09-07T21:29:23.360 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f... 2022-09-07T21:29:23.360 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stderr Getting image source signatures 2022-09-07T21:29:23.361 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stderr Copying blob sha256:df305547f7faebabb0d38c7e1906a4fd8264edcd18fcce61af66675bb804ad0f 2022-09-07T21:29:23.361 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stderr Copying blob sha256:99a39ffc60545d3551e6b2d1fa8868206fd1c0517c86346f930cd3902f2e17c6 2022-09-07T21:29:23.361 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stderr Copying blob sha256:5d1b43028e8e0373631944ff9016c99bdf03efc12705b9a423bc758fccf55806 2022-09-07T21:29:23.361 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stderr Copying blob sha256:bb868b3138d9139f452065e8d61f4c081341a6caa61b0ac6f12814f3a0dc362e 2022-09-07T21:29:23.361 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stderr Copying blob sha256:2252a7414fd07c781b1ed7bd7af688bb98409db8ecbad4aec3bacaa72fdc5aa7 2022-09-07T21:29:23.362 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stderr Copying config sha256:6405dbf5acb7a3494d2b44504da4bc6ac177ba034fa1511424e72538f0aa6923 2022-09-07T21:29:23.362 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stderr Writing manifest to image destination 2022-09-07T21:29:23.362 INFO:teuthology.orchestra.run.smithi047.stdout:/bin/podman: stderr Storing signatures 2022-09-07T21:29:24.559 INFO:teuthology.orchestra.run.smithi047.stdout:ceph: stdout ceph version 17.0.0-14857-g15219419 (152194198413033811b31149de909a7ec5852a7f) quincy (dev) 2022-09-07T21:29:24.559 INFO:teuthology.orchestra.run.smithi047.stdout:Ceph version: ceph version 17.0.0-14857-g15219419 (152194198413033811b31149de909a7ec5852a7f) quincy (dev) 2022-09-07T21:29:24.560 INFO:teuthology.orchestra.run.smithi047.stdout:Extracting ceph user uid/gid from container image... 2022-09-07T21:29:25.457 INFO:teuthology.orchestra.run.smithi047.stdout:stat: stdout 167 167 2022-09-07T21:29:25.457 INFO:teuthology.orchestra.run.smithi047.stdout:Creating initial keys... 2022-09-07T21:29:26.461 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph-authtool: stdout AQA1DRljQ/V4NxAAFqAEencfSw6QC/AuBATheA== 2022-09-07T21:29:27.461 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph-authtool: stdout AQA2DRljmc31ORAAK9fwap5fVxH4cK9Pcp/SNw== 2022-09-07T21:29:28.762 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph-authtool: stdout AQA3DRljfy93NhAA/FqPEaeky0yybQgH1NvFRg== 2022-09-07T21:29:28.763 INFO:teuthology.orchestra.run.smithi047.stdout:Creating initial monmap... 2022-09-07T21:29:29.764 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2022-09-07T21:29:29.764 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2022-09-07T21:29:29.764 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:29:29.765 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2022-09-07T21:29:29.765 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool for a [v2:172.21.15.47:3300,v1:172.21.15.47:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2022-09-07T21:29:29.765 INFO:teuthology.orchestra.run.smithi047.stdout:setting min_mon_release = octopus 2022-09-07T21:29:29.765 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/monmaptool: set fsid to 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:29:29.766 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2022-09-07T21:29:29.766 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:29:29.766 INFO:teuthology.orchestra.run.smithi047.stdout:Creating mon... 2022-09-07T21:29:30.763 INFO:teuthology.orchestra.run.smithi047.stdout:create mon.a on 2022-09-07T21:29:31.052 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2022-09-07T21:29:31.329 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2022-09-07T21:29:31.615 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-01b8396e-2ef4-11ed-8431-001a4aab830c.target → /etc/systemd/system/ceph-01b8396e-2ef4-11ed-8431-001a4aab830c.target. 2022-09-07T21:29:31.615 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-01b8396e-2ef4-11ed-8431-001a4aab830c.target → /etc/systemd/system/ceph-01b8396e-2ef4-11ed-8431-001a4aab830c.target. 2022-09-07T21:29:32.054 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mon.a 2022-09-07T21:29:32.055 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stderr Failed to reset failed state of unit ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mon.a.service: Unit ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mon.a.service not loaded. 2022-09-07T21:29:32.333 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-01b8396e-2ef4-11ed-8431-001a4aab830c.target.wants/ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@.service. 2022-09-07T21:29:32.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:32 smithi047 systemd[1]: Starting Ceph mon.a for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:29:33.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:32 smithi047 podman[89563]: 2022-09-07T21:29:34.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:33 smithi047 ceph-mon[89603]: mkfs 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:29:34.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:33 smithi047 ceph-mon[89603]: mon.a is new leader, mons a in quorum (ranks 0) 2022-09-07T21:29:35.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:34 smithi047 ceph-mon[89603]: mon.a is new leader, mons a in quorum (ranks 0) 2022-09-07T21:29:35.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:34 smithi047 ceph-mon[89603]: monmap e1: 1 mons at {a=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0]} 2022-09-07T21:29:35.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:34 smithi047 ceph-mon[89603]: fsmap 2022-09-07T21:29:35.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:34 smithi047 ceph-mon[89603]: osdmap e1: 0 total, 0 up, 0 in 2022-09-07T21:29:35.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:34 smithi047 ceph-mon[89603]: mgrmap e1: no daemons active 2022-09-07T21:29:35.376 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2022-09-07T21:29:35.376 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stdout disabled 2022-09-07T21:29:35.400 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2022-09-07T21:29:35.401 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stdout inactive 2022-09-07T21:29:35.401 INFO:teuthology.orchestra.run.smithi047.stdout:firewalld.service is not enabled 2022-09-07T21:29:35.401 INFO:teuthology.orchestra.run.smithi047.stdout:Not possible to enable service . firewalld.service is not available 2022-09-07T21:29:35.402 INFO:teuthology.orchestra.run.smithi047.stdout:Waiting for mon to start... 2022-09-07T21:29:35.403 INFO:teuthology.orchestra.run.smithi047.stdout:Waiting for mon... 2022-09-07T21:29:35.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:35 smithi047 bash[89563]: e273355988eb1b7cd344eedec5d99b616b86c91d6ffdc515759df294f847c78c 2022-09-07T21:29:35.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:35 smithi047 systemd[1]: Started Ceph mon.a for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:29:42.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:42 smithi047 ceph-mon[89603]: from='client.? 172.21.15.47:0/3022474574' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2022-09-07T21:29:42.938 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout cluster: 2022-09-07T21:29:42.945 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout id: 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:29:42.946 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2022-09-07T21:29:42.946 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 2022-09-07T21:29:42.946 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout services: 2022-09-07T21:29:42.946 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 8s) 2022-09-07T21:29:42.946 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2022-09-07T21:29:42.947 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2022-09-07T21:29:42.947 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 2022-09-07T21:29:42.947 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout data: 2022-09-07T21:29:42.947 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2022-09-07T21:29:42.948 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2022-09-07T21:29:42.948 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2022-09-07T21:29:42.948 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout pgs: 2022-09-07T21:29:42.948 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 2022-09-07T21:29:42.948 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.416+0000 7f48a6b4e700 1 Processor -- start 2022-09-07T21:29:42.949 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.416+0000 7f48a6b4e700 1 -- start start 2022-09-07T21:29:42.949 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.417+0000 7f48a6b4e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a0071ea0 0x7f48a0108650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:29:42.949 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.417+0000 7f48a6b4e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a0104040 con 0x7f48a0071ea0 2022-09-07T21:29:42.949 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.418+0000 7f48a48ea700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a0071ea0 0x7f48a0108650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:29:42.950 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.418+0000 7f48a48ea700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a0071ea0 0x7f48a0108650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51814/0 (socket says 172.21.15.47:51814) 2022-09-07T21:29:42.950 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.418+0000 7f48a48ea700 1 -- 172.21.15.47:0/4190875408 learned_addr learned my addr 172.21.15.47:0/4190875408 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:29:42.950 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.418+0000 7f48a48ea700 1 -- 172.21.15.47:0/4190875408 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a0108b90 con 0x7f48a0071ea0 2022-09-07T21:29:42.950 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.418+0000 7f48a48ea700 1 --2- 172.21.15.47:0/4190875408 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a0071ea0 0x7f48a0108650 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4894009700 tx=0x7f4894009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f55507326d46b9e2 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:29:42.950 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.419+0000 7f489f7fe700 1 -- 172.21.15.47:0/4190875408 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4894004160 con 0x7f48a0071ea0 2022-09-07T21:29:42.951 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.419+0000 7f489f7fe700 1 -- 172.21.15.47:0/4190875408 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48940042c0 con 0x7f48a0071ea0 2022-09-07T21:29:42.951 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.419+0000 7f489f7fe700 1 -- 172.21.15.47:0/4190875408 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4894011630 con 0x7f48a0071ea0 2022-09-07T21:29:42.951 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.419+0000 7f48a6b4e700 1 -- 172.21.15.47:0/4190875408 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a0071ea0 msgr2=0x7f48a0108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:29:42.952 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.419+0000 7f48a6b4e700 1 --2- 172.21.15.47:0/4190875408 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a0071ea0 0x7f48a0108650 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4894009700 tx=0x7f4894009a10 comp rx=0 tx=0).stop 2022-09-07T21:29:42.952 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.419+0000 7f48a6b4e700 1 -- 172.21.15.47:0/4190875408 shutdown_connections 2022-09-07T21:29:42.952 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.419+0000 7f48a6b4e700 1 --2- 172.21.15.47:0/4190875408 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a0071ea0 0x7f48a0108650 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:29:42.952 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.419+0000 7f48a6b4e700 1 -- 172.21.15.47:0/4190875408 >> 172.21.15.47:0/4190875408 conn(0x7f48a00fdc20 msgr2=0x7f48a0100050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:29:42.952 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.420+0000 7f48a6b4e700 1 -- 172.21.15.47:0/4190875408 shutdown_connections 2022-09-07T21:29:42.953 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.420+0000 7f48a6b4e700 1 -- 172.21.15.47:0/4190875408 wait complete. 2022-09-07T21:29:42.953 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.421+0000 7f48a6b4e700 1 Processor -- start 2022-09-07T21:29:42.953 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.421+0000 7f48a6b4e700 1 -- start start 2022-09-07T21:29:42.953 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.421+0000 7f48a6b4e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a0071ea0 0x7f48a0198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:29:42.954 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.421+0000 7f48a6b4e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a0199040 con 0x7f48a0071ea0 2022-09-07T21:29:42.954 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.422+0000 7f48a48ea700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a0071ea0 0x7f48a0198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:29:42.954 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.422+0000 7f48a48ea700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a0071ea0 0x7f48a0198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51826/0 (socket says 172.21.15.47:51826) 2022-09-07T21:29:42.954 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.422+0000 7f48a48ea700 1 -- 172.21.15.47:0/3022474574 learned_addr learned my addr 172.21.15.47:0/3022474574 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:29:42.954 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.422+0000 7f48a48ea700 1 -- 172.21.15.47:0/3022474574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48940093e0 con 0x7f48a0071ea0 2022-09-07T21:29:42.955 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.422+0000 7f48a48ea700 1 --2- 172.21.15.47:0/3022474574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a0071ea0 0x7f48a0198b00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4894000c00 tx=0x7f48940045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:29:42.955 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.423+0000 7f489dffb700 1 -- 172.21.15.47:0/3022474574 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4894003520 con 0x7f48a0071ea0 2022-09-07T21:29:42.955 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.423+0000 7f489dffb700 1 -- 172.21.15.47:0/3022474574 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4894003680 con 0x7f48a0071ea0 2022-09-07T21:29:42.955 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.423+0000 7f489dffb700 1 -- 172.21.15.47:0/3022474574 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4894003920 con 0x7f48a0071ea0 2022-09-07T21:29:42.956 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.423+0000 7f48a6b4e700 1 -- 172.21.15.47:0/3022474574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a0199240 con 0x7f48a0071ea0 2022-09-07T21:29:42.956 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.423+0000 7f48a6b4e700 1 -- 172.21.15.47:0/3022474574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a0199670 con 0x7f48a0071ea0 2022-09-07T21:29:42.956 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.423+0000 7f489dffb700 1 -- 172.21.15.47:0/3022474574 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f489402c020 con 0x7f48a0071ea0 2022-09-07T21:29:42.957 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.423+0000 7f489dffb700 1 -- 172.21.15.47:0/3022474574 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f489401cbf0 con 0x7f48a0071ea0 2022-09-07T21:29:42.957 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.424+0000 7f48a6b4e700 1 -- 172.21.15.47:0/3022474574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a0108de0 con 0x7f48a0071ea0 2022-09-07T21:29:42.957 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.427+0000 7f489dffb700 1 -- 172.21.15.47:0/3022474574 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7f489401c450 con 0x7f48a0071ea0 2022-09-07T21:29:42.957 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.505+0000 7f48a6b4e700 1 -- 172.21.15.47:0/3022474574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f48a005f8f0 con 0x7f48a0071ea0 2022-09-07T21:29:42.957 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.505+0000 7f489dffb700 1 -- 172.21.15.47:0/3022474574 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+310 (secure 0 0 0) 0x7f489401cdb0 con 0x7f48a0071ea0 2022-09-07T21:29:42.958 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.506+0000 7f48a6b4e700 1 -- 172.21.15.47:0/3022474574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a0071ea0 msgr2=0x7f48a0198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:29:42.958 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.506+0000 7f48a6b4e700 1 --2- 172.21.15.47:0/3022474574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a0071ea0 0x7f48a0198b00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4894000c00 tx=0x7f48940045d0 comp rx=0 tx=0).stop 2022-09-07T21:29:42.958 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.506+0000 7f48a6b4e700 1 -- 172.21.15.47:0/3022474574 shutdown_connections 2022-09-07T21:29:42.959 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.506+0000 7f48a6b4e700 1 --2- 172.21.15.47:0/3022474574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a0071ea0 0x7f48a0198b00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:29:42.959 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.506+0000 7f48a6b4e700 1 -- 172.21.15.47:0/3022474574 >> 172.21.15.47:0/3022474574 conn(0x7f48a00fdc20 msgr2=0x7f48a00ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:29:42.959 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.506+0000 7f48a6b4e700 1 -- 172.21.15.47:0/3022474574 shutdown_connections 2022-09-07T21:29:42.959 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:42.506+0000 7f48a6b4e700 1 -- 172.21.15.47:0/3022474574 wait complete. 2022-09-07T21:29:42.960 INFO:teuthology.orchestra.run.smithi047.stdout:mon is available 2022-09-07T21:29:42.960 INFO:teuthology.orchestra.run.smithi047.stdout:Assimilating anything we can from ceph.conf... 2022-09-07T21:29:44.153 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 2022-09-07T21:29:44.154 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout [global] 2022-09-07T21:29:44.155 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout fsid = 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:29:44.155 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.47:3300,v1:172.21.15.47:6789] 2022-09-07T21:29:44.155 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2022-09-07T21:29:44.155 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2022-09-07T21:29:44.155 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2022-09-07T21:29:44.156 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2022-09-07T21:29:44.156 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 2022-09-07T21:29:44.156 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout [mgr] 2022-09-07T21:29:44.156 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2022-09-07T21:29:44.157 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 2022-09-07T21:29:44.157 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout [osd] 2022-09-07T21:29:44.157 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2022-09-07T21:29:44.157 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2022-09-07T21:29:44.158 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2022-09-07T21:29:44.158 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2022-09-07T21:29:44.158 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2022-09-07T21:29:44.158 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2022-09-07T21:29:44.158 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.588+0000 7f0991fe4700 1 Processor -- start 2022-09-07T21:29:44.159 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.588+0000 7f0991fe4700 1 -- start start 2022-09-07T21:29:44.159 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.589+0000 7f0991fe4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f098c105860 0x7f098c105c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:29:44.159 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.589+0000 7f0991fe4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c1061b0 con 0x7f098c105860 2022-09-07T21:29:44.159 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.589+0000 7f098b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f098c105860 0x7f098c105c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:29:44.160 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.590+0000 7f098b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f098c105860 0x7f098c105c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51838/0 (socket says 172.21.15.47:51838) 2022-09-07T21:29:44.160 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.590+0000 7f098b7fe700 1 -- 172.21.15.47:0/3809395309 learned_addr learned my addr 172.21.15.47:0/3809395309 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:29:44.160 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.590+0000 7f098b7fe700 1 -- 172.21.15.47:0/3809395309 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098c1062f0 con 0x7f098c105860 2022-09-07T21:29:44.160 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.590+0000 7f098b7fe700 1 --2- 172.21.15.47:0/3809395309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f098c105860 0x7f098c105c70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f097c009700 tx=0x7f097c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d57c9d987e6d5a server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:29:44.160 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.591+0000 7f098a7fc700 1 -- 172.21.15.47:0/3809395309 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f097c004160 con 0x7f098c105860 2022-09-07T21:29:44.161 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.591+0000 7f098a7fc700 1 -- 172.21.15.47:0/3809395309 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097c0042c0 con 0x7f098c105860 2022-09-07T21:29:44.161 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.591+0000 7f098a7fc700 1 -- 172.21.15.47:0/3809395309 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f097c011630 con 0x7f098c105860 2022-09-07T21:29:44.161 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.591+0000 7f0991fe4700 1 -- 172.21.15.47:0/3809395309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f098c105860 msgr2=0x7f098c105c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:29:44.161 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.591+0000 7f0991fe4700 1 --2- 172.21.15.47:0/3809395309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f098c105860 0x7f098c105c70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f097c009700 tx=0x7f097c009a10 comp rx=0 tx=0).stop 2022-09-07T21:29:44.162 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.591+0000 7f0991fe4700 1 -- 172.21.15.47:0/3809395309 shutdown_connections 2022-09-07T21:29:44.162 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.591+0000 7f0991fe4700 1 --2- 172.21.15.47:0/3809395309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f098c105860 0x7f098c105c70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:29:44.162 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.591+0000 7f0991fe4700 1 -- 172.21.15.47:0/3809395309 >> 172.21.15.47:0/3809395309 conn(0x7f098c1013c0 msgr2=0x7f098c1037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:29:44.162 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.591+0000 7f0991fe4700 1 -- 172.21.15.47:0/3809395309 shutdown_connections 2022-09-07T21:29:44.162 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.591+0000 7f0991fe4700 1 -- 172.21.15.47:0/3809395309 wait complete. 2022-09-07T21:29:44.163 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.593+0000 7f0991fe4700 1 Processor -- start 2022-09-07T21:29:44.163 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.593+0000 7f0991fe4700 1 -- start start 2022-09-07T21:29:44.163 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.593+0000 7f0991fe4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f098c105860 0x7f098c198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:29:44.164 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.593+0000 7f0991fe4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c199070 con 0x7f098c105860 2022-09-07T21:29:44.164 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.594+0000 7f098b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f098c105860 0x7f098c198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:29:44.165 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.594+0000 7f098b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f098c105860 0x7f098c198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51854/0 (socket says 172.21.15.47:51854) 2022-09-07T21:29:44.165 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.594+0000 7f098b7fe700 1 -- 172.21.15.47:0/4284870200 learned_addr learned my addr 172.21.15.47:0/4284870200 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:29:44.166 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.594+0000 7f098b7fe700 1 -- 172.21.15.47:0/4284870200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c0093e0 con 0x7f098c105860 2022-09-07T21:29:44.166 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.594+0000 7f098b7fe700 1 --2- 172.21.15.47:0/4284870200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f098c105860 0x7f098c198b30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f097c004000 tx=0x7f097c0045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:29:44.166 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.594+0000 7f0988ff9700 1 -- 172.21.15.47:0/4284870200 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f097c003520 con 0x7f098c105860 2022-09-07T21:29:44.166 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.595+0000 7f0988ff9700 1 -- 172.21.15.47:0/4284870200 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097c003680 con 0x7f098c105860 2022-09-07T21:29:44.167 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.595+0000 7f0991fe4700 1 -- 172.21.15.47:0/4284870200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098c199270 con 0x7f098c105860 2022-09-07T21:29:44.167 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.595+0000 7f0988ff9700 1 -- 172.21.15.47:0/4284870200 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f097c003920 con 0x7f098c105860 2022-09-07T21:29:44.167 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.595+0000 7f0991fe4700 1 -- 172.21.15.47:0/4284870200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098c1996a0 con 0x7f098c105860 2022-09-07T21:29:44.168 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.595+0000 7f0988ff9700 1 -- 172.21.15.47:0/4284870200 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f097c02c020 con 0x7f098c105860 2022-09-07T21:29:44.168 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.595+0000 7f0988ff9700 1 -- 172.21.15.47:0/4284870200 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f097c01cbf0 con 0x7f098c105860 2022-09-07T21:29:44.168 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.596+0000 7f0991fe4700 1 -- 172.21.15.47:0/4284870200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f098c106c40 con 0x7f098c105860 2022-09-07T21:29:44.168 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.599+0000 7f0988ff9700 1 -- 172.21.15.47:0/4284870200 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7f097c006c00 con 0x7f098c105860 2022-09-07T21:29:44.168 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.671+0000 7f0991fe4700 1 -- 172.21.15.47:0/4284870200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f098c106dd0 con 0x7f098c105860 2022-09-07T21:29:44.169 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.676+0000 7f0988ff9700 1 -- 172.21.15.47:0/4284870200 <== mon.0 v2:172.21.15.47:3300/0 7 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f097c01c4b0 con 0x7f098c105860 2022-09-07T21:29:44.169 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.676+0000 7f0988ff9700 1 -- 172.21.15.47:0/4284870200 <== mon.0 v2:172.21.15.47:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+454 (secure 0 0 0) 0x7f098c106dd0 con 0x7f098c105860 2022-09-07T21:29:44.169 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.677+0000 7f0991fe4700 1 -- 172.21.15.47:0/4284870200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f098c105860 msgr2=0x7f098c198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:29:44.169 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.677+0000 7f0991fe4700 1 --2- 172.21.15.47:0/4284870200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f098c105860 0x7f098c198b30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f097c004000 tx=0x7f097c0045d0 comp rx=0 tx=0).stop 2022-09-07T21:29:44.170 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.677+0000 7f0991fe4700 1 -- 172.21.15.47:0/4284870200 shutdown_connections 2022-09-07T21:29:44.170 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.677+0000 7f0991fe4700 1 --2- 172.21.15.47:0/4284870200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f098c105860 0x7f098c198b30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:29:44.170 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.677+0000 7f0991fe4700 1 -- 172.21.15.47:0/4284870200 >> 172.21.15.47:0/4284870200 conn(0x7f098c1013c0 msgr2=0x7f098c18d320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:29:44.170 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.677+0000 7f0991fe4700 1 -- 172.21.15.47:0/4284870200 shutdown_connections 2022-09-07T21:29:44.171 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:43.677+0000 7f0991fe4700 1 -- 172.21.15.47:0/4284870200 wait complete. 2022-09-07T21:29:44.171 INFO:teuthology.orchestra.run.smithi047.stdout:Generating new minimal ceph.conf... 2022-09-07T21:29:44.172 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:43 smithi047 ceph-mon[89603]: from='client.? 172.21.15.47:0/4284870200' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2022-09-07T21:29:44.172 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:43 smithi047 ceph-mon[89603]: from='client.? 172.21.15.47:0/4284870200' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2022-09-07T21:29:45.379 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:44 smithi047 ceph-mon[89603]: from='client.? 172.21.15.47:0/1815132287' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:29:45.457 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.844+0000 7fd7aa8ee700 1 Processor -- start 2022-09-07T21:29:45.458 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.845+0000 7fd7aa8ee700 1 -- start start 2022-09-07T21:29:45.458 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.845+0000 7fd7aa8ee700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7a4078cc0 0x7fd7a4077240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:29:45.458 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.845+0000 7fd7aa8ee700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a4077780 con 0x7fd7a4078cc0 2022-09-07T21:29:45.459 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.845+0000 7fd7a3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7a4078cc0 0x7fd7a4077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:29:45.459 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.846+0000 7fd7a3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7a4078cc0 0x7fd7a4077240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51868/0 (socket says 172.21.15.47:51868) 2022-09-07T21:29:45.459 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.846+0000 7fd7a3fff700 1 -- 172.21.15.47:0/2523705808 learned_addr learned my addr 172.21.15.47:0/2523705808 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:29:45.459 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.846+0000 7fd7a3fff700 1 -- 172.21.15.47:0/2523705808 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a40778c0 con 0x7fd7a4078cc0 2022-09-07T21:29:45.460 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.846+0000 7fd7a3fff700 1 --2- 172.21.15.47:0/2523705808 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7a4078cc0 0x7fd7a4077240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd794009700 tx=0x7fd794009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b969d922c2bf236 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:29:45.460 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.847+0000 7fd7a2ffd700 1 -- 172.21.15.47:0/2523705808 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd794004160 con 0x7fd7a4078cc0 2022-09-07T21:29:45.460 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.847+0000 7fd7a2ffd700 1 -- 172.21.15.47:0/2523705808 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fd7940042c0 con 0x7fd7a4078cc0 2022-09-07T21:29:45.460 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.847+0000 7fd7a2ffd700 1 -- 172.21.15.47:0/2523705808 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd7940039d0 con 0x7fd7a4078cc0 2022-09-07T21:29:45.461 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.847+0000 7fd7aa8ee700 1 -- 172.21.15.47:0/2523705808 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7a4078cc0 msgr2=0x7fd7a4077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:29:45.461 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.847+0000 7fd7aa8ee700 1 --2- 172.21.15.47:0/2523705808 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7a4078cc0 0x7fd7a4077240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd794009700 tx=0x7fd794009a10 comp rx=0 tx=0).stop 2022-09-07T21:29:45.461 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.847+0000 7fd7aa8ee700 1 -- 172.21.15.47:0/2523705808 shutdown_connections 2022-09-07T21:29:45.461 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.847+0000 7fd7aa8ee700 1 --2- 172.21.15.47:0/2523705808 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7a4078cc0 0x7fd7a4077240 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:29:45.462 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.847+0000 7fd7aa8ee700 1 -- 172.21.15.47:0/2523705808 >> 172.21.15.47:0/2523705808 conn(0x7fd7a40fed50 msgr2=0x7fd7a4101180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:29:45.462 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.848+0000 7fd7aa8ee700 1 -- 172.21.15.47:0/2523705808 shutdown_connections 2022-09-07T21:29:45.462 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.848+0000 7fd7aa8ee700 1 -- 172.21.15.47:0/2523705808 wait complete. 2022-09-07T21:29:45.462 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.849+0000 7fd7aa8ee700 1 Processor -- start 2022-09-07T21:29:45.463 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.849+0000 7fd7aa8ee700 1 -- start start 2022-09-07T21:29:45.463 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.849+0000 7fd7aa8ee700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7a4078cc0 0x7fd7a419d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:29:45.463 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.849+0000 7fd7aa8ee700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a419db80 con 0x7fd7a4078cc0 2022-09-07T21:29:45.463 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.850+0000 7fd7a3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7a4078cc0 0x7fd7a419d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:29:45.464 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.850+0000 7fd7a3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7a4078cc0 0x7fd7a419d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51882/0 (socket says 172.21.15.47:51882) 2022-09-07T21:29:45.464 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.850+0000 7fd7a3fff700 1 -- 172.21.15.47:0/1815132287 learned_addr learned my addr 172.21.15.47:0/1815132287 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:29:45.464 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.850+0000 7fd7a3fff700 1 -- 172.21.15.47:0/1815132287 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7940093e0 con 0x7fd7a4078cc0 2022-09-07T21:29:45.464 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.850+0000 7fd7a3fff700 1 --2- 172.21.15.47:0/1815132287 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7a4078cc0 0x7fd7a419d640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd794009e90 tx=0x7fd794004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:29:45.464 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.850+0000 7fd7a17fa700 1 -- 172.21.15.47:0/1815132287 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd794011840 con 0x7fd7a4078cc0 2022-09-07T21:29:45.465 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.851+0000 7fd7a17fa700 1 -- 172.21.15.47:0/1815132287 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fd79401a430 con 0x7fd7a4078cc0 2022-09-07T21:29:45.465 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.851+0000 7fd7a17fa700 1 -- 172.21.15.47:0/1815132287 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd794019550 con 0x7fd7a4078cc0 2022-09-07T21:29:45.465 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.851+0000 7fd7aa8ee700 1 -- 172.21.15.47:0/1815132287 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a419dd80 con 0x7fd7a4078cc0 2022-09-07T21:29:45.465 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.851+0000 7fd7aa8ee700 1 -- 172.21.15.47:0/1815132287 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a419e1b0 con 0x7fd7a4078cc0 2022-09-07T21:29:45.466 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.851+0000 7fd7a17fa700 1 -- 172.21.15.47:0/1815132287 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fd79402a020 con 0x7fd7a4078cc0 2022-09-07T21:29:45.466 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.851+0000 7fd7a17fa700 1 -- 172.21.15.47:0/1815132287 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd79402d480 con 0x7fd7a4078cc0 2022-09-07T21:29:45.466 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.853+0000 7fd7aa8ee700 1 -- 172.21.15.47:0/1815132287 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a4060310 con 0x7fd7a4078cc0 2022-09-07T21:29:45.466 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.855+0000 7fd7a17fa700 1 -- 172.21.15.47:0/1815132287 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7fd79401a5a0 con 0x7fd7a4078cc0 2022-09-07T21:29:45.467 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.929+0000 7fd7aa8ee700 1 -- 172.21.15.47:0/1815132287 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd7a405f990 con 0x7fd7a4078cc0 2022-09-07T21:29:45.467 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.929+0000 7fd7a17fa700 1 -- 172.21.15.47:0/1815132287 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fd79401a740 con 0x7fd7a4078cc0 2022-09-07T21:29:45.467 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.930+0000 7fd7aa8ee700 1 -- 172.21.15.47:0/1815132287 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7a4078cc0 msgr2=0x7fd7a419d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:29:45.467 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.930+0000 7fd7aa8ee700 1 --2- 172.21.15.47:0/1815132287 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7a4078cc0 0x7fd7a419d640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd794009e90 tx=0x7fd794004040 comp rx=0 tx=0).stop 2022-09-07T21:29:45.468 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.930+0000 7fd7aa8ee700 1 -- 172.21.15.47:0/1815132287 shutdown_connections 2022-09-07T21:29:45.468 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.930+0000 7fd7aa8ee700 1 --2- 172.21.15.47:0/1815132287 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7a4078cc0 0x7fd7a419d640 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:29:45.468 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.930+0000 7fd7aa8ee700 1 -- 172.21.15.47:0/1815132287 >> 172.21.15.47:0/1815132287 conn(0x7fd7a40fed50 msgr2=0x7fd7a4104860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:29:45.468 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.930+0000 7fd7aa8ee700 1 -- 172.21.15.47:0/1815132287 shutdown_connections 2022-09-07T21:29:45.469 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:44.930+0000 7fd7aa8ee700 1 -- 172.21.15.47:0/1815132287 wait complete. 2022-09-07T21:29:45.469 INFO:teuthology.orchestra.run.smithi047.stdout:Restarting the monitor... 2022-09-07T21:29:45.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:45 smithi047 systemd[1]: Stopping Ceph mon.a for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:29:46.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[89579]: 2022-09-07T21:29:45.970+0000 7fd3c2032700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2022-09-07T21:29:46.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[89579]: 2022-09-07T21:29:45.970+0000 7fd3c2032700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-09-07T21:29:46.743 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:46 smithi047 bash[89945]: ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a 2022-09-07T21:29:47.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:46 smithi047 systemd[1]: ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mon.a.service: Succeeded. 2022-09-07T21:29:47.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:46 smithi047 systemd[1]: Stopped Ceph mon.a for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:29:47.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:46 smithi047 systemd[1]: Starting Ceph mon.a for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:29:47.792 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 podman[90047]: 2022-09-07T21:29:47.861 INFO:teuthology.orchestra.run.smithi047.stdout:Setting mon public_network to 172.21.0.0/20 2022-09-07T21:29:48.147 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 bash[90047]: 206ccff106127b9b25b545726cfff93eae8bc06aa090ce0c5ace3e2b8989f78b 2022-09-07T21:29:48.147 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: set uid:gid to 167:167 (ceph:ceph) 2022-09-07T21:29:48.147 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: ceph version 17.0.0-14857-g15219419 (152194198413033811b31149de909a7ec5852a7f) quincy (dev), process ceph-mon, pid 7 2022-09-07T21:29:48.147 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: pidfile_write: ignore empty --pid-file 2022-09-07T21:29:48.148 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: load: jerasure load: lrc 2022-09-07T21:29:48.148 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: RocksDB version: 6.15.5 2022-09-07T21:29:48.148 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-09-07T21:29:48.148 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Compile date Sep 7 2022 2022-09-07T21:29:48.149 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: DB SUMMARY 2022-09-07T21:29:48.149 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: DB Session ID: 8QR1AA8VXLSEZ7B9QEW0 2022-09-07T21:29:48.149 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: CURRENT file: CURRENT 2022-09-07T21:29:48.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: IDENTITY file: IDENTITY 2022-09-07T21:29:48.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2022-09-07T21:29:48.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2022-09-07T21:29:48.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 99453 ; 2022-09-07T21:29:48.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.error_if_exists: 0 2022-09-07T21:29:48.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.create_if_missing: 0 2022-09-07T21:29:48.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.paranoid_checks: 1 2022-09-07T21:29:48.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-09-07T21:29:48.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.env: 0x55a175867100 2022-09-07T21:29:48.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.fs: Posix File System 2022-09-07T21:29:48.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.info_log: 0x55a1776ebc20 2022-09-07T21:29:48.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_file_opening_threads: 16 2022-09-07T21:29:48.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.statistics: (nil) 2022-09-07T21:29:48.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.use_fsync: 0 2022-09-07T21:29:48.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_log_file_size: 0 2022-09-07T21:29:48.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-09-07T21:29:48.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.log_file_time_to_roll: 0 2022-09-07T21:29:48.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.keep_log_file_num: 1000 2022-09-07T21:29:48.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.recycle_log_file_num: 0 2022-09-07T21:29:48.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.allow_fallocate: 1 2022-09-07T21:29:48.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.allow_mmap_reads: 0 2022-09-07T21:29:48.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.allow_mmap_writes: 0 2022-09-07T21:29:48.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.use_direct_reads: 0 2022-09-07T21:29:48.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-09-07T21:29:48.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.create_missing_column_families: 0 2022-09-07T21:29:48.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.db_log_dir: 2022-09-07T21:29:48.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2022-09-07T21:29:48.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.table_cache_numshardbits: 6 2022-09-07T21:29:48.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.WAL_ttl_seconds: 0 2022-09-07T21:29:48.158 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.WAL_size_limit_MB: 0 2022-09-07T21:29:48.158 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-09-07T21:29:48.158 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-09-07T21:29:48.158 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.is_fd_close_on_exec: 1 2022-09-07T21:29:48.159 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.advise_random_on_open: 1 2022-09-07T21:29:48.159 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.db_write_buffer_size: 0 2022-09-07T21:29:48.159 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.write_buffer_manager: 0x55a1777e0b70 2022-09-07T21:29:48.159 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-09-07T21:29:48.159 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-09-07T21:29:48.160 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-09-07T21:29:48.160 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.use_adaptive_mutex: 0 2022-09-07T21:29:48.160 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.rate_limiter: (nil) 2022-09-07T21:29:48.161 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-09-07T21:29:48.161 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.wal_recovery_mode: 2 2022-09-07T21:29:48.161 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.enable_thread_tracking: 0 2022-09-07T21:29:48.161 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.enable_pipelined_write: 0 2022-09-07T21:29:48.162 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.unordered_write: 0 2022-09-07T21:29:48.162 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-09-07T21:29:48.162 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-09-07T21:29:48.162 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-09-07T21:29:48.163 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-09-07T21:29:48.163 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.row_cache: None 2022-09-07T21:29:48.163 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.wal_filter: None 2022-09-07T21:29:48.163 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-09-07T21:29:48.163 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.allow_ingest_behind: 0 2022-09-07T21:29:48.164 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.preserve_deletes: 0 2022-09-07T21:29:48.164 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.two_write_queues: 0 2022-09-07T21:29:48.164 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.manual_wal_flush: 0 2022-09-07T21:29:48.165 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.atomic_flush: 0 2022-09-07T21:29:48.165 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-09-07T21:29:48.166 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.persist_stats_to_disk: 0 2022-09-07T21:29:48.167 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.write_dbid_to_manifest: 0 2022-09-07T21:29:48.167 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.log_readahead_size: 0 2022-09-07T21:29:48.167 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-09-07T21:29:48.168 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.best_efforts_recovery: 0 2022-09-07T21:29:48.168 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-09-07T21:29:48.169 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-09-07T21:29:48.169 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.allow_data_in_errors: 0 2022-09-07T21:29:48.169 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.db_host_id: __hostname__ 2022-09-07T21:29:48.170 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_background_jobs: 2 2022-09-07T21:29:48.170 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_background_compactions: -1 2022-09-07T21:29:48.171 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_subcompactions: 1 2022-09-07T21:29:48.171 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-09-07T21:29:48.171 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-09-07T21:29:48.172 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.delayed_write_rate : 16777216 2022-09-07T21:29:48.172 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_total_wal_size: 0 2022-09-07T21:29:48.172 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-09-07T21:29:48.172 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.stats_dump_period_sec: 600 2022-09-07T21:29:48.173 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.stats_persist_period_sec: 600 2022-09-07T21:29:48.173 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-09-07T21:29:48.173 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_open_files: -1 2022-09-07T21:29:48.173 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.bytes_per_sync: 0 2022-09-07T21:29:48.173 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.wal_bytes_per_sync: 0 2022-09-07T21:29:48.174 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.strict_bytes_per_sync: 0 2022-09-07T21:29:48.174 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compaction_readahead_size: 0 2022-09-07T21:29:48.174 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_background_flushes: -1 2022-09-07T21:29:48.174 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Compression algorithms supported: 2022-09-07T21:29:48.175 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-09-07T21:29:48.175 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: kZSTD supported: 0 2022-09-07T21:29:48.175 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: kXpressCompression supported: 0 2022-09-07T21:29:48.175 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: kLZ4HCCompression supported: 1 2022-09-07T21:29:48.175 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: kZlibCompression supported: 1 2022-09-07T21:29:48.176 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: kSnappyCompression supported: 1 2022-09-07T21:29:48.176 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: kLZ4Compression supported: 1 2022-09-07T21:29:48.176 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: kBZip2Compression supported: 0 2022-09-07T21:29:48.176 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Fast CRC32 supported: Supported on x86 2022-09-07T21:29:48.176 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-09-07T21:29:48.177 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2022-09-07T21:29:48.177 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-09-07T21:29:48.177 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.merge_operator: 2022-09-07T21:29:48.177 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compaction_filter: None 2022-09-07T21:29:48.178 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compaction_filter_factory: None 2022-09-07T21:29:48.179 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.sst_partitioner_factory: None 2022-09-07T21:29:48.179 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.memtable_factory: SkipListFactory 2022-09-07T21:29:48.180 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.table_factory: BlockBasedTable 2022-09-07T21:29:48.180 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a1776e8d48) 2022-09-07T21:29:48.180 INFO:journalctl@ceph.mon.a.smithi047.stdout: cache_index_and_filter_blocks: 1 2022-09-07T21:29:48.180 INFO:journalctl@ceph.mon.a.smithi047.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-09-07T21:29:48.181 INFO:journalctl@ceph.mon.a.smithi047.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-09-07T21:29:48.181 INFO:journalctl@ceph.mon.a.smithi047.stdout: pin_top_level_index_and_filter: 1 2022-09-07T21:29:48.181 INFO:journalctl@ceph.mon.a.smithi047.stdout: index_type: 0 2022-09-07T21:29:48.182 INFO:journalctl@ceph.mon.a.smithi047.stdout: data_block_index_type: 0 2022-09-07T21:29:48.182 INFO:journalctl@ceph.mon.a.smithi047.stdout: index_shortening: 1 2022-09-07T21:29:48.183 INFO:journalctl@ceph.mon.a.smithi047.stdout: data_block_hash_table_util_ratio: 0.750000 2022-09-07T21:29:48.183 INFO:journalctl@ceph.mon.a.smithi047.stdout: hash_index_allow_collision: 1 2022-09-07T21:29:48.183 INFO:journalctl@ceph.mon.a.smithi047.stdout: checksum: 1 2022-09-07T21:29:48.183 INFO:journalctl@ceph.mon.a.smithi047.stdout: no_block_cache: 0 2022-09-07T21:29:48.183 INFO:journalctl@ceph.mon.a.smithi047.stdout: block_cache: 0x55a1777549b0 2022-09-07T21:29:48.184 INFO:journalctl@ceph.mon.a.smithi047.stdout: block_cache_name: BinnedLRUCache 2022-09-07T21:29:48.184 INFO:journalctl@ceph.mon.a.smithi047.stdout: block_cache_options: 2022-09-07T21:29:48.184 INFO:journalctl@ceph.mon.a.smithi047.stdout: capacity : 536870912 2022-09-07T21:29:48.184 INFO:journalctl@ceph.mon.a.smithi047.stdout: num_shard_bits : 4 2022-09-07T21:29:48.185 INFO:journalctl@ceph.mon.a.smithi047.stdout: strict_capacity_limit : 0 2022-09-07T21:29:48.185 INFO:journalctl@ceph.mon.a.smithi047.stdout: high_pri_pool_ratio: 0.000 2022-09-07T21:29:48.185 INFO:journalctl@ceph.mon.a.smithi047.stdout: block_cache_compressed: (nil) 2022-09-07T21:29:48.185 INFO:journalctl@ceph.mon.a.smithi047.stdout: persistent_cache: (nil) 2022-09-07T21:29:48.185 INFO:journalctl@ceph.mon.a.smithi047.stdout: block_size: 4096 2022-09-07T21:29:48.186 INFO:journalctl@ceph.mon.a.smithi047.stdout: block_size_deviation: 10 2022-09-07T21:29:48.186 INFO:journalctl@ceph.mon.a.smithi047.stdout: block_restart_interval: 16 2022-09-07T21:29:48.187 INFO:journalctl@ceph.mon.a.smithi047.stdout: index_block_restart_interval: 1 2022-09-07T21:29:48.187 INFO:journalctl@ceph.mon.a.smithi047.stdout: metadata_block_size: 4096 2022-09-07T21:29:48.187 INFO:journalctl@ceph.mon.a.smithi047.stdout: partition_filters: 0 2022-09-07T21:29:48.187 INFO:journalctl@ceph.mon.a.smithi047.stdout: use_delta_encoding: 1 2022-09-07T21:29:48.187 INFO:journalctl@ceph.mon.a.smithi047.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-09-07T21:29:48.188 INFO:journalctl@ceph.mon.a.smithi047.stdout: whole_key_filtering: 1 2022-09-07T21:29:48.188 INFO:journalctl@ceph.mon.a.smithi047.stdout: verify_compression: 0 2022-09-07T21:29:48.188 INFO:journalctl@ceph.mon.a.smithi047.stdout: read_amp_bytes_per_bit: 0 2022-09-07T21:29:48.188 INFO:journalctl@ceph.mon.a.smithi047.stdout: format_version: 4 2022-09-07T21:29:48.188 INFO:journalctl@ceph.mon.a.smithi047.stdout: enable_index_compression: 1 2022-09-07T21:29:48.189 INFO:journalctl@ceph.mon.a.smithi047.stdout: block_align: 0 2022-09-07T21:29:48.189 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.write_buffer_size: 33554432 2022-09-07T21:29:48.190 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_write_buffer_number: 2 2022-09-07T21:29:48.190 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compression: NoCompression 2022-09-07T21:29:48.190 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.bottommost_compression: Disabled 2022-09-07T21:29:48.190 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.prefix_extractor: nullptr 2022-09-07T21:29:48.191 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-09-07T21:29:48.191 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.num_levels: 7 2022-09-07T21:29:48.191 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-09-07T21:29:48.191 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-09-07T21:29:48.192 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-09-07T21:29:48.192 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-09-07T21:29:48.192 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-09-07T21:29:48.192 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-09-07T21:29:48.192 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-09-07T21:29:48.193 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-09-07T21:29:48.193 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-09-07T21:29:48.194 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-09-07T21:29:48.194 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compression_opts.window_bits: -14 2022-09-07T21:29:48.195 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compression_opts.level: 32767 2022-09-07T21:29:48.195 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compression_opts.strategy: 0 2022-09-07T21:29:48.195 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-09-07T21:29:48.196 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-09-07T21:29:48.196 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-09-07T21:29:48.196 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compression_opts.enabled: false 2022-09-07T21:29:48.196 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-09-07T21:29:48.197 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-09-07T21:29:48.197 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-09-07T21:29:48.197 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.target_file_size_base: 67108864 2022-09-07T21:29:48.197 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.target_file_size_multiplier: 1 2022-09-07T21:29:48.198 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-09-07T21:29:48.198 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-09-07T21:29:48.199 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-09-07T21:29:48.199 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-09-07T21:29:48.199 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-09-07T21:29:48.199 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-09-07T21:29:48.199 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-09-07T21:29:48.200 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-09-07T21:29:48.200 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-09-07T21:29:48.200 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-09-07T21:29:48.200 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-09-07T21:29:48.200 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-09-07T21:29:48.201 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.arena_block_size: 4194304 2022-09-07T21:29:48.201 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-09-07T21:29:48.201 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-09-07T21:29:48.201 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-09-07T21:29:48.202 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.disable_auto_compactions: 0 2022-09-07T21:29:48.202 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-09-07T21:29:48.202 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-09-07T21:29:48.203 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-09-07T21:29:48.203 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-09-07T21:29:48.203 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-09-07T21:29:48.204 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-09-07T21:29:48.204 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-09-07T21:29:48.204 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-09-07T21:29:48.204 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-09-07T21:29:48.204 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-09-07T21:29:48.206 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.table_properties_collectors: 2022-09-07T21:29:48.207 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.inplace_update_support: 0 2022-09-07T21:29:48.207 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.inplace_update_num_locks: 10000 2022-09-07T21:29:48.207 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-09-07T21:29:48.208 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-09-07T21:29:48.208 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.memtable_huge_page_size: 0 2022-09-07T21:29:48.208 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.bloom_locality: 0 2022-09-07T21:29:48.208 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.max_successive_merges: 0 2022-09-07T21:29:48.209 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.optimize_filters_for_hits: 0 2022-09-07T21:29:48.209 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.paranoid_file_checks: 0 2022-09-07T21:29:48.209 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.force_consistency_checks: 1 2022-09-07T21:29:48.209 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.report_bg_io_stats: 0 2022-09-07T21:29:48.209 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.ttl: 2592000 2022-09-07T21:29:48.210 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.periodic_compaction_seconds: 0 2022-09-07T21:29:48.210 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.enable_blob_files: false 2022-09-07T21:29:48.210 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.min_blob_size: 0 2022-09-07T21:29:48.211 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.blob_file_size: 268435456 2022-09-07T21:29:48.211 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.blob_compression_type: NoCompression 2022-09-07T21:29:48.211 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.enable_blob_garbage_collection: false 2022-09-07T21:29:48.212 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-09-07T21:29:48.212 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-09-07T21:29:48.212 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2022-09-07T21:29:48.213 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2022-09-07T21:29:48.213 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662586187814864, "job": 1, "event": "recovery_started", "wal_files": [10]} 2022-09-07T21:29:48.213 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2022-09-07T21:29:48.214 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-09-07T21:29:48.214 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662586187817808, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 93977, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 92064, "index_size": 273, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13837, "raw_average_key_size": 49, "raw_value_size": 84452, "raw_average_value_size": 302, "num_data_blocks": 11, "num_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1662586187, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "1d688e3a-328f-40d4-8448-5a0786adaf51", "db_session_id": "8QR1AA8VXLSEZ7B9QEW0"}} 2022-09-07T21:29:48.214 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2022-09-07T21:29:48.215 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662586187819075, "job": 1, "event": "recovery_finished"} 2022-09-07T21:29:48.215 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-09-07T21:29:48.216 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55a17773b880 2022-09-07T21:29:48.216 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: DB pointer 0x55a1777ae000 2022-09-07T21:29:48.216 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2022-09-07T21:29:48.217 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-09-07T21:29:48.217 INFO:journalctl@ceph.mon.a.smithi047.stdout: ** DB Stats ** 2022-09-07T21:29:48.217 INFO:journalctl@ceph.mon.a.smithi047.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-07T21:29:48.218 INFO:journalctl@ceph.mon.a.smithi047.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-09-07T21:29:48.218 INFO:journalctl@ceph.mon.a.smithi047.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-09-07T21:29:48.218 INFO:journalctl@ceph.mon.a.smithi047.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-09-07T21:29:48.219 INFO:journalctl@ceph.mon.a.smithi047.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-09-07T21:29:48.219 INFO:journalctl@ceph.mon.a.smithi047.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-09-07T21:29:48.219 INFO:journalctl@ceph.mon.a.smithi047.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-09-07T21:29:48.219 INFO:journalctl@ceph.mon.a.smithi047.stdout: 2022-09-07T21:29:48.220 INFO:journalctl@ceph.mon.a.smithi047.stdout: ** Compaction Stats [default] ** 2022-09-07T21:29:48.220 INFO:journalctl@ceph.mon.a.smithi047.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-07T21:29:48.221 INFO:journalctl@ceph.mon.a.smithi047.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-07T21:29:48.221 INFO:journalctl@ceph.mon.a.smithi047.stdout: L0 2/0 93.53 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 49.7 0.00 0.00 1 0.002 0 0 2022-09-07T21:29:48.221 INFO:journalctl@ceph.mon.a.smithi047.stdout: Sum 2/0 93.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 49.7 0.00 0.00 1 0.002 0 0 2022-09-07T21:29:48.221 INFO:journalctl@ceph.mon.a.smithi047.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 49.7 0.00 0.00 1 0.002 0 0 2022-09-07T21:29:48.222 INFO:journalctl@ceph.mon.a.smithi047.stdout: 2022-09-07T21:29:48.222 INFO:journalctl@ceph.mon.a.smithi047.stdout: ** Compaction Stats [default] ** 2022-09-07T21:29:48.222 INFO:journalctl@ceph.mon.a.smithi047.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-07T21:29:48.222 INFO:journalctl@ceph.mon.a.smithi047.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-07T21:29:48.223 INFO:journalctl@ceph.mon.a.smithi047.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 49.7 0.00 0.00 1 0.002 0 0 2022-09-07T21:29:48.223 INFO:journalctl@ceph.mon.a.smithi047.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-07T21:29:48.223 INFO:journalctl@ceph.mon.a.smithi047.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-09-07T21:29:48.224 INFO:journalctl@ceph.mon.a.smithi047.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-09-07T21:29:48.224 INFO:journalctl@ceph.mon.a.smithi047.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-09-07T21:29:48.224 INFO:journalctl@ceph.mon.a.smithi047.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-09-07T21:29:48.224 INFO:journalctl@ceph.mon.a.smithi047.stdout: AddFile(Keys): cumulative 0, interval 0 2022-09-07T21:29:48.225 INFO:journalctl@ceph.mon.a.smithi047.stdout: Cumulative compaction: 0.00 GB write, 7.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-07T21:29:48.225 INFO:journalctl@ceph.mon.a.smithi047.stdout: Interval compaction: 0.00 GB write, 7.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-07T21:29:48.225 INFO:journalctl@ceph.mon.a.smithi047.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-09-07T21:29:48.226 INFO:journalctl@ceph.mon.a.smithi047.stdout: 2022-09-07T21:29:48.226 INFO:journalctl@ceph.mon.a.smithi047.stdout: ** File Read Latency Histogram By Level [default] ** 2022-09-07T21:29:48.226 INFO:journalctl@ceph.mon.a.smithi047.stdout: 2022-09-07T21:29:48.227 INFO:journalctl@ceph.mon.a.smithi047.stdout: ** Compaction Stats [default] ** 2022-09-07T21:29:48.227 INFO:journalctl@ceph.mon.a.smithi047.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-07T21:29:48.228 INFO:journalctl@ceph.mon.a.smithi047.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-07T21:29:48.228 INFO:journalctl@ceph.mon.a.smithi047.stdout: L0 2/0 93.53 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 49.7 0.00 0.00 1 0.002 0 0 2022-09-07T21:29:48.229 INFO:journalctl@ceph.mon.a.smithi047.stdout: Sum 2/0 93.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 49.7 0.00 0.00 1 0.002 0 0 2022-09-07T21:29:48.229 INFO:journalctl@ceph.mon.a.smithi047.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-09-07T21:29:48.229 INFO:journalctl@ceph.mon.a.smithi047.stdout: 2022-09-07T21:29:48.230 INFO:journalctl@ceph.mon.a.smithi047.stdout: ** Compaction Stats [default] ** 2022-09-07T21:29:48.230 INFO:journalctl@ceph.mon.a.smithi047.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-07T21:29:48.231 INFO:journalctl@ceph.mon.a.smithi047.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-07T21:29:48.231 INFO:journalctl@ceph.mon.a.smithi047.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 49.7 0.00 0.00 1 0.002 0 0 2022-09-07T21:29:48.232 INFO:journalctl@ceph.mon.a.smithi047.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-07T21:29:48.232 INFO:journalctl@ceph.mon.a.smithi047.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-09-07T21:29:48.233 INFO:journalctl@ceph.mon.a.smithi047.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-09-07T21:29:48.233 INFO:journalctl@ceph.mon.a.smithi047.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-09-07T21:29:48.233 INFO:journalctl@ceph.mon.a.smithi047.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-09-07T21:29:48.234 INFO:journalctl@ceph.mon.a.smithi047.stdout: AddFile(Keys): cumulative 0, interval 0 2022-09-07T21:29:48.234 INFO:journalctl@ceph.mon.a.smithi047.stdout: Cumulative compaction: 0.00 GB write, 7.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-07T21:29:48.234 INFO:journalctl@ceph.mon.a.smithi047.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-07T21:29:48.234 INFO:journalctl@ceph.mon.a.smithi047.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-09-07T21:29:48.235 INFO:journalctl@ceph.mon.a.smithi047.stdout: 2022-09-07T21:29:48.235 INFO:journalctl@ceph.mon.a.smithi047.stdout: ** File Read Latency Histogram By Level [default] ** 2022-09-07T21:29:48.235 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: starting mon.a rank 0 at public addrs [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] at bind addrs [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:29:48.235 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: mon.a@-1(???) e1 preinit fsid 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:29:48.235 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: mon.a@-1(???).mds e1 new map 2022-09-07T21:29:48.236 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: mon.a@-1(???).mds e1 print_map 2022-09-07T21:29:48.236 INFO:journalctl@ceph.mon.a.smithi047.stdout: e1 2022-09-07T21:29:48.236 INFO:journalctl@ceph.mon.a.smithi047.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-09-07T21:29:48.237 INFO:journalctl@ceph.mon.a.smithi047.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-09-07T21:29:48.237 INFO:journalctl@ceph.mon.a.smithi047.stdout: legacy client fscid: -1 2022-09-07T21:29:48.237 INFO:journalctl@ceph.mon.a.smithi047.stdout: 2022-09-07T21:29:48.237 INFO:journalctl@ceph.mon.a.smithi047.stdout: No filesystems configured 2022-09-07T21:29:48.238 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2022-09-07T21:29:48.238 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-09-07T21:29:48.238 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-09-07T21:29:48.238 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-09-07T21:29:48.239 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2022-09-07T21:29:48.239 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 systemd[1]: Started Ceph mon.a for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:29:48.239 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: mon.a is new leader, mons a in quorum (ranks 0) 2022-09-07T21:29:48.239 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: monmap e1: 1 mons at {a=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0]} 2022-09-07T21:29:48.239 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: fsmap 2022-09-07T21:29:48.240 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: osdmap e1: 0 total, 0 up, 0 in 2022-09-07T21:29:48.240 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:47 smithi047 ceph-mon[90087]: mgrmap e1: no daemons active 2022-09-07T21:29:49.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.458+0000 7f7adde9f700 1 Processor -- start 2022-09-07T21:29:49.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.458+0000 7f7adde9f700 1 -- start start 2022-09-07T21:29:49.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.458+0000 7f7adde9f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ad8071e80 0x7f7ad81060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:29:49.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.458+0000 7f7adde9f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad81065f0 con 0x7f7ad8071e80 2022-09-07T21:29:49.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.459+0000 7f7adce9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ad8071e80 0x7f7ad81060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:29:49.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.460+0000 7f7adce9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ad8071e80 0x7f7ad81060b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51886/0 (socket says 172.21.15.47:51886) 2022-09-07T21:29:49.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.460+0000 7f7adce9d700 1 -- 172.21.15.47:0/3310416508 learned_addr learned my addr 172.21.15.47:0/3310416508 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:29:49.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.460+0000 7f7adce9d700 1 -- 172.21.15.47:0/3310416508 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad8106730 con 0x7f7ad8071e80 2022-09-07T21:29:49.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.460+0000 7f7adce9d700 1 --2- 172.21.15.47:0/3310416508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ad8071e80 0x7f7ad81060b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7acc009700 tx=0x7f7acc009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2195e051671cf029 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:29:49.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.461+0000 7f7ad77fe700 1 -- 172.21.15.47:0/3310416508 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7acc004160 con 0x7f7ad8071e80 2022-09-07T21:29:49.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.461+0000 7f7ad77fe700 1 -- 172.21.15.47:0/3310416508 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f7acc0042c0 con 0x7f7ad8071e80 2022-09-07T21:29:49.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.461+0000 7f7ad77fe700 1 -- 172.21.15.47:0/3310416508 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7acc0039d0 con 0x7f7ad8071e80 2022-09-07T21:29:49.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.461+0000 7f7adde9f700 1 -- 172.21.15.47:0/3310416508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ad8071e80 msgr2=0x7f7ad81060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:29:49.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.461+0000 7f7adde9f700 1 --2- 172.21.15.47:0/3310416508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ad8071e80 0x7f7ad81060b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7acc009700 tx=0x7f7acc009a10 comp rx=0 tx=0).stop 2022-09-07T21:29:49.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.461+0000 7f7adde9f700 1 -- 172.21.15.47:0/3310416508 shutdown_connections 2022-09-07T21:29:49.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.461+0000 7f7adde9f700 1 --2- 172.21.15.47:0/3310416508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ad8071e80 0x7f7ad81060b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:29:49.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.461+0000 7f7adde9f700 1 -- 172.21.15.47:0/3310416508 >> 172.21.15.47:0/3310416508 conn(0x7f7ad80fda80 msgr2=0x7f7ad80ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:29:49.076 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.462+0000 7f7adde9f700 1 -- 172.21.15.47:0/3310416508 shutdown_connections 2022-09-07T21:29:49.076 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.462+0000 7f7adde9f700 1 -- 172.21.15.47:0/3310416508 wait complete. 2022-09-07T21:29:49.076 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.463+0000 7f7adde9f700 1 Processor -- start 2022-09-07T21:29:49.076 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.463+0000 7f7adde9f700 1 -- start start 2022-09-07T21:29:49.077 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.463+0000 7f7adde9f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ad8071e80 0x7f7ad8190240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:29:49.077 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.463+0000 7f7adde9f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad8190780 con 0x7f7ad8071e80 2022-09-07T21:29:49.077 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.463+0000 7f7adce9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ad8071e80 0x7f7ad8190240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:29:49.077 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.464+0000 7f7adce9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ad8071e80 0x7f7ad8190240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51896/0 (socket says 172.21.15.47:51896) 2022-09-07T21:29:49.077 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.464+0000 7f7adce9d700 1 -- 172.21.15.47:0/3850474600 learned_addr learned my addr 172.21.15.47:0/3850474600 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:29:49.078 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.464+0000 7f7adce9d700 1 -- 172.21.15.47:0/3850474600 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc0093e0 con 0x7f7ad8071e80 2022-09-07T21:29:49.078 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.464+0000 7f7adce9d700 1 --2- 172.21.15.47:0/3850474600 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ad8071e80 0x7f7ad8190240 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7acc004000 tx=0x7f7acc0040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:29:49.078 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.465+0000 7f7ad5ffb700 1 -- 172.21.15.47:0/3850474600 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7acc011840 con 0x7f7ad8071e80 2022-09-07T21:29:49.078 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.465+0000 7f7ad5ffb700 1 -- 172.21.15.47:0/3850474600 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f7acc01a430 con 0x7f7ad8071e80 2022-09-07T21:29:49.079 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.465+0000 7f7ad5ffb700 1 -- 172.21.15.47:0/3850474600 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7acc019530 con 0x7f7ad8071e80 2022-09-07T21:29:49.079 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.465+0000 7f7adde9f700 1 -- 172.21.15.47:0/3850474600 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad8190980 con 0x7f7ad8071e80 2022-09-07T21:29:49.079 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.465+0000 7f7adde9f700 1 -- 172.21.15.47:0/3850474600 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad818ce90 con 0x7f7ad8071e80 2022-09-07T21:29:49.079 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.465+0000 7f7ad5ffb700 1 -- 172.21.15.47:0/3850474600 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f7acc02a020 con 0x7f7ad8071e80 2022-09-07T21:29:49.079 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.465+0000 7f7ad5ffb700 1 -- 172.21.15.47:0/3850474600 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7acc02d440 con 0x7f7ad8071e80 2022-09-07T21:29:49.080 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.466+0000 7f7adde9f700 1 -- 172.21.15.47:0/3850474600 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad8060310 con 0x7f7ad8071e80 2022-09-07T21:29:49.080 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.469+0000 7f7ad5ffb700 1 -- 172.21.15.47:0/3850474600 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7f7acc01a5a0 con 0x7f7ad8071e80 2022-09-07T21:29:49.080 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.544+0000 7f7adde9f700 1 -- 172.21.15.47:0/3850474600 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f7ad81074e0 con 0x7f7ad8071e80 2022-09-07T21:29:49.081 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.546+0000 7f7ad5ffb700 1 -- 172.21.15.47:0/3850474600 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7acc01a8a0 con 0x7f7ad8071e80 2022-09-07T21:29:49.081 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.547+0000 7f7adde9f700 1 -- 172.21.15.47:0/3850474600 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ad8071e80 msgr2=0x7f7ad8190240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:29:49.081 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.547+0000 7f7adde9f700 1 --2- 172.21.15.47:0/3850474600 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ad8071e80 0x7f7ad8190240 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7acc004000 tx=0x7f7acc0040e0 comp rx=0 tx=0).stop 2022-09-07T21:29:49.081 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.547+0000 7f7adde9f700 1 -- 172.21.15.47:0/3850474600 shutdown_connections 2022-09-07T21:29:49.081 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.547+0000 7f7adde9f700 1 --2- 172.21.15.47:0/3850474600 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ad8071e80 0x7f7ad8190240 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:29:49.082 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.548+0000 7f7adde9f700 1 -- 172.21.15.47:0/3850474600 >> 172.21.15.47:0/3850474600 conn(0x7f7ad80fda80 msgr2=0x7f7ad80fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:29:49.082 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.548+0000 7f7adde9f700 1 -- 172.21.15.47:0/3850474600 shutdown_connections 2022-09-07T21:29:49.082 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:48.548+0000 7f7adde9f700 1 -- 172.21.15.47:0/3850474600 wait complete. 2022-09-07T21:29:49.082 INFO:teuthology.orchestra.run.smithi047.stdout:Wrote config to /etc/ceph/ceph.conf 2022-09-07T21:29:49.083 INFO:teuthology.orchestra.run.smithi047.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2022-09-07T21:29:49.083 INFO:teuthology.orchestra.run.smithi047.stdout:Creating mgr... 2022-09-07T21:29:49.083 INFO:teuthology.orchestra.run.smithi047.stdout:Verifying port 9283 ... 2022-09-07T21:29:49.083 INFO:teuthology.orchestra.run.smithi047.stdout:Verifying port 8765 ... 2022-09-07T21:29:49.480 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mgr.x 2022-09-07T21:29:49.481 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stderr Failed to reset failed state of unit ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mgr.x.service: Unit ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mgr.x.service not loaded. 2022-09-07T21:29:49.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:49 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3850474600' entity='client.admin' 2022-09-07T21:29:49.759 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-01b8396e-2ef4-11ed-8431-001a4aab830c.target.wants/ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@.service. 2022-09-07T21:29:50.153 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:49 smithi047 systemd[1]: Starting Ceph mgr.x for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:29:50.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:50 smithi047 podman[90320]: 2022-09-07T21:29:50.789 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2022-09-07T21:29:50.789 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stdout disabled 2022-09-07T21:29:50.811 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2022-09-07T21:29:50.811 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stdout inactive 2022-09-07T21:29:50.811 INFO:teuthology.orchestra.run.smithi047.stdout:firewalld.service is not enabled 2022-09-07T21:29:50.812 INFO:teuthology.orchestra.run.smithi047.stdout:Not possible to enable service . firewalld.service is not available 2022-09-07T21:29:50.835 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2022-09-07T21:29:50.835 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stdout disabled 2022-09-07T21:29:50.861 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2022-09-07T21:29:50.861 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stdout inactive 2022-09-07T21:29:50.862 INFO:teuthology.orchestra.run.smithi047.stdout:firewalld.service is not enabled 2022-09-07T21:29:50.862 INFO:teuthology.orchestra.run.smithi047.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2022-09-07T21:29:50.863 INFO:teuthology.orchestra.run.smithi047.stdout:Waiting for mgr to start... 2022-09-07T21:29:50.863 INFO:teuthology.orchestra.run.smithi047.stdout:Waiting for mgr... 2022-09-07T21:29:51.017 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:50 smithi047 bash[90320]: fe9cb52dcc9ed0c1511494cbe65132ac83ca09689fe7b5e8bd5e9d342cc1fffd 2022-09-07T21:29:51.017 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:50 smithi047 systemd[1]: Started Ceph mgr.x for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:29:51.017 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:50.888+0000 7f2e9c0e6080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-07T21:29:51.018 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:51.014+0000 7f2e9c0e6080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-07T21:29:51.291 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:51.143+0000 7f2e9c0e6080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-07T21:29:51.569 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:51.286+0000 7f2e9c0e6080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-07T21:29:51.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:51.566+0000 7f2e9c0e6080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-07T21:29:51.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:51 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2939707848' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-09-07T21:29:52.206 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:52.066+0000 7f2e9c0e6080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-07T21:29:52.270 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 2022-09-07T21:29:52.283 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout { 2022-09-07T21:29:52.283 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "fsid": "01b8396e-2ef4-11ed-8431-001a4aab830c", 2022-09-07T21:29:52.284 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "health": { 2022-09-07T21:29:52.284 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2022-09-07T21:29:52.284 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "checks": {}, 2022-09-07T21:29:52.284 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "mutes": [] 2022-09-07T21:29:52.284 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:52.285 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2022-09-07T21:29:52.285 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "quorum": [ 2022-09-07T21:29:52.285 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 0 2022-09-07T21:29:52.285 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout ], 2022-09-07T21:29:52.286 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2022-09-07T21:29:52.286 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "a" 2022-09-07T21:29:52.286 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout ], 2022-09-07T21:29:52.286 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2022-09-07T21:29:52.287 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "monmap": { 2022-09-07T21:29:52.287 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:29:52.287 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2022-09-07T21:29:52.287 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_mons": 1 2022-09-07T21:29:52.287 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:52.288 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "osdmap": { 2022-09-07T21:29:52.288 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:29:52.288 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2022-09-07T21:29:52.289 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2022-09-07T21:29:52.289 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2022-09-07T21:29:52.289 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2022-09-07T21:29:52.290 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2022-09-07T21:29:52.290 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2022-09-07T21:29:52.291 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:52.291 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "pgmap": { 2022-09-07T21:29:52.291 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2022-09-07T21:29:52.292 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2022-09-07T21:29:52.292 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2022-09-07T21:29:52.293 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2022-09-07T21:29:52.293 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2022-09-07T21:29:52.294 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2022-09-07T21:29:52.294 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2022-09-07T21:29:52.294 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2022-09-07T21:29:52.295 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:52.295 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "fsmap": { 2022-09-07T21:29:52.296 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:29:52.296 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "by_rank": [], 2022-09-07T21:29:52.297 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "up:standby": 0 2022-09-07T21:29:52.297 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:52.298 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "mgrmap": { 2022-09-07T21:29:52.298 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "available": false, 2022-09-07T21:29:52.298 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2022-09-07T21:29:52.299 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "modules": [ 2022-09-07T21:29:52.299 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "iostat", 2022-09-07T21:29:52.300 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "nfs", 2022-09-07T21:29:52.300 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "restful" 2022-09-07T21:29:52.300 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout ], 2022-09-07T21:29:52.301 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "services": {} 2022-09-07T21:29:52.301 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:52.302 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "servicemap": { 2022-09-07T21:29:52.302 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:29:52.302 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "modified": "2022-09-07T21:29:33.860024+0000", 2022-09-07T21:29:52.303 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "services": {} 2022-09-07T21:29:52.303 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:52.303 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "progress_events": {} 2022-09-07T21:29:52.303 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout } 2022-09-07T21:29:52.304 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.537+0000 7f01cdca5700 1 Processor -- start 2022-09-07T21:29:52.304 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.538+0000 7f01cdca5700 1 -- start start 2022-09-07T21:29:52.304 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.538+0000 7f01cdca5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01c8105860 0x7f01c8105c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:29:52.304 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.538+0000 7f01cdca5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c81061b0 con 0x7f01c8105860 2022-09-07T21:29:52.304 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.539+0000 7f01c77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01c8105860 0x7f01c8105c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:29:52.305 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.539+0000 7f01c77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01c8105860 0x7f01c8105c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50364/0 (socket says 172.21.15.47:50364) 2022-09-07T21:29:52.305 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.539+0000 7f01c77fe700 1 -- 172.21.15.47:0/4093281823 learned_addr learned my addr 172.21.15.47:0/4093281823 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:29:52.305 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.539+0000 7f01c77fe700 1 -- 172.21.15.47:0/4093281823 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c81062f0 con 0x7f01c8105860 2022-09-07T21:29:52.305 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.540+0000 7f01c77fe700 1 --2- 172.21.15.47:0/4093281823 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01c8105860 0x7f01c8105c70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f01b8009700 tx=0x7f01b8009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5284089f6ac0ce7 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:29:52.306 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.540+0000 7f01c67fc700 1 -- 172.21.15.47:0/4093281823 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01b8004160 con 0x7f01c8105860 2022-09-07T21:29:52.306 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.540+0000 7f01c67fc700 1 -- 172.21.15.47:0/4093281823 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f01b80042c0 con 0x7f01c8105860 2022-09-07T21:29:52.306 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.540+0000 7f01c67fc700 1 -- 172.21.15.47:0/4093281823 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01b80039d0 con 0x7f01c8105860 2022-09-07T21:29:52.306 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.541+0000 7f01cdca5700 1 -- 172.21.15.47:0/4093281823 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01c8105860 msgr2=0x7f01c8105c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:29:52.307 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.541+0000 7f01cdca5700 1 --2- 172.21.15.47:0/4093281823 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01c8105860 0x7f01c8105c70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f01b8009700 tx=0x7f01b8009a10 comp rx=0 tx=0).stop 2022-09-07T21:29:52.307 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.541+0000 7f01cdca5700 1 -- 172.21.15.47:0/4093281823 shutdown_connections 2022-09-07T21:29:52.307 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.541+0000 7f01cdca5700 1 --2- 172.21.15.47:0/4093281823 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01c8105860 0x7f01c8105c70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:29:52.307 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.541+0000 7f01cdca5700 1 -- 172.21.15.47:0/4093281823 >> 172.21.15.47:0/4093281823 conn(0x7f01c81013c0 msgr2=0x7f01c81037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:29:52.307 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.541+0000 7f01cdca5700 1 -- 172.21.15.47:0/4093281823 shutdown_connections 2022-09-07T21:29:52.308 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.541+0000 7f01cdca5700 1 -- 172.21.15.47:0/4093281823 wait complete. 2022-09-07T21:29:52.308 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.542+0000 7f01cdca5700 1 Processor -- start 2022-09-07T21:29:52.308 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.543+0000 7f01cdca5700 1 -- start start 2022-09-07T21:29:52.308 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.543+0000 7f01cdca5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01c8105860 0x7f01c8199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:29:52.309 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.543+0000 7f01cdca5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c8199790 con 0x7f01c8105860 2022-09-07T21:29:52.309 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.543+0000 7f01c77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01c8105860 0x7f01c8199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:29:52.309 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.543+0000 7f01c77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01c8105860 0x7f01c8199250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50378/0 (socket says 172.21.15.47:50378) 2022-09-07T21:29:52.309 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.543+0000 7f01c77fe700 1 -- 172.21.15.47:0/2939707848 learned_addr learned my addr 172.21.15.47:0/2939707848 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:29:52.309 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.544+0000 7f01c77fe700 1 -- 172.21.15.47:0/2939707848 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b80093e0 con 0x7f01c8105860 2022-09-07T21:29:52.310 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.544+0000 7f01c77fe700 1 --2- 172.21.15.47:0/2939707848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01c8105860 0x7f01c8199250 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f01b8003fb0 tx=0x7f01b8004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:29:52.310 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.544+0000 7f01c4ff9700 1 -- 172.21.15.47:0/2939707848 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01b80118c0 con 0x7f01c8105860 2022-09-07T21:29:52.310 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.544+0000 7f01c4ff9700 1 -- 172.21.15.47:0/2939707848 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f01b8011a20 con 0x7f01c8105860 2022-09-07T21:29:52.310 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.544+0000 7f01c4ff9700 1 -- 172.21.15.47:0/2939707848 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01b8019550 con 0x7f01c8105860 2022-09-07T21:29:52.311 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.544+0000 7f01cdca5700 1 -- 172.21.15.47:0/2939707848 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c8199990 con 0x7f01c8105860 2022-09-07T21:29:52.311 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.544+0000 7f01cdca5700 1 -- 172.21.15.47:0/2939707848 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c8199dc0 con 0x7f01c8105860 2022-09-07T21:29:52.311 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.545+0000 7f01c4ff9700 1 -- 172.21.15.47:0/2939707848 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f01b802a020 con 0x7f01c8105860 2022-09-07T21:29:52.311 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.545+0000 7f01c4ff9700 1 -- 172.21.15.47:0/2939707848 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f01b802d780 con 0x7f01c8105860 2022-09-07T21:29:52.312 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.546+0000 7f01cdca5700 1 -- 172.21.15.47:0/2939707848 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c8060310 con 0x7f01c8105860 2022-09-07T21:29:52.312 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.549+0000 7f01c4ff9700 1 -- 172.21.15.47:0/2939707848 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7f01b8011b90 con 0x7f01c8105860 2022-09-07T21:29:52.312 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.655+0000 7f01cdca5700 1 -- 172.21.15.47:0/2939707848 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f01c81076a0 con 0x7f01c8105860 2022-09-07T21:29:52.312 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.655+0000 7f01c4ff9700 1 -- 172.21.15.47:0/2939707848 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f01b8019780 con 0x7f01c8105860 2022-09-07T21:29:52.312 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.656+0000 7f01cdca5700 1 -- 172.21.15.47:0/2939707848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01c8105860 msgr2=0x7f01c8199250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:29:52.313 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.656+0000 7f01cdca5700 1 --2- 172.21.15.47:0/2939707848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01c8105860 0x7f01c8199250 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f01b8003fb0 tx=0x7f01b8004090 comp rx=0 tx=0).stop 2022-09-07T21:29:52.313 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.656+0000 7f01cdca5700 1 -- 172.21.15.47:0/2939707848 shutdown_connections 2022-09-07T21:29:52.313 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.656+0000 7f01cdca5700 1 --2- 172.21.15.47:0/2939707848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01c8105860 0x7f01c8199250 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:29:52.314 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.656+0000 7f01cdca5700 1 -- 172.21.15.47:0/2939707848 >> 172.21.15.47:0/2939707848 conn(0x7f01c81013c0 msgr2=0x7f01c8102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:29:52.314 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.656+0000 7f01cdca5700 1 -- 172.21.15.47:0/2939707848 shutdown_connections 2022-09-07T21:29:52.314 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:51.656+0000 7f01cdca5700 1 -- 172.21.15.47:0/2939707848 wait complete. 2022-09-07T21:29:52.315 INFO:teuthology.orchestra.run.smithi047.stdout:mgr not available, waiting (1/15)... 2022-09-07T21:29:52.533 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:52.203+0000 7f2e9c0e6080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-07T21:29:52.533 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:52.388+0000 7f2e9c0e6080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-07T21:29:52.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:52.531+0000 7f2e9c0e6080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-07T21:29:53.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:53.161+0000 7f2e9c0e6080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-07T21:29:54.403 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:54.075+0000 7f2e9c0e6080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-07T21:29:54.403 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:54.315+0000 7f2e9c0e6080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-07T21:29:55.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:55 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/4216790045' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-09-07T21:29:55.608 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 2022-09-07T21:29:55.608 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout { 2022-09-07T21:29:55.609 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "fsid": "01b8396e-2ef4-11ed-8431-001a4aab830c", 2022-09-07T21:29:55.609 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "health": { 2022-09-07T21:29:55.610 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2022-09-07T21:29:55.610 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "checks": {}, 2022-09-07T21:29:55.611 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "mutes": [] 2022-09-07T21:29:55.611 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:55.612 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2022-09-07T21:29:55.612 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "quorum": [ 2022-09-07T21:29:55.612 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 0 2022-09-07T21:29:55.613 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout ], 2022-09-07T21:29:55.613 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2022-09-07T21:29:55.614 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "a" 2022-09-07T21:29:55.614 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout ], 2022-09-07T21:29:55.615 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2022-09-07T21:29:55.615 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "monmap": { 2022-09-07T21:29:55.615 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:29:55.616 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2022-09-07T21:29:55.616 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_mons": 1 2022-09-07T21:29:55.617 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:55.617 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "osdmap": { 2022-09-07T21:29:55.618 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:29:55.618 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2022-09-07T21:29:55.619 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2022-09-07T21:29:55.619 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2022-09-07T21:29:55.620 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2022-09-07T21:29:55.620 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2022-09-07T21:29:55.620 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2022-09-07T21:29:55.621 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:55.621 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "pgmap": { 2022-09-07T21:29:55.622 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2022-09-07T21:29:55.622 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2022-09-07T21:29:55.623 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2022-09-07T21:29:55.623 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2022-09-07T21:29:55.624 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2022-09-07T21:29:55.624 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2022-09-07T21:29:55.624 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2022-09-07T21:29:55.625 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2022-09-07T21:29:55.625 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:55.626 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "fsmap": { 2022-09-07T21:29:55.626 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:29:55.627 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "by_rank": [], 2022-09-07T21:29:55.627 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "up:standby": 0 2022-09-07T21:29:55.628 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:55.628 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "mgrmap": { 2022-09-07T21:29:55.628 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "available": false, 2022-09-07T21:29:55.629 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2022-09-07T21:29:55.629 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "modules": [ 2022-09-07T21:29:55.630 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "iostat", 2022-09-07T21:29:55.630 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "nfs", 2022-09-07T21:29:55.631 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "restful" 2022-09-07T21:29:55.631 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout ], 2022-09-07T21:29:55.632 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "services": {} 2022-09-07T21:29:55.632 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:55.632 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "servicemap": { 2022-09-07T21:29:55.633 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:29:55.633 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "modified": "2022-09-07T21:29:33.860024+0000", 2022-09-07T21:29:55.634 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "services": {} 2022-09-07T21:29:55.634 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:55.635 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "progress_events": {} 2022-09-07T21:29:55.635 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout } 2022-09-07T21:29:55.635 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.971+0000 7f15c759e700 1 Processor -- start 2022-09-07T21:29:55.636 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.972+0000 7f15c759e700 1 -- start start 2022-09-07T21:29:55.636 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.972+0000 7f15c759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15c8103480 0x7f15c8103890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:29:55.637 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.972+0000 7f15c759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c8103dd0 con 0x7f15c8103480 2022-09-07T21:29:55.637 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.973+0000 7f15c659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15c8103480 0x7f15c8103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:29:55.638 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.973+0000 7f15c659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15c8103480 0x7f15c8103890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50388/0 (socket says 172.21.15.47:50388) 2022-09-07T21:29:55.638 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.973+0000 7f15c659c700 1 -- 172.21.15.47:0/1100650922 learned_addr learned my addr 172.21.15.47:0/1100650922 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:29:55.639 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.973+0000 7f15c659c700 1 -- 172.21.15.47:0/1100650922 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c8103f10 con 0x7f15c8103480 2022-09-07T21:29:55.639 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.973+0000 7f15c659c700 1 --2- 172.21.15.47:0/1100650922 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15c8103480 0x7f15c8103890 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009700 tx=0x7f15b8009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9977d58eaeee514d server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:29:55.639 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.974+0000 7f15c559a700 1 -- 172.21.15.47:0/1100650922 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15b8004160 con 0x7f15c8103480 2022-09-07T21:29:55.640 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.974+0000 7f15c559a700 1 -- 172.21.15.47:0/1100650922 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f15b80042c0 con 0x7f15c8103480 2022-09-07T21:29:55.640 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.974+0000 7f15c559a700 1 -- 172.21.15.47:0/1100650922 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15b80039d0 con 0x7f15c8103480 2022-09-07T21:29:55.641 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.974+0000 7f15c759e700 1 -- 172.21.15.47:0/1100650922 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15c8103480 msgr2=0x7f15c8103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:29:55.641 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.974+0000 7f15c759e700 1 --2- 172.21.15.47:0/1100650922 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15c8103480 0x7f15c8103890 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009700 tx=0x7f15b8009a10 comp rx=0 tx=0).stop 2022-09-07T21:29:55.642 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.975+0000 7f15c759e700 1 -- 172.21.15.47:0/1100650922 shutdown_connections 2022-09-07T21:29:55.642 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.975+0000 7f15c759e700 1 --2- 172.21.15.47:0/1100650922 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15c8103480 0x7f15c8103890 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:29:55.642 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.975+0000 7f15c759e700 1 -- 172.21.15.47:0/1100650922 >> 172.21.15.47:0/1100650922 conn(0x7f15c80feb70 msgr2=0x7f15c8100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:29:55.643 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.975+0000 7f15c759e700 1 -- 172.21.15.47:0/1100650922 shutdown_connections 2022-09-07T21:29:55.643 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.975+0000 7f15c759e700 1 -- 172.21.15.47:0/1100650922 wait complete. 2022-09-07T21:29:55.643 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.976+0000 7f15c759e700 1 Processor -- start 2022-09-07T21:29:55.643 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.976+0000 7f15c759e700 1 -- start start 2022-09-07T21:29:55.643 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.976+0000 7f15c759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15c8103480 0x7f15c8199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:29:55.644 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.976+0000 7f15c759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c8199580 con 0x7f15c8103480 2022-09-07T21:29:55.644 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.977+0000 7f15c659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15c8103480 0x7f15c8199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:29:55.644 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.977+0000 7f15c659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15c8103480 0x7f15c8199040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50390/0 (socket says 172.21.15.47:50390) 2022-09-07T21:29:55.644 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.977+0000 7f15c659c700 1 -- 172.21.15.47:0/4216790045 learned_addr learned my addr 172.21.15.47:0/4216790045 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:29:55.645 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.977+0000 7f15c659c700 1 -- 172.21.15.47:0/4216790045 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b80093e0 con 0x7f15c8103480 2022-09-07T21:29:55.645 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.978+0000 7f15c659c700 1 --2- 172.21.15.47:0/4216790045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15c8103480 0x7f15c8199040 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f15b8003fb0 tx=0x7f15b8004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:29:55.645 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.978+0000 7f15b77fe700 1 -- 172.21.15.47:0/4216790045 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15b80118c0 con 0x7f15c8103480 2022-09-07T21:29:55.645 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.978+0000 7f15b77fe700 1 -- 172.21.15.47:0/4216790045 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f15b8011a20 con 0x7f15c8103480 2022-09-07T21:29:55.645 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.978+0000 7f15b77fe700 1 -- 172.21.15.47:0/4216790045 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15b8019550 con 0x7f15c8103480 2022-09-07T21:29:55.646 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.978+0000 7f15c759e700 1 -- 172.21.15.47:0/4216790045 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c8199780 con 0x7f15c8103480 2022-09-07T21:29:55.646 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.978+0000 7f15c759e700 1 -- 172.21.15.47:0/4216790045 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c8199bb0 con 0x7f15c8103480 2022-09-07T21:29:55.646 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.979+0000 7f15b77fe700 1 -- 172.21.15.47:0/4216790045 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f15b802a020 con 0x7f15c8103480 2022-09-07T21:29:55.646 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.979+0000 7f15b77fe700 1 -- 172.21.15.47:0/4216790045 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f15b802d780 con 0x7f15c8103480 2022-09-07T21:29:55.647 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.979+0000 7f15c759e700 1 -- 172.21.15.47:0/4216790045 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c8060310 con 0x7f15c8103480 2022-09-07T21:29:55.647 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:54.982+0000 7f15b77fe700 1 -- 172.21.15.47:0/4216790045 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7f15b8011b90 con 0x7f15c8103480 2022-09-07T21:29:55.647 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:55.059+0000 7f15c759e700 1 -- 172.21.15.47:0/4216790045 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f15c819c9f0 con 0x7f15c8103480 2022-09-07T21:29:55.647 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:55.060+0000 7f15b77fe700 1 -- 172.21.15.47:0/4216790045 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f15b8019780 con 0x7f15c8103480 2022-09-07T21:29:55.647 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:55.061+0000 7f15c759e700 1 -- 172.21.15.47:0/4216790045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15c8103480 msgr2=0x7f15c8199040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:29:55.648 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:55.061+0000 7f15c759e700 1 --2- 172.21.15.47:0/4216790045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15c8103480 0x7f15c8199040 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f15b8003fb0 tx=0x7f15b8004090 comp rx=0 tx=0).stop 2022-09-07T21:29:55.648 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:55.061+0000 7f15c759e700 1 -- 172.21.15.47:0/4216790045 shutdown_connections 2022-09-07T21:29:55.648 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:55.061+0000 7f15c759e700 1 --2- 172.21.15.47:0/4216790045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15c8103480 0x7f15c8199040 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:29:55.648 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:55.061+0000 7f15c759e700 1 -- 172.21.15.47:0/4216790045 >> 172.21.15.47:0/4216790045 conn(0x7f15c80feb70 msgr2=0x7f15c80ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:29:55.649 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:55.061+0000 7f15c759e700 1 -- 172.21.15.47:0/4216790045 shutdown_connections 2022-09-07T21:29:55.649 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:55.062+0000 7f15c759e700 1 -- 172.21.15.47:0/4216790045 wait complete. 2022-09-07T21:29:55.649 INFO:teuthology.orchestra.run.smithi047.stdout:mgr not available, waiting (2/15)... 2022-09-07T21:29:55.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:55.471+0000 7f2e9c0e6080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:29:55.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:55.630+0000 7f2e9c0e6080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-07T21:29:56.239 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:55.918+0000 7f2e9c0e6080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:29:56.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:56.236+0000 7f2e9c0e6080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-07T21:29:56.654 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:56.476+0000 7f2e9c0e6080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-07T21:29:57.403 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:57.089+0000 7f2e9c0e6080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-07T21:29:58.153 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:58.072+0000 7f2e9c0e6080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-07T21:29:58.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:58.523+0000 7f2e9c0e6080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-07T21:29:58.903 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 2022-09-07T21:29:58.904 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout { 2022-09-07T21:29:58.904 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "fsid": "01b8396e-2ef4-11ed-8431-001a4aab830c", 2022-09-07T21:29:58.904 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "health": { 2022-09-07T21:29:58.905 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2022-09-07T21:29:58.905 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "checks": {}, 2022-09-07T21:29:58.905 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "mutes": [] 2022-09-07T21:29:58.905 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:58.905 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2022-09-07T21:29:58.906 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "quorum": [ 2022-09-07T21:29:58.906 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 0 2022-09-07T21:29:58.906 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout ], 2022-09-07T21:29:58.906 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2022-09-07T21:29:58.907 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "a" 2022-09-07T21:29:58.907 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout ], 2022-09-07T21:29:58.907 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2022-09-07T21:29:58.907 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "monmap": { 2022-09-07T21:29:58.908 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:29:58.908 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2022-09-07T21:29:58.908 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_mons": 1 2022-09-07T21:29:58.908 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:58.909 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "osdmap": { 2022-09-07T21:29:58.909 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:29:58.909 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2022-09-07T21:29:58.909 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2022-09-07T21:29:58.909 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2022-09-07T21:29:58.922 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2022-09-07T21:29:58.922 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2022-09-07T21:29:58.922 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2022-09-07T21:29:58.923 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:58.923 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "pgmap": { 2022-09-07T21:29:58.923 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2022-09-07T21:29:58.923 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2022-09-07T21:29:58.924 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2022-09-07T21:29:58.924 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2022-09-07T21:29:58.924 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2022-09-07T21:29:58.925 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2022-09-07T21:29:58.926 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2022-09-07T21:29:58.926 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2022-09-07T21:29:58.927 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:58.927 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "fsmap": { 2022-09-07T21:29:58.928 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:29:58.928 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "by_rank": [], 2022-09-07T21:29:58.928 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "up:standby": 0 2022-09-07T21:29:58.929 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:58.929 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "mgrmap": { 2022-09-07T21:29:58.929 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "available": false, 2022-09-07T21:29:58.930 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2022-09-07T21:29:58.930 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "modules": [ 2022-09-07T21:29:58.931 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "iostat", 2022-09-07T21:29:58.931 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "nfs", 2022-09-07T21:29:58.931 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "restful" 2022-09-07T21:29:58.932 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout ], 2022-09-07T21:29:58.932 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "services": {} 2022-09-07T21:29:58.932 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:58.932 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "servicemap": { 2022-09-07T21:29:58.933 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:29:58.933 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "modified": "2022-09-07T21:29:33.860024+0000", 2022-09-07T21:29:58.933 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "services": {} 2022-09-07T21:29:58.934 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:29:58.934 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "progress_events": {} 2022-09-07T21:29:58.934 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout } 2022-09-07T21:29:58.934 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.192+0000 7fa5c44d3700 1 Processor -- start 2022-09-07T21:29:58.935 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.192+0000 7fa5c44d3700 1 -- start start 2022-09-07T21:29:58.935 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.193+0000 7fa5c44d3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5bc103660 0x7fa5bc103a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:29:58.935 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.193+0000 7fa5c44d3700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc103fb0 con 0x7fa5bc103660 2022-09-07T21:29:58.935 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.193+0000 7fa5c226f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5bc103660 0x7fa5bc103a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:29:58.936 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.193+0000 7fa5c226f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5bc103660 0x7fa5bc103a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50400/0 (socket says 172.21.15.47:50400) 2022-09-07T21:29:58.936 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.193+0000 7fa5c226f700 1 -- 172.21.15.47:0/2227573402 learned_addr learned my addr 172.21.15.47:0/2227573402 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:29:58.936 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.194+0000 7fa5c226f700 1 -- 172.21.15.47:0/2227573402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc1040f0 con 0x7fa5bc103660 2022-09-07T21:29:58.936 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.194+0000 7fa5c226f700 1 --2- 172.21.15.47:0/2227573402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5bc103660 0x7fa5bc103a70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009700 tx=0x7fa5b8009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d66e5e987652ce06 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:29:58.936 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.194+0000 7fa5c126d700 1 -- 172.21.15.47:0/2227573402 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5b8004160 con 0x7fa5bc103660 2022-09-07T21:29:58.937 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.194+0000 7fa5c126d700 1 -- 172.21.15.47:0/2227573402 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fa5b80042c0 con 0x7fa5bc103660 2022-09-07T21:29:58.937 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.194+0000 7fa5c126d700 1 -- 172.21.15.47:0/2227573402 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5b80039d0 con 0x7fa5bc103660 2022-09-07T21:29:58.937 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.195+0000 7fa5c44d3700 1 -- 172.21.15.47:0/2227573402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5bc103660 msgr2=0x7fa5bc103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:29:58.937 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.195+0000 7fa5c44d3700 1 --2- 172.21.15.47:0/2227573402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5bc103660 0x7fa5bc103a70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009700 tx=0x7fa5b8009a10 comp rx=0 tx=0).stop 2022-09-07T21:29:58.938 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.195+0000 7fa5c44d3700 1 -- 172.21.15.47:0/2227573402 shutdown_connections 2022-09-07T21:29:58.938 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.195+0000 7fa5c44d3700 1 --2- 172.21.15.47:0/2227573402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5bc103660 0x7fa5bc103a70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:29:58.938 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.195+0000 7fa5c44d3700 1 -- 172.21.15.47:0/2227573402 >> 172.21.15.47:0/2227573402 conn(0x7fa5bc0fed50 msgr2=0x7fa5bc101180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:29:58.938 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.195+0000 7fa5c44d3700 1 -- 172.21.15.47:0/2227573402 shutdown_connections 2022-09-07T21:29:58.938 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.196+0000 7fa5c44d3700 1 -- 172.21.15.47:0/2227573402 wait complete. 2022-09-07T21:29:58.939 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.197+0000 7fa5c44d3700 1 Processor -- start 2022-09-07T21:29:58.939 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.198+0000 7fa5c44d3700 1 -- start start 2022-09-07T21:29:58.939 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.198+0000 7fa5c44d3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5bc103660 0x7fa5bc199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:29:58.939 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.198+0000 7fa5c44d3700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc199780 con 0x7fa5bc103660 2022-09-07T21:29:58.940 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.198+0000 7fa5c226f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5bc103660 0x7fa5bc199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:29:58.940 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.198+0000 7fa5c226f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5bc103660 0x7fa5bc199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50406/0 (socket says 172.21.15.47:50406) 2022-09-07T21:29:58.940 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.198+0000 7fa5c226f700 1 -- 172.21.15.47:0/1331623754 learned_addr learned my addr 172.21.15.47:0/1331623754 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:29:58.940 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.199+0000 7fa5c226f700 1 -- 172.21.15.47:0/1331623754 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b80093e0 con 0x7fa5bc103660 2022-09-07T21:29:58.940 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.199+0000 7fa5c226f700 1 --2- 172.21.15.47:0/1331623754 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5bc103660 0x7fa5bc199240 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8003fb0 tx=0x7fa5b8004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:29:58.941 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.200+0000 7fa5ab7fe700 1 -- 172.21.15.47:0/1331623754 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5b80118c0 con 0x7fa5bc103660 2022-09-07T21:29:58.941 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.200+0000 7fa5c44d3700 1 -- 172.21.15.47:0/1331623754 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5bc199980 con 0x7fa5bc103660 2022-09-07T21:29:58.941 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.200+0000 7fa5ab7fe700 1 -- 172.21.15.47:0/1331623754 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fa5b8011a20 con 0x7fa5bc103660 2022-09-07T21:29:58.941 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.200+0000 7fa5ab7fe700 1 -- 172.21.15.47:0/1331623754 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5b8019550 con 0x7fa5bc103660 2022-09-07T21:29:58.942 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.200+0000 7fa5c44d3700 1 -- 172.21.15.47:0/1331623754 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5bc199db0 con 0x7fa5bc103660 2022-09-07T21:29:58.942 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.200+0000 7fa5ab7fe700 1 -- 172.21.15.47:0/1331623754 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fa5b802a020 con 0x7fa5bc103660 2022-09-07T21:29:58.942 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.201+0000 7fa5ab7fe700 1 -- 172.21.15.47:0/1331623754 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa5b802d440 con 0x7fa5bc103660 2022-09-07T21:29:58.942 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.201+0000 7fa5c44d3700 1 -- 172.21.15.47:0/1331623754 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5bc060310 con 0x7fa5bc103660 2022-09-07T21:29:58.943 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.204+0000 7fa5ab7fe700 1 -- 172.21.15.47:0/1331623754 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7fa5b8011b90 con 0x7fa5bc103660 2022-09-07T21:29:58.943 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.282+0000 7fa5c44d3700 1 -- 172.21.15.47:0/1331623754 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa5bc19cbf0 con 0x7fa5bc103660 2022-09-07T21:29:58.943 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.282+0000 7fa5ab7fe700 1 -- 172.21.15.47:0/1331623754 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1241 (secure 0 0 0) 0x7fa5b801a770 con 0x7fa5bc103660 2022-09-07T21:29:58.943 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.283+0000 7fa5c44d3700 1 -- 172.21.15.47:0/1331623754 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5bc103660 msgr2=0x7fa5bc199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:29:58.943 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.283+0000 7fa5c44d3700 1 --2- 172.21.15.47:0/1331623754 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5bc103660 0x7fa5bc199240 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8003fb0 tx=0x7fa5b8004090 comp rx=0 tx=0).stop 2022-09-07T21:29:58.944 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.284+0000 7fa5c44d3700 1 -- 172.21.15.47:0/1331623754 shutdown_connections 2022-09-07T21:29:58.944 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.284+0000 7fa5c44d3700 1 --2- 172.21.15.47:0/1331623754 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5bc103660 0x7fa5bc199240 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:29:58.944 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.284+0000 7fa5c44d3700 1 -- 172.21.15.47:0/1331623754 >> 172.21.15.47:0/1331623754 conn(0x7fa5bc0fed50 msgr2=0x7fa5bc0ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:29:58.944 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.284+0000 7fa5c44d3700 1 -- 172.21.15.47:0/1331623754 shutdown_connections 2022-09-07T21:29:58.945 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:29:58.284+0000 7fa5c44d3700 1 -- 172.21.15.47:0/1331623754 wait complete. 2022-09-07T21:29:58.945 INFO:teuthology.orchestra.run.smithi047.stdout:mgr not available, waiting (3/15)... 2022-09-07T21:29:59.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:29:58 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1331623754' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-09-07T21:29:59.153 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:58.860+0000 7f2e9c0e6080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-07T21:29:59.154 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:59.059+0000 7f2e9c0e6080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-07T21:29:59.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:29:59.433+0000 7f2e9c0e6080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-07T21:30:00.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:00.491+0000 7f2e9c0e6080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-07T21:30:00.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:00 smithi047 ceph-mon[90087]: Activating manager daemon x 2022-09-07T21:30:00.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:00 smithi047 ceph-mon[90087]: mgrmap e2: x(active, starting, since 0.00450085s) 2022-09-07T21:30:00.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:00 smithi047 ceph-mon[90087]: from='mgr.14100 172.21.15.47:0/4082691859' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-09-07T21:30:00.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:00 smithi047 ceph-mon[90087]: from='mgr.14100 172.21.15.47:0/4082691859' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-09-07T21:30:00.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:00 smithi047 ceph-mon[90087]: from='mgr.14100 172.21.15.47:0/4082691859' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-09-07T21:30:00.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:00 smithi047 ceph-mon[90087]: from='mgr.14100 172.21.15.47:0/4082691859' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:30:00.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:00 smithi047 ceph-mon[90087]: from='mgr.14100 172.21.15.47:0/4082691859' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-09-07T21:30:00.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:00 smithi047 ceph-mon[90087]: Manager daemon x is now available 2022-09-07T21:30:00.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:00 smithi047 ceph-mon[90087]: from='mgr.14100 172.21.15.47:0/4082691859' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:30:00.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:00 smithi047 ceph-mon[90087]: from='mgr.14100 172.21.15.47:0/4082691859' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:30:00.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:00 smithi047 ceph-mon[90087]: from='mgr.14100 172.21.15.47:0/4082691859' entity='mgr.x' 2022-09-07T21:30:00.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:00 smithi047 ceph-mon[90087]: from='mgr.14100 172.21.15.47:0/4082691859' entity='mgr.x' 2022-09-07T21:30:00.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:00 smithi047 ceph-mon[90087]: from='mgr.14100 172.21.15.47:0/4082691859' entity='mgr.x' 2022-09-07T21:30:02.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:02 smithi047 ceph-mon[90087]: mgrmap e3: x(active, since 1.00929s) 2022-09-07T21:30:02.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:02 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/52562789' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-09-07T21:30:03.029 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 2022-09-07T21:30:03.047 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout { 2022-09-07T21:30:03.047 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "fsid": "01b8396e-2ef4-11ed-8431-001a4aab830c", 2022-09-07T21:30:03.047 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "health": { 2022-09-07T21:30:03.047 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2022-09-07T21:30:03.048 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "checks": {}, 2022-09-07T21:30:03.048 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "mutes": [] 2022-09-07T21:30:03.048 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:30:03.049 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2022-09-07T21:30:03.049 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "quorum": [ 2022-09-07T21:30:03.049 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 0 2022-09-07T21:30:03.049 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout ], 2022-09-07T21:30:03.050 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2022-09-07T21:30:03.050 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "a" 2022-09-07T21:30:03.051 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout ], 2022-09-07T21:30:03.051 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2022-09-07T21:30:03.051 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "monmap": { 2022-09-07T21:30:03.051 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:30:03.052 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2022-09-07T21:30:03.052 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_mons": 1 2022-09-07T21:30:03.052 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:30:03.052 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "osdmap": { 2022-09-07T21:30:03.053 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:30:03.053 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2022-09-07T21:30:03.053 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2022-09-07T21:30:03.053 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2022-09-07T21:30:03.054 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2022-09-07T21:30:03.054 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2022-09-07T21:30:03.054 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2022-09-07T21:30:03.055 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:30:03.055 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "pgmap": { 2022-09-07T21:30:03.055 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2022-09-07T21:30:03.056 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2022-09-07T21:30:03.056 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2022-09-07T21:30:03.057 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2022-09-07T21:30:03.057 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2022-09-07T21:30:03.058 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2022-09-07T21:30:03.058 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2022-09-07T21:30:03.058 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2022-09-07T21:30:03.059 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:30:03.059 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "fsmap": { 2022-09-07T21:30:03.060 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:30:03.060 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "by_rank": [], 2022-09-07T21:30:03.060 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "up:standby": 0 2022-09-07T21:30:03.061 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:30:03.061 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "mgrmap": { 2022-09-07T21:30:03.062 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "available": true, 2022-09-07T21:30:03.062 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2022-09-07T21:30:03.062 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "modules": [ 2022-09-07T21:30:03.063 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "iostat", 2022-09-07T21:30:03.063 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "nfs", 2022-09-07T21:30:03.064 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "restful" 2022-09-07T21:30:03.064 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout ], 2022-09-07T21:30:03.064 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "services": {} 2022-09-07T21:30:03.064 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:30:03.064 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "servicemap": { 2022-09-07T21:30:03.065 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-07T21:30:03.065 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "modified": "2022-09-07T21:29:33.860024+0000", 2022-09-07T21:30:03.065 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "services": {} 2022-09-07T21:30:03.065 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout }, 2022-09-07T21:30:03.066 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "progress_events": {} 2022-09-07T21:30:03.066 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout } 2022-09-07T21:30:03.066 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.586+0000 7fa7b704b700 1 Processor -- start 2022-09-07T21:30:03.066 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.587+0000 7fa7b704b700 1 -- start start 2022-09-07T21:30:03.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.587+0000 7fa7b704b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7b0071e80 0x7fa7b0106270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:03.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.587+0000 7fa7b704b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b01067b0 con 0x7fa7b0071e80 2022-09-07T21:30:03.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.588+0000 7fa7b4de7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7b0071e80 0x7fa7b0106270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:03.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.588+0000 7fa7b4de7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7b0071e80 0x7fa7b0106270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:37418/0 (socket says 172.21.15.47:37418) 2022-09-07T21:30:03.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.588+0000 7fa7b4de7700 1 -- 172.21.15.47:0/678342016 learned_addr learned my addr 172.21.15.47:0/678342016 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:03.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.588+0000 7fa7b4de7700 1 -- 172.21.15.47:0/678342016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b01068f0 con 0x7fa7b0071e80 2022-09-07T21:30:03.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.588+0000 7fa7b4de7700 1 --2- 172.21.15.47:0/678342016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7b0071e80 0x7fa7b0106270 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4009700 tx=0x7fa7a4009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c701c7356f8152e server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:03.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.589+0000 7fa7af7fe700 1 -- 172.21.15.47:0/678342016 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7a4004160 con 0x7fa7b0071e80 2022-09-07T21:30:03.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.589+0000 7fa7af7fe700 1 -- 172.21.15.47:0/678342016 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fa7a40042c0 con 0x7fa7b0071e80 2022-09-07T21:30:03.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.589+0000 7fa7af7fe700 1 -- 172.21.15.47:0/678342016 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7a40039d0 con 0x7fa7b0071e80 2022-09-07T21:30:03.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.589+0000 7fa7b704b700 1 -- 172.21.15.47:0/678342016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7b0071e80 msgr2=0x7fa7b0106270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:03.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.589+0000 7fa7b704b700 1 --2- 172.21.15.47:0/678342016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7b0071e80 0x7fa7b0106270 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4009700 tx=0x7fa7a4009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:03.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.589+0000 7fa7b704b700 1 -- 172.21.15.47:0/678342016 shutdown_connections 2022-09-07T21:30:03.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.589+0000 7fa7b704b700 1 --2- 172.21.15.47:0/678342016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7b0071e80 0x7fa7b0106270 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:03.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.589+0000 7fa7b704b700 1 -- 172.21.15.47:0/678342016 >> 172.21.15.47:0/678342016 conn(0x7fa7b00fdc20 msgr2=0x7fa7b0100050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:03.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.590+0000 7fa7b704b700 1 -- 172.21.15.47:0/678342016 shutdown_connections 2022-09-07T21:30:03.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.590+0000 7fa7b704b700 1 -- 172.21.15.47:0/678342016 wait complete. 2022-09-07T21:30:03.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.591+0000 7fa7b704b700 1 Processor -- start 2022-09-07T21:30:03.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.591+0000 7fa7b704b700 1 -- start start 2022-09-07T21:30:03.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.591+0000 7fa7b704b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7b0071e80 0x7fa7b0199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:03.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.591+0000 7fa7b704b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b0199780 con 0x7fa7b0071e80 2022-09-07T21:30:03.072 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.591+0000 7fa7b4de7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7b0071e80 0x7fa7b0199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:03.072 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.591+0000 7fa7b4de7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7b0071e80 0x7fa7b0199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:37426/0 (socket says 172.21.15.47:37426) 2022-09-07T21:30:03.072 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.591+0000 7fa7b4de7700 1 -- 172.21.15.47:0/52562789 learned_addr learned my addr 172.21.15.47:0/52562789 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:03.072 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.592+0000 7fa7b4de7700 1 -- 172.21.15.47:0/52562789 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a40093e0 con 0x7fa7b0071e80 2022-09-07T21:30:03.072 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.592+0000 7fa7b4de7700 1 --2- 172.21.15.47:0/52562789 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7b0071e80 0x7fa7b0199240 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4004000 tx=0x7fa7a40040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:03.073 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.592+0000 7fa7adffb700 1 -- 172.21.15.47:0/52562789 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7a4011840 con 0x7fa7b0071e80 2022-09-07T21:30:03.073 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.592+0000 7fa7adffb700 1 -- 172.21.15.47:0/52562789 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fa7a401a430 con 0x7fa7b0071e80 2022-09-07T21:30:03.073 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.592+0000 7fa7b704b700 1 -- 172.21.15.47:0/52562789 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b0199980 con 0x7fa7b0071e80 2022-09-07T21:30:03.073 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.592+0000 7fa7adffb700 1 -- 172.21.15.47:0/52562789 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7a4019530 con 0x7fa7b0071e80 2022-09-07T21:30:03.074 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.593+0000 7fa7b704b700 1 -- 172.21.15.47:0/52562789 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b0199db0 con 0x7fa7b0071e80 2022-09-07T21:30:03.074 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.593+0000 7fa7adffb700 1 -- 172.21.15.47:0/52562789 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 3) v1 ==== 42599+0+0 (secure 0 0 0) 0x7fa7a402a020 con 0x7fa7b0071e80 2022-09-07T21:30:03.074 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.593+0000 7fa7b704b700 1 -- 172.21.15.47:0/52562789 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b0107670 con 0x7fa7b0071e80 2022-09-07T21:30:03.074 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.594+0000 7fa7adffb700 1 --2- 172.21.15.47:0/52562789 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fa794035810 0x7fa794037cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:03.074 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.594+0000 7fa7adffb700 1 -- 172.21.15.47:0/52562789 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa7a4048860 con 0x7fa7b0071e80 2022-09-07T21:30:03.075 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.594+0000 7fa7affff700 1 --2- 172.21.15.47:0/52562789 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fa794035810 0x7fa794037cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:03.075 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.595+0000 7fa7affff700 1 --2- 172.21.15.47:0/52562789 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fa794035810 0x7fa794037cc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa79c009750 tx=0x7fa79c006cc0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:03.075 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.601+0000 7fa7adffb700 1 -- 172.21.15.47:0/52562789 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fa7a40189a0 con 0x7fa7b0071e80 2022-09-07T21:30:03.075 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.896+0000 7fa7b704b700 1 -- 172.21.15.47:0/52562789 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa7b019cbf0 con 0x7fa7b0071e80 2022-09-07T21:30:03.075 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.896+0000 7fa7adffb700 1 -- 172.21.15.47:0/52562789 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7fa7a40187c0 con 0x7fa7b0071e80 2022-09-07T21:30:03.076 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.898+0000 7fa7b704b700 1 -- 172.21.15.47:0/52562789 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fa794035810 msgr2=0x7fa794037cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:03.076 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.898+0000 7fa7b704b700 1 --2- 172.21.15.47:0/52562789 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fa794035810 0x7fa794037cc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa79c009750 tx=0x7fa79c006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:03.076 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.898+0000 7fa7b704b700 1 -- 172.21.15.47:0/52562789 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7b0071e80 msgr2=0x7fa7b0199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:03.076 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.898+0000 7fa7b704b700 1 --2- 172.21.15.47:0/52562789 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7b0071e80 0x7fa7b0199240 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4004000 tx=0x7fa7a40040e0 comp rx=0 tx=0).stop 2022-09-07T21:30:03.077 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.899+0000 7fa7b704b700 1 -- 172.21.15.47:0/52562789 shutdown_connections 2022-09-07T21:30:03.077 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.899+0000 7fa7b704b700 1 --2- 172.21.15.47:0/52562789 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fa794035810 0x7fa794037cc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:03.077 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.899+0000 7fa7b704b700 1 --2- 172.21.15.47:0/52562789 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7b0071e80 0x7fa7b0199240 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:03.077 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.899+0000 7fa7b704b700 1 -- 172.21.15.47:0/52562789 >> 172.21.15.47:0/52562789 conn(0x7fa7b00fdc20 msgr2=0x7fa7b00fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:03.077 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.899+0000 7fa7b704b700 1 -- 172.21.15.47:0/52562789 shutdown_connections 2022-09-07T21:30:03.078 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:01.899+0000 7fa7b704b700 1 -- 172.21.15.47:0/52562789 wait complete. 2022-09-07T21:30:03.078 INFO:teuthology.orchestra.run.smithi047.stdout:mgr is available 2022-09-07T21:30:03.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:03 smithi047 ceph-mon[90087]: mgrmap e4: x(active, since 2s) 2022-09-07T21:30:04.544 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 2022-09-07T21:30:04.549 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout [global] 2022-09-07T21:30:04.549 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout fsid = 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:30:04.549 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2022-09-07T21:30:04.550 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2022-09-07T21:30:04.550 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2022-09-07T21:30:04.550 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2022-09-07T21:30:04.551 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 2022-09-07T21:30:04.551 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout [mgr] 2022-09-07T21:30:04.551 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2022-09-07T21:30:04.551 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 2022-09-07T21:30:04.552 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout [osd] 2022-09-07T21:30:04.552 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2022-09-07T21:30:04.552 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2022-09-07T21:30:04.552 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2022-09-07T21:30:04.552 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2022-09-07T21:30:04.553 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2022-09-07T21:30:04.553 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2022-09-07T21:30:04.553 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.702+0000 7f5c93ae7700 1 Processor -- start 2022-09-07T21:30:04.553 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.702+0000 7f5c93ae7700 1 -- start start 2022-09-07T21:30:04.554 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.702+0000 7f5c93ae7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c8c103660 0x7f5c8c103a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:04.554 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.702+0000 7f5c93ae7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c103fb0 con 0x7f5c8c103660 2022-09-07T21:30:04.554 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.703+0000 7f5c91883700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c8c103660 0x7f5c8c103a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:04.554 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.703+0000 7f5c91883700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c8c103660 0x7f5c8c103a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:37428/0 (socket says 172.21.15.47:37428) 2022-09-07T21:30:04.555 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.703+0000 7f5c91883700 1 -- 172.21.15.47:0/760756161 learned_addr learned my addr 172.21.15.47:0/760756161 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:04.555 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.703+0000 7f5c91883700 1 -- 172.21.15.47:0/760756161 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c1040f0 con 0x7f5c8c103660 2022-09-07T21:30:04.562 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.703+0000 7f5c91883700 1 --2- 172.21.15.47:0/760756161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c8c103660 0x7f5c8c103a70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009700 tx=0x7f5c88009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6478838c362859c5 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:04.562 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.704+0000 7f5c90881700 1 -- 172.21.15.47:0/760756161 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c88004160 con 0x7f5c8c103660 2022-09-07T21:30:04.563 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.704+0000 7f5c90881700 1 -- 172.21.15.47:0/760756161 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f5c880042c0 con 0x7f5c8c103660 2022-09-07T21:30:04.563 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.704+0000 7f5c90881700 1 -- 172.21.15.47:0/760756161 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c880039d0 con 0x7f5c8c103660 2022-09-07T21:30:04.563 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.704+0000 7f5c93ae7700 1 -- 172.21.15.47:0/760756161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c8c103660 msgr2=0x7f5c8c103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:04.564 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.704+0000 7f5c93ae7700 1 --2- 172.21.15.47:0/760756161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c8c103660 0x7f5c8c103a70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009700 tx=0x7f5c88009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:04.564 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.704+0000 7f5c93ae7700 1 -- 172.21.15.47:0/760756161 shutdown_connections 2022-09-07T21:30:04.564 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.704+0000 7f5c93ae7700 1 --2- 172.21.15.47:0/760756161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c8c103660 0x7f5c8c103a70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:04.564 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.705+0000 7f5c93ae7700 1 -- 172.21.15.47:0/760756161 >> 172.21.15.47:0/760756161 conn(0x7f5c8c0fed50 msgr2=0x7f5c8c101180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:04.565 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.705+0000 7f5c93ae7700 1 -- 172.21.15.47:0/760756161 shutdown_connections 2022-09-07T21:30:04.565 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.705+0000 7f5c93ae7700 1 -- 172.21.15.47:0/760756161 wait complete. 2022-09-07T21:30:04.565 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.706+0000 7f5c93ae7700 1 Processor -- start 2022-09-07T21:30:04.566 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.706+0000 7f5c93ae7700 1 -- start start 2022-09-07T21:30:04.566 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.706+0000 7f5c93ae7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c8c103660 0x7f5c8c199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:04.566 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.706+0000 7f5c93ae7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c199750 con 0x7f5c8c103660 2022-09-07T21:30:04.567 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.707+0000 7f5c91883700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c8c103660 0x7f5c8c199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:04.567 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.707+0000 7f5c91883700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c8c103660 0x7f5c8c199210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:37438/0 (socket says 172.21.15.47:37438) 2022-09-07T21:30:04.567 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.707+0000 7f5c91883700 1 -- 172.21.15.47:0/1263167813 learned_addr learned my addr 172.21.15.47:0/1263167813 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:04.567 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.707+0000 7f5c91883700 1 -- 172.21.15.47:0/1263167813 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c880093e0 con 0x7f5c8c103660 2022-09-07T21:30:04.568 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.707+0000 7f5c91883700 1 --2- 172.21.15.47:0/1263167813 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c8c103660 0x7f5c8c199210 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5c88003fb0 tx=0x7f5c88004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:04.568 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.708+0000 7f5c7affd700 1 -- 172.21.15.47:0/1263167813 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c880118c0 con 0x7f5c8c103660 2022-09-07T21:30:04.568 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.708+0000 7f5c93ae7700 1 -- 172.21.15.47:0/1263167813 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8c199950 con 0x7f5c8c103660 2022-09-07T21:30:04.568 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.708+0000 7f5c7affd700 1 -- 172.21.15.47:0/1263167813 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f5c88011a20 con 0x7f5c8c103660 2022-09-07T21:30:04.568 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.708+0000 7f5c7affd700 1 -- 172.21.15.47:0/1263167813 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c88019550 con 0x7f5c8c103660 2022-09-07T21:30:04.569 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.708+0000 7f5c93ae7700 1 -- 172.21.15.47:0/1263167813 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c199d80 con 0x7f5c8c103660 2022-09-07T21:30:04.569 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.709+0000 7f5c7affd700 1 -- 172.21.15.47:0/1263167813 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 4) v1 ==== 42679+0+0 (secure 0 0 0) 0x7f5c8802a020 con 0x7f5c8c103660 2022-09-07T21:30:04.569 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.709+0000 7f5c93ae7700 1 -- 172.21.15.47:0/1263167813 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8c1054a0 con 0x7f5c8c103660 2022-09-07T21:30:04.569 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.709+0000 7f5c7affd700 1 --2- 172.21.15.47:0/1263167813 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7f5c74035830 0x7f5c74037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:04.570 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.709+0000 7f5c7affd700 1 -- 172.21.15.47:0/1263167813 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5c880489b0 con 0x7f5c8c103660 2022-09-07T21:30:04.570 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.710+0000 7f5c91082700 1 --2- 172.21.15.47:0/1263167813 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7f5c74035830 0x7f5c74037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:04.570 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.710+0000 7f5c91082700 1 --2- 172.21.15.47:0/1263167813 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7f5c74035830 0x7f5c74037ce0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c009750 tx=0x7f5c7c006cc0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:04.570 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.715+0000 7f5c7affd700 1 -- 172.21.15.47:0/1263167813 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f5c88018b40 con 0x7f5c8c103660 2022-09-07T21:30:04.571 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.931+0000 7f5c93ae7700 1 -- 172.21.15.47:0/1263167813 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f5c8c19cbc0 con 0x7f5c8c103660 2022-09-07T21:30:04.571 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.932+0000 7f5c7affd700 1 -- 172.21.15.47:0/1263167813 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+398 (secure 0 0 0) 0x7f5c88018b40 con 0x7f5c8c103660 2022-09-07T21:30:04.571 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.934+0000 7f5c93ae7700 1 -- 172.21.15.47:0/1263167813 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7f5c74035830 msgr2=0x7f5c74037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:04.571 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.934+0000 7f5c93ae7700 1 --2- 172.21.15.47:0/1263167813 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7f5c74035830 0x7f5c74037ce0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c009750 tx=0x7f5c7c006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:04.571 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.934+0000 7f5c93ae7700 1 -- 172.21.15.47:0/1263167813 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c8c103660 msgr2=0x7f5c8c199210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:04.572 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.934+0000 7f5c93ae7700 1 --2- 172.21.15.47:0/1263167813 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c8c103660 0x7f5c8c199210 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5c88003fb0 tx=0x7f5c88004090 comp rx=0 tx=0).stop 2022-09-07T21:30:04.572 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.934+0000 7f5c93ae7700 1 -- 172.21.15.47:0/1263167813 shutdown_connections 2022-09-07T21:30:04.572 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.934+0000 7f5c93ae7700 1 --2- 172.21.15.47:0/1263167813 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7f5c74035830 0x7f5c74037ce0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:04.572 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.934+0000 7f5c93ae7700 1 --2- 172.21.15.47:0/1263167813 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c8c103660 0x7f5c8c199210 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:04.573 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.934+0000 7f5c93ae7700 1 -- 172.21.15.47:0/1263167813 >> 172.21.15.47:0/1263167813 conn(0x7f5c8c0fed50 msgr2=0x7f5c8c0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:04.573 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.934+0000 7f5c93ae7700 1 -- 172.21.15.47:0/1263167813 shutdown_connections 2022-09-07T21:30:04.573 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:03.934+0000 7f5c93ae7700 1 -- 172.21.15.47:0/1263167813 wait complete. 2022-09-07T21:30:04.573 INFO:teuthology.orchestra.run.smithi047.stdout:Enabling cephadm module... 2022-09-07T21:30:04.887 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:04 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1263167813' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2022-09-07T21:30:05.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:05 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1962923144' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2022-09-07T21:30:05.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: ignoring --setuser ceph since I am not root 2022-09-07T21:30:05.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: ignoring --setgroup ceph since I am not root 2022-09-07T21:30:06.060 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.203+0000 7fa71d883700 1 Processor -- start 2022-09-07T21:30:06.060 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.203+0000 7fa71d883700 1 -- start start 2022-09-07T21:30:06.060 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.203+0000 7fa71d883700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa718105860 0x7fa718105c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:06.061 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.203+0000 7fa71d883700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7181061b0 con 0x7fa718105860 2022-09-07T21:30:06.061 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.204+0000 7fa716ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa718105860 0x7fa718105c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:06.061 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.204+0000 7fa716ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa718105860 0x7fa718105c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:37444/0 (socket says 172.21.15.47:37444) 2022-09-07T21:30:06.062 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.204+0000 7fa716ffd700 1 -- 172.21.15.47:0/3387331051 learned_addr learned my addr 172.21.15.47:0/3387331051 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:06.062 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.204+0000 7fa716ffd700 1 -- 172.21.15.47:0/3387331051 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7181062f0 con 0x7fa718105860 2022-09-07T21:30:06.062 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.205+0000 7fa716ffd700 1 --2- 172.21.15.47:0/3387331051 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa718105860 0x7fa718105c70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa708009700 tx=0x7fa708009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692b06e36af81c62 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:06.062 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.205+0000 7fa715ffb700 1 -- 172.21.15.47:0/3387331051 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa708004160 con 0x7fa718105860 2022-09-07T21:30:06.063 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.205+0000 7fa715ffb700 1 -- 172.21.15.47:0/3387331051 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fa7080042c0 con 0x7fa718105860 2022-09-07T21:30:06.063 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.205+0000 7fa715ffb700 1 -- 172.21.15.47:0/3387331051 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7080039d0 con 0x7fa718105860 2022-09-07T21:30:06.063 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.206+0000 7fa71d883700 1 -- 172.21.15.47:0/3387331051 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa718105860 msgr2=0x7fa718105c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:06.063 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.206+0000 7fa71d883700 1 --2- 172.21.15.47:0/3387331051 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa718105860 0x7fa718105c70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa708009700 tx=0x7fa708009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:06.063 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.206+0000 7fa71d883700 1 -- 172.21.15.47:0/3387331051 shutdown_connections 2022-09-07T21:30:06.064 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.206+0000 7fa71d883700 1 --2- 172.21.15.47:0/3387331051 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa718105860 0x7fa718105c70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:06.064 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.206+0000 7fa71d883700 1 -- 172.21.15.47:0/3387331051 >> 172.21.15.47:0/3387331051 conn(0x7fa7181013c0 msgr2=0x7fa7181037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:06.064 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.206+0000 7fa71d883700 1 -- 172.21.15.47:0/3387331051 shutdown_connections 2022-09-07T21:30:06.064 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.206+0000 7fa71d883700 1 -- 172.21.15.47:0/3387331051 wait complete. 2022-09-07T21:30:06.065 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.207+0000 7fa71d883700 1 Processor -- start 2022-09-07T21:30:06.065 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.207+0000 7fa71d883700 1 -- start start 2022-09-07T21:30:06.065 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.208+0000 7fa71d883700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa718105860 0x7fa718199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:06.065 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.208+0000 7fa71d883700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa718199740 con 0x7fa718105860 2022-09-07T21:30:06.066 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.208+0000 7fa716ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa718105860 0x7fa718199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:06.066 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.208+0000 7fa716ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa718105860 0x7fa718199200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:37450/0 (socket says 172.21.15.47:37450) 2022-09-07T21:30:06.066 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.208+0000 7fa716ffd700 1 -- 172.21.15.47:0/1962923144 learned_addr learned my addr 172.21.15.47:0/1962923144 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:06.066 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.208+0000 7fa716ffd700 1 -- 172.21.15.47:0/1962923144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7080093e0 con 0x7fa718105860 2022-09-07T21:30:06.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.209+0000 7fa716ffd700 1 --2- 172.21.15.47:0/1962923144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa718105860 0x7fa718199200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa708003fb0 tx=0x7fa708004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:06.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.209+0000 7fa71c881700 1 -- 172.21.15.47:0/1962923144 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7080118c0 con 0x7fa718105860 2022-09-07T21:30:06.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.209+0000 7fa71c881700 1 -- 172.21.15.47:0/1962923144 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fa708011a20 con 0x7fa718105860 2022-09-07T21:30:06.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.209+0000 7fa71d883700 1 -- 172.21.15.47:0/1962923144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa718199940 con 0x7fa718105860 2022-09-07T21:30:06.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.209+0000 7fa71c881700 1 -- 172.21.15.47:0/1962923144 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa708019550 con 0x7fa718105860 2022-09-07T21:30:06.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.209+0000 7fa71d883700 1 -- 172.21.15.47:0/1962923144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa718199d70 con 0x7fa718105860 2022-09-07T21:30:06.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.210+0000 7fa71c881700 1 -- 172.21.15.47:0/1962923144 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 4) v1 ==== 42679+0+0 (secure 0 0 0) 0x7fa70802a020 con 0x7fa718105860 2022-09-07T21:30:06.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.210+0000 7fa71d883700 1 -- 172.21.15.47:0/1962923144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa718060310 con 0x7fa718105860 2022-09-07T21:30:06.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.211+0000 7fa71c881700 1 --2- 172.21.15.47:0/1962923144 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fa704035890 0x7fa704037d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:06.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.211+0000 7fa71c881700 1 -- 172.21.15.47:0/1962923144 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa708048990 con 0x7fa718105860 2022-09-07T21:30:06.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.211+0000 7fa7167fc700 1 --2- 172.21.15.47:0/1962923144 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fa704035890 0x7fa704037d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:06.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.212+0000 7fa7167fc700 1 --2- 172.21.15.47:0/1962923144 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fa704035890 0x7fa704037d40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa70c009750 tx=0x7fa70c006cc0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:06.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.215+0000 7fa71c881700 1 -- 172.21.15.47:0/1962923144 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fa708018b40 con 0x7fa718105860 2022-09-07T21:30:06.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.494+0000 7fa71d883700 1 -- 172.21.15.47:0/1962923144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fa7181076a0 con 0x7fa718105860 2022-09-07T21:30:06.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.509+0000 7fa71c881700 1 -- 172.21.15.47:0/1962923144 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 5) v1 ==== 42690+0+0 (secure 0 0 0) 0x7fa708017020 con 0x7fa718105860 2022-09-07T21:30:06.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.511+0000 7fa71c881700 1 -- 172.21.15.47:0/1962923144 <== mon.0 v2:172.21.15.47:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fa708023c00 con 0x7fa718105860 2022-09-07T21:30:06.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.514+0000 7fa71d883700 1 -- 172.21.15.47:0/1962923144 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fa704035890 msgr2=0x7fa704037d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:06.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.514+0000 7fa71d883700 1 --2- 172.21.15.47:0/1962923144 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fa704035890 0x7fa704037d40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa70c009750 tx=0x7fa70c006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:06.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.514+0000 7fa71d883700 1 -- 172.21.15.47:0/1962923144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa718105860 msgr2=0x7fa718199200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:06.072 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.514+0000 7fa71d883700 1 --2- 172.21.15.47:0/1962923144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa718105860 0x7fa718199200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa708003fb0 tx=0x7fa708004090 comp rx=0 tx=0).stop 2022-09-07T21:30:06.072 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.515+0000 7fa71d883700 1 -- 172.21.15.47:0/1962923144 shutdown_connections 2022-09-07T21:30:06.072 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.515+0000 7fa71d883700 1 --2- 172.21.15.47:0/1962923144 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fa704035890 0x7fa704037d40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:06.073 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.515+0000 7fa71d883700 1 --2- 172.21.15.47:0/1962923144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa718105860 0x7fa718199200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:06.073 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.515+0000 7fa71d883700 1 -- 172.21.15.47:0/1962923144 >> 172.21.15.47:0/1962923144 conn(0x7fa7181013c0 msgr2=0x7fa718103010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:06.073 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.515+0000 7fa71d883700 1 -- 172.21.15.47:0/1962923144 shutdown_connections 2022-09-07T21:30:06.073 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:05.515+0000 7fa71d883700 1 -- 172.21.15.47:0/1962923144 wait complete. 2022-09-07T21:30:06.170 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:05.910+0000 7f0a5269f080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-07T21:30:06.171 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:06.027+0000 7f0a5269f080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-07T21:30:06.463 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:06.167+0000 7f0a5269f080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-07T21:30:06.464 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:06.302+0000 7f0a5269f080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-07T21:30:06.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:06 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1962923144' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2022-09-07T21:30:06.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:06 smithi047 ceph-mon[90087]: mgrmap e5: x(active, since 5s) 2022-09-07T21:30:06.904 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:06.567+0000 7f0a5269f080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-07T21:30:07.397 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:07.062+0000 7f0a5269f080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-07T21:30:07.397 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:07.208+0000 7f0a5269f080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-07T21:30:07.573 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout { 2022-09-07T21:30:07.573 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 5, 2022-09-07T21:30:07.573 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "available": true, 2022-09-07T21:30:07.574 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "active_name": "x", 2022-09-07T21:30:07.574 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_standby": 0 2022-09-07T21:30:07.574 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout } 2022-09-07T21:30:07.575 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.723+0000 7efc2a718700 1 Processor -- start 2022-09-07T21:30:07.575 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.723+0000 7efc2a718700 1 -- start start 2022-09-07T21:30:07.575 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.724+0000 7efc2a718700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24078cc0 0x7efc24077240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:07.575 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.724+0000 7efc2a718700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc24077780 con 0x7efc24078cc0 2022-09-07T21:30:07.576 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.724+0000 7efc23fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24078cc0 0x7efc24077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:07.576 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.724+0000 7efc23fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24078cc0 0x7efc24077240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:37476/0 (socket says 172.21.15.47:37476) 2022-09-07T21:30:07.576 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.724+0000 7efc23fff700 1 -- 172.21.15.47:0/2339000578 learned_addr learned my addr 172.21.15.47:0/2339000578 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:07.576 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.725+0000 7efc23fff700 1 -- 172.21.15.47:0/2339000578 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc240778c0 con 0x7efc24078cc0 2022-09-07T21:30:07.577 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.725+0000 7efc23fff700 1 --2- 172.21.15.47:0/2339000578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24078cc0 0x7efc24077240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efc14009700 tx=0x7efc14009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d51e2bce0b4ae8c server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:07.577 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.726+0000 7efc22ffd700 1 -- 172.21.15.47:0/2339000578 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc14004160 con 0x7efc24078cc0 2022-09-07T21:30:07.577 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.726+0000 7efc22ffd700 1 -- 172.21.15.47:0/2339000578 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7efc140042c0 con 0x7efc24078cc0 2022-09-07T21:30:07.577 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.726+0000 7efc22ffd700 1 -- 172.21.15.47:0/2339000578 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc140039d0 con 0x7efc24078cc0 2022-09-07T21:30:07.578 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.726+0000 7efc2a718700 1 -- 172.21.15.47:0/2339000578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24078cc0 msgr2=0x7efc24077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:07.578 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.726+0000 7efc2a718700 1 --2- 172.21.15.47:0/2339000578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24078cc0 0x7efc24077240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efc14009700 tx=0x7efc14009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:07.578 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.727+0000 7efc2a718700 1 -- 172.21.15.47:0/2339000578 shutdown_connections 2022-09-07T21:30:07.578 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.727+0000 7efc2a718700 1 --2- 172.21.15.47:0/2339000578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24078cc0 0x7efc24077240 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:07.579 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.727+0000 7efc2a718700 1 -- 172.21.15.47:0/2339000578 >> 172.21.15.47:0/2339000578 conn(0x7efc240fed50 msgr2=0x7efc24101180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:07.579 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.727+0000 7efc2a718700 1 -- 172.21.15.47:0/2339000578 shutdown_connections 2022-09-07T21:30:07.579 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.727+0000 7efc2a718700 1 -- 172.21.15.47:0/2339000578 wait complete. 2022-09-07T21:30:07.579 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.729+0000 7efc2a718700 1 Processor -- start 2022-09-07T21:30:07.580 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.729+0000 7efc2a718700 1 -- start start 2022-09-07T21:30:07.580 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.729+0000 7efc2a718700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24078cc0 0x7efc24199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:07.580 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.730+0000 7efc2a718700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc24199770 con 0x7efc24078cc0 2022-09-07T21:30:07.580 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.730+0000 7efc23fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24078cc0 0x7efc24199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:07.580 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.730+0000 7efc23fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24078cc0 0x7efc24199230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:37492/0 (socket says 172.21.15.47:37492) 2022-09-07T21:30:07.581 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.730+0000 7efc23fff700 1 -- 172.21.15.47:0/1761853606 learned_addr learned my addr 172.21.15.47:0/1761853606 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:07.581 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.730+0000 7efc23fff700 1 -- 172.21.15.47:0/1761853606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc140093e0 con 0x7efc24078cc0 2022-09-07T21:30:07.581 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.731+0000 7efc23fff700 1 --2- 172.21.15.47:0/1761853606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24078cc0 0x7efc24199230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efc14009e90 tx=0x7efc14004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:07.581 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.731+0000 7efc217fa700 1 -- 172.21.15.47:0/1761853606 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc14011840 con 0x7efc24078cc0 2022-09-07T21:30:07.582 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.731+0000 7efc217fa700 1 -- 172.21.15.47:0/1761853606 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7efc1401a430 con 0x7efc24078cc0 2022-09-07T21:30:07.582 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.731+0000 7efc217fa700 1 -- 172.21.15.47:0/1761853606 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc14019550 con 0x7efc24078cc0 2022-09-07T21:30:07.582 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.731+0000 7efc2a718700 1 -- 172.21.15.47:0/1761853606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc24199970 con 0x7efc24078cc0 2022-09-07T21:30:07.582 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.731+0000 7efc2a718700 1 -- 172.21.15.47:0/1761853606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc24199da0 con 0x7efc24078cc0 2022-09-07T21:30:07.582 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.732+0000 7efc217fa700 1 -- 172.21.15.47:0/1761853606 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 42690+0+0 (secure 0 0 0) 0x7efc1402a020 con 0x7efc24078cc0 2022-09-07T21:30:07.583 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.733+0000 7efc217fa700 1 --2- 172.21.15.47:0/1761853606 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7efc100358d0 0x7efc10037d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:07.583 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.732+0000 7efc2a718700 1 -- 172.21.15.47:0/1761853606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc24060310 con 0x7efc24078cc0 2022-09-07T21:30:07.583 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.733+0000 7efc217fa700 1 -- 172.21.15.47:0/1761853606 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7efc140489b0 con 0x7efc24078cc0 2022-09-07T21:30:07.583 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.733+0000 7efc237fe700 1 -- 172.21.15.47:0/1761853606 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7efc100358d0 msgr2=0x7efc10037d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1435938941 2022-09-07T21:30:07.584 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.733+0000 7efc237fe700 1 --2- 172.21.15.47:0/1761853606 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7efc100358d0 0x7efc10037d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-07T21:30:07.584 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.738+0000 7efc217fa700 1 -- 172.21.15.47:0/1761853606 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7efc140119a0 con 0x7efc24078cc0 2022-09-07T21:30:07.584 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.934+0000 7efc237fe700 1 -- 172.21.15.47:0/1761853606 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7efc100358d0 msgr2=0x7efc10037d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1435938941 2022-09-07T21:30:07.584 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:06.934+0000 7efc237fe700 1 --2- 172.21.15.47:0/1761853606 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7efc100358d0 0x7efc10037d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-09-07T21:30:07.584 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:07.041+0000 7efc2a718700 1 -- 172.21.15.47:0/1761853606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7efc2419cfa0 con 0x7efc24078cc0 2022-09-07T21:30:07.585 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:07.041+0000 7efc217fa700 1 -- 172.21.15.47:0/1761853606 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7efc14018b40 con 0x7efc24078cc0 2022-09-07T21:30:07.585 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:07.043+0000 7efc2a718700 1 -- 172.21.15.47:0/1761853606 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7efc100358d0 msgr2=0x7efc10037d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-07T21:30:07.585 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:07.043+0000 7efc2a718700 1 --2- 172.21.15.47:0/1761853606 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7efc100358d0 0x7efc10037d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:07.585 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:07.043+0000 7efc2a718700 1 -- 172.21.15.47:0/1761853606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24078cc0 msgr2=0x7efc24199230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:07.586 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:07.043+0000 7efc2a718700 1 --2- 172.21.15.47:0/1761853606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24078cc0 0x7efc24199230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efc14009e90 tx=0x7efc14004040 comp rx=0 tx=0).stop 2022-09-07T21:30:07.586 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:07.044+0000 7efc2a718700 1 -- 172.21.15.47:0/1761853606 shutdown_connections 2022-09-07T21:30:07.586 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:07.044+0000 7efc2a718700 1 --2- 172.21.15.47:0/1761853606 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7efc100358d0 0x7efc10037d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:07.587 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:07.044+0000 7efc2a718700 1 --2- 172.21.15.47:0/1761853606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc24078cc0 0x7efc24199230 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:07.587 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:07.044+0000 7efc2a718700 1 -- 172.21.15.47:0/1761853606 >> 172.21.15.47:0/1761853606 conn(0x7efc240fed50 msgr2=0x7efc240ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:07.587 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:07.044+0000 7efc2a718700 1 -- 172.21.15.47:0/1761853606 shutdown_connections 2022-09-07T21:30:07.587 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:07.044+0000 7efc2a718700 1 -- 172.21.15.47:0/1761853606 wait complete. 2022-09-07T21:30:07.588 INFO:teuthology.orchestra.run.smithi047.stdout:Waiting for the mgr to restart... 2022-09-07T21:30:07.588 INFO:teuthology.orchestra.run.smithi047.stdout:Waiting for mgr epoch 5... 2022-09-07T21:30:07.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:07 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1761853606' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2022-09-07T21:30:07.657 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:07.394+0000 7f0a5269f080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-07T21:30:07.657 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:07.542+0000 7f0a5269f080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-07T21:30:08.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:08.207+0000 7f0a5269f080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-07T21:30:09.403 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:09.079+0000 7f0a5269f080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-07T21:30:09.404 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:09.320+0000 7f0a5269f080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-07T21:30:10.882 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:10.434+0000 7f0a5269f080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:30:10.883 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:10.589+0000 7f0a5269f080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-07T21:30:11.153 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:10.880+0000 7f0a5269f080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:30:11.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:11.199+0000 7f0a5269f080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-07T21:30:11.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:11.440+0000 7f0a5269f080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-07T21:30:12.403 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:12.057+0000 7f0a5269f080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-07T21:30:13.403 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:13.018+0000 7f0a5269f080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-07T21:30:13.788 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:13.449+0000 7f0a5269f080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-07T21:30:14.153 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:13.786+0000 7f0a5269f080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-07T21:30:14.153 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:13.982+0000 7f0a5269f080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-07T21:30:14.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:14.356+0000 7f0a5269f080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-07T21:30:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:15 smithi047 ceph-mon[90087]: Active manager daemon x restarted 2022-09-07T21:30:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:15 smithi047 ceph-mon[90087]: Activating manager daemon x 2022-09-07T21:30:15.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:15 smithi047 ceph-mon[90087]: osdmap e2: 0 total, 0 up, 0 in 2022-09-07T21:30:15.904 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:15.413+0000 7f0a5269f080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-07T21:30:16.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: mgrmap e6: x(active, starting, since 0.0531889s) 2022-09-07T21:30:16.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:30:16.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-09-07T21:30:16.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-09-07T21:30:16.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-09-07T21:30:16.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-09-07T21:30:16.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: Manager daemon x is now available 2022-09-07T21:30:16.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' 2022-09-07T21:30:16.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' 2022-09-07T21:30:16.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:30:16.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:30:16.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:30:16.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:30:16.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:30:16.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' 2022-09-07T21:30:16.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:16 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' 2022-09-07T21:30:16.991 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout { 2022-09-07T21:30:16.992 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2022-09-07T21:30:16.992 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "initialized": true 2022-09-07T21:30:16.992 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout } 2022-09-07T21:30:16.993 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.212+0000 7fae775bc700 1 Processor -- start 2022-09-07T21:30:16.993 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.212+0000 7fae775bc700 1 -- start start 2022-09-07T21:30:16.994 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.212+0000 7fae775bc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae70078ca0 0x7fae70077220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:16.994 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.212+0000 7fae775bc700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae700790b0 con 0x7fae70078ca0 2022-09-07T21:30:16.995 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.212+0000 7fae75358700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae70078ca0 0x7fae70077220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:16.995 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.213+0000 7fae75358700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae70078ca0 0x7fae70077220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:37508/0 (socket says 172.21.15.47:37508) 2022-09-07T21:30:16.995 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.213+0000 7fae75358700 1 -- 172.21.15.47:0/4138424488 learned_addr learned my addr 172.21.15.47:0/4138424488 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:16.995 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.213+0000 7fae75358700 1 -- 172.21.15.47:0/4138424488 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae70077760 con 0x7fae70078ca0 2022-09-07T21:30:16.996 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.213+0000 7fae75358700 1 --2- 172.21.15.47:0/4138424488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae70078ca0 0x7fae70077220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009700 tx=0x7fae6c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de50d66aa11a3076 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:16.996 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.214+0000 7fae5ffff700 1 -- 172.21.15.47:0/4138424488 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae6c004160 con 0x7fae70078ca0 2022-09-07T21:30:16.996 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.214+0000 7fae5ffff700 1 -- 172.21.15.47:0/4138424488 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fae6c0042c0 con 0x7fae70078ca0 2022-09-07T21:30:16.996 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.214+0000 7fae5ffff700 1 -- 172.21.15.47:0/4138424488 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae6c0039d0 con 0x7fae70078ca0 2022-09-07T21:30:16.997 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.214+0000 7fae775bc700 1 -- 172.21.15.47:0/4138424488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae70078ca0 msgr2=0x7fae70077220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:16.997 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.214+0000 7fae775bc700 1 --2- 172.21.15.47:0/4138424488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae70078ca0 0x7fae70077220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009700 tx=0x7fae6c009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:16.997 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.214+0000 7fae775bc700 1 -- 172.21.15.47:0/4138424488 shutdown_connections 2022-09-07T21:30:16.997 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.214+0000 7fae775bc700 1 --2- 172.21.15.47:0/4138424488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae70078ca0 0x7fae70077220 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:16.998 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.214+0000 7fae775bc700 1 -- 172.21.15.47:0/4138424488 >> 172.21.15.47:0/4138424488 conn(0x7fae700fed50 msgr2=0x7fae70101180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:16.998 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.215+0000 7fae775bc700 1 -- 172.21.15.47:0/4138424488 shutdown_connections 2022-09-07T21:30:16.998 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.215+0000 7fae775bc700 1 -- 172.21.15.47:0/4138424488 wait complete. 2022-09-07T21:30:16.998 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.216+0000 7fae775bc700 1 Processor -- start 2022-09-07T21:30:16.998 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.216+0000 7fae775bc700 1 -- start start 2022-09-07T21:30:16.999 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.216+0000 7fae775bc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae70078ca0 0x7fae70199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:16.999 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.216+0000 7fae775bc700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae70199790 con 0x7fae70078ca0 2022-09-07T21:30:16.999 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.216+0000 7fae75358700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae70078ca0 0x7fae70199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:17.000 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.217+0000 7fae75358700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae70078ca0 0x7fae70199250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:37522/0 (socket says 172.21.15.47:37522) 2022-09-07T21:30:17.001 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.217+0000 7fae75358700 1 -- 172.21.15.47:0/1219323078 learned_addr learned my addr 172.21.15.47:0/1219323078 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:17.001 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.217+0000 7fae75358700 1 -- 172.21.15.47:0/1219323078 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6c0093e0 con 0x7fae70078ca0 2022-09-07T21:30:17.001 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.217+0000 7fae75358700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae70078ca0 0x7fae70199250 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009e90 tx=0x7fae6c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:17.001 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.217+0000 7fae5e7fc700 1 -- 172.21.15.47:0/1219323078 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae6c011840 con 0x7fae70078ca0 2022-09-07T21:30:17.002 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.218+0000 7fae5e7fc700 1 -- 172.21.15.47:0/1219323078 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fae6c01a430 con 0x7fae70078ca0 2022-09-07T21:30:17.002 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.218+0000 7fae5e7fc700 1 -- 172.21.15.47:0/1219323078 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae6c019550 con 0x7fae70078ca0 2022-09-07T21:30:17.002 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.218+0000 7fae775bc700 1 -- 172.21.15.47:0/1219323078 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae70199990 con 0x7fae70078ca0 2022-09-07T21:30:17.002 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.218+0000 7fae775bc700 1 -- 172.21.15.47:0/1219323078 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae70199dc0 con 0x7fae70078ca0 2022-09-07T21:30:17.003 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.219+0000 7fae5e7fc700 1 -- 172.21.15.47:0/1219323078 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 42690+0+0 (secure 0 0 0) 0x7fae6c02a020 con 0x7fae70078ca0 2022-09-07T21:30:17.003 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.219+0000 7fae5e7fc700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 0x7fae58037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:17.003 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.219+0000 7fae5e7fc700 1 -- 172.21.15.47:0/1219323078 --> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae580383f0 con 0x7fae58035830 2022-09-07T21:30:17.004 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.219+0000 7fae5e7fc700 1 -- 172.21.15.47:0/1219323078 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fae6c0489b0 con 0x7fae70078ca0 2022-09-07T21:30:17.004 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.219+0000 7fae74b57700 1 -- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 msgr2=0x7fae58037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1435938941 2022-09-07T21:30:17.004 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.220+0000 7fae74b57700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 0x7fae58037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-07T21:30:17.004 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.421+0000 7fae74b57700 1 -- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 msgr2=0x7fae58037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1435938941 2022-09-07T21:30:17.004 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.421+0000 7fae74b57700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 0x7fae58037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-09-07T21:30:17.005 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.822+0000 7fae74b57700 1 -- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 msgr2=0x7fae58037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1435938941 2022-09-07T21:30:17.005 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:08.822+0000 7fae74b57700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 0x7fae58037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2022-09-07T21:30:17.005 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:09.623+0000 7fae74b57700 1 -- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 msgr2=0x7fae58037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1435938941 2022-09-07T21:30:17.006 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:09.623+0000 7fae74b57700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 0x7fae58037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2022-09-07T21:30:17.006 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:11.224+0000 7fae74b57700 1 -- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 msgr2=0x7fae58037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1435938941 2022-09-07T21:30:17.006 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:11.224+0000 7fae74b57700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 0x7fae58037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2022-09-07T21:30:17.006 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:14.427+0000 7fae74b57700 1 -- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 msgr2=0x7fae58037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1435938941 2022-09-07T21:30:17.007 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:14.427+0000 7fae74b57700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 0x7fae58037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2022-09-07T21:30:17.007 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:15.467+0000 7fae5e7fc700 1 -- 172.21.15.47:0/1219323078 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mgrmap(e 6) v1 ==== 42500+0+0 (secure 0 0 0) 0x7fae6c017020 con 0x7fae70078ca0 2022-09-07T21:30:17.007 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:15.467+0000 7fae5e7fc700 1 -- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 msgr2=0x7fae58037ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-07T21:30:17.007 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:15.467+0000 7fae5e7fc700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 0x7fae58037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:17.008 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.475+0000 7fae5e7fc700 1 -- 172.21.15.47:0/1219323078 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 7) v1 ==== 42610+0+0 (secure 0 0 0) 0x7fae6c00f050 con 0x7fae70078ca0 2022-09-07T21:30:17.008 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.475+0000 7fae5e7fc700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7fae58039060 0x7fae5803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:17.008 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.475+0000 7fae5e7fc700 1 -- 172.21.15.47:0/1219323078 --> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae580383f0 con 0x7fae58039060 2022-09-07T21:30:17.008 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.476+0000 7fae74b57700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7fae58039060 0x7fae5803b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:17.009 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.478+0000 7fae74b57700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7fae58039060 0x7fae5803b440 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fae60006fd0 tx=0x7fae60006dc0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:17.009 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.478+0000 7fae5e7fc700 1 -- 172.21.15.47:0/1219323078 <== mgr.14120 v2:172.21.15.47:6800/4097349239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6424 (secure 0 0 0) 0x7fae580383f0 con 0x7fae58039060 2022-09-07T21:30:17.009 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.485+0000 7fae775bc700 1 -- 172.21.15.47:0/1219323078 --> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fae7019a580 con 0x7fae58039060 2022-09-07T21:30:17.010 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.486+0000 7fae5e7fc700 1 -- 172.21.15.47:0/1219323078 <== mgr.14120 v2:172.21.15.47:6800/4097349239 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fae7019a580 con 0x7fae58039060 2022-09-07T21:30:17.010 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.486+0000 7fae775bc700 1 -- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7fae58039060 msgr2=0x7fae5803b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:17.010 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.486+0000 7fae775bc700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7fae58039060 0x7fae5803b440 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fae60006fd0 tx=0x7fae60006dc0 comp rx=0 tx=0).stop 2022-09-07T21:30:17.011 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.486+0000 7fae775bc700 1 -- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae70078ca0 msgr2=0x7fae70199250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:17.012 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.486+0000 7fae775bc700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae70078ca0 0x7fae70199250 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009e90 tx=0x7fae6c004040 comp rx=0 tx=0).stop 2022-09-07T21:30:17.012 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.486+0000 7fae775bc700 1 -- 172.21.15.47:0/1219323078 shutdown_connections 2022-09-07T21:30:17.012 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.486+0000 7fae775bc700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7fae58039060 0x7fae5803b440 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:17.012 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.486+0000 7fae775bc700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae70078ca0 0x7fae70199250 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:17.013 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.486+0000 7fae775bc700 1 --2- 172.21.15.47:0/1219323078 >> [v2:172.21.15.47:6800/1435938941,v1:172.21.15.47:6801/1435938941] conn(0x7fae58035830 0x7fae58037ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:17.013 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.486+0000 7fae775bc700 1 -- 172.21.15.47:0/1219323078 >> 172.21.15.47:0/1219323078 conn(0x7fae700fed50 msgr2=0x7fae700ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:17.013 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.487+0000 7fae775bc700 1 -- 172.21.15.47:0/1219323078 shutdown_connections 2022-09-07T21:30:17.013 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:16.487+0000 7fae775bc700 1 -- 172.21.15.47:0/1219323078 wait complete. 2022-09-07T21:30:17.014 INFO:teuthology.orchestra.run.smithi047.stdout:mgr epoch 5 is available 2022-09-07T21:30:17.014 INFO:teuthology.orchestra.run.smithi047.stdout:Setting orchestrator backend to cephadm... 2022-09-07T21:30:17.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:17 smithi047 ceph-mon[90087]: mgrmap e7: x(active, since 1.06293s) 2022-09-07T21:30:17.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:17 smithi047 ceph-mon[90087]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2022-09-07T21:30:17.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:17 smithi047 ceph-mon[90087]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2022-09-07T21:30:18.500 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.651+0000 7f385e9ae700 1 Processor -- start 2022-09-07T21:30:18.500 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.652+0000 7f385e9ae700 1 -- start start 2022-09-07T21:30:18.501 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.652+0000 7f385e9ae700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3858071ea0 0x7f3858108650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:18.502 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.652+0000 7f385e9ae700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3858104040 con 0x7f3858071ea0 2022-09-07T21:30:18.502 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.652+0000 7f3857fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3858071ea0 0x7f3858108650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:18.502 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.653+0000 7f3857fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3858071ea0 0x7f3858108650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50866/0 (socket says 172.21.15.47:50866) 2022-09-07T21:30:18.503 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.653+0000 7f3857fff700 1 -- 172.21.15.47:0/3635013882 learned_addr learned my addr 172.21.15.47:0/3635013882 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:18.503 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.653+0000 7f3857fff700 1 -- 172.21.15.47:0/3635013882 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3858108b90 con 0x7f3858071ea0 2022-09-07T21:30:18.503 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.653+0000 7f3857fff700 1 --2- 172.21.15.47:0/3635013882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3858071ea0 0x7f3858108650 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3848009700 tx=0x7f3848009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72acd2cf95d6dc5a server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:18.503 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.654+0000 7f3856ffd700 1 -- 172.21.15.47:0/3635013882 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3848004160 con 0x7f3858071ea0 2022-09-07T21:30:18.504 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.654+0000 7f3856ffd700 1 -- 172.21.15.47:0/3635013882 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f38480042c0 con 0x7f3858071ea0 2022-09-07T21:30:18.504 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.654+0000 7f3856ffd700 1 -- 172.21.15.47:0/3635013882 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38480039d0 con 0x7f3858071ea0 2022-09-07T21:30:18.504 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.654+0000 7f385e9ae700 1 -- 172.21.15.47:0/3635013882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3858071ea0 msgr2=0x7f3858108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:18.505 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.654+0000 7f385e9ae700 1 --2- 172.21.15.47:0/3635013882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3858071ea0 0x7f3858108650 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3848009700 tx=0x7f3848009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:18.505 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.654+0000 7f385e9ae700 1 -- 172.21.15.47:0/3635013882 shutdown_connections 2022-09-07T21:30:18.505 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.654+0000 7f385e9ae700 1 --2- 172.21.15.47:0/3635013882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3858071ea0 0x7f3858108650 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:18.505 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.654+0000 7f385e9ae700 1 -- 172.21.15.47:0/3635013882 >> 172.21.15.47:0/3635013882 conn(0x7f38580fdc20 msgr2=0x7f3858100050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:18.506 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.654+0000 7f385e9ae700 1 -- 172.21.15.47:0/3635013882 shutdown_connections 2022-09-07T21:30:18.506 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.655+0000 7f385e9ae700 1 -- 172.21.15.47:0/3635013882 wait complete. 2022-09-07T21:30:18.506 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.656+0000 7f385e9ae700 1 Processor -- start 2022-09-07T21:30:18.506 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.656+0000 7f385e9ae700 1 -- start start 2022-09-07T21:30:18.507 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.656+0000 7f385e9ae700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3858071ea0 0x7f3858199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:18.507 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.656+0000 7f385e9ae700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3858199790 con 0x7f3858071ea0 2022-09-07T21:30:18.507 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.656+0000 7f3857fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3858071ea0 0x7f3858199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:18.507 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.657+0000 7f3857fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3858071ea0 0x7f3858199250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50870/0 (socket says 172.21.15.47:50870) 2022-09-07T21:30:18.507 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.657+0000 7f3857fff700 1 -- 172.21.15.47:0/2631123874 learned_addr learned my addr 172.21.15.47:0/2631123874 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:18.508 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.657+0000 7f3857fff700 1 -- 172.21.15.47:0/2631123874 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38480093e0 con 0x7f3858071ea0 2022-09-07T21:30:18.508 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.657+0000 7f3857fff700 1 --2- 172.21.15.47:0/2631123874 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3858071ea0 0x7f3858199250 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3848003f10 tx=0x7f3848003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:18.508 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.658+0000 7f38557fa700 1 -- 172.21.15.47:0/2631123874 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3848011950 con 0x7f3858071ea0 2022-09-07T21:30:18.508 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.658+0000 7f38557fa700 1 -- 172.21.15.47:0/2631123874 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f3848011ab0 con 0x7f3858071ea0 2022-09-07T21:30:18.509 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.658+0000 7f385e9ae700 1 -- 172.21.15.47:0/2631123874 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3858199990 con 0x7f3858071ea0 2022-09-07T21:30:18.509 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.658+0000 7f38557fa700 1 -- 172.21.15.47:0/2631123874 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3848019550 con 0x7f3858071ea0 2022-09-07T21:30:18.509 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.658+0000 7f385e9ae700 1 -- 172.21.15.47:0/2631123874 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3858199dc0 con 0x7f3858071ea0 2022-09-07T21:30:18.509 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.659+0000 7f38557fa700 1 -- 172.21.15.47:0/2631123874 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f384802a020 con 0x7f3858071ea0 2022-09-07T21:30:18.510 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.659+0000 7f385e9ae700 1 -- 172.21.15.47:0/2631123874 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38581098a0 con 0x7f3858071ea0 2022-09-07T21:30:18.510 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.659+0000 7f38557fa700 1 --2- 172.21.15.47:0/2631123874 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f38440358a0 0x7f3844037d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:18.510 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.659+0000 7f38557fa700 1 -- 172.21.15.47:0/2631123874 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3848048af0 con 0x7f3858071ea0 2022-09-07T21:30:18.510 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.660+0000 7f38577fe700 1 --2- 172.21.15.47:0/2631123874 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f38440358a0 0x7f3844037d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:18.511 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.661+0000 7f38577fe700 1 --2- 172.21.15.47:0/2631123874 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f38440358a0 0x7f3844037d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f384c009750 tx=0x7f384c006cc0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:18.511 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.665+0000 7f38557fa700 1 -- 172.21.15.47:0/2631123874 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f3848018d20 con 0x7f3858071ea0 2022-09-07T21:30:18.511 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.906+0000 7f385e9ae700 1 -- 172.21.15.47:0/2631123874 --> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f38580fef70 con 0x7f38440358a0 2022-09-07T21:30:18.511 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.913+0000 7f38557fa700 1 -- 172.21.15.47:0/2631123874 <== mgr.14120 v2:172.21.15.47:6800/4097349239 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f38580fef70 con 0x7f38440358a0 2022-09-07T21:30:18.511 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.916+0000 7f385e9ae700 1 -- 172.21.15.47:0/2631123874 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f38440358a0 msgr2=0x7f3844037d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:18.512 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.916+0000 7f385e9ae700 1 --2- 172.21.15.47:0/2631123874 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f38440358a0 0x7f3844037d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f384c009750 tx=0x7f384c006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:18.512 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.916+0000 7f385e9ae700 1 -- 172.21.15.47:0/2631123874 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3858071ea0 msgr2=0x7f3858199250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:18.512 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.916+0000 7f385e9ae700 1 --2- 172.21.15.47:0/2631123874 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3858071ea0 0x7f3858199250 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3848003f10 tx=0x7f3848003ff0 comp rx=0 tx=0).stop 2022-09-07T21:30:18.512 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.916+0000 7f385e9ae700 1 -- 172.21.15.47:0/2631123874 shutdown_connections 2022-09-07T21:30:18.513 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.916+0000 7f385e9ae700 1 --2- 172.21.15.47:0/2631123874 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f38440358a0 0x7f3844037d50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:18.513 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.916+0000 7f385e9ae700 1 --2- 172.21.15.47:0/2631123874 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3858071ea0 0x7f3858199250 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:18.513 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.916+0000 7f385e9ae700 1 -- 172.21.15.47:0/2631123874 >> 172.21.15.47:0/2631123874 conn(0x7f38580fdc20 msgr2=0x7f38580fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:18.513 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.916+0000 7f385e9ae700 1 -- 172.21.15.47:0/2631123874 shutdown_connections 2022-09-07T21:30:18.514 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:17.916+0000 7f385e9ae700 1 -- 172.21.15.47:0/2631123874 wait complete. 2022-09-07T21:30:18.835 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:18 smithi047 ceph-mon[90087]: mgrmap e8: x(active, since 2s) 2022-09-07T21:30:18.836 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:18 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' 2022-09-07T21:30:18.836 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:18 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' 2022-09-07T21:30:18.836 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:18 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' 2022-09-07T21:30:18.836 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:18 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:30:18.837 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:18 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:30:19.899 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout value unchanged 2022-09-07T21:30:19.899 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.184+0000 7fd55bb6e700 1 Processor -- start 2022-09-07T21:30:19.900 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.184+0000 7fd55bb6e700 1 -- start start 2022-09-07T21:30:19.900 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.185+0000 7fd55bb6e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd554078ca0 0x7fd554077220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:19.900 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.185+0000 7fd55bb6e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5540790b0 con 0x7fd554078ca0 2022-09-07T21:30:19.900 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.185+0000 7fd55990a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd554078ca0 0x7fd554077220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:19.901 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.185+0000 7fd55990a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd554078ca0 0x7fd554077220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54380/0 (socket says 172.21.15.47:54380) 2022-09-07T21:30:19.901 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.185+0000 7fd55990a700 1 -- 172.21.15.47:0/177369386 learned_addr learned my addr 172.21.15.47:0/177369386 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:19.901 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.185+0000 7fd55990a700 1 -- 172.21.15.47:0/177369386 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd554077760 con 0x7fd554078ca0 2022-09-07T21:30:19.901 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.186+0000 7fd55990a700 1 --2- 172.21.15.47:0/177369386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd554078ca0 0x7fd554077220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd550009700 tx=0x7fd550009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c50705fb8f8893 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:19.901 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.186+0000 7fd558908700 1 -- 172.21.15.47:0/177369386 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd550004160 con 0x7fd554078ca0 2022-09-07T21:30:19.902 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.186+0000 7fd558908700 1 -- 172.21.15.47:0/177369386 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fd5500042c0 con 0x7fd554078ca0 2022-09-07T21:30:19.903 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.186+0000 7fd558908700 1 -- 172.21.15.47:0/177369386 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5500039d0 con 0x7fd554078ca0 2022-09-07T21:30:19.903 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.186+0000 7fd55bb6e700 1 -- 172.21.15.47:0/177369386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd554078ca0 msgr2=0x7fd554077220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:19.903 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.186+0000 7fd55bb6e700 1 --2- 172.21.15.47:0/177369386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd554078ca0 0x7fd554077220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd550009700 tx=0x7fd550009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:19.903 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.187+0000 7fd55bb6e700 1 -- 172.21.15.47:0/177369386 shutdown_connections 2022-09-07T21:30:19.904 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.187+0000 7fd55bb6e700 1 --2- 172.21.15.47:0/177369386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd554078ca0 0x7fd554077220 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:19.904 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.187+0000 7fd55bb6e700 1 -- 172.21.15.47:0/177369386 >> 172.21.15.47:0/177369386 conn(0x7fd5540fed50 msgr2=0x7fd554101180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:19.904 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.187+0000 7fd55bb6e700 1 -- 172.21.15.47:0/177369386 shutdown_connections 2022-09-07T21:30:19.904 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.187+0000 7fd55bb6e700 1 -- 172.21.15.47:0/177369386 wait complete. 2022-09-07T21:30:19.905 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.188+0000 7fd55bb6e700 1 Processor -- start 2022-09-07T21:30:19.905 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.188+0000 7fd55bb6e700 1 -- start start 2022-09-07T21:30:19.905 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.189+0000 7fd55bb6e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd554078ca0 0x7fd554199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:19.905 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.189+0000 7fd55bb6e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd554199790 con 0x7fd554078ca0 2022-09-07T21:30:19.905 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.189+0000 7fd55990a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd554078ca0 0x7fd554199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:19.906 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.189+0000 7fd55990a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd554078ca0 0x7fd554199250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54384/0 (socket says 172.21.15.47:54384) 2022-09-07T21:30:19.906 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.189+0000 7fd55990a700 1 -- 172.21.15.47:0/1121916934 learned_addr learned my addr 172.21.15.47:0/1121916934 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:19.906 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.189+0000 7fd55990a700 1 -- 172.21.15.47:0/1121916934 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5500093e0 con 0x7fd554078ca0 2022-09-07T21:30:19.906 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.190+0000 7fd55990a700 1 --2- 172.21.15.47:0/1121916934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd554078ca0 0x7fd554199250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd550009e90 tx=0x7fd550004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:19.907 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.190+0000 7fd542ffd700 1 -- 172.21.15.47:0/1121916934 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd550011840 con 0x7fd554078ca0 2022-09-07T21:30:19.907 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.190+0000 7fd542ffd700 1 -- 172.21.15.47:0/1121916934 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fd55001a430 con 0x7fd554078ca0 2022-09-07T21:30:19.907 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.190+0000 7fd55bb6e700 1 -- 172.21.15.47:0/1121916934 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd554199990 con 0x7fd554078ca0 2022-09-07T21:30:19.907 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.190+0000 7fd542ffd700 1 -- 172.21.15.47:0/1121916934 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd550019550 con 0x7fd554078ca0 2022-09-07T21:30:19.907 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.190+0000 7fd55bb6e700 1 -- 172.21.15.47:0/1121916934 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd554199dc0 con 0x7fd554078ca0 2022-09-07T21:30:19.908 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.191+0000 7fd542ffd700 1 -- 172.21.15.47:0/1121916934 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7fd55002a020 con 0x7fd554078ca0 2022-09-07T21:30:19.908 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.191+0000 7fd55bb6e700 1 -- 172.21.15.47:0/1121916934 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd554060310 con 0x7fd554078ca0 2022-09-07T21:30:19.908 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.192+0000 7fd542ffd700 1 --2- 172.21.15.47:0/1121916934 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7fd53c035890 0x7fd53c037d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:19.908 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.192+0000 7fd542ffd700 1 -- 172.21.15.47:0/1121916934 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd5500489d0 con 0x7fd554078ca0 2022-09-07T21:30:19.909 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.192+0000 7fd559109700 1 --2- 172.21.15.47:0/1121916934 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7fd53c035890 0x7fd53c037d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:19.909 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.193+0000 7fd559109700 1 --2- 172.21.15.47:0/1121916934 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7fd53c035890 0x7fd53c037d40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd544009750 tx=0x7fd544006cc0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:19.909 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.197+0000 7fd542ffd700 1 -- 172.21.15.47:0/1121916934 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fd5500119a0 con 0x7fd554078ca0 2022-09-07T21:30:19.909 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.420+0000 7fd55bb6e700 1 -- 172.21.15.47:0/1121916934 --> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5541000d0 con 0x7fd53c035890 2022-09-07T21:30:19.910 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.421+0000 7fd542ffd700 1 -- 172.21.15.47:0/1121916934 <== mgr.14120 v2:172.21.15.47:6800/4097349239 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fd5541000d0 con 0x7fd53c035890 2022-09-07T21:30:19.910 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.423+0000 7fd55bb6e700 1 -- 172.21.15.47:0/1121916934 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7fd53c035890 msgr2=0x7fd53c037d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:19.910 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.423+0000 7fd55bb6e700 1 --2- 172.21.15.47:0/1121916934 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7fd53c035890 0x7fd53c037d40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd544009750 tx=0x7fd544006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:19.910 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.423+0000 7fd55bb6e700 1 -- 172.21.15.47:0/1121916934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd554078ca0 msgr2=0x7fd554199250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:19.910 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.423+0000 7fd55bb6e700 1 --2- 172.21.15.47:0/1121916934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd554078ca0 0x7fd554199250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd550009e90 tx=0x7fd550004040 comp rx=0 tx=0).stop 2022-09-07T21:30:19.911 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.423+0000 7fd55bb6e700 1 -- 172.21.15.47:0/1121916934 shutdown_connections 2022-09-07T21:30:19.911 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.423+0000 7fd55bb6e700 1 --2- 172.21.15.47:0/1121916934 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7fd53c035890 0x7fd53c037d40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:19.911 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.423+0000 7fd55bb6e700 1 --2- 172.21.15.47:0/1121916934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd554078ca0 0x7fd554199250 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:19.911 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.423+0000 7fd55bb6e700 1 -- 172.21.15.47:0/1121916934 >> 172.21.15.47:0/1121916934 conn(0x7fd5540fed50 msgr2=0x7fd5540ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:19.912 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.423+0000 7fd55bb6e700 1 -- 172.21.15.47:0/1121916934 shutdown_connections 2022-09-07T21:30:19.912 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:19.423+0000 7fd55bb6e700 1 -- 172.21.15.47:0/1121916934 wait complete. 2022-09-07T21:30:19.912 INFO:teuthology.orchestra.run.smithi047.stdout:Generating ssh key... 2022-09-07T21:30:19.913 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:19 smithi047 ceph-mon[90087]: [07/Sep/2022:21:30:18] ENGINE Bus STARTING 2022-09-07T21:30:19.913 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:19 smithi047 ceph-mon[90087]: [07/Sep/2022:21:30:18] ENGINE Serving on https://:::8765 2022-09-07T21:30:19.913 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:19 smithi047 ceph-mon[90087]: [07/Sep/2022:21:30:18] ENGINE Serving on https://172.21.15.47:7150 2022-09-07T21:30:19.913 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:19 smithi047 ceph-mon[90087]: [07/Sep/2022:21:30:18] ENGINE Bus STARTED 2022-09-07T21:30:20.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:20 smithi047 ceph-mon[90087]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:30:21.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: Generating public/private rsa key pair. 2022-09-07T21:30:21.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: Your identification has been saved in /tmp/tmp1pymazt5/key. 2022-09-07T21:30:21.654 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: Your public key has been saved in /tmp/tmp1pymazt5/key.pub. 2022-09-07T21:30:21.654 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: The key fingerprint is: 2022-09-07T21:30:21.654 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: SHA256:Yaf28YePY/u3aT/aTQbpVDWoshpT1csEZIH92cGWMt0 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:30:21.654 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: The key's randomart image is: 2022-09-07T21:30:21.655 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: +---[RSA 3072]----+ 2022-09-07T21:30:21.655 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: | +=+ +.+| 2022-09-07T21:30:21.655 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: | ..o * *E| 2022-09-07T21:30:21.656 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: | o o = B o| 2022-09-07T21:30:21.656 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: | . * . = + | 2022-09-07T21:30:21.657 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: | S + + | 2022-09-07T21:30:21.657 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: | + o o + . | 2022-09-07T21:30:21.657 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: | + . o o o| 2022-09-07T21:30:21.658 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: | . o+.=+| 2022-09-07T21:30:21.658 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: | .o===*| 2022-09-07T21:30:21.659 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: +----[SHA256]-----+ 2022-09-07T21:30:21.906 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.530+0000 7f8a77ca4700 1 Processor -- start 2022-09-07T21:30:21.907 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.530+0000 7f8a77ca4700 1 -- start start 2022-09-07T21:30:21.907 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.531+0000 7f8a77ca4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a70105860 0x7f8a70105c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:21.907 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.531+0000 7f8a77ca4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a701061b0 con 0x7f8a70105860 2022-09-07T21:30:21.907 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.531+0000 7f8a75a40700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a70105860 0x7f8a70105c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:21.908 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.531+0000 7f8a75a40700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a70105860 0x7f8a70105c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54392/0 (socket says 172.21.15.47:54392) 2022-09-07T21:30:21.909 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.531+0000 7f8a75a40700 1 -- 172.21.15.47:0/963109795 learned_addr learned my addr 172.21.15.47:0/963109795 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:21.909 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.531+0000 7f8a75a40700 1 -- 172.21.15.47:0/963109795 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a701062f0 con 0x7f8a70105860 2022-09-07T21:30:21.909 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.532+0000 7f8a75a40700 1 --2- 172.21.15.47:0/963109795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a70105860 0x7f8a70105c70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c009700 tx=0x7f8a6c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882fcb80d2f5ebe1 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:21.910 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.532+0000 7f8a74a3e700 1 -- 172.21.15.47:0/963109795 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a6c004160 con 0x7f8a70105860 2022-09-07T21:30:21.910 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.532+0000 7f8a74a3e700 1 -- 172.21.15.47:0/963109795 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f8a6c0042c0 con 0x7f8a70105860 2022-09-07T21:30:21.910 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.532+0000 7f8a74a3e700 1 -- 172.21.15.47:0/963109795 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a6c0039d0 con 0x7f8a70105860 2022-09-07T21:30:21.910 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.533+0000 7f8a77ca4700 1 -- 172.21.15.47:0/963109795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a70105860 msgr2=0x7f8a70105c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.533+0000 7f8a77ca4700 1 --2- 172.21.15.47:0/963109795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a70105860 0x7f8a70105c70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c009700 tx=0x7f8a6c009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.533+0000 7f8a77ca4700 1 -- 172.21.15.47:0/963109795 shutdown_connections 2022-09-07T21:30:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.533+0000 7f8a77ca4700 1 --2- 172.21.15.47:0/963109795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a70105860 0x7f8a70105c70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.533+0000 7f8a77ca4700 1 -- 172.21.15.47:0/963109795 >> 172.21.15.47:0/963109795 conn(0x7f8a701013c0 msgr2=0x7f8a701037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:21.912 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.533+0000 7f8a77ca4700 1 -- 172.21.15.47:0/963109795 shutdown_connections 2022-09-07T21:30:21.912 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.533+0000 7f8a77ca4700 1 -- 172.21.15.47:0/963109795 wait complete. 2022-09-07T21:30:21.912 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.534+0000 7f8a77ca4700 1 Processor -- start 2022-09-07T21:30:21.912 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.534+0000 7f8a77ca4700 1 -- start start 2022-09-07T21:30:21.913 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.535+0000 7f8a77ca4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a70105860 0x7f8a70199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:21.913 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.535+0000 7f8a77ca4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a70199790 con 0x7f8a70105860 2022-09-07T21:30:21.913 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.535+0000 7f8a75a40700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a70105860 0x7f8a70199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:21.914 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.535+0000 7f8a75a40700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a70105860 0x7f8a70199250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54408/0 (socket says 172.21.15.47:54408) 2022-09-07T21:30:21.914 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.535+0000 7f8a75a40700 1 -- 172.21.15.47:0/3098742299 learned_addr learned my addr 172.21.15.47:0/3098742299 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:21.914 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.535+0000 7f8a75a40700 1 -- 172.21.15.47:0/3098742299 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6c0093e0 con 0x7f8a70105860 2022-09-07T21:30:21.914 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.536+0000 7f8a75a40700 1 --2- 172.21.15.47:0/3098742299 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a70105860 0x7f8a70199250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c003fb0 tx=0x7f8a6c004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:21.915 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.536+0000 7f8a5effd700 1 -- 172.21.15.47:0/3098742299 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a6c0118c0 con 0x7f8a70105860 2022-09-07T21:30:21.915 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.536+0000 7f8a5effd700 1 -- 172.21.15.47:0/3098742299 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f8a6c011a20 con 0x7f8a70105860 2022-09-07T21:30:21.915 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.536+0000 7f8a5effd700 1 -- 172.21.15.47:0/3098742299 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a6c019550 con 0x7f8a70105860 2022-09-07T21:30:21.915 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.536+0000 7f8a77ca4700 1 -- 172.21.15.47:0/3098742299 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a70199990 con 0x7f8a70105860 2022-09-07T21:30:21.916 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.536+0000 7f8a77ca4700 1 -- 172.21.15.47:0/3098742299 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a70199dc0 con 0x7f8a70105860 2022-09-07T21:30:21.916 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.537+0000 7f8a5effd700 1 -- 172.21.15.47:0/3098742299 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f8a6c02a020 con 0x7f8a70105860 2022-09-07T21:30:21.916 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.537+0000 7f8a77ca4700 1 -- 172.21.15.47:0/3098742299 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a70060310 con 0x7f8a70105860 2022-09-07T21:30:21.916 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.538+0000 7f8a5effd700 1 --2- 172.21.15.47:0/3098742299 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f8a580358d0 0x7f8a58037d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:21.917 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.538+0000 7f8a5effd700 1 -- 172.21.15.47:0/3098742299 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8a6c0489e0 con 0x7f8a70105860 2022-09-07T21:30:21.917 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.538+0000 7f8a7523f700 1 --2- 172.21.15.47:0/3098742299 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f8a580358d0 0x7f8a58037d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:21.917 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.539+0000 7f8a7523f700 1 --2- 172.21.15.47:0/3098742299 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f8a580358d0 0x7f8a58037d80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009750 tx=0x7f8a60006cc0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:21.917 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.543+0000 7f8a5effd700 1 -- 172.21.15.47:0/3098742299 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f8a6c018d20 con 0x7f8a70105860 2022-09-07T21:30:21.918 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:20.766+0000 7f8a77ca4700 1 -- 172.21.15.47:0/3098742299 --> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a701036f0 con 0x7f8a580358d0 2022-09-07T21:30:21.918 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:21.334+0000 7f8a5effd700 1 -- 172.21.15.47:0/3098742299 <== mgr.14120 v2:172.21.15.47:6800/4097349239 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8a701036f0 con 0x7f8a580358d0 2022-09-07T21:30:21.918 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:21.336+0000 7f8a77ca4700 1 -- 172.21.15.47:0/3098742299 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f8a580358d0 msgr2=0x7f8a58037d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:21.919 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:21.336+0000 7f8a77ca4700 1 --2- 172.21.15.47:0/3098742299 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f8a580358d0 0x7f8a58037d80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009750 tx=0x7f8a60006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:21.919 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:21.336+0000 7f8a77ca4700 1 -- 172.21.15.47:0/3098742299 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a70105860 msgr2=0x7f8a70199250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:21.919 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:21.336+0000 7f8a77ca4700 1 --2- 172.21.15.47:0/3098742299 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a70105860 0x7f8a70199250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c003fb0 tx=0x7f8a6c004090 comp rx=0 tx=0).stop 2022-09-07T21:30:21.919 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:21.336+0000 7f8a77ca4700 1 -- 172.21.15.47:0/3098742299 shutdown_connections 2022-09-07T21:30:21.920 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:21.336+0000 7f8a77ca4700 1 --2- 172.21.15.47:0/3098742299 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f8a580358d0 0x7f8a58037d80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:21.920 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:21.336+0000 7f8a77ca4700 1 --2- 172.21.15.47:0/3098742299 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a70105860 0x7f8a70199250 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:21.921 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:21.336+0000 7f8a77ca4700 1 -- 172.21.15.47:0/3098742299 >> 172.21.15.47:0/3098742299 conn(0x7f8a701013c0 msgr2=0x7f8a70102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:21.921 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:21.336+0000 7f8a77ca4700 1 -- 172.21.15.47:0/3098742299 shutdown_connections 2022-09-07T21:30:21.921 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:21.337+0000 7f8a77ca4700 1 -- 172.21.15.47:0/3098742299 wait complete. 2022-09-07T21:30:22.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:22 smithi047 ceph-mon[90087]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:30:22.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:22 smithi047 ceph-mon[90087]: Generating ssh key... 2022-09-07T21:30:22.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:22 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' 2022-09-07T21:30:22.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:22 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' 2022-09-07T21:30:23.308 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzw4GGLUr/epxtyWeQ0CCRdUvr7BLDy3yWP/sNTdatpsNGQQccVs3gNrK4cijTfxRelvBxTypbT7X40DfhoTAOLgUQ0LwEVCMKI6E1zrZogUvO0E+w+3hEc91fafY8hQTBEpc5zwqhQsM7CwdRhg61JV+d7Gry9XxZSZ/4RkwcXblVWZ2st48FUdl9fGPOKGy2UK3b7D3cA1Jxpd0cFlSBpjqHYoTB2MTsjj0pc8BeVsn1VyQ11xvlFpGGKZSWXLSXJMTpoc01Gn9LHtowt51gK8x6bGdZqVq8ntkKnemQBZWOjUPKNBD442kYDl7z5iCo812w3IfCkidUfQk9CAfAegnZhmw5VDihPfN7kSt4Drtak2cd20jLbYvrACZxH+Pt8fH8RTe5D2OPWZRFCCmlWLR/w1HMTJ/ABlFnD+aPOpdcRbkNpXSPpvctS5Bmy4cINS3IluvqPZyqqT0NA0i7bzkoN3YjvJEjGMZkFhy1Wr4CzMThtIiPVC0jIcyg95c= ceph-01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:30:23.309 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.556+0000 7f611771a700 1 Processor -- start 2022-09-07T21:30:23.309 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.556+0000 7f611771a700 1 -- start start 2022-09-07T21:30:23.309 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.557+0000 7f611771a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6110071e80 0x7f6110106270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:23.309 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.557+0000 7f611771a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61101067b0 con 0x7f6110071e80 2022-09-07T21:30:23.310 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.557+0000 7f61154b6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6110071e80 0x7f6110106270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:23.310 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.557+0000 7f61154b6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6110071e80 0x7f6110106270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54418/0 (socket says 172.21.15.47:54418) 2022-09-07T21:30:23.310 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.557+0000 7f61154b6700 1 -- 172.21.15.47:0/1827644487 learned_addr learned my addr 172.21.15.47:0/1827644487 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:23.310 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.558+0000 7f61154b6700 1 -- 172.21.15.47:0/1827644487 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61101068f0 con 0x7f6110071e80 2022-09-07T21:30:23.311 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.558+0000 7f61154b6700 1 --2- 172.21.15.47:0/1827644487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6110071e80 0x7f6110106270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f610c009700 tx=0x7f610c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9712bf0d433188e9 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:23.311 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.558+0000 7f60fffff700 1 -- 172.21.15.47:0/1827644487 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f610c004160 con 0x7f6110071e80 2022-09-07T21:30:23.311 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.558+0000 7f60fffff700 1 -- 172.21.15.47:0/1827644487 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f610c0042c0 con 0x7f6110071e80 2022-09-07T21:30:23.311 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.558+0000 7f60fffff700 1 -- 172.21.15.47:0/1827644487 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f610c0039d0 con 0x7f6110071e80 2022-09-07T21:30:23.311 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.559+0000 7f611771a700 1 -- 172.21.15.47:0/1827644487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6110071e80 msgr2=0x7f6110106270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:23.312 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.559+0000 7f611771a700 1 --2- 172.21.15.47:0/1827644487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6110071e80 0x7f6110106270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f610c009700 tx=0x7f610c009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:23.312 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.559+0000 7f611771a700 1 -- 172.21.15.47:0/1827644487 shutdown_connections 2022-09-07T21:30:23.312 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.559+0000 7f611771a700 1 --2- 172.21.15.47:0/1827644487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6110071e80 0x7f6110106270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:23.313 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.559+0000 7f611771a700 1 -- 172.21.15.47:0/1827644487 >> 172.21.15.47:0/1827644487 conn(0x7f61100fdc20 msgr2=0x7f6110100050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:23.313 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.559+0000 7f611771a700 1 -- 172.21.15.47:0/1827644487 shutdown_connections 2022-09-07T21:30:23.313 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.559+0000 7f611771a700 1 -- 172.21.15.47:0/1827644487 wait complete. 2022-09-07T21:30:23.314 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.560+0000 7f611771a700 1 Processor -- start 2022-09-07T21:30:23.314 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.561+0000 7f611771a700 1 -- start start 2022-09-07T21:30:23.314 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.561+0000 7f611771a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6110071e80 0x7f6110199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:23.314 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.561+0000 7f611771a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6110199780 con 0x7f6110071e80 2022-09-07T21:30:23.315 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.561+0000 7f61154b6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6110071e80 0x7f6110199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:23.315 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.561+0000 7f61154b6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6110071e80 0x7f6110199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54430/0 (socket says 172.21.15.47:54430) 2022-09-07T21:30:23.315 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.561+0000 7f61154b6700 1 -- 172.21.15.47:0/1984230407 learned_addr learned my addr 172.21.15.47:0/1984230407 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:23.316 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.561+0000 7f61154b6700 1 -- 172.21.15.47:0/1984230407 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610c0093e0 con 0x7f6110071e80 2022-09-07T21:30:23.316 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.562+0000 7f61154b6700 1 --2- 172.21.15.47:0/1984230407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6110071e80 0x7f6110199240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f610c009e90 tx=0x7f610c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:23.316 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.562+0000 7f60fe7fc700 1 -- 172.21.15.47:0/1984230407 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f610c011840 con 0x7f6110071e80 2022-09-07T21:30:23.316 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.562+0000 7f60fe7fc700 1 -- 172.21.15.47:0/1984230407 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f610c01a430 con 0x7f6110071e80 2022-09-07T21:30:23.317 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.562+0000 7f60fe7fc700 1 -- 172.21.15.47:0/1984230407 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f610c019550 con 0x7f6110071e80 2022-09-07T21:30:23.317 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.562+0000 7f611771a700 1 -- 172.21.15.47:0/1984230407 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6110199980 con 0x7f6110071e80 2022-09-07T21:30:23.317 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.562+0000 7f611771a700 1 -- 172.21.15.47:0/1984230407 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6110199db0 con 0x7f6110071e80 2022-09-07T21:30:23.318 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.563+0000 7f60fe7fc700 1 -- 172.21.15.47:0/1984230407 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f610c02a020 con 0x7f6110071e80 2022-09-07T21:30:23.318 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.563+0000 7f611771a700 1 -- 172.21.15.47:0/1984230407 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6110060310 con 0x7f6110071e80 2022-09-07T21:30:23.318 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.564+0000 7f60fe7fc700 1 --2- 172.21.15.47:0/1984230407 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f60f8035890 0x7f60f8037d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:23.318 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.564+0000 7f60fe7fc700 1 -- 172.21.15.47:0/1984230407 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f610c048a70 con 0x7f6110071e80 2022-09-07T21:30:23.319 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.564+0000 7f6114cb5700 1 --2- 172.21.15.47:0/1984230407 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f60f8035890 0x7f60f8037d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:23.319 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.565+0000 7f6114cb5700 1 --2- 172.21.15.47:0/1984230407 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f60f8035890 0x7f60f8037d40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6100009750 tx=0x7f6100006cc0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:23.319 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.569+0000 7f60fe7fc700 1 -- 172.21.15.47:0/1984230407 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f610c0119a0 con 0x7f6110071e80 2022-09-07T21:30:23.319 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.796+0000 7f611771a700 1 -- 172.21.15.47:0/1984230407 --> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f61100fef90 con 0x7f60f8035890 2022-09-07T21:30:23.320 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.797+0000 7f60fe7fc700 1 -- 172.21.15.47:0/1984230407 <== mgr.14120 v2:172.21.15.47:6800/4097349239 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f61100fef90 con 0x7f60f8035890 2022-09-07T21:30:23.320 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.799+0000 7f611771a700 1 -- 172.21.15.47:0/1984230407 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f60f8035890 msgr2=0x7f60f8037d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:23.320 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.799+0000 7f611771a700 1 --2- 172.21.15.47:0/1984230407 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f60f8035890 0x7f60f8037d40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6100009750 tx=0x7f6100006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:23.320 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.799+0000 7f611771a700 1 -- 172.21.15.47:0/1984230407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6110071e80 msgr2=0x7f6110199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:23.321 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.799+0000 7f611771a700 1 --2- 172.21.15.47:0/1984230407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6110071e80 0x7f6110199240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f610c009e90 tx=0x7f610c004040 comp rx=0 tx=0).stop 2022-09-07T21:30:23.321 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.799+0000 7f611771a700 1 -- 172.21.15.47:0/1984230407 shutdown_connections 2022-09-07T21:30:23.321 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.799+0000 7f611771a700 1 --2- 172.21.15.47:0/1984230407 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f60f8035890 0x7f60f8037d40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:23.321 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.799+0000 7f611771a700 1 --2- 172.21.15.47:0/1984230407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6110071e80 0x7f6110199240 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:23.321 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.799+0000 7f611771a700 1 -- 172.21.15.47:0/1984230407 >> 172.21.15.47:0/1984230407 conn(0x7f61100fdc20 msgr2=0x7f61100fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:23.322 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.800+0000 7f611771a700 1 -- 172.21.15.47:0/1984230407 shutdown_connections 2022-09-07T21:30:23.322 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:22.800+0000 7f611771a700 1 -- 172.21.15.47:0/1984230407 wait complete. 2022-09-07T21:30:23.322 INFO:teuthology.orchestra.run.smithi047.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2022-09-07T21:30:23.322 INFO:teuthology.orchestra.run.smithi047.stdout:Adding key to root@localhost authorized_keys... 2022-09-07T21:30:23.323 INFO:teuthology.orchestra.run.smithi047.stdout:Adding host smithi047... 2022-09-07T21:30:23.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:23 smithi047 ceph-mon[90087]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:30:25.055 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:24 smithi047 ceph-mon[90087]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi047", "addr": "172.21.15.47", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:30:25.884 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:25 smithi047 ceph-mon[90087]: Deploying cephadm binary to smithi047 2022-09-07T21:30:28.329 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout Added host 'smithi047' with addr '172.21.15.47' 2022-09-07T21:30:28.330 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.951+0000 7f23d16ae700 1 Processor -- start 2022-09-07T21:30:28.330 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.952+0000 7f23d16ae700 1 -- start start 2022-09-07T21:30:28.331 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.952+0000 7f23d16ae700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23cc071e80 0x7f23cc108670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:28.331 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.952+0000 7f23d16ae700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc104060 con 0x7f23cc071e80 2022-09-07T21:30:28.331 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.952+0000 7f23caffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23cc071e80 0x7f23cc108670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:28.332 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.953+0000 7f23caffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23cc071e80 0x7f23cc108670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54434/0 (socket says 172.21.15.47:54434) 2022-09-07T21:30:28.332 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.953+0000 7f23caffd700 1 -- 172.21.15.47:0/1487062310 learned_addr learned my addr 172.21.15.47:0/1487062310 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:28.333 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.953+0000 7f23caffd700 1 -- 172.21.15.47:0/1487062310 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc108bb0 con 0x7f23cc071e80 2022-09-07T21:30:28.333 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.953+0000 7f23caffd700 1 --2- 172.21.15.47:0/1487062310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23cc071e80 0x7f23cc108670 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009700 tx=0x7f23bc009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59da5ae0b5a0cace server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:28.334 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.954+0000 7f23c9ffb700 1 -- 172.21.15.47:0/1487062310 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f23bc004160 con 0x7f23cc071e80 2022-09-07T21:30:28.334 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.954+0000 7f23c9ffb700 1 -- 172.21.15.47:0/1487062310 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f23bc0042c0 con 0x7f23cc071e80 2022-09-07T21:30:28.335 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.954+0000 7f23c9ffb700 1 -- 172.21.15.47:0/1487062310 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f23bc0039d0 con 0x7f23cc071e80 2022-09-07T21:30:28.335 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.954+0000 7f23d16ae700 1 -- 172.21.15.47:0/1487062310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23cc071e80 msgr2=0x7f23cc108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:28.335 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.954+0000 7f23d16ae700 1 --2- 172.21.15.47:0/1487062310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23cc071e80 0x7f23cc108670 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009700 tx=0x7f23bc009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:28.336 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.954+0000 7f23d16ae700 1 -- 172.21.15.47:0/1487062310 shutdown_connections 2022-09-07T21:30:28.336 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.954+0000 7f23d16ae700 1 --2- 172.21.15.47:0/1487062310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23cc071e80 0x7f23cc108670 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:28.337 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.954+0000 7f23d16ae700 1 -- 172.21.15.47:0/1487062310 >> 172.21.15.47:0/1487062310 conn(0x7f23cc0fdc20 msgr2=0x7f23cc100050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:28.337 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.954+0000 7f23d16ae700 1 -- 172.21.15.47:0/1487062310 shutdown_connections 2022-09-07T21:30:28.338 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.955+0000 7f23d16ae700 1 -- 172.21.15.47:0/1487062310 wait complete. 2022-09-07T21:30:28.338 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.956+0000 7f23d16ae700 1 Processor -- start 2022-09-07T21:30:28.339 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.956+0000 7f23d16ae700 1 -- start start 2022-09-07T21:30:28.339 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.956+0000 7f23d16ae700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23cc071e80 0x7f23cc199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:28.340 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.956+0000 7f23d16ae700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc199780 con 0x7f23cc071e80 2022-09-07T21:30:28.340 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.957+0000 7f23caffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23cc071e80 0x7f23cc199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:28.341 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.957+0000 7f23caffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23cc071e80 0x7f23cc199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54450/0 (socket says 172.21.15.47:54450) 2022-09-07T21:30:28.341 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.957+0000 7f23caffd700 1 -- 172.21.15.47:0/2348517691 learned_addr learned my addr 172.21.15.47:0/2348517691 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:28.341 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.957+0000 7f23caffd700 1 -- 172.21.15.47:0/2348517691 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc0093e0 con 0x7f23cc071e80 2022-09-07T21:30:28.342 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.957+0000 7f23caffd700 1 --2- 172.21.15.47:0/2348517691 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23cc071e80 0x7f23cc199240 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009e90 tx=0x7f23bc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:28.342 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.958+0000 7f23b3fff700 1 -- 172.21.15.47:0/2348517691 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f23bc011840 con 0x7f23cc071e80 2022-09-07T21:30:28.343 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.958+0000 7f23d16ae700 1 -- 172.21.15.47:0/2348517691 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23cc199980 con 0x7f23cc071e80 2022-09-07T21:30:28.343 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.958+0000 7f23b3fff700 1 -- 172.21.15.47:0/2348517691 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f23bc01a430 con 0x7f23cc071e80 2022-09-07T21:30:28.344 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.958+0000 7f23b3fff700 1 -- 172.21.15.47:0/2348517691 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f23bc019550 con 0x7f23cc071e80 2022-09-07T21:30:28.344 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.958+0000 7f23d16ae700 1 -- 172.21.15.47:0/2348517691 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc199db0 con 0x7f23cc071e80 2022-09-07T21:30:28.345 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.959+0000 7f23b3fff700 1 -- 172.21.15.47:0/2348517691 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f23bc02a020 con 0x7f23cc071e80 2022-09-07T21:30:28.345 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.959+0000 7f23d16ae700 1 -- 172.21.15.47:0/2348517691 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23cc060310 con 0x7f23cc071e80 2022-09-07T21:30:28.345 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.959+0000 7f23b3fff700 1 --2- 172.21.15.47:0/2348517691 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f23b80358d0 0x7f23b8037d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:28.346 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.959+0000 7f23b3fff700 1 -- 172.21.15.47:0/2348517691 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f23bc048a70 con 0x7f23cc071e80 2022-09-07T21:30:28.346 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.960+0000 7f23ca7fc700 1 --2- 172.21.15.47:0/2348517691 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f23b80358d0 0x7f23b8037d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:28.347 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.960+0000 7f23ca7fc700 1 --2- 172.21.15.47:0/2348517691 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f23b80358d0 0x7f23b8037d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f23c0009750 tx=0x7f23c0006cc0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:28.347 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:23.965+0000 7f23b3fff700 1 -- 172.21.15.47:0/2348517691 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f23bc0119a0 con 0x7f23cc071e80 2022-09-07T21:30:28.348 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:24.193+0000 7f23d16ae700 1 -- 172.21.15.47:0/2348517691 --> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi047", "addr": "172.21.15.47", "target": ["mon-mgr", ""]}) v1 -- 0x7f23cc0fefb0 con 0x7f23b80358d0 2022-09-07T21:30:28.348 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:27.642+0000 7f23b3fff700 1 -- 172.21.15.47:0/2348517691 <== mgr.14120 v2:172.21.15.47:6800/4097349239 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f23cc0fefb0 con 0x7f23b80358d0 2022-09-07T21:30:28.349 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:27.645+0000 7f23d16ae700 1 -- 172.21.15.47:0/2348517691 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f23b80358d0 msgr2=0x7f23b8037d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:28.349 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:27.645+0000 7f23d16ae700 1 --2- 172.21.15.47:0/2348517691 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f23b80358d0 0x7f23b8037d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f23c0009750 tx=0x7f23c0006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:28.350 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:27.645+0000 7f23d16ae700 1 -- 172.21.15.47:0/2348517691 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23cc071e80 msgr2=0x7f23cc199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:28.350 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:27.645+0000 7f23d16ae700 1 --2- 172.21.15.47:0/2348517691 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23cc071e80 0x7f23cc199240 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009e90 tx=0x7f23bc004040 comp rx=0 tx=0).stop 2022-09-07T21:30:28.350 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:27.645+0000 7f23d16ae700 1 -- 172.21.15.47:0/2348517691 shutdown_connections 2022-09-07T21:30:28.351 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:27.645+0000 7f23d16ae700 1 --2- 172.21.15.47:0/2348517691 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f23b80358d0 0x7f23b8037d80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:28.351 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:27.645+0000 7f23d16ae700 1 --2- 172.21.15.47:0/2348517691 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23cc071e80 0x7f23cc199240 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:28.352 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:27.645+0000 7f23d16ae700 1 -- 172.21.15.47:0/2348517691 >> 172.21.15.47:0/2348517691 conn(0x7f23cc0fdc20 msgr2=0x7f23cc0fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:28.352 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:27.646+0000 7f23d16ae700 1 -- 172.21.15.47:0/2348517691 shutdown_connections 2022-09-07T21:30:28.353 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:27.646+0000 7f23d16ae700 1 -- 172.21.15.47:0/2348517691 wait complete. 2022-09-07T21:30:28.353 INFO:teuthology.orchestra.run.smithi047.stdout:Deploying unmanaged mon service... 2022-09-07T21:30:28.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:28 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' 2022-09-07T21:30:28.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:28 smithi047 ceph-mon[90087]: Added host smithi047 2022-09-07T21:30:28.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:28 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:30:29.732 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2022-09-07T21:30:29.744 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.969+0000 7f1bfd152700 1 Processor -- start 2022-09-07T21:30:29.745 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.969+0000 7f1bfd152700 1 -- start start 2022-09-07T21:30:29.745 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.970+0000 7f1bfd152700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bf8071e80 0x7f1bf8108670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:29.745 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.970+0000 7f1bfd152700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf8104060 con 0x7f1bf8071e80 2022-09-07T21:30:29.746 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.970+0000 7f1bf6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bf8071e80 0x7f1bf8108670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:29.746 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.970+0000 7f1bf6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bf8071e80 0x7f1bf8108670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53586/0 (socket says 172.21.15.47:53586) 2022-09-07T21:30:29.747 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.970+0000 7f1bf6d9d700 1 -- 172.21.15.47:0/3629688140 learned_addr learned my addr 172.21.15.47:0/3629688140 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:29.747 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.971+0000 7f1bf6d9d700 1 -- 172.21.15.47:0/3629688140 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf8108bb0 con 0x7f1bf8071e80 2022-09-07T21:30:29.747 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.971+0000 7f1bf6d9d700 1 --2- 172.21.15.47:0/3629688140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bf8071e80 0x7f1bf8108670 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009700 tx=0x7f1be8009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9154b8955cf43c55 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:29.748 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.971+0000 7f1bf5d9b700 1 -- 172.21.15.47:0/3629688140 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1be8004160 con 0x7f1bf8071e80 2022-09-07T21:30:29.748 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.971+0000 7f1bf5d9b700 1 -- 172.21.15.47:0/3629688140 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f1be80042c0 con 0x7f1bf8071e80 2022-09-07T21:30:29.748 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.971+0000 7f1bf5d9b700 1 -- 172.21.15.47:0/3629688140 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1be80039d0 con 0x7f1bf8071e80 2022-09-07T21:30:29.748 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.972+0000 7f1bfd152700 1 -- 172.21.15.47:0/3629688140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bf8071e80 msgr2=0x7f1bf8108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:29.749 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.972+0000 7f1bfd152700 1 --2- 172.21.15.47:0/3629688140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bf8071e80 0x7f1bf8108670 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009700 tx=0x7f1be8009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:29.749 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.972+0000 7f1bfd152700 1 -- 172.21.15.47:0/3629688140 shutdown_connections 2022-09-07T21:30:29.749 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.972+0000 7f1bfd152700 1 --2- 172.21.15.47:0/3629688140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bf8071e80 0x7f1bf8108670 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:29.750 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.972+0000 7f1bfd152700 1 -- 172.21.15.47:0/3629688140 >> 172.21.15.47:0/3629688140 conn(0x7f1bf80fdc20 msgr2=0x7f1bf8100050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:29.750 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.972+0000 7f1bfd152700 1 -- 172.21.15.47:0/3629688140 shutdown_connections 2022-09-07T21:30:29.750 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.972+0000 7f1bfd152700 1 -- 172.21.15.47:0/3629688140 wait complete. 2022-09-07T21:30:29.750 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.973+0000 7f1bfd152700 1 Processor -- start 2022-09-07T21:30:29.751 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.973+0000 7f1bfd152700 1 -- start start 2022-09-07T21:30:29.751 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.974+0000 7f1bfd152700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bf8071e80 0x7f1bf8199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:29.751 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.974+0000 7f1bfd152700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf8199780 con 0x7f1bf8071e80 2022-09-07T21:30:29.751 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.974+0000 7f1bf6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bf8071e80 0x7f1bf8199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:29.751 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.974+0000 7f1bf6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bf8071e80 0x7f1bf8199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53590/0 (socket says 172.21.15.47:53590) 2022-09-07T21:30:29.752 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.974+0000 7f1bf6d9d700 1 -- 172.21.15.47:0/387690235 learned_addr learned my addr 172.21.15.47:0/387690235 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:29.752 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.974+0000 7f1bf6d9d700 1 -- 172.21.15.47:0/387690235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be80093e0 con 0x7f1bf8071e80 2022-09-07T21:30:29.752 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.975+0000 7f1bf6d9d700 1 --2- 172.21.15.47:0/387690235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bf8071e80 0x7f1bf8199240 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009e90 tx=0x7f1be8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:29.752 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.975+0000 7f1bdffff700 1 -- 172.21.15.47:0/387690235 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1be8011840 con 0x7f1bf8071e80 2022-09-07T21:30:29.753 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.975+0000 7f1bfd152700 1 -- 172.21.15.47:0/387690235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf8199980 con 0x7f1bf8071e80 2022-09-07T21:30:29.753 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.975+0000 7f1bdffff700 1 -- 172.21.15.47:0/387690235 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f1be801a430 con 0x7f1bf8071e80 2022-09-07T21:30:29.753 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.975+0000 7f1bdffff700 1 -- 172.21.15.47:0/387690235 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1be8019550 con 0x7f1bf8071e80 2022-09-07T21:30:29.753 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.975+0000 7f1bfd152700 1 -- 172.21.15.47:0/387690235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf8199db0 con 0x7f1bf8071e80 2022-09-07T21:30:29.753 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.977+0000 7f1bdffff700 1 -- 172.21.15.47:0/387690235 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f1be802a020 con 0x7f1bf8071e80 2022-09-07T21:30:29.754 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.977+0000 7f1bfd152700 1 -- 172.21.15.47:0/387690235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf8060310 con 0x7f1bf8071e80 2022-09-07T21:30:29.754 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.977+0000 7f1bdffff700 1 --2- 172.21.15.47:0/387690235 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f1be4035890 0x7f1be4037d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:29.754 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.977+0000 7f1bdffff700 1 -- 172.21.15.47:0/387690235 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1be80489e0 con 0x7f1bf8071e80 2022-09-07T21:30:29.754 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.977+0000 7f1bf659c700 1 --2- 172.21.15.47:0/387690235 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f1be4035890 0x7f1be4037d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:29.755 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.978+0000 7f1bf659c700 1 --2- 172.21.15.47:0/387690235 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f1be4035890 0x7f1be4037d40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009750 tx=0x7f1bec006cc0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:29.755 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:28.982+0000 7f1bdffff700 1 -- 172.21.15.47:0/387690235 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f1be80119a0 con 0x7f1bf8071e80 2022-09-07T21:30:29.755 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:29.221+0000 7f1bfd152700 1 -- 172.21.15.47:0/387690235 --> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1bf80fefb0 con 0x7f1be4035890 2022-09-07T21:30:29.756 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:29.226+0000 7f1bdffff700 1 -- 172.21.15.47:0/387690235 <== mgr.14120 v2:172.21.15.47:6800/4097349239 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1bf80fefb0 con 0x7f1be4035890 2022-09-07T21:30:29.756 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:29.228+0000 7f1bfd152700 1 -- 172.21.15.47:0/387690235 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f1be4035890 msgr2=0x7f1be4037d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:29.756 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:29.228+0000 7f1bfd152700 1 --2- 172.21.15.47:0/387690235 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f1be4035890 0x7f1be4037d40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009750 tx=0x7f1bec006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:29.756 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:29.228+0000 7f1bfd152700 1 -- 172.21.15.47:0/387690235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bf8071e80 msgr2=0x7f1bf8199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:29.757 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:29.228+0000 7f1bfd152700 1 --2- 172.21.15.47:0/387690235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bf8071e80 0x7f1bf8199240 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009e90 tx=0x7f1be8004040 comp rx=0 tx=0).stop 2022-09-07T21:30:29.757 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:29.228+0000 7f1bfd152700 1 -- 172.21.15.47:0/387690235 shutdown_connections 2022-09-07T21:30:29.757 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:29.228+0000 7f1bfd152700 1 --2- 172.21.15.47:0/387690235 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f1be4035890 0x7f1be4037d40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:29.757 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:29.228+0000 7f1bfd152700 1 --2- 172.21.15.47:0/387690235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bf8071e80 0x7f1bf8199240 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:29.757 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:29.229+0000 7f1bfd152700 1 -- 172.21.15.47:0/387690235 >> 172.21.15.47:0/387690235 conn(0x7f1bf80fdc20 msgr2=0x7f1bf80fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:29.758 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:29.229+0000 7f1bfd152700 1 -- 172.21.15.47:0/387690235 shutdown_connections 2022-09-07T21:30:29.758 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:29.229+0000 7f1bfd152700 1 -- 172.21.15.47:0/387690235 wait complete. 2022-09-07T21:30:29.758 INFO:teuthology.orchestra.run.smithi047.stdout:Deploying unmanaged mgr service... 2022-09-07T21:30:30.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:30 smithi047 ceph-mon[90087]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:30:30.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:30 smithi047 ceph-mon[90087]: Saving service mon spec with placement count:5 2022-09-07T21:30:30.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:30 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' 2022-09-07T21:30:31.146 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2022-09-07T21:30:31.147 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.367+0000 7f3ccfaeb700 1 Processor -- start 2022-09-07T21:30:31.147 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.367+0000 7f3ccfaeb700 1 -- start start 2022-09-07T21:30:31.147 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.367+0000 7f3ccfaeb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cc8105850 0x7f3cc8105c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:31.147 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.367+0000 7f3ccfaeb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc81061a0 con 0x7f3cc8105850 2022-09-07T21:30:31.148 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.368+0000 7f3ccd887700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cc8105850 0x7f3cc8105c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:31.148 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.368+0000 7f3ccd887700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cc8105850 0x7f3cc8105c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53602/0 (socket says 172.21.15.47:53602) 2022-09-07T21:30:31.148 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.368+0000 7f3ccd887700 1 -- 172.21.15.47:0/1543725202 learned_addr learned my addr 172.21.15.47:0/1543725202 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:31.148 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.368+0000 7f3ccd887700 1 -- 172.21.15.47:0/1543725202 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc81062e0 con 0x7f3cc8105850 2022-09-07T21:30:31.149 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.369+0000 7f3ccd887700 1 --2- 172.21.15.47:0/1543725202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cc8105850 0x7f3cc8105c60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009700 tx=0x7f3cc4009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d4f22c6f7d2fc server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:31.149 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.369+0000 7f3ccc885700 1 -- 172.21.15.47:0/1543725202 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cc4004160 con 0x7f3cc8105850 2022-09-07T21:30:31.149 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.369+0000 7f3ccc885700 1 -- 172.21.15.47:0/1543725202 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f3cc40042c0 con 0x7f3cc8105850 2022-09-07T21:30:31.149 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.369+0000 7f3ccc885700 1 -- 172.21.15.47:0/1543725202 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cc40039d0 con 0x7f3cc8105850 2022-09-07T21:30:31.150 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.369+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/1543725202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cc8105850 msgr2=0x7f3cc8105c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:31.150 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.369+0000 7f3ccfaeb700 1 --2- 172.21.15.47:0/1543725202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cc8105850 0x7f3cc8105c60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009700 tx=0x7f3cc4009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:31.150 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.370+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/1543725202 shutdown_connections 2022-09-07T21:30:31.150 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.370+0000 7f3ccfaeb700 1 --2- 172.21.15.47:0/1543725202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cc8105850 0x7f3cc8105c60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:31.151 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.370+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/1543725202 >> 172.21.15.47:0/1543725202 conn(0x7f3cc8078d90 msgr2=0x7f3cc8079190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:31.151 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.370+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/1543725202 shutdown_connections 2022-09-07T21:30:31.151 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.370+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/1543725202 wait complete. 2022-09-07T21:30:31.151 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.371+0000 7f3ccfaeb700 1 Processor -- start 2022-09-07T21:30:31.151 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.371+0000 7f3ccfaeb700 1 -- start start 2022-09-07T21:30:31.152 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.371+0000 7f3ccfaeb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cc8105850 0x7f3cc8199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:31.152 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.371+0000 7f3ccfaeb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc81997a0 con 0x7f3cc8105850 2022-09-07T21:30:31.152 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.372+0000 7f3ccd887700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cc8105850 0x7f3cc8199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:31.152 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.372+0000 7f3ccd887700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cc8105850 0x7f3cc8199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53606/0 (socket says 172.21.15.47:53606) 2022-09-07T21:30:31.153 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.372+0000 7f3ccd887700 1 -- 172.21.15.47:0/3897540160 learned_addr learned my addr 172.21.15.47:0/3897540160 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:31.153 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.372+0000 7f3ccd887700 1 -- 172.21.15.47:0/3897540160 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc40093e0 con 0x7f3cc8105850 2022-09-07T21:30:31.153 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.372+0000 7f3ccd887700 1 --2- 172.21.15.47:0/3897540160 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cc8105850 0x7f3cc8199260 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4003f00 tx=0x7f3cc4003fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:31.153 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.373+0000 7f3cb6ffd700 1 -- 172.21.15.47:0/3897540160 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cc4011950 con 0x7f3cc8105850 2022-09-07T21:30:31.154 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.373+0000 7f3cb6ffd700 1 -- 172.21.15.47:0/3897540160 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f3cc4011ab0 con 0x7f3cc8105850 2022-09-07T21:30:31.154 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.373+0000 7f3cb6ffd700 1 -- 172.21.15.47:0/3897540160 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cc4019550 con 0x7f3cc8105850 2022-09-07T21:30:31.155 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.373+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/3897540160 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc81999a0 con 0x7f3cc8105850 2022-09-07T21:30:31.155 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.373+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/3897540160 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc8199dd0 con 0x7f3cc8105850 2022-09-07T21:30:31.155 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.374+0000 7f3cb6ffd700 1 -- 172.21.15.47:0/3897540160 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f3cc402a020 con 0x7f3cc8105850 2022-09-07T21:30:31.155 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.374+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/3897540160 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc8060310 con 0x7f3cc8105850 2022-09-07T21:30:31.156 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.375+0000 7f3cb6ffd700 1 --2- 172.21.15.47:0/3897540160 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f3cb00358f0 0x7f3cb0037da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:31.156 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.375+0000 7f3cb6ffd700 1 -- 172.21.15.47:0/3897540160 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3cc4048b40 con 0x7f3cc8105850 2022-09-07T21:30:31.156 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.375+0000 7f3ccd086700 1 --2- 172.21.15.47:0/3897540160 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f3cb00358f0 0x7f3cb0037da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:31.156 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.375+0000 7f3ccd086700 1 --2- 172.21.15.47:0/3897540160 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f3cb00358f0 0x7f3cb0037da0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009750 tx=0x7f3cb8006cc0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:31.157 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.379+0000 7f3cb6ffd700 1 -- 172.21.15.47:0/3897540160 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f3cc4018b40 con 0x7f3cc8105850 2022-09-07T21:30:31.157 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.621+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/3897540160 --> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3cc8060310 con 0x7f3cb00358f0 2022-09-07T21:30:31.157 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.626+0000 7f3cb6ffd700 1 -- 172.21.15.47:0/3897540160 <== mgr.14120 v2:172.21.15.47:6800/4097349239 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3cc8060310 con 0x7f3cb00358f0 2022-09-07T21:30:31.158 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.628+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/3897540160 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f3cb00358f0 msgr2=0x7f3cb0037da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:31.158 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.628+0000 7f3ccfaeb700 1 --2- 172.21.15.47:0/3897540160 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f3cb00358f0 0x7f3cb0037da0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009750 tx=0x7f3cb8006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:31.158 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.628+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/3897540160 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cc8105850 msgr2=0x7f3cc8199260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:31.159 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.628+0000 7f3ccfaeb700 1 --2- 172.21.15.47:0/3897540160 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cc8105850 0x7f3cc8199260 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4003f00 tx=0x7f3cc4003fe0 comp rx=0 tx=0).stop 2022-09-07T21:30:31.159 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.628+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/3897540160 shutdown_connections 2022-09-07T21:30:31.159 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.628+0000 7f3ccfaeb700 1 --2- 172.21.15.47:0/3897540160 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f3cb00358f0 0x7f3cb0037da0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:31.161 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.628+0000 7f3ccfaeb700 1 --2- 172.21.15.47:0/3897540160 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cc8105850 0x7f3cc8199260 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:31.161 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.628+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/3897540160 >> 172.21.15.47:0/3897540160 conn(0x7f3cc8078d90 msgr2=0x7f3cc8103620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:31.161 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.629+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/3897540160 shutdown_connections 2022-09-07T21:30:31.161 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:30.629+0000 7f3ccfaeb700 1 -- 172.21.15.47:0/3897540160 wait complete. 2022-09-07T21:30:31.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:31 smithi047 ceph-mon[90087]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:30:31.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:31 smithi047 ceph-mon[90087]: Saving service mgr spec with placement count:2 2022-09-07T21:30:31.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:31 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' 2022-09-07T21:30:32.649 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.754+0000 7ff236fac700 1 Processor -- start 2022-09-07T21:30:32.649 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.755+0000 7ff236fac700 1 -- start start 2022-09-07T21:30:32.650 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.755+0000 7ff236fac700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff230078c80 0x7ff230077200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:32.651 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.755+0000 7ff236fac700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff230079090 con 0x7ff230078c80 2022-09-07T21:30:32.651 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.755+0000 7ff234d48700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff230078c80 0x7ff230077200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:32.651 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.755+0000 7ff234d48700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff230078c80 0x7ff230077200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53620/0 (socket says 172.21.15.47:53620) 2022-09-07T21:30:32.651 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.755+0000 7ff234d48700 1 -- 172.21.15.47:0/2119946528 learned_addr learned my addr 172.21.15.47:0/2119946528 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:32.652 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.756+0000 7ff234d48700 1 -- 172.21.15.47:0/2119946528 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff230077740 con 0x7ff230078c80 2022-09-07T21:30:32.652 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.756+0000 7ff234d48700 1 --2- 172.21.15.47:0/2119946528 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff230078c80 0x7ff230077200 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff224009700 tx=0x7ff224009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d892a61fa569d3e4 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:32.652 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.756+0000 7ff22f7fe700 1 -- 172.21.15.47:0/2119946528 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff224004160 con 0x7ff230078c80 2022-09-07T21:30:32.653 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.756+0000 7ff22f7fe700 1 -- 172.21.15.47:0/2119946528 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7ff2240042c0 con 0x7ff230078c80 2022-09-07T21:30:32.653 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.756+0000 7ff22f7fe700 1 -- 172.21.15.47:0/2119946528 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff2240039d0 con 0x7ff230078c80 2022-09-07T21:30:32.653 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.757+0000 7ff236fac700 1 -- 172.21.15.47:0/2119946528 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff230078c80 msgr2=0x7ff230077200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:32.653 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.757+0000 7ff236fac700 1 --2- 172.21.15.47:0/2119946528 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff230078c80 0x7ff230077200 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff224009700 tx=0x7ff224009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:32.654 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.757+0000 7ff236fac700 1 -- 172.21.15.47:0/2119946528 shutdown_connections 2022-09-07T21:30:32.654 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.757+0000 7ff236fac700 1 --2- 172.21.15.47:0/2119946528 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff230078c80 0x7ff230077200 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:32.655 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.757+0000 7ff236fac700 1 -- 172.21.15.47:0/2119946528 >> 172.21.15.47:0/2119946528 conn(0x7ff2300fed50 msgr2=0x7ff230101180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:32.655 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.757+0000 7ff236fac700 1 -- 172.21.15.47:0/2119946528 shutdown_connections 2022-09-07T21:30:32.656 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.757+0000 7ff236fac700 1 -- 172.21.15.47:0/2119946528 wait complete. 2022-09-07T21:30:32.656 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.758+0000 7ff236fac700 1 Processor -- start 2022-09-07T21:30:32.657 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.759+0000 7ff236fac700 1 -- start start 2022-09-07T21:30:32.657 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.759+0000 7ff236fac700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff230078c80 0x7ff230199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:32.659 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.759+0000 7ff236fac700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2301997b0 con 0x7ff230078c80 2022-09-07T21:30:32.659 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.759+0000 7ff234d48700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff230078c80 0x7ff230199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:32.660 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.759+0000 7ff234d48700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff230078c80 0x7ff230199270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53622/0 (socket says 172.21.15.47:53622) 2022-09-07T21:30:32.660 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.759+0000 7ff234d48700 1 -- 172.21.15.47:0/2273897555 learned_addr learned my addr 172.21.15.47:0/2273897555 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:32.661 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.760+0000 7ff234d48700 1 -- 172.21.15.47:0/2273897555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2240093e0 con 0x7ff230078c80 2022-09-07T21:30:32.661 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.760+0000 7ff234d48700 1 --2- 172.21.15.47:0/2273897555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff230078c80 0x7ff230199270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff224009e90 tx=0x7ff224004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:32.663 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.760+0000 7ff22dffb700 1 -- 172.21.15.47:0/2273897555 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff224011840 con 0x7ff230078c80 2022-09-07T21:30:32.663 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.760+0000 7ff22dffb700 1 -- 172.21.15.47:0/2273897555 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7ff22401a430 con 0x7ff230078c80 2022-09-07T21:30:32.664 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.760+0000 7ff22dffb700 1 -- 172.21.15.47:0/2273897555 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff224019550 con 0x7ff230078c80 2022-09-07T21:30:32.664 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.760+0000 7ff236fac700 1 -- 172.21.15.47:0/2273897555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2301999b0 con 0x7ff230078c80 2022-09-07T21:30:32.665 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.760+0000 7ff236fac700 1 -- 172.21.15.47:0/2273897555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff230199de0 con 0x7ff230078c80 2022-09-07T21:30:32.665 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.761+0000 7ff22dffb700 1 -- 172.21.15.47:0/2273897555 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7ff22402a020 con 0x7ff230078c80 2022-09-07T21:30:32.665 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.761+0000 7ff236fac700 1 -- 172.21.15.47:0/2273897555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff230109940 con 0x7ff230078c80 2022-09-07T21:30:32.666 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.762+0000 7ff22dffb700 1 --2- 172.21.15.47:0/2273897555 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7ff214035890 0x7ff214037d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:32.666 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.762+0000 7ff22dffb700 1 -- 172.21.15.47:0/2273897555 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff224048a70 con 0x7ff230078c80 2022-09-07T21:30:32.667 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.762+0000 7ff22ffff700 1 --2- 172.21.15.47:0/2273897555 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7ff214035890 0x7ff214037d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:32.667 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.763+0000 7ff22ffff700 1 --2- 172.21.15.47:0/2273897555 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7ff214035890 0x7ff214037d40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff21c009750 tx=0x7ff21c006cc0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:32.667 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.767+0000 7ff22dffb700 1 -- 172.21.15.47:0/2273897555 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff2240119a0 con 0x7ff230078c80 2022-09-07T21:30:32.668 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:31.997+0000 7ff236fac700 1 -- 172.21.15.47:0/2273897555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7ff23019d070 con 0x7ff230078c80 2022-09-07T21:30:32.668 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:32.001+0000 7ff22dffb700 1 -- 172.21.15.47:0/2273897555 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) v1 ==== 142+0+0 (secure 0 0 0) 0x7ff224018b40 con 0x7ff230078c80 2022-09-07T21:30:32.668 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:32.003+0000 7ff236fac700 1 -- 172.21.15.47:0/2273897555 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7ff214035890 msgr2=0x7ff214037d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:32.669 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:32.003+0000 7ff236fac700 1 --2- 172.21.15.47:0/2273897555 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7ff214035890 0x7ff214037d40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff21c009750 tx=0x7ff21c006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:32.670 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:32.003+0000 7ff236fac700 1 -- 172.21.15.47:0/2273897555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff230078c80 msgr2=0x7ff230199270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:32.670 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:32.003+0000 7ff236fac700 1 --2- 172.21.15.47:0/2273897555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff230078c80 0x7ff230199270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff224009e90 tx=0x7ff224004040 comp rx=0 tx=0).stop 2022-09-07T21:30:32.670 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:32.004+0000 7ff236fac700 1 -- 172.21.15.47:0/2273897555 shutdown_connections 2022-09-07T21:30:32.670 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:32.004+0000 7ff236fac700 1 --2- 172.21.15.47:0/2273897555 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7ff214035890 0x7ff214037d40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:32.671 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:32.004+0000 7ff236fac700 1 --2- 172.21.15.47:0/2273897555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff230078c80 0x7ff230199270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:32.671 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:32.004+0000 7ff236fac700 1 -- 172.21.15.47:0/2273897555 >> 172.21.15.47:0/2273897555 conn(0x7ff2300fed50 msgr2=0x7ff2300ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:32.671 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:32.004+0000 7ff236fac700 1 -- 172.21.15.47:0/2273897555 shutdown_connections 2022-09-07T21:30:32.671 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:32.004+0000 7ff236fac700 1 -- 172.21.15.47:0/2273897555 wait complete. 2022-09-07T21:30:33.329 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:33 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2273897555' entity='client.admin' 2022-09-07T21:30:34.573 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.295+0000 7f00f2667700 1 Processor -- start 2022-09-07T21:30:34.574 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.295+0000 7f00f2667700 1 -- start start 2022-09-07T21:30:34.574 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.295+0000 7f00f2667700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00ec105850 0x7f00ec105c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:34.574 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.295+0000 7f00f2667700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec1061a0 con 0x7f00ec105850 2022-09-07T21:30:34.575 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.296+0000 7f00ebfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00ec105850 0x7f00ec105c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:34.575 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.296+0000 7f00ebfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00ec105850 0x7f00ec105c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53632/0 (socket says 172.21.15.47:53632) 2022-09-07T21:30:34.575 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.296+0000 7f00ebfff700 1 -- 172.21.15.47:0/4071785616 learned_addr learned my addr 172.21.15.47:0/4071785616 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:34.575 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.296+0000 7f00ebfff700 1 -- 172.21.15.47:0/4071785616 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec1062e0 con 0x7f00ec105850 2022-09-07T21:30:34.576 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.296+0000 7f00ebfff700 1 --2- 172.21.15.47:0/4071785616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00ec105850 0x7f00ec105c60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f00dc009700 tx=0x7f00dc009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3519e2eb989c9ff server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:34.576 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.297+0000 7f00eaffd700 1 -- 172.21.15.47:0/4071785616 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00dc004160 con 0x7f00ec105850 2022-09-07T21:30:34.576 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.297+0000 7f00eaffd700 1 -- 172.21.15.47:0/4071785616 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f00dc0042c0 con 0x7f00ec105850 2022-09-07T21:30:34.576 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.297+0000 7f00eaffd700 1 -- 172.21.15.47:0/4071785616 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00dc0039d0 con 0x7f00ec105850 2022-09-07T21:30:34.576 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.297+0000 7f00f2667700 1 -- 172.21.15.47:0/4071785616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00ec105850 msgr2=0x7f00ec105c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:34.577 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.297+0000 7f00f2667700 1 --2- 172.21.15.47:0/4071785616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00ec105850 0x7f00ec105c60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f00dc009700 tx=0x7f00dc009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:34.577 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.298+0000 7f00f2667700 1 -- 172.21.15.47:0/4071785616 shutdown_connections 2022-09-07T21:30:34.577 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.298+0000 7f00f2667700 1 --2- 172.21.15.47:0/4071785616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00ec105850 0x7f00ec105c60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:34.577 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.298+0000 7f00f2667700 1 -- 172.21.15.47:0/4071785616 >> 172.21.15.47:0/4071785616 conn(0x7f00ec078d90 msgr2=0x7f00ec079190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:34.578 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.298+0000 7f00f2667700 1 -- 172.21.15.47:0/4071785616 shutdown_connections 2022-09-07T21:30:34.578 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.298+0000 7f00f2667700 1 -- 172.21.15.47:0/4071785616 wait complete. 2022-09-07T21:30:34.578 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.299+0000 7f00f2667700 1 Processor -- start 2022-09-07T21:30:34.578 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.299+0000 7f00f2667700 1 -- start start 2022-09-07T21:30:34.579 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.299+0000 7f00f2667700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00ec105850 0x7f00ec199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:34.579 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.299+0000 7f00f2667700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec1997a0 con 0x7f00ec105850 2022-09-07T21:30:34.579 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.300+0000 7f00ebfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00ec105850 0x7f00ec199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:34.579 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.300+0000 7f00ebfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00ec105850 0x7f00ec199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53636/0 (socket says 172.21.15.47:53636) 2022-09-07T21:30:34.580 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.300+0000 7f00ebfff700 1 -- 172.21.15.47:0/2094061337 learned_addr learned my addr 172.21.15.47:0/2094061337 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:34.580 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.300+0000 7f00ebfff700 1 -- 172.21.15.47:0/2094061337 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00dc0093e0 con 0x7f00ec105850 2022-09-07T21:30:34.580 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.301+0000 7f00ebfff700 1 --2- 172.21.15.47:0/2094061337 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00ec105850 0x7f00ec199260 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f00dc003f00 tx=0x7f00dc003fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:34.580 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.301+0000 7f00e97fa700 1 -- 172.21.15.47:0/2094061337 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00dc011950 con 0x7f00ec105850 2022-09-07T21:30:34.580 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.301+0000 7f00e97fa700 1 -- 172.21.15.47:0/2094061337 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f00dc011ab0 con 0x7f00ec105850 2022-09-07T21:30:34.581 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.301+0000 7f00e97fa700 1 -- 172.21.15.47:0/2094061337 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00dc019550 con 0x7f00ec105850 2022-09-07T21:30:34.581 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.301+0000 7f00f2667700 1 -- 172.21.15.47:0/2094061337 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ec1999a0 con 0x7f00ec105850 2022-09-07T21:30:34.581 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.301+0000 7f00f2667700 1 -- 172.21.15.47:0/2094061337 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ec199dd0 con 0x7f00ec105850 2022-09-07T21:30:34.581 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.304+0000 7f00e97fa700 1 -- 172.21.15.47:0/2094061337 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f00dc02a020 con 0x7f00ec105850 2022-09-07T21:30:34.582 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.304+0000 7f00e97fa700 1 --2- 172.21.15.47:0/2094061337 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f00d8035930 0x7f00d8037de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:34.582 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.304+0000 7f00e97fa700 1 -- 172.21.15.47:0/2094061337 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f00dc048b40 con 0x7f00ec105850 2022-09-07T21:30:34.582 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.305+0000 7f00f2667700 1 -- 172.21.15.47:0/2094061337 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ec060310 con 0x7f00ec105850 2022-09-07T21:30:34.582 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.305+0000 7f00eb7fe700 1 --2- 172.21.15.47:0/2094061337 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f00d8035930 0x7f00d8037de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:34.583 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.306+0000 7f00eb7fe700 1 --2- 172.21.15.47:0/2094061337 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f00d8035930 0x7f00d8037de0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009750 tx=0x7f00e0006cc0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:34.583 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.313+0000 7f00e97fa700 1 -- 172.21.15.47:0/2094061337 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f00dc0197c0 con 0x7f00ec105850 2022-09-07T21:30:34.583 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.553+0000 7f00f2667700 1 -- 172.21.15.47:0/2094061337 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f00ec107690 con 0x7f00ec105850 2022-09-07T21:30:34.583 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.556+0000 7f00e97fa700 1 -- 172.21.15.47:0/2094061337 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v7)=0 v7) v1 ==== 130+0+0 (secure 0 0 0) 0x7f00dc018b40 con 0x7f00ec105850 2022-09-07T21:30:34.584 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.559+0000 7f00f2667700 1 -- 172.21.15.47:0/2094061337 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f00d8035930 msgr2=0x7f00d8037de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:34.584 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.559+0000 7f00f2667700 1 --2- 172.21.15.47:0/2094061337 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f00d8035930 0x7f00d8037de0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009750 tx=0x7f00e0006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:34.584 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.559+0000 7f00f2667700 1 -- 172.21.15.47:0/2094061337 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00ec105850 msgr2=0x7f00ec199260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:34.584 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.559+0000 7f00f2667700 1 --2- 172.21.15.47:0/2094061337 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00ec105850 0x7f00ec199260 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f00dc003f00 tx=0x7f00dc003fe0 comp rx=0 tx=0).stop 2022-09-07T21:30:34.585 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.559+0000 7f00f2667700 1 -- 172.21.15.47:0/2094061337 shutdown_connections 2022-09-07T21:30:34.585 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.559+0000 7f00f2667700 1 --2- 172.21.15.47:0/2094061337 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f00d8035930 0x7f00d8037de0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:34.585 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.559+0000 7f00f2667700 1 --2- 172.21.15.47:0/2094061337 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00ec105850 0x7f00ec199260 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:34.585 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.559+0000 7f00f2667700 1 -- 172.21.15.47:0/2094061337 >> 172.21.15.47:0/2094061337 conn(0x7f00ec078d90 msgr2=0x7f00ec103620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:34.586 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.562+0000 7f00f2667700 1 -- 172.21.15.47:0/2094061337 shutdown_connections 2022-09-07T21:30:34.586 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:33.562+0000 7f00f2667700 1 -- 172.21.15.47:0/2094061337 wait complete. 2022-09-07T21:30:34.586 INFO:teuthology.orchestra.run.smithi047.stdout:Enabling the dashboard module... 2022-09-07T21:30:34.881 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:34 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2094061337' entity='client.admin' 2022-09-07T21:30:34.881 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:34 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' 2022-09-07T21:30:36.532 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:36 smithi047 ceph-mon[90087]: from='mgr.14120 172.21.15.47:0/497359978' entity='mgr.x' 2022-09-07T21:30:36.532 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:36 smithi047 ceph-mon[90087]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:30:36.532 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:36 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/95532606' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2022-09-07T21:30:36.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: ignoring --setuser ceph since I am not root 2022-09-07T21:30:36.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: ignoring --setgroup ceph since I am not root 2022-09-07T21:30:37.016 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.230+0000 7f370f3ad700 1 Processor -- start 2022-09-07T21:30:37.016 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.231+0000 7f370f3ad700 1 -- start start 2022-09-07T21:30:37.017 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.231+0000 7f370f3ad700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3708103660 0x7f3708103a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:37.017 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.231+0000 7f370f3ad700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3708103fb0 con 0x7f3708103660 2022-09-07T21:30:37.017 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.232+0000 7f370d149700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3708103660 0x7f3708103a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:37.017 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.232+0000 7f370d149700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3708103660 0x7f3708103a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53640/0 (socket says 172.21.15.47:53640) 2022-09-07T21:30:37.018 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.232+0000 7f370d149700 1 -- 172.21.15.47:0/1509643617 learned_addr learned my addr 172.21.15.47:0/1509643617 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:37.018 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.232+0000 7f370d149700 1 -- 172.21.15.47:0/1509643617 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37081040f0 con 0x7f3708103660 2022-09-07T21:30:37.019 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.232+0000 7f370d149700 1 --2- 172.21.15.47:0/1509643617 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3708103660 0x7f3708103a70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3704009700 tx=0x7f3704009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4adbf35e3155241 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:37.019 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.232+0000 7f36f7fff700 1 -- 172.21.15.47:0/1509643617 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3704004160 con 0x7f3708103660 2022-09-07T21:30:37.019 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.233+0000 7f36f7fff700 1 -- 172.21.15.47:0/1509643617 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f37040042c0 con 0x7f3708103660 2022-09-07T21:30:37.020 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.233+0000 7f36f7fff700 1 -- 172.21.15.47:0/1509643617 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3704003a10 con 0x7f3708103660 2022-09-07T21:30:37.020 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.233+0000 7f370f3ad700 1 -- 172.21.15.47:0/1509643617 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3708103660 msgr2=0x7f3708103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:37.020 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.233+0000 7f370f3ad700 1 --2- 172.21.15.47:0/1509643617 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3708103660 0x7f3708103a70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3704009700 tx=0x7f3704009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:37.020 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.233+0000 7f370f3ad700 1 -- 172.21.15.47:0/1509643617 shutdown_connections 2022-09-07T21:30:37.020 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.233+0000 7f370f3ad700 1 --2- 172.21.15.47:0/1509643617 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3708103660 0x7f3708103a70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:37.021 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.233+0000 7f370f3ad700 1 -- 172.21.15.47:0/1509643617 >> 172.21.15.47:0/1509643617 conn(0x7f37080fed50 msgr2=0x7f3708101180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:37.021 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.233+0000 7f370f3ad700 1 -- 172.21.15.47:0/1509643617 shutdown_connections 2022-09-07T21:30:37.021 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.234+0000 7f370f3ad700 1 -- 172.21.15.47:0/1509643617 wait complete. 2022-09-07T21:30:37.021 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.235+0000 7f370f3ad700 1 Processor -- start 2022-09-07T21:30:37.022 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.235+0000 7f370f3ad700 1 -- start start 2022-09-07T21:30:37.022 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.235+0000 7f370f3ad700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3708103660 0x7f3708078b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:37.022 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.235+0000 7f370f3ad700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37080790a0 con 0x7f3708103660 2022-09-07T21:30:37.022 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.236+0000 7f370d149700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3708103660 0x7f3708078b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:37.023 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.236+0000 7f370d149700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3708103660 0x7f3708078b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53656/0 (socket says 172.21.15.47:53656) 2022-09-07T21:30:37.023 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.236+0000 7f370d149700 1 -- 172.21.15.47:0/95532606 learned_addr learned my addr 172.21.15.47:0/95532606 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:37.023 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.236+0000 7f370d149700 1 -- 172.21.15.47:0/95532606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37040093e0 con 0x7f3708103660 2022-09-07T21:30:37.023 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.236+0000 7f370d149700 1 --2- 172.21.15.47:0/95532606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3708103660 0x7f3708078b60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3704003560 tx=0x7f3704003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:37.024 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.237+0000 7f36f67fc700 1 -- 172.21.15.47:0/95532606 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3704011840 con 0x7f3708103660 2022-09-07T21:30:37.024 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.237+0000 7f36f67fc700 1 -- 172.21.15.47:0/95532606 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f37040119a0 con 0x7f3708103660 2022-09-07T21:30:37.025 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.237+0000 7f370f3ad700 1 -- 172.21.15.47:0/95532606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37080771a0 con 0x7f3708103660 2022-09-07T21:30:37.025 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.237+0000 7f36f67fc700 1 -- 172.21.15.47:0/95532606 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f37040193c0 con 0x7f3708103660 2022-09-07T21:30:37.026 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.237+0000 7f370f3ad700 1 -- 172.21.15.47:0/95532606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37080775d0 con 0x7f3708103660 2022-09-07T21:30:37.026 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.238+0000 7f370f3ad700 1 -- 172.21.15.47:0/95532606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37081054c0 con 0x7f3708103660 2022-09-07T21:30:37.027 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.238+0000 7f36f67fc700 1 -- 172.21.15.47:0/95532606 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f370402a020 con 0x7f3708103660 2022-09-07T21:30:37.027 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.239+0000 7f36f67fc700 1 --2- 172.21.15.47:0/95532606 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f36f00358f0 0x7f36f0037da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:37.027 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.239+0000 7f36f67fc700 1 -- 172.21.15.47:0/95532606 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f37040488a0 con 0x7f3708103660 2022-09-07T21:30:37.028 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.239+0000 7f370c948700 1 --2- 172.21.15.47:0/95532606 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f36f00358f0 0x7f36f0037da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:37.028 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.240+0000 7f370c948700 1 --2- 172.21.15.47:0/95532606 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f36f00358f0 0x7f36f0037da0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009750 tx=0x7f36f8006cc0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:37.029 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.244+0000 7f36f67fc700 1 -- 172.21.15.47:0/95532606 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f370402b940 con 0x7f3708103660 2022-09-07T21:30:37.029 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:35.526+0000 7f370f3ad700 1 -- 172.21.15.47:0/95532606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f3708077c80 con 0x7f3708103660 2022-09-07T21:30:37.029 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:36.478+0000 7f36f67fc700 1 -- 172.21.15.47:0/95532606 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 9) v1 ==== 42703+0+0 (secure 0 0 0) 0x7f3704017020 con 0x7f3708103660 2022-09-07T21:30:37.030 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:36.481+0000 7f36f67fc700 1 -- 172.21.15.47:0/95532606 <== mon.0 v2:172.21.15.47:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f37040058a0 con 0x7f3708103660 2022-09-07T21:30:37.030 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:36.484+0000 7f370f3ad700 1 -- 172.21.15.47:0/95532606 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f36f00358f0 msgr2=0x7f36f0037da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:37.031 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:36.484+0000 7f370f3ad700 1 --2- 172.21.15.47:0/95532606 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f36f00358f0 0x7f36f0037da0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009750 tx=0x7f36f8006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:37.031 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:36.484+0000 7f370f3ad700 1 -- 172.21.15.47:0/95532606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3708103660 msgr2=0x7f3708078b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:37.032 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:36.484+0000 7f370f3ad700 1 --2- 172.21.15.47:0/95532606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3708103660 0x7f3708078b60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3704003560 tx=0x7f3704003f60 comp rx=0 tx=0).stop 2022-09-07T21:30:37.032 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:36.484+0000 7f370f3ad700 1 -- 172.21.15.47:0/95532606 shutdown_connections 2022-09-07T21:30:37.033 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:36.484+0000 7f370f3ad700 1 --2- 172.21.15.47:0/95532606 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f36f00358f0 0x7f36f0037da0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:37.033 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:36.484+0000 7f370f3ad700 1 --2- 172.21.15.47:0/95532606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3708103660 0x7f3708078b60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:37.034 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:36.484+0000 7f370f3ad700 1 -- 172.21.15.47:0/95532606 >> 172.21.15.47:0/95532606 conn(0x7f37080fed50 msgr2=0x7f37080ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:37.034 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:36.484+0000 7f370f3ad700 1 -- 172.21.15.47:0/95532606 shutdown_connections 2022-09-07T21:30:37.035 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:36.484+0000 7f370f3ad700 1 -- 172.21.15.47:0/95532606 wait complete. 2022-09-07T21:30:37.184 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:36.902+0000 7fbda789c080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-07T21:30:37.185 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:37.029+0000 7fbda789c080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-07T21:30:37.481 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:37.181+0000 7fbda789c080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-07T21:30:37.482 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:37.319+0000 7fbda789c080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-07T21:30:37.732 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:37 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/95532606' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2022-09-07T21:30:37.733 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:37 smithi047 ceph-mon[90087]: mgrmap e9: x(active, since 21s) 2022-09-07T21:30:37.733 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:37.617+0000 7fbda789c080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-07T21:30:38.403 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:38.106+0000 7fbda789c080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-07T21:30:38.404 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:38.242+0000 7fbda789c080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-07T21:30:38.790 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout { 2022-09-07T21:30:38.791 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "epoch": 9, 2022-09-07T21:30:38.791 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "available": true, 2022-09-07T21:30:38.792 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "active_name": "x", 2022-09-07T21:30:38.793 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "num_standby": 0 2022-09-07T21:30:38.793 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout } 2022-09-07T21:30:38.794 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.972+0000 7f14b7923700 1 Processor -- start 2022-09-07T21:30:38.794 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.973+0000 7f14b7923700 1 -- start start 2022-09-07T21:30:38.795 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.973+0000 7f14b7923700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14b0103660 0x7f14b0103a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:38.796 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.973+0000 7f14b7923700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b0103fb0 con 0x7f14b0103660 2022-09-07T21:30:38.796 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.974+0000 7f14b56bf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14b0103660 0x7f14b0103a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:38.797 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.974+0000 7f14b56bf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14b0103660 0x7f14b0103a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53696/0 (socket says 172.21.15.47:53696) 2022-09-07T21:30:38.798 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.974+0000 7f14b56bf700 1 -- 172.21.15.47:0/4283220575 learned_addr learned my addr 172.21.15.47:0/4283220575 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:38.798 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.974+0000 7f14b56bf700 1 -- 172.21.15.47:0/4283220575 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b01040f0 con 0x7f14b0103660 2022-09-07T21:30:38.799 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.974+0000 7f14b56bf700 1 --2- 172.21.15.47:0/4283220575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14b0103660 0x7f14b0103a70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009700 tx=0x7f14ac009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27f589fc32d4921 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:38.799 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.975+0000 7f149ffff700 1 -- 172.21.15.47:0/4283220575 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14ac004160 con 0x7f14b0103660 2022-09-07T21:30:38.800 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.975+0000 7f149ffff700 1 -- 172.21.15.47:0/4283220575 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f14ac0042c0 con 0x7f14b0103660 2022-09-07T21:30:38.800 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.975+0000 7f149ffff700 1 -- 172.21.15.47:0/4283220575 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14ac003a10 con 0x7f14b0103660 2022-09-07T21:30:38.800 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.975+0000 7f14b7923700 1 -- 172.21.15.47:0/4283220575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14b0103660 msgr2=0x7f14b0103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:38.801 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.975+0000 7f14b7923700 1 --2- 172.21.15.47:0/4283220575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14b0103660 0x7f14b0103a70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009700 tx=0x7f14ac009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:38.801 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.976+0000 7f14b7923700 1 -- 172.21.15.47:0/4283220575 shutdown_connections 2022-09-07T21:30:38.802 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.976+0000 7f14b7923700 1 --2- 172.21.15.47:0/4283220575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14b0103660 0x7f14b0103a70 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:38.802 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.976+0000 7f14b7923700 1 -- 172.21.15.47:0/4283220575 >> 172.21.15.47:0/4283220575 conn(0x7f14b00fed50 msgr2=0x7f14b0101180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:38.803 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.976+0000 7f14b7923700 1 -- 172.21.15.47:0/4283220575 shutdown_connections 2022-09-07T21:30:38.803 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.976+0000 7f14b7923700 1 -- 172.21.15.47:0/4283220575 wait complete. 2022-09-07T21:30:38.804 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.978+0000 7f14b7923700 1 Processor -- start 2022-09-07T21:30:38.804 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.978+0000 7f14b7923700 1 -- start start 2022-09-07T21:30:38.804 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.978+0000 7f14b7923700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14b0103660 0x7f14b01904d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:38.805 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.978+0000 7f14b7923700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b0190a10 con 0x7f14b0103660 2022-09-07T21:30:38.805 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.979+0000 7f14b56bf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14b0103660 0x7f14b01904d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:38.806 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.979+0000 7f14b56bf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14b0103660 0x7f14b01904d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53706/0 (socket says 172.21.15.47:53706) 2022-09-07T21:30:38.806 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.979+0000 7f14b56bf700 1 -- 172.21.15.47:0/3254694710 learned_addr learned my addr 172.21.15.47:0/3254694710 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:38.806 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.979+0000 7f14b56bf700 1 -- 172.21.15.47:0/3254694710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac0093e0 con 0x7f14b0103660 2022-09-07T21:30:38.806 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.979+0000 7f14b56bf700 1 --2- 172.21.15.47:0/3254694710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14b0103660 0x7f14b01904d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f14ac003f00 tx=0x7f14ac003fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:38.807 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.980+0000 7f149e7fc700 1 -- 172.21.15.47:0/3254694710 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14ac011840 con 0x7f14b0103660 2022-09-07T21:30:38.807 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.980+0000 7f14b7923700 1 -- 172.21.15.47:0/3254694710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b0190c10 con 0x7f14b0103660 2022-09-07T21:30:38.807 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.980+0000 7f149e7fc700 1 -- 172.21.15.47:0/3254694710 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f14ac0119a0 con 0x7f14b0103660 2022-09-07T21:30:38.807 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.980+0000 7f149e7fc700 1 -- 172.21.15.47:0/3254694710 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14ac0193c0 con 0x7f14b0103660 2022-09-07T21:30:38.808 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.980+0000 7f14b7923700 1 -- 172.21.15.47:0/3254694710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b018d120 con 0x7f14b0103660 2022-09-07T21:30:38.808 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.981+0000 7f14b7923700 1 -- 172.21.15.47:0/3254694710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b0060310 con 0x7f14b0103660 2022-09-07T21:30:38.808 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.981+0000 7f149e7fc700 1 -- 172.21.15.47:0/3254694710 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 9) v1 ==== 42703+0+0 (secure 0 0 0) 0x7f14ac02a020 con 0x7f14b0103660 2022-09-07T21:30:38.809 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.982+0000 7f149e7fc700 1 --2- 172.21.15.47:0/3254694710 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f14980358f0 0x7f1498037da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:38.809 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.982+0000 7f149e7fc700 1 -- 172.21.15.47:0/3254694710 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f14ac048a00 con 0x7f14b0103660 2022-09-07T21:30:38.809 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.982+0000 7f14b4ebe700 1 -- 172.21.15.47:0/3254694710 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f14980358f0 msgr2=0x7f1498037da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/4097349239 2022-09-07T21:30:38.810 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.982+0000 7f14b4ebe700 1 --2- 172.21.15.47:0/3254694710 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f14980358f0 0x7f1498037da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-07T21:30:38.810 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:37.987+0000 7f149e7fc700 1 -- 172.21.15.47:0/3254694710 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f14ac0195f0 con 0x7f14b0103660 2022-09-07T21:30:38.811 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:38.183+0000 7f14b4ebe700 1 -- 172.21.15.47:0/3254694710 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f14980358f0 msgr2=0x7f1498037da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/4097349239 2022-09-07T21:30:38.811 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:38.183+0000 7f14b4ebe700 1 --2- 172.21.15.47:0/3254694710 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f14980358f0 0x7f1498037da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-09-07T21:30:38.812 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:38.272+0000 7f14b7923700 1 -- 172.21.15.47:0/3254694710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f14b018db80 con 0x7f14b0103660 2022-09-07T21:30:38.812 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:38.273+0000 7f149e7fc700 1 -- 172.21.15.47:0/3254694710 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f14ac018b40 con 0x7f14b0103660 2022-09-07T21:30:38.812 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:38.276+0000 7f14b7923700 1 -- 172.21.15.47:0/3254694710 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f14980358f0 msgr2=0x7f1498037da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-07T21:30:38.813 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:38.276+0000 7f14b7923700 1 --2- 172.21.15.47:0/3254694710 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f14980358f0 0x7f1498037da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:38.814 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:38.276+0000 7f14b7923700 1 -- 172.21.15.47:0/3254694710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14b0103660 msgr2=0x7f14b01904d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:38.814 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:38.276+0000 7f14b7923700 1 --2- 172.21.15.47:0/3254694710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14b0103660 0x7f14b01904d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f14ac003f00 tx=0x7f14ac003fe0 comp rx=0 tx=0).stop 2022-09-07T21:30:38.814 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:38.276+0000 7f14b7923700 1 -- 172.21.15.47:0/3254694710 shutdown_connections 2022-09-07T21:30:38.815 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:38.276+0000 7f14b7923700 1 --2- 172.21.15.47:0/3254694710 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f14980358f0 0x7f1498037da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:38.815 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:38.276+0000 7f14b7923700 1 --2- 172.21.15.47:0/3254694710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14b0103660 0x7f14b01904d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:38.815 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:38.276+0000 7f14b7923700 1 -- 172.21.15.47:0/3254694710 >> 172.21.15.47:0/3254694710 conn(0x7f14b00fed50 msgr2=0x7f14b00ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:38.815 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:38.276+0000 7f14b7923700 1 -- 172.21.15.47:0/3254694710 shutdown_connections 2022-09-07T21:30:38.815 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:38.276+0000 7f14b7923700 1 -- 172.21.15.47:0/3254694710 wait complete. 2022-09-07T21:30:38.816 INFO:teuthology.orchestra.run.smithi047.stdout:Waiting for the mgr to restart... 2022-09-07T21:30:38.816 INFO:teuthology.orchestra.run.smithi047.stdout:Waiting for mgr epoch 9... 2022-09-07T21:30:38.817 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:38.437+0000 7fbda789c080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-07T21:30:38.818 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:38.586+0000 7fbda789c080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-07T21:30:39.060 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:38 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3254694710' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2022-09-07T21:30:39.403 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:39.244+0000 7fbda789c080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-07T21:30:40.403 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:40.112+0000 7fbda789c080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-07T21:30:40.403 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:40.352+0000 7fbda789c080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-07T21:30:41.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:41.459+0000 7fbda789c080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:30:41.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:41.613+0000 7fbda789c080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-07T21:30:42.223 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:41.901+0000 7fbda789c080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:30:42.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:42.221+0000 7fbda789c080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-07T21:30:42.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:42.461+0000 7fbda789c080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-07T21:30:43.403 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:43.079+0000 7fbda789c080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-07T21:30:44.403 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:44.037+0000 7fbda789c080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-07T21:30:44.814 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:44.472+0000 7fbda789c080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-07T21:30:45.153 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:44.811+0000 7fbda789c080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-07T21:30:45.153 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:45.007+0000 7fbda789c080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-07T21:30:45.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:45.383+0000 7fbda789c080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-07T21:30:46.880 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:30:46.437+0000 7fbda789c080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-07T21:30:46.881 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:46 smithi047 ceph-mon[90087]: Active manager daemon x restarted 2022-09-07T21:30:46.881 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:46 smithi047 ceph-mon[90087]: Activating manager daemon x 2022-09-07T21:30:46.881 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:46 smithi047 ceph-mon[90087]: osdmap e3: 0 total, 0 up, 0 in 2022-09-07T21:30:47.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:47 smithi047 ceph-mon[90087]: mgrmap e10: x(active, starting, since 0.0533641s) 2022-09-07T21:30:47.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:30:47.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-09-07T21:30:47.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-09-07T21:30:47.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-09-07T21:30:47.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-09-07T21:30:47.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:47 smithi047 ceph-mon[90087]: Manager daemon x is now available 2022-09-07T21:30:47.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:30:47.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:30:47.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:30:47.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:30:48.015 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout { 2022-09-07T21:30:48.016 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2022-09-07T21:30:48.016 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout "initialized": true 2022-09-07T21:30:48.016 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout } 2022-09-07T21:30:48.017 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.418+0000 7f050914e700 1 Processor -- start 2022-09-07T21:30:48.017 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.418+0000 7f050914e700 1 -- start start 2022-09-07T21:30:48.017 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.419+0000 7f050914e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0504078cc0 0x7f0504077240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:48.017 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.419+0000 7f050914e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0504077780 con 0x7f0504078cc0 2022-09-07T21:30:48.018 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.419+0000 7f0502d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0504078cc0 0x7f0504077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:48.018 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.419+0000 7f0502d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0504078cc0 0x7f0504077240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36994/0 (socket says 172.21.15.47:36994) 2022-09-07T21:30:48.018 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.419+0000 7f0502d9d700 1 -- 172.21.15.47:0/3442677586 learned_addr learned my addr 172.21.15.47:0/3442677586 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:48.018 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.420+0000 7f0502d9d700 1 -- 172.21.15.47:0/3442677586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05040778c0 con 0x7f0504078cc0 2022-09-07T21:30:48.018 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.420+0000 7f0502d9d700 1 --2- 172.21.15.47:0/3442677586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0504078cc0 0x7f0504077240 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009700 tx=0x7f04f4009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbabe8c7f7d313a1 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:48.019 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.420+0000 7f0501d9b700 1 -- 172.21.15.47:0/3442677586 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04f4004160 con 0x7f0504078cc0 2022-09-07T21:30:48.019 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.420+0000 7f0501d9b700 1 -- 172.21.15.47:0/3442677586 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f04f40042c0 con 0x7f0504078cc0 2022-09-07T21:30:48.019 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.420+0000 7f0501d9b700 1 -- 172.21.15.47:0/3442677586 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04f4003a10 con 0x7f0504078cc0 2022-09-07T21:30:48.019 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.421+0000 7f050914e700 1 -- 172.21.15.47:0/3442677586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0504078cc0 msgr2=0x7f0504077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:48.020 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.421+0000 7f050914e700 1 --2- 172.21.15.47:0/3442677586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0504078cc0 0x7f0504077240 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009700 tx=0x7f04f4009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:48.020 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.421+0000 7f050914e700 1 -- 172.21.15.47:0/3442677586 shutdown_connections 2022-09-07T21:30:48.020 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.421+0000 7f050914e700 1 --2- 172.21.15.47:0/3442677586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0504078cc0 0x7f0504077240 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:48.020 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.421+0000 7f050914e700 1 -- 172.21.15.47:0/3442677586 >> 172.21.15.47:0/3442677586 conn(0x7f05040fed50 msgr2=0x7f0504101180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:48.020 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.421+0000 7f050914e700 1 -- 172.21.15.47:0/3442677586 shutdown_connections 2022-09-07T21:30:48.021 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.421+0000 7f050914e700 1 -- 172.21.15.47:0/3442677586 wait complete. 2022-09-07T21:30:48.021 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.423+0000 7f050914e700 1 Processor -- start 2022-09-07T21:30:48.021 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.423+0000 7f050914e700 1 -- start start 2022-09-07T21:30:48.021 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.424+0000 7f050914e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0504078cc0 0x7f050419d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:48.022 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.424+0000 7f050914e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050419dc10 con 0x7f0504078cc0 2022-09-07T21:30:48.022 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.424+0000 7f0502d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0504078cc0 0x7f050419d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:48.022 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.424+0000 7f0502d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0504078cc0 0x7f050419d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:37004/0 (socket says 172.21.15.47:37004) 2022-09-07T21:30:48.023 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.424+0000 7f0502d9d700 1 -- 172.21.15.47:0/2778654770 learned_addr learned my addr 172.21.15.47:0/2778654770 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:48.023 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.425+0000 7f0502d9d700 1 -- 172.21.15.47:0/2778654770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f40093e0 con 0x7f0504078cc0 2022-09-07T21:30:48.023 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.425+0000 7f0502d9d700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0504078cc0 0x7f050419d6d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f04f4003560 tx=0x7f04f4003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:48.023 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.425+0000 7f04ebfff700 1 -- 172.21.15.47:0/2778654770 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04f4011840 con 0x7f0504078cc0 2022-09-07T21:30:48.024 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.425+0000 7f04ebfff700 1 -- 172.21.15.47:0/2778654770 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f04f40119a0 con 0x7f0504078cc0 2022-09-07T21:30:48.024 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.425+0000 7f050914e700 1 -- 172.21.15.47:0/2778654770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050419de10 con 0x7f0504078cc0 2022-09-07T21:30:48.024 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.425+0000 7f04ebfff700 1 -- 172.21.15.47:0/2778654770 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04f40193c0 con 0x7f0504078cc0 2022-09-07T21:30:48.024 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.426+0000 7f050914e700 1 -- 172.21.15.47:0/2778654770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f050419e240 con 0x7f0504078cc0 2022-09-07T21:30:48.025 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.427+0000 7f04ebfff700 1 -- 172.21.15.47:0/2778654770 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 9) v1 ==== 42703+0+0 (secure 0 0 0) 0x7f04f402a020 con 0x7f0504078cc0 2022-09-07T21:30:48.025 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.427+0000 7f04ebfff700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 0x7f04f0037de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:48.025 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.427+0000 7f04ebfff700 1 -- 172.21.15.47:0/2778654770 --> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04f00384f0 con 0x7f04f0035930 2022-09-07T21:30:48.026 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.427+0000 7f04ebfff700 1 -- 172.21.15.47:0/2778654770 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f04f40488e0 con 0x7f0504078cc0 2022-09-07T21:30:48.026 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.427+0000 7f050259c700 1 -- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 msgr2=0x7f04f0037de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/4097349239 2022-09-07T21:30:48.026 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.427+0000 7f050259c700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 0x7f04f0037de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-07T21:30:48.026 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.628+0000 7f050259c700 1 -- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 msgr2=0x7f04f0037de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/4097349239 2022-09-07T21:30:48.027 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:39.628+0000 7f050259c700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 0x7f04f0037de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-09-07T21:30:48.027 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:40.029+0000 7f050259c700 1 -- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 msgr2=0x7f04f0037de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/4097349239 2022-09-07T21:30:48.027 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:40.029+0000 7f050259c700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 0x7f04f0037de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2022-09-07T21:30:48.027 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:40.830+0000 7f050259c700 1 -- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 msgr2=0x7f04f0037de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/4097349239 2022-09-07T21:30:48.028 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:40.830+0000 7f050259c700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 0x7f04f0037de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2022-09-07T21:30:48.028 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:42.432+0000 7f050259c700 1 -- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 msgr2=0x7f04f0037de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/4097349239 2022-09-07T21:30:48.029 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:42.432+0000 7f050259c700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 0x7f04f0037de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2022-09-07T21:30:48.029 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:45.635+0000 7f050259c700 1 -- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 msgr2=0x7f04f0037de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/4097349239 2022-09-07T21:30:48.029 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:45.635+0000 7f050259c700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 0x7f04f0037de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2022-09-07T21:30:48.029 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:46.492+0000 7f04ebfff700 1 -- 172.21.15.47:0/2778654770 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mgrmap(e 10) v1 ==== 42513+0+0 (secure 0 0 0) 0x7f04f4017020 con 0x7f0504078cc0 2022-09-07T21:30:48.030 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:46.492+0000 7f04ebfff700 1 -- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 msgr2=0x7f04f0037de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-07T21:30:48.031 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:46.492+0000 7f04ebfff700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 0x7f04f0037de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:48.031 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.500+0000 7f04ebfff700 1 -- 172.21.15.47:0/2778654770 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 11) v1 ==== 42623+0+0 (secure 0 0 0) 0x7f04f400f050 con 0x7f0504078cc0 2022-09-07T21:30:48.031 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.500+0000 7f04ebfff700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f04f00393d0 0x7f04f003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:48.032 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.501+0000 7f04ebfff700 1 -- 172.21.15.47:0/2778654770 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04f00384f0 con 0x7f04f00393d0 2022-09-07T21:30:48.032 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.501+0000 7f050259c700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f04f00393d0 0x7f04f003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:48.032 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.503+0000 7f050259c700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f04f00393d0 0x7f04f003b7b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f04f8003890 tx=0x7f04f8009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:48.032 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.504+0000 7f04ebfff700 1 -- 172.21.15.47:0/2778654770 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6424 (secure 0 0 0) 0x7f04f00384f0 con 0x7f04f00393d0 2022-09-07T21:30:48.032 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.514+0000 7f050914e700 1 -- 172.21.15.47:0/2778654770 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f050419ea00 con 0x7f04f00393d0 2022-09-07T21:30:48.033 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.514+0000 7f04ebfff700 1 -- 172.21.15.47:0/2778654770 <== mgr.14152 v2:172.21.15.47:6800/429234829 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f050419ea00 con 0x7f04f00393d0 2022-09-07T21:30:48.033 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.516+0000 7f050914e700 1 -- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f04f00393d0 msgr2=0x7f04f003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:48.033 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.516+0000 7f050914e700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f04f00393d0 0x7f04f003b7b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f04f8003890 tx=0x7f04f8009270 comp rx=0 tx=0).stop 2022-09-07T21:30:48.033 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.516+0000 7f050914e700 1 -- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0504078cc0 msgr2=0x7f050419d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:48.034 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.516+0000 7f050914e700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0504078cc0 0x7f050419d6d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f04f4003560 tx=0x7f04f4003f60 comp rx=0 tx=0).stop 2022-09-07T21:30:48.034 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.516+0000 7f050914e700 1 -- 172.21.15.47:0/2778654770 shutdown_connections 2022-09-07T21:30:48.034 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.516+0000 7f050914e700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f04f00393d0 0x7f04f003b7b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:48.034 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.516+0000 7f050914e700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0504078cc0 0x7f050419d6d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:48.035 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.516+0000 7f050914e700 1 --2- 172.21.15.47:0/2778654770 >> [v2:172.21.15.47:6800/4097349239,v1:172.21.15.47:6801/4097349239] conn(0x7f04f0035930 0x7f04f0037de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:48.035 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.516+0000 7f050914e700 1 -- 172.21.15.47:0/2778654770 >> 172.21.15.47:0/2778654770 conn(0x7f05040fed50 msgr2=0x7f0504104930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:48.036 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.516+0000 7f050914e700 1 -- 172.21.15.47:0/2778654770 shutdown_connections 2022-09-07T21:30:48.036 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:47.516+0000 7f050914e700 1 -- 172.21.15.47:0/2778654770 wait complete. 2022-09-07T21:30:48.036 INFO:teuthology.orchestra.run.smithi047.stdout:mgr epoch 9 is available 2022-09-07T21:30:48.036 INFO:teuthology.orchestra.run.smithi047.stdout:Generating a dashboard self-signed certificate... 2022-09-07T21:30:48.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:48 smithi047 ceph-mon[90087]: [07/Sep/2022:21:30:47] ENGINE Bus STARTING 2022-09-07T21:30:48.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:48 smithi047 ceph-mon[90087]: [07/Sep/2022:21:30:47] ENGINE Serving on https://172.21.15.47:7150 2022-09-07T21:30:48.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:48 smithi047 ceph-mon[90087]: mgrmap e11: x(active, since 1.06388s) 2022-09-07T21:30:48.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:48 smithi047 ceph-mon[90087]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2022-09-07T21:30:48.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:48 smithi047 ceph-mon[90087]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2022-09-07T21:30:48.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:48 smithi047 ceph-mon[90087]: [07/Sep/2022:21:30:47] ENGINE Serving on https://:::8765 2022-09-07T21:30:48.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:48 smithi047 ceph-mon[90087]: [07/Sep/2022:21:30:47] ENGINE Bus STARTED 2022-09-07T21:30:49.834 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2022-09-07T21:30:49.835 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.752+0000 7f08b72f3700 1 Processor -- start 2022-09-07T21:30:49.835 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.752+0000 7f08b72f3700 1 -- start start 2022-09-07T21:30:49.835 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.752+0000 7f08b72f3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08b0071ea0 0x7f08b0108490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:49.835 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.752+0000 7f08b72f3700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b0103e80 con 0x7f08b0071ea0 2022-09-07T21:30:49.836 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.753+0000 7f08b62f1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08b0071ea0 0x7f08b0108490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:49.836 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.753+0000 7f08b62f1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08b0071ea0 0x7f08b0108490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33654/0 (socket says 172.21.15.47:33654) 2022-09-07T21:30:49.836 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.753+0000 7f08b62f1700 1 -- 172.21.15.47:0/3451717800 learned_addr learned my addr 172.21.15.47:0/3451717800 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:49.837 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.753+0000 7f08b62f1700 1 -- 172.21.15.47:0/3451717800 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b01089d0 con 0x7f08b0071ea0 2022-09-07T21:30:49.837 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.754+0000 7f08b62f1700 1 --2- 172.21.15.47:0/3451717800 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08b0071ea0 0x7f08b0108490 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009700 tx=0x7f08ac009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7619707327e75d46 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:49.838 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.754+0000 7f08b52ef700 1 -- 172.21.15.47:0/3451717800 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08ac004160 con 0x7f08b0071ea0 2022-09-07T21:30:49.839 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.754+0000 7f08b52ef700 1 -- 172.21.15.47:0/3451717800 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f08ac0042c0 con 0x7f08b0071ea0 2022-09-07T21:30:49.839 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.754+0000 7f08b52ef700 1 -- 172.21.15.47:0/3451717800 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08ac003a10 con 0x7f08b0071ea0 2022-09-07T21:30:49.840 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.755+0000 7f08b72f3700 1 -- 172.21.15.47:0/3451717800 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08b0071ea0 msgr2=0x7f08b0108490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:49.840 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.755+0000 7f08b72f3700 1 --2- 172.21.15.47:0/3451717800 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08b0071ea0 0x7f08b0108490 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009700 tx=0x7f08ac009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:49.840 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.755+0000 7f08b72f3700 1 -- 172.21.15.47:0/3451717800 shutdown_connections 2022-09-07T21:30:49.841 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.755+0000 7f08b72f3700 1 --2- 172.21.15.47:0/3451717800 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08b0071ea0 0x7f08b0108490 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:49.841 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.755+0000 7f08b72f3700 1 -- 172.21.15.47:0/3451717800 >> 172.21.15.47:0/3451717800 conn(0x7f08b00fda80 msgr2=0x7f08b00ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:49.842 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.755+0000 7f08b72f3700 1 -- 172.21.15.47:0/3451717800 shutdown_connections 2022-09-07T21:30:49.842 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.755+0000 7f08b72f3700 1 -- 172.21.15.47:0/3451717800 wait complete. 2022-09-07T21:30:49.843 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.756+0000 7f08b72f3700 1 Processor -- start 2022-09-07T21:30:49.843 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.757+0000 7f08b72f3700 1 -- start start 2022-09-07T21:30:49.844 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.757+0000 7f08b72f3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08b0071ea0 0x7f08b0199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:49.845 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.757+0000 7f08b72f3700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b0199640 con 0x7f08b0071ea0 2022-09-07T21:30:49.845 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.757+0000 7f08b62f1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08b0071ea0 0x7f08b0199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:49.846 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.757+0000 7f08b62f1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08b0071ea0 0x7f08b0199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33656/0 (socket says 172.21.15.47:33656) 2022-09-07T21:30:49.846 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.757+0000 7f08b62f1700 1 -- 172.21.15.47:0/2327169448 learned_addr learned my addr 172.21.15.47:0/2327169448 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:49.847 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.758+0000 7f08b62f1700 1 -- 172.21.15.47:0/2327169448 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac0093e0 con 0x7f08b0071ea0 2022-09-07T21:30:49.847 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.758+0000 7f08b62f1700 1 --2- 172.21.15.47:0/2327169448 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08b0071ea0 0x7f08b0199100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f08ac003560 tx=0x7f08ac003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:49.847 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.758+0000 7f089f7fe700 1 -- 172.21.15.47:0/2327169448 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08ac011840 con 0x7f08b0071ea0 2022-09-07T21:30:49.848 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.758+0000 7f089f7fe700 1 -- 172.21.15.47:0/2327169448 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f08ac01a430 con 0x7f08b0071ea0 2022-09-07T21:30:49.848 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.758+0000 7f089f7fe700 1 -- 172.21.15.47:0/2327169448 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08ac0193c0 con 0x7f08b0071ea0 2022-09-07T21:30:49.849 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.758+0000 7f08b72f3700 1 -- 172.21.15.47:0/2327169448 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b0199840 con 0x7f08b0071ea0 2022-09-07T21:30:49.849 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.758+0000 7f08b72f3700 1 -- 172.21.15.47:0/2327169448 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b0199c70 con 0x7f08b0071ea0 2022-09-07T21:30:49.849 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.759+0000 7f089f7fe700 1 -- 172.21.15.47:0/2327169448 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 11) v1 ==== 42623+0+0 (secure 0 0 0) 0x7f08ac02a020 con 0x7f08b0071ea0 2022-09-07T21:30:49.850 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.759+0000 7f08b72f3700 1 -- 172.21.15.47:0/2327169448 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b0060310 con 0x7f08b0071ea0 2022-09-07T21:30:49.850 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.760+0000 7f089f7fe700 1 --2- 172.21.15.47:0/2327169448 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f08980358a0 0x7f0898037d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:49.850 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.760+0000 7f089f7fe700 1 -- 172.21.15.47:0/2327169448 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f08ac0487f0 con 0x7f08b0071ea0 2022-09-07T21:30:49.851 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.760+0000 7f08b5af0700 1 --2- 172.21.15.47:0/2327169448 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f08980358a0 0x7f0898037d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:49.851 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.761+0000 7f08b5af0700 1 --2- 172.21.15.47:0/2327169448 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f08980358a0 0x7f0898037d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f08a0009750 tx=0x7f08a0006cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:49.851 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.765+0000 7f089f7fe700 1 -- 172.21.15.47:0/2327169448 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f08ac01a5a0 con 0x7f08b0071ea0 2022-09-07T21:30:49.851 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:48.844+0000 7f089f7fe700 1 -- 172.21.15.47:0/2327169448 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 12) v1 ==== 42703+0+0 (secure 0 0 0) 0x7f08ac017020 con 0x7f08b0071ea0 2022-09-07T21:30:49.852 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:49.001+0000 7f08b72f3700 1 -- 172.21.15.47:0/2327169448 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f08b00fee60 con 0x7f08980358a0 2022-09-07T21:30:49.852 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:49.208+0000 7f089f7fe700 1 -- 172.21.15.47:0/2327169448 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f08b00fee60 con 0x7f08980358a0 2022-09-07T21:30:49.852 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:49.211+0000 7f08b72f3700 1 -- 172.21.15.47:0/2327169448 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f08980358a0 msgr2=0x7f0898037d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:49.852 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:49.211+0000 7f08b72f3700 1 --2- 172.21.15.47:0/2327169448 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f08980358a0 0x7f0898037d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f08a0009750 tx=0x7f08a0006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:49.853 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:49.211+0000 7f08b72f3700 1 -- 172.21.15.47:0/2327169448 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08b0071ea0 msgr2=0x7f08b0199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:49.853 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:49.211+0000 7f08b72f3700 1 --2- 172.21.15.47:0/2327169448 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08b0071ea0 0x7f08b0199100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f08ac003560 tx=0x7f08ac003f60 comp rx=0 tx=0).stop 2022-09-07T21:30:49.853 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:49.211+0000 7f08b72f3700 1 -- 172.21.15.47:0/2327169448 shutdown_connections 2022-09-07T21:30:49.853 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:49.211+0000 7f08b72f3700 1 --2- 172.21.15.47:0/2327169448 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f08980358a0 0x7f0898037d50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:49.854 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:49.211+0000 7f08b72f3700 1 --2- 172.21.15.47:0/2327169448 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08b0071ea0 0x7f08b0199100 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:49.854 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:49.211+0000 7f08b72f3700 1 -- 172.21.15.47:0/2327169448 >> 172.21.15.47:0/2327169448 conn(0x7f08b00fda80 msgr2=0x7f08b00fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:49.854 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:49.211+0000 7f08b72f3700 1 -- 172.21.15.47:0/2327169448 shutdown_connections 2022-09-07T21:30:49.854 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:49.211+0000 7f08b72f3700 1 -- 172.21.15.47:0/2327169448 wait complete. 2022-09-07T21:30:49.855 INFO:teuthology.orchestra.run.smithi047.stdout:Creating initial admin user... 2022-09-07T21:30:50.095 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:49 smithi047 ceph-mon[90087]: mgrmap e12: x(active, since 2s) 2022-09-07T21:30:50.096 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:49 smithi047 ceph-mon[90087]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:30:50.096 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:49 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:30:50.096 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:49 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:30:50.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:50 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:30:51.554 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$TUh3Uf5XOkfF7alTROhTbO0yj/scV4WjHIF060Iu5HWRYj5fY9Z/W", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1662586251, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2022-09-07T21:30:51.554 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.556+0000 7f0ba18ee700 1 Processor -- start 2022-09-07T21:30:51.555 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.556+0000 7f0ba18ee700 1 -- start start 2022-09-07T21:30:51.556 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.557+0000 7f0ba18ee700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b9c071ea0 0x7f0b9c108650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:51.556 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.557+0000 7f0ba18ee700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c104040 con 0x7f0b9c071ea0 2022-09-07T21:30:51.557 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.557+0000 7f0b9affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b9c071ea0 0x7f0b9c108650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:51.558 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.557+0000 7f0b9affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b9c071ea0 0x7f0b9c108650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33670/0 (socket says 172.21.15.47:33670) 2022-09-07T21:30:51.558 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.557+0000 7f0b9affd700 1 -- 172.21.15.47:0/3164904342 learned_addr learned my addr 172.21.15.47:0/3164904342 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:51.559 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.557+0000 7f0b9affd700 1 -- 172.21.15.47:0/3164904342 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c108b90 con 0x7f0b9c071ea0 2022-09-07T21:30:51.559 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.558+0000 7f0b9affd700 1 --2- 172.21.15.47:0/3164904342 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b9c071ea0 0x7f0b9c108650 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009700 tx=0x7f0b8c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9593548f20de034e server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:51.559 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.558+0000 7f0b99ffb700 1 -- 172.21.15.47:0/3164904342 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b8c004160 con 0x7f0b9c071ea0 2022-09-07T21:30:51.559 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.558+0000 7f0b99ffb700 1 -- 172.21.15.47:0/3164904342 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0b8c0042c0 con 0x7f0b9c071ea0 2022-09-07T21:30:51.560 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.558+0000 7f0b99ffb700 1 -- 172.21.15.47:0/3164904342 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b8c003a10 con 0x7f0b9c071ea0 2022-09-07T21:30:51.560 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.559+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3164904342 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b9c071ea0 msgr2=0x7f0b9c108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:51.560 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.559+0000 7f0ba18ee700 1 --2- 172.21.15.47:0/3164904342 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b9c071ea0 0x7f0b9c108650 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009700 tx=0x7f0b8c009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:51.561 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.559+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3164904342 shutdown_connections 2022-09-07T21:30:51.562 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.559+0000 7f0ba18ee700 1 --2- 172.21.15.47:0/3164904342 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b9c071ea0 0x7f0b9c108650 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:51.562 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.559+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3164904342 >> 172.21.15.47:0/3164904342 conn(0x7f0b9c0fdc20 msgr2=0x7f0b9c100050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:51.562 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.559+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3164904342 shutdown_connections 2022-09-07T21:30:51.563 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.559+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3164904342 wait complete. 2022-09-07T21:30:51.563 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.560+0000 7f0ba18ee700 1 Processor -- start 2022-09-07T21:30:51.563 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.560+0000 7f0ba18ee700 1 -- start start 2022-09-07T21:30:51.563 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.561+0000 7f0ba18ee700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b9c071ea0 0x7f0b9c1992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:51.563 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.561+0000 7f0ba18ee700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c199800 con 0x7f0b9c071ea0 2022-09-07T21:30:51.564 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.561+0000 7f0b9affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b9c071ea0 0x7f0b9c1992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:51.564 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.561+0000 7f0b9affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b9c071ea0 0x7f0b9c1992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33686/0 (socket says 172.21.15.47:33686) 2022-09-07T21:30:51.564 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.561+0000 7f0b9affd700 1 -- 172.21.15.47:0/3788445432 learned_addr learned my addr 172.21.15.47:0/3788445432 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:51.564 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.561+0000 7f0b9affd700 1 -- 172.21.15.47:0/3788445432 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c0093e0 con 0x7f0b9c071ea0 2022-09-07T21:30:51.565 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.561+0000 7f0b9affd700 1 --2- 172.21.15.47:0/3788445432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b9c071ea0 0x7f0b9c1992c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c003560 tx=0x7f0b8c003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:51.565 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.562+0000 7f0ba08ec700 1 -- 172.21.15.47:0/3788445432 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b8c011840 con 0x7f0b9c071ea0 2022-09-07T21:30:51.565 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.562+0000 7f0ba08ec700 1 -- 172.21.15.47:0/3788445432 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0b8c01a430 con 0x7f0b9c071ea0 2022-09-07T21:30:51.565 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.562+0000 7f0ba08ec700 1 -- 172.21.15.47:0/3788445432 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b8c0193c0 con 0x7f0b9c071ea0 2022-09-07T21:30:51.565 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.562+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3788445432 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9c199a00 con 0x7f0b9c071ea0 2022-09-07T21:30:51.566 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.562+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3788445432 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9c199e30 con 0x7f0b9c071ea0 2022-09-07T21:30:51.566 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.563+0000 7f0ba08ec700 1 -- 172.21.15.47:0/3788445432 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 12) v1 ==== 42703+0+0 (secure 0 0 0) 0x7f0b8c02a020 con 0x7f0b9c071ea0 2022-09-07T21:30:51.566 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.563+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3788445432 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b9c060310 con 0x7f0b9c071ea0 2022-09-07T21:30:51.566 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.564+0000 7f0ba08ec700 1 --2- 172.21.15.47:0/3788445432 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0b88035930 0x7f0b88037de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:51.567 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.564+0000 7f0ba08ec700 1 -- 172.21.15.47:0/3788445432 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0b8c0488f0 con 0x7f0b9c071ea0 2022-09-07T21:30:51.567 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.565+0000 7f0b9a7fc700 1 --2- 172.21.15.47:0/3788445432 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0b88035930 0x7f0b88037de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:51.567 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.565+0000 7f0b9a7fc700 1 --2- 172.21.15.47:0/3788445432 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0b88035930 0x7f0b88037de0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0b90009750 tx=0x7f0b90006cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:51.567 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.569+0000 7f0ba08ec700 1 -- 172.21.15.47:0/3788445432 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f0b8c01a5a0 con 0x7f0b9c071ea0 2022-09-07T21:30:51.568 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:50.819+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3788445432 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f0b9c19ce40 con 0x7f0b88035930 2022-09-07T21:30:51.568 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:51.068+0000 7f0ba08ec700 1 -- 172.21.15.47:0/3788445432 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f0b9c19ce40 con 0x7f0b88035930 2022-09-07T21:30:51.568 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:51.070+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3788445432 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0b88035930 msgr2=0x7f0b88037de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:51.568 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:51.070+0000 7f0ba18ee700 1 --2- 172.21.15.47:0/3788445432 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0b88035930 0x7f0b88037de0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0b90009750 tx=0x7f0b90006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:51.568 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:51.070+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3788445432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b9c071ea0 msgr2=0x7f0b9c1992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:51.569 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:51.070+0000 7f0ba18ee700 1 --2- 172.21.15.47:0/3788445432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b9c071ea0 0x7f0b9c1992c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c003560 tx=0x7f0b8c003f60 comp rx=0 tx=0).stop 2022-09-07T21:30:51.569 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:51.070+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3788445432 shutdown_connections 2022-09-07T21:30:51.569 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:51.071+0000 7f0ba18ee700 1 --2- 172.21.15.47:0/3788445432 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0b88035930 0x7f0b88037de0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:51.569 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:51.071+0000 7f0ba18ee700 1 --2- 172.21.15.47:0/3788445432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b9c071ea0 0x7f0b9c1992c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:51.570 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:51.071+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3788445432 >> 172.21.15.47:0/3788445432 conn(0x7f0b9c0fdc20 msgr2=0x7f0b9c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:51.570 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:51.071+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3788445432 shutdown_connections 2022-09-07T21:30:51.570 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:51.071+0000 7f0ba18ee700 1 -- 172.21.15.47:0/3788445432 wait complete. 2022-09-07T21:30:51.570 INFO:teuthology.orchestra.run.smithi047.stdout:Fetching dashboard port number... 2022-09-07T21:30:52.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:52 smithi047 ceph-mon[90087]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:30:52.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:52 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:30:52.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:52 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:30:53.065 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stdout 8443 2022-09-07T21:30:53.066 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.215+0000 7fb7d8dbf700 1 Processor -- start 2022-09-07T21:30:53.066 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.215+0000 7fb7d8dbf700 1 -- start start 2022-09-07T21:30:53.066 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.215+0000 7fb7d8dbf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7d4103660 0x7fb7d4103a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:53.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.215+0000 7fb7d8dbf700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d4103fb0 con 0x7fb7d4103660 2022-09-07T21:30:53.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.216+0000 7fb7d259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7d4103660 0x7fb7d4103a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:53.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.216+0000 7fb7d259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7d4103660 0x7fb7d4103a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33696/0 (socket says 172.21.15.47:33696) 2022-09-07T21:30:53.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.216+0000 7fb7d259c700 1 -- 172.21.15.47:0/3558306394 learned_addr learned my addr 172.21.15.47:0/3558306394 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:53.067 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.216+0000 7fb7d259c700 1 -- 172.21.15.47:0/3558306394 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d41040f0 con 0x7fb7d4103660 2022-09-07T21:30:53.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.217+0000 7fb7d259c700 1 --2- 172.21.15.47:0/3558306394 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7d4103660 0x7fb7d4103a70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009700 tx=0x7fb7c4009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15623d59f3cb219a server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:53.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.217+0000 7fb7d159a700 1 -- 172.21.15.47:0/3558306394 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7c4004160 con 0x7fb7d4103660 2022-09-07T21:30:53.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.217+0000 7fb7d159a700 1 -- 172.21.15.47:0/3558306394 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb7c40042c0 con 0x7fb7d4103660 2022-09-07T21:30:53.068 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.217+0000 7fb7d159a700 1 -- 172.21.15.47:0/3558306394 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7c4003a10 con 0x7fb7d4103660 2022-09-07T21:30:53.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.218+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/3558306394 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7d4103660 msgr2=0x7fb7d4103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:53.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.218+0000 7fb7d8dbf700 1 --2- 172.21.15.47:0/3558306394 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7d4103660 0x7fb7d4103a70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009700 tx=0x7fb7c4009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:53.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.218+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/3558306394 shutdown_connections 2022-09-07T21:30:53.069 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.218+0000 7fb7d8dbf700 1 --2- 172.21.15.47:0/3558306394 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7d4103660 0x7fb7d4103a70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:53.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.218+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/3558306394 >> 172.21.15.47:0/3558306394 conn(0x7fb7d40fed50 msgr2=0x7fb7d4101180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:53.070 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.218+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/3558306394 shutdown_connections 2022-09-07T21:30:53.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.218+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/3558306394 wait complete. 2022-09-07T21:30:53.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.219+0000 7fb7d8dbf700 1 Processor -- start 2022-09-07T21:30:53.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.220+0000 7fb7d8dbf700 1 -- start start 2022-09-07T21:30:53.071 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.220+0000 7fb7d8dbf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7d4103660 0x7fb7d41992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:53.072 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.220+0000 7fb7d8dbf700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d4199830 con 0x7fb7d4103660 2022-09-07T21:30:53.072 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.220+0000 7fb7d259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7d4103660 0x7fb7d41992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:53.072 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.220+0000 7fb7d259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7d4103660 0x7fb7d41992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33708/0 (socket says 172.21.15.47:33708) 2022-09-07T21:30:53.072 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.220+0000 7fb7d259c700 1 -- 172.21.15.47:0/4284999334 learned_addr learned my addr 172.21.15.47:0/4284999334 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:53.073 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.221+0000 7fb7d259c700 1 -- 172.21.15.47:0/4284999334 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c40093e0 con 0x7fb7d4103660 2022-09-07T21:30:53.073 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.221+0000 7fb7d259c700 1 --2- 172.21.15.47:0/4284999334 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7d4103660 0x7fb7d41992f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4003560 tx=0x7fb7c4003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:53.073 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.221+0000 7fb7c37fe700 1 -- 172.21.15.47:0/4284999334 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7c4011840 con 0x7fb7d4103660 2022-09-07T21:30:53.073 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.221+0000 7fb7c37fe700 1 -- 172.21.15.47:0/4284999334 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb7c40119a0 con 0x7fb7d4103660 2022-09-07T21:30:53.074 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.221+0000 7fb7c37fe700 1 -- 172.21.15.47:0/4284999334 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7c40193c0 con 0x7fb7d4103660 2022-09-07T21:30:53.074 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.221+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/4284999334 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d4199a30 con 0x7fb7d4103660 2022-09-07T21:30:53.074 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.222+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/4284999334 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d4199e60 con 0x7fb7d4103660 2022-09-07T21:30:53.074 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.222+0000 7fb7c37fe700 1 -- 172.21.15.47:0/4284999334 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 12) v1 ==== 42703+0+0 (secure 0 0 0) 0x7fb7c402a020 con 0x7fb7d4103660 2022-09-07T21:30:53.075 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.223+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/4284999334 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d4060310 con 0x7fb7d4103660 2022-09-07T21:30:53.075 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.223+0000 7fb7c37fe700 1 --2- 172.21.15.47:0/4284999334 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb7b8035930 0x7fb7b8037de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:53.075 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.223+0000 7fb7c37fe700 1 -- 172.21.15.47:0/4284999334 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb7c4048a00 con 0x7fb7d4103660 2022-09-07T21:30:53.075 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.224+0000 7fb7d1d9b700 1 --2- 172.21.15.47:0/4284999334 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb7b8035930 0x7fb7b8037de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:53.076 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.224+0000 7fb7d1d9b700 1 --2- 172.21.15.47:0/4284999334 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb7b8035930 0x7fb7b8037de0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009750 tx=0x7fb7c8006cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:53.076 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.229+0000 7fb7c37fe700 1 -- 172.21.15.47:0/4284999334 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fb7c4022700 con 0x7fb7d4103660 2022-09-07T21:30:53.076 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.455+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/4284999334 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fb7d419cca0 con 0x7fb7d4103660 2022-09-07T21:30:53.077 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.456+0000 7fb7c37fe700 1 -- 172.21.15.47:0/4284999334 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fb7c400ee60 con 0x7fb7d4103660 2022-09-07T21:30:53.077 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.458+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/4284999334 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb7b8035930 msgr2=0x7fb7b8037de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:53.077 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.458+0000 7fb7d8dbf700 1 --2- 172.21.15.47:0/4284999334 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb7b8035930 0x7fb7b8037de0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009750 tx=0x7fb7c8006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:53.077 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.458+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/4284999334 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7d4103660 msgr2=0x7fb7d41992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:53.078 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.458+0000 7fb7d8dbf700 1 --2- 172.21.15.47:0/4284999334 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7d4103660 0x7fb7d41992f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4003560 tx=0x7fb7c4003f60 comp rx=0 tx=0).stop 2022-09-07T21:30:53.078 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.458+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/4284999334 shutdown_connections 2022-09-07T21:30:53.078 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.458+0000 7fb7d8dbf700 1 --2- 172.21.15.47:0/4284999334 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb7b8035930 0x7fb7b8037de0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:53.078 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.458+0000 7fb7d8dbf700 1 --2- 172.21.15.47:0/4284999334 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7d4103660 0x7fb7d41992f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:53.079 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.458+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/4284999334 >> 172.21.15.47:0/4284999334 conn(0x7fb7d40fed50 msgr2=0x7fb7d40ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:53.079 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.458+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/4284999334 shutdown_connections 2022-09-07T21:30:53.079 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:52.458+0000 7fb7d8dbf700 1 -- 172.21.15.47:0/4284999334 wait complete. 2022-09-07T21:30:53.087 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2022-09-07T21:30:53.088 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stdout disabled 2022-09-07T21:30:53.111 INFO:teuthology.orchestra.run.smithi047.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2022-09-07T21:30:53.112 INFO:teuthology.orchestra.run.smithi047.stdout:systemctl: stdout inactive 2022-09-07T21:30:53.112 INFO:teuthology.orchestra.run.smithi047.stdout:firewalld.service is not enabled 2022-09-07T21:30:53.112 INFO:teuthology.orchestra.run.smithi047.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2022-09-07T21:30:53.114 INFO:teuthology.orchestra.run.smithi047.stdout:Ceph Dashboard is now available at: 2022-09-07T21:30:53.115 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:30:53.115 INFO:teuthology.orchestra.run.smithi047.stdout: URL: https://smithi047.front.sepia.ceph.com:8443/ 2022-09-07T21:30:53.115 INFO:teuthology.orchestra.run.smithi047.stdout: User: admin 2022-09-07T21:30:53.116 INFO:teuthology.orchestra.run.smithi047.stdout: Password: 6qf9o094dh 2022-09-07T21:30:53.116 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:30:53.118 INFO:teuthology.orchestra.run.smithi047.stdout:Saving cluster configuration to /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config directory 2022-09-07T21:30:53.118 INFO:teuthology.orchestra.run.smithi047.stdout:Enabling autotune for osd_memory_target 2022-09-07T21:30:53.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:53 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/4284999334' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2022-09-07T21:30:54.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:54 smithi047 ceph-mon[90087]: mgrmap e13: x(active, since 6s) 2022-09-07T21:30:54.825 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.865+0000 7f3970607700 1 Processor -- start 2022-09-07T21:30:54.826 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.865+0000 7f3970607700 1 -- start start 2022-09-07T21:30:54.826 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.866+0000 7f3970607700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3968078cc0 0x7f3968077240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:54.827 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.866+0000 7f3970607700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3968077780 con 0x7f3968078cc0 2022-09-07T21:30:54.827 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.867+0000 7f396e3a3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3968078cc0 0x7f3968077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:54.827 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.867+0000 7f396e3a3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3968078cc0 0x7f3968077240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33718/0 (socket says 172.21.15.47:33718) 2022-09-07T21:30:54.828 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.867+0000 7f396e3a3700 1 -- 172.21.15.47:0/312207265 learned_addr learned my addr 172.21.15.47:0/312207265 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:54.828 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.867+0000 7f396e3a3700 1 -- 172.21.15.47:0/312207265 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39680778c0 con 0x7f3968078cc0 2022-09-07T21:30:54.829 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.867+0000 7f396e3a3700 1 --2- 172.21.15.47:0/312207265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3968078cc0 0x7f3968077240 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3964009700 tx=0x7f3964009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a042e6328f2af9a server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:54.829 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.868+0000 7f396d3a1700 1 -- 172.21.15.47:0/312207265 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3964004160 con 0x7f3968078cc0 2022-09-07T21:30:54.830 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.868+0000 7f396d3a1700 1 -- 172.21.15.47:0/312207265 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f39640042c0 con 0x7f3968078cc0 2022-09-07T21:30:54.830 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.868+0000 7f396d3a1700 1 -- 172.21.15.47:0/312207265 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3964003a10 con 0x7f3968078cc0 2022-09-07T21:30:54.830 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.868+0000 7f3970607700 1 -- 172.21.15.47:0/312207265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3968078cc0 msgr2=0x7f3968077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:54.831 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.868+0000 7f3970607700 1 --2- 172.21.15.47:0/312207265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3968078cc0 0x7f3968077240 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3964009700 tx=0x7f3964009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:54.831 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.868+0000 7f3970607700 1 -- 172.21.15.47:0/312207265 shutdown_connections 2022-09-07T21:30:54.832 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.868+0000 7f3970607700 1 --2- 172.21.15.47:0/312207265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3968078cc0 0x7f3968077240 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:54.832 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.868+0000 7f3970607700 1 -- 172.21.15.47:0/312207265 >> 172.21.15.47:0/312207265 conn(0x7f39680fed50 msgr2=0x7f3968101180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:54.833 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.869+0000 7f3970607700 1 -- 172.21.15.47:0/312207265 shutdown_connections 2022-09-07T21:30:54.833 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.869+0000 7f3970607700 1 -- 172.21.15.47:0/312207265 wait complete. 2022-09-07T21:30:54.833 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.870+0000 7f3970607700 1 Processor -- start 2022-09-07T21:30:54.834 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.870+0000 7f3970607700 1 -- start start 2022-09-07T21:30:54.834 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.870+0000 7f3970607700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3968078cc0 0x7f396819d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:54.834 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.870+0000 7f3970607700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396819dc40 con 0x7f3968078cc0 2022-09-07T21:30:54.835 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.871+0000 7f396e3a3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3968078cc0 0x7f396819d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:54.835 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.871+0000 7f396e3a3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3968078cc0 0x7f396819d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33726/0 (socket says 172.21.15.47:33726) 2022-09-07T21:30:54.836 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.871+0000 7f396e3a3700 1 -- 172.21.15.47:0/1162415284 learned_addr learned my addr 172.21.15.47:0/1162415284 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:54.836 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.871+0000 7f396e3a3700 1 -- 172.21.15.47:0/1162415284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39640093e0 con 0x7f3968078cc0 2022-09-07T21:30:54.836 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.871+0000 7f396e3a3700 1 --2- 172.21.15.47:0/1162415284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3968078cc0 0x7f396819d700 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3964003560 tx=0x7f3964003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:54.837 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.872+0000 7f39577fe700 1 -- 172.21.15.47:0/1162415284 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3964011840 con 0x7f3968078cc0 2022-09-07T21:30:54.837 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.872+0000 7f39577fe700 1 -- 172.21.15.47:0/1162415284 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f39640119a0 con 0x7f3968078cc0 2022-09-07T21:30:54.837 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.872+0000 7f39577fe700 1 -- 172.21.15.47:0/1162415284 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39640193c0 con 0x7f3968078cc0 2022-09-07T21:30:54.838 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.872+0000 7f3970607700 1 -- 172.21.15.47:0/1162415284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396819de40 con 0x7f3968078cc0 2022-09-07T21:30:54.838 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.872+0000 7f3970607700 1 -- 172.21.15.47:0/1162415284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396819e270 con 0x7f3968078cc0 2022-09-07T21:30:54.839 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.873+0000 7f39577fe700 1 -- 172.21.15.47:0/1162415284 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f396402a020 con 0x7f3968078cc0 2022-09-07T21:30:54.839 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.873+0000 7f3970607700 1 -- 172.21.15.47:0/1162415284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3968060310 con 0x7f3968078cc0 2022-09-07T21:30:54.839 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.873+0000 7f39577fe700 1 --2- 172.21.15.47:0/1162415284 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f39500359d0 0x7f3950037e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:54.840 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.874+0000 7f39577fe700 1 -- 172.21.15.47:0/1162415284 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3964048a60 con 0x7f3968078cc0 2022-09-07T21:30:54.840 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.874+0000 7f396dba2700 1 --2- 172.21.15.47:0/1162415284 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f39500359d0 0x7f3950037e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:54.840 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.874+0000 7f396dba2700 1 --2- 172.21.15.47:0/1162415284 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f39500359d0 0x7f3950037e80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3958009750 tx=0x7f3958006cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:54.840 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:53.879+0000 7f39577fe700 1 -- 172.21.15.47:0/1162415284 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f3964011b10 con 0x7f3968078cc0 2022-09-07T21:30:54.841 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:54.121+0000 7f3970607700 1 -- 172.21.15.47:0/1162415284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f3968105500 con 0x7f3968078cc0 2022-09-07T21:30:54.841 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:54.122+0000 7f39577fe700 1 -- 172.21.15.47:0/1162415284 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f3968105500 con 0x7f3968078cc0 2022-09-07T21:30:54.841 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:54.124+0000 7f3970607700 1 -- 172.21.15.47:0/1162415284 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f39500359d0 msgr2=0x7f3950037e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:54.841 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:54.124+0000 7f3970607700 1 --2- 172.21.15.47:0/1162415284 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f39500359d0 0x7f3950037e80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3958009750 tx=0x7f3958006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:54.841 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:54.124+0000 7f3970607700 1 -- 172.21.15.47:0/1162415284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3968078cc0 msgr2=0x7f396819d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:54.842 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:54.124+0000 7f3970607700 1 --2- 172.21.15.47:0/1162415284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3968078cc0 0x7f396819d700 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3964003560 tx=0x7f3964003f60 comp rx=0 tx=0).stop 2022-09-07T21:30:54.842 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:54.124+0000 7f3970607700 1 -- 172.21.15.47:0/1162415284 shutdown_connections 2022-09-07T21:30:54.842 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:54.124+0000 7f3970607700 1 --2- 172.21.15.47:0/1162415284 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f39500359d0 0x7f3950037e80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:54.842 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:54.124+0000 7f3970607700 1 --2- 172.21.15.47:0/1162415284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3968078cc0 0x7f396819d700 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:54.843 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:54.124+0000 7f3970607700 1 -- 172.21.15.47:0/1162415284 >> 172.21.15.47:0/1162415284 conn(0x7f39680fed50 msgr2=0x7f3968104990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:54.843 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:54.124+0000 7f3970607700 1 -- 172.21.15.47:0/1162415284 shutdown_connections 2022-09-07T21:30:54.843 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:54.124+0000 7f3970607700 1 -- 172.21.15.47:0/1162415284 wait complete. 2022-09-07T21:30:56.225 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.456+0000 7f8d7f098700 1 Processor -- start 2022-09-07T21:30:56.226 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.457+0000 7f8d7f098700 1 -- start start 2022-09-07T21:30:56.226 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.457+0000 7f8d7f098700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d78105860 0x7f8d78105c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:56.226 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.457+0000 7f8d7f098700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d781061b0 con 0x7f8d78105860 2022-09-07T21:30:56.227 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.457+0000 7f8d7ce34700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d78105860 0x7f8d78105c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:56.227 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.457+0000 7f8d7ce34700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d78105860 0x7f8d78105c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33734/0 (socket says 172.21.15.47:33734) 2022-09-07T21:30:56.227 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.457+0000 7f8d7ce34700 1 -- 172.21.15.47:0/2509190784 learned_addr learned my addr 172.21.15.47:0/2509190784 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:56.227 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.458+0000 7f8d7ce34700 1 -- 172.21.15.47:0/2509190784 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d781062f0 con 0x7f8d78105860 2022-09-07T21:30:56.228 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.458+0000 7f8d7ce34700 1 --2- 172.21.15.47:0/2509190784 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d78105860 0x7f8d78105c70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c009700 tx=0x7f8d6c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5618455f525926 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:56.228 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.458+0000 7f8d777fe700 1 -- 172.21.15.47:0/2509190784 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d6c004160 con 0x7f8d78105860 2022-09-07T21:30:56.228 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.459+0000 7f8d777fe700 1 -- 172.21.15.47:0/2509190784 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8d6c0042c0 con 0x7f8d78105860 2022-09-07T21:30:56.228 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.459+0000 7f8d777fe700 1 -- 172.21.15.47:0/2509190784 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d6c003a10 con 0x7f8d78105860 2022-09-07T21:30:56.229 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.459+0000 7f8d7f098700 1 -- 172.21.15.47:0/2509190784 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d78105860 msgr2=0x7f8d78105c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:56.229 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.459+0000 7f8d7f098700 1 --2- 172.21.15.47:0/2509190784 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d78105860 0x7f8d78105c70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c009700 tx=0x7f8d6c009a10 comp rx=0 tx=0).stop 2022-09-07T21:30:56.229 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.459+0000 7f8d7f098700 1 -- 172.21.15.47:0/2509190784 shutdown_connections 2022-09-07T21:30:56.229 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.459+0000 7f8d7f098700 1 --2- 172.21.15.47:0/2509190784 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d78105860 0x7f8d78105c70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:56.230 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.459+0000 7f8d7f098700 1 -- 172.21.15.47:0/2509190784 >> 172.21.15.47:0/2509190784 conn(0x7f8d781013c0 msgr2=0x7f8d781037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:56.230 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.459+0000 7f8d7f098700 1 -- 172.21.15.47:0/2509190784 shutdown_connections 2022-09-07T21:30:56.230 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.459+0000 7f8d7f098700 1 -- 172.21.15.47:0/2509190784 wait complete. 2022-09-07T21:30:56.230 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.461+0000 7f8d7f098700 1 Processor -- start 2022-09-07T21:30:56.231 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.461+0000 7f8d7f098700 1 -- start start 2022-09-07T21:30:56.231 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.461+0000 7f8d7f098700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d78105860 0x7f8d781992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:56.231 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.461+0000 7f8d7f098700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d78199810 con 0x7f8d78105860 2022-09-07T21:30:56.231 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.461+0000 7f8d7ce34700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d78105860 0x7f8d781992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:56.231 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.461+0000 7f8d7ce34700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d78105860 0x7f8d781992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33744/0 (socket says 172.21.15.47:33744) 2022-09-07T21:30:56.232 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.462+0000 7f8d7ce34700 1 -- 172.21.15.47:0/460964923 learned_addr learned my addr 172.21.15.47:0/460964923 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:56.232 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.462+0000 7f8d7ce34700 1 -- 172.21.15.47:0/460964923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c0093e0 con 0x7f8d78105860 2022-09-07T21:30:56.232 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.462+0000 7f8d7ce34700 1 --2- 172.21.15.47:0/460964923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d78105860 0x7f8d781992d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c003560 tx=0x7f8d6c003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:56.232 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.462+0000 7f8d75ffb700 1 -- 172.21.15.47:0/460964923 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d6c011840 con 0x7f8d78105860 2022-09-07T21:30:56.233 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.462+0000 7f8d75ffb700 1 -- 172.21.15.47:0/460964923 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8d6c0119a0 con 0x7f8d78105860 2022-09-07T21:30:56.233 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.462+0000 7f8d75ffb700 1 -- 172.21.15.47:0/460964923 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d6c0193c0 con 0x7f8d78105860 2022-09-07T21:30:56.233 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.462+0000 7f8d7f098700 1 -- 172.21.15.47:0/460964923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d78199a10 con 0x7f8d78105860 2022-09-07T21:30:56.233 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.463+0000 7f8d7f098700 1 -- 172.21.15.47:0/460964923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d78199e40 con 0x7f8d78105860 2022-09-07T21:30:56.234 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.464+0000 7f8d7f098700 1 -- 172.21.15.47:0/460964923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d781076c0 con 0x7f8d78105860 2022-09-07T21:30:56.234 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.464+0000 7f8d75ffb700 1 -- 172.21.15.47:0/460964923 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f8d6c02a020 con 0x7f8d78105860 2022-09-07T21:30:56.234 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.464+0000 7f8d75ffb700 1 --2- 172.21.15.47:0/460964923 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8d5c0359d0 0x7f8d5c037e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:56.234 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.464+0000 7f8d75ffb700 1 -- 172.21.15.47:0/460964923 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8d6c0489f0 con 0x7f8d78105860 2022-09-07T21:30:56.235 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.465+0000 7f8d77fff700 1 --2- 172.21.15.47:0/460964923 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8d5c0359d0 0x7f8d5c037e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:56.235 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.465+0000 7f8d77fff700 1 --2- 172.21.15.47:0/460964923 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8d5c0359d0 0x7f8d5c037e80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009750 tx=0x7f8d64006cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:56.235 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.469+0000 7f8d75ffb700 1 -- 172.21.15.47:0/460964923 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f8d6c02b940 con 0x7f8d78105860 2022-09-07T21:30:56.235 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.803+0000 7f8d7f098700 1 -- 172.21.15.47:0/460964923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f8d7819cc80 con 0x7f8d78105860 2022-09-07T21:30:56.235 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.806+0000 7f8d75ffb700 1 -- 172.21.15.47:0/460964923 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f8d6c02b760 con 0x7f8d78105860 2022-09-07T21:30:56.236 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.809+0000 7f8d7f098700 1 -- 172.21.15.47:0/460964923 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8d5c0359d0 msgr2=0x7f8d5c037e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:56.236 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.809+0000 7f8d7f098700 1 --2- 172.21.15.47:0/460964923 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8d5c0359d0 0x7f8d5c037e80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009750 tx=0x7f8d64006cc0 comp rx=0 tx=0).stop 2022-09-07T21:30:56.236 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.809+0000 7f8d7f098700 1 -- 172.21.15.47:0/460964923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d78105860 msgr2=0x7f8d781992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:56.236 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.809+0000 7f8d7f098700 1 --2- 172.21.15.47:0/460964923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d78105860 0x7f8d781992d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c003560 tx=0x7f8d6c003f60 comp rx=0 tx=0).stop 2022-09-07T21:30:56.237 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.809+0000 7f8d7f098700 1 -- 172.21.15.47:0/460964923 shutdown_connections 2022-09-07T21:30:56.237 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.809+0000 7f8d7f098700 1 --2- 172.21.15.47:0/460964923 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8d5c0359d0 0x7f8d5c037e80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:56.237 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.809+0000 7f8d7f098700 1 --2- 172.21.15.47:0/460964923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d78105860 0x7f8d781992d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:56.237 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.809+0000 7f8d7f098700 1 -- 172.21.15.47:0/460964923 >> 172.21.15.47:0/460964923 conn(0x7f8d781013c0 msgr2=0x7f8d78102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:56.238 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.809+0000 7f8d7f098700 1 -- 172.21.15.47:0/460964923 shutdown_connections 2022-09-07T21:30:56.238 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr 2022-09-07T21:30:55.809+0000 7f8d7f098700 1 -- 172.21.15.47:0/460964923 wait complete. 2022-09-07T21:30:56.238 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2022-09-07T21:30:56.238 INFO:teuthology.orchestra.run.smithi047.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2022-09-07T21:30:56.239 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:30:56.239 INFO:teuthology.orchestra.run.smithi047.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2022-09-07T21:30:56.239 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:30:56.239 INFO:teuthology.orchestra.run.smithi047.stdout:Or, if you are only running a single cluster on this host: 2022-09-07T21:30:56.239 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:30:56.240 INFO:teuthology.orchestra.run.smithi047.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2022-09-07T21:30:56.240 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:30:56.240 INFO:teuthology.orchestra.run.smithi047.stdout:Please consider enabling telemetry to help improve Ceph: 2022-09-07T21:30:56.240 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:30:56.240 INFO:teuthology.orchestra.run.smithi047.stdout: ceph telemetry on 2022-09-07T21:30:56.241 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:30:56.241 INFO:teuthology.orchestra.run.smithi047.stdout:For more information see: 2022-09-07T21:30:56.241 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:30:56.241 INFO:teuthology.orchestra.run.smithi047.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2022-09-07T21:30:56.241 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:30:56.242 INFO:teuthology.orchestra.run.smithi047.stdout:Bootstrap complete. 2022-09-07T21:30:56.298 INFO:tasks.cephadm:Fetching config... 2022-09-07T21:30:56.299 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:30:56.300 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2022-09-07T21:30:56.366 INFO:tasks.cephadm:Fetching client.admin keyring... 2022-09-07T21:30:56.366 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:30:56.366 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2022-09-07T21:30:56.425 INFO:tasks.cephadm:Fetching mon keyring... 2022-09-07T21:30:56.425 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:30:56.426 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/keyring of=/dev/stdout 2022-09-07T21:30:56.516 INFO:tasks.cephadm:Fetching pub ssh key... 2022-09-07T21:30:56.517 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:30:56.517 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2022-09-07T21:30:56.579 INFO:tasks.cephadm:Installing pub ssh key for root users... 2022-09-07T21:30:56.580 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzw4GGLUr/epxtyWeQ0CCRdUvr7BLDy3yWP/sNTdatpsNGQQccVs3gNrK4cijTfxRelvBxTypbT7X40DfhoTAOLgUQ0LwEVCMKI6E1zrZogUvO0E+w+3hEc91fafY8hQTBEpc5zwqhQsM7CwdRhg61JV+d7Gry9XxZSZ/4RkwcXblVWZ2st48FUdl9fGPOKGy2UK3b7D3cA1Jxpd0cFlSBpjqHYoTB2MTsjj0pc8BeVsn1VyQ11xvlFpGGKZSWXLSXJMTpoc01Gn9LHtowt51gK8x6bGdZqVq8ntkKnemQBZWOjUPKNBD442kYDl7z5iCo812w3IfCkidUfQk9CAfAegnZhmw5VDihPfN7kSt4Drtak2cd20jLbYvrACZxH+Pt8fH8RTe5D2OPWZRFCCmlWLR/w1HMTJ/ABlFnD+aPOpdcRbkNpXSPpvctS5Bmy4cINS3IluvqPZyqqT0NA0i7bzkoN3YjvJEjGMZkFhy1Wr4CzMThtIiPVC0jIcyg95c= ceph-01b8396e-2ef4-11ed-8431-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-09-07T21:30:56.708 INFO:teuthology.orchestra.run.smithi047.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzw4GGLUr/epxtyWeQ0CCRdUvr7BLDy3yWP/sNTdatpsNGQQccVs3gNrK4cijTfxRelvBxTypbT7X40DfhoTAOLgUQ0LwEVCMKI6E1zrZogUvO0E+w+3hEc91fafY8hQTBEpc5zwqhQsM7CwdRhg61JV+d7Gry9XxZSZ/4RkwcXblVWZ2st48FUdl9fGPOKGy2UK3b7D3cA1Jxpd0cFlSBpjqHYoTB2MTsjj0pc8BeVsn1VyQ11xvlFpGGKZSWXLSXJMTpoc01Gn9LHtowt51gK8x6bGdZqVq8ntkKnemQBZWOjUPKNBD442kYDl7z5iCo812w3IfCkidUfQk9CAfAegnZhmw5VDihPfN7kSt4Drtak2cd20jLbYvrACZxH+Pt8fH8RTe5D2OPWZRFCCmlWLR/w1HMTJ/ABlFnD+aPOpdcRbkNpXSPpvctS5Bmy4cINS3IluvqPZyqqT0NA0i7bzkoN3YjvJEjGMZkFhy1Wr4CzMThtIiPVC0jIcyg95c= ceph-01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:30:56.745 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzw4GGLUr/epxtyWeQ0CCRdUvr7BLDy3yWP/sNTdatpsNGQQccVs3gNrK4cijTfxRelvBxTypbT7X40DfhoTAOLgUQ0LwEVCMKI6E1zrZogUvO0E+w+3hEc91fafY8hQTBEpc5zwqhQsM7CwdRhg61JV+d7Gry9XxZSZ/4RkwcXblVWZ2st48FUdl9fGPOKGy2UK3b7D3cA1Jxpd0cFlSBpjqHYoTB2MTsjj0pc8BeVsn1VyQ11xvlFpGGKZSWXLSXJMTpoc01Gn9LHtowt51gK8x6bGdZqVq8ntkKnemQBZWOjUPKNBD442kYDl7z5iCo812w3IfCkidUfQk9CAfAegnZhmw5VDihPfN7kSt4Drtak2cd20jLbYvrACZxH+Pt8fH8RTe5D2OPWZRFCCmlWLR/w1HMTJ/ABlFnD+aPOpdcRbkNpXSPpvctS5Bmy4cINS3IluvqPZyqqT0NA0i7bzkoN3YjvJEjGMZkFhy1Wr4CzMThtIiPVC0jIcyg95c= ceph-01b8396e-2ef4-11ed-8431-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-09-07T21:30:56.793 INFO:teuthology.orchestra.run.smithi055.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzw4GGLUr/epxtyWeQ0CCRdUvr7BLDy3yWP/sNTdatpsNGQQccVs3gNrK4cijTfxRelvBxTypbT7X40DfhoTAOLgUQ0LwEVCMKI6E1zrZogUvO0E+w+3hEc91fafY8hQTBEpc5zwqhQsM7CwdRhg61JV+d7Gry9XxZSZ/4RkwcXblVWZ2st48FUdl9fGPOKGy2UK3b7D3cA1Jxpd0cFlSBpjqHYoTB2MTsjj0pc8BeVsn1VyQ11xvlFpGGKZSWXLSXJMTpoc01Gn9LHtowt51gK8x6bGdZqVq8ntkKnemQBZWOjUPKNBD442kYDl7z5iCo812w3IfCkidUfQk9CAfAegnZhmw5VDihPfN7kSt4Drtak2cd20jLbYvrACZxH+Pt8fH8RTe5D2OPWZRFCCmlWLR/w1HMTJ/ABlFnD+aPOpdcRbkNpXSPpvctS5Bmy4cINS3IluvqPZyqqT0NA0i7bzkoN3YjvJEjGMZkFhy1Wr4CzMThtIiPVC0jIcyg95c= ceph-01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:30:56.806 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzw4GGLUr/epxtyWeQ0CCRdUvr7BLDy3yWP/sNTdatpsNGQQccVs3gNrK4cijTfxRelvBxTypbT7X40DfhoTAOLgUQ0LwEVCMKI6E1zrZogUvO0E+w+3hEc91fafY8hQTBEpc5zwqhQsM7CwdRhg61JV+d7Gry9XxZSZ/4RkwcXblVWZ2st48FUdl9fGPOKGy2UK3b7D3cA1Jxpd0cFlSBpjqHYoTB2MTsjj0pc8BeVsn1VyQ11xvlFpGGKZSWXLSXJMTpoc01Gn9LHtowt51gK8x6bGdZqVq8ntkKnemQBZWOjUPKNBD442kYDl7z5iCo812w3IfCkidUfQk9CAfAegnZhmw5VDihPfN7kSt4Drtak2cd20jLbYvrACZxH+Pt8fH8RTe5D2OPWZRFCCmlWLR/w1HMTJ/ABlFnD+aPOpdcRbkNpXSPpvctS5Bmy4cINS3IluvqPZyqqT0NA0i7bzkoN3YjvJEjGMZkFhy1Wr4CzMThtIiPVC0jIcyg95c= ceph-01b8396e-2ef4-11ed-8431-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-09-07T21:30:56.853 INFO:teuthology.orchestra.run.smithi059.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzw4GGLUr/epxtyWeQ0CCRdUvr7BLDy3yWP/sNTdatpsNGQQccVs3gNrK4cijTfxRelvBxTypbT7X40DfhoTAOLgUQ0LwEVCMKI6E1zrZogUvO0E+w+3hEc91fafY8hQTBEpc5zwqhQsM7CwdRhg61JV+d7Gry9XxZSZ/4RkwcXblVWZ2st48FUdl9fGPOKGy2UK3b7D3cA1Jxpd0cFlSBpjqHYoTB2MTsjj0pc8BeVsn1VyQ11xvlFpGGKZSWXLSXJMTpoc01Gn9LHtowt51gK8x6bGdZqVq8ntkKnemQBZWOjUPKNBD442kYDl7z5iCo812w3IfCkidUfQk9CAfAegnZhmw5VDihPfN7kSt4Drtak2cd20jLbYvrACZxH+Pt8fH8RTe5D2OPWZRFCCmlWLR/w1HMTJ/ABlFnD+aPOpdcRbkNpXSPpvctS5Bmy4cINS3IluvqPZyqqT0NA0i7bzkoN3YjvJEjGMZkFhy1Wr4CzMThtIiPVC0jIcyg95c= ceph-01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:30:56.867 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2022-09-07T21:30:57.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:56 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/460964923' entity='client.admin' 2022-09-07T21:30:58.097 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:30:59.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:59 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:30:59.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:59 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:30:59.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:59 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi047", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:30:59.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:59 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:30:59.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:59 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:30:59.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:30:59 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:30:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.776+0000 7fa27e1eb700 1 -- 172.21.15.47:0/3864396310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2780fe340 msgr2=0x7fa278100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:30:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.776+0000 7fa27e1eb700 1 --2- 172.21.15.47:0/3864396310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2780fe340 0x7fa278100720 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa268009790 tx=0x7fa268009aa0 comp rx=0 tx=0).stop 2022-09-07T21:30:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.776+0000 7fa27e1eb700 1 -- 172.21.15.47:0/3864396310 shutdown_connections 2022-09-07T21:30:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.776+0000 7fa27e1eb700 1 --2- 172.21.15.47:0/3864396310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2780fe340 0x7fa278100720 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:30:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.776+0000 7fa27e1eb700 1 -- 172.21.15.47:0/3864396310 >> 172.21.15.47:0/3864396310 conn(0x7fa2780f7d60 msgr2=0x7fa2780fa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:30:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.776+0000 7fa27e1eb700 1 -- 172.21.15.47:0/3864396310 shutdown_connections 2022-09-07T21:30:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.776+0000 7fa27e1eb700 1 -- 172.21.15.47:0/3864396310 wait complete. 2022-09-07T21:30:59.782 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.777+0000 7fa27e1eb700 1 Processor -- start 2022-09-07T21:30:59.782 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.778+0000 7fa27e1eb700 1 -- start start 2022-09-07T21:30:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.778+0000 7fa27e1eb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2780fe340 0x7fa2781949b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.778+0000 7fa27e1eb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa278194ef0 con 0x7fa2780fe340 2022-09-07T21:30:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.778+0000 7fa2777fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2780fe340 0x7fa2781949b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.778+0000 7fa2777fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2780fe340 0x7fa2781949b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46652/0 (socket says 172.21.15.47:46652) 2022-09-07T21:30:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.778+0000 7fa2777fe700 1 -- 172.21.15.47:0/2605781862 learned_addr learned my addr 172.21.15.47:0/2605781862 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:30:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.779+0000 7fa2777fe700 1 -- 172.21.15.47:0/2605781862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa268009470 con 0x7fa2780fe340 2022-09-07T21:30:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.779+0000 7fa2777fe700 1 --2- 172.21.15.47:0/2605781862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2780fe340 0x7fa2781949b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa2680059c0 tx=0x7fa26800f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.779+0000 7fa274ff9700 1 -- 172.21.15.47:0/2605781862 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa26801d070 con 0x7fa2780fe340 2022-09-07T21:30:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.779+0000 7fa274ff9700 1 -- 172.21.15.47:0/2605781862 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa26800fde0 con 0x7fa2780fe340 2022-09-07T21:30:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.779+0000 7fa274ff9700 1 -- 172.21.15.47:0/2605781862 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa26800b460 con 0x7fa2780fe340 2022-09-07T21:30:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.779+0000 7fa27e1eb700 1 -- 172.21.15.47:0/2605781862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2781950f0 con 0x7fa2780fe340 2022-09-07T21:30:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.780+0000 7fa27e1eb700 1 -- 172.21.15.47:0/2605781862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2781955a0 con 0x7fa2780fe340 2022-09-07T21:30:59.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.781+0000 7fa274ff9700 1 -- 172.21.15.47:0/2605781862 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7fa268028020 con 0x7fa2780fe340 2022-09-07T21:30:59.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.781+0000 7fa27e1eb700 1 -- 172.21.15.47:0/2605781862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa278103500 con 0x7fa2780fe340 2022-09-07T21:30:59.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.781+0000 7fa274ff9700 1 --2- 172.21.15.47:0/2605781862 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa260035910 0x7fa260037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:30:59.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.781+0000 7fa274ff9700 1 -- 172.21.15.47:0/2605781862 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa268048b50 con 0x7fa2780fe340 2022-09-07T21:30:59.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.781+0000 7fa276ffd700 1 --2- 172.21.15.47:0/2605781862 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa260035910 0x7fa260037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:30:59.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.782+0000 7fa276ffd700 1 --2- 172.21.15.47:0/2605781862 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa260035910 0x7fa260037dc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009750 tx=0x7fa26c006cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:30:59.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:30:59.786+0000 7fa274ff9700 1 -- 172.21.15.47:0/2605781862 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fa268017440 con 0x7fa2780fe340 2022-09-07T21:31:00.017 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:00.014+0000 7fa27e1eb700 1 -- 172.21.15.47:0/2605781862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fa27805fae0 con 0x7fa2780fe340 2022-09-07T21:31:00.020 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:00.018+0000 7fa274ff9700 1 -- 172.21.15.47:0/2605781862 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa268021bb0 con 0x7fa2780fe340 2022-09-07T21:31:00.023 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:00.020+0000 7fa27e1eb700 1 -- 172.21.15.47:0/2605781862 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa260035910 msgr2=0x7fa260037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:00.024 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:00.020+0000 7fa27e1eb700 1 --2- 172.21.15.47:0/2605781862 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa260035910 0x7fa260037dc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009750 tx=0x7fa26c006cc0 comp rx=0 tx=0).stop 2022-09-07T21:31:00.024 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:00.020+0000 7fa27e1eb700 1 -- 172.21.15.47:0/2605781862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2780fe340 msgr2=0x7fa2781949b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:00.024 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:00.020+0000 7fa27e1eb700 1 --2- 172.21.15.47:0/2605781862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2780fe340 0x7fa2781949b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa2680059c0 tx=0x7fa26800f460 comp rx=0 tx=0).stop 2022-09-07T21:31:00.025 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:00.021+0000 7fa27e1eb700 1 -- 172.21.15.47:0/2605781862 shutdown_connections 2022-09-07T21:31:00.026 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:00.021+0000 7fa27e1eb700 1 --2- 172.21.15.47:0/2605781862 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa260035910 0x7fa260037dc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:00.026 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:00.021+0000 7fa27e1eb700 1 --2- 172.21.15.47:0/2605781862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2780fe340 0x7fa2781949b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:00.026 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:00.021+0000 7fa27e1eb700 1 -- 172.21.15.47:0/2605781862 >> 172.21.15.47:0/2605781862 conn(0x7fa2780f7d60 msgr2=0x7fa2780fa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:00.027 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:00.021+0000 7fa27e1eb700 1 -- 172.21.15.47:0/2605781862 shutdown_connections 2022-09-07T21:31:00.027 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:00.021+0000 7fa27e1eb700 1 -- 172.21.15.47:0/2605781862 wait complete. 2022-09-07T21:31:00.576 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2022-09-07T21:31:00.577 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2022-09-07T21:31:01.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:01 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2605781862' entity='client.admin' 2022-09-07T21:31:01.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:01 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:01.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:01 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:01.455 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:31:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.111+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3670425638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc100fe180 msgr2=0x7fcc10100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:03.115 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.111+0000 7fcc0ffff700 1 --2- 172.21.15.47:0/3670425638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc100fe180 0x7fcc10100560 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcc00009790 tx=0x7fcc00009aa0 comp rx=0 tx=0).stop 2022-09-07T21:31:03.115 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.111+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3670425638 shutdown_connections 2022-09-07T21:31:03.115 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.111+0000 7fcc0ffff700 1 --2- 172.21.15.47:0/3670425638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc100fe180 0x7fcc10100560 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:03.116 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.111+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3670425638 >> 172.21.15.47:0/3670425638 conn(0x7fcc100f7bc0 msgr2=0x7fcc100f9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:03.116 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.112+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3670425638 shutdown_connections 2022-09-07T21:31:03.116 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.112+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3670425638 wait complete. 2022-09-07T21:31:03.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.113+0000 7fcc0ffff700 1 Processor -- start 2022-09-07T21:31:03.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.113+0000 7fcc0ffff700 1 -- start start 2022-09-07T21:31:03.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.113+0000 7fcc0ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc100fe180 0x7fcc101947a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:03.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.113+0000 7fcc0ffff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc10194ce0 con 0x7fcc100fe180 2022-09-07T21:31:03.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.114+0000 7fcc0effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc100fe180 0x7fcc101947a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:03.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.114+0000 7fcc0effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc100fe180 0x7fcc101947a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46676/0 (socket says 172.21.15.47:46676) 2022-09-07T21:31:03.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.114+0000 7fcc0effd700 1 -- 172.21.15.47:0/3231239498 learned_addr learned my addr 172.21.15.47:0/3231239498 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:31:03.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.114+0000 7fcc0effd700 1 -- 172.21.15.47:0/3231239498 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc00009470 con 0x7fcc100fe180 2022-09-07T21:31:03.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.114+0000 7fcc0effd700 1 --2- 172.21.15.47:0/3231239498 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc100fe180 0x7fcc101947a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcc000059c0 tx=0x7fcc0000f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:03.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.115+0000 7fcbf3fff700 1 -- 172.21.15.47:0/3231239498 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc0001d070 con 0x7fcc100fe180 2022-09-07T21:31:03.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.115+0000 7fcbf3fff700 1 -- 172.21.15.47:0/3231239498 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcc0000fde0 con 0x7fcc100fe180 2022-09-07T21:31:03.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.115+0000 7fcbf3fff700 1 -- 172.21.15.47:0/3231239498 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc0000b460 con 0x7fcc100fe180 2022-09-07T21:31:03.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.115+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3231239498 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc10194ee0 con 0x7fcc100fe180 2022-09-07T21:31:03.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.115+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3231239498 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc10195390 con 0x7fcc100fe180 2022-09-07T21:31:03.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.116+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3231239498 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc10103340 con 0x7fcc100fe180 2022-09-07T21:31:03.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.116+0000 7fcbf3fff700 1 -- 172.21.15.47:0/3231239498 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7fcc00028020 con 0x7fcc100fe180 2022-09-07T21:31:03.123 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.116+0000 7fcbf3fff700 1 --2- 172.21.15.47:0/3231239498 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcbf8035910 0x7fcbf8037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:03.123 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.116+0000 7fcbf3fff700 1 -- 172.21.15.47:0/3231239498 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcc00048c20 con 0x7fcc100fe180 2022-09-07T21:31:03.123 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.117+0000 7fcc0e7fc700 1 --2- 172.21.15.47:0/3231239498 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcbf8035910 0x7fcbf8037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:03.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.117+0000 7fcc0e7fc700 1 --2- 172.21.15.47:0/3231239498 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcbf8035910 0x7fcbf8037dc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009750 tx=0x7fcc04006cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:03.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.121+0000 7fcbf3fff700 1 -- 172.21.15.47:0/3231239498 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fcc00017440 con 0x7fcc100fe180 2022-09-07T21:31:03.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.342+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3231239498 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc1005f120 con 0x7fcbf8035910 2022-09-07T21:31:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.346+0000 7fcbf3fff700 1 -- 172.21.15.47:0/3231239498 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcc1005f120 con 0x7fcbf8035910 2022-09-07T21:31:03.351 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.348+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3231239498 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcbf8035910 msgr2=0x7fcbf8037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:03.351 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.348+0000 7fcc0ffff700 1 --2- 172.21.15.47:0/3231239498 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcbf8035910 0x7fcbf8037dc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009750 tx=0x7fcc04006cc0 comp rx=0 tx=0).stop 2022-09-07T21:31:03.352 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.348+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3231239498 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc100fe180 msgr2=0x7fcc101947a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:03.352 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.348+0000 7fcc0ffff700 1 --2- 172.21.15.47:0/3231239498 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc100fe180 0x7fcc101947a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcc000059c0 tx=0x7fcc0000f460 comp rx=0 tx=0).stop 2022-09-07T21:31:03.352 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.349+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3231239498 shutdown_connections 2022-09-07T21:31:03.352 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.349+0000 7fcc0ffff700 1 --2- 172.21.15.47:0/3231239498 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcbf8035910 0x7fcbf8037dc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:03.353 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.349+0000 7fcc0ffff700 1 --2- 172.21.15.47:0/3231239498 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc100fe180 0x7fcc101947a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:03.353 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.349+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3231239498 >> 172.21.15.47:0/3231239498 conn(0x7fcc100f7bc0 msgr2=0x7fcc100f9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:03.353 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.349+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3231239498 shutdown_connections 2022-09-07T21:31:03.353 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:03.349+0000 7fcc0ffff700 1 -- 172.21.15.47:0/3231239498 wait complete. 2022-09-07T21:31:03.926 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi055 2022-09-07T21:31:03.927 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2022-09-07T21:31:03.927 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/etc/ceph/ceph.conf 2022-09-07T21:31:03.950 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2022-09-07T21:31:03.951 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:31:04.012 INFO:tasks.cephadm:Adding host smithi055 to orchestrator... 2022-09-07T21:31:04.013 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch host add smithi055 2022-09-07T21:31:04.624 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:04 smithi047 ceph-mon[90087]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:31:04.625 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:04.626 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:04.626 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:04.627 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:04.627 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:04 smithi047 ceph-mon[90087]: Updating smithi047:/etc/ceph/ceph.conf 2022-09-07T21:31:04.911 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:31:05.509 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:05 smithi047 ceph-mon[90087]: Updating smithi047:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:31:06.386 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:06 smithi047 ceph-mon[90087]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:31:06.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.600+0000 7f6927d6c700 1 -- 172.21.15.47:0/3427909837 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6920105a60 msgr2=0x7f6920105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:06.603 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.600+0000 7f6927d6c700 1 --2- 172.21.15.47:0/3427909837 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6920105a60 0x7f6920105e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f691c009790 tx=0x7f691c009aa0 comp rx=0 tx=0).stop 2022-09-07T21:31:06.603 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.600+0000 7f6927d6c700 1 -- 172.21.15.47:0/3427909837 shutdown_connections 2022-09-07T21:31:06.603 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.600+0000 7f6927d6c700 1 --2- 172.21.15.47:0/3427909837 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6920105a60 0x7f6920105e30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:06.604 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.600+0000 7f6927d6c700 1 -- 172.21.15.47:0/3427909837 >> 172.21.15.47:0/3427909837 conn(0x7f69200fb6b0 msgr2=0x7f69200fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:06.604 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.600+0000 7f6927d6c700 1 -- 172.21.15.47:0/3427909837 shutdown_connections 2022-09-07T21:31:06.604 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.600+0000 7f6927d6c700 1 -- 172.21.15.47:0/3427909837 wait complete. 2022-09-07T21:31:06.605 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.601+0000 7f6927d6c700 1 Processor -- start 2022-09-07T21:31:06.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.601+0000 7f6927d6c700 1 -- start start 2022-09-07T21:31:06.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.602+0000 7f6927d6c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6920105a60 0x7f692019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:06.607 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.602+0000 7f6927d6c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692019af20 con 0x7f6920105a60 2022-09-07T21:31:06.607 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.602+0000 7f6925b08700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6920105a60 0x7f692019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:06.607 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.602+0000 7f6925b08700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6920105a60 0x7f692019a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46686/0 (socket says 172.21.15.47:46686) 2022-09-07T21:31:06.608 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.602+0000 7f6925b08700 1 -- 172.21.15.47:0/3286187480 learned_addr learned my addr 172.21.15.47:0/3286187480 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:31:06.608 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.603+0000 7f6925b08700 1 -- 172.21.15.47:0/3286187480 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c009470 con 0x7f6920105a60 2022-09-07T21:31:06.608 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.603+0000 7f6925b08700 1 --2- 172.21.15.47:0/3286187480 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6920105a60 0x7f692019a9e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f691c005dd0 tx=0x7f691c005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:06.608 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.603+0000 7f6916ffd700 1 -- 172.21.15.47:0/3286187480 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f691c01d070 con 0x7f6920105a60 2022-09-07T21:31:06.608 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.603+0000 7f6916ffd700 1 -- 172.21.15.47:0/3286187480 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f691c00fcc0 con 0x7f6920105a60 2022-09-07T21:31:06.609 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.603+0000 7f6927d6c700 1 -- 172.21.15.47:0/3286187480 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692019b120 con 0x7f6920105a60 2022-09-07T21:31:06.609 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.603+0000 7f6916ffd700 1 -- 172.21.15.47:0/3286187480 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f691c017e20 con 0x7f6920105a60 2022-09-07T21:31:06.611 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.604+0000 7f6927d6c700 1 -- 172.21.15.47:0/3286187480 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692019b550 con 0x7f6920105a60 2022-09-07T21:31:06.611 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.605+0000 7f6916ffd700 1 -- 172.21.15.47:0/3286187480 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f691c028020 con 0x7f6920105a60 2022-09-07T21:31:06.611 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.605+0000 7f6927d6c700 1 -- 172.21.15.47:0/3286187480 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6920060310 con 0x7f6920105a60 2022-09-07T21:31:06.612 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.605+0000 7f6916ffd700 1 --2- 172.21.15.47:0/3286187480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f690c0358d0 0x7f690c037d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:06.612 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.605+0000 7f6916ffd700 1 -- 172.21.15.47:0/3286187480 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f691c048ae0 con 0x7f6920105a60 2022-09-07T21:31:06.613 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.606+0000 7f6925307700 1 --2- 172.21.15.47:0/3286187480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f690c0358d0 0x7f690c037d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:06.613 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.606+0000 7f6925307700 1 --2- 172.21.15.47:0/3286187480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f690c0358d0 0x7f690c037d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6910009750 tx=0x7f6910006cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:06.614 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.611+0000 7f6916ffd700 1 -- 172.21.15.47:0/3286187480 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f691c021830 con 0x7f6920105a60 2022-09-07T21:31:06.879 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:06.877+0000 7f6927d6c700 1 -- 172.21.15.47:0/3286187480 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi055", "target": ["mon-mgr", ""]}) v1 -- 0x7f6920103ee0 con 0x7f690c0358d0 2022-09-07T21:31:07.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:07 smithi047 ceph-mon[90087]: Updating smithi047:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:31:07.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:07 smithi047 ceph-mon[90087]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:07.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:07 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:07.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:07 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:07.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:07 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:08.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:08 smithi047 ceph-mon[90087]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi055", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:31:08.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:08 smithi047 ceph-mon[90087]: Deploying cephadm binary to smithi055 2022-09-07T21:31:08.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:08.863+0000 7f6916ffd700 1 -- 172.21.15.47:0/3286187480 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f6920103ee0 con 0x7f690c0358d0 2022-09-07T21:31:08.867 INFO:teuthology.orchestra.run.smithi047.stdout:Added host 'smithi055' with addr '172.21.15.55' 2022-09-07T21:31:08.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:08.865+0000 7f6927d6c700 1 -- 172.21.15.47:0/3286187480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f690c0358d0 msgr2=0x7f690c037d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:08.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:08.865+0000 7f6927d6c700 1 --2- 172.21.15.47:0/3286187480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f690c0358d0 0x7f690c037d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6910009750 tx=0x7f6910006cc0 comp rx=0 tx=0).stop 2022-09-07T21:31:08.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:08.866+0000 7f6927d6c700 1 -- 172.21.15.47:0/3286187480 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6920105a60 msgr2=0x7f692019a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:08.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:08.866+0000 7f6927d6c700 1 --2- 172.21.15.47:0/3286187480 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6920105a60 0x7f692019a9e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f691c005dd0 tx=0x7f691c005390 comp rx=0 tx=0).stop 2022-09-07T21:31:08.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:08.866+0000 7f6927d6c700 1 -- 172.21.15.47:0/3286187480 shutdown_connections 2022-09-07T21:31:08.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:08.866+0000 7f6927d6c700 1 --2- 172.21.15.47:0/3286187480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f690c0358d0 0x7f690c037d80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:08.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:08.866+0000 7f6927d6c700 1 --2- 172.21.15.47:0/3286187480 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6920105a60 0x7f692019a9e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:08.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:08.866+0000 7f6927d6c700 1 -- 172.21.15.47:0/3286187480 >> 172.21.15.47:0/3286187480 conn(0x7f69200fb6b0 msgr2=0x7f6920101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:08.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:08.866+0000 7f6927d6c700 1 -- 172.21.15.47:0/3286187480 shutdown_connections 2022-09-07T21:31:08.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:08.866+0000 7f6927d6c700 1 -- 172.21.15.47:0/3286187480 wait complete. 2022-09-07T21:31:09.396 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch host ls --format=json 2022-09-07T21:31:09.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:09 smithi047 ceph-mon[90087]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:09.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:09 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:09.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:09 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:09.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:09 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:09.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:09 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:09.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:09 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:10.276 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:31:10.632 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:10 smithi047 ceph-mon[90087]: Added host smithi055 2022-09-07T21:31:10.632 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:10 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:11.628 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:11 smithi047 ceph-mon[90087]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:11.629 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:11 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:11.933 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.930+0000 7f950b508700 1 -- 172.21.15.47:0/1075971929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95041036a0 msgr2=0x7f9504103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:11.934 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.930+0000 7f950b508700 1 --2- 172.21.15.47:0/1075971929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95041036a0 0x7f9504103a70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9500009790 tx=0x7f9500009aa0 comp rx=0 tx=0).stop 2022-09-07T21:31:11.934 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.930+0000 7f950b508700 1 -- 172.21.15.47:0/1075971929 shutdown_connections 2022-09-07T21:31:11.934 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.930+0000 7f950b508700 1 --2- 172.21.15.47:0/1075971929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95041036a0 0x7f9504103a70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:11.934 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.930+0000 7f950b508700 1 -- 172.21.15.47:0/1075971929 >> 172.21.15.47:0/1075971929 conn(0x7f95040f9260 msgr2=0x7f95040fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:11.935 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.930+0000 7f950b508700 1 -- 172.21.15.47:0/1075971929 shutdown_connections 2022-09-07T21:31:11.935 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.930+0000 7f950b508700 1 -- 172.21.15.47:0/1075971929 wait complete. 2022-09-07T21:31:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.931+0000 7f950b508700 1 Processor -- start 2022-09-07T21:31:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.931+0000 7f950b508700 1 -- start start 2022-09-07T21:31:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.932+0000 7f950b508700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95041036a0 0x7f9504198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.932+0000 7f950b508700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9504198bc0 con 0x7f95041036a0 2022-09-07T21:31:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.932+0000 7f950a506700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95041036a0 0x7f9504198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.932+0000 7f950a506700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95041036a0 0x7f9504198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55574/0 (socket says 172.21.15.47:55574) 2022-09-07T21:31:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.932+0000 7f950a506700 1 -- 172.21.15.47:0/3331229211 learned_addr learned my addr 172.21.15.47:0/3331229211 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:31:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.932+0000 7f950a506700 1 -- 172.21.15.47:0/3331229211 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9500009470 con 0x7f95041036a0 2022-09-07T21:31:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.933+0000 7f950a506700 1 --2- 172.21.15.47:0/3331229211 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95041036a0 0x7f9504198680 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f95000059c0 tx=0x7f9500005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.933+0000 7f94fb7fe700 1 -- 172.21.15.47:0/3331229211 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f950001d070 con 0x7f95041036a0 2022-09-07T21:31:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.933+0000 7f94fb7fe700 1 -- 172.21.15.47:0/3331229211 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f950000fd80 con 0x7f95041036a0 2022-09-07T21:31:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.933+0000 7f94fb7fe700 1 -- 172.21.15.47:0/3331229211 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9500017df0 con 0x7f95041036a0 2022-09-07T21:31:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.933+0000 7f950b508700 1 -- 172.21.15.47:0/3331229211 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9504198d60 con 0x7f95041036a0 2022-09-07T21:31:11.940 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.933+0000 7f950b508700 1 -- 172.21.15.47:0/3331229211 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9504199190 con 0x7f95041036a0 2022-09-07T21:31:11.941 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.934+0000 7f94fb7fe700 1 -- 172.21.15.47:0/3331229211 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f9500028020 con 0x7f95041036a0 2022-09-07T21:31:11.941 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.934+0000 7f950b508700 1 -- 172.21.15.47:0/3331229211 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9504060310 con 0x7f95041036a0 2022-09-07T21:31:11.941 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.935+0000 7f94fb7fe700 1 --2- 172.21.15.47:0/3331229211 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f94f0035910 0x7f94f0037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:11.942 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.935+0000 7f94fb7fe700 1 -- 172.21.15.47:0/3331229211 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9500048a70 con 0x7f95041036a0 2022-09-07T21:31:11.942 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.935+0000 7f9509d05700 1 --2- 172.21.15.47:0/3331229211 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f94f0035910 0x7f94f0037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:11.942 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.936+0000 7f9509d05700 1 --2- 172.21.15.47:0/3331229211 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f94f0035910 0x7f94f0037dc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009750 tx=0x7f94f4006cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:11.943 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:11.940+0000 7f94fb7fe700 1 -- 172.21.15.47:0/3331229211 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f950000f8e0 con 0x7f95041036a0 2022-09-07T21:31:12.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:12.167+0000 7f950b508700 1 -- 172.21.15.47:0/3331229211 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95041028f0 con 0x7f94f0035910 2022-09-07T21:31:12.171 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:12.168+0000 7f94fb7fe700 1 -- 172.21.15.47:0/3331229211 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f95041028f0 con 0x7f94f0035910 2022-09-07T21:31:12.172 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:31:12.172 INFO:teuthology.orchestra.run.smithi047.stdout:[{"addr": "172.21.15.47", "hostname": "smithi047", "labels": [], "status": ""}, {"addr": "172.21.15.55", "hostname": "smithi055", "labels": [], "status": ""}] 2022-09-07T21:31:12.174 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:12.171+0000 7f950b508700 1 -- 172.21.15.47:0/3331229211 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f94f0035910 msgr2=0x7f94f0037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:12.175 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:12.171+0000 7f950b508700 1 --2- 172.21.15.47:0/3331229211 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f94f0035910 0x7f94f0037dc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009750 tx=0x7f94f4006cc0 comp rx=0 tx=0).stop 2022-09-07T21:31:12.175 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:12.171+0000 7f950b508700 1 -- 172.21.15.47:0/3331229211 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95041036a0 msgr2=0x7f9504198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:12.175 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:12.171+0000 7f950b508700 1 --2- 172.21.15.47:0/3331229211 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95041036a0 0x7f9504198680 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f95000059c0 tx=0x7f9500005390 comp rx=0 tx=0).stop 2022-09-07T21:31:12.175 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:12.171+0000 7f950b508700 1 -- 172.21.15.47:0/3331229211 shutdown_connections 2022-09-07T21:31:12.175 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:12.171+0000 7f950b508700 1 --2- 172.21.15.47:0/3331229211 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f94f0035910 0x7f94f0037dc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:12.176 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:12.171+0000 7f950b508700 1 --2- 172.21.15.47:0/3331229211 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95041036a0 0x7f9504198680 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:12.176 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:12.171+0000 7f950b508700 1 -- 172.21.15.47:0/3331229211 >> 172.21.15.47:0/3331229211 conn(0x7f95040f9260 msgr2=0x7f95040fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:12.176 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:12.171+0000 7f950b508700 1 -- 172.21.15.47:0/3331229211 shutdown_connections 2022-09-07T21:31:12.176 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:12.171+0000 7f950b508700 1 -- 172.21.15.47:0/3331229211 wait complete. 2022-09-07T21:31:12.749 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi059 2022-09-07T21:31:12.750 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-09-07T21:31:12.751 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/etc/ceph/ceph.conf 2022-09-07T21:31:12.775 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-09-07T21:31:12.776 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:31:12.836 INFO:tasks.cephadm:Adding host smithi059 to orchestrator... 2022-09-07T21:31:12.837 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch host add smithi059 2022-09-07T21:31:13.719 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:31:13.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:13 smithi047 ceph-mon[90087]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:31:13.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:13 smithi047 ceph-mon[90087]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:15.494 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.491+0000 7f25f270b700 1 -- 172.21.15.47:0/3523835329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ec102950 msgr2=0x7f25ec102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:15.495 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.491+0000 7f25f270b700 1 --2- 172.21.15.47:0/3523835329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ec102950 0x7f25ec102d20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f25dc009790 tx=0x7f25dc009aa0 comp rx=0 tx=0).stop 2022-09-07T21:31:15.495 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.491+0000 7f25f270b700 1 -- 172.21.15.47:0/3523835329 shutdown_connections 2022-09-07T21:31:15.496 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.491+0000 7f25f270b700 1 --2- 172.21.15.47:0/3523835329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ec102950 0x7f25ec102d20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:15.496 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.491+0000 7f25f270b700 1 -- 172.21.15.47:0/3523835329 >> 172.21.15.47:0/3523835329 conn(0x7f25ec0727f0 msgr2=0x7f25ec072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:15.497 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.491+0000 7f25f270b700 1 -- 172.21.15.47:0/3523835329 shutdown_connections 2022-09-07T21:31:15.497 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.491+0000 7f25f270b700 1 -- 172.21.15.47:0/3523835329 wait complete. 2022-09-07T21:31:15.499 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.492+0000 7f25f270b700 1 Processor -- start 2022-09-07T21:31:15.499 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.492+0000 7f25f270b700 1 -- start start 2022-09-07T21:31:15.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.493+0000 7f25f270b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ec102950 0x7f25ec198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:15.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.493+0000 7f25f270b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec199000 con 0x7f25ec102950 2022-09-07T21:31:15.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.493+0000 7f25ebfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ec102950 0x7f25ec198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:15.501 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.493+0000 7f25ebfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ec102950 0x7f25ec198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55606/0 (socket says 172.21.15.47:55606) 2022-09-07T21:31:15.501 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.493+0000 7f25ebfff700 1 -- 172.21.15.47:0/1856435091 learned_addr learned my addr 172.21.15.47:0/1856435091 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:31:15.501 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.494+0000 7f25ebfff700 1 -- 172.21.15.47:0/1856435091 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc009470 con 0x7f25ec102950 2022-09-07T21:31:15.502 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.494+0000 7f25ebfff700 1 --2- 172.21.15.47:0/1856435091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ec102950 0x7f25ec198ac0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f25dc006010 tx=0x7f25dc005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:15.502 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.494+0000 7f25e97fa700 1 -- 172.21.15.47:0/1856435091 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25dc01d070 con 0x7f25ec102950 2022-09-07T21:31:15.502 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.494+0000 7f25e97fa700 1 -- 172.21.15.47:0/1856435091 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f25dc00fc40 con 0x7f25ec102950 2022-09-07T21:31:15.502 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.494+0000 7f25f270b700 1 -- 172.21.15.47:0/1856435091 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ec199200 con 0x7f25ec102950 2022-09-07T21:31:15.503 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.494+0000 7f25e97fa700 1 -- 172.21.15.47:0/1856435091 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25dc017d00 con 0x7f25ec102950 2022-09-07T21:31:15.503 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.494+0000 7f25f270b700 1 -- 172.21.15.47:0/1856435091 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ec192aa0 con 0x7f25ec102950 2022-09-07T21:31:15.503 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.495+0000 7f25f270b700 1 -- 172.21.15.47:0/1856435091 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ec060310 con 0x7f25ec102950 2022-09-07T21:31:15.503 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.496+0000 7f25e97fa700 1 -- 172.21.15.47:0/1856435091 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f25dc028030 con 0x7f25ec102950 2022-09-07T21:31:15.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.496+0000 7f25e97fa700 1 --2- 172.21.15.47:0/1856435091 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f25d4035910 0x7f25d4037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:15.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.496+0000 7f25e97fa700 1 -- 172.21.15.47:0/1856435091 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f25dc0487b0 con 0x7f25ec102950 2022-09-07T21:31:15.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.496+0000 7f25eb7fe700 1 --2- 172.21.15.47:0/1856435091 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f25d4035910 0x7f25d4037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:15.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.497+0000 7f25eb7fe700 1 --2- 172.21.15.47:0/1856435091 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f25d4035910 0x7f25d4037dc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f25e0009750 tx=0x7f25e0006cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:15.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.501+0000 7f25e97fa700 1 -- 172.21.15.47:0/1856435091 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f25dc02abf0 con 0x7f25ec102950 2022-09-07T21:31:15.738 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:15.734+0000 7f25f270b700 1 -- 172.21.15.47:0/1856435091 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi059", "target": ["mon-mgr", ""]}) v1 -- 0x7f25ec199630 con 0x7f25d4035910 2022-09-07T21:31:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:15 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:15 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:15 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:15.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:15 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:15.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:15 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi055", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:31:15.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:15 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:15.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:15 smithi047 ceph-mon[90087]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:17.737 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:17.734+0000 7f25e97fa700 1 -- 172.21.15.47:0/1856435091 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f25ec199630 con 0x7f25d4035910 2022-09-07T21:31:17.738 INFO:teuthology.orchestra.run.smithi047.stdout:Added host 'smithi059' with addr '172.21.15.59' 2022-09-07T21:31:17.740 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:17.736+0000 7f25f270b700 1 -- 172.21.15.47:0/1856435091 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f25d4035910 msgr2=0x7f25d4037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:17.740 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:17.736+0000 7f25f270b700 1 --2- 172.21.15.47:0/1856435091 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f25d4035910 0x7f25d4037dc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f25e0009750 tx=0x7f25e0006cc0 comp rx=0 tx=0).stop 2022-09-07T21:31:17.740 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:17.736+0000 7f25f270b700 1 -- 172.21.15.47:0/1856435091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ec102950 msgr2=0x7f25ec198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:17.741 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:17.736+0000 7f25f270b700 1 --2- 172.21.15.47:0/1856435091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ec102950 0x7f25ec198ac0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f25dc006010 tx=0x7f25dc005390 comp rx=0 tx=0).stop 2022-09-07T21:31:17.741 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:17.736+0000 7f25f270b700 1 -- 172.21.15.47:0/1856435091 shutdown_connections 2022-09-07T21:31:17.741 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:17.736+0000 7f25f270b700 1 --2- 172.21.15.47:0/1856435091 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f25d4035910 0x7f25d4037dc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:17.741 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:17.736+0000 7f25f270b700 1 --2- 172.21.15.47:0/1856435091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ec102950 0x7f25ec198ac0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:17.742 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:17.736+0000 7f25f270b700 1 -- 172.21.15.47:0/1856435091 >> 172.21.15.47:0/1856435091 conn(0x7f25ec0727f0 msgr2=0x7f25ec0fb730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:17.742 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:17.736+0000 7f25f270b700 1 -- 172.21.15.47:0/1856435091 shutdown_connections 2022-09-07T21:31:17.743 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:17.737+0000 7f25f270b700 1 -- 172.21.15.47:0/1856435091 wait complete. 2022-09-07T21:31:17.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:17 smithi047 ceph-mon[90087]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi059", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:31:17.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:17 smithi047 ceph-mon[90087]: Deploying cephadm binary to smithi059 2022-09-07T21:31:17.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:17 smithi047 ceph-mon[90087]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:18.294 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch host ls --format=json 2022-09-07T21:31:19.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:19.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:18 smithi047 ceph-mon[90087]: Added host smithi059 2022-09-07T21:31:19.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:19.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:19.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:19.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:18 smithi047 ceph-mon[90087]: Updating smithi055:/etc/ceph/ceph.conf 2022-09-07T21:31:19.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:18 smithi047 ceph-mon[90087]: Updating smithi055:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:31:19.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:19.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:18 smithi047 ceph-mon[90087]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:31:19.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:18 smithi047 ceph-mon[90087]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:19.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:18 smithi047 ceph-mon[90087]: Updating smithi055:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:31:19.183 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:31:20.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:19 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:20.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:19 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:20.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:19 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:20.850 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.846+0000 7f3049ffb700 1 -- 172.21.15.47:0/57287007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3044103860 msgr2=0x7f3044103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:20.850 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.846+0000 7f3049ffb700 1 --2- 172.21.15.47:0/57287007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3044103860 0x7f3044103c30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3034009790 tx=0x7f3034009aa0 comp rx=0 tx=0).stop 2022-09-07T21:31:20.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.846+0000 7f3049ffb700 1 -- 172.21.15.47:0/57287007 shutdown_connections 2022-09-07T21:31:20.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.846+0000 7f3049ffb700 1 --2- 172.21.15.47:0/57287007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3044103860 0x7f3044103c30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:20.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.846+0000 7f3049ffb700 1 -- 172.21.15.47:0/57287007 >> 172.21.15.47:0/57287007 conn(0x7f30440f9420 msgr2=0x7f30440fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:20.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.846+0000 7f3049ffb700 1 -- 172.21.15.47:0/57287007 shutdown_connections 2022-09-07T21:31:20.852 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.846+0000 7f3049ffb700 1 -- 172.21.15.47:0/57287007 wait complete. 2022-09-07T21:31:20.853 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.847+0000 7f3049ffb700 1 Processor -- start 2022-09-07T21:31:20.853 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.848+0000 7f3049ffb700 1 -- start start 2022-09-07T21:31:20.854 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.848+0000 7f3049ffb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3044103860 0x7f3044198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:20.854 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.848+0000 7f3049ffb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3044198d80 con 0x7f3044103860 2022-09-07T21:31:20.854 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.848+0000 7f30437fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3044103860 0x7f3044198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:20.854 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.848+0000 7f30437fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3044103860 0x7f3044198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40566/0 (socket says 172.21.15.47:40566) 2022-09-07T21:31:20.855 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.848+0000 7f30437fe700 1 -- 172.21.15.47:0/3395979588 learned_addr learned my addr 172.21.15.47:0/3395979588 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:31:20.855 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.849+0000 7f30437fe700 1 -- 172.21.15.47:0/3395979588 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3034009470 con 0x7f3044103860 2022-09-07T21:31:20.855 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.849+0000 7f30437fe700 1 --2- 172.21.15.47:0/3395979588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3044103860 0x7f3044198840 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f30340059c0 tx=0x7f3034005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:20.855 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.849+0000 7f3040ff9700 1 -- 172.21.15.47:0/3395979588 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f303401d070 con 0x7f3044103860 2022-09-07T21:31:20.856 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.849+0000 7f3040ff9700 1 -- 172.21.15.47:0/3395979588 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f303400fd80 con 0x7f3044103860 2022-09-07T21:31:20.856 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.849+0000 7f3040ff9700 1 -- 172.21.15.47:0/3395979588 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3034017df0 con 0x7f3044103860 2022-09-07T21:31:20.856 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.849+0000 7f3049ffb700 1 -- 172.21.15.47:0/3395979588 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3044198f80 con 0x7f3044103860 2022-09-07T21:31:20.856 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.849+0000 7f3049ffb700 1 -- 172.21.15.47:0/3395979588 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3044192820 con 0x7f3044103860 2022-09-07T21:31:20.857 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.850+0000 7f3040ff9700 1 -- 172.21.15.47:0/3395979588 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f3034028020 con 0x7f3044103860 2022-09-07T21:31:20.858 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.850+0000 7f3049ffb700 1 -- 172.21.15.47:0/3395979588 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3044060310 con 0x7f3044103860 2022-09-07T21:31:20.859 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.851+0000 7f3040ff9700 1 --2- 172.21.15.47:0/3395979588 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3030035910 0x7f3030037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:20.859 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.851+0000 7f3040ff9700 1 -- 172.21.15.47:0/3395979588 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3034048a70 con 0x7f3044103860 2022-09-07T21:31:20.859 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.851+0000 7f3042ffd700 1 --2- 172.21.15.47:0/3395979588 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3030035910 0x7f3030037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:20.859 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.852+0000 7f3042ffd700 1 --2- 172.21.15.47:0/3395979588 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3030035910 0x7f3030037dc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3038009750 tx=0x7f3038006cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:20.860 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:20.856+0000 7f3040ff9700 1 -- 172.21.15.47:0/3395979588 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f303400f8e0 con 0x7f3044103860 2022-09-07T21:31:21.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:21.083+0000 7f3049ffb700 1 -- 172.21.15.47:0/3395979588 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3044193ec0 con 0x7f3030035910 2022-09-07T21:31:21.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:21.085+0000 7f3040ff9700 1 -- 172.21.15.47:0/3395979588 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+238 (secure 0 0 0) 0x7f3044193ec0 con 0x7f3030035910 2022-09-07T21:31:21.089 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:31:21.090 INFO:teuthology.orchestra.run.smithi047.stdout:[{"addr": "172.21.15.47", "hostname": "smithi047", "labels": [], "status": ""}, {"addr": "172.21.15.55", "hostname": "smithi055", "labels": [], "status": ""}, {"addr": "172.21.15.59", "hostname": "smithi059", "labels": [], "status": ""}] 2022-09-07T21:31:21.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:21.087+0000 7f3049ffb700 1 -- 172.21.15.47:0/3395979588 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3030035910 msgr2=0x7f3030037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:21.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:21.087+0000 7f3049ffb700 1 --2- 172.21.15.47:0/3395979588 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3030035910 0x7f3030037dc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3038009750 tx=0x7f3038006cc0 comp rx=0 tx=0).stop 2022-09-07T21:31:21.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:21.087+0000 7f3049ffb700 1 -- 172.21.15.47:0/3395979588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3044103860 msgr2=0x7f3044198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:21.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:21.087+0000 7f3049ffb700 1 --2- 172.21.15.47:0/3395979588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3044103860 0x7f3044198840 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f30340059c0 tx=0x7f3034005390 comp rx=0 tx=0).stop 2022-09-07T21:31:21.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:21.087+0000 7f3049ffb700 1 -- 172.21.15.47:0/3395979588 shutdown_connections 2022-09-07T21:31:21.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:21.087+0000 7f3049ffb700 1 --2- 172.21.15.47:0/3395979588 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3030035910 0x7f3030037dc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:21.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:21.087+0000 7f3049ffb700 1 --2- 172.21.15.47:0/3395979588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3044103860 0x7f3044198840 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:21.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:21.087+0000 7f3049ffb700 1 -- 172.21.15.47:0/3395979588 >> 172.21.15.47:0/3395979588 conn(0x7f30440f9420 msgr2=0x7f30440fb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:21.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:21.088+0000 7f3049ffb700 1 -- 172.21.15.47:0/3395979588 shutdown_connections 2022-09-07T21:31:21.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:21.088+0000 7f3049ffb700 1 -- 172.21.15.47:0/3395979588 wait complete. 2022-09-07T21:31:21.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:21.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:21 smithi047 ceph-mon[90087]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:21.554 INFO:tasks.cephadm:Setting crush tunables to default 2022-09-07T21:31:21.555 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd crush tunables default 2022-09-07T21:31:22.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:22 smithi047 ceph-mon[90087]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:31:22.438 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:31:23.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:23 smithi047 ceph-mon[90087]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:23.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:23.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:23.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:23.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:23.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:31:23.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:24.159 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.156+0000 7fe8fcf12700 1 -- 172.21.15.47:0/3267528796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f81036a0 msgr2=0x7fe8f8103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:24.160 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.156+0000 7fe8fcf12700 1 --2- 172.21.15.47:0/3267528796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f81036a0 0x7fe8f8103a70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8009790 tx=0x7fe8e8009aa0 comp rx=0 tx=0).stop 2022-09-07T21:31:24.160 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.156+0000 7fe8fcf12700 1 -- 172.21.15.47:0/3267528796 shutdown_connections 2022-09-07T21:31:24.160 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.156+0000 7fe8fcf12700 1 --2- 172.21.15.47:0/3267528796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f81036a0 0x7fe8f8103a70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:24.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.156+0000 7fe8fcf12700 1 -- 172.21.15.47:0/3267528796 >> 172.21.15.47:0/3267528796 conn(0x7fe8f80f9260 msgr2=0x7fe8f80fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:24.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.156+0000 7fe8fcf12700 1 -- 172.21.15.47:0/3267528796 shutdown_connections 2022-09-07T21:31:24.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.156+0000 7fe8fcf12700 1 -- 172.21.15.47:0/3267528796 wait complete. 2022-09-07T21:31:24.163 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.158+0000 7fe8fcf12700 1 Processor -- start 2022-09-07T21:31:24.163 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.158+0000 7fe8fcf12700 1 -- start start 2022-09-07T21:31:24.164 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.159+0000 7fe8fcf12700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f81036a0 0x7fe8f81986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:24.164 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.159+0000 7fe8fcf12700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f8198be0 con 0x7fe8f81036a0 2022-09-07T21:31:24.164 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.159+0000 7fe8f77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f81036a0 0x7fe8f81986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:24.164 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.159+0000 7fe8f77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f81036a0 0x7fe8f81986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40588/0 (socket says 172.21.15.47:40588) 2022-09-07T21:31:24.165 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.159+0000 7fe8f77fe700 1 -- 172.21.15.47:0/272893152 learned_addr learned my addr 172.21.15.47:0/272893152 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:31:24.165 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.159+0000 7fe8f77fe700 1 -- 172.21.15.47:0/272893152 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e8009470 con 0x7fe8f81036a0 2022-09-07T21:31:24.165 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.160+0000 7fe8f77fe700 1 --2- 172.21.15.47:0/272893152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f81036a0 0x7fe8f81986a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe8e80059c0 tx=0x7fe8e8005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:24.166 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.160+0000 7fe8f57fa700 1 -- 172.21.15.47:0/272893152 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8e801d070 con 0x7fe8f81036a0 2022-09-07T21:31:24.166 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.160+0000 7fe8f57fa700 1 -- 172.21.15.47:0/272893152 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe8e800fd80 con 0x7fe8f81036a0 2022-09-07T21:31:24.166 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.160+0000 7fe8fcf12700 1 -- 172.21.15.47:0/272893152 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f8198d80 con 0x7fe8f81036a0 2022-09-07T21:31:24.166 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.160+0000 7fe8f57fa700 1 -- 172.21.15.47:0/272893152 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8e8017df0 con 0x7fe8f81036a0 2022-09-07T21:31:24.167 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.160+0000 7fe8fcf12700 1 -- 172.21.15.47:0/272893152 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f81991b0 con 0x7fe8f81036a0 2022-09-07T21:31:24.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.161+0000 7fe8fcf12700 1 -- 172.21.15.47:0/272893152 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f8060310 con 0x7fe8f81036a0 2022-09-07T21:31:24.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.161+0000 7fe8f57fa700 1 -- 172.21.15.47:0/272893152 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7fe8e8028020 con 0x7fe8f81036a0 2022-09-07T21:31:24.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.162+0000 7fe8f57fa700 1 --2- 172.21.15.47:0/272893152 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe8e40358b0 0x7fe8e4037d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:24.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.162+0000 7fe8f57fa700 1 -- 172.21.15.47:0/272893152 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe8e8048b30 con 0x7fe8f81036a0 2022-09-07T21:31:24.169 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.162+0000 7fe8effff700 1 --2- 172.21.15.47:0/272893152 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe8e40358b0 0x7fe8e4037d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:24.169 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.163+0000 7fe8effff700 1 --2- 172.21.15.47:0/272893152 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe8e40358b0 0x7fe8e4037d60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009750 tx=0x7fe8e0006cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:24.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.167+0000 7fe8f57fa700 1 -- 172.21.15.47:0/272893152 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fe8e800f8e0 con 0x7fe8f81036a0 2022-09-07T21:31:24.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.405+0000 7fe8fcf12700 1 -- 172.21.15.47:0/272893152 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe8f81078e0 con 0x7fe8f81036a0 2022-09-07T21:31:24.462 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.458+0000 7fe8f57fa700 1 -- 172.21.15.47:0/272893152 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe8e8048e60 con 0x7fe8f81036a0 2022-09-07T21:31:24.465 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.461+0000 7fe8fcf12700 1 -- 172.21.15.47:0/272893152 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe8e40358b0 msgr2=0x7fe8e4037d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:24.465 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.461+0000 7fe8fcf12700 1 --2- 172.21.15.47:0/272893152 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe8e40358b0 0x7fe8e4037d60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009750 tx=0x7fe8e0006cc0 comp rx=0 tx=0).stop 2022-09-07T21:31:24.465 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.461+0000 7fe8fcf12700 1 -- 172.21.15.47:0/272893152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f81036a0 msgr2=0x7fe8f81986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:24.466 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.461+0000 7fe8fcf12700 1 --2- 172.21.15.47:0/272893152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f81036a0 0x7fe8f81986a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe8e80059c0 tx=0x7fe8e8005390 comp rx=0 tx=0).stop 2022-09-07T21:31:24.466 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.461+0000 7fe8fcf12700 1 -- 172.21.15.47:0/272893152 shutdown_connections 2022-09-07T21:31:24.466 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.461+0000 7fe8fcf12700 1 --2- 172.21.15.47:0/272893152 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe8e40358b0 0x7fe8e4037d60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:24.466 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.461+0000 7fe8fcf12700 1 --2- 172.21.15.47:0/272893152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8f81036a0 0x7fe8f81986a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:24.467 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.461+0000 7fe8fcf12700 1 -- 172.21.15.47:0/272893152 >> 172.21.15.47:0/272893152 conn(0x7fe8f80f9260 msgr2=0x7fe8f80fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:24.467 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.462+0000 7fe8fcf12700 1 -- 172.21.15.47:0/272893152 shutdown_connections 2022-09-07T21:31:24.467 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:24.462+0000 7fe8fcf12700 1 -- 172.21.15.47:0/272893152 wait complete. 2022-09-07T21:31:24.468 INFO:teuthology.orchestra.run.smithi047.stderr:adjusted tunables profile to default 2022-09-07T21:31:24.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:24 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/272893152' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-09-07T21:31:25.013 INFO:tasks.cephadm:Adding mon.a on smithi047 2022-09-07T21:31:25.014 INFO:tasks.cephadm:Adding mon.b on smithi055 2022-09-07T21:31:25.014 INFO:tasks.cephadm:Adding mon.c on smithi059 2022-09-07T21:31:25.015 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch apply mon '3;smithi047:172.21.15.47=a;smithi055:172.21.15.55=b;smithi059:172.21.15.59=c' 2022-09-07T21:31:25.649 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /etc/ceph/ceph.conf 2022-09-07T21:31:25.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:25 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/272893152' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-09-07T21:31:25.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:25 smithi047 ceph-mon[90087]: osdmap e4: 0 total, 0 up, 0 in 2022-09-07T21:31:25.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:25 smithi047 ceph-mon[90087]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:25.936 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /etc/ceph/ceph.conf 2022-09-07T21:31:27.140 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.132+0000 7fe613951700 1 -- 172.21.15.59:0/396403107 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe60c0fd840 msgr2=0x7fe60c0fdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.132+0000 7fe613951700 1 --2- 172.21.15.59:0/396403107 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe60c0fd840 0x7fe60c0fdc50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe608009790 tx=0x7fe608009aa0 comp rx=0 tx=0).stop 2022-09-07T21:31:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.132+0000 7fe613951700 1 -- 172.21.15.59:0/396403107 shutdown_connections 2022-09-07T21:31:27.142 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.132+0000 7fe613951700 1 --2- 172.21.15.59:0/396403107 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe60c0fd840 0x7fe60c0fdc50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:27.142 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.132+0000 7fe613951700 1 -- 172.21.15.59:0/396403107 >> 172.21.15.59:0/396403107 conn(0x7fe60c0f8e70 msgr2=0x7fe60c0fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:27.143 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.132+0000 7fe613951700 1 -- 172.21.15.59:0/396403107 shutdown_connections 2022-09-07T21:31:27.143 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.132+0000 7fe613951700 1 -- 172.21.15.59:0/396403107 wait complete. 2022-09-07T21:31:27.144 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.133+0000 7fe613951700 1 Processor -- start 2022-09-07T21:31:27.144 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.133+0000 7fe613951700 1 -- start start 2022-09-07T21:31:27.145 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.133+0000 7fe613951700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe60c0fd840 0x7fe60c1927e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:27.145 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.133+0000 7fe613951700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c192d20 con 0x7fe60c0fd840 2022-09-07T21:31:27.145 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.133+0000 7fe6116ed700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe60c0fd840 0x7fe60c1927e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:27.146 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.133+0000 7fe6116ed700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe60c0fd840 0x7fe60c1927e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.59:52846/0 (socket says 172.21.15.59:52846) 2022-09-07T21:31:27.146 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.133+0000 7fe6116ed700 1 -- 172.21.15.59:0/1300989889 learned_addr learned my addr 172.21.15.59:0/1300989889 (peer_addr_for_me v2:172.21.15.59:0/0) 2022-09-07T21:31:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.134+0000 7fe6116ed700 1 -- 172.21.15.59:0/1300989889 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe608009470 con 0x7fe60c0fd840 2022-09-07T21:31:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.134+0000 7fe6116ed700 1 --2- 172.21.15.59:0/1300989889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe60c0fd840 0x7fe60c1927e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe608005350 tx=0x7fe608005430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.134+0000 7fe6027fc700 1 -- 172.21.15.59:0/1300989889 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe60801d070 con 0x7fe60c0fd840 2022-09-07T21:31:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.134+0000 7fe6027fc700 1 -- 172.21.15.59:0/1300989889 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe60800fe30 con 0x7fe60c0fd840 2022-09-07T21:31:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.134+0000 7fe613951700 1 -- 172.21.15.59:0/1300989889 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60c192f20 con 0x7fe60c0fd840 2022-09-07T21:31:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.134+0000 7fe6027fc700 1 -- 172.21.15.59:0/1300989889 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe60800b460 con 0x7fe60c0fd840 2022-09-07T21:31:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.134+0000 7fe613951700 1 -- 172.21.15.59:0/1300989889 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60c1933d0 con 0x7fe60c0fd840 2022-09-07T21:31:27.150 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.135+0000 7fe613951700 1 -- 172.21.15.59:0/1300989889 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60c103510 con 0x7fe60c0fd840 2022-09-07T21:31:27.150 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.135+0000 7fe6027fc700 1 -- 172.21.15.59:0/1300989889 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7fe608028030 con 0x7fe60c0fd840 2022-09-07T21:31:27.151 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.135+0000 7fe6027fc700 1 --2- 172.21.15.59:0/1300989889 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe5f8035910 0x7fe5f8037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:27.151 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.136+0000 7fe6027fc700 1 -- 172.21.15.59:0/1300989889 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe608048830 con 0x7fe60c0fd840 2022-09-07T21:31:27.151 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.136+0000 7fe610eec700 1 --2- 172.21.15.59:0/1300989889 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe5f8035910 0x7fe5f8037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:27.152 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.136+0000 7fe610eec700 1 --2- 172.21.15.59:0/1300989889 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe5f8035910 0x7fe5f8037dc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009750 tx=0x7fe5fc006cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:27.153 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.141+0000 7fe6027fc700 1 -- 172.21.15.59:0/1300989889 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fe608029540 con 0x7fe60c0fd840 2022-09-07T21:31:27.352 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.349+0000 7fe613951700 1 -- 172.21.15.59:0/1300989889 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi047:172.21.15.47=a;smithi055:172.21.15.55=b;smithi059:172.21.15.59=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fe60c05fc50 con 0x7fe5f8035910 2022-09-07T21:31:27.359 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.356+0000 7fe6027fc700 1 -- 172.21.15.59:0/1300989889 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe60c05fc50 con 0x7fe5f8035910 2022-09-07T21:31:27.359 INFO:teuthology.orchestra.run.smithi059.stdout:Scheduled mon update... 2022-09-07T21:31:27.361 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.358+0000 7fe613951700 1 -- 172.21.15.59:0/1300989889 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe5f8035910 msgr2=0x7fe5f8037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:27.361 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.358+0000 7fe613951700 1 --2- 172.21.15.59:0/1300989889 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe5f8035910 0x7fe5f8037dc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009750 tx=0x7fe5fc006cc0 comp rx=0 tx=0).stop 2022-09-07T21:31:27.362 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.358+0000 7fe613951700 1 -- 172.21.15.59:0/1300989889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe60c0fd840 msgr2=0x7fe60c1927e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:27.362 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.358+0000 7fe613951700 1 --2- 172.21.15.59:0/1300989889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe60c0fd840 0x7fe60c1927e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe608005350 tx=0x7fe608005430 comp rx=0 tx=0).stop 2022-09-07T21:31:27.362 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.358+0000 7fe613951700 1 -- 172.21.15.59:0/1300989889 shutdown_connections 2022-09-07T21:31:27.362 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.358+0000 7fe613951700 1 --2- 172.21.15.59:0/1300989889 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe5f8035910 0x7fe5f8037dc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:27.363 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.358+0000 7fe613951700 1 --2- 172.21.15.59:0/1300989889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe60c0fd840 0x7fe60c1927e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:27.363 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.358+0000 7fe613951700 1 -- 172.21.15.59:0/1300989889 >> 172.21.15.59:0/1300989889 conn(0x7fe60c0f8e70 msgr2=0x7fe60c0f9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:27.363 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.358+0000 7fe613951700 1 -- 172.21.15.59:0/1300989889 shutdown_connections 2022-09-07T21:31:27.363 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:27.358+0000 7fe613951700 1 -- 172.21.15.59:0/1300989889 wait complete. 2022-09-07T21:31:27.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:27 smithi047 ceph-mon[90087]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:27.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:27.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:27.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:27.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:27.992 DEBUG:teuthology.orchestra.run.smithi055:mon.b> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mon.b.service 2022-09-07T21:31:27.995 DEBUG:teuthology.orchestra.run.smithi059:mon.c> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mon.c.service 2022-09-07T21:31:27.998 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-09-07T21:31:27.998 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph mon dump -f json 2022-09-07T21:31:28.026 INFO:journalctl@ceph.mon.c.smithi059.stdout:-- Logs begin at Wed 2022-09-07 21:07:37 UTC. -- 2022-09-07T21:31:28.029 INFO:journalctl@ceph.mon.b.smithi055.stdout:-- Logs begin at Wed 2022-09-07 21:07:35 UTC. -- 2022-09-07T21:31:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:31:28.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:28 smithi047 ceph-mon[90087]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi047:172.21.15.47=a;smithi055:172.21.15.55=b;smithi059:172.21.15.59=c", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:31:28.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:28 smithi047 ceph-mon[90087]: Saving service mon spec with placement smithi047:172.21.15.47=a;smithi055:172.21.15.55=b;smithi059:172.21.15.59=c;count:3 2022-09-07T21:31:28.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:28 smithi047 ceph-mon[90087]: Updating smithi059:/etc/ceph/ceph.conf 2022-09-07T21:31:28.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:28 smithi047 ceph-mon[90087]: Updating smithi059:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:31:28.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:28 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:28.909 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:28 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:29.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:29 smithi047 ceph-mon[90087]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:31:29.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:29 smithi047 ceph-mon[90087]: Updating smithi059:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:31:29.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:29 smithi047 ceph-mon[90087]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:29.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:29.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:29.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:29.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:29 smithi047 ceph-mon[90087]: Deploying daemon mon.c on smithi059 2022-09-07T21:31:30.511 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.503+0000 7f2452af5700 1 -- 172.21.15.59:0/2299898807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f244c1029c0 msgr2=0x7f244c102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:30.512 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.503+0000 7f2452af5700 1 --2- 172.21.15.59:0/2299898807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f244c1029c0 0x7f244c102d90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2440009790 tx=0x7f2440009aa0 comp rx=0 tx=0).stop 2022-09-07T21:31:30.512 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.503+0000 7f2452af5700 1 -- 172.21.15.59:0/2299898807 shutdown_connections 2022-09-07T21:31:30.512 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.503+0000 7f2452af5700 1 --2- 172.21.15.59:0/2299898807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f244c1029c0 0x7f244c102d90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:30.513 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.503+0000 7f2452af5700 1 -- 172.21.15.59:0/2299898807 >> 172.21.15.59:0/2299898807 conn(0x7f244c0727f0 msgr2=0x7f244c072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:30.513 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.503+0000 7f2452af5700 1 -- 172.21.15.59:0/2299898807 shutdown_connections 2022-09-07T21:31:30.514 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.503+0000 7f2452af5700 1 -- 172.21.15.59:0/2299898807 wait complete. 2022-09-07T21:31:30.514 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.504+0000 7f2452af5700 1 Processor -- start 2022-09-07T21:31:30.514 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.504+0000 7f2452af5700 1 -- start start 2022-09-07T21:31:30.515 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.505+0000 7f2452af5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f244c1029c0 0x7f244c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:30.515 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.505+0000 7f2452af5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c198f90 con 0x7f244c1029c0 2022-09-07T21:31:30.516 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.505+0000 7f2450891700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f244c1029c0 0x7f244c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:30.516 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.505+0000 7f2450891700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f244c1029c0 0x7f244c198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.59:52852/0 (socket says 172.21.15.59:52852) 2022-09-07T21:31:30.517 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.505+0000 7f2450891700 1 -- 172.21.15.59:0/3892081282 learned_addr learned my addr 172.21.15.59:0/3892081282 (peer_addr_for_me v2:172.21.15.59:0/0) 2022-09-07T21:31:30.517 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.506+0000 7f2450891700 1 -- 172.21.15.59:0/3892081282 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2440009470 con 0x7f244c1029c0 2022-09-07T21:31:30.517 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.506+0000 7f2450891700 1 --2- 172.21.15.59:0/3892081282 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f244c1029c0 0x7f244c198a50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2440006010 tx=0x7f2440005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:30.518 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.506+0000 7f2449ffb700 1 -- 172.21.15.59:0/3892081282 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f244001d070 con 0x7f244c1029c0 2022-09-07T21:31:30.518 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.506+0000 7f2452af5700 1 -- 172.21.15.59:0/3892081282 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f244c199190 con 0x7f244c1029c0 2022-09-07T21:31:30.518 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.506+0000 7f2452af5700 1 -- 172.21.15.59:0/3892081282 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f244c1995c0 con 0x7f244c1029c0 2022-09-07T21:31:30.518 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.506+0000 7f2449ffb700 1 -- 172.21.15.59:0/3892081282 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f244000fcc0 con 0x7f244c1029c0 2022-09-07T21:31:30.519 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.506+0000 7f2449ffb700 1 -- 172.21.15.59:0/3892081282 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2440017d90 con 0x7f244c1029c0 2022-09-07T21:31:30.519 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.507+0000 7f2449ffb700 1 -- 172.21.15.59:0/3892081282 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f2440028030 con 0x7f244c1029c0 2022-09-07T21:31:30.519 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.507+0000 7f2452af5700 1 -- 172.21.15.59:0/3892081282 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f244c060310 con 0x7f244c1029c0 2022-09-07T21:31:30.519 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.507+0000 7f2449ffb700 1 --2- 172.21.15.59:0/3892081282 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f243c0358b0 0x7f243c037d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:30.520 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.507+0000 7f2449ffb700 1 -- 172.21.15.59:0/3892081282 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2440048850 con 0x7f244c1029c0 2022-09-07T21:31:30.520 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.508+0000 7f244bfff700 1 --2- 172.21.15.59:0/3892081282 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f243c0358b0 0x7f243c037d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:30.520 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.508+0000 7f244bfff700 1 --2- 172.21.15.59:0/3892081282 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f243c0358b0 0x7f243c037d60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2438009750 tx=0x7f2438006cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:30.521 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.513+0000 7f2449ffb700 1 -- 172.21.15.59:0/3892081282 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f2440015740 con 0x7f244c1029c0 2022-09-07T21:31:30.820 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.817+0000 7f2452af5700 1 -- 172.21.15.59:0/3892081282 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f244c068530 con 0x7f244c1029c0 2022-09-07T21:31:30.821 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.818+0000 7f2449ffb700 1 -- 172.21.15.59:0/3892081282 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+710 (secure 0 0 0) 0x7f2440026030 con 0x7f244c1029c0 2022-09-07T21:31:30.822 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:31:30.822 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":1,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","modified":"2022-09-07T21:29:29.269079Z","created":"2022-09-07T21:29:29.269079Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-09-07T21:31:30.823 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.820+0000 7f2452af5700 1 -- 172.21.15.59:0/3892081282 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f243c0358b0 msgr2=0x7f243c037d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:30.823 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.820+0000 7f2452af5700 1 --2- 172.21.15.59:0/3892081282 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f243c0358b0 0x7f243c037d60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2438009750 tx=0x7f2438006cc0 comp rx=0 tx=0).stop 2022-09-07T21:31:30.823 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.820+0000 7f2452af5700 1 -- 172.21.15.59:0/3892081282 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f244c1029c0 msgr2=0x7f244c198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:30.824 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.820+0000 7f2452af5700 1 --2- 172.21.15.59:0/3892081282 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f244c1029c0 0x7f244c198a50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2440006010 tx=0x7f2440005390 comp rx=0 tx=0).stop 2022-09-07T21:31:30.824 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.820+0000 7f2452af5700 1 -- 172.21.15.59:0/3892081282 shutdown_connections 2022-09-07T21:31:30.824 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.820+0000 7f2452af5700 1 --2- 172.21.15.59:0/3892081282 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f243c0358b0 0x7f243c037d60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:30.824 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.820+0000 7f2452af5700 1 --2- 172.21.15.59:0/3892081282 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f244c1029c0 0x7f244c198a50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:30.825 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.820+0000 7f2452af5700 1 -- 172.21.15.59:0/3892081282 >> 172.21.15.59:0/3892081282 conn(0x7f244c0727f0 msgr2=0x7f244c065cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:30.825 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.820+0000 7f2452af5700 1 -- 172.21.15.59:0/3892081282 shutdown_connections 2022-09-07T21:31:30.825 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:30.820+0000 7f2452af5700 1 -- 172.21.15.59:0/3892081282 wait complete. 2022-09-07T21:31:30.826 INFO:teuthology.orchestra.run.smithi059.stderr:dumped monmap epoch 1 2022-09-07T21:31:31.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:31 smithi047 ceph-mon[90087]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:31.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:31 smithi047 ceph-mon[90087]: from='client.? 172.21.15.59:0/3892081282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-07T21:31:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:31 smithi059 systemd[1]: Starting Ceph mon.c for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:31:32.344 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 podman[97410]: 2022-09-07T21:31:32.344 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: set uid:gid to 167:167 (ceph:ceph) 2022-09-07T21:31:32.344 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: ceph version 17.0.0-14857-g15219419 (152194198413033811b31149de909a7ec5852a7f) quincy (dev), process ceph-mon, pid 7 2022-09-07T21:31:32.345 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: pidfile_write: ignore empty --pid-file 2022-09-07T21:31:32.345 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 bash[97410]: b7716b4d5dad0cd1037533987c973a4a468e22a7cff384c27f92f35dc21cf47d 2022-09-07T21:31:32.345 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: load: jerasure load: lrc 2022-09-07T21:31:32.345 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: RocksDB version: 6.15.5 2022-09-07T21:31:32.346 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-09-07T21:31:32.346 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Compile date Sep 7 2022 2022-09-07T21:31:32.346 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: DB SUMMARY 2022-09-07T21:31:32.346 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: DB Session ID: YSGS4CJAAULAW6MKRSKQ 2022-09-07T21:31:32.346 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: CURRENT file: CURRENT 2022-09-07T21:31:32.347 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: IDENTITY file: IDENTITY 2022-09-07T21:31:32.347 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2022-09-07T21:31:32.347 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2022-09-07T21:31:32.347 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2022-09-07T21:31:32.348 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.error_if_exists: 0 2022-09-07T21:31:32.348 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.create_if_missing: 0 2022-09-07T21:31:32.348 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.paranoid_checks: 1 2022-09-07T21:31:32.349 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-09-07T21:31:32.349 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.env: 0x556a2a67e100 2022-09-07T21:31:32.349 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.fs: Posix File System 2022-09-07T21:31:32.349 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.info_log: 0x556a2b32bc20 2022-09-07T21:31:32.349 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_file_opening_threads: 16 2022-09-07T21:31:32.350 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.statistics: (nil) 2022-09-07T21:31:32.350 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.use_fsync: 0 2022-09-07T21:31:32.350 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_log_file_size: 0 2022-09-07T21:31:32.350 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-09-07T21:31:32.351 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.log_file_time_to_roll: 0 2022-09-07T21:31:32.351 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.keep_log_file_num: 1000 2022-09-07T21:31:32.351 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.recycle_log_file_num: 0 2022-09-07T21:31:32.351 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.allow_fallocate: 1 2022-09-07T21:31:32.351 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.allow_mmap_reads: 0 2022-09-07T21:31:32.352 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.allow_mmap_writes: 0 2022-09-07T21:31:32.352 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.use_direct_reads: 0 2022-09-07T21:31:32.352 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-09-07T21:31:32.352 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.create_missing_column_families: 0 2022-09-07T21:31:32.352 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.db_log_dir: 2022-09-07T21:31:32.353 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2022-09-07T21:31:32.353 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.table_cache_numshardbits: 6 2022-09-07T21:31:32.353 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.WAL_ttl_seconds: 0 2022-09-07T21:31:32.353 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.WAL_size_limit_MB: 0 2022-09-07T21:31:32.353 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-09-07T21:31:32.354 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-09-07T21:31:32.354 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.is_fd_close_on_exec: 1 2022-09-07T21:31:32.354 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.advise_random_on_open: 1 2022-09-07T21:31:32.354 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.db_write_buffer_size: 0 2022-09-07T21:31:32.355 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.write_buffer_manager: 0x556a2b420b70 2022-09-07T21:31:32.355 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-09-07T21:31:32.355 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-09-07T21:31:32.355 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-09-07T21:31:32.355 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.use_adaptive_mutex: 0 2022-09-07T21:31:32.356 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.rate_limiter: (nil) 2022-09-07T21:31:32.356 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-09-07T21:31:32.356 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.wal_recovery_mode: 2 2022-09-07T21:31:32.356 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.enable_thread_tracking: 0 2022-09-07T21:31:32.357 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.enable_pipelined_write: 0 2022-09-07T21:31:32.357 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.unordered_write: 0 2022-09-07T21:31:32.357 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-09-07T21:31:32.357 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-09-07T21:31:32.358 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-09-07T21:31:32.358 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-09-07T21:31:32.358 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.row_cache: None 2022-09-07T21:31:32.359 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.wal_filter: None 2022-09-07T21:31:32.359 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-09-07T21:31:32.359 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.allow_ingest_behind: 0 2022-09-07T21:31:32.359 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.preserve_deletes: 0 2022-09-07T21:31:32.360 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.two_write_queues: 0 2022-09-07T21:31:32.360 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.manual_wal_flush: 0 2022-09-07T21:31:32.360 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.atomic_flush: 0 2022-09-07T21:31:32.360 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-09-07T21:31:32.360 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.persist_stats_to_disk: 0 2022-09-07T21:31:32.361 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.write_dbid_to_manifest: 0 2022-09-07T21:31:32.361 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.log_readahead_size: 0 2022-09-07T21:31:32.361 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-09-07T21:31:32.361 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.best_efforts_recovery: 0 2022-09-07T21:31:32.361 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-09-07T21:31:32.362 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-09-07T21:31:32.362 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.allow_data_in_errors: 0 2022-09-07T21:31:32.362 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.db_host_id: __hostname__ 2022-09-07T21:31:32.364 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_background_jobs: 2 2022-09-07T21:31:32.364 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_background_compactions: -1 2022-09-07T21:31:32.365 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_subcompactions: 1 2022-09-07T21:31:32.365 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-09-07T21:31:32.365 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-09-07T21:31:32.366 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.delayed_write_rate : 16777216 2022-09-07T21:31:32.366 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_total_wal_size: 0 2022-09-07T21:31:32.366 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-09-07T21:31:32.366 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.stats_dump_period_sec: 600 2022-09-07T21:31:32.367 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.stats_persist_period_sec: 600 2022-09-07T21:31:32.367 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-09-07T21:31:32.367 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_open_files: -1 2022-09-07T21:31:32.367 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.bytes_per_sync: 0 2022-09-07T21:31:32.367 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.wal_bytes_per_sync: 0 2022-09-07T21:31:32.368 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.strict_bytes_per_sync: 0 2022-09-07T21:31:32.368 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compaction_readahead_size: 0 2022-09-07T21:31:32.368 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_background_flushes: -1 2022-09-07T21:31:32.368 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Compression algorithms supported: 2022-09-07T21:31:32.369 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-09-07T21:31:32.369 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: kZSTD supported: 0 2022-09-07T21:31:32.369 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: kXpressCompression supported: 0 2022-09-07T21:31:32.369 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: kLZ4HCCompression supported: 1 2022-09-07T21:31:32.369 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: kZlibCompression supported: 1 2022-09-07T21:31:32.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: kSnappyCompression supported: 1 2022-09-07T21:31:32.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: kLZ4Compression supported: 1 2022-09-07T21:31:32.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: kBZip2Compression supported: 0 2022-09-07T21:31:32.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Fast CRC32 supported: Supported on x86 2022-09-07T21:31:32.371 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2022-09-07T21:31:32.371 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2022-09-07T21:31:32.371 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-09-07T21:31:32.371 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.merge_operator: 2022-09-07T21:31:32.371 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compaction_filter: None 2022-09-07T21:31:32.372 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compaction_filter_factory: None 2022-09-07T21:31:32.372 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.sst_partitioner_factory: None 2022-09-07T21:31:32.372 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.memtable_factory: SkipListFactory 2022-09-07T21:31:32.372 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.table_factory: BlockBasedTable 2022-09-07T21:31:32.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556a2b328d48) 2022-09-07T21:31:32.373 INFO:journalctl@ceph.mon.c.smithi059.stdout: cache_index_and_filter_blocks: 1 2022-09-07T21:31:32.373 INFO:journalctl@ceph.mon.c.smithi059.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-09-07T21:31:32.374 INFO:journalctl@ceph.mon.c.smithi059.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-09-07T21:31:32.374 INFO:journalctl@ceph.mon.c.smithi059.stdout: pin_top_level_index_and_filter: 1 2022-09-07T21:31:32.374 INFO:journalctl@ceph.mon.c.smithi059.stdout: index_type: 0 2022-09-07T21:31:32.374 INFO:journalctl@ceph.mon.c.smithi059.stdout: data_block_index_type: 0 2022-09-07T21:31:32.374 INFO:journalctl@ceph.mon.c.smithi059.stdout: index_shortening: 1 2022-09-07T21:31:32.375 INFO:journalctl@ceph.mon.c.smithi059.stdout: data_block_hash_table_util_ratio: 0.750000 2022-09-07T21:31:32.375 INFO:journalctl@ceph.mon.c.smithi059.stdout: hash_index_allow_collision: 1 2022-09-07T21:31:32.375 INFO:journalctl@ceph.mon.c.smithi059.stdout: checksum: 1 2022-09-07T21:31:32.375 INFO:journalctl@ceph.mon.c.smithi059.stdout: no_block_cache: 0 2022-09-07T21:31:32.376 INFO:journalctl@ceph.mon.c.smithi059.stdout: block_cache: 0x556a2b3949b0 2022-09-07T21:31:32.376 INFO:journalctl@ceph.mon.c.smithi059.stdout: block_cache_name: BinnedLRUCache 2022-09-07T21:31:32.376 INFO:journalctl@ceph.mon.c.smithi059.stdout: block_cache_options: 2022-09-07T21:31:32.376 INFO:journalctl@ceph.mon.c.smithi059.stdout: capacity : 536870912 2022-09-07T21:31:32.376 INFO:journalctl@ceph.mon.c.smithi059.stdout: num_shard_bits : 4 2022-09-07T21:31:32.377 INFO:journalctl@ceph.mon.c.smithi059.stdout: strict_capacity_limit : 0 2022-09-07T21:31:32.377 INFO:journalctl@ceph.mon.c.smithi059.stdout: high_pri_pool_ratio: 0.000 2022-09-07T21:31:32.377 INFO:journalctl@ceph.mon.c.smithi059.stdout: block_cache_compressed: (nil) 2022-09-07T21:31:32.377 INFO:journalctl@ceph.mon.c.smithi059.stdout: persistent_cache: (nil) 2022-09-07T21:31:32.377 INFO:journalctl@ceph.mon.c.smithi059.stdout: block_size: 4096 2022-09-07T21:31:32.378 INFO:journalctl@ceph.mon.c.smithi059.stdout: block_size_deviation: 10 2022-09-07T21:31:32.378 INFO:journalctl@ceph.mon.c.smithi059.stdout: block_restart_interval: 16 2022-09-07T21:31:32.378 INFO:journalctl@ceph.mon.c.smithi059.stdout: index_block_restart_interval: 1 2022-09-07T21:31:32.378 INFO:journalctl@ceph.mon.c.smithi059.stdout: metadata_block_size: 4096 2022-09-07T21:31:32.378 INFO:journalctl@ceph.mon.c.smithi059.stdout: partition_filters: 0 2022-09-07T21:31:32.379 INFO:journalctl@ceph.mon.c.smithi059.stdout: use_delta_encoding: 1 2022-09-07T21:31:32.379 INFO:journalctl@ceph.mon.c.smithi059.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-09-07T21:31:32.379 INFO:journalctl@ceph.mon.c.smithi059.stdout: whole_key_filtering: 1 2022-09-07T21:31:32.379 INFO:journalctl@ceph.mon.c.smithi059.stdout: verify_compression: 0 2022-09-07T21:31:32.380 INFO:journalctl@ceph.mon.c.smithi059.stdout: read_amp_bytes_per_bit: 0 2022-09-07T21:31:32.382 INFO:journalctl@ceph.mon.c.smithi059.stdout: format_version: 4 2022-09-07T21:31:32.382 INFO:journalctl@ceph.mon.c.smithi059.stdout: enable_index_compression: 1 2022-09-07T21:31:32.382 INFO:journalctl@ceph.mon.c.smithi059.stdout: block_align: 0 2022-09-07T21:31:32.383 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.write_buffer_size: 33554432 2022-09-07T21:31:32.383 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_write_buffer_number: 2 2022-09-07T21:31:32.384 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compression: NoCompression 2022-09-07T21:31:32.384 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.bottommost_compression: Disabled 2022-09-07T21:31:32.384 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.prefix_extractor: nullptr 2022-09-07T21:31:32.384 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-09-07T21:31:32.385 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.num_levels: 7 2022-09-07T21:31:32.385 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-09-07T21:31:32.385 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-09-07T21:31:32.385 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-09-07T21:31:32.385 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-09-07T21:31:32.386 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-09-07T21:31:32.387 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-09-07T21:31:32.387 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-09-07T21:31:32.388 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-09-07T21:31:32.388 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-09-07T21:31:32.388 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-09-07T21:31:32.389 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compression_opts.window_bits: -14 2022-09-07T21:31:32.389 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compression_opts.level: 32767 2022-09-07T21:31:32.389 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compression_opts.strategy: 0 2022-09-07T21:31:32.390 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-09-07T21:31:32.390 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-09-07T21:31:32.391 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-09-07T21:31:32.391 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compression_opts.enabled: false 2022-09-07T21:31:32.391 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-09-07T21:31:32.391 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-09-07T21:31:32.391 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-09-07T21:31:32.392 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.target_file_size_base: 67108864 2022-09-07T21:31:32.392 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.target_file_size_multiplier: 1 2022-09-07T21:31:32.392 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-09-07T21:31:32.392 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-09-07T21:31:32.393 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-09-07T21:31:32.393 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-09-07T21:31:32.393 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-09-07T21:31:32.393 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-09-07T21:31:32.393 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-09-07T21:31:32.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-09-07T21:31:32.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-09-07T21:31:32.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-09-07T21:31:32.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-09-07T21:31:32.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-09-07T21:31:32.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.arena_block_size: 4194304 2022-09-07T21:31:32.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-09-07T21:31:32.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-09-07T21:31:32.396 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-09-07T21:31:32.396 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.disable_auto_compactions: 0 2022-09-07T21:31:32.396 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-09-07T21:31:32.396 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-09-07T21:31:32.397 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-09-07T21:31:32.397 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-09-07T21:31:32.397 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-09-07T21:31:32.398 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-09-07T21:31:32.398 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-09-07T21:31:32.398 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-09-07T21:31:32.398 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-09-07T21:31:32.399 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-09-07T21:31:32.399 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.table_properties_collectors: 2022-09-07T21:31:32.399 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.inplace_update_support: 0 2022-09-07T21:31:32.399 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.inplace_update_num_locks: 10000 2022-09-07T21:31:32.400 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-09-07T21:31:32.400 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-09-07T21:31:32.400 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.memtable_huge_page_size: 0 2022-09-07T21:31:32.400 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.bloom_locality: 0 2022-09-07T21:31:32.401 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.max_successive_merges: 0 2022-09-07T21:31:32.401 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.optimize_filters_for_hits: 0 2022-09-07T21:31:32.401 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.paranoid_file_checks: 0 2022-09-07T21:31:32.401 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.force_consistency_checks: 1 2022-09-07T21:31:32.402 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.report_bg_io_stats: 0 2022-09-07T21:31:32.402 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.ttl: 2592000 2022-09-07T21:31:32.402 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.periodic_compaction_seconds: 0 2022-09-07T21:31:32.402 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.enable_blob_files: false 2022-09-07T21:31:32.403 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.min_blob_size: 0 2022-09-07T21:31:32.403 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.blob_file_size: 268435456 2022-09-07T21:31:32.403 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.blob_compression_type: NoCompression 2022-09-07T21:31:32.403 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.enable_blob_garbage_collection: false 2022-09-07T21:31:32.404 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-09-07T21:31:32.404 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-09-07T21:31:32.404 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2022-09-07T21:31:32.404 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2022-09-07T21:31:32.405 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662586292259859, "job": 1, "event": "recovery_started", "wal_files": [4]} 2022-09-07T21:31:32.405 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2022-09-07T21:31:32.405 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-09-07T21:31:32.405 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662586292260156, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1662586292, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "4654e5fd-8546-4e75-9fbe-26b093d5d536", "db_session_id": "YSGS4CJAAULAW6MKRSKQ"}} 2022-09-07T21:31:32.406 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2022-09-07T21:31:32.406 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662586292260486, "job": 1, "event": "recovery_finished"} 2022-09-07T21:31:32.406 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-09-07T21:31:32.406 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x556a2b37b880 2022-09-07T21:31:32.407 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: DB pointer 0x556a2b38a000 2022-09-07T21:31:32.407 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2022-09-07T21:31:32.407 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-09-07T21:31:32.407 INFO:journalctl@ceph.mon.c.smithi059.stdout: ** DB Stats ** 2022-09-07T21:31:32.408 INFO:journalctl@ceph.mon.c.smithi059.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-07T21:31:32.408 INFO:journalctl@ceph.mon.c.smithi059.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-09-07T21:31:32.409 INFO:journalctl@ceph.mon.c.smithi059.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-09-07T21:31:32.409 INFO:journalctl@ceph.mon.c.smithi059.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-09-07T21:31:32.410 INFO:journalctl@ceph.mon.c.smithi059.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-09-07T21:31:32.410 INFO:journalctl@ceph.mon.c.smithi059.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-09-07T21:31:32.410 INFO:journalctl@ceph.mon.c.smithi059.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-09-07T21:31:32.411 INFO:journalctl@ceph.mon.c.smithi059.stdout: 2022-09-07T21:31:32.411 INFO:journalctl@ceph.mon.c.smithi059.stdout: ** Compaction Stats [default] ** 2022-09-07T21:31:32.411 INFO:journalctl@ceph.mon.c.smithi059.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-07T21:31:32.411 INFO:journalctl@ceph.mon.c.smithi059.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-07T21:31:32.412 INFO:journalctl@ceph.mon.c.smithi059.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2022-09-07T21:31:32.412 INFO:journalctl@ceph.mon.c.smithi059.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2022-09-07T21:31:32.412 INFO:journalctl@ceph.mon.c.smithi059.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2022-09-07T21:31:32.413 INFO:journalctl@ceph.mon.c.smithi059.stdout: 2022-09-07T21:31:32.413 INFO:journalctl@ceph.mon.c.smithi059.stdout: ** Compaction Stats [default] ** 2022-09-07T21:31:32.413 INFO:journalctl@ceph.mon.c.smithi059.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-07T21:31:32.413 INFO:journalctl@ceph.mon.c.smithi059.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-07T21:31:32.414 INFO:journalctl@ceph.mon.c.smithi059.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2022-09-07T21:31:32.414 INFO:journalctl@ceph.mon.c.smithi059.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-07T21:31:32.414 INFO:journalctl@ceph.mon.c.smithi059.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-09-07T21:31:32.414 INFO:journalctl@ceph.mon.c.smithi059.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-09-07T21:31:32.415 INFO:journalctl@ceph.mon.c.smithi059.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-09-07T21:31:32.415 INFO:journalctl@ceph.mon.c.smithi059.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-09-07T21:31:32.415 INFO:journalctl@ceph.mon.c.smithi059.stdout: AddFile(Keys): cumulative 0, interval 0 2022-09-07T21:31:32.415 INFO:journalctl@ceph.mon.c.smithi059.stdout: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-07T21:31:32.415 INFO:journalctl@ceph.mon.c.smithi059.stdout: Interval compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-07T21:31:32.416 INFO:journalctl@ceph.mon.c.smithi059.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-09-07T21:31:32.416 INFO:journalctl@ceph.mon.c.smithi059.stdout: 2022-09-07T21:31:32.416 INFO:journalctl@ceph.mon.c.smithi059.stdout: ** File Read Latency Histogram By Level [default] ** 2022-09-07T21:31:32.417 INFO:journalctl@ceph.mon.c.smithi059.stdout: 2022-09-07T21:31:32.417 INFO:journalctl@ceph.mon.c.smithi059.stdout: ** Compaction Stats [default] ** 2022-09-07T21:31:32.417 INFO:journalctl@ceph.mon.c.smithi059.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-07T21:31:32.417 INFO:journalctl@ceph.mon.c.smithi059.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-07T21:31:32.418 INFO:journalctl@ceph.mon.c.smithi059.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2022-09-07T21:31:32.418 INFO:journalctl@ceph.mon.c.smithi059.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2022-09-07T21:31:32.418 INFO:journalctl@ceph.mon.c.smithi059.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-09-07T21:31:32.419 INFO:journalctl@ceph.mon.c.smithi059.stdout: 2022-09-07T21:31:32.419 INFO:journalctl@ceph.mon.c.smithi059.stdout: ** Compaction Stats [default] ** 2022-09-07T21:31:32.419 INFO:journalctl@ceph.mon.c.smithi059.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-07T21:31:32.420 INFO:journalctl@ceph.mon.c.smithi059.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-07T21:31:32.420 INFO:journalctl@ceph.mon.c.smithi059.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2022-09-07T21:31:32.420 INFO:journalctl@ceph.mon.c.smithi059.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-07T21:31:32.420 INFO:journalctl@ceph.mon.c.smithi059.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-09-07T21:31:32.421 INFO:journalctl@ceph.mon.c.smithi059.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-09-07T21:31:32.421 INFO:journalctl@ceph.mon.c.smithi059.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-09-07T21:31:32.422 INFO:journalctl@ceph.mon.c.smithi059.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-09-07T21:31:32.422 INFO:journalctl@ceph.mon.c.smithi059.stdout: AddFile(Keys): cumulative 0, interval 0 2022-09-07T21:31:32.423 INFO:journalctl@ceph.mon.c.smithi059.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-07T21:31:32.423 INFO:journalctl@ceph.mon.c.smithi059.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-07T21:31:32.424 INFO:journalctl@ceph.mon.c.smithi059.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-09-07T21:31:32.424 INFO:journalctl@ceph.mon.c.smithi059.stdout: 2022-09-07T21:31:32.425 INFO:journalctl@ceph.mon.c.smithi059.stdout: ** File Read Latency Histogram By Level [default] ** 2022-09-07T21:31:32.425 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c does not exist in monmap, will attempt to join an existing cluster 2022-09-07T21:31:32.426 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: using public_addr v2:172.21.15.59:0/0 -> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2022-09-07T21:31:32.426 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: starting mon.c rank -1 at public addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] at bind addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:31:32.427 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(???) e0 preinit fsid 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:31:32.427 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(synchronizing).mds e1 new map 2022-09-07T21:31:32.427 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(synchronizing).mds e1 print_map 2022-09-07T21:31:32.428 INFO:journalctl@ceph.mon.c.smithi059.stdout: e1 2022-09-07T21:31:32.428 INFO:journalctl@ceph.mon.c.smithi059.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-09-07T21:31:32.429 INFO:journalctl@ceph.mon.c.smithi059.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-09-07T21:31:32.429 INFO:journalctl@ceph.mon.c.smithi059.stdout: legacy client fscid: -1 2022-09-07T21:31:32.430 INFO:journalctl@ceph.mon.c.smithi059.stdout: 2022-09-07T21:31:32.430 INFO:journalctl@ceph.mon.c.smithi059.stdout: No filesystems configured 2022-09-07T21:31:32.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2022-09-07T21:31:32.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2022-09-07T21:31:32.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2022-09-07T21:31:32.432 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2022-09-07T21:31:32.432 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2022-09-07T21:31:32.433 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2022-09-07T21:31:32.433 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2022-09-07T21:31:32.433 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-09-07T21:31:32.434 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-09-07T21:31:32.434 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-09-07T21:31:32.435 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi055", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:31:32.435 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: Deploying cephadm binary to smithi055 2022-09-07T21:31:32.436 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:32.436 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.436 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:32.437 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:32.437 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:32.438 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.438 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: Added host smithi055 2022-09-07T21:31:32.438 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.439 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:32.439 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.440 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:31:32.440 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:32.440 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.441 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.441 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.441 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.442 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi055", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:31:32.442 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.443 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:32.443 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi059", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:31:32.443 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: Deploying cephadm binary to smithi059 2022-09-07T21:31:32.444 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:32.444 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.445 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: Added host smithi059 2022-09-07T21:31:32.445 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:32.446 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:32.446 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:32.447 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: Updating smithi055:/etc/ceph/ceph.conf 2022-09-07T21:31:32.447 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: Updating smithi055:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:31:32.447 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.448 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:31:32.448 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:32.449 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: Updating smithi055:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:31:32.449 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.449 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.450 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.450 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.451 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:32.451 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:31:32.451 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:32.452 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.452 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.452 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.453 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.453 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:31:32.454 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.454 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/272893152' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-09-07T21:31:32.455 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/272893152' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-09-07T21:31:32.455 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: osdmap e4: 0 total, 0 up, 0 in 2022-09-07T21:31:32.455 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:32.456 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:32.456 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.457 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:32.457 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:32.457 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:32.458 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi047:172.21.15.47=a;smithi055:172.21.15.55=b;smithi059:172.21.15.59=c", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:31:32.458 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: Saving service mon spec with placement smithi047:172.21.15.47=a;smithi055:172.21.15.55=b;smithi059:172.21.15.59=c;count:3 2022-09-07T21:31:32.458 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: Updating smithi059:/etc/ceph/ceph.conf 2022-09-07T21:31:32.458 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: Updating smithi059:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:31:32.459 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.459 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.459 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:31:32.460 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: Updating smithi059:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:31:32.460 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:32.460 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:32.460 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:32.460 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:32.461 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: Deploying daemon mon.c on smithi059 2022-09-07T21:31:32.461 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:32.461 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: from='client.? 172.21.15.59:0/3892081282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-07T21:31:32.461 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2022-09-07T21:31:32.461 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 ceph-mon[97450]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2022-09-07T21:31:32.462 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:32 smithi059 systemd[1]: Started Ceph mon.c for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:31:32.462 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-09-07T21:31:32.463 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph mon dump -f json 2022-09-07T21:31:33.067 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.c/config 2022-09-07T21:31:37.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:31:37.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: mon.a calling monitor election 2022-09-07T21:31:37.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:37.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:37.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:37.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:37.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: mon.c calling monitor election 2022-09-07T21:31:37.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:37.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:37.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:37.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:37.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:37.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2022-09-07T21:31:37.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: monmap e2: 2 mons at {a=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],c=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2022-09-07T21:31:37.658 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: fsmap 2022-09-07T21:31:37.658 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: osdmap e4: 0 total, 0 up, 0 in 2022-09-07T21:31:37.658 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: mgrmap e13: x(active, since 50s) 2022-09-07T21:31:37.659 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: overall HEALTH_OK 2022-09-07T21:31:37.660 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:37.660 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:37.660 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:37.661 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:37.666 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.651+0000 7fa43dfe5700 1 -- 172.21.15.59:0/3192520435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa420003440 msgr2=0x7fa4200058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:37.669 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.651+0000 7fa43dfe5700 1 --2- 172.21.15.59:0/3192520435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa420003440 0x7fa4200058c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa4380fdd90 tx=0x7fa428011d30 comp rx=0 tx=0).stop 2022-09-07T21:31:37.669 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.652+0000 7fa43dfe5700 1 -- 172.21.15.59:0/3192520435 shutdown_connections 2022-09-07T21:31:37.669 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.652+0000 7fa43dfe5700 1 --2- 172.21.15.59:0/3192520435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa420003440 0x7fa4200058c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:37.670 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.652+0000 7fa43dfe5700 1 -- 172.21.15.59:0/3192520435 >> 172.21.15.59:0/3192520435 conn(0x7fa438072700 msgr2=0x7fa438072b00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:37.670 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.652+0000 7fa43dfe5700 1 -- 172.21.15.59:0/3192520435 shutdown_connections 2022-09-07T21:31:37.671 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.652+0000 7fa43dfe5700 1 -- 172.21.15.59:0/3192520435 wait complete. 2022-09-07T21:31:37.671 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.653+0000 7fa43dfe5700 1 Processor -- start 2022-09-07T21:31:37.672 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.653+0000 7fa43dfe5700 1 -- start start 2022-09-07T21:31:37.672 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.654+0000 7fa43dfe5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa420003440 0x7fa438193a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:37.673 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.654+0000 7fa43dfe5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa438193fb0 0x7fa438194420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:37.673 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.654+0000 7fa43dfe5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43819cb70 con 0x7fa438193fb0 2022-09-07T21:31:37.674 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.654+0000 7fa43dfe5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43819ccb0 con 0x7fa420003440 2022-09-07T21:31:37.674 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.654+0000 7fa4377fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa420003440 0x7fa438193a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:37.674 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.654+0000 7fa4377fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa420003440 0x7fa438193a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40084/0 (socket says 172.21.15.59:40084) 2022-09-07T21:31:37.675 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.654+0000 7fa4377fe700 1 -- 172.21.15.59:0/2455948005 learned_addr learned my addr 172.21.15.59:0/2455948005 (peer_addr_for_me v2:172.21.15.59:0/0) 2022-09-07T21:31:37.676 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.654+0000 7fa4377fe700 1 -- 172.21.15.59:0/2455948005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa420003440 msgr2=0x7fa438193a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2022-09-07T21:31:37.676 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.654+0000 7fa4377fe700 1 -- 172.21.15.59:0/2455948005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa420003440 msgr2=0x7fa438193a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-09-07T21:31:37.677 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.654+0000 7fa4377fe700 1 --2- 172.21.15.59:0/2455948005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa420003440 0x7fa438193a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-09-07T21:31:37.677 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.654+0000 7fa4377fe700 1 --2- 172.21.15.59:0/2455948005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa420003440 0x7fa438193a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-07T21:31:37.678 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.654+0000 7fa436ffd700 1 --2- 172.21.15.59:0/2455948005 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa438193fb0 0x7fa438194420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:37.678 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.655+0000 7fa436ffd700 1 -- 172.21.15.59:0/2455948005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa420003440 msgr2=0x7fa438193a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-07T21:31:37.678 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.655+0000 7fa436ffd700 1 --2- 172.21.15.59:0/2455948005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa420003440 0x7fa438193a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:37.679 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.655+0000 7fa436ffd700 1 -- 172.21.15.59:0/2455948005 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa428011780 con 0x7fa438193fb0 2022-09-07T21:31:37.679 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.655+0000 7fa436ffd700 1 --2- 172.21.15.59:0/2455948005 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa438193fb0 0x7fa438194420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00b390 tx=0x7fa42c00b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:37.679 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.655+0000 7fa434ff9700 1 -- 172.21.15.59:0/2455948005 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fa42c011420 con 0x7fa438193fb0 2022-09-07T21:31:37.680 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.655+0000 7fa434ff9700 1 -- 172.21.15.59:0/2455948005 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa42c011a20 con 0x7fa438193fb0 2022-09-07T21:31:37.680 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.655+0000 7fa43dfe5700 1 -- 172.21.15.59:0/2455948005 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa43819ceb0 con 0x7fa438193fb0 2022-09-07T21:31:37.681 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.656+0000 7fa434ff9700 1 -- 172.21.15.59:0/2455948005 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fa42c010a90 con 0x7fa438193fb0 2022-09-07T21:31:37.681 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.656+0000 7fa43dfe5700 1 -- 172.21.15.59:0/2455948005 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43819d380 con 0x7fa438193fb0 2022-09-07T21:31:37.682 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.656+0000 7fa434ff9700 1 -- 172.21.15.59:0/2455948005 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7fa42c021070 con 0x7fa438193fb0 2022-09-07T21:31:37.682 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.657+0000 7fa43dfe5700 1 -- 172.21.15.59:0/2455948005 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa438060310 con 0x7fa438193fb0 2022-09-07T21:31:37.682 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.657+0000 7fa434ff9700 1 --2- 172.21.15.59:0/2455948005 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4240356b0 0x7fa424037b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:37.683 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.657+0000 7fa434ff9700 1 -- 172.21.15.59:0/2455948005 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa42c046ff0 con 0x7fa438193fb0 2022-09-07T21:31:37.683 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.657+0000 7fa4377fe700 1 --2- 172.21.15.59:0/2455948005 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4240356b0 0x7fa424037b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:37.684 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.658+0000 7fa4377fe700 1 --2- 172.21.15.59:0/2455948005 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4240356b0 0x7fa424037b60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa4280059c0 tx=0x7fa428005930 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:37.684 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.662+0000 7fa434ff9700 1 -- 172.21.15.59:0/2455948005 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fa42c00fcb0 con 0x7fa438193fb0 2022-09-07T21:31:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:31:37.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: mon.a calling monitor election 2022-09-07T21:31:37.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:37.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:37.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:37.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:37.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: mon.c calling monitor election 2022-09-07T21:31:37.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:37.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:37.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:37.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:37.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:37.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2022-09-07T21:31:37.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: monmap e2: 2 mons at {a=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],c=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2022-09-07T21:31:37.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: fsmap 2022-09-07T21:31:37.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: osdmap e4: 0 total, 0 up, 0 in 2022-09-07T21:31:37.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: mgrmap e13: x(active, since 50s) 2022-09-07T21:31:37.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: overall HEALTH_OK 2022-09-07T21:31:37.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:37.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:37.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:37.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:37.938 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.935+0000 7fa43dfe5700 1 -- 172.21.15.59:0/2455948005 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4381044b0 con 0x7fa438193fb0 2022-09-07T21:31:37.939 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.936+0000 7fa434ff9700 1 -- 172.21.15.59:0/2455948005 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+976 (secure 0 0 0) 0x7fa42c00fad0 con 0x7fa438193fb0 2022-09-07T21:31:37.939 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:31:37.939 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":2,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","modified":"2022-09-07T21:31:32.285336Z","created":"2022-09-07T21:29:29.269079Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2022-09-07T21:31:37.940 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.938+0000 7fa43dfe5700 1 -- 172.21.15.59:0/2455948005 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4240356b0 msgr2=0x7fa424037b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:37.941 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.938+0000 7fa43dfe5700 1 --2- 172.21.15.59:0/2455948005 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4240356b0 0x7fa424037b60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa4280059c0 tx=0x7fa428005930 comp rx=0 tx=0).stop 2022-09-07T21:31:37.941 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.938+0000 7fa43dfe5700 1 -- 172.21.15.59:0/2455948005 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa438193fb0 msgr2=0x7fa438194420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:37.941 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.938+0000 7fa43dfe5700 1 --2- 172.21.15.59:0/2455948005 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa438193fb0 0x7fa438194420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00b390 tx=0x7fa42c00b6a0 comp rx=0 tx=0).stop 2022-09-07T21:31:37.942 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.938+0000 7fa43dfe5700 1 -- 172.21.15.59:0/2455948005 shutdown_connections 2022-09-07T21:31:37.942 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.938+0000 7fa43dfe5700 1 --2- 172.21.15.59:0/2455948005 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4240356b0 0x7fa424037b60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:37.942 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.938+0000 7fa43dfe5700 1 --2- 172.21.15.59:0/2455948005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa420003440 0x7fa438193a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:37.942 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.938+0000 7fa43dfe5700 1 --2- 172.21.15.59:0/2455948005 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa438193fb0 0x7fa438194420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:37.943 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.938+0000 7fa43dfe5700 1 -- 172.21.15.59:0/2455948005 >> 172.21.15.59:0/2455948005 conn(0x7fa438072700 msgr2=0x7fa4380fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:37.943 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.938+0000 7fa43dfe5700 1 -- 172.21.15.59:0/2455948005 shutdown_connections 2022-09-07T21:31:37.943 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:37.938+0000 7fa43dfe5700 1 -- 172.21.15.59:0/2455948005 wait complete. 2022-09-07T21:31:37.944 INFO:teuthology.orchestra.run.smithi059.stderr:dumped monmap epoch 2 2022-09-07T21:31:38.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:38 smithi047 ceph-mon[90087]: Deploying daemon mon.b on smithi055 2022-09-07T21:31:38.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:38 smithi047 ceph-mon[90087]: from='client.? 172.21.15.59:0/2455948005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-07T21:31:38.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:38 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:38 smithi059 ceph-mon[97450]: Deploying daemon mon.b on smithi055 2022-09-07T21:31:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:38 smithi059 ceph-mon[97450]: from='client.? 172.21.15.59:0/2455948005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-07T21:31:38.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:38 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:39.410 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-09-07T21:31:39.411 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph mon dump -f json 2022-09-07T21:31:39.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:39 smithi047 ceph-mon[90087]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:39 smithi059 ceph-mon[97450]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.c/config 2022-09-07T21:31:40.441 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 systemd[1]: Starting Ceph mon.b for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:31:40.704 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 podman[96993]: 2022-09-07T21:31:40.704 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 bash[96993]: 0f3ba03b8a18dd03056efbe4c610727f5938d507b0631f69524f9dd6ec9dc7fe 2022-09-07T21:31:40.704 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: set uid:gid to 167:167 (ceph:ceph) 2022-09-07T21:31:40.705 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: ceph version 17.0.0-14857-g15219419 (152194198413033811b31149de909a7ec5852a7f) quincy (dev), process ceph-mon, pid 7 2022-09-07T21:31:40.705 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pidfile_write: ignore empty --pid-file 2022-09-07T21:31:40.705 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: load: jerasure load: lrc 2022-09-07T21:31:40.706 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: RocksDB version: 6.15.5 2022-09-07T21:31:40.706 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-09-07T21:31:40.706 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Compile date Sep 7 2022 2022-09-07T21:31:40.707 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: DB SUMMARY 2022-09-07T21:31:40.707 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: DB Session ID: 0JP9MVLUHIQ7QCEWZBNP 2022-09-07T21:31:40.707 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: CURRENT file: CURRENT 2022-09-07T21:31:40.707 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: IDENTITY file: IDENTITY 2022-09-07T21:31:40.708 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2022-09-07T21:31:40.708 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2022-09-07T21:31:40.708 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2022-09-07T21:31:40.709 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.error_if_exists: 0 2022-09-07T21:31:40.709 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.create_if_missing: 0 2022-09-07T21:31:40.709 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.paranoid_checks: 1 2022-09-07T21:31:40.710 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-09-07T21:31:40.710 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.env: 0x55f90b4d5100 2022-09-07T21:31:40.711 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.fs: Posix File System 2022-09-07T21:31:40.711 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.info_log: 0x55f90c769c20 2022-09-07T21:31:40.712 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_file_opening_threads: 16 2022-09-07T21:31:40.712 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.statistics: (nil) 2022-09-07T21:31:40.713 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.use_fsync: 0 2022-09-07T21:31:40.713 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_log_file_size: 0 2022-09-07T21:31:40.713 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-09-07T21:31:40.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.log_file_time_to_roll: 0 2022-09-07T21:31:40.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.keep_log_file_num: 1000 2022-09-07T21:31:40.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.recycle_log_file_num: 0 2022-09-07T21:31:40.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.allow_fallocate: 1 2022-09-07T21:31:40.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.allow_mmap_reads: 0 2022-09-07T21:31:40.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.allow_mmap_writes: 0 2022-09-07T21:31:40.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.use_direct_reads: 0 2022-09-07T21:31:40.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-09-07T21:31:40.717 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.create_missing_column_families: 0 2022-09-07T21:31:40.717 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.db_log_dir: 2022-09-07T21:31:40.718 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2022-09-07T21:31:40.718 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.table_cache_numshardbits: 6 2022-09-07T21:31:40.719 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.WAL_ttl_seconds: 0 2022-09-07T21:31:40.719 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.WAL_size_limit_MB: 0 2022-09-07T21:31:40.720 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-09-07T21:31:40.720 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-09-07T21:31:40.720 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.is_fd_close_on_exec: 1 2022-09-07T21:31:40.721 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.advise_random_on_open: 1 2022-09-07T21:31:40.721 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.db_write_buffer_size: 0 2022-09-07T21:31:40.721 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.write_buffer_manager: 0x55f90c85eb10 2022-09-07T21:31:40.722 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-09-07T21:31:40.722 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-09-07T21:31:40.723 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-09-07T21:31:40.723 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.use_adaptive_mutex: 0 2022-09-07T21:31:40.723 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.rate_limiter: (nil) 2022-09-07T21:31:40.723 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-09-07T21:31:40.724 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.wal_recovery_mode: 2 2022-09-07T21:31:40.724 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.enable_thread_tracking: 0 2022-09-07T21:31:40.724 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.enable_pipelined_write: 0 2022-09-07T21:31:40.725 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.unordered_write: 0 2022-09-07T21:31:40.725 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-09-07T21:31:40.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-09-07T21:31:40.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-09-07T21:31:40.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-09-07T21:31:40.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.row_cache: None 2022-09-07T21:31:40.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.wal_filter: None 2022-09-07T21:31:40.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-09-07T21:31:40.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.allow_ingest_behind: 0 2022-09-07T21:31:40.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.preserve_deletes: 0 2022-09-07T21:31:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.two_write_queues: 0 2022-09-07T21:31:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.manual_wal_flush: 0 2022-09-07T21:31:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.atomic_flush: 0 2022-09-07T21:31:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-09-07T21:31:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.persist_stats_to_disk: 0 2022-09-07T21:31:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.write_dbid_to_manifest: 0 2022-09-07T21:31:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.log_readahead_size: 0 2022-09-07T21:31:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-09-07T21:31:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.best_efforts_recovery: 0 2022-09-07T21:31:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-09-07T21:31:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-09-07T21:31:40.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.allow_data_in_errors: 0 2022-09-07T21:31:40.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.db_host_id: __hostname__ 2022-09-07T21:31:40.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_background_jobs: 2 2022-09-07T21:31:40.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_background_compactions: -1 2022-09-07T21:31:40.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_subcompactions: 1 2022-09-07T21:31:40.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-09-07T21:31:40.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-09-07T21:31:40.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.delayed_write_rate : 16777216 2022-09-07T21:31:40.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_total_wal_size: 0 2022-09-07T21:31:40.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-09-07T21:31:40.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.stats_dump_period_sec: 600 2022-09-07T21:31:40.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.stats_persist_period_sec: 600 2022-09-07T21:31:40.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-09-07T21:31:40.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_open_files: -1 2022-09-07T21:31:40.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.bytes_per_sync: 0 2022-09-07T21:31:40.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.wal_bytes_per_sync: 0 2022-09-07T21:31:40.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.strict_bytes_per_sync: 0 2022-09-07T21:31:40.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compaction_readahead_size: 0 2022-09-07T21:31:40.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_background_flushes: -1 2022-09-07T21:31:40.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Compression algorithms supported: 2022-09-07T21:31:40.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-09-07T21:31:40.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: kZSTD supported: 0 2022-09-07T21:31:40.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: kXpressCompression supported: 0 2022-09-07T21:31:40.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: kLZ4HCCompression supported: 1 2022-09-07T21:31:40.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: kZlibCompression supported: 1 2022-09-07T21:31:40.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: kSnappyCompression supported: 1 2022-09-07T21:31:40.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: kLZ4Compression supported: 1 2022-09-07T21:31:40.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: kBZip2Compression supported: 0 2022-09-07T21:31:40.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Fast CRC32 supported: Supported on x86 2022-09-07T21:31:40.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2022-09-07T21:31:40.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2022-09-07T21:31:40.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-09-07T21:31:40.739 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.merge_operator: 2022-09-07T21:31:40.739 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compaction_filter: None 2022-09-07T21:31:40.739 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compaction_filter_factory: None 2022-09-07T21:31:40.739 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.sst_partitioner_factory: None 2022-09-07T21:31:40.740 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.memtable_factory: SkipListFactory 2022-09-07T21:31:40.740 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.table_factory: BlockBasedTable 2022-09-07T21:31:40.740 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f90c766d48) 2022-09-07T21:31:40.740 INFO:journalctl@ceph.mon.b.smithi055.stdout: cache_index_and_filter_blocks: 1 2022-09-07T21:31:40.741 INFO:journalctl@ceph.mon.b.smithi055.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-09-07T21:31:40.741 INFO:journalctl@ceph.mon.b.smithi055.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-09-07T21:31:40.741 INFO:journalctl@ceph.mon.b.smithi055.stdout: pin_top_level_index_and_filter: 1 2022-09-07T21:31:40.741 INFO:journalctl@ceph.mon.b.smithi055.stdout: index_type: 0 2022-09-07T21:31:40.742 INFO:journalctl@ceph.mon.b.smithi055.stdout: data_block_index_type: 0 2022-09-07T21:31:40.742 INFO:journalctl@ceph.mon.b.smithi055.stdout: index_shortening: 1 2022-09-07T21:31:40.742 INFO:journalctl@ceph.mon.b.smithi055.stdout: data_block_hash_table_util_ratio: 0.750000 2022-09-07T21:31:40.742 INFO:journalctl@ceph.mon.b.smithi055.stdout: hash_index_allow_collision: 1 2022-09-07T21:31:40.742 INFO:journalctl@ceph.mon.b.smithi055.stdout: checksum: 1 2022-09-07T21:31:40.743 INFO:journalctl@ceph.mon.b.smithi055.stdout: no_block_cache: 0 2022-09-07T21:31:40.743 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_cache: 0x55f90c7d29b0 2022-09-07T21:31:40.743 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_cache_name: BinnedLRUCache 2022-09-07T21:31:40.743 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_cache_options: 2022-09-07T21:31:40.744 INFO:journalctl@ceph.mon.b.smithi055.stdout: capacity : 536870912 2022-09-07T21:31:40.744 INFO:journalctl@ceph.mon.b.smithi055.stdout: num_shard_bits : 4 2022-09-07T21:31:40.744 INFO:journalctl@ceph.mon.b.smithi055.stdout: strict_capacity_limit : 0 2022-09-07T21:31:40.744 INFO:journalctl@ceph.mon.b.smithi055.stdout: high_pri_pool_ratio: 0.000 2022-09-07T21:31:40.745 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_cache_compressed: (nil) 2022-09-07T21:31:40.745 INFO:journalctl@ceph.mon.b.smithi055.stdout: persistent_cache: (nil) 2022-09-07T21:31:40.745 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_size: 4096 2022-09-07T21:31:40.745 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_size_deviation: 10 2022-09-07T21:31:40.745 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_restart_interval: 16 2022-09-07T21:31:40.746 INFO:journalctl@ceph.mon.b.smithi055.stdout: index_block_restart_interval: 1 2022-09-07T21:31:40.746 INFO:journalctl@ceph.mon.b.smithi055.stdout: metadata_block_size: 4096 2022-09-07T21:31:40.746 INFO:journalctl@ceph.mon.b.smithi055.stdout: partition_filters: 0 2022-09-07T21:31:40.746 INFO:journalctl@ceph.mon.b.smithi055.stdout: use_delta_encoding: 1 2022-09-07T21:31:40.747 INFO:journalctl@ceph.mon.b.smithi055.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-09-07T21:31:40.747 INFO:journalctl@ceph.mon.b.smithi055.stdout: whole_key_filtering: 1 2022-09-07T21:31:40.747 INFO:journalctl@ceph.mon.b.smithi055.stdout: verify_compression: 0 2022-09-07T21:31:40.747 INFO:journalctl@ceph.mon.b.smithi055.stdout: read_amp_bytes_per_bit: 0 2022-09-07T21:31:40.748 INFO:journalctl@ceph.mon.b.smithi055.stdout: format_version: 4 2022-09-07T21:31:40.748 INFO:journalctl@ceph.mon.b.smithi055.stdout: enable_index_compression: 1 2022-09-07T21:31:40.748 INFO:journalctl@ceph.mon.b.smithi055.stdout: block_align: 0 2022-09-07T21:31:40.748 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.write_buffer_size: 33554432 2022-09-07T21:31:40.749 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_write_buffer_number: 2 2022-09-07T21:31:40.749 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compression: NoCompression 2022-09-07T21:31:40.749 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.bottommost_compression: Disabled 2022-09-07T21:31:40.749 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.prefix_extractor: nullptr 2022-09-07T21:31:40.749 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-09-07T21:31:40.750 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.num_levels: 7 2022-09-07T21:31:40.750 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-09-07T21:31:40.750 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-09-07T21:31:40.750 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-09-07T21:31:40.751 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-09-07T21:31:40.751 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-09-07T21:31:40.751 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-09-07T21:31:40.751 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-09-07T21:31:40.751 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-09-07T21:31:40.752 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-09-07T21:31:40.752 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-09-07T21:31:40.752 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compression_opts.window_bits: -14 2022-09-07T21:31:40.752 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compression_opts.level: 32767 2022-09-07T21:31:40.753 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compression_opts.strategy: 0 2022-09-07T21:31:40.753 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-09-07T21:31:40.753 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-09-07T21:31:40.753 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-09-07T21:31:40.754 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compression_opts.enabled: false 2022-09-07T21:31:40.754 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-09-07T21:31:40.754 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-09-07T21:31:40.754 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-09-07T21:31:40.755 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.target_file_size_base: 67108864 2022-09-07T21:31:40.755 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.target_file_size_multiplier: 1 2022-09-07T21:31:40.755 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-09-07T21:31:40.755 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-09-07T21:31:40.756 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-09-07T21:31:40.756 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-09-07T21:31:40.756 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-09-07T21:31:40.756 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-09-07T21:31:40.757 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-09-07T21:31:40.757 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-09-07T21:31:40.757 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-09-07T21:31:40.757 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-09-07T21:31:40.758 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-09-07T21:31:40.758 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-09-07T21:31:40.758 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.arena_block_size: 4194304 2022-09-07T21:31:40.758 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-09-07T21:31:40.758 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-09-07T21:31:40.759 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-09-07T21:31:40.759 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.disable_auto_compactions: 0 2022-09-07T21:31:40.759 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-09-07T21:31:40.759 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-09-07T21:31:40.760 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-09-07T21:31:40.760 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-09-07T21:31:40.760 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-09-07T21:31:40.760 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-09-07T21:31:40.761 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-09-07T21:31:40.761 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-09-07T21:31:40.761 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-09-07T21:31:40.761 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-09-07T21:31:40.762 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.table_properties_collectors: 2022-09-07T21:31:40.762 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.inplace_update_support: 0 2022-09-07T21:31:40.762 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.inplace_update_num_locks: 10000 2022-09-07T21:31:40.762 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-09-07T21:31:40.763 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-09-07T21:31:40.763 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.memtable_huge_page_size: 0 2022-09-07T21:31:40.763 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.bloom_locality: 0 2022-09-07T21:31:40.763 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.max_successive_merges: 0 2022-09-07T21:31:40.764 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.optimize_filters_for_hits: 0 2022-09-07T21:31:40.764 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.paranoid_file_checks: 0 2022-09-07T21:31:40.764 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.force_consistency_checks: 1 2022-09-07T21:31:40.764 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.report_bg_io_stats: 0 2022-09-07T21:31:40.764 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.ttl: 2592000 2022-09-07T21:31:40.765 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.periodic_compaction_seconds: 0 2022-09-07T21:31:40.765 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.enable_blob_files: false 2022-09-07T21:31:40.765 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.min_blob_size: 0 2022-09-07T21:31:40.765 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.blob_file_size: 268435456 2022-09-07T21:31:40.766 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.blob_compression_type: NoCompression 2022-09-07T21:31:40.766 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.enable_blob_garbage_collection: false 2022-09-07T21:31:40.766 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-09-07T21:31:40.766 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-09-07T21:31:40.767 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2022-09-07T21:31:40.767 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2022-09-07T21:31:40.767 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662586300680223, "job": 1, "event": "recovery_started", "wal_files": [4]} 2022-09-07T21:31:40.767 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2022-09-07T21:31:40.767 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-09-07T21:31:40.768 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662586300680635, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1662586300, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "9c090090-27c0-4325-99ba-1e716be81605", "db_session_id": "0JP9MVLUHIQ7QCEWZBNP"}} 2022-09-07T21:31:40.768 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2022-09-07T21:31:40.768 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662586300681223, "job": 1, "event": "recovery_finished"} 2022-09-07T21:31:40.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-09-07T21:31:40.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55f90c7b9880 2022-09-07T21:31:40.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: DB pointer 0x55f90c7c8000 2022-09-07T21:31:40.769 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b does not exist in monmap, will attempt to join an existing cluster 2022-09-07T21:31:40.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2022-09-07T21:31:40.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-09-07T21:31:40.770 INFO:journalctl@ceph.mon.b.smithi055.stdout: ** DB Stats ** 2022-09-07T21:31:40.770 INFO:journalctl@ceph.mon.b.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-07T21:31:40.771 INFO:journalctl@ceph.mon.b.smithi055.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-09-07T21:31:40.771 INFO:journalctl@ceph.mon.b.smithi055.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-09-07T21:31:40.771 INFO:journalctl@ceph.mon.b.smithi055.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-09-07T21:31:40.771 INFO:journalctl@ceph.mon.b.smithi055.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-09-07T21:31:40.772 INFO:journalctl@ceph.mon.b.smithi055.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-09-07T21:31:40.772 INFO:journalctl@ceph.mon.b.smithi055.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-09-07T21:31:40.772 INFO:journalctl@ceph.mon.b.smithi055.stdout: 2022-09-07T21:31:40.772 INFO:journalctl@ceph.mon.b.smithi055.stdout: ** Compaction Stats [default] ** 2022-09-07T21:31:40.772 INFO:journalctl@ceph.mon.b.smithi055.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-07T21:31:40.773 INFO:journalctl@ceph.mon.b.smithi055.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-07T21:31:40.773 INFO:journalctl@ceph.mon.b.smithi055.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 2022-09-07T21:31:40.773 INFO:journalctl@ceph.mon.b.smithi055.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 2022-09-07T21:31:40.773 INFO:journalctl@ceph.mon.b.smithi055.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 2022-09-07T21:31:40.774 INFO:journalctl@ceph.mon.b.smithi055.stdout: 2022-09-07T21:31:40.774 INFO:journalctl@ceph.mon.b.smithi055.stdout: ** Compaction Stats [default] ** 2022-09-07T21:31:40.774 INFO:journalctl@ceph.mon.b.smithi055.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-07T21:31:40.774 INFO:journalctl@ceph.mon.b.smithi055.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-07T21:31:40.775 INFO:journalctl@ceph.mon.b.smithi055.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 2022-09-07T21:31:40.775 INFO:journalctl@ceph.mon.b.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-07T21:31:40.775 INFO:journalctl@ceph.mon.b.smithi055.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-09-07T21:31:40.776 INFO:journalctl@ceph.mon.b.smithi055.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-09-07T21:31:40.776 INFO:journalctl@ceph.mon.b.smithi055.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-09-07T21:31:40.776 INFO:journalctl@ceph.mon.b.smithi055.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-09-07T21:31:40.776 INFO:journalctl@ceph.mon.b.smithi055.stdout: AddFile(Keys): cumulative 0, interval 0 2022-09-07T21:31:40.777 INFO:journalctl@ceph.mon.b.smithi055.stdout: Cumulative compaction: 0.00 GB write, 0.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-07T21:31:40.777 INFO:journalctl@ceph.mon.b.smithi055.stdout: Interval compaction: 0.00 GB write, 0.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-07T21:31:40.777 INFO:journalctl@ceph.mon.b.smithi055.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-09-07T21:31:40.777 INFO:journalctl@ceph.mon.b.smithi055.stdout: 2022-09-07T21:31:40.778 INFO:journalctl@ceph.mon.b.smithi055.stdout: ** File Read Latency Histogram By Level [default] ** 2022-09-07T21:31:40.778 INFO:journalctl@ceph.mon.b.smithi055.stdout: 2022-09-07T21:31:40.778 INFO:journalctl@ceph.mon.b.smithi055.stdout: ** Compaction Stats [default] ** 2022-09-07T21:31:40.778 INFO:journalctl@ceph.mon.b.smithi055.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-07T21:31:40.779 INFO:journalctl@ceph.mon.b.smithi055.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-07T21:31:40.779 INFO:journalctl@ceph.mon.b.smithi055.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 2022-09-07T21:31:40.779 INFO:journalctl@ceph.mon.b.smithi055.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 2022-09-07T21:31:40.779 INFO:journalctl@ceph.mon.b.smithi055.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-09-07T21:31:40.780 INFO:journalctl@ceph.mon.b.smithi055.stdout: 2022-09-07T21:31:40.780 INFO:journalctl@ceph.mon.b.smithi055.stdout: ** Compaction Stats [default] ** 2022-09-07T21:31:40.780 INFO:journalctl@ceph.mon.b.smithi055.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-07T21:31:40.780 INFO:journalctl@ceph.mon.b.smithi055.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-07T21:31:40.781 INFO:journalctl@ceph.mon.b.smithi055.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 2022-09-07T21:31:40.781 INFO:journalctl@ceph.mon.b.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-07T21:31:40.781 INFO:journalctl@ceph.mon.b.smithi055.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-09-07T21:31:40.782 INFO:journalctl@ceph.mon.b.smithi055.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-09-07T21:31:40.782 INFO:journalctl@ceph.mon.b.smithi055.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-09-07T21:31:40.782 INFO:journalctl@ceph.mon.b.smithi055.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-09-07T21:31:40.782 INFO:journalctl@ceph.mon.b.smithi055.stdout: AddFile(Keys): cumulative 0, interval 0 2022-09-07T21:31:40.783 INFO:journalctl@ceph.mon.b.smithi055.stdout: Cumulative compaction: 0.00 GB write, 0.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-07T21:31:40.783 INFO:journalctl@ceph.mon.b.smithi055.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-07T21:31:40.783 INFO:journalctl@ceph.mon.b.smithi055.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-09-07T21:31:40.783 INFO:journalctl@ceph.mon.b.smithi055.stdout: 2022-09-07T21:31:40.784 INFO:journalctl@ceph.mon.b.smithi055.stdout: ** File Read Latency Histogram By Level [default] ** 2022-09-07T21:31:40.784 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: using public_addr v2:172.21.15.55:0/0 -> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] 2022-09-07T21:31:40.784 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: starting mon.b rank -1 at public addrs [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] at bind addrs [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:31:40.784 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(???) e0 preinit fsid 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:31:40.785 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(synchronizing).mds e1 new map 2022-09-07T21:31:40.785 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(synchronizing).mds e1 print_map 2022-09-07T21:31:40.785 INFO:journalctl@ceph.mon.b.smithi055.stdout: e1 2022-09-07T21:31:40.785 INFO:journalctl@ceph.mon.b.smithi055.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-09-07T21:31:40.786 INFO:journalctl@ceph.mon.b.smithi055.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-09-07T21:31:40.786 INFO:journalctl@ceph.mon.b.smithi055.stdout: legacy client fscid: -1 2022-09-07T21:31:40.786 INFO:journalctl@ceph.mon.b.smithi055.stdout: 2022-09-07T21:31:40.786 INFO:journalctl@ceph.mon.b.smithi055.stdout: No filesystems configured 2022-09-07T21:31:40.787 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2022-09-07T21:31:40.787 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2022-09-07T21:31:40.787 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2022-09-07T21:31:40.787 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2022-09-07T21:31:40.788 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2022-09-07T21:31:40.788 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2022-09-07T21:31:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2022-09-07T21:31:40.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-09-07T21:31:40.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-09-07T21:31:40.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-09-07T21:31:40.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi055", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:31:40.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Deploying cephadm binary to smithi055 2022-09-07T21:31:40.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:40.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:40.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:40.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:40.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Added host smithi055 2022-09-07T21:31:40.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:40.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:31:40.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:40.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi055", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:31:40.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:40.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi059", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:31:40.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Deploying cephadm binary to smithi059 2022-09-07T21:31:40.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:40.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Added host smithi059 2022-09-07T21:31:40.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:40.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:40.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:40.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Updating smithi055:/etc/ceph/ceph.conf 2022-09-07T21:31:40.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Updating smithi055:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:31:40.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:31:40.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:40.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Updating smithi055:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:31:40.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:40.991 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:31:40.991 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:40.991 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.992 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.992 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.992 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.992 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:31:40.993 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.993 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/272893152' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-09-07T21:31:40.993 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/272893152' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-09-07T21:31:40.993 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: osdmap e4: 0 total, 0 up, 0 in 2022-09-07T21:31:40.994 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:40.994 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:40.994 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.995 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:40.995 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:40.995 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:40.995 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi047:172.21.15.47=a;smithi055:172.21.15.55=b;smithi059:172.21.15.59=c", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:31:40.996 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Saving service mon spec with placement smithi047:172.21.15.47=a;smithi055:172.21.15.55=b;smithi059:172.21.15.59=c;count:3 2022-09-07T21:31:40.996 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Updating smithi059:/etc/ceph/ceph.conf 2022-09-07T21:31:40.996 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Updating smithi059:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:31:40.996 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.996 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.997 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:31:40.997 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Updating smithi059:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:31:40.997 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:40.997 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:40.998 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:40.998 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:40.998 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Deploying daemon mon.c on smithi059 2022-09-07T21:31:40.998 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:40.998 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='client.? 172.21.15.59:0/3892081282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-07T21:31:40.999 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:31:40.999 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.a calling monitor election 2022-09-07T21:31:40.999 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:40.999 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:40.999 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:41.000 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:41.000 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.c calling monitor election 2022-09-07T21:31:41.000 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:41.000 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:41.001 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:41.001 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:41.001 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:41.001 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2022-09-07T21:31:41.002 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: monmap e2: 2 mons at {a=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],c=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2022-09-07T21:31:41.002 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: fsmap 2022-09-07T21:31:41.002 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: osdmap e4: 0 total, 0 up, 0 in 2022-09-07T21:31:41.002 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mgrmap e13: x(active, since 50s) 2022-09-07T21:31:41.002 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: overall HEALTH_OK 2022-09-07T21:31:41.003 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:41.003 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:41.003 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:41.003 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:41.004 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: Deploying daemon mon.b on smithi055 2022-09-07T21:31:41.004 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='client.? 172.21.15.59:0/2455948005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-07T21:31:41.004 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:41.004 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:41.005 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2022-09-07T21:31:41.005 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 ceph-mon[97033]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2022-09-07T21:31:41.005 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:40 smithi055 systemd[1]: Started Ceph mon.b for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:31:46.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:46.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:31:46.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:46.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: mon.a calling monitor election 2022-09-07T21:31:46.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: mon.c calling monitor election 2022-09-07T21:31:46.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:46.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:46.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:46.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:46.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:46.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:46.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:46.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:46.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2022-09-07T21:31:46.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: monmap e3: 3 mons at {a=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],b=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],c=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2022-09-07T21:31:46.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: fsmap 2022-09-07T21:31:46.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: osdmap e4: 0 total, 0 up, 0 in 2022-09-07T21:31:46.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: mgrmap e13: x(active, since 59s) 2022-09-07T21:31:46.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2022-09-07T21:31:46.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2022-09-07T21:31:46.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2022-09-07T21:31:46.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: mon.b (rank 2) addr [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] is down (out of quorum) 2022-09-07T21:31:46.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:46.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:45 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:46.086 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:46.086 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:31:46.086 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:46.086 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: mon.a calling monitor election 2022-09-07T21:31:46.087 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: mon.c calling monitor election 2022-09-07T21:31:46.087 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:46.087 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:46.087 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:46.088 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:46.088 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:46.088 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:46.089 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:46.089 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:46.089 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2022-09-07T21:31:46.089 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: monmap e3: 3 mons at {a=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],b=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],c=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2022-09-07T21:31:46.090 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: fsmap 2022-09-07T21:31:46.090 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: osdmap e4: 0 total, 0 up, 0 in 2022-09-07T21:31:46.090 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: mgrmap e13: x(active, since 59s) 2022-09-07T21:31:46.090 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2022-09-07T21:31:46.091 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2022-09-07T21:31:46.091 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2022-09-07T21:31:46.091 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: mon.b (rank 2) addr [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] is down (out of quorum) 2022-09-07T21:31:46.091 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:46.091 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:45 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:46.925 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:46.925 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:46.925 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:46.926 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:46 smithi047 ceph-mon[90087]: Updating smithi047:/etc/ceph/ceph.conf 2022-09-07T21:31:46.926 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:46 smithi047 ceph-mon[90087]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:46.926 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:46.926 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:31:46.927 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:31:46.927 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:47.217 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.214+0000 7f46ed428700 1 -- 172.21.15.59:0/948161885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46d00060a0 msgr2=0x7f46d0008500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:47.217 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.214+0000 7f46ed428700 1 --2- 172.21.15.59:0/948161885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46d00060a0 0x7f46d0008500 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f46e8066e60 tx=0x7f46dc012d70 comp rx=0 tx=0).stop 2022-09-07T21:31:47.218 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.214+0000 7f46ed428700 1 -- 172.21.15.59:0/948161885 shutdown_connections 2022-09-07T21:31:47.218 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.214+0000 7f46ed428700 1 --2- 172.21.15.59:0/948161885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46d00060a0 0x7f46d0008500 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:47.218 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.214+0000 7f46ed428700 1 -- 172.21.15.59:0/948161885 >> 172.21.15.59:0/948161885 conn(0x7f46e80f8450 msgr2=0x7f46e80fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:47.218 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.214+0000 7f46ed428700 1 -- 172.21.15.59:0/948161885 shutdown_connections 2022-09-07T21:31:47.219 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.214+0000 7f46ed428700 1 -- 172.21.15.59:0/948161885 wait complete. 2022-09-07T21:31:47.222 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.215+0000 7f46ed428700 1 Processor -- start 2022-09-07T21:31:47.222 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.215+0000 7f46ed428700 1 -- start start 2022-09-07T21:31:47.222 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.215+0000 7f46ed428700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00060a0 0x7f46e80fc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:47.223 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.215+0000 7f46ed428700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46e80fcd50 0x7f46e80fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:47.223 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.215+0000 7f46ed428700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46e819f000 0x7f46e81a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:47.223 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.215+0000 7f46ed428700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e81a1a70 con 0x7f46e819f000 2022-09-07T21:31:47.223 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.215+0000 7f46ed428700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e81a1be0 con 0x7f46e80fcd50 2022-09-07T21:31:47.224 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.215+0000 7f46ed428700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e81a1da0 con 0x7f46d00060a0 2022-09-07T21:31:47.224 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00060a0 0x7f46e80fc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:47.224 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00060a0 0x7f46e80fc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40112/0 (socket says 172.21.15.59:40112) 2022-09-07T21:31:47.224 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e6ffd700 1 -- 172.21.15.59:0/64425657 learned_addr learned my addr 172.21.15.59:0/64425657 (peer_addr_for_me v2:172.21.15.59:0/0) 2022-09-07T21:31:47.224 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e67fc700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46e80fcd50 0x7f46e80fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:47.225 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e77fe700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46e819f000 0x7f46e81a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:47.225 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e6ffd700 1 -- 172.21.15.59:0/64425657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00060a0 msgr2=0x7f46e80fc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2022-09-07T21:31:47.225 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e6ffd700 1 -- 172.21.15.59:0/64425657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00060a0 msgr2=0x7f46e80fc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-09-07T21:31:47.225 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e6ffd700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00060a0 0x7f46e80fc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-09-07T21:31:47.226 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e6ffd700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00060a0 0x7f46e80fc810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-07T21:31:47.226 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e67fc700 1 -- 172.21.15.59:0/64425657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46e80fcd50 msgr2=0x7f46e80fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2022-09-07T21:31:47.226 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e67fc700 1 -- 172.21.15.59:0/64425657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46e80fcd50 msgr2=0x7f46e80fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-09-07T21:31:47.226 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e67fc700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46e80fcd50 0x7f46e80fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-09-07T21:31:47.226 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e67fc700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46e80fcd50 0x7f46e80fd1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-07T21:31:47.227 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e77fe700 1 -- 172.21.15.59:0/64425657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00060a0 msgr2=0x7f46e80fc810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-07T21:31:47.227 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e77fe700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00060a0 0x7f46e80fc810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:47.227 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e77fe700 1 -- 172.21.15.59:0/64425657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46e80fcd50 msgr2=0x7f46e80fd1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-07T21:31:47.227 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e77fe700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46e80fcd50 0x7f46e80fd1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:47.228 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e77fe700 1 -- 172.21.15.59:0/64425657 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc0127c0 con 0x7f46e819f000 2022-09-07T21:31:47.228 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.216+0000 7f46e77fe700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46e819f000 0x7f46e81a13e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f46e000bd50 tx=0x7f46e000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:47.228 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.217+0000 7f46cffff700 1 -- 172.21.15.59:0/64425657 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e000cd10 con 0x7f46e819f000 2022-09-07T21:31:47.228 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.217+0000 7f46cffff700 1 -- 172.21.15.59:0/64425657 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f46e000ce70 con 0x7f46e819f000 2022-09-07T21:31:47.228 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.217+0000 7f46ed428700 1 -- 172.21.15.59:0/64425657 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e81a2080 con 0x7f46e819f000 2022-09-07T21:31:47.229 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.217+0000 7f46cffff700 1 -- 172.21.15.59:0/64425657 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e0012840 con 0x7f46e819f000 2022-09-07T21:31:47.229 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.217+0000 7f46ed428700 1 -- 172.21.15.59:0/64425657 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e81a2590 con 0x7f46e819f000 2022-09-07T21:31:47.229 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.217+0000 7f46cffff700 1 -- 172.21.15.59:0/64425657 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f46e001b030 con 0x7f46e819f000 2022-09-07T21:31:47.229 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.218+0000 7f46cffff700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f46d4035d90 0x7f46d4038240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:47.230 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.218+0000 7f46cffff700 1 -- 172.21.15.59:0/64425657 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f46e00480d0 con 0x7f46e819f000 2022-09-07T21:31:47.230 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.218+0000 7f46ed428700 1 -- 172.21.15.59:0/64425657 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e8060310 con 0x7f46e819f000 2022-09-07T21:31:47.230 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.218+0000 7f46e6ffd700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f46d4035d90 0x7f46d4038240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:47.230 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.218+0000 7f46e6ffd700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f46d4035d90 0x7f46d4038240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f46dc015040 tx=0x7f46dc012450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:47.231 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.224+0000 7f46cffff700 1 -- 172.21.15.59:0/64425657 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f46e0019070 con 0x7f46e819f000 2022-09-07T21:31:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:46 smithi059 ceph-mon[97450]: Updating smithi047:/etc/ceph/ceph.conf 2022-09-07T21:31:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:46 smithi059 ceph-mon[97450]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:47.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:31:47.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:31:47.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:47.510 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.507+0000 7f46ed428700 1 -- 172.21.15.59:0/64425657 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46e80fdd20 con 0x7f46e819f000 2022-09-07T21:31:47.511 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.508+0000 7f46cffff700 1 -- 172.21.15.59:0/64425657 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1240 (secure 0 0 0) 0x7f46e00116b0 con 0x7f46e819f000 2022-09-07T21:31:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-09-07T21:31:47.512 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":3,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","modified":"2022-09-07T21:31:40.708034Z","created":"2022-09-07T21:29:29.269079Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2022-09-07T21:31:47.514 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.510+0000 7f46ed428700 1 -- 172.21.15.59:0/64425657 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f46d4035d90 msgr2=0x7f46d4038240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:47.514 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.510+0000 7f46ed428700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f46d4035d90 0x7f46d4038240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f46dc015040 tx=0x7f46dc012450 comp rx=0 tx=0).stop 2022-09-07T21:31:47.515 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.510+0000 7f46ed428700 1 -- 172.21.15.59:0/64425657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46e819f000 msgr2=0x7f46e81a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:47.515 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.510+0000 7f46ed428700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46e819f000 0x7f46e81a13e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f46e000bd50 tx=0x7f46e000be30 comp rx=0 tx=0).stop 2022-09-07T21:31:47.515 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.510+0000 7f46ed428700 1 -- 172.21.15.59:0/64425657 shutdown_connections 2022-09-07T21:31:47.515 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.510+0000 7f46ed428700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f46d4035d90 0x7f46d4038240 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:47.515 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.510+0000 7f46ed428700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46e819f000 0x7f46e81a13e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:47.516 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.510+0000 7f46ed428700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00060a0 0x7f46e80fc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:47.516 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.510+0000 7f46ed428700 1 --2- 172.21.15.59:0/64425657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46e80fcd50 0x7f46e80fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:47.516 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.510+0000 7f46ed428700 1 -- 172.21.15.59:0/64425657 >> 172.21.15.59:0/64425657 conn(0x7f46e80f8450 msgr2=0x7f46e80ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:47.516 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.510+0000 7f46ed428700 1 -- 172.21.15.59:0/64425657 shutdown_connections 2022-09-07T21:31:47.517 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:47.510+0000 7f46ed428700 1 -- 172.21.15.59:0/64425657 wait complete. 2022-09-07T21:31:47.517 INFO:teuthology.orchestra.run.smithi059.stderr:dumped monmap epoch 3 2022-09-07T21:31:48.000 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:48.000 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:31:48.000 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:48.000 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: mon.a calling monitor election 2022-09-07T21:31:48.001 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: mon.c calling monitor election 2022-09-07T21:31:48.001 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:31:48.001 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:48.001 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:48.002 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:48.002 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:48.002 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:48.002 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:48.003 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:48.003 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2022-09-07T21:31:48.003 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: monmap e3: 3 mons at {a=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],b=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],c=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2022-09-07T21:31:48.003 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: fsmap 2022-09-07T21:31:48.004 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: osdmap e4: 0 total, 0 up, 0 in 2022-09-07T21:31:48.004 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: mgrmap e13: x(active, since 59s) 2022-09-07T21:31:48.004 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2022-09-07T21:31:48.004 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2022-09-07T21:31:48.005 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2022-09-07T21:31:48.005 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: mon.b (rank 2) addr [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] is down (out of quorum) 2022-09-07T21:31:48.005 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:48.005 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:48.005 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:48.006 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:48.006 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:48.006 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: Updating smithi047:/etc/ceph/ceph.conf 2022-09-07T21:31:48.006 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:48.007 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:48.007 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:31:48.007 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:31:48.008 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:48.071 INFO:tasks.cephadm:Generating final ceph.conf file... 2022-09-07T21:31:48.072 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph config generate-minimal-conf 2022-09-07T21:31:48.894 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:31:48.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: mon.b calling monitor election 2022-09-07T21:31:48.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: mon.b calling monitor election 2022-09-07T21:31:48.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: mon.c calling monitor election 2022-09-07T21:31:48.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: mon.a calling monitor election 2022-09-07T21:31:48.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-09-07T21:31:48.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: monmap e3: 3 mons at {a=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],b=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],c=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2022-09-07T21:31:48.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: fsmap 2022-09-07T21:31:48.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: osdmap e4: 0 total, 0 up, 0 in 2022-09-07T21:31:48.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: mgrmap e13: x(active, since 61s) 2022-09-07T21:31:48.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2022-09-07T21:31:48.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: Cluster is now healthy 2022-09-07T21:31:48.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: overall HEALTH_OK 2022-09-07T21:31:48.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:48.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:48.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:48.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:48.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:48.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:48.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-07T21:31:48.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: mon.b calling monitor election 2022-09-07T21:31:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: mon.b calling monitor election 2022-09-07T21:31:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: mon.c calling monitor election 2022-09-07T21:31:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: mon.a calling monitor election 2022-09-07T21:31:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-09-07T21:31:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: monmap e3: 3 mons at {a=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],b=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],c=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2022-09-07T21:31:48.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: fsmap 2022-09-07T21:31:48.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: osdmap e4: 0 total, 0 up, 0 in 2022-09-07T21:31:48.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: mgrmap e13: x(active, since 61s) 2022-09-07T21:31:48.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2022-09-07T21:31:48.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: Cluster is now healthy 2022-09-07T21:31:48.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: overall HEALTH_OK 2022-09-07T21:31:48.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:48.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:48.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:48.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:48.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:48.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:48.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-07T21:31:48.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: mon.b calling monitor election 2022-09-07T21:31:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: mon.b calling monitor election 2022-09-07T21:31:49.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: mon.c calling monitor election 2022-09-07T21:31:49.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: mon.a calling monitor election 2022-09-07T21:31:49.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-09-07T21:31:49.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: monmap e3: 3 mons at {a=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],b=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],c=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2022-09-07T21:31:49.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: fsmap 2022-09-07T21:31:49.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: osdmap e4: 0 total, 0 up, 0 in 2022-09-07T21:31:49.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: mgrmap e13: x(active, since 61s) 2022-09-07T21:31:49.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2022-09-07T21:31:49.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: Cluster is now healthy 2022-09-07T21:31:49.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: overall HEALTH_OK 2022-09-07T21:31:49.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:49.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:49.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:49.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:49.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:49.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:49.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-07T21:31:49.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:49.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:31:49.708+0000 7fbd5fee0700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-09-07T21:31:49.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:49 smithi047 ceph-mon[90087]: Reconfiguring mon.a (unknown last config time)... 2022-09-07T21:31:49.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:49 smithi047 ceph-mon[90087]: Reconfiguring daemon mon.a on smithi047 2022-09-07T21:31:49.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:49 smithi047 ceph-mon[90087]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:49.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:49 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:49 smithi055 ceph-mon[97033]: Reconfiguring mon.a (unknown last config time)... 2022-09-07T21:31:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:49 smithi055 ceph-mon[97033]: Reconfiguring daemon mon.a on smithi047 2022-09-07T21:31:49.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:49 smithi055 ceph-mon[97033]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:49.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:49 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:49 smithi059 ceph-mon[97450]: Reconfiguring mon.a (unknown last config time)... 2022-09-07T21:31:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:49 smithi059 ceph-mon[97450]: Reconfiguring daemon mon.a on smithi047 2022-09-07T21:31:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:49 smithi059 ceph-mon[97450]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:49 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:31:51.518 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.515+0000 7f33f1437700 1 -- 172.21.15.47:0/1868254394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33ec0fea70 msgr2=0x7f33ec102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:51.519 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.515+0000 7f33f1437700 1 --2- 172.21.15.47:0/1868254394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33ec0fea70 0x7f33ec102ac0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009840 tx=0x7f33e0009b50 comp rx=0 tx=0).stop 2022-09-07T21:31:51.519 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.515+0000 7f33f1437700 1 -- 172.21.15.47:0/1868254394 shutdown_connections 2022-09-07T21:31:51.519 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.515+0000 7f33f1437700 1 --2- 172.21.15.47:0/1868254394 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33ec1031d0 0x7f33ec10ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:51.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.515+0000 7f33f1437700 1 --2- 172.21.15.47:0/1868254394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33ec0fe0d0 0x7f33ec0fe4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:51.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.515+0000 7f33f1437700 1 --2- 172.21.15.47:0/1868254394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33ec0fea70 0x7f33ec102ac0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:51.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.515+0000 7f33f1437700 1 -- 172.21.15.47:0/1868254394 >> 172.21.15.47:0/1868254394 conn(0x7f33ec0f9a10 msgr2=0x7f33ec0fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:51.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.515+0000 7f33f1437700 1 -- 172.21.15.47:0/1868254394 shutdown_connections 2022-09-07T21:31:51.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.515+0000 7f33f1437700 1 -- 172.21.15.47:0/1868254394 wait complete. 2022-09-07T21:31:51.524 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.516+0000 7f33f1437700 1 Processor -- start 2022-09-07T21:31:51.524 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.516+0000 7f33f1437700 1 -- start start 2022-09-07T21:31:51.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.517+0000 7f33f1437700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33ec0fe0d0 0x7f33ec070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:51.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.517+0000 7f33f1437700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33ec0fea70 0x7f33ec071520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:51.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.517+0000 7f33f1437700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33ec1031d0 0x7f33ec1a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:51.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.517+0000 7f33f1437700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec071db0 con 0x7f33ec0fe0d0 2022-09-07T21:31:51.526 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.517+0000 7f33f1437700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec1a2450 con 0x7f33ec1031d0 2022-09-07T21:31:51.526 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.517+0000 7f33f1437700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec1a2590 con 0x7f33ec0fea70 2022-09-07T21:31:51.526 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.517+0000 7f33eaffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33ec0fe0d0 0x7f33ec070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:51.526 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.517+0000 7f33ea7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33ec0fea70 0x7f33ec071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:51.527 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.517+0000 7f33eaffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33ec0fe0d0 0x7f33ec070fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:37706/0 (socket says 172.21.15.47:37706) 2022-09-07T21:31:51.527 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.517+0000 7f33eaffd700 1 -- 172.21.15.47:0/3885755507 learned_addr learned my addr 172.21.15.47:0/3885755507 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:31:51.527 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.518+0000 7f33eb7fe700 1 --2- 172.21.15.47:0/3885755507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33ec1031d0 0x7f33ec1a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:51.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.518+0000 7f33ea7fc700 1 -- 172.21.15.47:0/3885755507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33ec1031d0 msgr2=0x7f33ec1a1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:51.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.518+0000 7f33ea7fc700 1 --2- 172.21.15.47:0/3885755507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33ec1031d0 0x7f33ec1a1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:51.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.518+0000 7f33ea7fc700 1 -- 172.21.15.47:0/3885755507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33ec0fe0d0 msgr2=0x7f33ec070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:51.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.518+0000 7f33ea7fc700 1 --2- 172.21.15.47:0/3885755507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33ec0fe0d0 0x7f33ec070fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:51.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.518+0000 7f33ea7fc700 1 -- 172.21.15.47:0/3885755507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e4009730 con 0x7f33ec0fea70 2022-09-07T21:31:51.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.518+0000 7f33eaffd700 1 --2- 172.21.15.47:0/3885755507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33ec0fe0d0 0x7f33ec070fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:31:51.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.518+0000 7f33ea7fc700 1 --2- 172.21.15.47:0/3885755507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33ec0fea70 0x7f33ec071520 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009760 tx=0x7f33e000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:51.530 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.518+0000 7f33d3fff700 1 -- 172.21.15.47:0/3885755507 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e001d070 con 0x7f33ec0fea70 2022-09-07T21:31:51.530 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.518+0000 7f33d3fff700 1 -- 172.21.15.47:0/3885755507 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f33e0005880 con 0x7f33ec0fea70 2022-09-07T21:31:51.530 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.518+0000 7f33f1437700 1 -- 172.21.15.47:0/3885755507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e0009470 con 0x7f33ec0fea70 2022-09-07T21:31:51.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.519+0000 7f33d3fff700 1 -- 172.21.15.47:0/3885755507 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e0004700 con 0x7f33ec0fea70 2022-09-07T21:31:51.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.519+0000 7f33f1437700 1 -- 172.21.15.47:0/3885755507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ec1a2b30 con 0x7f33ec0fea70 2022-09-07T21:31:51.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.520+0000 7f33d3fff700 1 -- 172.21.15.47:0/3885755507 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f33e0028030 con 0x7f33ec0fea70 2022-09-07T21:31:51.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.520+0000 7f33f1437700 1 -- 172.21.15.47:0/3885755507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33ec064ea0 con 0x7f33ec0fea70 2022-09-07T21:31:51.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.520+0000 7f33d3fff700 1 --2- 172.21.15.47:0/3885755507 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f33d4035b00 0x7f33d4037fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:51.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.520+0000 7f33d3fff700 1 -- 172.21.15.47:0/3885755507 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f33e0047fe0 con 0x7f33ec0fea70 2022-09-07T21:31:51.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.521+0000 7f33eaffd700 1 --2- 172.21.15.47:0/3885755507 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f33d4035b00 0x7f33d4037fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:51.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.521+0000 7f33eaffd700 1 --2- 172.21.15.47:0/3885755507 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f33d4035b00 0x7f33d4037fb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f33dc0057d0 tx=0x7f33dc005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:51.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.526+0000 7f33d3fff700 1 -- 172.21.15.47:0/3885755507 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f33e0029540 con 0x7f33ec0fea70 2022-09-07T21:31:51.588 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:51.588 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:51.588 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:51 smithi055 ceph-mon[97033]: Reconfiguring mon.b (monmap changed)... 2022-09-07T21:31:51.588 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:51.589 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-07T21:31:51.589 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:51.589 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:51 smithi055 ceph-mon[97033]: Reconfiguring daemon mon.b on smithi055 2022-09-07T21:31:51.589 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:51 smithi055 ceph-mon[97033]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:51.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:51.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:51.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:51 smithi047 ceph-mon[90087]: Reconfiguring mon.b (monmap changed)... 2022-09-07T21:31:51.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:51.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-07T21:31:51.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:51.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:51 smithi047 ceph-mon[90087]: Reconfiguring daemon mon.b on smithi055 2022-09-07T21:31:51.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:51 smithi047 ceph-mon[90087]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:51.688 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:51.688 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:51.688 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:51 smithi059 ceph-mon[97450]: Reconfiguring mon.b (monmap changed)... 2022-09-07T21:31:51.688 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:51.689 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-07T21:31:51.689 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:51.689 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:51 smithi059 ceph-mon[97450]: Reconfiguring daemon mon.b on smithi055 2022-09-07T21:31:51.689 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:51 smithi059 ceph-mon[97450]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:51.758 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.755+0000 7f33f1437700 1 -- 172.21.15.47:0/3885755507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f33ec10dce0 con 0x7f33ec0fea70 2022-09-07T21:31:51.758 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.756+0000 7f33d3fff700 1 -- 172.21.15.47:0/3885755507 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+271 (secure 0 0 0) 0x7f33e0026020 con 0x7f33ec0fea70 2022-09-07T21:31:51.759 INFO:teuthology.orchestra.run.smithi047.stdout:# minimal ceph.conf for 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:31:51.759 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2022-09-07T21:31:51.759 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:31:51.760 INFO:teuthology.orchestra.run.smithi047.stdout: mon_host = [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2022-09-07T21:31:51.761 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.758+0000 7f33f1437700 1 -- 172.21.15.47:0/3885755507 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f33d4035b00 msgr2=0x7f33d4037fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:51.761 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.758+0000 7f33f1437700 1 --2- 172.21.15.47:0/3885755507 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f33d4035b00 0x7f33d4037fb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f33dc0057d0 tx=0x7f33dc005ea0 comp rx=0 tx=0).stop 2022-09-07T21:31:51.761 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.758+0000 7f33f1437700 1 -- 172.21.15.47:0/3885755507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33ec0fea70 msgr2=0x7f33ec071520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:51.761 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.758+0000 7f33f1437700 1 --2- 172.21.15.47:0/3885755507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33ec0fea70 0x7f33ec071520 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009760 tx=0x7f33e000f690 comp rx=0 tx=0).stop 2022-09-07T21:31:51.762 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.758+0000 7f33f1437700 1 -- 172.21.15.47:0/3885755507 shutdown_connections 2022-09-07T21:31:51.762 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.758+0000 7f33f1437700 1 --2- 172.21.15.47:0/3885755507 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f33d4035b00 0x7f33d4037fb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:51.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.758+0000 7f33f1437700 1 --2- 172.21.15.47:0/3885755507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33ec1031d0 0x7f33ec1a1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:51.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.758+0000 7f33f1437700 1 --2- 172.21.15.47:0/3885755507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33ec0fe0d0 0x7f33ec070fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:51.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.758+0000 7f33f1437700 1 --2- 172.21.15.47:0/3885755507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33ec0fea70 0x7f33ec071520 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:51.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.758+0000 7f33f1437700 1 -- 172.21.15.47:0/3885755507 >> 172.21.15.47:0/3885755507 conn(0x7f33ec0f9a10 msgr2=0x7f33ec0fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:51.764 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.758+0000 7f33f1437700 1 -- 172.21.15.47:0/3885755507 shutdown_connections 2022-09-07T21:31:51.764 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:31:51.759+0000 7f33f1437700 1 -- 172.21.15.47:0/3885755507 wait complete. 2022-09-07T21:31:52.269 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2022-09-07T21:31:52.270 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:31:52.271 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.conf 2022-09-07T21:31:52.364 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:31:52.365 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:31:52.493 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2022-09-07T21:31:52.494 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.conf 2022-09-07T21:31:52.531 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2022-09-07T21:31:52.531 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:31:52.602 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-09-07T21:31:52.603 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.conf 2022-09-07T21:31:52.635 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-09-07T21:31:52.636 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:31:52.712 INFO:tasks.cephadm:Adding mgr.x on smithi047 2022-09-07T21:31:52.713 INFO:tasks.cephadm:Adding mgr.y on smithi055 2022-09-07T21:31:52.713 INFO:tasks.cephadm:Adding mgr.z on smithi059 2022-09-07T21:31:52.714 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch apply mgr '3;smithi047=x;smithi055=y;smithi059=z' 2022-09-07T21:31:52.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:52 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:52.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:52 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:52.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:52 smithi047 ceph-mon[90087]: Reconfiguring mon.c (monmap changed)... 2022-09-07T21:31:52.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:52 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:52.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:52 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-07T21:31:52.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:52 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:52.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:52 smithi047 ceph-mon[90087]: Reconfiguring daemon mon.c on smithi059 2022-09-07T21:31:52.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:52 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3885755507' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:52.935 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:52 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:52.935 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:52 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:52.936 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:52 smithi059 ceph-mon[97450]: Reconfiguring mon.c (monmap changed)... 2022-09-07T21:31:52.936 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:52 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:52.938 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:52 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-07T21:31:52.938 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:52 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:52.938 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:52 smithi059 ceph-mon[97450]: Reconfiguring daemon mon.c on smithi059 2022-09-07T21:31:52.938 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:52 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3885755507' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:52 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:52 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:52 smithi055 ceph-mon[97033]: Reconfiguring mon.c (monmap changed)... 2022-09-07T21:31:52.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:52 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-07T21:31:52.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:52 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-07T21:31:52.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:52 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:52.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:52 smithi055 ceph-mon[97033]: Reconfiguring daemon mon.c on smithi059 2022-09-07T21:31:52.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:52 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3885755507' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:53.348 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.c/config 2022-09-07T21:31:53.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:53 smithi059 ceph-mon[97450]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:53.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:53.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:53.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:53.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:53.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:53.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:53 smithi047 ceph-mon[90087]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:53.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:53.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:53.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:53.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:53.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:53 smithi055 ceph-mon[97033]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:53.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:53.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:53.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:53.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:53.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:54.534 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.526+0000 7f60254e9700 1 -- 172.21.15.59:0/1947881811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60200fe110 msgr2=0x7f60201049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:54.535 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.526+0000 7f60254e9700 1 --2- 172.21.15.59:0/1947881811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60200fe110 0x7f60201049d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6018009bb0 tx=0x7f6018009ec0 comp rx=0 tx=0).stop 2022-09-07T21:31:54.536 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.526+0000 7f60254e9700 1 -- 172.21.15.59:0/1947881811 shutdown_connections 2022-09-07T21:31:54.536 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.526+0000 7f60254e9700 1 --2- 172.21.15.59:0/1947881811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60200fe110 0x7f60201049d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:54.536 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.526+0000 7f60254e9700 1 --2- 172.21.15.59:0/1947881811 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60200fc500 0x7f60200fc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:54.537 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.526+0000 7f60254e9700 1 --2- 172.21.15.59:0/1947881811 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60200fd780 0x7f60200fdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:54.537 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.526+0000 7f60254e9700 1 -- 172.21.15.59:0/1947881811 >> 172.21.15.59:0/1947881811 conn(0x7f60200f7bf0 msgr2=0x7f60200fa020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:54.538 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.526+0000 7f60254e9700 1 -- 172.21.15.59:0/1947881811 shutdown_connections 2022-09-07T21:31:54.538 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.526+0000 7f60254e9700 1 -- 172.21.15.59:0/1947881811 wait complete. 2022-09-07T21:31:54.539 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.527+0000 7f60254e9700 1 Processor -- start 2022-09-07T21:31:54.539 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.527+0000 7f60254e9700 1 -- start start 2022-09-07T21:31:54.540 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f60254e9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60200fc500 0x7f60200fa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:54.540 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f60254e9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60200fd780 0x7f6020198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:54.542 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f60254e9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60200fe110 0x7f602019ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:54.542 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f60254e9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602019b5a0 con 0x7f60200fc500 2022-09-07T21:31:54.542 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f60254e9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602019b710 con 0x7f60200fe110 2022-09-07T21:31:54.543 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f60254e9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602019b8d0 con 0x7f60200fd780 2022-09-07T21:31:54.543 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f601e7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60200fd780 0x7f6020198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:54.543 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f601e7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60200fd780 0x7f6020198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40126/0 (socket says 172.21.15.59:40126) 2022-09-07T21:31:54.543 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f601e7fc700 1 -- 172.21.15.59:0/234639575 learned_addr learned my addr 172.21.15.59:0/234639575 (peer_addr_for_me v2:172.21.15.59:0/0) 2022-09-07T21:31:54.544 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f601e7fc700 1 -- 172.21.15.59:0/234639575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60200fe110 msgr2=0x7f602019ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:54.544 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f601e7fc700 1 --2- 172.21.15.59:0/234639575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60200fe110 0x7f602019ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:54.545 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f601e7fc700 1 -- 172.21.15.59:0/234639575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60200fc500 msgr2=0x7f60200fa0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:54.545 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f601effd700 1 --2- 172.21.15.59:0/234639575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60200fc500 0x7f60200fa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:54.545 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f601e7fc700 1 --2- 172.21.15.59:0/234639575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60200fc500 0x7f60200fa0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f601e7fc700 1 -- 172.21.15.59:0/234639575 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60180097e0 con 0x7f60200fd780 2022-09-07T21:31:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f601effd700 1 --2- 172.21.15.59:0/234639575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60200fc500 0x7f60200fa0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-07T21:31:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.528+0000 7f601e7fc700 1 --2- 172.21.15.59:0/234639575 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60200fd780 0x7f6020198700 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f601400d700 tx=0x7f601400dac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.529+0000 7f6003fff700 1 -- 172.21.15.59:0/234639575 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601400f890 con 0x7f60200fd780 2022-09-07T21:31:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.529+0000 7f60254e9700 1 -- 172.21.15.59:0/234639575 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602019bbb0 con 0x7f60200fd780 2022-09-07T21:31:54.547 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.529+0000 7f6003fff700 1 -- 172.21.15.59:0/234639575 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f601400fe90 con 0x7f60200fd780 2022-09-07T21:31:54.547 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.529+0000 7f6003fff700 1 -- 172.21.15.59:0/234639575 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601401a3e0 con 0x7f60200fd780 2022-09-07T21:31:54.547 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.529+0000 7f60254e9700 1 -- 172.21.15.59:0/234639575 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602019c0c0 con 0x7f60200fd780 2022-09-07T21:31:54.547 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.530+0000 7f6003fff700 1 -- 172.21.15.59:0/234639575 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f601400c040 con 0x7f60200fd780 2022-09-07T21:31:54.548 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.530+0000 7f6003fff700 1 --2- 172.21.15.59:0/234639575 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6008035b00 0x7f6008037fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:31:54.548 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.530+0000 7f6003fff700 1 -- 172.21.15.59:0/234639575 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6014016030 con 0x7f60200fd780 2022-09-07T21:31:54.548 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.530+0000 7f60254e9700 1 -- 172.21.15.59:0/234639575 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6020107cc0 con 0x7f60200fd780 2022-09-07T21:31:54.548 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.530+0000 7f601effd700 1 --2- 172.21.15.59:0/234639575 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6008035b00 0x7f6008037fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:31:54.548 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.531+0000 7f601effd700 1 --2- 172.21.15.59:0/234639575 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6008035b00 0x7f6008037fb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f60200fd5e0 tx=0x7f6010005c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:31:54.549 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.535+0000 7f6003fff700 1 -- 172.21.15.59:0/234639575 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f60140188d0 con 0x7f60200fd780 2022-09-07T21:31:54.745 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.743+0000 7f60254e9700 1 -- 172.21.15.59:0/234639575 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi047=x;smithi055=y;smithi059=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f602005fb70 con 0x7f6008035b00 2022-09-07T21:31:54.753 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.750+0000 7f6003fff700 1 -- 172.21.15.59:0/234639575 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f602005fb70 con 0x7f6008035b00 2022-09-07T21:31:54.753 INFO:teuthology.orchestra.run.smithi059.stdout:Scheduled mgr update... 2022-09-07T21:31:54.755 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.752+0000 7f60254e9700 1 -- 172.21.15.59:0/234639575 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6008035b00 msgr2=0x7f6008037fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:54.756 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.752+0000 7f60254e9700 1 --2- 172.21.15.59:0/234639575 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6008035b00 0x7f6008037fb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f60200fd5e0 tx=0x7f6010005c40 comp rx=0 tx=0).stop 2022-09-07T21:31:54.756 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.752+0000 7f60254e9700 1 -- 172.21.15.59:0/234639575 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60200fd780 msgr2=0x7f6020198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:31:54.756 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.752+0000 7f60254e9700 1 --2- 172.21.15.59:0/234639575 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60200fd780 0x7f6020198700 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f601400d700 tx=0x7f601400dac0 comp rx=0 tx=0).stop 2022-09-07T21:31:54.757 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.752+0000 7f60254e9700 1 -- 172.21.15.59:0/234639575 shutdown_connections 2022-09-07T21:31:54.757 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.752+0000 7f60254e9700 1 --2- 172.21.15.59:0/234639575 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6008035b00 0x7f6008037fb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:54.757 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.752+0000 7f60254e9700 1 --2- 172.21.15.59:0/234639575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60200fe110 0x7f602019ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:54.757 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.752+0000 7f60254e9700 1 --2- 172.21.15.59:0/234639575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60200fc500 0x7f60200fa0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:54.758 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.752+0000 7f60254e9700 1 --2- 172.21.15.59:0/234639575 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60200fd780 0x7f6020198700 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:31:54.758 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.752+0000 7f60254e9700 1 -- 172.21.15.59:0/234639575 >> 172.21.15.59:0/234639575 conn(0x7f60200f7bf0 msgr2=0x7f6020101220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:31:54.758 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.753+0000 7f60254e9700 1 -- 172.21.15.59:0/234639575 shutdown_connections 2022-09-07T21:31:54.758 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:31:54.753+0000 7f60254e9700 1 -- 172.21.15.59:0/234639575 wait complete. 2022-09-07T21:31:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:54 smithi059 ceph-mon[97450]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:55.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-07T21:31:55.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-09-07T21:31:55.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-07T21:31:55.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:55.186 DEBUG:teuthology.orchestra.run.smithi055:mgr.y> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mgr.y.service 2022-09-07T21:31:55.190 DEBUG:teuthology.orchestra.run.smithi059:mgr.z> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mgr.z.service 2022-09-07T21:31:55.193 INFO:tasks.cephadm:Deploying OSDs... 2022-09-07T21:31:55.193 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:31:55.194 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/scratch_devs of=/dev/stdout 2022-09-07T21:31:55.222 INFO:journalctl@ceph.mgr.y.smithi055.stdout:-- Logs begin at Wed 2022-09-07 21:07:35 UTC. -- 2022-09-07T21:31:55.223 INFO:journalctl@ceph.mgr.z.smithi059.stdout:-- Logs begin at Wed 2022-09-07 21:07:37 UTC. -- 2022-09-07T21:31:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:54 smithi055 ceph-mon[97033]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:55.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:55.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:55.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:55.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:55.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-07T21:31:55.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-09-07T21:31:55.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-07T21:31:55.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:55.256 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-09-07T21:31:55.257 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-09-07T21:31:55.317 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-09-07T21:31:55.318 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-07T21:31:55.318 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 448 Links: 1 2022-09-07T21:31:55.318 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-07T21:31:55.318 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2022-09-07T21:31:55.319 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2022-09-07 21:30:57.336227784 +0000 2022-09-07T21:31:55.319 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2022-09-07 21:23:25.649003019 +0000 2022-09-07T21:31:55.319 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2022-09-07 21:23:25.649003019 +0000 2022-09-07T21:31:55.319 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2022-09-07 21:23:25.615003721 +0000 2022-09-07T21:31:55.340 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:55.340 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:54 smithi047 ceph-mon[90087]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:55.341 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:55.341 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:31:55.341 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:55.341 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:31:55.342 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:55.342 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-07T21:31:55.342 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-09-07T21:31:55.342 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-07T21:31:55.343 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:55.348 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2022-09-07T21:31:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2022-09-07T21:31:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000326931 s, 1.6 MB/s 2022-09-07T21:31:55.359 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-09-07T21:31:55.418 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-09-07T21:31:55.419 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-07T21:31:55.419 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 454 Links: 1 2022-09-07T21:31:55.420 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-07T21:31:55.420 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2022-09-07T21:31:55.420 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2022-09-07 21:30:57.336227784 +0000 2022-09-07T21:31:55.420 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2022-09-07 21:23:26.709981106 +0000 2022-09-07T21:31:55.420 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2022-09-07 21:23:26.709981106 +0000 2022-09-07T21:31:55.421 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2022-09-07 21:23:26.680981705 +0000 2022-09-07T21:31:55.449 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2022-09-07T21:31:55.449 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2022-09-07T21:31:55.450 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000297429 s, 1.7 MB/s 2022-09-07T21:31:55.458 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-09-07T21:31:55.518 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-09-07T21:31:55.519 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-07T21:31:55.519 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 460 Links: 1 2022-09-07T21:31:55.519 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-07T21:31:55.519 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2022-09-07T21:31:55.520 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2022-09-07 21:30:57.336227784 +0000 2022-09-07T21:31:55.520 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2022-09-07 21:23:27.756959481 +0000 2022-09-07T21:31:55.520 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2022-09-07 21:23:27.756959481 +0000 2022-09-07T21:31:55.520 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2022-09-07 21:23:27.726960101 +0000 2022-09-07T21:31:55.548 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2022-09-07T21:31:55.549 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2022-09-07T21:31:55.549 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.00029921 s, 1.7 MB/s 2022-09-07T21:31:55.558 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-09-07T21:31:55.619 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-09-07T21:31:55.620 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-07T21:31:55.620 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 466 Links: 1 2022-09-07T21:31:55.621 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-07T21:31:55.621 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2022-09-07T21:31:55.621 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2022-09-07 21:30:57.336227784 +0000 2022-09-07T21:31:55.621 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2022-09-07 21:23:28.817937567 +0000 2022-09-07T21:31:55.622 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2022-09-07 21:23:28.817937567 +0000 2022-09-07T21:31:55.622 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2022-09-07 21:23:28.788938166 +0000 2022-09-07T21:31:55.647 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2022-09-07T21:31:55.648 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2022-09-07T21:31:55.648 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000333545 s, 1.5 MB/s 2022-09-07T21:31:55.657 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2022-09-07T21:31:55.658 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/scratch_devs of=/dev/stdout 2022-09-07T21:31:55.681 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-09-07T21:31:55.682 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-09-07T21:31:55.742 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-09-07T21:31:55.743 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-07T21:31:55.743 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 6h/6d Inode: 354851 Links: 1 2022-09-07T21:31:55.744 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-07T21:31:55.744 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2022-09-07T21:31:55.744 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2022-09-07 21:31:39.969111087 +0000 2022-09-07T21:31:55.744 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2022-09-07 21:19:43.931835075 +0000 2022-09-07T21:31:55.745 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2022-09-07 21:19:43.931835075 +0000 2022-09-07T21:31:55.745 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: - 2022-09-07T21:31:55.755 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2022-09-07T21:31:55.756 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2022-09-07T21:31:55.756 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000394261 s, 1.3 MB/s 2022-09-07T21:31:55.759 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-09-07T21:31:55.820 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-09-07T21:31:55.821 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-07T21:31:55.821 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 6h/6d Inode: 356735 Links: 1 2022-09-07T21:31:55.821 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-07T21:31:55.821 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2022-09-07T21:31:55.822 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2022-09-07 21:31:39.969111087 +0000 2022-09-07T21:31:55.822 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2022-09-07 21:19:44.495823673 +0000 2022-09-07T21:31:55.822 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2022-09-07 21:19:44.495823673 +0000 2022-09-07T21:31:55.822 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: - 2022-09-07T21:31:55.832 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2022-09-07T21:31:55.833 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2022-09-07T21:31:55.833 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000349752 s, 1.5 MB/s 2022-09-07T21:31:55.836 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-09-07T21:31:55.896 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-09-07T21:31:55.897 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-07T21:31:55.897 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 6h/6d Inode: 360653 Links: 1 2022-09-07T21:31:55.898 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-07T21:31:55.898 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2022-09-07T21:31:55.899 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2022-09-07 21:31:39.969111087 +0000 2022-09-07T21:31:55.899 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2022-09-07 21:19:45.093811584 +0000 2022-09-07T21:31:55.899 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2022-09-07 21:19:45.093811584 +0000 2022-09-07T21:31:55.900 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: - 2022-09-07T21:31:55.908 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2022-09-07T21:31:55.908 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2022-09-07T21:31:55.908 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000302385 s, 1.7 MB/s 2022-09-07T21:31:55.911 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-09-07T21:31:55.969 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-09-07T21:31:55.970 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-07T21:31:55.970 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 6h/6d Inode: 360372 Links: 1 2022-09-07T21:31:55.970 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-07T21:31:55.970 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2022-09-07T21:31:55.971 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2022-09-07 21:31:39.969111087 +0000 2022-09-07T21:31:55.971 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2022-09-07 21:19:45.659800142 +0000 2022-09-07T21:31:55.971 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2022-09-07 21:19:45.659800142 +0000 2022-09-07T21:31:55.971 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: - 2022-09-07T21:31:55.981 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2022-09-07T21:31:55.982 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2022-09-07T21:31:55.982 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000351414 s, 1.5 MB/s 2022-09-07T21:31:55.985 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-09-07T21:31:55.986 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2022-09-07T21:31:56.006 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-09-07T21:31:56.007 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-09-07T21:31:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:55 smithi059 ceph-mon[97450]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi047=x;smithi055=y;smithi059=z", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:31:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:55 smithi059 ceph-mon[97450]: Saving service mgr spec with placement smithi047=x;smithi055=y;smithi059=z;count:3 2022-09-07T21:31:56.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:55 smithi059 ceph-mon[97450]: Deploying daemon mgr.z on smithi059 2022-09-07T21:31:56.051 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-09-07T21:31:56.052 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-07T21:31:56.052 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 6h/6d Inode: 357739 Links: 1 2022-09-07T21:31:56.052 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-07T21:31:56.053 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2022-09-07T21:31:56.053 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2022-09-07 21:31:31.703743303 +0000 2022-09-07T21:31:56.053 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2022-09-07 21:19:43.936342637 +0000 2022-09-07T21:31:56.053 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2022-09-07 21:19:43.936342637 +0000 2022-09-07T21:31:56.053 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2022-09-07T21:31:56.063 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2022-09-07T21:31:56.064 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2022-09-07T21:31:56.064 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000207975 s, 2.5 MB/s 2022-09-07T21:31:56.067 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-09-07T21:31:56.126 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-09-07T21:31:56.126 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-07T21:31:56.127 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 6h/6d Inode: 354877 Links: 1 2022-09-07T21:31:56.127 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-07T21:31:56.127 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2022-09-07T21:31:56.127 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2022-09-07 21:31:31.703743303 +0000 2022-09-07T21:31:56.128 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2022-09-07 21:19:44.497335379 +0000 2022-09-07T21:31:56.128 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2022-09-07 21:19:44.497335379 +0000 2022-09-07T21:31:56.128 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2022-09-07T21:31:56.137 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2022-09-07T21:31:56.138 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2022-09-07T21:31:56.138 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000286315 s, 1.8 MB/s 2022-09-07T21:31:56.140 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-09-07T21:31:56.200 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-09-07T21:31:56.200 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-07T21:31:56.201 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 6h/6d Inode: 355863 Links: 1 2022-09-07T21:31:56.201 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-07T21:31:56.201 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2022-09-07T21:31:56.202 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2022-09-07 21:31:31.703743303 +0000 2022-09-07T21:31:56.203 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2022-09-07 21:19:45.066328017 +0000 2022-09-07T21:31:56.203 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2022-09-07 21:19:45.066328017 +0000 2022-09-07T21:31:56.203 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2022-09-07T21:31:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:55 smithi055 ceph-mon[97033]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi047=x;smithi055=y;smithi059=z", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:31:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:55 smithi055 ceph-mon[97033]: Saving service mgr spec with placement smithi047=x;smithi055=y;smithi059=z;count:3 2022-09-07T21:31:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:55 smithi055 ceph-mon[97033]: Deploying daemon mgr.z on smithi059 2022-09-07T21:31:56.299 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2022-09-07T21:31:56.299 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2022-09-07T21:31:56.299 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.00020077 s, 2.6 MB/s 2022-09-07T21:31:56.302 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-09-07T21:31:56.370 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-09-07T21:31:56.371 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-07T21:31:56.371 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 6h/6d Inode: 360702 Links: 1 2022-09-07T21:31:56.372 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-07T21:31:56.372 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2022-09-07T21:31:56.373 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2022-09-07 21:31:56.239429184 +0000 2022-09-07T21:31:56.373 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2022-09-07 21:19:45.607321018 +0000 2022-09-07T21:31:56.374 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2022-09-07 21:19:45.607321018 +0000 2022-09-07T21:31:56.374 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2022-09-07T21:31:56.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:55 smithi047 ceph-mon[90087]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi047=x;smithi055=y;smithi059=z", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:31:56.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:55 smithi047 ceph-mon[90087]: Saving service mgr spec with placement smithi047=x;smithi055=y;smithi059=z;count:3 2022-09-07T21:31:56.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:55 smithi047 ceph-mon[90087]: Deploying daemon mgr.z on smithi059 2022-09-07T21:31:56.437 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2022-09-07T21:31:56.438 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2022-09-07T21:31:56.438 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000206534 s, 2.5 MB/s 2022-09-07T21:31:56.441 INFO:tasks.cephadm:Deploying osd.0 on smithi047 with /dev/vg_nvme/lv_4... 2022-09-07T21:31:56.441 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-09-07T21:31:56.744 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:56 smithi059 systemd[1]: Starting Ceph mgr.z for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:31:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:56 smithi059 ceph-mon[97450]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:57.032 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:56 smithi059 podman[99491]: 2022-09-07T21:31:57.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:56 smithi055 ceph-mon[97033]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:31:57.399 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:57 smithi059 bash[99491]: 85ad55a4092ca2e5f3d62884db385b08dd1014f7356722751e13bdcbfead03d1 2022-09-07T21:31:57.400 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:57 smithi059 ceph-mgr[99530]: -- 172.21.15.59:0/1530265668 <== mon.1 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a465f5a820 con 0x55a465f2f000 2022-09-07T21:31:57.400 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:57 smithi059 systemd[1]: Started Ceph mgr.z for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:31:57.401 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:57 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:31:57.198+0000 7fc0cae75080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-07T21:31:57.401 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:57 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:31:57.291+0000 7fc0cae75080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-07T21:31:57.401 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:57 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:31:57.396+0000 7fc0cae75080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-07T21:31:57.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:56 smithi047 ceph-mon[90087]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:57.707 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:57 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:31:57.504+0000 7fc0cae75080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-07T21:31:58.031 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:57 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:31:57.704+0000 7fc0cae75080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-07T21:31:58.361 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:58 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:31:58.104+0000 7fc0cae75080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-07T21:31:58.362 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:58 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:31:58.209+0000 7fc0cae75080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-07T21:31:58.362 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:58 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:31:58.358+0000 7fc0cae75080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-07T21:31:58.362 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:58 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:58.362 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:58 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:58.363 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:58 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-07T21:31:58.363 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:58 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-09-07T21:31:58.363 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:58 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-07T21:31:58.363 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:58 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:58.364 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:58 smithi059 ceph-mon[97450]: Deploying daemon mgr.y on smithi055 2022-09-07T21:31:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:58 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:58 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:58 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-07T21:31:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:58 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-09-07T21:31:58.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:58 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-07T21:31:58.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:58 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:58.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:58 smithi055 ceph-mon[97033]: Deploying daemon mgr.y on smithi055 2022-09-07T21:31:58.527 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:58 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:58.527 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:58 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:31:58.528 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:58 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-07T21:31:58.528 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:58 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-09-07T21:31:58.528 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:58 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-07T21:31:58.528 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:58 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:31:58.528 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:58 smithi047 ceph-mon[90087]: Deploying daemon mgr.y on smithi055 2022-09-07T21:31:58.781 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:58 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:31:58.470+0000 7fc0cae75080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-07T21:31:58.906 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:31:58 smithi055 systemd[1]: Starting Ceph mgr.y for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:31:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:31:59 smithi059 ceph-mon[97450]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:59.283 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:58 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:31:58.974+0000 7fc0cae75080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-07T21:31:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:31:59 smithi055 ceph-mon[97033]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:59.481 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:31:59 smithi055 podman[98397]: 2022-09-07T21:31:59.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:31:59 smithi047 ceph-mon[90087]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:31:59.732 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-09-07T21:31:59.733 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-09-07T21:31:59.733 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2022-09-07T21:31:59.734 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2022-09-07T21:31:59.734 INFO:teuthology.orchestra.run.smithi047.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0443692 s, 236 MB/s 2022-09-07T21:31:59.735 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2022-09-07T21:31:59.960 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:31:59 smithi055 ceph-mgr[98436]: -- 172.21.15.55:0/4115502995 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f457d22820 con 0x55f457cf7000 2022-09-07T21:31:59.961 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:31:59 smithi055 bash[98397]: 4a34efdf74239e85bc12b621cbc6c6f6dd0c5522b26587a62f9d322508f5b461 2022-09-07T21:31:59.961 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:31:59 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:31:59.754+0000 7fd34c5c0080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-07T21:31:59.961 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:31:59 smithi055 systemd[1]: Started Ceph mgr.y for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:31:59.961 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:31:59 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:31:59.850+0000 7fd34c5c0080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-07T21:31:59.962 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:31:59 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:31:59.957+0000 7fd34c5c0080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-07T21:32:00.031 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:59 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:31:59.661+0000 7fc0cae75080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-07T21:32:00.032 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:31:59 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:31:59.852+0000 7fc0cae75080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-07T21:32:00.210 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi047:vg_nvme/lv_4 2022-09-07T21:32:00.230 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:00 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:00.067+0000 7fd34c5c0080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-07T21:32:00.700 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:00 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:00.273+0000 7fd34c5c0080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-07T21:32:00.700 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:00 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:00.696+0000 7fd34c5c0080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-07T21:32:00.963 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:00 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:00.805+0000 7fd34c5c0080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-07T21:32:00.963 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:00.963 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:00.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:00.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:00.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:00.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:00 smithi055 ceph-mon[97033]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:01.093 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:32:01.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:01.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:01.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:01.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:01.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:01.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:00 smithi047 ceph-mon[90087]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:01.155 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:32:00 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:32:00.826+0000 7fc0cae75080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:32:01.156 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:32:00 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:32:00.936+0000 7fc0cae75080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-07T21:32:01.156 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:01.157 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:01.158 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:01.158 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:01.159 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:01.159 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:00 smithi059 ceph-mon[97450]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:01.230 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:00 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:00.959+0000 7fd34c5c0080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-07T21:32:01.231 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:01.072+0000 7fd34c5c0080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-07T21:32:01.532 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:32:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:32:01.151+0000 7fc0cae75080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:32:01.532 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:32:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:32:01.380+0000 7fc0cae75080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-07T21:32:01.878 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:01.601+0000 7fd34c5c0080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-07T21:32:01.922 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:32:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:32:01.572+0000 7fc0cae75080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-07T21:32:02.281 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:32:02 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:32:02.041+0000 7fc0cae75080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-07T21:32:02.561 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:02 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:02.293+0000 7fd34c5c0080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-07T21:32:02.561 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:02 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:02.484+0000 7fd34c5c0080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-07T21:32:02.879 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.875+0000 7f4b398d0700 1 -- 172.21.15.47:0/4155399231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b34100f40 msgr2=0x7f4b341033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:32:02.879 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.875+0000 7f4b398d0700 1 --2- 172.21.15.47:0/4155399231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b34100f40 0x7f4b341033a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4b28009750 tx=0x7f4b28009a60 comp rx=0 tx=0).stop 2022-09-07T21:32:02.880 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.875+0000 7f4b398d0700 1 -- 172.21.15.47:0/4155399231 shutdown_connections 2022-09-07T21:32:02.880 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.875+0000 7f4b398d0700 1 --2- 172.21.15.47:0/4155399231 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b34103a30 0x7f4b34105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:02.880 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.875+0000 7f4b398d0700 1 --2- 172.21.15.47:0/4155399231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b340673d0 0x7f4b34100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:02.881 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.875+0000 7f4b398d0700 1 --2- 172.21.15.47:0/4155399231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b34100f40 0x7f4b341033a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:02.881 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.875+0000 7f4b398d0700 1 -- 172.21.15.47:0/4155399231 >> 172.21.15.47:0/4155399231 conn(0x7f4b340f8430 msgr2=0x7f4b340fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:32:02.881 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.875+0000 7f4b398d0700 1 -- 172.21.15.47:0/4155399231 shutdown_connections 2022-09-07T21:32:02.881 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.875+0000 7f4b398d0700 1 -- 172.21.15.47:0/4155399231 wait complete. 2022-09-07T21:32:02.885 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.877+0000 7f4b398d0700 1 Processor -- start 2022-09-07T21:32:02.885 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.877+0000 7f4b398d0700 1 -- start start 2022-09-07T21:32:02.885 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.877+0000 7f4b398d0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b340673d0 0x7f4b34195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:32:02.886 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.877+0000 7f4b398d0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b34100f40 0x7f4b34196200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:32:02.886 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.877+0000 7f4b398d0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b34103a30 0x7f4b3419d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:32:02.886 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.877+0000 7f4b398d0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3419d870 con 0x7f4b34103a30 2022-09-07T21:32:02.887 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.877+0000 7f4b398d0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3419d9e0 con 0x7f4b34100f40 2022-09-07T21:32:02.887 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.877+0000 7f4b398d0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3419dba0 con 0x7f4b340673d0 2022-09-07T21:32:02.887 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.877+0000 7f4b327fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b34100f40 0x7f4b34196200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:32:02.887 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.877+0000 7f4b337fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b34103a30 0x7f4b3419d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:32:02.888 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.877+0000 7f4b32ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b340673d0 0x7f4b34195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:32:02.888 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.877+0000 7f4b327fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b34100f40 0x7f4b34196200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:49776/0 (socket says 172.21.15.47:49776) 2022-09-07T21:32:02.888 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.877+0000 7f4b327fc700 1 -- 172.21.15.47:0/1122865515 learned_addr learned my addr 172.21.15.47:0/1122865515 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:32:02.888 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.878+0000 7f4b327fc700 1 -- 172.21.15.47:0/1122865515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b340673d0 msgr2=0x7f4b34195cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:32:02.889 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.878+0000 7f4b327fc700 1 --2- 172.21.15.47:0/1122865515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b340673d0 0x7f4b34195cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:02.889 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.878+0000 7f4b327fc700 1 -- 172.21.15.47:0/1122865515 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b34103a30 msgr2=0x7f4b3419d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:32:02.890 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.878+0000 7f4b327fc700 1 --2- 172.21.15.47:0/1122865515 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b34103a30 0x7f4b3419d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:02.891 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.878+0000 7f4b327fc700 1 -- 172.21.15.47:0/1122865515 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b280093b0 con 0x7f4b34100f40 2022-09-07T21:32:02.891 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.878+0000 7f4b327fc700 1 --2- 172.21.15.47:0/1122865515 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b34100f40 0x7f4b34196200 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4b28004230 tx=0x7f4b28004260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:32:02.891 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.878+0000 7f4b388ce700 1 -- 172.21.15.47:0/1122865515 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2801d070 con 0x7f4b34100f40 2022-09-07T21:32:02.891 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.879+0000 7f4b388ce700 1 -- 172.21.15.47:0/1122865515 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4b28003bb0 con 0x7f4b34100f40 2022-09-07T21:32:02.892 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.879+0000 7f4b398d0700 1 -- 172.21.15.47:0/1122865515 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3419de20 con 0x7f4b34100f40 2022-09-07T21:32:02.892 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.879+0000 7f4b388ce700 1 -- 172.21.15.47:0/1122865515 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b28017c10 con 0x7f4b34100f40 2022-09-07T21:32:02.892 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.879+0000 7f4b398d0700 1 -- 172.21.15.47:0/1122865515 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3419e250 con 0x7f4b34100f40 2022-09-07T21:32:02.892 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.880+0000 7f4b388ce700 1 -- 172.21.15.47:0/1122865515 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f4b28027020 con 0x7f4b34100f40 2022-09-07T21:32:02.893 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.880+0000 7f4b398d0700 1 -- 172.21.15.47:0/1122865515 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b34109640 con 0x7f4b34100f40 2022-09-07T21:32:02.893 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.880+0000 7f4b388ce700 1 --2- 172.21.15.47:0/1122865515 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4b1c035b00 0x7f4b1c037fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:32:02.893 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.880+0000 7f4b388ce700 1 -- 172.21.15.47:0/1122865515 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4b28048340 con 0x7f4b34100f40 2022-09-07T21:32:02.893 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.880+0000 7f4b32ffd700 1 --2- 172.21.15.47:0/1122865515 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4b1c035b00 0x7f4b1c037fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:32:02.894 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.881+0000 7f4b32ffd700 1 --2- 172.21.15.47:0/1122865515 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4b1c035b00 0x7f4b1c037fb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4b34196b90 tx=0x7f4b24005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:32:02.895 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:02.886+0000 7f4b388ce700 1 -- 172.21.15.47:0/1122865515 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f4b28020590 con 0x7f4b34100f40 2022-09-07T21:32:02.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:02.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:02.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:02.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:02.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:02.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:02.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-07T21:32:02.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-07T21:32:02.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:02.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:02.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-07T21:32:02.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-07T21:32:02.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:03.021 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:32:02 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:32:02.674+0000 7fc0cae75080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-07T21:32:03.022 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:03.022 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:03.023 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:03.023 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:03.023 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:03.023 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:03.024 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-07T21:32:03.024 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-07T21:32:03.024 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:03.145 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:03.142+0000 7f4b398d0700 1 -- 172.21.15.47:0/1122865515 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b3405fb70 con 0x7f4b1c035b00 2022-09-07T21:32:03.282 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:32:03 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:32:03.018+0000 7fc0cae75080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-07T21:32:03.283 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:32:03 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:32:03.252+0000 7fc0cae75080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-07T21:32:03.651 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:32:03 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:32:03.385+0000 7fc0cae75080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-07T21:32:03.651 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:32:03 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:32:03.647+0000 7fc0cae75080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-07T21:32:03.708 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:03 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:03.434+0000 7fd34c5c0080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:32:03.709 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:03 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:03.541+0000 7fd34c5c0080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-07T21:32:03.977 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:03 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:03.754+0000 7fd34c5c0080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:32:03.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:03 smithi055 ceph-mon[97033]: Reconfiguring mgr.x (unknown last config time)... 2022-09-07T21:32:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:03 smithi055 ceph-mon[97033]: Reconfiguring daemon mgr.x on smithi047 2022-09-07T21:32:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:03 smithi055 ceph-mon[97033]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:32:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:32:03.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:04.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:03 smithi059 ceph-mon[97450]: Reconfiguring mgr.x (unknown last config time)... 2022-09-07T21:32:04.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:03 smithi059 ceph-mon[97450]: Reconfiguring daemon mgr.x on smithi047 2022-09-07T21:32:04.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:03 smithi059 ceph-mon[97450]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:04.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:32:04.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:32:04.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:04.109 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:03 smithi047 ceph-mon[90087]: Reconfiguring mgr.x (unknown last config time)... 2022-09-07T21:32:04.109 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:03 smithi047 ceph-mon[90087]: Reconfiguring daemon mgr.x on smithi047 2022-09-07T21:32:04.109 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:03 smithi047 ceph-mon[90087]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:04.109 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:32:04.110 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:32:04.110 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:04.230 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:03 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:03.973+0000 7fd34c5c0080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-07T21:32:04.230 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:04 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:04.152+0000 7fd34c5c0080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-07T21:32:04.531 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:32:04 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:32:04.441+0000 7fc0cae75080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-07T21:32:04.980 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:04 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:04.618+0000 7fd34c5c0080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-07T21:32:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:04 smithi055 ceph-mon[97033]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:32:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:04 smithi055 ceph-mon[97033]: Standby manager daemon z started 2022-09-07T21:32:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:04 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.59:0/856072071' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-09-07T21:32:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:04 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.59:0/856072071' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-07T21:32:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:04 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.59:0/856072071' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-09-07T21:32:04.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:04 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.59:0/856072071' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-07T21:32:04.997 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:04 smithi047 ceph-mon[90087]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:32:04.997 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:04.998 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:04.998 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:04.998 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:04.998 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:04.999 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:04.999 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:04 smithi047 ceph-mon[90087]: Standby manager daemon z started 2022-09-07T21:32:04.999 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:04 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.59:0/856072071' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-09-07T21:32:04.999 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:04 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.59:0/856072071' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-07T21:32:04.999 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:04 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.59:0/856072071' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-09-07T21:32:05.000 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:04 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.59:0/856072071' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-07T21:32:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:04 smithi059 ceph-mon[97450]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:32:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:05.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:05.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:05.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:04 smithi059 ceph-mon[97450]: Standby manager daemon z started 2022-09-07T21:32:05.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:04 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.59:0/856072071' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-09-07T21:32:05.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:04 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.59:0/856072071' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-07T21:32:05.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:04 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.59:0/856072071' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-09-07T21:32:05.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:04 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.59:0/856072071' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-07T21:32:05.045 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:05.042+0000 7f4b388ce700 1 -- 172.21.15.47:0/1122865515 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 14) v1 ==== 85192+0+0 (secure 0 0 0) 0x7f4b2800c040 con 0x7f4b34100f40 2022-09-07T21:32:05.596 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:05 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:05.249+0000 7fd34c5c0080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-07T21:32:05.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:05 smithi047 ceph-mon[90087]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:05.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:05 smithi047 ceph-mon[90087]: mgrmap e14: x(active, since 78s), standbys: z 2022-09-07T21:32:05.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:05 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-09-07T21:32:05.963 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:05 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:05.593+0000 7fd34c5c0080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-07T21:32:05.964 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:05 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:05.826+0000 7fd34c5c0080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-07T21:32:05.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:05 smithi055 ceph-mon[97033]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:05.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:05 smithi055 ceph-mon[97033]: mgrmap e14: x(active, since 78s), standbys: z 2022-09-07T21:32:05.966 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:05 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-09-07T21:32:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:05 smithi059 ceph-mon[97450]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:05 smithi059 ceph-mon[97450]: mgrmap e14: x(active, since 78s), standbys: z 2022-09-07T21:32:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:05 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-09-07T21:32:06.224 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:05 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:05.959+0000 7fd34c5c0080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-07T21:32:06.480 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:06 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:06.221+0000 7fd34c5c0080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-07T21:32:07.480 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:32:07 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:32:07.011+0000 7fd34c5c0080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-07T21:32:07.732 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:07.728+0000 7f4b388ce700 1 -- 172.21.15.47:0/1122865515 <== mon.2 v2:172.21.15.55:3300/0 8 ==== mgrmap(e 15) v1 ==== 127638+0+0 (secure 0 0 0) 0x7f4b28017d70 con 0x7f4b34100f40 2022-09-07T21:32:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:07 smithi055 ceph-mon[97033]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:07 smithi055 ceph-mon[97033]: Standby manager daemon y started 2022-09-07T21:32:07.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:07 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.55:0/3240934932' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-09-07T21:32:07.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:07 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.55:0/3240934932' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-07T21:32:07.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:07 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.55:0/3240934932' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-09-07T21:32:07.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:07 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.55:0/3240934932' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-07T21:32:07.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:07 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2121368644' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c4477c8-f9db-4bd6-80f3-ac83b7c86010"}]: dispatch 2022-09-07T21:32:07.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:07 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2121368644' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c4477c8-f9db-4bd6-80f3-ac83b7c86010"}]': finished 2022-09-07T21:32:07.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:07 smithi055 ceph-mon[97033]: osdmap e5: 1 total, 0 up, 1 in 2022-09-07T21:32:07.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:07 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:07 smithi059 ceph-mon[97450]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:07 smithi059 ceph-mon[97450]: Standby manager daemon y started 2022-09-07T21:32:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:07 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.55:0/3240934932' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-09-07T21:32:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:07 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.55:0/3240934932' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-07T21:32:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:07 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.55:0/3240934932' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-09-07T21:32:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:07 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.55:0/3240934932' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-07T21:32:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:07 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2121368644' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c4477c8-f9db-4bd6-80f3-ac83b7c86010"}]: dispatch 2022-09-07T21:32:08.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:07 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2121368644' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c4477c8-f9db-4bd6-80f3-ac83b7c86010"}]': finished 2022-09-07T21:32:08.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:07 smithi059 ceph-mon[97450]: osdmap e5: 1 total, 0 up, 1 in 2022-09-07T21:32:08.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:07 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:08.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:07 smithi047 ceph-mon[90087]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:08.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:07 smithi047 ceph-mon[90087]: Standby manager daemon y started 2022-09-07T21:32:08.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:07 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.55:0/3240934932' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-09-07T21:32:08.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:07 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.55:0/3240934932' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-07T21:32:08.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:07 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.55:0/3240934932' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-09-07T21:32:08.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:07 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.55:0/3240934932' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-07T21:32:08.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:07 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2121368644' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c4477c8-f9db-4bd6-80f3-ac83b7c86010"}]: dispatch 2022-09-07T21:32:08.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:07 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2121368644' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c4477c8-f9db-4bd6-80f3-ac83b7c86010"}]': finished 2022-09-07T21:32:08.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:07 smithi047 ceph-mon[90087]: osdmap e5: 1 total, 0 up, 1 in 2022-09-07T21:32:08.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:07 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:08 smithi055 ceph-mon[97033]: mgrmap e15: x(active, since 81s), standbys: y, z 2022-09-07T21:32:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:08 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-09-07T21:32:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:08 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/690375188' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:32:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:08 smithi059 ceph-mon[97450]: mgrmap e15: x(active, since 81s), standbys: y, z 2022-09-07T21:32:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:08 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-09-07T21:32:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:08 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/690375188' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:32:09.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:08 smithi047 ceph-mon[90087]: mgrmap e15: x(active, since 81s), standbys: y, z 2022-09-07T21:32:09.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:08 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-09-07T21:32:09.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:08 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/690375188' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:32:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:09 smithi059 ceph-mon[97450]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:10.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:09 smithi047 ceph-mon[90087]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:09 smithi055 ceph-mon[97033]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:11 smithi059 ceph-mon[97450]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:12.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:11 smithi047 ceph-mon[90087]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:11 smithi055 ceph-mon[97033]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:13 smithi059 ceph-mon[97450]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:14.101 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:13 smithi047 ceph-mon[90087]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:13 smithi055 ceph-mon[97033]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:15.995 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:15 smithi047 ceph-mon[90087]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:15 smithi059 ceph-mon[97450]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:15 smithi055 ceph-mon[97033]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:16 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-09-07T21:32:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:16 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:17.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:16 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-09-07T21:32:17.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:16 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:16 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-09-07T21:32:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:16 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:17.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:17 smithi047 ceph-mon[90087]: Deploying daemon osd.0 on smithi047 2022-09-07T21:32:17.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:17 smithi047 ceph-mon[90087]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:17 smithi059 ceph-mon[97450]: Deploying daemon osd.0 on smithi047 2022-09-07T21:32:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:17 smithi059 ceph-mon[97450]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:17 smithi055 ceph-mon[97033]: Deploying daemon osd.0 on smithi047 2022-09-07T21:32:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:17 smithi055 ceph-mon[97033]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:19.135 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:18 smithi047 ceph-mon[90087]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:18 smithi055 ceph-mon[97033]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:18 smithi059 ceph-mon[97450]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:21.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:21 smithi047 ceph-mon[90087]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:21 smithi055 ceph-mon[97033]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:21 smithi059 ceph-mon[97450]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:23.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:23 smithi047 ceph-mon[90087]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:23 smithi055 ceph-mon[97033]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:23 smithi059 ceph-mon[97450]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:25.857 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:25 smithi047 ceph-mon[90087]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:25 smithi055 ceph-mon[97033]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:25 smithi059 ceph-mon[97450]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:27.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:27.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:27.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:27.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:27.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:27.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:27 smithi047 ceph-mon[90087]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:27.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:27.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:27 smithi055 ceph-mon[97033]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:27.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:27.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:27.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:27.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:27 smithi059 ceph-mon[97450]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:29.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:28 smithi047 ceph-mon[90087]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:28 smithi055 ceph-mon[97033]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:28 smithi059 ceph-mon[97450]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:30.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:30 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:30.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:30 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:30.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:30 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:30.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:30 smithi047 ceph-mon[90087]: from='osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-09-07T21:32:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:30 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:30 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:30 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:30 smithi055 ceph-mon[97033]: from='osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-09-07T21:32:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:30 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:30 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:30 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:30 smithi059 ceph-mon[97450]: from='osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-09-07T21:32:31.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:31 smithi047 ceph-mon[90087]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:31.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:31 smithi047 ceph-mon[90087]: from='osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-09-07T21:32:31.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:31 smithi047 ceph-mon[90087]: osdmap e6: 1 total, 0 up, 1 in 2022-09-07T21:32:31.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:31 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:31.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:31 smithi047 ceph-mon[90087]: from='osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:32:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:31 smithi055 ceph-mon[97033]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:31 smithi055 ceph-mon[97033]: from='osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-09-07T21:32:31.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:31 smithi055 ceph-mon[97033]: osdmap e6: 1 total, 0 up, 1 in 2022-09-07T21:32:31.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:31 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:31.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:31 smithi055 ceph-mon[97033]: from='osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:32:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:31 smithi059 ceph-mon[97450]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:31 smithi059 ceph-mon[97450]: from='osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-09-07T21:32:32.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:31 smithi059 ceph-mon[97450]: osdmap e6: 1 total, 0 up, 1 in 2022-09-07T21:32:32.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:31 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:32.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:31 smithi059 ceph-mon[97450]: from='osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:32:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:32 smithi055 ceph-mon[97033]: from='osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-09-07T21:32:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:32 smithi055 ceph-mon[97033]: osdmap e7: 1 total, 0 up, 1 in 2022-09-07T21:32:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:32.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:32.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:32 smithi059 ceph-mon[97450]: from='osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-09-07T21:32:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:32 smithi059 ceph-mon[97450]: osdmap e7: 1 total, 0 up, 1 in 2022-09-07T21:32:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:33.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:33.045 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:32 smithi047 ceph-mon[90087]: from='osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-09-07T21:32:33.045 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:32 smithi047 ceph-mon[90087]: osdmap e7: 1 total, 0 up, 1 in 2022-09-07T21:32:33.045 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:33.046 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:33.046 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:33.567 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:33.563+0000 7f4b388ce700 1 -- 172.21.15.47:0/1122865515 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4b3405fb70 con 0x7f4b1c035b00 2022-09-07T21:32:33.567 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 0 on host 'smithi047' 2022-09-07T21:32:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:33.566+0000 7f4b398d0700 1 -- 172.21.15.47:0/1122865515 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4b1c035b00 msgr2=0x7f4b1c037fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:32:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:33.566+0000 7f4b398d0700 1 --2- 172.21.15.47:0/1122865515 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4b1c035b00 0x7f4b1c037fb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4b34196b90 tx=0x7f4b24005f60 comp rx=0 tx=0).stop 2022-09-07T21:32:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:33.566+0000 7f4b398d0700 1 -- 172.21.15.47:0/1122865515 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b34100f40 msgr2=0x7f4b34196200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:32:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:33.566+0000 7f4b398d0700 1 --2- 172.21.15.47:0/1122865515 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b34100f40 0x7f4b34196200 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4b28004230 tx=0x7f4b28004260 comp rx=0 tx=0).stop 2022-09-07T21:32:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:33.566+0000 7f4b398d0700 1 -- 172.21.15.47:0/1122865515 shutdown_connections 2022-09-07T21:32:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:33.566+0000 7f4b398d0700 1 --2- 172.21.15.47:0/1122865515 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4b1c035b00 0x7f4b1c037fb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:33.566+0000 7f4b398d0700 1 --2- 172.21.15.47:0/1122865515 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b34103a30 0x7f4b3419d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:33.566+0000 7f4b398d0700 1 --2- 172.21.15.47:0/1122865515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b340673d0 0x7f4b34195cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:33.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:33.566+0000 7f4b398d0700 1 --2- 172.21.15.47:0/1122865515 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b34100f40 0x7f4b34196200 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:33.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:33.566+0000 7f4b398d0700 1 -- 172.21.15.47:0/1122865515 >> 172.21.15.47:0/1122865515 conn(0x7f4b340f8430 msgr2=0x7f4b340fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:32:33.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:33.566+0000 7f4b398d0700 1 -- 172.21.15.47:0/1122865515 shutdown_connections 2022-09-07T21:32:33.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:32:33.566+0000 7f4b398d0700 1 -- 172.21.15.47:0/1122865515 wait complete. 2022-09-07T21:32:33.811 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:33 smithi047 ceph-mon[90087]: purged_snaps scrub starts 2022-09-07T21:32:33.811 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:33 smithi047 ceph-mon[90087]: purged_snaps scrub ok 2022-09-07T21:32:33.811 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:33 smithi047 ceph-mon[90087]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:33.812 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:33 smithi047 ceph-mon[90087]: osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881] boot 2022-09-07T21:32:33.812 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:33 smithi047 ceph-mon[90087]: osdmap e8: 1 total, 1 up, 1 in 2022-09-07T21:32:33.812 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:33.812 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:33.813 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:33.813 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:33.813 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:33.813 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:33 smithi055 ceph-mon[97033]: purged_snaps scrub starts 2022-09-07T21:32:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:33 smithi055 ceph-mon[97033]: purged_snaps scrub ok 2022-09-07T21:32:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:33 smithi055 ceph-mon[97033]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:33.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:33 smithi055 ceph-mon[97033]: osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881] boot 2022-09-07T21:32:33.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:33 smithi055 ceph-mon[97033]: osdmap e8: 1 total, 1 up, 1 in 2022-09-07T21:32:33.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:33.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:33.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:33.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:33.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:33.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:33 smithi059 ceph-mon[97450]: purged_snaps scrub starts 2022-09-07T21:32:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:33 smithi059 ceph-mon[97450]: purged_snaps scrub ok 2022-09-07T21:32:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:33 smithi059 ceph-mon[97450]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-07T21:32:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:33 smithi059 ceph-mon[97450]: osd.0 [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881] boot 2022-09-07T21:32:34.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:33 smithi059 ceph-mon[97450]: osdmap e8: 1 total, 1 up, 1 in 2022-09-07T21:32:34.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:32:34.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:34.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:34.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:34.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:34.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:34.238 DEBUG:teuthology.orchestra.run.smithi047:osd.0> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@osd.0.service 2022-09-07T21:32:34.245 INFO:tasks.cephadm:Deploying osd.1 on smithi055 with /dev/vg_nvme/lv_4... 2022-09-07T21:32:34.246 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-09-07T21:32:34.335 INFO:journalctl@ceph.osd.0.smithi047.stdout:-- Logs begin at Wed 2022-09-07 21:11:20 UTC. -- 2022-09-07T21:32:34.856 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.b/config 2022-09-07T21:32:35.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:35 smithi047 ceph-mon[90087]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:35.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:35 smithi047 ceph-mon[90087]: osdmap e9: 1 total, 1 up, 1 in 2022-09-07T21:32:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:35 smithi055 ceph-mon[97033]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:35 smithi055 ceph-mon[97033]: osdmap e9: 1 total, 1 up, 1 in 2022-09-07T21:32:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:35 smithi059 ceph-mon[97450]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:35 smithi059 ceph-mon[97450]: osdmap e9: 1 total, 1 up, 1 in 2022-09-07T21:32:36.637 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-09-07T21:32:36.638 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-09-07T21:32:36.638 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2022-09-07T21:32:36.639 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2022-09-07T21:32:36.639 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00795687 s, 1.3 GB/s 2022-09-07T21:32:36.639 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2022-09-07T21:32:37.088 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi055:vg_nvme/lv_4 2022-09-07T21:32:37.691 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.b/config 2022-09-07T21:32:37.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:37 smithi047 ceph-mon[90087]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:37.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:37.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:37.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:32:37.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:37.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:37.916 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:37 smithi055 ceph-mon[97033]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:37.917 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:37.917 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:37.918 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:32:37.918 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:37.918 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:37 smithi059 ceph-mon[97450]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:38.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:38.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:38.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:32:38.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:38.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:38 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi047 2022-09-07T21:32:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:38 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi047 to 17276M 2022-09-07T21:32:38.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:38 smithi047 ceph-mon[90087]: Detected new or changed devices on smithi047 2022-09-07T21:32:38.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:38 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi047 to 17276M 2022-09-07T21:32:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:38 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi047 2022-09-07T21:32:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:38 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi047 to 17276M 2022-09-07T21:32:39.155 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.149+0000 7fd221ae3700 1 -- 172.21.15.55:0/3179801544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd21c0fde10 msgr2=0x7fd21c0fe220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:32:39.156 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.149+0000 7fd221ae3700 1 --2- 172.21.15.55:0/3179801544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd21c0fde10 0x7fd21c0fe220 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd204009810 tx=0x7fd204009b20 comp rx=0 tx=0).stop 2022-09-07T21:32:39.156 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.149+0000 7fd221ae3700 1 -- 172.21.15.55:0/3179801544 shutdown_connections 2022-09-07T21:32:39.156 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.149+0000 7fd221ae3700 1 --2- 172.21.15.55:0/3179801544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c0ffa30 0x7fd21c106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:39.156 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.149+0000 7fd221ae3700 1 --2- 172.21.15.55:0/3179801544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd21c0fde10 0x7fd21c0fe220 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:39.157 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.149+0000 7fd221ae3700 1 --2- 172.21.15.55:0/3179801544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd21c0ff080 0x7fd21c0ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:39.157 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.149+0000 7fd221ae3700 1 -- 172.21.15.55:0/3179801544 >> 172.21.15.55:0/3179801544 conn(0x7fd21c0f9550 msgr2=0x7fd21c0fb980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:32:39.157 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.149+0000 7fd221ae3700 1 -- 172.21.15.55:0/3179801544 shutdown_connections 2022-09-07T21:32:39.157 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.149+0000 7fd221ae3700 1 -- 172.21.15.55:0/3179801544 wait complete. 2022-09-07T21:32:39.158 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd221ae3700 1 Processor -- start 2022-09-07T21:32:39.158 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd221ae3700 1 -- start start 2022-09-07T21:32:39.158 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd221ae3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd21c0fde10 0x7fd21c072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:32:39.158 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd221ae3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd21c0ff080 0x7fd21c0710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:32:39.159 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd221ae3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c0ffa30 0x7fd21c071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:32:39.159 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd221ae3700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21c071e10 con 0x7fd21c0fde10 2022-09-07T21:32:39.159 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd221ae3700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21c071f80 con 0x7fd21c0ff080 2022-09-07T21:32:39.159 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd221ae3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21c19dbc0 con 0x7fd21c0ffa30 2022-09-07T21:32:39.160 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd21affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd21c0ff080 0x7fd21c0710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:32:39.160 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd21affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd21c0ff080 0x7fd21c0710d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53334/0 (socket says 172.21.15.55:53334) 2022-09-07T21:32:39.160 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd21affd700 1 -- 172.21.15.55:0/221449078 learned_addr learned my addr 172.21.15.55:0/221449078 (peer_addr_for_me v2:172.21.15.55:0/0) 2022-09-07T21:32:39.160 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd21bfff700 1 --2- 172.21.15.55:0/221449078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c0ffa30 0x7fd21c071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:32:39.161 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd21affd700 1 -- 172.21.15.55:0/221449078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c0ffa30 msgr2=0x7fd21c071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:32:39.161 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd21affd700 1 --2- 172.21.15.55:0/221449078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c0ffa30 0x7fd21c071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:39.161 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd21affd700 1 -- 172.21.15.55:0/221449078 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd21c0fde10 msgr2=0x7fd21c072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:32:39.161 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd21affd700 1 --2- 172.21.15.55:0/221449078 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd21c0fde10 0x7fd21c072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:39.162 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd21affd700 1 -- 172.21.15.55:0/221449078 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd204009470 con 0x7fd21c0ff080 2022-09-07T21:32:39.162 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.150+0000 7fd21affd700 1 --2- 172.21.15.55:0/221449078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd21c0ff080 0x7fd21c0710d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00b410 tx=0x7fd20c00b720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:32:39.162 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.151+0000 7fd218ff9700 1 -- 172.21.15.55:0/221449078 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c011830 con 0x7fd21c0ff080 2022-09-07T21:32:39.162 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.151+0000 7fd218ff9700 1 -- 172.21.15.55:0/221449078 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd20c00be70 con 0x7fd21c0ff080 2022-09-07T21:32:39.162 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.151+0000 7fd221ae3700 1 -- 172.21.15.55:0/221449078 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd21c19dea0 con 0x7fd21c0ff080 2022-09-07T21:32:39.163 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.151+0000 7fd218ff9700 1 -- 172.21.15.55:0/221449078 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c010db0 con 0x7fd21c0ff080 2022-09-07T21:32:39.163 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.151+0000 7fd221ae3700 1 -- 172.21.15.55:0/221449078 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd21c19e390 con 0x7fd21c0ff080 2022-09-07T21:32:39.163 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.152+0000 7fd221ae3700 1 -- 172.21.15.55:0/221449078 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd21c109760 con 0x7fd21c0ff080 2022-09-07T21:32:39.164 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.153+0000 7fd218ff9700 1 -- 172.21.15.55:0/221449078 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 15) v1 ==== 127638+0+0 (secure 0 0 0) 0x7fd20c021070 con 0x7fd21c0ff080 2022-09-07T21:32:39.165 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.153+0000 7fd218ff9700 1 --2- 172.21.15.55:0/221449078 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd208099040 0x7fd20809b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:32:39.165 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.153+0000 7fd218ff9700 1 -- 172.21.15.55:0/221449078 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1763+0+0 (secure 0 0 0) 0x7fd20c0c0270 con 0x7fd21c0ff080 2022-09-07T21:32:39.165 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.154+0000 7fd21b7fe700 1 --2- 172.21.15.55:0/221449078 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd208099040 0x7fd20809b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:32:39.166 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.154+0000 7fd21b7fe700 1 --2- 172.21.15.55:0/221449078 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd208099040 0x7fd20809b4f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd2040046a0 tx=0x7fd204000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:32:39.166 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.158+0000 7fd218ff9700 1 -- 172.21.15.55:0/221449078 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fd20c08e390 con 0x7fd21c0ff080 2022-09-07T21:32:39.365 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:39.362+0000 7fd221ae3700 1 -- 172.21.15.55:0/221449078 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd21c05fbd0 con 0x7fd208099040 2022-09-07T21:32:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:39 smithi055 ceph-mon[97033]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:39 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:32:39.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:39 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:32:39.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:39 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:39.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:39 smithi047 ceph-mon[90087]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:39.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:39 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:32:39.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:39 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:32:39.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:39 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:39 smithi059 ceph-mon[97450]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:39 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:32:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:39 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:32:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:39 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:40.874 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:40 smithi055 ceph-mon[97033]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:32:40.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:40 smithi047 ceph-mon[90087]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:32:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:40 smithi059 ceph-mon[97450]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:32:41.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:41 smithi047 ceph-mon[90087]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:41 smithi055 ceph-mon[97033]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:41 smithi059 ceph-mon[97450]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:42.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:42 smithi047 ceph-mon[90087]: from='client.? 172.21.15.55:0/1236795361' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "692ba70e-935d-4a34-847d-44a304850c5b"}]: dispatch 2022-09-07T21:32:42.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:42 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "692ba70e-935d-4a34-847d-44a304850c5b"}]: dispatch 2022-09-07T21:32:42.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:42 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "692ba70e-935d-4a34-847d-44a304850c5b"}]': finished 2022-09-07T21:32:42.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:42 smithi047 ceph-mon[90087]: osdmap e10: 2 total, 1 up, 2 in 2022-09-07T21:32:42.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:42 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:32:42.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:42 smithi047 ceph-mon[90087]: from='client.? 172.21.15.55:0/1969387789' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:32:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:42 smithi055 ceph-mon[97033]: from='client.? 172.21.15.55:0/1236795361' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "692ba70e-935d-4a34-847d-44a304850c5b"}]: dispatch 2022-09-07T21:32:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:42 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "692ba70e-935d-4a34-847d-44a304850c5b"}]: dispatch 2022-09-07T21:32:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:42 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "692ba70e-935d-4a34-847d-44a304850c5b"}]': finished 2022-09-07T21:32:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:42 smithi055 ceph-mon[97033]: osdmap e10: 2 total, 1 up, 2 in 2022-09-07T21:32:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:42 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:32:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:42 smithi055 ceph-mon[97033]: from='client.? 172.21.15.55:0/1969387789' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:32:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:42 smithi059 ceph-mon[97450]: from='client.? 172.21.15.55:0/1236795361' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "692ba70e-935d-4a34-847d-44a304850c5b"}]: dispatch 2022-09-07T21:32:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:42 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "692ba70e-935d-4a34-847d-44a304850c5b"}]: dispatch 2022-09-07T21:32:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:42 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "692ba70e-935d-4a34-847d-44a304850c5b"}]': finished 2022-09-07T21:32:43.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:42 smithi059 ceph-mon[97450]: osdmap e10: 2 total, 1 up, 2 in 2022-09-07T21:32:43.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:42 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:32:43.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:42 smithi059 ceph-mon[97450]: from='client.? 172.21.15.55:0/1969387789' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:32:43.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:43 smithi047 ceph-mon[90087]: pgmap v58: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:43 smithi055 ceph-mon[97033]: pgmap v58: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:43 smithi059 ceph-mon[97450]: pgmap v58: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:45.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:45 smithi047 ceph-mon[90087]: pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:45 smithi055 ceph-mon[97033]: pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:45 smithi059 ceph-mon[97450]: pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:46.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:32:46.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:32:46.975 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:32:46.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:32:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:32:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:32:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:47 smithi055 ceph-mon[97033]: pgmap v60: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:47.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:47 smithi047 ceph-mon[90087]: pgmap v60: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:47 smithi059 ceph-mon[97450]: pgmap v60: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:49.004 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:48 smithi055 ceph-mon[97033]: pgmap v61: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:49.161 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:48 smithi047 ceph-mon[90087]: pgmap v61: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:48 smithi059 ceph-mon[97450]: pgmap v61: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:50.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:49 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-09-07T21:32:50.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:49 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:50.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:49 smithi047 ceph-mon[90087]: Deploying daemon osd.1 on smithi055 2022-09-07T21:32:50.191 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:49 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-09-07T21:32:50.191 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:49 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:50.192 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:49 smithi055 ceph-mon[97033]: Deploying daemon osd.1 on smithi055 2022-09-07T21:32:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:49 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-09-07T21:32:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:49 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:49 smithi059 ceph-mon[97450]: Deploying daemon osd.1 on smithi055 2022-09-07T21:32:51.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:50 smithi047 ceph-mon[90087]: pgmap v62: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:51.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:50 smithi055 ceph-mon[97033]: pgmap v62: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:50 smithi059 ceph-mon[97450]: pgmap v62: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:53.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:53 smithi047 ceph-mon[90087]: pgmap v63: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:53 smithi055 ceph-mon[97033]: pgmap v63: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:53 smithi059 ceph-mon[97450]: pgmap v63: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:55.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:55 smithi047 ceph-mon[90087]: pgmap v64: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:55.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:55 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:55.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:55 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:55.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:55 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:55.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:55 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:55.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:55 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:55 smithi055 ceph-mon[97033]: pgmap v64: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:55 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:55 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:55.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:55 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:55.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:55 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:55.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:55 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:55 smithi059 ceph-mon[97450]: pgmap v64: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:55 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:55 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:55 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:56.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:55 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:56.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:55 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:32:57.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:57 smithi055 ceph-mon[97033]: pgmap v65: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:57.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:57.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:57.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:57.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:57 smithi047 ceph-mon[90087]: pgmap v65: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:57.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:57.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:57.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:57 smithi059 ceph-mon[97450]: pgmap v65: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:58.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:59.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:58 smithi047 ceph-mon[90087]: pgmap v66: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:58 smithi055 ceph-mon[97033]: pgmap v66: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:58 smithi059 ceph-mon[97450]: pgmap v66: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:32:59.725 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:59.722+0000 7fd218ff9700 1 -- 172.21.15.55:0/221449078 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd21c05fbd0 con 0x7fd208099040 2022-09-07T21:32:59.725 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 1 on host 'smithi055' 2022-09-07T21:32:59.727 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:59.724+0000 7fd221ae3700 1 -- 172.21.15.55:0/221449078 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd208099040 msgr2=0x7fd20809b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:32:59.727 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:59.724+0000 7fd221ae3700 1 --2- 172.21.15.55:0/221449078 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd208099040 0x7fd20809b4f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd2040046a0 tx=0x7fd204000bc0 comp rx=0 tx=0).stop 2022-09-07T21:32:59.728 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:59.724+0000 7fd221ae3700 1 -- 172.21.15.55:0/221449078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd21c0ff080 msgr2=0x7fd21c0710d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:32:59.728 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:59.724+0000 7fd221ae3700 1 --2- 172.21.15.55:0/221449078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd21c0ff080 0x7fd21c0710d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00b410 tx=0x7fd20c00b720 comp rx=0 tx=0).stop 2022-09-07T21:32:59.728 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:59.724+0000 7fd221ae3700 1 -- 172.21.15.55:0/221449078 shutdown_connections 2022-09-07T21:32:59.728 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:59.724+0000 7fd221ae3700 1 --2- 172.21.15.55:0/221449078 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd208099040 0x7fd20809b4f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:59.729 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:59.724+0000 7fd221ae3700 1 --2- 172.21.15.55:0/221449078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c0ffa30 0x7fd21c071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:59.729 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:59.724+0000 7fd221ae3700 1 --2- 172.21.15.55:0/221449078 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd21c0fde10 0x7fd21c072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:59.729 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:59.724+0000 7fd221ae3700 1 --2- 172.21.15.55:0/221449078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd21c0ff080 0x7fd21c0710d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:32:59.729 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:59.724+0000 7fd221ae3700 1 -- 172.21.15.55:0/221449078 >> 172.21.15.55:0/221449078 conn(0x7fd21c0f9550 msgr2=0x7fd21c0fb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:32:59.730 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:59.724+0000 7fd221ae3700 1 -- 172.21.15.55:0/221449078 shutdown_connections 2022-09-07T21:32:59.730 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:32:59.724+0000 7fd221ae3700 1 -- 172.21.15.55:0/221449078 wait complete. 2022-09-07T21:32:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:59 smithi055 ceph-mon[97033]: from='osd.1 [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-09-07T21:32:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:59 smithi055 ceph-mon[97033]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-09-07T21:32:59.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:59 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:59.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:59 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:32:59.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:59 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:32:59.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:59 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:32:59.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:32:59 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:33:00.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:59 smithi047 ceph-mon[90087]: from='osd.1 [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-09-07T21:33:00.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:59 smithi047 ceph-mon[90087]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-09-07T21:33:00.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:59 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:00.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:59 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:33:00.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:59 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:00.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:59 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:00.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:32:59 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:33:00.272 DEBUG:teuthology.orchestra.run.smithi055:osd.1> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@osd.1.service 2022-09-07T21:33:00.275 INFO:tasks.cephadm:Deploying osd.2 on smithi059 with /dev/vg_nvme/lv_4... 2022-09-07T21:33:00.275 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-09-07T21:33:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:59 smithi059 ceph-mon[97450]: from='osd.1 [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-09-07T21:33:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:59 smithi059 ceph-mon[97450]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-09-07T21:33:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:59 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:59 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:33:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:59 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:59 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:00.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:32:59 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:33:00.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:-- Logs begin at Wed 2022-09-07 21:07:35 UTC. -- 2022-09-07T21:33:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.c/config 2022-09-07T21:33:00.980 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 07 21:33:00 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-1[102223]: 2022-09-07T21:33:00.899+0000 7ff5d5c0a700 -1 osd.1 0 waiting for initial osdmap 2022-09-07T21:33:00.980 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 07 21:33:00 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-1[102223]: 2022-09-07T21:33:00.904+0000 7ff5d01fc700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-07T21:33:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:00 smithi055 ceph-mon[97033]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-09-07T21:33:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:00 smithi055 ceph-mon[97033]: osdmap e11: 2 total, 1 up, 2 in 2022-09-07T21:33:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:33:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:00 smithi055 ceph-mon[97033]: from='osd.1 [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:33:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:00 smithi055 ceph-mon[97033]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:33:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:00 smithi055 ceph-mon[97033]: pgmap v68: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:33:01.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:00 smithi047 ceph-mon[90087]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-09-07T21:33:01.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:00 smithi047 ceph-mon[90087]: osdmap e11: 2 total, 1 up, 2 in 2022-09-07T21:33:01.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:33:01.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:00 smithi047 ceph-mon[90087]: from='osd.1 [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:33:01.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:00 smithi047 ceph-mon[90087]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:33:01.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:00 smithi047 ceph-mon[90087]: pgmap v68: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:33:01.163 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:00 smithi059 ceph-mon[97450]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-09-07T21:33:01.164 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:00 smithi059 ceph-mon[97450]: osdmap e11: 2 total, 1 up, 2 in 2022-09-07T21:33:01.164 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:33:01.164 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:00 smithi059 ceph-mon[97450]: from='osd.1 [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:33:01.164 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:00 smithi059 ceph-mon[97450]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:33:01.165 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:00 smithi059 ceph-mon[97450]: pgmap v68: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:33:02.065 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:01 smithi059 ceph-mon[97450]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2022-09-07T21:33:02.066 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:01 smithi059 ceph-mon[97450]: osdmap e12: 2 total, 1 up, 2 in 2022-09-07T21:33:02.066 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:01 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:33:02.066 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:01 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:33:02.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:01 smithi047 ceph-mon[90087]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2022-09-07T21:33:02.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:01 smithi047 ceph-mon[90087]: osdmap e12: 2 total, 1 up, 2 in 2022-09-07T21:33:02.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:01 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:33:02.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:01 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:33:02.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:01 smithi055 ceph-mon[97033]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2022-09-07T21:33:02.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:01 smithi055 ceph-mon[97033]: osdmap e12: 2 total, 1 up, 2 in 2022-09-07T21:33:02.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:01 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:33:02.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:01 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:33:02.640 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-09-07T21:33:02.641 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-09-07T21:33:02.641 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2022-09-07T21:33:02.641 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2022-09-07T21:33:02.641 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123659 s, 848 MB/s 2022-09-07T21:33:02.642 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2022-09-07T21:33:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:02 smithi059 ceph-mon[97450]: purged_snaps scrub starts 2022-09-07T21:33:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:02 smithi059 ceph-mon[97450]: purged_snaps scrub ok 2022-09-07T21:33:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:02 smithi059 ceph-mon[97450]: osd.1 [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301] boot 2022-09-07T21:33:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:02 smithi059 ceph-mon[97450]: osdmap e13: 2 total, 2 up, 2 in 2022-09-07T21:33:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:33:03.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:02 smithi059 ceph-mon[97450]: pgmap v71: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:33:03.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:02 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-07T21:33:03.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:03.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:03.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:33:03.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:02 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 17173M 2022-09-07T21:33:03.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:03.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:03.216 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi059:vg_nvme/lv_4 2022-09-07T21:33:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:02 smithi055 ceph-mon[97033]: purged_snaps scrub starts 2022-09-07T21:33:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:02 smithi055 ceph-mon[97033]: purged_snaps scrub ok 2022-09-07T21:33:03.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:02 smithi055 ceph-mon[97033]: osd.1 [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301] boot 2022-09-07T21:33:03.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:02 smithi055 ceph-mon[97033]: osdmap e13: 2 total, 2 up, 2 in 2022-09-07T21:33:03.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:33:03.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:02 smithi055 ceph-mon[97033]: pgmap v71: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:33:03.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:02 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-07T21:33:03.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:03.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:03.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:33:03.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:02 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 17173M 2022-09-07T21:33:03.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:03.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:03.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:02 smithi047 ceph-mon[90087]: purged_snaps scrub starts 2022-09-07T21:33:03.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:02 smithi047 ceph-mon[90087]: purged_snaps scrub ok 2022-09-07T21:33:03.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:02 smithi047 ceph-mon[90087]: osd.1 [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301] boot 2022-09-07T21:33:03.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:02 smithi047 ceph-mon[90087]: osdmap e13: 2 total, 2 up, 2 in 2022-09-07T21:33:03.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:33:03.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:02 smithi047 ceph-mon[90087]: pgmap v71: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-07T21:33:03.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:02 smithi047 ceph-mon[90087]: Detected new or changed devices on smithi055 2022-09-07T21:33:03.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:03.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:03.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:33:03.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:02 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi055 to 17173M 2022-09-07T21:33:03.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:03.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:03.857 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.c/config 2022-09-07T21:33:04.171 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:03 smithi059 ceph-mon[97450]: osdmap e14: 2 total, 2 up, 2 in 2022-09-07T21:33:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:03 smithi055 ceph-mon[97033]: osdmap e14: 2 total, 2 up, 2 in 2022-09-07T21:33:04.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:03 smithi047 ceph-mon[90087]: osdmap e14: 2 total, 2 up, 2 in 2022-09-07T21:33:05.173 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:04 smithi059 ceph-mon[97450]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:04 smithi055 ceph-mon[97033]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:05.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:04 smithi047 ceph-mon[90087]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:05.514 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.508+0000 7faf52301700 1 -- 172.21.15.59:0/572168248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf4c0fe970 msgr2=0x7faf4c105210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:33:05.515 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.508+0000 7faf52301700 1 --2- 172.21.15.59:0/572168248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf4c0fe970 0x7faf4c105210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faf44009bb0 tx=0x7faf44009ec0 comp rx=0 tx=0).stop 2022-09-07T21:33:05.515 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.508+0000 7faf52301700 1 -- 172.21.15.59:0/572168248 shutdown_connections 2022-09-07T21:33:05.515 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.508+0000 7faf52301700 1 --2- 172.21.15.59:0/572168248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf4c0fe970 0x7faf4c105210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:33:05.516 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.508+0000 7faf52301700 1 --2- 172.21.15.59:0/572168248 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf4c0fcde0 0x7faf4c0fd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:33:05.516 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.508+0000 7faf52301700 1 --2- 172.21.15.59:0/572168248 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faf4c0fdfc0 0x7faf4c0fe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:33:05.516 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.508+0000 7faf52301700 1 -- 172.21.15.59:0/572168248 >> 172.21.15.59:0/572168248 conn(0x7faf4c0f8430 msgr2=0x7faf4c0fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:33:05.516 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.508+0000 7faf52301700 1 -- 172.21.15.59:0/572168248 shutdown_connections 2022-09-07T21:33:05.517 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.508+0000 7faf52301700 1 -- 172.21.15.59:0/572168248 wait complete. 2022-09-07T21:33:05.517 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.509+0000 7faf52301700 1 Processor -- start 2022-09-07T21:33:05.517 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.509+0000 7faf52301700 1 -- start start 2022-09-07T21:33:05.517 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.509+0000 7faf52301700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faf4c0fcde0 0x7faf4c193a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:33:05.517 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.509+0000 7faf52301700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf4c0fdfc0 0x7faf4c193f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:33:05.518 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.509+0000 7faf52301700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf4c0fe970 0x7faf4c19afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:33:05.518 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.509+0000 7faf52301700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c19b520 con 0x7faf4c0fdfc0 2022-09-07T21:33:05.518 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.509+0000 7faf52301700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c19b690 con 0x7faf4c0fcde0 2022-09-07T21:33:05.518 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf52301700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c19b850 con 0x7faf4c0fe970 2022-09-07T21:33:05.519 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf5089e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf4c0fe970 0x7faf4c19afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:33:05.519 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf5089e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf4c0fe970 0x7faf4c19afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40160/0 (socket says 172.21.15.59:40160) 2022-09-07T21:33:05.519 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf5089e700 1 -- 172.21.15.59:0/58015951 learned_addr learned my addr 172.21.15.59:0/58015951 (peer_addr_for_me v2:172.21.15.59:0/0) 2022-09-07T21:33:05.520 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf4bfff700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faf4c0fcde0 0x7faf4c193a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:33:05.520 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf4b7fe700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf4c0fdfc0 0x7faf4c193f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:33:05.520 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf5089e700 1 -- 172.21.15.59:0/58015951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faf4c0fcde0 msgr2=0x7faf4c193a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:33:05.521 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf5089e700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faf4c0fcde0 0x7faf4c193a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:33:05.521 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf5089e700 1 -- 172.21.15.59:0/58015951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf4c0fdfc0 msgr2=0x7faf4c193f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:33:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf5089e700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf4c0fdfc0 0x7faf4c193f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:33:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf5089e700 1 -- 172.21.15.59:0/58015951 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf440097e0 con 0x7faf4c0fe970 2022-09-07T21:33:05.523 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf4b7fe700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf4c0fdfc0 0x7faf4c193f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-07T21:33:05.523 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf4bfff700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faf4c0fcde0 0x7faf4c193a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:33:05.523 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf5089e700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf4c0fe970 0x7faf4c19afe0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faf440092d0 tx=0x7faf44005b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:33:05.524 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf497fa700 1 -- 172.21.15.59:0/58015951 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4401c070 con 0x7faf4c0fe970 2022-09-07T21:33:05.524 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf497fa700 1 -- 172.21.15.59:0/58015951 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7faf44003d20 con 0x7faf4c0fe970 2022-09-07T21:33:05.525 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf52301700 1 -- 172.21.15.59:0/58015951 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4c19bad0 con 0x7faf4c0fe970 2022-09-07T21:33:05.525 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf497fa700 1 -- 172.21.15.59:0/58015951 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf440174e0 con 0x7faf4c0fe970 2022-09-07T21:33:05.525 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.510+0000 7faf52301700 1 -- 172.21.15.59:0/58015951 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4c19bfb0 con 0x7faf4c0fe970 2022-09-07T21:33:05.525 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.511+0000 7faf52301700 1 -- 172.21.15.59:0/58015951 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4c108530 con 0x7faf4c0fe970 2022-09-07T21:33:05.527 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.512+0000 7faf497fa700 1 -- 172.21.15.59:0/58015951 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 15) v1 ==== 127638+0+0 (secure 0 0 0) 0x7faf44027020 con 0x7faf4c0fe970 2022-09-07T21:33:05.527 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.513+0000 7faf497fa700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7faf38098fc0 0x7faf3809b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:33:05.527 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.513+0000 7faf497fa700 1 -- 172.21.15.59:0/58015951 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2305+0+0 (secure 0 0 0) 0x7faf440c22c0 con 0x7faf4c0fe970 2022-09-07T21:33:05.527 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.513+0000 7faf4bfff700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7faf38098fc0 0x7faf3809b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:33:05.528 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.514+0000 7faf4bfff700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7faf38098fc0 0x7faf3809b470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faf3c006fd0 tx=0x7faf3c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:33:05.528 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.517+0000 7faf497fa700 1 -- 172.21.15.59:0/58015951 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7faf440901c0 con 0x7faf4c0fe970 2022-09-07T21:33:05.722 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:05.719+0000 7faf52301700 1 -- 172.21.15.59:0/58015951 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7faf4c066e50 con 0x7faf38098fc0 2022-09-07T21:33:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:05 smithi059 ceph-mon[97450]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:33:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:05 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:33:06.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:05 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:33:06.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:05 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:05 smithi055 ceph-mon[97033]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:33:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:05 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:33:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:05 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:33:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:05 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:06.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:05 smithi047 ceph-mon[90087]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:33:06.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:05 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:33:06.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:05 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:33:06.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:05 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:06 smithi059 ceph-mon[97450]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:06 smithi055 ceph-mon[97033]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:07.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:06 smithi047 ceph-mon[90087]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:09.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:08 smithi047 ceph-mon[90087]: from='client.? 172.21.15.59:0/94245400' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20bbe910-ab0c-4921-b54f-6d37be12f5c6"}]: dispatch 2022-09-07T21:33:09.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:08 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20bbe910-ab0c-4921-b54f-6d37be12f5c6"}]: dispatch 2022-09-07T21:33:09.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:08 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20bbe910-ab0c-4921-b54f-6d37be12f5c6"}]': finished 2022-09-07T21:33:09.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:08 smithi047 ceph-mon[90087]: osdmap e15: 3 total, 2 up, 3 in 2022-09-07T21:33:09.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:08 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:09.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:08 smithi047 ceph-mon[90087]: pgmap v76: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:08 smithi055 ceph-mon[97033]: from='client.? 172.21.15.59:0/94245400' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20bbe910-ab0c-4921-b54f-6d37be12f5c6"}]: dispatch 2022-09-07T21:33:09.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:08 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20bbe910-ab0c-4921-b54f-6d37be12f5c6"}]: dispatch 2022-09-07T21:33:09.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:08 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20bbe910-ab0c-4921-b54f-6d37be12f5c6"}]': finished 2022-09-07T21:33:09.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:08 smithi055 ceph-mon[97033]: osdmap e15: 3 total, 2 up, 3 in 2022-09-07T21:33:09.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:08 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:09.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:08 smithi055 ceph-mon[97033]: pgmap v76: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:08 smithi059 ceph-mon[97450]: from='client.? 172.21.15.59:0/94245400' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20bbe910-ab0c-4921-b54f-6d37be12f5c6"}]: dispatch 2022-09-07T21:33:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:08 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20bbe910-ab0c-4921-b54f-6d37be12f5c6"}]: dispatch 2022-09-07T21:33:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:08 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20bbe910-ab0c-4921-b54f-6d37be12f5c6"}]': finished 2022-09-07T21:33:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:08 smithi059 ceph-mon[97450]: osdmap e15: 3 total, 2 up, 3 in 2022-09-07T21:33:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:08 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:08 smithi059 ceph-mon[97450]: pgmap v76: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:10.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:09 smithi047 ceph-mon[90087]: from='client.? 172.21.15.59:0/13007776' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:33:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:09 smithi055 ceph-mon[97033]: from='client.? 172.21.15.59:0/13007776' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:33:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:09 smithi059 ceph-mon[97450]: from='client.? 172.21.15.59:0/13007776' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:33:11.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:10 smithi047 ceph-mon[90087]: pgmap v77: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:10 smithi055 ceph-mon[97033]: pgmap v77: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:10 smithi059 ceph-mon[97450]: pgmap v77: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:13.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:13 smithi047 ceph-mon[90087]: pgmap v78: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:13.961 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:13 smithi059 ceph-mon[97450]: pgmap v78: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:13 smithi055 ceph-mon[97033]: pgmap v78: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:15 smithi047 ceph-mon[90087]: pgmap v79: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:15.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:15 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-09-07T21:33:15.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:15 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:15 smithi055 ceph-mon[97033]: pgmap v79: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:15 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-09-07T21:33:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:15 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:15 smithi059 ceph-mon[97450]: pgmap v79: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:15 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-09-07T21:33:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:15 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:16.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:16 smithi047 ceph-mon[90087]: Deploying daemon osd.2 on smithi059 2022-09-07T21:33:16.922 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:16 smithi059 ceph-mon[97450]: Deploying daemon osd.2 on smithi059 2022-09-07T21:33:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:16 smithi055 ceph-mon[97033]: Deploying daemon osd.2 on smithi059 2022-09-07T21:33:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:17 smithi059 ceph-mon[97450]: pgmap v80: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:17.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:17 smithi047 ceph-mon[90087]: pgmap v80: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:17 smithi055 ceph-mon[97033]: pgmap v80: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:18 smithi059 ceph-mon[97450]: pgmap v81: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:19.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:18 smithi047 ceph-mon[90087]: pgmap v81: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:18 smithi055 ceph-mon[97033]: pgmap v81: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:21.689 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:21 smithi059 ceph-mon[97450]: pgmap v82: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:21.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:21 smithi047 ceph-mon[90087]: pgmap v82: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:21 smithi055 ceph-mon[97033]: pgmap v82: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:22.994 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:22 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:22.995 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:22 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:22.995 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:22 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:33:22.995 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:22 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:22.995 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:22 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:33:23.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:22 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:23.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:22 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:23.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:22 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:33:23.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:22 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:23.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:22 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:33:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:22 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:22 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:22 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:33:23.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:22 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:23.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:22 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:33:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:23 smithi059 ceph-mon[97450]: pgmap v83: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:24.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:23 smithi047 ceph-mon[90087]: pgmap v83: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:23 smithi055 ceph-mon[97033]: pgmap v83: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:25.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:25.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:25.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:25.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:25 smithi047 ceph-mon[90087]: pgmap v84: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:25 smithi055 ceph-mon[97033]: pgmap v84: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:25.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:25.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:25 smithi059 ceph-mon[97450]: pgmap v84: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:25.931 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:25.928+0000 7faf497fa700 1 -- 172.21.15.59:0/58015951 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faf4c066e50 con 0x7faf38098fc0 2022-09-07T21:33:25.932 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 2 on host 'smithi059' 2022-09-07T21:33:25.933 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:25.930+0000 7faf52301700 1 -- 172.21.15.59:0/58015951 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7faf38098fc0 msgr2=0x7faf3809b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:33:25.934 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:25.930+0000 7faf52301700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7faf38098fc0 0x7faf3809b470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faf3c006fd0 tx=0x7faf3c008040 comp rx=0 tx=0).stop 2022-09-07T21:33:25.934 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:25.930+0000 7faf52301700 1 -- 172.21.15.59:0/58015951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf4c0fe970 msgr2=0x7faf4c19afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:33:25.934 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:25.930+0000 7faf52301700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf4c0fe970 0x7faf4c19afe0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faf440092d0 tx=0x7faf44005b90 comp rx=0 tx=0).stop 2022-09-07T21:33:25.935 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:25.930+0000 7faf52301700 1 -- 172.21.15.59:0/58015951 shutdown_connections 2022-09-07T21:33:25.935 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:25.930+0000 7faf52301700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7faf38098fc0 0x7faf3809b470 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:33:25.935 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:25.931+0000 7faf52301700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf4c0fe970 0x7faf4c19afe0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:33:25.935 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:25.931+0000 7faf52301700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faf4c0fcde0 0x7faf4c193a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:33:25.935 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:25.931+0000 7faf52301700 1 --2- 172.21.15.59:0/58015951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf4c0fdfc0 0x7faf4c193f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:33:25.936 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:25.931+0000 7faf52301700 1 -- 172.21.15.59:0/58015951 >> 172.21.15.59:0/58015951 conn(0x7faf4c0f8430 msgr2=0x7faf4c066740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:33:25.936 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:25.931+0000 7faf52301700 1 -- 172.21.15.59:0/58015951 shutdown_connections 2022-09-07T21:33:25.936 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:33:25.931+0000 7faf52301700 1 -- 172.21.15.59:0/58015951 wait complete. 2022-09-07T21:33:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:26 smithi059 ceph-mon[97450]: from='osd.2 [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-09-07T21:33:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:26 smithi059 ceph-mon[97450]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-09-07T21:33:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:26 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:26 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:26.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:26 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:33:26.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:26 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:26.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:26 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:33:26.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:26 smithi047 ceph-mon[90087]: from='osd.2 [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-09-07T21:33:26.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:26 smithi047 ceph-mon[90087]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-09-07T21:33:26.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:26 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:26.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:26 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:26.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:26 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:33:26.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:26 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:26.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:26 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:33:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:26 smithi055 ceph-mon[97033]: from='osd.2 [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-09-07T21:33:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:26 smithi055 ceph-mon[97033]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-09-07T21:33:26.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:26 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:26.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:26 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:26.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:26 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:33:26.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:26 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:26.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:26 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:33:26.528 DEBUG:teuthology.orchestra.run.smithi059:osd.2> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@osd.2.service 2022-09-07T21:33:26.530 INFO:tasks.cephadm:Deploying osd.3 on smithi047 with /dev/vg_nvme/lv_3... 2022-09-07T21:33:26.531 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-09-07T21:33:26.561 INFO:journalctl@ceph.osd.2.smithi059.stdout:-- Logs begin at Wed 2022-09-07 21:07:37 UTC. -- 2022-09-07T21:33:27.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:27 smithi047 ceph-mon[90087]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-09-07T21:33:27.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:27 smithi047 ceph-mon[90087]: osdmap e16: 3 total, 2 up, 3 in 2022-09-07T21:33:27.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:27.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:27 smithi047 ceph-mon[90087]: from='osd.2 [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:33:27.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:27 smithi047 ceph-mon[90087]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:33:27.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:27 smithi047 ceph-mon[90087]: pgmap v86: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:27.416 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:33:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:27 smithi055 ceph-mon[97033]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-09-07T21:33:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:27 smithi055 ceph-mon[97033]: osdmap e16: 3 total, 2 up, 3 in 2022-09-07T21:33:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:27 smithi055 ceph-mon[97033]: from='osd.2 [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:33:27.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:27 smithi055 ceph-mon[97033]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:33:27.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:27 smithi055 ceph-mon[97033]: pgmap v86: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:27.493 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 07 21:33:27 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-2[103395]: 2022-09-07T21:33:27.142+0000 7fb848416700 -1 osd.2 0 waiting for initial osdmap 2022-09-07T21:33:27.494 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 07 21:33:27 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-2[103395]: 2022-09-07T21:33:27.146+0000 7fb841a06700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-07T21:33:27.494 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:27 smithi059 ceph-mon[97450]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-09-07T21:33:27.494 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:27 smithi059 ceph-mon[97450]: osdmap e16: 3 total, 2 up, 3 in 2022-09-07T21:33:27.495 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:27.495 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:27 smithi059 ceph-mon[97450]: from='osd.2 [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:33:27.495 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:27 smithi059 ceph-mon[97450]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:33:27.495 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:27 smithi059 ceph-mon[97450]: pgmap v86: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:28.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:28 smithi047 ceph-mon[90087]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2022-09-07T21:33:28.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:28 smithi047 ceph-mon[90087]: osdmap e17: 3 total, 2 up, 3 in 2022-09-07T21:33:28.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:28 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:28.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:28 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:28 smithi059 ceph-mon[97450]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2022-09-07T21:33:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:28 smithi059 ceph-mon[97450]: osdmap e17: 3 total, 2 up, 3 in 2022-09-07T21:33:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:28 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:28 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:28 smithi055 ceph-mon[97033]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2022-09-07T21:33:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:28 smithi055 ceph-mon[97033]: osdmap e17: 3 total, 2 up, 3 in 2022-09-07T21:33:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:28 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:28.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:28 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:29.603 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:29 smithi059 ceph-mon[97450]: osd.2 [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578] boot 2022-09-07T21:33:29.603 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:29 smithi059 ceph-mon[97450]: osdmap e18: 3 total, 3 up, 3 in 2022-09-07T21:33:29.603 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:29.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:29 smithi047 ceph-mon[90087]: osd.2 [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578] boot 2022-09-07T21:33:29.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:29 smithi047 ceph-mon[90087]: osdmap e18: 3 total, 3 up, 3 in 2022-09-07T21:33:29.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:29 smithi055 ceph-mon[97033]: osd.2 [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578] boot 2022-09-07T21:33:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:29 smithi055 ceph-mon[97033]: osdmap e18: 3 total, 3 up, 3 in 2022-09-07T21:33:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:33:29.963 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-09-07T21:33:29.964 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-09-07T21:33:29.964 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2022-09-07T21:33:29.964 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2022-09-07T21:33:29.964 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0445191 s, 236 MB/s 2022-09-07T21:33:29.965 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2022-09-07T21:33:30.493 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi047:vg_nvme/lv_3 2022-09-07T21:33:30.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:30 smithi047 ceph-mon[90087]: purged_snaps scrub starts 2022-09-07T21:33:30.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:30 smithi047 ceph-mon[90087]: purged_snaps scrub ok 2022-09-07T21:33:30.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:30 smithi047 ceph-mon[90087]: pgmap v89: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:30.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:30 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2022-09-07T21:33:30.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:30 smithi047 ceph-mon[90087]: Detected new or changed devices on smithi059 2022-09-07T21:33:30.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:30 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:30.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:30 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:30.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:30 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:33:30.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:30 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi059 to 17173M 2022-09-07T21:33:30.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:30 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:30.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:30 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:30.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:30 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2022-09-07T21:33:30.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:30 smithi047 ceph-mon[90087]: osdmap e19: 3 total, 3 up, 3 in 2022-09-07T21:33:30.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:30 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-09-07T21:33:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:30 smithi055 ceph-mon[97033]: purged_snaps scrub starts 2022-09-07T21:33:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:30 smithi055 ceph-mon[97033]: purged_snaps scrub ok 2022-09-07T21:33:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:30 smithi055 ceph-mon[97033]: pgmap v89: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:30 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2022-09-07T21:33:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:30 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-07T21:33:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:30 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:30 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:30.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:30 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:33:30.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:30 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 17173M 2022-09-07T21:33:30.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:30 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:30.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:30 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:30.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:30 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2022-09-07T21:33:30.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:30 smithi055 ceph-mon[97033]: osdmap e19: 3 total, 3 up, 3 in 2022-09-07T21:33:30.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:30 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-09-07T21:33:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:30 smithi059 ceph-mon[97450]: purged_snaps scrub starts 2022-09-07T21:33:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:30 smithi059 ceph-mon[97450]: purged_snaps scrub ok 2022-09-07T21:33:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:30 smithi059 ceph-mon[97450]: pgmap v89: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-07T21:33:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:30 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2022-09-07T21:33:30.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:30 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-07T21:33:30.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:30 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:30.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:30 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:30.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:30 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:33:30.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:30 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 17173M 2022-09-07T21:33:30.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:30 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:30.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:30 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:30.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:30 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2022-09-07T21:33:30.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:30 smithi059 ceph-mon[97450]: osdmap e19: 3 total, 3 up, 3 in 2022-09-07T21:33:30.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:30 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-09-07T21:33:31.382 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:33:31.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:31 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-09-07T21:33:31.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:31 smithi047 ceph-mon[90087]: osdmap e20: 3 total, 3 up, 3 in 2022-09-07T21:33:31.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:31 smithi047 ceph-mon[90087]: pgmap v92: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 21:33:31 smithi047 sudo[105370]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-07T21:33:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 21:33:31 smithi047 sudo[105370]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-07T21:33:31.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 21:33:31 smithi047 sudo[105370]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-07T21:33:31.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 21:33:31 smithi047 sudo[105370]: pam_unix(sudo:session): session closed for user root 2022-09-07T21:33:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:31 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-09-07T21:33:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:31 smithi055 ceph-mon[97033]: osdmap e20: 3 total, 3 up, 3 in 2022-09-07T21:33:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:31 smithi055 ceph-mon[97033]: pgmap v92: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:31 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-09-07T21:33:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:31 smithi059 ceph-mon[97450]: osdmap e20: 3 total, 3 up, 3 in 2022-09-07T21:33:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:31 smithi059 ceph-mon[97450]: pgmap v92: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 21:33:32 smithi047 sudo[105441]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-07T21:33:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 21:33:32 smithi047 sudo[105441]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-07T21:33:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 21:33:32 smithi047 sudo[105441]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-07T21:33:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 21:33:32 smithi047 sudo[105441]: pam_unix(sudo:session): session closed for user root 2022-09-07T21:33:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:32 smithi055 ceph-mon[97033]: mgrmap e16: x(active, since 2m), standbys: y, z 2022-09-07T21:33:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:32 smithi055 ceph-mon[97033]: osdmap e21: 3 total, 3 up, 3 in 2022-09-07T21:33:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:32.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 07 21:33:32 smithi055 sudo[104635]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-07T21:33:32.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 07 21:33:32 smithi055 sudo[104635]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-07T21:33:32.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 07 21:33:32 smithi055 sudo[104635]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-07T21:33:32.732 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 07 21:33:32 smithi055 sudo[104635]: pam_unix(sudo:session): session closed for user root 2022-09-07T21:33:32.746 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:32 smithi047 ceph-mon[90087]: mgrmap e16: x(active, since 2m), standbys: y, z 2022-09-07T21:33:32.747 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:32 smithi047 ceph-mon[90087]: osdmap e21: 3 total, 3 up, 3 in 2022-09-07T21:33:32.747 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:32 smithi059 ceph-mon[97450]: mgrmap e16: x(active, since 2m), standbys: y, z 2022-09-07T21:33:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:32 smithi059 ceph-mon[97450]: osdmap e21: 3 total, 3 up, 3 in 2022-09-07T21:33:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:33.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.047+0000 7f8ee4da8700 1 -- 172.21.15.47:0/3356774936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ee0072c30 msgr2=0x7f8ee0071110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:33:33.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.047+0000 7f8ee4da8700 1 --2- 172.21.15.47:0/3356774936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ee0072c30 0x7f8ee0071110 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009810 tx=0x7f8ed0009b20 comp rx=0 tx=0).stop 2022-09-07T21:33:33.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.047+0000 7f8ee4da8700 1 -- 172.21.15.47:0/3356774936 shutdown_connections 2022-09-07T21:33:33.052 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.047+0000 7f8ee4da8700 1 --2- 172.21.15.47:0/3356774936 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ee0105d30 0x7f8ee0108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:33:33.052 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.047+0000 7f8ee4da8700 1 --2- 172.21.15.47:0/3356774936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ee0072c30 0x7f8ee0071110 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:33:33.052 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.047+0000 7f8ee4da8700 1 --2- 172.21.15.47:0/3356774936 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee0071650 0x7f8ee0071ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:33:33.052 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.047+0000 7f8ee4da8700 1 -- 172.21.15.47:0/3356774936 >> 172.21.15.47:0/3356774936 conn(0x7f8ee00f9570 msgr2=0x7f8ee00fb980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:33:33.052 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.047+0000 7f8ee4da8700 1 -- 172.21.15.47:0/3356774936 shutdown_connections 2022-09-07T21:33:33.053 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.047+0000 7f8ee4da8700 1 -- 172.21.15.47:0/3356774936 wait complete. 2022-09-07T21:33:33.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.048+0000 7f8ee4da8700 1 Processor -- start 2022-09-07T21:33:33.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.048+0000 7f8ee4da8700 1 -- start start 2022-09-07T21:33:33.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.049+0000 7f8ee4da8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ee0071650 0x7f8ee0195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:33:33.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.049+0000 7f8ee4da8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ee0072c30 0x7f8ee01961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:33:33.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.049+0000 7f8ee4da8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee0105d30 0x7f8ee019d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:33:33.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.049+0000 7f8ee4da8700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee019d830 con 0x7f8ee0105d30 2022-09-07T21:33:33.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.049+0000 7f8ee4da8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee019d9a0 con 0x7f8ee0071650 2022-09-07T21:33:33.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.049+0000 7f8ee4da8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee019db60 con 0x7f8ee0072c30 2022-09-07T21:33:33.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.049+0000 7f8ede59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ee0071650 0x7f8ee0195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:33:33.057 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.049+0000 7f8ede59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ee0071650 0x7f8ee0195c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:53626/0 (socket says 172.21.15.47:53626) 2022-09-07T21:33:33.057 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.049+0000 7f8ede59c700 1 -- 172.21.15.47:0/1639077181 learned_addr learned my addr 172.21.15.47:0/1639077181 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:33:33.057 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.049+0000 7f8eddd9b700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ee0072c30 0x7f8ee01961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:33:33.057 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.049+0000 7f8eded9d700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee0105d30 0x7f8ee019d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:33:33.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.050+0000 7f8ede59c700 1 -- 172.21.15.47:0/1639077181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ee0072c30 msgr2=0x7f8ee01961b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:33:33.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.050+0000 7f8ede59c700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ee0072c30 0x7f8ee01961b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:33:33.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.050+0000 7f8ede59c700 1 -- 172.21.15.47:0/1639077181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee0105d30 msgr2=0x7f8ee019d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:33:33.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.050+0000 7f8ede59c700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee0105d30 0x7f8ee019d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:33:33.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.050+0000 7f8ede59c700 1 -- 172.21.15.47:0/1639077181 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed0009470 con 0x7f8ee0071650 2022-09-07T21:33:33.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.050+0000 7f8eded9d700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee0105d30 0x7f8ee019d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:33:33.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.050+0000 7f8ede59c700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ee0071650 0x7f8ee0195c70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8ed000f690 tx=0x7f8ed000f6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:33:33.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.050+0000 7f8ecf7fe700 1 -- 172.21.15.47:0/1639077181 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed001d070 con 0x7f8ee0071650 2022-09-07T21:33:33.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.050+0000 7f8ecf7fe700 1 -- 172.21.15.47:0/1639077181 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8ed00054f0 con 0x7f8ee0071650 2022-09-07T21:33:33.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.050+0000 7f8ee4da8700 1 -- 172.21.15.47:0/1639077181 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee019dde0 con 0x7f8ee0071650 2022-09-07T21:33:33.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.050+0000 7f8ecf7fe700 1 -- 172.21.15.47:0/1639077181 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed0004860 con 0x7f8ee0071650 2022-09-07T21:33:33.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.050+0000 7f8ee4da8700 1 -- 172.21.15.47:0/1639077181 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee019e220 con 0x7f8ee0071650 2022-09-07T21:33:33.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.051+0000 7f8ee4da8700 1 -- 172.21.15.47:0/1639077181 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee010b840 con 0x7f8ee0071650 2022-09-07T21:33:33.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.053+0000 7f8ecf7fe700 1 -- 172.21.15.47:0/1639077181 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f8ed0028030 con 0x7f8ee0071650 2022-09-07T21:33:33.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.054+0000 7f8ecf7fe700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8ec8099090 0x7f8ec809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:33:33.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.054+0000 7f8ecf7fe700 1 -- 172.21.15.47:0/1639077181 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3233+0+0 (secure 0 0 0) 0x7f8ed00c1ce0 con 0x7f8ee0071650 2022-09-07T21:33:33.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.055+0000 7f8eddd9b700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8ec8099090 0x7f8ec809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:33:33.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.055+0000 7f8eddd9b700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8ec8099090 0x7f8ec809b540 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4005fd0 tx=0x7f8ed4005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:33:33.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.058+0000 7f8ecf7fe700 1 -- 172.21.15.47:0/1639077181 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f8ed008f700 con 0x7f8ee0071650 2022-09-07T21:33:33.300 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:33:33.296+0000 7f8ee4da8700 1 -- 172.21.15.47:0/1639077181 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ee005fb30 con 0x7f8ec8099090 2022-09-07T21:33:33.419 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 07 21:33:33 smithi055 sudo[104638]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-07T21:33:33.419 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 07 21:33:33 smithi055 sudo[104638]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-07T21:33:33.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 07 21:33:33 smithi055 sudo[104638]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-07T21:33:33.420 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 07 21:33:33 smithi055 sudo[104638]: pam_unix(sudo:session): session closed for user root 2022-09-07T21:33:33.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:33 smithi047 ceph-mon[90087]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:33.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:33.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:33:33.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:33:33.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:33 smithi055 ceph-mon[97033]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:33:33.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:33:33.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:33 smithi059 ceph-mon[97450]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:33:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:33:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:33.783 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 07 21:33:33 smithi059 sudo[105741]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-07T21:33:33.783 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 07 21:33:33 smithi059 sudo[105741]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-07T21:33:33.783 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 07 21:33:33 smithi059 sudo[105741]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-07T21:33:33.783 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 07 21:33:33 smithi059 sudo[105741]: pam_unix(sudo:session): session closed for user root 2022-09-07T21:33:34.282 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 07 21:33:33 smithi059 sudo[105744]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-07T21:33:34.282 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 07 21:33:33 smithi059 sudo[105744]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-07T21:33:34.282 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 07 21:33:33 smithi059 sudo[105744]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-07T21:33:34.283 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 07 21:33:33 smithi059 sudo[105744]: pam_unix(sudo:session): session closed for user root 2022-09-07T21:33:35.115 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:34 smithi047 sudo[105659]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-07T21:33:35.116 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:34 smithi047 sudo[105659]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-07T21:33:35.116 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:34 smithi047 sudo[105659]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-07T21:33:35.116 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:34 smithi047 sudo[105659]: pam_unix(sudo:session): session closed for user root 2022-09-07T21:33:35.768 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:35 smithi047 sudo[105718]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-07T21:33:35.769 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:35 smithi047 sudo[105718]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-07T21:33:35.769 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:35 smithi047 sudo[105718]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-07T21:33:35.769 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:35 smithi047 sudo[105718]: pam_unix(sudo:session): session closed for user root 2022-09-07T21:33:35.769 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:35 smithi047 ceph-mon[90087]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:33:35.770 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:35 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:35.770 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:35 smithi047 ceph-mon[90087]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-07T21:33:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:35 smithi055 ceph-mon[97033]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:33:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:35 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:35 smithi055 ceph-mon[97033]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-07T21:33:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:35 smithi059 ceph-mon[97450]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:33:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:35 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:35 smithi059 ceph-mon[97450]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-07T21:33:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:36 smithi059 sudo[105747]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-07T21:33:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:36 smithi059 sudo[105747]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-07T21:33:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:36 smithi059 sudo[105747]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-07T21:33:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:36 smithi059 sudo[105747]: pam_unix(sudo:session): session closed for user root 2022-09-07T21:33:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:36 smithi059 sudo[105750]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-07T21:33:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:36 smithi059 sudo[105750]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-07T21:33:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:36 smithi059 sudo[105750]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-07T21:33:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:36 smithi059 sudo[105750]: pam_unix(sudo:session): session closed for user root 2022-09-07T21:33:37.450 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 sudo[104641]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-07T21:33:37.450 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 sudo[104641]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-07T21:33:37.451 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 sudo[104641]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-07T21:33:37.451 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 sudo[104641]: pam_unix(sudo:session): session closed for user root 2022-09-07T21:33:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 sudo[104644]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-07T21:33:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 sudo[104644]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-07T21:33:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 sudo[104644]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-07T21:33:37.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 sudo[104644]: pam_unix(sudo:session): session closed for user root 2022-09-07T21:33:37.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 ceph-mon[97033]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:37.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 ceph-mon[97033]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-07T21:33:37.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:33:37.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:33:37.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:33:37.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 ceph-mon[97033]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-07T21:33:37.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:33:37.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:33:37.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:33:37.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 ceph-mon[97033]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-07T21:33:37.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:33:37.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:33:37.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:33:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:37 smithi059 ceph-mon[97450]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:37 smithi059 ceph-mon[97450]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-07T21:33:37.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:33:37.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:33:37.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:33:37.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:37 smithi059 ceph-mon[97450]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-07T21:33:37.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:33:37.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:33:37.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:33:37.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:37 smithi059 ceph-mon[97450]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-07T21:33:37.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:33:37.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:33:37.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:33:37.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:37 smithi047 ceph-mon[90087]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:37.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:37 smithi047 ceph-mon[90087]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-07T21:33:37.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:33:37.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:33:37.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:33:37.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:37 smithi047 ceph-mon[90087]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-07T21:33:37.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:33:37.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:33:37.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:33:37.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:37 smithi047 ceph-mon[90087]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-07T21:33:37.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:33:37.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:33:37.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:33:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:38 smithi055 ceph-mon[97033]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:38 smithi055 ceph-mon[97033]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-07T21:33:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:38 smithi055 ceph-mon[97033]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-07T21:33:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:38 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2274769217' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4b5e6b0-5a09-4711-bd7f-5496b0547af2"}]: dispatch 2022-09-07T21:33:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:38 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4b5e6b0-5a09-4711-bd7f-5496b0547af2"}]: dispatch 2022-09-07T21:33:38.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:38 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4b5e6b0-5a09-4711-bd7f-5496b0547af2"}]': finished 2022-09-07T21:33:38.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:38 smithi055 ceph-mon[97033]: osdmap e22: 4 total, 3 up, 4 in 2022-09-07T21:33:38.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:38 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:33:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:38 smithi059 ceph-mon[97450]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:38 smithi059 ceph-mon[97450]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-07T21:33:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:38 smithi059 ceph-mon[97450]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-07T21:33:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:38 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2274769217' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4b5e6b0-5a09-4711-bd7f-5496b0547af2"}]: dispatch 2022-09-07T21:33:38.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:38 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4b5e6b0-5a09-4711-bd7f-5496b0547af2"}]: dispatch 2022-09-07T21:33:38.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:38 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4b5e6b0-5a09-4711-bd7f-5496b0547af2"}]': finished 2022-09-07T21:33:38.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:38 smithi059 ceph-mon[97450]: osdmap e22: 4 total, 3 up, 4 in 2022-09-07T21:33:38.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:38 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:33:38.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:38 smithi047 ceph-mon[90087]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:38.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:38 smithi047 ceph-mon[90087]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-07T21:33:38.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:38 smithi047 ceph-mon[90087]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-07T21:33:38.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:38 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2274769217' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4b5e6b0-5a09-4711-bd7f-5496b0547af2"}]: dispatch 2022-09-07T21:33:38.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:38 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4b5e6b0-5a09-4711-bd7f-5496b0547af2"}]: dispatch 2022-09-07T21:33:38.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:38 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4b5e6b0-5a09-4711-bd7f-5496b0547af2"}]': finished 2022-09-07T21:33:38.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:38 smithi047 ceph-mon[90087]: osdmap e22: 4 total, 3 up, 4 in 2022-09-07T21:33:38.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:38 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:33:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:39 smithi055 ceph-mon[97033]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 326 KiB/s wr, 22 op/s 2022-09-07T21:33:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:39 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2800075938' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:33:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:39 smithi059 ceph-mon[97450]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 326 KiB/s wr, 22 op/s 2022-09-07T21:33:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:39 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2800075938' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:33:39.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:39 smithi047 ceph-mon[90087]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 326 KiB/s wr, 22 op/s 2022-09-07T21:33:39.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:39 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2800075938' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:33:41.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:41 smithi047 ceph-mon[90087]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2022-09-07T21:33:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:41 smithi055 ceph-mon[97033]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2022-09-07T21:33:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:41 smithi059 ceph-mon[97450]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2022-09-07T21:33:43.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:43 smithi047 ceph-mon[90087]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2022-09-07T21:33:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:43 smithi055 ceph-mon[97033]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2022-09-07T21:33:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:43 smithi059 ceph-mon[97450]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2022-09-07T21:33:45.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:45 smithi047 ceph-mon[90087]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2022-09-07T21:33:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:45 smithi055 ceph-mon[97033]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2022-09-07T21:33:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:45 smithi059 ceph-mon[97450]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2022-09-07T21:33:46.718 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-09-07T21:33:46.719 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:46.719 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:33:46.719 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:33:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-09-07T21:33:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:46.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:33:46.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:33:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-09-07T21:33:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:33:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:33:47.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:47 smithi047 ceph-mon[90087]: Deploying daemon osd.3 on smithi047 2022-09-07T21:33:47.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:47 smithi047 ceph-mon[90087]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2022-09-07T21:33:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:47 smithi055 ceph-mon[97033]: Deploying daemon osd.3 on smithi047 2022-09-07T21:33:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:47 smithi055 ceph-mon[97033]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2022-09-07T21:33:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:47 smithi059 ceph-mon[97450]: Deploying daemon osd.3 on smithi047 2022-09-07T21:33:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:47 smithi059 ceph-mon[97450]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2022-09-07T21:33:49.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:48 smithi047 ceph-mon[90087]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 121 KiB/s wr, 8 op/s 2022-09-07T21:33:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:48 smithi055 ceph-mon[97033]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 121 KiB/s wr, 8 op/s 2022-09-07T21:33:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:48 smithi059 ceph-mon[97450]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 121 KiB/s wr, 8 op/s 2022-09-07T21:33:51.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:51 smithi047 ceph-mon[90087]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:51 smithi055 ceph-mon[97033]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:51 smithi059 ceph-mon[97450]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:53.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:53 smithi047 ceph-mon[90087]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:53 smithi055 ceph-mon[97033]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:53 smithi059 ceph-mon[97450]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:55.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:55 smithi047 ceph-mon[90087]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:55 smithi055 ceph-mon[97033]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:55 smithi059 ceph-mon[97450]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:57.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:57 smithi047 ceph-mon[90087]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:57.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:57.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:57.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:33:57.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:57.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:33:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:57 smithi055 ceph-mon[97033]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:57.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:57.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:57.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:33:57.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:57.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:33:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:57 smithi059 ceph-mon[97450]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:33:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:33:58.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:33:58.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:33:59.071 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:33:58 smithi047 ceph-mon[90087]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:33:58 smithi055 ceph-mon[97033]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:33:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:33:58 smithi059 ceph-mon[97450]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:34:01.587 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:01 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:01.587 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:01 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:01.588 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:01 smithi059 ceph-mon[97450]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:34:01.588 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:01 smithi059 ceph-mon[97450]: from='osd.3 [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-09-07T21:34:01.588 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:01 smithi059 ceph-mon[97450]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-09-07T21:34:01.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:01 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:01.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:01 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:01.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:01 smithi047 ceph-mon[90087]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:34:01.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:01 smithi047 ceph-mon[90087]: from='osd.3 [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-09-07T21:34:01.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:01 smithi047 ceph-mon[90087]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-09-07T21:34:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:01 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:01 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:01 smithi055 ceph-mon[97033]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:34:01.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:01 smithi055 ceph-mon[97033]: from='osd.3 [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-09-07T21:34:01.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:01 smithi055 ceph-mon[97033]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-09-07T21:34:02.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:02 smithi047 ceph-mon[90087]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-09-07T21:34:02.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:02 smithi047 ceph-mon[90087]: osdmap e23: 4 total, 3 up, 4 in 2022-09-07T21:34:02.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:34:02.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:02 smithi047 ceph-mon[90087]: from='osd.3 [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:34:02.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:02 smithi047 ceph-mon[90087]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:34:02.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:02 smithi055 ceph-mon[97033]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-09-07T21:34:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:02 smithi055 ceph-mon[97033]: osdmap e23: 4 total, 3 up, 4 in 2022-09-07T21:34:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:34:02.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:02 smithi055 ceph-mon[97033]: from='osd.3 [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:34:02.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:02 smithi055 ceph-mon[97033]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:34:02.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:02 smithi059 ceph-mon[97450]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-09-07T21:34:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:02 smithi059 ceph-mon[97450]: osdmap e23: 4 total, 3 up, 4 in 2022-09-07T21:34:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:34:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:02 smithi059 ceph-mon[97450]: from='osd.3 [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:34:02.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:02 smithi059 ceph-mon[97450]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:34:02.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:03.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:03 smithi047 ceph-mon[90087]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-09-07T21:34:03.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:03 smithi047 ceph-mon[90087]: osdmap e24: 4 total, 3 up, 4 in 2022-09-07T21:34:03.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:34:03.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:34:03.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:03 smithi047 ceph-mon[90087]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:34:03.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:03 smithi047 ceph-mon[90087]: osd.3 [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579] boot 2022-09-07T21:34:03.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:03 smithi047 ceph-mon[90087]: osdmap e25: 4 total, 4 up, 4 in 2022-09-07T21:34:03.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:34:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:03 smithi055 ceph-mon[97033]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-09-07T21:34:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:03 smithi055 ceph-mon[97033]: osdmap e24: 4 total, 3 up, 4 in 2022-09-07T21:34:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:34:03.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:34:03.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:03 smithi055 ceph-mon[97033]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:34:03.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:03 smithi055 ceph-mon[97033]: osd.3 [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579] boot 2022-09-07T21:34:03.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:03 smithi055 ceph-mon[97033]: osdmap e25: 4 total, 4 up, 4 in 2022-09-07T21:34:03.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:34:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:03 smithi059 ceph-mon[97450]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-09-07T21:34:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:03 smithi059 ceph-mon[97450]: osdmap e24: 4 total, 3 up, 4 in 2022-09-07T21:34:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:34:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:34:03.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:03 smithi059 ceph-mon[97450]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-09-07T21:34:03.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:03 smithi059 ceph-mon[97450]: osd.3 [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579] boot 2022-09-07T21:34:03.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:03 smithi059 ceph-mon[97450]: osdmap e25: 4 total, 4 up, 4 in 2022-09-07T21:34:03.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:34:04.242 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 3 on host 'smithi047' 2022-09-07T21:34:04.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:34:04.239+0000 7f8ecf7fe700 1 -- 172.21.15.47:0/1639077181 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8ee005fb30 con 0x7f8ec8099090 2022-09-07T21:34:04.244 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:34:04.242+0000 7f8ee4da8700 1 -- 172.21.15.47:0/1639077181 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8ec8099090 msgr2=0x7f8ec809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:34:04.245 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:34:04.242+0000 7f8ee4da8700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8ec8099090 0x7f8ec809b540 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4005fd0 tx=0x7f8ed4005ea0 comp rx=0 tx=0).stop 2022-09-07T21:34:04.245 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:34:04.242+0000 7f8ee4da8700 1 -- 172.21.15.47:0/1639077181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ee0071650 msgr2=0x7f8ee0195c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:34:04.246 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:34:04.242+0000 7f8ee4da8700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ee0071650 0x7f8ee0195c70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8ed000f690 tx=0x7f8ed000f6c0 comp rx=0 tx=0).stop 2022-09-07T21:34:04.246 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:34:04.242+0000 7f8ee4da8700 1 -- 172.21.15.47:0/1639077181 shutdown_connections 2022-09-07T21:34:04.246 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:34:04.242+0000 7f8ee4da8700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8ec8099090 0x7f8ec809b540 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:04.247 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:34:04.242+0000 7f8ee4da8700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee0105d30 0x7f8ee019d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:04.247 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:34:04.242+0000 7f8ee4da8700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ee0071650 0x7f8ee0195c70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:04.247 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:34:04.242+0000 7f8ee4da8700 1 --2- 172.21.15.47:0/1639077181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ee0072c30 0x7f8ee01961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:04.247 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:34:04.243+0000 7f8ee4da8700 1 -- 172.21.15.47:0/1639077181 >> 172.21.15.47:0/1639077181 conn(0x7f8ee00f9570 msgr2=0x7f8ee00fb950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:34:04.247 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:34:04.243+0000 7f8ee4da8700 1 -- 172.21.15.47:0/1639077181 shutdown_connections 2022-09-07T21:34:04.248 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:34:04.243+0000 7f8ee4da8700 1 -- 172.21.15.47:0/1639077181 wait complete. 2022-09-07T21:34:04.827 DEBUG:teuthology.orchestra.run.smithi047:osd.3> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@osd.3.service 2022-09-07T21:34:04.833 INFO:tasks.cephadm:Deploying osd.4 on smithi055 with /dev/vg_nvme/lv_3... 2022-09-07T21:34:04.833 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-09-07T21:34:04.927 INFO:journalctl@ceph.osd.3.smithi047.stdout:-- Logs begin at Wed 2022-09-07 21:11:20 UTC. -- 2022-09-07T21:34:05.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:04 smithi047 ceph-mon[90087]: purged_snaps scrub starts 2022-09-07T21:34:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:04 smithi047 ceph-mon[90087]: purged_snaps scrub ok 2022-09-07T21:34:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:04 smithi047 ceph-mon[90087]: osdmap e26: 4 total, 4 up, 4 in 2022-09-07T21:34:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:05.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:05.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:05.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:05.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:05.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:04 smithi047 ceph-mon[90087]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:04 smithi055 ceph-mon[97033]: purged_snaps scrub starts 2022-09-07T21:34:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:04 smithi055 ceph-mon[97033]: purged_snaps scrub ok 2022-09-07T21:34:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:04 smithi055 ceph-mon[97033]: osdmap e26: 4 total, 4 up, 4 in 2022-09-07T21:34:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:05.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:05.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:05.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:04 smithi055 ceph-mon[97033]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:04 smithi059 ceph-mon[97450]: purged_snaps scrub starts 2022-09-07T21:34:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:04 smithi059 ceph-mon[97450]: purged_snaps scrub ok 2022-09-07T21:34:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:04 smithi059 ceph-mon[97450]: osdmap e26: 4 total, 4 up, 4 in 2022-09-07T21:34:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:05.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:05.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:05.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:05.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:04 smithi059 ceph-mon[97450]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:05.444 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.b/config 2022-09-07T21:34:06.071 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:05 smithi047 ceph-mon[90087]: osdmap e27: 4 total, 4 up, 4 in 2022-09-07T21:34:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:05 smithi055 ceph-mon[97033]: osdmap e27: 4 total, 4 up, 4 in 2022-09-07T21:34:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:05 smithi059 ceph-mon[97450]: osdmap e27: 4 total, 4 up, 4 in 2022-09-07T21:34:07.121 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-09-07T21:34:07.121 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-09-07T21:34:07.122 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2022-09-07T21:34:07.122 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2022-09-07T21:34:07.122 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144373 s, 726 MB/s 2022-09-07T21:34:07.123 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2022-09-07T21:34:07.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:06 smithi047 ceph-mon[90087]: osdmap e28: 4 total, 4 up, 4 in 2022-09-07T21:34:07.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:06 smithi047 ceph-mon[90087]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:06 smithi055 ceph-mon[97033]: osdmap e28: 4 total, 4 up, 4 in 2022-09-07T21:34:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:06 smithi055 ceph-mon[97033]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:06 smithi059 ceph-mon[97450]: osdmap e28: 4 total, 4 up, 4 in 2022-09-07T21:34:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:06 smithi059 ceph-mon[97450]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:07.580 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi055:vg_nvme/lv_3 2022-09-07T21:34:08.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:07 smithi047 ceph-mon[90087]: osdmap e29: 4 total, 4 up, 4 in 2022-09-07T21:34:08.182 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.b/config 2022-09-07T21:34:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:07 smithi055 ceph-mon[97033]: osdmap e29: 4 total, 4 up, 4 in 2022-09-07T21:34:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:07 smithi059 ceph-mon[97450]: osdmap e29: 4 total, 4 up, 4 in 2022-09-07T21:34:09.404 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:09 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi047 2022-09-07T21:34:09.404 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:09 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:09.405 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:09 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:09.405 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:09 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:09.406 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:09 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:09.406 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:09 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi047 to 8638M 2022-09-07T21:34:09.406 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:09 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:09.407 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:09 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:09.407 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:09 smithi055 ceph-mon[97033]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2022-09-07T21:34:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:09 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi047 2022-09-07T21:34:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:09 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:09 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:09 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:09 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:09.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:09 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi047 to 8638M 2022-09-07T21:34:09.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:09 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:09.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:09 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:09.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:09 smithi059 ceph-mon[97450]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2022-09-07T21:34:09.555 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.552+0000 7f12d751f700 1 -- 172.21.15.55:0/1241699360 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12d00fdfc0 msgr2=0x7f12d00fe430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:34:09.555 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.552+0000 7f12d751f700 1 --2- 172.21.15.55:0/1241699360 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12d00fdfc0 0x7f12d00fe430 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f12c0009810 tx=0x7f12c0009b20 comp rx=0 tx=0).stop 2022-09-07T21:34:09.556 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.552+0000 7f12d751f700 1 -- 172.21.15.55:0/1241699360 shutdown_connections 2022-09-07T21:34:09.556 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.552+0000 7f12d751f700 1 --2- 172.21.15.55:0/1241699360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12d00fe970 0x7f12d0105210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:09.556 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.552+0000 7f12d751f700 1 --2- 172.21.15.55:0/1241699360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12d00fcde0 0x7f12d00fd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:09.556 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.552+0000 7f12d751f700 1 --2- 172.21.15.55:0/1241699360 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12d00fdfc0 0x7f12d00fe430 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:09.556 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.552+0000 7f12d751f700 1 -- 172.21.15.55:0/1241699360 >> 172.21.15.55:0/1241699360 conn(0x7f12d00f8430 msgr2=0x7f12d00fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:34:09.557 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.553+0000 7f12d751f700 1 -- 172.21.15.55:0/1241699360 shutdown_connections 2022-09-07T21:34:09.557 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.553+0000 7f12d751f700 1 -- 172.21.15.55:0/1241699360 wait complete. 2022-09-07T21:34:09.560 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.553+0000 7f12d751f700 1 Processor -- start 2022-09-07T21:34:09.560 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.553+0000 7f12d751f700 1 -- start start 2022-09-07T21:34:09.560 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d751f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12d00fcde0 0x7f12d0067230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:34:09.560 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d751f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12d00fdfc0 0x7f12d0198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:34:09.561 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d751f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12d00fe970 0x7f12d019ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:34:09.561 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d751f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d019b5a0 con 0x7f12d00fdfc0 2022-09-07T21:34:09.561 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d751f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d019b710 con 0x7f12d00fe970 2022-09-07T21:34:09.561 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d751f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d019b8d0 con 0x7f12d00fcde0 2022-09-07T21:34:09.561 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d5abc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12d00fe970 0x7f12d019ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:34:09.562 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d5abc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12d00fe970 0x7f12d019ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53410/0 (socket says 172.21.15.55:53410) 2022-09-07T21:34:09.562 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d5abc700 1 -- 172.21.15.55:0/850373901 learned_addr learned my addr 172.21.15.55:0/850373901 (peer_addr_for_me v2:172.21.15.55:0/0) 2022-09-07T21:34:09.562 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d4aba700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12d00fdfc0 0x7f12d0198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:34:09.562 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d52bb700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12d00fcde0 0x7f12d0067230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:34:09.563 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d5abc700 1 -- 172.21.15.55:0/850373901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12d00fcde0 msgr2=0x7f12d0067230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:34:09.563 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d5abc700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12d00fcde0 0x7f12d0067230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:09.563 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d5abc700 1 -- 172.21.15.55:0/850373901 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12d00fdfc0 msgr2=0x7f12d0198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:34:09.563 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d5abc700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12d00fdfc0 0x7f12d0198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:09.564 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d5abc700 1 -- 172.21.15.55:0/850373901 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c0009470 con 0x7f12d00fe970 2022-09-07T21:34:09.564 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d4aba700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12d00fdfc0 0x7f12d0198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:34:09.564 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d52bb700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12d00fcde0 0x7f12d0067230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-07T21:34:09.564 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d5abc700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12d00fe970 0x7f12d019ae00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f12c800e910 tx=0x7f12c800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:34:09.564 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12c67fc700 1 -- 172.21.15.55:0/850373901 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c8004c30 con 0x7f12d00fe970 2022-09-07T21:34:09.565 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12c67fc700 1 -- 172.21.15.55:0/850373901 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f12c800ce10 con 0x7f12d00fe970 2022-09-07T21:34:09.565 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.554+0000 7f12d751f700 1 -- 172.21.15.55:0/850373901 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d019bbb0 con 0x7f12d00fe970 2022-09-07T21:34:09.565 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.555+0000 7f12c67fc700 1 -- 172.21.15.55:0/850373901 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c8005810 con 0x7f12d00fe970 2022-09-07T21:34:09.565 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.555+0000 7f12d751f700 1 -- 172.21.15.55:0/850373901 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d019c140 con 0x7f12d00fe970 2022-09-07T21:34:09.565 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.555+0000 7f12d751f700 1 -- 172.21.15.55:0/850373901 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d01084d0 con 0x7f12d00fe970 2022-09-07T21:34:09.566 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.556+0000 7f12c67fc700 1 -- 172.21.15.55:0/850373901 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f12c800c970 con 0x7f12d00fe970 2022-09-07T21:34:09.566 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.557+0000 7f12c67fc700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f12bc099010 0x7f12bc09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:34:09.566 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.557+0000 7f12c67fc700 1 -- 172.21.15.55:0/850373901 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3665+0+0 (secure 0 0 0) 0x7f12c80c17e0 con 0x7f12d00fe970 2022-09-07T21:34:09.566 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.558+0000 7f12d52bb700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f12bc099010 0x7f12bc09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:34:09.567 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.558+0000 7f12d52bb700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f12bc099010 0x7f12bc09b4c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f12cc00a810 tx=0x7f12cc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:34:09.567 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.560+0000 7f12c67fc700 1 -- 172.21.15.55:0/850373901 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f12c808f140 con 0x7f12d00fe970 2022-09-07T21:34:09.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:09 smithi047 ceph-mon[90087]: Detected new or changed devices on smithi047 2022-09-07T21:34:09.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:09 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:09.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:09 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:09.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:09 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:09.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:09 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:09.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:09 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi047 to 8638M 2022-09-07T21:34:09.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:09 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:09.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:09 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:09.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:09 smithi047 ceph-mon[90087]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2022-09-07T21:34:09.768 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:09.766+0000 7f12d751f700 1 -- 172.21.15.55:0/850373901 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f12d005f120 con 0x7f12bc099010 2022-09-07T21:34:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:10 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:34:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:10 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:34:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:10 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:10.620 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:10 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:34:10.621 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:10 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:34:10.621 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:10 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:10.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:10 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:34:10.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:10 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:34:10.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:10 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:11 smithi055 ceph-mon[97033]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:34:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:11 smithi055 ceph-mon[97033]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-07T21:34:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:11 smithi059 ceph-mon[97450]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:34:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:11 smithi059 ceph-mon[97450]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-07T21:34:11.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:11 smithi047 ceph-mon[90087]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:34:11.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:11 smithi047 ceph-mon[90087]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-07T21:34:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:12 smithi055 ceph-mon[97033]: from='client.? 172.21.15.55:0/4145525311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f98aac6c-62be-4f57-94e1-4baa606d9e9e"}]: dispatch 2022-09-07T21:34:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:12 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f98aac6c-62be-4f57-94e1-4baa606d9e9e"}]: dispatch 2022-09-07T21:34:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:12 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f98aac6c-62be-4f57-94e1-4baa606d9e9e"}]': finished 2022-09-07T21:34:12.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:12 smithi055 ceph-mon[97033]: osdmap e30: 5 total, 4 up, 5 in 2022-09-07T21:34:12.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:12 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:12 smithi059 ceph-mon[97450]: from='client.? 172.21.15.55:0/4145525311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f98aac6c-62be-4f57-94e1-4baa606d9e9e"}]: dispatch 2022-09-07T21:34:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:12 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f98aac6c-62be-4f57-94e1-4baa606d9e9e"}]: dispatch 2022-09-07T21:34:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:12 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f98aac6c-62be-4f57-94e1-4baa606d9e9e"}]': finished 2022-09-07T21:34:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:12 smithi059 ceph-mon[97450]: osdmap e30: 5 total, 4 up, 5 in 2022-09-07T21:34:12.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:12 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:12.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:12 smithi047 ceph-mon[90087]: from='client.? 172.21.15.55:0/4145525311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f98aac6c-62be-4f57-94e1-4baa606d9e9e"}]: dispatch 2022-09-07T21:34:12.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:12 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f98aac6c-62be-4f57-94e1-4baa606d9e9e"}]: dispatch 2022-09-07T21:34:12.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:12 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f98aac6c-62be-4f57-94e1-4baa606d9e9e"}]': finished 2022-09-07T21:34:12.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:12 smithi047 ceph-mon[90087]: osdmap e30: 5 total, 4 up, 5 in 2022-09-07T21:34:12.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:12 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:13 smithi055 ceph-mon[97033]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2022-09-07T21:34:13.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:13 smithi055 ceph-mon[97033]: from='client.? 172.21.15.55:0/771599826' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:34:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:13 smithi059 ceph-mon[97450]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2022-09-07T21:34:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:13 smithi059 ceph-mon[97450]: from='client.? 172.21.15.55:0/771599826' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:34:13.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:13 smithi047 ceph-mon[90087]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2022-09-07T21:34:13.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:13 smithi047 ceph-mon[90087]: from='client.? 172.21.15.55:0/771599826' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:34:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:15 smithi047 ceph-mon[90087]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-07T21:34:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:15 smithi055 ceph-mon[97033]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-07T21:34:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:15 smithi059 ceph-mon[97450]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-07T21:34:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:17 smithi055 ceph-mon[97033]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2022-09-07T21:34:17.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:17 smithi047 ceph-mon[90087]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2022-09-07T21:34:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:17 smithi059 ceph-mon[97450]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2022-09-07T21:34:19.147 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:18 smithi055 ceph-mon[97033]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:19.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:18 smithi047 ceph-mon[90087]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:18 smithi059 ceph-mon[97450]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:20.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:19 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-09-07T21:34:20.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:19 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:20.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:19 smithi047 ceph-mon[90087]: Deploying daemon osd.4 on smithi055 2022-09-07T21:34:20.204 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:19 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-09-07T21:34:20.205 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:19 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:20.205 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:19 smithi055 ceph-mon[97033]: Deploying daemon osd.4 on smithi055 2022-09-07T21:34:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:19 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-09-07T21:34:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:19 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:19 smithi059 ceph-mon[97450]: Deploying daemon osd.4 on smithi055 2022-09-07T21:34:21.117 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:20 smithi055 ceph-mon[97033]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:20 smithi059 ceph-mon[97450]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:21.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:20 smithi047 ceph-mon[90087]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:23.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:23 smithi047 ceph-mon[90087]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:23 smithi055 ceph-mon[97033]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:23 smithi059 ceph-mon[97450]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:25 smithi055 ceph-mon[97033]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:25.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:25.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:25.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:25 smithi047 ceph-mon[90087]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:25.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:25.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:25.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:25.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:25.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:25 smithi059 ceph-mon[97450]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:26.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:26.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:27.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:27 smithi047 ceph-mon[90087]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:27.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:27.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:27.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:27 smithi055 ceph-mon[97033]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:27.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:27.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:27 smithi059 ceph-mon[97450]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:28.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:29.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:28 smithi047 ceph-mon[90087]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:28 smithi055 ceph-mon[97033]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:28 smithi059 ceph-mon[97450]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:29.388 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 4 on host 'smithi055' 2022-09-07T21:34:29.389 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:29.384+0000 7f12c67fc700 1 -- 172.21.15.55:0/850373901 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f12d005f120 con 0x7f12bc099010 2022-09-07T21:34:29.391 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:29.387+0000 7f12d751f700 1 -- 172.21.15.55:0/850373901 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f12bc099010 msgr2=0x7f12bc09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:34:29.391 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:29.387+0000 7f12d751f700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f12bc099010 0x7f12bc09b4c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f12cc00a810 tx=0x7f12cc008040 comp rx=0 tx=0).stop 2022-09-07T21:34:29.392 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:29.387+0000 7f12d751f700 1 -- 172.21.15.55:0/850373901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12d00fe970 msgr2=0x7f12d019ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:34:29.392 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:29.387+0000 7f12d751f700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12d00fe970 0x7f12d019ae00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f12c800e910 tx=0x7f12c800ecd0 comp rx=0 tx=0).stop 2022-09-07T21:34:29.393 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:29.387+0000 7f12d751f700 1 -- 172.21.15.55:0/850373901 shutdown_connections 2022-09-07T21:34:29.393 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:29.387+0000 7f12d751f700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f12bc099010 0x7f12bc09b4c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:29.393 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:29.387+0000 7f12d751f700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12d00fe970 0x7f12d019ae00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:29.394 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:29.387+0000 7f12d751f700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12d00fcde0 0x7f12d0067230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:29.394 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:29.387+0000 7f12d751f700 1 --2- 172.21.15.55:0/850373901 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12d00fdfc0 0x7f12d0198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:29.395 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:29.387+0000 7f12d751f700 1 -- 172.21.15.55:0/850373901 >> 172.21.15.55:0/850373901 conn(0x7f12d00f8430 msgr2=0x7f12d00fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:34:29.395 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:29.387+0000 7f12d751f700 1 -- 172.21.15.55:0/850373901 shutdown_connections 2022-09-07T21:34:29.396 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:34:29.387+0000 7f12d751f700 1 -- 172.21.15.55:0/850373901 wait complete. 2022-09-07T21:34:29.842 DEBUG:teuthology.orchestra.run.smithi055:osd.4> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@osd.4.service 2022-09-07T21:34:29.845 INFO:tasks.cephadm:Deploying osd.5 on smithi059 with /dev/vg_nvme/lv_3... 2022-09-07T21:34:29.845 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-09-07T21:34:29.877 INFO:journalctl@ceph.osd.4.smithi055.stdout:-- Logs begin at Wed 2022-09-07 21:07:35 UTC. -- 2022-09-07T21:34:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:29 smithi055 ceph-mon[97033]: from='osd.4 [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-09-07T21:34:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:29 smithi055 ceph-mon[97033]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-09-07T21:34:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:29 smithi059 ceph-mon[97450]: from='osd.4 [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-09-07T21:34:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:29 smithi059 ceph-mon[97450]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-09-07T21:34:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:30.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:30.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:30.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:30.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:29 smithi047 ceph-mon[90087]: from='osd.4 [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-09-07T21:34:30.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:29 smithi047 ceph-mon[90087]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-09-07T21:34:30.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:30.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:30.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:30.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:30.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:30.455 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.c/config 2022-09-07T21:34:30.980 INFO:journalctl@ceph.osd.4.smithi055.stdout:Sep 07 21:34:30 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-4[108024]: 2022-09-07T21:34:30.932+0000 7f4c6b10c700 -1 osd.4 0 waiting for initial osdmap 2022-09-07T21:34:30.980 INFO:journalctl@ceph.osd.4.smithi055.stdout:Sep 07 21:34:30 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-4[108024]: 2022-09-07T21:34:30.939+0000 7f4c636fa700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-07T21:34:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:30 smithi055 ceph-mon[97033]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-09-07T21:34:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:30 smithi055 ceph-mon[97033]: osdmap e31: 5 total, 4 up, 5 in 2022-09-07T21:34:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:30 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:30 smithi055 ceph-mon[97033]: from='osd.4 [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:34:30.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:30 smithi055 ceph-mon[97033]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:34:30.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:30 smithi055 ceph-mon[97033]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:31.223 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:30 smithi059 ceph-mon[97450]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-09-07T21:34:31.223 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:30 smithi059 ceph-mon[97450]: osdmap e31: 5 total, 4 up, 5 in 2022-09-07T21:34:31.224 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:30 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:31.224 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:30 smithi059 ceph-mon[97450]: from='osd.4 [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:34:31.224 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:30 smithi059 ceph-mon[97450]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:34:31.225 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:30 smithi059 ceph-mon[97450]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:31.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:30 smithi047 ceph-mon[90087]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-09-07T21:34:31.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:30 smithi047 ceph-mon[90087]: osdmap e31: 5 total, 4 up, 5 in 2022-09-07T21:34:31.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:30 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:31.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:30 smithi047 ceph-mon[90087]: from='osd.4 [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:34:31.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:30 smithi047 ceph-mon[90087]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:34:31.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:30 smithi047 ceph-mon[90087]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:32.051 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-09-07T21:34:32.140 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-09-07T21:34:32.140 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2022-09-07T21:34:32.140 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2022-09-07T21:34:32.140 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108133 s, 970 MB/s 2022-09-07T21:34:32.141 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2022-09-07T21:34:32.180 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:31 smithi055 ceph-mon[97033]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2022-09-07T21:34:32.181 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:31 smithi055 ceph-mon[97033]: osdmap e32: 5 total, 4 up, 5 in 2022-09-07T21:34:32.181 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:31 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:32.182 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:31 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:32.267 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:31 smithi047 ceph-mon[90087]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2022-09-07T21:34:32.268 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:31 smithi047 ceph-mon[90087]: osdmap e32: 5 total, 4 up, 5 in 2022-09-07T21:34:32.268 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:31 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:32.268 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:31 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:31 smithi059 ceph-mon[97450]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2022-09-07T21:34:32.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:31 smithi059 ceph-mon[97450]: osdmap e32: 5 total, 4 up, 5 in 2022-09-07T21:34:32.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:31 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:32.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:31 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:32.583 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi059:vg_nvme/lv_3 2022-09-07T21:34:33.182 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.c/config 2022-09-07T21:34:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:32 smithi055 ceph-mon[97033]: purged_snaps scrub starts 2022-09-07T21:34:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:32 smithi055 ceph-mon[97033]: purged_snaps scrub ok 2022-09-07T21:34:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:32 smithi055 ceph-mon[97033]: osd.4 [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956] boot 2022-09-07T21:34:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:32 smithi055 ceph-mon[97033]: osdmap e33: 5 total, 5 up, 5 in 2022-09-07T21:34:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:32 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-07T21:34:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:33.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:33.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:33.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:33.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:32 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 8586M 2022-09-07T21:34:33.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:33.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:33.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:32 smithi055 ceph-mon[97033]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:32 smithi059 ceph-mon[97450]: purged_snaps scrub starts 2022-09-07T21:34:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:32 smithi059 ceph-mon[97450]: purged_snaps scrub ok 2022-09-07T21:34:33.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:32 smithi059 ceph-mon[97450]: osd.4 [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956] boot 2022-09-07T21:34:33.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:32 smithi059 ceph-mon[97450]: osdmap e33: 5 total, 5 up, 5 in 2022-09-07T21:34:33.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:33.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:32 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-07T21:34:33.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:33.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:33.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:33.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:33.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:32 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 8586M 2022-09-07T21:34:33.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:33.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:33.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:32 smithi059 ceph-mon[97450]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:33.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:32 smithi047 ceph-mon[90087]: purged_snaps scrub starts 2022-09-07T21:34:33.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:32 smithi047 ceph-mon[90087]: purged_snaps scrub ok 2022-09-07T21:34:33.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:32 smithi047 ceph-mon[90087]: osd.4 [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956] boot 2022-09-07T21:34:33.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:32 smithi047 ceph-mon[90087]: osdmap e33: 5 total, 5 up, 5 in 2022-09-07T21:34:33.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:34:33.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:32 smithi047 ceph-mon[90087]: Detected new or changed devices on smithi055 2022-09-07T21:34:33.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:33.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:33.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:33.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:33.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:32 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi055 to 8586M 2022-09-07T21:34:33.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:33.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:33.407 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:32 smithi047 ceph-mon[90087]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-09-07T21:34:33.989 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:33 smithi059 ceph-mon[97450]: osdmap e34: 5 total, 5 up, 5 in 2022-09-07T21:34:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:33 smithi055 ceph-mon[97033]: osdmap e34: 5 total, 5 up, 5 in 2022-09-07T21:34:34.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:33 smithi047 ceph-mon[90087]: osdmap e34: 5 total, 5 up, 5 in 2022-09-07T21:34:34.426 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.423+0000 7f11d334f700 1 -- 172.21.15.59:0/968223667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11cc0fcde0 msgr2=0x7f11cc0fd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:34:34.427 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.423+0000 7f11d334f700 1 --2- 172.21.15.59:0/968223667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11cc0fcde0 0x7f11cc0fd1f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009810 tx=0x7f11bc009b20 comp rx=0 tx=0).stop 2022-09-07T21:34:34.427 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.423+0000 7f11d334f700 1 -- 172.21.15.59:0/968223667 shutdown_connections 2022-09-07T21:34:34.427 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.423+0000 7f11d334f700 1 --2- 172.21.15.59:0/968223667 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11cc0fe970 0x7f11cc105210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:34.427 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.423+0000 7f11d334f700 1 --2- 172.21.15.59:0/968223667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11cc0fcde0 0x7f11cc0fd1f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:34.428 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.423+0000 7f11d334f700 1 --2- 172.21.15.59:0/968223667 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11cc0fdfc0 0x7f11cc0fe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:34.428 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.423+0000 7f11d334f700 1 -- 172.21.15.59:0/968223667 >> 172.21.15.59:0/968223667 conn(0x7f11cc0f8430 msgr2=0x7f11cc0fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:34:34.428 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.423+0000 7f11d334f700 1 -- 172.21.15.59:0/968223667 shutdown_connections 2022-09-07T21:34:34.428 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.424+0000 7f11d334f700 1 -- 172.21.15.59:0/968223667 wait complete. 2022-09-07T21:34:34.431 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.424+0000 7f11d334f700 1 Processor -- start 2022-09-07T21:34:34.432 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.424+0000 7f11d334f700 1 -- start start 2022-09-07T21:34:34.432 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.424+0000 7f11d334f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11cc0fcde0 0x7f11cc193a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:34:34.432 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.424+0000 7f11d334f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11cc0fdfc0 0x7f11cc193f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:34:34.432 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.424+0000 7f11d334f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11cc0fe970 0x7f11cc19afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:34:34.433 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.424+0000 7f11d334f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc19b500 con 0x7f11cc0fe970 2022-09-07T21:34:34.433 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.424+0000 7f11d334f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc19b670 con 0x7f11cc0fdfc0 2022-09-07T21:34:34.433 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.424+0000 7f11d334f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc19b830 con 0x7f11cc0fcde0 2022-09-07T21:34:34.433 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.424+0000 7f11d10eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11cc0fcde0 0x7f11cc193a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:34:34.433 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.424+0000 7f11d10eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11cc0fcde0 0x7f11cc193a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40240/0 (socket says 172.21.15.59:40240) 2022-09-07T21:34:34.434 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.425+0000 7f11d10eb700 1 -- 172.21.15.59:0/868078309 learned_addr learned my addr 172.21.15.59:0/868078309 (peer_addr_for_me v2:172.21.15.59:0/0) 2022-09-07T21:34:34.434 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.425+0000 7f11d08ea700 1 --2- 172.21.15.59:0/868078309 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11cc0fdfc0 0x7f11cc193f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:34:34.434 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.425+0000 7f11d10eb700 1 -- 172.21.15.59:0/868078309 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11cc0fdfc0 msgr2=0x7f11cc193f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:34:34.434 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.425+0000 7f11d10eb700 1 --2- 172.21.15.59:0/868078309 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11cc0fdfc0 0x7f11cc193f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:34.435 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.425+0000 7f11d10eb700 1 -- 172.21.15.59:0/868078309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11cc0fe970 msgr2=0x7f11cc19afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:34:34.435 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.425+0000 7f11d10eb700 1 --2- 172.21.15.59:0/868078309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11cc0fe970 0x7f11cc19afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:34.435 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.425+0000 7f11d10eb700 1 -- 172.21.15.59:0/868078309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc009470 con 0x7f11cc0fcde0 2022-09-07T21:34:34.435 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.425+0000 7f11d08ea700 1 --2- 172.21.15.59:0/868078309 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11cc0fdfc0 0x7f11cc193f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-07T21:34:34.435 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.425+0000 7f11d10eb700 1 --2- 172.21.15.59:0/868078309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11cc0fcde0 0x7f11cc193a10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f11bc00f6c0 tx=0x7f11bc00f7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:34:34.436 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.425+0000 7f11c27fc700 1 -- 172.21.15.59:0/868078309 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc01d070 con 0x7f11cc0fcde0 2022-09-07T21:34:34.436 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.425+0000 7f11c27fc700 1 -- 172.21.15.59:0/868078309 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f11bc005760 con 0x7f11cc0fcde0 2022-09-07T21:34:34.436 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.425+0000 7f11c27fc700 1 -- 172.21.15.59:0/868078309 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc004ba0 con 0x7f11cc0fcde0 2022-09-07T21:34:34.436 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.425+0000 7f11d334f700 1 -- 172.21.15.59:0/868078309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11cc19bab0 con 0x7f11cc0fcde0 2022-09-07T21:34:34.437 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.425+0000 7f11d334f700 1 -- 172.21.15.59:0/868078309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11cc19bf10 con 0x7f11cc0fcde0 2022-09-07T21:34:34.437 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.426+0000 7f11d334f700 1 -- 172.21.15.59:0/868078309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11cc1084b0 con 0x7f11cc0fcde0 2022-09-07T21:34:34.437 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.427+0000 7f11c27fc700 1 -- 172.21.15.59:0/868078309 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f11bc028030 con 0x7f11cc0fcde0 2022-09-07T21:34:34.437 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.428+0000 7f11c27fc700 1 --2- 172.21.15.59:0/868078309 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f11b8099090 0x7f11b809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:34:34.438 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.428+0000 7f11c27fc700 1 -- 172.21.15.59:0/868078309 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4097+0+0 (secure 0 0 0) 0x7f11bc0c27b0 con 0x7f11cc0fcde0 2022-09-07T21:34:34.438 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.428+0000 7f11d08ea700 1 --2- 172.21.15.59:0/868078309 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f11b8099090 0x7f11b809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:34:34.438 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.429+0000 7f11d08ea700 1 --2- 172.21.15.59:0/868078309 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f11b8099090 0x7f11b809b540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f11c800ba20 tx=0x7f11c8005ca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:34:34.439 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.431+0000 7f11c27fc700 1 -- 172.21.15.59:0/868078309 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f11bc08fe70 con 0x7f11cc0fcde0 2022-09-07T21:34:34.637 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:34.635+0000 7f11d334f700 1 -- 172.21.15.59:0/868078309 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f11cc066e10 con 0x7f11b8099090 2022-09-07T21:34:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:34 smithi055 ceph-mon[97033]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:34 smithi055 ceph-mon[97033]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:34:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:34 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:34:35.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:34 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:34:35.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:34 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:34 smithi059 ceph-mon[97450]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:34 smithi059 ceph-mon[97450]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:34:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:34 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:34:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:34 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:34:35.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:34 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:35.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:34 smithi047 ceph-mon[90087]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:35.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:34 smithi047 ceph-mon[90087]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:34:35.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:34 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:34:35.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:34 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:34:35.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:34 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:37.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:37 smithi047 ceph-mon[90087]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:37.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:37 smithi047 ceph-mon[90087]: from='client.? 172.21.15.59:0/1199179657' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2b864da-e60b-4140-a6d9-30e4e375ef6e"}]: dispatch 2022-09-07T21:34:37.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:37 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2b864da-e60b-4140-a6d9-30e4e375ef6e"}]: dispatch 2022-09-07T21:34:37.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:37 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2b864da-e60b-4140-a6d9-30e4e375ef6e"}]': finished 2022-09-07T21:34:37.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:37 smithi047 ceph-mon[90087]: osdmap e35: 6 total, 5 up, 6 in 2022-09-07T21:34:37.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:37 smithi055 ceph-mon[97033]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:37 smithi055 ceph-mon[97033]: from='client.? 172.21.15.59:0/1199179657' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2b864da-e60b-4140-a6d9-30e4e375ef6e"}]: dispatch 2022-09-07T21:34:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:37 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2b864da-e60b-4140-a6d9-30e4e375ef6e"}]: dispatch 2022-09-07T21:34:37.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:37 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2b864da-e60b-4140-a6d9-30e4e375ef6e"}]': finished 2022-09-07T21:34:37.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:37 smithi055 ceph-mon[97033]: osdmap e35: 6 total, 5 up, 6 in 2022-09-07T21:34:37.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:37 smithi059 ceph-mon[97450]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:37 smithi059 ceph-mon[97450]: from='client.? 172.21.15.59:0/1199179657' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2b864da-e60b-4140-a6d9-30e4e375ef6e"}]: dispatch 2022-09-07T21:34:38.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:37 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2b864da-e60b-4140-a6d9-30e4e375ef6e"}]: dispatch 2022-09-07T21:34:38.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:37 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2b864da-e60b-4140-a6d9-30e4e375ef6e"}]': finished 2022-09-07T21:34:38.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:37 smithi059 ceph-mon[97450]: osdmap e35: 6 total, 5 up, 6 in 2022-09-07T21:34:38.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:38.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:38 smithi047 ceph-mon[90087]: from='client.? 172.21.15.59:0/2609994510' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:34:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:38 smithi055 ceph-mon[97033]: from='client.? 172.21.15.59:0/2609994510' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:34:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:38 smithi059 ceph-mon[97450]: from='client.? 172.21.15.59:0/2609994510' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:34:39.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:39 smithi047 ceph-mon[90087]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:39 smithi055 ceph-mon[97033]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:39 smithi059 ceph-mon[97450]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:41 smithi059 ceph-mon[97450]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:41.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:41 smithi047 ceph-mon[90087]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:41 smithi055 ceph-mon[97033]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:43.867 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:43 smithi059 ceph-mon[97450]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:43.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:43 smithi047 ceph-mon[90087]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:43 smithi055 ceph-mon[97033]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:44.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:44 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-09-07T21:34:44.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:44 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:44.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:44 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-09-07T21:34:44.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:44 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:44 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-09-07T21:34:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:44 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:45.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:45 smithi047 ceph-mon[90087]: Deploying daemon osd.5 on smithi059 2022-09-07T21:34:45.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:45 smithi047 ceph-mon[90087]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:45 smithi055 ceph-mon[97033]: Deploying daemon osd.5 on smithi059 2022-09-07T21:34:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:45 smithi055 ceph-mon[97033]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:45.997 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:45 smithi059 ceph-mon[97450]: Deploying daemon osd.5 on smithi059 2022-09-07T21:34:45.997 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:45 smithi059 ceph-mon[97450]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:46.628 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:34:46.629 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:34:46.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:34:46.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:34:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:34:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:34:47.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:47 smithi047 ceph-mon[90087]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:47 smithi055 ceph-mon[97033]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:47 smithi059 ceph-mon[97450]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:48 smithi055 ceph-mon[97033]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:48 smithi059 ceph-mon[97450]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:49.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:48 smithi047 ceph-mon[90087]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:51.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:51 smithi059 ceph-mon[97450]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:51.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:51.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:51.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:51.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:51.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:51.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:51 smithi047 ceph-mon[90087]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:51.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:51.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:51.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:51.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:51 smithi055 ceph-mon[97033]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:53 smithi059 ceph-mon[97450]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:54.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:53 smithi047 ceph-mon[90087]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:54.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:54.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:54.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:53 smithi055 ceph-mon[97033]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:54.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:54.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:54.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:54.477 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:54.474+0000 7f11c27fc700 1 -- 172.21.15.59:0/868078309 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f11cc066e10 con 0x7f11b8099090 2022-09-07T21:34:54.477 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 5 on host 'smithi059' 2022-09-07T21:34:54.479 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:54.476+0000 7f11d334f700 1 -- 172.21.15.59:0/868078309 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f11b8099090 msgr2=0x7f11b809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:34:54.480 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:54.476+0000 7f11d334f700 1 --2- 172.21.15.59:0/868078309 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f11b8099090 0x7f11b809b540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f11c800ba20 tx=0x7f11c8005ca0 comp rx=0 tx=0).stop 2022-09-07T21:34:54.480 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:54.476+0000 7f11d334f700 1 -- 172.21.15.59:0/868078309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11cc0fcde0 msgr2=0x7f11cc193a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:34:54.480 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:54.476+0000 7f11d334f700 1 --2- 172.21.15.59:0/868078309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11cc0fcde0 0x7f11cc193a10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f11bc00f6c0 tx=0x7f11bc00f7a0 comp rx=0 tx=0).stop 2022-09-07T21:34:54.480 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:54.476+0000 7f11d334f700 1 -- 172.21.15.59:0/868078309 shutdown_connections 2022-09-07T21:34:54.481 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:54.476+0000 7f11d334f700 1 --2- 172.21.15.59:0/868078309 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f11b8099090 0x7f11b809b540 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:54.481 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:54.476+0000 7f11d334f700 1 --2- 172.21.15.59:0/868078309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11cc0fe970 0x7f11cc19afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:54.481 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:54.476+0000 7f11d334f700 1 --2- 172.21.15.59:0/868078309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11cc0fcde0 0x7f11cc193a10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:54.482 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:54.476+0000 7f11d334f700 1 --2- 172.21.15.59:0/868078309 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11cc0fdfc0 0x7f11cc193f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:34:54.482 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:54.476+0000 7f11d334f700 1 -- 172.21.15.59:0/868078309 >> 172.21.15.59:0/868078309 conn(0x7f11cc0f8430 msgr2=0x7f11cc066700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:34:54.482 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:54.476+0000 7f11d334f700 1 -- 172.21.15.59:0/868078309 shutdown_connections 2022-09-07T21:34:54.482 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:34:54.476+0000 7f11d334f700 1 -- 172.21.15.59:0/868078309 wait complete. 2022-09-07T21:34:54.943 DEBUG:teuthology.orchestra.run.smithi059:osd.5> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@osd.5.service 2022-09-07T21:34:54.947 INFO:tasks.cephadm:Deploying osd.6 on smithi047 with /dev/vg_nvme/lv_2... 2022-09-07T21:34:54.947 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-09-07T21:34:54.977 INFO:journalctl@ceph.osd.5.smithi059.stdout:-- Logs begin at Wed 2022-09-07 21:07:37 UTC. -- 2022-09-07T21:34:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:54 smithi059 ceph-mon[97450]: from='osd.5 [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-09-07T21:34:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:54 smithi059 ceph-mon[97450]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-09-07T21:34:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:55.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:55.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:55.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:54 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:55.034 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:54 smithi047 ceph-mon[90087]: from='osd.5 [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-09-07T21:34:55.035 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:54 smithi047 ceph-mon[90087]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-09-07T21:34:55.035 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:55.035 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:55.035 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:55.036 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:55.036 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:54 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:55.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:54 smithi055 ceph-mon[97033]: from='osd.5 [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-09-07T21:34:55.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:54 smithi055 ceph-mon[97033]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-09-07T21:34:55.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:55.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:34:55.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:34:55.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:34:55.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:54 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:55.840 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:34:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:55 smithi059 ceph-mon[97450]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:55 smithi059 ceph-mon[97450]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-09-07T21:34:56.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:55 smithi059 ceph-mon[97450]: osdmap e36: 6 total, 5 up, 6 in 2022-09-07T21:34:56.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:55 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:56.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:55 smithi059 ceph-mon[97450]: from='osd.5 [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:34:56.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:55 smithi059 ceph-mon[97450]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:34:56.034 INFO:journalctl@ceph.osd.5.smithi059.stdout:Sep 07 21:34:55 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-5[109228]: 2022-09-07T21:34:55.766+0000 7f37d2825700 -1 osd.5 0 waiting for initial osdmap 2022-09-07T21:34:56.035 INFO:journalctl@ceph.osd.5.smithi059.stdout:Sep 07 21:34:55 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-5[109228]: 2022-09-07T21:34:55.773+0000 7f37cae13700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-07T21:34:56.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:55 smithi047 ceph-mon[90087]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:56.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:55 smithi047 ceph-mon[90087]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-09-07T21:34:56.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:55 smithi047 ceph-mon[90087]: osdmap e36: 6 total, 5 up, 6 in 2022-09-07T21:34:56.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:55 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:56.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:55 smithi047 ceph-mon[90087]: from='osd.5 [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:34:56.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:55 smithi047 ceph-mon[90087]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:34:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:55 smithi055 ceph-mon[97033]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:55 smithi055 ceph-mon[97033]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-09-07T21:34:56.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:55 smithi055 ceph-mon[97033]: osdmap e36: 6 total, 5 up, 6 in 2022-09-07T21:34:56.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:55 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:56.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:55 smithi055 ceph-mon[97033]: from='osd.5 [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:34:56.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:55 smithi055 ceph-mon[97033]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:34:56.852 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:56 smithi047 ceph-mon[90087]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2022-09-07T21:34:56.852 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:56 smithi047 ceph-mon[90087]: osdmap e37: 6 total, 5 up, 6 in 2022-09-07T21:34:56.853 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:56 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:56.853 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:56 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:56.853 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:56 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:56 smithi059 ceph-mon[97450]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2022-09-07T21:34:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:56 smithi059 ceph-mon[97450]: osdmap e37: 6 total, 5 up, 6 in 2022-09-07T21:34:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:56 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:57.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:56 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:57.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:56 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:56 smithi055 ceph-mon[97033]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2022-09-07T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:56 smithi055 ceph-mon[97033]: osdmap e37: 6 total, 5 up, 6 in 2022-09-07T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:56 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:56 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:56 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:57.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:57 smithi047 ceph-mon[90087]: purged_snaps scrub starts 2022-09-07T21:34:57.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:57 smithi047 ceph-mon[90087]: purged_snaps scrub ok 2022-09-07T21:34:57.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:57 smithi047 ceph-mon[90087]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:57.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:57 smithi047 ceph-mon[90087]: osd.5 [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539] boot 2022-09-07T21:34:57.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:57 smithi047 ceph-mon[90087]: osdmap e38: 6 total, 6 up, 6 in 2022-09-07T21:34:57.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:57.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:57 smithi047 ceph-mon[90087]: Detected new or changed devices on smithi059 2022-09-07T21:34:57.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:57.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:57.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:57.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:57.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:57 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi059 to 8586M 2022-09-07T21:34:57.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:57.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:57 smithi055 ceph-mon[97033]: purged_snaps scrub starts 2022-09-07T21:34:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:57 smithi055 ceph-mon[97033]: purged_snaps scrub ok 2022-09-07T21:34:57.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:57 smithi055 ceph-mon[97033]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:57.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:57 smithi055 ceph-mon[97033]: osd.5 [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539] boot 2022-09-07T21:34:57.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:57 smithi055 ceph-mon[97033]: osdmap e38: 6 total, 6 up, 6 in 2022-09-07T21:34:57.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:57.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:57 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-07T21:34:57.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:57.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:57.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:57.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:57.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:57 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 8586M 2022-09-07T21:34:57.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:57.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:57 smithi059 ceph-mon[97450]: purged_snaps scrub starts 2022-09-07T21:34:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:57 smithi059 ceph-mon[97450]: purged_snaps scrub ok 2022-09-07T21:34:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:57 smithi059 ceph-mon[97450]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-09-07T21:34:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:57 smithi059 ceph-mon[97450]: osd.5 [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539] boot 2022-09-07T21:34:58.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:57 smithi059 ceph-mon[97450]: osdmap e38: 6 total, 6 up, 6 in 2022-09-07T21:34:58.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:34:58.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:57 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-07T21:34:58.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:58.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:58.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:58.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:34:58.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:57 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 8586M 2022-09-07T21:34:58.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:58.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:34:58.191 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-09-07T21:34:58.192 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-09-07T21:34:58.192 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2022-09-07T21:34:58.192 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2022-09-07T21:34:58.193 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0619696 s, 169 MB/s 2022-09-07T21:34:58.193 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2022-09-07T21:34:58.819 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi047:vg_nvme/lv_2 2022-09-07T21:34:59.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:58 smithi047 ceph-mon[90087]: osdmap e39: 6 total, 6 up, 6 in 2022-09-07T21:34:59.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:58 smithi047 ceph-mon[90087]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:34:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:58 smithi055 ceph-mon[97033]: osdmap e39: 6 total, 6 up, 6 in 2022-09-07T21:34:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:58 smithi055 ceph-mon[97033]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:34:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:58 smithi059 ceph-mon[97450]: osdmap e39: 6 total, 6 up, 6 in 2022-09-07T21:34:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:58 smithi059 ceph-mon[97450]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:34:59.704 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:35:00.092 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:34:59 smithi047 ceph-mon[90087]: osdmap e40: 6 total, 6 up, 6 in 2022-09-07T21:35:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:34:59 smithi055 ceph-mon[97033]: osdmap e40: 6 total, 6 up, 6 in 2022-09-07T21:35:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:34:59 smithi059 ceph-mon[97450]: osdmap e40: 6 total, 6 up, 6 in 2022-09-07T21:35:01.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:00 smithi047 ceph-mon[90087]: osdmap e41: 6 total, 6 up, 6 in 2022-09-07T21:35:01.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:00 smithi047 ceph-mon[90087]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:00 smithi055 ceph-mon[97033]: osdmap e41: 6 total, 6 up, 6 in 2022-09-07T21:35:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:00 smithi055 ceph-mon[97033]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:00 smithi059 ceph-mon[97450]: osdmap e41: 6 total, 6 up, 6 in 2022-09-07T21:35:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:00 smithi059 ceph-mon[97450]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:01.658 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.655+0000 7f911aa97700 1 -- 172.21.15.47:0/1777229944 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9114071740 msgr2=0x7f9114071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:35:01.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.655+0000 7f911aa97700 1 --2- 172.21.15.47:0/1777229944 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9114071740 0x7f9114071bb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9104009750 tx=0x7f9104009a60 comp rx=0 tx=0).stop 2022-09-07T21:35:01.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.655+0000 7f911aa97700 1 -- 172.21.15.47:0/1777229944 shutdown_connections 2022-09-07T21:35:01.660 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.655+0000 7f911aa97700 1 --2- 172.21.15.47:0/1777229944 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9114105b90 0x7f9114107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:01.661 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.655+0000 7f911aa97700 1 --2- 172.21.15.47:0/1777229944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9114072c90 0x7f9114071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:01.661 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.655+0000 7f911aa97700 1 --2- 172.21.15.47:0/1777229944 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9114071740 0x7f9114071bb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:01.662 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.655+0000 7f911aa97700 1 -- 172.21.15.47:0/1777229944 >> 172.21.15.47:0/1777229944 conn(0x7f91140f93a0 msgr2=0x7f91140fb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:35:01.662 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.655+0000 7f911aa97700 1 -- 172.21.15.47:0/1777229944 shutdown_connections 2022-09-07T21:35:01.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.655+0000 7f911aa97700 1 -- 172.21.15.47:0/1777229944 wait complete. 2022-09-07T21:35:01.667 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.656+0000 7f911aa97700 1 Processor -- start 2022-09-07T21:35:01.668 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.657+0000 7f911aa97700 1 -- start start 2022-09-07T21:35:01.668 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.657+0000 7f911aa97700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9114071740 0x7f9114195ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:35:01.669 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.657+0000 7f911aa97700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9114072c90 0x7f9114196000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:35:01.669 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.657+0000 7f911aa97700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9114105b90 0x7f911419d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:35:01.670 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.657+0000 7f911aa97700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911419d640 con 0x7f9114105b90 2022-09-07T21:35:01.670 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.657+0000 7f911aa97700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911419d7b0 con 0x7f9114071740 2022-09-07T21:35:01.670 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.657+0000 7f911aa97700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911419d970 con 0x7f9114072c90 2022-09-07T21:35:01.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.657+0000 7f9119294700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9114072c90 0x7f9114196000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:35:01.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.657+0000 7f9119294700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9114072c90 0x7f9114196000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60732/0 (socket says 172.21.15.47:60732) 2022-09-07T21:35:01.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.657+0000 7f9119294700 1 -- 172.21.15.47:0/250613548 learned_addr learned my addr 172.21.15.47:0/250613548 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:35:01.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.657+0000 7f911a296700 1 --2- 172.21.15.47:0/250613548 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9114105b90 0x7f911419d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:35:01.673 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.657+0000 7f9119a95700 1 --2- 172.21.15.47:0/250613548 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9114071740 0x7f9114195ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:35:01.673 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.658+0000 7f9119294700 1 -- 172.21.15.47:0/250613548 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9114071740 msgr2=0x7f9114195ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:35:01.673 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.658+0000 7f9119294700 1 --2- 172.21.15.47:0/250613548 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9114071740 0x7f9114195ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:01.674 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.658+0000 7f9119294700 1 -- 172.21.15.47:0/250613548 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9114105b90 msgr2=0x7f911419d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:35:01.674 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.658+0000 7f9119294700 1 --2- 172.21.15.47:0/250613548 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9114105b90 0x7f911419d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:01.675 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.658+0000 7f9119294700 1 -- 172.21.15.47:0/250613548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91040093b0 con 0x7f9114072c90 2022-09-07T21:35:01.675 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.658+0000 7f9119294700 1 --2- 172.21.15.47:0/250613548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9114072c90 0x7f9114196000 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9104004230 tx=0x7f9104004260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:35:01.676 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.659+0000 7f910affd700 1 -- 172.21.15.47:0/250613548 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910401d070 con 0x7f9114072c90 2022-09-07T21:35:01.676 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.659+0000 7f911aa97700 1 -- 172.21.15.47:0/250613548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911419dbf0 con 0x7f9114072c90 2022-09-07T21:35:01.677 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.659+0000 7f911aa97700 1 -- 172.21.15.47:0/250613548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f911419e090 con 0x7f9114072c90 2022-09-07T21:35:01.677 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.659+0000 7f910affd700 1 -- 172.21.15.47:0/250613548 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f9104003bb0 con 0x7f9114072c90 2022-09-07T21:35:01.677 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.659+0000 7f910affd700 1 -- 172.21.15.47:0/250613548 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9104017b10 con 0x7f9114072c90 2022-09-07T21:35:01.678 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.660+0000 7f911aa97700 1 -- 172.21.15.47:0/250613548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f80051a0 con 0x7f9114072c90 2022-09-07T21:35:01.678 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.662+0000 7f910affd700 1 -- 172.21.15.47:0/250613548 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f910400c040 con 0x7f9114072c90 2022-09-07T21:35:01.679 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.663+0000 7f910affd700 1 --2- 172.21.15.47:0/250613548 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9100099010 0x7f910009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:35:01.679 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.663+0000 7f910affd700 1 -- 172.21.15.47:0/250613548 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4529+0+0 (secure 0 0 0) 0x7f91040c26b0 con 0x7f9114072c90 2022-09-07T21:35:01.680 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.663+0000 7f9119a95700 1 --2- 172.21.15.47:0/250613548 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9100099010 0x7f910009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:35:01.680 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.664+0000 7f9119a95700 1 --2- 172.21.15.47:0/250613548 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9100099010 0x7f910009b4c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9110005fd0 tx=0x7f9110005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:35:01.681 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.666+0000 7f910affd700 1 -- 172.21.15.47:0/250613548 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f910408fbc0 con 0x7f9114072c90 2022-09-07T21:35:01.910 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:01.908+0000 7f911aa97700 1 -- 172.21.15.47:0/250613548 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f90f8000bf0 con 0x7f9100099010 2022-09-07T21:35:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:01 smithi055 ceph-mon[97033]: osdmap e42: 6 total, 6 up, 6 in 2022-09-07T21:35:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:01 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:35:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:01 smithi059 ceph-mon[97450]: osdmap e42: 6 total, 6 up, 6 in 2022-09-07T21:35:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:01 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:35:02.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:01 smithi047 ceph-mon[90087]: osdmap e42: 6 total, 6 up, 6 in 2022-09-07T21:35:02.470 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:01 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:35:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:02 smithi055 ceph-mon[97033]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:35:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:35:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:03.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:02 smithi055 ceph-mon[97033]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:03.278 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:02 smithi047 ceph-mon[90087]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:35:03.279 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:35:03.279 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:03.279 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:02 smithi047 ceph-mon[90087]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:02 smithi059 ceph-mon[97450]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:35:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:35:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:02 smithi059 ceph-mon[97450]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:05.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:05 smithi047 ceph-mon[90087]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-07T21:35:05.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:05 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/4006766987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbe32a2e-570c-4f3b-a860-1c0e603ae96b"}]: dispatch 2022-09-07T21:35:05.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:05 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/4006766987' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbe32a2e-570c-4f3b-a860-1c0e603ae96b"}]': finished 2022-09-07T21:35:05.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:05 smithi047 ceph-mon[90087]: osdmap e43: 7 total, 6 up, 7 in 2022-09-07T21:35:05.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:05 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:05 smithi055 ceph-mon[97033]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-07T21:35:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:05 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/4006766987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbe32a2e-570c-4f3b-a860-1c0e603ae96b"}]: dispatch 2022-09-07T21:35:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:05 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/4006766987' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbe32a2e-570c-4f3b-a860-1c0e603ae96b"}]': finished 2022-09-07T21:35:05.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:05 smithi055 ceph-mon[97033]: osdmap e43: 7 total, 6 up, 7 in 2022-09-07T21:35:05.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:05 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:05 smithi059 ceph-mon[97450]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-07T21:35:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:05 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/4006766987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbe32a2e-570c-4f3b-a860-1c0e603ae96b"}]: dispatch 2022-09-07T21:35:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:05 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/4006766987' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbe32a2e-570c-4f3b-a860-1c0e603ae96b"}]': finished 2022-09-07T21:35:06.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:05 smithi059 ceph-mon[97450]: osdmap e43: 7 total, 6 up, 7 in 2022-09-07T21:35:06.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:05 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:06.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:06 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3731234582' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:35:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:06 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3731234582' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:35:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:06 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3731234582' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:35:07.909 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:07 smithi047 ceph-mon[90087]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2022-09-07T21:35:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:07 smithi055 ceph-mon[97033]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2022-09-07T21:35:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:07 smithi059 ceph-mon[97450]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2022-09-07T21:35:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:08 smithi055 ceph-mon[97033]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-07T21:35:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:08 smithi059 ceph-mon[97450]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-07T21:35:09.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:08 smithi047 ceph-mon[90087]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-07T21:35:11.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:11 smithi047 ceph-mon[90087]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2022-09-07T21:35:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:11 smithi055 ceph-mon[97033]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2022-09-07T21:35:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:11 smithi059 ceph-mon[97450]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2022-09-07T21:35:13.881 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:13 smithi047 ceph-mon[90087]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-07T21:35:13.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:13 smithi055 ceph-mon[97033]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-07T21:35:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:13 smithi059 ceph-mon[97450]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-07T21:35:14.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:14 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-09-07T21:35:14.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:14 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:14 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-09-07T21:35:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:14 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:14 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-09-07T21:35:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:14 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:15.887 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:15 smithi047 ceph-mon[90087]: Deploying daemon osd.6 on smithi047 2022-09-07T21:35:15.887 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:15 smithi047 ceph-mon[90087]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:15.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:15 smithi055 ceph-mon[97033]: Deploying daemon osd.6 on smithi047 2022-09-07T21:35:15.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:15 smithi055 ceph-mon[97033]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:15 smithi059 ceph-mon[97450]: Deploying daemon osd.6 on smithi047 2022-09-07T21:35:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:15 smithi059 ceph-mon[97450]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:17.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:17 smithi047 ceph-mon[90087]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:17 smithi055 ceph-mon[97033]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:17 smithi059 ceph-mon[97450]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:19.192 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:18 smithi047 ceph-mon[90087]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:18 smithi055 ceph-mon[97033]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:18 smithi059 ceph-mon[97450]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:21.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:21 smithi047 ceph-mon[90087]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:21.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:21 smithi055 ceph-mon[97033]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:21 smithi059 ceph-mon[97450]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:23.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:23 smithi047 ceph-mon[90087]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:23 smithi055 ceph-mon[97033]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:23 smithi059 ceph-mon[97450]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:35:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:35:25.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:25 smithi055 ceph-mon[97033]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:35:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:35:25.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:25 smithi059 ceph-mon[97450]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:25.896 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:25.896 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:25.896 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:35:25.896 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:25.897 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:35:25.898 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:25 smithi047 ceph-mon[90087]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:27.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:27 smithi047 ceph-mon[90087]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:27 smithi055 ceph-mon[97033]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:27 smithi059 ceph-mon[97450]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:29.428 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:29.429 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:29.429 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:29 smithi059 ceph-mon[97450]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:29.430 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:29 smithi059 ceph-mon[97450]: from='osd.6 [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-09-07T21:35:29.430 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:29 smithi059 ceph-mon[97450]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-09-07T21:35:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:29.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:29 smithi055 ceph-mon[97033]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:29.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:29 smithi055 ceph-mon[97033]: from='osd.6 [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-09-07T21:35:29.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:29 smithi055 ceph-mon[97033]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-09-07T21:35:29.659 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:29.659 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:29.660 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:29 smithi047 ceph-mon[90087]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:29.660 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:29 smithi047 ceph-mon[90087]: from='osd.6 [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-09-07T21:35:29.660 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:29 smithi047 ceph-mon[90087]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-09-07T21:35:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:30 smithi055 ceph-mon[97033]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-09-07T21:35:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:30 smithi055 ceph-mon[97033]: osdmap e44: 7 total, 6 up, 7 in 2022-09-07T21:35:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:30 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:30 smithi055 ceph-mon[97033]: from='osd.6 [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:35:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:30 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:30 smithi055 ceph-mon[97033]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:35:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:30 smithi059 ceph-mon[97450]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-09-07T21:35:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:30 smithi059 ceph-mon[97450]: osdmap e44: 7 total, 6 up, 7 in 2022-09-07T21:35:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:30 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:30.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:30 smithi059 ceph-mon[97450]: from='osd.6 [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:35:30.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:30 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:30.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:30 smithi059 ceph-mon[97450]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:35:30.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:30 smithi047 ceph-mon[90087]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-09-07T21:35:30.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:30 smithi047 ceph-mon[90087]: osdmap e44: 7 total, 6 up, 7 in 2022-09-07T21:35:30.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:30 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:30.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:30 smithi047 ceph-mon[90087]: from='osd.6 [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:35:30.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:30 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:30.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:30 smithi047 ceph-mon[90087]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:35:31.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:31 smithi047 ceph-mon[90087]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-09-07T21:35:31.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:31 smithi047 ceph-mon[90087]: osdmap e45: 7 total, 6 up, 7 in 2022-09-07T21:35:31.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:31 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:31.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:31 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:31.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:31 smithi047 ceph-mon[90087]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:31 smithi055 ceph-mon[97033]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-09-07T21:35:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:31 smithi055 ceph-mon[97033]: osdmap e45: 7 total, 6 up, 7 in 2022-09-07T21:35:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:31 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:31.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:31 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:31.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:31 smithi055 ceph-mon[97033]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:31 smithi059 ceph-mon[97450]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-09-07T21:35:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:31 smithi059 ceph-mon[97450]: osdmap e45: 7 total, 6 up, 7 in 2022-09-07T21:35:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:31 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:31 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:31.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:31 smithi059 ceph-mon[97450]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-09-07T21:35:31.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:31.561+0000 7f910affd700 1 -- 172.21.15.47:0/250613548 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f90f8000bf0 con 0x7f9100099010 2022-09-07T21:35:31.564 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 6 on host 'smithi047' 2022-09-07T21:35:31.567 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:31.563+0000 7f911aa97700 1 -- 172.21.15.47:0/250613548 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9100099010 msgr2=0x7f910009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:35:31.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:31.563+0000 7f911aa97700 1 --2- 172.21.15.47:0/250613548 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9100099010 0x7f910009b4c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9110005fd0 tx=0x7f9110005ea0 comp rx=0 tx=0).stop 2022-09-07T21:35:31.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:31.563+0000 7f911aa97700 1 -- 172.21.15.47:0/250613548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9114072c90 msgr2=0x7f9114196000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:35:31.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:31.563+0000 7f911aa97700 1 --2- 172.21.15.47:0/250613548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9114072c90 0x7f9114196000 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9104004230 tx=0x7f9104004260 comp rx=0 tx=0).stop 2022-09-07T21:35:31.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:31.564+0000 7f911aa97700 1 -- 172.21.15.47:0/250613548 shutdown_connections 2022-09-07T21:35:31.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:31.564+0000 7f911aa97700 1 --2- 172.21.15.47:0/250613548 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9100099010 0x7f910009b4c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:31.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:31.564+0000 7f911aa97700 1 --2- 172.21.15.47:0/250613548 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9114105b90 0x7f911419d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:31.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:31.564+0000 7f911aa97700 1 --2- 172.21.15.47:0/250613548 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9114071740 0x7f9114195ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:31.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:31.564+0000 7f911aa97700 1 --2- 172.21.15.47:0/250613548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9114072c90 0x7f9114196000 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:31.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:31.564+0000 7f911aa97700 1 -- 172.21.15.47:0/250613548 >> 172.21.15.47:0/250613548 conn(0x7f91140f93a0 msgr2=0x7f91140fb780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:35:31.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:31.564+0000 7f911aa97700 1 -- 172.21.15.47:0/250613548 shutdown_connections 2022-09-07T21:35:31.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:35:31.564+0000 7f911aa97700 1 -- 172.21.15.47:0/250613548 wait complete. 2022-09-07T21:35:32.056 DEBUG:teuthology.orchestra.run.smithi047:osd.6> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@osd.6.service 2022-09-07T21:35:32.063 INFO:tasks.cephadm:Deploying osd.7 on smithi055 with /dev/vg_nvme/lv_2... 2022-09-07T21:35:32.064 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-09-07T21:35:32.154 INFO:journalctl@ceph.osd.6.smithi047.stdout:-- Logs begin at Wed 2022-09-07 21:11:20 UTC. -- 2022-09-07T21:35:32.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:32 smithi047 ceph-mon[90087]: purged_snaps scrub starts 2022-09-07T21:35:32.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:32 smithi047 ceph-mon[90087]: purged_snaps scrub ok 2022-09-07T21:35:32.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:32 smithi047 ceph-mon[90087]: osd.6 [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165] boot 2022-09-07T21:35:32.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:32 smithi047 ceph-mon[90087]: osdmap e46: 7 total, 7 up, 7 in 2022-09-07T21:35:32.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:32.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:32.407 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:32.407 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:35:32.407 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:32.408 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:32 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:35:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:32 smithi055 ceph-mon[97033]: purged_snaps scrub starts 2022-09-07T21:35:32.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:32 smithi055 ceph-mon[97033]: purged_snaps scrub ok 2022-09-07T21:35:32.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:32 smithi055 ceph-mon[97033]: osd.6 [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165] boot 2022-09-07T21:35:32.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:32 smithi055 ceph-mon[97033]: osdmap e46: 7 total, 7 up, 7 in 2022-09-07T21:35:32.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:32.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:32.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:32.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:35:32.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:32.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:32 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:35:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:32 smithi059 ceph-mon[97450]: purged_snaps scrub starts 2022-09-07T21:35:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:32 smithi059 ceph-mon[97450]: purged_snaps scrub ok 2022-09-07T21:35:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:32 smithi059 ceph-mon[97450]: osd.6 [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165] boot 2022-09-07T21:35:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:32 smithi059 ceph-mon[97450]: osdmap e46: 7 total, 7 up, 7 in 2022-09-07T21:35:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:35:32.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:32.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:32.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:35:32.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:32.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:32 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:35:32.677 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.b/config 2022-09-07T21:35:33.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:33 smithi047 ceph-mon[90087]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:33.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:33 smithi047 ceph-mon[90087]: osdmap e47: 7 total, 7 up, 7 in 2022-09-07T21:35:33.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:33 smithi055 ceph-mon[97033]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:33.771 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:33 smithi055 ceph-mon[97033]: osdmap e47: 7 total, 7 up, 7 in 2022-09-07T21:35:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:33 smithi059 ceph-mon[97450]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:33 smithi059 ceph-mon[97450]: osdmap e47: 7 total, 7 up, 7 in 2022-09-07T21:35:34.358 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-09-07T21:35:34.358 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-09-07T21:35:34.358 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2022-09-07T21:35:34.359 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2022-09-07T21:35:34.359 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131518 s, 797 MB/s 2022-09-07T21:35:34.359 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2022-09-07T21:35:34.806 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi055:vg_nvme/lv_2 2022-09-07T21:35:35.405 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.b/config 2022-09-07T21:35:35.837 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:35 smithi055 ceph-mon[97033]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:35.838 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:35 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:35.838 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:35 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:35.838 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:35 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:35:35.839 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:35 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:35:35.839 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:35 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:35:35.839 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:35 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:35.839 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:35 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:35.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:35 smithi047 ceph-mon[90087]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:35.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:35 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:35.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:35 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:35.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:35 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:35:35.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:35 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:35:35.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:35 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:35:35.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:35 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:35.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:35 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:35 smithi059 ceph-mon[97450]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:36.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:35 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:36.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:35 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:36.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:35 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:35:36.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:35 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:35:36.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:35 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:35:36.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:35 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:36.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:35 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:36.739 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.729+0000 7f091c8d6700 1 -- 172.21.15.55:0/349646738 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09180fe300 msgr2=0x7f09180fe710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:35:36.740 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.729+0000 7f091c8d6700 1 --2- 172.21.15.55:0/349646738 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09180fe300 0x7f09180fe710 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0908009810 tx=0x7f0908009b20 comp rx=0 tx=0).stop 2022-09-07T21:35:36.740 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.729+0000 7f091c8d6700 1 -- 172.21.15.55:0/349646738 shutdown_connections 2022-09-07T21:35:36.741 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.729+0000 7f091c8d6700 1 --2- 172.21.15.55:0/349646738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09180fd990 0x7f0918105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:36.741 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.729+0000 7f091c8d6700 1 --2- 172.21.15.55:0/349646738 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09180fe300 0x7f09180fe710 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:36.741 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.729+0000 7f091c8d6700 1 --2- 172.21.15.55:0/349646738 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09180fcfe0 0x7f09180fd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:36.742 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.729+0000 7f091c8d6700 1 -- 172.21.15.55:0/349646738 >> 172.21.15.55:0/349646738 conn(0x7f09180f7c00 msgr2=0x7f09180fa010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:35:36.742 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.729+0000 7f091c8d6700 1 -- 172.21.15.55:0/349646738 shutdown_connections 2022-09-07T21:35:36.743 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.729+0000 7f091c8d6700 1 -- 172.21.15.55:0/349646738 wait complete. 2022-09-07T21:35:36.743 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f091c8d6700 1 Processor -- start 2022-09-07T21:35:36.743 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f091c8d6700 1 -- start start 2022-09-07T21:35:36.744 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f091c8d6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09180fcfe0 0x7f09180fca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:35:36.744 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f091c8d6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09180fd990 0x7f091819abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:35:36.745 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f091c8d6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09180fe300 0x7f091819d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:35:36.745 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f091c8d6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091819d820 con 0x7f09180fe300 2022-09-07T21:35:36.745 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f091c8d6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091819d990 con 0x7f09180fd990 2022-09-07T21:35:36.746 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f091c8d6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091819db50 con 0x7f09180fcfe0 2022-09-07T21:35:36.746 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f0915d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09180fd990 0x7f091819abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:35:36.746 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f0915d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09180fd990 0x7f091819abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53514/0 (socket says 172.21.15.55:53514) 2022-09-07T21:35:36.747 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f0915d9b700 1 -- 172.21.15.55:0/1607310868 learned_addr learned my addr 172.21.15.55:0/1607310868 (peer_addr_for_me v2:172.21.15.55:0/0) 2022-09-07T21:35:36.747 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f091659c700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09180fcfe0 0x7f09180fca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:35:36.748 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f0916d9d700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09180fe300 0x7f091819d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:35:36.748 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f0915d9b700 1 -- 172.21.15.55:0/1607310868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09180fcfe0 msgr2=0x7f09180fca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:35:36.749 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f0915d9b700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09180fcfe0 0x7f09180fca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:36.749 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f0915d9b700 1 -- 172.21.15.55:0/1607310868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09180fe300 msgr2=0x7f091819d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:35:36.749 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f0915d9b700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09180fe300 0x7f091819d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:36.750 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f0915d9b700 1 -- 172.21.15.55:0/1607310868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0908009470 con 0x7f09180fd990 2022-09-07T21:35:36.750 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f0916d9d700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09180fe300 0x7f091819d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-07T21:35:36.750 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.730+0000 7f0915d9b700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09180fd990 0x7f091819abd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f090c00b3c0 tx=0x7f090c00b780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:35:36.751 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.731+0000 7f09077fe700 1 -- 172.21.15.55:0/1607310868 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090c011830 con 0x7f09180fd990 2022-09-07T21:35:36.751 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.731+0000 7f09077fe700 1 -- 172.21.15.55:0/1607310868 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f090c011e30 con 0x7f09180fd990 2022-09-07T21:35:36.751 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.731+0000 7f09077fe700 1 -- 172.21.15.55:0/1607310868 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090c00f2e0 con 0x7f09180fd990 2022-09-07T21:35:36.751 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.731+0000 7f091c8d6700 1 -- 172.21.15.55:0/1607310868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091819de30 con 0x7f09180fd990 2022-09-07T21:35:36.751 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.731+0000 7f091c8d6700 1 -- 172.21.15.55:0/1607310868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091819e2e0 con 0x7f09180fd990 2022-09-07T21:35:36.752 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.732+0000 7f091c8d6700 1 -- 172.21.15.55:0/1607310868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0918108540 con 0x7f09180fd990 2022-09-07T21:35:36.752 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.733+0000 7f09077fe700 1 -- 172.21.15.55:0/1607310868 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f090c021070 con 0x7f09180fd990 2022-09-07T21:35:36.753 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.733+0000 7f09077fe700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0900099010 0x7f090009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:35:36.753 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.733+0000 7f09077fe700 1 -- 172.21.15.55:0/1607310868 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4961+0+0 (secure 0 0 0) 0x7f090c0c1450 con 0x7f09180fd990 2022-09-07T21:35:36.754 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.734+0000 7f091659c700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0900099010 0x7f090009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:35:36.754 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.734+0000 7f091659c700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0900099010 0x7f090009b4c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0908009440 tx=0x7f0908009370 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:35:36.755 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.736+0000 7f09077fe700 1 -- 172.21.15.55:0/1607310868 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f090c08e8a0 con 0x7f09180fd990 2022-09-07T21:35:36.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:36 smithi047 ceph-mon[90087]: Detected new or changed devices on smithi047 2022-09-07T21:35:36.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:36 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi047 to 5758M 2022-09-07T21:35:36.941 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:36.939+0000 7f091c8d6700 1 -- 172.21.15.55:0/1607310868 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f091805f120 con 0x7f0900099010 2022-09-07T21:35:36.974 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:36 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi047 2022-09-07T21:35:36.975 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:36 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi047 to 5758M 2022-09-07T21:35:37.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:36 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi047 2022-09-07T21:35:37.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:36 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi047 to 5758M 2022-09-07T21:35:37.843 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:37 smithi055 ceph-mon[97033]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:37.844 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:35:37.844 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:35:37.844 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:37 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:37.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:37 smithi047 ceph-mon[90087]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:37.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:35:37.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:35:37.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:37 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:37 smithi059 ceph-mon[97450]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:35:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:35:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:37 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:38.641 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:38 smithi055 ceph-mon[97033]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:35:38.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:38 smithi047 ceph-mon[90087]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:35:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:38 smithi059 ceph-mon[97450]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:35:39.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:39 smithi047 ceph-mon[90087]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:39.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:39 smithi047 ceph-mon[90087]: from='client.? 172.21.15.55:0/1978664842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85e70ce9-e721-4719-b309-e5011ddbbab3"}]: dispatch 2022-09-07T21:35:39.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:39 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85e70ce9-e721-4719-b309-e5011ddbbab3"}]: dispatch 2022-09-07T21:35:39.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:39 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85e70ce9-e721-4719-b309-e5011ddbbab3"}]': finished 2022-09-07T21:35:39.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:39 smithi047 ceph-mon[90087]: osdmap e48: 8 total, 7 up, 8 in 2022-09-07T21:35:39.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:39 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:39 smithi055 ceph-mon[97033]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:39.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:39 smithi055 ceph-mon[97033]: from='client.? 172.21.15.55:0/1978664842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85e70ce9-e721-4719-b309-e5011ddbbab3"}]: dispatch 2022-09-07T21:35:39.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:39 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85e70ce9-e721-4719-b309-e5011ddbbab3"}]: dispatch 2022-09-07T21:35:39.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:39 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85e70ce9-e721-4719-b309-e5011ddbbab3"}]': finished 2022-09-07T21:35:39.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:39 smithi055 ceph-mon[97033]: osdmap e48: 8 total, 7 up, 8 in 2022-09-07T21:35:39.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:39 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:39 smithi059 ceph-mon[97450]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:39 smithi059 ceph-mon[97450]: from='client.? 172.21.15.55:0/1978664842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85e70ce9-e721-4719-b309-e5011ddbbab3"}]: dispatch 2022-09-07T21:35:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:39 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85e70ce9-e721-4719-b309-e5011ddbbab3"}]: dispatch 2022-09-07T21:35:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:39 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85e70ce9-e721-4719-b309-e5011ddbbab3"}]': finished 2022-09-07T21:35:40.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:39 smithi059 ceph-mon[97450]: osdmap e48: 8 total, 7 up, 8 in 2022-09-07T21:35:40.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:39 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:40.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:40 smithi047 ceph-mon[90087]: from='client.? 172.21.15.55:0/1237041866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:35:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:40 smithi055 ceph-mon[97033]: from='client.? 172.21.15.55:0/1237041866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:35:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:40 smithi059 ceph-mon[97450]: from='client.? 172.21.15.55:0/1237041866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:35:41.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:41 smithi047 ceph-mon[90087]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:41 smithi055 ceph-mon[97033]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:41 smithi059 ceph-mon[97450]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:43.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:43 smithi047 ceph-mon[90087]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:43 smithi055 ceph-mon[97033]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:43 smithi059 ceph-mon[97450]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:45 smithi055 ceph-mon[97033]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:45.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:45 smithi047 ceph-mon[90087]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:45 smithi059 ceph-mon[97450]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:46.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-09-07T21:35:46.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:46.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:35:46.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:35:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-09-07T21:35:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:46.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:35:46.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:35:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-09-07T21:35:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:35:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:35:47.722 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:47 smithi055 ceph-mon[97033]: Deploying daemon osd.7 on smithi055 2022-09-07T21:35:47.723 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:47 smithi055 ceph-mon[97033]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:47.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:47 smithi047 ceph-mon[90087]: Deploying daemon osd.7 on smithi055 2022-09-07T21:35:47.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:47 smithi047 ceph-mon[90087]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:47 smithi059 ceph-mon[97450]: Deploying daemon osd.7 on smithi055 2022-09-07T21:35:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:47 smithi059 ceph-mon[97450]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:49.061 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:48 smithi055 ceph-mon[97033]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:48 smithi059 ceph-mon[97450]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:49.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:48 smithi047 ceph-mon[90087]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:51.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:51 smithi047 ceph-mon[90087]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:51.935 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:51 smithi055 ceph-mon[97033]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:51 smithi059 ceph-mon[97450]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:35:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:35:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:53 smithi055 ceph-mon[97033]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:35:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:53.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:35:53.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:53 smithi059 ceph-mon[97450]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:53.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:53.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:53.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:35:53.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:53.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:35:53.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:53 smithi047 ceph-mon[90087]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:55 smithi055 ceph-mon[97033]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:55.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:55 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:55.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:55 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:55.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:55 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:55 smithi059 ceph-mon[97450]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:55 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:55 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:56.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:55 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:56.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:55 smithi047 ceph-mon[90087]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:56.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:55 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:56.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:55 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:56.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:55 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:56.716 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:56.712+0000 7f09077fe700 1 -- 172.21.15.55:0/1607310868 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f091805f120 con 0x7f0900099010 2022-09-07T21:35:56.716 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 7 on host 'smithi055' 2022-09-07T21:35:56.718 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:56.714+0000 7f091c8d6700 1 -- 172.21.15.55:0/1607310868 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0900099010 msgr2=0x7f090009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:35:56.718 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:56.714+0000 7f091c8d6700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0900099010 0x7f090009b4c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0908009440 tx=0x7f0908009370 comp rx=0 tx=0).stop 2022-09-07T21:35:56.718 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:56.714+0000 7f091c8d6700 1 -- 172.21.15.55:0/1607310868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09180fd990 msgr2=0x7f091819abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:35:56.719 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:56.714+0000 7f091c8d6700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09180fd990 0x7f091819abd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f090c00b3c0 tx=0x7f090c00b780 comp rx=0 tx=0).stop 2022-09-07T21:35:56.719 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:56.714+0000 7f091c8d6700 1 -- 172.21.15.55:0/1607310868 shutdown_connections 2022-09-07T21:35:56.720 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:56.714+0000 7f091c8d6700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0900099010 0x7f090009b4c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:56.720 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:56.714+0000 7f091c8d6700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09180fe300 0x7f091819d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:56.720 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:56.714+0000 7f091c8d6700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09180fcfe0 0x7f09180fca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:56.720 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:56.714+0000 7f091c8d6700 1 --2- 172.21.15.55:0/1607310868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09180fd990 0x7f091819abd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:35:56.720 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:56.714+0000 7f091c8d6700 1 -- 172.21.15.55:0/1607310868 >> 172.21.15.55:0/1607310868 conn(0x7f09180f7c00 msgr2=0x7f09180faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:35:56.721 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:56.714+0000 7f091c8d6700 1 -- 172.21.15.55:0/1607310868 shutdown_connections 2022-09-07T21:35:56.721 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:35:56.714+0000 7f091c8d6700 1 -- 172.21.15.55:0/1607310868 wait complete. 2022-09-07T21:35:56.950 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:56 smithi055 ceph-mon[97033]: from='osd.7 [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-09-07T21:35:56.951 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:56 smithi055 ceph-mon[97033]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-09-07T21:35:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:56 smithi059 ceph-mon[97450]: from='osd.7 [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-09-07T21:35:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:56 smithi059 ceph-mon[97450]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-09-07T21:35:57.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:56 smithi047 ceph-mon[90087]: from='osd.7 [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-09-07T21:35:57.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:56 smithi047 ceph-mon[90087]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-09-07T21:35:57.278 DEBUG:teuthology.orchestra.run.smithi055:osd.7> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@osd.7.service 2022-09-07T21:35:57.282 INFO:tasks.cephadm:Deploying osd.8 on smithi059 with /dev/vg_nvme/lv_2... 2022-09-07T21:35:57.283 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-09-07T21:35:57.312 INFO:journalctl@ceph.osd.7.smithi055.stdout:-- Logs begin at Wed 2022-09-07 21:07:35 UTC. -- 2022-09-07T21:35:57.892 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.c/config 2022-09-07T21:35:57.980 INFO:journalctl@ceph.osd.7.smithi055.stdout:Sep 07 21:35:57 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-7[113964]: 2022-09-07T21:35:57.678+0000 7f17e852f700 -1 osd.7 0 waiting for initial osdmap 2022-09-07T21:35:57.981 INFO:journalctl@ceph.osd.7.smithi055.stdout:Sep 07 21:35:57 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-7[113964]: 2022-09-07T21:35:57.688+0000 7f17e131e700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-07T21:35:57.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:57 smithi055 ceph-mon[97033]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:57.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:57 smithi055 ceph-mon[97033]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-09-07T21:35:57.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:57 smithi055 ceph-mon[97033]: osdmap e49: 8 total, 7 up, 8 in 2022-09-07T21:35:57.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:57.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:57 smithi055 ceph-mon[97033]: from='osd.7 [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:35:57.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:57 smithi055 ceph-mon[97033]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:35:57.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:57.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:57.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:35:57.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:57.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:35:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:57 smithi059 ceph-mon[97450]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:57 smithi059 ceph-mon[97450]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-09-07T21:35:58.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:57 smithi059 ceph-mon[97450]: osdmap e49: 8 total, 7 up, 8 in 2022-09-07T21:35:58.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:58.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:57 smithi059 ceph-mon[97450]: from='osd.7 [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:35:58.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:57 smithi059 ceph-mon[97450]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:35:58.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:58.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:58.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:35:58.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:58.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:57 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:35:58.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:57 smithi047 ceph-mon[90087]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:58.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:57 smithi047 ceph-mon[90087]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-09-07T21:35:58.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:57 smithi047 ceph-mon[90087]: osdmap e49: 8 total, 7 up, 8 in 2022-09-07T21:35:58.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:58.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:57 smithi047 ceph-mon[90087]: from='osd.7 [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:35:58.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:57 smithi047 ceph-mon[90087]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:35:58.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:58.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:35:58.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:35:58.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:35:58.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:57 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:35:58.974 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:58 smithi059 ceph-mon[97450]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2022-09-07T21:35:58.975 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:58 smithi059 ceph-mon[97450]: osdmap e50: 8 total, 7 up, 8 in 2022-09-07T21:35:58.975 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:58 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:58.975 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:58 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:58.975 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:58 smithi059 ceph-mon[97450]: osd.7 [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712] boot 2022-09-07T21:35:58.976 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:58 smithi059 ceph-mon[97450]: osdmap e51: 8 total, 8 up, 8 in 2022-09-07T21:35:58.976 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:58 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:58 smithi055 ceph-mon[97033]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2022-09-07T21:35:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:58 smithi055 ceph-mon[97033]: osdmap e50: 8 total, 7 up, 8 in 2022-09-07T21:35:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:58 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:58.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:58 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:58.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:58 smithi055 ceph-mon[97033]: osd.7 [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712] boot 2022-09-07T21:35:58.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:58 smithi055 ceph-mon[97033]: osdmap e51: 8 total, 8 up, 8 in 2022-09-07T21:35:58.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:58 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:59.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:58 smithi047 ceph-mon[90087]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2022-09-07T21:35:59.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:58 smithi047 ceph-mon[90087]: osdmap e50: 8 total, 7 up, 8 in 2022-09-07T21:35:59.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:58 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:59.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:58 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:59.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:58 smithi047 ceph-mon[90087]: osd.7 [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712] boot 2022-09-07T21:35:59.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:58 smithi047 ceph-mon[90087]: osdmap e51: 8 total, 8 up, 8 in 2022-09-07T21:35:59.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:58 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:35:59.528 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-09-07T21:35:59.528 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-09-07T21:35:59.529 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2022-09-07T21:35:59.529 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2022-09-07T21:35:59.529 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113185 s, 926 MB/s 2022-09-07T21:35:59.529 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2022-09-07T21:35:59.910 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:59 smithi059 ceph-mon[97450]: purged_snaps scrub starts 2022-09-07T21:35:59.911 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:59 smithi059 ceph-mon[97450]: purged_snaps scrub ok 2022-09-07T21:35:59.911 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:59 smithi059 ceph-mon[97450]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:59.911 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:59 smithi059 ceph-mon[97450]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2022-09-07T21:35:59.911 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:35:59 smithi059 ceph-mon[97450]: osdmap e52: 8 total, 8 up, 8 in 2022-09-07T21:35:59.913 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi059:vg_nvme/lv_2 2022-09-07T21:35:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:59 smithi055 ceph-mon[97033]: purged_snaps scrub starts 2022-09-07T21:35:59.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:59 smithi055 ceph-mon[97033]: purged_snaps scrub ok 2022-09-07T21:35:59.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:59 smithi055 ceph-mon[97033]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:35:59.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:59 smithi055 ceph-mon[97033]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2022-09-07T21:35:59.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:35:59 smithi055 ceph-mon[97033]: osdmap e52: 8 total, 8 up, 8 in 2022-09-07T21:36:00.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:59 smithi047 ceph-mon[90087]: purged_snaps scrub starts 2022-09-07T21:36:00.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:59 smithi047 ceph-mon[90087]: purged_snaps scrub ok 2022-09-07T21:36:00.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:59 smithi047 ceph-mon[90087]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-09-07T21:36:00.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:59 smithi047 ceph-mon[90087]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2022-09-07T21:36:00.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:35:59 smithi047 ceph-mon[90087]: osdmap e52: 8 total, 8 up, 8 in 2022-09-07T21:36:00.516 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.c/config 2022-09-07T21:36:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:00 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-07T21:36:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:00 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 5724M 2022-09-07T21:36:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:00 smithi055 ceph-mon[97033]: osdmap e53: 8 total, 8 up, 8 in 2022-09-07T21:36:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:00 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-07T21:36:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:00 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 5724M 2022-09-07T21:36:01.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:01.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:01.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:00 smithi059 ceph-mon[97450]: osdmap e53: 8 total, 8 up, 8 in 2022-09-07T21:36:01.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:00 smithi047 ceph-mon[90087]: Detected new or changed devices on smithi055 2022-09-07T21:36:01.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:01.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:01.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:01.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:01.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:01.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:00 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi055 to 5724M 2022-09-07T21:36:01.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:01.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:01.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:00 smithi047 ceph-mon[90087]: osdmap e53: 8 total, 8 up, 8 in 2022-09-07T21:36:01.811 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.803+0000 7f95b7f06700 1 -- 172.21.15.59:0/1657836441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95b00ffa10 msgr2=0x7f95b0106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:36:01.811 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.803+0000 7f95b7f06700 1 --2- 172.21.15.59:0/1657836441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95b00ffa10 0x7f95b0106340 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009bb0 tx=0x7f95a8009ec0 comp rx=0 tx=0).stop 2022-09-07T21:36:01.812 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.804+0000 7f95b7f06700 1 -- 172.21.15.59:0/1657836441 shutdown_connections 2022-09-07T21:36:01.812 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.804+0000 7f95b7f06700 1 --2- 172.21.15.59:0/1657836441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95b00ffa10 0x7f95b0106340 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:01.812 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.804+0000 7f95b7f06700 1 --2- 172.21.15.59:0/1657836441 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95b00fde80 0x7f95b00fe290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:01.813 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.804+0000 7f95b7f06700 1 --2- 172.21.15.59:0/1657836441 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95b00ff060 0x7f95b00ff4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:01.813 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.804+0000 7f95b7f06700 1 -- 172.21.15.59:0/1657836441 >> 172.21.15.59:0/1657836441 conn(0x7f95b00f9570 msgr2=0x7f95b00fb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:36:01.813 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.804+0000 7f95b7f06700 1 -- 172.21.15.59:0/1657836441 shutdown_connections 2022-09-07T21:36:01.813 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.804+0000 7f95b7f06700 1 -- 172.21.15.59:0/1657836441 wait complete. 2022-09-07T21:36:01.814 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.804+0000 7f95b7f06700 1 Processor -- start 2022-09-07T21:36:01.814 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.804+0000 7f95b7f06700 1 -- start start 2022-09-07T21:36:01.814 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b7f06700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95b00fde80 0x7f95b0193a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:36:01.815 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b7f06700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95b00ff060 0x7f95b0193f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:36:01.815 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b7f06700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95b00ffa10 0x7f95b019afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:36:01.816 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b7f06700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b019b590 con 0x7f95b00ff060 2022-09-07T21:36:01.816 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b7f06700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b019b700 con 0x7f95b00ffa10 2022-09-07T21:36:01.816 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b7f06700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b019b8c0 con 0x7f95b00fde80 2022-09-07T21:36:01.816 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b5ca2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95b00fde80 0x7f95b0193a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:36:01.816 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b5ca2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95b00fde80 0x7f95b0193a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40358/0 (socket says 172.21.15.59:40358) 2022-09-07T21:36:01.817 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b5ca2700 1 -- 172.21.15.59:0/3515992405 learned_addr learned my addr 172.21.15.59:0/3515992405 (peer_addr_for_me v2:172.21.15.59:0/0) 2022-09-07T21:36:01.817 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b54a1700 1 --2- 172.21.15.59:0/3515992405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95b00ff060 0x7f95b0193f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:36:01.817 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b5ca2700 1 -- 172.21.15.59:0/3515992405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95b00ffa10 msgr2=0x7f95b019afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:36:01.818 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b5ca2700 1 --2- 172.21.15.59:0/3515992405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95b00ffa10 0x7f95b019afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:01.818 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b5ca2700 1 -- 172.21.15.59:0/3515992405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95b00ff060 msgr2=0x7f95b0193f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:36:01.818 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b5ca2700 1 --2- 172.21.15.59:0/3515992405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95b00ff060 0x7f95b0193f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:01.818 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b5ca2700 1 -- 172.21.15.59:0/3515992405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a80097e0 con 0x7f95b00fde80 2022-09-07T21:36:01.819 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b5ca2700 1 --2- 172.21.15.59:0/3515992405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95b00fde80 0x7f95b0193a10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f95ac00c7a0 tx=0x7f95ac00cb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:36:01.819 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95a6ffd700 1 -- 172.21.15.59:0/3515992405 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac005220 con 0x7f95b00fde80 2022-09-07T21:36:01.819 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b7f06700 1 -- 172.21.15.59:0/3515992405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b019bba0 con 0x7f95b00fde80 2022-09-07T21:36:01.819 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95a6ffd700 1 -- 172.21.15.59:0/3515992405 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f95ac005380 con 0x7f95b00fde80 2022-09-07T21:36:01.820 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95a6ffd700 1 -- 172.21.15.59:0/3515992405 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac004bb0 con 0x7f95b00fde80 2022-09-07T21:36:01.820 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.805+0000 7f95b7f06700 1 -- 172.21.15.59:0/3515992405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b019c0b0 con 0x7f95b00fde80 2022-09-07T21:36:01.820 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.806+0000 7f95b7f06700 1 -- 172.21.15.59:0/3515992405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b0109660 con 0x7f95b00fde80 2022-09-07T21:36:01.821 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.807+0000 7f95a6ffd700 1 -- 172.21.15.59:0/3515992405 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f95ac01f070 con 0x7f95b00fde80 2022-09-07T21:36:01.821 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.808+0000 7f95a6ffd700 1 --2- 172.21.15.59:0/3515992405 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f959c099010 0x7f959c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:36:01.822 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.808+0000 7f95a6ffd700 1 -- 172.21.15.59:0/3515992405 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5393+0+0 (secure 0 0 0) 0x7f95ac0c1810 con 0x7f95b00fde80 2022-09-07T21:36:01.822 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.808+0000 7f95b54a1700 1 --2- 172.21.15.59:0/3515992405 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f959c099010 0x7f959c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:36:01.824 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.809+0000 7f95b54a1700 1 --2- 172.21.15.59:0/3515992405 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f959c099010 0x7f959c09b4c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f95a000b7d0 tx=0x7f95a0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:36:01.824 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:01.811+0000 7f95a6ffd700 1 -- 172.21.15.59:0/3515992405 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f95ac08eab0 con 0x7f95b00fde80 2022-09-07T21:36:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:01 smithi055 ceph-mon[97033]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:01 smithi055 ceph-mon[97033]: osdmap e54: 8 total, 8 up, 8 in 2022-09-07T21:36:02.021 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:02.018+0000 7f95b7f06700 1 -- 172.21.15.59:0/3515992405 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f95b005f120 con 0x7f959c099010 2022-09-07T21:36:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:01 smithi059 ceph-mon[97450]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:01 smithi059 ceph-mon[97450]: osdmap e54: 8 total, 8 up, 8 in 2022-09-07T21:36:02.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:01 smithi047 ceph-mon[90087]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:02.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:01 smithi047 ceph-mon[90087]: osdmap e54: 8 total, 8 up, 8 in 2022-09-07T21:36:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:02 smithi055 ceph-mon[97033]: osdmap e55: 8 total, 8 up, 8 in 2022-09-07T21:36:03.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:02 smithi055 ceph-mon[97033]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:36:03.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:36:03.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:36:03.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:03.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:02 smithi055 ceph-mon[97033]: pgmap v203: 1 pgs: 1 unknown; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:03.278 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:02 smithi059 ceph-mon[97450]: osdmap e55: 8 total, 8 up, 8 in 2022-09-07T21:36:03.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:02 smithi059 ceph-mon[97450]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:36:03.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:36:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:36:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:02 smithi059 ceph-mon[97450]: pgmap v203: 1 pgs: 1 unknown; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:03.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:02 smithi047 ceph-mon[90087]: osdmap e55: 8 total, 8 up, 8 in 2022-09-07T21:36:03.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:02 smithi047 ceph-mon[90087]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:36:03.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:36:03.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:36:03.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:03.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:02 smithi047 ceph-mon[90087]: pgmap v203: 1 pgs: 1 unknown; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:04 smithi055 ceph-mon[97033]: from='client.? 172.21.15.59:0/1366159766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5b3728f-7ba6-4d25-a54c-d04bad536de1"}]: dispatch 2022-09-07T21:36:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:04 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5b3728f-7ba6-4d25-a54c-d04bad536de1"}]: dispatch 2022-09-07T21:36:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:04 smithi055 ceph-mon[97033]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2022-09-07T21:36:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:04 smithi055 ceph-mon[97033]: Cluster is now healthy 2022-09-07T21:36:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:04 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5b3728f-7ba6-4d25-a54c-d04bad536de1"}]': finished 2022-09-07T21:36:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:04 smithi055 ceph-mon[97033]: osdmap e56: 9 total, 8 up, 9 in 2022-09-07T21:36:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:04 smithi059 ceph-mon[97450]: from='client.? 172.21.15.59:0/1366159766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5b3728f-7ba6-4d25-a54c-d04bad536de1"}]: dispatch 2022-09-07T21:36:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:04 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5b3728f-7ba6-4d25-a54c-d04bad536de1"}]: dispatch 2022-09-07T21:36:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:04 smithi059 ceph-mon[97450]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2022-09-07T21:36:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:04 smithi059 ceph-mon[97450]: Cluster is now healthy 2022-09-07T21:36:05.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:04 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5b3728f-7ba6-4d25-a54c-d04bad536de1"}]': finished 2022-09-07T21:36:05.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:04 smithi059 ceph-mon[97450]: osdmap e56: 9 total, 8 up, 9 in 2022-09-07T21:36:05.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:05.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:04 smithi047 ceph-mon[90087]: from='client.? 172.21.15.59:0/1366159766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5b3728f-7ba6-4d25-a54c-d04bad536de1"}]: dispatch 2022-09-07T21:36:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:04 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5b3728f-7ba6-4d25-a54c-d04bad536de1"}]: dispatch 2022-09-07T21:36:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:04 smithi047 ceph-mon[90087]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2022-09-07T21:36:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:04 smithi047 ceph-mon[90087]: Cluster is now healthy 2022-09-07T21:36:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:04 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5b3728f-7ba6-4d25-a54c-d04bad536de1"}]': finished 2022-09-07T21:36:05.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:04 smithi047 ceph-mon[90087]: osdmap e56: 9 total, 8 up, 9 in 2022-09-07T21:36:05.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:05 smithi055 ceph-mon[97033]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:05 smithi055 ceph-mon[97033]: from='client.? 172.21.15.59:0/1290757758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:36:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:05 smithi059 ceph-mon[97450]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:05 smithi059 ceph-mon[97450]: from='client.? 172.21.15.59:0/1290757758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:36:06.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:05 smithi047 ceph-mon[90087]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:06.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:05 smithi047 ceph-mon[90087]: from='client.? 172.21.15.59:0/1290757758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:36:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:07 smithi055 ceph-mon[97033]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:07 smithi059 ceph-mon[97450]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:08.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:07 smithi047 ceph-mon[90087]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:08 smithi055 ceph-mon[97033]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:08 smithi059 ceph-mon[97450]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:09.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:08 smithi047 ceph-mon[90087]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:11.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:11 smithi047 ceph-mon[90087]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:11 smithi055 ceph-mon[97033]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:11 smithi059 ceph-mon[97450]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:12.863 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:12 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-09-07T21:36:12.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:12 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:12.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:12 smithi059 ceph-mon[97450]: Deploying daemon osd.8 on smithi059 2022-09-07T21:36:12.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:12 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-09-07T21:36:12.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:12 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:12.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:12 smithi047 ceph-mon[90087]: Deploying daemon osd.8 on smithi059 2022-09-07T21:36:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:12 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-09-07T21:36:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:12 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:12 smithi055 ceph-mon[97033]: Deploying daemon osd.8 on smithi059 2022-09-07T21:36:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:13 smithi059 ceph-mon[97450]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:13.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:13 smithi047 ceph-mon[90087]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:13 smithi055 ceph-mon[97033]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:15 smithi047 ceph-mon[90087]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:15 smithi055 ceph-mon[97033]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:15 smithi059 ceph-mon[97450]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:17.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:17 smithi047 ceph-mon[90087]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:17 smithi055 ceph-mon[97033]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:17 smithi059 ceph-mon[97450]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:18.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:18.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:18.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:36:18.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:18.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:36:18.922 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:18 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:18.923 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:18 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:18.923 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:18 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:36:18.923 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:18 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:18.923 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:18 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:36:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:18 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:18 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:18 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:36:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:18 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:18.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:18 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:36:19.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:19 smithi047 ceph-mon[90087]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:19 smithi055 ceph-mon[97033]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:19 smithi059 ceph-mon[97450]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:21.629 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:21 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:21.630 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:21 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:21.630 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:21 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:21.630 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:21 smithi059 ceph-mon[97450]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:21.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:21.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:21.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:21.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:21 smithi047 ceph-mon[90087]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:21 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:21.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:21 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:21.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:21 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:21.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:21 smithi055 ceph-mon[97033]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:21.960 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 8 on host 'smithi059' 2022-09-07T21:36:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:21.956+0000 7f95a6ffd700 1 -- 172.21.15.59:0/3515992405 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f95b005f120 con 0x7f959c099010 2022-09-07T21:36:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:21.959+0000 7f95b7f06700 1 -- 172.21.15.59:0/3515992405 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f959c099010 msgr2=0x7f959c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:36:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:21.959+0000 7f95b7f06700 1 --2- 172.21.15.59:0/3515992405 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f959c099010 0x7f959c09b4c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f95a000b7d0 tx=0x7f95a0008040 comp rx=0 tx=0).stop 2022-09-07T21:36:21.964 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:21.959+0000 7f95b7f06700 1 -- 172.21.15.59:0/3515992405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95b00fde80 msgr2=0x7f95b0193a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:36:21.964 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:21.959+0000 7f95b7f06700 1 --2- 172.21.15.59:0/3515992405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95b00fde80 0x7f95b0193a10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f95ac00c7a0 tx=0x7f95ac00cb60 comp rx=0 tx=0).stop 2022-09-07T21:36:21.964 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:21.959+0000 7f95b7f06700 1 -- 172.21.15.59:0/3515992405 shutdown_connections 2022-09-07T21:36:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:21.959+0000 7f95b7f06700 1 --2- 172.21.15.59:0/3515992405 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f959c099010 0x7f959c09b4c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:21.959+0000 7f95b7f06700 1 --2- 172.21.15.59:0/3515992405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95b00ffa10 0x7f95b019afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:21.959+0000 7f95b7f06700 1 --2- 172.21.15.59:0/3515992405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95b00fde80 0x7f95b0193a10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:21.959+0000 7f95b7f06700 1 --2- 172.21.15.59:0/3515992405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95b00ff060 0x7f95b0193f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:21.959+0000 7f95b7f06700 1 -- 172.21.15.59:0/3515992405 >> 172.21.15.59:0/3515992405 conn(0x7f95b00f9570 msgr2=0x7f95b00fb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:36:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:21.959+0000 7f95b7f06700 1 -- 172.21.15.59:0/3515992405 shutdown_connections 2022-09-07T21:36:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:36:21.959+0000 7f95b7f06700 1 -- 172.21.15.59:0/3515992405 wait complete. 2022-09-07T21:36:22.387 DEBUG:teuthology.orchestra.run.smithi059:osd.8> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@osd.8.service 2022-09-07T21:36:22.390 INFO:tasks.cephadm:Deploying osd.9 on smithi047 with /dev/vg_nvme/lv_1... 2022-09-07T21:36:22.390 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-09-07T21:36:22.419 INFO:journalctl@ceph.osd.8.smithi059.stdout:-- Logs begin at Wed 2022-09-07 21:07:37 UTC. -- 2022-09-07T21:36:22.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:22 smithi047 ceph-mon[90087]: from='osd.8 [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-09-07T21:36:22.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:22 smithi047 ceph-mon[90087]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-09-07T21:36:22.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:22 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:22.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:22 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:22.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:22 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:36:22.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:22 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:22.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:22 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:36:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:22 smithi055 ceph-mon[97033]: from='osd.8 [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-09-07T21:36:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:22 smithi055 ceph-mon[97033]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-09-07T21:36:22.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:22 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:22.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:22 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:22.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:22 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:36:22.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:22 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:22.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:22 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:36:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:22 smithi059 ceph-mon[97450]: from='osd.8 [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-09-07T21:36:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:22 smithi059 ceph-mon[97450]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-09-07T21:36:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:22 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:22 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:22.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:22 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:36:22.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:22 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:22.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:22 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:36:23.282 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:36:23.627 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:23 smithi047 ceph-mon[90087]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-09-07T21:36:23.628 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:23 smithi047 ceph-mon[90087]: osdmap e57: 9 total, 8 up, 9 in 2022-09-07T21:36:23.628 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:23.628 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:23 smithi047 ceph-mon[90087]: from='osd.8 [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:36:23.628 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:23 smithi047 ceph-mon[90087]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:36:23.629 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:23 smithi047 ceph-mon[90087]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:23.675 INFO:journalctl@ceph.osd.8.smithi059.stdout:Sep 07 21:36:23 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-8[115065]: 2022-09-07T21:36:23.379+0000 7feb040d4700 -1 osd.8 0 waiting for initial osdmap 2022-09-07T21:36:23.675 INFO:journalctl@ceph.osd.8.smithi059.stdout:Sep 07 21:36:23 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-8[115065]: 2022-09-07T21:36:23.391+0000 7feafc6c2700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-07T21:36:23.676 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:23 smithi059 ceph-mon[97450]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-09-07T21:36:23.676 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:23 smithi059 ceph-mon[97450]: osdmap e57: 9 total, 8 up, 9 in 2022-09-07T21:36:23.676 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:23 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:23.676 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:23 smithi059 ceph-mon[97450]: from='osd.8 [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:36:23.677 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:23 smithi059 ceph-mon[97450]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:36:23.677 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:23 smithi059 ceph-mon[97450]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:23 smithi055 ceph-mon[97033]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-09-07T21:36:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:23 smithi055 ceph-mon[97033]: osdmap e57: 9 total, 8 up, 9 in 2022-09-07T21:36:23.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:23 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:23.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:23 smithi055 ceph-mon[97033]: from='osd.8 [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:36:23.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:23 smithi055 ceph-mon[97033]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:36:23.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:23 smithi055 ceph-mon[97033]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:24.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:24 smithi047 ceph-mon[90087]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2022-09-07T21:36:24.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:24 smithi047 ceph-mon[90087]: osdmap e58: 9 total, 8 up, 9 in 2022-09-07T21:36:24.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:24 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:24.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:24 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:24.729 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:24 smithi059 ceph-mon[97450]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2022-09-07T21:36:24.729 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:24 smithi059 ceph-mon[97450]: osdmap e58: 9 total, 8 up, 9 in 2022-09-07T21:36:24.729 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:24 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:24.730 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:24 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:24 smithi055 ceph-mon[97033]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2022-09-07T21:36:24.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:24 smithi055 ceph-mon[97033]: osdmap e58: 9 total, 8 up, 9 in 2022-09-07T21:36:24.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:24 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:24.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:24 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:25 smithi055 ceph-mon[97033]: purged_snaps scrub starts 2022-09-07T21:36:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:25 smithi055 ceph-mon[97033]: purged_snaps scrub ok 2022-09-07T21:36:25.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:25 smithi055 ceph-mon[97033]: osd.8 [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220] boot 2022-09-07T21:36:25.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:25 smithi055 ceph-mon[97033]: osdmap e59: 9 total, 9 up, 9 in 2022-09-07T21:36:25.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:25.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:25 smithi055 ceph-mon[97033]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:25.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:25.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:25.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:25.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:25.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:25.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:25.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:25 smithi059 ceph-mon[97450]: purged_snaps scrub starts 2022-09-07T21:36:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:25 smithi059 ceph-mon[97450]: purged_snaps scrub ok 2022-09-07T21:36:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:25 smithi059 ceph-mon[97450]: osd.8 [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220] boot 2022-09-07T21:36:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:25 smithi059 ceph-mon[97450]: osdmap e59: 9 total, 9 up, 9 in 2022-09-07T21:36:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:25.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:25 smithi059 ceph-mon[97450]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:25.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:25.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:25.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:25.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:25.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:25.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:25.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:25.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:25 smithi047 ceph-mon[90087]: purged_snaps scrub starts 2022-09-07T21:36:25.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:25 smithi047 ceph-mon[90087]: purged_snaps scrub ok 2022-09-07T21:36:25.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:25 smithi047 ceph-mon[90087]: osd.8 [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220] boot 2022-09-07T21:36:25.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:25 smithi047 ceph-mon[90087]: osdmap e59: 9 total, 9 up, 9 in 2022-09-07T21:36:25.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:36:25.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:25 smithi047 ceph-mon[90087]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-09-07T21:36:25.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:25.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:25.658 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:25.658 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:25.659 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:36:25.659 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:25.659 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:25.767 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-09-07T21:36:25.768 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-09-07T21:36:25.768 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2022-09-07T21:36:25.769 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2022-09-07T21:36:25.769 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.071059 s, 148 MB/s 2022-09-07T21:36:25.770 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2022-09-07T21:36:26.257 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi047:vg_nvme/lv_1 2022-09-07T21:36:26.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:26 smithi047 ceph-mon[90087]: Detected new or changed devices on smithi059 2022-09-07T21:36:26.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:26 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi059 to 5724M 2022-09-07T21:36:26.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:26 smithi047 ceph-mon[90087]: osdmap e60: 9 total, 9 up, 9 in 2022-09-07T21:36:26.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:26 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-07T21:36:26.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:26 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 5724M 2022-09-07T21:36:26.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:26 smithi055 ceph-mon[97033]: osdmap e60: 9 total, 9 up, 9 in 2022-09-07T21:36:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:26 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-07T21:36:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:26 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 5724M 2022-09-07T21:36:26.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:26 smithi059 ceph-mon[97450]: osdmap e60: 9 total, 9 up, 9 in 2022-09-07T21:36:27.143 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:36:27.672 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:27 smithi047 ceph-mon[90087]: osdmap e61: 9 total, 9 up, 9 in 2022-09-07T21:36:27.673 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:27 smithi047 ceph-mon[90087]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:27 smithi055 ceph-mon[97033]: osdmap e61: 9 total, 9 up, 9 in 2022-09-07T21:36:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:27 smithi055 ceph-mon[97033]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:27 smithi059 ceph-mon[97450]: osdmap e61: 9 total, 9 up, 9 in 2022-09-07T21:36:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:27 smithi059 ceph-mon[97450]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:28.518 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:28 smithi047 ceph-mon[90087]: osdmap e62: 9 total, 9 up, 9 in 2022-09-07T21:36:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:28 smithi055 ceph-mon[97033]: osdmap e62: 9 total, 9 up, 9 in 2022-09-07T21:36:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:28 smithi059 ceph-mon[97450]: osdmap e62: 9 total, 9 up, 9 in 2022-09-07T21:36:28.838 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.835+0000 7fd393269700 1 -- 172.21.15.47:0/2528523114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd38c100fe0 msgr2=0x7fd38c1033c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:36:28.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.835+0000 7fd393269700 1 --2- 172.21.15.47:0/2528523114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd38c100fe0 0x7fd38c1033c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd37c009810 tx=0x7fd37c009b20 comp rx=0 tx=0).stop 2022-09-07T21:36:28.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.835+0000 7fd393269700 1 -- 172.21.15.47:0/2528523114 shutdown_connections 2022-09-07T21:36:28.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.835+0000 7fd393269700 1 --2- 172.21.15.47:0/2528523114 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd38c103a30 0x7fd38c105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:28.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.835+0000 7fd393269700 1 --2- 172.21.15.47:0/2528523114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd38c067330 0x7fd38c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:28.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.835+0000 7fd393269700 1 --2- 172.21.15.47:0/2528523114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd38c100fe0 0x7fd38c1033c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:28.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.835+0000 7fd393269700 1 -- 172.21.15.47:0/2528523114 >> 172.21.15.47:0/2528523114 conn(0x7fd38c0f8430 msgr2=0x7fd38c0fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:36:28.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.835+0000 7fd393269700 1 -- 172.21.15.47:0/2528523114 shutdown_connections 2022-09-07T21:36:28.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.835+0000 7fd393269700 1 -- 172.21.15.47:0/2528523114 wait complete. 2022-09-07T21:36:28.846 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.837+0000 7fd393269700 1 Processor -- start 2022-09-07T21:36:28.846 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.837+0000 7fd393269700 1 -- start start 2022-09-07T21:36:28.846 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.837+0000 7fd393269700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd38c067330 0x7fd38c195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:36:28.847 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.837+0000 7fd393269700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd38c100fe0 0x7fd38c1961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:36:28.847 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.837+0000 7fd393269700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd38c103a30 0x7fd38c19d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:36:28.847 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.837+0000 7fd393269700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c19d820 con 0x7fd38c103a30 2022-09-07T21:36:28.847 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.837+0000 7fd393269700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c19d990 con 0x7fd38c100fe0 2022-09-07T21:36:28.848 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.837+0000 7fd393269700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c19db50 con 0x7fd38c067330 2022-09-07T21:36:28.848 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.837+0000 7fd391005700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd38c067330 0x7fd38c195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:36:28.848 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.838+0000 7fd390804700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd38c100fe0 0x7fd38c1961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:36:28.848 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.838+0000 7fd391005700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd38c067330 0x7fd38c195c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60804/0 (socket says 172.21.15.47:60804) 2022-09-07T21:36:28.849 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.838+0000 7fd391005700 1 -- 172.21.15.47:0/4219252965 learned_addr learned my addr 172.21.15.47:0/4219252965 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:36:28.849 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.838+0000 7fd391806700 1 --2- 172.21.15.47:0/4219252965 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd38c103a30 0x7fd38c19d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:36:28.849 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.838+0000 7fd390804700 1 -- 172.21.15.47:0/4219252965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd38c067330 msgr2=0x7fd38c195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:36:28.850 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.838+0000 7fd390804700 1 --2- 172.21.15.47:0/4219252965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd38c067330 0x7fd38c195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:28.850 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.838+0000 7fd390804700 1 -- 172.21.15.47:0/4219252965 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd38c103a30 msgr2=0x7fd38c19d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:36:28.850 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.838+0000 7fd390804700 1 --2- 172.21.15.47:0/4219252965 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd38c103a30 0x7fd38c19d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:28.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.838+0000 7fd390804700 1 -- 172.21.15.47:0/4219252965 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37c009470 con 0x7fd38c100fe0 2022-09-07T21:36:28.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.838+0000 7fd391806700 1 --2- 172.21.15.47:0/4219252965 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd38c103a30 0x7fd38c19d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:36:28.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.838+0000 7fd390804700 1 --2- 172.21.15.47:0/4219252965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd38c100fe0 0x7fd38c1961b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd37c006010 tx=0x7fd37c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:36:28.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.839+0000 7fd3827fc700 1 -- 172.21.15.47:0/4219252965 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37c01d070 con 0x7fd38c100fe0 2022-09-07T21:36:28.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.839+0000 7fd3827fc700 1 -- 172.21.15.47:0/4219252965 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd37c00f930 con 0x7fd38c100fe0 2022-09-07T21:36:28.852 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.839+0000 7fd393269700 1 -- 172.21.15.47:0/4219252965 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd38c19ddd0 con 0x7fd38c100fe0 2022-09-07T21:36:28.852 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.839+0000 7fd3827fc700 1 -- 172.21.15.47:0/4219252965 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37c0047d0 con 0x7fd38c100fe0 2022-09-07T21:36:28.852 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.839+0000 7fd393269700 1 -- 172.21.15.47:0/4219252965 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd38c19e1d0 con 0x7fd38c100fe0 2022-09-07T21:36:28.852 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.840+0000 7fd393269700 1 -- 172.21.15.47:0/4219252965 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd38c109600 con 0x7fd38c100fe0 2022-09-07T21:36:28.853 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.842+0000 7fd3827fc700 1 -- 172.21.15.47:0/4219252965 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fd37c028030 con 0x7fd38c100fe0 2022-09-07T21:36:28.853 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.843+0000 7fd3827fc700 1 --2- 172.21.15.47:0/4219252965 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd378099010 0x7fd37809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:36:28.853 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.843+0000 7fd3827fc700 1 -- 172.21.15.47:0/4219252965 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd37c0c27e0 con 0x7fd38c100fe0 2022-09-07T21:36:28.853 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.843+0000 7fd391005700 1 --2- 172.21.15.47:0/4219252965 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd378099010 0x7fd37809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:36:28.854 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.844+0000 7fd391005700 1 --2- 172.21.15.47:0/4219252965 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd378099010 0x7fd37809b4c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd38c196b40 tx=0x7fd388005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:36:28.854 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:28.845+0000 7fd3827fc700 1 -- 172.21.15.47:0/4219252965 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fd37c08f7e0 con 0x7fd38c100fe0 2022-09-07T21:36:29.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:29.084+0000 7fd393269700 1 -- 172.21.15.47:0/4219252965 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd38c05f120 con 0x7fd378099010 2022-09-07T21:36:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:29 smithi055 ceph-mon[97033]: osdmap e63: 9 total, 9 up, 9 in 2022-09-07T21:36:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:29 smithi055 ceph-mon[97033]: pgmap v224: 1 pgs: 1 peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:36:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:36:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:29.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:29 smithi059 ceph-mon[97450]: osdmap e63: 9 total, 9 up, 9 in 2022-09-07T21:36:29.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:29 smithi059 ceph-mon[97450]: pgmap v224: 1 pgs: 1 peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:29.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:36:29.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:36:29.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:29.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:29 smithi047 ceph-mon[90087]: osdmap e63: 9 total, 9 up, 9 in 2022-09-07T21:36:29.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:29 smithi047 ceph-mon[90087]: pgmap v224: 1 pgs: 1 peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:29.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:36:29.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:36:29.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:30 smithi055 ceph-mon[97033]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:36:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:30 smithi059 ceph-mon[97450]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:36:30.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:30 smithi047 ceph-mon[90087]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:36:31.634 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:31 smithi047 ceph-mon[90087]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2022-09-07T21:36:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:31 smithi055 ceph-mon[97033]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2022-09-07T21:36:31.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:31 smithi059 ceph-mon[97450]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2022-09-07T21:36:33.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:33 smithi047 ceph-mon[90087]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2022-09-07T21:36:33.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:33 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/4257509074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f08f42a-4784-4e10-b862-16806fa5ce27"}]: dispatch 2022-09-07T21:36:33.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:33 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/4257509074' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f08f42a-4784-4e10-b862-16806fa5ce27"}]': finished 2022-09-07T21:36:33.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:33 smithi047 ceph-mon[90087]: osdmap e64: 10 total, 9 up, 10 in 2022-09-07T21:36:33.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:36:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:33 smithi055 ceph-mon[97033]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2022-09-07T21:36:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:33 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/4257509074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f08f42a-4784-4e10-b862-16806fa5ce27"}]: dispatch 2022-09-07T21:36:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:33 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/4257509074' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f08f42a-4784-4e10-b862-16806fa5ce27"}]': finished 2022-09-07T21:36:33.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:33 smithi055 ceph-mon[97033]: osdmap e64: 10 total, 9 up, 10 in 2022-09-07T21:36:33.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:36:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:33 smithi059 ceph-mon[97450]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2022-09-07T21:36:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:33 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/4257509074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f08f42a-4784-4e10-b862-16806fa5ce27"}]: dispatch 2022-09-07T21:36:34.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:33 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/4257509074' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f08f42a-4784-4e10-b862-16806fa5ce27"}]': finished 2022-09-07T21:36:34.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:33 smithi059 ceph-mon[97450]: osdmap e64: 10 total, 9 up, 10 in 2022-09-07T21:36:34.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:36:34.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:34 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1868309802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:36:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:34 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1868309802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:36:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:34 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1868309802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:36:35.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:35 smithi047 ceph-mon[90087]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2022-09-07T21:36:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:35 smithi055 ceph-mon[97033]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2022-09-07T21:36:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:35 smithi059 ceph-mon[97450]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2022-09-07T21:36:37.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:37 smithi047 ceph-mon[90087]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2022-09-07T21:36:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:37 smithi055 ceph-mon[97033]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2022-09-07T21:36:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:37 smithi059 ceph-mon[97450]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2022-09-07T21:36:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:38 smithi055 ceph-mon[97033]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-07T21:36:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:38 smithi059 ceph-mon[97450]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-07T21:36:39.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:38 smithi047 ceph-mon[90087]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-07T21:36:41.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:41 smithi047 ceph-mon[90087]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:41 smithi055 ceph-mon[97033]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:41 smithi059 ceph-mon[97450]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:42.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:42 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-09-07T21:36:42.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:42 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:42.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:42 smithi047 ceph-mon[90087]: Deploying daemon osd.9 on smithi047 2022-09-07T21:36:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:42 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-09-07T21:36:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:42 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:42 smithi055 ceph-mon[97033]: Deploying daemon osd.9 on smithi047 2022-09-07T21:36:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:42 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-09-07T21:36:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:42 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:42 smithi059 ceph-mon[97450]: Deploying daemon osd.9 on smithi047 2022-09-07T21:36:43.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:43 smithi047 ceph-mon[90087]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:43 smithi055 ceph-mon[97033]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:43 smithi059 ceph-mon[97450]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:45.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:45 smithi047 ceph-mon[90087]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:45 smithi055 ceph-mon[97033]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:45 smithi059 ceph-mon[97450]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:46.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:36:46.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:36:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:36:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:36:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:36:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:36:47.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:47 smithi047 ceph-mon[90087]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:47 smithi055 ceph-mon[97033]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:47 smithi059 ceph-mon[97450]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:48 smithi055 ceph-mon[97033]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:48 smithi059 ceph-mon[97450]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:49.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:48 smithi047 ceph-mon[90087]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:51.885 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:51 smithi047 ceph-mon[90087]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:51 smithi055 ceph-mon[97033]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:51 smithi059 ceph-mon[97450]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:53.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:53.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:53.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:36:53.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:53.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:53 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:36:53.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:53 smithi047 ceph-mon[90087]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:36:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:53 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:36:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:53 smithi055 ceph-mon[97033]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:36:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:36:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:53 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:36:53.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:53 smithi059 ceph-mon[97450]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:55.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:55 smithi047 ceph-mon[90087]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:55 smithi055 ceph-mon[97033]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:55 smithi059 ceph-mon[97450]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:56 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:56 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:56 smithi059 ceph-mon[97450]: from='osd.9 [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-09-07T21:36:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:56 smithi059 ceph-mon[97450]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-09-07T21:36:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:56 smithi059 ceph-mon[97450]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:57.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:56 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:57.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:56 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:57.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:56 smithi047 ceph-mon[90087]: from='osd.9 [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-09-07T21:36:57.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:56 smithi047 ceph-mon[90087]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-09-07T21:36:57.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:56 smithi047 ceph-mon[90087]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:57 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:57 smithi055 ceph-mon[97033]: from='osd.9 [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-09-07T21:36:57.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:57 smithi055 ceph-mon[97033]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-09-07T21:36:57.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:57 smithi055 ceph-mon[97033]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:58 smithi059 ceph-mon[97450]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-09-07T21:36:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:58 smithi059 ceph-mon[97450]: osdmap e65: 10 total, 9 up, 10 in 2022-09-07T21:36:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:58 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:36:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:58 smithi059 ceph-mon[97450]: from='osd.9 [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:36:58.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:58 smithi059 ceph-mon[97450]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:36:58.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:58 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:58.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:58 smithi047 ceph-mon[90087]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-09-07T21:36:58.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:58 smithi047 ceph-mon[90087]: osdmap e65: 10 total, 9 up, 10 in 2022-09-07T21:36:58.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:58 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:36:58.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:58 smithi047 ceph-mon[90087]: from='osd.9 [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:36:58.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:58 smithi047 ceph-mon[90087]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:36:58.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:58 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:58 smithi055 ceph-mon[97033]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-09-07T21:36:58.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:58 smithi055 ceph-mon[97033]: osdmap e65: 10 total, 9 up, 10 in 2022-09-07T21:36:58.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:58 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:36:58.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:58 smithi055 ceph-mon[97033]: from='osd.9 [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:36:58.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:58 smithi055 ceph-mon[97033]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-09-07T21:36:58.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:58 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:36:59.165 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 9 on host 'smithi047' 2022-09-07T21:36:59.167 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:59.161+0000 7fd3827fc700 1 -- 172.21.15.47:0/4219252965 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd38c05f120 con 0x7fd378099010 2022-09-07T21:36:59.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:59.164+0000 7fd393269700 1 -- 172.21.15.47:0/4219252965 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd378099010 msgr2=0x7fd37809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:36:59.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:59.164+0000 7fd393269700 1 --2- 172.21.15.47:0/4219252965 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd378099010 0x7fd37809b4c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd38c196b40 tx=0x7fd388005ea0 comp rx=0 tx=0).stop 2022-09-07T21:36:59.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:59.164+0000 7fd393269700 1 -- 172.21.15.47:0/4219252965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd38c100fe0 msgr2=0x7fd38c1961b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:36:59.169 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:59.164+0000 7fd393269700 1 --2- 172.21.15.47:0/4219252965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd38c100fe0 0x7fd38c1961b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd37c006010 tx=0x7fd37c00f690 comp rx=0 tx=0).stop 2022-09-07T21:36:59.169 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:59.164+0000 7fd393269700 1 -- 172.21.15.47:0/4219252965 shutdown_connections 2022-09-07T21:36:59.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:59.164+0000 7fd393269700 1 --2- 172.21.15.47:0/4219252965 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd378099010 0x7fd37809b4c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:59.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:59.164+0000 7fd393269700 1 --2- 172.21.15.47:0/4219252965 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd38c103a30 0x7fd38c19d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:59.171 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:59.164+0000 7fd393269700 1 --2- 172.21.15.47:0/4219252965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd38c067330 0x7fd38c195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:59.171 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:59.164+0000 7fd393269700 1 --2- 172.21.15.47:0/4219252965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd38c100fe0 0x7fd38c1961b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:36:59.172 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:59.164+0000 7fd393269700 1 -- 172.21.15.47:0/4219252965 >> 172.21.15.47:0/4219252965 conn(0x7fd38c0f8430 msgr2=0x7fd38c0ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:36:59.172 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:59.164+0000 7fd393269700 1 -- 172.21.15.47:0/4219252965 shutdown_connections 2022-09-07T21:36:59.173 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:36:59.165+0000 7fd393269700 1 -- 172.21.15.47:0/4219252965 wait complete. 2022-09-07T21:36:59.278 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:59 smithi047 ceph-mon[90087]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-09-07T21:36:59.278 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:59 smithi047 ceph-mon[90087]: osdmap e66: 10 total, 9 up, 10 in 2022-09-07T21:36:59.279 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:59 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:36:59.279 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:59 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:36:59.279 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:36:59 smithi047 ceph-mon[90087]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:59 smithi059 ceph-mon[97450]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-09-07T21:36:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:59 smithi059 ceph-mon[97450]: osdmap e66: 10 total, 9 up, 10 in 2022-09-07T21:36:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:59 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:36:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:59 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:36:59.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:36:59 smithi059 ceph-mon[97450]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:59 smithi055 ceph-mon[97033]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-09-07T21:36:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:59 smithi055 ceph-mon[97033]: osdmap e66: 10 total, 9 up, 10 in 2022-09-07T21:36:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:59 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:36:59.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:59 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:36:59.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:36:59 smithi055 ceph-mon[97033]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-07T21:36:59.877 DEBUG:teuthology.orchestra.run.smithi047:osd.9> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@osd.9.service 2022-09-07T21:36:59.883 INFO:tasks.cephadm:Deploying osd.10 on smithi055 with /dev/vg_nvme/lv_1... 2022-09-07T21:36:59.884 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-09-07T21:36:59.973 INFO:journalctl@ceph.osd.9.smithi047.stdout:-- Logs begin at Wed 2022-09-07 21:11:20 UTC. -- 2022-09-07T21:37:00.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:00 smithi047 ceph-mon[90087]: purged_snaps scrub starts 2022-09-07T21:37:00.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:00 smithi047 ceph-mon[90087]: purged_snaps scrub ok 2022-09-07T21:37:00.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:37:00.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:00 smithi047 ceph-mon[90087]: osd.9 [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795] boot 2022-09-07T21:37:00.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:00 smithi047 ceph-mon[90087]: osdmap e67: 10 total, 10 up, 10 in 2022-09-07T21:37:00.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:37:00.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:00.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:00.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:00.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:00.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:00 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:00 smithi055 ceph-mon[97033]: purged_snaps scrub starts 2022-09-07T21:37:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:00 smithi055 ceph-mon[97033]: purged_snaps scrub ok 2022-09-07T21:37:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:37:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:00 smithi055 ceph-mon[97033]: osd.9 [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795] boot 2022-09-07T21:37:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:00 smithi055 ceph-mon[97033]: osdmap e67: 10 total, 10 up, 10 in 2022-09-07T21:37:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:37:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:00.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:00.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:00.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:00.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:00 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:00 smithi059 ceph-mon[97450]: purged_snaps scrub starts 2022-09-07T21:37:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:00 smithi059 ceph-mon[97450]: purged_snaps scrub ok 2022-09-07T21:37:00.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:37:00.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:00 smithi059 ceph-mon[97450]: osd.9 [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795] boot 2022-09-07T21:37:00.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:00 smithi059 ceph-mon[97450]: osdmap e67: 10 total, 10 up, 10 in 2022-09-07T21:37:00.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:37:00.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:00.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:00.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:00.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:00.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:00 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:00.495 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.b/config 2022-09-07T21:37:01.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:01 smithi047 ceph-mon[90087]: osdmap e68: 10 total, 10 up, 10 in 2022-09-07T21:37:01.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:01 smithi047 ceph-mon[90087]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:01 smithi055 ceph-mon[97033]: osdmap e68: 10 total, 10 up, 10 in 2022-09-07T21:37:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:01 smithi055 ceph-mon[97033]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:01 smithi059 ceph-mon[97450]: osdmap e68: 10 total, 10 up, 10 in 2022-09-07T21:37:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:01 smithi059 ceph-mon[97450]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:02.131 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-09-07T21:37:02.131 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-09-07T21:37:02.131 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2022-09-07T21:37:02.131 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2022-09-07T21:37:02.132 INFO:teuthology.orchestra.run.smithi055.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0149227 s, 703 MB/s 2022-09-07T21:37:02.132 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 2022-09-07T21:37:02.132 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2022-09-07T21:37:02.615 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi055:vg_nvme/lv_1 2022-09-07T21:37:03.219 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.b/config 2022-09-07T21:37:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:03 smithi055 ceph-mon[97033]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:03.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:03.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:03.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:03.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:03.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:03.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:03 smithi059 ceph-mon[97450]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:03.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:03.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:03.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:03.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:03.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:03.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:03 smithi047 ceph-mon[90087]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:03.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:03.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:03.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:03.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:03.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:03.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:03.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:03.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:04.543 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.539+0000 7fd4aae79700 1 -- 172.21.15.55:0/1540618748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a40ffe10 msgr2=0x7fd4a40fbf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:04.543 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.539+0000 7fd4aae79700 1 --2- 172.21.15.55:0/1540618748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a40ffe10 0x7fd4a40fbf30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd490009810 tx=0x7fd490009b20 comp rx=0 tx=0).stop 2022-09-07T21:37:04.544 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.539+0000 7fd4aae79700 1 -- 172.21.15.55:0/1540618748 shutdown_connections 2022-09-07T21:37:04.544 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.539+0000 7fd4aae79700 1 --2- 172.21.15.55:0/1540618748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4a40fc5a0 0x7fd4a41064a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:04.544 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.539+0000 7fd4aae79700 1 --2- 172.21.15.55:0/1540618748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4a40ff430 0x7fd4a40ff840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:04.544 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.539+0000 7fd4aae79700 1 --2- 172.21.15.55:0/1540618748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a40ffe10 0x7fd4a40fbf30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:04.545 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.539+0000 7fd4aae79700 1 -- 172.21.15.55:0/1540618748 >> 172.21.15.55:0/1540618748 conn(0x7fd4a40f7c00 msgr2=0x7fd4a40fa010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:37:04.545 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.539+0000 7fd4aae79700 1 -- 172.21.15.55:0/1540618748 shutdown_connections 2022-09-07T21:37:04.545 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.539+0000 7fd4aae79700 1 -- 172.21.15.55:0/1540618748 wait complete. 2022-09-07T21:37:04.545 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4aae79700 1 Processor -- start 2022-09-07T21:37:04.545 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4aae79700 1 -- start start 2022-09-07T21:37:04.546 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4aae79700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a40fc5a0 0x7fd4a41939f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:04.546 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4aae79700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4a40ff430 0x7fd4a4193f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:04.546 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4aae79700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4a40ffe10 0x7fd4a419afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:04.546 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4aae79700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a419b4e0 con 0x7fd4a40ffe10 2022-09-07T21:37:04.547 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4aae79700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a419b650 con 0x7fd4a40fc5a0 2022-09-07T21:37:04.547 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4aae79700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a419b810 con 0x7fd4a40ff430 2022-09-07T21:37:04.547 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4a8c15700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a40fc5a0 0x7fd4a41939f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:04.547 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4a8c15700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a40fc5a0 0x7fd4a41939f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53656/0 (socket says 172.21.15.55:53656) 2022-09-07T21:37:04.547 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4a8c15700 1 -- 172.21.15.55:0/4141672985 learned_addr learned my addr 172.21.15.55:0/4141672985 (peer_addr_for_me v2:172.21.15.55:0/0) 2022-09-07T21:37:04.548 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4a3fff700 1 --2- 172.21.15.55:0/4141672985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4a40ff430 0x7fd4a4193f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:04.548 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4a8c15700 1 -- 172.21.15.55:0/4141672985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4a40ff430 msgr2=0x7fd4a4193f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:04.548 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4a8c15700 1 --2- 172.21.15.55:0/4141672985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4a40ff430 0x7fd4a4193f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:04.548 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4a8c15700 1 -- 172.21.15.55:0/4141672985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4a40ffe10 msgr2=0x7fd4a419afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:04.548 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4a8c15700 1 --2- 172.21.15.55:0/4141672985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4a40ffe10 0x7fd4a419afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:04.549 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.540+0000 7fd4a8c15700 1 -- 172.21.15.55:0/4141672985 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd490009470 con 0x7fd4a40fc5a0 2022-09-07T21:37:04.549 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.541+0000 7fd4a8c15700 1 --2- 172.21.15.55:0/4141672985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a40fc5a0 0x7fd4a41939f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd49800e710 tx=0x7fd49800ea20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:37:04.549 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.541+0000 7fd4a1ffb700 1 -- 172.21.15.55:0/4141672985 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498004220 con 0x7fd4a40fc5a0 2022-09-07T21:37:04.549 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.541+0000 7fd4a1ffb700 1 -- 172.21.15.55:0/4141672985 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd498004380 con 0x7fd4a40fc5a0 2022-09-07T21:37:04.550 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.541+0000 7fd4aae79700 1 -- 172.21.15.55:0/4141672985 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a419baf0 con 0x7fd4a40fc5a0 2022-09-07T21:37:04.550 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.541+0000 7fd4a1ffb700 1 -- 172.21.15.55:0/4141672985 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498003be0 con 0x7fd4a40fc5a0 2022-09-07T21:37:04.550 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.541+0000 7fd4aae79700 1 -- 172.21.15.55:0/4141672985 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a419c000 con 0x7fd4a40fc5a0 2022-09-07T21:37:04.551 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.542+0000 7fd4aae79700 1 -- 172.21.15.55:0/4141672985 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a41095b0 con 0x7fd4a40fc5a0 2022-09-07T21:37:04.552 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.543+0000 7fd4a1ffb700 1 -- 172.21.15.55:0/4141672985 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fd49801e020 con 0x7fd4a40fc5a0 2022-09-07T21:37:04.552 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.544+0000 7fd4a1ffb700 1 --2- 172.21.15.55:0/4141672985 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd494099010 0x7fd49409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:04.552 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.544+0000 7fd4a1ffb700 1 -- 172.21.15.55:0/4141672985 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6257+0+0 (secure 0 0 0) 0x7fd498014070 con 0x7fd4a40fc5a0 2022-09-07T21:37:04.552 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.544+0000 7fd4a3fff700 1 --2- 172.21.15.55:0/4141672985 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd494099010 0x7fd49409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:04.553 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.545+0000 7fd4a3fff700 1 --2- 172.21.15.55:0/4141672985 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd494099010 0x7fd49409b4c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd49000b4f0 tx=0x7fd490009e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:37:04.553 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.547+0000 7fd4a1ffb700 1 -- 172.21.15.55:0/4141672985 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fd49808f0f0 con 0x7fd4a40fc5a0 2022-09-07T21:37:04.756 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:04.754+0000 7fd4aae79700 1 -- 172.21.15.55:0/4141672985 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4a405f120 con 0x7fd494099010 2022-09-07T21:37:04.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:04 smithi047 ceph-mon[90087]: Detected new or changed devices on smithi047 2022-09-07T21:37:04.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:04 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi047 to 4319M 2022-09-07T21:37:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:04 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi047 2022-09-07T21:37:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:04 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi047 to 4319M 2022-09-07T21:37:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:04 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi047 2022-09-07T21:37:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:04 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi047 to 4319M 2022-09-07T21:37:05.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:05 smithi047 ceph-mon[90087]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:05.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:05 smithi047 ceph-mon[90087]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:37:05.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:05 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:37:05.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:05 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:37:05.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:05 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:05 smithi055 ceph-mon[97033]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:05 smithi055 ceph-mon[97033]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:37:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:05 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:37:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:05 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:37:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:05 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:05 smithi059 ceph-mon[97450]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:05 smithi059 ceph-mon[97450]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:37:06.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:05 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:37:06.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:05 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:37:06.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:05 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:07.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:07 smithi047 ceph-mon[90087]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:07.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:07 smithi047 ceph-mon[90087]: from='client.? 172.21.15.55:0/423084858' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "684aed4e-909e-470a-a8e2-a0b3a52aca2f"}]: dispatch 2022-09-07T21:37:07.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:07 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "684aed4e-909e-470a-a8e2-a0b3a52aca2f"}]: dispatch 2022-09-07T21:37:07.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:07 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "684aed4e-909e-470a-a8e2-a0b3a52aca2f"}]': finished 2022-09-07T21:37:07.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:07 smithi047 ceph-mon[90087]: osdmap e69: 11 total, 10 up, 11 in 2022-09-07T21:37:07.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:07 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:07 smithi055 ceph-mon[97033]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:07 smithi055 ceph-mon[97033]: from='client.? 172.21.15.55:0/423084858' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "684aed4e-909e-470a-a8e2-a0b3a52aca2f"}]: dispatch 2022-09-07T21:37:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:07 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "684aed4e-909e-470a-a8e2-a0b3a52aca2f"}]: dispatch 2022-09-07T21:37:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:07 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "684aed4e-909e-470a-a8e2-a0b3a52aca2f"}]': finished 2022-09-07T21:37:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:07 smithi055 ceph-mon[97033]: osdmap e69: 11 total, 10 up, 11 in 2022-09-07T21:37:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:07 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:08.041 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:07 smithi059 ceph-mon[97450]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:08.041 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:07 smithi059 ceph-mon[97450]: from='client.? 172.21.15.55:0/423084858' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "684aed4e-909e-470a-a8e2-a0b3a52aca2f"}]: dispatch 2022-09-07T21:37:08.042 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:07 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "684aed4e-909e-470a-a8e2-a0b3a52aca2f"}]: dispatch 2022-09-07T21:37:08.042 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:07 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "684aed4e-909e-470a-a8e2-a0b3a52aca2f"}]': finished 2022-09-07T21:37:08.042 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:07 smithi059 ceph-mon[97450]: osdmap e69: 11 total, 10 up, 11 in 2022-09-07T21:37:08.042 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:07 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:08 smithi055 ceph-mon[97033]: from='client.? 172.21.15.55:0/3659235401' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:37:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:08 smithi055 ceph-mon[97033]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:08 smithi059 ceph-mon[97450]: from='client.? 172.21.15.55:0/3659235401' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:37:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:08 smithi059 ceph-mon[97450]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:09.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:08 smithi047 ceph-mon[90087]: from='client.? 172.21.15.55:0/3659235401' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:37:09.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:08 smithi047 ceph-mon[90087]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:11 smithi055 ceph-mon[97033]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:11.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:11 smithi047 ceph-mon[90087]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:11 smithi059 ceph-mon[97450]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:13.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:13 smithi047 ceph-mon[90087]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:13.907 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:13 smithi055 ceph-mon[97033]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:13 smithi059 ceph-mon[97450]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:14.896 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:14 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-09-07T21:37:14.897 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:14 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:14.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:14 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-09-07T21:37:14.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:14 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:14 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-09-07T21:37:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:14 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:15 smithi047 ceph-mon[90087]: Deploying daemon osd.10 on smithi055 2022-09-07T21:37:15.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:15 smithi047 ceph-mon[90087]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:15 smithi055 ceph-mon[97033]: Deploying daemon osd.10 on smithi055 2022-09-07T21:37:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:15 smithi055 ceph-mon[97033]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:15 smithi059 ceph-mon[97450]: Deploying daemon osd.10 on smithi055 2022-09-07T21:37:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:15 smithi059 ceph-mon[97450]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:17.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:17 smithi047 ceph-mon[90087]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:17 smithi055 ceph-mon[97033]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:17 smithi059 ceph-mon[97450]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:18 smithi055 ceph-mon[97033]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:18 smithi059 ceph-mon[97450]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:19.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:18 smithi047 ceph-mon[90087]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:21.365 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:21 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:21.367 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:21 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:21.367 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:21 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:21.368 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:21 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:21.368 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:21 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:21.368 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:21 smithi055 ceph-mon[97033]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:21.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:21.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:21.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:21.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:21.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:21.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:21 smithi047 ceph-mon[90087]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:21 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:21 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:21 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:21 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:21 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:21 smithi059 ceph-mon[97450]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:23.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:23 smithi047 ceph-mon[90087]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:23.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:23.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:23.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:23 smithi055 ceph-mon[97033]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:23 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:23 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:23.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:23 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:23 smithi059 ceph-mon[97450]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:23 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:23 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:24.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:23 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:24.367 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:24.363+0000 7fd4a1ffb700 1 -- 172.21.15.55:0/4141672985 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fd4a405f120 con 0x7fd494099010 2022-09-07T21:37:24.367 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 10 on host 'smithi055' 2022-09-07T21:37:24.368 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:24.366+0000 7fd4aae79700 1 -- 172.21.15.55:0/4141672985 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd494099010 msgr2=0x7fd49409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:24.368 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:24.366+0000 7fd4aae79700 1 --2- 172.21.15.55:0/4141672985 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd494099010 0x7fd49409b4c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd49000b4f0 tx=0x7fd490009e70 comp rx=0 tx=0).stop 2022-09-07T21:37:24.369 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:24.366+0000 7fd4aae79700 1 -- 172.21.15.55:0/4141672985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a40fc5a0 msgr2=0x7fd4a41939f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:24.369 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:24.366+0000 7fd4aae79700 1 --2- 172.21.15.55:0/4141672985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a40fc5a0 0x7fd4a41939f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd49800e710 tx=0x7fd49800ea20 comp rx=0 tx=0).stop 2022-09-07T21:37:24.369 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:24.366+0000 7fd4aae79700 1 -- 172.21.15.55:0/4141672985 shutdown_connections 2022-09-07T21:37:24.369 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:24.366+0000 7fd4aae79700 1 --2- 172.21.15.55:0/4141672985 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd494099010 0x7fd49409b4c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:24.370 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:24.366+0000 7fd4aae79700 1 --2- 172.21.15.55:0/4141672985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4a40ffe10 0x7fd4a419afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:24.370 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:24.366+0000 7fd4aae79700 1 --2- 172.21.15.55:0/4141672985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a40fc5a0 0x7fd4a41939f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:24.370 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:24.366+0000 7fd4aae79700 1 --2- 172.21.15.55:0/4141672985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4a40ff430 0x7fd4a4193f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:24.370 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:24.366+0000 7fd4aae79700 1 -- 172.21.15.55:0/4141672985 >> 172.21.15.55:0/4141672985 conn(0x7fd4a40f7c00 msgr2=0x7fd4a40faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:37:24.371 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:24.366+0000 7fd4aae79700 1 -- 172.21.15.55:0/4141672985 shutdown_connections 2022-09-07T21:37:24.371 INFO:teuthology.orchestra.run.smithi055.stderr:2022-09-07T21:37:24.366+0000 7fd4aae79700 1 -- 172.21.15.55:0/4141672985 wait complete. 2022-09-07T21:37:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:24 smithi055 ceph-mon[97033]: from='osd.10 [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-09-07T21:37:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:24 smithi055 ceph-mon[97033]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-09-07T21:37:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:24 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:24.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:24 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:24.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:24 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:24.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:24 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:24.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:24 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:24.893 DEBUG:teuthology.orchestra.run.smithi055:osd.10> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@osd.10.service 2022-09-07T21:37:24.897 INFO:tasks.cephadm:Deploying osd.11 on smithi059 with /dev/vg_nvme/lv_1... 2022-09-07T21:37:24.898 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-09-07T21:37:24.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:24 smithi047 ceph-mon[90087]: from='osd.10 [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-09-07T21:37:24.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:24 smithi047 ceph-mon[90087]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-09-07T21:37:24.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:24 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:24.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:24 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:24.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:24 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:24.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:24 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:24.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:24 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:24.926 INFO:journalctl@ceph.osd.10.smithi055.stdout:-- Logs begin at Wed 2022-09-07 21:07:35 UTC. -- 2022-09-07T21:37:24.929 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:24 smithi059 ceph-mon[97450]: from='osd.10 [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-09-07T21:37:24.929 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:24 smithi059 ceph-mon[97450]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-09-07T21:37:24.929 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:24 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:24.929 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:24 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:24.930 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:24 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:24.930 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:24 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:24.930 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:24 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:25.510 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.c/config 2022-09-07T21:37:25.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:25 smithi047 ceph-mon[90087]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:25.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:25 smithi047 ceph-mon[90087]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-09-07T21:37:25.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:25 smithi047 ceph-mon[90087]: osdmap e70: 11 total, 10 up, 11 in 2022-09-07T21:37:25.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:25.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:25 smithi047 ceph-mon[90087]: from='osd.10 [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:37:25.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:25 smithi047 ceph-mon[90087]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:37:25.979 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:25 smithi059 ceph-mon[97450]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:25.980 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:25 smithi059 ceph-mon[97450]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-09-07T21:37:25.980 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:25 smithi059 ceph-mon[97450]: osdmap e70: 11 total, 10 up, 11 in 2022-09-07T21:37:25.980 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:25.981 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:25 smithi059 ceph-mon[97450]: from='osd.10 [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:37:25.981 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:25 smithi059 ceph-mon[97450]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:37:25.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:25 smithi055 ceph-mon[97033]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:25.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:25 smithi055 ceph-mon[97033]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-09-07T21:37:25.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:25 smithi055 ceph-mon[97033]: osdmap e70: 11 total, 10 up, 11 in 2022-09-07T21:37:25.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:25.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:25 smithi055 ceph-mon[97033]: from='osd.10 [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:37:25.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:25 smithi055 ceph-mon[97033]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2022-09-07T21:37:25.984 INFO:journalctl@ceph.osd.10.smithi055.stdout:Sep 07 21:37:25 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-10[119927]: 2022-09-07T21:37:25.649+0000 7fdb13a2e700 -1 osd.10 0 waiting for initial osdmap 2022-09-07T21:37:25.984 INFO:journalctl@ceph.osd.10.smithi055.stdout:Sep 07 21:37:25 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-10[119927]: 2022-09-07T21:37:25.663+0000 7fdb0e020700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-07T21:37:26.927 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-09-07T21:37:26.927 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-09-07T21:37:26.928 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2022-09-07T21:37:26.928 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2022-09-07T21:37:26.929 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122314 s, 857 MB/s 2022-09-07T21:37:26.929 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2022-09-07T21:37:26.939 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:26 smithi055 ceph-mon[97033]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2022-09-07T21:37:26.939 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:26 smithi055 ceph-mon[97033]: osdmap e71: 11 total, 10 up, 11 in 2022-09-07T21:37:26.940 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:26 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:26.940 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:26 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:26.962 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:26 smithi059 ceph-mon[97450]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2022-09-07T21:37:26.963 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:26 smithi059 ceph-mon[97450]: osdmap e71: 11 total, 10 up, 11 in 2022-09-07T21:37:26.963 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:26 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:26.963 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:26 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:27.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:26 smithi047 ceph-mon[90087]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2022-09-07T21:37:27.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:26 smithi047 ceph-mon[90087]: osdmap e71: 11 total, 10 up, 11 in 2022-09-07T21:37:27.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:26 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:27.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:26 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:27.339 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi059:vg_nvme/lv_1 2022-09-07T21:37:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:27 smithi055 ceph-mon[97033]: purged_snaps scrub starts 2022-09-07T21:37:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:27 smithi055 ceph-mon[97033]: purged_snaps scrub ok 2022-09-07T21:37:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:27 smithi055 ceph-mon[97033]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:27.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:27 smithi055 ceph-mon[97033]: osd.10 [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072] boot 2022-09-07T21:37:27.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:27 smithi055 ceph-mon[97033]: osdmap e72: 11 total, 11 up, 11 in 2022-09-07T21:37:27.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:27.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:27.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:27.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:27.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:27.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:27.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:27.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:27.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:27 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:27.986 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.c/config 2022-09-07T21:37:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:27 smithi059 ceph-mon[97450]: purged_snaps scrub starts 2022-09-07T21:37:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:27 smithi059 ceph-mon[97450]: purged_snaps scrub ok 2022-09-07T21:37:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:27 smithi059 ceph-mon[97450]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:28.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:27 smithi059 ceph-mon[97450]: osd.10 [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072] boot 2022-09-07T21:37:28.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:27 smithi059 ceph-mon[97450]: osdmap e72: 11 total, 11 up, 11 in 2022-09-07T21:37:28.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:28.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:28.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:28.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:28.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:28.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:28.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:28.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:28.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:27 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:28.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:27 smithi047 ceph-mon[90087]: purged_snaps scrub starts 2022-09-07T21:37:28.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:27 smithi047 ceph-mon[90087]: purged_snaps scrub ok 2022-09-07T21:37:28.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:27 smithi047 ceph-mon[90087]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-09-07T21:37:28.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:27 smithi047 ceph-mon[90087]: osd.10 [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072] boot 2022-09-07T21:37:28.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:27 smithi047 ceph-mon[90087]: osdmap e72: 11 total, 11 up, 11 in 2022-09-07T21:37:28.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:37:28.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:28.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:28.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:28.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:28.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:28.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:28.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:28.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:27 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:28 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-07T21:37:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:28 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 4293M 2022-09-07T21:37:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:28 smithi059 ceph-mon[97450]: osdmap e73: 11 total, 11 up, 11 in 2022-09-07T21:37:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:28 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-07T21:37:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:28 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 4293M 2022-09-07T21:37:28.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:28 smithi055 ceph-mon[97033]: osdmap e73: 11 total, 11 up, 11 in 2022-09-07T21:37:29.042 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.036+0000 7f91eb99d700 1 -- 172.21.15.59:0/3828951718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e40fe970 msgr2=0x7f91e4105210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:29.042 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.036+0000 7f91eb99d700 1 --2- 172.21.15.59:0/3828951718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e40fe970 0x7f91e4105210 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009bb0 tx=0x7f91dc009ec0 comp rx=0 tx=0).stop 2022-09-07T21:37:29.042 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.036+0000 7f91eb99d700 1 -- 172.21.15.59:0/3828951718 shutdown_connections 2022-09-07T21:37:29.043 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.036+0000 7f91eb99d700 1 --2- 172.21.15.59:0/3828951718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e40fe970 0x7f91e4105210 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:29.043 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.036+0000 7f91eb99d700 1 --2- 172.21.15.59:0/3828951718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e40fcde0 0x7f91e40fd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:29.043 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.036+0000 7f91eb99d700 1 --2- 172.21.15.59:0/3828951718 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91e40fdfc0 0x7f91e40fe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:29.043 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.036+0000 7f91eb99d700 1 -- 172.21.15.59:0/3828951718 >> 172.21.15.59:0/3828951718 conn(0x7f91e40f8430 msgr2=0x7f91e40fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:37:29.044 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.037+0000 7f91eb99d700 1 -- 172.21.15.59:0/3828951718 shutdown_connections 2022-09-07T21:37:29.044 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.037+0000 7f91eb99d700 1 -- 172.21.15.59:0/3828951718 wait complete. 2022-09-07T21:37:29.044 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.037+0000 7f91eb99d700 1 Processor -- start 2022-09-07T21:37:29.044 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.037+0000 7f91eb99d700 1 -- start start 2022-09-07T21:37:29.044 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.037+0000 7f91eb99d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e40fcde0 0x7f91e4067250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:29.045 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.037+0000 7f91eb99d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91e40fdfc0 0x7f91e41987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:29.045 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.037+0000 7f91eb99d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e40fe970 0x7f91e419ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:29.045 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.037+0000 7f91eb99d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e419b640 con 0x7f91e40fcde0 2022-09-07T21:37:29.045 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.037+0000 7f91eb99d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e419b7b0 con 0x7f91e40fdfc0 2022-09-07T21:37:29.045 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.037+0000 7f91eb99d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e419b970 con 0x7f91e40fe970 2022-09-07T21:37:29.046 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.037+0000 7f91e9f3a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e40fe970 0x7f91e419ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:29.046 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.037+0000 7f91e9f3a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e40fe970 0x7f91e419ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40518/0 (socket says 172.21.15.59:40518) 2022-09-07T21:37:29.046 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91e9f3a700 1 -- 172.21.15.59:0/2752644635 learned_addr learned my addr 172.21.15.59:0/2752644635 (peer_addr_for_me v2:172.21.15.59:0/0) 2022-09-07T21:37:29.046 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91e8f38700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91e40fdfc0 0x7f91e41987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:29.047 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91e9739700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e40fcde0 0x7f91e4067250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:29.047 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91e9f3a700 1 -- 172.21.15.59:0/2752644635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91e40fdfc0 msgr2=0x7f91e41987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:29.047 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91e9f3a700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91e40fdfc0 0x7f91e41987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:29.047 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91e9f3a700 1 -- 172.21.15.59:0/2752644635 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e40fcde0 msgr2=0x7f91e4067250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:29.047 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91e9f3a700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e40fcde0 0x7f91e4067250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:29.048 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91e9f3a700 1 -- 172.21.15.59:0/2752644635 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc0097e0 con 0x7f91e40fe970 2022-09-07T21:37:29.048 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91e9739700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e40fcde0 0x7f91e4067250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:37:29.048 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91e9f3a700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e40fe970 0x7f91e419ae70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0092d0 tx=0x7f91dc005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:37:29.048 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91da7fc700 1 -- 172.21.15.59:0/2752644635 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc01c070 con 0x7f91e40fe970 2022-09-07T21:37:29.048 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91da7fc700 1 -- 172.21.15.59:0/2752644635 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f91dc003d20 con 0x7f91e40fe970 2022-09-07T21:37:29.049 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91eb99d700 1 -- 172.21.15.59:0/2752644635 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e419bbf0 con 0x7f91e40fe970 2022-09-07T21:37:29.049 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91da7fc700 1 -- 172.21.15.59:0/2752644635 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc0056f0 con 0x7f91e40fe970 2022-09-07T21:37:29.049 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.038+0000 7f91eb99d700 1 -- 172.21.15.59:0/2752644635 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e419c0a0 con 0x7f91e40fe970 2022-09-07T21:37:29.049 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.039+0000 7f91eb99d700 1 -- 172.21.15.59:0/2752644635 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e4108530 con 0x7f91e40fe970 2022-09-07T21:37:29.050 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.040+0000 7f91da7fc700 1 -- 172.21.15.59:0/2752644635 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f91dc027020 con 0x7f91e40fe970 2022-09-07T21:37:29.051 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.041+0000 7f91da7fc700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f91d0099010 0x7f91d009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:29.051 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.041+0000 7f91da7fc700 1 -- 172.21.15.59:0/2752644635 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f91dc0c3490 con 0x7f91e40fe970 2022-09-07T21:37:29.051 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.041+0000 7f91e9739700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f91d0099010 0x7f91d009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:29.051 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.042+0000 7f91e9739700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f91d0099010 0x7f91d009b4c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f91e0006fd0 tx=0x7f91e0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:37:29.052 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.044+0000 7f91da7fc700 1 -- 172.21.15.59:0/2752644635 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f91dc090120 con 0x7f91e40fe970 2022-09-07T21:37:29.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:28 smithi047 ceph-mon[90087]: Detected new or changed devices on smithi055 2022-09-07T21:37:29.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:28 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi055 to 4293M 2022-09-07T21:37:29.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:28 smithi047 ceph-mon[90087]: osdmap e73: 11 total, 11 up, 11 in 2022-09-07T21:37:29.253 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:29.250+0000 7f91eb99d700 1 -- 172.21.15.59:0/2752644635 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f91e405fb70 con 0x7f91d0099010 2022-09-07T21:37:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:29 smithi055 ceph-mon[97033]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2022-09-07T21:37:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:29 smithi055 ceph-mon[97033]: osdmap e74: 11 total, 11 up, 11 in 2022-09-07T21:37:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:37:29.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:37:29.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:29 smithi059 ceph-mon[97450]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2022-09-07T21:37:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:29 smithi059 ceph-mon[97450]: osdmap e74: 11 total, 11 up, 11 in 2022-09-07T21:37:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:37:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:37:30.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:30.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:29 smithi047 ceph-mon[90087]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2022-09-07T21:37:30.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:29 smithi047 ceph-mon[90087]: osdmap e74: 11 total, 11 up, 11 in 2022-09-07T21:37:30.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-07T21:37:30.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-07T21:37:30.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:30 smithi055 ceph-mon[97033]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:37:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:30 smithi055 ceph-mon[97033]: osdmap e75: 11 total, 11 up, 11 in 2022-09-07T21:37:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:30 smithi059 ceph-mon[97450]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:37:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:30 smithi059 ceph-mon[97450]: osdmap e75: 11 total, 11 up, 11 in 2022-09-07T21:37:31.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:30 smithi047 ceph-mon[90087]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:37:31.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:30 smithi047 ceph-mon[90087]: osdmap e75: 11 total, 11 up, 11 in 2022-09-07T21:37:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:31 smithi055 ceph-mon[97033]: pgmap v267: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2022-09-07T21:37:31.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:31 smithi055 ceph-mon[97033]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2022-09-07T21:37:31.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:31 smithi055 ceph-mon[97033]: osdmap e76: 11 total, 11 up, 11 in 2022-09-07T21:37:31.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:31 smithi055 ceph-mon[97033]: from='client.? 172.21.15.59:0/3612564841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c3afb2-663c-434c-980a-37b1f230b170"}]: dispatch 2022-09-07T21:37:31.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:31 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c3afb2-663c-434c-980a-37b1f230b170"}]: dispatch 2022-09-07T21:37:31.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:31 smithi055 ceph-mon[97033]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5c3afb2-663c-434c-980a-37b1f230b170"}]': finished 2022-09-07T21:37:31.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:31 smithi055 ceph-mon[97033]: osdmap e77: 12 total, 11 up, 12 in 2022-09-07T21:37:31.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:31 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:31 smithi059 ceph-mon[97450]: pgmap v267: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2022-09-07T21:37:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:31 smithi059 ceph-mon[97450]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2022-09-07T21:37:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:31 smithi059 ceph-mon[97450]: osdmap e76: 11 total, 11 up, 11 in 2022-09-07T21:37:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:31 smithi059 ceph-mon[97450]: from='client.? 172.21.15.59:0/3612564841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c3afb2-663c-434c-980a-37b1f230b170"}]: dispatch 2022-09-07T21:37:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:31 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c3afb2-663c-434c-980a-37b1f230b170"}]: dispatch 2022-09-07T21:37:32.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:31 smithi059 ceph-mon[97450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5c3afb2-663c-434c-980a-37b1f230b170"}]': finished 2022-09-07T21:37:32.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:31 smithi059 ceph-mon[97450]: osdmap e77: 12 total, 11 up, 12 in 2022-09-07T21:37:32.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:31 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:32.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:31 smithi047 ceph-mon[90087]: pgmap v267: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2022-09-07T21:37:32.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:31 smithi047 ceph-mon[90087]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2022-09-07T21:37:32.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:31 smithi047 ceph-mon[90087]: osdmap e76: 11 total, 11 up, 11 in 2022-09-07T21:37:32.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:31 smithi047 ceph-mon[90087]: from='client.? 172.21.15.59:0/3612564841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c3afb2-663c-434c-980a-37b1f230b170"}]: dispatch 2022-09-07T21:37:32.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:31 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c3afb2-663c-434c-980a-37b1f230b170"}]: dispatch 2022-09-07T21:37:32.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:31 smithi047 ceph-mon[90087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5c3afb2-663c-434c-980a-37b1f230b170"}]': finished 2022-09-07T21:37:32.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:31 smithi047 ceph-mon[90087]: osdmap e77: 12 total, 11 up, 12 in 2022-09-07T21:37:32.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:31 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:32 smithi055 ceph-mon[97033]: from='client.? 172.21.15.59:0/3978480273' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:37:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:32 smithi059 ceph-mon[97450]: from='client.? 172.21.15.59:0/3978480273' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:37:33.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:32 smithi047 ceph-mon[90087]: from='client.? 172.21.15.59:0/3978480273' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-07T21:37:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:33 smithi055 ceph-mon[97033]: pgmap v270: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2022-09-07T21:37:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:33 smithi059 ceph-mon[97450]: pgmap v270: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2022-09-07T21:37:34.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:33 smithi047 ceph-mon[90087]: pgmap v270: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2022-09-07T21:37:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:34 smithi059 ceph-mon[97450]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2022-09-07T21:37:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:34 smithi059 ceph-mon[97450]: Cluster is now healthy 2022-09-07T21:37:35.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:34 smithi047 ceph-mon[90087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2022-09-07T21:37:35.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:34 smithi047 ceph-mon[90087]: Cluster is now healthy 2022-09-07T21:37:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:34 smithi055 ceph-mon[97033]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2022-09-07T21:37:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:34 smithi055 ceph-mon[97033]: Cluster is now healthy 2022-09-07T21:37:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:35 smithi059 ceph-mon[97450]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 98 KiB/s, 0 objects/s recovering 2022-09-07T21:37:36.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:35 smithi047 ceph-mon[90087]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 98 KiB/s, 0 objects/s recovering 2022-09-07T21:37:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:35 smithi055 ceph-mon[97033]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 98 KiB/s, 0 objects/s recovering 2022-09-07T21:37:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:37 smithi059 ceph-mon[97450]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 84 KiB/s, 0 objects/s recovering 2022-09-07T21:37:38.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:37 smithi047 ceph-mon[90087]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 84 KiB/s, 0 objects/s recovering 2022-09-07T21:37:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:37 smithi055 ceph-mon[97033]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 84 KiB/s, 0 objects/s recovering 2022-09-07T21:37:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:38 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-09-07T21:37:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:38 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:38 smithi055 ceph-mon[97033]: Deploying daemon osd.11 on smithi059 2022-09-07T21:37:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:38 smithi055 ceph-mon[97033]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-07T21:37:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:38 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-09-07T21:37:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:38 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:39.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:38 smithi059 ceph-mon[97450]: Deploying daemon osd.11 on smithi059 2022-09-07T21:37:39.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:38 smithi059 ceph-mon[97450]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-07T21:37:39.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:38 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-09-07T21:37:39.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:38 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:39.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:38 smithi047 ceph-mon[90087]: Deploying daemon osd.11 on smithi059 2022-09-07T21:37:39.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:38 smithi047 ceph-mon[90087]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-07T21:37:41.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:41 smithi047 ceph-mon[90087]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-07T21:37:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:41 smithi055 ceph-mon[97033]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-07T21:37:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:41 smithi059 ceph-mon[97450]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-07T21:37:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:43 smithi059 ceph-mon[97450]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-09-07T21:37:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:43 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:43 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:43.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:43 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:43.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:43 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:43.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:43 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:43.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:43 smithi047 ceph-mon[90087]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-09-07T21:37:43.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:43 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:43.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:43 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:43.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:43 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:43.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:43 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:43.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:43 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:43 smithi055 ceph-mon[97033]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-09-07T21:37:43.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:43 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:43.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:43 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:43.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:43 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:43.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:43 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:43.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:43 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:45.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:45 smithi047 ceph-mon[90087]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2022-09-07T21:37:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:45 smithi055 ceph-mon[97033]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2022-09-07T21:37:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:45 smithi059 ceph-mon[97450]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2022-09-07T21:37:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:47.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:37:47.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:37:47.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:46 smithi055 ceph-mon[97033]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-09-07T21:37:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:37:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:37:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:46 smithi059 ceph-mon[97450]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-09-07T21:37:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:47.355+0000 7f91da7fc700 1 -- 172.21.15.59:0/2752644635 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f91e405fb70 con 0x7f91d0099010 2022-09-07T21:37:47.359 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 11 on host 'smithi059' 2022-09-07T21:37:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:47.357+0000 7f91eb99d700 1 -- 172.21.15.59:0/2752644635 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f91d0099010 msgr2=0x7f91d009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:47.361 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:47.357+0000 7f91eb99d700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f91d0099010 0x7f91d009b4c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f91e0006fd0 tx=0x7f91e0008040 comp rx=0 tx=0).stop 2022-09-07T21:37:47.361 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:47.357+0000 7f91eb99d700 1 -- 172.21.15.59:0/2752644635 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e40fe970 msgr2=0x7f91e419ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:47.361 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:47.357+0000 7f91eb99d700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e40fe970 0x7f91e419ae70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0092d0 tx=0x7f91dc005c90 comp rx=0 tx=0).stop 2022-09-07T21:37:47.361 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:47.358+0000 7f91eb99d700 1 -- 172.21.15.59:0/2752644635 shutdown_connections 2022-09-07T21:37:47.362 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:47.358+0000 7f91eb99d700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f91d0099010 0x7f91d009b4c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:47.362 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:47.358+0000 7f91eb99d700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e40fe970 0x7f91e419ae70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:47.362 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:47.358+0000 7f91eb99d700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e40fcde0 0x7f91e4067250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:47.362 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:47.358+0000 7f91eb99d700 1 --2- 172.21.15.59:0/2752644635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91e40fdfc0 0x7f91e41987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:47.358+0000 7f91eb99d700 1 -- 172.21.15.59:0/2752644635 >> 172.21.15.59:0/2752644635 conn(0x7f91e40f8430 msgr2=0x7f91e41014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:37:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:47.358+0000 7f91eb99d700 1 -- 172.21.15.59:0/2752644635 shutdown_connections 2022-09-07T21:37:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:37:47.358+0000 7f91eb99d700 1 -- 172.21.15.59:0/2752644635 wait complete. 2022-09-07T21:37:47.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:47.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:47.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:47.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:37:47.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:37:47.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:46 smithi047 ceph-mon[90087]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-09-07T21:37:47.754 DEBUG:teuthology.orchestra.run.smithi059:osd.11> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@osd.11.service 2022-09-07T21:37:47.756 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2022-09-07T21:37:47.757 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd stat -f json 2022-09-07T21:37:47.785 INFO:journalctl@ceph.osd.11.smithi059.stdout:-- Logs begin at Wed 2022-09-07 21:07:37 UTC. -- 2022-09-07T21:37:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:48 smithi059 ceph-mon[97450]: from='osd.11 [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-09-07T21:37:48.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:48 smithi059 ceph-mon[97450]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-09-07T21:37:48.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:48.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:48.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:48.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:48.647 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:37:48.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:48.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:48 smithi047 ceph-mon[90087]: from='osd.11 [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-09-07T21:37:48.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:48 smithi047 ceph-mon[90087]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-09-07T21:37:48.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:48.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:48.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:48.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:48 smithi055 ceph-mon[97033]: from='osd.11 [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-09-07T21:37:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:48 smithi055 ceph-mon[97033]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-09-07T21:37:48.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:48.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:37:48.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:37:48.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:37:49.638 INFO:journalctl@ceph.osd.11.smithi059.stdout:Sep 07 21:37:49 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-11[120911]: 2022-09-07T21:37:49.355+0000 7f2dd5309700 -1 osd.11 0 waiting for initial osdmap 2022-09-07T21:37:49.638 INFO:journalctl@ceph.osd.11.smithi059.stdout:Sep 07 21:37:49 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-11[120911]: 2022-09-07T21:37:49.374+0000 7f2dcf0fa700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-07T21:37:49.639 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:49 smithi059 ceph-mon[97450]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-09-07T21:37:49.639 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:49 smithi059 ceph-mon[97450]: osdmap e78: 12 total, 11 up, 12 in 2022-09-07T21:37:49.639 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:49 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:49.639 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:49 smithi059 ceph-mon[97450]: from='osd.11 [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:37:49.640 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:49 smithi059 ceph-mon[97450]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:37:49.640 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:49 smithi059 ceph-mon[97450]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-09-07T21:37:49.640 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:49 smithi059 ceph-mon[97450]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2022-09-07T21:37:49.646 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:49 smithi059 ceph-mon[97450]: osdmap e79: 12 total, 11 up, 12 in 2022-09-07T21:37:49.647 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:49 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:49.647 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:49 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:49.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:49 smithi047 ceph-mon[90087]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-09-07T21:37:49.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:49 smithi047 ceph-mon[90087]: osdmap e78: 12 total, 11 up, 12 in 2022-09-07T21:37:49.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:49 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:49.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:49 smithi047 ceph-mon[90087]: from='osd.11 [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:37:49.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:49 smithi047 ceph-mon[90087]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:37:49.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:49 smithi047 ceph-mon[90087]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-09-07T21:37:49.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:49 smithi047 ceph-mon[90087]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2022-09-07T21:37:49.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:49 smithi047 ceph-mon[90087]: osdmap e79: 12 total, 11 up, 12 in 2022-09-07T21:37:49.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:49 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:49.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:49 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:49 smithi055 ceph-mon[97033]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-09-07T21:37:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:49 smithi055 ceph-mon[97033]: osdmap e78: 12 total, 11 up, 12 in 2022-09-07T21:37:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:49 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:49.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:49 smithi055 ceph-mon[97033]: from='osd.11 [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:37:49.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:49 smithi055 ceph-mon[97033]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2022-09-07T21:37:49.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:49 smithi055 ceph-mon[97033]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-09-07T21:37:49.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:49 smithi055 ceph-mon[97033]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2022-09-07T21:37:49.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:49 smithi055 ceph-mon[97033]: osdmap e79: 12 total, 11 up, 12 in 2022-09-07T21:37:49.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:49 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:49.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:49 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:50.567 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.564+0000 7f52bcea4700 1 -- 172.21.15.47:0/76223487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52b80ff080 msgr2=0x7f52b80ff4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:50.567 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.564+0000 7f52bcea4700 1 --2- 172.21.15.47:0/76223487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52b80ff080 0x7f52b80ff4f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f52ac009840 tx=0x7f52ac009b50 comp rx=0 tx=0).stop 2022-09-07T21:37:50.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.564+0000 7f52bcea4700 1 -- 172.21.15.47:0/76223487 shutdown_connections 2022-09-07T21:37:50.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.564+0000 7f52bcea4700 1 --2- 172.21.15.47:0/76223487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52b80ffa30 0x7f52b8106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:50.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.564+0000 7f52bcea4700 1 --2- 172.21.15.47:0/76223487 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52b80fde10 0x7f52b80fe220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:50.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.564+0000 7f52bcea4700 1 --2- 172.21.15.47:0/76223487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52b80ff080 0x7f52b80ff4f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:50.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.564+0000 7f52bcea4700 1 -- 172.21.15.47:0/76223487 >> 172.21.15.47:0/76223487 conn(0x7f52b80f9570 msgr2=0x7f52b80fb980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:37:50.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.564+0000 7f52bcea4700 1 -- 172.21.15.47:0/76223487 shutdown_connections 2022-09-07T21:37:50.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.564+0000 7f52bcea4700 1 -- 172.21.15.47:0/76223487 wait complete. 2022-09-07T21:37:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.565+0000 7f52bcea4700 1 Processor -- start 2022-09-07T21:37:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.566+0000 7f52bcea4700 1 -- start start 2022-09-07T21:37:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.566+0000 7f52bcea4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52b80fde10 0x7f52b8195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.566+0000 7f52bcea4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52b80ff080 0x7f52b81961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.566+0000 7f52bcea4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52b80ffa30 0x7f52b819d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.566+0000 7f52bcea4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b819d820 con 0x7f52b80ffa30 2022-09-07T21:37:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.566+0000 7f52bcea4700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b819d990 con 0x7f52b80fde10 2022-09-07T21:37:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.566+0000 7f52bcea4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b819db50 con 0x7f52b80ff080 2022-09-07T21:37:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.566+0000 7f52b5d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52b80ff080 0x7f52b81961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.566+0000 7f52b6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52b80ffa30 0x7f52b819d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.566+0000 7f52b5d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52b80ff080 0x7f52b81961b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:55450/0 (socket says 172.21.15.47:55450) 2022-09-07T21:37:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.566+0000 7f52b5d9b700 1 -- 172.21.15.47:0/1402341252 learned_addr learned my addr 172.21.15.47:0/1402341252 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:37:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.566+0000 7f52b659c700 1 --2- 172.21.15.47:0/1402341252 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52b80fde10 0x7f52b8195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:50.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.567+0000 7f52b5d9b700 1 -- 172.21.15.47:0/1402341252 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52b80fde10 msgr2=0x7f52b8195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:50.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.567+0000 7f52b5d9b700 1 --2- 172.21.15.47:0/1402341252 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52b80fde10 0x7f52b8195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:50.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.567+0000 7f52b5d9b700 1 -- 172.21.15.47:0/1402341252 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52b80ffa30 msgr2=0x7f52b819d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:50.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.567+0000 7f52b5d9b700 1 --2- 172.21.15.47:0/1402341252 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52b80ffa30 0x7f52b819d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:50.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.567+0000 7f52b5d9b700 1 -- 172.21.15.47:0/1402341252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b0009730 con 0x7f52b80ff080 2022-09-07T21:37:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.567+0000 7f52b5d9b700 1 --2- 172.21.15.47:0/1402341252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52b80ff080 0x7f52b81961b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00b590 tx=0x7f52ac00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:37:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.567+0000 7f52a77fe700 1 -- 172.21.15.47:0/1402341252 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac01d070 con 0x7f52b80ff080 2022-09-07T21:37:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.568+0000 7f52a77fe700 1 -- 172.21.15.47:0/1402341252 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f52ac00f830 con 0x7f52b80ff080 2022-09-07T21:37:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.568+0000 7f52bcea4700 1 -- 172.21.15.47:0/1402341252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ac009470 con 0x7f52b80ff080 2022-09-07T21:37:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.568+0000 7f52a77fe700 1 -- 172.21.15.47:0/1402341252 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac004860 con 0x7f52b80ff080 2022-09-07T21:37:50.577 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.568+0000 7f52bcea4700 1 -- 172.21.15.47:0/1402341252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b819e140 con 0x7f52b80ff080 2022-09-07T21:37:50.577 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.569+0000 7f52bcea4700 1 -- 172.21.15.47:0/1402341252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b8109630 con 0x7f52b80ff080 2022-09-07T21:37:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.571+0000 7f52a77fe700 1 -- 172.21.15.47:0/1402341252 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f52ac028030 con 0x7f52b80ff080 2022-09-07T21:37:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.572+0000 7f52a77fe700 1 --2- 172.21.15.47:0/1402341252 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f52a0099090 0x7f52a009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.572+0000 7f52a77fe700 1 -- 172.21.15.47:0/1402341252 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f52ac0c2ac0 con 0x7f52b80ff080 2022-09-07T21:37:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.572+0000 7f52b659c700 1 --2- 172.21.15.47:0/1402341252 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f52a0099090 0x7f52a009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.573+0000 7f52b659c700 1 --2- 172.21.15.47:0/1402341252 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f52a0099090 0x7f52a009b540 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f52b8196b40 tx=0x7f52a8005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:37:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.574+0000 7f52a77fe700 1 -- 172.21.15.47:0/1402341252 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f52ac08f5b0 con 0x7f52b80ff080 2022-09-07T21:37:50.799 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.797+0000 7f52bcea4700 1 -- 172.21.15.47:0/1402341252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f52b819e6f0 con 0x7f52b80ff080 2022-09-07T21:37:50.800 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.798+0000 7f52a77fe700 1 -- 172.21.15.47:0/1402341252 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) v1 ==== 74+0+133 (secure 0 0 0) 0x7f52ac092bd0 con 0x7f52b80ff080 2022-09-07T21:37:50.801 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:37:50.803 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.800+0000 7f52bcea4700 1 -- 172.21.15.47:0/1402341252 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f52a0099090 msgr2=0x7f52a009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:50.803 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.800+0000 7f52bcea4700 1 --2- 172.21.15.47:0/1402341252 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f52a0099090 0x7f52a009b540 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f52b8196b40 tx=0x7f52a8005ea0 comp rx=0 tx=0).stop 2022-09-07T21:37:50.803 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.800+0000 7f52bcea4700 1 -- 172.21.15.47:0/1402341252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52b80ff080 msgr2=0x7f52b81961b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:50.803 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.800+0000 7f52bcea4700 1 --2- 172.21.15.47:0/1402341252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52b80ff080 0x7f52b81961b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00b590 tx=0x7f52ac00f690 comp rx=0 tx=0).stop 2022-09-07T21:37:50.804 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.800+0000 7f52bcea4700 1 -- 172.21.15.47:0/1402341252 shutdown_connections 2022-09-07T21:37:50.804 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.800+0000 7f52bcea4700 1 --2- 172.21.15.47:0/1402341252 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f52a0099090 0x7f52a009b540 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:50.804 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.800+0000 7f52bcea4700 1 --2- 172.21.15.47:0/1402341252 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52b80ffa30 0x7f52b819d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:50.804 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.800+0000 7f52bcea4700 1 --2- 172.21.15.47:0/1402341252 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52b80fde10 0x7f52b8195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:50.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.800+0000 7f52bcea4700 1 --2- 172.21.15.47:0/1402341252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52b80ff080 0x7f52b81961b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:50.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.800+0000 7f52bcea4700 1 -- 172.21.15.47:0/1402341252 >> 172.21.15.47:0/1402341252 conn(0x7f52b80f9570 msgr2=0x7f52b80fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:37:50.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.800+0000 7f52bcea4700 1 -- 172.21.15.47:0/1402341252 shutdown_connections 2022-09-07T21:37:50.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:50.800+0000 7f52bcea4700 1 -- 172.21.15.47:0/1402341252 wait complete. 2022-09-07T21:37:51.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: purged_snaps scrub starts 2022-09-07T21:37:51.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: purged_snaps scrub ok 2022-09-07T21:37:51.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: Detected new or changed devices on smithi059 2022-09-07T21:37:51.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:51.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:51.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:51.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:51.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:51.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:51.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi059 to 4293M 2022-09-07T21:37:51.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:51.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:51.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:51.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: osd.11 [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315] boot 2022-09-07T21:37:51.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: osdmap e80: 12 total, 12 up, 12 in 2022-09-07T21:37:51.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:51.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-09-07T21:37:51.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:51 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1402341252' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-09-07T21:37:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: purged_snaps scrub starts 2022-09-07T21:37:51.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: purged_snaps scrub ok 2022-09-07T21:37:51.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-07T21:37:51.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:51.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:51.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:51.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:51.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:51.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:51.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 4293M 2022-09-07T21:37:51.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:51.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:51.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:51.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: osd.11 [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315] boot 2022-09-07T21:37:51.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: osdmap e80: 12 total, 12 up, 12 in 2022-09-07T21:37:51.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:51.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-09-07T21:37:51.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:51 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1402341252' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-09-07T21:37:51.320 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1662586670,"num_in_osds":12,"osd_in_since":1662586651,"num_remapped_pgs":0} 2022-09-07T21:37:51.321 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd dump --format=json 2022-09-07T21:37:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: purged_snaps scrub starts 2022-09-07T21:37:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: purged_snaps scrub ok 2022-09-07T21:37:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-07T21:37:51.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:51.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:51.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:51.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:51.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:51.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:37:51.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 4293M 2022-09-07T21:37:51.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:51.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:37:51.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:51.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: osd.11 [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315] boot 2022-09-07T21:37:51.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: osdmap e80: 12 total, 12 up, 12 in 2022-09-07T21:37:51.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:37:51.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-09-07T21:37:51.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:51 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1402341252' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-09-07T21:37:52.210 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:37:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:53 smithi059 ceph-mon[97450]: osdmap e81: 12 total, 12 up, 12 in 2022-09-07T21:37:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:53 smithi059 ceph-mon[97450]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:37:53.327 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:53 smithi047 ceph-mon[90087]: osdmap e81: 12 total, 12 up, 12 in 2022-09-07T21:37:53.328 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:53 smithi047 ceph-mon[90087]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:37:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:53 smithi055 ceph-mon[97033]: osdmap e81: 12 total, 12 up, 12 in 2022-09-07T21:37:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:53 smithi055 ceph-mon[97033]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:37:54.097 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.094+0000 7f345b4fd700 1 -- 172.21.15.47:0/2092483875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f345410aa10 msgr2=0x7f345410cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:54.098 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.094+0000 7f345b4fd700 1 --2- 172.21.15.47:0/2092483875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f345410aa10 0x7f345410cdf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f344c009bb0 tx=0x7f344c009ec0 comp rx=0 tx=0).stop 2022-09-07T21:37:54.098 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.094+0000 7f345b4fd700 1 -- 172.21.15.47:0/2092483875 shutdown_connections 2022-09-07T21:37:54.098 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.094+0000 7f345b4fd700 1 --2- 172.21.15.47:0/2092483875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f345410aa10 0x7f345410cdf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:54.098 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.094+0000 7f345b4fd700 1 --2- 172.21.15.47:0/2092483875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34540666c0 0x7f3454066a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:54.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.094+0000 7f345b4fd700 1 --2- 172.21.15.47:0/2092483875 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3454066fd0 0x7f345410a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:54.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.094+0000 7f345b4fd700 1 -- 172.21.15.47:0/2092483875 >> 172.21.15.47:0/2092483875 conn(0x7f3454073cf0 msgr2=0x7f34540740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:37:54.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.094+0000 7f345b4fd700 1 -- 172.21.15.47:0/2092483875 shutdown_connections 2022-09-07T21:37:54.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.094+0000 7f345b4fd700 1 -- 172.21.15.47:0/2092483875 wait complete. 2022-09-07T21:37:54.101 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.095+0000 7f345b4fd700 1 Processor -- start 2022-09-07T21:37:54.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.096+0000 7f345b4fd700 1 -- start start 2022-09-07T21:37:54.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.096+0000 7f345b4fd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34540666c0 0x7f3454195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:54.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.096+0000 7f345b4fd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3454066fd0 0x7f34541963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:54.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.096+0000 7f345b4fd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f345410aa10 0x7f345419a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:54.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.096+0000 7f345b4fd700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345419aee0 con 0x7f34540666c0 2022-09-07T21:37:54.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.096+0000 7f345b4fd700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345419b050 con 0x7f3454066fd0 2022-09-07T21:37:54.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.096+0000 7f345b4fd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345419b210 con 0x7f345410aa10 2022-09-07T21:37:54.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.096+0000 7f3459299700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34540666c0 0x7f3454195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:54.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.096+0000 7f3459299700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34540666c0 0x7f3454195eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42936/0 (socket says 172.21.15.47:42936) 2022-09-07T21:37:54.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.096+0000 7f3459299700 1 -- 172.21.15.47:0/527482318 learned_addr learned my addr 172.21.15.47:0/527482318 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:37:54.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.097+0000 7f3458a98700 1 --2- 172.21.15.47:0/527482318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3454066fd0 0x7f34541963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:54.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.097+0000 7f3459a9a700 1 --2- 172.21.15.47:0/527482318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f345410aa10 0x7f345419a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:54.105 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.097+0000 7f3459299700 1 -- 172.21.15.47:0/527482318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f345410aa10 msgr2=0x7f345419a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:54.105 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.097+0000 7f3459299700 1 --2- 172.21.15.47:0/527482318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f345410aa10 0x7f345419a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:54.105 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.097+0000 7f3459299700 1 -- 172.21.15.47:0/527482318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3454066fd0 msgr2=0x7f34541963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:54.105 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.097+0000 7f3459299700 1 --2- 172.21.15.47:0/527482318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3454066fd0 0x7f34541963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:54.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.097+0000 7f3459299700 1 -- 172.21.15.47:0/527482318 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c0097e0 con 0x7f34540666c0 2022-09-07T21:37:54.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.097+0000 7f3459299700 1 --2- 172.21.15.47:0/527482318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34540666c0 0x7f3454195eb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f345000e740 tx=0x7f345000ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:37:54.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.098+0000 7f344a7fc700 1 -- 172.21.15.47:0/527482318 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450005220 con 0x7f34540666c0 2022-09-07T21:37:54.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.098+0000 7f345b4fd700 1 -- 172.21.15.47:0/527482318 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f345419b4f0 con 0x7f34540666c0 2022-09-07T21:37:54.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.098+0000 7f344a7fc700 1 -- 172.21.15.47:0/527482318 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3450005380 con 0x7f34540666c0 2022-09-07T21:37:54.107 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.098+0000 7f344a7fc700 1 -- 172.21.15.47:0/527482318 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450004b70 con 0x7f34540666c0 2022-09-07T21:37:54.107 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.098+0000 7f345b4fd700 1 -- 172.21.15.47:0/527482318 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34541a2d30 con 0x7f34540666c0 2022-09-07T21:37:54.107 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.099+0000 7f345b4fd700 1 -- 172.21.15.47:0/527482318 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3454064ea0 con 0x7f34540666c0 2022-09-07T21:37:54.109 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.101+0000 7f344a7fc700 1 -- 172.21.15.47:0/527482318 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f345001e020 con 0x7f34540666c0 2022-09-07T21:37:54.109 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.102+0000 7f344a7fc700 1 --2- 172.21.15.47:0/527482318 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3440099090 0x7f344009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:54.109 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.102+0000 7f344a7fc700 1 -- 172.21.15.47:0/527482318 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f3450014070 con 0x7f34540666c0 2022-09-07T21:37:54.109 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.102+0000 7f3458a98700 1 --2- 172.21.15.47:0/527482318 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3440099090 0x7f344009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:54.110 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.103+0000 7f3458a98700 1 --2- 172.21.15.47:0/527482318 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3440099090 0x7f344009b540 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f34541974c0 tx=0x7f3444005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:37:54.110 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.106+0000 7f344a7fc700 1 -- 172.21.15.47:0/527482318 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f345008eb30 con 0x7f34540666c0 2022-09-07T21:37:54.332 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.330+0000 7f345b4fd700 1 -- 172.21.15.47:0/527482318 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34541a33a0 con 0x7f34540666c0 2022-09-07T21:37:54.333 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.331+0000 7f344a7fc700 1 -- 172.21.15.47:0/527482318 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18053 (secure 0 0 0) 0x7f3450092150 con 0x7f34540666c0 2022-09-07T21:37:54.334 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:37:54.334 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":81,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:37:52.015572+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:37:54.335 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.333+0000 7f345b4fd700 1 -- 172.21.15.47:0/527482318 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3440099090 msgr2=0x7f344009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:54.335 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.333+0000 7f345b4fd700 1 --2- 172.21.15.47:0/527482318 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3440099090 0x7f344009b540 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f34541974c0 tx=0x7f3444005ea0 comp rx=0 tx=0).stop 2022-09-07T21:37:54.336 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.333+0000 7f345b4fd700 1 -- 172.21.15.47:0/527482318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34540666c0 msgr2=0x7f3454195eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:54.336 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.333+0000 7f345b4fd700 1 --2- 172.21.15.47:0/527482318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34540666c0 0x7f3454195eb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f345000e740 tx=0x7f345000ea50 comp rx=0 tx=0).stop 2022-09-07T21:37:54.337 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.333+0000 7f345b4fd700 1 -- 172.21.15.47:0/527482318 shutdown_connections 2022-09-07T21:37:54.337 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.333+0000 7f345b4fd700 1 --2- 172.21.15.47:0/527482318 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3440099090 0x7f344009b540 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:54.337 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.333+0000 7f345b4fd700 1 --2- 172.21.15.47:0/527482318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f345410aa10 0x7f345419a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:54.338 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.333+0000 7f345b4fd700 1 --2- 172.21.15.47:0/527482318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34540666c0 0x7f3454195eb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:54.338 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.333+0000 7f345b4fd700 1 --2- 172.21.15.47:0/527482318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3454066fd0 0x7f34541963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:54.339 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.333+0000 7f345b4fd700 1 -- 172.21.15.47:0/527482318 >> 172.21.15.47:0/527482318 conn(0x7f3454073cf0 msgr2=0x7f34540fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:37:54.339 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.333+0000 7f345b4fd700 1 -- 172.21.15.47:0/527482318 shutdown_connections 2022-09-07T21:37:54.340 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:54.334+0000 7f345b4fd700 1 -- 172.21.15.47:0/527482318 wait complete. 2022-09-07T21:37:54.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:54 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/527482318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:37:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:54 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/527482318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:37:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:54 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/527482318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:37:54.880 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-07T21:33:29.342534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-09-07T21:37:54.881 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd pool get .mgr pg_num 2022-09-07T21:37:55.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:55 smithi047 ceph-mon[90087]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:37:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:55 smithi055 ceph-mon[97033]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:37:55.767 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:37:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:55 smithi059 ceph-mon[97450]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:37:57.548 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.545+0000 7fdf81cb7700 1 -- 172.21.15.47:0/560895018 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf7c073b20 msgr2=0x7fdf7c10e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:57.549 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.545+0000 7fdf81cb7700 1 --2- 172.21.15.47:0/560895018 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf7c073b20 0x7fdf7c10e860 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009810 tx=0x7fdf70009b20 comp rx=0 tx=0).stop 2022-09-07T21:37:57.549 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.545+0000 7fdf81cb7700 1 -- 172.21.15.47:0/560895018 shutdown_connections 2022-09-07T21:37:57.549 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.545+0000 7fdf81cb7700 1 --2- 172.21.15.47:0/560895018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7c10eda0 0x7fdf7c111180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.545+0000 7fdf81cb7700 1 --2- 172.21.15.47:0/560895018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf7c073210 0x7fdf7c0735e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.545+0000 7fdf81cb7700 1 --2- 172.21.15.47:0/560895018 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf7c073b20 0x7fdf7c10e860 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.545+0000 7fdf81cb7700 1 -- 172.21.15.47:0/560895018 >> 172.21.15.47:0/560895018 conn(0x7fdf7c0fbc20 msgr2=0x7fdf7c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:37:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.545+0000 7fdf81cb7700 1 -- 172.21.15.47:0/560895018 shutdown_connections 2022-09-07T21:37:57.551 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.545+0000 7fdf81cb7700 1 -- 172.21.15.47:0/560895018 wait complete. 2022-09-07T21:37:57.553 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.547+0000 7fdf81cb7700 1 Processor -- start 2022-09-07T21:37:57.553 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.547+0000 7fdf81cb7700 1 -- start start 2022-09-07T21:37:57.553 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.547+0000 7fdf81cb7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf7c073210 0x7fdf7c19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:57.553 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.547+0000 7fdf81cb7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7c073b20 0x7fdf7c19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:57.554 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.547+0000 7fdf81cb7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf7c10eda0 0x7fdf7c19ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:57.554 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.547+0000 7fdf81cb7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c19f310 con 0x7fdf7c073b20 2022-09-07T21:37:57.554 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.547+0000 7fdf81cb7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c19f480 con 0x7fdf7c073210 2022-09-07T21:37:57.554 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.547+0000 7fdf81cb7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c19f640 con 0x7fdf7c10eda0 2022-09-07T21:37:57.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.548+0000 7fdf7affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7c073b20 0x7fdf7c19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:57.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.548+0000 7fdf7b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf7c073210 0x7fdf7c19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:57.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.548+0000 7fdf7affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7c073b20 0x7fdf7c19a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42968/0 (socket says 172.21.15.47:42968) 2022-09-07T21:37:57.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.548+0000 7fdf7affd700 1 -- 172.21.15.47:0/558017258 learned_addr learned my addr 172.21.15.47:0/558017258 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:37:57.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.548+0000 7fdf7bfff700 1 --2- 172.21.15.47:0/558017258 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf7c10eda0 0x7fdf7c19ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:57.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.548+0000 7fdf7b7fe700 1 -- 172.21.15.47:0/558017258 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf7c10eda0 msgr2=0x7fdf7c19ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:57.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.548+0000 7fdf7b7fe700 1 --2- 172.21.15.47:0/558017258 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf7c10eda0 0x7fdf7c19ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:57.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.548+0000 7fdf7b7fe700 1 -- 172.21.15.47:0/558017258 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7c073b20 msgr2=0x7fdf7c19a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:57.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.548+0000 7fdf7b7fe700 1 --2- 172.21.15.47:0/558017258 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7c073b20 0x7fdf7c19a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:57.557 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.548+0000 7fdf7b7fe700 1 -- 172.21.15.47:0/558017258 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf70009470 con 0x7fdf7c073210 2022-09-07T21:37:57.557 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.548+0000 7fdf7affd700 1 --2- 172.21.15.47:0/558017258 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7c073b20 0x7fdf7c19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:37:57.557 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.548+0000 7fdf7b7fe700 1 --2- 172.21.15.47:0/558017258 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf7c073210 0x7fdf7c19a2e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c00e6c0 tx=0x7fdf6c00e9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:37:57.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.549+0000 7fdf78ff9700 1 -- 172.21.15.47:0/558017258 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c010900 con 0x7fdf7c073210 2022-09-07T21:37:57.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.549+0000 7fdf78ff9700 1 -- 172.21.15.47:0/558017258 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fdf6c018420 con 0x7fdf7c073210 2022-09-07T21:37:57.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.549+0000 7fdf81cb7700 1 -- 172.21.15.47:0/558017258 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7c19f920 con 0x7fdf7c073210 2022-09-07T21:37:57.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.549+0000 7fdf78ff9700 1 -- 172.21.15.47:0/558017258 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c017610 con 0x7fdf7c073210 2022-09-07T21:37:57.559 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.549+0000 7fdf81cb7700 1 -- 172.21.15.47:0/558017258 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7c1a7160 con 0x7fdf7c073210 2022-09-07T21:37:57.559 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.550+0000 7fdf81cb7700 1 -- 172.21.15.47:0/558017258 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7c064ea0 con 0x7fdf7c073210 2022-09-07T21:37:57.560 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.552+0000 7fdf78ff9700 1 -- 172.21.15.47:0/558017258 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fdf6c00c040 con 0x7fdf7c073210 2022-09-07T21:37:57.560 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.553+0000 7fdf78ff9700 1 --2- 172.21.15.47:0/558017258 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdf64099110 0x7fdf6409b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:37:57.560 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.553+0000 7fdf78ff9700 1 -- 172.21.15.47:0/558017258 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fdf6c014070 con 0x7fdf7c073210 2022-09-07T21:37:57.561 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.553+0000 7fdf7affd700 1 --2- 172.21.15.47:0/558017258 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdf64099110 0x7fdf6409b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:37:57.561 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.554+0000 7fdf7affd700 1 --2- 172.21.15.47:0/558017258 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdf64099110 0x7fdf6409b5c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c19b8f0 tx=0x7fdf70005910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:37:57.561 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.556+0000 7fdf78ff9700 1 -- 172.21.15.47:0/558017258 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fdf6c08ea50 con 0x7fdf7c073210 2022-09-07T21:37:57.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.781+0000 7fdf81cb7700 1 -- 172.21.15.47:0/558017258 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdf7c114330 con 0x7fdf7c073210 2022-09-07T21:37:57.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.781+0000 7fdf78ff9700 1 -- 172.21.15.47:0/558017258 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdf6c092070 con 0x7fdf7c073210 2022-09-07T21:37:57.785 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2022-09-07T21:37:57.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.784+0000 7fdf81cb7700 1 -- 172.21.15.47:0/558017258 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdf64099110 msgr2=0x7fdf6409b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:57.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.784+0000 7fdf81cb7700 1 --2- 172.21.15.47:0/558017258 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdf64099110 0x7fdf6409b5c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c19b8f0 tx=0x7fdf70005910 comp rx=0 tx=0).stop 2022-09-07T21:37:57.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.784+0000 7fdf81cb7700 1 -- 172.21.15.47:0/558017258 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf7c073210 msgr2=0x7fdf7c19a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:37:57.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.784+0000 7fdf81cb7700 1 --2- 172.21.15.47:0/558017258 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf7c073210 0x7fdf7c19a2e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c00e6c0 tx=0x7fdf6c00e9d0 comp rx=0 tx=0).stop 2022-09-07T21:37:57.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.784+0000 7fdf81cb7700 1 -- 172.21.15.47:0/558017258 shutdown_connections 2022-09-07T21:37:57.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.784+0000 7fdf81cb7700 1 --2- 172.21.15.47:0/558017258 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdf64099110 0x7fdf6409b5c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:57.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.784+0000 7fdf81cb7700 1 --2- 172.21.15.47:0/558017258 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf7c10eda0 0x7fdf7c19ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:57.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.784+0000 7fdf81cb7700 1 --2- 172.21.15.47:0/558017258 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf7c073210 0x7fdf7c19a2e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:57.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.784+0000 7fdf81cb7700 1 --2- 172.21.15.47:0/558017258 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7c073b20 0x7fdf7c19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:37:57.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.784+0000 7fdf81cb7700 1 -- 172.21.15.47:0/558017258 >> 172.21.15.47:0/558017258 conn(0x7fdf7c0fbc20 msgr2=0x7fdf7c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:37:57.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.784+0000 7fdf81cb7700 1 -- 172.21.15.47:0/558017258 shutdown_connections 2022-09-07T21:37:57.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:37:57.784+0000 7fdf81cb7700 1 -- 172.21.15.47:0/558017258 wait complete. 2022-09-07T21:37:57.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:57 smithi047 ceph-mon[90087]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:37:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:57 smithi055 ceph-mon[97033]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:37:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:57 smithi059 ceph-mon[97450]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:37:58.334 INFO:tasks.cephadm:Adding mds.a on smithi047 2022-09-07T21:37:58.334 INFO:tasks.cephadm:Adding mds.d on smithi047 2022-09-07T21:37:58.335 INFO:tasks.cephadm:Adding mds.g on smithi047 2022-09-07T21:37:58.336 INFO:tasks.cephadm:Adding mds.j on smithi047 2022-09-07T21:37:58.336 INFO:tasks.cephadm:Adding mds.b on smithi055 2022-09-07T21:37:58.337 INFO:tasks.cephadm:Adding mds.e on smithi055 2022-09-07T21:37:58.337 INFO:tasks.cephadm:Adding mds.h on smithi055 2022-09-07T21:37:58.338 INFO:tasks.cephadm:Adding mds.k on smithi055 2022-09-07T21:37:58.338 INFO:tasks.cephadm:Adding mds.c on smithi059 2022-09-07T21:37:58.339 INFO:tasks.cephadm:Adding mds.f on smithi059 2022-09-07T21:37:58.340 INFO:tasks.cephadm:Adding mds.i on smithi059 2022-09-07T21:37:58.340 INFO:tasks.cephadm:Adding mds.l on smithi059 2022-09-07T21:37:58.341 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph orch apply mds all '12;smithi047=a;smithi047=d;smithi047=g;smithi047=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi059=c;smithi059=f;smithi059=i;smithi059=l' 2022-09-07T21:37:58.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:58 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/558017258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:37:58.959 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.c/config 2022-09-07T21:37:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:58 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/558017258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:37:59.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:58 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/558017258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:37:59.714 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:37:59 smithi059 ceph-mon[97450]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:37:59.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:37:59 smithi047 ceph-mon[90087]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:37:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:37:59 smithi055 ceph-mon[97033]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:00.097 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.092+0000 7f5df6791700 1 -- 172.21.15.59:0/1372362268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5df00fe970 msgr2=0x7f5df0105210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:00.098 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.092+0000 7f5df6791700 1 --2- 172.21.15.59:0/1372362268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5df00fe970 0x7f5df0105210 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5de8009bb0 tx=0x7f5de8009ec0 comp rx=0 tx=0).stop 2022-09-07T21:38:00.098 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.092+0000 7f5df6791700 1 -- 172.21.15.59:0/1372362268 shutdown_connections 2022-09-07T21:38:00.098 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.092+0000 7f5df6791700 1 --2- 172.21.15.59:0/1372362268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5df00fe970 0x7f5df0105210 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:00.098 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.092+0000 7f5df6791700 1 --2- 172.21.15.59:0/1372362268 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5df00fcde0 0x7f5df00fd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:00.099 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.092+0000 7f5df6791700 1 --2- 172.21.15.59:0/1372362268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5df00fdfc0 0x7f5df00fe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:00.099 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.092+0000 7f5df6791700 1 -- 172.21.15.59:0/1372362268 >> 172.21.15.59:0/1372362268 conn(0x7f5df00f8430 msgr2=0x7f5df00fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:00.099 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.092+0000 7f5df6791700 1 -- 172.21.15.59:0/1372362268 shutdown_connections 2022-09-07T21:38:00.099 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.092+0000 7f5df6791700 1 -- 172.21.15.59:0/1372362268 wait complete. 2022-09-07T21:38:00.100 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.092+0000 7f5df6791700 1 Processor -- start 2022-09-07T21:38:00.100 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.092+0000 7f5df6791700 1 -- start start 2022-09-07T21:38:00.100 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df6791700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5df00fcde0 0x7f5df0100d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:00.100 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df6791700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5df00fdfc0 0x7f5df00ff3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:00.101 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df6791700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5df00fe970 0x7f5df00ff940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:00.101 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df6791700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df01000e0 con 0x7f5df00fcde0 2022-09-07T21:38:00.101 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df6791700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df0100250 con 0x7f5df00fdfc0 2022-09-07T21:38:00.101 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df6791700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df019db10 con 0x7f5df00fe970 2022-09-07T21:38:00.102 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df4d2e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5df00fe970 0x7f5df00ff940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:00.102 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df4d2e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5df00fe970 0x7f5df00ff940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40646/0 (socket says 172.21.15.59:40646) 2022-09-07T21:38:00.102 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df4d2e700 1 -- 172.21.15.59:0/4133565042 learned_addr learned my addr 172.21.15.59:0/4133565042 (peer_addr_for_me v2:172.21.15.59:0/0) 2022-09-07T21:38:00.102 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5def7fe700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5df00fdfc0 0x7f5df00ff3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:00.102 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5deffff700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5df00fcde0 0x7f5df0100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:00.103 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df4d2e700 1 -- 172.21.15.59:0/4133565042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5df00fdfc0 msgr2=0x7f5df00ff3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:00.103 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df4d2e700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5df00fdfc0 0x7f5df00ff3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:00.103 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df4d2e700 1 -- 172.21.15.59:0/4133565042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5df00fcde0 msgr2=0x7f5df0100d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:00.103 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df4d2e700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5df00fcde0 0x7f5df0100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:00.104 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df4d2e700 1 -- 172.21.15.59:0/4133565042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de80097e0 con 0x7f5df00fe970 2022-09-07T21:38:00.104 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5def7fe700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5df00fdfc0 0x7f5df00ff3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-07T21:38:00.104 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5df4d2e700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5df00fe970 0x7f5df00ff940 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5de80092d0 tx=0x7f5de8005ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:00.104 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.093+0000 7f5ded7fa700 1 -- 172.21.15.59:0/4133565042 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de801c070 con 0x7f5df00fe970 2022-09-07T21:38:00.104 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.094+0000 7f5df6791700 1 -- 172.21.15.59:0/4133565042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df019dd90 con 0x7f5df00fe970 2022-09-07T21:38:00.105 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.094+0000 7f5ded7fa700 1 -- 172.21.15.59:0/4133565042 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5de8003d20 con 0x7f5df00fe970 2022-09-07T21:38:00.105 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.094+0000 7f5ded7fa700 1 -- 172.21.15.59:0/4133565042 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de8005710 con 0x7f5df00fe970 2022-09-07T21:38:00.105 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.094+0000 7f5df6791700 1 -- 172.21.15.59:0/4133565042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df019e260 con 0x7f5df00fe970 2022-09-07T21:38:00.106 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.095+0000 7f5df6791700 1 -- 172.21.15.59:0/4133565042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df0108530 con 0x7f5df00fe970 2022-09-07T21:38:00.107 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.096+0000 7f5ded7fa700 1 -- 172.21.15.59:0/4133565042 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f5de8027020 con 0x7f5df00fe970 2022-09-07T21:38:00.107 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.096+0000 7f5ded7fa700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f5dd8099010 0x7f5dd809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:00.107 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.096+0000 7f5ded7fa700 1 -- 172.21.15.59:0/4133565042 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f5de80c3660 con 0x7f5df00fe970 2022-09-07T21:38:00.108 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.097+0000 7f5deffff700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f5dd8099010 0x7f5dd809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:00.108 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.097+0000 7f5deffff700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f5dd8099010 0x7f5dd809b4c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5de000a730 tx=0x7f5de0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:00.108 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.099+0000 7f5ded7fa700 1 -- 172.21.15.59:0/4133565042 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f5de8090240 con 0x7f5df00fe970 2022-09-07T21:38:00.305 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.302+0000 7f5df6791700 1 -- 172.21.15.59:0/4133565042 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi047=a;smithi047=d;smithi047=g;smithi047=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi059=c;smithi059=f;smithi059=i;smithi059=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f5df0066e50 con 0x7f5dd8099010 2022-09-07T21:38:00.314 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.311+0000 7f5ded7fa700 1 -- 172.21.15.59:0/4133565042 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f5df0066e50 con 0x7f5dd8099010 2022-09-07T21:38:00.315 INFO:teuthology.orchestra.run.smithi059.stdout:Scheduled mds.all update... 2022-09-07T21:38:00.317 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.313+0000 7f5df6791700 1 -- 172.21.15.59:0/4133565042 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f5dd8099010 msgr2=0x7f5dd809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:00.317 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.313+0000 7f5df6791700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f5dd8099010 0x7f5dd809b4c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5de000a730 tx=0x7f5de0008040 comp rx=0 tx=0).stop 2022-09-07T21:38:00.318 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.313+0000 7f5df6791700 1 -- 172.21.15.59:0/4133565042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5df00fe970 msgr2=0x7f5df00ff940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:00.318 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.313+0000 7f5df6791700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5df00fe970 0x7f5df00ff940 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5de80092d0 tx=0x7f5de8005ae0 comp rx=0 tx=0).stop 2022-09-07T21:38:00.319 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.313+0000 7f5df6791700 1 -- 172.21.15.59:0/4133565042 shutdown_connections 2022-09-07T21:38:00.319 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.313+0000 7f5df6791700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f5dd8099010 0x7f5dd809b4c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:00.320 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.313+0000 7f5df6791700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5df00fe970 0x7f5df00ff940 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:00.320 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.313+0000 7f5df6791700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5df00fcde0 0x7f5df0100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:00.320 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.313+0000 7f5df6791700 1 --2- 172.21.15.59:0/4133565042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5df00fdfc0 0x7f5df00ff3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:00.321 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.313+0000 7f5df6791700 1 -- 172.21.15.59:0/4133565042 >> 172.21.15.59:0/4133565042 conn(0x7f5df00f8430 msgr2=0x7f5df0066740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:00.321 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.313+0000 7f5df6791700 1 -- 172.21.15.59:0/4133565042 shutdown_connections 2022-09-07T21:38:00.322 INFO:teuthology.orchestra.run.smithi059.stderr:2022-09-07T21:38:00.313+0000 7f5df6791700 1 -- 172.21.15.59:0/4133565042 wait complete. 2022-09-07T21:38:00.675 DEBUG:teuthology.orchestra.run.smithi047:mds.a> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mds.a.service 2022-09-07T21:38:00.680 DEBUG:teuthology.orchestra.run.smithi047:mds.d> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mds.d.service 2022-09-07T21:38:00.727 DEBUG:teuthology.orchestra.run.smithi047:mds.g> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mds.g.service 2022-09-07T21:38:00.774 DEBUG:teuthology.orchestra.run.smithi047:mds.j> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mds.j.service 2022-09-07T21:38:00.775 INFO:journalctl@ceph.mds.a.smithi047.stdout:-- Logs begin at Wed 2022-09-07 21:11:20 UTC. -- 2022-09-07T21:38:00.822 DEBUG:teuthology.orchestra.run.smithi055:mds.b> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mds.b.service 2022-09-07T21:38:00.824 INFO:journalctl@ceph.mds.d.smithi047.stdout:-- Logs begin at Wed 2022-09-07 21:11:20 UTC. -- 2022-09-07T21:38:00.826 DEBUG:teuthology.orchestra.run.smithi055:mds.e> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mds.e.service 2022-09-07T21:38:00.854 INFO:journalctl@ceph.mds.b.smithi055.stdout:-- Logs begin at Wed 2022-09-07 21:07:35 UTC. -- 2022-09-07T21:38:00.856 DEBUG:teuthology.orchestra.run.smithi055:mds.h> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mds.h.service 2022-09-07T21:38:00.868 INFO:journalctl@ceph.mds.g.smithi047.stdout:-- Logs begin at Wed 2022-09-07 21:11:20 UTC. -- 2022-09-07T21:38:00.882 INFO:journalctl@ceph.mds.e.smithi055.stdout:-- Logs begin at Wed 2022-09-07 21:07:35 UTC. -- 2022-09-07T21:38:00.883 DEBUG:teuthology.orchestra.run.smithi055:mds.k> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mds.k.service 2022-09-07T21:38:00.909 INFO:journalctl@ceph.mds.h.smithi055.stdout:-- Logs begin at Wed 2022-09-07 21:07:35 UTC. -- 2022-09-07T21:38:00.910 DEBUG:teuthology.orchestra.run.smithi059:mds.c> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mds.c.service 2022-09-07T21:38:00.913 DEBUG:teuthology.orchestra.run.smithi059:mds.f> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mds.f.service 2022-09-07T21:38:00.914 INFO:journalctl@ceph.mds.j.smithi047.stdout:-- Logs begin at Wed 2022-09-07 21:11:20 UTC. -- 2022-09-07T21:38:00.936 INFO:journalctl@ceph.mds.k.smithi055.stdout:-- Logs begin at Wed 2022-09-07 21:07:35 UTC. -- 2022-09-07T21:38:00.940 INFO:journalctl@ceph.mds.c.smithi059.stdout:-- Logs begin at Wed 2022-09-07 21:07:37 UTC. -- 2022-09-07T21:38:00.942 DEBUG:teuthology.orchestra.run.smithi059:mds.i> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mds.i.service 2022-09-07T21:38:00.968 INFO:journalctl@ceph.mds.f.smithi059.stdout:-- Logs begin at Wed 2022-09-07 21:07:37 UTC. -- 2022-09-07T21:38:00.970 DEBUG:teuthology.orchestra.run.smithi059:mds.l> sudo journalctl -f -n 0 -u ceph-01b8396e-2ef4-11ed-8431-001a4aab830c@mds.l.service 2022-09-07T21:38:00.996 INFO:journalctl@ceph.mds.i.smithi059.stdout:-- Logs begin at Wed 2022-09-07 21:07:37 UTC. -- 2022-09-07T21:38:00.997 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2022-09-07T21:38:00.998 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-07T21:38:01.022 INFO:journalctl@ceph.mds.l.smithi059.stdout:-- Logs begin at Wed 2022-09-07 21:07:37 UTC. -- 2022-09-07T21:38:01.294 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.289+0000 7fc2941ff700 1 -- 172.21.15.47:0/2833953599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c100df0 msgr2=0x7fc28c103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:01.295 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.289+0000 7fc2941ff700 1 --2- 172.21.15.47:0/2833953599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c100df0 0x7fc28c103250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc27c009810 tx=0x7fc27c009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:01.295 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.289+0000 7fc2941ff700 1 -- 172.21.15.47:0/2833953599 shutdown_connections 2022-09-07T21:38:01.295 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.289+0000 7fc2941ff700 1 --2- 172.21.15.47:0/2833953599 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc28c1038e0 0x7fc28c105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.289+0000 7fc2941ff700 1 --2- 172.21.15.47:0/2833953599 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc28c066be0 0x7fc28c1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.289+0000 7fc2941ff700 1 --2- 172.21.15.47:0/2833953599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c100df0 0x7fc28c103250 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.289+0000 7fc2941ff700 1 -- 172.21.15.47:0/2833953599 >> 172.21.15.47:0/2833953599 conn(0x7fc28c0f82e0 msgr2=0x7fc28c0fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:01.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.289+0000 7fc2941ff700 1 -- 172.21.15.47:0/2833953599 shutdown_connections 2022-09-07T21:38:01.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.289+0000 7fc2941ff700 1 -- 172.21.15.47:0/2833953599 wait complete. 2022-09-07T21:38:01.297 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.290+0000 7fc2941ff700 1 Processor -- start 2022-09-07T21:38:01.297 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.291+0000 7fc2941ff700 1 -- start start 2022-09-07T21:38:01.297 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.291+0000 7fc2941ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc28c066be0 0x7fc28c195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:01.297 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.291+0000 7fc2941ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc28c100df0 0x7fc28c1960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:01.298 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.291+0000 7fc2941ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c1038e0 0x7fc28c19d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:01.298 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.291+0000 7fc2941ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c19d6f0 con 0x7fc28c066be0 2022-09-07T21:38:01.298 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.291+0000 7fc2941ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c19d860 con 0x7fc28c100df0 2022-09-07T21:38:01.299 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.291+0000 7fc2941ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c19da20 con 0x7fc28c1038e0 2022-09-07T21:38:01.299 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.291+0000 7fc291f9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc28c066be0 0x7fc28c195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:01.300 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.291+0000 7fc29279c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c1038e0 0x7fc28c19d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:01.300 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.291+0000 7fc291f9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc28c066be0 0x7fc28c195b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50428/0 (socket says 172.21.15.47:50428) 2022-09-07T21:38:01.301 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.291+0000 7fc291f9b700 1 -- 172.21.15.47:0/2781426625 learned_addr learned my addr 172.21.15.47:0/2781426625 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:01.301 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.291+0000 7fc29179a700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc28c100df0 0x7fc28c1960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:01.302 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.292+0000 7fc291f9b700 1 -- 172.21.15.47:0/2781426625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c1038e0 msgr2=0x7fc28c19d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:01.302 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.292+0000 7fc291f9b700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c1038e0 0x7fc28c19d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.303 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.292+0000 7fc291f9b700 1 -- 172.21.15.47:0/2781426625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc28c100df0 msgr2=0x7fc28c1960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:01.303 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.292+0000 7fc291f9b700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc28c100df0 0x7fc28c1960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.303 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.292+0000 7fc291f9b700 1 -- 172.21.15.47:0/2781426625 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc284009730 con 0x7fc28c066be0 2022-09-07T21:38:01.304 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.292+0000 7fc29279c700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c1038e0 0x7fc28c19d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:01.304 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.292+0000 7fc29179a700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc28c100df0 0x7fc28c1960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:01.305 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.292+0000 7fc291f9b700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc28c066be0 0x7fc28c195b70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc28800e6e0 tx=0x7fc28800e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:01.308 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.292+0000 7fc282ffd700 1 -- 172.21.15.47:0/2781426625 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288009500 con 0x7fc28c066be0 2022-09-07T21:38:01.309 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.292+0000 7fc2941ff700 1 -- 172.21.15.47:0/2781426625 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc27c009470 con 0x7fc28c066be0 2022-09-07T21:38:01.309 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.292+0000 7fc282ffd700 1 -- 172.21.15.47:0/2781426625 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fc288010ce0 con 0x7fc28c066be0 2022-09-07T21:38:01.309 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.293+0000 7fc282ffd700 1 -- 172.21.15.47:0/2781426625 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28800f420 con 0x7fc28c066be0 2022-09-07T21:38:01.310 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.293+0000 7fc2941ff700 1 -- 172.21.15.47:0/2781426625 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c19e0d0 con 0x7fc28c066be0 2022-09-07T21:38:01.310 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.294+0000 7fc2941ff700 1 -- 172.21.15.47:0/2781426625 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c1094f0 con 0x7fc28c066be0 2022-09-07T21:38:01.311 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.296+0000 7fc282ffd700 1 -- 172.21.15.47:0/2781426625 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fc28800c040 con 0x7fc28c066be0 2022-09-07T21:38:01.311 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.297+0000 7fc282ffd700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc278099080 0x7fc27809b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:01.311 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.297+0000 7fc282ffd700 1 -- 172.21.15.47:0/2781426625 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fc288014070 con 0x7fc28c066be0 2022-09-07T21:38:01.312 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.297+0000 7fc29179a700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc278099080 0x7fc27809b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:01.312 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.298+0000 7fc29179a700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc278099080 0x7fc27809b530 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc27c00b4f0 tx=0x7fc27c009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:01.312 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.301+0000 7fc282ffd700 1 -- 172.21.15.47:0/2781426625 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fc28804fc80 con 0x7fc28c066be0 2022-09-07T21:38:01.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:01 smithi047 ceph-mon[90087]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi047=a;smithi047=d;smithi047=g;smithi047=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi059=c;smithi059=f;smithi059=i;smithi059=l", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:38:01.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:01 smithi047 ceph-mon[90087]: Saving service mds.all spec with placement smithi047=a;smithi047=d;smithi047=g;smithi047=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi059=c;smithi059=f;smithi059=i;smithi059=l;count:12 2022-09-07T21:38:01.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:01 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:01.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:01 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:38:01.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:01 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:01.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:01 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:38:01.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:01 smithi047 ceph-mon[90087]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:01.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.525+0000 7fc2941ff700 1 -- 172.21.15.47:0/2781426625 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc28c05fa30 con 0x7fc28c066be0 2022-09-07T21:38:01.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.527+0000 7fc282ffd700 1 -- 172.21.15.47:0/2781426625 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18053 (secure 0 0 0) 0x7fc288092530 con 0x7fc28c066be0 2022-09-07T21:38:01.530 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:01.530 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":81,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:37:52.015572+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:38:01.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.529+0000 7fc2941ff700 1 -- 172.21.15.47:0/2781426625 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc278099080 msgr2=0x7fc27809b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:01.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.529+0000 7fc2941ff700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc278099080 0x7fc27809b530 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc27c00b4f0 tx=0x7fc27c009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.529+0000 7fc2941ff700 1 -- 172.21.15.47:0/2781426625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc28c066be0 msgr2=0x7fc28c195b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:01.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.529+0000 7fc2941ff700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc28c066be0 0x7fc28c195b70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc28800e6e0 tx=0x7fc28800e9f0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.529+0000 7fc2941ff700 1 -- 172.21.15.47:0/2781426625 shutdown_connections 2022-09-07T21:38:01.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.529+0000 7fc2941ff700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc278099080 0x7fc27809b530 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.529+0000 7fc2941ff700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c1038e0 0x7fc28c19d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.534 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.529+0000 7fc2941ff700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc28c066be0 0x7fc28c195b70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.534 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.529+0000 7fc2941ff700 1 --2- 172.21.15.47:0/2781426625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc28c100df0 0x7fc28c1960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.534 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.529+0000 7fc2941ff700 1 -- 172.21.15.47:0/2781426625 >> 172.21.15.47:0/2781426625 conn(0x7fc28c0f82e0 msgr2=0x7fc28c0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:01.534 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.529+0000 7fc2941ff700 1 -- 172.21.15.47:0/2781426625 shutdown_connections 2022-09-07T21:38:01.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.529+0000 7fc2941ff700 1 -- 172.21.15.47:0/2781426625 wait complete. 2022-09-07T21:38:01.560 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-07T21:33:29.342534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-09-07T21:38:01.562 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-09-07T21:38:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:01 smithi055 ceph-mon[97033]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi047=a;smithi047=d;smithi047=g;smithi047=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi059=c;smithi059=f;smithi059=i;smithi059=l", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:38:01.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:01 smithi055 ceph-mon[97033]: Saving service mds.all spec with placement smithi047=a;smithi047=d;smithi047=g;smithi047=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi059=c;smithi059=f;smithi059=i;smithi059=l;count:12 2022-09-07T21:38:01.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:01 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:01.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:01 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:38:01.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:01 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:01.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:01 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:38:01.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:01 smithi055 ceph-mon[97033]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:01 smithi059 ceph-mon[97450]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi047=a;smithi047=d;smithi047=g;smithi047=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi059=c;smithi059=f;smithi059=i;smithi059=l", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:38:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:01 smithi059 ceph-mon[97450]: Saving service mds.all spec with placement smithi047=a;smithi047=d;smithi047=g;smithi047=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi059=c;smithi059=f;smithi059=i;smithi059=l;count:12 2022-09-07T21:38:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:01 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:01.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:01 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:38:01.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:01 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:01.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:01 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:38:01.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:01 smithi059 ceph-mon[97450]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.795+0000 7f49833ff700 1 -- 172.21.15.47:0/1321931368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f497c0fef90 msgr2=0x7f497c0ff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.795+0000 7f49833ff700 1 --2- 172.21.15.47:0/1321931368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f497c0fef90 0x7f497c0ff400 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f496c009810 tx=0x7f496c009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:01.800 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.796+0000 7f49833ff700 1 -- 172.21.15.47:0/1321931368 shutdown_connections 2022-09-07T21:38:01.800 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.796+0000 7f49833ff700 1 --2- 172.21.15.47:0/1321931368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f497c0ff940 0x7f497c106270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.800 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.796+0000 7f49833ff700 1 --2- 172.21.15.47:0/1321931368 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c0fddb0 0x7f497c0fe1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.800 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.796+0000 7f49833ff700 1 --2- 172.21.15.47:0/1321931368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f497c0fef90 0x7f497c0ff400 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.801 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.796+0000 7f49833ff700 1 -- 172.21.15.47:0/1321931368 >> 172.21.15.47:0/1321931368 conn(0x7f497c0f9460 msgr2=0x7f497c0fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:01.801 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.796+0000 7f49833ff700 1 -- 172.21.15.47:0/1321931368 shutdown_connections 2022-09-07T21:38:01.801 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.796+0000 7f49833ff700 1 -- 172.21.15.47:0/1321931368 wait complete. 2022-09-07T21:38:01.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.797+0000 7f49833ff700 1 Processor -- start 2022-09-07T21:38:01.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.797+0000 7f49833ff700 1 -- start start 2022-09-07T21:38:01.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f49833ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f497c0fddb0 0x7f497c195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:01.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f49833ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f497c0fef90 0x7f497c1960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:01.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f49833ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c0ff940 0x7f497c19d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:01.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f49833ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c19d6f0 con 0x7f497c0fddb0 2022-09-07T21:38:01.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f49833ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c19d860 con 0x7f497c0ff940 2022-09-07T21:38:01.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f49833ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c19da20 con 0x7f497c0fef90 2022-09-07T21:38:01.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f498099a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f497c0fef90 0x7f497c1960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:01.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f498119b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f497c0fddb0 0x7f497c195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:01.808 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f498199c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c0ff940 0x7f497c19d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:01.808 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f498199c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c0ff940 0x7f497c19d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:45188/0 (socket says 172.21.15.47:45188) 2022-09-07T21:38:01.808 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f498099a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f497c0fef90 0x7f497c1960b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60648/0 (socket says 172.21.15.47:60648) 2022-09-07T21:38:01.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f498119b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f497c0fddb0 0x7f497c195b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50446/0 (socket says 172.21.15.47:50446) 2022-09-07T21:38:01.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f498199c700 1 -- 172.21.15.47:0/1414948042 learned_addr learned my addr 172.21.15.47:0/1414948042 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:01.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f498099a700 1 -- 172.21.15.47:0/1414948042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c0ff940 msgr2=0x7f497c19d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:01.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f498099a700 1 --2- 172.21.15.47:0/1414948042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c0ff940 0x7f497c19d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f498099a700 1 -- 172.21.15.47:0/1414948042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f497c0fddb0 msgr2=0x7f497c195b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:01.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f498099a700 1 --2- 172.21.15.47:0/1414948042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f497c0fddb0 0x7f497c195b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:01.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f498099a700 1 -- 172.21.15.47:0/1414948042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c009470 con 0x7f497c0fef90 2022-09-07T21:38:01.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f498199c700 1 --2- 172.21.15.47:0/1414948042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c0ff940 0x7f497c19d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:01.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.798+0000 7f498119b700 1 --2- 172.21.15.47:0/1414948042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f497c0fddb0 0x7f497c195b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:01.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.799+0000 7f498099a700 1 --2- 172.21.15.47:0/1414948042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f497c0fef90 0x7f497c1960b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f496c006010 tx=0x7f496c018630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:01.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.799+0000 7f49727fc700 1 -- 172.21.15.47:0/1414948042 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c01d070 con 0x7f497c0fef90 2022-09-07T21:38:01.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.799+0000 7f49833ff700 1 -- 172.21.15.47:0/1414948042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497c19dca0 con 0x7f497c0fef90 2022-09-07T21:38:01.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.799+0000 7f49727fc700 1 -- 172.21.15.47:0/1414948042 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f496c018820 con 0x7f497c0fef90 2022-09-07T21:38:01.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.799+0000 7f49727fc700 1 -- 172.21.15.47:0/1414948042 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c004760 con 0x7f497c0fef90 2022-09-07T21:38:01.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.799+0000 7f49833ff700 1 -- 172.21.15.47:0/1414948042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f497c19e1a0 con 0x7f497c0fef90 2022-09-07T21:38:01.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.800+0000 7f49833ff700 1 -- 172.21.15.47:0/1414948042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497c109560 con 0x7f497c0fef90 2022-09-07T21:38:01.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.802+0000 7f49727fc700 1 -- 172.21.15.47:0/1414948042 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f496c029030 con 0x7f497c0fef90 2022-09-07T21:38:01.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.803+0000 7f49727fc700 1 --2- 172.21.15.47:0/1414948042 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4968099010 0x7f496809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:01.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.804+0000 7f49727fc700 1 -- 172.21.15.47:0/1414948042 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f496c0c2e60 con 0x7f497c0fef90 2022-09-07T21:38:01.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.804+0000 7f498119b700 1 --2- 172.21.15.47:0/1414948042 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4968099010 0x7f496809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:01.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.804+0000 7f498119b700 1 --2- 172.21.15.47:0/1414948042 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4968099010 0x7f496809b4c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4978005fd0 tx=0x7f4978005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:01.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:01.806+0000 7f49727fc700 1 -- 172.21.15.47:0/1414948042 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f496c08f950 con 0x7f497c0fef90 2022-09-07T21:38:02.007 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.005+0000 7f49833ff700 1 -- 172.21.15.47:0/1414948042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f497c19e750 con 0x7f497c0fef90 2022-09-07T21:38:02.008 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.006+0000 7f49727fc700 1 -- 172.21.15.47:0/1414948042 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f496c0270a0 con 0x7f497c0fef90 2022-09-07T21:38:02.009 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2022-09-07T21:38:02.011 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.008+0000 7f49833ff700 1 -- 172.21.15.47:0/1414948042 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4968099010 msgr2=0x7f496809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:02.011 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.008+0000 7f49833ff700 1 --2- 172.21.15.47:0/1414948042 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4968099010 0x7f496809b4c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4978005fd0 tx=0x7f4978005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.011 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.008+0000 7f49833ff700 1 -- 172.21.15.47:0/1414948042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f497c0fef90 msgr2=0x7f497c1960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:02.012 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.008+0000 7f49833ff700 1 --2- 172.21.15.47:0/1414948042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f497c0fef90 0x7f497c1960b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f496c006010 tx=0x7f496c018630 comp rx=0 tx=0).stop 2022-09-07T21:38:02.012 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.009+0000 7f49833ff700 1 -- 172.21.15.47:0/1414948042 shutdown_connections 2022-09-07T21:38:02.012 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.009+0000 7f49833ff700 1 --2- 172.21.15.47:0/1414948042 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4968099010 0x7f496809b4c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.012 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.009+0000 7f49833ff700 1 --2- 172.21.15.47:0/1414948042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f497c0ff940 0x7f497c19d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.013 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.009+0000 7f49833ff700 1 --2- 172.21.15.47:0/1414948042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f497c0fddb0 0x7f497c195b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.013 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.009+0000 7f49833ff700 1 --2- 172.21.15.47:0/1414948042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f497c0fef90 0x7f497c1960b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.013 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.009+0000 7f49833ff700 1 -- 172.21.15.47:0/1414948042 >> 172.21.15.47:0/1414948042 conn(0x7f497c0f9460 msgr2=0x7f497c0fb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:02.013 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.009+0000 7f49833ff700 1 -- 172.21.15.47:0/1414948042 shutdown_connections 2022-09-07T21:38:02.013 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.009+0000 7f49833ff700 1 -- 172.21.15.47:0/1414948042 wait complete. 2022-09-07T21:38:02.041 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:02.273 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.271+0000 7f40723ff700 1 -- 172.21.15.47:0/1628623463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c100df0 msgr2=0x7f406c103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.271+0000 7f40723ff700 1 --2- 172.21.15.47:0/1628623463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c100df0 0x7f406c103250 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4060009810 tx=0x7f4060009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.271+0000 7f40723ff700 1 -- 172.21.15.47:0/1628623463 shutdown_connections 2022-09-07T21:38:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.271+0000 7f40723ff700 1 --2- 172.21.15.47:0/1628623463 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f406c1038e0 0x7f406c105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.271+0000 7f40723ff700 1 --2- 172.21.15.47:0/1628623463 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f406c066be0 0x7f406c1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.271+0000 7f40723ff700 1 --2- 172.21.15.47:0/1628623463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c100df0 0x7f406c103250 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.271+0000 7f40723ff700 1 -- 172.21.15.47:0/1628623463 >> 172.21.15.47:0/1628623463 conn(0x7f406c0f82e0 msgr2=0x7f406c0fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.271+0000 7f40723ff700 1 -- 172.21.15.47:0/1628623463 shutdown_connections 2022-09-07T21:38:02.276 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.271+0000 7f40723ff700 1 -- 172.21.15.47:0/1628623463 wait complete. 2022-09-07T21:38:02.278 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.272+0000 7f40723ff700 1 Processor -- start 2022-09-07T21:38:02.278 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.272+0000 7f40723ff700 1 -- start start 2022-09-07T21:38:02.278 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f40723ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f406c066be0 0x7f406c195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:02.279 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f40723ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f406c100df0 0x7f406c196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:02.279 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f40723ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c1038e0 0x7f406c19d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:02.279 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f40723ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c19d6c0 con 0x7f406c066be0 2022-09-07T21:38:02.279 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f40723ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c19d830 con 0x7f406c100df0 2022-09-07T21:38:02.280 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f40723ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c19d9f0 con 0x7f406c1038e0 2022-09-07T21:38:02.280 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f406bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f406c066be0 0x7f406c195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:02.280 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f407099c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c1038e0 0x7f406c19d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:02.280 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f406bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f406c066be0 0x7f406c195b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50472/0 (socket says 172.21.15.47:50472) 2022-09-07T21:38:02.280 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f406bfff700 1 -- 172.21.15.47:0/4114645825 learned_addr learned my addr 172.21.15.47:0/4114645825 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:02.281 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f406b7fe700 1 --2- 172.21.15.47:0/4114645825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f406c100df0 0x7f406c196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:02.281 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f406bfff700 1 -- 172.21.15.47:0/4114645825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c1038e0 msgr2=0x7f406c19d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:02.281 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f406bfff700 1 --2- 172.21.15.47:0/4114645825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c1038e0 0x7f406c19d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.281 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f406bfff700 1 -- 172.21.15.47:0/4114645825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f406c100df0 msgr2=0x7f406c196080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f406bfff700 1 --2- 172.21.15.47:0/4114645825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f406c100df0 0x7f406c196080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.273+0000 7f406bfff700 1 -- 172.21.15.47:0/4114645825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4060009470 con 0x7f406c066be0 2022-09-07T21:38:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.274+0000 7f406bfff700 1 --2- 172.21.15.47:0/4114645825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f406c066be0 0x7f406c195b40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f405c00e6e0 tx=0x7f405c00e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.274+0000 7f40697fa700 1 -- 172.21.15.47:0/4114645825 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c009500 con 0x7f406c066be0 2022-09-07T21:38:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.274+0000 7f40723ff700 1 -- 172.21.15.47:0/4114645825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f406c19dcd0 con 0x7f406c066be0 2022-09-07T21:38:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.274+0000 7f40697fa700 1 -- 172.21.15.47:0/4114645825 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f405c010da0 con 0x7f406c066be0 2022-09-07T21:38:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.274+0000 7f40697fa700 1 -- 172.21.15.47:0/4114645825 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c0173d0 con 0x7f406c066be0 2022-09-07T21:38:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.274+0000 7f40723ff700 1 -- 172.21.15.47:0/4114645825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406c19e180 con 0x7f406c066be0 2022-09-07T21:38:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.275+0000 7f40723ff700 1 -- 172.21.15.47:0/4114645825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406c1094f0 con 0x7f406c066be0 2022-09-07T21:38:02.284 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.277+0000 7f40697fa700 1 -- 172.21.15.47:0/4114645825 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f405c00c040 con 0x7f406c066be0 2022-09-07T21:38:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.279+0000 7f40697fa700 1 --2- 172.21.15.47:0/4114645825 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4054099000 0x7f405409b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.279+0000 7f40697fa700 1 -- 172.21.15.47:0/4114645825 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f405c014070 con 0x7f406c066be0 2022-09-07T21:38:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.280+0000 7f406b7fe700 1 --2- 172.21.15.47:0/4114645825 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4054099000 0x7f405409b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.280+0000 7f406b7fe700 1 --2- 172.21.15.47:0/4114645825 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4054099000 0x7f405409b4b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f406000b4f0 tx=0x7f4060009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:02.286 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.283+0000 7f40697fa700 1 -- 172.21.15.47:0/4114645825 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f405c04fc80 con 0x7f406c066be0 2022-09-07T21:38:02.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.532+0000 7f40723ff700 1 -- 172.21.15.47:0/4114645825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f406c19ea80 con 0x7f406c066be0 2022-09-07T21:38:02.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.533+0000 7f40697fa700 1 -- 172.21.15.47:0/4114645825 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f405c08ebc0 con 0x7f406c066be0 2022-09-07T21:38:02.536 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:02.537 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2022-09-07T21:38:02.538 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.535+0000 7f40723ff700 1 -- 172.21.15.47:0/4114645825 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4054099000 msgr2=0x7f405409b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:02.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.535+0000 7f40723ff700 1 --2- 172.21.15.47:0/4114645825 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4054099000 0x7f405409b4b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f406000b4f0 tx=0x7f4060009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.535+0000 7f40723ff700 1 -- 172.21.15.47:0/4114645825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f406c066be0 msgr2=0x7f406c195b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:02.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.535+0000 7f40723ff700 1 --2- 172.21.15.47:0/4114645825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f406c066be0 0x7f406c195b40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f405c00e6e0 tx=0x7f405c00e9f0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.535+0000 7f40723ff700 1 -- 172.21.15.47:0/4114645825 shutdown_connections 2022-09-07T21:38:02.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.535+0000 7f40723ff700 1 --2- 172.21.15.47:0/4114645825 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4054099000 0x7f405409b4b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.535+0000 7f40723ff700 1 --2- 172.21.15.47:0/4114645825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c1038e0 0x7f406c19d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.535+0000 7f40723ff700 1 --2- 172.21.15.47:0/4114645825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f406c066be0 0x7f406c195b40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.536+0000 7f40723ff700 1 --2- 172.21.15.47:0/4114645825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f406c100df0 0x7f406c196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:02.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.536+0000 7f40723ff700 1 -- 172.21.15.47:0/4114645825 >> 172.21.15.47:0/4114645825 conn(0x7f406c0f82e0 msgr2=0x7f406c0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:02.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.536+0000 7f40723ff700 1 -- 172.21.15.47:0/4114645825 shutdown_connections 2022-09-07T21:38:02.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:02.536+0000 7f40723ff700 1 -- 172.21.15.47:0/4114645825 wait complete. 2022-09-07T21:38:02.542 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 1 2022-09-07T21:38:02.580 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:02.581 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:02.582 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:02.582 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:02.582 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:02 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:02.583 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:02 smithi059 ceph-mon[97450]: Deploying daemon mds.l on smithi059 2022-09-07T21:38:02.583 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:02 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2781426625' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:02.584 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:02 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1414948042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:38:02.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:02.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:02.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:02.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:02.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:02 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:02.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:02 smithi047 ceph-mon[90087]: Deploying daemon mds.l on smithi059 2022-09-07T21:38:02.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:02 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2781426625' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:02.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:02 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1414948042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:38:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:02.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:02.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:02.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:02 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:02.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:02 smithi055 ceph-mon[97033]: Deploying daemon mds.l on smithi059 2022-09-07T21:38:02.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:02 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2781426625' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:02.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:02 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1414948042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:38:02.871 INFO:journalctl@ceph.mds.l.smithi059.stdout:Sep 07 21:38:02 smithi059 systemd[1]: Starting Ceph mds.l for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:38:03.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:03 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/4114645825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:03.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:03 smithi059 ceph-mon[97450]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:03.395 INFO:journalctl@ceph.mds.l.smithi059.stdout:Sep 07 21:38:03 smithi059 podman[124005]: 2022-09-07T21:38:03.395 INFO:journalctl@ceph.mds.l.smithi059.stdout:Sep 07 21:38:03 smithi059 bash[124005]: 33afe7eba40a0e24b59b4abdc95e5adab30153872d85231a5cfbd08325fd18d6 2022-09-07T21:38:03.396 INFO:journalctl@ceph.mds.l.smithi059.stdout:Sep 07 21:38:03 smithi059 ceph-mds[124045]: -- 172.21.15.59:0/1821694015 <== mon.1 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5608b0064820 con 0x5608b0038c00 2022-09-07T21:38:03.396 INFO:journalctl@ceph.mds.l.smithi059.stdout:Sep 07 21:38:03 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-l[124021]: starting mds.l at 2022-09-07T21:38:03.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:03 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/4114645825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:03.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:03 smithi047 ceph-mon[90087]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:03 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/4114645825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:03 smithi055 ceph-mon[97033]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:03.781 INFO:journalctl@ceph.mds.l.smithi059.stdout:Sep 07 21:38:03 smithi059 systemd[1]: Started Ceph mds.l for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:38:04.566 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:04.721 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:04 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.59:6832/1360705881,v1:172.21.15.59:6833/1360705881] up:boot 2022-09-07T21:38:04.722 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:04 smithi055 ceph-mon[97033]: fsmap 1 up:standby 2022-09-07T21:38:04.723 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-09-07T21:38:04.723 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:04.723 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:04.724 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:04.724 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:04.725 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:04 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:04.725 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:04 smithi055 ceph-mon[97033]: Deploying daemon mds.k on smithi055 2022-09-07T21:38:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:04 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.59:6832/1360705881,v1:172.21.15.59:6833/1360705881] up:boot 2022-09-07T21:38:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:04 smithi059 ceph-mon[97450]: fsmap 1 up:standby 2022-09-07T21:38:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-09-07T21:38:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:04 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:04.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:04 smithi059 ceph-mon[97450]: Deploying daemon mds.k on smithi055 2022-09-07T21:38:04.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.804+0000 7f8db7dff700 1 -- 172.21.15.47:0/3433475360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8db00fef90 msgr2=0x7f8db00ff3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:04.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.804+0000 7f8db7dff700 1 --2- 172.21.15.47:0/3433475360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8db00fef90 0x7f8db00ff3e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009810 tx=0x7f8da0009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:04.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.804+0000 7f8db7dff700 1 -- 172.21.15.47:0/3433475360 shutdown_connections 2022-09-07T21:38:04.808 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.804+0000 7f8db7dff700 1 --2- 172.21.15.47:0/3433475360 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00ff920 0x7f8db01061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:04.808 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.804+0000 7f8db7dff700 1 --2- 172.21.15.47:0/3433475360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8db00fdd10 0x7f8db00fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:04.808 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.804+0000 7f8db7dff700 1 --2- 172.21.15.47:0/3433475360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8db00fef90 0x7f8db00ff3e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:04.808 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.804+0000 7f8db7dff700 1 -- 172.21.15.47:0/3433475360 >> 172.21.15.47:0/3433475360 conn(0x7f8db00f93c0 msgr2=0x7f8db00fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:04.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.804+0000 7f8db7dff700 1 -- 172.21.15.47:0/3433475360 shutdown_connections 2022-09-07T21:38:04.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.804+0000 7f8db7dff700 1 -- 172.21.15.47:0/3433475360 wait complete. 2022-09-07T21:38:04.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.806+0000 7f8db7dff700 1 Processor -- start 2022-09-07T21:38:04.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.806+0000 7f8db7dff700 1 -- start start 2022-09-07T21:38:04.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.806+0000 7f8db7dff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8db00fdd10 0x7f8db0072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:04.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.806+0000 7f8db7dff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8db00fef90 0x7f8db00710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:04.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.806+0000 7f8db7dff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00ff920 0x7f8db0071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:04.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.806+0000 7f8db7dff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db0071e10 con 0x7f8db00fdd10 2022-09-07T21:38:04.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.806+0000 7f8db7dff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db0071f80 con 0x7f8db00ff920 2022-09-07T21:38:04.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.806+0000 7f8db7dff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db019da50 con 0x7f8db00fef90 2022-09-07T21:38:04.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.806+0000 7f8db5b9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8db00fdd10 0x7f8db0072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:04.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.807+0000 7f8db5b9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8db00fdd10 0x7f8db0072a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50490/0 (socket says 172.21.15.47:50490) 2022-09-07T21:38:04.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.807+0000 7f8db5b9b700 1 -- 172.21.15.47:0/3748813642 learned_addr learned my addr 172.21.15.47:0/3748813642 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:04.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.807+0000 7f8db539a700 1 --2- 172.21.15.47:0/3748813642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8db00fef90 0x7f8db00710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:04.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.807+0000 7f8db639c700 1 --2- 172.21.15.47:0/3748813642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00ff920 0x7f8db0071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:04.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.807+0000 7f8db5b9b700 1 -- 172.21.15.47:0/3748813642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8db00fef90 msgr2=0x7f8db00710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:04.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.807+0000 7f8db5b9b700 1 --2- 172.21.15.47:0/3748813642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8db00fef90 0x7f8db00710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:04.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.807+0000 7f8db5b9b700 1 -- 172.21.15.47:0/3748813642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00ff920 msgr2=0x7f8db0071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:04.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.807+0000 7f8db5b9b700 1 --2- 172.21.15.47:0/3748813642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00ff920 0x7f8db0071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:04.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.807+0000 7f8db5b9b700 1 -- 172.21.15.47:0/3748813642 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da0009470 con 0x7f8db00fdd10 2022-09-07T21:38:04.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.807+0000 7f8db5b9b700 1 --2- 172.21.15.47:0/3748813642 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8db00fdd10 0x7f8db0072a90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8dac00e710 tx=0x7f8dac00ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:04.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.807+0000 7f8da6ffd700 1 -- 172.21.15.47:0/3748813642 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac004220 con 0x7f8db00fdd10 2022-09-07T21:38:04.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.808+0000 7f8db7dff700 1 -- 172.21.15.47:0/3748813642 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db019dd30 con 0x7f8db00fdd10 2022-09-07T21:38:04.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.808+0000 7f8da6ffd700 1 -- 172.21.15.47:0/3748813642 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8dac004380 con 0x7f8db00fdd10 2022-09-07T21:38:04.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.808+0000 7f8da6ffd700 1 -- 172.21.15.47:0/3748813642 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac003cf0 con 0x7f8db00fdd10 2022-09-07T21:38:04.818 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.808+0000 7f8db7dff700 1 -- 172.21.15.47:0/3748813642 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db019e1d0 con 0x7f8db00fdd10 2022-09-07T21:38:04.818 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.809+0000 7f8db7dff700 1 -- 172.21.15.47:0/3748813642 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db0109530 con 0x7f8db00fdd10 2022-09-07T21:38:04.819 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.810+0000 7f8da6ffd700 1 -- 172.21.15.47:0/3748813642 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f8dac01e020 con 0x7f8db00fdd10 2022-09-07T21:38:04.819 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.812+0000 7f8da6ffd700 1 --2- 172.21.15.47:0/3748813642 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8d9c099080 0x7f8d9c09b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:04.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.812+0000 7f8da6ffd700 1 -- 172.21.15.47:0/3748813642 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f8dac014070 con 0x7f8db00fdd10 2022-09-07T21:38:04.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.812+0000 7f8db539a700 1 --2- 172.21.15.47:0/3748813642 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8d9c099080 0x7f8d9c09b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:04.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.813+0000 7f8db539a700 1 --2- 172.21.15.47:0/3748813642 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8d9c099080 0x7f8d9c09b530 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8da000b4f0 tx=0x7f8da0009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:04.821 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:04.816+0000 7f8da6ffd700 1 -- 172.21.15.47:0/3748813642 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f8dac08ebd0 con 0x7f8db00fdd10 2022-09-07T21:38:04.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:04 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.59:6832/1360705881,v1:172.21.15.59:6833/1360705881] up:boot 2022-09-07T21:38:04.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:04 smithi047 ceph-mon[90087]: fsmap 1 up:standby 2022-09-07T21:38:04.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-09-07T21:38:04.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:04.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:04.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:04.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:04.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:04 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:04.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:04 smithi047 ceph-mon[90087]: Deploying daemon mds.k on smithi055 2022-09-07T21:38:05.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:05.060+0000 7f8db7dff700 1 -- 172.21.15.47:0/3748813642 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8db0104bc0 con 0x7f8db00fdd10 2022-09-07T21:38:05.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:05.061+0000 7f8da6ffd700 1 -- 172.21.15.47:0/3748813642 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1257 (secure 0 0 0) 0x7f8dac0921f0 con 0x7f8db00fdd10 2022-09-07T21:38:05.064 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:05.064 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2022-09-07T21:38:05.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:05.063+0000 7f8db7dff700 1 -- 172.21.15.47:0/3748813642 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8d9c099080 msgr2=0x7f8d9c09b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:05.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:05.063+0000 7f8db7dff700 1 --2- 172.21.15.47:0/3748813642 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8d9c099080 0x7f8d9c09b530 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8da000b4f0 tx=0x7f8da0009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:05.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:05.063+0000 7f8db7dff700 1 -- 172.21.15.47:0/3748813642 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8db00fdd10 msgr2=0x7f8db0072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:05.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:05.063+0000 7f8db7dff700 1 --2- 172.21.15.47:0/3748813642 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8db00fdd10 0x7f8db0072a90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8dac00e710 tx=0x7f8dac00ea20 comp rx=0 tx=0).stop 2022-09-07T21:38:05.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:05.063+0000 7f8db7dff700 1 -- 172.21.15.47:0/3748813642 shutdown_connections 2022-09-07T21:38:05.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:05.063+0000 7f8db7dff700 1 --2- 172.21.15.47:0/3748813642 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8d9c099080 0x7f8d9c09b530 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:05.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:05.063+0000 7f8db7dff700 1 --2- 172.21.15.47:0/3748813642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00ff920 0x7f8db0071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:05.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:05.063+0000 7f8db7dff700 1 --2- 172.21.15.47:0/3748813642 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8db00fdd10 0x7f8db0072a90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:05.063+0000 7f8db7dff700 1 --2- 172.21.15.47:0/3748813642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8db00fef90 0x7f8db00710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:05.063+0000 7f8db7dff700 1 -- 172.21.15.47:0/3748813642 >> 172.21.15.47:0/3748813642 conn(0x7f8db00f93c0 msgr2=0x7f8db00fb730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:05.063+0000 7f8db7dff700 1 -- 172.21.15.47:0/3748813642 shutdown_connections 2022-09-07T21:38:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:05.063+0000 7f8db7dff700 1 -- 172.21.15.47:0/3748813642 wait complete. 2022-09-07T21:38:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 2 2022-09-07T21:38:05.309 INFO:journalctl@ceph.mds.k.smithi055.stdout:Sep 07 21:38:05 smithi055 systemd[1]: Starting Ceph mds.k for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:38:05.591 INFO:journalctl@ceph.mds.k.smithi055.stdout:Sep 07 21:38:05 smithi055 podman[122973]: 2022-09-07T21:38:05.591 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:05 smithi055 ceph-mon[97033]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:05.591 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:05 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3748813642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:05.732 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:05 smithi047 ceph-mon[90087]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:05.732 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:05 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3748813642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:05 smithi059 ceph-mon[97450]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:05 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3748813642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:05.980 INFO:journalctl@ceph.mds.k.smithi055.stdout:Sep 07 21:38:05 smithi055 bash[122973]: 0a5b12806a98e6f8cd86a96e203f0ff63d65a73cbeb507ca2d1c1dc89ea3a83f 2022-09-07T21:38:05.980 INFO:journalctl@ceph.mds.k.smithi055.stdout:Sep 07 21:38:05 smithi055 ceph-mds[123012]: -- 172.21.15.55:0/3240842081 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ee4b674820 con 0x55ee4b648c00 2022-09-07T21:38:05.980 INFO:journalctl@ceph.mds.k.smithi055.stdout:Sep 07 21:38:05 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-k[122989]: starting mds.k at 2022-09-07T21:38:05.981 INFO:journalctl@ceph.mds.k.smithi055.stdout:Sep 07 21:38:05 smithi055 systemd[1]: Started Ceph mds.k for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:38:06.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:06 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:06.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:06 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:06.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:06 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:06.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:06 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:06.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:06 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:06.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:06 smithi047 ceph-mon[90087]: Deploying daemon mds.j on smithi047 2022-09-07T21:38:06.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:06 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.55:6832/3763216833,v1:172.21.15.55:6833/3763216833] up:boot 2022-09-07T21:38:06.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:06 smithi047 ceph-mon[90087]: fsmap 2 up:standby 2022-09-07T21:38:06.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:06 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-09-07T21:38:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:06 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:06 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:06 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:06 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:06 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:06 smithi055 ceph-mon[97033]: Deploying daemon mds.j on smithi047 2022-09-07T21:38:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:06 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.55:6832/3763216833,v1:172.21.15.55:6833/3763216833] up:boot 2022-09-07T21:38:06.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:06 smithi055 ceph-mon[97033]: fsmap 2 up:standby 2022-09-07T21:38:06.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:06 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-09-07T21:38:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:06 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:06 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:06 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:06 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:06 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:06 smithi059 ceph-mon[97450]: Deploying daemon mds.j on smithi047 2022-09-07T21:38:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:06 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.55:6832/3763216833,v1:172.21.15.55:6833/3763216833] up:boot 2022-09-07T21:38:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:06 smithi059 ceph-mon[97450]: fsmap 2 up:standby 2022-09-07T21:38:07.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:06 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-09-07T21:38:07.094 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:07.340 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.338+0000 7fa4e3bff700 1 -- 172.21.15.47:0/4188716414 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4dc072c90 msgr2=0x7fa4dc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:07.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.338+0000 7fa4e3bff700 1 --2- 172.21.15.47:0/4188716414 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4dc072c90 0x7fa4dc071170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8009810 tx=0x7fa4d8009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:07.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.338+0000 7fa4e3bff700 1 -- 172.21.15.47:0/4188716414 shutdown_connections 2022-09-07T21:38:07.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.338+0000 7fa4e3bff700 1 --2- 172.21.15.47:0/4188716414 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4dc105bd0 0x7fa4dc108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:07.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.338+0000 7fa4e3bff700 1 --2- 172.21.15.47:0/4188716414 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4dc072c90 0x7fa4dc071170 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:07.342 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.338+0000 7fa4e3bff700 1 --2- 172.21.15.47:0/4188716414 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4dc071740 0x7fa4dc071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:07.342 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.338+0000 7fa4e3bff700 1 -- 172.21.15.47:0/4188716414 >> 172.21.15.47:0/4188716414 conn(0x7fa4dc0f93c0 msgr2=0x7fa4dc0fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:07.342 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.338+0000 7fa4e3bff700 1 -- 172.21.15.47:0/4188716414 shutdown_connections 2022-09-07T21:38:07.342 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.338+0000 7fa4e3bff700 1 -- 172.21.15.47:0/4188716414 wait complete. 2022-09-07T21:38:07.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.339+0000 7fa4e3bff700 1 Processor -- start 2022-09-07T21:38:07.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.339+0000 7fa4e3bff700 1 -- start start 2022-09-07T21:38:07.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e3bff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4dc071740 0x7fa4dc195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:07.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e3bff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4dc072c90 0x7fa4dc196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:07.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e3bff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4dc105bd0 0x7fa4dc19d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:07.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e3bff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc19d6b0 con 0x7fa4dc071740 2022-09-07T21:38:07.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e3bff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc19d820 con 0x7fa4dc072c90 2022-09-07T21:38:07.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e3bff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc19d9e0 con 0x7fa4dc105bd0 2022-09-07T21:38:07.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e199b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4dc071740 0x7fa4dc195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:07.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e119a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4dc072c90 0x7fa4dc196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:07.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e199b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4dc071740 0x7fa4dc195b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50508/0 (socket says 172.21.15.47:50508) 2022-09-07T21:38:07.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e199b700 1 -- 172.21.15.47:0/2732534749 learned_addr learned my addr 172.21.15.47:0/2732534749 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:07.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e219c700 1 --2- 172.21.15.47:0/2732534749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4dc105bd0 0x7fa4dc19d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:07.349 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e119a700 1 -- 172.21.15.47:0/2732534749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4dc105bd0 msgr2=0x7fa4dc19d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:07.349 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e119a700 1 --2- 172.21.15.47:0/2732534749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4dc105bd0 0x7fa4dc19d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:07.349 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e119a700 1 -- 172.21.15.47:0/2732534749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4dc071740 msgr2=0x7fa4dc195b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:07.349 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e119a700 1 --2- 172.21.15.47:0/2732534749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4dc071740 0x7fa4dc195b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:07.349 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e119a700 1 -- 172.21.15.47:0/2732534749 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d8009470 con 0x7fa4dc072c90 2022-09-07T21:38:07.350 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.340+0000 7fa4e199b700 1 --2- 172.21.15.47:0/2732534749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4dc071740 0x7fa4dc195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:07.350 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.341+0000 7fa4e119a700 1 --2- 172.21.15.47:0/2732534749 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4dc072c90 0x7fa4dc196070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc00e6c0 tx=0x7fa4cc00e9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:07.350 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.341+0000 7fa4d2ffd700 1 -- 172.21.15.47:0/2732534749 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc010900 con 0x7fa4dc072c90 2022-09-07T21:38:07.350 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.341+0000 7fa4e3bff700 1 -- 172.21.15.47:0/2732534749 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4dc19dcc0 con 0x7fa4dc072c90 2022-09-07T21:38:07.351 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.341+0000 7fa4d2ffd700 1 -- 172.21.15.47:0/2732534749 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa4cc018420 con 0x7fa4dc072c90 2022-09-07T21:38:07.351 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.341+0000 7fa4d2ffd700 1 -- 172.21.15.47:0/2732534749 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc017610 con 0x7fa4dc072c90 2022-09-07T21:38:07.351 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.341+0000 7fa4e3bff700 1 -- 172.21.15.47:0/2732534749 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4dc19e1d0 con 0x7fa4dc072c90 2022-09-07T21:38:07.351 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.342+0000 7fa4e3bff700 1 -- 172.21.15.47:0/2732534749 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4dc10b710 con 0x7fa4dc072c90 2022-09-07T21:38:07.352 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.344+0000 7fa4d2ffd700 1 -- 172.21.15.47:0/2732534749 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fa4cc00c040 con 0x7fa4dc072c90 2022-09-07T21:38:07.353 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.346+0000 7fa4d2ffd700 1 --2- 172.21.15.47:0/2732534749 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4c8099000 0x7fa4c809b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:07.353 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.346+0000 7fa4d2ffd700 1 -- 172.21.15.47:0/2732534749 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fa4cc014070 con 0x7fa4dc072c90 2022-09-07T21:38:07.353 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.346+0000 7fa4e199b700 1 --2- 172.21.15.47:0/2732534749 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4c8099000 0x7fa4c809b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:07.354 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.347+0000 7fa4e199b700 1 --2- 172.21.15.47:0/2732534749 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4c8099000 0x7fa4c809b4b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa4d800b4f0 tx=0x7fa4d8009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:07.354 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.350+0000 7fa4d2ffd700 1 -- 172.21.15.47:0/2732534749 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fa4cc04fc80 con 0x7fa4dc072c90 2022-09-07T21:38:07.643 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.640+0000 7fa4e3bff700 1 -- 172.21.15.47:0/2732534749 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4dc05fb40 con 0x7fa4dc072c90 2022-09-07T21:38:07.644 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.642+0000 7fa4d2ffd700 1 -- 172.21.15.47:0/2732534749 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2014 (secure 0 0 0) 0x7fa4cc08eaf0 con 0x7fa4dc072c90 2022-09-07T21:38:07.645 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:07.645 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2022-09-07T21:38:07.648 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.644+0000 7fa4e3bff700 1 -- 172.21.15.47:0/2732534749 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4c8099000 msgr2=0x7fa4c809b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:07.649 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.644+0000 7fa4e3bff700 1 --2- 172.21.15.47:0/2732534749 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4c8099000 0x7fa4c809b4b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa4d800b4f0 tx=0x7fa4d8009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:07.649 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.644+0000 7fa4e3bff700 1 -- 172.21.15.47:0/2732534749 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4dc072c90 msgr2=0x7fa4dc196070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:07.649 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.644+0000 7fa4e3bff700 1 --2- 172.21.15.47:0/2732534749 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4dc072c90 0x7fa4dc196070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc00e6c0 tx=0x7fa4cc00e9d0 comp rx=0 tx=0).stop 2022-09-07T21:38:07.650 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.644+0000 7fa4e3bff700 1 -- 172.21.15.47:0/2732534749 shutdown_connections 2022-09-07T21:38:07.650 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.644+0000 7fa4e3bff700 1 --2- 172.21.15.47:0/2732534749 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4c8099000 0x7fa4c809b4b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:07.651 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.644+0000 7fa4e3bff700 1 --2- 172.21.15.47:0/2732534749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4dc105bd0 0x7fa4dc19d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:07.651 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.645+0000 7fa4e3bff700 1 --2- 172.21.15.47:0/2732534749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4dc071740 0x7fa4dc195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:07.652 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.645+0000 7fa4e3bff700 1 --2- 172.21.15.47:0/2732534749 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4dc072c90 0x7fa4dc196070 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:07.652 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.645+0000 7fa4e3bff700 1 -- 172.21.15.47:0/2732534749 >> 172.21.15.47:0/2732534749 conn(0x7fa4dc0f93c0 msgr2=0x7fa4dc0fb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:07.653 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.645+0000 7fa4e3bff700 1 -- 172.21.15.47:0/2732534749 shutdown_connections 2022-09-07T21:38:07.653 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:07.645+0000 7fa4e3bff700 1 -- 172.21.15.47:0/2732534749 wait complete. 2022-09-07T21:38:07.653 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 3 2022-09-07T21:38:07.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:07 smithi047 ceph-mon[90087]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:07.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:07 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2732534749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:07 smithi055 ceph-mon[97033]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:07 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2732534749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:07 smithi059 ceph-mon[97450]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:07 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2732534749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 21:38:08 smithi047 systemd[1]: Starting Ceph mds.j for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:38:09.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:08 smithi047 ceph-mon[90087]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 21:38:08 smithi047 podman[124039]: 2022-09-07T21:38:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:08 smithi055 ceph-mon[97033]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:08 smithi059 ceph-mon[97450]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:09.453 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 21:38:09 smithi047 bash[124039]: 9e083b706bff647fdcb80b6ea2e3ff19dedd0c8641393c5dc19ff30ad61432e2 2022-09-07T21:38:09.454 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 21:38:09 smithi047 ceph-mds[124079]: -- 172.21.15.47:0/17727676 <== mon.0 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5610e8896820 con 0x5610e886ac00 2022-09-07T21:38:09.454 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 21:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: starting mds.j at 2022-09-07T21:38:09.454 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 21:38:09 smithi047 systemd[1]: Started Ceph mds.j for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:38:09.682 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:09.924 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.921+0000 7f899afff700 1 -- 172.21.15.47:0/1243449772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8994066be0 msgr2=0x7f89940fe6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:09.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.921+0000 7f899afff700 1 --2- 172.21.15.47:0/1243449772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8994066be0 0x7f89940fe6a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8988009810 tx=0x7f8988009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:09.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.922+0000 7f899afff700 1 -- 172.21.15.47:0/1243449772 shutdown_connections 2022-09-07T21:38:09.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.922+0000 7f899afff700 1 --2- 172.21.15.47:0/1243449772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89941016d0 0x7f8994101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:09.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.922+0000 7f899afff700 1 --2- 172.21.15.47:0/1243449772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8994066be0 0x7f89940fe6a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:09.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.922+0000 7f899afff700 1 --2- 172.21.15.47:0/1243449772 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89940febe0 0x7f8994101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:09.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.922+0000 7f899afff700 1 -- 172.21.15.47:0/1243449772 >> 172.21.15.47:0/1243449772 conn(0x7f89940f82e0 msgr2=0x7f89940fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:09.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.922+0000 7f899afff700 1 -- 172.21.15.47:0/1243449772 shutdown_connections 2022-09-07T21:38:09.927 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.922+0000 7f899afff700 1 -- 172.21.15.47:0/1243449772 wait complete. 2022-09-07T21:38:09.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.923+0000 7f899afff700 1 Processor -- start 2022-09-07T21:38:09.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.923+0000 7f899afff700 1 -- start start 2022-09-07T21:38:09.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.923+0000 7f899afff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8994066be0 0x7f8994199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:09.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.923+0000 7f899afff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89940febe0 0x7f899419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:09.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.924+0000 7f899afff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89941016d0 0x7f89941a14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:09.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.924+0000 7f899afff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89941a1ac0 con 0x7f89941016d0 2022-09-07T21:38:09.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.924+0000 7f899afff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89941a1c30 con 0x7f89940febe0 2022-09-07T21:38:09.931 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.924+0000 7f899afff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89941a1df0 con 0x7f8994066be0 2022-09-07T21:38:09.931 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.924+0000 7f8993fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89940febe0 0x7f899419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:09.931 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.924+0000 7f8993fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89940febe0 0x7f899419a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:57590/0 (socket says 172.21.15.47:57590) 2022-09-07T21:38:09.931 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.924+0000 7f8993fff700 1 -- 172.21.15.47:0/3827957077 learned_addr learned my addr 172.21.15.47:0/3827957077 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:09.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.924+0000 7f899959c700 1 --2- 172.21.15.47:0/3827957077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89941016d0 0x7f89941a14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:09.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.924+0000 7f8998d9b700 1 --2- 172.21.15.47:0/3827957077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8994066be0 0x7f8994199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:09.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.924+0000 7f8993fff700 1 -- 172.21.15.47:0/3827957077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8994066be0 msgr2=0x7f8994199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:09.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.924+0000 7f8993fff700 1 --2- 172.21.15.47:0/3827957077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8994066be0 0x7f8994199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:09.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.924+0000 7f8993fff700 1 -- 172.21.15.47:0/3827957077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89941016d0 msgr2=0x7f89941a14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:09.933 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.924+0000 7f8993fff700 1 --2- 172.21.15.47:0/3827957077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89941016d0 0x7f89941a14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:09.933 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.924+0000 7f8993fff700 1 -- 172.21.15.47:0/3827957077 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8988009470 con 0x7f89940febe0 2022-09-07T21:38:09.933 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.925+0000 7f8993fff700 1 --2- 172.21.15.47:0/3827957077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89940febe0 0x7f899419a480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f898000e740 tx=0x7f898000ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:09.933 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.925+0000 7f8991ffb700 1 -- 172.21.15.47:0/3827957077 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980005220 con 0x7f89940febe0 2022-09-07T21:38:09.934 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.925+0000 7f8991ffb700 1 -- 172.21.15.47:0/3827957077 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8980005380 con 0x7f89940febe0 2022-09-07T21:38:09.934 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.925+0000 7f899afff700 1 -- 172.21.15.47:0/3827957077 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89941a20d0 con 0x7f89940febe0 2022-09-07T21:38:09.934 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.925+0000 7f8991ffb700 1 -- 172.21.15.47:0/3827957077 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980004b70 con 0x7f89940febe0 2022-09-07T21:38:09.934 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.925+0000 7f899afff700 1 -- 172.21.15.47:0/3827957077 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89941a25e0 con 0x7f89940febe0 2022-09-07T21:38:09.935 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.926+0000 7f899afff700 1 -- 172.21.15.47:0/3827957077 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f899410fab0 con 0x7f89940febe0 2022-09-07T21:38:09.936 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.928+0000 7f8991ffb700 1 -- 172.21.15.47:0/3827957077 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f898001e020 con 0x7f89940febe0 2022-09-07T21:38:09.936 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.930+0000 7f8991ffb700 1 --2- 172.21.15.47:0/3827957077 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8984099000 0x7f898409b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:09.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.930+0000 7f8991ffb700 1 -- 172.21.15.47:0/3827957077 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f8980014070 con 0x7f89940febe0 2022-09-07T21:38:09.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.931+0000 7f8998d9b700 1 --2- 172.21.15.47:0/3827957077 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8984099000 0x7f898409b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:09.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.932+0000 7f8991ffb700 1 -- 172.21.15.47:0/3827957077 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f898008ef50 con 0x7f89940febe0 2022-09-07T21:38:09.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:09.932+0000 7f8998d9b700 1 --2- 172.21.15.47:0/3827957077 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8984099000 0x7f898409b4b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8988000c00 tx=0x7f8988009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:10.208 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:10.206+0000 7f899afff700 1 -- 172.21.15.47:0/3827957077 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89940f9cd0 con 0x7f89940febe0 2022-09-07T21:38:10.209 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:10.207+0000 7f8991ffb700 1 -- 172.21.15.47:0/3827957077 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2771 (secure 0 0 0) 0x7f8980092570 con 0x7f89940febe0 2022-09-07T21:38:10.209 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:10.210 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2022-09-07T21:38:10.212 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:10.209+0000 7f899afff700 1 -- 172.21.15.47:0/3827957077 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8984099000 msgr2=0x7f898409b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:10.212 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:10.209+0000 7f899afff700 1 --2- 172.21.15.47:0/3827957077 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8984099000 0x7f898409b4b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8988000c00 tx=0x7f8988009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:10.213 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:10.209+0000 7f899afff700 1 -- 172.21.15.47:0/3827957077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89940febe0 msgr2=0x7f899419a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:10.213 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:10.209+0000 7f899afff700 1 --2- 172.21.15.47:0/3827957077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89940febe0 0x7f899419a480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f898000e740 tx=0x7f898000ea50 comp rx=0 tx=0).stop 2022-09-07T21:38:10.213 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:10.209+0000 7f899afff700 1 -- 172.21.15.47:0/3827957077 shutdown_connections 2022-09-07T21:38:10.213 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:10.209+0000 7f899afff700 1 --2- 172.21.15.47:0/3827957077 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8984099000 0x7f898409b4b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:10.214 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:10.209+0000 7f899afff700 1 --2- 172.21.15.47:0/3827957077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89941016d0 0x7f89941a14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:10.214 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:10.209+0000 7f899afff700 1 --2- 172.21.15.47:0/3827957077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8994066be0 0x7f8994199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:10.214 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:10.209+0000 7f899afff700 1 --2- 172.21.15.47:0/3827957077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89940febe0 0x7f899419a480 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:10.214 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:10.209+0000 7f899afff700 1 -- 172.21.15.47:0/3827957077 >> 172.21.15.47:0/3827957077 conn(0x7f89940f82e0 msgr2=0x7f89940fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:10.215 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:10.210+0000 7f899afff700 1 -- 172.21.15.47:0/3827957077 shutdown_connections 2022-09-07T21:38:10.215 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:10.210+0000 7f899afff700 1 -- 172.21.15.47:0/3827957077 wait complete. 2022-09-07T21:38:10.215 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 4 2022-09-07T21:38:10.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:10 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:10.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:10 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:10.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:10 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:10.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:10 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:10.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:10 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:10.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:10 smithi047 ceph-mon[90087]: Deploying daemon mds.d on smithi047 2022-09-07T21:38:10.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:10 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.47:6834/3393333058,v1:172.21.15.47:6835/3393333058] up:boot 2022-09-07T21:38:10.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:10 smithi047 ceph-mon[90087]: fsmap 3 up:standby 2022-09-07T21:38:10.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:10 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:38:10.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:10 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3827957077' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:10 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:10 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:10 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:10 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:10 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:10 smithi055 ceph-mon[97033]: Deploying daemon mds.d on smithi047 2022-09-07T21:38:10.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:10 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.47:6834/3393333058,v1:172.21.15.47:6835/3393333058] up:boot 2022-09-07T21:38:10.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:10 smithi055 ceph-mon[97033]: fsmap 3 up:standby 2022-09-07T21:38:10.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:10 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:38:10.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:10 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3827957077' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:10 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:10 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:10 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:10 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:10 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:10 smithi059 ceph-mon[97450]: Deploying daemon mds.d on smithi047 2022-09-07T21:38:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:10 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.47:6834/3393333058,v1:172.21.15.47:6835/3393333058] up:boot 2022-09-07T21:38:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:10 smithi059 ceph-mon[97450]: fsmap 3 up:standby 2022-09-07T21:38:10.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:10 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:38:10.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:10 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3827957077' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:11.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:11 smithi047 ceph-mon[90087]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:11 smithi055 ceph-mon[97033]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:11 smithi059 ceph-mon[97450]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:12.241 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:12.498 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.495+0000 7fc7811ff700 1 -- 172.21.15.47:0/4202059277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc77c066be0 msgr2=0x7fc77c1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:12.498 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.495+0000 7fc7811ff700 1 --2- 172.21.15.47:0/4202059277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc77c066be0 0x7fc77c1008b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009810 tx=0x7fc76c009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:12.499 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.495+0000 7fc7811ff700 1 -- 172.21.15.47:0/4202059277 shutdown_connections 2022-09-07T21:38:12.499 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.495+0000 7fc7811ff700 1 --2- 172.21.15.47:0/4202059277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc77c1038e0 0x7fc77c105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:12.499 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.495+0000 7fc7811ff700 1 --2- 172.21.15.47:0/4202059277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc77c066be0 0x7fc77c1008b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:12.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.495+0000 7fc7811ff700 1 --2- 172.21.15.47:0/4202059277 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77c100df0 0x7fc77c103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:12.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.495+0000 7fc7811ff700 1 -- 172.21.15.47:0/4202059277 >> 172.21.15.47:0/4202059277 conn(0x7fc77c0f82e0 msgr2=0x7fc77c0fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:12.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.496+0000 7fc7811ff700 1 -- 172.21.15.47:0/4202059277 shutdown_connections 2022-09-07T21:38:12.501 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.496+0000 7fc7811ff700 1 -- 172.21.15.47:0/4202059277 wait complete. 2022-09-07T21:38:12.503 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.497+0000 7fc7811ff700 1 Processor -- start 2022-09-07T21:38:12.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.498+0000 7fc7811ff700 1 -- start start 2022-09-07T21:38:12.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.498+0000 7fc7811ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc77c066be0 0x7fc77c193950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:12.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.498+0000 7fc7811ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77c100df0 0x7fc77c193e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:12.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.498+0000 7fc7811ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc77c1038e0 0x7fc77c19af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:12.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.498+0000 7fc7811ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c19b4d0 con 0x7fc77c1038e0 2022-09-07T21:38:12.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.498+0000 7fc7811ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c19b640 con 0x7fc77c066be0 2022-09-07T21:38:12.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.498+0000 7fc7811ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c19b800 con 0x7fc77c100df0 2022-09-07T21:38:12.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.499+0000 7fc77ad9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc77c066be0 0x7fc77c193950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:12.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.499+0000 7fc77ad9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc77c066be0 0x7fc77c193950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:57602/0 (socket says 172.21.15.47:57602) 2022-09-07T21:38:12.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.499+0000 7fc77ad9d700 1 -- 172.21.15.47:0/1269111556 learned_addr learned my addr 172.21.15.47:0/1269111556 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:12.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.499+0000 7fc77a59c700 1 --2- 172.21.15.47:0/1269111556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77c100df0 0x7fc77c193e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:12.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.499+0000 7fc77b59e700 1 --2- 172.21.15.47:0/1269111556 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc77c1038e0 0x7fc77c19af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:12.507 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.499+0000 7fc77ad9d700 1 -- 172.21.15.47:0/1269111556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77c100df0 msgr2=0x7fc77c193e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:12.507 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.499+0000 7fc77ad9d700 1 --2- 172.21.15.47:0/1269111556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77c100df0 0x7fc77c193e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:12.507 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.499+0000 7fc77ad9d700 1 -- 172.21.15.47:0/1269111556 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc77c1038e0 msgr2=0x7fc77c19af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:12.507 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.499+0000 7fc77ad9d700 1 --2- 172.21.15.47:0/1269111556 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc77c1038e0 0x7fc77c19af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:12.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.499+0000 7fc77ad9d700 1 -- 172.21.15.47:0/1269111556 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76c009470 con 0x7fc77c066be0 2022-09-07T21:38:12.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.499+0000 7fc77ad9d700 1 --2- 172.21.15.47:0/1269111556 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc77c066be0 0x7fc77c193950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc76c006010 tx=0x7fc76c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:12.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.500+0000 7fc763fff700 1 -- 172.21.15.47:0/1269111556 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c01d070 con 0x7fc77c066be0 2022-09-07T21:38:12.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.500+0000 7fc7811ff700 1 -- 172.21.15.47:0/1269111556 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77c19ba80 con 0x7fc77c066be0 2022-09-07T21:38:12.509 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.500+0000 7fc763fff700 1 -- 172.21.15.47:0/1269111556 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fc76c00f880 con 0x7fc77c066be0 2022-09-07T21:38:12.509 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.500+0000 7fc763fff700 1 -- 172.21.15.47:0/1269111556 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c0047f0 con 0x7fc77c066be0 2022-09-07T21:38:12.509 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.500+0000 7fc7811ff700 1 -- 172.21.15.47:0/1269111556 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77c19bf80 con 0x7fc77c066be0 2022-09-07T21:38:12.509 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.501+0000 7fc7811ff700 1 -- 172.21.15.47:0/1269111556 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc77c1094b0 con 0x7fc77c066be0 2022-09-07T21:38:12.510 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.504+0000 7fc763fff700 1 -- 172.21.15.47:0/1269111556 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fc76c028030 con 0x7fc77c066be0 2022-09-07T21:38:12.511 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.505+0000 7fc763fff700 1 --2- 172.21.15.47:0/1269111556 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc764099000 0x7fc76409b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:12.511 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.505+0000 7fc763fff700 1 -- 172.21.15.47:0/1269111556 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fc76c0c2b50 con 0x7fc77c066be0 2022-09-07T21:38:12.511 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.505+0000 7fc77a59c700 1 --2- 172.21.15.47:0/1269111556 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc764099000 0x7fc76409b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:12.511 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.506+0000 7fc77a59c700 1 --2- 172.21.15.47:0/1269111556 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc764099000 0x7fc76409b4b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc770005fd0 tx=0x7fc770005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:12.512 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.507+0000 7fc763fff700 1 -- 172.21.15.47:0/1269111556 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fc76c08f640 con 0x7fc77c066be0 2022-09-07T21:38:12.653 INFO:journalctl@ceph.mds.d.smithi047.stdout:Sep 07 21:38:12 smithi047 systemd[1]: Starting Ceph mds.d for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:38:12.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.838+0000 7fc7811ff700 1 -- 172.21.15.47:0/1269111556 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc77c05f990 con 0x7fc77c066be0 2022-09-07T21:38:12.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.839+0000 7fc763fff700 1 -- 172.21.15.47:0/1269111556 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2771 (secure 0 0 0) 0x7fc76c092c60 con 0x7fc77c066be0 2022-09-07T21:38:12.842 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:12.842 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2022-09-07T21:38:12.844 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.841+0000 7fc7811ff700 1 -- 172.21.15.47:0/1269111556 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc764099000 msgr2=0x7fc76409b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:12.844 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.841+0000 7fc7811ff700 1 --2- 172.21.15.47:0/1269111556 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc764099000 0x7fc76409b4b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc770005fd0 tx=0x7fc770005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:12.844 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.841+0000 7fc7811ff700 1 -- 172.21.15.47:0/1269111556 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc77c066be0 msgr2=0x7fc77c193950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:12.845 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.841+0000 7fc7811ff700 1 --2- 172.21.15.47:0/1269111556 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc77c066be0 0x7fc77c193950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc76c006010 tx=0x7fc76c00f690 comp rx=0 tx=0).stop 2022-09-07T21:38:12.845 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.841+0000 7fc7811ff700 1 -- 172.21.15.47:0/1269111556 shutdown_connections 2022-09-07T21:38:12.845 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.841+0000 7fc7811ff700 1 --2- 172.21.15.47:0/1269111556 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc764099000 0x7fc76409b4b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:12.845 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.841+0000 7fc7811ff700 1 --2- 172.21.15.47:0/1269111556 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc77c1038e0 0x7fc77c19af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:12.846 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.841+0000 7fc7811ff700 1 --2- 172.21.15.47:0/1269111556 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc77c066be0 0x7fc77c193950 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:12.846 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.841+0000 7fc7811ff700 1 --2- 172.21.15.47:0/1269111556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77c100df0 0x7fc77c193e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:12.846 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.841+0000 7fc7811ff700 1 -- 172.21.15.47:0/1269111556 >> 172.21.15.47:0/1269111556 conn(0x7fc77c0f82e0 msgr2=0x7fc77c0ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:12.846 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.842+0000 7fc7811ff700 1 -- 172.21.15.47:0/1269111556 shutdown_connections 2022-09-07T21:38:12.847 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:12.842+0000 7fc7811ff700 1 -- 172.21.15.47:0/1269111556 wait complete. 2022-09-07T21:38:12.847 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 4 2022-09-07T21:38:13.134 INFO:journalctl@ceph.mds.d.smithi047.stdout:Sep 07 21:38:12 smithi047 podman[124469]: 2022-09-07T21:38:13.635 INFO:journalctl@ceph.mds.d.smithi047.stdout:Sep 07 21:38:13 smithi047 bash[124469]: a3eb588e683633b7e7397cc69f306e9d9cb386a5b2ada15e8731fbc691dd5b82 2022-09-07T21:38:13.635 INFO:journalctl@ceph.mds.d.smithi047.stdout:Sep 07 21:38:13 smithi047 ceph-mds[124511]: -- 172.21.15.47:0/3221512334 <== mon.0 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562676878820 con 0x56267684c800 2022-09-07T21:38:13.636 INFO:journalctl@ceph.mds.d.smithi047.stdout:Sep 07 21:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-d[124487]: starting mds.d at 2022-09-07T21:38:13.636 INFO:journalctl@ceph.mds.d.smithi047.stdout:Sep 07 21:38:13 smithi047 systemd[1]: Started Ceph mds.d for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:38:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:13 smithi055 ceph-mon[97033]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:13 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1269111556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:13.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:13 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:13.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:13 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:13.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:13 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:13.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:13 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:13.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:13 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:13.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:13 smithi047 ceph-mon[90087]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:13.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:13 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1269111556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:13.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:13 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:13.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:13 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:13.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:13 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:13.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:13 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:13.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:13 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:13 smithi059 ceph-mon[97450]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:13 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1269111556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:13 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:13 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:13 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:13 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:14.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:13 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:14.838 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:14 smithi055 ceph-mon[97033]: Deploying daemon mds.e on smithi055 2022-09-07T21:38:14.839 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:14 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.47:6836/139419378,v1:172.21.15.47:6837/139419378] up:boot 2022-09-07T21:38:14.839 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:14 smithi055 ceph-mon[97033]: fsmap 4 up:standby 2022-09-07T21:38:14.839 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:14 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-07T21:38:14.873 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:14.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:14 smithi047 ceph-mon[90087]: Deploying daemon mds.e on smithi055 2022-09-07T21:38:14.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:14 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.47:6836/139419378,v1:172.21.15.47:6837/139419378] up:boot 2022-09-07T21:38:14.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:14 smithi047 ceph-mon[90087]: fsmap 4 up:standby 2022-09-07T21:38:14.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:14 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-07T21:38:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:14 smithi059 ceph-mon[97450]: Deploying daemon mds.e on smithi055 2022-09-07T21:38:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:14 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.47:6836/139419378,v1:172.21.15.47:6837/139419378] up:boot 2022-09-07T21:38:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:14 smithi059 ceph-mon[97450]: fsmap 4 up:standby 2022-09-07T21:38:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:14 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-07T21:38:15.118 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.115+0000 7f1d345ff700 1 -- 172.21.15.47:0/533449588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c100df0 msgr2=0x7f1d2c103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:15.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.115+0000 7f1d345ff700 1 --2- 172.21.15.47:0/533449588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c100df0 0x7f1d2c103250 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009810 tx=0x7f1d1c009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:15.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.116+0000 7f1d345ff700 1 -- 172.21.15.47:0/533449588 shutdown_connections 2022-09-07T21:38:15.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.116+0000 7f1d345ff700 1 --2- 172.21.15.47:0/533449588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d2c1038e0 0x7f1d2c105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:15.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.116+0000 7f1d345ff700 1 --2- 172.21.15.47:0/533449588 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d2c066be0 0x7f1d2c1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:15.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.116+0000 7f1d345ff700 1 --2- 172.21.15.47:0/533449588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c100df0 0x7f1d2c103250 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:15.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.116+0000 7f1d345ff700 1 -- 172.21.15.47:0/533449588 >> 172.21.15.47:0/533449588 conn(0x7f1d2c0f82e0 msgr2=0x7f1d2c0fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:15.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.116+0000 7f1d345ff700 1 -- 172.21.15.47:0/533449588 shutdown_connections 2022-09-07T21:38:15.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.116+0000 7f1d345ff700 1 -- 172.21.15.47:0/533449588 wait complete. 2022-09-07T21:38:15.123 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.117+0000 7f1d345ff700 1 Processor -- start 2022-09-07T21:38:15.123 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.117+0000 7f1d345ff700 1 -- start start 2022-09-07T21:38:15.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.117+0000 7f1d345ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c066be0 0x7f1d2c0fe1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:15.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.117+0000 7f1d345ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d2c100df0 0x7f1d2c0fc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:15.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.117+0000 7f1d345ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d2c1038e0 0x7f1d2c0fcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:15.125 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.118+0000 7f1d345ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c0fd550 con 0x7f1d2c100df0 2022-09-07T21:38:15.125 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.118+0000 7f1d345ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c0fd6c0 con 0x7f1d2c1038e0 2022-09-07T21:38:15.126 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.118+0000 7f1d345ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c19da20 con 0x7f1d2c066be0 2022-09-07T21:38:15.126 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.118+0000 7f1d3239b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c066be0 0x7f1d2c0fe1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:15.127 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.118+0000 7f1d3239b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c066be0 0x7f1d2c0fe1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:57568/0 (socket says 172.21.15.47:57568) 2022-09-07T21:38:15.127 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.118+0000 7f1d3239b700 1 -- 172.21.15.47:0/605409567 learned_addr learned my addr 172.21.15.47:0/605409567 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:15.127 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.118+0000 7f1d31b9a700 1 --2- 172.21.15.47:0/605409567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d2c100df0 0x7f1d2c0fc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:15.128 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.118+0000 7f1d32b9c700 1 --2- 172.21.15.47:0/605409567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d2c1038e0 0x7f1d2c0fcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:15.128 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.118+0000 7f1d3239b700 1 -- 172.21.15.47:0/605409567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d2c1038e0 msgr2=0x7f1d2c0fcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:15.129 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.118+0000 7f1d3239b700 1 --2- 172.21.15.47:0/605409567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d2c1038e0 0x7f1d2c0fcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:15.129 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.118+0000 7f1d3239b700 1 -- 172.21.15.47:0/605409567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d2c100df0 msgr2=0x7f1d2c0fc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:15.130 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.118+0000 7f1d3239b700 1 --2- 172.21.15.47:0/605409567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d2c100df0 0x7f1d2c0fc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:15.130 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.118+0000 7f1d3239b700 1 -- 172.21.15.47:0/605409567 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1c009470 con 0x7f1d2c066be0 2022-09-07T21:38:15.131 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.119+0000 7f1d3239b700 1 --2- 172.21.15.47:0/605409567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c066be0 0x7f1d2c0fe1c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800e6e0 tx=0x7f1d2800e9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:15.131 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.119+0000 7f1d237fe700 1 -- 172.21.15.47:0/605409567 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d28009500 con 0x7f1d2c066be0 2022-09-07T21:38:15.132 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.119+0000 7f1d237fe700 1 -- 172.21.15.47:0/605409567 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1d28010da0 con 0x7f1d2c066be0 2022-09-07T21:38:15.132 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.119+0000 7f1d237fe700 1 -- 172.21.15.47:0/605409567 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d280173d0 con 0x7f1d2c066be0 2022-09-07T21:38:15.132 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.119+0000 7f1d345ff700 1 -- 172.21.15.47:0/605409567 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2c19dd00 con 0x7f1d2c066be0 2022-09-07T21:38:15.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.119+0000 7f1d345ff700 1 -- 172.21.15.47:0/605409567 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2c19e1c0 con 0x7f1d2c066be0 2022-09-07T21:38:15.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.120+0000 7f1d345ff700 1 -- 172.21.15.47:0/605409567 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2c1094f0 con 0x7f1d2c066be0 2022-09-07T21:38:15.135 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.122+0000 7f1d237fe700 1 -- 172.21.15.47:0/605409567 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f1d2800c040 con 0x7f1d2c066be0 2022-09-07T21:38:15.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.123+0000 7f1d237fe700 1 --2- 172.21.15.47:0/605409567 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1d18099080 0x7f1d1809b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:15.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.123+0000 7f1d237fe700 1 -- 172.21.15.47:0/605409567 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f1d28014070 con 0x7f1d2c066be0 2022-09-07T21:38:15.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.124+0000 7f1d31b9a700 1 --2- 172.21.15.47:0/605409567 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1d18099080 0x7f1d1809b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:15.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.124+0000 7f1d31b9a700 1 --2- 172.21.15.47:0/605409567 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1d18099080 0x7f1d1809b530 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c00b4f0 tx=0x7f1d1c009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:15.138 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.126+0000 7f1d237fe700 1 -- 172.21.15.47:0/605409567 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f1d2808eb50 con 0x7f1d2c066be0 2022-09-07T21:38:15.230 INFO:journalctl@ceph.mds.e.smithi055.stdout:Sep 07 21:38:14 smithi055 systemd[1]: Starting Ceph mds.e for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:38:15.376 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.374+0000 7f1d345ff700 1 -- 172.21.15.47:0/605409567 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d2c19eb30 con 0x7f1d2c066be0 2022-09-07T21:38:15.377 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.375+0000 7f1d237fe700 1 -- 172.21.15.47:0/605409567 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3525 (secure 0 0 0) 0x7f1d28092170 con 0x7f1d2c066be0 2022-09-07T21:38:15.378 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:15.378 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[]} 2022-09-07T21:38:15.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.377+0000 7f1d345ff700 1 -- 172.21.15.47:0/605409567 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1d18099080 msgr2=0x7f1d1809b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:15.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.377+0000 7f1d345ff700 1 --2- 172.21.15.47:0/605409567 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1d18099080 0x7f1d1809b530 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c00b4f0 tx=0x7f1d1c009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:15.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.377+0000 7f1d345ff700 1 -- 172.21.15.47:0/605409567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c066be0 msgr2=0x7f1d2c0fe1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:15.381 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.377+0000 7f1d345ff700 1 --2- 172.21.15.47:0/605409567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c066be0 0x7f1d2c0fe1c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800e6e0 tx=0x7f1d2800e9f0 comp rx=0 tx=0).stop 2022-09-07T21:38:15.381 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.377+0000 7f1d345ff700 1 -- 172.21.15.47:0/605409567 shutdown_connections 2022-09-07T21:38:15.381 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.377+0000 7f1d345ff700 1 --2- 172.21.15.47:0/605409567 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1d18099080 0x7f1d1809b530 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:15.381 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.377+0000 7f1d345ff700 1 --2- 172.21.15.47:0/605409567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d2c1038e0 0x7f1d2c0fcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:15.382 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.377+0000 7f1d345ff700 1 --2- 172.21.15.47:0/605409567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2c066be0 0x7f1d2c0fe1c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:15.382 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.377+0000 7f1d345ff700 1 --2- 172.21.15.47:0/605409567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d2c100df0 0x7f1d2c0fc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:15.382 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.377+0000 7f1d345ff700 1 -- 172.21.15.47:0/605409567 >> 172.21.15.47:0/605409567 conn(0x7f1d2c0f82e0 msgr2=0x7f1d2c0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:15.382 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.378+0000 7f1d345ff700 1 -- 172.21.15.47:0/605409567 shutdown_connections 2022-09-07T21:38:15.382 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:15.378+0000 7f1d345ff700 1 -- 172.21.15.47:0/605409567 wait complete. 2022-09-07T21:38:15.383 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 5 2022-09-07T21:38:15.580 INFO:journalctl@ceph.mds.e.smithi055.stdout:Sep 07 21:38:15 smithi055 podman[123325]: 2022-09-07T21:38:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:15 smithi047 ceph-mon[90087]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:15.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:15 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/605409567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:15.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:15 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:15.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:15 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:15 smithi055 ceph-mon[97033]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:15 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/605409567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:15.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:15 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:15.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:15 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:15.981 INFO:journalctl@ceph.mds.e.smithi055.stdout:Sep 07 21:38:15 smithi055 ceph-mds[123365]: -- 172.21.15.55:0/925990072 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559f340b6820 con 0x559f3408ac00 2022-09-07T21:38:15.981 INFO:journalctl@ceph.mds.e.smithi055.stdout:Sep 07 21:38:15 smithi055 bash[123325]: f5aa23b80a9b2be757c283b78c1738effefb09b8e3f77816ccd6d6b427ae7fe2 2022-09-07T21:38:15.982 INFO:journalctl@ceph.mds.e.smithi055.stdout:Sep 07 21:38:15 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-e[123341]: starting mds.e at 2022-09-07T21:38:15.982 INFO:journalctl@ceph.mds.e.smithi055.stdout:Sep 07 21:38:15 smithi055 systemd[1]: Started Ceph mds.e for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:38:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:15 smithi059 ceph-mon[97450]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:15 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/605409567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:15 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:15 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:16.894 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:16 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:16.894 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:16 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:16.895 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:16 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:16.895 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:16 smithi059 ceph-mon[97450]: Deploying daemon mds.f on smithi059 2022-09-07T21:38:16.895 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:16 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.55:6834/2559266585,v1:172.21.15.55:6835/2559266585] up:boot 2022-09-07T21:38:16.895 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:16 smithi059 ceph-mon[97450]: fsmap 5 up:standby 2022-09-07T21:38:16.896 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:16 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-09-07T21:38:16.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:16 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:16.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:16 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:16.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:16 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:16.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:16 smithi047 ceph-mon[90087]: Deploying daemon mds.f on smithi059 2022-09-07T21:38:16.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:16 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.55:6834/2559266585,v1:172.21.15.55:6835/2559266585] up:boot 2022-09-07T21:38:16.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:16 smithi047 ceph-mon[90087]: fsmap 5 up:standby 2022-09-07T21:38:16.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:16 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-09-07T21:38:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:16 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:16 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:16 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:16.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:16 smithi055 ceph-mon[97033]: Deploying daemon mds.f on smithi059 2022-09-07T21:38:16.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:16 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.55:6834/2559266585,v1:172.21.15.55:6835/2559266585] up:boot 2022-09-07T21:38:16.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:16 smithi055 ceph-mon[97033]: fsmap 5 up:standby 2022-09-07T21:38:16.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:16 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-09-07T21:38:17.157 INFO:journalctl@ceph.mds.f.smithi059.stdout:Sep 07 21:38:17 smithi059 systemd[1]: Starting Ceph mds.f for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:38:17.409 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:17.651 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.647+0000 7f72a9fff700 1 -- 172.21.15.47:0/3233145360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a40fdd10 msgr2=0x7f72a40fe120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:17.652 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.647+0000 7f72a9fff700 1 --2- 172.21.15.47:0/3233145360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a40fdd10 0x7f72a40fe120 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7294009810 tx=0x7f7294009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:17.652 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.647+0000 7f72a9fff700 1 -- 172.21.15.47:0/3233145360 shutdown_connections 2022-09-07T21:38:17.652 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.648+0000 7f72a9fff700 1 --2- 172.21.15.47:0/3233145360 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72a40ff920 0x7f72a41061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:17.653 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.648+0000 7f72a9fff700 1 --2- 172.21.15.47:0/3233145360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a40fdd10 0x7f72a40fe120 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:17.653 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.648+0000 7f72a9fff700 1 --2- 172.21.15.47:0/3233145360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72a40fef90 0x7f72a40ff3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:17.653 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.648+0000 7f72a9fff700 1 -- 172.21.15.47:0/3233145360 >> 172.21.15.47:0/3233145360 conn(0x7f72a40f93c0 msgr2=0x7f72a40fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:17.653 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.648+0000 7f72a9fff700 1 -- 172.21.15.47:0/3233145360 shutdown_connections 2022-09-07T21:38:17.654 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.648+0000 7f72a9fff700 1 -- 172.21.15.47:0/3233145360 wait complete. 2022-09-07T21:38:17.656 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.649+0000 7f72a9fff700 1 Processor -- start 2022-09-07T21:38:17.657 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.649+0000 7f72a9fff700 1 -- start start 2022-09-07T21:38:17.657 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.649+0000 7f72a9fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72a40fdd10 0x7f72a4195b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:17.657 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.649+0000 7f72a9fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a40fef90 0x7f72a4196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:17.657 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.649+0000 7f72a9fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72a40ff920 0x7f72a419d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:17.657 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.649+0000 7f72a9fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a419d680 con 0x7f72a40fdd10 2022-09-07T21:38:17.658 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.649+0000 7f72a9fff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a419d7f0 con 0x7f72a40ff920 2022-09-07T21:38:17.658 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.649+0000 7f72a9fff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a419d9b0 con 0x7f72a40fef90 2022-09-07T21:38:17.658 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.650+0000 7f72a2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a40fef90 0x7f72a4196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:17.658 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.650+0000 7f72a37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72a40fdd10 0x7f72a4195b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:17.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.650+0000 7f72a2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a40fef90 0x7f72a4196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:57586/0 (socket says 172.21.15.47:57586) 2022-09-07T21:38:17.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.650+0000 7f72a2ffd700 1 -- 172.21.15.47:0/301575432 learned_addr learned my addr 172.21.15.47:0/301575432 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:17.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.650+0000 7f72a3fff700 1 --2- 172.21.15.47:0/301575432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72a40ff920 0x7f72a419d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:17.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.650+0000 7f72a2ffd700 1 -- 172.21.15.47:0/301575432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72a40ff920 msgr2=0x7f72a419d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:17.660 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.650+0000 7f72a2ffd700 1 --2- 172.21.15.47:0/301575432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72a40ff920 0x7f72a419d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:17.660 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.650+0000 7f72a2ffd700 1 -- 172.21.15.47:0/301575432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72a40fdd10 msgr2=0x7f72a4195b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:17.660 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.650+0000 7f72a2ffd700 1 --2- 172.21.15.47:0/301575432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72a40fdd10 0x7f72a4195b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:17.660 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.650+0000 7f72a2ffd700 1 -- 172.21.15.47:0/301575432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72980093b0 con 0x7f72a40fef90 2022-09-07T21:38:17.660 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.650+0000 7f72a2ffd700 1 --2- 172.21.15.47:0/301575432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a40fef90 0x7f72a4196040 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f729800e9c0 tx=0x7f729800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:17.661 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.651+0000 7f72a0ff9700 1 -- 172.21.15.47:0/301575432 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72980097b0 con 0x7f72a40fef90 2022-09-07T21:38:17.661 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.651+0000 7f72a0ff9700 1 -- 172.21.15.47:0/301575432 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f729800ce70 con 0x7f72a40fef90 2022-09-07T21:38:17.661 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.651+0000 7f72a0ff9700 1 -- 172.21.15.47:0/301575432 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7298003cf0 con 0x7f72a40fef90 2022-09-07T21:38:17.661 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.651+0000 7f72a9fff700 1 -- 172.21.15.47:0/301575432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7294009470 con 0x7f72a40fef90 2022-09-07T21:38:17.662 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.651+0000 7f72a9fff700 1 -- 172.21.15.47:0/301575432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a419dfc0 con 0x7f72a40fef90 2022-09-07T21:38:17.662 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.652+0000 7f72a9fff700 1 -- 172.21.15.47:0/301575432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a41094c0 con 0x7f72a40fef90 2022-09-07T21:38:17.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.654+0000 7f72a0ff9700 1 -- 172.21.15.47:0/301575432 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f729801e020 con 0x7f72a40fef90 2022-09-07T21:38:17.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.655+0000 7f72a0ff9700 1 --2- 172.21.15.47:0/301575432 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f728c099000 0x7f728c09b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:17.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.655+0000 7f72a0ff9700 1 -- 172.21.15.47:0/301575432 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f7298014070 con 0x7f72a40fef90 2022-09-07T21:38:17.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.655+0000 7f72a37fe700 1 --2- 172.21.15.47:0/301575432 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f728c099000 0x7f728c09b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:17.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.656+0000 7f72a37fe700 1 --2- 172.21.15.47:0/301575432 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f728c099000 0x7f728c09b4b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f729400b4f0 tx=0x7f7294009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:17.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.659+0000 7f72a0ff9700 1 -- 172.21.15.47:0/301575432 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f729808eed0 con 0x7f72a40fef90 2022-09-07T21:38:17.745 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:17 smithi059 ceph-mon[97450]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:17.745 INFO:journalctl@ceph.mds.f.smithi059.stdout:Sep 07 21:38:17 smithi059 podman[124357]: 2022-09-07T21:38:17.746 INFO:journalctl@ceph.mds.f.smithi059.stdout:Sep 07 21:38:17 smithi059 bash[124357]: ddb264a795b4d9850c59740840dd429a47e1410ff7a3e16b28c510fd47547e50 2022-09-07T21:38:17.746 INFO:journalctl@ceph.mds.f.smithi059.stdout:Sep 07 21:38:17 smithi059 ceph-mds[124397]: -- 172.21.15.59:0/541238629 <== mon.1 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56135da84820 con 0x56135da58c00 2022-09-07T21:38:17.746 INFO:journalctl@ceph.mds.f.smithi059.stdout:Sep 07 21:38:17 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-f[124373]: starting mds.f at 2022-09-07T21:38:17.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:17 smithi047 ceph-mon[90087]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:17.913 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.910+0000 7f72a9fff700 1 -- 172.21.15.47:0/301575432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72a419db70 con 0x7f72a40fef90 2022-09-07T21:38:17.914 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.910+0000 7f72a0ff9700 1 -- 172.21.15.47:0/301575432 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4282 (secure 0 0 0) 0x7f72980924f0 con 0x7f72a40fef90 2022-09-07T21:38:17.914 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:17.914 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/2559266585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2559266585},{"type":"v1","addr":"172.21.15.55:6835","nonce":2559266585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[]} 2022-09-07T21:38:17.917 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.913+0000 7f72a9fff700 1 -- 172.21.15.47:0/301575432 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f728c099000 msgr2=0x7f728c09b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:17.917 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.913+0000 7f72a9fff700 1 --2- 172.21.15.47:0/301575432 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f728c099000 0x7f728c09b4b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f729400b4f0 tx=0x7f7294009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:17.917 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.913+0000 7f72a9fff700 1 -- 172.21.15.47:0/301575432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a40fef90 msgr2=0x7f72a4196040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:17.917 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.913+0000 7f72a9fff700 1 --2- 172.21.15.47:0/301575432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a40fef90 0x7f72a4196040 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f729800e9c0 tx=0x7f729800ecd0 comp rx=0 tx=0).stop 2022-09-07T21:38:17.918 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.913+0000 7f72a9fff700 1 -- 172.21.15.47:0/301575432 shutdown_connections 2022-09-07T21:38:17.918 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.913+0000 7f72a9fff700 1 --2- 172.21.15.47:0/301575432 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f728c099000 0x7f728c09b4b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:17.918 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.913+0000 7f72a9fff700 1 --2- 172.21.15.47:0/301575432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72a40ff920 0x7f72a419d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:17.918 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.913+0000 7f72a9fff700 1 --2- 172.21.15.47:0/301575432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72a40fdd10 0x7f72a4195b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:17.919 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.913+0000 7f72a9fff700 1 --2- 172.21.15.47:0/301575432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72a40fef90 0x7f72a4196040 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:17.919 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.913+0000 7f72a9fff700 1 -- 172.21.15.47:0/301575432 >> 172.21.15.47:0/301575432 conn(0x7f72a40f93c0 msgr2=0x7f72a4104910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:17.919 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.913+0000 7f72a9fff700 1 -- 172.21.15.47:0/301575432 shutdown_connections 2022-09-07T21:38:17.919 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:17.913+0000 7f72a9fff700 1 -- 172.21.15.47:0/301575432 wait complete. 2022-09-07T21:38:17.920 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 6 2022-09-07T21:38:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:17 smithi055 ceph-mon[97033]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:18.032 INFO:journalctl@ceph.mds.f.smithi059.stdout:Sep 07 21:38:17 smithi059 systemd[1]: Started Ceph mds.f for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:38:19.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:19.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:19.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:19.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:19.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:19.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:18 smithi047 ceph-mon[90087]: Deploying daemon mds.c on smithi059 2022-09-07T21:38:19.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:18 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/301575432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:19.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:18 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.59:6834/2445409240,v1:172.21.15.59:6835/2445409240] up:boot 2022-09-07T21:38:19.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:18 smithi047 ceph-mon[90087]: fsmap 6 up:standby 2022-09-07T21:38:19.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:18 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-09-07T21:38:19.159 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:18 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:19.159 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:18 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:19.159 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:18 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:19.160 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:18 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:19.160 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:18 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:19.160 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:18 smithi059 ceph-mon[97450]: Deploying daemon mds.c on smithi059 2022-09-07T21:38:19.160 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:18 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/301575432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:19.161 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:18 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.59:6834/2445409240,v1:172.21.15.59:6835/2445409240] up:boot 2022-09-07T21:38:19.161 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:18 smithi059 ceph-mon[97450]: fsmap 6 up:standby 2022-09-07T21:38:19.161 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:18 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-09-07T21:38:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:18 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:18 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:18 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:18 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:18 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:18 smithi055 ceph-mon[97033]: Deploying daemon mds.c on smithi059 2022-09-07T21:38:19.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:18 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/301575432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:19.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:18 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.59:6834/2445409240,v1:172.21.15.59:6835/2445409240] up:boot 2022-09-07T21:38:19.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:18 smithi055 ceph-mon[97033]: fsmap 6 up:standby 2022-09-07T21:38:19.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:18 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-09-07T21:38:19.782 INFO:journalctl@ceph.mds.c.smithi059.stdout:Sep 07 21:38:19 smithi059 systemd[1]: Starting Ceph mds.c for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:38:19.945 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:20.071 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:19 smithi059 ceph-mon[97450]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:20.071 INFO:journalctl@ceph.mds.c.smithi059.stdout:Sep 07 21:38:19 smithi059 podman[124703]: 2022-09-07T21:38:20.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:19 smithi047 ceph-mon[90087]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:20.179 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:19 smithi055 ceph-mon[97033]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.186+0000 7f78e11ff700 1 -- 172.21.15.47:0/4093702908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78dc105b10 msgr2=0x7f78dc107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.186+0000 7f78e11ff700 1 --2- 172.21.15.47:0/4093702908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78dc105b10 0x7f78dc107f70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f78d4009b00 tx=0x7f78d4009e10 comp rx=0 tx=0).stop 2022-09-07T21:38:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.186+0000 7f78e11ff700 1 -- 172.21.15.47:0/4093702908 shutdown_connections 2022-09-07T21:38:20.191 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.186+0000 7f78e11ff700 1 --2- 172.21.15.47:0/4093702908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78dc105b10 0x7f78dc107f70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:20.191 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.186+0000 7f78e11ff700 1 --2- 172.21.15.47:0/4093702908 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78dc066be0 0x7f78dc102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:20.191 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.186+0000 7f78e11ff700 1 --2- 172.21.15.47:0/4093702908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78dc1031f0 0x7f78dc1055d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:20.192 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.186+0000 7f78e11ff700 1 -- 172.21.15.47:0/4093702908 >> 172.21.15.47:0/4093702908 conn(0x7f78dc0f82e0 msgr2=0x7f78dc0fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:20.192 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.186+0000 7f78e11ff700 1 -- 172.21.15.47:0/4093702908 shutdown_connections 2022-09-07T21:38:20.192 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.186+0000 7f78e11ff700 1 -- 172.21.15.47:0/4093702908 wait complete. 2022-09-07T21:38:20.194 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.187+0000 7f78e11ff700 1 Processor -- start 2022-09-07T21:38:20.195 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.188+0000 7f78e11ff700 1 -- start start 2022-09-07T21:38:20.195 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.188+0000 7f78e11ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78dc066be0 0x7f78dc195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:20.195 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.188+0000 7f78e11ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78dc1031f0 0x7f78dc196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:20.195 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.188+0000 7f78e11ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78dc105b10 0x7f78dc19d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.188+0000 7f78e11ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc19d6c0 con 0x7f78dc066be0 2022-09-07T21:38:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.188+0000 7f78e11ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc19d830 con 0x7f78dc105b10 2022-09-07T21:38:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.188+0000 7f78e11ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc19d9f0 con 0x7f78dc1031f0 2022-09-07T21:38:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.188+0000 7f78db59e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78dc105b10 0x7f78dc19d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:20.197 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.188+0000 7f78dad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78dc066be0 0x7f78dc195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:20.197 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.188+0000 7f78db59e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78dc105b10 0x7f78dc19d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:50388/0 (socket says 172.21.15.47:50388) 2022-09-07T21:38:20.197 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.188+0000 7f78db59e700 1 -- 172.21.15.47:0/3180095182 learned_addr learned my addr 172.21.15.47:0/3180095182 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:20.198 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.188+0000 7f78da59c700 1 --2- 172.21.15.47:0/3180095182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78dc1031f0 0x7f78dc196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:20.198 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.189+0000 7f78db59e700 1 -- 172.21.15.47:0/3180095182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78dc1031f0 msgr2=0x7f78dc196080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:20.198 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.189+0000 7f78db59e700 1 --2- 172.21.15.47:0/3180095182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78dc1031f0 0x7f78dc196080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:20.198 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.189+0000 7f78db59e700 1 -- 172.21.15.47:0/3180095182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78dc066be0 msgr2=0x7f78dc195b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:20.199 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.189+0000 7f78db59e700 1 --2- 172.21.15.47:0/3180095182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78dc066be0 0x7f78dc195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:20.199 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.189+0000 7f78db59e700 1 -- 172.21.15.47:0/3180095182 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d4009730 con 0x7f78dc105b10 2022-09-07T21:38:20.199 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.189+0000 7f78db59e700 1 --2- 172.21.15.47:0/3180095182 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78dc105b10 0x7f78dc19d0f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78d400fd80 tx=0x7f78d400f870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:20.199 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.189+0000 7f78bffff700 1 -- 172.21.15.47:0/3180095182 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d400b9e0 con 0x7f78dc105b10 2022-09-07T21:38:20.200 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.189+0000 7f78bffff700 1 -- 172.21.15.47:0/3180095182 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f78d4005040 con 0x7f78dc105b10 2022-09-07T21:38:20.200 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.189+0000 7f78e11ff700 1 -- 172.21.15.47:0/3180095182 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78dc19dc70 con 0x7f78dc105b10 2022-09-07T21:38:20.200 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.189+0000 7f78bffff700 1 -- 172.21.15.47:0/3180095182 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4017620 con 0x7f78dc105b10 2022-09-07T21:38:20.200 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.190+0000 7f78e11ff700 1 -- 172.21.15.47:0/3180095182 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78dc19e170 con 0x7f78dc105b10 2022-09-07T21:38:20.201 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.190+0000 7f78e11ff700 1 -- 172.21.15.47:0/3180095182 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78dc10b700 con 0x7f78dc105b10 2022-09-07T21:38:20.202 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.192+0000 7f78bffff700 1 -- 172.21.15.47:0/3180095182 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f78d4027020 con 0x7f78dc105b10 2022-09-07T21:38:20.202 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.194+0000 7f78bffff700 1 --2- 172.21.15.47:0/3180095182 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f78c4099080 0x7f78c409b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:20.202 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.194+0000 7f78bffff700 1 -- 172.21.15.47:0/3180095182 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f78d40c3e40 con 0x7f78dc105b10 2022-09-07T21:38:20.203 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.194+0000 7f78dad9d700 1 --2- 172.21.15.47:0/3180095182 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f78c4099080 0x7f78c409b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:20.203 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.195+0000 7f78dad9d700 1 --2- 172.21.15.47:0/3180095182 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f78c4099080 0x7f78c409b530 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f78dc066fb0 tx=0x7f78cc005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:20.203 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.196+0000 7f78bffff700 1 -- 172.21.15.47:0/3180095182 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f78d4090930 con 0x7f78dc105b10 2022-09-07T21:38:20.447 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.444+0000 7f78e11ff700 1 -- 172.21.15.47:0/3180095182 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78dc0f9cd0 con 0x7f78dc105b10 2022-09-07T21:38:20.448 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.445+0000 7f78bffff700 1 -- 172.21.15.47:0/3180095182 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5039 (secure 0 0 0) 0x7f78d4093f50 con 0x7f78dc105b10 2022-09-07T21:38:20.448 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:20.449 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/2559266585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2559266585},{"type":"v1","addr":"172.21.15.55:6835","nonce":2559266585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24526,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2445409240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2445409240},{"type":"v1","addr":"172.21.15.59:6835","nonce":2445409240}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2022-09-07T21:38:20.451 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.447+0000 7f78e11ff700 1 -- 172.21.15.47:0/3180095182 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f78c4099080 msgr2=0x7f78c409b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:20.451 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.447+0000 7f78e11ff700 1 --2- 172.21.15.47:0/3180095182 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f78c4099080 0x7f78c409b530 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f78dc066fb0 tx=0x7f78cc005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:20.451 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.447+0000 7f78e11ff700 1 -- 172.21.15.47:0/3180095182 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78dc105b10 msgr2=0x7f78dc19d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:20.452 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.447+0000 7f78e11ff700 1 --2- 172.21.15.47:0/3180095182 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78dc105b10 0x7f78dc19d0f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78d400fd80 tx=0x7f78d400f870 comp rx=0 tx=0).stop 2022-09-07T21:38:20.452 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.447+0000 7f78e11ff700 1 -- 172.21.15.47:0/3180095182 shutdown_connections 2022-09-07T21:38:20.452 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.447+0000 7f78e11ff700 1 --2- 172.21.15.47:0/3180095182 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f78c4099080 0x7f78c409b530 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:20.452 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.447+0000 7f78e11ff700 1 --2- 172.21.15.47:0/3180095182 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78dc105b10 0x7f78dc19d0f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:20.453 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.447+0000 7f78e11ff700 1 --2- 172.21.15.47:0/3180095182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78dc066be0 0x7f78dc195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:20.453 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.447+0000 7f78e11ff700 1 --2- 172.21.15.47:0/3180095182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78dc1031f0 0x7f78dc196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:20.453 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.447+0000 7f78e11ff700 1 -- 172.21.15.47:0/3180095182 >> 172.21.15.47:0/3180095182 conn(0x7f78dc0f82e0 msgr2=0x7f78dc103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:20.453 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.448+0000 7f78e11ff700 1 -- 172.21.15.47:0/3180095182 shutdown_connections 2022-09-07T21:38:20.453 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:20.448+0000 7f78e11ff700 1 -- 172.21.15.47:0/3180095182 wait complete. 2022-09-07T21:38:20.454 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 7 2022-09-07T21:38:20.532 INFO:journalctl@ceph.mds.c.smithi059.stdout:Sep 07 21:38:20 smithi059 bash[124703]: a66993f2186d8e92ef39de10ec955f8437bb816992e2bc5ae5c4298fce58fbdd 2022-09-07T21:38:20.532 INFO:journalctl@ceph.mds.c.smithi059.stdout:Sep 07 21:38:20 smithi059 ceph-mds[124742]: -- 172.21.15.59:0/3441588953 <== mon.1 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559c32192820 con 0x559c32166800 2022-09-07T21:38:20.533 INFO:journalctl@ceph.mds.c.smithi059.stdout:Sep 07 21:38:20 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-c[124719]: starting mds.c at 2022-09-07T21:38:20.533 INFO:journalctl@ceph.mds.c.smithi059.stdout:Sep 07 21:38:20 smithi059 systemd[1]: Started Ceph mds.c for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:38:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:21 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:21.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:21 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:21.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:21 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:21.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:21 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:21.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:21 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:21.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:21 smithi055 ceph-mon[97033]: Deploying daemon mds.h on smithi055 2022-09-07T21:38:21.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:21 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3180095182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:21.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:21 smithi055 ceph-mon[97033]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:21.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:21 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.59:6836/3542466465,v1:172.21.15.59:6837/3542466465] up:boot 2022-09-07T21:38:21.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:21 smithi055 ceph-mon[97033]: fsmap 7 up:standby 2022-09-07T21:38:21.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:21 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-09-07T21:38:21.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:21.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:21.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:21.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:21.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:21.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:21 smithi047 ceph-mon[90087]: Deploying daemon mds.h on smithi055 2022-09-07T21:38:21.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:21 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3180095182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:21.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:21 smithi047 ceph-mon[90087]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:21.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:21 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.59:6836/3542466465,v1:172.21.15.59:6837/3542466465] up:boot 2022-09-07T21:38:21.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:21 smithi047 ceph-mon[90087]: fsmap 7 up:standby 2022-09-07T21:38:21.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:21 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-09-07T21:38:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:21 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:21 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:21 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:21 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:21 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:21 smithi059 ceph-mon[97450]: Deploying daemon mds.h on smithi055 2022-09-07T21:38:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:21 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3180095182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:21 smithi059 ceph-mon[97450]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:21 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.59:6836/3542466465,v1:172.21.15.59:6837/3542466465] up:boot 2022-09-07T21:38:21.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:21 smithi059 ceph-mon[97450]: fsmap 7 up:standby 2022-09-07T21:38:21.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:21 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-09-07T21:38:21.980 INFO:journalctl@ceph.mds.h.smithi055.stdout:Sep 07 21:38:21 smithi055 systemd[1]: Starting Ceph mds.h for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:38:22.478 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:22.480 INFO:journalctl@ceph.mds.h.smithi055.stdout:Sep 07 21:38:22 smithi055 podman[123672]: 2022-09-07T21:38:22.727 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.723+0000 7ff4b09ff700 1 -- 172.21.15.47:0/1011089794 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4ac071650 msgr2=0x7ff4ac071ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:22.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.723+0000 7ff4b09ff700 1 --2- 172.21.15.47:0/1011089794 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4ac071650 0x7ff4ac071ac0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0009810 tx=0x7ff4a0009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:22.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.724+0000 7ff4b09ff700 1 -- 172.21.15.47:0/1011089794 shutdown_connections 2022-09-07T21:38:22.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.724+0000 7ff4b09ff700 1 --2- 172.21.15.47:0/1011089794 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4ac105c20 0x7ff4ac108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:22.729 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.724+0000 7ff4b09ff700 1 --2- 172.21.15.47:0/1011089794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4ac072c30 0x7ff4ac071110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:22.729 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.724+0000 7ff4b09ff700 1 --2- 172.21.15.47:0/1011089794 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4ac071650 0x7ff4ac071ac0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:22.729 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.724+0000 7ff4b09ff700 1 -- 172.21.15.47:0/1011089794 >> 172.21.15.47:0/1011089794 conn(0x7ff4ac0f9460 msgr2=0x7ff4ac0fb870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:22.729 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.724+0000 7ff4b09ff700 1 -- 172.21.15.47:0/1011089794 shutdown_connections 2022-09-07T21:38:22.730 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.724+0000 7ff4b09ff700 1 -- 172.21.15.47:0/1011089794 wait complete. 2022-09-07T21:38:22.732 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.725+0000 7ff4b09ff700 1 Processor -- start 2022-09-07T21:38:22.732 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.725+0000 7ff4b09ff700 1 -- start start 2022-09-07T21:38:22.732 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.725+0000 7ff4b09ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4ac071650 0x7ff4ac195b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:22.732 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.725+0000 7ff4b09ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4ac072c30 0x7ff4ac196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:22.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.725+0000 7ff4b09ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4ac105c20 0x7ff4ac19d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:22.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.725+0000 7ff4b09ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac19d680 con 0x7ff4ac071650 2022-09-07T21:38:22.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.725+0000 7ff4b09ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac19d7f0 con 0x7ff4ac072c30 2022-09-07T21:38:22.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.725+0000 7ff4b09ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac19d9b0 con 0x7ff4ac105c20 2022-09-07T21:38:22.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.726+0000 7ff4a9d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4ac072c30 0x7ff4ac196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:22.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.726+0000 7ff4a9d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4ac072c30 0x7ff4ac196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:50418/0 (socket says 172.21.15.47:50418) 2022-09-07T21:38:22.735 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.726+0000 7ff4aa59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4ac071650 0x7ff4ac195b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:22.735 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.726+0000 7ff4a9d9b700 1 -- 172.21.15.47:0/3567363513 learned_addr learned my addr 172.21.15.47:0/3567363513 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:22.735 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.726+0000 7ff4aad9d700 1 --2- 172.21.15.47:0/3567363513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4ac105c20 0x7ff4ac19d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:22.735 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.726+0000 7ff4aad9d700 1 -- 172.21.15.47:0/3567363513 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4ac072c30 msgr2=0x7ff4ac196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:22.736 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.726+0000 7ff4aad9d700 1 --2- 172.21.15.47:0/3567363513 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4ac072c30 0x7ff4ac196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:22.736 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.726+0000 7ff4aad9d700 1 -- 172.21.15.47:0/3567363513 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4ac071650 msgr2=0x7ff4ac195b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:22.736 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.726+0000 7ff4aad9d700 1 --2- 172.21.15.47:0/3567363513 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4ac071650 0x7ff4ac195b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:22.736 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.726+0000 7ff4aad9d700 1 -- 172.21.15.47:0/3567363513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a0009470 con 0x7ff4ac105c20 2022-09-07T21:38:22.736 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.726+0000 7ff4aad9d700 1 --2- 172.21.15.47:0/3567363513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4ac105c20 0x7ff4ac19d0b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400ea80 tx=0x7ff4a400ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:22.737 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.727+0000 7ff49b7fe700 1 -- 172.21.15.47:0/3567363513 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a40098c0 con 0x7ff4ac105c20 2022-09-07T21:38:22.737 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.727+0000 7ff49b7fe700 1 -- 172.21.15.47:0/3567363513 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff4a400ce10 con 0x7ff4ac105c20 2022-09-07T21:38:22.737 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.727+0000 7ff4b09ff700 1 -- 172.21.15.47:0/3567363513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ac19dc90 con 0x7ff4ac105c20 2022-09-07T21:38:22.737 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.727+0000 7ff49b7fe700 1 -- 172.21.15.47:0/3567363513 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4004280 con 0x7ff4ac105c20 2022-09-07T21:38:22.738 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.727+0000 7ff4b09ff700 1 -- 172.21.15.47:0/3567363513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ac19e150 con 0x7ff4ac105c20 2022-09-07T21:38:22.738 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.728+0000 7ff4b09ff700 1 -- 172.21.15.47:0/3567363513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ac10b6e0 con 0x7ff4ac105c20 2022-09-07T21:38:22.739 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.730+0000 7ff49b7fe700 1 -- 172.21.15.47:0/3567363513 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7ff4a400c040 con 0x7ff4ac105c20 2022-09-07T21:38:22.740 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.731+0000 7ff49b7fe700 1 --2- 172.21.15.47:0/3567363513 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff494099000 0x7ff49409b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:22.740 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.731+0000 7ff49b7fe700 1 -- 172.21.15.47:0/3567363513 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7ff4a4014070 con 0x7ff4ac105c20 2022-09-07T21:38:22.740 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.732+0000 7ff4aa59c700 1 --2- 172.21.15.47:0/3567363513 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff494099000 0x7ff49409b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.732+0000 7ff4aa59c700 1 --2- 172.21.15.47:0/3567363513 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff494099000 0x7ff49409b4b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff49c005fd0 tx=0x7ff49c005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.734+0000 7ff49b7fe700 1 -- 172.21.15.47:0/3567363513 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff4a408f320 con 0x7ff4ac105c20 2022-09-07T21:38:22.980 INFO:journalctl@ceph.mds.h.smithi055.stdout:Sep 07 21:38:22 smithi055 ceph-mds[123711]: -- 172.21.15.55:0/1654108891 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d839870820 con 0x55d839844800 2022-09-07T21:38:22.981 INFO:journalctl@ceph.mds.h.smithi055.stdout:Sep 07 21:38:22 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-h[123688]: starting mds.h at 2022-09-07T21:38:22.981 INFO:journalctl@ceph.mds.h.smithi055.stdout:Sep 07 21:38:22 smithi055 bash[123672]: 16db67a062997a215420aa68461c264b4c12c98586d19ab9fbfc6aad8c5e78ab 2022-09-07T21:38:22.981 INFO:journalctl@ceph.mds.h.smithi055.stdout:Sep 07 21:38:22 smithi055 systemd[1]: Started Ceph mds.h for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:38:22.989 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.986+0000 7ff4b09ff700 1 -- 172.21.15.47:0/3567363513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4ac104230 con 0x7ff4ac105c20 2022-09-07T21:38:22.990 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.987+0000 7ff49b7fe700 1 -- 172.21.15.47:0/3567363513 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6553 (secure 0 0 0) 0x7ff4a4092940 con 0x7ff4ac105c20 2022-09-07T21:38:22.990 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:22.991 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/2559266585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2559266585},{"type":"v1","addr":"172.21.15.55:6835","nonce":2559266585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/2090942297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2090942297},{"type":"v1","addr":"172.21.15.55:6837","nonce":2090942297}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24526,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2445409240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2445409240},{"type":"v1","addr":"172.21.15.59:6835","nonce":2445409240}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/3542466465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":3542466465},{"type":"v1","addr":"172.21.15.59:6837","nonce":3542466465}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2022-09-07T21:38:22.993 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.989+0000 7ff4b09ff700 1 -- 172.21.15.47:0/3567363513 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff494099000 msgr2=0x7ff49409b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:22.993 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.989+0000 7ff4b09ff700 1 --2- 172.21.15.47:0/3567363513 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff494099000 0x7ff49409b4b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff49c005fd0 tx=0x7ff49c005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.989+0000 7ff4b09ff700 1 -- 172.21.15.47:0/3567363513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4ac105c20 msgr2=0x7ff4ac19d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.989+0000 7ff4b09ff700 1 --2- 172.21.15.47:0/3567363513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4ac105c20 0x7ff4ac19d0b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400ea80 tx=0x7ff4a400ed90 comp rx=0 tx=0).stop 2022-09-07T21:38:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.989+0000 7ff4b09ff700 1 -- 172.21.15.47:0/3567363513 shutdown_connections 2022-09-07T21:38:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.989+0000 7ff4b09ff700 1 --2- 172.21.15.47:0/3567363513 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff494099000 0x7ff49409b4b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:22.995 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.989+0000 7ff4b09ff700 1 --2- 172.21.15.47:0/3567363513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4ac105c20 0x7ff4ac19d0b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:22.995 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.989+0000 7ff4b09ff700 1 --2- 172.21.15.47:0/3567363513 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4ac071650 0x7ff4ac195b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:22.995 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.989+0000 7ff4b09ff700 1 --2- 172.21.15.47:0/3567363513 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff4ac072c30 0x7ff4ac196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:22.995 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.989+0000 7ff4b09ff700 1 -- 172.21.15.47:0/3567363513 >> 172.21.15.47:0/3567363513 conn(0x7ff4ac0f9460 msgr2=0x7ff4ac0fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:22.996 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.989+0000 7ff4b09ff700 1 -- 172.21.15.47:0/3567363513 shutdown_connections 2022-09-07T21:38:22.996 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:22.989+0000 7ff4b09ff700 1 -- 172.21.15.47:0/3567363513 wait complete. 2022-09-07T21:38:22.996 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 9 2022-09-07T21:38:23.672 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:23 smithi059 ceph-mon[97450]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:23.672 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:23 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.55:6836/2090942297,v1:172.21.15.55:6837/2090942297] up:boot 2022-09-07T21:38:23.672 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:23 smithi059 ceph-mon[97450]: fsmap 8 up:standby 2022-09-07T21:38:23.672 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:23 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-09-07T21:38:23.673 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:23 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:23.673 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:23 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:23.673 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:23 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:23.673 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:23 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:23.674 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:23 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:23.674 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:23 smithi059 ceph-mon[97450]: Deploying daemon mds.i on smithi059 2022-09-07T21:38:23.674 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:23 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3567363513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:23.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:23 smithi047 ceph-mon[90087]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:23.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:23 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.55:6836/2090942297,v1:172.21.15.55:6837/2090942297] up:boot 2022-09-07T21:38:23.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:23 smithi047 ceph-mon[90087]: fsmap 8 up:standby 2022-09-07T21:38:23.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-09-07T21:38:23.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:23.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:23.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:23.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:23.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:23 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:23.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:23 smithi047 ceph-mon[90087]: Deploying daemon mds.i on smithi059 2022-09-07T21:38:23.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:23 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3567363513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:23 smithi055 ceph-mon[97033]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:23.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:23 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.55:6836/2090942297,v1:172.21.15.55:6837/2090942297] up:boot 2022-09-07T21:38:23.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:23 smithi055 ceph-mon[97033]: fsmap 8 up:standby 2022-09-07T21:38:23.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:23 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-09-07T21:38:23.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:23 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:23.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:23 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:23.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:23 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:23.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:23 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:23.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:23 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:23.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:23 smithi055 ceph-mon[97033]: Deploying daemon mds.i on smithi059 2022-09-07T21:38:23.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:23 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3567363513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:24.282 INFO:journalctl@ceph.mds.i.smithi059.stdout:Sep 07 21:38:24 smithi059 systemd[1]: Starting Ceph mds.i for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:38:24.681 INFO:journalctl@ceph.mds.i.smithi059.stdout:Sep 07 21:38:24 smithi059 podman[125048]: 2022-09-07T21:38:25.021 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:25.031 INFO:journalctl@ceph.mds.i.smithi059.stdout:Sep 07 21:38:24 smithi059 ceph-mds[125087]: -- 172.21.15.59:0/953876432 <== mon.1 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5593cb9da820 con 0x5593cb9af000 2022-09-07T21:38:25.032 INFO:journalctl@ceph.mds.i.smithi059.stdout:Sep 07 21:38:24 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-i[125064]: starting mds.i at 2022-09-07T21:38:25.032 INFO:journalctl@ceph.mds.i.smithi059.stdout:Sep 07 21:38:24 smithi059 bash[125048]: 85e22203902f2cd4f4df2b303f0e233416552086c0dfbea9403f8666e11e539f 2022-09-07T21:38:25.033 INFO:journalctl@ceph.mds.i.smithi059.stdout:Sep 07 21:38:24 smithi059 systemd[1]: Started Ceph mds.i for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:38:25.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.281+0000 7efdf0fff700 1 -- 172.21.15.47:0/3710666125 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efdec0fdd50 msgr2=0x7efdec0fe160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:25.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.281+0000 7efdf0fff700 1 --2- 172.21.15.47:0/3710666125 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efdec0fdd50 0x7efdec0fe160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efddc009750 tx=0x7efddc009a60 comp rx=0 tx=0).stop 2022-09-07T21:38:25.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.281+0000 7efdf0fff700 1 -- 172.21.15.47:0/3710666125 shutdown_connections 2022-09-07T21:38:25.286 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.281+0000 7efdf0fff700 1 --2- 172.21.15.47:0/3710666125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdec0ff960 0x7efdec106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:25.286 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.281+0000 7efdf0fff700 1 --2- 172.21.15.47:0/3710666125 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efdec0fdd50 0x7efdec0fe160 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:25.286 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.281+0000 7efdf0fff700 1 --2- 172.21.15.47:0/3710666125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdec0fefd0 0x7efdec0ff420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:25.287 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.281+0000 7efdf0fff700 1 -- 172.21.15.47:0/3710666125 >> 172.21.15.47:0/3710666125 conn(0x7efdec0f9460 msgr2=0x7efdec0fb870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:25.287 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.281+0000 7efdf0fff700 1 -- 172.21.15.47:0/3710666125 shutdown_connections 2022-09-07T21:38:25.287 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.281+0000 7efdf0fff700 1 -- 172.21.15.47:0/3710666125 wait complete. 2022-09-07T21:38:25.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.283+0000 7efdf0fff700 1 Processor -- start 2022-09-07T21:38:25.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.283+0000 7efdf0fff700 1 -- start start 2022-09-07T21:38:25.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.283+0000 7efdf0fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efdec0fdd50 0x7efdec195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:25.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.283+0000 7efdf0fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdec0fefd0 0x7efdec1960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:25.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.283+0000 7efdf0fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdec0ff960 0x7efdec19d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:25.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.283+0000 7efdf0fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec19d6f0 con 0x7efdec0fefd0 2022-09-07T21:38:25.292 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.283+0000 7efdf0fff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec19d860 con 0x7efdec0fdd50 2022-09-07T21:38:25.292 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.283+0000 7efdf0fff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec19da20 con 0x7efdec0ff960 2022-09-07T21:38:25.292 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.284+0000 7efde9d9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdec0fefd0 0x7efdec1960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:25.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.284+0000 7efdea59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efdec0fdd50 0x7efdec195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:25.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.284+0000 7efde9d9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdec0fefd0 0x7efdec1960b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60532/0 (socket says 172.21.15.47:60532) 2022-09-07T21:38:25.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.284+0000 7efde9d9b700 1 -- 172.21.15.47:0/1346819928 learned_addr learned my addr 172.21.15.47:0/1346819928 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:25.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.284+0000 7efdead9d700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdec0ff960 0x7efdec19d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:25.294 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.284+0000 7efdead9d700 1 -- 172.21.15.47:0/1346819928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efdec0fdd50 msgr2=0x7efdec195b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:25.294 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.284+0000 7efdead9d700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efdec0fdd50 0x7efdec195b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:25.294 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.284+0000 7efdead9d700 1 -- 172.21.15.47:0/1346819928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdec0fefd0 msgr2=0x7efdec1960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:25.295 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.284+0000 7efdead9d700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdec0fefd0 0x7efdec1960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:25.295 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.284+0000 7efdead9d700 1 -- 172.21.15.47:0/1346819928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efddc0093b0 con 0x7efdec0ff960 2022-09-07T21:38:25.295 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.284+0000 7efde9d9b700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdec0fefd0 0x7efdec1960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:25.295 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.284+0000 7efdea59c700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efdec0fdd50 0x7efdec195b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:25.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.284+0000 7efdead9d700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdec0ff960 0x7efdec19d120 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efde400ea80 tx=0x7efde400ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:25.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.285+0000 7efddb7fe700 1 -- 172.21.15.47:0/1346819928 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde40098c0 con 0x7efdec0ff960 2022-09-07T21:38:25.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.285+0000 7efdf0fff700 1 -- 172.21.15.47:0/1346819928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdec19dd00 con 0x7efdec0ff960 2022-09-07T21:38:25.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.285+0000 7efddb7fe700 1 -- 172.21.15.47:0/1346819928 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7efde400ce10 con 0x7efdec0ff960 2022-09-07T21:38:25.297 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.285+0000 7efddb7fe700 1 -- 172.21.15.47:0/1346819928 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde4004280 con 0x7efdec0ff960 2022-09-07T21:38:25.297 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.285+0000 7efdf0fff700 1 -- 172.21.15.47:0/1346819928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdec19e1c0 con 0x7efdec0ff960 2022-09-07T21:38:25.297 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.286+0000 7efdf0fff700 1 -- 172.21.15.47:0/1346819928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdec109500 con 0x7efdec0ff960 2022-09-07T21:38:25.299 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.288+0000 7efddb7fe700 1 -- 172.21.15.47:0/1346819928 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7efde400c040 con 0x7efdec0ff960 2022-09-07T21:38:25.299 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.289+0000 7efddb7fe700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7efdd4099000 0x7efdd409b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:25.299 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.289+0000 7efddb7fe700 1 -- 172.21.15.47:0/1346819928 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7efde4014070 con 0x7efdec0ff960 2022-09-07T21:38:25.300 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.289+0000 7efdea59c700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7efdd4099000 0x7efdd409b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:25.300 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.290+0000 7efdea59c700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7efdd4099000 0x7efdd409b4b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efddc00f460 tx=0x7efddc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:25.301 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.293+0000 7efddb7fe700 1 -- 172.21.15.47:0/1346819928 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7efde408ec50 con 0x7efdec0ff960 2022-09-07T21:38:25.561 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.558+0000 7efdf0fff700 1 -- 172.21.15.47:0/1346819928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdec19edf0 con 0x7efdec0ff960 2022-09-07T21:38:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.559+0000 7efddb7fe700 1 -- 172.21.15.47:0/1346819928 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6553 (secure 0 0 0) 0x7efde4092270 con 0x7efdec0ff960 2022-09-07T21:38:25.563 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:25.564 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/2559266585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2559266585},{"type":"v1","addr":"172.21.15.55:6835","nonce":2559266585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/2090942297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2090942297},{"type":"v1","addr":"172.21.15.55:6837","nonce":2090942297}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24526,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2445409240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2445409240},{"type":"v1","addr":"172.21.15.59:6835","nonce":2445409240}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/3542466465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":3542466465},{"type":"v1","addr":"172.21.15.59:6837","nonce":3542466465}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2022-09-07T21:38:25.566 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.561+0000 7efdf0fff700 1 -- 172.21.15.47:0/1346819928 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7efdd4099000 msgr2=0x7efdd409b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.561+0000 7efdf0fff700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7efdd4099000 0x7efdd409b4b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efddc00f460 tx=0x7efddc005f90 comp rx=0 tx=0).stop 2022-09-07T21:38:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.561+0000 7efdf0fff700 1 -- 172.21.15.47:0/1346819928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdec0ff960 msgr2=0x7efdec19d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:25.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.561+0000 7efdf0fff700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdec0ff960 0x7efdec19d120 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efde400ea80 tx=0x7efde400ed90 comp rx=0 tx=0).stop 2022-09-07T21:38:25.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.562+0000 7efdf0fff700 1 -- 172.21.15.47:0/1346819928 shutdown_connections 2022-09-07T21:38:25.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.562+0000 7efdf0fff700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7efdd4099000 0x7efdd409b4b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:25.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.562+0000 7efdf0fff700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdec0ff960 0x7efdec19d120 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:25.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.562+0000 7efdf0fff700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efdec0fdd50 0x7efdec195b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:25.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.562+0000 7efdf0fff700 1 --2- 172.21.15.47:0/1346819928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdec0fefd0 0x7efdec1960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:25.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.562+0000 7efdf0fff700 1 -- 172.21.15.47:0/1346819928 >> 172.21.15.47:0/1346819928 conn(0x7efdec0f9460 msgr2=0x7efdec0fb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:25.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.562+0000 7efdf0fff700 1 -- 172.21.15.47:0/1346819928 shutdown_connections 2022-09-07T21:38:25.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:25.562+0000 7efdf0fff700 1 -- 172.21.15.47:0/1346819928 wait complete. 2022-09-07T21:38:25.571 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 9 2022-09-07T21:38:25.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:25 smithi047 ceph-mon[90087]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:25.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:25.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:25.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:25.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:25.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:25.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:25 smithi047 ceph-mon[90087]: Deploying daemon mds.g on smithi047 2022-09-07T21:38:25.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:25 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1346819928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:25.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:25 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] up:boot 2022-09-07T21:38:25.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:25 smithi047 ceph-mon[90087]: fsmap 9 up:standby 2022-09-07T21:38:25.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:25 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-09-07T21:38:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:25 smithi059 ceph-mon[97450]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:26.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:26.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:26.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:25 smithi059 ceph-mon[97450]: Deploying daemon mds.g on smithi047 2022-09-07T21:38:26.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:25 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1346819928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:26.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:25 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] up:boot 2022-09-07T21:38:26.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:25 smithi059 ceph-mon[97450]: fsmap 9 up:standby 2022-09-07T21:38:26.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:25 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-09-07T21:38:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:25 smithi055 ceph-mon[97033]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:26.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:26.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:26.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:26.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:26.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:25 smithi055 ceph-mon[97033]: Deploying daemon mds.g on smithi047 2022-09-07T21:38:26.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:25 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1346819928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:26.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:25 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] up:boot 2022-09-07T21:38:26.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:25 smithi055 ceph-mon[97033]: fsmap 9 up:standby 2022-09-07T21:38:26.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:25 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-09-07T21:38:27.597 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:27.860 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.856+0000 7eff62dff700 1 -- 172.21.15.47:0/4258270681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff5c0febe0 msgr2=0x7eff5c101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:27.861 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.856+0000 7eff62dff700 1 --2- 172.21.15.47:0/4258270681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff5c0febe0 0x7eff5c101040 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7eff48009810 tx=0x7eff48009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:27.861 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.857+0000 7eff62dff700 1 -- 172.21.15.47:0/4258270681 shutdown_connections 2022-09-07T21:38:27.861 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.857+0000 7eff62dff700 1 --2- 172.21.15.47:0/4258270681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff5c1016d0 0x7eff5c101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:27.861 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.857+0000 7eff62dff700 1 --2- 172.21.15.47:0/4258270681 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5c066be0 0x7eff5c0fe6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:27.861 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.857+0000 7eff62dff700 1 --2- 172.21.15.47:0/4258270681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff5c0febe0 0x7eff5c101040 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:27.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.857+0000 7eff62dff700 1 -- 172.21.15.47:0/4258270681 >> 172.21.15.47:0/4258270681 conn(0x7eff5c0f82e0 msgr2=0x7eff5c0fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:27.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.857+0000 7eff62dff700 1 -- 172.21.15.47:0/4258270681 shutdown_connections 2022-09-07T21:38:27.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.857+0000 7eff62dff700 1 -- 172.21.15.47:0/4258270681 wait complete. 2022-09-07T21:38:27.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.858+0000 7eff62dff700 1 Processor -- start 2022-09-07T21:38:27.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.858+0000 7eff62dff700 1 -- start start 2022-09-07T21:38:27.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.858+0000 7eff62dff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff5c066be0 0x7eff5c199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:27.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.858+0000 7eff62dff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5c0febe0 0x7eff5c19a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:27.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.859+0000 7eff62dff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff5c1016d0 0x7eff5c1a14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:27.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.859+0000 7eff62dff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c1a1a70 con 0x7eff5c0febe0 2022-09-07T21:38:27.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.859+0000 7eff62dff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c1a1be0 con 0x7eff5c066be0 2022-09-07T21:38:27.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.859+0000 7eff62dff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c1a1da0 con 0x7eff5c1016d0 2022-09-07T21:38:27.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.859+0000 7eff60b9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff5c066be0 0x7eff5c199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:27.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.859+0000 7eff60b9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff5c066be0 0x7eff5c199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:50460/0 (socket says 172.21.15.47:50460) 2022-09-07T21:38:27.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.859+0000 7eff60b9b700 1 -- 172.21.15.47:0/19685755 learned_addr learned my addr 172.21.15.47:0/19685755 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:27.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.859+0000 7eff5bfff700 1 --2- 172.21.15.47:0/19685755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5c0febe0 0x7eff5c19a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:27.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.859+0000 7eff6139c700 1 --2- 172.21.15.47:0/19685755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff5c1016d0 0x7eff5c1a14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:27.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.859+0000 7eff60b9b700 1 -- 172.21.15.47:0/19685755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff5c1016d0 msgr2=0x7eff5c1a14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:27.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.859+0000 7eff60b9b700 1 --2- 172.21.15.47:0/19685755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff5c1016d0 0x7eff5c1a14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:27.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.859+0000 7eff60b9b700 1 -- 172.21.15.47:0/19685755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5c0febe0 msgr2=0x7eff5c19a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:27.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.859+0000 7eff60b9b700 1 --2- 172.21.15.47:0/19685755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5c0febe0 0x7eff5c19a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:27.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.859+0000 7eff60b9b700 1 -- 172.21.15.47:0/19685755 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff48009470 con 0x7eff5c066be0 2022-09-07T21:38:27.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.860+0000 7eff60b9b700 1 --2- 172.21.15.47:0/19685755 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff5c066be0 0x7eff5c199ef0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7eff5000e6e0 tx=0x7eff5000e9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:27.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.860+0000 7eff59ffb700 1 -- 172.21.15.47:0/19685755 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50009500 con 0x7eff5c066be0 2022-09-07T21:38:27.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.860+0000 7eff62dff700 1 -- 172.21.15.47:0/19685755 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff5c1a2080 con 0x7eff5c066be0 2022-09-07T21:38:27.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.860+0000 7eff59ffb700 1 -- 172.21.15.47:0/19685755 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7eff50010da0 con 0x7eff5c066be0 2022-09-07T21:38:27.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.860+0000 7eff59ffb700 1 -- 172.21.15.47:0/19685755 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff500173d0 con 0x7eff5c066be0 2022-09-07T21:38:27.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.860+0000 7eff62dff700 1 -- 172.21.15.47:0/19685755 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff5c1a25e0 con 0x7eff5c066be0 2022-09-07T21:38:27.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.861+0000 7eff62dff700 1 -- 172.21.15.47:0/19685755 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff5c10faf0 con 0x7eff5c066be0 2022-09-07T21:38:27.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.863+0000 7eff59ffb700 1 -- 172.21.15.47:0/19685755 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7eff5000c040 con 0x7eff5c066be0 2022-09-07T21:38:27.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.865+0000 7eff59ffb700 1 --2- 172.21.15.47:0/19685755 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7eff4c099080 0x7eff4c09b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:27.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.865+0000 7eff59ffb700 1 -- 172.21.15.47:0/19685755 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7eff50014070 con 0x7eff5c066be0 2022-09-07T21:38:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.865+0000 7eff5bfff700 1 --2- 172.21.15.47:0/19685755 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7eff4c099080 0x7eff4c09b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.865+0000 7eff5bfff700 1 --2- 172.21.15.47:0/19685755 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7eff4c099080 0x7eff4c09b530 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7eff4800b4f0 tx=0x7eff48009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:27.867+0000 7eff59ffb700 1 -- 172.21.15.47:0/19685755 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7eff5008eb30 con 0x7eff5c066be0 2022-09-07T21:38:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:27 smithi059 ceph-mon[97450]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:28.132 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:28.129+0000 7eff62dff700 1 -- 172.21.15.47:0/19685755 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff5c0f9cd0 con 0x7eff5c066be0 2022-09-07T21:38:28.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:28.129+0000 7eff59ffb700 1 -- 172.21.15.47:0/19685755 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7312 (secure 0 0 0) 0x7eff50092150 con 0x7eff5c066be0 2022-09-07T21:38:28.134 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:28.134 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/2559266585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2559266585},{"type":"v1","addr":"172.21.15.55:6835","nonce":2559266585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/2090942297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2090942297},{"type":"v1","addr":"172.21.15.55:6837","nonce":2090942297}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24526,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2445409240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2445409240},{"type":"v1","addr":"172.21.15.59:6835","nonce":2445409240}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/3542466465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":3542466465},{"type":"v1","addr":"172.21.15.59:6837","nonce":3542466465}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6839/1006724740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":1006724740},{"type":"v1","addr":"172.21.15.59:6839","nonce":1006724740}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2022-09-07T21:38:28.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:28.132+0000 7eff62dff700 1 -- 172.21.15.47:0/19685755 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7eff4c099080 msgr2=0x7eff4c09b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:28.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:28.132+0000 7eff62dff700 1 --2- 172.21.15.47:0/19685755 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7eff4c099080 0x7eff4c09b530 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7eff4800b4f0 tx=0x7eff48009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:28.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:28.132+0000 7eff62dff700 1 -- 172.21.15.47:0/19685755 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff5c066be0 msgr2=0x7eff5c199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:28.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:28.132+0000 7eff62dff700 1 --2- 172.21.15.47:0/19685755 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff5c066be0 0x7eff5c199ef0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7eff5000e6e0 tx=0x7eff5000e9f0 comp rx=0 tx=0).stop 2022-09-07T21:38:28.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:28.132+0000 7eff62dff700 1 -- 172.21.15.47:0/19685755 shutdown_connections 2022-09-07T21:38:28.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:28.132+0000 7eff62dff700 1 --2- 172.21.15.47:0/19685755 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7eff4c099080 0x7eff4c09b530 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:28.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:28.132+0000 7eff62dff700 1 --2- 172.21.15.47:0/19685755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff5c1016d0 0x7eff5c1a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:28.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:28.132+0000 7eff62dff700 1 --2- 172.21.15.47:0/19685755 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff5c066be0 0x7eff5c199ef0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:28.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:28.132+0000 7eff62dff700 1 --2- 172.21.15.47:0/19685755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5c0febe0 0x7eff5c19a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:28.138 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:28.132+0000 7eff62dff700 1 -- 172.21.15.47:0/19685755 >> 172.21.15.47:0/19685755 conn(0x7eff5c0f82e0 msgr2=0x7eff5c0fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:28.138 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:28.132+0000 7eff62dff700 1 -- 172.21.15.47:0/19685755 shutdown_connections 2022-09-07T21:38:28.138 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:28.133+0000 7eff62dff700 1 -- 172.21.15.47:0/19685755 wait complete. 2022-09-07T21:38:28.139 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 10 2022-09-07T21:38:28.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:27 smithi047 ceph-mon[90087]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:28.154 INFO:journalctl@ceph.mds.g.smithi047.stdout:Sep 07 21:38:27 smithi047 systemd[1]: Starting Ceph mds.g for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:38:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:27 smithi055 ceph-mon[97033]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:28.442 INFO:journalctl@ceph.mds.g.smithi047.stdout:Sep 07 21:38:28 smithi047 podman[125087]: 2022-09-07T21:38:28.825 INFO:journalctl@ceph.mds.g.smithi047.stdout:Sep 07 21:38:28 smithi047 bash[125087]: 3eb8ecb25111ca18d2b15483385e4e407d537901eef4a206d45b1f4b6179ce76 2022-09-07T21:38:28.825 INFO:journalctl@ceph.mds.g.smithi047.stdout:Sep 07 21:38:28 smithi047 ceph-mds[125129]: -- 172.21.15.47:0/1970436270 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55afad022820 con 0x55afacff6c00 2022-09-07T21:38:28.826 INFO:journalctl@ceph.mds.g.smithi047.stdout:Sep 07 21:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-g[125105]: starting mds.g at 2022-09-07T21:38:28.826 INFO:journalctl@ceph.mds.g.smithi047.stdout:Sep 07 21:38:28 smithi047 systemd[1]: Started Ceph mds.g for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:38:29.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:28 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/19685755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:29.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:28 smithi047 ceph-mon[90087]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:29.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:28 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:29.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:28 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:29.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:28 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:29.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:28 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:29.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:28 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:29.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:28 smithi047 ceph-mon[90087]: Deploying daemon mds.a on smithi047 2022-09-07T21:38:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:28 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/19685755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:28 smithi055 ceph-mon[97033]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:28 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:29.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:28 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:29.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:28 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:29.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:28 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:29.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:28 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:29.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:28 smithi055 ceph-mon[97033]: Deploying daemon mds.a on smithi047 2022-09-07T21:38:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:28 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/19685755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:28 smithi059 ceph-mon[97450]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:28 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:28 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:29.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:28 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:29.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:28 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:29.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:28 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:29.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:28 smithi059 ceph-mon[97450]: Deploying daemon mds.a on smithi047 2022-09-07T21:38:30.164 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:29 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.47:6838/315505812,v1:172.21.15.47:6839/315505812] up:boot 2022-09-07T21:38:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:29 smithi055 ceph-mon[97033]: fsmap 10 up:standby 2022-09-07T21:38:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:29 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-07T21:38:30.252 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:29 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.47:6838/315505812,v1:172.21.15.47:6839/315505812] up:boot 2022-09-07T21:38:30.252 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:29 smithi047 ceph-mon[90087]: fsmap 10 up:standby 2022-09-07T21:38:30.253 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:29 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-07T21:38:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:29 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.47:6838/315505812,v1:172.21.15.47:6839/315505812] up:boot 2022-09-07T21:38:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:29 smithi059 ceph-mon[97450]: fsmap 10 up:standby 2022-09-07T21:38:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:29 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-07T21:38:30.416 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.412+0000 7ff5763ff700 1 -- 172.21.15.47:0/3533171686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff570100df0 msgr2=0x7ff570103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:30.416 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.412+0000 7ff5763ff700 1 --2- 172.21.15.47:0/3533171686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff570100df0 0x7ff570103250 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff564009810 tx=0x7ff564009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:30.416 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.412+0000 7ff5763ff700 1 -- 172.21.15.47:0/3533171686 shutdown_connections 2022-09-07T21:38:30.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.412+0000 7ff5763ff700 1 --2- 172.21.15.47:0/3533171686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff5701038e0 0x7ff570105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:30.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.412+0000 7ff5763ff700 1 --2- 172.21.15.47:0/3533171686 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff570066be0 0x7ff5701008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:30.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.412+0000 7ff5763ff700 1 --2- 172.21.15.47:0/3533171686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff570100df0 0x7ff570103250 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:30.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.412+0000 7ff5763ff700 1 -- 172.21.15.47:0/3533171686 >> 172.21.15.47:0/3533171686 conn(0x7ff5700f82e0 msgr2=0x7ff5700fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:30.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.412+0000 7ff5763ff700 1 -- 172.21.15.47:0/3533171686 shutdown_connections 2022-09-07T21:38:30.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.412+0000 7ff5763ff700 1 -- 172.21.15.47:0/3533171686 wait complete. 2022-09-07T21:38:30.420 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.414+0000 7ff5763ff700 1 Processor -- start 2022-09-07T21:38:30.420 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.414+0000 7ff5763ff700 1 -- start start 2022-09-07T21:38:30.421 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.414+0000 7ff5763ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff570066be0 0x7ff5700fe1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:30.421 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff5763ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff570100df0 0x7ff5700fc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:30.421 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff5763ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5701038e0 0x7ff5700fcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:30.421 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff5763ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5700fd550 con 0x7ff570066be0 2022-09-07T21:38:30.422 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff5763ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5700fd6c0 con 0x7ff5701038e0 2022-09-07T21:38:30.422 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff5763ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57019da20 con 0x7ff570100df0 2022-09-07T21:38:30.422 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff56ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff570066be0 0x7ff5700fe1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:30.422 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff56ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff570066be0 0x7ff5700fe1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45082/0 (socket says 172.21.15.47:45082) 2022-09-07T21:38:30.423 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff56ffff700 1 -- 172.21.15.47:0/221480516 learned_addr learned my addr 172.21.15.47:0/221480516 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:30.423 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff56f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff570100df0 0x7ff5700fc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:30.423 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff57499c700 1 --2- 172.21.15.47:0/221480516 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5701038e0 0x7ff5700fcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:30.423 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff56ffff700 1 -- 172.21.15.47:0/221480516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff570100df0 msgr2=0x7ff5700fc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:30.424 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff56ffff700 1 --2- 172.21.15.47:0/221480516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff570100df0 0x7ff5700fc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:30.424 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff56ffff700 1 -- 172.21.15.47:0/221480516 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5701038e0 msgr2=0x7ff5700fcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:30.424 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff56ffff700 1 --2- 172.21.15.47:0/221480516 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5701038e0 0x7ff5700fcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:30.424 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff56ffff700 1 -- 172.21.15.47:0/221480516 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff564009470 con 0x7ff570066be0 2022-09-07T21:38:30.425 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.415+0000 7ff57499c700 1 --2- 172.21.15.47:0/221480516 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5701038e0 0x7ff5700fcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:30.425 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.416+0000 7ff56ffff700 1 --2- 172.21.15.47:0/221480516 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff570066be0 0x7ff5700fe1c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff56000e6e0 tx=0x7ff56000e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:30.425 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.416+0000 7ff56d7fa700 1 -- 172.21.15.47:0/221480516 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff560009500 con 0x7ff570066be0 2022-09-07T21:38:30.426 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.416+0000 7ff56d7fa700 1 -- 172.21.15.47:0/221480516 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff560010da0 con 0x7ff570066be0 2022-09-07T21:38:30.426 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.416+0000 7ff5763ff700 1 -- 172.21.15.47:0/221480516 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57019dd00 con 0x7ff570066be0 2022-09-07T21:38:30.426 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.416+0000 7ff56d7fa700 1 -- 172.21.15.47:0/221480516 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5600173d0 con 0x7ff570066be0 2022-09-07T21:38:30.426 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.416+0000 7ff5763ff700 1 -- 172.21.15.47:0/221480516 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57019e200 con 0x7ff570066be0 2022-09-07T21:38:30.427 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.417+0000 7ff5763ff700 1 -- 172.21.15.47:0/221480516 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5701094f0 con 0x7ff570066be0 2022-09-07T21:38:30.428 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.419+0000 7ff56d7fa700 1 -- 172.21.15.47:0/221480516 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7ff56000c040 con 0x7ff570066be0 2022-09-07T21:38:30.428 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.421+0000 7ff56d7fa700 1 --2- 172.21.15.47:0/221480516 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff558099080 0x7ff55809b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:30.428 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.421+0000 7ff56d7fa700 1 -- 172.21.15.47:0/221480516 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7ff560014070 con 0x7ff570066be0 2022-09-07T21:38:30.429 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.421+0000 7ff56f7fe700 1 --2- 172.21.15.47:0/221480516 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff558099080 0x7ff55809b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:30.429 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.422+0000 7ff56f7fe700 1 --2- 172.21.15.47:0/221480516 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff558099080 0x7ff55809b530 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff56400b4f0 tx=0x7ff564009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:30.429 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.425+0000 7ff56d7fa700 1 -- 172.21.15.47:0/221480516 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff56004fc80 con 0x7ff570066be0 2022-09-07T21:38:30.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.685+0000 7ff5763ff700 1 -- 172.21.15.47:0/221480516 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff57019eb20 con 0x7ff570066be0 2022-09-07T21:38:30.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.686+0000 7ff56d7fa700 1 -- 172.21.15.47:0/221480516 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8067 (secure 0 0 0) 0x7ff56008eb30 con 0x7ff570066be0 2022-09-07T21:38:30.689 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:30.690 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/2559266585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2559266585},{"type":"v1","addr":"172.21.15.55:6835","nonce":2559266585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/2090942297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2090942297},{"type":"v1","addr":"172.21.15.55:6837","nonce":2090942297}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24526,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2445409240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2445409240},{"type":"v1","addr":"172.21.15.59:6835","nonce":2445409240}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/3542466465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":3542466465},{"type":"v1","addr":"172.21.15.59:6837","nonce":3542466465}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24548,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6839/315505812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":315505812},{"type":"v1","addr":"172.21.15.47:6839","nonce":315505812}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6839/1006724740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":1006724740},{"type":"v1","addr":"172.21.15.59:6839","nonce":1006724740}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2022-09-07T21:38:30.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.688+0000 7ff5763ff700 1 -- 172.21.15.47:0/221480516 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff558099080 msgr2=0x7ff55809b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:30.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.688+0000 7ff5763ff700 1 --2- 172.21.15.47:0/221480516 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff558099080 0x7ff55809b530 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff56400b4f0 tx=0x7ff564009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:30.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.689+0000 7ff5763ff700 1 -- 172.21.15.47:0/221480516 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff570066be0 msgr2=0x7ff5700fe1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:30.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.689+0000 7ff5763ff700 1 --2- 172.21.15.47:0/221480516 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff570066be0 0x7ff5700fe1c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff56000e6e0 tx=0x7ff56000e9f0 comp rx=0 tx=0).stop 2022-09-07T21:38:30.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.689+0000 7ff5763ff700 1 -- 172.21.15.47:0/221480516 shutdown_connections 2022-09-07T21:38:30.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.689+0000 7ff5763ff700 1 --2- 172.21.15.47:0/221480516 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff558099080 0x7ff55809b530 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:30.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.689+0000 7ff5763ff700 1 --2- 172.21.15.47:0/221480516 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5701038e0 0x7ff5700fcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:30.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.689+0000 7ff5763ff700 1 --2- 172.21.15.47:0/221480516 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff570066be0 0x7ff5700fe1c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:30.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.689+0000 7ff5763ff700 1 --2- 172.21.15.47:0/221480516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff570100df0 0x7ff5700fc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:30.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.689+0000 7ff5763ff700 1 -- 172.21.15.47:0/221480516 >> 172.21.15.47:0/221480516 conn(0x7ff5700f82e0 msgr2=0x7ff5700ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:30.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.689+0000 7ff5763ff700 1 -- 172.21.15.47:0/221480516 shutdown_connections 2022-09-07T21:38:30.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:30.689+0000 7ff5763ff700 1 -- 172.21.15.47:0/221480516 wait complete. 2022-09-07T21:38:30.695 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 11 2022-09-07T21:38:31.261 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:30 smithi047 ceph-mon[90087]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:31.261 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:30 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/221480516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:30 smithi059 ceph-mon[97450]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:30 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/221480516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:30 smithi055 ceph-mon[97033]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:30 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/221480516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:31.903 INFO:journalctl@ceph.mds.a.smithi047.stdout:Sep 07 21:38:31 smithi047 systemd[1]: Starting Ceph mds.a for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:38:32.490 INFO:journalctl@ceph.mds.a.smithi047.stdout:Sep 07 21:38:32 smithi047 podman[125484]: 2022-09-07T21:38:32.721 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:32.804 INFO:journalctl@ceph.mds.a.smithi047.stdout:Sep 07 21:38:32 smithi047 bash[125484]: 6f5e665bd681a71dfd8a13d043cef5e9c0bf86a0cfade3f1e30de7a71fe31255 2022-09-07T21:38:32.805 INFO:journalctl@ceph.mds.a.smithi047.stdout:Sep 07 21:38:32 smithi047 ceph-mds[125524]: -- 172.21.15.47:0/2727390037 <== mon.1 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55605e328820 con 0x55605e2fd000 2022-09-07T21:38:32.805 INFO:journalctl@ceph.mds.a.smithi047.stdout:Sep 07 21:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-a[125500]: starting mds.a at 2022-09-07T21:38:32.805 INFO:journalctl@ceph.mds.a.smithi047.stdout:Sep 07 21:38:32 smithi047 systemd[1]: Started Ceph mds.a for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:38:32.961 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.957+0000 7f87d4dff700 1 -- 172.21.15.47:0/3589222392 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87d01031f0 msgr2=0x7f87d01055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:32.961 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.957+0000 7f87d4dff700 1 --2- 172.21.15.47:0/3589222392 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87d01031f0 0x7f87d01055d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009810 tx=0x7f87c4009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:32.962 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.957+0000 7f87d4dff700 1 -- 172.21.15.47:0/3589222392 shutdown_connections 2022-09-07T21:38:32.962 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.957+0000 7f87d4dff700 1 --2- 172.21.15.47:0/3589222392 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87d0105b10 0x7f87d0107f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:32.962 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.957+0000 7f87d4dff700 1 --2- 172.21.15.47:0/3589222392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87d0066be0 0x7f87d0102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:32.962 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.957+0000 7f87d4dff700 1 --2- 172.21.15.47:0/3589222392 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87d01031f0 0x7f87d01055d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:32.963 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.957+0000 7f87d4dff700 1 -- 172.21.15.47:0/3589222392 >> 172.21.15.47:0/3589222392 conn(0x7f87d00f82e0 msgr2=0x7f87d00fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:32.963 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.957+0000 7f87d4dff700 1 -- 172.21.15.47:0/3589222392 shutdown_connections 2022-09-07T21:38:32.963 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.957+0000 7f87d4dff700 1 -- 172.21.15.47:0/3589222392 wait complete. 2022-09-07T21:38:32.964 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.959+0000 7f87d4dff700 1 Processor -- start 2022-09-07T21:38:32.964 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.959+0000 7f87d4dff700 1 -- start start 2022-09-07T21:38:32.965 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.960+0000 7f87d4dff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87d0066be0 0x7f87d0195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:32.965 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.960+0000 7f87d4dff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87d01031f0 0x7f87d0196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:32.965 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.960+0000 7f87d4dff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87d0105b10 0x7f87d019d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:32.965 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.960+0000 7f87d4dff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d019d8d0 con 0x7f87d0105b10 2022-09-07T21:38:32.966 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.960+0000 7f87d4dff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d019da40 con 0x7f87d0066be0 2022-09-07T21:38:32.966 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.960+0000 7f87d4dff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d019dc00 con 0x7f87d01031f0 2022-09-07T21:38:32.966 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.960+0000 7f87ce59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87d0066be0 0x7f87d0195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:32.966 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.960+0000 7f87ce59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87d0066be0 0x7f87d0195d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:47704/0 (socket says 172.21.15.47:47704) 2022-09-07T21:38:32.967 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.960+0000 7f87ce59c700 1 -- 172.21.15.47:0/1867288546 learned_addr learned my addr 172.21.15.47:0/1867288546 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:32.968 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.960+0000 7f87cdd9b700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87d01031f0 0x7f87d0196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:32.968 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.960+0000 7f87ced9d700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87d0105b10 0x7f87d019d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:32.969 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.961+0000 7f87ce59c700 1 -- 172.21.15.47:0/1867288546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87d01031f0 msgr2=0x7f87d0196250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:32.969 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.961+0000 7f87ce59c700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87d01031f0 0x7f87d0196250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:32.969 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.961+0000 7f87ce59c700 1 -- 172.21.15.47:0/1867288546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87d0105b10 msgr2=0x7f87d019d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:32.969 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.961+0000 7f87ce59c700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87d0105b10 0x7f87d019d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:32.970 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.961+0000 7f87ce59c700 1 -- 172.21.15.47:0/1867288546 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c4009470 con 0x7f87d0066be0 2022-09-07T21:38:32.970 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.961+0000 7f87cdd9b700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87d01031f0 0x7f87d0196250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:32.970 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.961+0000 7f87ce59c700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87d0066be0 0x7f87d0195d10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f87c000e6e0 tx=0x7f87c000e9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:32.970 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.963+0000 7f87bf7fe700 1 -- 172.21.15.47:0/1867288546 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c0009500 con 0x7f87d0066be0 2022-09-07T21:38:32.971 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.963+0000 7f87bf7fe700 1 -- 172.21.15.47:0/1867288546 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f87c0010da0 con 0x7f87d0066be0 2022-09-07T21:38:32.971 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.963+0000 7f87d4dff700 1 -- 172.21.15.47:0/1867288546 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d019dee0 con 0x7f87d0066be0 2022-09-07T21:38:32.971 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.964+0000 7f87bf7fe700 1 -- 172.21.15.47:0/1867288546 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c00173d0 con 0x7f87d0066be0 2022-09-07T21:38:32.971 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.964+0000 7f87d4dff700 1 -- 172.21.15.47:0/1867288546 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d019e310 con 0x7f87d0066be0 2022-09-07T21:38:32.972 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.965+0000 7f87d4dff700 1 -- 172.21.15.47:0/1867288546 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d010b720 con 0x7f87d0066be0 2022-09-07T21:38:32.972 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.966+0000 7f87bf7fe700 1 -- 172.21.15.47:0/1867288546 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f87c000c040 con 0x7f87d0066be0 2022-09-07T21:38:32.973 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.967+0000 7f87bf7fe700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f87b8099080 0x7f87b809b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:32.973 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.968+0000 7f87bf7fe700 1 -- 172.21.15.47:0/1867288546 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f87c0014070 con 0x7f87d0066be0 2022-09-07T21:38:32.973 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.968+0000 7f87cdd9b700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f87b8099080 0x7f87b809b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.968+0000 7f87cdd9b700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f87b8099080 0x7f87b809b530 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f87c400b4f0 tx=0x7f87c4009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:32.972+0000 7f87bf7fe700 1 -- 172.21.15.47:0/1867288546 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f87c004fc80 con 0x7f87d0066be0 2022-09-07T21:38:33.225 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:33.222+0000 7f87d4dff700 1 -- 172.21.15.47:0/1867288546 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87d010b8b0 con 0x7f87d0066be0 2022-09-07T21:38:33.227 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:33.223+0000 7f87bf7fe700 1 -- 172.21.15.47:0/1867288546 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8822 (secure 0 0 0) 0x7f87d010b8b0 con 0x7f87d0066be0 2022-09-07T21:38:33.227 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:33.228 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/2559266585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2559266585},{"type":"v1","addr":"172.21.15.55:6835","nonce":2559266585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/2090942297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2090942297},{"type":"v1","addr":"172.21.15.55:6837","nonce":2090942297}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24526,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2445409240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2445409240},{"type":"v1","addr":"172.21.15.59:6835","nonce":2445409240}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/3542466465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":3542466465},{"type":"v1","addr":"172.21.15.59:6837","nonce":3542466465}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24548,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6839/315505812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":315505812},{"type":"v1","addr":"172.21.15.47:6839","nonce":315505812}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6839/1006724740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":1006724740},{"type":"v1","addr":"172.21.15.59:6839","nonce":1006724740}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6841/793863162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6840","nonce":793863162},{"type":"v1","addr":"172.21.15.47:6841","nonce":793863162}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[]} 2022-09-07T21:38:33.230 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:33.225+0000 7f87d4dff700 1 -- 172.21.15.47:0/1867288546 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f87b8099080 msgr2=0x7f87b809b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:33.231 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:33.225+0000 7f87d4dff700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f87b8099080 0x7f87b809b530 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f87c400b4f0 tx=0x7f87c4009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:33.231 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:33.225+0000 7f87d4dff700 1 -- 172.21.15.47:0/1867288546 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87d0066be0 msgr2=0x7f87d0195d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:33.231 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:33.225+0000 7f87d4dff700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87d0066be0 0x7f87d0195d10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f87c000e6e0 tx=0x7f87c000e9f0 comp rx=0 tx=0).stop 2022-09-07T21:38:33.231 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:33.225+0000 7f87d4dff700 1 -- 172.21.15.47:0/1867288546 shutdown_connections 2022-09-07T21:38:33.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:33.225+0000 7f87d4dff700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f87b8099080 0x7f87b809b530 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:33.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:33.225+0000 7f87d4dff700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87d0105b10 0x7f87d019d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:33.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:33.225+0000 7f87d4dff700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87d0066be0 0x7f87d0195d10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:33.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:33.225+0000 7f87d4dff700 1 --2- 172.21.15.47:0/1867288546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87d01031f0 0x7f87d0196250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:33.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:33.225+0000 7f87d4dff700 1 -- 172.21.15.47:0/1867288546 >> 172.21.15.47:0/1867288546 conn(0x7f87d00f82e0 msgr2=0x7f87d0103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:33.225+0000 7f87d4dff700 1 -- 172.21.15.47:0/1867288546 shutdown_connections 2022-09-07T21:38:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:33.226+0000 7f87d4dff700 1 -- 172.21.15.47:0/1867288546 wait complete. 2022-09-07T21:38:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 12 2022-09-07T21:38:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:33 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] up:boot 2022-09-07T21:38:33.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:33 smithi055 ceph-mon[97033]: fsmap 11 up:standby 2022-09-07T21:38:33.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-09-07T21:38:33.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:33 smithi055 ceph-mon[97033]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:33.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:33.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:33.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:33.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:33.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:33 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:33.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:33 smithi055 ceph-mon[97033]: Deploying daemon mds.b on smithi055 2022-09-07T21:38:33.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:33 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1867288546' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:33.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:33 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] up:boot 2022-09-07T21:38:33.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:33 smithi047 ceph-mon[90087]: fsmap 11 up:standby 2022-09-07T21:38:33.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-09-07T21:38:33.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:33 smithi047 ceph-mon[90087]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:33.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:33.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:33.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:33.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:33.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:33 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:33.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:33 smithi047 ceph-mon[90087]: Deploying daemon mds.b on smithi055 2022-09-07T21:38:33.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:33 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1867288546' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:33 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] up:boot 2022-09-07T21:38:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:33 smithi059 ceph-mon[97450]: fsmap 11 up:standby 2022-09-07T21:38:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-09-07T21:38:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:33 smithi059 ceph-mon[97450]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:34.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:34.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:34.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-07T21:38:34.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-07T21:38:34.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:33 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:34.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:33 smithi059 ceph-mon[97450]: Deploying daemon mds.b on smithi055 2022-09-07T21:38:34.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:33 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1867288546' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:34.406 INFO:journalctl@ceph.mds.b.smithi055.stdout:Sep 07 21:38:34 smithi055 systemd[1]: Starting Ceph mds.b for 01b8396e-2ef4-11ed-8431-001a4aab830c... 2022-09-07T21:38:35.117 INFO:journalctl@ceph.mds.b.smithi055.stdout:Sep 07 21:38:34 smithi055 podman[124019]: 2022-09-07T21:38:35.258 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:35.426 INFO:journalctl@ceph.mds.b.smithi055.stdout:Sep 07 21:38:35 smithi055 ceph-mds[124058]: -- 172.21.15.55:0/2744815020 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55947bbfa820 con 0x55947bbcec00 2022-09-07T21:38:35.426 INFO:journalctl@ceph.mds.b.smithi055.stdout:Sep 07 21:38:35 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-b[124035]: starting mds.b at 2022-09-07T21:38:35.427 INFO:journalctl@ceph.mds.b.smithi055.stdout:Sep 07 21:38:35 smithi055 bash[124019]: d1bd93256a1d824704a17102997f1fb58939267073702e8be8c57ed52aae506b 2022-09-07T21:38:35.427 INFO:journalctl@ceph.mds.b.smithi055.stdout:Sep 07 21:38:35 smithi055 systemd[1]: Started Ceph mds.b for 01b8396e-2ef4-11ed-8431-001a4aab830c. 2022-09-07T21:38:35.513 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.510+0000 7f67eedff700 1 -- 172.21.15.47:0/3440506535 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67e8100df0 msgr2=0x7f67e8103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:35.514 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.510+0000 7f67eedff700 1 --2- 172.21.15.47:0/3440506535 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67e8100df0 0x7f67e8103250 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f67d4009750 tx=0x7f67d4009a60 comp rx=0 tx=0).stop 2022-09-07T21:38:35.514 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.510+0000 7f67eedff700 1 -- 172.21.15.47:0/3440506535 shutdown_connections 2022-09-07T21:38:35.514 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.510+0000 7f67eedff700 1 --2- 172.21.15.47:0/3440506535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f67e81038e0 0x7f67e8105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:35.514 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.510+0000 7f67eedff700 1 --2- 172.21.15.47:0/3440506535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e8066be0 0x7f67e81008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:35.515 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.510+0000 7f67eedff700 1 --2- 172.21.15.47:0/3440506535 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67e8100df0 0x7f67e8103250 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:35.515 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.510+0000 7f67eedff700 1 -- 172.21.15.47:0/3440506535 >> 172.21.15.47:0/3440506535 conn(0x7f67e80f82e0 msgr2=0x7f67e80fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:35.515 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.511+0000 7f67eedff700 1 -- 172.21.15.47:0/3440506535 shutdown_connections 2022-09-07T21:38:35.515 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.511+0000 7f67eedff700 1 -- 172.21.15.47:0/3440506535 wait complete. 2022-09-07T21:38:35.518 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.512+0000 7f67eedff700 1 Processor -- start 2022-09-07T21:38:35.518 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.512+0000 7f67eedff700 1 -- start start 2022-09-07T21:38:35.518 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.512+0000 7f67eedff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f67e8066be0 0x7f67e8195b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:35.518 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.512+0000 7f67eedff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e8100df0 0x7f67e8196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:35.519 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.513+0000 7f67eedff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67e81038e0 0x7f67e819d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:35.519 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.513+0000 7f67eedff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e819d690 con 0x7f67e8066be0 2022-09-07T21:38:35.519 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.513+0000 7f67eedff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e819d800 con 0x7f67e81038e0 2022-09-07T21:38:35.519 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.513+0000 7f67eedff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e819d9c0 con 0x7f67e8100df0 2022-09-07T21:38:35.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.513+0000 7f67ecb9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f67e8066be0 0x7f67e8195b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:35.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.513+0000 7f67e7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e8100df0 0x7f67e8196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:35.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.513+0000 7f67ed39c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67e81038e0 0x7f67e819d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:35.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.513+0000 7f67ecb9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f67e8066be0 0x7f67e8195b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45144/0 (socket says 172.21.15.47:45144) 2022-09-07T21:38:35.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.513+0000 7f67ecb9b700 1 -- 172.21.15.47:0/1199826031 learned_addr learned my addr 172.21.15.47:0/1199826031 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:35.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.513+0000 7f67ed39c700 1 -- 172.21.15.47:0/1199826031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e8100df0 msgr2=0x7f67e8196050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:35.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.513+0000 7f67ed39c700 1 --2- 172.21.15.47:0/1199826031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e8100df0 0x7f67e8196050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:35.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.513+0000 7f67ed39c700 1 -- 172.21.15.47:0/1199826031 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f67e8066be0 msgr2=0x7f67e8195b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:35.522 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.513+0000 7f67ed39c700 1 --2- 172.21.15.47:0/1199826031 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f67e8066be0 0x7f67e8195b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:35.522 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.513+0000 7f67ed39c700 1 -- 172.21.15.47:0/1199826031 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d40093b0 con 0x7f67e81038e0 2022-09-07T21:38:35.522 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.514+0000 7f67ed39c700 1 --2- 172.21.15.47:0/1199826031 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67e81038e0 0x7f67e819d0c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f67e000eab0 tx=0x7f67e000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:35.522 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.514+0000 7f67e5ffb700 1 -- 172.21.15.47:0/1199826031 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e00098c0 con 0x7f67e81038e0 2022-09-07T21:38:35.522 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.514+0000 7f67e5ffb700 1 -- 172.21.15.47:0/1199826031 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f67e000ce10 con 0x7f67e81038e0 2022-09-07T21:38:35.523 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.514+0000 7f67eedff700 1 -- 172.21.15.47:0/1199826031 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e819dca0 con 0x7f67e81038e0 2022-09-07T21:38:35.523 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.514+0000 7f67e5ffb700 1 -- 172.21.15.47:0/1199826031 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e0003ea0 con 0x7f67e81038e0 2022-09-07T21:38:35.523 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.514+0000 7f67eedff700 1 -- 172.21.15.47:0/1199826031 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e819e1b0 con 0x7f67e81038e0 2022-09-07T21:38:35.523 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.515+0000 7f67eedff700 1 -- 172.21.15.47:0/1199826031 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e81094f0 con 0x7f67e81038e0 2022-09-07T21:38:35.524 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.517+0000 7f67e5ffb700 1 -- 172.21.15.47:0/1199826031 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f67e000c040 con 0x7f67e81038e0 2022-09-07T21:38:35.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.518+0000 7f67e5ffb700 1 --2- 172.21.15.47:0/1199826031 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f67d8099080 0x7f67d809b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:35.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.518+0000 7f67e5ffb700 1 -- 172.21.15.47:0/1199826031 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f67e0014070 con 0x7f67e81038e0 2022-09-07T21:38:35.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.518+0000 7f67ecb9b700 1 --2- 172.21.15.47:0/1199826031 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f67d8099080 0x7f67d809b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:35.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.519+0000 7f67ecb9b700 1 --2- 172.21.15.47:0/1199826031 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f67d8099080 0x7f67d809b530 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f67dc005b90 tx=0x7f67dc005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:35.526 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.522+0000 7f67e5ffb700 1 -- 172.21.15.47:0/1199826031 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f67e008ebd0 con 0x7f67e81038e0 2022-09-07T21:38:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:35 smithi055 ceph-mon[97033]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:35 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:35 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:35.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:35 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:38:35.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:35 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:35.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:35 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:38:35.781 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.779+0000 7f67eedff700 1 -- 172.21.15.47:0/1199826031 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67e819ed40 con 0x7f67e81038e0 2022-09-07T21:38:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:35 smithi059 ceph-mon[97450]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:35 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:35.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:35 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:35.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:35 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:38:35.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:35 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:35.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:35 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:38:35.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.780+0000 7f67e5ffb700 1 -- 172.21.15.47:0/1199826031 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9577 (secure 0 0 0) 0x7f67e00921f0 con 0x7f67e81038e0 2022-09-07T21:38:35.784 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:35.784 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/2559266585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2559266585},{"type":"v1","addr":"172.21.15.55:6835","nonce":2559266585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/2090942297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2090942297},{"type":"v1","addr":"172.21.15.55:6837","nonce":2090942297}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24526,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2445409240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2445409240},{"type":"v1","addr":"172.21.15.59:6835","nonce":2445409240}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/3542466465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":3542466465},{"type":"v1","addr":"172.21.15.59:6837","nonce":3542466465}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24548,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6839/315505812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":315505812},{"type":"v1","addr":"172.21.15.47:6839","nonce":315505812}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6839/1006724740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":1006724740},{"type":"v1","addr":"172.21.15.59:6839","nonce":1006724740}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24572,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6839/517990300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":517990300},{"type":"v1","addr":"172.21.15.55:6839","nonce":517990300}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6841/793863162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6840","nonce":793863162},{"type":"v1","addr":"172.21.15.47:6841","nonce":793863162}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[]} 2022-09-07T21:38:35.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.782+0000 7f67eedff700 1 -- 172.21.15.47:0/1199826031 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f67d8099080 msgr2=0x7f67d809b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:35.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.782+0000 7f67eedff700 1 --2- 172.21.15.47:0/1199826031 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f67d8099080 0x7f67d809b530 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f67dc005b90 tx=0x7f67dc005f60 comp rx=0 tx=0).stop 2022-09-07T21:38:35.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.782+0000 7f67eedff700 1 -- 172.21.15.47:0/1199826031 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67e81038e0 msgr2=0x7f67e819d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:35.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.782+0000 7f67eedff700 1 --2- 172.21.15.47:0/1199826031 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67e81038e0 0x7f67e819d0c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f67e000eab0 tx=0x7f67e000edc0 comp rx=0 tx=0).stop 2022-09-07T21:38:35.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.783+0000 7f67eedff700 1 -- 172.21.15.47:0/1199826031 shutdown_connections 2022-09-07T21:38:35.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.783+0000 7f67eedff700 1 --2- 172.21.15.47:0/1199826031 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f67d8099080 0x7f67d809b530 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:35.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.783+0000 7f67eedff700 1 --2- 172.21.15.47:0/1199826031 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67e81038e0 0x7f67e819d0c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:35.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.783+0000 7f67eedff700 1 --2- 172.21.15.47:0/1199826031 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f67e8066be0 0x7f67e8195b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:35.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.783+0000 7f67eedff700 1 --2- 172.21.15.47:0/1199826031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e8100df0 0x7f67e8196050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:35.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.783+0000 7f67eedff700 1 -- 172.21.15.47:0/1199826031 >> 172.21.15.47:0/1199826031 conn(0x7f67e80f82e0 msgr2=0x7f67e80fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:35.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.783+0000 7f67eedff700 1 -- 172.21.15.47:0/1199826031 shutdown_connections 2022-09-07T21:38:35.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:35.783+0000 7f67eedff700 1 -- 172.21.15.47:0/1199826031 wait complete. 2022-09-07T21:38:35.788 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 13 2022-09-07T21:38:35.814 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-07T21:38:35.893 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:35 smithi047 ceph-mon[90087]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:35.893 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:35 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:35.893 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:35 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:35.894 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:35 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:38:35.894 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:35 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:35.894 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:35 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:38:36.053 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.050+0000 7ff929b9d700 1 -- 172.21.15.47:0/337860310 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff924100430 msgr2=0x7ff924108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:36.054 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.050+0000 7ff929b9d700 1 --2- 172.21.15.47:0/337860310 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff924100430 0x7ff924108210 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009bb0 tx=0x7ff91c009ec0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.054 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.050+0000 7ff929b9d700 1 -- 172.21.15.47:0/337860310 shutdown_connections 2022-09-07T21:38:36.054 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.050+0000 7ff929b9d700 1 --2- 172.21.15.47:0/337860310 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff924100430 0x7ff924108210 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.054 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.050+0000 7ff929b9d700 1 --2- 172.21.15.47:0/337860310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9241013f0 0x7ff924101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.051+0000 7ff929b9d700 1 --2- 172.21.15.47:0/337860310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9240ffa80 0x7ff9240ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.051+0000 7ff929b9d700 1 -- 172.21.15.47:0/337860310 >> 172.21.15.47:0/337860310 conn(0x7ff9240fb4d0 msgr2=0x7ff9240fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:36.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.051+0000 7ff929b9d700 1 -- 172.21.15.47:0/337860310 shutdown_connections 2022-09-07T21:38:36.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.051+0000 7ff929b9d700 1 -- 172.21.15.47:0/337860310 wait complete. 2022-09-07T21:38:36.057 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.052+0000 7ff929b9d700 1 Processor -- start 2022-09-07T21:38:36.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.052+0000 7ff929b9d700 1 -- start start 2022-09-07T21:38:36.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.052+0000 7ff929b9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9240ffa80 0x7ff924195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:36.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.052+0000 7ff929b9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff924100430 0x7ff924195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:36.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.052+0000 7ff929b9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9241013f0 0x7ff92419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:36.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.052+0000 7ff929b9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92419d490 con 0x7ff924100430 2022-09-07T21:38:36.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.052+0000 7ff929b9d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92419d600 con 0x7ff9241013f0 2022-09-07T21:38:36.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.052+0000 7ff929b9d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92419d7c0 con 0x7ff9240ffa80 2022-09-07T21:38:36.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.053+0000 7ff928b9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9240ffa80 0x7ff924195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:36.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.053+0000 7ff928b9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9240ffa80 0x7ff924195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:46184/0 (socket says 172.21.15.47:46184) 2022-09-07T21:38:36.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.053+0000 7ff928b9b700 1 -- 172.21.15.47:0/3374973752 learned_addr learned my addr 172.21.15.47:0/3374973752 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:36.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.053+0000 7ff92939c700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9241013f0 0x7ff92419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:36.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.053+0000 7ff923fff700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff924100430 0x7ff924195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:36.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.053+0000 7ff928b9b700 1 -- 172.21.15.47:0/3374973752 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9241013f0 msgr2=0x7ff92419cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:36.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.053+0000 7ff928b9b700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9241013f0 0x7ff92419cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.053+0000 7ff928b9b700 1 -- 172.21.15.47:0/3374973752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff924100430 msgr2=0x7ff924195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:36.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.053+0000 7ff928b9b700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff924100430 0x7ff924195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.053+0000 7ff928b9b700 1 -- 172.21.15.47:0/3374973752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c0097e0 con 0x7ff9240ffa80 2022-09-07T21:38:36.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.053+0000 7ff923fff700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff924100430 0x7ff924195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-07T21:38:36.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.054+0000 7ff928b9b700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9240ffa80 0x7ff924195910 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff91800e740 tx=0x7ff91800ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:36.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.054+0000 7ff921ffb700 1 -- 172.21.15.47:0/3374973752 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918005220 con 0x7ff9240ffa80 2022-09-07T21:38:36.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.054+0000 7ff929b9d700 1 -- 172.21.15.47:0/3374973752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92419daa0 con 0x7ff9240ffa80 2022-09-07T21:38:36.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.054+0000 7ff921ffb700 1 -- 172.21.15.47:0/3374973752 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff918005380 con 0x7ff9240ffa80 2022-09-07T21:38:36.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.054+0000 7ff921ffb700 1 -- 172.21.15.47:0/3374973752 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918004c10 con 0x7ff9240ffa80 2022-09-07T21:38:36.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.054+0000 7ff929b9d700 1 -- 172.21.15.47:0/3374973752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92419e000 con 0x7ff9240ffa80 2022-09-07T21:38:36.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.056+0000 7ff929b9d700 1 -- 172.21.15.47:0/3374973752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92410b5d0 con 0x7ff9240ffa80 2022-09-07T21:38:36.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.057+0000 7ff921ffb700 1 -- 172.21.15.47:0/3374973752 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7ff91801e020 con 0x7ff9240ffa80 2022-09-07T21:38:36.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.058+0000 7ff921ffb700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff914099090 0x7ff91409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:36.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.059+0000 7ff921ffb700 1 -- 172.21.15.47:0/3374973752 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7ff918014070 con 0x7ff9240ffa80 2022-09-07T21:38:36.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.059+0000 7ff923fff700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff914099090 0x7ff91409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:36.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.059+0000 7ff923fff700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff914099090 0x7ff91409b540 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff910005fd0 tx=0x7ff910005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:36.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.062+0000 7ff921ffb700 1 -- 172.21.15.47:0/3374973752 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff91808ed90 con 0x7ff9240ffa80 2022-09-07T21:38:36.283 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.281+0000 7ff929b9d700 1 -- 172.21.15.47:0/3374973752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff92405f990 con 0x7ff9240ffa80 2022-09-07T21:38:36.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.282+0000 7ff921ffb700 1 -- 172.21.15.47:0/3374973752 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18053 (secure 0 0 0) 0x7ff9180923b0 con 0x7ff9240ffa80 2022-09-07T21:38:36.286 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:36.286 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":81,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:37:52.015572+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:38:36.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.285+0000 7ff929b9d700 1 -- 172.21.15.47:0/3374973752 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff914099090 msgr2=0x7ff91409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:36.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.285+0000 7ff929b9d700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff914099090 0x7ff91409b540 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff910005fd0 tx=0x7ff910005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.285+0000 7ff929b9d700 1 -- 172.21.15.47:0/3374973752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9240ffa80 msgr2=0x7ff924195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:36.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.285+0000 7ff929b9d700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9240ffa80 0x7ff924195910 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff91800e740 tx=0x7ff91800ea50 comp rx=0 tx=0).stop 2022-09-07T21:38:36.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.285+0000 7ff929b9d700 1 -- 172.21.15.47:0/3374973752 shutdown_connections 2022-09-07T21:38:36.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.285+0000 7ff929b9d700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff914099090 0x7ff91409b540 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.285+0000 7ff929b9d700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9241013f0 0x7ff92419cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.292 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.285+0000 7ff929b9d700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9240ffa80 0x7ff924195910 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.292 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.285+0000 7ff929b9d700 1 --2- 172.21.15.47:0/3374973752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff924100430 0x7ff924195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.285+0000 7ff929b9d700 1 -- 172.21.15.47:0/3374973752 >> 172.21.15.47:0/3374973752 conn(0x7ff9240fb4d0 msgr2=0x7ff9241044d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:36.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.285+0000 7ff929b9d700 1 -- 172.21.15.47:0/3374973752 shutdown_connections 2022-09-07T21:38:36.294 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.285+0000 7ff929b9d700 1 -- 172.21.15.47:0/3374973752 wait complete. 2022-09-07T21:38:36.323 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-07T21:33:29.342534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-09-07T21:38:36.324 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-09-07T21:38:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.581+0000 7f595359e700 1 -- 172.21.15.47:0/3847573441 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59540ffe60 msgr2=0x7f5954100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.581+0000 7f595359e700 1 --2- 172.21.15.47:0/3847573441 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59540ffe60 0x7f5954100270 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5944009810 tx=0x7f5944009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:36.585 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.581+0000 7f595359e700 1 -- 172.21.15.47:0/3847573441 shutdown_connections 2022-09-07T21:38:36.585 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.581+0000 7f595359e700 1 --2- 172.21.15.47:0/3847573441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59541019a0 0x7f5954108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.585 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.581+0000 7f595359e700 1 --2- 172.21.15.47:0/3847573441 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59540ffe60 0x7f5954100270 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.581+0000 7f595359e700 1 --2- 172.21.15.47:0/3847573441 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5954100ff0 0x7f5954101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.581+0000 7f595359e700 1 -- 172.21.15.47:0/3847573441 >> 172.21.15.47:0/3847573441 conn(0x7f59540fb6b0 msgr2=0x7f59540fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.582+0000 7f595359e700 1 -- 172.21.15.47:0/3847573441 shutdown_connections 2022-09-07T21:38:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.582+0000 7f595359e700 1 -- 172.21.15.47:0/3847573441 wait complete. 2022-09-07T21:38:36.588 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.583+0000 7f595359e700 1 Processor -- start 2022-09-07T21:38:36.589 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.583+0000 7f595359e700 1 -- start start 2022-09-07T21:38:36.589 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.583+0000 7f595359e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59540ffe60 0x7f5954195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:36.589 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.583+0000 7f595359e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5954100ff0 0x7f5954195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:36.590 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.583+0000 7f595359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59541019a0 0x7f595419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:36.590 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.583+0000 7f595359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595419d490 con 0x7f59541019a0 2022-09-07T21:38:36.590 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.583+0000 7f595359e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595419d600 con 0x7f59540ffe60 2022-09-07T21:38:36.590 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.583+0000 7f595359e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595419d7c0 con 0x7f5954100ff0 2022-09-07T21:38:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.584+0000 7f595259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59540ffe60 0x7f5954195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.584+0000 7f5952d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59541019a0 0x7f595419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.584+0000 7f5951d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5954100ff0 0x7f5954195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.584+0000 7f595259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59540ffe60 0x7f5954195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:47754/0 (socket says 172.21.15.47:47754) 2022-09-07T21:38:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.584+0000 7f595259c700 1 -- 172.21.15.47:0/3803053771 learned_addr learned my addr 172.21.15.47:0/3803053771 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.584+0000 7f595259c700 1 -- 172.21.15.47:0/3803053771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5954100ff0 msgr2=0x7f5954195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.584+0000 7f595259c700 1 --2- 172.21.15.47:0/3803053771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5954100ff0 0x7f5954195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.584+0000 7f595259c700 1 -- 172.21.15.47:0/3803053771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59541019a0 msgr2=0x7f595419cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.584+0000 7f595259c700 1 --2- 172.21.15.47:0/3803053771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59541019a0 0x7f595419cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.584+0000 7f595259c700 1 -- 172.21.15.47:0/3803053771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5944009470 con 0x7f59540ffe60 2022-09-07T21:38:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.584+0000 7f595259c700 1 --2- 172.21.15.47:0/3803053771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59540ffe60 0x7f5954195910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f594400f690 tx=0x7f594400f6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.585+0000 7f59437fe700 1 -- 172.21.15.47:0/3803053771 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594401d070 con 0x7f59540ffe60 2022-09-07T21:38:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.585+0000 7f59437fe700 1 -- 172.21.15.47:0/3803053771 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f594400f8e0 con 0x7f59540ffe60 2022-09-07T21:38:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.585+0000 7f59437fe700 1 -- 172.21.15.47:0/3803053771 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5944004860 con 0x7f59540ffe60 2022-09-07T21:38:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.585+0000 7f595359e700 1 -- 172.21.15.47:0/3803053771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f595419da40 con 0x7f59540ffe60 2022-09-07T21:38:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.585+0000 7f595359e700 1 -- 172.21.15.47:0/3803053771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f595419df50 con 0x7f59540ffe60 2022-09-07T21:38:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.586+0000 7f595359e700 1 -- 172.21.15.47:0/3803053771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f595410b4d0 con 0x7f59540ffe60 2022-09-07T21:38:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.587+0000 7f59437fe700 1 -- 172.21.15.47:0/3803053771 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f5944028030 con 0x7f59540ffe60 2022-09-07T21:38:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.589+0000 7f59437fe700 1 --2- 172.21.15.47:0/3803053771 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f593c099090 0x7f593c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.589+0000 7f59437fe700 1 -- 172.21.15.47:0/3803053771 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f59440c2c60 con 0x7f59540ffe60 2022-09-07T21:38:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.589+0000 7f5951d9b700 1 --2- 172.21.15.47:0/3803053771 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f593c099090 0x7f593c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.590+0000 7f5951d9b700 1 --2- 172.21.15.47:0/3803053771 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f593c099090 0x7f593c09b540 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5948005fd0 tx=0x7f5948005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.593+0000 7f59437fe700 1 -- 172.21.15.47:0/3803053771 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f594408f750 con 0x7f59540ffe60 2022-09-07T21:38:36.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.810+0000 7f595359e700 1 -- 172.21.15.47:0/3803053771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f595419e7d0 con 0x7f59540ffe60 2022-09-07T21:38:36.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.811+0000 7f59437fe700 1 -- 172.21.15.47:0/3803053771 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5944092d70 con 0x7f59540ffe60 2022-09-07T21:38:36.815 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2022-09-07T21:38:36.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.813+0000 7f595359e700 1 -- 172.21.15.47:0/3803053771 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f593c099090 msgr2=0x7f593c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:36.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.813+0000 7f595359e700 1 --2- 172.21.15.47:0/3803053771 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f593c099090 0x7f593c09b540 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5948005fd0 tx=0x7f5948005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.818 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.813+0000 7f595359e700 1 -- 172.21.15.47:0/3803053771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59540ffe60 msgr2=0x7f5954195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:36.818 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.813+0000 7f595359e700 1 --2- 172.21.15.47:0/3803053771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59540ffe60 0x7f5954195910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f594400f690 tx=0x7f594400f6c0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.818 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.814+0000 7f595359e700 1 -- 172.21.15.47:0/3803053771 shutdown_connections 2022-09-07T21:38:36.819 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.814+0000 7f595359e700 1 --2- 172.21.15.47:0/3803053771 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f593c099090 0x7f593c09b540 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.819 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.814+0000 7f595359e700 1 --2- 172.21.15.47:0/3803053771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59541019a0 0x7f595419cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.814+0000 7f595359e700 1 --2- 172.21.15.47:0/3803053771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59540ffe60 0x7f5954195910 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.814+0000 7f595359e700 1 --2- 172.21.15.47:0/3803053771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5954100ff0 0x7f5954195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:36.821 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.814+0000 7f595359e700 1 -- 172.21.15.47:0/3803053771 >> 172.21.15.47:0/3803053771 conn(0x7f59540fb6b0 msgr2=0x7f59540fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:36.821 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.814+0000 7f595359e700 1 -- 172.21.15.47:0/3803053771 shutdown_connections 2022-09-07T21:38:36.822 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:36.814+0000 7f595359e700 1 -- 172.21.15.47:0/3803053771 wait complete. 2022-09-07T21:38:36.843 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-09-07T21:38:36.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:36 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] up:boot 2022-09-07T21:38:36.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:36 smithi047 ceph-mon[90087]: fsmap 12 up:standby 2022-09-07T21:38:36.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:36 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-09-07T21:38:36.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:36 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1199826031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:36.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:36 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3374973752' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:36.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:36 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:36 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] up:boot 2022-09-07T21:38:36.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:36 smithi055 ceph-mon[97033]: fsmap 12 up:standby 2022-09-07T21:38:36.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:36 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-09-07T21:38:36.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:36 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1199826031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:36.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:36 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3374973752' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:36.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:36 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:36 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] up:boot 2022-09-07T21:38:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:36 smithi059 ceph-mon[97450]: fsmap 12 up:standby 2022-09-07T21:38:37.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:36 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-09-07T21:38:37.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:36 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1199826031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:37.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:36 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3374973752' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:37.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:36 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:37.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.079+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/2867587188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2cc100f50 msgr2=0x7fe2cc103330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:37.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.079+0000 7fe2d1d9d700 1 --2- 172.21.15.47:0/2867587188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2cc100f50 0x7fe2cc103330 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009810 tx=0x7fe2b8009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:37.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.079+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/2867587188 shutdown_connections 2022-09-07T21:38:37.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.079+0000 7fe2d1d9d700 1 --2- 172.21.15.47:0/2867587188 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2cc103870 0x7fe2cc105c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:37.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.079+0000 7fe2d1d9d700 1 --2- 172.21.15.47:0/2867587188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2cc0fe630 0x7fe2cc100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:37.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.080+0000 7fe2d1d9d700 1 --2- 172.21.15.47:0/2867587188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2cc100f50 0x7fe2cc103330 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:37.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.080+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/2867587188 >> 172.21.15.47:0/2867587188 conn(0x7fe2cc0f80d0 msgr2=0x7fe2cc0fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:37.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.080+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/2867587188 shutdown_connections 2022-09-07T21:38:37.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.080+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/2867587188 wait complete. 2022-09-07T21:38:37.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.081+0000 7fe2d1d9d700 1 Processor -- start 2022-09-07T21:38:37.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.081+0000 7fe2d1d9d700 1 -- start start 2022-09-07T21:38:37.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.081+0000 7fe2d1d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2cc0fe630 0x7fe2cc1958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:37.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.081+0000 7fe2d1d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2cc100f50 0x7fe2cc195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:37.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.081+0000 7fe2d1d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2cc103870 0x7fe2cc19ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:37.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.081+0000 7fe2d1d9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc19d460 con 0x7fe2cc100f50 2022-09-07T21:38:37.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.081+0000 7fe2d1d9d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc19d5d0 con 0x7fe2cc0fe630 2022-09-07T21:38:37.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.081+0000 7fe2d1d9d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc19d790 con 0x7fe2cc103870 2022-09-07T21:38:37.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.082+0000 7fe2d0d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2cc0fe630 0x7fe2cc1958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:37.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.082+0000 7fe2d0d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2cc0fe630 0x7fe2cc1958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:47764/0 (socket says 172.21.15.47:47764) 2022-09-07T21:38:37.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.082+0000 7fe2d0d9b700 1 -- 172.21.15.47:0/3291265254 learned_addr learned my addr 172.21.15.47:0/3291265254 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:37.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.082+0000 7fe2cbfff700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2cc100f50 0x7fe2cc195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:37.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.082+0000 7fe2d159c700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2cc103870 0x7fe2cc19ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:37.097 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.082+0000 7fe2d0d9b700 1 -- 172.21.15.47:0/3291265254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2cc103870 msgr2=0x7fe2cc19ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:37.097 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.082+0000 7fe2d0d9b700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2cc103870 0x7fe2cc19ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:37.098 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.082+0000 7fe2d0d9b700 1 -- 172.21.15.47:0/3291265254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2cc100f50 msgr2=0x7fe2cc195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:37.098 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.082+0000 7fe2d0d9b700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2cc100f50 0x7fe2cc195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:37.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.082+0000 7fe2d0d9b700 1 -- 172.21.15.47:0/3291265254 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b8009470 con 0x7fe2cc0fe630 2022-09-07T21:38:37.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.082+0000 7fe2cbfff700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2cc100f50 0x7fe2cc195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:38:37.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.082+0000 7fe2d0d9b700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2cc0fe630 0x7fe2cc1958e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000e740 tx=0x7fe2c000eb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:37.100 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.083+0000 7fe2c9ffb700 1 -- 172.21.15.47:0/3291265254 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0005220 con 0x7fe2cc0fe630 2022-09-07T21:38:37.100 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.083+0000 7fe2c9ffb700 1 -- 172.21.15.47:0/3291265254 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe2c0005380 con 0x7fe2cc0fe630 2022-09-07T21:38:37.101 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.083+0000 7fe2c9ffb700 1 -- 172.21.15.47:0/3291265254 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0004c20 con 0x7fe2cc0fe630 2022-09-07T21:38:37.101 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.083+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/3291265254 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2cc19da70 con 0x7fe2cc0fe630 2022-09-07T21:38:37.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.083+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/3291265254 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2cc19df30 con 0x7fe2cc0fe630 2022-09-07T21:38:37.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.084+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/3291265254 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2cc1092b0 con 0x7fe2cc0fe630 2022-09-07T21:38:37.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.086+0000 7fe2c9ffb700 1 -- 172.21.15.47:0/3291265254 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fe2c001e020 con 0x7fe2cc0fe630 2022-09-07T21:38:37.105 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.088+0000 7fe2c9ffb700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe2bc099010 0x7fe2bc09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:37.105 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.088+0000 7fe2c9ffb700 1 -- 172.21.15.47:0/3291265254 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fe2c0014070 con 0x7fe2cc0fe630 2022-09-07T21:38:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.088+0000 7fe2cbfff700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe2bc099010 0x7fe2bc09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.089+0000 7fe2cbfff700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe2bc099010 0x7fe2bc09b4c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe2b800b4f0 tx=0x7fe2b8009e90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.091+0000 7fe2c9ffb700 1 -- 172.21.15.47:0/3291265254 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fe2c008eee0 con 0x7fe2cc0fe630 2022-09-07T21:38:37.306 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.303+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/3291265254 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe2cc05f9f0 con 0x7fe2cc0fe630 2022-09-07T21:38:37.307 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.304+0000 7fe2c9ffb700 1 -- 172.21.15.47:0/3291265254 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe2c0092500 con 0x7fe2cc0fe630 2022-09-07T21:38:37.307 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:37.308 INFO:teuthology.orchestra.run.smithi047.stdout:[ 2022-09-07T21:38:37.308 INFO:teuthology.orchestra.run.smithi047.stdout: { 2022-09-07T21:38:37.309 INFO:teuthology.orchestra.run.smithi047.stdout: "poolnum": 1, 2022-09-07T21:38:37.309 INFO:teuthology.orchestra.run.smithi047.stdout: "poolname": ".mgr" 2022-09-07T21:38:37.309 INFO:teuthology.orchestra.run.smithi047.stdout: } 2022-09-07T21:38:37.310 INFO:teuthology.orchestra.run.smithi047.stdout:] 2022-09-07T21:38:37.311 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.308+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/3291265254 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe2bc099010 msgr2=0x7fe2bc09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:37.312 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.308+0000 7fe2d1d9d700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe2bc099010 0x7fe2bc09b4c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe2b800b4f0 tx=0x7fe2b8009e90 comp rx=0 tx=0).stop 2022-09-07T21:38:37.312 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.308+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/3291265254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2cc0fe630 msgr2=0x7fe2cc1958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:37.312 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.308+0000 7fe2d1d9d700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2cc0fe630 0x7fe2cc1958e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000e740 tx=0x7fe2c000eb00 comp rx=0 tx=0).stop 2022-09-07T21:38:37.313 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.308+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/3291265254 shutdown_connections 2022-09-07T21:38:37.313 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.308+0000 7fe2d1d9d700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe2bc099010 0x7fe2bc09b4c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:37.313 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.308+0000 7fe2d1d9d700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2cc103870 0x7fe2cc19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:37.314 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.308+0000 7fe2d1d9d700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2cc0fe630 0x7fe2cc1958e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:37.314 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.308+0000 7fe2d1d9d700 1 --2- 172.21.15.47:0/3291265254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2cc100f50 0x7fe2cc195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:37.315 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.308+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/3291265254 >> 172.21.15.47:0/3291265254 conn(0x7fe2cc0f80d0 msgr2=0x7fe2cc0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:37.315 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.308+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/3291265254 shutdown_connections 2022-09-07T21:38:37.315 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.308+0000 7fe2d1d9d700 1 -- 172.21.15.47:0/3291265254 wait complete. 2022-09-07T21:38:37.346 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-09-07T21:38:37.346 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2022-09-07T21:38:37.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.598+0000 7f33dfdff700 1 -- 172.21.15.47:0/4255186063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33d80fefe0 msgr2=0x7f33d80ff450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:37.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.598+0000 7f33dfdff700 1 --2- 172.21.15.47:0/4255186063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33d80fefe0 0x7f33d80ff450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009810 tx=0x7f33c8009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:37.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.598+0000 7f33dfdff700 1 -- 172.21.15.47:0/4255186063 shutdown_connections 2022-09-07T21:38:37.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.598+0000 7f33dfdff700 1 --2- 172.21.15.47:0/4255186063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33d80ff990 0x7f33d81062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:37.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.598+0000 7f33dfdff700 1 --2- 172.21.15.47:0/4255186063 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33d80fde00 0x7f33d80fe210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:37.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.598+0000 7f33dfdff700 1 --2- 172.21.15.47:0/4255186063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33d80fefe0 0x7f33d80ff450 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:37.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.598+0000 7f33dfdff700 1 -- 172.21.15.47:0/4255186063 >> 172.21.15.47:0/4255186063 conn(0x7f33d80f9510 msgr2=0x7f33d80fb920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:37.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.598+0000 7f33dfdff700 1 -- 172.21.15.47:0/4255186063 shutdown_connections 2022-09-07T21:38:37.603 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.598+0000 7f33dfdff700 1 -- 172.21.15.47:0/4255186063 wait complete. 2022-09-07T21:38:37.605 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.599+0000 7f33dfdff700 1 Processor -- start 2022-09-07T21:38:37.605 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.599+0000 7f33dfdff700 1 -- start start 2022-09-07T21:38:37.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.600+0000 7f33dfdff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33d80fde00 0x7f33d8195b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:37.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.600+0000 7f33dfdff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33d80fefe0 0x7f33d8196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:37.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.600+0000 7f33dfdff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33d80ff990 0x7f33d819d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:37.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.600+0000 7f33dfdff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d819d690 con 0x7f33d80fefe0 2022-09-07T21:38:37.607 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.600+0000 7f33dfdff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d819d800 con 0x7f33d80ff990 2022-09-07T21:38:37.607 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.600+0000 7f33dfdff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d819d9c0 con 0x7f33d80fde00 2022-09-07T21:38:37.607 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.600+0000 7f33ddb9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33d80fde00 0x7f33d8195b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:37.607 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.600+0000 7f33ddb9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33d80fde00 0x7f33d8195b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:46230/0 (socket says 172.21.15.47:46230) 2022-09-07T21:38:37.608 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.600+0000 7f33ddb9b700 1 -- 172.21.15.47:0/2240351153 learned_addr learned my addr 172.21.15.47:0/2240351153 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:37.608 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.600+0000 7f33de39c700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33d80ff990 0x7f33d819d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:37.608 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.600+0000 7f33dd39a700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33d80fefe0 0x7f33d8196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:37.608 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.601+0000 7f33ddb9b700 1 -- 172.21.15.47:0/2240351153 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33d80ff990 msgr2=0x7f33d819d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:37.608 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.601+0000 7f33ddb9b700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33d80ff990 0x7f33d819d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:37.609 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.601+0000 7f33ddb9b700 1 -- 172.21.15.47:0/2240351153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33d80fefe0 msgr2=0x7f33d8196050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:37.609 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.601+0000 7f33ddb9b700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33d80fefe0 0x7f33d8196050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:37.609 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.601+0000 7f33ddb9b700 1 -- 172.21.15.47:0/2240351153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d40093b0 con 0x7f33d80fde00 2022-09-07T21:38:37.609 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.601+0000 7f33de39c700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33d80ff990 0x7f33d819d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:37.610 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.601+0000 7f33ddb9b700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33d80fde00 0x7f33d8195b10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f33d400e9c0 tx=0x7f33d400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:37.610 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.601+0000 7f33ceffd700 1 -- 172.21.15.47:0/2240351153 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d40097b0 con 0x7f33d80fde00 2022-09-07T21:38:37.610 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.601+0000 7f33ceffd700 1 -- 172.21.15.47:0/2240351153 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f33d400ce70 con 0x7f33d80fde00 2022-09-07T21:38:37.610 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.601+0000 7f33ceffd700 1 -- 172.21.15.47:0/2240351153 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d4003cf0 con 0x7f33d80fde00 2022-09-07T21:38:37.610 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.601+0000 7f33dfdff700 1 -- 172.21.15.47:0/2240351153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c8009470 con 0x7f33d80fde00 2022-09-07T21:38:37.611 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.602+0000 7f33dfdff700 1 -- 172.21.15.47:0/2240351153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d819e020 con 0x7f33d80fde00 2022-09-07T21:38:37.611 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.603+0000 7f33dfdff700 1 -- 172.21.15.47:0/2240351153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d8109550 con 0x7f33d80fde00 2022-09-07T21:38:37.612 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.604+0000 7f33ceffd700 1 -- 172.21.15.47:0/2240351153 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f33d401e020 con 0x7f33d80fde00 2022-09-07T21:38:37.612 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.605+0000 7f33ceffd700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f33c4099090 0x7f33c409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:37.613 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.605+0000 7f33ceffd700 1 -- 172.21.15.47:0/2240351153 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f33d4014070 con 0x7f33d80fde00 2022-09-07T21:38:37.613 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.606+0000 7f33dd39a700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f33c4099090 0x7f33c409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:37.613 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.606+0000 7f33dd39a700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f33c4099090 0x7f33c409b540 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f33c800b4f0 tx=0x7f33c8009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:37.613 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.609+0000 7f33ceffd700 1 -- 172.21.15.47:0/2240351153 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f33d408edd0 con 0x7f33d80fde00 2022-09-07T21:38:37.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:37.829+0000 7f33dfdff700 1 -- 172.21.15.47:0/2240351153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f33d805f9c0 con 0x7f33d80fde00 2022-09-07T21:38:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:37 smithi055 ceph-mon[97033]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:37 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3803053771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:38:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:37 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3291265254' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-09-07T21:38:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:37 smithi059 ceph-mon[97450]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:37 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3803053771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:38:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:37 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3291265254' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-09-07T21:38:38.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:37 smithi047 ceph-mon[90087]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:38.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:37 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3803053771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:38:38.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:37 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3291265254' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-09-07T21:38:38.666 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.663+0000 7f33ceffd700 1 -- 172.21.15.47:0/2240351153 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7f33d40923f0 con 0x7f33d80fde00 2022-09-07T21:38:38.670 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.668+0000 7f33dfdff700 1 -- 172.21.15.47:0/2240351153 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f33c4099090 msgr2=0x7f33c409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:38.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.668+0000 7f33dfdff700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f33c4099090 0x7f33c409b540 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f33c800b4f0 tx=0x7f33c8009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:38.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.668+0000 7f33dfdff700 1 -- 172.21.15.47:0/2240351153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33d80fde00 msgr2=0x7f33d8195b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:38.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.668+0000 7f33dfdff700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33d80fde00 0x7f33d8195b10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f33d400e9c0 tx=0x7f33d400ecd0 comp rx=0 tx=0).stop 2022-09-07T21:38:38.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.668+0000 7f33dfdff700 1 -- 172.21.15.47:0/2240351153 shutdown_connections 2022-09-07T21:38:38.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.668+0000 7f33dfdff700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f33c4099090 0x7f33c409b540 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:38.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.668+0000 7f33dfdff700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33d80ff990 0x7f33d819d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:38.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.668+0000 7f33dfdff700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33d80fde00 0x7f33d8195b10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:38.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.668+0000 7f33dfdff700 1 --2- 172.21.15.47:0/2240351153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33d80fefe0 0x7f33d8196050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:38.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.668+0000 7f33dfdff700 1 -- 172.21.15.47:0/2240351153 >> 172.21.15.47:0/2240351153 conn(0x7f33d80f9510 msgr2=0x7f33d8104950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:38.673 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.668+0000 7f33dfdff700 1 -- 172.21.15.47:0/2240351153 shutdown_connections 2022-09-07T21:38:38.673 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.668+0000 7f33dfdff700 1 -- 172.21.15.47:0/2240351153 wait complete. 2022-09-07T21:38:38.673 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' created 2022-09-07T21:38:38.701 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2022-09-07T21:38:38.961 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.958+0000 7f63e8b9d700 1 -- 172.21.15.47:0/2477021688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e40fc420 msgr2=0x7f63e40fe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:38.962 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.958+0000 7f63e8b9d700 1 --2- 172.21.15.47:0/2477021688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e40fc420 0x7f63e40fe840 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009810 tx=0x7f63d4009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:38.962 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.958+0000 7f63e8b9d700 1 -- 172.21.15.47:0/2477021688 shutdown_connections 2022-09-07T21:38:38.962 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.958+0000 7f63e8b9d700 1 --2- 172.21.15.47:0/2477021688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63e4101740 0x7f63e4101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:38.962 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.958+0000 7f63e8b9d700 1 --2- 172.21.15.47:0/2477021688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e40fc420 0x7f63e40fe840 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:38.963 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.958+0000 7f63e8b9d700 1 --2- 172.21.15.47:0/2477021688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63e40fed80 0x7f63e4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:38.963 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.958+0000 7f63e8b9d700 1 -- 172.21.15.47:0/2477021688 >> 172.21.15.47:0/2477021688 conn(0x7f63e40f80d0 msgr2=0x7f63e40fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:38.963 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.959+0000 7f63e8b9d700 1 -- 172.21.15.47:0/2477021688 shutdown_connections 2022-09-07T21:38:38.963 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.959+0000 7f63e8b9d700 1 -- 172.21.15.47:0/2477021688 wait complete. 2022-09-07T21:38:38.966 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.960+0000 7f63e8b9d700 1 Processor -- start 2022-09-07T21:38:38.966 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.960+0000 7f63e8b9d700 1 -- start start 2022-09-07T21:38:38.966 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.960+0000 7f63e8b9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63e40fc420 0x7f63e4199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:38.966 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.960+0000 7f63e8b9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e40fed80 0x7f63e419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:38.967 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.960+0000 7f63e8b9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63e4101740 0x7f63e41a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:38.967 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.960+0000 7f63e8b9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e41a18d0 con 0x7f63e40fed80 2022-09-07T21:38:38.967 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.960+0000 7f63e8b9d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e41a1a40 con 0x7f63e4101740 2022-09-07T21:38:38.967 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.960+0000 7f63e8b9d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e41a1c00 con 0x7f63e40fc420 2022-09-07T21:38:38.968 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.961+0000 7f63e3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63e4101740 0x7f63e41a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:38.968 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.961+0000 7f63e2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e40fed80 0x7f63e419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:38.968 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.961+0000 7f63e3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63e4101740 0x7f63e41a1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:45394/0 (socket says 172.21.15.47:45394) 2022-09-07T21:38:38.969 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.961+0000 7f63e2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e40fed80 0x7f63e419a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55296/0 (socket says 172.21.15.47:55296) 2022-09-07T21:38:38.969 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.961+0000 7f63e3fff700 1 -- 172.21.15.47:0/812842124 learned_addr learned my addr 172.21.15.47:0/812842124 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:38.969 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.961+0000 7f63e37fe700 1 --2- 172.21.15.47:0/812842124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63e40fc420 0x7f63e4199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:38.969 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.961+0000 7f63e3fff700 1 -- 172.21.15.47:0/812842124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63e40fc420 msgr2=0x7f63e4199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:38.970 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.961+0000 7f63e3fff700 1 --2- 172.21.15.47:0/812842124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63e40fc420 0x7f63e4199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:38.970 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.961+0000 7f63e3fff700 1 -- 172.21.15.47:0/812842124 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e40fed80 msgr2=0x7f63e419a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:38.970 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.961+0000 7f63e3fff700 1 --2- 172.21.15.47:0/812842124 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e40fed80 0x7f63e419a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:38.971 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.961+0000 7f63e3fff700 1 -- 172.21.15.47:0/812842124 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d80093b0 con 0x7f63e4101740 2022-09-07T21:38:38.971 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.962+0000 7f63e3fff700 1 --2- 172.21.15.47:0/812842124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63e4101740 0x7f63e41a1300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f63dc00eaf0 tx=0x7f63dc00ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:38.971 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.962+0000 7f63e0ff9700 1 -- 172.21.15.47:0/812842124 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc009960 con 0x7f63e4101740 2022-09-07T21:38:38.971 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.962+0000 7f63e8b9d700 1 -- 172.21.15.47:0/812842124 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d4009470 con 0x7f63e4101740 2022-09-07T21:38:38.972 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.962+0000 7f63e0ff9700 1 -- 172.21.15.47:0/812842124 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f63dc00ce70 con 0x7f63e4101740 2022-09-07T21:38:38.972 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.962+0000 7f63e0ff9700 1 -- 172.21.15.47:0/812842124 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc0058b0 con 0x7f63e4101740 2022-09-07T21:38:38.972 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.962+0000 7f63e8b9d700 1 -- 172.21.15.47:0/812842124 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e41a21b0 con 0x7f63e4101740 2022-09-07T21:38:38.972 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.963+0000 7f63e8b9d700 1 -- 172.21.15.47:0/812842124 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e410f8c0 con 0x7f63e4101740 2022-09-07T21:38:38.973 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.965+0000 7f63e0ff9700 1 -- 172.21.15.47:0/812842124 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f63dc01e020 con 0x7f63e4101740 2022-09-07T21:38:38.974 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.967+0000 7f63e0ff9700 1 --2- 172.21.15.47:0/812842124 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f63cc099010 0x7f63cc09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:38.974 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.967+0000 7f63e0ff9700 1 -- 172.21.15.47:0/812842124 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f63dc014070 con 0x7f63e4101740 2022-09-07T21:38:38.974 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.967+0000 7f63e37fe700 1 --2- 172.21.15.47:0/812842124 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f63cc099010 0x7f63cc09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:38.974 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.968+0000 7f63e37fe700 1 --2- 172.21.15.47:0/812842124 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f63cc099010 0x7f63cc09b4c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f63d400b4f0 tx=0x7f63d4009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:38.975 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:38.969+0000 7f63e0ff9700 1 -- 172.21.15.47:0/812842124 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f63dc08f770 con 0x7f63e4101740 2022-09-07T21:38:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:38 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2240351153' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-09-07T21:38:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:38 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-09-07T21:38:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:38 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:38 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:38.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:38 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:38.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:38 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:38 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2240351153' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-09-07T21:38:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:38 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-09-07T21:38:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:38 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:38 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:39.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:38 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:39.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:38 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:39.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:38 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2240351153' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-09-07T21:38:39.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:38 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-09-07T21:38:39.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:38 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:39.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:38 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:39.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:38 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:39.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:38 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:39.193 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:39.191+0000 7f63e8b9d700 1 -- 172.21.15.47:0/812842124 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f63e405f120 con 0x7f63e4101740 2022-09-07T21:38:39.960 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:39.957+0000 7f63e0ff9700 1 -- 172.21.15.47:0/812842124 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) v1 ==== 171+0+0 (secure 0 0 0) 0x7f63dc092d90 con 0x7f63e4101740 2022-09-07T21:38:39.972 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:39.969+0000 7f63e8b9d700 1 -- 172.21.15.47:0/812842124 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f63cc099010 msgr2=0x7f63cc09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:39.972 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:39.969+0000 7f63e8b9d700 1 --2- 172.21.15.47:0/812842124 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f63cc099010 0x7f63cc09b4c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f63d400b4f0 tx=0x7f63d4009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:39.973 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:39.969+0000 7f63e8b9d700 1 -- 172.21.15.47:0/812842124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63e4101740 msgr2=0x7f63e41a1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:39.973 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:39.969+0000 7f63e8b9d700 1 --2- 172.21.15.47:0/812842124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63e4101740 0x7f63e41a1300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f63dc00eaf0 tx=0x7f63dc00ee00 comp rx=0 tx=0).stop 2022-09-07T21:38:39.974 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:39.972+0000 7f63e8b9d700 1 -- 172.21.15.47:0/812842124 shutdown_connections 2022-09-07T21:38:39.975 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:39.972+0000 7f63e8b9d700 1 --2- 172.21.15.47:0/812842124 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f63cc099010 0x7f63cc09b4c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:39.975 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:39.972+0000 7f63e8b9d700 1 --2- 172.21.15.47:0/812842124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63e4101740 0x7f63e41a1300 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:39.976 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:39.972+0000 7f63e8b9d700 1 --2- 172.21.15.47:0/812842124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63e40fc420 0x7f63e4199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:39.976 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:39.972+0000 7f63e8b9d700 1 --2- 172.21.15.47:0/812842124 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e40fed80 0x7f63e419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:39.977 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:39.972+0000 7f63e8b9d700 1 -- 172.21.15.47:0/812842124 >> 172.21.15.47:0/812842124 conn(0x7f63e40f80d0 msgr2=0x7f63e40ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:39.978 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:39.974+0000 7f63e8b9d700 1 -- 172.21.15.47:0/812842124 shutdown_connections 2022-09-07T21:38:39.978 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:39.974+0000 7f63e8b9d700 1 -- 172.21.15.47:0/812842124 wait complete. 2022-09-07T21:38:39.981 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' created 2022-09-07T21:38:40.010 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-09-07T21:38:40.204 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:39 smithi059 ceph-mon[97450]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:40.205 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:39 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-09-07T21:38:40.205 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:39 smithi059 ceph-mon[97450]: osdmap e82: 12 total, 12 up, 12 in 2022-09-07T21:38:40.206 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:39 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/812842124' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2022-09-07T21:38:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:39 smithi055 ceph-mon[97033]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:39 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-09-07T21:38:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:39 smithi055 ceph-mon[97033]: osdmap e82: 12 total, 12 up, 12 in 2022-09-07T21:38:40.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:39 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/812842124' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2022-09-07T21:38:40.260 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.257+0000 7f72b599d700 1 -- 172.21.15.47:0/3866924615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b0100f50 msgr2=0x7f72b0103330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:40.261 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.257+0000 7f72b599d700 1 --2- 172.21.15.47:0/3866924615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b0100f50 0x7f72b0103330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f729c009810 tx=0x7f729c009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:40.261 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.257+0000 7f72b599d700 1 -- 172.21.15.47:0/3866924615 shutdown_connections 2022-09-07T21:38:40.262 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.257+0000 7f72b599d700 1 --2- 172.21.15.47:0/3866924615 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72b0103870 0x7f72b0105c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.262 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.257+0000 7f72b599d700 1 --2- 172.21.15.47:0/3866924615 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72b00fe630 0x7f72b0100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.262 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.257+0000 7f72b599d700 1 --2- 172.21.15.47:0/3866924615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b0100f50 0x7f72b0103330 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.263 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.257+0000 7f72b599d700 1 -- 172.21.15.47:0/3866924615 >> 172.21.15.47:0/3866924615 conn(0x7f72b00f80d0 msgr2=0x7f72b00fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:40.263 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.257+0000 7f72b599d700 1 -- 172.21.15.47:0/3866924615 shutdown_connections 2022-09-07T21:38:40.263 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.257+0000 7f72b599d700 1 -- 172.21.15.47:0/3866924615 wait complete. 2022-09-07T21:38:40.264 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.258+0000 7f72b599d700 1 Processor -- start 2022-09-07T21:38:40.266 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.258+0000 7f72b599d700 1 -- start start 2022-09-07T21:38:40.266 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.259+0000 7f72b599d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72b00fe630 0x7f72b00fdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:40.267 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.259+0000 7f72b599d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72b0100f50 0x7f72b00fc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:40.267 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.259+0000 7f72b599d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b0103870 0x7f72b00fcb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:40.267 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.259+0000 7f72b599d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b00fd320 con 0x7f72b00fe630 2022-09-07T21:38:40.267 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.259+0000 7f72b599d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b00fd490 con 0x7f72b0100f50 2022-09-07T21:38:40.268 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.259+0000 7f72b599d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b019d7b0 con 0x7f72b0103870 2022-09-07T21:38:40.268 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.259+0000 7f72b519c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b0103870 0x7f72b00fcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:40.268 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.259+0000 7f72b499b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72b00fe630 0x7f72b00fdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:40.268 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.259+0000 7f72affff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72b0100f50 0x7f72b00fc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:40.268 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.259+0000 7f72b499b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72b00fe630 0x7f72b00fdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55312/0 (socket says 172.21.15.47:55312) 2022-09-07T21:38:40.269 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.259+0000 7f72b499b700 1 -- 172.21.15.47:0/760561082 learned_addr learned my addr 172.21.15.47:0/760561082 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:40.269 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.259+0000 7f72b519c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b0103870 0x7f72b00fcb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60354/0 (socket says 172.21.15.47:60354) 2022-09-07T21:38:40.269 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.260+0000 7f72b519c700 1 -- 172.21.15.47:0/760561082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72b0100f50 msgr2=0x7f72b00fc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:40.269 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.260+0000 7f72b519c700 1 --2- 172.21.15.47:0/760561082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72b0100f50 0x7f72b00fc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.260+0000 7f72b519c700 1 -- 172.21.15.47:0/760561082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72b00fe630 msgr2=0x7f72b00fdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.260+0000 7f72b519c700 1 --2- 172.21.15.47:0/760561082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72b00fe630 0x7f72b00fdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.260+0000 7f72b519c700 1 -- 172.21.15.47:0/760561082 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c009470 con 0x7f72b0103870 2022-09-07T21:38:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.260+0000 7f72b519c700 1 --2- 172.21.15.47:0/760561082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b0103870 0x7f72b00fcb80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f72a800eab0 tx=0x7f72a800edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.260+0000 7f72adffb700 1 -- 172.21.15.47:0/760561082 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a80098c0 con 0x7f72b0103870 2022-09-07T21:38:40.271 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.260+0000 7f72adffb700 1 -- 172.21.15.47:0/760561082 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f72a800ce10 con 0x7f72b0103870 2022-09-07T21:38:40.271 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.260+0000 7f72b599d700 1 -- 172.21.15.47:0/760561082 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b019da90 con 0x7f72b0103870 2022-09-07T21:38:40.271 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.260+0000 7f72adffb700 1 -- 172.21.15.47:0/760561082 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8004280 con 0x7f72b0103870 2022-09-07T21:38:40.271 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.260+0000 7f72b599d700 1 -- 172.21.15.47:0/760561082 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b019df50 con 0x7f72b0103870 2022-09-07T21:38:40.271 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.261+0000 7f72b599d700 1 -- 172.21.15.47:0/760561082 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b01092b0 con 0x7f72b0103870 2022-09-07T21:38:40.272 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.264+0000 7f72adffb700 1 -- 172.21.15.47:0/760561082 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f72a800c040 con 0x7f72b0103870 2022-09-07T21:38:40.273 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.265+0000 7f72adffb700 1 --2- 172.21.15.47:0/760561082 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f72a0099010 0x7f72a009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:40.273 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.265+0000 7f72adffb700 1 -- 172.21.15.47:0/760561082 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7875+0+0 (secure 0 0 0) 0x7f72a8014070 con 0x7f72b0103870 2022-09-07T21:38:40.273 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.265+0000 7f72b499b700 1 --2- 172.21.15.47:0/760561082 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f72a0099010 0x7f72a009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:40.274 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.266+0000 7f72b499b700 1 --2- 172.21.15.47:0/760561082 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f72a0099010 0x7f72a009b4c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f72a40057d0 tx=0x7f72a4005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:40.274 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.267+0000 7f72adffb700 1 -- 172.21.15.47:0/760561082 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f72a808f320 con 0x7f72b0103870 2022-09-07T21:38:40.317 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:39 smithi047 ceph-mon[90087]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:40.318 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:39 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-09-07T21:38:40.318 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:39 smithi047 ceph-mon[90087]: osdmap e82: 12 total, 12 up, 12 in 2022-09-07T21:38:40.318 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:39 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/812842124' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2022-09-07T21:38:40.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.541+0000 7f72b599d700 1 -- 172.21.15.47:0/760561082 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f72b00f9ac0 con 0x7f72b0103870 2022-09-07T21:38:40.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.561+0000 7f72adffb700 1 -- 172.21.15.47:0/760561082 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f72a8092940 con 0x7f72b0103870 2022-09-07T21:38:40.590 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.587+0000 7f72b599d700 1 -- 172.21.15.47:0/760561082 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f72a0099010 msgr2=0x7f72a009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:40.590 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.587+0000 7f72b599d700 1 --2- 172.21.15.47:0/760561082 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f72a0099010 0x7f72a009b4c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f72a40057d0 tx=0x7f72a4005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.590 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.587+0000 7f72b599d700 1 -- 172.21.15.47:0/760561082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b0103870 msgr2=0x7f72b00fcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:40.591 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.587+0000 7f72b599d700 1 --2- 172.21.15.47:0/760561082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b0103870 0x7f72b00fcb80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f72a800eab0 tx=0x7f72a800edc0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.591 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.587+0000 7f72b599d700 1 -- 172.21.15.47:0/760561082 shutdown_connections 2022-09-07T21:38:40.591 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.587+0000 7f72b599d700 1 --2- 172.21.15.47:0/760561082 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f72a0099010 0x7f72a009b4c0 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f72a40057d0 tx=0x7f72a4005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.591 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.587+0000 7f72b599d700 1 --2- 172.21.15.47:0/760561082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b0103870 0x7f72b00fcb80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.592 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.587+0000 7f72b599d700 1 --2- 172.21.15.47:0/760561082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72b00fe630 0x7f72b00fdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.592 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.587+0000 7f72b599d700 1 --2- 172.21.15.47:0/760561082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72b0100f50 0x7f72b00fc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.592 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.587+0000 7f72b599d700 1 -- 172.21.15.47:0/760561082 >> 172.21.15.47:0/760561082 conn(0x7f72b00f80d0 msgr2=0x7f72b00ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:40.593 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.590+0000 7f72b599d700 1 -- 172.21.15.47:0/760561082 shutdown_connections 2022-09-07T21:38:40.593 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.590+0000 7f72b599d700 1 -- 172.21.15.47:0/760561082 wait complete. 2022-09-07T21:38:40.595 INFO:teuthology.orchestra.run.smithi047.stderr:new fs with metadata pool 2 and data pool 3 2022-09-07T21:38:40.623 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2022-09-07T21:38:40.624 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2022-09-07T21:38:40.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.866+0000 7fddd3fff700 1 -- 172.21.15.47:0/3415973709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddd40ffe60 msgr2=0x7fddd4100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:40.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.866+0000 7fddd3fff700 1 --2- 172.21.15.47:0/3415973709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddd40ffe60 0x7fddd4100270 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009810 tx=0x7fddc4009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:40.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.866+0000 7fddd3fff700 1 -- 172.21.15.47:0/3415973709 shutdown_connections 2022-09-07T21:38:40.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.866+0000 7fddd3fff700 1 --2- 172.21.15.47:0/3415973709 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fddd41019a0 0x7fddd4108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.866+0000 7fddd3fff700 1 --2- 172.21.15.47:0/3415973709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddd40ffe60 0x7fddd4100270 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.866+0000 7fddd3fff700 1 --2- 172.21.15.47:0/3415973709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddd4100ff0 0x7fddd4101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.866+0000 7fddd3fff700 1 -- 172.21.15.47:0/3415973709 >> 172.21.15.47:0/3415973709 conn(0x7fddd40fb6b0 msgr2=0x7fddd40fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:40.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.866+0000 7fddd3fff700 1 -- 172.21.15.47:0/3415973709 shutdown_connections 2022-09-07T21:38:40.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.866+0000 7fddd3fff700 1 -- 172.21.15.47:0/3415973709 wait complete. 2022-09-07T21:38:40.876 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.867+0000 7fddd3fff700 1 Processor -- start 2022-09-07T21:38:40.877 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.867+0000 7fddd3fff700 1 -- start start 2022-09-07T21:38:40.878 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddd40ffe60 0x7fddd4195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:40.878 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddd4100ff0 0x7fddd4195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:40.879 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fddd41019a0 0x7fddd419ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:40.879 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd3fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd419d480 con 0x7fddd40ffe60 2022-09-07T21:38:40.879 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd3fff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd419d5f0 con 0x7fddd41019a0 2022-09-07T21:38:40.879 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd3fff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd419d7b0 con 0x7fddd4100ff0 2022-09-07T21:38:40.880 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddd40ffe60 0x7fddd4195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:40.880 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddd40ffe60 0x7fddd4195900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55328/0 (socket says 172.21.15.47:55328) 2022-09-07T21:38:40.880 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd2ffd700 1 -- 172.21.15.47:0/1475426787 learned_addr learned my addr 172.21.15.47:0/1475426787 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:40.880 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd27fc700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddd4100ff0 0x7fddd4195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:40.880 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd37fe700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fddd41019a0 0x7fddd419ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:40.881 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd2ffd700 1 -- 172.21.15.47:0/1475426787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddd4100ff0 msgr2=0x7fddd4195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:40.881 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd2ffd700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddd4100ff0 0x7fddd4195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.881 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd2ffd700 1 -- 172.21.15.47:0/1475426787 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fddd41019a0 msgr2=0x7fddd419ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:40.881 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.868+0000 7fddd2ffd700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fddd41019a0 0x7fddd419ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:40.882 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.869+0000 7fddd2ffd700 1 -- 172.21.15.47:0/1475426787 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc4009470 con 0x7fddd40ffe60 2022-09-07T21:38:40.882 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.869+0000 7fddd37fe700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fddd41019a0 0x7fddd419ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:38:40.882 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.869+0000 7fddd2ffd700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddd40ffe60 0x7fddd4195900 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fddc4018630 tx=0x7fddc4018660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:40.883 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.869+0000 7fddbbfff700 1 -- 172.21.15.47:0/1475426787 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc401d070 con 0x7fddd40ffe60 2022-09-07T21:38:40.883 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.869+0000 7fddbbfff700 1 -- 172.21.15.47:0/1475426787 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fddc4018930 con 0x7fddd40ffe60 2022-09-07T21:38:40.883 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.869+0000 7fddd3fff700 1 -- 172.21.15.47:0/1475426787 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd419da30 con 0x7fddd40ffe60 2022-09-07T21:38:40.883 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.869+0000 7fddbbfff700 1 -- 172.21.15.47:0/1475426787 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc40047d0 con 0x7fddd40ffe60 2022-09-07T21:38:40.884 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.869+0000 7fddd3fff700 1 -- 172.21.15.47:0/1475426787 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd419dee0 con 0x7fddd40ffe60 2022-09-07T21:38:40.884 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.870+0000 7fddd3fff700 1 -- 172.21.15.47:0/1475426787 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd410b530 con 0x7fddd40ffe60 2022-09-07T21:38:40.884 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.873+0000 7fddbbfff700 1 -- 172.21.15.47:0/1475426787 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fddc4029030 con 0x7fddd40ffe60 2022-09-07T21:38:40.884 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.874+0000 7fddbbfff700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fddbc099010 0x7fddbc09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:40.884 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.874+0000 7fddbbfff700 1 -- 172.21.15.47:0/1475426787 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fddc40c3850 con 0x7fddd40ffe60 2022-09-07T21:38:40.885 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.874+0000 7fddd27fc700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fddbc099010 0x7fddbc09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:40.885 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.875+0000 7fddd27fc700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fddbc099010 0x7fddbc09b4c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fddc8005fd0 tx=0x7fddc8005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:40.886 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:40.877+0000 7fddbbfff700 1 -- 172.21.15.47:0/1475426787 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fddc408fff0 con 0x7fddd40ffe60 2022-09-07T21:38:41.135 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.132+0000 7fddd3fff700 1 -- 172.21.15.47:0/1475426787 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fddd4106bb0 con 0x7fddd40ffe60 2022-09-07T21:38:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2022-09-07T21:38:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2022-09-07T21:38:41.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: osdmap e83: 12 total, 12 up, 12 in 2022-09-07T21:38:41.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:41.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:41.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:41.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:38:41.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:41.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:38:41.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/760561082' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-09-07T21:38:41.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-09-07T21:38:41.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: daemon mds.a assigned to filesystem cephfs as rank 0 2022-09-07T21:38:41.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: osdmap e84: 12 total, 12 up, 12 in 2022-09-07T21:38:41.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-09-07T21:38:41.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2022-09-07T21:38:41.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: pgmap v311: 97 pgs: 12 creating+peering, 75 unknown, 10 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:41.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:40 smithi055 ceph-mon[97033]: daemon mds.a is now active in filesystem cephfs as rank 0 2022-09-07T21:38:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2022-09-07T21:38:41.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2022-09-07T21:38:41.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: osdmap e83: 12 total, 12 up, 12 in 2022-09-07T21:38:41.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:41.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:41.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:41.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:38:41.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:41.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:38:41.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/760561082' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-09-07T21:38:41.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-09-07T21:38:41.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: daemon mds.a assigned to filesystem cephfs as rank 0 2022-09-07T21:38:41.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: osdmap e84: 12 total, 12 up, 12 in 2022-09-07T21:38:41.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-09-07T21:38:41.289 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2022-09-07T21:38:41.289 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: pgmap v311: 97 pgs: 12 creating+peering, 75 unknown, 10 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:41.290 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:40 smithi059 ceph-mon[97450]: daemon mds.a is now active in filesystem cephfs as rank 0 2022-09-07T21:38:41.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2022-09-07T21:38:41.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2022-09-07T21:38:41.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: osdmap e83: 12 total, 12 up, 12 in 2022-09-07T21:38:41.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:41.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:41.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:41.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:38:41.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:38:41.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:38:41.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/760561082' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-09-07T21:38:41.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-09-07T21:38:41.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: daemon mds.a assigned to filesystem cephfs as rank 0 2022-09-07T21:38:41.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: osdmap e84: 12 total, 12 up, 12 in 2022-09-07T21:38:41.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-09-07T21:38:41.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2022-09-07T21:38:41.407 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: pgmap v311: 97 pgs: 12 creating+peering, 75 unknown, 10 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:41.407 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:40 smithi047 ceph-mon[90087]: daemon mds.a is now active in filesystem cephfs as rank 0 2022-09-07T21:38:41.561 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.558+0000 7fddbbfff700 1 -- 172.21.15.47:0/1475426787 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) v1 ==== 157+0+0 (secure 0 0 0) 0x7fddc40270e0 con 0x7fddd40ffe60 2022-09-07T21:38:41.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.561+0000 7fddd3fff700 1 -- 172.21.15.47:0/1475426787 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fddbc099010 msgr2=0x7fddbc09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:41.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.561+0000 7fddd3fff700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fddbc099010 0x7fddbc09b4c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fddc8005fd0 tx=0x7fddc8005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:41.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.561+0000 7fddd3fff700 1 -- 172.21.15.47:0/1475426787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddd40ffe60 msgr2=0x7fddd4195900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:41.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.561+0000 7fddd3fff700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddd40ffe60 0x7fddd4195900 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fddc4018630 tx=0x7fddc4018660 comp rx=0 tx=0).stop 2022-09-07T21:38:41.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.561+0000 7fddd3fff700 1 -- 172.21.15.47:0/1475426787 shutdown_connections 2022-09-07T21:38:41.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.561+0000 7fddd3fff700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fddbc099010 0x7fddbc09b4c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:41.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.561+0000 7fddd3fff700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fddd41019a0 0x7fddd419ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:41.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.561+0000 7fddd3fff700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddd40ffe60 0x7fddd4195900 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:41.566 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.561+0000 7fddd3fff700 1 --2- 172.21.15.47:0/1475426787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddd4100ff0 0x7fddd4195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:41.566 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.561+0000 7fddd3fff700 1 -- 172.21.15.47:0/1475426787 >> 172.21.15.47:0/1475426787 conn(0x7fddd40fb6b0 msgr2=0x7fddd40fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:41.567 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.561+0000 7fddd3fff700 1 -- 172.21.15.47:0/1475426787 shutdown_connections 2022-09-07T21:38:41.567 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.561+0000 7fddd3fff700 1 -- 172.21.15.47:0/1475426787 wait complete. 2022-09-07T21:38:41.592 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2022-09-07T21:38:41.852 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.848+0000 7f7718bff700 1 -- 172.21.15.47:0/2024847549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77140fde00 msgr2=0x7f77140fe210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:41.852 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.848+0000 7f7718bff700 1 --2- 172.21.15.47:0/2024847549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77140fde00 0x7f77140fe210 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7704009810 tx=0x7f7704009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:41.853 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.848+0000 7f7718bff700 1 -- 172.21.15.47:0/2024847549 shutdown_connections 2022-09-07T21:38:41.853 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.848+0000 7f7718bff700 1 --2- 172.21.15.47:0/2024847549 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f77140ff990 0x7f77141062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:41.854 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.848+0000 7f7718bff700 1 --2- 172.21.15.47:0/2024847549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77140fde00 0x7f77140fe210 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.848+0000 7f7718bff700 1 --2- 172.21.15.47:0/2024847549 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77140fefe0 0x7f77140ff450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.848+0000 7f7718bff700 1 -- 172.21.15.47:0/2024847549 >> 172.21.15.47:0/2024847549 conn(0x7f77140f9510 msgr2=0x7f77140fb920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:41.856 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.848+0000 7f7718bff700 1 -- 172.21.15.47:0/2024847549 shutdown_connections 2022-09-07T21:38:41.856 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.848+0000 7f7718bff700 1 -- 172.21.15.47:0/2024847549 wait complete. 2022-09-07T21:38:41.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.850+0000 7f7718bff700 1 Processor -- start 2022-09-07T21:38:41.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.850+0000 7f7718bff700 1 -- start start 2022-09-07T21:38:41.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.851+0000 7f7718bff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77140fde00 0x7f7714072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:41.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.851+0000 7f7718bff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77140fefe0 0x7f77140710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:41.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.851+0000 7f7718bff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f77140ff990 0x7f7714071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:41.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.851+0000 7f7718bff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7714071e10 con 0x7f77140ff990 2022-09-07T21:38:41.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.851+0000 7f7718bff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7714071f80 con 0x7f77140fefe0 2022-09-07T21:38:41.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.851+0000 7f7718bff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771419da50 con 0x7f77140fde00 2022-09-07T21:38:41.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.851+0000 7f771259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77140fde00 0x7f7714072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:41.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.851+0000 7f771259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77140fde00 0x7f7714072a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60386/0 (socket says 172.21.15.47:60386) 2022-09-07T21:38:41.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.851+0000 7f771259c700 1 -- 172.21.15.47:0/1841511102 learned_addr learned my addr 172.21.15.47:0/1841511102 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:41.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.851+0000 7f7712d9d700 1 --2- 172.21.15.47:0/1841511102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f77140ff990 0x7f7714071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:41.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.852+0000 7f7711d9b700 1 --2- 172.21.15.47:0/1841511102 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77140fefe0 0x7f77140710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:41.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.852+0000 7f771259c700 1 -- 172.21.15.47:0/1841511102 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77140fefe0 msgr2=0x7f77140710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:41.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.852+0000 7f771259c700 1 --2- 172.21.15.47:0/1841511102 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77140fefe0 0x7f77140710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:41.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.852+0000 7f771259c700 1 -- 172.21.15.47:0/1841511102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f77140ff990 msgr2=0x7f7714071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:41.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.852+0000 7f771259c700 1 --2- 172.21.15.47:0/1841511102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f77140ff990 0x7f7714071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:41.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.852+0000 7f771259c700 1 -- 172.21.15.47:0/1841511102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7704009470 con 0x7f77140fde00 2022-09-07T21:38:41.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.852+0000 7f771259c700 1 --2- 172.21.15.47:0/1841511102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77140fde00 0x7f7714072a90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f770400f690 tx=0x7f7704009400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:41.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.852+0000 7f77037fe700 1 -- 172.21.15.47:0/1841511102 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770401d070 con 0x7f77140fde00 2022-09-07T21:38:41.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.853+0000 7f77037fe700 1 -- 172.21.15.47:0/1841511102 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f7704005030 con 0x7f77140fde00 2022-09-07T21:38:41.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.853+0000 7f7718bff700 1 -- 172.21.15.47:0/1841511102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771419dcd0 con 0x7f77140fde00 2022-09-07T21:38:41.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.853+0000 7f77037fe700 1 -- 172.21.15.47:0/1841511102 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77040048a0 con 0x7f77140fde00 2022-09-07T21:38:41.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.853+0000 7f7718bff700 1 -- 172.21.15.47:0/1841511102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771419e1d0 con 0x7f77140fde00 2022-09-07T21:38:41.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.854+0000 7f7718bff700 1 -- 172.21.15.47:0/1841511102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77141095b0 con 0x7f77140fde00 2022-09-07T21:38:41.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.856+0000 7f77037fe700 1 -- 172.21.15.47:0/1841511102 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f7704028030 con 0x7f77140fde00 2022-09-07T21:38:41.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.858+0000 7f77037fe700 1 --2- 172.21.15.47:0/1841511102 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f76fc099090 0x7f76fc09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:41.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.858+0000 7f77037fe700 1 -- 172.21.15.47:0/1841511102 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8201+0+0 (secure 0 0 0) 0x7f77040c2ff0 con 0x7f77140fde00 2022-09-07T21:38:41.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.858+0000 7f7711d9b700 1 --2- 172.21.15.47:0/1841511102 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f76fc099090 0x7f76fc09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:41.873 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.859+0000 7f7711d9b700 1 --2- 172.21.15.47:0/1841511102 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f76fc099090 0x7f76fc09b540 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f770800baf0 tx=0x7f770800b2e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:41.873 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:41.859+0000 7f77037fe700 1 -- 172.21.15.47:0/1841511102 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f770408f6c0 con 0x7f77140fde00 2022-09-07T21:38:42.072 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:42.070+0000 7f7718bff700 1 -- 172.21.15.47:0/1841511102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7714104c80 con 0x7f77140fde00 2022-09-07T21:38:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:41 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1475426787' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2022-09-07T21:38:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:41 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] up:active 2022-09-07T21:38:42.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:41 smithi055 ceph-mon[97033]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:38:42.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:41 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1475426787' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2022-09-07T21:38:42.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:41 smithi055 ceph-mon[97033]: osdmap e85: 12 total, 12 up, 12 in 2022-09-07T21:38:42.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:41 smithi055 ceph-mon[97033]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:38:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:41 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1475426787' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2022-09-07T21:38:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:41 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] up:active 2022-09-07T21:38:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:41 smithi059 ceph-mon[97450]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:38:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:41 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1475426787' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2022-09-07T21:38:42.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:41 smithi059 ceph-mon[97450]: osdmap e85: 12 total, 12 up, 12 in 2022-09-07T21:38:42.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:41 smithi059 ceph-mon[97450]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:38:42.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:41 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1475426787' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2022-09-07T21:38:42.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:41 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] up:active 2022-09-07T21:38:42.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:41 smithi047 ceph-mon[90087]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:38:42.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:41 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1475426787' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2022-09-07T21:38:42.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:41 smithi047 ceph-mon[90087]: osdmap e85: 12 total, 12 up, 12 in 2022-09-07T21:38:42.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:41 smithi047 ceph-mon[90087]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:38:42.903 INFO:journalctl@ceph.mds.a.smithi047.stdout:Sep 07 21:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-a[125500]: 2022-09-07T21:38:42.576+0000 7fdbbb178700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-09-07T21:38:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:42 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1841511102' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-09-07T21:38:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:42 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-09-07T21:38:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:42 smithi055 ceph-mon[97033]: pgmap v313: 97 pgs: 12 creating+peering, 59 unknown, 26 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-07T21:38:43.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:42 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1841511102' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-09-07T21:38:43.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:42 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-09-07T21:38:43.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:42 smithi059 ceph-mon[97450]: pgmap v313: 97 pgs: 12 creating+peering, 59 unknown, 26 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-07T21:38:43.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:42 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1841511102' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-09-07T21:38:43.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:42 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-09-07T21:38:43.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:42 smithi047 ceph-mon[90087]: pgmap v313: 97 pgs: 12 creating+peering, 59 unknown, 26 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-07T21:38:43.974 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:43.968+0000 7f77037fe700 1 -- 172.21.15.47:0/1841511102 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) v1 ==== 228+0+0 (secure 0 0 0) 0x7f7704026100 con 0x7f77140fde00 2022-09-07T21:38:43.981 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:43.978+0000 7f7718bff700 1 -- 172.21.15.47:0/1841511102 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f76fc099090 msgr2=0x7f76fc09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:43.981 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:43.978+0000 7f7718bff700 1 --2- 172.21.15.47:0/1841511102 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f76fc099090 0x7f76fc09b540 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f770800baf0 tx=0x7f770800b2e0 comp rx=0 tx=0).stop 2022-09-07T21:38:43.981 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:43.978+0000 7f7718bff700 1 -- 172.21.15.47:0/1841511102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77140fde00 msgr2=0x7f7714072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:43.982 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:43.978+0000 7f7718bff700 1 --2- 172.21.15.47:0/1841511102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77140fde00 0x7f7714072a90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f770400f690 tx=0x7f7704009400 comp rx=0 tx=0).stop 2022-09-07T21:38:43.982 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:43.978+0000 7f7718bff700 1 -- 172.21.15.47:0/1841511102 shutdown_connections 2022-09-07T21:38:43.982 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:43.978+0000 7f7718bff700 1 --2- 172.21.15.47:0/1841511102 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f76fc099090 0x7f76fc09b540 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:43.982 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:43.978+0000 7f7718bff700 1 --2- 172.21.15.47:0/1841511102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f77140ff990 0x7f7714071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:43.983 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:43.978+0000 7f7718bff700 1 --2- 172.21.15.47:0/1841511102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77140fde00 0x7f7714072a90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:43.983 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:43.978+0000 7f7718bff700 1 --2- 172.21.15.47:0/1841511102 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77140fefe0 0x7f77140710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:43.983 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:43.978+0000 7f7718bff700 1 -- 172.21.15.47:0/1841511102 >> 172.21.15.47:0/1841511102 conn(0x7f77140f9510 msgr2=0x7f77140fb760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:43.983 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:43.978+0000 7f7718bff700 1 -- 172.21.15.47:0/1841511102 shutdown_connections 2022-09-07T21:38:43.983 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:43.978+0000 7f7718bff700 1 -- 172.21.15.47:0/1841511102 wait complete. 2022-09-07T21:38:43.984 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data_ec' created 2022-09-07T21:38:44.008 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2022-09-07T21:38:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:43 smithi055 ceph-mon[97033]: osdmap e86: 12 total, 12 up, 12 in 2022-09-07T21:38:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:43 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:43 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:44.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:43 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:44.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:43 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:44.255 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.252+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/1075451436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc640fef90 msgr2=0x7fdc640ff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:44.255 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.252+0000 7fdc6a7ff700 1 --2- 172.21.15.47:0/1075451436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc640fef90 0x7fdc640ff400 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009810 tx=0x7fdc58009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:44.255 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.252+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/1075451436 shutdown_connections 2022-09-07T21:38:44.256 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.252+0000 7fdc6a7ff700 1 --2- 172.21.15.47:0/1075451436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdc640ff940 0x7fdc64106270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:44.256 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.252+0000 7fdc6a7ff700 1 --2- 172.21.15.47:0/1075451436 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc640fddb0 0x7fdc640fe1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:44.256 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.252+0000 7fdc6a7ff700 1 --2- 172.21.15.47:0/1075451436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc640fef90 0x7fdc640ff400 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:44.256 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.252+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/1075451436 >> 172.21.15.47:0/1075451436 conn(0x7fdc640f9460 msgr2=0x7fdc640fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:44.257 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.252+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/1075451436 shutdown_connections 2022-09-07T21:38:44.257 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.252+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/1075451436 wait complete. 2022-09-07T21:38:44.259 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.253+0000 7fdc6a7ff700 1 Processor -- start 2022-09-07T21:38:44.259 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.253+0000 7fdc6a7ff700 1 -- start start 2022-09-07T21:38:44.260 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.254+0000 7fdc6a7ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc640fddb0 0x7fdc64195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:44.260 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.254+0000 7fdc6a7ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc640fef90 0x7fdc641960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:44.260 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.254+0000 7fdc6a7ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdc640ff940 0x7fdc6419d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:44.260 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.254+0000 7fdc6a7ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6419d6f0 con 0x7fdc640fef90 2022-09-07T21:38:44.261 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.254+0000 7fdc6a7ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6419d860 con 0x7fdc640ff940 2022-09-07T21:38:44.261 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.254+0000 7fdc6a7ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6419da20 con 0x7fdc640fddb0 2022-09-07T21:38:44.261 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.254+0000 7fdc63fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc640fddb0 0x7fdc64195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:44.261 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.254+0000 7fdc637fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc640fef90 0x7fdc641960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:44.261 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.254+0000 7fdc63fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc640fddb0 0x7fdc64195b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60402/0 (socket says 172.21.15.47:60402) 2022-09-07T21:38:44.262 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.254+0000 7fdc63fff700 1 -- 172.21.15.47:0/532227924 learned_addr learned my addr 172.21.15.47:0/532227924 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:44.262 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.254+0000 7fdc68d9c700 1 --2- 172.21.15.47:0/532227924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdc640ff940 0x7fdc6419d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:44.262 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.255+0000 7fdc63fff700 1 -- 172.21.15.47:0/532227924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdc640ff940 msgr2=0x7fdc6419d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:44.262 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.255+0000 7fdc63fff700 1 --2- 172.21.15.47:0/532227924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdc640ff940 0x7fdc6419d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:44.263 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.255+0000 7fdc63fff700 1 -- 172.21.15.47:0/532227924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc640fef90 msgr2=0x7fdc641960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:44.263 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.255+0000 7fdc63fff700 1 --2- 172.21.15.47:0/532227924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc640fef90 0x7fdc641960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:44.263 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.255+0000 7fdc63fff700 1 -- 172.21.15.47:0/532227924 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc58009470 con 0x7fdc640fddb0 2022-09-07T21:38:44.263 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.255+0000 7fdc68d9c700 1 --2- 172.21.15.47:0/532227924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdc640ff940 0x7fdc6419d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:44.264 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.255+0000 7fdc63fff700 1 --2- 172.21.15.47:0/532227924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc640fddb0 0x7fdc64195b70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdc5400e740 tx=0x7fdc5400ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:44.264 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.255+0000 7fdc617fa700 1 -- 172.21.15.47:0/532227924 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc54005220 con 0x7fdc640fddb0 2022-09-07T21:38:44.264 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.255+0000 7fdc617fa700 1 -- 172.21.15.47:0/532227924 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fdc54005380 con 0x7fdc640fddb0 2022-09-07T21:38:44.264 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.255+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/532227924 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6419dd00 con 0x7fdc640fddb0 2022-09-07T21:38:44.264 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.255+0000 7fdc617fa700 1 -- 172.21.15.47:0/532227924 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc54004b70 con 0x7fdc640fddb0 2022-09-07T21:38:44.265 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.255+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/532227924 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6419e1b0 con 0x7fdc640fddb0 2022-09-07T21:38:44.265 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.256+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/532227924 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc64109560 con 0x7fdc640fddb0 2022-09-07T21:38:44.266 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.258+0000 7fdc617fa700 1 -- 172.21.15.47:0/532227924 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fdc5401e020 con 0x7fdc640fddb0 2022-09-07T21:38:44.266 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.260+0000 7fdc617fa700 1 --2- 172.21.15.47:0/532227924 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdc4c099090 0x7fdc4c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:44.267 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.260+0000 7fdc617fa700 1 -- 172.21.15.47:0/532227924 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8694+0+0 (secure 0 0 0) 0x7fdc54014070 con 0x7fdc640fddb0 2022-09-07T21:38:44.267 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.260+0000 7fdc637fe700 1 --2- 172.21.15.47:0/532227924 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdc4c099090 0x7fdc4c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:44.267 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.261+0000 7fdc637fe700 1 --2- 172.21.15.47:0/532227924 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdc4c099090 0x7fdc4c09b540 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdc5800f4d0 tx=0x7fdc58009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:44.267 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.262+0000 7fdc617fa700 1 -- 172.21.15.47:0/532227924 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fdc5408eeb0 con 0x7fdc640fddb0 2022-09-07T21:38:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:43 smithi059 ceph-mon[97450]: osdmap e86: 12 total, 12 up, 12 in 2022-09-07T21:38:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:43 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:43 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:43 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:43 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:44.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:43 smithi047 ceph-mon[90087]: osdmap e86: 12 total, 12 up, 12 in 2022-09-07T21:38:44.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:43 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:44.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:43 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:44.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:43 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:44.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:43 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:44.472 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.470+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/532227924 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fdc6419ed80 con 0x7fdc640fddb0 2022-09-07T21:38:44.975 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.972+0000 7fdc617fa700 1 -- 172.21.15.47:0/532227924 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdc540924d0 con 0x7fdc640fddb0 2022-09-07T21:38:44.980 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.978+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/532227924 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdc4c099090 msgr2=0x7fdc4c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:44.980 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.978+0000 7fdc6a7ff700 1 --2- 172.21.15.47:0/532227924 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdc4c099090 0x7fdc4c09b540 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdc5800f4d0 tx=0x7fdc58009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.978+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/532227924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc640fddb0 msgr2=0x7fdc64195b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:44.982 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.978+0000 7fdc6a7ff700 1 --2- 172.21.15.47:0/532227924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc640fddb0 0x7fdc64195b70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdc5400e740 tx=0x7fdc5400ea50 comp rx=0 tx=0).stop 2022-09-07T21:38:44.983 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.980+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/532227924 shutdown_connections 2022-09-07T21:38:44.984 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.980+0000 7fdc6a7ff700 1 --2- 172.21.15.47:0/532227924 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdc4c099090 0x7fdc4c09b540 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:44.984 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.980+0000 7fdc6a7ff700 1 --2- 172.21.15.47:0/532227924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdc640ff940 0x7fdc6419d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:44.985 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.980+0000 7fdc6a7ff700 1 --2- 172.21.15.47:0/532227924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc640fddb0 0x7fdc64195b70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:44.985 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.980+0000 7fdc6a7ff700 1 --2- 172.21.15.47:0/532227924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc640fef90 0x7fdc641960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:44.986 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.980+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/532227924 >> 172.21.15.47:0/532227924 conn(0x7fdc640f9460 msgr2=0x7fdc640fb740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:44.986 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.981+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/532227924 shutdown_connections 2022-09-07T21:38:44.986 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:44.981+0000 7fdc6a7ff700 1 -- 172.21.15.47:0/532227924 wait complete. 2022-09-07T21:38:44.994 INFO:teuthology.orchestra.run.smithi047.stderr:set pool 4 allow_ec_overwrites to true 2022-09-07T21:38:45.025 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2022-09-07T21:38:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:44 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-09-07T21:38:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:44 smithi059 ceph-mon[97450]: osdmap e87: 12 total, 12 up, 12 in 2022-09-07T21:38:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:44 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/532227924' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2022-09-07T21:38:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:44 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2022-09-07T21:38:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:44 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:45.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:44 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:45.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:44 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:45.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:44 smithi059 ceph-mon[97450]: pgmap v316: 129 pgs: 10 creating+peering, 28 unknown, 91 active+clean; 578 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 4 op/s 2022-09-07T21:38:45.266 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.258+0000 7f948cd9d700 1 -- 172.21.15.47:0/3897484202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9488100f50 msgr2=0x7f9488103330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:45.266 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.258+0000 7f948cd9d700 1 --2- 172.21.15.47:0/3897484202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9488100f50 0x7f9488103330 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f947c009810 tx=0x7f947c009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:45.267 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.258+0000 7f948cd9d700 1 -- 172.21.15.47:0/3897484202 shutdown_connections 2022-09-07T21:38:45.267 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.258+0000 7f948cd9d700 1 --2- 172.21.15.47:0/3897484202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9488103870 0x7f9488105c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.267 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.258+0000 7f948cd9d700 1 --2- 172.21.15.47:0/3897484202 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94880fe630 0x7f9488100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.268 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.258+0000 7f948cd9d700 1 --2- 172.21.15.47:0/3897484202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9488100f50 0x7f9488103330 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.268 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.258+0000 7f948cd9d700 1 -- 172.21.15.47:0/3897484202 >> 172.21.15.47:0/3897484202 conn(0x7f94880f80d0 msgr2=0x7f94880fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:45.269 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.258+0000 7f948cd9d700 1 -- 172.21.15.47:0/3897484202 shutdown_connections 2022-09-07T21:38:45.269 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.259+0000 7f948cd9d700 1 -- 172.21.15.47:0/3897484202 wait complete. 2022-09-07T21:38:45.270 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.260+0000 7f948cd9d700 1 Processor -- start 2022-09-07T21:38:45.270 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.260+0000 7f948cd9d700 1 -- start start 2022-09-07T21:38:45.270 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.260+0000 7f948cd9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94880fe630 0x7f94881958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:45.271 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.260+0000 7f948cd9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9488100f50 0x7f9488195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:45.271 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.260+0000 7f948cd9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9488103870 0x7f948819ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:45.272 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.260+0000 7f948cd9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948819d430 con 0x7f9488103870 2022-09-07T21:38:45.272 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.260+0000 7f948cd9d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948819d5a0 con 0x7f94880fe630 2022-09-07T21:38:45.273 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.260+0000 7f948cd9d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948819d760 con 0x7f9488100f50 2022-09-07T21:38:45.273 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.260+0000 7f9487fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9488103870 0x7f948819ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:45.273 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.261+0000 7f9487fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9488103870 0x7f948819ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55396/0 (socket says 172.21.15.47:55396) 2022-09-07T21:38:45.274 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.261+0000 7f9487fff700 1 -- 172.21.15.47:0/2008854989 learned_addr learned my addr 172.21.15.47:0/2008854989 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:45.274 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.261+0000 7f94877fe700 1 --2- 172.21.15.47:0/2008854989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94880fe630 0x7f94881958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:45.274 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.261+0000 7f9486ffd700 1 --2- 172.21.15.47:0/2008854989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9488100f50 0x7f9488195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:45.275 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.261+0000 7f9487fff700 1 -- 172.21.15.47:0/2008854989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9488100f50 msgr2=0x7f9488195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:45.275 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.261+0000 7f9487fff700 1 --2- 172.21.15.47:0/2008854989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9488100f50 0x7f9488195df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.275 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.261+0000 7f9487fff700 1 -- 172.21.15.47:0/2008854989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94880fe630 msgr2=0x7f94881958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:45.275 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.261+0000 7f9487fff700 1 --2- 172.21.15.47:0/2008854989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94880fe630 0x7f94881958b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.276 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.261+0000 7f9487fff700 1 -- 172.21.15.47:0/2008854989 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947c009470 con 0x7f9488103870 2022-09-07T21:38:45.276 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.261+0000 7f9487fff700 1 --2- 172.21.15.47:0/2008854989 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9488103870 0x7f948819ce60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f948000ebf0 tx=0x7f948000ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:45.276 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.262+0000 7f9484ff9700 1 -- 172.21.15.47:0/2008854989 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94800099c0 con 0x7f9488103870 2022-09-07T21:38:45.276 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.262+0000 7f948cd9d700 1 -- 172.21.15.47:0/2008854989 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f948819da40 con 0x7f9488103870 2022-09-07T21:38:45.277 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.262+0000 7f9484ff9700 1 -- 172.21.15.47:0/2008854989 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f948000cc90 con 0x7f9488103870 2022-09-07T21:38:45.277 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.262+0000 7f9484ff9700 1 -- 172.21.15.47:0/2008854989 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9480010430 con 0x7f9488103870 2022-09-07T21:38:45.277 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.262+0000 7f948cd9d700 1 -- 172.21.15.47:0/2008854989 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f948819df50 con 0x7f9488103870 2022-09-07T21:38:45.277 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.263+0000 7f948cd9d700 1 -- 172.21.15.47:0/2008854989 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94881092b0 con 0x7f9488103870 2022-09-07T21:38:45.279 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.266+0000 7f9484ff9700 1 -- 172.21.15.47:0/2008854989 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f948001e020 con 0x7f9488103870 2022-09-07T21:38:45.279 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.267+0000 7f9484ff9700 1 --2- 172.21.15.47:0/2008854989 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9470099010 0x7f947009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:45.279 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.267+0000 7f9484ff9700 1 -- 172.21.15.47:0/2008854989 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8694+0+0 (secure 0 0 0) 0x7f9480014070 con 0x7f9488103870 2022-09-07T21:38:45.279 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.268+0000 7f94877fe700 1 --2- 172.21.15.47:0/2008854989 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9470099010 0x7f947009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:45.280 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.268+0000 7f94877fe700 1 --2- 172.21.15.47:0/2008854989 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9470099010 0x7f947009b4c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9478005fd0 tx=0x7f9478005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:45.280 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.271+0000 7f9484ff9700 1 -- 172.21.15.47:0/2008854989 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f948008f1a0 con 0x7f9488103870 2022-09-07T21:38:45.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:44 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-09-07T21:38:45.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:44 smithi047 ceph-mon[90087]: osdmap e87: 12 total, 12 up, 12 in 2022-09-07T21:38:45.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:44 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/532227924' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2022-09-07T21:38:45.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:44 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2022-09-07T21:38:45.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:44 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:45.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:44 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:45.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:44 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:45.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:44 smithi047 ceph-mon[90087]: pgmap v316: 129 pgs: 10 creating+peering, 28 unknown, 91 active+clean; 578 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 4 op/s 2022-09-07T21:38:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:44 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-09-07T21:38:45.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:44 smithi055 ceph-mon[97033]: osdmap e87: 12 total, 12 up, 12 in 2022-09-07T21:38:45.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:44 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/532227924' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2022-09-07T21:38:45.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:44 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2022-09-07T21:38:45.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:44 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:45.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:44 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:45.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:44 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:38:45.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:44 smithi055 ceph-mon[97033]: pgmap v316: 129 pgs: 10 creating+peering, 28 unknown, 91 active+clean; 578 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 4 op/s 2022-09-07T21:38:45.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.528+0000 7f948cd9d700 1 -- 172.21.15.47:0/2008854989 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f9488109440 con 0x7f9488103870 2022-09-07T21:38:45.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.534+0000 7f9484ff9700 1 -- 172.21.15.47:0/2008854989 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f9488109440 con 0x7f9488103870 2022-09-07T21:38:45.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.536+0000 7f948cd9d700 1 -- 172.21.15.47:0/2008854989 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9470099010 msgr2=0x7f947009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:45.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.536+0000 7f948cd9d700 1 --2- 172.21.15.47:0/2008854989 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9470099010 0x7f947009b4c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9478005fd0 tx=0x7f9478005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.536+0000 7f948cd9d700 1 -- 172.21.15.47:0/2008854989 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9488103870 msgr2=0x7f948819ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:45.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.536+0000 7f948cd9d700 1 --2- 172.21.15.47:0/2008854989 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9488103870 0x7f948819ce60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f948000ebf0 tx=0x7f948000ef00 comp rx=0 tx=0).stop 2022-09-07T21:38:45.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.536+0000 7f948cd9d700 1 -- 172.21.15.47:0/2008854989 shutdown_connections 2022-09-07T21:38:45.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.536+0000 7f948cd9d700 1 --2- 172.21.15.47:0/2008854989 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9470099010 0x7f947009b4c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.536+0000 7f948cd9d700 1 --2- 172.21.15.47:0/2008854989 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9488103870 0x7f948819ce60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.536+0000 7f948cd9d700 1 --2- 172.21.15.47:0/2008854989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94880fe630 0x7f94881958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.536+0000 7f948cd9d700 1 --2- 172.21.15.47:0/2008854989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9488100f50 0x7f9488195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.536+0000 7f948cd9d700 1 -- 172.21.15.47:0/2008854989 >> 172.21.15.47:0/2008854989 conn(0x7f94880f80d0 msgr2=0x7f94880fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:45.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.537+0000 7f948cd9d700 1 -- 172.21.15.47:0/2008854989 shutdown_connections 2022-09-07T21:38:45.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.537+0000 7f948cd9d700 1 -- 172.21.15.47:0/2008854989 wait complete. 2022-09-07T21:38:45.542 INFO:teuthology.orchestra.run.smithi047.stderr:added data pool 4 to fsmap 2022-09-07T21:38:45.569 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.801+0000 7fcbe25ff700 1 -- 172.21.15.47:0/517165499 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbdc105bd0 msgr2=0x7fcbdc108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.801+0000 7fcbe25ff700 1 --2- 172.21.15.47:0/517165499 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbdc105bd0 0x7fcbdc108010 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009bb0 tx=0x7fcbd4009ec0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.801+0000 7fcbe25ff700 1 -- 172.21.15.47:0/517165499 shutdown_connections 2022-09-07T21:38:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.801+0000 7fcbe25ff700 1 --2- 172.21.15.47:0/517165499 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbdc105bd0 0x7fcbdc108010 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.801+0000 7fcbe25ff700 1 --2- 172.21.15.47:0/517165499 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbdc072c90 0x7fcbdc071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.801+0000 7fcbe25ff700 1 --2- 172.21.15.47:0/517165499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbdc071740 0x7fcbdc071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.801+0000 7fcbe25ff700 1 -- 172.21.15.47:0/517165499 >> 172.21.15.47:0/517165499 conn(0x7fcbdc0f93c0 msgr2=0x7fcbdc0fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.802+0000 7fcbe25ff700 1 -- 172.21.15.47:0/517165499 shutdown_connections 2022-09-07T21:38:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.802+0000 7fcbe25ff700 1 -- 172.21.15.47:0/517165499 wait complete. 2022-09-07T21:38:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.803+0000 7fcbe25ff700 1 Processor -- start 2022-09-07T21:38:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.803+0000 7fcbe25ff700 1 -- start start 2022-09-07T21:38:45.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.803+0000 7fcbe25ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbdc071740 0x7fcbdc195b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.803+0000 7fcbe25ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbdc072c90 0x7fcbdc196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.803+0000 7fcbe25ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbdc105bd0 0x7fcbdc19d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.803+0000 7fcbe25ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc19d680 con 0x7fcbdc071740 2022-09-07T21:38:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.803+0000 7fcbe25ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc19d7f0 con 0x7fcbdc072c90 2022-09-07T21:38:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.803+0000 7fcbe25ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc19d9b0 con 0x7fcbdc105bd0 2022-09-07T21:38:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.804+0000 7fcbdbfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbdc071740 0x7fcbdc195b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.804+0000 7fcbdbfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbdc071740 0x7fcbdc195b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55408/0 (socket says 172.21.15.47:55408) 2022-09-07T21:38:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.804+0000 7fcbdbfff700 1 -- 172.21.15.47:0/2027217935 learned_addr learned my addr 172.21.15.47:0/2027217935 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.804+0000 7fcbdb7fe700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbdc072c90 0x7fcbdc196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.804+0000 7fcbe0b9c700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbdc105bd0 0x7fcbdc19d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.804+0000 7fcbdbfff700 1 -- 172.21.15.47:0/2027217935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbdc105bd0 msgr2=0x7fcbdc19d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.804+0000 7fcbdbfff700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbdc105bd0 0x7fcbdc19d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.804+0000 7fcbdbfff700 1 -- 172.21.15.47:0/2027217935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbdc072c90 msgr2=0x7fcbdc196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.804+0000 7fcbdbfff700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbdc072c90 0x7fcbdc196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.804+0000 7fcbdbfff700 1 -- 172.21.15.47:0/2027217935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd40097e0 con 0x7fcbdc071740 2022-09-07T21:38:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.804+0000 7fcbe0b9c700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbdc105bd0 0x7fcbdc19d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:38:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.804+0000 7fcbdbfff700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbdc071740 0x7fcbdc195b00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00e740 tx=0x7fcbcc00ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.805+0000 7fcbd97fa700 1 -- 172.21.15.47:0/2027217935 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc005220 con 0x7fcbdc071740 2022-09-07T21:38:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.805+0000 7fcbd97fa700 1 -- 172.21.15.47:0/2027217935 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcbcc005380 con 0x7fcbdc071740 2022-09-07T21:38:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.805+0000 7fcbd97fa700 1 -- 172.21.15.47:0/2027217935 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc004c10 con 0x7fcbdc071740 2022-09-07T21:38:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.805+0000 7fcbe25ff700 1 -- 172.21.15.47:0/2027217935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbdc19dc90 con 0x7fcbdc071740 2022-09-07T21:38:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.805+0000 7fcbe25ff700 1 -- 172.21.15.47:0/2027217935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbdc19e1a0 con 0x7fcbdc071740 2022-09-07T21:38:45.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.806+0000 7fcbe25ff700 1 -- 172.21.15.47:0/2027217935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbdc10b6c0 con 0x7fcbdc071740 2022-09-07T21:38:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.808+0000 7fcbd97fa700 1 -- 172.21.15.47:0/2027217935 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fcbcc01e020 con 0x7fcbdc071740 2022-09-07T21:38:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.809+0000 7fcbd97fa700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcbc4099080 0x7fcbc409b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.809+0000 7fcbd97fa700 1 -- 172.21.15.47:0/2027217935 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fcbcc014070 con 0x7fcbdc071740 2022-09-07T21:38:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.809+0000 7fcbdb7fe700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcbc4099080 0x7fcbc409b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.810+0000 7fcbdb7fe700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcbc4099080 0x7fcbc409b530 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0005fd0 tx=0x7fcbd0005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:45.813+0000 7fcbd97fa700 1 -- 172.21.15.47:0/2027217935 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fcbcc08ed90 con 0x7fcbdc071740 2022-09-07T21:38:46.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.063+0000 7fcbe25ff700 1 -- 172.21.15.47:0/2027217935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbdc05f8f0 con 0x7fcbdc071740 2022-09-07T21:38:46.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.064+0000 7fcbd97fa700 1 -- 172.21.15.47:0/2027217935 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10649 (secure 0 0 0) 0x7fcbcc0923b0 con 0x7fcbdc071740 2022-09-07T21:38:46.067 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:46.067 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/2559266585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2559266585},{"type":"v1","addr":"172.21.15.55:6835","nonce":2559266585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/2090942297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2090942297},{"type":"v1","addr":"172.21.15.55:6837","nonce":2090942297}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24526,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2445409240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2445409240},{"type":"v1","addr":"172.21.15.59:6835","nonce":2445409240}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/3542466465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":3542466465},{"type":"v1","addr":"172.21.15.59:6837","nonce":3542466465}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24548,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6839/315505812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":315505812},{"type":"v1","addr":"172.21.15.47:6839","nonce":315505812}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6839/1006724740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":1006724740},{"type":"v1","addr":"172.21.15.59:6839","nonce":1006724740}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24572,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6839/517990300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":517990300},{"type":"v1","addr":"172.21.15.55:6839","nonce":517990300}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-09-07T21:38:40.544389+0000","modified":"2022-09-07T21:38:45.529512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24586":{"gid":24586,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6841/793863162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6840","nonce":793863162},{"type":"v1","addr":"172.21.15.47:6841","nonce":793863162}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2022-09-07T21:38:46.069 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.066+0000 7fcbe25ff700 1 -- 172.21.15.47:0/2027217935 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcbc4099080 msgr2=0x7fcbc409b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:46.069 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.066+0000 7fcbe25ff700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcbc4099080 0x7fcbc409b530 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0005fd0 tx=0x7fcbd0005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.066+0000 7fcbe25ff700 1 -- 172.21.15.47:0/2027217935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbdc071740 msgr2=0x7fcbdc195b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:46.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.066+0000 7fcbe25ff700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbdc071740 0x7fcbdc195b00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00e740 tx=0x7fcbcc00ea50 comp rx=0 tx=0).stop 2022-09-07T21:38:46.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.067+0000 7fcbe25ff700 1 -- 172.21.15.47:0/2027217935 shutdown_connections 2022-09-07T21:38:46.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.067+0000 7fcbe25ff700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcbc4099080 0x7fcbc409b530 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.067+0000 7fcbe25ff700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbdc105bd0 0x7fcbdc19d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.067+0000 7fcbe25ff700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbdc071740 0x7fcbdc195b00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.067+0000 7fcbe25ff700 1 --2- 172.21.15.47:0/2027217935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbdc072c90 0x7fcbdc196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.067+0000 7fcbe25ff700 1 -- 172.21.15.47:0/2027217935 >> 172.21.15.47:0/2027217935 conn(0x7fcbdc0f93c0 msgr2=0x7fcbdc0fb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:46.072 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.067+0000 7fcbe25ff700 1 -- 172.21.15.47:0/2027217935 shutdown_connections 2022-09-07T21:38:46.072 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.067+0000 7fcbe25ff700 1 -- 172.21.15.47:0/2027217935 wait complete. 2022-09-07T21:38:46.073 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 17 2022-09-07T21:38:46.097 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-07T21:38:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:45 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2022-09-07T21:38:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:45 smithi059 ceph-mon[97450]: osdmap e88: 12 total, 12 up, 12 in 2022-09-07T21:38:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:45 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2008854989' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2022-09-07T21:38:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:45 smithi059 ceph-mon[97450]: osdmap e89: 12 total, 12 up, 12 in 2022-09-07T21:38:46.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:45 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2008854989' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2022-09-07T21:38:46.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:45 smithi059 ceph-mon[97450]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:38:46.329 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.326+0000 7fd7579ff700 1 -- 172.21.15.47:0/4003261086 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd75010a370 msgr2=0x7fd75010c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:46.329 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.326+0000 7fd7579ff700 1 --2- 172.21.15.47:0/4003261086 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd75010a370 0x7fd75010c750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd748009bb0 tx=0x7fd748009ec0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.330 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.326+0000 7fd7579ff700 1 -- 172.21.15.47:0/4003261086 shutdown_connections 2022-09-07T21:38:46.330 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.326+0000 7fd7579ff700 1 --2- 172.21.15.47:0/4003261086 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd75010a370 0x7fd75010c750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.330 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.326+0000 7fd7579ff700 1 --2- 172.21.15.47:0/4003261086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd750072c90 0x7fd750071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.330 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.326+0000 7fd7579ff700 1 --2- 172.21.15.47:0/4003261086 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd750071740 0x7fd750071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.331 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.326+0000 7fd7579ff700 1 -- 172.21.15.47:0/4003261086 >> 172.21.15.47:0/4003261086 conn(0x7fd7500f93c0 msgr2=0x7fd7500fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:46.331 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.326+0000 7fd7579ff700 1 -- 172.21.15.47:0/4003261086 shutdown_connections 2022-09-07T21:38:46.331 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.326+0000 7fd7579ff700 1 -- 172.21.15.47:0/4003261086 wait complete. 2022-09-07T21:38:46.333 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.327+0000 7fd7579ff700 1 Processor -- start 2022-09-07T21:38:46.334 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.328+0000 7fd7579ff700 1 -- start start 2022-09-07T21:38:46.334 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.328+0000 7fd7579ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd750071740 0x7fd750199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:46.334 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.328+0000 7fd7579ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd750072c90 0x7fd75019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:46.334 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.328+0000 7fd7579ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd75010a370 0x7fd7501a14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:46.335 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.328+0000 7fd7579ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7501a1ac0 con 0x7fd750071740 2022-09-07T21:38:46.335 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.328+0000 7fd7579ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7501a1c30 con 0x7fd75010a370 2022-09-07T21:38:46.335 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.328+0000 7fd7579ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7501a1df0 con 0x7fd750072c90 2022-09-07T21:38:46.336 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.328+0000 7fd75579b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd750071740 0x7fd750199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:46.336 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.328+0000 7fd754f9a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd750072c90 0x7fd75019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:46.336 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.328+0000 7fd75579b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd750071740 0x7fd750199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55434/0 (socket says 172.21.15.47:55434) 2022-09-07T21:38:46.336 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.328+0000 7fd75579b700 1 -- 172.21.15.47:0/968628206 learned_addr learned my addr 172.21.15.47:0/968628206 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:46.336 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.328+0000 7fd755f9c700 1 --2- 172.21.15.47:0/968628206 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd75010a370 0x7fd7501a14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:46.337 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.329+0000 7fd75579b700 1 -- 172.21.15.47:0/968628206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd750072c90 msgr2=0x7fd75019a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:46.337 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.329+0000 7fd75579b700 1 --2- 172.21.15.47:0/968628206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd750072c90 0x7fd75019a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.337 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.329+0000 7fd75579b700 1 -- 172.21.15.47:0/968628206 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd75010a370 msgr2=0x7fd7501a14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:46.337 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.329+0000 7fd75579b700 1 --2- 172.21.15.47:0/968628206 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd75010a370 0x7fd7501a14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.338 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.329+0000 7fd75579b700 1 -- 172.21.15.47:0/968628206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7480097e0 con 0x7fd750071740 2022-09-07T21:38:46.338 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.329+0000 7fd75579b700 1 --2- 172.21.15.47:0/968628206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd750071740 0x7fd750199f40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd74c00e740 tx=0x7fd74c00ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:46.338 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.329+0000 7fd7467fc700 1 -- 172.21.15.47:0/968628206 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c005220 con 0x7fd750071740 2022-09-07T21:38:46.338 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.329+0000 7fd7579ff700 1 -- 172.21.15.47:0/968628206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7501a20d0 con 0x7fd750071740 2022-09-07T21:38:46.339 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.329+0000 7fd7467fc700 1 -- 172.21.15.47:0/968628206 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd74c005380 con 0x7fd750071740 2022-09-07T21:38:46.339 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.329+0000 7fd7467fc700 1 -- 172.21.15.47:0/968628206 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c004b70 con 0x7fd750071740 2022-09-07T21:38:46.339 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.329+0000 7fd7579ff700 1 -- 172.21.15.47:0/968628206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7501a2630 con 0x7fd750071740 2022-09-07T21:38:46.339 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.330+0000 7fd7579ff700 1 -- 172.21.15.47:0/968628206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd75010fae0 con 0x7fd750071740 2022-09-07T21:38:46.340 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.332+0000 7fd7467fc700 1 -- 172.21.15.47:0/968628206 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fd74c01e020 con 0x7fd750071740 2022-09-07T21:38:46.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.334+0000 7fd7467fc700 1 --2- 172.21.15.47:0/968628206 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd73c099080 0x7fd73c09b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:46.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.335+0000 7fd7467fc700 1 -- 172.21.15.47:0/968628206 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fd74c014070 con 0x7fd750071740 2022-09-07T21:38:46.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.335+0000 7fd754f9a700 1 --2- 172.21.15.47:0/968628206 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd73c099080 0x7fd73c09b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:46.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.335+0000 7fd754f9a700 1 --2- 172.21.15.47:0/968628206 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd73c099080 0x7fd73c09b530 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd740005fd0 tx=0x7fd740005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:46.342 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.338+0000 7fd7467fc700 1 -- 172.21.15.47:0/968628206 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fd74c08eb30 con 0x7fd750071740 2022-09-07T21:38:46.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:45 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2022-09-07T21:38:46.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:45 smithi047 ceph-mon[90087]: osdmap e88: 12 total, 12 up, 12 in 2022-09-07T21:38:46.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:45 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2008854989' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2022-09-07T21:38:46.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:45 smithi047 ceph-mon[90087]: osdmap e89: 12 total, 12 up, 12 in 2022-09-07T21:38:46.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:45 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2008854989' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2022-09-07T21:38:46.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:45 smithi047 ceph-mon[90087]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:38:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:45 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2022-09-07T21:38:46.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:45 smithi055 ceph-mon[97033]: osdmap e88: 12 total, 12 up, 12 in 2022-09-07T21:38:46.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:45 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2008854989' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2022-09-07T21:38:46.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:45 smithi055 ceph-mon[97033]: osdmap e89: 12 total, 12 up, 12 in 2022-09-07T21:38:46.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:45 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2008854989' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2022-09-07T21:38:46.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:45 smithi055 ceph-mon[97033]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:38:46.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.541+0000 7fd7579ff700 1 -- 172.21.15.47:0/968628206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7500ffd40 con 0x7fd750071740 2022-09-07T21:38:46.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.542+0000 7fd7467fc700 1 -- 172.21.15.47:0/968628206 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23119 (secure 0 0 0) 0x7fd74c092150 con 0x7fd750071740 2022-09-07T21:38:46.545 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:46.545 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":89,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:38:45.529264+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-07T21:38:38.608937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-07T21:38:39.936252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-07T21:38:43.731797+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:38:46.547 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.545+0000 7fd7579ff700 1 -- 172.21.15.47:0/968628206 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd73c099080 msgr2=0x7fd73c09b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:46.547 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.545+0000 7fd7579ff700 1 --2- 172.21.15.47:0/968628206 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd73c099080 0x7fd73c09b530 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd740005fd0 tx=0x7fd740005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.548 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.545+0000 7fd7579ff700 1 -- 172.21.15.47:0/968628206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd750071740 msgr2=0x7fd750199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:46.548 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.545+0000 7fd7579ff700 1 --2- 172.21.15.47:0/968628206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd750071740 0x7fd750199f40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd74c00e740 tx=0x7fd74c00ea50 comp rx=0 tx=0).stop 2022-09-07T21:38:46.548 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.545+0000 7fd7579ff700 1 -- 172.21.15.47:0/968628206 shutdown_connections 2022-09-07T21:38:46.548 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.545+0000 7fd7579ff700 1 --2- 172.21.15.47:0/968628206 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd73c099080 0x7fd73c09b530 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.549 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.545+0000 7fd7579ff700 1 --2- 172.21.15.47:0/968628206 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd75010a370 0x7fd7501a14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.549 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.545+0000 7fd7579ff700 1 --2- 172.21.15.47:0/968628206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd750071740 0x7fd750199f40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.549 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.545+0000 7fd7579ff700 1 --2- 172.21.15.47:0/968628206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd750072c90 0x7fd75019a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.549 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.545+0000 7fd7579ff700 1 -- 172.21.15.47:0/968628206 >> 172.21.15.47:0/968628206 conn(0x7fd7500f93c0 msgr2=0x7fd7500fb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:46.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.545+0000 7fd7579ff700 1 -- 172.21.15.47:0/968628206 shutdown_connections 2022-09-07T21:38:46.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.545+0000 7fd7579ff700 1 -- 172.21.15.47:0/968628206 wait complete. 2022-09-07T21:38:46.576 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-07T21:38:46.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.806+0000 7f1fd759d700 1 -- 172.21.15.47:0/2506398833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fd00fc420 msgr2=0x7f1fd00fc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:46.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.806+0000 7f1fd759d700 1 --2- 172.21.15.47:0/2506398833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fd00fc420 0x7f1fd00fc830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009810 tx=0x7f1fcc009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:46.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.807+0000 7f1fd759d700 1 -- 172.21.15.47:0/2506398833 shutdown_connections 2022-09-07T21:38:46.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.807+0000 7f1fd759d700 1 --2- 172.21.15.47:0/2506398833 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fd00fd720 0x7f1fd00fdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.807+0000 7f1fd759d700 1 --2- 172.21.15.47:0/2506398833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fd00fc420 0x7f1fd00fc830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.807+0000 7f1fd759d700 1 --2- 172.21.15.47:0/2506398833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fd00fcd70 0x7f1fd00fd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.807+0000 7f1fd759d700 1 -- 172.21.15.47:0/2506398833 >> 172.21.15.47:0/2506398833 conn(0x7f1fd00f80d0 msgr2=0x7f1fd00fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:46.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.807+0000 7f1fd759d700 1 -- 172.21.15.47:0/2506398833 shutdown_connections 2022-09-07T21:38:46.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.807+0000 7f1fd759d700 1 -- 172.21.15.47:0/2506398833 wait complete. 2022-09-07T21:38:46.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.808+0000 7f1fd759d700 1 Processor -- start 2022-09-07T21:38:46.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.808+0000 7f1fd759d700 1 -- start start 2022-09-07T21:38:46.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd759d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fd00fc420 0x7f1fd0195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:46.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd759d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fd00fcd70 0x7f1fd0195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:46.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd759d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fd00fd720 0x7f1fd019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:46.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd759d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd019d4c0 con 0x7f1fd00fc420 2022-09-07T21:38:46.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd759d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd019d630 con 0x7f1fd00fcd70 2022-09-07T21:38:46.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd759d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd019d7f0 con 0x7f1fd00fd720 2022-09-07T21:38:46.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd659b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fd00fc420 0x7f1fd0195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:46.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd659b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fd00fc420 0x7f1fd0195940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55448/0 (socket says 172.21.15.47:55448) 2022-09-07T21:38:46.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd659b700 1 -- 172.21.15.47:0/434150464 learned_addr learned my addr 172.21.15.47:0/434150464 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:46.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd5d9a700 1 --2- 172.21.15.47:0/434150464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fd00fcd70 0x7f1fd0195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:46.818 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd6d9c700 1 --2- 172.21.15.47:0/434150464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fd00fd720 0x7f1fd019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:46.818 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd659b700 1 -- 172.21.15.47:0/434150464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fd00fd720 msgr2=0x7f1fd019cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:46.818 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd659b700 1 --2- 172.21.15.47:0/434150464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fd00fd720 0x7f1fd019cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.819 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd659b700 1 -- 172.21.15.47:0/434150464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fd00fcd70 msgr2=0x7f1fd0195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:46.819 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd659b700 1 --2- 172.21.15.47:0/434150464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fd00fcd70 0x7f1fd0195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:46.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.809+0000 7f1fd659b700 1 -- 172.21.15.47:0/434150464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc009470 con 0x7f1fd00fc420 2022-09-07T21:38:46.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.810+0000 7f1fd659b700 1 --2- 172.21.15.47:0/434150464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fd00fc420 0x7f1fd0195940 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc000c00 tx=0x7f1fcc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:46.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.810+0000 7f1fc77fe700 1 -- 172.21.15.47:0/434150464 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc01d070 con 0x7f1fd00fc420 2022-09-07T21:38:46.821 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.810+0000 7f1fd759d700 1 -- 172.21.15.47:0/434150464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd019da70 con 0x7f1fd00fc420 2022-09-07T21:38:46.821 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.810+0000 7f1fc77fe700 1 -- 172.21.15.47:0/434150464 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1fcc00f8b0 con 0x7f1fd00fc420 2022-09-07T21:38:46.822 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.810+0000 7f1fc77fe700 1 -- 172.21.15.47:0/434150464 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc0047f0 con 0x7f1fd00fc420 2022-09-07T21:38:46.822 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.810+0000 7f1fd759d700 1 -- 172.21.15.47:0/434150464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd019df70 con 0x7f1fd00fc420 2022-09-07T21:38:46.823 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.811+0000 7f1fd759d700 1 -- 172.21.15.47:0/434150464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd010b4c0 con 0x7f1fd00fc420 2022-09-07T21:38:46.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.813+0000 7f1fc77fe700 1 -- 172.21.15.47:0/434150464 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f1fcc028030 con 0x7f1fd00fc420 2022-09-07T21:38:46.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.814+0000 7f1fc77fe700 1 --2- 172.21.15.47:0/434150464 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1fbc099010 0x7f1fbc09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:46.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.815+0000 7f1fc77fe700 1 -- 172.21.15.47:0/434150464 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f1fcc0c3650 con 0x7f1fd00fc420 2022-09-07T21:38:46.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.815+0000 7f1fd5d9a700 1 --2- 172.21.15.47:0/434150464 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1fbc099010 0x7f1fbc09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:46.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.816+0000 7f1fd5d9a700 1 --2- 172.21.15.47:0/434150464 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1fbc099010 0x7f1fbc09b4c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0005fd0 tx=0x7f1fc0005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:46.827 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:46.817+0000 7f1fc77fe700 1 -- 172.21.15.47:0/434150464 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f1fcc08faf0 con 0x7f1fd00fc420 2022-09-07T21:38:47.023 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.021+0000 7f1fd759d700 1 -- 172.21.15.47:0/434150464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fd0107080 con 0x7f1fd00fc420 2022-09-07T21:38:47.024 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.022+0000 7f1fc77fe700 1 -- 172.21.15.47:0/434150464 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23119 (secure 0 0 0) 0x7f1fcc026020 con 0x7f1fd00fc420 2022-09-07T21:38:47.025 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:47.025 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":89,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:38:45.529264+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-07T21:38:38.608937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-07T21:38:39.936252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-07T21:38:43.731797+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:38:47.027 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.025+0000 7f1fd759d700 1 -- 172.21.15.47:0/434150464 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1fbc099010 msgr2=0x7f1fbc09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.028 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.025+0000 7f1fd759d700 1 --2- 172.21.15.47:0/434150464 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1fbc099010 0x7f1fbc09b4c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0005fd0 tx=0x7f1fc0005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.028 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.025+0000 7f1fd759d700 1 -- 172.21.15.47:0/434150464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fd00fc420 msgr2=0x7f1fd0195940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.028 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.025+0000 7f1fd759d700 1 --2- 172.21.15.47:0/434150464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fd00fc420 0x7f1fd0195940 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc000c00 tx=0x7f1fcc00f690 comp rx=0 tx=0).stop 2022-09-07T21:38:47.029 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.025+0000 7f1fd759d700 1 -- 172.21.15.47:0/434150464 shutdown_connections 2022-09-07T21:38:47.029 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.025+0000 7f1fd759d700 1 --2- 172.21.15.47:0/434150464 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1fbc099010 0x7f1fbc09b4c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.029 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.025+0000 7f1fd759d700 1 --2- 172.21.15.47:0/434150464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fd00fd720 0x7f1fd019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.029 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.025+0000 7f1fd759d700 1 --2- 172.21.15.47:0/434150464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fd00fc420 0x7f1fd0195940 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.029 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.025+0000 7f1fd759d700 1 --2- 172.21.15.47:0/434150464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fd00fcd70 0x7f1fd0195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.030 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.025+0000 7f1fd759d700 1 -- 172.21.15.47:0/434150464 >> 172.21.15.47:0/434150464 conn(0x7f1fd00f80d0 msgr2=0x7f1fd00f9ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:47.030 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.025+0000 7f1fd759d700 1 -- 172.21.15.47:0/434150464 shutdown_connections 2022-09-07T21:38:47.030 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.025+0000 7f1fd759d700 1 -- 172.21.15.47:0/434150464 wait complete. 2022-09-07T21:38:47.075 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:38:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:46 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2027217935' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:46 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/968628206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:38:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:38:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:46 smithi059 ceph-mon[97450]: pgmap v319: 129 pgs: 4 creating+peering, 17 unknown, 108 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2022-09-07T21:38:47.308 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.304+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1275321866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0fef90 msgr2=0x7fe59c0ff3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.309 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.304+0000 7fe5a1dff700 1 --2- 172.21.15.47:0/1275321866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0fef90 0x7fe59c0ff3e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe590009810 tx=0x7fe590009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:47.309 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.304+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1275321866 shutdown_connections 2022-09-07T21:38:47.309 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.304+0000 7fe5a1dff700 1 --2- 172.21.15.47:0/1275321866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe59c0ff920 0x7fe59c1061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.310 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.304+0000 7fe5a1dff700 1 --2- 172.21.15.47:0/1275321866 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe59c0fdd10 0x7fe59c0fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.310 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.304+0000 7fe5a1dff700 1 --2- 172.21.15.47:0/1275321866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0fef90 0x7fe59c0ff3e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.311 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.304+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1275321866 >> 172.21.15.47:0/1275321866 conn(0x7fe59c0f93c0 msgr2=0x7fe59c0fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:47.311 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.304+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1275321866 shutdown_connections 2022-09-07T21:38:47.311 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.305+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1275321866 wait complete. 2022-09-07T21:38:47.318 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.306+0000 7fe5a1dff700 1 Processor -- start 2022-09-07T21:38:47.318 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.306+0000 7fe5a1dff700 1 -- start start 2022-09-07T21:38:47.318 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.307+0000 7fe5a1dff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe59c0fdd10 0x7fe59c193980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.319 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.307+0000 7fe5a1dff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0fef90 0x7fe59c193ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.319 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.307+0000 7fe5a1dff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe59c0ff920 0x7fe59c19af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.319 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.307+0000 7fe5a1dff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c19b500 con 0x7fe59c0fdd10 2022-09-07T21:38:47.319 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.307+0000 7fe5a1dff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c19b670 con 0x7fe59c0ff920 2022-09-07T21:38:47.320 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.307+0000 7fe5a1dff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c19b830 con 0x7fe59c0fef90 2022-09-07T21:38:47.320 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.307+0000 7fe59b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe59c0fdd10 0x7fe59c193980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.320 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.307+0000 7fe59affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0fef90 0x7fe59c193ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.320 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.307+0000 7fe59affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0fef90 0x7fe59c193ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60482/0 (socket says 172.21.15.47:60482) 2022-09-07T21:38:47.320 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.307+0000 7fe59b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe59c0fdd10 0x7fe59c193980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55472/0 (socket says 172.21.15.47:55472) 2022-09-07T21:38:47.321 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.307+0000 7fe59affd700 1 -- 172.21.15.47:0/1776796671 learned_addr learned my addr 172.21.15.47:0/1776796671 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:47.321 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.307+0000 7fe59bfff700 1 --2- 172.21.15.47:0/1776796671 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe59c0ff920 0x7fe59c19af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.321 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.308+0000 7fe59affd700 1 -- 172.21.15.47:0/1776796671 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe59c0ff920 msgr2=0x7fe59c19af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.321 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.308+0000 7fe59affd700 1 --2- 172.21.15.47:0/1776796671 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe59c0ff920 0x7fe59c19af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.322 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.308+0000 7fe59affd700 1 -- 172.21.15.47:0/1776796671 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe59c0fdd10 msgr2=0x7fe59c193980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.322 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.308+0000 7fe59affd700 1 --2- 172.21.15.47:0/1776796671 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe59c0fdd10 0x7fe59c193980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.322 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.308+0000 7fe59affd700 1 -- 172.21.15.47:0/1776796671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe590009470 con 0x7fe59c0fef90 2022-09-07T21:38:47.322 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.308+0000 7fe59b7fe700 1 --2- 172.21.15.47:0/1776796671 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe59c0fdd10 0x7fe59c193980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:47.322 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.308+0000 7fe59affd700 1 --2- 172.21.15.47:0/1776796671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0fef90 0x7fe59c193ec0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe59000f690 tx=0x7fe59000f6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:47.323 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.308+0000 7fe598ff9700 1 -- 172.21.15.47:0/1776796671 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59001d070 con 0x7fe59c0fef90 2022-09-07T21:38:47.323 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.308+0000 7fe598ff9700 1 -- 172.21.15.47:0/1776796671 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe59000f8e0 con 0x7fe59c0fef90 2022-09-07T21:38:47.323 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.308+0000 7fe598ff9700 1 -- 172.21.15.47:0/1776796671 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5900047f0 con 0x7fe59c0fef90 2022-09-07T21:38:47.323 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.308+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1776796671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe59c19bab0 con 0x7fe59c0fef90 2022-09-07T21:38:47.324 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.309+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1776796671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59c19bf10 con 0x7fe59c0fef90 2022-09-07T21:38:47.324 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.310+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1776796671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe59c1094c0 con 0x7fe59c0fef90 2022-09-07T21:38:47.324 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.312+0000 7fe598ff9700 1 -- 172.21.15.47:0/1776796671 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fe590028030 con 0x7fe59c0fef90 2022-09-07T21:38:47.324 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.313+0000 7fe598ff9700 1 --2- 172.21.15.47:0/1776796671 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe584099080 0x7fe58409b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.324 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.313+0000 7fe598ff9700 1 -- 172.21.15.47:0/1776796671 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fe5900c31d0 con 0x7fe59c0fef90 2022-09-07T21:38:47.325 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.313+0000 7fe59b7fe700 1 --2- 172.21.15.47:0/1776796671 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe584099080 0x7fe58409b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.325 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.314+0000 7fe59b7fe700 1 --2- 172.21.15.47:0/1776796671 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe584099080 0x7fe58409b530 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0fedf0 tx=0x7fe58c005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:47.325 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.315+0000 7fe598ff9700 1 -- 172.21.15.47:0/1776796671 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fe59008f670 con 0x7fe59c0fef90 2022-09-07T21:38:47.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:46 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2027217935' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:47.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:46 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/968628206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:47.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:38:47.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:38:47.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:46 smithi047 ceph-mon[90087]: pgmap v319: 129 pgs: 4 creating+peering, 17 unknown, 108 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2022-09-07T21:38:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:46 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2027217935' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:46 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/968628206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:47.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:38:47.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:38:47.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:46 smithi055 ceph-mon[97033]: pgmap v319: 129 pgs: 4 creating+peering, 17 unknown, 108 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2022-09-07T21:38:47.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.562+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1776796671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe59c19c5d0 con 0x7fe59c0fef90 2022-09-07T21:38:47.566 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.563+0000 7fe598ff9700 1 -- 172.21.15.47:0/1776796671 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10649 (secure 0 0 0) 0x7fe590092c90 con 0x7fe59c0fef90 2022-09-07T21:38:47.566 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:47.566 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/2559266585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2559266585},{"type":"v1","addr":"172.21.15.55:6835","nonce":2559266585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/2090942297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2090942297},{"type":"v1","addr":"172.21.15.55:6837","nonce":2090942297}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24526,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2445409240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2445409240},{"type":"v1","addr":"172.21.15.59:6835","nonce":2445409240}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6837/3542466465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":3542466465},{"type":"v1","addr":"172.21.15.59:6837","nonce":3542466465}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24548,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6839/315505812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":315505812},{"type":"v1","addr":"172.21.15.47:6839","nonce":315505812}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6839/1006724740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":1006724740},{"type":"v1","addr":"172.21.15.59:6839","nonce":1006724740}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24572,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6839/517990300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":517990300},{"type":"v1","addr":"172.21.15.55:6839","nonce":517990300}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-09-07T21:38:40.544389+0000","modified":"2022-09-07T21:38:45.529512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24586":{"gid":24586,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6841/793863162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6840","nonce":793863162},{"type":"v1","addr":"172.21.15.47:6841","nonce":793863162}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2022-09-07T21:38:47.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.566+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1776796671 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe584099080 msgr2=0x7fe58409b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.566+0000 7fe5a1dff700 1 --2- 172.21.15.47:0/1776796671 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe584099080 0x7fe58409b530 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0fedf0 tx=0x7fe58c005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.566+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1776796671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0fef90 msgr2=0x7fe59c193ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.566+0000 7fe5a1dff700 1 --2- 172.21.15.47:0/1776796671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0fef90 0x7fe59c193ec0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe59000f690 tx=0x7fe59000f6c0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.566+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1776796671 shutdown_connections 2022-09-07T21:38:47.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.566+0000 7fe5a1dff700 1 --2- 172.21.15.47:0/1776796671 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe584099080 0x7fe58409b530 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.566+0000 7fe5a1dff700 1 --2- 172.21.15.47:0/1776796671 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe59c0ff920 0x7fe59c19af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.566+0000 7fe5a1dff700 1 --2- 172.21.15.47:0/1776796671 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe59c0fdd10 0x7fe59c193980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.566+0000 7fe5a1dff700 1 --2- 172.21.15.47:0/1776796671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0fef90 0x7fe59c193ec0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.566+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1776796671 >> 172.21.15.47:0/1776796671 conn(0x7fe59c0f93c0 msgr2=0x7fe59c0fb700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:47.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.566+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1776796671 shutdown_connections 2022-09-07T21:38:47.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.566+0000 7fe5a1dff700 1 -- 172.21.15.47:0/1776796671 wait complete. 2022-09-07T21:38:47.572 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 17 2022-09-07T21:38:47.596 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-09-07T21:38:40.544389+0000', 'modified': '2022-09-07T21:38:45.529512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24586}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24586': {'gid': 24586, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.47:6841/793863162', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6840', 'nonce': 793863162}, {'type': 'v1', 'addr': '172.21.15.47:6841', 'nonce': 793863162}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2022-09-07T21:38:47.597 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-09-07T21:38:47.598 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2022-09-07T21:38:47.832 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.829+0000 7fdfe25ff700 1 -- 172.21.15.47:0/3055295675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfdc066be0 msgr2=0x7fdfdc100960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.832 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.829+0000 7fdfe25ff700 1 --2- 172.21.15.47:0/3055295675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfdc066be0 0x7fdfdc100960 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009810 tx=0x7fdfcc009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:47.832 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.829+0000 7fdfe25ff700 1 -- 172.21.15.47:0/3055295675 shutdown_connections 2022-09-07T21:38:47.833 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.829+0000 7fdfe25ff700 1 --2- 172.21.15.47:0/3055295675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdfdc103990 0x7fdfdc105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.833 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.829+0000 7fdfe25ff700 1 --2- 172.21.15.47:0/3055295675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfdc066be0 0x7fdfdc100960 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.833 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.829+0000 7fdfe25ff700 1 --2- 172.21.15.47:0/3055295675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfdc100ea0 0x7fdfdc103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.833 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.829+0000 7fdfe25ff700 1 -- 172.21.15.47:0/3055295675 >> 172.21.15.47:0/3055295675 conn(0x7fdfdc0f8370 msgr2=0x7fdfdc0fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:47.834 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.829+0000 7fdfe25ff700 1 -- 172.21.15.47:0/3055295675 shutdown_connections 2022-09-07T21:38:47.834 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.829+0000 7fdfe25ff700 1 -- 172.21.15.47:0/3055295675 wait complete. 2022-09-07T21:38:47.837 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.830+0000 7fdfe25ff700 1 Processor -- start 2022-09-07T21:38:47.838 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.830+0000 7fdfe25ff700 1 -- start start 2022-09-07T21:38:47.838 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.831+0000 7fdfe25ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfdc066be0 0x7fdfdc195c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.839 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.831+0000 7fdfe25ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfdc100ea0 0x7fdfdc196160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.839 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.831+0000 7fdfe25ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdfdc103990 0x7fdfdc19d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.831+0000 7fdfe25ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc19d7e0 con 0x7fdfdc103990 2022-09-07T21:38:47.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.831+0000 7fdfe25ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc19d950 con 0x7fdfdc066be0 2022-09-07T21:38:47.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.831+0000 7fdfe25ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc19db10 con 0x7fdfdc100ea0 2022-09-07T21:38:47.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.831+0000 7fdfe0b9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdfdc103990 0x7fdfdc19d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.831+0000 7fdfe0b9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdfdc103990 0x7fdfdc19d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55488/0 (socket says 172.21.15.47:55488) 2022-09-07T21:38:47.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.831+0000 7fdfe0b9c700 1 -- 172.21.15.47:0/108438029 learned_addr learned my addr 172.21.15.47:0/108438029 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:47.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.831+0000 7fdfdbfff700 1 --2- 172.21.15.47:0/108438029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfdc066be0 0x7fdfdc195c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.831+0000 7fdfdb7fe700 1 --2- 172.21.15.47:0/108438029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfdc100ea0 0x7fdfdc196160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.844 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.832+0000 7fdfe0b9c700 1 -- 172.21.15.47:0/108438029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfdc100ea0 msgr2=0x7fdfdc196160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.844 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.832+0000 7fdfe0b9c700 1 --2- 172.21.15.47:0/108438029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfdc100ea0 0x7fdfdc196160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.845 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.832+0000 7fdfe0b9c700 1 -- 172.21.15.47:0/108438029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfdc066be0 msgr2=0x7fdfdc195c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.845 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.832+0000 7fdfe0b9c700 1 --2- 172.21.15.47:0/108438029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfdc066be0 0x7fdfdc195c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.846 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.832+0000 7fdfe0b9c700 1 -- 172.21.15.47:0/108438029 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc009470 con 0x7fdfdc103990 2022-09-07T21:38:47.846 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.832+0000 7fdfe0b9c700 1 --2- 172.21.15.47:0/108438029 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdfdc103990 0x7fdfdc19d190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400ebf0 tx=0x7fdfd400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:47.847 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.832+0000 7fdfd97fa700 1 -- 172.21.15.47:0/108438029 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd40099c0 con 0x7fdfdc103990 2022-09-07T21:38:47.847 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.832+0000 7fdfd97fa700 1 -- 172.21.15.47:0/108438029 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fdfd400cc90 con 0x7fdfdc103990 2022-09-07T21:38:47.847 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.832+0000 7fdfe25ff700 1 -- 172.21.15.47:0/108438029 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfdc19ddf0 con 0x7fdfdc103990 2022-09-07T21:38:47.848 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.832+0000 7fdfd97fa700 1 -- 172.21.15.47:0/108438029 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4010430 con 0x7fdfdc103990 2022-09-07T21:38:47.849 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.833+0000 7fdfe25ff700 1 -- 172.21.15.47:0/108438029 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc19e1c0 con 0x7fdfdc103990 2022-09-07T21:38:47.856 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.836+0000 7fdfd97fa700 1 -- 172.21.15.47:0/108438029 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fdfd401e020 con 0x7fdfdc103990 2022-09-07T21:38:47.857 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.837+0000 7fdfd97fa700 1 --2- 172.21.15.47:0/108438029 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdfc4099010 0x7fdfc409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.857 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.837+0000 7fdfd97fa700 1 -- 172.21.15.47:0/108438029 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fdfd4014070 con 0x7fdfdc103990 2022-09-07T21:38:47.857 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.838+0000 7fdff5ac8b80 1 Processor -- start 2022-09-07T21:38:47.857 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.838+0000 7fdfdbfff700 1 --2- 172.21.15.47:0/108438029 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdfc4099010 0x7fdfc409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.858 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.838+0000 7fdff5ac8b80 1 -- start start 2022-09-07T21:38:47.858 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.838+0000 7fdfdbfff700 1 --2- 172.21.15.47:0/108438029 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdfc4099010 0x7fdfc409b4c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009760 tx=0x7fdfcc009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:47.858 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.838+0000 7fdff5ac8b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3baae0 0x55f25b3baef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.858 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.838+0000 7fdff5ac8b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3bb430 0x55f25b3bbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.859 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.838+0000 7fdff5ac8b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bc1f0 0x55f25b3bc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.859 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.839+0000 7fdff5ac8b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25b2a5080 con 0x55f25b3baae0 2022-09-07T21:38:47.859 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.839+0000 7fdff5ac8b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25b30f9e0 con 0x55f25b3bc1f0 2022-09-07T21:38:47.859 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.839+0000 7fdff5ac8b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25b309180 con 0x55f25b3bb430 2022-09-07T21:38:47.860 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.839+0000 7fdfdbfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3baae0 0x55f25b3baef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.860 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.839+0000 7fdfdbfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3baae0 0x55f25b3baef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55500/0 (socket says 172.21.15.47:55500) 2022-09-07T21:38:47.860 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.839+0000 7fdfdbfff700 1 -- 172.21.15.47:0/417762584 learned_addr learned my addr 172.21.15.47:0/417762584 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:47.860 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.839+0000 7fdfe0b9c700 1 --2- 172.21.15.47:0/417762584 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bc1f0 0x55f25b3bc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.860 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.839+0000 7fdfdb7fe700 1 --2- 172.21.15.47:0/417762584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3bb430 0x55f25b3bbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.861 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.839+0000 7fdfe0b9c700 1 -- 172.21.15.47:0/417762584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3bb430 msgr2=0x55f25b3bbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:38:47.861 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.839+0000 7fdfe0b9c700 1 --2- 172.21.15.47:0/417762584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3bb430 0x55f25b3bbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.861 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.839+0000 7fdfe0b9c700 1 -- 172.21.15.47:0/417762584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3baae0 msgr2=0x55f25b3baef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:38:47.861 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.839+0000 7fdfe0b9c700 1 --2- 172.21.15.47:0/417762584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3baae0 0x55f25b3baef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.839+0000 7fdfe0b9c700 1 -- 172.21.15.47:0/417762584 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f25b1915e0 con 0x55f25b3bc1f0 2022-09-07T21:38:47.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.839+0000 7fdfdbfff700 1 --2- 172.21.15.47:0/417762584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3baae0 0x55f25b3baef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:47.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.840+0000 7fdfe0b9c700 1 --2- 172.21.15.47:0/417762584 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bc1f0 0x55f25b3bc640 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400c5b0 tx=0x7fdfd40927d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9db7134245c9060b server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:47.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.840+0000 7fdfc27fc700 1 -- 172.21.15.47:0/417762584 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd40980c0 con 0x55f25b3bc1f0 2022-09-07T21:38:47.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.840+0000 7fdfc27fc700 1 -- 172.21.15.47:0/417762584 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fdfd4092c90 con 0x55f25b3bc1f0 2022-09-07T21:38:47.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.840+0000 7fdfc27fc700 1 -- 172.21.15.47:0/417762584 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4016330 con 0x55f25b3bc1f0 2022-09-07T21:38:47.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.840+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/417762584 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bc1f0 msgr2=0x55f25b3bc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.840+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/417762584 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bc1f0 0x55f25b3bc640 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400c5b0 tx=0x7fdfd40927d0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.841+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/417762584 shutdown_connections 2022-09-07T21:38:47.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.841+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/417762584 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bc1f0 0x55f25b3bc640 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.841+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/417762584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3baae0 0x55f25b3baef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.841+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/417762584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3bb430 0x55f25b3bbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.841+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/417762584 >> 172.21.15.47:0/417762584 conn(0x55f25b3b6fa0 msgr2=0x55f25b3b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:47.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.841+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/417762584 shutdown_connections 2022-09-07T21:38:47.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.841+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/417762584 wait complete. 2022-09-07T21:38:47.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.841+0000 7fdff5ac8b80 1 -- start start 2022-09-07T21:38:47.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.842+0000 7fdff5ac8b80 1 Processor -- start 2022-09-07T21:38:47.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.842+0000 7fdff5ac8b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3baae0 0x55f25b3c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.842+0000 7fdff5ac8b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bb430 0x55f25b3c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.842+0000 7fdff5ac8b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bc1f0 0x55f25b3cba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.842+0000 7fdff5ac8b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25b1f70c0 con 0x55f25b3bc1f0 2022-09-07T21:38:47.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.842+0000 7fdff5ac8b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25b3cbfb0 con 0x55f25b3bb430 2022-09-07T21:38:47.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.842+0000 7fdff5ac8b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25b3cc140 con 0x55f25b3baae0 2022-09-07T21:38:47.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.842+0000 7fdfdbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3baae0 0x55f25b3c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.842+0000 7fdff5ac8b80 10 client.0 _ll_register_callbacks cb 0x55f25b2b4370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-07T21:38:47.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.842+0000 7fdfdb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bb430 0x55f25b3c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.842+0000 7fdfdbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3baae0 0x55f25b3c44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60516/0 (socket says 172.21.15.47:60516) 2022-09-07T21:38:47.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.842+0000 7fdfdbfff700 1 -- 172.21.15.47:0/2520174229 learned_addr learned my addr 172.21.15.47:0/2520174229 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:47.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.842+0000 7fdfe0b9c700 1 --2- 172.21.15.47:0/2520174229 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bc1f0 0x55f25b3cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.843+0000 7fdfdbfff700 1 -- 172.21.15.47:0/2520174229 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bb430 msgr2=0x55f25b3c4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:38:47.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.843+0000 7fdfdbfff700 1 --2- 172.21.15.47:0/2520174229 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bb430 0x55f25b3c4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.843+0000 7fdfdbfff700 1 -- 172.21.15.47:0/2520174229 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bc1f0 msgr2=0x55f25b3cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:38:47.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.843+0000 7fdfdbfff700 1 --2- 172.21.15.47:0/2520174229 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bc1f0 0x55f25b3cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.843+0000 7fdfdbfff700 1 -- 172.21.15.47:0/2520174229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc106ad0 con 0x55f25b3baae0 2022-09-07T21:38:47.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.843+0000 7fdfdb7fe700 1 --2- 172.21.15.47:0/2520174229 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bb430 0x55f25b3c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:47.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.843+0000 7fdfe0b9c700 1 --2- 172.21.15.47:0/2520174229 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bc1f0 0x55f25b3cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:47.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.843+0000 7fdfdbfff700 1 --2- 172.21.15.47:0/2520174229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3baae0 0x55f25b3c44f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc006010 tx=0x7fdfcc01a050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8ebabec6d1d78eb server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:47.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.843+0000 7fdfb77fe700 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2022-09-07T21:38:47.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.843+0000 7fdfb77fe700 1 -- 172.21.15.47:0/2520174229 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc02b050 con 0x55f25b3baae0 2022-09-07T21:38:47.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.844+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/2520174229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f25b3adaa0 con 0x55f25b3baae0 2022-09-07T21:38:47.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.844+0000 7fdfb77fe700 1 -- 172.21.15.47:0/2520174229 <== mon.1 v2:172.21.15.59:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fdfcc004dc0 con 0x55f25b3baae0 2022-09-07T21:38:47.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.844+0000 7fdfb77fe700 1 -- 172.21.15.47:0/2520174229 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfcc00bed0 con 0x55f25b3baae0 2022-09-07T21:38:47.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.844+0000 7fdff5ac8b80 10 client.24700 fetch_fsmap learned FSMap version 17 2022-09-07T21:38:47.873 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.844+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/2520174229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x7fdfdc19e1c0 con 0x55f25b3baae0 2022-09-07T21:38:47.873 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.844+0000 7fdfb77fe700 1 -- 172.21.15.47:0/2520174229 <== mon.1 v2:172.21.15.59:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x55f25b3adaa0 con 0x55f25b3baae0 2022-09-07T21:38:47.874 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.844+0000 7fdfb77fe700 1 -- 172.21.15.47:0/2520174229 <== mon.1 v2:172.21.15.59:3300/0 5 ==== fsmap(e 17) v1 ==== 7468+0+0 (secure 0 0 0) 0x7fdfcc024410 con 0x55f25b3baae0 2022-09-07T21:38:47.874 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.844+0000 7fdff5ac8b80 10 client.24700 fetch_fsmap finished waiting for FSMap version 17 2022-09-07T21:38:47.874 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.845+0000 7fdff5ac8b80 10 client.24700 resolve_mds: resolved name 'a' to daemon mds.a 2022-09-07T21:38:47.875 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.845+0000 7fdff5ac8b80 20 client.24700 populate_metadata read hostname 'smithi047' 2022-09-07T21:38:47.875 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.845+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/2520174229 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55f25b3cf820 0x55f25b3d4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.876 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.845+0000 7fdff5ac8b80 4 client.24700 mds_command: new command op to 24586 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-07T21:38:47.876 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.845+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/2520174229 --> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f25b38a4c0 con 0x55f25b3cf820 2022-09-07T21:38:47.877 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.845+0000 7fdfdb7fe700 1 --2- 172.21.15.47:0/2520174229 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55f25b3cf820 0x55f25b3d4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.877 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.845+0000 7fdfdb7fe700 1 --2- 172.21.15.47:0/2520174229 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55f25b3cf820 0x55f25b3d4b20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e3ed88b06fb7282 server_cookie=925f77b829de24df in_seq=0 out_seq=0 2022-09-07T21:38:47.878 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.845+0000 7fdfb77fe700 10 client.24700 ms_handle_connect on v2:172.21.15.47:6840/793863162 2022-09-07T21:38:47.878 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.846+0000 7fdfb77fe700 1 -- 172.21.15.47:0/2520174229 <== mds.0 v2:172.21.15.47:6840/793863162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55f25b38a4c0 con 0x55f25b3cf820 2022-09-07T21:38:47.879 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.846+0000 7fdfb77fe700 10 client.24700 handle_command_reply: tid=0 2022-09-07T21:38:47.879 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.846+0000 7fdff5ac8b80 1 client.24700 shutdown 2022-09-07T21:38:47.880 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.846+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/2520174229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3baae0 msgr2=0x55f25b3c44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.880 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.846+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/2520174229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3baae0 0x55f25b3c44f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc006010 tx=0x7fdfcc01a050 comp rx=0 tx=0).stop 2022-09-07T21:38:47.880 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.847+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/2520174229 shutdown_connections 2022-09-07T21:38:47.881 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.847+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/2520174229 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bc1f0 0x55f25b3cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.881 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.847+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/2520174229 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55f25b3cf820 0x55f25b3d4b20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.882 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.847+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/2520174229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3baae0 0x55f25b3c44f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.882 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.847+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/2520174229 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bb430 0x55f25b3c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.882 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.847+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/2520174229 >> 172.21.15.47:0/2520174229 conn(0x55f25b3b6fa0 msgr2=0x55f25b3b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:47.883 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.847+0000 7fdfb77fe700 0 client.24700 ms_handle_reset on v2:172.21.15.47:6840/793863162 2022-09-07T21:38:47.883 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.847+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/2520174229 shutdown_connections 2022-09-07T21:38:47.884 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.847+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/2520174229 wait complete. 2022-09-07T21:38:47.884 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.847+0000 7fdff5ac8b80 20 client.24700 trim_cache size 0 max 16384 2022-09-07T21:38:47.891 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.865+0000 7fdff5ac8b80 1 Processor -- start 2022-09-07T21:38:47.891 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.865+0000 7fdff5ac8b80 1 -- start start 2022-09-07T21:38:47.891 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.865+0000 7fdff5ac8b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3baae0 0x55f25b3c2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.891 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.865+0000 7fdff5ac8b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bb430 0x55f25b3c2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.892 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.865+0000 7fdff5ac8b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bc1f0 0x55f25b3c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.892 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.865+0000 7fdff5ac8b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25b3c56e0 con 0x55f25b3bb430 2022-09-07T21:38:47.892 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.865+0000 7fdff5ac8b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25b2cd050 con 0x55f25b3bc1f0 2022-09-07T21:38:47.892 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.865+0000 7fdff5ac8b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25b195a30 con 0x55f25b3baae0 2022-09-07T21:38:47.893 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.866+0000 7fdfdb7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bb430 0x55f25b3c2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.893 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.866+0000 7fdfdb7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bb430 0x55f25b3c2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55524/0 (socket says 172.21.15.47:55524) 2022-09-07T21:38:47.893 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.866+0000 7fdfdb7fe700 1 -- 172.21.15.47:0/1397162691 learned_addr learned my addr 172.21.15.47:0/1397162691 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:47.893 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.866+0000 7fdfdbfff700 1 --2- 172.21.15.47:0/1397162691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3baae0 0x55f25b3c2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.893 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.866+0000 7fdfe0b9c700 1 --2- 172.21.15.47:0/1397162691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bc1f0 0x55f25b3c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.894 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.866+0000 7fdfdb7fe700 1 -- 172.21.15.47:0/1397162691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3baae0 msgr2=0x55f25b3c2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:38:47.894 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.866+0000 7fdfdb7fe700 1 --2- 172.21.15.47:0/1397162691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3baae0 0x55f25b3c2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.894 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.866+0000 7fdfdb7fe700 1 -- 172.21.15.47:0/1397162691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bc1f0 msgr2=0x55f25b3c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:38:47.895 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.866+0000 7fdfdb7fe700 1 --2- 172.21.15.47:0/1397162691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bc1f0 0x55f25b3c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.895 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.866+0000 7fdfdb7fe700 1 -- 172.21.15.47:0/1397162691 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f25b13dd90 con 0x55f25b3bb430 2022-09-07T21:38:47.895 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.866+0000 7fdfdb7fe700 1 --2- 172.21.15.47:0/1397162691 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bb430 0x55f25b3c2760 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x55f25b3d0da0 tx=0x7fdfd0005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf6544638578ee9 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:47.896 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.867+0000 7fdfc2ffd700 1 -- 172.21.15.47:0/1397162691 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0017050 con 0x55f25b3bb430 2022-09-07T21:38:47.896 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.867+0000 7fdfc2ffd700 1 -- 172.21.15.47:0/1397162691 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fdfd0018040 con 0x55f25b3bb430 2022-09-07T21:38:47.896 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.867+0000 7fdfc2ffd700 1 -- 172.21.15.47:0/1397162691 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd00108f0 con 0x55f25b3bb430 2022-09-07T21:38:47.896 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.867+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/1397162691 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bb430 msgr2=0x55f25b3c2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.896 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.867+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/1397162691 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bb430 0x55f25b3c2760 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x55f25b3d0da0 tx=0x7fdfd0005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.897 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.867+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/1397162691 shutdown_connections 2022-09-07T21:38:47.897 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.867+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/1397162691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3bc1f0 0x55f25b3c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.897 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.867+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/1397162691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3baae0 0x55f25b3c2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.897 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.867+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/1397162691 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bb430 0x55f25b3c2760 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.898 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.867+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/1397162691 >> 172.21.15.47:0/1397162691 conn(0x55f25b3b6fa0 msgr2=0x55f25b3bee90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:47.898 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.867+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/1397162691 shutdown_connections 2022-09-07T21:38:47.898 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.868+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/1397162691 wait complete. 2022-09-07T21:38:47.898 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.868+0000 7fdff5ac8b80 1 -- start start 2022-09-07T21:38:47.899 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.868+0000 7fdff5ac8b80 1 Processor -- start 2022-09-07T21:38:47.899 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdff5ac8b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3baae0 0x55f25b3c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.899 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdff5ac8b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3bb430 0x55f25b3c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.899 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdff5ac8b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bc1f0 0x55f25b421ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.899 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdff5ac8b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25b422000 con 0x55f25b3bc1f0 2022-09-07T21:38:47.900 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdff5ac8b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25b422140 con 0x55f25b3baae0 2022-09-07T21:38:47.900 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdff5ac8b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f25b4222d0 con 0x55f25b3bb430 2022-09-07T21:38:47.900 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdff5ac8b80 10 client.0 _ll_register_callbacks cb 0x55f25b2b4370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-07T21:38:47.900 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdfdbfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3baae0 0x55f25b3c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.901 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdfe0b9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bc1f0 0x55f25b421ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.901 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdfdbfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3baae0 0x55f25b3c88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:45600/0 (socket says 172.21.15.47:45600) 2022-09-07T21:38:47.901 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdfdbfff700 1 -- 172.21.15.47:0/3679627633 learned_addr learned my addr 172.21.15.47:0/3679627633 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:47.901 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdfdb7fe700 1 --2- 172.21.15.47:0/3679627633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3bb430 0x55f25b3c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.902 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdfe0b9c700 1 -- 172.21.15.47:0/3679627633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3bb430 msgr2=0x55f25b3c8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:38:47.902 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdfe0b9c700 1 --2- 172.21.15.47:0/3679627633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3bb430 0x55f25b3c8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.902 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdfe0b9c700 1 -- 172.21.15.47:0/3679627633 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3baae0 msgr2=0x55f25b3c88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:38:47.902 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdfe0b9c700 1 --2- 172.21.15.47:0/3679627633 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3baae0 0x55f25b3c88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.902 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdfe0b9c700 1 -- 172.21.15.47:0/3679627633 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd0009e60 con 0x55f25b3bc1f0 2022-09-07T21:38:47.903 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.869+0000 7fdfdbfff700 1 --2- 172.21.15.47:0/3679627633 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3baae0 0x55f25b3c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:47.903 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.870+0000 7fdfe0b9c700 1 --2- 172.21.15.47:0/3679627633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bc1f0 0x55f25b421ac0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400c5b0 tx=0x7fdfd4005930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2147da556660c6 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:47.903 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.870+0000 7fdfb77fe700 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2022-09-07T21:38:47.903 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.870+0000 7fdfb77fe700 1 -- 172.21.15.47:0/3679627633 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd40c0200 con 0x55f25b3bc1f0 2022-09-07T21:38:47.904 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.870+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/3679627633 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdfd4092c90 con 0x55f25b3bc1f0 2022-09-07T21:38:47.904 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.870+0000 7fdfb77fe700 1 -- 172.21.15.47:0/3679627633 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fdfd4099f30 con 0x55f25b3bc1f0 2022-09-07T21:38:47.904 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.870+0000 7fdfb77fe700 1 -- 172.21.15.47:0/3679627633 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfd408efd0 con 0x55f25b3bc1f0 2022-09-07T21:38:47.905 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.870+0000 7fdff5ac8b80 10 client.14619 fetch_fsmap learned FSMap version 17 2022-09-07T21:38:47.905 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.870+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/3679627633 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55f25b13c260 con 0x55f25b3bc1f0 2022-09-07T21:38:47.905 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.870+0000 7fdfb77fe700 1 -- 172.21.15.47:0/3679627633 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fdfd405f080 con 0x55f25b3bc1f0 2022-09-07T21:38:47.905 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.870+0000 7fdfb77fe700 1 -- 172.21.15.47:0/3679627633 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 17) v1 ==== 7468+0+0 (secure 0 0 0) 0x7fdfd40961b0 con 0x55f25b3bc1f0 2022-09-07T21:38:47.906 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.871+0000 7fdff5ac8b80 10 client.14619 fetch_fsmap finished waiting for FSMap version 17 2022-09-07T21:38:47.906 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.871+0000 7fdff5ac8b80 10 client.14619 resolve_mds: resolved name 'a' to daemon mds.a 2022-09-07T21:38:47.906 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.871+0000 7fdff5ac8b80 20 client.14619 populate_metadata read hostname 'smithi047' 2022-09-07T21:38:47.906 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.871+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/3679627633 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55f25b426cc0 0x55f25b4236a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:47.906 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.871+0000 7fdff5ac8b80 4 client.14619 mds_command: new command op to 24586 tid=0[{"prefix": "status"}] 2022-09-07T21:38:47.907 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.871+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/3679627633 --> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f25b3026f0 con 0x55f25b426cc0 2022-09-07T21:38:47.907 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.871+0000 7fdfdbfff700 1 --2- 172.21.15.47:0/3679627633 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55f25b426cc0 0x55f25b4236a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:47.907 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.871+0000 7fdfdbfff700 1 --2- 172.21.15.47:0/3679627633 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55f25b426cc0 0x55f25b4236a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b2e4b420be6f1cc server_cookie=4d009be011f20455 in_seq=0 out_seq=0 2022-09-07T21:38:47.907 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.871+0000 7fdfb77fe700 10 client.14619 ms_handle_connect on v2:172.21.15.47:6840/793863162 2022-09-07T21:38:47.908 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdfb77fe700 1 -- 172.21.15.47:0/3679627633 <== mds.0 v2:172.21.15.47:6840/793863162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55f25b3026f0 con 0x55f25b426cc0 2022-09-07T21:38:47.908 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdfb77fe700 10 client.14619 handle_command_reply: tid=0 2022-09-07T21:38:47.908 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdff5ac8b80 1 client.14619 shutdown 2022-09-07T21:38:47.908 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/3679627633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bc1f0 msgr2=0x55f25b421ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.908 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/3679627633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bc1f0 0x55f25b421ac0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400c5b0 tx=0x7fdfd4005930 comp rx=0 tx=0).stop 2022-09-07T21:38:47.909 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/3679627633 shutdown_connections 2022-09-07T21:38:47.909 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/3679627633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f25b3bc1f0 0x55f25b421ac0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.909 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/3679627633 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f25b3baae0 0x55f25b3c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.910 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/3679627633 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55f25b426cc0 0x55f25b4236a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.910 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdff5ac8b80 1 --2- 172.21.15.47:0/3679627633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f25b3bb430 0x55f25b3c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.910 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/3679627633 >> 172.21.15.47:0/3679627633 conn(0x55f25b3b6fa0 msgr2=0x55f25b3c2220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:47.911 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdfb77fe700 0 client.14619 ms_handle_reset on v2:172.21.15.47:6840/793863162 2022-09-07T21:38:47.911 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/3679627633 shutdown_connections 2022-09-07T21:38:47.911 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdff5ac8b80 1 -- 172.21.15.47:0/3679627633 wait complete. 2022-09-07T21:38:47.911 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.872+0000 7fdff5ac8b80 20 client.14619 trim_cache size 0 max 16384 2022-09-07T21:38:47.912 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2022-09-07T21:38:47.912 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "01b8396e-2ef4-11ed-8431-001a4aab830c", 2022-09-07T21:38:47.912 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 0, 2022-09-07T21:38:47.913 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 24586, 2022-09-07T21:38:47.913 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2022-09-07T21:38:47.913 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2022-09-07T21:38:47.913 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2022-09-07T21:38:47.914 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 7.3242093029999999, 2022-09-07T21:38:47.914 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 17, 2022-09-07T21:38:47.914 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 89, 2022-09-07T21:38:47.914 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 88, 2022-09-07T21:38:47.915 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 15.363758903000001 2022-09-07T21:38:47.915 INFO:teuthology.orchestra.run.smithi047.stdout:} 2022-09-07T21:38:47.916 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.873+0000 7fdfe25ff700 1 -- 172.21.15.47:0/108438029 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdfc4099010 msgr2=0x7fdfc409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.917 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.873+0000 7fdfe25ff700 1 --2- 172.21.15.47:0/108438029 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdfc4099010 0x7fdfc409b4c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009760 tx=0x7fdfcc009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.917 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.873+0000 7fdfe25ff700 1 -- 172.21.15.47:0/108438029 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdfdc103990 msgr2=0x7fdfdc19d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:47.917 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.873+0000 7fdfe25ff700 1 --2- 172.21.15.47:0/108438029 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdfdc103990 0x7fdfdc19d190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400ebf0 tx=0x7fdfd400ef00 comp rx=0 tx=0).stop 2022-09-07T21:38:47.917 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.873+0000 7fdfe25ff700 1 -- 172.21.15.47:0/108438029 shutdown_connections 2022-09-07T21:38:47.918 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.873+0000 7fdfe25ff700 1 --2- 172.21.15.47:0/108438029 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdfc4099010 0x7fdfc409b4c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.918 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.873+0000 7fdfe25ff700 1 --2- 172.21.15.47:0/108438029 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdfdc103990 0x7fdfdc19d190 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.918 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.873+0000 7fdfe25ff700 1 --2- 172.21.15.47:0/108438029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfdc066be0 0x7fdfdc195c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.918 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.873+0000 7fdfe25ff700 1 --2- 172.21.15.47:0/108438029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfdc100ea0 0x7fdfdc196160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:47.919 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.873+0000 7fdfe25ff700 1 -- 172.21.15.47:0/108438029 >> 172.21.15.47:0/108438029 conn(0x7fdfdc0f8370 msgr2=0x7fdfdc0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:47.919 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.873+0000 7fdfe25ff700 1 -- 172.21.15.47:0/108438029 shutdown_connections 2022-09-07T21:38:47.919 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:47.873+0000 7fdfe25ff700 1 -- 172.21.15.47:0/108438029 wait complete. 2022-09-07T21:38:47.920 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2022-09-07T21:38:47.921 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2022-09-07T21:38:47.921 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph fs ls 2022-09-07T21:38:48.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:47 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/434150464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:48.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:47 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1776796671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:47 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/434150464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:47 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1776796671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:47 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/434150464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:48.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:47 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1776796671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:38:48.800 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:38:49.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:49 smithi047 ceph-mon[90087]: pgmap v320: 129 pgs: 4 creating+peering, 3 unknown, 122 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 4 op/s 2022-09-07T21:38:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:49 smithi059 ceph-mon[97450]: pgmap v320: 129 pgs: 4 creating+peering, 3 unknown, 122 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 4 op/s 2022-09-07T21:38:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:49 smithi055 ceph-mon[97033]: pgmap v320: 129 pgs: 4 creating+peering, 3 unknown, 122 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 4 op/s 2022-09-07T21:38:51.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.079+0000 7fed2441a700 1 -- 172.21.15.47:0/764426250 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed1c1053e0 msgr2=0x7fed1c10cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:51.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.079+0000 7fed2441a700 1 --2- 172.21.15.47:0/764426250 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed1c1053e0 0x7fed1c10cf80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fed14009be0 tx=0x7fed14009ef0 comp rx=0 tx=0).stop 2022-09-07T21:38:51.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.079+0000 7fed2441a700 1 -- 172.21.15.47:0/764426250 shutdown_connections 2022-09-07T21:38:51.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.079+0000 7fed2441a700 1 --2- 172.21.15.47:0/764426250 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed1c1053e0 0x7fed1c10cf80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:51.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.079+0000 7fed2441a700 1 --2- 172.21.15.47:0/764426250 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed1c1002e0 0x7fed1c1006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:51.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.079+0000 7fed2441a700 1 --2- 172.21.15.47:0/764426250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed1c100c80 0x7fed1c104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:51.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.079+0000 7fed2441a700 1 -- 172.21.15.47:0/764426250 >> 172.21.15.47:0/764426250 conn(0x7fed1c0fbc20 msgr2=0x7fed1c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:51.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.079+0000 7fed2441a700 1 -- 172.21.15.47:0/764426250 shutdown_connections 2022-09-07T21:38:51.085 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.079+0000 7fed2441a700 1 -- 172.21.15.47:0/764426250 wait complete. 2022-09-07T21:38:51.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.081+0000 7fed2441a700 1 Processor -- start 2022-09-07T21:38:51.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.082+0000 7fed2441a700 1 -- start start 2022-09-07T21:38:51.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.082+0000 7fed2441a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed1c1002e0 0x7fed1c195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:51.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.082+0000 7fed2441a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed1c100c80 0x7fed1c196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:51.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.082+0000 7fed2441a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed1c1053e0 0x7fed1c19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:51.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.082+0000 7fed2441a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c19ae60 con 0x7fed1c1053e0 2022-09-07T21:38:51.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.082+0000 7fed2441a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c19afd0 con 0x7fed1c100c80 2022-09-07T21:38:51.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.082+0000 7fed2441a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c19b190 con 0x7fed1c1002e0 2022-09-07T21:38:51.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.083+0000 7fed221b6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed1c1002e0 0x7fed1c195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:51.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.083+0000 7fed221b6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed1c1002e0 0x7fed1c195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:38522/0 (socket says 172.21.15.47:38522) 2022-09-07T21:38:51.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.083+0000 7fed221b6700 1 -- 172.21.15.47:0/193642581 learned_addr learned my addr 172.21.15.47:0/193642581 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:51.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.083+0000 7fed219b5700 1 --2- 172.21.15.47:0/193642581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed1c100c80 0x7fed1c196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:51.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.083+0000 7fed229b7700 1 --2- 172.21.15.47:0/193642581 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed1c1053e0 0x7fed1c19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:51.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.083+0000 7fed221b6700 1 -- 172.21.15.47:0/193642581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed1c100c80 msgr2=0x7fed1c196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:51.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.083+0000 7fed221b6700 1 --2- 172.21.15.47:0/193642581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed1c100c80 0x7fed1c196410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:51.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.083+0000 7fed221b6700 1 -- 172.21.15.47:0/193642581 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed1c1053e0 msgr2=0x7fed1c19a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:51.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.083+0000 7fed221b6700 1 --2- 172.21.15.47:0/193642581 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed1c1053e0 0x7fed1c19a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:51.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.083+0000 7fed221b6700 1 -- 172.21.15.47:0/193642581 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed180093b0 con 0x7fed1c1002e0 2022-09-07T21:38:51.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.083+0000 7fed221b6700 1 --2- 172.21.15.47:0/193642581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed1c1002e0 0x7fed1c195ed0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fed1800e9c0 tx=0x7fed1800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:51.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.084+0000 7fed137fe700 1 -- 172.21.15.47:0/193642581 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed180097b0 con 0x7fed1c1002e0 2022-09-07T21:38:51.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.084+0000 7fed2441a700 1 -- 172.21.15.47:0/193642581 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed140097e0 con 0x7fed1c1002e0 2022-09-07T21:38:51.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.084+0000 7fed137fe700 1 -- 172.21.15.47:0/193642581 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fed1800ce70 con 0x7fed1c1002e0 2022-09-07T21:38:51.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.084+0000 7fed137fe700 1 -- 172.21.15.47:0/193642581 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18003cf0 con 0x7fed1c1002e0 2022-09-07T21:38:51.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.084+0000 7fed2441a700 1 -- 172.21.15.47:0/193642581 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1c19b610 con 0x7fed1c1002e0 2022-09-07T21:38:51.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.085+0000 7fed2441a700 1 -- 172.21.15.47:0/193642581 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1c064ea0 con 0x7fed1c1002e0 2022-09-07T21:38:51.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.087+0000 7fed137fe700 1 -- 172.21.15.47:0/193642581 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fed1801e020 con 0x7fed1c1002e0 2022-09-07T21:38:51.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.088+0000 7fed137fe700 1 --2- 172.21.15.47:0/193642581 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fed08099010 0x7fed0809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:51.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.088+0000 7fed137fe700 1 -- 172.21.15.47:0/193642581 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fed18014070 con 0x7fed1c1002e0 2022-09-07T21:38:51.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.088+0000 7fed219b5700 1 --2- 172.21.15.47:0/193642581 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fed08099010 0x7fed0809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:51.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.089+0000 7fed219b5700 1 --2- 172.21.15.47:0/193642581 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fed08099010 0x7fed0809b4c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed1c1974e0 tx=0x7fed0c005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:51.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.090+0000 7fed137fe700 1 -- 172.21.15.47:0/193642581 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fed1808f5b0 con 0x7fed1c1002e0 2022-09-07T21:38:51.363 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.360+0000 7fed2441a700 1 -- 172.21.15.47:0/193642581 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fed1c1a3140 con 0x7fed1c1002e0 2022-09-07T21:38:51.364 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.361+0000 7fed137fe700 1 -- 172.21.15.47:0/193642581 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7fed18092bd0 con 0x7fed1c1002e0 2022-09-07T21:38:51.364 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2022-09-07T21:38:51.366 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.363+0000 7fed2441a700 1 -- 172.21.15.47:0/193642581 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fed08099010 msgr2=0x7fed0809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:51.367 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.363+0000 7fed2441a700 1 --2- 172.21.15.47:0/193642581 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fed08099010 0x7fed0809b4c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed1c1974e0 tx=0x7fed0c005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:51.367 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.363+0000 7fed2441a700 1 -- 172.21.15.47:0/193642581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed1c1002e0 msgr2=0x7fed1c195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:51.367 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.363+0000 7fed2441a700 1 --2- 172.21.15.47:0/193642581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed1c1002e0 0x7fed1c195ed0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fed1800e9c0 tx=0x7fed1800ecd0 comp rx=0 tx=0).stop 2022-09-07T21:38:51.367 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.364+0000 7fed2441a700 1 -- 172.21.15.47:0/193642581 shutdown_connections 2022-09-07T21:38:51.368 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.364+0000 7fed2441a700 1 --2- 172.21.15.47:0/193642581 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fed08099010 0x7fed0809b4c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:51.368 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.364+0000 7fed2441a700 1 --2- 172.21.15.47:0/193642581 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed1c1053e0 0x7fed1c19a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:51.368 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.364+0000 7fed2441a700 1 --2- 172.21.15.47:0/193642581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed1c1002e0 0x7fed1c195ed0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:51.368 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.364+0000 7fed2441a700 1 --2- 172.21.15.47:0/193642581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed1c100c80 0x7fed1c196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:51.369 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.364+0000 7fed2441a700 1 -- 172.21.15.47:0/193642581 >> 172.21.15.47:0/193642581 conn(0x7fed1c0fbc20 msgr2=0x7fed1c0fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:51.369 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.364+0000 7fed2441a700 1 -- 172.21.15.47:0/193642581 shutdown_connections 2022-09-07T21:38:51.369 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:51.364+0000 7fed2441a700 1 -- 172.21.15.47:0/193642581 wait complete. 2022-09-07T21:38:51.872 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2022-09-07T21:38:51.873 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2022-09-07T21:38:51.873 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi047.front.sepia.ceph.com 2022-09-07T21:38:51.874 INFO:tasks.cephfs.mount:self.client.name = client.admin 2022-09-07T21:38:51.874 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2022-09-07T21:38:51.875 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2022-09-07T21:38:51.875 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2022-09-07T21:38:51.875 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2022-09-07T21:38:51.876 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2022-09-07T21:38:51.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:51.877 DEBUG:teuthology.orchestra.run.smithi047:> ip addr 2022-09-07T21:38:51.936 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-09-07T21:38:51.937 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-09-07T21:38:51.937 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2022-09-07T21:38:51.937 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2022-09-07T21:38:51.938 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2022-09-07T21:38:51.938 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2022-09-07T21:38:51.938 INFO:teuthology.orchestra.run.smithi047.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-09-07T21:38:51.938 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:78 brd ff:ff:ff:ff:ff:ff 2022-09-07T21:38:51.939 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f0 2022-09-07T21:38:51.939 INFO:teuthology.orchestra.run.smithi047.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-09-07T21:38:51.939 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:79 brd ff:ff:ff:ff:ff:ff 2022-09-07T21:38:51.939 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f1 2022-09-07T21:38:51.940 INFO:teuthology.orchestra.run.smithi047.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-09-07T21:38:51.940 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8e brd ff:ff:ff:ff:ff:ff 2022-09-07T21:38:51.940 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f0 2022-09-07T21:38:51.940 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.21.15.47/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2022-09-07T21:38:51.941 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft 41567sec preferred_lft 41567sec 2022-09-07T21:38:51.941 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2022-09-07T21:38:51.941 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2022-09-07T21:38:51.941 INFO:teuthology.orchestra.run.smithi047.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-09-07T21:38:51.942 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8f brd ff:ff:ff:ff:ff:ff 2022-09-07T21:38:51.943 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f1 2022-09-07T21:38:51.945 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2022-09-07T21:38:51.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:51.946 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2022-09-07T21:38:51.946 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2022-09-07T21:38:51.947 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link add name ceph-brx type bridge 2022-09-07T21:38:51.947 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip addr flush dev ceph-brx 2022-09-07T21:38:51.948 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set ceph-brx up 2022-09-07T21:38:51.948 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2022-09-07T21:38:51.949 DEBUG:teuthology.orchestra.run.smithi047:> ') 2022-09-07T21:38:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:51 smithi059 ceph-mon[97450]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2022-09-07T21:38:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:51 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/193642581' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-09-07T21:38:52.082 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:51 smithi047 ceph-mon[90087]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2022-09-07T21:38:52.082 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:51 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/193642581' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-09-07T21:38:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:51 smithi055 ceph-mon[97033]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2022-09-07T21:38:52.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:51 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/193642581' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-09-07T21:38:52.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:52.435 DEBUG:teuthology.orchestra.run.smithi047:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2022-09-07T21:38:52.528 INFO:teuthology.orchestra.run.smithi047.stdout:1 2022-09-07T21:38:52.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:52.534 DEBUG:teuthology.orchestra.run.smithi047:> route 2022-09-07T21:38:52.601 INFO:teuthology.orchestra.run.smithi047.stdout:Kernel IP routing table 2022-09-07T21:38:52.601 INFO:teuthology.orchestra.run.smithi047.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2022-09-07T21:38:52.601 INFO:teuthology.orchestra.run.smithi047.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2022-09-07T21:38:52.602 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2022-09-07T21:38:52.602 INFO:teuthology.orchestra.run.smithi047.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2022-09-07T21:38:52.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:52.604 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2022-09-07T21:38:52.604 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2022-09-07T21:38:52.605 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2022-09-07T21:38:52.605 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2022-09-07T21:38:52.606 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2022-09-07T21:38:52.606 DEBUG:teuthology.orchestra.run.smithi047:> ') 2022-09-07T21:38:53.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:53.377 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2022-09-07T21:38:53.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:53.438 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list-id 2022-09-07T21:38:53.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:53.498 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2022-09-07T21:38:53.498 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2022-09-07T21:38:53.498 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2022-09-07T21:38:53.499 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2022-09-07T21:38:53.499 DEBUG:teuthology.orchestra.run.smithi047:> ') 2022-09-07T21:38:53.687 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2022-09-07T21:38:53.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:53.689 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2022-09-07T21:38:53.689 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2022-09-07T21:38:53.689 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2022-09-07T21:38:53.690 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2022-09-07T21:38:53.690 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2022-09-07T21:38:53.691 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2022-09-07T21:38:53.691 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2022-09-07T21:38:53.691 DEBUG:teuthology.orchestra.run.smithi047:> ') 2022-09-07T21:38:53.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:53 smithi047 ceph-mon[90087]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 3 op/s 2022-09-07T21:38:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:53 smithi059 ceph-mon[97450]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 3 op/s 2022-09-07T21:38:54.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:54.054 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2022-09-07T21:38:54.054 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2022-09-07T21:38:54.055 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set brx.0 up 2022-09-07T21:38:54.055 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set dev brx.0 master ceph-brx 2022-09-07T21:38:54.056 DEBUG:teuthology.orchestra.run.smithi047:> ') 2022-09-07T21:38:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:53 smithi055 ceph-mon[97033]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 3 op/s 2022-09-07T21:38:54.272 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2022-09-07T21:38:54.273 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-09-07T21:38:54.273 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2022-09-07T21:38:54.332 INFO:teuthology.orchestra.run.smithi047.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2022-09-07T21:38:54.332 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-09-07T21:38:54.333 DEBUG:teuthology.orchestra.run.smithi047:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2022-09-07T21:38:54.391 DEBUG:teuthology.orchestra.run.smithi047:> sudo modprobe fuse 2022-09-07T21:38:54.481 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-09-07T21:38:54.482 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-09-07T21:38:54.568 INFO:teuthology.orchestra.run.smithi047.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-09-07T21:38:54.572 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-09-07T21:38:54.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:54.574 DEBUG:teuthology.orchestra.run.smithi047:> ls /sys/fs/fuse/connections 2022-09-07T21:38:54.636 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-09-07T21:38:54.637 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_mountpoint=/ --client_fs=cephfs) 2022-09-07T21:38:54.643 DEBUG:teuthology.orchestra.run.smithi047:> sudo modprobe fuse 2022-09-07T21:38:54.754 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-09-07T21:38:54.755 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-09-07T21:38:54.846 INFO:teuthology.orchestra.run.smithi047.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-09-07T21:38:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-09-07T21:38:54.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:54.929 DEBUG:teuthology.orchestra.run.smithi047:> ls /sys/fs/fuse/connections 2022-09-07T21:38:54.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi047.stderr:2022-09-07T21:38:54.878+0000 7ff5d5dce400 -1 init, newargv = 0x55e643703750 newargc=15 2022-09-07T21:38:54.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi047.stderr:ceph-fuse[127948]: starting ceph client 2022-09-07T21:38:54.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi047.stderr:ceph-fuse[127948]: starting fuse 2022-09-07T21:38:54.989 INFO:teuthology.orchestra.run.smithi047.stdout:120 2022-09-07T21:38:54.989 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [120] 2022-09-07T21:38:54.990 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-09-07T21:38:54.990 DEBUG:teuthology.orchestra.run.smithi047:> import glob 2022-09-07T21:38:54.991 DEBUG:teuthology.orchestra.run.smithi047:> import re 2022-09-07T21:38:54.991 DEBUG:teuthology.orchestra.run.smithi047:> import os 2022-09-07T21:38:54.992 DEBUG:teuthology.orchestra.run.smithi047:> import subprocess 2022-09-07T21:38:54.992 DEBUG:teuthology.orchestra.run.smithi047:> 2022-09-07T21:38:54.992 DEBUG:teuthology.orchestra.run.smithi047:> def _find_admin_socket(client_name): 2022-09-07T21:38:54.993 DEBUG:teuthology.orchestra.run.smithi047:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2022-09-07T21:38:54.993 DEBUG:teuthology.orchestra.run.smithi047:> files = glob.glob(asok_path) 2022-09-07T21:38:54.993 DEBUG:teuthology.orchestra.run.smithi047:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2022-09-07T21:38:54.995 DEBUG:teuthology.orchestra.run.smithi047:> 2022-09-07T21:38:54.995 DEBUG:teuthology.orchestra.run.smithi047:> # Given a non-glob path, it better be there 2022-09-07T21:38:54.995 DEBUG:teuthology.orchestra.run.smithi047:> if "*" not in asok_path: 2022-09-07T21:38:54.996 DEBUG:teuthology.orchestra.run.smithi047:> assert(len(files) == 1) 2022-09-07T21:38:54.996 DEBUG:teuthology.orchestra.run.smithi047:> return files[0] 2022-09-07T21:38:54.997 DEBUG:teuthology.orchestra.run.smithi047:> 2022-09-07T21:38:54.997 DEBUG:teuthology.orchestra.run.smithi047:> for f in files: 2022-09-07T21:38:54.998 DEBUG:teuthology.orchestra.run.smithi047:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-09-07T21:38:54.998 DEBUG:teuthology.orchestra.run.smithi047:> if os.path.exists("/proc/{0}".format(pid)): 2022-09-07T21:38:54.998 DEBUG:teuthology.orchestra.run.smithi047:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2022-09-07T21:38:54.999 DEBUG:teuthology.orchestra.run.smithi047:> contents = proc_f.read() 2022-09-07T21:38:54.999 DEBUG:teuthology.orchestra.run.smithi047:> if mountpoint in contents: 2022-09-07T21:38:55.000 DEBUG:teuthology.orchestra.run.smithi047:> return f 2022-09-07T21:38:55.000 DEBUG:teuthology.orchestra.run.smithi047:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-09-07T21:38:55.001 DEBUG:teuthology.orchestra.run.smithi047:> 2022-09-07T21:38:55.001 DEBUG:teuthology.orchestra.run.smithi047:> print(_find_admin_socket("client.admin")) 2022-09-07T21:38:55.002 DEBUG:teuthology.orchestra.run.smithi047:> ' 2022-09-07T21:38:55.176 INFO:teuthology.orchestra.run.smithi047.stdout:/var/run/ceph/ceph-client.admin.127948.asok 2022-09-07T21:38:55.353 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.127948.asok 2022-09-07T21:38:55.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:55.354 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.127948.asok status 2022-09-07T21:38:55.611 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2022-09-07T21:38:55.611 INFO:teuthology.orchestra.run.smithi047.stdout: "metadata": { 2022-09-07T21:38:55.612 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph_sha1": "152194198413033811b31149de909a7ec5852a7f", 2022-09-07T21:38:55.612 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph_version": "ceph version 17.0.0-14857-g15219419 (152194198413033811b31149de909a7ec5852a7f) quincy (dev)", 2022-09-07T21:38:55.612 INFO:teuthology.orchestra.run.smithi047.stdout: "entity_id": "admin", 2022-09-07T21:38:55.612 INFO:teuthology.orchestra.run.smithi047.stdout: "hostname": "smithi047", 2022-09-07T21:38:55.613 INFO:teuthology.orchestra.run.smithi047.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2022-09-07T21:38:55.613 INFO:teuthology.orchestra.run.smithi047.stdout: "pid": "127948", 2022-09-07T21:38:55.613 INFO:teuthology.orchestra.run.smithi047.stdout: "root": "/" 2022-09-07T21:38:55.613 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2022-09-07T21:38:55.614 INFO:teuthology.orchestra.run.smithi047.stdout: "dentry_count": 0, 2022-09-07T21:38:55.614 INFO:teuthology.orchestra.run.smithi047.stdout: "dentry_pinned_count": 0, 2022-09-07T21:38:55.614 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 24718, 2022-09-07T21:38:55.614 INFO:teuthology.orchestra.run.smithi047.stdout: "inst": { 2022-09-07T21:38:55.615 INFO:teuthology.orchestra.run.smithi047.stdout: "name": { 2022-09-07T21:38:55.615 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "client", 2022-09-07T21:38:55.615 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 24718 2022-09-07T21:38:55.615 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2022-09-07T21:38:55.616 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": { 2022-09-07T21:38:55.616 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2022-09-07T21:38:55.616 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:0", 2022-09-07T21:38:55.616 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 2200606581 2022-09-07T21:38:55.617 INFO:teuthology.orchestra.run.smithi047.stdout: } 2022-09-07T21:38:55.617 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2022-09-07T21:38:55.617 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": { 2022-09-07T21:38:55.617 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2022-09-07T21:38:55.617 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:0", 2022-09-07T21:38:55.618 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 2200606581 2022-09-07T21:38:55.618 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2022-09-07T21:38:55.618 INFO:teuthology.orchestra.run.smithi047.stdout: "inst_str": "client.24718 172.21.15.47:0/2200606581", 2022-09-07T21:38:55.618 INFO:teuthology.orchestra.run.smithi047.stdout: "addr_str": "172.21.15.47:0/2200606581", 2022-09-07T21:38:55.619 INFO:teuthology.orchestra.run.smithi047.stdout: "inode_count": 1, 2022-09-07T21:38:55.619 INFO:teuthology.orchestra.run.smithi047.stdout: "mds_epoch": 17, 2022-09-07T21:38:55.619 INFO:teuthology.orchestra.run.smithi047.stdout: "osd_epoch": 89, 2022-09-07T21:38:55.619 INFO:teuthology.orchestra.run.smithi047.stdout: "osd_epoch_barrier": 0, 2022-09-07T21:38:55.620 INFO:teuthology.orchestra.run.smithi047.stdout: "blocklisted": false, 2022-09-07T21:38:55.620 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs" 2022-09-07T21:38:55.620 INFO:teuthology.orchestra.run.smithi047.stdout:} 2022-09-07T21:38:55.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:55.632 DEBUG:teuthology.orchestra.run.smithi047:> stat --file-system '--printf=%T 2022-09-07T21:38:55.632 DEBUG:teuthology.orchestra.run.smithi047:> ' -- /home/ubuntu/cephtest/mnt.admin 2022-09-07T21:38:55.692 INFO:teuthology.orchestra.run.smithi047.stdout:fuseblk 2022-09-07T21:38:55.692 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2022-09-07T21:38:55.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:55.693 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2022-09-07T21:38:55.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:38:55.784 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2022-09-07T21:38:55.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:55 smithi047 ceph-mon[90087]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 212 B/s wr, 2 op/s 2022-09-07T21:38:55.942 INFO:teuthology.orchestra.run.smithi047.stdout:# file: . 2022-09-07T21:38:55.942 INFO:teuthology.orchestra.run.smithi047.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2022-09-07T21:38:55.943 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:55.944 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:38:55.944 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2022-09-07T21:38:56.004 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-07T21:38:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:55 smithi059 ceph-mon[97450]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 212 B/s wr, 2 op/s 2022-09-07T21:38:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:55 smithi055 ceph-mon[97033]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 212 B/s wr, 2 op/s 2022-09-07T21:38:56.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.233+0000 7f188959d700 1 -- 172.21.15.47:0/738614366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18840fede0 msgr2=0x7f18840ff230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:56.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.233+0000 7f188959d700 1 --2- 172.21.15.47:0/738614366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18840fede0 0x7f18840ff230 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1878009810 tx=0x7f1878009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:56.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.234+0000 7f188959d700 1 -- 172.21.15.47:0/738614366 shutdown_connections 2022-09-07T21:38:56.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.234+0000 7f188959d700 1 --2- 172.21.15.47:0/738614366 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18840ff770 0x7f1884106030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.234+0000 7f188959d700 1 --2- 172.21.15.47:0/738614366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18840fdb60 0x7f18840fdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.234+0000 7f188959d700 1 --2- 172.21.15.47:0/738614366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18840fede0 0x7f18840ff230 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.234+0000 7f188959d700 1 -- 172.21.15.47:0/738614366 >> 172.21.15.47:0/738614366 conn(0x7f18840f9250 msgr2=0x7f18840fb680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:56.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.234+0000 7f188959d700 1 -- 172.21.15.47:0/738614366 shutdown_connections 2022-09-07T21:38:56.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.234+0000 7f188959d700 1 -- 172.21.15.47:0/738614366 wait complete. 2022-09-07T21:38:56.241 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.235+0000 7f188959d700 1 Processor -- start 2022-09-07T21:38:56.241 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.235+0000 7f188959d700 1 -- start start 2022-09-07T21:38:56.241 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.235+0000 7f188959d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18840fdb60 0x7f18841936f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:56.241 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f188959d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18840fede0 0x7f1884193c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:56.242 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f188959d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18840ff770 0x7f188419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:56.242 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f188959d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188419b270 con 0x7f18840ff770 2022-09-07T21:38:56.242 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f188959d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188419b3e0 con 0x7f18840fede0 2022-09-07T21:38:56.242 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f188959d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188419b5a0 con 0x7f18840fdb60 2022-09-07T21:38:56.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f18837fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18840fede0 0x7f1884193c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:56.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f1888d9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18840ff770 0x7f188419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:56.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f18837fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18840fede0 0x7f1884193c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:41988/0 (socket says 172.21.15.47:41988) 2022-09-07T21:38:56.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f18837fe700 1 -- 172.21.15.47:0/903737757 learned_addr learned my addr 172.21.15.47:0/903737757 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:56.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f1883fff700 1 --2- 172.21.15.47:0/903737757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18840fdb60 0x7f18841936f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:56.244 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f18837fe700 1 -- 172.21.15.47:0/903737757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18840fdb60 msgr2=0x7f18841936f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:56.244 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f18837fe700 1 --2- 172.21.15.47:0/903737757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18840fdb60 0x7f18841936f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.244 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f18837fe700 1 -- 172.21.15.47:0/903737757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18840ff770 msgr2=0x7f188419aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:56.244 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f18837fe700 1 --2- 172.21.15.47:0/903737757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18840ff770 0x7f188419aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.236+0000 7f18837fe700 1 -- 172.21.15.47:0/903737757 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1878009470 con 0x7f18840fede0 2022-09-07T21:38:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.237+0000 7f18837fe700 1 --2- 172.21.15.47:0/903737757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18840fede0 0x7f1884193c30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f187800f690 tx=0x7f187800f6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.237+0000 7f18817fa700 1 -- 172.21.15.47:0/903737757 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187801d070 con 0x7f18840fede0 2022-09-07T21:38:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.237+0000 7f188959d700 1 -- 172.21.15.47:0/903737757 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f188419b820 con 0x7f18840fede0 2022-09-07T21:38:56.246 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.237+0000 7f18817fa700 1 -- 172.21.15.47:0/903737757 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f187800f8e0 con 0x7f18840fede0 2022-09-07T21:38:56.246 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.237+0000 7f18817fa700 1 -- 172.21.15.47:0/903737757 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18780047f0 con 0x7f18840fede0 2022-09-07T21:38:56.246 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.237+0000 7f188959d700 1 -- 172.21.15.47:0/903737757 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f188419bd20 con 0x7f18840fede0 2022-09-07T21:38:56.246 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.238+0000 7f188959d700 1 -- 172.21.15.47:0/903737757 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1884109310 con 0x7f18840fede0 2022-09-07T21:38:56.247 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.240+0000 7f18817fa700 1 -- 172.21.15.47:0/903737757 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f1878028030 con 0x7f18840fede0 2022-09-07T21:38:56.248 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.241+0000 7f18817fa700 1 --2- 172.21.15.47:0/903737757 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f186c099010 0x7f186c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:56.248 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.241+0000 7f18817fa700 1 -- 172.21.15.47:0/903737757 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f18780c3250 con 0x7f18840fede0 2022-09-07T21:38:56.248 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.242+0000 7f1883fff700 1 --2- 172.21.15.47:0/903737757 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f186c099010 0x7f186c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:56.248 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.242+0000 7f1883fff700 1 --2- 172.21.15.47:0/903737757 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f186c099010 0x7f186c09b4c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f18840fec40 tx=0x7f1874005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:56.249 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.244+0000 7f18817fa700 1 -- 172.21.15.47:0/903737757 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f187808f6f0 con 0x7f18840fede0 2022-09-07T21:38:56.448 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.446+0000 7f188959d700 1 -- 172.21.15.47:0/903737757 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1884105000 con 0x7f18840fede0 2022-09-07T21:38:56.449 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.447+0000 7f18817fa700 1 -- 172.21.15.47:0/903737757 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23119 (secure 0 0 0) 0x7f1878092d10 con 0x7f18840fede0 2022-09-07T21:38:56.450 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:56.451 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":89,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:38:45.529264+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-07T21:38:38.608937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-07T21:38:39.936252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-07T21:38:43.731797+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:38:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.450+0000 7f188959d700 1 -- 172.21.15.47:0/903737757 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f186c099010 msgr2=0x7f186c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.450+0000 7f188959d700 1 --2- 172.21.15.47:0/903737757 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f186c099010 0x7f186c09b4c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f18840fec40 tx=0x7f1874005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.450+0000 7f188959d700 1 -- 172.21.15.47:0/903737757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18840fede0 msgr2=0x7f1884193c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.450+0000 7f188959d700 1 --2- 172.21.15.47:0/903737757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18840fede0 0x7f1884193c30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f187800f690 tx=0x7f187800f6c0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.450+0000 7f188959d700 1 -- 172.21.15.47:0/903737757 shutdown_connections 2022-09-07T21:38:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.450+0000 7f188959d700 1 --2- 172.21.15.47:0/903737757 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f186c099010 0x7f186c09b4c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.450+0000 7f188959d700 1 --2- 172.21.15.47:0/903737757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18840ff770 0x7f188419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.450+0000 7f188959d700 1 --2- 172.21.15.47:0/903737757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18840fdb60 0x7f18841936f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.450+0000 7f188959d700 1 --2- 172.21.15.47:0/903737757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18840fede0 0x7f1884193c30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.450+0000 7f188959d700 1 -- 172.21.15.47:0/903737757 >> 172.21.15.47:0/903737757 conn(0x7f18840f9250 msgr2=0x7f18841022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.450+0000 7f188959d700 1 -- 172.21.15.47:0/903737757 shutdown_connections 2022-09-07T21:38:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.450+0000 7f188959d700 1 -- 172.21.15.47:0/903737757 wait complete. 2022-09-07T21:38:56.479 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-07T21:33:29.342534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-09-07T21:38:38.608937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-09-07T21:38:39.936252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2022-09-07T21:38:43.731797+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-09-07T21:38:56.480 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-09-07T21:38:56.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.715+0000 7fe003fff700 1 -- 172.21.15.47:0/2003321433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe004103870 msgr2=0x7fe004105c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:56.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.715+0000 7fe003fff700 1 --2- 172.21.15.47:0/2003321433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe004103870 0x7fe004105c50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdffc009bb0 tx=0x7fdffc009ec0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.715+0000 7fe003fff700 1 -- 172.21.15.47:0/2003321433 shutdown_connections 2022-09-07T21:38:56.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.715+0000 7fe003fff700 1 --2- 172.21.15.47:0/2003321433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe004103870 0x7fe004105c50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.715+0000 7fe003fff700 1 --2- 172.21.15.47:0/2003321433 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0040fe630 0x7fe004100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.715+0000 7fe003fff700 1 --2- 172.21.15.47:0/2003321433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe004100f50 0x7fe004103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.715+0000 7fe003fff700 1 -- 172.21.15.47:0/2003321433 >> 172.21.15.47:0/2003321433 conn(0x7fe0040f80d0 msgr2=0x7fe0040fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:56.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.715+0000 7fe003fff700 1 -- 172.21.15.47:0/2003321433 shutdown_connections 2022-09-07T21:38:56.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.715+0000 7fe003fff700 1 -- 172.21.15.47:0/2003321433 wait complete. 2022-09-07T21:38:56.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.716+0000 7fe003fff700 1 Processor -- start 2022-09-07T21:38:56.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.716+0000 7fe003fff700 1 -- start start 2022-09-07T21:38:56.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.717+0000 7fe003fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0040fe630 0x7fe004195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:56.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.717+0000 7fe003fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004100f50 0x7fe004195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:56.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.717+0000 7fe003fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe004103870 0x7fe00419ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:56.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.717+0000 7fe003fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00419d480 con 0x7fe0040fe630 2022-09-07T21:38:56.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.717+0000 7fe003fff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00419d5f0 con 0x7fe004100f50 2022-09-07T21:38:56.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.717+0000 7fe003fff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00419d7b0 con 0x7fe004103870 2022-09-07T21:38:56.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.717+0000 7fe002ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0040fe630 0x7fe004195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:56.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.717+0000 7fe0037fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe004103870 0x7fe00419ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:56.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.717+0000 7fe002ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0040fe630 0x7fe004195900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52862/0 (socket says 172.21.15.47:52862) 2022-09-07T21:38:56.727 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.717+0000 7fe002ffd700 1 -- 172.21.15.47:0/1091138254 learned_addr learned my addr 172.21.15.47:0/1091138254 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:56.727 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.717+0000 7fe0027fc700 1 --2- 172.21.15.47:0/1091138254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004100f50 0x7fe004195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:56.727 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.718+0000 7fe0037fe700 1 -- 172.21.15.47:0/1091138254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004100f50 msgr2=0x7fe004195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:56.727 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.718+0000 7fe0037fe700 1 --2- 172.21.15.47:0/1091138254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004100f50 0x7fe004195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.718+0000 7fe0037fe700 1 -- 172.21.15.47:0/1091138254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0040fe630 msgr2=0x7fe004195900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:56.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.718+0000 7fe0037fe700 1 --2- 172.21.15.47:0/1091138254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0040fe630 0x7fe004195900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.718+0000 7fe0037fe700 1 -- 172.21.15.47:0/1091138254 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc0097e0 con 0x7fe004103870 2022-09-07T21:38:56.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.718+0000 7fe002ffd700 1 --2- 172.21.15.47:0/1091138254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0040fe630 0x7fe004195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:56.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.718+0000 7fe0037fe700 1 --2- 172.21.15.47:0/1091138254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe004103870 0x7fe00419ceb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdffc009fd0 tx=0x7fdffc00b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:56.729 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.718+0000 7fdfebfff700 1 -- 172.21.15.47:0/1091138254 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc00f870 con 0x7fe004103870 2022-09-07T21:38:56.729 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.718+0000 7fdfebfff700 1 -- 172.21.15.47:0/1091138254 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fdffc00ba80 con 0x7fe004103870 2022-09-07T21:38:56.729 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.718+0000 7fe003fff700 1 -- 172.21.15.47:0/1091138254 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00419da30 con 0x7fe004103870 2022-09-07T21:38:56.729 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.718+0000 7fdfebfff700 1 -- 172.21.15.47:0/1091138254 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc017470 con 0x7fe004103870 2022-09-07T21:38:56.730 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.718+0000 7fe003fff700 1 -- 172.21.15.47:0/1091138254 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00419de90 con 0x7fe004103870 2022-09-07T21:38:56.730 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.719+0000 7fe003fff700 1 -- 172.21.15.47:0/1091138254 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0041092e0 con 0x7fe004103870 2022-09-07T21:38:56.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.721+0000 7fdfebfff700 1 -- 172.21.15.47:0/1091138254 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fdffc020070 con 0x7fe004103870 2022-09-07T21:38:56.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.723+0000 7fdfebfff700 1 --2- 172.21.15.47:0/1091138254 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdfec099010 0x7fdfec09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:56.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.723+0000 7fdfebfff700 1 -- 172.21.15.47:0/1091138254 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fdffc0c4400 con 0x7fe004103870 2022-09-07T21:38:56.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.723+0000 7fe002ffd700 1 --2- 172.21.15.47:0/1091138254 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdfec099010 0x7fdfec09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:56.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.723+0000 7fe002ffd700 1 --2- 172.21.15.47:0/1091138254 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdfec099010 0x7fdfec09b4c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdff40057d0 tx=0x7fdff4005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:56.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.725+0000 7fdfebfff700 1 -- 172.21.15.47:0/1091138254 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fdffc0908a0 con 0x7fe004103870 2022-09-07T21:38:56.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.925+0000 7fe003fff700 1 -- 172.21.15.47:0/1091138254 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe00405fb10 con 0x7fe004103870 2022-09-07T21:38:56.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.926+0000 7fdfebfff700 1 -- 172.21.15.47:0/1091138254 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdffc0c3020 con 0x7fe004103870 2022-09-07T21:38:56.929 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2022-09-07T21:38:56.931 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.929+0000 7fe003fff700 1 -- 172.21.15.47:0/1091138254 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdfec099010 msgr2=0x7fdfec09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.929+0000 7fe003fff700 1 --2- 172.21.15.47:0/1091138254 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdfec099010 0x7fdfec09b4c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdff40057d0 tx=0x7fdff4005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.929+0000 7fe003fff700 1 -- 172.21.15.47:0/1091138254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe004103870 msgr2=0x7fe00419ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.929+0000 7fe003fff700 1 --2- 172.21.15.47:0/1091138254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe004103870 0x7fe00419ceb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdffc009fd0 tx=0x7fdffc00b890 comp rx=0 tx=0).stop 2022-09-07T21:38:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.929+0000 7fe003fff700 1 -- 172.21.15.47:0/1091138254 shutdown_connections 2022-09-07T21:38:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.929+0000 7fe003fff700 1 --2- 172.21.15.47:0/1091138254 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fdfec099010 0x7fdfec09b4c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.929+0000 7fe003fff700 1 --2- 172.21.15.47:0/1091138254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe004103870 0x7fe00419ceb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.929+0000 7fe003fff700 1 --2- 172.21.15.47:0/1091138254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0040fe630 0x7fe004195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.929+0000 7fe003fff700 1 --2- 172.21.15.47:0/1091138254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004100f50 0x7fe004195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.929+0000 7fe003fff700 1 -- 172.21.15.47:0/1091138254 >> 172.21.15.47:0/1091138254 conn(0x7fe0040f80d0 msgr2=0x7fe0040fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.929+0000 7fe003fff700 1 -- 172.21.15.47:0/1091138254 shutdown_connections 2022-09-07T21:38:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:56.929+0000 7fe003fff700 1 -- 172.21.15.47:0/1091138254 wait complete. 2022-09-07T21:38:56.967 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-09-07T21:38:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:56 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/903737757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:57.045 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:56 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/903737757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:57.204 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.201+0000 7f60645ff700 1 -- 172.21.15.47:0/4071742754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f605c071740 msgr2=0x7f605c071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:57.204 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.201+0000 7f60645ff700 1 --2- 172.21.15.47:0/4071742754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f605c071740 0x7f605c071bb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f604c009810 tx=0x7f604c009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:57.205 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.201+0000 7f60645ff700 1 -- 172.21.15.47:0/4071742754 shutdown_connections 2022-09-07T21:38:57.205 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.201+0000 7f60645ff700 1 --2- 172.21.15.47:0/4071742754 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f605c105c70 0x7f605c1080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.205 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.201+0000 7f60645ff700 1 --2- 172.21.15.47:0/4071742754 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f605c072c90 0x7f605c071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.205 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.201+0000 7f60645ff700 1 --2- 172.21.15.47:0/4071742754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f605c071740 0x7f605c071bb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.206 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.201+0000 7f60645ff700 1 -- 172.21.15.47:0/4071742754 >> 172.21.15.47:0/4071742754 conn(0x7f605c0f9460 msgr2=0x7f605c0fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:57.206 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.201+0000 7f60645ff700 1 -- 172.21.15.47:0/4071742754 shutdown_connections 2022-09-07T21:38:57.206 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.201+0000 7f60645ff700 1 -- 172.21.15.47:0/4071742754 wait complete. 2022-09-07T21:38:57.212 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.202+0000 7f60645ff700 1 Processor -- start 2022-09-07T21:38:57.212 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.202+0000 7f60645ff700 1 -- start start 2022-09-07T21:38:57.212 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.203+0000 7f60645ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f605c071740 0x7f605c195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:57.213 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.203+0000 7f60645ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f605c072c90 0x7f605c1962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:57.213 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.203+0000 7f60645ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f605c105c70 0x7f605c19d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:57.213 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.203+0000 7f60645ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c19d970 con 0x7f605c072c90 2022-09-07T21:38:57.213 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.203+0000 7f60645ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c19dae0 con 0x7f605c105c70 2022-09-07T21:38:57.213 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.203+0000 7f60645ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c19dca0 con 0x7f605c071740 2022-09-07T21:38:57.214 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.203+0000 7f606239b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f605c071740 0x7f605c195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:57.214 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.203+0000 7f6061b9a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f605c072c90 0x7f605c1962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:57.214 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.203+0000 7f606239b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f605c071740 0x7f605c195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:38578/0 (socket says 172.21.15.47:38578) 2022-09-07T21:38:57.215 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.203+0000 7f606239b700 1 -- 172.21.15.47:0/2802453957 learned_addr learned my addr 172.21.15.47:0/2802453957 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:57.215 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.203+0000 7f6062b9c700 1 --2- 172.21.15.47:0/2802453957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f605c105c70 0x7f605c19d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:57.215 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.203+0000 7f606239b700 1 -- 172.21.15.47:0/2802453957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f605c105c70 msgr2=0x7f605c19d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:57.215 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.204+0000 7f606239b700 1 --2- 172.21.15.47:0/2802453957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f605c105c70 0x7f605c19d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.215 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.204+0000 7f606239b700 1 -- 172.21.15.47:0/2802453957 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f605c072c90 msgr2=0x7f605c1962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.204+0000 7f606239b700 1 --2- 172.21.15.47:0/2802453957 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f605c072c90 0x7f605c1962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.204+0000 7f606239b700 1 -- 172.21.15.47:0/2802453957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c009470 con 0x7f605c071740 2022-09-07T21:38:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.204+0000 7f6062b9c700 1 --2- 172.21.15.47:0/2802453957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f605c105c70 0x7f605c19d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-07T21:38:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.204+0000 7f606239b700 1 --2- 172.21.15.47:0/2802453957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f605c071740 0x7f605c195db0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f605800e740 tx=0x7f605800ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.204+0000 7f60537fe700 1 -- 172.21.15.47:0/2802453957 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6058005220 con 0x7f605c071740 2022-09-07T21:38:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.204+0000 7f60537fe700 1 -- 172.21.15.47:0/2802453957 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6058005380 con 0x7f605c071740 2022-09-07T21:38:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.204+0000 7f60537fe700 1 -- 172.21.15.47:0/2802453957 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6058004c10 con 0x7f605c071740 2022-09-07T21:38:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.204+0000 7f60645ff700 1 -- 172.21.15.47:0/2802453957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605c19df80 con 0x7f605c071740 2022-09-07T21:38:57.218 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.204+0000 7f60645ff700 1 -- 172.21.15.47:0/2802453957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605c19e2f0 con 0x7f605c071740 2022-09-07T21:38:57.218 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.205+0000 7f60645ff700 1 -- 172.21.15.47:0/2802453957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605c10b760 con 0x7f605c071740 2022-09-07T21:38:57.218 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.207+0000 7f60537fe700 1 -- 172.21.15.47:0/2802453957 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f605801e020 con 0x7f605c071740 2022-09-07T21:38:57.218 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.209+0000 7f60537fe700 1 --2- 172.21.15.47:0/2802453957 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6048099090 0x7f604809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:57.219 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.209+0000 7f60537fe700 1 -- 172.21.15.47:0/2802453957 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f6058014070 con 0x7f605c071740 2022-09-07T21:38:57.219 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.209+0000 7f6061b9a700 1 --2- 172.21.15.47:0/2802453957 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6048099090 0x7f604809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:57.219 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.210+0000 7f6061b9a700 1 --2- 172.21.15.47:0/2802453957 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6048099090 0x7f604809b540 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f604c009ef0 tx=0x7f604c005aa0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:57.221 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.211+0000 7f60537fe700 1 -- 172.21.15.47:0/2802453957 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f605808f350 con 0x7f605c071740 2022-09-07T21:38:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:56 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/903737757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:38:57.416 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.413+0000 7f60645ff700 1 -- 172.21.15.47:0/2802453957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f605c19e9b0 con 0x7f605c071740 2022-09-07T21:38:57.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.414+0000 7f60537fe700 1 -- 172.21.15.47:0/2802453957 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6058092970 con 0x7f605c071740 2022-09-07T21:38:57.417 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2022-09-07T21:38:57.419 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.417+0000 7f60645ff700 1 -- 172.21.15.47:0/2802453957 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6048099090 msgr2=0x7f604809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:57.420 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.417+0000 7f60645ff700 1 --2- 172.21.15.47:0/2802453957 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6048099090 0x7f604809b540 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f604c009ef0 tx=0x7f604c005aa0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.420 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.417+0000 7f60645ff700 1 -- 172.21.15.47:0/2802453957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f605c071740 msgr2=0x7f605c195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:57.420 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.417+0000 7f60645ff700 1 --2- 172.21.15.47:0/2802453957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f605c071740 0x7f605c195db0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f605800e740 tx=0x7f605800ea50 comp rx=0 tx=0).stop 2022-09-07T21:38:57.421 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.417+0000 7f60645ff700 1 -- 172.21.15.47:0/2802453957 shutdown_connections 2022-09-07T21:38:57.421 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.417+0000 7f60645ff700 1 --2- 172.21.15.47:0/2802453957 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6048099090 0x7f604809b540 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.421 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.417+0000 7f60645ff700 1 --2- 172.21.15.47:0/2802453957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f605c105c70 0x7f605c19d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.421 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.417+0000 7f60645ff700 1 --2- 172.21.15.47:0/2802453957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f605c071740 0x7f605c195db0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.421 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.417+0000 7f60645ff700 1 --2- 172.21.15.47:0/2802453957 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f605c072c90 0x7f605c1962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.422 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.417+0000 7f60645ff700 1 -- 172.21.15.47:0/2802453957 >> 172.21.15.47:0/2802453957 conn(0x7f605c0f9460 msgr2=0x7f605c0fa0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:57.422 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.417+0000 7f60645ff700 1 -- 172.21.15.47:0/2802453957 shutdown_connections 2022-09-07T21:38:57.422 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.417+0000 7f60645ff700 1 -- 172.21.15.47:0/2802453957 wait complete. 2022-09-07T21:38:57.446 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-09-07T21:38:57.683 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.679+0000 7f3def59e700 1 -- 172.21.15.47:0/3986924277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df00fdb10 msgr2=0x7f3df00fdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:57.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.679+0000 7f3def59e700 1 --2- 172.21.15.47:0/3986924277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df00fdb10 0x7f3df00fdf20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3de0009810 tx=0x7f3de0009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:57.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.679+0000 7f3def59e700 1 -- 172.21.15.47:0/3986924277 shutdown_connections 2022-09-07T21:38:57.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.679+0000 7f3def59e700 1 --2- 172.21.15.47:0/3986924277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3df00ff4e0 0x7f3df0106080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.679+0000 7f3def59e700 1 --2- 172.21.15.47:0/3986924277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df00fdb10 0x7f3df00fdf20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.679+0000 7f3def59e700 1 --2- 172.21.15.47:0/3986924277 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3df00fe460 0x7f3df00fee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.679+0000 7f3def59e700 1 -- 172.21.15.47:0/3986924277 >> 172.21.15.47:0/3986924277 conn(0x7f3df00f91c0 msgr2=0x7f3df00fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:57.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.679+0000 7f3def59e700 1 -- 172.21.15.47:0/3986924277 shutdown_connections 2022-09-07T21:38:57.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.679+0000 7f3def59e700 1 -- 172.21.15.47:0/3986924277 wait complete. 2022-09-07T21:38:57.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.680+0000 7f3def59e700 1 Processor -- start 2022-09-07T21:38:57.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.681+0000 7f3def59e700 1 -- start start 2022-09-07T21:38:57.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.681+0000 7f3def59e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3df00fdb10 0x7f3df01959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:57.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.681+0000 7f3def59e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3df00fe460 0x7f3df0195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:57.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.681+0000 7f3def59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df00ff4e0 0x7f3df019cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:57.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.681+0000 7f3def59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df019d560 con 0x7f3df00ff4e0 2022-09-07T21:38:57.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.681+0000 7f3def59e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df019d6d0 con 0x7f3df00fe460 2022-09-07T21:38:57.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.681+0000 7f3def59e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df019d890 con 0x7f3df00fdb10 2022-09-07T21:38:57.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.681+0000 7f3deed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df00ff4e0 0x7f3df019cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:57.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.681+0000 7f3dedd9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3df00fe460 0x7f3df0195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:57.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.681+0000 7f3dee59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3df00fdb10 0x7f3df01959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:57.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.682+0000 7f3deed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df00ff4e0 0x7f3df019cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52902/0 (socket says 172.21.15.47:52902) 2022-09-07T21:38:57.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.682+0000 7f3deed9d700 1 -- 172.21.15.47:0/3913834132 learned_addr learned my addr 172.21.15.47:0/3913834132 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:57.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.682+0000 7f3dedd9b700 1 -- 172.21.15.47:0/3913834132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3df00fdb10 msgr2=0x7f3df01959e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:57.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.682+0000 7f3dedd9b700 1 --2- 172.21.15.47:0/3913834132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3df00fdb10 0x7f3df01959e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.682+0000 7f3dedd9b700 1 -- 172.21.15.47:0/3913834132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df00ff4e0 msgr2=0x7f3df019cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:57.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.682+0000 7f3dedd9b700 1 --2- 172.21.15.47:0/3913834132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df00ff4e0 0x7f3df019cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.682+0000 7f3dedd9b700 1 -- 172.21.15.47:0/3913834132 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de0009470 con 0x7f3df00fe460 2022-09-07T21:38:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.682+0000 7f3dee59c700 1 --2- 172.21.15.47:0/3913834132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3df00fdb10 0x7f3df01959e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.682+0000 7f3dedd9b700 1 --2- 172.21.15.47:0/3913834132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3df00fe460 0x7f3df0195f20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3de4005fd0 tx=0x7f3de400e9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.683+0000 7f3ddf7fe700 1 -- 172.21.15.47:0/3913834132 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de4010900 con 0x7f3df00fe460 2022-09-07T21:38:57.696 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.683+0000 7f3ddf7fe700 1 -- 172.21.15.47:0/3913834132 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3de4018420 con 0x7f3df00fe460 2022-09-07T21:38:57.696 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.683+0000 7f3def59e700 1 -- 172.21.15.47:0/3913834132 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df019db70 con 0x7f3df00fe460 2022-09-07T21:38:57.696 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.683+0000 7f3ddf7fe700 1 -- 172.21.15.47:0/3913834132 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de4017610 con 0x7f3df00fe460 2022-09-07T21:38:57.696 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.683+0000 7f3def59e700 1 -- 172.21.15.47:0/3913834132 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df019e080 con 0x7f3df00fe460 2022-09-07T21:38:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.684+0000 7f3def59e700 1 -- 172.21.15.47:0/3913834132 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df0109300 con 0x7f3df00fe460 2022-09-07T21:38:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.686+0000 7f3ddf7fe700 1 -- 172.21.15.47:0/3913834132 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f3de400c040 con 0x7f3df00fe460 2022-09-07T21:38:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.687+0000 7f3ddf7fe700 1 --2- 172.21.15.47:0/3913834132 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3dd8099010 0x7f3dd809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.687+0000 7f3ddf7fe700 1 -- 172.21.15.47:0/3913834132 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f3de4014070 con 0x7f3df00fe460 2022-09-07T21:38:57.698 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.687+0000 7f3dee59c700 1 --2- 172.21.15.47:0/3913834132 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3dd8099010 0x7f3dd809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:57.698 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.688+0000 7f3dee59c700 1 --2- 172.21.15.47:0/3913834132 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3dd8099010 0x7f3dd809b4c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3de000b4f0 tx=0x7f3de0009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:57.699 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.689+0000 7f3ddf7fe700 1 -- 172.21.15.47:0/3913834132 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f3de408ed30 con 0x7f3df00fe460 2022-09-07T21:38:57.894 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.892+0000 7f3def59e700 1 -- 172.21.15.47:0/3913834132 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3df018ff10 con 0x7f3df00fe460 2022-09-07T21:38:57.895 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.893+0000 7f3ddf7fe700 1 -- 172.21.15.47:0/3913834132 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3de4026020 con 0x7f3df00fe460 2022-09-07T21:38:57.895 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2022-09-07T21:38:57.898 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.895+0000 7f3def59e700 1 -- 172.21.15.47:0/3913834132 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3dd8099010 msgr2=0x7f3dd809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:57.898 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.895+0000 7f3def59e700 1 --2- 172.21.15.47:0/3913834132 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3dd8099010 0x7f3dd809b4c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3de000b4f0 tx=0x7f3de0009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.898 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.895+0000 7f3def59e700 1 -- 172.21.15.47:0/3913834132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3df00fe460 msgr2=0x7f3df0195f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:57.899 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.895+0000 7f3def59e700 1 --2- 172.21.15.47:0/3913834132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3df00fe460 0x7f3df0195f20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3de4005fd0 tx=0x7f3de400e9a0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.899 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.895+0000 7f3def59e700 1 -- 172.21.15.47:0/3913834132 shutdown_connections 2022-09-07T21:38:57.899 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.895+0000 7f3def59e700 1 --2- 172.21.15.47:0/3913834132 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3dd8099010 0x7f3dd809b4c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.899 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.895+0000 7f3def59e700 1 --2- 172.21.15.47:0/3913834132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df00ff4e0 0x7f3df019cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.899 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.895+0000 7f3def59e700 1 --2- 172.21.15.47:0/3913834132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3df00fdb10 0x7f3df01959e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.900 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.895+0000 7f3def59e700 1 --2- 172.21.15.47:0/3913834132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3df00fe460 0x7f3df0195f20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:57.900 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.895+0000 7f3def59e700 1 -- 172.21.15.47:0/3913834132 >> 172.21.15.47:0/3913834132 conn(0x7f3df00f91c0 msgr2=0x7f3df01047b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:57.900 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.895+0000 7f3def59e700 1 -- 172.21.15.47:0/3913834132 shutdown_connections 2022-09-07T21:38:57.900 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:57.896+0000 7f3def59e700 1 -- 172.21.15.47:0/3913834132 wait complete. 2022-09-07T21:38:57.925 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2022-09-07T21:38:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:57 smithi059 ceph-mon[97450]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:57 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1091138254' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:38:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:57 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2802453957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:38:58.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:57 smithi047 ceph-mon[90087]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:58.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:57 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1091138254' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:38:58.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:57 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2802453957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:38:58.160 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.158+0000 7f84b8bff700 1 -- 172.21.15.47:0/3907749763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84b4103990 msgr2=0x7f84b4105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:58.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.158+0000 7f84b8bff700 1 --2- 172.21.15.47:0/3907749763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84b4103990 0x7f84b4105e10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f84ac009b00 tx=0x7f84ac009e10 comp rx=0 tx=0).stop 2022-09-07T21:38:58.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.158+0000 7f84b8bff700 1 -- 172.21.15.47:0/3907749763 shutdown_connections 2022-09-07T21:38:58.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.158+0000 7f84b8bff700 1 --2- 172.21.15.47:0/3907749763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84b4103990 0x7f84b4105e10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.158+0000 7f84b8bff700 1 --2- 172.21.15.47:0/3907749763 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84b4066be0 0x7f84b4100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.158+0000 7f84b8bff700 1 --2- 172.21.15.47:0/3907749763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b4100ea0 0x7f84b4103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.162 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.158+0000 7f84b8bff700 1 -- 172.21.15.47:0/3907749763 >> 172.21.15.47:0/3907749763 conn(0x7f84b40f8370 msgr2=0x7f84b40fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:58.162 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.158+0000 7f84b8bff700 1 -- 172.21.15.47:0/3907749763 shutdown_connections 2022-09-07T21:38:58.162 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.158+0000 7f84b8bff700 1 -- 172.21.15.47:0/3907749763 wait complete. 2022-09-07T21:38:58.164 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.159+0000 7f84b8bff700 1 Processor -- start 2022-09-07T21:38:58.164 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.159+0000 7f84b8bff700 1 -- start start 2022-09-07T21:38:58.165 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b8bff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b4066be0 0x7f84b4195bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:58.165 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b8bff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84b4100ea0 0x7f84b4196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:58.165 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b8bff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84b4103990 0x7f84b419d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:58.165 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b8bff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b419d770 con 0x7f84b4103990 2022-09-07T21:38:58.166 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b8bff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b419d8e0 con 0x7f84b4100ea0 2022-09-07T21:38:58.166 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b8bff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b419daa0 con 0x7f84b4066be0 2022-09-07T21:38:58.166 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b1d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84b4100ea0 0x7f84b4196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:58.166 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b2d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84b4103990 0x7f84b419d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:58.167 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b1d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84b4100ea0 0x7f84b4196130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:42038/0 (socket says 172.21.15.47:42038) 2022-09-07T21:38:58.167 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b4066be0 0x7f84b4195bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:58.167 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b1d9b700 1 -- 172.21.15.47:0/3092252927 learned_addr learned my addr 172.21.15.47:0/3092252927 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:58.167 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b2d9d700 1 -- 172.21.15.47:0/3092252927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b4066be0 msgr2=0x7f84b4195bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:58.167 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b2d9d700 1 --2- 172.21.15.47:0/3092252927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b4066be0 0x7f84b4195bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b2d9d700 1 -- 172.21.15.47:0/3092252927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84b4100ea0 msgr2=0x7f84b4196130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:58.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b2d9d700 1 --2- 172.21.15.47:0/3092252927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84b4100ea0 0x7f84b4196130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.160+0000 7f84b2d9d700 1 -- 172.21.15.47:0/3092252927 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a4009470 con 0x7f84b4103990 2022-09-07T21:38:58.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.161+0000 7f84b1d9b700 1 --2- 172.21.15.47:0/3092252927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84b4100ea0 0x7f84b4196130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:58.169 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.161+0000 7f84b2d9d700 1 --2- 172.21.15.47:0/3092252927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84b4103990 0x7f84b419d1a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00fd80 tx=0x7f84ac00f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:58.169 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.161+0000 7f84a37fe700 1 -- 172.21.15.47:0/3092252927 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac00bbe0 con 0x7f84b4103990 2022-09-07T21:38:58.169 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.161+0000 7f84b8bff700 1 -- 172.21.15.47:0/3092252927 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84ac009730 con 0x7f84b4103990 2022-09-07T21:38:58.169 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.161+0000 7f84a37fe700 1 -- 172.21.15.47:0/3092252927 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f84ac00fb20 con 0x7f84b4103990 2022-09-07T21:38:58.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.161+0000 7f84a37fe700 1 -- 172.21.15.47:0/3092252927 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac017660 con 0x7f84b4103990 2022-09-07T21:38:58.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.161+0000 7f84b8bff700 1 -- 172.21.15.47:0/3092252927 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b419e040 con 0x7f84b4103990 2022-09-07T21:38:58.171 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.162+0000 7f84b8bff700 1 -- 172.21.15.47:0/3092252927 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b41095a0 con 0x7f84b4103990 2022-09-07T21:38:58.171 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.164+0000 7f84a37fe700 1 -- 172.21.15.47:0/3092252927 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f84ac027020 con 0x7f84b4103990 2022-09-07T21:38:58.171 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.165+0000 7f84a37fe700 1 --2- 172.21.15.47:0/3092252927 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f849c099010 0x7f849c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:58.172 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.166+0000 7f84a37fe700 1 -- 172.21.15.47:0/3092252927 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f84ac0c3ca0 con 0x7f84b4103990 2022-09-07T21:38:58.172 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.166+0000 7f84b259c700 1 --2- 172.21.15.47:0/3092252927 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f849c099010 0x7f849c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:58.172 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.166+0000 7f84b259c700 1 --2- 172.21.15.47:0/3092252927 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f849c099010 0x7f849c09b4c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f84a4009440 tx=0x7f84a4005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:58.173 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.170+0000 7f84a37fe700 1 -- 172.21.15.47:0/3092252927 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f84ac090140 con 0x7f84b4103990 2022-09-07T21:38:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:57 smithi055 ceph-mon[97033]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:38:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:57 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1091138254' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:38:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:57 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2802453957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:38:58.373 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.370+0000 7f84b8bff700 1 -- 172.21.15.47:0/3092252927 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f84b405fa30 con 0x7f84b4103990 2022-09-07T21:38:58.373 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.371+0000 7f84a37fe700 1 -- 172.21.15.47:0/3092252927 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f84ac093760 con 0x7f84b4103990 2022-09-07T21:38:58.376 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2022-09-07T21:38:58.377 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.373+0000 7f84b8bff700 1 -- 172.21.15.47:0/3092252927 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f849c099010 msgr2=0x7f849c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:58.378 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.373+0000 7f84b8bff700 1 --2- 172.21.15.47:0/3092252927 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f849c099010 0x7f849c09b4c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f84a4009440 tx=0x7f84a4005f60 comp rx=0 tx=0).stop 2022-09-07T21:38:58.378 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.373+0000 7f84b8bff700 1 -- 172.21.15.47:0/3092252927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84b4103990 msgr2=0x7f84b419d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:58.378 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.373+0000 7f84b8bff700 1 --2- 172.21.15.47:0/3092252927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84b4103990 0x7f84b419d1a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00fd80 tx=0x7f84ac00f870 comp rx=0 tx=0).stop 2022-09-07T21:38:58.378 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.373+0000 7f84b8bff700 1 -- 172.21.15.47:0/3092252927 shutdown_connections 2022-09-07T21:38:58.379 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.373+0000 7f84b8bff700 1 --2- 172.21.15.47:0/3092252927 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f849c099010 0x7f849c09b4c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.379 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.373+0000 7f84b8bff700 1 --2- 172.21.15.47:0/3092252927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84b4103990 0x7f84b419d1a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.379 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.373+0000 7f84b8bff700 1 --2- 172.21.15.47:0/3092252927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b4066be0 0x7f84b4195bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.373+0000 7f84b8bff700 1 --2- 172.21.15.47:0/3092252927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84b4100ea0 0x7f84b4196130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.373+0000 7f84b8bff700 1 -- 172.21.15.47:0/3092252927 >> 172.21.15.47:0/3092252927 conn(0x7f84b40f8370 msgr2=0x7f84b40ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.374+0000 7f84b8bff700 1 -- 172.21.15.47:0/3092252927 shutdown_connections 2022-09-07T21:38:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.374+0000 7f84b8bff700 1 -- 172.21.15.47:0/3092252927 wait complete. 2022-09-07T21:38:58.403 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2022-09-07T21:38:58.636 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.633+0000 7f99b87ff700 1 -- 172.21.15.47:0/2901173133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99b0066be0 msgr2=0x7f99b0100960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:58.637 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.633+0000 7f99b87ff700 1 --2- 172.21.15.47:0/2901173133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99b0066be0 0x7f99b0100960 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f99ac009810 tx=0x7f99ac009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:58.637 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.633+0000 7f99b87ff700 1 -- 172.21.15.47:0/2901173133 shutdown_connections 2022-09-07T21:38:58.637 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.633+0000 7f99b87ff700 1 --2- 172.21.15.47:0/2901173133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99b0103990 0x7f99b0105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.637 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.633+0000 7f99b87ff700 1 --2- 172.21.15.47:0/2901173133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99b0066be0 0x7f99b0100960 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.638 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.633+0000 7f99b87ff700 1 --2- 172.21.15.47:0/2901173133 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99b0100ea0 0x7f99b0103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.638 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.633+0000 7f99b87ff700 1 -- 172.21.15.47:0/2901173133 >> 172.21.15.47:0/2901173133 conn(0x7f99b00f8370 msgr2=0x7f99b00fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:58.638 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.633+0000 7f99b87ff700 1 -- 172.21.15.47:0/2901173133 shutdown_connections 2022-09-07T21:38:58.638 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.633+0000 7f99b87ff700 1 -- 172.21.15.47:0/2901173133 wait complete. 2022-09-07T21:38:58.640 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.635+0000 7f99b87ff700 1 Processor -- start 2022-09-07T21:38:58.641 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.635+0000 7f99b87ff700 1 -- start start 2022-09-07T21:38:58.641 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.635+0000 7f99b87ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99b0066be0 0x7f99b00fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:58.641 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.635+0000 7f99b87ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99b0100ea0 0x7f99b00fc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:58.641 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.635+0000 7f99b87ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99b0103990 0x7f99b00fcea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:58.642 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.635+0000 7f99b87ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00fd640 con 0x7f99b0066be0 2022-09-07T21:38:58.642 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.635+0000 7f99b87ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00fd7b0 con 0x7f99b0103990 2022-09-07T21:38:58.642 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.635+0000 7f99b87ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b019dac0 con 0x7f99b0100ea0 2022-09-07T21:38:58.642 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.635+0000 7f99b659b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99b0066be0 0x7f99b00fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:58.643 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.635+0000 7f99b5d9a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99b0100ea0 0x7f99b00fc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:58.643 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.635+0000 7f99b659b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99b0066be0 0x7f99b00fe2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52934/0 (socket says 172.21.15.47:52934) 2022-09-07T21:38:58.643 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.635+0000 7f99b659b700 1 -- 172.21.15.47:0/1272127315 learned_addr learned my addr 172.21.15.47:0/1272127315 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:58.643 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.636+0000 7f99b6d9c700 1 --2- 172.21.15.47:0/1272127315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99b0103990 0x7f99b00fcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:58.644 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.636+0000 7f99b5d9a700 1 -- 172.21.15.47:0/1272127315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99b0103990 msgr2=0x7f99b00fcea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:58.644 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.636+0000 7f99b5d9a700 1 --2- 172.21.15.47:0/1272127315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99b0103990 0x7f99b00fcea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.644 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.636+0000 7f99b5d9a700 1 -- 172.21.15.47:0/1272127315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99b0066be0 msgr2=0x7f99b00fe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:58.644 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.636+0000 7f99b5d9a700 1 --2- 172.21.15.47:0/1272127315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99b0066be0 0x7f99b00fe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.645 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.636+0000 7f99b5d9a700 1 -- 172.21.15.47:0/1272127315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ac009470 con 0x7f99b0100ea0 2022-09-07T21:38:58.645 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.636+0000 7f99b6d9c700 1 --2- 172.21.15.47:0/1272127315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99b0103990 0x7f99b00fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:38:58.645 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.636+0000 7f99b5d9a700 1 --2- 172.21.15.47:0/1272127315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99b0100ea0 0x7f99b00fc900 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f99a000e6c0 tx=0x7f99a000e9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:58.645 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.637+0000 7f99a77fe700 1 -- 172.21.15.47:0/1272127315 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0010900 con 0x7f99b0100ea0 2022-09-07T21:38:58.645 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.637+0000 7f99b87ff700 1 -- 172.21.15.47:0/1272127315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b019dda0 con 0x7f99b0100ea0 2022-09-07T21:38:58.646 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.637+0000 7f99a77fe700 1 -- 172.21.15.47:0/1272127315 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f99a0018420 con 0x7f99b0100ea0 2022-09-07T21:38:58.646 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.637+0000 7f99a77fe700 1 -- 172.21.15.47:0/1272127315 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a000f660 con 0x7f99b0100ea0 2022-09-07T21:38:58.646 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.637+0000 7f99b87ff700 1 -- 172.21.15.47:0/1272127315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b019e270 con 0x7f99b0100ea0 2022-09-07T21:38:58.646 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.638+0000 7f99b87ff700 1 -- 172.21.15.47:0/1272127315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b0109560 con 0x7f99b0100ea0 2022-09-07T21:38:58.648 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.640+0000 7f99a77fe700 1 -- 172.21.15.47:0/1272127315 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f99a000c040 con 0x7f99b0100ea0 2022-09-07T21:38:58.648 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.641+0000 7f99a77fe700 1 --2- 172.21.15.47:0/1272127315 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f999c099090 0x7f999c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:58.648 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.641+0000 7f99a77fe700 1 -- 172.21.15.47:0/1272127315 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f99a0014070 con 0x7f99b0100ea0 2022-09-07T21:38:58.648 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.641+0000 7f99b659b700 1 --2- 172.21.15.47:0/1272127315 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f999c099090 0x7f999c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:58.649 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.642+0000 7f99b659b700 1 --2- 172.21.15.47:0/1272127315 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f999c099090 0x7f999c09b540 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f99ac00b4f0 tx=0x7f99ac009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:58.649 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.643+0000 7f99a77fe700 1 -- 172.21.15.47:0/1272127315 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f99a008eb20 con 0x7f99b0100ea0 2022-09-07T21:38:58.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.861+0000 7f99b87ff700 1 -- 172.21.15.47:0/1272127315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f99b00f9d60 con 0x7f99b0100ea0 2022-09-07T21:38:58.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.862+0000 7f99a77fe700 1 -- 172.21.15.47:0/1272127315 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+592 (secure 0 0 0) 0x7f99a0092140 con 0x7f99b0100ea0 2022-09-07T21:38:58.865 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:0/2838000558 2022-09-08T21:30:46.439577+0000 2022-09-07T21:38:58.865 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:6801/4097349239 2022-09-08T21:30:46.439577+0000 2022-09-07T21:38:58.866 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:6800/4097349239 2022-09-08T21:30:46.439577+0000 2022-09-07T21:38:58.866 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:6801/1435938941 2022-09-08T21:30:15.415366+0000 2022-09-07T21:38:58.866 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:6800/1435938941 2022-09-08T21:30:15.415366+0000 2022-09-07T21:38:58.866 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:0/2303715505 2022-09-08T21:30:15.415366+0000 2022-09-07T21:38:58.867 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:0/1816663715 2022-09-08T21:30:46.439577+0000 2022-09-07T21:38:58.867 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:0/3909884596 2022-09-08T21:30:15.415366+0000 2022-09-07T21:38:58.867 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:0/2780837564 2022-09-08T21:30:15.415366+0000 2022-09-07T21:38:58.867 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:0/1253653766 2022-09-08T21:30:46.439577+0000 2022-09-07T21:38:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.865+0000 7f99b87ff700 1 -- 172.21.15.47:0/1272127315 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f999c099090 msgr2=0x7f999c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.865+0000 7f99b87ff700 1 --2- 172.21.15.47:0/1272127315 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f999c099090 0x7f999c09b540 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f99ac00b4f0 tx=0x7f99ac009eb0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.865+0000 7f99b87ff700 1 -- 172.21.15.47:0/1272127315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99b0100ea0 msgr2=0x7f99b00fc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.865+0000 7f99b87ff700 1 --2- 172.21.15.47:0/1272127315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99b0100ea0 0x7f99b00fc900 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f99a000e6c0 tx=0x7f99a000e9d0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.865+0000 7f99b87ff700 1 -- 172.21.15.47:0/1272127315 shutdown_connections 2022-09-07T21:38:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.865+0000 7f99b87ff700 1 --2- 172.21.15.47:0/1272127315 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f999c099090 0x7f999c09b540 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.865+0000 7f99b87ff700 1 --2- 172.21.15.47:0/1272127315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99b0103990 0x7f99b00fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.865+0000 7f99b87ff700 1 --2- 172.21.15.47:0/1272127315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99b0066be0 0x7f99b00fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.865+0000 7f99b87ff700 1 --2- 172.21.15.47:0/1272127315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99b0100ea0 0x7f99b00fc900 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.865+0000 7f99b87ff700 1 -- 172.21.15.47:0/1272127315 >> 172.21.15.47:0/1272127315 conn(0x7f99b00f8370 msgr2=0x7f99b00ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.865+0000 7f99b87ff700 1 -- 172.21.15.47:0/1272127315 shutdown_connections 2022-09-07T21:38:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:58.865+0000 7f99b87ff700 1 -- 172.21.15.47:0/1272127315 wait complete. 2022-09-07T21:38:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:listed 10 entries 2022-09-07T21:38:58.894 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:38:58.895 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2022-09-07T21:38:58.955 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-07T21:38:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:58 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3913834132' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:38:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:58 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3092252927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:38:59.075 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:58 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3913834132' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:38:59.076 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:58 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3092252927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:38:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:58 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3913834132' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:38:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:58 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3092252927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:38:59.231 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.229+0000 7f1302fff700 1 -- 172.21.15.47:0/1355100291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12fc066be0 msgr2=0x7f12fc1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:59.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.229+0000 7f1302fff700 1 --2- 172.21.15.47:0/1355100291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12fc066be0 0x7f12fc1008b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f12f0009810 tx=0x7f12f0009b20 comp rx=0 tx=0).stop 2022-09-07T21:38:59.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.229+0000 7f1302fff700 1 -- 172.21.15.47:0/1355100291 shutdown_connections 2022-09-07T21:38:59.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.229+0000 7f1302fff700 1 --2- 172.21.15.47:0/1355100291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12fc1038e0 0x7f12fc105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.229+0000 7f1302fff700 1 --2- 172.21.15.47:0/1355100291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12fc066be0 0x7f12fc1008b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.229+0000 7f1302fff700 1 --2- 172.21.15.47:0/1355100291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12fc100df0 0x7f12fc103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.229+0000 7f1302fff700 1 -- 172.21.15.47:0/1355100291 >> 172.21.15.47:0/1355100291 conn(0x7f12fc0f82e0 msgr2=0x7f12fc0fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:59.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.229+0000 7f1302fff700 1 -- 172.21.15.47:0/1355100291 shutdown_connections 2022-09-07T21:38:59.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.229+0000 7f1302fff700 1 -- 172.21.15.47:0/1355100291 wait complete. 2022-09-07T21:38:59.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.230+0000 7f1302fff700 1 Processor -- start 2022-09-07T21:38:59.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.230+0000 7f1302fff700 1 -- start start 2022-09-07T21:38:59.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.231+0000 7f1302fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12fc066be0 0x7f12fc195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:59.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.231+0000 7f1302fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12fc100df0 0x7f12fc196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:59.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.231+0000 7f1302fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12fc1038e0 0x7f12fc19d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:59.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.231+0000 7f1302fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc19d6b0 con 0x7f12fc1038e0 2022-09-07T21:38:59.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.231+0000 7f1302fff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc19d820 con 0x7f12fc100df0 2022-09-07T21:38:59.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.231+0000 7f1302fff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc19d9e0 con 0x7f12fc066be0 2022-09-07T21:38:59.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.231+0000 7f130159c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12fc1038e0 0x7f12fc19d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:59.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.231+0000 7f130159c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12fc1038e0 0x7f12fc19d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33382/0 (socket says 172.21.15.47:33382) 2022-09-07T21:38:59.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.231+0000 7f1300d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12fc066be0 0x7f12fc195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:59.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.231+0000 7f12fbfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12fc100df0 0x7f12fc196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:59.239 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.231+0000 7f130159c700 1 -- 172.21.15.47:0/3074305523 learned_addr learned my addr 172.21.15.47:0/3074305523 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:59.239 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.232+0000 7f1300d9b700 1 -- 172.21.15.47:0/3074305523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12fc100df0 msgr2=0x7f12fc196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:59.239 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.232+0000 7f1300d9b700 1 --2- 172.21.15.47:0/3074305523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12fc100df0 0x7f12fc196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.239 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.232+0000 7f1300d9b700 1 -- 172.21.15.47:0/3074305523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12fc1038e0 msgr2=0x7f12fc19d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:59.239 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.232+0000 7f1300d9b700 1 --2- 172.21.15.47:0/3074305523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12fc1038e0 0x7f12fc19d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.240 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.232+0000 7f1300d9b700 1 -- 172.21.15.47:0/3074305523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f0009470 con 0x7f12fc066be0 2022-09-07T21:38:59.240 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.232+0000 7f130159c700 1 --2- 172.21.15.47:0/3074305523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12fc1038e0 0x7f12fc19d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:59.240 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.232+0000 7f1300d9b700 1 --2- 172.21.15.47:0/3074305523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12fc066be0 0x7f12fc195b30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f12f0009760 tx=0x7f12f000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:59.240 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.232+0000 7f12f9ffb700 1 -- 172.21.15.47:0/3074305523 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f001d070 con 0x7f12fc066be0 2022-09-07T21:38:59.241 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.232+0000 7f1302fff700 1 -- 172.21.15.47:0/3074305523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12fc19dc60 con 0x7f12fc066be0 2022-09-07T21:38:59.241 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.232+0000 7f12f9ffb700 1 -- 172.21.15.47:0/3074305523 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f12f000f880 con 0x7f12fc066be0 2022-09-07T21:38:59.241 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.232+0000 7f12f9ffb700 1 -- 172.21.15.47:0/3074305523 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f0004740 con 0x7f12fc066be0 2022-09-07T21:38:59.241 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.232+0000 7f1302fff700 1 -- 172.21.15.47:0/3074305523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12fc19e130 con 0x7f12fc066be0 2022-09-07T21:38:59.241 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.233+0000 7f1302fff700 1 -- 172.21.15.47:0/3074305523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12fc1094b0 con 0x7f12fc066be0 2022-09-07T21:38:59.242 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.235+0000 7f12f9ffb700 1 -- 172.21.15.47:0/3074305523 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f12f0028030 con 0x7f12fc066be0 2022-09-07T21:38:59.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.236+0000 7f12f9ffb700 1 --2- 172.21.15.47:0/3074305523 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f12ec099080 0x7f12ec09b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:59.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.237+0000 7f12f9ffb700 1 -- 172.21.15.47:0/3074305523 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f12f00c3520 con 0x7f12fc066be0 2022-09-07T21:38:59.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.237+0000 7f12fbfff700 1 --2- 172.21.15.47:0/3074305523 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f12ec099080 0x7f12ec09b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:59.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.238+0000 7f12fbfff700 1 --2- 172.21.15.47:0/3074305523 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f12ec099080 0x7f12ec09b530 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f12e8005fd0 tx=0x7f12e8005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:59.244 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.239+0000 7f12f9ffb700 1 -- 172.21.15.47:0/3074305523 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f12f008f9c0 con 0x7f12fc066be0 2022-09-07T21:38:59.445 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.443+0000 7f1302fff700 1 -- 172.21.15.47:0/3074305523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12fc0fdcb0 con 0x7f12fc066be0 2022-09-07T21:38:59.446 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.444+0000 7f12f9ffb700 1 -- 172.21.15.47:0/3074305523 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23119 (secure 0 0 0) 0x7f12f0026090 con 0x7f12fc066be0 2022-09-07T21:38:59.447 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:38:59.447 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":89,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:38:45.529264+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-07T21:38:38.608937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-07T21:38:39.936252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-07T21:38:43.731797+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:38:59.449 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.447+0000 7f1302fff700 1 -- 172.21.15.47:0/3074305523 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f12ec099080 msgr2=0x7f12ec09b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:59.450 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.447+0000 7f1302fff700 1 --2- 172.21.15.47:0/3074305523 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f12ec099080 0x7f12ec09b530 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f12e8005fd0 tx=0x7f12e8005ea0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.450 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.447+0000 7f1302fff700 1 -- 172.21.15.47:0/3074305523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12fc066be0 msgr2=0x7f12fc195b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:59.450 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.447+0000 7f1302fff700 1 --2- 172.21.15.47:0/3074305523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12fc066be0 0x7f12fc195b30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f12f0009760 tx=0x7f12f000f690 comp rx=0 tx=0).stop 2022-09-07T21:38:59.450 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.447+0000 7f1302fff700 1 -- 172.21.15.47:0/3074305523 shutdown_connections 2022-09-07T21:38:59.451 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.447+0000 7f1302fff700 1 --2- 172.21.15.47:0/3074305523 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f12ec099080 0x7f12ec09b530 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.451 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.447+0000 7f1302fff700 1 --2- 172.21.15.47:0/3074305523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12fc1038e0 0x7f12fc19d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.451 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.447+0000 7f1302fff700 1 --2- 172.21.15.47:0/3074305523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12fc066be0 0x7f12fc195b30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.451 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.447+0000 7f1302fff700 1 --2- 172.21.15.47:0/3074305523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12fc100df0 0x7f12fc196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.452 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.447+0000 7f1302fff700 1 -- 172.21.15.47:0/3074305523 >> 172.21.15.47:0/3074305523 conn(0x7f12fc0f82e0 msgr2=0x7f12fc0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:59.452 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.447+0000 7f1302fff700 1 -- 172.21.15.47:0/3074305523 shutdown_connections 2022-09-07T21:38:59.452 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.447+0000 7f1302fff700 1 -- 172.21.15.47:0/3074305523 wait complete. 2022-09-07T21:38:59.476 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-07T21:33:29.342534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-09-07T21:38:38.608937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-09-07T21:38:39.936252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2022-09-07T21:38:43.731797+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-09-07T21:38:59.477 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-09-07T21:38:59.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.710+0000 7fae8cdff700 1 -- 172.21.15.47:0/1618057216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae88103990 msgr2=0x7fae88105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:59.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.710+0000 7fae8cdff700 1 --2- 172.21.15.47:0/1618057216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae88103990 0x7fae88105e10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fae80009b00 tx=0x7fae80009e10 comp rx=0 tx=0).stop 2022-09-07T21:38:59.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.710+0000 7fae8cdff700 1 -- 172.21.15.47:0/1618057216 shutdown_connections 2022-09-07T21:38:59.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.710+0000 7fae8cdff700 1 --2- 172.21.15.47:0/1618057216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae88103990 0x7fae88105e10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.710+0000 7fae8cdff700 1 --2- 172.21.15.47:0/1618057216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae88066be0 0x7fae88100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.710+0000 7fae8cdff700 1 --2- 172.21.15.47:0/1618057216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae88100ea0 0x7fae88103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.710+0000 7fae8cdff700 1 -- 172.21.15.47:0/1618057216 >> 172.21.15.47:0/1618057216 conn(0x7fae880f8370 msgr2=0x7fae880fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:59.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.710+0000 7fae8cdff700 1 -- 172.21.15.47:0/1618057216 shutdown_connections 2022-09-07T21:38:59.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.710+0000 7fae8cdff700 1 -- 172.21.15.47:0/1618057216 wait complete. 2022-09-07T21:38:59.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.711+0000 7fae8cdff700 1 Processor -- start 2022-09-07T21:38:59.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.711+0000 7fae8cdff700 1 -- start start 2022-09-07T21:38:59.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae8cdff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae88066be0 0x7fae880fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:59.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae8cdff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae88100ea0 0x7fae880fc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:59.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae8cdff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae88103990 0x7fae880fcea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:59.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae8cdff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae880fd640 con 0x7fae88103990 2022-09-07T21:38:59.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae8cdff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae880fd7b0 con 0x7fae88066be0 2022-09-07T21:38:59.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae8cdff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8819daa0 con 0x7fae88100ea0 2022-09-07T21:38:59.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae86d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae88103990 0x7fae880fcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:59.721 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae8659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae88066be0 0x7fae880fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:59.721 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae8659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae88066be0 0x7fae880fe2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:58028/0 (socket says 172.21.15.47:58028) 2022-09-07T21:38:59.721 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae8659c700 1 -- 172.21.15.47:0/1008512552 learned_addr learned my addr 172.21.15.47:0/1008512552 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:38:59.721 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae85d9b700 1 --2- 172.21.15.47:0/1008512552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae88100ea0 0x7fae880fc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:59.721 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae86d9d700 1 -- 172.21.15.47:0/1008512552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae88100ea0 msgr2=0x7fae880fc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:59.722 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae86d9d700 1 --2- 172.21.15.47:0/1008512552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae88100ea0 0x7fae880fc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.722 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae86d9d700 1 -- 172.21.15.47:0/1008512552 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae88066be0 msgr2=0x7fae880fe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:59.722 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae86d9d700 1 --2- 172.21.15.47:0/1008512552 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae88066be0 0x7fae880fe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.722 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae86d9d700 1 -- 172.21.15.47:0/1008512552 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae78009470 con 0x7fae88103990 2022-09-07T21:38:59.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.712+0000 7fae8659c700 1 --2- 172.21.15.47:0/1008512552 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae88066be0 0x7fae880fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:38:59.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.713+0000 7fae86d9d700 1 --2- 172.21.15.47:0/1008512552 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae88103990 0x7fae880fcea0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fae8000fd80 tx=0x7fae8000f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:59.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.713+0000 7fae777fe700 1 -- 172.21.15.47:0/1008512552 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8000bbe0 con 0x7fae88103990 2022-09-07T21:38:59.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.713+0000 7fae8cdff700 1 -- 172.21.15.47:0/1008512552 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae80009730 con 0x7fae88103990 2022-09-07T21:38:59.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.713+0000 7fae777fe700 1 -- 172.21.15.47:0/1008512552 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fae8000bd40 con 0x7fae88103990 2022-09-07T21:38:59.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.713+0000 7fae777fe700 1 -- 172.21.15.47:0/1008512552 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae800176f0 con 0x7fae88103990 2022-09-07T21:38:59.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.713+0000 7fae8cdff700 1 -- 172.21.15.47:0/1008512552 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8819e090 con 0x7fae88103990 2022-09-07T21:38:59.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.714+0000 7fae8cdff700 1 -- 172.21.15.47:0/1008512552 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae881095a0 con 0x7fae88103990 2022-09-07T21:38:59.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.716+0000 7fae777fe700 1 -- 172.21.15.47:0/1008512552 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fae80027020 con 0x7fae88103990 2022-09-07T21:38:59.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.717+0000 7fae777fe700 1 --2- 172.21.15.47:0/1008512552 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fae70099090 0x7fae7009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:38:59.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.717+0000 7fae777fe700 1 -- 172.21.15.47:0/1008512552 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fae800c3cf0 con 0x7fae88103990 2022-09-07T21:38:59.727 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.718+0000 7fae8659c700 1 --2- 172.21.15.47:0/1008512552 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fae70099090 0x7fae7009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:38:59.727 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.718+0000 7fae8659c700 1 --2- 172.21.15.47:0/1008512552 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fae70099090 0x7fae7009b540 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fae78009440 tx=0x7fae78005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:38:59.727 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.721+0000 7fae777fe700 1 -- 172.21.15.47:0/1008512552 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fae80090190 con 0x7fae88103990 2022-09-07T21:38:59.923 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.920+0000 7fae8cdff700 1 -- 172.21.15.47:0/1008512552 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fae8819e640 con 0x7fae88103990 2022-09-07T21:38:59.924 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.921+0000 7fae777fe700 1 -- 172.21.15.47:0/1008512552 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fae800937b0 con 0x7fae88103990 2022-09-07T21:38:59.925 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2022-09-07T21:38:59.927 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.923+0000 7fae8cdff700 1 -- 172.21.15.47:0/1008512552 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fae70099090 msgr2=0x7fae7009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:59.927 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.923+0000 7fae8cdff700 1 --2- 172.21.15.47:0/1008512552 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fae70099090 0x7fae7009b540 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fae78009440 tx=0x7fae78005f60 comp rx=0 tx=0).stop 2022-09-07T21:38:59.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.923+0000 7fae8cdff700 1 -- 172.21.15.47:0/1008512552 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae88103990 msgr2=0x7fae880fcea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:38:59.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.923+0000 7fae8cdff700 1 --2- 172.21.15.47:0/1008512552 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae88103990 0x7fae880fcea0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fae8000fd80 tx=0x7fae8000f870 comp rx=0 tx=0).stop 2022-09-07T21:38:59.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.924+0000 7fae8cdff700 1 -- 172.21.15.47:0/1008512552 shutdown_connections 2022-09-07T21:38:59.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.924+0000 7fae8cdff700 1 --2- 172.21.15.47:0/1008512552 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fae70099090 0x7fae7009b540 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.924+0000 7fae8cdff700 1 --2- 172.21.15.47:0/1008512552 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae88103990 0x7fae880fcea0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.924+0000 7fae8cdff700 1 --2- 172.21.15.47:0/1008512552 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae88066be0 0x7fae880fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.924+0000 7fae8cdff700 1 --2- 172.21.15.47:0/1008512552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae88100ea0 0x7fae880fc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:38:59.931 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.924+0000 7fae8cdff700 1 -- 172.21.15.47:0/1008512552 >> 172.21.15.47:0/1008512552 conn(0x7fae880f8370 msgr2=0x7fae880ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:38:59.931 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.924+0000 7fae8cdff700 1 -- 172.21.15.47:0/1008512552 shutdown_connections 2022-09-07T21:38:59.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:38:59.924+0000 7fae8cdff700 1 -- 172.21.15.47:0/1008512552 wait complete. 2022-09-07T21:38:59.953 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-09-07T21:39:00.031 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:59 smithi047 ceph-mon[90087]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2022-09-07T21:39:00.031 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:59 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1272127315' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2022-09-07T21:39:00.031 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:38:59 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3074305523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:59 smithi059 ceph-mon[97450]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2022-09-07T21:39:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:59 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1272127315' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2022-09-07T21:39:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:38:59 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3074305523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:00.189 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.186+0000 7f04767ff700 1 -- 172.21.15.47:0/918800649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0470103990 msgr2=0x7f0470105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:00.190 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.186+0000 7f04767ff700 1 --2- 172.21.15.47:0/918800649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0470103990 0x7f0470105e10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0468009b00 tx=0x7f0468009e10 comp rx=0 tx=0).stop 2022-09-07T21:39:00.190 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.186+0000 7f04767ff700 1 -- 172.21.15.47:0/918800649 shutdown_connections 2022-09-07T21:39:00.190 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.186+0000 7f04767ff700 1 --2- 172.21.15.47:0/918800649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0470103990 0x7f0470105e10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.191 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.186+0000 7f04767ff700 1 --2- 172.21.15.47:0/918800649 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0470066b40 0x7f0470100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.191 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.186+0000 7f04767ff700 1 --2- 172.21.15.47:0/918800649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0470100f40 0x7f0470103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.191 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.186+0000 7f04767ff700 1 -- 172.21.15.47:0/918800649 >> 172.21.15.47:0/918800649 conn(0x7f04700f8370 msgr2=0x7f04700fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:00.191 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.186+0000 7f04767ff700 1 -- 172.21.15.47:0/918800649 shutdown_connections 2022-09-07T21:39:00.192 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.186+0000 7f04767ff700 1 -- 172.21.15.47:0/918800649 wait complete. 2022-09-07T21:39:00.194 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.188+0000 7f04767ff700 1 Processor -- start 2022-09-07T21:39:00.194 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.188+0000 7f04767ff700 1 -- start start 2022-09-07T21:39:00.195 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.188+0000 7f04767ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0470066b40 0x7f0470195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:00.195 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.188+0000 7f04767ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0470100f40 0x7f0470196100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:00.196 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.188+0000 7f04767ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0470103990 0x7f047019d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:00.196 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.188+0000 7f04767ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047019d740 con 0x7f0470066b40 2022-09-07T21:39:00.196 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.188+0000 7f04767ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047019d8b0 con 0x7f0470100f40 2022-09-07T21:39:00.196 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.188+0000 7f04767ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047019da70 con 0x7f0470103990 2022-09-07T21:39:00.197 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.189+0000 7f046ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0470066b40 0x7f0470195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:00.197 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.189+0000 7f046ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0470066b40 0x7f0470195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33434/0 (socket says 172.21.15.47:33434) 2022-09-07T21:39:00.197 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.189+0000 7f046f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0470100f40 0x7f0470196100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:00.197 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.189+0000 7f046ffff700 1 -- 172.21.15.47:0/370582209 learned_addr learned my addr 172.21.15.47:0/370582209 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:00.198 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.189+0000 7f0474d9c700 1 --2- 172.21.15.47:0/370582209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0470103990 0x7f047019d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:00.198 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.189+0000 7f046ffff700 1 -- 172.21.15.47:0/370582209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0470103990 msgr2=0x7f047019d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:00.198 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.189+0000 7f046ffff700 1 --2- 172.21.15.47:0/370582209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0470103990 0x7f047019d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.198 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.189+0000 7f046ffff700 1 -- 172.21.15.47:0/370582209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0470100f40 msgr2=0x7f0470196100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:00.198 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.189+0000 7f046ffff700 1 --2- 172.21.15.47:0/370582209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0470100f40 0x7f0470196100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.199 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.189+0000 7f046ffff700 1 -- 172.21.15.47:0/370582209 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0468009730 con 0x7f0470066b40 2022-09-07T21:39:00.199 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.189+0000 7f046ffff700 1 --2- 172.21.15.47:0/370582209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0470066b40 0x7f0470195bc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f046000e710 tx=0x7f046000ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:00.199 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.190+0000 7f046d7fa700 1 -- 172.21.15.47:0/370582209 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0460004220 con 0x7f0470066b40 2022-09-07T21:39:00.200 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.190+0000 7f046d7fa700 1 -- 172.21.15.47:0/370582209 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0460004380 con 0x7f0470066b40 2022-09-07T21:39:00.200 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.190+0000 7f04767ff700 1 -- 172.21.15.47:0/370582209 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f047019dd50 con 0x7f0470066b40 2022-09-07T21:39:00.200 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.190+0000 7f046d7fa700 1 -- 172.21.15.47:0/370582209 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0460003c80 con 0x7f0470066b40 2022-09-07T21:39:00.201 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.190+0000 7f04767ff700 1 -- 172.21.15.47:0/370582209 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047019e1f0 con 0x7f0470066b40 2022-09-07T21:39:00.201 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.191+0000 7f04767ff700 1 -- 172.21.15.47:0/370582209 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0470109580 con 0x7f0470066b40 2022-09-07T21:39:00.202 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.193+0000 7f046d7fa700 1 -- 172.21.15.47:0/370582209 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f046001e020 con 0x7f0470066b40 2022-09-07T21:39:00.202 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.194+0000 7f046d7fa700 1 --2- 172.21.15.47:0/370582209 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0458099090 0x7f045809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:00.203 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.194+0000 7f046d7fa700 1 -- 172.21.15.47:0/370582209 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f0460014070 con 0x7f0470066b40 2022-09-07T21:39:00.203 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.195+0000 7f046f7fe700 1 --2- 172.21.15.47:0/370582209 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0458099090 0x7f045809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:00.203 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.195+0000 7f046f7fe700 1 --2- 172.21.15.47:0/370582209 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0458099090 0x7f045809b540 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0470066fb0 tx=0x7f0464005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:00.203 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.198+0000 7f046d7fa700 1 -- 172.21.15.47:0/370582209 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f046008eb00 con 0x7f0470066b40 2022-09-07T21:39:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:59 smithi055 ceph-mon[97033]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2022-09-07T21:39:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:59 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1272127315' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2022-09-07T21:39:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:38:59 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3074305523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:00.402 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.400+0000 7f04767ff700 1 -- 172.21.15.47:0/370582209 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f04700fde00 con 0x7f0470066b40 2022-09-07T21:39:00.403 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.400+0000 7f046d7fa700 1 -- 172.21.15.47:0/370582209 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0460092120 con 0x7f0470066b40 2022-09-07T21:39:00.403 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2022-09-07T21:39:00.406 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.403+0000 7f04767ff700 1 -- 172.21.15.47:0/370582209 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0458099090 msgr2=0x7f045809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:00.406 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.403+0000 7f04767ff700 1 --2- 172.21.15.47:0/370582209 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0458099090 0x7f045809b540 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0470066fb0 tx=0x7f0464005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.403+0000 7f04767ff700 1 -- 172.21.15.47:0/370582209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0470066b40 msgr2=0x7f0470195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:00.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.403+0000 7f04767ff700 1 --2- 172.21.15.47:0/370582209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0470066b40 0x7f0470195bc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f046000e710 tx=0x7f046000ea20 comp rx=0 tx=0).stop 2022-09-07T21:39:00.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.403+0000 7f04767ff700 1 -- 172.21.15.47:0/370582209 shutdown_connections 2022-09-07T21:39:00.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.403+0000 7f04767ff700 1 --2- 172.21.15.47:0/370582209 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0458099090 0x7f045809b540 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.403+0000 7f04767ff700 1 --2- 172.21.15.47:0/370582209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0470103990 0x7f047019d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.403+0000 7f04767ff700 1 --2- 172.21.15.47:0/370582209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0470066b40 0x7f0470195bc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.403+0000 7f04767ff700 1 --2- 172.21.15.47:0/370582209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0470100f40 0x7f0470196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.403+0000 7f04767ff700 1 -- 172.21.15.47:0/370582209 >> 172.21.15.47:0/370582209 conn(0x7f04700f8370 msgr2=0x7f04700fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:00.409 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.404+0000 7f04767ff700 1 -- 172.21.15.47:0/370582209 shutdown_connections 2022-09-07T21:39:00.409 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.404+0000 7f04767ff700 1 -- 172.21.15.47:0/370582209 wait complete. 2022-09-07T21:39:00.433 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-09-07T21:39:00.669 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.665+0000 7fe20279d700 1 -- 172.21.15.47:0/603418831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1fc072bf0 msgr2=0x7fe1fc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:00.669 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.666+0000 7fe20279d700 1 --2- 172.21.15.47:0/603418831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1fc072bf0 0x7fe1fc071170 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8009810 tx=0x7fe1f8009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:00.670 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.666+0000 7fe20279d700 1 -- 172.21.15.47:0/603418831 shutdown_connections 2022-09-07T21:39:00.670 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.666+0000 7fe20279d700 1 --2- 172.21.15.47:0/603418831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1fc105a20 0x7fe1fc107e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.666+0000 7fe20279d700 1 --2- 172.21.15.47:0/603418831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1fc072bf0 0x7fe1fc071170 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.666+0000 7fe20279d700 1 --2- 172.21.15.47:0/603418831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1fc0716b0 0x7fe1fc071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.666+0000 7fe20279d700 1 -- 172.21.15.47:0/603418831 >> 172.21.15.47:0/603418831 conn(0x7fe1fc0f91c0 msgr2=0x7fe1fc0fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:00.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.666+0000 7fe20279d700 1 -- 172.21.15.47:0/603418831 shutdown_connections 2022-09-07T21:39:00.673 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.666+0000 7fe20279d700 1 -- 172.21.15.47:0/603418831 wait complete. 2022-09-07T21:39:00.677 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.667+0000 7fe20279d700 1 Processor -- start 2022-09-07T21:39:00.677 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.667+0000 7fe20279d700 1 -- start start 2022-09-07T21:39:00.678 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.667+0000 7fe20279d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1fc0716b0 0x7fe1fc195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:00.678 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.667+0000 7fe20279d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1fc072bf0 0x7fe1fc195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe20279d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1fc105a20 0x7fe1fc19cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe20279d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc19d4b0 con 0x7fe1fc072bf0 2022-09-07T21:39:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe20279d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc19d620 con 0x7fe1fc105a20 2022-09-07T21:39:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe20279d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc19d7e0 con 0x7fe1fc0716b0 2022-09-07T21:39:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe200f9a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1fc072bf0 0x7fe1fc195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:00.680 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe200f9a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1fc072bf0 0x7fe1fc195e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33440/0 (socket says 172.21.15.47:33440) 2022-09-07T21:39:00.680 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe200f9a700 1 -- 172.21.15.47:0/1894564774 learned_addr learned my addr 172.21.15.47:0/1894564774 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:00.680 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe20179b700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1fc0716b0 0x7fe1fc195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:00.680 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe201f9c700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1fc105a20 0x7fe1fc19cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:00.681 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe20179b700 1 -- 172.21.15.47:0/1894564774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1fc105a20 msgr2=0x7fe1fc19cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:00.681 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe20179b700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1fc105a20 0x7fe1fc19cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.681 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe20179b700 1 -- 172.21.15.47:0/1894564774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1fc072bf0 msgr2=0x7fe1fc195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:00.681 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe20179b700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1fc072bf0 0x7fe1fc195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.682 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe20179b700 1 -- 172.21.15.47:0/1894564774 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f8009470 con 0x7fe1fc0716b0 2022-09-07T21:39:00.682 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.668+0000 7fe200f9a700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1fc072bf0 0x7fe1fc195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:00.682 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.669+0000 7fe20179b700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1fc0716b0 0x7fe1fc195930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8006010 tx=0x7fe1f800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:00.682 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.669+0000 7fe1f27fc700 1 -- 172.21.15.47:0/1894564774 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f801d070 con 0x7fe1fc0716b0 2022-09-07T21:39:00.683 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.669+0000 7fe20279d700 1 -- 172.21.15.47:0/1894564774 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1fc19da60 con 0x7fe1fc0716b0 2022-09-07T21:39:00.683 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.669+0000 7fe1f27fc700 1 -- 172.21.15.47:0/1894564774 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe1f800be50 con 0x7fe1fc0716b0 2022-09-07T21:39:00.683 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.669+0000 7fe1f27fc700 1 -- 172.21.15.47:0/1894564774 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8004500 con 0x7fe1fc0716b0 2022-09-07T21:39:00.683 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.669+0000 7fe20279d700 1 -- 172.21.15.47:0/1894564774 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1fc19df00 con 0x7fe1fc0716b0 2022-09-07T21:39:00.683 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.670+0000 7fe20279d700 1 -- 172.21.15.47:0/1894564774 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1fc10b4d0 con 0x7fe1fc0716b0 2022-09-07T21:39:00.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.672+0000 7fe1f27fc700 1 -- 172.21.15.47:0/1894564774 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fe1f800c040 con 0x7fe1fc0716b0 2022-09-07T21:39:00.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.674+0000 7fe1f27fc700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe1e8099010 0x7fe1e809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:00.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.674+0000 7fe1f27fc700 1 -- 172.21.15.47:0/1894564774 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fe1f80c3830 con 0x7fe1fc0716b0 2022-09-07T21:39:00.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.674+0000 7fe200f9a700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe1e8099010 0x7fe1e809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:00.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.675+0000 7fe200f9a700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe1e8099010 0x7fe1e809b4c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec0057d0 tx=0x7fe1ec00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:00.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.676+0000 7fe1f27fc700 1 -- 172.21.15.47:0/1894564774 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fe1f808fcd0 con 0x7fe1fc0716b0 2022-09-07T21:39:00.886 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.882+0000 7fe20279d700 1 -- 172.21.15.47:0/1894564774 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe1fc104030 con 0x7fe1fc0716b0 2022-09-07T21:39:00.886 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.883+0000 7fe1f27fc700 1 -- 172.21.15.47:0/1894564774 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe1f80932f0 con 0x7fe1fc0716b0 2022-09-07T21:39:00.887 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2022-09-07T21:39:00.889 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.885+0000 7fe20279d700 1 -- 172.21.15.47:0/1894564774 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe1e8099010 msgr2=0x7fe1e809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:00.889 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.885+0000 7fe20279d700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe1e8099010 0x7fe1e809b4c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec0057d0 tx=0x7fe1ec00b390 comp rx=0 tx=0).stop 2022-09-07T21:39:00.889 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.885+0000 7fe20279d700 1 -- 172.21.15.47:0/1894564774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1fc0716b0 msgr2=0x7fe1fc195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:00.889 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.885+0000 7fe20279d700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1fc0716b0 0x7fe1fc195930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8006010 tx=0x7fe1f800f690 comp rx=0 tx=0).stop 2022-09-07T21:39:00.890 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.885+0000 7fe20279d700 1 -- 172.21.15.47:0/1894564774 shutdown_connections 2022-09-07T21:39:00.890 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.885+0000 7fe20279d700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fe1e8099010 0x7fe1e809b4c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.890 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.885+0000 7fe20279d700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1fc105a20 0x7fe1fc19cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.891 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.885+0000 7fe20279d700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1fc0716b0 0x7fe1fc195930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.891 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.885+0000 7fe20279d700 1 --2- 172.21.15.47:0/1894564774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1fc072bf0 0x7fe1fc195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:00.891 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.885+0000 7fe20279d700 1 -- 172.21.15.47:0/1894564774 >> 172.21.15.47:0/1894564774 conn(0x7fe1fc0f91c0 msgr2=0x7fe1fc0fb5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:00.891 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.885+0000 7fe20279d700 1 -- 172.21.15.47:0/1894564774 shutdown_connections 2022-09-07T21:39:00.892 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:00.885+0000 7fe20279d700 1 -- 172.21.15.47:0/1894564774 wait complete. 2022-09-07T21:39:00.914 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2022-09-07T21:39:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:00 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1008512552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:39:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:00 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/370582209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:39:01.149 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.146+0000 7f0973dff700 1 -- 172.21.15.47:0/689636980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f096c066be0 msgr2=0x7f096c100960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:01.149 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.146+0000 7f0973dff700 1 --2- 172.21.15.47:0/689636980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f096c066be0 0x7f096c100960 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0968009810 tx=0x7f0968009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:01.149 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.146+0000 7f0973dff700 1 -- 172.21.15.47:0/689636980 shutdown_connections 2022-09-07T21:39:01.150 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.146+0000 7f0973dff700 1 --2- 172.21.15.47:0/689636980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f096c103990 0x7f096c105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.150 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.146+0000 7f0973dff700 1 --2- 172.21.15.47:0/689636980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f096c066be0 0x7f096c100960 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.150 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.146+0000 7f0973dff700 1 --2- 172.21.15.47:0/689636980 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f096c100ea0 0x7f096c103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.150 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.146+0000 7f0973dff700 1 -- 172.21.15.47:0/689636980 >> 172.21.15.47:0/689636980 conn(0x7f096c0f8370 msgr2=0x7f096c0fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:01.151 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.146+0000 7f0973dff700 1 -- 172.21.15.47:0/689636980 shutdown_connections 2022-09-07T21:39:01.151 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.146+0000 7f0973dff700 1 -- 172.21.15.47:0/689636980 wait complete. 2022-09-07T21:39:01.153 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.147+0000 7f0973dff700 1 Processor -- start 2022-09-07T21:39:01.153 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.148+0000 7f0973dff700 1 -- start start 2022-09-07T21:39:01.154 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.148+0000 7f0973dff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f096c066be0 0x7f096c195c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:01.154 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.148+0000 7f0973dff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f096c100ea0 0x7f096c196160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:01.154 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.148+0000 7f0973dff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f096c103990 0x7f096c19d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:01.154 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.148+0000 7f0973dff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c19d7e0 con 0x7f096c066be0 2022-09-07T21:39:01.154 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.148+0000 7f0973dff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c19d950 con 0x7f096c100ea0 2022-09-07T21:39:01.155 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.148+0000 7f0973dff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c19db10 con 0x7f096c103990 2022-09-07T21:39:01.155 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.148+0000 7f0971b9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f096c066be0 0x7f096c195c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:01.155 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.148+0000 7f0971b9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f096c066be0 0x7f096c195c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33450/0 (socket says 172.21.15.47:33450) 2022-09-07T21:39:01.155 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.148+0000 7f0971b9b700 1 -- 172.21.15.47:0/341130238 learned_addr learned my addr 172.21.15.47:0/341130238 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:01.156 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.148+0000 7f097139a700 1 --2- 172.21.15.47:0/341130238 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f096c100ea0 0x7f096c196160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:01.156 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.148+0000 7f097239c700 1 --2- 172.21.15.47:0/341130238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f096c103990 0x7f096c19d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:01.156 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.149+0000 7f0971b9b700 1 -- 172.21.15.47:0/341130238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f096c103990 msgr2=0x7f096c19d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:01.156 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.149+0000 7f0971b9b700 1 --2- 172.21.15.47:0/341130238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f096c103990 0x7f096c19d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.156 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.149+0000 7f0971b9b700 1 -- 172.21.15.47:0/341130238 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f096c100ea0 msgr2=0x7f096c196160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:01.157 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.149+0000 7f0971b9b700 1 --2- 172.21.15.47:0/341130238 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f096c100ea0 0x7f096c196160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.157 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.149+0000 7f0971b9b700 1 -- 172.21.15.47:0/341130238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0968009470 con 0x7f096c066be0 2022-09-07T21:39:01.157 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.149+0000 7f0971b9b700 1 --2- 172.21.15.47:0/341130238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f096c066be0 0x7f096c195c20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0968000c00 tx=0x7f096800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:01.157 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.149+0000 7f0962ffd700 1 -- 172.21.15.47:0/341130238 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096801d070 con 0x7f096c066be0 2022-09-07T21:39:01.158 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.149+0000 7f0962ffd700 1 -- 172.21.15.47:0/341130238 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f096800f880 con 0x7f096c066be0 2022-09-07T21:39:01.158 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.149+0000 7f0973dff700 1 -- 172.21.15.47:0/341130238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096c19dd90 con 0x7f096c066be0 2022-09-07T21:39:01.158 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.149+0000 7f0962ffd700 1 -- 172.21.15.47:0/341130238 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09680047f0 con 0x7f096c066be0 2022-09-07T21:39:01.158 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.150+0000 7f0973dff700 1 -- 172.21.15.47:0/341130238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096c19e190 con 0x7f096c066be0 2022-09-07T21:39:01.159 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:00 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1008512552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:39:01.159 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:00 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/370582209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:39:01.160 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.151+0000 7f0973dff700 1 -- 172.21.15.47:0/341130238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096c109560 con 0x7f096c066be0 2022-09-07T21:39:01.160 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.153+0000 7f0962ffd700 1 -- 172.21.15.47:0/341130238 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f0968028030 con 0x7f096c066be0 2022-09-07T21:39:01.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.154+0000 7f0962ffd700 1 --2- 172.21.15.47:0/341130238 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0958099010 0x7f095809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:01.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.154+0000 7f0962ffd700 1 -- 172.21.15.47:0/341130238 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f09680c3220 con 0x7f096c066be0 2022-09-07T21:39:01.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.154+0000 7f097139a700 1 --2- 172.21.15.47:0/341130238 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0958099010 0x7f095809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:01.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.155+0000 7f097139a700 1 --2- 172.21.15.47:0/341130238 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0958099010 0x7f095809b4c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f095c005fd0 tx=0x7f095c005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:01.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.158+0000 7f0962ffd700 1 -- 172.21.15.47:0/341130238 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f096808f6c0 con 0x7f096c066be0 2022-09-07T21:39:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:00 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1008512552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:39:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:00 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/370582209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:39:01.366 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.364+0000 7f0973dff700 1 -- 172.21.15.47:0/341130238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f096c05f9c0 con 0x7f096c066be0 2022-09-07T21:39:01.367 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.364+0000 7f0962ffd700 1 -- 172.21.15.47:0/341130238 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0968059e60 con 0x7f096c066be0 2022-09-07T21:39:01.368 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2022-09-07T21:39:01.369 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.367+0000 7f0973dff700 1 -- 172.21.15.47:0/341130238 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0958099010 msgr2=0x7f095809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:01.370 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.367+0000 7f0973dff700 1 --2- 172.21.15.47:0/341130238 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0958099010 0x7f095809b4c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f095c005fd0 tx=0x7f095c005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.370 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.367+0000 7f0973dff700 1 -- 172.21.15.47:0/341130238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f096c066be0 msgr2=0x7f096c195c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:01.370 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.367+0000 7f0973dff700 1 --2- 172.21.15.47:0/341130238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f096c066be0 0x7f096c195c20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0968000c00 tx=0x7f096800f690 comp rx=0 tx=0).stop 2022-09-07T21:39:01.371 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.367+0000 7f0973dff700 1 -- 172.21.15.47:0/341130238 shutdown_connections 2022-09-07T21:39:01.371 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.367+0000 7f0973dff700 1 --2- 172.21.15.47:0/341130238 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0958099010 0x7f095809b4c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.371 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.367+0000 7f0973dff700 1 --2- 172.21.15.47:0/341130238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f096c103990 0x7f096c19d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.371 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.367+0000 7f0973dff700 1 --2- 172.21.15.47:0/341130238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f096c066be0 0x7f096c195c20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.371 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.367+0000 7f0973dff700 1 --2- 172.21.15.47:0/341130238 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f096c100ea0 0x7f096c196160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.372 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.367+0000 7f0973dff700 1 -- 172.21.15.47:0/341130238 >> 172.21.15.47:0/341130238 conn(0x7f096c0f8370 msgr2=0x7f096c0fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:01.372 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.367+0000 7f0973dff700 1 -- 172.21.15.47:0/341130238 shutdown_connections 2022-09-07T21:39:01.372 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.367+0000 7f0973dff700 1 -- 172.21.15.47:0/341130238 wait complete. 2022-09-07T21:39:01.397 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2022-09-07T21:39:01.631 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.628+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/2322896458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a840fef90 msgr2=0x7f6a840ff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:01.631 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.628+0000 7f6a8c1ff700 1 --2- 172.21.15.47:0/2322896458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a840fef90 0x7f6a840ff400 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6a74009810 tx=0x7f6a74009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:01.632 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.628+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/2322896458 shutdown_connections 2022-09-07T21:39:01.632 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.628+0000 7f6a8c1ff700 1 --2- 172.21.15.47:0/2322896458 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a840ff940 0x7f6a84106270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.632 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.628+0000 7f6a8c1ff700 1 --2- 172.21.15.47:0/2322896458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a840fddb0 0x7f6a840fe1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.632 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.628+0000 7f6a8c1ff700 1 --2- 172.21.15.47:0/2322896458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a840fef90 0x7f6a840ff400 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.633 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.628+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/2322896458 >> 172.21.15.47:0/2322896458 conn(0x7f6a840f9460 msgr2=0x7f6a840fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:01.633 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.628+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/2322896458 shutdown_connections 2022-09-07T21:39:01.633 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.628+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/2322896458 wait complete. 2022-09-07T21:39:01.636 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.630+0000 7f6a8c1ff700 1 Processor -- start 2022-09-07T21:39:01.636 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.630+0000 7f6a8c1ff700 1 -- start start 2022-09-07T21:39:01.636 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.630+0000 7f6a8c1ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a840fddb0 0x7f6a84195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:01.636 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.630+0000 7f6a8c1ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a840fef90 0x7f6a84196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:01.637 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.630+0000 7f6a8c1ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a840ff940 0x7f6a8419d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:01.637 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.630+0000 7f6a8c1ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8419d6c0 con 0x7f6a840ff940 2022-09-07T21:39:01.637 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.630+0000 7f6a8c1ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8419d830 con 0x7f6a840fef90 2022-09-07T21:39:01.637 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.630+0000 7f6a8c1ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8419d9f0 con 0x7f6a840fddb0 2022-09-07T21:39:01.638 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.630+0000 7f6a89f9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a840fddb0 0x7f6a84195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:01.638 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.630+0000 7f6a89f9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a840fddb0 0x7f6a84195b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:42752/0 (socket says 172.21.15.47:42752) 2022-09-07T21:39:01.638 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.630+0000 7f6a89f9b700 1 -- 172.21.15.47:0/4282763058 learned_addr learned my addr 172.21.15.47:0/4282763058 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:01.638 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.631+0000 7f6a8979a700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a840fef90 0x7f6a84196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:01.639 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.631+0000 7f6a8a79c700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a840ff940 0x7f6a8419d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:01.639 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.631+0000 7f6a89f9b700 1 -- 172.21.15.47:0/4282763058 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a840fef90 msgr2=0x7f6a84196080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:01.639 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.631+0000 7f6a89f9b700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a840fef90 0x7f6a84196080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.640 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.631+0000 7f6a89f9b700 1 -- 172.21.15.47:0/4282763058 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a840ff940 msgr2=0x7f6a8419d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:01.640 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.631+0000 7f6a89f9b700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a840ff940 0x7f6a8419d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.640 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.631+0000 7f6a89f9b700 1 -- 172.21.15.47:0/4282763058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a800093b0 con 0x7f6a840fddb0 2022-09-07T21:39:01.641 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.631+0000 7f6a8979a700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a840fef90 0x7f6a84196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:39:01.641 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.631+0000 7f6a8a79c700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a840ff940 0x7f6a8419d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:39:01.641 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.631+0000 7f6a89f9b700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a840fddb0 0x7f6a84195b40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6a8000e9c0 tx=0x7f6a8000ed80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:01.641 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.631+0000 7f6a7affd700 1 -- 172.21.15.47:0/4282763058 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a800097b0 con 0x7f6a840fddb0 2022-09-07T21:39:01.642 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.631+0000 7f6a7affd700 1 -- 172.21.15.47:0/4282763058 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6a8000ce70 con 0x7f6a840fddb0 2022-09-07T21:39:01.642 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.631+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/4282763058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a74009470 con 0x7f6a840fddb0 2022-09-07T21:39:01.642 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.631+0000 7f6a7affd700 1 -- 172.21.15.47:0/4282763058 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a80003ea0 con 0x7f6a840fddb0 2022-09-07T21:39:01.642 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.632+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/4282763058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a8419e050 con 0x7f6a840fddb0 2022-09-07T21:39:01.643 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.633+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/4282763058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a84109560 con 0x7f6a840fddb0 2022-09-07T21:39:01.644 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.635+0000 7f6a7affd700 1 -- 172.21.15.47:0/4282763058 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f6a8001e020 con 0x7f6a840fddb0 2022-09-07T21:39:01.645 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.636+0000 7f6a7affd700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6a70099010 0x7f6a7009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:01.645 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.636+0000 7f6a7affd700 1 -- 172.21.15.47:0/4282763058 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f6a80014070 con 0x7f6a840fddb0 2022-09-07T21:39:01.646 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.636+0000 7f6a8979a700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6a70099010 0x7f6a7009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:01.646 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.637+0000 7f6a8979a700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6a70099010 0x7f6a7009b4c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400f4d0 tx=0x7f6a74009e90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:01.647 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.640+0000 7f6a7affd700 1 -- 172.21.15.47:0/4282763058 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f6a8008f050 con 0x7f6a840fddb0 2022-09-07T21:39:01.861 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.859+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/4282763058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6a8419e8b0 con 0x7f6a840fddb0 2022-09-07T21:39:01.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.860+0000 7f6a7affd700 1 -- 172.21.15.47:0/4282763058 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+592 (secure 0 0 0) 0x7f6a80092670 con 0x7f6a840fddb0 2022-09-07T21:39:01.863 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:0/2838000558 2022-09-08T21:30:46.439577+0000 2022-09-07T21:39:01.863 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:6801/4097349239 2022-09-08T21:30:46.439577+0000 2022-09-07T21:39:01.863 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:6800/4097349239 2022-09-08T21:30:46.439577+0000 2022-09-07T21:39:01.864 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:6801/1435938941 2022-09-08T21:30:15.415366+0000 2022-09-07T21:39:01.864 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:6800/1435938941 2022-09-08T21:30:15.415366+0000 2022-09-07T21:39:01.864 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:0/2303715505 2022-09-08T21:30:15.415366+0000 2022-09-07T21:39:01.864 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:0/1816663715 2022-09-08T21:30:46.439577+0000 2022-09-07T21:39:01.865 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:0/3909884596 2022-09-08T21:30:15.415366+0000 2022-09-07T21:39:01.865 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:0/2780837564 2022-09-08T21:30:15.415366+0000 2022-09-07T21:39:01.865 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:0/1253653766 2022-09-08T21:30:46.439577+0000 2022-09-07T21:39:01.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.862+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/4282763058 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6a70099010 msgr2=0x7f6a7009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:01.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.862+0000 7f6a8c1ff700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6a70099010 0x7f6a7009b4c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400f4d0 tx=0x7f6a74009e90 comp rx=0 tx=0).stop 2022-09-07T21:39:01.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.862+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/4282763058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a840fddb0 msgr2=0x7f6a84195b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:01.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.862+0000 7f6a8c1ff700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a840fddb0 0x7f6a84195b40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6a8000e9c0 tx=0x7f6a8000ed80 comp rx=0 tx=0).stop 2022-09-07T21:39:01.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.862+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/4282763058 shutdown_connections 2022-09-07T21:39:01.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.862+0000 7f6a8c1ff700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6a70099010 0x7f6a7009b4c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.862+0000 7f6a8c1ff700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a840ff940 0x7f6a8419d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.862+0000 7f6a8c1ff700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a840fddb0 0x7f6a84195b40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.862+0000 7f6a8c1ff700 1 --2- 172.21.15.47:0/4282763058 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a840fef90 0x7f6a84196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:01.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.862+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/4282763058 >> 172.21.15.47:0/4282763058 conn(0x7f6a840f9460 msgr2=0x7f6a840fb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:01.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.862+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/4282763058 shutdown_connections 2022-09-07T21:39:01.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:01.862+0000 7f6a8c1ff700 1 -- 172.21.15.47:0/4282763058 wait complete. 2022-09-07T21:39:01.870 INFO:teuthology.orchestra.run.smithi047.stderr:listed 10 entries 2022-09-07T21:39:01.892 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi047.front.sepia.ceph.com... 2022-09-07T21:39:01.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:39:01.893 DEBUG:teuthology.orchestra.run.smithi047:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2022-09-07T21:39:01.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi047.stderr:ceph-fuse[127948]: fuse finished with error 0 and tester_r 0 2022-09-07T21:39:01.984 INFO:teuthology.orchestra.run:waiting for 300 2022-09-07T21:39:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:01 smithi059 ceph-mon[97450]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2022-09-07T21:39:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:01 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1894564774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:39:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:01 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/341130238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:39:02.051 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:01 smithi047 ceph-mon[90087]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2022-09-07T21:39:02.051 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:01 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1894564774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:39:02.051 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:01 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/341130238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:39:02.059 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi047.front.sepia.ceph.com 2022-09-07T21:39:02.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:39:02.060 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2022-09-07T21:39:02.137 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-07T21:39:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:01 smithi055 ceph-mon[97033]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2022-09-07T21:39:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:01 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1894564774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:39:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:01 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/341130238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:39:02.369 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.366+0000 7f98d899d700 1 -- 172.21.15.47:0/3962131719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98d40fe630 msgr2=0x7f98d4100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:02.370 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.366+0000 7f98d899d700 1 --2- 172.21.15.47:0/3962131719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98d40fe630 0x7f98d4100a10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009810 tx=0x7f98c4009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:02.370 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.366+0000 7f98d899d700 1 -- 172.21.15.47:0/3962131719 shutdown_connections 2022-09-07T21:39:02.371 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.367+0000 7f98d899d700 1 --2- 172.21.15.47:0/3962131719 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98d4103870 0x7f98d4105c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.371 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.367+0000 7f98d899d700 1 --2- 172.21.15.47:0/3962131719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98d40fe630 0x7f98d4100a10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.371 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.367+0000 7f98d899d700 1 --2- 172.21.15.47:0/3962131719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98d4100f50 0x7f98d4103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.371 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.367+0000 7f98d899d700 1 -- 172.21.15.47:0/3962131719 >> 172.21.15.47:0/3962131719 conn(0x7f98d40f80d0 msgr2=0x7f98d40fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:02.372 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.367+0000 7f98d899d700 1 -- 172.21.15.47:0/3962131719 shutdown_connections 2022-09-07T21:39:02.372 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.367+0000 7f98d899d700 1 -- 172.21.15.47:0/3962131719 wait complete. 2022-09-07T21:39:02.374 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.368+0000 7f98d899d700 1 Processor -- start 2022-09-07T21:39:02.375 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.368+0000 7f98d899d700 1 -- start start 2022-09-07T21:39:02.375 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.368+0000 7f98d899d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98d40fe630 0x7f98d4195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:02.375 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.368+0000 7f98d899d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98d4100f50 0x7f98d4195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:02.375 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.368+0000 7f98d899d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98d4103870 0x7f98d419ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:02.376 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.369+0000 7f98d899d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d419d3e0 con 0x7f98d4100f50 2022-09-07T21:39:02.376 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.369+0000 7f98d899d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d419d550 con 0x7f98d40fe630 2022-09-07T21:39:02.376 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.369+0000 7f98d899d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d419d710 con 0x7f98d4103870 2022-09-07T21:39:02.377 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.369+0000 7f98d37fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98d40fe630 0x7f98d4195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:02.377 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.369+0000 7f98d37fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98d40fe630 0x7f98d4195860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:58120/0 (socket says 172.21.15.47:58120) 2022-09-07T21:39:02.377 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.369+0000 7f98d37fe700 1 -- 172.21.15.47:0/1819648566 learned_addr learned my addr 172.21.15.47:0/1819648566 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:02.378 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.369+0000 7f98d2ffd700 1 --2- 172.21.15.47:0/1819648566 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98d4100f50 0x7f98d4195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:02.378 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.369+0000 7f98d3fff700 1 --2- 172.21.15.47:0/1819648566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98d4103870 0x7f98d419ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:02.378 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.369+0000 7f98d37fe700 1 -- 172.21.15.47:0/1819648566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98d4103870 msgr2=0x7f98d419ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:02.378 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.369+0000 7f98d37fe700 1 --2- 172.21.15.47:0/1819648566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98d4103870 0x7f98d419ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.379 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.369+0000 7f98d37fe700 1 -- 172.21.15.47:0/1819648566 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98d4100f50 msgr2=0x7f98d4195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:02.379 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.369+0000 7f98d37fe700 1 --2- 172.21.15.47:0/1819648566 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98d4100f50 0x7f98d4195da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.379 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.369+0000 7f98d37fe700 1 -- 172.21.15.47:0/1819648566 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c4009470 con 0x7f98d40fe630 2022-09-07T21:39:02.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.370+0000 7f98d37fe700 1 --2- 172.21.15.47:0/1819648566 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98d40fe630 0x7f98d4195860 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f98c4000c00 tx=0x7f98c400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:02.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.370+0000 7f98d0ff9700 1 -- 172.21.15.47:0/1819648566 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c401d070 con 0x7f98d40fe630 2022-09-07T21:39:02.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.370+0000 7f98d899d700 1 -- 172.21.15.47:0/1819648566 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d419d990 con 0x7f98d40fe630 2022-09-07T21:39:02.381 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.370+0000 7f98d0ff9700 1 -- 172.21.15.47:0/1819648566 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f98c400f8b0 con 0x7f98d40fe630 2022-09-07T21:39:02.381 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.370+0000 7f98d0ff9700 1 -- 172.21.15.47:0/1819648566 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4004880 con 0x7f98d40fe630 2022-09-07T21:39:02.381 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.370+0000 7f98d899d700 1 -- 172.21.15.47:0/1819648566 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d419de90 con 0x7f98d40fe630 2022-09-07T21:39:02.382 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.371+0000 7f98d899d700 1 -- 172.21.15.47:0/1819648566 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d4109260 con 0x7f98d40fe630 2022-09-07T21:39:02.383 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.373+0000 7f98d0ff9700 1 -- 172.21.15.47:0/1819648566 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f98c4028030 con 0x7f98d40fe630 2022-09-07T21:39:02.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.374+0000 7f98d0ff9700 1 --2- 172.21.15.47:0/1819648566 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f98bc099010 0x7f98bc09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:02.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.374+0000 7f98d0ff9700 1 -- 172.21.15.47:0/1819648566 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f98c40c32c0 con 0x7f98d40fe630 2022-09-07T21:39:02.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.375+0000 7f98d2ffd700 1 --2- 172.21.15.47:0/1819648566 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f98bc099010 0x7f98bc09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:02.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.375+0000 7f98d2ffd700 1 --2- 172.21.15.47:0/1819648566 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f98bc099010 0x7f98bc09b4c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f98c8005fd0 tx=0x7f98c8005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:02.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.377+0000 7f98d0ff9700 1 -- 172.21.15.47:0/1819648566 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f98c408f760 con 0x7f98d40fe630 2022-09-07T21:39:02.589 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.587+0000 7f98d899d700 1 -- 172.21.15.47:0/1819648566 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98d4105260 con 0x7f98d40fe630 2022-09-07T21:39:02.591 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.588+0000 7f98d0ff9700 1 -- 172.21.15.47:0/1819648566 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23119 (secure 0 0 0) 0x7f98c4092d80 con 0x7f98d40fe630 2022-09-07T21:39:02.591 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:39:02.592 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":89,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:38:45.529264+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-07T21:38:38.608937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-07T21:38:39.936252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-07T21:38:43.731797+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:39:02.593 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.591+0000 7f98d899d700 1 -- 172.21.15.47:0/1819648566 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f98bc099010 msgr2=0x7f98bc09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:02.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.591+0000 7f98d899d700 1 --2- 172.21.15.47:0/1819648566 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f98bc099010 0x7f98bc09b4c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f98c8005fd0 tx=0x7f98c8005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.591+0000 7f98d899d700 1 -- 172.21.15.47:0/1819648566 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98d40fe630 msgr2=0x7f98d4195860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:02.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.591+0000 7f98d899d700 1 --2- 172.21.15.47:0/1819648566 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98d40fe630 0x7f98d4195860 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f98c4000c00 tx=0x7f98c400f690 comp rx=0 tx=0).stop 2022-09-07T21:39:02.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.591+0000 7f98d899d700 1 -- 172.21.15.47:0/1819648566 shutdown_connections 2022-09-07T21:39:02.595 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.591+0000 7f98d899d700 1 --2- 172.21.15.47:0/1819648566 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f98bc099010 0x7f98bc09b4c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.595 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.591+0000 7f98d899d700 1 --2- 172.21.15.47:0/1819648566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98d4103870 0x7f98d419ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.595 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.591+0000 7f98d899d700 1 --2- 172.21.15.47:0/1819648566 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98d40fe630 0x7f98d4195860 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.595 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.591+0000 7f98d899d700 1 --2- 172.21.15.47:0/1819648566 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98d4100f50 0x7f98d4195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.596 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.591+0000 7f98d899d700 1 -- 172.21.15.47:0/1819648566 >> 172.21.15.47:0/1819648566 conn(0x7f98d40f80d0 msgr2=0x7f98d40ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:02.596 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.591+0000 7f98d899d700 1 -- 172.21.15.47:0/1819648566 shutdown_connections 2022-09-07T21:39:02.596 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.591+0000 7f98d899d700 1 -- 172.21.15.47:0/1819648566 wait complete. 2022-09-07T21:39:02.622 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-07T21:39:02.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.855+0000 7ffbc17ff700 1 -- 172.21.15.47:0/3782546495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbbc100df0 msgr2=0x7ffbbc103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:02.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.855+0000 7ffbc17ff700 1 --2- 172.21.15.47:0/3782546495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbbc100df0 0x7ffbbc103250 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009810 tx=0x7ffbb0009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:02.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.855+0000 7ffbc17ff700 1 -- 172.21.15.47:0/3782546495 shutdown_connections 2022-09-07T21:39:02.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.855+0000 7ffbc17ff700 1 --2- 172.21.15.47:0/3782546495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc1038e0 0x7ffbbc105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.855+0000 7ffbc17ff700 1 --2- 172.21.15.47:0/3782546495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbbc066be0 0x7ffbbc1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.855+0000 7ffbc17ff700 1 --2- 172.21.15.47:0/3782546495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbbc100df0 0x7ffbbc103250 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.855+0000 7ffbc17ff700 1 -- 172.21.15.47:0/3782546495 >> 172.21.15.47:0/3782546495 conn(0x7ffbbc0f82e0 msgr2=0x7ffbbc0fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:02.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.855+0000 7ffbc17ff700 1 -- 172.21.15.47:0/3782546495 shutdown_connections 2022-09-07T21:39:02.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.855+0000 7ffbc17ff700 1 -- 172.21.15.47:0/3782546495 wait complete. 2022-09-07T21:39:02.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.856+0000 7ffbc17ff700 1 Processor -- start 2022-09-07T21:39:02.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.857+0000 7ffbc17ff700 1 -- start start 2022-09-07T21:39:02.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.857+0000 7ffbc17ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc066be0 0x7ffbbc195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:02.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.857+0000 7ffbc17ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbbc100df0 0x7ffbbc196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:02.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.857+0000 7ffbc17ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbbc1038e0 0x7ffbbc19d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:02.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.857+0000 7ffbc17ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc19d6c0 con 0x7ffbbc100df0 2022-09-07T21:39:02.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.857+0000 7ffbc17ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc19d830 con 0x7ffbbc066be0 2022-09-07T21:39:02.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.857+0000 7ffbc17ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc19d9f0 con 0x7ffbbc1038e0 2022-09-07T21:39:02.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.857+0000 7ffbba7fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbbc100df0 0x7ffbbc196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:02.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.857+0000 7ffbba7fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbbc100df0 0x7ffbbc196080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33508/0 (socket says 172.21.15.47:33508) 2022-09-07T21:39:02.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.857+0000 7ffbba7fc700 1 -- 172.21.15.47:0/1633183003 learned_addr learned my addr 172.21.15.47:0/1633183003 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:02.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.857+0000 7ffbbaffd700 1 --2- 172.21.15.47:0/1633183003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc066be0 0x7ffbbc195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:02.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.857+0000 7ffbbb7fe700 1 --2- 172.21.15.47:0/1633183003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbbc1038e0 0x7ffbbc19d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:02.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.858+0000 7ffbba7fc700 1 -- 172.21.15.47:0/1633183003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbbc1038e0 msgr2=0x7ffbbc19d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:02.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.858+0000 7ffbba7fc700 1 --2- 172.21.15.47:0/1633183003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbbc1038e0 0x7ffbbc19d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.858+0000 7ffbba7fc700 1 -- 172.21.15.47:0/1633183003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc066be0 msgr2=0x7ffbbc195b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:02.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.858+0000 7ffbba7fc700 1 --2- 172.21.15.47:0/1633183003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc066be0 0x7ffbbc195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:02.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.858+0000 7ffbba7fc700 1 -- 172.21.15.47:0/1633183003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb0009470 con 0x7ffbbc100df0 2022-09-07T21:39:02.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.858+0000 7ffbba7fc700 1 --2- 172.21.15.47:0/1633183003 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbbc100df0 0x7ffbbc196080 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00059c0 tx=0x7ffbb000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:02.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.858+0000 7ffb9ffff700 1 -- 172.21.15.47:0/1633183003 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb001d070 con 0x7ffbbc100df0 2022-09-07T21:39:02.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.858+0000 7ffb9ffff700 1 -- 172.21.15.47:0/1633183003 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ffbb000f880 con 0x7ffbbc100df0 2022-09-07T21:39:02.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.858+0000 7ffbc17ff700 1 -- 172.21.15.47:0/1633183003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbbc19dc70 con 0x7ffbbc100df0 2022-09-07T21:39:02.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.858+0000 7ffb9ffff700 1 -- 172.21.15.47:0/1633183003 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00047d0 con 0x7ffbbc100df0 2022-09-07T21:39:02.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.859+0000 7ffbc17ff700 1 -- 172.21.15.47:0/1633183003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbbc19e120 con 0x7ffbbc100df0 2022-09-07T21:39:02.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.860+0000 7ffbc17ff700 1 -- 172.21.15.47:0/1633183003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbbc1094f0 con 0x7ffbbc100df0 2022-09-07T21:39:02.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.861+0000 7ffb9ffff700 1 -- 172.21.15.47:0/1633183003 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7ffbb0028030 con 0x7ffbbc100df0 2022-09-07T21:39:02.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.863+0000 7ffb9ffff700 1 --2- 172.21.15.47:0/1633183003 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ffba4099000 0x7ffba409b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:02.871 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.863+0000 7ffb9ffff700 1 -- 172.21.15.47:0/1633183003 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7ffbb00c3270 con 0x7ffbbc100df0 2022-09-07T21:39:02.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.863+0000 7ffbbaffd700 1 --2- 172.21.15.47:0/1633183003 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ffba4099000 0x7ffba409b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:02.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.864+0000 7ffbbaffd700 1 --2- 172.21.15.47:0/1633183003 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ffba4099000 0x7ffba409b4b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc066fb0 tx=0x7ffbac005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:02.872 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:02.866+0000 7ffb9ffff700 1 -- 172.21.15.47:0/1633183003 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ffbb008f710 con 0x7ffbbc100df0 2022-09-07T21:39:02.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:02 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/4282763058' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2022-09-07T21:39:02.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:02 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1819648566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:02 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/4282763058' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2022-09-07T21:39:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:02 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1819648566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:03.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.068+0000 7ffbc17ff700 1 -- 172.21.15.47:0/1633183003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbbc05fb40 con 0x7ffbbc100df0 2022-09-07T21:39:03.072 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.069+0000 7ffb9ffff700 1 -- 172.21.15.47:0/1633183003 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23119 (secure 0 0 0) 0x7ffbb0092d30 con 0x7ffbbc100df0 2022-09-07T21:39:03.073 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:39:03.074 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":89,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:38:45.529264+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-07T21:38:38.608937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-07T21:38:39.936252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-07T21:38:43.731797+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:39:03.076 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.071+0000 7ffbc17ff700 1 -- 172.21.15.47:0/1633183003 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ffba4099000 msgr2=0x7ffba409b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:03.076 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.071+0000 7ffbc17ff700 1 --2- 172.21.15.47:0/1633183003 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ffba4099000 0x7ffba409b4b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc066fb0 tx=0x7ffbac005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.077 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.071+0000 7ffbc17ff700 1 -- 172.21.15.47:0/1633183003 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbbc100df0 msgr2=0x7ffbbc196080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:03.077 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.071+0000 7ffbc17ff700 1 --2- 172.21.15.47:0/1633183003 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbbc100df0 0x7ffbbc196080 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00059c0 tx=0x7ffbb000f690 comp rx=0 tx=0).stop 2022-09-07T21:39:03.078 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.072+0000 7ffbc17ff700 1 -- 172.21.15.47:0/1633183003 shutdown_connections 2022-09-07T21:39:03.078 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.072+0000 7ffbc17ff700 1 --2- 172.21.15.47:0/1633183003 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ffba4099000 0x7ffba409b4b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.072+0000 7ffbc17ff700 1 --2- 172.21.15.47:0/1633183003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbbc1038e0 0x7ffbbc19d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.072+0000 7ffbc17ff700 1 --2- 172.21.15.47:0/1633183003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffbbc066be0 0x7ffbbc195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.072+0000 7ffbc17ff700 1 --2- 172.21.15.47:0/1633183003 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbbc100df0 0x7ffbbc196080 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.072+0000 7ffbc17ff700 1 -- 172.21.15.47:0/1633183003 >> 172.21.15.47:0/1633183003 conn(0x7ffbbc0f82e0 msgr2=0x7ffbbc0ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:03.081 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.072+0000 7ffbc17ff700 1 -- 172.21.15.47:0/1633183003 shutdown_connections 2022-09-07T21:39:03.081 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.072+0000 7ffbc17ff700 1 -- 172.21.15.47:0/1633183003 wait complete. 2022-09-07T21:39:03.102 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-09-07T21:39:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:02 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/4282763058' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2022-09-07T21:39:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:02 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1819648566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:03.337 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.334+0000 7f36175ff700 1 -- 172.21.15.47:0/1571200462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3610066be0 msgr2=0x7f3610100960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:03.337 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.334+0000 7f36175ff700 1 --2- 172.21.15.47:0/1571200462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3610066be0 0x7f3610100960 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f360c009810 tx=0x7f360c009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:03.338 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.334+0000 7f36175ff700 1 -- 172.21.15.47:0/1571200462 shutdown_connections 2022-09-07T21:39:03.338 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.334+0000 7f36175ff700 1 --2- 172.21.15.47:0/1571200462 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3610103990 0x7f3610105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.338 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.334+0000 7f36175ff700 1 --2- 172.21.15.47:0/1571200462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3610066be0 0x7f3610100960 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.338 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.334+0000 7f36175ff700 1 --2- 172.21.15.47:0/1571200462 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3610100ea0 0x7f3610103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.340 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.334+0000 7f36175ff700 1 -- 172.21.15.47:0/1571200462 >> 172.21.15.47:0/1571200462 conn(0x7f36100f8370 msgr2=0x7f36100fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:03.340 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.334+0000 7f36175ff700 1 -- 172.21.15.47:0/1571200462 shutdown_connections 2022-09-07T21:39:03.340 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.334+0000 7f36175ff700 1 -- 172.21.15.47:0/1571200462 wait complete. 2022-09-07T21:39:03.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.336+0000 7f36175ff700 1 Processor -- start 2022-09-07T21:39:03.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.336+0000 7f36175ff700 1 -- start start 2022-09-07T21:39:03.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.336+0000 7f36175ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3610066be0 0x7f3610195c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:03.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.336+0000 7f36175ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3610100ea0 0x7f3610196160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.336+0000 7f36175ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3610103990 0x7f361019d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.336+0000 7f36175ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361019d7e0 con 0x7f3610103990 2022-09-07T21:39:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.336+0000 7f36175ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361019d950 con 0x7f3610066be0 2022-09-07T21:39:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.336+0000 7f36175ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361019db10 con 0x7f3610100ea0 2022-09-07T21:39:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.336+0000 7f3614b9a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3610100ea0 0x7f3610196160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:03.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.336+0000 7f3615b9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3610103990 0x7f361019d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:03.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.336+0000 7f3614b9a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3610100ea0 0x7f3610196160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:42798/0 (socket says 172.21.15.47:42798) 2022-09-07T21:39:03.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.336+0000 7f361539b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3610066be0 0x7f3610195c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:03.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.336+0000 7f3614b9a700 1 -- 172.21.15.47:0/3745255617 learned_addr learned my addr 172.21.15.47:0/3745255617 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:03.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.337+0000 7f3614b9a700 1 -- 172.21.15.47:0/3745255617 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3610066be0 msgr2=0x7f3610195c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:03.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.337+0000 7f3614b9a700 1 --2- 172.21.15.47:0/3745255617 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3610066be0 0x7f3610195c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.337+0000 7f3614b9a700 1 -- 172.21.15.47:0/3745255617 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3610103990 msgr2=0x7f361019d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:03.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.337+0000 7f3614b9a700 1 --2- 172.21.15.47:0/3745255617 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3610103990 0x7f361019d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.337+0000 7f3614b9a700 1 -- 172.21.15.47:0/3745255617 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c009470 con 0x7f3610100ea0 2022-09-07T21:39:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.337+0000 7f3615b9c700 1 --2- 172.21.15.47:0/3745255617 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3610103990 0x7f361019d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.337+0000 7f3614b9a700 1 --2- 172.21.15.47:0/3745255617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3610100ea0 0x7f3610196160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f360000e740 tx=0x7f360000ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.337+0000 7f36067fc700 1 -- 172.21.15.47:0/3745255617 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3600005220 con 0x7f3610100ea0 2022-09-07T21:39:03.349 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.338+0000 7f36175ff700 1 -- 172.21.15.47:0/3745255617 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361019ddf0 con 0x7f3610100ea0 2022-09-07T21:39:03.349 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.338+0000 7f36067fc700 1 -- 172.21.15.47:0/3745255617 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3600005380 con 0x7f3610100ea0 2022-09-07T21:39:03.349 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.338+0000 7f36067fc700 1 -- 172.21.15.47:0/3745255617 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3600004b70 con 0x7f3610100ea0 2022-09-07T21:39:03.349 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.338+0000 7f36175ff700 1 -- 172.21.15.47:0/3745255617 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f361019e210 con 0x7f3610100ea0 2022-09-07T21:39:03.349 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.339+0000 7f36175ff700 1 -- 172.21.15.47:0/3745255617 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3610109560 con 0x7f3610100ea0 2022-09-07T21:39:03.350 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.341+0000 7f36067fc700 1 -- 172.21.15.47:0/3745255617 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f360001e020 con 0x7f3610100ea0 2022-09-07T21:39:03.351 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.342+0000 7f36067fc700 1 --2- 172.21.15.47:0/3745255617 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f35fc099010 0x7f35fc09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:03.352 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.342+0000 7f36067fc700 1 -- 172.21.15.47:0/3745255617 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f3600014070 con 0x7f3610100ea0 2022-09-07T21:39:03.352 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.342+0000 7f361539b700 1 --2- 172.21.15.47:0/3745255617 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f35fc099010 0x7f35fc09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:03.352 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.343+0000 7f361539b700 1 --2- 172.21.15.47:0/3745255617 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f35fc099010 0x7f35fc09b4c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f360c000c00 tx=0x7f360c009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:03.352 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.344+0000 7f36067fc700 1 -- 172.21.15.47:0/3745255617 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f360008ef40 con 0x7f3610100ea0 2022-09-07T21:39:03.596 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.594+0000 7f36175ff700 1 -- 172.21.15.47:0/3745255617 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3610105420 con 0x7f3610100ea0 2022-09-07T21:39:03.782 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.779+0000 7f36067fc700 1 -- 172.21.15.47:0/3745255617 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3600092560 con 0x7f3610100ea0 2022-09-07T21:39:03.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.782+0000 7f36175ff700 1 -- 172.21.15.47:0/3745255617 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f35fc099010 msgr2=0x7f35fc09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:03.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.782+0000 7f36175ff700 1 --2- 172.21.15.47:0/3745255617 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f35fc099010 0x7f35fc09b4c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f360c000c00 tx=0x7f360c009eb0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.782+0000 7f36175ff700 1 -- 172.21.15.47:0/3745255617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3610100ea0 msgr2=0x7f3610196160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:03.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.782+0000 7f36175ff700 1 --2- 172.21.15.47:0/3745255617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3610100ea0 0x7f3610196160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f360000e740 tx=0x7f360000ea50 comp rx=0 tx=0).stop 2022-09-07T21:39:03.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.782+0000 7f36175ff700 1 -- 172.21.15.47:0/3745255617 shutdown_connections 2022-09-07T21:39:03.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.782+0000 7f36175ff700 1 --2- 172.21.15.47:0/3745255617 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f35fc099010 0x7f35fc09b4c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.782+0000 7f36175ff700 1 --2- 172.21.15.47:0/3745255617 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3610103990 0x7f361019d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.782+0000 7f36175ff700 1 --2- 172.21.15.47:0/3745255617 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3610066be0 0x7f3610195c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.782+0000 7f36175ff700 1 --2- 172.21.15.47:0/3745255617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3610100ea0 0x7f3610196160 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:03.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.782+0000 7f36175ff700 1 -- 172.21.15.47:0/3745255617 >> 172.21.15.47:0/3745255617 conn(0x7f36100f8370 msgr2=0x7f36100ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:03.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.783+0000 7f36175ff700 1 -- 172.21.15.47:0/3745255617 shutdown_connections 2022-09-07T21:39:03.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:03.783+0000 7f36175ff700 1 -- 172.21.15.47:0/3745255617 wait complete. 2022-09-07T21:39:03.812 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2022-09-07T21:39:03.813 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2022-09-07T21:39:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.047+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/2064871843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c54100ea0 msgr2=0x7f6c54103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.047+0000 7f6c5a9ff700 1 --2- 172.21.15.47:0/2064871843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c54100ea0 0x7f6c54103300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6c48009810 tx=0x7f6c48009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.047+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/2064871843 shutdown_connections 2022-09-07T21:39:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.047+0000 7f6c5a9ff700 1 --2- 172.21.15.47:0/2064871843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c54103990 0x7f6c54105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.047+0000 7f6c5a9ff700 1 --2- 172.21.15.47:0/2064871843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c54066be0 0x7f6c54100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.047+0000 7f6c5a9ff700 1 --2- 172.21.15.47:0/2064871843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c54100ea0 0x7f6c54103300 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.047+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/2064871843 >> 172.21.15.47:0/2064871843 conn(0x7f6c540f8370 msgr2=0x7f6c540fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.048+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/2064871843 shutdown_connections 2022-09-07T21:39:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.048+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/2064871843 wait complete. 2022-09-07T21:39:04.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.049+0000 7f6c5a9ff700 1 Processor -- start 2022-09-07T21:39:04.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.049+0000 7f6c5a9ff700 1 -- start start 2022-09-07T21:39:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.049+0000 7f6c5a9ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c54066be0 0x7f6c54195ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.049+0000 7f6c5a9ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c54100ea0 0x7f6c541960e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.049+0000 7f6c5a9ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c54103990 0x7f6c5419d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.049+0000 7f6c5a9ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5419d720 con 0x7f6c54066be0 2022-09-07T21:39:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.049+0000 7f6c5a9ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5419d890 con 0x7f6c54103990 2022-09-07T21:39:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.049+0000 7f6c5a9ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5419da50 con 0x7f6c54100ea0 2022-09-07T21:39:04.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.050+0000 7f6c53fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c54066be0 0x7f6c54195ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:04.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.050+0000 7f6c58f9c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c54103990 0x7f6c5419d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:04.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.050+0000 7f6c58f9c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c54103990 0x7f6c5419d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:58160/0 (socket says 172.21.15.47:58160) 2022-09-07T21:39:04.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.050+0000 7f6c58f9c700 1 -- 172.21.15.47:0/3528299651 learned_addr learned my addr 172.21.15.47:0/3528299651 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:04.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.050+0000 7f6c537fe700 1 --2- 172.21.15.47:0/3528299651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c54100ea0 0x7f6c541960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:04.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.050+0000 7f6c58f9c700 1 -- 172.21.15.47:0/3528299651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c54100ea0 msgr2=0x7f6c541960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:04.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.050+0000 7f6c58f9c700 1 --2- 172.21.15.47:0/3528299651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c54100ea0 0x7f6c541960e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:04.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.050+0000 7f6c58f9c700 1 -- 172.21.15.47:0/3528299651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c54066be0 msgr2=0x7f6c54195ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:04.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.050+0000 7f6c58f9c700 1 --2- 172.21.15.47:0/3528299651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c54066be0 0x7f6c54195ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:04.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.050+0000 7f6c58f9c700 1 -- 172.21.15.47:0/3528299651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c48009470 con 0x7f6c54103990 2022-09-07T21:39:04.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.051+0000 7f6c58f9c700 1 --2- 172.21.15.47:0/3528299651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c54103990 0x7f6c5419d150 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c00ebf0 tx=0x7f6c4c00ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:04.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.051+0000 7f6c517fa700 1 -- 172.21.15.47:0/3528299651 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c0099c0 con 0x7f6c54103990 2022-09-07T21:39:04.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.051+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/3528299651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c5419dd30 con 0x7f6c54103990 2022-09-07T21:39:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.051+0000 7f6c517fa700 1 -- 172.21.15.47:0/3528299651 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6c4c00cc90 con 0x7f6c54103990 2022-09-07T21:39:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.051+0000 7f6c517fa700 1 -- 172.21.15.47:0/3528299651 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c010430 con 0x7f6c54103990 2022-09-07T21:39:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.051+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/3528299651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5419e1e0 con 0x7f6c54103990 2022-09-07T21:39:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.052+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/3528299651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c541095a0 con 0x7f6c54103990 2022-09-07T21:39:04.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.054+0000 7f6c517fa700 1 -- 172.21.15.47:0/3528299651 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f6c4c01e020 con 0x7f6c54103990 2022-09-07T21:39:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.055+0000 7f6c517fa700 1 --2- 172.21.15.47:0/3528299651 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6c3c099010 0x7f6c3c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.056+0000 7f6c517fa700 1 -- 172.21.15.47:0/3528299651 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f6c4c014070 con 0x7f6c54103990 2022-09-07T21:39:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.056+0000 7f6c53fff700 1 --2- 172.21.15.47:0/3528299651 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6c3c099010 0x7f6c3c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.057+0000 7f6c53fff700 1 --2- 172.21.15.47:0/3528299651 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6c3c099010 0x7f6c3c09b4c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c54066fb0 tx=0x7f6c44005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:04.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.058+0000 7f6c517fa700 1 -- 172.21.15.47:0/3528299651 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f6c4c08f730 con 0x7f6c54103990 2022-09-07T21:39:04.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:03 smithi047 ceph-mon[90087]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2022-09-07T21:39:04.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:03 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1633183003' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:04.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:03 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3745255617' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-09-07T21:39:04.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:03 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-09-07T21:39:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:03 smithi055 ceph-mon[97033]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2022-09-07T21:39:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:03 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1633183003' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:04.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:03 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3745255617' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-09-07T21:39:04.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:03 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-09-07T21:39:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:03 smithi059 ceph-mon[97450]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2022-09-07T21:39:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:03 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1633183003' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:03 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3745255617' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-09-07T21:39:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:03 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-09-07T21:39:04.310 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.308+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/3528299651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f6c5419e850 con 0x7f6c54103990 2022-09-07T21:39:04.796 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.794+0000 7f6c517fa700 1 -- 172.21.15.47:0/3528299651 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f6c4c092d50 con 0x7f6c54103990 2022-09-07T21:39:04.800 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.797+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/3528299651 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6c3c099010 msgr2=0x7f6c3c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:04.801 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.797+0000 7f6c5a9ff700 1 --2- 172.21.15.47:0/3528299651 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6c3c099010 0x7f6c3c09b4c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c54066fb0 tx=0x7f6c44005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:04.801 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.797+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/3528299651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c54103990 msgr2=0x7f6c5419d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:04.802 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.797+0000 7f6c5a9ff700 1 --2- 172.21.15.47:0/3528299651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c54103990 0x7f6c5419d150 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c00ebf0 tx=0x7f6c4c00ef00 comp rx=0 tx=0).stop 2022-09-07T21:39:04.802 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.797+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/3528299651 shutdown_connections 2022-09-07T21:39:04.803 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.797+0000 7f6c5a9ff700 1 --2- 172.21.15.47:0/3528299651 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6c3c099010 0x7f6c3c09b4c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:04.803 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.797+0000 7f6c5a9ff700 1 --2- 172.21.15.47:0/3528299651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c54103990 0x7f6c5419d150 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:04.804 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.797+0000 7f6c5a9ff700 1 --2- 172.21.15.47:0/3528299651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c54066be0 0x7f6c54195ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:04.804 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.797+0000 7f6c5a9ff700 1 --2- 172.21.15.47:0/3528299651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c54100ea0 0x7f6c541960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:04.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.797+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/3528299651 >> 172.21.15.47:0/3528299651 conn(0x7f6c540f8370 msgr2=0x7f6c540fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:04.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.797+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/3528299651 shutdown_connections 2022-09-07T21:39:04.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:04.797+0000 7f6c5a9ff700 1 -- 172.21.15.47:0/3528299651 wait complete. 2022-09-07T21:39:04.833 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2022-09-07T21:39:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.066+0000 7f13785ff700 1 -- 172.21.15.47:0/2262675182 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1370100ea0 msgr2=0x7f1370103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.066+0000 7f13785ff700 1 --2- 172.21.15.47:0/2262675182 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1370100ea0 0x7f1370103300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1360009810 tx=0x7f1360009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.066+0000 7f13785ff700 1 -- 172.21.15.47:0/2262675182 shutdown_connections 2022-09-07T21:39:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.066+0000 7f13785ff700 1 --2- 172.21.15.47:0/2262675182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1370103990 0x7f1370105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.066+0000 7f13785ff700 1 --2- 172.21.15.47:0/2262675182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1370066be0 0x7f1370100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.066+0000 7f13785ff700 1 --2- 172.21.15.47:0/2262675182 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1370100ea0 0x7f1370103300 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.066+0000 7f13785ff700 1 -- 172.21.15.47:0/2262675182 >> 172.21.15.47:0/2262675182 conn(0x7f13700f8370 msgr2=0x7f13700fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:05.072 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.066+0000 7f13785ff700 1 -- 172.21.15.47:0/2262675182 shutdown_connections 2022-09-07T21:39:05.072 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.066+0000 7f13785ff700 1 -- 172.21.15.47:0/2262675182 wait complete. 2022-09-07T21:39:05.074 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.067+0000 7f13785ff700 1 Processor -- start 2022-09-07T21:39:05.075 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.067+0000 7f13785ff700 1 -- start start 2022-09-07T21:39:05.075 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.068+0000 7f13785ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1370066be0 0x7f1370195c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:05.075 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.068+0000 7f13785ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1370100ea0 0x7f1370196160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:05.075 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.068+0000 7f13785ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1370103990 0x7f137019d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:05.076 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.068+0000 7f13785ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137019d7e0 con 0x7f1370103990 2022-09-07T21:39:05.076 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.068+0000 7f13785ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137019d950 con 0x7f1370100ea0 2022-09-07T21:39:05.076 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.068+0000 7f13785ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137019db10 con 0x7f1370066be0 2022-09-07T21:39:05.076 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.068+0000 7f1376b9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1370103990 0x7f137019d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.068+0000 7f137639b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1370066be0 0x7f1370195c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.068+0000 7f1376b9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1370103990 0x7f137019d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33566/0 (socket says 172.21.15.47:33566) 2022-09-07T21:39:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.068+0000 7f1376b9c700 1 -- 172.21.15.47:0/1301712041 learned_addr learned my addr 172.21.15.47:0/1301712041 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.068+0000 7f1375b9a700 1 --2- 172.21.15.47:0/1301712041 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1370100ea0 0x7f1370196160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:05.078 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.069+0000 7f1376b9c700 1 -- 172.21.15.47:0/1301712041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1370066be0 msgr2=0x7f1370195c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:05.078 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.069+0000 7f1376b9c700 1 --2- 172.21.15.47:0/1301712041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1370066be0 0x7f1370195c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:05.078 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.069+0000 7f1376b9c700 1 -- 172.21.15.47:0/1301712041 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1370100ea0 msgr2=0x7f1370196160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:05.078 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.069+0000 7f1376b9c700 1 --2- 172.21.15.47:0/1301712041 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1370100ea0 0x7f1370196160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.069+0000 7f1376b9c700 1 -- 172.21.15.47:0/1301712041 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1360009470 con 0x7f1370103990 2022-09-07T21:39:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.069+0000 7f1375b9a700 1 --2- 172.21.15.47:0/1301712041 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1370100ea0 0x7f1370196160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.069+0000 7f1376b9c700 1 --2- 172.21.15.47:0/1301712041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1370103990 0x7f137019d190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f136800ebf0 tx=0x7f136800ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.069+0000 7f13677fe700 1 -- 172.21.15.47:0/1301712041 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13680099c0 con 0x7f1370103990 2022-09-07T21:39:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.069+0000 7f13785ff700 1 -- 172.21.15.47:0/1301712041 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137019ddf0 con 0x7f1370103990 2022-09-07T21:39:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.069+0000 7f13677fe700 1 -- 172.21.15.47:0/1301712041 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f136800cc90 con 0x7f1370103990 2022-09-07T21:39:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.069+0000 7f13677fe700 1 -- 172.21.15.47:0/1301712041 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1368010430 con 0x7f1370103990 2022-09-07T21:39:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.070+0000 7f13785ff700 1 -- 172.21.15.47:0/1301712041 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137019e270 con 0x7f1370103990 2022-09-07T21:39:05.081 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.071+0000 7f13785ff700 1 -- 172.21.15.47:0/1301712041 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13701095a0 con 0x7f1370103990 2022-09-07T21:39:05.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.073+0000 7f13677fe700 1 -- 172.21.15.47:0/1301712041 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f136801e020 con 0x7f1370103990 2022-09-07T21:39:05.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.074+0000 7f13677fe700 1 --2- 172.21.15.47:0/1301712041 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f135c099090 0x7f135c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:05.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.074+0000 7f13677fe700 1 -- 172.21.15.47:0/1301712041 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f1368014070 con 0x7f1370103990 2022-09-07T21:39:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.074+0000 7f137639b700 1 --2- 172.21.15.47:0/1301712041 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f135c099090 0x7f135c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.075+0000 7f137639b700 1 --2- 172.21.15.47:0/1301712041 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f135c099090 0x7f135c09b540 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f13701970f0 tx=0x7f136c005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.078+0000 7f13677fe700 1 -- 172.21.15.47:0/1301712041 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f136808ef90 con 0x7f1370103990 2022-09-07T21:39:05.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:04 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-09-07T21:39:05.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:04 smithi047 ceph-mon[90087]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:39:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:04 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3528299651' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2022-09-07T21:39:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:04 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2022-09-07T21:39:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:04 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-09-07T21:39:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:04 smithi055 ceph-mon[97033]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:39:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:04 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3528299651' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2022-09-07T21:39:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:04 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2022-09-07T21:39:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:04 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-09-07T21:39:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:04 smithi059 ceph-mon[97450]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:39:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:04 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3528299651' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2022-09-07T21:39:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:04 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2022-09-07T21:39:05.329 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.325+0000 7f13785ff700 1 -- 172.21.15.47:0/1301712041 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f137005f9f0 con 0x7f1370103990 2022-09-07T21:39:05.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.806+0000 7f13677fe700 1 -- 172.21.15.47:0/1301712041 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7f13680925b0 con 0x7f1370103990 2022-09-07T21:39:05.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.809+0000 7f13785ff700 1 -- 172.21.15.47:0/1301712041 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f135c099090 msgr2=0x7f135c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:05.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.809+0000 7f13785ff700 1 --2- 172.21.15.47:0/1301712041 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f135c099090 0x7f135c09b540 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f13701970f0 tx=0x7f136c005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:05.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.809+0000 7f13785ff700 1 -- 172.21.15.47:0/1301712041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1370103990 msgr2=0x7f137019d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:05.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.809+0000 7f13785ff700 1 --2- 172.21.15.47:0/1301712041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1370103990 0x7f137019d190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f136800ebf0 tx=0x7f136800ef00 comp rx=0 tx=0).stop 2022-09-07T21:39:05.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.809+0000 7f13785ff700 1 -- 172.21.15.47:0/1301712041 shutdown_connections 2022-09-07T21:39:05.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.809+0000 7f13785ff700 1 --2- 172.21.15.47:0/1301712041 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f135c099090 0x7f135c09b540 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:05.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.809+0000 7f13785ff700 1 --2- 172.21.15.47:0/1301712041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1370103990 0x7f137019d190 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:05.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.809+0000 7f13785ff700 1 --2- 172.21.15.47:0/1301712041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1370066be0 0x7f1370195c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:05.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.809+0000 7f13785ff700 1 --2- 172.21.15.47:0/1301712041 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1370100ea0 0x7f1370196160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:05.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.809+0000 7f13785ff700 1 -- 172.21.15.47:0/1301712041 >> 172.21.15.47:0/1301712041 conn(0x7f13700f8370 msgr2=0x7f13700ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:05.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.809+0000 7f13785ff700 1 -- 172.21.15.47:0/1301712041 shutdown_connections 2022-09-07T21:39:05.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:05.809+0000 7f13785ff700 1 -- 172.21.15.47:0/1301712041 wait complete. 2022-09-07T21:39:05.846 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2022-09-07T21:39:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.079+0000 7fc8b79ff700 1 -- 172.21.15.47:0/203856376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b0100ea0 msgr2=0x7fc8b0103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.079+0000 7fc8b79ff700 1 --2- 172.21.15.47:0/203856376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b0100ea0 0x7fc8b0103300 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009810 tx=0x7fc8a0009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.079+0000 7fc8b79ff700 1 -- 172.21.15.47:0/203856376 shutdown_connections 2022-09-07T21:39:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.079+0000 7fc8b79ff700 1 --2- 172.21.15.47:0/203856376 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b0103990 0x7fc8b0105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.079+0000 7fc8b79ff700 1 --2- 172.21.15.47:0/203856376 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8b0066be0 0x7fc8b0100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:06.085 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.080+0000 7fc8b79ff700 1 --2- 172.21.15.47:0/203856376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b0100ea0 0x7fc8b0103300 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:06.085 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.080+0000 7fc8b79ff700 1 -- 172.21.15.47:0/203856376 >> 172.21.15.47:0/203856376 conn(0x7fc8b00f8370 msgr2=0x7fc8b00fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:06.085 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.080+0000 7fc8b79ff700 1 -- 172.21.15.47:0/203856376 shutdown_connections 2022-09-07T21:39:06.085 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.080+0000 7fc8b79ff700 1 -- 172.21.15.47:0/203856376 wait complete. 2022-09-07T21:39:06.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.081+0000 7fc8b79ff700 1 Processor -- start 2022-09-07T21:39:06.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.081+0000 7fc8b79ff700 1 -- start start 2022-09-07T21:39:06.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.081+0000 7fc8b79ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8b0066be0 0x7fc8b0195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:06.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.081+0000 7fc8b79ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b0100ea0 0x7fc8b0196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:06.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.081+0000 7fc8b79ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b0103990 0x7fc8b019d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:06.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.081+0000 7fc8b79ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b019d750 con 0x7fc8b0100ea0 2022-09-07T21:39:06.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.081+0000 7fc8b79ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b019d8c0 con 0x7fc8b0066be0 2022-09-07T21:39:06.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.081+0000 7fc8b79ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b019da80 con 0x7fc8b0103990 2022-09-07T21:39:06.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.082+0000 7fc8b579b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8b0066be0 0x7fc8b0195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:06.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.082+0000 7fc8b579b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8b0066be0 0x7fc8b0195bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:58198/0 (socket says 172.21.15.47:58198) 2022-09-07T21:39:06.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.082+0000 7fc8b4f9a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b0100ea0 0x7fc8b0196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:06.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.082+0000 7fc8b579b700 1 -- 172.21.15.47:0/290683197 learned_addr learned my addr 172.21.15.47:0/290683197 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:06.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.082+0000 7fc8b5f9c700 1 --2- 172.21.15.47:0/290683197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b0103990 0x7fc8b019d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:06.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.082+0000 7fc8b579b700 1 -- 172.21.15.47:0/290683197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b0103990 msgr2=0x7fc8b019d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:06.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.082+0000 7fc8b579b700 1 --2- 172.21.15.47:0/290683197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b0103990 0x7fc8b019d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:06.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.082+0000 7fc8b579b700 1 -- 172.21.15.47:0/290683197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b0100ea0 msgr2=0x7fc8b0196110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:06.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.082+0000 7fc8b579b700 1 --2- 172.21.15.47:0/290683197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b0100ea0 0x7fc8b0196110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:06.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.082+0000 7fc8b579b700 1 -- 172.21.15.47:0/290683197 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a0009470 con 0x7fc8b0066be0 2022-09-07T21:39:06.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.083+0000 7fc8b579b700 1 --2- 172.21.15.47:0/290683197 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8b0066be0 0x7fc8b0195bd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00e6e0 tx=0x7fc8ac00e9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:06.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.083+0000 7fc8a67fc700 1 -- 172.21.15.47:0/290683197 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac009500 con 0x7fc8b0066be0 2022-09-07T21:39:06.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.083+0000 7fc8a67fc700 1 -- 172.21.15.47:0/290683197 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fc8ac010da0 con 0x7fc8b0066be0 2022-09-07T21:39:06.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.083+0000 7fc8a67fc700 1 -- 172.21.15.47:0/290683197 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac0173d0 con 0x7fc8b0066be0 2022-09-07T21:39:06.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.083+0000 7fc8b79ff700 1 -- 172.21.15.47:0/290683197 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b019dd60 con 0x7fc8b0066be0 2022-09-07T21:39:06.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.083+0000 7fc8b79ff700 1 -- 172.21.15.47:0/290683197 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b019e1c0 con 0x7fc8b0066be0 2022-09-07T21:39:06.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.084+0000 7fc8b79ff700 1 -- 172.21.15.47:0/290683197 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b01095a0 con 0x7fc8b0066be0 2022-09-07T21:39:06.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.087+0000 7fc8a67fc700 1 -- 172.21.15.47:0/290683197 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fc8ac00c040 con 0x7fc8b0066be0 2022-09-07T21:39:06.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.088+0000 7fc8a67fc700 1 --2- 172.21.15.47:0/290683197 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc89c099090 0x7fc89c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:06.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.088+0000 7fc8a67fc700 1 -- 172.21.15.47:0/290683197 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fc8ac014070 con 0x7fc8b0066be0 2022-09-07T21:39:06.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.088+0000 7fc8b4f9a700 1 --2- 172.21.15.47:0/290683197 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc89c099090 0x7fc89c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:06.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.089+0000 7fc8b4f9a700 1 --2- 172.21.15.47:0/290683197 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc89c099090 0x7fc89c09b540 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc8a000b4f0 tx=0x7fc8a0009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:06.097 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.090+0000 7fc8a67fc700 1 -- 172.21.15.47:0/290683197 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fc8ac08eff0 con 0x7fc8b0066be0 2022-09-07T21:39:06.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:05 smithi047 ceph-mon[90087]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2022-09-07T21:39:06.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:05 smithi047 ceph-mon[90087]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2022-09-07T21:39:06.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:05 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2022-09-07T21:39:06.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:05 smithi047 ceph-mon[90087]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2022-09-07T21:39:06.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:05 smithi047 ceph-mon[90087]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:39:06.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:05 smithi047 ceph-mon[90087]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2022-09-07T21:39:06.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:05 smithi047 ceph-mon[90087]: daemon mds.b is now active in filesystem cephfs as rank 1 2022-09-07T21:39:06.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:05 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1301712041' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-09-07T21:39:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:05 smithi055 ceph-mon[97033]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2022-09-07T21:39:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:05 smithi055 ceph-mon[97033]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2022-09-07T21:39:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:05 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2022-09-07T21:39:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:05 smithi055 ceph-mon[97033]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2022-09-07T21:39:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:05 smithi055 ceph-mon[97033]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:39:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:05 smithi055 ceph-mon[97033]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2022-09-07T21:39:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:05 smithi055 ceph-mon[97033]: daemon mds.b is now active in filesystem cephfs as rank 1 2022-09-07T21:39:06.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:05 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1301712041' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-09-07T21:39:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:05 smithi059 ceph-mon[97450]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2022-09-07T21:39:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:05 smithi059 ceph-mon[97450]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2022-09-07T21:39:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:05 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2022-09-07T21:39:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:05 smithi059 ceph-mon[97450]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2022-09-07T21:39:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:05 smithi059 ceph-mon[97450]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-07T21:39:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:05 smithi059 ceph-mon[97450]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2022-09-07T21:39:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:05 smithi059 ceph-mon[97450]: daemon mds.b is now active in filesystem cephfs as rank 1 2022-09-07T21:39:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:05 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1301712041' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-09-07T21:39:06.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.338+0000 7fc8b79ff700 1 -- 172.21.15.47:0/290683197 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fc8b0105450 con 0x7fc8b0066be0 2022-09-07T21:39:06.836 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.832+0000 7fc8a67fc700 1 -- 172.21.15.47:0/290683197 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7fc8ac0260b0 con 0x7fc8b0066be0 2022-09-07T21:39:06.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.836+0000 7fc8b79ff700 1 -- 172.21.15.47:0/290683197 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc89c099090 msgr2=0x7fc89c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:06.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.836+0000 7fc8b79ff700 1 --2- 172.21.15.47:0/290683197 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc89c099090 0x7fc89c09b540 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc8a000b4f0 tx=0x7fc8a0009eb0 comp rx=0 tx=0).stop 2022-09-07T21:39:06.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.836+0000 7fc8b79ff700 1 -- 172.21.15.47:0/290683197 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8b0066be0 msgr2=0x7fc8b0195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:06.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.836+0000 7fc8b79ff700 1 --2- 172.21.15.47:0/290683197 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8b0066be0 0x7fc8b0195bd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00e6e0 tx=0x7fc8ac00e9f0 comp rx=0 tx=0).stop 2022-09-07T21:39:06.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.836+0000 7fc8b79ff700 1 -- 172.21.15.47:0/290683197 shutdown_connections 2022-09-07T21:39:06.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.836+0000 7fc8b79ff700 1 --2- 172.21.15.47:0/290683197 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc89c099090 0x7fc89c09b540 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:06.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.836+0000 7fc8b79ff700 1 --2- 172.21.15.47:0/290683197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b0103990 0x7fc8b019d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:06.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.836+0000 7fc8b79ff700 1 --2- 172.21.15.47:0/290683197 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8b0066be0 0x7fc8b0195bd0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:06.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.836+0000 7fc8b79ff700 1 --2- 172.21.15.47:0/290683197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b0100ea0 0x7fc8b0196110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:06.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.836+0000 7fc8b79ff700 1 -- 172.21.15.47:0/290683197 >> 172.21.15.47:0/290683197 conn(0x7fc8b00f8370 msgr2=0x7fc8b00ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:06.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.837+0000 7fc8b79ff700 1 -- 172.21.15.47:0/290683197 shutdown_connections 2022-09-07T21:39:06.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:06.837+0000 7fc8b79ff700 1 -- 172.21.15.47:0/290683197 wait complete. 2022-09-07T21:39:06.872 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2022-09-07T21:39:06.873 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2022-09-07T21:39:07.113 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.109+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/2037060018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bb80fddb0 msgr2=0x7f3bb80fe1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:07.114 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.109+0000 7f3bbe9ff700 1 --2- 172.21.15.47:0/2037060018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bb80fddb0 0x7f3bb80fe1c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8009810 tx=0x7f3ba8009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:07.114 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.109+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/2037060018 shutdown_connections 2022-09-07T21:39:07.114 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.109+0000 7f3bbe9ff700 1 --2- 172.21.15.47:0/2037060018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb80ff940 0x7f3bb8106270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.115 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.109+0000 7f3bbe9ff700 1 --2- 172.21.15.47:0/2037060018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bb80fddb0 0x7f3bb80fe1c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.115 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.109+0000 7f3bbe9ff700 1 --2- 172.21.15.47:0/2037060018 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bb80fef90 0x7f3bb80ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.116 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.109+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/2037060018 >> 172.21.15.47:0/2037060018 conn(0x7f3bb80f9460 msgr2=0x7f3bb80fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:07.116 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.110+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/2037060018 shutdown_connections 2022-09-07T21:39:07.116 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.110+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/2037060018 wait complete. 2022-09-07T21:39:07.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.111+0000 7f3bbe9ff700 1 Processor -- start 2022-09-07T21:39:07.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.112+0000 7f3bbe9ff700 1 -- start start 2022-09-07T21:39:07.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.112+0000 7f3bbe9ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bb80fddb0 0x7f3bb81939e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:07.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.112+0000 7f3bbe9ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb80fef90 0x7f3bb8193f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:07.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.112+0000 7f3bbe9ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bb80ff940 0x7f3bb819af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:07.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.112+0000 7f3bbe9ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb819b560 con 0x7f3bb80fddb0 2022-09-07T21:39:07.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.112+0000 7f3bbe9ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb819b6d0 con 0x7f3bb80ff940 2022-09-07T21:39:07.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.112+0000 7f3bbe9ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb819b890 con 0x7f3bb80fef90 2022-09-07T21:39:07.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.112+0000 7f3bb77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb80fef90 0x7f3bb8193f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:07.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.112+0000 7f3bb7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bb80fddb0 0x7f3bb81939e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:07.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.112+0000 7f3bb77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb80fef90 0x7f3bb8193f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:42886/0 (socket says 172.21.15.47:42886) 2022-09-07T21:39:07.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.112+0000 7f3bb77fe700 1 -- 172.21.15.47:0/3694315139 learned_addr learned my addr 172.21.15.47:0/3694315139 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:07.123 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.113+0000 7f3bbcf9c700 1 --2- 172.21.15.47:0/3694315139 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bb80ff940 0x7f3bb819af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:07.123 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.113+0000 7f3bb77fe700 1 -- 172.21.15.47:0/3694315139 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bb80ff940 msgr2=0x7f3bb819af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:07.123 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.113+0000 7f3bb77fe700 1 --2- 172.21.15.47:0/3694315139 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bb80ff940 0x7f3bb819af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.123 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.113+0000 7f3bb77fe700 1 -- 172.21.15.47:0/3694315139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bb80fddb0 msgr2=0x7f3bb81939e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:07.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.113+0000 7f3bb77fe700 1 --2- 172.21.15.47:0/3694315139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bb80fddb0 0x7f3bb81939e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.113+0000 7f3bb77fe700 1 -- 172.21.15.47:0/3694315139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba8009470 con 0x7f3bb80fef90 2022-09-07T21:39:07.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.113+0000 7f3bb7fff700 1 --2- 172.21.15.47:0/3694315139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bb80fddb0 0x7f3bb81939e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:07.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.113+0000 7f3bb77fe700 1 --2- 172.21.15.47:0/3694315139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb80fef90 0x7f3bb8193f20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3bac00e740 tx=0x7f3bac00ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:07.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.114+0000 7f3bb57fa700 1 -- 172.21.15.47:0/3694315139 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac005220 con 0x7f3bb80fef90 2022-09-07T21:39:07.125 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.114+0000 7f3bb57fa700 1 -- 172.21.15.47:0/3694315139 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3bac005380 con 0x7f3bb80fef90 2022-09-07T21:39:07.125 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.114+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/3694315139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb819bb70 con 0x7f3bb80fef90 2022-09-07T21:39:07.125 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.114+0000 7f3bb57fa700 1 -- 172.21.15.47:0/3694315139 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac004b70 con 0x7f3bb80fef90 2022-09-07T21:39:07.126 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.114+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/3694315139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb819c0d0 con 0x7f3bb80fef90 2022-09-07T21:39:07.126 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.115+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/3694315139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb8109560 con 0x7f3bb80fef90 2022-09-07T21:39:07.127 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.117+0000 7f3bb57fa700 1 -- 172.21.15.47:0/3694315139 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f3bac01e020 con 0x7f3bb80fef90 2022-09-07T21:39:07.127 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.118+0000 7f3bb57fa700 1 --2- 172.21.15.47:0/3694315139 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3ba0099090 0x7f3ba009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:07.128 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.118+0000 7f3bb57fa700 1 -- 172.21.15.47:0/3694315139 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f3bac014070 con 0x7f3bb80fef90 2022-09-07T21:39:07.128 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.118+0000 7f3bb7fff700 1 --2- 172.21.15.47:0/3694315139 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3ba0099090 0x7f3ba009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:07.128 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.119+0000 7f3bb7fff700 1 --2- 172.21.15.47:0/3694315139 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3ba0099090 0x7f3ba009b540 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3ba800f4d0 tx=0x7f3ba8009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:07.128 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.120+0000 7f3bb57fa700 1 -- 172.21.15.47:0/3694315139 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f3bac08eeb0 con 0x7f3bb80fef90 2022-09-07T21:39:07.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:06 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] up:active 2022-09-07T21:39:07.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:06 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1301712041' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-09-07T21:39:07.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:06 smithi047 ceph-mon[90087]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2022-09-07T21:39:07.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:06 smithi047 ceph-mon[90087]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2022-09-07T21:39:07.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:06 smithi047 ceph-mon[90087]: Cluster is now healthy 2022-09-07T21:39:07.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:06 smithi047 ceph-mon[90087]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2022-09-07T21:39:07.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:06 smithi047 ceph-mon[90087]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2022-09-07T21:39:07.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:06 smithi047 ceph-mon[90087]: daemon mds.i is now active in filesystem cephfs as rank 2 2022-09-07T21:39:07.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:06 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/290683197' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-09-07T21:39:07.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:06 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-09-07T21:39:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:06 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] up:active 2022-09-07T21:39:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:06 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1301712041' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-09-07T21:39:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:06 smithi055 ceph-mon[97033]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2022-09-07T21:39:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:06 smithi055 ceph-mon[97033]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2022-09-07T21:39:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:06 smithi055 ceph-mon[97033]: Cluster is now healthy 2022-09-07T21:39:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:06 smithi055 ceph-mon[97033]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2022-09-07T21:39:07.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:06 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2022-09-07T21:39:07.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:06 smithi055 ceph-mon[97033]: daemon mds.i is now active in filesystem cephfs as rank 2 2022-09-07T21:39:07.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:06 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/290683197' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-09-07T21:39:07.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:06 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-09-07T21:39:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:06 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] up:active 2022-09-07T21:39:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:06 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1301712041' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-09-07T21:39:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:06 smithi059 ceph-mon[97450]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2022-09-07T21:39:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:06 smithi059 ceph-mon[97450]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2022-09-07T21:39:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:06 smithi059 ceph-mon[97450]: Cluster is now healthy 2022-09-07T21:39:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:06 smithi059 ceph-mon[97450]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2022-09-07T21:39:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:06 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2022-09-07T21:39:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:06 smithi059 ceph-mon[97450]: daemon mds.i is now active in filesystem cephfs as rank 2 2022-09-07T21:39:07.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:06 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/290683197' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-09-07T21:39:07.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:06 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-09-07T21:39:07.364 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.360+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/3694315139 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bb805fae0 con 0x7f3ba0099090 2022-09-07T21:39:07.414 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.410+0000 7f3bb57fa700 1 -- 172.21.15.47:0/3694315139 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3bb805fae0 con 0x7f3ba0099090 2022-09-07T21:39:07.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.413+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/3694315139 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3ba0099090 msgr2=0x7f3ba009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:07.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.413+0000 7f3bbe9ff700 1 --2- 172.21.15.47:0/3694315139 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3ba0099090 0x7f3ba009b540 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3ba800f4d0 tx=0x7f3ba8009eb0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.413+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/3694315139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb80fef90 msgr2=0x7f3bb8193f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:07.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.413+0000 7f3bbe9ff700 1 --2- 172.21.15.47:0/3694315139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb80fef90 0x7f3bb8193f20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3bac00e740 tx=0x7f3bac00ea50 comp rx=0 tx=0).stop 2022-09-07T21:39:07.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.413+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/3694315139 shutdown_connections 2022-09-07T21:39:07.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.413+0000 7f3bbe9ff700 1 --2- 172.21.15.47:0/3694315139 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3ba0099090 0x7f3ba009b540 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.413+0000 7f3bbe9ff700 1 --2- 172.21.15.47:0/3694315139 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bb80ff940 0x7f3bb819af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.413+0000 7f3bbe9ff700 1 --2- 172.21.15.47:0/3694315139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bb80fddb0 0x7f3bb81939e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.419 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.413+0000 7f3bbe9ff700 1 --2- 172.21.15.47:0/3694315139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb80fef90 0x7f3bb8193f20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.419 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.413+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/3694315139 >> 172.21.15.47:0/3694315139 conn(0x7f3bb80f9460 msgr2=0x7f3bb8102500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:07.419 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.413+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/3694315139 shutdown_connections 2022-09-07T21:39:07.419 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.413+0000 7f3bbe9ff700 1 -- 172.21.15.47:0/3694315139 wait complete. 2022-09-07T21:39:07.444 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2022-09-07T21:39:07.653 INFO:journalctl@ceph.mds.a.smithi047.stdout:Sep 07 21:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-a[125500]: 2022-09-07T21:39:07.576+0000 7fdbbb178700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2022-09-07T21:39:07.654 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:39:07.367+0000 7fbd576df700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:39:07.654 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:39:07.367+0000 7fbd576df700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:39:07.654 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:39:07.367+0000 7fbd576df700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:39:07.655 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:39:07.367+0000 7fbd576df700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:39:07.655 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:39:07.367+0000 7fbd576df700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:39:07.683 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.679+0000 7f8fe27ff700 1 -- 172.21.15.47:0/4201549657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fdc103990 msgr2=0x7f8fdc105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:07.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.679+0000 7f8fe27ff700 1 --2- 172.21.15.47:0/4201549657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fdc103990 0x7f8fdc105e10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009bb0 tx=0x7f8fd4009ec0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.679+0000 7f8fe27ff700 1 -- 172.21.15.47:0/4201549657 shutdown_connections 2022-09-07T21:39:07.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.679+0000 7f8fe27ff700 1 --2- 172.21.15.47:0/4201549657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fdc103990 0x7f8fdc105e10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.679+0000 7f8fe27ff700 1 --2- 172.21.15.47:0/4201549657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fdc066be0 0x7f8fdc100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.679+0000 7f8fe27ff700 1 --2- 172.21.15.47:0/4201549657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fdc100ea0 0x7f8fdc103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.679+0000 7f8fe27ff700 1 -- 172.21.15.47:0/4201549657 >> 172.21.15.47:0/4201549657 conn(0x7f8fdc0f8370 msgr2=0x7f8fdc0fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:07.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.680+0000 7f8fe27ff700 1 -- 172.21.15.47:0/4201549657 shutdown_connections 2022-09-07T21:39:07.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.680+0000 7f8fe27ff700 1 -- 172.21.15.47:0/4201549657 wait complete. 2022-09-07T21:39:07.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.681+0000 7f8fe27ff700 1 Processor -- start 2022-09-07T21:39:07.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.681+0000 7f8fe27ff700 1 -- start start 2022-09-07T21:39:07.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.681+0000 7f8fe27ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fdc066be0 0x7f8fdc195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:07.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.681+0000 7f8fe27ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fdc100ea0 0x7f8fdc196100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:07.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.681+0000 7f8fe27ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fdc103990 0x7f8fdc19d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:07.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.681+0000 7f8fe27ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc19d740 con 0x7f8fdc100ea0 2022-09-07T21:39:07.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.681+0000 7f8fe27ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc19d8b0 con 0x7f8fdc103990 2022-09-07T21:39:07.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.681+0000 7f8fe27ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc19da70 con 0x7f8fdc066be0 2022-09-07T21:39:07.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.682+0000 7f8fdb7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fdc100ea0 0x7f8fdc196100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:07.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.682+0000 7f8fe0d9c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fdc103990 0x7f8fdc19d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:07.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.682+0000 7f8fdb7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fdc100ea0 0x7f8fdc196100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33622/0 (socket says 172.21.15.47:33622) 2022-09-07T21:39:07.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.682+0000 7f8fdb7fe700 1 -- 172.21.15.47:0/3175845301 learned_addr learned my addr 172.21.15.47:0/3175845301 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:07.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.682+0000 7f8fe0d9c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fdc103990 0x7f8fdc19d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:58254/0 (socket says 172.21.15.47:58254) 2022-09-07T21:39:07.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.682+0000 7f8fdbfff700 1 --2- 172.21.15.47:0/3175845301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fdc066be0 0x7f8fdc195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:07.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.682+0000 7f8fe0d9c700 1 -- 172.21.15.47:0/3175845301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fdc066be0 msgr2=0x7f8fdc195bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:07.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.682+0000 7f8fe0d9c700 1 --2- 172.21.15.47:0/3175845301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fdc066be0 0x7f8fdc195bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.682+0000 7f8fe0d9c700 1 -- 172.21.15.47:0/3175845301 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fdc100ea0 msgr2=0x7f8fdc196100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:07.696 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.682+0000 7f8fe0d9c700 1 --2- 172.21.15.47:0/3175845301 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fdc100ea0 0x7f8fdc196100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.696 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.682+0000 7f8fe0d9c700 1 -- 172.21.15.47:0/3175845301 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40097e0 con 0x7f8fdc103990 2022-09-07T21:39:07.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.682+0000 7f8fdb7fe700 1 --2- 172.21.15.47:0/3175845301 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fdc100ea0 0x7f8fdc196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:07.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.683+0000 7f8fe0d9c700 1 --2- 172.21.15.47:0/3175845301 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fdc103990 0x7f8fdc19d170 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009fd0 tx=0x7f8fd400b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:07.698 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.683+0000 7f8fd97fa700 1 -- 172.21.15.47:0/3175845301 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd400f870 con 0x7f8fdc103990 2022-09-07T21:39:07.698 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.683+0000 7f8fe27ff700 1 -- 172.21.15.47:0/3175845301 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fdc19dcf0 con 0x7f8fdc103990 2022-09-07T21:39:07.699 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.683+0000 7f8fd97fa700 1 -- 172.21.15.47:0/3175845301 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8fd400ba80 con 0x7f8fdc103990 2022-09-07T21:39:07.700 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.683+0000 7f8fd97fa700 1 -- 172.21.15.47:0/3175845301 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4017470 con 0x7f8fdc103990 2022-09-07T21:39:07.700 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.683+0000 7f8fe27ff700 1 -- 172.21.15.47:0/3175845301 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fdc19e1a0 con 0x7f8fdc103990 2022-09-07T21:39:07.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.684+0000 7f8fe27ff700 1 -- 172.21.15.47:0/3175845301 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fdc1095a0 con 0x7f8fdc103990 2022-09-07T21:39:07.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.686+0000 7f8fd97fa700 1 -- 172.21.15.47:0/3175845301 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f8fd4020070 con 0x7f8fdc103990 2022-09-07T21:39:07.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.687+0000 7f8fd97fa700 1 --2- 172.21.15.47:0/3175845301 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8fc4099090 0x7f8fc409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:07.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.688+0000 7f8fd97fa700 1 -- 172.21.15.47:0/3175845301 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f8fd40c3e50 con 0x7f8fdc103990 2022-09-07T21:39:07.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.688+0000 7f8fdbfff700 1 --2- 172.21.15.47:0/3175845301 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8fc4099090 0x7f8fc409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:07.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.688+0000 7f8fdbfff700 1 --2- 172.21.15.47:0/3175845301 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8fc4099090 0x7f8fc409b540 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc066fb0 tx=0x7f8fcc005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:07.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.690+0000 7f8fd97fa700 1 -- 172.21.15.47:0/3175845301 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f8fd405cc80 con 0x7f8fdc103990 2022-09-07T21:39:07.936 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.932+0000 7f8fe27ff700 1 -- 172.21.15.47:0/3175845301 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fdc05f120 con 0x7f8fc4099090 2022-09-07T21:39:07.939 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.936+0000 7f8fd97fa700 1 -- 172.21.15.47:0/3175845301 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f8fdc05f120 con 0x7f8fc4099090 2022-09-07T21:39:07.940 INFO:teuthology.orchestra.run.smithi047.stdout:/volumes/_nogroup/sv_0/54bb6cf3-11e9-47a1-9245-f6ade7730d94 2022-09-07T21:39:07.943 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.939+0000 7f8fe27ff700 1 -- 172.21.15.47:0/3175845301 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8fc4099090 msgr2=0x7f8fc409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:07.943 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.939+0000 7f8fe27ff700 1 --2- 172.21.15.47:0/3175845301 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8fc4099090 0x7f8fc409b540 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc066fb0 tx=0x7f8fcc005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.944 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.939+0000 7f8fe27ff700 1 -- 172.21.15.47:0/3175845301 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fdc103990 msgr2=0x7f8fdc19d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:07.944 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.939+0000 7f8fe27ff700 1 --2- 172.21.15.47:0/3175845301 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fdc103990 0x7f8fdc19d170 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009fd0 tx=0x7f8fd400b890 comp rx=0 tx=0).stop 2022-09-07T21:39:07.944 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.939+0000 7f8fe27ff700 1 -- 172.21.15.47:0/3175845301 shutdown_connections 2022-09-07T21:39:07.944 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.939+0000 7f8fe27ff700 1 --2- 172.21.15.47:0/3175845301 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8fc4099090 0x7f8fc409b540 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.945 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.939+0000 7f8fe27ff700 1 --2- 172.21.15.47:0/3175845301 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fdc103990 0x7f8fdc19d170 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.945 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.939+0000 7f8fe27ff700 1 --2- 172.21.15.47:0/3175845301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fdc066be0 0x7f8fdc195bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.945 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.939+0000 7f8fe27ff700 1 --2- 172.21.15.47:0/3175845301 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fdc100ea0 0x7f8fdc196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:07.945 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.939+0000 7f8fe27ff700 1 -- 172.21.15.47:0/3175845301 >> 172.21.15.47:0/3175845301 conn(0x7f8fdc0f8370 msgr2=0x7f8fdc0fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:07.945 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.940+0000 7f8fe27ff700 1 -- 172.21.15.47:0/3175845301 shutdown_connections 2022-09-07T21:39:07.946 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:07.940+0000 7f8fe27ff700 1 -- 172.21.15.47:0/3175845301 wait complete. 2022-09-07T21:39:07.970 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2022-09-07T21:39:08.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:07 smithi047 ceph-mon[90087]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s wr, 1 op/s 2022-09-07T21:39:08.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:07 smithi047 ceph-mon[90087]: mds.? [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] up:active 2022-09-07T21:39:08.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:07 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-09-07T21:39:08.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:07 smithi047 ceph-mon[90087]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2022-09-07T21:39:08.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:07 smithi047 ceph-mon[90087]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:39:08.207 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.203+0000 7fde36b9d700 1 -- 172.21.15.47:0/1761243812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde30072bf0 msgr2=0x7fde30071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:08.207 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.203+0000 7fde36b9d700 1 --2- 172.21.15.47:0/1761243812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde30072bf0 0x7fde30071170 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009810 tx=0x7fde2c009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:08.207 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.203+0000 7fde36b9d700 1 -- 172.21.15.47:0/1761243812 shutdown_connections 2022-09-07T21:39:08.208 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.203+0000 7fde36b9d700 1 --2- 172.21.15.47:0/1761243812 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde30105a20 0x7fde30107e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.208 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.203+0000 7fde36b9d700 1 --2- 172.21.15.47:0/1761243812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde30072bf0 0x7fde30071170 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.208 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.203+0000 7fde36b9d700 1 --2- 172.21.15.47:0/1761243812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde300716b0 0x7fde30071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.208 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.203+0000 7fde36b9d700 1 -- 172.21.15.47:0/1761243812 >> 172.21.15.47:0/1761243812 conn(0x7fde300f91c0 msgr2=0x7fde300fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:08.209 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.203+0000 7fde36b9d700 1 -- 172.21.15.47:0/1761243812 shutdown_connections 2022-09-07T21:39:08.209 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.203+0000 7fde36b9d700 1 -- 172.21.15.47:0/1761243812 wait complete. 2022-09-07T21:39:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.204+0000 7fde36b9d700 1 Processor -- start 2022-09-07T21:39:08.212 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.205+0000 7fde36b9d700 1 -- start start 2022-09-07T21:39:08.212 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.205+0000 7fde36b9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde300716b0 0x7fde30195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:08.212 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.205+0000 7fde36b9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde30072bf0 0x7fde30195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:08.212 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.205+0000 7fde36b9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde30105a20 0x7fde3019cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:08.213 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.205+0000 7fde36b9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3019d4d0 con 0x7fde30105a20 2022-09-07T21:39:08.213 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.205+0000 7fde36b9d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3019d640 con 0x7fde300716b0 2022-09-07T21:39:08.213 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.205+0000 7fde36b9d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3019d800 con 0x7fde30072bf0 2022-09-07T21:39:08.213 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.205+0000 7fde35b9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde300716b0 0x7fde30195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.205+0000 7fde35b9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde300716b0 0x7fde30195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:58268/0 (socket says 172.21.15.47:58268) 2022-09-07T21:39:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.205+0000 7fde35b9b700 1 -- 172.21.15.47:0/3786985554 learned_addr learned my addr 172.21.15.47:0/3786985554 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.205+0000 7fde3639c700 1 --2- 172.21.15.47:0/3786985554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde30105a20 0x7fde3019cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.205+0000 7fde3539a700 1 --2- 172.21.15.47:0/3786985554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde30072bf0 0x7fde30195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.206+0000 7fde35b9b700 1 -- 172.21.15.47:0/3786985554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde30072bf0 msgr2=0x7fde30195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.206+0000 7fde35b9b700 1 --2- 172.21.15.47:0/3786985554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde30072bf0 0x7fde30195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.206+0000 7fde35b9b700 1 -- 172.21.15.47:0/3786985554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde30105a20 msgr2=0x7fde3019cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.206+0000 7fde35b9b700 1 --2- 172.21.15.47:0/3786985554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde30105a20 0x7fde3019cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.206+0000 7fde35b9b700 1 -- 172.21.15.47:0/3786985554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c009470 con 0x7fde300716b0 2022-09-07T21:39:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.206+0000 7fde35b9b700 1 --2- 172.21.15.47:0/3786985554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde300716b0 0x7fde30195950 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fde2c006010 tx=0x7fde2c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.206+0000 7fde26ffd700 1 -- 172.21.15.47:0/3786985554 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c01d070 con 0x7fde300716b0 2022-09-07T21:39:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.206+0000 7fde26ffd700 1 -- 172.21.15.47:0/3786985554 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fde2c00f8b0 con 0x7fde300716b0 2022-09-07T21:39:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.206+0000 7fde36b9d700 1 -- 172.21.15.47:0/3786985554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde3019da80 con 0x7fde300716b0 2022-09-07T21:39:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.206+0000 7fde26ffd700 1 -- 172.21.15.47:0/3786985554 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c0047f0 con 0x7fde300716b0 2022-09-07T21:39:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.207+0000 7fde36b9d700 1 -- 172.21.15.47:0/3786985554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde3019df30 con 0x7fde300716b0 2022-09-07T21:39:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.208+0000 7fde36b9d700 1 -- 172.21.15.47:0/3786985554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde3010b4d0 con 0x7fde300716b0 2022-09-07T21:39:08.218 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.209+0000 7fde26ffd700 1 -- 172.21.15.47:0/3786985554 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fde2c028030 con 0x7fde300716b0 2022-09-07T21:39:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.211+0000 7fde26ffd700 1 --2- 172.21.15.47:0/3786985554 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fde1c099010 0x7fde1c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.211+0000 7fde26ffd700 1 -- 172.21.15.47:0/3786985554 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fde2c0c3250 con 0x7fde300716b0 2022-09-07T21:39:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.211+0000 7fde3539a700 1 --2- 172.21.15.47:0/3786985554 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fde1c099010 0x7fde1c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.212+0000 7fde3539a700 1 --2- 172.21.15.47:0/3786985554 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fde1c099010 0x7fde1c09b4c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fde20005fd0 tx=0x7fde20005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.215+0000 7fde26ffd700 1 -- 172.21.15.47:0/3786985554 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fde2c08f6f0 con 0x7fde300716b0 2022-09-07T21:39:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:07 smithi055 ceph-mon[97033]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s wr, 1 op/s 2022-09-07T21:39:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:07 smithi055 ceph-mon[97033]: mds.? [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] up:active 2022-09-07T21:39:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:07 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-09-07T21:39:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:07 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2022-09-07T21:39:08.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:07 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:39:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:07 smithi059 ceph-mon[97450]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s wr, 1 op/s 2022-09-07T21:39:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:07 smithi059 ceph-mon[97450]: mds.? [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] up:active 2022-09-07T21:39:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:07 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-09-07T21:39:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:07 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2022-09-07T21:39:08.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:07 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:39:08.454 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.451+0000 7fde36b9d700 1 -- 172.21.15.47:0/3786985554 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7fde3005fca0 con 0x7fde1c099010 2022-09-07T21:39:08.478 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.475+0000 7fde26ffd700 1 -- 172.21.15.47:0/3786985554 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fde3005fca0 con 0x7fde1c099010 2022-09-07T21:39:08.481 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.478+0000 7fde36b9d700 1 -- 172.21.15.47:0/3786985554 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fde1c099010 msgr2=0x7fde1c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:08.482 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.478+0000 7fde36b9d700 1 --2- 172.21.15.47:0/3786985554 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fde1c099010 0x7fde1c09b4c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fde20005fd0 tx=0x7fde20005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.482 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.478+0000 7fde36b9d700 1 -- 172.21.15.47:0/3786985554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde300716b0 msgr2=0x7fde30195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:08.482 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.478+0000 7fde36b9d700 1 --2- 172.21.15.47:0/3786985554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde300716b0 0x7fde30195950 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fde2c006010 tx=0x7fde2c00f690 comp rx=0 tx=0).stop 2022-09-07T21:39:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.478+0000 7fde36b9d700 1 -- 172.21.15.47:0/3786985554 shutdown_connections 2022-09-07T21:39:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.478+0000 7fde36b9d700 1 --2- 172.21.15.47:0/3786985554 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fde1c099010 0x7fde1c09b4c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.478+0000 7fde36b9d700 1 --2- 172.21.15.47:0/3786985554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde30105a20 0x7fde3019cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.478+0000 7fde36b9d700 1 --2- 172.21.15.47:0/3786985554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde300716b0 0x7fde30195950 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.478+0000 7fde36b9d700 1 --2- 172.21.15.47:0/3786985554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde30072bf0 0x7fde30195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.484 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.478+0000 7fde36b9d700 1 -- 172.21.15.47:0/3786985554 >> 172.21.15.47:0/3786985554 conn(0x7fde300f91c0 msgr2=0x7fde300fb5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:08.484 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.478+0000 7fde36b9d700 1 -- 172.21.15.47:0/3786985554 shutdown_connections 2022-09-07T21:39:08.484 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.478+0000 7fde36b9d700 1 -- 172.21.15.47:0/3786985554 wait complete. 2022-09-07T21:39:08.508 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2022-09-07T21:39:08.747 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.743+0000 7f2055dff700 1 -- 172.21.15.47:0/1436494174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2050071740 msgr2=0x7f2050071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:08.747 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.743+0000 7f2055dff700 1 --2- 172.21.15.47:0/1436494174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2050071740 0x7f2050071bb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2044009810 tx=0x7f2044009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:08.748 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.743+0000 7f2055dff700 1 -- 172.21.15.47:0/1436494174 shutdown_connections 2022-09-07T21:39:08.748 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.743+0000 7f2055dff700 1 --2- 172.21.15.47:0/1436494174 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f205010a410 0x7f205010c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.748 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.743+0000 7f2055dff700 1 --2- 172.21.15.47:0/1436494174 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2050072c90 0x7f2050071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.748 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.743+0000 7f2055dff700 1 --2- 172.21.15.47:0/1436494174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2050071740 0x7f2050071bb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.749 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.743+0000 7f2055dff700 1 -- 172.21.15.47:0/1436494174 >> 172.21.15.47:0/1436494174 conn(0x7f20500f9460 msgr2=0x7f20500fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:08.749 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.743+0000 7f2055dff700 1 -- 172.21.15.47:0/1436494174 shutdown_connections 2022-09-07T21:39:08.749 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.744+0000 7f2055dff700 1 -- 172.21.15.47:0/1436494174 wait complete. 2022-09-07T21:39:08.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.745+0000 7f2055dff700 1 Processor -- start 2022-09-07T21:39:08.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.745+0000 7f2055dff700 1 -- start start 2022-09-07T21:39:08.754 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.745+0000 7f2055dff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2050071740 0x7f2050199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:08.754 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.745+0000 7f2055dff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2050072c90 0x7f205019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:08.755 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.745+0000 7f2055dff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f205010a410 0x7f20501a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:08.755 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.745+0000 7f2055dff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20501a1b60 con 0x7f205010a410 2022-09-07T21:39:08.755 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.745+0000 7f2055dff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20501a1cd0 con 0x7f2050071740 2022-09-07T21:39:08.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.745+0000 7f2055dff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20501a1e90 con 0x7f2050072c90 2022-09-07T21:39:08.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.745+0000 7f204ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f205010a410 0x7f20501a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:08.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.746+0000 7f204effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2050072c90 0x7f205019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:08.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.746+0000 7f204ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f205010a410 0x7f20501a1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:41886/0 (socket says 172.21.15.47:41886) 2022-09-07T21:39:08.757 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.746+0000 7f204f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2050071740 0x7f2050199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:08.757 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.746+0000 7f204ffff700 1 -- 172.21.15.47:0/390658631 learned_addr learned my addr 172.21.15.47:0/390658631 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:08.757 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.746+0000 7f204f7fe700 1 -- 172.21.15.47:0/390658631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2050072c90 msgr2=0x7f205019a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:08.757 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.746+0000 7f204f7fe700 1 --2- 172.21.15.47:0/390658631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2050072c90 0x7f205019a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.758 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.746+0000 7f204f7fe700 1 -- 172.21.15.47:0/390658631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f205010a410 msgr2=0x7f20501a1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:08.758 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.746+0000 7f204f7fe700 1 --2- 172.21.15.47:0/390658631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f205010a410 0x7f20501a1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:08.758 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.746+0000 7f204f7fe700 1 -- 172.21.15.47:0/390658631 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2044009470 con 0x7f2050071740 2022-09-07T21:39:08.758 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.746+0000 7f204ffff700 1 --2- 172.21.15.47:0/390658631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f205010a410 0x7f20501a1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:08.759 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.746+0000 7f204f7fe700 1 --2- 172.21.15.47:0/390658631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2050071740 0x7f2050199fe0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f204000e6c0 tx=0x7f204000e9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:08.759 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.747+0000 7f204cff9700 1 -- 172.21.15.47:0/390658631 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040010900 con 0x7f2050071740 2022-09-07T21:39:08.759 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.747+0000 7f2055dff700 1 -- 172.21.15.47:0/390658631 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20501a2170 con 0x7f2050071740 2022-09-07T21:39:08.759 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.747+0000 7f204cff9700 1 -- 172.21.15.47:0/390658631 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2040018420 con 0x7f2050071740 2022-09-07T21:39:08.760 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.747+0000 7f204cff9700 1 -- 172.21.15.47:0/390658631 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040017610 con 0x7f2050071740 2022-09-07T21:39:08.760 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.747+0000 7f2055dff700 1 -- 172.21.15.47:0/390658631 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20501a26d0 con 0x7f2050071740 2022-09-07T21:39:08.760 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.748+0000 7f2055dff700 1 -- 172.21.15.47:0/390658631 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f205010fbc0 con 0x7f2050071740 2022-09-07T21:39:08.760 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.750+0000 7f204cff9700 1 -- 172.21.15.47:0/390658631 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f204000c040 con 0x7f2050071740 2022-09-07T21:39:08.762 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.751+0000 7f204cff9700 1 --2- 172.21.15.47:0/390658631 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2038099090 0x7f203809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:08.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.751+0000 7f204cff9700 1 -- 172.21.15.47:0/390658631 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f2040014070 con 0x7f2050071740 2022-09-07T21:39:08.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.751+0000 7f204effd700 1 --2- 172.21.15.47:0/390658631 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2038099090 0x7f203809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:08.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.752+0000 7f204effd700 1 --2- 172.21.15.47:0/390658631 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2038099090 0x7f203809b540 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f204400b4f0 tx=0x7f2044009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:08.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.755+0000 7f204cff9700 1 -- 172.21.15.47:0/390658631 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f204004fc80 con 0x7f2050071740 2022-09-07T21:39:08.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:08.824+0000 7f204cff9700 1 -- 172.21.15.47:0/390658631 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f20400188e0 con 0x7f2050071740 2022-09-07T21:39:09.008 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.005+0000 7f2055dff700 1 -- 172.21.15.47:0/390658631 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f205005f120 con 0x7f2038099090 2022-09-07T21:39:09.012 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.009+0000 7f204cff9700 1 -- 172.21.15.47:0/390658631 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f205005f120 con 0x7f2038099090 2022-09-07T21:39:09.012 INFO:teuthology.orchestra.run.smithi047.stdout:/volumes/_nogroup/sv_1/3c62772f-2f12-4471-ad76-516abd941b3f 2022-09-07T21:39:09.015 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.011+0000 7f2055dff700 1 -- 172.21.15.47:0/390658631 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2038099090 msgr2=0x7f203809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:09.015 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.011+0000 7f2055dff700 1 --2- 172.21.15.47:0/390658631 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2038099090 0x7f203809b540 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f204400b4f0 tx=0x7f2044009eb0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.015 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.011+0000 7f2055dff700 1 -- 172.21.15.47:0/390658631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2050071740 msgr2=0x7f2050199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:09.016 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.011+0000 7f2055dff700 1 --2- 172.21.15.47:0/390658631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2050071740 0x7f2050199fe0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f204000e6c0 tx=0x7f204000e9d0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.016 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.011+0000 7f2055dff700 1 -- 172.21.15.47:0/390658631 shutdown_connections 2022-09-07T21:39:09.016 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.011+0000 7f2055dff700 1 --2- 172.21.15.47:0/390658631 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2038099090 0x7f203809b540 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.016 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.011+0000 7f2055dff700 1 --2- 172.21.15.47:0/390658631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f205010a410 0x7f20501a1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.016 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.011+0000 7f2055dff700 1 --2- 172.21.15.47:0/390658631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2050071740 0x7f2050199fe0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.017 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.011+0000 7f2055dff700 1 --2- 172.21.15.47:0/390658631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2050072c90 0x7f205019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.017 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.011+0000 7f2055dff700 1 -- 172.21.15.47:0/390658631 >> 172.21.15.47:0/390658631 conn(0x7f20500f9460 msgr2=0x7f20500fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:09.017 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.012+0000 7f2055dff700 1 -- 172.21.15.47:0/390658631 shutdown_connections 2022-09-07T21:39:09.017 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.012+0000 7f2055dff700 1 -- 172.21.15.47:0/390658631 wait complete. 2022-09-07T21:39:09.043 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:39:09.119 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:08 smithi047 ceph-mon[90087]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:39:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:08 smithi055 ceph-mon[97033]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:39:09.278 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.271+0000 7ff9469ff700 1 -- 172.21.15.47:0/122776949 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9401011d0 msgr2=0x7ff940101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:09.279 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.271+0000 7ff9469ff700 1 --2- 172.21.15.47:0/122776949 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9401011d0 0x7ff940101620 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff934009810 tx=0x7ff934009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:09.280 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.271+0000 7ff9469ff700 1 -- 172.21.15.47:0/122776949 shutdown_connections 2022-09-07T21:39:09.280 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.271+0000 7ff9469ff700 1 --2- 172.21.15.47:0/122776949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff940101b60 0x7ff940108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.281 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.271+0000 7ff9469ff700 1 --2- 172.21.15.47:0/122776949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9400fff50 0x7ff940100360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.281 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.271+0000 7ff9469ff700 1 --2- 172.21.15.47:0/122776949 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9401011d0 0x7ff940101620 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.281 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.271+0000 7ff9469ff700 1 -- 172.21.15.47:0/122776949 >> 172.21.15.47:0/122776949 conn(0x7ff9400fb6b0 msgr2=0x7ff9400fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:09.282 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.271+0000 7ff9469ff700 1 -- 172.21.15.47:0/122776949 shutdown_connections 2022-09-07T21:39:09.282 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.271+0000 7ff9469ff700 1 -- 172.21.15.47:0/122776949 wait complete. 2022-09-07T21:39:09.283 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.272+0000 7ff9469ff700 1 Processor -- start 2022-09-07T21:39:09.283 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.272+0000 7ff9469ff700 1 -- start start 2022-09-07T21:39:09.284 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.273+0000 7ff9469ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9400fff50 0x7ff940195af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:09.284 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.273+0000 7ff9469ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9401011d0 0x7ff940196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:09.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.273+0000 7ff9469ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff940101b60 0x7ff94019d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:09.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.273+0000 7ff9469ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94019d670 con 0x7ff9400fff50 2022-09-07T21:39:09.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.273+0000 7ff9469ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94019d7e0 con 0x7ff940101b60 2022-09-07T21:39:09.286 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.273+0000 7ff9469ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94019d9a0 con 0x7ff9401011d0 2022-09-07T21:39:09.286 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.273+0000 7ff93ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9400fff50 0x7ff940195af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:09.287 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.273+0000 7ff93ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9400fff50 0x7ff940195af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:41904/0 (socket says 172.21.15.47:41904) 2022-09-07T21:39:09.287 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.273+0000 7ff93ffff700 1 -- 172.21.15.47:0/1231874437 learned_addr learned my addr 172.21.15.47:0/1231874437 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:09.288 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.273+0000 7ff93f7fe700 1 --2- 172.21.15.47:0/1231874437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9401011d0 0x7ff940196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:09.288 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.273+0000 7ff944f9c700 1 --2- 172.21.15.47:0/1231874437 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff940101b60 0x7ff94019d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:09.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.274+0000 7ff93ffff700 1 -- 172.21.15.47:0/1231874437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9401011d0 msgr2=0x7ff940196030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:09.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.274+0000 7ff93ffff700 1 --2- 172.21.15.47:0/1231874437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9401011d0 0x7ff940196030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.274+0000 7ff93ffff700 1 -- 172.21.15.47:0/1231874437 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff940101b60 msgr2=0x7ff94019d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:09.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.274+0000 7ff93ffff700 1 --2- 172.21.15.47:0/1231874437 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff940101b60 0x7ff94019d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.274+0000 7ff93ffff700 1 -- 172.21.15.47:0/1231874437 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff934009470 con 0x7ff9400fff50 2022-09-07T21:39:09.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.274+0000 7ff93ffff700 1 --2- 172.21.15.47:0/1231874437 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9400fff50 0x7ff940195af0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff93000e990 tx=0x7ff93000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:09.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.274+0000 7ff93d7fa700 1 -- 172.21.15.47:0/1231874437 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930003a80 con 0x7ff9400fff50 2022-09-07T21:39:09.292 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.274+0000 7ff93d7fa700 1 -- 172.21.15.47:0/1231874437 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff93000cc50 con 0x7ff9400fff50 2022-09-07T21:39:09.292 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.274+0000 7ff9469ff700 1 -- 172.21.15.47:0/1231874437 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94019dc80 con 0x7ff9400fff50 2022-09-07T21:39:09.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.274+0000 7ff93d7fa700 1 -- 172.21.15.47:0/1231874437 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930004560 con 0x7ff9400fff50 2022-09-07T21:39:09.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.274+0000 7ff9469ff700 1 -- 172.21.15.47:0/1231874437 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94019e190 con 0x7ff9400fff50 2022-09-07T21:39:09.295 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:08 smithi059 ceph-mon[97450]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:39:09.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.276+0000 7ff9469ff700 1 -- 172.21.15.47:0/1231874437 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94010b700 con 0x7ff9400fff50 2022-09-07T21:39:09.297 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.277+0000 7ff93d7fa700 1 -- 172.21.15.47:0/1231874437 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7ff930021070 con 0x7ff9400fff50 2022-09-07T21:39:09.297 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.278+0000 7ff93d7fa700 1 --2- 172.21.15.47:0/1231874437 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff9280990c0 0x7ff92809b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:09.298 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.278+0000 7ff93d7fa700 1 -- 172.21.15.47:0/1231874437 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7ff9300c2520 con 0x7ff9400fff50 2022-09-07T21:39:09.298 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.279+0000 7ff93f7fe700 1 --2- 172.21.15.47:0/1231874437 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff9280990c0 0x7ff92809b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:09.299 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.279+0000 7ff93f7fe700 1 --2- 172.21.15.47:0/1231874437 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff9280990c0 0x7ff92809b570 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff934003610 tx=0x7ff934003500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:09.299 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.283+0000 7ff93d7fa700 1 -- 172.21.15.47:0/1231874437 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff93008e980 con 0x7ff9400fff50 2022-09-07T21:39:09.534 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.531+0000 7ff9469ff700 1 -- 172.21.15.47:0/1231874437 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff94019ead0 con 0x7ff9400fff50 2022-09-07T21:39:09.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.532+0000 7ff93d7fa700 1 -- 172.21.15.47:0/1231874437 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10707 (secure 0 0 0) 0x7ff93001f020 con 0x7ff9400fff50 2022-09-07T21:39:09.536 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:39:09.536 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/2559266585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2559266585},{"type":"v1","addr":"172.21.15.55:6835","nonce":2559266585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/2090942297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2090942297},{"type":"v1","addr":"172.21.15.55:6837","nonce":2090942297}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-09-07T21:38:40.544389+0000","modified":"2022-09-07T21:39:06.833800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24586,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6835/2445409240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2445409240},{"type":"v1","addr":"172.21.15.59:6835","nonce":2445409240}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/3542466465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":3542466465},{"type":"v1","addr":"172.21.15.59:6837","nonce":3542466465}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6839/315505812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":315505812},{"type":"v1","addr":"172.21.15.47:6839","nonce":315505812}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.59:6839/1006724740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":1006724740},{"type":"v1","addr":"172.21.15.59:6839","nonce":1006724740}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6839/517990300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":517990300},{"type":"v1","addr":"172.21.15.55:6839","nonce":517990300}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24586":{"gid":24586,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6841/793863162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6840","nonce":793863162},{"type":"v1","addr":"172.21.15.47:6841","nonce":793863162}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-09-07T21:39:09.538 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.535+0000 7ff9469ff700 1 -- 172.21.15.47:0/1231874437 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff9280990c0 msgr2=0x7ff92809b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:09.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.535+0000 7ff9469ff700 1 --2- 172.21.15.47:0/1231874437 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff9280990c0 0x7ff92809b570 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff934003610 tx=0x7ff934003500 comp rx=0 tx=0).stop 2022-09-07T21:39:09.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.535+0000 7ff9469ff700 1 -- 172.21.15.47:0/1231874437 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9400fff50 msgr2=0x7ff940195af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:09.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.535+0000 7ff9469ff700 1 --2- 172.21.15.47:0/1231874437 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9400fff50 0x7ff940195af0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff93000e990 tx=0x7ff93000eca0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.535+0000 7ff9469ff700 1 -- 172.21.15.47:0/1231874437 shutdown_connections 2022-09-07T21:39:09.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.535+0000 7ff9469ff700 1 --2- 172.21.15.47:0/1231874437 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff9280990c0 0x7ff92809b570 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.535+0000 7ff9469ff700 1 --2- 172.21.15.47:0/1231874437 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff940101b60 0x7ff94019d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.535+0000 7ff9469ff700 1 --2- 172.21.15.47:0/1231874437 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9400fff50 0x7ff940195af0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.535+0000 7ff9469ff700 1 --2- 172.21.15.47:0/1231874437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9401011d0 0x7ff940196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.535+0000 7ff9469ff700 1 -- 172.21.15.47:0/1231874437 >> 172.21.15.47:0/1231874437 conn(0x7ff9400fb6b0 msgr2=0x7ff9400fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:09.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.535+0000 7ff9469ff700 1 -- 172.21.15.47:0/1231874437 shutdown_connections 2022-09-07T21:39:09.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.535+0000 7ff9469ff700 1 -- 172.21.15.47:0/1231874437 wait complete. 2022-09-07T21:39:09.542 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 24 2022-09-07T21:39:09.566 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-07T21:39:09.800 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.796+0000 7f1256bff700 1 -- 172.21.15.47:0/4255033526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1250066be0 msgr2=0x7f1250102cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:09.800 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.796+0000 7f1256bff700 1 --2- 172.21.15.47:0/4255033526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1250066be0 0x7f1250102cb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1244009810 tx=0x7f1244009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:09.800 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.796+0000 7f1256bff700 1 -- 172.21.15.47:0/4255033526 shutdown_connections 2022-09-07T21:39:09.801 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.796+0000 7f1256bff700 1 --2- 172.21.15.47:0/4255033526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1250105b10 0x7f1250107f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.801 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.796+0000 7f1256bff700 1 --2- 172.21.15.47:0/4255033526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1250066be0 0x7f1250102cb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.801 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.796+0000 7f1256bff700 1 --2- 172.21.15.47:0/4255033526 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12501031f0 0x7f12501055d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.801 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.796+0000 7f1256bff700 1 -- 172.21.15.47:0/4255033526 >> 172.21.15.47:0/4255033526 conn(0x7f12500f82e0 msgr2=0x7f12500fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:09.801 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.796+0000 7f1256bff700 1 -- 172.21.15.47:0/4255033526 shutdown_connections 2022-09-07T21:39:09.802 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.796+0000 7f1256bff700 1 -- 172.21.15.47:0/4255033526 wait complete. 2022-09-07T21:39:09.804 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.798+0000 7f1256bff700 1 Processor -- start 2022-09-07T21:39:09.804 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.798+0000 7f1256bff700 1 -- start start 2022-09-07T21:39:09.804 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.798+0000 7f1256bff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1250066be0 0x7f1250195ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:09.804 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.798+0000 7f1256bff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12501031f0 0x7f12501960e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:09.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.799+0000 7f1256bff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1250105b10 0x7f125019d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:09.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.799+0000 7f1256bff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125019d720 con 0x7f1250066be0 2022-09-07T21:39:09.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.799+0000 7f1256bff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125019d890 con 0x7f1250105b10 2022-09-07T21:39:09.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.799+0000 7f1256bff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125019da50 con 0x7f12501031f0 2022-09-07T21:39:09.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.799+0000 7f125499b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1250066be0 0x7f1250195ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:09.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.799+0000 7f125519c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1250105b10 0x7f125019d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:09.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.799+0000 7f125499b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1250066be0 0x7f1250195ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:41934/0 (socket says 172.21.15.47:41934) 2022-09-07T21:39:09.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.799+0000 7f125499b700 1 -- 172.21.15.47:0/2211518304 learned_addr learned my addr 172.21.15.47:0/2211518304 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:09.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.799+0000 7f124ffff700 1 --2- 172.21.15.47:0/2211518304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12501031f0 0x7f12501960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:09.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.799+0000 7f125519c700 1 -- 172.21.15.47:0/2211518304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12501031f0 msgr2=0x7f12501960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:09.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.799+0000 7f125519c700 1 --2- 172.21.15.47:0/2211518304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12501031f0 0x7f12501960e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.799+0000 7f125519c700 1 -- 172.21.15.47:0/2211518304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1250066be0 msgr2=0x7f1250195ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:09.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.799+0000 7f125519c700 1 --2- 172.21.15.47:0/2211518304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1250066be0 0x7f1250195ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:09.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.799+0000 7f125519c700 1 -- 172.21.15.47:0/2211518304 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1244009470 con 0x7f1250105b10 2022-09-07T21:39:09.808 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.800+0000 7f125519c700 1 --2- 172.21.15.47:0/2211518304 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1250105b10 0x7f125019d150 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f124800ea80 tx=0x7f124800ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:09.808 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.800+0000 7f124dffb700 1 -- 172.21.15.47:0/2211518304 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12480098c0 con 0x7f1250105b10 2022-09-07T21:39:09.808 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.800+0000 7f1256bff700 1 -- 172.21.15.47:0/2211518304 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f125019dd30 con 0x7f1250105b10 2022-09-07T21:39:09.808 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.800+0000 7f124dffb700 1 -- 172.21.15.47:0/2211518304 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f124800ce10 con 0x7f1250105b10 2022-09-07T21:39:09.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.800+0000 7f124dffb700 1 -- 172.21.15.47:0/2211518304 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248004280 con 0x7f1250105b10 2022-09-07T21:39:09.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.800+0000 7f1256bff700 1 -- 172.21.15.47:0/2211518304 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f125019e1e0 con 0x7f1250105b10 2022-09-07T21:39:09.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.801+0000 7f1256bff700 1 -- 172.21.15.47:0/2211518304 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f125010b6e0 con 0x7f1250105b10 2022-09-07T21:39:09.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.803+0000 7f124dffb700 1 -- 172.21.15.47:0/2211518304 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f124800c040 con 0x7f1250105b10 2022-09-07T21:39:09.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.805+0000 7f124dffb700 1 --2- 172.21.15.47:0/2211518304 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1240099040 0x7f124009b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.805+0000 7f124dffb700 1 -- 172.21.15.47:0/2211518304 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f1248014070 con 0x7f1250105b10 2022-09-07T21:39:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.805+0000 7f125499b700 1 --2- 172.21.15.47:0/2211518304 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1240099040 0x7f124009b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.806+0000 7f125499b700 1 --2- 172.21.15.47:0/2211518304 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1240099040 0x7f124009b4f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f124400b4f0 tx=0x7f1244009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:09.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:09.807+0000 7f124dffb700 1 -- 172.21.15.47:0/2211518304 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f124808f350 con 0x7f1250105b10 2022-09-07T21:39:10.017 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.014+0000 7f1256bff700 1 -- 172.21.15.47:0/2211518304 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12500f9cd0 con 0x7f1250105b10 2022-09-07T21:39:10.019 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.016+0000 7f124dffb700 1 -- 172.21.15.47:0/2211518304 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23119 (secure 0 0 0) 0x7f124801c090 con 0x7f1250105b10 2022-09-07T21:39:10.019 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:39:10.020 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":89,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:38:45.529264+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-07T21:38:38.608937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-07T21:38:39.936252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-07T21:38:43.731797+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:39:10.022 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.018+0000 7f1256bff700 1 -- 172.21.15.47:0/2211518304 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1240099040 msgr2=0x7f124009b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:10.022 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.018+0000 7f1256bff700 1 --2- 172.21.15.47:0/2211518304 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1240099040 0x7f124009b4f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f124400b4f0 tx=0x7f1244009eb0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.022 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.018+0000 7f1256bff700 1 -- 172.21.15.47:0/2211518304 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1250105b10 msgr2=0x7f125019d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:10.023 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.018+0000 7f1256bff700 1 --2- 172.21.15.47:0/2211518304 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1250105b10 0x7f125019d150 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f124800ea80 tx=0x7f124800ed90 comp rx=0 tx=0).stop 2022-09-07T21:39:10.023 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.018+0000 7f1256bff700 1 -- 172.21.15.47:0/2211518304 shutdown_connections 2022-09-07T21:39:10.023 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.018+0000 7f1256bff700 1 --2- 172.21.15.47:0/2211518304 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1240099040 0x7f124009b4f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.023 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.018+0000 7f1256bff700 1 --2- 172.21.15.47:0/2211518304 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1250105b10 0x7f125019d150 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.023 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.018+0000 7f1256bff700 1 --2- 172.21.15.47:0/2211518304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1250066be0 0x7f1250195ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.024 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.018+0000 7f1256bff700 1 --2- 172.21.15.47:0/2211518304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12501031f0 0x7f12501960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.024 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.018+0000 7f1256bff700 1 -- 172.21.15.47:0/2211518304 >> 172.21.15.47:0/2211518304 conn(0x7f12500f82e0 msgr2=0x7f1250103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:10.024 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.018+0000 7f1256bff700 1 -- 172.21.15.47:0/2211518304 shutdown_connections 2022-09-07T21:39:10.024 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.018+0000 7f1256bff700 1 -- 172.21.15.47:0/2211518304 wait complete. 2022-09-07T21:39:10.050 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2022-09-07T21:39:10.051 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-09-07T21:39:10.129 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:09 smithi047 ceph-mon[90087]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:39:10.129 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:09 smithi047 ceph-mon[90087]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:39:10.129 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:09 smithi047 ceph-mon[90087]: pgmap v330: 129 pgs: 129 active+clean; 605 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2022-09-07T21:39:10.130 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:09 smithi047 ceph-mon[90087]: mgrmap e17: x(active, since 8m), standbys: y, z 2022-09-07T21:39:10.130 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:09 smithi047 ceph-mon[90087]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:39:10.130 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:09 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1231874437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:39:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:09 smithi055 ceph-mon[97033]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:39:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:09 smithi055 ceph-mon[97033]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:39:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:09 smithi055 ceph-mon[97033]: pgmap v330: 129 pgs: 129 active+clean; 605 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2022-09-07T21:39:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:09 smithi055 ceph-mon[97033]: mgrmap e17: x(active, since 8m), standbys: y, z 2022-09-07T21:39:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:09 smithi055 ceph-mon[97033]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:39:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:09 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1231874437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:39:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:09 smithi059 ceph-mon[97450]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:39:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:09 smithi059 ceph-mon[97450]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:39:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:09 smithi059 ceph-mon[97450]: pgmap v330: 129 pgs: 129 active+clean; 605 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2022-09-07T21:39:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:09 smithi059 ceph-mon[97450]: mgrmap e17: x(active, since 8m), standbys: y, z 2022-09-07T21:39:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:09 smithi059 ceph-mon[97450]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:39:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:09 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1231874437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:39:10.284 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.280+0000 7f978619d700 1 -- 172.21.15.47:0/3731930209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9780101740 msgr2=0x7f9780101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.280+0000 7f978619d700 1 --2- 172.21.15.47:0/3731930209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9780101740 0x7f9780101bb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9778009bb0 tx=0x7f9778009ec0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.281+0000 7f978619d700 1 -- 172.21.15.47:0/3731930209 shutdown_connections 2022-09-07T21:39:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.281+0000 7f978619d700 1 --2- 172.21.15.47:0/3731930209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9780101740 0x7f9780101bb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.281+0000 7f978619d700 1 --2- 172.21.15.47:0/3731930209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97800fc420 0x7f97800fe840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.281+0000 7f978619d700 1 --2- 172.21.15.47:0/3731930209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97800fed80 0x7f9780101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.281+0000 7f978619d700 1 -- 172.21.15.47:0/3731930209 >> 172.21.15.47:0/3731930209 conn(0x7f97800f80d0 msgr2=0x7f97800fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.281+0000 7f978619d700 1 -- 172.21.15.47:0/3731930209 shutdown_connections 2022-09-07T21:39:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.281+0000 7f978619d700 1 -- 172.21.15.47:0/3731930209 wait complete. 2022-09-07T21:39:10.288 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.282+0000 7f978619d700 1 Processor -- start 2022-09-07T21:39:10.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.282+0000 7f978619d700 1 -- start start 2022-09-07T21:39:10.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.282+0000 7f978619d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97800fc420 0x7f9780199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:10.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.282+0000 7f978619d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97800fed80 0x7f978019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:10.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.282+0000 7f978619d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9780101740 0x7f97801a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.282+0000 7f978619d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97801a1900 con 0x7f97800fed80 2022-09-07T21:39:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.282+0000 7f978619d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97801a1a70 con 0x7f97800fc420 2022-09-07T21:39:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.282+0000 7f978619d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97801a1c30 con 0x7f9780101740 2022-09-07T21:39:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.283+0000 7f978499a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97800fed80 0x7f978019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.283+0000 7f978519b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97800fc420 0x7f9780199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:10.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.283+0000 7f978499a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97800fed80 0x7f978019a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:41952/0 (socket says 172.21.15.47:41952) 2022-09-07T21:39:10.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.283+0000 7f978499a700 1 -- 172.21.15.47:0/3632652070 learned_addr learned my addr 172.21.15.47:0/3632652070 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:10.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.283+0000 7f978599c700 1 --2- 172.21.15.47:0/3632652070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9780101740 0x7f97801a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:10.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.283+0000 7f978519b700 1 -- 172.21.15.47:0/3632652070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9780101740 msgr2=0x7f97801a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:10.292 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.283+0000 7f978519b700 1 --2- 172.21.15.47:0/3632652070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9780101740 0x7f97801a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.292 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.283+0000 7f978519b700 1 -- 172.21.15.47:0/3632652070 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97800fed80 msgr2=0x7f978019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:10.292 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.283+0000 7f978519b700 1 --2- 172.21.15.47:0/3632652070 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97800fed80 0x7f978019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.292 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.283+0000 7f978519b700 1 -- 172.21.15.47:0/3632652070 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97780097e0 con 0x7f97800fc420 2022-09-07T21:39:10.292 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.283+0000 7f978499a700 1 --2- 172.21.15.47:0/3632652070 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97800fed80 0x7f978019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:10.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.284+0000 7f978519b700 1 --2- 172.21.15.47:0/3632652070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97800fc420 0x7f9780199d40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f977c00e6c0 tx=0x7f977c00e9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:10.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.284+0000 7f97767fc700 1 -- 172.21.15.47:0/3632652070 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c010900 con 0x7f97800fc420 2022-09-07T21:39:10.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.284+0000 7f97767fc700 1 -- 172.21.15.47:0/3632652070 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f977c018420 con 0x7f97800fc420 2022-09-07T21:39:10.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.284+0000 7f97767fc700 1 -- 172.21.15.47:0/3632652070 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c017610 con 0x7f97800fc420 2022-09-07T21:39:10.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.284+0000 7f978619d700 1 -- 172.21.15.47:0/3632652070 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97801a1f10 con 0x7f97800fc420 2022-09-07T21:39:10.294 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.284+0000 7f978619d700 1 -- 172.21.15.47:0/3632652070 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97801a22d0 con 0x7f97800fc420 2022-09-07T21:39:10.294 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.285+0000 7f978619d700 1 -- 172.21.15.47:0/3632652070 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978010f8c0 con 0x7f97800fc420 2022-09-07T21:39:10.295 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.287+0000 7f97767fc700 1 -- 172.21.15.47:0/3632652070 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f977c00c040 con 0x7f97800fc420 2022-09-07T21:39:10.295 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.288+0000 7f97767fc700 1 --2- 172.21.15.47:0/3632652070 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f976c099050 0x7f976c09b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:10.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.288+0000 7f97767fc700 1 -- 172.21.15.47:0/3632652070 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f977c014070 con 0x7f97800fc420 2022-09-07T21:39:10.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.289+0000 7f978499a700 1 --2- 172.21.15.47:0/3632652070 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f976c099050 0x7f976c09b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:10.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.289+0000 7f978499a700 1 --2- 172.21.15.47:0/3632652070 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f976c099050 0x7f976c09b500 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f97700057d0 tx=0x7f9770005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:10.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.291+0000 7f97767fc700 1 -- 172.21.15.47:0/3632652070 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f977c08eb00 con 0x7f97800fc420 2022-09-07T21:39:10.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.487+0000 7f978619d700 1 -- 172.21.15.47:0/3632652070 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f978005fbd0 con 0x7f976c099050 2022-09-07T21:39:10.507 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.504+0000 7f97767fc700 1 -- 172.21.15.47:0/3632652070 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422534 (secure 0 0 0) 0x7f978005fbd0 con 0x7f976c099050 2022-09-07T21:39:10.515 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:39:10.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.509+0000 7f978619d700 1 -- 172.21.15.47:0/3632652070 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f976c099050 msgr2=0x7f976c09b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:10.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.509+0000 7f978619d700 1 --2- 172.21.15.47:0/3632652070 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f976c099050 0x7f976c09b500 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f97700057d0 tx=0x7f9770005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.509+0000 7f978619d700 1 -- 172.21.15.47:0/3632652070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97800fc420 msgr2=0x7f9780199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:10.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.509+0000 7f978619d700 1 --2- 172.21.15.47:0/3632652070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97800fc420 0x7f9780199d40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f977c00e6c0 tx=0x7f977c00e9d0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.509+0000 7f978619d700 1 -- 172.21.15.47:0/3632652070 shutdown_connections 2022-09-07T21:39:10.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.509+0000 7f978619d700 1 --2- 172.21.15.47:0/3632652070 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f976c099050 0x7f976c09b500 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.509+0000 7f978619d700 1 --2- 172.21.15.47:0/3632652070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9780101740 0x7f97801a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.509+0000 7f978619d700 1 --2- 172.21.15.47:0/3632652070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97800fc420 0x7f9780199d40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.509+0000 7f978619d700 1 --2- 172.21.15.47:0/3632652070 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97800fed80 0x7f978019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.522 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.509+0000 7f978619d700 1 -- 172.21.15.47:0/3632652070 >> 172.21.15.47:0/3632652070 conn(0x7f97800f80d0 msgr2=0x7f97800fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:10.522 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.509+0000 7f978619d700 1 -- 172.21.15.47:0/3632652070 shutdown_connections 2022-09-07T21:39:10.522 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.510+0000 7f978619d700 1 -- 172.21.15.47:0/3632652070 wait complete. 2022-09-07T21:39:10.522 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2022-09-07T21:39:10.541 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2022-09-07T21:39:08.594291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":619592,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":155,"num_read_kb":137,"num_write":285,"num_write_kb":2997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":79392,"kb_used_data":11584,"kb_used_omap":0,"kb_used_meta":67712,"kb_avail":1124862432,"statfs":{"total":1151940427776,"available":1151859130368,"internally_reserved":0,"allocated":11862016,"data_stored":7493746,"data_compressed":250677,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":0,"internal_metadata":69337088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26894,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":19,"num_read_kb":18,"num_write":59,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003480"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974670+0000","last_change":"2022-09-07T21:38:40.576826+0000","last_active":"2022-09-07T21:38:44.974670+0000","last_peered":"2022-09-07T21:38:44.974670+0000","last_clean":"2022-09-07T21:38:44.974670+0000","last_became_active":"2022-09-07T21:38:40.576429+0000","last_became_peered":"2022-09-07T21:38:40.576429+0000","last_unstale":"2022-09-07T21:38:44.974670+0000","last_undegraded":"2022-09-07T21:38:44.974670+0000","last_fullsized":"2022-09-07T21:38:44.974670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:40:45.777536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.970047+0000","last_change":"2022-09-07T21:38:40.562924+0000","last_active":"2022-09-07T21:38:44.970047+0000","last_peered":"2022-09-07T21:38:44.970047+0000","last_clean":"2022-09-07T21:38:44.970047+0000","last_became_active":"2022-09-07T21:38:40.562509+0000","last_became_peered":"2022-09-07T21:38:40.562509+0000","last_unstale":"2022-09-07T21:38:44.970047+0000","last_undegraded":"2022-09-07T21:38:44.970047+0000","last_fullsized":"2022-09-07T21:38:44.970047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:27:16.781072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.969745+0000","last_change":"2022-09-07T21:38:40.562123+0000","last_active":"2022-09-07T21:38:44.969745+0000","last_peered":"2022-09-07T21:38:44.969745+0000","last_clean":"2022-09-07T21:38:44.969745+0000","last_became_active":"2022-09-07T21:38:40.561632+0000","last_became_peered":"2022-09-07T21:38:40.561632+0000","last_unstale":"2022-09-07T21:38:44.969745+0000","last_undegraded":"2022-09-07T21:38:44.969745+0000","last_fullsized":"2022-09-07T21:38:44.969745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:28:10.763115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.977488+0000","last_change":"2022-09-07T21:38:40.560968+0000","last_active":"2022-09-07T21:38:44.977488+0000","last_peered":"2022-09-07T21:38:44.977488+0000","last_clean":"2022-09-07T21:38:44.977488+0000","last_became_active":"2022-09-07T21:38:40.560685+0000","last_became_peered":"2022-09-07T21:38:40.560685+0000","last_unstale":"2022-09-07T21:38:44.977488+0000","last_undegraded":"2022-09-07T21:38:44.977488+0000","last_fullsized":"2022-09-07T21:38:44.977488+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:49:41.627456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975256+0000","last_change":"2022-09-07T21:38:40.572935+0000","last_active":"2022-09-07T21:38:44.975256+0000","last_peered":"2022-09-07T21:38:44.975256+0000","last_clean":"2022-09-07T21:38:44.975256+0000","last_became_active":"2022-09-07T21:38:40.572605+0000","last_became_peered":"2022-09-07T21:38:40.572605+0000","last_unstale":"2022-09-07T21:38:44.975256+0000","last_undegraded":"2022-09-07T21:38:44.975256+0000","last_fullsized":"2022-09-07T21:38:44.975256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:55:47.267936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.972241+0000","last_change":"2022-09-07T21:38:40.577737+0000","last_active":"2022-09-07T21:38:44.972241+0000","last_peered":"2022-09-07T21:38:44.972241+0000","last_clean":"2022-09-07T21:38:44.972241+0000","last_became_active":"2022-09-07T21:38:40.577425+0000","last_became_peered":"2022-09-07T21:38:40.577425+0000","last_unstale":"2022-09-07T21:38:44.972241+0000","last_undegraded":"2022-09-07T21:38:44.972241+0000","last_fullsized":"2022-09-07T21:38:44.972241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:46:32.369263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.981768+0000","last_change":"2022-09-07T21:38:40.577255+0000","last_active":"2022-09-07T21:38:44.981768+0000","last_peered":"2022-09-07T21:38:44.981768+0000","last_clean":"2022-09-07T21:38:44.981768+0000","last_became_active":"2022-09-07T21:38:40.576750+0000","last_became_peered":"2022-09-07T21:38:40.576750+0000","last_unstale":"2022-09-07T21:38:44.981768+0000","last_undegraded":"2022-09-07T21:38:44.981768+0000","last_fullsized":"2022-09-07T21:38:44.981768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:03:19.663251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974664+0000","last_change":"2022-09-07T21:38:40.577599+0000","last_active":"2022-09-07T21:38:44.974664+0000","last_peered":"2022-09-07T21:38:44.974664+0000","last_clean":"2022-09-07T21:38:44.974664+0000","last_became_active":"2022-09-07T21:38:40.577239+0000","last_became_peered":"2022-09-07T21:38:40.577239+0000","last_unstale":"2022-09-07T21:38:44.974664+0000","last_undegraded":"2022-09-07T21:38:44.974664+0000","last_fullsized":"2022-09-07T21:38:44.974664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:39:23.857698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.980029+0000","last_change":"2022-09-07T21:38:40.559102+0000","last_active":"2022-09-07T21:38:44.980029+0000","last_peered":"2022-09-07T21:38:44.980029+0000","last_clean":"2022-09-07T21:38:44.980029+0000","last_became_active":"2022-09-07T21:38:40.558907+0000","last_became_peered":"2022-09-07T21:38:40.558907+0000","last_unstale":"2022-09-07T21:38:44.980029+0000","last_undegraded":"2022-09-07T21:38:44.980029+0000","last_fullsized":"2022-09-07T21:38:44.980029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:08:03.644593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.972741+0000","last_change":"2022-09-07T21:38:40.566427+0000","last_active":"2022-09-07T21:38:44.972741+0000","last_peered":"2022-09-07T21:38:44.972741+0000","last_clean":"2022-09-07T21:38:44.972741+0000","last_became_active":"2022-09-07T21:38:40.566068+0000","last_became_peered":"2022-09-07T21:38:40.566068+0000","last_unstale":"2022-09-07T21:38:44.972741+0000","last_undegraded":"2022-09-07T21:38:44.972741+0000","last_fullsized":"2022-09-07T21:38:44.972741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:40:01.825654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.981081+0000","last_change":"2022-09-07T21:38:40.568455+0000","last_active":"2022-09-07T21:38:44.981081+0000","last_peered":"2022-09-07T21:38:44.981081+0000","last_clean":"2022-09-07T21:38:44.981081+0000","last_became_active":"2022-09-07T21:38:40.567941+0000","last_became_peered":"2022-09-07T21:38:40.567941+0000","last_unstale":"2022-09-07T21:38:44.981081+0000","last_undegraded":"2022-09-07T21:38:44.981081+0000","last_fullsized":"2022-09-07T21:38:44.981081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:49:16.624335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.977402+0000","last_change":"2022-09-07T21:38:40.563200+0000","last_active":"2022-09-07T21:38:44.977402+0000","last_peered":"2022-09-07T21:38:44.977402+0000","last_clean":"2022-09-07T21:38:44.977402+0000","last_became_active":"2022-09-07T21:38:40.562901+0000","last_became_peered":"2022-09-07T21:38:40.562901+0000","last_unstale":"2022-09-07T21:38:44.977402+0000","last_undegraded":"2022-09-07T21:38:44.977402+0000","last_fullsized":"2022-09-07T21:38:44.977402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:49:26.745940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.984739+0000","last_change":"2022-09-07T21:38:40.570403+0000","last_active":"2022-09-07T21:38:44.984739+0000","last_peered":"2022-09-07T21:38:44.984739+0000","last_clean":"2022-09-07T21:38:44.984739+0000","last_became_active":"2022-09-07T21:38:40.567326+0000","last_became_peered":"2022-09-07T21:38:40.567326+0000","last_unstale":"2022-09-07T21:38:44.984739+0000","last_undegraded":"2022-09-07T21:38:44.984739+0000","last_fullsized":"2022-09-07T21:38:44.984739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:19:31.437519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.969433+0000","last_change":"2022-09-07T21:38:40.562990+0000","last_active":"2022-09-07T21:38:44.969433+0000","last_peered":"2022-09-07T21:38:44.969433+0000","last_clean":"2022-09-07T21:38:44.969433+0000","last_became_active":"2022-09-07T21:38:40.562651+0000","last_became_peered":"2022-09-07T21:38:40.562651+0000","last_unstale":"2022-09-07T21:38:44.969433+0000","last_undegraded":"2022-09-07T21:38:44.969433+0000","last_fullsized":"2022-09-07T21:38:44.969433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:49:07.935459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.969293+0000","last_change":"2022-09-07T21:38:40.570849+0000","last_active":"2022-09-07T21:38:44.969293+0000","last_peered":"2022-09-07T21:38:44.969293+0000","last_clean":"2022-09-07T21:38:44.969293+0000","last_became_active":"2022-09-07T21:38:40.570543+0000","last_became_peered":"2022-09-07T21:38:40.570543+0000","last_unstale":"2022-09-07T21:38:44.969293+0000","last_undegraded":"2022-09-07T21:38:44.969293+0000","last_fullsized":"2022-09-07T21:38:44.969293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:33:30.564522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.977322+0000","last_change":"2022-09-07T21:38:40.568348+0000","last_active":"2022-09-07T21:38:44.977322+0000","last_peered":"2022-09-07T21:38:44.977322+0000","last_clean":"2022-09-07T21:38:44.977322+0000","last_became_active":"2022-09-07T21:38:40.568031+0000","last_became_peered":"2022-09-07T21:38:40.568031+0000","last_unstale":"2022-09-07T21:38:44.977322+0000","last_undegraded":"2022-09-07T21:38:44.977322+0000","last_fullsized":"2022-09-07T21:38:44.977322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:24:21.978277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974421+0000","last_change":"2022-09-07T21:38:40.577289+0000","last_active":"2022-09-07T21:38:44.974421+0000","last_peered":"2022-09-07T21:38:44.974421+0000","last_clean":"2022-09-07T21:38:44.974421+0000","last_became_active":"2022-09-07T21:38:40.576927+0000","last_became_peered":"2022-09-07T21:38:40.576927+0000","last_unstale":"2022-09-07T21:38:44.974421+0000","last_undegraded":"2022-09-07T21:38:44.974421+0000","last_fullsized":"2022-09-07T21:38:44.974421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:02:53.553226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.971786+0000","last_change":"2022-09-07T21:38:40.575886+0000","last_active":"2022-09-07T21:38:44.971786+0000","last_peered":"2022-09-07T21:38:44.971786+0000","last_clean":"2022-09-07T21:38:44.971786+0000","last_became_active":"2022-09-07T21:38:40.575463+0000","last_became_peered":"2022-09-07T21:38:40.575463+0000","last_unstale":"2022-09-07T21:38:44.971786+0000","last_undegraded":"2022-09-07T21:38:44.971786+0000","last_fullsized":"2022-09-07T21:38:44.971786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:10:20.091899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.979794+0000","last_change":"2022-09-07T21:38:40.572579+0000","last_active":"2022-09-07T21:38:44.979794+0000","last_peered":"2022-09-07T21:38:44.979794+0000","last_clean":"2022-09-07T21:38:44.979794+0000","last_became_active":"2022-09-07T21:38:40.572302+0000","last_became_peered":"2022-09-07T21:38:40.572302+0000","last_unstale":"2022-09-07T21:38:44.979794+0000","last_undegraded":"2022-09-07T21:38:44.979794+0000","last_fullsized":"2022-09-07T21:38:44.979794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:41:19.938512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.996782+0000","last_change":"2022-09-07T21:38:44.996073+0000","last_active":"2022-09-07T21:38:44.996782+0000","last_peered":"2022-09-07T21:38:44.996782+0000","last_clean":"2022-09-07T21:38:44.996782+0000","last_became_active":"2022-09-07T21:38:44.995571+0000","last_became_peered":"2022-09-07T21:38:44.995571+0000","last_unstale":"2022-09-07T21:38:44.996782+0000","last_undegraded":"2022-09-07T21:38:44.996782+0000","last_fullsized":"2022-09-07T21:38:44.996782+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:41:58.745823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.972053+0000","last_change":"2022-09-07T21:38:39.974715+0000","last_active":"2022-09-07T21:38:44.972053+0000","last_peered":"2022-09-07T21:38:44.972053+0000","last_clean":"2022-09-07T21:38:44.972053+0000","last_became_active":"2022-09-07T21:38:39.974413+0000","last_became_peered":"2022-09-07T21:38:39.974413+0000","last_unstale":"2022-09-07T21:38:44.972053+0000","last_undegraded":"2022-09-07T21:38:44.972053+0000","last_fullsized":"2022-09-07T21:38:44.972053+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:33:41.200805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974793+0000","last_change":"2022-09-07T21:38:40.578853+0000","last_active":"2022-09-07T21:38:44.974793+0000","last_peered":"2022-09-07T21:38:44.974793+0000","last_clean":"2022-09-07T21:38:44.974793+0000","last_became_active":"2022-09-07T21:38:40.578229+0000","last_became_peered":"2022-09-07T21:38:40.578229+0000","last_unstale":"2022-09-07T21:38:44.974793+0000","last_undegraded":"2022-09-07T21:38:44.974793+0000","last_fullsized":"2022-09-07T21:38:44.974793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:39:28.259219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771619+0000","last_change":"2022-09-07T21:38:44.995929+0000","last_active":"2022-09-07T21:38:45.771619+0000","last_peered":"2022-09-07T21:38:45.771619+0000","last_clean":"2022-09-07T21:38:45.771619+0000","last_became_active":"2022-09-07T21:38:44.995265+0000","last_became_peered":"2022-09-07T21:38:44.995265+0000","last_unstale":"2022-09-07T21:38:45.771619+0000","last_undegraded":"2022-09-07T21:38:45.771619+0000","last_fullsized":"2022-09-07T21:38:45.771619+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:11:30.103628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771465+0000","last_change":"2022-09-07T21:38:39.971598+0000","last_active":"2022-09-07T21:38:45.771465+0000","last_peered":"2022-09-07T21:38:45.771465+0000","last_clean":"2022-09-07T21:38:45.771465+0000","last_became_active":"2022-09-07T21:38:39.971079+0000","last_became_peered":"2022-09-07T21:38:39.971079+0000","last_unstale":"2022-09-07T21:38:45.771465+0000","last_undegraded":"2022-09-07T21:38:45.771465+0000","last_fullsized":"2022-09-07T21:38:45.771465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:01:22.960635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.969544+0000","last_change":"2022-09-07T21:38:40.577589+0000","last_active":"2022-09-07T21:38:44.969544+0000","last_peered":"2022-09-07T21:38:44.969544+0000","last_clean":"2022-09-07T21:38:44.969544+0000","last_became_active":"2022-09-07T21:38:40.577192+0000","last_became_peered":"2022-09-07T21:38:40.577192+0000","last_unstale":"2022-09-07T21:38:44.969544+0000","last_undegraded":"2022-09-07T21:38:44.969544+0000","last_fullsized":"2022-09-07T21:38:44.969544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:41:03.746467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770985+0000","last_change":"2022-09-07T21:38:44.997384+0000","last_active":"2022-09-07T21:38:45.770985+0000","last_peered":"2022-09-07T21:38:45.770985+0000","last_clean":"2022-09-07T21:38:45.770985+0000","last_became_active":"2022-09-07T21:38:44.996150+0000","last_became_peered":"2022-09-07T21:38:44.996150+0000","last_unstale":"2022-09-07T21:38:45.770985+0000","last_undegraded":"2022-09-07T21:38:45.770985+0000","last_fullsized":"2022-09-07T21:38:45.770985+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:30:05.633718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974821+0000","last_change":"2022-09-07T21:38:39.970057+0000","last_active":"2022-09-07T21:38:44.974821+0000","last_peered":"2022-09-07T21:38:44.974821+0000","last_clean":"2022-09-07T21:38:44.974821+0000","last_became_active":"2022-09-07T21:38:39.967832+0000","last_became_peered":"2022-09-07T21:38:39.967832+0000","last_unstale":"2022-09-07T21:38:44.974821+0000","last_undegraded":"2022-09-07T21:38:44.974821+0000","last_fullsized":"2022-09-07T21:38:44.974821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:33:13.316063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.980694+0000","last_change":"2022-09-07T21:38:40.572694+0000","last_active":"2022-09-07T21:38:44.980694+0000","last_peered":"2022-09-07T21:38:44.980694+0000","last_clean":"2022-09-07T21:38:44.980694+0000","last_became_active":"2022-09-07T21:38:40.572407+0000","last_became_peered":"2022-09-07T21:38:40.572407+0000","last_unstale":"2022-09-07T21:38:44.980694+0000","last_undegraded":"2022-09-07T21:38:44.980694+0000","last_fullsized":"2022-09-07T21:38:44.980694+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:40:17.238549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771762+0000","last_change":"2022-09-07T21:38:44.988445+0000","last_active":"2022-09-07T21:38:45.771762+0000","last_peered":"2022-09-07T21:38:45.771762+0000","last_clean":"2022-09-07T21:38:45.771762+0000","last_became_active":"2022-09-07T21:38:44.987966+0000","last_became_peered":"2022-09-07T21:38:44.987966+0000","last_unstale":"2022-09-07T21:38:45.771762+0000","last_undegraded":"2022-09-07T21:38:45.771762+0000","last_fullsized":"2022-09-07T21:38:45.771762+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:36:16.050494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975449+0000","last_change":"2022-09-07T21:38:39.963456+0000","last_active":"2022-09-07T21:38:44.975449+0000","last_peered":"2022-09-07T21:38:44.975449+0000","last_clean":"2022-09-07T21:38:44.975449+0000","last_became_active":"2022-09-07T21:38:39.963059+0000","last_became_peered":"2022-09-07T21:38:39.963059+0000","last_unstale":"2022-09-07T21:38:44.975449+0000","last_undegraded":"2022-09-07T21:38:44.975449+0000","last_fullsized":"2022-09-07T21:38:44.975449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:51:09.061725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.979895+0000","last_change":"2022-09-07T21:38:40.559585+0000","last_active":"2022-09-07T21:38:44.979895+0000","last_peered":"2022-09-07T21:38:44.979895+0000","last_clean":"2022-09-07T21:38:44.979895+0000","last_became_active":"2022-09-07T21:38:40.559307+0000","last_became_peered":"2022-09-07T21:38:40.559307+0000","last_unstale":"2022-09-07T21:38:44.979895+0000","last_undegraded":"2022-09-07T21:38:44.979895+0000","last_fullsized":"2022-09-07T21:38:44.979895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:20:32.332429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.986450+0000","last_change":"2022-09-07T21:38:44.985596+0000","last_active":"2022-09-07T21:38:44.986450+0000","last_peered":"2022-09-07T21:38:44.986450+0000","last_clean":"2022-09-07T21:38:44.986450+0000","last_became_active":"2022-09-07T21:38:44.985112+0000","last_became_peered":"2022-09-07T21:38:44.985112+0000","last_unstale":"2022-09-07T21:38:44.986450+0000","last_undegraded":"2022-09-07T21:38:44.986450+0000","last_fullsized":"2022-09-07T21:38:44.986450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:24:06.456838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:06.844632+0000","last_change":"2022-09-07T21:38:39.973295+0000","last_active":"2022-09-07T21:39:06.844632+0000","last_peered":"2022-09-07T21:39:06.844632+0000","last_clean":"2022-09-07T21:39:06.844632+0000","last_became_active":"2022-09-07T21:38:39.972932+0000","last_became_peered":"2022-09-07T21:38:39.972932+0000","last_unstale":"2022-09-07T21:39:06.844632+0000","last_undegraded":"2022-09-07T21:39:06.844632+0000","last_fullsized":"2022-09-07T21:39:06.844632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:11:53.533171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974789+0000","last_change":"2022-09-07T21:38:40.577546+0000","last_active":"2022-09-07T21:38:44.974789+0000","last_peered":"2022-09-07T21:38:44.974789+0000","last_clean":"2022-09-07T21:38:44.974789+0000","last_became_active":"2022-09-07T21:38:40.577229+0000","last_became_peered":"2022-09-07T21:38:40.577229+0000","last_unstale":"2022-09-07T21:38:44.974789+0000","last_undegraded":"2022-09-07T21:38:44.974789+0000","last_fullsized":"2022-09-07T21:38:44.974789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:57:36.947488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.995700+0000","last_change":"2022-09-07T21:38:44.993976+0000","last_active":"2022-09-07T21:38:44.995700+0000","last_peered":"2022-09-07T21:38:44.995700+0000","last_clean":"2022-09-07T21:38:44.995700+0000","last_became_active":"2022-09-07T21:38:44.993554+0000","last_became_peered":"2022-09-07T21:38:44.993554+0000","last_unstale":"2022-09-07T21:38:44.995700+0000","last_undegraded":"2022-09-07T21:38:44.995700+0000","last_fullsized":"2022-09-07T21:38:44.995700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:21:13.466677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:05.824799+0000","last_change":"2022-09-07T21:38:39.970361+0000","last_active":"2022-09-07T21:39:05.824799+0000","last_peered":"2022-09-07T21:39:05.824799+0000","last_clean":"2022-09-07T21:39:05.824799+0000","last_became_active":"2022-09-07T21:38:39.969420+0000","last_became_peered":"2022-09-07T21:38:39.969420+0000","last_unstale":"2022-09-07T21:39:05.824799+0000","last_undegraded":"2022-09-07T21:39:05.824799+0000","last_fullsized":"2022-09-07T21:39:05.824799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:53:46.903275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974959+0000","last_change":"2022-09-07T21:38:40.567273+0000","last_active":"2022-09-07T21:38:44.974959+0000","last_peered":"2022-09-07T21:38:44.974959+0000","last_clean":"2022-09-07T21:38:44.974959+0000","last_became_active":"2022-09-07T21:38:40.566942+0000","last_became_peered":"2022-09-07T21:38:40.566942+0000","last_unstale":"2022-09-07T21:38:44.974959+0000","last_undegraded":"2022-09-07T21:38:44.974959+0000","last_fullsized":"2022-09-07T21:38:44.974959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:29:38.009443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770562+0000","last_change":"2022-09-07T21:38:44.998836+0000","last_active":"2022-09-07T21:38:45.770562+0000","last_peered":"2022-09-07T21:38:45.770562+0000","last_clean":"2022-09-07T21:38:45.770562+0000","last_became_active":"2022-09-07T21:38:44.998461+0000","last_became_peered":"2022-09-07T21:38:44.998461+0000","last_unstale":"2022-09-07T21:38:45.770562+0000","last_undegraded":"2022-09-07T21:38:45.770562+0000","last_fullsized":"2022-09-07T21:38:45.770562+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:37:37.203792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:05.826963+0000","last_change":"2022-09-07T21:38:39.970746+0000","last_active":"2022-09-07T21:39:05.826963+0000","last_peered":"2022-09-07T21:39:05.826963+0000","last_clean":"2022-09-07T21:39:05.826963+0000","last_became_active":"2022-09-07T21:38:39.970438+0000","last_became_peered":"2022-09-07T21:38:39.970438+0000","last_unstale":"2022-09-07T21:39:05.826963+0000","last_undegraded":"2022-09-07T21:39:05.826963+0000","last_fullsized":"2022-09-07T21:39:05.826963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:51:07.849231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974556+0000","last_change":"2022-09-07T21:38:40.576245+0000","last_active":"2022-09-07T21:38:44.974556+0000","last_peered":"2022-09-07T21:38:44.974556+0000","last_clean":"2022-09-07T21:38:44.974556+0000","last_became_active":"2022-09-07T21:38:40.575916+0000","last_became_peered":"2022-09-07T21:38:40.575916+0000","last_unstale":"2022-09-07T21:38:44.974556+0000","last_undegraded":"2022-09-07T21:38:44.974556+0000","last_fullsized":"2022-09-07T21:38:44.974556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:35:52.025507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.533897+0000","last_change":"2022-09-07T21:38:44.991494+0000","last_active":"2022-09-07T21:38:45.533897+0000","last_peered":"2022-09-07T21:38:45.533897+0000","last_clean":"2022-09-07T21:38:45.533897+0000","last_became_active":"2022-09-07T21:38:44.988446+0000","last_became_peered":"2022-09-07T21:38:44.988446+0000","last_unstale":"2022-09-07T21:38:45.533897+0000","last_undegraded":"2022-09-07T21:38:45.533897+0000","last_fullsized":"2022-09-07T21:38:45.533897+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:22:52.937965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771325+0000","last_change":"2022-09-07T21:38:39.966535+0000","last_active":"2022-09-07T21:38:45.771325+0000","last_peered":"2022-09-07T21:38:45.771325+0000","last_clean":"2022-09-07T21:38:45.771325+0000","last_became_active":"2022-09-07T21:38:39.965895+0000","last_became_peered":"2022-09-07T21:38:39.965895+0000","last_unstale":"2022-09-07T21:38:45.771325+0000","last_undegraded":"2022-09-07T21:38:45.771325+0000","last_fullsized":"2022-09-07T21:38:45.771325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:16:11.146779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.971237+0000","last_change":"2022-09-07T21:38:40.590979+0000","last_active":"2022-09-07T21:38:44.971237+0000","last_peered":"2022-09-07T21:38:44.971237+0000","last_clean":"2022-09-07T21:38:44.971237+0000","last_became_active":"2022-09-07T21:38:40.590024+0000","last_became_peered":"2022-09-07T21:38:40.590024+0000","last_unstale":"2022-09-07T21:38:44.971237+0000","last_undegraded":"2022-09-07T21:38:44.971237+0000","last_fullsized":"2022-09-07T21:38:44.971237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:58:11.726364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770147+0000","last_change":"2022-09-07T21:38:44.988252+0000","last_active":"2022-09-07T21:38:45.770147+0000","last_peered":"2022-09-07T21:38:45.770147+0000","last_clean":"2022-09-07T21:38:45.770147+0000","last_became_active":"2022-09-07T21:38:44.987851+0000","last_became_peered":"2022-09-07T21:38:44.987851+0000","last_unstale":"2022-09-07T21:38:45.770147+0000","last_undegraded":"2022-09-07T21:38:45.770147+0000","last_fullsized":"2022-09-07T21:38:45.770147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:53:42.207358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.808370+0000","last_change":"2022-09-07T21:38:39.969970+0000","last_active":"2022-09-07T21:39:04.808370+0000","last_peered":"2022-09-07T21:39:04.808370+0000","last_clean":"2022-09-07T21:39:04.808370+0000","last_became_active":"2022-09-07T21:38:39.969627+0000","last_became_peered":"2022-09-07T21:38:39.969627+0000","last_unstale":"2022-09-07T21:39:04.808370+0000","last_undegraded":"2022-09-07T21:39:04.808370+0000","last_fullsized":"2022-09-07T21:39:04.808370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:14:11.990138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.980466+0000","last_change":"2022-09-07T21:38:40.577175+0000","last_active":"2022-09-07T21:38:44.980466+0000","last_peered":"2022-09-07T21:38:44.980466+0000","last_clean":"2022-09-07T21:38:44.980466+0000","last_became_active":"2022-09-07T21:38:40.576873+0000","last_became_peered":"2022-09-07T21:38:40.576873+0000","last_unstale":"2022-09-07T21:38:44.980466+0000","last_undegraded":"2022-09-07T21:38:44.980466+0000","last_fullsized":"2022-09-07T21:38:44.980466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:25:17.911241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.029838+0000","last_change":"2022-09-07T21:38:44.984456+0000","last_active":"2022-09-07T21:38:45.029838+0000","last_peered":"2022-09-07T21:38:45.029838+0000","last_clean":"2022-09-07T21:38:45.029838+0000","last_became_active":"2022-09-07T21:38:44.984070+0000","last_became_peered":"2022-09-07T21:38:44.984070+0000","last_unstale":"2022-09-07T21:38:45.029838+0000","last_undegraded":"2022-09-07T21:38:45.029838+0000","last_fullsized":"2022-09-07T21:38:45.029838+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:36:14.150189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.978172+0000","last_change":"2022-09-07T21:38:39.975863+0000","last_active":"2022-09-07T21:38:44.978172+0000","last_peered":"2022-09-07T21:38:44.978172+0000","last_clean":"2022-09-07T21:38:44.978172+0000","last_became_active":"2022-09-07T21:38:39.975490+0000","last_became_peered":"2022-09-07T21:38:39.975490+0000","last_unstale":"2022-09-07T21:38:44.978172+0000","last_undegraded":"2022-09-07T21:38:44.978172+0000","last_fullsized":"2022-09-07T21:38:44.978172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:46:54.710925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.969816+0000","last_change":"2022-09-07T21:38:40.571594+0000","last_active":"2022-09-07T21:38:44.969816+0000","last_peered":"2022-09-07T21:38:44.969816+0000","last_clean":"2022-09-07T21:38:44.969816+0000","last_became_active":"2022-09-07T21:38:40.571188+0000","last_became_peered":"2022-09-07T21:38:40.571188+0000","last_unstale":"2022-09-07T21:38:44.969816+0000","last_undegraded":"2022-09-07T21:38:44.969816+0000","last_fullsized":"2022-09-07T21:38:44.969816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:25:49.474602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771857+0000","last_change":"2022-09-07T21:38:44.996134+0000","last_active":"2022-09-07T21:38:45.771857+0000","last_peered":"2022-09-07T21:38:45.771857+0000","last_clean":"2022-09-07T21:38:45.771857+0000","last_became_active":"2022-09-07T21:38:44.995696+0000","last_became_peered":"2022-09-07T21:38:44.995696+0000","last_unstale":"2022-09-07T21:38:45.771857+0000","last_undegraded":"2022-09-07T21:38:45.771857+0000","last_fullsized":"2022-09-07T21:38:45.771857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:52:03.648431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.809728+0000","last_change":"2022-09-07T21:38:39.964670+0000","last_active":"2022-09-07T21:39:04.809728+0000","last_peered":"2022-09-07T21:39:04.809728+0000","last_clean":"2022-09-07T21:39:04.809728+0000","last_became_active":"2022-09-07T21:38:39.964305+0000","last_became_peered":"2022-09-07T21:38:39.964305+0000","last_unstale":"2022-09-07T21:39:04.809728+0000","last_undegraded":"2022-09-07T21:39:04.809728+0000","last_fullsized":"2022-09-07T21:39:04.809728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:15:57.041815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.980818+0000","last_change":"2022-09-07T21:38:40.565156+0000","last_active":"2022-09-07T21:38:44.980818+0000","last_peered":"2022-09-07T21:38:44.980818+0000","last_clean":"2022-09-07T21:38:44.980818+0000","last_became_active":"2022-09-07T21:38:40.564843+0000","last_became_peered":"2022-09-07T21:38:40.564843+0000","last_unstale":"2022-09-07T21:38:44.980818+0000","last_undegraded":"2022-09-07T21:38:44.980818+0000","last_fullsized":"2022-09-07T21:38:44.980818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:11:09.741599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.029862+0000","last_change":"2022-09-07T21:38:44.986167+0000","last_active":"2022-09-07T21:38:45.029862+0000","last_peered":"2022-09-07T21:38:45.029862+0000","last_clean":"2022-09-07T21:38:45.029862+0000","last_became_active":"2022-09-07T21:38:44.985778+0000","last_became_peered":"2022-09-07T21:38:44.985778+0000","last_unstale":"2022-09-07T21:38:45.029862+0000","last_undegraded":"2022-09-07T21:38:45.029862+0000","last_fullsized":"2022-09-07T21:38:45.029862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:21:01.441205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'19","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:07.408184+0000","last_change":"2022-09-07T21:38:39.968660+0000","last_active":"2022-09-07T21:39:07.408184+0000","last_peered":"2022-09-07T21:39:07.408184+0000","last_clean":"2022-09-07T21:39:07.408184+0000","last_became_active":"2022-09-07T21:38:39.968285+0000","last_became_peered":"2022-09-07T21:38:39.968285+0000","last_unstale":"2022-09-07T21:39:07.408184+0000","last_undegraded":"2022-09-07T21:39:07.408184+0000","last_fullsized":"2022-09-07T21:39:07.408184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:16:16.027615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25650,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":20,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974554+0000","last_change":"2022-09-07T21:38:40.566145+0000","last_active":"2022-09-07T21:38:44.974554+0000","last_peered":"2022-09-07T21:38:44.974554+0000","last_clean":"2022-09-07T21:38:44.974554+0000","last_became_active":"2022-09-07T21:38:40.565823+0000","last_became_peered":"2022-09-07T21:38:40.565823+0000","last_unstale":"2022-09-07T21:38:44.974554+0000","last_undegraded":"2022-09-07T21:38:44.974554+0000","last_fullsized":"2022-09-07T21:38:44.974554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:13:28.826066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771597+0000","last_change":"2022-09-07T21:38:44.995599+0000","last_active":"2022-09-07T21:38:45.771597+0000","last_peered":"2022-09-07T21:38:45.771597+0000","last_clean":"2022-09-07T21:38:45.771597+0000","last_became_active":"2022-09-07T21:38:44.994627+0000","last_became_peered":"2022-09-07T21:38:44.994627+0000","last_unstale":"2022-09-07T21:38:45.771597+0000","last_undegraded":"2022-09-07T21:38:45.771597+0000","last_fullsized":"2022-09-07T21:38:45.771597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:17:30.608595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975870+0000","last_change":"2022-09-07T21:38:39.968773+0000","last_active":"2022-09-07T21:38:44.975870+0000","last_peered":"2022-09-07T21:38:44.975870+0000","last_clean":"2022-09-07T21:38:44.975870+0000","last_became_active":"2022-09-07T21:38:39.967912+0000","last_became_peered":"2022-09-07T21:38:39.967912+0000","last_unstale":"2022-09-07T21:38:44.975870+0000","last_undegraded":"2022-09-07T21:38:44.975870+0000","last_fullsized":"2022-09-07T21:38:44.975870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:51:12.940327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.976273+0000","last_change":"2022-09-07T21:38:40.573548+0000","last_active":"2022-09-07T21:38:44.976273+0000","last_peered":"2022-09-07T21:38:44.976273+0000","last_clean":"2022-09-07T21:38:44.976273+0000","last_became_active":"2022-09-07T21:38:40.573273+0000","last_became_peered":"2022-09-07T21:38:40.573273+0000","last_unstale":"2022-09-07T21:38:44.976273+0000","last_undegraded":"2022-09-07T21:38:44.976273+0000","last_fullsized":"2022-09-07T21:38:44.976273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:03:52.778107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770487+0000","last_change":"2022-09-07T21:38:44.987366+0000","last_active":"2022-09-07T21:38:45.770487+0000","last_peered":"2022-09-07T21:38:45.770487+0000","last_clean":"2022-09-07T21:38:45.770487+0000","last_became_active":"2022-09-07T21:38:44.986775+0000","last_became_peered":"2022-09-07T21:38:44.986775+0000","last_unstale":"2022-09-07T21:38:45.770487+0000","last_undegraded":"2022-09-07T21:38:45.770487+0000","last_fullsized":"2022-09-07T21:38:45.770487+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:07:14.746521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.809894+0000","last_change":"2022-09-07T21:38:39.969123+0000","last_active":"2022-09-07T21:39:04.809894+0000","last_peered":"2022-09-07T21:39:04.809894+0000","last_clean":"2022-09-07T21:39:04.809894+0000","last_became_active":"2022-09-07T21:38:39.968744+0000","last_became_peered":"2022-09-07T21:38:39.968744+0000","last_unstale":"2022-09-07T21:39:04.809894+0000","last_undegraded":"2022-09-07T21:39:04.809894+0000","last_fullsized":"2022-09-07T21:39:04.809894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:54:34.616856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.977308+0000","last_change":"2022-09-07T21:38:40.578056+0000","last_active":"2022-09-07T21:38:44.977308+0000","last_peered":"2022-09-07T21:38:44.977308+0000","last_clean":"2022-09-07T21:38:44.977308+0000","last_became_active":"2022-09-07T21:38:40.577727+0000","last_became_peered":"2022-09-07T21:38:40.577727+0000","last_unstale":"2022-09-07T21:38:44.977308+0000","last_undegraded":"2022-09-07T21:38:44.977308+0000","last_fullsized":"2022-09-07T21:38:44.977308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:10:08.645724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.029910+0000","last_change":"2022-09-07T21:38:44.993544+0000","last_active":"2022-09-07T21:38:45.029910+0000","last_peered":"2022-09-07T21:38:45.029910+0000","last_clean":"2022-09-07T21:38:45.029910+0000","last_became_active":"2022-09-07T21:38:44.986520+0000","last_became_peered":"2022-09-07T21:38:44.986520+0000","last_unstale":"2022-09-07T21:38:45.029910+0000","last_undegraded":"2022-09-07T21:38:45.029910+0000","last_fullsized":"2022-09-07T21:38:45.029910+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:54:18.417488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:07.847298+0000","last_change":"2022-09-07T21:38:39.962760+0000","last_active":"2022-09-07T21:39:07.847298+0000","last_peered":"2022-09-07T21:39:07.847298+0000","last_clean":"2022-09-07T21:39:07.847298+0000","last_became_active":"2022-09-07T21:38:39.962327+0000","last_became_peered":"2022-09-07T21:38:39.962327+0000","last_unstale":"2022-09-07T21:39:07.847298+0000","last_undegraded":"2022-09-07T21:39:07.847298+0000","last_fullsized":"2022-09-07T21:39:07.847298+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:17:27.180736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.983608+0000","last_change":"2022-09-07T21:38:40.569532+0000","last_active":"2022-09-07T21:38:44.983608+0000","last_peered":"2022-09-07T21:38:44.983608+0000","last_clean":"2022-09-07T21:38:44.983608+0000","last_became_active":"2022-09-07T21:38:40.569037+0000","last_became_peered":"2022-09-07T21:38:40.569037+0000","last_unstale":"2022-09-07T21:38:44.983608+0000","last_undegraded":"2022-09-07T21:38:44.983608+0000","last_fullsized":"2022-09-07T21:38:44.983608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:37:05.114735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770904+0000","last_change":"2022-09-07T21:38:44.995323+0000","last_active":"2022-09-07T21:38:45.770904+0000","last_peered":"2022-09-07T21:38:45.770904+0000","last_clean":"2022-09-07T21:38:45.770904+0000","last_became_active":"2022-09-07T21:38:44.994915+0000","last_became_peered":"2022-09-07T21:38:44.994915+0000","last_unstale":"2022-09-07T21:38:45.770904+0000","last_undegraded":"2022-09-07T21:38:45.770904+0000","last_fullsized":"2022-09-07T21:38:45.770904+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:17:10.419180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'10","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.805649+0000","last_change":"2022-09-07T21:38:39.967903+0000","last_active":"2022-09-07T21:39:04.805649+0000","last_peered":"2022-09-07T21:39:04.805649+0000","last_clean":"2022-09-07T21:39:04.805649+0000","last_became_active":"2022-09-07T21:38:39.967392+0000","last_became_peered":"2022-09-07T21:38:39.967392+0000","last_unstale":"2022-09-07T21:39:04.805649+0000","last_undegraded":"2022-09-07T21:39:04.805649+0000","last_fullsized":"2022-09-07T21:39:04.805649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:31:39.581561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.976347+0000","last_change":"2022-09-07T21:38:40.566956+0000","last_active":"2022-09-07T21:38:44.976347+0000","last_peered":"2022-09-07T21:38:44.976347+0000","last_clean":"2022-09-07T21:38:44.976347+0000","last_became_active":"2022-09-07T21:38:40.566655+0000","last_became_peered":"2022-09-07T21:38:40.566655+0000","last_unstale":"2022-09-07T21:38:44.976347+0000","last_undegraded":"2022-09-07T21:38:44.976347+0000","last_fullsized":"2022-09-07T21:38:44.976347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:37:16.353162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771274+0000","last_change":"2022-09-07T21:38:44.988605+0000","last_active":"2022-09-07T21:38:45.771274+0000","last_peered":"2022-09-07T21:38:45.771274+0000","last_clean":"2022-09-07T21:38:45.771274+0000","last_became_active":"2022-09-07T21:38:44.988082+0000","last_became_peered":"2022-09-07T21:38:44.988082+0000","last_unstale":"2022-09-07T21:38:45.771274+0000","last_undegraded":"2022-09-07T21:38:45.771274+0000","last_fullsized":"2022-09-07T21:38:45.771274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:30:06.975077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975833+0000","last_change":"2022-09-07T21:38:39.961740+0000","last_active":"2022-09-07T21:38:44.975833+0000","last_peered":"2022-09-07T21:38:44.975833+0000","last_clean":"2022-09-07T21:38:44.975833+0000","last_became_active":"2022-09-07T21:38:39.961219+0000","last_became_peered":"2022-09-07T21:38:39.961219+0000","last_unstale":"2022-09-07T21:38:44.975833+0000","last_undegraded":"2022-09-07T21:38:44.975833+0000","last_fullsized":"2022-09-07T21:38:44.975833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:51:24.649401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.973894+0000","last_change":"2022-09-07T21:38:40.569594+0000","last_active":"2022-09-07T21:38:44.973894+0000","last_peered":"2022-09-07T21:38:44.973894+0000","last_clean":"2022-09-07T21:38:44.973894+0000","last_became_active":"2022-09-07T21:38:40.569251+0000","last_became_peered":"2022-09-07T21:38:40.569251+0000","last_unstale":"2022-09-07T21:38:44.973894+0000","last_undegraded":"2022-09-07T21:38:44.973894+0000","last_fullsized":"2022-09-07T21:38:44.973894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:37:43.337560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.392327+0000","last_change":"2022-09-07T21:38:44.993534+0000","last_active":"2022-09-07T21:38:45.392327+0000","last_peered":"2022-09-07T21:38:45.392327+0000","last_clean":"2022-09-07T21:38:45.392327+0000","last_became_active":"2022-09-07T21:38:44.992966+0000","last_became_peered":"2022-09-07T21:38:44.992966+0000","last_unstale":"2022-09-07T21:38:45.392327+0000","last_undegraded":"2022-09-07T21:38:45.392327+0000","last_fullsized":"2022-09-07T21:38:45.392327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:12:27.991839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.971397+0000","last_change":"2022-09-07T21:38:39.978436+0000","last_active":"2022-09-07T21:38:44.971397+0000","last_peered":"2022-09-07T21:38:44.971397+0000","last_clean":"2022-09-07T21:38:44.971397+0000","last_became_active":"2022-09-07T21:38:39.977613+0000","last_became_peered":"2022-09-07T21:38:39.977613+0000","last_unstale":"2022-09-07T21:38:44.971397+0000","last_undegraded":"2022-09-07T21:38:44.971397+0000","last_fullsized":"2022-09-07T21:38:44.971397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:23:08.765467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.972585+0000","last_change":"2022-09-07T21:38:40.577968+0000","last_active":"2022-09-07T21:38:44.972585+0000","last_peered":"2022-09-07T21:38:44.972585+0000","last_clean":"2022-09-07T21:38:44.972585+0000","last_became_active":"2022-09-07T21:38:40.577632+0000","last_became_peered":"2022-09-07T21:38:40.577632+0000","last_unstale":"2022-09-07T21:38:44.972585+0000","last_undegraded":"2022-09-07T21:38:44.972585+0000","last_fullsized":"2022-09-07T21:38:44.972585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:57:31.599202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770320+0000","last_change":"2022-09-07T21:38:44.989341+0000","last_active":"2022-09-07T21:38:45.770320+0000","last_peered":"2022-09-07T21:38:45.770320+0000","last_clean":"2022-09-07T21:38:45.770320+0000","last_became_active":"2022-09-07T21:38:44.988977+0000","last_became_peered":"2022-09-07T21:38:44.988977+0000","last_unstale":"2022-09-07T21:38:45.770320+0000","last_undegraded":"2022-09-07T21:38:45.770320+0000","last_fullsized":"2022-09-07T21:38:45.770320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:34:52.397025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.979194+0000","last_change":"2022-09-07T21:38:39.970365+0000","last_active":"2022-09-07T21:38:44.979194+0000","last_peered":"2022-09-07T21:38:44.979194+0000","last_clean":"2022-09-07T21:38:44.979194+0000","last_became_active":"2022-09-07T21:38:39.970044+0000","last_became_peered":"2022-09-07T21:38:39.970044+0000","last_unstale":"2022-09-07T21:38:44.979194+0000","last_undegraded":"2022-09-07T21:38:44.979194+0000","last_fullsized":"2022-09-07T21:38:44.979194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:39:04.408591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.972235+0000","last_change":"2022-09-07T21:38:40.564714+0000","last_active":"2022-09-07T21:38:44.972235+0000","last_peered":"2022-09-07T21:38:44.972235+0000","last_clean":"2022-09-07T21:38:44.972235+0000","last_became_active":"2022-09-07T21:38:40.563744+0000","last_became_peered":"2022-09-07T21:38:40.563744+0000","last_unstale":"2022-09-07T21:38:44.972235+0000","last_undegraded":"2022-09-07T21:38:44.972235+0000","last_fullsized":"2022-09-07T21:38:44.972235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:46:29.965821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.769987+0000","last_change":"2022-09-07T21:38:44.999889+0000","last_active":"2022-09-07T21:38:45.769987+0000","last_peered":"2022-09-07T21:38:45.769987+0000","last_clean":"2022-09-07T21:38:45.769987+0000","last_became_active":"2022-09-07T21:38:44.999453+0000","last_became_peered":"2022-09-07T21:38:44.999453+0000","last_unstale":"2022-09-07T21:38:45.769987+0000","last_undegraded":"2022-09-07T21:38:45.769987+0000","last_fullsized":"2022-09-07T21:38:45.769987+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:26:24.497901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.977617+0000","last_change":"2022-09-07T21:38:40.567008+0000","last_active":"2022-09-07T21:38:44.977617+0000","last_peered":"2022-09-07T21:38:44.977617+0000","last_clean":"2022-09-07T21:38:44.977617+0000","last_became_active":"2022-09-07T21:38:40.566598+0000","last_became_peered":"2022-09-07T21:38:40.566598+0000","last_unstale":"2022-09-07T21:38:44.977617+0000","last_undegraded":"2022-09-07T21:38:44.977617+0000","last_fullsized":"2022-09-07T21:38:44.977617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:06:16.999794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:05.826220+0000","last_change":"2022-09-07T21:38:39.966446+0000","last_active":"2022-09-07T21:39:05.826220+0000","last_peered":"2022-09-07T21:39:05.826220+0000","last_clean":"2022-09-07T21:39:05.826220+0000","last_became_active":"2022-09-07T21:38:39.966095+0000","last_became_peered":"2022-09-07T21:38:39.966095+0000","last_unstale":"2022-09-07T21:39:05.826220+0000","last_undegraded":"2022-09-07T21:39:05.826220+0000","last_fullsized":"2022-09-07T21:39:05.826220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:08:54.360894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.536749+0000","last_change":"2022-09-07T21:38:44.993551+0000","last_active":"2022-09-07T21:38:45.536749+0000","last_peered":"2022-09-07T21:38:45.536749+0000","last_clean":"2022-09-07T21:38:45.536749+0000","last_became_active":"2022-09-07T21:38:44.993235+0000","last_became_peered":"2022-09-07T21:38:44.993235+0000","last_unstale":"2022-09-07T21:38:45.536749+0000","last_undegraded":"2022-09-07T21:38:45.536749+0000","last_fullsized":"2022-09-07T21:38:45.536749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:06:31.660680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974097+0000","last_change":"2022-09-07T21:38:40.578195+0000","last_active":"2022-09-07T21:38:44.974097+0000","last_peered":"2022-09-07T21:38:44.974097+0000","last_clean":"2022-09-07T21:38:44.974097+0000","last_became_active":"2022-09-07T21:38:40.577888+0000","last_became_peered":"2022-09-07T21:38:40.577888+0000","last_unstale":"2022-09-07T21:38:44.974097+0000","last_undegraded":"2022-09-07T21:38:44.974097+0000","last_fullsized":"2022-09-07T21:38:44.974097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:09:25.600697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:06.840520+0000","last_change":"2022-09-07T21:38:39.970599+0000","last_active":"2022-09-07T21:39:06.840520+0000","last_peered":"2022-09-07T21:39:06.840520+0000","last_clean":"2022-09-07T21:39:06.840520+0000","last_became_active":"2022-09-07T21:38:39.970216+0000","last_became_peered":"2022-09-07T21:38:39.970216+0000","last_unstale":"2022-09-07T21:39:06.840520+0000","last_undegraded":"2022-09-07T21:39:06.840520+0000","last_fullsized":"2022-09-07T21:39:06.840520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:01:36.517769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":379,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.981509+0000","last_change":"2022-09-07T21:37:30.699119+0000","last_active":"2022-09-07T21:38:44.981509+0000","last_peered":"2022-09-07T21:38:44.981509+0000","last_clean":"2022-09-07T21:38:44.981509+0000","last_became_active":"2022-09-07T21:37:30.698640+0000","last_became_peered":"2022-09-07T21:37:30.698640+0000","last_unstale":"2022-09-07T21:38:44.981509+0000","last_undegraded":"2022-09-07T21:38:44.981509+0000","last_fullsized":"2022-09-07T21:38:44.981509+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:33:29.392715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:33:29.392715+0000","last_clean_scrub_stamp":"2022-09-07T21:33:29.392715+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:10:27.439063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771114+0000","last_change":"2022-09-07T21:38:44.996255+0000","last_active":"2022-09-07T21:38:45.771114+0000","last_peered":"2022-09-07T21:38:45.771114+0000","last_clean":"2022-09-07T21:38:45.771114+0000","last_became_active":"2022-09-07T21:38:44.995597+0000","last_became_peered":"2022-09-07T21:38:44.995597+0000","last_unstale":"2022-09-07T21:38:45.771114+0000","last_undegraded":"2022-09-07T21:38:45.771114+0000","last_fullsized":"2022-09-07T21:38:45.771114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:59:56.494229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975808+0000","last_change":"2022-09-07T21:38:40.567961+0000","last_active":"2022-09-07T21:38:44.975808+0000","last_peered":"2022-09-07T21:38:44.975808+0000","last_clean":"2022-09-07T21:38:44.975808+0000","last_became_active":"2022-09-07T21:38:40.567436+0000","last_became_peered":"2022-09-07T21:38:40.567436+0000","last_unstale":"2022-09-07T21:38:44.975808+0000","last_undegraded":"2022-09-07T21:38:44.975808+0000","last_fullsized":"2022-09-07T21:38:44.975808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:25:13.025231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.983654+0000","last_change":"2022-09-07T21:38:39.971039+0000","last_active":"2022-09-07T21:38:44.983654+0000","last_peered":"2022-09-07T21:38:44.983654+0000","last_clean":"2022-09-07T21:38:44.983654+0000","last_became_active":"2022-09-07T21:38:39.970675+0000","last_became_peered":"2022-09-07T21:38:39.970675+0000","last_unstale":"2022-09-07T21:38:44.983654+0000","last_undegraded":"2022-09-07T21:38:44.983654+0000","last_fullsized":"2022-09-07T21:38:44.983654+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:36:47.865699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770309+0000","last_change":"2022-09-07T21:38:44.992311+0000","last_active":"2022-09-07T21:38:45.770309+0000","last_peered":"2022-09-07T21:38:45.770309+0000","last_clean":"2022-09-07T21:38:45.770309+0000","last_became_active":"2022-09-07T21:38:44.991846+0000","last_became_peered":"2022-09-07T21:38:44.991846+0000","last_unstale":"2022-09-07T21:38:45.770309+0000","last_undegraded":"2022-09-07T21:38:45.770309+0000","last_fullsized":"2022-09-07T21:38:45.770309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:42:43.968482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770114+0000","last_change":"2022-09-07T21:38:39.970220+0000","last_active":"2022-09-07T21:38:45.770114+0000","last_peered":"2022-09-07T21:38:45.770114+0000","last_clean":"2022-09-07T21:38:45.770114+0000","last_became_active":"2022-09-07T21:38:39.969573+0000","last_became_peered":"2022-09-07T21:38:39.969573+0000","last_unstale":"2022-09-07T21:38:45.770114+0000","last_undegraded":"2022-09-07T21:38:45.770114+0000","last_fullsized":"2022-09-07T21:38:45.770114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:22:05.375623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.982395+0000","last_change":"2022-09-07T21:38:40.585697+0000","last_active":"2022-09-07T21:38:44.982395+0000","last_peered":"2022-09-07T21:38:44.982395+0000","last_clean":"2022-09-07T21:38:44.982395+0000","last_became_active":"2022-09-07T21:38:40.585293+0000","last_became_peered":"2022-09-07T21:38:40.585293+0000","last_unstale":"2022-09-07T21:38:44.982395+0000","last_undegraded":"2022-09-07T21:38:44.982395+0000","last_fullsized":"2022-09-07T21:38:44.982395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:22:21.707916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.769655+0000","last_change":"2022-09-07T21:38:44.985777+0000","last_active":"2022-09-07T21:38:45.769655+0000","last_peered":"2022-09-07T21:38:45.769655+0000","last_clean":"2022-09-07T21:38:45.769655+0000","last_became_active":"2022-09-07T21:38:44.985312+0000","last_became_peered":"2022-09-07T21:38:44.985312+0000","last_unstale":"2022-09-07T21:38:45.769655+0000","last_undegraded":"2022-09-07T21:38:45.769655+0000","last_fullsized":"2022-09-07T21:38:45.769655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:08:04.011094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.971789+0000","last_change":"2022-09-07T21:38:39.973742+0000","last_active":"2022-09-07T21:38:44.971789+0000","last_peered":"2022-09-07T21:38:44.971789+0000","last_clean":"2022-09-07T21:38:44.971789+0000","last_became_active":"2022-09-07T21:38:39.973324+0000","last_became_peered":"2022-09-07T21:38:39.973324+0000","last_unstale":"2022-09-07T21:38:44.971789+0000","last_undegraded":"2022-09-07T21:38:44.971789+0000","last_fullsized":"2022-09-07T21:38:44.971789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:49:54.331302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975725+0000","last_change":"2022-09-07T21:38:40.565122+0000","last_active":"2022-09-07T21:38:44.975725+0000","last_peered":"2022-09-07T21:38:44.975725+0000","last_clean":"2022-09-07T21:38:44.975725+0000","last_became_active":"2022-09-07T21:38:40.564813+0000","last_became_peered":"2022-09-07T21:38:40.564813+0000","last_unstale":"2022-09-07T21:38:44.975725+0000","last_undegraded":"2022-09-07T21:38:44.975725+0000","last_fullsized":"2022-09-07T21:38:44.975725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:03:02.178226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.988379+0000","last_change":"2022-09-07T21:38:44.987707+0000","last_active":"2022-09-07T21:38:44.988379+0000","last_peered":"2022-09-07T21:38:44.988379+0000","last_clean":"2022-09-07T21:38:44.988379+0000","last_became_active":"2022-09-07T21:38:44.987296+0000","last_became_peered":"2022-09-07T21:38:44.987296+0000","last_unstale":"2022-09-07T21:38:44.988379+0000","last_undegraded":"2022-09-07T21:38:44.988379+0000","last_fullsized":"2022-09-07T21:38:44.988379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:53:07.004127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.806531+0000","last_change":"2022-09-07T21:38:39.967664+0000","last_active":"2022-09-07T21:39:04.806531+0000","last_peered":"2022-09-07T21:39:04.806531+0000","last_clean":"2022-09-07T21:39:04.806531+0000","last_became_active":"2022-09-07T21:38:39.967283+0000","last_became_peered":"2022-09-07T21:38:39.967283+0000","last_unstale":"2022-09-07T21:39:04.806531+0000","last_undegraded":"2022-09-07T21:39:04.806531+0000","last_fullsized":"2022-09-07T21:39:04.806531+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:47:31.095692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.970015+0000","last_change":"2022-09-07T21:38:40.579398+0000","last_active":"2022-09-07T21:38:44.970015+0000","last_peered":"2022-09-07T21:38:44.970015+0000","last_clean":"2022-09-07T21:38:44.970015+0000","last_became_active":"2022-09-07T21:38:40.579112+0000","last_became_peered":"2022-09-07T21:38:40.579112+0000","last_unstale":"2022-09-07T21:38:44.970015+0000","last_undegraded":"2022-09-07T21:38:44.970015+0000","last_fullsized":"2022-09-07T21:38:44.970015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:50:13.063073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771478+0000","last_change":"2022-09-07T21:38:44.996219+0000","last_active":"2022-09-07T21:38:45.771478+0000","last_peered":"2022-09-07T21:38:45.771478+0000","last_clean":"2022-09-07T21:38:45.771478+0000","last_became_active":"2022-09-07T21:38:44.995738+0000","last_became_peered":"2022-09-07T21:38:44.995738+0000","last_unstale":"2022-09-07T21:38:45.771478+0000","last_undegraded":"2022-09-07T21:38:45.771478+0000","last_fullsized":"2022-09-07T21:38:45.771478+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:36:49.744364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.808043+0000","last_change":"2022-09-07T21:38:39.965087+0000","last_active":"2022-09-07T21:39:04.808043+0000","last_peered":"2022-09-07T21:39:04.808043+0000","last_clean":"2022-09-07T21:39:04.808043+0000","last_became_active":"2022-09-07T21:38:39.964681+0000","last_became_peered":"2022-09-07T21:38:39.964681+0000","last_unstale":"2022-09-07T21:39:04.808043+0000","last_undegraded":"2022-09-07T21:39:04.808043+0000","last_fullsized":"2022-09-07T21:39:04.808043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:47:50.028036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975103+0000","last_change":"2022-09-07T21:38:40.564639+0000","last_active":"2022-09-07T21:38:44.975103+0000","last_peered":"2022-09-07T21:38:44.975103+0000","last_clean":"2022-09-07T21:38:44.975103+0000","last_became_active":"2022-09-07T21:38:40.564315+0000","last_became_peered":"2022-09-07T21:38:40.564315+0000","last_unstale":"2022-09-07T21:38:44.975103+0000","last_undegraded":"2022-09-07T21:38:44.975103+0000","last_fullsized":"2022-09-07T21:38:44.975103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:20:27.670309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.029920+0000","last_change":"2022-09-07T21:38:44.993278+0000","last_active":"2022-09-07T21:38:45.029920+0000","last_peered":"2022-09-07T21:38:45.029920+0000","last_clean":"2022-09-07T21:38:45.029920+0000","last_became_active":"2022-09-07T21:38:44.987798+0000","last_became_peered":"2022-09-07T21:38:44.987798+0000","last_unstale":"2022-09-07T21:38:45.029920+0000","last_undegraded":"2022-09-07T21:38:45.029920+0000","last_fullsized":"2022-09-07T21:38:45.029920+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:09:21.015154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.811419+0000","last_change":"2022-09-07T21:38:39.967488+0000","last_active":"2022-09-07T21:39:04.811419+0000","last_peered":"2022-09-07T21:39:04.811419+0000","last_clean":"2022-09-07T21:39:04.811419+0000","last_became_active":"2022-09-07T21:38:39.967162+0000","last_became_peered":"2022-09-07T21:38:39.967162+0000","last_unstale":"2022-09-07T21:39:04.811419+0000","last_undegraded":"2022-09-07T21:39:04.811419+0000","last_fullsized":"2022-09-07T21:39:04.811419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:43:43.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.973371+0000","last_change":"2022-09-07T21:38:40.568656+0000","last_active":"2022-09-07T21:38:44.973371+0000","last_peered":"2022-09-07T21:38:44.973371+0000","last_clean":"2022-09-07T21:38:44.973371+0000","last_became_active":"2022-09-07T21:38:40.568357+0000","last_became_peered":"2022-09-07T21:38:40.568357+0000","last_unstale":"2022-09-07T21:38:44.973371+0000","last_undegraded":"2022-09-07T21:38:44.973371+0000","last_fullsized":"2022-09-07T21:38:44.973371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:31:32.498945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770704+0000","last_change":"2022-09-07T21:38:44.995472+0000","last_active":"2022-09-07T21:38:45.770704+0000","last_peered":"2022-09-07T21:38:45.770704+0000","last_clean":"2022-09-07T21:38:45.770704+0000","last_became_active":"2022-09-07T21:38:44.995030+0000","last_became_peered":"2022-09-07T21:38:44.995030+0000","last_unstale":"2022-09-07T21:38:45.770704+0000","last_undegraded":"2022-09-07T21:38:45.770704+0000","last_fullsized":"2022-09-07T21:38:45.770704+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:39:41.557547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.976148+0000","last_change":"2022-09-07T21:38:39.969316+0000","last_active":"2022-09-07T21:38:44.976148+0000","last_peered":"2022-09-07T21:38:44.976148+0000","last_clean":"2022-09-07T21:38:44.976148+0000","last_became_active":"2022-09-07T21:38:39.968958+0000","last_became_peered":"2022-09-07T21:38:39.968958+0000","last_unstale":"2022-09-07T21:38:44.976148+0000","last_undegraded":"2022-09-07T21:38:44.976148+0000","last_fullsized":"2022-09-07T21:38:44.976148+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:51:01.568029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974827+0000","last_change":"2022-09-07T21:38:40.579479+0000","last_active":"2022-09-07T21:38:44.974827+0000","last_peered":"2022-09-07T21:38:44.974827+0000","last_clean":"2022-09-07T21:38:44.974827+0000","last_became_active":"2022-09-07T21:38:40.578452+0000","last_became_peered":"2022-09-07T21:38:40.578452+0000","last_unstale":"2022-09-07T21:38:44.974827+0000","last_undegraded":"2022-09-07T21:38:44.974827+0000","last_fullsized":"2022-09-07T21:38:44.974827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:34:24.096814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771309+0000","last_change":"2022-09-07T21:38:44.988455+0000","last_active":"2022-09-07T21:38:45.771309+0000","last_peered":"2022-09-07T21:38:45.771309+0000","last_clean":"2022-09-07T21:38:45.771309+0000","last_became_active":"2022-09-07T21:38:44.988070+0000","last_became_peered":"2022-09-07T21:38:44.988070+0000","last_unstale":"2022-09-07T21:38:45.771309+0000","last_undegraded":"2022-09-07T21:38:45.771309+0000","last_fullsized":"2022-09-07T21:38:45.771309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:52:37.094892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.806817+0000","last_change":"2022-09-07T21:38:39.972219+0000","last_active":"2022-09-07T21:39:04.806817+0000","last_peered":"2022-09-07T21:39:04.806817+0000","last_clean":"2022-09-07T21:39:04.806817+0000","last_became_active":"2022-09-07T21:38:39.971484+0000","last_became_peered":"2022-09-07T21:38:39.971484+0000","last_unstale":"2022-09-07T21:39:04.806817+0000","last_undegraded":"2022-09-07T21:39:04.806817+0000","last_fullsized":"2022-09-07T21:39:04.806817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:54:17.947316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.979313+0000","last_change":"2022-09-07T21:38:40.578389+0000","last_active":"2022-09-07T21:38:44.979313+0000","last_peered":"2022-09-07T21:38:44.979313+0000","last_clean":"2022-09-07T21:38:44.979313+0000","last_became_active":"2022-09-07T21:38:40.577886+0000","last_became_peered":"2022-09-07T21:38:40.577886+0000","last_unstale":"2022-09-07T21:38:44.979313+0000","last_undegraded":"2022-09-07T21:38:44.979313+0000","last_fullsized":"2022-09-07T21:38:44.979313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:48:07.590464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770599+0000","last_change":"2022-09-07T21:38:44.994968+0000","last_active":"2022-09-07T21:38:45.770599+0000","last_peered":"2022-09-07T21:38:45.770599+0000","last_clean":"2022-09-07T21:38:45.770599+0000","last_became_active":"2022-09-07T21:38:44.994516+0000","last_became_peered":"2022-09-07T21:38:44.994516+0000","last_unstale":"2022-09-07T21:38:45.770599+0000","last_undegraded":"2022-09-07T21:38:45.770599+0000","last_fullsized":"2022-09-07T21:38:45.770599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:49:10.172571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.981453+0000","last_change":"2022-09-07T21:38:39.970117+0000","last_active":"2022-09-07T21:38:44.981453+0000","last_peered":"2022-09-07T21:38:44.981453+0000","last_clean":"2022-09-07T21:38:44.981453+0000","last_became_active":"2022-09-07T21:38:39.966606+0000","last_became_peered":"2022-09-07T21:38:39.966606+0000","last_unstale":"2022-09-07T21:38:44.981453+0000","last_undegraded":"2022-09-07T21:38:44.981453+0000","last_fullsized":"2022-09-07T21:38:44.981453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:59:13.103878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.981182+0000","last_change":"2022-09-07T21:38:40.563232+0000","last_active":"2022-09-07T21:38:44.981182+0000","last_peered":"2022-09-07T21:38:44.981182+0000","last_clean":"2022-09-07T21:38:44.981182+0000","last_became_active":"2022-09-07T21:38:40.562945+0000","last_became_peered":"2022-09-07T21:38:40.562945+0000","last_unstale":"2022-09-07T21:38:44.981182+0000","last_undegraded":"2022-09-07T21:38:44.981182+0000","last_fullsized":"2022-09-07T21:38:44.981182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:42:44.271490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.533534+0000","last_change":"2022-09-07T21:38:44.996941+0000","last_active":"2022-09-07T21:38:45.533534+0000","last_peered":"2022-09-07T21:38:45.533534+0000","last_clean":"2022-09-07T21:38:45.533534+0000","last_became_active":"2022-09-07T21:38:44.988680+0000","last_became_peered":"2022-09-07T21:38:44.988680+0000","last_unstale":"2022-09-07T21:38:45.533534+0000","last_undegraded":"2022-09-07T21:38:45.533534+0000","last_fullsized":"2022-09-07T21:38:45.533534+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:33:12.017654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771465+0000","last_change":"2022-09-07T21:38:39.969588+0000","last_active":"2022-09-07T21:38:45.771465+0000","last_peered":"2022-09-07T21:38:45.771465+0000","last_clean":"2022-09-07T21:38:45.771465+0000","last_became_active":"2022-09-07T21:38:39.969222+0000","last_became_peered":"2022-09-07T21:38:39.969222+0000","last_unstale":"2022-09-07T21:38:45.771465+0000","last_undegraded":"2022-09-07T21:38:45.771465+0000","last_fullsized":"2022-09-07T21:38:45.771465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:46:36.580497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.980314+0000","last_change":"2022-09-07T21:38:40.569663+0000","last_active":"2022-09-07T21:38:44.980314+0000","last_peered":"2022-09-07T21:38:44.980314+0000","last_clean":"2022-09-07T21:38:44.980314+0000","last_became_active":"2022-09-07T21:38:40.569376+0000","last_became_peered":"2022-09-07T21:38:40.569376+0000","last_unstale":"2022-09-07T21:38:44.980314+0000","last_undegraded":"2022-09-07T21:38:44.980314+0000","last_fullsized":"2022-09-07T21:38:44.980314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:06:01.895953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771052+0000","last_change":"2022-09-07T21:38:44.987761+0000","last_active":"2022-09-07T21:38:45.771052+0000","last_peered":"2022-09-07T21:38:45.771052+0000","last_clean":"2022-09-07T21:38:45.771052+0000","last_became_active":"2022-09-07T21:38:44.987376+0000","last_became_peered":"2022-09-07T21:38:44.987376+0000","last_unstale":"2022-09-07T21:38:45.771052+0000","last_undegraded":"2022-09-07T21:38:45.771052+0000","last_fullsized":"2022-09-07T21:38:45.771052+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:37:19.993838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.983087+0000","last_change":"2022-09-07T21:38:39.968703+0000","last_active":"2022-09-07T21:38:44.983087+0000","last_peered":"2022-09-07T21:38:44.983087+0000","last_clean":"2022-09-07T21:38:44.983087+0000","last_became_active":"2022-09-07T21:38:39.968393+0000","last_became_peered":"2022-09-07T21:38:39.968393+0000","last_unstale":"2022-09-07T21:38:44.983087+0000","last_undegraded":"2022-09-07T21:38:44.983087+0000","last_fullsized":"2022-09-07T21:38:44.983087+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:46:36.042794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.980263+0000","last_change":"2022-09-07T21:38:40.567044+0000","last_active":"2022-09-07T21:38:44.980263+0000","last_peered":"2022-09-07T21:38:44.980263+0000","last_clean":"2022-09-07T21:38:44.980263+0000","last_became_active":"2022-09-07T21:38:40.566732+0000","last_became_peered":"2022-09-07T21:38:40.566732+0000","last_unstale":"2022-09-07T21:38:44.980263+0000","last_undegraded":"2022-09-07T21:38:44.980263+0000","last_fullsized":"2022-09-07T21:38:44.980263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:12:52.569238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.973985+0000","last_change":"2022-09-07T21:38:40.563301+0000","last_active":"2022-09-07T21:38:44.973985+0000","last_peered":"2022-09-07T21:38:44.973985+0000","last_clean":"2022-09-07T21:38:44.973985+0000","last_became_active":"2022-09-07T21:38:40.562979+0000","last_became_peered":"2022-09-07T21:38:40.562979+0000","last_unstale":"2022-09-07T21:38:44.973985+0000","last_undegraded":"2022-09-07T21:38:44.973985+0000","last_fullsized":"2022-09-07T21:38:44.973985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:21:46.453495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.976194+0000","last_change":"2022-09-07T21:38:40.565046+0000","last_active":"2022-09-07T21:38:44.976194+0000","last_peered":"2022-09-07T21:38:44.976194+0000","last_clean":"2022-09-07T21:38:44.976194+0000","last_became_active":"2022-09-07T21:38:40.564777+0000","last_became_peered":"2022-09-07T21:38:40.564777+0000","last_unstale":"2022-09-07T21:38:44.976194+0000","last_undegraded":"2022-09-07T21:38:44.976194+0000","last_fullsized":"2022-09-07T21:38:44.976194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:44:16.894070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.976040+0000","last_change":"2022-09-07T21:38:40.576602+0000","last_active":"2022-09-07T21:38:44.976040+0000","last_peered":"2022-09-07T21:38:44.976040+0000","last_clean":"2022-09-07T21:38:44.976040+0000","last_became_active":"2022-09-07T21:38:40.576296+0000","last_became_peered":"2022-09-07T21:38:40.576296+0000","last_unstale":"2022-09-07T21:38:44.976040+0000","last_undegraded":"2022-09-07T21:38:44.976040+0000","last_fullsized":"2022-09-07T21:38:44.976040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:11:43.175430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.969769+0000","last_change":"2022-09-07T21:38:40.584214+0000","last_active":"2022-09-07T21:38:44.969769+0000","last_peered":"2022-09-07T21:38:44.969769+0000","last_clean":"2022-09-07T21:38:44.969769+0000","last_became_active":"2022-09-07T21:38:40.583908+0000","last_became_peered":"2022-09-07T21:38:40.583908+0000","last_unstale":"2022-09-07T21:38:44.969769+0000","last_undegraded":"2022-09-07T21:38:44.969769+0000","last_fullsized":"2022-09-07T21:38:44.969769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:40:54.023025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.982479+0000","last_change":"2022-09-07T21:38:40.571163+0000","last_active":"2022-09-07T21:38:44.982479+0000","last_peered":"2022-09-07T21:38:44.982479+0000","last_clean":"2022-09-07T21:38:44.982479+0000","last_became_active":"2022-09-07T21:38:40.570802+0000","last_became_peered":"2022-09-07T21:38:40.570802+0000","last_unstale":"2022-09-07T21:38:44.982479+0000","last_undegraded":"2022-09-07T21:38:44.982479+0000","last_fullsized":"2022-09-07T21:38:44.982479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:03:14.067136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.973816+0000","last_change":"2022-09-07T21:38:40.561957+0000","last_active":"2022-09-07T21:38:44.973816+0000","last_peered":"2022-09-07T21:38:44.973816+0000","last_clean":"2022-09-07T21:38:44.973816+0000","last_became_active":"2022-09-07T21:38:40.561600+0000","last_became_peered":"2022-09-07T21:38:40.561600+0000","last_unstale":"2022-09-07T21:38:44.973816+0000","last_undegraded":"2022-09-07T21:38:44.973816+0000","last_fullsized":"2022-09-07T21:38:44.973816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:28:37.479711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975919+0000","last_change":"2022-09-07T21:38:40.562779+0000","last_active":"2022-09-07T21:38:44.975919+0000","last_peered":"2022-09-07T21:38:44.975919+0000","last_clean":"2022-09-07T21:38:44.975919+0000","last_became_active":"2022-09-07T21:38:40.562472+0000","last_became_peered":"2022-09-07T21:38:40.562472+0000","last_unstale":"2022-09-07T21:38:44.975919+0000","last_undegraded":"2022-09-07T21:38:44.975919+0000","last_fullsized":"2022-09-07T21:38:44.975919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:13:22.395242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975807+0000","last_change":"2022-09-07T21:38:40.570257+0000","last_active":"2022-09-07T21:38:44.975807+0000","last_peered":"2022-09-07T21:38:44.975807+0000","last_clean":"2022-09-07T21:38:44.975807+0000","last_became_active":"2022-09-07T21:38:40.569930+0000","last_became_peered":"2022-09-07T21:38:40.569930+0000","last_unstale":"2022-09-07T21:38:44.975807+0000","last_undegraded":"2022-09-07T21:38:44.975807+0000","last_fullsized":"2022-09-07T21:38:44.975807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:52:03.810386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.977520+0000","last_change":"2022-09-07T21:38:40.578796+0000","last_active":"2022-09-07T21:38:44.977520+0000","last_peered":"2022-09-07T21:38:44.977520+0000","last_clean":"2022-09-07T21:38:44.977520+0000","last_became_active":"2022-09-07T21:38:40.578492+0000","last_became_peered":"2022-09-07T21:38:40.578492+0000","last_unstale":"2022-09-07T21:38:44.977520+0000","last_undegraded":"2022-09-07T21:38:44.977520+0000","last_fullsized":"2022-09-07T21:38:44.977520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:49:50.549224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974337+0000","last_change":"2022-09-07T21:38:40.571320+0000","last_active":"2022-09-07T21:38:44.974337+0000","last_peered":"2022-09-07T21:38:44.974337+0000","last_clean":"2022-09-07T21:38:44.974337+0000","last_became_active":"2022-09-07T21:38:40.570401+0000","last_became_peered":"2022-09-07T21:38:40.570401+0000","last_unstale":"2022-09-07T21:38:44.974337+0000","last_undegraded":"2022-09-07T21:38:44.974337+0000","last_fullsized":"2022-09-07T21:38:44.974337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:24:33.254833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.982507+0000","last_change":"2022-09-07T21:38:40.571386+0000","last_active":"2022-09-07T21:38:44.982507+0000","last_peered":"2022-09-07T21:38:44.982507+0000","last_clean":"2022-09-07T21:38:44.982507+0000","last_became_active":"2022-09-07T21:38:40.571130+0000","last_became_peered":"2022-09-07T21:38:40.571130+0000","last_unstale":"2022-09-07T21:38:44.982507+0000","last_undegraded":"2022-09-07T21:38:44.982507+0000","last_fullsized":"2022-09-07T21:38:44.982507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:57:47.083547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.984651+0000","last_change":"2022-09-07T21:38:40.584706+0000","last_active":"2022-09-07T21:38:44.984651+0000","last_peered":"2022-09-07T21:38:44.984651+0000","last_clean":"2022-09-07T21:38:44.984651+0000","last_became_active":"2022-09-07T21:38:40.584193+0000","last_became_peered":"2022-09-07T21:38:40.584193+0000","last_unstale":"2022-09-07T21:38:44.984651+0000","last_undegraded":"2022-09-07T21:38:44.984651+0000","last_fullsized":"2022-09-07T21:38:44.984651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:10:32.597608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974513+0000","last_change":"2022-09-07T21:38:40.569545+0000","last_active":"2022-09-07T21:38:44.974513+0000","last_peered":"2022-09-07T21:38:44.974513+0000","last_clean":"2022-09-07T21:38:44.974513+0000","last_became_active":"2022-09-07T21:38:40.569218+0000","last_became_peered":"2022-09-07T21:38:40.569218+0000","last_unstale":"2022-09-07T21:38:44.974513+0000","last_undegraded":"2022-09-07T21:38:44.974513+0000","last_fullsized":"2022-09-07T21:38:44.974513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:40:39.949203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":219246,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":29224,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":19,"num_read_kb":18,"num_write":104,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":62758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6004,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739148,"statfs":{"total":95995035648,"available":95988887552,"internally_reserved":0,"allocated":962560,"data_stored":475487,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56000000000000005}]},{"osd":1,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.755},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.77400000000000002}]},{"osd":2,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.58299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":4,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.71199999999999997}]},{"osd":5,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.68899999999999995}]},{"osd":6,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.83299999999999996}]},{"osd":7,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.85399999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":9,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.97799999999999998}]},{"osd":10,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.89300000000000002}]}]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6352,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738800,"statfs":{"total":95995035648,"available":95988531200,"internally_reserved":0,"allocated":1056768,"data_stored":1071118,"data_compressed":48501,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":1,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.52800000000000002}]},{"osd":2,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.55500000000000005}]},{"osd":3,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.624},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.58299999999999996}]},{"osd":4,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.70199999999999996}]},{"osd":5,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.59399999999999997}]},{"osd":6,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.85699999999999998}]},{"osd":7,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.42399999999999999}]},{"osd":8,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":1.022}]},{"osd":9,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":1.0740000000000001}]},{"osd":11,"last update":"Wed Sep 7 21:38:54 2022","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.92900000000000005}]}]},{"osd":9,"up_from":67,"seq":287762808859,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6212,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738940,"statfs":{"total":95995035648,"available":95988674560,"internally_reserved":0,"allocated":978944,"data_stored":476255,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.69999999999999996}]},{"osd":1,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":2.2490000000000001}]},{"osd":2,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.748},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.499},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.56200000000000006}]},{"osd":4,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.55500000000000005}]},{"osd":5,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.77800000000000002}]},{"osd":6,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":0.81200000000000006}]},{"osd":7,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.8380000000000001,"5min":1.8380000000000001,"15min":1.8380000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":0.67600000000000005}]},{"osd":8,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":2.3010000000000002},{"interface":"front","average":{"1min":1.083,"5min":1.083,"15min":1.083},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":0.81399999999999995}]},{"osd":10,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":2.2669999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3180000000000001}]}]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6336,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":974848,"data_stored":476097,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.622,"15min":0.63200000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59299999999999997,"15min":0.59599999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56200000000000006}]},{"osd":1,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.61699999999999999,"15min":0.627},"min":{"1min":0.36799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78700000000000003,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.61399999999999999,"15min":0.622},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81499999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.78600000000000003}]},{"osd":2,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.48499999999999999,"15min":0.495},"min":{"1min":0.311,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.437,"5min":0.50700000000000001,"15min":0.51900000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.75700000000000001,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.66100000000000003}]},{"osd":3,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63900000000000001,"15min":0.64100000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.65700000000000003,"15min":0.66600000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.65000000000000002}]},{"osd":4,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65200000000000002,"15min":0.66000000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.82799999999999996}]},{"osd":5,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.63500000000000001,"15min":0.65300000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.86599999999999999,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.60199999999999998,"15min":0.61499999999999999},"min":{"1min":0.31900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.83599999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.72799999999999998}]},{"osd":6,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.72999999999999998,"15min":0.74099999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90200000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.77400000000000002,"15min":0.78400000000000003},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.97999999999999998,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.89100000000000001}]},{"osd":7,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.746,"15min":0.76000000000000001},"min":{"1min":0.45400000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.72799999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.93799999999999994}]},{"osd":9,"last update":"Wed Sep 7 21:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.755,"15min":0.748},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.75900000000000001,"15min":0.752},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.92500000000000004}]},{"osd":10,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.879}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]}]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6440,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738712,"statfs":{"total":95995035648,"available":95988441088,"internally_reserved":0,"allocated":1015808,"data_stored":1066421,"data_compressed":48501,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58499999999999996,"15min":0.57299999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.54200000000000004}]},{"osd":1,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.439,"15min":0.433},"min":{"1min":0.33000000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.45300000000000001,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.46999999999999997}]},{"osd":2,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83499999999999996,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.67100000000000004,"15min":0.627},"min":{"1min":0.40899999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.4580000000000002,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":0.59299999999999997}]},{"osd":3,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.629,"15min":0.60899999999999999},"min":{"1min":0.49099999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0149999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64100000000000001,"15min":0.627},"min":{"1min":0.42799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.68400000000000005}]},{"osd":4,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.35999999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.623},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52700000000000002,"15min":0.52800000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79400000000000004,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.58299999999999996}]},{"osd":5,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.66000000000000003,"15min":0.66400000000000003},"min":{"1min":0.48099999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.85399999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.45400000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.873,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.71699999999999997}]},{"osd":6,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73599999999999999,"15min":0.73799999999999999},"min":{"1min":0.50600000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0049999999999999,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.71299999999999997,"15min":0.70099999999999996},"min":{"1min":0.55300000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.73899999999999999}]},{"osd":8,"last update":"Wed Sep 7 21:38:26 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.71299999999999997,"15min":0.71699999999999997},"min":{"1min":0.48499999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72699999999999998,"15min":0.72799999999999998},"min":{"1min":0.503,"5min":0.443,"15min":0.443},"max":{"1min":1.0509999999999999,"5min":1.123,"15min":1.123},"last":0.69799999999999995}]},{"osd":9,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80000000000000004,"15min":0.80300000000000005},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.99399999999999999,"5min":1.036,"15min":1.036},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.83299999999999996,"15min":0.84099999999999997},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0980000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.77800000000000002}]},{"osd":10,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.75700000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]}]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6324,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738828,"statfs":{"total":95995035648,"available":95988559872,"internally_reserved":0,"allocated":962560,"data_stored":475487,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:38 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54700000000000004,"15min":0.53800000000000003},"min":{"1min":0.35699999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81100000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.34599999999999997,"5min":0.308,"15min":0.308},"max":{"1min":0.91900000000000004,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.442}]},{"osd":1,"last update":"Wed Sep 7 21:38:38 2022","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.65500000000000003,"15min":0.66100000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93999999999999995,"5min":1.127,"15min":1.127},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.48899999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.83199999999999996,"5min":1.159,"15min":1.159},"last":0.66900000000000004}]},{"osd":2,"last update":"Wed Sep 7 21:38:38 2022","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71099999999999997,"15min":0.71199999999999997},"min":{"1min":0.50700000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90500000000000003,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69999999999999996,"15min":0.70099999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0289999999999999,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.60599999999999998}]},{"osd":3,"last update":"Wed Sep 7 21:38:38 2022","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.67700000000000005,"15min":0.65700000000000003},"min":{"1min":0.45000000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.123,"5min":1.151,"15min":1.151},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.44400000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.62}]},{"osd":4,"last update":"Wed Sep 7 21:38:38 2022","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.746,"15min":0.754},"min":{"1min":0.54600000000000004,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.86399999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.76600000000000001,"15min":0.78000000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.96799999999999997,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.84599999999999997}]},{"osd":5,"last update":"Wed Sep 7 21:38:38 2022","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.76600000000000001,"15min":0.75800000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94399999999999995,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.82399999999999995,"15min":0.83199999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95399999999999996,"5min":1.341,"15min":1.341},"last":0.64100000000000001}]},{"osd":7,"last update":"Wed Sep 7 21:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.81100000000000005,"15min":0.80700000000000005},"min":{"1min":0.57199999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.139,"5min":1.212,"15min":1.212},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.82599999999999996,"15min":0.81999999999999995},"min":{"1min":0.61599999999999999,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.78600000000000003}]},{"osd":8,"last update":"Wed Sep 7 21:38:28 2022","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.84399999999999997,"15min":0.83499999999999996},"min":{"1min":0.60499999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.107,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.86499999999999999,"15min":0.85399999999999998},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1359999999999999,"5min":1.268,"15min":1.268},"last":0.752}]},{"osd":9,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.96799999999999997,"15min":0.97399999999999998},"min":{"1min":0.77700000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.133,"5min":1.254,"15min":1.254},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.88800000000000001,"15min":0.89000000000000001},"min":{"1min":0.65300000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1439999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.88100000000000001}]},{"osd":10,"last update":"Wed Sep 7 21:38:28 2022","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.82999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7296,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737856,"statfs":{"total":95995035648,"available":95987564544,"internally_reserved":0,"allocated":974848,"data_stored":479236,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:12 2022","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.53600000000000003,"15min":0.50900000000000001},"min":{"1min":0.42799999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81799999999999995,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.54800000000000004,"15min":0.52300000000000002},"min":{"1min":0.35699999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65200000000000002}]},{"osd":2,"last update":"Wed Sep 7 21:38:39 2022","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55500000000000005,"15min":0.52300000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83599999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53500000000000003,"15min":0.51300000000000001},"min":{"1min":0.372,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.874}]},{"osd":3,"last update":"Wed Sep 7 21:38:12 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.56499999999999995,"15min":0.54900000000000004},"min":{"1min":0.433,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.59499999999999997,"15min":0.58399999999999996},"min":{"1min":0.39200000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.504}]},{"osd":4,"last update":"Wed Sep 7 21:38:39 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.45300000000000001,"15min":0.44400000000000001},"min":{"1min":0.31900000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.77000000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.48399999999999999,"15min":0.46300000000000002},"min":{"1min":0.311,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.75600000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.59199999999999997}]},{"osd":5,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.64400000000000002,"15min":0.61099999999999999},"min":{"1min":0.38900000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.63300000000000001,"15min":0.59799999999999998},"min":{"1min":0.51400000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82099999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.81599999999999995}]},{"osd":6,"last update":"Wed Sep 7 21:38:39 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.67700000000000005,"15min":0.66300000000000003},"min":{"1min":0.48999999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.69599999999999995,"15min":0.68700000000000006},"min":{"1min":0.497,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.58399999999999996}]},{"osd":7,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.622,"15min":0.622},"min":{"1min":0.45800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82899999999999996,"5min":1.232,"15min":1.232},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.66100000000000003},"min":{"1min":0.40400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.874,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.67400000000000004}]},{"osd":8,"last update":"Wed Sep 7 21:38:31 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.70999999999999996,"15min":0.69999999999999996},"min":{"1min":0.51400000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.69099999999999995,"15min":0.67700000000000005},"min":{"1min":0.55200000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.83199999999999996}]},{"osd":9,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.57599999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.90200000000000002,"5min":1.032,"15min":1.032},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.77700000000000002,"15min":0.77900000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.91600000000000004,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.747}]},{"osd":10,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79000000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7288,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737864,"statfs":{"total":95995035648,"available":95987572736,"internally_reserved":0,"allocated":966656,"data_stored":475577,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 7 21:38:10 2022","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.59399999999999997,"15min":0.53800000000000003},"min":{"1min":0.39200000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.59599999999999997,"15min":0.55200000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0649999999999999,"5min":1.26,"15min":1.26},"last":0.78500000000000003}]},{"osd":2,"last update":"Wed Sep 7 21:38:31 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.626,"15min":0.60399999999999998},"min":{"1min":0.47699999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.871,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.64000000000000001,"15min":0.61199999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.96099999999999997,"5min":1.075,"15min":1.075},"last":0.58999999999999997}]},{"osd":3,"last update":"Wed Sep 7 21:37:10 2022","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.379,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.53300000000000003,"15min":0.51100000000000001},"min":{"1min":0.32300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.72699999999999998}]},{"osd":4,"last update":"Wed Sep 7 21:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.67600000000000005,"15min":0.65000000000000002},"min":{"1min":0.56100000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.70799999999999996,"15min":0.65900000000000003},"min":{"1min":0.55400000000000005,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.84199999999999997}]},{"osd":5,"last update":"Wed Sep 7 21:38:06 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.72999999999999998,"15min":0.70799999999999996},"min":{"1min":0.64300000000000002,"5min":0.5,"15min":0.5},"max":{"1min":1.1020000000000001,"5min":1.115,"15min":1.115},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.67600000000000005,"15min":0.66100000000000003},"min":{"1min":0.50600000000000001,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0109999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.48299999999999998}]},{"osd":6,"last update":"Wed Sep 7 21:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.71699999999999997,"15min":0.70399999999999996},"min":{"1min":0.48599999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.03,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.754,"15min":0.75800000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0669999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.84799999999999998}]},{"osd":7,"last update":"Wed Sep 7 21:38:02 2022","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.81100000000000005,"15min":0.80200000000000005},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.153,"5min":1.177,"15min":1.177},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.84399999999999997,"15min":0.82299999999999995},"min":{"1min":0.72399999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.871}]},{"osd":8,"last update":"Wed Sep 7 21:38:26 2022","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.89800000000000002,"15min":0.89400000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.875,"5min":0.84899999999999998,"15min":0.84499999999999997},"min":{"1min":0.67500000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.1200000000000001,"5min":1.143,"15min":1.143},"last":0.68500000000000005}]},{"osd":9,"last update":"Wed Sep 7 21:39:03 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.93700000000000006,"15min":0.95099999999999996},"min":{"1min":0.625,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0509999999999999,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.746},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.89700000000000002,"15min":0.90500000000000003},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.88700000000000001}]},{"osd":10,"last update":"Wed Sep 7 21:38:31 2022","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.90400000000000003}]},{"osd":11,"last update":"Wed Sep 7 21:38:51 2022","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.82699999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7300,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737852,"statfs":{"total":95995035648,"available":95987560448,"internally_reserved":0,"allocated":978944,"data_stored":479270,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:42 2022","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53500000000000003,"15min":0.51300000000000001},"min":{"1min":0.38700000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.78900000000000003,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.53600000000000003,"15min":0.52000000000000002},"min":{"1min":0.39000000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.93500000000000005,"5min":0.999,"15min":0.999},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 7 21:38:42 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.42499999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.53300000000000003,"15min":0.495},"min":{"1min":0.40799999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.89400000000000002}]},{"osd":3,"last update":"Wed Sep 7 21:38:11 2022","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.64100000000000001,"15min":0.57299999999999995},"min":{"1min":0.434,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":7.649,"5min":7.649,"15min":7.649},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.56399999999999995,"15min":0.53600000000000003},"min":{"1min":0.442,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.75900000000000001}]},{"osd":4,"last update":"Wed Sep 7 21:37:37 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.53200000000000003,"15min":0.51000000000000001},"min":{"1min":0.34000000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.55400000000000005,"15min":0.52500000000000002},"min":{"1min":0.38100000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.63900000000000001}]},{"osd":5,"last update":"Wed Sep 7 21:38:05 2022","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.442,"15min":0.42599999999999999},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.499,"15min":0.47899999999999998},"min":{"1min":0.34999999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.94299999999999995}]},{"osd":6,"last update":"Wed Sep 7 21:38:42 2022","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.69399999999999995},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.93000000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.55400000000000005,"5min":0.498,"15min":0.498},"max":{"1min":0.90100000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.878}]},{"osd":7,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.60199999999999998,"15min":0.59599999999999997},"min":{"1min":0.42299999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.66100000000000003,"15min":0.65700000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.98199999999999998}]},{"osd":8,"last update":"Wed Sep 7 21:38:30 2022","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.63600000000000001,"15min":0.64300000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64400000000000002,"15min":0.64900000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.73199999999999998}]},{"osd":9,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":1.0109999999999999}]},{"osd":10,"last update":"Wed Sep 7 21:38:30 2022","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":1.024}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182462,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6660,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738492,"statfs":{"total":95995035648,"available":95988215808,"internally_reserved":0,"allocated":978944,"data_stored":475771,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:13 2022","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.47999999999999998,"15min":0.442},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84099999999999997,"5min":1.167,"15min":1.167},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.48499999999999999,"15min":0.45700000000000002},"min":{"1min":0.379,"5min":0.307,"15min":0.307},"max":{"1min":0.876,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.46500000000000002}]},{"osd":1,"last update":"Wed Sep 7 21:38:13 2022","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.45900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.65200000000000002,"15min":0.60099999999999998},"min":{"1min":0.51300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.048,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.79600000000000004}]},{"osd":2,"last update":"Wed Sep 7 21:38:13 2022","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.66700000000000004,"15min":0.63500000000000001},"min":{"1min":0.53800000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.99199999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.64400000000000002,"15min":0.62},"min":{"1min":0.50700000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97699999999999998,"5min":1.302,"15min":1.302},"last":0.78000000000000003}]},{"osd":4,"last update":"Wed Sep 7 21:38:39 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.68600000000000005,"15min":0.64200000000000002},"min":{"1min":0.498,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.92800000000000005,"5min":1.286,"15min":1.286},"last":0.755},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.66600000000000004,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.99399999999999999,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.66100000000000003}]},{"osd":5,"last update":"Wed Sep 7 21:38:07 2022","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.67600000000000005,"15min":0.66400000000000003},"min":{"1min":0.53900000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.125,"5min":1.181,"15min":1.181},"last":0.746},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.75900000000000001,"15min":0.73199999999999998},"min":{"1min":0.505,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.1819999999999999,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.70599999999999996}]},{"osd":6,"last update":"Wed Sep 7 21:38:39 2022","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79500000000000004,"15min":0.78000000000000003},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.995,"5min":1.399,"15min":1.399},"last":0.629},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.74199999999999999,"15min":0.73399999999999999},"min":{"1min":0.51100000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.1259999999999999,"5min":1.157,"15min":1.157},"last":0.51200000000000001}]},{"osd":7,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.73199999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.034,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.874,"15min":0.86299999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.0169999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.82899999999999996}]},{"osd":8,"last update":"Wed Sep 7 21:38:30 2022","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82999999999999996,"15min":0.83099999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81299999999999994,"15min":0.81200000000000006},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.78400000000000003}]},{"osd":9,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.97799999999999998,"15min":0.99199999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.093,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.90400000000000003,"15min":0.91800000000000004},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1539999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.54500000000000004}]},{"osd":10,"last update":"Wed Sep 7 21:38:30 2022","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.86299999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]}]},{"osd":4,"up_from":33,"seq":141733920825,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6528,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738624,"statfs":{"total":95995035648,"available":95988350976,"internally_reserved":0,"allocated":974848,"data_stored":476083,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:37:36 2022","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54400000000000004,"15min":0.52900000000000003},"min":{"1min":0.41199999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78100000000000003,"5min":0.879,"15min":0.879},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.52700000000000002,"15min":0.503},"min":{"1min":0.40699999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.35199999999999998}]},{"osd":1,"last update":"Wed Sep 7 21:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38,"15min":0.35799999999999998},"min":{"1min":0.29199999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54800000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.39200000000000002,"15min":0.372},"min":{"1min":0.27300000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.53100000000000003,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.441}]},{"osd":2,"last update":"Wed Sep 7 21:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.53800000000000003},"min":{"1min":0.378,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77800000000000002,"5min":0.879,"15min":0.879},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56599999999999995,"15min":0.55100000000000005},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82399999999999995,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Sep 7 21:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58399999999999996,"15min":0.56000000000000005},"min":{"1min":0.40200000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.40300000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.79300000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.64800000000000002}]},{"osd":5,"last update":"Wed Sep 7 21:38:06 2022","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81799999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.57799999999999996},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.79600000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.51600000000000001}]},{"osd":6,"last update":"Wed Sep 7 21:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.64600000000000002,"15min":0.64200000000000002},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.95399999999999996,"5min":1.024,"15min":1.024},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64800000000000002,"15min":0.64400000000000002},"min":{"1min":0.45200000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.80400000000000005,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.77500000000000002}]},{"osd":7,"last update":"Wed Sep 7 21:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61699999999999999,"15min":0.61599999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.872,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76300000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.502}]},{"osd":8,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73099999999999998,"15min":0.72999999999999998},"min":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.94899999999999995,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.72299999999999998,"15min":0.72899999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.93200000000000005,"5min":1.198,"15min":1.198},"last":0.56699999999999995}]},{"osd":9,"last update":"Wed Sep 7 21:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72099999999999997,"15min":0.72299999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.78100000000000003,"15min":0.78600000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.048,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.74399999999999999}]},{"osd":10,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.67800000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757301,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6652,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738500,"statfs":{"total":95995035648,"available":95988224000,"internally_reserved":0,"allocated":1036288,"data_stored":1066944,"data_compressed":48501,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77800000000000002,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.56599999999999995,"15min":0.55600000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88400000000000001,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.39400000000000002}]},{"osd":1,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.56399999999999995,"15min":0.57699999999999996},"min":{"1min":0.38300000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.77500000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.39700000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.67100000000000004}]},{"osd":2,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.42999999999999999,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.38},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.28599999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.70899999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.41299999999999998}]},{"osd":3,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80000000000000004,"5min":1.238,"15min":1.238},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61099999999999999,"15min":0.59199999999999997},"min":{"1min":0.437,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.84799999999999998,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.47199999999999998}]},{"osd":4,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60999999999999999,"15min":0.61599999999999999},"min":{"1min":0.40400000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.85999999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.39200000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75700000000000001,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.76200000000000001}]},{"osd":6,"last update":"Wed Sep 7 21:38:40 2022","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64900000000000002,"15min":0.627},"min":{"1min":0.5,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66700000000000004,"15min":0.65900000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80800000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.502}]},{"osd":7,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.64200000000000002,"15min":0.65100000000000002},"min":{"1min":0.46400000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.94699999999999995,"5min":1.002,"15min":1.002},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66600000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.72299999999999998}]},{"osd":8,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.64600000000000002,"15min":0.66200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.748},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.90300000000000002,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.70699999999999996}]},{"osd":9,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.79800000000000004,"15min":0.80900000000000005},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":0.90000000000000002,"5min":0.998,"15min":0.998},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.78100000000000003,"15min":0.78900000000000003},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.89600000000000002,"5min":1.042,"15min":1.042},"last":0.63600000000000001}]},{"osd":10,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.69399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35171,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":25650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":27557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-07T21:39:10.553 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-09-07T21:39:10.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.810+0000 7fa5d0dff700 1 -- 172.21.15.47:0/160810102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5cc0ff920 msgr2=0x7fa5cc1061e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:10.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.810+0000 7fa5d0dff700 1 --2- 172.21.15.47:0/160810102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5cc0ff920 0x7fa5cc1061e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009b00 tx=0x7fa5c4009e10 comp rx=0 tx=0).stop 2022-09-07T21:39:10.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.811+0000 7fa5d0dff700 1 -- 172.21.15.47:0/160810102 shutdown_connections 2022-09-07T21:39:10.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.811+0000 7fa5d0dff700 1 --2- 172.21.15.47:0/160810102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5cc0ff920 0x7fa5cc1061e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.811+0000 7fa5d0dff700 1 --2- 172.21.15.47:0/160810102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5cc0fdd10 0x7fa5cc0fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.811+0000 7fa5d0dff700 1 --2- 172.21.15.47:0/160810102 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc0fef90 0x7fa5cc0ff3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.811+0000 7fa5d0dff700 1 -- 172.21.15.47:0/160810102 >> 172.21.15.47:0/160810102 conn(0x7fa5cc0f93c0 msgr2=0x7fa5cc0fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:10.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.811+0000 7fa5d0dff700 1 -- 172.21.15.47:0/160810102 shutdown_connections 2022-09-07T21:39:10.817 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.811+0000 7fa5d0dff700 1 -- 172.21.15.47:0/160810102 wait complete. 2022-09-07T21:39:10.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.812+0000 7fa5d0dff700 1 Processor -- start 2022-09-07T21:39:10.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.812+0000 7fa5d0dff700 1 -- start start 2022-09-07T21:39:10.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.812+0000 7fa5d0dff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5cc0fdd10 0x7fa5cc072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:10.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.812+0000 7fa5d0dff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5cc0fef90 0x7fa5cc0710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:10.821 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.812+0000 7fa5d0dff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc0ff920 0x7fa5cc071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:10.821 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.812+0000 7fa5d0dff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc071e10 con 0x7fa5cc0fef90 2022-09-07T21:39:10.821 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.812+0000 7fa5d0dff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc071f80 con 0x7fa5cc0ff920 2022-09-07T21:39:10.821 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.812+0000 7fa5d0dff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc19d9b0 con 0x7fa5cc0fdd10 2022-09-07T21:39:10.822 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.813+0000 7fa5c9d9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5cc0fef90 0x7fa5cc0710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:10.822 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.813+0000 7fa5ca59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5cc0fdd10 0x7fa5cc072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:10.822 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.813+0000 7fa5c9d9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5cc0fef90 0x7fa5cc0710d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:41984/0 (socket says 172.21.15.47:41984) 2022-09-07T21:39:10.822 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.813+0000 7fa5cad9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc0ff920 0x7fa5cc071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:10.823 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.813+0000 7fa5c9d9b700 1 -- 172.21.15.47:0/3007108962 learned_addr learned my addr 172.21.15.47:0/3007108962 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:10.823 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.813+0000 7fa5cad9d700 1 -- 172.21.15.47:0/3007108962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5cc0fdd10 msgr2=0x7fa5cc072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:10.823 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.813+0000 7fa5cad9d700 1 --2- 172.21.15.47:0/3007108962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5cc0fdd10 0x7fa5cc072a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.823 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.813+0000 7fa5cad9d700 1 -- 172.21.15.47:0/3007108962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5cc0fef90 msgr2=0x7fa5cc0710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:10.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.813+0000 7fa5cad9d700 1 --2- 172.21.15.47:0/3007108962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5cc0fef90 0x7fa5cc0710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:10.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.813+0000 7fa5cad9d700 1 -- 172.21.15.47:0/3007108962 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c4009730 con 0x7fa5cc0ff920 2022-09-07T21:39:10.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.813+0000 7fa5ca59c700 1 --2- 172.21.15.47:0/3007108962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5cc0fdd10 0x7fa5cc072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:10.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.813+0000 7fa5c9d9b700 1 --2- 172.21.15.47:0/3007108962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5cc0fef90 0x7fa5cc0710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:10.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.814+0000 7fa5cad9d700 1 --2- 172.21.15.47:0/3007108962 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc0ff920 0x7fa5cc071670 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009a20 tx=0x7fa5c400f870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:10.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.814+0000 7fa5bb7fe700 1 -- 172.21.15.47:0/3007108962 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c400b9e0 con 0x7fa5cc0ff920 2022-09-07T21:39:10.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.814+0000 7fa5bb7fe700 1 -- 172.21.15.47:0/3007108962 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa5c4005040 con 0x7fa5cc0ff920 2022-09-07T21:39:10.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.814+0000 7fa5bb7fe700 1 -- 172.21.15.47:0/3007108962 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4017660 con 0x7fa5cc0ff920 2022-09-07T21:39:10.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.814+0000 7fa5d0dff700 1 -- 172.21.15.47:0/3007108962 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5cc19dc30 con 0x7fa5cc0ff920 2022-09-07T21:39:10.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.814+0000 7fa5d0dff700 1 -- 172.21.15.47:0/3007108962 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5cc19e060 con 0x7fa5cc0ff920 2022-09-07T21:39:10.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.815+0000 7fa5d0dff700 1 -- 172.21.15.47:0/3007108962 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5cc1094d0 con 0x7fa5cc0ff920 2022-09-07T21:39:10.827 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.817+0000 7fa5bb7fe700 1 -- 172.21.15.47:0/3007108962 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7fa5c4027020 con 0x7fa5cc0ff920 2022-09-07T21:39:10.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.818+0000 7fa5bb7fe700 1 --2- 172.21.15.47:0/3007108962 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa5b40990c0 0x7fa5b409b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:10.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.819+0000 7fa5bb7fe700 1 -- 172.21.15.47:0/3007108962 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fa5c40c3eb0 con 0x7fa5cc0ff920 2022-09-07T21:39:10.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.819+0000 7fa5ca59c700 1 --2- 172.21.15.47:0/3007108962 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa5b40990c0 0x7fa5b409b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:10.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.820+0000 7fa5ca59c700 1 --2- 172.21.15.47:0/3007108962 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa5b40990c0 0x7fa5b409b570 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc0fedf0 tx=0x7fa5bc005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:10.829 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:10.821+0000 7fa5bb7fe700 1 -- 172.21.15.47:0/3007108962 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fa5c4090310 con 0x7fa5cc0ff920 2022-09-07T21:39:11.026 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:11.023+0000 7fa5d0dff700 1 -- 172.21.15.47:0/3007108962 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa5cc05fbe0 con 0x7fa5b40990c0 2022-09-07T21:39:11.039 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:11.036+0000 7fa5bb7fe700 1 -- 172.21.15.47:0/3007108962 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422583 (secure 0 0 0) 0x7fa5cc05fbe0 con 0x7fa5b40990c0 2022-09-07T21:39:11.046 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:39:11.049 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:11.041+0000 7fa5d0dff700 1 -- 172.21.15.47:0/3007108962 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa5b40990c0 msgr2=0x7fa5b409b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:11.050 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:11.041+0000 7fa5d0dff700 1 --2- 172.21.15.47:0/3007108962 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa5b40990c0 0x7fa5b409b570 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc0fedf0 tx=0x7fa5bc005f60 comp rx=0 tx=0).stop 2022-09-07T21:39:11.050 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:11.041+0000 7fa5d0dff700 1 -- 172.21.15.47:0/3007108962 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc0ff920 msgr2=0x7fa5cc071670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:11.050 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:11.041+0000 7fa5d0dff700 1 --2- 172.21.15.47:0/3007108962 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc0ff920 0x7fa5cc071670 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009a20 tx=0x7fa5c400f870 comp rx=0 tx=0).stop 2022-09-07T21:39:11.050 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:11.041+0000 7fa5d0dff700 1 -- 172.21.15.47:0/3007108962 shutdown_connections 2022-09-07T21:39:11.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:11.041+0000 7fa5d0dff700 1 --2- 172.21.15.47:0/3007108962 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa5b40990c0 0x7fa5b409b570 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:11.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:11.041+0000 7fa5d0dff700 1 --2- 172.21.15.47:0/3007108962 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc0ff920 0x7fa5cc071670 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:11.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:11.041+0000 7fa5d0dff700 1 --2- 172.21.15.47:0/3007108962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5cc0fdd10 0x7fa5cc072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:11.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:11.041+0000 7fa5d0dff700 1 --2- 172.21.15.47:0/3007108962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5cc0fef90 0x7fa5cc0710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:11.052 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:11.041+0000 7fa5d0dff700 1 -- 172.21.15.47:0/3007108962 >> 172.21.15.47:0/3007108962 conn(0x7fa5cc0f93c0 msgr2=0x7fa5cc1024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:11.052 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:11.041+0000 7fa5d0dff700 1 -- 172.21.15.47:0/3007108962 shutdown_connections 2022-09-07T21:39:11.052 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:11.042+0000 7fa5d0dff700 1 -- 172.21.15.47:0/3007108962 wait complete. 2022-09-07T21:39:11.052 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2022-09-07T21:39:11.075 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2022-09-07T21:39:10.594858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":620393,"num_objects":62,"num_object_clones":0,"num_object_copies":187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":163,"num_read_kb":145,"num_write":321,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260,"ondisk_log_size":260,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":79704,"kb_used_data":11704,"kb_used_omap":0,"kb_used_meta":67904,"kb_avail":1124862120,"statfs":{"total":1151940427776,"available":1151858810880,"internally_reserved":0,"allocated":11984896,"data_stored":7551144,"data_compressed":250677,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":0,"internal_metadata":69533696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25635,"num_objects":38,"num_object_clones":0,"num_object_copies":115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":27,"num_read_kb":26,"num_write":92,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003578"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974670+0000","last_change":"2022-09-07T21:38:40.576826+0000","last_active":"2022-09-07T21:38:44.974670+0000","last_peered":"2022-09-07T21:38:44.974670+0000","last_clean":"2022-09-07T21:38:44.974670+0000","last_became_active":"2022-09-07T21:38:40.576429+0000","last_became_peered":"2022-09-07T21:38:40.576429+0000","last_unstale":"2022-09-07T21:38:44.974670+0000","last_undegraded":"2022-09-07T21:38:44.974670+0000","last_fullsized":"2022-09-07T21:38:44.974670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:40:45.777536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.970047+0000","last_change":"2022-09-07T21:38:40.562924+0000","last_active":"2022-09-07T21:38:44.970047+0000","last_peered":"2022-09-07T21:38:44.970047+0000","last_clean":"2022-09-07T21:38:44.970047+0000","last_became_active":"2022-09-07T21:38:40.562509+0000","last_became_peered":"2022-09-07T21:38:40.562509+0000","last_unstale":"2022-09-07T21:38:44.970047+0000","last_undegraded":"2022-09-07T21:38:44.970047+0000","last_fullsized":"2022-09-07T21:38:44.970047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:27:16.781072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.969745+0000","last_change":"2022-09-07T21:38:40.562123+0000","last_active":"2022-09-07T21:38:44.969745+0000","last_peered":"2022-09-07T21:38:44.969745+0000","last_clean":"2022-09-07T21:38:44.969745+0000","last_became_active":"2022-09-07T21:38:40.561632+0000","last_became_peered":"2022-09-07T21:38:40.561632+0000","last_unstale":"2022-09-07T21:38:44.969745+0000","last_undegraded":"2022-09-07T21:38:44.969745+0000","last_fullsized":"2022-09-07T21:38:44.969745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:28:10.763115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.977488+0000","last_change":"2022-09-07T21:38:40.560968+0000","last_active":"2022-09-07T21:38:44.977488+0000","last_peered":"2022-09-07T21:38:44.977488+0000","last_clean":"2022-09-07T21:38:44.977488+0000","last_became_active":"2022-09-07T21:38:40.560685+0000","last_became_peered":"2022-09-07T21:38:40.560685+0000","last_unstale":"2022-09-07T21:38:44.977488+0000","last_undegraded":"2022-09-07T21:38:44.977488+0000","last_fullsized":"2022-09-07T21:38:44.977488+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:49:41.627456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975256+0000","last_change":"2022-09-07T21:38:40.572935+0000","last_active":"2022-09-07T21:38:44.975256+0000","last_peered":"2022-09-07T21:38:44.975256+0000","last_clean":"2022-09-07T21:38:44.975256+0000","last_became_active":"2022-09-07T21:38:40.572605+0000","last_became_peered":"2022-09-07T21:38:40.572605+0000","last_unstale":"2022-09-07T21:38:44.975256+0000","last_undegraded":"2022-09-07T21:38:44.975256+0000","last_fullsized":"2022-09-07T21:38:44.975256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:55:47.267936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.972241+0000","last_change":"2022-09-07T21:38:40.577737+0000","last_active":"2022-09-07T21:38:44.972241+0000","last_peered":"2022-09-07T21:38:44.972241+0000","last_clean":"2022-09-07T21:38:44.972241+0000","last_became_active":"2022-09-07T21:38:40.577425+0000","last_became_peered":"2022-09-07T21:38:40.577425+0000","last_unstale":"2022-09-07T21:38:44.972241+0000","last_undegraded":"2022-09-07T21:38:44.972241+0000","last_fullsized":"2022-09-07T21:38:44.972241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:46:32.369263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.981768+0000","last_change":"2022-09-07T21:38:40.577255+0000","last_active":"2022-09-07T21:38:44.981768+0000","last_peered":"2022-09-07T21:38:44.981768+0000","last_clean":"2022-09-07T21:38:44.981768+0000","last_became_active":"2022-09-07T21:38:40.576750+0000","last_became_peered":"2022-09-07T21:38:40.576750+0000","last_unstale":"2022-09-07T21:38:44.981768+0000","last_undegraded":"2022-09-07T21:38:44.981768+0000","last_fullsized":"2022-09-07T21:38:44.981768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:03:19.663251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974664+0000","last_change":"2022-09-07T21:38:40.577599+0000","last_active":"2022-09-07T21:38:44.974664+0000","last_peered":"2022-09-07T21:38:44.974664+0000","last_clean":"2022-09-07T21:38:44.974664+0000","last_became_active":"2022-09-07T21:38:40.577239+0000","last_became_peered":"2022-09-07T21:38:40.577239+0000","last_unstale":"2022-09-07T21:38:44.974664+0000","last_undegraded":"2022-09-07T21:38:44.974664+0000","last_fullsized":"2022-09-07T21:38:44.974664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:39:23.857698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.980029+0000","last_change":"2022-09-07T21:38:40.559102+0000","last_active":"2022-09-07T21:38:44.980029+0000","last_peered":"2022-09-07T21:38:44.980029+0000","last_clean":"2022-09-07T21:38:44.980029+0000","last_became_active":"2022-09-07T21:38:40.558907+0000","last_became_peered":"2022-09-07T21:38:40.558907+0000","last_unstale":"2022-09-07T21:38:44.980029+0000","last_undegraded":"2022-09-07T21:38:44.980029+0000","last_fullsized":"2022-09-07T21:38:44.980029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:08:03.644593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.972741+0000","last_change":"2022-09-07T21:38:40.566427+0000","last_active":"2022-09-07T21:38:44.972741+0000","last_peered":"2022-09-07T21:38:44.972741+0000","last_clean":"2022-09-07T21:38:44.972741+0000","last_became_active":"2022-09-07T21:38:40.566068+0000","last_became_peered":"2022-09-07T21:38:40.566068+0000","last_unstale":"2022-09-07T21:38:44.972741+0000","last_undegraded":"2022-09-07T21:38:44.972741+0000","last_fullsized":"2022-09-07T21:38:44.972741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:40:01.825654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.981081+0000","last_change":"2022-09-07T21:38:40.568455+0000","last_active":"2022-09-07T21:38:44.981081+0000","last_peered":"2022-09-07T21:38:44.981081+0000","last_clean":"2022-09-07T21:38:44.981081+0000","last_became_active":"2022-09-07T21:38:40.567941+0000","last_became_peered":"2022-09-07T21:38:40.567941+0000","last_unstale":"2022-09-07T21:38:44.981081+0000","last_undegraded":"2022-09-07T21:38:44.981081+0000","last_fullsized":"2022-09-07T21:38:44.981081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:49:16.624335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.977402+0000","last_change":"2022-09-07T21:38:40.563200+0000","last_active":"2022-09-07T21:38:44.977402+0000","last_peered":"2022-09-07T21:38:44.977402+0000","last_clean":"2022-09-07T21:38:44.977402+0000","last_became_active":"2022-09-07T21:38:40.562901+0000","last_became_peered":"2022-09-07T21:38:40.562901+0000","last_unstale":"2022-09-07T21:38:44.977402+0000","last_undegraded":"2022-09-07T21:38:44.977402+0000","last_fullsized":"2022-09-07T21:38:44.977402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:49:26.745940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.984739+0000","last_change":"2022-09-07T21:38:40.570403+0000","last_active":"2022-09-07T21:38:44.984739+0000","last_peered":"2022-09-07T21:38:44.984739+0000","last_clean":"2022-09-07T21:38:44.984739+0000","last_became_active":"2022-09-07T21:38:40.567326+0000","last_became_peered":"2022-09-07T21:38:40.567326+0000","last_unstale":"2022-09-07T21:38:44.984739+0000","last_undegraded":"2022-09-07T21:38:44.984739+0000","last_fullsized":"2022-09-07T21:38:44.984739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:19:31.437519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.969433+0000","last_change":"2022-09-07T21:38:40.562990+0000","last_active":"2022-09-07T21:38:44.969433+0000","last_peered":"2022-09-07T21:38:44.969433+0000","last_clean":"2022-09-07T21:38:44.969433+0000","last_became_active":"2022-09-07T21:38:40.562651+0000","last_became_peered":"2022-09-07T21:38:40.562651+0000","last_unstale":"2022-09-07T21:38:44.969433+0000","last_undegraded":"2022-09-07T21:38:44.969433+0000","last_fullsized":"2022-09-07T21:38:44.969433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:49:07.935459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.969293+0000","last_change":"2022-09-07T21:38:40.570849+0000","last_active":"2022-09-07T21:38:44.969293+0000","last_peered":"2022-09-07T21:38:44.969293+0000","last_clean":"2022-09-07T21:38:44.969293+0000","last_became_active":"2022-09-07T21:38:40.570543+0000","last_became_peered":"2022-09-07T21:38:40.570543+0000","last_unstale":"2022-09-07T21:38:44.969293+0000","last_undegraded":"2022-09-07T21:38:44.969293+0000","last_fullsized":"2022-09-07T21:38:44.969293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:33:30.564522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.977322+0000","last_change":"2022-09-07T21:38:40.568348+0000","last_active":"2022-09-07T21:38:44.977322+0000","last_peered":"2022-09-07T21:38:44.977322+0000","last_clean":"2022-09-07T21:38:44.977322+0000","last_became_active":"2022-09-07T21:38:40.568031+0000","last_became_peered":"2022-09-07T21:38:40.568031+0000","last_unstale":"2022-09-07T21:38:44.977322+0000","last_undegraded":"2022-09-07T21:38:44.977322+0000","last_fullsized":"2022-09-07T21:38:44.977322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:24:21.978277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974421+0000","last_change":"2022-09-07T21:38:40.577289+0000","last_active":"2022-09-07T21:38:44.974421+0000","last_peered":"2022-09-07T21:38:44.974421+0000","last_clean":"2022-09-07T21:38:44.974421+0000","last_became_active":"2022-09-07T21:38:40.576927+0000","last_became_peered":"2022-09-07T21:38:40.576927+0000","last_unstale":"2022-09-07T21:38:44.974421+0000","last_undegraded":"2022-09-07T21:38:44.974421+0000","last_fullsized":"2022-09-07T21:38:44.974421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:02:53.553226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.971786+0000","last_change":"2022-09-07T21:38:40.575886+0000","last_active":"2022-09-07T21:38:44.971786+0000","last_peered":"2022-09-07T21:38:44.971786+0000","last_clean":"2022-09-07T21:38:44.971786+0000","last_became_active":"2022-09-07T21:38:40.575463+0000","last_became_peered":"2022-09-07T21:38:40.575463+0000","last_unstale":"2022-09-07T21:38:44.971786+0000","last_undegraded":"2022-09-07T21:38:44.971786+0000","last_fullsized":"2022-09-07T21:38:44.971786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:10:20.091899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.979794+0000","last_change":"2022-09-07T21:38:40.572579+0000","last_active":"2022-09-07T21:38:44.979794+0000","last_peered":"2022-09-07T21:38:44.979794+0000","last_clean":"2022-09-07T21:38:44.979794+0000","last_became_active":"2022-09-07T21:38:40.572302+0000","last_became_peered":"2022-09-07T21:38:40.572302+0000","last_unstale":"2022-09-07T21:38:44.979794+0000","last_undegraded":"2022-09-07T21:38:44.979794+0000","last_fullsized":"2022-09-07T21:38:44.979794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:41:19.938512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.996782+0000","last_change":"2022-09-07T21:38:44.996073+0000","last_active":"2022-09-07T21:38:44.996782+0000","last_peered":"2022-09-07T21:38:44.996782+0000","last_clean":"2022-09-07T21:38:44.996782+0000","last_became_active":"2022-09-07T21:38:44.995571+0000","last_became_peered":"2022-09-07T21:38:44.995571+0000","last_unstale":"2022-09-07T21:38:44.996782+0000","last_undegraded":"2022-09-07T21:38:44.996782+0000","last_fullsized":"2022-09-07T21:38:44.996782+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:41:58.745823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:05.826471+0000","last_change":"2022-09-07T21:38:39.974715+0000","last_active":"2022-09-07T21:39:05.826471+0000","last_peered":"2022-09-07T21:39:05.826471+0000","last_clean":"2022-09-07T21:39:05.826471+0000","last_became_active":"2022-09-07T21:38:39.974413+0000","last_became_peered":"2022-09-07T21:38:39.974413+0000","last_unstale":"2022-09-07T21:39:05.826471+0000","last_undegraded":"2022-09-07T21:39:05.826471+0000","last_fullsized":"2022-09-07T21:39:05.826471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:33:41.200805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974793+0000","last_change":"2022-09-07T21:38:40.578853+0000","last_active":"2022-09-07T21:38:44.974793+0000","last_peered":"2022-09-07T21:38:44.974793+0000","last_clean":"2022-09-07T21:38:44.974793+0000","last_became_active":"2022-09-07T21:38:40.578229+0000","last_became_peered":"2022-09-07T21:38:40.578229+0000","last_unstale":"2022-09-07T21:38:44.974793+0000","last_undegraded":"2022-09-07T21:38:44.974793+0000","last_fullsized":"2022-09-07T21:38:44.974793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:39:28.259219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771619+0000","last_change":"2022-09-07T21:38:44.995929+0000","last_active":"2022-09-07T21:38:45.771619+0000","last_peered":"2022-09-07T21:38:45.771619+0000","last_clean":"2022-09-07T21:38:45.771619+0000","last_became_active":"2022-09-07T21:38:44.995265+0000","last_became_peered":"2022-09-07T21:38:44.995265+0000","last_unstale":"2022-09-07T21:38:45.771619+0000","last_undegraded":"2022-09-07T21:38:45.771619+0000","last_fullsized":"2022-09-07T21:38:45.771619+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:11:30.103628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771465+0000","last_change":"2022-09-07T21:38:39.971598+0000","last_active":"2022-09-07T21:38:45.771465+0000","last_peered":"2022-09-07T21:38:45.771465+0000","last_clean":"2022-09-07T21:38:45.771465+0000","last_became_active":"2022-09-07T21:38:39.971079+0000","last_became_peered":"2022-09-07T21:38:39.971079+0000","last_unstale":"2022-09-07T21:38:45.771465+0000","last_undegraded":"2022-09-07T21:38:45.771465+0000","last_fullsized":"2022-09-07T21:38:45.771465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:01:22.960635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.969544+0000","last_change":"2022-09-07T21:38:40.577589+0000","last_active":"2022-09-07T21:38:44.969544+0000","last_peered":"2022-09-07T21:38:44.969544+0000","last_clean":"2022-09-07T21:38:44.969544+0000","last_became_active":"2022-09-07T21:38:40.577192+0000","last_became_peered":"2022-09-07T21:38:40.577192+0000","last_unstale":"2022-09-07T21:38:44.969544+0000","last_undegraded":"2022-09-07T21:38:44.969544+0000","last_fullsized":"2022-09-07T21:38:44.969544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:41:03.746467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770985+0000","last_change":"2022-09-07T21:38:44.997384+0000","last_active":"2022-09-07T21:38:45.770985+0000","last_peered":"2022-09-07T21:38:45.770985+0000","last_clean":"2022-09-07T21:38:45.770985+0000","last_became_active":"2022-09-07T21:38:44.996150+0000","last_became_peered":"2022-09-07T21:38:44.996150+0000","last_unstale":"2022-09-07T21:38:45.770985+0000","last_undegraded":"2022-09-07T21:38:45.770985+0000","last_fullsized":"2022-09-07T21:38:45.770985+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:30:05.633718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:05.824297+0000","last_change":"2022-09-07T21:38:39.970057+0000","last_active":"2022-09-07T21:39:05.824297+0000","last_peered":"2022-09-07T21:39:05.824297+0000","last_clean":"2022-09-07T21:39:05.824297+0000","last_became_active":"2022-09-07T21:38:39.967832+0000","last_became_peered":"2022-09-07T21:38:39.967832+0000","last_unstale":"2022-09-07T21:39:05.824297+0000","last_undegraded":"2022-09-07T21:39:05.824297+0000","last_fullsized":"2022-09-07T21:39:05.824297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:33:13.316063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.980694+0000","last_change":"2022-09-07T21:38:40.572694+0000","last_active":"2022-09-07T21:38:44.980694+0000","last_peered":"2022-09-07T21:38:44.980694+0000","last_clean":"2022-09-07T21:38:44.980694+0000","last_became_active":"2022-09-07T21:38:40.572407+0000","last_became_peered":"2022-09-07T21:38:40.572407+0000","last_unstale":"2022-09-07T21:38:44.980694+0000","last_undegraded":"2022-09-07T21:38:44.980694+0000","last_fullsized":"2022-09-07T21:38:44.980694+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:40:17.238549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771762+0000","last_change":"2022-09-07T21:38:44.988445+0000","last_active":"2022-09-07T21:38:45.771762+0000","last_peered":"2022-09-07T21:38:45.771762+0000","last_clean":"2022-09-07T21:38:45.771762+0000","last_became_active":"2022-09-07T21:38:44.987966+0000","last_became_peered":"2022-09-07T21:38:44.987966+0000","last_unstale":"2022-09-07T21:38:45.771762+0000","last_undegraded":"2022-09-07T21:38:45.771762+0000","last_fullsized":"2022-09-07T21:38:45.771762+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:36:16.050494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975449+0000","last_change":"2022-09-07T21:38:39.963456+0000","last_active":"2022-09-07T21:38:44.975449+0000","last_peered":"2022-09-07T21:38:44.975449+0000","last_clean":"2022-09-07T21:38:44.975449+0000","last_became_active":"2022-09-07T21:38:39.963059+0000","last_became_peered":"2022-09-07T21:38:39.963059+0000","last_unstale":"2022-09-07T21:38:44.975449+0000","last_undegraded":"2022-09-07T21:38:44.975449+0000","last_fullsized":"2022-09-07T21:38:44.975449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:51:09.061725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.979895+0000","last_change":"2022-09-07T21:38:40.559585+0000","last_active":"2022-09-07T21:38:44.979895+0000","last_peered":"2022-09-07T21:38:44.979895+0000","last_clean":"2022-09-07T21:38:44.979895+0000","last_became_active":"2022-09-07T21:38:40.559307+0000","last_became_peered":"2022-09-07T21:38:40.559307+0000","last_unstale":"2022-09-07T21:38:44.979895+0000","last_undegraded":"2022-09-07T21:38:44.979895+0000","last_fullsized":"2022-09-07T21:38:44.979895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:20:32.332429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.986450+0000","last_change":"2022-09-07T21:38:44.985596+0000","last_active":"2022-09-07T21:38:44.986450+0000","last_peered":"2022-09-07T21:38:44.986450+0000","last_clean":"2022-09-07T21:38:44.986450+0000","last_became_active":"2022-09-07T21:38:44.985112+0000","last_became_peered":"2022-09-07T21:38:44.985112+0000","last_unstale":"2022-09-07T21:38:44.986450+0000","last_undegraded":"2022-09-07T21:38:44.986450+0000","last_fullsized":"2022-09-07T21:38:44.986450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:24:06.456838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:06.844632+0000","last_change":"2022-09-07T21:38:39.973295+0000","last_active":"2022-09-07T21:39:06.844632+0000","last_peered":"2022-09-07T21:39:06.844632+0000","last_clean":"2022-09-07T21:39:06.844632+0000","last_became_active":"2022-09-07T21:38:39.972932+0000","last_became_peered":"2022-09-07T21:38:39.972932+0000","last_unstale":"2022-09-07T21:39:06.844632+0000","last_undegraded":"2022-09-07T21:39:06.844632+0000","last_fullsized":"2022-09-07T21:39:06.844632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:11:53.533171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974789+0000","last_change":"2022-09-07T21:38:40.577546+0000","last_active":"2022-09-07T21:38:44.974789+0000","last_peered":"2022-09-07T21:38:44.974789+0000","last_clean":"2022-09-07T21:38:44.974789+0000","last_became_active":"2022-09-07T21:38:40.577229+0000","last_became_peered":"2022-09-07T21:38:40.577229+0000","last_unstale":"2022-09-07T21:38:44.974789+0000","last_undegraded":"2022-09-07T21:38:44.974789+0000","last_fullsized":"2022-09-07T21:38:44.974789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:57:36.947488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.995700+0000","last_change":"2022-09-07T21:38:44.993976+0000","last_active":"2022-09-07T21:38:44.995700+0000","last_peered":"2022-09-07T21:38:44.995700+0000","last_clean":"2022-09-07T21:38:44.995700+0000","last_became_active":"2022-09-07T21:38:44.993554+0000","last_became_peered":"2022-09-07T21:38:44.993554+0000","last_unstale":"2022-09-07T21:38:44.995700+0000","last_undegraded":"2022-09-07T21:38:44.995700+0000","last_fullsized":"2022-09-07T21:38:44.995700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:21:13.466677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:05.824799+0000","last_change":"2022-09-07T21:38:39.970361+0000","last_active":"2022-09-07T21:39:05.824799+0000","last_peered":"2022-09-07T21:39:05.824799+0000","last_clean":"2022-09-07T21:39:05.824799+0000","last_became_active":"2022-09-07T21:38:39.969420+0000","last_became_peered":"2022-09-07T21:38:39.969420+0000","last_unstale":"2022-09-07T21:39:05.824799+0000","last_undegraded":"2022-09-07T21:39:05.824799+0000","last_fullsized":"2022-09-07T21:39:05.824799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:53:46.903275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974959+0000","last_change":"2022-09-07T21:38:40.567273+0000","last_active":"2022-09-07T21:38:44.974959+0000","last_peered":"2022-09-07T21:38:44.974959+0000","last_clean":"2022-09-07T21:38:44.974959+0000","last_became_active":"2022-09-07T21:38:40.566942+0000","last_became_peered":"2022-09-07T21:38:40.566942+0000","last_unstale":"2022-09-07T21:38:44.974959+0000","last_undegraded":"2022-09-07T21:38:44.974959+0000","last_fullsized":"2022-09-07T21:38:44.974959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:29:38.009443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770562+0000","last_change":"2022-09-07T21:38:44.998836+0000","last_active":"2022-09-07T21:38:45.770562+0000","last_peered":"2022-09-07T21:38:45.770562+0000","last_clean":"2022-09-07T21:38:45.770562+0000","last_became_active":"2022-09-07T21:38:44.998461+0000","last_became_peered":"2022-09-07T21:38:44.998461+0000","last_unstale":"2022-09-07T21:38:45.770562+0000","last_undegraded":"2022-09-07T21:38:45.770562+0000","last_fullsized":"2022-09-07T21:38:45.770562+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:37:37.203792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:05.826963+0000","last_change":"2022-09-07T21:38:39.970746+0000","last_active":"2022-09-07T21:39:05.826963+0000","last_peered":"2022-09-07T21:39:05.826963+0000","last_clean":"2022-09-07T21:39:05.826963+0000","last_became_active":"2022-09-07T21:38:39.970438+0000","last_became_peered":"2022-09-07T21:38:39.970438+0000","last_unstale":"2022-09-07T21:39:05.826963+0000","last_undegraded":"2022-09-07T21:39:05.826963+0000","last_fullsized":"2022-09-07T21:39:05.826963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:51:07.849231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974556+0000","last_change":"2022-09-07T21:38:40.576245+0000","last_active":"2022-09-07T21:38:44.974556+0000","last_peered":"2022-09-07T21:38:44.974556+0000","last_clean":"2022-09-07T21:38:44.974556+0000","last_became_active":"2022-09-07T21:38:40.575916+0000","last_became_peered":"2022-09-07T21:38:40.575916+0000","last_unstale":"2022-09-07T21:38:44.974556+0000","last_undegraded":"2022-09-07T21:38:44.974556+0000","last_fullsized":"2022-09-07T21:38:44.974556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:35:52.025507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.533897+0000","last_change":"2022-09-07T21:38:44.991494+0000","last_active":"2022-09-07T21:38:45.533897+0000","last_peered":"2022-09-07T21:38:45.533897+0000","last_clean":"2022-09-07T21:38:45.533897+0000","last_became_active":"2022-09-07T21:38:44.988446+0000","last_became_peered":"2022-09-07T21:38:44.988446+0000","last_unstale":"2022-09-07T21:38:45.533897+0000","last_undegraded":"2022-09-07T21:38:45.533897+0000","last_fullsized":"2022-09-07T21:38:45.533897+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:22:52.937965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771325+0000","last_change":"2022-09-07T21:38:39.966535+0000","last_active":"2022-09-07T21:38:45.771325+0000","last_peered":"2022-09-07T21:38:45.771325+0000","last_clean":"2022-09-07T21:38:45.771325+0000","last_became_active":"2022-09-07T21:38:39.965895+0000","last_became_peered":"2022-09-07T21:38:39.965895+0000","last_unstale":"2022-09-07T21:38:45.771325+0000","last_undegraded":"2022-09-07T21:38:45.771325+0000","last_fullsized":"2022-09-07T21:38:45.771325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:16:11.146779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.971237+0000","last_change":"2022-09-07T21:38:40.590979+0000","last_active":"2022-09-07T21:38:44.971237+0000","last_peered":"2022-09-07T21:38:44.971237+0000","last_clean":"2022-09-07T21:38:44.971237+0000","last_became_active":"2022-09-07T21:38:40.590024+0000","last_became_peered":"2022-09-07T21:38:40.590024+0000","last_unstale":"2022-09-07T21:38:44.971237+0000","last_undegraded":"2022-09-07T21:38:44.971237+0000","last_fullsized":"2022-09-07T21:38:44.971237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:58:11.726364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770147+0000","last_change":"2022-09-07T21:38:44.988252+0000","last_active":"2022-09-07T21:38:45.770147+0000","last_peered":"2022-09-07T21:38:45.770147+0000","last_clean":"2022-09-07T21:38:45.770147+0000","last_became_active":"2022-09-07T21:38:44.987851+0000","last_became_peered":"2022-09-07T21:38:44.987851+0000","last_unstale":"2022-09-07T21:38:45.770147+0000","last_undegraded":"2022-09-07T21:38:45.770147+0000","last_fullsized":"2022-09-07T21:38:45.770147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:53:42.207358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.808370+0000","last_change":"2022-09-07T21:38:39.969970+0000","last_active":"2022-09-07T21:39:04.808370+0000","last_peered":"2022-09-07T21:39:04.808370+0000","last_clean":"2022-09-07T21:39:04.808370+0000","last_became_active":"2022-09-07T21:38:39.969627+0000","last_became_peered":"2022-09-07T21:38:39.969627+0000","last_unstale":"2022-09-07T21:39:04.808370+0000","last_undegraded":"2022-09-07T21:39:04.808370+0000","last_fullsized":"2022-09-07T21:39:04.808370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:14:11.990138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.980466+0000","last_change":"2022-09-07T21:38:40.577175+0000","last_active":"2022-09-07T21:38:44.980466+0000","last_peered":"2022-09-07T21:38:44.980466+0000","last_clean":"2022-09-07T21:38:44.980466+0000","last_became_active":"2022-09-07T21:38:40.576873+0000","last_became_peered":"2022-09-07T21:38:40.576873+0000","last_unstale":"2022-09-07T21:38:44.980466+0000","last_undegraded":"2022-09-07T21:38:44.980466+0000","last_fullsized":"2022-09-07T21:38:44.980466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:25:17.911241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.029838+0000","last_change":"2022-09-07T21:38:44.984456+0000","last_active":"2022-09-07T21:38:45.029838+0000","last_peered":"2022-09-07T21:38:45.029838+0000","last_clean":"2022-09-07T21:38:45.029838+0000","last_became_active":"2022-09-07T21:38:44.984070+0000","last_became_peered":"2022-09-07T21:38:44.984070+0000","last_unstale":"2022-09-07T21:38:45.029838+0000","last_undegraded":"2022-09-07T21:38:45.029838+0000","last_fullsized":"2022-09-07T21:38:45.029838+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:36:14.150189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.806795+0000","last_change":"2022-09-07T21:38:39.975863+0000","last_active":"2022-09-07T21:39:04.806795+0000","last_peered":"2022-09-07T21:39:04.806795+0000","last_clean":"2022-09-07T21:39:04.806795+0000","last_became_active":"2022-09-07T21:38:39.975490+0000","last_became_peered":"2022-09-07T21:38:39.975490+0000","last_unstale":"2022-09-07T21:39:04.806795+0000","last_undegraded":"2022-09-07T21:39:04.806795+0000","last_fullsized":"2022-09-07T21:39:04.806795+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:46:54.710925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.969816+0000","last_change":"2022-09-07T21:38:40.571594+0000","last_active":"2022-09-07T21:38:44.969816+0000","last_peered":"2022-09-07T21:38:44.969816+0000","last_clean":"2022-09-07T21:38:44.969816+0000","last_became_active":"2022-09-07T21:38:40.571188+0000","last_became_peered":"2022-09-07T21:38:40.571188+0000","last_unstale":"2022-09-07T21:38:44.969816+0000","last_undegraded":"2022-09-07T21:38:44.969816+0000","last_fullsized":"2022-09-07T21:38:44.969816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:25:49.474602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771857+0000","last_change":"2022-09-07T21:38:44.996134+0000","last_active":"2022-09-07T21:38:45.771857+0000","last_peered":"2022-09-07T21:38:45.771857+0000","last_clean":"2022-09-07T21:38:45.771857+0000","last_became_active":"2022-09-07T21:38:44.995696+0000","last_became_peered":"2022-09-07T21:38:44.995696+0000","last_unstale":"2022-09-07T21:38:45.771857+0000","last_undegraded":"2022-09-07T21:38:45.771857+0000","last_fullsized":"2022-09-07T21:38:45.771857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:52:03.648431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.809728+0000","last_change":"2022-09-07T21:38:39.964670+0000","last_active":"2022-09-07T21:39:04.809728+0000","last_peered":"2022-09-07T21:39:04.809728+0000","last_clean":"2022-09-07T21:39:04.809728+0000","last_became_active":"2022-09-07T21:38:39.964305+0000","last_became_peered":"2022-09-07T21:38:39.964305+0000","last_unstale":"2022-09-07T21:39:04.809728+0000","last_undegraded":"2022-09-07T21:39:04.809728+0000","last_fullsized":"2022-09-07T21:39:04.809728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:15:57.041815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.980818+0000","last_change":"2022-09-07T21:38:40.565156+0000","last_active":"2022-09-07T21:38:44.980818+0000","last_peered":"2022-09-07T21:38:44.980818+0000","last_clean":"2022-09-07T21:38:44.980818+0000","last_became_active":"2022-09-07T21:38:40.564843+0000","last_became_peered":"2022-09-07T21:38:40.564843+0000","last_unstale":"2022-09-07T21:38:44.980818+0000","last_undegraded":"2022-09-07T21:38:44.980818+0000","last_fullsized":"2022-09-07T21:38:44.980818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:11:09.741599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.029862+0000","last_change":"2022-09-07T21:38:44.986167+0000","last_active":"2022-09-07T21:38:45.029862+0000","last_peered":"2022-09-07T21:38:45.029862+0000","last_clean":"2022-09-07T21:38:45.029862+0000","last_became_active":"2022-09-07T21:38:44.985778+0000","last_became_peered":"2022-09-07T21:38:44.985778+0000","last_unstale":"2022-09-07T21:38:45.029862+0000","last_undegraded":"2022-09-07T21:38:45.029862+0000","last_fullsized":"2022-09-07T21:38:45.029862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:21:01.441205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'19","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:07.408184+0000","last_change":"2022-09-07T21:38:39.968660+0000","last_active":"2022-09-07T21:39:07.408184+0000","last_peered":"2022-09-07T21:39:07.408184+0000","last_clean":"2022-09-07T21:39:07.408184+0000","last_became_active":"2022-09-07T21:38:39.968285+0000","last_became_peered":"2022-09-07T21:38:39.968285+0000","last_unstale":"2022-09-07T21:39:07.408184+0000","last_undegraded":"2022-09-07T21:39:07.408184+0000","last_fullsized":"2022-09-07T21:39:07.408184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:16:16.027615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25650,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":20,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974554+0000","last_change":"2022-09-07T21:38:40.566145+0000","last_active":"2022-09-07T21:38:44.974554+0000","last_peered":"2022-09-07T21:38:44.974554+0000","last_clean":"2022-09-07T21:38:44.974554+0000","last_became_active":"2022-09-07T21:38:40.565823+0000","last_became_peered":"2022-09-07T21:38:40.565823+0000","last_unstale":"2022-09-07T21:38:44.974554+0000","last_undegraded":"2022-09-07T21:38:44.974554+0000","last_fullsized":"2022-09-07T21:38:44.974554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:13:28.826066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771597+0000","last_change":"2022-09-07T21:38:44.995599+0000","last_active":"2022-09-07T21:38:45.771597+0000","last_peered":"2022-09-07T21:38:45.771597+0000","last_clean":"2022-09-07T21:38:45.771597+0000","last_became_active":"2022-09-07T21:38:44.994627+0000","last_became_peered":"2022-09-07T21:38:44.994627+0000","last_unstale":"2022-09-07T21:38:45.771597+0000","last_undegraded":"2022-09-07T21:38:45.771597+0000","last_fullsized":"2022-09-07T21:38:45.771597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:17:30.608595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:05.826546+0000","last_change":"2022-09-07T21:38:39.968773+0000","last_active":"2022-09-07T21:39:05.826546+0000","last_peered":"2022-09-07T21:39:05.826546+0000","last_clean":"2022-09-07T21:39:05.826546+0000","last_became_active":"2022-09-07T21:38:39.967912+0000","last_became_peered":"2022-09-07T21:38:39.967912+0000","last_unstale":"2022-09-07T21:39:05.826546+0000","last_undegraded":"2022-09-07T21:39:05.826546+0000","last_fullsized":"2022-09-07T21:39:05.826546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:51:12.940327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.976273+0000","last_change":"2022-09-07T21:38:40.573548+0000","last_active":"2022-09-07T21:38:44.976273+0000","last_peered":"2022-09-07T21:38:44.976273+0000","last_clean":"2022-09-07T21:38:44.976273+0000","last_became_active":"2022-09-07T21:38:40.573273+0000","last_became_peered":"2022-09-07T21:38:40.573273+0000","last_unstale":"2022-09-07T21:38:44.976273+0000","last_undegraded":"2022-09-07T21:38:44.976273+0000","last_fullsized":"2022-09-07T21:38:44.976273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:03:52.778107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770487+0000","last_change":"2022-09-07T21:38:44.987366+0000","last_active":"2022-09-07T21:38:45.770487+0000","last_peered":"2022-09-07T21:38:45.770487+0000","last_clean":"2022-09-07T21:38:45.770487+0000","last_became_active":"2022-09-07T21:38:44.986775+0000","last_became_peered":"2022-09-07T21:38:44.986775+0000","last_unstale":"2022-09-07T21:38:45.770487+0000","last_undegraded":"2022-09-07T21:38:45.770487+0000","last_fullsized":"2022-09-07T21:38:45.770487+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:07:14.746521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.809894+0000","last_change":"2022-09-07T21:38:39.969123+0000","last_active":"2022-09-07T21:39:04.809894+0000","last_peered":"2022-09-07T21:39:04.809894+0000","last_clean":"2022-09-07T21:39:04.809894+0000","last_became_active":"2022-09-07T21:38:39.968744+0000","last_became_peered":"2022-09-07T21:38:39.968744+0000","last_unstale":"2022-09-07T21:39:04.809894+0000","last_undegraded":"2022-09-07T21:39:04.809894+0000","last_fullsized":"2022-09-07T21:39:04.809894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:54:34.616856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.977308+0000","last_change":"2022-09-07T21:38:40.578056+0000","last_active":"2022-09-07T21:38:44.977308+0000","last_peered":"2022-09-07T21:38:44.977308+0000","last_clean":"2022-09-07T21:38:44.977308+0000","last_became_active":"2022-09-07T21:38:40.577727+0000","last_became_peered":"2022-09-07T21:38:40.577727+0000","last_unstale":"2022-09-07T21:38:44.977308+0000","last_undegraded":"2022-09-07T21:38:44.977308+0000","last_fullsized":"2022-09-07T21:38:44.977308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:10:08.645724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.029910+0000","last_change":"2022-09-07T21:38:44.993544+0000","last_active":"2022-09-07T21:38:45.029910+0000","last_peered":"2022-09-07T21:38:45.029910+0000","last_clean":"2022-09-07T21:38:45.029910+0000","last_became_active":"2022-09-07T21:38:44.986520+0000","last_became_peered":"2022-09-07T21:38:44.986520+0000","last_unstale":"2022-09-07T21:38:45.029910+0000","last_undegraded":"2022-09-07T21:38:45.029910+0000","last_fullsized":"2022-09-07T21:38:45.029910+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:54:18.417488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:07.847298+0000","last_change":"2022-09-07T21:38:39.962760+0000","last_active":"2022-09-07T21:39:07.847298+0000","last_peered":"2022-09-07T21:39:07.847298+0000","last_clean":"2022-09-07T21:39:07.847298+0000","last_became_active":"2022-09-07T21:38:39.962327+0000","last_became_peered":"2022-09-07T21:38:39.962327+0000","last_unstale":"2022-09-07T21:39:07.847298+0000","last_undegraded":"2022-09-07T21:39:07.847298+0000","last_fullsized":"2022-09-07T21:39:07.847298+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:17:27.180736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.983608+0000","last_change":"2022-09-07T21:38:40.569532+0000","last_active":"2022-09-07T21:38:44.983608+0000","last_peered":"2022-09-07T21:38:44.983608+0000","last_clean":"2022-09-07T21:38:44.983608+0000","last_became_active":"2022-09-07T21:38:40.569037+0000","last_became_peered":"2022-09-07T21:38:40.569037+0000","last_unstale":"2022-09-07T21:38:44.983608+0000","last_undegraded":"2022-09-07T21:38:44.983608+0000","last_fullsized":"2022-09-07T21:38:44.983608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:37:05.114735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770904+0000","last_change":"2022-09-07T21:38:44.995323+0000","last_active":"2022-09-07T21:38:45.770904+0000","last_peered":"2022-09-07T21:38:45.770904+0000","last_clean":"2022-09-07T21:38:45.770904+0000","last_became_active":"2022-09-07T21:38:44.994915+0000","last_became_peered":"2022-09-07T21:38:44.994915+0000","last_unstale":"2022-09-07T21:38:45.770904+0000","last_undegraded":"2022-09-07T21:38:45.770904+0000","last_fullsized":"2022-09-07T21:38:45.770904+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:17:10.419180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:05.825783+0000","last_change":"2022-09-07T21:38:39.967903+0000","last_active":"2022-09-07T21:39:05.825783+0000","last_peered":"2022-09-07T21:39:05.825783+0000","last_clean":"2022-09-07T21:39:05.825783+0000","last_became_active":"2022-09-07T21:38:39.967392+0000","last_became_peered":"2022-09-07T21:38:39.967392+0000","last_unstale":"2022-09-07T21:39:05.825783+0000","last_undegraded":"2022-09-07T21:39:05.825783+0000","last_fullsized":"2022-09-07T21:39:05.825783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:31:39.581561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.976347+0000","last_change":"2022-09-07T21:38:40.566956+0000","last_active":"2022-09-07T21:38:44.976347+0000","last_peered":"2022-09-07T21:38:44.976347+0000","last_clean":"2022-09-07T21:38:44.976347+0000","last_became_active":"2022-09-07T21:38:40.566655+0000","last_became_peered":"2022-09-07T21:38:40.566655+0000","last_unstale":"2022-09-07T21:38:44.976347+0000","last_undegraded":"2022-09-07T21:38:44.976347+0000","last_fullsized":"2022-09-07T21:38:44.976347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:37:16.353162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771274+0000","last_change":"2022-09-07T21:38:44.988605+0000","last_active":"2022-09-07T21:38:45.771274+0000","last_peered":"2022-09-07T21:38:45.771274+0000","last_clean":"2022-09-07T21:38:45.771274+0000","last_became_active":"2022-09-07T21:38:44.988082+0000","last_became_peered":"2022-09-07T21:38:44.988082+0000","last_unstale":"2022-09-07T21:38:45.771274+0000","last_undegraded":"2022-09-07T21:38:45.771274+0000","last_fullsized":"2022-09-07T21:38:45.771274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:30:06.975077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:06.842401+0000","last_change":"2022-09-07T21:38:39.961740+0000","last_active":"2022-09-07T21:39:06.842401+0000","last_peered":"2022-09-07T21:39:06.842401+0000","last_clean":"2022-09-07T21:39:06.842401+0000","last_became_active":"2022-09-07T21:38:39.961219+0000","last_became_peered":"2022-09-07T21:38:39.961219+0000","last_unstale":"2022-09-07T21:39:06.842401+0000","last_undegraded":"2022-09-07T21:39:06.842401+0000","last_fullsized":"2022-09-07T21:39:06.842401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:51:24.649401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.973894+0000","last_change":"2022-09-07T21:38:40.569594+0000","last_active":"2022-09-07T21:38:44.973894+0000","last_peered":"2022-09-07T21:38:44.973894+0000","last_clean":"2022-09-07T21:38:44.973894+0000","last_became_active":"2022-09-07T21:38:40.569251+0000","last_became_peered":"2022-09-07T21:38:40.569251+0000","last_unstale":"2022-09-07T21:38:44.973894+0000","last_undegraded":"2022-09-07T21:38:44.973894+0000","last_fullsized":"2022-09-07T21:38:44.973894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:37:43.337560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.392327+0000","last_change":"2022-09-07T21:38:44.993534+0000","last_active":"2022-09-07T21:38:45.392327+0000","last_peered":"2022-09-07T21:38:45.392327+0000","last_clean":"2022-09-07T21:38:45.392327+0000","last_became_active":"2022-09-07T21:38:44.992966+0000","last_became_peered":"2022-09-07T21:38:44.992966+0000","last_unstale":"2022-09-07T21:38:45.392327+0000","last_undegraded":"2022-09-07T21:38:45.392327+0000","last_fullsized":"2022-09-07T21:38:45.392327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:12:27.991839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.971397+0000","last_change":"2022-09-07T21:38:39.978436+0000","last_active":"2022-09-07T21:38:44.971397+0000","last_peered":"2022-09-07T21:38:44.971397+0000","last_clean":"2022-09-07T21:38:44.971397+0000","last_became_active":"2022-09-07T21:38:39.977613+0000","last_became_peered":"2022-09-07T21:38:39.977613+0000","last_unstale":"2022-09-07T21:38:44.971397+0000","last_undegraded":"2022-09-07T21:38:44.971397+0000","last_fullsized":"2022-09-07T21:38:44.971397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:23:08.765467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.972585+0000","last_change":"2022-09-07T21:38:40.577968+0000","last_active":"2022-09-07T21:38:44.972585+0000","last_peered":"2022-09-07T21:38:44.972585+0000","last_clean":"2022-09-07T21:38:44.972585+0000","last_became_active":"2022-09-07T21:38:40.577632+0000","last_became_peered":"2022-09-07T21:38:40.577632+0000","last_unstale":"2022-09-07T21:38:44.972585+0000","last_undegraded":"2022-09-07T21:38:44.972585+0000","last_fullsized":"2022-09-07T21:38:44.972585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:57:31.599202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770320+0000","last_change":"2022-09-07T21:38:44.989341+0000","last_active":"2022-09-07T21:38:45.770320+0000","last_peered":"2022-09-07T21:38:45.770320+0000","last_clean":"2022-09-07T21:38:45.770320+0000","last_became_active":"2022-09-07T21:38:44.988977+0000","last_became_peered":"2022-09-07T21:38:44.988977+0000","last_unstale":"2022-09-07T21:38:45.770320+0000","last_undegraded":"2022-09-07T21:38:45.770320+0000","last_fullsized":"2022-09-07T21:38:45.770320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:34:52.397025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.536381+0000","last_change":"2022-09-07T21:38:39.970365+0000","last_active":"2022-09-07T21:38:45.536381+0000","last_peered":"2022-09-07T21:38:45.536381+0000","last_clean":"2022-09-07T21:38:45.536381+0000","last_became_active":"2022-09-07T21:38:39.970044+0000","last_became_peered":"2022-09-07T21:38:39.970044+0000","last_unstale":"2022-09-07T21:38:45.536381+0000","last_undegraded":"2022-09-07T21:38:45.536381+0000","last_fullsized":"2022-09-07T21:38:45.536381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:39:04.408591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.972235+0000","last_change":"2022-09-07T21:38:40.564714+0000","last_active":"2022-09-07T21:38:44.972235+0000","last_peered":"2022-09-07T21:38:44.972235+0000","last_clean":"2022-09-07T21:38:44.972235+0000","last_became_active":"2022-09-07T21:38:40.563744+0000","last_became_peered":"2022-09-07T21:38:40.563744+0000","last_unstale":"2022-09-07T21:38:44.972235+0000","last_undegraded":"2022-09-07T21:38:44.972235+0000","last_fullsized":"2022-09-07T21:38:44.972235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:46:29.965821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.769987+0000","last_change":"2022-09-07T21:38:44.999889+0000","last_active":"2022-09-07T21:38:45.769987+0000","last_peered":"2022-09-07T21:38:45.769987+0000","last_clean":"2022-09-07T21:38:45.769987+0000","last_became_active":"2022-09-07T21:38:44.999453+0000","last_became_peered":"2022-09-07T21:38:44.999453+0000","last_unstale":"2022-09-07T21:38:45.769987+0000","last_undegraded":"2022-09-07T21:38:45.769987+0000","last_fullsized":"2022-09-07T21:38:45.769987+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:26:24.497901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.977617+0000","last_change":"2022-09-07T21:38:40.567008+0000","last_active":"2022-09-07T21:38:44.977617+0000","last_peered":"2022-09-07T21:38:44.977617+0000","last_clean":"2022-09-07T21:38:44.977617+0000","last_became_active":"2022-09-07T21:38:40.566598+0000","last_became_peered":"2022-09-07T21:38:40.566598+0000","last_unstale":"2022-09-07T21:38:44.977617+0000","last_undegraded":"2022-09-07T21:38:44.977617+0000","last_fullsized":"2022-09-07T21:38:44.977617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:06:16.999794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:05.826220+0000","last_change":"2022-09-07T21:38:39.966446+0000","last_active":"2022-09-07T21:39:05.826220+0000","last_peered":"2022-09-07T21:39:05.826220+0000","last_clean":"2022-09-07T21:39:05.826220+0000","last_became_active":"2022-09-07T21:38:39.966095+0000","last_became_peered":"2022-09-07T21:38:39.966095+0000","last_unstale":"2022-09-07T21:39:05.826220+0000","last_undegraded":"2022-09-07T21:39:05.826220+0000","last_fullsized":"2022-09-07T21:39:05.826220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:08:54.360894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:48.691696+0000","last_change":"2022-09-07T21:38:44.993551+0000","last_active":"2022-09-07T21:38:48.691696+0000","last_peered":"2022-09-07T21:38:48.691696+0000","last_clean":"2022-09-07T21:38:48.691696+0000","last_became_active":"2022-09-07T21:38:44.993235+0000","last_became_peered":"2022-09-07T21:38:44.993235+0000","last_unstale":"2022-09-07T21:38:48.691696+0000","last_undegraded":"2022-09-07T21:38:48.691696+0000","last_fullsized":"2022-09-07T21:38:48.691696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:06:31.660680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974097+0000","last_change":"2022-09-07T21:38:40.578195+0000","last_active":"2022-09-07T21:38:44.974097+0000","last_peered":"2022-09-07T21:38:44.974097+0000","last_clean":"2022-09-07T21:38:44.974097+0000","last_became_active":"2022-09-07T21:38:40.577888+0000","last_became_peered":"2022-09-07T21:38:40.577888+0000","last_unstale":"2022-09-07T21:38:44.974097+0000","last_undegraded":"2022-09-07T21:38:44.974097+0000","last_fullsized":"2022-09-07T21:38:44.974097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:09:25.600697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:06.840520+0000","last_change":"2022-09-07T21:38:39.970599+0000","last_active":"2022-09-07T21:39:06.840520+0000","last_peered":"2022-09-07T21:39:06.840520+0000","last_clean":"2022-09-07T21:39:06.840520+0000","last_became_active":"2022-09-07T21:38:39.970216+0000","last_became_peered":"2022-09-07T21:38:39.970216+0000","last_unstale":"2022-09-07T21:39:06.840520+0000","last_undegraded":"2022-09-07T21:39:06.840520+0000","last_fullsized":"2022-09-07T21:39:06.840520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:01:36.517769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":379,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.981509+0000","last_change":"2022-09-07T21:37:30.699119+0000","last_active":"2022-09-07T21:38:44.981509+0000","last_peered":"2022-09-07T21:38:44.981509+0000","last_clean":"2022-09-07T21:38:44.981509+0000","last_became_active":"2022-09-07T21:37:30.698640+0000","last_became_peered":"2022-09-07T21:37:30.698640+0000","last_unstale":"2022-09-07T21:38:44.981509+0000","last_undegraded":"2022-09-07T21:38:44.981509+0000","last_fullsized":"2022-09-07T21:38:44.981509+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:33:29.392715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:33:29.392715+0000","last_clean_scrub_stamp":"2022-09-07T21:33:29.392715+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:10:27.439063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771114+0000","last_change":"2022-09-07T21:38:44.996255+0000","last_active":"2022-09-07T21:38:45.771114+0000","last_peered":"2022-09-07T21:38:45.771114+0000","last_clean":"2022-09-07T21:38:45.771114+0000","last_became_active":"2022-09-07T21:38:44.995597+0000","last_became_peered":"2022-09-07T21:38:44.995597+0000","last_unstale":"2022-09-07T21:38:45.771114+0000","last_undegraded":"2022-09-07T21:38:45.771114+0000","last_fullsized":"2022-09-07T21:38:45.771114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:59:56.494229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975808+0000","last_change":"2022-09-07T21:38:40.567961+0000","last_active":"2022-09-07T21:38:44.975808+0000","last_peered":"2022-09-07T21:38:44.975808+0000","last_clean":"2022-09-07T21:38:44.975808+0000","last_became_active":"2022-09-07T21:38:40.567436+0000","last_became_peered":"2022-09-07T21:38:40.567436+0000","last_unstale":"2022-09-07T21:38:44.975808+0000","last_undegraded":"2022-09-07T21:38:44.975808+0000","last_fullsized":"2022-09-07T21:38:44.975808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:25:13.025231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:05.822977+0000","last_change":"2022-09-07T21:38:39.971039+0000","last_active":"2022-09-07T21:39:05.822977+0000","last_peered":"2022-09-07T21:39:05.822977+0000","last_clean":"2022-09-07T21:39:05.822977+0000","last_became_active":"2022-09-07T21:38:39.970675+0000","last_became_peered":"2022-09-07T21:38:39.970675+0000","last_unstale":"2022-09-07T21:39:05.822977+0000","last_undegraded":"2022-09-07T21:39:05.822977+0000","last_fullsized":"2022-09-07T21:39:05.822977+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:36:47.865699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770309+0000","last_change":"2022-09-07T21:38:44.992311+0000","last_active":"2022-09-07T21:38:45.770309+0000","last_peered":"2022-09-07T21:38:45.770309+0000","last_clean":"2022-09-07T21:38:45.770309+0000","last_became_active":"2022-09-07T21:38:44.991846+0000","last_became_peered":"2022-09-07T21:38:44.991846+0000","last_unstale":"2022-09-07T21:38:45.770309+0000","last_undegraded":"2022-09-07T21:38:45.770309+0000","last_fullsized":"2022-09-07T21:38:45.770309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:42:43.968482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.806340+0000","last_change":"2022-09-07T21:38:39.970220+0000","last_active":"2022-09-07T21:39:04.806340+0000","last_peered":"2022-09-07T21:39:04.806340+0000","last_clean":"2022-09-07T21:39:04.806340+0000","last_became_active":"2022-09-07T21:38:39.969573+0000","last_became_peered":"2022-09-07T21:38:39.969573+0000","last_unstale":"2022-09-07T21:39:04.806340+0000","last_undegraded":"2022-09-07T21:39:04.806340+0000","last_fullsized":"2022-09-07T21:39:04.806340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:22:05.375623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.982395+0000","last_change":"2022-09-07T21:38:40.585697+0000","last_active":"2022-09-07T21:38:44.982395+0000","last_peered":"2022-09-07T21:38:44.982395+0000","last_clean":"2022-09-07T21:38:44.982395+0000","last_became_active":"2022-09-07T21:38:40.585293+0000","last_became_peered":"2022-09-07T21:38:40.585293+0000","last_unstale":"2022-09-07T21:38:44.982395+0000","last_undegraded":"2022-09-07T21:38:44.982395+0000","last_fullsized":"2022-09-07T21:38:44.982395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:22:21.707916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.769655+0000","last_change":"2022-09-07T21:38:44.985777+0000","last_active":"2022-09-07T21:38:45.769655+0000","last_peered":"2022-09-07T21:38:45.769655+0000","last_clean":"2022-09-07T21:38:45.769655+0000","last_became_active":"2022-09-07T21:38:44.985312+0000","last_became_peered":"2022-09-07T21:38:44.985312+0000","last_unstale":"2022-09-07T21:38:45.769655+0000","last_undegraded":"2022-09-07T21:38:45.769655+0000","last_fullsized":"2022-09-07T21:38:45.769655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:08:04.011094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'4","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:05.826695+0000","last_change":"2022-09-07T21:38:39.973742+0000","last_active":"2022-09-07T21:39:05.826695+0000","last_peered":"2022-09-07T21:39:05.826695+0000","last_clean":"2022-09-07T21:39:05.826695+0000","last_became_active":"2022-09-07T21:38:39.973324+0000","last_became_peered":"2022-09-07T21:38:39.973324+0000","last_unstale":"2022-09-07T21:39:05.826695+0000","last_undegraded":"2022-09-07T21:39:05.826695+0000","last_fullsized":"2022-09-07T21:39:05.826695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:49:54.331302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975725+0000","last_change":"2022-09-07T21:38:40.565122+0000","last_active":"2022-09-07T21:38:44.975725+0000","last_peered":"2022-09-07T21:38:44.975725+0000","last_clean":"2022-09-07T21:38:44.975725+0000","last_became_active":"2022-09-07T21:38:40.564813+0000","last_became_peered":"2022-09-07T21:38:40.564813+0000","last_unstale":"2022-09-07T21:38:44.975725+0000","last_undegraded":"2022-09-07T21:38:44.975725+0000","last_fullsized":"2022-09-07T21:38:44.975725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:03:02.178226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.988379+0000","last_change":"2022-09-07T21:38:44.987707+0000","last_active":"2022-09-07T21:38:44.988379+0000","last_peered":"2022-09-07T21:38:44.988379+0000","last_clean":"2022-09-07T21:38:44.988379+0000","last_became_active":"2022-09-07T21:38:44.987296+0000","last_became_peered":"2022-09-07T21:38:44.987296+0000","last_unstale":"2022-09-07T21:38:44.988379+0000","last_undegraded":"2022-09-07T21:38:44.988379+0000","last_fullsized":"2022-09-07T21:38:44.988379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:53:07.004127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.806531+0000","last_change":"2022-09-07T21:38:39.967664+0000","last_active":"2022-09-07T21:39:04.806531+0000","last_peered":"2022-09-07T21:39:04.806531+0000","last_clean":"2022-09-07T21:39:04.806531+0000","last_became_active":"2022-09-07T21:38:39.967283+0000","last_became_peered":"2022-09-07T21:38:39.967283+0000","last_unstale":"2022-09-07T21:39:04.806531+0000","last_undegraded":"2022-09-07T21:39:04.806531+0000","last_fullsized":"2022-09-07T21:39:04.806531+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:47:31.095692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.970015+0000","last_change":"2022-09-07T21:38:40.579398+0000","last_active":"2022-09-07T21:38:44.970015+0000","last_peered":"2022-09-07T21:38:44.970015+0000","last_clean":"2022-09-07T21:38:44.970015+0000","last_became_active":"2022-09-07T21:38:40.579112+0000","last_became_peered":"2022-09-07T21:38:40.579112+0000","last_unstale":"2022-09-07T21:38:44.970015+0000","last_undegraded":"2022-09-07T21:38:44.970015+0000","last_fullsized":"2022-09-07T21:38:44.970015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:50:13.063073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771478+0000","last_change":"2022-09-07T21:38:44.996219+0000","last_active":"2022-09-07T21:38:45.771478+0000","last_peered":"2022-09-07T21:38:45.771478+0000","last_clean":"2022-09-07T21:38:45.771478+0000","last_became_active":"2022-09-07T21:38:44.995738+0000","last_became_peered":"2022-09-07T21:38:44.995738+0000","last_unstale":"2022-09-07T21:38:45.771478+0000","last_undegraded":"2022-09-07T21:38:45.771478+0000","last_fullsized":"2022-09-07T21:38:45.771478+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:36:49.744364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.808043+0000","last_change":"2022-09-07T21:38:39.965087+0000","last_active":"2022-09-07T21:39:04.808043+0000","last_peered":"2022-09-07T21:39:04.808043+0000","last_clean":"2022-09-07T21:39:04.808043+0000","last_became_active":"2022-09-07T21:38:39.964681+0000","last_became_peered":"2022-09-07T21:38:39.964681+0000","last_unstale":"2022-09-07T21:39:04.808043+0000","last_undegraded":"2022-09-07T21:39:04.808043+0000","last_fullsized":"2022-09-07T21:39:04.808043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:47:50.028036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975103+0000","last_change":"2022-09-07T21:38:40.564639+0000","last_active":"2022-09-07T21:38:44.975103+0000","last_peered":"2022-09-07T21:38:44.975103+0000","last_clean":"2022-09-07T21:38:44.975103+0000","last_became_active":"2022-09-07T21:38:40.564315+0000","last_became_peered":"2022-09-07T21:38:40.564315+0000","last_unstale":"2022-09-07T21:38:44.975103+0000","last_undegraded":"2022-09-07T21:38:44.975103+0000","last_fullsized":"2022-09-07T21:38:44.975103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:20:27.670309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.029920+0000","last_change":"2022-09-07T21:38:44.993278+0000","last_active":"2022-09-07T21:38:45.029920+0000","last_peered":"2022-09-07T21:38:45.029920+0000","last_clean":"2022-09-07T21:38:45.029920+0000","last_became_active":"2022-09-07T21:38:44.987798+0000","last_became_peered":"2022-09-07T21:38:44.987798+0000","last_unstale":"2022-09-07T21:38:45.029920+0000","last_undegraded":"2022-09-07T21:38:45.029920+0000","last_fullsized":"2022-09-07T21:38:45.029920+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:09:21.015154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.811419+0000","last_change":"2022-09-07T21:38:39.967488+0000","last_active":"2022-09-07T21:39:04.811419+0000","last_peered":"2022-09-07T21:39:04.811419+0000","last_clean":"2022-09-07T21:39:04.811419+0000","last_became_active":"2022-09-07T21:38:39.967162+0000","last_became_peered":"2022-09-07T21:38:39.967162+0000","last_unstale":"2022-09-07T21:39:04.811419+0000","last_undegraded":"2022-09-07T21:39:04.811419+0000","last_fullsized":"2022-09-07T21:39:04.811419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:43:43.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.973371+0000","last_change":"2022-09-07T21:38:40.568656+0000","last_active":"2022-09-07T21:38:44.973371+0000","last_peered":"2022-09-07T21:38:44.973371+0000","last_clean":"2022-09-07T21:38:44.973371+0000","last_became_active":"2022-09-07T21:38:40.568357+0000","last_became_peered":"2022-09-07T21:38:40.568357+0000","last_unstale":"2022-09-07T21:38:44.973371+0000","last_undegraded":"2022-09-07T21:38:44.973371+0000","last_fullsized":"2022-09-07T21:38:44.973371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:31:32.498945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770704+0000","last_change":"2022-09-07T21:38:44.995472+0000","last_active":"2022-09-07T21:38:45.770704+0000","last_peered":"2022-09-07T21:38:45.770704+0000","last_clean":"2022-09-07T21:38:45.770704+0000","last_became_active":"2022-09-07T21:38:44.995030+0000","last_became_peered":"2022-09-07T21:38:44.995030+0000","last_unstale":"2022-09-07T21:38:45.770704+0000","last_undegraded":"2022-09-07T21:38:45.770704+0000","last_fullsized":"2022-09-07T21:38:45.770704+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:39:41.557547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.976148+0000","last_change":"2022-09-07T21:38:39.969316+0000","last_active":"2022-09-07T21:38:44.976148+0000","last_peered":"2022-09-07T21:38:44.976148+0000","last_clean":"2022-09-07T21:38:44.976148+0000","last_became_active":"2022-09-07T21:38:39.968958+0000","last_became_peered":"2022-09-07T21:38:39.968958+0000","last_unstale":"2022-09-07T21:38:44.976148+0000","last_undegraded":"2022-09-07T21:38:44.976148+0000","last_fullsized":"2022-09-07T21:38:44.976148+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:51:01.568029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974827+0000","last_change":"2022-09-07T21:38:40.579479+0000","last_active":"2022-09-07T21:38:44.974827+0000","last_peered":"2022-09-07T21:38:44.974827+0000","last_clean":"2022-09-07T21:38:44.974827+0000","last_became_active":"2022-09-07T21:38:40.578452+0000","last_became_peered":"2022-09-07T21:38:40.578452+0000","last_unstale":"2022-09-07T21:38:44.974827+0000","last_undegraded":"2022-09-07T21:38:44.974827+0000","last_fullsized":"2022-09-07T21:38:44.974827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:34:24.096814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771309+0000","last_change":"2022-09-07T21:38:44.988455+0000","last_active":"2022-09-07T21:38:45.771309+0000","last_peered":"2022-09-07T21:38:45.771309+0000","last_clean":"2022-09-07T21:38:45.771309+0000","last_became_active":"2022-09-07T21:38:44.988070+0000","last_became_peered":"2022-09-07T21:38:44.988070+0000","last_unstale":"2022-09-07T21:38:45.771309+0000","last_undegraded":"2022-09-07T21:38:45.771309+0000","last_fullsized":"2022-09-07T21:38:45.771309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:52:37.094892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:04.806982+0000","last_change":"2022-09-07T21:38:39.972219+0000","last_active":"2022-09-07T21:39:04.806982+0000","last_peered":"2022-09-07T21:39:04.806982+0000","last_clean":"2022-09-07T21:39:04.806982+0000","last_became_active":"2022-09-07T21:38:39.971484+0000","last_became_peered":"2022-09-07T21:38:39.971484+0000","last_unstale":"2022-09-07T21:39:04.806982+0000","last_undegraded":"2022-09-07T21:39:04.806982+0000","last_fullsized":"2022-09-07T21:39:04.806982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:54:17.947316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.979313+0000","last_change":"2022-09-07T21:38:40.578389+0000","last_active":"2022-09-07T21:38:44.979313+0000","last_peered":"2022-09-07T21:38:44.979313+0000","last_clean":"2022-09-07T21:38:44.979313+0000","last_became_active":"2022-09-07T21:38:40.577886+0000","last_became_peered":"2022-09-07T21:38:40.577886+0000","last_unstale":"2022-09-07T21:38:44.979313+0000","last_undegraded":"2022-09-07T21:38:44.979313+0000","last_fullsized":"2022-09-07T21:38:44.979313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:48:07.590464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.770599+0000","last_change":"2022-09-07T21:38:44.994968+0000","last_active":"2022-09-07T21:38:45.770599+0000","last_peered":"2022-09-07T21:38:45.770599+0000","last_clean":"2022-09-07T21:38:45.770599+0000","last_became_active":"2022-09-07T21:38:44.994516+0000","last_became_peered":"2022-09-07T21:38:44.994516+0000","last_unstale":"2022-09-07T21:38:45.770599+0000","last_undegraded":"2022-09-07T21:38:45.770599+0000","last_fullsized":"2022-09-07T21:38:45.770599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:49:10.172571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:39:05.826218+0000","last_change":"2022-09-07T21:38:39.970117+0000","last_active":"2022-09-07T21:39:05.826218+0000","last_peered":"2022-09-07T21:39:05.826218+0000","last_clean":"2022-09-07T21:39:05.826218+0000","last_became_active":"2022-09-07T21:38:39.966606+0000","last_became_peered":"2022-09-07T21:38:39.966606+0000","last_unstale":"2022-09-07T21:39:05.826218+0000","last_undegraded":"2022-09-07T21:39:05.826218+0000","last_fullsized":"2022-09-07T21:39:05.826218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:59:13.103878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.981182+0000","last_change":"2022-09-07T21:38:40.563232+0000","last_active":"2022-09-07T21:38:44.981182+0000","last_peered":"2022-09-07T21:38:44.981182+0000","last_clean":"2022-09-07T21:38:44.981182+0000","last_became_active":"2022-09-07T21:38:40.562945+0000","last_became_peered":"2022-09-07T21:38:40.562945+0000","last_unstale":"2022-09-07T21:38:44.981182+0000","last_undegraded":"2022-09-07T21:38:44.981182+0000","last_fullsized":"2022-09-07T21:38:44.981182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:42:44.271490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.533534+0000","last_change":"2022-09-07T21:38:44.996941+0000","last_active":"2022-09-07T21:38:45.533534+0000","last_peered":"2022-09-07T21:38:45.533534+0000","last_clean":"2022-09-07T21:38:45.533534+0000","last_became_active":"2022-09-07T21:38:44.988680+0000","last_became_peered":"2022-09-07T21:38:44.988680+0000","last_unstale":"2022-09-07T21:38:45.533534+0000","last_undegraded":"2022-09-07T21:38:45.533534+0000","last_fullsized":"2022-09-07T21:38:45.533534+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:33:12.017654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771465+0000","last_change":"2022-09-07T21:38:39.969588+0000","last_active":"2022-09-07T21:38:45.771465+0000","last_peered":"2022-09-07T21:38:45.771465+0000","last_clean":"2022-09-07T21:38:45.771465+0000","last_became_active":"2022-09-07T21:38:39.969222+0000","last_became_peered":"2022-09-07T21:38:39.969222+0000","last_unstale":"2022-09-07T21:38:45.771465+0000","last_undegraded":"2022-09-07T21:38:45.771465+0000","last_fullsized":"2022-09-07T21:38:45.771465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:46:36.580497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.980314+0000","last_change":"2022-09-07T21:38:40.569663+0000","last_active":"2022-09-07T21:38:44.980314+0000","last_peered":"2022-09-07T21:38:44.980314+0000","last_clean":"2022-09-07T21:38:44.980314+0000","last_became_active":"2022-09-07T21:38:40.569376+0000","last_became_peered":"2022-09-07T21:38:40.569376+0000","last_unstale":"2022-09-07T21:38:44.980314+0000","last_undegraded":"2022-09-07T21:38:44.980314+0000","last_fullsized":"2022-09-07T21:38:44.980314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:06:01.895953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-07T21:38:45.771052+0000","last_change":"2022-09-07T21:38:44.987761+0000","last_active":"2022-09-07T21:38:45.771052+0000","last_peered":"2022-09-07T21:38:45.771052+0000","last_clean":"2022-09-07T21:38:45.771052+0000","last_became_active":"2022-09-07T21:38:44.987376+0000","last_became_peered":"2022-09-07T21:38:44.987376+0000","last_unstale":"2022-09-07T21:38:45.771052+0000","last_undegraded":"2022-09-07T21:38:45.771052+0000","last_fullsized":"2022-09-07T21:38:45.771052+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:37:19.993838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.983087+0000","last_change":"2022-09-07T21:38:39.968703+0000","last_active":"2022-09-07T21:38:44.983087+0000","last_peered":"2022-09-07T21:38:44.983087+0000","last_clean":"2022-09-07T21:38:44.983087+0000","last_became_active":"2022-09-07T21:38:39.968393+0000","last_became_peered":"2022-09-07T21:38:39.968393+0000","last_unstale":"2022-09-07T21:38:44.983087+0000","last_undegraded":"2022-09-07T21:38:44.983087+0000","last_fullsized":"2022-09-07T21:38:44.983087+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:46:36.042794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.980263+0000","last_change":"2022-09-07T21:38:40.567044+0000","last_active":"2022-09-07T21:38:44.980263+0000","last_peered":"2022-09-07T21:38:44.980263+0000","last_clean":"2022-09-07T21:38:44.980263+0000","last_became_active":"2022-09-07T21:38:40.566732+0000","last_became_peered":"2022-09-07T21:38:40.566732+0000","last_unstale":"2022-09-07T21:38:44.980263+0000","last_undegraded":"2022-09-07T21:38:44.980263+0000","last_fullsized":"2022-09-07T21:38:44.980263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:12:52.569238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.973985+0000","last_change":"2022-09-07T21:38:40.563301+0000","last_active":"2022-09-07T21:38:44.973985+0000","last_peered":"2022-09-07T21:38:44.973985+0000","last_clean":"2022-09-07T21:38:44.973985+0000","last_became_active":"2022-09-07T21:38:40.562979+0000","last_became_peered":"2022-09-07T21:38:40.562979+0000","last_unstale":"2022-09-07T21:38:44.973985+0000","last_undegraded":"2022-09-07T21:38:44.973985+0000","last_fullsized":"2022-09-07T21:38:44.973985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:21:46.453495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.976194+0000","last_change":"2022-09-07T21:38:40.565046+0000","last_active":"2022-09-07T21:38:44.976194+0000","last_peered":"2022-09-07T21:38:44.976194+0000","last_clean":"2022-09-07T21:38:44.976194+0000","last_became_active":"2022-09-07T21:38:40.564777+0000","last_became_peered":"2022-09-07T21:38:40.564777+0000","last_unstale":"2022-09-07T21:38:44.976194+0000","last_undegraded":"2022-09-07T21:38:44.976194+0000","last_fullsized":"2022-09-07T21:38:44.976194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:44:16.894070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.976040+0000","last_change":"2022-09-07T21:38:40.576602+0000","last_active":"2022-09-07T21:38:44.976040+0000","last_peered":"2022-09-07T21:38:44.976040+0000","last_clean":"2022-09-07T21:38:44.976040+0000","last_became_active":"2022-09-07T21:38:40.576296+0000","last_became_peered":"2022-09-07T21:38:40.576296+0000","last_unstale":"2022-09-07T21:38:44.976040+0000","last_undegraded":"2022-09-07T21:38:44.976040+0000","last_fullsized":"2022-09-07T21:38:44.976040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:11:43.175430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.969769+0000","last_change":"2022-09-07T21:38:40.584214+0000","last_active":"2022-09-07T21:38:44.969769+0000","last_peered":"2022-09-07T21:38:44.969769+0000","last_clean":"2022-09-07T21:38:44.969769+0000","last_became_active":"2022-09-07T21:38:40.583908+0000","last_became_peered":"2022-09-07T21:38:40.583908+0000","last_unstale":"2022-09-07T21:38:44.969769+0000","last_undegraded":"2022-09-07T21:38:44.969769+0000","last_fullsized":"2022-09-07T21:38:44.969769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:40:54.023025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.982479+0000","last_change":"2022-09-07T21:38:40.571163+0000","last_active":"2022-09-07T21:38:44.982479+0000","last_peered":"2022-09-07T21:38:44.982479+0000","last_clean":"2022-09-07T21:38:44.982479+0000","last_became_active":"2022-09-07T21:38:40.570802+0000","last_became_peered":"2022-09-07T21:38:40.570802+0000","last_unstale":"2022-09-07T21:38:44.982479+0000","last_undegraded":"2022-09-07T21:38:44.982479+0000","last_fullsized":"2022-09-07T21:38:44.982479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:03:14.067136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.973816+0000","last_change":"2022-09-07T21:38:40.561957+0000","last_active":"2022-09-07T21:38:44.973816+0000","last_peered":"2022-09-07T21:38:44.973816+0000","last_clean":"2022-09-07T21:38:44.973816+0000","last_became_active":"2022-09-07T21:38:40.561600+0000","last_became_peered":"2022-09-07T21:38:40.561600+0000","last_unstale":"2022-09-07T21:38:44.973816+0000","last_undegraded":"2022-09-07T21:38:44.973816+0000","last_fullsized":"2022-09-07T21:38:44.973816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:28:37.479711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975919+0000","last_change":"2022-09-07T21:38:40.562779+0000","last_active":"2022-09-07T21:38:44.975919+0000","last_peered":"2022-09-07T21:38:44.975919+0000","last_clean":"2022-09-07T21:38:44.975919+0000","last_became_active":"2022-09-07T21:38:40.562472+0000","last_became_peered":"2022-09-07T21:38:40.562472+0000","last_unstale":"2022-09-07T21:38:44.975919+0000","last_undegraded":"2022-09-07T21:38:44.975919+0000","last_fullsized":"2022-09-07T21:38:44.975919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:13:22.395242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.975807+0000","last_change":"2022-09-07T21:38:40.570257+0000","last_active":"2022-09-07T21:38:44.975807+0000","last_peered":"2022-09-07T21:38:44.975807+0000","last_clean":"2022-09-07T21:38:44.975807+0000","last_became_active":"2022-09-07T21:38:40.569930+0000","last_became_peered":"2022-09-07T21:38:40.569930+0000","last_unstale":"2022-09-07T21:38:44.975807+0000","last_undegraded":"2022-09-07T21:38:44.975807+0000","last_fullsized":"2022-09-07T21:38:44.975807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:52:03.810386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.977520+0000","last_change":"2022-09-07T21:38:40.578796+0000","last_active":"2022-09-07T21:38:44.977520+0000","last_peered":"2022-09-07T21:38:44.977520+0000","last_clean":"2022-09-07T21:38:44.977520+0000","last_became_active":"2022-09-07T21:38:40.578492+0000","last_became_peered":"2022-09-07T21:38:40.578492+0000","last_unstale":"2022-09-07T21:38:44.977520+0000","last_undegraded":"2022-09-07T21:38:44.977520+0000","last_fullsized":"2022-09-07T21:38:44.977520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:49:50.549224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974337+0000","last_change":"2022-09-07T21:38:40.571320+0000","last_active":"2022-09-07T21:38:44.974337+0000","last_peered":"2022-09-07T21:38:44.974337+0000","last_clean":"2022-09-07T21:38:44.974337+0000","last_became_active":"2022-09-07T21:38:40.570401+0000","last_became_peered":"2022-09-07T21:38:40.570401+0000","last_unstale":"2022-09-07T21:38:44.974337+0000","last_undegraded":"2022-09-07T21:38:44.974337+0000","last_fullsized":"2022-09-07T21:38:44.974337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:24:33.254833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.982507+0000","last_change":"2022-09-07T21:38:40.571386+0000","last_active":"2022-09-07T21:38:44.982507+0000","last_peered":"2022-09-07T21:38:44.982507+0000","last_clean":"2022-09-07T21:38:44.982507+0000","last_became_active":"2022-09-07T21:38:40.571130+0000","last_became_peered":"2022-09-07T21:38:40.571130+0000","last_unstale":"2022-09-07T21:38:44.982507+0000","last_undegraded":"2022-09-07T21:38:44.982507+0000","last_fullsized":"2022-09-07T21:38:44.982507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:57:47.083547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.984651+0000","last_change":"2022-09-07T21:38:40.584706+0000","last_active":"2022-09-07T21:38:44.984651+0000","last_peered":"2022-09-07T21:38:44.984651+0000","last_clean":"2022-09-07T21:38:44.984651+0000","last_became_active":"2022-09-07T21:38:40.584193+0000","last_became_peered":"2022-09-07T21:38:40.584193+0000","last_unstale":"2022-09-07T21:38:44.984651+0000","last_undegraded":"2022-09-07T21:38:44.984651+0000","last_fullsized":"2022-09-07T21:38:44.984651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:10:32.597608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2022-09-07T21:38:44.974513+0000","last_change":"2022-09-07T21:38:40.569545+0000","last_active":"2022-09-07T21:38:44.974513+0000","last_peered":"2022-09-07T21:38:44.974513+0000","last_clean":"2022-09-07T21:38:44.974513+0000","last_became_active":"2022-09-07T21:38:40.569218+0000","last_became_peered":"2022-09-07T21:38:40.569218+0000","last_unstale":"2022-09-07T21:38:44.974513+0000","last_undegraded":"2022-09-07T21:38:44.974513+0000","last_fullsized":"2022-09-07T21:38:44.974513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:40:39.949203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":219246,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":29902,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":27,"num_read_kb":26,"num_write":139,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":103772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122,"ondisk_log_size":122,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6004,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739148,"statfs":{"total":95995035648,"available":95988887552,"internally_reserved":0,"allocated":962560,"data_stored":475487,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56000000000000005}]},{"osd":1,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.755},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.77400000000000002}]},{"osd":2,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.58299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":4,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.71199999999999997}]},{"osd":5,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.68899999999999995}]},{"osd":6,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.83299999999999996}]},{"osd":7,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.85399999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":9,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.97799999999999998}]},{"osd":10,"last update":"Wed Sep 7 21:38:50 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.89300000000000002}]}]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6352,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738800,"statfs":{"total":95995035648,"available":95988531200,"internally_reserved":0,"allocated":1056768,"data_stored":1071118,"data_compressed":48501,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":1,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.52800000000000002}]},{"osd":2,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.55500000000000005}]},{"osd":3,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.624},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.58299999999999996}]},{"osd":4,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.70199999999999996}]},{"osd":5,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.59399999999999997}]},{"osd":6,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.85699999999999998}]},{"osd":7,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.42399999999999999}]},{"osd":8,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":1.022}]},{"osd":9,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":1.0740000000000001}]},{"osd":11,"last update":"Wed Sep 7 21:38:54 2022","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.92900000000000005}]}]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6224,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738928,"statfs":{"total":95995035648,"available":95988662272,"internally_reserved":0,"allocated":991232,"data_stored":484481,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81999999999999995,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.67300000000000004,"15min":0.67900000000000005},"min":{"1min":0.47499999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.012,"5min":1.175,"15min":1.175},"last":0.64500000000000002}]},{"osd":1,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.77000000000000002,"15min":0.77600000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.236,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.84799999999999998,"15min":0.85399999999999998},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.751,"15min":0.75700000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.999,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.86299999999999999,"15min":0.88600000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0029999999999999,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.66100000000000003}]},{"osd":3,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.81000000000000005,"15min":0.81599999999999995},"min":{"1min":0.499,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0980000000000001,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.79300000000000004,"15min":0.80600000000000005},"min":{"1min":0.48999999999999999,"5min":0.434,"15min":0.434},"max":{"1min":1.04,"5min":1.613,"15min":1.613},"last":0.51700000000000002}]},{"osd":4,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.70499999999999996}]},{"osd":5,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.90400000000000003,"15min":0.91700000000000004},"min":{"1min":0.58499999999999996,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0609999999999999,"5min":1.55,"15min":1.55},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.88100000000000001,"15min":0.89200000000000002},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.083,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.58599999999999997}]},{"osd":6,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":0.81200000000000006}]},{"osd":7,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.0609999999999999,"15min":1.0529999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":5.6529999999999996,"5min":5.6529999999999996,"15min":5.6529999999999996},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.98099999999999998,"15min":0.98699999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.337,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":0.71199999999999997}]},{"osd":8,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.0169999999999999,"15min":1.026},"min":{"1min":0.66400000000000003,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.3010000000000002,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.052,"15min":1.0720000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.1479999999999999,"5min":1.76,"15min":1.76},"last":0.72399999999999998}]},{"osd":10,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.79400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]}]},{"osd":8,"up_from":59,"seq":253403070499,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6412,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738740,"statfs":{"total":95995035648,"available":95988469760,"internally_reserved":0,"allocated":987136,"data_stored":480305,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.622,"15min":0.63200000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59299999999999997,"15min":0.59599999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.42399999999999999}]},{"osd":1,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.61699999999999999,"15min":0.627},"min":{"1min":0.36799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78700000000000003,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.61399999999999999,"15min":0.622},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81499999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.64400000000000002}]},{"osd":2,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.48499999999999999,"15min":0.495},"min":{"1min":0.311,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.50700000000000001,"15min":0.51900000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.75700000000000001,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52600000000000002}]},{"osd":3,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63900000000000001,"15min":0.64100000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.62},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.65700000000000003,"15min":0.66600000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.67600000000000005}]},{"osd":4,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65200000000000002,"15min":0.66000000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.73499999999999999}]},{"osd":5,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.63500000000000001,"15min":0.65300000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.86599999999999999,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.60199999999999998,"15min":0.61499999999999999},"min":{"1min":0.31900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.83599999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.79200000000000004}]},{"osd":6,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.72999999999999998,"15min":0.74099999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90200000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.77400000000000002,"15min":0.78400000000000003},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.97999999999999998,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.77600000000000002}]},{"osd":7,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.746,"15min":0.76000000000000001},"min":{"1min":0.45400000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.745},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.72799999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.85099999999999998}]},{"osd":9,"last update":"Wed Sep 7 21:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.755,"15min":0.748},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.75900000000000001,"15min":0.752},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.83299999999999996}]},{"osd":10,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.78800000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]}]},{"osd":7,"up_from":51,"seq":219043332136,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6520,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738632,"statfs":{"total":95995035648,"available":95988359168,"internally_reserved":0,"allocated":1032192,"data_stored":1072417,"data_compressed":48501,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58499999999999996,"15min":0.57299999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.54200000000000004}]},{"osd":1,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.439,"15min":0.433},"min":{"1min":0.33000000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.45300000000000001,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.46600000000000003}]},{"osd":2,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83499999999999996,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.67100000000000004,"15min":0.627},"min":{"1min":0.40899999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.4580000000000002,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":0.60699999999999998}]},{"osd":3,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.629,"15min":0.60899999999999999},"min":{"1min":0.49099999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0149999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64100000000000001,"15min":0.627},"min":{"1min":0.42799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.53000000000000003}]},{"osd":4,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.35999999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52700000000000002,"15min":0.52800000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79400000000000004,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.50800000000000001}]},{"osd":5,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.66000000000000003,"15min":0.66400000000000003},"min":{"1min":0.48099999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.85399999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.624},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.45400000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.873,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.76200000000000001}]},{"osd":6,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73599999999999999,"15min":0.73799999999999999},"min":{"1min":0.50600000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0049999999999999,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.71299999999999997,"15min":0.70099999999999996},"min":{"1min":0.55300000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.73499999999999999}]},{"osd":8,"last update":"Wed Sep 7 21:38:26 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.71299999999999997,"15min":0.71699999999999997},"min":{"1min":0.48499999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.747},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72699999999999998,"15min":0.72799999999999998},"min":{"1min":0.503,"5min":0.443,"15min":0.443},"max":{"1min":1.0509999999999999,"5min":1.123,"15min":1.123},"last":0.67700000000000005}]},{"osd":9,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80000000000000004,"15min":0.80300000000000005},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.99399999999999999,"5min":1.036,"15min":1.036},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.83299999999999996,"15min":0.84099999999999997},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0980000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.79000000000000004}]},{"osd":10,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.72299999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]}]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6324,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738828,"statfs":{"total":95995035648,"available":95988559872,"internally_reserved":0,"allocated":962560,"data_stored":475487,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:38 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54700000000000004,"15min":0.53800000000000003},"min":{"1min":0.35699999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81100000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.34599999999999997,"5min":0.308,"15min":0.308},"max":{"1min":0.91900000000000004,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.442}]},{"osd":1,"last update":"Wed Sep 7 21:38:38 2022","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.65500000000000003,"15min":0.66100000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93999999999999995,"5min":1.127,"15min":1.127},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.48899999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.83199999999999996,"5min":1.159,"15min":1.159},"last":0.66900000000000004}]},{"osd":2,"last update":"Wed Sep 7 21:38:38 2022","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71099999999999997,"15min":0.71199999999999997},"min":{"1min":0.50700000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90500000000000003,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69999999999999996,"15min":0.70099999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0289999999999999,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.60599999999999998}]},{"osd":3,"last update":"Wed Sep 7 21:38:38 2022","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.67700000000000005,"15min":0.65700000000000003},"min":{"1min":0.45000000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.123,"5min":1.151,"15min":1.151},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.44400000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.62}]},{"osd":4,"last update":"Wed Sep 7 21:38:38 2022","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.746,"15min":0.754},"min":{"1min":0.54600000000000004,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.86399999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.76600000000000001,"15min":0.78000000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.96799999999999997,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.84599999999999997}]},{"osd":5,"last update":"Wed Sep 7 21:38:38 2022","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.76600000000000001,"15min":0.75800000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94399999999999995,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.82399999999999995,"15min":0.83199999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95399999999999996,"5min":1.341,"15min":1.341},"last":0.64100000000000001}]},{"osd":7,"last update":"Wed Sep 7 21:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.81100000000000005,"15min":0.80700000000000005},"min":{"1min":0.57199999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.139,"5min":1.212,"15min":1.212},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.82599999999999996,"15min":0.81999999999999995},"min":{"1min":0.61599999999999999,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.78600000000000003}]},{"osd":8,"last update":"Wed Sep 7 21:38:28 2022","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.84399999999999997,"15min":0.83499999999999996},"min":{"1min":0.60499999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.107,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.86499999999999999,"15min":0.85399999999999998},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1359999999999999,"5min":1.268,"15min":1.268},"last":0.752}]},{"osd":9,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.96799999999999997,"15min":0.97399999999999998},"min":{"1min":0.77700000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.133,"5min":1.254,"15min":1.254},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.88800000000000001,"15min":0.89000000000000001},"min":{"1min":0.65300000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1439999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.88100000000000001}]},{"osd":10,"last update":"Wed Sep 7 21:38:28 2022","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.82999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7296,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737856,"statfs":{"total":95995035648,"available":95987564544,"internally_reserved":0,"allocated":974848,"data_stored":479236,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:12 2022","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.53600000000000003,"15min":0.50900000000000001},"min":{"1min":0.42799999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81799999999999995,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.54800000000000004,"15min":0.52300000000000002},"min":{"1min":0.35699999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65200000000000002}]},{"osd":2,"last update":"Wed Sep 7 21:38:39 2022","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55500000000000005,"15min":0.52300000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83599999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53500000000000003,"15min":0.51300000000000001},"min":{"1min":0.372,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.874}]},{"osd":3,"last update":"Wed Sep 7 21:38:12 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.56499999999999995,"15min":0.54900000000000004},"min":{"1min":0.433,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.59499999999999997,"15min":0.58399999999999996},"min":{"1min":0.39200000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.504}]},{"osd":4,"last update":"Wed Sep 7 21:38:39 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.45300000000000001,"15min":0.44400000000000001},"min":{"1min":0.31900000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.77000000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.48399999999999999,"15min":0.46300000000000002},"min":{"1min":0.311,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.75600000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.59199999999999997}]},{"osd":5,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.64400000000000002,"15min":0.61099999999999999},"min":{"1min":0.38900000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.63300000000000001,"15min":0.59799999999999998},"min":{"1min":0.51400000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82099999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.81599999999999995}]},{"osd":6,"last update":"Wed Sep 7 21:38:39 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.67700000000000005,"15min":0.66300000000000003},"min":{"1min":0.48999999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.69599999999999995,"15min":0.68700000000000006},"min":{"1min":0.497,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.58399999999999996}]},{"osd":7,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.622,"15min":0.622},"min":{"1min":0.45800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82899999999999996,"5min":1.232,"15min":1.232},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.66100000000000003},"min":{"1min":0.40400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.874,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.67400000000000004}]},{"osd":8,"last update":"Wed Sep 7 21:38:31 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.70999999999999996,"15min":0.69999999999999996},"min":{"1min":0.51400000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.69099999999999995,"15min":0.67700000000000005},"min":{"1min":0.55200000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.83199999999999996}]},{"osd":9,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.57599999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.90200000000000002,"5min":1.032,"15min":1.032},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.77700000000000002,"15min":0.77900000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.91600000000000004,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.747}]},{"osd":10,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79000000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7312,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737840,"statfs":{"total":95995035648,"available":95987548160,"internally_reserved":0,"allocated":991232,"data_stored":476933,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 7 21:38:10 2022","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.59399999999999997,"15min":0.53800000000000003},"min":{"1min":0.39200000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.748},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.59599999999999997,"15min":0.55200000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0649999999999999,"5min":1.26,"15min":1.26},"last":0.52800000000000002}]},{"osd":2,"last update":"Wed Sep 7 21:38:31 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.626,"15min":0.60399999999999998},"min":{"1min":0.47699999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.871,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.498},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.64000000000000001,"15min":0.61199999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.96099999999999997,"5min":1.075,"15min":1.075},"last":0.85399999999999998}]},{"osd":3,"last update":"Wed Sep 7 21:37:10 2022","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.379,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":1.004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.53300000000000003,"15min":0.51100000000000001},"min":{"1min":0.32300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.56699999999999995}]},{"osd":4,"last update":"Wed Sep 7 21:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.67600000000000005,"15min":0.65000000000000002},"min":{"1min":0.56100000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.70799999999999996,"15min":0.65900000000000003},"min":{"1min":0.55400000000000005,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.65500000000000003}]},{"osd":5,"last update":"Wed Sep 7 21:39:07 2022","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":0.877,"15min":0.75700000000000001},"min":{"1min":0.68300000000000005,"5min":0.5,"15min":0.5},"max":{"1min":12.776999999999999,"5min":12.776999999999999,"15min":12.776999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.68700000000000006,"15min":0.66500000000000004},"min":{"1min":0.47199999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95699999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.91400000000000003}]},{"osd":6,"last update":"Wed Sep 7 21:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.71699999999999997,"15min":0.70399999999999996},"min":{"1min":0.48599999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.03,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.754,"15min":0.75800000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0669999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.80300000000000005}]},{"osd":7,"last update":"Wed Sep 7 21:38:02 2022","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.81100000000000005,"15min":0.80200000000000005},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.153,"5min":1.177,"15min":1.177},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.84399999999999997,"15min":0.82299999999999995},"min":{"1min":0.72399999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.88700000000000001}]},{"osd":8,"last update":"Wed Sep 7 21:38:26 2022","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.89800000000000002,"15min":0.89400000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.88},{"interface":"front","average":{"1min":0.875,"5min":0.84899999999999998,"15min":0.84499999999999997},"min":{"1min":0.67500000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.1200000000000001,"5min":1.143,"15min":1.143},"last":0.83399999999999996}]},{"osd":9,"last update":"Wed Sep 7 21:39:03 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.93700000000000006,"15min":0.95099999999999996},"min":{"1min":0.625,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0509999999999999,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.89700000000000002,"15min":0.90500000000000003},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.97199999999999998}]},{"osd":10,"last update":"Wed Sep 7 21:38:31 2022","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.89800000000000002}]},{"osd":11,"last update":"Wed Sep 7 21:38:51 2022","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.94499999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7420,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737732,"statfs":{"total":95995035648,"available":95987437568,"internally_reserved":0,"allocated":1036288,"data_stored":516882,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:42 2022","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53500000000000003,"15min":0.51300000000000001},"min":{"1min":0.38700000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.78900000000000003,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.53600000000000003,"15min":0.52000000000000002},"min":{"1min":0.39000000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.93500000000000005,"5min":0.999,"15min":0.999},"last":0.438}]},{"osd":1,"last update":"Wed Sep 7 21:38:42 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.42499999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.53300000000000003,"15min":0.495},"min":{"1min":0.40799999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.60099999999999998}]},{"osd":3,"last update":"Wed Sep 7 21:38:11 2022","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.64100000000000001,"15min":0.57299999999999995},"min":{"1min":0.434,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":7.649,"5min":7.649,"15min":7.649},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.56399999999999995,"15min":0.53600000000000003},"min":{"1min":0.442,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.61299999999999999}]},{"osd":4,"last update":"Wed Sep 7 21:37:37 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.53200000000000003,"15min":0.51000000000000001},"min":{"1min":0.34000000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.55400000000000005,"15min":0.52500000000000002},"min":{"1min":0.38100000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.73099999999999998}]},{"osd":5,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.47499999999999998,"15min":0.437},"min":{"1min":0.312,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.53600000000000003,"15min":0.49099999999999999},"min":{"1min":0.35599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.51300000000000001}]},{"osd":6,"last update":"Wed Sep 7 21:38:42 2022","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.69399999999999995},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.93000000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.55400000000000005,"5min":0.498,"15min":0.498},"max":{"1min":0.90100000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.76400000000000001}]},{"osd":7,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.60199999999999998,"15min":0.59599999999999997},"min":{"1min":0.42299999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.66100000000000003,"15min":0.65700000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.69899999999999995}]},{"osd":8,"last update":"Wed Sep 7 21:38:30 2022","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.63600000000000001,"15min":0.64300000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64400000000000002,"15min":0.64900000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.68300000000000005}]},{"osd":9,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.77700000000000002,"15min":0.77000000000000002},"min":{"1min":0.54800000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76100000000000001,"15min":0.755},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.84799999999999998}]},{"osd":10,"last update":"Wed Sep 7 21:38:30 2022","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.75}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182462,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6660,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738492,"statfs":{"total":95995035648,"available":95988215808,"internally_reserved":0,"allocated":978944,"data_stored":475771,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:13 2022","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.47999999999999998,"15min":0.442},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84099999999999997,"5min":1.167,"15min":1.167},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.48499999999999999,"15min":0.45700000000000002},"min":{"1min":0.379,"5min":0.307,"15min":0.307},"max":{"1min":0.876,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.46500000000000002}]},{"osd":1,"last update":"Wed Sep 7 21:38:13 2022","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.45900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.65200000000000002,"15min":0.60099999999999998},"min":{"1min":0.51300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.048,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.79600000000000004}]},{"osd":2,"last update":"Wed Sep 7 21:38:13 2022","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.66700000000000004,"15min":0.63500000000000001},"min":{"1min":0.53800000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.99199999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.64400000000000002,"15min":0.62},"min":{"1min":0.50700000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97699999999999998,"5min":1.302,"15min":1.302},"last":0.78000000000000003}]},{"osd":4,"last update":"Wed Sep 7 21:38:39 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.68600000000000005,"15min":0.64200000000000002},"min":{"1min":0.498,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.92800000000000005,"5min":1.286,"15min":1.286},"last":0.755},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.66600000000000004,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.99399999999999999,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.66100000000000003}]},{"osd":5,"last update":"Wed Sep 7 21:38:07 2022","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.67600000000000005,"15min":0.66400000000000003},"min":{"1min":0.53900000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.125,"5min":1.181,"15min":1.181},"last":0.746},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.75900000000000001,"15min":0.73199999999999998},"min":{"1min":0.505,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.1819999999999999,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.70599999999999996}]},{"osd":6,"last update":"Wed Sep 7 21:38:39 2022","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79500000000000004,"15min":0.78000000000000003},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.995,"5min":1.399,"15min":1.399},"last":0.629},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.74199999999999999,"15min":0.73399999999999999},"min":{"1min":0.51100000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.1259999999999999,"5min":1.157,"15min":1.157},"last":0.51200000000000001}]},{"osd":7,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.73199999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.034,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.874,"15min":0.86299999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.0169999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.82899999999999996}]},{"osd":8,"last update":"Wed Sep 7 21:38:30 2022","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82999999999999996,"15min":0.83099999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81299999999999994,"15min":0.81200000000000006},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.78400000000000003}]},{"osd":9,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.97799999999999998,"15min":0.99199999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.093,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.90400000000000003,"15min":0.91800000000000004},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1539999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.54500000000000004}]},{"osd":10,"last update":"Wed Sep 7 21:38:30 2022","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.86299999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]}]},{"osd":4,"up_from":33,"seq":141733920825,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6528,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738624,"statfs":{"total":95995035648,"available":95988350976,"internally_reserved":0,"allocated":974848,"data_stored":476083,"data_compressed":11686,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:37:36 2022","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54400000000000004,"15min":0.52900000000000003},"min":{"1min":0.41199999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78100000000000003,"5min":0.879,"15min":0.879},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.52700000000000002,"15min":0.503},"min":{"1min":0.40699999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.35199999999999998}]},{"osd":1,"last update":"Wed Sep 7 21:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38,"15min":0.35799999999999998},"min":{"1min":0.29199999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54800000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.39200000000000002,"15min":0.372},"min":{"1min":0.27300000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.53100000000000003,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.441}]},{"osd":2,"last update":"Wed Sep 7 21:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.53800000000000003},"min":{"1min":0.378,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77800000000000002,"5min":0.879,"15min":0.879},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56599999999999995,"15min":0.55100000000000005},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82399999999999995,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Sep 7 21:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58399999999999996,"15min":0.56000000000000005},"min":{"1min":0.40200000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.40300000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.79300000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.64800000000000002}]},{"osd":5,"last update":"Wed Sep 7 21:38:06 2022","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81799999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.57799999999999996},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.79600000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.51600000000000001}]},{"osd":6,"last update":"Wed Sep 7 21:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.64600000000000002,"15min":0.64200000000000002},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.95399999999999996,"5min":1.024,"15min":1.024},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64800000000000002,"15min":0.64400000000000002},"min":{"1min":0.45200000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.80400000000000005,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.77500000000000002}]},{"osd":7,"last update":"Wed Sep 7 21:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61699999999999999,"15min":0.61599999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.872,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76300000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.502}]},{"osd":8,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73099999999999998,"15min":0.72999999999999998},"min":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.94899999999999995,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.72299999999999998,"15min":0.72899999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.93200000000000005,"5min":1.198,"15min":1.198},"last":0.56699999999999995}]},{"osd":9,"last update":"Wed Sep 7 21:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72099999999999997,"15min":0.72299999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.78100000000000003,"15min":0.78600000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.048,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.74399999999999999}]},{"osd":10,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.67800000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757301,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6652,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738500,"statfs":{"total":95995035648,"available":95988224000,"internally_reserved":0,"allocated":1036288,"data_stored":1066944,"data_compressed":48501,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77800000000000002,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.56599999999999995,"15min":0.55600000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88400000000000001,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.39400000000000002}]},{"osd":1,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.56399999999999995,"15min":0.57699999999999996},"min":{"1min":0.38300000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.77500000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.39700000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.67100000000000004}]},{"osd":2,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.42999999999999999,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.38},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.28599999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.70899999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.41299999999999998}]},{"osd":3,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80000000000000004,"5min":1.238,"15min":1.238},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61099999999999999,"15min":0.59199999999999997},"min":{"1min":0.437,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.84799999999999998,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.47199999999999998}]},{"osd":4,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60999999999999999,"15min":0.61599999999999999},"min":{"1min":0.40400000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.85999999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.39200000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75700000000000001,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.76200000000000001}]},{"osd":6,"last update":"Wed Sep 7 21:38:40 2022","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64900000000000002,"15min":0.627},"min":{"1min":0.5,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66700000000000004,"15min":0.65900000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80800000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.502}]},{"osd":7,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.64200000000000002,"15min":0.65100000000000002},"min":{"1min":0.46400000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.94699999999999995,"5min":1.002,"15min":1.002},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66600000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.72299999999999998}]},{"osd":8,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.64600000000000002,"15min":0.66200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.748},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.90300000000000002,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.70699999999999996}]},{"osd":9,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.79800000000000004,"15min":0.80900000000000005},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":0.90000000000000002,"5min":0.998,"15min":0.998},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.78100000000000003,"15min":0.78900000000000003},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.89600000000000002,"5min":1.042,"15min":1.042},"last":0.63600000000000001}]},{"osd":10,"last update":"Wed Sep 7 21:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.69399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35171,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":25650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":41395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":27557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-07T21:39:11.085 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2022-09-07T21:39:11.086 INFO:tasks.cephadm:Setting up client nodes... 2022-09-07T21:39:11.086 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-09-07T21:39:11.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:10 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2211518304' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:11.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:10 smithi047 ceph-mon[90087]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:39:11.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:10 smithi047 ceph-mon[90087]: pgmap v331: 129 pgs: 129 active+clean; 606 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4.3 KiB/s wr, 9 op/s 2022-09-07T21:39:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:10 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2211518304' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:10 smithi055 ceph-mon[97033]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:39:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:10 smithi055 ceph-mon[97033]: pgmap v331: 129 pgs: 129 active+clean; 606 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4.3 KiB/s wr, 9 op/s 2022-09-07T21:39:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:10 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2211518304' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:10 smithi059 ceph-mon[97450]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:39:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:10 smithi059 ceph-mon[97450]: pgmap v331: 129 pgs: 129 active+clean; 606 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4.3 KiB/s wr, 9 op/s 2022-09-07T21:39:12.005 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:12.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:11 smithi047 ceph-mon[90087]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:39:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:11 smithi055 ceph-mon[97033]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:39:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:11 smithi059 ceph-mon[97450]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:39:12.945 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:12 smithi047 ceph-mon[90087]: pgmap v332: 129 pgs: 129 active+clean; 606 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.3 KiB/s wr, 10 op/s 2022-09-07T21:39:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:12 smithi055 ceph-mon[97033]: pgmap v332: 129 pgs: 129 active+clean; 606 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.3 KiB/s wr, 10 op/s 2022-09-07T21:39:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:12 smithi059 ceph-mon[97450]: pgmap v332: 129 pgs: 129 active+clean; 606 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.3 KiB/s wr, 10 op/s 2022-09-07T21:39:13.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.685+0000 7f909ab1b700 1 -- 172.21.15.47:0/732255819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9094103a30 msgr2=0x7f9094105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:13.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.685+0000 7f909ab1b700 1 --2- 172.21.15.47:0/732255819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9094103a30 0x7f9094105eb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f908c009bb0 tx=0x7f908c009ec0 comp rx=0 tx=0).stop 2022-09-07T21:39:13.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.685+0000 7f909ab1b700 1 -- 172.21.15.47:0/732255819 shutdown_connections 2022-09-07T21:39:13.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.685+0000 7f909ab1b700 1 --2- 172.21.15.47:0/732255819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9094103a30 0x7f9094105eb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.685+0000 7f909ab1b700 1 --2- 172.21.15.47:0/732255819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90940673d0 0x7f9094100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.685+0000 7f909ab1b700 1 --2- 172.21.15.47:0/732255819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9094100f40 0x7f90941033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.685+0000 7f909ab1b700 1 -- 172.21.15.47:0/732255819 >> 172.21.15.47:0/732255819 conn(0x7f90940f8430 msgr2=0x7f90940fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.685+0000 7f909ab1b700 1 -- 172.21.15.47:0/732255819 shutdown_connections 2022-09-07T21:39:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.685+0000 7f909ab1b700 1 -- 172.21.15.47:0/732255819 wait complete. 2022-09-07T21:39:13.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.686+0000 7f909ab1b700 1 Processor -- start 2022-09-07T21:39:13.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.686+0000 7f909ab1b700 1 -- start start 2022-09-07T21:39:13.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.687+0000 7f909ab1b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90940673d0 0x7f9094193a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:13.698 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.687+0000 7f909ab1b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9094100f40 0x7f9094193f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:13.699 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.687+0000 7f909ab1b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9094103a30 0x7f909419afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:13.699 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.687+0000 7f909ab1b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909419b5b0 con 0x7f9094100f40 2022-09-07T21:39:13.699 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.687+0000 7f909ab1b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909419b720 con 0x7f9094103a30 2022-09-07T21:39:13.700 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.687+0000 7f909ab1b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909419b8e0 con 0x7f90940673d0 2022-09-07T21:39:13.700 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.687+0000 7f9093fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9094100f40 0x7f9094193f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.687+0000 7f90988b7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90940673d0 0x7f9094193a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.687+0000 7f90990b8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9094103a30 0x7f909419afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.687+0000 7f90988b7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90940673d0 0x7f9094193a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:50270/0 (socket says 172.21.15.47:50270) 2022-09-07T21:39:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.687+0000 7f90988b7700 1 -- 172.21.15.47:0/3148658626 learned_addr learned my addr 172.21.15.47:0/3148658626 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.687+0000 7f9093fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9094100f40 0x7f9094193f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42014/0 (socket says 172.21.15.47:42014) 2022-09-07T21:39:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.688+0000 7f90988b7700 1 -- 172.21.15.47:0/3148658626 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9094103a30 msgr2=0x7f909419afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.688+0000 7f90988b7700 1 --2- 172.21.15.47:0/3148658626 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9094103a30 0x7f909419afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:13.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.688+0000 7f90988b7700 1 -- 172.21.15.47:0/3148658626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9094100f40 msgr2=0x7f9094193f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:13.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.688+0000 7f90988b7700 1 --2- 172.21.15.47:0/3148658626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9094100f40 0x7f9094193f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:13.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.688+0000 7f90988b7700 1 -- 172.21.15.47:0/3148658626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c0097e0 con 0x7f90940673d0 2022-09-07T21:39:13.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.688+0000 7f90990b8700 1 --2- 172.21.15.47:0/3148658626 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9094103a30 0x7f909419afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:13.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.688+0000 7f90988b7700 1 --2- 172.21.15.47:0/3148658626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90940673d0 0x7f9094193a30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f908800e6e0 tx=0x7f908800e9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:13.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.688+0000 7f9091ffb700 1 -- 172.21.15.47:0/3148658626 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9088009500 con 0x7f90940673d0 2022-09-07T21:39:13.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.688+0000 7f909ab1b700 1 -- 172.21.15.47:0/3148658626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909419bbc0 con 0x7f90940673d0 2022-09-07T21:39:13.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.688+0000 7f9091ffb700 1 -- 172.21.15.47:0/3148658626 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f9088010da0 con 0x7f90940673d0 2022-09-07T21:39:13.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.688+0000 7f9091ffb700 1 -- 172.21.15.47:0/3148658626 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90880173d0 con 0x7f90940673d0 2022-09-07T21:39:13.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.689+0000 7f909ab1b700 1 -- 172.21.15.47:0/3148658626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909419c120 con 0x7f90940673d0 2022-09-07T21:39:13.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.690+0000 7f909ab1b700 1 -- 172.21.15.47:0/3148658626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9094109640 con 0x7f90940673d0 2022-09-07T21:39:13.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.692+0000 7f9091ffb700 1 -- 172.21.15.47:0/3148658626 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f908800c040 con 0x7f90940673d0 2022-09-07T21:39:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.693+0000 7f9091ffb700 1 --2- 172.21.15.47:0/3148658626 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f90840990d0 0x7f908409b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.693+0000 7f9091ffb700 1 -- 172.21.15.47:0/3148658626 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f9088014070 con 0x7f90940673d0 2022-09-07T21:39:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.693+0000 7f9093fff700 1 --2- 172.21.15.47:0/3148658626 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f90840990d0 0x7f908409b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.694+0000 7f9093fff700 1 --2- 172.21.15.47:0/3148658626 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f90840990d0 0x7f908409b580 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9080005fd0 tx=0x7f9080005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:13.695+0000 7f9091ffb700 1 -- 172.21.15.47:0/3148658626 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f908808eb60 con 0x7f90940673d0 2022-09-07T21:39:14.019 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:14.016+0000 7f909ab1b700 1 -- 172.21.15.47:0/3148658626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f90940f9e20 con 0x7f90940673d0 2022-09-07T21:39:14.025 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:14.022+0000 7f9091ffb700 1 -- 172.21.15.47:0/3148658626 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f9088092180 con 0x7f90940673d0 2022-09-07T21:39:14.026 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2022-09-07T21:39:14.026 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQCCDxljVpMhARAAaqs8ha/nhrCMTiys10rtvg== 2022-09-07T21:39:14.028 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:14.024+0000 7f909ab1b700 1 -- 172.21.15.47:0/3148658626 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f90840990d0 msgr2=0x7f908409b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:14.028 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:14.024+0000 7f909ab1b700 1 --2- 172.21.15.47:0/3148658626 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f90840990d0 0x7f908409b580 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9080005fd0 tx=0x7f9080005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:14.029 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:14.025+0000 7f909ab1b700 1 -- 172.21.15.47:0/3148658626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90940673d0 msgr2=0x7f9094193a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:14.029 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:14.025+0000 7f909ab1b700 1 --2- 172.21.15.47:0/3148658626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90940673d0 0x7f9094193a30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f908800e6e0 tx=0x7f908800e9f0 comp rx=0 tx=0).stop 2022-09-07T21:39:14.029 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:14.025+0000 7f909ab1b700 1 -- 172.21.15.47:0/3148658626 shutdown_connections 2022-09-07T21:39:14.030 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:14.025+0000 7f909ab1b700 1 --2- 172.21.15.47:0/3148658626 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f90840990d0 0x7f908409b580 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:14.030 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:14.025+0000 7f909ab1b700 1 --2- 172.21.15.47:0/3148658626 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9094103a30 0x7f909419afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:14.030 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:14.025+0000 7f909ab1b700 1 --2- 172.21.15.47:0/3148658626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90940673d0 0x7f9094193a30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:14.031 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:14.025+0000 7f909ab1b700 1 --2- 172.21.15.47:0/3148658626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9094100f40 0x7f9094193f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:14.031 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:14.025+0000 7f909ab1b700 1 -- 172.21.15.47:0/3148658626 >> 172.21.15.47:0/3148658626 conn(0x7f90940f8430 msgr2=0x7f90940ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:14.031 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:14.025+0000 7f909ab1b700 1 -- 172.21.15.47:0/3148658626 shutdown_connections 2022-09-07T21:39:14.031 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:14.025+0000 7f909ab1b700 1 -- 172.21.15.47:0/3148658626 wait complete. 2022-09-07T21:39:14.577 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:39:14.577 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2022-09-07T21:39:14.577 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-09-07T21:39:14.744 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-09-07T21:39:14.745 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2022-09-07T21:39:14.745 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph mgr dump --format=json 2022-09-07T21:39:14.863 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:14 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3148658626' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-09-07T21:39:14.863 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:14 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-09-07T21:39:14.864 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:14 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-09-07T21:39:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:14 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3148658626' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-09-07T21:39:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:14 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-09-07T21:39:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:14 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-09-07T21:39:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:14 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3148658626' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-09-07T21:39:15.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:14 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-09-07T21:39:15.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:14 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-09-07T21:39:15.667 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:16.006 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:15 smithi047 ceph-mon[90087]: pgmap v333: 129 pgs: 129 active+clean; 628 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2022-09-07T21:39:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:15 smithi059 ceph-mon[97450]: pgmap v333: 129 pgs: 129 active+clean; 628 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2022-09-07T21:39:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:15 smithi055 ceph-mon[97033]: pgmap v333: 129 pgs: 129 active+clean; 628 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2022-09-07T21:39:17.530 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.526+0000 7f3963368700 1 -- 172.21.15.47:0/2202027657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f395c1053e0 msgr2=0x7f395c10cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:17.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.526+0000 7f3963368700 1 --2- 172.21.15.47:0/2202027657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f395c1053e0 0x7f395c10cf80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3954009be0 tx=0x7f3954009ef0 comp rx=0 tx=0).stop 2022-09-07T21:39:17.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.526+0000 7f3963368700 1 -- 172.21.15.47:0/2202027657 shutdown_connections 2022-09-07T21:39:17.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.526+0000 7f3963368700 1 --2- 172.21.15.47:0/2202027657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f395c1053e0 0x7f395c10cf80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:17.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.526+0000 7f3963368700 1 --2- 172.21.15.47:0/2202027657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f395c1002e0 0x7f395c1006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:17.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.526+0000 7f3963368700 1 --2- 172.21.15.47:0/2202027657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f395c100c80 0x7f395c104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:17.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.526+0000 7f3963368700 1 -- 172.21.15.47:0/2202027657 >> 172.21.15.47:0/2202027657 conn(0x7f395c0fbc20 msgr2=0x7f395c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:17.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.526+0000 7f3963368700 1 -- 172.21.15.47:0/2202027657 shutdown_connections 2022-09-07T21:39:17.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.527+0000 7f3963368700 1 -- 172.21.15.47:0/2202027657 wait complete. 2022-09-07T21:39:17.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.528+0000 7f3963368700 1 Processor -- start 2022-09-07T21:39:17.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.528+0000 7f3963368700 1 -- start start 2022-09-07T21:39:17.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.528+0000 7f3963368700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f395c1002e0 0x7f395c195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:17.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.528+0000 7f3963368700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f395c100c80 0x7f395c196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:17.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.528+0000 7f3963368700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f395c1053e0 0x7f395c19a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:17.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.528+0000 7f3963368700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c19aef0 con 0x7f395c1053e0 2022-09-07T21:39:17.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.528+0000 7f3963368700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c19b060 con 0x7f395c1002e0 2022-09-07T21:39:17.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.528+0000 7f3963368700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c19b220 con 0x7f395c100c80 2022-09-07T21:39:17.537 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.529+0000 7f3961104700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f395c1002e0 0x7f395c195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:17.537 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.529+0000 7f3961905700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f395c1053e0 0x7f395c19a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:17.537 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.529+0000 7f3961104700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f395c1002e0 0x7f395c195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:42956/0 (socket says 172.21.15.47:42956) 2022-09-07T21:39:17.537 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.529+0000 7f3961104700 1 -- 172.21.15.47:0/1489936815 learned_addr learned my addr 172.21.15.47:0/1489936815 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:17.538 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.529+0000 7f3960903700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f395c100c80 0x7f395c196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:17.538 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.529+0000 7f3961905700 1 -- 172.21.15.47:0/1489936815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f395c100c80 msgr2=0x7f395c196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:17.538 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.529+0000 7f3961905700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f395c100c80 0x7f395c196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:17.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.529+0000 7f3961905700 1 -- 172.21.15.47:0/1489936815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f395c1002e0 msgr2=0x7f395c195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:17.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.529+0000 7f3961905700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f395c1002e0 0x7f395c195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:17.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.529+0000 7f3961905700 1 -- 172.21.15.47:0/1489936815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39540097e0 con 0x7f395c1053e0 2022-09-07T21:39:17.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.529+0000 7f3960903700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f395c100c80 0x7f395c196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:17.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.529+0000 7f3961104700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f395c1002e0 0x7f395c195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:17.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.530+0000 7f3961905700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f395c1053e0 0x7f395c19a860 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3954009be0 tx=0x7f395400bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:17.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.530+0000 7f39527fc700 1 -- 172.21.15.47:0/1489936815 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395400ba10 con 0x7f395c1053e0 2022-09-07T21:39:17.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.530+0000 7f39527fc700 1 -- 172.21.15.47:0/1489936815 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f39540056f0 con 0x7f395c1053e0 2022-09-07T21:39:17.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.530+0000 7f3963368700 1 -- 172.21.15.47:0/1489936815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f395c19b4a0 con 0x7f395c1053e0 2022-09-07T21:39:17.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.530+0000 7f39527fc700 1 -- 172.21.15.47:0/1489936815 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395400fc50 con 0x7f395c1053e0 2022-09-07T21:39:17.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.530+0000 7f3963368700 1 -- 172.21.15.47:0/1489936815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395c1a2cf0 con 0x7f395c1053e0 2022-09-07T21:39:17.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.531+0000 7f3963368700 1 -- 172.21.15.47:0/1489936815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f395c064ea0 con 0x7f395c1053e0 2022-09-07T21:39:17.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.533+0000 7f39527fc700 1 -- 172.21.15.47:0/1489936815 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f3954027030 con 0x7f395c1053e0 2022-09-07T21:39:17.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.534+0000 7f39527fc700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3948099050 0x7f394809b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:17.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.535+0000 7f39527fc700 1 -- 172.21.15.47:0/1489936815 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f39540c34d0 con 0x7f395c1053e0 2022-09-07T21:39:17.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.535+0000 7f3961104700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3948099050 0x7f394809b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:17.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.535+0000 7f3961104700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3948099050 0x7f394809b500 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f39580057d0 tx=0x7f3958005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:17.545 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.539+0000 7f39527fc700 1 -- 172.21.15.47:0/1489936815 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f395408f930 con 0x7f395c1053e0 2022-09-07T21:39:17.816 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.813+0000 7f3963368700 1 -- 172.21.15.47:0/1489936815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f395c197310 con 0x7f395c1053e0 2022-09-07T21:39:17.822 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.818+0000 7f39527fc700 1 -- 172.21.15.47:0/1489936815 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+246062 (secure 0 0 0) 0x7f3954092f50 con 0x7f395c1053e0 2022-09-07T21:39:17.827 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:39:17.829 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.823+0000 7f3963368700 1 -- 172.21.15.47:0/1489936815 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3948099050 msgr2=0x7f394809b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:17.829 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.823+0000 7f3963368700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3948099050 0x7f394809b500 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f39580057d0 tx=0x7f3958005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:17.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.823+0000 7f3963368700 1 -- 172.21.15.47:0/1489936815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f395c1053e0 msgr2=0x7f395c19a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:17.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.823+0000 7f3963368700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f395c1053e0 0x7f395c19a860 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3954009be0 tx=0x7f395400bfc0 comp rx=0 tx=0).stop 2022-09-07T21:39:17.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.823+0000 7f3963368700 1 -- 172.21.15.47:0/1489936815 shutdown_connections 2022-09-07T21:39:17.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.823+0000 7f3963368700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3948099050 0x7f394809b500 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:17.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.823+0000 7f3963368700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f395c1053e0 0x7f395c19a860 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:17.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.823+0000 7f3963368700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f395c1002e0 0x7f395c195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:17.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.823+0000 7f3963368700 1 --2- 172.21.15.47:0/1489936815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f395c100c80 0x7f395c196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:17.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.823+0000 7f3963368700 1 -- 172.21.15.47:0/1489936815 >> 172.21.15.47:0/1489936815 conn(0x7f395c0fbc20 msgr2=0x7f395c0fc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:17.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.823+0000 7f3963368700 1 -- 172.21.15.47:0/1489936815 shutdown_connections 2022-09-07T21:39:17.832 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:17.824+0000 7f3963368700 1 -- 172.21.15.47:0/1489936815 wait complete. 2022-09-07T21:39:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:17 smithi059 ceph-mon[97450]: pgmap v334: 129 pgs: 129 active+clean; 628 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 6.6 KiB/s wr, 14 op/s 2022-09-07T21:39:18.046 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:17 smithi047 ceph-mon[90087]: pgmap v334: 129 pgs: 129 active+clean; 628 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 6.6 KiB/s wr, 14 op/s 2022-09-07T21:39:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:17 smithi055 ceph-mon[97033]: pgmap v334: 129 pgs: 129 active+clean; 628 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 6.6 KiB/s wr, 14 op/s 2022-09-07T21:39:18.339 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":17,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":429234829},{"type":"v1","addr":"172.21.15.47:6801","nonce":429234829}]},"active_addr":"172.21.15.47:6801/429234829","active_change":"2022-09-07T21:30:46.440166+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24110,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24115,"name":"z","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.47:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":2376545074}]},{"addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":101590997}]},{"addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":1295922839}]},{"addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":2271337006}]},{"addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":2999789307}]}]}} 2022-09-07T21:39:18.347 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2022-09-07T21:39:18.348 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2022-09-07T21:39:18.349 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd dump --format=json 2022-09-07T21:39:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:18 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1489936815' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-09-07T21:39:19.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:18 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1489936815' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-09-07T21:39:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:18 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1489936815' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-09-07T21:39:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:19.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:19 smithi047 ceph-mon[90087]: pgmap v335: 129 pgs: 129 active+clean; 636 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.2 KiB/s wr, 15 op/s 2022-09-07T21:39:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:19 smithi059 ceph-mon[97450]: pgmap v335: 129 pgs: 129 active+clean; 636 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.2 KiB/s wr, 15 op/s 2022-09-07T21:39:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:19 smithi055 ceph-mon[97033]: pgmap v335: 129 pgs: 129 active+clean; 636 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.2 KiB/s wr, 15 op/s 2022-09-07T21:39:20.948 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.943+0000 7fb28f59e700 1 -- 172.21.15.47:0/1926551721 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb290102fa0 msgr2=0x7fb29010ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:20.948 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.943+0000 7fb28f59e700 1 --2- 172.21.15.47:0/1926551721 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb290102fa0 0x7fb29010ab40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb288009be0 tx=0x7fb288009ef0 comp rx=0 tx=0).stop 2022-09-07T21:39:20.948 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.944+0000 7fb28f59e700 1 -- 172.21.15.47:0/1926551721 shutdown_connections 2022-09-07T21:39:20.949 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.944+0000 7fb28f59e700 1 --2- 172.21.15.47:0/1926551721 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb290102fa0 0x7fb29010ab40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:20.949 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.944+0000 7fb28f59e700 1 --2- 172.21.15.47:0/1926551721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2900fdea0 0x7fb2900fe270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:20.949 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.944+0000 7fb28f59e700 1 --2- 172.21.15.47:0/1926551721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2900fe840 0x7fb290102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:20.950 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.944+0000 7fb28f59e700 1 -- 172.21.15.47:0/1926551721 >> 172.21.15.47:0/1926551721 conn(0x7fb2900f9800 msgr2=0x7fb2900fbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:20.950 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.944+0000 7fb28f59e700 1 -- 172.21.15.47:0/1926551721 shutdown_connections 2022-09-07T21:39:20.950 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.944+0000 7fb28f59e700 1 -- 172.21.15.47:0/1926551721 wait complete. 2022-09-07T21:39:20.954 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.945+0000 7fb28f59e700 1 Processor -- start 2022-09-07T21:39:20.954 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.945+0000 7fb28f59e700 1 -- start start 2022-09-07T21:39:20.954 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.946+0000 7fb28f59e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2900fdea0 0x7fb290070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:20.955 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.946+0000 7fb28f59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2900fe840 0x7fb290071520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:20.955 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.946+0000 7fb28f59e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb290102fa0 0x7fb2901a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:20.955 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.946+0000 7fb28f59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb290071db0 con 0x7fb2900fe840 2022-09-07T21:39:20.955 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.946+0000 7fb28f59e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2901a2210 con 0x7fb290102fa0 2022-09-07T21:39:20.956 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.946+0000 7fb28f59e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2901a2350 con 0x7fb2900fdea0 2022-09-07T21:39:20.956 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.946+0000 7fb28dd9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2900fe840 0x7fb290071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:20.956 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.946+0000 7fb28e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2900fdea0 0x7fb290070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:20.956 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.946+0000 7fb28ed9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb290102fa0 0x7fb2901a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:20.957 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.946+0000 7fb28dd9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2900fe840 0x7fb290071520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60580/0 (socket says 172.21.15.47:60580) 2022-09-07T21:39:20.957 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.946+0000 7fb28e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2900fdea0 0x7fb290070fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:45296/0 (socket says 172.21.15.47:45296) 2022-09-07T21:39:20.957 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.946+0000 7fb28dd9b700 1 -- 172.21.15.47:0/773931894 learned_addr learned my addr 172.21.15.47:0/773931894 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:20.957 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.947+0000 7fb28e59c700 1 -- 172.21.15.47:0/773931894 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb290102fa0 msgr2=0x7fb2901a1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:20.958 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.947+0000 7fb28e59c700 1 --2- 172.21.15.47:0/773931894 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb290102fa0 0x7fb2901a1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:20.958 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.947+0000 7fb28e59c700 1 -- 172.21.15.47:0/773931894 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2900fe840 msgr2=0x7fb290071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:20.958 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.947+0000 7fb28e59c700 1 --2- 172.21.15.47:0/773931894 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2900fe840 0x7fb290071520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:20.959 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.947+0000 7fb28e59c700 1 -- 172.21.15.47:0/773931894 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2880097e0 con 0x7fb2900fdea0 2022-09-07T21:39:20.959 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.947+0000 7fb28ed9d700 1 --2- 172.21.15.47:0/773931894 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb290102fa0 0x7fb2901a1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:20.959 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.947+0000 7fb28dd9b700 1 --2- 172.21.15.47:0/773931894 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2900fe840 0x7fb290071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:20.960 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.947+0000 7fb28e59c700 1 --2- 172.21.15.47:0/773931894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2900fdea0 0x7fb290070fe0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb28000e6c0 tx=0x7fb28000e9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:20.960 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.947+0000 7fb27f7fe700 1 -- 172.21.15.47:0/773931894 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb280010900 con 0x7fb2900fdea0 2022-09-07T21:39:20.960 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.947+0000 7fb27f7fe700 1 -- 172.21.15.47:0/773931894 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb280018420 con 0x7fb2900fdea0 2022-09-07T21:39:20.961 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.947+0000 7fb27f7fe700 1 -- 172.21.15.47:0/773931894 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb280017610 con 0x7fb2900fdea0 2022-09-07T21:39:20.961 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.948+0000 7fb28f59e700 1 -- 172.21.15.47:0/773931894 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2901a25e0 con 0x7fb2900fdea0 2022-09-07T21:39:20.961 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.948+0000 7fb28f59e700 1 -- 172.21.15.47:0/773931894 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2901a2aa0 con 0x7fb2900fdea0 2022-09-07T21:39:20.962 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.949+0000 7fb28f59e700 1 -- 172.21.15.47:0/773931894 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29004d480 con 0x7fb2900fdea0 2022-09-07T21:39:20.962 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.951+0000 7fb27f7fe700 1 -- 172.21.15.47:0/773931894 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7fb28000c040 con 0x7fb2900fdea0 2022-09-07T21:39:20.964 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.952+0000 7fb27f7fe700 1 --2- 172.21.15.47:0/773931894 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb278099050 0x7fb27809b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:20.965 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.952+0000 7fb27f7fe700 1 -- 172.21.15.47:0/773931894 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fb280014070 con 0x7fb2900fdea0 2022-09-07T21:39:20.965 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.952+0000 7fb28dd9b700 1 --2- 172.21.15.47:0/773931894 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb278099050 0x7fb27809b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:20.966 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.953+0000 7fb28dd9b700 1 --2- 172.21.15.47:0/773931894 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb278099050 0x7fb27809b500 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb2900727d0 tx=0x7fb284005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:20.966 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:20.957+0000 7fb27f7fe700 1 -- 172.21.15.47:0/773931894 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fb28004fc80 con 0x7fb2900fdea0 2022-09-07T21:39:21.184 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:21.180+0000 7fb28f59e700 1 -- 172.21.15.47:0/773931894 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb29010dac0 con 0x7fb2900fdea0 2022-09-07T21:39:21.185 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:21.183+0000 7fb27f7fe700 1 -- 172.21.15.47:0/773931894 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23119 (secure 0 0 0) 0x7fb28008eb10 con 0x7fb2900fdea0 2022-09-07T21:39:21.187 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:39:21.187 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":89,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:38:45.529264+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-07T21:38:38.608937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-07T21:38:39.936252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-07T21:38:43.731797+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:39:21.188 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:21.184+0000 7fb28f59e700 1 -- 172.21.15.47:0/773931894 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb278099050 msgr2=0x7fb27809b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:21.188 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:21.184+0000 7fb28f59e700 1 --2- 172.21.15.47:0/773931894 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb278099050 0x7fb27809b500 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb2900727d0 tx=0x7fb284005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:21.189 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:21.184+0000 7fb28f59e700 1 -- 172.21.15.47:0/773931894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2900fdea0 msgr2=0x7fb290070fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:21.189 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:21.184+0000 7fb28f59e700 1 --2- 172.21.15.47:0/773931894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2900fdea0 0x7fb290070fe0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb28000e6c0 tx=0x7fb28000e9d0 comp rx=0 tx=0).stop 2022-09-07T21:39:21.189 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:21.184+0000 7fb28f59e700 1 -- 172.21.15.47:0/773931894 shutdown_connections 2022-09-07T21:39:21.190 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:21.184+0000 7fb28f59e700 1 --2- 172.21.15.47:0/773931894 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb278099050 0x7fb27809b500 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:21.190 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:21.184+0000 7fb28f59e700 1 --2- 172.21.15.47:0/773931894 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb290102fa0 0x7fb2901a1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:21.190 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:21.184+0000 7fb28f59e700 1 --2- 172.21.15.47:0/773931894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2900fdea0 0x7fb290070fe0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:21.191 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:21.184+0000 7fb28f59e700 1 --2- 172.21.15.47:0/773931894 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2900fe840 0x7fb290071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:21.191 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:21.184+0000 7fb28f59e700 1 -- 172.21.15.47:0/773931894 >> 172.21.15.47:0/773931894 conn(0x7fb2900f9800 msgr2=0x7fb2900fa300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:21.191 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:21.185+0000 7fb28f59e700 1 -- 172.21.15.47:0/773931894 shutdown_connections 2022-09-07T21:39:21.191 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:21.185+0000 7fb28f59e700 1 -- 172.21.15.47:0/773931894 wait complete. 2022-09-07T21:39:21.707 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2022-09-07T21:39:21.707 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd dump --format=json 2022-09-07T21:39:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:21 smithi059 ceph-mon[97450]: pgmap v336: 129 pgs: 129 active+clean; 636 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 4.0 KiB/s wr, 10 op/s 2022-09-07T21:39:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:21 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/773931894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:22.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:21 smithi047 ceph-mon[90087]: pgmap v336: 129 pgs: 129 active+clean; 636 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 4.0 KiB/s wr, 10 op/s 2022-09-07T21:39:22.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:21 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/773931894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:21 smithi055 ceph-mon[97033]: pgmap v336: 129 pgs: 129 active+clean; 636 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 4.0 KiB/s wr, 10 op/s 2022-09-07T21:39:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:21 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/773931894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:22.599 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:23.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:23 smithi047 ceph-mon[90087]: pgmap v337: 129 pgs: 129 active+clean; 636 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2022-09-07T21:39:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:23 smithi059 ceph-mon[97450]: pgmap v337: 129 pgs: 129 active+clean; 636 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2022-09-07T21:39:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:23 smithi055 ceph-mon[97033]: pgmap v337: 129 pgs: 129 active+clean; 636 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2022-09-07T21:39:24.415 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.409+0000 7f0e011c6700 1 -- 172.21.15.47:0/1108594041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dfc10a500 msgr2=0x7f0dfc10c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:24.415 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.409+0000 7f0e011c6700 1 --2- 172.21.15.47:0/1108594041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dfc10a500 0x7f0dfc10c8e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009bb0 tx=0x7f0df4009ec0 comp rx=0 tx=0).stop 2022-09-07T21:39:24.416 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.409+0000 7f0e011c6700 1 -- 172.21.15.47:0/1108594041 shutdown_connections 2022-09-07T21:39:24.416 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.409+0000 7f0e011c6700 1 --2- 172.21.15.47:0/1108594041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dfc10a500 0x7f0dfc10c8e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:24.416 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.409+0000 7f0e011c6700 1 --2- 172.21.15.47:0/1108594041 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0dfc071080 0x7f0dfc071450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:24.416 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.409+0000 7f0e011c6700 1 --2- 172.21.15.47:0/1108594041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0dfc071990 0x7f0dfc109fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:24.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.409+0000 7f0e011c6700 1 -- 172.21.15.47:0/1108594041 >> 172.21.15.47:0/1108594041 conn(0x7f0dfc0f9800 msgr2=0x7f0dfc0fbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:24.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.409+0000 7f0e011c6700 1 -- 172.21.15.47:0/1108594041 shutdown_connections 2022-09-07T21:39:24.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.409+0000 7f0e011c6700 1 -- 172.21.15.47:0/1108594041 wait complete. 2022-09-07T21:39:24.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.410+0000 7f0e011c6700 1 Processor -- start 2022-09-07T21:39:24.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.411+0000 7f0e011c6700 1 -- start start 2022-09-07T21:39:24.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.411+0000 7f0e011c6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dfc071080 0x7f0dfc195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:24.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.411+0000 7f0e011c6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0dfc071990 0x7f0dfc196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:24.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.411+0000 7f0e011c6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0dfc10a500 0x7f0dfc19a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:24.419 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.411+0000 7f0e011c6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc19ad80 con 0x7f0dfc071080 2022-09-07T21:39:24.419 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.411+0000 7f0e011c6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc19aef0 con 0x7f0dfc10a500 2022-09-07T21:39:24.419 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.411+0000 7f0e011c6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc19b0b0 con 0x7f0dfc071990 2022-09-07T21:39:24.419 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.411+0000 7f0dfbfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dfc071080 0x7f0dfc195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:24.420 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.411+0000 7f0e009c5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0dfc10a500 0x7f0dfc19a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:24.420 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.412+0000 7f0dfbfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dfc071080 0x7f0dfc195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60600/0 (socket says 172.21.15.47:60600) 2022-09-07T21:39:24.420 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.412+0000 7f0dfbfff700 1 -- 172.21.15.47:0/4197953359 learned_addr learned my addr 172.21.15.47:0/4197953359 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:24.420 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.412+0000 7f0e009c5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0dfc10a500 0x7f0dfc19a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:52110/0 (socket says 172.21.15.47:52110) 2022-09-07T21:39:24.420 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.412+0000 7f0dfb7fe700 1 --2- 172.21.15.47:0/4197953359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0dfc071990 0x7f0dfc196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:24.421 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.412+0000 7f0e009c5700 1 -- 172.21.15.47:0/4197953359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0dfc071990 msgr2=0x7f0dfc196290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:24.421 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.412+0000 7f0e009c5700 1 --2- 172.21.15.47:0/4197953359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0dfc071990 0x7f0dfc196290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:24.421 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.412+0000 7f0e009c5700 1 -- 172.21.15.47:0/4197953359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dfc071080 msgr2=0x7f0dfc195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:24.421 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.412+0000 7f0e009c5700 1 --2- 172.21.15.47:0/4197953359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dfc071080 0x7f0dfc195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:24.422 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.412+0000 7f0e009c5700 1 -- 172.21.15.47:0/4197953359 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dec0093b0 con 0x7f0dfc10a500 2022-09-07T21:39:24.422 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.412+0000 7f0e009c5700 1 --2- 172.21.15.47:0/4197953359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0dfc10a500 0x7f0dfc19a6f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009ad0 tx=0x7f0df400b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:24.422 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.413+0000 7f0df97fa700 1 -- 172.21.15.47:0/4197953359 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4005450 con 0x7f0dfc10a500 2022-09-07T21:39:24.423 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.413+0000 7f0df97fa700 1 -- 172.21.15.47:0/4197953359 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0df400f460 con 0x7f0dfc10a500 2022-09-07T21:39:24.423 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.413+0000 7f0df97fa700 1 -- 172.21.15.47:0/4197953359 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df40176f0 con 0x7f0dfc10a500 2022-09-07T21:39:24.423 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.413+0000 7f0e011c6700 1 -- 172.21.15.47:0/4197953359 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df40097e0 con 0x7f0dfc10a500 2022-09-07T21:39:24.423 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.413+0000 7f0e011c6700 1 -- 172.21.15.47:0/4197953359 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dfc1a29a0 con 0x7f0dfc10a500 2022-09-07T21:39:24.424 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.414+0000 7f0e011c6700 1 -- 172.21.15.47:0/4197953359 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dfc04d480 con 0x7f0dfc10a500 2022-09-07T21:39:24.425 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.416+0000 7f0df97fa700 1 -- 172.21.15.47:0/4197953359 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f0df4020070 con 0x7f0dfc10a500 2022-09-07T21:39:24.425 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.417+0000 7f0df97fa700 1 --2- 172.21.15.47:0/4197953359 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0de40990d0 0x7f0de409b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:24.425 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.417+0000 7f0df97fa700 1 -- 172.21.15.47:0/4197953359 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f0df40c4180 con 0x7f0dfc10a500 2022-09-07T21:39:24.425 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.418+0000 7f0dfbfff700 1 --2- 172.21.15.47:0/4197953359 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0de40990d0 0x7f0de409b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:24.426 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.418+0000 7f0dfbfff700 1 --2- 172.21.15.47:0/4197953359 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0de40990d0 0x7f0de409b580 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009d40 tx=0x7f0dec005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:24.426 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.419+0000 7f0df97fa700 1 -- 172.21.15.47:0/4197953359 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f0df40905e0 con 0x7f0dfc10a500 2022-09-07T21:39:24.646 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.643+0000 7f0e011c6700 1 -- 172.21.15.47:0/4197953359 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0dfc1a2f50 con 0x7f0dfc10a500 2022-09-07T21:39:24.647 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.645+0000 7f0df97fa700 1 -- 172.21.15.47:0/4197953359 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23119 (secure 0 0 0) 0x7f0df4093c00 con 0x7f0dfc10a500 2022-09-07T21:39:24.648 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:39:24.648 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":89,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:38:45.529264+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-07T21:38:38.608937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-07T21:38:39.936252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-07T21:38:43.731797+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:39:24.650 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.647+0000 7f0e011c6700 1 -- 172.21.15.47:0/4197953359 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0de40990d0 msgr2=0x7f0de409b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:24.650 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.647+0000 7f0e011c6700 1 --2- 172.21.15.47:0/4197953359 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0de40990d0 0x7f0de409b580 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009d40 tx=0x7f0dec005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:24.650 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.647+0000 7f0e011c6700 1 -- 172.21.15.47:0/4197953359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0dfc10a500 msgr2=0x7f0dfc19a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:24.651 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.647+0000 7f0e011c6700 1 --2- 172.21.15.47:0/4197953359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0dfc10a500 0x7f0dfc19a6f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009ad0 tx=0x7f0df400b890 comp rx=0 tx=0).stop 2022-09-07T21:39:24.651 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.647+0000 7f0e011c6700 1 -- 172.21.15.47:0/4197953359 shutdown_connections 2022-09-07T21:39:24.652 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.647+0000 7f0e011c6700 1 --2- 172.21.15.47:0/4197953359 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0de40990d0 0x7f0de409b580 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:24.652 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.647+0000 7f0e011c6700 1 --2- 172.21.15.47:0/4197953359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0dfc10a500 0x7f0dfc19a6f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:24.653 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.647+0000 7f0e011c6700 1 --2- 172.21.15.47:0/4197953359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0dfc071080 0x7f0dfc195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:24.653 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.647+0000 7f0e011c6700 1 --2- 172.21.15.47:0/4197953359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0dfc071990 0x7f0dfc196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:24.653 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.647+0000 7f0e011c6700 1 -- 172.21.15.47:0/4197953359 >> 172.21.15.47:0/4197953359 conn(0x7f0dfc0f9800 msgr2=0x7f0dfc10c750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:24.654 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.647+0000 7f0e011c6700 1 -- 172.21.15.47:0/4197953359 shutdown_connections 2022-09-07T21:39:24.654 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:24.647+0000 7f0e011c6700 1 -- 172.21.15.47:0/4197953359 wait complete. 2022-09-07T21:39:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:24 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/4197953359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:25.103 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:24 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/4197953359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:25.174 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2022-09-07T21:39:25.175 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2022-09-07T21:39:25.176 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2022-09-07T21:39:25.176 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2022-09-07T21:39:25.177 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2022-09-07T21:39:25.178 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2022-09-07T21:39:25.179 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2022-09-07T21:39:25.179 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2022-09-07T21:39:25.180 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2022-09-07T21:39:25.181 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2022-09-07T21:39:25.181 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2022-09-07T21:39:25.182 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2022-09-07T21:39:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:24 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/4197953359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:39:25.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:25 smithi047 ceph-mon[90087]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2022-09-07T21:39:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:25 smithi055 ceph-mon[97033]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2022-09-07T21:39:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:25 smithi059 ceph-mon[97450]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2022-09-07T21:39:27.070 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:27.358 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:27.666 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:27.929 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:28.015 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:27 smithi047 ceph-mon[90087]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2022-09-07T21:39:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:27 smithi055 ceph-mon[97033]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2022-09-07T21:39:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:27 smithi059 ceph-mon[97450]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2022-09-07T21:39:28.295 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:29.238 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:29.341 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:29.734 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:29.894 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:29.926 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:30.090 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:29 smithi047 ceph-mon[90087]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2022-09-07T21:39:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:29 smithi055 ceph-mon[97033]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2022-09-07T21:39:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:29 smithi059 ceph-mon[97450]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2022-09-07T21:39:30.433 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.430+0000 7f94b7c3c700 1 -- 172.21.15.47:0/3117859280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b0066c00 msgr2=0x7f94b0067050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:30.434 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.430+0000 7f94b7c3c700 1 --2- 172.21.15.47:0/3117859280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b0066c00 0x7f94b0067050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f94a0009810 tx=0x7f94a0009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:30.434 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.430+0000 7f94b7c3c700 1 -- 172.21.15.47:0/3117859280 shutdown_connections 2022-09-07T21:39:30.434 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.430+0000 7f94b7c3c700 1 --2- 172.21.15.47:0/3117859280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94b01008d0 0x7f94b0100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:30.435 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.430+0000 7f94b7c3c700 1 --2- 172.21.15.47:0/3117859280 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94b0102d40 0x7f94b00666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:30.435 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.430+0000 7f94b7c3c700 1 --2- 172.21.15.47:0/3117859280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b0066c00 0x7f94b0067050 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:30.435 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.430+0000 7f94b7c3c700 1 -- 172.21.15.47:0/3117859280 >> 172.21.15.47:0/3117859280 conn(0x7f94b0073cf0 msgr2=0x7f94b00740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:30.436 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.430+0000 7f94b7c3c700 1 -- 172.21.15.47:0/3117859280 shutdown_connections 2022-09-07T21:39:30.436 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.430+0000 7f94b7c3c700 1 -- 172.21.15.47:0/3117859280 wait complete. 2022-09-07T21:39:30.439 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.432+0000 7f94b7c3c700 1 Processor -- start 2022-09-07T21:39:30.439 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.432+0000 7f94b7c3c700 1 -- start start 2022-09-07T21:39:30.439 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.433+0000 7f94b7c3c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94b0066c00 0x7f94b019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:30.440 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.433+0000 7f94b7c3c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b01008d0 0x7f94b019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:30.440 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.433+0000 7f94b7c3c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94b0102d40 0x7f94b019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:30.440 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.433+0000 7f94b7c3c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b019f270 con 0x7f94b0102d40 2022-09-07T21:39:30.440 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.433+0000 7f94b7c3c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b019f3e0 con 0x7f94b0066c00 2022-09-07T21:39:30.441 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.433+0000 7f94b7c3c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b019f5a0 con 0x7f94b01008d0 2022-09-07T21:39:30.441 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.433+0000 7f94b59d8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94b0066c00 0x7f94b019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:30.441 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.433+0000 7f94b51d7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b01008d0 0x7f94b019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:30.442 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.433+0000 7f94b59d8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94b0066c00 0x7f94b019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:47840/0 (socket says 172.21.15.47:47840) 2022-09-07T21:39:30.442 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.433+0000 7f94b59d8700 1 -- 172.21.15.47:0/1146621119 learned_addr learned my addr 172.21.15.47:0/1146621119 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:30.443 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.433+0000 7f94b61d9700 1 --2- 172.21.15.47:0/1146621119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94b0102d40 0x7f94b019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:30.443 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.434+0000 7f94b51d7700 1 -- 172.21.15.47:0/1146621119 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94b0066c00 msgr2=0x7f94b019a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:30.443 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.434+0000 7f94b51d7700 1 --2- 172.21.15.47:0/1146621119 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94b0066c00 0x7f94b019a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:30.444 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.434+0000 7f94b51d7700 1 -- 172.21.15.47:0/1146621119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94b0102d40 msgr2=0x7f94b019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:30.444 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.434+0000 7f94b51d7700 1 --2- 172.21.15.47:0/1146621119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94b0102d40 0x7f94b019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:30.445 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.434+0000 7f94b51d7700 1 -- 172.21.15.47:0/1146621119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a0009470 con 0x7f94b01008d0 2022-09-07T21:39:30.445 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.434+0000 7f94b61d9700 1 --2- 172.21.15.47:0/1146621119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94b0102d40 0x7f94b019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:30.446 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.434+0000 7f94b51d7700 1 --2- 172.21.15.47:0/1146621119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b01008d0 0x7f94b019a780 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f94a0000c00 tx=0x7f94a000bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:30.446 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.434+0000 7f94a6ffd700 1 -- 172.21.15.47:0/1146621119 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a001d070 con 0x7f94b01008d0 2022-09-07T21:39:30.446 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.434+0000 7f94a6ffd700 1 -- 172.21.15.47:0/1146621119 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f94a000f810 con 0x7f94b01008d0 2022-09-07T21:39:30.446 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.435+0000 7f94b7c3c700 1 -- 172.21.15.47:0/1146621119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b019f880 con 0x7f94b01008d0 2022-09-07T21:39:30.446 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.435+0000 7f94a6ffd700 1 -- 172.21.15.47:0/1146621119 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a0004740 con 0x7f94b01008d0 2022-09-07T21:39:30.447 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.435+0000 7f94b7c3c700 1 -- 172.21.15.47:0/1146621119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b01a70c0 con 0x7f94b01008d0 2022-09-07T21:39:30.447 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.436+0000 7f94b7c3c700 1 -- 172.21.15.47:0/1146621119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f94b01002c0 con 0x7f94b01008d0 2022-09-07T21:39:30.448 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.438+0000 7f94a6ffd700 1 -- 172.21.15.47:0/1146621119 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f94a0028030 con 0x7f94b01008d0 2022-09-07T21:39:30.449 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.439+0000 7f94a6ffd700 1 --2- 172.21.15.47:0/1146621119 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f949c0990d0 0x7f949c09b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:30.449 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.439+0000 7f94a6ffd700 1 -- 172.21.15.47:0/1146621119 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f94a00c38c0 con 0x7f94b01008d0 2022-09-07T21:39:30.449 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.439+0000 7f94b59d8700 1 --2- 172.21.15.47:0/1146621119 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f949c0990d0 0x7f949c09b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:30.449 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.440+0000 7f94a6ffd700 1 --2- 172.21.15.47:0/1146621119 >> [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956] conn(0x7f949c0a3bc0 0x7f949c0a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:30.449 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.440+0000 7f94a6ffd700 1 -- 172.21.15.47:0/1146621119 --> [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f949c0a66c0 con 0x7f949c0a3bc0 2022-09-07T21:39:30.450 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.440+0000 7f94a6ffd700 1 -- 172.21.15.47:0/1146621119 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94a00c3c30 con 0x7f94b01008d0 2022-09-07T21:39:30.450 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.440+0000 7f94b61d9700 1 --2- 172.21.15.47:0/1146621119 >> [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956] conn(0x7f949c0a3bc0 0x7f949c0a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:30.450 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.440+0000 7f94b59d8700 1 --2- 172.21.15.47:0/1146621119 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f949c0990d0 0x7f949c09b580 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0057d0 tx=0x7f94ac005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:30.451 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.441+0000 7f94b61d9700 1 --2- 172.21.15.47:0/1146621119 >> [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956] conn(0x7f949c0a3bc0 0x7f949c0a5ff0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:30.451 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.442+0000 7f94a6ffd700 1 -- 172.21.15.47:0/1146621119 <== osd.4 v2:172.21.15.55:6808/4055716956 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f949c0a66c0 con 0x7f949c0a3bc0 2022-09-07T21:39:30.463 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.460+0000 7f94b7c3c700 1 -- 172.21.15.47:0/1146621119 --> [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f94b0064ea0 con 0x7f949c0a3bc0 2022-09-07T21:39:30.463 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.461+0000 7f94a6ffd700 1 -- 172.21.15.47:0/1146621119 <== osd.4 v2:172.21.15.55:6808/4055716956 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f94b0064ea0 con 0x7f949c0a3bc0 2022-09-07T21:39:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.461+0000 7f94b7c3c700 1 -- 172.21.15.47:0/1146621119 >> [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956] conn(0x7f949c0a3bc0 msgr2=0x7f949c0a5ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.461+0000 7f94b7c3c700 1 --2- 172.21.15.47:0/1146621119 >> [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956] conn(0x7f949c0a3bc0 0x7f949c0a5ff0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.461+0000 7f94b7c3c700 1 -- 172.21.15.47:0/1146621119 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f949c0990d0 msgr2=0x7f949c09b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.461+0000 7f94b7c3c700 1 --2- 172.21.15.47:0/1146621119 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f949c0990d0 0x7f949c09b580 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0057d0 tx=0x7f94ac005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.461+0000 7f94b7c3c700 1 -- 172.21.15.47:0/1146621119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b01008d0 msgr2=0x7f94b019a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.461+0000 7f94b7c3c700 1 --2- 172.21.15.47:0/1146621119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b01008d0 0x7f94b019a780 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f94a0000c00 tx=0x7f94a000bfd0 comp rx=0 tx=0).stop 2022-09-07T21:39:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.461+0000 7f94b61d9700 1 -- 172.21.15.47:0/1146621119 reap_dead start 2022-09-07T21:39:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.461+0000 7f94b7c3c700 1 -- 172.21.15.47:0/1146621119 shutdown_connections 2022-09-07T21:39:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.461+0000 7f94b7c3c700 1 -- 172.21.15.47:0/1146621119 >> 172.21.15.47:0/1146621119 conn(0x7f94b0073cf0 msgr2=0x7f94b00ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.462+0000 7f94b7c3c700 1 -- 172.21.15.47:0/1146621119 shutdown_connections 2022-09-07T21:39:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:30.462+0000 7f94b7c3c700 1 -- 172.21.15.47:0/1146621119 wait complete. 2022-09-07T21:39:30.714 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:31.013 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:30 smithi047 ceph-mon[90087]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 5 op/s 2022-09-07T21:39:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:30 smithi055 ceph-mon[97033]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 5 op/s 2022-09-07T21:39:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:30 smithi059 ceph-mon[97450]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 5 op/s 2022-09-07T21:39:31.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.245+0000 7fb5b69f1700 1 -- 172.21.15.47:0/269274429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5b00fd510 msgr2=0x7fb5b0101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:31.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.245+0000 7fb5b69f1700 1 --2- 172.21.15.47:0/269274429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5b00fd510 0x7fb5b0101a70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009750 tx=0x7fb5a0009a60 comp rx=0 tx=0).stop 2022-09-07T21:39:31.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.245+0000 7fb5b69f1700 1 -- 172.21.15.47:0/269274429 shutdown_connections 2022-09-07T21:39:31.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.245+0000 7fb5b69f1700 1 --2- 172.21.15.47:0/269274429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5b0101fb0 0x7fb5b0102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:31.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.245+0000 7fb5b69f1700 1 --2- 172.21.15.47:0/269274429 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5b00fcc00 0x7fb5b00fcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:31.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.245+0000 7fb5b69f1700 1 --2- 172.21.15.47:0/269274429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5b00fd510 0x7fb5b0101a70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:31.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.245+0000 7fb5b69f1700 1 -- 172.21.15.47:0/269274429 >> 172.21.15.47:0/269274429 conn(0x7fb5b00f8710 msgr2=0x7fb5b00fab20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:31.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.245+0000 7fb5b69f1700 1 -- 172.21.15.47:0/269274429 shutdown_connections 2022-09-07T21:39:31.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.245+0000 7fb5b69f1700 1 -- 172.21.15.47:0/269274429 wait complete. 2022-09-07T21:39:31.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.246+0000 7fb5b69f1700 1 Processor -- start 2022-09-07T21:39:31.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.247+0000 7fb5b69f1700 1 -- start start 2022-09-07T21:39:31.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.247+0000 7fb5b69f1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5b00fcc00 0x7fb5b019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:31.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.247+0000 7fb5b69f1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5b00fd510 0x7fb5b019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:31.573 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.247+0000 7fb5b69f1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5b0101fb0 0x7fb5b019eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:31.573 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.247+0000 7fb5b69f1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b019f150 con 0x7fb5b00fd510 2022-09-07T21:39:31.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.247+0000 7fb5b69f1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b019f2c0 con 0x7fb5b0101fb0 2022-09-07T21:39:31.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.247+0000 7fb5b69f1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b019f480 con 0x7fb5b00fcc00 2022-09-07T21:39:31.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.247+0000 7fb5b51ee700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5b00fd510 0x7fb5b019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:31.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.247+0000 7fb5b59ef700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5b00fcc00 0x7fb5b019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:31.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.247+0000 7fb5b51ee700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5b00fd510 0x7fb5b019a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52402/0 (socket says 172.21.15.47:52402) 2022-09-07T21:39:31.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.247+0000 7fb5b59ef700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5b00fcc00 0x7fb5b019a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:41554/0 (socket says 172.21.15.47:41554) 2022-09-07T21:39:31.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.247+0000 7fb5b51ee700 1 -- 172.21.15.47:0/1896186618 learned_addr learned my addr 172.21.15.47:0/1896186618 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:31.577 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.248+0000 7fb5b61f0700 1 --2- 172.21.15.47:0/1896186618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5b0101fb0 0x7fb5b019eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:31.577 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.248+0000 7fb5b59ef700 1 -- 172.21.15.47:0/1896186618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5b0101fb0 msgr2=0x7fb5b019eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:31.577 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.248+0000 7fb5b59ef700 1 --2- 172.21.15.47:0/1896186618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5b0101fb0 0x7fb5b019eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:31.578 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.248+0000 7fb5b59ef700 1 -- 172.21.15.47:0/1896186618 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5b00fd510 msgr2=0x7fb5b019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:31.578 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.248+0000 7fb5b59ef700 1 --2- 172.21.15.47:0/1896186618 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5b00fd510 0x7fb5b019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:31.579 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.248+0000 7fb5b59ef700 1 -- 172.21.15.47:0/1896186618 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac009470 con 0x7fb5b00fcc00 2022-09-07T21:39:31.579 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.248+0000 7fb5b59ef700 1 --2- 172.21.15.47:0/1896186618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5b00fcc00 0x7fb5b019a120 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac005fd0 tx=0x7fb5ac00eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:31.579 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.249+0000 7fb5a6ffd700 1 -- 172.21.15.47:0/1896186618 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac004220 con 0x7fb5b00fcc00 2022-09-07T21:39:31.579 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.249+0000 7fb5b69f1700 1 -- 172.21.15.47:0/1896186618 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a00093b0 con 0x7fb5b00fcc00 2022-09-07T21:39:31.580 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.249+0000 7fb5a6ffd700 1 -- 172.21.15.47:0/1896186618 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb5ac004380 con 0x7fb5b00fcc00 2022-09-07T21:39:31.580 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.249+0000 7fb5a6ffd700 1 -- 172.21.15.47:0/1896186618 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac003c10 con 0x7fb5b00fcc00 2022-09-07T21:39:31.580 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.249+0000 7fb5b69f1700 1 -- 172.21.15.47:0/1896186618 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b01a6f50 con 0x7fb5b00fcc00 2022-09-07T21:39:31.580 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.250+0000 7fb5b69f1700 1 -- 172.21.15.47:0/1896186618 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb5b0101500 con 0x7fb5b00fcc00 2022-09-07T21:39:31.580 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.252+0000 7fb5a6ffd700 1 -- 172.21.15.47:0/1896186618 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7fb5ac01e020 con 0x7fb5b00fcc00 2022-09-07T21:39:31.581 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.253+0000 7fb5a6ffd700 1 --2- 172.21.15.47:0/1896186618 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb59c0990d0 0x7fb59c09b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:31.581 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.253+0000 7fb5a6ffd700 1 -- 172.21.15.47:0/1896186618 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fb5ac014070 con 0x7fb5b00fcc00 2022-09-07T21:39:31.581 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.253+0000 7fb5b51ee700 1 --2- 172.21.15.47:0/1896186618 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb59c0990d0 0x7fb59c09b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:31.581 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.254+0000 7fb5a6ffd700 1 --2- 172.21.15.47:0/1896186618 >> [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539] conn(0x7fb59c0a3bc0 0x7fb59c0a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:31.582 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.254+0000 7fb5a6ffd700 1 -- 172.21.15.47:0/1896186618 --> [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb59c0a66c0 con 0x7fb59c0a3bc0 2022-09-07T21:39:31.582 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.254+0000 7fb5a6ffd700 1 -- 172.21.15.47:0/1896186618 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5ac0c3080 con 0x7fb5b00fcc00 2022-09-07T21:39:31.582 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.254+0000 7fb5b61f0700 1 --2- 172.21.15.47:0/1896186618 >> [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539] conn(0x7fb59c0a3bc0 0x7fb59c0a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:31.582 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.254+0000 7fb5b51ee700 1 --2- 172.21.15.47:0/1896186618 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb59c0990d0 0x7fb59c09b580 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb5b019b730 tx=0x7fb5a00092f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:31.583 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.255+0000 7fb5b61f0700 1 --2- 172.21.15.47:0/1896186618 >> [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539] conn(0x7fb59c0a3bc0 0x7fb59c0a5ff0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:31.583 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.255+0000 7fb5a6ffd700 1 -- 172.21.15.47:0/1896186618 <== osd.5 v2:172.21.15.59:6808/725179539 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7fb59c0a66c0 con 0x7fb59c0a3bc0 2022-09-07T21:39:31.583 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.272+0000 7fb5b69f1700 1 -- 172.21.15.47:0/1896186618 --> [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb5b004d480 con 0x7fb59c0a3bc0 2022-09-07T21:39:31.583 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.273+0000 7fb5a6ffd700 1 -- 172.21.15.47:0/1896186618 <== osd.5 v2:172.21.15.59:6808/725179539 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb5b004d480 con 0x7fb59c0a3bc0 2022-09-07T21:39:31.584 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.273+0000 7fb5b69f1700 1 -- 172.21.15.47:0/1896186618 >> [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539] conn(0x7fb59c0a3bc0 msgr2=0x7fb59c0a5ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:31.585 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.273+0000 7fb5b69f1700 1 --2- 172.21.15.47:0/1896186618 >> [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539] conn(0x7fb59c0a3bc0 0x7fb59c0a5ff0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:31.585 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.274+0000 7fb5b69f1700 1 -- 172.21.15.47:0/1896186618 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb59c0990d0 msgr2=0x7fb59c09b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:31.585 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.274+0000 7fb5b69f1700 1 --2- 172.21.15.47:0/1896186618 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fb59c0990d0 0x7fb59c09b580 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb5b019b730 tx=0x7fb5a00092f0 comp rx=0 tx=0).stop 2022-09-07T21:39:31.585 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.274+0000 7fb5b69f1700 1 -- 172.21.15.47:0/1896186618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5b00fcc00 msgr2=0x7fb5b019a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:31.586 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.274+0000 7fb5b69f1700 1 --2- 172.21.15.47:0/1896186618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5b00fcc00 0x7fb5b019a120 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac005fd0 tx=0x7fb5ac00eca0 comp rx=0 tx=0).stop 2022-09-07T21:39:31.586 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.274+0000 7fb5b61f0700 1 -- 172.21.15.47:0/1896186618 reap_dead start 2022-09-07T21:39:31.586 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.274+0000 7fb5b69f1700 1 -- 172.21.15.47:0/1896186618 shutdown_connections 2022-09-07T21:39:31.586 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.274+0000 7fb5b69f1700 1 -- 172.21.15.47:0/1896186618 >> 172.21.15.47:0/1896186618 conn(0x7fb5b00f8710 msgr2=0x7fb5b00ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:31.587 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.274+0000 7fb5b69f1700 1 -- 172.21.15.47:0/1896186618 shutdown_connections 2022-09-07T21:39:31.587 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:31.274+0000 7fb5b69f1700 1 -- 172.21.15.47:0/1896186618 wait complete. 2022-09-07T21:39:32.000 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:33.456 INFO:teuthology.orchestra.run.smithi047.stdout:141733920830 2022-09-07T21:39:33.457 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.4 2022-09-07T21:39:33.925 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:33 smithi047 ceph-mon[90087]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 6 op/s 2022-09-07T21:39:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:33 smithi055 ceph-mon[97033]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 6 op/s 2022-09-07T21:39:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:33 smithi059 ceph-mon[97450]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 6 op/s 2022-09-07T21:39:35.071 INFO:teuthology.orchestra.run.smithi047.stdout:163208757306 2022-09-07T21:39:35.072 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.5 2022-09-07T21:39:35.879 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:35 smithi047 ceph-mon[90087]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 7 op/s 2022-09-07T21:39:35.891 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.535+0000 7f90508f0700 1 -- 172.21.15.47:0/3958386085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c066fd0 msgr2=0x7f904c10a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:35.891 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.535+0000 7f90508f0700 1 --2- 172.21.15.47:0/3958386085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c066fd0 0x7f904c10a4d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9040009750 tx=0x7f9040009a60 comp rx=0 tx=0).stop 2022-09-07T21:39:35.892 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.535+0000 7f90508f0700 1 -- 172.21.15.47:0/3958386085 shutdown_connections 2022-09-07T21:39:35.892 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.535+0000 7f90508f0700 1 --2- 172.21.15.47:0/3958386085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f904c10aa10 0x7f904c10cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:35.892 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.535+0000 7f90508f0700 1 --2- 172.21.15.47:0/3958386085 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f904c0666c0 0x7f904c066a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:35.892 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.535+0000 7f90508f0700 1 --2- 172.21.15.47:0/3958386085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c066fd0 0x7f904c10a4d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:35.893 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.535+0000 7f90508f0700 1 -- 172.21.15.47:0/3958386085 >> 172.21.15.47:0/3958386085 conn(0x7f904c073cf0 msgr2=0x7f904c0740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:35.893 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.536+0000 7f90508f0700 1 -- 172.21.15.47:0/3958386085 shutdown_connections 2022-09-07T21:39:35.893 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.536+0000 7f90508f0700 1 -- 172.21.15.47:0/3958386085 wait complete. 2022-09-07T21:39:35.893 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.537+0000 7f90508f0700 1 Processor -- start 2022-09-07T21:39:35.893 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.538+0000 7f90508f0700 1 -- start start 2022-09-07T21:39:35.894 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.538+0000 7f90508f0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f904c0666c0 0x7f904c195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:35.894 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.538+0000 7f90508f0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f904c066fd0 0x7f904c196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:35.894 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.538+0000 7f90508f0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c10aa10 0x7f904c19a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:35.894 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.538+0000 7f90508f0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c19af30 con 0x7f904c0666c0 2022-09-07T21:39:35.895 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.538+0000 7f90508f0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c19b0a0 con 0x7f904c066fd0 2022-09-07T21:39:35.895 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.538+0000 7f90508f0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c19b260 con 0x7f904c10aa10 2022-09-07T21:39:35.895 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.539+0000 7f904ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c10aa10 0x7f904c19a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:35.895 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.539+0000 7f9049d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f904c066fd0 0x7f904c196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:35.895 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.539+0000 7f904ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c10aa10 0x7f904c19a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:41584/0 (socket says 172.21.15.47:41584) 2022-09-07T21:39:35.896 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.539+0000 7f904ad9d700 1 -- 172.21.15.47:0/1851013259 learned_addr learned my addr 172.21.15.47:0/1851013259 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:35.896 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.539+0000 7f904a59c700 1 --2- 172.21.15.47:0/1851013259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f904c0666c0 0x7f904c195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:35.896 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.540+0000 7f904ad9d700 1 -- 172.21.15.47:0/1851013259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f904c066fd0 msgr2=0x7f904c196440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:35.896 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.540+0000 7f904ad9d700 1 --2- 172.21.15.47:0/1851013259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f904c066fd0 0x7f904c196440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:35.897 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.540+0000 7f904ad9d700 1 -- 172.21.15.47:0/1851013259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f904c0666c0 msgr2=0x7f904c195f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:35.897 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.540+0000 7f904ad9d700 1 --2- 172.21.15.47:0/1851013259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f904c0666c0 0x7f904c195f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:35.897 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.540+0000 7f904ad9d700 1 -- 172.21.15.47:0/1851013259 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90440097e0 con 0x7f904c10aa10 2022-09-07T21:39:35.897 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.540+0000 7f904ad9d700 1 --2- 172.21.15.47:0/1851013259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c10aa10 0x7f904c19a8a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9044009fd0 tx=0x7f904400eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:35.898 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.540+0000 7f903b7fe700 1 -- 172.21.15.47:0/1851013259 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904400cc90 con 0x7f904c10aa10 2022-09-07T21:39:35.898 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.540+0000 7f90508f0700 1 -- 172.21.15.47:0/1851013259 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90400093b0 con 0x7f904c10aa10 2022-09-07T21:39:35.898 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.541+0000 7f903b7fe700 1 -- 172.21.15.47:0/1851013259 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f904400cdf0 con 0x7f904c10aa10 2022-09-07T21:39:35.898 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.541+0000 7f903b7fe700 1 -- 172.21.15.47:0/1851013259 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90440042b0 con 0x7f904c10aa10 2022-09-07T21:39:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.541+0000 7f90508f0700 1 -- 172.21.15.47:0/1851013259 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904c19b6e0 con 0x7f904c10aa10 2022-09-07T21:39:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.542+0000 7f90508f0700 1 -- 172.21.15.47:0/1851013259 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f904c0fb860 con 0x7f904c10aa10 2022-09-07T21:39:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.544+0000 7f903b7fe700 1 -- 172.21.15.47:0/1851013259 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f904400c040 con 0x7f904c10aa10 2022-09-07T21:39:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.546+0000 7f903b7fe700 1 --2- 172.21.15.47:0/1851013259 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9034099050 0x7f903409b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:35.900 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.546+0000 7f903b7fe700 1 -- 172.21.15.47:0/1851013259 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f9044014070 con 0x7f904c10aa10 2022-09-07T21:39:35.900 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.546+0000 7f904a59c700 1 --2- 172.21.15.47:0/1851013259 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9034099050 0x7f903409b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:35.900 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.546+0000 7f903b7fe700 1 --2- 172.21.15.47:0/1851013259 >> [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578] conn(0x7f90340a3b40 0x7f90340a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:35.900 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.546+0000 7f903b7fe700 1 -- 172.21.15.47:0/1851013259 --> [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90340a6640 con 0x7f90340a3b40 2022-09-07T21:39:35.900 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.546+0000 7f903b7fe700 1 -- 172.21.15.47:0/1851013259 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90440c3080 con 0x7f904c10aa10 2022-09-07T21:39:35.901 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.547+0000 7f9049d9b700 1 --2- 172.21.15.47:0/1851013259 >> [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578] conn(0x7f90340a3b40 0x7f90340a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:35.901 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.547+0000 7f904a59c700 1 --2- 172.21.15.47:0/1851013259 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9034099050 0x7f903409b500 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f903c005fd0 tx=0x7f903c005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:35.901 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.547+0000 7f9049d9b700 1 --2- 172.21.15.47:0/1851013259 >> [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578] conn(0x7f90340a3b40 0x7f90340a5f70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:35.901 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.548+0000 7f903b7fe700 1 -- 172.21.15.47:0/1851013259 <== osd.2 v2:172.21.15.59:6800/2729613578 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f90340a6640 con 0x7f90340a3b40 2022-09-07T21:39:35.902 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.568+0000 7f90508f0700 1 -- 172.21.15.47:0/1851013259 --> [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f904c064ea0 con 0x7f90340a3b40 2022-09-07T21:39:35.902 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.569+0000 7f903b7fe700 1 -- 172.21.15.47:0/1851013259 <== osd.2 v2:172.21.15.59:6800/2729613578 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f904c064ea0 con 0x7f90340a3b40 2022-09-07T21:39:35.903 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.569+0000 7f90508f0700 1 -- 172.21.15.47:0/1851013259 >> [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578] conn(0x7f90340a3b40 msgr2=0x7f90340a5f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:35.903 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.569+0000 7f90508f0700 1 --2- 172.21.15.47:0/1851013259 >> [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578] conn(0x7f90340a3b40 0x7f90340a5f70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.569+0000 7f90508f0700 1 -- 172.21.15.47:0/1851013259 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9034099050 msgr2=0x7f903409b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.569+0000 7f90508f0700 1 --2- 172.21.15.47:0/1851013259 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9034099050 0x7f903409b500 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f903c005fd0 tx=0x7f903c005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.569+0000 7f90508f0700 1 -- 172.21.15.47:0/1851013259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c10aa10 msgr2=0x7f904c19a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.569+0000 7f90508f0700 1 --2- 172.21.15.47:0/1851013259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c10aa10 0x7f904c19a8a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9044009fd0 tx=0x7f904400eea0 comp rx=0 tx=0).stop 2022-09-07T21:39:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.569+0000 7f904ad9d700 1 -- 172.21.15.47:0/1851013259 reap_dead start 2022-09-07T21:39:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.569+0000 7f90508f0700 1 -- 172.21.15.47:0/1851013259 shutdown_connections 2022-09-07T21:39:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.569+0000 7f90508f0700 1 -- 172.21.15.47:0/1851013259 >> 172.21.15.47:0/1851013259 conn(0x7f904c073cf0 msgr2=0x7f904c0fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.570+0000 7f90508f0700 1 -- 172.21.15.47:0/1851013259 shutdown_connections 2022-09-07T21:39:35.906 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.570+0000 7f90508f0700 1 -- 172.21.15.47:0/1851013259 wait complete. 2022-09-07T21:39:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:35 smithi059 ceph-mon[97450]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 7 op/s 2022-09-07T21:39:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:35 smithi055 ceph-mon[97033]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 7 op/s 2022-09-07T21:39:36.404 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.480+0000 7f31534b5700 1 -- 172.21.15.47:0/3041683879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f314c105210 msgr2=0x7f314c10cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:36.404 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.480+0000 7f31534b5700 1 --2- 172.21.15.47:0/3041683879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f314c105210 0x7f314c10cdb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3144009bb0 tx=0x7f3144009ec0 comp rx=0 tx=0).stop 2022-09-07T21:39:36.404 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.481+0000 7f31534b5700 1 -- 172.21.15.47:0/3041683879 shutdown_connections 2022-09-07T21:39:36.405 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.481+0000 7f31534b5700 1 --2- 172.21.15.47:0/3041683879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f314c105210 0x7f314c10cdb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:36.405 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.481+0000 7f31534b5700 1 --2- 172.21.15.47:0/3041683879 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f314c100110 0x7f314c1004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:36.405 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.481+0000 7f31534b5700 1 --2- 172.21.15.47:0/3041683879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f314c100ab0 0x7f314c104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:36.405 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.481+0000 7f31534b5700 1 -- 172.21.15.47:0/3041683879 >> 172.21.15.47:0/3041683879 conn(0x7f314c0fba90 msgr2=0x7f314c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:36.406 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.481+0000 7f31534b5700 1 -- 172.21.15.47:0/3041683879 shutdown_connections 2022-09-07T21:39:36.406 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.481+0000 7f31534b5700 1 -- 172.21.15.47:0/3041683879 wait complete. 2022-09-07T21:39:36.406 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.483+0000 7f31534b5700 1 Processor -- start 2022-09-07T21:39:36.406 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.483+0000 7f31534b5700 1 -- start start 2022-09-07T21:39:36.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.483+0000 7f31534b5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f314c100110 0x7f314c19ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:36.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.483+0000 7f31534b5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f314c100ab0 0x7f314c19bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:36.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.484+0000 7f31534b5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f314c105210 0x7f314c195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:36.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.484+0000 7f31534b5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c196240 con 0x7f314c105210 2022-09-07T21:39:36.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.484+0000 7f31534b5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c1963b0 con 0x7f314c100ab0 2022-09-07T21:39:36.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.484+0000 7f31534b5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c196570 con 0x7f314c100110 2022-09-07T21:39:36.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.484+0000 7f31524b3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f314c100110 0x7f314c19ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:36.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.484+0000 7f3152cb4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f314c105210 0x7f314c195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:36.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.484+0000 7f3151cb2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f314c100ab0 0x7f314c19bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:36.409 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.484+0000 7f31524b3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f314c100110 0x7f314c19ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:41564/0 (socket says 172.21.15.47:41564) 2022-09-07T21:39:36.409 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.484+0000 7f31524b3700 1 -- 172.21.15.47:0/2352729220 learned_addr learned my addr 172.21.15.47:0/2352729220 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:36.409 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.485+0000 7f31524b3700 1 -- 172.21.15.47:0/2352729220 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f314c100ab0 msgr2=0x7f314c19bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:36.409 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.485+0000 7f31524b3700 1 --2- 172.21.15.47:0/2352729220 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f314c100ab0 0x7f314c19bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:36.410 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.485+0000 7f31524b3700 1 -- 172.21.15.47:0/2352729220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f314c105210 msgr2=0x7f314c195aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:36.410 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.485+0000 7f31524b3700 1 --2- 172.21.15.47:0/2352729220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f314c105210 0x7f314c195aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:36.410 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.485+0000 7f31524b3700 1 -- 172.21.15.47:0/2352729220 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31480093b0 con 0x7f314c100110 2022-09-07T21:39:36.410 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.485+0000 7f3151cb2700 1 --2- 172.21.15.47:0/2352729220 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f314c100ab0 0x7f314c19bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:36.411 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.485+0000 7f31524b3700 1 --2- 172.21.15.47:0/2352729220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f314c100110 0x7f314c19ba30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f314800e9c0 tx=0x7f314800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:36.411 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.485+0000 7f31437fe700 1 -- 172.21.15.47:0/2352729220 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31480097b0 con 0x7f314c100110 2022-09-07T21:39:36.411 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.485+0000 7f31437fe700 1 -- 172.21.15.47:0/2352729220 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f314800ce70 con 0x7f314c100110 2022-09-07T21:39:36.411 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.485+0000 7f31534b5700 1 -- 172.21.15.47:0/2352729220 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31440097e0 con 0x7f314c100110 2022-09-07T21:39:36.411 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.485+0000 7f31437fe700 1 -- 172.21.15.47:0/2352729220 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3148003cf0 con 0x7f314c100110 2022-09-07T21:39:36.412 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.485+0000 7f31534b5700 1 -- 172.21.15.47:0/2352729220 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f314c1a2a20 con 0x7f314c100110 2022-09-07T21:39:36.412 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.486+0000 7f31534b5700 1 -- 172.21.15.47:0/2352729220 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f314c0fcf10 con 0x7f314c100110 2022-09-07T21:39:36.412 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.489+0000 7f31437fe700 1 -- 172.21.15.47:0/2352729220 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f314801e020 con 0x7f314c100110 2022-09-07T21:39:36.412 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.491+0000 7f31437fe700 1 --2- 172.21.15.47:0/2352729220 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3138099050 0x7f313809b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:36.413 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.491+0000 7f31437fe700 1 -- 172.21.15.47:0/2352729220 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f3148014070 con 0x7f314c100110 2022-09-07T21:39:36.413 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.491+0000 7f31437fe700 1 --2- 172.21.15.47:0/2352729220 >> [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301] conn(0x7f31380a3b40 0x7f31380a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:36.413 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.491+0000 7f3151cb2700 1 --2- 172.21.15.47:0/2352729220 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3138099050 0x7f313809b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:36.413 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.491+0000 7f31437fe700 1 -- 172.21.15.47:0/2352729220 --> [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f31380a6640 con 0x7f31380a3b40 2022-09-07T21:39:36.414 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.491+0000 7f31437fe700 1 -- 172.21.15.47:0/2352729220 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31480c2fc0 con 0x7f314c100110 2022-09-07T21:39:36.414 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.492+0000 7f3152cb4700 1 --2- 172.21.15.47:0/2352729220 >> [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301] conn(0x7f31380a3b40 0x7f31380a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:36.415 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.492+0000 7f3151cb2700 1 --2- 172.21.15.47:0/2352729220 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3138099050 0x7f313809b500 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f314c0fefa0 tx=0x7f313c005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:36.415 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.492+0000 7f3152cb4700 1 --2- 172.21.15.47:0/2352729220 >> [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301] conn(0x7f31380a3b40 0x7f31380a5f70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:36.415 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.493+0000 7f31437fe700 1 -- 172.21.15.47:0/2352729220 <== osd.1 v2:172.21.15.55:6800/990490301 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f31380a6640 con 0x7f31380a3b40 2022-09-07T21:39:36.415 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.517+0000 7f31534b5700 1 -- 172.21.15.47:0/2352729220 --> [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f314c04d480 con 0x7f31380a3b40 2022-09-07T21:39:36.415 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.518+0000 7f31437fe700 1 -- 172.21.15.47:0/2352729220 <== osd.1 v2:172.21.15.55:6800/990490301 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f314c04d480 con 0x7f31380a3b40 2022-09-07T21:39:36.416 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.518+0000 7f31534b5700 1 -- 172.21.15.47:0/2352729220 >> [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301] conn(0x7f31380a3b40 msgr2=0x7f31380a5f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:36.416 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.518+0000 7f31534b5700 1 --2- 172.21.15.47:0/2352729220 >> [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301] conn(0x7f31380a3b40 0x7f31380a5f70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:36.416 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.518+0000 7f31534b5700 1 -- 172.21.15.47:0/2352729220 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3138099050 msgr2=0x7f313809b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:36.416 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.518+0000 7f31534b5700 1 --2- 172.21.15.47:0/2352729220 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3138099050 0x7f313809b500 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f314c0fefa0 tx=0x7f313c005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:36.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.518+0000 7f31534b5700 1 -- 172.21.15.47:0/2352729220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f314c100110 msgr2=0x7f314c19ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:36.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.518+0000 7f31534b5700 1 --2- 172.21.15.47:0/2352729220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f314c100110 0x7f314c19ba30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f314800e9c0 tx=0x7f314800ecd0 comp rx=0 tx=0).stop 2022-09-07T21:39:36.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.518+0000 7f3152cb4700 1 -- 172.21.15.47:0/2352729220 reap_dead start 2022-09-07T21:39:36.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.518+0000 7f31534b5700 1 -- 172.21.15.47:0/2352729220 shutdown_connections 2022-09-07T21:39:36.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.518+0000 7f31534b5700 1 -- 172.21.15.47:0/2352729220 >> 172.21.15.47:0/2352729220 conn(0x7f314c0fba90 msgr2=0x7f314c0fc500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:36.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.518+0000 7f31534b5700 1 -- 172.21.15.47:0/2352729220 shutdown_connections 2022-09-07T21:39:36.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:35.519+0000 7f31534b5700 1 -- 172.21.15.47:0/2352729220 wait complete. 2022-09-07T21:39:37.133 INFO:teuthology.orchestra.run.smithi047.stdout:77309411405 2022-09-07T21:39:37.133 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.2 2022-09-07T21:39:37.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.411+0000 7f60df435700 1 -- 172.21.15.47:0/3041815195 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60d810a580 msgr2=0x7f60d810c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:37.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.411+0000 7f60df435700 1 --2- 172.21.15.47:0/3041815195 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60d810a580 0x7f60d810c960 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009bb0 tx=0x7f60d0009ec0 comp rx=0 tx=0).stop 2022-09-07T21:39:37.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.412+0000 7f60df435700 1 -- 172.21.15.47:0/3041815195 shutdown_connections 2022-09-07T21:39:37.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.412+0000 7f60df435700 1 --2- 172.21.15.47:0/3041815195 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60d810a580 0x7f60d810c960 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:37.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.412+0000 7f60df435700 1 --2- 172.21.15.47:0/3041815195 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60d8071000 0x7f60d80713d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:37.530 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.412+0000 7f60df435700 1 --2- 172.21.15.47:0/3041815195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8071910 0x7f60d810a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:37.530 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.412+0000 7f60df435700 1 -- 172.21.15.47:0/3041815195 >> 172.21.15.47:0/3041815195 conn(0x7f60d80f9800 msgr2=0x7f60d80fbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:37.530 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.412+0000 7f60df435700 1 -- 172.21.15.47:0/3041815195 shutdown_connections 2022-09-07T21:39:37.530 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.412+0000 7f60df435700 1 -- 172.21.15.47:0/3041815195 wait complete. 2022-09-07T21:39:37.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.413+0000 7f60df435700 1 Processor -- start 2022-09-07T21:39:37.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.414+0000 7f60df435700 1 -- start start 2022-09-07T21:39:37.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.414+0000 7f60df435700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8071000 0x7f60d8195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:37.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.414+0000 7f60df435700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60d8071910 0x7f60d8196210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:37.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.414+0000 7f60df435700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60d810a580 0x7f60d81995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:37.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.414+0000 7f60df435700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d8199c20 con 0x7f60d8071910 2022-09-07T21:39:37.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.414+0000 7f60df435700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d8199d90 con 0x7f60d810a580 2022-09-07T21:39:37.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.414+0000 7f60df435700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d8199f50 con 0x7f60d8071000 2022-09-07T21:39:37.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.415+0000 7f60ddc32700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60d8071910 0x7f60d8196210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:37.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.415+0000 7f60ddc32700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60d8071910 0x7f60d8196210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52454/0 (socket says 172.21.15.47:52454) 2022-09-07T21:39:37.534 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.415+0000 7f60ddc32700 1 -- 172.21.15.47:0/1924941890 learned_addr learned my addr 172.21.15.47:0/1924941890 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:37.534 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.415+0000 7f60dec34700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60d810a580 0x7f60d81995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:37.534 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.415+0000 7f60de433700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8071000 0x7f60d8195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:37.534 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.415+0000 7f60dec34700 1 -- 172.21.15.47:0/1924941890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8071000 msgr2=0x7f60d8195cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:37.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.415+0000 7f60dec34700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8071000 0x7f60d8195cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:37.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.415+0000 7f60dec34700 1 -- 172.21.15.47:0/1924941890 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60d8071910 msgr2=0x7f60d8196210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:37.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.415+0000 7f60dec34700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60d8071910 0x7f60d8196210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:37.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.415+0000 7f60dec34700 1 -- 172.21.15.47:0/1924941890 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d40093b0 con 0x7f60d810a580 2022-09-07T21:39:37.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.415+0000 7f60ddc32700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60d8071910 0x7f60d8196210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:37.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.416+0000 7f60dec34700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60d810a580 0x7f60d81995c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009ad0 tx=0x7f60d000b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:37.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.416+0000 7f60cf7fe700 1 -- 172.21.15.47:0/1924941890 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d0005450 con 0x7f60d810a580 2022-09-07T21:39:37.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.416+0000 7f60df435700 1 -- 172.21.15.47:0/1924941890 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d00097e0 con 0x7f60d810a580 2022-09-07T21:39:37.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.416+0000 7f60cf7fe700 1 -- 172.21.15.47:0/1924941890 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f60d000f460 con 0x7f60d810a580 2022-09-07T21:39:37.537 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.416+0000 7f60cf7fe700 1 -- 172.21.15.47:0/1924941890 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d00176f0 con 0x7f60d810a580 2022-09-07T21:39:37.537 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.416+0000 7f60df435700 1 -- 172.21.15.47:0/1924941890 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d81a2950 con 0x7f60d810a580 2022-09-07T21:39:37.537 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.417+0000 7f60df435700 1 -- 172.21.15.47:0/1924941890 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f60d80fb790 con 0x7f60d810a580 2022-09-07T21:39:37.537 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.419+0000 7f60cf7fe700 1 -- 172.21.15.47:0/1924941890 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f60d0020070 con 0x7f60d810a580 2022-09-07T21:39:37.538 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.420+0000 7f60cf7fe700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f60c4099050 0x7f60c409b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:37.538 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.421+0000 7f60cf7fe700 1 -- 172.21.15.47:0/1924941890 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f60d00c44e0 con 0x7f60d810a580 2022-09-07T21:39:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.421+0000 7f60de433700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f60c4099050 0x7f60c409b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.421+0000 7f60cf7fe700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165] conn(0x7f60c40a3b40 0x7f60c40a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.421+0000 7f60cf7fe700 1 -- 172.21.15.47:0/1924941890 --> [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60c40a6640 con 0x7f60c40a3b40 2022-09-07T21:39:37.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.421+0000 7f60cf7fe700 1 -- 172.21.15.47:0/1924941890 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60d00c4850 con 0x7f60d810a580 2022-09-07T21:39:37.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.421+0000 7f60ddc32700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165] conn(0x7f60c40a3b40 0x7f60c40a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:37.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.422+0000 7f60de433700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f60c4099050 0x7f60c409b500 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f60d4005fd0 tx=0x7f60d4005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:37.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.422+0000 7f60ddc32700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165] conn(0x7f60c40a3b40 0x7f60c40a5f70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:37.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.423+0000 7f60cf7fe700 1 -- 172.21.15.47:0/1924941890 <== osd.6 v2:172.21.15.47:6818/4053448165 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f60c40a6640 con 0x7f60c40a3b40 2022-09-07T21:39:37.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.440+0000 7f60df435700 1 -- 172.21.15.47:0/1924941890 --> [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f60d804d480 con 0x7f60c40a3b40 2022-09-07T21:39:37.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.440+0000 7f60cf7fe700 1 -- 172.21.15.47:0/1924941890 <== osd.6 v2:172.21.15.47:6818/4053448165 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f60d804d480 con 0x7f60c40a3b40 2022-09-07T21:39:37.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.441+0000 7f60df435700 1 -- 172.21.15.47:0/1924941890 >> [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165] conn(0x7f60c40a3b40 msgr2=0x7f60c40a5f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:37.542 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.441+0000 7f60df435700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165] conn(0x7f60c40a3b40 0x7f60c40a5f70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:37.542 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.441+0000 7f60df435700 1 -- 172.21.15.47:0/1924941890 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f60c4099050 msgr2=0x7f60c409b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:37.542 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.441+0000 7f60df435700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f60c4099050 0x7f60c409b500 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f60d4005fd0 tx=0x7f60d4005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:37.542 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.441+0000 7f60df435700 1 -- 172.21.15.47:0/1924941890 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60d810a580 msgr2=0x7f60d81995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:37.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.441+0000 7f60df435700 1 --2- 172.21.15.47:0/1924941890 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60d810a580 0x7f60d81995c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009ad0 tx=0x7f60d000b890 comp rx=0 tx=0).stop 2022-09-07T21:39:37.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.441+0000 7f60dec34700 1 -- 172.21.15.47:0/1924941890 reap_dead start 2022-09-07T21:39:37.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.441+0000 7f60df435700 1 -- 172.21.15.47:0/1924941890 shutdown_connections 2022-09-07T21:39:37.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.441+0000 7f60df435700 1 -- 172.21.15.47:0/1924941890 >> 172.21.15.47:0/1924941890 conn(0x7f60d80f9800 msgr2=0x7f60d810c770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:37.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.441+0000 7f60df435700 1 -- 172.21.15.47:0/1924941890 shutdown_connections 2022-09-07T21:39:37.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:37.441+0000 7f60df435700 1 -- 172.21.15.47:0/1924941890 wait complete. 2022-09-07T21:39:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:37 smithi059 ceph-mon[97450]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-07T21:39:38.086 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:37 smithi047 ceph-mon[90087]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-07T21:39:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:37 smithi055 ceph-mon[97033]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-07T21:39:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.564+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/4184384447 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2dd00fe0d0 msgr2=0x7f2dd00fe4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.564+0000 7f2dd4a8c700 1 --2- 172.21.15.47:0/4184384447 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2dd00fe0d0 0x7f2dd00fe4a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0009840 tx=0x7f2dc0009b50 comp rx=0 tx=0).stop 2022-09-07T21:39:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.564+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/4184384447 shutdown_connections 2022-09-07T21:39:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.564+0000 7f2dd4a8c700 1 --2- 172.21.15.47:0/4184384447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dd01031d0 0x7f2dd010ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.564+0000 7f2dd4a8c700 1 --2- 172.21.15.47:0/4184384447 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2dd00fe0d0 0x7f2dd00fe4a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.564+0000 7f2dd4a8c700 1 --2- 172.21.15.47:0/4184384447 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dd00fea70 0x7f2dd0102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.564+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/4184384447 >> 172.21.15.47:0/4184384447 conn(0x7f2dd00f9a10 msgr2=0x7f2dd00fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.564+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/4184384447 shutdown_connections 2022-09-07T21:39:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.565+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/4184384447 wait complete. 2022-09-07T21:39:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.566+0000 7f2dd4a8c700 1 Processor -- start 2022-09-07T21:39:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.567+0000 7f2dd4a8c700 1 -- start start 2022-09-07T21:39:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.567+0000 7f2dd4a8c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2dd00fe0d0 0x7f2dd0195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.567+0000 7f2dd4a8c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dd00fea70 0x7f2dd0196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.567+0000 7f2dd4a8c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dd01031d0 0x7f2dd019a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.567+0000 7f2dd4a8c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd019af50 con 0x7f2dd00fe0d0 2022-09-07T21:39:38.573 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.567+0000 7f2dd4a8c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd019b0c0 con 0x7f2dd01031d0 2022-09-07T21:39:38.573 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.567+0000 7f2dd4a8c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd019b280 con 0x7f2dd00fea70 2022-09-07T21:39:38.573 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.568+0000 7f2dce59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2dd00fe0d0 0x7f2dd0195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.573 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.568+0000 7f2dce59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2dd00fe0d0 0x7f2dd0195f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52484/0 (socket says 172.21.15.47:52484) 2022-09-07T21:39:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.568+0000 7f2dce59c700 1 -- 172.21.15.47:0/372880007 learned_addr learned my addr 172.21.15.47:0/372880007 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.568+0000 7f2dced9d700 1 --2- 172.21.15.47:0/372880007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dd01031d0 0x7f2dd019a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.568+0000 7f2dcdd9b700 1 --2- 172.21.15.47:0/372880007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dd00fea70 0x7f2dd0196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.568+0000 7f2dce59c700 1 -- 172.21.15.47:0/372880007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dd00fea70 msgr2=0x7f2dd0196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.568+0000 7f2dce59c700 1 --2- 172.21.15.47:0/372880007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dd00fea70 0x7f2dd0196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.568+0000 7f2dce59c700 1 -- 172.21.15.47:0/372880007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dd01031d0 msgr2=0x7f2dd019a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.568+0000 7f2dce59c700 1 --2- 172.21.15.47:0/372880007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dd01031d0 0x7f2dd019a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.568+0000 7f2dce59c700 1 -- 172.21.15.47:0/372880007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc0009470 con 0x7f2dd00fe0d0 2022-09-07T21:39:38.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.569+0000 7f2dce59c700 1 --2- 172.21.15.47:0/372880007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2dd00fe0d0 0x7f2dd0195f20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c010 tx=0x7f2dc000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:38.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.569+0000 7f2dbf7fe700 1 -- 172.21.15.47:0/372880007 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc001d070 con 0x7f2dd00fe0d0 2022-09-07T21:39:38.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.569+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/372880007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd019b500 con 0x7f2dd00fe0d0 2022-09-07T21:39:38.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.569+0000 7f2dbf7fe700 1 -- 172.21.15.47:0/372880007 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2dc000b480 con 0x7f2dd00fe0d0 2022-09-07T21:39:38.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.569+0000 7f2dbf7fe700 1 -- 172.21.15.47:0/372880007 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0004760 con 0x7f2dd00fe0d0 2022-09-07T21:39:38.579 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.569+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/372880007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd01a2d50 con 0x7f2dd00fe0d0 2022-09-07T21:39:38.579 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.570+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/372880007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2dd00faed0 con 0x7f2dd00fe0d0 2022-09-07T21:39:38.579 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.572+0000 7f2dbf7fe700 1 -- 172.21.15.47:0/372880007 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f2dc0028030 con 0x7f2dd00fe0d0 2022-09-07T21:39:38.579 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.573+0000 7f2dbf7fe700 1 --2- 172.21.15.47:0/372880007 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2db80990d0 0x7f2db809b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.580 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.574+0000 7f2dbf7fe700 1 -- 172.21.15.47:0/372880007 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f2dc00c3880 con 0x7f2dd00fe0d0 2022-09-07T21:39:38.580 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.574+0000 7f2dcdd9b700 1 --2- 172.21.15.47:0/372880007 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2db80990d0 0x7f2db809b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.580 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.574+0000 7f2dbf7fe700 1 --2- 172.21.15.47:0/372880007 >> [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579] conn(0x7f2db80a3bc0 0x7f2db80a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.580 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.574+0000 7f2dbf7fe700 1 -- 172.21.15.47:0/372880007 --> [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2db80a66c0 con 0x7f2db80a3bc0 2022-09-07T21:39:38.581 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.574+0000 7f2dbf7fe700 1 -- 172.21.15.47:0/372880007 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2dc00c3bf0 con 0x7f2dd00fe0d0 2022-09-07T21:39:38.581 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.575+0000 7f2dced9d700 1 --2- 172.21.15.47:0/372880007 >> [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579] conn(0x7f2db80a3bc0 0x7f2db80a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.581 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.575+0000 7f2dcdd9b700 1 --2- 172.21.15.47:0/372880007 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2db80990d0 0x7f2db809b580 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0197530 tx=0x7f2dc4005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:38.581 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.575+0000 7f2dced9d700 1 --2- 172.21.15.47:0/372880007 >> [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579] conn(0x7f2db80a3bc0 0x7f2db80a5ff0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:38.581 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.576+0000 7f2dbf7fe700 1 -- 172.21.15.47:0/372880007 <== osd.3 v2:172.21.15.47:6810/1063754579 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f2db80a66c0 con 0x7f2db80a3bc0 2022-09-07T21:39:38.597 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.594+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/372880007 --> [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2dd0064ea0 con 0x7f2db80a3bc0 2022-09-07T21:39:38.597 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.594+0000 7f2dbf7fe700 1 -- 172.21.15.47:0/372880007 <== osd.3 v2:172.21.15.47:6810/1063754579 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2dd0064ea0 con 0x7f2db80a3bc0 2022-09-07T21:39:38.598 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.595+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/372880007 >> [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579] conn(0x7f2db80a3bc0 msgr2=0x7f2db80a5ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.595+0000 7f2dd4a8c700 1 --2- 172.21.15.47:0/372880007 >> [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579] conn(0x7f2db80a3bc0 0x7f2db80a5ff0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.595+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/372880007 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2db80990d0 msgr2=0x7f2db809b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.595+0000 7f2dd4a8c700 1 --2- 172.21.15.47:0/372880007 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2db80990d0 0x7f2db809b580 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0197530 tx=0x7f2dc4005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.595+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/372880007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2dd00fe0d0 msgr2=0x7f2dd0195f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.595+0000 7f2dd4a8c700 1 --2- 172.21.15.47:0/372880007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2dd00fe0d0 0x7f2dd0195f20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c010 tx=0x7f2dc000f690 comp rx=0 tx=0).stop 2022-09-07T21:39:38.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.595+0000 7f2dced9d700 1 -- 172.21.15.47:0/372880007 reap_dead start 2022-09-07T21:39:38.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.595+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/372880007 shutdown_connections 2022-09-07T21:39:38.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.595+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/372880007 >> 172.21.15.47:0/372880007 conn(0x7f2dd00f9a10 msgr2=0x7f2dd00fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:38.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.595+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/372880007 shutdown_connections 2022-09-07T21:39:38.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.595+0000 7f2dd4a8c700 1 -- 172.21.15.47:0/372880007 wait complete. 2022-09-07T21:39:38.710 INFO:teuthology.orchestra.run.smithi047.stdout:55834574928 2022-09-07T21:39:38.711 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.1 2022-09-07T21:39:38.873 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.870+0000 7f68c0e5d700 1 -- 172.21.15.47:0/858493036 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68bc102d40 msgr2=0x7f68bc0666c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.874 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.870+0000 7f68c0e5d700 1 --2- 172.21.15.47:0/858493036 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68bc102d40 0x7f68bc0666c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009810 tx=0x7f68ac009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:38.874 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.870+0000 7f68c0e5d700 1 -- 172.21.15.47:0/858493036 shutdown_connections 2022-09-07T21:39:38.875 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.870+0000 7f68c0e5d700 1 --2- 172.21.15.47:0/858493036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68bc1008d0 0x7f68bc100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.875 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.870+0000 7f68c0e5d700 1 --2- 172.21.15.47:0/858493036 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68bc102d40 0x7f68bc0666c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.875 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.870+0000 7f68c0e5d700 1 --2- 172.21.15.47:0/858493036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68bc066c00 0x7f68bc067050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.876 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.870+0000 7f68c0e5d700 1 -- 172.21.15.47:0/858493036 >> 172.21.15.47:0/858493036 conn(0x7f68bc073cf0 msgr2=0x7f68bc0740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:38.876 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.870+0000 7f68c0e5d700 1 -- 172.21.15.47:0/858493036 shutdown_connections 2022-09-07T21:39:38.876 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.870+0000 7f68c0e5d700 1 -- 172.21.15.47:0/858493036 wait complete. 2022-09-07T21:39:38.877 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.872+0000 7f68c0e5d700 1 Processor -- start 2022-09-07T21:39:38.878 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.872+0000 7f68c0e5d700 1 -- start start 2022-09-07T21:39:38.878 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.873+0000 7f68c0e5d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68bc066c00 0x7f68bc19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.878 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.873+0000 7f68c0e5d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68bc1008d0 0x7f68bc19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.878 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.873+0000 7f68c0e5d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68bc102d40 0x7f68bc19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.879 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.873+0000 7f68c0e5d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc19f240 con 0x7f68bc102d40 2022-09-07T21:39:38.879 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.873+0000 7f68c0e5d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc19f3b0 con 0x7f68bc1008d0 2022-09-07T21:39:38.879 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.873+0000 7f68c0e5d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc19f570 con 0x7f68bc066c00 2022-09-07T21:39:38.879 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.874+0000 7f68bad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68bc102d40 0x7f68bc19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.880 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.874+0000 7f68bad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68bc102d40 0x7f68bc19ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38266/0 (socket says 172.21.15.47:38266) 2022-09-07T21:39:38.880 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.874+0000 7f68bad9d700 1 -- 172.21.15.47:0/1595160303 learned_addr learned my addr 172.21.15.47:0/1595160303 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:38.880 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.874+0000 7f68b9d9b700 1 --2- 172.21.15.47:0/1595160303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68bc1008d0 0x7f68bc19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.881 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.875+0000 7f68ba59c700 1 --2- 172.21.15.47:0/1595160303 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68bc066c00 0x7f68bc19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.881 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.875+0000 7f68bad9d700 1 -- 172.21.15.47:0/1595160303 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68bc066c00 msgr2=0x7f68bc19a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.881 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.875+0000 7f68bad9d700 1 --2- 172.21.15.47:0/1595160303 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68bc066c00 0x7f68bc19a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.881 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.875+0000 7f68bad9d700 1 -- 172.21.15.47:0/1595160303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68bc1008d0 msgr2=0x7f68bc19a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.882 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.875+0000 7f68bad9d700 1 --2- 172.21.15.47:0/1595160303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68bc1008d0 0x7f68bc19a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.882 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.875+0000 7f68bad9d700 1 -- 172.21.15.47:0/1595160303 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac009470 con 0x7f68bc102d40 2022-09-07T21:39:38.883 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.875+0000 7f68bad9d700 1 --2- 172.21.15.47:0/1595160303 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68bc102d40 0x7f68bc19ec40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f68b400e910 tx=0x7f68b400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:38.884 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.876+0000 7f68ab7fe700 1 -- 172.21.15.47:0/1595160303 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4004c30 con 0x7f68bc102d40 2022-09-07T21:39:38.884 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.876+0000 7f68ab7fe700 1 -- 172.21.15.47:0/1595160303 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f68b400ee60 con 0x7f68bc102d40 2022-09-07T21:39:38.884 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.876+0000 7f68ab7fe700 1 -- 172.21.15.47:0/1595160303 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b40057e0 con 0x7f68bc102d40 2022-09-07T21:39:38.884 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.876+0000 7f68c0e5d700 1 -- 172.21.15.47:0/1595160303 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68bc19f850 con 0x7f68bc102d40 2022-09-07T21:39:38.885 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.876+0000 7f68c0e5d700 1 -- 172.21.15.47:0/1595160303 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68bc1a70d0 con 0x7f68bc102d40 2022-09-07T21:39:38.885 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.877+0000 7f68c0e5d700 1 -- 172.21.15.47:0/1595160303 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f68bc100240 con 0x7f68bc102d40 2022-09-07T21:39:38.885 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.880+0000 7f68ab7fe700 1 -- 172.21.15.47:0/1595160303 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f68b4021070 con 0x7f68bc102d40 2022-09-07T21:39:38.886 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.881+0000 7f68ab7fe700 1 --2- 172.21.15.47:0/1595160303 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f68a40991a0 0x7f68a409b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.886 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.881+0000 7f68ab7fe700 1 -- 172.21.15.47:0/1595160303 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f68b40c2880 con 0x7f68bc102d40 2022-09-07T21:39:38.887 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.882+0000 7f68ab7fe700 1 --2- 172.21.15.47:0/1595160303 >> [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315] conn(0x7f68a40a3c90 0x7f68a40a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.887 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.882+0000 7f68ab7fe700 1 -- 172.21.15.47:0/1595160303 --> [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68a40a6790 con 0x7f68a40a3c90 2022-09-07T21:39:38.887 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.882+0000 7f68ab7fe700 1 -- 172.21.15.47:0/1595160303 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68b40c2bf0 con 0x7f68bc102d40 2022-09-07T21:39:38.887 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.882+0000 7f68b9d9b700 1 --2- 172.21.15.47:0/1595160303 >> [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315] conn(0x7f68a40a3c90 0x7f68a40a60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.888 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.883+0000 7f68b9d9b700 1 --2- 172.21.15.47:0/1595160303 >> [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315] conn(0x7f68a40a3c90 0x7f68a40a60c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:38.888 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.883+0000 7f68ba59c700 1 --2- 172.21.15.47:0/1595160303 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f68a40991a0 0x7f68a409b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.888 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.884+0000 7f68ab7fe700 1 -- 172.21.15.47:0/1595160303 <== osd.11 v2:172.21.15.59:6824/1569847315 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f68a40a6790 con 0x7f68a40a3c90 2022-09-07T21:39:38.889 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.885+0000 7f68ba59c700 1 --2- 172.21.15.47:0/1595160303 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f68a40991a0 0x7f68a409b650 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f68ac006010 tx=0x7f68ac005760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:38.913 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.911+0000 7f68c0e5d700 1 -- 172.21.15.47:0/1595160303 --> [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f68bc064ea0 con 0x7f68a40a3c90 2022-09-07T21:39:38.916 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.482+0000 7f5d33e28700 1 -- 172.21.15.47:0/2921132603 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d2c0ff9c0 msgr2=0x7f5d2c107f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.917 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.482+0000 7f5d33e28700 1 --2- 172.21.15.47:0/2921132603 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d2c0ff9c0 0x7f5d2c107f40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009810 tx=0x7f5d1c009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:38.917 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.482+0000 7f5d33e28700 1 -- 172.21.15.47:0/2921132603 shutdown_connections 2022-09-07T21:39:38.918 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.482+0000 7f5d33e28700 1 --2- 172.21.15.47:0/2921132603 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d2c108480 0x7f5d2c10a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.918 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.482+0000 7f5d33e28700 1 --2- 172.21.15.47:0/2921132603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d2c0ff0b0 0x7f5d2c0ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.918 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.482+0000 7f5d33e28700 1 --2- 172.21.15.47:0/2921132603 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d2c0ff9c0 0x7f5d2c107f40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.919 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.482+0000 7f5d33e28700 1 -- 172.21.15.47:0/2921132603 >> 172.21.15.47:0/2921132603 conn(0x7f5d2c0f89d0 msgr2=0x7f5d2c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:38.919 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.482+0000 7f5d33e28700 1 -- 172.21.15.47:0/2921132603 shutdown_connections 2022-09-07T21:39:38.920 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.482+0000 7f5d33e28700 1 -- 172.21.15.47:0/2921132603 wait complete. 2022-09-07T21:39:38.920 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.484+0000 7f5d33e28700 1 Processor -- start 2022-09-07T21:39:38.920 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.484+0000 7f5d33e28700 1 -- start start 2022-09-07T21:39:38.921 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.484+0000 7f5d33e28700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d2c0ff0b0 0x7f5d2c195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.921 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.484+0000 7f5d33e28700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d2c0ff9c0 0x7f5d2c1963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.921 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.484+0000 7f5d33e28700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d2c108480 0x7f5d2c19a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.921 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.484+0000 7f5d33e28700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c19aee0 con 0x7f5d2c108480 2022-09-07T21:39:38.922 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.484+0000 7f5d33e28700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c19b050 con 0x7f5d2c0ff0b0 2022-09-07T21:39:38.922 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.484+0000 7f5d33e28700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c19b210 con 0x7f5d2c0ff9c0 2022-09-07T21:39:38.922 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.485+0000 7f5d323c5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d2c108480 0x7f5d2c19a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.923 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.485+0000 7f5d31bc4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d2c0ff0b0 0x7f5d2c195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.923 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.485+0000 7f5d313c3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d2c0ff9c0 0x7f5d2c1963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.923 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.485+0000 7f5d31bc4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d2c0ff0b0 0x7f5d2c195eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:47936/0 (socket says 172.21.15.47:47936) 2022-09-07T21:39:38.923 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.485+0000 7f5d31bc4700 1 -- 172.21.15.47:0/184160845 learned_addr learned my addr 172.21.15.47:0/184160845 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:38.924 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.485+0000 7f5d323c5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d2c108480 0x7f5d2c19a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52466/0 (socket says 172.21.15.47:52466) 2022-09-07T21:39:38.924 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.485+0000 7f5d323c5700 1 -- 172.21.15.47:0/184160845 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d2c0ff9c0 msgr2=0x7f5d2c1963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.924 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.485+0000 7f5d323c5700 1 --2- 172.21.15.47:0/184160845 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d2c0ff9c0 0x7f5d2c1963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.924 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.485+0000 7f5d323c5700 1 -- 172.21.15.47:0/184160845 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d2c0ff0b0 msgr2=0x7f5d2c195eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.485+0000 7f5d323c5700 1 --2- 172.21.15.47:0/184160845 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d2c0ff0b0 0x7f5d2c195eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.485+0000 7f5d323c5700 1 -- 172.21.15.47:0/184160845 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1c009470 con 0x7f5d2c108480 2022-09-07T21:39:38.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.485+0000 7f5d31bc4700 1 --2- 172.21.15.47:0/184160845 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d2c0ff0b0 0x7f5d2c195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:38.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.485+0000 7f5d323c5700 1 --2- 172.21.15.47:0/184160845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d2c108480 0x7f5d2c19a850 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400eab0 tx=0x7f5d2400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:38.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.486+0000 7f5d22ffd700 1 -- 172.21.15.47:0/184160845 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d240098c0 con 0x7f5d2c108480 2022-09-07T21:39:38.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.486+0000 7f5d22ffd700 1 -- 172.21.15.47:0/184160845 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5d2400ce10 con 0x7f5d2c108480 2022-09-07T21:39:38.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.486+0000 7f5d33e28700 1 -- 172.21.15.47:0/184160845 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2c19b4f0 con 0x7f5d2c108480 2022-09-07T21:39:38.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.486+0000 7f5d22ffd700 1 -- 172.21.15.47:0/184160845 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24004280 con 0x7f5d2c108480 2022-09-07T21:39:38.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.486+0000 7f5d33e28700 1 -- 172.21.15.47:0/184160845 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2c1a2ce0 con 0x7f5d2c108480 2022-09-07T21:39:38.927 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.487+0000 7f5d33e28700 1 -- 172.21.15.47:0/184160845 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5d2c0fe840 con 0x7f5d2c108480 2022-09-07T21:39:38.927 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.489+0000 7f5d22ffd700 1 -- 172.21.15.47:0/184160845 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f5d2400c040 con 0x7f5d2c108480 2022-09-07T21:39:38.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.490+0000 7f5d22ffd700 1 --2- 172.21.15.47:0/184160845 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f5d180990d0 0x7f5d1809b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.490+0000 7f5d22ffd700 1 -- 172.21.15.47:0/184160845 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f5d24014070 con 0x7f5d2c108480 2022-09-07T21:39:38.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.490+0000 7f5d31bc4700 1 --2- 172.21.15.47:0/184160845 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f5d180990d0 0x7f5d1809b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.491+0000 7f5d22ffd700 1 --2- 172.21.15.47:0/184160845 >> [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795] conn(0x7f5d180a3bc0 0x7f5d180a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:38.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.491+0000 7f5d22ffd700 1 -- 172.21.15.47:0/184160845 --> [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d180a66c0 con 0x7f5d180a3bc0 2022-09-07T21:39:38.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.491+0000 7f5d22ffd700 1 -- 172.21.15.47:0/184160845 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d240c2be0 con 0x7f5d2c108480 2022-09-07T21:39:38.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.491+0000 7f5d313c3700 1 --2- 172.21.15.47:0/184160845 >> [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795] conn(0x7f5d180a3bc0 0x7f5d180a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:38.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.491+0000 7f5d31bc4700 1 --2- 172.21.15.47:0/184160845 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f5d180990d0 0x7f5d1809b580 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5d280057d0 tx=0x7f5d28005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:38.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.492+0000 7f5d313c3700 1 --2- 172.21.15.47:0/184160845 >> [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795] conn(0x7f5d180a3bc0 0x7f5d180a5ff0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:38.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.493+0000 7f5d22ffd700 1 -- 172.21.15.47:0/184160845 <== osd.9 v2:172.21.15.47:6826/3489379795 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f5d180a66c0 con 0x7f5d180a3bc0 2022-09-07T21:39:38.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.510+0000 7f5d33e28700 1 -- 172.21.15.47:0/184160845 --> [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5d2c0669f0 con 0x7f5d180a3bc0 2022-09-07T21:39:38.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.511+0000 7f5d22ffd700 1 -- 172.21.15.47:0/184160845 <== osd.9 v2:172.21.15.47:6826/3489379795 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5d2c0669f0 con 0x7f5d180a3bc0 2022-09-07T21:39:38.931 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.511+0000 7f5d33e28700 1 -- 172.21.15.47:0/184160845 >> [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795] conn(0x7f5d180a3bc0 msgr2=0x7f5d180a5ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.511+0000 7f5d33e28700 1 --2- 172.21.15.47:0/184160845 >> [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795] conn(0x7f5d180a3bc0 0x7f5d180a5ff0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.511+0000 7f5d33e28700 1 -- 172.21.15.47:0/184160845 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f5d180990d0 msgr2=0x7f5d1809b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.511+0000 7f5d33e28700 1 --2- 172.21.15.47:0/184160845 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f5d180990d0 0x7f5d1809b580 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5d280057d0 tx=0x7f5d28005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.511+0000 7f5d33e28700 1 -- 172.21.15.47:0/184160845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d2c108480 msgr2=0x7f5d2c19a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.933 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.511+0000 7f5d33e28700 1 --2- 172.21.15.47:0/184160845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d2c108480 0x7f5d2c19a850 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400eab0 tx=0x7f5d2400edc0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.933 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.511+0000 7f5d323c5700 1 -- 172.21.15.47:0/184160845 reap_dead start 2022-09-07T21:39:38.933 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.511+0000 7f5d33e28700 1 -- 172.21.15.47:0/184160845 shutdown_connections 2022-09-07T21:39:38.933 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.511+0000 7f5d33e28700 1 -- 172.21.15.47:0/184160845 >> 172.21.15.47:0/184160845 conn(0x7f5d2c0f89d0 msgr2=0x7f5d2c10a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:38.934 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.512+0000 7f5d33e28700 1 -- 172.21.15.47:0/184160845 shutdown_connections 2022-09-07T21:39:38.934 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.512+0000 7f5d33e28700 1 -- 172.21.15.47:0/184160845 wait complete. 2022-09-07T21:39:38.935 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.913+0000 7f68ab7fe700 1 -- 172.21.15.47:0/1595160303 <== osd.11 v2:172.21.15.59:6824/1569847315 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f68bc064ea0 con 0x7f68a40a3c90 2022-09-07T21:39:38.935 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.914+0000 7f68c0e5d700 1 -- 172.21.15.47:0/1595160303 >> [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315] conn(0x7f68a40a3c90 msgr2=0x7f68a40a60c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.936 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.914+0000 7f68c0e5d700 1 --2- 172.21.15.47:0/1595160303 >> [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315] conn(0x7f68a40a3c90 0x7f68a40a60c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:38.936 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.914+0000 7f68c0e5d700 1 -- 172.21.15.47:0/1595160303 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f68a40991a0 msgr2=0x7f68a409b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.936 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.914+0000 7f68c0e5d700 1 --2- 172.21.15.47:0/1595160303 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f68a40991a0 0x7f68a409b650 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f68ac006010 tx=0x7f68ac005760 comp rx=0 tx=0).stop 2022-09-07T21:39:38.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.914+0000 7f68c0e5d700 1 -- 172.21.15.47:0/1595160303 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68bc102d40 msgr2=0x7f68bc19ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:38.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.914+0000 7f68c0e5d700 1 --2- 172.21.15.47:0/1595160303 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68bc102d40 0x7f68bc19ec40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f68b400e910 tx=0x7f68b400ec20 comp rx=0 tx=0).stop 2022-09-07T21:39:38.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.916+0000 7f68bad9d700 1 -- 172.21.15.47:0/1595160303 reap_dead start 2022-09-07T21:39:38.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.916+0000 7f68c0e5d700 1 -- 172.21.15.47:0/1595160303 shutdown_connections 2022-09-07T21:39:38.938 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.916+0000 7f68c0e5d700 1 -- 172.21.15.47:0/1595160303 >> 172.21.15.47:0/1595160303 conn(0x7f68bc073cf0 msgr2=0x7f68bc0ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:38.938 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.916+0000 7f68c0e5d700 1 -- 172.21.15.47:0/1595160303 shutdown_connections 2022-09-07T21:39:38.939 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:38.917+0000 7f68c0e5d700 1 -- 172.21.15.47:0/1595160303 wait complete. 2022-09-07T21:39:39.880 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:39 smithi047 ceph-mon[90087]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T21:39:40.025 INFO:teuthology.orchestra.run.smithi047.stdout:287762808866 2022-09-07T21:39:40.026 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.9 2022-09-07T21:39:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:39 smithi059 ceph-mon[97450]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T21:39:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:39 smithi055 ceph-mon[97033]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T21:39:40.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.543+0000 7f1e9c80b700 1 -- 172.21.15.47:0/3601719550 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e981002e0 msgr2=0x7f1e981006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.543+0000 7f1e9c80b700 1 --2- 172.21.15.47:0/3601719550 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e981002e0 0x7f1e981006b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1e88009840 tx=0x7f1e88009b50 comp rx=0 tx=0).stop 2022-09-07T21:39:40.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.543+0000 7f1e9c80b700 1 -- 172.21.15.47:0/3601719550 shutdown_connections 2022-09-07T21:39:40.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.543+0000 7f1e9c80b700 1 --2- 172.21.15.47:0/3601719550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e981053e0 0x7f1e9810cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.543+0000 7f1e9c80b700 1 --2- 172.21.15.47:0/3601719550 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e981002e0 0x7f1e981006b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.291 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.543+0000 7f1e9c80b700 1 --2- 172.21.15.47:0/3601719550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e98100c80 0x7f1e98104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.292 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.543+0000 7f1e9c80b700 1 -- 172.21.15.47:0/3601719550 >> 172.21.15.47:0/3601719550 conn(0x7f1e980fbc20 msgr2=0x7f1e980fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:40.292 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.543+0000 7f1e9c80b700 1 -- 172.21.15.47:0/3601719550 shutdown_connections 2022-09-07T21:39:40.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.544+0000 7f1e9c80b700 1 -- 172.21.15.47:0/3601719550 wait complete. 2022-09-07T21:39:40.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.545+0000 7f1e9c80b700 1 Processor -- start 2022-09-07T21:39:40.293 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.545+0000 7f1e9c80b700 1 -- start start 2022-09-07T21:39:40.294 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.545+0000 7f1e9c80b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e981002e0 0x7f1e98195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.294 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.545+0000 7f1e9c80b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e98100c80 0x7f1e98196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.294 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.545+0000 7f1e9c80b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e981053e0 0x7f1e9819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.294 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.545+0000 7f1e9c80b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9819ae70 con 0x7f1e981053e0 2022-09-07T21:39:40.295 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.545+0000 7f1e9c80b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9819afe0 con 0x7f1e98100c80 2022-09-07T21:39:40.295 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.545+0000 7f1e9c80b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9819b1a0 con 0x7f1e981002e0 2022-09-07T21:39:40.295 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.546+0000 7f1e96d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e981053e0 0x7f1e9819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:40.295 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.546+0000 7f1e9659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e981002e0 0x7f1e98195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:40.295 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.546+0000 7f1e96d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e981053e0 0x7f1e9819a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38286/0 (socket says 172.21.15.47:38286) 2022-09-07T21:39:40.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.546+0000 7f1e96d9d700 1 -- 172.21.15.47:0/1599151264 learned_addr learned my addr 172.21.15.47:0/1599151264 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:40.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.546+0000 7f1e95d9b700 1 --2- 172.21.15.47:0/1599151264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e98100c80 0x7f1e98196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:40.296 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.546+0000 7f1e9659c700 1 -- 172.21.15.47:0/1599151264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e98100c80 msgr2=0x7f1e98196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.297 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.546+0000 7f1e9659c700 1 --2- 172.21.15.47:0/1599151264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e98100c80 0x7f1e98196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.297 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.546+0000 7f1e9659c700 1 -- 172.21.15.47:0/1599151264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e981053e0 msgr2=0x7f1e9819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.298 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.546+0000 7f1e9659c700 1 --2- 172.21.15.47:0/1599151264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e981053e0 0x7f1e9819a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.298 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.546+0000 7f1e9659c700 1 -- 172.21.15.47:0/1599151264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e90009730 con 0x7f1e981002e0 2022-09-07T21:39:40.298 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.546+0000 7f1e96d9d700 1 --2- 172.21.15.47:0/1599151264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e981053e0 0x7f1e9819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:40.298 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.546+0000 7f1e9659c700 1 --2- 172.21.15.47:0/1599151264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e981002e0 0x7f1e98195e40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1e88009760 tx=0x7f1e8800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:40.299 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.547+0000 7f1e877fe700 1 -- 172.21.15.47:0/1599151264 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8801d070 con 0x7f1e981002e0 2022-09-07T21:39:40.300 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.547+0000 7f1e9c80b700 1 -- 172.21.15.47:0/1599151264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e88009470 con 0x7f1e981002e0 2022-09-07T21:39:40.300 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.547+0000 7f1e9c80b700 1 -- 172.21.15.47:0/1599151264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e981a2a90 con 0x7f1e981002e0 2022-09-07T21:39:40.300 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.547+0000 7f1e877fe700 1 -- 172.21.15.47:0/1599151264 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1e88005860 con 0x7f1e981002e0 2022-09-07T21:39:40.300 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.547+0000 7f1e877fe700 1 -- 172.21.15.47:0/1599151264 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88004760 con 0x7f1e981002e0 2022-09-07T21:39:40.300 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.548+0000 7f1e9c80b700 1 -- 172.21.15.47:0/1599151264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1e70000ff0 con 0x7f1e981002e0 2022-09-07T21:39:40.301 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.550+0000 7f1e877fe700 1 -- 172.21.15.47:0/1599151264 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f1e88028030 con 0x7f1e981002e0 2022-09-07T21:39:40.301 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.551+0000 7f1e877fe700 1 --2- 172.21.15.47:0/1599151264 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1e800990d0 0x7f1e8009b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.301 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.551+0000 7f1e877fe700 1 -- 172.21.15.47:0/1599151264 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f1e880c3340 con 0x7f1e981002e0 2022-09-07T21:39:40.301 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.552+0000 7f1e95d9b700 1 --2- 172.21.15.47:0/1599151264 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1e800990d0 0x7f1e8009b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:40.302 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.552+0000 7f1e877fe700 1 --2- 172.21.15.47:0/1599151264 >> [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072] conn(0x7f1e800a3bc0 0x7f1e800a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.302 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.552+0000 7f1e877fe700 1 -- 172.21.15.47:0/1599151264 --> [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e800a66c0 con 0x7f1e800a3bc0 2022-09-07T21:39:40.302 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.552+0000 7f1e877fe700 1 -- 172.21.15.47:0/1599151264 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e880c36b0 con 0x7f1e981002e0 2022-09-07T21:39:40.302 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.552+0000 7f1e96d9d700 1 --2- 172.21.15.47:0/1599151264 >> [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072] conn(0x7f1e800a3bc0 0x7f1e800a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:40.303 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.552+0000 7f1e95d9b700 1 --2- 172.21.15.47:0/1599151264 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1e800990d0 0x7f1e8009b580 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1e98197450 tx=0x7f1e8c005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:40.303 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.553+0000 7f1e96d9d700 1 --2- 172.21.15.47:0/1599151264 >> [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072] conn(0x7f1e800a3bc0 0x7f1e800a5ff0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:40.303 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.554+0000 7f1e877fe700 1 -- 172.21.15.47:0/1599151264 <== osd.10 v2:172.21.15.55:6824/2413766072 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f1e800a66c0 con 0x7f1e800a3bc0 2022-09-07T21:39:40.303 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.570+0000 7f1e9c80b700 1 -- 172.21.15.47:0/1599151264 --> [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1e70002bf0 con 0x7f1e800a3bc0 2022-09-07T21:39:40.304 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.571+0000 7f1e877fe700 1 -- 172.21.15.47:0/1599151264 <== osd.10 v2:172.21.15.55:6824/2413766072 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1e70002bf0 con 0x7f1e800a3bc0 2022-09-07T21:39:40.304 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.571+0000 7f1e9c80b700 1 -- 172.21.15.47:0/1599151264 >> [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072] conn(0x7f1e800a3bc0 msgr2=0x7f1e800a5ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.304 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.571+0000 7f1e9c80b700 1 --2- 172.21.15.47:0/1599151264 >> [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072] conn(0x7f1e800a3bc0 0x7f1e800a5ff0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.304 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.571+0000 7f1e9c80b700 1 -- 172.21.15.47:0/1599151264 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1e800990d0 msgr2=0x7f1e8009b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.305 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.571+0000 7f1e9c80b700 1 --2- 172.21.15.47:0/1599151264 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1e800990d0 0x7f1e8009b580 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1e98197450 tx=0x7f1e8c005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.305 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.571+0000 7f1e9c80b700 1 -- 172.21.15.47:0/1599151264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e981002e0 msgr2=0x7f1e98195e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.305 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.571+0000 7f1e9c80b700 1 --2- 172.21.15.47:0/1599151264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e981002e0 0x7f1e98195e40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1e88009760 tx=0x7f1e8800f690 comp rx=0 tx=0).stop 2022-09-07T21:39:40.305 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.571+0000 7f1e96d9d700 1 -- 172.21.15.47:0/1599151264 reap_dead start 2022-09-07T21:39:40.305 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.571+0000 7f1e9c80b700 1 -- 172.21.15.47:0/1599151264 shutdown_connections 2022-09-07T21:39:40.306 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.571+0000 7f1e9c80b700 1 -- 172.21.15.47:0/1599151264 >> 172.21.15.47:0/1599151264 conn(0x7f1e980fbc20 msgr2=0x7f1e98103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:40.306 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.572+0000 7f1e9c80b700 1 -- 172.21.15.47:0/1599151264 shutdown_connections 2022-09-07T21:39:40.306 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.572+0000 7f1e9c80b700 1 -- 172.21.15.47:0/1599151264 wait complete. 2022-09-07T21:39:40.332 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.903+0000 7fcaf4860700 1 -- 172.21.15.47:0/3536067619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcaf0105210 msgr2=0x7fcaf010cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.333 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.903+0000 7fcaf4860700 1 --2- 172.21.15.47:0/3536067619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcaf0105210 0x7fcaf010cdb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcae8009bb0 tx=0x7fcae8009ec0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.333 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.903+0000 7fcaf4860700 1 -- 172.21.15.47:0/3536067619 shutdown_connections 2022-09-07T21:39:40.333 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.903+0000 7fcaf4860700 1 --2- 172.21.15.47:0/3536067619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcaf0105210 0x7fcaf010cdb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.334 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.903+0000 7fcaf4860700 1 --2- 172.21.15.47:0/3536067619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaf0100110 0x7fcaf01004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.334 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.903+0000 7fcaf4860700 1 --2- 172.21.15.47:0/3536067619 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcaf0100ab0 0x7fcaf0104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.334 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.903+0000 7fcaf4860700 1 -- 172.21.15.47:0/3536067619 >> 172.21.15.47:0/3536067619 conn(0x7fcaf00fba90 msgr2=0x7fcaf00fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:40.334 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.903+0000 7fcaf4860700 1 -- 172.21.15.47:0/3536067619 shutdown_connections 2022-09-07T21:39:40.335 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.903+0000 7fcaf4860700 1 -- 172.21.15.47:0/3536067619 wait complete. 2022-09-07T21:39:40.335 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.904+0000 7fcaf4860700 1 Processor -- start 2022-09-07T21:39:40.336 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.905+0000 7fcaf4860700 1 -- start start 2022-09-07T21:39:40.336 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.905+0000 7fcaf4860700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcaf0100110 0x7fcaf0195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.336 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.905+0000 7fcaf4860700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcaf0100ab0 0x7fcaf01962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.337 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.905+0000 7fcaf4860700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaf0105210 0x7fcaf019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.337 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.905+0000 7fcaf4860700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf019ad90 con 0x7fcaf0105210 2022-09-07T21:39:40.338 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.905+0000 7fcaf4860700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf019af00 con 0x7fcaf0100110 2022-09-07T21:39:40.338 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.905+0000 7fcaf4860700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf019b0c0 con 0x7fcaf0100ab0 2022-09-07T21:39:40.338 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.906+0000 7fcaeeffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcaf0100ab0 0x7fcaf01962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:40.338 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.906+0000 7fcaeeffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcaf0100ab0 0x7fcaf01962f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60210/0 (socket says 172.21.15.47:60210) 2022-09-07T21:39:40.339 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.906+0000 7fcaeeffd700 1 -- 172.21.15.47:0/480690959 learned_addr learned my addr 172.21.15.47:0/480690959 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:40.339 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.906+0000 7fcaeffff700 1 --2- 172.21.15.47:0/480690959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaf0105210 0x7fcaf019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:40.339 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.906+0000 7fcaeeffd700 1 -- 172.21.15.47:0/480690959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcaf0100110 msgr2=0x7fcaf0195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.339 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.906+0000 7fcaeeffd700 1 --2- 172.21.15.47:0/480690959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcaf0100110 0x7fcaf0195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.339 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.906+0000 7fcaeeffd700 1 -- 172.21.15.47:0/480690959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaf0105210 msgr2=0x7fcaf019a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.340 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.906+0000 7fcaeeffd700 1 --2- 172.21.15.47:0/480690959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaf0105210 0x7fcaf019a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.340 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.906+0000 7fcaeeffd700 1 -- 172.21.15.47:0/480690959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae80097e0 con 0x7fcaf0100ab0 2022-09-07T21:39:40.340 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.906+0000 7fcaeffff700 1 --2- 172.21.15.47:0/480690959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcaf0105210 0x7fcaf019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:40.340 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.907+0000 7fcaeeffd700 1 --2- 172.21.15.47:0/480690959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcaf0100ab0 0x7fcaf01962f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcae400e7c0 tx=0x7fcae400ead0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:40.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.907+0000 7fcaecff9700 1 -- 172.21.15.47:0/480690959 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae4005220 con 0x7fcaf0100ab0 2022-09-07T21:39:40.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.907+0000 7fcaf4860700 1 -- 172.21.15.47:0/480690959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf019b3a0 con 0x7fcaf0100ab0 2022-09-07T21:39:40.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.907+0000 7fcaf4860700 1 -- 172.21.15.47:0/480690959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf01a2b90 con 0x7fcaf0100ab0 2022-09-07T21:39:40.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.908+0000 7fcaecff9700 1 -- 172.21.15.47:0/480690959 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcae4005380 con 0x7fcaf0100ab0 2022-09-07T21:39:40.342 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.908+0000 7fcaecff9700 1 -- 172.21.15.47:0/480690959 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae4004c80 con 0x7fcaf0100ab0 2022-09-07T21:39:40.342 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.908+0000 7fcaf4860700 1 -- 172.21.15.47:0/480690959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcadc000ff0 con 0x7fcaf0100ab0 2022-09-07T21:39:40.343 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.911+0000 7fcaecff9700 1 -- 172.21.15.47:0/480690959 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7fcae401e020 con 0x7fcaf0100ab0 2022-09-07T21:39:40.343 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.912+0000 7fcaecff9700 1 --2- 172.21.15.47:0/480690959 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcad80990d0 0x7fcad809b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.912+0000 7fcaecff9700 1 -- 172.21.15.47:0/480690959 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fcae4014070 con 0x7fcaf0100ab0 2022-09-07T21:39:40.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.912+0000 7fcaecff9700 1 --2- 172.21.15.47:0/480690959 >> [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881] conn(0x7fcad80a3bc0 0x7fcad80a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.912+0000 7fcaecff9700 1 -- 172.21.15.47:0/480690959 --> [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcad80a66c0 con 0x7fcad80a3bc0 2022-09-07T21:39:40.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.912+0000 7fcaecff9700 1 -- 172.21.15.47:0/480690959 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcae40c2d80 con 0x7fcaf0100ab0 2022-09-07T21:39:40.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.913+0000 7fcaef7fe700 1 --2- 172.21.15.47:0/480690959 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcad80990d0 0x7fcad809b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:40.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.913+0000 7fcaeffff700 1 --2- 172.21.15.47:0/480690959 >> [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881] conn(0x7fcad80a3bc0 0x7fcad80a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:40.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.914+0000 7fcaef7fe700 1 --2- 172.21.15.47:0/480690959 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcad80990d0 0x7fcad809b580 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcae0009630 tx=0x7fcae0006ba0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:40.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.914+0000 7fcaeffff700 1 --2- 172.21.15.47:0/480690959 >> [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881] conn(0x7fcad80a3bc0 0x7fcad80a5ff0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:40.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.915+0000 7fcaecff9700 1 -- 172.21.15.47:0/480690959 <== osd.0 v2:172.21.15.47:6802/3092904881 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7fcad80a66c0 con 0x7fcad80a3bc0 2022-09-07T21:39:40.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.935+0000 7fcaf4860700 1 -- 172.21.15.47:0/480690959 --> [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcadc002bf0 con 0x7fcad80a3bc0 2022-09-07T21:39:40.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.935+0000 7fcaecff9700 1 -- 172.21.15.47:0/480690959 <== osd.0 v2:172.21.15.47:6802/3092904881 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcadc002bf0 con 0x7fcad80a3bc0 2022-09-07T21:39:40.349 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.936+0000 7fcaf4860700 1 -- 172.21.15.47:0/480690959 >> [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881] conn(0x7fcad80a3bc0 msgr2=0x7fcad80a5ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.349 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.936+0000 7fcaf4860700 1 --2- 172.21.15.47:0/480690959 >> [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881] conn(0x7fcad80a3bc0 0x7fcad80a5ff0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.350 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.936+0000 7fcaf4860700 1 -- 172.21.15.47:0/480690959 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcad80990d0 msgr2=0x7fcad809b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.350 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.936+0000 7fcaf4860700 1 --2- 172.21.15.47:0/480690959 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fcad80990d0 0x7fcad809b580 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcae0009630 tx=0x7fcae0006ba0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.350 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.936+0000 7fcaf4860700 1 -- 172.21.15.47:0/480690959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcaf0100ab0 msgr2=0x7fcaf01962f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.350 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.936+0000 7fcaf4860700 1 --2- 172.21.15.47:0/480690959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcaf0100ab0 0x7fcaf01962f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcae400e7c0 tx=0x7fcae400ead0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.351 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.936+0000 7fcaeffff700 1 -- 172.21.15.47:0/480690959 reap_dead start 2022-09-07T21:39:40.351 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.936+0000 7fcaf4860700 1 -- 172.21.15.47:0/480690959 shutdown_connections 2022-09-07T21:39:40.351 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.936+0000 7fcaf4860700 1 -- 172.21.15.47:0/480690959 >> 172.21.15.47:0/480690959 conn(0x7fcaf00fba90 msgr2=0x7fcaf00fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:40.351 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.936+0000 7fcaf4860700 1 -- 172.21.15.47:0/480690959 shutdown_connections 2022-09-07T21:39:40.351 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:39.936+0000 7fcaf4860700 1 -- 172.21.15.47:0/480690959 wait complete. 2022-09-07T21:39:40.698 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.593+0000 7ff71d506700 1 -- 172.21.15.47:0/52315797 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff718071080 msgr2=0x7ff718071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.699 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.593+0000 7ff71d506700 1 --2- 172.21.15.47:0/52315797 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff718071080 0x7ff718071450 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff708009810 tx=0x7ff708009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:40.699 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.593+0000 7ff71d506700 1 -- 172.21.15.47:0/52315797 shutdown_connections 2022-09-07T21:39:40.699 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.593+0000 7ff71d506700 1 --2- 172.21.15.47:0/52315797 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff71810ed00 0x7ff7181110e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.699 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.593+0000 7ff71d506700 1 --2- 172.21.15.47:0/52315797 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff718071080 0x7ff718071450 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.700 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.593+0000 7ff71d506700 1 --2- 172.21.15.47:0/52315797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff718071990 0x7ff71810e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.700 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.593+0000 7ff71d506700 1 -- 172.21.15.47:0/52315797 >> 172.21.15.47:0/52315797 conn(0x7ff7180f9830 msgr2=0x7ff7180fbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:40.700 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.593+0000 7ff71d506700 1 -- 172.21.15.47:0/52315797 shutdown_connections 2022-09-07T21:39:40.700 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.594+0000 7ff71d506700 1 -- 172.21.15.47:0/52315797 wait complete. 2022-09-07T21:39:40.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.595+0000 7ff71d506700 1 Processor -- start 2022-09-07T21:39:40.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.596+0000 7ff71d506700 1 -- start start 2022-09-07T21:39:40.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.596+0000 7ff71d506700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff718071080 0x7ff71819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.596+0000 7ff71d506700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff718071990 0x7ff71819a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.596+0000 7ff71d506700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff71810ed00 0x7ff71819eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.596+0000 7ff71d506700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71819f130 con 0x7ff718071990 2022-09-07T21:39:40.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.596+0000 7ff71d506700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71819f2a0 con 0x7ff71810ed00 2022-09-07T21:39:40.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.596+0000 7ff71d506700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71819f460 con 0x7ff718071080 2022-09-07T21:39:40.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.596+0000 7ff717fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff718071080 0x7ff71819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:40.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.596+0000 7ff7177fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff718071990 0x7ff71819a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:40.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.597+0000 7ff717fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff718071080 0x7ff71819a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60240/0 (socket says 172.21.15.47:60240) 2022-09-07T21:39:40.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.597+0000 7ff717fff700 1 -- 172.21.15.47:0/390765846 learned_addr learned my addr 172.21.15.47:0/390765846 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:40.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.597+0000 7ff71cd05700 1 --2- 172.21.15.47:0/390765846 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff71810ed00 0x7ff71819eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:40.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.597+0000 7ff717fff700 1 -- 172.21.15.47:0/390765846 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff71810ed00 msgr2=0x7ff71819eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.597+0000 7ff717fff700 1 --2- 172.21.15.47:0/390765846 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff71810ed00 0x7ff71819eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.597+0000 7ff717fff700 1 -- 172.21.15.47:0/390765846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff718071990 msgr2=0x7ff71819a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.597+0000 7ff717fff700 1 --2- 172.21.15.47:0/390765846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff718071990 0x7ff71819a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.597+0000 7ff717fff700 1 -- 172.21.15.47:0/390765846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70c0093b0 con 0x7ff718071080 2022-09-07T21:39:40.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.597+0000 7ff7177fe700 1 --2- 172.21.15.47:0/390765846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff718071990 0x7ff71819a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:40.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.597+0000 7ff717fff700 1 --2- 172.21.15.47:0/390765846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff718071080 0x7ff71819a100 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff7080059c0 tx=0x7ff70800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:40.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.598+0000 7ff7157fa700 1 -- 172.21.15.47:0/390765846 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70801d070 con 0x7ff718071080 2022-09-07T21:39:40.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.598+0000 7ff71d506700 1 -- 172.21.15.47:0/390765846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff708009470 con 0x7ff718071080 2022-09-07T21:39:40.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.598+0000 7ff7157fa700 1 -- 172.21.15.47:0/390765846 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff70800f800 con 0x7ff718071080 2022-09-07T21:39:40.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.598+0000 7ff7157fa700 1 -- 172.21.15.47:0/390765846 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7080047f0 con 0x7ff718071080 2022-09-07T21:39:40.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.598+0000 7ff71d506700 1 -- 172.21.15.47:0/390765846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7181a6d50 con 0x7ff718071080 2022-09-07T21:39:40.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.598+0000 7ff71d506700 1 -- 172.21.15.47:0/390765846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff71805f460 con 0x7ff718071080 2022-09-07T21:39:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.601+0000 7ff7157fa700 1 -- 172.21.15.47:0/390765846 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7ff708028030 con 0x7ff718071080 2022-09-07T21:39:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.602+0000 7ff7157fa700 1 --2- 172.21.15.47:0/390765846 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff700099050 0x7ff70009b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.602+0000 7ff7157fa700 1 -- 172.21.15.47:0/390765846 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7ff7080c3a00 con 0x7ff718071080 2022-09-07T21:39:40.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.602+0000 7ff7177fe700 1 --2- 172.21.15.47:0/390765846 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff700099050 0x7ff70009b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:40.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.603+0000 7ff7157fa700 1 --2- 172.21.15.47:0/390765846 >> [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220] conn(0x7ff7000a3b40 0x7ff7000a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:40.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.603+0000 7ff7157fa700 1 -- 172.21.15.47:0/390765846 --> [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff7000a6640 con 0x7ff7000a3b40 2022-09-07T21:39:40.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.603+0000 7ff7157fa700 1 -- 172.21.15.47:0/390765846 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7080c3d70 con 0x7ff718071080 2022-09-07T21:39:40.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.603+0000 7ff71cd05700 1 --2- 172.21.15.47:0/390765846 >> [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220] conn(0x7ff7000a3b40 0x7ff7000a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:40.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.603+0000 7ff7177fe700 1 --2- 172.21.15.47:0/390765846 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff700099050 0x7ff70009b500 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff71819b710 tx=0x7ff70c005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:40.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.604+0000 7ff71cd05700 1 --2- 172.21.15.47:0/390765846 >> [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220] conn(0x7ff7000a3b40 0x7ff7000a5f70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:40.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.605+0000 7ff7157fa700 1 -- 172.21.15.47:0/390765846 <== osd.8 v2:172.21.15.59:6816/1132606220 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7ff7000a6640 con 0x7ff7000a3b40 2022-09-07T21:39:40.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.621+0000 7ff71d506700 1 -- 172.21.15.47:0/390765846 --> [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff71804d480 con 0x7ff7000a3b40 2022-09-07T21:39:40.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.622+0000 7ff7157fa700 1 -- 172.21.15.47:0/390765846 <== osd.8 v2:172.21.15.59:6816/1132606220 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff71804d480 con 0x7ff7000a3b40 2022-09-07T21:39:40.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.622+0000 7ff71d506700 1 -- 172.21.15.47:0/390765846 >> [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220] conn(0x7ff7000a3b40 msgr2=0x7ff7000a5f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.622+0000 7ff71d506700 1 --2- 172.21.15.47:0/390765846 >> [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220] conn(0x7ff7000a3b40 0x7ff7000a5f70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.622+0000 7ff71d506700 1 -- 172.21.15.47:0/390765846 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff700099050 msgr2=0x7ff70009b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.622+0000 7ff71d506700 1 --2- 172.21.15.47:0/390765846 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff700099050 0x7ff70009b500 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff71819b710 tx=0x7ff70c005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:40.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.622+0000 7ff71d506700 1 -- 172.21.15.47:0/390765846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff718071080 msgr2=0x7ff71819a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:40.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.623+0000 7ff71d506700 1 --2- 172.21.15.47:0/390765846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff718071080 0x7ff71819a100 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff7080059c0 tx=0x7ff70800f690 comp rx=0 tx=0).stop 2022-09-07T21:39:40.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.623+0000 7ff71cd05700 1 -- 172.21.15.47:0/390765846 reap_dead start 2022-09-07T21:39:40.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.623+0000 7ff71d506700 1 -- 172.21.15.47:0/390765846 shutdown_connections 2022-09-07T21:39:40.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.623+0000 7ff71d506700 1 -- 172.21.15.47:0/390765846 >> 172.21.15.47:0/390765846 conn(0x7ff7180f9830 msgr2=0x7ff7180fa470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:40.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.623+0000 7ff71d506700 1 -- 172.21.15.47:0/390765846 shutdown_connections 2022-09-07T21:39:40.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:40.623+0000 7ff71d506700 1 -- 172.21.15.47:0/390765846 wait complete. 2022-09-07T21:39:40.779 INFO:teuthology.orchestra.run.smithi047.stdout:107374182469 2022-09-07T21:39:40.779 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.3 2022-09-07T21:39:40.907 INFO:teuthology.orchestra.run.smithi047.stdout:197568495668 2022-09-07T21:39:40.908 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.6 2022-09-07T21:39:41.558 INFO:teuthology.orchestra.run.smithi047.stdout:34359738456 2022-09-07T21:39:41.559 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.0 2022-09-07T21:39:41.648 INFO:teuthology.orchestra.run.smithi047.stdout:343597383704 2022-09-07T21:39:41.649 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.11 2022-09-07T21:39:41.823 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.047+0000 7f0f99fcf700 1 -- 172.21.15.47:0/2275095357 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f941053e0 msgr2=0x7f0f9410cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:41.823 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.047+0000 7f0f99fcf700 1 --2- 172.21.15.47:0/2275095357 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f941053e0 0x7f0f9410cf80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c009be0 tx=0x7f0f8c009ef0 comp rx=0 tx=0).stop 2022-09-07T21:39:41.823 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.047+0000 7f0f99fcf700 1 -- 172.21.15.47:0/2275095357 shutdown_connections 2022-09-07T21:39:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.047+0000 7f0f99fcf700 1 --2- 172.21.15.47:0/2275095357 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f941053e0 0x7f0f9410cf80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.047+0000 7f0f99fcf700 1 --2- 172.21.15.47:0/2275095357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f941002e0 0x7f0f941006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.048+0000 7f0f99fcf700 1 --2- 172.21.15.47:0/2275095357 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f94100c80 0x7f0f94104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.048+0000 7f0f99fcf700 1 -- 172.21.15.47:0/2275095357 >> 172.21.15.47:0/2275095357 conn(0x7f0f940fbc20 msgr2=0x7f0f940fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.048+0000 7f0f99fcf700 1 -- 172.21.15.47:0/2275095357 shutdown_connections 2022-09-07T21:39:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.048+0000 7f0f99fcf700 1 -- 172.21.15.47:0/2275095357 wait complete. 2022-09-07T21:39:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.049+0000 7f0f99fcf700 1 Processor -- start 2022-09-07T21:39:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.049+0000 7f0f99fcf700 1 -- start start 2022-09-07T21:39:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.050+0000 7f0f99fcf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f941002e0 0x7f0f94195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:41.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.050+0000 7f0f99fcf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f94100c80 0x7f0f941963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:41.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.050+0000 7f0f99fcf700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f941053e0 0x7f0f9419a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:41.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.050+0000 7f0f99fcf700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9419aeb0 con 0x7f0f941002e0 2022-09-07T21:39:41.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.050+0000 7f0f99fcf700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9419b020 con 0x7f0f941053e0 2022-09-07T21:39:41.827 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.050+0000 7f0f99fcf700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9419b1e0 con 0x7f0f94100c80 2022-09-07T21:39:41.827 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.050+0000 7f0f937fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f941002e0 0x7f0f94195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:41.827 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.050+0000 7f0f937fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f941002e0 0x7f0f94195e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38332/0 (socket says 172.21.15.47:38332) 2022-09-07T21:39:41.827 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.050+0000 7f0f937fe700 1 -- 172.21.15.47:0/284774403 learned_addr learned my addr 172.21.15.47:0/284774403 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:41.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.050+0000 7f0f93fff700 1 --2- 172.21.15.47:0/284774403 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f941053e0 0x7f0f9419a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:41.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.050+0000 7f0f92ffd700 1 --2- 172.21.15.47:0/284774403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f94100c80 0x7f0f941963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:41.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.051+0000 7f0f937fe700 1 -- 172.21.15.47:0/284774403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f94100c80 msgr2=0x7f0f941963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:41.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.051+0000 7f0f937fe700 1 --2- 172.21.15.47:0/284774403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f94100c80 0x7f0f941963c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:41.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.051+0000 7f0f937fe700 1 -- 172.21.15.47:0/284774403 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f941053e0 msgr2=0x7f0f9419a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:41.829 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.051+0000 7f0f937fe700 1 --2- 172.21.15.47:0/284774403 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0f941053e0 0x7f0f9419a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:41.829 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.051+0000 7f0f937fe700 1 -- 172.21.15.47:0/284774403 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f840093b0 con 0x7f0f941002e0 2022-09-07T21:39:41.829 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.051+0000 7f0f937fe700 1 --2- 172.21.15.47:0/284774403 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f941002e0 0x7f0f94195e80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0f8400e9c0 tx=0x7f0f8400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:41.829 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.051+0000 7f0f90ff9700 1 -- 172.21.15.47:0/284774403 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f840097b0 con 0x7f0f941002e0 2022-09-07T21:39:41.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.051+0000 7f0f99fcf700 1 -- 172.21.15.47:0/284774403 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8c0097e0 con 0x7f0f941002e0 2022-09-07T21:39:41.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.051+0000 7f0f90ff9700 1 -- 172.21.15.47:0/284774403 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0f8400ce70 con 0x7f0f941002e0 2022-09-07T21:39:41.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.051+0000 7f0f90ff9700 1 -- 172.21.15.47:0/284774403 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f84003e30 con 0x7f0f941002e0 2022-09-07T21:39:41.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.052+0000 7f0f99fcf700 1 -- 172.21.15.47:0/284774403 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f941a2cb0 con 0x7f0f941002e0 2022-09-07T21:39:41.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.053+0000 7f0f99fcf700 1 -- 172.21.15.47:0/284774403 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0f940fd120 con 0x7f0f941002e0 2022-09-07T21:39:41.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.055+0000 7f0f90ff9700 1 -- 172.21.15.47:0/284774403 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f0f8401e020 con 0x7f0f941002e0 2022-09-07T21:39:41.832 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.056+0000 7f0f90ff9700 1 --2- 172.21.15.47:0/284774403 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0f7c099050 0x7f0f7c09b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:41.832 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.056+0000 7f0f90ff9700 1 -- 172.21.15.47:0/284774403 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f0f84014070 con 0x7f0f941002e0 2022-09-07T21:39:41.832 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.056+0000 7f0f90ff9700 1 --2- 172.21.15.47:0/284774403 >> [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712] conn(0x7f0f7c0a3b40 0x7f0f7c0a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:41.833 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.056+0000 7f0f92ffd700 1 --2- 172.21.15.47:0/284774403 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0f7c099050 0x7f0f7c09b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:41.833 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.056+0000 7f0f90ff9700 1 -- 172.21.15.47:0/284774403 --> [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f7c0a6640 con 0x7f0f7c0a3b40 2022-09-07T21:39:41.833 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.056+0000 7f0f90ff9700 1 -- 172.21.15.47:0/284774403 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f840c72a0 con 0x7f0f941002e0 2022-09-07T21:39:41.833 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.057+0000 7f0f93fff700 1 --2- 172.21.15.47:0/284774403 >> [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712] conn(0x7f0f7c0a3b40 0x7f0f7c0a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:41.833 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.057+0000 7f0f92ffd700 1 --2- 172.21.15.47:0/284774403 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0f7c099050 0x7f0f7c09b500 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0f880057d0 tx=0x7f0f88005760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:41.834 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.057+0000 7f0f93fff700 1 --2- 172.21.15.47:0/284774403 >> [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712] conn(0x7f0f7c0a3b40 0x7f0f7c0a5f70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:41.834 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.059+0000 7f0f90ff9700 1 -- 172.21.15.47:0/284774403 <== osd.7 v2:172.21.15.55:6816/199887712 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f0f7c0a6640 con 0x7f0f7c0a3b40 2022-09-07T21:39:41.834 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.076+0000 7f0f99fcf700 1 -- 172.21.15.47:0/284774403 --> [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0f94064ea0 con 0x7f0f7c0a3b40 2022-09-07T21:39:41.834 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.076+0000 7f0f90ff9700 1 -- 172.21.15.47:0/284774403 <== osd.7 v2:172.21.15.55:6816/199887712 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0f94064ea0 con 0x7f0f7c0a3b40 2022-09-07T21:39:41.835 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.077+0000 7f0f99fcf700 1 -- 172.21.15.47:0/284774403 >> [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712] conn(0x7f0f7c0a3b40 msgr2=0x7f0f7c0a5f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:41.836 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.077+0000 7f0f99fcf700 1 --2- 172.21.15.47:0/284774403 >> [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712] conn(0x7f0f7c0a3b40 0x7f0f7c0a5f70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:41.836 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.077+0000 7f0f99fcf700 1 -- 172.21.15.47:0/284774403 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0f7c099050 msgr2=0x7f0f7c09b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:41.836 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.077+0000 7f0f99fcf700 1 --2- 172.21.15.47:0/284774403 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f0f7c099050 0x7f0f7c09b500 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0f880057d0 tx=0x7f0f88005760 comp rx=0 tx=0).stop 2022-09-07T21:39:41.836 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.077+0000 7f0f99fcf700 1 -- 172.21.15.47:0/284774403 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f941002e0 msgr2=0x7f0f94195e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:41.837 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.077+0000 7f0f99fcf700 1 --2- 172.21.15.47:0/284774403 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f941002e0 0x7f0f94195e80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0f8400e9c0 tx=0x7f0f8400ecd0 comp rx=0 tx=0).stop 2022-09-07T21:39:41.837 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.077+0000 7f0f93fff700 1 -- 172.21.15.47:0/284774403 reap_dead start 2022-09-07T21:39:41.837 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.077+0000 7f0f99fcf700 1 -- 172.21.15.47:0/284774403 shutdown_connections 2022-09-07T21:39:41.838 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.077+0000 7f0f99fcf700 1 -- 172.21.15.47:0/284774403 >> 172.21.15.47:0/284774403 conn(0x7f0f940fbc20 msgr2=0x7f0f940fc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:41.838 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.078+0000 7f0f99fcf700 1 -- 172.21.15.47:0/284774403 shutdown_connections 2022-09-07T21:39:41.838 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:41.078+0000 7f0f99fcf700 1 -- 172.21.15.47:0/284774403 wait complete. 2022-09-07T21:39:41.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:41 smithi047 ceph-mon[90087]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-07T21:39:41.997 INFO:teuthology.orchestra.run.smithi047.stdout:253403070506 2022-09-07T21:39:41.998 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.8 2022-09-07T21:39:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:41 smithi059 ceph-mon[97450]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-07T21:39:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:41 smithi055 ceph-mon[97033]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-07T21:39:42.373 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:42.432 INFO:teuthology.orchestra.run.smithi047.stdout:309237645341 2022-09-07T21:39:42.432 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.10 2022-09-07T21:39:42.987 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:43.121 INFO:teuthology.orchestra.run.smithi047.stdout:219043332143 2022-09-07T21:39:43.122 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.7 2022-09-07T21:39:43.172 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:43.190 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:43.721 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:43.728 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:44.028 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:43 smithi047 ceph-mon[90087]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-07T21:39:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:43 smithi059 ceph-mon[97450]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-07T21:39:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:43 smithi055 ceph-mon[97033]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-07T21:39:45.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:45 smithi047 ceph-mon[90087]: pgmap v348: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-07T21:39:45.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:45 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:39:45.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:45 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:39:45.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:45 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:39:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:45 smithi059 ceph-mon[97450]: pgmap v348: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-07T21:39:46.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:45 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:39:46.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:45 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:39:46.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:45 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:39:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:45 smithi055 ceph-mon[97033]: pgmap v348: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-07T21:39:46.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:45 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:39:46.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:45 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:39:46.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:45 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:39:46.288 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:46.922 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.842+0000 7f66b2414700 1 -- 172.21.15.47:0/4113499098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66ac100c80 msgr2=0x7f66ac104cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:46.923 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.842+0000 7f66b2414700 1 --2- 172.21.15.47:0/4113499098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66ac100c80 0x7f66ac104cd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009810 tx=0x7f66a0009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:46.923 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.842+0000 7f66b2414700 1 -- 172.21.15.47:0/4113499098 shutdown_connections 2022-09-07T21:39:46.923 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.842+0000 7f66b2414700 1 --2- 172.21.15.47:0/4113499098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66ac1053e0 0x7f66ac10cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:46.924 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.842+0000 7f66b2414700 1 --2- 172.21.15.47:0/4113499098 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66ac1002e0 0x7f66ac1006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:46.924 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.842+0000 7f66b2414700 1 --2- 172.21.15.47:0/4113499098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66ac100c80 0x7f66ac104cd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:46.924 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.842+0000 7f66b2414700 1 -- 172.21.15.47:0/4113499098 >> 172.21.15.47:0/4113499098 conn(0x7f66ac0fbc20 msgr2=0x7f66ac0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:46.924 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.843+0000 7f66b2414700 1 -- 172.21.15.47:0/4113499098 shutdown_connections 2022-09-07T21:39:46.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.843+0000 7f66b2414700 1 -- 172.21.15.47:0/4113499098 wait complete. 2022-09-07T21:39:46.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.844+0000 7f66b2414700 1 Processor -- start 2022-09-07T21:39:46.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.844+0000 7f66b2414700 1 -- start start 2022-09-07T21:39:46.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.844+0000 7f66b2414700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66ac1002e0 0x7f66ac195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:46.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.844+0000 7f66b2414700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66ac100c80 0x7f66ac196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:46.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.844+0000 7f66b2414700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66ac1053e0 0x7f66ac19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:46.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66b2414700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac19af50 con 0x7f66ac100c80 2022-09-07T21:39:46.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66b2414700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac19b0c0 con 0x7f66ac1053e0 2022-09-07T21:39:46.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66b2414700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac19b280 con 0x7f66ac1002e0 2022-09-07T21:39:46.927 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66abfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66ac1002e0 0x7f66ac195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:46.927 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66ab7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66ac100c80 0x7f66ac196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:46.927 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66abfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66ac1002e0 0x7f66ac195f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60266/0 (socket says 172.21.15.47:60266) 2022-09-07T21:39:46.927 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66abfff700 1 -- 172.21.15.47:0/2680974384 learned_addr learned my addr 172.21.15.47:0/2680974384 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:46.927 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66b09b1700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66ac1053e0 0x7f66ac19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:46.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66abfff700 1 -- 172.21.15.47:0/2680974384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66ac1053e0 msgr2=0x7f66ac19a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:46.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66abfff700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66ac1053e0 0x7f66ac19a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:46.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66abfff700 1 -- 172.21.15.47:0/2680974384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66ac100c80 msgr2=0x7f66ac196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:46.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66abfff700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66ac100c80 0x7f66ac196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:46.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66abfff700 1 -- 172.21.15.47:0/2680974384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a0009470 con 0x7f66ac1002e0 2022-09-07T21:39:46.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66ab7fe700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66ac100c80 0x7f66ac196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:46.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.845+0000 7f66b09b1700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66ac1053e0 0x7f66ac19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:46.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.846+0000 7f66abfff700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66ac1002e0 0x7f66ac195f20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f669c00e740 tx=0x7f669c00ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:46.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.846+0000 7f66a97fa700 1 -- 172.21.15.47:0/2680974384 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c005220 con 0x7f66ac1002e0 2022-09-07T21:39:46.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.846+0000 7f66a97fa700 1 -- 172.21.15.47:0/2680974384 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f669c005380 con 0x7f66ac1002e0 2022-09-07T21:39:46.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.846+0000 7f66a97fa700 1 -- 172.21.15.47:0/2680974384 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c004b70 con 0x7f66ac1002e0 2022-09-07T21:39:46.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.846+0000 7f66b2414700 1 -- 172.21.15.47:0/2680974384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ac19b560 con 0x7f66ac1002e0 2022-09-07T21:39:46.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.846+0000 7f66b2414700 1 -- 172.21.15.47:0/2680974384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ac1a2d50 con 0x7f66ac1002e0 2022-09-07T21:39:46.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.847+0000 7f66b2414700 1 -- 172.21.15.47:0/2680974384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ac064ea0 con 0x7f66ac1002e0 2022-09-07T21:39:46.931 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.849+0000 7f66a97fa700 1 -- 172.21.15.47:0/2680974384 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f669c01e020 con 0x7f66ac1002e0 2022-09-07T21:39:46.931 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.850+0000 7f66a97fa700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6694099050 0x7f669409b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:46.931 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.850+0000 7f66a97fa700 1 -- 172.21.15.47:0/2680974384 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f669c014070 con 0x7f66ac1002e0 2022-09-07T21:39:46.931 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.850+0000 7f66ab7fe700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6694099050 0x7f669409b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:46.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.851+0000 7f66ab7fe700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6694099050 0x7f669409b500 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f66ac197530 tx=0x7f66a0009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:46.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:46.854+0000 7f66a97fa700 1 -- 172.21.15.47:0/2680974384 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f669c08ec40 con 0x7f66ac1002e0 2022-09-07T21:39:47.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:47.099+0000 7f66b2414700 1 -- 172.21.15.47:0/2680974384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f66ac10fef0 con 0x7f66ac1002e0 2022-09-07T21:39:47.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:47.100+0000 7f66a97fa700 1 -- 172.21.15.47:0/2680974384 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f669c092260 con 0x7f66ac1002e0 2022-09-07T21:39:47.103 INFO:teuthology.orchestra.run.smithi047.stdout:141733920833 2022-09-07T21:39:47.105 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:47.102+0000 7f66b2414700 1 -- 172.21.15.47:0/2680974384 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6694099050 msgr2=0x7f669409b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:47.105 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:47.102+0000 7f66b2414700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6694099050 0x7f669409b500 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f66ac197530 tx=0x7f66a0009eb0 comp rx=0 tx=0).stop 2022-09-07T21:39:47.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:47.102+0000 7f66b2414700 1 -- 172.21.15.47:0/2680974384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66ac1002e0 msgr2=0x7f66ac195f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:47.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:47.102+0000 7f66b2414700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66ac1002e0 0x7f66ac195f20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f669c00e740 tx=0x7f669c00ea50 comp rx=0 tx=0).stop 2022-09-07T21:39:47.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:47.103+0000 7f66b2414700 1 -- 172.21.15.47:0/2680974384 shutdown_connections 2022-09-07T21:39:47.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:47.103+0000 7f66b2414700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6694099050 0x7f669409b500 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:47.107 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:47.103+0000 7f66b2414700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66ac1053e0 0x7f66ac19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:47.107 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:47.103+0000 7f66b2414700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66ac1002e0 0x7f66ac195f20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:47.107 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:47.103+0000 7f66b2414700 1 --2- 172.21.15.47:0/2680974384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66ac100c80 0x7f66ac196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:47.107 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:47.103+0000 7f66b2414700 1 -- 172.21.15.47:0/2680974384 >> 172.21.15.47:0/2680974384 conn(0x7f66ac0fbc20 msgr2=0x7f66ac0fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:47.108 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:47.103+0000 7f66b2414700 1 -- 172.21.15.47:0/2680974384 shutdown_connections 2022-09-07T21:39:47.108 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:47.103+0000 7f66b2414700 1 -- 172.21.15.47:0/2680974384 wait complete. 2022-09-07T21:39:47.135 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2022-09-07T21:39:47.136 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [5, 11]}]: dispatch 2022-09-07T21:39:47.136 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2022-09-07T21:39:47.137 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2022-09-07T21:39:47.137 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2022-09-07T21:39:47.138 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2022-09-07T21:39:47.138 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2022-09-07T21:39:47.139 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2022-09-07T21:39:47.139 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2022-09-07T21:39:47.140 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2022-09-07T21:39:47.140 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:39:47.140 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:39:47.202 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2022-09-07T21:39:47.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [5, 11]}]: dispatch 2022-09-07T21:39:47.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2022-09-07T21:39:47.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2022-09-07T21:39:47.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2022-09-07T21:39:47.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2022-09-07T21:39:47.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2022-09-07T21:39:47.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2022-09-07T21:39:47.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2022-09-07T21:39:47.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2022-09-07T21:39:47.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:39:47.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:39:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2022-09-07T21:39:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [5, 11]}]: dispatch 2022-09-07T21:39:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2022-09-07T21:39:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2022-09-07T21:39:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2022-09-07T21:39:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2022-09-07T21:39:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2022-09-07T21:39:47.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2022-09-07T21:39:47.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2022-09-07T21:39:47.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2022-09-07T21:39:47.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:39:47.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:39:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2022-09-07T21:39:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2022-09-07T21:39:47.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [5, 11]}]': finished 2022-09-07T21:39:47.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2022-09-07T21:39:47.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2022-09-07T21:39:47.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2022-09-07T21:39:47.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2022-09-07T21:39:47.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2022-09-07T21:39:47.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2022-09-07T21:39:47.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2022-09-07T21:39:47.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2022-09-07T21:39:47.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: osdmap e90: 12 total, 12 up, 12 in 2022-09-07T21:39:47.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2680974384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-09-07T21:39:47.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:39:47.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:39:47.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:39:47.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:39:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2022-09-07T21:39:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2022-09-07T21:39:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [5, 11]}]': finished 2022-09-07T21:39:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2022-09-07T21:39:48.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2022-09-07T21:39:48.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2022-09-07T21:39:48.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2022-09-07T21:39:48.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2022-09-07T21:39:48.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2022-09-07T21:39:48.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2022-09-07T21:39:48.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2022-09-07T21:39:48.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: osdmap e90: 12 total, 12 up, 12 in 2022-09-07T21:39:48.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2680974384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-09-07T21:39:48.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:39:48.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:39:48.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:39:48.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:39:48.099 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2022-09-07T21:39:48.100 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2022-09-07T21:39:48.100 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [5, 11]}]': finished 2022-09-07T21:39:48.100 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2022-09-07T21:39:48.100 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2022-09-07T21:39:48.101 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2022-09-07T21:39:48.101 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2022-09-07T21:39:48.101 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2022-09-07T21:39:48.101 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2022-09-07T21:39:48.101 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2022-09-07T21:39:48.102 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2022-09-07T21:39:48.102 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: osdmap e90: 12 total, 12 up, 12 in 2022-09-07T21:39:48.102 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2680974384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-09-07T21:39:48.102 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:39:48.103 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:39:48.103 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:39:48.103 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:39:48.339 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.068+0000 7f480f276700 1 -- 172.21.15.47:0/826428931 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4808073b20 msgr2=0x7f480810e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:48.340 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.068+0000 7f480f276700 1 --2- 172.21.15.47:0/826428931 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4808073b20 0x7f480810e690 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f47f8009810 tx=0x7f47f8009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:48.340 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.068+0000 7f480f276700 1 -- 172.21.15.47:0/826428931 shutdown_connections 2022-09-07T21:39:48.340 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.068+0000 7f480f276700 1 --2- 172.21.15.47:0/826428931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f480810ebd0 0x7f4808110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.340 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.068+0000 7f480f276700 1 --2- 172.21.15.47:0/826428931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4808073210 0x7f48080735e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.068+0000 7f480f276700 1 --2- 172.21.15.47:0/826428931 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4808073b20 0x7f480810e690 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.068+0000 7f480f276700 1 -- 172.21.15.47:0/826428931 >> 172.21.15.47:0/826428931 conn(0x7f48080fba90 msgr2=0x7f48080fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:48.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.068+0000 7f480f276700 1 -- 172.21.15.47:0/826428931 shutdown_connections 2022-09-07T21:39:48.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.068+0000 7f480f276700 1 -- 172.21.15.47:0/826428931 wait complete. 2022-09-07T21:39:48.342 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.070+0000 7f480f276700 1 Processor -- start 2022-09-07T21:39:48.342 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.071+0000 7f480f276700 1 -- start start 2022-09-07T21:39:48.342 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.071+0000 7f480f276700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4808073210 0x7f480819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:48.342 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.071+0000 7f480f276700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4808073b20 0x7f480819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:48.342 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.071+0000 7f480f276700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f480810ebd0 0x7f480819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:48.343 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.071+0000 7f480f276700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480819f160 con 0x7f480810ebd0 2022-09-07T21:39:48.343 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.071+0000 7f480f276700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480819f2d0 con 0x7f4808073210 2022-09-07T21:39:48.343 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.071+0000 7f480f276700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480819f490 con 0x7f4808073b20 2022-09-07T21:39:48.343 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.072+0000 7f480da73700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4808073b20 0x7f480819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:48.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.072+0000 7f480da73700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4808073b20 0x7f480819a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60282/0 (socket says 172.21.15.47:60282) 2022-09-07T21:39:48.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.072+0000 7f480da73700 1 -- 172.21.15.47:0/4195313283 learned_addr learned my addr 172.21.15.47:0/4195313283 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:48.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.072+0000 7f480e274700 1 --2- 172.21.15.47:0/4195313283 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4808073210 0x7f480819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:48.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.072+0000 7f480ea75700 1 --2- 172.21.15.47:0/4195313283 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f480810ebd0 0x7f480819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:48.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.072+0000 7f480da73700 1 -- 172.21.15.47:0/4195313283 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4808073210 msgr2=0x7f480819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:48.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.072+0000 7f480da73700 1 --2- 172.21.15.47:0/4195313283 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4808073210 0x7f480819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.072+0000 7f480da73700 1 -- 172.21.15.47:0/4195313283 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f480810ebd0 msgr2=0x7f480819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:48.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.072+0000 7f480da73700 1 --2- 172.21.15.47:0/4195313283 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f480810ebd0 0x7f480819ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.072+0000 7f480da73700 1 -- 172.21.15.47:0/4195313283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f8009470 con 0x7f4808073b20 2022-09-07T21:39:48.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.073+0000 7f480da73700 1 --2- 172.21.15.47:0/4195313283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4808073b20 0x7f480819a670 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f47f800f690 tx=0x7f47f800f6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:48.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.073+0000 7f47ff7fe700 1 -- 172.21.15.47:0/4195313283 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f801d070 con 0x7f4808073b20 2022-09-07T21:39:48.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.073+0000 7f47ff7fe700 1 -- 172.21.15.47:0/4195313283 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f47f8005c20 con 0x7f4808073b20 2022-09-07T21:39:48.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.073+0000 7f47ff7fe700 1 -- 172.21.15.47:0/4195313283 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f80048c0 con 0x7f4808073b20 2022-09-07T21:39:48.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.073+0000 7f480f276700 1 -- 172.21.15.47:0/4195313283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f480819f710 con 0x7f4808073b20 2022-09-07T21:39:48.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.073+0000 7f480f276700 1 -- 172.21.15.47:0/4195313283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48081a6f60 con 0x7f4808073b20 2022-09-07T21:39:48.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.074+0000 7f480f276700 1 -- 172.21.15.47:0/4195313283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f480804d480 con 0x7f4808073b20 2022-09-07T21:39:48.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.077+0000 7f47ff7fe700 1 -- 172.21.15.47:0/4195313283 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f47f8028020 con 0x7f4808073b20 2022-09-07T21:39:48.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.079+0000 7f47ff7fe700 1 --2- 172.21.15.47:0/4195313283 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f47f4099050 0x7f47f409b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:48.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.079+0000 7f47ff7fe700 1 -- 172.21.15.47:0/4195313283 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f47f80c36a0 con 0x7f4808073b20 2022-09-07T21:39:48.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.079+0000 7f480e274700 1 --2- 172.21.15.47:0/4195313283 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f47f4099050 0x7f47f409b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:48.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.080+0000 7f480e274700 1 --2- 172.21.15.47:0/4195313283 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f47f4099050 0x7f47f409b500 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4800009e70 tx=0x7f4800009410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:48.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.083+0000 7f47ff7fe700 1 -- 172.21.15.47:0/4195313283 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f47f808fa40 con 0x7f4808073b20 2022-09-07T21:39:48.366 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.364+0000 7f480f276700 1 -- 172.21.15.47:0/4195313283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f480819f8a0 con 0x7f4808073b20 2022-09-07T21:39:48.367 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.364+0000 7f47ff7fe700 1 -- 172.21.15.47:0/4195313283 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f47f8093060 con 0x7f4808073b20 2022-09-07T21:39:48.368 INFO:teuthology.orchestra.run.smithi047.stdout:163208757309 2022-09-07T21:39:48.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.368+0000 7f480f276700 1 -- 172.21.15.47:0/4195313283 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f47f4099050 msgr2=0x7f47f409b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:48.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.368+0000 7f480f276700 1 --2- 172.21.15.47:0/4195313283 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f47f4099050 0x7f47f409b500 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4800009e70 tx=0x7f4800009410 comp rx=0 tx=0).stop 2022-09-07T21:39:48.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.368+0000 7f480f276700 1 -- 172.21.15.47:0/4195313283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4808073b20 msgr2=0x7f480819a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:48.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.368+0000 7f480f276700 1 --2- 172.21.15.47:0/4195313283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4808073b20 0x7f480819a670 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f47f800f690 tx=0x7f47f800f6c0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.368+0000 7f480f276700 1 -- 172.21.15.47:0/4195313283 shutdown_connections 2022-09-07T21:39:48.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.368+0000 7f480f276700 1 --2- 172.21.15.47:0/4195313283 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f47f4099050 0x7f47f409b500 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.386 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.368+0000 7f480f276700 1 --2- 172.21.15.47:0/4195313283 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f480810ebd0 0x7f480819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.386 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.368+0000 7f480f276700 1 --2- 172.21.15.47:0/4195313283 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4808073210 0x7f480819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.386 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.368+0000 7f480f276700 1 --2- 172.21.15.47:0/4195313283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4808073b20 0x7f480819a670 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f47f800f690 tx=0x7f47f800f6c0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.387 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.368+0000 7f480f276700 1 -- 172.21.15.47:0/4195313283 >> 172.21.15.47:0/4195313283 conn(0x7f48080fba90 msgr2=0x7f48080fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:48.387 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.369+0000 7f480f276700 1 -- 172.21.15.47:0/4195313283 shutdown_connections 2022-09-07T21:39:48.387 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.369+0000 7f480f276700 1 -- 172.21.15.47:0/4195313283 wait complete. 2022-09-07T21:39:48.838 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.835+0000 7f9b9388e700 1 -- 172.21.15.47:0/3829540295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b8c0fea70 msgr2=0x7f9b8c102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:48.839 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.835+0000 7f9b9388e700 1 --2- 172.21.15.47:0/3829540295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b8c0fea70 0x7f9b8c102ac0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c009810 tx=0x7f9b7c009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:48.839 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.835+0000 7f9b9388e700 1 -- 172.21.15.47:0/3829540295 shutdown_connections 2022-09-07T21:39:48.839 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.835+0000 7f9b9388e700 1 --2- 172.21.15.47:0/3829540295 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b8c1031d0 0x7f9b8c10ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.839 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.835+0000 7f9b9388e700 1 --2- 172.21.15.47:0/3829540295 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b8c0fe0d0 0x7f9b8c0fe4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.835+0000 7f9b9388e700 1 --2- 172.21.15.47:0/3829540295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b8c0fea70 0x7f9b8c102ac0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.835+0000 7f9b9388e700 1 -- 172.21.15.47:0/3829540295 >> 172.21.15.47:0/3829540295 conn(0x7f9b8c0f9a10 msgr2=0x7f9b8c0fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:48.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.836+0000 7f9b9388e700 1 -- 172.21.15.47:0/3829540295 shutdown_connections 2022-09-07T21:39:48.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.836+0000 7f9b9388e700 1 -- 172.21.15.47:0/3829540295 wait complete. 2022-09-07T21:39:48.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.837+0000 7f9b9388e700 1 Processor -- start 2022-09-07T21:39:48.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.837+0000 7f9b9388e700 1 -- start start 2022-09-07T21:39:48.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b9388e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b8c0fe0d0 0x7f9b8c195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:48.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b9388e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b8c0fea70 0x7f9b8c196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:48.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b9388e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b8c1031d0 0x7f9b8c19a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:48.844 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b9388e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c19af20 con 0x7f9b8c0fea70 2022-09-07T21:39:48.844 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b9388e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c19b090 con 0x7f9b8c0fe0d0 2022-09-07T21:39:48.844 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b9388e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c19b250 con 0x7f9b8c1031d0 2022-09-07T21:39:48.844 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b9162a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b8c0fe0d0 0x7f9b8c195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:48.845 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b9162a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b8c0fe0d0 0x7f9b8c195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:41954/0 (socket says 172.21.15.47:41954) 2022-09-07T21:39:48.845 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b9162a700 1 -- 172.21.15.47:0/2713551006 learned_addr learned my addr 172.21.15.47:0/2713551006 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:48.845 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b91e2b700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b8c1031d0 0x7f9b8c19a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:48.845 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b90e29700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b8c0fea70 0x7f9b8c196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:48.845 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b9162a700 1 -- 172.21.15.47:0/2713551006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b8c1031d0 msgr2=0x7f9b8c19a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:48.846 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b9162a700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b8c1031d0 0x7f9b8c19a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.846 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b9162a700 1 -- 172.21.15.47:0/2713551006 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b8c0fea70 msgr2=0x7f9b8c196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:48.846 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b9162a700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b8c0fea70 0x7f9b8c196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:48.846 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b9162a700 1 -- 172.21.15.47:0/2713551006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7c009470 con 0x7f9b8c0fe0d0 2022-09-07T21:39:48.847 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.838+0000 7f9b91e2b700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b8c1031d0 0x7f9b8c19a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:39:48.847 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.839+0000 7f9b9162a700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b8c0fe0d0 0x7f9b8c195ef0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800e740 tx=0x7f9b8800eb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:48.847 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.839+0000 7f9b827fc700 1 -- 172.21.15.47:0/2713551006 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88005220 con 0x7f9b8c0fe0d0 2022-09-07T21:39:48.847 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.839+0000 7f9b827fc700 1 -- 172.21.15.47:0/2713551006 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f9b88005380 con 0x7f9b8c0fe0d0 2022-09-07T21:39:48.848 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.839+0000 7f9b827fc700 1 -- 172.21.15.47:0/2713551006 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88004c20 con 0x7f9b8c0fe0d0 2022-09-07T21:39:48.848 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.839+0000 7f9b9388e700 1 -- 172.21.15.47:0/2713551006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8c19b530 con 0x7f9b8c0fe0d0 2022-09-07T21:39:48.848 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.839+0000 7f9b9388e700 1 -- 172.21.15.47:0/2713551006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b8c1a2d20 con 0x7f9b8c0fe0d0 2022-09-07T21:39:48.849 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.841+0000 7f9b9388e700 1 -- 172.21.15.47:0/2713551006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b8c064ea0 con 0x7f9b8c0fe0d0 2022-09-07T21:39:48.849 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.843+0000 7f9b827fc700 1 -- 172.21.15.47:0/2713551006 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f9b8801e020 con 0x7f9b8c0fe0d0 2022-09-07T21:39:48.850 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.844+0000 7f9b827fc700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9b780990d0 0x7f9b7809b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:48.850 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.844+0000 7f9b827fc700 1 -- 172.21.15.47:0/2713551006 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f9b88014070 con 0x7f9b8c0fe0d0 2022-09-07T21:39:48.850 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.845+0000 7f9b90e29700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9b780990d0 0x7f9b7809b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:48.850 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.845+0000 7f9b90e29700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9b780990d0 0x7f9b7809b580 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c009760 tx=0x7f9b7c009370 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:48.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:48.848+0000 7f9b827fc700 1 -- 172.21.15.47:0/2713551006 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f9b8808ee50 con 0x7f9b8c0fe0d0 2022-09-07T21:39:48.935 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:49.036 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:48 smithi047 ceph-mon[90087]: osdmap e91: 12 total, 12 up, 12 in 2022-09-07T21:39:49.036 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:48 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/4195313283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-09-07T21:39:49.125 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.123+0000 7f9b9388e700 1 -- 172.21.15.47:0/2713551006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f9b8c10dca0 con 0x7f9b8c0fe0d0 2022-09-07T21:39:49.126 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.123+0000 7f9b827fc700 1 -- 172.21.15.47:0/2713551006 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9b88092470 con 0x7f9b8c0fe0d0 2022-09-07T21:39:49.126 INFO:teuthology.orchestra.run.smithi047.stdout:77309411407 2022-09-07T21:39:49.131 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.128+0000 7f9b9388e700 1 -- 172.21.15.47:0/2713551006 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9b780990d0 msgr2=0x7f9b7809b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:49.131 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.128+0000 7f9b9388e700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9b780990d0 0x7f9b7809b580 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c009760 tx=0x7f9b7c009370 comp rx=0 tx=0).stop 2022-09-07T21:39:49.132 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.128+0000 7f9b9388e700 1 -- 172.21.15.47:0/2713551006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b8c0fe0d0 msgr2=0x7f9b8c195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:49.132 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.128+0000 7f9b9388e700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b8c0fe0d0 0x7f9b8c195ef0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800e740 tx=0x7f9b8800eb00 comp rx=0 tx=0).stop 2022-09-07T21:39:49.132 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.129+0000 7f9b9388e700 1 -- 172.21.15.47:0/2713551006 shutdown_connections 2022-09-07T21:39:49.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.129+0000 7f9b9388e700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9b780990d0 0x7f9b7809b580 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:49.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.129+0000 7f9b9388e700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b8c1031d0 0x7f9b8c19a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:49.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.129+0000 7f9b9388e700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b8c0fe0d0 0x7f9b8c195ef0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:49.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.129+0000 7f9b9388e700 1 --2- 172.21.15.47:0/2713551006 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b8c0fea70 0x7f9b8c196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:49.134 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.129+0000 7f9b9388e700 1 -- 172.21.15.47:0/2713551006 >> 172.21.15.47:0/2713551006 conn(0x7f9b8c0f9a10 msgr2=0x7f9b8c0fa870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:49.134 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.130+0000 7f9b9388e700 1 -- 172.21.15.47:0/2713551006 shutdown_connections 2022-09-07T21:39:49.134 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.130+0000 7f9b9388e700 1 -- 172.21.15.47:0/2713551006 wait complete. 2022-09-07T21:39:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:48 smithi055 ceph-mon[97033]: osdmap e91: 12 total, 12 up, 12 in 2022-09-07T21:39:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:48 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/4195313283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-09-07T21:39:49.277 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:48 smithi059 ceph-mon[97450]: osdmap e91: 12 total, 12 up, 12 in 2022-09-07T21:39:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:48 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/4195313283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-09-07T21:39:49.871 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920830 got 141733920833 for osd.4 2022-09-07T21:39:49.871 DEBUG:teuthology.parallel:result is None 2022-09-07T21:39:50.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.361+0000 7f641d14e700 1 -- 172.21.15.47:0/287463471 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6418066c00 msgr2=0x7f6418067050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:50.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.361+0000 7f641d14e700 1 --2- 172.21.15.47:0/287463471 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6418066c00 0x7f6418067050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f640c009810 tx=0x7f640c009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:50.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.362+0000 7f641d14e700 1 -- 172.21.15.47:0/287463471 shutdown_connections 2022-09-07T21:39:50.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.362+0000 7f641d14e700 1 --2- 172.21.15.47:0/287463471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64181008d0 0x7f6418100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:50.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.362+0000 7f641d14e700 1 --2- 172.21.15.47:0/287463471 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6418102d40 0x7f64180666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:50.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.362+0000 7f641d14e700 1 --2- 172.21.15.47:0/287463471 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6418066c00 0x7f6418067050 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:50.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.362+0000 7f641d14e700 1 -- 172.21.15.47:0/287463471 >> 172.21.15.47:0/287463471 conn(0x7f6418073cf0 msgr2=0x7f64180740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:50.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.362+0000 7f641d14e700 1 -- 172.21.15.47:0/287463471 shutdown_connections 2022-09-07T21:39:50.069 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.362+0000 7f641d14e700 1 -- 172.21.15.47:0/287463471 wait complete. 2022-09-07T21:39:50.069 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.363+0000 7f641d14e700 1 Processor -- start 2022-09-07T21:39:50.069 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.363+0000 7f641d14e700 1 -- start start 2022-09-07T21:39:50.069 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.364+0000 7f641d14e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6418066c00 0x7f641819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:50.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.364+0000 7f641d14e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64181008d0 0x7f641819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:50.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.364+0000 7f641d14e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6418102d40 0x7f641819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:50.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.364+0000 7f641d14e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641819f270 con 0x7f64181008d0 2022-09-07T21:39:50.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.364+0000 7f641d14e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641819f3e0 con 0x7f6418102d40 2022-09-07T21:39:50.072 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.364+0000 7f641d14e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641819f5a0 con 0x7f6418066c00 2022-09-07T21:39:50.072 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.364+0000 7f6416d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6418066c00 0x7f641819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:50.073 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.364+0000 7f641659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64181008d0 0x7f641819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:50.073 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.364+0000 7f6416d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6418066c00 0x7f641819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60012/0 (socket says 172.21.15.47:60012) 2022-09-07T21:39:50.073 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.364+0000 7f6416d9d700 1 -- 172.21.15.47:0/3916601501 learned_addr learned my addr 172.21.15.47:0/3916601501 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:50.074 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.364+0000 7f641759e700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6418102d40 0x7f641819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:50.074 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.364+0000 7f641759e700 1 -- 172.21.15.47:0/3916601501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6418066c00 msgr2=0x7f641819a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:50.075 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.364+0000 7f641759e700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6418066c00 0x7f641819a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:50.075 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.364+0000 7f641759e700 1 -- 172.21.15.47:0/3916601501 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64181008d0 msgr2=0x7f641819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:50.076 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.365+0000 7f641759e700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64181008d0 0x7f641819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:50.076 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.365+0000 7f641759e700 1 -- 172.21.15.47:0/3916601501 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c009470 con 0x7f6418102d40 2022-09-07T21:39:50.077 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.365+0000 7f641659c700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64181008d0 0x7f641819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:50.077 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.365+0000 7f6416d9d700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6418066c00 0x7f641819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:50.078 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.365+0000 7f641759e700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6418102d40 0x7f641819ebe0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f641000eab0 tx=0x7f641000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:50.078 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.365+0000 7f63fffff700 1 -- 172.21.15.47:0/3916601501 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64100098c0 con 0x7f6418102d40 2022-09-07T21:39:50.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.365+0000 7f641d14e700 1 -- 172.21.15.47:0/3916601501 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641819f880 con 0x7f6418102d40 2022-09-07T21:39:50.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.365+0000 7f63fffff700 1 -- 172.21.15.47:0/3916601501 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f641000ce10 con 0x7f6418102d40 2022-09-07T21:39:50.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.365+0000 7f63fffff700 1 -- 172.21.15.47:0/3916601501 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6410004280 con 0x7f6418102d40 2022-09-07T21:39:50.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.365+0000 7f641d14e700 1 -- 172.21.15.47:0/3916601501 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64181a7070 con 0x7f6418102d40 2022-09-07T21:39:50.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.366+0000 7f641d14e700 1 -- 172.21.15.47:0/3916601501 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6418064ea0 con 0x7f6418102d40 2022-09-07T21:39:50.081 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.368+0000 7f63fffff700 1 -- 172.21.15.47:0/3916601501 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f641000c040 con 0x7f6418102d40 2022-09-07T21:39:50.081 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.370+0000 7f63fffff700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6400099050 0x7f640009b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:50.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.370+0000 7f63fffff700 1 -- 172.21.15.47:0/3916601501 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f6410014070 con 0x7f6418102d40 2022-09-07T21:39:50.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.370+0000 7f6416d9d700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6400099050 0x7f640009b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.371+0000 7f6416d9d700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6400099050 0x7f640009b500 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f64080057d0 tx=0x7f6408005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.372+0000 7f63fffff700 1 -- 172.21.15.47:0/3916601501 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f641008f360 con 0x7f6418102d40 2022-09-07T21:39:50.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.613+0000 7f641d14e700 1 -- 172.21.15.47:0/3916601501 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6418114280 con 0x7f6418102d40 2022-09-07T21:39:50.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.614+0000 7f63fffff700 1 -- 172.21.15.47:0/3916601501 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f641001c020 con 0x7f6418102d40 2022-09-07T21:39:50.085 INFO:teuthology.orchestra.run.smithi047.stdout:55834574931 2022-09-07T21:39:50.086 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.616+0000 7f641d14e700 1 -- 172.21.15.47:0/3916601501 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6400099050 msgr2=0x7f640009b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:50.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.616+0000 7f641d14e700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6400099050 0x7f640009b500 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f64080057d0 tx=0x7f6408005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:50.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.616+0000 7f641d14e700 1 -- 172.21.15.47:0/3916601501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6418102d40 msgr2=0x7f641819ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:50.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.616+0000 7f641d14e700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6418102d40 0x7f641819ebe0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f641000eab0 tx=0x7f641000edc0 comp rx=0 tx=0).stop 2022-09-07T21:39:50.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.616+0000 7f641d14e700 1 -- 172.21.15.47:0/3916601501 shutdown_connections 2022-09-07T21:39:50.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.616+0000 7f641d14e700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6400099050 0x7f640009b500 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:50.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.616+0000 7f641d14e700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6418102d40 0x7f641819ebe0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.616+0000 7f641d14e700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6418066c00 0x7f641819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.616+0000 7f641d14e700 1 --2- 172.21.15.47:0/3916601501 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64181008d0 0x7f641819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:50.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.616+0000 7f641d14e700 1 -- 172.21.15.47:0/3916601501 >> 172.21.15.47:0/3916601501 conn(0x7f6418073cf0 msgr2=0x7f64180ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:50.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.617+0000 7f641d14e700 1 -- 172.21.15.47:0/3916601501 shutdown_connections 2022-09-07T21:39:50.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:49.617+0000 7f641d14e700 1 -- 172.21.15.47:0/3916601501 wait complete. 2022-09-07T21:39:50.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:49 smithi047 ceph-mon[90087]: pgmap v352: 129 pgs: 1 activating, 1 peering, 127 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 4 op/s; 2/194 objects degraded (1.031%) 2022-09-07T21:39:50.291 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:49 smithi047 ceph-mon[90087]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2022-09-07T21:39:50.291 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:49 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2713551006' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-09-07T21:39:50.292 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:49 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3916601501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-09-07T21:39:50.293 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:49 smithi059 ceph-mon[97450]: pgmap v352: 129 pgs: 1 activating, 1 peering, 127 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 4 op/s; 2/194 objects degraded (1.031%) 2022-09-07T21:39:50.293 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:49 smithi059 ceph-mon[97450]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2022-09-07T21:39:50.293 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:49 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2713551006' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-09-07T21:39:50.294 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:49 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3916601501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-09-07T21:39:50.294 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:49 smithi055 ceph-mon[97033]: pgmap v352: 129 pgs: 1 activating, 1 peering, 127 active+clean; 640 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 4 op/s; 2/194 objects degraded (1.031%) 2022-09-07T21:39:50.294 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:49 smithi055 ceph-mon[97033]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2022-09-07T21:39:50.294 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:49 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2713551006' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-09-07T21:39:50.295 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:49 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3916601501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-09-07T21:39:50.339 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411405 got 77309411407 for osd.2 2022-09-07T21:39:50.339 DEBUG:teuthology.parallel:result is None 2022-09-07T21:39:52.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:51 smithi047 ceph-mon[90087]: pgmap v353: 129 pgs: 1 activating, 1 peering, 127 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s; 2/194 objects degraded (1.031%); 70 B/s, 0 objects/s recovering 2022-09-07T21:39:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:51 smithi055 ceph-mon[97033]: pgmap v353: 129 pgs: 1 activating, 1 peering, 127 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s; 2/194 objects degraded (1.031%); 70 B/s, 0 objects/s recovering 2022-09-07T21:39:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:51 smithi059 ceph-mon[97450]: pgmap v353: 129 pgs: 1 activating, 1 peering, 127 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3 op/s; 2/194 objects degraded (1.031%); 70 B/s, 0 objects/s recovering 2022-09-07T21:39:52.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.742+0000 7f94373a2700 1 -- 172.21.15.47:0/2676458149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9430100120 msgr2=0x7f94301004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.742+0000 7f94373a2700 1 --2- 172.21.15.47:0/2676458149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9430100120 0x7f94301004f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f942c009810 tx=0x7f942c009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.743+0000 7f94373a2700 1 -- 172.21.15.47:0/2676458149 shutdown_connections 2022-09-07T21:39:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.743+0000 7f94373a2700 1 --2- 172.21.15.47:0/2676458149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94301051d0 0x7f943010cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.743+0000 7f94373a2700 1 --2- 172.21.15.47:0/2676458149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9430100120 0x7f94301004f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:52.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.743+0000 7f94373a2700 1 --2- 172.21.15.47:0/2676458149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9430100ac0 0x7f9430104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:52.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.743+0000 7f94373a2700 1 -- 172.21.15.47:0/2676458149 >> 172.21.15.47:0/2676458149 conn(0x7f94300fba20 msgr2=0x7f94300fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:52.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.743+0000 7f94373a2700 1 -- 172.21.15.47:0/2676458149 shutdown_connections 2022-09-07T21:39:52.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.743+0000 7f94373a2700 1 -- 172.21.15.47:0/2676458149 wait complete. 2022-09-07T21:39:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.745+0000 7f94373a2700 1 Processor -- start 2022-09-07T21:39:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.746+0000 7f94373a2700 1 -- start start 2022-09-07T21:39:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.746+0000 7f94373a2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9430100120 0x7f9430195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.746+0000 7f94373a2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9430100ac0 0x7f9430196200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.746+0000 7f94373a2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94301051d0 0x7f94301995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.746+0000 7f94373a2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9430199c10 con 0x7f94301051d0 2022-09-07T21:39:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.746+0000 7f94373a2700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9430199d80 con 0x7f9430100ac0 2022-09-07T21:39:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.746+0000 7f94373a2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9430199f40 con 0x7f9430100120 2022-09-07T21:39:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.747+0000 7f94363a0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9430100120 0x7f9430195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:52.793 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.747+0000 7f94363a0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9430100120 0x7f9430195cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60024/0 (socket says 172.21.15.47:60024) 2022-09-07T21:39:52.793 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.747+0000 7f94363a0700 1 -- 172.21.15.47:0/989161104 learned_addr learned my addr 172.21.15.47:0/989161104 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:52.793 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.747+0000 7f9435b9f700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9430100ac0 0x7f9430196200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:52.793 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.747+0000 7f9436ba1700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94301051d0 0x7f94301995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:52.794 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.747+0000 7f94363a0700 1 -- 172.21.15.47:0/989161104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9430100ac0 msgr2=0x7f9430196200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:52.794 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.747+0000 7f94363a0700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9430100ac0 0x7f9430196200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:52.794 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.747+0000 7f94363a0700 1 -- 172.21.15.47:0/989161104 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94301051d0 msgr2=0x7f94301995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:52.794 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.747+0000 7f94363a0700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94301051d0 0x7f94301995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:52.795 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.747+0000 7f94363a0700 1 -- 172.21.15.47:0/989161104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942c009470 con 0x7f9430100120 2022-09-07T21:39:52.795 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.747+0000 7f9435b9f700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9430100ac0 0x7f9430196200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:52.795 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.748+0000 7f94363a0700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9430100120 0x7f9430195cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f942c009ef0 tx=0x7f942c00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:52.795 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.748+0000 7f94277fe700 1 -- 172.21.15.47:0/989161104 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c01d070 con 0x7f9430100120 2022-09-07T21:39:52.795 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.748+0000 7f94373a2700 1 -- 172.21.15.47:0/989161104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f943019a1c0 con 0x7f9430100120 2022-09-07T21:39:52.796 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.749+0000 7f94373a2700 1 -- 172.21.15.47:0/989161104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94301a2b20 con 0x7f9430100120 2022-09-07T21:39:52.796 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.749+0000 7f94277fe700 1 -- 172.21.15.47:0/989161104 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f942c00f8b0 con 0x7f9430100120 2022-09-07T21:39:52.796 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.749+0000 7f94277fe700 1 -- 172.21.15.47:0/989161104 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c004760 con 0x7f9430100120 2022-09-07T21:39:52.796 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.750+0000 7f94373a2700 1 -- 172.21.15.47:0/989161104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f943004d480 con 0x7f9430100120 2022-09-07T21:39:52.797 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.752+0000 7f94277fe700 1 -- 172.21.15.47:0/989161104 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f942c028030 con 0x7f9430100120 2022-09-07T21:39:52.797 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.754+0000 7f94277fe700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f941c0990d0 0x7f941c09b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:52.797 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.754+0000 7f94277fe700 1 -- 172.21.15.47:0/989161104 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f942c0c3760 con 0x7f9430100120 2022-09-07T21:39:52.797 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.754+0000 7f9435b9f700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f941c0990d0 0x7f941c09b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:52.797 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.755+0000 7f9435b9f700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f941c0990d0 0x7f941c09b580 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9430197290 tx=0x7f9420005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:52.798 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.755+0000 7f94277fe700 1 -- 172.21.15.47:0/989161104 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f942c08faa0 con 0x7f9430100120 2022-09-07T21:39:52.997 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.994+0000 7f94373a2700 1 -- 172.21.15.47:0/989161104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f94301970d0 con 0x7f9430100120 2022-09-07T21:39:52.998 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.995+0000 7f94277fe700 1 -- 172.21.15.47:0/989161104 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f942c026090 con 0x7f9430100120 2022-09-07T21:39:52.998 INFO:teuthology.orchestra.run.smithi047.stdout:197568495671 2022-09-07T21:39:53.001 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.997+0000 7f94373a2700 1 -- 172.21.15.47:0/989161104 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f941c0990d0 msgr2=0x7f941c09b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:53.001 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.997+0000 7f94373a2700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f941c0990d0 0x7f941c09b580 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9430197290 tx=0x7f9420005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:53.001 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.997+0000 7f94373a2700 1 -- 172.21.15.47:0/989161104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9430100120 msgr2=0x7f9430195cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:53.002 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.997+0000 7f94373a2700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9430100120 0x7f9430195cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f942c009ef0 tx=0x7f942c00f690 comp rx=0 tx=0).stop 2022-09-07T21:39:53.002 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.997+0000 7f94373a2700 1 -- 172.21.15.47:0/989161104 shutdown_connections 2022-09-07T21:39:53.002 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.997+0000 7f94373a2700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f941c0990d0 0x7f941c09b580 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:53.002 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.997+0000 7f94373a2700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94301051d0 0x7f94301995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:53.003 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.997+0000 7f94373a2700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9430100120 0x7f9430195cc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:53.003 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.997+0000 7f94373a2700 1 --2- 172.21.15.47:0/989161104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9430100ac0 0x7f9430196200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:53.003 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.997+0000 7f94373a2700 1 -- 172.21.15.47:0/989161104 >> 172.21.15.47:0/989161104 conn(0x7f94300fba20 msgr2=0x7f9430103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:53.003 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.997+0000 7f94373a2700 1 -- 172.21.15.47:0/989161104 shutdown_connections 2022-09-07T21:39:53.003 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.997+0000 7f94373a2700 1 -- 172.21.15.47:0/989161104 wait complete. 2022-09-07T21:39:53.384 INFO:teuthology.orchestra.run.smithi047.stdout:107374182471 2022-09-07T21:39:53.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.903+0000 7f47317a7700 1 -- 172.21.15.47:0/3271227551 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f472c10eb90 msgr2=0x7f472c110f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:53.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.903+0000 7f47317a7700 1 --2- 172.21.15.47:0/3271227551 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f472c10eb90 0x7f472c110f70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4724009b00 tx=0x7f4724009e10 comp rx=0 tx=0).stop 2022-09-07T21:39:53.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.903+0000 7f47317a7700 1 -- 172.21.15.47:0/3271227551 shutdown_connections 2022-09-07T21:39:53.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.903+0000 7f47317a7700 1 --2- 172.21.15.47:0/3271227551 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f472c10eb90 0x7f472c110f70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:53.386 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.903+0000 7f47317a7700 1 --2- 172.21.15.47:0/3271227551 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f472c073210 0x7f472c0735e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:53.386 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.903+0000 7f47317a7700 1 --2- 172.21.15.47:0/3271227551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f472c073b20 0x7f472c10e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:53.386 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.903+0000 7f47317a7700 1 -- 172.21.15.47:0/3271227551 >> 172.21.15.47:0/3271227551 conn(0x7f472c0fb9f0 msgr2=0x7f472c0fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:53.387 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.903+0000 7f47317a7700 1 -- 172.21.15.47:0/3271227551 shutdown_connections 2022-09-07T21:39:53.387 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.903+0000 7f47317a7700 1 -- 172.21.15.47:0/3271227551 wait complete. 2022-09-07T21:39:53.387 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.905+0000 7f47317a7700 1 Processor -- start 2022-09-07T21:39:53.387 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.905+0000 7f47317a7700 1 -- start start 2022-09-07T21:39:53.388 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.906+0000 7f47317a7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f472c073210 0x7f472c19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:53.388 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.906+0000 7f47317a7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f472c073b20 0x7f472c19a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:53.388 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.906+0000 7f47317a7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f472c10eb90 0x7f472c19ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:53.388 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.906+0000 7f47317a7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c19f0a0 con 0x7f472c073210 2022-09-07T21:39:53.389 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.906+0000 7f47317a7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c19f210 con 0x7f472c073b20 2022-09-07T21:39:53.389 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.906+0000 7f47317a7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c19f3d0 con 0x7f472c10eb90 2022-09-07T21:39:53.389 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.906+0000 7f472b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f472c073b20 0x7f472c19a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:53.389 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.906+0000 7f472bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f472c073210 0x7f472c19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:53.390 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.906+0000 7f4730fa6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f472c10eb90 0x7f472c19ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:53.390 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.906+0000 7f472b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f472c073b20 0x7f472c19a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:42034/0 (socket says 172.21.15.47:42034) 2022-09-07T21:39:53.390 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.906+0000 7f472b7fe700 1 -- 172.21.15.47:0/146350972 learned_addr learned my addr 172.21.15.47:0/146350972 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:53.390 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.907+0000 7f472b7fe700 1 -- 172.21.15.47:0/146350972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f472c10eb90 msgr2=0x7f472c19ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:53.391 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.907+0000 7f472b7fe700 1 --2- 172.21.15.47:0/146350972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f472c10eb90 0x7f472c19ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:53.391 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.907+0000 7f472b7fe700 1 -- 172.21.15.47:0/146350972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f472c073210 msgr2=0x7f472c19a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:53.391 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.907+0000 7f472b7fe700 1 --2- 172.21.15.47:0/146350972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f472c073210 0x7f472c19a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:53.391 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.907+0000 7f472b7fe700 1 -- 172.21.15.47:0/146350972 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4724009730 con 0x7f472c073b20 2022-09-07T21:39:53.391 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.907+0000 7f4730fa6700 1 --2- 172.21.15.47:0/146350972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f472c10eb90 0x7f472c19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:53.392 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.907+0000 7f472bfff700 1 --2- 172.21.15.47:0/146350972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f472c073210 0x7f472c19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:53.392 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.907+0000 7f472b7fe700 1 --2- 172.21.15.47:0/146350972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f472c073b20 0x7f472c19a5b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f472000e740 tx=0x7f472000ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:53.392 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.907+0000 7f47297fa700 1 -- 172.21.15.47:0/146350972 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4720005220 con 0x7f472c073b20 2022-09-07T21:39:53.392 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.907+0000 7f47297fa700 1 -- 172.21.15.47:0/146350972 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4720005380 con 0x7f472c073b20 2022-09-07T21:39:53.393 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.907+0000 7f47317a7700 1 -- 172.21.15.47:0/146350972 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472c19f6b0 con 0x7f472c073b20 2022-09-07T21:39:53.393 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.907+0000 7f47297fa700 1 -- 172.21.15.47:0/146350972 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4720010ba0 con 0x7f472c073b20 2022-09-07T21:39:53.393 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.908+0000 7f47317a7700 1 -- 172.21.15.47:0/146350972 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c1a6ea0 con 0x7f472c073b20 2022-09-07T21:39:53.393 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.909+0000 7f47317a7700 1 -- 172.21.15.47:0/146350972 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472c04d480 con 0x7f472c073b20 2022-09-07T21:39:53.394 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.910+0000 7f47297fa700 1 -- 172.21.15.47:0/146350972 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f472001e020 con 0x7f472c073b20 2022-09-07T21:39:53.394 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.912+0000 7f47297fa700 1 --2- 172.21.15.47:0/146350972 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4714099050 0x7f471409b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:53.394 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.912+0000 7f47297fa700 1 -- 172.21.15.47:0/146350972 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f4720014070 con 0x7f472c073b20 2022-09-07T21:39:53.394 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.912+0000 7f472bfff700 1 --2- 172.21.15.47:0/146350972 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4714099050 0x7f471409b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:53.395 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.913+0000 7f472bfff700 1 --2- 172.21.15.47:0/146350972 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4714099050 0x7f471409b500 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f471c005fd0 tx=0x7f471c005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:53.395 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:52.916+0000 7f47297fa700 1 -- 172.21.15.47:0/146350972 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f472008ecc0 con 0x7f472c073b20 2022-09-07T21:39:53.395 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.160+0000 7f47317a7700 1 -- 172.21.15.47:0/146350972 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f472c1140e0 con 0x7f472c073b20 2022-09-07T21:39:53.395 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.161+0000 7f47297fa700 1 -- 172.21.15.47:0/146350972 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f47200922e0 con 0x7f472c073b20 2022-09-07T21:39:53.396 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.163+0000 7f47317a7700 1 -- 172.21.15.47:0/146350972 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4714099050 msgr2=0x7f471409b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:53.396 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.163+0000 7f47317a7700 1 --2- 172.21.15.47:0/146350972 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4714099050 0x7f471409b500 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f471c005fd0 tx=0x7f471c005f60 comp rx=0 tx=0).stop 2022-09-07T21:39:53.396 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.163+0000 7f47317a7700 1 -- 172.21.15.47:0/146350972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f472c073b20 msgr2=0x7f472c19a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:53.396 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.163+0000 7f47317a7700 1 --2- 172.21.15.47:0/146350972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f472c073b20 0x7f472c19a5b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f472000e740 tx=0x7f472000ea50 comp rx=0 tx=0).stop 2022-09-07T21:39:53.397 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.164+0000 7f47317a7700 1 -- 172.21.15.47:0/146350972 shutdown_connections 2022-09-07T21:39:53.397 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.164+0000 7f47317a7700 1 --2- 172.21.15.47:0/146350972 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4714099050 0x7f471409b500 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:53.397 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.164+0000 7f47317a7700 1 --2- 172.21.15.47:0/146350972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f472c10eb90 0x7f472c19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:53.397 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.164+0000 7f47317a7700 1 --2- 172.21.15.47:0/146350972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f472c073210 0x7f472c19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:53.397 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.164+0000 7f47317a7700 1 --2- 172.21.15.47:0/146350972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f472c073b20 0x7f472c19a5b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:53.398 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.164+0000 7f47317a7700 1 -- 172.21.15.47:0/146350972 >> 172.21.15.47:0/146350972 conn(0x7f472c0fb9f0 msgr2=0x7f472c0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:53.398 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.164+0000 7f47317a7700 1 -- 172.21.15.47:0/146350972 shutdown_connections 2022-09-07T21:39:53.398 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.164+0000 7f47317a7700 1 -- 172.21.15.47:0/146350972 wait complete. 2022-09-07T21:39:53.399 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757306 got 163208757309 for osd.5 2022-09-07T21:39:53.400 DEBUG:teuthology.parallel:result is None 2022-09-07T21:39:53.769 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574928 got 55834574931 for osd.1 2022-09-07T21:39:53.769 DEBUG:teuthology.parallel:result is None 2022-09-07T21:39:54.013 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:53 smithi047 ceph-mon[90087]: pgmap v354: 129 pgs: 1 activating, 1 peering, 127 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1 op/s; 2/194 objects degraded (1.031%); 70 B/s, 1 keys/s, 0 objects/s recovering 2022-09-07T21:39:54.014 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:53 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/989161104' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-09-07T21:39:54.014 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:53 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/146350972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-09-07T21:39:54.014 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:53 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/614189971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-09-07T21:39:54.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.207+0000 7f62fb77b700 1 -- 172.21.15.47:0/3840857149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f410a4b0 msgr2=0x7f62f410a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:54.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.207+0000 7f62fb77b700 1 --2- 172.21.15.47:0/3840857149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f410a4b0 0x7f62f410a880 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f62f0009810 tx=0x7f62f0009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:54.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.207+0000 7f62fb77b700 1 -- 172.21.15.47:0/3840857149 shutdown_connections 2022-09-07T21:39:54.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.207+0000 7f62fb77b700 1 --2- 172.21.15.47:0/3840857149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62f406fdf0 0x7f62f4112440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.207+0000 7f62fb77b700 1 --2- 172.21.15.47:0/3840857149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f410a4b0 0x7f62f410a880 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.207+0000 7f62fb77b700 1 --2- 172.21.15.47:0/3840857149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f410ae50 0x7f62f406f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.207+0000 7f62fb77b700 1 -- 172.21.15.47:0/3840857149 >> 172.21.15.47:0/3840857149 conn(0x7f62f406a070 msgr2=0x7f62f406d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:54.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.207+0000 7f62fb77b700 1 -- 172.21.15.47:0/3840857149 shutdown_connections 2022-09-07T21:39:54.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.207+0000 7f62fb77b700 1 -- 172.21.15.47:0/3840857149 wait complete. 2022-09-07T21:39:54.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.209+0000 7f62fb77b700 1 Processor -- start 2022-09-07T21:39:54.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.209+0000 7f62fb77b700 1 -- start start 2022-09-07T21:39:54.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.209+0000 7f62fb77b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f406fdf0 0x7f62f419e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:54.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.209+0000 7f62fb77b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f410a4b0 0x7f62f419eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:54.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.209+0000 7f62fb77b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62f41a2f60 0x7f62f41a33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:54.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.209+0000 7f62fb77b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41a39d0 con 0x7f62f406fdf0 2022-09-07T21:39:54.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.209+0000 7f62fb77b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41a3b40 con 0x7f62f41a2f60 2022-09-07T21:39:54.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.209+0000 7f62fb77b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41a3d00 con 0x7f62f410a4b0 2022-09-07T21:39:54.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.210+0000 7f62f9517700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f406fdf0 0x7f62f419e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:54.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.210+0000 7f62f9517700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f406fdf0 0x7f62f419e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52410/0 (socket says 172.21.15.47:52410) 2022-09-07T21:39:54.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.210+0000 7f62f9517700 1 -- 172.21.15.47:0/614189971 learned_addr learned my addr 172.21.15.47:0/614189971 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:54.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.210+0000 7f62f9d18700 1 --2- 172.21.15.47:0/614189971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62f41a2f60 0x7f62f41a33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:54.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.210+0000 7f62f8d16700 1 --2- 172.21.15.47:0/614189971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f410a4b0 0x7f62f419eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:54.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.210+0000 7f62f9517700 1 -- 172.21.15.47:0/614189971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f410a4b0 msgr2=0x7f62f419eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:54.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.210+0000 7f62f9517700 1 --2- 172.21.15.47:0/614189971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f410a4b0 0x7f62f419eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.210+0000 7f62f9517700 1 -- 172.21.15.47:0/614189971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62f41a2f60 msgr2=0x7f62f41a33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:54.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.210+0000 7f62f9517700 1 --2- 172.21.15.47:0/614189971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62f41a2f60 0x7f62f41a33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.210+0000 7f62f9517700 1 -- 172.21.15.47:0/614189971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f0009470 con 0x7f62f406fdf0 2022-09-07T21:39:54.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.211+0000 7f62f9517700 1 --2- 172.21.15.47:0/614189971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f406fdf0 0x7f62f419e5c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f62f000f690 tx=0x7f62f000f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:54.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.211+0000 7f62ea7fc700 1 -- 172.21.15.47:0/614189971 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f001d070 con 0x7f62f406fdf0 2022-09-07T21:39:54.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.211+0000 7f62fb77b700 1 -- 172.21.15.47:0/614189971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f41ab270 con 0x7f62f406fdf0 2022-09-07T21:39:54.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.211+0000 7f62fb77b700 1 -- 172.21.15.47:0/614189971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f41ab720 con 0x7f62f406fdf0 2022-09-07T21:39:54.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.211+0000 7f62ea7fc700 1 -- 172.21.15.47:0/614189971 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f62f0003500 con 0x7f62f406fdf0 2022-09-07T21:39:54.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.211+0000 7f62ea7fc700 1 -- 172.21.15.47:0/614189971 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0004760 con 0x7f62f406fdf0 2022-09-07T21:39:54.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.212+0000 7f62fb77b700 1 -- 172.21.15.47:0/614189971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f4064ea0 con 0x7f62f406fdf0 2022-09-07T21:39:54.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.215+0000 7f62ea7fc700 1 -- 172.21.15.47:0/614189971 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f62f0028030 con 0x7f62f406fdf0 2022-09-07T21:39:54.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.216+0000 7f62ea7fc700 1 --2- 172.21.15.47:0/614189971 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f62e00990d0 0x7f62e009b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:54.097 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.216+0000 7f62ea7fc700 1 -- 172.21.15.47:0/614189971 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f62f00c38b0 con 0x7f62f406fdf0 2022-09-07T21:39:54.097 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.217+0000 7f62f8d16700 1 --2- 172.21.15.47:0/614189971 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f62e00990d0 0x7f62e009b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:54.097 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.218+0000 7f62f8d16700 1 --2- 172.21.15.47:0/614189971 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f62e00990d0 0x7f62e009b580 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f62f419fbd0 tx=0x7f62e400d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:54.097 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.219+0000 7f62ea7fc700 1 -- 172.21.15.47:0/614189971 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f62f008fbf0 con 0x7f62f406fdf0 2022-09-07T21:39:54.098 INFO:teuthology.orchestra.run.smithi047.stdout:287762808869 2022-09-07T21:39:54.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.488+0000 7f62fb77b700 1 -- 172.21.15.47:0/614189971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f62f419f9b0 con 0x7f62f406fdf0 2022-09-07T21:39:54.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.488+0000 7f62ea7fc700 1 -- 172.21.15.47:0/614189971 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f62f0026070 con 0x7f62f406fdf0 2022-09-07T21:39:54.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.491+0000 7f62fb77b700 1 -- 172.21.15.47:0/614189971 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f62e00990d0 msgr2=0x7f62e009b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:54.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.491+0000 7f62fb77b700 1 --2- 172.21.15.47:0/614189971 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f62e00990d0 0x7f62e009b580 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f62f419fbd0 tx=0x7f62e400d040 comp rx=0 tx=0).stop 2022-09-07T21:39:54.100 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.491+0000 7f62fb77b700 1 -- 172.21.15.47:0/614189971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f406fdf0 msgr2=0x7f62f419e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:54.100 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.491+0000 7f62fb77b700 1 --2- 172.21.15.47:0/614189971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f406fdf0 0x7f62f419e5c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f62f000f690 tx=0x7f62f000f6c0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.101 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.491+0000 7f62fb77b700 1 -- 172.21.15.47:0/614189971 shutdown_connections 2022-09-07T21:39:54.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.491+0000 7f62fb77b700 1 --2- 172.21.15.47:0/614189971 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f62e00990d0 0x7f62e009b580 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.491+0000 7f62fb77b700 1 --2- 172.21.15.47:0/614189971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62f41a2f60 0x7f62f41a33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.491+0000 7f62fb77b700 1 --2- 172.21.15.47:0/614189971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f406fdf0 0x7f62f419e5c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.491+0000 7f62fb77b700 1 --2- 172.21.15.47:0/614189971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f410a4b0 0x7f62f419eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.491+0000 7f62fb77b700 1 -- 172.21.15.47:0/614189971 >> 172.21.15.47:0/614189971 conn(0x7f62f406a070 msgr2=0x7f62f406a440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:54.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.491+0000 7f62fb77b700 1 -- 172.21.15.47:0/614189971 shutdown_connections 2022-09-07T21:39:54.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:53.491+0000 7f62fb77b700 1 -- 172.21.15.47:0/614189971 wait complete. 2022-09-07T21:39:54.213 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495668 got 197568495671 for osd.6 2022-09-07T21:39:54.213 DEBUG:teuthology.parallel:result is None 2022-09-07T21:39:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:53 smithi055 ceph-mon[97033]: pgmap v354: 129 pgs: 1 activating, 1 peering, 127 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1 op/s; 2/194 objects degraded (1.031%); 70 B/s, 1 keys/s, 0 objects/s recovering 2022-09-07T21:39:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:53 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/989161104' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-09-07T21:39:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:53 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/146350972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-09-07T21:39:54.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:53 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/614189971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-09-07T21:39:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:53 smithi059 ceph-mon[97450]: pgmap v354: 129 pgs: 1 activating, 1 peering, 127 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1 op/s; 2/194 objects degraded (1.031%); 70 B/s, 1 keys/s, 0 objects/s recovering 2022-09-07T21:39:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:53 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/989161104' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-09-07T21:39:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:53 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/146350972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-09-07T21:39:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:53 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/614189971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-09-07T21:39:54.614 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.180+0000 7f5a076ef700 1 -- 172.21.15.47:0/989618788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a00066da0 msgr2=0x7f5a0010ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:54.614 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.180+0000 7f5a076ef700 1 --2- 172.21.15.47:0/989618788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a00066da0 0x7f5a0010ad60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009b00 tx=0x7f59f8009e10 comp rx=0 tx=0).stop 2022-09-07T21:39:54.614 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.181+0000 7f5a076ef700 1 -- 172.21.15.47:0/989618788 shutdown_connections 2022-09-07T21:39:54.614 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.181+0000 7f5a076ef700 1 --2- 172.21.15.47:0/989618788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a00066da0 0x7f5a0010ad60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.615 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.181+0000 7f5a076ef700 1 --2- 172.21.15.47:0/989618788 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a00103ea0 0x7f5a00104270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.615 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.181+0000 7f5a076ef700 1 --2- 172.21.15.47:0/989618788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a00104840 0x7f5a000666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.615 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.181+0000 7f5a076ef700 1 -- 172.21.15.47:0/989618788 >> 172.21.15.47:0/989618788 conn(0x7f5a00073cf0 msgr2=0x7f5a000740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:54.615 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.181+0000 7f5a076ef700 1 -- 172.21.15.47:0/989618788 shutdown_connections 2022-09-07T21:39:54.616 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.181+0000 7f5a076ef700 1 -- 172.21.15.47:0/989618788 wait complete. 2022-09-07T21:39:54.616 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.182+0000 7f5a076ef700 1 Processor -- start 2022-09-07T21:39:54.616 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.182+0000 7f5a076ef700 1 -- start start 2022-09-07T21:39:54.616 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a076ef700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a00066da0 0x7f5a00195fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:54.617 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a076ef700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a00103ea0 0x7f5a00196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:54.617 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a076ef700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a00104840 0x7f5a0019a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:54.617 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a076ef700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0019aff0 con 0x7f5a00103ea0 2022-09-07T21:39:54.617 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a076ef700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0019b160 con 0x7f5a00066da0 2022-09-07T21:39:54.617 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a076ef700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0019b320 con 0x7f5a00104840 2022-09-07T21:39:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a04c8a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a00103ea0 0x7f5a00196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a05c8c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a00104840 0x7f5a0019a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a0548b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a00066da0 0x7f5a00195fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a05c8c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a00104840 0x7f5a0019a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60090/0 (socket says 172.21.15.47:60090) 2022-09-07T21:39:54.619 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a0548b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a00066da0 0x7f5a00195fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:42076/0 (socket says 172.21.15.47:42076) 2022-09-07T21:39:54.619 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a05c8c700 1 -- 172.21.15.47:0/3389853989 learned_addr learned my addr 172.21.15.47:0/3389853989 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:54.619 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a05c8c700 1 -- 172.21.15.47:0/3389853989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a00066da0 msgr2=0x7f5a00195fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:54.620 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a05c8c700 1 --2- 172.21.15.47:0/3389853989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a00066da0 0x7f5a00195fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.620 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a05c8c700 1 -- 172.21.15.47:0/3389853989 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a00103ea0 msgr2=0x7f5a00196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:54.620 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a05c8c700 1 --2- 172.21.15.47:0/3389853989 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a00103ea0 0x7f5a00196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.620 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a05c8c700 1 -- 172.21.15.47:0/3389853989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f8009730 con 0x7f5a00104840 2022-09-07T21:39:54.620 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.183+0000 7f5a0548b700 1 --2- 172.21.15.47:0/3389853989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a00066da0 0x7f5a00195fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:54.621 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.184+0000 7f5a05c8c700 1 --2- 172.21.15.47:0/3389853989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a00104840 0x7f5a0019a960 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009fd0 tx=0x7f59f800bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:54.621 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.184+0000 7f59f67fc700 1 -- 172.21.15.47:0/3389853989 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f800f7d0 con 0x7f5a00104840 2022-09-07T21:39:54.621 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.184+0000 7f5a076ef700 1 -- 172.21.15.47:0/3389853989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0019b5a0 con 0x7f5a00104840 2022-09-07T21:39:54.621 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.185+0000 7f59f67fc700 1 -- 172.21.15.47:0/3389853989 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f59f8005040 con 0x7f5a00104840 2022-09-07T21:39:54.621 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.185+0000 7f59f67fc700 1 -- 172.21.15.47:0/3389853989 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f8017660 con 0x7f5a00104840 2022-09-07T21:39:54.622 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.185+0000 7f5a076ef700 1 -- 172.21.15.47:0/3389853989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a001a2df0 con 0x7f5a00104840 2022-09-07T21:39:54.622 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.186+0000 7f5a076ef700 1 -- 172.21.15.47:0/3389853989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a00064ea0 con 0x7f5a00104840 2022-09-07T21:39:54.622 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.189+0000 7f59f67fc700 1 -- 172.21.15.47:0/3389853989 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f59f8027020 con 0x7f5a00104840 2022-09-07T21:39:54.623 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.192+0000 7f59f67fc700 1 --2- 172.21.15.47:0/3389853989 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f59ec099050 0x7f59ec09b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:54.623 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.192+0000 7f59f67fc700 1 -- 172.21.15.47:0/3389853989 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f59f80c42d0 con 0x7f5a00104840 2022-09-07T21:39:54.623 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.192+0000 7f5a0548b700 1 --2- 172.21.15.47:0/3389853989 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f59ec099050 0x7f59ec09b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:54.623 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.193+0000 7f59f67fc700 1 -- 172.21.15.47:0/3389853989 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f59f8090610 con 0x7f5a00104840 2022-09-07T21:39:54.624 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.193+0000 7f5a0548b700 1 --2- 172.21.15.47:0/3389853989 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f59ec099050 0x7f59ec09b500 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f59fc005fd0 tx=0x7f59fc005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:54.624 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.422+0000 7f5a076ef700 1 -- 172.21.15.47:0/3389853989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f5a001973b0 con 0x7f5a00104840 2022-09-07T21:39:54.624 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.423+0000 7f59f67fc700 1 -- 172.21.15.47:0/3389853989 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f59f8093c30 con 0x7f5a00104840 2022-09-07T21:39:54.625 INFO:teuthology.orchestra.run.smithi047.stdout:253403070508 2022-09-07T21:39:54.625 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.425+0000 7f5a076ef700 1 -- 172.21.15.47:0/3389853989 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f59ec099050 msgr2=0x7f59ec09b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:54.626 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.425+0000 7f5a076ef700 1 --2- 172.21.15.47:0/3389853989 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f59ec099050 0x7f59ec09b500 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f59fc005fd0 tx=0x7f59fc005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.626 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.425+0000 7f5a076ef700 1 -- 172.21.15.47:0/3389853989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a00104840 msgr2=0x7f5a0019a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:54.626 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.425+0000 7f5a076ef700 1 --2- 172.21.15.47:0/3389853989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a00104840 0x7f5a0019a960 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009fd0 tx=0x7f59f800bb40 comp rx=0 tx=0).stop 2022-09-07T21:39:54.626 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.425+0000 7f5a076ef700 1 -- 172.21.15.47:0/3389853989 shutdown_connections 2022-09-07T21:39:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.425+0000 7f5a076ef700 1 --2- 172.21.15.47:0/3389853989 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f59ec099050 0x7f59ec09b500 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.425+0000 7f5a076ef700 1 --2- 172.21.15.47:0/3389853989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a00104840 0x7f5a0019a960 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.425+0000 7f5a076ef700 1 --2- 172.21.15.47:0/3389853989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a00066da0 0x7f5a00195fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.425+0000 7f5a076ef700 1 --2- 172.21.15.47:0/3389853989 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a00103ea0 0x7f5a00196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.425+0000 7f5a076ef700 1 -- 172.21.15.47:0/3389853989 >> 172.21.15.47:0/3389853989 conn(0x7f5a00073cf0 msgr2=0x7f5a000fb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:54.628 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.425+0000 7f5a076ef700 1 -- 172.21.15.47:0/3389853989 shutdown_connections 2022-09-07T21:39:54.628 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:54.425+0000 7f5a076ef700 1 -- 172.21.15.47:0/3389853989 wait complete. 2022-09-07T21:39:55.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:54 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3389853989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-09-07T21:39:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:54 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3389853989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-09-07T21:39:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:54 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3389853989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-09-07T21:39:55.632 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182469 got 107374182471 for osd.3 2022-09-07T21:39:55.633 DEBUG:teuthology.parallel:result is None 2022-09-07T21:39:55.718 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:39:55.873 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808866 got 287762808869 for osd.9 2022-09-07T21:39:55.874 DEBUG:teuthology.parallel:result is None 2022-09-07T21:39:55.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:55 smithi047 ceph-mon[90087]: pgmap v355: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 4 op/s; 191 B/s, 1 keys/s, 1 objects/s recovering 2022-09-07T21:39:55.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:55 smithi047 ceph-mon[90087]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2022-09-07T21:39:55.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:55 smithi047 ceph-mon[90087]: Cluster is now healthy 2022-09-07T21:39:56.057 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.053+0000 7fc3183f7700 1 -- 172.21.15.47:0/1924685713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc310071080 msgr2=0x7fc310071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.053+0000 7fc3183f7700 1 --2- 172.21.15.47:0/1924685713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc310071080 0x7fc310071450 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc30c009810 tx=0x7fc30c009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:56.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.054+0000 7fc3183f7700 1 -- 172.21.15.47:0/1924685713 shutdown_connections 2022-09-07T21:39:56.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.054+0000 7fc3183f7700 1 --2- 172.21.15.47:0/1924685713 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc31010ef00 0x7fc3101112e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.054+0000 7fc3183f7700 1 --2- 172.21.15.47:0/1924685713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc310071080 0x7fc310071450 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.054+0000 7fc3183f7700 1 --2- 172.21.15.47:0/1924685713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc310071990 0x7fc31010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.054+0000 7fc3183f7700 1 -- 172.21.15.47:0/1924685713 >> 172.21.15.47:0/1924685713 conn(0x7fc3100f9a10 msgr2=0x7fc3100fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:56.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.054+0000 7fc3183f7700 1 -- 172.21.15.47:0/1924685713 shutdown_connections 2022-09-07T21:39:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.055+0000 7fc3183f7700 1 -- 172.21.15.47:0/1924685713 wait complete. 2022-09-07T21:39:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.056+0000 7fc3183f7700 1 Processor -- start 2022-09-07T21:39:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.056+0000 7fc3183f7700 1 -- start start 2022-09-07T21:39:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.056+0000 7fc3183f7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc310071080 0x7fc31019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.056+0000 7fc3183f7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc310071990 0x7fc31019a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.056+0000 7fc3183f7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc31010ef00 0x7fc31019ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.056+0000 7fc3183f7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31019f320 con 0x7fc31010ef00 2022-09-07T21:39:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.056+0000 7fc3183f7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31019f490 con 0x7fc310071080 2022-09-07T21:39:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.056+0000 7fc3183f7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31019f650 con 0x7fc310071990 2022-09-07T21:39:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.057+0000 7fc316193700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc310071080 0x7fc31019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.057+0000 7fc316193700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc310071080 0x7fc31019a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:42088/0 (socket says 172.21.15.47:42088) 2022-09-07T21:39:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.057+0000 7fc316193700 1 -- 172.21.15.47:0/1539891262 learned_addr learned my addr 172.21.15.47:0/1539891262 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.057+0000 7fc316994700 1 --2- 172.21.15.47:0/1539891262 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc31010ef00 0x7fc31019ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:56.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.057+0000 7fc315992700 1 --2- 172.21.15.47:0/1539891262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc310071990 0x7fc31019a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:56.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.057+0000 7fc316193700 1 -- 172.21.15.47:0/1539891262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc310071990 msgr2=0x7fc31019a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.057+0000 7fc316193700 1 --2- 172.21.15.47:0/1539891262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc310071990 0x7fc31019a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.057+0000 7fc316193700 1 -- 172.21.15.47:0/1539891262 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc31010ef00 msgr2=0x7fc31019ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.057+0000 7fc316193700 1 --2- 172.21.15.47:0/1539891262 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc31010ef00 0x7fc31019ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.057+0000 7fc316193700 1 -- 172.21.15.47:0/1539891262 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30c009470 con 0x7fc310071080 2022-09-07T21:39:56.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.058+0000 7fc316193700 1 --2- 172.21.15.47:0/1539891262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc310071080 0x7fc31019a2f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0059c0 tx=0x7fc30c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:56.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.058+0000 7fc3077fe700 1 -- 172.21.15.47:0/1539891262 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30c01d070 con 0x7fc310071080 2022-09-07T21:39:56.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.058+0000 7fc3183f7700 1 -- 172.21.15.47:0/1539891262 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc31019f8d0 con 0x7fc310071080 2022-09-07T21:39:56.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.058+0000 7fc3077fe700 1 -- 172.21.15.47:0/1539891262 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fc30c00f800 con 0x7fc310071080 2022-09-07T21:39:56.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.058+0000 7fc3077fe700 1 -- 172.21.15.47:0/1539891262 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30c004820 con 0x7fc310071080 2022-09-07T21:39:56.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.058+0000 7fc3183f7700 1 -- 172.21.15.47:0/1539891262 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3101a7120 con 0x7fc310071080 2022-09-07T21:39:56.069 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.060+0000 7fc3057fa700 1 -- 172.21.15.47:0/1539891262 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f0005170 con 0x7fc310071080 2022-09-07T21:39:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.063+0000 7fc3077fe700 1 -- 172.21.15.47:0/1539891262 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7fc30c028030 con 0x7fc310071080 2022-09-07T21:39:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.064+0000 7fc3077fe700 1 --2- 172.21.15.47:0/1539891262 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc2fc0990d0 0x7fc2fc09b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.064+0000 7fc3077fe700 1 -- 172.21.15.47:0/1539891262 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fc30c0c33e0 con 0x7fc310071080 2022-09-07T21:39:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.065+0000 7fc315992700 1 --2- 172.21.15.47:0/1539891262 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc2fc0990d0 0x7fc2fc09b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:56.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.065+0000 7fc315992700 1 --2- 172.21.15.47:0/1539891262 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc2fc0990d0 0x7fc2fc09b580 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc31019b900 tx=0x7fc300005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:56.072 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.068+0000 7fc3077fe700 1 -- 172.21.15.47:0/1539891262 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fc30c08f780 con 0x7fc310071080 2022-09-07T21:39:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:55 smithi055 ceph-mon[97033]: pgmap v355: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 4 op/s; 191 B/s, 1 keys/s, 1 objects/s recovering 2022-09-07T21:39:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:55 smithi055 ceph-mon[97033]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2022-09-07T21:39:56.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:55 smithi055 ceph-mon[97033]: Cluster is now healthy 2022-09-07T21:39:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:55 smithi059 ceph-mon[97450]: pgmap v355: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 4 op/s; 191 B/s, 1 keys/s, 1 objects/s recovering 2022-09-07T21:39:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:55 smithi059 ceph-mon[97450]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2022-09-07T21:39:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:55 smithi059 ceph-mon[97450]: Cluster is now healthy 2022-09-07T21:39:56.330 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.327+0000 7fc3057fa700 1 -- 172.21.15.47:0/1539891262 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fc2f0004fe0 con 0x7fc310071080 2022-09-07T21:39:56.331 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.327+0000 7fc3077fe700 1 -- 172.21.15.47:0/1539891262 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fc30c026020 con 0x7fc310071080 2022-09-07T21:39:56.331 INFO:teuthology.orchestra.run.smithi047.stdout:343597383707 2022-09-07T21:39:56.334 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.330+0000 7fc3057fa700 1 -- 172.21.15.47:0/1539891262 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc2fc0990d0 msgr2=0x7fc2fc09b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.334 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.330+0000 7fc3057fa700 1 --2- 172.21.15.47:0/1539891262 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc2fc0990d0 0x7fc2fc09b580 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc31019b900 tx=0x7fc300005ea0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.334 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.330+0000 7fc3057fa700 1 -- 172.21.15.47:0/1539891262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc310071080 msgr2=0x7fc31019a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.335 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.330+0000 7fc3057fa700 1 --2- 172.21.15.47:0/1539891262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc310071080 0x7fc31019a2f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0059c0 tx=0x7fc30c00f690 comp rx=0 tx=0).stop 2022-09-07T21:39:56.335 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.330+0000 7fc3057fa700 1 -- 172.21.15.47:0/1539891262 shutdown_connections 2022-09-07T21:39:56.335 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.330+0000 7fc3057fa700 1 --2- 172.21.15.47:0/1539891262 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc2fc0990d0 0x7fc2fc09b580 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.335 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.330+0000 7fc3057fa700 1 --2- 172.21.15.47:0/1539891262 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc31010ef00 0x7fc31019ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.336 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.330+0000 7fc3057fa700 1 --2- 172.21.15.47:0/1539891262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc310071080 0x7fc31019a2f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.336 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.330+0000 7fc3057fa700 1 --2- 172.21.15.47:0/1539891262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc310071990 0x7fc31019a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.336 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.330+0000 7fc3057fa700 1 -- 172.21.15.47:0/1539891262 >> 172.21.15.47:0/1539891262 conn(0x7fc3100f9a10 msgr2=0x7fc3100fa650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:56.336 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.330+0000 7fc3057fa700 1 -- 172.21.15.47:0/1539891262 shutdown_connections 2022-09-07T21:39:56.337 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.330+0000 7fc3057fa700 1 -- 172.21.15.47:0/1539891262 wait complete. 2022-09-07T21:39:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.208+0000 7f3a81385700 1 -- 172.21.15.47:0/76544186 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a7c102d40 msgr2=0x7f3a7c0666c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.208+0000 7f3a81385700 1 --2- 172.21.15.47:0/76544186 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a7c102d40 0x7f3a7c0666c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009750 tx=0x7f3a6c009a60 comp rx=0 tx=0).stop 2022-09-07T21:39:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.208+0000 7f3a81385700 1 -- 172.21.15.47:0/76544186 shutdown_connections 2022-09-07T21:39:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.208+0000 7f3a81385700 1 --2- 172.21.15.47:0/76544186 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a7c1008d0 0x7f3a7c100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.460 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.209+0000 7f3a81385700 1 --2- 172.21.15.47:0/76544186 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a7c102d40 0x7f3a7c0666c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.460 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.209+0000 7f3a81385700 1 --2- 172.21.15.47:0/76544186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a7c066c00 0x7f3a7c067050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.460 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.209+0000 7f3a81385700 1 -- 172.21.15.47:0/76544186 >> 172.21.15.47:0/76544186 conn(0x7f3a7c073cf0 msgr2=0x7f3a7c0740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:56.460 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.209+0000 7f3a81385700 1 -- 172.21.15.47:0/76544186 shutdown_connections 2022-09-07T21:39:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.209+0000 7f3a81385700 1 -- 172.21.15.47:0/76544186 wait complete. 2022-09-07T21:39:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.211+0000 7f3a81385700 1 Processor -- start 2022-09-07T21:39:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.211+0000 7f3a81385700 1 -- start start 2022-09-07T21:39:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.212+0000 7f3a81385700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a7c066c00 0x7f3a7c10a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.212+0000 7f3a81385700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a7c1008d0 0x7f3a7c10a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.212+0000 7f3a81385700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a7c102d40 0x7f3a7c1ae790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.212+0000 7f3a81385700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c1aedf0 con 0x7f3a7c1008d0 2022-09-07T21:39:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.212+0000 7f3a81385700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c068700 con 0x7f3a7c066c00 2022-09-07T21:39:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.212+0000 7f3a81385700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c0688c0 con 0x7f3a7c102d40 2022-09-07T21:39:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.213+0000 7f3a7affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a7c066c00 0x7f3a7c10a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.213+0000 7f3a7affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a7c066c00 0x7f3a7c10a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:42106/0 (socket says 172.21.15.47:42106) 2022-09-07T21:39:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.213+0000 7f3a7affd700 1 -- 172.21.15.47:0/3225128243 learned_addr learned my addr 172.21.15.47:0/3225128243 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:56.464 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.213+0000 7f3a7b7fe700 1 --2- 172.21.15.47:0/3225128243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a7c102d40 0x7f3a7c1ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:56.464 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.213+0000 7f3a7a7fc700 1 --2- 172.21.15.47:0/3225128243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a7c1008d0 0x7f3a7c10a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:56.464 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.213+0000 7f3a7affd700 1 -- 172.21.15.47:0/3225128243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a7c102d40 msgr2=0x7f3a7c1ae790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.464 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.213+0000 7f3a7affd700 1 --2- 172.21.15.47:0/3225128243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a7c102d40 0x7f3a7c1ae790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.465 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.213+0000 7f3a7affd700 1 -- 172.21.15.47:0/3225128243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a7c1008d0 msgr2=0x7f3a7c10a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.465 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.213+0000 7f3a7affd700 1 --2- 172.21.15.47:0/3225128243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a7c1008d0 0x7f3a7c10a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.465 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.213+0000 7f3a7affd700 1 -- 172.21.15.47:0/3225128243 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c0093b0 con 0x7f3a7c066c00 2022-09-07T21:39:56.465 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.214+0000 7f3a7affd700 1 --2- 172.21.15.47:0/3225128243 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a7c066c00 0x7f3a7c10a040 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c018630 tx=0x7f3a6c018660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:56.466 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.214+0000 7f3a63fff700 1 -- 172.21.15.47:0/3225128243 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c01d070 con 0x7f3a7c066c00 2022-09-07T21:39:56.466 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.214+0000 7f3a63fff700 1 -- 172.21.15.47:0/3225128243 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3a6c004500 con 0x7f3a7c066c00 2022-09-07T21:39:56.466 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.214+0000 7f3a63fff700 1 -- 172.21.15.47:0/3225128243 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c021c60 con 0x7f3a7c066c00 2022-09-07T21:39:56.466 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.215+0000 7f3a81385700 1 -- 172.21.15.47:0/3225128243 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7c068b40 con 0x7f3a7c066c00 2022-09-07T21:39:56.467 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.216+0000 7f3a81385700 1 -- 172.21.15.47:0/3225128243 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7c068fd0 con 0x7f3a7c066c00 2022-09-07T21:39:56.467 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.218+0000 7f3a81385700 1 -- 172.21.15.47:0/3225128243 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7c064ea0 con 0x7f3a7c066c00 2022-09-07T21:39:56.467 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.219+0000 7f3a63fff700 1 -- 172.21.15.47:0/3225128243 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f3a6c027070 con 0x7f3a7c066c00 2022-09-07T21:39:56.467 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.221+0000 7f3a63fff700 1 --2- 172.21.15.47:0/3225128243 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3a64099050 0x7f3a6409b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:56.468 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.221+0000 7f3a63fff700 1 -- 172.21.15.47:0/3225128243 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f3a6c0c3a30 con 0x7f3a7c066c00 2022-09-07T21:39:56.468 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.222+0000 7f3a7a7fc700 1 --2- 172.21.15.47:0/3225128243 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3a64099050 0x7f3a6409b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:56.468 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.223+0000 7f3a7a7fc700 1 --2- 172.21.15.47:0/3225128243 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3a64099050 0x7f3a6409b500 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c10b600 tx=0x7f3a70005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:56.468 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.225+0000 7f3a63fff700 1 -- 172.21.15.47:0/3225128243 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f3a6c08fd70 con 0x7f3a7c066c00 2022-09-07T21:39:56.469 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.465+0000 7f3a81385700 1 -- 172.21.15.47:0/3225128243 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3a7c10b410 con 0x7f3a7c066c00 2022-09-07T21:39:56.469 INFO:teuthology.orchestra.run.smithi047.stdout:34359738459 2022-09-07T21:39:56.470 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.465+0000 7f3a63fff700 1 -- 172.21.15.47:0/3225128243 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3a6c093390 con 0x7f3a7c066c00 2022-09-07T21:39:56.471 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.468+0000 7f3a81385700 1 -- 172.21.15.47:0/3225128243 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3a64099050 msgr2=0x7f3a6409b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.471 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.468+0000 7f3a81385700 1 --2- 172.21.15.47:0/3225128243 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3a64099050 0x7f3a6409b500 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c10b600 tx=0x7f3a70005f60 comp rx=0 tx=0).stop 2022-09-07T21:39:56.472 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.468+0000 7f3a81385700 1 -- 172.21.15.47:0/3225128243 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a7c066c00 msgr2=0x7f3a7c10a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.472 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.468+0000 7f3a81385700 1 --2- 172.21.15.47:0/3225128243 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a7c066c00 0x7f3a7c10a040 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c018630 tx=0x7f3a6c018660 comp rx=0 tx=0).stop 2022-09-07T21:39:56.472 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.468+0000 7f3a81385700 1 -- 172.21.15.47:0/3225128243 shutdown_connections 2022-09-07T21:39:56.472 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.468+0000 7f3a81385700 1 --2- 172.21.15.47:0/3225128243 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3a64099050 0x7f3a6409b500 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.473 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.468+0000 7f3a81385700 1 --2- 172.21.15.47:0/3225128243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a7c102d40 0x7f3a7c1ae790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.473 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.468+0000 7f3a81385700 1 --2- 172.21.15.47:0/3225128243 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a7c066c00 0x7f3a7c10a040 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.473 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.468+0000 7f3a81385700 1 --2- 172.21.15.47:0/3225128243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a7c1008d0 0x7f3a7c10a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.473 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.468+0000 7f3a81385700 1 -- 172.21.15.47:0/3225128243 >> 172.21.15.47:0/3225128243 conn(0x7f3a7c073cf0 msgr2=0x7f3a7c0ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:56.474 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.468+0000 7f3a81385700 1 -- 172.21.15.47:0/3225128243 shutdown_connections 2022-09-07T21:39:56.474 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.468+0000 7f3a81385700 1 -- 172.21.15.47:0/3225128243 wait complete. 2022-09-07T21:39:56.598 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.487+0000 7f0930c4d700 1 -- 172.21.15.47:0/40448043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f092c104840 msgr2=0x7f092c0666c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.598 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.487+0000 7f0930c4d700 1 --2- 172.21.15.47:0/40448043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f092c104840 0x7f092c0666c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0920009810 tx=0x7f0920009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:56.598 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.488+0000 7f0930c4d700 1 -- 172.21.15.47:0/40448043 shutdown_connections 2022-09-07T21:39:56.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.488+0000 7f0930c4d700 1 --2- 172.21.15.47:0/40448043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c066da0 0x7f092c10ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.488+0000 7f0930c4d700 1 --2- 172.21.15.47:0/40448043 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f092c103ea0 0x7f092c104270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.488+0000 7f0930c4d700 1 --2- 172.21.15.47:0/40448043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f092c104840 0x7f092c0666c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.488+0000 7f0930c4d700 1 -- 172.21.15.47:0/40448043 >> 172.21.15.47:0/40448043 conn(0x7f092c073cf0 msgr2=0x7f092c0740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.488+0000 7f0930c4d700 1 -- 172.21.15.47:0/40448043 shutdown_connections 2022-09-07T21:39:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.488+0000 7f0930c4d700 1 -- 172.21.15.47:0/40448043 wait complete. 2022-09-07T21:39:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.489+0000 7f0930c4d700 1 Processor -- start 2022-09-07T21:39:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.490+0000 7f0930c4d700 1 -- start start 2022-09-07T21:39:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.491+0000 7f0930c4d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f092c066da0 0x7f092c193d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.491+0000 7f0930c4d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f092c103ea0 0x7f092c1942c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.491+0000 7f0930c4d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c104840 0x7f092c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.491+0000 7f0930c4d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c198db0 con 0x7f092c066da0 2022-09-07T21:39:56.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.491+0000 7f0930c4d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c198f20 con 0x7f092c103ea0 2022-09-07T21:39:56.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.491+0000 7f0930c4d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c1990e0 con 0x7f092c104840 2022-09-07T21:39:56.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.491+0000 7f092ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c104840 0x7f092c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:56.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.491+0000 7f092ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c104840 0x7f092c198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:60142/0 (socket says 172.21.15.47:60142) 2022-09-07T21:39:56.603 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.491+0000 7f092ad9d700 1 -- 172.21.15.47:0/133797160 learned_addr learned my addr 172.21.15.47:0/133797160 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:56.603 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.491+0000 7f0929d9b700 1 --2- 172.21.15.47:0/133797160 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f092c103ea0 0x7f092c1942c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:56.603 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.491+0000 7f092ad9d700 1 -- 172.21.15.47:0/133797160 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f092c103ea0 msgr2=0x7f092c1942c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.603 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.492+0000 7f092ad9d700 1 --2- 172.21.15.47:0/133797160 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f092c103ea0 0x7f092c1942c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.604 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.492+0000 7f092ad9d700 1 -- 172.21.15.47:0/133797160 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f092c066da0 msgr2=0x7f092c193d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.604 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.492+0000 7f092ad9d700 1 --2- 172.21.15.47:0/133797160 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f092c066da0 0x7f092c193d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.604 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.492+0000 7f092ad9d700 1 -- 172.21.15.47:0/133797160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0924009730 con 0x7f092c104840 2022-09-07T21:39:56.605 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.492+0000 7f0929d9b700 1 --2- 172.21.15.47:0/133797160 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f092c103ea0 0x7f092c1942c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-07T21:39:56.605 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.492+0000 7f092ad9d700 1 --2- 172.21.15.47:0/133797160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c104840 0x7f092c198720 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f092400ec50 tx=0x7f092400c5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:56.605 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.492+0000 7f091b7fe700 1 -- 172.21.15.47:0/133797160 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092400ee50 con 0x7f092c104840 2022-09-07T21:39:56.605 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.492+0000 7f0930c4d700 1 -- 172.21.15.47:0/133797160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0920009470 con 0x7f092c104840 2022-09-07T21:39:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.492+0000 7f091b7fe700 1 -- 172.21.15.47:0/133797160 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0924003d20 con 0x7f092c104840 2022-09-07T21:39:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.492+0000 7f091b7fe700 1 -- 172.21.15.47:0/133797160 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924010690 con 0x7f092c104840 2022-09-07T21:39:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.493+0000 7f0930c4d700 1 -- 172.21.15.47:0/133797160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092c0fce60 con 0x7f092c104840 2022-09-07T21:39:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.494+0000 7f0930c4d700 1 -- 172.21.15.47:0/133797160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092c064ea0 con 0x7f092c104840 2022-09-07T21:39:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.497+0000 7f091b7fe700 1 -- 172.21.15.47:0/133797160 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f092400c040 con 0x7f092c104840 2022-09-07T21:39:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.498+0000 7f091b7fe700 1 --2- 172.21.15.47:0/133797160 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f091409d460 0x7f091409f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.499+0000 7f091b7fe700 1 -- 172.21.15.47:0/133797160 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f0924014070 con 0x7f092c104840 2022-09-07T21:39:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.499+0000 7f092a59c700 1 --2- 172.21.15.47:0/133797160 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f091409d460 0x7f091409f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.499+0000 7f091b7fe700 1 -- 172.21.15.47:0/133797160 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f092404ec80 con 0x7f092c104840 2022-09-07T21:39:56.608 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.502+0000 7f092a59c700 1 --2- 172.21.15.47:0/133797160 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f091409d460 0x7f091409f910 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f091c009630 tx=0x7f091c006ba0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:56.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.748+0000 7f0930c4d700 1 -- 172.21.15.47:0/133797160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f092c0fd560 con 0x7f092c104840 2022-09-07T21:39:56.752 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.748+0000 7f091b7fe700 1 -- 172.21.15.47:0/133797160 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f092408f3b0 con 0x7f092c104840 2022-09-07T21:39:56.752 INFO:teuthology.orchestra.run.smithi047.stdout:309237645344 2022-09-07T21:39:56.755 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.751+0000 7f0930c4d700 1 -- 172.21.15.47:0/133797160 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f091409d460 msgr2=0x7f091409f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.755 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.751+0000 7f0930c4d700 1 --2- 172.21.15.47:0/133797160 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f091409d460 0x7f091409f910 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f091c009630 tx=0x7f091c006ba0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.755 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.751+0000 7f0930c4d700 1 -- 172.21.15.47:0/133797160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c104840 msgr2=0x7f092c198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:56.755 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.751+0000 7f0930c4d700 1 --2- 172.21.15.47:0/133797160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c104840 0x7f092c198720 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f092400ec50 tx=0x7f092400c5b0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.752+0000 7f0930c4d700 1 -- 172.21.15.47:0/133797160 shutdown_connections 2022-09-07T21:39:56.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.752+0000 7f0930c4d700 1 --2- 172.21.15.47:0/133797160 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f091409d460 0x7f091409f910 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.757 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.752+0000 7f0930c4d700 1 --2- 172.21.15.47:0/133797160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c104840 0x7f092c198720 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.757 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.752+0000 7f0930c4d700 1 --2- 172.21.15.47:0/133797160 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f092c066da0 0x7f092c193d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.757 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.752+0000 7f0930c4d700 1 --2- 172.21.15.47:0/133797160 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f092c103ea0 0x7f092c1942c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:56.757 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.752+0000 7f0930c4d700 1 -- 172.21.15.47:0/133797160 >> 172.21.15.47:0/133797160 conn(0x7f092c073cf0 msgr2=0x7f092c0fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:56.758 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.752+0000 7f0930c4d700 1 -- 172.21.15.47:0/133797160 shutdown_connections 2022-09-07T21:39:56.758 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:56.752+0000 7f0930c4d700 1 -- 172.21.15.47:0/133797160 wait complete. 2022-09-07T21:39:57.118 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:56 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1539891262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-09-07T21:39:57.119 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:56 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3225128243' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-09-07T21:39:57.119 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:56 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/133797160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-09-07T21:39:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:56 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1539891262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-09-07T21:39:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:56 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3225128243' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-09-07T21:39:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:56 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/133797160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-09-07T21:39:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:56 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1539891262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-09-07T21:39:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:56 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3225128243' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-09-07T21:39:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:56 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/133797160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-09-07T21:39:57.414 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070506 got 253403070508 for osd.8 2022-09-07T21:39:57.415 DEBUG:teuthology.parallel:result is None 2022-09-07T21:39:57.468 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738456 got 34359738459 for osd.0 2022-09-07T21:39:57.469 DEBUG:teuthology.parallel:result is None 2022-09-07T21:39:57.973 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645341 got 309237645344 for osd.10 2022-09-07T21:39:57.973 DEBUG:teuthology.parallel:result is None 2022-09-07T21:39:58.080 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:57 smithi047 ceph-mon[90087]: pgmap v356: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 3 op/s; 155 B/s, 1 keys/s, 0 objects/s recovering 2022-09-07T21:39:58.084 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383704 got 343597383707 for osd.11 2022-09-07T21:39:58.085 DEBUG:teuthology.parallel:result is None 2022-09-07T21:39:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:57 smithi055 ceph-mon[97033]: pgmap v356: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 3 op/s; 155 B/s, 1 keys/s, 0 objects/s recovering 2022-09-07T21:39:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:57 smithi059 ceph-mon[97450]: pgmap v356: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 3 op/s; 155 B/s, 1 keys/s, 0 objects/s recovering 2022-09-07T21:39:59.311 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.307+0000 7f6318d43700 1 -- 172.21.15.47:0/1200903325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63140fef70 msgr2=0x7f63140ff340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:59.311 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.307+0000 7f6318d43700 1 --2- 172.21.15.47:0/1200903325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63140fef70 0x7f63140ff340 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6304009810 tx=0x7f6304009b20 comp rx=0 tx=0).stop 2022-09-07T21:39:59.311 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.307+0000 7f6318d43700 1 -- 172.21.15.47:0/1200903325 shutdown_connections 2022-09-07T21:39:59.311 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.307+0000 7f6318d43700 1 --2- 172.21.15.47:0/1200903325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6314108340 0x7f631410a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.307+0000 7f6318d43700 1 --2- 172.21.15.47:0/1200903325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63140fef70 0x7f63140ff340 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.307+0000 7f6318d43700 1 --2- 172.21.15.47:0/1200903325 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63140ff880 0x7f6314107e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.307+0000 7f6318d43700 1 -- 172.21.15.47:0/1200903325 >> 172.21.15.47:0/1200903325 conn(0x7f6314073ec0 msgr2=0x7f63140742c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.307+0000 7f6318d43700 1 -- 172.21.15.47:0/1200903325 shutdown_connections 2022-09-07T21:39:59.313 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.307+0000 7f6318d43700 1 -- 172.21.15.47:0/1200903325 wait complete. 2022-09-07T21:39:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.308+0000 7f6318d43700 1 Processor -- start 2022-09-07T21:39:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.309+0000 7f6318d43700 1 -- start start 2022-09-07T21:39:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.309+0000 7f6318d43700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63140fef70 0x7f6314195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.309+0000 7f6318d43700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63140ff880 0x7f6314196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.309+0000 7f6318d43700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6314108340 0x7f631419a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.309+0000 7f6318d43700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631419aef0 con 0x7f6314108340 2022-09-07T21:39:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.309+0000 7f6318d43700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631419b060 con 0x7f63140ff880 2022-09-07T21:39:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.309+0000 7f6318d43700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631419b220 con 0x7f63140fef70 2022-09-07T21:39:59.317 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.310+0000 7f6312d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6314108340 0x7f631419a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:59.317 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.310+0000 7f631259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63140fef70 0x7f6314195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:59.317 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.310+0000 7f6311d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63140ff880 0x7f6314196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:59.317 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.310+0000 7f631259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63140fef70 0x7f6314195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:33754/0 (socket says 172.21.15.47:33754) 2022-09-07T21:39:59.317 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.310+0000 7f631259c700 1 -- 172.21.15.47:0/1704680985 learned_addr learned my addr 172.21.15.47:0/1704680985 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:39:59.318 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.310+0000 7f6311d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63140ff880 0x7f6314196400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:51920/0 (socket says 172.21.15.47:51920) 2022-09-07T21:39:59.318 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.310+0000 7f6312d9d700 1 --2- 172.21.15.47:0/1704680985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6314108340 0x7f631419a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:35404/0 (socket says 172.21.15.47:35404) 2022-09-07T21:39:59.318 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.310+0000 7f631259c700 1 -- 172.21.15.47:0/1704680985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63140ff880 msgr2=0x7f6314196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:59.318 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.310+0000 7f631259c700 1 --2- 172.21.15.47:0/1704680985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63140ff880 0x7f6314196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:59.319 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.310+0000 7f631259c700 1 -- 172.21.15.47:0/1704680985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6314108340 msgr2=0x7f631419a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:59.319 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.310+0000 7f631259c700 1 --2- 172.21.15.47:0/1704680985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6314108340 0x7f631419a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:59.319 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.310+0000 7f631259c700 1 -- 172.21.15.47:0/1704680985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6304009470 con 0x7f63140fef70 2022-09-07T21:39:59.319 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.310+0000 7f6312d9d700 1 --2- 172.21.15.47:0/1704680985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6314108340 0x7f631419a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:39:59.320 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.310+0000 7f631259c700 1 --2- 172.21.15.47:0/1704680985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63140fef70 0x7f6314195ec0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f630400f690 tx=0x7f630400f6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:59.320 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.311+0000 7f63037fe700 1 -- 172.21.15.47:0/1704680985 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630401d070 con 0x7f63140fef70 2022-09-07T21:39:59.320 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.311+0000 7f6318d43700 1 -- 172.21.15.47:0/1704680985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631419b4a0 con 0x7f63140fef70 2022-09-07T21:39:59.320 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.311+0000 7f63037fe700 1 -- 172.21.15.47:0/1704680985 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6304003500 con 0x7f63140fef70 2022-09-07T21:39:59.321 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.311+0000 7f63037fe700 1 -- 172.21.15.47:0/1704680985 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6304004760 con 0x7f63140fef70 2022-09-07T21:39:59.321 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.311+0000 7f6318d43700 1 -- 172.21.15.47:0/1704680985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63141a2cf0 con 0x7f63140fef70 2022-09-07T21:39:59.322 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.312+0000 7f6318d43700 1 -- 172.21.15.47:0/1704680985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f631404d480 con 0x7f63140fef70 2022-09-07T21:39:59.322 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.314+0000 7f63037fe700 1 -- 172.21.15.47:0/1704680985 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f6304028030 con 0x7f63140fef70 2022-09-07T21:39:59.323 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.315+0000 7f63037fe700 1 --2- 172.21.15.47:0/1704680985 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f62fc0990d0 0x7f62fc09b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:39:59.323 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.316+0000 7f63037fe700 1 -- 172.21.15.47:0/1704680985 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f63040c37f0 con 0x7f63140fef70 2022-09-07T21:39:59.323 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.316+0000 7f6311d9b700 1 --2- 172.21.15.47:0/1704680985 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f62fc0990d0 0x7f62fc09b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:39:59.323 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.316+0000 7f6311d9b700 1 --2- 172.21.15.47:0/1704680985 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f62fc0990d0 0x7f62fc09b580 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f63141974d0 tx=0x7f6308005f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:39:59.323 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.317+0000 7f63037fe700 1 -- 172.21.15.47:0/1704680985 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f630408fb30 con 0x7f63140fef70 2022-09-07T21:39:59.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.567+0000 7f6318d43700 1 -- 172.21.15.47:0/1704680985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f6314197210 con 0x7f63140fef70 2022-09-07T21:39:59.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.568+0000 7f63037fe700 1 -- 172.21.15.47:0/1704680985 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6304026020 con 0x7f63140fef70 2022-09-07T21:39:59.572 INFO:teuthology.orchestra.run.smithi047.stdout:219043332146 2022-09-07T21:39:59.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.570+0000 7f6318d43700 1 -- 172.21.15.47:0/1704680985 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f62fc0990d0 msgr2=0x7f62fc09b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:59.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.570+0000 7f6318d43700 1 --2- 172.21.15.47:0/1704680985 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f62fc0990d0 0x7f62fc09b580 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f63141974d0 tx=0x7f6308005f70 comp rx=0 tx=0).stop 2022-09-07T21:39:59.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.570+0000 7f6318d43700 1 -- 172.21.15.47:0/1704680985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63140fef70 msgr2=0x7f6314195ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:39:59.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.570+0000 7f6318d43700 1 --2- 172.21.15.47:0/1704680985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63140fef70 0x7f6314195ec0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f630400f690 tx=0x7f630400f6c0 comp rx=0 tx=0).stop 2022-09-07T21:39:59.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.570+0000 7f6318d43700 1 -- 172.21.15.47:0/1704680985 shutdown_connections 2022-09-07T21:39:59.575 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.570+0000 7f6318d43700 1 --2- 172.21.15.47:0/1704680985 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f62fc0990d0 0x7f62fc09b580 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:59.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.570+0000 7f6318d43700 1 --2- 172.21.15.47:0/1704680985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6314108340 0x7f631419a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:59.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.570+0000 7f6318d43700 1 --2- 172.21.15.47:0/1704680985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63140fef70 0x7f6314195ec0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:59.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.570+0000 7f6318d43700 1 --2- 172.21.15.47:0/1704680985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63140ff880 0x7f6314196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:39:59.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.570+0000 7f6318d43700 1 -- 172.21.15.47:0/1704680985 >> 172.21.15.47:0/1704680985 conn(0x7f6314073ec0 msgr2=0x7f63140fcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:39:59.577 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.570+0000 7f6318d43700 1 -- 172.21.15.47:0/1704680985 shutdown_connections 2022-09-07T21:39:59.577 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:39:59.570+0000 7f6318d43700 1 -- 172.21.15.47:0/1704680985 wait complete. 2022-09-07T21:39:59.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:59 smithi047 ceph-mon[90087]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s; 141 B/s, 0 keys/s, 0 objects/s recovering 2022-09-07T21:39:59.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:39:59 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1704680985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-09-07T21:40:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:59 smithi059 ceph-mon[97450]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s; 141 B/s, 0 keys/s, 0 objects/s recovering 2022-09-07T21:40:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:39:59 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1704680985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-09-07T21:40:00.094 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332143 got 219043332146 for osd.7 2022-09-07T21:40:00.095 DEBUG:teuthology.parallel:result is None 2022-09-07T21:40:00.095 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2022-09-07T21:40:00.096 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph pg dump --format=json 2022-09-07T21:40:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:59 smithi055 ceph-mon[97033]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s; 141 B/s, 0 keys/s, 0 objects/s recovering 2022-09-07T21:40:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:39:59 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1704680985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-09-07T21:40:01.002 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:40:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:00 smithi059 ceph-mon[97450]: overall HEALTH_OK 2022-09-07T21:40:01.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:00 smithi047 ceph-mon[90087]: overall HEALTH_OK 2022-09-07T21:40:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:00 smithi055 ceph-mon[97033]: overall HEALTH_OK 2022-09-07T21:40:02.071 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:01 smithi047 ceph-mon[90087]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s; 127 B/s, 0 keys/s, 0 objects/s recovering 2022-09-07T21:40:02.071 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:01 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:40:02.071 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:01 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:40:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:01 smithi055 ceph-mon[97033]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s; 127 B/s, 0 keys/s, 0 objects/s recovering 2022-09-07T21:40:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:01 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:40:02.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:01 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:40:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:01 smithi059 ceph-mon[97450]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s; 127 B/s, 0 keys/s, 0 objects/s recovering 2022-09-07T21:40:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:01 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:40:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:01 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:40:02.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.706+0000 7f4bad72e700 1 -- 172.21.15.47:0/4063818363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ba8102d40 msgr2=0x7f4ba80666c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:02.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.706+0000 7f4bad72e700 1 --2- 172.21.15.47:0/4063818363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ba8102d40 0x7f4ba80666c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009810 tx=0x7f4b98009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.706+0000 7f4bad72e700 1 -- 172.21.15.47:0/4063818363 shutdown_connections 2022-09-07T21:40:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.706+0000 7f4bad72e700 1 --2- 172.21.15.47:0/4063818363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba81008d0 0x7f4ba8100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.706+0000 7f4bad72e700 1 --2- 172.21.15.47:0/4063818363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ba8102d40 0x7f4ba80666c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:02.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.706+0000 7f4bad72e700 1 --2- 172.21.15.47:0/4063818363 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba8066c00 0x7f4ba8067050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:02.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.706+0000 7f4bad72e700 1 -- 172.21.15.47:0/4063818363 >> 172.21.15.47:0/4063818363 conn(0x7f4ba8073cf0 msgr2=0x7f4ba80740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:02.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.707+0000 7f4bad72e700 1 -- 172.21.15.47:0/4063818363 shutdown_connections 2022-09-07T21:40:02.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.707+0000 7f4bad72e700 1 -- 172.21.15.47:0/4063818363 wait complete. 2022-09-07T21:40:02.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.708+0000 7f4bad72e700 1 Processor -- start 2022-09-07T21:40:02.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.708+0000 7f4bad72e700 1 -- start start 2022-09-07T21:40:02.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.708+0000 7f4bad72e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ba8066c00 0x7f4ba819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:02.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.708+0000 7f4bad72e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba81008d0 0x7f4ba819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:02.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.708+0000 7f4bad72e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba8102d40 0x7f4ba819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:02.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.708+0000 7f4bad72e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba819f270 con 0x7f4ba81008d0 2022-09-07T21:40:02.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.708+0000 7f4bad72e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba819f3e0 con 0x7f4ba8066c00 2022-09-07T21:40:02.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.708+0000 7f4bad72e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba819f5a0 con 0x7f4ba8102d40 2022-09-07T21:40:02.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.709+0000 7f4ba6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ba8066c00 0x7f4ba819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:02.717 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.709+0000 7f4ba67fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba81008d0 0x7f4ba819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:02.717 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.709+0000 7f4ba6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ba8066c00 0x7f4ba819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:51942/0 (socket says 172.21.15.47:51942) 2022-09-07T21:40:02.717 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.709+0000 7f4ba6ffd700 1 -- 172.21.15.47:0/2587822928 learned_addr learned my addr 172.21.15.47:0/2587822928 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:02.717 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.709+0000 7f4ba77fe700 1 --2- 172.21.15.47:0/2587822928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba8102d40 0x7f4ba819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:02.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.709+0000 7f4ba67fc700 1 -- 172.21.15.47:0/2587822928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba8102d40 msgr2=0x7f4ba819ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:02.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.709+0000 7f4ba67fc700 1 --2- 172.21.15.47:0/2587822928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba8102d40 0x7f4ba819ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:02.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.709+0000 7f4ba67fc700 1 -- 172.21.15.47:0/2587822928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ba8066c00 msgr2=0x7f4ba819a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:02.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.709+0000 7f4ba67fc700 1 --2- 172.21.15.47:0/2587822928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ba8066c00 0x7f4ba819a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:02.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.709+0000 7f4ba67fc700 1 -- 172.21.15.47:0/2587822928 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b98009470 con 0x7f4ba81008d0 2022-09-07T21:40:02.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.709+0000 7f4ba6ffd700 1 --2- 172.21.15.47:0/2587822928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ba8066c00 0x7f4ba819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:02.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.710+0000 7f4ba67fc700 1 --2- 172.21.15.47:0/2587822928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba81008d0 0x7f4ba819a780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00e6c0 tx=0x7f4b9c00e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:02.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.710+0000 7f4b8ffff700 1 -- 172.21.15.47:0/2587822928 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c010900 con 0x7f4ba81008d0 2022-09-07T21:40:02.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.710+0000 7f4bad72e700 1 -- 172.21.15.47:0/2587822928 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba819f880 con 0x7f4ba81008d0 2022-09-07T21:40:02.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.710+0000 7f4b8ffff700 1 -- 172.21.15.47:0/2587822928 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4b9c018420 con 0x7f4ba81008d0 2022-09-07T21:40:02.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.710+0000 7f4b8ffff700 1 -- 172.21.15.47:0/2587822928 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c017610 con 0x7f4ba81008d0 2022-09-07T21:40:02.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.710+0000 7f4bad72e700 1 -- 172.21.15.47:0/2587822928 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba81a70c0 con 0x7f4ba81008d0 2022-09-07T21:40:02.721 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.711+0000 7f4bad72e700 1 -- 172.21.15.47:0/2587822928 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba8064ea0 con 0x7f4ba81008d0 2022-09-07T21:40:02.722 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.714+0000 7f4b8ffff700 1 -- 172.21.15.47:0/2587822928 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f4b9c00c040 con 0x7f4ba81008d0 2022-09-07T21:40:02.722 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.715+0000 7f4b8ffff700 1 --2- 172.21.15.47:0/2587822928 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4b900990d0 0x7f4b9009b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:02.722 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.715+0000 7f4b8ffff700 1 -- 172.21.15.47:0/2587822928 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f4b9c014070 con 0x7f4ba81008d0 2022-09-07T21:40:02.722 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.716+0000 7f4ba6ffd700 1 --2- 172.21.15.47:0/2587822928 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4b900990d0 0x7f4b9009b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:02.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.716+0000 7f4ba6ffd700 1 --2- 172.21.15.47:0/2587822928 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4b900990d0 0x7f4b9009b580 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009760 tx=0x7f4b9800b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:02.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.720+0000 7f4b8ffff700 1 -- 172.21.15.47:0/2587822928 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f4b9c04fc80 con 0x7f4ba81008d0 2022-09-07T21:40:02.941 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.938+0000 7f4bad72e700 1 -- 172.21.15.47:0/2587822928 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ba80fffc0 con 0x7f4b900990d0 2022-09-07T21:40:02.955 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.952+0000 7f4b8ffff700 1 -- 172.21.15.47:0/2587822928 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426687 (secure 0 0 0) 0x7f4ba80fffc0 con 0x7f4b900990d0 2022-09-07T21:40:02.965 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:40:02.969 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.959+0000 7f4bad72e700 1 -- 172.21.15.47:0/2587822928 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4b900990d0 msgr2=0x7f4b9009b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:02.969 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.959+0000 7f4bad72e700 1 --2- 172.21.15.47:0/2587822928 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4b900990d0 0x7f4b9009b580 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009760 tx=0x7f4b9800b480 comp rx=0 tx=0).stop 2022-09-07T21:40:02.969 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.959+0000 7f4bad72e700 1 -- 172.21.15.47:0/2587822928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba81008d0 msgr2=0x7f4ba819a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:02.969 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.959+0000 7f4bad72e700 1 --2- 172.21.15.47:0/2587822928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba81008d0 0x7f4ba819a780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00e6c0 tx=0x7f4b9c00e9d0 comp rx=0 tx=0).stop 2022-09-07T21:40:02.970 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.959+0000 7f4bad72e700 1 -- 172.21.15.47:0/2587822928 shutdown_connections 2022-09-07T21:40:02.970 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.959+0000 7f4bad72e700 1 --2- 172.21.15.47:0/2587822928 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4b900990d0 0x7f4b9009b580 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:02.970 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.959+0000 7f4bad72e700 1 --2- 172.21.15.47:0/2587822928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba8102d40 0x7f4ba819ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:02.970 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.959+0000 7f4bad72e700 1 --2- 172.21.15.47:0/2587822928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ba8066c00 0x7f4ba819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:02.971 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.959+0000 7f4bad72e700 1 --2- 172.21.15.47:0/2587822928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba81008d0 0x7f4ba819a780 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:02.971 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.959+0000 7f4bad72e700 1 -- 172.21.15.47:0/2587822928 >> 172.21.15.47:0/2587822928 conn(0x7f4ba8073cf0 msgr2=0x7f4ba80ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:02.971 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.960+0000 7f4bad72e700 1 -- 172.21.15.47:0/2587822928 shutdown_connections 2022-09-07T21:40:02.971 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:02.960+0000 7f4bad72e700 1 -- 172.21.15.47:0/2587822928 wait complete. 2022-09-07T21:40:02.972 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2022-09-07T21:40:03.487 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2022-09-07T21:40:02.610861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655731,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":453,"num_read_kb":358,"num_write":341,"num_write_kb":3052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1182269,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80852,"kb_used_data":12148,"kb_used_omap":0,"kb_used_meta":68608,"kb_avail":1124860972,"statfs":{"total":1151940427776,"available":1151857635328,"internally_reserved":0,"allocated":12439552,"data_stored":7930080,"data_compressed":321765,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":0,"internal_metadata":70254592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":33,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":967,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003635"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.792064+0000","last_change":"2022-09-07T21:38:40.576826+0000","last_active":"2022-09-07T21:39:46.792064+0000","last_peered":"2022-09-07T21:39:46.792064+0000","last_clean":"2022-09-07T21:39:46.792064+0000","last_became_active":"2022-09-07T21:38:40.576429+0000","last_became_peered":"2022-09-07T21:38:40.576429+0000","last_unstale":"2022-09-07T21:39:46.792064+0000","last_undegraded":"2022-09-07T21:39:46.792064+0000","last_fullsized":"2022-09-07T21:39:46.792064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:40:45.777536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784539+0000","last_change":"2022-09-07T21:38:40.562924+0000","last_active":"2022-09-07T21:39:46.784539+0000","last_peered":"2022-09-07T21:39:46.784539+0000","last_clean":"2022-09-07T21:39:46.784539+0000","last_became_active":"2022-09-07T21:38:40.562509+0000","last_became_peered":"2022-09-07T21:38:40.562509+0000","last_unstale":"2022-09-07T21:39:46.784539+0000","last_undegraded":"2022-09-07T21:39:46.784539+0000","last_fullsized":"2022-09-07T21:39:46.784539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:27:16.781072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784192+0000","last_change":"2022-09-07T21:38:40.562123+0000","last_active":"2022-09-07T21:39:46.784192+0000","last_peered":"2022-09-07T21:39:46.784192+0000","last_clean":"2022-09-07T21:39:46.784192+0000","last_became_active":"2022-09-07T21:38:40.561632+0000","last_became_peered":"2022-09-07T21:38:40.561632+0000","last_unstale":"2022-09-07T21:39:46.784192+0000","last_undegraded":"2022-09-07T21:39:46.784192+0000","last_fullsized":"2022-09-07T21:39:46.784192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:28:10.763115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793271+0000","last_change":"2022-09-07T21:38:40.560968+0000","last_active":"2022-09-07T21:39:46.793271+0000","last_peered":"2022-09-07T21:39:46.793271+0000","last_clean":"2022-09-07T21:39:46.793271+0000","last_became_active":"2022-09-07T21:38:40.560685+0000","last_became_peered":"2022-09-07T21:38:40.560685+0000","last_unstale":"2022-09-07T21:39:46.793271+0000","last_undegraded":"2022-09-07T21:39:46.793271+0000","last_fullsized":"2022-09-07T21:39:46.793271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:49:41.627456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.786853+0000","last_change":"2022-09-07T21:38:40.572935+0000","last_active":"2022-09-07T21:39:46.786853+0000","last_peered":"2022-09-07T21:39:46.786853+0000","last_clean":"2022-09-07T21:39:46.786853+0000","last_became_active":"2022-09-07T21:38:40.572605+0000","last_became_peered":"2022-09-07T21:38:40.572605+0000","last_unstale":"2022-09-07T21:39:46.786853+0000","last_undegraded":"2022-09-07T21:39:46.786853+0000","last_fullsized":"2022-09-07T21:39:46.786853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:55:47.267936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790461+0000","last_change":"2022-09-07T21:38:40.577737+0000","last_active":"2022-09-07T21:39:46.790461+0000","last_peered":"2022-09-07T21:39:46.790461+0000","last_clean":"2022-09-07T21:39:46.790461+0000","last_became_active":"2022-09-07T21:38:40.577425+0000","last_became_peered":"2022-09-07T21:38:40.577425+0000","last_unstale":"2022-09-07T21:39:46.790461+0000","last_undegraded":"2022-09-07T21:39:46.790461+0000","last_fullsized":"2022-09-07T21:39:46.790461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:46:32.369263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791725+0000","last_change":"2022-09-07T21:38:40.577255+0000","last_active":"2022-09-07T21:39:46.791725+0000","last_peered":"2022-09-07T21:39:46.791725+0000","last_clean":"2022-09-07T21:39:46.791725+0000","last_became_active":"2022-09-07T21:38:40.576750+0000","last_became_peered":"2022-09-07T21:38:40.576750+0000","last_unstale":"2022-09-07T21:39:46.791725+0000","last_undegraded":"2022-09-07T21:39:46.791725+0000","last_fullsized":"2022-09-07T21:39:46.791725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:03:19.663251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784380+0000","last_change":"2022-09-07T21:38:40.577599+0000","last_active":"2022-09-07T21:39:46.784380+0000","last_peered":"2022-09-07T21:39:46.784380+0000","last_clean":"2022-09-07T21:39:46.784380+0000","last_became_active":"2022-09-07T21:38:40.577239+0000","last_became_peered":"2022-09-07T21:38:40.577239+0000","last_unstale":"2022-09-07T21:39:46.784380+0000","last_undegraded":"2022-09-07T21:39:46.784380+0000","last_fullsized":"2022-09-07T21:39:46.784380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:39:23.857698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.794199+0000","last_change":"2022-09-07T21:38:40.559102+0000","last_active":"2022-09-07T21:39:46.794199+0000","last_peered":"2022-09-07T21:39:46.794199+0000","last_clean":"2022-09-07T21:39:46.794199+0000","last_became_active":"2022-09-07T21:38:40.558907+0000","last_became_peered":"2022-09-07T21:38:40.558907+0000","last_unstale":"2022-09-07T21:39:46.794199+0000","last_undegraded":"2022-09-07T21:39:46.794199+0000","last_fullsized":"2022-09-07T21:39:46.794199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:08:03.644593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791478+0000","last_change":"2022-09-07T21:38:40.566427+0000","last_active":"2022-09-07T21:39:46.791478+0000","last_peered":"2022-09-07T21:39:46.791478+0000","last_clean":"2022-09-07T21:39:46.791478+0000","last_became_active":"2022-09-07T21:38:40.566068+0000","last_became_peered":"2022-09-07T21:38:40.566068+0000","last_unstale":"2022-09-07T21:39:46.791478+0000","last_undegraded":"2022-09-07T21:39:46.791478+0000","last_fullsized":"2022-09-07T21:39:46.791478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:40:01.825654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791305+0000","last_change":"2022-09-07T21:38:40.568455+0000","last_active":"2022-09-07T21:39:46.791305+0000","last_peered":"2022-09-07T21:39:46.791305+0000","last_clean":"2022-09-07T21:39:46.791305+0000","last_became_active":"2022-09-07T21:38:40.567941+0000","last_became_peered":"2022-09-07T21:38:40.567941+0000","last_unstale":"2022-09-07T21:39:46.791305+0000","last_undegraded":"2022-09-07T21:39:46.791305+0000","last_fullsized":"2022-09-07T21:39:46.791305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:49:16.624335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793197+0000","last_change":"2022-09-07T21:38:40.563200+0000","last_active":"2022-09-07T21:39:46.793197+0000","last_peered":"2022-09-07T21:39:46.793197+0000","last_clean":"2022-09-07T21:39:46.793197+0000","last_became_active":"2022-09-07T21:38:40.562901+0000","last_became_peered":"2022-09-07T21:38:40.562901+0000","last_unstale":"2022-09-07T21:39:46.793197+0000","last_undegraded":"2022-09-07T21:39:46.793197+0000","last_fullsized":"2022-09-07T21:39:46.793197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:49:26.745940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788724+0000","last_change":"2022-09-07T21:38:40.570403+0000","last_active":"2022-09-07T21:39:46.788724+0000","last_peered":"2022-09-07T21:39:46.788724+0000","last_clean":"2022-09-07T21:39:46.788724+0000","last_became_active":"2022-09-07T21:38:40.567326+0000","last_became_peered":"2022-09-07T21:38:40.567326+0000","last_unstale":"2022-09-07T21:39:46.788724+0000","last_undegraded":"2022-09-07T21:39:46.788724+0000","last_fullsized":"2022-09-07T21:39:46.788724+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:19:31.437519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.783735+0000","last_change":"2022-09-07T21:38:40.562990+0000","last_active":"2022-09-07T21:39:46.783735+0000","last_peered":"2022-09-07T21:39:46.783735+0000","last_clean":"2022-09-07T21:39:46.783735+0000","last_became_active":"2022-09-07T21:38:40.562651+0000","last_became_peered":"2022-09-07T21:38:40.562651+0000","last_unstale":"2022-09-07T21:39:46.783735+0000","last_undegraded":"2022-09-07T21:39:46.783735+0000","last_fullsized":"2022-09-07T21:39:46.783735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:49:07.935459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.783491+0000","last_change":"2022-09-07T21:38:40.570849+0000","last_active":"2022-09-07T21:39:46.783491+0000","last_peered":"2022-09-07T21:39:46.783491+0000","last_clean":"2022-09-07T21:39:46.783491+0000","last_became_active":"2022-09-07T21:38:40.570543+0000","last_became_peered":"2022-09-07T21:38:40.570543+0000","last_unstale":"2022-09-07T21:39:46.783491+0000","last_undegraded":"2022-09-07T21:39:46.783491+0000","last_fullsized":"2022-09-07T21:39:46.783491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:33:30.564522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788070+0000","last_change":"2022-09-07T21:38:40.568348+0000","last_active":"2022-09-07T21:39:46.788070+0000","last_peered":"2022-09-07T21:39:46.788070+0000","last_clean":"2022-09-07T21:39:46.788070+0000","last_became_active":"2022-09-07T21:38:40.568031+0000","last_became_peered":"2022-09-07T21:38:40.568031+0000","last_unstale":"2022-09-07T21:39:46.788070+0000","last_undegraded":"2022-09-07T21:39:46.788070+0000","last_fullsized":"2022-09-07T21:39:46.788070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:24:21.978277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791902+0000","last_change":"2022-09-07T21:38:40.577289+0000","last_active":"2022-09-07T21:39:46.791902+0000","last_peered":"2022-09-07T21:39:46.791902+0000","last_clean":"2022-09-07T21:39:46.791902+0000","last_became_active":"2022-09-07T21:38:40.576927+0000","last_became_peered":"2022-09-07T21:38:40.576927+0000","last_unstale":"2022-09-07T21:39:46.791902+0000","last_undegraded":"2022-09-07T21:39:46.791902+0000","last_fullsized":"2022-09-07T21:39:46.791902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:02:53.553226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790196+0000","last_change":"2022-09-07T21:38:40.575886+0000","last_active":"2022-09-07T21:39:46.790196+0000","last_peered":"2022-09-07T21:39:46.790196+0000","last_clean":"2022-09-07T21:39:46.790196+0000","last_became_active":"2022-09-07T21:38:40.575463+0000","last_became_peered":"2022-09-07T21:38:40.575463+0000","last_unstale":"2022-09-07T21:39:46.790196+0000","last_undegraded":"2022-09-07T21:39:46.790196+0000","last_fullsized":"2022-09-07T21:39:46.790196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:10:20.091899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793516+0000","last_change":"2022-09-07T21:38:40.572579+0000","last_active":"2022-09-07T21:39:46.793516+0000","last_peered":"2022-09-07T21:39:46.793516+0000","last_clean":"2022-09-07T21:39:46.793516+0000","last_became_active":"2022-09-07T21:38:40.572302+0000","last_became_peered":"2022-09-07T21:38:40.572302+0000","last_unstale":"2022-09-07T21:39:46.793516+0000","last_undegraded":"2022-09-07T21:39:46.793516+0000","last_fullsized":"2022-09-07T21:39:46.793516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:41:19.938512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.792583+0000","last_change":"2022-09-07T21:38:44.996073+0000","last_active":"2022-09-07T21:39:46.792583+0000","last_peered":"2022-09-07T21:39:46.792583+0000","last_clean":"2022-09-07T21:39:46.792583+0000","last_became_active":"2022-09-07T21:38:44.995571+0000","last_became_peered":"2022-09-07T21:38:44.995571+0000","last_unstale":"2022-09-07T21:39:46.792583+0000","last_undegraded":"2022-09-07T21:39:46.792583+0000","last_fullsized":"2022-09-07T21:39:46.792583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:41:58.745823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:48.148181+0000","last_change":"2022-09-07T21:39:48.148181+0000","last_active":"2022-09-07T21:39:48.148181+0000","last_peered":"2022-09-07T21:39:48.148181+0000","last_clean":"2022-09-07T21:39:48.148181+0000","last_became_active":"2022-09-07T21:39:47.794055+0000","last_became_peered":"2022-09-07T21:39:47.794055+0000","last_unstale":"2022-09-07T21:39:48.148181+0000","last_undegraded":"2022-09-07T21:39:48.148181+0000","last_fullsized":"2022-09-07T21:39:48.148181+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:43:35.756474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":566,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.785552+0000","last_change":"2022-09-07T21:38:40.578853+0000","last_active":"2022-09-07T21:39:46.785552+0000","last_peered":"2022-09-07T21:39:46.785552+0000","last_clean":"2022-09-07T21:39:46.785552+0000","last_became_active":"2022-09-07T21:38:40.578229+0000","last_became_peered":"2022-09-07T21:38:40.578229+0000","last_unstale":"2022-09-07T21:39:46.785552+0000","last_undegraded":"2022-09-07T21:39:46.785552+0000","last_fullsized":"2022-09-07T21:39:46.785552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:39:28.259219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.792264+0000","last_change":"2022-09-07T21:38:44.995929+0000","last_active":"2022-09-07T21:39:46.792264+0000","last_peered":"2022-09-07T21:39:46.792264+0000","last_clean":"2022-09-07T21:39:46.792264+0000","last_became_active":"2022-09-07T21:38:44.995265+0000","last_became_peered":"2022-09-07T21:38:44.995265+0000","last_unstale":"2022-09-07T21:39:46.792264+0000","last_undegraded":"2022-09-07T21:39:46.792264+0000","last_fullsized":"2022-09-07T21:39:46.792264+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:11:30.103628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.785590+0000","last_change":"2022-09-07T21:38:39.971598+0000","last_active":"2022-09-07T21:39:46.785590+0000","last_peered":"2022-09-07T21:39:46.785590+0000","last_clean":"2022-09-07T21:39:46.785590+0000","last_became_active":"2022-09-07T21:38:39.971079+0000","last_became_peered":"2022-09-07T21:38:39.971079+0000","last_unstale":"2022-09-07T21:39:46.785590+0000","last_undegraded":"2022-09-07T21:39:46.785590+0000","last_fullsized":"2022-09-07T21:39:46.785590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:01:22.960635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.785131+0000","last_change":"2022-09-07T21:38:40.577589+0000","last_active":"2022-09-07T21:39:46.785131+0000","last_peered":"2022-09-07T21:39:46.785131+0000","last_clean":"2022-09-07T21:39:46.785131+0000","last_became_active":"2022-09-07T21:38:40.577192+0000","last_became_peered":"2022-09-07T21:38:40.577192+0000","last_unstale":"2022-09-07T21:39:46.785131+0000","last_undegraded":"2022-09-07T21:39:46.785131+0000","last_fullsized":"2022-09-07T21:39:46.785131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:41:03.746467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.792888+0000","last_change":"2022-09-07T21:38:44.997384+0000","last_active":"2022-09-07T21:39:46.792888+0000","last_peered":"2022-09-07T21:39:46.792888+0000","last_clean":"2022-09-07T21:39:46.792888+0000","last_became_active":"2022-09-07T21:38:44.996150+0000","last_became_peered":"2022-09-07T21:38:44.996150+0000","last_unstale":"2022-09-07T21:39:46.792888+0000","last_undegraded":"2022-09-07T21:39:46.792888+0000","last_fullsized":"2022-09-07T21:39:46.792888+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:30:05.633718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:48.140438+0000","last_change":"2022-09-07T21:39:48.139937+0000","last_active":"2022-09-07T21:39:48.140438+0000","last_peered":"2022-09-07T21:39:48.140438+0000","last_clean":"2022-09-07T21:39:48.140438+0000","last_became_active":"2022-09-07T21:39:47.792095+0000","last_became_peered":"2022-09-07T21:39:47.792095+0000","last_unstale":"2022-09-07T21:39:48.140438+0000","last_undegraded":"2022-09-07T21:39:48.140438+0000","last_fullsized":"2022-09-07T21:39:48.140438+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:33:13.259619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789240+0000","last_change":"2022-09-07T21:38:40.572694+0000","last_active":"2022-09-07T21:39:46.789240+0000","last_peered":"2022-09-07T21:39:46.789240+0000","last_clean":"2022-09-07T21:39:46.789240+0000","last_became_active":"2022-09-07T21:38:40.572407+0000","last_became_peered":"2022-09-07T21:38:40.572407+0000","last_unstale":"2022-09-07T21:39:46.789240+0000","last_undegraded":"2022-09-07T21:39:46.789240+0000","last_fullsized":"2022-09-07T21:39:46.789240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:40:17.238549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.792057+0000","last_change":"2022-09-07T21:38:44.988445+0000","last_active":"2022-09-07T21:39:46.792057+0000","last_peered":"2022-09-07T21:39:46.792057+0000","last_clean":"2022-09-07T21:39:46.792057+0000","last_became_active":"2022-09-07T21:38:44.987966+0000","last_became_peered":"2022-09-07T21:38:44.987966+0000","last_unstale":"2022-09-07T21:39:46.792057+0000","last_undegraded":"2022-09-07T21:39:46.792057+0000","last_fullsized":"2022-09-07T21:39:46.792057+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:36:16.050494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.787147+0000","last_change":"2022-09-07T21:38:39.963456+0000","last_active":"2022-09-07T21:39:46.787147+0000","last_peered":"2022-09-07T21:39:46.787147+0000","last_clean":"2022-09-07T21:39:46.787147+0000","last_became_active":"2022-09-07T21:38:39.963059+0000","last_became_peered":"2022-09-07T21:38:39.963059+0000","last_unstale":"2022-09-07T21:39:46.787147+0000","last_undegraded":"2022-09-07T21:39:46.787147+0000","last_fullsized":"2022-09-07T21:39:46.787147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:51:09.061725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790283+0000","last_change":"2022-09-07T21:38:40.559585+0000","last_active":"2022-09-07T21:39:46.790283+0000","last_peered":"2022-09-07T21:39:46.790283+0000","last_clean":"2022-09-07T21:39:46.790283+0000","last_became_active":"2022-09-07T21:38:40.559307+0000","last_became_peered":"2022-09-07T21:38:40.559307+0000","last_unstale":"2022-09-07T21:39:46.790283+0000","last_undegraded":"2022-09-07T21:39:46.790283+0000","last_fullsized":"2022-09-07T21:39:46.790283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:20:32.332429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793369+0000","last_change":"2022-09-07T21:38:44.985596+0000","last_active":"2022-09-07T21:39:46.793369+0000","last_peered":"2022-09-07T21:39:46.793369+0000","last_clean":"2022-09-07T21:39:46.793369+0000","last_became_active":"2022-09-07T21:38:44.985112+0000","last_became_peered":"2022-09-07T21:38:44.985112+0000","last_unstale":"2022-09-07T21:39:46.793369+0000","last_undegraded":"2022-09-07T21:39:46.793369+0000","last_fullsized":"2022-09-07T21:39:46.793369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:24:06.456838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:55.809350+0000","last_change":"2022-09-07T21:39:48.142234+0000","last_active":"2022-09-07T21:39:55.809350+0000","last_peered":"2022-09-07T21:39:55.809350+0000","last_clean":"2022-09-07T21:39:55.809350+0000","last_became_active":"2022-09-07T21:39:47.795907+0000","last_became_peered":"2022-09-07T21:39:47.795907+0000","last_unstale":"2022-09-07T21:39:55.809350+0000","last_undegraded":"2022-09-07T21:39:55.809350+0000","last_fullsized":"2022-09-07T21:39:55.809350+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:42:26.284246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.786886+0000","last_change":"2022-09-07T21:38:40.577546+0000","last_active":"2022-09-07T21:39:46.786886+0000","last_peered":"2022-09-07T21:39:46.786886+0000","last_clean":"2022-09-07T21:39:46.786886+0000","last_became_active":"2022-09-07T21:38:40.577229+0000","last_became_peered":"2022-09-07T21:38:40.577229+0000","last_unstale":"2022-09-07T21:39:46.786886+0000","last_undegraded":"2022-09-07T21:39:46.786886+0000","last_fullsized":"2022-09-07T21:39:46.786886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:57:36.947488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793428+0000","last_change":"2022-09-07T21:38:44.993976+0000","last_active":"2022-09-07T21:39:46.793428+0000","last_peered":"2022-09-07T21:39:46.793428+0000","last_clean":"2022-09-07T21:39:46.793428+0000","last_became_active":"2022-09-07T21:38:44.993554+0000","last_became_peered":"2022-09-07T21:38:44.993554+0000","last_unstale":"2022-09-07T21:39:46.793428+0000","last_undegraded":"2022-09-07T21:39:46.793428+0000","last_fullsized":"2022-09-07T21:39:46.793428+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:21:13.466677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790366+0000","last_change":"2022-09-07T21:38:39.970361+0000","last_active":"2022-09-07T21:39:46.790366+0000","last_peered":"2022-09-07T21:39:46.790366+0000","last_clean":"2022-09-07T21:39:46.790366+0000","last_became_active":"2022-09-07T21:38:39.969420+0000","last_became_peered":"2022-09-07T21:38:39.969420+0000","last_unstale":"2022-09-07T21:39:46.790366+0000","last_undegraded":"2022-09-07T21:39:46.790366+0000","last_fullsized":"2022-09-07T21:39:46.790366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:53:46.903275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791431+0000","last_change":"2022-09-07T21:38:40.567273+0000","last_active":"2022-09-07T21:39:46.791431+0000","last_peered":"2022-09-07T21:39:46.791431+0000","last_clean":"2022-09-07T21:39:46.791431+0000","last_became_active":"2022-09-07T21:38:40.566942+0000","last_became_peered":"2022-09-07T21:38:40.566942+0000","last_unstale":"2022-09-07T21:39:46.791431+0000","last_undegraded":"2022-09-07T21:39:46.791431+0000","last_fullsized":"2022-09-07T21:39:46.791431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:29:38.009443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791804+0000","last_change":"2022-09-07T21:38:44.998836+0000","last_active":"2022-09-07T21:39:46.791804+0000","last_peered":"2022-09-07T21:39:46.791804+0000","last_clean":"2022-09-07T21:39:46.791804+0000","last_became_active":"2022-09-07T21:38:44.998461+0000","last_became_peered":"2022-09-07T21:38:44.998461+0000","last_unstale":"2022-09-07T21:39:46.791804+0000","last_undegraded":"2022-09-07T21:39:46.791804+0000","last_fullsized":"2022-09-07T21:39:46.791804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:37:37.203792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788074+0000","last_change":"2022-09-07T21:38:39.970746+0000","last_active":"2022-09-07T21:39:46.788074+0000","last_peered":"2022-09-07T21:39:46.788074+0000","last_clean":"2022-09-07T21:39:46.788074+0000","last_became_active":"2022-09-07T21:38:39.970438+0000","last_became_peered":"2022-09-07T21:38:39.970438+0000","last_unstale":"2022-09-07T21:39:46.788074+0000","last_undegraded":"2022-09-07T21:39:46.788074+0000","last_fullsized":"2022-09-07T21:39:46.788074+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:51:07.849231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784303+0000","last_change":"2022-09-07T21:38:40.576245+0000","last_active":"2022-09-07T21:39:46.784303+0000","last_peered":"2022-09-07T21:39:46.784303+0000","last_clean":"2022-09-07T21:39:46.784303+0000","last_became_active":"2022-09-07T21:38:40.575916+0000","last_became_peered":"2022-09-07T21:38:40.575916+0000","last_unstale":"2022-09-07T21:39:46.784303+0000","last_undegraded":"2022-09-07T21:39:46.784303+0000","last_fullsized":"2022-09-07T21:39:46.784303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:35:52.025507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784407+0000","last_change":"2022-09-07T21:38:44.991494+0000","last_active":"2022-09-07T21:39:46.784407+0000","last_peered":"2022-09-07T21:39:46.784407+0000","last_clean":"2022-09-07T21:39:46.784407+0000","last_became_active":"2022-09-07T21:38:44.988446+0000","last_became_peered":"2022-09-07T21:38:44.988446+0000","last_unstale":"2022-09-07T21:39:46.784407+0000","last_undegraded":"2022-09-07T21:39:46.784407+0000","last_fullsized":"2022-09-07T21:39:46.784407+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:22:52.937965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:47.795341+0000","last_change":"2022-09-07T21:39:47.794537+0000","last_active":"2022-09-07T21:39:47.795341+0000","last_peered":"2022-09-07T21:39:47.795341+0000","last_clean":"2022-09-07T21:39:47.795341+0000","last_became_active":"2022-09-07T21:39:47.794107+0000","last_became_peered":"2022-09-07T21:39:47.794107+0000","last_unstale":"2022-09-07T21:39:47.795341+0000","last_undegraded":"2022-09-07T21:39:47.795341+0000","last_fullsized":"2022-09-07T21:39:47.795341+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:44:58.430301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789269+0000","last_change":"2022-09-07T21:38:40.590979+0000","last_active":"2022-09-07T21:39:46.789269+0000","last_peered":"2022-09-07T21:39:46.789269+0000","last_clean":"2022-09-07T21:39:46.789269+0000","last_became_active":"2022-09-07T21:38:40.590024+0000","last_became_peered":"2022-09-07T21:38:40.590024+0000","last_unstale":"2022-09-07T21:39:46.789269+0000","last_undegraded":"2022-09-07T21:39:46.789269+0000","last_fullsized":"2022-09-07T21:39:46.789269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:58:11.726364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784894+0000","last_change":"2022-09-07T21:38:44.988252+0000","last_active":"2022-09-07T21:39:46.784894+0000","last_peered":"2022-09-07T21:39:46.784894+0000","last_clean":"2022-09-07T21:39:46.784894+0000","last_became_active":"2022-09-07T21:38:44.987851+0000","last_became_peered":"2022-09-07T21:38:44.987851+0000","last_unstale":"2022-09-07T21:39:46.784894+0000","last_undegraded":"2022-09-07T21:39:46.784894+0000","last_fullsized":"2022-09-07T21:39:46.784894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:53:42.207358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:55.808678+0000","last_change":"2022-09-07T21:38:39.969970+0000","last_active":"2022-09-07T21:39:55.808678+0000","last_peered":"2022-09-07T21:39:55.808678+0000","last_clean":"2022-09-07T21:39:55.808678+0000","last_became_active":"2022-09-07T21:38:39.969627+0000","last_became_peered":"2022-09-07T21:38:39.969627+0000","last_unstale":"2022-09-07T21:39:55.808678+0000","last_undegraded":"2022-09-07T21:39:55.808678+0000","last_fullsized":"2022-09-07T21:39:55.808678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:14:11.990138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":50,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.792576+0000","last_change":"2022-09-07T21:38:40.577175+0000","last_active":"2022-09-07T21:39:46.792576+0000","last_peered":"2022-09-07T21:39:46.792576+0000","last_clean":"2022-09-07T21:39:46.792576+0000","last_became_active":"2022-09-07T21:38:40.576873+0000","last_became_peered":"2022-09-07T21:38:40.576873+0000","last_unstale":"2022-09-07T21:39:46.792576+0000","last_undegraded":"2022-09-07T21:39:46.792576+0000","last_fullsized":"2022-09-07T21:39:46.792576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:25:17.911241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789266+0000","last_change":"2022-09-07T21:38:44.984456+0000","last_active":"2022-09-07T21:39:46.789266+0000","last_peered":"2022-09-07T21:39:46.789266+0000","last_clean":"2022-09-07T21:39:46.789266+0000","last_became_active":"2022-09-07T21:38:44.984070+0000","last_became_peered":"2022-09-07T21:38:44.984070+0000","last_unstale":"2022-09-07T21:39:46.789266+0000","last_undegraded":"2022-09-07T21:39:46.789266+0000","last_fullsized":"2022-09-07T21:39:46.789266+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:36:14.150189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:47.881379+0000","last_change":"2022-09-07T21:39:47.881379+0000","last_active":"2022-09-07T21:39:47.881379+0000","last_peered":"2022-09-07T21:39:47.881379+0000","last_clean":"2022-09-07T21:39:47.881379+0000","last_became_active":"2022-09-07T21:39:47.792120+0000","last_became_peered":"2022-09-07T21:39:47.792120+0000","last_unstale":"2022-09-07T21:39:47.881379+0000","last_undegraded":"2022-09-07T21:39:47.881379+0000","last_fullsized":"2022-09-07T21:39:47.881379+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:42:47.731825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784902+0000","last_change":"2022-09-07T21:38:40.571594+0000","last_active":"2022-09-07T21:39:46.784902+0000","last_peered":"2022-09-07T21:39:46.784902+0000","last_clean":"2022-09-07T21:39:46.784902+0000","last_became_active":"2022-09-07T21:38:40.571188+0000","last_became_peered":"2022-09-07T21:38:40.571188+0000","last_unstale":"2022-09-07T21:39:46.784902+0000","last_undegraded":"2022-09-07T21:39:46.784902+0000","last_fullsized":"2022-09-07T21:39:46.784902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:25:49.474602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:47.794303+0000","last_change":"2022-09-07T21:39:47.794303+0000","last_active":"2022-09-07T21:39:47.794303+0000","last_peered":"2022-09-07T21:39:47.794303+0000","last_clean":"2022-09-07T21:39:47.794303+0000","last_became_active":"2022-09-07T21:39:47.793887+0000","last_became_peered":"2022-09-07T21:39:47.793887+0000","last_unstale":"2022-09-07T21:39:47.794303+0000","last_undegraded":"2022-09-07T21:39:47.794303+0000","last_fullsized":"2022-09-07T21:39:47.794303+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:11:36.134842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790182+0000","last_change":"2022-09-07T21:38:39.964670+0000","last_active":"2022-09-07T21:39:46.790182+0000","last_peered":"2022-09-07T21:39:46.790182+0000","last_clean":"2022-09-07T21:39:46.790182+0000","last_became_active":"2022-09-07T21:38:39.964305+0000","last_became_peered":"2022-09-07T21:38:39.964305+0000","last_unstale":"2022-09-07T21:39:46.790182+0000","last_undegraded":"2022-09-07T21:39:46.790182+0000","last_fullsized":"2022-09-07T21:39:46.790182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:15:57.041815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789403+0000","last_change":"2022-09-07T21:38:40.565156+0000","last_active":"2022-09-07T21:39:46.789403+0000","last_peered":"2022-09-07T21:39:46.789403+0000","last_clean":"2022-09-07T21:39:46.789403+0000","last_became_active":"2022-09-07T21:38:40.564843+0000","last_became_peered":"2022-09-07T21:38:40.564843+0000","last_unstale":"2022-09-07T21:39:46.789403+0000","last_undegraded":"2022-09-07T21:39:46.789403+0000","last_fullsized":"2022-09-07T21:39:46.789403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:11:09.741599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789546+0000","last_change":"2022-09-07T21:38:44.986167+0000","last_active":"2022-09-07T21:39:46.789546+0000","last_peered":"2022-09-07T21:39:46.789546+0000","last_clean":"2022-09-07T21:39:46.789546+0000","last_became_active":"2022-09-07T21:38:44.985778+0000","last_became_peered":"2022-09-07T21:38:44.985778+0000","last_unstale":"2022-09-07T21:39:46.789546+0000","last_undegraded":"2022-09-07T21:39:46.789546+0000","last_fullsized":"2022-09-07T21:39:46.789546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:21:01.441205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'33","reported_seq":159,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:56.810222+0000","last_change":"2022-09-07T21:38:39.968660+0000","last_active":"2022-09-07T21:39:56.810222+0000","last_peered":"2022-09-07T21:39:56.810222+0000","last_clean":"2022-09-07T21:39:56.810222+0000","last_became_active":"2022-09-07T21:38:39.968285+0000","last_became_peered":"2022-09-07T21:38:39.968285+0000","last_unstale":"2022-09-07T21:39:56.810222+0000","last_undegraded":"2022-09-07T21:39:56.810222+0000","last_fullsized":"2022-09-07T21:39:56.810222+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:16:16.027615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60865,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":108,"num_read_kb":114,"num_write":34,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.787175+0000","last_change":"2022-09-07T21:38:40.566145+0000","last_active":"2022-09-07T21:39:46.787175+0000","last_peered":"2022-09-07T21:39:46.787175+0000","last_clean":"2022-09-07T21:39:46.787175+0000","last_became_active":"2022-09-07T21:38:40.565823+0000","last_became_peered":"2022-09-07T21:38:40.565823+0000","last_unstale":"2022-09-07T21:39:46.787175+0000","last_undegraded":"2022-09-07T21:39:46.787175+0000","last_fullsized":"2022-09-07T21:39:46.787175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:13:28.826066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790639+0000","last_change":"2022-09-07T21:38:44.995599+0000","last_active":"2022-09-07T21:39:46.790639+0000","last_peered":"2022-09-07T21:39:46.790639+0000","last_clean":"2022-09-07T21:39:46.790639+0000","last_became_active":"2022-09-07T21:38:44.994627+0000","last_became_peered":"2022-09-07T21:38:44.994627+0000","last_unstale":"2022-09-07T21:39:46.790639+0000","last_undegraded":"2022-09-07T21:39:46.790639+0000","last_fullsized":"2022-09-07T21:39:46.790639+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:17:30.608595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.794164+0000","last_change":"2022-09-07T21:38:39.968773+0000","last_active":"2022-09-07T21:39:46.794164+0000","last_peered":"2022-09-07T21:39:46.794164+0000","last_clean":"2022-09-07T21:39:46.794164+0000","last_became_active":"2022-09-07T21:38:39.967912+0000","last_became_peered":"2022-09-07T21:38:39.967912+0000","last_unstale":"2022-09-07T21:39:46.794164+0000","last_undegraded":"2022-09-07T21:39:46.794164+0000","last_fullsized":"2022-09-07T21:39:46.794164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:51:12.940327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788800+0000","last_change":"2022-09-07T21:38:40.573548+0000","last_active":"2022-09-07T21:39:46.788800+0000","last_peered":"2022-09-07T21:39:46.788800+0000","last_clean":"2022-09-07T21:39:46.788800+0000","last_became_active":"2022-09-07T21:38:40.573273+0000","last_became_peered":"2022-09-07T21:38:40.573273+0000","last_unstale":"2022-09-07T21:39:46.788800+0000","last_undegraded":"2022-09-07T21:39:46.788800+0000","last_fullsized":"2022-09-07T21:39:46.788800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:03:52.778107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789661+0000","last_change":"2022-09-07T21:38:44.987366+0000","last_active":"2022-09-07T21:39:46.789661+0000","last_peered":"2022-09-07T21:39:46.789661+0000","last_clean":"2022-09-07T21:39:46.789661+0000","last_became_active":"2022-09-07T21:38:44.986775+0000","last_became_peered":"2022-09-07T21:38:44.986775+0000","last_unstale":"2022-09-07T21:39:46.789661+0000","last_undegraded":"2022-09-07T21:39:46.789661+0000","last_fullsized":"2022-09-07T21:39:46.789661+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:07:14.746521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790197+0000","last_change":"2022-09-07T21:38:39.969123+0000","last_active":"2022-09-07T21:39:46.790197+0000","last_peered":"2022-09-07T21:39:46.790197+0000","last_clean":"2022-09-07T21:39:46.790197+0000","last_became_active":"2022-09-07T21:38:39.968744+0000","last_became_peered":"2022-09-07T21:38:39.968744+0000","last_unstale":"2022-09-07T21:39:46.790197+0000","last_undegraded":"2022-09-07T21:39:46.790197+0000","last_fullsized":"2022-09-07T21:39:46.790197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:54:34.616856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789635+0000","last_change":"2022-09-07T21:38:40.578056+0000","last_active":"2022-09-07T21:39:46.789635+0000","last_peered":"2022-09-07T21:39:46.789635+0000","last_clean":"2022-09-07T21:39:46.789635+0000","last_became_active":"2022-09-07T21:38:40.577727+0000","last_became_peered":"2022-09-07T21:38:40.577727+0000","last_unstale":"2022-09-07T21:39:46.789635+0000","last_undegraded":"2022-09-07T21:39:46.789635+0000","last_fullsized":"2022-09-07T21:39:46.789635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:10:08.645724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:48.011128+0000","last_change":"2022-09-07T21:39:48.010666+0000","last_active":"2022-09-07T21:39:48.011128+0000","last_peered":"2022-09-07T21:39:48.011128+0000","last_clean":"2022-09-07T21:39:48.011128+0000","last_became_active":"2022-09-07T21:39:48.010254+0000","last_became_peered":"2022-09-07T21:39:48.010254+0000","last_unstale":"2022-09-07T21:39:48.011128+0000","last_undegraded":"2022-09-07T21:39:48.011128+0000","last_fullsized":"2022-09-07T21:39:48.011128+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:54:08.005552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:56.941296+0000","last_change":"2022-09-07T21:38:39.962760+0000","last_active":"2022-09-07T21:39:56.941296+0000","last_peered":"2022-09-07T21:39:56.941296+0000","last_clean":"2022-09-07T21:39:56.941296+0000","last_became_active":"2022-09-07T21:38:39.962327+0000","last_became_peered":"2022-09-07T21:38:39.962327+0000","last_unstale":"2022-09-07T21:39:56.941296+0000","last_undegraded":"2022-09-07T21:39:56.941296+0000","last_fullsized":"2022-09-07T21:39:56.941296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:17:27.180736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788943+0000","last_change":"2022-09-07T21:38:40.569532+0000","last_active":"2022-09-07T21:39:46.788943+0000","last_peered":"2022-09-07T21:39:46.788943+0000","last_clean":"2022-09-07T21:39:46.788943+0000","last_became_active":"2022-09-07T21:38:40.569037+0000","last_became_peered":"2022-09-07T21:38:40.569037+0000","last_unstale":"2022-09-07T21:39:46.788943+0000","last_undegraded":"2022-09-07T21:39:46.788943+0000","last_fullsized":"2022-09-07T21:39:46.788943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:37:05.114735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788554+0000","last_change":"2022-09-07T21:38:44.995323+0000","last_active":"2022-09-07T21:39:46.788554+0000","last_peered":"2022-09-07T21:39:46.788554+0000","last_clean":"2022-09-07T21:39:46.788554+0000","last_became_active":"2022-09-07T21:38:44.994915+0000","last_became_peered":"2022-09-07T21:38:44.994915+0000","last_unstale":"2022-09-07T21:39:46.788554+0000","last_undegraded":"2022-09-07T21:39:46.788554+0000","last_fullsized":"2022-09-07T21:39:46.788554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:17:10.419180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.786701+0000","last_change":"2022-09-07T21:38:39.967903+0000","last_active":"2022-09-07T21:39:46.786701+0000","last_peered":"2022-09-07T21:39:46.786701+0000","last_clean":"2022-09-07T21:39:46.786701+0000","last_became_active":"2022-09-07T21:38:39.967392+0000","last_became_peered":"2022-09-07T21:38:39.967392+0000","last_unstale":"2022-09-07T21:39:46.786701+0000","last_undegraded":"2022-09-07T21:39:46.786701+0000","last_fullsized":"2022-09-07T21:39:46.786701+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:31:39.581561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790435+0000","last_change":"2022-09-07T21:38:40.566956+0000","last_active":"2022-09-07T21:39:46.790435+0000","last_peered":"2022-09-07T21:39:46.790435+0000","last_clean":"2022-09-07T21:39:46.790435+0000","last_became_active":"2022-09-07T21:38:40.566655+0000","last_became_peered":"2022-09-07T21:38:40.566655+0000","last_unstale":"2022-09-07T21:39:46.790435+0000","last_undegraded":"2022-09-07T21:39:46.790435+0000","last_fullsized":"2022-09-07T21:39:46.790435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:37:16.353162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790328+0000","last_change":"2022-09-07T21:38:44.988605+0000","last_active":"2022-09-07T21:39:46.790328+0000","last_peered":"2022-09-07T21:39:46.790328+0000","last_clean":"2022-09-07T21:39:46.790328+0000","last_became_active":"2022-09-07T21:38:44.988082+0000","last_became_peered":"2022-09-07T21:38:44.988082+0000","last_unstale":"2022-09-07T21:39:46.790328+0000","last_undegraded":"2022-09-07T21:39:46.790328+0000","last_fullsized":"2022-09-07T21:39:46.790328+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:30:06.975077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.787324+0000","last_change":"2022-09-07T21:38:39.961740+0000","last_active":"2022-09-07T21:39:46.787324+0000","last_peered":"2022-09-07T21:39:46.787324+0000","last_clean":"2022-09-07T21:39:46.787324+0000","last_became_active":"2022-09-07T21:38:39.961219+0000","last_became_peered":"2022-09-07T21:38:39.961219+0000","last_unstale":"2022-09-07T21:39:46.787324+0000","last_undegraded":"2022-09-07T21:39:46.787324+0000","last_fullsized":"2022-09-07T21:39:46.787324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:51:24.649401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791871+0000","last_change":"2022-09-07T21:38:40.569594+0000","last_active":"2022-09-07T21:39:46.791871+0000","last_peered":"2022-09-07T21:39:46.791871+0000","last_clean":"2022-09-07T21:39:46.791871+0000","last_became_active":"2022-09-07T21:38:40.569251+0000","last_became_peered":"2022-09-07T21:38:40.569251+0000","last_unstale":"2022-09-07T21:39:46.791871+0000","last_undegraded":"2022-09-07T21:39:46.791871+0000","last_fullsized":"2022-09-07T21:39:46.791871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:37:43.337560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.786814+0000","last_change":"2022-09-07T21:38:44.993534+0000","last_active":"2022-09-07T21:39:46.786814+0000","last_peered":"2022-09-07T21:39:46.786814+0000","last_clean":"2022-09-07T21:39:46.786814+0000","last_became_active":"2022-09-07T21:38:44.992966+0000","last_became_peered":"2022-09-07T21:38:44.992966+0000","last_unstale":"2022-09-07T21:39:46.786814+0000","last_undegraded":"2022-09-07T21:39:46.786814+0000","last_fullsized":"2022-09-07T21:39:46.786814+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:12:27.991839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790045+0000","last_change":"2022-09-07T21:38:39.978436+0000","last_active":"2022-09-07T21:39:46.790045+0000","last_peered":"2022-09-07T21:39:46.790045+0000","last_clean":"2022-09-07T21:39:46.790045+0000","last_became_active":"2022-09-07T21:38:39.977613+0000","last_became_peered":"2022-09-07T21:38:39.977613+0000","last_unstale":"2022-09-07T21:39:46.790045+0000","last_undegraded":"2022-09-07T21:39:46.790045+0000","last_fullsized":"2022-09-07T21:39:46.790045+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:23:08.765467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790791+0000","last_change":"2022-09-07T21:38:40.577968+0000","last_active":"2022-09-07T21:39:46.790791+0000","last_peered":"2022-09-07T21:39:46.790791+0000","last_clean":"2022-09-07T21:39:46.790791+0000","last_became_active":"2022-09-07T21:38:40.577632+0000","last_became_peered":"2022-09-07T21:38:40.577632+0000","last_unstale":"2022-09-07T21:39:46.790791+0000","last_undegraded":"2022-09-07T21:39:46.790791+0000","last_fullsized":"2022-09-07T21:39:46.790791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:57:31.599202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788466+0000","last_change":"2022-09-07T21:38:44.989341+0000","last_active":"2022-09-07T21:39:46.788466+0000","last_peered":"2022-09-07T21:39:46.788466+0000","last_clean":"2022-09-07T21:39:46.788466+0000","last_became_active":"2022-09-07T21:38:44.988977+0000","last_became_peered":"2022-09-07T21:38:44.988977+0000","last_unstale":"2022-09-07T21:39:46.788466+0000","last_undegraded":"2022-09-07T21:39:46.788466+0000","last_fullsized":"2022-09-07T21:39:46.788466+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:34:52.397025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789613+0000","last_change":"2022-09-07T21:38:39.970365+0000","last_active":"2022-09-07T21:39:46.789613+0000","last_peered":"2022-09-07T21:39:46.789613+0000","last_clean":"2022-09-07T21:39:46.789613+0000","last_became_active":"2022-09-07T21:38:39.970044+0000","last_became_peered":"2022-09-07T21:38:39.970044+0000","last_unstale":"2022-09-07T21:39:46.789613+0000","last_undegraded":"2022-09-07T21:39:46.789613+0000","last_fullsized":"2022-09-07T21:39:46.789613+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:39:04.408591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790853+0000","last_change":"2022-09-07T21:38:40.564714+0000","last_active":"2022-09-07T21:39:46.790853+0000","last_peered":"2022-09-07T21:39:46.790853+0000","last_clean":"2022-09-07T21:39:46.790853+0000","last_became_active":"2022-09-07T21:38:40.563744+0000","last_became_peered":"2022-09-07T21:38:40.563744+0000","last_unstale":"2022-09-07T21:39:46.790853+0000","last_undegraded":"2022-09-07T21:39:46.790853+0000","last_fullsized":"2022-09-07T21:39:46.790853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:46:29.965821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:47.794770+0000","last_change":"2022-09-07T21:39:47.794770+0000","last_active":"2022-09-07T21:39:47.794770+0000","last_peered":"2022-09-07T21:39:47.794770+0000","last_clean":"2022-09-07T21:39:47.794770+0000","last_became_active":"2022-09-07T21:39:47.794311+0000","last_became_peered":"2022-09-07T21:39:47.794311+0000","last_unstale":"2022-09-07T21:39:47.794770+0000","last_undegraded":"2022-09-07T21:39:47.794770+0000","last_fullsized":"2022-09-07T21:39:47.794770+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:36:29.299981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791062+0000","last_change":"2022-09-07T21:38:40.567008+0000","last_active":"2022-09-07T21:39:46.791062+0000","last_peered":"2022-09-07T21:39:46.791062+0000","last_clean":"2022-09-07T21:39:46.791062+0000","last_became_active":"2022-09-07T21:38:40.566598+0000","last_became_peered":"2022-09-07T21:38:40.566598+0000","last_unstale":"2022-09-07T21:39:46.791062+0000","last_undegraded":"2022-09-07T21:39:46.791062+0000","last_fullsized":"2022-09-07T21:39:46.791062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:06:16.999794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:59.945811+0000","last_change":"2022-09-07T21:38:39.966446+0000","last_active":"2022-09-07T21:39:59.945811+0000","last_peered":"2022-09-07T21:39:59.945811+0000","last_clean":"2022-09-07T21:39:59.945811+0000","last_became_active":"2022-09-07T21:38:39.966095+0000","last_became_peered":"2022-09-07T21:38:39.966095+0000","last_unstale":"2022-09-07T21:39:59.945811+0000","last_undegraded":"2022-09-07T21:39:59.945811+0000","last_fullsized":"2022-09-07T21:39:59.945811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:08:54.360894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":55,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790628+0000","last_change":"2022-09-07T21:38:44.993551+0000","last_active":"2022-09-07T21:39:46.790628+0000","last_peered":"2022-09-07T21:39:46.790628+0000","last_clean":"2022-09-07T21:39:46.790628+0000","last_became_active":"2022-09-07T21:38:44.993235+0000","last_became_peered":"2022-09-07T21:38:44.993235+0000","last_unstale":"2022-09-07T21:39:46.790628+0000","last_undegraded":"2022-09-07T21:39:46.790628+0000","last_fullsized":"2022-09-07T21:39:46.790628+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:06:31.660680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784919+0000","last_change":"2022-09-07T21:38:40.578195+0000","last_active":"2022-09-07T21:39:46.784919+0000","last_peered":"2022-09-07T21:39:46.784919+0000","last_clean":"2022-09-07T21:39:46.784919+0000","last_became_active":"2022-09-07T21:38:40.577888+0000","last_became_peered":"2022-09-07T21:38:40.577888+0000","last_unstale":"2022-09-07T21:39:46.784919+0000","last_undegraded":"2022-09-07T21:39:46.784919+0000","last_fullsized":"2022-09-07T21:39:46.784919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:09:25.600697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793053+0000","last_change":"2022-09-07T21:38:39.970599+0000","last_active":"2022-09-07T21:39:46.793053+0000","last_peered":"2022-09-07T21:39:46.793053+0000","last_clean":"2022-09-07T21:39:46.793053+0000","last_became_active":"2022-09-07T21:38:39.970216+0000","last_became_peered":"2022-09-07T21:38:39.970216+0000","last_unstale":"2022-09-07T21:39:46.793053+0000","last_undegraded":"2022-09-07T21:39:46.793053+0000","last_fullsized":"2022-09-07T21:39:46.793053+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:01:36.517769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":381,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788723+0000","last_change":"2022-09-07T21:37:30.699119+0000","last_active":"2022-09-07T21:39:46.788723+0000","last_peered":"2022-09-07T21:39:46.788723+0000","last_clean":"2022-09-07T21:39:46.788723+0000","last_became_active":"2022-09-07T21:37:30.698640+0000","last_became_peered":"2022-09-07T21:37:30.698640+0000","last_unstale":"2022-09-07T21:39:46.788723+0000","last_undegraded":"2022-09-07T21:39:46.788723+0000","last_fullsized":"2022-09-07T21:39:46.788723+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:33:29.392715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:33:29.392715+0000","last_clean_scrub_stamp":"2022-09-07T21:33:29.392715+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:10:27.439063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790737+0000","last_change":"2022-09-07T21:38:44.996255+0000","last_active":"2022-09-07T21:39:46.790737+0000","last_peered":"2022-09-07T21:39:46.790737+0000","last_clean":"2022-09-07T21:39:46.790737+0000","last_became_active":"2022-09-07T21:38:44.995597+0000","last_became_peered":"2022-09-07T21:38:44.995597+0000","last_unstale":"2022-09-07T21:39:46.790737+0000","last_undegraded":"2022-09-07T21:39:46.790737+0000","last_fullsized":"2022-09-07T21:39:46.790737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:59:56.494229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.787878+0000","last_change":"2022-09-07T21:38:40.567961+0000","last_active":"2022-09-07T21:39:46.787878+0000","last_peered":"2022-09-07T21:39:46.787878+0000","last_clean":"2022-09-07T21:39:46.787878+0000","last_became_active":"2022-09-07T21:38:40.567436+0000","last_became_peered":"2022-09-07T21:38:40.567436+0000","last_unstale":"2022-09-07T21:39:46.787878+0000","last_undegraded":"2022-09-07T21:39:46.787878+0000","last_fullsized":"2022-09-07T21:39:46.787878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:25:13.025231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788853+0000","last_change":"2022-09-07T21:38:39.971039+0000","last_active":"2022-09-07T21:39:46.788853+0000","last_peered":"2022-09-07T21:39:46.788853+0000","last_clean":"2022-09-07T21:39:46.788853+0000","last_became_active":"2022-09-07T21:38:39.970675+0000","last_became_peered":"2022-09-07T21:38:39.970675+0000","last_unstale":"2022-09-07T21:39:46.788853+0000","last_undegraded":"2022-09-07T21:39:46.788853+0000","last_fullsized":"2022-09-07T21:39:46.788853+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:36:47.865699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789711+0000","last_change":"2022-09-07T21:38:44.992311+0000","last_active":"2022-09-07T21:39:46.789711+0000","last_peered":"2022-09-07T21:39:46.789711+0000","last_clean":"2022-09-07T21:39:46.789711+0000","last_became_active":"2022-09-07T21:38:44.991846+0000","last_became_peered":"2022-09-07T21:38:44.991846+0000","last_unstale":"2022-09-07T21:39:46.789711+0000","last_undegraded":"2022-09-07T21:39:46.789711+0000","last_fullsized":"2022-09-07T21:39:46.789711+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:42:43.968482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.787249+0000","last_change":"2022-09-07T21:38:39.970220+0000","last_active":"2022-09-07T21:39:46.787249+0000","last_peered":"2022-09-07T21:39:46.787249+0000","last_clean":"2022-09-07T21:39:46.787249+0000","last_became_active":"2022-09-07T21:38:39.969573+0000","last_became_peered":"2022-09-07T21:38:39.969573+0000","last_unstale":"2022-09-07T21:39:46.787249+0000","last_undegraded":"2022-09-07T21:39:46.787249+0000","last_fullsized":"2022-09-07T21:39:46.787249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:22:05.375623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788624+0000","last_change":"2022-09-07T21:38:40.585697+0000","last_active":"2022-09-07T21:39:46.788624+0000","last_peered":"2022-09-07T21:39:46.788624+0000","last_clean":"2022-09-07T21:39:46.788624+0000","last_became_active":"2022-09-07T21:38:40.585293+0000","last_became_peered":"2022-09-07T21:38:40.585293+0000","last_unstale":"2022-09-07T21:39:46.788624+0000","last_undegraded":"2022-09-07T21:39:46.788624+0000","last_fullsized":"2022-09-07T21:39:46.788624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:22:21.707916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784222+0000","last_change":"2022-09-07T21:38:44.985777+0000","last_active":"2022-09-07T21:39:46.784222+0000","last_peered":"2022-09-07T21:39:46.784222+0000","last_clean":"2022-09-07T21:39:46.784222+0000","last_became_active":"2022-09-07T21:38:44.985312+0000","last_became_peered":"2022-09-07T21:38:44.985312+0000","last_unstale":"2022-09-07T21:39:46.784222+0000","last_undegraded":"2022-09-07T21:39:46.784222+0000","last_fullsized":"2022-09-07T21:39:46.784222+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:08:04.011094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'6","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790103+0000","last_change":"2022-09-07T21:38:39.973742+0000","last_active":"2022-09-07T21:39:46.790103+0000","last_peered":"2022-09-07T21:39:46.790103+0000","last_clean":"2022-09-07T21:39:46.790103+0000","last_became_active":"2022-09-07T21:38:39.973324+0000","last_became_peered":"2022-09-07T21:38:39.973324+0000","last_unstale":"2022-09-07T21:39:46.790103+0000","last_undegraded":"2022-09-07T21:39:46.790103+0000","last_fullsized":"2022-09-07T21:39:46.790103+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:49:54.331302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.785668+0000","last_change":"2022-09-07T21:38:40.565122+0000","last_active":"2022-09-07T21:39:46.785668+0000","last_peered":"2022-09-07T21:39:46.785668+0000","last_clean":"2022-09-07T21:39:46.785668+0000","last_became_active":"2022-09-07T21:38:40.564813+0000","last_became_peered":"2022-09-07T21:38:40.564813+0000","last_unstale":"2022-09-07T21:39:46.785668+0000","last_undegraded":"2022-09-07T21:39:46.785668+0000","last_fullsized":"2022-09-07T21:39:46.785668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:03:02.178226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791171+0000","last_change":"2022-09-07T21:38:44.987707+0000","last_active":"2022-09-07T21:39:46.791171+0000","last_peered":"2022-09-07T21:39:46.791171+0000","last_clean":"2022-09-07T21:39:46.791171+0000","last_became_active":"2022-09-07T21:38:44.987296+0000","last_became_peered":"2022-09-07T21:38:44.987296+0000","last_unstale":"2022-09-07T21:39:46.791171+0000","last_undegraded":"2022-09-07T21:39:46.791171+0000","last_fullsized":"2022-09-07T21:39:46.791171+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:53:07.004127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793725+0000","last_change":"2022-09-07T21:38:39.967664+0000","last_active":"2022-09-07T21:39:46.793725+0000","last_peered":"2022-09-07T21:39:46.793725+0000","last_clean":"2022-09-07T21:39:46.793725+0000","last_became_active":"2022-09-07T21:38:39.967283+0000","last_became_peered":"2022-09-07T21:38:39.967283+0000","last_unstale":"2022-09-07T21:39:46.793725+0000","last_undegraded":"2022-09-07T21:39:46.793725+0000","last_fullsized":"2022-09-07T21:39:46.793725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:47:31.095692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784474+0000","last_change":"2022-09-07T21:38:40.579398+0000","last_active":"2022-09-07T21:39:46.784474+0000","last_peered":"2022-09-07T21:39:46.784474+0000","last_clean":"2022-09-07T21:39:46.784474+0000","last_became_active":"2022-09-07T21:38:40.579112+0000","last_became_peered":"2022-09-07T21:38:40.579112+0000","last_unstale":"2022-09-07T21:39:46.784474+0000","last_undegraded":"2022-09-07T21:39:46.784474+0000","last_fullsized":"2022-09-07T21:39:46.784474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:50:13.063073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793498+0000","last_change":"2022-09-07T21:38:44.996219+0000","last_active":"2022-09-07T21:39:46.793498+0000","last_peered":"2022-09-07T21:39:46.793498+0000","last_clean":"2022-09-07T21:39:46.793498+0000","last_became_active":"2022-09-07T21:38:44.995738+0000","last_became_peered":"2022-09-07T21:38:44.995738+0000","last_unstale":"2022-09-07T21:39:46.793498+0000","last_undegraded":"2022-09-07T21:39:46.793498+0000","last_fullsized":"2022-09-07T21:39:46.793498+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:36:49.744364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788166+0000","last_change":"2022-09-07T21:38:39.965087+0000","last_active":"2022-09-07T21:39:46.788166+0000","last_peered":"2022-09-07T21:39:46.788166+0000","last_clean":"2022-09-07T21:39:46.788166+0000","last_became_active":"2022-09-07T21:38:39.964681+0000","last_became_peered":"2022-09-07T21:38:39.964681+0000","last_unstale":"2022-09-07T21:39:46.788166+0000","last_undegraded":"2022-09-07T21:39:46.788166+0000","last_fullsized":"2022-09-07T21:39:46.788166+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:47:50.028036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.786720+0000","last_change":"2022-09-07T21:38:40.564639+0000","last_active":"2022-09-07T21:39:46.786720+0000","last_peered":"2022-09-07T21:39:46.786720+0000","last_clean":"2022-09-07T21:39:46.786720+0000","last_became_active":"2022-09-07T21:38:40.564315+0000","last_became_peered":"2022-09-07T21:38:40.564315+0000","last_unstale":"2022-09-07T21:39:46.786720+0000","last_undegraded":"2022-09-07T21:39:46.786720+0000","last_fullsized":"2022-09-07T21:39:46.786720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:20:27.670309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:48.010329+0000","last_change":"2022-09-07T21:39:48.010329+0000","last_active":"2022-09-07T21:39:48.010329+0000","last_peered":"2022-09-07T21:39:48.010329+0000","last_clean":"2022-09-07T21:39:48.010329+0000","last_became_active":"2022-09-07T21:39:48.009972+0000","last_became_peered":"2022-09-07T21:39:48.009972+0000","last_unstale":"2022-09-07T21:39:48.010329+0000","last_undegraded":"2022-09-07T21:39:48.010329+0000","last_fullsized":"2022-09-07T21:39:48.010329+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:59:50.655757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784630+0000","last_change":"2022-09-07T21:38:39.967488+0000","last_active":"2022-09-07T21:39:46.784630+0000","last_peered":"2022-09-07T21:39:46.784630+0000","last_clean":"2022-09-07T21:39:46.784630+0000","last_became_active":"2022-09-07T21:38:39.967162+0000","last_became_peered":"2022-09-07T21:38:39.967162+0000","last_unstale":"2022-09-07T21:39:46.784630+0000","last_undegraded":"2022-09-07T21:39:46.784630+0000","last_fullsized":"2022-09-07T21:39:46.784630+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:43:43.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790023+0000","last_change":"2022-09-07T21:38:40.568656+0000","last_active":"2022-09-07T21:39:46.790023+0000","last_peered":"2022-09-07T21:39:46.790023+0000","last_clean":"2022-09-07T21:39:46.790023+0000","last_became_active":"2022-09-07T21:38:40.568357+0000","last_became_peered":"2022-09-07T21:38:40.568357+0000","last_unstale":"2022-09-07T21:39:46.790023+0000","last_undegraded":"2022-09-07T21:39:46.790023+0000","last_fullsized":"2022-09-07T21:39:46.790023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:31:32.498945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793066+0000","last_change":"2022-09-07T21:38:44.995472+0000","last_active":"2022-09-07T21:39:46.793066+0000","last_peered":"2022-09-07T21:39:46.793066+0000","last_clean":"2022-09-07T21:39:46.793066+0000","last_became_active":"2022-09-07T21:38:44.995030+0000","last_became_peered":"2022-09-07T21:38:44.995030+0000","last_unstale":"2022-09-07T21:39:46.793066+0000","last_undegraded":"2022-09-07T21:39:46.793066+0000","last_fullsized":"2022-09-07T21:39:46.793066+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:39:41.557547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793396+0000","last_change":"2022-09-07T21:38:39.969316+0000","last_active":"2022-09-07T21:39:46.793396+0000","last_peered":"2022-09-07T21:39:46.793396+0000","last_clean":"2022-09-07T21:39:46.793396+0000","last_became_active":"2022-09-07T21:38:39.968958+0000","last_became_peered":"2022-09-07T21:38:39.968958+0000","last_unstale":"2022-09-07T21:39:46.793396+0000","last_undegraded":"2022-09-07T21:39:46.793396+0000","last_fullsized":"2022-09-07T21:39:46.793396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:51:01.568029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.785307+0000","last_change":"2022-09-07T21:38:40.579479+0000","last_active":"2022-09-07T21:39:46.785307+0000","last_peered":"2022-09-07T21:39:46.785307+0000","last_clean":"2022-09-07T21:39:46.785307+0000","last_became_active":"2022-09-07T21:38:40.578452+0000","last_became_peered":"2022-09-07T21:38:40.578452+0000","last_unstale":"2022-09-07T21:39:46.785307+0000","last_undegraded":"2022-09-07T21:39:46.785307+0000","last_fullsized":"2022-09-07T21:39:46.785307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:34:24.096814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793294+0000","last_change":"2022-09-07T21:38:44.988455+0000","last_active":"2022-09-07T21:39:46.793294+0000","last_peered":"2022-09-07T21:39:46.793294+0000","last_clean":"2022-09-07T21:39:46.793294+0000","last_became_active":"2022-09-07T21:38:44.988070+0000","last_became_peered":"2022-09-07T21:38:44.988070+0000","last_unstale":"2022-09-07T21:39:46.793294+0000","last_undegraded":"2022-09-07T21:39:46.793294+0000","last_fullsized":"2022-09-07T21:39:46.793294+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:52:37.094892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.786944+0000","last_change":"2022-09-07T21:38:39.972219+0000","last_active":"2022-09-07T21:39:46.786944+0000","last_peered":"2022-09-07T21:39:46.786944+0000","last_clean":"2022-09-07T21:39:46.786944+0000","last_became_active":"2022-09-07T21:38:39.971484+0000","last_became_peered":"2022-09-07T21:38:39.971484+0000","last_unstale":"2022-09-07T21:39:46.786944+0000","last_undegraded":"2022-09-07T21:39:46.786944+0000","last_fullsized":"2022-09-07T21:39:46.786944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:54:17.947316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789667+0000","last_change":"2022-09-07T21:38:40.578389+0000","last_active":"2022-09-07T21:39:46.789667+0000","last_peered":"2022-09-07T21:39:46.789667+0000","last_clean":"2022-09-07T21:39:46.789667+0000","last_became_active":"2022-09-07T21:38:40.577886+0000","last_became_peered":"2022-09-07T21:38:40.577886+0000","last_unstale":"2022-09-07T21:39:46.789667+0000","last_undegraded":"2022-09-07T21:39:46.789667+0000","last_fullsized":"2022-09-07T21:39:46.789667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:48:07.590464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.787982+0000","last_change":"2022-09-07T21:38:44.994968+0000","last_active":"2022-09-07T21:39:46.787982+0000","last_peered":"2022-09-07T21:39:46.787982+0000","last_clean":"2022-09-07T21:39:46.787982+0000","last_became_active":"2022-09-07T21:38:44.994516+0000","last_became_peered":"2022-09-07T21:38:44.994516+0000","last_unstale":"2022-09-07T21:39:46.787982+0000","last_undegraded":"2022-09-07T21:39:46.787982+0000","last_fullsized":"2022-09-07T21:39:46.787982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:49:10.172571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793679+0000","last_change":"2022-09-07T21:38:39.970117+0000","last_active":"2022-09-07T21:39:46.793679+0000","last_peered":"2022-09-07T21:39:46.793679+0000","last_clean":"2022-09-07T21:39:46.793679+0000","last_became_active":"2022-09-07T21:38:39.966606+0000","last_became_peered":"2022-09-07T21:38:39.966606+0000","last_unstale":"2022-09-07T21:39:46.793679+0000","last_undegraded":"2022-09-07T21:39:46.793679+0000","last_fullsized":"2022-09-07T21:39:46.793679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:59:13.103878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791392+0000","last_change":"2022-09-07T21:38:40.563232+0000","last_active":"2022-09-07T21:39:46.791392+0000","last_peered":"2022-09-07T21:39:46.791392+0000","last_clean":"2022-09-07T21:39:46.791392+0000","last_became_active":"2022-09-07T21:38:40.562945+0000","last_became_peered":"2022-09-07T21:38:40.562945+0000","last_unstale":"2022-09-07T21:39:46.791392+0000","last_undegraded":"2022-09-07T21:39:46.791392+0000","last_fullsized":"2022-09-07T21:39:46.791392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:42:44.271490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.783840+0000","last_change":"2022-09-07T21:38:44.996941+0000","last_active":"2022-09-07T21:39:46.783840+0000","last_peered":"2022-09-07T21:39:46.783840+0000","last_clean":"2022-09-07T21:39:46.783840+0000","last_became_active":"2022-09-07T21:38:44.988680+0000","last_became_peered":"2022-09-07T21:38:44.988680+0000","last_unstale":"2022-09-07T21:39:46.783840+0000","last_undegraded":"2022-09-07T21:39:46.783840+0000","last_fullsized":"2022-09-07T21:39:46.783840+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:33:12.017654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790387+0000","last_change":"2022-09-07T21:38:39.969588+0000","last_active":"2022-09-07T21:39:46.790387+0000","last_peered":"2022-09-07T21:39:46.790387+0000","last_clean":"2022-09-07T21:39:46.790387+0000","last_became_active":"2022-09-07T21:38:39.969222+0000","last_became_peered":"2022-09-07T21:38:39.969222+0000","last_unstale":"2022-09-07T21:39:46.790387+0000","last_undegraded":"2022-09-07T21:39:46.790387+0000","last_fullsized":"2022-09-07T21:39:46.790387+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:46:36.580497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790363+0000","last_change":"2022-09-07T21:38:40.569663+0000","last_active":"2022-09-07T21:39:46.790363+0000","last_peered":"2022-09-07T21:39:46.790363+0000","last_clean":"2022-09-07T21:39:46.790363+0000","last_became_active":"2022-09-07T21:38:40.569376+0000","last_became_peered":"2022-09-07T21:38:40.569376+0000","last_unstale":"2022-09-07T21:39:46.790363+0000","last_undegraded":"2022-09-07T21:39:46.790363+0000","last_fullsized":"2022-09-07T21:39:46.790363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:06:01.895953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790372+0000","last_change":"2022-09-07T21:38:44.987761+0000","last_active":"2022-09-07T21:39:46.790372+0000","last_peered":"2022-09-07T21:39:46.790372+0000","last_clean":"2022-09-07T21:39:46.790372+0000","last_became_active":"2022-09-07T21:38:44.987376+0000","last_became_peered":"2022-09-07T21:38:44.987376+0000","last_unstale":"2022-09-07T21:39:46.790372+0000","last_undegraded":"2022-09-07T21:39:46.790372+0000","last_fullsized":"2022-09-07T21:39:46.790372+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:37:19.993838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788457+0000","last_change":"2022-09-07T21:38:39.968703+0000","last_active":"2022-09-07T21:39:46.788457+0000","last_peered":"2022-09-07T21:39:46.788457+0000","last_clean":"2022-09-07T21:39:46.788457+0000","last_became_active":"2022-09-07T21:38:39.968393+0000","last_became_peered":"2022-09-07T21:38:39.968393+0000","last_unstale":"2022-09-07T21:39:46.788457+0000","last_undegraded":"2022-09-07T21:39:46.788457+0000","last_fullsized":"2022-09-07T21:39:46.788457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:46:36.042794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789617+0000","last_change":"2022-09-07T21:38:40.567044+0000","last_active":"2022-09-07T21:39:46.789617+0000","last_peered":"2022-09-07T21:39:46.789617+0000","last_clean":"2022-09-07T21:39:46.789617+0000","last_became_active":"2022-09-07T21:38:40.566732+0000","last_became_peered":"2022-09-07T21:38:40.566732+0000","last_unstale":"2022-09-07T21:39:46.789617+0000","last_undegraded":"2022-09-07T21:39:46.789617+0000","last_fullsized":"2022-09-07T21:39:46.789617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:12:52.569238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790103+0000","last_change":"2022-09-07T21:38:40.563301+0000","last_active":"2022-09-07T21:39:46.790103+0000","last_peered":"2022-09-07T21:39:46.790103+0000","last_clean":"2022-09-07T21:39:46.790103+0000","last_became_active":"2022-09-07T21:38:40.562979+0000","last_became_peered":"2022-09-07T21:38:40.562979+0000","last_unstale":"2022-09-07T21:39:46.790103+0000","last_undegraded":"2022-09-07T21:39:46.790103+0000","last_fullsized":"2022-09-07T21:39:46.790103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:21:46.453495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:47.793099+0000","last_change":"2022-09-07T21:39:47.793099+0000","last_active":"2022-09-07T21:39:47.793099+0000","last_peered":"2022-09-07T21:39:47.793099+0000","last_clean":"2022-09-07T21:39:47.793099+0000","last_became_active":"2022-09-07T21:39:47.792757+0000","last_became_peered":"2022-09-07T21:39:47.792757+0000","last_unstale":"2022-09-07T21:39:47.793099+0000","last_undegraded":"2022-09-07T21:39:47.793099+0000","last_fullsized":"2022-09-07T21:39:47.793099+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:40:12.701597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793295+0000","last_change":"2022-09-07T21:38:40.576602+0000","last_active":"2022-09-07T21:39:46.793295+0000","last_peered":"2022-09-07T21:39:46.793295+0000","last_clean":"2022-09-07T21:39:46.793295+0000","last_became_active":"2022-09-07T21:38:40.576296+0000","last_became_peered":"2022-09-07T21:38:40.576296+0000","last_unstale":"2022-09-07T21:39:46.793295+0000","last_undegraded":"2022-09-07T21:39:46.793295+0000","last_fullsized":"2022-09-07T21:39:46.793295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:11:43.175430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784035+0000","last_change":"2022-09-07T21:38:40.584214+0000","last_active":"2022-09-07T21:39:46.784035+0000","last_peered":"2022-09-07T21:39:46.784035+0000","last_clean":"2022-09-07T21:39:46.784035+0000","last_became_active":"2022-09-07T21:38:40.583908+0000","last_became_peered":"2022-09-07T21:38:40.583908+0000","last_unstale":"2022-09-07T21:39:46.784035+0000","last_undegraded":"2022-09-07T21:39:46.784035+0000","last_fullsized":"2022-09-07T21:39:46.784035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:40:54.023025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788732+0000","last_change":"2022-09-07T21:38:40.571163+0000","last_active":"2022-09-07T21:39:46.788732+0000","last_peered":"2022-09-07T21:39:46.788732+0000","last_clean":"2022-09-07T21:39:46.788732+0000","last_became_active":"2022-09-07T21:38:40.570802+0000","last_became_peered":"2022-09-07T21:38:40.570802+0000","last_unstale":"2022-09-07T21:39:46.788732+0000","last_undegraded":"2022-09-07T21:39:46.788732+0000","last_fullsized":"2022-09-07T21:39:46.788732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:03:14.067136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.792173+0000","last_change":"2022-09-07T21:38:40.561957+0000","last_active":"2022-09-07T21:39:46.792173+0000","last_peered":"2022-09-07T21:39:46.792173+0000","last_clean":"2022-09-07T21:39:46.792173+0000","last_became_active":"2022-09-07T21:38:40.561600+0000","last_became_peered":"2022-09-07T21:38:40.561600+0000","last_unstale":"2022-09-07T21:39:46.792173+0000","last_undegraded":"2022-09-07T21:39:46.792173+0000","last_fullsized":"2022-09-07T21:39:46.792173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:28:37.479711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793515+0000","last_change":"2022-09-07T21:38:40.562779+0000","last_active":"2022-09-07T21:39:46.793515+0000","last_peered":"2022-09-07T21:39:46.793515+0000","last_clean":"2022-09-07T21:39:46.793515+0000","last_became_active":"2022-09-07T21:38:40.562472+0000","last_became_peered":"2022-09-07T21:38:40.562472+0000","last_unstale":"2022-09-07T21:39:46.793515+0000","last_undegraded":"2022-09-07T21:39:46.793515+0000","last_fullsized":"2022-09-07T21:39:46.793515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:13:22.395242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.785978+0000","last_change":"2022-09-07T21:38:40.570257+0000","last_active":"2022-09-07T21:39:46.785978+0000","last_peered":"2022-09-07T21:39:46.785978+0000","last_clean":"2022-09-07T21:39:46.785978+0000","last_became_active":"2022-09-07T21:38:40.569930+0000","last_became_peered":"2022-09-07T21:38:40.569930+0000","last_unstale":"2022-09-07T21:39:46.785978+0000","last_undegraded":"2022-09-07T21:39:46.785978+0000","last_fullsized":"2022-09-07T21:39:46.785978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:52:03.810386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790960+0000","last_change":"2022-09-07T21:38:40.578796+0000","last_active":"2022-09-07T21:39:46.790960+0000","last_peered":"2022-09-07T21:39:46.790960+0000","last_clean":"2022-09-07T21:39:46.790960+0000","last_became_active":"2022-09-07T21:38:40.578492+0000","last_became_peered":"2022-09-07T21:38:40.578492+0000","last_unstale":"2022-09-07T21:39:46.790960+0000","last_undegraded":"2022-09-07T21:39:46.790960+0000","last_fullsized":"2022-09-07T21:39:46.790960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:49:50.549224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784727+0000","last_change":"2022-09-07T21:38:40.571320+0000","last_active":"2022-09-07T21:39:46.784727+0000","last_peered":"2022-09-07T21:39:46.784727+0000","last_clean":"2022-09-07T21:39:46.784727+0000","last_became_active":"2022-09-07T21:38:40.570401+0000","last_became_peered":"2022-09-07T21:38:40.570401+0000","last_unstale":"2022-09-07T21:39:46.784727+0000","last_undegraded":"2022-09-07T21:39:46.784727+0000","last_fullsized":"2022-09-07T21:39:46.784727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:24:33.254833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788295+0000","last_change":"2022-09-07T21:38:40.571386+0000","last_active":"2022-09-07T21:39:46.788295+0000","last_peered":"2022-09-07T21:39:46.788295+0000","last_clean":"2022-09-07T21:39:46.788295+0000","last_became_active":"2022-09-07T21:38:40.571130+0000","last_became_peered":"2022-09-07T21:38:40.571130+0000","last_unstale":"2022-09-07T21:39:46.788295+0000","last_undegraded":"2022-09-07T21:39:46.788295+0000","last_fullsized":"2022-09-07T21:39:46.788295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:57:47.083547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788559+0000","last_change":"2022-09-07T21:38:40.584706+0000","last_active":"2022-09-07T21:39:46.788559+0000","last_peered":"2022-09-07T21:39:46.788559+0000","last_clean":"2022-09-07T21:39:46.788559+0000","last_became_active":"2022-09-07T21:38:40.584193+0000","last_became_peered":"2022-09-07T21:38:40.584193+0000","last_unstale":"2022-09-07T21:39:46.788559+0000","last_undegraded":"2022-09-07T21:39:46.788559+0000","last_fullsized":"2022-09-07T21:39:46.788559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:10:32.597608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790426+0000","last_change":"2022-09-07T21:38:40.569545+0000","last_active":"2022-09-07T21:39:46.790426+0000","last_peered":"2022-09-07T21:39:46.790426+0000","last_clean":"2022-09-07T21:39:46.790426+0000","last_became_active":"2022-09-07T21:38:40.569218+0000","last_became_peered":"2022-09-07T21:38:40.569218+0000","last_unstale":"2022-09-07T21:39:46.790426+0000","last_undegraded":"2022-09-07T21:39:46.790426+0000","last_fullsized":"2022-09-07T21:39:46.790426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:40:39.949203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":219246,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65117,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":317,"num_read_kb":239,"num_write":158,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1533,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":195351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383709,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6100,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739052,"statfs":{"total":95995035648,"available":95988789248,"internally_reserved":0,"allocated":995328,"data_stored":498927,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58699999999999997,"15min":0.58799999999999997},"min":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60599999999999998,"15min":0.61099999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.755,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.55000000000000004}]},{"osd":1,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.625,"15min":0.61599999999999999},"min":{"1min":0.45000000000000001,"5min":0.436,"15min":0.436},"max":{"1min":0.875,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.61199999999999999,"15min":0.59799999999999998},"min":{"1min":0.441,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.67600000000000005}]},{"osd":2,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.498,"15min":0.49099999999999999},"min":{"1min":0.376,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.50600000000000001,"15min":0.50900000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72999999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.43099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":4,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.64600000000000002,"15min":0.64100000000000001},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.67300000000000004,"15min":0.65900000000000003},"min":{"1min":0.49099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.64600000000000002}]},{"osd":5,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.61299999999999999,"15min":0.61499999999999999},"min":{"1min":0.46100000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80700000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.38500000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80800000000000005,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.53300000000000003}]},{"osd":6,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.49199999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.749,"5min":0.80900000000000005,"15min":0.81899999999999995},"min":{"1min":0.52700000000000002,"5min":0.5,"15min":0.5},"max":{"1min":0.93799999999999994,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.621}]},{"osd":7,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.75700000000000001,"15min":0.755},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.97199999999999998,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.79900000000000004,"15min":0.79200000000000004},"min":{"1min":0.52200000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":0.70599999999999996}]},{"osd":8,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.66000000000000003}]},{"osd":9,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81799999999999995,"15min":0.81999999999999995},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.98499999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.83399999999999996,"15min":0.82399999999999995},"min":{"1min":0.65500000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":0.78300000000000003}]},{"osd":10,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.83199999999999996,"15min":0.81999999999999995},"min":{"1min":0.63,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.54800000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.052,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.72999999999999998}]}]},{"osd":10,"up_from":72,"seq":309237645345,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6560,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738592,"statfs":{"total":95995035648,"available":95988318208,"internally_reserved":0,"allocated":1138688,"data_stored":1147405,"data_compressed":54425,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:44 2022","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.68300000000000005}]},{"osd":1,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.46100000000000002,"15min":0.46500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.497},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.52000000000000002}]},{"osd":2,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59199999999999997,"15min":0.58399999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.59399999999999997,"15min":0.58399999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70199999999999996}]},{"osd":3,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.64900000000000002,"15min":0.64300000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.69499999999999995,"15min":0.67500000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":3.794,"5min":3.794,"15min":3.794},"last":0.66900000000000004}]},{"osd":4,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.57699999999999996,"15min":0.58399999999999996},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78400000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.626},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.877,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.61599999999999999}]},{"osd":5,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67800000000000005,"15min":0.67600000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69699999999999995,"15min":0.69599999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.81999999999999995}]},{"osd":6,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.80300000000000005,"15min":0.78900000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":3.8900000000000001,"5min":3.8900000000000001,"15min":3.8900000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.79200000000000004,"15min":0.79000000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.88700000000000001}]},{"osd":7,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.42399999999999999}]},{"osd":8,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.75800000000000001,"15min":0.75900000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.98199999999999998,"5min":0.997,"15min":0.997},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.79700000000000004,"15min":0.79900000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.83899999999999997}]},{"osd":9,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.85899999999999999,"15min":0.83299999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":4.9989999999999997,"5min":4.9989999999999997,"15min":4.9989999999999997},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84999999999999998,"15min":0.85299999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.86799999999999999}]},{"osd":11,"last update":"Wed Sep 7 21:39:57 2022","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.78000000000000003,"15min":0.78400000000000003},"min":{"1min":0.51200000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0069999999999999,"5min":1.103,"15min":1.103},"last":0.878},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80600000000000005,"15min":0.80700000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98899999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.91100000000000003}]}]},{"osd":9,"up_from":67,"seq":287762808871,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6312,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738840,"statfs":{"total":95995035648,"available":95988572160,"internally_reserved":0,"allocated":1015808,"data_stored":503259,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81999999999999995,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.67300000000000004,"15min":0.67900000000000005},"min":{"1min":0.47499999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.012,"5min":1.175,"15min":1.175},"last":0.71099999999999997}]},{"osd":1,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.77000000000000002,"15min":0.77600000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.236,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.84799999999999998,"15min":0.85399999999999998},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.751,"15min":0.75700000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.999,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.86299999999999999,"15min":0.88600000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0029999999999999,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.81299999999999994}]},{"osd":3,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.81000000000000005,"15min":0.81599999999999995},"min":{"1min":0.499,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0980000000000001,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.79300000000000004,"15min":0.80600000000000005},"min":{"1min":0.48999999999999999,"5min":0.434,"15min":0.434},"max":{"1min":1.04,"5min":1.613,"15min":1.613},"last":0.51300000000000001}]},{"osd":4,"last update":"Wed Sep 7 21:39:46 2022","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.75},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.734,"5min":1.734,"15min":1.734},"last":0.79600000000000004}]},{"osd":5,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.90400000000000003,"15min":0.91700000000000004},"min":{"1min":0.58499999999999996,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0609999999999999,"5min":1.55,"15min":1.55},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.88100000000000001,"15min":0.89200000000000002},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.083,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.73599999999999999}]},{"osd":6,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":0.81200000000000006}]},{"osd":7,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.0609999999999999,"15min":1.0529999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":5.6529999999999996,"5min":5.6529999999999996,"15min":5.6529999999999996},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.98099999999999998,"15min":0.98699999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.337,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":0.67900000000000005}]},{"osd":8,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.0169999999999999,"15min":1.026},"min":{"1min":0.66400000000000003,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.3010000000000002,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.052,"15min":1.0720000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.1479999999999999,"5min":1.76,"15min":1.76},"last":0.88}]},{"osd":10,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.97299999999999998,"15min":0.98499999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.224,"5min":1.768,"15min":1.768},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.071,"5min":1.044,"15min":1.04},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"last":0.92500000000000004}]},{"osd":11,"last update":"Wed Sep 7 21:39:46 2022","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.3180000000000001,"5min":2.3180000000000001,"15min":2.3180000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.89000000000000001}]}]},{"osd":8,"up_from":59,"seq":253403070510,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6432,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738720,"statfs":{"total":95995035648,"available":95988449280,"internally_reserved":0,"allocated":1007616,"data_stored":498517,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.63100000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78600000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.58499999999999996,"15min":0.59299999999999997},"min":{"1min":0.42399999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.75600000000000001,"5min":0.876,"15min":0.876},"last":1.383}]},{"osd":1,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.622,"15min":0.628},"min":{"1min":0.39900000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.623,"15min":0.625},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.89700000000000002,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.63900000000000001}]},{"osd":2,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48499999999999999,"15min":0.495},"min":{"1min":0.34100000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70699999999999996,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.439},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.50700000000000001,"15min":0.51900000000000002},"min":{"1min":0.314,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68799999999999994,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.47999999999999998}]},{"osd":3,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.68899999999999995,"15min":0.65700000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":4.4969999999999999,"5min":4.4969999999999999,"15min":4.4969999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.71399999999999997,"15min":0.68500000000000005},"min":{"1min":0.42099999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":5.4059999999999997,"5min":5.4059999999999997,"15min":5.4059999999999997},"last":0.68799999999999994}]},{"osd":4,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.51300000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.66200000000000003,"15min":0.66400000000000003},"min":{"1min":0.52600000000000002,"5min":0.433,"15min":0.433},"max":{"1min":0.91100000000000003,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.624}]},{"osd":5,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.628,"15min":0.65100000000000002},"min":{"1min":0.34999999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.84499999999999997,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60699999999999998,"15min":0.61699999999999999},"min":{"1min":0.378,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.85599999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.46400000000000002}]},{"osd":6,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.78000000000000003,"15min":0.75800000000000001},"min":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":4.7300000000000004,"5min":4.7300000000000004,"15min":4.7300000000000004},"last":1.5940000000000001},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.80200000000000005,"15min":0.79300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.9729999999999999,"5min":2.9729999999999999,"15min":2.9729999999999999},"last":0.67700000000000005}]},{"osd":7,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.745,"15min":0.75900000000000001},"min":{"1min":0.437,"5min":0.378,"15min":0.378},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72699999999999998,"15min":0.73099999999999998},"min":{"1min":0.497,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.97499999999999998,"5min":1,"15min":1},"last":0.70899999999999996}]},{"osd":9,"last update":"Wed Sep 7 21:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.755,"15min":0.748},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.75900000000000001,"15min":0.752},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.79200000000000004}]},{"osd":10,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.73299999999999998,"15min":0.71699999999999997},"min":{"1min":0.51700000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.755},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.749,"15min":0.73899999999999999},"min":{"1min":0.52300000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0169999999999999,"5min":1.02,"15min":1.02},"last":0.73899999999999999}]},{"osd":11,"last update":"Wed Sep 7 21:39:42 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.82399999999999995}]}]},{"osd":7,"up_from":51,"seq":219043332147,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6612,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738540,"statfs":{"total":95995035648,"available":95988264960,"internally_reserved":0,"allocated":1060864,"data_stored":1095291,"data_compressed":54425,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58499999999999996,"15min":0.57299999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.54200000000000004}]},{"osd":1,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.439,"15min":0.433},"min":{"1min":0.33000000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.45300000000000001,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.42699999999999999}]},{"osd":2,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83499999999999996,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.67100000000000004,"15min":0.627},"min":{"1min":0.40899999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.4580000000000002,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":0.55700000000000005}]},{"osd":3,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.629,"15min":0.60899999999999999},"min":{"1min":0.49099999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0149999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64100000000000001,"15min":0.627},"min":{"1min":0.42799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.58999999999999997}]},{"osd":4,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.35999999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52700000000000002,"15min":0.52800000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79400000000000004,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.47099999999999997}]},{"osd":5,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.66000000000000003,"15min":0.66400000000000003},"min":{"1min":0.48099999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.85399999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.497},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.45400000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.873,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.627}]},{"osd":6,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73599999999999999,"15min":0.73799999999999999},"min":{"1min":0.50600000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0049999999999999,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.71299999999999997,"15min":0.70099999999999996},"min":{"1min":0.55300000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.52200000000000002}]},{"osd":8,"last update":"Wed Sep 7 21:39:27 2022","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71299999999999997,"15min":0.71699999999999997},"min":{"1min":0.46600000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.86099999999999999,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.71899999999999997,"15min":0.72499999999999998},"min":{"1min":0.51300000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.83199999999999996,"5min":1.123,"15min":1.123},"last":0.53700000000000003}]},{"osd":9,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80000000000000004,"15min":0.80300000000000005},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.99399999999999999,"5min":1.036,"15min":1.036},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.83299999999999996,"15min":0.84099999999999997},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0980000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.67100000000000004}]},{"osd":10,"last update":"Wed Sep 7 21:39:32 2022","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.73199999999999998,"15min":0.73999999999999999},"min":{"1min":0.51100000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.92000000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.93600000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.755}]},{"osd":11,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.746}]}]},{"osd":6,"up_from":46,"seq":197568495673,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6416,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738736,"statfs":{"total":95995035648,"available":95988465664,"internally_reserved":0,"allocated":991232,"data_stored":494355,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56000000000000005,"15min":0.54200000000000004},"min":{"1min":0.40999999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.40000000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.78300000000000003,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.45900000000000002}]},{"osd":1,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.65500000000000003,"15min":0.66100000000000003},"min":{"1min":0.48299999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86099999999999999,"5min":1.127,"15min":1.127},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67100000000000004,"15min":0.66100000000000003},"min":{"1min":0.41899999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.94099999999999995,"5min":1.159,"15min":1.159},"last":0.66200000000000003}]},{"osd":2,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.71899999999999997,"15min":0.71499999999999997},"min":{"1min":0.496,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.97099999999999997,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.034,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.70799999999999996}]},{"osd":3,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68500000000000005,"15min":0.65900000000000003},"min":{"1min":0.47499999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.90500000000000003,"5min":1.151,"15min":1.151},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68799999999999994,"15min":0.68999999999999995},"min":{"1min":0.46700000000000003,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.879,"5min":1.153,"15min":1.153},"last":0.89100000000000001}]},{"osd":4,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.746,"15min":0.754},"min":{"1min":0.51900000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.98199999999999998,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.54800000000000004,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.90500000000000003,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.81599999999999995}]},{"osd":5,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.77300000000000002,"15min":0.76100000000000001},"min":{"1min":0.61799999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96899999999999997,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.81799999999999995,"15min":0.82999999999999996},"min":{"1min":0.49099999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.99399999999999999,"5min":1.341,"15min":1.341},"last":0.79200000000000004}]},{"osd":7,"last update":"Wed Sep 7 21:39:07 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.81000000000000005,"15min":0.80600000000000005},"min":{"1min":0.57099999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.044,"5min":1.212,"15min":1.212},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.82999999999999996,"15min":0.82199999999999995},"min":{"1min":0.64800000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.98699999999999999,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.90900000000000003}]},{"osd":8,"last update":"Wed Sep 7 21:39:28 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84599999999999997,"15min":0.83599999999999997},"min":{"1min":0.63800000000000001,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.016,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.871,"15min":0.85599999999999998},"min":{"1min":0.69099999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.026,"5min":1.268,"15min":1.268},"last":0.83099999999999996}]},{"osd":9,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.96799999999999997,"15min":0.97399999999999998},"min":{"1min":0.77700000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.133,"5min":1.254,"15min":1.254},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.877,"5min":0.88800000000000001,"15min":0.89000000000000001},"min":{"1min":0.65300000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1439999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.0549999999999999}]},{"osd":10,"last update":"Wed Sep 7 21:39:28 2022","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.93100000000000005,"15min":0.94599999999999995},"min":{"1min":0.70999999999999996,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":0.97899999999999998,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.98699999999999999,"15min":0.995},"min":{"1min":0.77500000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.0820000000000001,"5min":1.218,"15min":1.218},"last":0.94599999999999995}]},{"osd":11,"last update":"Wed Sep 7 21:39:44 2022","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.873},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.92300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7508,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737644,"statfs":{"total":95995035648,"available":95987347456,"internally_reserved":0,"allocated":1060864,"data_stored":555696,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:13 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55300000000000005,"15min":0.51500000000000001},"min":{"1min":0.38100000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.78200000000000003,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55800000000000005,"15min":0.52600000000000002},"min":{"1min":0.39100000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.73399999999999999,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.51600000000000001}]},{"osd":2,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.57999999999999996,"15min":0.53100000000000003},"min":{"1min":0.38200000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83199999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.56599999999999995,"15min":0.52300000000000002},"min":{"1min":0.39400000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.61199999999999999}]},{"osd":3,"last update":"Wed Sep 7 21:39:13 2022","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.57999999999999996,"15min":0.55400000000000005},"min":{"1min":0.41999999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.83799999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59999999999999998,"15min":0.58599999999999997},"min":{"1min":0.42999999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79400000000000004,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.58899999999999997}]},{"osd":4,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46300000000000002,"15min":0.44700000000000001},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72999999999999998,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.498,"15min":0.46800000000000003},"min":{"1min":0.309,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81399999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.443}]},{"osd":5,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.64400000000000002,"15min":0.61099999999999999},"min":{"1min":0.38900000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.63300000000000001,"15min":0.59799999999999998},"min":{"1min":0.51400000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82099999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.94199999999999995}]},{"osd":6,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68200000000000005,"15min":0.66500000000000004},"min":{"1min":0.49299999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85799999999999998,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.69699999999999995,"15min":0.68700000000000006},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.873,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.73199999999999998}]},{"osd":7,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.622,"15min":0.622},"min":{"1min":0.45800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82899999999999996,"5min":1.232,"15min":1.232},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.66100000000000003},"min":{"1min":0.40400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.874,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.34999999999999998}]},{"osd":8,"last update":"Wed Sep 7 21:39:32 2022","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.71799999999999997,"15min":0.70199999999999996},"min":{"1min":0.54900000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.96199999999999997,"5min":1.091,"15min":1.091},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.71499999999999997,"15min":0.68500000000000005},"min":{"1min":0.56000000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94499999999999995,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.84799999999999998}]},{"osd":9,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.57599999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.90200000000000002,"5min":1.032,"15min":1.032},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.77700000000000002,"15min":0.77900000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.91600000000000004,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.625}]},{"osd":10,"last update":"Wed Sep 7 21:39:31 2022","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.79000000000000004,"15min":0.80000000000000004},"min":{"1min":0.49099999999999999,"5min":0.437,"15min":0.437},"max":{"1min":0.92300000000000004,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.78100000000000003,"15min":0.78500000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.98699999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.70199999999999996}]},{"osd":11,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.89700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7400,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737752,"statfs":{"total":95995035648,"available":95987458048,"internally_reserved":0,"allocated":1015808,"data_stored":495711,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 7 21:39:13 2022","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.54500000000000004},"min":{"1min":0.5,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76600000000000001,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.623,"15min":0.56100000000000005},"min":{"1min":0.41499999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.2509999999999999,"5min":1.26,"15min":1.26},"last":0.86199999999999999}]},{"osd":2,"last update":"Wed Sep 7 21:39:37 2022","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":0.76200000000000001,"15min":0.65000000000000002},"min":{"1min":0.44400000000000001,"5min":0.316,"15min":0.316},"max":{"1min":12.795,"5min":12.795,"15min":12.795},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.66900000000000004,"15min":0.621},"min":{"1min":0.52500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.92200000000000004,"5min":1.075,"15min":1.075},"last":0.61899999999999999}]},{"osd":3,"last update":"Wed Sep 7 21:39:43 2022","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.56299999999999994}]},{"osd":4,"last update":"Wed Sep 7 21:39:43 2022","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.71799999999999997,"15min":0.66400000000000003},"min":{"1min":0.48699999999999999,"5min":0.438,"15min":0.438},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.72099999999999997,"15min":0.66300000000000003},"min":{"1min":0.50700000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93999999999999995,"5min":1.109,"15min":1.109},"last":0.58099999999999996}]},{"osd":5,"last update":"Wed Sep 7 21:39:07 2022","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":0.877,"15min":0.75700000000000001},"min":{"1min":0.68300000000000005,"5min":0.5,"15min":0.5},"max":{"1min":12.776999999999999,"5min":12.776999999999999,"15min":12.776999999999999},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.68700000000000006,"15min":0.66500000000000004},"min":{"1min":0.47199999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95699999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.81200000000000006}]},{"osd":6,"last update":"Wed Sep 7 21:39:43 2022","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.73099999999999998,"15min":0.70899999999999996},"min":{"1min":0.48599999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.78500000000000003,"15min":0.76900000000000002},"min":{"1min":0.58599999999999997,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.94899999999999995}]},{"osd":7,"last update":"Wed Sep 7 21:39:47 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":12.808,"5min":12.808,"15min":12.808},"last":0.89600000000000002}]},{"osd":8,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.89000000000000001,"15min":0.89200000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0349999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.84699999999999998,"15min":0.84399999999999997},"min":{"1min":0.65100000000000002,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0349999999999999,"5min":1.143,"15min":1.143},"last":0.88}]},{"osd":9,"last update":"Wed Sep 7 21:39:03 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.93700000000000006,"15min":0.95099999999999996},"min":{"1min":0.625,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0509999999999999,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.89700000000000002,"15min":0.90500000000000003},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.91500000000000004}]},{"osd":10,"last update":"Wed Sep 7 21:39:37 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.95099999999999996,"15min":0.95899999999999996},"min":{"1min":0.73499999999999999,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.2150000000000001,"5min":1.413,"15min":1.413},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.0960000000000001,"15min":1.0229999999999999},"min":{"1min":0.73199999999999998,"5min":0.504,"15min":0.504},"max":{"1min":12.837999999999999,"5min":12.837999999999999,"15min":12.837999999999999},"last":0.84799999999999998}]},{"osd":11,"last update":"Wed Sep 7 21:39:54 2022","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.89000000000000001,"15min":0.872},"min":{"1min":0.77000000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.76700000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.96999999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7584,"kb_used_data":1048,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737568,"statfs":{"total":95995035648,"available":95987269632,"internally_reserved":0,"allocated":1073152,"data_stored":555842,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:42 2022","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.55900000000000005,"15min":0.52100000000000002},"min":{"1min":0.41499999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.58099999999999996,"15min":0.53500000000000003},"min":{"1min":0.38900000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.6899999999999999,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":0.58599999999999997}]},{"osd":1,"last update":"Wed Sep 7 21:39:42 2022","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.54200000000000004,"15min":0.51500000000000001},"min":{"1min":0.36699999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.58299999999999996,"15min":0.51100000000000001},"min":{"1min":0.47599999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.82399999999999995}]},{"osd":3,"last update":"Wed Sep 7 21:39:12 2022","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.65700000000000003,"15min":0.57899999999999996},"min":{"1min":0.40300000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.95799999999999996,"5min":7.649,"15min":7.649},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.58199999999999996,"15min":0.54200000000000004},"min":{"1min":0.442,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.79900000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.64100000000000001}]},{"osd":4,"last update":"Wed Sep 7 21:39:42 2022","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.70099999999999996}]},{"osd":5,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.47499999999999998,"15min":0.437},"min":{"1min":0.312,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.53600000000000003,"15min":0.49099999999999999},"min":{"1min":0.35599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.45000000000000001}]},{"osd":6,"last update":"Wed Sep 7 21:39:42 2022","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.72899999999999998,"15min":0.70299999999999996},"min":{"1min":0.63100000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.72099999999999997,"15min":0.69699999999999995},"min":{"1min":0.60099999999999998,"5min":0.498,"15min":0.498},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.73799999999999999}]},{"osd":7,"last update":"Wed Sep 7 21:39:48 2022","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.85699999999999998}]},{"osd":8,"last update":"Wed Sep 7 21:39:31 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.65900000000000003,"15min":0.65100000000000002},"min":{"1min":0.48199999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.93600000000000005,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.65800000000000003,"15min":0.65400000000000003},"min":{"1min":0.497,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.65000000000000002}]},{"osd":9,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.77700000000000002,"15min":0.77000000000000002},"min":{"1min":0.54800000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76100000000000001,"15min":0.755},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.79700000000000004}]},{"osd":10,"last update":"Wed Sep 7 21:39:31 2022","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.75700000000000001,"15min":0.747},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.81299999999999994,"15min":0.79400000000000004},"min":{"1min":0.67400000000000004,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.89500000000000002}]},{"osd":11,"last update":"Wed Sep 7 21:39:42 2022","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.754}]}]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6684,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738468,"statfs":{"total":95995035648,"available":95988191232,"internally_reserved":0,"allocated":1003520,"data_stored":494549,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:14 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.495,"15min":0.44700000000000001},"min":{"1min":0.375,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67500000000000004,"5min":1.167,"15min":1.167},"last":1.3680000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.499,"15min":0.46100000000000002},"min":{"1min":0.36099999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.80000000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.72399999999999998}]},{"osd":1,"last update":"Wed Sep 7 21:39:14 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.60299999999999998,"15min":0.57699999999999996},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.67900000000000005,"15min":0.60999999999999999},"min":{"1min":0.47399999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.92200000000000004,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.59099999999999997}]},{"osd":2,"last update":"Wed Sep 7 21:39:14 2022","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.68700000000000006,"15min":0.64200000000000002},"min":{"1min":0.48299999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94699999999999995,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.66400000000000003,"15min":0.627},"min":{"1min":0.42399999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.89800000000000002,"5min":1.302,"15min":1.302},"last":0.73499999999999999}]},{"osd":4,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70599999999999996,"15min":0.64900000000000002},"min":{"1min":0.54100000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90600000000000003,"5min":1.286,"15min":1.286},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67300000000000004,"15min":0.63},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.84399999999999997,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.68100000000000005}]},{"osd":5,"last update":"Wed Sep 7 21:39:09 2022","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":0.80900000000000005,"15min":0.70899999999999996},"min":{"1min":0.48899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":11.808,"5min":11.808,"15min":11.808},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.77600000000000002,"15min":0.73799999999999999},"min":{"1min":0.501,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.97399999999999998,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.67500000000000004}]},{"osd":6,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.80500000000000005,"15min":0.78300000000000003},"min":{"1min":0.629,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.96199999999999997,"5min":1.399,"15min":1.399},"last":1.3049999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.74399999999999999,"15min":0.73399999999999999},"min":{"1min":0.51200000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.995,"5min":1.157,"15min":1.157},"last":1.5169999999999999}]},{"osd":7,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.73199999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.034,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.622},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.874,"15min":0.86299999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.0169999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.70699999999999996}]},{"osd":8,"last update":"Wed Sep 7 21:38:30 2022","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82999999999999996,"15min":0.83099999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81299999999999994,"15min":0.81200000000000006},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.78400000000000003}]},{"osd":9,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.97799999999999998,"15min":0.99199999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.093,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":1.409},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.90400000000000003,"15min":0.91800000000000004},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1539999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":1.383}]},{"osd":10,"last update":"Wed Sep 7 21:39:33 2022","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.88500000000000001,"15min":0.89900000000000002},"min":{"1min":0.60399999999999998,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.1359999999999999,"5min":1.228,"15min":1.228},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.98299999999999998,"15min":0.99299999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.1160000000000001,"5min":1.486,"15min":1.486},"last":0.745}]},{"osd":11,"last update":"Wed Sep 7 21:39:43 2022","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":1.0569999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.3380000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":1011712,"data_stored":500317,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:37:36 2022","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54400000000000004,"15min":0.52900000000000003},"min":{"1min":0.41199999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78100000000000003,"5min":0.879,"15min":0.879},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.52700000000000002,"15min":0.503},"min":{"1min":0.40699999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.35199999999999998}]},{"osd":1,"last update":"Wed Sep 7 21:39:41 2022","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.40500000000000003,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.41699999999999998,"15min":0.38100000000000001},"min":{"1min":0.34699999999999998,"5min":0.216,"15min":0.216},"max":{"1min":0.69299999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.40999999999999998}]},{"osd":2,"last update":"Wed Sep 7 21:39:41 2022","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56899999999999995,"15min":0.54700000000000004},"min":{"1min":0.42099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.59299999999999997,"15min":0.56000000000000005},"min":{"1min":0.55200000000000005,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.89100000000000001,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.65100000000000002}]},{"osd":3,"last update":"Wed Sep 7 21:39:41 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.61299999999999999,"15min":0.56999999999999995},"min":{"1min":0.42899999999999999,"5min":0.37,"15min":0.37},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.61499999999999999,"15min":0.59599999999999997},"min":{"1min":0.44600000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.54200000000000004}]},{"osd":5,"last update":"Wed Sep 7 21:39:10 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57199999999999995,"15min":0.55400000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.60899999999999999,"15min":0.58299999999999996},"min":{"1min":0.45500000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91200000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.70499999999999996}]},{"osd":6,"last update":"Wed Sep 7 21:39:41 2022","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.66400000000000003,"15min":0.64800000000000002},"min":{"1min":0.499,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.67500000000000004,"15min":0.65300000000000002},"min":{"1min":0.46400000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":0.57099999999999995}]},{"osd":7,"last update":"Wed Sep 7 21:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61699999999999999,"15min":0.61599999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.872,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76300000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.57699999999999996}]},{"osd":8,"last update":"Wed Sep 7 21:39:32 2022","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73399999999999999,"15min":0.73099999999999998},"min":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93300000000000005,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72099999999999997,"15min":0.72799999999999998},"min":{"1min":0.50800000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.878,"5min":1.198,"15min":1.198},"last":0.68600000000000005}]},{"osd":9,"last update":"Wed Sep 7 21:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72099999999999997,"15min":0.72299999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.78100000000000003,"15min":0.78600000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.048,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.877}]},{"osd":10,"last update":"Wed Sep 7 21:39:32 2022","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.74199999999999999,"15min":0.745},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.99199999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.67900000000000005,"15min":0.67100000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.67200000000000004}]},{"osd":11,"last update":"Wed Sep 7 21:39:39 2022","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.89000000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6680,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738472,"statfs":{"total":95995035648,"available":95988195328,"internally_reserved":0,"allocated":1064960,"data_stored":1090211,"data_compressed":54425,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77800000000000002,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.56599999999999995,"15min":0.55600000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88400000000000001,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.61699999999999999}]},{"osd":1,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.56399999999999995,"15min":0.57699999999999996},"min":{"1min":0.38300000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.77500000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.39700000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.56799999999999995}]},{"osd":2,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.42999999999999999,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.28599999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.70899999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.45400000000000001}]},{"osd":3,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80000000000000004,"5min":1.238,"15min":1.238},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61099999999999999,"15min":0.59199999999999997},"min":{"1min":0.437,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.84799999999999998,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.747}]},{"osd":4,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60999999999999999,"15min":0.61599999999999999},"min":{"1min":0.40400000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.85999999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.39200000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75700000000000001,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.503}]},{"osd":6,"last update":"Wed Sep 7 21:39:43 2022","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.66300000000000003,"15min":0.63100000000000001},"min":{"1min":0.442,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92400000000000004,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.68400000000000005,"15min":0.66400000000000003},"min":{"1min":0.502,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":0.78900000000000003}]},{"osd":7,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.64200000000000002,"15min":0.65100000000000002},"min":{"1min":0.46400000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.94699999999999995,"5min":1.002,"15min":1.002},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66600000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.67900000000000005}]},{"osd":8,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.64200000000000002,"15min":0.66000000000000003},"min":{"1min":0.38800000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85199999999999998,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68500000000000005,"15min":0.69699999999999995},"min":{"1min":0.53800000000000003,"5min":0.372,"15min":0.372},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.66800000000000004}]},{"osd":9,"last update":"Wed Sep 7 21:39:00 2022","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.79800000000000004,"15min":0.80900000000000005},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":0.90000000000000002,"5min":0.998,"15min":0.998},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.78100000000000003,"15min":0.78900000000000003},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.89600000000000002,"5min":1.042,"15min":1.042},"last":0.879}]},{"osd":10,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.72799999999999998,"15min":0.71799999999999997},"min":{"1min":0.63,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.996,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.61899999999999999,"5min":0.499,"15min":0.499},"max":{"1min":0.97199999999999998,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.83299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35171,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":61431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":61577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-07T21:40:03.499 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph pg dump --format=json 2022-09-07T21:40:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:03 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:40:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:03 smithi059 ceph-mon[97450]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 80 B/s, 0 keys/s, 0 objects/s recovering 2022-09-07T21:40:03.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:03 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:40:03.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:03 smithi047 ceph-mon[90087]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 80 B/s, 0 keys/s, 0 objects/s recovering 2022-09-07T21:40:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:03 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' 2022-09-07T21:40:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:03 smithi055 ceph-mon[97033]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 80 B/s, 0 keys/s, 0 objects/s recovering 2022-09-07T21:40:04.384 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:40:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:04 smithi059 ceph-mon[97450]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:40:05.131 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:04 smithi047 ceph-mon[90087]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:40:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:04 smithi055 ceph-mon[97033]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:40:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:05 smithi059 ceph-mon[97450]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 80 B/s, 0 objects/s recovering 2022-09-07T21:40:06.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:05 smithi047 ceph-mon[90087]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 80 B/s, 0 objects/s recovering 2022-09-07T21:40:06.188 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.185+0000 7f1051d4f700 1 -- 172.21.15.47:0/2497625096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f104c066fd0 msgr2=0x7f104c10a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:06.189 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.185+0000 7f1051d4f700 1 --2- 172.21.15.47:0/2497625096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f104c066fd0 0x7f104c10a4d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1040009750 tx=0x7f1040009a60 comp rx=0 tx=0).stop 2022-09-07T21:40:06.189 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.185+0000 7f1051d4f700 1 -- 172.21.15.47:0/2497625096 shutdown_connections 2022-09-07T21:40:06.189 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.185+0000 7f1051d4f700 1 --2- 172.21.15.47:0/2497625096 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f104c10aa10 0x7f104c10cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:06.189 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.185+0000 7f1051d4f700 1 --2- 172.21.15.47:0/2497625096 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f104c0666c0 0x7f104c066a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:06.190 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.185+0000 7f1051d4f700 1 --2- 172.21.15.47:0/2497625096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f104c066fd0 0x7f104c10a4d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:06.190 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.185+0000 7f1051d4f700 1 -- 172.21.15.47:0/2497625096 >> 172.21.15.47:0/2497625096 conn(0x7f104c073cf0 msgr2=0x7f104c0740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:06.190 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.185+0000 7f1051d4f700 1 -- 172.21.15.47:0/2497625096 shutdown_connections 2022-09-07T21:40:06.190 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.185+0000 7f1051d4f700 1 -- 172.21.15.47:0/2497625096 wait complete. 2022-09-07T21:40:06.192 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.186+0000 7f1051d4f700 1 Processor -- start 2022-09-07T21:40:06.193 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.186+0000 7f1051d4f700 1 -- start start 2022-09-07T21:40:06.193 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.187+0000 7f1051d4f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f104c0666c0 0x7f104c195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:06.193 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.187+0000 7f1051d4f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f104c066fd0 0x7f104c1963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:06.193 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.187+0000 7f1051d4f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f104c10aa10 0x7f104c19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:06.194 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.187+0000 7f1051d4f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104c19aea0 con 0x7f104c10aa10 2022-09-07T21:40:06.194 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.187+0000 7f1051d4f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104c19b010 con 0x7f104c0666c0 2022-09-07T21:40:06.194 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.187+0000 7f1051d4f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104c19b1d0 con 0x7f104c066fd0 2022-09-07T21:40:06.194 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.187+0000 7f104bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f104c10aa10 0x7f104c19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:06.195 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.187+0000 7f104bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f104c10aa10 0x7f104c19a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:35430/0 (socket says 172.21.15.47:35430) 2022-09-07T21:40:06.195 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.187+0000 7f104bfff700 1 -- 172.21.15.47:0/1877480866 learned_addr learned my addr 172.21.15.47:0/1877480866 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:06.195 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.187+0000 7f104b7fe700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f104c0666c0 0x7f104c195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:06.195 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.187+0000 7f104affd700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f104c066fd0 0x7f104c1963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:06.196 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.188+0000 7f104b7fe700 1 -- 172.21.15.47:0/1877480866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f104c066fd0 msgr2=0x7f104c1963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:06.196 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.188+0000 7f104b7fe700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f104c066fd0 0x7f104c1963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:06.196 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.188+0000 7f104b7fe700 1 -- 172.21.15.47:0/1877480866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f104c10aa10 msgr2=0x7f104c19a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:06.196 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.188+0000 7f104b7fe700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f104c10aa10 0x7f104c19a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:06.197 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.188+0000 7f104b7fe700 1 -- 172.21.15.47:0/1877480866 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10400093b0 con 0x7f104c0666c0 2022-09-07T21:40:06.197 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.188+0000 7f104affd700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f104c066fd0 0x7f104c1963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:06.197 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.188+0000 7f104b7fe700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f104c0666c0 0x7f104c195e70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f103c009fd0 tx=0x7f103c00e9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:06.197 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.188+0000 7f1048ff9700 1 -- 172.21.15.47:0/1877480866 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c010900 con 0x7f104c0666c0 2022-09-07T21:40:06.197 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.188+0000 7f1051d4f700 1 -- 172.21.15.47:0/1877480866 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f104c19b4b0 con 0x7f104c0666c0 2022-09-07T21:40:06.198 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.188+0000 7f1048ff9700 1 -- 172.21.15.47:0/1877480866 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f103c018420 con 0x7f104c0666c0 2022-09-07T21:40:06.198 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.188+0000 7f1048ff9700 1 -- 172.21.15.47:0/1877480866 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c017610 con 0x7f104c0666c0 2022-09-07T21:40:06.198 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.189+0000 7f1051d4f700 1 -- 172.21.15.47:0/1877480866 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f104c1a2cf0 con 0x7f104c0666c0 2022-09-07T21:40:06.199 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.190+0000 7f1051d4f700 1 -- 172.21.15.47:0/1877480866 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f104c064ea0 con 0x7f104c0666c0 2022-09-07T21:40:06.200 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.192+0000 7f1048ff9700 1 -- 172.21.15.47:0/1877480866 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f103c00c040 con 0x7f104c0666c0 2022-09-07T21:40:06.200 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.193+0000 7f1048ff9700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1034099050 0x7f103409b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:06.200 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.193+0000 7f1048ff9700 1 -- 172.21.15.47:0/1877480866 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f103c014070 con 0x7f104c0666c0 2022-09-07T21:40:06.200 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.193+0000 7f104affd700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1034099050 0x7f103409b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:06.200 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.194+0000 7f104affd700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1034099050 0x7f103409b500 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f104c197480 tx=0x7f1040005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:06.201 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.195+0000 7f1048ff9700 1 -- 172.21.15.47:0/1877480866 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f103c08ed20 con 0x7f104c0666c0 2022-09-07T21:40:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:05 smithi055 ceph-mon[97033]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 80 B/s, 0 objects/s recovering 2022-09-07T21:40:06.416 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.413+0000 7f1051d4f700 1 -- 172.21.15.47:0/1877480866 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f104c197300 con 0x7f1034099050 2022-09-07T21:40:06.429 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.426+0000 7f1048ff9700 1 -- 172.21.15.47:0/1877480866 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426644 (secure 0 0 0) 0x7f104001e040 con 0x7f1034099050 2022-09-07T21:40:06.438 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:40:06.442 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.433+0000 7f1051d4f700 1 -- 172.21.15.47:0/1877480866 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1034099050 msgr2=0x7f103409b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:06.442 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.433+0000 7f1051d4f700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1034099050 0x7f103409b500 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f104c197480 tx=0x7f1040005fd0 comp rx=0 tx=0).stop 2022-09-07T21:40:06.442 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.433+0000 7f1051d4f700 1 -- 172.21.15.47:0/1877480866 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f104c0666c0 msgr2=0x7f104c195e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:06.443 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.433+0000 7f1051d4f700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f104c0666c0 0x7f104c195e70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f103c009fd0 tx=0x7f103c00e9a0 comp rx=0 tx=0).stop 2022-09-07T21:40:06.443 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.433+0000 7f1051d4f700 1 -- 172.21.15.47:0/1877480866 shutdown_connections 2022-09-07T21:40:06.443 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.433+0000 7f1051d4f700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1034099050 0x7f103409b500 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:06.443 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.433+0000 7f1051d4f700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f104c10aa10 0x7f104c19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:06.444 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.433+0000 7f1051d4f700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f104c0666c0 0x7f104c195e70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:06.444 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.433+0000 7f1051d4f700 1 --2- 172.21.15.47:0/1877480866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f104c066fd0 0x7f104c1963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:06.444 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.433+0000 7f1051d4f700 1 -- 172.21.15.47:0/1877480866 >> 172.21.15.47:0/1877480866 conn(0x7f104c073cf0 msgr2=0x7f104c0fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:06.444 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.433+0000 7f1051d4f700 1 -- 172.21.15.47:0/1877480866 shutdown_connections 2022-09-07T21:40:06.444 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:06.433+0000 7f1051d4f700 1 -- 172.21.15.47:0/1877480866 wait complete. 2022-09-07T21:40:06.445 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2022-09-07T21:40:06.954 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2022-09-07T21:40:04.611361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655731,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":478,"num_read_kb":368,"num_write":341,"num_write_kb":3052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1182269,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80852,"kb_used_data":12148,"kb_used_omap":0,"kb_used_meta":68608,"kb_avail":1124860972,"statfs":{"total":1151940427776,"available":1151857635328,"internally_reserved":0,"allocated":12439552,"data_stored":7930080,"data_compressed":321765,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":0,"internal_metadata":70254592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":38,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003585"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.792064+0000","last_change":"2022-09-07T21:38:40.576826+0000","last_active":"2022-09-07T21:39:46.792064+0000","last_peered":"2022-09-07T21:39:46.792064+0000","last_clean":"2022-09-07T21:39:46.792064+0000","last_became_active":"2022-09-07T21:38:40.576429+0000","last_became_peered":"2022-09-07T21:38:40.576429+0000","last_unstale":"2022-09-07T21:39:46.792064+0000","last_undegraded":"2022-09-07T21:39:46.792064+0000","last_fullsized":"2022-09-07T21:39:46.792064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:40:45.777536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784539+0000","last_change":"2022-09-07T21:38:40.562924+0000","last_active":"2022-09-07T21:39:46.784539+0000","last_peered":"2022-09-07T21:39:46.784539+0000","last_clean":"2022-09-07T21:39:46.784539+0000","last_became_active":"2022-09-07T21:38:40.562509+0000","last_became_peered":"2022-09-07T21:38:40.562509+0000","last_unstale":"2022-09-07T21:39:46.784539+0000","last_undegraded":"2022-09-07T21:39:46.784539+0000","last_fullsized":"2022-09-07T21:39:46.784539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:27:16.781072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784192+0000","last_change":"2022-09-07T21:38:40.562123+0000","last_active":"2022-09-07T21:39:46.784192+0000","last_peered":"2022-09-07T21:39:46.784192+0000","last_clean":"2022-09-07T21:39:46.784192+0000","last_became_active":"2022-09-07T21:38:40.561632+0000","last_became_peered":"2022-09-07T21:38:40.561632+0000","last_unstale":"2022-09-07T21:39:46.784192+0000","last_undegraded":"2022-09-07T21:39:46.784192+0000","last_fullsized":"2022-09-07T21:39:46.784192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:28:10.763115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793271+0000","last_change":"2022-09-07T21:38:40.560968+0000","last_active":"2022-09-07T21:39:46.793271+0000","last_peered":"2022-09-07T21:39:46.793271+0000","last_clean":"2022-09-07T21:39:46.793271+0000","last_became_active":"2022-09-07T21:38:40.560685+0000","last_became_peered":"2022-09-07T21:38:40.560685+0000","last_unstale":"2022-09-07T21:39:46.793271+0000","last_undegraded":"2022-09-07T21:39:46.793271+0000","last_fullsized":"2022-09-07T21:39:46.793271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:49:41.627456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.786853+0000","last_change":"2022-09-07T21:38:40.572935+0000","last_active":"2022-09-07T21:39:46.786853+0000","last_peered":"2022-09-07T21:39:46.786853+0000","last_clean":"2022-09-07T21:39:46.786853+0000","last_became_active":"2022-09-07T21:38:40.572605+0000","last_became_peered":"2022-09-07T21:38:40.572605+0000","last_unstale":"2022-09-07T21:39:46.786853+0000","last_undegraded":"2022-09-07T21:39:46.786853+0000","last_fullsized":"2022-09-07T21:39:46.786853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:55:47.267936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790461+0000","last_change":"2022-09-07T21:38:40.577737+0000","last_active":"2022-09-07T21:39:46.790461+0000","last_peered":"2022-09-07T21:39:46.790461+0000","last_clean":"2022-09-07T21:39:46.790461+0000","last_became_active":"2022-09-07T21:38:40.577425+0000","last_became_peered":"2022-09-07T21:38:40.577425+0000","last_unstale":"2022-09-07T21:39:46.790461+0000","last_undegraded":"2022-09-07T21:39:46.790461+0000","last_fullsized":"2022-09-07T21:39:46.790461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:46:32.369263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791725+0000","last_change":"2022-09-07T21:38:40.577255+0000","last_active":"2022-09-07T21:39:46.791725+0000","last_peered":"2022-09-07T21:39:46.791725+0000","last_clean":"2022-09-07T21:39:46.791725+0000","last_became_active":"2022-09-07T21:38:40.576750+0000","last_became_peered":"2022-09-07T21:38:40.576750+0000","last_unstale":"2022-09-07T21:39:46.791725+0000","last_undegraded":"2022-09-07T21:39:46.791725+0000","last_fullsized":"2022-09-07T21:39:46.791725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:03:19.663251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784380+0000","last_change":"2022-09-07T21:38:40.577599+0000","last_active":"2022-09-07T21:39:46.784380+0000","last_peered":"2022-09-07T21:39:46.784380+0000","last_clean":"2022-09-07T21:39:46.784380+0000","last_became_active":"2022-09-07T21:38:40.577239+0000","last_became_peered":"2022-09-07T21:38:40.577239+0000","last_unstale":"2022-09-07T21:39:46.784380+0000","last_undegraded":"2022-09-07T21:39:46.784380+0000","last_fullsized":"2022-09-07T21:39:46.784380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:39:23.857698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.794199+0000","last_change":"2022-09-07T21:38:40.559102+0000","last_active":"2022-09-07T21:39:46.794199+0000","last_peered":"2022-09-07T21:39:46.794199+0000","last_clean":"2022-09-07T21:39:46.794199+0000","last_became_active":"2022-09-07T21:38:40.558907+0000","last_became_peered":"2022-09-07T21:38:40.558907+0000","last_unstale":"2022-09-07T21:39:46.794199+0000","last_undegraded":"2022-09-07T21:39:46.794199+0000","last_fullsized":"2022-09-07T21:39:46.794199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:08:03.644593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791478+0000","last_change":"2022-09-07T21:38:40.566427+0000","last_active":"2022-09-07T21:39:46.791478+0000","last_peered":"2022-09-07T21:39:46.791478+0000","last_clean":"2022-09-07T21:39:46.791478+0000","last_became_active":"2022-09-07T21:38:40.566068+0000","last_became_peered":"2022-09-07T21:38:40.566068+0000","last_unstale":"2022-09-07T21:39:46.791478+0000","last_undegraded":"2022-09-07T21:39:46.791478+0000","last_fullsized":"2022-09-07T21:39:46.791478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:40:01.825654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791305+0000","last_change":"2022-09-07T21:38:40.568455+0000","last_active":"2022-09-07T21:39:46.791305+0000","last_peered":"2022-09-07T21:39:46.791305+0000","last_clean":"2022-09-07T21:39:46.791305+0000","last_became_active":"2022-09-07T21:38:40.567941+0000","last_became_peered":"2022-09-07T21:38:40.567941+0000","last_unstale":"2022-09-07T21:39:46.791305+0000","last_undegraded":"2022-09-07T21:39:46.791305+0000","last_fullsized":"2022-09-07T21:39:46.791305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:49:16.624335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793197+0000","last_change":"2022-09-07T21:38:40.563200+0000","last_active":"2022-09-07T21:39:46.793197+0000","last_peered":"2022-09-07T21:39:46.793197+0000","last_clean":"2022-09-07T21:39:46.793197+0000","last_became_active":"2022-09-07T21:38:40.562901+0000","last_became_peered":"2022-09-07T21:38:40.562901+0000","last_unstale":"2022-09-07T21:39:46.793197+0000","last_undegraded":"2022-09-07T21:39:46.793197+0000","last_fullsized":"2022-09-07T21:39:46.793197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:49:26.745940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788724+0000","last_change":"2022-09-07T21:38:40.570403+0000","last_active":"2022-09-07T21:39:46.788724+0000","last_peered":"2022-09-07T21:39:46.788724+0000","last_clean":"2022-09-07T21:39:46.788724+0000","last_became_active":"2022-09-07T21:38:40.567326+0000","last_became_peered":"2022-09-07T21:38:40.567326+0000","last_unstale":"2022-09-07T21:39:46.788724+0000","last_undegraded":"2022-09-07T21:39:46.788724+0000","last_fullsized":"2022-09-07T21:39:46.788724+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:19:31.437519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.783735+0000","last_change":"2022-09-07T21:38:40.562990+0000","last_active":"2022-09-07T21:39:46.783735+0000","last_peered":"2022-09-07T21:39:46.783735+0000","last_clean":"2022-09-07T21:39:46.783735+0000","last_became_active":"2022-09-07T21:38:40.562651+0000","last_became_peered":"2022-09-07T21:38:40.562651+0000","last_unstale":"2022-09-07T21:39:46.783735+0000","last_undegraded":"2022-09-07T21:39:46.783735+0000","last_fullsized":"2022-09-07T21:39:46.783735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:49:07.935459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.783491+0000","last_change":"2022-09-07T21:38:40.570849+0000","last_active":"2022-09-07T21:39:46.783491+0000","last_peered":"2022-09-07T21:39:46.783491+0000","last_clean":"2022-09-07T21:39:46.783491+0000","last_became_active":"2022-09-07T21:38:40.570543+0000","last_became_peered":"2022-09-07T21:38:40.570543+0000","last_unstale":"2022-09-07T21:39:46.783491+0000","last_undegraded":"2022-09-07T21:39:46.783491+0000","last_fullsized":"2022-09-07T21:39:46.783491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:33:30.564522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788070+0000","last_change":"2022-09-07T21:38:40.568348+0000","last_active":"2022-09-07T21:39:46.788070+0000","last_peered":"2022-09-07T21:39:46.788070+0000","last_clean":"2022-09-07T21:39:46.788070+0000","last_became_active":"2022-09-07T21:38:40.568031+0000","last_became_peered":"2022-09-07T21:38:40.568031+0000","last_unstale":"2022-09-07T21:39:46.788070+0000","last_undegraded":"2022-09-07T21:39:46.788070+0000","last_fullsized":"2022-09-07T21:39:46.788070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:24:21.978277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791902+0000","last_change":"2022-09-07T21:38:40.577289+0000","last_active":"2022-09-07T21:39:46.791902+0000","last_peered":"2022-09-07T21:39:46.791902+0000","last_clean":"2022-09-07T21:39:46.791902+0000","last_became_active":"2022-09-07T21:38:40.576927+0000","last_became_peered":"2022-09-07T21:38:40.576927+0000","last_unstale":"2022-09-07T21:39:46.791902+0000","last_undegraded":"2022-09-07T21:39:46.791902+0000","last_fullsized":"2022-09-07T21:39:46.791902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:02:53.553226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790196+0000","last_change":"2022-09-07T21:38:40.575886+0000","last_active":"2022-09-07T21:39:46.790196+0000","last_peered":"2022-09-07T21:39:46.790196+0000","last_clean":"2022-09-07T21:39:46.790196+0000","last_became_active":"2022-09-07T21:38:40.575463+0000","last_became_peered":"2022-09-07T21:38:40.575463+0000","last_unstale":"2022-09-07T21:39:46.790196+0000","last_undegraded":"2022-09-07T21:39:46.790196+0000","last_fullsized":"2022-09-07T21:39:46.790196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:10:20.091899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793516+0000","last_change":"2022-09-07T21:38:40.572579+0000","last_active":"2022-09-07T21:39:46.793516+0000","last_peered":"2022-09-07T21:39:46.793516+0000","last_clean":"2022-09-07T21:39:46.793516+0000","last_became_active":"2022-09-07T21:38:40.572302+0000","last_became_peered":"2022-09-07T21:38:40.572302+0000","last_unstale":"2022-09-07T21:39:46.793516+0000","last_undegraded":"2022-09-07T21:39:46.793516+0000","last_fullsized":"2022-09-07T21:39:46.793516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:41:19.938512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.792583+0000","last_change":"2022-09-07T21:38:44.996073+0000","last_active":"2022-09-07T21:39:46.792583+0000","last_peered":"2022-09-07T21:39:46.792583+0000","last_clean":"2022-09-07T21:39:46.792583+0000","last_became_active":"2022-09-07T21:38:44.995571+0000","last_became_peered":"2022-09-07T21:38:44.995571+0000","last_unstale":"2022-09-07T21:39:46.792583+0000","last_undegraded":"2022-09-07T21:39:46.792583+0000","last_fullsized":"2022-09-07T21:39:46.792583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:41:58.745823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:48.148181+0000","last_change":"2022-09-07T21:39:48.148181+0000","last_active":"2022-09-07T21:39:48.148181+0000","last_peered":"2022-09-07T21:39:48.148181+0000","last_clean":"2022-09-07T21:39:48.148181+0000","last_became_active":"2022-09-07T21:39:47.794055+0000","last_became_peered":"2022-09-07T21:39:47.794055+0000","last_unstale":"2022-09-07T21:39:48.148181+0000","last_undegraded":"2022-09-07T21:39:48.148181+0000","last_fullsized":"2022-09-07T21:39:48.148181+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:43:35.756474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":566,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.785552+0000","last_change":"2022-09-07T21:38:40.578853+0000","last_active":"2022-09-07T21:39:46.785552+0000","last_peered":"2022-09-07T21:39:46.785552+0000","last_clean":"2022-09-07T21:39:46.785552+0000","last_became_active":"2022-09-07T21:38:40.578229+0000","last_became_peered":"2022-09-07T21:38:40.578229+0000","last_unstale":"2022-09-07T21:39:46.785552+0000","last_undegraded":"2022-09-07T21:39:46.785552+0000","last_fullsized":"2022-09-07T21:39:46.785552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:39:28.259219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.792264+0000","last_change":"2022-09-07T21:38:44.995929+0000","last_active":"2022-09-07T21:39:46.792264+0000","last_peered":"2022-09-07T21:39:46.792264+0000","last_clean":"2022-09-07T21:39:46.792264+0000","last_became_active":"2022-09-07T21:38:44.995265+0000","last_became_peered":"2022-09-07T21:38:44.995265+0000","last_unstale":"2022-09-07T21:39:46.792264+0000","last_undegraded":"2022-09-07T21:39:46.792264+0000","last_fullsized":"2022-09-07T21:39:46.792264+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:11:30.103628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.785590+0000","last_change":"2022-09-07T21:38:39.971598+0000","last_active":"2022-09-07T21:39:46.785590+0000","last_peered":"2022-09-07T21:39:46.785590+0000","last_clean":"2022-09-07T21:39:46.785590+0000","last_became_active":"2022-09-07T21:38:39.971079+0000","last_became_peered":"2022-09-07T21:38:39.971079+0000","last_unstale":"2022-09-07T21:39:46.785590+0000","last_undegraded":"2022-09-07T21:39:46.785590+0000","last_fullsized":"2022-09-07T21:39:46.785590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:01:22.960635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.785131+0000","last_change":"2022-09-07T21:38:40.577589+0000","last_active":"2022-09-07T21:39:46.785131+0000","last_peered":"2022-09-07T21:39:46.785131+0000","last_clean":"2022-09-07T21:39:46.785131+0000","last_became_active":"2022-09-07T21:38:40.577192+0000","last_became_peered":"2022-09-07T21:38:40.577192+0000","last_unstale":"2022-09-07T21:39:46.785131+0000","last_undegraded":"2022-09-07T21:39:46.785131+0000","last_fullsized":"2022-09-07T21:39:46.785131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:41:03.746467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.792888+0000","last_change":"2022-09-07T21:38:44.997384+0000","last_active":"2022-09-07T21:39:46.792888+0000","last_peered":"2022-09-07T21:39:46.792888+0000","last_clean":"2022-09-07T21:39:46.792888+0000","last_became_active":"2022-09-07T21:38:44.996150+0000","last_became_peered":"2022-09-07T21:38:44.996150+0000","last_unstale":"2022-09-07T21:39:46.792888+0000","last_undegraded":"2022-09-07T21:39:46.792888+0000","last_fullsized":"2022-09-07T21:39:46.792888+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:30:05.633718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:48.140438+0000","last_change":"2022-09-07T21:39:48.139937+0000","last_active":"2022-09-07T21:39:48.140438+0000","last_peered":"2022-09-07T21:39:48.140438+0000","last_clean":"2022-09-07T21:39:48.140438+0000","last_became_active":"2022-09-07T21:39:47.792095+0000","last_became_peered":"2022-09-07T21:39:47.792095+0000","last_unstale":"2022-09-07T21:39:48.140438+0000","last_undegraded":"2022-09-07T21:39:48.140438+0000","last_fullsized":"2022-09-07T21:39:48.140438+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:33:13.259619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789240+0000","last_change":"2022-09-07T21:38:40.572694+0000","last_active":"2022-09-07T21:39:46.789240+0000","last_peered":"2022-09-07T21:39:46.789240+0000","last_clean":"2022-09-07T21:39:46.789240+0000","last_became_active":"2022-09-07T21:38:40.572407+0000","last_became_peered":"2022-09-07T21:38:40.572407+0000","last_unstale":"2022-09-07T21:39:46.789240+0000","last_undegraded":"2022-09-07T21:39:46.789240+0000","last_fullsized":"2022-09-07T21:39:46.789240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:40:17.238549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.792057+0000","last_change":"2022-09-07T21:38:44.988445+0000","last_active":"2022-09-07T21:39:46.792057+0000","last_peered":"2022-09-07T21:39:46.792057+0000","last_clean":"2022-09-07T21:39:46.792057+0000","last_became_active":"2022-09-07T21:38:44.987966+0000","last_became_peered":"2022-09-07T21:38:44.987966+0000","last_unstale":"2022-09-07T21:39:46.792057+0000","last_undegraded":"2022-09-07T21:39:46.792057+0000","last_fullsized":"2022-09-07T21:39:46.792057+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:36:16.050494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.787147+0000","last_change":"2022-09-07T21:38:39.963456+0000","last_active":"2022-09-07T21:39:46.787147+0000","last_peered":"2022-09-07T21:39:46.787147+0000","last_clean":"2022-09-07T21:39:46.787147+0000","last_became_active":"2022-09-07T21:38:39.963059+0000","last_became_peered":"2022-09-07T21:38:39.963059+0000","last_unstale":"2022-09-07T21:39:46.787147+0000","last_undegraded":"2022-09-07T21:39:46.787147+0000","last_fullsized":"2022-09-07T21:39:46.787147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:51:09.061725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790283+0000","last_change":"2022-09-07T21:38:40.559585+0000","last_active":"2022-09-07T21:39:46.790283+0000","last_peered":"2022-09-07T21:39:46.790283+0000","last_clean":"2022-09-07T21:39:46.790283+0000","last_became_active":"2022-09-07T21:38:40.559307+0000","last_became_peered":"2022-09-07T21:38:40.559307+0000","last_unstale":"2022-09-07T21:39:46.790283+0000","last_undegraded":"2022-09-07T21:39:46.790283+0000","last_fullsized":"2022-09-07T21:39:46.790283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:20:32.332429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793369+0000","last_change":"2022-09-07T21:38:44.985596+0000","last_active":"2022-09-07T21:39:46.793369+0000","last_peered":"2022-09-07T21:39:46.793369+0000","last_clean":"2022-09-07T21:39:46.793369+0000","last_became_active":"2022-09-07T21:38:44.985112+0000","last_became_peered":"2022-09-07T21:38:44.985112+0000","last_unstale":"2022-09-07T21:39:46.793369+0000","last_undegraded":"2022-09-07T21:39:46.793369+0000","last_fullsized":"2022-09-07T21:39:46.793369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:24:06.456838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":99,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:40:00.817352+0000","last_change":"2022-09-07T21:39:48.142234+0000","last_active":"2022-09-07T21:40:00.817352+0000","last_peered":"2022-09-07T21:40:00.817352+0000","last_clean":"2022-09-07T21:40:00.817352+0000","last_became_active":"2022-09-07T21:39:47.795907+0000","last_became_peered":"2022-09-07T21:39:47.795907+0000","last_unstale":"2022-09-07T21:40:00.817352+0000","last_undegraded":"2022-09-07T21:40:00.817352+0000","last_fullsized":"2022-09-07T21:40:00.817352+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:42:26.284246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.786886+0000","last_change":"2022-09-07T21:38:40.577546+0000","last_active":"2022-09-07T21:39:46.786886+0000","last_peered":"2022-09-07T21:39:46.786886+0000","last_clean":"2022-09-07T21:39:46.786886+0000","last_became_active":"2022-09-07T21:38:40.577229+0000","last_became_peered":"2022-09-07T21:38:40.577229+0000","last_unstale":"2022-09-07T21:39:46.786886+0000","last_undegraded":"2022-09-07T21:39:46.786886+0000","last_fullsized":"2022-09-07T21:39:46.786886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:57:36.947488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793428+0000","last_change":"2022-09-07T21:38:44.993976+0000","last_active":"2022-09-07T21:39:46.793428+0000","last_peered":"2022-09-07T21:39:46.793428+0000","last_clean":"2022-09-07T21:39:46.793428+0000","last_became_active":"2022-09-07T21:38:44.993554+0000","last_became_peered":"2022-09-07T21:38:44.993554+0000","last_unstale":"2022-09-07T21:39:46.793428+0000","last_undegraded":"2022-09-07T21:39:46.793428+0000","last_fullsized":"2022-09-07T21:39:46.793428+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:21:13.466677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790366+0000","last_change":"2022-09-07T21:38:39.970361+0000","last_active":"2022-09-07T21:39:46.790366+0000","last_peered":"2022-09-07T21:39:46.790366+0000","last_clean":"2022-09-07T21:39:46.790366+0000","last_became_active":"2022-09-07T21:38:39.969420+0000","last_became_peered":"2022-09-07T21:38:39.969420+0000","last_unstale":"2022-09-07T21:39:46.790366+0000","last_undegraded":"2022-09-07T21:39:46.790366+0000","last_fullsized":"2022-09-07T21:39:46.790366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:53:46.903275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791431+0000","last_change":"2022-09-07T21:38:40.567273+0000","last_active":"2022-09-07T21:39:46.791431+0000","last_peered":"2022-09-07T21:39:46.791431+0000","last_clean":"2022-09-07T21:39:46.791431+0000","last_became_active":"2022-09-07T21:38:40.566942+0000","last_became_peered":"2022-09-07T21:38:40.566942+0000","last_unstale":"2022-09-07T21:39:46.791431+0000","last_undegraded":"2022-09-07T21:39:46.791431+0000","last_fullsized":"2022-09-07T21:39:46.791431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:29:38.009443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791804+0000","last_change":"2022-09-07T21:38:44.998836+0000","last_active":"2022-09-07T21:39:46.791804+0000","last_peered":"2022-09-07T21:39:46.791804+0000","last_clean":"2022-09-07T21:39:46.791804+0000","last_became_active":"2022-09-07T21:38:44.998461+0000","last_became_peered":"2022-09-07T21:38:44.998461+0000","last_unstale":"2022-09-07T21:39:46.791804+0000","last_undegraded":"2022-09-07T21:39:46.791804+0000","last_fullsized":"2022-09-07T21:39:46.791804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:37:37.203792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788074+0000","last_change":"2022-09-07T21:38:39.970746+0000","last_active":"2022-09-07T21:39:46.788074+0000","last_peered":"2022-09-07T21:39:46.788074+0000","last_clean":"2022-09-07T21:39:46.788074+0000","last_became_active":"2022-09-07T21:38:39.970438+0000","last_became_peered":"2022-09-07T21:38:39.970438+0000","last_unstale":"2022-09-07T21:39:46.788074+0000","last_undegraded":"2022-09-07T21:39:46.788074+0000","last_fullsized":"2022-09-07T21:39:46.788074+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:51:07.849231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784303+0000","last_change":"2022-09-07T21:38:40.576245+0000","last_active":"2022-09-07T21:39:46.784303+0000","last_peered":"2022-09-07T21:39:46.784303+0000","last_clean":"2022-09-07T21:39:46.784303+0000","last_became_active":"2022-09-07T21:38:40.575916+0000","last_became_peered":"2022-09-07T21:38:40.575916+0000","last_unstale":"2022-09-07T21:39:46.784303+0000","last_undegraded":"2022-09-07T21:39:46.784303+0000","last_fullsized":"2022-09-07T21:39:46.784303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:35:52.025507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784407+0000","last_change":"2022-09-07T21:38:44.991494+0000","last_active":"2022-09-07T21:39:46.784407+0000","last_peered":"2022-09-07T21:39:46.784407+0000","last_clean":"2022-09-07T21:39:46.784407+0000","last_became_active":"2022-09-07T21:38:44.988446+0000","last_became_peered":"2022-09-07T21:38:44.988446+0000","last_unstale":"2022-09-07T21:39:46.784407+0000","last_undegraded":"2022-09-07T21:39:46.784407+0000","last_fullsized":"2022-09-07T21:39:46.784407+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:22:52.937965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:47.795341+0000","last_change":"2022-09-07T21:39:47.794537+0000","last_active":"2022-09-07T21:39:47.795341+0000","last_peered":"2022-09-07T21:39:47.795341+0000","last_clean":"2022-09-07T21:39:47.795341+0000","last_became_active":"2022-09-07T21:39:47.794107+0000","last_became_peered":"2022-09-07T21:39:47.794107+0000","last_unstale":"2022-09-07T21:39:47.795341+0000","last_undegraded":"2022-09-07T21:39:47.795341+0000","last_fullsized":"2022-09-07T21:39:47.795341+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:44:58.430301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789269+0000","last_change":"2022-09-07T21:38:40.590979+0000","last_active":"2022-09-07T21:39:46.789269+0000","last_peered":"2022-09-07T21:39:46.789269+0000","last_clean":"2022-09-07T21:39:46.789269+0000","last_became_active":"2022-09-07T21:38:40.590024+0000","last_became_peered":"2022-09-07T21:38:40.590024+0000","last_unstale":"2022-09-07T21:39:46.789269+0000","last_undegraded":"2022-09-07T21:39:46.789269+0000","last_fullsized":"2022-09-07T21:39:46.789269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:58:11.726364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784894+0000","last_change":"2022-09-07T21:38:44.988252+0000","last_active":"2022-09-07T21:39:46.784894+0000","last_peered":"2022-09-07T21:39:46.784894+0000","last_clean":"2022-09-07T21:39:46.784894+0000","last_became_active":"2022-09-07T21:38:44.987851+0000","last_became_peered":"2022-09-07T21:38:44.987851+0000","last_unstale":"2022-09-07T21:39:46.784894+0000","last_undegraded":"2022-09-07T21:39:46.784894+0000","last_fullsized":"2022-09-07T21:39:46.784894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:53:42.207358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:40:00.816719+0000","last_change":"2022-09-07T21:38:39.969970+0000","last_active":"2022-09-07T21:40:00.816719+0000","last_peered":"2022-09-07T21:40:00.816719+0000","last_clean":"2022-09-07T21:40:00.816719+0000","last_became_active":"2022-09-07T21:38:39.969627+0000","last_became_peered":"2022-09-07T21:38:39.969627+0000","last_unstale":"2022-09-07T21:40:00.816719+0000","last_undegraded":"2022-09-07T21:40:00.816719+0000","last_fullsized":"2022-09-07T21:40:00.816719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:14:11.990138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":55,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.792576+0000","last_change":"2022-09-07T21:38:40.577175+0000","last_active":"2022-09-07T21:39:46.792576+0000","last_peered":"2022-09-07T21:39:46.792576+0000","last_clean":"2022-09-07T21:39:46.792576+0000","last_became_active":"2022-09-07T21:38:40.576873+0000","last_became_peered":"2022-09-07T21:38:40.576873+0000","last_unstale":"2022-09-07T21:39:46.792576+0000","last_undegraded":"2022-09-07T21:39:46.792576+0000","last_fullsized":"2022-09-07T21:39:46.792576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:25:17.911241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789266+0000","last_change":"2022-09-07T21:38:44.984456+0000","last_active":"2022-09-07T21:39:46.789266+0000","last_peered":"2022-09-07T21:39:46.789266+0000","last_clean":"2022-09-07T21:39:46.789266+0000","last_became_active":"2022-09-07T21:38:44.984070+0000","last_became_peered":"2022-09-07T21:38:44.984070+0000","last_unstale":"2022-09-07T21:39:46.789266+0000","last_undegraded":"2022-09-07T21:39:46.789266+0000","last_fullsized":"2022-09-07T21:39:46.789266+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:36:14.150189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:47.881379+0000","last_change":"2022-09-07T21:39:47.881379+0000","last_active":"2022-09-07T21:39:47.881379+0000","last_peered":"2022-09-07T21:39:47.881379+0000","last_clean":"2022-09-07T21:39:47.881379+0000","last_became_active":"2022-09-07T21:39:47.792120+0000","last_became_peered":"2022-09-07T21:39:47.792120+0000","last_unstale":"2022-09-07T21:39:47.881379+0000","last_undegraded":"2022-09-07T21:39:47.881379+0000","last_fullsized":"2022-09-07T21:39:47.881379+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:42:47.731825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784902+0000","last_change":"2022-09-07T21:38:40.571594+0000","last_active":"2022-09-07T21:39:46.784902+0000","last_peered":"2022-09-07T21:39:46.784902+0000","last_clean":"2022-09-07T21:39:46.784902+0000","last_became_active":"2022-09-07T21:38:40.571188+0000","last_became_peered":"2022-09-07T21:38:40.571188+0000","last_unstale":"2022-09-07T21:39:46.784902+0000","last_undegraded":"2022-09-07T21:39:46.784902+0000","last_fullsized":"2022-09-07T21:39:46.784902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:25:49.474602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:47.794303+0000","last_change":"2022-09-07T21:39:47.794303+0000","last_active":"2022-09-07T21:39:47.794303+0000","last_peered":"2022-09-07T21:39:47.794303+0000","last_clean":"2022-09-07T21:39:47.794303+0000","last_became_active":"2022-09-07T21:39:47.793887+0000","last_became_peered":"2022-09-07T21:39:47.793887+0000","last_unstale":"2022-09-07T21:39:47.794303+0000","last_undegraded":"2022-09-07T21:39:47.794303+0000","last_fullsized":"2022-09-07T21:39:47.794303+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:11:36.134842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790182+0000","last_change":"2022-09-07T21:38:39.964670+0000","last_active":"2022-09-07T21:39:46.790182+0000","last_peered":"2022-09-07T21:39:46.790182+0000","last_clean":"2022-09-07T21:39:46.790182+0000","last_became_active":"2022-09-07T21:38:39.964305+0000","last_became_peered":"2022-09-07T21:38:39.964305+0000","last_unstale":"2022-09-07T21:39:46.790182+0000","last_undegraded":"2022-09-07T21:39:46.790182+0000","last_fullsized":"2022-09-07T21:39:46.790182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:15:57.041815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789403+0000","last_change":"2022-09-07T21:38:40.565156+0000","last_active":"2022-09-07T21:39:46.789403+0000","last_peered":"2022-09-07T21:39:46.789403+0000","last_clean":"2022-09-07T21:39:46.789403+0000","last_became_active":"2022-09-07T21:38:40.564843+0000","last_became_peered":"2022-09-07T21:38:40.564843+0000","last_unstale":"2022-09-07T21:39:46.789403+0000","last_undegraded":"2022-09-07T21:39:46.789403+0000","last_fullsized":"2022-09-07T21:39:46.789403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:11:09.741599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789546+0000","last_change":"2022-09-07T21:38:44.986167+0000","last_active":"2022-09-07T21:39:46.789546+0000","last_peered":"2022-09-07T21:39:46.789546+0000","last_clean":"2022-09-07T21:39:46.789546+0000","last_became_active":"2022-09-07T21:38:44.985778+0000","last_became_peered":"2022-09-07T21:38:44.985778+0000","last_unstale":"2022-09-07T21:39:46.789546+0000","last_undegraded":"2022-09-07T21:39:46.789546+0000","last_fullsized":"2022-09-07T21:39:46.789546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:21:01.441205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'33","reported_seq":169,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:40:01.818371+0000","last_change":"2022-09-07T21:38:39.968660+0000","last_active":"2022-09-07T21:40:01.818371+0000","last_peered":"2022-09-07T21:40:01.818371+0000","last_clean":"2022-09-07T21:40:01.818371+0000","last_became_active":"2022-09-07T21:38:39.968285+0000","last_became_peered":"2022-09-07T21:38:39.968285+0000","last_unstale":"2022-09-07T21:40:01.818371+0000","last_undegraded":"2022-09-07T21:40:01.818371+0000","last_fullsized":"2022-09-07T21:40:01.818371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:16:16.027615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60865,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":118,"num_read_kb":119,"num_write":34,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.787175+0000","last_change":"2022-09-07T21:38:40.566145+0000","last_active":"2022-09-07T21:39:46.787175+0000","last_peered":"2022-09-07T21:39:46.787175+0000","last_clean":"2022-09-07T21:39:46.787175+0000","last_became_active":"2022-09-07T21:38:40.565823+0000","last_became_peered":"2022-09-07T21:38:40.565823+0000","last_unstale":"2022-09-07T21:39:46.787175+0000","last_undegraded":"2022-09-07T21:39:46.787175+0000","last_fullsized":"2022-09-07T21:39:46.787175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:13:28.826066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790639+0000","last_change":"2022-09-07T21:38:44.995599+0000","last_active":"2022-09-07T21:39:46.790639+0000","last_peered":"2022-09-07T21:39:46.790639+0000","last_clean":"2022-09-07T21:39:46.790639+0000","last_became_active":"2022-09-07T21:38:44.994627+0000","last_became_peered":"2022-09-07T21:38:44.994627+0000","last_unstale":"2022-09-07T21:39:46.790639+0000","last_undegraded":"2022-09-07T21:39:46.790639+0000","last_fullsized":"2022-09-07T21:39:46.790639+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:17:30.608595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.794164+0000","last_change":"2022-09-07T21:38:39.968773+0000","last_active":"2022-09-07T21:39:46.794164+0000","last_peered":"2022-09-07T21:39:46.794164+0000","last_clean":"2022-09-07T21:39:46.794164+0000","last_became_active":"2022-09-07T21:38:39.967912+0000","last_became_peered":"2022-09-07T21:38:39.967912+0000","last_unstale":"2022-09-07T21:39:46.794164+0000","last_undegraded":"2022-09-07T21:39:46.794164+0000","last_fullsized":"2022-09-07T21:39:46.794164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:51:12.940327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788800+0000","last_change":"2022-09-07T21:38:40.573548+0000","last_active":"2022-09-07T21:39:46.788800+0000","last_peered":"2022-09-07T21:39:46.788800+0000","last_clean":"2022-09-07T21:39:46.788800+0000","last_became_active":"2022-09-07T21:38:40.573273+0000","last_became_peered":"2022-09-07T21:38:40.573273+0000","last_unstale":"2022-09-07T21:39:46.788800+0000","last_undegraded":"2022-09-07T21:39:46.788800+0000","last_fullsized":"2022-09-07T21:39:46.788800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:03:52.778107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789661+0000","last_change":"2022-09-07T21:38:44.987366+0000","last_active":"2022-09-07T21:39:46.789661+0000","last_peered":"2022-09-07T21:39:46.789661+0000","last_clean":"2022-09-07T21:39:46.789661+0000","last_became_active":"2022-09-07T21:38:44.986775+0000","last_became_peered":"2022-09-07T21:38:44.986775+0000","last_unstale":"2022-09-07T21:39:46.789661+0000","last_undegraded":"2022-09-07T21:39:46.789661+0000","last_fullsized":"2022-09-07T21:39:46.789661+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:07:14.746521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790197+0000","last_change":"2022-09-07T21:38:39.969123+0000","last_active":"2022-09-07T21:39:46.790197+0000","last_peered":"2022-09-07T21:39:46.790197+0000","last_clean":"2022-09-07T21:39:46.790197+0000","last_became_active":"2022-09-07T21:38:39.968744+0000","last_became_peered":"2022-09-07T21:38:39.968744+0000","last_unstale":"2022-09-07T21:39:46.790197+0000","last_undegraded":"2022-09-07T21:39:46.790197+0000","last_fullsized":"2022-09-07T21:39:46.790197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:54:34.616856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789635+0000","last_change":"2022-09-07T21:38:40.578056+0000","last_active":"2022-09-07T21:39:46.789635+0000","last_peered":"2022-09-07T21:39:46.789635+0000","last_clean":"2022-09-07T21:39:46.789635+0000","last_became_active":"2022-09-07T21:38:40.577727+0000","last_became_peered":"2022-09-07T21:38:40.577727+0000","last_unstale":"2022-09-07T21:39:46.789635+0000","last_undegraded":"2022-09-07T21:39:46.789635+0000","last_fullsized":"2022-09-07T21:39:46.789635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:10:08.645724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:48.011128+0000","last_change":"2022-09-07T21:39:48.010666+0000","last_active":"2022-09-07T21:39:48.011128+0000","last_peered":"2022-09-07T21:39:48.011128+0000","last_clean":"2022-09-07T21:39:48.011128+0000","last_became_active":"2022-09-07T21:39:48.010254+0000","last_became_peered":"2022-09-07T21:39:48.010254+0000","last_unstale":"2022-09-07T21:39:48.011128+0000","last_undegraded":"2022-09-07T21:39:48.011128+0000","last_fullsized":"2022-09-07T21:39:48.011128+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:54:08.005552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:40:01.950426+0000","last_change":"2022-09-07T21:38:39.962760+0000","last_active":"2022-09-07T21:40:01.950426+0000","last_peered":"2022-09-07T21:40:01.950426+0000","last_clean":"2022-09-07T21:40:01.950426+0000","last_became_active":"2022-09-07T21:38:39.962327+0000","last_became_peered":"2022-09-07T21:38:39.962327+0000","last_unstale":"2022-09-07T21:40:01.950426+0000","last_undegraded":"2022-09-07T21:40:01.950426+0000","last_fullsized":"2022-09-07T21:40:01.950426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:17:27.180736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":61,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788943+0000","last_change":"2022-09-07T21:38:40.569532+0000","last_active":"2022-09-07T21:39:46.788943+0000","last_peered":"2022-09-07T21:39:46.788943+0000","last_clean":"2022-09-07T21:39:46.788943+0000","last_became_active":"2022-09-07T21:38:40.569037+0000","last_became_peered":"2022-09-07T21:38:40.569037+0000","last_unstale":"2022-09-07T21:39:46.788943+0000","last_undegraded":"2022-09-07T21:39:46.788943+0000","last_fullsized":"2022-09-07T21:39:46.788943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:37:05.114735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788554+0000","last_change":"2022-09-07T21:38:44.995323+0000","last_active":"2022-09-07T21:39:46.788554+0000","last_peered":"2022-09-07T21:39:46.788554+0000","last_clean":"2022-09-07T21:39:46.788554+0000","last_became_active":"2022-09-07T21:38:44.994915+0000","last_became_peered":"2022-09-07T21:38:44.994915+0000","last_unstale":"2022-09-07T21:39:46.788554+0000","last_undegraded":"2022-09-07T21:39:46.788554+0000","last_fullsized":"2022-09-07T21:39:46.788554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:17:10.419180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.786701+0000","last_change":"2022-09-07T21:38:39.967903+0000","last_active":"2022-09-07T21:39:46.786701+0000","last_peered":"2022-09-07T21:39:46.786701+0000","last_clean":"2022-09-07T21:39:46.786701+0000","last_became_active":"2022-09-07T21:38:39.967392+0000","last_became_peered":"2022-09-07T21:38:39.967392+0000","last_unstale":"2022-09-07T21:39:46.786701+0000","last_undegraded":"2022-09-07T21:39:46.786701+0000","last_fullsized":"2022-09-07T21:39:46.786701+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:31:39.581561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790435+0000","last_change":"2022-09-07T21:38:40.566956+0000","last_active":"2022-09-07T21:39:46.790435+0000","last_peered":"2022-09-07T21:39:46.790435+0000","last_clean":"2022-09-07T21:39:46.790435+0000","last_became_active":"2022-09-07T21:38:40.566655+0000","last_became_peered":"2022-09-07T21:38:40.566655+0000","last_unstale":"2022-09-07T21:39:46.790435+0000","last_undegraded":"2022-09-07T21:39:46.790435+0000","last_fullsized":"2022-09-07T21:39:46.790435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:37:16.353162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790328+0000","last_change":"2022-09-07T21:38:44.988605+0000","last_active":"2022-09-07T21:39:46.790328+0000","last_peered":"2022-09-07T21:39:46.790328+0000","last_clean":"2022-09-07T21:39:46.790328+0000","last_became_active":"2022-09-07T21:38:44.988082+0000","last_became_peered":"2022-09-07T21:38:44.988082+0000","last_unstale":"2022-09-07T21:39:46.790328+0000","last_undegraded":"2022-09-07T21:39:46.790328+0000","last_fullsized":"2022-09-07T21:39:46.790328+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:30:06.975077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.787324+0000","last_change":"2022-09-07T21:38:39.961740+0000","last_active":"2022-09-07T21:39:46.787324+0000","last_peered":"2022-09-07T21:39:46.787324+0000","last_clean":"2022-09-07T21:39:46.787324+0000","last_became_active":"2022-09-07T21:38:39.961219+0000","last_became_peered":"2022-09-07T21:38:39.961219+0000","last_unstale":"2022-09-07T21:39:46.787324+0000","last_undegraded":"2022-09-07T21:39:46.787324+0000","last_fullsized":"2022-09-07T21:39:46.787324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:51:24.649401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791871+0000","last_change":"2022-09-07T21:38:40.569594+0000","last_active":"2022-09-07T21:39:46.791871+0000","last_peered":"2022-09-07T21:39:46.791871+0000","last_clean":"2022-09-07T21:39:46.791871+0000","last_became_active":"2022-09-07T21:38:40.569251+0000","last_became_peered":"2022-09-07T21:38:40.569251+0000","last_unstale":"2022-09-07T21:39:46.791871+0000","last_undegraded":"2022-09-07T21:39:46.791871+0000","last_fullsized":"2022-09-07T21:39:46.791871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:37:43.337560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.786814+0000","last_change":"2022-09-07T21:38:44.993534+0000","last_active":"2022-09-07T21:39:46.786814+0000","last_peered":"2022-09-07T21:39:46.786814+0000","last_clean":"2022-09-07T21:39:46.786814+0000","last_became_active":"2022-09-07T21:38:44.992966+0000","last_became_peered":"2022-09-07T21:38:44.992966+0000","last_unstale":"2022-09-07T21:39:46.786814+0000","last_undegraded":"2022-09-07T21:39:46.786814+0000","last_fullsized":"2022-09-07T21:39:46.786814+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:12:27.991839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790045+0000","last_change":"2022-09-07T21:38:39.978436+0000","last_active":"2022-09-07T21:39:46.790045+0000","last_peered":"2022-09-07T21:39:46.790045+0000","last_clean":"2022-09-07T21:39:46.790045+0000","last_became_active":"2022-09-07T21:38:39.977613+0000","last_became_peered":"2022-09-07T21:38:39.977613+0000","last_unstale":"2022-09-07T21:39:46.790045+0000","last_undegraded":"2022-09-07T21:39:46.790045+0000","last_fullsized":"2022-09-07T21:39:46.790045+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T03:23:08.765467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790791+0000","last_change":"2022-09-07T21:38:40.577968+0000","last_active":"2022-09-07T21:39:46.790791+0000","last_peered":"2022-09-07T21:39:46.790791+0000","last_clean":"2022-09-07T21:39:46.790791+0000","last_became_active":"2022-09-07T21:38:40.577632+0000","last_became_peered":"2022-09-07T21:38:40.577632+0000","last_unstale":"2022-09-07T21:39:46.790791+0000","last_undegraded":"2022-09-07T21:39:46.790791+0000","last_fullsized":"2022-09-07T21:39:46.790791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:57:31.599202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788466+0000","last_change":"2022-09-07T21:38:44.989341+0000","last_active":"2022-09-07T21:39:46.788466+0000","last_peered":"2022-09-07T21:39:46.788466+0000","last_clean":"2022-09-07T21:39:46.788466+0000","last_became_active":"2022-09-07T21:38:44.988977+0000","last_became_peered":"2022-09-07T21:38:44.988977+0000","last_unstale":"2022-09-07T21:39:46.788466+0000","last_undegraded":"2022-09-07T21:39:46.788466+0000","last_fullsized":"2022-09-07T21:39:46.788466+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:34:52.397025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789613+0000","last_change":"2022-09-07T21:38:39.970365+0000","last_active":"2022-09-07T21:39:46.789613+0000","last_peered":"2022-09-07T21:39:46.789613+0000","last_clean":"2022-09-07T21:39:46.789613+0000","last_became_active":"2022-09-07T21:38:39.970044+0000","last_became_peered":"2022-09-07T21:38:39.970044+0000","last_unstale":"2022-09-07T21:39:46.789613+0000","last_undegraded":"2022-09-07T21:39:46.789613+0000","last_fullsized":"2022-09-07T21:39:46.789613+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:39:04.408591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790853+0000","last_change":"2022-09-07T21:38:40.564714+0000","last_active":"2022-09-07T21:39:46.790853+0000","last_peered":"2022-09-07T21:39:46.790853+0000","last_clean":"2022-09-07T21:39:46.790853+0000","last_became_active":"2022-09-07T21:38:40.563744+0000","last_became_peered":"2022-09-07T21:38:40.563744+0000","last_unstale":"2022-09-07T21:39:46.790853+0000","last_undegraded":"2022-09-07T21:39:46.790853+0000","last_fullsized":"2022-09-07T21:39:46.790853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:46:29.965821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:47.794770+0000","last_change":"2022-09-07T21:39:47.794770+0000","last_active":"2022-09-07T21:39:47.794770+0000","last_peered":"2022-09-07T21:39:47.794770+0000","last_clean":"2022-09-07T21:39:47.794770+0000","last_became_active":"2022-09-07T21:39:47.794311+0000","last_became_peered":"2022-09-07T21:39:47.794311+0000","last_unstale":"2022-09-07T21:39:47.794770+0000","last_undegraded":"2022-09-07T21:39:47.794770+0000","last_fullsized":"2022-09-07T21:39:47.794770+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:36:29.299981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791062+0000","last_change":"2022-09-07T21:38:40.567008+0000","last_active":"2022-09-07T21:39:46.791062+0000","last_peered":"2022-09-07T21:39:46.791062+0000","last_clean":"2022-09-07T21:39:46.791062+0000","last_became_active":"2022-09-07T21:38:40.566598+0000","last_became_peered":"2022-09-07T21:38:40.566598+0000","last_unstale":"2022-09-07T21:39:46.791062+0000","last_undegraded":"2022-09-07T21:39:46.791062+0000","last_fullsized":"2022-09-07T21:39:46.791062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:06:16.999794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:59.945811+0000","last_change":"2022-09-07T21:38:39.966446+0000","last_active":"2022-09-07T21:39:59.945811+0000","last_peered":"2022-09-07T21:39:59.945811+0000","last_clean":"2022-09-07T21:39:59.945811+0000","last_became_active":"2022-09-07T21:38:39.966095+0000","last_became_peered":"2022-09-07T21:38:39.966095+0000","last_unstale":"2022-09-07T21:39:59.945811+0000","last_undegraded":"2022-09-07T21:39:59.945811+0000","last_fullsized":"2022-09-07T21:39:59.945811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:08:54.360894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":55,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790628+0000","last_change":"2022-09-07T21:38:44.993551+0000","last_active":"2022-09-07T21:39:46.790628+0000","last_peered":"2022-09-07T21:39:46.790628+0000","last_clean":"2022-09-07T21:39:46.790628+0000","last_became_active":"2022-09-07T21:38:44.993235+0000","last_became_peered":"2022-09-07T21:38:44.993235+0000","last_unstale":"2022-09-07T21:39:46.790628+0000","last_undegraded":"2022-09-07T21:39:46.790628+0000","last_fullsized":"2022-09-07T21:39:46.790628+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:06:31.660680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784919+0000","last_change":"2022-09-07T21:38:40.578195+0000","last_active":"2022-09-07T21:39:46.784919+0000","last_peered":"2022-09-07T21:39:46.784919+0000","last_clean":"2022-09-07T21:39:46.784919+0000","last_became_active":"2022-09-07T21:38:40.577888+0000","last_became_peered":"2022-09-07T21:38:40.577888+0000","last_unstale":"2022-09-07T21:39:46.784919+0000","last_undegraded":"2022-09-07T21:39:46.784919+0000","last_fullsized":"2022-09-07T21:39:46.784919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:09:25.600697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793053+0000","last_change":"2022-09-07T21:38:39.970599+0000","last_active":"2022-09-07T21:39:46.793053+0000","last_peered":"2022-09-07T21:39:46.793053+0000","last_clean":"2022-09-07T21:39:46.793053+0000","last_became_active":"2022-09-07T21:38:39.970216+0000","last_became_peered":"2022-09-07T21:38:39.970216+0000","last_unstale":"2022-09-07T21:39:46.793053+0000","last_undegraded":"2022-09-07T21:39:46.793053+0000","last_fullsized":"2022-09-07T21:39:46.793053+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:01:36.517769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":382,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:47.788108+0000","last_change":"2022-09-07T21:37:30.699119+0000","last_active":"2022-09-07T21:39:47.788108+0000","last_peered":"2022-09-07T21:39:47.788108+0000","last_clean":"2022-09-07T21:39:47.788108+0000","last_became_active":"2022-09-07T21:37:30.698640+0000","last_became_peered":"2022-09-07T21:37:30.698640+0000","last_unstale":"2022-09-07T21:39:47.788108+0000","last_undegraded":"2022-09-07T21:39:47.788108+0000","last_fullsized":"2022-09-07T21:39:47.788108+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:33:29.392715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:33:29.392715+0000","last_clean_scrub_stamp":"2022-09-07T21:33:29.392715+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:10:27.439063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790737+0000","last_change":"2022-09-07T21:38:44.996255+0000","last_active":"2022-09-07T21:39:46.790737+0000","last_peered":"2022-09-07T21:39:46.790737+0000","last_clean":"2022-09-07T21:39:46.790737+0000","last_became_active":"2022-09-07T21:38:44.995597+0000","last_became_peered":"2022-09-07T21:38:44.995597+0000","last_unstale":"2022-09-07T21:39:46.790737+0000","last_undegraded":"2022-09-07T21:39:46.790737+0000","last_fullsized":"2022-09-07T21:39:46.790737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:59:56.494229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.787878+0000","last_change":"2022-09-07T21:38:40.567961+0000","last_active":"2022-09-07T21:39:46.787878+0000","last_peered":"2022-09-07T21:39:46.787878+0000","last_clean":"2022-09-07T21:39:46.787878+0000","last_became_active":"2022-09-07T21:38:40.567436+0000","last_became_peered":"2022-09-07T21:38:40.567436+0000","last_unstale":"2022-09-07T21:39:46.787878+0000","last_undegraded":"2022-09-07T21:39:46.787878+0000","last_fullsized":"2022-09-07T21:39:46.787878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:25:13.025231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788853+0000","last_change":"2022-09-07T21:38:39.971039+0000","last_active":"2022-09-07T21:39:46.788853+0000","last_peered":"2022-09-07T21:39:46.788853+0000","last_clean":"2022-09-07T21:39:46.788853+0000","last_became_active":"2022-09-07T21:38:39.970675+0000","last_became_peered":"2022-09-07T21:38:39.970675+0000","last_unstale":"2022-09-07T21:39:46.788853+0000","last_undegraded":"2022-09-07T21:39:46.788853+0000","last_fullsized":"2022-09-07T21:39:46.788853+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:36:47.865699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789711+0000","last_change":"2022-09-07T21:38:44.992311+0000","last_active":"2022-09-07T21:39:46.789711+0000","last_peered":"2022-09-07T21:39:46.789711+0000","last_clean":"2022-09-07T21:39:46.789711+0000","last_became_active":"2022-09-07T21:38:44.991846+0000","last_became_peered":"2022-09-07T21:38:44.991846+0000","last_unstale":"2022-09-07T21:39:46.789711+0000","last_undegraded":"2022-09-07T21:39:46.789711+0000","last_fullsized":"2022-09-07T21:39:46.789711+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:42:43.968482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.787249+0000","last_change":"2022-09-07T21:38:39.970220+0000","last_active":"2022-09-07T21:39:46.787249+0000","last_peered":"2022-09-07T21:39:46.787249+0000","last_clean":"2022-09-07T21:39:46.787249+0000","last_became_active":"2022-09-07T21:38:39.969573+0000","last_became_peered":"2022-09-07T21:38:39.969573+0000","last_unstale":"2022-09-07T21:39:46.787249+0000","last_undegraded":"2022-09-07T21:39:46.787249+0000","last_fullsized":"2022-09-07T21:39:46.787249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:22:05.375623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788624+0000","last_change":"2022-09-07T21:38:40.585697+0000","last_active":"2022-09-07T21:39:46.788624+0000","last_peered":"2022-09-07T21:39:46.788624+0000","last_clean":"2022-09-07T21:39:46.788624+0000","last_became_active":"2022-09-07T21:38:40.585293+0000","last_became_peered":"2022-09-07T21:38:40.585293+0000","last_unstale":"2022-09-07T21:39:46.788624+0000","last_undegraded":"2022-09-07T21:39:46.788624+0000","last_fullsized":"2022-09-07T21:39:46.788624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:22:21.707916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784222+0000","last_change":"2022-09-07T21:38:44.985777+0000","last_active":"2022-09-07T21:39:46.784222+0000","last_peered":"2022-09-07T21:39:46.784222+0000","last_clean":"2022-09-07T21:39:46.784222+0000","last_became_active":"2022-09-07T21:38:44.985312+0000","last_became_peered":"2022-09-07T21:38:44.985312+0000","last_unstale":"2022-09-07T21:39:46.784222+0000","last_undegraded":"2022-09-07T21:39:46.784222+0000","last_fullsized":"2022-09-07T21:39:46.784222+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:08:04.011094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'6","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790103+0000","last_change":"2022-09-07T21:38:39.973742+0000","last_active":"2022-09-07T21:39:46.790103+0000","last_peered":"2022-09-07T21:39:46.790103+0000","last_clean":"2022-09-07T21:39:46.790103+0000","last_became_active":"2022-09-07T21:38:39.973324+0000","last_became_peered":"2022-09-07T21:38:39.973324+0000","last_unstale":"2022-09-07T21:39:46.790103+0000","last_undegraded":"2022-09-07T21:39:46.790103+0000","last_fullsized":"2022-09-07T21:39:46.790103+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T21:49:54.331302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.785668+0000","last_change":"2022-09-07T21:38:40.565122+0000","last_active":"2022-09-07T21:39:46.785668+0000","last_peered":"2022-09-07T21:39:46.785668+0000","last_clean":"2022-09-07T21:39:46.785668+0000","last_became_active":"2022-09-07T21:38:40.564813+0000","last_became_peered":"2022-09-07T21:38:40.564813+0000","last_unstale":"2022-09-07T21:39:46.785668+0000","last_undegraded":"2022-09-07T21:39:46.785668+0000","last_fullsized":"2022-09-07T21:39:46.785668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:03:02.178226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791171+0000","last_change":"2022-09-07T21:38:44.987707+0000","last_active":"2022-09-07T21:39:46.791171+0000","last_peered":"2022-09-07T21:39:46.791171+0000","last_clean":"2022-09-07T21:39:46.791171+0000","last_became_active":"2022-09-07T21:38:44.987296+0000","last_became_peered":"2022-09-07T21:38:44.987296+0000","last_unstale":"2022-09-07T21:39:46.791171+0000","last_undegraded":"2022-09-07T21:39:46.791171+0000","last_fullsized":"2022-09-07T21:39:46.791171+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T06:53:07.004127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793725+0000","last_change":"2022-09-07T21:38:39.967664+0000","last_active":"2022-09-07T21:39:46.793725+0000","last_peered":"2022-09-07T21:39:46.793725+0000","last_clean":"2022-09-07T21:39:46.793725+0000","last_became_active":"2022-09-07T21:38:39.967283+0000","last_became_peered":"2022-09-07T21:38:39.967283+0000","last_unstale":"2022-09-07T21:39:46.793725+0000","last_undegraded":"2022-09-07T21:39:46.793725+0000","last_fullsized":"2022-09-07T21:39:46.793725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:47:31.095692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784474+0000","last_change":"2022-09-07T21:38:40.579398+0000","last_active":"2022-09-07T21:39:46.784474+0000","last_peered":"2022-09-07T21:39:46.784474+0000","last_clean":"2022-09-07T21:39:46.784474+0000","last_became_active":"2022-09-07T21:38:40.579112+0000","last_became_peered":"2022-09-07T21:38:40.579112+0000","last_unstale":"2022-09-07T21:39:46.784474+0000","last_undegraded":"2022-09-07T21:39:46.784474+0000","last_fullsized":"2022-09-07T21:39:46.784474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:50:13.063073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793498+0000","last_change":"2022-09-07T21:38:44.996219+0000","last_active":"2022-09-07T21:39:46.793498+0000","last_peered":"2022-09-07T21:39:46.793498+0000","last_clean":"2022-09-07T21:39:46.793498+0000","last_became_active":"2022-09-07T21:38:44.995738+0000","last_became_peered":"2022-09-07T21:38:44.995738+0000","last_unstale":"2022-09-07T21:39:46.793498+0000","last_undegraded":"2022-09-07T21:39:46.793498+0000","last_fullsized":"2022-09-07T21:39:46.793498+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:36:49.744364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788166+0000","last_change":"2022-09-07T21:38:39.965087+0000","last_active":"2022-09-07T21:39:46.788166+0000","last_peered":"2022-09-07T21:39:46.788166+0000","last_clean":"2022-09-07T21:39:46.788166+0000","last_became_active":"2022-09-07T21:38:39.964681+0000","last_became_peered":"2022-09-07T21:38:39.964681+0000","last_unstale":"2022-09-07T21:39:46.788166+0000","last_undegraded":"2022-09-07T21:39:46.788166+0000","last_fullsized":"2022-09-07T21:39:46.788166+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:47:50.028036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.786720+0000","last_change":"2022-09-07T21:38:40.564639+0000","last_active":"2022-09-07T21:39:46.786720+0000","last_peered":"2022-09-07T21:39:46.786720+0000","last_clean":"2022-09-07T21:39:46.786720+0000","last_became_active":"2022-09-07T21:38:40.564315+0000","last_became_peered":"2022-09-07T21:38:40.564315+0000","last_unstale":"2022-09-07T21:39:46.786720+0000","last_undegraded":"2022-09-07T21:39:46.786720+0000","last_fullsized":"2022-09-07T21:39:46.786720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:20:27.670309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:48.010329+0000","last_change":"2022-09-07T21:39:48.010329+0000","last_active":"2022-09-07T21:39:48.010329+0000","last_peered":"2022-09-07T21:39:48.010329+0000","last_clean":"2022-09-07T21:39:48.010329+0000","last_became_active":"2022-09-07T21:39:48.009972+0000","last_became_peered":"2022-09-07T21:39:48.009972+0000","last_unstale":"2022-09-07T21:39:48.010329+0000","last_undegraded":"2022-09-07T21:39:48.010329+0000","last_fullsized":"2022-09-07T21:39:48.010329+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:59:50.655757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784630+0000","last_change":"2022-09-07T21:38:39.967488+0000","last_active":"2022-09-07T21:39:46.784630+0000","last_peered":"2022-09-07T21:39:46.784630+0000","last_clean":"2022-09-07T21:39:46.784630+0000","last_became_active":"2022-09-07T21:38:39.967162+0000","last_became_peered":"2022-09-07T21:38:39.967162+0000","last_unstale":"2022-09-07T21:39:46.784630+0000","last_undegraded":"2022-09-07T21:39:46.784630+0000","last_fullsized":"2022-09-07T21:39:46.784630+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:43:43.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790023+0000","last_change":"2022-09-07T21:38:40.568656+0000","last_active":"2022-09-07T21:39:46.790023+0000","last_peered":"2022-09-07T21:39:46.790023+0000","last_clean":"2022-09-07T21:39:46.790023+0000","last_became_active":"2022-09-07T21:38:40.568357+0000","last_became_peered":"2022-09-07T21:38:40.568357+0000","last_unstale":"2022-09-07T21:39:46.790023+0000","last_undegraded":"2022-09-07T21:39:46.790023+0000","last_fullsized":"2022-09-07T21:39:46.790023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:31:32.498945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793066+0000","last_change":"2022-09-07T21:38:44.995472+0000","last_active":"2022-09-07T21:39:46.793066+0000","last_peered":"2022-09-07T21:39:46.793066+0000","last_clean":"2022-09-07T21:39:46.793066+0000","last_became_active":"2022-09-07T21:38:44.995030+0000","last_became_peered":"2022-09-07T21:38:44.995030+0000","last_unstale":"2022-09-07T21:39:46.793066+0000","last_undegraded":"2022-09-07T21:39:46.793066+0000","last_fullsized":"2022-09-07T21:39:46.793066+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:39:41.557547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793396+0000","last_change":"2022-09-07T21:38:39.969316+0000","last_active":"2022-09-07T21:39:46.793396+0000","last_peered":"2022-09-07T21:39:46.793396+0000","last_clean":"2022-09-07T21:39:46.793396+0000","last_became_active":"2022-09-07T21:38:39.968958+0000","last_became_peered":"2022-09-07T21:38:39.968958+0000","last_unstale":"2022-09-07T21:39:46.793396+0000","last_undegraded":"2022-09-07T21:39:46.793396+0000","last_fullsized":"2022-09-07T21:39:46.793396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:51:01.568029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.785307+0000","last_change":"2022-09-07T21:38:40.579479+0000","last_active":"2022-09-07T21:39:46.785307+0000","last_peered":"2022-09-07T21:39:46.785307+0000","last_clean":"2022-09-07T21:39:46.785307+0000","last_became_active":"2022-09-07T21:38:40.578452+0000","last_became_peered":"2022-09-07T21:38:40.578452+0000","last_unstale":"2022-09-07T21:39:46.785307+0000","last_undegraded":"2022-09-07T21:39:46.785307+0000","last_fullsized":"2022-09-07T21:39:46.785307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:34:24.096814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793294+0000","last_change":"2022-09-07T21:38:44.988455+0000","last_active":"2022-09-07T21:39:46.793294+0000","last_peered":"2022-09-07T21:39:46.793294+0000","last_clean":"2022-09-07T21:39:46.793294+0000","last_became_active":"2022-09-07T21:38:44.988070+0000","last_became_peered":"2022-09-07T21:38:44.988070+0000","last_unstale":"2022-09-07T21:39:46.793294+0000","last_undegraded":"2022-09-07T21:39:46.793294+0000","last_fullsized":"2022-09-07T21:39:46.793294+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:52:37.094892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.786944+0000","last_change":"2022-09-07T21:38:39.972219+0000","last_active":"2022-09-07T21:39:46.786944+0000","last_peered":"2022-09-07T21:39:46.786944+0000","last_clean":"2022-09-07T21:39:46.786944+0000","last_became_active":"2022-09-07T21:38:39.971484+0000","last_became_peered":"2022-09-07T21:38:39.971484+0000","last_unstale":"2022-09-07T21:39:46.786944+0000","last_undegraded":"2022-09-07T21:39:46.786944+0000","last_fullsized":"2022-09-07T21:39:46.786944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:54:17.947316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789667+0000","last_change":"2022-09-07T21:38:40.578389+0000","last_active":"2022-09-07T21:39:46.789667+0000","last_peered":"2022-09-07T21:39:46.789667+0000","last_clean":"2022-09-07T21:39:46.789667+0000","last_became_active":"2022-09-07T21:38:40.577886+0000","last_became_peered":"2022-09-07T21:38:40.577886+0000","last_unstale":"2022-09-07T21:39:46.789667+0000","last_undegraded":"2022-09-07T21:39:46.789667+0000","last_fullsized":"2022-09-07T21:39:46.789667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:48:07.590464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.787982+0000","last_change":"2022-09-07T21:38:44.994968+0000","last_active":"2022-09-07T21:39:46.787982+0000","last_peered":"2022-09-07T21:39:46.787982+0000","last_clean":"2022-09-07T21:39:46.787982+0000","last_became_active":"2022-09-07T21:38:44.994516+0000","last_became_peered":"2022-09-07T21:38:44.994516+0000","last_unstale":"2022-09-07T21:39:46.787982+0000","last_undegraded":"2022-09-07T21:39:46.787982+0000","last_fullsized":"2022-09-07T21:39:46.787982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T00:49:10.172571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793679+0000","last_change":"2022-09-07T21:38:39.970117+0000","last_active":"2022-09-07T21:39:46.793679+0000","last_peered":"2022-09-07T21:39:46.793679+0000","last_clean":"2022-09-07T21:39:46.793679+0000","last_became_active":"2022-09-07T21:38:39.966606+0000","last_became_peered":"2022-09-07T21:38:39.966606+0000","last_unstale":"2022-09-07T21:39:46.793679+0000","last_undegraded":"2022-09-07T21:39:46.793679+0000","last_fullsized":"2022-09-07T21:39:46.793679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:59:13.103878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.791392+0000","last_change":"2022-09-07T21:38:40.563232+0000","last_active":"2022-09-07T21:39:46.791392+0000","last_peered":"2022-09-07T21:39:46.791392+0000","last_clean":"2022-09-07T21:39:46.791392+0000","last_became_active":"2022-09-07T21:38:40.562945+0000","last_became_peered":"2022-09-07T21:38:40.562945+0000","last_unstale":"2022-09-07T21:39:46.791392+0000","last_undegraded":"2022-09-07T21:39:46.791392+0000","last_fullsized":"2022-09-07T21:39:46.791392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:42:44.271490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.783840+0000","last_change":"2022-09-07T21:38:44.996941+0000","last_active":"2022-09-07T21:39:46.783840+0000","last_peered":"2022-09-07T21:39:46.783840+0000","last_clean":"2022-09-07T21:39:46.783840+0000","last_became_active":"2022-09-07T21:38:44.988680+0000","last_became_peered":"2022-09-07T21:38:44.988680+0000","last_unstale":"2022-09-07T21:39:46.783840+0000","last_undegraded":"2022-09-07T21:39:46.783840+0000","last_fullsized":"2022-09-07T21:39:46.783840+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:33:12.017654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790387+0000","last_change":"2022-09-07T21:38:39.969588+0000","last_active":"2022-09-07T21:39:46.790387+0000","last_peered":"2022-09-07T21:39:46.790387+0000","last_clean":"2022-09-07T21:39:46.790387+0000","last_became_active":"2022-09-07T21:38:39.969222+0000","last_became_peered":"2022-09-07T21:38:39.969222+0000","last_unstale":"2022-09-07T21:39:46.790387+0000","last_undegraded":"2022-09-07T21:39:46.790387+0000","last_fullsized":"2022-09-07T21:39:46.790387+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:46:36.580497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790363+0000","last_change":"2022-09-07T21:38:40.569663+0000","last_active":"2022-09-07T21:39:46.790363+0000","last_peered":"2022-09-07T21:39:46.790363+0000","last_clean":"2022-09-07T21:39:46.790363+0000","last_became_active":"2022-09-07T21:38:40.569376+0000","last_became_peered":"2022-09-07T21:38:40.569376+0000","last_unstale":"2022-09-07T21:39:46.790363+0000","last_undegraded":"2022-09-07T21:39:46.790363+0000","last_fullsized":"2022-09-07T21:39:46.790363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T08:06:01.895953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790372+0000","last_change":"2022-09-07T21:38:44.987761+0000","last_active":"2022-09-07T21:39:46.790372+0000","last_peered":"2022-09-07T21:39:46.790372+0000","last_clean":"2022-09-07T21:39:46.790372+0000","last_became_active":"2022-09-07T21:38:44.987376+0000","last_became_peered":"2022-09-07T21:38:44.987376+0000","last_unstale":"2022-09-07T21:39:46.790372+0000","last_undegraded":"2022-09-07T21:39:46.790372+0000","last_fullsized":"2022-09-07T21:39:46.790372+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:43.955705+0000","last_clean_scrub_stamp":"2022-09-07T21:38:43.955705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:37:19.993838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788457+0000","last_change":"2022-09-07T21:38:39.968703+0000","last_active":"2022-09-07T21:39:46.788457+0000","last_peered":"2022-09-07T21:39:46.788457+0000","last_clean":"2022-09-07T21:39:46.788457+0000","last_became_active":"2022-09-07T21:38:39.968393+0000","last_became_peered":"2022-09-07T21:38:39.968393+0000","last_unstale":"2022-09-07T21:39:46.788457+0000","last_undegraded":"2022-09-07T21:39:46.788457+0000","last_fullsized":"2022-09-07T21:39:46.788457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:38.651764+0000","last_clean_scrub_stamp":"2022-09-07T21:38:38.651764+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:46:36.042794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.789617+0000","last_change":"2022-09-07T21:38:40.567044+0000","last_active":"2022-09-07T21:39:46.789617+0000","last_peered":"2022-09-07T21:39:46.789617+0000","last_clean":"2022-09-07T21:39:46.789617+0000","last_became_active":"2022-09-07T21:38:40.566732+0000","last_became_peered":"2022-09-07T21:38:40.566732+0000","last_unstale":"2022-09-07T21:39:46.789617+0000","last_undegraded":"2022-09-07T21:39:46.789617+0000","last_fullsized":"2022-09-07T21:39:46.789617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T05:12:52.569238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790103+0000","last_change":"2022-09-07T21:38:40.563301+0000","last_active":"2022-09-07T21:39:46.790103+0000","last_peered":"2022-09-07T21:39:46.790103+0000","last_clean":"2022-09-07T21:39:46.790103+0000","last_became_active":"2022-09-07T21:38:40.562979+0000","last_became_peered":"2022-09-07T21:38:40.562979+0000","last_unstale":"2022-09-07T21:39:46.790103+0000","last_undegraded":"2022-09-07T21:39:46.790103+0000","last_fullsized":"2022-09-07T21:39:46.790103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:21:46.453495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-07T21:39:47.793099+0000","last_change":"2022-09-07T21:39:47.793099+0000","last_active":"2022-09-07T21:39:47.793099+0000","last_peered":"2022-09-07T21:39:47.793099+0000","last_clean":"2022-09-07T21:39:47.793099+0000","last_became_active":"2022-09-07T21:39:47.792757+0000","last_became_peered":"2022-09-07T21:39:47.792757+0000","last_unstale":"2022-09-07T21:39:47.793099+0000","last_undegraded":"2022-09-07T21:39:47.793099+0000","last_fullsized":"2022-09-07T21:39:47.793099+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:40:12.701597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793295+0000","last_change":"2022-09-07T21:38:40.576602+0000","last_active":"2022-09-07T21:39:46.793295+0000","last_peered":"2022-09-07T21:39:46.793295+0000","last_clean":"2022-09-07T21:39:46.793295+0000","last_became_active":"2022-09-07T21:38:40.576296+0000","last_became_peered":"2022-09-07T21:38:40.576296+0000","last_unstale":"2022-09-07T21:39:46.793295+0000","last_undegraded":"2022-09-07T21:39:46.793295+0000","last_fullsized":"2022-09-07T21:39:46.793295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:11:43.175430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784035+0000","last_change":"2022-09-07T21:38:40.584214+0000","last_active":"2022-09-07T21:39:46.784035+0000","last_peered":"2022-09-07T21:39:46.784035+0000","last_clean":"2022-09-07T21:39:46.784035+0000","last_became_active":"2022-09-07T21:38:40.583908+0000","last_became_peered":"2022-09-07T21:38:40.583908+0000","last_unstale":"2022-09-07T21:39:46.784035+0000","last_undegraded":"2022-09-07T21:39:46.784035+0000","last_fullsized":"2022-09-07T21:39:46.784035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:40:54.023025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788732+0000","last_change":"2022-09-07T21:38:40.571163+0000","last_active":"2022-09-07T21:39:46.788732+0000","last_peered":"2022-09-07T21:39:46.788732+0000","last_clean":"2022-09-07T21:39:46.788732+0000","last_became_active":"2022-09-07T21:38:40.570802+0000","last_became_peered":"2022-09-07T21:38:40.570802+0000","last_unstale":"2022-09-07T21:39:46.788732+0000","last_undegraded":"2022-09-07T21:39:46.788732+0000","last_fullsized":"2022-09-07T21:39:46.788732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:03:14.067136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.792173+0000","last_change":"2022-09-07T21:38:40.561957+0000","last_active":"2022-09-07T21:39:46.792173+0000","last_peered":"2022-09-07T21:39:46.792173+0000","last_clean":"2022-09-07T21:39:46.792173+0000","last_became_active":"2022-09-07T21:38:40.561600+0000","last_became_peered":"2022-09-07T21:38:40.561600+0000","last_unstale":"2022-09-07T21:39:46.792173+0000","last_undegraded":"2022-09-07T21:39:46.792173+0000","last_fullsized":"2022-09-07T21:39:46.792173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:28:37.479711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.793515+0000","last_change":"2022-09-07T21:38:40.562779+0000","last_active":"2022-09-07T21:39:46.793515+0000","last_peered":"2022-09-07T21:39:46.793515+0000","last_clean":"2022-09-07T21:39:46.793515+0000","last_became_active":"2022-09-07T21:38:40.562472+0000","last_became_peered":"2022-09-07T21:38:40.562472+0000","last_unstale":"2022-09-07T21:39:46.793515+0000","last_undegraded":"2022-09-07T21:39:46.793515+0000","last_fullsized":"2022-09-07T21:39:46.793515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T09:13:22.395242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.785978+0000","last_change":"2022-09-07T21:38:40.570257+0000","last_active":"2022-09-07T21:39:46.785978+0000","last_peered":"2022-09-07T21:39:46.785978+0000","last_clean":"2022-09-07T21:39:46.785978+0000","last_became_active":"2022-09-07T21:38:40.569930+0000","last_became_peered":"2022-09-07T21:38:40.569930+0000","last_unstale":"2022-09-07T21:39:46.785978+0000","last_undegraded":"2022-09-07T21:39:46.785978+0000","last_fullsized":"2022-09-07T21:39:46.785978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T01:52:03.810386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790960+0000","last_change":"2022-09-07T21:38:40.578796+0000","last_active":"2022-09-07T21:39:46.790960+0000","last_peered":"2022-09-07T21:39:46.790960+0000","last_clean":"2022-09-07T21:39:46.790960+0000","last_became_active":"2022-09-07T21:38:40.578492+0000","last_became_peered":"2022-09-07T21:38:40.578492+0000","last_unstale":"2022-09-07T21:39:46.790960+0000","last_undegraded":"2022-09-07T21:39:46.790960+0000","last_fullsized":"2022-09-07T21:39:46.790960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T22:49:50.549224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.784727+0000","last_change":"2022-09-07T21:38:40.571320+0000","last_active":"2022-09-07T21:39:46.784727+0000","last_peered":"2022-09-07T21:39:46.784727+0000","last_clean":"2022-09-07T21:39:46.784727+0000","last_became_active":"2022-09-07T21:38:40.570401+0000","last_became_peered":"2022-09-07T21:38:40.570401+0000","last_unstale":"2022-09-07T21:39:46.784727+0000","last_undegraded":"2022-09-07T21:39:46.784727+0000","last_fullsized":"2022-09-07T21:39:46.784727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T07:24:33.254833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788295+0000","last_change":"2022-09-07T21:38:40.571386+0000","last_active":"2022-09-07T21:39:46.788295+0000","last_peered":"2022-09-07T21:39:46.788295+0000","last_clean":"2022-09-07T21:39:46.788295+0000","last_became_active":"2022-09-07T21:38:40.571130+0000","last_became_peered":"2022-09-07T21:38:40.571130+0000","last_unstale":"2022-09-07T21:39:46.788295+0000","last_undegraded":"2022-09-07T21:39:46.788295+0000","last_fullsized":"2022-09-07T21:39:46.788295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T02:57:47.083547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.788559+0000","last_change":"2022-09-07T21:38:40.584706+0000","last_active":"2022-09-07T21:39:46.788559+0000","last_peered":"2022-09-07T21:39:46.788559+0000","last_clean":"2022-09-07T21:39:46.788559+0000","last_became_active":"2022-09-07T21:38:40.584193+0000","last_became_peered":"2022-09-07T21:38:40.584193+0000","last_unstale":"2022-09-07T21:39:46.788559+0000","last_undegraded":"2022-09-07T21:39:46.788559+0000","last_fullsized":"2022-09-07T21:39:46.788559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-08T23:10:32.597608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-07T21:39:46.790426+0000","last_change":"2022-09-07T21:38:40.569545+0000","last_active":"2022-09-07T21:39:46.790426+0000","last_peered":"2022-09-07T21:39:46.790426+0000","last_clean":"2022-09-07T21:39:46.790426+0000","last_became_active":"2022-09-07T21:38:40.569218+0000","last_became_peered":"2022-09-07T21:38:40.569218+0000","last_unstale":"2022-09-07T21:39:46.790426+0000","last_undegraded":"2022-09-07T21:39:46.790426+0000","last_fullsized":"2022-09-07T21:39:46.790426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-07T21:38:39.936880+0000","last_clean_scrub_stamp":"2022-09-07T21:38:39.936880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-09T04:40:39.949203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":219246,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65117,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":342,"num_read_kb":249,"num_write":158,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1533,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":195351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383709,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6100,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739052,"statfs":{"total":95995035648,"available":95988789248,"internally_reserved":0,"allocated":995328,"data_stored":498927,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58699999999999997,"15min":0.58799999999999997},"min":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60599999999999998,"15min":0.61099999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.755,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.55000000000000004}]},{"osd":1,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.625,"15min":0.61599999999999999},"min":{"1min":0.45000000000000001,"5min":0.436,"15min":0.436},"max":{"1min":0.875,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.61199999999999999,"15min":0.59799999999999998},"min":{"1min":0.441,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.67600000000000005}]},{"osd":2,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.498,"15min":0.49099999999999999},"min":{"1min":0.376,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.50600000000000001,"15min":0.50900000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72999999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.43099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":4,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.64600000000000002,"15min":0.64100000000000001},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.67300000000000004,"15min":0.65900000000000003},"min":{"1min":0.49099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.64600000000000002}]},{"osd":5,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.61299999999999999,"15min":0.61499999999999999},"min":{"1min":0.46100000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80700000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.38500000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80800000000000005,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.53300000000000003}]},{"osd":6,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.49199999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.749,"5min":0.80900000000000005,"15min":0.81899999999999995},"min":{"1min":0.52700000000000002,"5min":0.5,"15min":0.5},"max":{"1min":0.93799999999999994,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.621}]},{"osd":7,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.75700000000000001,"15min":0.755},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.97199999999999998,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.79900000000000004,"15min":0.79200000000000004},"min":{"1min":0.52200000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":0.70599999999999996}]},{"osd":8,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.66000000000000003}]},{"osd":9,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81799999999999995,"15min":0.81999999999999995},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.98499999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.83399999999999996,"15min":0.82399999999999995},"min":{"1min":0.65500000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":0.78300000000000003}]},{"osd":10,"last update":"Wed Sep 7 21:39:52 2022","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.83199999999999996,"15min":0.81999999999999995},"min":{"1min":0.63,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.54800000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.052,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.72999999999999998}]}]},{"osd":10,"up_from":72,"seq":309237645346,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6560,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738592,"statfs":{"total":95995035648,"available":95988318208,"internally_reserved":0,"allocated":1138688,"data_stored":1147405,"data_compressed":54425,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:44 2022","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.71399999999999997}]},{"osd":1,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.46100000000000002,"15min":0.46500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.60499999999999998}]},{"osd":2,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59199999999999997,"15min":0.58399999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.59399999999999997,"15min":0.58399999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70599999999999996}]},{"osd":3,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.64900000000000002,"15min":0.64300000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.69499999999999995,"15min":0.67500000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":3.794,"5min":3.794,"15min":3.794},"last":0.78100000000000003}]},{"osd":4,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.57699999999999996,"15min":0.58399999999999996},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78400000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.877,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.63500000000000001}]},{"osd":5,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67800000000000005,"15min":0.67600000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.879},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69699999999999995,"15min":0.69599999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.77200000000000002}]},{"osd":6,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.80300000000000005,"15min":0.78900000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":3.8900000000000001,"5min":3.8900000000000001,"15min":3.8900000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.79200000000000004,"15min":0.79000000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.82899999999999996}]},{"osd":7,"last update":"Wed Sep 7 21:38:27 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.42399999999999999}]},{"osd":8,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.75800000000000001,"15min":0.75900000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.98199999999999998,"5min":0.997,"15min":0.997},"last":0.871},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.79700000000000004,"15min":0.79900000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.91900000000000004}]},{"osd":9,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.85899999999999999,"15min":0.83299999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":4.9989999999999997,"5min":4.9989999999999997,"15min":4.9989999999999997},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84999999999999998,"15min":0.85299999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.95899999999999996}]},{"osd":11,"last update":"Wed Sep 7 21:39:57 2022","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.78000000000000003,"15min":0.78400000000000003},"min":{"1min":0.51200000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0069999999999999,"5min":1.103,"15min":1.103},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80600000000000005,"15min":0.80700000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98899999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.90600000000000003}]}]},{"osd":9,"up_from":67,"seq":287762808871,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6312,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738840,"statfs":{"total":95995035648,"available":95988572160,"internally_reserved":0,"allocated":1015808,"data_stored":503259,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81999999999999995,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.67300000000000004,"15min":0.67900000000000005},"min":{"1min":0.47499999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.012,"5min":1.175,"15min":1.175},"last":0.71099999999999997}]},{"osd":1,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.77000000000000002,"15min":0.77600000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.236,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.84799999999999998,"15min":0.85399999999999998},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.751,"15min":0.75700000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.999,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.86299999999999999,"15min":0.88600000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0029999999999999,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.81299999999999994}]},{"osd":3,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.81000000000000005,"15min":0.81599999999999995},"min":{"1min":0.499,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0980000000000001,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.79300000000000004,"15min":0.80600000000000005},"min":{"1min":0.48999999999999999,"5min":0.434,"15min":0.434},"max":{"1min":1.04,"5min":1.613,"15min":1.613},"last":0.51300000000000001}]},{"osd":4,"last update":"Wed Sep 7 21:39:46 2022","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.75},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.734,"5min":1.734,"15min":1.734},"last":0.79600000000000004}]},{"osd":5,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.90400000000000003,"15min":0.91700000000000004},"min":{"1min":0.58499999999999996,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0609999999999999,"5min":1.55,"15min":1.55},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.88100000000000001,"15min":0.89200000000000002},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.083,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.73599999999999999}]},{"osd":6,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":0.81200000000000006}]},{"osd":7,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.0609999999999999,"15min":1.0529999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":5.6529999999999996,"5min":5.6529999999999996,"15min":5.6529999999999996},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.98099999999999998,"15min":0.98699999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.337,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":0.67900000000000005}]},{"osd":8,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.0169999999999999,"15min":1.026},"min":{"1min":0.66400000000000003,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.3010000000000002,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.052,"15min":1.0720000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.1479999999999999,"5min":1.76,"15min":1.76},"last":0.88}]},{"osd":10,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.97299999999999998,"15min":0.98499999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.224,"5min":1.768,"15min":1.768},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.071,"5min":1.044,"15min":1.04},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"last":0.92500000000000004}]},{"osd":11,"last update":"Wed Sep 7 21:39:46 2022","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.3180000000000001,"5min":2.3180000000000001,"15min":2.3180000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.89000000000000001}]}]},{"osd":8,"up_from":59,"seq":253403070510,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6432,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738720,"statfs":{"total":95995035648,"available":95988449280,"internally_reserved":0,"allocated":1007616,"data_stored":498517,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.63100000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78600000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.58499999999999996,"15min":0.59299999999999997},"min":{"1min":0.42399999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.75600000000000001,"5min":0.876,"15min":0.876},"last":1.383}]},{"osd":1,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.622,"15min":0.628},"min":{"1min":0.39900000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.623,"15min":0.625},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.89700000000000002,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.63900000000000001}]},{"osd":2,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48499999999999999,"15min":0.495},"min":{"1min":0.34100000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70699999999999996,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.439},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.50700000000000001,"15min":0.51900000000000002},"min":{"1min":0.314,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68799999999999994,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.47999999999999998}]},{"osd":3,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.68899999999999995,"15min":0.65700000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":4.4969999999999999,"5min":4.4969999999999999,"15min":4.4969999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.71399999999999997,"15min":0.68500000000000005},"min":{"1min":0.42099999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":5.4059999999999997,"5min":5.4059999999999997,"15min":5.4059999999999997},"last":0.68799999999999994}]},{"osd":4,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.51300000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.66200000000000003,"15min":0.66400000000000003},"min":{"1min":0.52600000000000002,"5min":0.433,"15min":0.433},"max":{"1min":0.91100000000000003,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.624}]},{"osd":5,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.628,"15min":0.65100000000000002},"min":{"1min":0.34999999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.84499999999999997,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60699999999999998,"15min":0.61699999999999999},"min":{"1min":0.378,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.85599999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.46400000000000002}]},{"osd":6,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.78000000000000003,"15min":0.75800000000000001},"min":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":4.7300000000000004,"5min":4.7300000000000004,"15min":4.7300000000000004},"last":1.5940000000000001},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.80200000000000005,"15min":0.79300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.9729999999999999,"5min":2.9729999999999999,"15min":2.9729999999999999},"last":0.67700000000000005}]},{"osd":7,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.745,"15min":0.75900000000000001},"min":{"1min":0.437,"5min":0.378,"15min":0.378},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72699999999999998,"15min":0.73099999999999998},"min":{"1min":0.497,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.97499999999999998,"5min":1,"15min":1},"last":0.70899999999999996}]},{"osd":9,"last update":"Wed Sep 7 21:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.755,"15min":0.748},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.75900000000000001,"15min":0.752},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.79200000000000004}]},{"osd":10,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.73299999999999998,"15min":0.71699999999999997},"min":{"1min":0.51700000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.755},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.749,"15min":0.73899999999999999},"min":{"1min":0.52300000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0169999999999999,"5min":1.02,"15min":1.02},"last":0.73899999999999999}]},{"osd":11,"last update":"Wed Sep 7 21:39:42 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.82399999999999995}]}]},{"osd":7,"up_from":51,"seq":219043332148,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6612,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738540,"statfs":{"total":95995035648,"available":95988264960,"internally_reserved":0,"allocated":1060864,"data_stored":1095291,"data_compressed":54425,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58499999999999996,"15min":0.57299999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.54200000000000004}]},{"osd":1,"last update":"Wed Sep 7 21:40:03 2022","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.44,"15min":0.434},"min":{"1min":0.307,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.52000000000000002,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.307},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45500000000000002,"15min":0.45200000000000001},"min":{"1min":0.37,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.59999999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.505}]},{"osd":2,"last update":"Wed Sep 7 21:40:03 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.39300000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.69499999999999995,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.67600000000000005,"15min":0.629},"min":{"1min":0.55000000000000004,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78200000000000003,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":0.61699999999999999}]},{"osd":3,"last update":"Wed Sep 7 21:40:03 2022","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.629,"15min":0.60899999999999999},"min":{"1min":0.45400000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.70299999999999996,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63900000000000001,"15min":0.626},"min":{"1min":0.52400000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.74199999999999999,"5min":1.091,"15min":1.091},"last":0.74199999999999999}]},{"osd":4,"last update":"Wed Sep 7 21:40:03 2022","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.38300000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.72699999999999998,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52000000000000002,"15min":0.52600000000000002},"min":{"1min":0.33700000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65300000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.33700000000000002}]},{"osd":5,"last update":"Wed Sep 7 21:40:03 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65200000000000002,"15min":0.66100000000000003},"min":{"1min":0.45200000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.77400000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67100000000000004},"min":{"1min":0.46600000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83099999999999996,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.46600000000000003}]},{"osd":6,"last update":"Wed Sep 7 21:40:03 2022","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72699999999999998,"15min":0.73499999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.86099999999999999,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.70599999999999996,"15min":0.69799999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.88,"5min":1.079,"15min":1.079},"last":0.65300000000000002}]},{"osd":8,"last update":"Wed Sep 7 21:39:27 2022","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71299999999999997,"15min":0.71699999999999997},"min":{"1min":0.46600000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.86099999999999999,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.71899999999999997,"15min":0.72499999999999998},"min":{"1min":0.51300000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.83199999999999996,"5min":1.123,"15min":1.123},"last":0.72199999999999998}]},{"osd":9,"last update":"Wed Sep 7 21:40:03 2022","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.78300000000000003,"15min":0.79700000000000004},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.84899999999999998,"5min":1.036,"15min":1.036},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.81699999999999995,"15min":0.83499999999999996},"min":{"1min":0.64100000000000001,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.90700000000000003,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.82299999999999995}]},{"osd":10,"last update":"Wed Sep 7 21:39:32 2022","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.73199999999999998,"15min":0.73999999999999999},"min":{"1min":0.51100000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.92000000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.93600000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.71099999999999997}]},{"osd":11,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.80900000000000005}]}]},{"osd":6,"up_from":46,"seq":197568495673,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6416,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738736,"statfs":{"total":95995035648,"available":95988465664,"internally_reserved":0,"allocated":991232,"data_stored":494355,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56000000000000005,"15min":0.54200000000000004},"min":{"1min":0.40999999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.40000000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.78300000000000003,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.45900000000000002}]},{"osd":1,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.65500000000000003,"15min":0.66100000000000003},"min":{"1min":0.48299999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86099999999999999,"5min":1.127,"15min":1.127},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67100000000000004,"15min":0.66100000000000003},"min":{"1min":0.41899999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.94099999999999995,"5min":1.159,"15min":1.159},"last":0.66200000000000003}]},{"osd":2,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.71899999999999997,"15min":0.71499999999999997},"min":{"1min":0.496,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.97099999999999997,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.034,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.70799999999999996}]},{"osd":3,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68500000000000005,"15min":0.65900000000000003},"min":{"1min":0.47499999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.90500000000000003,"5min":1.151,"15min":1.151},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68799999999999994,"15min":0.68999999999999995},"min":{"1min":0.46700000000000003,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.879,"5min":1.153,"15min":1.153},"last":0.89100000000000001}]},{"osd":4,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.746,"15min":0.754},"min":{"1min":0.51900000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.98199999999999998,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.54800000000000004,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.90500000000000003,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.81599999999999995}]},{"osd":5,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.77300000000000002,"15min":0.76100000000000001},"min":{"1min":0.61799999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96899999999999997,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.81799999999999995,"15min":0.82999999999999996},"min":{"1min":0.49099999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.99399999999999999,"5min":1.341,"15min":1.341},"last":0.79200000000000004}]},{"osd":7,"last update":"Wed Sep 7 21:39:07 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.81000000000000005,"15min":0.80600000000000005},"min":{"1min":0.57099999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.044,"5min":1.212,"15min":1.212},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.82999999999999996,"15min":0.82199999999999995},"min":{"1min":0.64800000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.98699999999999999,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.90900000000000003}]},{"osd":8,"last update":"Wed Sep 7 21:39:28 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84599999999999997,"15min":0.83599999999999997},"min":{"1min":0.63800000000000001,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.016,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.871,"15min":0.85599999999999998},"min":{"1min":0.69099999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.026,"5min":1.268,"15min":1.268},"last":0.83099999999999996}]},{"osd":9,"last update":"Wed Sep 7 21:39:02 2022","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.96799999999999997,"15min":0.97399999999999998},"min":{"1min":0.77700000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.133,"5min":1.254,"15min":1.254},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.877,"5min":0.88800000000000001,"15min":0.89000000000000001},"min":{"1min":0.65300000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1439999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.0549999999999999}]},{"osd":10,"last update":"Wed Sep 7 21:39:28 2022","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.93100000000000005,"15min":0.94599999999999995},"min":{"1min":0.70999999999999996,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":0.97899999999999998,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.98699999999999999,"15min":0.995},"min":{"1min":0.77500000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.0820000000000001,"5min":1.218,"15min":1.218},"last":0.94599999999999995}]},{"osd":11,"last update":"Wed Sep 7 21:39:44 2022","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.873},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.92300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7508,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737644,"statfs":{"total":95995035648,"available":95987347456,"internally_reserved":0,"allocated":1060864,"data_stored":555696,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:13 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55300000000000005,"15min":0.51500000000000001},"min":{"1min":0.38100000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.78200000000000003,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55800000000000005,"15min":0.52600000000000002},"min":{"1min":0.39100000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.73399999999999999,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.61499999999999999}]},{"osd":2,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.57999999999999996,"15min":0.53100000000000003},"min":{"1min":0.38200000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83199999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.56599999999999995,"15min":0.52300000000000002},"min":{"1min":0.39400000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.73299999999999998}]},{"osd":3,"last update":"Wed Sep 7 21:39:13 2022","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.57999999999999996,"15min":0.55400000000000005},"min":{"1min":0.41999999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.83799999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59999999999999998,"15min":0.58599999999999997},"min":{"1min":0.42999999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79400000000000004,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.71999999999999997}]},{"osd":4,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46300000000000002,"15min":0.44700000000000001},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72999999999999998,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.504},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.498,"15min":0.46800000000000003},"min":{"1min":0.309,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81399999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.54500000000000004}]},{"osd":5,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.64400000000000002,"15min":0.61099999999999999},"min":{"1min":0.38900000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.63300000000000001,"15min":0.59799999999999998},"min":{"1min":0.51400000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82099999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.64200000000000002}]},{"osd":6,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68200000000000005,"15min":0.66500000000000004},"min":{"1min":0.49299999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85799999999999998,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.69699999999999995,"15min":0.68700000000000006},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.873,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.81599999999999995}]},{"osd":7,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.622,"15min":0.622},"min":{"1min":0.45800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82899999999999996,"5min":1.232,"15min":1.232},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.66100000000000003},"min":{"1min":0.40400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.874,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.75700000000000001}]},{"osd":8,"last update":"Wed Sep 7 21:39:32 2022","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.71799999999999997,"15min":0.70199999999999996},"min":{"1min":0.54900000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.96199999999999997,"5min":1.091,"15min":1.091},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.71499999999999997,"15min":0.68500000000000005},"min":{"1min":0.56000000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94499999999999995,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.77800000000000002}]},{"osd":9,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.57599999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.90200000000000002,"5min":1.032,"15min":1.032},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.77700000000000002,"15min":0.77900000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.91600000000000004,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.872}]},{"osd":10,"last update":"Wed Sep 7 21:39:31 2022","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.79000000000000004,"15min":0.80000000000000004},"min":{"1min":0.49099999999999999,"5min":0.437,"15min":0.437},"max":{"1min":0.92300000000000004,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.78100000000000003,"15min":0.78500000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.98699999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.80500000000000005}]},{"osd":11,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.83099999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7400,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737752,"statfs":{"total":95995035648,"available":95987458048,"internally_reserved":0,"allocated":1015808,"data_stored":495711,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 7 21:39:13 2022","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.54500000000000004},"min":{"1min":0.5,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76600000000000001,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.623,"15min":0.56100000000000005},"min":{"1min":0.41499999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.2509999999999999,"5min":1.26,"15min":1.26},"last":0.54500000000000004}]},{"osd":2,"last update":"Wed Sep 7 21:39:37 2022","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":0.76200000000000001,"15min":0.65000000000000002},"min":{"1min":0.44400000000000001,"5min":0.316,"15min":0.316},"max":{"1min":12.795,"5min":12.795,"15min":12.795},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.66900000000000004,"15min":0.621},"min":{"1min":0.52500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.92200000000000004,"5min":1.075,"15min":1.075},"last":0.69399999999999995}]},{"osd":3,"last update":"Wed Sep 7 21:39:43 2022","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.98099999999999998}]},{"osd":4,"last update":"Wed Sep 7 21:39:43 2022","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.71799999999999997,"15min":0.66400000000000003},"min":{"1min":0.48699999999999999,"5min":0.438,"15min":0.438},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.72099999999999997,"15min":0.66300000000000003},"min":{"1min":0.50700000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93999999999999995,"5min":1.109,"15min":1.109},"last":0.66000000000000003}]},{"osd":5,"last update":"Wed Sep 7 21:39:07 2022","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":0.877,"15min":0.75700000000000001},"min":{"1min":0.68300000000000005,"5min":0.5,"15min":0.5},"max":{"1min":12.776999999999999,"5min":12.776999999999999,"15min":12.776999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.68700000000000006,"15min":0.66500000000000004},"min":{"1min":0.47199999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95699999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.751}]},{"osd":6,"last update":"Wed Sep 7 21:39:43 2022","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.73099999999999998,"15min":0.70899999999999996},"min":{"1min":0.48599999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.78500000000000003,"15min":0.76900000000000002},"min":{"1min":0.58599999999999997,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.0209999999999999}]},{"osd":7,"last update":"Wed Sep 7 21:39:47 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":12.808,"5min":12.808,"15min":12.808},"last":1.139}]},{"osd":8,"last update":"Wed Sep 7 21:39:30 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.89000000000000001,"15min":0.89200000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0349999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.84699999999999998,"15min":0.84399999999999997},"min":{"1min":0.65100000000000002,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0349999999999999,"5min":1.143,"15min":1.143},"last":0.78400000000000003}]},{"osd":9,"last update":"Wed Sep 7 21:40:03 2022","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92900000000000005,"15min":0.94799999999999995},"min":{"1min":0.65100000000000002,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.01,"5min":0.91700000000000004,"15min":0.91200000000000003},"min":{"1min":0.75,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":1.0700000000000001}]},{"osd":10,"last update":"Wed Sep 7 21:39:37 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.95099999999999996,"15min":0.95899999999999996},"min":{"1min":0.73499999999999999,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.2150000000000001,"5min":1.413,"15min":1.413},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.0960000000000001,"15min":1.0229999999999999},"min":{"1min":0.73199999999999998,"5min":0.504,"15min":0.504},"max":{"1min":12.837999999999999,"5min":12.837999999999999,"15min":12.837999999999999},"last":0.98699999999999999}]},{"osd":11,"last update":"Wed Sep 7 21:39:54 2022","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.89000000000000001,"15min":0.872},"min":{"1min":0.77000000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":1.105},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.76700000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.88200000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7584,"kb_used_data":1048,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737568,"statfs":{"total":95995035648,"available":95987269632,"internally_reserved":0,"allocated":1073152,"data_stored":555842,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:42 2022","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.55900000000000005,"15min":0.52100000000000002},"min":{"1min":0.41499999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.58099999999999996,"15min":0.53500000000000003},"min":{"1min":0.38900000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.6899999999999999,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":0.59699999999999998}]},{"osd":1,"last update":"Wed Sep 7 21:39:42 2022","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.54200000000000004,"15min":0.51500000000000001},"min":{"1min":0.36699999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.628},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.58299999999999996,"15min":0.51100000000000001},"min":{"1min":0.47599999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.80400000000000005}]},{"osd":3,"last update":"Wed Sep 7 21:39:12 2022","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.65700000000000003,"15min":0.57899999999999996},"min":{"1min":0.40300000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.95799999999999996,"5min":7.649,"15min":7.649},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.58199999999999996,"15min":0.54200000000000004},"min":{"1min":0.442,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.79900000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.77500000000000002}]},{"osd":4,"last update":"Wed Sep 7 21:39:42 2022","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.73799999999999999}]},{"osd":5,"last update":"Wed Sep 7 21:39:06 2022","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.47499999999999998,"15min":0.437},"min":{"1min":0.312,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.53600000000000003,"15min":0.49099999999999999},"min":{"1min":0.35599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":1.0660000000000001}]},{"osd":6,"last update":"Wed Sep 7 21:39:42 2022","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.72899999999999998,"15min":0.70299999999999996},"min":{"1min":0.63100000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.093},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.72099999999999997,"15min":0.69699999999999995},"min":{"1min":0.60099999999999998,"5min":0.498,"15min":0.498},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":1.056}]},{"osd":7,"last update":"Wed Sep 7 21:39:48 2022","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.94899999999999995}]},{"osd":8,"last update":"Wed Sep 7 21:39:31 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.65900000000000003,"15min":0.65100000000000002},"min":{"1min":0.48199999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.93600000000000005,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.65800000000000003,"15min":0.65400000000000003},"min":{"1min":0.497,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.67800000000000005}]},{"osd":9,"last update":"Wed Sep 7 21:39:05 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.77700000000000002,"15min":0.77000000000000002},"min":{"1min":0.54800000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76100000000000001,"15min":0.755},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":1.014}]},{"osd":10,"last update":"Wed Sep 7 21:39:31 2022","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.75700000000000001,"15min":0.747},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.81299999999999994,"15min":0.79400000000000004},"min":{"1min":0.67400000000000004,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":1.0309999999999999}]},{"osd":11,"last update":"Wed Sep 7 21:39:42 2022","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.84899999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6684,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738468,"statfs":{"total":95995035648,"available":95988191232,"internally_reserved":0,"allocated":1003520,"data_stored":494549,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:39:14 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.495,"15min":0.44700000000000001},"min":{"1min":0.375,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67500000000000004,"5min":1.167,"15min":1.167},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.499,"15min":0.46100000000000002},"min":{"1min":0.36099999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.80000000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.88500000000000001}]},{"osd":1,"last update":"Wed Sep 7 21:39:14 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.60299999999999998,"15min":0.57699999999999996},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.67900000000000005,"15min":0.60999999999999999},"min":{"1min":0.47399999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.92200000000000004,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.57199999999999995}]},{"osd":2,"last update":"Wed Sep 7 21:39:14 2022","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.68700000000000006,"15min":0.64200000000000002},"min":{"1min":0.48299999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94699999999999995,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.66400000000000003,"15min":0.627},"min":{"1min":0.42399999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.89800000000000002,"5min":1.302,"15min":1.302},"last":0.622}]},{"osd":4,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70599999999999996,"15min":0.64900000000000002},"min":{"1min":0.54100000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90600000000000003,"5min":1.286,"15min":1.286},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67300000000000004,"15min":0.63},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.84399999999999997,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.83899999999999997}]},{"osd":5,"last update":"Wed Sep 7 21:39:09 2022","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":0.80900000000000005,"15min":0.70899999999999996},"min":{"1min":0.48899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":11.808,"5min":11.808,"15min":11.808},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.77600000000000002,"15min":0.73799999999999999},"min":{"1min":0.501,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.97399999999999998,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.49199999999999999}]},{"osd":6,"last update":"Wed Sep 7 21:39:40 2022","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.80500000000000005,"15min":0.78300000000000003},"min":{"1min":0.629,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.96199999999999997,"5min":1.399,"15min":1.399},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.74399999999999999,"15min":0.73399999999999999},"min":{"1min":0.51200000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.995,"5min":1.157,"15min":1.157},"last":0.94599999999999995}]},{"osd":7,"last update":"Wed Sep 7 21:40:02 2022","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.56899999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.86499999999999999,"15min":0.85999999999999999},"min":{"1min":0.65700000000000003,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.083,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.68000000000000005}]},{"osd":8,"last update":"Wed Sep 7 21:38:30 2022","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82999999999999996,"15min":0.83099999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81299999999999994,"15min":0.81200000000000006},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.78400000000000003}]},{"osd":9,"last update":"Wed Sep 7 21:40:02 2022","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.95699999999999996,"15min":0.98499999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.409,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89800000000000002,"15min":0.91600000000000004},"min":{"1min":0.54500000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.98899999999999999}]},{"osd":10,"last update":"Wed Sep 7 21:39:33 2022","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.88500000000000001,"15min":0.89900000000000002},"min":{"1min":0.60399999999999998,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.1359999999999999,"5min":1.228,"15min":1.228},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.98299999999999998,"15min":0.99299999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.1160000000000001,"5min":1.486,"15min":1.486},"last":0.73399999999999999}]},{"osd":11,"last update":"Wed Sep 7 21:39:43 2022","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.90700000000000003}]}]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":1011712,"data_stored":500317,"data_compressed":17610,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:37:36 2022","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54400000000000004,"15min":0.52900000000000003},"min":{"1min":0.41199999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78100000000000003,"5min":0.879,"15min":0.879},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.52700000000000002,"15min":0.503},"min":{"1min":0.40699999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.35199999999999998}]},{"osd":1,"last update":"Wed Sep 7 21:39:41 2022","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.40500000000000003,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.41699999999999998,"15min":0.38100000000000001},"min":{"1min":0.34699999999999998,"5min":0.216,"15min":0.216},"max":{"1min":0.69299999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.54800000000000004}]},{"osd":2,"last update":"Wed Sep 7 21:39:41 2022","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56899999999999995,"15min":0.54700000000000004},"min":{"1min":0.42099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.59299999999999997,"15min":0.56000000000000005},"min":{"1min":0.55200000000000005,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.89100000000000001,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.71199999999999997}]},{"osd":3,"last update":"Wed Sep 7 21:39:41 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.61299999999999999,"15min":0.56999999999999995},"min":{"1min":0.42899999999999999,"5min":0.37,"15min":0.37},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.747},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.61499999999999999,"15min":0.59599999999999997},"min":{"1min":0.44600000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.65100000000000002}]},{"osd":5,"last update":"Wed Sep 7 21:39:10 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57199999999999995,"15min":0.55400000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.60899999999999999,"15min":0.58299999999999996},"min":{"1min":0.45500000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91200000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.66100000000000003}]},{"osd":6,"last update":"Wed Sep 7 21:39:41 2022","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.66400000000000003,"15min":0.64800000000000002},"min":{"1min":0.499,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.67500000000000004,"15min":0.65300000000000002},"min":{"1min":0.46400000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":0.73099999999999998}]},{"osd":7,"last update":"Wed Sep 7 21:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61699999999999999,"15min":0.61599999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.872,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.871},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76300000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.60599999999999998}]},{"osd":8,"last update":"Wed Sep 7 21:39:32 2022","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73399999999999999,"15min":0.73099999999999998},"min":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93300000000000005,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72099999999999997,"15min":0.72799999999999998},"min":{"1min":0.50800000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.878,"5min":1.198,"15min":1.198},"last":0.80300000000000005}]},{"osd":9,"last update":"Wed Sep 7 21:39:04 2022","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72099999999999997,"15min":0.72299999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.753,"5min":0.78100000000000003,"15min":0.78600000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.048,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.78200000000000003}]},{"osd":10,"last update":"Wed Sep 7 21:39:32 2022","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.74199999999999999,"15min":0.745},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.99199999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.67900000000000005,"15min":0.67100000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.82099999999999995}]},{"osd":11,"last update":"Wed Sep 7 21:39:39 2022","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.88300000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6680,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738472,"statfs":{"total":95995035648,"available":95988195328,"internally_reserved":0,"allocated":1064960,"data_stored":1090211,"data_compressed":54425,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 7 21:40:01 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56399999999999995,"15min":0.56299999999999994},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70999999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66500000000000004,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.57199999999999995}]},{"osd":1,"last update":"Wed Sep 7 21:40:01 2022","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.56599999999999995,"15min":0.57799999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.83899999999999997,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.35299999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75700000000000001,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.61099999999999999}]},{"osd":2,"last update":"Wed Sep 7 21:40:01 2022","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.432,"15min":0.438},"min":{"1min":0.27000000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.66700000000000004,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.501},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66600000000000004,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.441}]},{"osd":3,"last update":"Wed Sep 7 21:40:01 2022","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.80600000000000005,"5min":1.238,"15min":1.238},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.59499999999999997},"min":{"1min":0.435,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78400000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.78400000000000003}]},{"osd":4,"last update":"Wed Sep 7 21:40:01 2022","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61399999999999999,"15min":0.61699999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82399999999999995,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.752},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.60499999999999998,"15min":0.59799999999999998},"min":{"1min":0.41199999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82199999999999995,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.80200000000000005}]},{"osd":6,"last update":"Wed Sep 7 21:39:43 2022","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.66300000000000003,"15min":0.63100000000000001},"min":{"1min":0.442,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92400000000000004,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.68400000000000005,"15min":0.66400000000000003},"min":{"1min":0.502,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":0.69499999999999995}]},{"osd":7,"last update":"Wed Sep 7 21:40:01 2022","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.96499999999999997,"5min":1.002,"15min":1.002},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.66600000000000004,"15min":0.66800000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85799999999999998,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.55100000000000005}]},{"osd":8,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.64200000000000002,"15min":0.66000000000000003},"min":{"1min":0.38800000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85199999999999998,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68500000000000005,"15min":0.69699999999999995},"min":{"1min":0.53800000000000003,"5min":0.372,"15min":0.372},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.72599999999999998}]},{"osd":9,"last update":"Wed Sep 7 21:40:01 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.80300000000000005,"15min":0.81100000000000005},"min":{"1min":0.60099999999999998,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.78400000000000003,"15min":0.79000000000000004},"min":{"1min":0.54700000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.83899999999999997}]},{"osd":10,"last update":"Wed Sep 7 21:39:29 2022","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.72799999999999998,"15min":0.71799999999999997},"min":{"1min":0.63,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.996,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.61899999999999999,"5min":0.499,"15min":0.499},"max":{"1min":0.97199999999999998,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.872}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35171,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36815,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":61431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":61577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-07T21:40:06.964 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2022-09-07T21:40:06.964 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-09-07T21:40:06.965 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2022-09-07T21:40:06.965 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph health --format=json 2022-09-07T21:40:07.848 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:40:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:07 smithi059 ceph-mon[97450]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:40:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:07 smithi059 ceph-mon[97450]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:08.123 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:07 smithi047 ceph-mon[90087]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:40:08.124 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:07 smithi047 ceph-mon[90087]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:07 smithi055 ceph-mon[97033]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-07T21:40:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:07 smithi055 ceph-mon[97033]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:09.430 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.425+0000 7f20dbfff700 1 -- 172.21.15.47:0/2056353734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20dc0fee10 msgr2=0x7f20dc0ff1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:09.430 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.425+0000 7f20dbfff700 1 --2- 172.21.15.47:0/2056353734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20dc0fee10 0x7f20dc0ff1e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009810 tx=0x7f20cc009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:09.430 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.425+0000 7f20dbfff700 1 -- 172.21.15.47:0/2056353734 shutdown_connections 2022-09-07T21:40:09.431 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.425+0000 7f20dbfff700 1 --2- 172.21.15.47:0/2056353734 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20dc1081e0 0x7f20dc10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:09.431 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.425+0000 7f20dbfff700 1 --2- 172.21.15.47:0/2056353734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20dc0fee10 0x7f20dc0ff1e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:09.431 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.425+0000 7f20dbfff700 1 --2- 172.21.15.47:0/2056353734 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20dc0ff720 0x7f20dc107ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:09.431 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.425+0000 7f20dbfff700 1 -- 172.21.15.47:0/2056353734 >> 172.21.15.47:0/2056353734 conn(0x7f20dc0f8710 msgr2=0x7f20dc0fab20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:09.431 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.425+0000 7f20dbfff700 1 -- 172.21.15.47:0/2056353734 shutdown_connections 2022-09-07T21:40:09.432 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.425+0000 7f20dbfff700 1 -- 172.21.15.47:0/2056353734 wait complete. 2022-09-07T21:40:09.432 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.426+0000 7f20dbfff700 1 Processor -- start 2022-09-07T21:40:09.432 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.426+0000 7f20dbfff700 1 -- start start 2022-09-07T21:40:09.434 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.427+0000 7f20dbfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20dc0fee10 0x7f20dc0fcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:09.434 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.427+0000 7f20dbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20dc0ff720 0x7f20dc0fd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:09.435 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.427+0000 7f20dbfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20dc1081e0 0x7f20dc1a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:09.435 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.427+0000 7f20dbfff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc0fd8c0 con 0x7f20dc0fee10 2022-09-07T21:40:09.435 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.427+0000 7f20dbfff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc1a21d0 con 0x7f20dc1081e0 2022-09-07T21:40:09.435 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.427+0000 7f20dbfff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc1a2310 con 0x7f20dc0ff720 2022-09-07T21:40:09.435 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.427+0000 7f20daffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20dc0fee10 0x7f20dc0fcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:09.436 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.427+0000 7f20da7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20dc0ff720 0x7f20dc0fd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:09.436 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.427+0000 7f20daffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20dc0fee10 0x7f20dc0fcaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56804/0 (socket says 172.21.15.47:56804) 2022-09-07T21:40:09.436 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.427+0000 7f20daffd700 1 -- 172.21.15.47:0/938004537 learned_addr learned my addr 172.21.15.47:0/938004537 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:09.436 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.427+0000 7f20db7fe700 1 --2- 172.21.15.47:0/938004537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20dc1081e0 0x7f20dc1a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:09.437 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.428+0000 7f20da7fc700 1 -- 172.21.15.47:0/938004537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20dc1081e0 msgr2=0x7f20dc1a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:09.437 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.428+0000 7f20da7fc700 1 --2- 172.21.15.47:0/938004537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20dc1081e0 0x7f20dc1a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:09.437 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.428+0000 7f20da7fc700 1 -- 172.21.15.47:0/938004537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20dc0fee10 msgr2=0x7f20dc0fcaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:09.437 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.428+0000 7f20da7fc700 1 --2- 172.21.15.47:0/938004537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20dc0fee10 0x7f20dc0fcaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:09.438 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.428+0000 7f20da7fc700 1 -- 172.21.15.47:0/938004537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc009470 con 0x7f20dc0ff720 2022-09-07T21:40:09.438 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.428+0000 7f20da7fc700 1 --2- 172.21.15.47:0/938004537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20dc0ff720 0x7f20dc0fd030 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f20d000e5d0 tx=0x7f20d000e8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:09.438 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.428+0000 7f20bffff700 1 -- 172.21.15.47:0/938004537 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d0010840 con 0x7f20dc0ff720 2022-09-07T21:40:09.438 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.428+0000 7f20dbfff700 1 -- 172.21.15.47:0/938004537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20dc1a25a0 con 0x7f20dc0ff720 2022-09-07T21:40:09.439 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.428+0000 7f20bffff700 1 -- 172.21.15.47:0/938004537 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f20d0010e40 con 0x7f20dc0ff720 2022-09-07T21:40:09.439 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.428+0000 7f20bffff700 1 -- 172.21.15.47:0/938004537 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d00173d0 con 0x7f20dc0ff720 2022-09-07T21:40:09.439 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.429+0000 7f20dbfff700 1 -- 172.21.15.47:0/938004537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20dc1a2ab0 con 0x7f20dc0ff720 2022-09-07T21:40:09.439 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.430+0000 7f20dbfff700 1 -- 172.21.15.47:0/938004537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20dc04d480 con 0x7f20dc0ff720 2022-09-07T21:40:09.440 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.432+0000 7f20bffff700 1 -- 172.21.15.47:0/938004537 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f20d000c040 con 0x7f20dc0ff720 2022-09-07T21:40:09.441 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.434+0000 7f20bffff700 1 --2- 172.21.15.47:0/938004537 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f20c4099050 0x7f20c409b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:09.441 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.434+0000 7f20bffff700 1 -- 172.21.15.47:0/938004537 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f20d0014070 con 0x7f20dc0ff720 2022-09-07T21:40:09.441 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.434+0000 7f20daffd700 1 --2- 172.21.15.47:0/938004537 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f20c4099050 0x7f20c409b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:09.441 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.434+0000 7f20daffd700 1 --2- 172.21.15.47:0/938004537 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f20c4099050 0x7f20c409b500 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009760 tx=0x7f20cc009e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:09.442 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.436+0000 7f20bffff700 1 -- 172.21.15.47:0/938004537 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f20d008ef60 con 0x7f20dc0ff720 2022-09-07T21:40:09.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.702+0000 7f20dbfff700 1 -- 172.21.15.47:0/938004537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f20dc10da70 con 0x7f20dc0ff720 2022-09-07T21:40:09.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.702+0000 7f20bffff700 1 -- 172.21.15.47:0/938004537 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f20d0092580 con 0x7f20dc0ff720 2022-09-07T21:40:09.707 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:40:09.707 INFO:teuthology.orchestra.run.smithi047.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-09-07T21:40:09.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.705+0000 7f20dbfff700 1 -- 172.21.15.47:0/938004537 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f20c4099050 msgr2=0x7f20c409b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:09.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.705+0000 7f20dbfff700 1 --2- 172.21.15.47:0/938004537 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f20c4099050 0x7f20c409b500 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009760 tx=0x7f20cc009e70 comp rx=0 tx=0).stop 2022-09-07T21:40:09.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.705+0000 7f20dbfff700 1 -- 172.21.15.47:0/938004537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20dc0ff720 msgr2=0x7f20dc0fd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:09.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.705+0000 7f20dbfff700 1 --2- 172.21.15.47:0/938004537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20dc0ff720 0x7f20dc0fd030 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f20d000e5d0 tx=0x7f20d000e8e0 comp rx=0 tx=0).stop 2022-09-07T21:40:09.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.705+0000 7f20dbfff700 1 -- 172.21.15.47:0/938004537 shutdown_connections 2022-09-07T21:40:09.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.705+0000 7f20dbfff700 1 --2- 172.21.15.47:0/938004537 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f20c4099050 0x7f20c409b500 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:09.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.705+0000 7f20dbfff700 1 --2- 172.21.15.47:0/938004537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20dc1081e0 0x7f20dc1a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:09.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.705+0000 7f20dbfff700 1 --2- 172.21.15.47:0/938004537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20dc0fee10 0x7f20dc0fcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:09.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.705+0000 7f20dbfff700 1 --2- 172.21.15.47:0/938004537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20dc0ff720 0x7f20dc0fd030 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:09.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.705+0000 7f20dbfff700 1 -- 172.21.15.47:0/938004537 >> 172.21.15.47:0/938004537 conn(0x7f20dc0f8710 msgr2=0x7f20dc10a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:09.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.705+0000 7f20dbfff700 1 -- 172.21.15.47:0/938004537 shutdown_connections 2022-09-07T21:40:09.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:09.705+0000 7f20dbfff700 1 -- 172.21.15.47:0/938004537 wait complete. 2022-09-07T21:40:10.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:09 smithi047 ceph-mon[90087]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-07T21:40:10.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:09 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/938004537' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-09-07T21:40:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:09 smithi055 ceph-mon[97033]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-07T21:40:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:09 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/938004537' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-09-07T21:40:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:09 smithi059 ceph-mon[97450]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-07T21:40:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:09 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/938004537' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-09-07T21:40:10.324 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2022-09-07T21:40:10.325 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-07T21:40:10.563 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.560+0000 7f7af4dff700 1 -- 172.21.15.47:0/250929315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af0105c20 msgr2=0x7f7af0108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:10.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.560+0000 7f7af4dff700 1 --2- 172.21.15.47:0/250929315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af0105c20 0x7f7af0108040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8009b00 tx=0x7f7ae8009e10 comp rx=0 tx=0).stop 2022-09-07T21:40:10.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.560+0000 7f7af4dff700 1 -- 172.21.15.47:0/250929315 shutdown_connections 2022-09-07T21:40:10.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.560+0000 7f7af4dff700 1 --2- 172.21.15.47:0/250929315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af0105c20 0x7f7af0108040 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:10.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.560+0000 7f7af4dff700 1 --2- 172.21.15.47:0/250929315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7af0072c30 0x7f7af0071110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:10.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.560+0000 7f7af4dff700 1 --2- 172.21.15.47:0/250929315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7af0071650 0x7f7af0071ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:10.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.560+0000 7f7af4dff700 1 -- 172.21.15.47:0/250929315 >> 172.21.15.47:0/250929315 conn(0x7f7af00f9460 msgr2=0x7f7af00fb870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:10.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.560+0000 7f7af4dff700 1 -- 172.21.15.47:0/250929315 shutdown_connections 2022-09-07T21:40:10.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.561+0000 7f7af4dff700 1 -- 172.21.15.47:0/250929315 wait complete. 2022-09-07T21:40:10.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.563+0000 7f7af4dff700 1 Processor -- start 2022-09-07T21:40:10.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.563+0000 7f7af4dff700 1 -- start start 2022-09-07T21:40:10.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.563+0000 7f7af4dff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af0071650 0x7f7af0195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:10.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.563+0000 7f7af4dff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7af0072c30 0x7f7af0196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:10.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.563+0000 7f7af4dff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7af0105c20 0x7f7af019d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:10.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.563+0000 7f7af4dff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af019d6a0 con 0x7f7af0072c30 2022-09-07T21:40:10.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.563+0000 7f7af4dff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af019d810 con 0x7f7af0105c20 2022-09-07T21:40:10.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.563+0000 7f7af4dff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af019d9d0 con 0x7f7af0071650 2022-09-07T21:40:10.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.564+0000 7f7aee59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af0071650 0x7f7af0195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:10.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.564+0000 7f7aee59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af0071650 0x7f7af0195b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:42178/0 (socket says 172.21.15.47:42178) 2022-09-07T21:40:10.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.564+0000 7f7aee59c700 1 -- 172.21.15.47:0/2476367908 learned_addr learned my addr 172.21.15.47:0/2476367908 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:10.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.564+0000 7f7aeed9d700 1 --2- 172.21.15.47:0/2476367908 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7af0105c20 0x7f7af019d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:10.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.564+0000 7f7aedd9b700 1 --2- 172.21.15.47:0/2476367908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7af0072c30 0x7f7af0196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:10.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.564+0000 7f7aee59c700 1 -- 172.21.15.47:0/2476367908 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7af0105c20 msgr2=0x7f7af019d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:10.571 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.564+0000 7f7aee59c700 1 --2- 172.21.15.47:0/2476367908 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7af0105c20 0x7f7af019d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:10.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.564+0000 7f7aee59c700 1 -- 172.21.15.47:0/2476367908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7af0072c30 msgr2=0x7f7af0196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:10.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.564+0000 7f7aee59c700 1 --2- 172.21.15.47:0/2476367908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7af0072c30 0x7f7af0196060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:10.572 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.564+0000 7f7aee59c700 1 -- 172.21.15.47:0/2476367908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae4009470 con 0x7f7af0071650 2022-09-07T21:40:10.573 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.564+0000 7f7aee59c700 1 --2- 172.21.15.47:0/2476367908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af0071650 0x7f7af0195b20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000e740 tx=0x7f7ae000ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:10.573 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.565+0000 7f7adf7fe700 1 -- 172.21.15.47:0/2476367908 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0005220 con 0x7f7af0071650 2022-09-07T21:40:10.573 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.565+0000 7f7adf7fe700 1 -- 172.21.15.47:0/2476367908 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f7ae0005380 con 0x7f7af0071650 2022-09-07T21:40:10.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.565+0000 7f7af4dff700 1 -- 172.21.15.47:0/2476367908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae8009730 con 0x7f7af0071650 2022-09-07T21:40:10.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.565+0000 7f7adf7fe700 1 -- 172.21.15.47:0/2476367908 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0004b70 con 0x7f7af0071650 2022-09-07T21:40:10.574 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.565+0000 7f7af4dff700 1 -- 172.21.15.47:0/2476367908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af019e080 con 0x7f7af0071650 2022-09-07T21:40:10.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.566+0000 7f7af4dff700 1 -- 172.21.15.47:0/2476367908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af010b6f0 con 0x7f7af0071650 2022-09-07T21:40:10.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.568+0000 7f7adf7fe700 1 -- 172.21.15.47:0/2476367908 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f7ae001e020 con 0x7f7af0071650 2022-09-07T21:40:10.576 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.569+0000 7f7adf7fe700 1 --2- 172.21.15.47:0/2476367908 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f7ad80990c0 0x7f7ad809b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:10.577 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.569+0000 7f7adf7fe700 1 -- 172.21.15.47:0/2476367908 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f7ae0014070 con 0x7f7af0071650 2022-09-07T21:40:10.577 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.570+0000 7f7aedd9b700 1 --2- 172.21.15.47:0/2476367908 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f7ad80990c0 0x7f7ad809b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:10.577 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.570+0000 7f7aedd9b700 1 --2- 172.21.15.47:0/2476367908 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f7ad80990c0 0x7f7ad809b570 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009440 tx=0x7f7ae4005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:10.577 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.572+0000 7f7adf7fe700 1 -- 172.21.15.47:0/2476367908 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f7ae008ef60 con 0x7f7af0071650 2022-09-07T21:40:10.779 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.776+0000 7f7af4dff700 1 -- 172.21.15.47:0/2476367908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7af0104230 con 0x7f7af0071650 2022-09-07T21:40:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.778+0000 7f7adf7fe700 1 -- 172.21.15.47:0/2476367908 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+23592 (secure 0 0 0) 0x7f7ae0092580 con 0x7f7af0071650 2022-09-07T21:40:10.781 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:40:10.781 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":91,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:39:47.780661+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-07T21:38:38.608937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-07T21:38:39.936252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-07T21:38:43.731797+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.18","mappings":[{"from":5,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:40:10.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.780+0000 7f7af4dff700 1 -- 172.21.15.47:0/2476367908 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f7ad80990c0 msgr2=0x7f7ad809b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:10.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.780+0000 7f7af4dff700 1 --2- 172.21.15.47:0/2476367908 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f7ad80990c0 0x7f7ad809b570 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009440 tx=0x7f7ae4005f60 comp rx=0 tx=0).stop 2022-09-07T21:40:10.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.780+0000 7f7af4dff700 1 -- 172.21.15.47:0/2476367908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af0071650 msgr2=0x7f7af0195b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:10.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.780+0000 7f7af4dff700 1 --2- 172.21.15.47:0/2476367908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af0071650 0x7f7af0195b20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000e740 tx=0x7f7ae000ea50 comp rx=0 tx=0).stop 2022-09-07T21:40:10.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.781+0000 7f7af4dff700 1 -- 172.21.15.47:0/2476367908 shutdown_connections 2022-09-07T21:40:10.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.781+0000 7f7af4dff700 1 --2- 172.21.15.47:0/2476367908 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f7ad80990c0 0x7f7ad809b570 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:10.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.781+0000 7f7af4dff700 1 --2- 172.21.15.47:0/2476367908 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7af0105c20 0x7f7af019d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:10.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.781+0000 7f7af4dff700 1 --2- 172.21.15.47:0/2476367908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af0071650 0x7f7af0195b20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:10.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.781+0000 7f7af4dff700 1 --2- 172.21.15.47:0/2476367908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7af0072c30 0x7f7af0196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:10.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.781+0000 7f7af4dff700 1 -- 172.21.15.47:0/2476367908 >> 172.21.15.47:0/2476367908 conn(0x7f7af00f9460 msgr2=0x7f7af00fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:10.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.781+0000 7f7af4dff700 1 -- 172.21.15.47:0/2476367908 shutdown_connections 2022-09-07T21:40:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:10.781+0000 7f7af4dff700 1 -- 172.21.15.47:0/2476367908 wait complete. 2022-09-07T21:40:10.816 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-07T21:33:29.342534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-09-07T21:38:38.608937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-09-07T21:38:39.936252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2022-09-07T21:38:43.731797+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-09-07T21:40:10.817 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-09-07T21:40:11.054 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.051+0000 7fef32bff700 1 -- 172.21.15.47:0/493671424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef2c0fddb0 msgr2=0x7fef2c0fe1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:11.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.051+0000 7fef32bff700 1 --2- 172.21.15.47:0/493671424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef2c0fddb0 0x7fef2c0fe1c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fef20009750 tx=0x7fef20009a60 comp rx=0 tx=0).stop 2022-09-07T21:40:11.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.052+0000 7fef32bff700 1 -- 172.21.15.47:0/493671424 shutdown_connections 2022-09-07T21:40:11.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.052+0000 7fef32bff700 1 --2- 172.21.15.47:0/493671424 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef2c0ff940 0x7fef2c106270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.052+0000 7fef32bff700 1 --2- 172.21.15.47:0/493671424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef2c0fddb0 0x7fef2c0fe1c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.052+0000 7fef32bff700 1 --2- 172.21.15.47:0/493671424 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef2c0fef90 0x7fef2c0ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.052+0000 7fef32bff700 1 -- 172.21.15.47:0/493671424 >> 172.21.15.47:0/493671424 conn(0x7fef2c0f9460 msgr2=0x7fef2c0fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:11.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.052+0000 7fef32bff700 1 -- 172.21.15.47:0/493671424 shutdown_connections 2022-09-07T21:40:11.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.052+0000 7fef32bff700 1 -- 172.21.15.47:0/493671424 wait complete. 2022-09-07T21:40:11.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.053+0000 7fef32bff700 1 Processor -- start 2022-09-07T21:40:11.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.054+0000 7fef32bff700 1 -- start start 2022-09-07T21:40:11.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.054+0000 7fef32bff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef2c0fddb0 0x7fef2c1939c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:11.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.054+0000 7fef32bff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef2c0fef90 0x7fef2c193f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:11.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.054+0000 7fef32bff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef2c0ff940 0x7fef2c19af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:11.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.054+0000 7fef32bff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c19b540 con 0x7fef2c0ff940 2022-09-07T21:40:11.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.054+0000 7fef32bff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c19b6b0 con 0x7fef2c0fef90 2022-09-07T21:40:11.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.054+0000 7fef32bff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c19b870 con 0x7fef2c0fddb0 2022-09-07T21:40:11.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.055+0000 7fef2bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef2c0fef90 0x7fef2c193f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:11.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.055+0000 7fef3119c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef2c0ff940 0x7fef2c19af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:11.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.055+0000 7fef2bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef2c0fef90 0x7fef2c193f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:46302/0 (socket says 172.21.15.47:46302) 2022-09-07T21:40:11.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.055+0000 7fef2bfff700 1 -- 172.21.15.47:0/3938372562 learned_addr learned my addr 172.21.15.47:0/3938372562 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:11.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.055+0000 7fef3099b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef2c0fddb0 0x7fef2c1939c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:11.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.055+0000 7fef3119c700 1 -- 172.21.15.47:0/3938372562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef2c0fddb0 msgr2=0x7fef2c1939c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:11.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.055+0000 7fef3119c700 1 --2- 172.21.15.47:0/3938372562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef2c0fddb0 0x7fef2c1939c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.055+0000 7fef3119c700 1 -- 172.21.15.47:0/3938372562 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef2c0fef90 msgr2=0x7fef2c193f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:11.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.055+0000 7fef3119c700 1 --2- 172.21.15.47:0/3938372562 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef2c0fef90 0x7fef2c193f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.055+0000 7fef3119c700 1 -- 172.21.15.47:0/3938372562 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef200093b0 con 0x7fef2c0ff940 2022-09-07T21:40:11.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.055+0000 7fef3119c700 1 --2- 172.21.15.47:0/3938372562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef2c0ff940 0x7fef2c19af70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fef2400eab0 tx=0x7fef2400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:11.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.056+0000 7fef29ffb700 1 -- 172.21.15.47:0/3938372562 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef240098c0 con 0x7fef2c0ff940 2022-09-07T21:40:11.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.056+0000 7fef29ffb700 1 -- 172.21.15.47:0/3938372562 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fef2400ce10 con 0x7fef2c0ff940 2022-09-07T21:40:11.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.056+0000 7fef32bff700 1 -- 172.21.15.47:0/3938372562 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2c19bb50 con 0x7fef2c0ff940 2022-09-07T21:40:11.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.056+0000 7fef29ffb700 1 -- 172.21.15.47:0/3938372562 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24004280 con 0x7fef2c0ff940 2022-09-07T21:40:11.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.056+0000 7fef32bff700 1 -- 172.21.15.47:0/3938372562 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2c19c060 con 0x7fef2c0ff940 2022-09-07T21:40:11.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.057+0000 7fef32bff700 1 -- 172.21.15.47:0/3938372562 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef2c109560 con 0x7fef2c0ff940 2022-09-07T21:40:11.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.059+0000 7fef29ffb700 1 -- 172.21.15.47:0/3938372562 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7fef2400c040 con 0x7fef2c0ff940 2022-09-07T21:40:11.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.061+0000 7fef29ffb700 1 --2- 172.21.15.47:0/3938372562 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fef1c0990d0 0x7fef1c09b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:11.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.061+0000 7fef29ffb700 1 -- 172.21.15.47:0/3938372562 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fef24014070 con 0x7fef2c0ff940 2022-09-07T21:40:11.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.061+0000 7fef3099b700 1 --2- 172.21.15.47:0/3938372562 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fef1c0990d0 0x7fef1c09b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:11.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.062+0000 7fef3099b700 1 --2- 172.21.15.47:0/3938372562 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fef1c0990d0 0x7fef1c09b580 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fef20009e30 tx=0x7fef20005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:11.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.065+0000 7fef29ffb700 1 -- 172.21.15.47:0/3938372562 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fef2408ec10 con 0x7fef2c0ff940 2022-09-07T21:40:11.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:10 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2476367908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:40:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:10 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2476367908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:40:11.268 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.266+0000 7fef32bff700 1 -- 172.21.15.47:0/3938372562 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fef2c104c30 con 0x7fef2c0ff940 2022-09-07T21:40:11.269 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.267+0000 7fef29ffb700 1 -- 172.21.15.47:0/3938372562 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fef2401c0a0 con 0x7fef2c0ff940 2022-09-07T21:40:11.270 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2022-09-07T21:40:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.269+0000 7fef32bff700 1 -- 172.21.15.47:0/3938372562 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fef1c0990d0 msgr2=0x7fef1c09b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.269+0000 7fef32bff700 1 --2- 172.21.15.47:0/3938372562 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fef1c0990d0 0x7fef1c09b580 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fef20009e30 tx=0x7fef20005fb0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.269+0000 7fef32bff700 1 -- 172.21.15.47:0/3938372562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef2c0ff940 msgr2=0x7fef2c19af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:11.273 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.269+0000 7fef32bff700 1 --2- 172.21.15.47:0/3938372562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef2c0ff940 0x7fef2c19af70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fef2400eab0 tx=0x7fef2400edc0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.273 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.269+0000 7fef32bff700 1 -- 172.21.15.47:0/3938372562 shutdown_connections 2022-09-07T21:40:11.273 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.269+0000 7fef32bff700 1 --2- 172.21.15.47:0/3938372562 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fef1c0990d0 0x7fef1c09b580 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.273 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.269+0000 7fef32bff700 1 --2- 172.21.15.47:0/3938372562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef2c0ff940 0x7fef2c19af70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.274 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.269+0000 7fef32bff700 1 --2- 172.21.15.47:0/3938372562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef2c0fddb0 0x7fef2c1939c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.274 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.269+0000 7fef32bff700 1 --2- 172.21.15.47:0/3938372562 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef2c0fef90 0x7fef2c193f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.274 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.269+0000 7fef32bff700 1 -- 172.21.15.47:0/3938372562 >> 172.21.15.47:0/3938372562 conn(0x7fef2c0f9460 msgr2=0x7fef2c0fb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:11.274 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.270+0000 7fef32bff700 1 -- 172.21.15.47:0/3938372562 shutdown_connections 2022-09-07T21:40:11.275 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.270+0000 7fef32bff700 1 -- 172.21.15.47:0/3938372562 wait complete. 2022-09-07T21:40:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:10 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2476367908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:40:11.299 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-09-07T21:40:11.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.533+0000 7f47b57ff700 1 -- 172.21.15.47:0/2668921308 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b0100ea0 msgr2=0x7f47b0103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:11.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.533+0000 7f47b57ff700 1 --2- 172.21.15.47:0/2668921308 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b0100ea0 0x7f47b0103300 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f47a4009810 tx=0x7f47a4009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:11.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.533+0000 7f47b57ff700 1 -- 172.21.15.47:0/2668921308 shutdown_connections 2022-09-07T21:40:11.537 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.533+0000 7f47b57ff700 1 --2- 172.21.15.47:0/2668921308 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47b0103990 0x7f47b0105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.537 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.533+0000 7f47b57ff700 1 --2- 172.21.15.47:0/2668921308 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47b0066be0 0x7f47b0100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.537 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.533+0000 7f47b57ff700 1 --2- 172.21.15.47:0/2668921308 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b0100ea0 0x7f47b0103300 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.538 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.533+0000 7f47b57ff700 1 -- 172.21.15.47:0/2668921308 >> 172.21.15.47:0/2668921308 conn(0x7f47b00f8370 msgr2=0x7f47b00fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:11.538 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.533+0000 7f47b57ff700 1 -- 172.21.15.47:0/2668921308 shutdown_connections 2022-09-07T21:40:11.538 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.533+0000 7f47b57ff700 1 -- 172.21.15.47:0/2668921308 wait complete. 2022-09-07T21:40:11.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.534+0000 7f47b57ff700 1 Processor -- start 2022-09-07T21:40:11.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.534+0000 7f47b57ff700 1 -- start start 2022-09-07T21:40:11.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.535+0000 7f47b57ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47b0066be0 0x7f47b00fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:11.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.535+0000 7f47b57ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b0100ea0 0x7f47b00fc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:11.542 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.535+0000 7f47b57ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47b0103990 0x7f47b00fce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:11.542 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.535+0000 7f47b57ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b00fd380 con 0x7f47b0066be0 2022-09-07T21:40:11.542 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.535+0000 7f47b57ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b00fd4f0 con 0x7f47b0100ea0 2022-09-07T21:40:11.542 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.535+0000 7f47b57ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b00fd6b0 con 0x7f47b0103990 2022-09-07T21:40:11.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.535+0000 7f47ae7fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b0100ea0 0x7f47b00fc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:11.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.535+0000 7f47ae7fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b0100ea0 0x7f47b00fc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:46320/0 (socket says 172.21.15.47:46320) 2022-09-07T21:40:11.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.535+0000 7f47ae7fc700 1 -- 172.21.15.47:0/1192481050 learned_addr learned my addr 172.21.15.47:0/1192481050 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:11.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.535+0000 7f47af7fe700 1 --2- 172.21.15.47:0/1192481050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47b0103990 0x7f47b00fce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:11.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.535+0000 7f47aeffd700 1 --2- 172.21.15.47:0/1192481050 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47b0066be0 0x7f47b00fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:11.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.536+0000 7f47ae7fc700 1 -- 172.21.15.47:0/1192481050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47b0103990 msgr2=0x7f47b00fce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:11.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.536+0000 7f47ae7fc700 1 --2- 172.21.15.47:0/1192481050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47b0103990 0x7f47b00fce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.536+0000 7f47ae7fc700 1 -- 172.21.15.47:0/1192481050 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47b0066be0 msgr2=0x7f47b00fe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:11.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.536+0000 7f47ae7fc700 1 --2- 172.21.15.47:0/1192481050 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47b0066be0 0x7f47b00fe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.545 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.536+0000 7f47ae7fc700 1 -- 172.21.15.47:0/1192481050 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a00093b0 con 0x7f47b0100ea0 2022-09-07T21:40:11.545 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.536+0000 7f47ae7fc700 1 --2- 172.21.15.47:0/1192481050 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b0100ea0 0x7f47b00fc900 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f47a40059c0 tx=0x7f47a400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:11.545 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.536+0000 7f4797fff700 1 -- 172.21.15.47:0/1192481050 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a401d070 con 0x7f47b0100ea0 2022-09-07T21:40:11.545 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.536+0000 7f4797fff700 1 -- 172.21.15.47:0/1192481050 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f47a400f880 con 0x7f47b0100ea0 2022-09-07T21:40:11.546 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.536+0000 7f4797fff700 1 -- 172.21.15.47:0/1192481050 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a40047d0 con 0x7f47b0100ea0 2022-09-07T21:40:11.546 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.536+0000 7f47b57ff700 1 -- 172.21.15.47:0/1192481050 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a4009470 con 0x7f47b0100ea0 2022-09-07T21:40:11.546 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.536+0000 7f47b57ff700 1 -- 172.21.15.47:0/1192481050 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b01a2460 con 0x7f47b0100ea0 2022-09-07T21:40:11.546 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.537+0000 7f47b57ff700 1 -- 172.21.15.47:0/1192481050 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b01095a0 con 0x7f47b0100ea0 2022-09-07T21:40:11.547 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.539+0000 7f4797fff700 1 -- 172.21.15.47:0/1192481050 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f47a4028030 con 0x7f47b0100ea0 2022-09-07T21:40:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.541+0000 7f4797fff700 1 --2- 172.21.15.47:0/1192481050 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4798099050 0x7f479809b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.541+0000 7f4797fff700 1 -- 172.21.15.47:0/1192481050 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f47a40c37c0 con 0x7f47b0100ea0 2022-09-07T21:40:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.541+0000 7f47aeffd700 1 --2- 172.21.15.47:0/1192481050 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4798099050 0x7f479809b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.542+0000 7f47aeffd700 1 --2- 172.21.15.47:0/1192481050 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4798099050 0x7f479809b500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f47b0066fb0 tx=0x7f47a0005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.543+0000 7f4797fff700 1 -- 172.21.15.47:0/1192481050 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f47a408fb00 con 0x7f47b0100ea0 2022-09-07T21:40:11.746 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.744+0000 7f47b57ff700 1 -- 172.21.15.47:0/1192481050 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f47b00f9d60 con 0x7f47b0100ea0 2022-09-07T21:40:11.747 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.745+0000 7f4797fff700 1 -- 172.21.15.47:0/1192481050 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f47a4026020 con 0x7f47b0100ea0 2022-09-07T21:40:11.748 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2022-09-07T21:40:11.750 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.747+0000 7f47b57ff700 1 -- 172.21.15.47:0/1192481050 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4798099050 msgr2=0x7f479809b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:11.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.747+0000 7f47b57ff700 1 --2- 172.21.15.47:0/1192481050 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4798099050 0x7f479809b500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f47b0066fb0 tx=0x7f47a0005ea0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.747+0000 7f47b57ff700 1 -- 172.21.15.47:0/1192481050 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b0100ea0 msgr2=0x7f47b00fc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:11.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.747+0000 7f47b57ff700 1 --2- 172.21.15.47:0/1192481050 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b0100ea0 0x7f47b00fc900 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f47a40059c0 tx=0x7f47a400f690 comp rx=0 tx=0).stop 2022-09-07T21:40:11.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.747+0000 7f47b57ff700 1 -- 172.21.15.47:0/1192481050 shutdown_connections 2022-09-07T21:40:11.752 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.747+0000 7f47b57ff700 1 --2- 172.21.15.47:0/1192481050 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4798099050 0x7f479809b500 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.752 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.747+0000 7f47b57ff700 1 --2- 172.21.15.47:0/1192481050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47b0103990 0x7f47b00fce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.752 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.747+0000 7f47b57ff700 1 --2- 172.21.15.47:0/1192481050 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47b0066be0 0x7f47b00fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.752 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.747+0000 7f47b57ff700 1 --2- 172.21.15.47:0/1192481050 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47b0100ea0 0x7f47b00fc900 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.748+0000 7f47b57ff700 1 -- 172.21.15.47:0/1192481050 >> 172.21.15.47:0/1192481050 conn(0x7f47b00f8370 msgr2=0x7f47b00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.748+0000 7f47b57ff700 1 -- 172.21.15.47:0/1192481050 shutdown_connections 2022-09-07T21:40:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:11.748+0000 7f47b57ff700 1 -- 172.21.15.47:0/1192481050 wait complete. 2022-09-07T21:40:11.777 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-09-07T21:40:12.014 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.011+0000 7fc2b1bff700 1 -- 172.21.15.47:0/631783246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2ac0fddb0 msgr2=0x7fc2ac0fe1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:12.014 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.011+0000 7fc2b1bff700 1 --2- 172.21.15.47:0/631783246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2ac0fddb0 0x7fc2ac0fe1c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009810 tx=0x7fc29c009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:12.015 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.011+0000 7fc2b1bff700 1 -- 172.21.15.47:0/631783246 shutdown_connections 2022-09-07T21:40:12.015 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.011+0000 7fc2b1bff700 1 --2- 172.21.15.47:0/631783246 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2ac0ff940 0x7fc2ac106270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.015 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.011+0000 7fc2b1bff700 1 --2- 172.21.15.47:0/631783246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2ac0fddb0 0x7fc2ac0fe1c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.016 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.011+0000 7fc2b1bff700 1 --2- 172.21.15.47:0/631783246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0fef90 0x7fc2ac0ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.016 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.011+0000 7fc2b1bff700 1 -- 172.21.15.47:0/631783246 >> 172.21.15.47:0/631783246 conn(0x7fc2ac0f9460 msgr2=0x7fc2ac0fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:12.016 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.011+0000 7fc2b1bff700 1 -- 172.21.15.47:0/631783246 shutdown_connections 2022-09-07T21:40:12.017 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.012+0000 7fc2b1bff700 1 -- 172.21.15.47:0/631783246 wait complete. 2022-09-07T21:40:12.019 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.013+0000 7fc2b1bff700 1 Processor -- start 2022-09-07T21:40:12.020 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.013+0000 7fc2b1bff700 1 -- start start 2022-09-07T21:40:12.020 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.013+0000 7fc2b1bff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2ac0fddb0 0x7fc2ac195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:12.020 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.013+0000 7fc2b1bff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0fef90 0x7fc2ac196100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:12.020 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.013+0000 7fc2b1bff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2ac0ff940 0x7fc2ac19d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:12.020 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.013+0000 7fc2b1bff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac19d740 con 0x7fc2ac0fef90 2022-09-07T21:40:12.021 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.013+0000 7fc2b1bff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac19d8b0 con 0x7fc2ac0ff940 2022-09-07T21:40:12.021 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.013+0000 7fc2b1bff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac19da70 con 0x7fc2ac0fddb0 2022-09-07T21:40:12.021 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.013+0000 7fc2aaffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0fef90 0x7fc2ac196100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:12.022 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.013+0000 7fc2ab7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2ac0fddb0 0x7fc2ac195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:12.022 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.014+0000 7fc2ab7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2ac0fddb0 0x7fc2ac195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:42236/0 (socket says 172.21.15.47:42236) 2022-09-07T21:40:12.022 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.014+0000 7fc2aaffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0fef90 0x7fc2ac196100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56870/0 (socket says 172.21.15.47:56870) 2022-09-07T21:40:12.022 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.014+0000 7fc2ab7fe700 1 -- 172.21.15.47:0/616414862 learned_addr learned my addr 172.21.15.47:0/616414862 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:12.023 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.014+0000 7fc2abfff700 1 --2- 172.21.15.47:0/616414862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2ac0ff940 0x7fc2ac19d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:12.023 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.014+0000 7fc2ab7fe700 1 -- 172.21.15.47:0/616414862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2ac0ff940 msgr2=0x7fc2ac19d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:12.023 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.014+0000 7fc2ab7fe700 1 --2- 172.21.15.47:0/616414862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2ac0ff940 0x7fc2ac19d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.023 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.014+0000 7fc2ab7fe700 1 -- 172.21.15.47:0/616414862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0fef90 msgr2=0x7fc2ac196100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:12.024 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.014+0000 7fc2ab7fe700 1 --2- 172.21.15.47:0/616414862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0fef90 0x7fc2ac196100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.024 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.014+0000 7fc2ab7fe700 1 -- 172.21.15.47:0/616414862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c009470 con 0x7fc2ac0fddb0 2022-09-07T21:40:12.024 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.014+0000 7fc2aaffd700 1 --2- 172.21.15.47:0/616414862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0fef90 0x7fc2ac196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:12.024 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.014+0000 7fc2ab7fe700 1 --2- 172.21.15.47:0/616414862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2ac0fddb0 0x7fc2ac195bc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc29c018630 tx=0x7fc29c018660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:12.025 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.015+0000 7fc2a8ff9700 1 -- 172.21.15.47:0/616414862 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c01d070 con 0x7fc2ac0fddb0 2022-09-07T21:40:12.025 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.015+0000 7fc2a8ff9700 1 -- 172.21.15.47:0/616414862 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fc29c018880 con 0x7fc2ac0fddb0 2022-09-07T21:40:12.025 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.015+0000 7fc2b1bff700 1 -- 172.21.15.47:0/616414862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2ac19dcf0 con 0x7fc2ac0fddb0 2022-09-07T21:40:12.025 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.015+0000 7fc2a8ff9700 1 -- 172.21.15.47:0/616414862 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c0047d0 con 0x7fc2ac0fddb0 2022-09-07T21:40:12.026 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.015+0000 7fc2b1bff700 1 -- 172.21.15.47:0/616414862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ac19e1a0 con 0x7fc2ac0fddb0 2022-09-07T21:40:12.026 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.016+0000 7fc2b1bff700 1 -- 172.21.15.47:0/616414862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ac109560 con 0x7fc2ac0fddb0 2022-09-07T21:40:12.027 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.018+0000 7fc2a8ff9700 1 -- 172.21.15.47:0/616414862 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7fc29c029030 con 0x7fc2ac0fddb0 2022-09-07T21:40:12.027 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.019+0000 7fc2a8ff9700 1 --2- 172.21.15.47:0/616414862 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc294099050 0x7fc29409b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:12.028 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.019+0000 7fc2a8ff9700 1 -- 172.21.15.47:0/616414862 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fc29c0c3820 con 0x7fc2ac0fddb0 2022-09-07T21:40:12.028 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.019+0000 7fc2aaffd700 1 --2- 172.21.15.47:0/616414862 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc294099050 0x7fc29409b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:12.028 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.020+0000 7fc2aaffd700 1 --2- 172.21.15.47:0/616414862 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc294099050 0x7fc29409b500 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0005fd0 tx=0x7fc2a0005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:12.028 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.021+0000 7fc2a8ff9700 1 -- 172.21.15.47:0/616414862 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fc29c08fb60 con 0x7fc2ac0fddb0 2022-09-07T21:40:12.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:11 smithi047 ceph-mon[90087]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:40:12.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:11 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3938372562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:40:12.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:11 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1192481050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:40:12.227 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.225+0000 7fc2b1bff700 1 -- 172.21.15.47:0/616414862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc2ac190170 con 0x7fc2ac0fddb0 2022-09-07T21:40:12.228 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.226+0000 7fc2a8ff9700 1 -- 172.21.15.47:0/616414862 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc29c027020 con 0x7fc2ac0fddb0 2022-09-07T21:40:12.228 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2022-09-07T21:40:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:11 smithi055 ceph-mon[97033]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:40:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:11 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3938372562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:40:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:11 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1192481050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:40:12.231 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.228+0000 7fc2b1bff700 1 -- 172.21.15.47:0/616414862 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc294099050 msgr2=0x7fc29409b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:12.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.228+0000 7fc2b1bff700 1 --2- 172.21.15.47:0/616414862 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc294099050 0x7fc29409b500 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0005fd0 tx=0x7fc2a0005ea0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.228+0000 7fc2b1bff700 1 -- 172.21.15.47:0/616414862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2ac0fddb0 msgr2=0x7fc2ac195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:12.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.228+0000 7fc2b1bff700 1 --2- 172.21.15.47:0/616414862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2ac0fddb0 0x7fc2ac195bc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc29c018630 tx=0x7fc29c018660 comp rx=0 tx=0).stop 2022-09-07T21:40:12.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.228+0000 7fc2b1bff700 1 -- 172.21.15.47:0/616414862 shutdown_connections 2022-09-07T21:40:12.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.228+0000 7fc2b1bff700 1 --2- 172.21.15.47:0/616414862 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc294099050 0x7fc29409b500 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.228+0000 7fc2b1bff700 1 --2- 172.21.15.47:0/616414862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2ac0ff940 0x7fc2ac19d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.228+0000 7fc2b1bff700 1 --2- 172.21.15.47:0/616414862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2ac0fddb0 0x7fc2ac195bc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.228+0000 7fc2b1bff700 1 --2- 172.21.15.47:0/616414862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0fef90 0x7fc2ac196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.228+0000 7fc2b1bff700 1 -- 172.21.15.47:0/616414862 >> 172.21.15.47:0/616414862 conn(0x7fc2ac0f9460 msgr2=0x7fc2ac1049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:12.234 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.228+0000 7fc2b1bff700 1 -- 172.21.15.47:0/616414862 shutdown_connections 2022-09-07T21:40:12.234 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.229+0000 7fc2b1bff700 1 -- 172.21.15.47:0/616414862 wait complete. 2022-09-07T21:40:12.259 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2022-09-07T21:40:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:11 smithi059 ceph-mon[97450]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:40:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:11 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3938372562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:40:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:11 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1192481050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:40:12.495 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.492+0000 7f142ffff700 1 -- 172.21.15.47:0/2774675471 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14301013f0 msgr2=0x7f1430101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:12.495 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.492+0000 7f142ffff700 1 --2- 172.21.15.47:0/2774675471 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14301013f0 0x7f1430101800 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1420009810 tx=0x7f1420009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:12.495 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.492+0000 7f142ffff700 1 -- 172.21.15.47:0/2774675471 shutdown_connections 2022-09-07T21:40:12.495 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.492+0000 7f142ffff700 1 --2- 172.21.15.47:0/2774675471 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1430100430 0x7f1430108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.496 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.492+0000 7f142ffff700 1 --2- 172.21.15.47:0/2774675471 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14301013f0 0x7f1430101800 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.496 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.492+0000 7f142ffff700 1 --2- 172.21.15.47:0/2774675471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14300ffa80 0x7f14300ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.496 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.492+0000 7f142ffff700 1 -- 172.21.15.47:0/2774675471 >> 172.21.15.47:0/2774675471 conn(0x7f14300fb6b0 msgr2=0x7f14300fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:12.497 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.492+0000 7f142ffff700 1 -- 172.21.15.47:0/2774675471 shutdown_connections 2022-09-07T21:40:12.497 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.492+0000 7f142ffff700 1 -- 172.21.15.47:0/2774675471 wait complete. 2022-09-07T21:40:12.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.494+0000 7f142ffff700 1 Processor -- start 2022-09-07T21:40:12.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.494+0000 7f142ffff700 1 -- start start 2022-09-07T21:40:12.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.494+0000 7f142ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14300ffa80 0x7f14301959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:12.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.494+0000 7f142ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1430100430 0x7f1430195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:12.501 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.494+0000 7f142ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14301013f0 0x7f143019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:12.501 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142ffff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143019d550 con 0x7f14300ffa80 2022-09-07T21:40:12.501 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142ffff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143019d6c0 con 0x7f14301013f0 2022-09-07T21:40:12.501 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142ffff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143019d880 con 0x7f1430100430 2022-09-07T21:40:12.502 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14300ffa80 0x7f14301959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:12.502 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142e7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1430100430 0x7f1430195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:12.502 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14300ffa80 0x7f14301959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56888/0 (socket says 172.21.15.47:56888) 2022-09-07T21:40:12.502 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142effd700 1 -- 172.21.15.47:0/3306136646 learned_addr learned my addr 172.21.15.47:0/3306136646 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:12.503 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142f7fe700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14301013f0 0x7f143019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:12.503 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142e7fc700 1 -- 172.21.15.47:0/3306136646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14301013f0 msgr2=0x7f143019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:12.503 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142e7fc700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14301013f0 0x7f143019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.503 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142e7fc700 1 -- 172.21.15.47:0/3306136646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14300ffa80 msgr2=0x7f14301959d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:12.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142e7fc700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14300ffa80 0x7f14301959d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142e7fc700 1 -- 172.21.15.47:0/3306136646 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1420009470 con 0x7f1430100430 2022-09-07T21:40:12.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142effd700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14300ffa80 0x7f14301959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:12.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.495+0000 7f142f7fe700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14301013f0 0x7f143019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:12.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.496+0000 7f142e7fc700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1430100430 0x7f1430195f10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f142400e6c0 tx=0x7f142400e9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:12.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.496+0000 7f1417fff700 1 -- 172.21.15.47:0/3306136646 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1424010900 con 0x7f1430100430 2022-09-07T21:40:12.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.496+0000 7f142ffff700 1 -- 172.21.15.47:0/3306136646 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143019db60 con 0x7f1430100430 2022-09-07T21:40:12.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.496+0000 7f1417fff700 1 -- 172.21.15.47:0/3306136646 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1424018420 con 0x7f1430100430 2022-09-07T21:40:12.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.496+0000 7f1417fff700 1 -- 172.21.15.47:0/3306136646 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1424017610 con 0x7f1430100430 2022-09-07T21:40:12.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.496+0000 7f142ffff700 1 -- 172.21.15.47:0/3306136646 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f143019e0c0 con 0x7f1430100430 2022-09-07T21:40:12.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.497+0000 7f142ffff700 1 -- 172.21.15.47:0/3306136646 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1430065eb0 con 0x7f1430100430 2022-09-07T21:40:12.507 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.499+0000 7f1417fff700 1 -- 172.21.15.47:0/3306136646 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f142400c040 con 0x7f1430100430 2022-09-07T21:40:12.507 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.501+0000 7f1417fff700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1418099050 0x7f141809b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:12.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.501+0000 7f1417fff700 1 -- 172.21.15.47:0/3306136646 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f1424014070 con 0x7f1430100430 2022-09-07T21:40:12.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.501+0000 7f142effd700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1418099050 0x7f141809b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:12.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.501+0000 7f142effd700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1418099050 0x7f141809b500 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f142000f4d0 tx=0x7f1420009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:12.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.503+0000 7f1417fff700 1 -- 172.21.15.47:0/3306136646 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f142408eb00 con 0x7f1430100430 2022-09-07T21:40:12.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.706+0000 7f142ffff700 1 -- 172.21.15.47:0/3306136646 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1430107350 con 0x7f1430100430 2022-09-07T21:40:12.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.707+0000 7f1417fff700 1 -- 172.21.15.47:0/3306136646 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1424092120 con 0x7f1430100430 2022-09-07T21:40:12.710 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2022-09-07T21:40:12.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.710+0000 7f142ffff700 1 -- 172.21.15.47:0/3306136646 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1418099050 msgr2=0x7f141809b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:12.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.710+0000 7f142ffff700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1418099050 0x7f141809b500 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f142000f4d0 tx=0x7f1420009eb0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.710+0000 7f142ffff700 1 -- 172.21.15.47:0/3306136646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1430100430 msgr2=0x7f1430195f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:12.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.710+0000 7f142ffff700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1430100430 0x7f1430195f10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f142400e6c0 tx=0x7f142400e9d0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.710+0000 7f142ffff700 1 -- 172.21.15.47:0/3306136646 shutdown_connections 2022-09-07T21:40:12.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.710+0000 7f142ffff700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1418099050 0x7f141809b500 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.710+0000 7f142ffff700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14301013f0 0x7f143019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.710+0000 7f142ffff700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14300ffa80 0x7f14301959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.710+0000 7f142ffff700 1 --2- 172.21.15.47:0/3306136646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1430100430 0x7f1430195f10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.710+0000 7f142ffff700 1 -- 172.21.15.47:0/3306136646 >> 172.21.15.47:0/3306136646 conn(0x7f14300fb6b0 msgr2=0x7f14301044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:12.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.710+0000 7f142ffff700 1 -- 172.21.15.47:0/3306136646 shutdown_connections 2022-09-07T21:40:12.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.710+0000 7f142ffff700 1 -- 172.21.15.47:0/3306136646 wait complete. 2022-09-07T21:40:12.740 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:40:12.974 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.971+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/1364766001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b740fef90 msgr2=0x7f1b740ff3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:12.975 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.971+0000 7f1b7b7ff700 1 --2- 172.21.15.47:0/1364766001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b740fef90 0x7f1b740ff3e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009750 tx=0x7f1b64009a60 comp rx=0 tx=0).stop 2022-09-07T21:40:12.975 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.971+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/1364766001 shutdown_connections 2022-09-07T21:40:12.975 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.971+0000 7f1b7b7ff700 1 --2- 172.21.15.47:0/1364766001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b740ff920 0x7f1b741061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.975 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.971+0000 7f1b7b7ff700 1 --2- 172.21.15.47:0/1364766001 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b740fdd10 0x7f1b740fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.976 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.971+0000 7f1b7b7ff700 1 --2- 172.21.15.47:0/1364766001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b740fef90 0x7f1b740ff3e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.976 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.971+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/1364766001 >> 172.21.15.47:0/1364766001 conn(0x7f1b740f93c0 msgr2=0x7f1b740fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:12.976 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.972+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/1364766001 shutdown_connections 2022-09-07T21:40:12.977 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.972+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/1364766001 wait complete. 2022-09-07T21:40:12.979 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.973+0000 7f1b7b7ff700 1 Processor -- start 2022-09-07T21:40:12.979 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.973+0000 7f1b7b7ff700 1 -- start start 2022-09-07T21:40:12.979 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.973+0000 7f1b7b7ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b740fdd10 0x7f1b74195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:12.980 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.973+0000 7f1b7b7ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b740fef90 0x7f1b74196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:12.980 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.973+0000 7f1b7b7ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b740ff920 0x7f1b7419d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:12.980 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.973+0000 7f1b7b7ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7419d650 con 0x7f1b740ff920 2022-09-07T21:40:12.980 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.973+0000 7f1b7b7ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7419d7c0 con 0x7f1b740fef90 2022-09-07T21:40:12.981 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.973+0000 7f1b7b7ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7419d980 con 0x7f1b740fdd10 2022-09-07T21:40:12.981 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.974+0000 7f1b78d9a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b740fef90 0x7f1b74196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:12.981 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.974+0000 7f1b7959b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b740fdd10 0x7f1b74195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:12.981 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.974+0000 7f1b78d9a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b740fef90 0x7f1b74196010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:46354/0 (socket says 172.21.15.47:46354) 2022-09-07T21:40:12.982 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.974+0000 7f1b78d9a700 1 -- 172.21.15.47:0/848731489 learned_addr learned my addr 172.21.15.47:0/848731489 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:12.982 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.974+0000 7f1b79d9c700 1 --2- 172.21.15.47:0/848731489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b740ff920 0x7f1b7419d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:12.982 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.974+0000 7f1b78d9a700 1 -- 172.21.15.47:0/848731489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b740fdd10 msgr2=0x7f1b74195ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:12.982 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.974+0000 7f1b78d9a700 1 --2- 172.21.15.47:0/848731489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b740fdd10 0x7f1b74195ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.982 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.974+0000 7f1b78d9a700 1 -- 172.21.15.47:0/848731489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b740ff920 msgr2=0x7f1b7419d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:12.983 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.974+0000 7f1b78d9a700 1 --2- 172.21.15.47:0/848731489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b740ff920 0x7f1b7419d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:12.983 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.974+0000 7f1b78d9a700 1 -- 172.21.15.47:0/848731489 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b640093b0 con 0x7f1b740fef90 2022-09-07T21:40:12.983 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.975+0000 7f1b78d9a700 1 --2- 172.21.15.47:0/848731489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b740fef90 0x7f1b74196010 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1b64004230 tx=0x7f1b64004260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:12.983 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.975+0000 7f1b6a7fc700 1 -- 172.21.15.47:0/848731489 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6401d070 con 0x7f1b740fef90 2022-09-07T21:40:12.984 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.975+0000 7f1b6a7fc700 1 -- 172.21.15.47:0/848731489 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1b64003bb0 con 0x7f1b740fef90 2022-09-07T21:40:12.984 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.975+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/848731489 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7419dc00 con 0x7f1b740fef90 2022-09-07T21:40:12.984 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.975+0000 7f1b6a7fc700 1 -- 172.21.15.47:0/848731489 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64017b80 con 0x7f1b740fef90 2022-09-07T21:40:12.984 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.975+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/848731489 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7419e050 con 0x7f1b740fef90 2022-09-07T21:40:12.984 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.976+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/848731489 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b74109530 con 0x7f1b740fef90 2022-09-07T21:40:12.985 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.978+0000 7f1b6a7fc700 1 -- 172.21.15.47:0/848731489 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f1b6400c040 con 0x7f1b740fef90 2022-09-07T21:40:12.986 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.979+0000 7f1b6a7fc700 1 --2- 172.21.15.47:0/848731489 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1b60099040 0x7f1b6009b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:12.986 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.979+0000 7f1b6a7fc700 1 -- 172.21.15.47:0/848731489 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f1b640c3a00 con 0x7f1b740fef90 2022-09-07T21:40:12.986 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.980+0000 7f1b7959b700 1 --2- 172.21.15.47:0/848731489 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1b60099040 0x7f1b6009b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:12.986 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.980+0000 7f1b7959b700 1 --2- 172.21.15.47:0/848731489 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1b60099040 0x7f1b6009b4f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1b740fedf0 tx=0x7f1b70005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:12.987 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:12.982+0000 7f1b6a7fc700 1 -- 172.21.15.47:0/848731489 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f1b6408fd40 con 0x7f1b740fef90 2022-09-07T21:40:13.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:12 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/616414862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:40:13.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:12 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3306136646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:40:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:12 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/616414862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:40:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:12 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3306136646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:40:13.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.231+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/848731489 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b7419e660 con 0x7f1b740fef90 2022-09-07T21:40:13.234 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.232+0000 7f1b6a7fc700 1 -- 172.21.15.47:0/848731489 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10707 (secure 0 0 0) 0x7f1b64093360 con 0x7f1b740fef90 2022-09-07T21:40:13.234 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:40:13.235 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/2559266585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2559266585},{"type":"v1","addr":"172.21.15.55:6835","nonce":2559266585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/2090942297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2090942297},{"type":"v1","addr":"172.21.15.55:6837","nonce":2090942297}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-09-07T21:38:40.544389+0000","modified":"2022-09-07T21:39:06.833800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24586,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6835/2445409240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2445409240},{"type":"v1","addr":"172.21.15.59:6835","nonce":2445409240}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/3542466465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":3542466465},{"type":"v1","addr":"172.21.15.59:6837","nonce":3542466465}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6839/315505812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":315505812},{"type":"v1","addr":"172.21.15.47:6839","nonce":315505812}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.59:6839/1006724740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":1006724740},{"type":"v1","addr":"172.21.15.59:6839","nonce":1006724740}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6839/517990300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":517990300},{"type":"v1","addr":"172.21.15.55:6839","nonce":517990300}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24586":{"gid":24586,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6841/793863162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6840","nonce":793863162},{"type":"v1","addr":"172.21.15.47:6841","nonce":793863162}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-09-07T21:40:13.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.234+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/848731489 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1b60099040 msgr2=0x7f1b6009b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.234+0000 7f1b7b7ff700 1 --2- 172.21.15.47:0/848731489 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1b60099040 0x7f1b6009b4f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1b740fedf0 tx=0x7f1b70005ea0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.234+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/848731489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b740fef90 msgr2=0x7f1b74196010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.234+0000 7f1b7b7ff700 1 --2- 172.21.15.47:0/848731489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b740fef90 0x7f1b74196010 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1b64004230 tx=0x7f1b64004260 comp rx=0 tx=0).stop 2022-09-07T21:40:13.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.234+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/848731489 shutdown_connections 2022-09-07T21:40:13.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.234+0000 7f1b7b7ff700 1 --2- 172.21.15.47:0/848731489 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1b60099040 0x7f1b6009b4f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.234+0000 7f1b7b7ff700 1 --2- 172.21.15.47:0/848731489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b740ff920 0x7f1b7419d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.234+0000 7f1b7b7ff700 1 --2- 172.21.15.47:0/848731489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b740fdd10 0x7f1b74195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.239 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.234+0000 7f1b7b7ff700 1 --2- 172.21.15.47:0/848731489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b740fef90 0x7f1b74196010 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.239 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.234+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/848731489 >> 172.21.15.47:0/848731489 conn(0x7f1b740f93c0 msgr2=0x7f1b740fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:13.239 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.234+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/848731489 shutdown_connections 2022-09-07T21:40:13.239 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.234+0000 7f1b7b7ff700 1 -- 172.21.15.47:0/848731489 wait complete. 2022-09-07T21:40:13.240 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 24 2022-09-07T21:40:13.265 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2022-09-07T21:38:40.544389+0000', 'modified': '2022-09-07T21:39:06.833800+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24586, 'mds_1': 24572, 'mds_2': 24556}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24526': {'gid': 24526, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.59:6835/2445409240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 2445409240}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 2445409240}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24532': {'gid': 24532, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.59:6837/3542466465', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 3542466465}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 3542466465}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24548': {'gid': 24548, 'name': 'g', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6839/315505812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6838', 'nonce': 315505812}, {'type': 'v1', 'addr': '172.21.15.47:6839', 'nonce': 315505812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24556': {'gid': 24556, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.59:6839/1006724740', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6838', 'nonce': 1006724740}, {'type': 'v1', 'addr': '172.21.15.59:6839', 'nonce': 1006724740}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6839/517990300', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6838', 'nonce': 517990300}, {'type': 'v1', 'addr': '172.21.15.55:6839', 'nonce': 517990300}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24586': {'gid': 24586, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.47:6841/793863162', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6840', 'nonce': 793863162}, {'type': 'v1', 'addr': '172.21.15.47:6841', 'nonce': 793863162}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-09-07T21:40:13.265 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2022-09-07T21:40:13.265 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2022-09-07T21:40:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:12 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/616414862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:40:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:12 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3306136646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:40:13.499 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.496+0000 7f3c389ff700 1 -- 172.21.15.47:0/81102759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c34066be0 msgr2=0x7f3c34100960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.496+0000 7f3c389ff700 1 --2- 172.21.15.47:0/81102759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c34066be0 0x7f3c34100960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3c24009810 tx=0x7f3c24009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:13.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.496+0000 7f3c389ff700 1 -- 172.21.15.47:0/81102759 shutdown_connections 2022-09-07T21:40:13.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.496+0000 7f3c389ff700 1 --2- 172.21.15.47:0/81102759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c34103990 0x7f3c34105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.496+0000 7f3c389ff700 1 --2- 172.21.15.47:0/81102759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c34066be0 0x7f3c34100960 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.500 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.496+0000 7f3c389ff700 1 --2- 172.21.15.47:0/81102759 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34100ea0 0x7f3c34103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.501 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.496+0000 7f3c389ff700 1 -- 172.21.15.47:0/81102759 >> 172.21.15.47:0/81102759 conn(0x7f3c340f8370 msgr2=0x7f3c340fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:13.501 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.497+0000 7f3c389ff700 1 -- 172.21.15.47:0/81102759 shutdown_connections 2022-09-07T21:40:13.501 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.497+0000 7f3c389ff700 1 -- 172.21.15.47:0/81102759 wait complete. 2022-09-07T21:40:13.503 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.498+0000 7f3c389ff700 1 Processor -- start 2022-09-07T21:40:13.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.498+0000 7f3c389ff700 1 -- start start 2022-09-07T21:40:13.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.498+0000 7f3c389ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c34066be0 0x7f3c34195be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.498+0000 7f3c389ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c34100ea0 0x7f3c34196120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.498+0000 7f3c389ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34103990 0x7f3c3419d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.498+0000 7f3c389ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3419d760 con 0x7f3c34100ea0 2022-09-07T21:40:13.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.498+0000 7f3c389ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3419d8d0 con 0x7f3c34103990 2022-09-07T21:40:13.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.498+0000 7f3c389ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3419da90 con 0x7f3c34066be0 2022-09-07T21:40:13.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.499+0000 7f3c31d9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c34100ea0 0x7f3c34196120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.499+0000 7f3c31d9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c34100ea0 0x7f3c34196120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56950/0 (socket says 172.21.15.47:56950) 2022-09-07T21:40:13.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.499+0000 7f3c3259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c34066be0 0x7f3c34195be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.499+0000 7f3c31d9b700 1 -- 172.21.15.47:0/1742616838 learned_addr learned my addr 172.21.15.47:0/1742616838 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:13.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.499+0000 7f3c32d9d700 1 --2- 172.21.15.47:0/1742616838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34103990 0x7f3c3419d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.499+0000 7f3c32d9d700 1 -- 172.21.15.47:0/1742616838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c34066be0 msgr2=0x7f3c34195be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.507 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.499+0000 7f3c32d9d700 1 --2- 172.21.15.47:0/1742616838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c34066be0 0x7f3c34195be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.507 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.499+0000 7f3c32d9d700 1 -- 172.21.15.47:0/1742616838 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c34100ea0 msgr2=0x7f3c34196120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.507 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.499+0000 7f3c32d9d700 1 --2- 172.21.15.47:0/1742616838 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c34100ea0 0x7f3c34196120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.507 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.499+0000 7f3c32d9d700 1 -- 172.21.15.47:0/1742616838 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c280093b0 con 0x7f3c34103990 2022-09-07T21:40:13.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.499+0000 7f3c32d9d700 1 --2- 172.21.15.47:0/1742616838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34103990 0x7f3c3419d190 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00eab0 tx=0x7f3c2c00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:13.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.500+0000 7f3c237fe700 1 -- 172.21.15.47:0/1742616838 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c0098c0 con 0x7f3c34103990 2022-09-07T21:40:13.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.500+0000 7f3c389ff700 1 -- 172.21.15.47:0/1742616838 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c24009470 con 0x7f3c34103990 2022-09-07T21:40:13.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.500+0000 7f3c237fe700 1 -- 172.21.15.47:0/1742616838 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3c2c00ce10 con 0x7f3c34103990 2022-09-07T21:40:13.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.500+0000 7f3c237fe700 1 -- 172.21.15.47:0/1742616838 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c004280 con 0x7f3c34103990 2022-09-07T21:40:13.509 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.500+0000 7f3c389ff700 1 -- 172.21.15.47:0/1742616838 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c3419e0f0 con 0x7f3c34103990 2022-09-07T21:40:13.514 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.503+0000 7f3c237fe700 1 -- 172.21.15.47:0/1742616838 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f3c2c00c040 con 0x7f3c34103990 2022-09-07T21:40:13.514 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.504+0000 7f3c237fe700 1 --2- 172.21.15.47:0/1742616838 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3c1c0990d0 0x7f3c1c09b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.514 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.504+0000 7f3c4bf81b80 1 Processor -- start 2022-09-07T21:40:13.515 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.505+0000 7f3c237fe700 1 -- 172.21.15.47:0/1742616838 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f3c2c014070 con 0x7f3c34103990 2022-09-07T21:40:13.515 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.505+0000 7f3c4bf81b80 1 -- start start 2022-09-07T21:40:13.515 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.505+0000 7f3c3259c700 1 --2- 172.21.15.47:0/1742616838 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3c1c0990d0 0x7f3c1c09b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.515 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.505+0000 7f3c4bf81b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd3ae0 0x5629d1fd3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.516 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.505+0000 7f3c4bf81b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fd4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.516 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.505+0000 7f3c4bf81b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 0x5629d1fd5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.516 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.505+0000 7f3c4bf81b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d1ebe080 con 0x5629d1fd4430 2022-09-07T21:40:13.516 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.505+0000 7f3c4bf81b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d1f289e0 con 0x5629d1fd51f0 2022-09-07T21:40:13.516 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.505+0000 7f3c4bf81b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d1f22180 con 0x5629d1fd3ae0 2022-09-07T21:40:13.517 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.506+0000 7f3c31d9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fd4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.517 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.506+0000 7f3c32d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 0x5629d1fd5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.517 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.506+0000 7f3c32d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 0x5629d1fd5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:46382/0 (socket says 172.21.15.47:46382) 2022-09-07T21:40:13.517 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.506+0000 7f3c3259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd3ae0 0x5629d1fd3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.517 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.506+0000 7f3c32d9d700 1 -- 172.21.15.47:0/1945457652 learned_addr learned my addr 172.21.15.47:0/1945457652 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:13.518 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.506+0000 7f3c31d9b700 1 --2- 172.21.15.47:0/1945457652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fd4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56958/0 (socket says 172.21.15.47:56958) 2022-09-07T21:40:13.518 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.506+0000 7f3c3259c700 1 --2- 172.21.15.47:0/1742616838 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3c1c0990d0 0x7f3c1c09b580 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3c24009760 tx=0x7f3c24009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:13.518 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.506+0000 7f3c32d9d700 1 -- 172.21.15.47:0/1945457652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd3ae0 msgr2=0x5629d1fd3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.518 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.506+0000 7f3c32d9d700 1 --2- 172.21.15.47:0/1945457652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd3ae0 0x5629d1fd3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.519 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.506+0000 7f3c32d9d700 1 -- 172.21.15.47:0/1945457652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 msgr2=0x5629d1fd4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.519 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.506+0000 7f3c32d9d700 1 --2- 172.21.15.47:0/1945457652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fd4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.519 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.506+0000 7f3c32d9d700 1 -- 172.21.15.47:0/1945457652 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629d1daa5e0 con 0x5629d1fd51f0 2022-09-07T21:40:13.519 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.506+0000 7f3c32d9d700 1 --2- 172.21.15.47:0/1945457652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 0x5629d1fd5640 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00c5b0 tx=0x7f3c2c08ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4ab157c8576561f server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:13.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.507+0000 7f3c20ff9700 1 -- 172.21.15.47:0/1945457652 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c003880 con 0x5629d1fd51f0 2022-09-07T21:40:13.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.507+0000 7f3c20ff9700 1 -- 172.21.15.47:0/1945457652 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3c2c092fd0 con 0x5629d1fd51f0 2022-09-07T21:40:13.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.507+0000 7f3c20ff9700 1 -- 172.21.15.47:0/1945457652 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c0043e0 con 0x5629d1fd51f0 2022-09-07T21:40:13.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.507+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/1945457652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 msgr2=0x5629d1fd5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.520 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.507+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/1945457652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 0x5629d1fd5640 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00c5b0 tx=0x7f3c2c08ed40 comp rx=0 tx=0).stop 2022-09-07T21:40:13.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.507+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/1945457652 shutdown_connections 2022-09-07T21:40:13.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.507+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/1945457652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 0x5629d1fd5640 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.507+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/1945457652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd3ae0 0x5629d1fd3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.507+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/1945457652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fd4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.521 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.507+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/1945457652 >> 172.21.15.47:0/1945457652 conn(0x5629d1fcffa0 msgr2=0x5629d1fce370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:13.522 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.508+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/1945457652 shutdown_connections 2022-09-07T21:40:13.522 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.508+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/1945457652 wait complete. 2022-09-07T21:40:13.522 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.508+0000 7f3c4bf81b80 1 -- start start 2022-09-07T21:40:13.522 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.508+0000 7f3c4bf81b80 1 Processor -- start 2022-09-07T21:40:13.522 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.509+0000 7f3c4bf81b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd3ae0 0x5629d1fdd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.523 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.509+0000 7f3c4bf81b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fdd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.523 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.509+0000 7f3c4bf81b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd51f0 0x5629d1fe49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.523 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.509+0000 7f3c4bf81b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d1e100c0 con 0x5629d1fd4430 2022-09-07T21:40:13.523 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.509+0000 7f3c4bf81b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d1fe4f20 con 0x5629d1fd3ae0 2022-09-07T21:40:13.524 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.509+0000 7f3c4bf81b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d1fe50b0 con 0x5629d1fd51f0 2022-09-07T21:40:13.524 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.509+0000 7f3c4bf81b80 10 client.0 _ll_register_callbacks cb 0x5629d1ecd370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-07T21:40:13.524 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.509+0000 7f3c31d9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fdd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.524 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.509+0000 7f3c3259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd3ae0 0x5629d1fdd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.524 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.509+0000 7f3c32d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd51f0 0x5629d1fe49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.509+0000 7f3c3259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd3ae0 0x5629d1fdd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:46392/0 (socket says 172.21.15.47:46392) 2022-09-07T21:40:13.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.509+0000 7f3c3259c700 1 -- 172.21.15.47:0/811272760 learned_addr learned my addr 172.21.15.47:0/811272760 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:13.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.510+0000 7f3c32d9d700 1 -- 172.21.15.47:0/811272760 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd3ae0 msgr2=0x5629d1fdd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.510+0000 7f3c32d9d700 1 --2- 172.21.15.47:0/811272760 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd3ae0 0x5629d1fdd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.510+0000 7f3c32d9d700 1 -- 172.21.15.47:0/811272760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 msgr2=0x5629d1fdd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.510+0000 7f3c32d9d700 1 --2- 172.21.15.47:0/811272760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fdd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.526 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.510+0000 7f3c32d9d700 1 -- 172.21.15.47:0/811272760 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c3419e0f0 con 0x5629d1fd51f0 2022-09-07T21:40:13.526 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.510+0000 7f3c32d9d700 1 --2- 172.21.15.47:0/811272760 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd51f0 0x5629d1fe49e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c08f340 tx=0x7f3c2c093140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f92e463eb55e501 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:13.526 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.510+0000 7f3c19ffb700 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2022-09-07T21:40:13.526 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.510+0000 7f3c19ffb700 1 -- 172.21.15.47:0/811272760 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c0b3d10 con 0x5629d1fd51f0 2022-09-07T21:40:13.527 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.510+0000 7f3c19ffb700 1 -- 172.21.15.47:0/811272760 <== mon.1 v2:172.21.15.59:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f3c2c003880 con 0x5629d1fd51f0 2022-09-07T21:40:13.527 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.510+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/811272760 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5629d1fc6aa0 con 0x5629d1fd51f0 2022-09-07T21:40:13.527 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.511+0000 7f3c19ffb700 1 -- 172.21.15.47:0/811272760 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c2c092b30 con 0x5629d1fd51f0 2022-09-07T21:40:13.527 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.511+0000 7f3c4bf81b80 10 client.25108 fetch_fsmap learned FSMap version 24 2022-09-07T21:40:13.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.511+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/811272760 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5629d1d56d90 con 0x5629d1fd51f0 2022-09-07T21:40:13.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.511+0000 7f3c19ffb700 1 -- 172.21.15.47:0/811272760 <== mon.1 v2:172.21.15.59:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3c2c0b7080 con 0x5629d1fd51f0 2022-09-07T21:40:13.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.511+0000 7f3c19ffb700 1 -- 172.21.15.47:0/811272760 <== mon.1 v2:172.21.15.59:3300/0 5 ==== fsmap(e 24) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f3c2c0996e0 con 0x5629d1fd51f0 2022-09-07T21:40:13.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.511+0000 7f3c4bf81b80 10 client.25108 fetch_fsmap finished waiting for FSMap version 24 2022-09-07T21:40:13.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.511+0000 7f3c4bf81b80 10 client.25108 resolve_mds: resolved name 'i' to daemon mds.i 2022-09-07T21:40:13.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.511+0000 7f3c4bf81b80 20 client.25108 populate_metadata read hostname 'smithi047' 2022-09-07T21:40:13.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.511+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/811272760 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x5629d1fe8340 0x5629d1fe6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.511+0000 7f3c4bf81b80 4 client.25108 mds_command: new command op to 24556 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-07T21:40:13.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.511+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/811272760 --> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5629d1fa34c0 con 0x5629d1fe8340 2022-09-07T21:40:13.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.512+0000 7f3c3259c700 1 --2- 172.21.15.47:0/811272760 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x5629d1fe8340 0x5629d1fe6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.512+0000 7f3c3259c700 1 --2- 172.21.15.47:0/811272760 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x5629d1fe8340 0x5629d1fe6aa0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ebd8ff24b74d321b server_cookie=3e7c558bafb821dd in_seq=0 out_seq=0 2022-09-07T21:40:13.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.512+0000 7f3c19ffb700 10 client.25108 ms_handle_connect on v2:172.21.15.59:6838/1006724740 2022-09-07T21:40:13.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.513+0000 7f3c19ffb700 1 -- 172.21.15.47:0/811272760 <== mds.2 v2:172.21.15.59:6838/1006724740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x5629d1fa34c0 con 0x5629d1fe8340 2022-09-07T21:40:13.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.513+0000 7f3c19ffb700 10 client.25108 handle_command_reply: tid=0 2022-09-07T21:40:13.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.513+0000 7f3c4bf81b80 1 client.25108 shutdown 2022-09-07T21:40:13.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.514+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/811272760 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd51f0 msgr2=0x5629d1fe49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.514+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/811272760 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd51f0 0x5629d1fe49e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c08f340 tx=0x7f3c2c093140 comp rx=0 tx=0).stop 2022-09-07T21:40:13.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.514+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/811272760 shutdown_connections 2022-09-07T21:40:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.514+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/811272760 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x5629d1fe8340 0x5629d1fe6aa0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.514+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/811272760 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd51f0 0x5629d1fe49e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.514+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/811272760 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd3ae0 0x5629d1fdd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.514+0000 7f3c19ffb700 0 client.25108 ms_handle_reset on v2:172.21.15.59:6838/1006724740 2022-09-07T21:40:13.534 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.514+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/811272760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fdd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.534 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.514+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/811272760 >> 172.21.15.47:0/811272760 conn(0x5629d1fcffa0 msgr2=0x5629d1fcda40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:13.534 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.514+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/811272760 shutdown_connections 2022-09-07T21:40:13.534 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.514+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/811272760 wait complete. 2022-09-07T21:40:13.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.514+0000 7f3c4bf81b80 20 client.25108 trim_cache size 0 max 16384 2022-09-07T21:40:13.537 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.532+0000 7f3c4bf81b80 1 Processor -- start 2022-09-07T21:40:13.538 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.532+0000 7f3c4bf81b80 1 -- start start 2022-09-07T21:40:13.538 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.532+0000 7f3c4bf81b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd3ae0 0x5629d1fdd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.538 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.532+0000 7f3c4bf81b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fdb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.538 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.532+0000 7f3c4bf81b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 0x5629d1fdb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.532+0000 7f3c4bf81b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d1fd4b10 con 0x5629d1fd4430 2022-09-07T21:40:13.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.532+0000 7f3c4bf81b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d1fe9bd0 con 0x5629d1fd51f0 2022-09-07T21:40:13.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.532+0000 7f3c4bf81b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d1fc9330 con 0x5629d1fd3ae0 2022-09-07T21:40:13.539 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.533+0000 7f3c31d9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fdb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.533+0000 7f3c3259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd3ae0 0x5629d1fdd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.533+0000 7f3c31d9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fdb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56982/0 (socket says 172.21.15.47:56982) 2022-09-07T21:40:13.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.533+0000 7f3c31d9b700 1 -- 172.21.15.47:0/3118679999 learned_addr learned my addr 172.21.15.47:0/3118679999 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:13.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.533+0000 7f3c32d9d700 1 --2- 172.21.15.47:0/3118679999 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 0x5629d1fdb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.540 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.533+0000 7f3c3259c700 1 -- 172.21.15.47:0/3118679999 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 msgr2=0x5629d1fdb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.533+0000 7f3c3259c700 1 --2- 172.21.15.47:0/3118679999 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 0x5629d1fdb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.533+0000 7f3c3259c700 1 -- 172.21.15.47:0/3118679999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 msgr2=0x5629d1fdb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.533+0000 7f3c3259c700 1 --2- 172.21.15.47:0/3118679999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fdb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.541 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.533+0000 7f3c3259c700 1 -- 172.21.15.47:0/3118679999 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629d1d55260 con 0x5629d1fd3ae0 2022-09-07T21:40:13.542 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.533+0000 7f3c31d9b700 1 --2- 172.21.15.47:0/3118679999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fdb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:13.542 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.534+0000 7f3c3259c700 1 --2- 172.21.15.47:0/3118679999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd3ae0 0x5629d1fdd920 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x5629d1fe9af0 tx=0x7f3c24020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d9b5acf2e60deea server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:13.542 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.534+0000 7f3c217fa700 1 -- 172.21.15.47:0/3118679999 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2400bd40 con 0x5629d1fd3ae0 2022-09-07T21:40:13.542 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.534+0000 7f3c217fa700 1 -- 172.21.15.47:0/3118679999 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3c2402a070 con 0x5629d1fd3ae0 2022-09-07T21:40:13.542 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.534+0000 7f3c217fa700 1 -- 172.21.15.47:0/3118679999 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c24004e70 con 0x5629d1fd3ae0 2022-09-07T21:40:13.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.534+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/3118679999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd3ae0 msgr2=0x5629d1fdd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.534+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/3118679999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd3ae0 0x5629d1fdd920 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x5629d1fe9af0 tx=0x7f3c24020000 comp rx=0 tx=0).stop 2022-09-07T21:40:13.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.534+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/3118679999 shutdown_connections 2022-09-07T21:40:13.543 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.534+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/3118679999 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 0x5629d1fdb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.534+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/3118679999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd3ae0 0x5629d1fdd920 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.534+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/3118679999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd4430 0x5629d1fdb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.534+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/3118679999 >> 172.21.15.47:0/3118679999 conn(0x5629d1fcffa0 msgr2=0x5629d1fdd210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.535+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/3118679999 shutdown_connections 2022-09-07T21:40:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.535+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/3118679999 wait complete. 2022-09-07T21:40:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.535+0000 7f3c4bf81b80 1 -- start start 2022-09-07T21:40:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.535+0000 7f3c4bf81b80 1 Processor -- start 2022-09-07T21:40:13.548 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c4bf81b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd3ae0 0x5629d1fee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.548 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c4bf81b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd4430 0x5629d1feec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.549 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c4bf81b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 0x5629d1ff1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.549 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c4bf81b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d1ff2190 con 0x5629d1fd3ae0 2022-09-07T21:40:13.549 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c4bf81b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d1ff2300 con 0x5629d1fd51f0 2022-09-07T21:40:13.549 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c4bf81b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d1ff24c0 con 0x5629d1fd4430 2022-09-07T21:40:13.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c4bf81b80 10 client.0 _ll_register_callbacks cb 0x5629d1ecd370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-07T21:40:13.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c31d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd4430 0x5629d1feec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c3259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd3ae0 0x5629d1fee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c31d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd4430 0x5629d1feec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:42300/0 (socket says 172.21.15.47:42300) 2022-09-07T21:40:13.551 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c31d9b700 1 -- 172.21.15.47:0/431182369 learned_addr learned my addr 172.21.15.47:0/431182369 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:13.551 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c32d9d700 1 --2- 172.21.15.47:0/431182369 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 0x5629d1ff1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.551 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c31d9b700 1 -- 172.21.15.47:0/431182369 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 msgr2=0x5629d1ff1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.551 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c31d9b700 1 --2- 172.21.15.47:0/431182369 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 0x5629d1ff1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.551 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c31d9b700 1 -- 172.21.15.47:0/431182369 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd3ae0 msgr2=0x5629d1fee6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.552 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c31d9b700 1 --2- 172.21.15.47:0/431182369 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd3ae0 0x5629d1fee6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.552 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.536+0000 7f3c31d9b700 1 -- 172.21.15.47:0/431182369 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c28009b40 con 0x5629d1fd4430 2022-09-07T21:40:13.552 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.537+0000 7f3c31d9b700 1 --2- 172.21.15.47:0/431182369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd4430 0x5629d1feec30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3c28012c80 tx=0x7f3c28012f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ed2a858d50beea6 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:13.552 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.537+0000 7f3c19ffb700 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2022-09-07T21:40:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.537+0000 7f3c19ffb700 1 -- 172.21.15.47:0/431182369 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c28022070 con 0x5629d1fd4430 2022-09-07T21:40:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.537+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/431182369 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c2c092fd0 con 0x5629d1fd4430 2022-09-07T21:40:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.537+0000 7f3c19ffb700 1 -- 172.21.15.47:0/431182369 <== mon.1 v2:172.21.15.59:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f3c28005020 con 0x5629d1fd4430 2022-09-07T21:40:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.537+0000 7f3c19ffb700 1 -- 172.21.15.47:0/431182369 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c28017040 con 0x5629d1fd4430 2022-09-07T21:40:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.538+0000 7f3c4bf81b80 10 client.25114 fetch_fsmap learned FSMap version 24 2022-09-07T21:40:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.538+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/431182369 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7f3c34106ad0 con 0x5629d1fd4430 2022-09-07T21:40:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.538+0000 7f3c19ffb700 1 -- 172.21.15.47:0/431182369 <== mon.1 v2:172.21.15.59:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3c2c092fd0 con 0x5629d1fd4430 2022-09-07T21:40:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.538+0000 7f3c19ffb700 1 -- 172.21.15.47:0/431182369 <== mon.1 v2:172.21.15.59:3300/0 5 ==== fsmap(e 24) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f3c28025020 con 0x5629d1fd4430 2022-09-07T21:40:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.538+0000 7f3c4bf81b80 10 client.25114 fetch_fsmap finished waiting for FSMap version 24 2022-09-07T21:40:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.538+0000 7f3c4bf81b80 10 client.25114 resolve_mds: resolved name 'i' to daemon mds.i 2022-09-07T21:40:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.538+0000 7f3c4bf81b80 20 client.25114 populate_metadata read hostname 'smithi047' 2022-09-07T21:40:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.538+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/431182369 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x5629d203f930 0x5629d203c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.538+0000 7f3c4bf81b80 4 client.25114 mds_command: new command op to 24556 tid=0[{"prefix": "status"}] 2022-09-07T21:40:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.538+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/431182369 --> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5629d1f1b6f0 con 0x5629d203f930 2022-09-07T21:40:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.539+0000 7f3c3259c700 1 --2- 172.21.15.47:0/431182369 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x5629d203f930 0x5629d203c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.539+0000 7f3c3259c700 1 --2- 172.21.15.47:0/431182369 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x5629d203f930 0x5629d203c2a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d7907f92c1108719 server_cookie=b7e12acce0001010 in_seq=0 out_seq=0 2022-09-07T21:40:13.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.539+0000 7f3c19ffb700 10 client.25114 ms_handle_connect on v2:172.21.15.59:6838/1006724740 2022-09-07T21:40:13.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.539+0000 7f3c19ffb700 1 -- 172.21.15.47:0/431182369 <== mds.2 v2:172.21.15.59:6838/1006724740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5629d1f1b6f0 con 0x5629d203f930 2022-09-07T21:40:13.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.539+0000 7f3c19ffb700 10 client.25114 handle_command_reply: tid=0 2022-09-07T21:40:13.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.539+0000 7f3c4bf81b80 1 client.25114 shutdown 2022-09-07T21:40:13.557 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.539+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/431182369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd4430 msgr2=0x5629d1feec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.557 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.539+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/431182369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd4430 0x5629d1feec30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3c28012c80 tx=0x7f3c28012f90 comp rx=0 tx=0).stop 2022-09-07T21:40:13.557 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.540+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/431182369 shutdown_connections 2022-09-07T21:40:13.557 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.540+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/431182369 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x5629d203f930 0x5629d203c2a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.540+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/431182369 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5629d1fd51f0 0x5629d1ff1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.540+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/431182369 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5629d1fd3ae0 0x5629d1fee6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.540+0000 7f3c4bf81b80 1 --2- 172.21.15.47:0/431182369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5629d1fd4430 0x5629d1feec30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.540+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/431182369 >> 172.21.15.47:0/431182369 conn(0x5629d1fcffa0 msgr2=0x5629d1fcda40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:13.559 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.540+0000 7f3c19ffb700 0 client.25114 ms_handle_reset on v2:172.21.15.59:6838/1006724740 2022-09-07T21:40:13.559 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.540+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/431182369 shutdown_connections 2022-09-07T21:40:13.559 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.540+0000 7f3c4bf81b80 1 -- 172.21.15.47:0/431182369 wait complete. 2022-09-07T21:40:13.559 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.540+0000 7f3c4bf81b80 20 client.25114 trim_cache size 0 max 16384 2022-09-07T21:40:13.560 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2022-09-07T21:40:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "01b8396e-2ef4-11ed-8431-001a4aab830c", 2022-09-07T21:40:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 2, 2022-09-07T21:40:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 24556, 2022-09-07T21:40:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2022-09-07T21:40:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2022-09-07T21:40:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2022-09-07T21:40:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 67.725554736000007, 2022-09-07T21:40:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 24, 2022-09-07T21:40:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 91, 2022-09-07T21:40:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 89, 2022-09-07T21:40:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 108.857996204 2022-09-07T21:40:13.562 INFO:teuthology.orchestra.run.smithi047.stdout:} 2022-09-07T21:40:13.563 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.541+0000 7f3c389ff700 1 -- 172.21.15.47:0/1742616838 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3c1c0990d0 msgr2=0x7f3c1c09b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.563 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.541+0000 7f3c389ff700 1 --2- 172.21.15.47:0/1742616838 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3c1c0990d0 0x7f3c1c09b580 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3c24009760 tx=0x7f3c24009eb0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.541+0000 7f3c389ff700 1 -- 172.21.15.47:0/1742616838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34103990 msgr2=0x7f3c3419d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.541+0000 7f3c389ff700 1 --2- 172.21.15.47:0/1742616838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34103990 0x7f3c3419d190 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00eab0 tx=0x7f3c2c00edc0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.541+0000 7f3c389ff700 1 -- 172.21.15.47:0/1742616838 shutdown_connections 2022-09-07T21:40:13.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.541+0000 7f3c389ff700 1 --2- 172.21.15.47:0/1742616838 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3c1c0990d0 0x7f3c1c09b580 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.541+0000 7f3c389ff700 1 --2- 172.21.15.47:0/1742616838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34103990 0x7f3c3419d190 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.541+0000 7f3c389ff700 1 --2- 172.21.15.47:0/1742616838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c34066be0 0x7f3c34195be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.541+0000 7f3c389ff700 1 --2- 172.21.15.47:0/1742616838 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c34100ea0 0x7f3c34196120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.541+0000 7f3c389ff700 1 -- 172.21.15.47:0/1742616838 >> 172.21.15.47:0/1742616838 conn(0x7f3c340f8370 msgr2=0x7f3c340ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:13.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.541+0000 7f3c389ff700 1 -- 172.21.15.47:0/1742616838 shutdown_connections 2022-09-07T21:40:13.566 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.541+0000 7f3c389ff700 1 -- 172.21.15.47:0/1742616838 wait complete. 2022-09-07T21:40:13.570 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2022-09-07T21:40:13.804 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.801+0000 7f6d03dff700 1 -- 172.21.15.47:0/2008688172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cfc0fef90 msgr2=0x7f6cfc0ff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.801+0000 7f6d03dff700 1 --2- 172.21.15.47:0/2008688172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cfc0fef90 0x7f6cfc0ff400 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6cec009810 tx=0x7f6cec009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:13.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.801+0000 7f6d03dff700 1 -- 172.21.15.47:0/2008688172 shutdown_connections 2022-09-07T21:40:13.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.801+0000 7f6d03dff700 1 --2- 172.21.15.47:0/2008688172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc0ff940 0x7f6cfc106270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.801+0000 7f6d03dff700 1 --2- 172.21.15.47:0/2008688172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6cfc0fddb0 0x7f6cfc0fe1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.801+0000 7f6d03dff700 1 --2- 172.21.15.47:0/2008688172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cfc0fef90 0x7f6cfc0ff400 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.801+0000 7f6d03dff700 1 -- 172.21.15.47:0/2008688172 >> 172.21.15.47:0/2008688172 conn(0x7f6cfc0f9460 msgr2=0x7f6cfc0fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:13.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.801+0000 7f6d03dff700 1 -- 172.21.15.47:0/2008688172 shutdown_connections 2022-09-07T21:40:13.807 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.802+0000 7f6d03dff700 1 -- 172.21.15.47:0/2008688172 wait complete. 2022-09-07T21:40:13.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.803+0000 7f6d03dff700 1 Processor -- start 2022-09-07T21:40:13.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.804+0000 7f6d03dff700 1 -- start start 2022-09-07T21:40:13.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.804+0000 7f6d03dff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc0fddb0 0x7f6cfc195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.804+0000 7f6d03dff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6cfc0fef90 0x7f6cfc1960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.804+0000 7f6d03dff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cfc0ff940 0x7f6cfc19d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.804+0000 7f6d03dff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc19d6f0 con 0x7f6cfc0ff940 2022-09-07T21:40:13.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.804+0000 7f6d03dff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc19d860 con 0x7f6cfc0fef90 2022-09-07T21:40:13.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.804+0000 7f6d03dff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc19da20 con 0x7f6cfc0fddb0 2022-09-07T21:40:13.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.804+0000 7f6d0239c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cfc0ff940 0x7f6cfc19d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.804+0000 7f6d0239c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cfc0ff940 0x7f6cfc19d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57010/0 (socket says 172.21.15.47:57010) 2022-09-07T21:40:13.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.804+0000 7f6d0239c700 1 -- 172.21.15.47:0/1894513684 learned_addr learned my addr 172.21.15.47:0/1894513684 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:13.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.804+0000 7f6d01b9b700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc0fddb0 0x7f6cfc195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.805+0000 7f6d0139a700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6cfc0fef90 0x7f6cfc1960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.805+0000 7f6d0239c700 1 -- 172.21.15.47:0/1894513684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc0fddb0 msgr2=0x7f6cfc195b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.805+0000 7f6d0239c700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc0fddb0 0x7f6cfc195b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.805+0000 7f6d0239c700 1 -- 172.21.15.47:0/1894513684 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6cfc0fef90 msgr2=0x7f6cfc1960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.805+0000 7f6d0239c700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6cfc0fef90 0x7f6cfc1960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.805+0000 7f6d0239c700 1 -- 172.21.15.47:0/1894513684 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec009470 con 0x7f6cfc0ff940 2022-09-07T21:40:13.813 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.805+0000 7f6d0139a700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6cfc0fef90 0x7f6cfc1960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:13.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.805+0000 7f6d0239c700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cfc0ff940 0x7f6cfc19d120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400ebf0 tx=0x7f6cf400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:13.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.805+0000 7f6cf2ffd700 1 -- 172.21.15.47:0/1894513684 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf40099c0 con 0x7f6cfc0ff940 2022-09-07T21:40:13.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.806+0000 7f6cf2ffd700 1 -- 172.21.15.47:0/1894513684 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6cf400cc90 con 0x7f6cfc0ff940 2022-09-07T21:40:13.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.806+0000 7f6d03dff700 1 -- 172.21.15.47:0/1894513684 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cfc19dd00 con 0x7f6cfc0ff940 2022-09-07T21:40:13.814 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.806+0000 7f6cf2ffd700 1 -- 172.21.15.47:0/1894513684 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4010430 con 0x7f6cfc0ff940 2022-09-07T21:40:13.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.806+0000 7f6d03dff700 1 -- 172.21.15.47:0/1894513684 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cfc19e1c0 con 0x7f6cfc0ff940 2022-09-07T21:40:13.819 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.809+0000 7f6cf2ffd700 1 -- 172.21.15.47:0/1894513684 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f6cf401e020 con 0x7f6cfc0ff940 2022-09-07T21:40:13.819 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.811+0000 7f6d173b3b80 1 Processor -- start 2022-09-07T21:40:13.819 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.811+0000 7f6cf2ffd700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6ce8099050 0x7f6ce809b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.819 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.811+0000 7f6d173b3b80 1 -- start start 2022-09-07T21:40:13.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.811+0000 7f6cf2ffd700 1 -- 172.21.15.47:0/1894513684 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f6cf4014070 con 0x7f6cfc0ff940 2022-09-07T21:40:13.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.811+0000 7f6d0139a700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6ce8099050 0x7f6ce809b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.811+0000 7f6d173b3b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a11ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.811+0000 7f6d173b3b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a12430 0x55d4d6a12c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.820 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.811+0000 7f6d173b3b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a131f0 0x55d4d6a13640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.821 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.811+0000 7f6d173b3b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d68fc080 con 0x55d4d6a11ae0 2022-09-07T21:40:13.821 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.811+0000 7f6d173b3b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d69669e0 con 0x55d4d6a131f0 2022-09-07T21:40:13.821 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.811+0000 7f6d173b3b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d6960180 con 0x55d4d6a12430 2022-09-07T21:40:13.821 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.812+0000 7f6d0139a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a12430 0x55d4d6a12c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.822 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.812+0000 7f6d01b9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a11ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.822 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.812+0000 7f6d0239c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a131f0 0x55d4d6a13640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.822 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.812+0000 7f6d0139a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a12430 0x55d4d6a12c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:42332/0 (socket says 172.21.15.47:42332) 2022-09-07T21:40:13.822 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.812+0000 7f6d0239c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a131f0 0x55d4d6a13640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:46446/0 (socket says 172.21.15.47:46446) 2022-09-07T21:40:13.823 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.812+0000 7f6d0139a700 1 -- 172.21.15.47:0/3104308839 learned_addr learned my addr 172.21.15.47:0/3104308839 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:13.823 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.812+0000 7f6d01b9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a11ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57024/0 (socket says 172.21.15.47:57024) 2022-09-07T21:40:13.823 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.812+0000 7f6d0239c700 1 -- 172.21.15.47:0/3104308839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a12430 msgr2=0x55d4d6a12c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.823 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.812+0000 7f6d0239c700 1 --2- 172.21.15.47:0/3104308839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a12430 0x55d4d6a12c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.812+0000 7f6d0239c700 1 -- 172.21.15.47:0/3104308839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a11ae0 msgr2=0x55d4d6a11ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.812+0000 7f6d0239c700 1 --2- 172.21.15.47:0/3104308839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a11ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.812+0000 7f6d0239c700 1 -- 172.21.15.47:0/3104308839 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4d67e85e0 con 0x55d4d6a131f0 2022-09-07T21:40:13.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.812+0000 7f6d01b9b700 1 --2- 172.21.15.47:0/3104308839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a11ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:13.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.813+0000 7f6d0139a700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6ce8099050 0x7f6ce809b500 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6cec005b50 tx=0x7f6cec00fec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:13.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.813+0000 7f6d0239c700 1 --2- 172.21.15.47:0/3104308839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a131f0 0x55d4d6a13640 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6cf408f1c0 tx=0x7f6cf4092800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14776138014218d5 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:13.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.813+0000 7f6ce7fff700 1 -- 172.21.15.47:0/3104308839 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf40980c0 con 0x55d4d6a131f0 2022-09-07T21:40:13.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.813+0000 7f6ce7fff700 1 -- 172.21.15.47:0/3104308839 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6cf4092cc0 con 0x55d4d6a131f0 2022-09-07T21:40:13.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.813+0000 7f6ce7fff700 1 -- 172.21.15.47:0/3104308839 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4016330 con 0x55d4d6a131f0 2022-09-07T21:40:13.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.814+0000 7f6d173b3b80 1 -- 172.21.15.47:0/3104308839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a131f0 msgr2=0x55d4d6a13640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.814+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/3104308839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a131f0 0x55d4d6a13640 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6cf408f1c0 tx=0x7f6cf4092800 comp rx=0 tx=0).stop 2022-09-07T21:40:13.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.814+0000 7f6d173b3b80 1 -- 172.21.15.47:0/3104308839 shutdown_connections 2022-09-07T21:40:13.827 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.814+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/3104308839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a131f0 0x55d4d6a13640 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.827 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.814+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/3104308839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a11ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.827 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.814+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/3104308839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a12430 0x55d4d6a12c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.827 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.814+0000 7f6d173b3b80 1 -- 172.21.15.47:0/3104308839 >> 172.21.15.47:0/3104308839 conn(0x55d4d6a0dfa0 msgr2=0x55d4d6a0c370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:13.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.814+0000 7f6d173b3b80 1 -- 172.21.15.47:0/3104308839 shutdown_connections 2022-09-07T21:40:13.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.814+0000 7f6d173b3b80 1 -- 172.21.15.47:0/3104308839 wait complete. 2022-09-07T21:40:13.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.815+0000 7f6d173b3b80 1 -- start start 2022-09-07T21:40:13.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.815+0000 7f6d173b3b80 1 Processor -- start 2022-09-07T21:40:13.828 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.816+0000 7f6d173b3b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a1b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.829 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.816+0000 7f6d173b3b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a12430 0x55d4d6a1ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.829 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.816+0000 7f6d173b3b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a131f0 0x55d4d6a22aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.829 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.816+0000 7f6d173b3b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d684e0c0 con 0x55d4d6a12430 2022-09-07T21:40:13.829 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.816+0000 7f6d173b3b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d6a22fe0 con 0x55d4d6a11ae0 2022-09-07T21:40:13.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.816+0000 7f6d173b3b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d6a23170 con 0x55d4d6a131f0 2022-09-07T21:40:13.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.816+0000 7f6d173b3b80 10 client.0 _ll_register_callbacks cb 0x55d4d690b370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-07T21:40:13.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.816+0000 7f6d0139a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a12430 0x55d4d6a1ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.816+0000 7f6d01b9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a1b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.816+0000 7f6d01b9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a1b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:46452/0 (socket says 172.21.15.47:46452) 2022-09-07T21:40:13.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.817+0000 7f6d0239c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a131f0 0x55d4d6a22aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.817+0000 7f6d01b9b700 1 -- 172.21.15.47:0/325139111 learned_addr learned my addr 172.21.15.47:0/325139111 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:13.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.816+0000 7f6d0139a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a12430 0x55d4d6a1ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57028/0 (socket says 172.21.15.47:57028) 2022-09-07T21:40:13.835 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.817+0000 7f6d0139a700 1 -- 172.21.15.47:0/325139111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a131f0 msgr2=0x55d4d6a22aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.835 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.817+0000 7f6d0139a700 1 --2- 172.21.15.47:0/325139111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a131f0 0x55d4d6a22aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.835 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.817+0000 7f6d0139a700 1 -- 172.21.15.47:0/325139111 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a11ae0 msgr2=0x55d4d6a1b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.835 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.817+0000 7f6d0139a700 1 --2- 172.21.15.47:0/325139111 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a1b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.836 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.817+0000 7f6d0139a700 1 -- 172.21.15.47:0/325139111 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf8009790 con 0x55d4d6a12430 2022-09-07T21:40:13.836 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.817+0000 7f6d0239c700 1 --2- 172.21.15.47:0/325139111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a131f0 0x55d4d6a22aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:13.836 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.817+0000 7f6d01b9b700 1 --2- 172.21.15.47:0/325139111 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a1b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:13.836 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.817+0000 7f6d0139a700 1 --2- 172.21.15.47:0/325139111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a12430 0x55d4d6a1ba60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6cec005dd0 tx=0x7f6cec009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b57e42fa12d460c server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:13.837 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.817+0000 7f6ce57fa700 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2022-09-07T21:40:13.837 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.818+0000 7f6ce57fa700 1 -- 172.21.15.47:0/325139111 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec004cf0 con 0x55d4d6a12430 2022-09-07T21:40:13.837 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.818+0000 7f6ce57fa700 1 -- 172.21.15.47:0/325139111 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f6cec018400 con 0x55d4d6a12430 2022-09-07T21:40:13.837 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.818+0000 7f6d173b3b80 1 -- 172.21.15.47:0/325139111 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4d6a04aa0 con 0x55d4d6a12430 2022-09-07T21:40:13.838 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.818+0000 7f6ce57fa700 1 -- 172.21.15.47:0/325139111 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6cec004e50 con 0x55d4d6a12430 2022-09-07T21:40:13.838 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.818+0000 7f6d173b3b80 10 client.15027 fetch_fsmap learned FSMap version 24 2022-09-07T21:40:13.838 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.818+0000 7f6d173b3b80 1 -- 172.21.15.47:0/325139111 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7f6cfc19e1c0 con 0x55d4d6a12430 2022-09-07T21:40:13.838 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.818+0000 7f6ce57fa700 1 -- 172.21.15.47:0/325139111 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x55d4d6a04aa0 con 0x55d4d6a12430 2022-09-07T21:40:13.839 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.818+0000 7f6ce57fa700 1 -- 172.21.15.47:0/325139111 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 24) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f6cec0304f0 con 0x55d4d6a12430 2022-09-07T21:40:13.839 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.819+0000 7f6d173b3b80 10 client.15027 fetch_fsmap finished waiting for FSMap version 24 2022-09-07T21:40:13.839 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.819+0000 7f6d173b3b80 10 client.15027 resolve_mds: resolved name 'b' to daemon mds.b 2022-09-07T21:40:13.839 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.819+0000 7f6d173b3b80 20 client.15027 populate_metadata read hostname 'smithi047' 2022-09-07T21:40:13.839 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.819+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/325139111 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x55d4d6a266b0 0x55d4d6a2b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.819+0000 7f6d173b3b80 4 client.15027 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-07T21:40:13.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.819+0000 7f6d173b3b80 1 -- 172.21.15.47:0/325139111 --> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d4d69e14c0 con 0x55d4d6a266b0 2022-09-07T21:40:13.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.819+0000 7f6d01b9b700 1 --2- 172.21.15.47:0/325139111 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x55d4d6a266b0 0x55d4d6a2b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.840 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.820+0000 7f6d01b9b700 1 --2- 172.21.15.47:0/325139111 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x55d4d6a266b0 0x55d4d6a2b990 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6765ffe4956f97db server_cookie=b99e003a01cc8a5a in_seq=0 out_seq=0 2022-09-07T21:40:13.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.820+0000 7f6ce57fa700 10 client.15027 ms_handle_connect on v2:172.21.15.55:6838/517990300 2022-09-07T21:40:13.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.821+0000 7f6ce57fa700 1 -- 172.21.15.47:0/325139111 <== mds.1 v2:172.21.15.55:6838/517990300 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55d4d69e14c0 con 0x55d4d6a266b0 2022-09-07T21:40:13.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.821+0000 7f6ce57fa700 10 client.15027 handle_command_reply: tid=0 2022-09-07T21:40:13.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.821+0000 7f6d173b3b80 1 client.15027 shutdown 2022-09-07T21:40:13.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.821+0000 7f6d173b3b80 1 -- 172.21.15.47:0/325139111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a12430 msgr2=0x55d4d6a1ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.821+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/325139111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a12430 0x55d4d6a1ba60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6cec005dd0 tx=0x7f6cec009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:13.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.821+0000 7f6d173b3b80 1 -- 172.21.15.47:0/325139111 shutdown_connections 2022-09-07T21:40:13.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.821+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/325139111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a131f0 0x55d4d6a22aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.821+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/325139111 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x55d4d6a266b0 0x55d4d6a2b990 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.821+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/325139111 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a1b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.821+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/325139111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a12430 0x55d4d6a1ba60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.821+0000 7f6d173b3b80 1 -- 172.21.15.47:0/325139111 >> 172.21.15.47:0/325139111 conn(0x55d4d6a0dfa0 msgr2=0x55d4d6a0ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:13.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.821+0000 7f6ce57fa700 0 client.15027 ms_handle_reset on v2:172.21.15.55:6838/517990300 2022-09-07T21:40:13.844 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.821+0000 7f6d173b3b80 1 -- 172.21.15.47:0/325139111 shutdown_connections 2022-09-07T21:40:13.844 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.821+0000 7f6d173b3b80 1 -- 172.21.15.47:0/325139111 wait complete. 2022-09-07T21:40:13.844 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.822+0000 7f6d173b3b80 20 client.15027 trim_cache size 0 max 16384 2022-09-07T21:40:13.850 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.840+0000 7f6d173b3b80 1 Processor -- start 2022-09-07T21:40:13.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.840+0000 7f6d173b3b80 1 -- start start 2022-09-07T21:40:13.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.840+0000 7f6d173b3b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a2f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.840+0000 7f6d173b3b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a12430 0x55d4d6a15e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.840+0000 7f6d173b3b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a131f0 0x55d4d6a1b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.852 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.840+0000 7f6d173b3b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d6a07330 con 0x55d4d6a12430 2022-09-07T21:40:13.852 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.840+0000 7f6d173b3b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d69e14c0 con 0x55d4d6a11ae0 2022-09-07T21:40:13.852 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.840+0000 7f6d173b3b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d6a04aa0 con 0x55d4d6a131f0 2022-09-07T21:40:13.852 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.841+0000 7f6d0139a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a12430 0x55d4d6a15e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.852 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.841+0000 7f6d01b9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a2f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.853 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.841+0000 7f6d01b9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a2f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:46468/0 (socket says 172.21.15.47:46468) 2022-09-07T21:40:13.853 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.841+0000 7f6d01b9b700 1 -- 172.21.15.47:0/1882450172 learned_addr learned my addr 172.21.15.47:0/1882450172 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:13.853 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.841+0000 7f6d0239c700 1 --2- 172.21.15.47:0/1882450172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a131f0 0x55d4d6a1b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.853 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.841+0000 7f6d01b9b700 1 -- 172.21.15.47:0/1882450172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a131f0 msgr2=0x55d4d6a1b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.841+0000 7f6d01b9b700 1 --2- 172.21.15.47:0/1882450172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a131f0 0x55d4d6a1b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.841+0000 7f6d01b9b700 1 -- 172.21.15.47:0/1882450172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a12430 msgr2=0x55d4d6a15e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.841+0000 7f6d01b9b700 1 --2- 172.21.15.47:0/1882450172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a12430 0x55d4d6a15e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.841+0000 7f6d01b9b700 1 -- 172.21.15.47:0/1882450172 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4d6794d90 con 0x55d4d6a11ae0 2022-09-07T21:40:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.841+0000 7f6d0139a700 1 --2- 172.21.15.47:0/1882450172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a12430 0x55d4d6a15e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.841+0000 7f6d01b9b700 1 --2- 172.21.15.47:0/1882450172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a2f9b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8000c00 tx=0x7f6cf8005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=370792ae48406db0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.842+0000 7f6cf0bff700 1 -- 172.21.15.47:0/1882450172 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8017070 con 0x55d4d6a11ae0 2022-09-07T21:40:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.842+0000 7f6cf0bff700 1 -- 172.21.15.47:0/1882450172 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6cf8018040 con 0x55d4d6a11ae0 2022-09-07T21:40:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.842+0000 7f6cf0bff700 1 -- 172.21.15.47:0/1882450172 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf80046b0 con 0x55d4d6a11ae0 2022-09-07T21:40:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.842+0000 7f6d173b3b80 1 -- 172.21.15.47:0/1882450172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a11ae0 msgr2=0x55d4d6a2f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.842+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/1882450172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a2f9b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8000c00 tx=0x7f6cf8005ea0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.842+0000 7f6d173b3b80 1 -- 172.21.15.47:0/1882450172 shutdown_connections 2022-09-07T21:40:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.842+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/1882450172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a131f0 0x55d4d6a1b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.842+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/1882450172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a2f9b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.842+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/1882450172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a12430 0x55d4d6a15e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.842+0000 7f6d173b3b80 1 -- 172.21.15.47:0/1882450172 >> 172.21.15.47:0/1882450172 conn(0x55d4d6a0dfa0 msgr2=0x55d4d6a0ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.842+0000 7f6d173b3b80 1 -- 172.21.15.47:0/1882450172 shutdown_connections 2022-09-07T21:40:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.842+0000 7f6d173b3b80 1 -- 172.21.15.47:0/1882450172 wait complete. 2022-09-07T21:40:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.843+0000 7f6d173b3b80 1 -- start start 2022-09-07T21:40:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.843+0000 7f6d173b3b80 1 Processor -- start 2022-09-07T21:40:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.843+0000 7f6d173b3b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a1fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.843+0000 7f6d173b3b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a12430 0x55d4d6a201f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.843+0000 7f6d173b3b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a131f0 0x55d4d6a78930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.843+0000 7f6d173b3b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d6a20920 con 0x55d4d6a11ae0 2022-09-07T21:40:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.843+0000 7f6d173b3b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d6a78ea0 con 0x55d4d6a131f0 2022-09-07T21:40:13.860 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.844+0000 7f6d173b3b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d6a79060 con 0x55d4d6a12430 2022-09-07T21:40:13.860 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.844+0000 7f6d173b3b80 10 client.0 _ll_register_callbacks cb 0x55d4d690b370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-07T21:40:13.860 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.844+0000 7f6d01b9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a1fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.860 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.844+0000 7f6d0239c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a131f0 0x55d4d6a78930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.860 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.844+0000 7f6d01b9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a1fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57038/0 (socket says 172.21.15.47:57038) 2022-09-07T21:40:13.861 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.844+0000 7f6d01b9b700 1 -- 172.21.15.47:0/373882264 learned_addr learned my addr 172.21.15.47:0/373882264 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:13.861 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.844+0000 7f6d0139a700 1 --2- 172.21.15.47:0/373882264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a12430 0x55d4d6a201f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.861 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.844+0000 7f6d0239c700 1 -- 172.21.15.47:0/373882264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a12430 msgr2=0x55d4d6a201f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.861 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.844+0000 7f6d0239c700 1 --2- 172.21.15.47:0/373882264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a12430 0x55d4d6a201f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.844+0000 7f6d0239c700 1 -- 172.21.15.47:0/373882264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a11ae0 msgr2=0x55d4d6a1fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:13.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.844+0000 7f6d0239c700 1 --2- 172.21.15.47:0/373882264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a1fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.844+0000 7f6d0239c700 1 -- 172.21.15.47:0/373882264 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cfc19e1c0 con 0x55d4d6a131f0 2022-09-07T21:40:13.862 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.844+0000 7f6d0139a700 1 --2- 172.21.15.47:0/373882264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a12430 0x55d4d6a201f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:13.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.844+0000 7f6d01b9b700 1 --2- 172.21.15.47:0/373882264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a1fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:13.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.845+0000 7f6d0239c700 1 --2- 172.21.15.47:0/373882264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a131f0 0x55d4d6a78930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4092c50 tx=0x7f6cf4009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=981a4c486ca58e05 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:13.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.845+0000 7f6ce57fa700 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2022-09-07T21:40:13.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.845+0000 7f6ce57fa700 1 -- 172.21.15.47:0/373882264 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4098050 con 0x55d4d6a131f0 2022-09-07T21:40:13.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.845+0000 7f6ce57fa700 1 -- 172.21.15.47:0/373882264 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f6cf40ad070 con 0x55d4d6a131f0 2022-09-07T21:40:13.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.845+0000 7f6d173b3b80 1 -- 172.21.15.47:0/373882264 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6cf4092cc0 con 0x55d4d6a131f0 2022-09-07T21:40:13.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.845+0000 7f6ce57fa700 1 -- 172.21.15.47:0/373882264 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6cf40c0380 con 0x55d4d6a131f0 2022-09-07T21:40:13.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.845+0000 7f6d173b3b80 10 client.25103 fetch_fsmap learned FSMap version 24 2022-09-07T21:40:13.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.845+0000 7f6d173b3b80 1 -- 172.21.15.47:0/373882264 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7f6cf8012040 con 0x55d4d6a131f0 2022-09-07T21:40:13.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.846+0000 7f6ce57fa700 1 -- 172.21.15.47:0/373882264 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6cf4092cc0 con 0x55d4d6a131f0 2022-09-07T21:40:13.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.846+0000 7f6ce57fa700 1 -- 172.21.15.47:0/373882264 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 24) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f6cf4079020 con 0x55d4d6a131f0 2022-09-07T21:40:13.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.846+0000 7f6d173b3b80 10 client.25103 fetch_fsmap finished waiting for FSMap version 24 2022-09-07T21:40:13.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.846+0000 7f6d173b3b80 10 client.25103 resolve_mds: resolved name 'b' to daemon mds.b 2022-09-07T21:40:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.846+0000 7f6d173b3b80 20 client.25103 populate_metadata read hostname 'smithi047' 2022-09-07T21:40:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.846+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/373882264 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x55d4d6a7db20 0x55d4d6a7a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.846+0000 7f6d173b3b80 4 client.25103 mds_command: new command op to 24572 tid=0[{"prefix": "status"}] 2022-09-07T21:40:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.846+0000 7f6d173b3b80 1 -- 172.21.15.47:0/373882264 --> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d4d69596f0 con 0x55d4d6a7db20 2022-09-07T21:40:13.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.846+0000 7f6d01b9b700 1 --2- 172.21.15.47:0/373882264 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x55d4d6a7db20 0x55d4d6a7a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:13.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.847+0000 7f6d01b9b700 1 --2- 172.21.15.47:0/373882264 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x55d4d6a7db20 0x55d4d6a7a490 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=defda69541941cf0 server_cookie=db2a7a2f16d11188 in_seq=0 out_seq=0 2022-09-07T21:40:13.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.847+0000 7f6ce57fa700 10 client.25103 ms_handle_connect on v2:172.21.15.55:6838/517990300 2022-09-07T21:40:13.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.847+0000 7f6ce57fa700 1 -- 172.21.15.47:0/373882264 <== mds.1 v2:172.21.15.55:6838/517990300 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55d4d69596f0 con 0x55d4d6a7db20 2022-09-07T21:40:13.867 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.847+0000 7f6ce57fa700 10 client.25103 handle_command_reply: tid=0 2022-09-07T21:40:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.847+0000 7f6d173b3b80 1 client.25103 shutdown 2022-09-07T21:40:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.847+0000 7f6d173b3b80 1 -- 172.21.15.47:0/373882264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a131f0 msgr2=0x55d4d6a78930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.847+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/373882264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a131f0 0x55d4d6a78930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4092c50 tx=0x7f6cf4009f60 comp rx=0 tx=0).stop 2022-09-07T21:40:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.847+0000 7f6d173b3b80 1 -- 172.21.15.47:0/373882264 shutdown_connections 2022-09-07T21:40:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.847+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/373882264 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x55d4d6a7db20 0x55d4d6a7a490 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.847+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/373882264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d4d6a131f0 0x55d4d6a78930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.847+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/373882264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d4d6a11ae0 0x55d4d6a1fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.847+0000 7f6d173b3b80 1 --2- 172.21.15.47:0/373882264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d4d6a12430 0x55d4d6a201f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.847+0000 7f6ce57fa700 0 client.25103 ms_handle_reset on v2:172.21.15.55:6838/517990300 2022-09-07T21:40:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.847+0000 7f6d173b3b80 1 -- 172.21.15.47:0/373882264 >> 172.21.15.47:0/373882264 conn(0x55d4d6a0dfa0 msgr2=0x55d4d6a2f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.848+0000 7f6d173b3b80 1 -- 172.21.15.47:0/373882264 shutdown_connections 2022-09-07T21:40:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.848+0000 7f6d173b3b80 1 -- 172.21.15.47:0/373882264 wait complete. 2022-09-07T21:40:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.848+0000 7f6d173b3b80 20 client.25103 trim_cache size 0 max 16384 2022-09-07T21:40:13.871 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2022-09-07T21:40:13.871 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "01b8396e-2ef4-11ed-8431-001a4aab830c", 2022-09-07T21:40:13.872 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 1, 2022-09-07T21:40:13.872 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 24572, 2022-09-07T21:40:13.872 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2022-09-07T21:40:13.872 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2022-09-07T21:40:13.872 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2022-09-07T21:40:13.873 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 69.049938620000006, 2022-09-07T21:40:13.873 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 24, 2022-09-07T21:40:13.873 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 91, 2022-09-07T21:40:13.873 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 89, 2022-09-07T21:40:13.874 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 98.730005852999994 2022-09-07T21:40:13.874 INFO:teuthology.orchestra.run.smithi047.stdout:} 2022-09-07T21:40:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.848+0000 7f6d03dff700 1 -- 172.21.15.47:0/1894513684 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6ce8099050 msgr2=0x7f6ce809b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.848+0000 7f6d03dff700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6ce8099050 0x7f6ce809b500 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6cec005b50 tx=0x7f6cec00fec0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.848+0000 7f6d03dff700 1 -- 172.21.15.47:0/1894513684 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cfc0ff940 msgr2=0x7f6cfc19d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.848+0000 7f6d03dff700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cfc0ff940 0x7f6cfc19d120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400ebf0 tx=0x7f6cf400ef00 comp rx=0 tx=0).stop 2022-09-07T21:40:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.849+0000 7f6d03dff700 1 -- 172.21.15.47:0/1894513684 shutdown_connections 2022-09-07T21:40:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.849+0000 7f6d03dff700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f6ce8099050 0x7f6ce809b500 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.849+0000 7f6d03dff700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cfc0ff940 0x7f6cfc19d120 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.849+0000 7f6d03dff700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc0fddb0 0x7f6cfc195b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.849+0000 7f6d03dff700 1 --2- 172.21.15.47:0/1894513684 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6cfc0fef90 0x7f6cfc1960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.849+0000 7f6d03dff700 1 -- 172.21.15.47:0/1894513684 >> 172.21.15.47:0/1894513684 conn(0x7f6cfc0f9460 msgr2=0x7f6cfc0fb6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.849+0000 7f6d03dff700 1 -- 172.21.15.47:0/1894513684 shutdown_connections 2022-09-07T21:40:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:13.849+0000 7f6d03dff700 1 -- 172.21.15.47:0/1894513684 wait complete. 2022-09-07T21:40:13.878 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:13 smithi047 ceph-mon[90087]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:13.878 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:13 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/848731489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:40:13.879 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2022-09-07T21:40:14.113 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.110+0000 7fa4da3ff700 1 -- 172.21.15.47:0/3232752140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4d4100ea0 msgr2=0x7fa4d4103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:14.113 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.110+0000 7fa4da3ff700 1 --2- 172.21.15.47:0/3232752140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4d4100ea0 0x7fa4d4103300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009810 tx=0x7fa4c8009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:14.113 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.110+0000 7fa4da3ff700 1 -- 172.21.15.47:0/3232752140 shutdown_connections 2022-09-07T21:40:14.113 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.110+0000 7fa4da3ff700 1 --2- 172.21.15.47:0/3232752140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4d4103990 0x7fa4d4105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.114 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.110+0000 7fa4da3ff700 1 --2- 172.21.15.47:0/3232752140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d4066be0 0x7fa4d4100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.114 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.110+0000 7fa4da3ff700 1 --2- 172.21.15.47:0/3232752140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4d4100ea0 0x7fa4d4103300 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.114 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.110+0000 7fa4da3ff700 1 -- 172.21.15.47:0/3232752140 >> 172.21.15.47:0/3232752140 conn(0x7fa4d40f8370 msgr2=0x7fa4d40fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:14.114 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.110+0000 7fa4da3ff700 1 -- 172.21.15.47:0/3232752140 shutdown_connections 2022-09-07T21:40:14.115 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.110+0000 7fa4da3ff700 1 -- 172.21.15.47:0/3232752140 wait complete. 2022-09-07T21:40:14.117 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.111+0000 7fa4da3ff700 1 Processor -- start 2022-09-07T21:40:14.117 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.111+0000 7fa4da3ff700 1 -- start start 2022-09-07T21:40:14.117 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.112+0000 7fa4da3ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d4066be0 0x7fa4d4195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.118 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.112+0000 7fa4da3ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4d4100ea0 0x7fa4d4196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.118 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.112+0000 7fa4da3ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4d4103990 0x7fa4d419d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.118 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.112+0000 7fa4da3ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d419d750 con 0x7fa4d4103990 2022-09-07T21:40:14.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.112+0000 7fa4da3ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d419d8c0 con 0x7fa4d4066be0 2022-09-07T21:40:14.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.112+0000 7fa4da3ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d419da80 con 0x7fa4d4100ea0 2022-09-07T21:40:14.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.112+0000 7fa4d899c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4d4103990 0x7fa4d419d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.112+0000 7fa4d3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d4066be0 0x7fa4d4195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.112+0000 7fa4d899c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4d4103990 0x7fa4d419d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57056/0 (socket says 172.21.15.47:57056) 2022-09-07T21:40:14.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.112+0000 7fa4d899c700 1 -- 172.21.15.47:0/20883469 learned_addr learned my addr 172.21.15.47:0/20883469 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:14.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.112+0000 7fa4d37fe700 1 --2- 172.21.15.47:0/20883469 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4d4100ea0 0x7fa4d4196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.112+0000 7fa4d3fff700 1 -- 172.21.15.47:0/20883469 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4d4100ea0 msgr2=0x7fa4d4196110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:14.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.112+0000 7fa4d3fff700 1 --2- 172.21.15.47:0/20883469 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4d4100ea0 0x7fa4d4196110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.113+0000 7fa4d3fff700 1 -- 172.21.15.47:0/20883469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4d4103990 msgr2=0x7fa4d419d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:14.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.113+0000 7fa4d3fff700 1 --2- 172.21.15.47:0/20883469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4d4103990 0x7fa4d419d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.113+0000 7fa4d3fff700 1 -- 172.21.15.47:0/20883469 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c8009470 con 0x7fa4d4066be0 2022-09-07T21:40:14.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.113+0000 7fa4d899c700 1 --2- 172.21.15.47:0/20883469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4d4103990 0x7fa4d419d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:14.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.113+0000 7fa4d3fff700 1 --2- 172.21.15.47:0/20883469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d4066be0 0x7fa4d4195bd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4066990 tx=0x7fa4c400e9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:14.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.113+0000 7fa4d17fa700 1 -- 172.21.15.47:0/20883469 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4010840 con 0x7fa4d4066be0 2022-09-07T21:40:14.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.113+0000 7fa4d17fa700 1 -- 172.21.15.47:0/20883469 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa4c4010e40 con 0x7fa4d4066be0 2022-09-07T21:40:14.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.113+0000 7fa4da3ff700 1 -- 172.21.15.47:0/20883469 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d419dd60 con 0x7fa4d4066be0 2022-09-07T21:40:14.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.113+0000 7fa4d17fa700 1 -- 172.21.15.47:0/20883469 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4017630 con 0x7fa4d4066be0 2022-09-07T21:40:14.123 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.113+0000 7fa4da3ff700 1 -- 172.21.15.47:0/20883469 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d419e200 con 0x7fa4d4066be0 2022-09-07T21:40:14.130 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.117+0000 7fa4d17fa700 1 -- 172.21.15.47:0/20883469 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7fa4c400c040 con 0x7fa4d4066be0 2022-09-07T21:40:14.130 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.118+0000 7fa4ed8e1b80 1 Processor -- start 2022-09-07T21:40:14.130 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.118+0000 7fa4ed8e1b80 1 -- start start 2022-09-07T21:40:14.130 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.118+0000 7fa4ed8e1b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2eae0 0x55fab2f2eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.131 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.118+0000 7fa4ed8e1b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f2f430 0x55fab2f2fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.131 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.118+0000 7fa4ed8e1b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f301f0 0x55fab2f30640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.131 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.118+0000 7fa4ed8e1b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab2e19080 con 0x55fab2f2eae0 2022-09-07T21:40:14.131 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.118+0000 7fa4ed8e1b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab2e839e0 con 0x55fab2f301f0 2022-09-07T21:40:14.131 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.118+0000 7fa4ed8e1b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab2e7d180 con 0x55fab2f2f430 2022-09-07T21:40:14.132 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.118+0000 7fa4d17fa700 1 --2- 172.21.15.47:0/20883469 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4bc0990d0 0x7fa4bc09b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.132 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.119+0000 7fa4d17fa700 1 -- 172.21.15.47:0/20883469 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fa4c4014070 con 0x7fa4d4066be0 2022-09-07T21:40:14.132 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.119+0000 7fa4d37fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f301f0 0x55fab2f30640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.132 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.119+0000 7fa4d37fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f301f0 0x55fab2f30640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:46498/0 (socket says 172.21.15.47:46498) 2022-09-07T21:40:14.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.119+0000 7fa4d37fe700 1 -- 172.21.15.47:0/3848844314 learned_addr learned my addr 172.21.15.47:0/3848844314 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:14.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.119+0000 7fa4d3fff700 1 --2- 172.21.15.47:0/3848844314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f2f430 0x55fab2f2fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.119+0000 7fa4d899c700 1 --2- 172.21.15.47:0/3848844314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2eae0 0x55fab2f2eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.119+0000 7fa4d899c700 1 --2- 172.21.15.47:0/20883469 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4bc0990d0 0x7fa4bc09b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.119+0000 7fa4d37fe700 1 -- 172.21.15.47:0/3848844314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f2f430 msgr2=0x55fab2f2fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:14.134 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.119+0000 7fa4d37fe700 1 --2- 172.21.15.47:0/3848844314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f2f430 0x55fab2f2fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.134 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.119+0000 7fa4d37fe700 1 -- 172.21.15.47:0/3848844314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2eae0 msgr2=0x55fab2f2eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:14.134 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.119+0000 7fa4d37fe700 1 --2- 172.21.15.47:0/3848844314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2eae0 0x55fab2f2eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.134 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.119+0000 7fa4d37fe700 1 -- 172.21.15.47:0/3848844314 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fab2d055e0 con 0x55fab2f301f0 2022-09-07T21:40:14.135 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.120+0000 7fa4d37fe700 1 --2- 172.21.15.47:0/3848844314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f301f0 0x55fab2f30640 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8015000 tx=0x7fa4c8005d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3236b155008e7860 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:14.135 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.120+0000 7fa4d899c700 1 --2- 172.21.15.47:0/20883469 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4bc0990d0 0x7fa4bc09b580 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x55fab2f2e880 tx=0x7fa4cc009410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:14.135 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.120+0000 7fa4b67fc700 1 -- 172.21.15.47:0/3848844314 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c801d070 con 0x55fab2f301f0 2022-09-07T21:40:14.135 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.120+0000 7fa4b67fc700 1 -- 172.21.15.47:0/3848844314 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa4c801e040 con 0x55fab2f301f0 2022-09-07T21:40:14.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.120+0000 7fa4b67fc700 1 -- 172.21.15.47:0/3848844314 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8004760 con 0x55fab2f301f0 2022-09-07T21:40:14.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.121+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/3848844314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f301f0 msgr2=0x55fab2f30640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:14.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.121+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/3848844314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f301f0 0x55fab2f30640 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8015000 tx=0x7fa4c8005d00 comp rx=0 tx=0).stop 2022-09-07T21:40:14.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.121+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/3848844314 shutdown_connections 2022-09-07T21:40:14.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.121+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/3848844314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f301f0 0x55fab2f30640 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.121+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/3848844314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2eae0 0x55fab2f2eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.121+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/3848844314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f2f430 0x55fab2f2fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.121+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/3848844314 >> 172.21.15.47:0/3848844314 conn(0x55fab2f2afa0 msgr2=0x55fab2f29370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:14.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.121+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/3848844314 shutdown_connections 2022-09-07T21:40:14.138 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.121+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/3848844314 wait complete. 2022-09-07T21:40:14.138 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.122+0000 7fa4ed8e1b80 1 -- start start 2022-09-07T21:40:14.138 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.122+0000 7fa4ed8e1b80 1 Processor -- start 2022-09-07T21:40:14.138 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.122+0000 7fa4ed8e1b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f2eae0 0x55fab2f38470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.139 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.122+0000 7fa4ed8e1b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f389b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.139 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.122+0000 7fa4ed8e1b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f301f0 0x55fab2f3f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.139 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.122+0000 7fa4ed8e1b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab2d6b0c0 con 0x55fab2f2f430 2022-09-07T21:40:14.139 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.122+0000 7fa4ed8e1b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab2f3ff30 con 0x55fab2f301f0 2022-09-07T21:40:14.140 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.122+0000 7fa4ed8e1b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab2f400c0 con 0x55fab2f2eae0 2022-09-07T21:40:14.140 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.122+0000 7fa4ed8e1b80 10 client.0 _ll_register_callbacks cb 0x55fab2e28370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-07T21:40:14.140 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.123+0000 7fa4d37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f389b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.140 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.123+0000 7fa4d899c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f301f0 0x55fab2f3f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.141 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.123+0000 7fa4d3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f2eae0 0x55fab2f38470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.141 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.123+0000 7fa4d37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f389b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57072/0 (socket says 172.21.15.47:57072) 2022-09-07T21:40:14.141 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.123+0000 7fa4d37fe700 1 -- 172.21.15.47:0/1437212839 learned_addr learned my addr 172.21.15.47:0/1437212839 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:14.141 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.123+0000 7fa4d3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f2eae0 0x55fab2f38470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:42392/0 (socket says 172.21.15.47:42392) 2022-09-07T21:40:14.141 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.123+0000 7fa4d3fff700 1 -- 172.21.15.47:0/1437212839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f301f0 msgr2=0x55fab2f3f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:14.142 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.123+0000 7fa4d3fff700 1 --2- 172.21.15.47:0/1437212839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f301f0 0x55fab2f3f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.142 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.123+0000 7fa4d3fff700 1 -- 172.21.15.47:0/1437212839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 msgr2=0x55fab2f389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:14.142 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.123+0000 7fa4d3fff700 1 --2- 172.21.15.47:0/1437212839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f389b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.142 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.123+0000 7fa4d3fff700 1 -- 172.21.15.47:0/1437212839 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d4106ad0 con 0x55fab2f2eae0 2022-09-07T21:40:14.143 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.123+0000 7fa4d3fff700 1 --2- 172.21.15.47:0/1437212839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f2eae0 0x55fab2f38470 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa4c408f260 tx=0x7fa4c4092150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deb7b592e3082a6f server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:14.143 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.123+0000 7fa4af7fe700 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2022-09-07T21:40:14.143 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.124+0000 7fa4af7fe700 1 -- 172.21.15.47:0/1437212839 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4099a90 con 0x55fab2f2eae0 2022-09-07T21:40:14.143 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.124+0000 7fa4af7fe700 1 -- 172.21.15.47:0/1437212839 <== mon.1 v2:172.21.15.59:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fa4c40a08a0 con 0x55fab2f2eae0 2022-09-07T21:40:14.143 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.124+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/1437212839 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fab2f21aa0 con 0x55fab2f2eae0 2022-09-07T21:40:14.144 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.124+0000 7fa4af7fe700 1 -- 172.21.15.47:0/1437212839 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4c40a0a60 con 0x55fab2f2eae0 2022-09-07T21:40:14.144 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.124+0000 7fa4ed8e1b80 10 client.25144 fetch_fsmap learned FSMap version 24 2022-09-07T21:40:14.144 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.124+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/1437212839 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55fab2cb1d90 con 0x55fab2f2eae0 2022-09-07T21:40:14.144 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.124+0000 7fa4af7fe700 1 -- 172.21.15.47:0/1437212839 <== mon.1 v2:172.21.15.59:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x55fab2f21aa0 con 0x55fab2f2eae0 2022-09-07T21:40:14.145 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.124+0000 7fa4af7fe700 1 -- 172.21.15.47:0/1437212839 <== mon.1 v2:172.21.15.59:3300/0 5 ==== fsmap(e 24) v1 ==== 7456+0+0 (secure 0 0 0) 0x7fa4c4099600 con 0x55fab2f2eae0 2022-09-07T21:40:14.145 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.124+0000 7fa4ed8e1b80 10 client.25144 fetch_fsmap finished waiting for FSMap version 24 2022-09-07T21:40:14.145 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.124+0000 7fa4ed8e1b80 10 client.25144 resolve_mds: resolved name 'a' to daemon mds.a 2022-09-07T21:40:14.145 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.124+0000 7fa4ed8e1b80 20 client.25144 populate_metadata read hostname 'smithi047' 2022-09-07T21:40:14.146 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.125+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/1437212839 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55fab2f436f0 0x55fab2f48a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.146 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.125+0000 7fa4ed8e1b80 4 client.25144 mds_command: new command op to 24586 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-07T21:40:14.146 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.125+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/1437212839 --> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fab2efe4c0 con 0x55fab2f436f0 2022-09-07T21:40:14.146 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.125+0000 7fa4d37fe700 1 --2- 172.21.15.47:0/1437212839 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55fab2f436f0 0x55fab2f48a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.147 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.125+0000 7fa4d37fe700 1 --2- 172.21.15.47:0/1437212839 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55fab2f436f0 0x55fab2f48a10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e684923779286922 server_cookie=99de91bedeed4796 in_seq=0 out_seq=0 2022-09-07T21:40:14.147 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.125+0000 7fa4af7fe700 10 client.25144 ms_handle_connect on v2:172.21.15.47:6840/793863162 2022-09-07T21:40:14.147 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.126+0000 7fa4af7fe700 1 -- 172.21.15.47:0/1437212839 <== mds.0 v2:172.21.15.47:6840/793863162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55fab2efe4c0 con 0x55fab2f436f0 2022-09-07T21:40:14.147 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.126+0000 7fa4af7fe700 10 client.25144 handle_command_reply: tid=0 2022-09-07T21:40:14.147 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.126+0000 7fa4ed8e1b80 1 client.25144 shutdown 2022-09-07T21:40:14.148 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.126+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/1437212839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f2eae0 msgr2=0x55fab2f38470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:14.148 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.126+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/1437212839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f2eae0 0x55fab2f38470 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa4c408f260 tx=0x7fa4c4092150 comp rx=0 tx=0).stop 2022-09-07T21:40:14.148 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.127+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/1437212839 shutdown_connections 2022-09-07T21:40:14.148 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.127+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/1437212839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f301f0 0x55fab2f3f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.149 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.127+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/1437212839 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55fab2f436f0 0x55fab2f48a10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.149 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.127+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/1437212839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f2eae0 0x55fab2f38470 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.149 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.127+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/1437212839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f389b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.149 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.127+0000 7fa4af7fe700 0 client.25144 ms_handle_reset on v2:172.21.15.47:6840/793863162 2022-09-07T21:40:14.150 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.127+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/1437212839 >> 172.21.15.47:0/1437212839 conn(0x55fab2f2afa0 msgr2=0x55fab2f28a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:14.150 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.127+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/1437212839 shutdown_connections 2022-09-07T21:40:14.150 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.127+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/1437212839 wait complete. 2022-09-07T21:40:14.150 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.127+0000 7fa4ed8e1b80 20 client.25144 trim_cache size 0 max 16384 2022-09-07T21:40:14.156 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.145+0000 7fa4ed8e1b80 1 Processor -- start 2022-09-07T21:40:14.156 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.145+0000 7fa4ed8e1b80 1 -- start start 2022-09-07T21:40:14.157 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.145+0000 7fa4ed8e1b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f2eae0 0x55fab2f4ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.157 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.145+0000 7fa4ed8e1b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f32e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.157 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.145+0000 7fa4ed8e1b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f301f0 0x55fab2f36220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.157 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.145+0000 7fa4ed8e1b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab2e41050 con 0x55fab2f2f430 2022-09-07T21:40:14.157 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.145+0000 7fa4ed8e1b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab2e1f4f0 con 0x55fab2f2eae0 2022-09-07T21:40:14.158 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.145+0000 7fa4ed8e1b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab2d09a30 con 0x55fab2f301f0 2022-09-07T21:40:14.158 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.145+0000 7fa4d37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f32e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.158 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.145+0000 7fa4d37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f32e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57082/0 (socket says 172.21.15.47:57082) 2022-09-07T21:40:14.158 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.146+0000 7fa4d37fe700 1 -- 172.21.15.47:0/3788639559 learned_addr learned my addr 172.21.15.47:0/3788639559 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:14.159 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.146+0000 7fa4d899c700 1 --2- 172.21.15.47:0/3788639559 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f301f0 0x55fab2f36220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.159 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.146+0000 7fa4d3fff700 1 --2- 172.21.15.47:0/3788639559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f2eae0 0x55fab2f4ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.159 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.146+0000 7fa4d37fe700 1 -- 172.21.15.47:0/3788639559 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f301f0 msgr2=0x55fab2f36220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:14.159 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.146+0000 7fa4d37fe700 1 --2- 172.21.15.47:0/3788639559 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f301f0 0x55fab2f36220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.160 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.146+0000 7fa4d37fe700 1 -- 172.21.15.47:0/3788639559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f2eae0 msgr2=0x55fab2f4ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:14.160 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.146+0000 7fa4d37fe700 1 --2- 172.21.15.47:0/3788639559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f2eae0 0x55fab2f4ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.160 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.146+0000 7fa4d37fe700 1 -- 172.21.15.47:0/3788639559 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fab2cb0260 con 0x55fab2f2f430 2022-09-07T21:40:14.160 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.146+0000 7fa4d37fe700 1 --2- 172.21.15.47:0/3788639559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f32e90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8000c00 tx=0x7fa4c800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bbbcaeffd23d440 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:14.160 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.146+0000 7fa4b6ffd700 1 -- 172.21.15.47:0/3788639559 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c801d040 con 0x55fab2f2f430 2022-09-07T21:40:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.147+0000 7fa4b6ffd700 1 -- 172.21.15.47:0/3788639559 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa4c8009260 con 0x55fab2f2f430 2022-09-07T21:40:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.147+0000 7fa4b6ffd700 1 -- 172.21.15.47:0/3788639559 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c800f920 con 0x55fab2f2f430 2022-09-07T21:40:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.147+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/3788639559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 msgr2=0x55fab2f32e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.147+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/3788639559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f32e90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8000c00 tx=0x7fa4c800b260 comp rx=0 tx=0).stop 2022-09-07T21:40:14.162 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.147+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/3788639559 shutdown_connections 2022-09-07T21:40:14.162 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.147+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/3788639559 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f301f0 0x55fab2f36220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.162 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.147+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/3788639559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f2eae0 0x55fab2f4ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.162 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.147+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/3788639559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f32e90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.163 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.147+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/3788639559 >> 172.21.15.47:0/3788639559 conn(0x55fab2f2afa0 msgr2=0x55fab2f404a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:14.163 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.147+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/3788639559 shutdown_connections 2022-09-07T21:40:14.163 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.147+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/3788639559 wait complete. 2022-09-07T21:40:14.163 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.148+0000 7fa4ed8e1b80 1 -- start start 2022-09-07T21:40:14.163 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.148+0000 7fa4ed8e1b80 1 Processor -- start 2022-09-07T21:40:14.164 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.148+0000 7fa4ed8e1b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f2eae0 0x55fab2f3c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.164 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.148+0000 7fa4ed8e1b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f3cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.164 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4ed8e1b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f301f0 0x55fab2f3d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.164 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4ed8e1b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab2f95a70 con 0x55fab2f2f430 2022-09-07T21:40:14.165 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4ed8e1b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab2f95be0 con 0x55fab2f2eae0 2022-09-07T21:40:14.165 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4ed8e1b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab2f95da0 con 0x55fab2f301f0 2022-09-07T21:40:14.165 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4ed8e1b80 10 client.0 _ll_register_callbacks cb 0x55fab2e28370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-07T21:40:14.165 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4d37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f3cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.165 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4d3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f2eae0 0x55fab2f3c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.166 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4d3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f2eae0 0x55fab2f3c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:46528/0 (socket says 172.21.15.47:46528) 2022-09-07T21:40:14.166 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4d3fff700 1 -- 172.21.15.47:0/2941636727 learned_addr learned my addr 172.21.15.47:0/2941636727 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:14.166 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4d899c700 1 --2- 172.21.15.47:0/2941636727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f301f0 0x55fab2f3d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.166 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4d3fff700 1 -- 172.21.15.47:0/2941636727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f301f0 msgr2=0x55fab2f3d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:14.167 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4d3fff700 1 --2- 172.21.15.47:0/2941636727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f301f0 0x55fab2f3d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.167 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4d3fff700 1 -- 172.21.15.47:0/2941636727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 msgr2=0x55fab2f3cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:14.167 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4d3fff700 1 --2- 172.21.15.47:0/2941636727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f3cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.167 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.149+0000 7fa4d3fff700 1 -- 172.21.15.47:0/2941636727 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fab2d055e0 con 0x55fab2f2eae0 2022-09-07T21:40:14.167 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.150+0000 7fa4d3fff700 1 --2- 172.21.15.47:0/2941636727 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f2eae0 0x55fab2f3c9b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4005fd0 tx=0x7fa4c4091f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9886133db29cb1eb server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:14.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.150+0000 7fa4af7fe700 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2022-09-07T21:40:14.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.150+0000 7fa4af7fe700 1 -- 172.21.15.47:0/2941636727 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c40af270 con 0x55fab2f2eae0 2022-09-07T21:40:14.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.150+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/2941636727 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4c801e040 con 0x55fab2f2eae0 2022-09-07T21:40:14.168 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.150+0000 7fa4af7fe700 1 -- 172.21.15.47:0/2941636727 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fa4c4099a90 con 0x55fab2f2eae0 2022-09-07T21:40:14.169 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.150+0000 7fa4af7fe700 1 -- 172.21.15.47:0/2941636727 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4c4099810 con 0x55fab2f2eae0 2022-09-07T21:40:14.169 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.150+0000 7fa4ed8e1b80 10 client.25121 fetch_fsmap learned FSMap version 24 2022-09-07T21:40:14.169 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.150+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/2941636727 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55fab2d7ea30 con 0x55fab2f2eae0 2022-09-07T21:40:14.169 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.150+0000 7fa4af7fe700 1 -- 172.21.15.47:0/2941636727 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa4c801e040 con 0x55fab2f2eae0 2022-09-07T21:40:14.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.151+0000 7fa4af7fe700 1 -- 172.21.15.47:0/2941636727 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 24) v1 ==== 7456+0+0 (secure 0 0 0) 0x7fa4c409bd80 con 0x55fab2f2eae0 2022-09-07T21:40:14.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.151+0000 7fa4ed8e1b80 10 client.25121 fetch_fsmap finished waiting for FSMap version 24 2022-09-07T21:40:14.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.151+0000 7fa4ed8e1b80 10 client.25121 resolve_mds: resolved name 'a' to daemon mds.a 2022-09-07T21:40:14.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.151+0000 7fa4ed8e1b80 20 client.25121 populate_metadata read hostname 'smithi047' 2022-09-07T21:40:14.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.151+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/2941636727 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55fab2f9a780 0x55fab2f97110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:14.171 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.151+0000 7fa4ed8e1b80 4 client.25121 mds_command: new command op to 24586 tid=0[{"prefix": "status"}] 2022-09-07T21:40:14.171 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.151+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/2941636727 --> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fab2e766f0 con 0x55fab2f9a780 2022-09-07T21:40:14.171 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.151+0000 7fa4d37fe700 1 --2- 172.21.15.47:0/2941636727 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55fab2f9a780 0x55fab2f97110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:14.171 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.152+0000 7fa4d37fe700 1 --2- 172.21.15.47:0/2941636727 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55fab2f9a780 0x55fab2f97110 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17d7794427577569 server_cookie=8de57528ef4a35f2 in_seq=0 out_seq=0 2022-09-07T21:40:14.172 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.152+0000 7fa4af7fe700 10 client.25121 ms_handle_connect on v2:172.21.15.47:6840/793863162 2022-09-07T21:40:14.172 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.152+0000 7fa4af7fe700 1 -- 172.21.15.47:0/2941636727 <== mds.0 v2:172.21.15.47:6840/793863162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55fab2e766f0 con 0x55fab2f9a780 2022-09-07T21:40:14.172 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.152+0000 7fa4af7fe700 10 client.25121 handle_command_reply: tid=0 2022-09-07T21:40:14.172 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.152+0000 7fa4ed8e1b80 1 client.25121 shutdown 2022-09-07T21:40:14.172 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.152+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/2941636727 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f2eae0 msgr2=0x55fab2f3c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:14.173 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.152+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/2941636727 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f2eae0 0x55fab2f3c9b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4005fd0 tx=0x7fa4c4091f10 comp rx=0 tx=0).stop 2022-09-07T21:40:14.173 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.152+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/2941636727 shutdown_connections 2022-09-07T21:40:14.173 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.152+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/2941636727 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x55fab2f9a780 0x55fab2f97110 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.173 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.152+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/2941636727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55fab2f301f0 0x55fab2f3d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.174 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.152+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/2941636727 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fab2f2eae0 0x55fab2f3c9b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.174 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.152+0000 7fa4ed8e1b80 1 --2- 172.21.15.47:0/2941636727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55fab2f2f430 0x55fab2f3cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.174 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.152+0000 7fa4af7fe700 0 client.25121 ms_handle_reset on v2:172.21.15.47:6840/793863162 2022-09-07T21:40:14.174 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.152+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/2941636727 >> 172.21.15.47:0/2941636727 conn(0x55fab2f2afa0 msgr2=0x55fab2f404a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:14.174 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.153+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/2941636727 shutdown_connections 2022-09-07T21:40:14.175 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.153+0000 7fa4ed8e1b80 1 -- 172.21.15.47:0/2941636727 wait complete. 2022-09-07T21:40:14.175 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.153+0000 7fa4ed8e1b80 20 client.25121 trim_cache size 0 max 16384 2022-09-07T21:40:14.175 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2022-09-07T21:40:14.176 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "01b8396e-2ef4-11ed-8431-001a4aab830c", 2022-09-07T21:40:14.176 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 0, 2022-09-07T21:40:14.176 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 24586, 2022-09-07T21:40:14.176 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2022-09-07T21:40:14.176 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2022-09-07T21:40:14.177 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2022-09-07T21:40:14.177 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 93.604648019999999, 2022-09-07T21:40:14.177 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 24, 2022-09-07T21:40:14.177 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 91, 2022-09-07T21:40:14.177 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 89, 2022-09-07T21:40:14.178 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 101.644223504 2022-09-07T21:40:14.178 INFO:teuthology.orchestra.run.smithi047.stdout:} 2022-09-07T21:40:14.179 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.153+0000 7fa4da3ff700 1 -- 172.21.15.47:0/20883469 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4bc0990d0 msgr2=0x7fa4bc09b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:14.179 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.153+0000 7fa4da3ff700 1 --2- 172.21.15.47:0/20883469 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4bc0990d0 0x7fa4bc09b580 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x55fab2f2e880 tx=0x7fa4cc009410 comp rx=0 tx=0).stop 2022-09-07T21:40:14.179 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.153+0000 7fa4da3ff700 1 -- 172.21.15.47:0/20883469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d4066be0 msgr2=0x7fa4d4195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.153+0000 7fa4da3ff700 1 --2- 172.21.15.47:0/20883469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d4066be0 0x7fa4d4195bd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4066990 tx=0x7fa4c400e9a0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.154+0000 7fa4da3ff700 1 -- 172.21.15.47:0/20883469 shutdown_connections 2022-09-07T21:40:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.154+0000 7fa4da3ff700 1 --2- 172.21.15.47:0/20883469 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fa4bc0990d0 0x7fa4bc09b580 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.154+0000 7fa4da3ff700 1 --2- 172.21.15.47:0/20883469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4d4103990 0x7fa4d419d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.154+0000 7fa4da3ff700 1 --2- 172.21.15.47:0/20883469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d4066be0 0x7fa4d4195bd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.181 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.154+0000 7fa4da3ff700 1 --2- 172.21.15.47:0/20883469 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4d4100ea0 0x7fa4d4196110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:14.181 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.154+0000 7fa4da3ff700 1 -- 172.21.15.47:0/20883469 >> 172.21.15.47:0/20883469 conn(0x7fa4d40f8370 msgr2=0x7fa4d40fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:14.181 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.154+0000 7fa4da3ff700 1 -- 172.21.15.47:0/20883469 shutdown_connections 2022-09-07T21:40:14.181 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:14.154+0000 7fa4da3ff700 1 -- 172.21.15.47:0/20883469 wait complete. 2022-09-07T21:40:14.183 INFO:tasks.cephadm:Setup complete, yielding 2022-09-07T21:40:14.183 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-09-07T21:40:14.194 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi047.front.sepia.ceph.com 2022-09-07T21:40:14.195 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- bash -c 'ceph orch status' 2022-09-07T21:40:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:13 smithi055 ceph-mon[97033]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:13 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/848731489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:40:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:13 smithi059 ceph-mon[97450]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:13 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/848731489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:40:15.078 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:40:16.139 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:15 smithi047 ceph-mon[90087]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-07T21:40:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:15 smithi055 ceph-mon[97033]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-07T21:40:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:15 smithi059 ceph-mon[97450]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-07T21:40:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.780+0000 7f41623f7700 1 -- 172.21.15.47:0/1935558043 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415c101070 msgr2=0x7f415c1014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.781+0000 7f41623f7700 1 --2- 172.21.15.47:0/1935558043 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415c101070 0x7f415c1014e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f414c009810 tx=0x7f414c009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.781+0000 7f41623f7700 1 -- 172.21.15.47:0/1935558043 shutdown_connections 2022-09-07T21:40:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.781+0000 7f41623f7700 1 --2- 172.21.15.47:0/1935558043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f415c101a20 0x7f415c108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.781+0000 7f41623f7700 1 --2- 172.21.15.47:0/1935558043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f415c0ffe90 0x7f415c1002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.781+0000 7f41623f7700 1 --2- 172.21.15.47:0/1935558043 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415c101070 0x7f415c1014e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:16.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.781+0000 7f41623f7700 1 -- 172.21.15.47:0/1935558043 >> 172.21.15.47:0/1935558043 conn(0x7f415c0fb6b0 msgr2=0x7f415c0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:16.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.781+0000 7f41623f7700 1 -- 172.21.15.47:0/1935558043 shutdown_connections 2022-09-07T21:40:16.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.781+0000 7f41623f7700 1 -- 172.21.15.47:0/1935558043 wait complete. 2022-09-07T21:40:16.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.782+0000 7f41623f7700 1 Processor -- start 2022-09-07T21:40:16.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.782+0000 7f41623f7700 1 -- start start 2022-09-07T21:40:16.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.783+0000 7f41623f7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415c0ffe90 0x7f415c1959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:16.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.783+0000 7f41623f7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f415c101070 0x7f415c195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:16.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.783+0000 7f41623f7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f415c101a20 0x7f415c19cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:16.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.783+0000 7f41623f7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c19d560 con 0x7f415c0ffe90 2022-09-07T21:40:16.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.783+0000 7f41623f7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c19d6d0 con 0x7f415c101a20 2022-09-07T21:40:16.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.783+0000 7f41623f7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c19d890 con 0x7f415c101070 2022-09-07T21:40:16.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.783+0000 7f4160bf4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f415c101070 0x7f415c195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:16.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.783+0000 7f4160bf4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f415c101070 0x7f415c195f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:42444/0 (socket says 172.21.15.47:42444) 2022-09-07T21:40:16.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.783+0000 7f4160bf4700 1 -- 172.21.15.47:0/851572695 learned_addr learned my addr 172.21.15.47:0/851572695 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:16.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.783+0000 7f4161bf6700 1 --2- 172.21.15.47:0/851572695 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f415c101a20 0x7f415c19cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:16.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.783+0000 7f41613f5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415c0ffe90 0x7f415c1959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:16.792 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.784+0000 7f4160bf4700 1 -- 172.21.15.47:0/851572695 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f415c101a20 msgr2=0x7f415c19cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:16.792 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.784+0000 7f4160bf4700 1 --2- 172.21.15.47:0/851572695 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f415c101a20 0x7f415c19cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:16.792 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.784+0000 7f4160bf4700 1 -- 172.21.15.47:0/851572695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415c0ffe90 msgr2=0x7f415c1959e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:16.792 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.784+0000 7f4160bf4700 1 --2- 172.21.15.47:0/851572695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415c0ffe90 0x7f415c1959e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:16.793 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.784+0000 7f4160bf4700 1 -- 172.21.15.47:0/851572695 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414c009470 con 0x7f415c101070 2022-09-07T21:40:16.793 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.784+0000 7f41613f5700 1 --2- 172.21.15.47:0/851572695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415c0ffe90 0x7f415c1959e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:40:16.793 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.784+0000 7f4160bf4700 1 --2- 172.21.15.47:0/851572695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f415c101070 0x7f415c195f20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f414c018660 tx=0x7f414c018690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:16.793 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.784+0000 7f41527fc700 1 -- 172.21.15.47:0/851572695 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c01d070 con 0x7f415c101070 2022-09-07T21:40:16.793 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.784+0000 7f41527fc700 1 -- 172.21.15.47:0/851572695 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f414c0054f0 con 0x7f415c101070 2022-09-07T21:40:16.794 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.784+0000 7f41623f7700 1 -- 172.21.15.47:0/851572695 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415c19db10 con 0x7f415c101070 2022-09-07T21:40:16.794 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.784+0000 7f41527fc700 1 -- 172.21.15.47:0/851572695 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c004900 con 0x7f415c101070 2022-09-07T21:40:16.794 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.785+0000 7f41623f7700 1 -- 172.21.15.47:0/851572695 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415c19e010 con 0x7f415c101070 2022-09-07T21:40:16.794 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.785+0000 7f41623f7700 1 -- 172.21.15.47:0/851572695 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415c10b5f0 con 0x7f415c101070 2022-09-07T21:40:16.796 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.787+0000 7f41527fc700 1 -- 172.21.15.47:0/851572695 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 127718+0+0 (secure 0 0 0) 0x7f414c029030 con 0x7f415c101070 2022-09-07T21:40:16.796 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.789+0000 7f41527fc700 1 --2- 172.21.15.47:0/851572695 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f41480990d0 0x7f414809b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:16.797 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.789+0000 7f41527fc700 1 -- 172.21.15.47:0/851572695 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f414c0c3730 con 0x7f415c101070 2022-09-07T21:40:16.797 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.789+0000 7f41613f5700 1 --2- 172.21.15.47:0/851572695 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f41480990d0 0x7f414809b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:16.797 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.790+0000 7f41613f5700 1 --2- 172.21.15.47:0/851572695 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f41480990d0 0x7f414809b580 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4158005fd0 tx=0x7f4158005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:16.797 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.791+0000 7f41527fc700 1 -- 172.21.15.47:0/851572695 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f414c08fa70 con 0x7f415c101070 2022-09-07T21:40:16.832 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:16.830+0000 7f41527fc700 1 -- 172.21.15.47:0/851572695 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f414c00c040 con 0x7f415c101070 2022-09-07T21:40:17.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:17.054+0000 7f41623f7700 1 -- 172.21.15.47:0/851572695 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f415c05f120 con 0x7f41480990d0 2022-09-07T21:40:17.057 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:17.055+0000 7f41527fc700 1 -- 172.21.15.47:0/851572695 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f415c05f120 con 0x7f41480990d0 2022-09-07T21:40:17.058 INFO:teuthology.orchestra.run.smithi047.stdout:Backend: cephadm 2022-09-07T21:40:17.058 INFO:teuthology.orchestra.run.smithi047.stdout:Available: Yes 2022-09-07T21:40:17.058 INFO:teuthology.orchestra.run.smithi047.stdout:Paused: No 2022-09-07T21:40:17.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:17.057+0000 7f41623f7700 1 -- 172.21.15.47:0/851572695 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f41480990d0 msgr2=0x7f414809b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:17.060 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:17.057+0000 7f41623f7700 1 --2- 172.21.15.47:0/851572695 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f41480990d0 0x7f414809b580 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4158005fd0 tx=0x7f4158005ea0 comp rx=0 tx=0).stop 2022-09-07T21:40:17.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:17.057+0000 7f41623f7700 1 -- 172.21.15.47:0/851572695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f415c101070 msgr2=0x7f415c195f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:17.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:17.057+0000 7f41623f7700 1 --2- 172.21.15.47:0/851572695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f415c101070 0x7f415c195f20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f414c018660 tx=0x7f414c018690 comp rx=0 tx=0).stop 2022-09-07T21:40:17.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:17.057+0000 7f41623f7700 1 -- 172.21.15.47:0/851572695 shutdown_connections 2022-09-07T21:40:17.061 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:17.057+0000 7f41623f7700 1 --2- 172.21.15.47:0/851572695 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f41480990d0 0x7f414809b580 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:17.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:17.057+0000 7f41623f7700 1 --2- 172.21.15.47:0/851572695 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f415c101a20 0x7f415c19cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:17.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:17.057+0000 7f41623f7700 1 --2- 172.21.15.47:0/851572695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415c0ffe90 0x7f415c1959e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:17.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:17.057+0000 7f41623f7700 1 --2- 172.21.15.47:0/851572695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f415c101070 0x7f415c195f20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:17.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:17.057+0000 7f41623f7700 1 -- 172.21.15.47:0/851572695 >> 172.21.15.47:0/851572695 conn(0x7f415c0fb6b0 msgr2=0x7f415c1045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:17.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:17.058+0000 7f41623f7700 1 -- 172.21.15.47:0/851572695 shutdown_connections 2022-09-07T21:40:17.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:17.058+0000 7f41623f7700 1 -- 172.21.15.47:0/851572695 wait complete. 2022-09-07T21:40:17.556 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- bash -c 'ceph orch ps' 2022-09-07T21:40:18.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:17 smithi047 ceph-mon[90087]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:18.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:17 smithi047 ceph-mon[90087]: mgrmap e18: x(active, since 9m), standbys: y, z 2022-09-07T21:40:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:17 smithi055 ceph-mon[97033]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:17 smithi055 ceph-mon[97033]: mgrmap e18: x(active, since 9m), standbys: y, z 2022-09-07T21:40:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:17 smithi059 ceph-mon[97450]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:17 smithi059 ceph-mon[97450]: mgrmap e18: x(active, since 9m), standbys: y, z 2022-09-07T21:40:18.446 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:40:19.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:18 smithi047 ceph-mon[90087]: from='client.25156 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:19.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:18 smithi047 ceph-mon[90087]: pgmap v367: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2022-09-07T21:40:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:18 smithi055 ceph-mon[97033]: from='client.25156 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:18 smithi055 ceph-mon[97033]: pgmap v367: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2022-09-07T21:40:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:18 smithi059 ceph-mon[97450]: from='client.25156 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:18 smithi059 ceph-mon[97450]: pgmap v367: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2022-09-07T21:40:20.224 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.220+0000 7f70f33d4700 1 -- 172.21.15.47:0/4057357992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70ec0673d0 msgr2=0x7f70ec100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:20.224 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.220+0000 7f70f33d4700 1 --2- 172.21.15.47:0/4057357992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70ec0673d0 0x7f70ec100a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f70e8009810 tx=0x7f70e8009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:20.225 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.221+0000 7f70f33d4700 1 -- 172.21.15.47:0/4057357992 shutdown_connections 2022-09-07T21:40:20.225 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.221+0000 7f70f33d4700 1 --2- 172.21.15.47:0/4057357992 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70ec103a30 0x7f70ec105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:20.225 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.221+0000 7f70f33d4700 1 --2- 172.21.15.47:0/4057357992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70ec0673d0 0x7f70ec100a00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:20.225 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.221+0000 7f70f33d4700 1 --2- 172.21.15.47:0/4057357992 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70ec100f40 0x7f70ec1033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:20.226 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.221+0000 7f70f33d4700 1 -- 172.21.15.47:0/4057357992 >> 172.21.15.47:0/4057357992 conn(0x7f70ec0f8430 msgr2=0x7f70ec0fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:20.226 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.221+0000 7f70f33d4700 1 -- 172.21.15.47:0/4057357992 shutdown_connections 2022-09-07T21:40:20.226 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.221+0000 7f70f33d4700 1 -- 172.21.15.47:0/4057357992 wait complete. 2022-09-07T21:40:20.228 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.222+0000 7f70f33d4700 1 Processor -- start 2022-09-07T21:40:20.228 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.222+0000 7f70f33d4700 1 -- start start 2022-09-07T21:40:20.229 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.223+0000 7f70f33d4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70ec0673d0 0x7f70ec195c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:20.229 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.223+0000 7f70f33d4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70ec100f40 0x7f70ec196180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:20.229 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.223+0000 7f70f33d4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70ec103a30 0x7f70ec19d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:20.229 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.223+0000 7f70f33d4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec19d800 con 0x7f70ec100f40 2022-09-07T21:40:20.229 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.223+0000 7f70f33d4700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec19d970 con 0x7f70ec0673d0 2022-09-07T21:40:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.223+0000 7f70f33d4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec19db30 con 0x7f70ec103a30 2022-09-07T21:40:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.223+0000 7f70f096f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70ec100f40 0x7f70ec196180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.223+0000 7f70f1170700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70ec0673d0 0x7f70ec195c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.223+0000 7f70f1170700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70ec0673d0 0x7f70ec195c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:50062/0 (socket says 172.21.15.47:50062) 2022-09-07T21:40:20.231 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.223+0000 7f70f1170700 1 -- 172.21.15.47:0/3685227747 learned_addr learned my addr 172.21.15.47:0/3685227747 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:20.231 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.223+0000 7f70f1971700 1 --2- 172.21.15.47:0/3685227747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70ec103a30 0x7f70ec19d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:20.231 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.224+0000 7f70f1170700 1 -- 172.21.15.47:0/3685227747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70ec103a30 msgr2=0x7f70ec19d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:20.231 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.224+0000 7f70f1170700 1 --2- 172.21.15.47:0/3685227747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70ec103a30 0x7f70ec19d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:20.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.224+0000 7f70f1170700 1 -- 172.21.15.47:0/3685227747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70ec100f40 msgr2=0x7f70ec196180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:20.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.224+0000 7f70f1170700 1 --2- 172.21.15.47:0/3685227747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70ec100f40 0x7f70ec196180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:20.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.224+0000 7f70f1170700 1 -- 172.21.15.47:0/3685227747 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc0093b0 con 0x7f70ec0673d0 2022-09-07T21:40:20.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.224+0000 7f70f096f700 1 --2- 172.21.15.47:0/3685227747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70ec100f40 0x7f70ec196180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:20.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.224+0000 7f70f1170700 1 --2- 172.21.15.47:0/3685227747 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70ec0673d0 0x7f70ec195c40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f70e8009760 tx=0x7f70e800f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:20.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.224+0000 7f70e27fc700 1 -- 172.21.15.47:0/3685227747 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e801d070 con 0x7f70ec0673d0 2022-09-07T21:40:20.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.224+0000 7f70f33d4700 1 -- 172.21.15.47:0/3685227747 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e4009730 con 0x7f70ec0673d0 2022-09-07T21:40:20.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.224+0000 7f70e27fc700 1 -- 172.21.15.47:0/3685227747 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f70e800f780 con 0x7f70ec0673d0 2022-09-07T21:40:20.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.224+0000 7f70e27fc700 1 -- 172.21.15.47:0/3685227747 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8004760 con 0x7f70ec0673d0 2022-09-07T21:40:20.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.225+0000 7f70f33d4700 1 -- 172.21.15.47:0/3685227747 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e8009470 con 0x7f70ec0673d0 2022-09-07T21:40:20.234 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.225+0000 7f70f33d4700 1 -- 172.21.15.47:0/3685227747 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70ec1095b0 con 0x7f70ec0673d0 2022-09-07T21:40:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.227+0000 7f70e27fc700 1 -- 172.21.15.47:0/3685227747 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f70e8028030 con 0x7f70ec0673d0 2022-09-07T21:40:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.229+0000 7f70e27fc700 1 --2- 172.21.15.47:0/3685227747 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f70d8099010 0x7f70d809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.229+0000 7f70e27fc700 1 -- 172.21.15.47:0/3685227747 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f70e80c37b0 con 0x7f70ec0673d0 2022-09-07T21:40:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.229+0000 7f70f096f700 1 --2- 172.21.15.47:0/3685227747 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f70d8099010 0x7f70d809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.230+0000 7f70f096f700 1 --2- 172.21.15.47:0/3685227747 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f70d8099010 0x7f70d809b4c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f70dc005fd0 tx=0x7f70dc005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.231+0000 7f70e27fc700 1 -- 172.21.15.47:0/3685227747 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f70e808fb30 con 0x7f70ec0673d0 2022-09-07T21:40:20.469 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.466+0000 7f70f33d4700 1 -- 172.21.15.47:0/3685227747 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f70ec05f120 con 0x7f70d8099010 2022-09-07T21:40:20.482 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.480+0000 7f70e27fc700 1 -- 172.21.15.47:0/3685227747 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4216 (secure 0 0 0) 0x7f70ec05f120 con 0x7f70d8099010 2022-09-07T21:40:20.482 INFO:teuthology.orchestra.run.smithi047.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-09-07T21:40:20.483 INFO:teuthology.orchestra.run.smithi047.stdout:mds.a smithi047 running (108s) 19s ago 107s 30.1M - 17.0.0-14857-g15219419 6405dbf5acb7 6f5e665bd681 2022-09-07T21:40:20.483 INFO:teuthology.orchestra.run.smithi047.stdout:mds.b smithi055 running (105s) 32s ago 105s 21.2M - 17.0.0-14857-g15219419 6405dbf5acb7 d1bd93256a1d 2022-09-07T21:40:20.483 INFO:teuthology.orchestra.run.smithi047.stdout:mds.c smithi059 running (2m) 32s ago 2m 17.6M - 17.0.0-14857-g15219419 6405dbf5acb7 a66993f2186d 2022-09-07T21:40:20.483 INFO:teuthology.orchestra.run.smithi047.stdout:mds.d smithi047 running (2m) 19s ago 2m 13.8M - 17.0.0-14857-g15219419 6405dbf5acb7 a3eb588e6836 2022-09-07T21:40:20.484 INFO:teuthology.orchestra.run.smithi047.stdout:mds.e smithi055 running (2m) 32s ago 2m 16.5M - 17.0.0-14857-g15219419 6405dbf5acb7 f5aa23b80a9b 2022-09-07T21:40:20.484 INFO:teuthology.orchestra.run.smithi047.stdout:mds.f smithi059 running (2m) 32s ago 2m 18.8M - 17.0.0-14857-g15219419 6405dbf5acb7 ddb264a795b4 2022-09-07T21:40:20.484 INFO:teuthology.orchestra.run.smithi047.stdout:mds.g smithi047 running (112s) 19s ago 111s 17.6M - 17.0.0-14857-g15219419 6405dbf5acb7 3eb8ecb25111 2022-09-07T21:40:20.484 INFO:teuthology.orchestra.run.smithi047.stdout:mds.h smithi055 running (118s) 32s ago 117s 17.0M - 17.0.0-14857-g15219419 6405dbf5acb7 16db67a06299 2022-09-07T21:40:20.485 INFO:teuthology.orchestra.run.smithi047.stdout:mds.i smithi059 running (116s) 32s ago 115s 21.2M - 17.0.0-14857-g15219419 6405dbf5acb7 85e22203902f 2022-09-07T21:40:20.485 INFO:teuthology.orchestra.run.smithi047.stdout:mds.j smithi047 running (2m) 19s ago 2m 13.7M - 17.0.0-14857-g15219419 6405dbf5acb7 9e083b706bff 2022-09-07T21:40:20.485 INFO:teuthology.orchestra.run.smithi047.stdout:mds.k smithi055 running (2m) 32s ago 2m 14.9M - 17.0.0-14857-g15219419 6405dbf5acb7 0a5b12806a98 2022-09-07T21:40:20.485 INFO:teuthology.orchestra.run.smithi047.stdout:mds.l smithi059 running (2m) 32s ago 2m 14.8M - 17.0.0-14857-g15219419 6405dbf5acb7 33afe7eba40a 2022-09-07T21:40:20.485 INFO:teuthology.orchestra.run.smithi047.stdout:mgr.x smithi047 *:9283,8765 running (10m) 19s ago 10m 502M - 17.0.0-14857-g15219419 6405dbf5acb7 fe9cb52dcc9e 2022-09-07T21:40:20.486 INFO:teuthology.orchestra.run.smithi047.stdout:mgr.y smithi055 *:8443 running (8m) 32s ago 8m 429M - 17.0.0-14857-g15219419 6405dbf5acb7 4a34efdf7423 2022-09-07T21:40:20.486 INFO:teuthology.orchestra.run.smithi047.stdout:mgr.z smithi059 *:8443 running (8m) 32s ago 8m 431M - 17.0.0-14857-g15219419 6405dbf5acb7 85ad55a4092c 2022-09-07T21:40:20.486 INFO:teuthology.orchestra.run.smithi047.stdout:mon.a smithi047 running (10m) 19s ago 10m 100M 2048M 17.0.0-14857-g15219419 6405dbf5acb7 206ccff10612 2022-09-07T21:40:20.486 INFO:teuthology.orchestra.run.smithi047.stdout:mon.b smithi055 running (8m) 32s ago 8m 67.1M 2048M 17.0.0-14857-g15219419 6405dbf5acb7 0f3ba03b8a18 2022-09-07T21:40:20.487 INFO:teuthology.orchestra.run.smithi047.stdout:mon.c smithi059 running (8m) 32s ago 8m 69.0M 2048M 17.0.0-14857-g15219419 6405dbf5acb7 b7716b4d5dad 2022-09-07T21:40:20.487 INFO:teuthology.orchestra.run.smithi047.stdout:osd.0 smithi047 running (7m) 19s ago 7m 71.0M 4319M 17.0.0-14857-g15219419 6405dbf5acb7 5cd176885012 2022-09-07T21:40:20.487 INFO:teuthology.orchestra.run.smithi047.stdout:osd.1 smithi055 running (7m) 32s ago 7m 74.8M 4293M 17.0.0-14857-g15219419 6405dbf5acb7 e54797203c6c 2022-09-07T21:40:20.487 INFO:teuthology.orchestra.run.smithi047.stdout:osd.2 smithi059 running (6m) 32s ago 6m 71.3M 4293M 17.0.0-14857-g15219419 6405dbf5acb7 1a2d37b14416 2022-09-07T21:40:20.488 INFO:teuthology.orchestra.run.smithi047.stdout:osd.3 smithi047 running (6m) 19s ago 6m 63.6M 4319M 17.0.0-14857-g15219419 6405dbf5acb7 19107de0050c 2022-09-07T21:40:20.488 INFO:teuthology.orchestra.run.smithi047.stdout:osd.4 smithi055 running (5m) 32s ago 5m 60.7M 4293M 17.0.0-14857-g15219419 6405dbf5acb7 b4e4ee3f9936 2022-09-07T21:40:20.488 INFO:teuthology.orchestra.run.smithi047.stdout:osd.5 smithi059 running (5m) 32s ago 5m 66.9M 4293M 17.0.0-14857-g15219419 6405dbf5acb7 f3a38b4fd8dc 2022-09-07T21:40:20.488 INFO:teuthology.orchestra.run.smithi047.stdout:osd.6 smithi047 running (4m) 19s ago 4m 62.0M 4319M 17.0.0-14857-g15219419 6405dbf5acb7 f47b8aa5f72b 2022-09-07T21:40:20.488 INFO:teuthology.orchestra.run.smithi047.stdout:osd.7 smithi055 running (4m) 32s ago 4m 63.2M 4293M 17.0.0-14857-g15219419 6405dbf5acb7 b02e7be5c190 2022-09-07T21:40:20.489 INFO:teuthology.orchestra.run.smithi047.stdout:osd.8 smithi059 running (4m) 32s ago 4m 59.8M 4293M 17.0.0-14857-g15219419 6405dbf5acb7 1d0e465b8706 2022-09-07T21:40:20.489 INFO:teuthology.orchestra.run.smithi047.stdout:osd.9 smithi047 running (3m) 19s ago 3m 58.3M 4319M 17.0.0-14857-g15219419 6405dbf5acb7 4af5e8793171 2022-09-07T21:40:20.489 INFO:teuthology.orchestra.run.smithi047.stdout:osd.10 smithi055 running (3m) 32s ago 3m 61.2M 4293M 17.0.0-14857-g15219419 6405dbf5acb7 aa761c776368 2022-09-07T21:40:20.489 INFO:teuthology.orchestra.run.smithi047.stdout:osd.11 smithi059 running (2m) 32s ago 2m 52.8M 4293M 17.0.0-14857-g15219419 6405dbf5acb7 9e8cf7dd4e6f 2022-09-07T21:40:20.490 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.482+0000 7f70f33d4700 1 -- 172.21.15.47:0/3685227747 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f70d8099010 msgr2=0x7f70d809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:20.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.482+0000 7f70f33d4700 1 --2- 172.21.15.47:0/3685227747 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f70d8099010 0x7f70d809b4c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f70dc005fd0 tx=0x7f70dc005ea0 comp rx=0 tx=0).stop 2022-09-07T21:40:20.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.482+0000 7f70f33d4700 1 -- 172.21.15.47:0/3685227747 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70ec0673d0 msgr2=0x7f70ec195c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:20.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.482+0000 7f70f33d4700 1 --2- 172.21.15.47:0/3685227747 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70ec0673d0 0x7f70ec195c40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f70e8009760 tx=0x7f70e800f690 comp rx=0 tx=0).stop 2022-09-07T21:40:20.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.482+0000 7f70f33d4700 1 -- 172.21.15.47:0/3685227747 shutdown_connections 2022-09-07T21:40:20.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.482+0000 7f70f33d4700 1 --2- 172.21.15.47:0/3685227747 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f70d8099010 0x7f70d809b4c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:20.492 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.482+0000 7f70f33d4700 1 --2- 172.21.15.47:0/3685227747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70ec103a30 0x7f70ec19d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:20.492 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.482+0000 7f70f33d4700 1 --2- 172.21.15.47:0/3685227747 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70ec0673d0 0x7f70ec195c40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:20.492 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.482+0000 7f70f33d4700 1 --2- 172.21.15.47:0/3685227747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70ec100f40 0x7f70ec196180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:20.492 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.482+0000 7f70f33d4700 1 -- 172.21.15.47:0/3685227747 >> 172.21.15.47:0/3685227747 conn(0x7f70ec0f8430 msgr2=0x7f70ec0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:20.493 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.482+0000 7f70f33d4700 1 -- 172.21.15.47:0/3685227747 shutdown_connections 2022-09-07T21:40:20.493 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:20.483+0000 7f70f33d4700 1 -- 172.21.15.47:0/3685227747 wait complete. 2022-09-07T21:40:21.026 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- bash -c 'ceph orch ls' 2022-09-07T21:40:21.911 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:40:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:21 smithi055 ceph-mon[97033]: from='client.25133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:21 smithi055 ceph-mon[97033]: pgmap v368: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2022-09-07T21:40:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:21 smithi059 ceph-mon[97450]: from='client.25133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:21 smithi059 ceph-mon[97450]: pgmap v368: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2022-09-07T21:40:22.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:21 smithi047 ceph-mon[90087]: from='client.25133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:22.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:21 smithi047 ceph-mon[90087]: pgmap v368: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2022-09-07T21:40:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:23 smithi055 ceph-mon[97033]: pgmap v369: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2022-09-07T21:40:24.001 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:23.998+0000 7fbfe6092700 1 -- 172.21.15.47:0/3396325253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfe0101300 msgr2=0x7fbfe0101750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:24.001 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:23.998+0000 7fbfe6092700 1 --2- 172.21.15.47:0/3396325253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfe0101300 0x7fbfe0101750 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009750 tx=0x7fbfd4009a60 comp rx=0 tx=0).stop 2022-09-07T21:40:24.001 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:23.998+0000 7fbfe6092700 1 -- 172.21.15.47:0/3396325253 shutdown_connections 2022-09-07T21:40:24.002 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:23.998+0000 7fbfe6092700 1 --2- 172.21.15.47:0/3396325253 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfe0101c90 0x7fbfe0108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:24.002 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:23.998+0000 7fbfe6092700 1 --2- 172.21.15.47:0/3396325253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe0100080 0x7fbfe0100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:24.002 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:23.998+0000 7fbfe6092700 1 --2- 172.21.15.47:0/3396325253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfe0101300 0x7fbfe0101750 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:24.002 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:23.998+0000 7fbfe6092700 1 -- 172.21.15.47:0/3396325253 >> 172.21.15.47:0/3396325253 conn(0x7fbfe00fb890 msgr2=0x7fbfe00fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:24.003 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:23.998+0000 7fbfe6092700 1 -- 172.21.15.47:0/3396325253 shutdown_connections 2022-09-07T21:40:24.003 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:23.998+0000 7fbfe6092700 1 -- 172.21.15.47:0/3396325253 wait complete. 2022-09-07T21:40:24.005 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:23.999+0000 7fbfe6092700 1 Processor -- start 2022-09-07T21:40:24.005 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:23.999+0000 7fbfe6092700 1 -- start start 2022-09-07T21:40:24.006 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.000+0000 7fbfe6092700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe0100080 0x7fbfe0195bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:24.006 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.000+0000 7fbfe6092700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfe0101300 0x7fbfe01960f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:24.006 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.000+0000 7fbfe6092700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfe0101c90 0x7fbfe019d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:24.007 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.000+0000 7fbfe6092700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe019d730 con 0x7fbfe0101c90 2022-09-07T21:40:24.007 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.000+0000 7fbfe6092700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe019d8a0 con 0x7fbfe0101300 2022-09-07T21:40:24.007 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.000+0000 7fbfe6092700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe019da60 con 0x7fbfe0100080 2022-09-07T21:40:24.007 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.000+0000 7fbfdffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfe0101c90 0x7fbfe019d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:24.007 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.000+0000 7fbfdffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfe0101c90 0x7fbfe019d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55532/0 (socket says 172.21.15.47:55532) 2022-09-07T21:40:24.008 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.000+0000 7fbfdffff700 1 -- 172.21.15.47:0/564633040 learned_addr learned my addr 172.21.15.47:0/564633040 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:24.008 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.000+0000 7fbfdeffd700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfe0101300 0x7fbfe01960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:24.008 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.000+0000 7fbfdf7fe700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe0100080 0x7fbfe0195bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:24.008 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.001+0000 7fbfdf7fe700 1 -- 172.21.15.47:0/564633040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfe0101300 msgr2=0x7fbfe01960f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:24.008 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.001+0000 7fbfdf7fe700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfe0101300 0x7fbfe01960f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:24.009 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.001+0000 7fbfdf7fe700 1 -- 172.21.15.47:0/564633040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfe0101c90 msgr2=0x7fbfe019d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:24.009 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.001+0000 7fbfdf7fe700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfe0101c90 0x7fbfe019d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:24.009 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.001+0000 7fbfdf7fe700 1 -- 172.21.15.47:0/564633040 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd40093b0 con 0x7fbfe0100080 2022-09-07T21:40:24.009 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.001+0000 7fbfdffff700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfe0101c90 0x7fbfe019d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:24.010 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.001+0000 7fbfdf7fe700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe0100080 0x7fbfe0195bb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0005fd0 tx=0x7fbfd000e9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:24.010 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.001+0000 7fbfdcff9700 1 -- 172.21.15.47:0/564633040 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0004220 con 0x7fbfe0100080 2022-09-07T21:40:24.010 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.001+0000 7fbfdcff9700 1 -- 172.21.15.47:0/564633040 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fbfd0004380 con 0x7fbfe0100080 2022-09-07T21:40:24.010 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.001+0000 7fbfdcff9700 1 -- 172.21.15.47:0/564633040 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0003c70 con 0x7fbfe0100080 2022-09-07T21:40:24.010 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.001+0000 7fbfe6092700 1 -- 172.21.15.47:0/564633040 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe019dd40 con 0x7fbfe0100080 2022-09-07T21:40:24.011 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.001+0000 7fbfe6092700 1 -- 172.21.15.47:0/564633040 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe019e1a0 con 0x7fbfe0100080 2022-09-07T21:40:24.011 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.002+0000 7fbfe6092700 1 -- 172.21.15.47:0/564633040 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe010b860 con 0x7fbfe0100080 2022-09-07T21:40:24.012 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.005+0000 7fbfdcff9700 1 -- 172.21.15.47:0/564633040 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fbfd001e020 con 0x7fbfe0100080 2022-09-07T21:40:24.012 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.006+0000 7fbfdcff9700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fbfc8099090 0x7fbfc809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:24.013 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.006+0000 7fbfdcff9700 1 -- 172.21.15.47:0/564633040 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fbfd0014070 con 0x7fbfe0100080 2022-09-07T21:40:24.013 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.006+0000 7fbfdeffd700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fbfc8099090 0x7fbfc809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:24.013 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.007+0000 7fbfdeffd700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fbfc8099090 0x7fbfc809b540 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400f500 tx=0x7fbfd4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:24.013 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.008+0000 7fbfdcff9700 1 -- 172.21.15.47:0/564633040 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fbfd008ef80 con 0x7fbfe0100080 2022-09-07T21:40:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:23 smithi059 ceph-mon[97450]: pgmap v369: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2022-09-07T21:40:24.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:23 smithi047 ceph-mon[90087]: pgmap v369: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2022-09-07T21:40:24.258 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.256+0000 7fbfe6092700 1 -- 172.21.15.47:0/564633040 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbfe005fc50 con 0x7fbfc8099090 2022-09-07T21:40:24.264 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.262+0000 7fbfdcff9700 1 -- 172.21.15.47:0/564633040 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7fbfe005fc50 con 0x7fbfc8099090 2022-09-07T21:40:24.265 INFO:teuthology.orchestra.run.smithi047.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-09-07T21:40:24.265 INFO:teuthology.orchestra.run.smithi047.stdout:mds.all 12/12 36s ago 2m smithi047=a;smithi047=d;smithi047=g;smithi047=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi059=c;smithi059=f;smithi059=i;smithi059=l;count:12 2022-09-07T21:40:24.265 INFO:teuthology.orchestra.run.smithi047.stdout:mgr 3/3 36s ago 8m smithi047=x;smithi055=y;smithi059=z;count:3 2022-09-07T21:40:24.265 INFO:teuthology.orchestra.run.smithi047.stdout:mon 3/3 36s ago 8m smithi047:172.21.15.47=a;smithi055:172.21.15.55=b;smithi059:172.21.15.59=c;count:3 2022-09-07T21:40:24.266 INFO:teuthology.orchestra.run.smithi047.stdout:osd 12 36s ago - 2022-09-07T21:40:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.264+0000 7fbfe6092700 1 -- 172.21.15.47:0/564633040 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fbfc8099090 msgr2=0x7fbfc809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.264+0000 7fbfe6092700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fbfc8099090 0x7fbfc809b540 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400f500 tx=0x7fbfd4005f90 comp rx=0 tx=0).stop 2022-09-07T21:40:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.264+0000 7fbfe6092700 1 -- 172.21.15.47:0/564633040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe0100080 msgr2=0x7fbfe0195bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.264+0000 7fbfe6092700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe0100080 0x7fbfe0195bb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0005fd0 tx=0x7fbfd000e9a0 comp rx=0 tx=0).stop 2022-09-07T21:40:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.265+0000 7fbfe6092700 1 -- 172.21.15.47:0/564633040 shutdown_connections 2022-09-07T21:40:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.265+0000 7fbfe6092700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fbfc8099090 0x7fbfc809b540 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.265+0000 7fbfe6092700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfe0101c90 0x7fbfe019d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.265+0000 7fbfe6092700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe0100080 0x7fbfe0195bb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.265+0000 7fbfe6092700 1 --2- 172.21.15.47:0/564633040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfe0101300 0x7fbfe01960f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.265+0000 7fbfe6092700 1 -- 172.21.15.47:0/564633040 >> 172.21.15.47:0/564633040 conn(0x7fbfe00fb890 msgr2=0x7fbfe00fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.265+0000 7fbfe6092700 1 -- 172.21.15.47:0/564633040 shutdown_connections 2022-09-07T21:40:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:24.265+0000 7fbfe6092700 1 -- 172.21.15.47:0/564633040 wait complete. 2022-09-07T21:40:24.779 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- bash -c 'ceph orch host ls' 2022-09-07T21:40:25.664 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:40:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:25 smithi059 ceph-mon[97450]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:25 smithi059 ceph-mon[97450]: pgmap v370: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2022-09-07T21:40:26.057 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:25 smithi047 ceph-mon[90087]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:26.057 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:25 smithi047 ceph-mon[90087]: pgmap v370: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2022-09-07T21:40:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:25 smithi055 ceph-mon[97033]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:25 smithi055 ceph-mon[97033]: pgmap v370: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2022-09-07T21:40:27.481 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.478+0000 7f1836ef7700 1 -- 172.21.15.47:0/911543278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18300feed0 msgr2=0x7f18300ff320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:27.482 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.478+0000 7f1836ef7700 1 --2- 172.21.15.47:0/911543278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18300feed0 0x7f18300ff320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1820009810 tx=0x7f1820009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:27.482 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.478+0000 7f1836ef7700 1 -- 172.21.15.47:0/911543278 shutdown_connections 2022-09-07T21:40:27.482 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.478+0000 7f1836ef7700 1 --2- 172.21.15.47:0/911543278 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18300ff860 0x7f1830106120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:27.482 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.478+0000 7f1836ef7700 1 --2- 172.21.15.47:0/911543278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18300fdc50 0x7f18300fe060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:27.483 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.478+0000 7f1836ef7700 1 --2- 172.21.15.47:0/911543278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18300feed0 0x7f18300ff320 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:27.483 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.478+0000 7f1836ef7700 1 -- 172.21.15.47:0/911543278 >> 172.21.15.47:0/911543278 conn(0x7f18300f9320 msgr2=0x7f18300fb770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:27.483 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.478+0000 7f1836ef7700 1 -- 172.21.15.47:0/911543278 shutdown_connections 2022-09-07T21:40:27.483 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.479+0000 7f1836ef7700 1 -- 172.21.15.47:0/911543278 wait complete. 2022-09-07T21:40:27.485 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.480+0000 7f1836ef7700 1 Processor -- start 2022-09-07T21:40:27.486 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.480+0000 7f1836ef7700 1 -- start start 2022-09-07T21:40:27.486 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.480+0000 7f1836ef7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18300fdc50 0x7f1830195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:27.486 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.480+0000 7f1836ef7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18300feed0 0x7f1830195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:27.486 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.480+0000 7f1836ef7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18300ff860 0x7f183019d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:27.487 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.481+0000 7f1836ef7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183019d610 con 0x7f18300fdc50 2022-09-07T21:40:27.487 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.481+0000 7f1836ef7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183019d780 con 0x7f18300feed0 2022-09-07T21:40:27.487 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.481+0000 7f1836ef7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183019d940 con 0x7f18300ff860 2022-09-07T21:40:27.487 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.481+0000 7f18356f4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18300feed0 0x7f1830195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:27.487 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.481+0000 7f18356f4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18300feed0 0x7f1830195fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:50114/0 (socket says 172.21.15.47:50114) 2022-09-07T21:40:27.488 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.481+0000 7f18356f4700 1 -- 172.21.15.47:0/3676177459 learned_addr learned my addr 172.21.15.47:0/3676177459 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:27.488 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.481+0000 7f1835ef5700 1 --2- 172.21.15.47:0/3676177459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18300fdc50 0x7f1830195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:27.488 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.481+0000 7f18366f6700 1 --2- 172.21.15.47:0/3676177459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18300ff860 0x7f183019d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:27.488 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.481+0000 7f18356f4700 1 -- 172.21.15.47:0/3676177459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18300ff860 msgr2=0x7f183019d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.481+0000 7f18356f4700 1 --2- 172.21.15.47:0/3676177459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18300ff860 0x7f183019d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.481+0000 7f18356f4700 1 -- 172.21.15.47:0/3676177459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18300fdc50 msgr2=0x7f1830195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.481+0000 7f18356f4700 1 --2- 172.21.15.47:0/3676177459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18300fdc50 0x7f1830195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.481+0000 7f18356f4700 1 -- 172.21.15.47:0/3676177459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1820009470 con 0x7f18300feed0 2022-09-07T21:40:27.490 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.482+0000 7f18356f4700 1 --2- 172.21.15.47:0/3676177459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18300feed0 0x7f1830195fd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f182000f690 tx=0x7f182000f6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:27.490 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.482+0000 7f1826ffd700 1 -- 172.21.15.47:0/3676177459 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182001d070 con 0x7f18300feed0 2022-09-07T21:40:27.490 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.482+0000 7f1836ef7700 1 -- 172.21.15.47:0/3676177459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183019dbc0 con 0x7f18300feed0 2022-09-07T21:40:27.490 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.482+0000 7f1826ffd700 1 -- 172.21.15.47:0/3676177459 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f182000f8e0 con 0x7f18300feed0 2022-09-07T21:40:27.490 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.482+0000 7f1826ffd700 1 -- 172.21.15.47:0/3676177459 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18200047f0 con 0x7f18300feed0 2022-09-07T21:40:27.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.482+0000 7f1836ef7700 1 -- 172.21.15.47:0/3676177459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183019e070 con 0x7f18300feed0 2022-09-07T21:40:27.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.483+0000 7f1836ef7700 1 -- 172.21.15.47:0/3676177459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1830109400 con 0x7f18300feed0 2022-09-07T21:40:27.492 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.485+0000 7f1826ffd700 1 -- 172.21.15.47:0/3676177459 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f1820028030 con 0x7f18300feed0 2022-09-07T21:40:27.492 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.486+0000 7f1826ffd700 1 --2- 172.21.15.47:0/3676177459 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f181c099010 0x7f181c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:27.492 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.487+0000 7f1826ffd700 1 -- 172.21.15.47:0/3676177459 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f18200c3750 con 0x7f18300feed0 2022-09-07T21:40:27.493 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.487+0000 7f1835ef5700 1 --2- 172.21.15.47:0/3676177459 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f181c099010 0x7f181c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:27.493 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.487+0000 7f1835ef5700 1 --2- 172.21.15.47:0/3676177459 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f181c099010 0x7f181c09b4c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f18300fed30 tx=0x7f182c005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:27.493 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.489+0000 7f1826ffd700 1 -- 172.21.15.47:0/3676177459 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f182008fad0 con 0x7f18300feed0 2022-09-07T21:40:27.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.725+0000 7f1836ef7700 1 -- 172.21.15.47:0/3676177459 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f183005fbb0 con 0x7f181c099010 2022-09-07T21:40:27.729 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.727+0000 7f1826ffd700 1 -- 172.21.15.47:0/3676177459 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+187 (secure 0 0 0) 0x7f183005fbb0 con 0x7f181c099010 2022-09-07T21:40:27.730 INFO:teuthology.orchestra.run.smithi047.stdout:HOST ADDR LABELS STATUS 2022-09-07T21:40:27.730 INFO:teuthology.orchestra.run.smithi047.stdout:smithi047 172.21.15.47 2022-09-07T21:40:27.730 INFO:teuthology.orchestra.run.smithi047.stdout:smithi055 172.21.15.55 2022-09-07T21:40:27.731 INFO:teuthology.orchestra.run.smithi047.stdout:smithi059 172.21.15.59 2022-09-07T21:40:27.731 INFO:teuthology.orchestra.run.smithi047.stdout:3 hosts in cluster 2022-09-07T21:40:27.732 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.730+0000 7f1836ef7700 1 -- 172.21.15.47:0/3676177459 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f181c099010 msgr2=0x7f181c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:27.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.730+0000 7f1836ef7700 1 --2- 172.21.15.47:0/3676177459 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f181c099010 0x7f181c09b4c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f18300fed30 tx=0x7f182c005ea0 comp rx=0 tx=0).stop 2022-09-07T21:40:27.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.730+0000 7f1836ef7700 1 -- 172.21.15.47:0/3676177459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18300feed0 msgr2=0x7f1830195fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:27.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.730+0000 7f1836ef7700 1 --2- 172.21.15.47:0/3676177459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18300feed0 0x7f1830195fd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f182000f690 tx=0x7f182000f6c0 comp rx=0 tx=0).stop 2022-09-07T21:40:27.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.730+0000 7f1836ef7700 1 -- 172.21.15.47:0/3676177459 shutdown_connections 2022-09-07T21:40:27.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.730+0000 7f1836ef7700 1 --2- 172.21.15.47:0/3676177459 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f181c099010 0x7f181c09b4c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:27.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.730+0000 7f1836ef7700 1 --2- 172.21.15.47:0/3676177459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18300ff860 0x7f183019d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:27.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.730+0000 7f1836ef7700 1 --2- 172.21.15.47:0/3676177459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18300fdc50 0x7f1830195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:27.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.730+0000 7f1836ef7700 1 --2- 172.21.15.47:0/3676177459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18300feed0 0x7f1830195fd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:27.735 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.730+0000 7f1836ef7700 1 -- 172.21.15.47:0/3676177459 >> 172.21.15.47:0/3676177459 conn(0x7f18300f9320 msgr2=0x7f18301048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:27.735 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.730+0000 7f1836ef7700 1 -- 172.21.15.47:0/3676177459 shutdown_connections 2022-09-07T21:40:27.735 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:27.730+0000 7f1836ef7700 1 -- 172.21.15.47:0/3676177459 wait complete. 2022-09-07T21:40:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:27 smithi059 ceph-mon[97450]: pgmap v371: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2022-09-07T21:40:28.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:27 smithi047 ceph-mon[90087]: pgmap v371: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2022-09-07T21:40:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:27 smithi055 ceph-mon[97033]: pgmap v371: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2022-09-07T21:40:28.233 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- bash -c 'ceph orch device ls' 2022-09-07T21:40:29.119 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:40:29.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:28 smithi047 ceph-mon[90087]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:28 smithi055 ceph-mon[97033]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:28 smithi059 ceph-mon[97450]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:30.146 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:29 smithi047 ceph-mon[90087]: pgmap v372: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2022-09-07T21:40:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:29 smithi055 ceph-mon[97033]: pgmap v372: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2022-09-07T21:40:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:29 smithi059 ceph-mon[97450]: pgmap v372: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2022-09-07T21:40:30.776 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.773+0000 7f98c4b1e700 1 -- 172.21.15.47:0/1625763664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98c00fde10 msgr2=0x7f98c00fe220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:30.776 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.773+0000 7f98c4b1e700 1 --2- 172.21.15.47:0/1625763664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98c00fde10 0x7f98c00fe220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f98b0009840 tx=0x7f98b0009b50 comp rx=0 tx=0).stop 2022-09-07T21:40:30.776 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.773+0000 7f98c4b1e700 1 -- 172.21.15.47:0/1625763664 shutdown_connections 2022-09-07T21:40:30.776 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.773+0000 7f98c4b1e700 1 --2- 172.21.15.47:0/1625763664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98c00ffa30 0x7f98c0106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:30.777 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.773+0000 7f98c4b1e700 1 --2- 172.21.15.47:0/1625763664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98c00fde10 0x7f98c00fe220 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:30.777 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.773+0000 7f98c4b1e700 1 --2- 172.21.15.47:0/1625763664 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98c00ff080 0x7f98c00ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:30.777 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.773+0000 7f98c4b1e700 1 -- 172.21.15.47:0/1625763664 >> 172.21.15.47:0/1625763664 conn(0x7f98c00f9570 msgr2=0x7f98c00fb980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:30.777 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.773+0000 7f98c4b1e700 1 -- 172.21.15.47:0/1625763664 shutdown_connections 2022-09-07T21:40:30.778 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.773+0000 7f98c4b1e700 1 -- 172.21.15.47:0/1625763664 wait complete. 2022-09-07T21:40:30.779 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.774+0000 7f98c4b1e700 1 Processor -- start 2022-09-07T21:40:30.780 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.774+0000 7f98c4b1e700 1 -- start start 2022-09-07T21:40:30.780 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.775+0000 7f98c4b1e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98c00fde10 0x7f98c0193a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:30.780 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.775+0000 7f98c4b1e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98c00ff080 0x7f98c0193fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:30.780 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.775+0000 7f98c4b1e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98c00ffa30 0x7f98c019b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:30.781 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.775+0000 7f98c4b1e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c019b5e0 con 0x7f98c00ff080 2022-09-07T21:40:30.781 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.775+0000 7f98c4b1e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c019b750 con 0x7f98c00fde10 2022-09-07T21:40:30.781 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.775+0000 7f98c4b1e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c019b910 con 0x7f98c00ffa30 2022-09-07T21:40:30.781 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.775+0000 7f98bed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98c00ffa30 0x7f98c019b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:30.781 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.775+0000 7f98be59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98c00fde10 0x7f98c0193a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:30.782 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.775+0000 7f98bed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98c00ffa30 0x7f98c019b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:37968/0 (socket says 172.21.15.47:37968) 2022-09-07T21:40:30.782 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.775+0000 7f98bed9d700 1 -- 172.21.15.47:0/201705633 learned_addr learned my addr 172.21.15.47:0/201705633 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:30.782 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.775+0000 7f98bdd9b700 1 --2- 172.21.15.47:0/201705633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98c00ff080 0x7f98c0193fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:30.782 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.776+0000 7f98bed9d700 1 -- 172.21.15.47:0/201705633 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98c00fde10 msgr2=0x7f98c0193a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:30.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.776+0000 7f98bed9d700 1 --2- 172.21.15.47:0/201705633 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98c00fde10 0x7f98c0193a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:30.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.776+0000 7f98bed9d700 1 -- 172.21.15.47:0/201705633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98c00ff080 msgr2=0x7f98c0193fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:30.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.776+0000 7f98bed9d700 1 --2- 172.21.15.47:0/201705633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98c00ff080 0x7f98c0193fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:30.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.776+0000 7f98bed9d700 1 -- 172.21.15.47:0/201705633 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b40093b0 con 0x7f98c00ffa30 2022-09-07T21:40:30.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.776+0000 7f98bed9d700 1 --2- 172.21.15.47:0/201705633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98c00ffa30 0x7f98c019b010 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f98b800ebf0 tx=0x7f98b800ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:30.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.776+0000 7f98af7fe700 1 -- 172.21.15.47:0/201705633 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b80099c0 con 0x7f98c00ffa30 2022-09-07T21:40:30.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.776+0000 7f98c4b1e700 1 -- 172.21.15.47:0/201705633 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b0009470 con 0x7f98c00ffa30 2022-09-07T21:40:30.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.776+0000 7f98af7fe700 1 -- 172.21.15.47:0/201705633 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f98b800cc90 con 0x7f98c00ffa30 2022-09-07T21:40:30.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.776+0000 7f98af7fe700 1 -- 172.21.15.47:0/201705633 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8010430 con 0x7f98c00ffa30 2022-09-07T21:40:30.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.777+0000 7f98c4b1e700 1 -- 172.21.15.47:0/201705633 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c019bf70 con 0x7f98c00ffa30 2022-09-07T21:40:30.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.778+0000 7f98c4b1e700 1 -- 172.21.15.47:0/201705633 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c0109630 con 0x7f98c00ffa30 2022-09-07T21:40:30.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.780+0000 7f98af7fe700 1 -- 172.21.15.47:0/201705633 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f98b801e020 con 0x7f98c00ffa30 2022-09-07T21:40:30.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.781+0000 7f98af7fe700 1 --2- 172.21.15.47:0/201705633 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f98a8099010 0x7f98a809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:30.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.781+0000 7f98af7fe700 1 -- 172.21.15.47:0/201705633 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f98b8014070 con 0x7f98c00ffa30 2022-09-07T21:40:30.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.781+0000 7f98be59c700 1 --2- 172.21.15.47:0/201705633 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f98a8099010 0x7f98a809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:30.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.782+0000 7f98be59c700 1 --2- 172.21.15.47:0/201705633 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f98a8099010 0x7f98a809b4c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f98b0009fd0 tx=0x7f98b0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:30.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:30.783+0000 7f98af7fe700 1 -- 172.21.15.47:0/201705633 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f98b808f730 con 0x7f98c00ffa30 2022-09-07T21:40:31.020 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:31.017+0000 7f98c4b1e700 1 -- 172.21.15.47:0/201705633 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f98c005f120 con 0x7f98a8099010 2022-09-07T21:40:31.022 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:31.019+0000 7f98af7fe700 1 -- 172.21.15.47:0/201705633 <== mgr.14152 v2:172.21.15.47:6800/429234829 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f98c005f120 con 0x7f98a8099010 2022-09-07T21:40:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-09-07T21:40:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:smithi047 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB75130022480DGN 480G 3m ago LVM detected, locked 2022-09-07T21:40:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:smithi055 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736102VS4P0KGN 4000G 3m ago LVM detected, locked 2022-09-07T21:40:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:smithi059 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009L400BGN 400G 2m ago LVM detected, locked 2022-09-07T21:40:31.024 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:31.022+0000 7f98c4b1e700 1 -- 172.21.15.47:0/201705633 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f98a8099010 msgr2=0x7f98a809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:31.025 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:31.022+0000 7f98c4b1e700 1 --2- 172.21.15.47:0/201705633 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f98a8099010 0x7f98a809b4c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f98b0009fd0 tx=0x7f98b0000bc0 comp rx=0 tx=0).stop 2022-09-07T21:40:31.025 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:31.022+0000 7f98c4b1e700 1 -- 172.21.15.47:0/201705633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98c00ffa30 msgr2=0x7f98c019b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:31.025 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:31.022+0000 7f98c4b1e700 1 --2- 172.21.15.47:0/201705633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98c00ffa30 0x7f98c019b010 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f98b800ebf0 tx=0x7f98b800ef00 comp rx=0 tx=0).stop 2022-09-07T21:40:31.025 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:31.022+0000 7f98c4b1e700 1 -- 172.21.15.47:0/201705633 shutdown_connections 2022-09-07T21:40:31.026 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:31.022+0000 7f98c4b1e700 1 --2- 172.21.15.47:0/201705633 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f98a8099010 0x7f98a809b4c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:31.026 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:31.022+0000 7f98c4b1e700 1 --2- 172.21.15.47:0/201705633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98c00ffa30 0x7f98c019b010 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:31.026 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:31.022+0000 7f98c4b1e700 1 --2- 172.21.15.47:0/201705633 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98c00fde10 0x7f98c0193a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:31.026 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:31.022+0000 7f98c4b1e700 1 --2- 172.21.15.47:0/201705633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98c00ff080 0x7f98c0193fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:31.027 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:31.022+0000 7f98c4b1e700 1 -- 172.21.15.47:0/201705633 >> 172.21.15.47:0/201705633 conn(0x7f98c00f9570 msgr2=0x7f98c00fb860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:31.027 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:31.022+0000 7f98c4b1e700 1 -- 172.21.15.47:0/201705633 shutdown_connections 2022-09-07T21:40:31.027 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:31.022+0000 7f98c4b1e700 1 -- 172.21.15.47:0/201705633 wait complete. 2022-09-07T21:40:31.496 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-09-07T21:40:31.506 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi047.front.sepia.ceph.com 2022-09-07T21:40:31.506 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- bash -c 'ceph fs dump' 2022-09-07T21:40:32.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:31 smithi047 ceph-mon[90087]: pgmap v373: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:40:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:31 smithi055 ceph-mon[97033]: pgmap v373: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:40:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:31 smithi059 ceph-mon[97450]: pgmap v373: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:40:32.395 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:40:33.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:32 smithi047 ceph-mon[90087]: from='client.25180 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:32 smithi055 ceph-mon[97033]: from='client.25180 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:32 smithi059 ceph-mon[97450]: from='client.25180 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:40:34.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:33 smithi047 ceph-mon[90087]: pgmap v374: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:40:34.200 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.197+0000 7f317be15700 1 -- 172.21.15.47:0/638969215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3174105d30 msgr2=0x7f3174108150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:34.200 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.197+0000 7f317be15700 1 --2- 172.21.15.47:0/638969215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3174105d30 0x7f3174108150 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f316c009bb0 tx=0x7f316c009ec0 comp rx=0 tx=0).stop 2022-09-07T21:40:34.201 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.197+0000 7f317be15700 1 -- 172.21.15.47:0/638969215 shutdown_connections 2022-09-07T21:40:34.201 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.197+0000 7f317be15700 1 --2- 172.21.15.47:0/638969215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3174105d30 0x7f3174108150 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:34.201 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.197+0000 7f317be15700 1 --2- 172.21.15.47:0/638969215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3174072c30 0x7f3174071110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:34.201 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.197+0000 7f317be15700 1 --2- 172.21.15.47:0/638969215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3174071650 0x7f3174071ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:34.202 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.197+0000 7f317be15700 1 -- 172.21.15.47:0/638969215 >> 172.21.15.47:0/638969215 conn(0x7f31740f9570 msgr2=0x7f31740fb980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:34.202 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.197+0000 7f317be15700 1 -- 172.21.15.47:0/638969215 shutdown_connections 2022-09-07T21:40:34.202 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.197+0000 7f317be15700 1 -- 172.21.15.47:0/638969215 wait complete. 2022-09-07T21:40:34.205 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.199+0000 7f317be15700 1 Processor -- start 2022-09-07T21:40:34.205 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.199+0000 7f317be15700 1 -- start start 2022-09-07T21:40:34.205 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.199+0000 7f317be15700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3174071650 0x7f3174195be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:34.205 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.199+0000 7f317be15700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3174072c30 0x7f3174196120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:34.206 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.199+0000 7f317be15700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3174105d30 0x7f317419d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:34.206 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.199+0000 7f317be15700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317419d760 con 0x7f3174072c30 2022-09-07T21:40:34.206 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.199+0000 7f317be15700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317419d8d0 con 0x7f3174071650 2022-09-07T21:40:34.206 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.199+0000 7f317be15700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317419da90 con 0x7f3174105d30 2022-09-07T21:40:34.206 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.200+0000 7f31793b0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3174072c30 0x7f3174196120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:34.207 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.200+0000 7f317a3b2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3174105d30 0x7f317419d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:34.207 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.200+0000 7f3179bb1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3174071650 0x7f3174195be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:34.207 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.200+0000 7f317a3b2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3174105d30 0x7f317419d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:37990/0 (socket says 172.21.15.47:37990) 2022-09-07T21:40:34.207 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.200+0000 7f317a3b2700 1 -- 172.21.15.47:0/3854208836 learned_addr learned my addr 172.21.15.47:0/3854208836 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:34.208 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.200+0000 7f317a3b2700 1 -- 172.21.15.47:0/3854208836 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3174071650 msgr2=0x7f3174195be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:34.208 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.200+0000 7f317a3b2700 1 --2- 172.21.15.47:0/3854208836 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3174071650 0x7f3174195be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:34.208 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.200+0000 7f317a3b2700 1 -- 172.21.15.47:0/3854208836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3174072c30 msgr2=0x7f3174196120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:34.208 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.200+0000 7f317a3b2700 1 --2- 172.21.15.47:0/3854208836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3174072c30 0x7f3174196120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:34.208 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.200+0000 7f317a3b2700 1 -- 172.21.15.47:0/3854208836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31640093b0 con 0x7f3174105d30 2022-09-07T21:40:34.209 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.201+0000 7f317a3b2700 1 --2- 172.21.15.47:0/3854208836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3174105d30 0x7f317419d190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f316c004fd0 tx=0x7f316c0092a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:34.209 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.201+0000 7f316affd700 1 -- 172.21.15.47:0/3854208836 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316c00f870 con 0x7f3174105d30 2022-09-07T21:40:34.209 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.201+0000 7f317be15700 1 -- 172.21.15.47:0/3854208836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316c0097e0 con 0x7f3174105d30 2022-09-07T21:40:34.209 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.201+0000 7f316affd700 1 -- 172.21.15.47:0/3854208836 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f316c00ba10 con 0x7f3174105d30 2022-09-07T21:40:34.209 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.201+0000 7f316affd700 1 -- 172.21.15.47:0/3854208836 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316c0176d0 con 0x7f3174105d30 2022-09-07T21:40:34.210 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.201+0000 7f317be15700 1 -- 172.21.15.47:0/3854208836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f317419e030 con 0x7f3174105d30 2022-09-07T21:40:34.210 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.202+0000 7f317be15700 1 -- 172.21.15.47:0/3854208836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f317410b7c0 con 0x7f3174105d30 2022-09-07T21:40:34.211 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.204+0000 7f316affd700 1 -- 172.21.15.47:0/3854208836 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f316c020070 con 0x7f3174105d30 2022-09-07T21:40:34.211 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.205+0000 7f316affd700 1 --2- 172.21.15.47:0/3854208836 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3160099010 0x7f316009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:34.211 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.205+0000 7f316affd700 1 -- 172.21.15.47:0/3854208836 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f316c0c3da0 con 0x7f3174105d30 2022-09-07T21:40:34.212 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.206+0000 7f3179bb1700 1 --2- 172.21.15.47:0/3854208836 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3160099010 0x7f316009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:34.212 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.206+0000 7f3179bb1700 1 --2- 172.21.15.47:0/3854208836 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3160099010 0x7f316009b4c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3170005fd0 tx=0x7f3170005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:34.212 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.209+0000 7f316affd700 1 -- 172.21.15.47:0/3854208836 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f316c090120 con 0x7f3174105d30 2022-09-07T21:40:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:33 smithi055 ceph-mon[97033]: pgmap v374: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:40:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:33 smithi059 ceph-mon[97450]: pgmap v374: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:40:34.484 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.482+0000 7f317be15700 1 -- 172.21.15.47:0/3854208836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f317405fb40 con 0x7f3174105d30 2022-09-07T21:40:34.485 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.482+0000 7f316affd700 1 -- 172.21.15.47:0/3854208836 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+2846 (secure 0 0 0) 0x7f316c093740 con 0x7f3174105d30 2022-09-07T21:40:34.485 INFO:teuthology.orchestra.run.smithi047.stdout:e24 2022-09-07T21:40:34.486 INFO:teuthology.orchestra.run.smithi047.stdout:enable_multiple, ever_enabled_multiple: 1,1 2022-09-07T21:40:34.486 INFO:teuthology.orchestra.run.smithi047.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-09-07T21:40:34.486 INFO:teuthology.orchestra.run.smithi047.stdout:legacy client fscid: 1 2022-09-07T21:40:34.486 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:40:34.487 INFO:teuthology.orchestra.run.smithi047.stdout:Filesystem 'cephfs' (1) 2022-09-07T21:40:34.487 INFO:teuthology.orchestra.run.smithi047.stdout:fs_name cephfs 2022-09-07T21:40:34.487 INFO:teuthology.orchestra.run.smithi047.stdout:epoch 24 2022-09-07T21:40:34.487 INFO:teuthology.orchestra.run.smithi047.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-09-07T21:40:34.488 INFO:teuthology.orchestra.run.smithi047.stdout:created 2022-09-07T21:38:40.544389+0000 2022-09-07T21:40:34.488 INFO:teuthology.orchestra.run.smithi047.stdout:modified 2022-09-07T21:39:06.833800+0000 2022-09-07T21:40:34.488 INFO:teuthology.orchestra.run.smithi047.stdout:tableserver 0 2022-09-07T21:40:34.488 INFO:teuthology.orchestra.run.smithi047.stdout:root 0 2022-09-07T21:40:34.488 INFO:teuthology.orchestra.run.smithi047.stdout:session_timeout 300 2022-09-07T21:40:34.489 INFO:teuthology.orchestra.run.smithi047.stdout:session_autoclose 300 2022-09-07T21:40:34.489 INFO:teuthology.orchestra.run.smithi047.stdout:max_file_size 1099511627776 2022-09-07T21:40:34.489 INFO:teuthology.orchestra.run.smithi047.stdout:required_client_features {} 2022-09-07T21:40:34.489 INFO:teuthology.orchestra.run.smithi047.stdout:last_failure 0 2022-09-07T21:40:34.489 INFO:teuthology.orchestra.run.smithi047.stdout:last_failure_osd_epoch 0 2022-09-07T21:40:34.490 INFO:teuthology.orchestra.run.smithi047.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-09-07T21:40:34.490 INFO:teuthology.orchestra.run.smithi047.stdout:max_mds 3 2022-09-07T21:40:34.490 INFO:teuthology.orchestra.run.smithi047.stdout:in 0,1,2 2022-09-07T21:40:34.490 INFO:teuthology.orchestra.run.smithi047.stdout:up {0=24586,1=24572,2=24556} 2022-09-07T21:40:34.491 INFO:teuthology.orchestra.run.smithi047.stdout:failed 2022-09-07T21:40:34.491 INFO:teuthology.orchestra.run.smithi047.stdout:damaged 2022-09-07T21:40:34.491 INFO:teuthology.orchestra.run.smithi047.stdout:stopped 2022-09-07T21:40:34.491 INFO:teuthology.orchestra.run.smithi047.stdout:data_pools [3,4] 2022-09-07T21:40:34.492 INFO:teuthology.orchestra.run.smithi047.stdout:metadata_pool 2 2022-09-07T21:40:34.492 INFO:teuthology.orchestra.run.smithi047.stdout:inline_data disabled 2022-09-07T21:40:34.492 INFO:teuthology.orchestra.run.smithi047.stdout:balancer 2022-09-07T21:40:34.492 INFO:teuthology.orchestra.run.smithi047.stdout:standby_count_wanted 0 2022-09-07T21:40:34.492 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.a{0:24586} state up:active seq 4 addr [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] compat {c=[1],r=[1],i=[7ff]}] 2022-09-07T21:40:34.493 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.g{0:24548} state up:standby-replay seq 1 addr [v2:172.21.15.47:6838/315505812,v1:172.21.15.47:6839/315505812] compat {c=[1],r=[1],i=[7ff]}] 2022-09-07T21:40:34.493 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.b{1:24572} state up:active seq 9 addr [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] compat {c=[1],r=[1],i=[7ff]}] 2022-09-07T21:40:34.493 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.c{1:24532} state up:standby-replay seq 1 addr [v2:172.21.15.59:6836/3542466465,v1:172.21.15.59:6837/3542466465] compat {c=[1],r=[1],i=[7ff]}] 2022-09-07T21:40:34.493 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.i{2:24556} state up:active seq 12 addr [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] compat {c=[1],r=[1],i=[7ff]}] 2022-09-07T21:40:34.493 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.f{2:24526} state up:standby-replay seq 1 addr [v2:172.21.15.59:6834/2445409240,v1:172.21.15.59:6835/2445409240] compat {c=[1],r=[1],i=[7ff]}] 2022-09-07T21:40:34.494 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:40:34.494 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:40:34.494 INFO:teuthology.orchestra.run.smithi047.stdout:Standby daemons: 2022-09-07T21:40:34.494 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:40:34.495 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.j{-1:14415} state up:standby seq 1 addr [v2:172.21.15.47:6834/3393333058,v1:172.21.15.47:6835/3393333058] compat {c=[1],r=[1],i=[7ff]}] 2022-09-07T21:40:34.495 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.k{-1:24458} state up:standby seq 1 addr [v2:172.21.15.55:6832/3763216833,v1:172.21.15.55:6833/3763216833] compat {c=[1],r=[1],i=[7ff]}] 2022-09-07T21:40:34.495 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.l{-1:24466} state up:standby seq 1 addr [v2:172.21.15.59:6832/1360705881,v1:172.21.15.59:6833/1360705881] compat {c=[1],r=[1],i=[7ff]}] 2022-09-07T21:40:34.495 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.e{-1:24500} state up:standby seq 1 addr [v2:172.21.15.55:6834/2559266585,v1:172.21.15.55:6835/2559266585] compat {c=[1],r=[1],i=[7ff]}] 2022-09-07T21:40:34.495 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.d{-1:24505} state up:standby seq 1 addr [v2:172.21.15.47:6836/139419378,v1:172.21.15.47:6837/139419378] compat {c=[1],r=[1],i=[7ff]}] 2022-09-07T21:40:34.496 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.h{-1:24524} state up:standby seq 1 addr [v2:172.21.15.55:6836/2090942297,v1:172.21.15.55:6837/2090942297] compat {c=[1],r=[1],i=[7ff]}] 2022-09-07T21:40:34.497 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.485+0000 7f317be15700 1 -- 172.21.15.47:0/3854208836 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3160099010 msgr2=0x7f316009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:34.497 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.485+0000 7f317be15700 1 --2- 172.21.15.47:0/3854208836 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3160099010 0x7f316009b4c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3170005fd0 tx=0x7f3170005ea0 comp rx=0 tx=0).stop 2022-09-07T21:40:34.497 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.485+0000 7f317be15700 1 -- 172.21.15.47:0/3854208836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3174105d30 msgr2=0x7f317419d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:34.498 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.485+0000 7f317be15700 1 --2- 172.21.15.47:0/3854208836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3174105d30 0x7f317419d190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f316c004fd0 tx=0x7f316c0092a0 comp rx=0 tx=0).stop 2022-09-07T21:40:34.498 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.485+0000 7f317be15700 1 -- 172.21.15.47:0/3854208836 shutdown_connections 2022-09-07T21:40:34.498 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.485+0000 7f317be15700 1 --2- 172.21.15.47:0/3854208836 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f3160099010 0x7f316009b4c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:34.498 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.485+0000 7f317be15700 1 --2- 172.21.15.47:0/3854208836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3174105d30 0x7f317419d190 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:34.499 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.485+0000 7f317be15700 1 --2- 172.21.15.47:0/3854208836 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3174071650 0x7f3174195be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:34.499 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.485+0000 7f317be15700 1 --2- 172.21.15.47:0/3854208836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3174072c30 0x7f3174196120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:34.499 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.485+0000 7f317be15700 1 -- 172.21.15.47:0/3854208836 >> 172.21.15.47:0/3854208836 conn(0x7f31740f9570 msgr2=0x7f31740fb950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:34.499 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.485+0000 7f317be15700 1 -- 172.21.15.47:0/3854208836 shutdown_connections 2022-09-07T21:40:34.499 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:34.485+0000 7f317be15700 1 -- 172.21.15.47:0/3854208836 wait complete. 2022-09-07T21:40:34.500 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 24 2022-09-07T21:40:34.969 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- bash -c 'ceph osd dump' 2022-09-07T21:40:35.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:34 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3854208836' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-09-07T21:40:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:34 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3854208836' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-09-07T21:40:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:34 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3854208836' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-09-07T21:40:35.855 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:40:36.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:35 smithi047 ceph-mon[90087]: pgmap v375: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2022-09-07T21:40:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:35 smithi055 ceph-mon[97033]: pgmap v375: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2022-09-07T21:40:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:35 smithi059 ceph-mon[97450]: pgmap v375: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2022-09-07T21:40:37.680 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.677+0000 7f94389a4700 1 -- 172.21.15.47:0/3462253563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94340feed0 msgr2=0x7f94340ff320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:37.680 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.677+0000 7f94389a4700 1 --2- 172.21.15.47:0/3462253563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94340feed0 0x7f94340ff320 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9428009750 tx=0x7f9428009a60 comp rx=0 tx=0).stop 2022-09-07T21:40:37.681 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.677+0000 7f94389a4700 1 -- 172.21.15.47:0/3462253563 shutdown_connections 2022-09-07T21:40:37.681 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.677+0000 7f94389a4700 1 --2- 172.21.15.47:0/3462253563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94340ff860 0x7f9434106120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:37.681 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.677+0000 7f94389a4700 1 --2- 172.21.15.47:0/3462253563 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94340fdc50 0x7f94340fe060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:37.681 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.677+0000 7f94389a4700 1 --2- 172.21.15.47:0/3462253563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94340feed0 0x7f94340ff320 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:37.682 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.677+0000 7f94389a4700 1 -- 172.21.15.47:0/3462253563 >> 172.21.15.47:0/3462253563 conn(0x7f94340f9320 msgr2=0x7f94340fb770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:37.682 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.677+0000 7f94389a4700 1 -- 172.21.15.47:0/3462253563 shutdown_connections 2022-09-07T21:40:37.682 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.677+0000 7f94389a4700 1 -- 172.21.15.47:0/3462253563 wait complete. 2022-09-07T21:40:37.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.679+0000 7f94389a4700 1 Processor -- start 2022-09-07T21:40:37.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.679+0000 7f94389a4700 1 -- start start 2022-09-07T21:40:37.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.679+0000 7f94389a4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94340fdc50 0x7f9434193860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:37.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.679+0000 7f94389a4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94340feed0 0x7f9434193da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:37.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.679+0000 7f94389a4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94340ff860 0x7f943419ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:37.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.679+0000 7f94389a4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943419b3e0 con 0x7f94340fdc50 2022-09-07T21:40:37.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.679+0000 7f94389a4700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943419b550 con 0x7f94340feed0 2022-09-07T21:40:37.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.679+0000 7f94389a4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943419b710 con 0x7f94340ff860 2022-09-07T21:40:37.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.679+0000 7f9432d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94340fdc50 0x7f9434193860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:37.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.679+0000 7f9432d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94340fdc50 0x7f9434193860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44818/0 (socket says 172.21.15.47:44818) 2022-09-07T21:40:37.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.679+0000 7f9432d9d700 1 -- 172.21.15.47:0/1984818010 learned_addr learned my addr 172.21.15.47:0/1984818010 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:37.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.680+0000 7f943259c700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94340feed0 0x7f9434193da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:37.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.680+0000 7f943359e700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94340ff860 0x7f943419ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:37.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.680+0000 7f9432d9d700 1 -- 172.21.15.47:0/1984818010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94340ff860 msgr2=0x7f943419ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:37.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.680+0000 7f9432d9d700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94340ff860 0x7f943419ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:37.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.680+0000 7f9432d9d700 1 -- 172.21.15.47:0/1984818010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94340feed0 msgr2=0x7f9434193da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:37.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.680+0000 7f9432d9d700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94340feed0 0x7f9434193da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:37.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.680+0000 7f9432d9d700 1 -- 172.21.15.47:0/1984818010 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942c0097e0 con 0x7f94340fdc50 2022-09-07T21:40:37.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.680+0000 7f943259c700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94340feed0 0x7f9434193da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:37.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.680+0000 7f9432d9d700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94340fdc50 0x7f9434193860 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f942400e710 tx=0x7f942400ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:37.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.681+0000 7f941bfff700 1 -- 172.21.15.47:0/1984818010 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9424004220 con 0x7f94340fdc50 2022-09-07T21:40:37.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.681+0000 7f94389a4700 1 -- 172.21.15.47:0/1984818010 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94280093b0 con 0x7f94340fdc50 2022-09-07T21:40:37.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.681+0000 7f941bfff700 1 -- 172.21.15.47:0/1984818010 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f9424004380 con 0x7f94340fdc50 2022-09-07T21:40:37.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.681+0000 7f941bfff700 1 -- 172.21.15.47:0/1984818010 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9424003cf0 con 0x7f94340fdc50 2022-09-07T21:40:37.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.681+0000 7f94389a4700 1 -- 172.21.15.47:0/1984818010 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f943419bdc0 con 0x7f94340fdc50 2022-09-07T21:40:37.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.682+0000 7f94389a4700 1 -- 172.21.15.47:0/1984818010 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9434109400 con 0x7f94340fdc50 2022-09-07T21:40:37.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.684+0000 7f941bfff700 1 -- 172.21.15.47:0/1984818010 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f942401e020 con 0x7f94340fdc50 2022-09-07T21:40:37.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.685+0000 7f941bfff700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f941c099010 0x7f941c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:37.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.685+0000 7f941bfff700 1 -- 172.21.15.47:0/1984818010 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f9424014070 con 0x7f94340fdc50 2022-09-07T21:40:37.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.685+0000 7f943259c700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f941c099010 0x7f941c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:37.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.686+0000 7f943259c700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f941c099010 0x7f941c09b4c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f942800f500 tx=0x7f9428005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:37.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.689+0000 7f941bfff700 1 -- 172.21.15.47:0/1984818010 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f942408eca0 con 0x7f94340fdc50 2022-09-07T21:40:37.923 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.920+0000 7f94389a4700 1 -- 172.21.15.47:0/1984818010 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f943405fa30 con 0x7f94340fdc50 2022-09-07T21:40:37.923 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.920+0000 7f941bfff700 1 -- 172.21.15.47:0/1984818010 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v91) v1 ==== 56+0+5502 (secure 0 0 0) 0x7f94240922c0 con 0x7f94340fdc50 2022-09-07T21:40:37.924 INFO:teuthology.orchestra.run.smithi047.stdout:epoch 91 2022-09-07T21:40:37.924 INFO:teuthology.orchestra.run.smithi047.stdout:fsid 01b8396e-2ef4-11ed-8431-001a4aab830c 2022-09-07T21:40:37.924 INFO:teuthology.orchestra.run.smithi047.stdout:created 2022-09-07T21:29:33.859557+0000 2022-09-07T21:40:37.924 INFO:teuthology.orchestra.run.smithi047.stdout:modified 2022-09-07T21:39:47.780661+0000 2022-09-07T21:40:37.925 INFO:teuthology.orchestra.run.smithi047.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2022-09-07T21:40:37.925 INFO:teuthology.orchestra.run.smithi047.stdout:crush_version 27 2022-09-07T21:40:37.925 INFO:teuthology.orchestra.run.smithi047.stdout:full_ratio 0.95 2022-09-07T21:40:37.925 INFO:teuthology.orchestra.run.smithi047.stdout:backfillfull_ratio 0.9 2022-09-07T21:40:37.926 INFO:teuthology.orchestra.run.smithi047.stdout:nearfull_ratio 0.85 2022-09-07T21:40:37.926 INFO:teuthology.orchestra.run.smithi047.stdout:require_min_compat_client luminous 2022-09-07T21:40:37.926 INFO:teuthology.orchestra.run.smithi047.stdout:min_compat_client luminous 2022-09-07T21:40:37.926 INFO:teuthology.orchestra.run.smithi047.stdout:require_osd_release quincy 2022-09-07T21:40:37.927 INFO:teuthology.orchestra.run.smithi047.stdout:stretch_mode_enabled false 2022-09-07T21:40:37.927 INFO:teuthology.orchestra.run.smithi047.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2022-09-07T21:40:37.927 INFO:teuthology.orchestra.run.smithi047.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs 2022-09-07T21:40:37.927 INFO:teuthology.orchestra.run.smithi047.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs 2022-09-07T21:40:37.928 INFO:teuthology.orchestra.run.smithi047.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2022-09-07T21:40:37.928 INFO:teuthology.orchestra.run.smithi047.stdout:max_osd 12 2022-09-07T21:40:37.928 INFO:teuthology.orchestra.run.smithi047.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.47:6802/3092904881,v1:172.21.15.47:6803/3092904881] [v2:172.21.15.47:6804/3092904881,v1:172.21.15.47:6805/3092904881] exists,up 7c4477c8-f9db-4bd6-80f3-ac83b7c86010 2022-09-07T21:40:37.928 INFO:teuthology.orchestra.run.smithi047.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.55:6800/990490301,v1:172.21.15.55:6801/990490301] [v2:172.21.15.55:6802/990490301,v1:172.21.15.55:6803/990490301] exists,up 692ba70e-935d-4a34-847d-44a304850c5b 2022-09-07T21:40:37.928 INFO:teuthology.orchestra.run.smithi047.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6800/2729613578,v1:172.21.15.59:6801/2729613578] [v2:172.21.15.59:6802/2729613578,v1:172.21.15.59:6803/2729613578] exists,up 20bbe910-ab0c-4921-b54f-6d37be12f5c6 2022-09-07T21:40:37.929 INFO:teuthology.orchestra.run.smithi047.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.47:6810/1063754579,v1:172.21.15.47:6811/1063754579] [v2:172.21.15.47:6812/1063754579,v1:172.21.15.47:6813/1063754579] exists,up b4b5e6b0-5a09-4711-bd7f-5496b0547af2 2022-09-07T21:40:37.929 INFO:teuthology.orchestra.run.smithi047.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.55:6808/4055716956,v1:172.21.15.55:6809/4055716956] [v2:172.21.15.55:6810/4055716956,v1:172.21.15.55:6811/4055716956] exists,up f98aac6c-62be-4f57-94e1-4baa606d9e9e 2022-09-07T21:40:37.929 INFO:teuthology.orchestra.run.smithi047.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6808/725179539,v1:172.21.15.59:6809/725179539] [v2:172.21.15.59:6810/725179539,v1:172.21.15.59:6811/725179539] exists,up c2b864da-e60b-4140-a6d9-30e4e375ef6e 2022-09-07T21:40:37.929 INFO:teuthology.orchestra.run.smithi047.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.47:6818/4053448165,v1:172.21.15.47:6819/4053448165] [v2:172.21.15.47:6820/4053448165,v1:172.21.15.47:6821/4053448165] exists,up fbe32a2e-570c-4f3b-a860-1c0e603ae96b 2022-09-07T21:40:37.930 INFO:teuthology.orchestra.run.smithi047.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.55:6816/199887712,v1:172.21.15.55:6817/199887712] [v2:172.21.15.55:6818/199887712,v1:172.21.15.55:6819/199887712] exists,up 85e70ce9-e721-4719-b309-e5011ddbbab3 2022-09-07T21:40:37.930 INFO:teuthology.orchestra.run.smithi047.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6816/1132606220,v1:172.21.15.59:6817/1132606220] [v2:172.21.15.59:6818/1132606220,v1:172.21.15.59:6819/1132606220] exists,up e5b3728f-7ba6-4d25-a54c-d04bad536de1 2022-09-07T21:40:37.930 INFO:teuthology.orchestra.run.smithi047.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.47:6826/3489379795,v1:172.21.15.47:6827/3489379795] [v2:172.21.15.47:6828/3489379795,v1:172.21.15.47:6829/3489379795] exists,up 3f08f42a-4784-4e10-b862-16806fa5ce27 2022-09-07T21:40:37.930 INFO:teuthology.orchestra.run.smithi047.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.55:6824/2413766072,v1:172.21.15.55:6825/2413766072] [v2:172.21.15.55:6826/2413766072,v1:172.21.15.55:6827/2413766072] exists,up 684aed4e-909e-470a-a8e2-a0b3a52aca2f 2022-09-07T21:40:37.930 INFO:teuthology.orchestra.run.smithi047.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6824/1569847315,v1:172.21.15.59:6825/1569847315] [v2:172.21.15.59:6826/1569847315,v1:172.21.15.59:6827/1569847315] exists,up d5c3afb2-663c-434c-980a-37b1f230b170 2022-09-07T21:40:37.931 INFO:teuthology.orchestra.run.smithi047.stdout:pg_upmap_items 2.16 [2,4] 2022-09-07T21:40:37.931 INFO:teuthology.orchestra.run.smithi047.stdout:pg_upmap_items 2.18 [5,11] 2022-09-07T21:40:37.931 INFO:teuthology.orchestra.run.smithi047.stdout:pg_upmap_items 2.1b [2,4] 2022-09-07T21:40:37.932 INFO:teuthology.orchestra.run.smithi047.stdout:pg_upmap_items 2.1d [8,11] 2022-09-07T21:40:37.932 INFO:teuthology.orchestra.run.smithi047.stdout:pg_upmap_items 2.1f [8,1] 2022-09-07T21:40:37.932 INFO:teuthology.orchestra.run.smithi047.stdout:pg_upmap_items 3.21 [4,8] 2022-09-07T21:40:37.932 INFO:teuthology.orchestra.run.smithi047.stdout:pg_upmap_items 4.7 [5,8] 2022-09-07T21:40:37.932 INFO:teuthology.orchestra.run.smithi047.stdout:pg_upmap_items 4.f [9,10] 2022-09-07T21:40:37.933 INFO:teuthology.orchestra.run.smithi047.stdout:pg_upmap_items 4.13 [9,8] 2022-09-07T21:40:37.933 INFO:teuthology.orchestra.run.smithi047.stdout:pg_upmap_items 4.17 [9,8] 2022-09-07T21:40:37.933 INFO:teuthology.orchestra.run.smithi047.stdout:blocklist 172.21.15.47:0/2838000558 expires 2022-09-08T21:30:46.439577+0000 2022-09-07T21:40:37.933 INFO:teuthology.orchestra.run.smithi047.stdout:blocklist 172.21.15.47:6801/4097349239 expires 2022-09-08T21:30:46.439577+0000 2022-09-07T21:40:37.934 INFO:teuthology.orchestra.run.smithi047.stdout:blocklist 172.21.15.47:6800/4097349239 expires 2022-09-08T21:30:46.439577+0000 2022-09-07T21:40:37.934 INFO:teuthology.orchestra.run.smithi047.stdout:blocklist 172.21.15.47:6801/1435938941 expires 2022-09-08T21:30:15.415366+0000 2022-09-07T21:40:37.934 INFO:teuthology.orchestra.run.smithi047.stdout:blocklist 172.21.15.47:6800/1435938941 expires 2022-09-08T21:30:15.415366+0000 2022-09-07T21:40:37.934 INFO:teuthology.orchestra.run.smithi047.stdout:blocklist 172.21.15.47:0/2303715505 expires 2022-09-08T21:30:15.415366+0000 2022-09-07T21:40:37.934 INFO:teuthology.orchestra.run.smithi047.stdout:blocklist 172.21.15.47:0/1816663715 expires 2022-09-08T21:30:46.439577+0000 2022-09-07T21:40:37.935 INFO:teuthology.orchestra.run.smithi047.stdout:blocklist 172.21.15.47:0/3909884596 expires 2022-09-08T21:30:15.415366+0000 2022-09-07T21:40:37.935 INFO:teuthology.orchestra.run.smithi047.stdout:blocklist 172.21.15.47:0/2780837564 expires 2022-09-08T21:30:15.415366+0000 2022-09-07T21:40:37.935 INFO:teuthology.orchestra.run.smithi047.stdout:blocklist 172.21.15.47:0/1253653766 expires 2022-09-08T21:30:46.439577+0000 2022-09-07T21:40:37.936 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.923+0000 7f94389a4700 1 -- 172.21.15.47:0/1984818010 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f941c099010 msgr2=0x7f941c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:37.936 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.923+0000 7f94389a4700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f941c099010 0x7f941c09b4c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f942800f500 tx=0x7f9428005f90 comp rx=0 tx=0).stop 2022-09-07T21:40:37.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.923+0000 7f94389a4700 1 -- 172.21.15.47:0/1984818010 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94340fdc50 msgr2=0x7f9434193860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:37.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.923+0000 7f94389a4700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94340fdc50 0x7f9434193860 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f942400e710 tx=0x7f942400ea20 comp rx=0 tx=0).stop 2022-09-07T21:40:37.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.923+0000 7f94389a4700 1 -- 172.21.15.47:0/1984818010 shutdown_connections 2022-09-07T21:40:37.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.923+0000 7f94389a4700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f941c099010 0x7f941c09b4c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:37.938 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.923+0000 7f94389a4700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94340ff860 0x7f943419ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:37.938 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.923+0000 7f94389a4700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94340fdc50 0x7f9434193860 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:37.938 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.923+0000 7f94389a4700 1 --2- 172.21.15.47:0/1984818010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94340feed0 0x7f9434193da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:37.938 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.923+0000 7f94389a4700 1 -- 172.21.15.47:0/1984818010 >> 172.21.15.47:0/1984818010 conn(0x7f94340f9320 msgr2=0x7f94340fb600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:37.938 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.924+0000 7f94389a4700 1 -- 172.21.15.47:0/1984818010 shutdown_connections 2022-09-07T21:40:37.939 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:37.924+0000 7f94389a4700 1 -- 172.21.15.47:0/1984818010 wait complete. 2022-09-07T21:40:38.094 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:37 smithi047 ceph-mon[90087]: pgmap v376: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:37 smithi055 ceph-mon[97033]: pgmap v376: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:37 smithi059 ceph-mon[97450]: pgmap v376: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:38.431 INFO:teuthology.run_tasks:Running task fs.ready... 2022-09-07T21:40:38.444 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-07T21:40:38.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.682+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2822578756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cbc066be0 msgr2=0x7f2cbc1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:38.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.682+0000 7f2cc1bff700 1 --2- 172.21.15.47:0/2822578756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cbc066be0 0x7f2cbc1008b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009810 tx=0x7f2cac009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:38.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.682+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2822578756 shutdown_connections 2022-09-07T21:40:38.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.682+0000 7f2cc1bff700 1 --2- 172.21.15.47:0/2822578756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2cbc1038e0 0x7f2cbc105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:38.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.682+0000 7f2cc1bff700 1 --2- 172.21.15.47:0/2822578756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cbc066be0 0x7f2cbc1008b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:38.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.682+0000 7f2cc1bff700 1 --2- 172.21.15.47:0/2822578756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cbc100df0 0x7f2cbc103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:38.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.682+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2822578756 >> 172.21.15.47:0/2822578756 conn(0x7f2cbc0f82e0 msgr2=0x7f2cbc0fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:38.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.682+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2822578756 shutdown_connections 2022-09-07T21:40:38.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.682+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2822578756 wait complete. 2022-09-07T21:40:38.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.683+0000 7f2cc1bff700 1 Processor -- start 2022-09-07T21:40:38.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.683+0000 7f2cc1bff700 1 -- start start 2022-09-07T21:40:38.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.684+0000 7f2cc1bff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cbc066be0 0x7f2cbc195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.684+0000 7f2cc1bff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cbc100df0 0x7f2cbc196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.684+0000 7f2cc1bff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2cbc1038e0 0x7f2cbc19d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.684+0000 7f2cc1bff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc19d6b0 con 0x7f2cbc100df0 2022-09-07T21:40:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.684+0000 7f2cc1bff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc19d820 con 0x7f2cbc1038e0 2022-09-07T21:40:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.684+0000 7f2cc1bff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc19d9e0 con 0x7f2cbc066be0 2022-09-07T21:40:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.684+0000 7f2cbb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cbc066be0 0x7f2cbc195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.684+0000 7f2cbb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cbc066be0 0x7f2cbc195b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:56294/0 (socket says 172.21.15.47:56294) 2022-09-07T21:40:38.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.684+0000 7f2cbb7fe700 1 -- 172.21.15.47:0/2226033300 learned_addr learned my addr 172.21.15.47:0/2226033300 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:38.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.684+0000 7f2cbaffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cbc100df0 0x7f2cbc196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:38.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.684+0000 7f2cbbfff700 1 --2- 172.21.15.47:0/2226033300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2cbc1038e0 0x7f2cbc19d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:38.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.685+0000 7f2cbb7fe700 1 -- 172.21.15.47:0/2226033300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2cbc1038e0 msgr2=0x7f2cbc19d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:38.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.685+0000 7f2cbb7fe700 1 --2- 172.21.15.47:0/2226033300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2cbc1038e0 0x7f2cbc19d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:38.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.685+0000 7f2cbb7fe700 1 -- 172.21.15.47:0/2226033300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cbc100df0 msgr2=0x7f2cbc196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:38.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.685+0000 7f2cbb7fe700 1 --2- 172.21.15.47:0/2226033300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cbc100df0 0x7f2cbc196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:38.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.685+0000 7f2cbb7fe700 1 -- 172.21.15.47:0/2226033300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb00093b0 con 0x7f2cbc066be0 2022-09-07T21:40:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.685+0000 7f2cbb7fe700 1 --2- 172.21.15.47:0/2226033300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cbc066be0 0x7f2cbc195b30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2cac000c00 tx=0x7f2cac00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.685+0000 7f2cb8ff9700 1 -- 172.21.15.47:0/2226033300 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac01d070 con 0x7f2cbc066be0 2022-09-07T21:40:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.685+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2226033300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cac009470 con 0x7f2cbc066be0 2022-09-07T21:40:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.685+0000 7f2cb8ff9700 1 -- 172.21.15.47:0/2226033300 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2cac00f880 con 0x7f2cbc066be0 2022-09-07T21:40:38.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.685+0000 7f2cb8ff9700 1 -- 172.21.15.47:0/2226033300 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac004740 con 0x7f2cbc066be0 2022-09-07T21:40:38.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.685+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2226033300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cbc19dfd0 con 0x7f2cbc066be0 2022-09-07T21:40:38.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.687+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2226033300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cbc109460 con 0x7f2cbc066be0 2022-09-07T21:40:38.696 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.689+0000 7f2cb8ff9700 1 -- 172.21.15.47:0/2226033300 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f2cac028030 con 0x7f2cbc066be0 2022-09-07T21:40:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.690+0000 7f2cb8ff9700 1 --2- 172.21.15.47:0/2226033300 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2ca4099000 0x7f2ca409b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.690+0000 7f2cb8ff9700 1 -- 172.21.15.47:0/2226033300 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f2cac0c3350 con 0x7f2cbc066be0 2022-09-07T21:40:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.690+0000 7f2cbaffd700 1 --2- 172.21.15.47:0/2226033300 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2ca4099000 0x7f2ca409b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.691+0000 7f2cbaffd700 1 --2- 172.21.15.47:0/2226033300 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2ca4099000 0x7f2ca409b4b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0005fd0 tx=0x7f2cb0005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:38.698 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.692+0000 7f2cb8ff9700 1 -- 172.21.15.47:0/2226033300 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f2cac08f6d0 con 0x7f2cbc066be0 2022-09-07T21:40:38.900 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.897+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2226033300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cbc05f950 con 0x7f2cbc066be0 2022-09-07T21:40:38.901 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.900+0000 7f2cb8ff9700 1 -- 172.21.15.47:0/2226033300 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+23592 (secure 0 0 0) 0x7f2cac092cf0 con 0x7f2cbc066be0 2022-09-07T21:40:38.902 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:40:38.903 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":91,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:39:47.780661+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-07T21:38:38.608937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-07T21:38:39.936252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-07T21:38:43.731797+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.18","mappings":[{"from":5,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:40:38.904 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.902+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2226033300 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2ca4099000 msgr2=0x7f2ca409b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:38.905 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.902+0000 7f2cc1bff700 1 --2- 172.21.15.47:0/2226033300 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2ca4099000 0x7f2ca409b4b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0005fd0 tx=0x7f2cb0005ea0 comp rx=0 tx=0).stop 2022-09-07T21:40:38.905 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.902+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2226033300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cbc066be0 msgr2=0x7f2cbc195b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:38.905 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.902+0000 7f2cc1bff700 1 --2- 172.21.15.47:0/2226033300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cbc066be0 0x7f2cbc195b30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2cac000c00 tx=0x7f2cac00f690 comp rx=0 tx=0).stop 2022-09-07T21:40:38.905 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.902+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2226033300 shutdown_connections 2022-09-07T21:40:38.906 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.902+0000 7f2cc1bff700 1 --2- 172.21.15.47:0/2226033300 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f2ca4099000 0x7f2ca409b4b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:38.906 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.902+0000 7f2cc1bff700 1 --2- 172.21.15.47:0/2226033300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2cbc1038e0 0x7f2cbc19d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:38.906 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.902+0000 7f2cc1bff700 1 --2- 172.21.15.47:0/2226033300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cbc066be0 0x7f2cbc195b30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:38.906 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.902+0000 7f2cc1bff700 1 --2- 172.21.15.47:0/2226033300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cbc100df0 0x7f2cbc196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:38.907 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.902+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2226033300 >> 172.21.15.47:0/2226033300 conn(0x7f2cbc0f82e0 msgr2=0x7f2cbc104640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:38.907 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.902+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2226033300 shutdown_connections 2022-09-07T21:40:38.907 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:38.902+0000 7f2cc1bff700 1 -- 172.21.15.47:0/2226033300 wait complete. 2022-09-07T21:40:38.933 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-07T21:33:29.342534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-09-07T21:38:38.608937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-09-07T21:38:39.936252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2022-09-07T21:38:43.731797+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-09-07T21:40:38.934 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-09-07T21:40:39.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:38 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1984818010' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-09-07T21:40:39.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.167+0000 7f723edff700 1 -- 172.21.15.47:0/811574464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7238066be0 msgr2=0x7f7238100960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:39.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.167+0000 7f723edff700 1 --2- 172.21.15.47:0/811574464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7238066be0 0x7f7238100960 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f722c009810 tx=0x7f722c009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:39.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.167+0000 7f723edff700 1 -- 172.21.15.47:0/811574464 shutdown_connections 2022-09-07T21:40:39.170 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.167+0000 7f723edff700 1 --2- 172.21.15.47:0/811574464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7238103990 0x7f7238105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.171 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.167+0000 7f723edff700 1 --2- 172.21.15.47:0/811574464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7238066be0 0x7f7238100960 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.171 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.167+0000 7f723edff700 1 --2- 172.21.15.47:0/811574464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7238100ea0 0x7f7238103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.171 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.167+0000 7f723edff700 1 -- 172.21.15.47:0/811574464 >> 172.21.15.47:0/811574464 conn(0x7f72380f8370 msgr2=0x7f72380fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:39.171 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.167+0000 7f723edff700 1 -- 172.21.15.47:0/811574464 shutdown_connections 2022-09-07T21:40:39.172 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.167+0000 7f723edff700 1 -- 172.21.15.47:0/811574464 wait complete. 2022-09-07T21:40:39.174 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.169+0000 7f723edff700 1 Processor -- start 2022-09-07T21:40:39.174 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.169+0000 7f723edff700 1 -- start start 2022-09-07T21:40:39.175 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.169+0000 7f723edff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7238066be0 0x7f72380fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:39.175 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.169+0000 7f723edff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7238100ea0 0x7f72380fc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:39.175 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.169+0000 7f723edff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7238103990 0x7f72380fcea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:39.175 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.169+0000 7f723edff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72380fd640 con 0x7f7238103990 2022-09-07T21:40:39.175 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.169+0000 7f723edff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72380fd7b0 con 0x7f7238100ea0 2022-09-07T21:40:39.176 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.169+0000 7f723edff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723819dac0 con 0x7f7238066be0 2022-09-07T21:40:39.176 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.170+0000 7f723cb9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7238066be0 0x7f72380fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:39.176 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.170+0000 7f723cb9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7238066be0 0x7f72380fe2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:56306/0 (socket says 172.21.15.47:56306) 2022-09-07T21:40:39.176 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.170+0000 7f723cb9b700 1 -- 172.21.15.47:0/3817166981 learned_addr learned my addr 172.21.15.47:0/3817166981 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:39.177 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.170+0000 7f7237fff700 1 --2- 172.21.15.47:0/3817166981 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7238100ea0 0x7f72380fc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:39.177 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.170+0000 7f723d39c700 1 --2- 172.21.15.47:0/3817166981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7238103990 0x7f72380fcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:39.177 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.170+0000 7f723cb9b700 1 -- 172.21.15.47:0/3817166981 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7238100ea0 msgr2=0x7f72380fc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:39.177 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.170+0000 7f723cb9b700 1 --2- 172.21.15.47:0/3817166981 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7238100ea0 0x7f72380fc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.177 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.170+0000 7f723cb9b700 1 -- 172.21.15.47:0/3817166981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7238103990 msgr2=0x7f72380fcea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:39.178 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.170+0000 7f723cb9b700 1 --2- 172.21.15.47:0/3817166981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7238103990 0x7f72380fcea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.178 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.170+0000 7f723cb9b700 1 -- 172.21.15.47:0/3817166981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c009470 con 0x7f7238066be0 2022-09-07T21:40:39.178 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.170+0000 7f723cb9b700 1 --2- 172.21.15.47:0/3817166981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7238066be0 0x7f72380fe2c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f722c000c00 tx=0x7f722c00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:39.178 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.171+0000 7f7235ffb700 1 -- 172.21.15.47:0/3817166981 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c01d070 con 0x7f7238066be0 2022-09-07T21:40:39.179 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.171+0000 7f723edff700 1 -- 172.21.15.47:0/3817166981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f723819dd40 con 0x7f7238066be0 2022-09-07T21:40:39.179 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.171+0000 7f7235ffb700 1 -- 172.21.15.47:0/3817166981 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f722c00f880 con 0x7f7238066be0 2022-09-07T21:40:39.179 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.171+0000 7f7235ffb700 1 -- 172.21.15.47:0/3817166981 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c0047f0 con 0x7f7238066be0 2022-09-07T21:40:39.179 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.171+0000 7f723edff700 1 -- 172.21.15.47:0/3817166981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723819e190 con 0x7f7238066be0 2022-09-07T21:40:39.180 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.172+0000 7f723edff700 1 -- 172.21.15.47:0/3817166981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7238109560 con 0x7f7238066be0 2022-09-07T21:40:39.180 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.174+0000 7f7235ffb700 1 -- 172.21.15.47:0/3817166981 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f722c028030 con 0x7f7238066be0 2022-09-07T21:40:39.181 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.175+0000 7f7235ffb700 1 --2- 172.21.15.47:0/3817166981 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f7228099010 0x7f722809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:39.181 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.175+0000 7f7235ffb700 1 -- 172.21.15.47:0/3817166981 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f722c0c3740 con 0x7f7238066be0 2022-09-07T21:40:39.181 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.175+0000 7f7237fff700 1 --2- 172.21.15.47:0/3817166981 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f7228099010 0x7f722809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:39.181 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.176+0000 7f7237fff700 1 --2- 172.21.15.47:0/3817166981 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f7228099010 0x7f722809b4c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7224005fd0 tx=0x7f7224005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:39.182 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.177+0000 7f7235ffb700 1 -- 172.21.15.47:0/3817166981 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f722c08fac0 con 0x7f7238066be0 2022-09-07T21:40:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:38 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1984818010' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-09-07T21:40:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:38 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1984818010' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-09-07T21:40:39.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.378+0000 7f723edff700 1 -- 172.21.15.47:0/3817166981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f72380f9d60 con 0x7f7238066be0 2022-09-07T21:40:39.381 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.378+0000 7f7235ffb700 1 -- 172.21.15.47:0/3817166981 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f722c0260a0 con 0x7f7238066be0 2022-09-07T21:40:39.382 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2022-09-07T21:40:39.383 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.381+0000 7f723edff700 1 -- 172.21.15.47:0/3817166981 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f7228099010 msgr2=0x7f722809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:39.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.381+0000 7f723edff700 1 --2- 172.21.15.47:0/3817166981 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f7228099010 0x7f722809b4c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7224005fd0 tx=0x7f7224005ea0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.381+0000 7f723edff700 1 -- 172.21.15.47:0/3817166981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7238066be0 msgr2=0x7f72380fe2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:39.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.381+0000 7f723edff700 1 --2- 172.21.15.47:0/3817166981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7238066be0 0x7f72380fe2c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f722c000c00 tx=0x7f722c00f690 comp rx=0 tx=0).stop 2022-09-07T21:40:39.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.381+0000 7f723edff700 1 -- 172.21.15.47:0/3817166981 shutdown_connections 2022-09-07T21:40:39.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.381+0000 7f723edff700 1 --2- 172.21.15.47:0/3817166981 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f7228099010 0x7f722809b4c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.381+0000 7f723edff700 1 --2- 172.21.15.47:0/3817166981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7238103990 0x7f72380fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.381+0000 7f723edff700 1 --2- 172.21.15.47:0/3817166981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7238066be0 0x7f72380fe2c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.386 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.381+0000 7f723edff700 1 --2- 172.21.15.47:0/3817166981 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7238100ea0 0x7f72380fc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.386 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.381+0000 7f723edff700 1 -- 172.21.15.47:0/3817166981 >> 172.21.15.47:0/3817166981 conn(0x7f72380f8370 msgr2=0x7f72380ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:39.386 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.381+0000 7f723edff700 1 -- 172.21.15.47:0/3817166981 shutdown_connections 2022-09-07T21:40:39.386 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.381+0000 7f723edff700 1 -- 172.21.15.47:0/3817166981 wait complete. 2022-09-07T21:40:39.411 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-09-07T21:40:39.651 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.644+0000 7f02593ff700 1 -- 172.21.15.47:0/3677670955 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0254066be0 msgr2=0x7f02541008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:39.652 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.644+0000 7f02593ff700 1 --2- 172.21.15.47:0/3677670955 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0254066be0 0x7f02541008e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0244009750 tx=0x7f0244009a60 comp rx=0 tx=0).stop 2022-09-07T21:40:39.652 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.644+0000 7f02593ff700 1 -- 172.21.15.47:0/3677670955 shutdown_connections 2022-09-07T21:40:39.653 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.644+0000 7f02593ff700 1 --2- 172.21.15.47:0/3677670955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0254103910 0x7f0254105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.653 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.644+0000 7f02593ff700 1 --2- 172.21.15.47:0/3677670955 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0254066be0 0x7f02541008e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.654 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.644+0000 7f02593ff700 1 --2- 172.21.15.47:0/3677670955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0254100e20 0x7f0254103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.654 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.644+0000 7f02593ff700 1 -- 172.21.15.47:0/3677670955 >> 172.21.15.47:0/3677670955 conn(0x7f02540f8310 msgr2=0x7f02540fa740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:39.654 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.644+0000 7f02593ff700 1 -- 172.21.15.47:0/3677670955 shutdown_connections 2022-09-07T21:40:39.655 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.644+0000 7f02593ff700 1 -- 172.21.15.47:0/3677670955 wait complete. 2022-09-07T21:40:39.655 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.645+0000 7f02593ff700 1 Processor -- start 2022-09-07T21:40:39.656 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.646+0000 7f02593ff700 1 -- start start 2022-09-07T21:40:39.656 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.646+0000 7f02593ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0254066be0 0x7f02540fe240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:39.656 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.646+0000 7f02593ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0254100e20 0x7f02540fc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:39.656 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.646+0000 7f02593ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0254103910 0x7f02540fce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:39.657 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.646+0000 7f02593ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02540fd5c0 con 0x7f0254100e20 2022-09-07T21:40:39.657 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.646+0000 7f02593ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02540fd730 con 0x7f0254066be0 2022-09-07T21:40:39.657 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.646+0000 7f02593ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025419da30 con 0x7f0254103910 2022-09-07T21:40:39.657 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.646+0000 7f02527fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0254100e20 0x7f02540fc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:39.657 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.646+0000 7f02527fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0254100e20 0x7f02540fc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33948/0 (socket says 172.21.15.47:33948) 2022-09-07T21:40:39.658 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.646+0000 7f02527fc700 1 -- 172.21.15.47:0/1841830705 learned_addr learned my addr 172.21.15.47:0/1841830705 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:39.658 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.646+0000 7f0252ffd700 1 --2- 172.21.15.47:0/1841830705 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0254066be0 0x7f02540fe240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:39.658 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.646+0000 7f02537fe700 1 --2- 172.21.15.47:0/1841830705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0254103910 0x7f02540fce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:39.658 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.647+0000 7f02527fc700 1 -- 172.21.15.47:0/1841830705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0254103910 msgr2=0x7f02540fce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:39.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.647+0000 7f02527fc700 1 --2- 172.21.15.47:0/1841830705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0254103910 0x7f02540fce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.647+0000 7f02527fc700 1 -- 172.21.15.47:0/1841830705 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0254066be0 msgr2=0x7f02540fe240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:39.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.647+0000 7f02527fc700 1 --2- 172.21.15.47:0/1841830705 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0254066be0 0x7f02540fe240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.647+0000 7f02527fc700 1 -- 172.21.15.47:0/1841830705 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02440093b0 con 0x7f0254100e20 2022-09-07T21:40:39.660 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.647+0000 7f02527fc700 1 --2- 172.21.15.47:0/1841830705 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0254100e20 0x7f02540fc880 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f024800e740 tx=0x7f024800ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:39.660 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.647+0000 7f0237fff700 1 -- 172.21.15.47:0/1841830705 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0248005220 con 0x7f0254100e20 2022-09-07T21:40:39.660 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.647+0000 7f02593ff700 1 -- 172.21.15.47:0/1841830705 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f025419dd10 con 0x7f0254100e20 2022-09-07T21:40:39.660 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.647+0000 7f0237fff700 1 -- 172.21.15.47:0/1841830705 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0248005380 con 0x7f0254100e20 2022-09-07T21:40:39.661 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.647+0000 7f0237fff700 1 -- 172.21.15.47:0/1841830705 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0248010ba0 con 0x7f0254100e20 2022-09-07T21:40:39.661 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.647+0000 7f02593ff700 1 -- 172.21.15.47:0/1841830705 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f025419e210 con 0x7f0254100e20 2022-09-07T21:40:39.661 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.649+0000 7f02593ff700 1 -- 172.21.15.47:0/1841830705 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02541094e0 con 0x7f0254100e20 2022-09-07T21:40:39.662 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.650+0000 7f0237fff700 1 -- 172.21.15.47:0/1841830705 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f024801e020 con 0x7f0254100e20 2022-09-07T21:40:39.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.652+0000 7f0237fff700 1 --2- 172.21.15.47:0/1841830705 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f023c099090 0x7f023c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:39.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.652+0000 7f0237fff700 1 -- 172.21.15.47:0/1841830705 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f0248014070 con 0x7f0254100e20 2022-09-07T21:40:39.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.652+0000 7f0252ffd700 1 --2- 172.21.15.47:0/1841830705 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f023c099090 0x7f023c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:39.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.653+0000 7f0252ffd700 1 --2- 172.21.15.47:0/1841830705 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f023c099090 0x7f023c09b540 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0244009e30 tx=0x7f0244005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:39.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.656+0000 7f0237fff700 1 -- 172.21.15.47:0/1841830705 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f024808ecd0 con 0x7f0254100e20 2022-09-07T21:40:39.860 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.857+0000 7f02593ff700 1 -- 172.21.15.47:0/1841830705 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f025405fb00 con 0x7f0254100e20 2022-09-07T21:40:39.860 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.858+0000 7f0237fff700 1 -- 172.21.15.47:0/1841830705 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f02480922f0 con 0x7f0254100e20 2022-09-07T21:40:39.861 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2022-09-07T21:40:39.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.860+0000 7f02593ff700 1 -- 172.21.15.47:0/1841830705 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f023c099090 msgr2=0x7f023c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:39.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.860+0000 7f02593ff700 1 --2- 172.21.15.47:0/1841830705 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f023c099090 0x7f023c09b540 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0244009e30 tx=0x7f0244005f90 comp rx=0 tx=0).stop 2022-09-07T21:40:39.863 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.860+0000 7f02593ff700 1 -- 172.21.15.47:0/1841830705 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0254100e20 msgr2=0x7f02540fc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:39.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.860+0000 7f02593ff700 1 --2- 172.21.15.47:0/1841830705 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0254100e20 0x7f02540fc880 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f024800e740 tx=0x7f024800ea50 comp rx=0 tx=0).stop 2022-09-07T21:40:39.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.860+0000 7f02593ff700 1 -- 172.21.15.47:0/1841830705 shutdown_connections 2022-09-07T21:40:39.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.860+0000 7f02593ff700 1 --2- 172.21.15.47:0/1841830705 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f023c099090 0x7f023c09b540 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.864 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.860+0000 7f02593ff700 1 --2- 172.21.15.47:0/1841830705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0254103910 0x7f02540fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.860+0000 7f02593ff700 1 --2- 172.21.15.47:0/1841830705 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0254066be0 0x7f02540fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.860+0000 7f02593ff700 1 --2- 172.21.15.47:0/1841830705 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0254100e20 0x7f02540fc880 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:39.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.860+0000 7f02593ff700 1 -- 172.21.15.47:0/1841830705 >> 172.21.15.47:0/1841830705 conn(0x7f02540f8310 msgr2=0x7f02540ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:39.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.861+0000 7f02593ff700 1 -- 172.21.15.47:0/1841830705 shutdown_connections 2022-09-07T21:40:39.865 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:39.861+0000 7f02593ff700 1 -- 172.21.15.47:0/1841830705 wait complete. 2022-09-07T21:40:39.890 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-09-07T21:40:40.128 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.124+0000 7f9126dff700 1 -- 172.21.15.47:0/4087511303 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91200fddb0 msgr2=0x7f91200fe1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:40.128 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.124+0000 7f9126dff700 1 --2- 172.21.15.47:0/4087511303 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91200fddb0 0x7f91200fe1c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9114009810 tx=0x7f9114009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:40.129 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.124+0000 7f9126dff700 1 -- 172.21.15.47:0/4087511303 shutdown_connections 2022-09-07T21:40:40.129 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.124+0000 7f9126dff700 1 --2- 172.21.15.47:0/4087511303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91200ff940 0x7f9120106270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.129 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.124+0000 7f9126dff700 1 --2- 172.21.15.47:0/4087511303 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91200fddb0 0x7f91200fe1c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.129 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.124+0000 7f9126dff700 1 --2- 172.21.15.47:0/4087511303 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91200fef90 0x7f91200ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.130 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.124+0000 7f9126dff700 1 -- 172.21.15.47:0/4087511303 >> 172.21.15.47:0/4087511303 conn(0x7f91200f9460 msgr2=0x7f91200fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:40.130 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.124+0000 7f9126dff700 1 -- 172.21.15.47:0/4087511303 shutdown_connections 2022-09-07T21:40:40.130 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.124+0000 7f9126dff700 1 -- 172.21.15.47:0/4087511303 wait complete. 2022-09-07T21:40:40.132 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.126+0000 7f9126dff700 1 Processor -- start 2022-09-07T21:40:40.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.126+0000 7f9126dff700 1 -- start start 2022-09-07T21:40:40.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.126+0000 7f9126dff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91200fddb0 0x7f9120195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:40.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.126+0000 7f9126dff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91200fef90 0x7f9120196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:40.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.126+0000 7f9126dff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91200ff940 0x7f912019d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:40.133 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.126+0000 7f9126dff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912019d750 con 0x7f91200fddb0 2022-09-07T21:40:40.134 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.126+0000 7f9126dff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912019d8c0 con 0x7f91200ff940 2022-09-07T21:40:40.134 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.126+0000 7f9126dff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912019da80 con 0x7f91200fef90 2022-09-07T21:40:40.134 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.126+0000 7f9124b9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91200fddb0 0x7f9120195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:40.134 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.127+0000 7f912539c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91200ff940 0x7f912019d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:40.135 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.127+0000 7f911ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91200fef90 0x7f9120196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:40.135 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.127+0000 7f9124b9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91200fddb0 0x7f9120195bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33968/0 (socket says 172.21.15.47:33968) 2022-09-07T21:40:40.135 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.127+0000 7f9124b9b700 1 -- 172.21.15.47:0/4272354148 learned_addr learned my addr 172.21.15.47:0/4272354148 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:40.135 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.127+0000 7f9124b9b700 1 -- 172.21.15.47:0/4272354148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91200fef90 msgr2=0x7f9120196110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:40.135 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.127+0000 7f9124b9b700 1 --2- 172.21.15.47:0/4272354148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91200fef90 0x7f9120196110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.127+0000 7f9124b9b700 1 -- 172.21.15.47:0/4272354148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91200ff940 msgr2=0x7f912019d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:40.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.127+0000 7f9124b9b700 1 --2- 172.21.15.47:0/4272354148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91200ff940 0x7f912019d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.127+0000 7f9124b9b700 1 -- 172.21.15.47:0/4272354148 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9114009470 con 0x7f91200fddb0 2022-09-07T21:40:40.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.127+0000 7f9124b9b700 1 --2- 172.21.15.47:0/4272354148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91200fddb0 0x7f9120195bd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f91140059c0 tx=0x7f911400ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:40.136 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.128+0000 7f911dffb700 1 -- 172.21.15.47:0/4272354148 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911401d070 con 0x7f91200fddb0 2022-09-07T21:40:40.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.128+0000 7f911dffb700 1 -- 172.21.15.47:0/4272354148 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f91140187b0 con 0x7f91200fddb0 2022-09-07T21:40:40.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.128+0000 7f9126dff700 1 -- 172.21.15.47:0/4272354148 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912019dd00 con 0x7f91200fddb0 2022-09-07T21:40:40.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.128+0000 7f911dffb700 1 -- 172.21.15.47:0/4272354148 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114004840 con 0x7f91200fddb0 2022-09-07T21:40:40.137 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.128+0000 7f9126dff700 1 -- 172.21.15.47:0/4272354148 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f912019e1b0 con 0x7f91200fddb0 2022-09-07T21:40:40.138 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.129+0000 7f9126dff700 1 -- 172.21.15.47:0/4272354148 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9120109560 con 0x7f91200fddb0 2022-09-07T21:40:40.139 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.131+0000 7f911dffb700 1 -- 172.21.15.47:0/4272354148 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f9114029030 con 0x7f91200fddb0 2022-09-07T21:40:40.139 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.132+0000 7f911dffb700 1 --2- 172.21.15.47:0/4272354148 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9110099010 0x7f911009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:40.139 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.132+0000 7f911dffb700 1 -- 172.21.15.47:0/4272354148 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f91140c3800 con 0x7f91200fddb0 2022-09-07T21:40:40.140 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.132+0000 7f911ffff700 1 --2- 172.21.15.47:0/4272354148 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9110099010 0x7f911009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:40.140 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.133+0000 7f911ffff700 1 --2- 172.21.15.47:0/4272354148 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9110099010 0x7f911009b4c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f910c005fd0 tx=0x7f910c005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:40.140 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.136+0000 7f911dffb700 1 -- 172.21.15.47:0/4272354148 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f911408fb80 con 0x7f91200fddb0 2022-09-07T21:40:40.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:39 smithi047 ceph-mon[90087]: pgmap v377: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2022-09-07T21:40:40.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:39 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2226033300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:40:40.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:39 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3817166981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:40:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:39 smithi055 ceph-mon[97033]: pgmap v377: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2022-09-07T21:40:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:39 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2226033300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:40:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:39 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3817166981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:40:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:39 smithi059 ceph-mon[97450]: pgmap v377: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2022-09-07T21:40:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:39 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2226033300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:40:40.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:39 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3817166981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:40:40.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.337+0000 7f9126dff700 1 -- 172.21.15.47:0/4272354148 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f912019e760 con 0x7f91200fddb0 2022-09-07T21:40:40.341 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.338+0000 7f911dffb700 1 -- 172.21.15.47:0/4272354148 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f911405ae60 con 0x7f91200fddb0 2022-09-07T21:40:40.342 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2022-09-07T21:40:40.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.340+0000 7f9126dff700 1 -- 172.21.15.47:0/4272354148 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9110099010 msgr2=0x7f911009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:40.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.340+0000 7f9126dff700 1 --2- 172.21.15.47:0/4272354148 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9110099010 0x7f911009b4c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f910c005fd0 tx=0x7f910c005ea0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.340+0000 7f9126dff700 1 -- 172.21.15.47:0/4272354148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91200fddb0 msgr2=0x7f9120195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:40.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.340+0000 7f9126dff700 1 --2- 172.21.15.47:0/4272354148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91200fddb0 0x7f9120195bd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f91140059c0 tx=0x7f911400ff90 comp rx=0 tx=0).stop 2022-09-07T21:40:40.345 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.340+0000 7f9126dff700 1 -- 172.21.15.47:0/4272354148 shutdown_connections 2022-09-07T21:40:40.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.340+0000 7f9126dff700 1 --2- 172.21.15.47:0/4272354148 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9110099010 0x7f911009b4c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.340+0000 7f9126dff700 1 --2- 172.21.15.47:0/4272354148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91200ff940 0x7f912019d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.346 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.340+0000 7f9126dff700 1 --2- 172.21.15.47:0/4272354148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91200fddb0 0x7f9120195bd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.340+0000 7f9126dff700 1 --2- 172.21.15.47:0/4272354148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91200fef90 0x7f9120196110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.340+0000 7f9126dff700 1 -- 172.21.15.47:0/4272354148 >> 172.21.15.47:0/4272354148 conn(0x7f91200f9460 msgr2=0x7f91200fb770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:40.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.341+0000 7f9126dff700 1 -- 172.21.15.47:0/4272354148 shutdown_connections 2022-09-07T21:40:40.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.341+0000 7f9126dff700 1 -- 172.21.15.47:0/4272354148 wait complete. 2022-09-07T21:40:40.372 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2022-09-07T21:40:40.612 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.608+0000 7f1a03fff700 1 -- 172.21.15.47:0/730763701 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a040fe630 msgr2=0x7f1a04100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:40.613 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.608+0000 7f1a03fff700 1 --2- 172.21.15.47:0/730763701 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a040fe630 0x7f1a04100a10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009810 tx=0x7f19f4009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:40.613 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.609+0000 7f1a03fff700 1 -- 172.21.15.47:0/730763701 shutdown_connections 2022-09-07T21:40:40.614 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.609+0000 7f1a03fff700 1 --2- 172.21.15.47:0/730763701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a04103870 0x7f1a04105c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.614 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.609+0000 7f1a03fff700 1 --2- 172.21.15.47:0/730763701 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a040fe630 0x7f1a04100a10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.614 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.609+0000 7f1a03fff700 1 --2- 172.21.15.47:0/730763701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a04100f50 0x7f1a04103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.614 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.609+0000 7f1a03fff700 1 -- 172.21.15.47:0/730763701 >> 172.21.15.47:0/730763701 conn(0x7f1a040f80d0 msgr2=0x7f1a040fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:40.615 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.609+0000 7f1a03fff700 1 -- 172.21.15.47:0/730763701 shutdown_connections 2022-09-07T21:40:40.615 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.609+0000 7f1a03fff700 1 -- 172.21.15.47:0/730763701 wait complete. 2022-09-07T21:40:40.617 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.610+0000 7f1a03fff700 1 Processor -- start 2022-09-07T21:40:40.617 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.610+0000 7f1a03fff700 1 -- start start 2022-09-07T21:40:40.617 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.610+0000 7f1a03fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a040fe630 0x7f1a041958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:40.618 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.610+0000 7f1a03fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a04100f50 0x7f1a04195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:40.618 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.610+0000 7f1a03fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a04103870 0x7f1a0419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:40.618 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.610+0000 7f1a03fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0419d460 con 0x7f1a04103870 2022-09-07T21:40:40.619 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.610+0000 7f1a03fff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0419d5d0 con 0x7f1a04100f50 2022-09-07T21:40:40.619 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.610+0000 7f1a03fff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0419d790 con 0x7f1a040fe630 2022-09-07T21:40:40.619 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.611+0000 7f1a027fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a04100f50 0x7f1a04195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:40.619 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.611+0000 7f1a027fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a04100f50 0x7f1a04195e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:39926/0 (socket says 172.21.15.47:39926) 2022-09-07T21:40:40.620 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.611+0000 7f1a02ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a040fe630 0x7f1a041958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:40.620 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.611+0000 7f1a027fc700 1 -- 172.21.15.47:0/2644259497 learned_addr learned my addr 172.21.15.47:0/2644259497 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:40.620 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.611+0000 7f1a037fe700 1 --2- 172.21.15.47:0/2644259497 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a04103870 0x7f1a0419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:40.620 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.611+0000 7f1a027fc700 1 -- 172.21.15.47:0/2644259497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a040fe630 msgr2=0x7f1a041958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:40.620 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.611+0000 7f1a027fc700 1 --2- 172.21.15.47:0/2644259497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a040fe630 0x7f1a041958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.621 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.611+0000 7f1a027fc700 1 -- 172.21.15.47:0/2644259497 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a04103870 msgr2=0x7f1a0419ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:40.621 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.611+0000 7f1a027fc700 1 --2- 172.21.15.47:0/2644259497 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a04103870 0x7f1a0419ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.621 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.611+0000 7f1a027fc700 1 -- 172.21.15.47:0/2644259497 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f4009470 con 0x7f1a04100f50 2022-09-07T21:40:40.621 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.611+0000 7f1a02ffd700 1 --2- 172.21.15.47:0/2644259497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a040fe630 0x7f1a041958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:40.622 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.611+0000 7f1a027fc700 1 --2- 172.21.15.47:0/2644259497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a04100f50 0x7f1a04195e20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f19f800d700 tx=0x7f19f800da10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:40.622 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.612+0000 7f19ebfff700 1 -- 172.21.15.47:0/2644259497 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f800f890 con 0x7f1a04100f50 2022-09-07T21:40:40.622 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.612+0000 7f19ebfff700 1 -- 172.21.15.47:0/2644259497 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f19f800fe90 con 0x7f1a04100f50 2022-09-07T21:40:40.622 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.612+0000 7f1a03fff700 1 -- 172.21.15.47:0/2644259497 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0419da70 con 0x7f1a04100f50 2022-09-07T21:40:40.623 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.612+0000 7f19ebfff700 1 -- 172.21.15.47:0/2644259497 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f8019360 con 0x7f1a04100f50 2022-09-07T21:40:40.623 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.612+0000 7f1a03fff700 1 -- 172.21.15.47:0/2644259497 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0419dfd0 con 0x7f1a04100f50 2022-09-07T21:40:40.623 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.613+0000 7f1a03fff700 1 -- 172.21.15.47:0/2644259497 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a041092b0 con 0x7f1a04100f50 2022-09-07T21:40:40.624 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.615+0000 7f19ebfff700 1 -- 172.21.15.47:0/2644259497 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f19f8022070 con 0x7f1a04100f50 2022-09-07T21:40:40.625 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.616+0000 7f19ebfff700 1 --2- 172.21.15.47:0/2644259497 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f19ec099090 0x7f19ec09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:40.625 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.616+0000 7f19ebfff700 1 -- 172.21.15.47:0/2644259497 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f19f80c2350 con 0x7f1a04100f50 2022-09-07T21:40:40.625 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.617+0000 7f1a02ffd700 1 --2- 172.21.15.47:0/2644259497 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f19ec099090 0x7f19ec09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:40.625 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.617+0000 7f1a02ffd700 1 --2- 172.21.15.47:0/2644259497 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f19ec099090 0x7f19ec09b540 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f19f400b4f0 tx=0x7f19f4009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:40.626 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.618+0000 7f19ebfff700 1 -- 172.21.15.47:0/2644259497 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f19f808e6d0 con 0x7f1a04100f50 2022-09-07T21:40:40.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.821+0000 7f1a03fff700 1 -- 172.21.15.47:0/2644259497 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1a04109440 con 0x7f1a04100f50 2022-09-07T21:40:40.837 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.834+0000 7f19ebfff700 1 -- 172.21.15.47:0/2644259497 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f19f8020020 con 0x7f1a04100f50 2022-09-07T21:40:40.838 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2022-09-07T21:40:40.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.837+0000 7f1a03fff700 1 -- 172.21.15.47:0/2644259497 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f19ec099090 msgr2=0x7f19ec09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:40.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.837+0000 7f1a03fff700 1 --2- 172.21.15.47:0/2644259497 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f19ec099090 0x7f19ec09b540 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f19f400b4f0 tx=0x7f19f4009eb0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.837+0000 7f1a03fff700 1 -- 172.21.15.47:0/2644259497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a04100f50 msgr2=0x7f1a04195e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:40.841 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.837+0000 7f1a03fff700 1 --2- 172.21.15.47:0/2644259497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a04100f50 0x7f1a04195e20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f19f800d700 tx=0x7f19f800da10 comp rx=0 tx=0).stop 2022-09-07T21:40:40.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.837+0000 7f1a03fff700 1 -- 172.21.15.47:0/2644259497 shutdown_connections 2022-09-07T21:40:40.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.837+0000 7f1a03fff700 1 --2- 172.21.15.47:0/2644259497 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f19ec099090 0x7f19ec09b540 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.837+0000 7f1a03fff700 1 --2- 172.21.15.47:0/2644259497 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a04103870 0x7f1a0419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.842 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.837+0000 7f1a03fff700 1 --2- 172.21.15.47:0/2644259497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a040fe630 0x7f1a041958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.837+0000 7f1a03fff700 1 --2- 172.21.15.47:0/2644259497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a04100f50 0x7f1a04195e20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:40.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.837+0000 7f1a03fff700 1 -- 172.21.15.47:0/2644259497 >> 172.21.15.47:0/2644259497 conn(0x7f1a040f80d0 msgr2=0x7f1a040ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:40.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.837+0000 7f1a03fff700 1 -- 172.21.15.47:0/2644259497 shutdown_connections 2022-09-07T21:40:40.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:40.837+0000 7f1a03fff700 1 -- 172.21.15.47:0/2644259497 wait complete. 2022-09-07T21:40:40.868 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:40:41.101 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.097+0000 7fd3f27ff700 1 -- 172.21.15.47:0/3784744183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3ec066be0 msgr2=0x7fd3ec1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:41.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.097+0000 7fd3f27ff700 1 --2- 172.21.15.47:0/3784744183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3ec066be0 0x7fd3ec1008b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009810 tx=0x7fd3dc009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:41.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.098+0000 7fd3f27ff700 1 -- 172.21.15.47:0/3784744183 shutdown_connections 2022-09-07T21:40:41.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.098+0000 7fd3f27ff700 1 --2- 172.21.15.47:0/3784744183 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3ec1038e0 0x7fd3ec105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.098+0000 7fd3f27ff700 1 --2- 172.21.15.47:0/3784744183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3ec066be0 0x7fd3ec1008b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.098+0000 7fd3f27ff700 1 --2- 172.21.15.47:0/3784744183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3ec100df0 0x7fd3ec103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.098+0000 7fd3f27ff700 1 -- 172.21.15.47:0/3784744183 >> 172.21.15.47:0/3784744183 conn(0x7fd3ec0f82e0 msgr2=0x7fd3ec0fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:41.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.098+0000 7fd3f27ff700 1 -- 172.21.15.47:0/3784744183 shutdown_connections 2022-09-07T21:40:41.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.098+0000 7fd3f27ff700 1 -- 172.21.15.47:0/3784744183 wait complete. 2022-09-07T21:40:41.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.099+0000 7fd3f27ff700 1 Processor -- start 2022-09-07T21:40:41.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.099+0000 7fd3f27ff700 1 -- start start 2022-09-07T21:40:41.107 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.099+0000 7fd3f27ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3ec066be0 0x7fd3ec195af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:41.107 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.099+0000 7fd3f27ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3ec100df0 0x7fd3ec196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:41.107 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.099+0000 7fd3f27ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3ec1038e0 0x7fd3ec19d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:41.107 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.099+0000 7fd3f27ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec19d670 con 0x7fd3ec1038e0 2022-09-07T21:40:41.108 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.099+0000 7fd3f27ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec19d7e0 con 0x7fd3ec066be0 2022-09-07T21:40:41.108 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.099+0000 7fd3f27ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec19d9a0 con 0x7fd3ec100df0 2022-09-07T21:40:41.108 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.100+0000 7fd3ebfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3ec066be0 0x7fd3ec195af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:41.109 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.100+0000 7fd3eb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3ec100df0 0x7fd3ec196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:41.109 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.100+0000 7fd3ebfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3ec066be0 0x7fd3ec195af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:39936/0 (socket says 172.21.15.47:39936) 2022-09-07T21:40:41.109 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.100+0000 7fd3ebfff700 1 -- 172.21.15.47:0/2782023368 learned_addr learned my addr 172.21.15.47:0/2782023368 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:41.109 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.100+0000 7fd3f0d9c700 1 --2- 172.21.15.47:0/2782023368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3ec1038e0 0x7fd3ec19d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:41.109 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.100+0000 7fd3eb7fe700 1 -- 172.21.15.47:0/2782023368 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3ec066be0 msgr2=0x7fd3ec195af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:41.110 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.100+0000 7fd3eb7fe700 1 --2- 172.21.15.47:0/2782023368 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3ec066be0 0x7fd3ec195af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.110 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.100+0000 7fd3eb7fe700 1 -- 172.21.15.47:0/2782023368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3ec1038e0 msgr2=0x7fd3ec19d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:41.110 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.100+0000 7fd3eb7fe700 1 --2- 172.21.15.47:0/2782023368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3ec1038e0 0x7fd3ec19d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.110 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.100+0000 7fd3eb7fe700 1 -- 172.21.15.47:0/2782023368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e00093b0 con 0x7fd3ec100df0 2022-09-07T21:40:41.111 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.100+0000 7fd3ebfff700 1 --2- 172.21.15.47:0/2782023368 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3ec066be0 0x7fd3ec195af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:41.111 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.101+0000 7fd3eb7fe700 1 --2- 172.21.15.47:0/2782023368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3ec100df0 0x7fd3ec196030 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000e9c0 tx=0x7fd3e000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:41.111 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.101+0000 7fd3e97fa700 1 -- 172.21.15.47:0/2782023368 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0004220 con 0x7fd3ec100df0 2022-09-07T21:40:41.111 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.101+0000 7fd3f27ff700 1 -- 172.21.15.47:0/2782023368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3dc009470 con 0x7fd3ec100df0 2022-09-07T21:40:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.101+0000 7fd3f27ff700 1 -- 172.21.15.47:0/2782023368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ec19e050 con 0x7fd3ec100df0 2022-09-07T21:40:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.101+0000 7fd3e97fa700 1 -- 172.21.15.47:0/2782023368 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd3e0004380 con 0x7fd3ec100df0 2022-09-07T21:40:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.101+0000 7fd3e97fa700 1 -- 172.21.15.47:0/2782023368 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0003d00 con 0x7fd3ec100df0 2022-09-07T21:40:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.102+0000 7fd3f27ff700 1 -- 172.21.15.47:0/2782023368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d00051a0 con 0x7fd3ec100df0 2022-09-07T21:40:41.114 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.104+0000 7fd3e97fa700 1 -- 172.21.15.47:0/2782023368 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fd3e001e020 con 0x7fd3ec100df0 2022-09-07T21:40:41.114 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.105+0000 7fd3e97fa700 1 --2- 172.21.15.47:0/2782023368 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd3d4099080 0x7fd3d409b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:41.114 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.106+0000 7fd3e97fa700 1 -- 172.21.15.47:0/2782023368 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fd3e0014070 con 0x7fd3ec100df0 2022-09-07T21:40:41.114 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.106+0000 7fd3ebfff700 1 --2- 172.21.15.47:0/2782023368 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd3d4099080 0x7fd3d409b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:41.115 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.106+0000 7fd3ebfff700 1 --2- 172.21.15.47:0/2782023368 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd3d4099080 0x7fd3d409b530 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00b4f0 tx=0x7fd3dc009e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:41.115 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.108+0000 7fd3e97fa700 1 -- 172.21.15.47:0/2782023368 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fd3e008f3c0 con 0x7fd3ec100df0 2022-09-07T21:40:41.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:40 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1841830705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:40:41.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:40 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/4272354148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:40:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:40 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1841830705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:40:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:40 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/4272354148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:40:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:40 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1841830705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:40:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:40 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/4272354148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:40:41.360 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.357+0000 7fd3f27ff700 1 -- 172.21.15.47:0/2782023368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3d0005010 con 0x7fd3ec100df0 2022-09-07T21:40:41.361 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.358+0000 7fd3e97fa700 1 -- 172.21.15.47:0/2782023368 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10707 (secure 0 0 0) 0x7fd3e0003e60 con 0x7fd3ec100df0 2022-09-07T21:40:41.362 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:40:41.362 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/2559266585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2559266585},{"type":"v1","addr":"172.21.15.55:6835","nonce":2559266585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/2090942297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2090942297},{"type":"v1","addr":"172.21.15.55:6837","nonce":2090942297}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-09-07T21:38:40.544389+0000","modified":"2022-09-07T21:39:06.833800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24586,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6835/2445409240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2445409240},{"type":"v1","addr":"172.21.15.59:6835","nonce":2445409240}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/3542466465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":3542466465},{"type":"v1","addr":"172.21.15.59:6837","nonce":3542466465}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6839/315505812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":315505812},{"type":"v1","addr":"172.21.15.47:6839","nonce":315505812}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.59:6839/1006724740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":1006724740},{"type":"v1","addr":"172.21.15.59:6839","nonce":1006724740}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6839/517990300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":517990300},{"type":"v1","addr":"172.21.15.55:6839","nonce":517990300}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24586":{"gid":24586,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6841/793863162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6840","nonce":793863162},{"type":"v1","addr":"172.21.15.47:6841","nonce":793863162}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-09-07T21:40:41.364 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.360+0000 7fd3f27ff700 1 -- 172.21.15.47:0/2782023368 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd3d4099080 msgr2=0x7fd3d409b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:41.364 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.360+0000 7fd3f27ff700 1 --2- 172.21.15.47:0/2782023368 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd3d4099080 0x7fd3d409b530 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00b4f0 tx=0x7fd3dc009e70 comp rx=0 tx=0).stop 2022-09-07T21:40:41.365 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.360+0000 7fd3f27ff700 1 -- 172.21.15.47:0/2782023368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3ec100df0 msgr2=0x7fd3ec196030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:41.365 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.360+0000 7fd3f27ff700 1 --2- 172.21.15.47:0/2782023368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3ec100df0 0x7fd3ec196030 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000e9c0 tx=0x7fd3e000ecd0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.365 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.360+0000 7fd3f27ff700 1 -- 172.21.15.47:0/2782023368 shutdown_connections 2022-09-07T21:40:41.365 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.360+0000 7fd3f27ff700 1 --2- 172.21.15.47:0/2782023368 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd3d4099080 0x7fd3d409b530 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.366 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.361+0000 7fd3f27ff700 1 --2- 172.21.15.47:0/2782023368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3ec1038e0 0x7fd3ec19d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.366 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.361+0000 7fd3f27ff700 1 --2- 172.21.15.47:0/2782023368 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3ec066be0 0x7fd3ec195af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.366 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.361+0000 7fd3f27ff700 1 --2- 172.21.15.47:0/2782023368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3ec100df0 0x7fd3ec196030 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.366 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.361+0000 7fd3f27ff700 1 -- 172.21.15.47:0/2782023368 >> 172.21.15.47:0/2782023368 conn(0x7fd3ec0f82e0 msgr2=0x7fd3ec0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:41.367 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.361+0000 7fd3f27ff700 1 -- 172.21.15.47:0/2782023368 shutdown_connections 2022-09-07T21:40:41.367 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.361+0000 7fd3f27ff700 1 -- 172.21.15.47:0/2782023368 wait complete. 2022-09-07T21:40:41.368 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 24 2022-09-07T21:40:41.392 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-07T21:40:41.624 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.621+0000 7f115599d700 1 -- 172.21.15.47:0/2278106357 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11500716b0 msgr2=0x7f1150071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:41.625 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.621+0000 7f115599d700 1 --2- 172.21.15.47:0/2278106357 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11500716b0 0x7f1150071b20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f113c009810 tx=0x7f113c009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:41.625 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.621+0000 7f115599d700 1 -- 172.21.15.47:0/2278106357 shutdown_connections 2022-09-07T21:40:41.625 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.621+0000 7f115599d700 1 --2- 172.21.15.47:0/2278106357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1150101610 0x7f115010c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.626 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.621+0000 7f115599d700 1 --2- 172.21.15.47:0/2278106357 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1150072bf0 0x7f1150071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.626 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.621+0000 7f115599d700 1 --2- 172.21.15.47:0/2278106357 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11500716b0 0x7f1150071b20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.626 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.621+0000 7f115599d700 1 -- 172.21.15.47:0/2278106357 >> 172.21.15.47:0/2278106357 conn(0x7f11500f91c0 msgr2=0x7f11500fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:41.626 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.621+0000 7f115599d700 1 -- 172.21.15.47:0/2278106357 shutdown_connections 2022-09-07T21:40:41.627 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.621+0000 7f115599d700 1 -- 172.21.15.47:0/2278106357 wait complete. 2022-09-07T21:40:41.629 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.622+0000 7f115599d700 1 Processor -- start 2022-09-07T21:40:41.629 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.622+0000 7f115599d700 1 -- start start 2022-09-07T21:40:41.629 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.623+0000 7f115599d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11500716b0 0x7f1150199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:41.630 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.623+0000 7f115599d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1150072bf0 0x7f115019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:41.630 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.623+0000 7f115599d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1150101610 0x7f11501a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:41.630 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.623+0000 7f115599d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11501a18f0 con 0x7f1150101610 2022-09-07T21:40:41.630 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.623+0000 7f115599d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11501a1a60 con 0x7f11500716b0 2022-09-07T21:40:41.631 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.623+0000 7f115599d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11501a1c20 con 0x7f1150072bf0 2022-09-07T21:40:41.631 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.623+0000 7f115519c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1150101610 0x7f11501a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:41.631 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.623+0000 7f114ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1150072bf0 0x7f115019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:41.631 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.623+0000 7f115499b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11500716b0 0x7f1150199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:41.632 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.623+0000 7f114ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1150072bf0 0x7f115019a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:56382/0 (socket says 172.21.15.47:56382) 2022-09-07T21:40:41.632 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.623+0000 7f114ffff700 1 -- 172.21.15.47:0/3358406407 learned_addr learned my addr 172.21.15.47:0/3358406407 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:41.632 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.623+0000 7f115499b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11500716b0 0x7f1150199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:39956/0 (socket says 172.21.15.47:39956) 2022-09-07T21:40:41.632 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.624+0000 7f115499b700 1 -- 172.21.15.47:0/3358406407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1150072bf0 msgr2=0x7f115019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:41.633 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.624+0000 7f115499b700 1 --2- 172.21.15.47:0/3358406407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1150072bf0 0x7f115019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.633 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.624+0000 7f115499b700 1 -- 172.21.15.47:0/3358406407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1150101610 msgr2=0x7f11501a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:41.633 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.624+0000 7f115499b700 1 --2- 172.21.15.47:0/3358406407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1150101610 0x7f11501a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.633 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.624+0000 7f115499b700 1 -- 172.21.15.47:0/3358406407 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c009470 con 0x7f11500716b0 2022-09-07T21:40:41.633 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.624+0000 7f114ffff700 1 --2- 172.21.15.47:0/3358406407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1150072bf0 0x7f115019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:41.634 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.624+0000 7f115519c700 1 --2- 172.21.15.47:0/3358406407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1150101610 0x7f11501a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:41.634 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.624+0000 7f115499b700 1 --2- 172.21.15.47:0/3358406407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11500716b0 0x7f1150199d30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1144005fd0 tx=0x7f114400e9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:41.634 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.624+0000 7f114dffb700 1 -- 172.21.15.47:0/3358406407 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1144004220 con 0x7f11500716b0 2022-09-07T21:40:41.634 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.624+0000 7f114dffb700 1 -- 172.21.15.47:0/3358406407 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1144004380 con 0x7f11500716b0 2022-09-07T21:40:41.635 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.624+0000 7f114dffb700 1 -- 172.21.15.47:0/3358406407 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1144003b60 con 0x7f11500716b0 2022-09-07T21:40:41.635 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.624+0000 7f115599d700 1 -- 172.21.15.47:0/3358406407 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11501a1f00 con 0x7f11500716b0 2022-09-07T21:40:41.635 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.624+0000 7f115599d700 1 -- 172.21.15.47:0/3358406407 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11501a22d0 con 0x7f11500716b0 2022-09-07T21:40:41.635 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.625+0000 7f115599d700 1 -- 172.21.15.47:0/3358406407 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f115010f900 con 0x7f11500716b0 2022-09-07T21:40:41.636 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.627+0000 7f114dffb700 1 -- 172.21.15.47:0/3358406407 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f114401e020 con 0x7f11500716b0 2022-09-07T21:40:41.637 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.629+0000 7f114dffb700 1 --2- 172.21.15.47:0/3358406407 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1140099010 0x7f114009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:41.637 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.629+0000 7f114dffb700 1 -- 172.21.15.47:0/3358406407 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f1144014070 con 0x7f11500716b0 2022-09-07T21:40:41.637 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.629+0000 7f114ffff700 1 --2- 172.21.15.47:0/3358406407 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1140099010 0x7f114009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:41.638 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.630+0000 7f114ffff700 1 --2- 172.21.15.47:0/3358406407 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1140099010 0x7f114009b4c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f113c00b4f0 tx=0x7f113c009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:41.638 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.631+0000 7f114dffb700 1 -- 172.21.15.47:0/3358406407 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f114408ed10 con 0x7f11500716b0 2022-09-07T21:40:41.843 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.840+0000 7f115599d700 1 -- 172.21.15.47:0/3358406407 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11500ffbe0 con 0x7f11500716b0 2022-09-07T21:40:41.845 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.842+0000 7f114dffb700 1 -- 172.21.15.47:0/3358406407 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+23592 (secure 0 0 0) 0x7f114400eea0 con 0x7f11500716b0 2022-09-07T21:40:41.846 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:40:41.846 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":91,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:39:47.780661+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-07T21:38:38.608937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-07T21:38:39.936252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-07T21:38:43.731797+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.18","mappings":[{"from":5,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:40:41.848 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.844+0000 7f115599d700 1 -- 172.21.15.47:0/3358406407 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1140099010 msgr2=0x7f114009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:41.848 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.844+0000 7f115599d700 1 --2- 172.21.15.47:0/3358406407 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1140099010 0x7f114009b4c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f113c00b4f0 tx=0x7f113c009eb0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.849 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.844+0000 7f115599d700 1 -- 172.21.15.47:0/3358406407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11500716b0 msgr2=0x7f1150199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:41.849 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.844+0000 7f115599d700 1 --2- 172.21.15.47:0/3358406407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11500716b0 0x7f1150199d30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1144005fd0 tx=0x7f114400e9a0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.844+0000 7f115599d700 1 -- 172.21.15.47:0/3358406407 shutdown_connections 2022-09-07T21:40:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.844+0000 7f115599d700 1 --2- 172.21.15.47:0/3358406407 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1140099010 0x7f114009b4c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.844+0000 7f115599d700 1 --2- 172.21.15.47:0/3358406407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1150101610 0x7f11501a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.844+0000 7f115599d700 1 --2- 172.21.15.47:0/3358406407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11500716b0 0x7f1150199d30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.844+0000 7f115599d700 1 --2- 172.21.15.47:0/3358406407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1150072bf0 0x7f115019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:41.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.844+0000 7f115599d700 1 -- 172.21.15.47:0/3358406407 >> 172.21.15.47:0/3358406407 conn(0x7f11500f91c0 msgr2=0x7f11500fb5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:41.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.844+0000 7f115599d700 1 -- 172.21.15.47:0/3358406407 shutdown_connections 2022-09-07T21:40:41.851 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:41.844+0000 7f115599d700 1 -- 172.21.15.47:0/3358406407 wait complete. 2022-09-07T21:40:41.875 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-07T21:33:29.342534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-09-07T21:38:38.608937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-09-07T21:38:39.936252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2022-09-07T21:38:43.731797+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-09-07T21:40:41.876 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-09-07T21:40:42.114 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.110+0000 7f8387fff700 1 -- 172.21.15.47:0/2068964181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f83880fdb60 msgr2=0x7f83880fdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:42.115 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.110+0000 7f8387fff700 1 --2- 172.21.15.47:0/2068964181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f83880fdb60 0x7f83880fdf70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8378009750 tx=0x7f8378009a60 comp rx=0 tx=0).stop 2022-09-07T21:40:42.115 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.111+0000 7f8387fff700 1 -- 172.21.15.47:0/2068964181 shutdown_connections 2022-09-07T21:40:42.115 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.111+0000 7f8387fff700 1 --2- 172.21.15.47:0/2068964181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83880ff530 0x7f83881060d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.115 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.111+0000 7f8387fff700 1 --2- 172.21.15.47:0/2068964181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f83880fdb60 0x7f83880fdf70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.116 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.111+0000 7f8387fff700 1 --2- 172.21.15.47:0/2068964181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880fe4b0 0x7f83880feea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.116 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.111+0000 7f8387fff700 1 -- 172.21.15.47:0/2068964181 >> 172.21.15.47:0/2068964181 conn(0x7f83880f9230 msgr2=0x7f83880fb680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:42.116 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.111+0000 7f8387fff700 1 -- 172.21.15.47:0/2068964181 shutdown_connections 2022-09-07T21:40:42.116 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.111+0000 7f8387fff700 1 -- 172.21.15.47:0/2068964181 wait complete. 2022-09-07T21:40:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.112+0000 7f8387fff700 1 Processor -- start 2022-09-07T21:40:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.113+0000 7f8387fff700 1 -- start start 2022-09-07T21:40:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.113+0000 7f8387fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880fdb60 0x7f8388072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.113+0000 7f8387fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83880fe4b0 0x7f83880710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:42.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.113+0000 7f8387fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f83880ff530 0x7f8388071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:42.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.113+0000 7f8387fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8388071e10 con 0x7f83880fe4b0 2022-09-07T21:40:42.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.113+0000 7f8387fff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8388071f80 con 0x7f83880ff530 2022-09-07T21:40:42.120 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.113+0000 7f8387fff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838819d800 con 0x7f83880fdb60 2022-09-07T21:40:42.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.114+0000 7f83867fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83880fe4b0 0x7f83880710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:42.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.114+0000 7f8386ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880fdb60 0x7f8388072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:42.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.114+0000 7f83867fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83880fe4b0 0x7f83880710d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34044/0 (socket says 172.21.15.47:34044) 2022-09-07T21:40:42.121 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.114+0000 7f8386ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880fdb60 0x7f8388072a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:56404/0 (socket says 172.21.15.47:56404) 2022-09-07T21:40:42.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.114+0000 7f83867fc700 1 -- 172.21.15.47:0/2181872456 learned_addr learned my addr 172.21.15.47:0/2181872456 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:42.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.114+0000 7f83877fe700 1 --2- 172.21.15.47:0/2181872456 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f83880ff530 0x7f8388071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:42.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.114+0000 7f8386ffd700 1 -- 172.21.15.47:0/2181872456 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f83880ff530 msgr2=0x7f8388071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:42.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.114+0000 7f8386ffd700 1 --2- 172.21.15.47:0/2181872456 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f83880ff530 0x7f8388071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.122 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.114+0000 7f8386ffd700 1 -- 172.21.15.47:0/2181872456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83880fe4b0 msgr2=0x7f83880710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:42.123 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.114+0000 7f8386ffd700 1 --2- 172.21.15.47:0/2181872456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83880fe4b0 0x7f83880710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.123 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.114+0000 7f8386ffd700 1 -- 172.21.15.47:0/2181872456 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83780093b0 con 0x7f83880fdb60 2022-09-07T21:40:42.123 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.114+0000 7f83877fe700 1 --2- 172.21.15.47:0/2181872456 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f83880ff530 0x7f8388071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:42.123 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.114+0000 7f8386ffd700 1 --2- 172.21.15.47:0/2181872456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880fdb60 0x7f8388072a90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f83780096a0 tx=0x7f8378004230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:42.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.115+0000 7f836ffff700 1 -- 172.21.15.47:0/2181872456 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837801d070 con 0x7f83880fdb60 2022-09-07T21:40:42.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.115+0000 7f8387fff700 1 -- 172.21.15.47:0/2181872456 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838819da80 con 0x7f83880fdb60 2022-09-07T21:40:42.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.115+0000 7f836ffff700 1 -- 172.21.15.47:0/2181872456 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8378005590 con 0x7f83880fdb60 2022-09-07T21:40:42.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.115+0000 7f8387fff700 1 -- 172.21.15.47:0/2181872456 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838819df20 con 0x7f83880fdb60 2022-09-07T21:40:42.124 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.115+0000 7f836ffff700 1 -- 172.21.15.47:0/2181872456 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378017a50 con 0x7f83880fdb60 2022-09-07T21:40:42.125 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.116+0000 7f8387fff700 1 -- 172.21.15.47:0/2181872456 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8388109350 con 0x7f83880fdb60 2022-09-07T21:40:42.126 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.118+0000 7f836ffff700 1 -- 172.21.15.47:0/2181872456 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f837800c040 con 0x7f83880fdb60 2022-09-07T21:40:42.126 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.119+0000 7f836ffff700 1 --2- 172.21.15.47:0/2181872456 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8370099010 0x7f837009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:42.126 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.119+0000 7f836ffff700 1 -- 172.21.15.47:0/2181872456 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f83780c32b0 con 0x7f83880fdb60 2022-09-07T21:40:42.126 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.120+0000 7f83867fc700 1 --2- 172.21.15.47:0/2181872456 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8370099010 0x7f837009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:42.127 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.120+0000 7f83867fc700 1 --2- 172.21.15.47:0/2181872456 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8370099010 0x7f837009b4c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f837c009b80 tx=0x7f837c005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:42.127 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.123+0000 7f836ffff700 1 -- 172.21.15.47:0/2181872456 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f837808f630 con 0x7f83880fdb60 2022-09-07T21:40:42.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:41 smithi047 ceph-mon[90087]: pgmap v378: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:40:42.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:41 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2644259497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:40:42.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:41 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2782023368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:40:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:41 smithi055 ceph-mon[97033]: pgmap v378: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:40:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:41 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2644259497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:40:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:41 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2782023368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:40:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:41 smithi059 ceph-mon[97450]: pgmap v378: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:40:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:41 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2644259497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:40:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:41 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2782023368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:40:42.326 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.323+0000 7f8387fff700 1 -- 172.21.15.47:0/2181872456 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f838819e590 con 0x7f83880fdb60 2022-09-07T21:40:42.327 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.324+0000 7f836ffff700 1 -- 172.21.15.47:0/2181872456 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8378092c50 con 0x7f83880fdb60 2022-09-07T21:40:42.328 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2022-09-07T21:40:42.330 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.326+0000 7f8387fff700 1 -- 172.21.15.47:0/2181872456 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8370099010 msgr2=0x7f837009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:42.330 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.326+0000 7f8387fff700 1 --2- 172.21.15.47:0/2181872456 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8370099010 0x7f837009b4c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f837c009b80 tx=0x7f837c005f60 comp rx=0 tx=0).stop 2022-09-07T21:40:42.331 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.326+0000 7f8387fff700 1 -- 172.21.15.47:0/2181872456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880fdb60 msgr2=0x7f8388072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:42.331 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.326+0000 7f8387fff700 1 --2- 172.21.15.47:0/2181872456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880fdb60 0x7f8388072a90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f83780096a0 tx=0x7f8378004230 comp rx=0 tx=0).stop 2022-09-07T21:40:42.331 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.326+0000 7f8387fff700 1 -- 172.21.15.47:0/2181872456 shutdown_connections 2022-09-07T21:40:42.331 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.327+0000 7f8387fff700 1 --2- 172.21.15.47:0/2181872456 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f8370099010 0x7f837009b4c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.332 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.327+0000 7f8387fff700 1 --2- 172.21.15.47:0/2181872456 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f83880ff530 0x7f8388071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.332 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.327+0000 7f8387fff700 1 --2- 172.21.15.47:0/2181872456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880fdb60 0x7f8388072a90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.332 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.327+0000 7f8387fff700 1 --2- 172.21.15.47:0/2181872456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83880fe4b0 0x7f83880710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.332 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.327+0000 7f8387fff700 1 -- 172.21.15.47:0/2181872456 >> 172.21.15.47:0/2181872456 conn(0x7f83880f9230 msgr2=0x7f83880fb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:42.332 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.327+0000 7f8387fff700 1 -- 172.21.15.47:0/2181872456 shutdown_connections 2022-09-07T21:40:42.333 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.327+0000 7f8387fff700 1 -- 172.21.15.47:0/2181872456 wait complete. 2022-09-07T21:40:42.357 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-09-07T21:40:42.593 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.589+0000 7fda035ff700 1 -- 172.21.15.47:0/2385027304 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9fc103990 msgr2=0x7fd9fc105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:42.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.589+0000 7fda035ff700 1 --2- 172.21.15.47:0/2385027304 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9fc103990 0x7fd9fc105e10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4009b00 tx=0x7fd9f4009e10 comp rx=0 tx=0).stop 2022-09-07T21:40:42.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.590+0000 7fda035ff700 1 -- 172.21.15.47:0/2385027304 shutdown_connections 2022-09-07T21:40:42.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.590+0000 7fda035ff700 1 --2- 172.21.15.47:0/2385027304 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9fc103990 0x7fd9fc105e10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.595 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.590+0000 7fda035ff700 1 --2- 172.21.15.47:0/2385027304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9fc066be0 0x7fd9fc100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.595 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.590+0000 7fda035ff700 1 --2- 172.21.15.47:0/2385027304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9fc100ea0 0x7fd9fc103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.595 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.590+0000 7fda035ff700 1 -- 172.21.15.47:0/2385027304 >> 172.21.15.47:0/2385027304 conn(0x7fd9fc0f8370 msgr2=0x7fd9fc0fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:42.595 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.590+0000 7fda035ff700 1 -- 172.21.15.47:0/2385027304 shutdown_connections 2022-09-07T21:40:42.596 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.590+0000 7fda035ff700 1 -- 172.21.15.47:0/2385027304 wait complete. 2022-09-07T21:40:42.598 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.591+0000 7fda035ff700 1 Processor -- start 2022-09-07T21:40:42.598 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.591+0000 7fda035ff700 1 -- start start 2022-09-07T21:40:42.598 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.591+0000 7fda035ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9fc066be0 0x7fd9fc195be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:42.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.591+0000 7fda035ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9fc100ea0 0x7fd9fc196120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:42.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.591+0000 7fda035ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9fc103990 0x7fd9fc19d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:42.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.591+0000 7fda035ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9fc19d760 con 0x7fd9fc103990 2022-09-07T21:40:42.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.591+0000 7fda035ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9fc19d8d0 con 0x7fd9fc100ea0 2022-09-07T21:40:42.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.591+0000 7fda035ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9fc19da90 con 0x7fd9fc066be0 2022-09-07T21:40:42.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.592+0000 7fda01b9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9fc103990 0x7fd9fc19d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:42.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.592+0000 7fda00b9a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9fc100ea0 0x7fd9fc196120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:42.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.592+0000 7fda01b9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9fc103990 0x7fd9fc19d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34054/0 (socket says 172.21.15.47:34054) 2022-09-07T21:40:42.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.592+0000 7fda01b9c700 1 -- 172.21.15.47:0/4185960024 learned_addr learned my addr 172.21.15.47:0/4185960024 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:42.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.592+0000 7fda0139b700 1 --2- 172.21.15.47:0/4185960024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9fc066be0 0x7fd9fc195be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:42.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.592+0000 7fda00b9a700 1 -- 172.21.15.47:0/4185960024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9fc066be0 msgr2=0x7fd9fc195be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:42.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.592+0000 7fda00b9a700 1 --2- 172.21.15.47:0/4185960024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9fc066be0 0x7fd9fc195be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.592+0000 7fda00b9a700 1 -- 172.21.15.47:0/4185960024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9fc103990 msgr2=0x7fd9fc19d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:42.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.592+0000 7fda00b9a700 1 --2- 172.21.15.47:0/4185960024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9fc103990 0x7fd9fc19d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.592+0000 7fda00b9a700 1 -- 172.21.15.47:0/4185960024 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f4009730 con 0x7fd9fc100ea0 2022-09-07T21:40:42.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.592+0000 7fda01b9c700 1 --2- 172.21.15.47:0/4185960024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9fc103990 0x7fd9fc19d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:42.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.592+0000 7fda00b9a700 1 --2- 172.21.15.47:0/4185960024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9fc100ea0 0x7fd9fc196120 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00e6c0 tx=0x7fd9ec00e9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:42.603 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.593+0000 7fd9f27fc700 1 -- 172.21.15.47:0/4185960024 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec010840 con 0x7fd9fc100ea0 2022-09-07T21:40:42.603 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.593+0000 7fd9f27fc700 1 -- 172.21.15.47:0/4185960024 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd9ec010e40 con 0x7fd9fc100ea0 2022-09-07T21:40:42.603 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.593+0000 7fda035ff700 1 -- 172.21.15.47:0/4185960024 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9fc19dd70 con 0x7fd9fc100ea0 2022-09-07T21:40:42.603 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.593+0000 7fd9f27fc700 1 -- 172.21.15.47:0/4185960024 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec00f420 con 0x7fd9fc100ea0 2022-09-07T21:40:42.604 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.593+0000 7fda035ff700 1 -- 172.21.15.47:0/4185960024 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9fc19e220 con 0x7fd9fc100ea0 2022-09-07T21:40:42.604 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.594+0000 7fda035ff700 1 -- 172.21.15.47:0/4185960024 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9fc1095a0 con 0x7fd9fc100ea0 2022-09-07T21:40:42.605 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.596+0000 7fd9f27fc700 1 -- 172.21.15.47:0/4185960024 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fd9ec00c040 con 0x7fd9fc100ea0 2022-09-07T21:40:42.605 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.597+0000 7fd9f27fc700 1 --2- 172.21.15.47:0/4185960024 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd9e8099010 0x7fd9e809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:42.605 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.597+0000 7fd9f27fc700 1 -- 172.21.15.47:0/4185960024 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fd9ec014070 con 0x7fd9fc100ea0 2022-09-07T21:40:42.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.598+0000 7fda0139b700 1 --2- 172.21.15.47:0/4185960024 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd9e8099010 0x7fd9e809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:42.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.598+0000 7fda0139b700 1 --2- 172.21.15.47:0/4185960024 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd9e8099010 0x7fd9e809b4c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc066fb0 tx=0x7fd9f8005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:42.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.599+0000 7fd9f27fc700 1 -- 172.21.15.47:0/4185960024 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fd9ec08efe0 con 0x7fd9fc100ea0 2022-09-07T21:40:42.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.802+0000 7fda035ff700 1 -- 172.21.15.47:0/4185960024 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd9fc19e830 con 0x7fd9fc100ea0 2022-09-07T21:40:42.806 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.803+0000 7fd9f27fc700 1 -- 172.21.15.47:0/4185960024 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd9ec092600 con 0x7fd9fc100ea0 2022-09-07T21:40:42.807 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2022-09-07T21:40:42.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.805+0000 7fda035ff700 1 -- 172.21.15.47:0/4185960024 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd9e8099010 msgr2=0x7fd9e809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:42.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.805+0000 7fda035ff700 1 --2- 172.21.15.47:0/4185960024 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd9e8099010 0x7fd9e809b4c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc066fb0 tx=0x7fd9f8005f60 comp rx=0 tx=0).stop 2022-09-07T21:40:42.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.805+0000 7fda035ff700 1 -- 172.21.15.47:0/4185960024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9fc100ea0 msgr2=0x7fd9fc196120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:42.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.805+0000 7fda035ff700 1 --2- 172.21.15.47:0/4185960024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9fc100ea0 0x7fd9fc196120 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00e6c0 tx=0x7fd9ec00e9d0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.805+0000 7fda035ff700 1 -- 172.21.15.47:0/4185960024 shutdown_connections 2022-09-07T21:40:42.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.805+0000 7fda035ff700 1 --2- 172.21.15.47:0/4185960024 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fd9e8099010 0x7fd9e809b4c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.805+0000 7fda035ff700 1 --2- 172.21.15.47:0/4185960024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9fc103990 0x7fd9fc19d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.805+0000 7fda035ff700 1 --2- 172.21.15.47:0/4185960024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9fc066be0 0x7fd9fc195be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.805+0000 7fda035ff700 1 --2- 172.21.15.47:0/4185960024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9fc100ea0 0x7fd9fc196120 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:42.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.805+0000 7fda035ff700 1 -- 172.21.15.47:0/4185960024 >> 172.21.15.47:0/4185960024 conn(0x7fd9fc0f8370 msgr2=0x7fd9fc0ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:42.811 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.806+0000 7fda035ff700 1 -- 172.21.15.47:0/4185960024 shutdown_connections 2022-09-07T21:40:42.812 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:42.806+0000 7fda035ff700 1 -- 172.21.15.47:0/4185960024 wait complete. 2022-09-07T21:40:42.837 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-09-07T21:40:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.070+0000 7fbcb11ff700 1 -- 172.21.15.47:0/3312815620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbcac0ff940 msgr2=0x7fbcac106270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.070+0000 7fbcb11ff700 1 --2- 172.21.15.47:0/3312815620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbcac0ff940 0x7fbcac106270 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbca4009b00 tx=0x7fbca4009e10 comp rx=0 tx=0).stop 2022-09-07T21:40:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.070+0000 7fbcb11ff700 1 -- 172.21.15.47:0/3312815620 shutdown_connections 2022-09-07T21:40:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.070+0000 7fbcb11ff700 1 --2- 172.21.15.47:0/3312815620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbcac0ff940 0x7fbcac106270 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.070+0000 7fbcb11ff700 1 --2- 172.21.15.47:0/3312815620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcac0fddb0 0x7fbcac0fe1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.070+0000 7fbcb11ff700 1 --2- 172.21.15.47:0/3312815620 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac0fef90 0x7fbcac0ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.070+0000 7fbcb11ff700 1 -- 172.21.15.47:0/3312815620 >> 172.21.15.47:0/3312815620 conn(0x7fbcac0f9460 msgr2=0x7fbcac0fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.070+0000 7fbcb11ff700 1 -- 172.21.15.47:0/3312815620 shutdown_connections 2022-09-07T21:40:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.070+0000 7fbcb11ff700 1 -- 172.21.15.47:0/3312815620 wait complete. 2022-09-07T21:40:43.078 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.071+0000 7fbcb11ff700 1 Processor -- start 2022-09-07T21:40:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.072+0000 7fbcb11ff700 1 -- start start 2022-09-07T21:40:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.072+0000 7fbcb11ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbcac0fddb0 0x7fbcac195b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.072+0000 7fbcb11ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcac0fef90 0x7fbcac1960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:43.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.072+0000 7fbcb11ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac0ff940 0x7fbcac19d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:43.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.072+0000 7fbcb11ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac19d700 con 0x7fbcac0fddb0 2022-09-07T21:40:43.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.072+0000 7fbcb11ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac19d870 con 0x7fbcac0ff940 2022-09-07T21:40:43.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.072+0000 7fbcb11ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac19da30 con 0x7fbcac0fef90 2022-09-07T21:40:43.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.072+0000 7fbcaad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbcac0fddb0 0x7fbcac195b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.072+0000 7fbcaa59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcac0fef90 0x7fbcac1960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.072+0000 7fbcaad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbcac0fddb0 0x7fbcac195b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34076/0 (socket says 172.21.15.47:34076) 2022-09-07T21:40:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.072+0000 7fbcaad9d700 1 -- 172.21.15.47:0/1312499536 learned_addr learned my addr 172.21.15.47:0/1312499536 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.072+0000 7fbcab59e700 1 --2- 172.21.15.47:0/1312499536 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac0ff940 0x7fbcac19d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.073+0000 7fbcaa59c700 1 -- 172.21.15.47:0/1312499536 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac0ff940 msgr2=0x7fbcac19d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.073+0000 7fbcaa59c700 1 --2- 172.21.15.47:0/1312499536 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac0ff940 0x7fbcac19d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.073+0000 7fbcaa59c700 1 -- 172.21.15.47:0/1312499536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbcac0fddb0 msgr2=0x7fbcac195b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.073+0000 7fbcaa59c700 1 --2- 172.21.15.47:0/1312499536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbcac0fddb0 0x7fbcac195b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.073+0000 7fbcaa59c700 1 -- 172.21.15.47:0/1312499536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca0009470 con 0x7fbcac0fef90 2022-09-07T21:40:43.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.073+0000 7fbcaa59c700 1 --2- 172.21.15.47:0/1312499536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcac0fef90 0x7fbcac1960c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbca0005fd0 tx=0x7fbca000eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:43.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.073+0000 7fbc8ffff700 1 -- 172.21.15.47:0/1312499536 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca0004220 con 0x7fbcac0fef90 2022-09-07T21:40:43.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.073+0000 7fbc8ffff700 1 -- 172.21.15.47:0/1312499536 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fbca0004380 con 0x7fbcac0fef90 2022-09-07T21:40:43.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.073+0000 7fbcb11ff700 1 -- 172.21.15.47:0/1312499536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca4009730 con 0x7fbcac0fef90 2022-09-07T21:40:43.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.073+0000 7fbc8ffff700 1 -- 172.21.15.47:0/1312499536 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca0003c70 con 0x7fbcac0fef90 2022-09-07T21:40:43.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.074+0000 7fbcb11ff700 1 -- 172.21.15.47:0/1312499536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcac19e090 con 0x7fbcac0fef90 2022-09-07T21:40:43.085 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.075+0000 7fbcb11ff700 1 -- 172.21.15.47:0/1312499536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcac109560 con 0x7fbcac0fef90 2022-09-07T21:40:43.086 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.077+0000 7fbc8ffff700 1 -- 172.21.15.47:0/1312499536 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fbca001e020 con 0x7fbcac0fef90 2022-09-07T21:40:43.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.078+0000 7fbc8ffff700 1 --2- 172.21.15.47:0/1312499536 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fbc94099010 0x7fbc9409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:43.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.078+0000 7fbc8ffff700 1 -- 172.21.15.47:0/1312499536 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fbca0014070 con 0x7fbcac0fef90 2022-09-07T21:40:43.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.078+0000 7fbcaad9d700 1 --2- 172.21.15.47:0/1312499536 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fbc94099010 0x7fbc9409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:43.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.079+0000 7fbcaad9d700 1 --2- 172.21.15.47:0/1312499536 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fbc94099010 0x7fbc9409b4c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0057d0 tx=0x7fbc9c005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:43.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.080+0000 7fbc8ffff700 1 -- 172.21.15.47:0/1312499536 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fbca008f3f0 con 0x7fbcac0fef90 2022-09-07T21:40:43.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:42 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3358406407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:40:43.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:42 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/2181872456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:40:43.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:42 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/4185960024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:40:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:42 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3358406407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:40:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:42 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/2181872456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:40:43.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:42 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/4185960024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:40:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:42 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3358406407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:40:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:42 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/2181872456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-07T21:40:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:42 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/4185960024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-07T21:40:43.283 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.279+0000 7fbcb11ff700 1 -- 172.21.15.47:0/1312499536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbcac19e6a0 con 0x7fbcac0fef90 2022-09-07T21:40:43.284 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.280+0000 7fbc8ffff700 1 -- 172.21.15.47:0/1312499536 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbca0003dd0 con 0x7fbcac0fef90 2022-09-07T21:40:43.284 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2022-09-07T21:40:43.286 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.282+0000 7fbcb11ff700 1 -- 172.21.15.47:0/1312499536 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fbc94099010 msgr2=0x7fbc9409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:43.287 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.282+0000 7fbcb11ff700 1 --2- 172.21.15.47:0/1312499536 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fbc94099010 0x7fbc9409b4c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0057d0 tx=0x7fbc9c005ea0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.287 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.282+0000 7fbcb11ff700 1 -- 172.21.15.47:0/1312499536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcac0fef90 msgr2=0x7fbcac1960c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:43.287 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.282+0000 7fbcb11ff700 1 --2- 172.21.15.47:0/1312499536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcac0fef90 0x7fbcac1960c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbca0005fd0 tx=0x7fbca000eca0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.287 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.282+0000 7fbcb11ff700 1 -- 172.21.15.47:0/1312499536 shutdown_connections 2022-09-07T21:40:43.288 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.282+0000 7fbcb11ff700 1 --2- 172.21.15.47:0/1312499536 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fbc94099010 0x7fbc9409b4c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.288 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.282+0000 7fbcb11ff700 1 --2- 172.21.15.47:0/1312499536 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcac0ff940 0x7fbcac19d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.288 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.282+0000 7fbcb11ff700 1 --2- 172.21.15.47:0/1312499536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbcac0fddb0 0x7fbcac195b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.288 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.282+0000 7fbcb11ff700 1 --2- 172.21.15.47:0/1312499536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcac0fef90 0x7fbcac1960c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.282+0000 7fbcb11ff700 1 -- 172.21.15.47:0/1312499536 >> 172.21.15.47:0/1312499536 conn(0x7fbcac0f9460 msgr2=0x7fbcac0fb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:43.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.283+0000 7fbcb11ff700 1 -- 172.21.15.47:0/1312499536 shutdown_connections 2022-09-07T21:40:43.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.283+0000 7fbcb11ff700 1 -- 172.21.15.47:0/1312499536 wait complete. 2022-09-07T21:40:43.313 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2022-09-07T21:40:43.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.546+0000 7f4466dff700 1 -- 172.21.15.47:0/3597846874 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4460100ea0 msgr2=0x7f4460103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:43.551 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.546+0000 7f4466dff700 1 --2- 172.21.15.47:0/3597846874 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4460100ea0 0x7f4460103300 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f444c009750 tx=0x7f444c009a60 comp rx=0 tx=0).stop 2022-09-07T21:40:43.551 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.546+0000 7f4466dff700 1 -- 172.21.15.47:0/3597846874 shutdown_connections 2022-09-07T21:40:43.551 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.546+0000 7f4466dff700 1 --2- 172.21.15.47:0/3597846874 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4460103990 0x7f4460105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.552 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.546+0000 7f4466dff700 1 --2- 172.21.15.47:0/3597846874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4460066be0 0x7f4460100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.552 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.546+0000 7f4466dff700 1 --2- 172.21.15.47:0/3597846874 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4460100ea0 0x7f4460103300 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.552 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.546+0000 7f4466dff700 1 -- 172.21.15.47:0/3597846874 >> 172.21.15.47:0/3597846874 conn(0x7f44600f8370 msgr2=0x7f44600fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:43.553 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.546+0000 7f4466dff700 1 -- 172.21.15.47:0/3597846874 shutdown_connections 2022-09-07T21:40:43.553 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.546+0000 7f4466dff700 1 -- 172.21.15.47:0/3597846874 wait complete. 2022-09-07T21:40:43.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.548+0000 7f4466dff700 1 Processor -- start 2022-09-07T21:40:43.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.548+0000 7f4466dff700 1 -- start start 2022-09-07T21:40:43.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.548+0000 7f4466dff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4460066be0 0x7f44601939d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:43.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.548+0000 7f4466dff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4460100ea0 0x7f4460193f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:43.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.548+0000 7f4466dff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4460103990 0x7f446019af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:43.557 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.548+0000 7f4466dff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446019b550 con 0x7f4460066be0 2022-09-07T21:40:43.557 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.548+0000 7f4466dff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446019b6c0 con 0x7f4460100ea0 2022-09-07T21:40:43.557 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.548+0000 7f4466dff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446019b880 con 0x7f4460103990 2022-09-07T21:40:43.557 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.548+0000 7f4464b9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4460066be0 0x7f44601939d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:43.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.548+0000 7f445ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4460100ea0 0x7f4460193f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:43.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.548+0000 7f446539c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4460103990 0x7f446019af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:43.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.549+0000 7f445ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4460100ea0 0x7f4460193f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:40034/0 (socket says 172.21.15.47:40034) 2022-09-07T21:40:43.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.549+0000 7f445ffff700 1 -- 172.21.15.47:0/1787885954 learned_addr learned my addr 172.21.15.47:0/1787885954 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:43.559 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.549+0000 7f445ffff700 1 -- 172.21.15.47:0/1787885954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4460103990 msgr2=0x7f446019af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:43.559 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.549+0000 7f445ffff700 1 --2- 172.21.15.47:0/1787885954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4460103990 0x7f446019af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.559 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.549+0000 7f445ffff700 1 -- 172.21.15.47:0/1787885954 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4460066be0 msgr2=0x7f44601939d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:43.559 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.549+0000 7f445ffff700 1 --2- 172.21.15.47:0/1787885954 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4460066be0 0x7f44601939d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.560 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.549+0000 7f445ffff700 1 -- 172.21.15.47:0/1787885954 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444c0093b0 con 0x7f4460100ea0 2022-09-07T21:40:43.560 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.549+0000 7f446539c700 1 --2- 172.21.15.47:0/1787885954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4460103990 0x7f446019af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:43.560 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.549+0000 7f4464b9b700 1 --2- 172.21.15.47:0/1787885954 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4460066be0 0x7f44601939d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:43.560 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.549+0000 7f445ffff700 1 --2- 172.21.15.47:0/1787885954 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4460100ea0 0x7f4460193f10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f444c004230 tx=0x7f444c004260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:43.560 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.550+0000 7f445dffb700 1 -- 172.21.15.47:0/1787885954 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c01d070 con 0x7f4460100ea0 2022-09-07T21:40:43.561 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.550+0000 7f445dffb700 1 -- 172.21.15.47:0/1787885954 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f444c003bb0 con 0x7f4460100ea0 2022-09-07T21:40:43.561 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.550+0000 7f4466dff700 1 -- 172.21.15.47:0/1787885954 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446019bb00 con 0x7f4460100ea0 2022-09-07T21:40:43.561 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.550+0000 7f445dffb700 1 -- 172.21.15.47:0/1787885954 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c017b80 con 0x7f4460100ea0 2022-09-07T21:40:43.561 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.550+0000 7f4466dff700 1 -- 172.21.15.47:0/1787885954 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446019bf50 con 0x7f4460100ea0 2022-09-07T21:40:43.562 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.551+0000 7f4466dff700 1 -- 172.21.15.47:0/1787885954 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44601095a0 con 0x7f4460100ea0 2022-09-07T21:40:43.563 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.553+0000 7f445dffb700 1 -- 172.21.15.47:0/1787885954 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f444c00c040 con 0x7f4460100ea0 2022-09-07T21:40:43.563 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.554+0000 7f445dffb700 1 --2- 172.21.15.47:0/1787885954 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4450099010 0x7f445009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:43.563 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.554+0000 7f445dffb700 1 -- 172.21.15.47:0/1787885954 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f444c0c3390 con 0x7f4460100ea0 2022-09-07T21:40:43.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.555+0000 7f4464b9b700 1 --2- 172.21.15.47:0/1787885954 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4450099010 0x7f445009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:43.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.555+0000 7f4464b9b700 1 --2- 172.21.15.47:0/1787885954 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4450099010 0x7f445009b4c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4460066fb0 tx=0x7f4454005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:43.564 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.558+0000 7f445dffb700 1 -- 172.21.15.47:0/1787885954 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f444c08f710 con 0x7f4460100ea0 2022-09-07T21:40:43.764 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.761+0000 7f4466dff700 1 -- 172.21.15.47:0/1787885954 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f4460109730 con 0x7f4460100ea0 2022-09-07T21:40:43.765 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.762+0000 7f445dffb700 1 -- 172.21.15.47:0/1787885954 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4460109730 con 0x7f4460100ea0 2022-09-07T21:40:43.766 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2022-09-07T21:40:43.768 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.764+0000 7f4466dff700 1 -- 172.21.15.47:0/1787885954 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4450099010 msgr2=0x7f445009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:43.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.764+0000 7f4466dff700 1 --2- 172.21.15.47:0/1787885954 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4450099010 0x7f445009b4c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4460066fb0 tx=0x7f4454005f60 comp rx=0 tx=0).stop 2022-09-07T21:40:43.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.765+0000 7f4466dff700 1 -- 172.21.15.47:0/1787885954 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4460100ea0 msgr2=0x7f4460193f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:43.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.765+0000 7f4466dff700 1 --2- 172.21.15.47:0/1787885954 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4460100ea0 0x7f4460193f10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f444c004230 tx=0x7f444c004260 comp rx=0 tx=0).stop 2022-09-07T21:40:43.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.765+0000 7f4466dff700 1 -- 172.21.15.47:0/1787885954 shutdown_connections 2022-09-07T21:40:43.770 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.765+0000 7f4466dff700 1 --2- 172.21.15.47:0/1787885954 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4450099010 0x7f445009b4c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.770 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.765+0000 7f4466dff700 1 --2- 172.21.15.47:0/1787885954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4460103990 0x7f446019af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.770 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.765+0000 7f4466dff700 1 --2- 172.21.15.47:0/1787885954 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4460066be0 0x7f44601939d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.770 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.765+0000 7f4466dff700 1 --2- 172.21.15.47:0/1787885954 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4460100ea0 0x7f4460193f10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:43.771 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.765+0000 7f4466dff700 1 -- 172.21.15.47:0/1787885954 >> 172.21.15.47:0/1787885954 conn(0x7f44600f8370 msgr2=0x7f44600ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:43.771 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.765+0000 7f4466dff700 1 -- 172.21.15.47:0/1787885954 shutdown_connections 2022-09-07T21:40:43.771 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:43.765+0000 7f4466dff700 1 -- 172.21.15.47:0/1787885954 wait complete. 2022-09-07T21:40:43.796 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-07T21:40:44.029 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.025+0000 7f45d7fff700 1 -- 172.21.15.47:0/66069780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d01011d0 msgr2=0x7f45d0101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:44.030 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.025+0000 7f45d7fff700 1 --2- 172.21.15.47:0/66069780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d01011d0 0x7f45d0101620 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009810 tx=0x7f45c0009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:44.030 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.025+0000 7f45d7fff700 1 -- 172.21.15.47:0/66069780 shutdown_connections 2022-09-07T21:40:44.030 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.025+0000 7f45d7fff700 1 --2- 172.21.15.47:0/66069780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45d0101b60 0x7f45d0108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.030 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.025+0000 7f45d7fff700 1 --2- 172.21.15.47:0/66069780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45d00fff50 0x7f45d0100360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.031 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.025+0000 7f45d7fff700 1 --2- 172.21.15.47:0/66069780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d01011d0 0x7f45d0101620 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.031 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.025+0000 7f45d7fff700 1 -- 172.21.15.47:0/66069780 >> 172.21.15.47:0/66069780 conn(0x7f45d00fb6b0 msgr2=0x7f45d00fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:44.031 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.025+0000 7f45d7fff700 1 -- 172.21.15.47:0/66069780 shutdown_connections 2022-09-07T21:40:44.032 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.026+0000 7f45d7fff700 1 -- 172.21.15.47:0/66069780 wait complete. 2022-09-07T21:40:44.034 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.027+0000 7f45d7fff700 1 Processor -- start 2022-09-07T21:40:44.034 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.027+0000 7f45d7fff700 1 -- start start 2022-09-07T21:40:44.034 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.028+0000 7f45d7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45d00fff50 0x7f45d0195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:44.035 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.028+0000 7f45d7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45d01011d0 0x7f45d0196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:44.035 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.028+0000 7f45d7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d0101b60 0x7f45d019d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:44.035 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.028+0000 7f45d7fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d019d6a0 con 0x7f45d01011d0 2022-09-07T21:40:44.035 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.028+0000 7f45d7fff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d019d810 con 0x7f45d00fff50 2022-09-07T21:40:44.035 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.028+0000 7f45d7fff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d019d9d0 con 0x7f45d0101b60 2022-09-07T21:40:44.036 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.028+0000 7f45d5d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45d00fff50 0x7f45d0195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:44.036 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.028+0000 7f45d5d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45d00fff50 0x7f45d0195b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:40050/0 (socket says 172.21.15.47:40050) 2022-09-07T21:40:44.036 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.028+0000 7f45d5d9b700 1 -- 172.21.15.47:0/3228083687 learned_addr learned my addr 172.21.15.47:0/3228083687 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:44.036 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.028+0000 7f45d659c700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d0101b60 0x7f45d019d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:44.037 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.028+0000 7f45d559a700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45d01011d0 0x7f45d0196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:44.037 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.029+0000 7f45d5d9b700 1 -- 172.21.15.47:0/3228083687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d0101b60 msgr2=0x7f45d019d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:44.037 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.029+0000 7f45d5d9b700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d0101b60 0x7f45d019d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.037 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.029+0000 7f45d5d9b700 1 -- 172.21.15.47:0/3228083687 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45d01011d0 msgr2=0x7f45d0196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:44.038 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.029+0000 7f45d5d9b700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45d01011d0 0x7f45d0196060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.038 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.029+0000 7f45d5d9b700 1 -- 172.21.15.47:0/3228083687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c0009470 con 0x7f45d00fff50 2022-09-07T21:40:44.038 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.029+0000 7f45d659c700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d0101b60 0x7f45d019d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:44.038 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.029+0000 7f45d5d9b700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45d00fff50 0x7f45d0195b20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00e710 tx=0x7f45cc00ea20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:44.039 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.029+0000 7f45c6ffd700 1 -- 172.21.15.47:0/3228083687 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc004220 con 0x7f45d00fff50 2022-09-07T21:40:44.039 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.029+0000 7f45d7fff700 1 -- 172.21.15.47:0/3228083687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d019dcb0 con 0x7f45d00fff50 2022-09-07T21:40:44.039 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.029+0000 7f45c6ffd700 1 -- 172.21.15.47:0/3228083687 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f45cc004380 con 0x7f45d00fff50 2022-09-07T21:40:44.039 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.029+0000 7f45c6ffd700 1 -- 172.21.15.47:0/3228083687 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc003cf0 con 0x7f45d00fff50 2022-09-07T21:40:44.039 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.029+0000 7f45d7fff700 1 -- 172.21.15.47:0/3228083687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d019e1c0 con 0x7f45d00fff50 2022-09-07T21:40:44.040 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.030+0000 7f45d7fff700 1 -- 172.21.15.47:0/3228083687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d010b700 con 0x7f45d00fff50 2022-09-07T21:40:44.041 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.032+0000 7f45c6ffd700 1 -- 172.21.15.47:0/3228083687 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f45cc01e020 con 0x7f45d00fff50 2022-09-07T21:40:44.041 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.033+0000 7f45c6ffd700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f45bc099080 0x7f45bc09b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:44.042 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.034+0000 7f45c6ffd700 1 -- 172.21.15.47:0/3228083687 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f45cc014070 con 0x7f45d00fff50 2022-09-07T21:40:44.042 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.034+0000 7f45d559a700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f45bc099080 0x7f45bc09b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:44.042 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.034+0000 7f45d559a700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f45bc099080 0x7f45bc09b530 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f45c000b4f0 tx=0x7f45c0009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:44.042 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.036+0000 7f45c6ffd700 1 -- 172.21.15.47:0/3228083687 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f45cc08f240 con 0x7f45d00fff50 2022-09-07T21:40:44.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:43 smithi047 ceph-mon[90087]: pgmap v379: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:44.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:43 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1312499536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:40:44.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:43 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1787885954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:40:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:43 smithi055 ceph-mon[97033]: pgmap v379: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:43 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1312499536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:40:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:43 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1787885954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:40:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:43 smithi059 ceph-mon[97450]: pgmap v379: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:43 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1312499536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-07T21:40:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:43 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1787885954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-07T21:40:44.283 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.280+0000 7f45d7fff700 1 -- 172.21.15.47:0/3228083687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45d0106de0 con 0x7f45d00fff50 2022-09-07T21:40:44.284 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.282+0000 7f45c6ffd700 1 -- 172.21.15.47:0/3228083687 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10707 (secure 0 0 0) 0x7f45cc042e60 con 0x7f45d00fff50 2022-09-07T21:40:44.285 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:40:44.285 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3393333058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3393333058},{"type":"v1","addr":"172.21.15.47:6835","nonce":3393333058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3763216833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3763216833},{"type":"v1","addr":"172.21.15.55:6833","nonce":3763216833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1360705881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1360705881},{"type":"v1","addr":"172.21.15.59:6833","nonce":1360705881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/2559266585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":2559266585},{"type":"v1","addr":"172.21.15.55:6835","nonce":2559266585}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/139419378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":139419378},{"type":"v1","addr":"172.21.15.47:6837","nonce":139419378}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/2090942297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2090942297},{"type":"v1","addr":"172.21.15.55:6837","nonce":2090942297}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-09-07T21:38:40.544389+0000","modified":"2022-09-07T21:39:06.833800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24586,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6835/2445409240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2445409240},{"type":"v1","addr":"172.21.15.59:6835","nonce":2445409240}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6837/3542466465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6836","nonce":3542466465},{"type":"v1","addr":"172.21.15.59:6837","nonce":3542466465}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6839/315505812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":315505812},{"type":"v1","addr":"172.21.15.47:6839","nonce":315505812}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.59:6839/1006724740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6838","nonce":1006724740},{"type":"v1","addr":"172.21.15.59:6839","nonce":1006724740}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6839/517990300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":517990300},{"type":"v1","addr":"172.21.15.55:6839","nonce":517990300}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24586":{"gid":24586,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6841/793863162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6840","nonce":793863162},{"type":"v1","addr":"172.21.15.47:6841","nonce":793863162}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-09-07T21:40:44.287 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.283+0000 7f45d7fff700 1 -- 172.21.15.47:0/3228083687 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f45bc099080 msgr2=0x7f45bc09b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:44.288 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.283+0000 7f45d7fff700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f45bc099080 0x7f45bc09b530 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f45c000b4f0 tx=0x7f45c0009eb0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.288 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.283+0000 7f45d7fff700 1 -- 172.21.15.47:0/3228083687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45d00fff50 msgr2=0x7f45d0195b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:44.288 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.283+0000 7f45d7fff700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45d00fff50 0x7f45d0195b20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00e710 tx=0x7f45cc00ea20 comp rx=0 tx=0).stop 2022-09-07T21:40:44.288 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.284+0000 7f45d7fff700 1 -- 172.21.15.47:0/3228083687 shutdown_connections 2022-09-07T21:40:44.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.284+0000 7f45d7fff700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f45bc099080 0x7f45bc09b530 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.284+0000 7f45d7fff700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d0101b60 0x7f45d019d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.284+0000 7f45d7fff700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45d00fff50 0x7f45d0195b20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.289 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.284+0000 7f45d7fff700 1 --2- 172.21.15.47:0/3228083687 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45d01011d0 0x7f45d0196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.284+0000 7f45d7fff700 1 -- 172.21.15.47:0/3228083687 >> 172.21.15.47:0/3228083687 conn(0x7f45d00fb6b0 msgr2=0x7f45d00fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:44.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.284+0000 7f45d7fff700 1 -- 172.21.15.47:0/3228083687 shutdown_connections 2022-09-07T21:40:44.290 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.284+0000 7f45d7fff700 1 -- 172.21.15.47:0/3228083687 wait complete. 2022-09-07T21:40:44.291 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 24 2022-09-07T21:40:44.314 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-07T21:40:44.548 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.544+0000 7ff7ea39d700 1 -- 172.21.15.47:0/2719787824 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7e40fe460 msgr2=0x7ff7e40fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:44.548 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.544+0000 7ff7ea39d700 1 --2- 172.21.15.47:0/2719787824 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7e40fe460 0x7ff7e40fee50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4009810 tx=0x7ff7d4009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:44.549 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.544+0000 7ff7ea39d700 1 -- 172.21.15.47:0/2719787824 shutdown_connections 2022-09-07T21:40:44.549 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.544+0000 7ff7ea39d700 1 --2- 172.21.15.47:0/2719787824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7e40ff4e0 0x7ff7e4106080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.549 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.544+0000 7ff7ea39d700 1 --2- 172.21.15.47:0/2719787824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7e40fdb10 0x7ff7e40fdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.544+0000 7ff7ea39d700 1 --2- 172.21.15.47:0/2719787824 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7e40fe460 0x7ff7e40fee50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.544+0000 7ff7ea39d700 1 -- 172.21.15.47:0/2719787824 >> 172.21.15.47:0/2719787824 conn(0x7ff7e40f91c0 msgr2=0x7ff7e40fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:44.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.544+0000 7ff7ea39d700 1 -- 172.21.15.47:0/2719787824 shutdown_connections 2022-09-07T21:40:44.550 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.544+0000 7ff7ea39d700 1 -- 172.21.15.47:0/2719787824 wait complete. 2022-09-07T21:40:44.552 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.546+0000 7ff7ea39d700 1 Processor -- start 2022-09-07T21:40:44.553 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.546+0000 7ff7ea39d700 1 -- start start 2022-09-07T21:40:44.553 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.546+0000 7ff7ea39d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7e40fdb10 0x7ff7e4195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:44.553 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.546+0000 7ff7ea39d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7e40fe460 0x7ff7e4195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:44.554 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.546+0000 7ff7ea39d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7e40ff4e0 0x7ff7e419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:44.554 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.546+0000 7ff7ea39d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e419d490 con 0x7ff7e40fe460 2022-09-07T21:40:44.554 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.546+0000 7ff7ea39d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e419d600 con 0x7ff7e40ff4e0 2022-09-07T21:40:44.554 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.546+0000 7ff7ea39d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e419d7c0 con 0x7ff7e40fdb10 2022-09-07T21:40:44.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.546+0000 7ff7e8b9a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7e40fe460 0x7ff7e4195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:44.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.546+0000 7ff7e939b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7e40fdb10 0x7ff7e4195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:44.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.547+0000 7ff7e939b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7e40fdb10 0x7ff7e4195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:56502/0 (socket says 172.21.15.47:56502) 2022-09-07T21:40:44.555 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.547+0000 7ff7e939b700 1 -- 172.21.15.47:0/1645855249 learned_addr learned my addr 172.21.15.47:0/1645855249 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:44.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.547+0000 7ff7e9b9c700 1 --2- 172.21.15.47:0/1645855249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7e40ff4e0 0x7ff7e419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:44.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.547+0000 7ff7e939b700 1 -- 172.21.15.47:0/1645855249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7e40ff4e0 msgr2=0x7ff7e419cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:44.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.547+0000 7ff7e939b700 1 --2- 172.21.15.47:0/1645855249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7e40ff4e0 0x7ff7e419cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.547+0000 7ff7e939b700 1 -- 172.21.15.47:0/1645855249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7e40fe460 msgr2=0x7ff7e4195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:44.556 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.547+0000 7ff7e939b700 1 --2- 172.21.15.47:0/1645855249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7e40fe460 0x7ff7e4195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.557 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.547+0000 7ff7e939b700 1 -- 172.21.15.47:0/1645855249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e00093b0 con 0x7ff7e40fdb10 2022-09-07T21:40:44.557 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.547+0000 7ff7e8b9a700 1 --2- 172.21.15.47:0/1645855249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7e40fe460 0x7ff7e4195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:44.557 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.547+0000 7ff7e939b700 1 --2- 172.21.15.47:0/1645855249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7e40fdb10 0x7ff7e4195910 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000e9c0 tx=0x7ff7e000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:44.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.548+0000 7ff7da7fc700 1 -- 172.21.15.47:0/1645855249 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e00097b0 con 0x7ff7e40fdb10 2022-09-07T21:40:44.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.548+0000 7ff7da7fc700 1 -- 172.21.15.47:0/1645855249 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff7e000ce70 con 0x7ff7e40fdb10 2022-09-07T21:40:44.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.548+0000 7ff7da7fc700 1 -- 172.21.15.47:0/1645855249 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e0003cf0 con 0x7ff7e40fdb10 2022-09-07T21:40:44.558 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.548+0000 7ff7ea39d700 1 -- 172.21.15.47:0/1645855249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d4009470 con 0x7ff7e40fdb10 2022-09-07T21:40:44.559 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.548+0000 7ff7ea39d700 1 -- 172.21.15.47:0/1645855249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e419ddd0 con 0x7ff7e40fdb10 2022-09-07T21:40:44.559 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.549+0000 7ff7ea39d700 1 -- 172.21.15.47:0/1645855249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e41092d0 con 0x7ff7e40fdb10 2022-09-07T21:40:44.560 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.551+0000 7ff7da7fc700 1 -- 172.21.15.47:0/1645855249 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7ff7e001e020 con 0x7ff7e40fdb10 2022-09-07T21:40:44.561 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.552+0000 7ff7da7fc700 1 --2- 172.21.15.47:0/1645855249 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff7d0099090 0x7ff7d009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:44.561 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.552+0000 7ff7da7fc700 1 -- 172.21.15.47:0/1645855249 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7ff7e0014070 con 0x7ff7e40fdb10 2022-09-07T21:40:44.561 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.553+0000 7ff7e8b9a700 1 --2- 172.21.15.47:0/1645855249 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff7d0099090 0x7ff7d009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:44.561 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.553+0000 7ff7e8b9a700 1 --2- 172.21.15.47:0/1645855249 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff7d0099090 0x7ff7d009b540 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff7d400b4f0 tx=0x7ff7d4009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:44.562 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.555+0000 7ff7da7fc700 1 -- 172.21.15.47:0/1645855249 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff7e008f540 con 0x7ff7e40fdb10 2022-09-07T21:40:44.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.759+0000 7ff7ea39d700 1 -- 172.21.15.47:0/1645855249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7e419e4b0 con 0x7ff7e40fdb10 2022-09-07T21:40:44.764 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.761+0000 7ff7da7fc700 1 -- 172.21.15.47:0/1645855249 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+23592 (secure 0 0 0) 0x7ff7e0092b60 con 0x7ff7e40fdb10 2022-09-07T21:40:44.765 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-09-07T21:40:44.765 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":91,"fsid":"01b8396e-2ef4-11ed-8431-001a4aab830c","created":"2022-09-07T21:29:33.859557+0000","modified":"2022-09-07T21:39:47.780661+0000","last_up_change":"2022-09-07T21:37:50.366879+0000","last_in_change":"2022-09-07T21:37:31.536099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-07T21:33:29.342534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-07T21:38:38.608937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-07T21:38:39.936252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-07T21:38:43.731797+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7c4477c8-f9db-4bd6-80f3-ac83b7c86010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6803","nonce":3092904881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6805","nonce":3092904881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6809","nonce":3092904881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3092904881},{"type":"v1","addr":"172.21.15.47:6807","nonce":3092904881}]},"public_addr":"172.21.15.47:6803/3092904881","cluster_addr":"172.21.15.47:6805/3092904881","heartbeat_back_addr":"172.21.15.47:6809/3092904881","heartbeat_front_addr":"172.21.15.47:6807/3092904881","state":["exists","up"]},{"osd":1,"uuid":"692ba70e-935d-4a34-847d-44a304850c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6801","nonce":990490301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6803","nonce":990490301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6807","nonce":990490301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":990490301},{"type":"v1","addr":"172.21.15.55:6805","nonce":990490301}]},"public_addr":"172.21.15.55:6801/990490301","cluster_addr":"172.21.15.55:6803/990490301","heartbeat_back_addr":"172.21.15.55:6807/990490301","heartbeat_front_addr":"172.21.15.55:6805/990490301","state":["exists","up"]},{"osd":2,"uuid":"20bbe910-ab0c-4921-b54f-6d37be12f5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6801","nonce":2729613578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6803","nonce":2729613578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6807","nonce":2729613578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2729613578},{"type":"v1","addr":"172.21.15.59:6805","nonce":2729613578}]},"public_addr":"172.21.15.59:6801/2729613578","cluster_addr":"172.21.15.59:6803/2729613578","heartbeat_back_addr":"172.21.15.59:6807/2729613578","heartbeat_front_addr":"172.21.15.59:6805/2729613578","state":["exists","up"]},{"osd":3,"uuid":"b4b5e6b0-5a09-4711-bd7f-5496b0547af2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6811","nonce":1063754579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6813","nonce":1063754579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6817","nonce":1063754579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1063754579},{"type":"v1","addr":"172.21.15.47:6815","nonce":1063754579}]},"public_addr":"172.21.15.47:6811/1063754579","cluster_addr":"172.21.15.47:6813/1063754579","heartbeat_back_addr":"172.21.15.47:6817/1063754579","heartbeat_front_addr":"172.21.15.47:6815/1063754579","state":["exists","up"]},{"osd":4,"uuid":"f98aac6c-62be-4f57-94e1-4baa606d9e9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6809","nonce":4055716956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6811","nonce":4055716956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6815","nonce":4055716956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":4055716956},{"type":"v1","addr":"172.21.15.55:6813","nonce":4055716956}]},"public_addr":"172.21.15.55:6809/4055716956","cluster_addr":"172.21.15.55:6811/4055716956","heartbeat_back_addr":"172.21.15.55:6815/4055716956","heartbeat_front_addr":"172.21.15.55:6813/4055716956","state":["exists","up"]},{"osd":5,"uuid":"c2b864da-e60b-4140-a6d9-30e4e375ef6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6809","nonce":725179539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6811","nonce":725179539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6815","nonce":725179539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":725179539},{"type":"v1","addr":"172.21.15.59:6813","nonce":725179539}]},"public_addr":"172.21.15.59:6809/725179539","cluster_addr":"172.21.15.59:6811/725179539","heartbeat_back_addr":"172.21.15.59:6815/725179539","heartbeat_front_addr":"172.21.15.59:6813/725179539","state":["exists","up"]},{"osd":6,"uuid":"fbe32a2e-570c-4f3b-a860-1c0e603ae96b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6819","nonce":4053448165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6821","nonce":4053448165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6825","nonce":4053448165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":4053448165},{"type":"v1","addr":"172.21.15.47:6823","nonce":4053448165}]},"public_addr":"172.21.15.47:6819/4053448165","cluster_addr":"172.21.15.47:6821/4053448165","heartbeat_back_addr":"172.21.15.47:6825/4053448165","heartbeat_front_addr":"172.21.15.47:6823/4053448165","state":["exists","up"]},{"osd":7,"uuid":"85e70ce9-e721-4719-b309-e5011ddbbab3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6817","nonce":199887712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6819","nonce":199887712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6823","nonce":199887712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":199887712},{"type":"v1","addr":"172.21.15.55:6821","nonce":199887712}]},"public_addr":"172.21.15.55:6817/199887712","cluster_addr":"172.21.15.55:6819/199887712","heartbeat_back_addr":"172.21.15.55:6823/199887712","heartbeat_front_addr":"172.21.15.55:6821/199887712","state":["exists","up"]},{"osd":8,"uuid":"e5b3728f-7ba6-4d25-a54c-d04bad536de1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6817","nonce":1132606220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6819","nonce":1132606220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6823","nonce":1132606220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1132606220},{"type":"v1","addr":"172.21.15.59:6821","nonce":1132606220}]},"public_addr":"172.21.15.59:6817/1132606220","cluster_addr":"172.21.15.59:6819/1132606220","heartbeat_back_addr":"172.21.15.59:6823/1132606220","heartbeat_front_addr":"172.21.15.59:6821/1132606220","state":["exists","up"]},{"osd":9,"uuid":"3f08f42a-4784-4e10-b862-16806fa5ce27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6827","nonce":3489379795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6829","nonce":3489379795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6833","nonce":3489379795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3489379795},{"type":"v1","addr":"172.21.15.47:6831","nonce":3489379795}]},"public_addr":"172.21.15.47:6827/3489379795","cluster_addr":"172.21.15.47:6829/3489379795","heartbeat_back_addr":"172.21.15.47:6833/3489379795","heartbeat_front_addr":"172.21.15.47:6831/3489379795","state":["exists","up"]},{"osd":10,"uuid":"684aed4e-909e-470a-a8e2-a0b3a52aca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6825","nonce":2413766072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6827","nonce":2413766072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6831","nonce":2413766072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2413766072},{"type":"v1","addr":"172.21.15.55:6829","nonce":2413766072}]},"public_addr":"172.21.15.55:6825/2413766072","cluster_addr":"172.21.15.55:6827/2413766072","heartbeat_back_addr":"172.21.15.55:6831/2413766072","heartbeat_front_addr":"172.21.15.55:6829/2413766072","state":["exists","up"]},{"osd":11,"uuid":"d5c3afb2-663c-434c-980a-37b1f230b170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6825","nonce":1569847315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6827","nonce":1569847315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6831","nonce":1569847315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1569847315},{"type":"v1","addr":"172.21.15.59:6829","nonce":1569847315}]},"public_addr":"172.21.15.59:6825/1569847315","cluster_addr":"172.21.15.59:6827/1569847315","heartbeat_back_addr":"172.21.15.59:6831/1569847315","heartbeat_front_addr":"172.21.15.59:6829/1569847315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:32:31.523841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:00.254971+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:33:26.732146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:02.056846+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:30.004927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:34:55.071193+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:29.698414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:35:57.239555+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:22.620732+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:36:57.156136+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:24.997837+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-07T21:37:48.333528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.18","mappings":[{"from":5,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:0/2838000558":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6800/4097349239":"2022-09-08T21:30:46.439577+0000","172.21.15.47:6801/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:6800/1435938941":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2303715505":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1816663715":"2022-09-08T21:30:46.439577+0000","172.21.15.47:0/3909884596":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/2780837564":"2022-09-08T21:30:15.415366+0000","172.21.15.47:0/1253653766":"2022-09-08T21:30:46.439577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-07T21:40:44.767 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.764+0000 7ff7ea39d700 1 -- 172.21.15.47:0/1645855249 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff7d0099090 msgr2=0x7ff7d009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:44.767 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.764+0000 7ff7ea39d700 1 --2- 172.21.15.47:0/1645855249 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff7d0099090 0x7ff7d009b540 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff7d400b4f0 tx=0x7ff7d4009eb0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.768 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.764+0000 7ff7ea39d700 1 -- 172.21.15.47:0/1645855249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7e40fdb10 msgr2=0x7ff7e4195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:44.768 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.764+0000 7ff7ea39d700 1 --2- 172.21.15.47:0/1645855249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7e40fdb10 0x7ff7e4195910 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000e9c0 tx=0x7ff7e000ecd0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.768 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.764+0000 7ff7ea39d700 1 -- 172.21.15.47:0/1645855249 shutdown_connections 2022-09-07T21:40:44.768 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.764+0000 7ff7ea39d700 1 --2- 172.21.15.47:0/1645855249 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7ff7d0099090 0x7ff7d009b540 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.764+0000 7ff7ea39d700 1 --2- 172.21.15.47:0/1645855249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7e40ff4e0 0x7ff7e419cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.764+0000 7ff7ea39d700 1 --2- 172.21.15.47:0/1645855249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7e40fdb10 0x7ff7e4195910 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.764+0000 7ff7ea39d700 1 --2- 172.21.15.47:0/1645855249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7e40fe460 0x7ff7e4195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:44.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.764+0000 7ff7ea39d700 1 -- 172.21.15.47:0/1645855249 >> 172.21.15.47:0/1645855249 conn(0x7ff7e40f91c0 msgr2=0x7ff7e40fb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:44.770 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.764+0000 7ff7ea39d700 1 -- 172.21.15.47:0/1645855249 shutdown_connections 2022-09-07T21:40:44.770 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:44.764+0000 7ff7ea39d700 1 -- 172.21.15.47:0/1645855249 wait complete. 2022-09-07T21:40:44.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2022-09-07T21:38:40.544389+0000', 'modified': '2022-09-07T21:39:06.833800+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24586, 'mds_1': 24572, 'mds_2': 24556}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24526': {'gid': 24526, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.59:6835/2445409240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 2445409240}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 2445409240}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24532': {'gid': 24532, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.59:6837/3542466465', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6836', 'nonce': 3542466465}, {'type': 'v1', 'addr': '172.21.15.59:6837', 'nonce': 3542466465}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24548': {'gid': 24548, 'name': 'g', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6839/315505812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6838', 'nonce': 315505812}, {'type': 'v1', 'addr': '172.21.15.47:6839', 'nonce': 315505812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24556': {'gid': 24556, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.59:6839/1006724740', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6838', 'nonce': 1006724740}, {'type': 'v1', 'addr': '172.21.15.59:6839', 'nonce': 1006724740}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6839/517990300', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6838', 'nonce': 517990300}, {'type': 'v1', 'addr': '172.21.15.55:6839', 'nonce': 517990300}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24586': {'gid': 24586, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.47:6841/793863162', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6840', 'nonce': 793863162}, {'type': 'v1', 'addr': '172.21.15.47:6841', 'nonce': 793863162}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-09-07T21:40:44.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2022-09-07T21:40:44.796 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2022-09-07T21:40:45.034 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.030+0000 7efc8d9ff700 1 -- 172.21.15.47:0/149422563 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc880fef90 msgr2=0x7efc880ff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.034 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.030+0000 7efc8d9ff700 1 --2- 172.21.15.47:0/149422563 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc880fef90 0x7efc880ff400 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efc7c009810 tx=0x7efc7c009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:45.034 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.030+0000 7efc8d9ff700 1 -- 172.21.15.47:0/149422563 shutdown_connections 2022-09-07T21:40:45.034 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.030+0000 7efc8d9ff700 1 --2- 172.21.15.47:0/149422563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc880ff940 0x7efc88106270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.035 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.030+0000 7efc8d9ff700 1 --2- 172.21.15.47:0/149422563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc880fddb0 0x7efc880fe1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.035 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.030+0000 7efc8d9ff700 1 --2- 172.21.15.47:0/149422563 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc880fef90 0x7efc880ff400 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.035 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.030+0000 7efc8d9ff700 1 -- 172.21.15.47:0/149422563 >> 172.21.15.47:0/149422563 conn(0x7efc880f9460 msgr2=0x7efc880fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.035 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.030+0000 7efc8d9ff700 1 -- 172.21.15.47:0/149422563 shutdown_connections 2022-09-07T21:40:45.036 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.030+0000 7efc8d9ff700 1 -- 172.21.15.47:0/149422563 wait complete. 2022-09-07T21:40:45.038 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.031+0000 7efc8d9ff700 1 Processor -- start 2022-09-07T21:40:45.038 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.031+0000 7efc8d9ff700 1 -- start start 2022-09-07T21:40:45.038 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.032+0000 7efc8d9ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc880fddb0 0x7efc88195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.038 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.032+0000 7efc8d9ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc880fef90 0x7efc88195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.039 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.032+0000 7efc8d9ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc880ff940 0x7efc8819d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.039 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.032+0000 7efc8d9ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8819d610 con 0x7efc880fef90 2022-09-07T21:40:45.039 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.032+0000 7efc8d9ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8819d780 con 0x7efc880fddb0 2022-09-07T21:40:45.039 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.032+0000 7efc8d9ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8819d940 con 0x7efc880ff940 2022-09-07T21:40:45.039 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.032+0000 7efc867fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc880fef90 0x7efc88195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.040 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.032+0000 7efc86ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc880fddb0 0x7efc88195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.040 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.032+0000 7efc867fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc880fef90 0x7efc88195fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34144/0 (socket says 172.21.15.47:34144) 2022-09-07T21:40:45.040 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.032+0000 7efc86ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc880fddb0 0x7efc88195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:40092/0 (socket says 172.21.15.47:40092) 2022-09-07T21:40:45.040 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.032+0000 7efc867fc700 1 -- 172.21.15.47:0/125358825 learned_addr learned my addr 172.21.15.47:0/125358825 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.041 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.032+0000 7efc877fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc880ff940 0x7efc8819d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.041 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.032+0000 7efc867fc700 1 -- 172.21.15.47:0/125358825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc880ff940 msgr2=0x7efc8819d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.041 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.033+0000 7efc867fc700 1 --2- 172.21.15.47:0/125358825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc880ff940 0x7efc8819d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.041 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.033+0000 7efc867fc700 1 -- 172.21.15.47:0/125358825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc880fddb0 msgr2=0x7efc88195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.041 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.033+0000 7efc867fc700 1 --2- 172.21.15.47:0/125358825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc880fddb0 0x7efc88195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.042 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.033+0000 7efc867fc700 1 -- 172.21.15.47:0/125358825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c009470 con 0x7efc880fef90 2022-09-07T21:40:45.042 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.033+0000 7efc86ffd700 1 --2- 172.21.15.47:0/125358825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc880fddb0 0x7efc88195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:45.042 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.033+0000 7efc867fc700 1 --2- 172.21.15.47:0/125358825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc880fef90 0x7efc88195fd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efc7c00f6c0 tx=0x7efc7c00f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.042 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.033+0000 7efc8c9fd700 1 -- 172.21.15.47:0/125358825 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c01d070 con 0x7efc880fef90 2022-09-07T21:40:45.042 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.033+0000 7efc8d9ff700 1 -- 172.21.15.47:0/125358825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc8819dbc0 con 0x7efc880fef90 2022-09-07T21:40:45.043 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.033+0000 7efc8c9fd700 1 -- 172.21.15.47:0/125358825 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7efc7c00f910 con 0x7efc880fef90 2022-09-07T21:40:45.043 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.033+0000 7efc8c9fd700 1 -- 172.21.15.47:0/125358825 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c004840 con 0x7efc880fef90 2022-09-07T21:40:45.043 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.033+0000 7efc8d9ff700 1 -- 172.21.15.47:0/125358825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8819e120 con 0x7efc880fef90 2022-09-07T21:40:45.047 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.037+0000 7efc8c9fd700 1 -- 172.21.15.47:0/125358825 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7efc7c028030 con 0x7efc880fef90 2022-09-07T21:40:45.047 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.038+0000 7efc8c9fd700 1 --2- 172.21.15.47:0/125358825 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7efc70099010 0x7efc7009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.048 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.038+0000 7efca0f44b80 1 Processor -- start 2022-09-07T21:40:45.048 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.038+0000 7efc8c9fd700 1 -- 172.21.15.47:0/125358825 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7efc7c0c33e0 con 0x7efc880fef90 2022-09-07T21:40:45.048 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.039+0000 7efc86ffd700 1 --2- 172.21.15.47:0/125358825 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7efc70099010 0x7efc7009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.048 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.039+0000 7efca0f44b80 1 -- start start 2022-09-07T21:40:45.049 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.039+0000 7efca0f44b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c4934682f0 0x55c493468700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.049 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.039+0000 7efca0f44b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 0x55c493469470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.049 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.039+0000 7efca0f44b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c493469a00 0x55c493469e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.049 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.039+0000 7efca0f44b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49334a080 con 0x55c4934682f0 2022-09-07T21:40:45.049 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.040+0000 7efca0f44b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4933b49e0 con 0x55c493469a00 2022-09-07T21:40:45.050 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.040+0000 7efca0f44b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4933ae180 con 0x55c493468c40 2022-09-07T21:40:45.050 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.040+0000 7efc86ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c4934682f0 0x55c493468700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.050 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.040+0000 7efc86ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c4934682f0 0x55c493468700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34146/0 (socket says 172.21.15.47:34146) 2022-09-07T21:40:45.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.040+0000 7efc86ffd700 1 -- 172.21.15.47:0/724557156 learned_addr learned my addr 172.21.15.47:0/724557156 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.040+0000 7efc877fe700 1 --2- 172.21.15.47:0/724557156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c493469a00 0x55c493469e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.040+0000 7efc867fc700 1 --2- 172.21.15.47:0/724557156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 0x55c493469470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.040+0000 7efc86ffd700 1 --2- 172.21.15.47:0/125358825 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7efc70099010 0x7efc7009b4c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efc78005fd0 tx=0x7efc78005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.051 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.040+0000 7efc86ffd700 1 -- 172.21.15.47:0/724557156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 msgr2=0x55c493469470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.052 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.040+0000 7efc86ffd700 1 --2- 172.21.15.47:0/724557156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 0x55c493469470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.052 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.040+0000 7efc86ffd700 1 -- 172.21.15.47:0/724557156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c493469a00 msgr2=0x55c493469e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.052 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.040+0000 7efc86ffd700 1 --2- 172.21.15.47:0/724557156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c493469a00 0x55c493469e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.052 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.040+0000 7efc86ffd700 1 -- 172.21.15.47:0/724557156 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4932365e0 con 0x55c4934682f0 2022-09-07T21:40:45.053 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.040+0000 7efc877fe700 1 --2- 172.21.15.47:0/724557156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c493469a00 0x55c493469e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:45.053 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.041+0000 7efc86ffd700 1 --2- 172.21.15.47:0/724557156 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c4934682f0 0x55c493468700 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efc78004330 tx=0x7efc78014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8fc187c79a5d7d server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.053 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.041+0000 7efc69ffb700 1 -- 172.21.15.47:0/724557156 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc78003990 con 0x55c4934682f0 2022-09-07T21:40:45.053 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.041+0000 7efc69ffb700 1 -- 172.21.15.47:0/724557156 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7efc7801e3f0 con 0x55c4934682f0 2022-09-07T21:40:45.054 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.041+0000 7efc69ffb700 1 -- 172.21.15.47:0/724557156 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7800e610 con 0x55c4934682f0 2022-09-07T21:40:45.054 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.042+0000 7efca0f44b80 1 -- 172.21.15.47:0/724557156 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c4934682f0 msgr2=0x55c493468700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.054 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.042+0000 7efca0f44b80 1 --2- 172.21.15.47:0/724557156 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c4934682f0 0x55c493468700 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efc78004330 tx=0x7efc78014000 comp rx=0 tx=0).stop 2022-09-07T21:40:45.054 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.042+0000 7efca0f44b80 1 -- 172.21.15.47:0/724557156 shutdown_connections 2022-09-07T21:40:45.054 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.042+0000 7efca0f44b80 1 --2- 172.21.15.47:0/724557156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c493469a00 0x55c493469e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.042+0000 7efca0f44b80 1 --2- 172.21.15.47:0/724557156 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c4934682f0 0x55c493468700 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.042+0000 7efca0f44b80 1 --2- 172.21.15.47:0/724557156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 0x55c493469470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.042+0000 7efca0f44b80 1 -- 172.21.15.47:0/724557156 >> 172.21.15.47:0/724557156 conn(0x55c49345bfa0 msgr2=0x55c49345a370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.055 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.042+0000 7efca0f44b80 1 -- 172.21.15.47:0/724557156 shutdown_connections 2022-09-07T21:40:45.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.042+0000 7efca0f44b80 1 -- 172.21.15.47:0/724557156 wait complete. 2022-09-07T21:40:45.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.043+0000 7efca0f44b80 1 -- start start 2022-09-07T21:40:45.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.043+0000 7efca0f44b80 1 Processor -- start 2022-09-07T21:40:45.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.043+0000 7efca0f44b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c4934682f0 0x55c493471d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.056 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.043+0000 7efca0f44b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 0x55c493472250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.057 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efca0f44b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c493469a00 0x55c493479290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.057 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efca0f44b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49329c0c0 con 0x55c493469a00 2022-09-07T21:40:45.057 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efca0f44b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4934797d0 con 0x55c4934682f0 2022-09-07T21:40:45.057 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efca0f44b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c493479960 con 0x55c493468c40 2022-09-07T21:40:45.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efca0f44b80 10 client.0 _ll_register_callbacks cb 0x55c493359370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-07T21:40:45.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efc877fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c493469a00 0x55c493479290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efc877fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c493469a00 0x55c493479290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34158/0 (socket says 172.21.15.47:34158) 2022-09-07T21:40:45.058 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efc877fe700 1 -- 172.21.15.47:0/3760322085 learned_addr learned my addr 172.21.15.47:0/3760322085 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efc86ffd700 1 --2- 172.21.15.47:0/3760322085 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c4934682f0 0x55c493471d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.059 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efc867fc700 1 --2- 172.21.15.47:0/3760322085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 0x55c493472250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efc86ffd700 1 -- 172.21.15.47:0/3760322085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 msgr2=0x55c493472250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.062 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efc86ffd700 1 --2- 172.21.15.47:0/3760322085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 0x55c493472250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efc86ffd700 1 -- 172.21.15.47:0/3760322085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c493469a00 msgr2=0x55c493479290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efc86ffd700 1 --2- 172.21.15.47:0/3760322085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c493469a00 0x55c493479290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efc86ffd700 1 -- 172.21.15.47:0/3760322085 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc800099b0 con 0x55c4934682f0 2022-09-07T21:40:45.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.044+0000 7efc877fe700 1 --2- 172.21.15.47:0/3760322085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c493469a00 0x55c493479290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:45.063 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.045+0000 7efc86ffd700 1 --2- 172.21.15.47:0/3760322085 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c4934682f0 0x55c493471d10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efc780037e0 tx=0x7efc78003bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed87a09310636946 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.045+0000 7efc62ffd700 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2022-09-07T21:40:45.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.045+0000 7efc62ffd700 1 -- 172.21.15.47:0/3760322085 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7802b070 con 0x55c4934682f0 2022-09-07T21:40:45.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.045+0000 7efc62ffd700 1 -- 172.21.15.47:0/3760322085 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7efc78004360 con 0x55c4934682f0 2022-09-07T21:40:45.064 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.045+0000 7efca0f44b80 1 -- 172.21.15.47:0/3760322085 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c493452aa0 con 0x55c4934682f0 2022-09-07T21:40:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.045+0000 7efc62ffd700 1 -- 172.21.15.47:0/3760322085 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc7801ead0 con 0x55c4934682f0 2022-09-07T21:40:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.045+0000 7efca0f44b80 10 client.25253 fetch_fsmap learned FSMap version 24 2022-09-07T21:40:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.045+0000 7efca0f44b80 1 -- 172.21.15.47:0/3760322085 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7efc78017070 con 0x55c4934682f0 2022-09-07T21:40:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.046+0000 7efc62ffd700 1 -- 172.21.15.47:0/3760322085 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x55c493452aa0 con 0x55c4934682f0 2022-09-07T21:40:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.046+0000 7efc62ffd700 1 -- 172.21.15.47:0/3760322085 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 24) v1 ==== 7456+0+0 (secure 0 0 0) 0x7efc78024020 con 0x55c4934682f0 2022-09-07T21:40:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.046+0000 7efca0f44b80 10 client.25253 fetch_fsmap finished waiting for FSMap version 24 2022-09-07T21:40:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.046+0000 7efca0f44b80 10 client.25253 resolve_mds: resolved name 'i' to daemon mds.i 2022-09-07T21:40:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.046+0000 7efca0f44b80 20 client.25253 populate_metadata read hostname 'smithi047' 2022-09-07T21:40:45.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.046+0000 7efca0f44b80 1 --2- 172.21.15.47:0/3760322085 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x55c49347cf60 0x55c493482200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.046+0000 7efca0f44b80 4 client.25253 mds_command: new command op to 24556 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-07T21:40:45.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.046+0000 7efca0f44b80 1 -- 172.21.15.47:0/3760322085 --> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c49342f4c0 con 0x55c49347cf60 2022-09-07T21:40:45.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.046+0000 7efc867fc700 1 --2- 172.21.15.47:0/3760322085 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x55c49347cf60 0x55c493482200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.067 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.047+0000 7efc867fc700 1 --2- 172.21.15.47:0/3760322085 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x55c49347cf60 0x55c493482200 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f451b9047150958c server_cookie=d78dab984dc6112d in_seq=0 out_seq=0 2022-09-07T21:40:45.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.047+0000 7efc62ffd700 10 client.25253 ms_handle_connect on v2:172.21.15.59:6838/1006724740 2022-09-07T21:40:45.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.048+0000 7efc62ffd700 1 -- 172.21.15.47:0/3760322085 <== mds.2 v2:172.21.15.59:6838/1006724740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55c49342f4c0 con 0x55c49347cf60 2022-09-07T21:40:45.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.048+0000 7efc62ffd700 10 client.25253 handle_command_reply: tid=0 2022-09-07T21:40:45.068 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.048+0000 7efca0f44b80 1 client.25253 shutdown 2022-09-07T21:40:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.048+0000 7efca0f44b80 1 -- 172.21.15.47:0/3760322085 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c4934682f0 msgr2=0x55c493471d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.048+0000 7efca0f44b80 1 --2- 172.21.15.47:0/3760322085 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c4934682f0 0x55c493471d10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efc780037e0 tx=0x7efc78003bc0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.048+0000 7efca0f44b80 1 -- 172.21.15.47:0/3760322085 shutdown_connections 2022-09-07T21:40:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.048+0000 7efca0f44b80 1 --2- 172.21.15.47:0/3760322085 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x55c49347cf60 0x55c493482200 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.048+0000 7efca0f44b80 1 --2- 172.21.15.47:0/3760322085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c493469a00 0x55c493479290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.048+0000 7efca0f44b80 1 --2- 172.21.15.47:0/3760322085 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c4934682f0 0x55c493471d10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.048+0000 7efc62ffd700 0 client.25253 ms_handle_reset on v2:172.21.15.59:6838/1006724740 2022-09-07T21:40:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.048+0000 7efca0f44b80 1 --2- 172.21.15.47:0/3760322085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 0x55c493472250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.048+0000 7efca0f44b80 1 -- 172.21.15.47:0/3760322085 >> 172.21.15.47:0/3760322085 conn(0x55c49345bfa0 msgr2=0x55c493459a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.048+0000 7efca0f44b80 1 -- 172.21.15.47:0/3760322085 shutdown_connections 2022-09-07T21:40:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.048+0000 7efca0f44b80 1 -- 172.21.15.47:0/3760322085 wait complete. 2022-09-07T21:40:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.049+0000 7efca0f44b80 20 client.25253 trim_cache size 0 max 16384 2022-09-07T21:40:45.078 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.066+0000 7efca0f44b80 1 Processor -- start 2022-09-07T21:40:45.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.067+0000 7efca0f44b80 1 -- start start 2022-09-07T21:40:45.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.067+0000 7efca0f44b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c4934682f0 0x55c49346c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.067+0000 7efca0f44b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c493468c40 0x55c49346fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.067+0000 7efca0f44b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493469a00 0x55c49346fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.079 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.067+0000 7efca0f44b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c493479670 con 0x55c4934682f0 2022-09-07T21:40:45.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.067+0000 7efca0f44b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c493470530 con 0x55c493468c40 2022-09-07T21:40:45.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.067+0000 7efca0f44b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49342f4c0 con 0x55c493469a00 2022-09-07T21:40:45.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.067+0000 7efc867fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c493468c40 0x55c49346fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.080 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.067+0000 7efc867fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c493468c40 0x55c49346fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:40124/0 (socket says 172.21.15.47:40124) 2022-09-07T21:40:45.081 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.067+0000 7efc867fc700 1 -- 172.21.15.47:0/1130135306 learned_addr learned my addr 172.21.15.47:0/1130135306 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.081 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.067+0000 7efc877fe700 1 --2- 172.21.15.47:0/1130135306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493469a00 0x55c49346fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.081 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.068+0000 7efc86ffd700 1 --2- 172.21.15.47:0/1130135306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c4934682f0 0x55c49346c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.081 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.068+0000 7efc867fc700 1 -- 172.21.15.47:0/1130135306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493469a00 msgr2=0x55c49346fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.081 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.068+0000 7efc867fc700 1 --2- 172.21.15.47:0/1130135306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493469a00 0x55c49346fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.068+0000 7efc867fc700 1 -- 172.21.15.47:0/1130135306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c4934682f0 msgr2=0x55c49346c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.068+0000 7efc867fc700 1 --2- 172.21.15.47:0/1130135306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c4934682f0 0x55c49346c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.068+0000 7efc867fc700 1 -- 172.21.15.47:0/1130135306 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4931e2d90 con 0x55c493468c40 2022-09-07T21:40:45.082 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.068+0000 7efc877fe700 1 --2- 172.21.15.47:0/1130135306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493469a00 0x55c49346fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:45.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.068+0000 7efc867fc700 1 --2- 172.21.15.47:0/1130135306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c493468c40 0x55c49346fab0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efc7c028000 tx=0x7efc7c009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90650782f00e9486 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.068+0000 7efc6a7fc700 1 -- 172.21.15.47:0/1130135306 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c09a6c0 con 0x55c493468c40 2022-09-07T21:40:45.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.068+0000 7efc6a7fc700 1 -- 172.21.15.47:0/1130135306 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7efc7c0a9070 con 0x55c493468c40 2022-09-07T21:40:45.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.069+0000 7efc6a7fc700 1 -- 172.21.15.47:0/1130135306 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c08fb30 con 0x55c493468c40 2022-09-07T21:40:45.083 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.069+0000 7efca0f44b80 1 -- 172.21.15.47:0/1130135306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c493468c40 msgr2=0x55c49346fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.069+0000 7efca0f44b80 1 --2- 172.21.15.47:0/1130135306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c493468c40 0x55c49346fab0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efc7c028000 tx=0x7efc7c009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:45.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.069+0000 7efca0f44b80 1 -- 172.21.15.47:0/1130135306 shutdown_connections 2022-09-07T21:40:45.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.069+0000 7efca0f44b80 1 --2- 172.21.15.47:0/1130135306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493469a00 0x55c49346fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.069+0000 7efca0f44b80 1 --2- 172.21.15.47:0/1130135306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c4934682f0 0x55c49346c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.084 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.069+0000 7efca0f44b80 1 --2- 172.21.15.47:0/1130135306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c493468c40 0x55c49346fab0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.085 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.069+0000 7efca0f44b80 1 -- 172.21.15.47:0/1130135306 >> 172.21.15.47:0/1130135306 conn(0x55c49345bfa0 msgr2=0x55c493479ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.085 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.069+0000 7efca0f44b80 1 -- 172.21.15.47:0/1130135306 shutdown_connections 2022-09-07T21:40:45.085 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.069+0000 7efca0f44b80 1 -- 172.21.15.47:0/1130135306 wait complete. 2022-09-07T21:40:45.085 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.070+0000 7efca0f44b80 1 -- start start 2022-09-07T21:40:45.085 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.070+0000 7efca0f44b80 1 Processor -- start 2022-09-07T21:40:45.086 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.070+0000 7efca0f44b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c4934682f0 0x55c4934759e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.086 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.070+0000 7efca0f44b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 0x55c493475f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.086 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.070+0000 7efca0f44b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c493469a00 0x55c493476460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.086 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.071+0000 7efca0f44b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c493477520 con 0x55c493469a00 2022-09-07T21:40:45.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.071+0000 7efca0f44b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4934cf1d0 con 0x55c4934682f0 2022-09-07T21:40:45.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.071+0000 7efca0f44b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4934cf390 con 0x55c493468c40 2022-09-07T21:40:45.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.071+0000 7efca0f44b80 10 client.0 _ll_register_callbacks cb 0x55c493359370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-07T21:40:45.087 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.071+0000 7efc86ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c4934682f0 0x55c4934759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.071+0000 7efc877fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c493469a00 0x55c493476460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.071+0000 7efc867fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 0x55c493475f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.071+0000 7efc867fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 0x55c493475f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:56564/0 (socket says 172.21.15.47:56564) 2022-09-07T21:40:45.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.071+0000 7efc867fc700 1 -- 172.21.15.47:0/935525394 learned_addr learned my addr 172.21.15.47:0/935525394 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.088 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.071+0000 7efc867fc700 1 -- 172.21.15.47:0/935525394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c4934682f0 msgr2=0x55c4934759e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.071+0000 7efc867fc700 1 --2- 172.21.15.47:0/935525394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c4934682f0 0x55c4934759e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.071+0000 7efc867fc700 1 -- 172.21.15.47:0/935525394 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c493469a00 msgr2=0x55c493476460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.071+0000 7efc867fc700 1 --2- 172.21.15.47:0/935525394 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c493469a00 0x55c493476460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.089 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.071+0000 7efc867fc700 1 -- 172.21.15.47:0/935525394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc80009d80 con 0x55c493468c40 2022-09-07T21:40:45.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.072+0000 7efc867fc700 1 --2- 172.21.15.47:0/935525394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 0x55c493475f20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efc7c08fe90 tx=0x7efc7c0c0d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a1b3f42fd3feecc server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.090 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.072+0000 7efc62ffd700 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2022-09-07T21:40:45.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.072+0000 7efc62ffd700 1 -- 172.21.15.47:0/935525394 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c0ae070 con 0x55c493468c40 2022-09-07T21:40:45.091 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.072+0000 7efca0f44b80 1 -- 172.21.15.47:0/935525394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efc7801e3f0 con 0x55c493468c40 2022-09-07T21:40:45.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.072+0000 7efc62ffd700 1 -- 172.21.15.47:0/935525394 <== mon.1 v2:172.21.15.59:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7efc7c0c3110 con 0x55c493468c40 2022-09-07T21:40:45.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.072+0000 7efc62ffd700 1 -- 172.21.15.47:0/935525394 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc7c08fb30 con 0x55c493468c40 2022-09-07T21:40:45.092 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.072+0000 7efca0f44b80 10 client.25288 fetch_fsmap learned FSMap version 24 2022-09-07T21:40:45.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.072+0000 7efca0f44b80 1 -- 172.21.15.47:0/935525394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7efc8819e120 con 0x55c493468c40 2022-09-07T21:40:45.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.072+0000 7efc62ffd700 1 -- 172.21.15.47:0/935525394 <== mon.1 v2:172.21.15.59:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7efc7801e3f0 con 0x55c493468c40 2022-09-07T21:40:45.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.073+0000 7efc62ffd700 1 -- 172.21.15.47:0/935525394 <== mon.1 v2:172.21.15.59:3300/0 5 ==== fsmap(e 24) v1 ==== 7456+0+0 (secure 0 0 0) 0x7efc7c05dd80 con 0x55c493468c40 2022-09-07T21:40:45.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.073+0000 7efca0f44b80 10 client.25288 fetch_fsmap finished waiting for FSMap version 24 2022-09-07T21:40:45.093 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.073+0000 7efca0f44b80 10 client.25288 resolve_mds: resolved name 'i' to daemon mds.i 2022-09-07T21:40:45.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.073+0000 7efca0f44b80 20 client.25288 populate_metadata read hostname 'smithi047' 2022-09-07T21:40:45.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.073+0000 7efca0f44b80 1 --2- 172.21.15.47:0/935525394 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x55c4934d3cc0 0x55c4934d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.073+0000 7efca0f44b80 4 client.25288 mds_command: new command op to 24556 tid=0[{"prefix": "status"}] 2022-09-07T21:40:45.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.073+0000 7efca0f44b80 1 -- 172.21.15.47:0/935525394 --> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c4933a76f0 con 0x55c4934d3cc0 2022-09-07T21:40:45.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.073+0000 7efc86ffd700 1 --2- 172.21.15.47:0/935525394 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x55c4934d3cc0 0x55c4934d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efc86ffd700 1 --2- 172.21.15.47:0/935525394 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x55c4934d3cc0 0x55c4934d0670 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f5e15adf64a87174 server_cookie=c6da5ad1b79ad185 in_seq=0 out_seq=0 2022-09-07T21:40:45.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efc62ffd700 10 client.25288 ms_handle_connect on v2:172.21.15.59:6838/1006724740 2022-09-07T21:40:45.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efc62ffd700 1 -- 172.21.15.47:0/935525394 <== mds.2 v2:172.21.15.59:6838/1006724740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55c4933a76f0 con 0x55c4934d3cc0 2022-09-07T21:40:45.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efc62ffd700 10 client.25288 handle_command_reply: tid=0 2022-09-07T21:40:45.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efca0f44b80 1 client.25288 shutdown 2022-09-07T21:40:45.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efca0f44b80 1 -- 172.21.15.47:0/935525394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 msgr2=0x55c493475f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.097 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efca0f44b80 1 --2- 172.21.15.47:0/935525394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 0x55c493475f20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efc7c08fe90 tx=0x7efc7c0c0d10 comp rx=0 tx=0).stop 2022-09-07T21:40:45.097 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efca0f44b80 1 -- 172.21.15.47:0/935525394 shutdown_connections 2022-09-07T21:40:45.097 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efca0f44b80 1 --2- 172.21.15.47:0/935525394 >> [v2:172.21.15.59:6838/1006724740,v1:172.21.15.59:6839/1006724740] conn(0x55c4934d3cc0 0x55c4934d0670 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.097 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efca0f44b80 1 --2- 172.21.15.47:0/935525394 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c493469a00 0x55c493476460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.097 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efca0f44b80 1 --2- 172.21.15.47:0/935525394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c4934682f0 0x55c4934759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.098 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efc62ffd700 0 client.25288 ms_handle_reset on v2:172.21.15.59:6838/1006724740 2022-09-07T21:40:45.098 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efca0f44b80 1 --2- 172.21.15.47:0/935525394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c493468c40 0x55c493475f20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.098 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efca0f44b80 1 -- 172.21.15.47:0/935525394 >> 172.21.15.47:0/935525394 conn(0x55c49345bfa0 msgr2=0x55c49346c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.098 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efca0f44b80 1 -- 172.21.15.47:0/935525394 shutdown_connections 2022-09-07T21:40:45.098 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.074+0000 7efca0f44b80 1 -- 172.21.15.47:0/935525394 wait complete. 2022-09-07T21:40:45.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.075+0000 7efca0f44b80 20 client.25288 trim_cache size 0 max 16384 2022-09-07T21:40:45.099 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2022-09-07T21:40:45.099 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "01b8396e-2ef4-11ed-8431-001a4aab830c", 2022-09-07T21:40:45.100 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 2, 2022-09-07T21:40:45.100 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 24556, 2022-09-07T21:40:45.100 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2022-09-07T21:40:45.100 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2022-09-07T21:40:45.101 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2022-09-07T21:40:45.101 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 99.261204555999996, 2022-09-07T21:40:45.101 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 24, 2022-09-07T21:40:45.101 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 91, 2022-09-07T21:40:45.101 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 89, 2022-09-07T21:40:45.102 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 140.39364978200001 2022-09-07T21:40:45.102 INFO:teuthology.orchestra.run.smithi047.stdout:} 2022-09-07T21:40:45.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.075+0000 7efc8d9ff700 1 -- 172.21.15.47:0/125358825 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7efc70099010 msgr2=0x7efc7009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.075+0000 7efc8d9ff700 1 --2- 172.21.15.47:0/125358825 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7efc70099010 0x7efc7009b4c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efc78005fd0 tx=0x7efc78005ea0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.075+0000 7efc8d9ff700 1 -- 172.21.15.47:0/125358825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc880fef90 msgr2=0x7efc88195fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.075+0000 7efc8d9ff700 1 --2- 172.21.15.47:0/125358825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc880fef90 0x7efc88195fd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efc7c00f6c0 tx=0x7efc7c00f6f0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.075+0000 7efc8d9ff700 1 -- 172.21.15.47:0/125358825 shutdown_connections 2022-09-07T21:40:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.075+0000 7efc8d9ff700 1 --2- 172.21.15.47:0/125358825 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7efc70099010 0x7efc7009b4c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.075+0000 7efc8d9ff700 1 --2- 172.21.15.47:0/125358825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc880ff940 0x7efc8819d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.075+0000 7efc8d9ff700 1 --2- 172.21.15.47:0/125358825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc880fddb0 0x7efc88195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.105 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.075+0000 7efc8d9ff700 1 --2- 172.21.15.47:0/125358825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc880fef90 0x7efc88195fd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.105 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.075+0000 7efc8d9ff700 1 -- 172.21.15.47:0/125358825 >> 172.21.15.47:0/125358825 conn(0x7efc880f9460 msgr2=0x7efc881048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.105 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.075+0000 7efc8d9ff700 1 -- 172.21.15.47:0/125358825 shutdown_connections 2022-09-07T21:40:45.105 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.076+0000 7efc8d9ff700 1 -- 172.21.15.47:0/125358825 wait complete. 2022-09-07T21:40:45.106 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:44 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3228083687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:40:45.106 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:44 smithi047 ceph-mon[90087]: pgmap v380: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2022-09-07T21:40:45.106 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:44 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/1645855249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:40:45.107 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2022-09-07T21:40:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:44 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3228083687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:40:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:44 smithi055 ceph-mon[97033]: pgmap v380: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2022-09-07T21:40:45.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:44 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/1645855249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:40:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:44 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3228083687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-07T21:40:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:44 smithi059 ceph-mon[97450]: pgmap v380: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2022-09-07T21:40:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:44 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/1645855249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-07T21:40:45.342 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.338+0000 7f4e25fff700 1 -- 172.21.15.47:0/2016028900 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e20066be0 msgr2=0x7f4e20100960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.342 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.338+0000 7f4e25fff700 1 --2- 172.21.15.47:0/2016028900 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e20066be0 0x7f4e20100960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4e10009750 tx=0x7f4e10009a60 comp rx=0 tx=0).stop 2022-09-07T21:40:45.343 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.338+0000 7f4e25fff700 1 -- 172.21.15.47:0/2016028900 shutdown_connections 2022-09-07T21:40:45.343 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.338+0000 7f4e25fff700 1 --2- 172.21.15.47:0/2016028900 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e20103990 0x7f4e20105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.343 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.338+0000 7f4e25fff700 1 --2- 172.21.15.47:0/2016028900 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e20066be0 0x7f4e20100960 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.338+0000 7f4e25fff700 1 --2- 172.21.15.47:0/2016028900 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e20100ea0 0x7f4e20103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.338+0000 7f4e25fff700 1 -- 172.21.15.47:0/2016028900 >> 172.21.15.47:0/2016028900 conn(0x7f4e200f8370 msgr2=0x7f4e200fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.338+0000 7f4e25fff700 1 -- 172.21.15.47:0/2016028900 shutdown_connections 2022-09-07T21:40:45.344 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.339+0000 7f4e25fff700 1 -- 172.21.15.47:0/2016028900 wait complete. 2022-09-07T21:40:45.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.340+0000 7f4e25fff700 1 Processor -- start 2022-09-07T21:40:45.347 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.340+0000 7f4e25fff700 1 -- start start 2022-09-07T21:40:45.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.340+0000 7f4e25fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e20066be0 0x7f4e20195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.340+0000 7f4e25fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e20100ea0 0x7f4e201960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.340+0000 7f4e25fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e20103990 0x7f4e2019d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.348 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.340+0000 7f4e25fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2019d710 con 0x7f4e20066be0 2022-09-07T21:40:45.349 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.340+0000 7f4e25fff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2019d880 con 0x7f4e20103990 2022-09-07T21:40:45.349 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.340+0000 7f4e25fff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2019da40 con 0x7f4e20100ea0 2022-09-07T21:40:45.350 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.341+0000 7f4e1f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e20066be0 0x7f4e20195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.350 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.341+0000 7f4e1ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e20103990 0x7f4e2019d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.351 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.341+0000 7f4e1effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e20100ea0 0x7f4e201960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.351 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.341+0000 7f4e1ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e20103990 0x7f4e2019d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:40156/0 (socket says 172.21.15.47:40156) 2022-09-07T21:40:45.352 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.341+0000 7f4e1ffff700 1 -- 172.21.15.47:0/59673809 learned_addr learned my addr 172.21.15.47:0/59673809 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.352 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.341+0000 7f4e1ffff700 1 -- 172.21.15.47:0/59673809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e20100ea0 msgr2=0x7f4e201960d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.352 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.341+0000 7f4e1ffff700 1 --2- 172.21.15.47:0/59673809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e20100ea0 0x7f4e201960d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.341+0000 7f4e1ffff700 1 -- 172.21.15.47:0/59673809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e20066be0 msgr2=0x7f4e20195b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.341+0000 7f4e1ffff700 1 --2- 172.21.15.47:0/59673809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e20066be0 0x7f4e20195b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.354 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.341+0000 7f4e1ffff700 1 -- 172.21.15.47:0/59673809 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e100093b0 con 0x7f4e20103990 2022-09-07T21:40:45.354 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.341+0000 7f4e1ffff700 1 --2- 172.21.15.47:0/59673809 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e20103990 0x7f4e2019d140 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800ebf0 tx=0x7f4e1800ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.354 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.342+0000 7f4e1cff9700 1 -- 172.21.15.47:0/59673809 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e180099c0 con 0x7f4e20103990 2022-09-07T21:40:45.355 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.342+0000 7f4e25fff700 1 -- 172.21.15.47:0/59673809 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2019dd20 con 0x7f4e20103990 2022-09-07T21:40:45.355 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.342+0000 7f4e1cff9700 1 -- 172.21.15.47:0/59673809 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4e1800cc80 con 0x7f4e20103990 2022-09-07T21:40:45.355 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.342+0000 7f4e1cff9700 1 -- 172.21.15.47:0/59673809 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18005920 con 0x7f4e20103990 2022-09-07T21:40:45.356 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.342+0000 7f4e25fff700 1 -- 172.21.15.47:0/59673809 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2019e1d0 con 0x7f4e20103990 2022-09-07T21:40:45.363 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.346+0000 7f4e1cff9700 1 -- 172.21.15.47:0/59673809 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f4e1801e020 con 0x7f4e20103990 2022-09-07T21:40:45.364 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.347+0000 7f4e3944bb80 1 Processor -- start 2022-09-07T21:40:45.364 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.347+0000 7f4e3944bb80 1 -- start start 2022-09-07T21:40:45.364 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.348+0000 7f4e3944bb80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x5561468fdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.364 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.348+0000 7f4e3944bb80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fe430 0x5561468fec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.365 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.348+0000 7f4e3944bb80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468ff1f0 0x5561468ff640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.365 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.348+0000 7f4e3944bb80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561467e8080 con 0x5561468fe430 2022-09-07T21:40:45.365 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.348+0000 7f4e3944bb80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561468529e0 con 0x5561468fdae0 2022-09-07T21:40:45.365 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.348+0000 7f4e3944bb80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55614684c180 con 0x5561468ff1f0 2022-09-07T21:40:45.366 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.348+0000 7f4e1f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fe430 0x5561468fec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.366 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.348+0000 7f4e1effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x5561468fdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.366 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.348+0000 7f4e1effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x5561468fdef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:40170/0 (socket says 172.21.15.47:40170) 2022-09-07T21:40:45.367 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.348+0000 7f4e1f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fe430 0x5561468fec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34184/0 (socket says 172.21.15.47:34184) 2022-09-07T21:40:45.367 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.348+0000 7f4e1effd700 1 -- 172.21.15.47:0/1149561479 learned_addr learned my addr 172.21.15.47:0/1149561479 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.367 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.348+0000 7f4e1effd700 1 --2- 172.21.15.47:0/1149561479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468ff1f0 0x5561468ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.367 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.349+0000 7f4e1cff9700 1 --2- 172.21.15.47:0/59673809 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4e08099010 0x7f4e0809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.367 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.349+0000 7f4e1f7fe700 1 -- 172.21.15.47:0/1149561479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468ff1f0 msgr2=0x5561468ff640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.368 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.349+0000 7f4e1f7fe700 1 --2- 172.21.15.47:0/1149561479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468ff1f0 0x5561468ff640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.368 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.349+0000 7f4e1f7fe700 1 -- 172.21.15.47:0/1149561479 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 msgr2=0x5561468fdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.368 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.349+0000 7f4e1f7fe700 1 --2- 172.21.15.47:0/1149561479 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x5561468fdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.368 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.349+0000 7f4e1f7fe700 1 -- 172.21.15.47:0/1149561479 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561466d45e0 con 0x5561468fe430 2022-09-07T21:40:45.369 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.349+0000 7f4e1effd700 1 --2- 172.21.15.47:0/1149561479 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x5561468fdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:45.369 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.349+0000 7f4e1cff9700 1 -- 172.21.15.47:0/59673809 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f4e18014070 con 0x7f4e20103990 2022-09-07T21:40:45.369 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.349+0000 7f4e1ffff700 1 --2- 172.21.15.47:0/59673809 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4e08099010 0x7f4e0809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.369 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.349+0000 7f4e1f7fe700 1 --2- 172.21.15.47:0/1149561479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fe430 0x5561468fec60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4e10000c00 tx=0x7f4e10005ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed36aa4b8b1c191 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.370 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.350+0000 7f4e05ffb700 1 -- 172.21.15.47:0/1149561479 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1001d070 con 0x5561468fe430 2022-09-07T21:40:45.370 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.350+0000 7f4e05ffb700 1 -- 172.21.15.47:0/1149561479 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4e1001e040 con 0x5561468fe430 2022-09-07T21:40:45.370 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.350+0000 7f4e05ffb700 1 -- 172.21.15.47:0/1149561479 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e10004780 con 0x5561468fe430 2022-09-07T21:40:45.370 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.350+0000 7f4e3944bb80 1 -- 172.21.15.47:0/1149561479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fe430 msgr2=0x5561468fec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.371 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.350+0000 7f4e1ffff700 1 --2- 172.21.15.47:0/59673809 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4e08099010 0x7f4e0809b4c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4e1808f1c0 tx=0x7f4e18009f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.371 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.350+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/1149561479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fe430 0x5561468fec60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4e10000c00 tx=0x7f4e10005ca0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.371 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.350+0000 7f4e3944bb80 1 -- 172.21.15.47:0/1149561479 shutdown_connections 2022-09-07T21:40:45.371 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.350+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/1149561479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468ff1f0 0x5561468ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.372 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.350+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/1149561479 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x5561468fdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.372 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.350+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/1149561479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fe430 0x5561468fec60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.372 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.350+0000 7f4e3944bb80 1 -- 172.21.15.47:0/1149561479 >> 172.21.15.47:0/1149561479 conn(0x5561468f9fa0 msgr2=0x5561468f8370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.372 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.351+0000 7f4e3944bb80 1 -- 172.21.15.47:0/1149561479 shutdown_connections 2022-09-07T21:40:45.373 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.351+0000 7f4e3944bb80 1 -- 172.21.15.47:0/1149561479 wait complete. 2022-09-07T21:40:45.373 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.351+0000 7f4e3944bb80 1 -- start start 2022-09-07T21:40:45.373 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.351+0000 7f4e3944bb80 1 Processor -- start 2022-09-07T21:40:45.374 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.352+0000 7f4e3944bb80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fdae0 0x556146907430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.374 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.352+0000 7f4e3944bb80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468fe430 0x556146907970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.374 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.352+0000 7f4e3944bb80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468ff1f0 0x55614690e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.374 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.352+0000 7f4e3944bb80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55614673a0c0 con 0x5561468fdae0 2022-09-07T21:40:45.374 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.352+0000 7f4e3944bb80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55614690eef0 con 0x5561468ff1f0 2022-09-07T21:40:45.375 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.352+0000 7f4e3944bb80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55614690f080 con 0x5561468fe430 2022-09-07T21:40:45.375 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.352+0000 7f4e3944bb80 10 client.0 _ll_register_callbacks cb 0x5561467f7370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-07T21:40:45.375 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.352+0000 7f4e1effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fdae0 0x556146907430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.375 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.352+0000 7f4e1f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468fe430 0x556146907970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.376 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.352+0000 7f4e1effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fdae0 0x556146907430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34198/0 (socket says 172.21.15.47:34198) 2022-09-07T21:40:45.376 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.352+0000 7f4e1effd700 1 -- 172.21.15.47:0/1163587762 learned_addr learned my addr 172.21.15.47:0/1163587762 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.376 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.353+0000 7f4e1effd700 1 --2- 172.21.15.47:0/1163587762 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468ff1f0 0x55614690e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.376 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.353+0000 7f4e1effd700 1 -- 172.21.15.47:0/1163587762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468fe430 msgr2=0x556146907970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.376 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.353+0000 7f4e1effd700 1 --2- 172.21.15.47:0/1163587762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468fe430 0x556146907970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.377 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.353+0000 7f4e1effd700 1 -- 172.21.15.47:0/1163587762 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468ff1f0 msgr2=0x55614690e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.377 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.353+0000 7f4e1effd700 1 --2- 172.21.15.47:0/1163587762 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468ff1f0 0x55614690e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.377 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.353+0000 7f4e1effd700 1 -- 172.21.15.47:0/1163587762 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e14019040 con 0x5561468fdae0 2022-09-07T21:40:45.377 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.353+0000 7f4e1f7fe700 1 --2- 172.21.15.47:0/1163587762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468fe430 0x556146907970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:45.378 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.353+0000 7f4e1effd700 1 --2- 172.21.15.47:0/1163587762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fdae0 0x556146907430 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4e14012bd0 tx=0x7f4e14004640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f021371868cf3f08 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.378 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.353+0000 7f4dfaffd700 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2022-09-07T21:40:45.378 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.353+0000 7f4dfaffd700 1 -- 172.21.15.47:0/1163587762 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e140040f0 con 0x5561468fdae0 2022-09-07T21:40:45.378 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.353+0000 7f4dfaffd700 1 -- 172.21.15.47:0/1163587762 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f4e14004250 con 0x5561468fdae0 2022-09-07T21:40:45.378 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.353+0000 7f4e3944bb80 1 -- 172.21.15.47:0/1163587762 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5561468f0aa0 con 0x5561468fdae0 2022-09-07T21:40:45.379 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.354+0000 7f4dfaffd700 1 -- 172.21.15.47:0/1163587762 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e14004410 con 0x5561468fdae0 2022-09-07T21:40:45.379 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.354+0000 7f4e3944bb80 10 client.15201 fetch_fsmap learned FSMap version 24 2022-09-07T21:40:45.379 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.354+0000 7f4e3944bb80 1 -- 172.21.15.47:0/1163587762 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7f4e20106ad0 con 0x5561468fdae0 2022-09-07T21:40:45.379 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.354+0000 7f4dfaffd700 1 -- 172.21.15.47:0/1163587762 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x5561468f0aa0 con 0x5561468fdae0 2022-09-07T21:40:45.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.354+0000 7f4dfaffd700 1 -- 172.21.15.47:0/1163587762 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 24) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f4e14003c00 con 0x5561468fdae0 2022-09-07T21:40:45.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.354+0000 7f4e3944bb80 10 client.15201 fetch_fsmap finished waiting for FSMap version 24 2022-09-07T21:40:45.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.354+0000 7f4e3944bb80 10 client.15201 resolve_mds: resolved name 'b' to daemon mds.b 2022-09-07T21:40:45.380 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.354+0000 7f4e3944bb80 20 client.15201 populate_metadata read hostname 'smithi047' 2022-09-07T21:40:45.381 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.354+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/1163587762 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x5561469126e0 0x5561469179e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.381 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.354+0000 7f4e3944bb80 4 client.15201 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-07T21:40:45.381 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.354+0000 7f4e3944bb80 1 -- 172.21.15.47:0/1163587762 --> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5561468cd4c0 con 0x5561469126e0 2022-09-07T21:40:45.381 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.355+0000 7f4e1f7fe700 1 --2- 172.21.15.47:0/1163587762 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x5561469126e0 0x5561469179e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.381 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.355+0000 7f4e1f7fe700 1 --2- 172.21.15.47:0/1163587762 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x5561469126e0 0x5561469179e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f2396cabe7393851 server_cookie=768ae4c0d736a4a1 in_seq=0 out_seq=0 2022-09-07T21:40:45.382 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.355+0000 7f4dfaffd700 10 client.15201 ms_handle_connect on v2:172.21.15.55:6838/517990300 2022-09-07T21:40:45.382 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.356+0000 7f4dfaffd700 1 -- 172.21.15.47:0/1163587762 <== mds.1 v2:172.21.15.55:6838/517990300 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x5561468cd4c0 con 0x5561469126e0 2022-09-07T21:40:45.382 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.356+0000 7f4dfaffd700 10 client.15201 handle_command_reply: tid=0 2022-09-07T21:40:45.382 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.356+0000 7f4e3944bb80 1 client.15201 shutdown 2022-09-07T21:40:45.383 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.357+0000 7f4e3944bb80 1 -- 172.21.15.47:0/1163587762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fdae0 msgr2=0x556146907430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.383 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.357+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/1163587762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fdae0 0x556146907430 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4e14012bd0 tx=0x7f4e14004640 comp rx=0 tx=0).stop 2022-09-07T21:40:45.383 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.357+0000 7f4e3944bb80 1 -- 172.21.15.47:0/1163587762 shutdown_connections 2022-09-07T21:40:45.383 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.357+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/1163587762 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x5561469126e0 0x5561469179e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.357+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/1163587762 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468ff1f0 0x55614690e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.357+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/1163587762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fdae0 0x556146907430 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.357+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/1163587762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468fe430 0x556146907970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.357+0000 7f4dfaffd700 0 client.15201 ms_handle_reset on v2:172.21.15.55:6838/517990300 2022-09-07T21:40:45.384 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.357+0000 7f4e3944bb80 1 -- 172.21.15.47:0/1163587762 >> 172.21.15.47:0/1163587762 conn(0x5561468f9fa0 msgr2=0x5561468f7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.357+0000 7f4e3944bb80 1 -- 172.21.15.47:0/1163587762 shutdown_connections 2022-09-07T21:40:45.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.357+0000 7f4e3944bb80 1 -- 172.21.15.47:0/1163587762 wait complete. 2022-09-07T21:40:45.385 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.357+0000 7f4e3944bb80 20 client.15201 trim_cache size 0 max 16384 2022-09-07T21:40:45.391 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.375+0000 7f4e3944bb80 1 Processor -- start 2022-09-07T21:40:45.391 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.376+0000 7f4e3944bb80 1 -- start start 2022-09-07T21:40:45.392 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.376+0000 7f4e3944bb80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x556146906440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.392 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.376+0000 7f4e3944bb80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468fe430 0x5561468f8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.392 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.376+0000 7f4e3944bb80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468ff1f0 0x5561468f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.392 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.376+0000 7f4e3944bb80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x556146810050 con 0x5561468ff1f0 2022-09-07T21:40:45.392 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.376+0000 7f4e3944bb80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561467ee4f0 con 0x5561468fdae0 2022-09-07T21:40:45.393 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.376+0000 7f4e3944bb80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561466d8a30 con 0x5561468fe430 2022-09-07T21:40:45.393 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.376+0000 7f4e1effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468ff1f0 0x5561468f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.393 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.376+0000 7f4e1f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468fe430 0x5561468f8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.393 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.376+0000 7f4e1f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468fe430 0x5561468f8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:56606/0 (socket says 172.21.15.47:56606) 2022-09-07T21:40:45.394 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.376+0000 7f4e1f7fe700 1 -- 172.21.15.47:0/2529031465 learned_addr learned my addr 172.21.15.47:0/2529031465 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.394 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.376+0000 7f4e1effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468ff1f0 0x5561468f7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34212/0 (socket says 172.21.15.47:34212) 2022-09-07T21:40:45.394 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.376+0000 7f4e1effd700 1 --2- 172.21.15.47:0/2529031465 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x556146906440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.395 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.377+0000 7f4e1f7fe700 1 -- 172.21.15.47:0/2529031465 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 msgr2=0x556146906440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.395 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.377+0000 7f4e1f7fe700 1 --2- 172.21.15.47:0/2529031465 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x556146906440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.395 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.377+0000 7f4e1f7fe700 1 -- 172.21.15.47:0/2529031465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468ff1f0 msgr2=0x5561468f7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.395 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.377+0000 7f4e1f7fe700 1 --2- 172.21.15.47:0/2529031465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468ff1f0 0x5561468f7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.395 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.377+0000 7f4e1f7fe700 1 -- 172.21.15.47:0/2529031465 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556146680d90 con 0x5561468fe430 2022-09-07T21:40:45.396 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.377+0000 7f4e1effd700 1 --2- 172.21.15.47:0/2529031465 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x556146906440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:40:45.396 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.377+0000 7f4e1f7fe700 1 --2- 172.21.15.47:0/2529031465 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468fe430 0x5561468f8340 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4e10000c00 tx=0x7f4e10009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9e9a27e2bb5836 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.396 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.377+0000 7f4e067fc700 1 -- 172.21.15.47:0/2529031465 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1001d040 con 0x5561468fe430 2022-09-07T21:40:45.396 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.377+0000 7f4e067fc700 1 -- 172.21.15.47:0/2529031465 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4e10005760 con 0x5561468fe430 2022-09-07T21:40:45.396 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.377+0000 7f4e067fc700 1 -- 172.21.15.47:0/2529031465 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e10033b80 con 0x5561468fe430 2022-09-07T21:40:45.397 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.378+0000 7f4e3944bb80 1 -- 172.21.15.47:0/2529031465 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468fe430 msgr2=0x5561468f8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.397 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.378+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/2529031465 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468fe430 0x5561468f8340 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4e10000c00 tx=0x7f4e10009f60 comp rx=0 tx=0).stop 2022-09-07T21:40:45.397 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.378+0000 7f4e3944bb80 1 -- 172.21.15.47:0/2529031465 shutdown_connections 2022-09-07T21:40:45.397 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.378+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/2529031465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468ff1f0 0x5561468f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.398 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.378+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/2529031465 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x556146906440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.398 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.378+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/2529031465 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468fe430 0x5561468f8340 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.398 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.378+0000 7f4e3944bb80 1 -- 172.21.15.47:0/2529031465 >> 172.21.15.47:0/2529031465 conn(0x5561468f9fa0 msgr2=0x5561468f4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.398 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.378+0000 7f4e3944bb80 1 -- 172.21.15.47:0/2529031465 shutdown_connections 2022-09-07T21:40:45.399 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.378+0000 7f4e3944bb80 1 -- 172.21.15.47:0/2529031465 wait complete. 2022-09-07T21:40:45.399 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.379+0000 7f4e3944bb80 1 -- start start 2022-09-07T21:40:45.399 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.379+0000 7f4e3944bb80 1 Processor -- start 2022-09-07T21:40:45.399 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.379+0000 7f4e3944bb80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x55614690c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.400 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.379+0000 7f4e3944bb80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fe430 0x55614690c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.400 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.379+0000 7f4e3944bb80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468ff1f0 0x55614690a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.400 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.379+0000 7f4e3944bb80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55614690ad70 con 0x5561468fe430 2022-09-07T21:40:45.400 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.379+0000 7f4e3944bb80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x55614690aee0 con 0x5561468fdae0 2022-09-07T21:40:45.401 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.379+0000 7f4e3944bb80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55614690b0a0 con 0x5561468ff1f0 2022-09-07T21:40:45.401 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.379+0000 7f4e3944bb80 10 client.0 _ll_register_callbacks cb 0x5561467f7370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-07T21:40:45.401 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.380+0000 7f4e1f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fe430 0x55614690c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.401 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.380+0000 7f4e1f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fe430 0x55614690c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34218/0 (socket says 172.21.15.47:34218) 2022-09-07T21:40:45.402 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.380+0000 7f4e1f7fe700 1 -- 172.21.15.47:0/3572263133 learned_addr learned my addr 172.21.15.47:0/3572263133 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.402 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.380+0000 7f4e1effd700 1 --2- 172.21.15.47:0/3572263133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468ff1f0 0x55614690a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.402 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.380+0000 7f4e1effd700 1 --2- 172.21.15.47:0/3572263133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x55614690c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.402 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.380+0000 7f4e1f7fe700 1 -- 172.21.15.47:0/3572263133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468ff1f0 msgr2=0x55614690a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.403 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.380+0000 7f4e1f7fe700 1 --2- 172.21.15.47:0/3572263133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468ff1f0 0x55614690a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.403 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.380+0000 7f4e1f7fe700 1 -- 172.21.15.47:0/3572263133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 msgr2=0x55614690c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.403 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.380+0000 7f4e1f7fe700 1 --2- 172.21.15.47:0/3572263133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x55614690c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.403 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.380+0000 7f4e1f7fe700 1 -- 172.21.15.47:0/3572263133 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561466d45e0 con 0x5561468fe430 2022-09-07T21:40:45.403 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.380+0000 7f4e1effd700 1 --2- 172.21.15.47:0/3572263133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468ff1f0 0x55614690a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:40:45.404 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.380+0000 7f4e1f7fe700 1 --2- 172.21.15.47:0/3572263133 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fe430 0x55614690c650 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000f460 tx=0x7f4e10009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495566c666489ab3 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.404 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.380+0000 7f4dfaffd700 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2022-09-07T21:40:45.404 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.380+0000 7f4dfaffd700 1 -- 172.21.15.47:0/3572263133 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1001d070 con 0x5561468fe430 2022-09-07T21:40:45.404 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.381+0000 7f4e3944bb80 1 -- 172.21.15.47:0/3572263133 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e1001e040 con 0x5561468fe430 2022-09-07T21:40:45.405 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.381+0000 7f4dfaffd700 1 -- 172.21.15.47:0/3572263133 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f4e1000fad0 con 0x5561468fe430 2022-09-07T21:40:45.405 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.381+0000 7f4dfaffd700 1 -- 172.21.15.47:0/3572263133 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e1000fc90 con 0x5561468fe430 2022-09-07T21:40:45.405 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.381+0000 7f4e3944bb80 10 client.15207 fetch_fsmap learned FSMap version 24 2022-09-07T21:40:45.405 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.381+0000 7f4e3944bb80 1 -- 172.21.15.47:0/3572263133 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55614667f260 con 0x5561468fe430 2022-09-07T21:40:45.405 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.381+0000 7f4dfaffd700 1 -- 172.21.15.47:0/3572263133 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4e1001e040 con 0x5561468fe430 2022-09-07T21:40:45.406 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.381+0000 7f4dfaffd700 1 -- 172.21.15.47:0/3572263133 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 24) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f4e1002a050 con 0x5561468fe430 2022-09-07T21:40:45.406 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.381+0000 7f4e3944bb80 10 client.15207 fetch_fsmap finished waiting for FSMap version 24 2022-09-07T21:40:45.406 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.381+0000 7f4e3944bb80 10 client.15207 resolve_mds: resolved name 'b' to daemon mds.b 2022-09-07T21:40:45.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.381+0000 7f4e3944bb80 20 client.15207 populate_metadata read hostname 'smithi047' 2022-09-07T21:40:45.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.381+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/3572263133 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x556146968c00 0x5561469655d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.381+0000 7f4e3944bb80 4 client.15207 mds_command: new command op to 24572 tid=0[{"prefix": "status"}] 2022-09-07T21:40:45.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.381+0000 7f4e3944bb80 1 -- 172.21.15.47:0/3572263133 --> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5561468456f0 con 0x556146968c00 2022-09-07T21:40:45.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.382+0000 7f4e1effd700 1 --2- 172.21.15.47:0/3572263133 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x556146968c00 0x5561469655d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.382+0000 7f4e1effd700 1 --2- 172.21.15.47:0/3572263133 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x556146968c00 0x5561469655d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b0bebcf3b1d5fb5f server_cookie=7ec209edef2d7310 in_seq=0 out_seq=0 2022-09-07T21:40:45.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.382+0000 7f4dfaffd700 10 client.15207 ms_handle_connect on v2:172.21.15.55:6838/517990300 2022-09-07T21:40:45.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4dfaffd700 1 -- 172.21.15.47:0/3572263133 <== mds.1 v2:172.21.15.55:6838/517990300 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5561468456f0 con 0x556146968c00 2022-09-07T21:40:45.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4dfaffd700 10 client.15207 handle_command_reply: tid=0 2022-09-07T21:40:45.409 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4e3944bb80 1 client.15207 shutdown 2022-09-07T21:40:45.409 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4e3944bb80 1 -- 172.21.15.47:0/3572263133 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fe430 msgr2=0x55614690c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.409 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/3572263133 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fe430 0x55614690c650 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000f460 tx=0x7f4e10009ee0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.409 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4e3944bb80 1 -- 172.21.15.47:0/3572263133 shutdown_connections 2022-09-07T21:40:45.410 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/3572263133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561468ff1f0 0x55614690a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.410 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/3572263133 >> [v2:172.21.15.55:6838/517990300,v1:172.21.15.55:6839/517990300] conn(0x556146968c00 0x5561469655d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.410 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/3572263133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5561468fdae0 0x55614690c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.410 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4dfaffd700 0 client.15207 ms_handle_reset on v2:172.21.15.55:6838/517990300 2022-09-07T21:40:45.411 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4e3944bb80 1 --2- 172.21.15.47:0/3572263133 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5561468fe430 0x55614690c650 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.411 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4e3944bb80 1 -- 172.21.15.47:0/3572263133 >> 172.21.15.47:0/3572263133 conn(0x5561468f9fa0 msgr2=0x55614691ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.411 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4e3944bb80 1 -- 172.21.15.47:0/3572263133 shutdown_connections 2022-09-07T21:40:45.411 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4e3944bb80 1 -- 172.21.15.47:0/3572263133 wait complete. 2022-09-07T21:40:45.412 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.383+0000 7f4e3944bb80 20 client.15207 trim_cache size 0 max 16384 2022-09-07T21:40:45.412 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2022-09-07T21:40:45.412 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "01b8396e-2ef4-11ed-8431-001a4aab830c", 2022-09-07T21:40:45.413 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 1, 2022-09-07T21:40:45.413 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 24572, 2022-09-07T21:40:45.413 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2022-09-07T21:40:45.413 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2022-09-07T21:40:45.414 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2022-09-07T21:40:45.414 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 100.58632277300001, 2022-09-07T21:40:45.414 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 24, 2022-09-07T21:40:45.414 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 91, 2022-09-07T21:40:45.414 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 89, 2022-09-07T21:40:45.415 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 130.266390172 2022-09-07T21:40:45.415 INFO:teuthology.orchestra.run.smithi047.stdout:} 2022-09-07T21:40:45.416 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.384+0000 7f4e25fff700 1 -- 172.21.15.47:0/59673809 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4e08099010 msgr2=0x7f4e0809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.384+0000 7f4e25fff700 1 --2- 172.21.15.47:0/59673809 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4e08099010 0x7f4e0809b4c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4e1808f1c0 tx=0x7f4e18009f60 comp rx=0 tx=0).stop 2022-09-07T21:40:45.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.384+0000 7f4e25fff700 1 -- 172.21.15.47:0/59673809 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e20103990 msgr2=0x7f4e2019d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.384+0000 7f4e25fff700 1 --2- 172.21.15.47:0/59673809 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e20103990 0x7f4e2019d140 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800ebf0 tx=0x7f4e1800ef00 comp rx=0 tx=0).stop 2022-09-07T21:40:45.417 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.384+0000 7f4e25fff700 1 -- 172.21.15.47:0/59673809 shutdown_connections 2022-09-07T21:40:45.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.384+0000 7f4e25fff700 1 --2- 172.21.15.47:0/59673809 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f4e08099010 0x7f4e0809b4c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.384+0000 7f4e25fff700 1 --2- 172.21.15.47:0/59673809 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e20103990 0x7f4e2019d140 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.418 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.384+0000 7f4e25fff700 1 --2- 172.21.15.47:0/59673809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e20066be0 0x7f4e20195b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.419 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.384+0000 7f4e25fff700 1 --2- 172.21.15.47:0/59673809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e20100ea0 0x7f4e201960d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.419 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.384+0000 7f4e25fff700 1 -- 172.21.15.47:0/59673809 >> 172.21.15.47:0/59673809 conn(0x7f4e200f8370 msgr2=0x7f4e200fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.419 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.384+0000 7f4e25fff700 1 -- 172.21.15.47:0/59673809 shutdown_connections 2022-09-07T21:40:45.419 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.384+0000 7f4e25fff700 1 -- 172.21.15.47:0/59673809 wait complete. 2022-09-07T21:40:45.420 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2022-09-07T21:40:45.653 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.649+0000 7f959159d700 1 -- 172.21.15.47:0/513842302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f958c100f50 msgr2=0x7f958c103330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.654 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.649+0000 7f959159d700 1 --2- 172.21.15.47:0/513842302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f958c100f50 0x7f958c103330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9580009810 tx=0x7f9580009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:45.654 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.649+0000 7f959159d700 1 -- 172.21.15.47:0/513842302 shutdown_connections 2022-09-07T21:40:45.654 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.649+0000 7f959159d700 1 --2- 172.21.15.47:0/513842302 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f958c103870 0x7f958c105c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.655 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.649+0000 7f959159d700 1 --2- 172.21.15.47:0/513842302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f958c0fe630 0x7f958c100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.655 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.649+0000 7f959159d700 1 --2- 172.21.15.47:0/513842302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f958c100f50 0x7f958c103330 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.655 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.650+0000 7f959159d700 1 -- 172.21.15.47:0/513842302 >> 172.21.15.47:0/513842302 conn(0x7f958c0f80d0 msgr2=0x7f958c0fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.655 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.650+0000 7f959159d700 1 -- 172.21.15.47:0/513842302 shutdown_connections 2022-09-07T21:40:45.656 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.650+0000 7f959159d700 1 -- 172.21.15.47:0/513842302 wait complete. 2022-09-07T21:40:45.658 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.651+0000 7f959159d700 1 Processor -- start 2022-09-07T21:40:45.658 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.651+0000 7f959159d700 1 -- start start 2022-09-07T21:40:45.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.651+0000 7f959159d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f958c0fe630 0x7f958c195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.651+0000 7f959159d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f958c100f50 0x7f958c195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.651+0000 7f959159d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f958c103870 0x7f958c19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.651+0000 7f959159d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c19d480 con 0x7f958c100f50 2022-09-07T21:40:45.660 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.651+0000 7f959159d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c19d5f0 con 0x7f958c103870 2022-09-07T21:40:45.660 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.651+0000 7f959159d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c19d7b0 con 0x7f958c0fe630 2022-09-07T21:40:45.660 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.652+0000 7f958b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f958c100f50 0x7f958c195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.660 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.652+0000 7f958b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f958c100f50 0x7f958c195e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34244/0 (socket says 172.21.15.47:34244) 2022-09-07T21:40:45.661 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.652+0000 7f958b7fe700 1 -- 172.21.15.47:0/1249133384 learned_addr learned my addr 172.21.15.47:0/1249133384 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.661 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.652+0000 7f958bfff700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f958c0fe630 0x7f958c195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.661 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.652+0000 7f9590d9c700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f958c103870 0x7f958c19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.661 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.652+0000 7f958bfff700 1 -- 172.21.15.47:0/1249133384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f958c103870 msgr2=0x7f958c19ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.662 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.652+0000 7f958bfff700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f958c103870 0x7f958c19ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.662 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.652+0000 7f958bfff700 1 -- 172.21.15.47:0/1249133384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f958c100f50 msgr2=0x7f958c195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.662 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.652+0000 7f958bfff700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f958c100f50 0x7f958c195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.662 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.652+0000 7f958bfff700 1 -- 172.21.15.47:0/1249133384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9580009470 con 0x7f958c0fe630 2022-09-07T21:40:45.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.652+0000 7f958b7fe700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f958c100f50 0x7f958c195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:45.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.653+0000 7f958bfff700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f958c0fe630 0x7f958c195900 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f957c00e6c0 tx=0x7f957c00e9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.653+0000 7f95897fa700 1 -- 172.21.15.47:0/1249133384 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c010900 con 0x7f958c0fe630 2022-09-07T21:40:45.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.653+0000 7f959159d700 1 -- 172.21.15.47:0/1249133384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958c19da90 con 0x7f958c0fe630 2022-09-07T21:40:45.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.653+0000 7f95897fa700 1 -- 172.21.15.47:0/1249133384 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f957c018420 con 0x7f958c0fe630 2022-09-07T21:40:45.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.653+0000 7f95897fa700 1 -- 172.21.15.47:0/1249133384 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c017610 con 0x7f958c0fe630 2022-09-07T21:40:45.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.653+0000 7f959159d700 1 -- 172.21.15.47:0/1249133384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958c19dff0 con 0x7f958c0fe630 2022-09-07T21:40:45.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.656+0000 7f95897fa700 1 -- 172.21.15.47:0/1249133384 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f957c00c040 con 0x7f958c0fe630 2022-09-07T21:40:45.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.658+0000 7f95a5db4b80 1 Processor -- start 2022-09-07T21:40:45.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.658+0000 7f95a5db4b80 1 -- start start 2022-09-07T21:40:45.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.658+0000 7f95897fa700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9574099010 0x7f957409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.658+0000 7f95897fa700 1 -- 172.21.15.47:0/1249133384 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f957c014070 con 0x7f958c0fe630 2022-09-07T21:40:45.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.658+0000 7f95a5db4b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233000 0x556d39233410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.673 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.658+0000 7f95a5db4b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d39233950 0x556d39231be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.673 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.658+0000 7f95a5db4b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d392321c0 0x556d39232630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.673 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.658+0000 7f95a5db4b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d3911c080 con 0x556d39233950 2022-09-07T21:40:45.673 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.658+0000 7f95a5db4b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d391869e0 con 0x556d392321c0 2022-09-07T21:40:45.673 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.658+0000 7f95a5db4b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d39180180 con 0x556d39233000 2022-09-07T21:40:45.674 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.658+0000 7f9590d9c700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9574099010 0x7f957409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.674 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.659+0000 7f958bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233000 0x556d39233410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.674 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.659+0000 7f958b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d39233950 0x556d39231be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.674 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.659+0000 7f958bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233000 0x556d39233410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:56642/0 (socket says 172.21.15.47:56642) 2022-09-07T21:40:45.674 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.659+0000 7f958bfff700 1 -- 172.21.15.47:0/1560527188 learned_addr learned my addr 172.21.15.47:0/1560527188 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.675 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.659+0000 7f9590d9c700 1 --2- 172.21.15.47:0/1560527188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d392321c0 0x556d39232630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.675 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.659+0000 7f958bfff700 1 -- 172.21.15.47:0/1560527188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d392321c0 msgr2=0x556d39232630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.675 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.659+0000 7f958bfff700 1 --2- 172.21.15.47:0/1560527188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d392321c0 0x556d39232630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.675 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.659+0000 7f958bfff700 1 -- 172.21.15.47:0/1560527188 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d39233950 msgr2=0x556d39231be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.676 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.659+0000 7f958bfff700 1 --2- 172.21.15.47:0/1560527188 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d39233950 0x556d39231be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.676 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.659+0000 7f958bfff700 1 -- 172.21.15.47:0/1560527188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d390085e0 con 0x556d39233000 2022-09-07T21:40:45.676 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.659+0000 7f9590d9c700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9574099010 0x7f957409b4c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9584013690 tx=0x7f9584013a50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.676 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.659+0000 7f958bfff700 1 --2- 172.21.15.47:0/1560527188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233000 0x556d39233410 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f957c092640 tx=0x7f957c092670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76ba87b4455e59b5 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.676 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.660+0000 7f956e7fc700 1 -- 172.21.15.47:0/1560527188 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c094070 con 0x556d39233000 2022-09-07T21:40:45.677 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.660+0000 7f956e7fc700 1 -- 172.21.15.47:0/1560527188 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f957c092ea0 con 0x556d39233000 2022-09-07T21:40:45.677 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.660+0000 7f956e7fc700 1 -- 172.21.15.47:0/1560527188 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c092a00 con 0x556d39233000 2022-09-07T21:40:45.677 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.660+0000 7f95a5db4b80 1 -- 172.21.15.47:0/1560527188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233000 msgr2=0x556d39233410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.677 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.660+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/1560527188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233000 0x556d39233410 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f957c092640 tx=0x7f957c092670 comp rx=0 tx=0).stop 2022-09-07T21:40:45.677 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.660+0000 7f95a5db4b80 1 -- 172.21.15.47:0/1560527188 shutdown_connections 2022-09-07T21:40:45.678 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.660+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/1560527188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d392321c0 0x556d39232630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.678 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.660+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/1560527188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233000 0x556d39233410 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.678 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.660+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/1560527188 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d39233950 0x556d39231be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.678 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.660+0000 7f95a5db4b80 1 -- 172.21.15.47:0/1560527188 >> 172.21.15.47:0/1560527188 conn(0x556d3922e170 msgr2=0x556d3922c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.679 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.660+0000 7f95a5db4b80 1 -- 172.21.15.47:0/1560527188 shutdown_connections 2022-09-07T21:40:45.679 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.660+0000 7f95a5db4b80 1 -- 172.21.15.47:0/1560527188 wait complete. 2022-09-07T21:40:45.679 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.661+0000 7f95a5db4b80 1 -- start start 2022-09-07T21:40:45.680 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.661+0000 7f95a5db4b80 1 Processor -- start 2022-09-07T21:40:45.680 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.662+0000 7f95a5db4b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d392321c0 0x556d3923b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.681 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.662+0000 7f95a5db4b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233000 0x556d3923bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.681 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.662+0000 7f95a5db4b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d39233950 0x556d39242c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.681 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.662+0000 7f95a5db4b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d3906e0c0 con 0x556d392321c0 2022-09-07T21:40:45.682 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.662+0000 7f95a5db4b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d39243140 con 0x556d39233950 2022-09-07T21:40:45.682 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.662+0000 7f95a5db4b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d392432d0 con 0x556d39233000 2022-09-07T21:40:45.683 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.662+0000 7f95a5db4b80 10 client.0 _ll_register_callbacks cb 0x556d3912b370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-07T21:40:45.683 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.662+0000 7f958bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d392321c0 0x556d3923b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.662+0000 7f958bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d392321c0 0x556d3923b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34256/0 (socket says 172.21.15.47:34256) 2022-09-07T21:40:45.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.662+0000 7f958bfff700 1 -- 172.21.15.47:0/3446038291 learned_addr learned my addr 172.21.15.47:0/3446038291 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.662+0000 7f958b7fe700 1 --2- 172.21.15.47:0/3446038291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233000 0x556d3923bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.662+0000 7f9590d9c700 1 --2- 172.21.15.47:0/3446038291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d39233950 0x556d39242c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.663+0000 7f958bfff700 1 -- 172.21.15.47:0/3446038291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233000 msgr2=0x556d3923bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.663+0000 7f958bfff700 1 --2- 172.21.15.47:0/3446038291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233000 0x556d3923bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.663+0000 7f958bfff700 1 -- 172.21.15.47:0/3446038291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d39233950 msgr2=0x556d39242c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.663+0000 7f958bfff700 1 --2- 172.21.15.47:0/3446038291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d39233950 0x556d39242c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.663+0000 7f958bfff700 1 -- 172.21.15.47:0/3446038291 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958c19dff0 con 0x556d392321c0 2022-09-07T21:40:45.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.663+0000 7f958bfff700 1 --2- 172.21.15.47:0/3446038291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d392321c0 0x556d3923b680 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f957c018890 tx=0x7f957c091ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d24dcb8a99c9327 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.663+0000 7f95677fe700 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2022-09-07T21:40:45.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.663+0000 7f95677fe700 1 -- 172.21.15.47:0/3446038291 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c094040 con 0x556d392321c0 2022-09-07T21:40:45.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.663+0000 7f95677fe700 1 -- 172.21.15.47:0/3446038291 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f957c099a40 con 0x556d392321c0 2022-09-07T21:40:45.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.663+0000 7f95a5db4b80 1 -- 172.21.15.47:0/3446038291 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556d39224cf0 con 0x556d392321c0 2022-09-07T21:40:45.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.663+0000 7f95677fe700 1 -- 172.21.15.47:0/3446038291 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f957c091f30 con 0x556d392321c0 2022-09-07T21:40:45.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.664+0000 7f95a5db4b80 10 client.15219 fetch_fsmap learned FSMap version 24 2022-09-07T21:40:45.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.664+0000 7f95a5db4b80 1 -- 172.21.15.47:0/3446038291 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556d38fb4d90 con 0x556d392321c0 2022-09-07T21:40:45.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.664+0000 7f95677fe700 1 -- 172.21.15.47:0/3446038291 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f957c076020 con 0x556d392321c0 2022-09-07T21:40:45.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.664+0000 7f95677fe700 1 -- 172.21.15.47:0/3446038291 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 24) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f957c091700 con 0x556d392321c0 2022-09-07T21:40:45.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.664+0000 7f95a5db4b80 10 client.15219 fetch_fsmap finished waiting for FSMap version 24 2022-09-07T21:40:45.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.664+0000 7f95a5db4b80 10 client.15219 resolve_mds: resolved name 'a' to daemon mds.a 2022-09-07T21:40:45.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.664+0000 7f95a5db4b80 20 client.15219 populate_metadata read hostname 'smithi047' 2022-09-07T21:40:45.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.664+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/3446038291 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x556d39246610 0x556d3924b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.664+0000 7f95a5db4b80 4 client.15219 mds_command: new command op to 24586 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-07T21:40:45.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.664+0000 7f95a5db4b80 1 -- 172.21.15.47:0/3446038291 --> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d392014c0 con 0x556d39246610 2022-09-07T21:40:45.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.665+0000 7f958b7fe700 1 --2- 172.21.15.47:0/3446038291 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x556d39246610 0x556d3924b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.665+0000 7f958b7fe700 1 --2- 172.21.15.47:0/3446038291 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x556d39246610 0x556d3924b8d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c34fc767481cb19a server_cookie=3c38622a428b2fc in_seq=0 out_seq=0 2022-09-07T21:40:45.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.665+0000 7f95677fe700 10 client.15219 ms_handle_connect on v2:172.21.15.47:6840/793863162 2022-09-07T21:40:45.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.666+0000 7f95677fe700 1 -- 172.21.15.47:0/3446038291 <== mds.0 v2:172.21.15.47:6840/793863162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x556d392014c0 con 0x556d39246610 2022-09-07T21:40:45.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.666+0000 7f95677fe700 10 client.15219 handle_command_reply: tid=0 2022-09-07T21:40:45.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.666+0000 7f95a5db4b80 1 client.15219 shutdown 2022-09-07T21:40:45.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.666+0000 7f95a5db4b80 1 -- 172.21.15.47:0/3446038291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d392321c0 msgr2=0x556d3923b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.666+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/3446038291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d392321c0 0x556d3923b680 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f957c018890 tx=0x7f957c091ec0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.666+0000 7f95a5db4b80 1 -- 172.21.15.47:0/3446038291 shutdown_connections 2022-09-07T21:40:45.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.666+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/3446038291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d39233950 0x556d39242c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.666+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/3446038291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d392321c0 0x556d3923b680 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.666+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/3446038291 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x556d39246610 0x556d3924b8d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.666+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/3446038291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233000 0x556d3923bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.666+0000 7f95a5db4b80 1 -- 172.21.15.47:0/3446038291 >> 172.21.15.47:0/3446038291 conn(0x556d3922e170 msgr2=0x556d3922bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.666+0000 7f95677fe700 0 client.15219 ms_handle_reset on v2:172.21.15.47:6840/793863162 2022-09-07T21:40:45.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.667+0000 7f95a5db4b80 1 -- 172.21.15.47:0/3446038291 shutdown_connections 2022-09-07T21:40:45.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.667+0000 7f95a5db4b80 1 -- 172.21.15.47:0/3446038291 wait complete. 2022-09-07T21:40:45.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.667+0000 7f95a5db4b80 20 client.15219 trim_cache size 0 max 16384 2022-09-07T21:40:45.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.685+0000 7f95a5db4b80 1 Processor -- start 2022-09-07T21:40:45.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.685+0000 7f95a5db4b80 1 -- start start 2022-09-07T21:40:45.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.685+0000 7f95a5db4b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d392321c0 0x556d3924f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.685+0000 7f95a5db4b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d39233000 0x556d39236040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.685+0000 7f95a5db4b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233950 0x556d3923b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.685+0000 7f95a5db4b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d3922ae50 con 0x556d392321c0 2022-09-07T21:40:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.685+0000 7f95a5db4b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d3923c580 con 0x556d39233000 2022-09-07T21:40:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.685+0000 7f95a5db4b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d39242fe0 con 0x556d39233950 2022-09-07T21:40:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.685+0000 7f958bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d392321c0 0x556d3924f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.685+0000 7f9590d9c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233950 0x556d3923b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.685+0000 7f9590d9c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233950 0x556d3923b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:56660/0 (socket says 172.21.15.47:56660) 2022-09-07T21:40:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.685+0000 7f9590d9c700 1 -- 172.21.15.47:0/4262648770 learned_addr learned my addr 172.21.15.47:0/4262648770 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.686+0000 7f958b7fe700 1 --2- 172.21.15.47:0/4262648770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d39233000 0x556d39236040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.686+0000 7f958bfff700 1 -- 172.21.15.47:0/4262648770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233950 msgr2=0x556d3923b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.686+0000 7f958bfff700 1 --2- 172.21.15.47:0/4262648770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233950 0x556d3923b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.686+0000 7f958bfff700 1 -- 172.21.15.47:0/4262648770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d39233000 msgr2=0x556d39236040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.686+0000 7f958bfff700 1 --2- 172.21.15.47:0/4262648770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d39233000 0x556d39236040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.686+0000 7f958bfff700 1 -- 172.21.15.47:0/4262648770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d38fb3260 con 0x556d392321c0 2022-09-07T21:40:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.686+0000 7f9590d9c700 1 --2- 172.21.15.47:0/4262648770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233950 0x556d3923b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.686+0000 7f958bfff700 1 --2- 172.21.15.47:0/4262648770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d392321c0 0x556d3924f8f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f957c08eca0 tx=0x7f957c092410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4142a227f641b4f server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.686+0000 7f956effd700 1 -- 172.21.15.47:0/4262648770 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c094070 con 0x556d392321c0 2022-09-07T21:40:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.686+0000 7f956effd700 1 -- 172.21.15.47:0/4262648770 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f957c076070 con 0x556d392321c0 2022-09-07T21:40:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.686+0000 7f956effd700 1 -- 172.21.15.47:0/4262648770 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c05d030 con 0x556d392321c0 2022-09-07T21:40:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.687+0000 7f95a5db4b80 1 -- 172.21.15.47:0/4262648770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d392321c0 msgr2=0x556d3924f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.687+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/4262648770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d392321c0 0x556d3924f8f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f957c08eca0 tx=0x7f957c092410 comp rx=0 tx=0).stop 2022-09-07T21:40:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.687+0000 7f95a5db4b80 1 -- 172.21.15.47:0/4262648770 shutdown_connections 2022-09-07T21:40:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.687+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/4262648770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233950 0x556d3923b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.687+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/4262648770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d392321c0 0x556d3924f8f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.687+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/4262648770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d39233000 0x556d39236040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.687+0000 7f95a5db4b80 1 -- 172.21.15.47:0/4262648770 >> 172.21.15.47:0/4262648770 conn(0x556d3922e170 msgr2=0x556d3922bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.687+0000 7f95a5db4b80 1 -- 172.21.15.47:0/4262648770 shutdown_connections 2022-09-07T21:40:45.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.687+0000 7f95a5db4b80 1 -- 172.21.15.47:0/4262648770 wait complete. 2022-09-07T21:40:45.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.688+0000 7f95a5db4b80 1 -- start start 2022-09-07T21:40:45.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.688+0000 7f95a5db4b80 1 Processor -- start 2022-09-07T21:40:45.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.688+0000 7f95a5db4b80 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d392321c0 0x556d3923f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.688+0000 7f95a5db4b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d39233000 0x556d3923fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.688+0000 7f95a5db4b80 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233950 0x556d39298870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.688+0000 7f95a5db4b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d39240200 con 0x556d39233000 2022-09-07T21:40:45.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.688+0000 7f95a5db4b80 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d39240370 con 0x556d392321c0 2022-09-07T21:40:45.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.688+0000 7f95a5db4b80 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d39298db0 con 0x556d39233950 2022-09-07T21:40:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.688+0000 7f95a5db4b80 10 client.0 _ll_register_callbacks cb 0x556d3912b370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-07T21:40:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.689+0000 7f958b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d39233000 0x556d3923fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.689+0000 7f958b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d39233000 0x556d3923fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34278/0 (socket says 172.21.15.47:34278) 2022-09-07T21:40:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.689+0000 7f958b7fe700 1 -- 172.21.15.47:0/4184272129 learned_addr learned my addr 172.21.15.47:0/4184272129 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:45.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.689+0000 7f958bfff700 1 --2- 172.21.15.47:0/4184272129 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d392321c0 0x556d3923f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.689+0000 7f9590d9c700 1 --2- 172.21.15.47:0/4184272129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233950 0x556d39298870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.689+0000 7f958b7fe700 1 -- 172.21.15.47:0/4184272129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233950 msgr2=0x556d39298870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.689+0000 7f958b7fe700 1 --2- 172.21.15.47:0/4184272129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233950 0x556d39298870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.689+0000 7f958b7fe700 1 -- 172.21.15.47:0/4184272129 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d392321c0 msgr2=0x556d3923f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-07T21:40:45.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.689+0000 7f958b7fe700 1 --2- 172.21.15.47:0/4184272129 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d392321c0 0x556d3923f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.689+0000 7f958b7fe700 1 -- 172.21.15.47:0/4184272129 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9584012150 con 0x556d39233000 2022-09-07T21:40:45.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.689+0000 7f958b7fe700 1 --2- 172.21.15.47:0/4184272129 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d39233000 0x556d3923fa70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9580000c00 tx=0x7f958000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff58cfd8424420d server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:45.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.689+0000 7f95677fe700 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2022-09-07T21:40:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.690+0000 7f95677fe700 1 -- 172.21.15.47:0/4184272129 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958001d040 con 0x556d39233000 2022-09-07T21:40:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.690+0000 7f95a5db4b80 1 -- 172.21.15.47:0/4184272129 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f957c092ea0 con 0x556d39233000 2022-09-07T21:40:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.690+0000 7f95677fe700 1 -- 172.21.15.47:0/4184272129 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f9580005240 con 0x556d39233000 2022-09-07T21:40:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.690+0000 7f95677fe700 1 -- 172.21.15.47:0/4184272129 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9580005400 con 0x556d39233000 2022-09-07T21:40:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.690+0000 7f95a5db4b80 10 client.15225 fetch_fsmap learned FSMap version 24 2022-09-07T21:40:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.690+0000 7f95a5db4b80 1 -- 172.21.15.47:0/4184272129 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556d38fb4d90 con 0x556d39233000 2022-09-07T21:40:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.690+0000 7f95677fe700 1 -- 172.21.15.47:0/4184272129 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f957c092ea0 con 0x556d39233000 2022-09-07T21:40:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.691+0000 7f95677fe700 1 -- 172.21.15.47:0/4184272129 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 24) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f9580009840 con 0x556d39233000 2022-09-07T21:40:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.691+0000 7f95a5db4b80 10 client.15225 fetch_fsmap finished waiting for FSMap version 24 2022-09-07T21:40:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.691+0000 7f95a5db4b80 10 client.15225 resolve_mds: resolved name 'a' to daemon mds.a 2022-09-07T21:40:45.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.691+0000 7f95a5db4b80 20 client.15225 populate_metadata read hostname 'smithi047' 2022-09-07T21:40:45.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.691+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/4184272129 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x556d3929db20 0x556d3929a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:45.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.691+0000 7f95a5db4b80 4 client.15225 mds_command: new command op to 24586 tid=0[{"prefix": "status"}] 2022-09-07T21:40:45.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.691+0000 7f95a5db4b80 1 -- 172.21.15.47:0/4184272129 --> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556d391796f0 con 0x556d3929db20 2022-09-07T21:40:45.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.691+0000 7f958bfff700 1 --2- 172.21.15.47:0/4184272129 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x556d3929db20 0x556d3929a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f958bfff700 1 --2- 172.21.15.47:0/4184272129 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x556d3929db20 0x556d3929a460 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad595a74d520c3d3 server_cookie=d4b942dd4ac81738 in_seq=0 out_seq=0 2022-09-07T21:40:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95677fe700 10 client.15225 ms_handle_connect on v2:172.21.15.47:6840/793863162 2022-09-07T21:40:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95677fe700 1 -- 172.21.15.47:0/4184272129 <== mds.0 v2:172.21.15.47:6840/793863162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x556d391796f0 con 0x556d3929db20 2022-09-07T21:40:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95677fe700 10 client.15225 handle_command_reply: tid=0 2022-09-07T21:40:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95a5db4b80 1 client.15225 shutdown 2022-09-07T21:40:45.717 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95a5db4b80 1 -- 172.21.15.47:0/4184272129 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d39233000 msgr2=0x556d3923fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.717 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/4184272129 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d39233000 0x556d3923fa70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9580000c00 tx=0x7f958000be50 comp rx=0 tx=0).stop 2022-09-07T21:40:45.717 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95a5db4b80 1 -- 172.21.15.47:0/4184272129 shutdown_connections 2022-09-07T21:40:45.717 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/4184272129 >> [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162] conn(0x556d3929db20 0x556d3929a460 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/4184272129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556d39233950 0x556d39298870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/4184272129 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556d392321c0 0x556d3923f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95a5db4b80 1 --2- 172.21.15.47:0/4184272129 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556d39233000 0x556d3923fa70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95a5db4b80 1 -- 172.21.15.47:0/4184272129 >> 172.21.15.47:0/4184272129 conn(0x556d3922e170 msgr2=0x556d3924f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95677fe700 0 client.15225 ms_handle_reset on v2:172.21.15.47:6840/793863162 2022-09-07T21:40:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95a5db4b80 1 -- 172.21.15.47:0/4184272129 shutdown_connections 2022-09-07T21:40:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.692+0000 7f95a5db4b80 1 -- 172.21.15.47:0/4184272129 wait complete. 2022-09-07T21:40:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.693+0000 7f95a5db4b80 20 client.15225 trim_cache size 0 max 16384 2022-09-07T21:40:45.720 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2022-09-07T21:40:45.720 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "01b8396e-2ef4-11ed-8431-001a4aab830c", 2022-09-07T21:40:45.720 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 0, 2022-09-07T21:40:45.721 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 24586, 2022-09-07T21:40:45.721 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2022-09-07T21:40:45.721 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2022-09-07T21:40:45.721 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2022-09-07T21:40:45.722 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 125.14537692899999, 2022-09-07T21:40:45.722 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 24, 2022-09-07T21:40:45.722 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 91, 2022-09-07T21:40:45.722 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 89, 2022-09-07T21:40:45.723 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 133.184926609 2022-09-07T21:40:45.723 INFO:teuthology.orchestra.run.smithi047.stdout:} 2022-09-07T21:40:45.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.693+0000 7f959159d700 1 -- 172.21.15.47:0/1249133384 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9574099010 msgr2=0x7f957409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.693+0000 7f959159d700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9574099010 0x7f957409b4c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9584013690 tx=0x7f9584013a50 comp rx=0 tx=0).stop 2022-09-07T21:40:45.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.693+0000 7f959159d700 1 -- 172.21.15.47:0/1249133384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f958c0fe630 msgr2=0x7f958c195900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:45.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.693+0000 7f959159d700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f958c0fe630 0x7f958c195900 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f957c00e6c0 tx=0x7f957c00e9d0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.693+0000 7f959159d700 1 -- 172.21.15.47:0/1249133384 shutdown_connections 2022-09-07T21:40:45.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.693+0000 7f959159d700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f9574099010 0x7f957409b4c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.693+0000 7f959159d700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f958c103870 0x7f958c19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.693+0000 7f959159d700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f958c0fe630 0x7f958c195900 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.693+0000 7f959159d700 1 --2- 172.21.15.47:0/1249133384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f958c100f50 0x7f958c195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:45.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.693+0000 7f959159d700 1 -- 172.21.15.47:0/1249133384 >> 172.21.15.47:0/1249133384 conn(0x7f958c0f80d0 msgr2=0x7f958c0ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:45.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.693+0000 7f959159d700 1 -- 172.21.15.47:0/1249133384 shutdown_connections 2022-09-07T21:40:45.727 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:45.693+0000 7f959159d700 1 -- 172.21.15.47:0/1249133384 wait complete. 2022-09-07T21:40:45.727 INFO:teuthology.run_tasks:Running task kclient... 2022-09-07T21:40:45.740 INFO:tasks.kclient:Mounting kernel clients... 2022-09-07T21:40:45.741 INFO:tasks.kclient:config is {'client.0': {}} 2022-09-07T21:40:45.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:45.742 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2022-09-07T21:40:45.800 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2022-09-07T21:40:45.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:45.802 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2022-09-07T21:40:45.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:45.895 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link delete ceph-brx 2022-09-07T21:40:46.033 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2022-09-07T21:40:46.033 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2022-09-07T21:40:46.034 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2022-09-07T21:40:46.034 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph fs ls 2022-09-07T21:40:46.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:40:46.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:46 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:40:46.928 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T21:40:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:40:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:46 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:40:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:40:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:46 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:40:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:47 smithi055 ceph-mon[97033]: pgmap v381: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2022-09-07T21:40:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2022-09-07T21:40:47.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:47 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2022-09-07T21:40:48.029 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:47 smithi047 ceph-mon[90087]: pgmap v381: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:48.029 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2022-09-07T21:40:48.030 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2022-09-07T21:40:48.030 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:47 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2022-09-07T21:40:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:47 smithi059 ceph-mon[97450]: pgmap v381: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:40:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2022-09-07T21:40:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2022-09-07T21:40:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:47 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2022-09-07T21:40:48.592 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.588+0000 7fc46f427700 1 -- 172.21.15.47:0/152296582 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc468108480 msgr2=0x7fc46810a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:48.593 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.588+0000 7fc46f427700 1 --2- 172.21.15.47:0/152296582 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc468108480 0x7fc46810a8a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc460009bb0 tx=0x7fc460009ec0 comp rx=0 tx=0).stop 2022-09-07T21:40:48.593 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.589+0000 7fc46f427700 1 -- 172.21.15.47:0/152296582 shutdown_connections 2022-09-07T21:40:48.593 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.589+0000 7fc46f427700 1 --2- 172.21.15.47:0/152296582 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc468108480 0x7fc46810a8a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:48.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.589+0000 7fc46f427700 1 --2- 172.21.15.47:0/152296582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4680ff0b0 0x7fc4680ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:48.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.589+0000 7fc46f427700 1 --2- 172.21.15.47:0/152296582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4680ff9c0 0x7fc468107f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:48.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.589+0000 7fc46f427700 1 -- 172.21.15.47:0/152296582 >> 172.21.15.47:0/152296582 conn(0x7fc4680f89d0 msgr2=0x7fc4680fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:48.594 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.589+0000 7fc46f427700 1 -- 172.21.15.47:0/152296582 shutdown_connections 2022-09-07T21:40:48.595 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.589+0000 7fc46f427700 1 -- 172.21.15.47:0/152296582 wait complete. 2022-09-07T21:40:48.597 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.590+0000 7fc46f427700 1 Processor -- start 2022-09-07T21:40:48.597 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.590+0000 7fc46f427700 1 -- start start 2022-09-07T21:40:48.597 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.591+0000 7fc46f427700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4680ff0b0 0x7fc468199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:48.597 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.591+0000 7fc46f427700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4680ff9c0 0x7fc468199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:48.598 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.591+0000 7fc46f427700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc468108480 0x7fc468193ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:48.598 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.591+0000 7fc46f427700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc468194250 con 0x7fc4680ff0b0 2022-09-07T21:40:48.598 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.591+0000 7fc46f427700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4681943c0 con 0x7fc4680ff9c0 2022-09-07T21:40:48.598 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.591+0000 7fc46f427700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc468194580 con 0x7fc468108480 2022-09-07T21:40:48.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.591+0000 7fc46d1c3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4680ff0b0 0x7fc468199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:48.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.591+0000 7fc46d1c3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4680ff0b0 0x7fc468199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34304/0 (socket says 172.21.15.47:34304) 2022-09-07T21:40:48.599 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.591+0000 7fc46d1c3700 1 -- 172.21.15.47:0/334378193 learned_addr learned my addr 172.21.15.47:0/334378193 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:48.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.591+0000 7fc46d9c4700 1 --2- 172.21.15.47:0/334378193 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc468108480 0x7fc468193ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:48.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.591+0000 7fc46c9c2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4680ff9c0 0x7fc468199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:48.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.592+0000 7fc46d1c3700 1 -- 172.21.15.47:0/334378193 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc468108480 msgr2=0x7fc468193ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:48.600 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.592+0000 7fc46d1c3700 1 --2- 172.21.15.47:0/334378193 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc468108480 0x7fc468193ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:48.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.592+0000 7fc46d1c3700 1 -- 172.21.15.47:0/334378193 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4680ff9c0 msgr2=0x7fc468199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:48.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.592+0000 7fc46d1c3700 1 --2- 172.21.15.47:0/334378193 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4680ff9c0 0x7fc468199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:48.601 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.592+0000 7fc46d1c3700 1 -- 172.21.15.47:0/334378193 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4600097e0 con 0x7fc4680ff0b0 2022-09-07T21:40:48.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.592+0000 7fc46d1c3700 1 --2- 172.21.15.47:0/334378193 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4680ff0b0 0x7fc468199a40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc46400e7c0 tx=0x7fc46400ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:48.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.592+0000 7fc45e7fc700 1 -- 172.21.15.47:0/334378193 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc464005220 con 0x7fc4680ff0b0 2022-09-07T21:40:48.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.593+0000 7fc45e7fc700 1 -- 172.21.15.47:0/334378193 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fc464005380 con 0x7fc4680ff0b0 2022-09-07T21:40:48.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.593+0000 7fc46f427700 1 -- 172.21.15.47:0/334378193 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc468194860 con 0x7fc4680ff0b0 2022-09-07T21:40:48.602 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.593+0000 7fc45e7fc700 1 -- 172.21.15.47:0/334378193 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc464004bd0 con 0x7fc4680ff0b0 2022-09-07T21:40:48.603 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.593+0000 7fc46f427700 1 -- 172.21.15.47:0/334378193 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4681a0a80 con 0x7fc4680ff0b0 2022-09-07T21:40:48.604 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.594+0000 7fc46f427700 1 -- 172.21.15.47:0/334378193 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4680669f0 con 0x7fc4680ff0b0 2022-09-07T21:40:48.604 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.596+0000 7fc45e7fc700 1 -- 172.21.15.47:0/334378193 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fc46401e020 con 0x7fc4680ff0b0 2022-09-07T21:40:48.604 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.597+0000 7fc45e7fc700 1 --2- 172.21.15.47:0/334378193 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc454099090 0x7fc45409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:48.605 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.597+0000 7fc45e7fc700 1 -- 172.21.15.47:0/334378193 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9103+0+0 (secure 0 0 0) 0x7fc464014070 con 0x7fc4680ff0b0 2022-09-07T21:40:48.605 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.598+0000 7fc46c9c2700 1 --2- 172.21.15.47:0/334378193 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc454099090 0x7fc45409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:48.605 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.598+0000 7fc46c9c2700 1 --2- 172.21.15.47:0/334378193 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc454099090 0x7fc45409b540 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc468109b10 tx=0x7fc458005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:48.606 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.602+0000 7fc45e7fc700 1 -- 172.21.15.47:0/334378193 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fc46408ec50 con 0x7fc4680ff0b0 2022-09-07T21:40:48.882 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.878+0000 7fc46f427700 1 -- 172.21.15.47:0/334378193 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc4681a10f0 con 0x7fc4680ff0b0 2022-09-07T21:40:48.882 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.879+0000 7fc45e7fc700 1 -- 172.21.15.47:0/334378193 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+88 (secure 0 0 0) 0x7fc464092270 con 0x7fc4680ff0b0 2022-09-07T21:40:48.883 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2022-09-07T21:40:48.885 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.881+0000 7fc46f427700 1 -- 172.21.15.47:0/334378193 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc454099090 msgr2=0x7fc45409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:48.885 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.881+0000 7fc46f427700 1 --2- 172.21.15.47:0/334378193 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc454099090 0x7fc45409b540 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc468109b10 tx=0x7fc458005ea0 comp rx=0 tx=0).stop 2022-09-07T21:40:48.885 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.881+0000 7fc46f427700 1 -- 172.21.15.47:0/334378193 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4680ff0b0 msgr2=0x7fc468199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:48.885 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.881+0000 7fc46f427700 1 --2- 172.21.15.47:0/334378193 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4680ff0b0 0x7fc468199a40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc46400e7c0 tx=0x7fc46400ead0 comp rx=0 tx=0).stop 2022-09-07T21:40:48.886 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.881+0000 7fc46f427700 1 -- 172.21.15.47:0/334378193 shutdown_connections 2022-09-07T21:40:48.886 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.881+0000 7fc46f427700 1 --2- 172.21.15.47:0/334378193 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7fc454099090 0x7fc45409b540 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:48.886 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.881+0000 7fc46f427700 1 --2- 172.21.15.47:0/334378193 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc468108480 0x7fc468193ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:48.887 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.881+0000 7fc46f427700 1 --2- 172.21.15.47:0/334378193 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4680ff0b0 0x7fc468199a40 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:48.887 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.881+0000 7fc46f427700 1 --2- 172.21.15.47:0/334378193 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4680ff9c0 0x7fc468199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:48.887 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.881+0000 7fc46f427700 1 -- 172.21.15.47:0/334378193 >> 172.21.15.47:0/334378193 conn(0x7fc4680f89d0 msgr2=0x7fc46810a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:48.887 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.881+0000 7fc46f427700 1 -- 172.21.15.47:0/334378193 shutdown_connections 2022-09-07T21:40:48.887 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:48.881+0000 7fc46f427700 1 -- 172.21.15.47:0/334378193 wait complete. 2022-09-07T21:40:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2022-09-07T21:40:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2022-09-07T21:40:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:48 smithi055 ceph-mon[97033]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2022-09-07T21:40:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:48 smithi055 ceph-mon[97033]: osdmap e92: 12 total, 12 up, 12 in 2022-09-07T21:40:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2022-09-07T21:40:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2022-09-07T21:40:49.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:48 smithi059 ceph-mon[97450]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2022-09-07T21:40:49.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:48 smithi059 ceph-mon[97450]: osdmap e92: 12 total, 12 up, 12 in 2022-09-07T21:40:49.066 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2022-09-07T21:40:49.067 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2022-09-07T21:40:49.067 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:48 smithi047 ceph-mon[90087]: from='mgr.14152 172.21.15.47:0/3796549205' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2022-09-07T21:40:49.068 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:48 smithi047 ceph-mon[90087]: osdmap e92: 12 total, 12 up, 12 in 2022-09-07T21:40:49.402 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2022-09-07T21:40:49.403 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2022-09-07T21:40:49.403 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi047.front.sepia.ceph.com 2022-09-07T21:40:49.404 INFO:tasks.cephfs.mount:self.client.name = client.0 2022-09-07T21:40:49.404 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2022-09-07T21:40:49.404 INFO:tasks.cephfs.mount:self.cephfs_name = None 2022-09-07T21:40:49.405 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2022-09-07T21:40:49.405 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2022-09-07T21:40:49.405 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2022-09-07T21:40:49.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:49.406 DEBUG:teuthology.orchestra.run.smithi047:> ip addr 2022-09-07T21:40:49.469 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-09-07T21:40:49.469 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-09-07T21:40:49.470 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2022-09-07T21:40:49.470 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2022-09-07T21:40:49.470 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2022-09-07T21:40:49.470 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2022-09-07T21:40:49.471 INFO:teuthology.orchestra.run.smithi047.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-09-07T21:40:49.471 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:78 brd ff:ff:ff:ff:ff:ff 2022-09-07T21:40:49.471 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f0 2022-09-07T21:40:49.471 INFO:teuthology.orchestra.run.smithi047.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-09-07T21:40:49.471 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:79 brd ff:ff:ff:ff:ff:ff 2022-09-07T21:40:49.472 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f1 2022-09-07T21:40:49.472 INFO:teuthology.orchestra.run.smithi047.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-09-07T21:40:49.472 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8e brd ff:ff:ff:ff:ff:ff 2022-09-07T21:40:49.472 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f0 2022-09-07T21:40:49.473 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.21.15.47/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2022-09-07T21:40:49.473 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft 41450sec preferred_lft 41450sec 2022-09-07T21:40:49.473 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2022-09-07T21:40:49.473 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2022-09-07T21:40:49.474 INFO:teuthology.orchestra.run.smithi047.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-09-07T21:40:49.474 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8f brd ff:ff:ff:ff:ff:ff 2022-09-07T21:40:49.474 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f1 2022-09-07T21:40:49.476 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2022-09-07T21:40:49.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:49.477 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2022-09-07T21:40:49.477 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2022-09-07T21:40:49.477 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link add name ceph-brx type bridge 2022-09-07T21:40:49.478 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip addr flush dev ceph-brx 2022-09-07T21:40:49.478 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set ceph-brx up 2022-09-07T21:40:49.478 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2022-09-07T21:40:49.479 DEBUG:teuthology.orchestra.run.smithi047:> ') 2022-09-07T21:40:49.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:49.743 DEBUG:teuthology.orchestra.run.smithi047:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2022-09-07T21:40:49.833 INFO:teuthology.orchestra.run.smithi047.stdout:1 2022-09-07T21:40:49.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:49.838 DEBUG:teuthology.orchestra.run.smithi047:> route 2022-09-07T21:40:49.900 INFO:teuthology.orchestra.run.smithi047.stdout:Kernel IP routing table 2022-09-07T21:40:49.901 INFO:teuthology.orchestra.run.smithi047.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2022-09-07T21:40:49.901 INFO:teuthology.orchestra.run.smithi047.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2022-09-07T21:40:49.901 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2022-09-07T21:40:49.901 INFO:teuthology.orchestra.run.smithi047.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2022-09-07T21:40:49.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:49.908 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2022-09-07T21:40:49.908 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2022-09-07T21:40:49.908 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2022-09-07T21:40:49.909 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2022-09-07T21:40:49.909 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2022-09-07T21:40:49.909 DEBUG:teuthology.orchestra.run.smithi047:> ') 2022-09-07T21:40:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:49 smithi055 ceph-mon[97033]: pgmap v383: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:40:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:49 smithi055 ceph-mon[97033]: osdmap e93: 12 total, 12 up, 12 in 2022-09-07T21:40:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:49 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/334378193' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-09-07T21:40:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:49 smithi059 ceph-mon[97450]: pgmap v383: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:40:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:49 smithi059 ceph-mon[97450]: osdmap e93: 12 total, 12 up, 12 in 2022-09-07T21:40:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:49 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/334378193' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-09-07T21:40:50.042 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:49 smithi047 ceph-mon[90087]: pgmap v383: 129 pgs: 129 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:40:50.042 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:49 smithi047 ceph-mon[90087]: osdmap e93: 12 total, 12 up, 12 in 2022-09-07T21:40:50.043 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:49 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/334378193' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-09-07T21:40:50.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:50.131 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2022-09-07T21:40:50.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:50.191 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list-id 2022-09-07T21:40:50.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:50.250 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2022-09-07T21:40:50.251 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2022-09-07T21:40:50.251 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2022-09-07T21:40:50.252 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2022-09-07T21:40:50.252 DEBUG:teuthology.orchestra.run.smithi047:> ') 2022-09-07T21:40:50.437 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2022-09-07T21:40:50.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:50.438 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2022-09-07T21:40:50.438 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2022-09-07T21:40:50.439 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2022-09-07T21:40:50.439 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2022-09-07T21:40:50.439 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2022-09-07T21:40:50.440 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2022-09-07T21:40:50.440 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2022-09-07T21:40:50.440 DEBUG:teuthology.orchestra.run.smithi047:> ') 2022-09-07T21:40:50.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:50.760 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2022-09-07T21:40:50.760 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2022-09-07T21:40:50.761 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set brx.0 up 2022-09-07T21:40:50.761 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set dev brx.0 master ceph-brx 2022-09-07T21:40:50.762 DEBUG:teuthology.orchestra.run.smithi047:> ') 2022-09-07T21:40:50.978 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-09-07T21:40:50.979 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2022-09-07T21:40:51.037 INFO:teuthology.orchestra.run.smithi047.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2022-09-07T21:40:51.038 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-09-07T21:40:51.038 DEBUG:teuthology.orchestra.run.smithi047:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2022-09-07T21:40:51.095 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/_nogroup/sv_1/3c62772f-2f12-4471-ad76-516abd941b3f 2022-09-07T21:40:51.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:51.096 DEBUG:teuthology.orchestra.run.smithi047:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/_nogroup/sv_1/3c62772f-2f12-4471-ad76-516abd941b3f /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2022-09-07T21:40:51.227 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2022-09-07T21:40:51.227 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2022-09-07T21:40:51.228 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2022-09-07T21:40:51.712 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2022-09-07T21:40:51.713 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2022-09-07T21:40:51.713 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2022-09-07T21:40:51.713 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.55:3300,172.21.15.59:3300" 2022-09-07T21:40:51.713 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:3300,172.21.15.55:3300,172.21.15.59:3300:/volumes/_nogroup/sv_1/3c62772f-2f12-4471-ad76-516abd941b3f 2022-09-07T21:40:51.714 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,key=0,fsid=01b8396e-2ef4-11ed-8431-001a4aab830c" will pass to kernel 2022-09-07T21:40:51.726 INFO:tasks.cephfs.kernel_mount:mount command passed 2022-09-07T21:40:51.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:51.727 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2022-09-07T21:40:51.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:51.820 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2022-09-07T21:40:51.937 INFO:teuthology.orchestra.run.smithi047.stdout:client25304 2022-09-07T21:40:51.937 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:40:51.938 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2022-09-07T21:40:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:51 smithi055 ceph-mon[97033]: pgmap v385: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-07T21:40:51.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:51.997 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2022-09-07T21:40:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:51 smithi059 ceph-mon[97450]: pgmap v385: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-07T21:40:52.114 INFO:teuthology.orchestra.run.smithi047.stdout:client25304 2022-09-07T21:40:52.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:52.115 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2022-09-07T21:40:52.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:51 smithi047 ceph-mon[90087]: pgmap v385: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-07T21:40:52.230 INFO:teuthology.orchestra.run.smithi047.stdout:client25304 2022-09-07T21:40:52.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-07T21:40:52.231 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c.client25304/status') 2022-09-07T21:40:52.409 INFO:teuthology.orchestra.run.smithi047.stdout:instance: client.25304 (3)172.21.15.47:0/4287182335 2022-09-07T21:40:52.410 INFO:teuthology.orchestra.run.smithi047.stdout:blocklisted: false 2022-09-07T21:40:52.410 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2022-09-07T21:40:52.410 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2022-09-07T21:40:52.411 INFO:teuthology.orchestra.run.smithi047.stderr:71 bytes copied, 7.3826e-05 s, 962 kB/s 2022-09-07T21:40:52.414 INFO:teuthology.run_tasks:Running task check-counter... 2022-09-07T21:40:52.426 INFO:teuthology.run_tasks:Running task exec... 2022-09-07T21:40:52.436 INFO:teuthology.task.exec:Executing custom commands... 2022-09-07T21:40:52.437 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi047.front.sepia.ceph.com 2022-09-07T21:40:52.437 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2022-09-07T21:40:52.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.659+0000 7f50c219d700 1 -- 172.21.15.47:0/2141846829 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50bc0fee80 msgr2=0x7f50bc101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:52.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.659+0000 7f50c219d700 1 --2- 172.21.15.47:0/2141846829 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50bc0fee80 0x7f50bc101300 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009810 tx=0x7f50ac009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:52.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.659+0000 7f50c219d700 1 -- 172.21.15.47:0/2141846829 shutdown_connections 2022-09-07T21:40:52.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.659+0000 7f50c219d700 1 --2- 172.21.15.47:0/2141846829 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50bc101840 0x7f50bc101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:52.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.659+0000 7f50c219d700 1 --2- 172.21.15.47:0/2141846829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fc520 0x7f50bc0fe940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:52.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.659+0000 7f50c219d700 1 --2- 172.21.15.47:0/2141846829 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50bc0fee80 0x7f50bc101300 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:52.665 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.659+0000 7f50c219d700 1 -- 172.21.15.47:0/2141846829 >> 172.21.15.47:0/2141846829 conn(0x7f50bc0f8230 msgr2=0x7f50bc0fa640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:52.665 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.659+0000 7f50c219d700 1 -- 172.21.15.47:0/2141846829 shutdown_connections 2022-09-07T21:40:52.665 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.659+0000 7f50c219d700 1 -- 172.21.15.47:0/2141846829 wait complete. 2022-09-07T21:40:52.667 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.660+0000 7f50c219d700 1 Processor -- start 2022-09-07T21:40:52.668 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.660+0000 7f50c219d700 1 -- start start 2022-09-07T21:40:52.668 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.661+0000 7f50c219d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fc520 0x7f50bc199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:52.668 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.661+0000 7f50c219d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50bc0fee80 0x7f50bc19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:52.668 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.661+0000 7f50c219d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50bc101840 0x7f50bc1a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:52.669 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.661+0000 7f50c219d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc1a1830 con 0x7f50bc0fee80 2022-09-07T21:40:52.669 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.661+0000 7f50c219d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc1a19a0 con 0x7f50bc101840 2022-09-07T21:40:52.669 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.661+0000 7f50c219d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc1a1b60 con 0x7f50bc0fc520 2022-09-07T21:40:52.669 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.661+0000 7f50c099a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50bc0fee80 0x7f50bc19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:52.670 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.661+0000 7f50c119b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fc520 0x7f50bc199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:52.670 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.661+0000 7f50c119b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fc520 0x7f50bc199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:40376/0 (socket says 172.21.15.47:40376) 2022-09-07T21:40:52.670 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.661+0000 7f50c119b700 1 -- 172.21.15.47:0/3737653914 learned_addr learned my addr 172.21.15.47:0/3737653914 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:52.670 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.661+0000 7f50c199c700 1 --2- 172.21.15.47:0/3737653914 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50bc101840 0x7f50bc1a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:52.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.662+0000 7f50c119b700 1 -- 172.21.15.47:0/3737653914 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50bc101840 msgr2=0x7f50bc1a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:52.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.662+0000 7f50c119b700 1 --2- 172.21.15.47:0/3737653914 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50bc101840 0x7f50bc1a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:52.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.662+0000 7f50c119b700 1 -- 172.21.15.47:0/3737653914 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50bc0fee80 msgr2=0x7f50bc19a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:52.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.662+0000 7f50c119b700 1 --2- 172.21.15.47:0/3737653914 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50bc0fee80 0x7f50bc19a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:52.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.662+0000 7f50c119b700 1 -- 172.21.15.47:0/3737653914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac009470 con 0x7f50bc0fc520 2022-09-07T21:40:52.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.662+0000 7f50c119b700 1 --2- 172.21.15.47:0/3737653914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fc520 0x7f50bc199c80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f50b800e740 tx=0x7f50b800ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:52.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.662+0000 7f50b27fc700 1 -- 172.21.15.47:0/3737653914 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b8005220 con 0x7f50bc0fc520 2022-09-07T21:40:52.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.662+0000 7f50c219d700 1 -- 172.21.15.47:0/3737653914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50bc1a1e40 con 0x7f50bc0fc520 2022-09-07T21:40:52.672 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.662+0000 7f50b27fc700 1 -- 172.21.15.47:0/3737653914 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f50b8005380 con 0x7f50bc0fc520 2022-09-07T21:40:52.673 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.662+0000 7f50b27fc700 1 -- 172.21.15.47:0/3737653914 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b8004b70 con 0x7f50bc0fc520 2022-09-07T21:40:52.673 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.662+0000 7f50c219d700 1 -- 172.21.15.47:0/3737653914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50bc1a2270 con 0x7f50bc0fc520 2022-09-07T21:40:52.673 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.663+0000 7f50c219d700 1 -- 172.21.15.47:0/3737653914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50bc10f9d0 con 0x7f50bc0fc520 2022-09-07T21:40:52.674 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.665+0000 7f50b27fc700 1 -- 172.21.15.47:0/3737653914 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f50b801e020 con 0x7f50bc0fc520 2022-09-07T21:40:52.674 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.667+0000 7f50b27fc700 1 --2- 172.21.15.47:0/3737653914 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f50a80990c0 0x7f50a809b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:52.675 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.667+0000 7f50b27fc700 1 -- 172.21.15.47:0/3737653914 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9103+0+0 (secure 0 0 0) 0x7f50b8014070 con 0x7f50bc0fc520 2022-09-07T21:40:52.675 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.667+0000 7f50c099a700 1 --2- 172.21.15.47:0/3737653914 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f50a80990c0 0x7f50a809b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:52.675 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.668+0000 7f50c099a700 1 --2- 172.21.15.47:0/3737653914 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f50a80990c0 0x7f50a809b570 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f50bc19b180 tx=0x7f50ac009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:52.675 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.669+0000 7f50b27fc700 1 -- 172.21.15.47:0/3737653914 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f50b808eef0 con 0x7f50bc0fc520 2022-09-07T21:40:52.937 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:52.933+0000 7f50c219d700 1 -- 172.21.15.47:0/3737653914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f50bc05f9d0 con 0x7f50bc0fc520 2022-09-07T21:40:53.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:53.714+0000 7f50b27fc700 1 -- 172.21.15.47:0/3737653914 <== mon.1 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 19) v1 ==== 127695+0+0 (secure 0 0 0) 0x7f50b800c040 con 0x7f50bc0fc520 2022-09-07T21:40:53.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:53.716+0000 7f50b27fc700 1 -- 172.21.15.47:0/3737653914 <== mon.1 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f50b8004cd0 con 0x7f50bc0fc520 2022-09-07T21:40:53.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:53.720+0000 7f50c219d700 1 -- 172.21.15.47:0/3737653914 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f50a80990c0 msgr2=0x7f50a809b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:53.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:53.720+0000 7f50c219d700 1 --2- 172.21.15.47:0/3737653914 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f50a80990c0 0x7f50a809b570 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f50bc19b180 tx=0x7f50ac009eb0 comp rx=0 tx=0).stop 2022-09-07T21:40:53.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:53.720+0000 7f50c219d700 1 -- 172.21.15.47:0/3737653914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fc520 msgr2=0x7f50bc199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:53.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:53.720+0000 7f50c219d700 1 --2- 172.21.15.47:0/3737653914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fc520 0x7f50bc199c80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f50b800e740 tx=0x7f50b800ea50 comp rx=0 tx=0).stop 2022-09-07T21:40:53.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:53.720+0000 7f50c219d700 1 -- 172.21.15.47:0/3737653914 shutdown_connections 2022-09-07T21:40:53.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:53.720+0000 7f50c219d700 1 --2- 172.21.15.47:0/3737653914 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f50a80990c0 0x7f50a809b570 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:53.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:53.720+0000 7f50c219d700 1 --2- 172.21.15.47:0/3737653914 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50bc101840 0x7f50bc1a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:53.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:53.720+0000 7f50c219d700 1 --2- 172.21.15.47:0/3737653914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fc520 0x7f50bc199c80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:53.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:53.720+0000 7f50c219d700 1 --2- 172.21.15.47:0/3737653914 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50bc0fee80 0x7f50bc19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:53.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:53.720+0000 7f50c219d700 1 -- 172.21.15.47:0/3737653914 >> 172.21.15.47:0/3737653914 conn(0x7f50bc0f8230 msgr2=0x7f50bc0fd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:53.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:53.720+0000 7f50c219d700 1 -- 172.21.15.47:0/3737653914 shutdown_connections 2022-09-07T21:40:53.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:53.720+0000 7f50c219d700 1 -- 172.21.15.47:0/3737653914 wait complete. 2022-09-07T21:40:53.764 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2022-09-07T21:40:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:53 smithi055 ceph-mon[97033]: pgmap v386: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:40:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:53 smithi055 ceph-mon[97033]: from='client.? 172.21.15.47:0/3737653914' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2022-09-07T21:40:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:53 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2022-09-07T21:40:53.981 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:53 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: ignoring --setuser ceph since I am not root 2022-09-07T21:40:53.981 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:53 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: ignoring --setgroup ceph since I am not root 2022-09-07T21:40:53.981 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:53 smithi055 ceph-mgr[98436]: -- 172.21.15.55:0/1783513830 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c8e376a820 con 0x55c8e373ec00 2022-09-07T21:40:53.982 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:53 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:53.902+0000 7f4315f1b080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-07T21:40:54.005 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:53 smithi059 ceph-mon[97450]: pgmap v386: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:40:54.006 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:53 smithi059 ceph-mon[97450]: from='client.? 172.21.15.47:0/3737653914' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2022-09-07T21:40:54.006 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:53 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2022-09-07T21:40:54.007 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:53 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: ignoring --setuser ceph since I am not root 2022-09-07T21:40:54.007 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:53 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: ignoring --setgroup ceph since I am not root 2022-09-07T21:40:54.007 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:53 smithi059 ceph-mgr[99530]: -- 172.21.15.59:0/1408968140 <== mon.1 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5611a676c820 con 0x5611a6740800 2022-09-07T21:40:54.007 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:53 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:53.898+0000 7f038fa6e080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-07T21:40:54.008 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:54 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:54.002+0000 7f038fa6e080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-07T21:40:54.011 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.004+0000 7f01249ff700 1 -- 172.21.15.47:0/2470747468 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0120072c90 msgr2=0x7f0120071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:54.011 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.004+0000 7f01249ff700 1 --2- 172.21.15.47:0/2470747468 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0120072c90 0x7f0120071170 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0110009750 tx=0x7f0110009a60 comp rx=0 tx=0).stop 2022-09-07T21:40:54.011 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.004+0000 7f01249ff700 1 -- 172.21.15.47:0/2470747468 shutdown_connections 2022-09-07T21:40:54.012 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.004+0000 7f01249ff700 1 --2- 172.21.15.47:0/2470747468 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f012010a430 0x7f012010c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.012 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.004+0000 7f01249ff700 1 --2- 172.21.15.47:0/2470747468 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0120072c90 0x7f0120071170 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.012 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.004+0000 7f01249ff700 1 --2- 172.21.15.47:0/2470747468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0120071740 0x7f0120071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.013 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.004+0000 7f01249ff700 1 -- 172.21.15.47:0/2470747468 >> 172.21.15.47:0/2470747468 conn(0x7f01200f9480 msgr2=0x7f01200fb8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:54.013 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.004+0000 7f01249ff700 1 -- 172.21.15.47:0/2470747468 shutdown_connections 2022-09-07T21:40:54.013 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.004+0000 7f01249ff700 1 -- 172.21.15.47:0/2470747468 wait complete. 2022-09-07T21:40:54.013 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.005+0000 7f01249ff700 1 Processor -- start 2022-09-07T21:40:54.013 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.005+0000 7f01249ff700 1 -- start start 2022-09-07T21:40:54.014 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.005+0000 7f01249ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0120071740 0x7f012019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:54.014 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.005+0000 7f01249ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0120072c90 0x7f012019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:54.014 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f01249ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f012010a430 0x7f01201a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:54.014 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f01249ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01201a1c30 con 0x7f012010a430 2022-09-07T21:40:54.015 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f01249ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01201a1da0 con 0x7f0120072c90 2022-09-07T21:40:54.015 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f01249ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01201a1f60 con 0x7f0120071740 2022-09-07T21:40:54.015 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f011e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0120071740 0x7f012019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:54.015 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f011e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0120071740 0x7f012019a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:40398/0 (socket says 172.21.15.47:40398) 2022-09-07T21:40:54.016 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f011e59c700 1 -- 172.21.15.47:0/2859299950 learned_addr learned my addr 172.21.15.47:0/2859299950 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:54.016 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f011dd9b700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0120072c90 0x7f012019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:54.016 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f011ed9d700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f012010a430 0x7f01201a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:54.016 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f011dd9b700 1 -- 172.21.15.47:0/2859299950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0120071740 msgr2=0x7f012019a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:54.017 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f011dd9b700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0120071740 0x7f012019a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.017 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f011dd9b700 1 -- 172.21.15.47:0/2859299950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f012010a430 msgr2=0x7f01201a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:54.017 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f011dd9b700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f012010a430 0x7f01201a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.017 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f011dd9b700 1 -- 172.21.15.47:0/2859299950 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01100093b0 con 0x7f0120072c90 2022-09-07T21:40:54.017 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.006+0000 7f011e59c700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0120071740 0x7f012019a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:40:54.018 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.007+0000 7f011dd9b700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0120072c90 0x7f012019a5f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0114009fd0 tx=0x7f011400e9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:54.018 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.007+0000 7f010f7fe700 1 -- 172.21.15.47:0/2859299950 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0114010900 con 0x7f0120072c90 2022-09-07T21:40:54.018 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.007+0000 7f010f7fe700 1 -- 172.21.15.47:0/2859299950 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0114018420 con 0x7f0120072c90 2022-09-07T21:40:54.018 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.007+0000 7f01249ff700 1 -- 172.21.15.47:0/2859299950 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01201a2240 con 0x7f0120072c90 2022-09-07T21:40:54.019 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.007+0000 7f010f7fe700 1 -- 172.21.15.47:0/2859299950 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01140175f0 con 0x7f0120072c90 2022-09-07T21:40:54.019 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.007+0000 7f01249ff700 1 -- 172.21.15.47:0/2859299950 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01201a27a0 con 0x7f0120072c90 2022-09-07T21:40:54.020 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.008+0000 7f01249ff700 1 -- 172.21.15.47:0/2859299950 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012010fbd0 con 0x7f0120072c90 2022-09-07T21:40:54.020 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.010+0000 7f010f7fe700 1 -- 172.21.15.47:0/2859299950 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 127695+0+0 (secure 0 0 0) 0x7f011400c040 con 0x7f0120072c90 2022-09-07T21:40:54.020 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.011+0000 7f010f7fe700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f01080990e0 0x7f010809b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:54.021 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.011+0000 7f010f7fe700 1 -- 172.21.15.47:0/2859299950 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9103+0+0 (secure 0 0 0) 0x7f0114014070 con 0x7f0120072c90 2022-09-07T21:40:54.021 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.011+0000 7f011e59c700 1 -- 172.21.15.47:0/2859299950 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f01080990e0 msgr2=0x7f010809b590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/429234829 2022-09-07T21:40:54.021 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.011+0000 7f011e59c700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f01080990e0 0x7f010809b590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-07T21:40:54.021 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.015+0000 7f010f7fe700 1 -- 172.21.15.47:0/2859299950 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f011404fc80 con 0x7f0120072c90 2022-09-07T21:40:54.125 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:53 smithi047 ceph-mon[90087]: pgmap v386: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:40:54.126 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:53 smithi047 ceph-mon[90087]: from='client.? 172.21.15.47:0/3737653914' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2022-09-07T21:40:54.126 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:53 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2022-09-07T21:40:54.126 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: ignoring --setuser ceph since I am not root 2022-09-07T21:40:54.127 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: ignoring --setgroup ceph since I am not root 2022-09-07T21:40:54.127 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:53 smithi047 ceph-mgr[90360]: -- 172.21.15.47:0/3960736433 <== mon.1 v2:172.21.15.59:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5574324ce820 con 0x5574324a3000 2022-09-07T21:40:54.127 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:54.121+0000 7f3feec2b080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-07T21:40:54.215 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.212+0000 7f011e59c700 1 -- 172.21.15.47:0/2859299950 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f01080990e0 msgr2=0x7f010809b590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/429234829 2022-09-07T21:40:54.216 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.212+0000 7f011e59c700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f01080990e0 0x7f010809b590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-09-07T21:40:54.222 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.219+0000 7f01249ff700 1 -- 172.21.15.47:0/2859299950 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f012005f990 con 0x7f0120072c90 2022-09-07T21:40:54.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.229+0000 7f010f7fe700 1 -- 172.21.15.47:0/2859299950 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f011408eaf0 con 0x7f0120072c90 2022-09-07T21:40:54.235 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.231+0000 7f01249ff700 1 -- 172.21.15.47:0/2859299950 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f01080990e0 msgr2=0x7f010809b590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-07T21:40:54.235 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.231+0000 7f01249ff700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f01080990e0 0x7f010809b590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.231+0000 7f01249ff700 1 -- 172.21.15.47:0/2859299950 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0120072c90 msgr2=0x7f012019a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:54.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.231+0000 7f01249ff700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0120072c90 0x7f012019a5f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0114009fd0 tx=0x7f011400e9a0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.231+0000 7f01249ff700 1 -- 172.21.15.47:0/2859299950 shutdown_connections 2022-09-07T21:40:54.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.231+0000 7f01249ff700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f01080990e0 0x7f010809b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.231+0000 7f01249ff700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f012010a430 0x7f01201a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.231+0000 7f01249ff700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0120071740 0x7f012019a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.231+0000 7f01249ff700 1 --2- 172.21.15.47:0/2859299950 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0120072c90 0x7f012019a5f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.231+0000 7f01249ff700 1 -- 172.21.15.47:0/2859299950 >> 172.21.15.47:0/2859299950 conn(0x7f01200f9480 msgr2=0x7f01200fb860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:54.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.231+0000 7f01249ff700 1 -- 172.21.15.47:0/2859299950 shutdown_connections 2022-09-07T21:40:54.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.232+0000 7f01249ff700 1 -- 172.21.15.47:0/2859299950 wait complete. 2022-09-07T21:40:54.263 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2022-09-07T21:40:54.281 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:54 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:54.111+0000 7f038fa6e080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-07T21:40:54.281 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:54 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:54.220+0000 7f038fa6e080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-07T21:40:54.379 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:54.236+0000 7f3feec2b080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-07T21:40:54.422 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:54 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:54.003+0000 7f4315f1b080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-07T21:40:54.423 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:54 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:54.110+0000 7f4315f1b080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-07T21:40:54.423 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:54 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:54.218+0000 7f4315f1b080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-07T21:40:54.496 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.492+0000 7f118d79d700 1 -- 172.21.15.47:0/715257759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1188073610 msgr2=0x7f1188073a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:54.497 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.492+0000 7f118d79d700 1 --2- 172.21.15.47:0/715257759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1188073610 0x7f1188073a20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1178009810 tx=0x7f1178009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:54.497 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.493+0000 7f118d79d700 1 -- 172.21.15.47:0/715257759 shutdown_connections 2022-09-07T21:40:54.497 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.493+0000 7f118d79d700 1 --2- 172.21.15.47:0/715257759 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1188075290 0x7f1188075b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.498 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.493+0000 7f118d79d700 1 --2- 172.21.15.47:0/715257759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1188073610 0x7f1188073a20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.498 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.493+0000 7f118d79d700 1 --2- 172.21.15.47:0/715257759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11880748e0 0x7f1188074d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.498 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.493+0000 7f118d79d700 1 -- 172.21.15.47:0/715257759 >> 172.21.15.47:0/715257759 conn(0x7f11880fb6b0 msgr2=0x7f11880fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.493+0000 7f118d79d700 1 -- 172.21.15.47:0/715257759 shutdown_connections 2022-09-07T21:40:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.493+0000 7f118d79d700 1 -- 172.21.15.47:0/715257759 wait complete. 2022-09-07T21:40:54.501 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.494+0000 7f118d79d700 1 Processor -- start 2022-09-07T21:40:54.502 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.494+0000 7f118d79d700 1 -- start start 2022-09-07T21:40:54.502 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.494+0000 7f118d79d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1188073610 0x7f1188199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:54.502 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f118d79d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11880748e0 0x7f118819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:54.503 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f118d79d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1188075290 0x7f11881a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:54.503 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f118d79d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11881a18e0 con 0x7f1188073610 2022-09-07T21:40:54.503 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f118d79d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11881a1a50 con 0x7f11880748e0 2022-09-07T21:40:54.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f118d79d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11881a1c10 con 0x7f1188075290 2022-09-07T21:40:54.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f11877fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11880748e0 0x7f118819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:54.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f1187fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1188073610 0x7f1188199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:54.504 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f11877fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11880748e0 0x7f118819a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:57186/0 (socket says 172.21.15.47:57186) 2022-09-07T21:40:54.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f11877fe700 1 -- 172.21.15.47:0/1853746373 learned_addr learned my addr 172.21.15.47:0/1853746373 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:54.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f118cf9c700 1 --2- 172.21.15.47:0/1853746373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1188075290 0x7f11881a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:54.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f11877fe700 1 -- 172.21.15.47:0/1853746373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1188075290 msgr2=0x7f11881a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:54.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f11877fe700 1 --2- 172.21.15.47:0/1853746373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1188075290 0x7f11881a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.505 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f11877fe700 1 -- 172.21.15.47:0/1853746373 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1188073610 msgr2=0x7f1188199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:54.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f11877fe700 1 --2- 172.21.15.47:0/1853746373 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1188073610 0x7f1188199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f11877fe700 1 -- 172.21.15.47:0/1853746373 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1178009470 con 0x7f11880748e0 2022-09-07T21:40:54.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.495+0000 7f118cf9c700 1 --2- 172.21.15.47:0/1853746373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1188075290 0x7f11881a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:40:54.506 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.496+0000 7f11877fe700 1 --2- 172.21.15.47:0/1853746373 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11880748e0 0x7f118819a260 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f117c00e740 tx=0x7f117c00ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.496+0000 7f11857fa700 1 -- 172.21.15.47:0/1853746373 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c005220 con 0x7f11880748e0 2022-09-07T21:40:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.496+0000 7f11857fa700 1 -- 172.21.15.47:0/1853746373 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f117c005380 con 0x7f11880748e0 2022-09-07T21:40:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.496+0000 7f118d79d700 1 -- 172.21.15.47:0/1853746373 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11881a1ef0 con 0x7f11880748e0 2022-09-07T21:40:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.496+0000 7f11857fa700 1 -- 172.21.15.47:0/1853746373 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c004c10 con 0x7f11880748e0 2022-09-07T21:40:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.496+0000 7f118d79d700 1 -- 172.21.15.47:0/1853746373 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11881a2310 con 0x7f11880748e0 2022-09-07T21:40:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.497+0000 7f118d79d700 1 -- 172.21.15.47:0/1853746373 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118810fa60 con 0x7f11880748e0 2022-09-07T21:40:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.499+0000 7f11857fa700 1 -- 172.21.15.47:0/1853746373 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 127695+0+0 (secure 0 0 0) 0x7f117c01e020 con 0x7f11880748e0 2022-09-07T21:40:54.510 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.501+0000 7f11857fa700 1 --2- 172.21.15.47:0/1853746373 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1170099110 0x7f117009b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:54.510 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.501+0000 7f1187fff700 1 -- 172.21.15.47:0/1853746373 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1170099110 msgr2=0x7f117009b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/429234829 2022-09-07T21:40:54.510 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.501+0000 7f11857fa700 1 -- 172.21.15.47:0/1853746373 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9103+0+0 (secure 0 0 0) 0x7f117c014070 con 0x7f11880748e0 2022-09-07T21:40:54.510 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.501+0000 7f1187fff700 1 --2- 172.21.15.47:0/1853746373 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1170099110 0x7f117009b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-07T21:40:54.511 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.504+0000 7f11857fa700 1 -- 172.21.15.47:0/1853746373 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f117c08ee50 con 0x7f11880748e0 2022-09-07T21:40:54.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:54.375+0000 7f3feec2b080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-07T21:40:54.654 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:54.524+0000 7f3feec2b080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-07T21:40:54.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.702+0000 7f1187fff700 1 -- 172.21.15.47:0/1853746373 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1170099110 msgr2=0x7f117009b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/429234829 2022-09-07T21:40:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.702+0000 7f1187fff700 1 --2- 172.21.15.47:0/1853746373 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1170099110 0x7f117009b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-09-07T21:40:54.722 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:54 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:54.421+0000 7f038fa6e080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-07T21:40:54.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.750+0000 7f118d79d700 1 -- 172.21.15.47:0/1853746373 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f118805f120 con 0x7f11880748e0 2022-09-07T21:40:54.759 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.755+0000 7f11857fa700 1 -- 172.21.15.47:0/1853746373 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f117c092470 con 0x7f11880748e0 2022-09-07T21:40:54.762 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.758+0000 7f118d79d700 1 -- 172.21.15.47:0/1853746373 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1170099110 msgr2=0x7f117009b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-07T21:40:54.762 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.758+0000 7f118d79d700 1 --2- 172.21.15.47:0/1853746373 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1170099110 0x7f117009b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.762 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.758+0000 7f118d79d700 1 -- 172.21.15.47:0/1853746373 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11880748e0 msgr2=0x7f118819a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:54.762 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.758+0000 7f118d79d700 1 --2- 172.21.15.47:0/1853746373 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11880748e0 0x7f118819a260 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f117c00e740 tx=0x7f117c00ea50 comp rx=0 tx=0).stop 2022-09-07T21:40:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.758+0000 7f118d79d700 1 -- 172.21.15.47:0/1853746373 shutdown_connections 2022-09-07T21:40:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.758+0000 7f118d79d700 1 --2- 172.21.15.47:0/1853746373 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f1170099110 0x7f117009b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.758+0000 7f118d79d700 1 --2- 172.21.15.47:0/1853746373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1188075290 0x7f11881a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.758+0000 7f118d79d700 1 --2- 172.21.15.47:0/1853746373 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1188073610 0x7f1188199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.764 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.758+0000 7f118d79d700 1 --2- 172.21.15.47:0/1853746373 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11880748e0 0x7f118819a260 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:54.764 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.758+0000 7f118d79d700 1 -- 172.21.15.47:0/1853746373 >> 172.21.15.47:0/1853746373 conn(0x7f11880fb6b0 msgr2=0x7f11880fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:54.764 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.758+0000 7f118d79d700 1 -- 172.21.15.47:0/1853746373 shutdown_connections 2022-09-07T21:40:54.765 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:54.758+0000 7f118d79d700 1 -- 172.21.15.47:0/1853746373 wait complete. 2022-09-07T21:40:54.788 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M' 2022-09-07T21:40:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:54 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2022-09-07T21:40:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:54 smithi055 ceph-mon[97033]: mgrmap e19: x(active, since 10m), standbys: y, z 2022-09-07T21:40:54.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:54 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' 2022-09-07T21:40:54.981 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:54 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:54.419+0000 7f4315f1b080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-07T21:40:54.981 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:54 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:54.816+0000 7f4315f1b080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-07T21:40:54.982 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:54 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:54.922+0000 7f4315f1b080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-07T21:40:55.020 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.016+0000 7f02c73ff700 1 -- 172.21.15.47:0/1510705136 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02c00ff080 msgr2=0x7f02c00ff4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:55.021 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.016+0000 7f02c73ff700 1 --2- 172.21.15.47:0/1510705136 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02c00ff080 0x7f02c00ff4f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009810 tx=0x7f02b0009b20 comp rx=0 tx=0).stop 2022-09-07T21:40:55.021 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.017+0000 7f02c73ff700 1 -- 172.21.15.47:0/1510705136 shutdown_connections 2022-09-07T21:40:55.021 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.017+0000 7f02c73ff700 1 --2- 172.21.15.47:0/1510705136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02c00ffa30 0x7f02c01062d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:55.022 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.017+0000 7f02c73ff700 1 --2- 172.21.15.47:0/1510705136 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02c00fdea0 0x7f02c00fe2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:55.022 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.017+0000 7f02c73ff700 1 --2- 172.21.15.47:0/1510705136 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02c00ff080 0x7f02c00ff4f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:55.022 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.017+0000 7f02c73ff700 1 -- 172.21.15.47:0/1510705136 >> 172.21.15.47:0/1510705136 conn(0x7f02c00f9510 msgr2=0x7f02c00fb920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:40:55.023 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.017+0000 7f02c73ff700 1 -- 172.21.15.47:0/1510705136 shutdown_connections 2022-09-07T21:40:55.023 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.017+0000 7f02c73ff700 1 -- 172.21.15.47:0/1510705136 wait complete. 2022-09-07T21:40:55.025 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.019+0000 7f02c73ff700 1 Processor -- start 2022-09-07T21:40:55.025 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.019+0000 7f02c73ff700 1 -- start start 2022-09-07T21:40:55.026 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.019+0000 7f02c73ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02c00fdea0 0x7f02c0072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:55.026 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.019+0000 7f02c73ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02c00ff080 0x7f02c00710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:55.027 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.020+0000 7f02c73ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02c00ffa30 0x7f02c0071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:55.027 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.020+0000 7f02c73ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c0071e10 con 0x7f02c00ff080 2022-09-07T21:40:55.027 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.020+0000 7f02c73ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c0071f80 con 0x7f02c00fdea0 2022-09-07T21:40:55.028 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.020+0000 7f02c73ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c019da10 con 0x7f02c00ffa30 2022-09-07T21:40:55.028 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.020+0000 7f02c519b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02c00fdea0 0x7f02c0072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:55.029 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.020+0000 7f02c519b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02c00fdea0 0x7f02c0072a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:57202/0 (socket says 172.21.15.47:57202) 2022-09-07T21:40:55.029 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.020+0000 7f02c519b700 1 -- 172.21.15.47:0/1492268480 learned_addr learned my addr 172.21.15.47:0/1492268480 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:40:55.029 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.020+0000 7f02c599c700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02c00ffa30 0x7f02c0071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:55.030 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.020+0000 7f02c499a700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02c00ff080 0x7f02c00710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:40:55.030 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.020+0000 7f02c519b700 1 -- 172.21.15.47:0/1492268480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02c00ffa30 msgr2=0x7f02c0071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:55.031 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.020+0000 7f02c519b700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02c00ffa30 0x7f02c0071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:55.031 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.020+0000 7f02c519b700 1 -- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02c00ff080 msgr2=0x7f02c00710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:40:55.032 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.020+0000 7f02c519b700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02c00ff080 0x7f02c00710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:40:55.032 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.020+0000 7f02c519b700 1 -- 172.21.15.47:0/1492268480 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b0009470 con 0x7f02c00fdea0 2022-09-07T21:40:55.033 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.021+0000 7f02c519b700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02c00fdea0 0x7f02c0072a90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00e740 tx=0x7f02bc00ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:40:55.033 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.021+0000 7f02b67fc700 1 -- 172.21.15.47:0/1492268480 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc005220 con 0x7f02c00fdea0 2022-09-07T21:40:55.033 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.021+0000 7f02b67fc700 1 -- 172.21.15.47:0/1492268480 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f02bc005380 con 0x7f02c00fdea0 2022-09-07T21:40:55.033 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.021+0000 7f02c73ff700 1 -- 172.21.15.47:0/1492268480 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c019dcf0 con 0x7f02c00fdea0 2022-09-07T21:40:55.034 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.021+0000 7f02b67fc700 1 -- 172.21.15.47:0/1492268480 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc004c10 con 0x7f02c00fdea0 2022-09-07T21:40:55.034 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.021+0000 7f02c73ff700 1 -- 172.21.15.47:0/1492268480 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c019e1a0 con 0x7f02c00fdea0 2022-09-07T21:40:55.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:54 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2022-09-07T21:40:55.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:54 smithi059 ceph-mon[97450]: mgrmap e19: x(active, since 10m), standbys: y, z 2022-09-07T21:40:55.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:54 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' 2022-09-07T21:40:55.036 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:54 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:54.824+0000 7f038fa6e080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-07T21:40:55.036 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:54 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:54.930+0000 7f038fa6e080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-07T21:40:55.037 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.023+0000 7f02c73ff700 1 -- 172.21.15.47:0/1492268480 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c0109560 con 0x7f02c00fdea0 2022-09-07T21:40:55.037 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.024+0000 7f02b67fc700 1 -- 172.21.15.47:0/1492268480 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 127695+0+0 (secure 0 0 0) 0x7f02bc01e020 con 0x7f02c00fdea0 2022-09-07T21:40:55.038 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.026+0000 7f02b67fc700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 0x7f02ac09b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:40:55.038 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.026+0000 7f02b67fc700 1 -- 172.21.15.47:0/1492268480 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9103+0+0 (secure 0 0 0) 0x7f02bc014070 con 0x7f02c00fdea0 2022-09-07T21:40:55.038 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.026+0000 7f02c499a700 1 -- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 msgr2=0x7f02ac09b590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/429234829 2022-09-07T21:40:55.038 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.026+0000 7f02c499a700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 0x7f02ac09b590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-07T21:40:55.038 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.029+0000 7f02b67fc700 1 -- 172.21.15.47:0/1492268480 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f02bc08edf0 con 0x7f02c00fdea0 2022-09-07T21:40:55.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:54 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2022-09-07T21:40:55.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:54 smithi047 ceph-mon[90087]: mgrmap e19: x(active, since 10m), standbys: y, z 2022-09-07T21:40:55.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:54 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' 2022-09-07T21:40:55.154 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:54.810+0000 7f3feec2b080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-07T21:40:55.230 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.227+0000 7f02c499a700 1 -- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 msgr2=0x7f02ac09b590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/429234829 2022-09-07T21:40:55.231 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.227+0000 7f02c499a700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 0x7f02ac09b590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-09-07T21:40:55.274 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.271+0000 7f02c73ff700 1 -- 172.21.15.47:0/1492268480 --> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f02c005f120 con 0x7f02ac0990e0 2022-09-07T21:40:55.480 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:55 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:55.071+0000 7f4315f1b080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-07T21:40:55.480 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:55 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:55.183+0000 7f4315f1b080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-07T21:40:55.531 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:55 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:55.080+0000 7f038fa6e080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-07T21:40:55.532 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:55 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:55.192+0000 7f038fa6e080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-07T21:40:55.605 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:55.283+0000 7f3feec2b080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-07T21:40:55.606 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:55.419+0000 7f3feec2b080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-07T21:40:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.628+0000 7f02c499a700 1 -- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 msgr2=0x7f02ac09b590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/429234829 2022-09-07T21:40:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:55.628+0000 7f02c499a700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 0x7f02ac09b590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2022-09-07T21:40:55.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:40:55 smithi047 ceph-mon[90087]: from='client.? ' entity='client.admin' 2022-09-07T21:40:55.904 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:55.602+0000 7f3feec2b080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-07T21:40:55.904 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:55.746+0000 7f3feec2b080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-07T21:40:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:40:55 smithi055 ceph-mon[97033]: from='client.? ' entity='client.admin' 2022-09-07T21:40:55.980 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:55 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:55.686+0000 7f4315f1b080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-07T21:40:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:40:55 smithi059 ceph-mon[97450]: from='client.? ' entity='client.admin' 2022-09-07T21:40:56.032 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:55 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:55.696+0000 7f038fa6e080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-07T21:40:56.432 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:56.429+0000 7f02c499a700 1 -- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 msgr2=0x7f02ac09b590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/429234829 2022-09-07T21:40:56.432 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:56.429+0000 7f02c499a700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 0x7f02ac09b590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2022-09-07T21:40:56.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:56.371+0000 7f3feec2b080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-07T21:40:56.730 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:56 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:56.362+0000 7f4315f1b080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-07T21:40:56.730 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:56 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:56.553+0000 7f4315f1b080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-07T21:40:56.781 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:56 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:56.375+0000 7f038fa6e080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-07T21:40:56.782 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:56 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:56.567+0000 7f038fa6e080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-07T21:40:57.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:57.233+0000 7f3feec2b080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-07T21:40:57.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:57.475+0000 7f3feec2b080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-07T21:40:57.781 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:57 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:57.509+0000 7f038fa6e080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:40:57.782 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:57 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:57.617+0000 7f038fa6e080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-07T21:40:57.819 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:57 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:57.496+0000 7f4315f1b080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:40:57.819 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:57 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:57.603+0000 7f4315f1b080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-07T21:40:58.033 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:58.031+0000 7f02c499a700 1 -- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 msgr2=0x7f02ac09b590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/429234829 2022-09-07T21:40:58.033 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:40:58.031+0000 7f02c499a700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 0x7f02ac09b590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2022-09-07T21:40:58.220 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:57 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:57.815+0000 7f4315f1b080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:40:58.220 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:58 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:58.036+0000 7f4315f1b080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-07T21:40:58.233 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:57 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:57.829+0000 7f038fa6e080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:40:58.233 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:58 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:58.051+0000 7f038fa6e080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-07T21:40:58.480 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:58 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:58.216+0000 7f4315f1b080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-07T21:40:58.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:58.603+0000 7f3feec2b080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:40:58.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:58.757+0000 7f3feec2b080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-07T21:40:58.980 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:58 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:58.680+0000 7f4315f1b080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-07T21:40:59.031 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:58 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:58.230+0000 7f038fa6e080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-07T21:40:59.032 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:58 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:58.695+0000 7f038fa6e080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-07T21:40:59.373 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:59.049+0000 7f3feec2b080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-07T21:40:59.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:59.370+0000 7f3feec2b080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-07T21:40:59.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:40:59.612+0000 7f3feec2b080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-07T21:40:59.656 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:59 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:59.310+0000 7f4315f1b080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-07T21:40:59.672 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:59 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:59.326+0000 7f038fa6e080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-07T21:40:59.980 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:59 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:59.653+0000 7f4315f1b080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-07T21:40:59.980 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:40:59 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:40:59.888+0000 7f4315f1b080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-07T21:41:00.031 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:59 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:59.669+0000 7f038fa6e080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-07T21:41:00.032 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:40:59 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:40:59.904+0000 7f038fa6e080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-07T21:41:00.288 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:41:00 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:41:00.022+0000 7f4315f1b080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-07T21:41:00.304 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:41:00 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:41:00.038+0000 7f038fa6e080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-07T21:41:00.653 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:00.232+0000 7f3feec2b080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-07T21:41:00.730 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:41:00 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:41:00.285+0000 7f4315f1b080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-07T21:41:01.156 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:01.154+0000 7f02b67fc700 1 -- 172.21.15.47:0/1492268480 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 20) v1 ==== 127695+0+0 (secure 0 0 0) 0x7f02bc00c040 con 0x7f02c00fdea0 2022-09-07T21:41:01.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:01.234+0000 7f02c499a700 1 -- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 msgr2=0x7f02ac09b590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/429234829 2022-09-07T21:41:01.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:01.234+0000 7f02c499a700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 0x7f02ac09b590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2022-09-07T21:41:01.480 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 07 21:41:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-07T21:41:01.076+0000 7f4315f1b080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-07T21:41:01.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:01 smithi055 ceph-mon[97033]: Standby manager daemon y restarted 2022-09-07T21:41:01.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:01 smithi055 ceph-mon[97033]: Standby manager daemon y started 2022-09-07T21:41:01.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:01 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.55:0/3961654595' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-09-07T21:41:01.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:01 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.55:0/3961654595' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-07T21:41:01.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:01 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.55:0/3961654595' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-09-07T21:41:01.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:01 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.55:0/3961654595' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-07T21:41:01.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:01 smithi055 ceph-mon[97033]: Standby manager daemon z restarted 2022-09-07T21:41:01.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:01 smithi055 ceph-mon[97033]: Standby manager daemon z started 2022-09-07T21:41:01.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:01 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.59:0/757973815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-09-07T21:41:01.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:01 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.59:0/757973815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-07T21:41:01.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:01 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.59:0/757973815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-09-07T21:41:01.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:01 smithi055 ceph-mon[97033]: from='mgr.? 172.21.15.59:0/757973815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-07T21:41:01.531 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:41:00 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:41:00.301+0000 7f038fa6e080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-07T21:41:01.532 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 07 21:41:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-07T21:41:01.093+0000 7f038fa6e080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-07T21:41:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:01 smithi059 ceph-mon[97450]: Standby manager daemon y restarted 2022-09-07T21:41:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:01 smithi059 ceph-mon[97450]: Standby manager daemon y started 2022-09-07T21:41:01.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:01 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.55:0/3961654595' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-09-07T21:41:01.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:01 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.55:0/3961654595' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-07T21:41:01.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:01 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.55:0/3961654595' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-09-07T21:41:01.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:01 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.55:0/3961654595' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-07T21:41:01.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:01 smithi059 ceph-mon[97450]: Standby manager daemon z restarted 2022-09-07T21:41:01.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:01 smithi059 ceph-mon[97450]: Standby manager daemon z started 2022-09-07T21:41:01.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:01 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.59:0/757973815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-09-07T21:41:01.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:01 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.59:0/757973815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-07T21:41:01.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:01 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.59:0/757973815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-09-07T21:41:01.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:01 smithi059 ceph-mon[97450]: from='mgr.? 172.21.15.59:0/757973815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-07T21:41:01.638 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:01.199+0000 7f3feec2b080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-07T21:41:01.638 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-mon[90087]: Standby manager daemon y restarted 2022-09-07T21:41:01.639 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-mon[90087]: Standby manager daemon y started 2022-09-07T21:41:01.639 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.55:0/3961654595' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-09-07T21:41:01.639 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.55:0/3961654595' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-07T21:41:01.639 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.55:0/3961654595' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-09-07T21:41:01.640 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.55:0/3961654595' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-07T21:41:01.640 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-mon[90087]: Standby manager daemon z restarted 2022-09-07T21:41:01.640 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-mon[90087]: Standby manager daemon z started 2022-09-07T21:41:01.640 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.59:0/757973815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-09-07T21:41:01.640 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.59:0/757973815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-07T21:41:01.641 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.59:0/757973815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-09-07T21:41:01.641 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-mon[90087]: from='mgr.? 172.21.15.59:0/757973815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-07T21:41:01.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:01.635+0000 7f3feec2b080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-07T21:41:02.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:02 smithi047 ceph-mon[90087]: mgrmap e20: x(active, since 10m), standbys: y, z 2022-09-07T21:41:02.404 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:01.982+0000 7f3feec2b080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-07T21:41:02.404 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:02.180+0000 7f3feec2b080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-07T21:41:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:02 smithi055 ceph-mon[97033]: mgrmap e20: x(active, since 10m), standbys: y, z 2022-09-07T21:41:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:02 smithi059 ceph-mon[97450]: mgrmap e20: x(active, since 10m), standbys: y, z 2022-09-07T21:41:02.903 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:02.557+0000 7f3feec2b080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-07T21:41:03.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:03.687+0000 7f02b67fc700 1 -- 172.21.15.47:0/1492268480 <== mon.2 v2:172.21.15.55:3300/0 8 ==== mgrmap(e 21) v1 ==== 127422+0+0 (secure 0 0 0) 0x7f02bc010c30 con 0x7f02c00fdea0 2022-09-07T21:41:03.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:03.687+0000 7f02b67fc700 1 -- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 msgr2=0x7f02ac09b590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-07T21:41:03.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:03.687+0000 7f02b67fc700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/429234829,v1:172.21.15.47:6801/429234829] conn(0x7f02ac0990e0 0x7f02ac09b590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:03.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:03 smithi047 ceph-mon[90087]: Active manager daemon x restarted 2022-09-07T21:41:03.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:03 smithi047 ceph-mon[90087]: Activating manager daemon x 2022-09-07T21:41:03.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:03 smithi047 ceph-mon[90087]: osdmap e94: 12 total, 12 up, 12 in 2022-09-07T21:41:03.904 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:03.620+0000 7f3feec2b080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-07T21:41:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:03 smithi055 ceph-mon[97033]: Active manager daemon x restarted 2022-09-07T21:41:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:03 smithi055 ceph-mon[97033]: Activating manager daemon x 2022-09-07T21:41:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:03 smithi055 ceph-mon[97033]: osdmap e94: 12 total, 12 up, 12 in 2022-09-07T21:41:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:03 smithi059 ceph-mon[97450]: Active manager daemon x restarted 2022-09-07T21:41:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:03 smithi059 ceph-mon[97450]: Activating manager daemon x 2022-09-07T21:41:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:03 smithi059 ceph-mon[97450]: osdmap e94: 12 total, 12 up, 12 in 2022-09-07T21:41:04.394 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:04.181+0000 7f3f54714700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:41:04.394 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:04.181+0000 7f3f54714700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:41:04.394 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:04.181+0000 7f3f54714700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:41:04.395 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:04.181+0000 7f3f54714700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:41:04.395 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:04.181+0000 7f3f54714700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:41:04.395 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:04.182+0000 7f3f51ecf700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:41:04.395 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:04.182+0000 7f3f51ecf700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:41:04.395 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:04.182+0000 7f3f51ecf700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:41:04.396 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:04.182+0000 7f3f51ecf700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:41:04.396 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:41:04.182+0000 7f3f51ecf700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:41:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.717+0000 7f02b67fc700 1 -- 172.21.15.47:0/1492268480 <== mon.2 v2:172.21.15.55:3300/0 9 ==== mgrmap(e 22) v1 ==== 127532+0+0 (secure 0 0 0) 0x7f02bc004d70 con 0x7f02c00fdea0 2022-09-07T21:41:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.718+0000 7f02b67fc700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f02ac02eff0 0x7f02ac02f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:41:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.718+0000 7f02b67fc700 1 -- 172.21.15.47:0/1492268480 --> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f02c005f120 con 0x7f02ac02eff0 2022-09-07T21:41:04.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.721+0000 7f02c499a700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f02ac02eff0 0x7f02ac02f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:41:04.731 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.729+0000 7f02c499a700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f02ac02eff0 0x7f02ac02f3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f02b000b4f0 tx=0x7f02b00050e0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:41:04.805 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.802+0000 7f02b67fc700 1 -- 172.21.15.47:0/1492268480 <== mgr.25354 v2:172.21.15.47:6800/4004859621 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f02c005f120 con 0x7f02ac02eff0 2022-09-07T21:41:04.805 INFO:teuthology.orchestra.run.smithi047.stdout:Schedule set for path / 2022-09-07T21:41:04.808 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.806+0000 7f02c73ff700 1 -- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f02ac02eff0 msgr2=0x7f02ac02f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:04.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.806+0000 7f02c73ff700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f02ac02eff0 0x7f02ac02f3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f02b000b4f0 tx=0x7f02b00050e0 comp rx=0 tx=0).stop 2022-09-07T21:41:04.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.806+0000 7f02c73ff700 1 -- 172.21.15.47:0/1492268480 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02c00fdea0 msgr2=0x7f02c0072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:04.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.806+0000 7f02c73ff700 1 --2- 172.21.15.47:0/1492268480 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02c00fdea0 0x7f02c0072a90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00e740 tx=0x7f02bc00ea50 comp rx=0 tx=0).stop 2022-09-07T21:41:04.809 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.806+0000 7f02c599c700 1 -- 172.21.15.47:0/1492268480 reap_dead start 2022-09-07T21:41:04.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.806+0000 7f02c73ff700 1 -- 172.21.15.47:0/1492268480 shutdown_connections 2022-09-07T21:41:04.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.806+0000 7f02c73ff700 1 -- 172.21.15.47:0/1492268480 >> 172.21.15.47:0/1492268480 conn(0x7f02c00f9510 msgr2=0x7f02c00fb780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:41:04.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.806+0000 7f02c73ff700 1 -- 172.21.15.47:0/1492268480 shutdown_connections 2022-09-07T21:41:04.810 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:04.806+0000 7f02c73ff700 1 -- 172.21.15.47:0/1492268480 wait complete. 2022-09-07T21:41:04.854 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h' 2022-09-07T21:41:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: mgrmap e21: x(active, starting, since 0.0584606s), standbys: y, z 2022-09-07T21:41:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:41:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:41:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:41:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:41:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-09-07T21:41:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-09-07T21:41:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-09-07T21:41:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-07T21:41:04.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-09-07T21:41:04.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-09-07T21:41:04.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-09-07T21:41:04.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-07T21:41:04.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-09-07T21:41:04.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-09-07T21:41:04.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-09-07T21:41:04.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-09-07T21:41:04.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-09-07T21:41:04.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-09-07T21:41:04.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:41:04.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:41:04.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:41:04.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:41:04.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:41:04.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:41:04.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:41:04.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:41:04.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:41:04.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:41:04.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:41:04.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:41:04.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-09-07T21:41:04.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-09-07T21:41:04.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-09-07T21:41:04.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: Manager daemon x is now available 2022-09-07T21:41:04.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:41:04.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:41:04.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:41:04.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:41:04.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:41:04.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:41:04.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:41:04.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:04 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:41:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: mgrmap e21: x(active, starting, since 0.0584606s), standbys: y, z 2022-09-07T21:41:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:41:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:41:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:41:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:41:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-09-07T21:41:05.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-09-07T21:41:05.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-09-07T21:41:05.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-07T21:41:05.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-09-07T21:41:05.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-09-07T21:41:05.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-09-07T21:41:05.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-07T21:41:05.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-09-07T21:41:05.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-09-07T21:41:05.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-09-07T21:41:05.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-09-07T21:41:05.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-09-07T21:41:05.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-09-07T21:41:05.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:41:05.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:41:05.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:41:05.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:41:05.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:41:05.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:41:05.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:41:05.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:41:05.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:41:05.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:41:05.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:41:05.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:41:05.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-09-07T21:41:05.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-09-07T21:41:05.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-09-07T21:41:05.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: Manager daemon x is now available 2022-09-07T21:41:05.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:41:05.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:41:05.041 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:41:05.041 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:41:05.041 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:41:05.041 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:41:05.041 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:41:05.042 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:04 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:41:05.094 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.091+0000 7f721719d700 1 -- 172.21.15.47:0/262008700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72100fc520 msgr2=0x7f72100fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:05.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.091+0000 7f721719d700 1 --2- 172.21.15.47:0/262008700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72100fc520 0x7f72100fe940 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f720c009810 tx=0x7f720c009b20 comp rx=0 tx=0).stop 2022-09-07T21:41:05.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.091+0000 7f721719d700 1 -- 172.21.15.47:0/262008700 shutdown_connections 2022-09-07T21:41:05.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.091+0000 7f721719d700 1 --2- 172.21.15.47:0/262008700 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7210101840 0x7f7210101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.095 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.091+0000 7f721719d700 1 --2- 172.21.15.47:0/262008700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72100fc520 0x7f72100fe940 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.091+0000 7f721719d700 1 --2- 172.21.15.47:0/262008700 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72100fee80 0x7f7210101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.092+0000 7f721719d700 1 -- 172.21.15.47:0/262008700 >> 172.21.15.47:0/262008700 conn(0x7f72100f8230 msgr2=0x7f72100fa640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:41:05.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.092+0000 7f721719d700 1 -- 172.21.15.47:0/262008700 shutdown_connections 2022-09-07T21:41:05.096 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.092+0000 7f721719d700 1 -- 172.21.15.47:0/262008700 wait complete. 2022-09-07T21:41:05.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.093+0000 7f721719d700 1 Processor -- start 2022-09-07T21:41:05.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.093+0000 7f721719d700 1 -- start start 2022-09-07T21:41:05.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.093+0000 7f721719d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72100fc520 0x7f7210199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:41:05.099 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.093+0000 7f721719d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72100fee80 0x7f721019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:41:05.100 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721719d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7210101840 0x7f72101a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:41:05.100 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721719d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72101a1900 con 0x7f72100fee80 2022-09-07T21:41:05.100 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721719d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72101a1a70 con 0x7f7210101840 2022-09-07T21:41:05.100 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721719d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72101a1c30 con 0x7f72100fc520 2022-09-07T21:41:05.100 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721699c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7210101840 0x7f72101a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:41:05.101 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721699c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7210101840 0x7f72101a12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:52822/0 (socket says 172.21.15.47:52822) 2022-09-07T21:41:05.101 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721699c700 1 -- 172.21.15.47:0/3538558863 learned_addr learned my addr 172.21.15.47:0/3538558863 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:41:05.101 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721619b700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72100fc520 0x7f7210199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:41:05.101 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721599a700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72100fee80 0x7f721019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:41:05.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721699c700 1 -- 172.21.15.47:0/3538558863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72100fc520 msgr2=0x7f7210199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:05.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721699c700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72100fc520 0x7f7210199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721699c700 1 -- 172.21.15.47:0/3538558863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72100fee80 msgr2=0x7f721019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:05.102 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721699c700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72100fee80 0x7f721019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721699c700 1 -- 172.21.15.47:0/3538558863 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72000093b0 con 0x7f7210101840 2022-09-07T21:41:05.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.094+0000 7f721619b700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72100fc520 0x7f7210199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:41:05.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.095+0000 7f721599a700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72100fee80 0x7f721019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-07T21:41:05.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.095+0000 7f721699c700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7210101840 0x7f72101a12b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f720800eba0 tx=0x7f720800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:41:05.103 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.095+0000 7f72077fe700 1 -- 172.21.15.47:0/3538558863 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720800ccc0 con 0x7f7210101840 2022-09-07T21:41:05.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.095+0000 7f72077fe700 1 -- 172.21.15.47:0/3538558863 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f720800ce20 con 0x7f7210101840 2022-09-07T21:41:05.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.095+0000 7f72077fe700 1 -- 172.21.15.47:0/3538558863 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7208010430 con 0x7f7210101840 2022-09-07T21:41:05.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.095+0000 7f721719d700 1 -- 172.21.15.47:0/3538558863 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7208009730 con 0x7f7210101840 2022-09-07T21:41:05.104 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.095+0000 7f721719d700 1 -- 172.21.15.47:0/3538558863 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f720c009470 con 0x7f7210101840 2022-09-07T21:41:05.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.097+0000 7f721719d700 1 -- 172.21.15.47:0/3538558863 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721010f9c0 con 0x7f7210101840 2022-09-07T21:41:05.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.098+0000 7f72077fe700 1 -- 172.21.15.47:0/3538558863 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 22) v1 ==== 127532+0+0 (secure 0 0 0) 0x7f720801e020 con 0x7f7210101840 2022-09-07T21:41:05.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.099+0000 7f72077fe700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f71fc099000 0x7f71fc09b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:41:05.106 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.100+0000 7f72077fe700 1 -- 172.21.15.47:0/3538558863 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9361+0+0 (secure 0 0 0) 0x7f7208014070 con 0x7f7210101840 2022-09-07T21:41:05.107 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.100+0000 7f721619b700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f71fc099000 0x7f71fc09b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:41:05.107 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.101+0000 7f721619b700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f71fc099000 0x7f71fc09b4b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f720c009760 tx=0x7f720c009e70 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:41:05.107 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.103+0000 7f72077fe700 1 -- 172.21.15.47:0/3538558863 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f720808f030 con 0x7f7210101840 2022-09-07T21:41:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: mgrmap e21: x(active, starting, since 0.0584606s), standbys: y, z 2022-09-07T21:41:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-07T21:41:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-07T21:41:05.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-07T21:41:05.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:41:05.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-09-07T21:41:05.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-09-07T21:41:05.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-09-07T21:41:05.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-07T21:41:05.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-09-07T21:41:05.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-09-07T21:41:05.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-09-07T21:41:05.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-07T21:41:05.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-09-07T21:41:05.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-09-07T21:41:05.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-09-07T21:41:05.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-09-07T21:41:05.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-09-07T21:41:05.158 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-09-07T21:41:05.158 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-07T21:41:05.158 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-07T21:41:05.158 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-07T21:41:05.158 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-07T21:41:05.159 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-07T21:41:05.159 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-07T21:41:05.159 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-07T21:41:05.159 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-07T21:41:05.160 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-07T21:41:05.160 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-07T21:41:05.160 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-07T21:41:05.160 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-07T21:41:05.160 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-09-07T21:41:05.161 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-09-07T21:41:05.161 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-09-07T21:41:05.161 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: Manager daemon x is now available 2022-09-07T21:41:05.161 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:41:05.162 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:41:05.162 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:41:05.162 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:41:05.162 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:41:05.163 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:41:05.163 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:41:05.163 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:04 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:41:05.354 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.351+0000 7f721719d700 1 -- 172.21.15.47:0/3538558863 --> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f721005f1c0 con 0x7f71fc099000 2022-09-07T21:41:05.409 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.407+0000 7f72077fe700 1 -- 172.21.15.47:0/3538558863 <== mgr.25354 v2:172.21.15.47:6800/4004859621 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f721005f1c0 con 0x7f71fc099000 2022-09-07T21:41:05.410 INFO:teuthology.orchestra.run.smithi047.stdout:Retention added to path / 2022-09-07T21:41:05.412 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.410+0000 7f721719d700 1 -- 172.21.15.47:0/3538558863 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f71fc099000 msgr2=0x7f71fc09b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:05.413 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.410+0000 7f721719d700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f71fc099000 0x7f71fc09b4b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f720c009760 tx=0x7f720c009e70 comp rx=0 tx=0).stop 2022-09-07T21:41:05.413 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.410+0000 7f721719d700 1 -- 172.21.15.47:0/3538558863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7210101840 msgr2=0x7f72101a12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:05.413 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.410+0000 7f721719d700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7210101840 0x7f72101a12b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f720800eba0 tx=0x7f720800eeb0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.413 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.410+0000 7f721719d700 1 -- 172.21.15.47:0/3538558863 shutdown_connections 2022-09-07T21:41:05.414 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.410+0000 7f721719d700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f71fc099000 0x7f71fc09b4b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.414 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.410+0000 7f721719d700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7210101840 0x7f72101a12b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.414 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.410+0000 7f721719d700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72100fc520 0x7f7210199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.414 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.410+0000 7f721719d700 1 --2- 172.21.15.47:0/3538558863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72100fee80 0x7f721019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.415 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.410+0000 7f721719d700 1 -- 172.21.15.47:0/3538558863 >> 172.21.15.47:0/3538558863 conn(0x7f72100f8230 msgr2=0x7f72100ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:41:05.415 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.410+0000 7f721719d700 1 -- 172.21.15.47:0/3538558863 shutdown_connections 2022-09-07T21:41:05.415 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.410+0000 7f721719d700 1 -- 172.21.15.47:0/3538558863 wait complete. 2022-09-07T21:41:05.442 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2022-09-07T21:41:05.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.703+0000 7f209c9fe700 1 -- 172.21.15.47:0/1632678082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f209810a280 msgr2=0x7f209810c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:05.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.703+0000 7f209c9fe700 1 --2- 172.21.15.47:0/1632678082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f209810a280 0x7f209810c660 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2090009bb0 tx=0x7f2090009ec0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.703+0000 7f209c9fe700 1 -- 172.21.15.47:0/1632678082 shutdown_connections 2022-09-07T21:41:05.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.703+0000 7f209c9fe700 1 --2- 172.21.15.47:0/1632678082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f209810a280 0x7f209810c660 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.703+0000 7f209c9fe700 1 --2- 172.21.15.47:0/1632678082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2098072c90 0x7f2098071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.703+0000 7f209c9fe700 1 --2- 172.21.15.47:0/1632678082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2098071740 0x7f2098071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.703+0000 7f209c9fe700 1 -- 172.21.15.47:0/1632678082 >> 172.21.15.47:0/1632678082 conn(0x7f20980f9330 msgr2=0x7f20980fb740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:41:05.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.703+0000 7f209c9fe700 1 -- 172.21.15.47:0/1632678082 shutdown_connections 2022-09-07T21:41:05.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.703+0000 7f209c9fe700 1 -- 172.21.15.47:0/1632678082 wait complete. 2022-09-07T21:41:05.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.704+0000 7f209c9fe700 1 Processor -- start 2022-09-07T21:41:05.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.705+0000 7f209c9fe700 1 -- start start 2022-09-07T21:41:05.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.705+0000 7f209c9fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2098071740 0x7f2098199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:41:05.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.705+0000 7f209c9fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2098072c90 0x7f209819a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:41:05.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.705+0000 7f209c9fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f209810a280 0x7f20981a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:41:05.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.705+0000 7f209c9fe700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20981a1950 con 0x7f2098071740 2022-09-07T21:41:05.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.705+0000 7f209c9fe700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20981a1ac0 con 0x7f209810a280 2022-09-07T21:41:05.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.705+0000 7f209c9fe700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20981a1c80 con 0x7f2098072c90 2022-09-07T21:41:05.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.705+0000 7f209659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2098072c90 0x7f209819a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:41:05.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.705+0000 7f209759e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f209810a280 0x7f20981a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:41:05.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.705+0000 7f209659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2098072c90 0x7f209819a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:35058/0 (socket says 172.21.15.47:35058) 2022-09-07T21:41:05.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.705+0000 7f209659c700 1 -- 172.21.15.47:0/4130819907 learned_addr learned my addr 172.21.15.47:0/4130819907 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:41:05.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.706+0000 7f209759e700 1 -- 172.21.15.47:0/4130819907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2098072c90 msgr2=0x7f209819a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:05.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.706+0000 7f209759e700 1 --2- 172.21.15.47:0/4130819907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2098072c90 0x7f209819a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.706+0000 7f209759e700 1 -- 172.21.15.47:0/4130819907 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2098071740 msgr2=0x7f2098199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:05.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.706+0000 7f209759e700 1 --2- 172.21.15.47:0/4130819907 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2098071740 0x7f2098199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.706+0000 7f209759e700 1 -- 172.21.15.47:0/4130819907 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20900097e0 con 0x7f209810a280 2022-09-07T21:41:05.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.706+0000 7f209759e700 1 --2- 172.21.15.47:0/4130819907 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f209810a280 0x7f20981a1380 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2090009ad0 tx=0x7f209000b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:41:05.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.706+0000 7f207ffff700 1 -- 172.21.15.47:0/4130819907 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209000f870 con 0x7f209810a280 2022-09-07T21:41:05.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.707+0000 7f207ffff700 1 -- 172.21.15.47:0/4130819907 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f209000f9d0 con 0x7f209810a280 2022-09-07T21:41:05.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.707+0000 7f209c9fe700 1 -- 172.21.15.47:0/4130819907 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20981a1f00 con 0x7f209810a280 2022-09-07T21:41:05.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.707+0000 7f207ffff700 1 -- 172.21.15.47:0/4130819907 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900176f0 con 0x7f209810a280 2022-09-07T21:41:05.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.707+0000 7f209c9fe700 1 -- 172.21.15.47:0/4130819907 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20981a2320 con 0x7f209810a280 2022-09-07T21:41:05.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.708+0000 7f209c9fe700 1 -- 172.21.15.47:0/4130819907 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209810f9e0 con 0x7f209810a280 2022-09-07T21:41:05.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.711+0000 7f207ffff700 1 -- 172.21.15.47:0/4130819907 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 22) v1 ==== 127532+0+0 (secure 0 0 0) 0x7f2090029070 con 0x7f209810a280 2022-09-07T21:41:05.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.713+0000 7f207ffff700 1 --2- 172.21.15.47:0/4130819907 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f2080099000 0x7f208009b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:41:05.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.713+0000 7f2096d9d700 1 --2- 172.21.15.47:0/4130819907 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f2080099000 0x7f208009b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:41:05.717 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.714+0000 7f2096d9d700 1 --2- 172.21.15.47:0/4130819907 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f2080099000 0x7f208009b4b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2088005fd0 tx=0x7f20880094d0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:41:05.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.716+0000 7f207ffff700 1 -- 172.21.15.47:0/4130819907 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9361+0+0 (secure 0 0 0) 0x7f2090093d70 con 0x7f209810a280 2022-09-07T21:41:05.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.721+0000 7f207ffff700 1 -- 172.21.15.47:0/4130819907 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f209008f9b0 con 0x7f209810a280 2022-09-07T21:41:05.960 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.958+0000 7f209c9fe700 1 -- 172.21.15.47:0/4130819907 --> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f209805f120 con 0x7f2080099000 2022-09-07T21:41:05.970 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.968+0000 7f207ffff700 1 -- 172.21.15.47:0/4130819907 <== mgr.25354 v2:172.21.15.47:6800/4004859621 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+283 (secure 0 0 0) 0x7f209805f120 con 0x7f2080099000 2022-09-07T21:41:05.970 INFO:teuthology.orchestra.run.smithi047.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2022-09-07T00:00:00", "created": "2022-09-07T21:41:04", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2022-09-07T21:41:05.973 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.970+0000 7f209c9fe700 1 -- 172.21.15.47:0/4130819907 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f2080099000 msgr2=0x7f208009b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:05.973 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.970+0000 7f209c9fe700 1 --2- 172.21.15.47:0/4130819907 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f2080099000 0x7f208009b4b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2088005fd0 tx=0x7f20880094d0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.974 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.970+0000 7f209c9fe700 1 -- 172.21.15.47:0/4130819907 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f209810a280 msgr2=0x7f20981a1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:05.974 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.970+0000 7f209c9fe700 1 --2- 172.21.15.47:0/4130819907 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f209810a280 0x7f20981a1380 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2090009ad0 tx=0x7f209000b890 comp rx=0 tx=0).stop 2022-09-07T21:41:05.974 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.970+0000 7f209c9fe700 1 -- 172.21.15.47:0/4130819907 shutdown_connections 2022-09-07T21:41:05.974 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.970+0000 7f209c9fe700 1 --2- 172.21.15.47:0/4130819907 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f2080099000 0x7f208009b4b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.975 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.970+0000 7f209c9fe700 1 --2- 172.21.15.47:0/4130819907 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f209810a280 0x7f20981a1380 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.975 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.970+0000 7f209c9fe700 1 --2- 172.21.15.47:0/4130819907 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2098071740 0x7f2098199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.975 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.970+0000 7f209c9fe700 1 --2- 172.21.15.47:0/4130819907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2098072c90 0x7f209819a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:05.975 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.970+0000 7f209c9fe700 1 -- 172.21.15.47:0/4130819907 >> 172.21.15.47:0/4130819907 conn(0x7f20980f9330 msgr2=0x7f20980fb710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:41:05.975 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.970+0000 7f209c9fe700 1 -- 172.21.15.47:0/4130819907 shutdown_connections 2022-09-07T21:41:05.976 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:05.970+0000 7f209c9fe700 1 -- 172.21.15.47:0/4130819907 wait complete. 2022-09-07T21:41:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:05 smithi055 ceph-mon[97033]: mgrmap e22: x(active, since 1.08827s), standbys: y, z 2022-09-07T21:41:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:05 smithi055 ceph-mon[97033]: from='client.25346 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:41:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:05 smithi055 ceph-mon[97033]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:41:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:05 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:05 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:05.996 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:05 smithi047 ceph-mon[90087]: mgrmap e22: x(active, since 1.08827s), standbys: y, z 2022-09-07T21:41:05.996 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:05 smithi047 ceph-mon[90087]: from='client.25346 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:41:05.997 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:05 smithi047 ceph-mon[90087]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:41:05.997 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:05 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:05.997 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:05 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:05.999 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2022-09-07T21:41:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:05 smithi059 ceph-mon[97450]: mgrmap e22: x(active, since 1.08827s), standbys: y, z 2022-09-07T21:41:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:05 smithi059 ceph-mon[97450]: from='client.25346 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:41:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:05 smithi059 ceph-mon[97450]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:41:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:05 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:06.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:05 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:06.235 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.232+0000 7f62bddff700 1 -- 172.21.15.47:0/4226781766 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62b81011a0 msgr2=0x7f62b81015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:06.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.232+0000 7f62bddff700 1 --2- 172.21.15.47:0/4226781766 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62b81011a0 0x7f62b81015f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f62ac009750 tx=0x7f62ac009a60 comp rx=0 tx=0).stop 2022-09-07T21:41:06.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.233+0000 7f62bddff700 1 -- 172.21.15.47:0/4226781766 shutdown_connections 2022-09-07T21:41:06.236 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.233+0000 7f62bddff700 1 --2- 172.21.15.47:0/4226781766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62b8101b30 0x7f62b81083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:06.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.233+0000 7f62bddff700 1 --2- 172.21.15.47:0/4226781766 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62b80fff20 0x7f62b8100330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:06.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.233+0000 7f62bddff700 1 --2- 172.21.15.47:0/4226781766 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62b81011a0 0x7f62b81015f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:06.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.233+0000 7f62bddff700 1 -- 172.21.15.47:0/4226781766 >> 172.21.15.47:0/4226781766 conn(0x7f62b80fb6b0 msgr2=0x7f62b80fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:41:06.237 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.233+0000 7f62bddff700 1 -- 172.21.15.47:0/4226781766 shutdown_connections 2022-09-07T21:41:06.238 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.233+0000 7f62bddff700 1 -- 172.21.15.47:0/4226781766 wait complete. 2022-09-07T21:41:06.240 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.234+0000 7f62bddff700 1 Processor -- start 2022-09-07T21:41:06.240 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.234+0000 7f62bddff700 1 -- start start 2022-09-07T21:41:06.240 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.234+0000 7f62bddff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62b80fff20 0x7f62b8195ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:41:06.241 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.234+0000 7f62bddff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62b81011a0 0x7f62b8196000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:41:06.241 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.234+0000 7f62bddff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62b8101b30 0x7f62b819d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:41:06.241 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.235+0000 7f62bddff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b819d640 con 0x7f62b80fff20 2022-09-07T21:41:06.241 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.235+0000 7f62bddff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b819d7b0 con 0x7f62b8101b30 2022-09-07T21:41:06.242 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.235+0000 7f62bddff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b819d970 con 0x7f62b81011a0 2022-09-07T21:41:06.242 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.235+0000 7f62b77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62b80fff20 0x7f62b8195ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:41:06.242 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.235+0000 7f62b7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62b8101b30 0x7f62b819d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:41:06.242 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.235+0000 7f62b7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62b8101b30 0x7f62b819d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:52840/0 (socket says 172.21.15.47:52840) 2022-09-07T21:41:06.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.235+0000 7f62b7fff700 1 -- 172.21.15.47:0/2317688216 learned_addr learned my addr 172.21.15.47:0/2317688216 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:41:06.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.235+0000 7f62b6ffd700 1 --2- 172.21.15.47:0/2317688216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62b81011a0 0x7f62b8196000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:41:06.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.235+0000 7f62b7fff700 1 -- 172.21.15.47:0/2317688216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62b81011a0 msgr2=0x7f62b8196000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:06.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.235+0000 7f62b7fff700 1 --2- 172.21.15.47:0/2317688216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62b81011a0 0x7f62b8196000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:06.244 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.235+0000 7f62b7fff700 1 -- 172.21.15.47:0/2317688216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62b80fff20 msgr2=0x7f62b8195ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:06.244 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.235+0000 7f62b7fff700 1 --2- 172.21.15.47:0/2317688216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62b80fff20 0x7f62b8195ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:06.244 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.235+0000 7f62b7fff700 1 -- 172.21.15.47:0/2317688216 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac0093b0 con 0x7f62b8101b30 2022-09-07T21:41:06.244 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.236+0000 7f62b7fff700 1 --2- 172.21.15.47:0/2317688216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62b8101b30 0x7f62b819d070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f62b000eab0 tx=0x7f62b000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:41:06.244 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.236+0000 7f62b4ff9700 1 -- 172.21.15.47:0/2317688216 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0009880 con 0x7f62b8101b30 2022-09-07T21:41:06.245 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.236+0000 7f62b4ff9700 1 -- 172.21.15.47:0/2317688216 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f62b000ce70 con 0x7f62b8101b30 2022-09-07T21:41:06.245 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.236+0000 7f62bddff700 1 -- 172.21.15.47:0/2317688216 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b819dc50 con 0x7f62b8101b30 2022-09-07T21:41:06.245 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.236+0000 7f62b4ff9700 1 -- 172.21.15.47:0/2317688216 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0005840 con 0x7f62b8101b30 2022-09-07T21:41:06.245 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.236+0000 7f62bddff700 1 -- 172.21.15.47:0/2317688216 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b819e160 con 0x7f62b8101b30 2022-09-07T21:41:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.237+0000 7f62bddff700 1 -- 172.21.15.47:0/2317688216 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b810b6d0 con 0x7f62b8101b30 2022-09-07T21:41:06.247 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.239+0000 7f62b4ff9700 1 -- 172.21.15.47:0/2317688216 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 22) v1 ==== 127532+0+0 (secure 0 0 0) 0x7f62b001e020 con 0x7f62b8101b30 2022-09-07T21:41:06.247 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.241+0000 7f62b4ff9700 1 --2- 172.21.15.47:0/2317688216 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f62a0098f50 0x7f62a009b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:41:06.247 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.241+0000 7f62b4ff9700 1 -- 172.21.15.47:0/2317688216 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9361+0+0 (secure 0 0 0) 0x7f62b0014070 con 0x7f62b8101b30 2022-09-07T21:41:06.247 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.241+0000 7f62b77fe700 1 --2- 172.21.15.47:0/2317688216 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f62a0098f50 0x7f62a009b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:41:06.248 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.242+0000 7f62b77fe700 1 --2- 172.21.15.47:0/2317688216 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f62a0098f50 0x7f62a009b400 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f62b8101000 tx=0x7f62a8005ea0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:41:06.248 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.243+0000 7f62b4ff9700 1 -- 172.21.15.47:0/2317688216 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f62b008f410 con 0x7f62b8101b30 2022-09-07T21:41:06.478 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.476+0000 7f62bddff700 1 -- 172.21.15.47:0/2317688216 --> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f62b805fba0 con 0x7f62a0098f50 2022-09-07T21:41:06.486 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.484+0000 7f62b4ff9700 1 -- 172.21.15.47:0/2317688216 <== mgr.25354 v2:172.21.15.47:6800/4004859621 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f62b805fba0 con 0x7f62a0098f50 2022-09-07T21:41:06.487 INFO:teuthology.orchestra.run.smithi047.stdout:/ 1M 6M3h 2022-09-07T21:41:06.490 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.487+0000 7f62bddff700 1 -- 172.21.15.47:0/2317688216 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f62a0098f50 msgr2=0x7f62a009b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:06.490 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.487+0000 7f62bddff700 1 --2- 172.21.15.47:0/2317688216 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f62a0098f50 0x7f62a009b400 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f62b8101000 tx=0x7f62a8005ea0 comp rx=0 tx=0).stop 2022-09-07T21:41:06.490 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.487+0000 7f62bddff700 1 -- 172.21.15.47:0/2317688216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62b8101b30 msgr2=0x7f62b819d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:41:06.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.487+0000 7f62bddff700 1 --2- 172.21.15.47:0/2317688216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62b8101b30 0x7f62b819d070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f62b000eab0 tx=0x7f62b000edc0 comp rx=0 tx=0).stop 2022-09-07T21:41:06.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.487+0000 7f62bddff700 1 -- 172.21.15.47:0/2317688216 shutdown_connections 2022-09-07T21:41:06.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.487+0000 7f62bddff700 1 --2- 172.21.15.47:0/2317688216 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f62a0098f50 0x7f62a009b400 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:06.491 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.487+0000 7f62bddff700 1 --2- 172.21.15.47:0/2317688216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62b8101b30 0x7f62b819d070 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:06.492 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.487+0000 7f62bddff700 1 --2- 172.21.15.47:0/2317688216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62b80fff20 0x7f62b8195ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:06.492 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.487+0000 7f62bddff700 1 --2- 172.21.15.47:0/2317688216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62b81011a0 0x7f62b8196000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:41:06.492 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.487+0000 7f62bddff700 1 -- 172.21.15.47:0/2317688216 >> 172.21.15.47:0/2317688216 conn(0x7f62b80fb6b0 msgr2=0x7f62b81046d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:41:06.492 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.487+0000 7f62bddff700 1 -- 172.21.15.47:0/2317688216 shutdown_connections 2022-09-07T21:41:06.493 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:41:06.487+0000 7f62bddff700 1 -- 172.21.15.47:0/2317688216 wait complete. 2022-09-07T21:41:06.521 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2022-09-07T21:41:06.625 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2022-09-07T21:41:06.637 INFO:teuthology.run_tasks:Running task workunit... 2022-09-07T21:41:06.651 INFO:tasks.workunit:Pulling workunits from ref 75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d 2022-09-07T21:41:06.653 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-09-07T21:41:06.653 INFO:tasks.workunit:timeout=3h 2022-09-07T21:41:06.654 INFO:tasks.workunit:cleanup=True 2022-09-07T21:41:06.654 DEBUG:teuthology.orchestra.run.smithi047:> stat -- /home/ubuntu/cephtest/mnt.0 2022-09-07T21:41:06.716 INFO:teuthology.orchestra.run.smithi047.stdout: File: /home/ubuntu/cephtest/mnt.0 2022-09-07T21:41:06.717 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2022-09-07T21:41:06.717 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 78h/120d Inode: 1099511628283 Links: 2 2022-09-07T21:41:06.717 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-07T21:41:06.717 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:cephfs_t:s0 2022-09-07T21:41:06.717 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2022-09-07 21:39:08.456589267 +0000 2022-09-07T21:41:06.718 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2022-09-07 21:39:08.456589267 +0000 2022-09-07T21:41:06.718 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2022-09-07 21:40:51.809754615 +0000 2022-09-07T21:41:06.718 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2022-09-07 21:39:08.456589267 +0000 2022-09-07T21:41:06.719 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2022-09-07T21:41:06.719 DEBUG:teuthology.orchestra.run.smithi047:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2022-09-07T21:41:06.800 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:06 smithi047 ceph-mon[90087]: [07/Sep/2022:21:41:05] ENGINE Bus STARTING 2022-09-07T21:41:06.801 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:06 smithi047 ceph-mon[90087]: [07/Sep/2022:21:41:05] ENGINE Serving on https://172.21.15.47:7150 2022-09-07T21:41:06.801 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:06 smithi047 ceph-mon[90087]: mgrmap e23: x(active, since 2s), standbys: y, z 2022-09-07T21:41:06.823 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d 2022-09-07T21:41:06.885 INFO:tasks.workunit.client.0.smithi047.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-09-07T21:41:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:06 smithi055 ceph-mon[97033]: [07/Sep/2022:21:41:05] ENGINE Bus STARTING 2022-09-07T21:41:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:06 smithi055 ceph-mon[97033]: [07/Sep/2022:21:41:05] ENGINE Serving on https://172.21.15.47:7150 2022-09-07T21:41:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:06 smithi055 ceph-mon[97033]: mgrmap e23: x(active, since 2s), standbys: y, z 2022-09-07T21:41:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:06 smithi059 ceph-mon[97450]: [07/Sep/2022:21:41:05] ENGINE Bus STARTING 2022-09-07T21:41:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:06 smithi059 ceph-mon[97450]: [07/Sep/2022:21:41:05] ENGINE Serving on https://172.21.15.47:7150 2022-09-07T21:41:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:06 smithi059 ceph-mon[97450]: mgrmap e23: x(active, since 2s), standbys: y, z 2022-09-07T21:41:07.723 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:07 smithi059 ceph-mon[97450]: from='client.25394 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:41:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:07 smithi055 ceph-mon[97033]: from='client.25394 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:41:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:07 smithi055 ceph-mon[97033]: [07/Sep/2022:21:41:05] ENGINE Serving on https://:::8765 2022-09-07T21:41:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:07 smithi055 ceph-mon[97033]: [07/Sep/2022:21:41:05] ENGINE Bus STARTED 2022-09-07T21:41:07.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:07 smithi055 ceph-mon[97033]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:41:07.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:07 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:07.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:07 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:07.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:07 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:07.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:07 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:07 smithi059 ceph-mon[97450]: [07/Sep/2022:21:41:05] ENGINE Serving on https://:::8765 2022-09-07T21:41:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:07 smithi059 ceph-mon[97450]: [07/Sep/2022:21:41:05] ENGINE Bus STARTED 2022-09-07T21:41:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:07 smithi059 ceph-mon[97450]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:41:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:07 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:07 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:07 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:07 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:07 smithi047 ceph-mon[90087]: from='client.25394 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:41:08.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:07 smithi047 ceph-mon[90087]: [07/Sep/2022:21:41:05] ENGINE Serving on https://:::8765 2022-09-07T21:41:08.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:07 smithi047 ceph-mon[90087]: [07/Sep/2022:21:41:05] ENGINE Bus STARTED 2022-09-07T21:41:08.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:07 smithi047 ceph-mon[90087]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:41:08.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:07 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:07 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:07 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:07 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.771 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:41:08.771 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.772 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.772 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.772 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.772 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.773 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.773 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.773 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.773 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.774 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.774 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.774 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.774 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.775 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.775 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.775 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.775 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.776 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.776 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.776 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.955 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:41:08.955 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.956 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.956 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.956 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.957 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.957 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.957 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.958 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.958 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.958 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.958 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.959 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:08.959 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.959 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.959 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.960 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.960 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.960 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.960 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:08.961 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:41:09.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:09.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:09.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:09.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:09.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.158 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.158 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:09.158 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:08 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:10.133 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:09 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T21:41:10.133 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:09 smithi047 ceph-mon[90087]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T21:41:10.134 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:09 smithi047 ceph-mon[90087]: Updating smithi055:/etc/ceph/ceph.conf 2022-09-07T21:41:10.134 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:09 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T21:41:10.134 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:09 smithi047 ceph-mon[90087]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T21:41:10.134 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:09 smithi047 ceph-mon[90087]: Updating smithi059:/etc/ceph/ceph.conf 2022-09-07T21:41:10.135 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:09 smithi047 ceph-mon[90087]: Updating smithi055:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:41:10.135 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:09 smithi047 ceph-mon[90087]: Updating smithi059:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:41:10.135 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:09 smithi047 ceph-mon[90087]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:41:10.135 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:09 smithi047 ceph-mon[90087]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:41:10.135 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:09 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:10.136 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:09 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:10.136 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:09 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:10.136 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:09 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:09 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T21:41:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:09 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T21:41:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:09 smithi055 ceph-mon[97033]: Updating smithi055:/etc/ceph/ceph.conf 2022-09-07T21:41:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:09 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T21:41:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:09 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T21:41:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:09 smithi055 ceph-mon[97033]: Updating smithi059:/etc/ceph/ceph.conf 2022-09-07T21:41:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:09 smithi055 ceph-mon[97033]: Updating smithi055:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:41:10.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:09 smithi055 ceph-mon[97033]: Updating smithi059:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:41:10.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:09 smithi055 ceph-mon[97033]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:41:10.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:09 smithi055 ceph-mon[97033]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:41:10.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:09 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:10.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:09 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:10.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:09 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:10.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:09 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:09 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T21:41:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:09 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T21:41:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:09 smithi059 ceph-mon[97450]: Updating smithi055:/etc/ceph/ceph.conf 2022-09-07T21:41:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:09 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T21:41:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:09 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T21:41:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:09 smithi059 ceph-mon[97450]: Updating smithi059:/etc/ceph/ceph.conf 2022-09-07T21:41:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:09 smithi059 ceph-mon[97450]: Updating smithi055:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:41:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:09 smithi059 ceph-mon[97450]: Updating smithi059:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:41:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:09 smithi059 ceph-mon[97450]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:41:10.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:09 smithi059 ceph-mon[97450]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:41:10.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:09 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:10.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:09 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:10.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:09 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:10.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:09 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:11.207 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:10 smithi047 ceph-mon[90087]: Updating smithi055:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:41:11.208 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:10 smithi047 ceph-mon[90087]: Updating smithi059:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:41:11.208 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:10 smithi047 ceph-mon[90087]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:41:11.208 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:10 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:11.209 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:10 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:10 smithi055 ceph-mon[97033]: Updating smithi055:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:41:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:10 smithi055 ceph-mon[97033]: Updating smithi059:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:41:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:10 smithi055 ceph-mon[97033]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:41:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:10 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:10 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:10 smithi059 ceph-mon[97450]: Updating smithi055:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:41:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:10 smithi059 ceph-mon[97450]: Updating smithi059:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:41:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:10 smithi059 ceph-mon[97450]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-07T21:41:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:10 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:10 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:12.940 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:12 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:12.940 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:12 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:12.940 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:12 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.941 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:12 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.941 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:12 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.941 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:12 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.941 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:12 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.942 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:12 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.942 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:12 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.942 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:12 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.942 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:12 smithi047 ceph-mon[90087]: Adjusting osd_memory_target on smithi047 to 223.0M 2022-09-07T21:41:12.942 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:12 smithi047 ceph-mon[90087]: Unable to set osd_memory_target on smithi047 to 233835929: error parsing value: Value '233835929' is below minimum 939524096 2022-09-07T21:41:12.943 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:12 smithi047 ceph-mon[90087]: Updating smithi047:/etc/ceph/ceph.conf 2022-09-07T21:41:12.943 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:12 smithi047 ceph-mon[90087]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 133 KiB/s wr, 51 op/s 2022-09-07T21:41:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:12 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:12 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:12 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:12 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:12 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:12 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:12.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:12 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi047 to 223.0M 2022-09-07T21:41:12.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:12 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi047 to 233835929: error parsing value: Value '233835929' is below minimum 939524096 2022-09-07T21:41:12.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:12 smithi055 ceph-mon[97033]: Updating smithi047:/etc/ceph/ceph.conf 2022-09-07T21:41:12.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:12 smithi055 ceph-mon[97033]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 133 KiB/s wr, 51 op/s 2022-09-07T21:41:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:12 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:12 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:12 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:13.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:12 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:13.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:13.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:12 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:13.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:13.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:12 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:41:13.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:12 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi047 to 223.0M 2022-09-07T21:41:13.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:12 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi047 to 233835929: error parsing value: Value '233835929' is below minimum 939524096 2022-09-07T21:41:13.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:12 smithi059 ceph-mon[97450]: Updating smithi047:/etc/ceph/ceph.conf 2022-09-07T21:41:13.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:12 smithi059 ceph-mon[97450]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 133 KiB/s wr, 51 op/s 2022-09-07T21:41:13.861 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:13 smithi047 ceph-mon[90087]: Updating smithi047:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:41:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:13 smithi055 ceph-mon[97033]: Updating smithi047:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:41:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:13 smithi059 ceph-mon[97450]: Updating smithi047:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-07T21:41:15.123 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:14 smithi047 ceph-mon[90087]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:41:15.123 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:14 smithi047 ceph-mon[90087]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 103 KiB/s wr, 40 op/s 2022-09-07T21:41:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:14 smithi055 ceph-mon[97033]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:41:15.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:14 smithi055 ceph-mon[97033]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 103 KiB/s wr, 40 op/s 2022-09-07T21:41:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:14 smithi059 ceph-mon[97450]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2022-09-07T21:41:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:14 smithi059 ceph-mon[97450]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 103 KiB/s wr, 40 op/s 2022-09-07T21:41:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:15 smithi059 ceph-mon[97450]: Updating smithi047:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:41:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:15 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:15 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:15 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:41:16.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:41:16.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:41:16.140 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:15 smithi047 ceph-mon[90087]: Updating smithi047:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:41:16.140 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:15 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:16.141 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:15 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:16.141 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:15 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:16.141 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:15 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:41:16.141 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:15 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:41:16.142 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:15 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:41:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:15 smithi055 ceph-mon[97033]: Updating smithi047:/var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-07T21:41:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:15 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:16.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:15 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:16.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:15 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:16.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:41:16.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:41:16.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:41:17.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:16 smithi047 ceph-mon[90087]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 84 KiB/s wr, 35 op/s 2022-09-07T21:41:17.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:16 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:17.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:16 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:17.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:16 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:16 smithi055 ceph-mon[97033]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 84 KiB/s wr, 35 op/s 2022-09-07T21:41:17.251 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:16 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:17.251 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:16 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:17.251 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:16 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:16 smithi059 ceph-mon[97450]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 84 KiB/s wr, 35 op/s 2022-09-07T21:41:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:16 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:16 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:17.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:16 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:41:19.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:18 smithi047 ceph-mon[90087]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2022-09-07T21:41:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:18 smithi055 ceph-mon[97033]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2022-09-07T21:41:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:18 smithi059 ceph-mon[97450]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2022-09-07T21:41:21.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:20 smithi047 ceph-mon[90087]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2022-09-07T21:41:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:20 smithi055 ceph-mon[97033]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2022-09-07T21:41:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:20 smithi059 ceph-mon[97450]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2022-09-07T21:41:23.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:22 smithi047 ceph-mon[90087]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 35 op/s 2022-09-07T21:41:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:22 smithi055 ceph-mon[97033]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 35 op/s 2022-09-07T21:41:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:22 smithi059 ceph-mon[97450]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 35 op/s 2022-09-07T21:41:25.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:24 smithi047 ceph-mon[90087]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:24 smithi055 ceph-mon[97033]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:24 smithi059 ceph-mon[97450]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:27.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:26 smithi047 ceph-mon[90087]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:41:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:26 smithi055 ceph-mon[97033]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:41:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:26 smithi059 ceph-mon[97450]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:41:29.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:28 smithi047 ceph-mon[90087]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:28 smithi055 ceph-mon[97033]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:28 smithi059 ceph-mon[97450]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:31.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:30 smithi047 ceph-mon[90087]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:30 smithi055 ceph-mon[97033]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:30 smithi059 ceph-mon[97450]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:33.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:32 smithi047 ceph-mon[90087]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:41:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:32 smithi055 ceph-mon[97033]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:41:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:32 smithi059 ceph-mon[97450]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:41:35.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:34 smithi047 ceph-mon[90087]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:34 smithi055 ceph-mon[97033]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:34 smithi059 ceph-mon[97450]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:37.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:36 smithi047 ceph-mon[90087]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:41:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:36 smithi055 ceph-mon[97033]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:41:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:36 smithi059 ceph-mon[97450]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:41:39.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:38 smithi047 ceph-mon[90087]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:38 smithi055 ceph-mon[97033]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:38 smithi059 ceph-mon[97450]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:41.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:40 smithi047 ceph-mon[90087]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:40 smithi055 ceph-mon[97033]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:40 smithi059 ceph-mon[97450]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:43.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:42 smithi047 ceph-mon[90087]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:41:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:42 smithi055 ceph-mon[97033]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:41:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:42 smithi059 ceph-mon[97450]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:41:45.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:44 smithi047 ceph-mon[90087]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:44 smithi055 ceph-mon[97033]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:44 smithi059 ceph-mon[97450]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:41:47.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:46 smithi047 ceph-mon[90087]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2022-09-07T21:41:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:46 smithi055 ceph-mon[97033]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2022-09-07T21:41:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:46 smithi059 ceph-mon[97450]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2022-09-07T21:41:47.404 INFO:tasks.workunit.client.0.smithi047.stderr:Note: switching to '75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d'. 2022-09-07T21:41:47.404 INFO:tasks.workunit.client.0.smithi047.stderr: 2022-09-07T21:41:47.405 INFO:tasks.workunit.client.0.smithi047.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-09-07T21:41:47.405 INFO:tasks.workunit.client.0.smithi047.stderr:changes and commit them, and you can discard any commits you make in this 2022-09-07T21:41:47.405 INFO:tasks.workunit.client.0.smithi047.stderr:state without impacting any branches by switching back to a branch. 2022-09-07T21:41:47.405 INFO:tasks.workunit.client.0.smithi047.stderr: 2022-09-07T21:41:47.406 INFO:tasks.workunit.client.0.smithi047.stderr:If you want to create a new branch to retain commits you create, you may 2022-09-07T21:41:47.406 INFO:tasks.workunit.client.0.smithi047.stderr:do so (now or later) by using -c with the switch command. Example: 2022-09-07T21:41:47.406 INFO:tasks.workunit.client.0.smithi047.stderr: 2022-09-07T21:41:47.406 INFO:tasks.workunit.client.0.smithi047.stderr: git switch -c 2022-09-07T21:41:47.407 INFO:tasks.workunit.client.0.smithi047.stderr: 2022-09-07T21:41:47.407 INFO:tasks.workunit.client.0.smithi047.stderr:Or undo this operation with: 2022-09-07T21:41:47.407 INFO:tasks.workunit.client.0.smithi047.stderr: 2022-09-07T21:41:47.407 INFO:tasks.workunit.client.0.smithi047.stderr: git switch - 2022-09-07T21:41:47.407 INFO:tasks.workunit.client.0.smithi047.stderr: 2022-09-07T21:41:47.408 INFO:tasks.workunit.client.0.smithi047.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-09-07T21:41:47.408 INFO:tasks.workunit.client.0.smithi047.stderr: 2022-09-07T21:41:47.408 INFO:tasks.workunit.client.0.smithi047.stderr:HEAD is now at 75ae46d5f37 fix 2022-09-07T21:41:47.449 DEBUG:teuthology.orchestra.run.smithi047:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-09-07T21:41:47.513 INFO:tasks.workunit.client.0.smithi047.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-09-07T21:41:47.520 INFO:tasks.workunit.client.0.smithi047.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-09-07T21:41:47.520 INFO:tasks.workunit.client.0.smithi047.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-09-07T21:41:48.091 INFO:tasks.workunit.client.0.smithi047.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-09-07T21:41:48.225 INFO:tasks.workunit.client.0.smithi047.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-09-07T21:41:48.306 INFO:tasks.workunit.client.0.smithi047.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-09-07T21:41:48.312 INFO:tasks.workunit.client.0.smithi047.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-09-07T21:41:48.312 INFO:tasks.workunit.client.0.smithi047.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-09-07T21:41:48.392 INFO:tasks.workunit.client.0.smithi047.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-09-07T21:41:48.404 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-09-07T21:41:48.404 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2022-09-07T21:41:48.463 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2022-09-07T21:41:48.465 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2022-09-07T21:41:48.466 DEBUG:teuthology.orchestra.run.smithi047:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2022-09-07T21:41:48.557 INFO:tasks.workunit.client.0.smithi047.stderr:Cloning into 'ffsb'... 2022-09-07T21:41:49.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:48 smithi047 ceph-mon[90087]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2022-09-07T21:41:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:48 smithi055 ceph-mon[97033]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2022-09-07T21:41:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:48 smithi059 ceph-mon[97450]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2022-09-07T21:41:49.294 INFO:tasks.workunit.client.0.smithi047.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2022-09-07T21:41:50.309 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether build environment is sane... yes 2022-09-07T21:41:50.335 INFO:tasks.workunit.client.0.smithi047.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2022-09-07T21:41:50.335 INFO:tasks.workunit.client.0.smithi047.stdout:checking for gawk... gawk 2022-09-07T21:41:50.380 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether make sets $(MAKE)... yes 2022-09-07T21:41:50.481 INFO:tasks.workunit.client.0.smithi047.stdout:checking build system type... x86_64-unknown-linux-gnu 2022-09-07T21:41:50.481 INFO:tasks.workunit.client.0.smithi047.stdout:checking host system type... x86_64-unknown-linux-gnu 2022-09-07T21:41:50.499 INFO:tasks.workunit.client.0.smithi047.stdout:checking for style of include used by make... GNU 2022-09-07T21:41:50.504 INFO:tasks.workunit.client.0.smithi047.stdout:checking for gcc... gcc 2022-09-07T21:41:50.632 INFO:tasks.workunit.client.0.smithi047.stdout:checking for C compiler default output file name... a.out 2022-09-07T21:41:50.638 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether the C compiler works... yes 2022-09-07T21:41:50.643 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether we are cross compiling... no 2022-09-07T21:41:50.721 INFO:tasks.workunit.client.0.smithi047.stdout:checking for suffix of executables... 2022-09-07T21:41:50.794 INFO:tasks.workunit.client.0.smithi047.stdout:checking for suffix of object files... o 2022-09-07T21:41:50.865 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether we are using the GNU C compiler... yes 2022-09-07T21:41:50.935 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether gcc accepts -g... yes 2022-09-07T21:41:51.026 INFO:tasks.workunit.client.0.smithi047.stdout:checking for gcc option to accept ISO C89... none needed 2022-09-07T21:41:51.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:50 smithi047 ceph-mon[90087]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2022-09-07T21:41:51.187 INFO:tasks.workunit.client.0.smithi047.stdout:checking dependency style of gcc... gcc3 2022-09-07T21:41:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:50 smithi055 ceph-mon[97033]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2022-09-07T21:41:51.276 INFO:tasks.workunit.client.0.smithi047.stdout:checking how to run the C preprocessor... gcc -E 2022-09-07T21:41:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:50 smithi059 ceph-mon[97450]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2022-09-07T21:41:51.368 INFO:tasks.workunit.client.0.smithi047.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2022-09-07T21:41:51.372 INFO:tasks.workunit.client.0.smithi047.stdout:checking for egrep... /usr/bin/grep -E 2022-09-07T21:41:51.397 INFO:tasks.workunit.client.0.smithi047.stdout:checking for AIX... no 2022-09-07T21:41:51.416 INFO:tasks.workunit.client.0.smithi047.stdout:checking for gcc... (cached) gcc 2022-09-07T21:41:51.445 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether we are using the GNU C compiler... (cached) yes 2022-09-07T21:41:51.447 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether gcc accepts -g... (cached) yes 2022-09-07T21:41:51.448 INFO:tasks.workunit.client.0.smithi047.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2022-09-07T21:41:51.449 INFO:tasks.workunit.client.0.smithi047.stdout:checking dependency style of gcc... (cached) gcc3 2022-09-07T21:41:51.593 INFO:tasks.workunit.client.0.smithi047.stdout:checking for main in -lm... yes 2022-09-07T21:41:51.698 INFO:tasks.workunit.client.0.smithi047.stdout:checking for main in -lpthread... yes 2022-09-07T21:41:52.012 INFO:tasks.workunit.client.0.smithi047.stdout:checking for ANSI C header files... yes 2022-09-07T21:41:52.103 INFO:tasks.workunit.client.0.smithi047.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2022-09-07T21:41:52.200 INFO:tasks.workunit.client.0.smithi047.stdout:checking for sys/types.h... yes 2022-09-07T21:41:52.310 INFO:tasks.workunit.client.0.smithi047.stdout:checking for sys/stat.h... yes 2022-09-07T21:41:52.419 INFO:tasks.workunit.client.0.smithi047.stdout:checking for stdlib.h... yes 2022-09-07T21:41:52.606 INFO:tasks.workunit.client.0.smithi047.stdout:checking for string.h... yes 2022-09-07T21:41:52.717 INFO:tasks.workunit.client.0.smithi047.stdout:checking for memory.h... yes 2022-09-07T21:41:52.823 INFO:tasks.workunit.client.0.smithi047.stdout:checking for strings.h... yes 2022-09-07T21:41:52.932 INFO:tasks.workunit.client.0.smithi047.stdout:checking for inttypes.h... yes 2022-09-07T21:41:53.040 INFO:tasks.workunit.client.0.smithi047.stdout:checking for stdint.h... yes 2022-09-07T21:41:53.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:52 smithi047 ceph-mon[90087]: pgmap v27: 129 pgs: 129 active+clean; 1.9 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 52 KiB/s wr, 10 op/s 2022-09-07T21:41:53.153 INFO:tasks.workunit.client.0.smithi047.stdout:checking for unistd.h... yes 2022-09-07T21:41:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:52 smithi055 ceph-mon[97033]: pgmap v27: 129 pgs: 129 active+clean; 1.9 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 52 KiB/s wr, 10 op/s 2022-09-07T21:41:53.272 INFO:tasks.workunit.client.0.smithi047.stdout:checking pthread.h usability... yes 2022-09-07T21:41:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:52 smithi059 ceph-mon[97450]: pgmap v27: 129 pgs: 129 active+clean; 1.9 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 52 KiB/s wr, 10 op/s 2022-09-07T21:41:53.318 INFO:tasks.workunit.client.0.smithi047.stdout:checking pthread.h presence... yes 2022-09-07T21:41:53.320 INFO:tasks.workunit.client.0.smithi047.stdout:checking for pthread.h... yes 2022-09-07T21:41:53.433 INFO:tasks.workunit.client.0.smithi047.stdout:checking fcntl.h usability... yes 2022-09-07T21:41:53.478 INFO:tasks.workunit.client.0.smithi047.stdout:checking fcntl.h presence... yes 2022-09-07T21:41:53.480 INFO:tasks.workunit.client.0.smithi047.stdout:checking for fcntl.h... yes 2022-09-07T21:41:53.589 INFO:tasks.workunit.client.0.smithi047.stdout:checking limits.h usability... yes 2022-09-07T21:41:53.631 INFO:tasks.workunit.client.0.smithi047.stdout:checking limits.h presence... yes 2022-09-07T21:41:53.633 INFO:tasks.workunit.client.0.smithi047.stdout:checking for limits.h... yes 2022-09-07T21:41:53.665 INFO:tasks.workunit.client.0.smithi047.stdout:checking for stdint.h... (cached) yes 2022-09-07T21:41:53.780 INFO:tasks.workunit.client.0.smithi047.stdout:checking sys/time.h usability... yes 2022-09-07T21:41:53.822 INFO:tasks.workunit.client.0.smithi047.stdout:checking sys/time.h presence... yes 2022-09-07T21:41:53.824 INFO:tasks.workunit.client.0.smithi047.stdout:checking for sys/time.h... yes 2022-09-07T21:41:53.856 INFO:tasks.workunit.client.0.smithi047.stdout:checking for unistd.h... (cached) yes 2022-09-07T21:41:53.965 INFO:tasks.workunit.client.0.smithi047.stdout:checking sys/vfs.h usability... yes 2022-09-07T21:41:54.102 INFO:tasks.workunit.client.0.smithi047.stdout:checking sys/vfs.h presence... yes 2022-09-07T21:41:54.104 INFO:tasks.workunit.client.0.smithi047.stdout:checking for sys/vfs.h... yes 2022-09-07T21:41:54.228 INFO:tasks.workunit.client.0.smithi047.stdout:checking sys/limits.h usability... no 2022-09-07T21:41:54.273 INFO:tasks.workunit.client.0.smithi047.stdout:checking sys/limits.h presence... no 2022-09-07T21:41:54.274 INFO:tasks.workunit.client.0.smithi047.stdout:checking for sys/limits.h... no 2022-09-07T21:41:54.348 INFO:tasks.workunit.client.0.smithi047.stdout:checking for an ANSI C-conforming const... yes 2022-09-07T21:41:54.425 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether time.h and sys/time.h may both be included... yes 2022-09-07T21:41:54.428 INFO:tasks.workunit.client.0.smithi047.stdout:checking for function prototypes... yes 2022-09-07T21:41:54.491 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether setvbuf arguments are reversed... no 2022-09-07T21:41:54.631 INFO:tasks.workunit.client.0.smithi047.stdout:checking for system... yes 2022-09-07T21:41:54.798 INFO:tasks.workunit.client.0.smithi047.stdout:checking for gettimeofday... yes 2022-09-07T21:41:54.939 INFO:tasks.workunit.client.0.smithi047.stdout:checking for mkdir... yes 2022-09-07T21:41:55.074 INFO:tasks.workunit.client.0.smithi047.stdout:checking for strerror... yes 2022-09-07T21:41:55.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:54 smithi047 ceph-mon[90087]: pgmap v28: 129 pgs: 129 active+clean; 1.9 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 52 KiB/s wr, 7 op/s 2022-09-07T21:41:55.209 INFO:tasks.workunit.client.0.smithi047.stdout:checking for open64... yes 2022-09-07T21:41:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:54 smithi055 ceph-mon[97033]: pgmap v28: 129 pgs: 129 active+clean; 1.9 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 52 KiB/s wr, 7 op/s 2022-09-07T21:41:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:54 smithi059 ceph-mon[97450]: pgmap v28: 129 pgs: 129 active+clean; 1.9 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 52 KiB/s wr, 7 op/s 2022-09-07T21:41:55.346 INFO:tasks.workunit.client.0.smithi047.stdout:checking for stat64... yes 2022-09-07T21:41:55.484 INFO:tasks.workunit.client.0.smithi047.stdout:checking for fseeko64... yes 2022-09-07T21:41:55.619 INFO:tasks.workunit.client.0.smithi047.stdout:checking for lrand48_r... yes 2022-09-07T21:41:55.754 INFO:tasks.workunit.client.0.smithi047.stdout:checking for srand48_r... yes 2022-09-07T21:41:55.813 INFO:tasks.workunit.client.0.smithi047.stdout:configure: creating ./config.status 2022-09-07T21:41:56.101 INFO:tasks.workunit.client.0.smithi047.stdout:config.status: creating Makefile 2022-09-07T21:41:56.140 INFO:tasks.workunit.client.0.smithi047.stdout:config.status: creating config.h 2022-09-07T21:41:56.181 INFO:tasks.workunit.client.0.smithi047.stdout:config.status: executing depfiles commands 2022-09-07T21:41:56.301 INFO:tasks.workunit.client.0.smithi047.stdout:config.status: executing default commands 2022-09-07T21:41:56.398 INFO:tasks.workunit.client.0.smithi047.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2022-09-07T21:41:56.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2022-09-07T21:41:56.407 INFO:tasks.workunit.client.0.smithi047.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2022-09-07T21:41:56.408 INFO:tasks.workunit.client.0.smithi047.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2022-09-07T21:41:56.408 INFO:tasks.workunit.client.0.smithi047.stderr: to install the `Automake' and `Perl' packages. Grab them from 2022-09-07T21:41:56.408 INFO:tasks.workunit.client.0.smithi047.stderr: any GNU archive site. 2022-09-07T21:41:56.417 INFO:tasks.workunit.client.0.smithi047.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2022-09-07T21:41:56.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2022-09-07T21:41:56.424 INFO:tasks.workunit.client.0.smithi047.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2022-09-07T21:41:56.424 INFO:tasks.workunit.client.0.smithi047.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2022-09-07T21:41:56.425 INFO:tasks.workunit.client.0.smithi047.stderr: You might want to install the `Automake' and `Perl' packages. 2022-09-07T21:41:56.425 INFO:tasks.workunit.client.0.smithi047.stderr: Grab them from any GNU archive site. 2022-09-07T21:41:56.455 INFO:tasks.workunit.client.0.smithi047.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2022-09-07T21:41:56.757 INFO:tasks.workunit.client.0.smithi047.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2022-09-07T21:41:56.757 INFO:tasks.workunit.client.0.smithi047.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2022-09-07T21:41:56.758 INFO:tasks.workunit.client.0.smithi047.stderr:If you have problems, you may need to regenerate the build system entirely. 2022-09-07T21:41:56.758 INFO:tasks.workunit.client.0.smithi047.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2022-09-07T21:41:56.903 INFO:tasks.workunit.client.0.smithi047.stdout:/bin/sh ./config.status --recheck 2022-09-07T21:41:57.075 INFO:tasks.workunit.client.0.smithi047.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2022-09-07T21:41:57.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:56 smithi047 ceph-mon[90087]: pgmap v29: 129 pgs: 129 active+clean; 4.0 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 243 KiB/s wr, 28 op/s 2022-09-07T21:41:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:56 smithi055 ceph-mon[97033]: pgmap v29: 129 pgs: 129 active+clean; 4.0 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 243 KiB/s wr, 28 op/s 2022-09-07T21:41:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:56 smithi059 ceph-mon[97450]: pgmap v29: 129 pgs: 129 active+clean; 4.0 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 243 KiB/s wr, 28 op/s 2022-09-07T21:41:57.330 INFO:tasks.workunit.client.0.smithi047.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2022-09-07T21:41:58.345 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether build environment is sane... yes 2022-09-07T21:41:58.362 INFO:tasks.workunit.client.0.smithi047.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2022-09-07T21:41:58.363 INFO:tasks.workunit.client.0.smithi047.stdout:checking for gawk... gawk 2022-09-07T21:41:58.396 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether make sets $(MAKE)... yes 2022-09-07T21:41:58.496 INFO:tasks.workunit.client.0.smithi047.stdout:checking build system type... x86_64-unknown-linux-gnu 2022-09-07T21:41:58.496 INFO:tasks.workunit.client.0.smithi047.stdout:checking host system type... x86_64-unknown-linux-gnu 2022-09-07T21:41:58.515 INFO:tasks.workunit.client.0.smithi047.stdout:checking for style of include used by make... GNU 2022-09-07T21:41:58.519 INFO:tasks.workunit.client.0.smithi047.stdout:checking for gcc... gcc 2022-09-07T21:41:58.672 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether the C compiler works... yes 2022-09-07T21:41:58.672 INFO:tasks.workunit.client.0.smithi047.stdout:checking for C compiler default output file name... a.out 2022-09-07T21:41:58.747 INFO:tasks.workunit.client.0.smithi047.stdout:checking for suffix of executables... 2022-09-07T21:41:58.826 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether we are cross compiling... no 2022-09-07T21:41:58.893 INFO:tasks.workunit.client.0.smithi047.stdout:checking for suffix of object files... o 2022-09-07T21:41:58.943 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether we are using the GNU C compiler... yes 2022-09-07T21:41:58.994 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether gcc accepts -g... yes 2022-09-07T21:41:59.060 INFO:tasks.workunit.client.0.smithi047.stdout:checking for gcc option to accept ISO C89... none needed 2022-09-07T21:41:59.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:41:58 smithi047 ceph-mon[90087]: pgmap v30: 129 pgs: 129 active+clean; 4.0 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 242 KiB/s wr, 25 op/s 2022-09-07T21:41:59.207 INFO:tasks.workunit.client.0.smithi047.stdout:checking dependency style of gcc... gcc3 2022-09-07T21:41:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:41:58 smithi055 ceph-mon[97033]: pgmap v30: 129 pgs: 129 active+clean; 4.0 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 242 KiB/s wr, 25 op/s 2022-09-07T21:41:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:41:58 smithi059 ceph-mon[97450]: pgmap v30: 129 pgs: 129 active+clean; 4.0 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 242 KiB/s wr, 25 op/s 2022-09-07T21:41:59.283 INFO:tasks.workunit.client.0.smithi047.stdout:checking how to run the C preprocessor... gcc -E 2022-09-07T21:41:59.363 INFO:tasks.workunit.client.0.smithi047.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2022-09-07T21:41:59.366 INFO:tasks.workunit.client.0.smithi047.stdout:checking for egrep... /usr/bin/grep -E 2022-09-07T21:41:59.600 INFO:tasks.workunit.client.0.smithi047.stdout:checking for ANSI C header files... yes 2022-09-07T21:41:59.670 INFO:tasks.workunit.client.0.smithi047.stdout:checking for sys/types.h... yes 2022-09-07T21:41:59.756 INFO:tasks.workunit.client.0.smithi047.stdout:checking for sys/stat.h... yes 2022-09-07T21:41:59.841 INFO:tasks.workunit.client.0.smithi047.stdout:checking for stdlib.h... yes 2022-09-07T21:41:59.927 INFO:tasks.workunit.client.0.smithi047.stdout:checking for string.h... yes 2022-09-07T21:42:00.011 INFO:tasks.workunit.client.0.smithi047.stdout:checking for memory.h... yes 2022-09-07T21:42:00.096 INFO:tasks.workunit.client.0.smithi047.stdout:checking for strings.h... yes 2022-09-07T21:42:00.182 INFO:tasks.workunit.client.0.smithi047.stdout:checking for inttypes.h... yes 2022-09-07T21:42:00.267 INFO:tasks.workunit.client.0.smithi047.stdout:checking for stdint.h... yes 2022-09-07T21:42:00.356 INFO:tasks.workunit.client.0.smithi047.stdout:checking for unistd.h... yes 2022-09-07T21:42:00.449 INFO:tasks.workunit.client.0.smithi047.stdout:checking minix/config.h usability... no 2022-09-07T21:42:00.506 INFO:tasks.workunit.client.0.smithi047.stdout:checking minix/config.h presence... no 2022-09-07T21:42:00.507 INFO:tasks.workunit.client.0.smithi047.stdout:checking for minix/config.h... no 2022-09-07T21:42:00.576 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2022-09-07T21:42:00.581 INFO:tasks.workunit.client.0.smithi047.stdout:checking for gcc... (cached) gcc 2022-09-07T21:42:00.642 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether we are using the GNU C compiler... (cached) yes 2022-09-07T21:42:00.642 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether gcc accepts -g... (cached) yes 2022-09-07T21:42:00.642 INFO:tasks.workunit.client.0.smithi047.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2022-09-07T21:42:00.643 INFO:tasks.workunit.client.0.smithi047.stdout:checking dependency style of gcc... (cached) gcc3 2022-09-07T21:42:00.727 INFO:tasks.workunit.client.0.smithi047.stdout:checking for main in -lm... yes 2022-09-07T21:42:00.817 INFO:tasks.workunit.client.0.smithi047.stdout:checking for main in -lpthread... yes 2022-09-07T21:42:00.820 INFO:tasks.workunit.client.0.smithi047.stdout:checking for ANSI C header files... (cached) yes 2022-09-07T21:42:00.871 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:42:00.420+0000 7f3f3ec29700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:42:00.871 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:42:00.420+0000 7f3f3ec29700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:42:00.872 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:42:00.420+0000 7f3f3ec29700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:42:00.872 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:42:00.420+0000 7f3f3ec29700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:42:00.872 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 07 21:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-07T21:42:00.420+0000 7f3f3ec29700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-07T21:42:00.885 INFO:tasks.workunit.client.0.smithi047.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2022-09-07T21:42:00.987 INFO:tasks.workunit.client.0.smithi047.stdout:checking pthread.h usability... yes 2022-09-07T21:42:01.017 INFO:tasks.workunit.client.0.smithi047.stdout:checking pthread.h presence... yes 2022-09-07T21:42:01.018 INFO:tasks.workunit.client.0.smithi047.stdout:checking for pthread.h... yes 2022-09-07T21:42:01.113 INFO:tasks.workunit.client.0.smithi047.stdout:checking fcntl.h usability... yes 2022-09-07T21:42:01.140 INFO:tasks.workunit.client.0.smithi047.stdout:checking fcntl.h presence... yes 2022-09-07T21:42:01.141 INFO:tasks.workunit.client.0.smithi047.stdout:checking for fcntl.h... yes 2022-09-07T21:42:01.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:00 smithi047 ceph-mon[90087]: pgmap v31: 129 pgs: 129 active+clean; 4.0 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 242 KiB/s wr, 25 op/s 2022-09-07T21:42:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:00 smithi055 ceph-mon[97033]: pgmap v31: 129 pgs: 129 active+clean; 4.0 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 242 KiB/s wr, 25 op/s 2022-09-07T21:42:01.239 INFO:tasks.workunit.client.0.smithi047.stdout:checking limits.h usability... yes 2022-09-07T21:42:01.268 INFO:tasks.workunit.client.0.smithi047.stdout:checking limits.h presence... yes 2022-09-07T21:42:01.268 INFO:tasks.workunit.client.0.smithi047.stdout:checking for limits.h... yes 2022-09-07T21:42:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:00 smithi059 ceph-mon[97450]: pgmap v31: 129 pgs: 129 active+clean; 4.0 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 242 KiB/s wr, 25 op/s 2022-09-07T21:42:01.296 INFO:tasks.workunit.client.0.smithi047.stdout:checking for stdint.h... (cached) yes 2022-09-07T21:42:01.381 INFO:tasks.workunit.client.0.smithi047.stdout:checking sys/time.h usability... yes 2022-09-07T21:42:01.411 INFO:tasks.workunit.client.0.smithi047.stdout:checking sys/time.h presence... yes 2022-09-07T21:42:01.411 INFO:tasks.workunit.client.0.smithi047.stdout:checking for sys/time.h... yes 2022-09-07T21:42:01.440 INFO:tasks.workunit.client.0.smithi047.stdout:checking for unistd.h... (cached) yes 2022-09-07T21:42:01.524 INFO:tasks.workunit.client.0.smithi047.stdout:checking sys/vfs.h usability... yes 2022-09-07T21:42:01.551 INFO:tasks.workunit.client.0.smithi047.stdout:checking sys/vfs.h presence... yes 2022-09-07T21:42:01.551 INFO:tasks.workunit.client.0.smithi047.stdout:checking for sys/vfs.h... yes 2022-09-07T21:42:01.653 INFO:tasks.workunit.client.0.smithi047.stdout:checking sys/limits.h usability... no 2022-09-07T21:42:01.698 INFO:tasks.workunit.client.0.smithi047.stdout:checking sys/limits.h presence... no 2022-09-07T21:42:01.698 INFO:tasks.workunit.client.0.smithi047.stdout:checking for sys/limits.h... no 2022-09-07T21:42:01.753 INFO:tasks.workunit.client.0.smithi047.stdout:checking for an ANSI C-conforming const... yes 2022-09-07T21:42:01.813 INFO:tasks.workunit.client.0.smithi047.stdout:checking whether time.h and sys/time.h may both be included... yes 2022-09-07T21:42:01.926 INFO:tasks.workunit.client.0.smithi047.stdout:checking for system... yes 2022-09-07T21:42:02.042 INFO:tasks.workunit.client.0.smithi047.stdout:checking for gettimeofday... yes 2022-09-07T21:42:02.158 INFO:tasks.workunit.client.0.smithi047.stdout:checking for mkdir... yes 2022-09-07T21:42:02.279 INFO:tasks.workunit.client.0.smithi047.stdout:checking for strerror... yes 2022-09-07T21:42:02.400 INFO:tasks.workunit.client.0.smithi047.stdout:checking for open64... yes 2022-09-07T21:42:02.522 INFO:tasks.workunit.client.0.smithi047.stdout:checking for stat64... yes 2022-09-07T21:42:02.635 INFO:tasks.workunit.client.0.smithi047.stdout:checking for fseeko64... yes 2022-09-07T21:42:02.787 INFO:tasks.workunit.client.0.smithi047.stdout:checking for lrand48_r... yes 2022-09-07T21:42:02.955 INFO:tasks.workunit.client.0.smithi047.stdout:checking for srand48_r... yes 2022-09-07T21:42:03.006 INFO:tasks.workunit.client.0.smithi047.stdout:configure: creating ./config.status 2022-09-07T21:42:03.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:02 smithi047 ceph-mon[90087]: pgmap v32: 129 pgs: 129 active+clean; 4.5 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 326 KiB/s wr, 41 op/s 2022-09-07T21:42:03.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:02 smithi047 ceph-mon[90087]: mgrmap e24: x(active, since 58s), standbys: y, z 2022-09-07T21:42:03.171 INFO:tasks.workunit.client.0.smithi047.stdout: /bin/sh ./config.status 2022-09-07T21:42:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:02 smithi055 ceph-mon[97033]: pgmap v32: 129 pgs: 129 active+clean; 4.5 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 326 KiB/s wr, 41 op/s 2022-09-07T21:42:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:02 smithi055 ceph-mon[97033]: mgrmap e24: x(active, since 58s), standbys: y, z 2022-09-07T21:42:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:02 smithi059 ceph-mon[97450]: pgmap v32: 129 pgs: 129 active+clean; 4.5 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 326 KiB/s wr, 41 op/s 2022-09-07T21:42:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:02 smithi059 ceph-mon[97450]: mgrmap e24: x(active, since 58s), standbys: y, z 2022-09-07T21:42:03.296 INFO:tasks.workunit.client.0.smithi047.stdout:config.status: creating Makefile 2022-09-07T21:42:03.329 INFO:tasks.workunit.client.0.smithi047.stdout:config.status: creating config.h 2022-09-07T21:42:03.370 INFO:tasks.workunit.client.0.smithi047.stdout:config.status: executing depfiles commands 2022-09-07T21:42:03.427 INFO:tasks.workunit.client.0.smithi047.stdout:config.status: executing default commands 2022-09-07T21:42:03.470 INFO:tasks.workunit.client.0.smithi047.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2022-09-07T21:42:03.713 INFO:tasks.workunit.client.0.smithi047.stdout:rm -f stamp-h1 2022-09-07T21:42:03.717 INFO:tasks.workunit.client.0.smithi047.stdout:touch config.h.in 2022-09-07T21:42:03.722 INFO:tasks.workunit.client.0.smithi047.stdout:cd . && /bin/sh ./config.status config.h 2022-09-07T21:42:03.826 INFO:tasks.workunit.client.0.smithi047.stdout:config.status: creating config.h 2022-09-07T21:42:03.871 INFO:tasks.workunit.client.0.smithi047.stdout:make all-am 2022-09-07T21:42:03.880 INFO:tasks.workunit.client.0.smithi047.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2022-09-07T21:42:03.880 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2022-09-07T21:42:04.116 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2022-09-07T21:42:04.123 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2022-09-07T21:42:04.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:03 smithi047 ceph-mon[90087]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:42:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:03 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:42:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:03 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:42:04.303 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2022-09-07T21:42:04.321 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2022-09-07T21:42:04.447 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/main.Tpo .deps/main.Po 2022-09-07T21:42:04.453 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2022-09-07T21:42:04.590 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2022-09-07T21:42:04.596 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2022-09-07T21:42:04.756 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2022-09-07T21:42:04.764 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2022-09-07T21:42:04.888 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2022-09-07T21:42:04.895 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2022-09-07T21:42:04.969 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2022-09-07T21:42:04.976 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2022-09-07T21:42:05.057 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2022-09-07T21:42:05.063 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2022-09-07T21:42:05.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:04 smithi047 ceph-mon[90087]: pgmap v33: 129 pgs: 129 active+clean; 4.5 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 275 KiB/s wr, 37 op/s 2022-09-07T21:42:05.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:42:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:04 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:42:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:42:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:04 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:42:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:04 smithi055 ceph-mon[97033]: pgmap v33: 129 pgs: 129 active+clean; 4.5 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 275 KiB/s wr, 37 op/s 2022-09-07T21:42:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:42:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:04 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:42:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:42:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:04 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:42:05.268 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2022-09-07T21:42:05.275 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2022-09-07T21:42:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:04 smithi059 ceph-mon[97450]: pgmap v33: 129 pgs: 129 active+clean; 4.5 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 275 KiB/s wr, 37 op/s 2022-09-07T21:42:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:42:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:04 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:42:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:42:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:04 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:42:05.432 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2022-09-07T21:42:05.439 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2022-09-07T21:42:05.627 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2022-09-07T21:42:05.634 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2022-09-07T21:42:05.738 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2022-09-07T21:42:05.745 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2022-09-07T21:42:05.852 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2022-09-07T21:42:05.858 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2022-09-07T21:42:06.004 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/util.Tpo .deps/util.Po 2022-09-07T21:42:06.010 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2022-09-07T21:42:06.057 INFO:tasks.workunit.client.0.smithi047.stderr:In file included from parser.h:22, 2022-09-07T21:42:06.057 INFO:tasks.workunit.client.0.smithi047.stderr: from parser.c:25: 2022-09-07T21:42:06.058 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c: In function ‘init_filesys’: 2022-09-07T21:42:06.058 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2022-09-07T21:42:06.058 INFO:tasks.workunit.client.0.smithi047.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2022-09-07T21:42:06.058 INFO:tasks.workunit.client.0.smithi047.stderr: ^ 2022-09-07T21:42:06.059 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2022-09-07T21:42:06.059 INFO:tasks.workunit.client.0.smithi047.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2022-09-07T21:42:06.059 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~ 2022-09-07T21:42:06.059 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2022-09-07T21:42:06.060 INFO:tasks.workunit.client.0.smithi047.stderr: container_of(ptr, type, member) 2022-09-07T21:42:06.060 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~ 2022-09-07T21:42:06.060 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:39:13: note: in expansion of macro ‘list_entry’ 2022-09-07T21:42:06.060 INFO:tasks.workunit.client.0.smithi047.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2022-09-07T21:42:06.060 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~ 2022-09-07T21:42:06.061 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c:934:3: note: in expansion of macro ‘list_for_each_entry’ 2022-09-07T21:42:06.061 INFO:tasks.workunit.client.0.smithi047.stderr: list_for_each_entry(tmp_list, &list_head->list, list) 2022-09-07T21:42:06.061 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~~~~~~~~ 2022-09-07T21:42:06.061 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2022-09-07T21:42:06.062 INFO:tasks.workunit.client.0.smithi047.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2022-09-07T21:42:06.062 INFO:tasks.workunit.client.0.smithi047.stderr: ^ 2022-09-07T21:42:06.062 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2022-09-07T21:42:06.062 INFO:tasks.workunit.client.0.smithi047.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2022-09-07T21:42:06.063 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~ 2022-09-07T21:42:06.063 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2022-09-07T21:42:06.063 INFO:tasks.workunit.client.0.smithi047.stderr: container_of(ptr, type, member) 2022-09-07T21:42:06.063 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~ 2022-09-07T21:42:06.064 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:41:13: note: in expansion of macro ‘list_entry’ 2022-09-07T21:42:06.064 INFO:tasks.workunit.client.0.smithi047.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2022-09-07T21:42:06.064 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~ 2022-09-07T21:42:06.065 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c:934:3: note: in expansion of macro ‘list_for_each_entry’ 2022-09-07T21:42:06.065 INFO:tasks.workunit.client.0.smithi047.stderr: list_for_each_entry(tmp_list, &list_head->list, list) 2022-09-07T21:42:06.065 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~~~~~~~~ 2022-09-07T21:42:06.066 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2022-09-07T21:42:06.066 INFO:tasks.workunit.client.0.smithi047.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2022-09-07T21:42:06.066 INFO:tasks.workunit.client.0.smithi047.stderr: ^ 2022-09-07T21:42:06.066 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2022-09-07T21:42:06.067 INFO:tasks.workunit.client.0.smithi047.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2022-09-07T21:42:06.067 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~ 2022-09-07T21:42:06.067 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2022-09-07T21:42:06.067 INFO:tasks.workunit.client.0.smithi047.stderr: container_of(ptr, type, member) 2022-09-07T21:42:06.067 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~ 2022-09-07T21:42:06.068 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:39:13: note: in expansion of macro ‘list_entry’ 2022-09-07T21:42:06.068 INFO:tasks.workunit.client.0.smithi047.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2022-09-07T21:42:06.068 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~ 2022-09-07T21:42:06.068 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c:941:3: note: in expansion of macro ‘list_for_each_entry’ 2022-09-07T21:42:06.069 INFO:tasks.workunit.client.0.smithi047.stderr: list_for_each_entry(tmp_list, &list_head->list, list) { 2022-09-07T21:42:06.069 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~~~~~~~~ 2022-09-07T21:42:06.069 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2022-09-07T21:42:06.069 INFO:tasks.workunit.client.0.smithi047.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2022-09-07T21:42:06.070 INFO:tasks.workunit.client.0.smithi047.stderr: ^ 2022-09-07T21:42:06.070 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2022-09-07T21:42:06.071 INFO:tasks.workunit.client.0.smithi047.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2022-09-07T21:42:06.071 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~ 2022-09-07T21:42:06.071 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2022-09-07T21:42:06.071 INFO:tasks.workunit.client.0.smithi047.stderr: container_of(ptr, type, member) 2022-09-07T21:42:06.072 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~ 2022-09-07T21:42:06.072 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:41:13: note: in expansion of macro ‘list_entry’ 2022-09-07T21:42:06.072 INFO:tasks.workunit.client.0.smithi047.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2022-09-07T21:42:06.072 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~ 2022-09-07T21:42:06.072 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c:941:3: note: in expansion of macro ‘list_for_each_entry’ 2022-09-07T21:42:06.073 INFO:tasks.workunit.client.0.smithi047.stderr: list_for_each_entry(tmp_list, &list_head->list, list) { 2022-09-07T21:42:06.073 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~~~~~~~~ 2022-09-07T21:42:06.073 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c: In function ‘init_tg_stats’: 2022-09-07T21:42:06.073 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2022-09-07T21:42:06.074 INFO:tasks.workunit.client.0.smithi047.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2022-09-07T21:42:06.074 INFO:tasks.workunit.client.0.smithi047.stderr: ^ 2022-09-07T21:42:06.074 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2022-09-07T21:42:06.074 INFO:tasks.workunit.client.0.smithi047.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2022-09-07T21:42:06.075 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~ 2022-09-07T21:42:06.075 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2022-09-07T21:42:06.075 INFO:tasks.workunit.client.0.smithi047.stderr: container_of(ptr, type, member) 2022-09-07T21:42:06.075 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~ 2022-09-07T21:42:06.076 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:39:13: note: in expansion of macro ‘list_entry’ 2022-09-07T21:42:06.076 INFO:tasks.workunit.client.0.smithi047.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2022-09-07T21:42:06.076 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~ 2022-09-07T21:42:06.076 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c:971:6: note: in expansion of macro ‘list_for_each_entry’ 2022-09-07T21:42:06.077 INFO:tasks.workunit.client.0.smithi047.stderr: list_for_each_entry(tmp_list, 2022-09-07T21:42:06.077 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~~~~~~~~ 2022-09-07T21:42:06.077 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2022-09-07T21:42:06.077 INFO:tasks.workunit.client.0.smithi047.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2022-09-07T21:42:06.077 INFO:tasks.workunit.client.0.smithi047.stderr: ^ 2022-09-07T21:42:06.078 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2022-09-07T21:42:06.078 INFO:tasks.workunit.client.0.smithi047.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2022-09-07T21:42:06.078 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~ 2022-09-07T21:42:06.078 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2022-09-07T21:42:06.079 INFO:tasks.workunit.client.0.smithi047.stderr: container_of(ptr, type, member) 2022-09-07T21:42:06.079 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~ 2022-09-07T21:42:06.079 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:41:13: note: in expansion of macro ‘list_entry’ 2022-09-07T21:42:06.079 INFO:tasks.workunit.client.0.smithi047.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2022-09-07T21:42:06.080 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~ 2022-09-07T21:42:06.080 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c:971:6: note: in expansion of macro ‘list_for_each_entry’ 2022-09-07T21:42:06.080 INFO:tasks.workunit.client.0.smithi047.stderr: list_for_each_entry(tmp_list, 2022-09-07T21:42:06.080 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~~~~~~~~ 2022-09-07T21:42:06.080 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2022-09-07T21:42:06.081 INFO:tasks.workunit.client.0.smithi047.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2022-09-07T21:42:06.081 INFO:tasks.workunit.client.0.smithi047.stderr: ^ 2022-09-07T21:42:06.081 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2022-09-07T21:42:06.081 INFO:tasks.workunit.client.0.smithi047.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2022-09-07T21:42:06.082 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~ 2022-09-07T21:42:06.082 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2022-09-07T21:42:06.082 INFO:tasks.workunit.client.0.smithi047.stderr: container_of(ptr, type, member) 2022-09-07T21:42:06.082 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~ 2022-09-07T21:42:06.082 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:39:13: note: in expansion of macro ‘list_entry’ 2022-09-07T21:42:06.083 INFO:tasks.workunit.client.0.smithi047.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2022-09-07T21:42:06.083 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~ 2022-09-07T21:42:06.083 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c:980:6: note: in expansion of macro ‘list_for_each_entry’ 2022-09-07T21:42:06.083 INFO:tasks.workunit.client.0.smithi047.stderr: list_for_each_entry(tmp_list, 2022-09-07T21:42:06.084 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~~~~~~~~ 2022-09-07T21:42:06.084 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2022-09-07T21:42:06.084 INFO:tasks.workunit.client.0.smithi047.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2022-09-07T21:42:06.084 INFO:tasks.workunit.client.0.smithi047.stderr: ^ 2022-09-07T21:42:06.085 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2022-09-07T21:42:06.085 INFO:tasks.workunit.client.0.smithi047.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2022-09-07T21:42:06.086 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~ 2022-09-07T21:42:06.086 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2022-09-07T21:42:06.086 INFO:tasks.workunit.client.0.smithi047.stderr: container_of(ptr, type, member) 2022-09-07T21:42:06.087 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~ 2022-09-07T21:42:06.087 INFO:tasks.workunit.client.0.smithi047.stderr:list.h:41:13: note: in expansion of macro ‘list_entry’ 2022-09-07T21:42:06.088 INFO:tasks.workunit.client.0.smithi047.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2022-09-07T21:42:06.088 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~ 2022-09-07T21:42:06.089 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c:980:6: note: in expansion of macro ‘list_for_each_entry’ 2022-09-07T21:42:06.089 INFO:tasks.workunit.client.0.smithi047.stderr: list_for_each_entry(tmp_list, 2022-09-07T21:42:06.090 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~~~~~~~~ 2022-09-07T21:42:06.179 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c: In function ‘set_weight’: 2022-09-07T21:42:06.179 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c:623:4: warning: ‘strncpy’ specified bound depends on the length of the source argument [-Wstringop-overflow=] 2022-09-07T21:42:06.180 INFO:tasks.workunit.client.0.smithi047.stderr: strncpy (op, tmp_config->name, len - 7); 2022-09-07T21:42:06.180 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-09-07T21:42:06.180 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c:620:10: note: length computed here 2022-09-07T21:42:06.180 INFO:tasks.workunit.client.0.smithi047.stderr: len = strlen(tmp_config->name); 2022-09-07T21:42:06.181 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~ 2022-09-07T21:42:06.184 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c: In function ‘get_weight_total’: 2022-09-07T21:42:06.184 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c:645:4: warning: ‘strncpy’ specified bound depends on the length of the source argument [-Wstringop-overflow=] 2022-09-07T21:42:06.184 INFO:tasks.workunit.client.0.smithi047.stderr: strncpy (op, tmp_config->name, len - 7); 2022-09-07T21:42:06.184 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-09-07T21:42:06.185 INFO:tasks.workunit.client.0.smithi047.stderr:parser.c:642:10: note: length computed here 2022-09-07T21:42:06.185 INFO:tasks.workunit.client.0.smithi047.stderr: len = strlen(tmp_config->name); 2022-09-07T21:42:06.185 INFO:tasks.workunit.client.0.smithi047.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~ 2022-09-07T21:42:06.341 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2022-09-07T21:42:06.347 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2022-09-07T21:42:06.439 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2022-09-07T21:42:06.445 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2022-09-07T21:42:06.573 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2022-09-07T21:42:06.579 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2022-09-07T21:42:06.638 INFO:tasks.workunit.client.0.smithi047.stdout:mv -f .deps/list.Tpo .deps/list.Po 2022-09-07T21:42:06.646 INFO:tasks.workunit.client.0.smithi047.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2022-09-07T21:42:06.705 INFO:tasks.workunit.client.0.smithi047.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2022-09-07T21:42:06.713 INFO:tasks.workunit.client.0.smithi047.stdout:FFSB version 6.0-RC2 started 2022-09-07T21:42:06.714 INFO:tasks.workunit.client.0.smithi047.stdout: 2022-09-07T21:42:06.721 INFO:tasks.workunit.client.0.smithi047.stdout:benchmark time = 300 2022-09-07T21:42:06.722 INFO:tasks.workunit.client.0.smithi047.stdout:ThreadGroup 0 2022-09-07T21:42:06.722 INFO:tasks.workunit.client.0.smithi047.stdout:================ 2022-09-07T21:42:06.722 INFO:tasks.workunit.client.0.smithi047.stdout: num_threads = 32 2022-09-07T21:42:06.722 INFO:tasks.workunit.client.0.smithi047.stdout: 2022-09-07T21:42:06.723 INFO:tasks.workunit.client.0.smithi047.stdout: read_random = off 2022-09-07T21:42:06.723 INFO:tasks.workunit.client.0.smithi047.stdout: read_size = 0 (0B) 2022-09-07T21:42:06.723 INFO:tasks.workunit.client.0.smithi047.stdout: read_blocksize = 0 (0B) 2022-09-07T21:42:06.723 INFO:tasks.workunit.client.0.smithi047.stdout: read_skip = off 2022-09-07T21:42:06.723 INFO:tasks.workunit.client.0.smithi047.stdout: read_skipsize = 0 (0B) 2022-09-07T21:42:06.724 INFO:tasks.workunit.client.0.smithi047.stdout: 2022-09-07T21:42:06.724 INFO:tasks.workunit.client.0.smithi047.stdout: write_random = on 2022-09-07T21:42:06.724 INFO:tasks.workunit.client.0.smithi047.stdout: write_size = 5242880 (5MB) 2022-09-07T21:42:06.724 INFO:tasks.workunit.client.0.smithi047.stdout: fsync_file = 0 2022-09-07T21:42:06.725 INFO:tasks.workunit.client.0.smithi047.stdout: write_blocksize = 4096 (4KB) 2022-09-07T21:42:06.725 INFO:tasks.workunit.client.0.smithi047.stdout: wait time = 0 2022-09-07T21:42:06.725 INFO:tasks.workunit.client.0.smithi047.stdout: 2022-09-07T21:42:06.725 INFO:tasks.workunit.client.0.smithi047.stdout: op weights 2022-09-07T21:42:06.726 INFO:tasks.workunit.client.0.smithi047.stdout: read = 0 (0.00%) 2022-09-07T21:42:06.726 INFO:tasks.workunit.client.0.smithi047.stdout: readall = 0 (0.00%) 2022-09-07T21:42:06.726 INFO:tasks.workunit.client.0.smithi047.stdout: write = 1 (100.00%) 2022-09-07T21:42:06.726 INFO:tasks.workunit.client.0.smithi047.stdout: create = 0 (0.00%) 2022-09-07T21:42:06.727 INFO:tasks.workunit.client.0.smithi047.stdout: append = 0 (0.00%) 2022-09-07T21:42:06.727 INFO:tasks.workunit.client.0.smithi047.stdout: delete = 0 (0.00%) 2022-09-07T21:42:06.727 INFO:tasks.workunit.client.0.smithi047.stdout: metaop = 0 (0.00%) 2022-09-07T21:42:06.727 INFO:tasks.workunit.client.0.smithi047.stdout: createdir = 0 (0.00%) 2022-09-07T21:42:06.727 INFO:tasks.workunit.client.0.smithi047.stdout: stat = 0 (0.00%) 2022-09-07T21:42:06.728 INFO:tasks.workunit.client.0.smithi047.stdout: writeall = 0 (0.00%) 2022-09-07T21:42:06.728 INFO:tasks.workunit.client.0.smithi047.stdout: writeall_fsync = 0 (0.00%) 2022-09-07T21:42:06.728 INFO:tasks.workunit.client.0.smithi047.stdout: open_close = 0 (0.00%) 2022-09-07T21:42:06.728 INFO:tasks.workunit.client.0.smithi047.stdout: write_fsync = 0 (0.00%) 2022-09-07T21:42:06.729 INFO:tasks.workunit.client.0.smithi047.stdout: create_fsync = 0 (0.00%) 2022-09-07T21:42:06.729 INFO:tasks.workunit.client.0.smithi047.stdout: append_fsync = 0 (0.00%) 2022-09-07T21:42:06.729 INFO:tasks.workunit.client.0.smithi047.stdout: 2022-09-07T21:42:06.729 INFO:tasks.workunit.client.0.smithi047.stdout:FileSystem . 2022-09-07T21:42:06.729 INFO:tasks.workunit.client.0.smithi047.stdout:========== 2022-09-07T21:42:06.730 INFO:tasks.workunit.client.0.smithi047.stdout: num_dirs = 0 2022-09-07T21:42:06.730 INFO:tasks.workunit.client.0.smithi047.stdout: starting files = 128 2022-09-07T21:42:06.730 INFO:tasks.workunit.client.0.smithi047.stdout: 2022-09-07T21:42:06.730 INFO:tasks.workunit.client.0.smithi047.stdout: min file size = 104857600 (100MB) 2022-09-07T21:42:06.731 INFO:tasks.workunit.client.0.smithi047.stdout: max file size = 104857600 (100MB) 2022-09-07T21:42:06.731 INFO:tasks.workunit.client.0.smithi047.stdout: directio = off 2022-09-07T21:42:06.731 INFO:tasks.workunit.client.0.smithi047.stdout: alignedio = on 2022-09-07T21:42:06.731 INFO:tasks.workunit.client.0.smithi047.stdout: bufferedio = off 2022-09-07T21:42:06.732 INFO:tasks.workunit.client.0.smithi047.stdout: 2022-09-07T21:42:06.732 INFO:tasks.workunit.client.0.smithi047.stdout: aging is off 2022-09-07T21:42:06.732 INFO:tasks.workunit.client.0.smithi047.stdout: current utilization = 0.01% 2022-09-07T21:42:06.732 INFO:tasks.workunit.client.0.smithi047.stdout: 2022-09-07T21:42:06.733 INFO:tasks.workunit.client.0.smithi047.stdout:checking existing fs: . 2022-09-07T21:42:06.733 INFO:tasks.workunit.client.0.smithi047.stdout:recreating new fileset 2022-09-07T21:42:06.734 INFO:tasks.workunit.client.0.smithi047.stderr:opendir: No such file or directory 2022-09-07T21:42:07.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:06 smithi047 ceph-mon[90087]: pgmap v34: 129 pgs: 129 active+clean; 7.6 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 579 KiB/s wr, 79 op/s 2022-09-07T21:42:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:06 smithi055 ceph-mon[97033]: pgmap v34: 129 pgs: 129 active+clean; 7.6 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 579 KiB/s wr, 79 op/s 2022-09-07T21:42:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:06 smithi059 ceph-mon[97450]: pgmap v34: 129 pgs: 129 active+clean; 7.6 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 579 KiB/s wr, 79 op/s 2022-09-07T21:42:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:08 smithi055 ceph-mon[97033]: pgmap v35: 129 pgs: 129 active+clean; 7.6 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 388 KiB/s wr, 58 op/s 2022-09-07T21:42:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:08 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:42:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:08 smithi059 ceph-mon[97450]: pgmap v35: 129 pgs: 129 active+clean; 7.6 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 388 KiB/s wr, 58 op/s 2022-09-07T21:42:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:08 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:42:09.415 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:08 smithi047 ceph-mon[90087]: pgmap v35: 129 pgs: 129 active+clean; 7.6 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 388 KiB/s wr, 58 op/s 2022-09-07T21:42:09.415 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:08 smithi047 ceph-mon[90087]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:42:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:10 smithi055 ceph-mon[97033]: pgmap v36: 129 pgs: 129 active+clean; 7.6 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 388 KiB/s wr, 58 op/s 2022-09-07T21:42:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:10 smithi059 ceph-mon[97450]: pgmap v36: 129 pgs: 129 active+clean; 7.6 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 388 KiB/s wr, 58 op/s 2022-09-07T21:42:11.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:10 smithi047 ceph-mon[90087]: pgmap v36: 129 pgs: 129 active+clean; 7.6 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 388 KiB/s wr, 58 op/s 2022-09-07T21:42:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:12 smithi055 ceph-mon[97033]: pgmap v37: 129 pgs: 129 active+clean; 65 MiB data, 116 MiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 5.2 MiB/s wr, 91 op/s 2022-09-07T21:42:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:12 smithi059 ceph-mon[97450]: pgmap v37: 129 pgs: 129 active+clean; 65 MiB data, 116 MiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 5.2 MiB/s wr, 91 op/s 2022-09-07T21:42:13.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:12 smithi047 ceph-mon[90087]: pgmap v37: 129 pgs: 129 active+clean; 65 MiB data, 116 MiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 5.2 MiB/s wr, 91 op/s 2022-09-07T21:42:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:14 smithi055 ceph-mon[97033]: pgmap v38: 129 pgs: 129 active+clean; 65 MiB data, 116 MiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 5.1 MiB/s wr, 74 op/s 2022-09-07T21:42:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:14 smithi059 ceph-mon[97450]: pgmap v38: 129 pgs: 129 active+clean; 65 MiB data, 116 MiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 5.1 MiB/s wr, 74 op/s 2022-09-07T21:42:15.409 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:14 smithi047 ceph-mon[90087]: pgmap v38: 129 pgs: 129 active+clean; 65 MiB data, 116 MiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 5.1 MiB/s wr, 74 op/s 2022-09-07T21:42:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:16 smithi055 ceph-mon[97033]: pgmap v39: 129 pgs: 129 active+clean; 121 MiB data, 138 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 9.8 MiB/s wr, 91 op/s 2022-09-07T21:42:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:42:17.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:42:17.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:42:17.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:16 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:42:17.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:16 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:42:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:16 smithi059 ceph-mon[97450]: pgmap v39: 129 pgs: 129 active+clean; 121 MiB data, 138 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 9.8 MiB/s wr, 91 op/s 2022-09-07T21:42:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:42:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:42:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:42:17.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:16 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:42:17.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:16 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:42:17.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:16 smithi047 ceph-mon[90087]: pgmap v39: 129 pgs: 129 active+clean; 121 MiB data, 138 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 9.8 MiB/s wr, 91 op/s 2022-09-07T21:42:17.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:16 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:42:17.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:16 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:42:17.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:16 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:42:17.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:16 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:42:17.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:16 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:42:18.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:18 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:42:18.666 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:18 smithi047 ceph-mon[90087]: pgmap v40: 129 pgs: 129 active+clean; 121 MiB data, 138 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 9.6 MiB/s wr, 48 op/s 2022-09-07T21:42:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:18 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:42:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:18 smithi055 ceph-mon[97033]: pgmap v40: 129 pgs: 129 active+clean; 121 MiB data, 138 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 9.6 MiB/s wr, 48 op/s 2022-09-07T21:42:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:18 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:42:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:18 smithi059 ceph-mon[97450]: pgmap v40: 129 pgs: 129 active+clean; 121 MiB data, 138 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 9.6 MiB/s wr, 48 op/s 2022-09-07T21:42:21.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:20 smithi047 ceph-mon[90087]: pgmap v41: 129 pgs: 129 active+clean; 121 MiB data, 138 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 9.6 MiB/s wr, 48 op/s 2022-09-07T21:42:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:20 smithi055 ceph-mon[97033]: pgmap v41: 129 pgs: 129 active+clean; 121 MiB data, 138 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 9.6 MiB/s wr, 48 op/s 2022-09-07T21:42:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:20 smithi059 ceph-mon[97450]: pgmap v41: 129 pgs: 129 active+clean; 121 MiB data, 138 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 9.6 MiB/s wr, 48 op/s 2022-09-07T21:42:23.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:22 smithi047 ceph-mon[90087]: pgmap v42: 129 pgs: 129 active+clean; 478 MiB data, 188 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 39 MiB/s wr, 68 op/s 2022-09-07T21:42:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:22 smithi055 ceph-mon[97033]: pgmap v42: 129 pgs: 129 active+clean; 478 MiB data, 188 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 39 MiB/s wr, 68 op/s 2022-09-07T21:42:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:22 smithi059 ceph-mon[97450]: pgmap v42: 129 pgs: 129 active+clean; 478 MiB data, 188 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 39 MiB/s wr, 68 op/s 2022-09-07T21:42:25.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:24 smithi047 ceph-mon[90087]: pgmap v43: 129 pgs: 129 active+clean; 478 MiB data, 188 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 34 MiB/s wr, 35 op/s 2022-09-07T21:42:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:24 smithi055 ceph-mon[97033]: pgmap v43: 129 pgs: 129 active+clean; 478 MiB data, 188 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 34 MiB/s wr, 35 op/s 2022-09-07T21:42:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:24 smithi059 ceph-mon[97450]: pgmap v43: 129 pgs: 129 active+clean; 478 MiB data, 188 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 34 MiB/s wr, 35 op/s 2022-09-07T21:42:27.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:26 smithi047 ceph-mon[90087]: pgmap v44: 129 pgs: 129 active+clean; 1.6 GiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 127 MiB/s wr, 71 op/s 2022-09-07T21:42:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:26 smithi055 ceph-mon[97033]: pgmap v44: 129 pgs: 129 active+clean; 1.6 GiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 127 MiB/s wr, 71 op/s 2022-09-07T21:42:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:26 smithi059 ceph-mon[97450]: pgmap v44: 129 pgs: 129 active+clean; 1.6 GiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 127 MiB/s wr, 71 op/s 2022-09-07T21:42:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:28 smithi055 ceph-mon[97033]: pgmap v45: 129 pgs: 129 active+clean; 1.6 GiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 123 MiB/s wr, 54 op/s 2022-09-07T21:42:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:28 smithi059 ceph-mon[97450]: pgmap v45: 129 pgs: 129 active+clean; 1.6 GiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 123 MiB/s wr, 54 op/s 2022-09-07T21:42:29.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:28 smithi047 ceph-mon[90087]: pgmap v45: 129 pgs: 129 active+clean; 1.6 GiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 123 MiB/s wr, 54 op/s 2022-09-07T21:42:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:30 smithi055 ceph-mon[97033]: pgmap v46: 129 pgs: 129 active+clean; 1.6 GiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 123 MiB/s wr, 54 op/s 2022-09-07T21:42:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:30 smithi059 ceph-mon[97450]: pgmap v46: 129 pgs: 129 active+clean; 1.6 GiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 123 MiB/s wr, 54 op/s 2022-09-07T21:42:31.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:30 smithi047 ceph-mon[90087]: pgmap v46: 129 pgs: 129 active+clean; 1.6 GiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 123 MiB/s wr, 54 op/s 2022-09-07T21:42:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:32 smithi055 ceph-mon[97033]: pgmap v47: 129 pgs: 129 active+clean; 2.4 GiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 193 MiB/s wr, 84 op/s 2022-09-07T21:42:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:32 smithi059 ceph-mon[97450]: pgmap v47: 129 pgs: 129 active+clean; 2.4 GiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 193 MiB/s wr, 84 op/s 2022-09-07T21:42:33.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:32 smithi047 ceph-mon[90087]: pgmap v47: 129 pgs: 129 active+clean; 2.4 GiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 193 MiB/s wr, 84 op/s 2022-09-07T21:42:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:34 smithi059 ceph-mon[97450]: pgmap v48: 129 pgs: 129 active+clean; 2.4 GiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 164 MiB/s wr, 64 op/s 2022-09-07T21:42:35.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:34 smithi047 ceph-mon[90087]: pgmap v48: 129 pgs: 129 active+clean; 2.4 GiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 164 MiB/s wr, 64 op/s 2022-09-07T21:42:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:34 smithi055 ceph-mon[97033]: pgmap v48: 129 pgs: 129 active+clean; 2.4 GiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 164 MiB/s wr, 64 op/s 2022-09-07T21:42:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:36 smithi059 ceph-mon[97450]: pgmap v49: 129 pgs: 129 active+clean; 4.2 GiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 320 MiB/s wr, 113 op/s 2022-09-07T21:42:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:36 smithi059 ceph-mon[97450]: mgrmap e25: x(active, since 92s), standbys: y, z 2022-09-07T21:42:37.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:36 smithi047 ceph-mon[90087]: pgmap v49: 129 pgs: 129 active+clean; 4.2 GiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 320 MiB/s wr, 113 op/s 2022-09-07T21:42:37.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:36 smithi047 ceph-mon[90087]: mgrmap e25: x(active, since 92s), standbys: y, z 2022-09-07T21:42:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:36 smithi055 ceph-mon[97033]: pgmap v49: 129 pgs: 129 active+clean; 4.2 GiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 320 MiB/s wr, 113 op/s 2022-09-07T21:42:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:36 smithi055 ceph-mon[97033]: mgrmap e25: x(active, since 92s), standbys: y, z 2022-09-07T21:42:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:38 smithi059 ceph-mon[97450]: pgmap v50: 129 pgs: 129 active+clean; 4.2 GiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 227 MiB/s wr, 77 op/s 2022-09-07T21:42:39.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:38 smithi047 ceph-mon[90087]: pgmap v50: 129 pgs: 129 active+clean; 4.2 GiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 227 MiB/s wr, 77 op/s 2022-09-07T21:42:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:38 smithi055 ceph-mon[97033]: pgmap v50: 129 pgs: 129 active+clean; 4.2 GiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 227 MiB/s wr, 77 op/s 2022-09-07T21:42:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:40 smithi059 ceph-mon[97450]: pgmap v51: 129 pgs: 129 active+clean; 4.2 GiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 227 MiB/s wr, 77 op/s 2022-09-07T21:42:41.409 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:40 smithi047 ceph-mon[90087]: pgmap v51: 129 pgs: 129 active+clean; 4.2 GiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 227 MiB/s wr, 77 op/s 2022-09-07T21:42:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:40 smithi055 ceph-mon[97033]: pgmap v51: 129 pgs: 129 active+clean; 4.2 GiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 227 MiB/s wr, 77 op/s 2022-09-07T21:42:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:42 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:42:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:42 smithi059 ceph-mon[97450]: pgmap v52: 129 pgs: 129 active+clean; 4.5 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 251 MiB/s wr, 95 op/s 2022-09-07T21:42:42.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:42 smithi047 ceph-mon[90087]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:42:42.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:42 smithi047 ceph-mon[90087]: pgmap v52: 129 pgs: 129 active+clean; 4.5 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 251 MiB/s wr, 95 op/s 2022-09-07T21:42:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:42 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:42:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:42 smithi055 ceph-mon[97033]: pgmap v52: 129 pgs: 129 active+clean; 4.5 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 251 MiB/s wr, 95 op/s 2022-09-07T21:42:45.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:44 smithi047 ceph-mon[90087]: pgmap v53: 129 pgs: 129 active+clean; 4.5 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 181 MiB/s wr, 66 op/s 2022-09-07T21:42:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:44 smithi055 ceph-mon[97033]: pgmap v53: 129 pgs: 129 active+clean; 4.5 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 181 MiB/s wr, 66 op/s 2022-09-07T21:42:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:44 smithi059 ceph-mon[97450]: pgmap v53: 129 pgs: 129 active+clean; 4.5 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 181 MiB/s wr, 66 op/s 2022-09-07T21:42:47.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:46 smithi047 ceph-mon[90087]: pgmap v54: 129 pgs: 129 active+clean; 5.5 GiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 265 MiB/s wr, 99 op/s 2022-09-07T21:42:47.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:46 smithi047 ceph-mon[90087]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:42:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:46 smithi055 ceph-mon[97033]: pgmap v54: 129 pgs: 129 active+clean; 5.5 GiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 265 MiB/s wr, 99 op/s 2022-09-07T21:42:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:46 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:42:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:46 smithi059 ceph-mon[97450]: pgmap v54: 129 pgs: 129 active+clean; 5.5 GiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 265 MiB/s wr, 99 op/s 2022-09-07T21:42:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:46 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:42:49.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:48 smithi047 ceph-mon[90087]: pgmap v55: 129 pgs: 129 active+clean; 5.5 GiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 109 MiB/s wr, 50 op/s 2022-09-07T21:42:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:48 smithi055 ceph-mon[97033]: pgmap v55: 129 pgs: 129 active+clean; 5.5 GiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 109 MiB/s wr, 50 op/s 2022-09-07T21:42:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:48 smithi059 ceph-mon[97450]: pgmap v55: 129 pgs: 129 active+clean; 5.5 GiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 109 MiB/s wr, 50 op/s 2022-09-07T21:42:51.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:50 smithi047 ceph-mon[90087]: pgmap v56: 129 pgs: 129 active+clean; 5.5 GiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 109 MiB/s wr, 50 op/s 2022-09-07T21:42:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:50 smithi055 ceph-mon[97033]: pgmap v56: 129 pgs: 129 active+clean; 5.5 GiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 109 MiB/s wr, 50 op/s 2022-09-07T21:42:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:50 smithi059 ceph-mon[97450]: pgmap v56: 129 pgs: 129 active+clean; 5.5 GiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 109 MiB/s wr, 50 op/s 2022-09-07T21:42:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:52 smithi055 ceph-mon[97033]: pgmap v57: 129 pgs: 129 active+clean; 7.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 234 MiB/s wr, 89 op/s 2022-09-07T21:42:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:52 smithi059 ceph-mon[97450]: pgmap v57: 129 pgs: 129 active+clean; 7.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 234 MiB/s wr, 89 op/s 2022-09-07T21:42:53.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:52 smithi047 ceph-mon[90087]: pgmap v57: 129 pgs: 129 active+clean; 7.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 234 MiB/s wr, 89 op/s 2022-09-07T21:42:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:54 smithi055 ceph-mon[97033]: pgmap v58: 129 pgs: 129 active+clean; 7.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 209 MiB/s wr, 71 op/s 2022-09-07T21:42:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:54 smithi059 ceph-mon[97450]: pgmap v58: 129 pgs: 129 active+clean; 7.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 209 MiB/s wr, 71 op/s 2022-09-07T21:42:55.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:54 smithi047 ceph-mon[90087]: pgmap v58: 129 pgs: 129 active+clean; 7.0 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 209 MiB/s wr, 71 op/s 2022-09-07T21:42:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:56 smithi055 ceph-mon[97033]: pgmap v59: 129 pgs: 129 active+clean; 7.3 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 242 MiB/s wr, 91 op/s 2022-09-07T21:42:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:56 smithi059 ceph-mon[97450]: pgmap v59: 129 pgs: 129 active+clean; 7.3 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 242 MiB/s wr, 91 op/s 2022-09-07T21:42:57.407 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:56 smithi047 ceph-mon[90087]: pgmap v59: 129 pgs: 129 active+clean; 7.3 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 242 MiB/s wr, 91 op/s 2022-09-07T21:42:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:42:58 smithi055 ceph-mon[97033]: pgmap v60: 129 pgs: 129 active+clean; 7.3 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 157 MiB/s wr, 58 op/s 2022-09-07T21:42:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:42:58 smithi059 ceph-mon[97450]: pgmap v60: 129 pgs: 129 active+clean; 7.3 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 157 MiB/s wr, 58 op/s 2022-09-07T21:42:59.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:42:58 smithi047 ceph-mon[90087]: pgmap v60: 129 pgs: 129 active+clean; 7.3 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 157 MiB/s wr, 58 op/s 2022-09-07T21:43:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:00 smithi055 ceph-mon[97033]: pgmap v61: 129 pgs: 129 active+clean; 7.3 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 157 MiB/s wr, 58 op/s 2022-09-07T21:43:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:00 smithi059 ceph-mon[97450]: pgmap v61: 129 pgs: 129 active+clean; 7.3 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 157 MiB/s wr, 58 op/s 2022-09-07T21:43:01.407 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:00 smithi047 ceph-mon[90087]: pgmap v61: 129 pgs: 129 active+clean; 7.3 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 157 MiB/s wr, 58 op/s 2022-09-07T21:43:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:02 smithi059 ceph-mon[97450]: pgmap v62: 129 pgs: 129 active+clean; 8.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 240 MiB/s wr, 92 op/s 2022-09-07T21:43:03.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:02 smithi047 ceph-mon[90087]: pgmap v62: 129 pgs: 129 active+clean; 8.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 240 MiB/s wr, 92 op/s 2022-09-07T21:43:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:02 smithi055 ceph-mon[97033]: pgmap v62: 129 pgs: 129 active+clean; 8.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 240 MiB/s wr, 92 op/s 2022-09-07T21:43:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:04 smithi059 ceph-mon[97450]: pgmap v63: 129 pgs: 129 active+clean; 8.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 116 MiB/s wr, 53 op/s 2022-09-07T21:43:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:43:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:04 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:43:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:43:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:04 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:43:05.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:04 smithi047 ceph-mon[90087]: pgmap v63: 129 pgs: 129 active+clean; 8.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 116 MiB/s wr, 53 op/s 2022-09-07T21:43:05.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:43:05.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:04 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:43:05.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:43:05.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:04 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:43:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:04 smithi055 ceph-mon[97033]: pgmap v63: 129 pgs: 129 active+clean; 8.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 116 MiB/s wr, 53 op/s 2022-09-07T21:43:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:43:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:04 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:43:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:43:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:04 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:43:06.565 INFO:tasks.workunit.client.0.smithi047.stdout:fs setup took 59 secs 2022-09-07T21:43:07.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:07 smithi047 ceph-mon[90087]: pgmap v64: 129 pgs: 129 active+clean; 9.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 241 MiB/s wr, 106 op/s 2022-09-07T21:43:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:07 smithi055 ceph-mon[97033]: pgmap v64: 129 pgs: 129 active+clean; 9.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 241 MiB/s wr, 106 op/s 2022-09-07T21:43:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:07 smithi059 ceph-mon[97450]: pgmap v64: 129 pgs: 129 active+clean; 9.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 241 MiB/s wr, 106 op/s 2022-09-07T21:43:08.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:08 smithi047 ceph-mon[90087]: pgmap v65: 129 pgs: 129 active+clean; 9.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 208 MiB/s wr, 87 op/s 2022-09-07T21:43:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:08 smithi055 ceph-mon[97033]: pgmap v65: 129 pgs: 129 active+clean; 9.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 208 MiB/s wr, 87 op/s 2022-09-07T21:43:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:08 smithi059 ceph-mon[97450]: pgmap v65: 129 pgs: 129 active+clean; 9.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 208 MiB/s wr, 87 op/s 2022-09-07T21:43:11.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:10 smithi047 ceph-mon[90087]: pgmap v66: 129 pgs: 129 active+clean; 11 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 291 MiB/s wr, 114 op/s 2022-09-07T21:43:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:10 smithi055 ceph-mon[97033]: pgmap v66: 129 pgs: 129 active+clean; 11 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 291 MiB/s wr, 114 op/s 2022-09-07T21:43:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:10 smithi059 ceph-mon[97450]: pgmap v66: 129 pgs: 129 active+clean; 11 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 291 MiB/s wr, 114 op/s 2022-09-07T21:43:13.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:12 smithi047 ceph-mon[90087]: pgmap v67: 129 pgs: 129 active+clean; 11 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 324 MiB/s wr, 123 op/s 2022-09-07T21:43:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:12 smithi055 ceph-mon[97033]: pgmap v67: 129 pgs: 129 active+clean; 11 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 324 MiB/s wr, 123 op/s 2022-09-07T21:43:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:12 smithi059 ceph-mon[97450]: pgmap v67: 129 pgs: 129 active+clean; 11 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 324 MiB/s wr, 123 op/s 2022-09-07T21:43:15.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:14 smithi047 ceph-mon[90087]: pgmap v68: 129 pgs: 129 active+clean; 11 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 240 MiB/s wr, 89 op/s 2022-09-07T21:43:15.165 INFO:tasks.workunit.client.0.smithi047.stdout:Syncing()...8 sec 2022-09-07T21:43:15.167 INFO:tasks.workunit.client.0.smithi047.stdout:Starting Actual Benchmark At: Wed Sep 7 21:43:15 2022 2022-09-07T21:43:15.167 INFO:tasks.workunit.client.0.smithi047.stdout: 2022-09-07T21:43:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:14 smithi055 ceph-mon[97033]: pgmap v68: 129 pgs: 129 active+clean; 11 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 240 MiB/s wr, 89 op/s 2022-09-07T21:43:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:14 smithi059 ceph-mon[97450]: pgmap v68: 129 pgs: 129 active+clean; 11 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 240 MiB/s wr, 89 op/s 2022-09-07T21:43:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:16 smithi055 ceph-mon[97033]: pgmap v69: 129 pgs: 129 active+clean; 12 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 354 MiB/s wr, 121 op/s 2022-09-07T21:43:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:16 smithi059 ceph-mon[97450]: pgmap v69: 129 pgs: 129 active+clean; 12 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 354 MiB/s wr, 121 op/s 2022-09-07T21:43:17.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:16 smithi047 ceph-mon[90087]: pgmap v69: 129 pgs: 129 active+clean; 12 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 354 MiB/s wr, 121 op/s 2022-09-07T21:43:18.192 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:43:18.192 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:43:18.192 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:43:18.219 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:43:18.219 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:43:18.220 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:43:18.663 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:18 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:43:18.664 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:18 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:43:18.664 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:18 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:43:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:19 smithi055 ceph-mon[97033]: pgmap v70: 129 pgs: 129 active+clean; 12 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 229 MiB/s wr, 67 op/s 2022-09-07T21:43:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:19 smithi059 ceph-mon[97450]: pgmap v70: 129 pgs: 129 active+clean; 12 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 229 MiB/s wr, 67 op/s 2022-09-07T21:43:19.582 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:19 smithi047 ceph-mon[90087]: pgmap v70: 129 pgs: 129 active+clean; 12 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 229 MiB/s wr, 67 op/s 2022-09-07T21:43:20.658 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:20 smithi047 ceph-mon[90087]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 238 MiB/s wr, 873 op/s 2022-09-07T21:43:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:20 smithi055 ceph-mon[97033]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 238 MiB/s wr, 873 op/s 2022-09-07T21:43:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:20 smithi059 ceph-mon[97450]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 238 MiB/s wr, 873 op/s 2022-09-07T21:43:22.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:22 smithi047 ceph-mon[90087]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 156 MiB/s wr, 1.07k op/s 2022-09-07T21:43:22.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:22 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:43:22.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:22 smithi055 ceph-mon[97033]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 156 MiB/s wr, 1.07k op/s 2022-09-07T21:43:22.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:22 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:43:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:22 smithi059 ceph-mon[97450]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 156 MiB/s wr, 1.07k op/s 2022-09-07T21:43:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:22 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:43:25.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:25 smithi059 ceph-mon[97450]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 123 MiB/s wr, 1.06k op/s 2022-09-07T21:43:25.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:25 smithi047 ceph-mon[90087]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 123 MiB/s wr, 1.06k op/s 2022-09-07T21:43:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:25 smithi055 ceph-mon[97033]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 123 MiB/s wr, 1.06k op/s 2022-09-07T21:43:27.407 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:27 smithi047 ceph-mon[90087]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 148 MiB/s wr, 5.13k op/s 2022-09-07T21:43:27.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:27 smithi055 ceph-mon[97033]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 148 MiB/s wr, 5.13k op/s 2022-09-07T21:43:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:27 smithi059 ceph-mon[97450]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 148 MiB/s wr, 5.13k op/s 2022-09-07T21:43:29.407 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:29 smithi047 ceph-mon[90087]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 38 MiB/s wr, 5.65k op/s 2022-09-07T21:43:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:29 smithi055 ceph-mon[97033]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 38 MiB/s wr, 5.65k op/s 2022-09-07T21:43:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:29 smithi059 ceph-mon[97450]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 38 MiB/s wr, 5.65k op/s 2022-09-07T21:43:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:30 smithi055 ceph-mon[97033]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 64 MiB/s wr, 9.25k op/s 2022-09-07T21:43:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:30 smithi059 ceph-mon[97450]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 64 MiB/s wr, 9.25k op/s 2022-09-07T21:43:30.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:30 smithi047 ceph-mon[90087]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 64 MiB/s wr, 9.25k op/s 2022-09-07T21:43:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:32 smithi055 ceph-mon[97033]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 60 MiB/s wr, 9.04k op/s 2022-09-07T21:43:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:32 smithi059 ceph-mon[97450]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 60 MiB/s wr, 9.04k op/s 2022-09-07T21:43:33.409 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:32 smithi047 ceph-mon[90087]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 60 MiB/s wr, 9.04k op/s 2022-09-07T21:43:35.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:34 smithi055 ceph-mon[97033]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 62 MiB/s wr, 9.33k op/s 2022-09-07T21:43:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:34 smithi059 ceph-mon[97450]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 62 MiB/s wr, 9.33k op/s 2022-09-07T21:43:35.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:34 smithi047 ceph-mon[90087]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 62 MiB/s wr, 9.33k op/s 2022-09-07T21:43:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:36 smithi059 ceph-mon[97450]: pgmap v79: 129 pgs: 129 active+clean; 14 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 89 MiB/s wr, 12.59k op/s 2022-09-07T21:43:37.411 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:36 smithi047 ceph-mon[90087]: pgmap v79: 129 pgs: 129 active+clean; 14 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 89 MiB/s wr, 12.59k op/s 2022-09-07T21:43:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:36 smithi055 ceph-mon[97033]: pgmap v79: 129 pgs: 129 active+clean; 14 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 89 MiB/s wr, 12.59k op/s 2022-09-07T21:43:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:39 smithi059 ceph-mon[97450]: pgmap v80: 129 pgs: 129 active+clean; 14 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 70 MiB/s wr, 9.23k op/s 2022-09-07T21:43:39.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:39 smithi047 ceph-mon[90087]: pgmap v80: 129 pgs: 129 active+clean; 14 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 70 MiB/s wr, 9.23k op/s 2022-09-07T21:43:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:39 smithi055 ceph-mon[97033]: pgmap v80: 129 pgs: 129 active+clean; 14 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 70 MiB/s wr, 9.23k op/s 2022-09-07T21:43:41.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:41 smithi047 ceph-mon[90087]: pgmap v81: 129 pgs: 129 active+clean; 14 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 98 MiB/s wr, 12.36k op/s 2022-09-07T21:43:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:41 smithi055 ceph-mon[97033]: pgmap v81: 129 pgs: 129 active+clean; 14 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 98 MiB/s wr, 12.36k op/s 2022-09-07T21:43:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:41 smithi059 ceph-mon[97450]: pgmap v81: 129 pgs: 129 active+clean; 14 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 98 MiB/s wr, 12.36k op/s 2022-09-07T21:43:43.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:43 smithi047 ceph-mon[90087]: pgmap v82: 129 pgs: 129 active+clean; 14 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 74 MiB/s wr, 9.09k op/s 2022-09-07T21:43:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:43 smithi055 ceph-mon[97033]: pgmap v82: 129 pgs: 129 active+clean; 14 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 74 MiB/s wr, 9.09k op/s 2022-09-07T21:43:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:43 smithi059 ceph-mon[97450]: pgmap v82: 129 pgs: 129 active+clean; 14 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 74 MiB/s wr, 9.09k op/s 2022-09-07T21:43:45.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:45 smithi055 ceph-mon[97033]: pgmap v83: 129 pgs: 129 active+clean; 14 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 79 MiB/s wr, 9.45k op/s 2022-09-07T21:43:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:45 smithi059 ceph-mon[97450]: pgmap v83: 129 pgs: 129 active+clean; 14 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 79 MiB/s wr, 9.45k op/s 2022-09-07T21:43:45.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:45 smithi047 ceph-mon[90087]: pgmap v83: 129 pgs: 129 active+clean; 14 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 79 MiB/s wr, 9.45k op/s 2022-09-07T21:43:47.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:47 smithi055 ceph-mon[97033]: pgmap v84: 129 pgs: 129 active+clean; 14 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 104 MiB/s wr, 11.96k op/s 2022-09-07T21:43:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:47 smithi059 ceph-mon[97450]: pgmap v84: 129 pgs: 129 active+clean; 14 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 104 MiB/s wr, 11.96k op/s 2022-09-07T21:43:47.665 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:47 smithi047 ceph-mon[90087]: pgmap v84: 129 pgs: 129 active+clean; 14 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 104 MiB/s wr, 11.96k op/s 2022-09-07T21:43:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:49 smithi059 ceph-mon[97450]: pgmap v85: 129 pgs: 129 active+clean; 14 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 81 MiB/s wr, 8.94k op/s 2022-09-07T21:43:49.661 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:49 smithi047 ceph-mon[90087]: pgmap v85: 129 pgs: 129 active+clean; 14 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 81 MiB/s wr, 8.94k op/s 2022-09-07T21:43:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:49 smithi055 ceph-mon[97033]: pgmap v85: 129 pgs: 129 active+clean; 14 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 81 MiB/s wr, 8.94k op/s 2022-09-07T21:43:50.658 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:50 smithi047 ceph-mon[90087]: pgmap v86: 129 pgs: 129 active+clean; 15 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 112 MiB/s wr, 11.92k op/s 2022-09-07T21:43:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:50 smithi055 ceph-mon[97033]: pgmap v86: 129 pgs: 129 active+clean; 15 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 112 MiB/s wr, 11.92k op/s 2022-09-07T21:43:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:50 smithi059 ceph-mon[97450]: pgmap v86: 129 pgs: 129 active+clean; 15 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 112 MiB/s wr, 11.92k op/s 2022-09-07T21:43:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:52 smithi055 ceph-mon[97033]: pgmap v87: 129 pgs: 129 active+clean; 15 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 84 MiB/s wr, 8.53k op/s 2022-09-07T21:43:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:52 smithi059 ceph-mon[97450]: pgmap v87: 129 pgs: 129 active+clean; 15 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 84 MiB/s wr, 8.53k op/s 2022-09-07T21:43:53.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:52 smithi047 ceph-mon[90087]: pgmap v87: 129 pgs: 129 active+clean; 15 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 84 MiB/s wr, 8.53k op/s 2022-09-07T21:43:55.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:55 smithi047 ceph-mon[90087]: pgmap v88: 129 pgs: 129 active+clean; 15 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 83 MiB/s wr, 8.36k op/s 2022-09-07T21:43:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:55 smithi055 ceph-mon[97033]: pgmap v88: 129 pgs: 129 active+clean; 15 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 83 MiB/s wr, 8.36k op/s 2022-09-07T21:43:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:55 smithi059 ceph-mon[97450]: pgmap v88: 129 pgs: 129 active+clean; 15 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 83 MiB/s wr, 8.36k op/s 2022-09-07T21:43:57.413 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:57 smithi047 ceph-mon[90087]: pgmap v89: 129 pgs: 129 active+clean; 15 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 107 MiB/s wr, 10.38k op/s 2022-09-07T21:43:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:57 smithi055 ceph-mon[97033]: pgmap v89: 129 pgs: 129 active+clean; 15 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 107 MiB/s wr, 10.38k op/s 2022-09-07T21:43:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:57 smithi059 ceph-mon[97450]: pgmap v89: 129 pgs: 129 active+clean; 15 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 107 MiB/s wr, 10.38k op/s 2022-09-07T21:43:59.428 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:43:59 smithi047 ceph-mon[90087]: pgmap v90: 129 pgs: 129 active+clean; 15 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 80 MiB/s wr, 7.63k op/s 2022-09-07T21:43:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:43:59 smithi055 ceph-mon[97033]: pgmap v90: 129 pgs: 129 active+clean; 15 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 80 MiB/s wr, 7.63k op/s 2022-09-07T21:43:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:43:59 smithi059 ceph-mon[97450]: pgmap v90: 129 pgs: 129 active+clean; 15 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 80 MiB/s wr, 7.63k op/s 2022-09-07T21:44:01.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:01 smithi047 ceph-mon[90087]: pgmap v91: 129 pgs: 129 active+clean; 16 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 114 MiB/s wr, 10.10k op/s 2022-09-07T21:44:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:01 smithi055 ceph-mon[97033]: pgmap v91: 129 pgs: 129 active+clean; 16 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 114 MiB/s wr, 10.10k op/s 2022-09-07T21:44:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:01 smithi059 ceph-mon[97450]: pgmap v91: 129 pgs: 129 active+clean; 16 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 114 MiB/s wr, 10.10k op/s 2022-09-07T21:44:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:03 smithi055 ceph-mon[97033]: pgmap v92: 129 pgs: 129 active+clean; 16 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 88 MiB/s wr, 7.16k op/s 2022-09-07T21:44:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:03 smithi059 ceph-mon[97450]: pgmap v92: 129 pgs: 129 active+clean; 16 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 88 MiB/s wr, 7.16k op/s 2022-09-07T21:44:03.658 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:03 smithi047 ceph-mon[90087]: pgmap v92: 129 pgs: 129 active+clean; 16 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 88 MiB/s wr, 7.16k op/s 2022-09-07T21:44:05.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:05 smithi047 ceph-mon[90087]: pgmap v93: 129 pgs: 129 active+clean; 16 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 88 MiB/s wr, 7.04k op/s 2022-09-07T21:44:05.407 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:05 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:44:05.407 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:05 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:44:05.408 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:05 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:44:05.408 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:05 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:44:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:05 smithi055 ceph-mon[97033]: pgmap v93: 129 pgs: 129 active+clean; 16 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 88 MiB/s wr, 7.04k op/s 2022-09-07T21:44:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:44:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:44:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:05 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:44:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:05 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:44:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:05 smithi059 ceph-mon[97450]: pgmap v93: 129 pgs: 129 active+clean; 16 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 88 MiB/s wr, 7.04k op/s 2022-09-07T21:44:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:44:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:05 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:05 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:44:07.414 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:07 smithi047 ceph-mon[90087]: pgmap v94: 129 pgs: 129 active+clean; 16 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 133 MiB/s wr, 9.68k op/s 2022-09-07T21:44:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:07 smithi055 ceph-mon[97033]: pgmap v94: 129 pgs: 129 active+clean; 16 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 133 MiB/s wr, 9.68k op/s 2022-09-07T21:44:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:07 smithi059 ceph-mon[97450]: pgmap v94: 129 pgs: 129 active+clean; 16 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 133 MiB/s wr, 9.68k op/s 2022-09-07T21:44:09.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:09 smithi047 ceph-mon[90087]: pgmap v95: 129 pgs: 129 active+clean; 16 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 110 MiB/s wr, 7.26k op/s 2022-09-07T21:44:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:09 smithi055 ceph-mon[97033]: pgmap v95: 129 pgs: 129 active+clean; 16 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 110 MiB/s wr, 7.26k op/s 2022-09-07T21:44:09.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:09 smithi059 ceph-mon[97450]: pgmap v95: 129 pgs: 129 active+clean; 16 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 110 MiB/s wr, 7.26k op/s 2022-09-07T21:44:11.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:11 smithi047 ceph-mon[90087]: pgmap v96: 129 pgs: 129 active+clean; 17 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 158 MiB/s wr, 10.28k op/s 2022-09-07T21:44:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:11 smithi055 ceph-mon[97033]: pgmap v96: 129 pgs: 129 active+clean; 17 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 158 MiB/s wr, 10.28k op/s 2022-09-07T21:44:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:11 smithi059 ceph-mon[97450]: pgmap v96: 129 pgs: 129 active+clean; 17 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 158 MiB/s wr, 10.28k op/s 2022-09-07T21:44:13.540 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:13 smithi059 ceph-mon[97450]: pgmap v97: 129 pgs: 129 active+clean; 17 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 135 MiB/s wr, 8.36k op/s 2022-09-07T21:44:13.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:13 smithi047 ceph-mon[90087]: pgmap v97: 129 pgs: 129 active+clean; 17 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 135 MiB/s wr, 8.36k op/s 2022-09-07T21:44:13.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:13 smithi055 ceph-mon[97033]: pgmap v97: 129 pgs: 129 active+clean; 17 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 135 MiB/s wr, 8.36k op/s 2022-09-07T21:44:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:14 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:44:14.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:14 smithi047 ceph-mon[90087]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:44:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:14 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:44:15.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:15 smithi047 ceph-mon[90087]: pgmap v98: 129 pgs: 129 active+clean; 17 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 128 MiB/s wr, 7.81k op/s 2022-09-07T21:44:15.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:15 smithi055 ceph-mon[97033]: pgmap v98: 129 pgs: 129 active+clean; 17 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 128 MiB/s wr, 7.81k op/s 2022-09-07T21:44:15.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:15 smithi059 ceph-mon[97450]: pgmap v98: 129 pgs: 129 active+clean; 17 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 128 MiB/s wr, 7.81k op/s 2022-09-07T21:44:17.664 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:17 smithi047 ceph-mon[90087]: pgmap v99: 129 pgs: 129 active+clean; 17 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 165 MiB/s wr, 10.64k op/s 2022-09-07T21:44:17.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:17 smithi055 ceph-mon[97033]: pgmap v99: 129 pgs: 129 active+clean; 17 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 165 MiB/s wr, 10.64k op/s 2022-09-07T21:44:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:17 smithi059 ceph-mon[97450]: pgmap v99: 129 pgs: 129 active+clean; 17 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 165 MiB/s wr, 10.64k op/s 2022-09-07T21:44:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:19 smithi055 ceph-mon[97033]: pgmap v100: 129 pgs: 129 active+clean; 17 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 128 MiB/s wr, 8.69k op/s 2022-09-07T21:44:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:19 smithi059 ceph-mon[97450]: pgmap v100: 129 pgs: 129 active+clean; 17 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 128 MiB/s wr, 8.69k op/s 2022-09-07T21:44:19.910 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:19 smithi047 ceph-mon[90087]: pgmap v100: 129 pgs: 129 active+clean; 17 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 128 MiB/s wr, 8.69k op/s 2022-09-07T21:44:20.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:20 smithi059 ceph-mon[97450]: pgmap v101: 129 pgs: 129 active+clean; 18 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 154 MiB/s wr, 11.37k op/s 2022-09-07T21:44:20.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:20 smithi047 ceph-mon[90087]: pgmap v101: 129 pgs: 129 active+clean; 18 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 154 MiB/s wr, 11.37k op/s 2022-09-07T21:44:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:20 smithi055 ceph-mon[97033]: pgmap v101: 129 pgs: 129 active+clean; 18 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 154 MiB/s wr, 11.37k op/s 2022-09-07T21:44:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:23 smithi055 ceph-mon[97033]: pgmap v102: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 107 MiB/s wr, 8.38k op/s 2022-09-07T21:44:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:44:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:44:23.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:44:23.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:23 smithi059 ceph-mon[97450]: pgmap v102: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 107 MiB/s wr, 8.38k op/s 2022-09-07T21:44:23.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:44:23.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:44:23.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:44:23.412 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:23 smithi047 ceph-mon[90087]: pgmap v102: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 107 MiB/s wr, 8.38k op/s 2022-09-07T21:44:23.412 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:23 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:44:23.413 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:23 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:44:23.413 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:23 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:44:25.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:25 smithi047 ceph-mon[90087]: pgmap v103: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 97 MiB/s wr, 7.89k op/s 2022-09-07T21:44:25.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:25 smithi055 ceph-mon[97033]: pgmap v103: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 97 MiB/s wr, 7.89k op/s 2022-09-07T21:44:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:25 smithi059 ceph-mon[97450]: pgmap v103: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 97 MiB/s wr, 7.89k op/s 2022-09-07T21:44:26.215 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:26 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:44:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:26 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:44:26.661 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:26 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:44:27.665 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:27 smithi047 ceph-mon[90087]: pgmap v104: 129 pgs: 129 active+clean; 18 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 134 MiB/s wr, 10.99k op/s 2022-09-07T21:44:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:27 smithi055 ceph-mon[97033]: pgmap v104: 129 pgs: 129 active+clean; 18 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 134 MiB/s wr, 10.99k op/s 2022-09-07T21:44:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:27 smithi059 ceph-mon[97450]: pgmap v104: 129 pgs: 129 active+clean; 18 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 134 MiB/s wr, 10.99k op/s 2022-09-07T21:44:29.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:29 smithi047 ceph-mon[90087]: pgmap v105: 129 pgs: 129 active+clean; 18 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 100 MiB/s wr, 8.48k op/s 2022-09-07T21:44:29.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:29 smithi055 ceph-mon[97033]: pgmap v105: 129 pgs: 129 active+clean; 18 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 100 MiB/s wr, 8.48k op/s 2022-09-07T21:44:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:29 smithi059 ceph-mon[97450]: pgmap v105: 129 pgs: 129 active+clean; 18 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 100 MiB/s wr, 8.48k op/s 2022-09-07T21:44:31.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:31 smithi047 ceph-mon[90087]: pgmap v106: 129 pgs: 129 active+clean; 19 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 130 MiB/s wr, 10.69k op/s 2022-09-07T21:44:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:31 smithi055 ceph-mon[97033]: pgmap v106: 129 pgs: 129 active+clean; 19 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 130 MiB/s wr, 10.69k op/s 2022-09-07T21:44:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:31 smithi059 ceph-mon[97450]: pgmap v106: 129 pgs: 129 active+clean; 19 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 130 MiB/s wr, 10.69k op/s 2022-09-07T21:44:33.660 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:33 smithi047 ceph-mon[90087]: pgmap v107: 129 pgs: 129 active+clean; 19 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 99 MiB/s wr, 7.79k op/s 2022-09-07T21:44:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:33 smithi055 ceph-mon[97033]: pgmap v107: 129 pgs: 129 active+clean; 19 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 99 MiB/s wr, 7.79k op/s 2022-09-07T21:44:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:33 smithi059 ceph-mon[97450]: pgmap v107: 129 pgs: 129 active+clean; 19 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 99 MiB/s wr, 7.79k op/s 2022-09-07T21:44:35.662 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:35 smithi047 ceph-mon[90087]: pgmap v108: 129 pgs: 129 active+clean; 19 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 96 MiB/s wr, 7.52k op/s 2022-09-07T21:44:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:35 smithi055 ceph-mon[97033]: pgmap v108: 129 pgs: 129 active+clean; 19 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 96 MiB/s wr, 7.52k op/s 2022-09-07T21:44:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:35 smithi059 ceph-mon[97450]: pgmap v108: 129 pgs: 129 active+clean; 19 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 96 MiB/s wr, 7.52k op/s 2022-09-07T21:44:37.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:37 smithi047 ceph-mon[90087]: pgmap v109: 129 pgs: 129 active+clean; 19 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 125 MiB/s wr, 9.47k op/s 2022-09-07T21:44:37.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:37 smithi055 ceph-mon[97033]: pgmap v109: 129 pgs: 129 active+clean; 19 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 125 MiB/s wr, 9.47k op/s 2022-09-07T21:44:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:37 smithi059 ceph-mon[97450]: pgmap v109: 129 pgs: 129 active+clean; 19 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 125 MiB/s wr, 9.47k op/s 2022-09-07T21:44:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:39 smithi055 ceph-mon[97033]: pgmap v110: 129 pgs: 129 active+clean; 19 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 91 MiB/s wr, 6.72k op/s 2022-09-07T21:44:39.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:39 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:44:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:39 smithi059 ceph-mon[97450]: pgmap v110: 129 pgs: 129 active+clean; 19 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 91 MiB/s wr, 6.72k op/s 2022-09-07T21:44:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:39 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:44:39.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:39 smithi047 ceph-mon[90087]: pgmap v110: 129 pgs: 129 active+clean; 19 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 91 MiB/s wr, 6.72k op/s 2022-09-07T21:44:39.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:39 smithi047 ceph-mon[90087]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:44:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:40 smithi059 ceph-mon[97450]: pgmap v111: 129 pgs: 129 active+clean; 19 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 114 MiB/s wr, 8.37k op/s 2022-09-07T21:44:40.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:40 smithi047 ceph-mon[90087]: pgmap v111: 129 pgs: 129 active+clean; 19 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 114 MiB/s wr, 8.37k op/s 2022-09-07T21:44:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:40 smithi055 ceph-mon[97033]: pgmap v111: 129 pgs: 129 active+clean; 19 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 114 MiB/s wr, 8.37k op/s 2022-09-07T21:44:43.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:43 smithi047 ceph-mon[90087]: pgmap v112: 129 pgs: 129 active+clean; 19 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 74 MiB/s wr, 5.43k op/s 2022-09-07T21:44:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:43 smithi055 ceph-mon[97033]: pgmap v112: 129 pgs: 129 active+clean; 19 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 74 MiB/s wr, 5.43k op/s 2022-09-07T21:44:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:43 smithi059 ceph-mon[97450]: pgmap v112: 129 pgs: 129 active+clean; 19 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 74 MiB/s wr, 5.43k op/s 2022-09-07T21:44:45.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:45 smithi047 ceph-mon[90087]: pgmap v113: 129 pgs: 129 active+clean; 19 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 77 MiB/s wr, 5.56k op/s 2022-09-07T21:44:45.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:45 smithi055 ceph-mon[97033]: pgmap v113: 129 pgs: 129 active+clean; 19 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 77 MiB/s wr, 5.56k op/s 2022-09-07T21:44:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:45 smithi059 ceph-mon[97450]: pgmap v113: 129 pgs: 129 active+clean; 19 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 77 MiB/s wr, 5.56k op/s 2022-09-07T21:44:47.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:47 smithi047 ceph-mon[90087]: pgmap v114: 129 pgs: 129 active+clean; 20 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 113 MiB/s wr, 7.94k op/s 2022-09-07T21:44:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:47 smithi055 ceph-mon[97033]: pgmap v114: 129 pgs: 129 active+clean; 20 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 113 MiB/s wr, 7.94k op/s 2022-09-07T21:44:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:47 smithi059 ceph-mon[97450]: pgmap v114: 129 pgs: 129 active+clean; 20 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 113 MiB/s wr, 7.94k op/s 2022-09-07T21:44:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:49 smithi055 ceph-mon[97033]: pgmap v115: 129 pgs: 129 active+clean; 20 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 83 MiB/s wr, 5.90k op/s 2022-09-07T21:44:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:49 smithi059 ceph-mon[97450]: pgmap v115: 129 pgs: 129 active+clean; 20 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 83 MiB/s wr, 5.90k op/s 2022-09-07T21:44:49.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:49 smithi047 ceph-mon[90087]: pgmap v115: 129 pgs: 129 active+clean; 20 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 83 MiB/s wr, 5.90k op/s 2022-09-07T21:44:51.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:51 smithi047 ceph-mon[90087]: pgmap v116: 129 pgs: 129 active+clean; 20 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 107 MiB/s wr, 7.91k op/s 2022-09-07T21:44:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:51 smithi055 ceph-mon[97033]: pgmap v116: 129 pgs: 129 active+clean; 20 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 107 MiB/s wr, 7.91k op/s 2022-09-07T21:44:51.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:51 smithi059 ceph-mon[97450]: pgmap v116: 129 pgs: 129 active+clean; 20 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 107 MiB/s wr, 7.91k op/s 2022-09-07T21:44:53.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:53 smithi047 ceph-mon[90087]: pgmap v117: 129 pgs: 129 active+clean; 20 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 81 MiB/s wr, 5.92k op/s 2022-09-07T21:44:53.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:53 smithi055 ceph-mon[97033]: pgmap v117: 129 pgs: 129 active+clean; 20 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 81 MiB/s wr, 5.92k op/s 2022-09-07T21:44:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:53 smithi059 ceph-mon[97450]: pgmap v117: 129 pgs: 129 active+clean; 20 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 81 MiB/s wr, 5.92k op/s 2022-09-07T21:44:55.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:55 smithi047 ceph-mon[90087]: pgmap v118: 129 pgs: 129 active+clean; 20 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 80 MiB/s wr, 5.82k op/s 2022-09-07T21:44:55.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:55 smithi055 ceph-mon[97033]: pgmap v118: 129 pgs: 129 active+clean; 20 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 80 MiB/s wr, 5.82k op/s 2022-09-07T21:44:55.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:55 smithi059 ceph-mon[97450]: pgmap v118: 129 pgs: 129 active+clean; 20 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 80 MiB/s wr, 5.82k op/s 2022-09-07T21:44:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:57 smithi055 ceph-mon[97033]: pgmap v119: 129 pgs: 129 active+clean; 20 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 118 MiB/s wr, 9.25k op/s 2022-09-07T21:44:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:57 smithi059 ceph-mon[97450]: pgmap v119: 129 pgs: 129 active+clean; 20 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 118 MiB/s wr, 9.25k op/s 2022-09-07T21:44:57.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:57 smithi047 ceph-mon[90087]: pgmap v119: 129 pgs: 129 active+clean; 20 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 118 MiB/s wr, 9.25k op/s 2022-09-07T21:44:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:44:59 smithi055 ceph-mon[97033]: pgmap v120: 129 pgs: 129 active+clean; 20 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 84 MiB/s wr, 7.08k op/s 2022-09-07T21:44:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:44:59 smithi059 ceph-mon[97450]: pgmap v120: 129 pgs: 129 active+clean; 20 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 84 MiB/s wr, 7.08k op/s 2022-09-07T21:44:59.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:44:59 smithi047 ceph-mon[90087]: pgmap v120: 129 pgs: 129 active+clean; 20 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 84 MiB/s wr, 7.08k op/s 2022-09-07T21:45:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:01 smithi055 ceph-mon[97033]: pgmap v121: 129 pgs: 129 active+clean; 21 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 MiB/s wr, 8.82k op/s 2022-09-07T21:45:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:01 smithi059 ceph-mon[97450]: pgmap v121: 129 pgs: 129 active+clean; 21 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 MiB/s wr, 8.82k op/s 2022-09-07T21:45:01.913 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:01 smithi047 ceph-mon[90087]: pgmap v121: 129 pgs: 129 active+clean; 21 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 MiB/s wr, 8.82k op/s 2022-09-07T21:45:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:03 smithi055 ceph-mon[97033]: pgmap v122: 129 pgs: 129 active+clean; 21 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 MiB/s wr, 6.87k op/s 2022-09-07T21:45:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:03 smithi059 ceph-mon[97450]: pgmap v122: 129 pgs: 129 active+clean; 21 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 MiB/s wr, 6.87k op/s 2022-09-07T21:45:03.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:03 smithi047 ceph-mon[90087]: pgmap v122: 129 pgs: 129 active+clean; 21 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 MiB/s wr, 6.87k op/s 2022-09-07T21:45:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:04 smithi059 ceph-mon[97450]: pgmap v123: 129 pgs: 129 active+clean; 21 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 MiB/s wr, 6.87k op/s 2022-09-07T21:45:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:45:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:45:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:04 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:45:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:04 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:45:04.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:04 smithi047 ceph-mon[90087]: pgmap v123: 129 pgs: 129 active+clean; 21 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 MiB/s wr, 6.87k op/s 2022-09-07T21:45:04.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:45:04.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:45:04.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:04 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:45:04.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:04 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:45:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:04 smithi055 ceph-mon[97033]: pgmap v123: 129 pgs: 129 active+clean; 21 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 MiB/s wr, 6.87k op/s 2022-09-07T21:45:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:45:04.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:45:04.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:04 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:45:04.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:04 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:45:07.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:07 smithi047 ceph-mon[90087]: pgmap v124: 129 pgs: 129 active+clean; 21 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 99 MiB/s wr, 8.65k op/s 2022-09-07T21:45:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:07 smithi055 ceph-mon[97033]: pgmap v124: 129 pgs: 129 active+clean; 21 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 99 MiB/s wr, 8.65k op/s 2022-09-07T21:45:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:07 smithi059 ceph-mon[97450]: pgmap v124: 129 pgs: 129 active+clean; 21 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 99 MiB/s wr, 8.65k op/s 2022-09-07T21:45:09.407 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:09 smithi047 ceph-mon[90087]: pgmap v125: 129 pgs: 129 active+clean; 21 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 54 MiB/s wr, 4.75k op/s 2022-09-07T21:45:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:09 smithi055 ceph-mon[97033]: pgmap v125: 129 pgs: 129 active+clean; 21 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 54 MiB/s wr, 4.75k op/s 2022-09-07T21:45:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:09 smithi059 ceph-mon[97450]: pgmap v125: 129 pgs: 129 active+clean; 21 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 54 MiB/s wr, 4.75k op/s 2022-09-07T21:45:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:11 smithi055 ceph-mon[97033]: pgmap v126: 129 pgs: 129 active+clean; 21 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 57 MiB/s wr, 5.20k op/s 2022-09-07T21:45:11.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:11 smithi059 ceph-mon[97450]: pgmap v126: 129 pgs: 129 active+clean; 21 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 57 MiB/s wr, 5.20k op/s 2022-09-07T21:45:11.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:11 smithi047 ceph-mon[90087]: pgmap v126: 129 pgs: 129 active+clean; 21 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 57 MiB/s wr, 5.20k op/s 2022-09-07T21:45:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:13 smithi055 ceph-mon[97033]: pgmap v127: 129 pgs: 129 active+clean; 21 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 50 MiB/s wr, 4.94k op/s 2022-09-07T21:45:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:13 smithi059 ceph-mon[97450]: pgmap v127: 129 pgs: 129 active+clean; 21 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 50 MiB/s wr, 4.94k op/s 2022-09-07T21:45:13.665 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:13 smithi047 ceph-mon[90087]: pgmap v127: 129 pgs: 129 active+clean; 21 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 50 MiB/s wr, 4.94k op/s 2022-09-07T21:45:15.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:15 smithi059 ceph-mon[97450]: pgmap v128: 129 pgs: 129 active+clean; 21 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 45 MiB/s wr, 4.41k op/s 2022-09-07T21:45:15.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:15 smithi047 ceph-mon[90087]: pgmap v128: 129 pgs: 129 active+clean; 21 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 45 MiB/s wr, 4.41k op/s 2022-09-07T21:45:15.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:15 smithi055 ceph-mon[97033]: pgmap v128: 129 pgs: 129 active+clean; 21 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 45 MiB/s wr, 4.41k op/s 2022-09-07T21:45:17.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:17 smithi047 ceph-mon[90087]: pgmap v129: 129 pgs: 129 active+clean; 21 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 61 MiB/s wr, 6.19k op/s 2022-09-07T21:45:17.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:17 smithi055 ceph-mon[97033]: pgmap v129: 129 pgs: 129 active+clean; 21 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 61 MiB/s wr, 6.19k op/s 2022-09-07T21:45:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:17 smithi059 ceph-mon[97450]: pgmap v129: 129 pgs: 129 active+clean; 21 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 61 MiB/s wr, 6.19k op/s 2022-09-07T21:45:19.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:19 smithi047 ceph-mon[90087]: pgmap v130: 129 pgs: 129 active+clean; 21 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 38 MiB/s wr, 4.41k op/s 2022-09-07T21:45:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:19 smithi055 ceph-mon[97033]: pgmap v130: 129 pgs: 129 active+clean; 21 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 38 MiB/s wr, 4.41k op/s 2022-09-07T21:45:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:19 smithi059 ceph-mon[97450]: pgmap v130: 129 pgs: 129 active+clean; 21 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 38 MiB/s wr, 4.41k op/s 2022-09-07T21:45:21.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:21 smithi047 ceph-mon[90087]: pgmap v131: 129 pgs: 129 active+clean; 21 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 45 MiB/s wr, 5.24k op/s 2022-09-07T21:45:21.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:21 smithi055 ceph-mon[97033]: pgmap v131: 129 pgs: 129 active+clean; 21 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 45 MiB/s wr, 5.24k op/s 2022-09-07T21:45:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:21 smithi059 ceph-mon[97450]: pgmap v131: 129 pgs: 129 active+clean; 21 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 45 MiB/s wr, 5.24k op/s 2022-09-07T21:45:23.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:23 smithi055 ceph-mon[97033]: pgmap v132: 129 pgs: 129 active+clean; 21 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 MiB/s wr, 6.55k op/s 2022-09-07T21:45:23.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:23 smithi059 ceph-mon[97450]: pgmap v132: 129 pgs: 129 active+clean; 21 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 MiB/s wr, 6.55k op/s 2022-09-07T21:45:23.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:23 smithi047 ceph-mon[90087]: pgmap v132: 129 pgs: 129 active+clean; 21 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 MiB/s wr, 6.55k op/s 2022-09-07T21:45:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:25 smithi059 ceph-mon[97450]: pgmap v133: 129 pgs: 129 active+clean; 21 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 40 MiB/s wr, 5.13k op/s 2022-09-07T21:45:25.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:25 smithi047 ceph-mon[90087]: pgmap v133: 129 pgs: 129 active+clean; 21 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 40 MiB/s wr, 5.13k op/s 2022-09-07T21:45:25.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:25 smithi055 ceph-mon[97033]: pgmap v133: 129 pgs: 129 active+clean; 21 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 40 MiB/s wr, 5.13k op/s 2022-09-07T21:45:26.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:26 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:45:26.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:45:26.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:45:26.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:45:26.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:26 smithi047 ceph-mon[90087]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:45:26.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:26 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:45:26.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:26 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:45:26.909 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:26 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:45:26.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:26 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:45:26.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:45:26.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:45:26.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:45:27.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:27 smithi047 ceph-mon[90087]: pgmap v134: 129 pgs: 129 active+clean; 21 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 55 MiB/s wr, 7.61k op/s 2022-09-07T21:45:27.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:27 smithi055 ceph-mon[97033]: pgmap v134: 129 pgs: 129 active+clean; 21 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 55 MiB/s wr, 7.61k op/s 2022-09-07T21:45:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:27 smithi059 ceph-mon[97450]: pgmap v134: 129 pgs: 129 active+clean; 21 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 55 MiB/s wr, 7.61k op/s 2022-09-07T21:45:28.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:28 smithi055 ceph-mon[97033]: pgmap v135: 129 pgs: 129 active+clean; 21 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 39 MiB/s wr, 5.83k op/s 2022-09-07T21:45:29.061 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:28 smithi059 ceph-mon[97450]: pgmap v135: 129 pgs: 129 active+clean; 21 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 39 MiB/s wr, 5.83k op/s 2022-09-07T21:45:29.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:28 smithi047 ceph-mon[90087]: pgmap v135: 129 pgs: 129 active+clean; 21 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 39 MiB/s wr, 5.83k op/s 2022-09-07T21:45:31.407 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:31 smithi047 ceph-mon[90087]: pgmap v136: 129 pgs: 129 active+clean; 21 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 41 MiB/s wr, 6.23k op/s 2022-09-07T21:45:31.408 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:31 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:45:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:31 smithi055 ceph-mon[97033]: pgmap v136: 129 pgs: 129 active+clean; 21 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 41 MiB/s wr, 6.23k op/s 2022-09-07T21:45:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:31 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:45:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:31 smithi059 ceph-mon[97450]: pgmap v136: 129 pgs: 129 active+clean; 21 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 41 MiB/s wr, 6.23k op/s 2022-09-07T21:45:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:31 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:45:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:32 smithi059 ceph-mon[97450]: pgmap v137: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 51 MiB/s wr, 7.95k op/s 2022-09-07T21:45:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:32 smithi059 ceph-mon[97450]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2022-09-07T21:45:32.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:32 smithi047 ceph-mon[90087]: pgmap v137: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 51 MiB/s wr, 7.95k op/s 2022-09-07T21:45:32.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:32 smithi047 ceph-mon[90087]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2022-09-07T21:45:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:32 smithi055 ceph-mon[97033]: pgmap v137: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 51 MiB/s wr, 7.95k op/s 2022-09-07T21:45:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:32 smithi055 ceph-mon[97033]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2022-09-07T21:45:34.740 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:34 smithi055 ceph-mon[97033]: pgmap v138: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 36 MiB/s wr, 5.96k op/s 2022-09-07T21:45:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:34 smithi059 ceph-mon[97450]: pgmap v138: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 36 MiB/s wr, 5.96k op/s 2022-09-07T21:45:36.085 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:34 smithi047 ceph-mon[90087]: pgmap v138: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 36 MiB/s wr, 5.96k op/s 2022-09-07T21:45:36.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:36 smithi047 ceph-mon[90087]: pgmap v139: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 MiB/s wr, 6.08k op/s 2022-09-07T21:45:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:36 smithi055 ceph-mon[97033]: pgmap v139: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 MiB/s wr, 6.08k op/s 2022-09-07T21:45:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:36 smithi059 ceph-mon[97450]: pgmap v139: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 MiB/s wr, 6.08k op/s 2022-09-07T21:45:38.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:38 smithi047 ceph-mon[90087]: pgmap v140: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 MiB/s wr, 5.17k op/s 2022-09-07T21:45:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:38 smithi055 ceph-mon[97033]: pgmap v140: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 MiB/s wr, 5.17k op/s 2022-09-07T21:45:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:38 smithi059 ceph-mon[97450]: pgmap v140: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 MiB/s wr, 5.17k op/s 2022-09-07T21:45:40.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:40 smithi055 ceph-mon[97033]: pgmap v141: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 MiB/s wr, 5.10k op/s 2022-09-07T21:45:40.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:40 smithi059 ceph-mon[97450]: pgmap v141: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 MiB/s wr, 5.10k op/s 2022-09-07T21:45:40.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:40 smithi047 ceph-mon[90087]: pgmap v141: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 MiB/s wr, 5.10k op/s 2022-09-07T21:45:42.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:42 smithi059 ceph-mon[97450]: pgmap v142: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 51 MiB/s wr, 6.76k op/s 2022-09-07T21:45:42.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:42 smithi047 ceph-mon[90087]: pgmap v142: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 51 MiB/s wr, 6.76k op/s 2022-09-07T21:45:42.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:42 smithi055 ceph-mon[97033]: pgmap v142: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 51 MiB/s wr, 6.76k op/s 2022-09-07T21:45:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:44 smithi059 ceph-mon[97450]: pgmap v143: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 38 MiB/s wr, 4.64k op/s 2022-09-07T21:45:44.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:44 smithi047 ceph-mon[90087]: pgmap v143: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 38 MiB/s wr, 4.64k op/s 2022-09-07T21:45:44.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:44 smithi055 ceph-mon[97033]: pgmap v143: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 38 MiB/s wr, 4.64k op/s 2022-09-07T21:45:46.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:46 smithi047 ceph-mon[90087]: pgmap v144: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 38 MiB/s wr, 4.75k op/s 2022-09-07T21:45:46.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:46 smithi055 ceph-mon[97033]: pgmap v144: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 38 MiB/s wr, 4.75k op/s 2022-09-07T21:45:47.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:46 smithi059 ceph-mon[97450]: pgmap v144: 129 pgs: 129 active+clean; 22 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 38 MiB/s wr, 4.75k op/s 2022-09-07T21:45:48.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:48 smithi047 ceph-mon[90087]: pgmap v145: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 44 MiB/s wr, 5.73k op/s 2022-09-07T21:45:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:48 smithi055 ceph-mon[97033]: pgmap v145: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 44 MiB/s wr, 5.73k op/s 2022-09-07T21:45:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:48 smithi059 ceph-mon[97450]: pgmap v145: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 44 MiB/s wr, 5.73k op/s 2022-09-07T21:45:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:50 smithi055 ceph-mon[97033]: pgmap v146: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 33 MiB/s wr, 4.32k op/s 2022-09-07T21:45:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:50 smithi059 ceph-mon[97450]: pgmap v146: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 33 MiB/s wr, 4.32k op/s 2022-09-07T21:45:51.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:50 smithi047 ceph-mon[90087]: pgmap v146: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 33 MiB/s wr, 4.32k op/s 2022-09-07T21:45:51.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:51 smithi055 ceph-mon[97033]: pgmap v147: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 44 MiB/s wr, 5.96k op/s 2022-09-07T21:45:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:51 smithi059 ceph-mon[97450]: pgmap v147: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 44 MiB/s wr, 5.96k op/s 2022-09-07T21:45:52.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:51 smithi047 ceph-mon[90087]: pgmap v147: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 44 MiB/s wr, 5.96k op/s 2022-09-07T21:45:54.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:54 smithi055 ceph-mon[97033]: pgmap v148: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 MiB/s wr, 4.26k op/s 2022-09-07T21:45:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:54 smithi059 ceph-mon[97450]: pgmap v148: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 MiB/s wr, 4.26k op/s 2022-09-07T21:45:54.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:54 smithi047 ceph-mon[90087]: pgmap v148: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 27 MiB/s wr, 4.26k op/s 2022-09-07T21:45:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:56 smithi059 ceph-mon[97450]: pgmap v149: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 MiB/s wr, 4.21k op/s 2022-09-07T21:45:56.612 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:56 smithi047 ceph-mon[90087]: pgmap v149: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 MiB/s wr, 4.21k op/s 2022-09-07T21:45:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:56 smithi055 ceph-mon[97033]: pgmap v149: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 MiB/s wr, 4.21k op/s 2022-09-07T21:45:58.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:45:58 smithi047 ceph-mon[90087]: pgmap v150: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 39 MiB/s wr, 5.98k op/s 2022-09-07T21:45:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:45:58 smithi055 ceph-mon[97033]: pgmap v150: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 39 MiB/s wr, 5.98k op/s 2022-09-07T21:45:58.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:45:58 smithi059 ceph-mon[97450]: pgmap v150: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 39 MiB/s wr, 5.98k op/s 2022-09-07T21:46:00.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:00 smithi047 ceph-mon[90087]: pgmap v151: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 32 MiB/s wr, 4.67k op/s 2022-09-07T21:46:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:00 smithi055 ceph-mon[97033]: pgmap v151: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 32 MiB/s wr, 4.67k op/s 2022-09-07T21:46:00.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:00 smithi059 ceph-mon[97450]: pgmap v151: 129 pgs: 129 active+clean; 22 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 32 MiB/s wr, 4.67k op/s 2022-09-07T21:46:01.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:01 smithi047 ceph-mon[90087]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:46:01.739 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:01 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:46:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:01 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:46:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:02 smithi055 ceph-mon[97033]: pgmap v152: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 39 MiB/s wr, 5.99k op/s 2022-09-07T21:46:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:02 smithi059 ceph-mon[97450]: pgmap v152: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 39 MiB/s wr, 5.99k op/s 2022-09-07T21:46:02.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:02 smithi047 ceph-mon[90087]: pgmap v152: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 39 MiB/s wr, 5.99k op/s 2022-09-07T21:46:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:04 smithi055 ceph-mon[97033]: pgmap v153: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 29 MiB/s wr, 4.75k op/s 2022-09-07T21:46:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:46:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:46:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:04 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:46:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:04 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:46:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:04 smithi059 ceph-mon[97450]: pgmap v153: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 29 MiB/s wr, 4.75k op/s 2022-09-07T21:46:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:46:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:46:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:04 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:46:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:04 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:46:04.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:04 smithi047 ceph-mon[90087]: pgmap v153: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 29 MiB/s wr, 4.75k op/s 2022-09-07T21:46:04.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:46:04.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:04 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:46:04.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:04 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:46:04.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:04 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:46:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:06 smithi059 ceph-mon[97450]: pgmap v154: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 29 MiB/s wr, 4.68k op/s 2022-09-07T21:46:06.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:06 smithi047 ceph-mon[90087]: pgmap v154: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 29 MiB/s wr, 4.68k op/s 2022-09-07T21:46:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:06 smithi055 ceph-mon[97033]: pgmap v154: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 29 MiB/s wr, 4.68k op/s 2022-09-07T21:46:08.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:08 smithi047 ceph-mon[90087]: pgmap v155: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 35 MiB/s wr, 6.03k op/s 2022-09-07T21:46:08.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:08 smithi055 ceph-mon[97033]: pgmap v155: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 35 MiB/s wr, 6.03k op/s 2022-09-07T21:46:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:08 smithi059 ceph-mon[97450]: pgmap v155: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 35 MiB/s wr, 6.03k op/s 2022-09-07T21:46:10.018 INFO:tasks.workunit.client.0.smithi047.stdout:Wrote -1 instead of 4096 bytes. 2022-09-07T21:46:10.018 INFO:tasks.workunit.client.0.smithi047.stdout:Probably out of disk space 2022-09-07T21:46:10.019 INFO:tasks.workunit.client.0.smithi047.stderr:write: Resource temporarily unavailable 2022-09-07T21:46:10.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:10 smithi047 ceph-mon[90087]: pgmap v156: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 MiB/s wr, 4.20k op/s 2022-09-07T21:46:10.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:10 smithi055 ceph-mon[97033]: pgmap v156: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 MiB/s wr, 4.20k op/s 2022-09-07T21:46:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:10 smithi059 ceph-mon[97450]: pgmap v156: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 MiB/s wr, 4.20k op/s 2022-09-07T21:46:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-07T21:46:12.162 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2022-09-07T21:46:12.163 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2022-09-07T21:46:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:12 smithi055 ceph-mon[97033]: pgmap v157: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 30 MiB/s wr, 5.95k op/s 2022-09-07T21:46:13.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:12 smithi059 ceph-mon[97450]: pgmap v157: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 30 MiB/s wr, 5.95k op/s 2022-09-07T21:46:13.160 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:12 smithi047 ceph-mon[90087]: pgmap v157: 129 pgs: 129 active+clean; 22 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 30 MiB/s wr, 5.95k op/s 2022-09-07T21:46:15.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:14 smithi059 ceph-mon[97450]: pgmap v158: 129 pgs: 129 active+clean; 22 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 23 MiB/s wr, 4.65k op/s 2022-09-07T21:46:15.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:14 smithi047 ceph-mon[90087]: pgmap v158: 129 pgs: 129 active+clean; 22 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 23 MiB/s wr, 4.65k op/s 2022-09-07T21:46:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:14 smithi055 ceph-mon[97033]: pgmap v158: 129 pgs: 129 active+clean; 22 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 23 MiB/s wr, 4.65k op/s 2022-09-07T21:46:16.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:15 smithi047 ceph-mon[90087]: pgmap v159: 129 pgs: 129 active+clean; 22 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 MiB/s wr, 4.74k op/s 2022-09-07T21:46:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:15 smithi055 ceph-mon[97033]: pgmap v159: 129 pgs: 129 active+clean; 22 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 MiB/s wr, 4.74k op/s 2022-09-07T21:46:16.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:15 smithi059 ceph-mon[97450]: pgmap v159: 129 pgs: 129 active+clean; 22 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 MiB/s wr, 4.74k op/s 2022-09-07T21:46:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:18 smithi055 ceph-mon[97033]: pgmap v160: 129 pgs: 129 active+clean; 22 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 MiB/s wr, 6.01k op/s 2022-09-07T21:46:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:18 smithi059 ceph-mon[97450]: pgmap v160: 129 pgs: 129 active+clean; 22 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 MiB/s wr, 6.01k op/s 2022-09-07T21:46:18.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:18 smithi047 ceph-mon[90087]: pgmap v160: 129 pgs: 129 active+clean; 22 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 MiB/s wr, 6.01k op/s 2022-09-07T21:46:20.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:20 smithi047 ceph-mon[90087]: pgmap v161: 129 pgs: 129 active+clean; 22 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 MiB/s wr, 4.66k op/s 2022-09-07T21:46:20.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:20 smithi055 ceph-mon[97033]: pgmap v161: 129 pgs: 129 active+clean; 22 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 MiB/s wr, 4.66k op/s 2022-09-07T21:46:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:20 smithi059 ceph-mon[97450]: pgmap v161: 129 pgs: 129 active+clean; 22 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 MiB/s wr, 4.66k op/s 2022-09-07T21:46:22.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:22 smithi047 ceph-mon[90087]: pgmap v162: 129 pgs: 129 active+clean; 23 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 MiB/s wr, 5.99k op/s 2022-09-07T21:46:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:22 smithi055 ceph-mon[97033]: pgmap v162: 129 pgs: 129 active+clean; 23 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 MiB/s wr, 5.99k op/s 2022-09-07T21:46:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:22 smithi059 ceph-mon[97450]: pgmap v162: 129 pgs: 129 active+clean; 23 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 MiB/s wr, 5.99k op/s 2022-09-07T21:46:24.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:24 smithi047 ceph-mon[90087]: pgmap v163: 129 pgs: 129 active+clean; 23 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 MiB/s wr, 4.31k op/s 2022-09-07T21:46:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:24 smithi055 ceph-mon[97033]: pgmap v163: 129 pgs: 129 active+clean; 23 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 MiB/s wr, 4.31k op/s 2022-09-07T21:46:24.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:24 smithi059 ceph-mon[97450]: pgmap v163: 129 pgs: 129 active+clean; 23 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 MiB/s wr, 4.31k op/s 2022-09-07T21:46:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:26 smithi055 ceph-mon[97033]: pgmap v164: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 MiB/s wr, 4.66k op/s 2022-09-07T21:46:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:26 smithi059 ceph-mon[97450]: pgmap v164: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 MiB/s wr, 4.66k op/s 2022-09-07T21:46:26.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:26 smithi047 ceph-mon[90087]: pgmap v164: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 MiB/s wr, 4.66k op/s 2022-09-07T21:46:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:28 smithi055 ceph-mon[97033]: pgmap v165: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 24 MiB/s wr, 5.38k op/s 2022-09-07T21:46:28.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:28 smithi059 ceph-mon[97450]: pgmap v165: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 24 MiB/s wr, 5.38k op/s 2022-09-07T21:46:28.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:28 smithi047 ceph-mon[90087]: pgmap v165: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 24 MiB/s wr, 5.38k op/s 2022-09-07T21:46:30.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:30 smithi059 ceph-mon[97450]: pgmap v166: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 4.08k op/s 2022-09-07T21:46:30.912 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:30 smithi047 ceph-mon[90087]: pgmap v166: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 4.08k op/s 2022-09-07T21:46:30.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:30 smithi055 ceph-mon[97033]: pgmap v166: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 4.08k op/s 2022-09-07T21:46:31.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:31 smithi047 ceph-mon[90087]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:46:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:31 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:46:32.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:31 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:46:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:32 smithi059 ceph-mon[97450]: pgmap v167: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 MiB/s wr, 4.83k op/s 2022-09-07T21:46:32.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:46:32.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:46:32.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:46:32.943 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:32 smithi055 ceph-mon[97033]: pgmap v167: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 MiB/s wr, 4.83k op/s 2022-09-07T21:46:32.943 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:46:32.943 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:46:32.944 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:46:33.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:32 smithi047 ceph-mon[90087]: pgmap v167: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 MiB/s wr, 4.83k op/s 2022-09-07T21:46:33.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:32 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:46:33.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:32 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:46:33.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:32 smithi047 ceph-mon[90087]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:46:34.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:34 smithi055 ceph-mon[97033]: pgmap v168: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 18 MiB/s wr, 4.06k op/s 2022-09-07T21:46:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:34 smithi059 ceph-mon[97450]: pgmap v168: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 18 MiB/s wr, 4.06k op/s 2022-09-07T21:46:35.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:34 smithi047 ceph-mon[90087]: pgmap v168: 129 pgs: 129 active+clean; 23 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 18 MiB/s wr, 4.06k op/s 2022-09-07T21:46:37.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:36 smithi059 ceph-mon[97450]: pgmap v169: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 4.44k op/s 2022-09-07T21:46:37.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:36 smithi047 ceph-mon[90087]: pgmap v169: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 4.44k op/s 2022-09-07T21:46:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:36 smithi055 ceph-mon[97033]: pgmap v169: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 4.44k op/s 2022-09-07T21:46:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:38 smithi059 ceph-mon[97450]: pgmap v170: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 MiB/s wr, 4.97k op/s 2022-09-07T21:46:39.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:38 smithi059 ceph-mon[97450]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:46:39.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:38 smithi047 ceph-mon[90087]: pgmap v170: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 MiB/s wr, 4.97k op/s 2022-09-07T21:46:39.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:38 smithi047 ceph-mon[90087]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:46:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:38 smithi055 ceph-mon[97033]: pgmap v170: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 MiB/s wr, 4.97k op/s 2022-09-07T21:46:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:38 smithi055 ceph-mon[97033]: from='mgr.25354 ' entity='mgr.x' 2022-09-07T21:46:40.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:39 smithi059 ceph-mon[97450]: pgmap v171: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 MiB/s wr, 3.93k op/s 2022-09-07T21:46:40.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:39 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2022-09-07T21:46:40.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:39 smithi059 ceph-mon[97450]: Cluster is now healthy 2022-09-07T21:46:40.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:39 smithi047 ceph-mon[90087]: pgmap v171: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 MiB/s wr, 3.93k op/s 2022-09-07T21:46:40.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:39 smithi047 ceph-mon[90087]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2022-09-07T21:46:40.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:39 smithi047 ceph-mon[90087]: Cluster is now healthy 2022-09-07T21:46:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:39 smithi055 ceph-mon[97033]: pgmap v171: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 MiB/s wr, 3.93k op/s 2022-09-07T21:46:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:39 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2022-09-07T21:46:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:39 smithi055 ceph-mon[97033]: Cluster is now healthy 2022-09-07T21:46:42.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:41 smithi047 ceph-mon[90087]: pgmap v172: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 MiB/s wr, 4.27k op/s 2022-09-07T21:46:42.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:41 smithi047 ceph-mon[90087]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:46:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:41 smithi055 ceph-mon[97033]: pgmap v172: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 MiB/s wr, 4.27k op/s 2022-09-07T21:46:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:41 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:46:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:41 smithi059 ceph-mon[97450]: pgmap v172: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 MiB/s wr, 4.27k op/s 2022-09-07T21:46:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:41 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:46:44.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:43 smithi047 ceph-mon[90087]: pgmap v173: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 4.08k op/s 2022-09-07T21:46:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:43 smithi055 ceph-mon[97033]: pgmap v173: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 4.08k op/s 2022-09-07T21:46:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:43 smithi059 ceph-mon[97450]: pgmap v173: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 4.08k op/s 2022-09-07T21:46:45.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:45 smithi047 ceph-mon[90087]: pgmap v174: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 3.29k op/s 2022-09-07T21:46:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:45 smithi055 ceph-mon[97033]: pgmap v174: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 3.29k op/s 2022-09-07T21:46:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:45 smithi059 ceph-mon[97450]: pgmap v174: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 3.29k op/s 2022-09-07T21:46:47.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:47 smithi047 ceph-mon[90087]: pgmap v175: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 2.83k op/s 2022-09-07T21:46:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:47 smithi055 ceph-mon[97033]: pgmap v175: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 2.83k op/s 2022-09-07T21:46:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:47 smithi059 ceph-mon[97450]: pgmap v175: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 2.83k op/s 2022-09-07T21:46:49.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:49 smithi047 ceph-mon[90087]: pgmap v176: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.9 MiB/s wr, 2.27k op/s 2022-09-07T21:46:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:49 smithi055 ceph-mon[97033]: pgmap v176: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.9 MiB/s wr, 2.27k op/s 2022-09-07T21:46:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:49 smithi059 ceph-mon[97450]: pgmap v176: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.9 MiB/s wr, 2.27k op/s 2022-09-07T21:46:51.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:51 smithi055 ceph-mon[97033]: pgmap v177: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 10 MiB/s wr, 2.31k op/s 2022-09-07T21:46:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:51 smithi059 ceph-mon[97450]: pgmap v177: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 10 MiB/s wr, 2.31k op/s 2022-09-07T21:46:52.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:51 smithi047 ceph-mon[90087]: pgmap v177: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 10 MiB/s wr, 2.31k op/s 2022-09-07T21:46:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:53 smithi055 ceph-mon[97033]: pgmap v178: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 2.67k op/s 2022-09-07T21:46:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:53 smithi059 ceph-mon[97450]: pgmap v178: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 2.67k op/s 2022-09-07T21:46:54.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:53 smithi047 ceph-mon[90087]: pgmap v178: 129 pgs: 129 active+clean; 23 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 2.67k op/s 2022-09-07T21:46:55.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:55 smithi047 ceph-mon[90087]: pgmap v179: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.5 MiB/s wr, 1.95k op/s 2022-09-07T21:46:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:55 smithi055 ceph-mon[97033]: pgmap v179: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.5 MiB/s wr, 1.95k op/s 2022-09-07T21:46:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:55 smithi059 ceph-mon[97450]: pgmap v179: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.5 MiB/s wr, 1.95k op/s 2022-09-07T21:46:57.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:57 smithi047 ceph-mon[90087]: pgmap v180: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 2.64k op/s 2022-09-07T21:46:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:57 smithi055 ceph-mon[97033]: pgmap v180: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 2.64k op/s 2022-09-07T21:46:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:57 smithi059 ceph-mon[97450]: pgmap v180: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 2.64k op/s 2022-09-07T21:46:59.244 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/run_tasks.py", line 103, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/run_tasks.py", line 82, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d/qa/tasks/workunit.py", line 148, in task cleanup=cleanup) File "/home/teuthworker/src/github.com_batrick_ceph_75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d/qa/tasks/workunit.py", line 298, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d/qa/tasks/workunit.py", line 427, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/remote.py", line 510, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi047 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2022-09-07T21:46:59.377 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4e643983750a4889a84984af6629283a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/run_tasks.py", line 103, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/run_tasks.py", line 82, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d/qa/tasks/workunit.py", line 148, in task cleanup=cleanup) File "/home/teuthworker/src/github.com_batrick_ceph_75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d/qa/tasks/workunit.py", line 298, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d/qa/tasks/workunit.py", line 427, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/remote.py", line 510, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi047 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=75ae46d5f37dd4924c2d5cff1f4a7ab89ec92d1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2022-09-07T21:46:59.381 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2022-09-07T21:46:59.394 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2022-09-07T21:46:59.395 INFO:teuthology.task.exec:Executing custom commands... 2022-09-07T21:46:59.395 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi047.front.sepia.ceph.com 2022-09-07T21:46:59.396 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2022-09-07T21:46:59.513 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2022-09-07T21:46:59.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:46:59 smithi047 ceph-mon[90087]: pgmap v181: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 MiB/s wr, 3.00k op/s 2022-09-07T21:46:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:46:59 smithi055 ceph-mon[97033]: pgmap v181: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 MiB/s wr, 3.00k op/s 2022-09-07T21:47:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:46:59 smithi059 ceph-mon[97450]: pgmap v181: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 MiB/s wr, 3.00k op/s 2022-09-07T21:47:04.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 21:47:04 smithi047 ceph-mon[90087]: pgmap v182: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 2.40k op/s 2022-09-07T21:47:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:04 smithi055 ceph-mon[97033]: pgmap v182: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 2.40k op/s 2022-09-07T21:47:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:04 smithi059 ceph-mon[97450]: pgmap v182: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 2.40k op/s 2022-09-07T21:47:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: pgmap v183: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.0 MiB/s wr, 2.10k op/s 2022-09-07T21:47:19.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:47:19.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:47:19.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: pgmap v184: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.2 MiB/s wr, 1.45k op/s 2022-09-07T21:47:19.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: pgmap v185: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 1.36k op/s 2022-09-07T21:47:19.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: pgmap v186: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.1 MiB/s wr, 734 op/s 2022-09-07T21:47:19.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: pgmap v187: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.4 MiB/s wr, 321 op/s 2022-09-07T21:47:19.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: pgmap v188: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.1 MiB/s wr, 250 op/s 2022-09-07T21:47:19.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: mon.b calling monitor election 2022-09-07T21:47:19.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: mon.c calling monitor election 2022-09-07T21:47:19.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: pgmap v189: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 1023 KiB/s wr, 239 op/s 2022-09-07T21:47:19.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: pgmap v190: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 425 KiB/s wr, 99 op/s 2022-09-07T21:47:19.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: mon.c is new leader, mons c,b in quorum (ranks 1,2) 2022-09-07T21:47:19.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: monmap e3: 3 mons at {a=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],b=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],c=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2022-09-07T21:47:19.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:47:19.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: osdmap e94: 12 total, 12 up, 12 in 2022-09-07T21:47:19.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: mgrmap e25: x(active, since 6m), standbys: y, z 2022-09-07T21:47:19.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: Health check failed: 1/3 mons down, quorum c,b (MON_DOWN) 2022-09-07T21:47:19.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-07T21:47:19.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T21:47:19.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:19 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T21:47:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: pgmap v183: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.0 MiB/s wr, 2.10k op/s 2022-09-07T21:47:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:47:19.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:47:19.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: pgmap v184: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.2 MiB/s wr, 1.45k op/s 2022-09-07T21:47:19.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: pgmap v185: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 1.36k op/s 2022-09-07T21:47:19.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: pgmap v186: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.1 MiB/s wr, 734 op/s 2022-09-07T21:47:19.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: pgmap v187: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.4 MiB/s wr, 321 op/s 2022-09-07T21:47:19.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: pgmap v188: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.1 MiB/s wr, 250 op/s 2022-09-07T21:47:19.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: mon.b calling monitor election 2022-09-07T21:47:19.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: mon.c calling monitor election 2022-09-07T21:47:19.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: pgmap v189: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 1023 KiB/s wr, 239 op/s 2022-09-07T21:47:19.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: pgmap v190: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 425 KiB/s wr, 99 op/s 2022-09-07T21:47:19.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: mon.c is new leader, mons c,b in quorum (ranks 1,2) 2022-09-07T21:47:19.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: monmap e3: 3 mons at {a=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],b=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],c=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2022-09-07T21:47:19.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:47:19.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: osdmap e94: 12 total, 12 up, 12 in 2022-09-07T21:47:19.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: mgrmap e25: x(active, since 6m), standbys: y, z 2022-09-07T21:47:19.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: Health check failed: 1/3 mons down, quorum c,b (MON_DOWN) 2022-09-07T21:47:19.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-07T21:47:19.790 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T21:47:19.790 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:19 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T21:47:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:20 smithi055 ceph-mon[97033]: pgmap v191: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 171 KiB/s wr, 44 op/s 2022-09-07T21:47:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:20 smithi059 ceph-mon[97450]: pgmap v191: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 171 KiB/s wr, 44 op/s 2022-09-07T21:47:22.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:22 smithi055 ceph-mon[97033]: pgmap v192: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T21:47:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:22 smithi059 ceph-mon[97450]: pgmap v192: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T21:47:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:24 smithi055 ceph-mon[97033]: pgmap v193: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:24 smithi059 ceph-mon[97450]: pgmap v193: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:26 smithi055 ceph-mon[97033]: pgmap v194: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:47:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:26 smithi059 ceph-mon[97450]: pgmap v194: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:47:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:28 smithi055 ceph-mon[97033]: pgmap v195: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:28 smithi059 ceph-mon[97450]: pgmap v195: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:30 smithi055 ceph-mon[97033]: pgmap v196: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:30 smithi059 ceph-mon[97450]: pgmap v196: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:32 smithi055 ceph-mon[97033]: pgmap v197: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:47:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:32 smithi059 ceph-mon[97450]: pgmap v197: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:47:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:34 smithi055 ceph-mon[97033]: pgmap v198: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:34 smithi059 ceph-mon[97450]: pgmap v198: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:36 smithi055 ceph-mon[97033]: pgmap v199: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:47:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:36 smithi059 ceph-mon[97450]: pgmap v199: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:47:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:38 smithi055 ceph-mon[97033]: pgmap v200: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:38 smithi059 ceph-mon[97450]: pgmap v200: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:40 smithi055 ceph-mon[97033]: pgmap v201: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:40.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:47:40.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:47:40.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:47:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:40 smithi059 ceph-mon[97450]: pgmap v201: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:40.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:47:40.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:47:40.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:47:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:42 smithi055 ceph-mon[97033]: pgmap v202: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:47:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:47:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:42 smithi059 ceph-mon[97450]: pgmap v202: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:47:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:47:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:44 smithi055 ceph-mon[97033]: pgmap v203: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:44 smithi059 ceph-mon[97450]: pgmap v203: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:46 smithi055 ceph-mon[97033]: pgmap v204: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:47:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:46 smithi059 ceph-mon[97450]: pgmap v204: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:47:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:48 smithi055 ceph-mon[97033]: pgmap v205: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:48 smithi059 ceph-mon[97450]: pgmap v205: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:50 smithi055 ceph-mon[97033]: pgmap v206: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:50.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:50 smithi059 ceph-mon[97450]: pgmap v206: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:51.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:51 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:47:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:51 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:47:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:52 smithi055 ceph-mon[97033]: pgmap v207: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:47:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:52 smithi059 ceph-mon[97450]: pgmap v207: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:47:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:54 smithi055 ceph-mon[97033]: pgmap v208: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:54 smithi059 ceph-mon[97450]: pgmap v208: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:56 smithi055 ceph-mon[97033]: pgmap v209: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:47:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:56 smithi059 ceph-mon[97450]: pgmap v209: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:47:58.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:47:58 smithi055 ceph-mon[97033]: pgmap v210: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:47:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:47:58 smithi059 ceph-mon[97450]: pgmap v210: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:00 smithi055 ceph-mon[97033]: pgmap v211: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:00 smithi059 ceph-mon[97450]: pgmap v211: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:01 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:48:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:01 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2022-09-07T21:48:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:02 smithi055 ceph-mon[97033]: pgmap v212: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 853 B/s wr, 7 op/s 2022-09-07T21:48:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:02 smithi059 ceph-mon[97450]: pgmap v212: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 853 B/s wr, 7 op/s 2022-09-07T21:48:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:04 smithi055 ceph-mon[97033]: pgmap v213: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:48:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:48:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:48:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:04 smithi059 ceph-mon[97450]: pgmap v213: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:48:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:48:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:48:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:06 smithi055 ceph-mon[97033]: pgmap v214: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2022-09-07T21:48:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:06 smithi059 ceph-mon[97450]: pgmap v214: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2022-09-07T21:48:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:08 smithi055 ceph-mon[97033]: pgmap v215: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2022-09-07T21:48:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:08 smithi059 ceph-mon[97450]: pgmap v215: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2022-09-07T21:48:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:09 smithi055 ceph-mon[97033]: osdmap e95: 12 total, 12 up, 12 in 2022-09-07T21:48:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:09 smithi059 ceph-mon[97450]: osdmap e95: 12 total, 12 up, 12 in 2022-09-07T21:48:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:10 smithi055 ceph-mon[97033]: pgmap v217: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2022-09-07T21:48:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:10 smithi059 ceph-mon[97450]: pgmap v217: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2022-09-07T21:48:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:12 smithi055 ceph-mon[97033]: pgmap v218: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:48:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:12 smithi059 ceph-mon[97450]: pgmap v218: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:48:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:14 smithi055 ceph-mon[97033]: pgmap v219: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T21:48:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:14 smithi059 ceph-mon[97450]: pgmap v219: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T21:48:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:16 smithi055 ceph-mon[97033]: pgmap v220: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:48:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:16 smithi059 ceph-mon[97450]: pgmap v220: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:48:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:18 smithi055 ceph-mon[97033]: pgmap v221: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:48:18.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:18 smithi055 ceph-mon[97033]: osdmap e96: 12 total, 12 up, 12 in 2022-09-07T21:48:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:18 smithi059 ceph-mon[97450]: pgmap v221: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:48:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:18 smithi059 ceph-mon[97450]: osdmap e96: 12 total, 12 up, 12 in 2022-09-07T21:48:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:20 smithi055 ceph-mon[97033]: pgmap v223: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:48:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:20 smithi059 ceph-mon[97450]: pgmap v223: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:48:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:22 smithi055 ceph-mon[97033]: pgmap v224: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:48:22.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:22 smithi055 ceph-mon[97033]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2022-09-07T21:48:22.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:22 smithi055 ceph-mon[97033]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2022-09-07T21:48:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:22 smithi059 ceph-mon[97450]: pgmap v224: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:48:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:22 smithi059 ceph-mon[97450]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2022-09-07T21:48:22.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:22 smithi059 ceph-mon[97450]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 5 up:standby 2022-09-07T21:48:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:24 smithi055 ceph-mon[97033]: pgmap v225: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:48:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:24 smithi059 ceph-mon[97450]: pgmap v225: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:48:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:25.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:26 smithi055 ceph-mon[97033]: pgmap v226: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:48:26.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:26 smithi059 ceph-mon[97450]: pgmap v226: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:48:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:28 smithi055 ceph-mon[97033]: pgmap v227: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:48:28.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:28 smithi059 ceph-mon[97450]: pgmap v227: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:48:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:30 smithi055 ceph-mon[97033]: pgmap v228: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:30 smithi059 ceph-mon[97450]: pgmap v228: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:32 smithi055 ceph-mon[97033]: pgmap v229: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:48:32.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:32 smithi059 ceph-mon[97450]: pgmap v229: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:48:32.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:34.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:34 smithi055 ceph-mon[97033]: pgmap v230: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:34.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:34 smithi059 ceph-mon[97450]: pgmap v230: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:34.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:36 smithi055 ceph-mon[97033]: pgmap v231: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:48:36.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:36 smithi059 ceph-mon[97450]: pgmap v231: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:48:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:38 smithi055 ceph-mon[97033]: pgmap v232: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:38 smithi059 ceph-mon[97450]: pgmap v232: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:40 smithi055 ceph-mon[97033]: pgmap v233: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:40.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:40 smithi059 ceph-mon[97450]: pgmap v233: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:40.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:48:41.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:48:41.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:48:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:48:41.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:48:41.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:48:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:42 smithi055 ceph-mon[97033]: pgmap v234: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:48:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:42.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:48:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:42 smithi059 ceph-mon[97450]: pgmap v234: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:48:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:48:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:44 smithi059 ceph-mon[97450]: pgmap v235: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:44 smithi055 ceph-mon[97033]: pgmap v235: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:44.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:46 smithi059 ceph-mon[97450]: pgmap v236: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:48:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:46 smithi055 ceph-mon[97033]: pgmap v236: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:48:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:48 smithi059 ceph-mon[97450]: pgmap v237: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:48 smithi055 ceph-mon[97033]: pgmap v237: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:50.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:50 smithi059 ceph-mon[97450]: pgmap v238: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:50.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:50 smithi055 ceph-mon[97033]: pgmap v238: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:52 smithi059 ceph-mon[97450]: pgmap v239: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:48:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:52 smithi055 ceph-mon[97033]: pgmap v239: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:48:52.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:54 smithi059 ceph-mon[97450]: pgmap v240: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:54 smithi055 ceph-mon[97033]: pgmap v240: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:56 smithi055 ceph-mon[97033]: pgmap v241: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:48:56.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:56 smithi059 ceph-mon[97450]: pgmap v241: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:48:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:58 smithi055 ceph-mon[97033]: pgmap v242: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:58.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:59.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:58 smithi059 ceph-mon[97450]: pgmap v242: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:48:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:48:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:48:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:48:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:00 smithi055 ceph-mon[97033]: pgmap v243: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:49:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:00 smithi059 ceph-mon[97450]: pgmap v243: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:49:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:02 smithi055 ceph-mon[97033]: pgmap v244: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2022-09-07T21:49:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:03.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:02 smithi059 ceph-mon[97450]: pgmap v244: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2022-09-07T21:49:03.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:04 smithi055 ceph-mon[97033]: pgmap v245: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:49:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:49:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:49:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:04 smithi059 ceph-mon[97450]: pgmap v245: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:49:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:49:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:49:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:06 smithi055 ceph-mon[97033]: pgmap v246: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:49:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:06 smithi059 ceph-mon[97450]: pgmap v246: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:49:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:08 smithi055 ceph-mon[97033]: pgmap v247: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 20 op/s 2022-09-07T21:49:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:08 smithi059 ceph-mon[97450]: pgmap v247: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 20 op/s 2022-09-07T21:49:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:09 smithi055 ceph-mon[97033]: osdmap e97: 12 total, 12 up, 12 in 2022-09-07T21:49:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:10.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:09 smithi059 ceph-mon[97450]: osdmap e97: 12 total, 12 up, 12 in 2022-09-07T21:49:10.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:10 smithi055 ceph-mon[97033]: pgmap v249: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2022-09-07T21:49:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:11.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:10 smithi059 ceph-mon[97450]: pgmap v249: 129 pgs: 129 active+clean; 23 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2022-09-07T21:49:11.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:12.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:12 smithi055 ceph-mon[97033]: pgmap v250: 129 pgs: 10 active+clean+snaptrim, 101 active+clean, 18 active+clean+snaptrim_wait; 22 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:49:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:12 smithi059 ceph-mon[97450]: pgmap v250: 129 pgs: 10 active+clean+snaptrim, 101 active+clean, 18 active+clean+snaptrim_wait; 22 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:49:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:14 smithi055 ceph-mon[97033]: pgmap v251: 129 pgs: 18 active+clean+snaptrim, 65 active+clean, 46 active+clean+snaptrim_wait; 20 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T21:49:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:14 smithi059 ceph-mon[97450]: pgmap v251: 129 pgs: 18 active+clean+snaptrim, 65 active+clean, 46 active+clean+snaptrim_wait; 20 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T21:49:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:16 smithi055 ceph-mon[97033]: pgmap v252: 129 pgs: 10 active+clean+snaptrim, 87 active+clean, 32 active+clean+snaptrim_wait; 18 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 716 B/s wr, 6 op/s 2022-09-07T21:49:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:16 smithi059 ceph-mon[97450]: pgmap v252: 129 pgs: 10 active+clean+snaptrim, 87 active+clean, 32 active+clean+snaptrim_wait; 18 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 716 B/s wr, 6 op/s 2022-09-07T21:49:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:18 smithi055 ceph-mon[97033]: pgmap v253: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:49:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:18 smithi059 ceph-mon[97450]: pgmap v253: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:49:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:20 smithi055 ceph-mon[97033]: pgmap v254: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 15 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 193 B/s wr, 6 op/s 2022-09-07T21:49:20.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:20 smithi059 ceph-mon[97450]: pgmap v254: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 15 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 193 B/s wr, 6 op/s 2022-09-07T21:49:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:22 smithi055 ceph-mon[97033]: pgmap v255: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 15 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T21:49:22.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:22 smithi059 ceph-mon[97450]: pgmap v255: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 15 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T21:49:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:23.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:24 smithi055 ceph-mon[97033]: pgmap v256: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:49:24.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:24 smithi059 ceph-mon[97450]: pgmap v256: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:49:25.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:26 smithi055 ceph-mon[97033]: pgmap v257: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 6 op/s 2022-09-07T21:49:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:26 smithi059 ceph-mon[97450]: pgmap v257: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 6 op/s 2022-09-07T21:49:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:27.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:27 smithi055 ceph-mon[97033]: osdmap e98: 12 total, 12 up, 12 in 2022-09-07T21:49:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:28.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:27 smithi059 ceph-mon[97450]: osdmap e98: 12 total, 12 up, 12 in 2022-09-07T21:49:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:28 smithi055 ceph-mon[97033]: pgmap v258: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:49:28.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:28 smithi059 ceph-mon[97450]: pgmap v258: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:49:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:30 smithi055 ceph-mon[97033]: pgmap v260: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:49:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:31.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:30 smithi059 ceph-mon[97450]: pgmap v260: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:49:31.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:32.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:32 smithi059 ceph-mon[97450]: pgmap v261: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:49:33.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:32 smithi055 ceph-mon[97033]: pgmap v261: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:49:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:34.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:34 smithi059 ceph-mon[97450]: pgmap v262: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:49:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:34 smithi055 ceph-mon[97033]: pgmap v262: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:49:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:37.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:36 smithi059 ceph-mon[97450]: pgmap v263: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:49:37.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:36 smithi055 ceph-mon[97033]: pgmap v263: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:49:37.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:38 smithi059 ceph-mon[97450]: pgmap v264: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:49:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:38 smithi055 ceph-mon[97033]: pgmap v264: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:49:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:40 smithi059 ceph-mon[97450]: pgmap v265: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:49:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:40 smithi055 ceph-mon[97033]: pgmap v265: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:49:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:42.113 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:42.201 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:42 smithi055 ceph-mon[97033]: pgmap v266: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:49:43.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:43.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:49:43.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:49:43.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:49:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:42 smithi059 ceph-mon[97450]: pgmap v266: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:49:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:43.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:49:43.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:49:43.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:49:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:43.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:49:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:49:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:44 smithi055 ceph-mon[97033]: pgmap v267: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:49:45.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:44 smithi059 ceph-mon[97450]: pgmap v267: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:49:45.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:46 smithi055 ceph-mon[97033]: pgmap v268: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:49:47.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:46 smithi059 ceph-mon[97450]: pgmap v268: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:49:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:48 smithi055 ceph-mon[97033]: pgmap v269: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:49:49.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:48 smithi059 ceph-mon[97450]: pgmap v269: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:49:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:50 smithi055 ceph-mon[97033]: pgmap v270: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:49:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:50 smithi059 ceph-mon[97450]: pgmap v270: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:49:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:52 smithi055 ceph-mon[97033]: pgmap v271: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:49:53.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:52 smithi059 ceph-mon[97450]: pgmap v271: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:49:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:54.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:54 smithi055 ceph-mon[97033]: pgmap v272: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:49:55.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:54 smithi059 ceph-mon[97450]: pgmap v272: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:49:55.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:56 smithi055 ceph-mon[97033]: pgmap v273: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:49:57.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:56 smithi059 ceph-mon[97450]: pgmap v273: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:49:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:58 smithi055 ceph-mon[97033]: pgmap v274: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:49:59.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:49:59.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:58 smithi059 ceph-mon[97450]: pgmap v274: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:49:59.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:49:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:49:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:00 smithi055 ceph-mon[97033]: pgmap v275: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:50:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:00 smithi055 ceph-mon[97033]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-07T21:50:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:00 smithi059 ceph-mon[97450]: pgmap v275: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:50:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:00 smithi059 ceph-mon[97450]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-07T21:50:02.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:02 smithi055 ceph-mon[97033]: pgmap v276: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:50:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:02 smithi059 ceph-mon[97450]: pgmap v276: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:50:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:04 smithi055 ceph-mon[97033]: pgmap v277: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-07T21:50:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:05.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:50:05.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:50:05.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:04 smithi059 ceph-mon[97450]: pgmap v277: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-07T21:50:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:50:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:50:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:06 smithi055 ceph-mon[97033]: pgmap v278: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:50:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:06 smithi059 ceph-mon[97450]: pgmap v278: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:50:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:08 smithi055 ceph-mon[97033]: pgmap v279: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:50:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:08 smithi059 ceph-mon[97450]: pgmap v279: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:50:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:09.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:09 smithi055 ceph-mon[97033]: osdmap e99: 12 total, 12 up, 12 in 2022-09-07T21:50:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-07T21:50:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:09 smithi059 ceph-mon[97450]: osdmap e99: 12 total, 12 up, 12 in 2022-09-07T21:50:11.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:10 smithi055 ceph-mon[97033]: pgmap v281: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:50:11.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:10 smithi059 ceph-mon[97450]: pgmap v281: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:50:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:12 smithi055 ceph-mon[97033]: pgmap v282: 129 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 99 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:50:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:12 smithi059 ceph-mon[97450]: pgmap v282: 129 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 99 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:50:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:14 smithi055 ceph-mon[97033]: pgmap v283: 129 pgs: 21 active+clean+snaptrim, 56 active+clean+snaptrim_wait, 52 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 21 op/s 2022-09-07T21:50:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:14 smithi059 ceph-mon[97450]: pgmap v283: 129 pgs: 21 active+clean+snaptrim, 56 active+clean+snaptrim_wait, 52 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 21 op/s 2022-09-07T21:50:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:16 smithi055 ceph-mon[97033]: pgmap v284: 129 pgs: 15 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 74 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1023 B/s wr, 6 op/s 2022-09-07T21:50:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:16 smithi059 ceph-mon[97450]: pgmap v284: 129 pgs: 15 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 74 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1023 B/s wr, 6 op/s 2022-09-07T21:50:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:18 smithi055 ceph-mon[97033]: pgmap v285: 129 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 98 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:50:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:18 smithi059 ceph-mon[97450]: pgmap v285: 129 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 98 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:50:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:20 smithi055 ceph-mon[97033]: pgmap v286: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 199 B/s wr, 6 op/s 2022-09-07T21:50:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:20 smithi059 ceph-mon[97450]: pgmap v286: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 199 B/s wr, 6 op/s 2022-09-07T21:50:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:22 smithi055 ceph-mon[97033]: pgmap v287: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T21:50:23.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:22 smithi055 ceph-mon[97033]: osdmap e100: 12 total, 12 up, 12 in 2022-09-07T21:50:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:22 smithi059 ceph-mon[97450]: pgmap v287: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T21:50:23.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:22 smithi059 ceph-mon[97450]: osdmap e100: 12 total, 12 up, 12 in 2022-09-07T21:50:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:24 smithi055 ceph-mon[97033]: pgmap v289: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:50:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:24 smithi059 ceph-mon[97450]: pgmap v289: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:50:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:26 smithi055 ceph-mon[97033]: pgmap v290: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:50:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:26 smithi059 ceph-mon[97450]: pgmap v290: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:50:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:28 smithi055 ceph-mon[97033]: pgmap v291: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:50:28.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:28 smithi055 ceph-mon[97033]: osdmap e101: 12 total, 12 up, 12 in 2022-09-07T21:50:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:28 smithi059 ceph-mon[97450]: pgmap v291: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:50:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:28 smithi059 ceph-mon[97450]: osdmap e101: 12 total, 12 up, 12 in 2022-09-07T21:50:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:30 smithi055 ceph-mon[97033]: pgmap v293: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2022-09-07T21:50:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:30 smithi059 ceph-mon[97450]: pgmap v293: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2022-09-07T21:50:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:32 smithi055 ceph-mon[97033]: pgmap v294: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-07T21:50:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:32 smithi059 ceph-mon[97450]: pgmap v294: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-07T21:50:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:34 smithi055 ceph-mon[97033]: pgmap v295: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:50:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:34 smithi059 ceph-mon[97450]: pgmap v295: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:50:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:36 smithi055 ceph-mon[97033]: pgmap v296: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:50:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:36 smithi059 ceph-mon[97450]: pgmap v296: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:50:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:38 smithi055 ceph-mon[97033]: pgmap v297: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:50:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:38 smithi059 ceph-mon[97450]: pgmap v297: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:50:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:40 smithi055 ceph-mon[97033]: pgmap v298: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:50:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:40 smithi059 ceph-mon[97450]: pgmap v298: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:50:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:42 smithi055 ceph-mon[97033]: pgmap v299: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:50:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:42 smithi059 ceph-mon[97450]: pgmap v299: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:50:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:44 smithi055 ceph-mon[97033]: pgmap v300: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:50:44.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:50:44.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:50:44.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:50:44.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:50:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:44 smithi059 ceph-mon[97450]: pgmap v300: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:50:44.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:50:44.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:50:44.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:50:44.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:50:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:46 smithi055 ceph-mon[97033]: pgmap v301: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:50:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:46 smithi059 ceph-mon[97450]: pgmap v301: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:50:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:48 smithi055 ceph-mon[97033]: pgmap v302: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:50:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:48 smithi059 ceph-mon[97450]: pgmap v302: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:50:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:50 smithi055 ceph-mon[97033]: pgmap v303: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:50:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:50 smithi059 ceph-mon[97450]: pgmap v303: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:50:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:52 smithi055 ceph-mon[97033]: pgmap v304: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:50:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:52 smithi059 ceph-mon[97450]: pgmap v304: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:50:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:54 smithi055 ceph-mon[97033]: pgmap v305: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:50:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:54 smithi059 ceph-mon[97450]: pgmap v305: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:50:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:56 smithi055 ceph-mon[97033]: pgmap v306: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:50:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:56 smithi059 ceph-mon[97450]: pgmap v306: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:50:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:50:58 smithi055 ceph-mon[97033]: pgmap v307: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:50:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:50:58 smithi059 ceph-mon[97450]: pgmap v307: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:51:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:00 smithi055 ceph-mon[97033]: pgmap v308: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:51:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:00 smithi059 ceph-mon[97450]: pgmap v308: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:51:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:02 smithi055 ceph-mon[97033]: pgmap v309: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2022-09-07T21:51:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:02 smithi059 ceph-mon[97450]: pgmap v309: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2022-09-07T21:51:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:04 smithi059 ceph-mon[97450]: pgmap v310: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:51:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:51:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:51:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:04 smithi055 ceph-mon[97033]: pgmap v310: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:51:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:51:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:51:06.633 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2022-09-07T21:51:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:06 smithi059 ceph-mon[97450]: pgmap v311: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:51:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:06 smithi055 ceph-mon[97033]: pgmap v311: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:51:08.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:08 smithi059 ceph-mon[97450]: pgmap v312: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:51:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:08 smithi055 ceph-mon[97033]: pgmap v312: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:51:09.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:09 smithi059 ceph-mon[97450]: osdmap e102: 12 total, 12 up, 12 in 2022-09-07T21:51:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:09 smithi055 ceph-mon[97033]: osdmap e102: 12 total, 12 up, 12 in 2022-09-07T21:51:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:10 smithi059 ceph-mon[97450]: pgmap v314: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:51:10.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:10 smithi055 ceph-mon[97033]: pgmap v314: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:51:12.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:12 smithi055 ceph-mon[97033]: pgmap v315: 129 pgs: 10 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 98 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:51:13.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:12 smithi059 ceph-mon[97450]: pgmap v315: 129 pgs: 10 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 98 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:51:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:14 smithi055 ceph-mon[97033]: pgmap v316: 129 pgs: 20 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 60 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T21:51:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:14 smithi059 ceph-mon[97450]: pgmap v316: 129 pgs: 20 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 60 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T21:51:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:16 smithi055 ceph-mon[97033]: pgmap v317: 129 pgs: 15 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 76 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:51:17.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:16 smithi059 ceph-mon[97450]: pgmap v317: 129 pgs: 15 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 76 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:51:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:18 smithi055 ceph-mon[97033]: pgmap v318: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:51:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:18 smithi059 ceph-mon[97450]: pgmap v318: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:51:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:20 smithi055 ceph-mon[97033]: pgmap v319: 129 pgs: 9 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 190 B/s wr, 5 op/s 2022-09-07T21:51:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:20 smithi059 ceph-mon[97450]: pgmap v319: 129 pgs: 9 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 190 B/s wr, 5 op/s 2022-09-07T21:51:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:22 smithi055 ceph-mon[97033]: pgmap v320: 129 pgs: 6 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 103 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T21:51:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:22 smithi059 ceph-mon[97450]: pgmap v320: 129 pgs: 6 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 103 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T21:51:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:24 smithi055 ceph-mon[97033]: pgmap v321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:51:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:24 smithi059 ceph-mon[97450]: pgmap v321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:51:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:26 smithi055 ceph-mon[97033]: pgmap v322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 6 op/s 2022-09-07T21:51:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:26 smithi059 ceph-mon[97450]: pgmap v322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 6 op/s 2022-09-07T21:51:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:28 smithi055 ceph-mon[97033]: pgmap v323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:51:28.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:28 smithi055 ceph-mon[97033]: osdmap e103: 12 total, 12 up, 12 in 2022-09-07T21:51:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:28 smithi059 ceph-mon[97450]: pgmap v323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:51:28.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:28 smithi059 ceph-mon[97450]: osdmap e103: 12 total, 12 up, 12 in 2022-09-07T21:51:30.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:30 smithi055 ceph-mon[97033]: pgmap v325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:51:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:30 smithi059 ceph-mon[97450]: pgmap v325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:51:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:32 smithi055 ceph-mon[97033]: pgmap v326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:51:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:32 smithi059 ceph-mon[97450]: pgmap v326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:51:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:34 smithi055 ceph-mon[97033]: pgmap v327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:51:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:34 smithi059 ceph-mon[97450]: pgmap v327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:51:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:36 smithi055 ceph-mon[97033]: pgmap v328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:51:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:36 smithi059 ceph-mon[97450]: pgmap v328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:51:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:38 smithi055 ceph-mon[97033]: pgmap v329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:51:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:38 smithi059 ceph-mon[97450]: pgmap v329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:51:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:40 smithi055 ceph-mon[97033]: pgmap v330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:51:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:40 smithi059 ceph-mon[97450]: pgmap v330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:51:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:42 smithi055 ceph-mon[97033]: pgmap v331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:51:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:42 smithi059 ceph-mon[97450]: pgmap v331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:51:44.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:44 smithi055 ceph-mon[97033]: pgmap v332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:51:44.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:51:44.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:51:44.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:51:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:44 smithi059 ceph-mon[97450]: pgmap v332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:51:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:51:44.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:51:44.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:51:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:51:46.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:51:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:51:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:51:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:46 smithi055 ceph-mon[97033]: pgmap v333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:51:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:46 smithi059 ceph-mon[97450]: pgmap v333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:51:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: pgmap v334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:51:48.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:51:48.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:51:48.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:51:48.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:51:48.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T21:51:48.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T21:51:48.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: pgmap v334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:51:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:51:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:51:48.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:51:48.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:51:48.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T21:51:48.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T21:51:48.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:48.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T21:51:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:49 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T21:51:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:49 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T21:51:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:49 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T21:51:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:49 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T21:51:50.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:50 smithi059 ceph-mon[97450]: pgmap v335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:51:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:50 smithi055 ceph-mon[97033]: pgmap v335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:51:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:52 smithi059 ceph-mon[97450]: pgmap v336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:51:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:52 smithi055 ceph-mon[97033]: pgmap v336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:51:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:54 smithi055 ceph-mon[97033]: pgmap v337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:51:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:54 smithi059 ceph-mon[97450]: pgmap v337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:51:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:56 smithi055 ceph-mon[97033]: pgmap v338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:51:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:56 smithi059 ceph-mon[97450]: pgmap v338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:51:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:51:58 smithi055 ceph-mon[97033]: pgmap v339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:51:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:51:58 smithi059 ceph-mon[97450]: pgmap v339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:52:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:00 smithi055 ceph-mon[97033]: pgmap v340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:52:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:00 smithi059 ceph-mon[97450]: pgmap v340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:52:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:02 smithi055 ceph-mon[97033]: pgmap v341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2022-09-07T21:52:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:02 smithi059 ceph-mon[97450]: pgmap v341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2022-09-07T21:52:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:03 smithi055 ceph-mon[97033]: evicting unresponsive client smithi047:0 (25304), after 363.399 seconds 2022-09-07T21:52:03.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:03 smithi055 ceph-mon[97033]: Evicting (and blocklisting) client session 25304 (172.21.15.47:0/4287182335) 2022-09-07T21:52:03.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:03 smithi055 ceph-mon[97033]: from='mds.? [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162]' entity='mds.a' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.47:0/4287182335"}]: dispatch 2022-09-07T21:52:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:03 smithi059 ceph-mon[97450]: evicting unresponsive client smithi047:0 (25304), after 363.399 seconds 2022-09-07T21:52:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:03 smithi059 ceph-mon[97450]: Evicting (and blocklisting) client session 25304 (172.21.15.47:0/4287182335) 2022-09-07T21:52:04.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:03 smithi059 ceph-mon[97450]: from='mds.? [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162]' entity='mds.a' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.47:0/4287182335"}]: dispatch 2022-09-07T21:52:05.005 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:04 smithi055 ceph-mon[97033]: pgmap v342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:52:05.006 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:04 smithi055 ceph-mon[97033]: from='mds.? [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162]' entity='mds.a' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.47:0/4287182335"}]': finished 2022-09-07T21:52:05.007 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:04 smithi055 ceph-mon[97033]: osdmap e104: 12 total, 12 up, 12 in 2022-09-07T21:52:05.007 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:52:05.008 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:52:05.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:04 smithi059 ceph-mon[97450]: pgmap v342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:52:05.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:04 smithi059 ceph-mon[97450]: from='mds.? [v2:172.21.15.47:6840/793863162,v1:172.21.15.47:6841/793863162]' entity='mds.a' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.47:0/4287182335"}]': finished 2022-09-07T21:52:05.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:04 smithi059 ceph-mon[97450]: osdmap e104: 12 total, 12 up, 12 in 2022-09-07T21:52:05.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:52:05.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:52:05.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:05 smithi055 ceph-mon[97033]: osdmap e105: 12 total, 12 up, 12 in 2022-09-07T21:52:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:05 smithi059 ceph-mon[97450]: osdmap e105: 12 total, 12 up, 12 in 2022-09-07T21:52:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:06 smithi055 ceph-mon[97033]: pgmap v345: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 35 KiB/s wr, 13 op/s 2022-09-07T21:52:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:06 smithi059 ceph-mon[97450]: pgmap v345: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 35 KiB/s wr, 13 op/s 2022-09-07T21:52:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:08 smithi055 ceph-mon[97033]: pgmap v346: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 67 KiB/s wr, 36 op/s 2022-09-07T21:52:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:08 smithi059 ceph-mon[97450]: pgmap v346: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 67 KiB/s wr, 36 op/s 2022-09-07T21:52:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:10 smithi055 ceph-mon[97033]: pgmap v347: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 67 KiB/s wr, 37 op/s 2022-09-07T21:52:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:10 smithi059 ceph-mon[97450]: pgmap v347: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 67 KiB/s wr, 37 op/s 2022-09-07T21:52:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:12 smithi055 ceph-mon[97033]: pgmap v348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 39 KiB/s wr, 37 op/s 2022-09-07T21:52:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:12 smithi055 ceph-mon[97033]: osdmap e106: 12 total, 12 up, 12 in 2022-09-07T21:52:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:12 smithi059 ceph-mon[97450]: pgmap v348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 39 KiB/s wr, 37 op/s 2022-09-07T21:52:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:12 smithi059 ceph-mon[97450]: osdmap e106: 12 total, 12 up, 12 in 2022-09-07T21:52:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:14 smithi055 ceph-mon[97033]: pgmap v350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 36 KiB/s wr, 35 op/s 2022-09-07T21:52:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:14 smithi059 ceph-mon[97450]: pgmap v350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 36 KiB/s wr, 35 op/s 2022-09-07T21:52:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:16 smithi059 ceph-mon[97450]: pgmap v351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 31 KiB/s wr, 28 op/s 2022-09-07T21:52:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:16 smithi059 ceph-mon[97450]: pgmap v352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.5 KiB/s wr, 11 op/s 2022-09-07T21:52:17.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:52:17.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:16 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-07T21:52:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:16 smithi055 ceph-mon[97033]: pgmap v351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 31 KiB/s wr, 28 op/s 2022-09-07T21:52:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:16 smithi055 ceph-mon[97033]: pgmap v352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.5 KiB/s wr, 11 op/s 2022-09-07T21:52:17.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:52:17.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:16 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-07T21:52:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:18 smithi055 ceph-mon[97033]: pgmap v353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.0 KiB/s wr, 8 op/s 2022-09-07T21:52:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:18 smithi059 ceph-mon[97450]: pgmap v353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.0 KiB/s wr, 8 op/s 2022-09-07T21:52:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:21 smithi055 ceph-mon[97033]: pgmap v354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 234 B/s wr, 5 op/s 2022-09-07T21:52:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:21 smithi059 ceph-mon[97450]: pgmap v354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 234 B/s wr, 5 op/s 2022-09-07T21:52:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:23 smithi055 ceph-mon[97033]: pgmap v355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 321 B/s wr, 6 op/s 2022-09-07T21:52:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:23 smithi059 ceph-mon[97450]: pgmap v355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 321 B/s wr, 6 op/s 2022-09-07T21:52:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:25 smithi055 ceph-mon[97033]: pgmap v356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 285 B/s wr, 6 op/s 2022-09-07T21:52:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:25 smithi059 ceph-mon[97450]: pgmap v356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 285 B/s wr, 6 op/s 2022-09-07T21:52:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:27 smithi055 ceph-mon[97033]: pgmap v357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 285 B/s wr, 6 op/s 2022-09-07T21:52:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:27 smithi059 ceph-mon[97450]: pgmap v357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 285 B/s wr, 6 op/s 2022-09-07T21:52:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:28 smithi055 ceph-mon[97033]: pgmap v358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-07T21:52:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:28 smithi059 ceph-mon[97450]: pgmap v358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-07T21:52:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:31 smithi055 ceph-mon[97033]: pgmap v359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:52:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:31 smithi059 ceph-mon[97450]: pgmap v359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T21:52:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:33 smithi055 ceph-mon[97033]: pgmap v360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2022-09-07T21:52:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:33 smithi059 ceph-mon[97450]: pgmap v360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2022-09-07T21:52:35.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:35 smithi055 ceph-mon[97033]: pgmap v361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:52:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:35 smithi059 ceph-mon[97450]: pgmap v361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:52:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:37 smithi055 ceph-mon[97033]: pgmap v362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:52:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:37 smithi059 ceph-mon[97450]: pgmap v362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:52:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:38 smithi055 ceph-mon[97033]: pgmap v363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-07T21:52:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:38 smithi059 ceph-mon[97450]: pgmap v363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-07T21:52:41.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:41 smithi055 ceph-mon[97033]: pgmap v364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2022-09-07T21:52:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:41 smithi059 ceph-mon[97450]: pgmap v364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2022-09-07T21:52:43.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:43 smithi055 ceph-mon[97033]: pgmap v365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2022-09-07T21:52:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:43 smithi059 ceph-mon[97450]: pgmap v365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2022-09-07T21:52:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:45 smithi055 ceph-mon[97033]: pgmap v366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-07T21:52:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:45 smithi059 ceph-mon[97450]: pgmap v366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-07T21:52:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:47 smithi055 ceph-mon[97033]: pgmap v367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:52:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:47 smithi059 ceph-mon[97450]: pgmap v367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:52:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:48 smithi055 ceph-mon[97033]: pgmap v368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:52:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:48 smithi059 ceph-mon[97450]: pgmap v368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:52:51.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:51 smithi055 ceph-mon[97033]: pgmap v369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:52:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:51 smithi059 ceph-mon[97450]: pgmap v369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:52:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:53 smithi055 ceph-mon[97033]: pgmap v370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T21:52:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:53 smithi059 ceph-mon[97450]: pgmap v370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T21:52:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:55 smithi055 ceph-mon[97033]: pgmap v371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:52:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:55 smithi059 ceph-mon[97450]: pgmap v371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:52:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:57 smithi055 ceph-mon[97033]: pgmap v372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:52:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:57 smithi059 ceph-mon[97450]: pgmap v372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:52:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:52:58 smithi055 ceph-mon[97033]: pgmap v373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:52:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:52:58 smithi059 ceph-mon[97450]: pgmap v373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:53:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:01 smithi055 ceph-mon[97033]: pgmap v374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:53:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:01 smithi059 ceph-mon[97450]: pgmap v374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:53:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:03 smithi055 ceph-mon[97033]: pgmap v375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2022-09-07T21:53:03.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:03 smithi059 ceph-mon[97450]: pgmap v375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2022-09-07T21:53:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:05 smithi055 ceph-mon[97033]: pgmap v376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:53:05.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:53:05.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:53:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:05 smithi059 ceph-mon[97450]: pgmap v376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:53:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:53:05.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:53:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:07 smithi055 ceph-mon[97033]: pgmap v377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:53:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:07 smithi059 ceph-mon[97450]: pgmap v377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:53:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:08 smithi055 ceph-mon[97033]: pgmap v378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:53:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:08 smithi059 ceph-mon[97450]: pgmap v378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:53:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:09 smithi055 ceph-mon[97033]: osdmap e107: 12 total, 12 up, 12 in 2022-09-07T21:53:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:09 smithi059 ceph-mon[97450]: osdmap e107: 12 total, 12 up, 12 in 2022-09-07T21:53:10.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:10 smithi055 ceph-mon[97033]: pgmap v380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:53:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:10 smithi059 ceph-mon[97450]: pgmap v380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:53:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:13 smithi055 ceph-mon[97033]: pgmap v381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:53:13.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:13 smithi059 ceph-mon[97450]: pgmap v381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:53:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:15 smithi055 ceph-mon[97033]: pgmap v382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T21:53:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:15 smithi059 ceph-mon[97450]: pgmap v382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T21:53:17.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:17 smithi055 ceph-mon[97033]: pgmap v383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:53:17.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:53:17.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:53:17.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:53:17.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:53:17.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:53:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:17 smithi059 ceph-mon[97450]: pgmap v383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:53:17.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:53:17.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:53:17.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:53:17.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:53:17.538 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:53:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:18 smithi055 ceph-mon[97033]: osdmap e108: 12 total, 12 up, 12 in 2022-09-07T21:53:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:18 smithi055 ceph-mon[97033]: pgmap v385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-07T21:53:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:18 smithi059 ceph-mon[97450]: osdmap e108: 12 total, 12 up, 12 in 2022-09-07T21:53:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:18 smithi059 ceph-mon[97450]: pgmap v385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-07T21:53:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:21 smithi055 ceph-mon[97033]: pgmap v386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:53:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:21 smithi059 ceph-mon[97450]: pgmap v386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:53:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:23 smithi055 ceph-mon[97033]: pgmap v387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:53:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:23 smithi059 ceph-mon[97450]: pgmap v387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:53:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:25 smithi055 ceph-mon[97033]: pgmap v388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:53:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:25 smithi059 ceph-mon[97450]: pgmap v388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:53:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:27 smithi055 ceph-mon[97033]: pgmap v389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:53:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:27 smithi059 ceph-mon[97450]: pgmap v389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:53:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:28 smithi055 ceph-mon[97033]: pgmap v390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-07T21:53:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:28 smithi059 ceph-mon[97450]: pgmap v390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-07T21:53:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:31 smithi055 ceph-mon[97033]: pgmap v391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2022-09-07T21:53:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:31 smithi059 ceph-mon[97450]: pgmap v391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2022-09-07T21:53:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:33 smithi055 ceph-mon[97033]: pgmap v392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2022-09-07T21:53:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:33 smithi059 ceph-mon[97450]: pgmap v392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2022-09-07T21:53:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:35 smithi055 ceph-mon[97033]: pgmap v393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-07T21:53:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:35 smithi059 ceph-mon[97450]: pgmap v393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-07T21:53:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:37 smithi055 ceph-mon[97033]: pgmap v394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:53:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:37 smithi059 ceph-mon[97450]: pgmap v394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:53:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:38 smithi055 ceph-mon[97033]: pgmap v395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:53:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:38 smithi059 ceph-mon[97450]: pgmap v395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:53:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:41 smithi055 ceph-mon[97033]: pgmap v396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:53:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:41 smithi059 ceph-mon[97450]: pgmap v396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:53:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:43 smithi055 ceph-mon[97033]: pgmap v397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T21:53:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:43 smithi059 ceph-mon[97450]: pgmap v397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T21:53:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:45 smithi055 ceph-mon[97033]: pgmap v398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:53:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:45 smithi059 ceph-mon[97450]: pgmap v398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:53:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:47 smithi055 ceph-mon[97033]: pgmap v399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:53:47.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:53:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:47 smithi059 ceph-mon[97450]: pgmap v399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:53:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:53:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:48 smithi055 ceph-mon[97033]: pgmap v400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:53:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:48 smithi055 ceph-mon[97033]: pgmap v401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T21:53:48.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:48 smithi059 ceph-mon[97450]: pgmap v400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:53:48.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:48 smithi059 ceph-mon[97450]: pgmap v401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T21:53:50.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:50 smithi055 ceph-mon[97033]: pgmap v402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-07T21:53:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:50 smithi059 ceph-mon[97450]: pgmap v402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-07T21:53:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:52 smithi055 ceph-mon[97033]: pgmap v403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T21:53:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:52 smithi059 ceph-mon[97450]: pgmap v403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T21:53:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:54 smithi055 ceph-mon[97033]: pgmap v404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T21:53:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:54 smithi059 ceph-mon[97450]: pgmap v404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T21:53:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:56 smithi055 ceph-mon[97033]: pgmap v405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-07T21:53:56.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:56 smithi059 ceph-mon[97450]: pgmap v405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-07T21:53:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:53:58 smithi055 ceph-mon[97033]: pgmap v406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:53:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:53:58 smithi059 ceph-mon[97450]: pgmap v406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:54:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:00 smithi055 ceph-mon[97033]: pgmap v407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:00 smithi059 ceph-mon[97450]: pgmap v407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:02 smithi055 ceph-mon[97033]: pgmap v408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:54:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:02 smithi059 ceph-mon[97450]: pgmap v408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:54:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:04 smithi055 ceph-mon[97033]: pgmap v409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-07T21:54:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:04 smithi059 ceph-mon[97450]: pgmap v409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-07T21:54:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:54:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:54:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:54:05.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:54:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:06 smithi055 ceph-mon[97033]: pgmap v410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:54:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:06 smithi059 ceph-mon[97450]: pgmap v410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:54:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:08 smithi055 ceph-mon[97033]: pgmap v411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:54:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:08 smithi059 ceph-mon[97450]: pgmap v411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:54:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:10 smithi055 ceph-mon[97033]: pgmap v412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:54:10.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:10 smithi055 ceph-mon[97033]: osdmap e109: 12 total, 12 up, 12 in 2022-09-07T21:54:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:10 smithi059 ceph-mon[97450]: pgmap v412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:54:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:10 smithi059 ceph-mon[97450]: osdmap e109: 12 total, 12 up, 12 in 2022-09-07T21:54:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:12 smithi055 ceph-mon[97033]: pgmap v414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:54:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:12 smithi059 ceph-mon[97450]: pgmap v414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:54:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:14 smithi055 ceph-mon[97033]: pgmap v415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:54:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:14 smithi059 ceph-mon[97450]: pgmap v415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:54:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:16 smithi055 ceph-mon[97033]: pgmap v416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T21:54:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:16 smithi059 ceph-mon[97450]: pgmap v416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T21:54:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:18 smithi055 ceph-mon[97033]: pgmap v417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:54:18.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:18 smithi055 ceph-mon[97033]: osdmap e110: 12 total, 12 up, 12 in 2022-09-07T21:54:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:18 smithi059 ceph-mon[97450]: pgmap v417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:54:18.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:18 smithi059 ceph-mon[97450]: osdmap e110: 12 total, 12 up, 12 in 2022-09-07T21:54:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:20 smithi055 ceph-mon[97033]: pgmap v419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 224 B/s wr, 7 op/s 2022-09-07T21:54:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:20 smithi059 ceph-mon[97450]: pgmap v419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 224 B/s wr, 7 op/s 2022-09-07T21:54:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:22 smithi055 ceph-mon[97033]: pgmap v420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:54:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:22 smithi059 ceph-mon[97450]: pgmap v420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:54:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:24 smithi055 ceph-mon[97033]: pgmap v421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:54:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:24 smithi059 ceph-mon[97450]: pgmap v421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:54:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:26 smithi055 ceph-mon[97033]: pgmap v422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:54:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:26 smithi059 ceph-mon[97450]: pgmap v422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:54:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:27 smithi055 ceph-mon[97033]: pgmap v423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:54:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:27 smithi059 ceph-mon[97450]: pgmap v423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:54:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:29 smithi059 ceph-mon[97450]: pgmap v424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T21:54:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:29 smithi055 ceph-mon[97033]: pgmap v424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T21:54:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:31 smithi059 ceph-mon[97450]: pgmap v425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:54:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:31 smithi055 ceph-mon[97033]: pgmap v425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:54:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:33 smithi059 ceph-mon[97450]: pgmap v426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:33 smithi055 ceph-mon[97033]: pgmap v426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:35 smithi059 ceph-mon[97450]: pgmap v427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:35 smithi055 ceph-mon[97033]: pgmap v427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:37 smithi059 ceph-mon[97450]: pgmap v428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:54:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:37 smithi055 ceph-mon[97033]: pgmap v428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:54:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:39 smithi059 ceph-mon[97450]: pgmap v429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:39 smithi055 ceph-mon[97033]: pgmap v429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:41 smithi059 ceph-mon[97450]: pgmap v430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:54:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:41 smithi055 ceph-mon[97033]: pgmap v430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:54:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:43 smithi059 ceph-mon[97450]: pgmap v431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:43 smithi055 ceph-mon[97033]: pgmap v431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:45 smithi059 ceph-mon[97450]: pgmap v432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:45 smithi055 ceph-mon[97033]: pgmap v432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:54:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:54:46.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:54:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:54:46.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:54:46.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:54:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:47 smithi059 ceph-mon[97450]: pgmap v433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:54:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:47 smithi055 ceph-mon[97033]: pgmap v433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:54:49.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:49 smithi059 ceph-mon[97450]: pgmap v434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:49 smithi055 ceph-mon[97033]: pgmap v434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:51 smithi059 ceph-mon[97450]: pgmap v435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:54:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:51 smithi055 ceph-mon[97033]: pgmap v435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:54:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:53 smithi059 ceph-mon[97450]: pgmap v436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:53 smithi055 ceph-mon[97033]: pgmap v436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:55 smithi059 ceph-mon[97450]: pgmap v437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:55 smithi055 ceph-mon[97033]: pgmap v437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:54:57.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:57 smithi055 ceph-mon[97033]: pgmap v438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:54:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:57 smithi059 ceph-mon[97450]: pgmap v438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:54:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:54:59 smithi055 ceph-mon[97033]: pgmap v439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:55:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:54:59 smithi059 ceph-mon[97450]: pgmap v439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:55:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:01 smithi055 ceph-mon[97033]: pgmap v440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 17 op/s 2022-09-07T21:55:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:01 smithi059 ceph-mon[97450]: pgmap v440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 17 op/s 2022-09-07T21:55:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:03 smithi055 ceph-mon[97033]: pgmap v441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 11 KiB/s wr, 16 op/s 2022-09-07T21:55:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:03 smithi059 ceph-mon[97450]: pgmap v441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 11 KiB/s wr, 16 op/s 2022-09-07T21:55:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:55:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:55:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:55:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:55:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:05 smithi055 ceph-mon[97033]: pgmap v442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 33 KiB/s wr, 18 op/s 2022-09-07T21:55:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:05 smithi059 ceph-mon[97450]: pgmap v442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 33 KiB/s wr, 18 op/s 2022-09-07T21:55:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:07 smithi055 ceph-mon[97033]: pgmap v443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:55:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:07 smithi059 ceph-mon[97450]: pgmap v443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:55:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:09 smithi055 ceph-mon[97033]: pgmap v444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:55:09.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:09 smithi055 ceph-mon[97033]: osdmap e111: 12 total, 12 up, 12 in 2022-09-07T21:55:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:09 smithi059 ceph-mon[97450]: pgmap v444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:55:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:09 smithi059 ceph-mon[97450]: osdmap e111: 12 total, 12 up, 12 in 2022-09-07T21:55:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:11 smithi055 ceph-mon[97033]: pgmap v446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 40 KiB/s wr, 10 op/s 2022-09-07T21:55:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:11 smithi059 ceph-mon[97450]: pgmap v446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 40 KiB/s wr, 10 op/s 2022-09-07T21:55:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:13 smithi055 ceph-mon[97033]: pgmap v447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 40 KiB/s wr, 10 op/s 2022-09-07T21:55:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:13 smithi059 ceph-mon[97450]: pgmap v447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 40 KiB/s wr, 10 op/s 2022-09-07T21:55:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:15 smithi055 ceph-mon[97033]: pgmap v448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 13 KiB/s wr, 8 op/s 2022-09-07T21:55:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:15 smithi059 ceph-mon[97450]: pgmap v448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 13 KiB/s wr, 8 op/s 2022-09-07T21:55:17.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:17 smithi055 ceph-mon[97033]: pgmap v449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:55:17.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:17 smithi055 ceph-mon[97033]: pgmap v450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-07T21:55:17.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:17 smithi055 ceph-mon[97033]: pgmap v451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 257 B/s wr, 7 op/s 2022-09-07T21:55:17.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:55:17.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:17 smithi055 ceph-mon[97033]: osdmap e112: 12 total, 12 up, 12 in 2022-09-07T21:55:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:17 smithi059 ceph-mon[97450]: pgmap v449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:55:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:17 smithi059 ceph-mon[97450]: pgmap v450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-07T21:55:18.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:17 smithi059 ceph-mon[97450]: pgmap v451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 257 B/s wr, 7 op/s 2022-09-07T21:55:18.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:55:18.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:17 smithi059 ceph-mon[97450]: osdmap e112: 12 total, 12 up, 12 in 2022-09-07T21:55:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:19 smithi055 ceph-mon[97033]: pgmap v453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T21:55:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:19 smithi059 ceph-mon[97450]: pgmap v453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T21:55:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:21 smithi055 ceph-mon[97033]: pgmap v454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 163 B/s wr, 8 op/s 2022-09-07T21:55:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:21 smithi059 ceph-mon[97450]: pgmap v454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 163 B/s wr, 8 op/s 2022-09-07T21:55:23.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:23 smithi059 ceph-mon[97450]: pgmap v455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-07T21:55:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:23 smithi055 ceph-mon[97033]: pgmap v455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-07T21:55:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:25 smithi055 ceph-mon[97033]: pgmap v456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2022-09-07T21:55:26.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:25 smithi059 ceph-mon[97450]: pgmap v456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2022-09-07T21:55:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:27 smithi055 ceph-mon[97033]: pgmap v457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:55:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:27 smithi059 ceph-mon[97450]: pgmap v457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:55:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:29 smithi055 ceph-mon[97033]: pgmap v458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-07T21:55:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:29 smithi059 ceph-mon[97450]: pgmap v458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-07T21:55:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:31 smithi055 ceph-mon[97033]: pgmap v459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:55:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:31 smithi059 ceph-mon[97450]: pgmap v459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:55:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:33 smithi059 ceph-mon[97450]: pgmap v460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:55:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:33 smithi055 ceph-mon[97033]: pgmap v460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:55:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:35 smithi055 ceph-mon[97033]: pgmap v461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:55:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:35 smithi059 ceph-mon[97450]: pgmap v461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:55:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:37 smithi055 ceph-mon[97033]: pgmap v462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:55:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:37 smithi059 ceph-mon[97450]: pgmap v462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:55:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:39 smithi055 ceph-mon[97033]: pgmap v463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:55:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:39 smithi059 ceph-mon[97450]: pgmap v463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:55:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:41 smithi055 ceph-mon[97033]: pgmap v464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:55:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:41 smithi059 ceph-mon[97450]: pgmap v464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:55:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:43 smithi059 ceph-mon[97450]: pgmap v465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:55:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:43 smithi055 ceph-mon[97033]: pgmap v465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:55:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:45 smithi055 ceph-mon[97033]: pgmap v466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:55:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:45 smithi059 ceph-mon[97450]: pgmap v466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:55:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:47 smithi055 ceph-mon[97033]: pgmap v467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:55:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:47 smithi059 ceph-mon[97450]: pgmap v467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:55:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:49 smithi055 ceph-mon[97033]: pgmap v468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:55:50.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:49 smithi059 ceph-mon[97450]: pgmap v468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:55:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:51 smithi055 ceph-mon[97033]: pgmap v469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:55:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:51 smithi059 ceph-mon[97450]: pgmap v469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:55:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:53 smithi059 ceph-mon[97450]: pgmap v470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:55:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:53 smithi055 ceph-mon[97033]: pgmap v470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:55:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:55 smithi055 ceph-mon[97033]: pgmap v471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:55:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:55 smithi059 ceph-mon[97450]: pgmap v471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:55:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:57 smithi055 ceph-mon[97033]: pgmap v472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:55:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:57 smithi059 ceph-mon[97450]: pgmap v472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:55:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:55:59 smithi055 ceph-mon[97033]: pgmap v473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:56:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:55:59 smithi059 ceph-mon[97450]: pgmap v473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:56:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:01 smithi055 ceph-mon[97033]: pgmap v474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 0 B/s wr, 6 op/s 2022-09-07T21:56:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:01 smithi059 ceph-mon[97450]: pgmap v474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 0 B/s wr, 6 op/s 2022-09-07T21:56:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:03 smithi059 ceph-mon[97450]: pgmap v475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 23 KiB/s wr, 8 op/s 2022-09-07T21:56:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:03 smithi055 ceph-mon[97033]: pgmap v475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 23 KiB/s wr, 8 op/s 2022-09-07T21:56:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:56:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:56:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:56:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:56:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:05 smithi059 ceph-mon[97450]: pgmap v476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:56:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:05 smithi055 ceph-mon[97033]: pgmap v476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:56:07.228 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.224+0000 7f1038fff700 -1 monclient: get_monmap_and_config failed to get config 2022-09-07T21:56:07.231 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.227+0000 7f1038fff700 1 -- 172.21.15.47:0/2457347211 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10340fd920 msgr2=0x7f1034066160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:56:07.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.227+0000 7f1038fff700 1 --2- 172.21.15.47:0/2457347211 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10340fd920 0x7f1034066160 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f102c009d70 tx=0x7f102c00e630 comp rx=0 tx=0).stop 2022-09-07T21:56:07.232 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.227+0000 7f1038fff700 1 -- 172.21.15.47:0/2457347211 shutdown_connections 2022-09-07T21:56:07.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.227+0000 7f1038fff700 1 --2- 172.21.15.47:0/2457347211 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10340fd920 0x7f1034066160 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:56:07.233 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.227+0000 7f1038fff700 1 --2- 172.21.15.47:0/2457347211 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1034102030 0x7f1034109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:56:07.234 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.227+0000 7f1038fff700 1 --2- 172.21.15.47:0/2457347211 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10340fcf80 0x7f10340fd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:56:07.234 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.227+0000 7f1038fff700 1 -- 172.21.15.47:0/2457347211 >> 172.21.15.47:0/2457347211 conn(0x7f1034073e90 msgr2=0x7f1034074290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:56:07.235 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.227+0000 7f1038fff700 1 -- 172.21.15.47:0/2457347211 shutdown_connections 2022-09-07T21:56:07.235 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.227+0000 7f1038fff700 1 -- 172.21.15.47:0/2457347211 wait complete. 2022-09-07T21:56:07.242 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.229+0000 7f1038fff700 1 Processor -- start 2022-09-07T21:56:07.242 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.230+0000 7f1038fff700 1 -- start start 2022-09-07T21:56:07.243 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.230+0000 7f1038fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10340fcf80 0x7f103410a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:56:07.244 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.230+0000 7f1038fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10340fd920 0x7f103410abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:56:07.244 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.230+0000 7f1038fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1034102030 0x7f103410b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:56:07.245 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.230+0000 7f1038fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10341a2920 con 0x7f1034102030 2022-09-07T21:56:07.245 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.230+0000 7f1038fff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10341a2a90 con 0x7f10340fd920 2022-09-07T21:56:07.246 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.230+0000 7f1038fff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10341a2c50 con 0x7f10340fcf80 2022-09-07T21:56:07.247 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.230+0000 7f103259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10340fcf80 0x7f103410a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:56:07.247 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.230+0000 7f1029d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10340fd920 0x7f103410abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:56:07.248 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.230+0000 7f103259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10340fcf80 0x7f103410a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.47:51962/0 (socket says 172.21.15.47:51962) 2022-09-07T21:56:07.248 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.230+0000 7f1032d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1034102030 0x7f103410b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:56:07.249 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.230+0000 7f1029d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10340fd920 0x7f103410abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.47:41924/0 (socket says 172.21.15.47:41924) 2022-09-07T21:56:07.250 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.230+0000 7f103259c700 1 -- 172.21.15.47:0/2196953395 learned_addr learned my addr 172.21.15.47:0/2196953395 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T21:56:07.250 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.231+0000 7f103259c700 1 -- 172.21.15.47:0/2196953395 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10340fd920 msgr2=0x7f103410abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:56:07.251 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.231+0000 7f103259c700 1 --2- 172.21.15.47:0/2196953395 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10340fd920 0x7f103410abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:56:07.252 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.231+0000 7f103259c700 1 -- 172.21.15.47:0/2196953395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1034102030 msgr2=0x7f103410b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:56:07.252 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.231+0000 7f103259c700 1 --2- 172.21.15.47:0/2196953395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1034102030 0x7f103410b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:56:07.253 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.231+0000 7f103259c700 1 -- 172.21.15.47:0/2196953395 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103410a390 con 0x7f10340fcf80 2022-09-07T21:56:07.253 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.231+0000 7f1029d9b700 1 --2- 172.21.15.47:0/2196953395 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10340fd920 0x7f103410abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T21:56:07.254 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.231+0000 7f103259c700 1 --2- 172.21.15.47:0/2196953395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10340fcf80 0x7f103410a6b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1034102870 tx=0x7f10240045d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:56:07.255 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.231+0000 7f102bfff700 1 -- 172.21.15.47:0/2196953395 <== mon.1 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1024003cc0 con 0x7f10340fcf80 2022-09-07T21:56:07.255 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.231+0000 7f102bfff700 1 -- 172.21.15.47:0/2196953395 <== mon.1 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1024003e20 con 0x7f10340fcf80 2022-09-07T21:56:07.256 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.231+0000 7f1038fff700 1 -- 172.21.15.47:0/2196953395 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102c0099a0 con 0x7f10340fcf80 2022-09-07T21:56:07.256 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.231+0000 7f102bfff700 1 -- 172.21.15.47:0/2196953395 <== mon.1 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1024004d80 con 0x7f10340fcf80 2022-09-07T21:56:07.257 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.232+0000 7f1038fff700 1 -- 172.21.15.47:0/2196953395 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10341a3170 con 0x7f10340fcf80 2022-09-07T21:56:07.258 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.233+0000 7f1038fff700 1 -- 172.21.15.47:0/2196953395 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1034065cf0 con 0x7f10340fcf80 2022-09-07T21:56:07.258 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.234+0000 7f102bfff700 1 -- 172.21.15.47:0/2196953395 <== mon.1 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 25) v1 ==== 127775+0+0 (secure 0 0 0) 0x7f102400c040 con 0x7f10340fcf80 2022-09-07T21:56:07.259 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.235+0000 7f102bfff700 1 --2- 172.21.15.47:0/2196953395 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f10200990e0 0x7f102009b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T21:56:07.260 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.235+0000 7f102bfff700 1 -- 172.21.15.47:0/2196953395 <== mon.1 v2:172.21.15.59:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 9492+0+0 (secure 0 0 0) 0x7f1024000e40 con 0x7f10340fcf80 2022-09-07T21:56:07.260 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.235+0000 7f1029d9b700 1 --2- 172.21.15.47:0/2196953395 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f10200990e0 0x7f102009b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T21:56:07.261 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.236+0000 7f1029d9b700 1 --2- 172.21.15.47:0/2196953395 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f10200990e0 0x7f102009b590 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f103410bc90 tx=0x7f101c005ea0 comp rx=0 tx=0).ready entity=mgr.25354 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T21:56:07.261 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.238+0000 7f102bfff700 1 -- 172.21.15.47:0/2196953395 <== mon.1 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f1024091670 con 0x7f10340fcf80 2022-09-07T21:56:07.463 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.460+0000 7f1038fff700 1 -- 172.21.15.47:0/2196953395 --> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f103410e7c0 con 0x7f10200990e0 2022-09-07T21:56:07.488 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.484+0000 7f102bfff700 1 -- 172.21.15.47:0/2196953395 <== mgr.25354 v2:172.21.15.47:6800/4004859621 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+335 (secure 0 0 0) 0x7f103410e7c0 con 0x7f10200990e0 2022-09-07T21:56:07.489 INFO:teuthology.orchestra.run.smithi047.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2022-09-07T00:00:00", "created": "2022-09-07T21:41:04", "first": "2022-09-07T21:42:00", "last": "2022-09-07T21:56:00", "last_pruned": "2022-09-07T21:56:00", "created_count": 15, "pruned_count": 9, "active": true} 2022-09-07T21:56:07.490 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.487+0000 7f1038fff700 1 -- 172.21.15.47:0/2196953395 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f10200990e0 msgr2=0x7f102009b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:56:07.492 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.487+0000 7f1038fff700 1 --2- 172.21.15.47:0/2196953395 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f10200990e0 0x7f102009b590 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f103410bc90 tx=0x7f101c005ea0 comp rx=0 tx=0).stop 2022-09-07T21:56:07.492 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.487+0000 7f1038fff700 1 -- 172.21.15.47:0/2196953395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10340fcf80 msgr2=0x7f103410a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T21:56:07.493 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.487+0000 7f1038fff700 1 --2- 172.21.15.47:0/2196953395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10340fcf80 0x7f103410a6b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1034102870 tx=0x7f10240045d0 comp rx=0 tx=0).stop 2022-09-07T21:56:07.493 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.487+0000 7f1038fff700 1 -- 172.21.15.47:0/2196953395 shutdown_connections 2022-09-07T21:56:07.494 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.487+0000 7f1038fff700 1 --2- 172.21.15.47:0/2196953395 >> [v2:172.21.15.47:6800/4004859621,v1:172.21.15.47:6801/4004859621] conn(0x7f10200990e0 0x7f102009b590 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:56:07.495 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.487+0000 7f1038fff700 1 --2- 172.21.15.47:0/2196953395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1034102030 0x7f103410b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:56:07.495 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.487+0000 7f1038fff700 1 --2- 172.21.15.47:0/2196953395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10340fcf80 0x7f103410a6b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:56:07.495 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.487+0000 7f1038fff700 1 --2- 172.21.15.47:0/2196953395 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10340fd920 0x7f103410abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T21:56:07.496 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.487+0000 7f1038fff700 1 -- 172.21.15.47:0/2196953395 >> 172.21.15.47:0/2196953395 conn(0x7f1034073e90 msgr2=0x7f10340745f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T21:56:07.496 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.487+0000 7f1038fff700 1 -- 172.21.15.47:0/2196953395 shutdown_connections 2022-09-07T21:56:07.497 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T21:56:07.487+0000 7f1038fff700 1 -- 172.21.15.47:0/2196953395 wait complete. 2022-09-07T21:56:07.510 DEBUG:teuthology.orchestra.run.smithi047:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2022-09-07T21:56:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:07 smithi055 ceph-mon[97033]: pgmap v477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:56:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:07 smithi059 ceph-mon[97450]: pgmap v477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:56:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:08 smithi055 ceph-mon[97033]: from='client.25441 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:56:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:08 smithi059 ceph-mon[97450]: from='client.25441 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2022-09-07T21:56:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:09 smithi055 ceph-mon[97033]: pgmap v478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:56:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:09 smithi055 ceph-mon[97033]: osdmap e113: 12 total, 12 up, 12 in 2022-09-07T21:56:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:09 smithi059 ceph-mon[97450]: pgmap v478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:56:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:09 smithi059 ceph-mon[97450]: osdmap e113: 12 total, 12 up, 12 in 2022-09-07T21:56:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:11 smithi055 ceph-mon[97033]: pgmap v480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T21:56:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:11 smithi059 ceph-mon[97450]: pgmap v480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T21:56:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:13 smithi059 ceph-mon[97450]: pgmap v481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 22 op/s 2022-09-07T21:56:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:13 smithi055 ceph-mon[97033]: pgmap v481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 22 op/s 2022-09-07T21:56:15.992 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:15 smithi055 ceph-mon[97033]: pgmap v482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 22 op/s 2022-09-07T21:56:16.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:15 smithi059 ceph-mon[97450]: pgmap v482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 22 op/s 2022-09-07T21:56:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:17 smithi055 ceph-mon[97033]: pgmap v483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 204 B/s wr, 7 op/s 2022-09-07T21:56:17.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:56:17.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:56:17.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:56:17.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:17 smithi055 ceph-mon[97033]: osdmap e114: 12 total, 12 up, 12 in 2022-09-07T21:56:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:17 smithi059 ceph-mon[97450]: pgmap v483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 204 B/s wr, 7 op/s 2022-09-07T21:56:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:56:18.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:56:18.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:56:18.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:17 smithi059 ceph-mon[97450]: osdmap e114: 12 total, 12 up, 12 in 2022-09-07T21:56:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:19 smithi055 ceph-mon[97033]: pgmap v485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 205 B/s wr, 7 op/s 2022-09-07T21:56:20.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:19 smithi059 ceph-mon[97450]: pgmap v485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 205 B/s wr, 7 op/s 2022-09-07T21:56:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:21 smithi055 ceph-mon[97033]: pgmap v486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:56:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:21 smithi059 ceph-mon[97450]: pgmap v486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:56:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:23 smithi059 ceph-mon[97450]: pgmap v487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:56:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:23 smithi055 ceph-mon[97033]: pgmap v487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:56:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:25 smithi055 ceph-mon[97033]: pgmap v488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:56:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:25 smithi059 ceph-mon[97450]: pgmap v488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:56:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:27 smithi055 ceph-mon[97033]: pgmap v489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:56:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:27 smithi059 ceph-mon[97450]: pgmap v489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:56:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:29 smithi055 ceph-mon[97033]: pgmap v490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-07T21:56:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:29 smithi059 ceph-mon[97450]: pgmap v490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-07T21:56:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:31 smithi055 ceph-mon[97033]: pgmap v491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:56:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:31 smithi059 ceph-mon[97450]: pgmap v491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:56:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:33 smithi059 ceph-mon[97450]: pgmap v492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:56:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:33 smithi055 ceph-mon[97033]: pgmap v492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:56:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:35 smithi055 ceph-mon[97033]: pgmap v493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:56:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:35 smithi059 ceph-mon[97450]: pgmap v493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:56:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:37 smithi055 ceph-mon[97033]: pgmap v494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:56:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:37 smithi059 ceph-mon[97450]: pgmap v494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:56:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:39 smithi055 ceph-mon[97033]: pgmap v495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:56:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:39 smithi059 ceph-mon[97450]: pgmap v495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:56:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:41 smithi055 ceph-mon[97033]: pgmap v496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:56:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:41 smithi059 ceph-mon[97450]: pgmap v496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:56:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:43 smithi059 ceph-mon[97450]: pgmap v497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:56:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:43 smithi055 ceph-mon[97033]: pgmap v497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:56:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:45 smithi055 ceph-mon[97033]: pgmap v498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:56:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:45 smithi059 ceph-mon[97450]: pgmap v498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:56:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:47 smithi055 ceph-mon[97033]: pgmap v499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:56:48.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:47 smithi055 ceph-mon[97033]: pgmap v500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:56:48.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:47 smithi055 ceph-mon[97033]: pgmap v501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T21:56:48.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:56:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:47 smithi059 ceph-mon[97450]: pgmap v499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:56:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:47 smithi059 ceph-mon[97450]: pgmap v500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:56:48.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:47 smithi059 ceph-mon[97450]: pgmap v501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T21:56:48.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:56:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:49 smithi055 ceph-mon[97033]: pgmap v502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-07T21:56:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:49 smithi059 ceph-mon[97450]: pgmap v502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-07T21:56:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:51 smithi055 ceph-mon[97033]: pgmap v503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2022-09-07T21:56:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:51 smithi059 ceph-mon[97450]: pgmap v503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2022-09-07T21:56:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:53 smithi055 ceph-mon[97033]: pgmap v504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T21:56:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:53 smithi059 ceph-mon[97450]: pgmap v504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T21:56:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:55 smithi055 ceph-mon[97033]: pgmap v505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-07T21:56:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:55 smithi059 ceph-mon[97450]: pgmap v505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-07T21:56:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:57 smithi055 ceph-mon[97033]: pgmap v506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:56:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:57 smithi059 ceph-mon[97450]: pgmap v506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:57:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:56:59 smithi055 ceph-mon[97033]: pgmap v507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:57:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:56:59 smithi059 ceph-mon[97450]: pgmap v507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:57:02.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:01 smithi055 ceph-mon[97033]: pgmap v508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:57:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:01 smithi059 ceph-mon[97450]: pgmap v508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:57:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:03 smithi059 ceph-mon[97450]: pgmap v509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 8 op/s 2022-09-07T21:57:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:03 smithi055 ceph-mon[97033]: pgmap v509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 8 op/s 2022-09-07T21:57:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:57:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:57:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:57:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:57:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:05 smithi055 ceph-mon[97033]: pgmap v510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:57:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:05 smithi059 ceph-mon[97450]: pgmap v510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:57:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:07 smithi055 ceph-mon[97033]: pgmap v511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:57:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:07 smithi059 ceph-mon[97450]: pgmap v511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:57:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:09 smithi055 ceph-mon[97033]: pgmap v512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:57:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:09 smithi055 ceph-mon[97033]: osdmap e115: 12 total, 12 up, 12 in 2022-09-07T21:57:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:09 smithi059 ceph-mon[97450]: pgmap v512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:57:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:09 smithi059 ceph-mon[97450]: osdmap e115: 12 total, 12 up, 12 in 2022-09-07T21:57:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:11 smithi055 ceph-mon[97033]: pgmap v514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:57:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:11 smithi059 ceph-mon[97450]: pgmap v514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:57:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:13 smithi055 ceph-mon[97033]: pgmap v515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T21:57:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:13 smithi059 ceph-mon[97450]: pgmap v515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T21:57:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:15 smithi055 ceph-mon[97033]: pgmap v516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T21:57:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:15 smithi059 ceph-mon[97450]: pgmap v516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T21:57:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:17 smithi055 ceph-mon[97033]: pgmap v517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:57:18.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:17 smithi055 ceph-mon[97033]: osdmap e116: 12 total, 12 up, 12 in 2022-09-07T21:57:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:17 smithi059 ceph-mon[97450]: pgmap v517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:57:18.364 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:17 smithi059 ceph-mon[97450]: osdmap e116: 12 total, 12 up, 12 in 2022-09-07T21:57:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:19 smithi055 ceph-mon[97033]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T21:57:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:19 smithi059 ceph-mon[97450]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T21:57:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:21 smithi055 ceph-mon[97033]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:57:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:21 smithi059 ceph-mon[97450]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:57:23.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:23 smithi055 ceph-mon[97033]: pgmap v521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:57:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:23 smithi059 ceph-mon[97450]: pgmap v521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:57:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:25 smithi055 ceph-mon[97033]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:57:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:25 smithi059 ceph-mon[97450]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:57:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:27 smithi055 ceph-mon[97033]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:57:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:27 smithi059 ceph-mon[97450]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:57:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:29 smithi055 ceph-mon[97033]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-07T21:57:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:29 smithi059 ceph-mon[97450]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-07T21:57:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:31 smithi055 ceph-mon[97033]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:57:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:31 smithi059 ceph-mon[97450]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:57:33.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:33 smithi055 ceph-mon[97033]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:57:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:33 smithi059 ceph-mon[97450]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:57:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:35 smithi055 ceph-mon[97033]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:57:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:35 smithi059 ceph-mon[97450]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:57:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:37 smithi055 ceph-mon[97033]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:57:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:37 smithi059 ceph-mon[97450]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:57:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:39 smithi055 ceph-mon[97033]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:57:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:39 smithi059 ceph-mon[97450]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:57:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:41 smithi055 ceph-mon[97033]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:57:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:41 smithi059 ceph-mon[97450]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:57:43.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:43 smithi055 ceph-mon[97033]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:57:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:43 smithi059 ceph-mon[97450]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:57:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:45 smithi055 ceph-mon[97033]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:57:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:45 smithi059 ceph-mon[97450]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:57:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:47 smithi055 ceph-mon[97033]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:57:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:57:48.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:57:48.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:57:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:47 smithi059 ceph-mon[97450]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:57:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:57:48.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:57:48.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:57:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:49 smithi055 ceph-mon[97033]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:57:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:49 smithi059 ceph-mon[97450]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:57:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:51 smithi055 ceph-mon[97033]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:57:52.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:51 smithi059 ceph-mon[97450]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:57:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:53 smithi055 ceph-mon[97033]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:57:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:53 smithi059 ceph-mon[97450]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:57:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:55 smithi055 ceph-mon[97033]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:57:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:55 smithi059 ceph-mon[97450]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T21:57:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:57 smithi055 ceph-mon[97033]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:57:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:57 smithi059 ceph-mon[97450]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:58:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:57:59 smithi055 ceph-mon[97033]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:58:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:57:59 smithi059 ceph-mon[97450]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:58:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:01 smithi055 ceph-mon[97033]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:58:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:01 smithi059 ceph-mon[97450]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T21:58:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:03 smithi055 ceph-mon[97033]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 8 op/s 2022-09-07T21:58:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:03 smithi059 ceph-mon[97450]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 8 op/s 2022-09-07T21:58:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:58:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:58:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:58:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:58:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:05 smithi055 ceph-mon[97033]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:58:06.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:05 smithi059 ceph-mon[97450]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T21:58:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:07 smithi055 ceph-mon[97033]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:58:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:07 smithi059 ceph-mon[97450]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:58:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:09 smithi055 ceph-mon[97033]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:58:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:09 smithi055 ceph-mon[97033]: osdmap e117: 12 total, 12 up, 12 in 2022-09-07T21:58:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:09 smithi059 ceph-mon[97450]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:58:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:09 smithi059 ceph-mon[97450]: osdmap e117: 12 total, 12 up, 12 in 2022-09-07T21:58:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:11 smithi055 ceph-mon[97033]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:58:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:11 smithi059 ceph-mon[97450]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:58:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:13 smithi055 ceph-mon[97033]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T21:58:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:13 smithi059 ceph-mon[97450]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T21:58:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:15 smithi055 ceph-mon[97033]: pgmap v548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T21:58:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:15 smithi059 ceph-mon[97450]: pgmap v548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T21:58:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:18 smithi055 ceph-mon[97033]: pgmap v549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:58:18.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:18 smithi055 ceph-mon[97033]: pgmap v550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-07T21:58:18.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:18 smithi055 ceph-mon[97033]: pgmap v551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-07T21:58:18.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:58:18.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:18 smithi055 ceph-mon[97033]: osdmap e118: 12 total, 12 up, 12 in 2022-09-07T21:58:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:18 smithi059 ceph-mon[97450]: pgmap v549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:58:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:18 smithi059 ceph-mon[97450]: pgmap v550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-07T21:58:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:18 smithi059 ceph-mon[97450]: pgmap v551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-07T21:58:18.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:58:18.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:18 smithi059 ceph-mon[97450]: osdmap e118: 12 total, 12 up, 12 in 2022-09-07T21:58:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:20 smithi055 ceph-mon[97033]: pgmap v553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-07T21:58:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:20 smithi059 ceph-mon[97450]: pgmap v553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-07T21:58:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:22 smithi055 ceph-mon[97033]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 164 B/s wr, 9 op/s 2022-09-07T21:58:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:22 smithi059 ceph-mon[97450]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 164 B/s wr, 9 op/s 2022-09-07T21:58:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:23 smithi055 ceph-mon[97033]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-07T21:58:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:23 smithi059 ceph-mon[97450]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-07T21:58:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:26 smithi055 ceph-mon[97033]: pgmap v556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2022-09-07T21:58:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:26 smithi059 ceph-mon[97450]: pgmap v556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2022-09-07T21:58:28.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:28 smithi055 ceph-mon[97033]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:58:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:28 smithi059 ceph-mon[97450]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:58:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:30 smithi055 ceph-mon[97033]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T21:58:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:30 smithi059 ceph-mon[97450]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T21:58:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:32 smithi055 ceph-mon[97033]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:58:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:32 smithi059 ceph-mon[97450]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:58:33.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:33 smithi055 ceph-mon[97033]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:58:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:33 smithi059 ceph-mon[97450]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:58:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:36 smithi055 ceph-mon[97033]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:58:36.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:36 smithi059 ceph-mon[97450]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:58:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:38 smithi055 ceph-mon[97033]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T21:58:38.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:38 smithi059 ceph-mon[97450]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T21:58:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:40 smithi055 ceph-mon[97033]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:58:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:40 smithi059 ceph-mon[97450]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:58:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:42 smithi055 ceph-mon[97033]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:58:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:42 smithi059 ceph-mon[97450]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:58:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:43 smithi055 ceph-mon[97033]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:58:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:43 smithi059 ceph-mon[97450]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:58:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:46 smithi055 ceph-mon[97033]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:58:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:46 smithi059 ceph-mon[97450]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:58:48.488 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:48 smithi055 ceph-mon[97033]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T21:58:48.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:48 smithi059 ceph-mon[97450]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T21:58:50.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:50 smithi055 ceph-mon[97033]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:58:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:50 smithi059 ceph-mon[97450]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:58:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:52 smithi055 ceph-mon[97033]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:58:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:52 smithi059 ceph-mon[97450]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:58:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:53 smithi055 ceph-mon[97033]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:58:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:53 smithi059 ceph-mon[97450]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:58:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:56 smithi055 ceph-mon[97033]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:58:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:56 smithi059 ceph-mon[97450]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:58:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:58:58 smithi055 ceph-mon[97033]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T21:58:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:58:58 smithi059 ceph-mon[97450]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T21:59:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:00 smithi055 ceph-mon[97033]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:59:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:00 smithi059 ceph-mon[97450]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:59:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:02 smithi055 ceph-mon[97033]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:59:02.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:02 smithi059 ceph-mon[97450]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:59:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:03 smithi055 ceph-mon[97033]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-07T21:59:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:03 smithi059 ceph-mon[97450]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-07T21:59:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:59:04.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:59:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T21:59:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T21:59:05.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:05 smithi055 ceph-mon[97033]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 8 op/s 2022-09-07T21:59:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:05 smithi059 ceph-mon[97450]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 8 op/s 2022-09-07T21:59:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:08 smithi055 ceph-mon[97033]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:59:08.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:08 smithi059 ceph-mon[97450]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T21:59:10.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:10 smithi055 ceph-mon[97033]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:59:10.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:10 smithi055 ceph-mon[97033]: osdmap e119: 12 total, 12 up, 12 in 2022-09-07T21:59:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:10 smithi059 ceph-mon[97450]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T21:59:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:10 smithi059 ceph-mon[97450]: osdmap e119: 12 total, 12 up, 12 in 2022-09-07T21:59:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:12 smithi055 ceph-mon[97033]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:59:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:12 smithi059 ceph-mon[97450]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T21:59:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:13 smithi055 ceph-mon[97033]: pgmap v581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T21:59:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:13 smithi059 ceph-mon[97450]: pgmap v581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T21:59:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:16 smithi055 ceph-mon[97033]: pgmap v582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T21:59:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:16 smithi059 ceph-mon[97450]: pgmap v582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T21:59:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:18 smithi055 ceph-mon[97033]: pgmap v583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:59:18.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:59:18.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:59:18.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:59:18.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:18 smithi055 ceph-mon[97033]: osdmap e120: 12 total, 12 up, 12 in 2022-09-07T21:59:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:18 smithi059 ceph-mon[97450]: pgmap v583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T21:59:18.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T21:59:18.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T21:59:18.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T21:59:18.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:18 smithi059 ceph-mon[97450]: osdmap e120: 12 total, 12 up, 12 in 2022-09-07T21:59:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:20 smithi055 ceph-mon[97033]: pgmap v585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T21:59:20.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:20 smithi059 ceph-mon[97450]: pgmap v585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T21:59:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:22 smithi055 ceph-mon[97033]: pgmap v586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:59:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:22 smithi059 ceph-mon[97450]: pgmap v586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:59:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:23 smithi055 ceph-mon[97033]: pgmap v587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:59:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:23 smithi059 ceph-mon[97450]: pgmap v587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:59:26.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:26 smithi055 ceph-mon[97033]: pgmap v588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:59:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:26 smithi059 ceph-mon[97450]: pgmap v588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T21:59:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:28 smithi055 ceph-mon[97033]: pgmap v589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:59:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:28 smithi059 ceph-mon[97450]: pgmap v589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:59:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:30 smithi055 ceph-mon[97033]: pgmap v590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T21:59:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:30 smithi059 ceph-mon[97450]: pgmap v590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T21:59:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:32 smithi055 ceph-mon[97033]: pgmap v591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:59:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:32 smithi059 ceph-mon[97450]: pgmap v591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:59:33.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:33 smithi055 ceph-mon[97033]: pgmap v592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:59:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:33 smithi059 ceph-mon[97450]: pgmap v592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:59:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:36 smithi055 ceph-mon[97033]: pgmap v593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:59:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:36 smithi059 ceph-mon[97450]: pgmap v593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:59:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:38 smithi055 ceph-mon[97033]: pgmap v594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T21:59:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:38 smithi059 ceph-mon[97450]: pgmap v594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T21:59:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:40 smithi055 ceph-mon[97033]: pgmap v595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:59:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:40 smithi059 ceph-mon[97450]: pgmap v595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:59:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:42 smithi055 ceph-mon[97033]: pgmap v596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:59:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:42 smithi059 ceph-mon[97450]: pgmap v596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T21:59:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:43 smithi055 ceph-mon[97033]: pgmap v597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:59:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:43 smithi059 ceph-mon[97450]: pgmap v597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T21:59:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:46 smithi055 ceph-mon[97033]: pgmap v598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:59:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:46 smithi059 ceph-mon[97450]: pgmap v598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T21:59:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:48 smithi055 ceph-mon[97033]: pgmap v599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T21:59:48.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:59:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:48 smithi059 ceph-mon[97450]: pgmap v599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T21:59:48.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T21:59:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:49 smithi055 ceph-mon[97033]: pgmap v600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:59:49.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:49 smithi055 ceph-mon[97033]: pgmap v601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T21:59:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:49 smithi059 ceph-mon[97450]: pgmap v600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T21:59:49.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:49 smithi059 ceph-mon[97450]: pgmap v601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T21:59:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:51 smithi055 ceph-mon[97033]: pgmap v602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-07T21:59:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:51 smithi059 ceph-mon[97450]: pgmap v602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-07T21:59:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:53 smithi055 ceph-mon[97033]: pgmap v603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T21:59:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:53 smithi059 ceph-mon[97450]: pgmap v603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T21:59:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:55 smithi055 ceph-mon[97033]: pgmap v604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2022-09-07T21:59:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:55 smithi059 ceph-mon[97450]: pgmap v604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2022-09-07T21:59:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:57 smithi055 ceph-mon[97033]: pgmap v605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T21:59:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:57 smithi059 ceph-mon[97450]: pgmap v605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T21:59:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 21:59:59 smithi055 ceph-mon[97033]: pgmap v606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T21:59:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 21:59:59 smithi059 ceph-mon[97450]: pgmap v606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:00:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-07T22:00:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-07T22:00:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-07T22:00:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T22:00:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T22:00:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-07T22:00:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-07T22:00:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-07T22:00:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T22:00:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T22:00:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:01 smithi055 ceph-mon[97033]: pgmap v607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:01 smithi059 ceph-mon[97450]: pgmap v607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:03 smithi055 ceph-mon[97033]: pgmap v608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 KiB/s wr, 11 op/s 2022-09-07T22:00:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:03 smithi059 ceph-mon[97450]: pgmap v608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 KiB/s wr, 11 op/s 2022-09-07T22:00:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:05 smithi055 ceph-mon[97033]: pgmap v609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 33 KiB/s wr, 12 op/s 2022-09-07T22:00:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:00:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:00:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:05 smithi059 ceph-mon[97450]: pgmap v609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 33 KiB/s wr, 12 op/s 2022-09-07T22:00:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:00:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:00:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:07 smithi055 ceph-mon[97033]: pgmap v610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T22:00:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:07 smithi059 ceph-mon[97450]: pgmap v610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T22:00:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:09 smithi055 ceph-mon[97033]: pgmap v611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:00:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:09 smithi055 ceph-mon[97033]: osdmap e121: 12 total, 12 up, 12 in 2022-09-07T22:00:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:09 smithi059 ceph-mon[97450]: pgmap v611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:00:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:09 smithi059 ceph-mon[97450]: osdmap e121: 12 total, 12 up, 12 in 2022-09-07T22:00:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:11 smithi055 ceph-mon[97033]: pgmap v613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:00:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:11 smithi059 ceph-mon[97450]: pgmap v613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:00:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:13 smithi055 ceph-mon[97033]: pgmap v614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 40 KiB/s wr, 18 op/s 2022-09-07T22:00:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:13 smithi059 ceph-mon[97450]: pgmap v614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 40 KiB/s wr, 18 op/s 2022-09-07T22:00:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:15 smithi055 ceph-mon[97033]: pgmap v615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 13 KiB/s wr, 15 op/s 2022-09-07T22:00:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:15 smithi059 ceph-mon[97450]: pgmap v615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 13 KiB/s wr, 15 op/s 2022-09-07T22:00:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:16 smithi059 ceph-mon[97450]: pgmap v616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:00:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:16 smithi055 ceph-mon[97033]: pgmap v616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:00:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:18 smithi055 ceph-mon[97033]: pgmap v617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:00:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:18 smithi055 ceph-mon[97033]: osdmap e122: 12 total, 12 up, 12 in 2022-09-07T22:00:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:18 smithi059 ceph-mon[97450]: pgmap v617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:00:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:18 smithi059 ceph-mon[97450]: osdmap e122: 12 total, 12 up, 12 in 2022-09-07T22:00:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:20 smithi055 ceph-mon[97033]: pgmap v619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:00:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:20 smithi059 ceph-mon[97450]: pgmap v619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:00:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:22 smithi055 ceph-mon[97033]: pgmap v620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:00:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:22 smithi059 ceph-mon[97450]: pgmap v620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:00:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:24 smithi055 ceph-mon[97033]: pgmap v621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:00:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:24 smithi059 ceph-mon[97450]: pgmap v621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:00:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:26 smithi055 ceph-mon[97033]: pgmap v622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:00:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:26 smithi059 ceph-mon[97450]: pgmap v622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:00:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:28 smithi055 ceph-mon[97033]: pgmap v623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:00:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:28 smithi059 ceph-mon[97450]: pgmap v623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:00:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:30 smithi055 ceph-mon[97033]: pgmap v624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:30 smithi059 ceph-mon[97450]: pgmap v624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:32 smithi055 ceph-mon[97033]: pgmap v625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:00:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:32 smithi059 ceph-mon[97450]: pgmap v625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:00:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:34 smithi055 ceph-mon[97033]: pgmap v626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:34 smithi059 ceph-mon[97450]: pgmap v626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:36 smithi055 ceph-mon[97033]: pgmap v627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:00:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:36 smithi059 ceph-mon[97450]: pgmap v627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:00:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:38 smithi055 ceph-mon[97033]: pgmap v628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:38 smithi059 ceph-mon[97450]: pgmap v628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:40 smithi055 ceph-mon[97033]: pgmap v629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:40 smithi059 ceph-mon[97450]: pgmap v629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:42 smithi055 ceph-mon[97033]: pgmap v630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:00:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:42 smithi059 ceph-mon[97450]: pgmap v630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:00:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:44 smithi055 ceph-mon[97033]: pgmap v631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:44 smithi059 ceph-mon[97450]: pgmap v631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:46 smithi055 ceph-mon[97033]: pgmap v632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:00:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:46 smithi059 ceph-mon[97450]: pgmap v632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:00:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:00:47.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:00:47.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:00:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:00:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:00:48.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:00:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:48 smithi055 ceph-mon[97033]: pgmap v633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:48 smithi059 ceph-mon[97450]: pgmap v633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:50 smithi055 ceph-mon[97033]: pgmap v634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:50 smithi059 ceph-mon[97450]: pgmap v634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:52 smithi055 ceph-mon[97033]: pgmap v635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:00:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:52 smithi059 ceph-mon[97450]: pgmap v635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:00:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:54 smithi055 ceph-mon[97033]: pgmap v636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:54 smithi059 ceph-mon[97450]: pgmap v636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:56.991 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:56 smithi055 ceph-mon[97033]: pgmap v637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:00:57.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:56 smithi059 ceph-mon[97450]: pgmap v637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:00:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:00:58 smithi055 ceph-mon[97033]: pgmap v638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:00:59.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:00:58 smithi059 ceph-mon[97450]: pgmap v638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:01:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:00 smithi055 ceph-mon[97033]: pgmap v639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:01:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:00 smithi059 ceph-mon[97450]: pgmap v639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:01:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:02 smithi055 ceph-mon[97033]: pgmap v640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2022-09-07T22:01:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:02 smithi059 ceph-mon[97450]: pgmap v640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2022-09-07T22:01:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:04 smithi055 ceph-mon[97033]: pgmap v641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T22:01:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:01:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:01:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:04 smithi059 ceph-mon[97450]: pgmap v641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T22:01:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:01:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:01:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:06 smithi055 ceph-mon[97033]: pgmap v642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T22:01:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:06 smithi059 ceph-mon[97450]: pgmap v642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T22:01:07.671 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.668+0000 7f450b7ff700 -1 monclient: get_monmap_and_config failed to get config 2022-09-07T22:01:07.674 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.670+0000 7f450b7ff700 1 -- 172.21.15.47:0/1060448711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4504067610 msgr2=0x7f45040679e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T22:01:07.675 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.670+0000 7f450b7ff700 1 --2- 172.21.15.47:0/1060448711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4504067610 0x7f45040679e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f450000b990 tx=0x7f450000bca0 comp rx=0 tx=0).stop 2022-09-07T22:01:07.675 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.671+0000 7f450b7ff700 1 -- 172.21.15.47:0/1060448711 shutdown_connections 2022-09-07T22:01:07.676 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.671+0000 7f450b7ff700 1 --2- 172.21.15.47:0/1060448711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4504067fb0 0x7f45040ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T22:01:07.677 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.671+0000 7f450b7ff700 1 --2- 172.21.15.47:0/1060448711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4504100110 0x7f450410ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T22:01:07.677 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.671+0000 7f450b7ff700 1 --2- 172.21.15.47:0/1060448711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4504067610 0x7f45040679e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T22:01:07.678 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.671+0000 7f450b7ff700 1 -- 172.21.15.47:0/1060448711 >> 172.21.15.47:0/1060448711 conn(0x7f4504073ef0 msgr2=0x7f45040742f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T22:01:07.679 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.671+0000 7f450b7ff700 1 -- 172.21.15.47:0/1060448711 shutdown_connections 2022-09-07T22:01:07.679 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.671+0000 7f450b7ff700 1 -- 172.21.15.47:0/1060448711 wait complete. 2022-09-07T22:01:07.682 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.672+0000 7f450b7ff700 1 Processor -- start 2022-09-07T22:01:07.683 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.672+0000 7f450b7ff700 1 -- start start 2022-09-07T22:01:07.683 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.673+0000 7f450b7ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4504067610 0x7f450419c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T22:01:07.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.673+0000 7f450b7ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4504067fb0 0x7f450419ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T22:01:07.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.673+0000 7f450b7ff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4504100110 0x7f45041967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-07T22:01:07.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.673+0000 7f450b7ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4504196f50 con 0x7f4504067fb0 2022-09-07T22:01:07.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.673+0000 7f450b7ff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45041970c0 con 0x7f4504100110 2022-09-07T22:01:07.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.673+0000 7f450b7ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4504197280 con 0x7f4504067610 2022-09-07T22:01:07.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.673+0000 7f4508d9a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4504067fb0 0x7f450419ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T22:01:07.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.673+0000 7f4508d9a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4504067fb0 0x7f450419ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:41496/0 (socket says 172.21.15.47:41496) 2022-09-07T22:01:07.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.673+0000 7f4508d9a700 1 -- 172.21.15.47:0/1459530205 learned_addr learned my addr 172.21.15.47:0/1459530205 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-09-07T22:01:07.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.673+0000 7f4509d9c700 1 --2- 172.21.15.47:0/1459530205 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4504100110 0x7f45041967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T22:01:07.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.673+0000 7f450959b700 1 --2- 172.21.15.47:0/1459530205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4504067610 0x7f450419c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-07T22:01:07.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.674+0000 7f4508d9a700 1 -- 172.21.15.47:0/1459530205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4504067610 msgr2=0x7f450419c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T22:01:07.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.674+0000 7f4508d9a700 1 --2- 172.21.15.47:0/1459530205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4504067610 0x7f450419c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T22:01:07.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.674+0000 7f4508d9a700 1 -- 172.21.15.47:0/1459530205 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4504100110 msgr2=0x7f45041967b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T22:01:07.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.674+0000 7f4508d9a700 1 --2- 172.21.15.47:0/1459530205 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4504100110 0x7f45041967b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T22:01:07.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.674+0000 7f4508d9a700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450000b5c0 con 0x7f4504067fb0 2022-09-07T22:01:07.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.674+0000 7f4509d9c700 1 --2- 172.21.15.47:0/1459530205 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4504100110 0x7f45041967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T22:01:07.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.674+0000 7f450959b700 1 --2- 172.21.15.47:0/1459530205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4504067610 0x7f450419c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-07T22:01:07.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:01:07.674+0000 7f4508d9a700 1 --2- 172.21.15.47:0/1459530205 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4504067fb0 0x7f450419ca70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f44f4024040 tx=0x7f44f4004640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-07T22:01:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:08 smithi055 ceph-mon[97033]: pgmap v643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:01:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:08 smithi059 ceph-mon[97450]: pgmap v643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:01:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:09 smithi055 ceph-mon[97033]: osdmap e123: 12 total, 12 up, 12 in 2022-09-07T22:01:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:09 smithi059 ceph-mon[97450]: osdmap e123: 12 total, 12 up, 12 in 2022-09-07T22:01:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:10 smithi055 ceph-mon[97033]: pgmap v645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:01:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:10 smithi059 ceph-mon[97450]: pgmap v645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:01:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:12 smithi055 ceph-mon[97033]: pgmap v646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:01:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:12 smithi059 ceph-mon[97450]: pgmap v646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:01:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:14 smithi055 ceph-mon[97033]: pgmap v647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:01:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:14 smithi059 ceph-mon[97450]: pgmap v647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:01:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:16 smithi055 ceph-mon[97033]: pgmap v648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:01:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:16 smithi059 ceph-mon[97450]: pgmap v648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:01:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:18 smithi055 ceph-mon[97033]: pgmap v649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:01:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:18 smithi055 ceph-mon[97033]: osdmap e124: 12 total, 12 up, 12 in 2022-09-07T22:01:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:18 smithi059 ceph-mon[97450]: pgmap v649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:01:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:18 smithi059 ceph-mon[97450]: osdmap e124: 12 total, 12 up, 12 in 2022-09-07T22:01:20.862 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:20 smithi059 ceph-mon[97450]: pgmap v651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:01:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:20 smithi055 ceph-mon[97033]: pgmap v651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:01:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:01:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:01:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:21 smithi059 ceph-mon[97450]: pgmap v652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 109 B/s wr, 6 op/s 2022-09-07T22:01:22.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:01:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:01:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:01:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:21 smithi055 ceph-mon[97033]: pgmap v652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 109 B/s wr, 6 op/s 2022-09-07T22:01:22.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:01:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:23 smithi055 ceph-mon[97033]: pgmap v653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 109 B/s wr, 6 op/s 2022-09-07T22:01:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:23 smithi059 ceph-mon[97450]: pgmap v653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 109 B/s wr, 6 op/s 2022-09-07T22:01:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:25 smithi055 ceph-mon[97033]: pgmap v654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 109 B/s wr, 4 op/s 2022-09-07T22:01:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:25 smithi059 ceph-mon[97450]: pgmap v654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 109 B/s wr, 4 op/s 2022-09-07T22:01:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:27 smithi055 ceph-mon[97033]: pgmap v655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-07T22:01:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:27 smithi059 ceph-mon[97450]: pgmap v655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-07T22:01:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:29 smithi055 ceph-mon[97033]: pgmap v656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:01:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:29 smithi059 ceph-mon[97450]: pgmap v656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:01:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:31 smithi055 ceph-mon[97033]: pgmap v657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2022-09-07T22:01:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:31 smithi059 ceph-mon[97450]: pgmap v657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2022-09-07T22:01:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:33 smithi055 ceph-mon[97033]: pgmap v658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-07T22:01:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:33 smithi059 ceph-mon[97450]: pgmap v658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-07T22:01:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:35 smithi055 ceph-mon[97033]: pgmap v659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:01:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:35 smithi059 ceph-mon[97450]: pgmap v659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:01:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:37 smithi055 ceph-mon[97033]: pgmap v660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:01:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:37 smithi059 ceph-mon[97450]: pgmap v660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:01:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:39 smithi055 ceph-mon[97033]: pgmap v661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:01:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:39 smithi059 ceph-mon[97450]: pgmap v661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:01:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:41 smithi055 ceph-mon[97033]: pgmap v662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:01:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:41 smithi059 ceph-mon[97450]: pgmap v662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:01:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:43 smithi055 ceph-mon[97033]: pgmap v663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:01:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:43 smithi059 ceph-mon[97450]: pgmap v663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:01:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:45 smithi055 ceph-mon[97033]: pgmap v664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:01:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:45 smithi059 ceph-mon[97450]: pgmap v664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:01:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:47 smithi055 ceph-mon[97033]: pgmap v665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:01:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:47 smithi059 ceph-mon[97450]: pgmap v665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:01:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:49 smithi055 ceph-mon[97033]: pgmap v666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:01:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:49 smithi059 ceph-mon[97450]: pgmap v666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:01:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:51 smithi055 ceph-mon[97033]: pgmap v667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:01:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:51 smithi059 ceph-mon[97450]: pgmap v667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:01:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:53 smithi055 ceph-mon[97033]: pgmap v668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:01:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:53 smithi059 ceph-mon[97450]: pgmap v668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:01:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:55 smithi055 ceph-mon[97033]: pgmap v669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:01:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:55 smithi059 ceph-mon[97450]: pgmap v669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:01:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:57 smithi055 ceph-mon[97033]: pgmap v670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:01:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:57 smithi059 ceph-mon[97450]: pgmap v670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:02:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:01:59 smithi055 ceph-mon[97033]: pgmap v671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:02:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:01:59 smithi059 ceph-mon[97450]: pgmap v671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:02:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:01 smithi055 ceph-mon[97033]: pgmap v672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:02:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:01 smithi059 ceph-mon[97450]: pgmap v672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:02:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:03 smithi055 ceph-mon[97033]: pgmap v673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 8 op/s 2022-09-07T22:02:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:03 smithi059 ceph-mon[97450]: pgmap v673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 8 op/s 2022-09-07T22:02:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:02:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:02:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:02:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:02:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:05 smithi055 ceph-mon[97033]: pgmap v674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T22:02:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:05 smithi059 ceph-mon[97450]: pgmap v674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T22:02:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:07 smithi055 ceph-mon[97033]: pgmap v675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T22:02:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:07 smithi059 ceph-mon[97450]: pgmap v675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T22:02:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:09 smithi055 ceph-mon[97033]: pgmap v676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:02:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:09 smithi055 ceph-mon[97033]: osdmap e125: 12 total, 12 up, 12 in 2022-09-07T22:02:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:09 smithi059 ceph-mon[97450]: pgmap v676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:02:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:09 smithi059 ceph-mon[97450]: osdmap e125: 12 total, 12 up, 12 in 2022-09-07T22:02:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:11 smithi055 ceph-mon[97033]: pgmap v678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:02:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:11 smithi059 ceph-mon[97450]: pgmap v678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:02:12.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:12.683+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0033d0 con 0x7f4504067fb0 2022-09-07T22:02:13.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:13.683+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0045f0 con 0x7f4504067fb0 2022-09-07T22:02:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:13 smithi055 ceph-mon[97033]: pgmap v679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:02:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:13 smithi059 ceph-mon[97450]: pgmap v679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:02:14.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:14.683+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0045f0 con 0x7f4504067fb0 2022-09-07T22:02:15.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:15.684+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0045f0 con 0x7f4504067fb0 2022-09-07T22:02:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:15 smithi055 ceph-mon[97033]: pgmap v680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:02:16.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:15 smithi059 ceph-mon[97450]: pgmap v680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:02:16.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:16.682+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0045f0 con 0x7f4504067fb0 2022-09-07T22:02:17.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:17.683+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0045f0 con 0x7f4504067fb0 2022-09-07T22:02:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:17 smithi055 ceph-mon[97033]: pgmap v681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:02:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:17 smithi055 ceph-mon[97033]: osdmap e126: 12 total, 12 up, 12 in 2022-09-07T22:02:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:17 smithi059 ceph-mon[97450]: pgmap v681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:02:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:17 smithi059 ceph-mon[97450]: osdmap e126: 12 total, 12 up, 12 in 2022-09-07T22:02:18.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:18.683+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0045f0 con 0x7f4504067fb0 2022-09-07T22:02:19.686 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:19.683+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0045f0 con 0x7f4504067fb0 2022-09-07T22:02:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:19 smithi055 ceph-mon[97033]: pgmap v683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T22:02:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:19 smithi059 ceph-mon[97450]: pgmap v683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T22:02:20.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:20.684+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0045f0 con 0x7f4504067fb0 2022-09-07T22:02:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:21.684+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0045f0 con 0x7f4504067fb0 2022-09-07T22:02:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:21 smithi055 ceph-mon[97033]: pgmap v684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:02:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:02:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:02:22.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:02:22.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:02:22.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:02:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:21 smithi059 ceph-mon[97450]: pgmap v684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:02:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:02:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:02:22.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:02:22.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:02:22.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:02:22.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:22.684+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0045f0 con 0x7f4504067fb0 2022-09-07T22:02:23.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:23.684+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:23 smithi055 ceph-mon[97033]: pgmap v685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:02:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:23 smithi059 ceph-mon[97450]: pgmap v685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:02:24.687 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:24.685+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0045a0 con 0x7f4504067fb0 2022-09-07T22:02:24.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:02:24.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:02:24.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:24.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:24.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:24.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:24.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:24 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T22:02:24.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:24 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T22:02:24.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:02:24.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:02:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:02:25.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:02:25.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:25.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:25.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:25.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:25.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:24 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T22:02:25.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:24 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T22:02:25.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:02:25.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:02:25.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:25.685+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:25 smithi059 ceph-mon[97450]: pgmap v686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:02:26.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:26.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:26.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:26.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:26.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:25 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T22:02:26.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:25 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T22:02:26.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:25 smithi059 ceph-mon[97450]: pgmap v687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:02:26.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:02:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:25 smithi055 ceph-mon[97033]: pgmap v686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:02:26.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:26.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:26.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:26.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:02:26.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:25 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T22:02:26.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:25 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T22:02:26.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:25 smithi055 ceph-mon[97033]: pgmap v687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:02:26.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:02:26.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:26.684+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0045a0 con 0x7f4504067fb0 2022-09-07T22:02:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:26 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-07T22:02:27.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:26 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-07T22:02:27.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:27.685+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0045a0 con 0x7f4504067fb0 2022-09-07T22:02:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:27 smithi059 ceph-mon[97450]: pgmap v688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:02:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:27 smithi055 ceph-mon[97033]: pgmap v688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:02:28.688 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:28.686+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec0045a0 con 0x7f4504067fb0 2022-09-07T22:02:29.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:29.686+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:30 smithi055 ceph-mon[97033]: pgmap v689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:02:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:30 smithi059 ceph-mon[97450]: pgmap v689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:02:30.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:30.686+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:31.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:31.686+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:32 smithi059 ceph-mon[97450]: pgmap v690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2022-09-07T22:02:32.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:32.687+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:32 smithi055 ceph-mon[97033]: pgmap v690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2022-09-07T22:02:33.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:33.687+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:33 smithi055 ceph-mon[97033]: pgmap v691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:02:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:33 smithi059 ceph-mon[97450]: pgmap v691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:02:34.689 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:34.687+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:35.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:35.687+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:36 smithi055 ceph-mon[97033]: pgmap v692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2022-09-07T22:02:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:36 smithi059 ceph-mon[97450]: pgmap v692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2022-09-07T22:02:36.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:36.687+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:37.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:37.688+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:38 smithi059 ceph-mon[97450]: pgmap v693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2022-09-07T22:02:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:38.688+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:38 smithi055 ceph-mon[97033]: pgmap v693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2022-09-07T22:02:39.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:39.688+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:40 smithi059 ceph-mon[97450]: pgmap v694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:02:40.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:40.688+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:40 smithi055 ceph-mon[97033]: pgmap v694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:02:41.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:41.689+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:42 smithi059 ceph-mon[97450]: pgmap v695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:02:42.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:42.689+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:42 smithi055 ceph-mon[97033]: pgmap v695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:02:43.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:43.689+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:43 smithi055 ceph-mon[97033]: pgmap v696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:02:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:43 smithi059 ceph-mon[97450]: pgmap v696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:02:44.691 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:44.689+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:45.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:45.689+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:46 smithi059 ceph-mon[97450]: pgmap v697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T22:02:46.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:46.690+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:46 smithi055 ceph-mon[97033]: pgmap v697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T22:02:47.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:47.690+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:48 smithi059 ceph-mon[97450]: pgmap v698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T22:02:48.692 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:48.690+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:48 smithi055 ceph-mon[97033]: pgmap v698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T22:02:49.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:49.690+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:50 smithi059 ceph-mon[97450]: pgmap v699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:02:50.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:50.691+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:50 smithi055 ceph-mon[97033]: pgmap v699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:02:51.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:51.691+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:52.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:52 smithi059 ceph-mon[97450]: pgmap v700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:02:52.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:52.691+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:52 smithi055 ceph-mon[97033]: pgmap v700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:02:53.693 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:53.691+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:53.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:53 smithi055 ceph-mon[97033]: pgmap v701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:02:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:53 smithi059 ceph-mon[97450]: pgmap v701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:02:54.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:54.692+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:55.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:55.692+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:56 smithi059 ceph-mon[97450]: pgmap v702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T22:02:56.694 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:56.692+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:56 smithi055 ceph-mon[97033]: pgmap v702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T22:02:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:57.692+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:02:58 smithi059 ceph-mon[97450]: pgmap v703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T22:02:58.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:58.692+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:02:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:02:58 smithi055 ceph-mon[97033]: pgmap v703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T22:02:59.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:02:59.693+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:00 smithi059 ceph-mon[97450]: pgmap v704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:00.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:00.693+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:00 smithi055 ceph-mon[97033]: pgmap v704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:01.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:01.693+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:02 smithi059 ceph-mon[97450]: pgmap v705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:03:02.695 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:02.693+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:02 smithi055 ceph-mon[97033]: pgmap v705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:03:03.696 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:03.693+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:03 smithi055 ceph-mon[97033]: pgmap v706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T22:03:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:03 smithi059 ceph-mon[97450]: pgmap v706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T22:03:04.696 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:04.692+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:03:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:03:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:03:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:03:05.696 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:05.693+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:05 smithi055 ceph-mon[97033]: pgmap v707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 23 KiB/s wr, 9 op/s 2022-09-07T22:03:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:05 smithi059 ceph-mon[97450]: pgmap v707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 23 KiB/s wr, 9 op/s 2022-09-07T22:03:06.696 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:06.693+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:07.700 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:07.694+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:08 smithi059 ceph-mon[97450]: pgmap v708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T22:03:08.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:08.693+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:08 smithi055 ceph-mon[97033]: pgmap v708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T22:03:09.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:09.693+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:09 smithi055 ceph-mon[97033]: osdmap e127: 12 total, 12 up, 12 in 2022-09-07T22:03:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:09 smithi055 ceph-mon[97033]: pgmap v710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:03:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:09 smithi059 ceph-mon[97450]: osdmap e127: 12 total, 12 up, 12 in 2022-09-07T22:03:10.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:09 smithi059 ceph-mon[97450]: pgmap v710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:03:10.698 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:10.694+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:11.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:11.695+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:12 smithi059 ceph-mon[97450]: pgmap v711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:03:12.697 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:12.695+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:12 smithi055 ceph-mon[97033]: pgmap v711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:03:13.698 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:13.695+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:13 smithi055 ceph-mon[97033]: pgmap v712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T22:03:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:13 smithi059 ceph-mon[97450]: pgmap v712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T22:03:14.698 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:14.695+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:15.698 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:15.696+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:16 smithi059 ceph-mon[97450]: pgmap v713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T22:03:16.699 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:16.696+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:16 smithi055 ceph-mon[97033]: pgmap v713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T22:03:17.699 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:17.696+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:18 smithi059 ceph-mon[97450]: pgmap v714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:03:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:18 smithi059 ceph-mon[97450]: osdmap e128: 12 total, 12 up, 12 in 2022-09-07T22:03:18.700 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:18.696+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:18 smithi055 ceph-mon[97033]: pgmap v714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:03:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:18 smithi055 ceph-mon[97033]: osdmap e128: 12 total, 12 up, 12 in 2022-09-07T22:03:19.699 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:19.696+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:20 smithi059 ceph-mon[97450]: pgmap v716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:03:20.699 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:20.697+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:20 smithi055 ceph-mon[97033]: pgmap v716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:03:21.699 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:21.697+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:22 smithi059 ceph-mon[97450]: pgmap v717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:03:22.700 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:22.697+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:22 smithi055 ceph-mon[97033]: pgmap v717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:03:23.700 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:23.697+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:23 smithi055 ceph-mon[97033]: pgmap v718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:03:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:23 smithi059 ceph-mon[97450]: pgmap v718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:03:24.700 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:24.698+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:25.700 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:25.698+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:26 smithi059 ceph-mon[97450]: pgmap v719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:03:26.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:03:26.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:03:26.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:03:26.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:03:26.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:03:26.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:26.698+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:26.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:26 smithi055 ceph-mon[97033]: pgmap v719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:03:26.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:03:26.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:03:26.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:03:26.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:03:26.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:03:27.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:27.698+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:03:27.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:27 smithi055 ceph-mon[97033]: pgmap v720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:03:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:03:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:27 smithi059 ceph-mon[97450]: pgmap v720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:03:28.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:28.698+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:29.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:29.699+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:30 smithi059 ceph-mon[97450]: pgmap v721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:03:30.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:30.699+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:30 smithi055 ceph-mon[97033]: pgmap v721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:03:31.701 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:31.699+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:32 smithi059 ceph-mon[97450]: pgmap v722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:03:32.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:32.699+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:32 smithi055 ceph-mon[97033]: pgmap v722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:03:33.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:33.699+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:33.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:33 smithi055 ceph-mon[97033]: pgmap v723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:33 smithi059 ceph-mon[97450]: pgmap v723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:34.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:34.700+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:35.702 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:35.700+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:36 smithi059 ceph-mon[97450]: pgmap v724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:03:36.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:36.700+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:36 smithi055 ceph-mon[97033]: pgmap v724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:03:37.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:37.700+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:38 smithi059 ceph-mon[97450]: pgmap v725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:38.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:38.701+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:38 smithi055 ceph-mon[97033]: pgmap v725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:39.703 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:39.701+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:40 smithi059 ceph-mon[97450]: pgmap v726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:40.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:40.701+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:40 smithi055 ceph-mon[97033]: pgmap v726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:41.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:41.701+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:41.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:41 smithi055 ceph-mon[97033]: pgmap v727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:03:41.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:41 smithi059 ceph-mon[97450]: pgmap v727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:03:42.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:42.702+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:43.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:43.702+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:43 smithi055 ceph-mon[97033]: pgmap v728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:43 smithi059 ceph-mon[97450]: pgmap v728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:44.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:44.702+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:45.702+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:46 smithi059 ceph-mon[97450]: pgmap v729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:03:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:46.702+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:46 smithi055 ceph-mon[97033]: pgmap v729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:03:47.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:47.703+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:48 smithi059 ceph-mon[97450]: pgmap v730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:48.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:48.702+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:48 smithi055 ceph-mon[97033]: pgmap v730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:49.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:49.703+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:50 smithi059 ceph-mon[97450]: pgmap v731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:50.705 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:50.702+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:50 smithi055 ceph-mon[97033]: pgmap v731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:51 smithi059 ceph-mon[97450]: pgmap v732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:03:51.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:51.702+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:51 smithi055 ceph-mon[97033]: pgmap v732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:03:52.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:52.702+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:53.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:53.703+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:53 smithi055 ceph-mon[97033]: pgmap v733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:53 smithi059 ceph-mon[97450]: pgmap v733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:54.704+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:55.706 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:55.703+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:56 smithi059 ceph-mon[97450]: pgmap v734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:03:56.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:56.704+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:56 smithi055 ceph-mon[97033]: pgmap v734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:03:57.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:57.704+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:58 smithi059 ceph-mon[97450]: pgmap v735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:58.704+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:58 smithi055 ceph-mon[97033]: pgmap v735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:59.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:03:59.705+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:03:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:03:59 smithi055 ceph-mon[97033]: pgmap v736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:03:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:03:59 smithi059 ceph-mon[97450]: pgmap v736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:00.707 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:00.705+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:01.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:01.705+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:02 smithi059 ceph-mon[97450]: pgmap v737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2022-09-07T22:04:02.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:02.705+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:02 smithi055 ceph-mon[97033]: pgmap v737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2022-09-07T22:04:03.708 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:03.705+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:03 smithi055 ceph-mon[97033]: pgmap v738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 33 KiB/s wr, 18 op/s 2022-09-07T22:04:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:03 smithi059 ceph-mon[97450]: pgmap v738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 33 KiB/s wr, 18 op/s 2022-09-07T22:04:04.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:04.706+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:04:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:04:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:04:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:04:05.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:05.706+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:05 smithi055 ceph-mon[97033]: pgmap v739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 34 KiB/s wr, 19 op/s 2022-09-07T22:04:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:05 smithi059 ceph-mon[97450]: pgmap v739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 34 KiB/s wr, 19 op/s 2022-09-07T22:04:06.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:06.706+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:07.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:07.707+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:08 smithi059 ceph-mon[97450]: pgmap v740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:04:08.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:08.707+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:08 smithi055 ceph-mon[97033]: pgmap v740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:04:09.709 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:09.707+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:09 smithi055 ceph-mon[97033]: osdmap e129: 12 total, 12 up, 12 in 2022-09-07T22:04:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:09 smithi055 ceph-mon[97033]: pgmap v742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:04:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:09 smithi059 ceph-mon[97450]: osdmap e129: 12 total, 12 up, 12 in 2022-09-07T22:04:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:09 smithi059 ceph-mon[97450]: pgmap v742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:04:10.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:10.707+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:11.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:11.707+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:12 smithi059 ceph-mon[97450]: pgmap v743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 40 KiB/s wr, 10 op/s 2022-09-07T22:04:12.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:12.708+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:12 smithi055 ceph-mon[97033]: pgmap v743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 40 KiB/s wr, 10 op/s 2022-09-07T22:04:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:13.708+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:13 smithi055 ceph-mon[97033]: pgmap v744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 13 KiB/s wr, 8 op/s 2022-09-07T22:04:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:13 smithi059 ceph-mon[97450]: pgmap v744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 13 KiB/s wr, 8 op/s 2022-09-07T22:04:14.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:14.708+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:15.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:15.708+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:16 smithi059 ceph-mon[97450]: pgmap v745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 13 KiB/s wr, 7 op/s 2022-09-07T22:04:16.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:16.708+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:16 smithi055 ceph-mon[97033]: pgmap v745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 13 KiB/s wr, 7 op/s 2022-09-07T22:04:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:17 smithi059 ceph-mon[97450]: pgmap v746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:04:17.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:17.708+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:17 smithi055 ceph-mon[97033]: pgmap v746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:04:18.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:18.709+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:18 smithi055 ceph-mon[97033]: osdmap e130: 12 total, 12 up, 12 in 2022-09-07T22:04:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:18 smithi059 ceph-mon[97450]: osdmap e130: 12 total, 12 up, 12 in 2022-09-07T22:04:19.711 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:19.709+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:19 smithi055 ceph-mon[97033]: pgmap v748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:04:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:19 smithi059 ceph-mon[97450]: pgmap v748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:04:20.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:20.709+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:21.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:21.709+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:21 smithi055 ceph-mon[97033]: pgmap v749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:04:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:21 smithi059 ceph-mon[97450]: pgmap v749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:04:22.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:22.709+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:23.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:23.710+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:23 smithi055 ceph-mon[97033]: pgmap v750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:04:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:23 smithi059 ceph-mon[97450]: pgmap v750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:04:24.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:24.710+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:25.712 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:25.710+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:25 smithi055 ceph-mon[97033]: pgmap v751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:04:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:25 smithi059 ceph-mon[97450]: pgmap v751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:04:26.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:26.710+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:04:26.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:04:26.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:04:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:04:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:04:26.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:04:27.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:27.710+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:04:28.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:27 smithi055 ceph-mon[97033]: pgmap v752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:04:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:04:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:27 smithi059 ceph-mon[97450]: pgmap v752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:04:28.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:28.710+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:29.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:29.711+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:29 smithi055 ceph-mon[97033]: pgmap v753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:04:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:29 smithi059 ceph-mon[97450]: pgmap v753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:04:30.713 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:30.711+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:31.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:31.711+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:31 smithi055 ceph-mon[97033]: pgmap v754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:04:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:31 smithi059 ceph-mon[97450]: pgmap v754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:04:32.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:32.711+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:33.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:33.712+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:33 smithi055 ceph-mon[97033]: pgmap v755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:33 smithi059 ceph-mon[97450]: pgmap v755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:34.714 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:34.712+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:35.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:35.712+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:35 smithi055 ceph-mon[97033]: pgmap v756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:04:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:35 smithi059 ceph-mon[97450]: pgmap v756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:04:36.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:36.711+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:37.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:37.711+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:37 smithi055 ceph-mon[97033]: pgmap v757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:37 smithi059 ceph-mon[97450]: pgmap v757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:38.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:38.712+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:39.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:39.712+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:39 smithi055 ceph-mon[97033]: pgmap v758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:39 smithi059 ceph-mon[97450]: pgmap v758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:40.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:40.712+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:41.715 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:41.713+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:41 smithi055 ceph-mon[97033]: pgmap v759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:04:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:41 smithi059 ceph-mon[97450]: pgmap v759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:04:42.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:42.713+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:43.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:43.713+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:43 smithi055 ceph-mon[97033]: pgmap v760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:43 smithi059 ceph-mon[97450]: pgmap v760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:44.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:44.713+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:45.713+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:45 smithi055 ceph-mon[97033]: pgmap v761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:04:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:45 smithi059 ceph-mon[97450]: pgmap v761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:04:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:46.714+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:47.717 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:47.714+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:47 smithi055 ceph-mon[97033]: pgmap v762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:47 smithi059 ceph-mon[97450]: pgmap v762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:48.717 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:48.714+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:49.717 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:49.714+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:49 smithi055 ceph-mon[97033]: pgmap v763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:49 smithi059 ceph-mon[97450]: pgmap v763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:50.717 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:50.715+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:51.717 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:51.715+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:51 smithi055 ceph-mon[97033]: pgmap v764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:04:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:51 smithi059 ceph-mon[97450]: pgmap v764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:04:52.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:52.715+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:53.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:53.715+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:53 smithi055 ceph-mon[97033]: pgmap v765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:53 smithi059 ceph-mon[97450]: pgmap v765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:54.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:54.716+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:55.718 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:55.716+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:55 smithi055 ceph-mon[97033]: pgmap v766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:04:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:55 smithi059 ceph-mon[97450]: pgmap v766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:04:56.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:56.716+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:57.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:57.716+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:57 smithi055 ceph-mon[97033]: pgmap v767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:57 smithi059 ceph-mon[97450]: pgmap v767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:58.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:58.716+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:59.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:04:59.717+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:04:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:04:59 smithi055 ceph-mon[97033]: pgmap v768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:04:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:04:59 smithi059 ceph-mon[97450]: pgmap v768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:00.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:00.717+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:01.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:01.717+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:01 smithi055 ceph-mon[97033]: pgmap v769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2022-09-07T22:05:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:01 smithi059 ceph-mon[97450]: pgmap v769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2022-09-07T22:05:02.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:02.717+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:03.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:03.718+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:03 smithi055 ceph-mon[97033]: pgmap v770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 11 KiB/s wr, 7 op/s 2022-09-07T22:05:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:03 smithi059 ceph-mon[97450]: pgmap v770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 11 KiB/s wr, 7 op/s 2022-09-07T22:05:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:04.718+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:05:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:05:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:05:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:05:05.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:05.718+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:05 smithi055 ceph-mon[97033]: pgmap v771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 11 KiB/s wr, 9 op/s 2022-09-07T22:05:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:05 smithi059 ceph-mon[97450]: pgmap v771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 11 KiB/s wr, 9 op/s 2022-09-07T22:05:06.720 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:06.718+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:07.721 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:07.719+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:07 smithi055 ceph-mon[97033]: pgmap v772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-07T22:05:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:07 smithi059 ceph-mon[97450]: pgmap v772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-07T22:05:08.721 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:08.718+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:09.721 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:09.719+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:09 smithi055 ceph-mon[97033]: pgmap v773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-07T22:05:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:09 smithi059 ceph-mon[97450]: pgmap v773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-07T22:05:10.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:10.719+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:11.722 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:11.719+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:11 smithi055 ceph-mon[97033]: pgmap v774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2022-09-07T22:05:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:11 smithi059 ceph-mon[97450]: pgmap v774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2022-09-07T22:05:12.722 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:12.719+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:13.722 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:13.719+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:13 smithi055 ceph-mon[97033]: pgmap v775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-07T22:05:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:13 smithi059 ceph-mon[97450]: pgmap v775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-07T22:05:14.722 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:14.720+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:15.722 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:15.720+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:15 smithi055 ceph-mon[97033]: pgmap v776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 16 op/s 2022-09-07T22:05:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:15 smithi059 ceph-mon[97450]: pgmap v776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 16 op/s 2022-09-07T22:05:16.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:16.720+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:17.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:17.720+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:17 smithi055 ceph-mon[97033]: pgmap v777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 11 KiB/s wr, 15 op/s 2022-09-07T22:05:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:17 smithi059 ceph-mon[97450]: pgmap v777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 11 KiB/s wr, 15 op/s 2022-09-07T22:05:18.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:18.721+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:19.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:19.721+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:19 smithi055 ceph-mon[97033]: pgmap v778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:05:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:19 smithi059 ceph-mon[97450]: pgmap v778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:05:20.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:20.721+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:21.723 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:21.721+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:21 smithi055 ceph-mon[97033]: pgmap v779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:05:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:21 smithi059 ceph-mon[97450]: pgmap v779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:05:22.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:22.722+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:23.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:23.721+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:23 smithi055 ceph-mon[97033]: pgmap v780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:23 smithi059 ceph-mon[97450]: pgmap v780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:24.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:24.721+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:25.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:25.721+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:25 smithi055 ceph-mon[97033]: pgmap v781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:05:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:25 smithi059 ceph-mon[97450]: pgmap v781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:05:26.724 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:26.722+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:05:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:05:27.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:05:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:05:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:05:27.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:05:27.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:27.722+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:28 smithi055 ceph-mon[97033]: pgmap v782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:28.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:05:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:28 smithi059 ceph-mon[97450]: pgmap v782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:05:28.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:28.721+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:29.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:29.722+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:29 smithi055 ceph-mon[97033]: pgmap v783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:29 smithi059 ceph-mon[97450]: pgmap v783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:30.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:30.722+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:31.725 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:31.723+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:31 smithi055 ceph-mon[97033]: pgmap v784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:05:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:31 smithi059 ceph-mon[97450]: pgmap v784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:05:32.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:32.723+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:33.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:33.722+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:33 smithi055 ceph-mon[97033]: pgmap v785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:33 smithi059 ceph-mon[97450]: pgmap v785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:34.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:34.723+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:35.726 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:35.724+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:35 smithi055 ceph-mon[97033]: pgmap v786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:05:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:35 smithi059 ceph-mon[97450]: pgmap v786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:05:36.727 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:36.724+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:37.727 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:37.724+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:37 smithi055 ceph-mon[97033]: pgmap v787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:37 smithi059 ceph-mon[97450]: pgmap v787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:38.727 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:38.723+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:39.727 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:39.725+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:39 smithi055 ceph-mon[97033]: pgmap v788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:39 smithi059 ceph-mon[97450]: pgmap v788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:40.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:40.725+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:41.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:41.725+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:41 smithi055 ceph-mon[97033]: pgmap v789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:05:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:41 smithi059 ceph-mon[97450]: pgmap v789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:05:42.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:42.725+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:43.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:43.726+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:43 smithi055 ceph-mon[97033]: pgmap v790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:43 smithi059 ceph-mon[97450]: pgmap v790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:44.726+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:45.728 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:45.726+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:45 smithi055 ceph-mon[97033]: pgmap v791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:05:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:45 smithi059 ceph-mon[97450]: pgmap v791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:05:46.729 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:46.726+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:47.729 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:47.726+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:47 smithi055 ceph-mon[97033]: pgmap v792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:47 smithi059 ceph-mon[97450]: pgmap v792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:48.729 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:48.727+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:49.729 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:49.727+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:49.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:49 smithi055 ceph-mon[97033]: pgmap v793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:49 smithi059 ceph-mon[97450]: pgmap v793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:50.730 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:50.727+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:51 smithi055 ceph-mon[97033]: pgmap v794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:05:51.731 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:51.727+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:51 smithi059 ceph-mon[97450]: pgmap v794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:05:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:52.728+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:53.730 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:53.728+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:53 smithi055 ceph-mon[97033]: pgmap v795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:53 smithi059 ceph-mon[97450]: pgmap v795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:54.730 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:54.728+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:55 smithi055 ceph-mon[97033]: pgmap v796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:05:55.731 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:55.728+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:55 smithi059 ceph-mon[97450]: pgmap v796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:05:56.731 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:56.728+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:57 smithi055 ceph-mon[97033]: pgmap v797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:57.731 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:57.729+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:57 smithi059 ceph-mon[97450]: pgmap v797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:58.731 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:58.729+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:05:59 smithi055 ceph-mon[97033]: pgmap v798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:05:59.731 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:05:59.729+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:05:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:05:59 smithi059 ceph-mon[97450]: pgmap v798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:00.732 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:00.729+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:01 smithi055 ceph-mon[97033]: pgmap v799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2022-09-07T22:06:01.731 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:01.730+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:01 smithi059 ceph-mon[97450]: pgmap v799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2022-09-07T22:06:02.732 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:02.730+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:03.732 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:03.730+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:03 smithi055 ceph-mon[97033]: pgmap v800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T22:06:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:03 smithi059 ceph-mon[97450]: pgmap v800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T22:06:04.732 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:04.730+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:06:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:06:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:06:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:06:05.732 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:05.730+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:05 smithi055 ceph-mon[97033]: pgmap v801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 23 KiB/s wr, 9 op/s 2022-09-07T22:06:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:05 smithi059 ceph-mon[97450]: pgmap v801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 23 KiB/s wr, 9 op/s 2022-09-07T22:06:06.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:06.731+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:07 smithi055 ceph-mon[97033]: pgmap v802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:06:07.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:07.731+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:07 smithi059 ceph-mon[97450]: pgmap v802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:06:08.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:08.731+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:09.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:09.731+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:09 smithi055 ceph-mon[97033]: osdmap e131: 12 total, 12 up, 12 in 2022-09-07T22:06:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:09 smithi055 ceph-mon[97033]: pgmap v804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:06:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:09 smithi059 ceph-mon[97450]: osdmap e131: 12 total, 12 up, 12 in 2022-09-07T22:06:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:09 smithi059 ceph-mon[97450]: pgmap v804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:06:10.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:10.730+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:11 smithi055 ceph-mon[97033]: pgmap v805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:06:11.733 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:11.730+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:11 smithi059 ceph-mon[97450]: pgmap v805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:06:12.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:12.730+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:13.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:13.730+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:13 smithi055 ceph-mon[97033]: pgmap v806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:06:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:13 smithi059 ceph-mon[97450]: pgmap v806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:06:14.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:14.731+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:15 smithi055 ceph-mon[97033]: pgmap v807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:06:15.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:15.731+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:15 smithi059 ceph-mon[97450]: pgmap v807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:06:16.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:16.732+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:17 smithi055 ceph-mon[97033]: pgmap v808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:06:17.734 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:17.731+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:17 smithi059 ceph-mon[97450]: pgmap v808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:06:18.735 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:18.731+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:18 smithi055 ceph-mon[97033]: osdmap e132: 12 total, 12 up, 12 in 2022-09-07T22:06:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:18 smithi059 ceph-mon[97450]: osdmap e132: 12 total, 12 up, 12 in 2022-09-07T22:06:19.735 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:19.732+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:19 smithi055 ceph-mon[97033]: pgmap v810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:06:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:19 smithi059 ceph-mon[97450]: pgmap v810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:06:20.736 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:20.733+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:21 smithi055 ceph-mon[97033]: pgmap v811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:06:21.736 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:21.733+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:21 smithi059 ceph-mon[97450]: pgmap v811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:06:22.736 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:22.733+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:23.736 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:23.732+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:23 smithi055 ceph-mon[97033]: pgmap v812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:06:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:23 smithi059 ceph-mon[97450]: pgmap v812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:06:24.737 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:24.734+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:25 smithi055 ceph-mon[97033]: pgmap v813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:06:25.736 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:25.734+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:25 smithi059 ceph-mon[97450]: pgmap v813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:06:26.736 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:26.734+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:27.688 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:27 smithi059 ceph-mon[97450]: pgmap v814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:06:27.689 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:27 smithi055 ceph-mon[97033]: pgmap v814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:06:27.737 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:27.734+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:06:28.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:06:28.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:06:28.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:06:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:06:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:06:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:06:28.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:06:28.737 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:28.734+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:29 smithi055 ceph-mon[97033]: pgmap v815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:06:29.737 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:29.735+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:29 smithi059 ceph-mon[97450]: pgmap v815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:06:30.737 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:30.735+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:31 smithi055 ceph-mon[97033]: pgmap v816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:06:31.737 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:31.735+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:31 smithi059 ceph-mon[97450]: pgmap v816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:06:32.738 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:32.735+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:33.738 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:33.735+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:33 smithi055 ceph-mon[97033]: pgmap v817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:33 smithi059 ceph-mon[97450]: pgmap v817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:34.738 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:34.736+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:35 smithi055 ceph-mon[97033]: pgmap v818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:06:35.738 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:35.736+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:35 smithi059 ceph-mon[97450]: pgmap v818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:06:36.738 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:36.736+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:37 smithi055 ceph-mon[97033]: pgmap v819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:37.739 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:37.736+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:37 smithi059 ceph-mon[97450]: pgmap v819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:38.739 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:38.736+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:39 smithi055 ceph-mon[97033]: pgmap v820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:39.739 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:39.737+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:39 smithi059 ceph-mon[97450]: pgmap v820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:40.740 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:40.737+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:41 smithi055 ceph-mon[97033]: pgmap v821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:06:41.740 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:41.737+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:41 smithi059 ceph-mon[97450]: pgmap v821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:06:42.739 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:42.737+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:43.740 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:43.737+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:43 smithi055 ceph-mon[97033]: pgmap v822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:43 smithi059 ceph-mon[97450]: pgmap v822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:44.740 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:44.738+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:45 smithi055 ceph-mon[97033]: pgmap v823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:06:45.740 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:45.738+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:45 smithi059 ceph-mon[97450]: pgmap v823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:06:46.740 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:46.738+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:47 smithi055 ceph-mon[97033]: pgmap v824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:47.741 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:47.739+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:47 smithi059 ceph-mon[97450]: pgmap v824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:48.741 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:48.739+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:49 smithi055 ceph-mon[97033]: pgmap v825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:49.741 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:49.739+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:49 smithi059 ceph-mon[97450]: pgmap v825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:50.741 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:50.739+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:51 smithi055 ceph-mon[97033]: pgmap v826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:06:51.741 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:51.740+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:51 smithi059 ceph-mon[97450]: pgmap v826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:06:52.742 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:52.739+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:53.742 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:53.740+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:53 smithi055 ceph-mon[97033]: pgmap v827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:53 smithi059 ceph-mon[97450]: pgmap v827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:54.742 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:54.740+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:55 smithi055 ceph-mon[97033]: pgmap v828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:06:55.742 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:55.740+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:55 smithi059 ceph-mon[97450]: pgmap v828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:06:56.743 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:56.740+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:57 smithi055 ceph-mon[97033]: pgmap v829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:57.743 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:57.740+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:57 smithi059 ceph-mon[97450]: pgmap v829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:58.743 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:58.740+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:06:59 smithi055 ceph-mon[97033]: pgmap v830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:06:59.743 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:06:59.740+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:06:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:06:59 smithi059 ceph-mon[97450]: pgmap v830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:00.743 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:00.740+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:01 smithi055 ceph-mon[97033]: pgmap v831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:07:01.744 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:01.741+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:01 smithi059 ceph-mon[97450]: pgmap v831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:07:02.744 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:02.741+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:03.750 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:03.741+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:03 smithi055 ceph-mon[97033]: pgmap v832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T22:07:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:03 smithi059 ceph-mon[97450]: pgmap v832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T22:07:04.744 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:04.741+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:07:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:07:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:07:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:07:05.744 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:05.741+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:05 smithi055 ceph-mon[97033]: pgmap v833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 23 KiB/s wr, 9 op/s 2022-09-07T22:07:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:05 smithi059 ceph-mon[97450]: pgmap v833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 23 KiB/s wr, 9 op/s 2022-09-07T22:07:06.744 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:06.742+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:07.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:07 smithi055 ceph-mon[97033]: pgmap v834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:07:07.745 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:07.742+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:07 smithi059 ceph-mon[97450]: pgmap v834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:07:08.746 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:08.742+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:09.745 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:09.742+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:09 smithi055 ceph-mon[97033]: osdmap e133: 12 total, 12 up, 12 in 2022-09-07T22:07:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:09 smithi055 ceph-mon[97033]: pgmap v836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:07:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:09 smithi059 ceph-mon[97450]: osdmap e133: 12 total, 12 up, 12 in 2022-09-07T22:07:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:09 smithi059 ceph-mon[97450]: pgmap v836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:07:10.745 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:10.742+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:11 smithi055 ceph-mon[97033]: pgmap v837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:07:11.746 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:11.743+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:11 smithi059 ceph-mon[97450]: pgmap v837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:07:12.747 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:12.743+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:13.746 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:13.743+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:13 smithi055 ceph-mon[97033]: pgmap v838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T22:07:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:13 smithi059 ceph-mon[97450]: pgmap v838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T22:07:14.746 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:14.744+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:15 smithi055 ceph-mon[97033]: pgmap v839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T22:07:15.746 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:15.744+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:15 smithi059 ceph-mon[97450]: pgmap v839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-07T22:07:16.746 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:16.744+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:17 smithi055 ceph-mon[97033]: pgmap v840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:07:17.758 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:17.744+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:17 smithi059 ceph-mon[97450]: pgmap v840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:07:18.747 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:18.744+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:18 smithi055 ceph-mon[97033]: osdmap e134: 12 total, 12 up, 12 in 2022-09-07T22:07:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:18 smithi059 ceph-mon[97450]: osdmap e134: 12 total, 12 up, 12 in 2022-09-07T22:07:19.747 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:19.745+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:19 smithi055 ceph-mon[97033]: pgmap v842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:07:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:19 smithi059 ceph-mon[97450]: pgmap v842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:07:20.747 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:20.745+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:21 smithi055 ceph-mon[97033]: pgmap v843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:07:21.747 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:21.745+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:21 smithi059 ceph-mon[97450]: pgmap v843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:07:22.748 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:22.745+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:23.748 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:23.745+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:23 smithi055 ceph-mon[97033]: pgmap v844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:07:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:23 smithi059 ceph-mon[97450]: pgmap v844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:07:24.748 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:24.746+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:25 smithi055 ceph-mon[97033]: pgmap v845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:07:25.749 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:25.746+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:25 smithi059 ceph-mon[97450]: pgmap v845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:07:26.748 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:26.746+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:27 smithi055 ceph-mon[97033]: pgmap v846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:07:27.749 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:27.746+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:27 smithi059 ceph-mon[97450]: pgmap v846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:07:28.699 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:07:28.699 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:07:28.700 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:07:28.749 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:28.746+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:28.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:07:28.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:07:28.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:07:29.749 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:29.747+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:29.951 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:29 smithi055 ceph-mon[97033]: pgmap v847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:07:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:29 smithi059 ceph-mon[97450]: pgmap v847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:07:30.749 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:30.747+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:31.608 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:31 smithi055 ceph-mon[97033]: pgmap v848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:07:31.749 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:31.747+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:31 smithi059 ceph-mon[97450]: pgmap v848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:07:32.750 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:32.747+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:32 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-07T22:07:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:07:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:07:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:07:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:32 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-07T22:07:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:07:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:07:33.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:07:33.750 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:33.747+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:33 smithi055 ceph-mon[97033]: pgmap v849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:33 smithi059 ceph-mon[97450]: pgmap v849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:34.750 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:34.748+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:35 smithi055 ceph-mon[97033]: pgmap v850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:07:35.750 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:35.748+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:35 smithi059 ceph-mon[97450]: pgmap v850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:07:36.750 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:36.748+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:37 smithi055 ceph-mon[97033]: pgmap v851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:37.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:37.748+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:37 smithi059 ceph-mon[97450]: pgmap v851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:38.748+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:39 smithi055 ceph-mon[97033]: pgmap v852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:39.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:39.749+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:39 smithi059 ceph-mon[97450]: pgmap v852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:40.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:40.749+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:41 smithi055 ceph-mon[97033]: pgmap v853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:07:41.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:41.749+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:41 smithi059 ceph-mon[97450]: pgmap v853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:07:42.752 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:42.750+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:43.752 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:43.749+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:43 smithi055 ceph-mon[97033]: pgmap v854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:43 smithi059 ceph-mon[97450]: pgmap v854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:44.752 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:44.749+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:45 smithi055 ceph-mon[97033]: pgmap v855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:07:45.752 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:45.749+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:45 smithi059 ceph-mon[97450]: pgmap v855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:07:46.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:46.750+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:47 smithi055 ceph-mon[97033]: pgmap v856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:47.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:47.750+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:47 smithi059 ceph-mon[97450]: pgmap v856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:48.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:48.750+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:49 smithi055 ceph-mon[97033]: pgmap v857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:49.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:49.750+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:49.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:49 smithi059 ceph-mon[97450]: pgmap v857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:50.754 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:50.750+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:51 smithi055 ceph-mon[97033]: pgmap v858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:07:51.754 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:51.750+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:51 smithi059 ceph-mon[97450]: pgmap v858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:07:52.754 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:52.750+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:53.750+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:53 smithi055 ceph-mon[97033]: pgmap v859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:53 smithi059 ceph-mon[97450]: pgmap v859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:54.754 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:54.751+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:55 smithi055 ceph-mon[97033]: pgmap v860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:07:55.754 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:55.751+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:55 smithi059 ceph-mon[97450]: pgmap v860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:07:56.754 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:56.752+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:57 smithi055 ceph-mon[97033]: pgmap v861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:57.755 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:57.752+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:57 smithi059 ceph-mon[97450]: pgmap v861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:58.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:58.751+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:07:59 smithi055 ceph-mon[97033]: pgmap v862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:07:59.755 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:07:59.753+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:07:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:07:59 smithi059 ceph-mon[97450]: pgmap v862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:00.755 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:00.753+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:01 smithi055 ceph-mon[97033]: pgmap v863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2022-09-07T22:08:01.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:01.753+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:01 smithi059 ceph-mon[97450]: pgmap v863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2022-09-07T22:08:02.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:02.753+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:03.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:03.753+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:03 smithi055 ceph-mon[97033]: pgmap v864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:08:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:03 smithi059 ceph-mon[97450]: pgmap v864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:08:04.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:04.754+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:08:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:08:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:08:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:08:05.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:05.754+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:05 smithi055 ceph-mon[97033]: pgmap v865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T22:08:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:05 smithi059 ceph-mon[97450]: pgmap v865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T22:08:06.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:06.754+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:07 smithi055 ceph-mon[97033]: pgmap v866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:08:07.757 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:07.754+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:07 smithi059 ceph-mon[97450]: pgmap v866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:08:08.757 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:08.755+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:09.755+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:09 smithi055 ceph-mon[97033]: osdmap e135: 12 total, 12 up, 12 in 2022-09-07T22:08:09.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:09 smithi055 ceph-mon[97033]: pgmap v868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:08:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:09 smithi059 ceph-mon[97450]: osdmap e135: 12 total, 12 up, 12 in 2022-09-07T22:08:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:09 smithi059 ceph-mon[97450]: pgmap v868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:08:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:10.755+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:11 smithi055 ceph-mon[97033]: pgmap v869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2022-09-07T22:08:11.758 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:11.755+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:11 smithi059 ceph-mon[97450]: pgmap v869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2022-09-07T22:08:12.758 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:12.756+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:13.758 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:13.756+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:13 smithi055 ceph-mon[97033]: pgmap v870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:08:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:13 smithi059 ceph-mon[97450]: pgmap v870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:08:14.758 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:14.756+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:15 smithi055 ceph-mon[97033]: pgmap v871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:08:15.759 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:15.756+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:15 smithi059 ceph-mon[97450]: pgmap v871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:08:16.759 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:16.757+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:17 smithi055 ceph-mon[97033]: pgmap v872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:08:17.759 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:17.757+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:17 smithi059 ceph-mon[97450]: pgmap v872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:08:18.760 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:18.757+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:18 smithi055 ceph-mon[97033]: osdmap e136: 12 total, 12 up, 12 in 2022-09-07T22:08:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:18 smithi059 ceph-mon[97450]: osdmap e136: 12 total, 12 up, 12 in 2022-09-07T22:08:19.760 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:19.757+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:19 smithi055 ceph-mon[97033]: pgmap v874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:08:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:19 smithi059 ceph-mon[97450]: pgmap v874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:08:20.760 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:20.757+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:21 smithi055 ceph-mon[97033]: pgmap v875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:08:21.760 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:21.758+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:21 smithi059 ceph-mon[97450]: pgmap v875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:08:22.760 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:22.758+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:23.760 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:23.758+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:23 smithi055 ceph-mon[97033]: pgmap v876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:08:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:23 smithi059 ceph-mon[97450]: pgmap v876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:08:24.761 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:24.758+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:25 smithi055 ceph-mon[97033]: pgmap v877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:08:25.760 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:25.758+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:25 smithi059 ceph-mon[97450]: pgmap v877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:08:26.761 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:26.759+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:27 smithi055 ceph-mon[97033]: pgmap v878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:08:27.761 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:27.759+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:27 smithi059 ceph-mon[97450]: pgmap v878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:08:28.761 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:28.759+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:29 smithi055 ceph-mon[97033]: pgmap v879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:08:29.761 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:29.759+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:29 smithi059 ceph-mon[97450]: pgmap v879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:08:30.762 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:30.758+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:31 smithi055 ceph-mon[97033]: pgmap v880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:08:31.762 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:31.758+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:31 smithi059 ceph-mon[97450]: pgmap v880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:08:32.607 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:08:32.608 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:08:32.608 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:08:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:08:32.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:08:32.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:08:32.762 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:32.759+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:33.762 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:33.759+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:33.975 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:33 smithi059 ceph-mon[97450]: pgmap v881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:33 smithi055 ceph-mon[97033]: pgmap v881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:34.759+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:35.518 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:35 smithi059 ceph-mon[97450]: pgmap v882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:08:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:35 smithi055 ceph-mon[97033]: pgmap v882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:08:35.762 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:35.759+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:36.760+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:36 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-07T22:08:37.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:08:37.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:08:37.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:08:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:36 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-07T22:08:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:08:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:08:37.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:08:37.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:37.760+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:37 smithi055 ceph-mon[97033]: pgmap v883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:37 smithi059 ceph-mon[97450]: pgmap v883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:38.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:38.759+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:39 smithi055 ceph-mon[97033]: pgmap v884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:39.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:39.760+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:39 smithi059 ceph-mon[97450]: pgmap v884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:40.763 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:40.760+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:41 smithi055 ceph-mon[97033]: pgmap v885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:08:41.764 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:41.760+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:41 smithi059 ceph-mon[97450]: pgmap v885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:08:42.764 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:42.761+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:43.764 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:43.760+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:43 smithi055 ceph-mon[97033]: pgmap v886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:43 smithi059 ceph-mon[97450]: pgmap v886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:44.764 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:44.761+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:45 smithi055 ceph-mon[97033]: pgmap v887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:08:45.764 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:45.762+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:45 smithi059 ceph-mon[97450]: pgmap v887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:08:46.765 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:46.761+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:47 smithi055 ceph-mon[97033]: pgmap v888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:47.765 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:47.762+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:47 smithi059 ceph-mon[97450]: pgmap v888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:48.765 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:48.762+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:49 smithi055 ceph-mon[97033]: pgmap v889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:49.765 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:49.763+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:49 smithi059 ceph-mon[97450]: pgmap v889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:50.765 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:50.763+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:51 smithi055 ceph-mon[97033]: pgmap v890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:08:51.765 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:51.763+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:51 smithi059 ceph-mon[97450]: pgmap v890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:08:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:52.763+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:53.766 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:53.763+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:53 smithi059 ceph-mon[97450]: pgmap v891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:53 smithi055 ceph-mon[97033]: pgmap v891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:54.766 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:54.764+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:55 smithi055 ceph-mon[97033]: pgmap v892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:08:55.766 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:55.764+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:55 smithi059 ceph-mon[97450]: pgmap v892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:08:56.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:56.764+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:57 smithi055 ceph-mon[97033]: pgmap v893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:57.767 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:57.764+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:57 smithi059 ceph-mon[97450]: pgmap v893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:58.767 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:58.764+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:08:59 smithi055 ceph-mon[97033]: pgmap v894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:08:59.767 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:08:59.765+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:08:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:08:59 smithi059 ceph-mon[97450]: pgmap v894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:00.767 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:00.765+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:01 smithi055 ceph-mon[97033]: pgmap v895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 KiB/s wr, 10 op/s 2022-09-07T22:09:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:01.765+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:01 smithi059 ceph-mon[97450]: pgmap v895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 KiB/s wr, 10 op/s 2022-09-07T22:09:02.768 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:02.765+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:03.768 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:03.765+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:03 smithi059 ceph-mon[97450]: pgmap v896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 33 KiB/s wr, 11 op/s 2022-09-07T22:09:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:03 smithi055 ceph-mon[97033]: pgmap v896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 33 KiB/s wr, 11 op/s 2022-09-07T22:09:04.768 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:04.766+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:09:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:09:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:09:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:09:05.768 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:05.766+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:05 smithi059 ceph-mon[97450]: pgmap v897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 34 KiB/s wr, 12 op/s 2022-09-07T22:09:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:05 smithi055 ceph-mon[97033]: pgmap v897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 34 KiB/s wr, 12 op/s 2022-09-07T22:09:06.768 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:06.766+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:07 smithi055 ceph-mon[97033]: pgmap v898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:09:07.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:07.766+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:07 smithi059 ceph-mon[97450]: pgmap v898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:09:08.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:08.766+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:09.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:09.767+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:09 smithi059 ceph-mon[97450]: osdmap e137: 12 total, 12 up, 12 in 2022-09-07T22:09:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:09 smithi059 ceph-mon[97450]: pgmap v900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:09:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:09 smithi055 ceph-mon[97033]: osdmap e137: 12 total, 12 up, 12 in 2022-09-07T22:09:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:09 smithi055 ceph-mon[97033]: pgmap v900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:09:10.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:10.767+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:11 smithi055 ceph-mon[97033]: pgmap v901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 40 KiB/s wr, 19 op/s 2022-09-07T22:09:11.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:11.767+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:11 smithi059 ceph-mon[97450]: pgmap v901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 40 KiB/s wr, 19 op/s 2022-09-07T22:09:12.769 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:12.767+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:13.770 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:13.767+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:13 smithi059 ceph-mon[97450]: pgmap v902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 KiB/s wr, 16 op/s 2022-09-07T22:09:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:13 smithi055 ceph-mon[97033]: pgmap v902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 KiB/s wr, 16 op/s 2022-09-07T22:09:14.770 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:14.767+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:15 smithi055 ceph-mon[97033]: pgmap v903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 KiB/s wr, 16 op/s 2022-09-07T22:09:15.770 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:15.768+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:15 smithi059 ceph-mon[97450]: pgmap v903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 KiB/s wr, 16 op/s 2022-09-07T22:09:16.770 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:16.768+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:17 smithi055 ceph-mon[97033]: pgmap v904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:09:17.771 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:17.768+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:17 smithi059 ceph-mon[97450]: pgmap v904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:09:18.771 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:18.767+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:18 smithi055 ceph-mon[97033]: osdmap e138: 12 total, 12 up, 12 in 2022-09-07T22:09:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:18 smithi059 ceph-mon[97450]: osdmap e138: 12 total, 12 up, 12 in 2022-09-07T22:09:19.771 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:19.767+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:19 smithi059 ceph-mon[97450]: pgmap v906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:09:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:19 smithi055 ceph-mon[97033]: pgmap v906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:09:20.771 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:20.768+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:21.771 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:21.768+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:22 smithi055 ceph-mon[97033]: pgmap v907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:09:22.771 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:22.769+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:22 smithi059 ceph-mon[97450]: pgmap v907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:09:23.772 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:23.769+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:24.772 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:24.768+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:24 smithi059 ceph-mon[97450]: pgmap v908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:09:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:24 smithi055 ceph-mon[97033]: pgmap v908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:09:25.772 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:25.769+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:26.772 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:26.769+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:26 smithi059 ceph-mon[97450]: pgmap v909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:09:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:26 smithi055 ceph-mon[97033]: pgmap v909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:09:27.772 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:27.770+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:28.773 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:28.770+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:28 smithi059 ceph-mon[97450]: pgmap v910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:09:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:28 smithi055 ceph-mon[97033]: pgmap v910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:09:29.773 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:29.770+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:30.773 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:30.770+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:30 smithi059 ceph-mon[97450]: pgmap v911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:09:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:30 smithi055 ceph-mon[97033]: pgmap v911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:09:31.773 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:31.771+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:32.774 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:32.770+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:32 smithi059 ceph-mon[97450]: pgmap v912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:09:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:32 smithi055 ceph-mon[97033]: pgmap v912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:09:33.774 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:33.771+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:34.771+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:34 smithi059 ceph-mon[97450]: pgmap v913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:34 smithi055 ceph-mon[97033]: pgmap v913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:35.774 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:35.772+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:36.757 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:36 smithi059 ceph-mon[97450]: pgmap v914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:09:36.758 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:09:36.760 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:36 smithi055 ceph-mon[97033]: pgmap v914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:09:36.761 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:09:36.774 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:36.772+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:09:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:09:36.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:09:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:09:37.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:09:37.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:09:37.775 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:37.772+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:38.775 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:38.772+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:38 smithi059 ceph-mon[97450]: pgmap v915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:38 smithi055 ceph-mon[97033]: pgmap v915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:39.775 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:39.772+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:40.775 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:40.773+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:40 smithi059 ceph-mon[97450]: pgmap v916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:40 smithi055 ceph-mon[97033]: pgmap v916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:41.775 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:41.773+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:42.775 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:42.773+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:42 smithi059 ceph-mon[97450]: pgmap v917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:09:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:42 smithi055 ceph-mon[97033]: pgmap v917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:09:43.776 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:43.773+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:44.776 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:44.773+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:44 smithi059 ceph-mon[97450]: pgmap v918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:44 smithi055 ceph-mon[97033]: pgmap v918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:45.776 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:45.773+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:46.776 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:46.774+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:46 smithi059 ceph-mon[97450]: pgmap v919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:09:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:46 smithi055 ceph-mon[97033]: pgmap v919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:09:47.777 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:47.774+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:48.777 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:48.774+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:48 smithi059 ceph-mon[97450]: pgmap v920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:48 smithi055 ceph-mon[97033]: pgmap v920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:49.777 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:49.775+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:50.777 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:50.775+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:50 smithi059 ceph-mon[97450]: pgmap v921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:50 smithi055 ceph-mon[97033]: pgmap v921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:51.777 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:51.775+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:52.777 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:52.775+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:52 smithi055 ceph-mon[97033]: pgmap v922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:09:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:52 smithi059 ceph-mon[97450]: pgmap v922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:09:53.778 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:53.775+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:54.778 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:54.776+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:54 smithi055 ceph-mon[97033]: pgmap v923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:54 smithi059 ceph-mon[97450]: pgmap v923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:55.778 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:55.776+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:56.776+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:56 smithi055 ceph-mon[97033]: pgmap v924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:09:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:56 smithi059 ceph-mon[97450]: pgmap v924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:09:57.778 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:57.776+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:58.778 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:58.776+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:09:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:09:58 smithi055 ceph-mon[97033]: pgmap v925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:09:58 smithi059 ceph-mon[97450]: pgmap v925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:09:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:09:59.777+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:00.779 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:00.777+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:00 smithi055 ceph-mon[97033]: pgmap v926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-07T22:10:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T22:10:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T22:10:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:00 smithi059 ceph-mon[97450]: pgmap v926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-07T22:10:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T22:10:01.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T22:10:01.779 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:01.777+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:02.779 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:02.777+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:02 smithi055 ceph-mon[97033]: pgmap v927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2022-09-07T22:10:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:02 smithi059 ceph-mon[97450]: pgmap v927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2022-09-07T22:10:03.780 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:03.778+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:04.780 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:04.777+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:04 smithi055 ceph-mon[97033]: pgmap v928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T22:10:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:10:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:10:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:04 smithi059 ceph-mon[97450]: pgmap v928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T22:10:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:10:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:10:05.780 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:05.777+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:06.780 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:06.777+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:06 smithi055 ceph-mon[97033]: pgmap v929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 23 KiB/s wr, 9 op/s 2022-09-07T22:10:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:06 smithi059 ceph-mon[97450]: pgmap v929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 23 KiB/s wr, 9 op/s 2022-09-07T22:10:07.780 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:07.777+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:08.781 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:08.778+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:08 smithi055 ceph-mon[97033]: pgmap v930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:10:09.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:08 smithi059 ceph-mon[97450]: pgmap v930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:10:09.781 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:09.777+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:09 smithi055 ceph-mon[97033]: osdmap e139: 12 total, 12 up, 12 in 2022-09-07T22:10:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:09 smithi059 ceph-mon[97450]: osdmap e139: 12 total, 12 up, 12 in 2022-09-07T22:10:10.781 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:10.778+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:10 smithi055 ceph-mon[97033]: pgmap v932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:10:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:10 smithi059 ceph-mon[97450]: pgmap v932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:10:11.781 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:11.778+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:12.782 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:12.779+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:12 smithi055 ceph-mon[97033]: pgmap v933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:10:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:12 smithi059 ceph-mon[97450]: pgmap v933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:10:13.782 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:13.778+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:14.782 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:14.779+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:14 smithi055 ceph-mon[97033]: pgmap v934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:10:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:14 smithi059 ceph-mon[97450]: pgmap v934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:10:15.782 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:15.779+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:16.782 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:16.780+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:16 smithi055 ceph-mon[97033]: pgmap v935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:10:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:16 smithi059 ceph-mon[97450]: pgmap v935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:10:17.782 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:17.780+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:18.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:18.780+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:18 smithi059 ceph-mon[97450]: pgmap v936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:10:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:18 smithi059 ceph-mon[97450]: osdmap e140: 12 total, 12 up, 12 in 2022-09-07T22:10:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:18 smithi055 ceph-mon[97033]: pgmap v936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:10:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:18 smithi055 ceph-mon[97033]: osdmap e140: 12 total, 12 up, 12 in 2022-09-07T22:10:19.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:19.780+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:20.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:20.781+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:20 smithi059 ceph-mon[97450]: pgmap v938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:10:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:20 smithi055 ceph-mon[97033]: pgmap v938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:10:21.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:21.781+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:22.783 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:22.781+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:22 smithi059 ceph-mon[97450]: pgmap v939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:10:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:22 smithi055 ceph-mon[97033]: pgmap v939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:10:23.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:23.781+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:24.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:24.781+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:24 smithi059 ceph-mon[97450]: pgmap v940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:10:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:24 smithi055 ceph-mon[97033]: pgmap v940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:10:25.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:25.782+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:26.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:26.782+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:26 smithi059 ceph-mon[97450]: pgmap v941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:10:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:26 smithi055 ceph-mon[97033]: pgmap v941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:10:27.784 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:27.782+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:28.782+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:28 smithi059 ceph-mon[97450]: pgmap v942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:10:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:28 smithi055 ceph-mon[97033]: pgmap v942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:10:29.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:29.782+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:30.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:30.782+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:30 smithi059 ceph-mon[97450]: pgmap v943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:10:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:30 smithi055 ceph-mon[97033]: pgmap v943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:10:31.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:31.783+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:32.783+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:32 smithi059 ceph-mon[97450]: pgmap v944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:10:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:32 smithi055 ceph-mon[97033]: pgmap v944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:10:33.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:33.783+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:34.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:34.783+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:34 smithi059 ceph-mon[97450]: pgmap v945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:34 smithi055 ceph-mon[97033]: pgmap v945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:35.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:35.783+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:36.786 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:36.783+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:36 smithi055 ceph-mon[97033]: pgmap v946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:10:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:10:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:10:36.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:10:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:36 smithi059 ceph-mon[97450]: pgmap v946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:10:37.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:10:37.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:10:37.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:10:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:37.784+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:10:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:10:38.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:38.784+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:39 smithi059 ceph-mon[97450]: pgmap v947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:39 smithi055 ceph-mon[97033]: pgmap v947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:39.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:39.784+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:40.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:40.784+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:41 smithi059 ceph-mon[97450]: pgmap v948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:41 smithi055 ceph-mon[97033]: pgmap v948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:41.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:41.785+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:42.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:42.785+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:43 smithi059 ceph-mon[97450]: pgmap v949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:10:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:43 smithi055 ceph-mon[97033]: pgmap v949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:10:43.787 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:43.785+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:44.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:44.786+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:45 smithi059 ceph-mon[97450]: pgmap v950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:45 smithi055 ceph-mon[97033]: pgmap v950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:45.793 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:45.785+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:46.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:46.786+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:47 smithi059 ceph-mon[97450]: pgmap v951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:10:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:47 smithi055 ceph-mon[97033]: pgmap v951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:10:47.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:47.786+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:48.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:48.786+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:49 smithi059 ceph-mon[97450]: pgmap v952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:49 smithi055 ceph-mon[97033]: pgmap v952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:49.788 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:49.786+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:50.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:50.787+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:51 smithi059 ceph-mon[97450]: pgmap v953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:51 smithi055 ceph-mon[97033]: pgmap v953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:51.786+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:52.786+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:53 smithi059 ceph-mon[97450]: pgmap v954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:10:53.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:53 smithi055 ceph-mon[97033]: pgmap v954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:10:53.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:53.786+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:54.787+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:55 smithi059 ceph-mon[97450]: pgmap v955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:55 smithi055 ceph-mon[97033]: pgmap v955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:10:55.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:55.787+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:56.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:56.787+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:57.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:57 smithi059 ceph-mon[97450]: pgmap v956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T22:10:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:57 smithi055 ceph-mon[97033]: pgmap v956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T22:10:57.790 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:57.787+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:58.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:58.787+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:10:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:10:59 smithi059 ceph-mon[97450]: pgmap v957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2022-09-07T22:10:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:10:59 smithi055 ceph-mon[97033]: pgmap v957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2022-09-07T22:10:59.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:10:59.787+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:11:00.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:00.787+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:11:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:01 smithi055 ceph-mon[97033]: pgmap v958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2022-09-07T22:11:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:01 smithi059 ceph-mon[97450]: pgmap v958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2022-09-07T22:11:01.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:01.788+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:11:02.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:02.788+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:11:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:03 smithi055 ceph-mon[97033]: pgmap v959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2022-09-07T22:11:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:03 smithi059 ceph-mon[97450]: pgmap v959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2022-09-07T22:11:03.791 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:03.789+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:11:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:11:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:11:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:11:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:11:04.792 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:04.789+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:11:05.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:11:04.724+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:11:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:11:04.925+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:11:05.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:11:05.326+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:11:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:05 smithi055 ceph-mon[97033]: pgmap v960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T22:11:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:05 smithi059 ceph-mon[97450]: pgmap v960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 23 KiB/s wr, 7 op/s 2022-09-07T22:11:05.792 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:05.788+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:11:06.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:11:06.128+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:11:06.792 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:06.789+0000 7f44f97fa700 1 -- 172.21.15.47:0/1459530205 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f44ec003380 con 0x7f4504067fb0 2022-09-07T22:11:07.679 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:07.673+0000 7f450b7ff700 0 librados: client.admin authentication error (1) Operation not permitted 2022-09-07T22:11:07.680 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:07.673+0000 7f450b7ff700 1 -- 172.21.15.47:0/1459530205 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4504067fb0 msgr2=0x7f450419ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-07T22:11:07.681 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:07.673+0000 7f450b7ff700 1 --2- 172.21.15.47:0/1459530205 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4504067fb0 0x7f450419ca70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f44f4024040 tx=0x7f44f4004640 comp rx=0 tx=0).stop 2022-09-07T22:11:07.682 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:07.673+0000 7f450b7ff700 1 -- 172.21.15.47:0/1459530205 shutdown_connections 2022-09-07T22:11:07.682 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:07.673+0000 7f450b7ff700 1 --2- 172.21.15.47:0/1459530205 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4504100110 0x7f45041967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T22:11:07.683 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:07.673+0000 7f450b7ff700 1 --2- 172.21.15.47:0/1459530205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4504067610 0x7f450419c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T22:11:07.683 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:07.673+0000 7f450b7ff700 1 --2- 172.21.15.47:0/1459530205 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4504067fb0 0x7f450419ca70 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-07T22:11:07.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:07.673+0000 7f450b7ff700 1 -- 172.21.15.47:0/1459530205 >> 172.21.15.47:0/1459530205 conn(0x7f4504073ef0 msgr2=0x7f45040fab20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-07T22:11:07.684 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:07.673+0000 7f450b7ff700 1 -- 172.21.15.47:0/1459530205 shutdown_connections 2022-09-07T22:11:07.685 INFO:teuthology.orchestra.run.smithi047.stderr:2022-09-07T22:11:07.673+0000 7f450b7ff700 1 -- 172.21.15.47:0/1459530205 wait complete. 2022-09-07T22:11:07.685 INFO:teuthology.orchestra.run.smithi047.stderr:[errno 1] RADOS permission error (error connecting to the cluster) 2022-09-07T22:11:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-07T22:11:07.690 ERROR:teuthology.run_tasks:Manager failed: full_sequential_finally Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/task/full_sequential_finally.py", line 45, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/run_tasks.py", line 82, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/task/exec.py", line 72, in task c], File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/remote.py", line 510, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi047 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true'" 2022-09-07T22:11:07.706 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2022-09-07T22:11:07.725 DEBUG:tasks.check_counter:Getting stats from a 2022-09-07T22:11:07.726 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell --fsid 01b8396e-2ef4-11ed-8431-001a4aab830c -- ceph daemon mds.a perf dump 2022-09-07T22:11:07.751 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:07 smithi055 ceph-mon[97033]: pgmap v961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T22:11:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:07 smithi059 ceph-mon[97450]: pgmap v961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T22:11:08.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:11:07.730+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:11:08.676 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/01b8396e-2ef4-11ed-8431-001a4aab830c/mon.a/config 2022-09-07T22:11:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:09 smithi055 ceph-mon[97033]: pgmap v962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:11:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:09 smithi055 ceph-mon[97033]: osdmap e141: 12 total, 12 up, 12 in 2022-09-07T22:11:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:09 smithi059 ceph-mon[97450]: pgmap v962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T22:11:09.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:09 smithi059 ceph-mon[97450]: osdmap e141: 12 total, 12 up, 12 in 2022-09-07T22:11:11.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:11:10.934+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:11:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:11 smithi055 ceph-mon[97033]: pgmap v964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:11:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:11 smithi059 ceph-mon[97450]: pgmap v964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:11:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:13 smithi055 ceph-mon[97033]: pgmap v965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:11:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:13 smithi059 ceph-mon[97450]: pgmap v965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:11:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:15 smithi055 ceph-mon[97033]: pgmap v966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:11:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:15 smithi059 ceph-mon[97450]: pgmap v966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2022-09-07T22:11:17.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:11:17.339+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:11:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:17 smithi055 ceph-mon[97033]: pgmap v967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:11:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:17 smithi059 ceph-mon[97450]: pgmap v967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:11:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:19 smithi055 ceph-mon[97033]: pgmap v968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:11:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:19 smithi059 ceph-mon[97450]: pgmap v968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:11:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:20 smithi055 ceph-mon[97033]: pgmap v969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 193 B/s wr, 6 op/s 2022-09-07T22:11:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:20 smithi059 ceph-mon[97450]: pgmap v969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 193 B/s wr, 6 op/s 2022-09-07T22:11:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:22 smithi055 ceph-mon[97033]: pgmap v970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T22:11:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:22 smithi059 ceph-mon[97450]: pgmap v970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T22:11:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:24 smithi059 ceph-mon[97450]: pgmap v971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:11:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:24 smithi055 ceph-mon[97033]: pgmap v971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:11:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:26 smithi059 ceph-mon[97450]: pgmap v972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:11:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:26 smithi055 ceph-mon[97033]: pgmap v972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:11:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:28 smithi059 ceph-mon[97450]: pgmap v973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:28 smithi055 ceph-mon[97033]: pgmap v973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:30.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:11:30.152+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:11:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:30 smithi059 ceph-mon[97450]: pgmap v974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:30 smithi055 ceph-mon[97033]: pgmap v974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:32 smithi059 ceph-mon[97450]: pgmap v975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:11:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:32 smithi055 ceph-mon[97033]: pgmap v975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:11:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:34 smithi055 ceph-mon[97033]: pgmap v976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:34 smithi059 ceph-mon[97450]: pgmap v976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:36 smithi055 ceph-mon[97033]: pgmap v977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:11:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:36 smithi059 ceph-mon[97450]: pgmap v977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:11:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:11:38.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:11:38.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:11:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:11:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:11:38.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:11:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:39 smithi055 ceph-mon[97033]: pgmap v978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:39.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:11:39.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:11:39.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:11:39.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:11:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:39 smithi059 ceph-mon[97450]: pgmap v978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:39.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:11:39.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:11:39.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:11:39.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:11:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:41 smithi055 ceph-mon[97033]: pgmap v979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:41 smithi059 ceph-mon[97450]: pgmap v979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:43 smithi055 ceph-mon[97033]: pgmap v980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:11:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:43 smithi059 ceph-mon[97450]: pgmap v980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:11:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:44 smithi055 ceph-mon[97033]: pgmap v981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:44 smithi059 ceph-mon[97450]: pgmap v981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:45.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:11:45.159+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:11:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:46 smithi055 ceph-mon[97033]: pgmap v982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:11:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:46 smithi059 ceph-mon[97450]: pgmap v982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:11:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:48 smithi059 ceph-mon[97450]: pgmap v983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:48 smithi055 ceph-mon[97033]: pgmap v983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:50 smithi059 ceph-mon[97450]: pgmap v984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:50 smithi055 ceph-mon[97033]: pgmap v984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:52 smithi059 ceph-mon[97450]: pgmap v985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:11:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:52 smithi055 ceph-mon[97033]: pgmap v985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:11:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:54 smithi059 ceph-mon[97450]: pgmap v986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:54 smithi055 ceph-mon[97033]: pgmap v986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:56 smithi059 ceph-mon[97450]: pgmap v987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:11:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:56 smithi055 ceph-mon[97033]: pgmap v987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:11:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:11:58 smithi055 ceph-mon[97033]: pgmap v988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:11:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:11:58 smithi059 ceph-mon[97450]: pgmap v988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:00.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:12:00.175+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:12:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:00 smithi055 ceph-mon[97033]: pgmap v989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:00 smithi059 ceph-mon[97450]: pgmap v989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:02 smithi055 ceph-mon[97033]: pgmap v990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:12:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:02 smithi059 ceph-mon[97450]: pgmap v990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:12:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:04 smithi055 ceph-mon[97033]: pgmap v991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:12:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:12:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:04 smithi059 ceph-mon[97450]: pgmap v991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:12:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:12:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:06 smithi055 ceph-mon[97033]: pgmap v992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:12:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:06 smithi059 ceph-mon[97450]: pgmap v992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:12:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:08 smithi059 ceph-mon[97450]: pgmap v993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:12:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:08 smithi059 ceph-mon[97450]: pgmap v994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-07T22:12:08.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:12:08.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:08 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-07T22:12:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:08 smithi055 ceph-mon[97033]: pgmap v993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:12:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:08 smithi055 ceph-mon[97033]: pgmap v994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-07T22:12:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:12:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:08 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-07T22:12:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:09 smithi055 ceph-mon[97033]: osdmap e142: 12 total, 12 up, 12 in 2022-09-07T22:12:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:09 smithi059 ceph-mon[97450]: osdmap e142: 12 total, 12 up, 12 in 2022-09-07T22:12:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:10 smithi055 ceph-mon[97033]: pgmap v996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 65 KiB/s wr, 25 op/s 2022-09-07T22:12:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:10 smithi059 ceph-mon[97450]: pgmap v996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 65 KiB/s wr, 25 op/s 2022-09-07T22:12:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:12 smithi055 ceph-mon[97033]: pgmap v997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 66 KiB/s wr, 29 op/s 2022-09-07T22:12:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:12 smithi059 ceph-mon[97450]: pgmap v997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 66 KiB/s wr, 29 op/s 2022-09-07T22:12:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:14 smithi055 ceph-mon[97033]: pgmap v998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 125 B/s wr, 3 op/s 2022-09-07T22:12:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:14 smithi059 ceph-mon[97450]: pgmap v998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 125 B/s wr, 3 op/s 2022-09-07T22:12:15.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:12:15.187+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:12:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:16 smithi055 ceph-mon[97033]: pgmap v999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 251 B/s wr, 7 op/s 2022-09-07T22:12:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:16 smithi059 ceph-mon[97450]: pgmap v999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 251 B/s wr, 7 op/s 2022-09-07T22:12:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:18 smithi055 ceph-mon[97033]: pgmap v1000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:12:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:18 smithi059 ceph-mon[97450]: pgmap v1000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:12:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:20 smithi055 ceph-mon[97033]: pgmap v1001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 191 B/s wr, 5 op/s 2022-09-07T22:12:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:20 smithi059 ceph-mon[97450]: pgmap v1001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 191 B/s wr, 5 op/s 2022-09-07T22:12:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:22 smithi055 ceph-mon[97033]: pgmap v1002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T22:12:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:22 smithi059 ceph-mon[97450]: pgmap v1002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T22:12:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:24 smithi055 ceph-mon[97033]: pgmap v1003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:12:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:24 smithi059 ceph-mon[97450]: pgmap v1003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:12:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:26 smithi055 ceph-mon[97033]: pgmap v1004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:12:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:26 smithi059 ceph-mon[97450]: pgmap v1004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:12:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:28 smithi055 ceph-mon[97033]: pgmap v1005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:28 smithi059 ceph-mon[97450]: pgmap v1005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:30.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:12:30.201+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:12:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:30 smithi055 ceph-mon[97033]: pgmap v1006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:30 smithi059 ceph-mon[97450]: pgmap v1006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:33.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:32 smithi055 ceph-mon[97033]: pgmap v1007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:12:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:32 smithi059 ceph-mon[97450]: pgmap v1007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:12:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:34 smithi055 ceph-mon[97033]: pgmap v1008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:34 smithi059 ceph-mon[97450]: pgmap v1008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:36 smithi055 ceph-mon[97033]: pgmap v1009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:12:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:36 smithi059 ceph-mon[97450]: pgmap v1009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:12:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:38 smithi055 ceph-mon[97033]: pgmap v1010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:38 smithi059 ceph-mon[97450]: pgmap v1010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:40 smithi055 ceph-mon[97033]: pgmap v1011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:40 smithi059 ceph-mon[97450]: pgmap v1011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:42 smithi055 ceph-mon[97033]: pgmap v1012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:12:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:42 smithi059 ceph-mon[97450]: pgmap v1012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:12:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:44 smithi055 ceph-mon[97033]: pgmap v1013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:44 smithi059 ceph-mon[97450]: pgmap v1013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:45.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:12:45.217+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:12:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:46 smithi055 ceph-mon[97033]: pgmap v1014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:12:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:46 smithi059 ceph-mon[97450]: pgmap v1014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:12:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:48 smithi055 ceph-mon[97033]: pgmap v1015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:48 smithi059 ceph-mon[97450]: pgmap v1015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:50 smithi055 ceph-mon[97033]: pgmap v1016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:50 smithi059 ceph-mon[97450]: pgmap v1016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:52 smithi055 ceph-mon[97033]: pgmap v1017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:12:53.290 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:52 smithi059 ceph-mon[97450]: pgmap v1017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:12:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:54 smithi055 ceph-mon[97033]: pgmap v1018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:54 smithi059 ceph-mon[97450]: pgmap v1018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:56 smithi055 ceph-mon[97033]: pgmap v1019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:12:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:56 smithi059 ceph-mon[97450]: pgmap v1019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:12:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:12:58 smithi055 ceph-mon[97033]: pgmap v1020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:12:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:12:58 smithi059 ceph-mon[97450]: pgmap v1020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:00.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:13:00.232+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:13:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:00 smithi055 ceph-mon[97033]: pgmap v1021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:00 smithi059 ceph-mon[97450]: pgmap v1021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:02 smithi055 ceph-mon[97033]: pgmap v1022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:13:03.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:02 smithi059 ceph-mon[97450]: pgmap v1022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:13:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:04 smithi055 ceph-mon[97033]: pgmap v1023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:13:05.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:13:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:04 smithi059 ceph-mon[97450]: pgmap v1023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:13:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:13:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:06 smithi055 ceph-mon[97033]: pgmap v1024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:13:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:06 smithi059 ceph-mon[97450]: pgmap v1024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:13:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:13:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:13:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:13:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:13:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:13:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:13:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:09 smithi055 ceph-mon[97033]: pgmap v1025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:13:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:13:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:13:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:09 smithi059 ceph-mon[97450]: pgmap v1025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:13:09.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:13:09.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:13:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:10 smithi055 ceph-mon[97033]: osdmap e143: 12 total, 12 up, 12 in 2022-09-07T22:13:10.512 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:10 smithi059 ceph-mon[97450]: osdmap e143: 12 total, 12 up, 12 in 2022-09-07T22:13:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:11 smithi055 ceph-mon[97033]: pgmap v1027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:13:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:13:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:13:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:11.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:11.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:11.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:11.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:13:11.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:13:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:11 smithi059 ceph-mon[97450]: pgmap v1027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:13:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:13:11.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:13:11.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:11.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:11.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:11.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:11.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:13:11.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:13:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:12 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T22:13:12.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:12 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T22:13:12.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:12.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:12.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:12.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:12 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T22:13:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:12 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T22:13:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:12.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:12.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:12.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:13:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:13 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T22:13:13.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:13 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T22:13:13.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:13 smithi055 ceph-mon[97033]: pgmap v1028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:13:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:13 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T22:13:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:13 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T22:13:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:13 smithi059 ceph-mon[97450]: pgmap v1028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:13:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:15 smithi055 ceph-mon[97033]: pgmap v1029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:13:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:15 smithi059 ceph-mon[97450]: pgmap v1029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:13:15.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:13:15.239+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:13:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:17 smithi055 ceph-mon[97033]: pgmap v1030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:13:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:17 smithi059 ceph-mon[97450]: pgmap v1030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:13:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:19 smithi055 ceph-mon[97033]: pgmap v1031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:13:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:19 smithi059 ceph-mon[97450]: pgmap v1031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:13:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:21 smithi055 ceph-mon[97033]: pgmap v1032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 197 B/s wr, 6 op/s 2022-09-07T22:13:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:21 smithi059 ceph-mon[97450]: pgmap v1032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 197 B/s wr, 6 op/s 2022-09-07T22:13:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:23 smithi055 ceph-mon[97033]: pgmap v1033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T22:13:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:23 smithi059 ceph-mon[97450]: pgmap v1033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T22:13:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:25 smithi055 ceph-mon[97033]: pgmap v1034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:13:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:25 smithi059 ceph-mon[97450]: pgmap v1034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:13:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:27 smithi055 ceph-mon[97033]: pgmap v1035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:13:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:27 smithi059 ceph-mon[97450]: pgmap v1035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:13:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:29 smithi055 ceph-mon[97033]: pgmap v1036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:29.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:29 smithi059 ceph-mon[97450]: pgmap v1036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:30.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:13:30.256+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:13:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:31 smithi055 ceph-mon[97033]: pgmap v1037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:31 smithi059 ceph-mon[97450]: pgmap v1037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:33 smithi055 ceph-mon[97033]: pgmap v1038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:13:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:33 smithi059 ceph-mon[97450]: pgmap v1038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:13:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:35 smithi055 ceph-mon[97033]: pgmap v1039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:35 smithi059 ceph-mon[97450]: pgmap v1039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:37 smithi055 ceph-mon[97033]: pgmap v1040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:13:37.538 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:37 smithi059 ceph-mon[97450]: pgmap v1040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:13:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:38 smithi059 ceph-mon[97450]: pgmap v1041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:38 smithi059 ceph-mon[97450]: pgmap v1042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:13:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:38 smithi059 ceph-mon[97450]: pgmap v1043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:13:39.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:13:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:38 smithi055 ceph-mon[97033]: pgmap v1041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:38 smithi055 ceph-mon[97033]: pgmap v1042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:13:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:38 smithi055 ceph-mon[97033]: pgmap v1043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:13:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:13:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:40 smithi059 ceph-mon[97450]: pgmap v1044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:13:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:40 smithi055 ceph-mon[97033]: pgmap v1044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:13:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:42 smithi055 ceph-mon[97033]: pgmap v1045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:13:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:42 smithi059 ceph-mon[97450]: pgmap v1045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:13:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:44 smithi055 ceph-mon[97033]: pgmap v1046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:13:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:44 smithi059 ceph-mon[97450]: pgmap v1046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:13:45.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:13:45.267+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:13:47.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:46 smithi055 ceph-mon[97033]: pgmap v1047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:13:47.289 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:46 smithi059 ceph-mon[97450]: pgmap v1047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:13:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:48 smithi055 ceph-mon[97033]: pgmap v1048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:13:49.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:48 smithi059 ceph-mon[97450]: pgmap v1048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:13:51.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:50 smithi055 ceph-mon[97033]: pgmap v1049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:13:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:50 smithi059 ceph-mon[97450]: pgmap v1049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:13:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:52 smithi055 ceph-mon[97033]: pgmap v1050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:52 smithi059 ceph-mon[97450]: pgmap v1050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:54 smithi055 ceph-mon[97033]: pgmap v1051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:54 smithi059 ceph-mon[97450]: pgmap v1051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:56 smithi055 ceph-mon[97033]: pgmap v1052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:13:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:56 smithi059 ceph-mon[97450]: pgmap v1052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:13:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:13:58 smithi055 ceph-mon[97033]: pgmap v1053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:13:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:13:58 smithi059 ceph-mon[97450]: pgmap v1053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:00.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:14:00.281+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:14:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:00 smithi055 ceph-mon[97033]: pgmap v1054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:14:01.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:00 smithi059 ceph-mon[97450]: pgmap v1054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:14:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:02 smithi055 ceph-mon[97033]: pgmap v1055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:02 smithi059 ceph-mon[97450]: pgmap v1055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:04 smithi055 ceph-mon[97033]: pgmap v1056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:14:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:14:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:04 smithi059 ceph-mon[97450]: pgmap v1056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:14:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:14:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:06 smithi055 ceph-mon[97033]: pgmap v1057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:14:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:06 smithi059 ceph-mon[97450]: pgmap v1057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:14:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:08 smithi055 ceph-mon[97033]: pgmap v1058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:14:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:08 smithi059 ceph-mon[97450]: pgmap v1058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:14:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:09 smithi055 ceph-mon[97033]: osdmap e144: 12 total, 12 up, 12 in 2022-09-07T22:14:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:09 smithi059 ceph-mon[97450]: osdmap e144: 12 total, 12 up, 12 in 2022-09-07T22:14:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:10 smithi055 ceph-mon[97033]: pgmap v1060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:14:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:10 smithi059 ceph-mon[97450]: pgmap v1060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:14:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:12 smithi055 ceph-mon[97033]: pgmap v1061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:14:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:12 smithi059 ceph-mon[97450]: pgmap v1061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:14:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:14 smithi055 ceph-mon[97033]: pgmap v1062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:14:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:14 smithi059 ceph-mon[97450]: pgmap v1062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:14:15.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:14:15.283+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:14:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:16 smithi055 ceph-mon[97033]: pgmap v1063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:14:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:16 smithi059 ceph-mon[97450]: pgmap v1063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:14:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:18 smithi055 ceph-mon[97033]: pgmap v1064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:14:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:18 smithi059 ceph-mon[97450]: pgmap v1064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:14:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:20 smithi055 ceph-mon[97033]: pgmap v1065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-07T22:14:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:20 smithi059 ceph-mon[97450]: pgmap v1065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-07T22:14:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:22 smithi055 ceph-mon[97033]: pgmap v1066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:14:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:22 smithi059 ceph-mon[97450]: pgmap v1066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:14:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:24 smithi055 ceph-mon[97033]: pgmap v1067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:14:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:24 smithi059 ceph-mon[97450]: pgmap v1067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:14:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:26 smithi055 ceph-mon[97033]: pgmap v1068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:14:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:26 smithi059 ceph-mon[97450]: pgmap v1068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:14:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:28 smithi055 ceph-mon[97033]: pgmap v1069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:28 smithi059 ceph-mon[97450]: pgmap v1069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:30.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:14:30.297+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:14:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:30 smithi055 ceph-mon[97033]: pgmap v1070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:14:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:30 smithi059 ceph-mon[97450]: pgmap v1070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:14:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:32 smithi055 ceph-mon[97033]: pgmap v1071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:32 smithi059 ceph-mon[97450]: pgmap v1071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:34 smithi055 ceph-mon[97033]: pgmap v1072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:34 smithi059 ceph-mon[97450]: pgmap v1072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:36 smithi055 ceph-mon[97033]: pgmap v1073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:14:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:36 smithi059 ceph-mon[97450]: pgmap v1073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:14:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:38 smithi055 ceph-mon[97033]: pgmap v1074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:14:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:14:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:14:39.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:14:39.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:14:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:38 smithi059 ceph-mon[97450]: pgmap v1074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:14:39.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:14:39.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:14:39.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:14:39.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:14:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:40 smithi055 ceph-mon[97033]: pgmap v1075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:14:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:40 smithi059 ceph-mon[97450]: pgmap v1075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:14:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:42 smithi055 ceph-mon[97033]: pgmap v1076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:42 smithi059 ceph-mon[97450]: pgmap v1076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:44 smithi055 ceph-mon[97033]: pgmap v1077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:44 smithi059 ceph-mon[97450]: pgmap v1077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:45.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:14:45.304+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:14:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:46 smithi055 ceph-mon[97033]: pgmap v1078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:14:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:46 smithi059 ceph-mon[97450]: pgmap v1078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:14:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:48 smithi055 ceph-mon[97033]: pgmap v1079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:48 smithi059 ceph-mon[97450]: pgmap v1079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:50 smithi055 ceph-mon[97033]: pgmap v1080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:14:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:50 smithi059 ceph-mon[97450]: pgmap v1080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:14:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:52 smithi055 ceph-mon[97033]: pgmap v1081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:52 smithi059 ceph-mon[97450]: pgmap v1081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:54 smithi055 ceph-mon[97033]: pgmap v1082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:54 smithi059 ceph-mon[97450]: pgmap v1082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:56 smithi055 ceph-mon[97033]: pgmap v1083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:14:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:56 smithi059 ceph-mon[97450]: pgmap v1083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:14:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:14:58 smithi055 ceph-mon[97033]: pgmap v1084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:14:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:14:58 smithi059 ceph-mon[97450]: pgmap v1084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:00.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:15:00.316+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:15:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:00 smithi055 ceph-mon[97033]: pgmap v1085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:15:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:00 smithi059 ceph-mon[97450]: pgmap v1085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:15:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:02 smithi055 ceph-mon[97033]: pgmap v1086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:02 smithi059 ceph-mon[97450]: pgmap v1086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:04 smithi055 ceph-mon[97033]: pgmap v1087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:15:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:15:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:04 smithi059 ceph-mon[97450]: pgmap v1087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:15:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:15:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:06 smithi055 ceph-mon[97033]: pgmap v1088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:15:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:06 smithi059 ceph-mon[97450]: pgmap v1088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:15:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:09 smithi055 ceph-mon[97033]: pgmap v1089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:15:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:09 smithi055 ceph-mon[97033]: pgmap v1090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-07T22:15:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:09 smithi055 ceph-mon[97033]: pgmap v1091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-07T22:15:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:15:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:09 smithi059 ceph-mon[97450]: pgmap v1089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:15:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:09 smithi059 ceph-mon[97450]: pgmap v1090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-07T22:15:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:09 smithi059 ceph-mon[97450]: pgmap v1091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-07T22:15:09.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:15:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:10 smithi055 ceph-mon[97033]: osdmap e145: 12 total, 12 up, 12 in 2022-09-07T22:15:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:10 smithi059 ceph-mon[97450]: osdmap e145: 12 total, 12 up, 12 in 2022-09-07T22:15:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:11 smithi055 ceph-mon[97033]: pgmap v1093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-07T22:15:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:11 smithi059 ceph-mon[97450]: pgmap v1093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-07T22:15:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:13 smithi055 ceph-mon[97033]: pgmap v1094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T22:15:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:13 smithi059 ceph-mon[97450]: pgmap v1094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T22:15:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:15 smithi055 ceph-mon[97033]: pgmap v1095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T22:15:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:15 smithi059 ceph-mon[97450]: pgmap v1095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T22:15:15.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:15:15.333+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:15:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:17 smithi055 ceph-mon[97033]: pgmap v1096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:15:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:17 smithi059 ceph-mon[97450]: pgmap v1096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:15:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:18 smithi055 ceph-mon[97033]: pgmap v1097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:15:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:18 smithi059 ceph-mon[97450]: pgmap v1097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:15:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:21 smithi055 ceph-mon[97033]: pgmap v1098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 186 B/s wr, 8 op/s 2022-09-07T22:15:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:21 smithi059 ceph-mon[97450]: pgmap v1098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 186 B/s wr, 8 op/s 2022-09-07T22:15:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:23 smithi055 ceph-mon[97033]: pgmap v1099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:15:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:23 smithi059 ceph-mon[97450]: pgmap v1099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:15:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:25 smithi055 ceph-mon[97033]: pgmap v1100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:15:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:25 smithi059 ceph-mon[97450]: pgmap v1100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:15:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:27 smithi055 ceph-mon[97033]: pgmap v1101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:15:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:27 smithi059 ceph-mon[97450]: pgmap v1101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:15:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:28 smithi055 ceph-mon[97033]: pgmap v1102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:28 smithi059 ceph-mon[97450]: pgmap v1102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:30.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:15:30.346+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:15:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:31 smithi055 ceph-mon[97033]: pgmap v1103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:15:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:31 smithi059 ceph-mon[97450]: pgmap v1103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:15:33.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:33 smithi055 ceph-mon[97033]: pgmap v1104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:33 smithi059 ceph-mon[97450]: pgmap v1104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:35 smithi055 ceph-mon[97033]: pgmap v1105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:35 smithi059 ceph-mon[97450]: pgmap v1105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:37 smithi055 ceph-mon[97033]: pgmap v1106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:15:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:37 smithi059 ceph-mon[97450]: pgmap v1106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:15:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:38 smithi055 ceph-mon[97033]: pgmap v1107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:38 smithi059 ceph-mon[97450]: pgmap v1107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:41 smithi055 ceph-mon[97033]: pgmap v1108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:15:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:41 smithi059 ceph-mon[97450]: pgmap v1108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:15:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:43 smithi055 ceph-mon[97033]: pgmap v1109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:43 smithi059 ceph-mon[97450]: pgmap v1109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:45 smithi055 ceph-mon[97033]: pgmap v1110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:45 smithi059 ceph-mon[97450]: pgmap v1110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:45.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:15:45.360+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:15:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:47 smithi055 ceph-mon[97033]: pgmap v1111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:15:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:47 smithi059 ceph-mon[97450]: pgmap v1111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:15:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:48 smithi055 ceph-mon[97033]: pgmap v1112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:48 smithi059 ceph-mon[97450]: pgmap v1112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:51 smithi055 ceph-mon[97033]: pgmap v1113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:15:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:51 smithi059 ceph-mon[97450]: pgmap v1113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:15:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:53 smithi055 ceph-mon[97033]: pgmap v1114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:53 smithi059 ceph-mon[97450]: pgmap v1114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:55 smithi055 ceph-mon[97033]: pgmap v1115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:55 smithi059 ceph-mon[97450]: pgmap v1115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:57 smithi055 ceph-mon[97033]: pgmap v1116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:15:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:57 smithi059 ceph-mon[97450]: pgmap v1116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:15:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:15:58 smithi055 ceph-mon[97033]: pgmap v1117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:15:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:15:58 smithi059 ceph-mon[97450]: pgmap v1117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:16:00.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:16:00.371+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:16:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:01 smithi055 ceph-mon[97033]: pgmap v1118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:16:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:01 smithi059 ceph-mon[97450]: pgmap v1118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:16:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:03 smithi055 ceph-mon[97033]: pgmap v1119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:16:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:03 smithi059 ceph-mon[97450]: pgmap v1119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:16:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:05 smithi055 ceph-mon[97033]: pgmap v1120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:16:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:16:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:16:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:05 smithi059 ceph-mon[97450]: pgmap v1120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:16:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:16:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:16:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:07 smithi055 ceph-mon[97033]: pgmap v1121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:16:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:07 smithi059 ceph-mon[97450]: pgmap v1121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:16:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:08 smithi055 ceph-mon[97033]: pgmap v1122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:16:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:16:09.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:16:09.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:16:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:08 smithi059 ceph-mon[97450]: pgmap v1122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:16:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:16:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:16:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:16:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:09 smithi055 ceph-mon[97033]: osdmap e146: 12 total, 12 up, 12 in 2022-09-07T22:16:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:09 smithi059 ceph-mon[97450]: osdmap e146: 12 total, 12 up, 12 in 2022-09-07T22:16:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:10 smithi055 ceph-mon[97033]: pgmap v1124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:16:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:10 smithi059 ceph-mon[97450]: pgmap v1124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:16:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:13 smithi055 ceph-mon[97033]: pgmap v1125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:16:13.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:13 smithi059 ceph-mon[97450]: pgmap v1125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:16:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:15 smithi055 ceph-mon[97033]: pgmap v1126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:16:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:15 smithi059 ceph-mon[97450]: pgmap v1126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:16:15.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:16:15.379+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:16:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:17 smithi055 ceph-mon[97033]: pgmap v1127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:16:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:17 smithi059 ceph-mon[97450]: pgmap v1127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:16:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:18 smithi055 ceph-mon[97033]: pgmap v1128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:16:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:18 smithi059 ceph-mon[97450]: pgmap v1128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:16:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:21 smithi055 ceph-mon[97033]: pgmap v1129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 182 B/s wr, 8 op/s 2022-09-07T22:16:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:21 smithi059 ceph-mon[97450]: pgmap v1129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 182 B/s wr, 8 op/s 2022-09-07T22:16:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:23 smithi055 ceph-mon[97033]: pgmap v1130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:16:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:23 smithi059 ceph-mon[97450]: pgmap v1130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:16:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:25 smithi055 ceph-mon[97033]: pgmap v1131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:16:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:25 smithi059 ceph-mon[97450]: pgmap v1131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:16:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:27 smithi055 ceph-mon[97033]: pgmap v1132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:16:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:27 smithi059 ceph-mon[97450]: pgmap v1132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:16:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:28 smithi055 ceph-mon[97033]: pgmap v1133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:16:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:28 smithi059 ceph-mon[97450]: pgmap v1133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:16:30.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:16:30.393+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:16:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:31 smithi055 ceph-mon[97033]: pgmap v1134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:16:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:31 smithi059 ceph-mon[97450]: pgmap v1134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:16:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:33 smithi055 ceph-mon[97033]: pgmap v1135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:16:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:33 smithi059 ceph-mon[97450]: pgmap v1135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:16:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:35 smithi055 ceph-mon[97033]: pgmap v1136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:16:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:35 smithi059 ceph-mon[97450]: pgmap v1136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:16:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:37 smithi055 ceph-mon[97033]: pgmap v1137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:16:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:37 smithi059 ceph-mon[97450]: pgmap v1137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:16:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:39 smithi055 ceph-mon[97033]: pgmap v1138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:16:39.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:39 smithi055 ceph-mon[97033]: pgmap v1139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:16:39.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:39 smithi055 ceph-mon[97033]: pgmap v1140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:16:39.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:16:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:39 smithi059 ceph-mon[97450]: pgmap v1138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:16:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:39 smithi059 ceph-mon[97450]: pgmap v1139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:16:39.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:39 smithi059 ceph-mon[97450]: pgmap v1140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:16:39.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:16:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:41 smithi055 ceph-mon[97033]: pgmap v1141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:16:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:41 smithi059 ceph-mon[97450]: pgmap v1141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:16:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:43 smithi055 ceph-mon[97033]: pgmap v1142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:16:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:43 smithi059 ceph-mon[97450]: pgmap v1142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:16:45.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:16:45.396+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:16:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:45 smithi055 ceph-mon[97033]: pgmap v1143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:16:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:45 smithi059 ceph-mon[97450]: pgmap v1143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:16:47.741 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:47 smithi055 ceph-mon[97033]: pgmap v1144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:16:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:47 smithi059 ceph-mon[97450]: pgmap v1144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:16:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:48 smithi055 ceph-mon[97033]: pgmap v1145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:16:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:48 smithi059 ceph-mon[97450]: pgmap v1145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:16:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:51 smithi055 ceph-mon[97033]: pgmap v1146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:16:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:51 smithi059 ceph-mon[97450]: pgmap v1146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:16:53.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:53 smithi055 ceph-mon[97033]: pgmap v1147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:16:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:53 smithi059 ceph-mon[97450]: pgmap v1147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:16:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:55 smithi055 ceph-mon[97033]: pgmap v1148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:16:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:55 smithi059 ceph-mon[97450]: pgmap v1148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:16:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:56 smithi055 ceph-mon[97033]: pgmap v1149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:16:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:56 smithi059 ceph-mon[97450]: pgmap v1149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:16:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:16:58 smithi055 ceph-mon[97033]: pgmap v1150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:16:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:16:58 smithi059 ceph-mon[97450]: pgmap v1150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:17:00.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:17:00.407+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:17:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:01 smithi055 ceph-mon[97033]: pgmap v1151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:17:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:01 smithi059 ceph-mon[97450]: pgmap v1151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:17:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:03 smithi055 ceph-mon[97033]: pgmap v1152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-07T22:17:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:03 smithi059 ceph-mon[97450]: pgmap v1152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-07T22:17:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:17:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:17:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:17:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:17:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:05 smithi055 ceph-mon[97033]: pgmap v1153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-07T22:17:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:05 smithi059 ceph-mon[97450]: pgmap v1153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-07T22:17:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:06 smithi055 ceph-mon[97033]: pgmap v1154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T22:17:06.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:06 smithi059 ceph-mon[97450]: pgmap v1154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T22:17:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:08 smithi055 ceph-mon[97033]: pgmap v1155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:17:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:08 smithi059 ceph-mon[97450]: pgmap v1155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:17:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:09 smithi055 ceph-mon[97033]: osdmap e147: 12 total, 12 up, 12 in 2022-09-07T22:17:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:09 smithi059 ceph-mon[97450]: osdmap e147: 12 total, 12 up, 12 in 2022-09-07T22:17:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:10 smithi055 ceph-mon[97033]: pgmap v1157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:17:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:10 smithi059 ceph-mon[97450]: pgmap v1157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:17:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:13 smithi055 ceph-mon[97033]: pgmap v1158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:17:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:13 smithi059 ceph-mon[97450]: pgmap v1158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:17:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:15 smithi055 ceph-mon[97033]: pgmap v1159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:17:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:15 smithi059 ceph-mon[97450]: pgmap v1159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:17:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:17:15.422+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:17:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:16 smithi055 ceph-mon[97033]: pgmap v1160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:17:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:16 smithi059 ceph-mon[97450]: pgmap v1160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:17:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:18 smithi055 ceph-mon[97033]: pgmap v1161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:17:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:18 smithi059 ceph-mon[97450]: pgmap v1161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:17:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:21 smithi055 ceph-mon[97033]: pgmap v1162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 178 B/s wr, 8 op/s 2022-09-07T22:17:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:21 smithi059 ceph-mon[97450]: pgmap v1162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 178 B/s wr, 8 op/s 2022-09-07T22:17:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:22 smithi055 ceph-mon[97033]: pgmap v1163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:17:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:22 smithi059 ceph-mon[97450]: pgmap v1163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:17:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:22 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-c[97426]: 2022-09-07T22:17:22.851+0000 7fe6ce761700 -1 mon.c@1(leader).osd e147 no beacon from osd.0 since 2022-09-07T22:02:22.603742+0000, 900.248647 seconds ago. marking down 2022-09-07T22:17:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:23 smithi055 ceph-mon[97033]: osd.0 marked down after no beacon for 900.248647 seconds 2022-09-07T22:17:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:23 smithi055 ceph-mon[97033]: Health check failed: 1 osds down (OSD_DOWN) 2022-09-07T22:17:23.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:23 smithi055 ceph-mon[97033]: osdmap e148: 12 total, 11 up, 12 in 2022-09-07T22:17:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:23 smithi059 ceph-mon[97450]: osd.0 marked down after no beacon for 900.248647 seconds 2022-09-07T22:17:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:23 smithi059 ceph-mon[97450]: Health check failed: 1 osds down (OSD_DOWN) 2022-09-07T22:17:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:23 smithi059 ceph-mon[97450]: osdmap e148: 12 total, 11 up, 12 in 2022-09-07T22:17:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:24 smithi055 ceph-mon[97033]: osdmap e149: 12 total, 11 up, 12 in 2022-09-07T22:17:25.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:24 smithi055 ceph-mon[97033]: pgmap v1166: 129 pgs: 10 stale+active+clean, 119 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2022-09-07T22:17:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:24 smithi059 ceph-mon[97450]: osdmap e149: 12 total, 11 up, 12 in 2022-09-07T22:17:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:24 smithi059 ceph-mon[97450]: pgmap v1166: 129 pgs: 10 stale+active+clean, 119 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2022-09-07T22:17:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:26 smithi055 ceph-mon[97033]: pgmap v1167: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:26.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:26 smithi055 ceph-mon[97033]: Health check failed: Degraded data redundancy: 987/13572 objects degraded (7.272%), 16 pgs degraded (PG_DEGRADED) 2022-09-07T22:17:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:26 smithi059 ceph-mon[97450]: pgmap v1167: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:26.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:26 smithi059 ceph-mon[97450]: Health check failed: Degraded data redundancy: 987/13572 objects degraded (7.272%), 16 pgs degraded (PG_DEGRADED) 2022-09-07T22:17:29.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:28 smithi055 ceph-mon[97033]: osdmap e150: 12 total, 11 up, 12 in 2022-09-07T22:17:29.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:28 smithi055 ceph-mon[97033]: pgmap v1169: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:28 smithi059 ceph-mon[97450]: osdmap e150: 12 total, 11 up, 12 in 2022-09-07T22:17:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:28 smithi059 ceph-mon[97450]: pgmap v1169: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:30.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:17:30.432+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:17:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:31 smithi055 ceph-mon[97033]: pgmap v1170: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 685 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:31 smithi059 ceph-mon[97450]: pgmap v1170: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 685 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:32 smithi055 ceph-mon[97033]: pgmap v1171: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 605 B/s rd, 0 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:32 smithi059 ceph-mon[97450]: pgmap v1171: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 605 B/s rd, 0 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:34 smithi055 ceph-mon[97033]: pgmap v1172: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 0 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:34 smithi059 ceph-mon[97450]: pgmap v1172: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 0 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:36 smithi055 ceph-mon[97033]: pgmap v1173: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:36 smithi059 ceph-mon[97450]: pgmap v1173: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:38 smithi055 ceph-mon[97033]: pgmap v1174: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 977 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:17:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:17:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:17:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:38 smithi059 ceph-mon[97450]: pgmap v1174: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 977 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:17:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:17:39.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:17:40.698 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:40 smithi055 ceph-mon[97033]: pgmap v1175: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:40 smithi059 ceph-mon[97450]: pgmap v1175: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:42 smithi055 ceph-mon[97033]: pgmap v1176: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:42 smithi059 ceph-mon[97450]: pgmap v1176: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:44 smithi055 ceph-mon[97033]: pgmap v1177: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:44 smithi059 ceph-mon[97450]: pgmap v1177: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:45.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:17:45.444+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:17:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:46 smithi055 ceph-mon[97033]: pgmap v1178: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:46 smithi059 ceph-mon[97450]: pgmap v1178: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:48 smithi055 ceph-mon[97033]: pgmap v1179: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:49.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:48 smithi059 ceph-mon[97450]: pgmap v1179: 129 pgs: 19 active+undersized+wait, 16 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:50 smithi055 ceph-mon[97033]: pgmap v1180: 129 pgs: 2 active+undersized+degraded, 19 active+undersized+wait, 14 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:50 smithi059 ceph-mon[97450]: pgmap v1180: 129 pgs: 2 active+undersized+degraded, 19 active+undersized+wait, 14 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:52 smithi055 ceph-mon[97033]: pgmap v1181: 129 pgs: 2 active+undersized+degraded, 19 active+undersized+wait, 14 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:52 smithi059 ceph-mon[97450]: pgmap v1181: 129 pgs: 2 active+undersized+degraded, 19 active+undersized+wait, 14 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:54 smithi055 ceph-mon[97033]: pgmap v1182: 129 pgs: 2 active+undersized+degraded, 19 active+undersized+wait, 14 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:54 smithi059 ceph-mon[97450]: pgmap v1182: 129 pgs: 2 active+undersized+degraded, 19 active+undersized+wait, 14 active+undersized+degraded+wait, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:56 smithi055 ceph-mon[97033]: pgmap v1183: 129 pgs: 8 active+undersized+degraded, 8 active+undersized+degraded+wait, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:56 smithi059 ceph-mon[97450]: pgmap v1183: 129 pgs: 8 active+undersized+degraded, 8 active+undersized+degraded+wait, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:17:58 smithi055 ceph-mon[97033]: pgmap v1184: 129 pgs: 8 active+undersized+degraded, 8 active+undersized+degraded+wait, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:17:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:17:58 smithi059 ceph-mon[97450]: pgmap v1184: 129 pgs: 8 active+undersized+degraded, 8 active+undersized+degraded+wait, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:00 smithi055 ceph-mon[97033]: pgmap v1185: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:00 smithi059 ceph-mon[97450]: pgmap v1185: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:00.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:18:00.458+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:18:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:02 smithi055 ceph-mon[97033]: pgmap v1186: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:02 smithi059 ceph-mon[97450]: pgmap v1186: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:18:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:18:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:18:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:18:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:05 smithi055 ceph-mon[97033]: pgmap v1187: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:05 smithi059 ceph-mon[97450]: pgmap v1187: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:06 smithi055 ceph-mon[97033]: pgmap v1188: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:06 smithi059 ceph-mon[97450]: pgmap v1188: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:08 smithi059 ceph-mon[97450]: pgmap v1189: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:18:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:08 smithi055 ceph-mon[97033]: pgmap v1189: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:18:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:09 smithi055 ceph-mon[97033]: pgmap v1190: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:09 smithi055 ceph-mon[97033]: pgmap v1191: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 65 KiB/s wr, 25 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:09 smithi055 ceph-mon[97033]: osdmap e151: 12 total, 11 up, 12 in 2022-09-07T22:18:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:09 smithi059 ceph-mon[97450]: pgmap v1190: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:09 smithi059 ceph-mon[97450]: pgmap v1191: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 65 KiB/s wr, 25 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:09 smithi059 ceph-mon[97450]: osdmap e151: 12 total, 11 up, 12 in 2022-09-07T22:18:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:11 smithi055 ceph-mon[97033]: pgmap v1193: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 86 KiB/s wr, 38 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:11 smithi059 ceph-mon[97450]: pgmap v1193: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 86 KiB/s wr, 38 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:13 smithi055 ceph-mon[97033]: pgmap v1194: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:13 smithi059 ceph-mon[97450]: pgmap v1194: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:18:15.474+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:18:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:15 smithi055 ceph-mon[97033]: pgmap v1195: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:15 smithi059 ceph-mon[97450]: pgmap v1195: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:17 smithi055 ceph-mon[97033]: pgmap v1196: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:17 smithi059 ceph-mon[97450]: pgmap v1196: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:19 smithi055 ceph-mon[97033]: pgmap v1197: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:19 smithi059 ceph-mon[97450]: pgmap v1197: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:21 smithi055 ceph-mon[97033]: pgmap v1198: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 175 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:21 smithi059 ceph-mon[97450]: pgmap v1198: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 175 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:23 smithi055 ceph-mon[97033]: pgmap v1199: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:23 smithi059 ceph-mon[97450]: pgmap v1199: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:24 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 987/13572 objects degraded (7.272%), 16 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2022-09-07T22:18:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:24 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 987/13572 objects degraded (7.272%), 16 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2022-09-07T22:18:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:25 smithi055 ceph-mon[97033]: pgmap v1200: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:25 smithi059 ceph-mon[97450]: pgmap v1200: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:27 smithi055 ceph-mon[97033]: pgmap v1201: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:27 smithi059 ceph-mon[97450]: pgmap v1201: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:29 smithi055 ceph-mon[97033]: pgmap v1202: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:29 smithi059 ceph-mon[97450]: pgmap v1202: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:30.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:18:30.490+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:18:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:31 smithi055 ceph-mon[97033]: pgmap v1203: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:31 smithi059 ceph-mon[97450]: pgmap v1203: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:33 smithi055 ceph-mon[97033]: pgmap v1204: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:33 smithi059 ceph-mon[97450]: pgmap v1204: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:35 smithi059 ceph-mon[97450]: pgmap v1205: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:35 smithi055 ceph-mon[97033]: pgmap v1205: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:37 smithi059 ceph-mon[97450]: pgmap v1206: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:37 smithi055 ceph-mon[97033]: pgmap v1206: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:39 smithi059 ceph-mon[97450]: pgmap v1207: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:39 smithi055 ceph-mon[97033]: pgmap v1207: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:41 smithi059 ceph-mon[97450]: pgmap v1208: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:41 smithi055 ceph-mon[97033]: pgmap v1208: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:43 smithi059 ceph-mon[97450]: pgmap v1209: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:43 smithi055 ceph-mon[97033]: pgmap v1209: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:45.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:18:45.491+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:18:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:45 smithi059 ceph-mon[97450]: pgmap v1210: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:46 smithi055 ceph-mon[97033]: pgmap v1210: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:48 smithi059 ceph-mon[97450]: pgmap v1211: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:48 smithi055 ceph-mon[97033]: pgmap v1211: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:50 smithi059 ceph-mon[97450]: pgmap v1212: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:50 smithi055 ceph-mon[97033]: pgmap v1212: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:52 smithi059 ceph-mon[97450]: pgmap v1213: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:52 smithi055 ceph-mon[97033]: pgmap v1213: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:54 smithi059 ceph-mon[97450]: pgmap v1214: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:54 smithi055 ceph-mon[97033]: pgmap v1214: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:56 smithi059 ceph-mon[97450]: pgmap v1215: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:56 smithi055 ceph-mon[97033]: pgmap v1215: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:18:58 smithi059 ceph-mon[97450]: pgmap v1216: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:18:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:18:58 smithi055 ceph-mon[97033]: pgmap v1216: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:00 smithi059 ceph-mon[97450]: pgmap v1217: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:00 smithi055 ceph-mon[97033]: pgmap v1217: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:00.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:19:00.494+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:19:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:02 smithi059 ceph-mon[97450]: pgmap v1218: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:02 smithi055 ceph-mon[97033]: pgmap v1218: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:04 smithi059 ceph-mon[97450]: pgmap v1219: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:04 smithi055 ceph-mon[97033]: pgmap v1219: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:19:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:19:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:19:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:19:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:06 smithi055 ceph-mon[97033]: pgmap v1220: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:06 smithi059 ceph-mon[97450]: pgmap v1220: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:08 smithi055 ceph-mon[97033]: pgmap v1221: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:08 smithi059 ceph-mon[97450]: pgmap v1221: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:19:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:19:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:19:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:19:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:19:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:19:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:10 smithi055 ceph-mon[97033]: pgmap v1222: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:10 smithi055 ceph-mon[97033]: osdmap e152: 12 total, 11 up, 12 in 2022-09-07T22:19:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:10 smithi059 ceph-mon[97450]: pgmap v1222: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:10 smithi059 ceph-mon[97450]: osdmap e152: 12 total, 11 up, 12 in 2022-09-07T22:19:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:12 smithi055 ceph-mon[97033]: pgmap v1224: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:12 smithi059 ceph-mon[97450]: pgmap v1224: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:14 smithi055 ceph-mon[97033]: pgmap v1225: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:14 smithi059 ceph-mon[97450]: pgmap v1225: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:19:15.508+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:19:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:16 smithi055 ceph-mon[97033]: pgmap v1226: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:16 smithi059 ceph-mon[97450]: pgmap v1226: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:18 smithi055 ceph-mon[97033]: pgmap v1227: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:18.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:18 smithi059 ceph-mon[97450]: pgmap v1227: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:20 smithi055 ceph-mon[97033]: pgmap v1228: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:20 smithi059 ceph-mon[97450]: pgmap v1228: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:22 smithi055 ceph-mon[97033]: pgmap v1229: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 177 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:22 smithi059 ceph-mon[97450]: pgmap v1229: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 177 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:24 smithi055 ceph-mon[97033]: pgmap v1230: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:24 smithi059 ceph-mon[97450]: pgmap v1230: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:26 smithi055 ceph-mon[97033]: pgmap v1231: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:26 smithi059 ceph-mon[97450]: pgmap v1231: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:28 smithi055 ceph-mon[97033]: pgmap v1232: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:28 smithi059 ceph-mon[97450]: pgmap v1232: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:30 smithi055 ceph-mon[97033]: pgmap v1233: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:30 smithi059 ceph-mon[97450]: pgmap v1233: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:30.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:19:30.520+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:19:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:32 smithi055 ceph-mon[97033]: pgmap v1234: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:32 smithi059 ceph-mon[97450]: pgmap v1234: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:34 smithi055 ceph-mon[97033]: pgmap v1235: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:34 smithi059 ceph-mon[97450]: pgmap v1235: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:36 smithi055 ceph-mon[97033]: pgmap v1236: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:36 smithi059 ceph-mon[97450]: pgmap v1236: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:38 smithi055 ceph-mon[97033]: pgmap v1237: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:38 smithi059 ceph-mon[97450]: pgmap v1237: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:39 smithi055 ceph-mon[97033]: pgmap v1238: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:39 smithi055 ceph-mon[97033]: pgmap v1239: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:39 smithi055 ceph-mon[97033]: pgmap v1240: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:40.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:19:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:39 smithi059 ceph-mon[97450]: pgmap v1238: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:40.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:39 smithi059 ceph-mon[97450]: pgmap v1239: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:40.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:39 smithi059 ceph-mon[97450]: pgmap v1240: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:40.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:19:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:41 smithi055 ceph-mon[97033]: pgmap v1241: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:41 smithi059 ceph-mon[97450]: pgmap v1241: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:43 smithi055 ceph-mon[97033]: pgmap v1242: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:43 smithi059 ceph-mon[97450]: pgmap v1242: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:45.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:19:45.533+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:19:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:45 smithi055 ceph-mon[97033]: pgmap v1243: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:45 smithi059 ceph-mon[97450]: pgmap v1243: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:47 smithi055 ceph-mon[97033]: pgmap v1244: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:47 smithi059 ceph-mon[97450]: pgmap v1244: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:49 smithi055 ceph-mon[97033]: pgmap v1245: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:49 smithi059 ceph-mon[97450]: pgmap v1245: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:51 smithi055 ceph-mon[97033]: pgmap v1246: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:51 smithi059 ceph-mon[97450]: pgmap v1246: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:53 smithi055 ceph-mon[97033]: pgmap v1247: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:53 smithi059 ceph-mon[97450]: pgmap v1247: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:55 smithi055 ceph-mon[97033]: pgmap v1248: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:55 smithi059 ceph-mon[97450]: pgmap v1248: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:57 smithi055 ceph-mon[97033]: pgmap v1249: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:19:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:57 smithi059 ceph-mon[97450]: pgmap v1249: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:19:59 smithi055 ceph-mon[97033]: pgmap v1250: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:19:59 smithi059 ceph-mon[97450]: pgmap v1250: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:00.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:20:00.541+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:20:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b; 1 osds down; Degraded data redundancy: 987/13572 objects degraded (7.272%), 16 pgs degraded, 35 pgs undersized 2022-09-07T22:20:01.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-07T22:20:01.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-07T22:20:01.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T22:20:01.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T22:20:01.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: [WRN] OSD_DOWN: 1 osds down 2022-09-07T22:20:01.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: osd.0 (root=default,host=smithi047) is down 2022-09-07T22:20:01.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: [WRN] PG_DEGRADED: Degraded data redundancy: 987/13572 objects degraded (7.272%), 16 pgs degraded, 35 pgs undersized 2022-09-07T22:20:01.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 2.1 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,3] 2022-09-07T22:20:01.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 2.4 is stuck undersized for 2m, current state active+undersized, last acting [9,2] 2022-09-07T22:20:01.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 2.5 is stuck undersized for 2m, current state active+undersized+degraded, last acting [8,9] 2022-09-07T22:20:01.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 2.8 is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,2] 2022-09-07T22:20:01.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 2.c is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,11] 2022-09-07T22:20:01.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 2.10 is stuck undersized for 2m, current state active+undersized+degraded, last acting [8,2] 2022-09-07T22:20:01.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 2.15 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,10] 2022-09-07T22:20:01.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 2.1e is stuck undersized for 2m, current state active+undersized, last acting [3,5] 2022-09-07T22:20:01.239 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.3 is stuck undersized for 2m, current state active+undersized, last acting [10,4] 2022-09-07T22:20:01.239 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.6 is stuck undersized for 2m, current state active+undersized, last acting [3,4] 2022-09-07T22:20:01.240 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.8 is stuck undersized for 2m, current state active+undersized, last acting [11,2] 2022-09-07T22:20:01.240 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.b is stuck undersized for 2m, current state active+undersized, last acting [3,10] 2022-09-07T22:20:01.240 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.17 is stuck undersized for 2m, current state active+undersized, last acting [4,5] 2022-09-07T22:20:01.241 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.1f is stuck undersized for 2m, current state active+undersized, last acting [1,9] 2022-09-07T22:20:01.241 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.22 is stuck undersized for 2m, current state active+undersized, last acting [2,3] 2022-09-07T22:20:01.242 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.23 is stuck undersized for 2m, current state active+undersized, last acting [6,4] 2022-09-07T22:20:01.242 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.24 is stuck undersized for 2m, current state active+undersized, last acting [6,4] 2022-09-07T22:20:01.243 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.2b is stuck undersized for 2m, current state active+undersized, last acting [6,7] 2022-09-07T22:20:01.243 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.2f is stuck undersized for 2m, current state active+undersized, last acting [11,1] 2022-09-07T22:20:01.243 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.30 is stuck undersized for 2m, current state active+undersized, last acting [1,7] 2022-09-07T22:20:01.244 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.31 is stuck undersized for 2m, current state active+undersized, last acting [4,2] 2022-09-07T22:20:01.244 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.32 is stuck undersized for 2m, current state active+undersized, last acting [8,10] 2022-09-07T22:20:01.244 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.3d is stuck undersized for 2m, current state active+undersized, last acting [2,1] 2022-09-07T22:20:01.245 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.3e is stuck undersized for 2m, current state active+undersized, last acting [11,10] 2022-09-07T22:20:01.245 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 3.3f is stuck undersized for 2m, current state active+undersized, last acting [11,1] 2022-09-07T22:20:01.245 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 4.0 is stuck undersized for 2m, current state active+undersized+degraded, last acting [3,10,NONE,7] 2022-09-07T22:20:01.246 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 4.4 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,NONE,1,8] 2022-09-07T22:20:01.246 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 4.7 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,8,10,NONE] 2022-09-07T22:20:01.247 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 4.9 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,NONE,3,1] 2022-09-07T22:20:01.247 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 4.b is stuck undersized for 2m, current state active+undersized+degraded, last acting [NONE,9,1,4] 2022-09-07T22:20:01.247 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 4.10 is stuck undersized for 2m, current state active+undersized+degraded, last acting [9,NONE,2,4] 2022-09-07T22:20:01.248 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 4.17 is stuck undersized for 2m, current state active+undersized+degraded, last acting [8,7,NONE,3] 2022-09-07T22:20:01.248 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 4.19 is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,3,9,NONE] 2022-09-07T22:20:01.248 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 4.1e is stuck undersized for 2m, current state active+undersized+degraded, last acting [NONE,7,6,1] 2022-09-07T22:20:01.249 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:00 smithi055 ceph-mon[97033]: pg 4.1f is stuck undersized for 2m, current state active+undersized+degraded, last acting [11,4,9,NONE] 2022-09-07T22:20:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b; 1 osds down; Degraded data redundancy: 987/13572 objects degraded (7.272%), 16 pgs degraded, 35 pgs undersized 2022-09-07T22:20:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-07T22:20:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-07T22:20:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T22:20:01.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T22:20:01.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: [WRN] OSD_DOWN: 1 osds down 2022-09-07T22:20:01.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: osd.0 (root=default,host=smithi047) is down 2022-09-07T22:20:01.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: [WRN] PG_DEGRADED: Degraded data redundancy: 987/13572 objects degraded (7.272%), 16 pgs degraded, 35 pgs undersized 2022-09-07T22:20:01.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 2.1 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,3] 2022-09-07T22:20:01.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 2.4 is stuck undersized for 2m, current state active+undersized, last acting [9,2] 2022-09-07T22:20:01.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 2.5 is stuck undersized for 2m, current state active+undersized+degraded, last acting [8,9] 2022-09-07T22:20:01.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 2.8 is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,2] 2022-09-07T22:20:01.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 2.c is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,11] 2022-09-07T22:20:01.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 2.10 is stuck undersized for 2m, current state active+undersized+degraded, last acting [8,2] 2022-09-07T22:20:01.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 2.15 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,10] 2022-09-07T22:20:01.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 2.1e is stuck undersized for 2m, current state active+undersized, last acting [3,5] 2022-09-07T22:20:01.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.3 is stuck undersized for 2m, current state active+undersized, last acting [10,4] 2022-09-07T22:20:01.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.6 is stuck undersized for 2m, current state active+undersized, last acting [3,4] 2022-09-07T22:20:01.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.8 is stuck undersized for 2m, current state active+undersized, last acting [11,2] 2022-09-07T22:20:01.289 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.b is stuck undersized for 2m, current state active+undersized, last acting [3,10] 2022-09-07T22:20:01.289 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.17 is stuck undersized for 2m, current state active+undersized, last acting [4,5] 2022-09-07T22:20:01.289 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.1f is stuck undersized for 2m, current state active+undersized, last acting [1,9] 2022-09-07T22:20:01.290 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.22 is stuck undersized for 2m, current state active+undersized, last acting [2,3] 2022-09-07T22:20:01.290 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.23 is stuck undersized for 2m, current state active+undersized, last acting [6,4] 2022-09-07T22:20:01.290 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.24 is stuck undersized for 2m, current state active+undersized, last acting [6,4] 2022-09-07T22:20:01.291 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.2b is stuck undersized for 2m, current state active+undersized, last acting [6,7] 2022-09-07T22:20:01.291 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.2f is stuck undersized for 2m, current state active+undersized, last acting [11,1] 2022-09-07T22:20:01.291 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.30 is stuck undersized for 2m, current state active+undersized, last acting [1,7] 2022-09-07T22:20:01.292 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.31 is stuck undersized for 2m, current state active+undersized, last acting [4,2] 2022-09-07T22:20:01.292 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.32 is stuck undersized for 2m, current state active+undersized, last acting [8,10] 2022-09-07T22:20:01.292 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.3d is stuck undersized for 2m, current state active+undersized, last acting [2,1] 2022-09-07T22:20:01.293 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.3e is stuck undersized for 2m, current state active+undersized, last acting [11,10] 2022-09-07T22:20:01.293 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 3.3f is stuck undersized for 2m, current state active+undersized, last acting [11,1] 2022-09-07T22:20:01.293 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 4.0 is stuck undersized for 2m, current state active+undersized+degraded, last acting [3,10,NONE,7] 2022-09-07T22:20:01.294 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 4.4 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,NONE,1,8] 2022-09-07T22:20:01.294 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 4.7 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,8,10,NONE] 2022-09-07T22:20:01.294 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 4.9 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,NONE,3,1] 2022-09-07T22:20:01.295 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 4.b is stuck undersized for 2m, current state active+undersized+degraded, last acting [NONE,9,1,4] 2022-09-07T22:20:01.295 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 4.10 is stuck undersized for 2m, current state active+undersized+degraded, last acting [9,NONE,2,4] 2022-09-07T22:20:01.295 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 4.17 is stuck undersized for 2m, current state active+undersized+degraded, last acting [8,7,NONE,3] 2022-09-07T22:20:01.296 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 4.19 is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,3,9,NONE] 2022-09-07T22:20:01.296 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 4.1e is stuck undersized for 2m, current state active+undersized+degraded, last acting [NONE,7,6,1] 2022-09-07T22:20:01.296 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:00 smithi059 ceph-mon[97450]: pg 4.1f is stuck undersized for 2m, current state active+undersized+degraded, last acting [11,4,9,NONE] 2022-09-07T22:20:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:01 smithi055 ceph-mon[97033]: pgmap v1251: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:01 smithi059 ceph-mon[97450]: pgmap v1251: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:03 smithi055 ceph-mon[97033]: pgmap v1252: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:03 smithi059 ceph-mon[97450]: pgmap v1252: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:20:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:20:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:20:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:20:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:05 smithi055 ceph-mon[97033]: pgmap v1253: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:05 smithi059 ceph-mon[97450]: pgmap v1253: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:07 smithi055 ceph-mon[97033]: pgmap v1254: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:07 smithi059 ceph-mon[97450]: pgmap v1254: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:09 smithi055 ceph-mon[97033]: pgmap v1255: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:09 smithi055 ceph-mon[97033]: osdmap e153: 12 total, 11 up, 12 in 2022-09-07T22:20:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:09 smithi059 ceph-mon[97450]: pgmap v1255: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:09 smithi059 ceph-mon[97450]: osdmap e153: 12 total, 11 up, 12 in 2022-09-07T22:20:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:11 smithi055 ceph-mon[97033]: pgmap v1257: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:12.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:11 smithi059 ceph-mon[97450]: pgmap v1257: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:13 smithi055 ceph-mon[97033]: pgmap v1258: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:13 smithi059 ceph-mon[97450]: pgmap v1258: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:20:15.549+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:20:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:15 smithi055 ceph-mon[97033]: pgmap v1259: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:15 smithi059 ceph-mon[97450]: pgmap v1259: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:17 smithi055 ceph-mon[97033]: pgmap v1260: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:17 smithi059 ceph-mon[97450]: pgmap v1260: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:19 smithi055 ceph-mon[97033]: pgmap v1261: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:19 smithi059 ceph-mon[97450]: pgmap v1261: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:21 smithi055 ceph-mon[97033]: pgmap v1262: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 172 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:21 smithi059 ceph-mon[97450]: pgmap v1262: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 172 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:23 smithi055 ceph-mon[97033]: pgmap v1263: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:23 smithi059 ceph-mon[97450]: pgmap v1263: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:25 smithi059 ceph-mon[97450]: pgmap v1264: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:26.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:25 smithi055 ceph-mon[97033]: pgmap v1264: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:27 smithi059 ceph-mon[97450]: pgmap v1265: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:27 smithi055 ceph-mon[97033]: pgmap v1265: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:30.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:29 smithi059 ceph-mon[97450]: pgmap v1266: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:29 smithi055 ceph-mon[97033]: pgmap v1266: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:30.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:20:30.555+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:20:32.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:31 smithi059 ceph-mon[97450]: pgmap v1267: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:31 smithi055 ceph-mon[97033]: pgmap v1267: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:33 smithi059 ceph-mon[97450]: pgmap v1268: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:33 smithi055 ceph-mon[97033]: pgmap v1268: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:35 smithi059 ceph-mon[97450]: pgmap v1269: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:35 smithi055 ceph-mon[97033]: pgmap v1269: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:37 smithi059 ceph-mon[97450]: pgmap v1270: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:37 smithi055 ceph-mon[97033]: pgmap v1270: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:20:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:20:39.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:20:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:20:39.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:20:39.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:20:40.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:40 smithi059 ceph-mon[97450]: pgmap v1271: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:40 smithi055 ceph-mon[97033]: pgmap v1271: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:42 smithi059 ceph-mon[97450]: pgmap v1272: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:42 smithi055 ceph-mon[97033]: pgmap v1272: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:44 smithi059 ceph-mon[97450]: pgmap v1273: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:44 smithi055 ceph-mon[97033]: pgmap v1273: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:45.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:20:45.561+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:20:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:46 smithi059 ceph-mon[97450]: pgmap v1274: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:46.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:46 smithi055 ceph-mon[97033]: pgmap v1274: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:48 smithi059 ceph-mon[97450]: pgmap v1275: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:48.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:48 smithi055 ceph-mon[97033]: pgmap v1275: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:50 smithi059 ceph-mon[97450]: pgmap v1276: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:50 smithi055 ceph-mon[97033]: pgmap v1276: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:52 smithi059 ceph-mon[97450]: pgmap v1277: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:52 smithi055 ceph-mon[97033]: pgmap v1277: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:54 smithi055 ceph-mon[97033]: pgmap v1278: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:54 smithi059 ceph-mon[97450]: pgmap v1278: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:56 smithi055 ceph-mon[97033]: pgmap v1279: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:56 smithi059 ceph-mon[97450]: pgmap v1279: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:20:58 smithi055 ceph-mon[97033]: pgmap v1280: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:20:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:20:58 smithi059 ceph-mon[97450]: pgmap v1280: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:00 smithi055 ceph-mon[97033]: pgmap v1281: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:00 smithi059 ceph-mon[97450]: pgmap v1281: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:00.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:21:00.566+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:21:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:02 smithi055 ceph-mon[97033]: pgmap v1282: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:02.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:02 smithi059 ceph-mon[97450]: pgmap v1282: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:04 smithi055 ceph-mon[97033]: pgmap v1283: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:04 smithi059 ceph-mon[97450]: pgmap v1283: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:21:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:21:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:21:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:21:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:06 smithi055 ceph-mon[97033]: pgmap v1284: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:06 smithi059 ceph-mon[97450]: pgmap v1284: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:08 smithi055 ceph-mon[97033]: pgmap v1285: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:08 smithi059 ceph-mon[97450]: pgmap v1285: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:10 smithi055 ceph-mon[97033]: pgmap v1286: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:10.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:10 smithi055 ceph-mon[97033]: osdmap e154: 12 total, 11 up, 12 in 2022-09-07T22:21:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:10 smithi059 ceph-mon[97450]: pgmap v1286: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:10 smithi059 ceph-mon[97450]: osdmap e154: 12 total, 11 up, 12 in 2022-09-07T22:21:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:12 smithi055 ceph-mon[97033]: pgmap v1288: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:12 smithi059 ceph-mon[97450]: pgmap v1288: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:14 smithi059 ceph-mon[97450]: pgmap v1289: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:14.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:21:14.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:21:14.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:21:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:14 smithi055 ceph-mon[97033]: pgmap v1289: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:21:14.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:21:14.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:21:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:15 smithi055 ceph-mon[97033]: pgmap v1290: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 114 B/s wr, 3 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:15 smithi059 ceph-mon[97450]: pgmap v1290: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 114 B/s wr, 3 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:21:15.581+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:21:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:17 smithi055 ceph-mon[97033]: pgmap v1291: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:17 smithi059 ceph-mon[97450]: pgmap v1291: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:19 smithi055 ceph-mon[97033]: pgmap v1292: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:19 smithi059 ceph-mon[97450]: pgmap v1292: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:21 smithi055 ceph-mon[97033]: pgmap v1293: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 191 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:21 smithi059 ceph-mon[97450]: pgmap v1293: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 191 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:23 smithi055 ceph-mon[97033]: pgmap v1294: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:23 smithi059 ceph-mon[97450]: pgmap v1294: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:25 smithi055 ceph-mon[97033]: pgmap v1295: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:25.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:25 smithi059 ceph-mon[97450]: pgmap v1295: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:27 smithi055 ceph-mon[97033]: pgmap v1296: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:27 smithi059 ceph-mon[97450]: pgmap v1296: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:29 smithi055 ceph-mon[97033]: pgmap v1297: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:29 smithi059 ceph-mon[97450]: pgmap v1297: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:30.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:21:30.597+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:21:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:31 smithi055 ceph-mon[97033]: pgmap v1298: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:31.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:31 smithi059 ceph-mon[97450]: pgmap v1298: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:33 smithi055 ceph-mon[97033]: pgmap v1299: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:33 smithi059 ceph-mon[97450]: pgmap v1299: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:35 smithi055 ceph-mon[97033]: pgmap v1300: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:35 smithi059 ceph-mon[97450]: pgmap v1300: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:37 smithi055 ceph-mon[97033]: pgmap v1301: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:37 smithi059 ceph-mon[97450]: pgmap v1301: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:39.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:39 smithi055 ceph-mon[97033]: pgmap v1302: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:39 smithi059 ceph-mon[97450]: pgmap v1302: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:41 smithi055 ceph-mon[97033]: pgmap v1303: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:41 smithi059 ceph-mon[97450]: pgmap v1303: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:43 smithi055 ceph-mon[97033]: pgmap v1304: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:43 smithi059 ceph-mon[97450]: pgmap v1304: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:45 smithi055 ceph-mon[97033]: pgmap v1305: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:45 smithi059 ceph-mon[97450]: pgmap v1305: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:45.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:21:45.598+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:21:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:47 smithi055 ceph-mon[97033]: pgmap v1306: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:47.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:47 smithi059 ceph-mon[97450]: pgmap v1306: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:49 smithi055 ceph-mon[97033]: pgmap v1307: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:49 smithi059 ceph-mon[97450]: pgmap v1307: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:51 smithi055 ceph-mon[97033]: pgmap v1308: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:51 smithi059 ceph-mon[97450]: pgmap v1308: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:53 smithi055 ceph-mon[97033]: pgmap v1309: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:53 smithi059 ceph-mon[97450]: pgmap v1309: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:55.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:55 smithi055 ceph-mon[97033]: pgmap v1310: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:55 smithi059 ceph-mon[97450]: pgmap v1310: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:57 smithi055 ceph-mon[97033]: pgmap v1311: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:57 smithi059 ceph-mon[97450]: pgmap v1311: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:21:59 smithi055 ceph-mon[97033]: pgmap v1312: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:21:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:21:59 smithi059 ceph-mon[97450]: pgmap v1312: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:00.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:22:00.612+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:22:01.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:01 smithi055 ceph-mon[97033]: pgmap v1313: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:01 smithi059 ceph-mon[97450]: pgmap v1313: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:03 smithi055 ceph-mon[97033]: pgmap v1314: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:03 smithi059 ceph-mon[97450]: pgmap v1314: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:05 smithi055 ceph-mon[97033]: pgmap v1315: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:22:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:22:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:05 smithi059 ceph-mon[97450]: pgmap v1315: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:22:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:22:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:07 smithi055 ceph-mon[97033]: pgmap v1316: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:07 smithi059 ceph-mon[97450]: pgmap v1316: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:09 smithi055 ceph-mon[97033]: pgmap v1317: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:09.542 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:09 smithi059 ceph-mon[97450]: pgmap v1317: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:10 smithi055 ceph-mon[97033]: osdmap e155: 12 total, 11 up, 12 in 2022-09-07T22:22:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:10 smithi059 ceph-mon[97450]: osdmap e155: 12 total, 11 up, 12 in 2022-09-07T22:22:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:11 smithi055 ceph-mon[97033]: pgmap v1319: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:11.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:11 smithi059 ceph-mon[97450]: pgmap v1319: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:13 smithi055 ceph-mon[97033]: pgmap v1320: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:13 smithi059 ceph-mon[97450]: pgmap v1320: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:22:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:22:14.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:22:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:22:14.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:22:14.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:22:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:15 smithi055 ceph-mon[97033]: pgmap v1321: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:22:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:15 smithi059 ceph-mon[97450]: pgmap v1321: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:15.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:22:15.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:22:15.628+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:22:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:16 smithi055 ceph-mon[97033]: pgmap v1322: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 118 B/s wr, 3 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:16 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-07T22:22:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:16 smithi059 ceph-mon[97450]: pgmap v1322: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 118 B/s wr, 3 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:16 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-07T22:22:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:18 smithi055 ceph-mon[97033]: pgmap v1323: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:18 smithi059 ceph-mon[97450]: pgmap v1323: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:20 smithi055 ceph-mon[97033]: pgmap v1324: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:20 smithi059 ceph-mon[97450]: pgmap v1324: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:22 smithi055 ceph-mon[97033]: pgmap v1325: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 192 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:22 smithi059 ceph-mon[97450]: pgmap v1325: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 192 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:24 smithi055 ceph-mon[97033]: pgmap v1326: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:24 smithi059 ceph-mon[97450]: pgmap v1326: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:26 smithi059 ceph-mon[97450]: pgmap v1327: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:26.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:26 smithi055 ceph-mon[97033]: pgmap v1327: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:28 smithi059 ceph-mon[97450]: pgmap v1328: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:28 smithi055 ceph-mon[97033]: pgmap v1328: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:30 smithi059 ceph-mon[97450]: pgmap v1329: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:30 smithi055 ceph-mon[97033]: pgmap v1329: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:30.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:22:30.642+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:22:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:32 smithi059 ceph-mon[97450]: pgmap v1330: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:32 smithi055 ceph-mon[97033]: pgmap v1330: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:34 smithi059 ceph-mon[97450]: pgmap v1331: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:34 smithi055 ceph-mon[97033]: pgmap v1331: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:36 smithi059 ceph-mon[97450]: pgmap v1332: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:36 smithi055 ceph-mon[97033]: pgmap v1332: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:38 smithi059 ceph-mon[97450]: pgmap v1333: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:38 smithi055 ceph-mon[97033]: pgmap v1333: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:40 smithi059 ceph-mon[97450]: pgmap v1334: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:40 smithi055 ceph-mon[97033]: pgmap v1334: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:42 smithi059 ceph-mon[97450]: pgmap v1335: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:42 smithi055 ceph-mon[97033]: pgmap v1335: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:44.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:44 smithi059 ceph-mon[97450]: pgmap v1336: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:44 smithi055 ceph-mon[97033]: pgmap v1336: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:45.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:22:45.644+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:22:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:46 smithi059 ceph-mon[97450]: pgmap v1337: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:46 smithi055 ceph-mon[97033]: pgmap v1337: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:48 smithi059 ceph-mon[97450]: pgmap v1338: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:48 smithi055 ceph-mon[97033]: pgmap v1338: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:50 smithi059 ceph-mon[97450]: pgmap v1339: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:50 smithi055 ceph-mon[97033]: pgmap v1339: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:52 smithi059 ceph-mon[97450]: pgmap v1340: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:52 smithi055 ceph-mon[97033]: pgmap v1340: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:54 smithi059 ceph-mon[97450]: pgmap v1341: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:54 smithi055 ceph-mon[97033]: pgmap v1341: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:56 smithi055 ceph-mon[97033]: pgmap v1342: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:56 smithi059 ceph-mon[97450]: pgmap v1342: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:22:58 smithi055 ceph-mon[97033]: pgmap v1343: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:22:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:22:58 smithi059 ceph-mon[97450]: pgmap v1343: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:00 smithi055 ceph-mon[97033]: pgmap v1344: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:00 smithi059 ceph-mon[97450]: pgmap v1344: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:01.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:23:00.660+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:23:02.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:02 smithi055 ceph-mon[97033]: pgmap v1345: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:02 smithi059 ceph-mon[97450]: pgmap v1345: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:04 smithi055 ceph-mon[97033]: pgmap v1346: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:23:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:23:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:04 smithi059 ceph-mon[97450]: pgmap v1346: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:23:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:23:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:06 smithi055 ceph-mon[97033]: pgmap v1347: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:06 smithi059 ceph-mon[97450]: pgmap v1347: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:08 smithi055 ceph-mon[97033]: pgmap v1348: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:08 smithi059 ceph-mon[97450]: pgmap v1348: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:09 smithi055 ceph-mon[97033]: pgmap v1349: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:09 smithi055 ceph-mon[97033]: osdmap e156: 12 total, 11 up, 12 in 2022-09-07T22:23:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:09 smithi059 ceph-mon[97450]: pgmap v1349: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:09 smithi059 ceph-mon[97450]: osdmap e156: 12 total, 11 up, 12 in 2022-09-07T22:23:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:11 smithi055 ceph-mon[97033]: pgmap v1351: 129 pgs: 3 active+clean+snaptrim_wait, 16 active+undersized+degraded, 19 active+undersized, 91 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:11 smithi059 ceph-mon[97450]: pgmap v1351: 129 pgs: 3 active+clean+snaptrim_wait, 16 active+undersized+degraded, 19 active+undersized, 91 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:13 smithi055 ceph-mon[97033]: pgmap v1352: 129 pgs: 3 active+clean+snaptrim_wait, 16 active+undersized+degraded, 19 active+undersized, 91 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:13 smithi059 ceph-mon[97450]: pgmap v1352: 129 pgs: 3 active+clean+snaptrim_wait, 16 active+undersized+degraded, 19 active+undersized, 91 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:23:15.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:23:15.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:23:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:23:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:23:15.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:23:16.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:23:15.665+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:23:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:16 smithi055 ceph-mon[97033]: pgmap v1353: 129 pgs: 3 active+clean+snaptrim_wait, 16 active+undersized+degraded, 19 active+undersized, 91 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:23:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:23:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:16 smithi059 ceph-mon[97450]: pgmap v1353: 129 pgs: 3 active+clean+snaptrim_wait, 16 active+undersized+degraded, 19 active+undersized, 91 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:23:16.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:23:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:18 smithi055 ceph-mon[97033]: pgmap v1354: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:18.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:23:18.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:23:18.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:23:18.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:23:18.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:23:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:18 smithi059 ceph-mon[97450]: pgmap v1354: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:18.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:23:18.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:23:18.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:23:18.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:23:18.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:23:18.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:23:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:19 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T22:23:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:19 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T22:23:19.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:19 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T22:23:19.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:19 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T22:23:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:19 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T22:23:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:19 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T22:23:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:19 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T22:23:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:19 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T22:23:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:20 smithi055 ceph-mon[97033]: pgmap v1355: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:20 smithi059 ceph-mon[97450]: pgmap v1355: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:22 smithi055 ceph-mon[97033]: pgmap v1356: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 177 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:22 smithi059 ceph-mon[97450]: pgmap v1356: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 177 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:24 smithi055 ceph-mon[97033]: pgmap v1357: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:24 smithi059 ceph-mon[97450]: pgmap v1357: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:26 smithi055 ceph-mon[97033]: pgmap v1358: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:26 smithi059 ceph-mon[97450]: pgmap v1358: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:28 smithi055 ceph-mon[97033]: pgmap v1359: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:28 smithi059 ceph-mon[97450]: pgmap v1359: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:30 smithi055 ceph-mon[97033]: pgmap v1360: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:30 smithi059 ceph-mon[97450]: pgmap v1360: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:31.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:23:30.680+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:23:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:32 smithi055 ceph-mon[97033]: pgmap v1361: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:32 smithi059 ceph-mon[97450]: pgmap v1361: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:34 smithi055 ceph-mon[97033]: pgmap v1362: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:34 smithi059 ceph-mon[97450]: pgmap v1362: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:36 smithi055 ceph-mon[97033]: pgmap v1363: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:36 smithi059 ceph-mon[97450]: pgmap v1363: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:38 smithi055 ceph-mon[97033]: pgmap v1364: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:38 smithi059 ceph-mon[97450]: pgmap v1364: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:40 smithi055 ceph-mon[97033]: pgmap v1365: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:40 smithi059 ceph-mon[97450]: pgmap v1365: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:42 smithi055 ceph-mon[97033]: pgmap v1366: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:42 smithi059 ceph-mon[97450]: pgmap v1366: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:44 smithi055 ceph-mon[97033]: pgmap v1367: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:44 smithi059 ceph-mon[97450]: pgmap v1367: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:46.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:23:45.691+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:23:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:46 smithi055 ceph-mon[97033]: pgmap v1368: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:46 smithi059 ceph-mon[97450]: pgmap v1368: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:48 smithi055 ceph-mon[97033]: pgmap v1369: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:48 smithi059 ceph-mon[97450]: pgmap v1369: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:49 smithi055 ceph-mon[97033]: pgmap v1370: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:49 smithi059 ceph-mon[97450]: pgmap v1370: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:51 smithi055 ceph-mon[97033]: pgmap v1371: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:51 smithi059 ceph-mon[97450]: pgmap v1371: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:53 smithi055 ceph-mon[97033]: pgmap v1372: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:53 smithi059 ceph-mon[97450]: pgmap v1372: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:56.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:55 smithi055 ceph-mon[97033]: pgmap v1373: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:55 smithi059 ceph-mon[97450]: pgmap v1373: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:57 smithi055 ceph-mon[97033]: pgmap v1374: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:23:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:57 smithi059 ceph-mon[97450]: pgmap v1374: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:23:59 smithi055 ceph-mon[97033]: pgmap v1375: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:23:59 smithi059 ceph-mon[97450]: pgmap v1375: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:01.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:24:00.702+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:24:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:01 smithi055 ceph-mon[97033]: pgmap v1376: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:01 smithi059 ceph-mon[97450]: pgmap v1376: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:03 smithi059 ceph-mon[97450]: pgmap v1377: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:03 smithi055 ceph-mon[97033]: pgmap v1377: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:24:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:24:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:24:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:24:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:05 smithi059 ceph-mon[97450]: pgmap v1378: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:05 smithi055 ceph-mon[97033]: pgmap v1378: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:07 smithi059 ceph-mon[97450]: pgmap v1379: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:07 smithi055 ceph-mon[97033]: pgmap v1379: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:10 smithi059 ceph-mon[97450]: pgmap v1380: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:10 smithi059 ceph-mon[97450]: osdmap e157: 12 total, 11 up, 12 in 2022-09-07T22:24:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:10 smithi055 ceph-mon[97033]: pgmap v1380: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:10 smithi055 ceph-mon[97033]: osdmap e157: 12 total, 11 up, 12 in 2022-09-07T22:24:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:12 smithi059 ceph-mon[97450]: pgmap v1382: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:12 smithi055 ceph-mon[97033]: pgmap v1382: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:14 smithi059 ceph-mon[97450]: pgmap v1383: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:14 smithi055 ceph-mon[97033]: pgmap v1383: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:16.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:24:15.714+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:24:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:16 smithi059 ceph-mon[97450]: pgmap v1384: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:16 smithi055 ceph-mon[97033]: pgmap v1384: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:18 smithi059 ceph-mon[97450]: pgmap v1385: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:18.461 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:18 smithi055 ceph-mon[97033]: pgmap v1385: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:24:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:24:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:24:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:24:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:24:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:24:19.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:24:19.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:24:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:20 smithi055 ceph-mon[97033]: pgmap v1386: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:20 smithi059 ceph-mon[97450]: pgmap v1386: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:22 smithi055 ceph-mon[97033]: pgmap v1387: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 178 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:22 smithi059 ceph-mon[97450]: pgmap v1387: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 178 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:24 smithi055 ceph-mon[97033]: pgmap v1388: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:24 smithi059 ceph-mon[97450]: pgmap v1388: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:26 smithi055 ceph-mon[97033]: pgmap v1389: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:26 smithi059 ceph-mon[97450]: pgmap v1389: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:28 smithi055 ceph-mon[97033]: pgmap v1390: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:28 smithi059 ceph-mon[97450]: pgmap v1390: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:30 smithi055 ceph-mon[97033]: pgmap v1391: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:30 smithi059 ceph-mon[97450]: pgmap v1391: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:31.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:24:30.718+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:24:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:32 smithi055 ceph-mon[97033]: pgmap v1392: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:32 smithi059 ceph-mon[97450]: pgmap v1392: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:34 smithi055 ceph-mon[97033]: pgmap v1393: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:34 smithi059 ceph-mon[97450]: pgmap v1393: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:36 smithi055 ceph-mon[97033]: pgmap v1394: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:36 smithi059 ceph-mon[97450]: pgmap v1394: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:38 smithi055 ceph-mon[97033]: pgmap v1395: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:38 smithi059 ceph-mon[97450]: pgmap v1395: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:40 smithi055 ceph-mon[97033]: pgmap v1396: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:40 smithi059 ceph-mon[97450]: pgmap v1396: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:42 smithi055 ceph-mon[97033]: pgmap v1397: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:42 smithi059 ceph-mon[97450]: pgmap v1397: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:44 smithi055 ceph-mon[97033]: pgmap v1398: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:44 smithi059 ceph-mon[97450]: pgmap v1398: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:46.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:24:45.720+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:24:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:46 smithi055 ceph-mon[97033]: pgmap v1399: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:46 smithi059 ceph-mon[97450]: pgmap v1399: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:48 smithi055 ceph-mon[97033]: pgmap v1400: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:48.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:48 smithi059 ceph-mon[97450]: pgmap v1400: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:50 smithi055 ceph-mon[97033]: pgmap v1401: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:50 smithi059 ceph-mon[97450]: pgmap v1401: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:52 smithi055 ceph-mon[97033]: pgmap v1402: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:52 smithi059 ceph-mon[97450]: pgmap v1402: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:54 smithi055 ceph-mon[97033]: pgmap v1403: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:54 smithi059 ceph-mon[97450]: pgmap v1403: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:56 smithi055 ceph-mon[97033]: pgmap v1404: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:56 smithi059 ceph-mon[97450]: pgmap v1404: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:24:58 smithi055 ceph-mon[97033]: pgmap v1405: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:24:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:24:58 smithi059 ceph-mon[97450]: pgmap v1405: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:00 smithi055 ceph-mon[97033]: pgmap v1406: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:00 smithi059 ceph-mon[97450]: pgmap v1406: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:01.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:25:00.735+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:25:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:02 smithi055 ceph-mon[97033]: pgmap v1407: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:02 smithi059 ceph-mon[97450]: pgmap v1407: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:04 smithi055 ceph-mon[97033]: pgmap v1408: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:04 smithi059 ceph-mon[97450]: pgmap v1408: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:25:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:25:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:25:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:25:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:06 smithi055 ceph-mon[97033]: pgmap v1409: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:06 smithi059 ceph-mon[97450]: pgmap v1409: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:08 smithi055 ceph-mon[97033]: pgmap v1410: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:08 smithi059 ceph-mon[97450]: pgmap v1410: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:10 smithi055 ceph-mon[97033]: pgmap v1411: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:10 smithi055 ceph-mon[97033]: osdmap e158: 12 total, 11 up, 12 in 2022-09-07T22:25:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:10 smithi059 ceph-mon[97450]: pgmap v1411: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:10 smithi059 ceph-mon[97450]: osdmap e158: 12 total, 11 up, 12 in 2022-09-07T22:25:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:12 smithi055 ceph-mon[97033]: pgmap v1413: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:12 smithi059 ceph-mon[97450]: pgmap v1413: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:14 smithi055 ceph-mon[97033]: pgmap v1414: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:14 smithi059 ceph-mon[97450]: pgmap v1414: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:16.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:25:15.744+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:25:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:16 smithi055 ceph-mon[97033]: pgmap v1415: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:16 smithi059 ceph-mon[97450]: pgmap v1415: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:18 smithi055 ceph-mon[97033]: pgmap v1416: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:18 smithi059 ceph-mon[97450]: pgmap v1416: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:25:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:25:19.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:25:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:25:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:25:19.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:25:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:20 smithi055 ceph-mon[97033]: pgmap v1417: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:25:20.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:25:20.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:25:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:20 smithi059 ceph-mon[97450]: pgmap v1417: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:25:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:25:20.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:25:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:22 smithi055 ceph-mon[97033]: pgmap v1418: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 179 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:22 smithi059 ceph-mon[97450]: pgmap v1418: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 179 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:24 smithi055 ceph-mon[97033]: pgmap v1419: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:24 smithi059 ceph-mon[97450]: pgmap v1419: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:26 smithi055 ceph-mon[97033]: pgmap v1420: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:26 smithi059 ceph-mon[97450]: pgmap v1420: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:28 smithi055 ceph-mon[97033]: pgmap v1421: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:28 smithi059 ceph-mon[97450]: pgmap v1421: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:30 smithi055 ceph-mon[97033]: pgmap v1422: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:30 smithi059 ceph-mon[97450]: pgmap v1422: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:31.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:25:30.752+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:25:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:32 smithi055 ceph-mon[97033]: pgmap v1423: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:32 smithi059 ceph-mon[97450]: pgmap v1423: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:34 smithi055 ceph-mon[97033]: pgmap v1424: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:34 smithi059 ceph-mon[97450]: pgmap v1424: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:36 smithi055 ceph-mon[97033]: pgmap v1425: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:36 smithi059 ceph-mon[97450]: pgmap v1425: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:38 smithi055 ceph-mon[97033]: pgmap v1426: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:38 smithi059 ceph-mon[97450]: pgmap v1426: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:40 smithi055 ceph-mon[97033]: pgmap v1427: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:40 smithi059 ceph-mon[97450]: pgmap v1427: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:42 smithi055 ceph-mon[97033]: pgmap v1428: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:42 smithi059 ceph-mon[97450]: pgmap v1428: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:44 smithi055 ceph-mon[97033]: pgmap v1429: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:44 smithi059 ceph-mon[97450]: pgmap v1429: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:46.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:25:45.762+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:25:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:46 smithi055 ceph-mon[97033]: pgmap v1430: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:46 smithi059 ceph-mon[97450]: pgmap v1430: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:48 smithi055 ceph-mon[97033]: pgmap v1431: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:48 smithi059 ceph-mon[97450]: pgmap v1431: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:50 smithi055 ceph-mon[97033]: pgmap v1432: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:50 smithi059 ceph-mon[97450]: pgmap v1432: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:52 smithi055 ceph-mon[97033]: pgmap v1433: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:52 smithi059 ceph-mon[97450]: pgmap v1433: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:54 smithi055 ceph-mon[97033]: pgmap v1434: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:54 smithi059 ceph-mon[97450]: pgmap v1434: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:56 smithi055 ceph-mon[97033]: pgmap v1435: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:56 smithi059 ceph-mon[97450]: pgmap v1435: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:25:58 smithi055 ceph-mon[97033]: pgmap v1436: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:25:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:25:58 smithi059 ceph-mon[97450]: pgmap v1436: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:00 smithi055 ceph-mon[97033]: pgmap v1437: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:00 smithi059 ceph-mon[97450]: pgmap v1437: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:01.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:26:00.772+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:26:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:02 smithi055 ceph-mon[97033]: pgmap v1438: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:02 smithi059 ceph-mon[97450]: pgmap v1438: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:04 smithi059 ceph-mon[97450]: pgmap v1439: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:04 smithi055 ceph-mon[97033]: pgmap v1439: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:26:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:26:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:26:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:26:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:06 smithi059 ceph-mon[97450]: pgmap v1440: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:06 smithi055 ceph-mon[97033]: pgmap v1440: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:08 smithi059 ceph-mon[97450]: pgmap v1441: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:08 smithi055 ceph-mon[97033]: pgmap v1441: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:10 smithi059 ceph-mon[97450]: pgmap v1442: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:10 smithi059 ceph-mon[97450]: osdmap e159: 12 total, 11 up, 12 in 2022-09-07T22:26:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:10 smithi055 ceph-mon[97033]: pgmap v1442: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:10 smithi055 ceph-mon[97033]: osdmap e159: 12 total, 11 up, 12 in 2022-09-07T22:26:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:12 smithi059 ceph-mon[97450]: pgmap v1444: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:12 smithi055 ceph-mon[97033]: pgmap v1444: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:14 smithi059 ceph-mon[97450]: pgmap v1445: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:14 smithi055 ceph-mon[97033]: pgmap v1445: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:16.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:26:15.784+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:26:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:16 smithi059 ceph-mon[97450]: pgmap v1446: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:16 smithi055 ceph-mon[97033]: pgmap v1446: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:18 smithi059 ceph-mon[97450]: pgmap v1447: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:18 smithi055 ceph-mon[97033]: pgmap v1447: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:20 smithi059 ceph-mon[97450]: pgmap v1448: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:20 smithi055 ceph-mon[97033]: pgmap v1448: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:26:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:26:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:26:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:26:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:26:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:26:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:26:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:26:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:22 smithi055 ceph-mon[97033]: pgmap v1449: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 177 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:22 smithi059 ceph-mon[97450]: pgmap v1449: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 177 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:24 smithi055 ceph-mon[97033]: pgmap v1450: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:24 smithi059 ceph-mon[97450]: pgmap v1450: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:26 smithi055 ceph-mon[97033]: pgmap v1451: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:26 smithi059 ceph-mon[97450]: pgmap v1451: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:28 smithi055 ceph-mon[97033]: pgmap v1452: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:28 smithi059 ceph-mon[97450]: pgmap v1452: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:30 smithi055 ceph-mon[97033]: pgmap v1453: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:30 smithi059 ceph-mon[97450]: pgmap v1453: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:31.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:26:30.798+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:26:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:32 smithi055 ceph-mon[97033]: pgmap v1454: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:32 smithi059 ceph-mon[97450]: pgmap v1454: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:34 smithi055 ceph-mon[97033]: pgmap v1455: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:34 smithi059 ceph-mon[97450]: pgmap v1455: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:36 smithi055 ceph-mon[97033]: pgmap v1456: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:36 smithi059 ceph-mon[97450]: pgmap v1456: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:38 smithi055 ceph-mon[97033]: pgmap v1457: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:38 smithi059 ceph-mon[97450]: pgmap v1457: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:40 smithi055 ceph-mon[97033]: pgmap v1458: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:40 smithi059 ceph-mon[97450]: pgmap v1458: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:42 smithi055 ceph-mon[97033]: pgmap v1459: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:42 smithi059 ceph-mon[97450]: pgmap v1459: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:44 smithi055 ceph-mon[97033]: pgmap v1460: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:44 smithi059 ceph-mon[97450]: pgmap v1460: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:46.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:26:45.813+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:26:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:46 smithi055 ceph-mon[97033]: pgmap v1461: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:46 smithi059 ceph-mon[97450]: pgmap v1461: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:48 smithi055 ceph-mon[97033]: pgmap v1462: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:48 smithi059 ceph-mon[97450]: pgmap v1462: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:50 smithi055 ceph-mon[97033]: pgmap v1463: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:50 smithi059 ceph-mon[97450]: pgmap v1463: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:52 smithi055 ceph-mon[97033]: pgmap v1464: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:52 smithi059 ceph-mon[97450]: pgmap v1464: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:54 smithi055 ceph-mon[97033]: pgmap v1465: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:54 smithi059 ceph-mon[97450]: pgmap v1465: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:56 smithi055 ceph-mon[97033]: pgmap v1466: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:56 smithi059 ceph-mon[97450]: pgmap v1466: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:26:58 smithi055 ceph-mon[97033]: pgmap v1467: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:26:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:26:58 smithi059 ceph-mon[97450]: pgmap v1467: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:00 smithi055 ceph-mon[97033]: pgmap v1468: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:00 smithi059 ceph-mon[97450]: pgmap v1468: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:01.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:27:00.827+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:27:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:02 smithi055 ceph-mon[97033]: pgmap v1469: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:02 smithi059 ceph-mon[97450]: pgmap v1469: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:04 smithi055 ceph-mon[97033]: pgmap v1470: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:04 smithi059 ceph-mon[97450]: pgmap v1470: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:27:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:27:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:27:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:27:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:06 smithi055 ceph-mon[97033]: pgmap v1471: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:06 smithi059 ceph-mon[97450]: pgmap v1471: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:08 smithi055 ceph-mon[97033]: pgmap v1472: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:08 smithi059 ceph-mon[97450]: pgmap v1472: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:10 smithi055 ceph-mon[97033]: pgmap v1473: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:10 smithi055 ceph-mon[97033]: osdmap e160: 12 total, 11 up, 12 in 2022-09-07T22:27:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:10 smithi059 ceph-mon[97450]: pgmap v1473: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:10 smithi059 ceph-mon[97450]: osdmap e160: 12 total, 11 up, 12 in 2022-09-07T22:27:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:12 smithi055 ceph-mon[97033]: pgmap v1475: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:12 smithi059 ceph-mon[97450]: pgmap v1475: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:14 smithi055 ceph-mon[97033]: pgmap v1476: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:14 smithi059 ceph-mon[97450]: pgmap v1476: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:16.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:27:15.829+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:27:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:16 smithi055 ceph-mon[97033]: pgmap v1477: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:16 smithi059 ceph-mon[97450]: pgmap v1477: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:18 smithi055 ceph-mon[97033]: pgmap v1478: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:18 smithi059 ceph-mon[97450]: pgmap v1478: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:20 smithi055 ceph-mon[97033]: pgmap v1479: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:20 smithi059 ceph-mon[97450]: pgmap v1479: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:27:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:27:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:27:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:27:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:27:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:27:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:22 smithi059 ceph-mon[97450]: pgmap v1480: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 178 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:27:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:22 smithi055 ceph-mon[97033]: pgmap v1480: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 178 B/s wr, 8 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:27:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:23 smithi059 ceph-mon[97450]: Marking osd.0 out (has been down for 600 seconds) 2022-09-07T22:27:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:23 smithi059 ceph-mon[97450]: Health check cleared: OSD_DOWN (was: 1 osds down) 2022-09-07T22:27:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:23 smithi059 ceph-mon[97450]: osdmap e161: 12 total, 11 up, 11 in 2022-09-07T22:27:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:23 smithi055 ceph-mon[97033]: Marking osd.0 out (has been down for 600 seconds) 2022-09-07T22:27:23.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:23 smithi055 ceph-mon[97033]: Health check cleared: OSD_DOWN (was: 1 osds down) 2022-09-07T22:27:23.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:23 smithi055 ceph-mon[97033]: osdmap e161: 12 total, 11 up, 11 in 2022-09-07T22:27:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:24 smithi059 ceph-mon[97450]: pgmap v1481: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:24 smithi059 ceph-mon[97450]: osdmap e162: 12 total, 11 up, 11 in 2022-09-07T22:27:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:24 smithi055 ceph-mon[97033]: pgmap v1481: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:24 smithi055 ceph-mon[97033]: osdmap e162: 12 total, 11 up, 11 in 2022-09-07T22:27:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:26 smithi059 ceph-mon[97450]: pgmap v1484: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:26 smithi059 ceph-mon[97450]: osdmap e163: 12 total, 11 up, 11 in 2022-09-07T22:27:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:26 smithi055 ceph-mon[97033]: pgmap v1484: 129 pgs: 16 active+undersized+degraded, 19 active+undersized, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 987/13572 objects degraded (7.272%) 2022-09-07T22:27:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:26 smithi055 ceph-mon[97033]: osdmap e163: 12 total, 11 up, 11 in 2022-09-07T22:27:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:27 smithi059 ceph-mon[97450]: osdmap e164: 12 total, 11 up, 11 in 2022-09-07T22:27:27.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:27 smithi055 ceph-mon[97033]: osdmap e164: 12 total, 11 up, 11 in 2022-09-07T22:27:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:28 smithi059 ceph-mon[97450]: pgmap v1487: 129 pgs: 1 active+recovering+degraded+wait, 11 remapped+peering, 23 active+clean+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3/13572 objects degraded (0.022%); 316 B/s, 6 keys/s, 3 objects/s recovering 2022-09-07T22:27:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:28 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 3/13572 objects degraded (0.022%), 1 pg degraded (PG_DEGRADED) 2022-09-07T22:27:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:28 smithi059 ceph-mon[97450]: osdmap e165: 12 total, 11 up, 11 in 2022-09-07T22:27:28.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:28 smithi059 ceph-mon[97450]: osdmap e166: 12 total, 11 up, 11 in 2022-09-07T22:27:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:28 smithi055 ceph-mon[97033]: pgmap v1487: 129 pgs: 1 active+recovering+degraded+wait, 11 remapped+peering, 23 active+clean+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3/13572 objects degraded (0.022%); 316 B/s, 6 keys/s, 3 objects/s recovering 2022-09-07T22:27:28.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:28 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 3/13572 objects degraded (0.022%), 1 pg degraded (PG_DEGRADED) 2022-09-07T22:27:28.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:28 smithi055 ceph-mon[97033]: osdmap e165: 12 total, 11 up, 11 in 2022-09-07T22:27:28.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:28 smithi055 ceph-mon[97033]: osdmap e166: 12 total, 11 up, 11 in 2022-09-07T22:27:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:30 smithi055 ceph-mon[97033]: pgmap v1490: 129 pgs: 1 active+recovering+degraded+wait, 11 remapped+peering, 23 active+clean+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3/13572 objects degraded (0.022%); 316 B/s, 6 keys/s, 3 objects/s recovering 2022-09-07T22:27:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:30 smithi059 ceph-mon[97450]: pgmap v1490: 129 pgs: 1 active+recovering+degraded+wait, 11 remapped+peering, 23 active+clean+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3/13572 objects degraded (0.022%); 316 B/s, 6 keys/s, 3 objects/s recovering 2022-09-07T22:27:31.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:27:30.840+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:27:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:31 smithi055 ceph-mon[97033]: osdmap e167: 12 total, 11 up, 11 in 2022-09-07T22:27:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:31 smithi059 ceph-mon[97450]: osdmap e167: 12 total, 11 up, 11 in 2022-09-07T22:27:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:32 smithi055 ceph-mon[97033]: pgmap v1492: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+wait, 25 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.2 KiB/s rd, 2 op/s; 2759/13572 objects degraded (20.329%); 78 MiB/s, 7 keys/s, 83 objects/s recovering 2022-09-07T22:27:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:32 smithi055 ceph-mon[97033]: osdmap e168: 12 total, 11 up, 11 in 2022-09-07T22:27:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:32 smithi059 ceph-mon[97450]: pgmap v1492: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+wait, 25 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.2 KiB/s rd, 2 op/s; 2759/13572 objects degraded (20.329%); 78 MiB/s, 7 keys/s, 83 objects/s recovering 2022-09-07T22:27:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:32 smithi059 ceph-mon[97450]: osdmap e168: 12 total, 11 up, 11 in 2022-09-07T22:27:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:33 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 2759/13572 objects degraded (20.329%), 10 pgs degraded (PG_DEGRADED) 2022-09-07T22:27:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:33 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 2759/13572 objects degraded (20.329%), 10 pgs degraded (PG_DEGRADED) 2022-09-07T22:27:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:34 smithi055 ceph-mon[97033]: pgmap v1494: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+wait, 25 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 964 B/s rd, 1 op/s; 2759/13572 objects degraded (20.329%); 64 MiB/s, 6 keys/s, 67 objects/s recovering 2022-09-07T22:27:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:34 smithi059 ceph-mon[97450]: pgmap v1494: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+wait, 25 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 964 B/s rd, 1 op/s; 2759/13572 objects degraded (20.329%); 64 MiB/s, 6 keys/s, 67 objects/s recovering 2022-09-07T22:27:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:36 smithi055 ceph-mon[97033]: pgmap v1495: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+wait, 25 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 780 B/s rd, 1 op/s; 2759/13572 objects degraded (20.329%); 51 MiB/s, 5 keys/s, 54 objects/s recovering 2022-09-07T22:27:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:36 smithi059 ceph-mon[97450]: pgmap v1495: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+wait, 25 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 780 B/s rd, 1 op/s; 2759/13572 objects degraded (20.329%); 51 MiB/s, 5 keys/s, 54 objects/s recovering 2022-09-07T22:27:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:38 smithi055 ceph-mon[97033]: pgmap v1496: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 26 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.2 KiB/s rd, 2 op/s; 2752/13572 objects degraded (20.277%); 46 MiB/s, 4 keys/s, 52 objects/s recovering 2022-09-07T22:27:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:38 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 2752/13572 objects degraded (20.277%), 9 pgs degraded (PG_DEGRADED) 2022-09-07T22:27:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:38 smithi059 ceph-mon[97450]: pgmap v1496: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 26 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.2 KiB/s rd, 2 op/s; 2752/13572 objects degraded (20.277%); 46 MiB/s, 4 keys/s, 52 objects/s recovering 2022-09-07T22:27:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:38 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 2752/13572 objects degraded (20.277%), 9 pgs degraded (PG_DEGRADED) 2022-09-07T22:27:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:40 smithi055 ceph-mon[97033]: pgmap v1497: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 26 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 616 B/s rd, 1 op/s; 2752/13572 objects degraded (20.277%); 3.4 MiB/s, 7 objects/s recovering 2022-09-07T22:27:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:40 smithi059 ceph-mon[97450]: pgmap v1497: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 26 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 616 B/s rd, 1 op/s; 2752/13572 objects degraded (20.277%); 3.4 MiB/s, 7 objects/s recovering 2022-09-07T22:27:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:42 smithi055 ceph-mon[97033]: pgmap v1498: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 26 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1023 B/s rd, 1 op/s; 2742/13572 objects degraded (20.203%); 6.8 MiB/s, 12 objects/s recovering 2022-09-07T22:27:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:42 smithi059 ceph-mon[97450]: pgmap v1498: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 26 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1023 B/s rd, 1 op/s; 2742/13572 objects degraded (20.203%); 6.8 MiB/s, 12 objects/s recovering 2022-09-07T22:27:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:43 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 2742/13572 objects degraded (20.203%), 9 pgs degraded (PG_DEGRADED) 2022-09-07T22:27:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:43 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 2742/13572 objects degraded (20.203%), 9 pgs degraded (PG_DEGRADED) 2022-09-07T22:27:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:44 smithi055 ceph-mon[97033]: pgmap v1499: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 26 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 907 B/s rd, 1 op/s; 2742/13572 objects degraded (20.203%); 6.0 MiB/s, 10 objects/s recovering 2022-09-07T22:27:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:44 smithi059 ceph-mon[97450]: pgmap v1499: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 26 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 907 B/s rd, 1 op/s; 2742/13572 objects degraded (20.203%); 6.0 MiB/s, 10 objects/s recovering 2022-09-07T22:27:46.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:27:45.853+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:27:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:46 smithi055 ceph-mon[97033]: pgmap v1500: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 26 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 853 B/s rd, 1 op/s; 2742/13572 objects degraded (20.203%); 5.7 MiB/s, 10 objects/s recovering 2022-09-07T22:27:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:46 smithi059 ceph-mon[97450]: pgmap v1500: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 26 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 853 B/s rd, 1 op/s; 2742/13572 objects degraded (20.203%); 5.7 MiB/s, 10 objects/s recovering 2022-09-07T22:27:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:48 smithi055 ceph-mon[97033]: pgmap v1501: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 26 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.2 KiB/s rd, 2 op/s; 2714/13572 objects degraded (19.997%); 15 MiB/s, 14 objects/s recovering 2022-09-07T22:27:48.741 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:48 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 2714/13572 objects degraded (19.997%), 9 pgs degraded (PG_DEGRADED) 2022-09-07T22:27:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:48 smithi059 ceph-mon[97450]: pgmap v1501: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 26 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.2 KiB/s rd, 2 op/s; 2714/13572 objects degraded (19.997%); 15 MiB/s, 14 objects/s recovering 2022-09-07T22:27:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:48 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 2714/13572 objects degraded (19.997%), 9 pgs degraded (PG_DEGRADED) 2022-09-07T22:27:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:50 smithi055 ceph-mon[97033]: pgmap v1502: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 26 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 852 B/s rd, 1 op/s; 2714/13572 objects degraded (19.997%); 13 MiB/s, 9 objects/s recovering 2022-09-07T22:27:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:50 smithi059 ceph-mon[97450]: pgmap v1502: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 26 active+clean+wait, 6 active+recovery_wait+undersized+degraded+remapped+wait, 94 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 852 B/s rd, 1 op/s; 2714/13572 objects degraded (19.997%); 13 MiB/s, 9 objects/s recovering 2022-09-07T22:27:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:52 smithi055 ceph-mon[97033]: pgmap v1503: 129 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 120 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.5 KiB/s rd, 3 op/s; 2687/13572 objects degraded (19.798%); 22 MiB/s, 14 objects/s recovering 2022-09-07T22:27:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:52 smithi059 ceph-mon[97450]: pgmap v1503: 129 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 120 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.5 KiB/s rd, 3 op/s; 2687/13572 objects degraded (19.798%); 22 MiB/s, 14 objects/s recovering 2022-09-07T22:27:53.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:53 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 2687/13572 objects degraded (19.798%), 9 pgs degraded (PG_DEGRADED) 2022-09-07T22:27:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:53 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 2687/13572 objects degraded (19.798%), 9 pgs degraded (PG_DEGRADED) 2022-09-07T22:27:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:54 smithi055 ceph-mon[97033]: pgmap v1504: 129 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 120 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.1 KiB/s rd, 2 op/s; 2687/13572 objects degraded (19.798%); 18 MiB/s, 9 objects/s recovering 2022-09-07T22:27:54.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:54 smithi059 ceph-mon[97450]: pgmap v1504: 129 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 120 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.1 KiB/s rd, 2 op/s; 2687/13572 objects degraded (19.798%); 18 MiB/s, 9 objects/s recovering 2022-09-07T22:27:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:55 smithi055 ceph-mon[97033]: osdmap e169: 12 total, 11 up, 11 in 2022-09-07T22:27:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:55 smithi059 ceph-mon[97450]: osdmap e169: 12 total, 11 up, 11 in 2022-09-07T22:27:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:56 smithi055 ceph-mon[97033]: pgmap v1506: 129 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 120 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.3 KiB/s rd, 2 op/s; 2687/13572 objects degraded (19.798%); 22 MiB/s, 11 objects/s recovering 2022-09-07T22:27:56.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:56 smithi055 ceph-mon[97033]: osdmap e170: 12 total, 11 up, 11 in 2022-09-07T22:27:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:56 smithi059 ceph-mon[97450]: pgmap v1506: 129 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 120 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.3 KiB/s rd, 2 op/s; 2687/13572 objects degraded (19.798%); 22 MiB/s, 11 objects/s recovering 2022-09-07T22:27:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:56 smithi059 ceph-mon[97450]: osdmap e170: 12 total, 11 up, 11 in 2022-09-07T22:27:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:58 smithi055 ceph-mon[97033]: pgmap v1508: 129 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 120 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s; 2427/13572 objects degraded (17.882%); 28 MiB/s, 21 objects/s recovering 2022-09-07T22:27:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:27:58 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 2427/13572 objects degraded (17.882%), 8 pgs degraded (PG_DEGRADED) 2022-09-07T22:27:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:58 smithi059 ceph-mon[97450]: pgmap v1508: 129 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 120 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s; 2427/13572 objects degraded (17.882%); 28 MiB/s, 21 objects/s recovering 2022-09-07T22:27:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:27:58 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 2427/13572 objects degraded (17.882%), 8 pgs degraded (PG_DEGRADED) 2022-09-07T22:28:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:00 smithi055 ceph-mon[97033]: pgmap v1509: 129 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 120 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 2427/13572 objects degraded (17.882%); 14 MiB/s, 14 objects/s recovering 2022-09-07T22:28:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:00 smithi055 ceph-mon[97033]: osdmap e171: 12 total, 11 up, 11 in 2022-09-07T22:28:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:00 smithi059 ceph-mon[97450]: pgmap v1509: 129 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 120 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 2427/13572 objects degraded (17.882%); 14 MiB/s, 14 objects/s recovering 2022-09-07T22:28:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:00 smithi059 ceph-mon[97450]: osdmap e171: 12 total, 11 up, 11 in 2022-09-07T22:28:01.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:28:00.868+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:28:01.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:01 smithi055 ceph-mon[97033]: osdmap e172: 12 total, 11 up, 11 in 2022-09-07T22:28:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:01 smithi059 ceph-mon[97450]: osdmap e172: 12 total, 11 up, 11 in 2022-09-07T22:28:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:02 smithi055 ceph-mon[97033]: pgmap v1512: 129 pgs: 1 peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 121 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 5.0 KiB/s rd, 9 op/s; 2067/13572 objects degraded (15.230%); 79 MiB/s, 65 objects/s recovering 2022-09-07T22:28:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:02 smithi059 ceph-mon[97450]: pgmap v1512: 129 pgs: 1 peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 121 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 5.0 KiB/s rd, 9 op/s; 2067/13572 objects degraded (15.230%); 79 MiB/s, 65 objects/s recovering 2022-09-07T22:28:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:03 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 2067/13572 objects degraded (15.230%), 7 pgs degraded (PG_DEGRADED) 2022-09-07T22:28:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:03 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 2067/13572 objects degraded (15.230%), 7 pgs degraded (PG_DEGRADED) 2022-09-07T22:28:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:04 smithi055 ceph-mon[97033]: pgmap v1513: 129 pgs: 1 peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 121 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.1 KiB/s rd, 4 op/s; 2067/13572 objects degraded (15.230%); 49 MiB/s, 37 objects/s recovering 2022-09-07T22:28:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:04 smithi059 ceph-mon[97450]: pgmap v1513: 129 pgs: 1 peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 121 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.1 KiB/s rd, 4 op/s; 2067/13572 objects degraded (15.230%); 49 MiB/s, 37 objects/s recovering 2022-09-07T22:28:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:28:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:28:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:28:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:28:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:06 smithi055 ceph-mon[97033]: pgmap v1514: 129 pgs: 1 peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 121 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 2067/13572 objects degraded (15.230%); 45 MiB/s, 34 objects/s recovering 2022-09-07T22:28:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:06 smithi059 ceph-mon[97450]: pgmap v1514: 129 pgs: 1 peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 121 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 2067/13572 objects degraded (15.230%); 45 MiB/s, 34 objects/s recovering 2022-09-07T22:28:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:08 smithi055 ceph-mon[97033]: pgmap v1515: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 66 KiB/s wr, 29 op/s; 2062/13572 objects degraded (15.193%); 47 MiB/s, 41 objects/s recovering 2022-09-07T22:28:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:08 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 2062/13572 objects degraded (15.193%), 7 pgs degraded (PG_DEGRADED) 2022-09-07T22:28:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:08 smithi059 ceph-mon[97450]: pgmap v1515: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 66 KiB/s wr, 29 op/s; 2062/13572 objects degraded (15.193%); 47 MiB/s, 41 objects/s recovering 2022-09-07T22:28:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:08 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 2062/13572 objects degraded (15.193%), 7 pgs degraded (PG_DEGRADED) 2022-09-07T22:28:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:10 smithi055 ceph-mon[97033]: pgmap v1516: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 56 KiB/s wr, 24 op/s; 2062/13572 objects degraded (15.193%); 39 MiB/s, 34 objects/s recovering 2022-09-07T22:28:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:10 smithi055 ceph-mon[97033]: osdmap e173: 12 total, 11 up, 11 in 2022-09-07T22:28:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:10 smithi059 ceph-mon[97450]: pgmap v1516: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 56 KiB/s wr, 24 op/s; 2062/13572 objects degraded (15.193%); 39 MiB/s, 34 objects/s recovering 2022-09-07T22:28:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:10 smithi059 ceph-mon[97450]: osdmap e173: 12 total, 11 up, 11 in 2022-09-07T22:28:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:12 smithi055 ceph-mon[97033]: pgmap v1518: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 2061/13572 objects degraded (15.186%); 2.4 MiB/s, 11 objects/s recovering 2022-09-07T22:28:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:12 smithi059 ceph-mon[97450]: pgmap v1518: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 2061/13572 objects degraded (15.186%); 2.4 MiB/s, 11 objects/s recovering 2022-09-07T22:28:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:13 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 2061/13572 objects degraded (15.186%), 7 pgs degraded (PG_DEGRADED) 2022-09-07T22:28:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:13 smithi055 ceph-mon[97033]: osdmap e174: 12 total, 11 up, 11 in 2022-09-07T22:28:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:13 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 2061/13572 objects degraded (15.186%), 7 pgs degraded (PG_DEGRADED) 2022-09-07T22:28:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:13 smithi059 ceph-mon[97450]: osdmap e174: 12 total, 11 up, 11 in 2022-09-07T22:28:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:14 smithi055 ceph-mon[97033]: pgmap v1519: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 2061/13572 objects degraded (15.186%); 2.4 MiB/s, 11 objects/s recovering 2022-09-07T22:28:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:14 smithi059 ceph-mon[97450]: pgmap v1519: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 2061/13572 objects degraded (15.186%); 2.4 MiB/s, 11 objects/s recovering 2022-09-07T22:28:16.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:28:15.879+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:28:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:16 smithi055 ceph-mon[97033]: pgmap v1521: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s; 2061/13572 objects degraded (15.186%); 512 KiB/s, 7 objects/s recovering 2022-09-07T22:28:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:16 smithi059 ceph-mon[97450]: pgmap v1521: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s; 2061/13572 objects degraded (15.186%); 512 KiB/s, 7 objects/s recovering 2022-09-07T22:28:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:18 smithi055 ceph-mon[97033]: pgmap v1522: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s; 2053/13572 objects degraded (15.127%); 4.5 MiB/s, 14 objects/s recovering 2022-09-07T22:28:18.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:18 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 2053/13572 objects degraded (15.127%), 7 pgs degraded (PG_DEGRADED) 2022-09-07T22:28:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:18 smithi059 ceph-mon[97450]: pgmap v1522: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s; 2053/13572 objects degraded (15.127%); 4.5 MiB/s, 14 objects/s recovering 2022-09-07T22:28:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:18 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 2053/13572 objects degraded (15.127%), 7 pgs degraded (PG_DEGRADED) 2022-09-07T22:28:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:20 smithi055 ceph-mon[97033]: pgmap v1523: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s; 2053/13572 objects degraded (15.127%); 3.8 MiB/s, 12 objects/s recovering 2022-09-07T22:28:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:20 smithi059 ceph-mon[97450]: pgmap v1523: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s; 2053/13572 objects degraded (15.127%); 3.8 MiB/s, 12 objects/s recovering 2022-09-07T22:28:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:22 smithi055 ceph-mon[97033]: pgmap v1524: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s; 2024/13572 objects degraded (14.913%); 15 MiB/s, 11 objects/s recovering 2022-09-07T22:28:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:28:22.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:28:22.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:28:22.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:28:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:22 smithi059 ceph-mon[97450]: pgmap v1524: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s; 2024/13572 objects degraded (14.913%); 15 MiB/s, 11 objects/s recovering 2022-09-07T22:28:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:28:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:28:22.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:28:22.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:28:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:23 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 2024/13572 objects degraded (14.913%), 7 pgs degraded (PG_DEGRADED) 2022-09-07T22:28:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:23 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 2024/13572 objects degraded (14.913%), 7 pgs degraded (PG_DEGRADED) 2022-09-07T22:28:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:24 smithi055 ceph-mon[97033]: pgmap v1525: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s; 2024/13572 objects degraded (14.913%); 15 MiB/s, 11 objects/s recovering 2022-09-07T22:28:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:24 smithi059 ceph-mon[97450]: pgmap v1525: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s; 2024/13572 objects degraded (14.913%); 15 MiB/s, 11 objects/s recovering 2022-09-07T22:28:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:26 smithi055 ceph-mon[97033]: pgmap v1526: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 88 B/s wr, 5 op/s; 2024/13572 objects degraded (14.913%); 13 MiB/s, 10 objects/s recovering 2022-09-07T22:28:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:26 smithi059 ceph-mon[97450]: pgmap v1526: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 88 B/s wr, 5 op/s; 2024/13572 objects degraded (14.913%); 13 MiB/s, 10 objects/s recovering 2022-09-07T22:28:28.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:28 smithi055 ceph-mon[97033]: pgmap v1527: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 1996/13572 objects degraded (14.707%); 22 MiB/s, 14 objects/s recovering 2022-09-07T22:28:28.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:28 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 1996/13572 objects degraded (14.707%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2022-09-07T22:28:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:28 smithi059 ceph-mon[97450]: pgmap v1527: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 1996/13572 objects degraded (14.707%); 22 MiB/s, 14 objects/s recovering 2022-09-07T22:28:28.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:28 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 1996/13572 objects degraded (14.707%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2022-09-07T22:28:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:30 smithi055 ceph-mon[97033]: pgmap v1528: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 1996/13572 objects degraded (14.707%); 19 MiB/s, 9 objects/s recovering 2022-09-07T22:28:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:30 smithi059 ceph-mon[97450]: pgmap v1528: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 1996/13572 objects degraded (14.707%); 19 MiB/s, 9 objects/s recovering 2022-09-07T22:28:31.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:28:30.895+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:28:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:32 smithi055 ceph-mon[97033]: pgmap v1529: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 1968/13572 objects degraded (14.500%); 28 MiB/s, 14 objects/s recovering 2022-09-07T22:28:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:32 smithi055 ceph-mon[97033]: osdmap e175: 12 total, 11 up, 11 in 2022-09-07T22:28:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:32 smithi059 ceph-mon[97450]: pgmap v1529: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 1968/13572 objects degraded (14.500%); 28 MiB/s, 14 objects/s recovering 2022-09-07T22:28:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:32 smithi059 ceph-mon[97450]: osdmap e175: 12 total, 11 up, 11 in 2022-09-07T22:28:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:33 smithi055 ceph-mon[97033]: osdmap e176: 12 total, 11 up, 11 in 2022-09-07T22:28:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:33 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 1968/13572 objects degraded (14.500%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2022-09-07T22:28:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:33 smithi059 ceph-mon[97450]: osdmap e176: 12 total, 11 up, 11 in 2022-09-07T22:28:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:33 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 1968/13572 objects degraded (14.500%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2022-09-07T22:28:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:34 smithi055 ceph-mon[97033]: pgmap v1532: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 1968/13572 objects degraded (14.500%); 28 MiB/s, 14 objects/s recovering 2022-09-07T22:28:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:34 smithi059 ceph-mon[97450]: pgmap v1532: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 1968/13572 objects degraded (14.500%); 28 MiB/s, 14 objects/s recovering 2022-09-07T22:28:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:36 smithi055 ceph-mon[97033]: pgmap v1533: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 1968/13572 objects degraded (14.500%); 14 MiB/s, 7 objects/s recovering 2022-09-07T22:28:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:36 smithi059 ceph-mon[97450]: pgmap v1533: 129 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 122 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 1968/13572 objects degraded (14.500%); 14 MiB/s, 7 objects/s recovering 2022-09-07T22:28:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:38 smithi055 ceph-mon[97033]: pgmap v1534: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 1759/13572 objects degraded (12.961%); 17 MiB/s, 14 objects/s recovering 2022-09-07T22:28:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:38 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 1759/13572 objects degraded (12.961%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2022-09-07T22:28:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:38 smithi059 ceph-mon[97450]: pgmap v1534: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 1759/13572 objects degraded (12.961%); 17 MiB/s, 14 objects/s recovering 2022-09-07T22:28:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:38 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 1759/13572 objects degraded (12.961%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2022-09-07T22:28:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:40 smithi055 ceph-mon[97033]: pgmap v1535: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 1759/13572 objects degraded (12.961%); 3.1 MiB/s, 7 objects/s recovering 2022-09-07T22:28:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:40 smithi059 ceph-mon[97450]: pgmap v1535: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 1759/13572 objects degraded (12.961%); 3.1 MiB/s, 7 objects/s recovering 2022-09-07T22:28:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:42 smithi055 ceph-mon[97033]: pgmap v1536: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s; 1756/13572 objects degraded (12.938%); 3.8 MiB/s, 12 objects/s recovering 2022-09-07T22:28:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:42 smithi059 ceph-mon[97450]: pgmap v1536: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s; 1756/13572 objects degraded (12.938%); 3.8 MiB/s, 12 objects/s recovering 2022-09-07T22:28:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:43 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 1756/13572 objects degraded (12.938%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2022-09-07T22:28:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:43 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 1756/13572 objects degraded (12.938%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2022-09-07T22:28:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:44 smithi055 ceph-mon[97033]: pgmap v1537: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s; 1756/13572 objects degraded (12.938%); 3.5 MiB/s, 11 objects/s recovering 2022-09-07T22:28:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:44 smithi059 ceph-mon[97450]: pgmap v1537: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s; 1756/13572 objects degraded (12.938%); 3.5 MiB/s, 11 objects/s recovering 2022-09-07T22:28:46.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:28:45.910+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:28:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:46 smithi055 ceph-mon[97033]: pgmap v1538: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 1756/13572 objects degraded (12.938%); 3.0 MiB/s, 9 objects/s recovering 2022-09-07T22:28:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:46 smithi059 ceph-mon[97450]: pgmap v1538: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 1756/13572 objects degraded (12.938%); 3.0 MiB/s, 9 objects/s recovering 2022-09-07T22:28:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:48 smithi055 ceph-mon[97033]: pgmap v1539: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 1736/13572 objects degraded (12.791%); 9.7 MiB/s, 14 objects/s recovering 2022-09-07T22:28:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:48 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 1736/13572 objects degraded (12.791%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2022-09-07T22:28:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:48 smithi059 ceph-mon[97450]: pgmap v1539: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 1736/13572 objects degraded (12.791%); 9.7 MiB/s, 14 objects/s recovering 2022-09-07T22:28:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:48 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 1736/13572 objects degraded (12.791%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2022-09-07T22:28:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:50 smithi055 ceph-mon[97033]: pgmap v1540: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 1736/13572 objects degraded (12.791%); 7.7 MiB/s, 9 objects/s recovering 2022-09-07T22:28:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:50 smithi059 ceph-mon[97450]: pgmap v1540: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 1736/13572 objects degraded (12.791%); 7.7 MiB/s, 9 objects/s recovering 2022-09-07T22:28:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:52 smithi055 ceph-mon[97033]: pgmap v1541: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 1708/13572 objects degraded (12.585%); 17 MiB/s, 14 objects/s recovering 2022-09-07T22:28:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:52 smithi059 ceph-mon[97450]: pgmap v1541: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 1708/13572 objects degraded (12.585%); 17 MiB/s, 14 objects/s recovering 2022-09-07T22:28:53.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:53 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 1708/13572 objects degraded (12.585%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2022-09-07T22:28:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:53 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 1708/13572 objects degraded (12.585%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2022-09-07T22:28:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:54 smithi055 ceph-mon[97033]: pgmap v1542: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 1708/13572 objects degraded (12.585%); 16 MiB/s, 9 objects/s recovering 2022-09-07T22:28:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:54 smithi059 ceph-mon[97450]: pgmap v1542: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 1708/13572 objects degraded (12.585%); 16 MiB/s, 9 objects/s recovering 2022-09-07T22:28:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:56 smithi055 ceph-mon[97033]: pgmap v1543: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 1708/13572 objects degraded (12.585%); 16 MiB/s, 9 objects/s recovering 2022-09-07T22:28:56.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:56 smithi055 ceph-mon[97033]: osdmap e177: 12 total, 11 up, 11 in 2022-09-07T22:28:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:56 smithi059 ceph-mon[97450]: pgmap v1543: 129 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 1708/13572 objects degraded (12.585%); 16 MiB/s, 9 objects/s recovering 2022-09-07T22:28:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:56 smithi059 ceph-mon[97450]: osdmap e177: 12 total, 11 up, 11 in 2022-09-07T22:28:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:57 smithi055 ceph-mon[97033]: osdmap e178: 12 total, 11 up, 11 in 2022-09-07T22:28:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:57 smithi059 ceph-mon[97450]: osdmap e178: 12 total, 11 up, 11 in 2022-09-07T22:28:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:58 smithi055 ceph-mon[97033]: pgmap v1546: 129 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 1681/13572 objects degraded (12.386%); 27 MiB/s, 14 objects/s recovering 2022-09-07T22:28:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:28:58 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 1681/13572 objects degraded (12.386%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2022-09-07T22:28:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:58 smithi059 ceph-mon[97450]: pgmap v1546: 129 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 1681/13572 objects degraded (12.386%); 27 MiB/s, 14 objects/s recovering 2022-09-07T22:28:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:28:58 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 1681/13572 objects degraded (12.386%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:00 smithi055 ceph-mon[97033]: pgmap v1547: 129 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 1681/13572 objects degraded (12.386%); 13 MiB/s, 7 objects/s recovering 2022-09-07T22:29:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:00 smithi059 ceph-mon[97450]: pgmap v1547: 129 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 123 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 1681/13572 objects degraded (12.386%); 13 MiB/s, 7 objects/s recovering 2022-09-07T22:29:01.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:29:00.924+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:29:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:01 smithi055 ceph-mon[97033]: osdmap e179: 12 total, 11 up, 11 in 2022-09-07T22:29:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:01 smithi059 ceph-mon[97450]: osdmap e179: 12 total, 11 up, 11 in 2022-09-07T22:29:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:02 smithi055 ceph-mon[97033]: pgmap v1549: 129 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 124 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 5.0 KiB/s rd, 9 op/s; 1095/13572 objects degraded (8.068%); 31 MiB/s, 21 objects/s recovering 2022-09-07T22:29:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:02 smithi055 ceph-mon[97033]: osdmap e180: 12 total, 11 up, 11 in 2022-09-07T22:29:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:02 smithi059 ceph-mon[97450]: pgmap v1549: 129 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 124 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 5.0 KiB/s rd, 9 op/s; 1095/13572 objects degraded (8.068%); 31 MiB/s, 21 objects/s recovering 2022-09-07T22:29:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:02 smithi059 ceph-mon[97450]: osdmap e180: 12 total, 11 up, 11 in 2022-09-07T22:29:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:03 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 1095/13572 objects degraded (8.068%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:03.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:03 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 1095/13572 objects degraded (8.068%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:04 smithi059 ceph-mon[97450]: pgmap v1551: 129 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 124 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s; 1095/13572 objects degraded (8.068%); 12 MiB/s, 11 objects/s recovering 2022-09-07T22:29:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:04 smithi055 ceph-mon[97033]: pgmap v1551: 129 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 124 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s; 1095/13572 objects degraded (8.068%); 12 MiB/s, 11 objects/s recovering 2022-09-07T22:29:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:29:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:29:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:29:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:29:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:06 smithi059 ceph-mon[97450]: pgmap v1552: 129 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 124 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 1095/13572 objects degraded (8.068%); 9.5 MiB/s, 9 objects/s recovering 2022-09-07T22:29:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:06 smithi055 ceph-mon[97033]: pgmap v1552: 129 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 124 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 1095/13572 objects degraded (8.068%); 9.5 MiB/s, 9 objects/s recovering 2022-09-07T22:29:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:08 smithi059 ceph-mon[97450]: pgmap v1553: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 66 KiB/s wr, 29 op/s; 1095/13572 objects degraded (8.068%); 9.5 MiB/s, 16 objects/s recovering 2022-09-07T22:29:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:08 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 1095/13572 objects degraded (8.068%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:08 smithi055 ceph-mon[97033]: pgmap v1553: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 66 KiB/s wr, 29 op/s; 1095/13572 objects degraded (8.068%); 9.5 MiB/s, 16 objects/s recovering 2022-09-07T22:29:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:08 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 1095/13572 objects degraded (8.068%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:10 smithi055 ceph-mon[97033]: pgmap v1554: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 63 KiB/s wr, 24 op/s; 1095/13572 objects degraded (8.068%); 0 B/s, 6 objects/s recovering 2022-09-07T22:29:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:10 smithi055 ceph-mon[97033]: osdmap e181: 12 total, 11 up, 11 in 2022-09-07T22:29:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:10 smithi059 ceph-mon[97450]: pgmap v1554: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 63 KiB/s wr, 24 op/s; 1095/13572 objects degraded (8.068%); 0 B/s, 6 objects/s recovering 2022-09-07T22:29:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:10 smithi059 ceph-mon[97450]: osdmap e181: 12 total, 11 up, 11 in 2022-09-07T22:29:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:12 smithi055 ceph-mon[97033]: pgmap v1556: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 57 KiB/s wr, 25 op/s; 1090/13572 objects degraded (8.031%); 2.1 MiB/s, 12 objects/s recovering 2022-09-07T22:29:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:12 smithi059 ceph-mon[97450]: pgmap v1556: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 57 KiB/s wr, 25 op/s; 1090/13572 objects degraded (8.031%); 2.1 MiB/s, 12 objects/s recovering 2022-09-07T22:29:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:13 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 1090/13572 objects degraded (8.031%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:13 smithi055 ceph-mon[97033]: osdmap e182: 12 total, 11 up, 11 in 2022-09-07T22:29:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:13 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 1090/13572 objects degraded (8.031%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:13 smithi059 ceph-mon[97450]: osdmap e182: 12 total, 11 up, 11 in 2022-09-07T22:29:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:14 smithi055 ceph-mon[97033]: pgmap v1557: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 1090/13572 objects degraded (8.031%); 2.0 MiB/s, 11 objects/s recovering 2022-09-07T22:29:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:14 smithi059 ceph-mon[97450]: pgmap v1557: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s; 1090/13572 objects degraded (8.031%); 2.0 MiB/s, 11 objects/s recovering 2022-09-07T22:29:16.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:29:15.939+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:29:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:16 smithi055 ceph-mon[97033]: pgmap v1559: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s; 1090/13572 objects degraded (8.031%); 2.5 MiB/s, 7 objects/s recovering 2022-09-07T22:29:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:16 smithi059 ceph-mon[97450]: pgmap v1559: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s; 1090/13572 objects degraded (8.031%); 2.5 MiB/s, 7 objects/s recovering 2022-09-07T22:29:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:18 smithi055 ceph-mon[97033]: pgmap v1560: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s; 1074/13572 objects degraded (7.913%); 10 MiB/s, 14 objects/s recovering 2022-09-07T22:29:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:18 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 1074/13572 objects degraded (7.913%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:18 smithi059 ceph-mon[97450]: pgmap v1560: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s; 1074/13572 objects degraded (7.913%); 10 MiB/s, 14 objects/s recovering 2022-09-07T22:29:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:18 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 1074/13572 objects degraded (7.913%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:20 smithi055 ceph-mon[97033]: pgmap v1561: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s; 1074/13572 objects degraded (7.913%); 8.8 MiB/s, 12 objects/s recovering 2022-09-07T22:29:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:20 smithi059 ceph-mon[97450]: pgmap v1561: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s; 1074/13572 objects degraded (7.913%); 8.8 MiB/s, 12 objects/s recovering 2022-09-07T22:29:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:22 smithi059 ceph-mon[97450]: pgmap v1562: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s; 1047/13572 objects degraded (7.714%); 17 MiB/s, 11 objects/s recovering 2022-09-07T22:29:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:29:22.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:29:22.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:29:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:22 smithi055 ceph-mon[97033]: pgmap v1562: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s; 1047/13572 objects degraded (7.714%); 17 MiB/s, 11 objects/s recovering 2022-09-07T22:29:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:29:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:29:22.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:29:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:29:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:23 smithi055 ceph-mon[97033]: pgmap v1563: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s; 1047/13572 objects degraded (7.714%); 17 MiB/s, 11 objects/s recovering 2022-09-07T22:29:23.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:23 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 1047/13572 objects degraded (7.714%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:29:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:23 smithi059 ceph-mon[97450]: pgmap v1563: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s; 1047/13572 objects degraded (7.714%); 17 MiB/s, 11 objects/s recovering 2022-09-07T22:29:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:23 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 1047/13572 objects degraded (7.714%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:25 smithi055 ceph-mon[97033]: pgmap v1564: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 88 B/s wr, 5 op/s; 1047/13572 objects degraded (7.714%); 15 MiB/s, 10 objects/s recovering 2022-09-07T22:29:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:25 smithi059 ceph-mon[97450]: pgmap v1564: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 88 B/s wr, 5 op/s; 1047/13572 objects degraded (7.714%); 15 MiB/s, 10 objects/s recovering 2022-09-07T22:29:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:27 smithi055 ceph-mon[97033]: pgmap v1565: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 1019/13572 objects degraded (7.508%); 24 MiB/s, 14 objects/s recovering 2022-09-07T22:29:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:27 smithi059 ceph-mon[97450]: pgmap v1565: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s; 1019/13572 objects degraded (7.508%); 24 MiB/s, 14 objects/s recovering 2022-09-07T22:29:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:28 smithi055 ceph-mon[97033]: osdmap e183: 12 total, 11 up, 11 in 2022-09-07T22:29:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:28 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 1019/13572 objects degraded (7.508%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:28 smithi059 ceph-mon[97450]: osdmap e183: 12 total, 11 up, 11 in 2022-09-07T22:29:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:28 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 1019/13572 objects degraded (7.508%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:29 smithi055 ceph-mon[97033]: pgmap v1567: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s; 1019/13572 objects degraded (7.508%); 22 MiB/s, 11 objects/s recovering 2022-09-07T22:29:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:29 smithi055 ceph-mon[97033]: osdmap e184: 12 total, 11 up, 11 in 2022-09-07T22:29:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:29 smithi059 ceph-mon[97450]: pgmap v1567: 129 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 125 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s; 1019/13572 objects degraded (7.508%); 22 MiB/s, 11 objects/s recovering 2022-09-07T22:29:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:29 smithi059 ceph-mon[97450]: osdmap e184: 12 total, 11 up, 11 in 2022-09-07T22:29:31.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:29:30.954+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:29:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:31 smithi055 ceph-mon[97033]: pgmap v1569: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 126 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 726/13572 objects degraded (5.349%); 25 MiB/s, 32 objects/s recovering 2022-09-07T22:29:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:31 smithi059 ceph-mon[97450]: pgmap v1569: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 126 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 726/13572 objects degraded (5.349%); 25 MiB/s, 32 objects/s recovering 2022-09-07T22:29:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:33 smithi055 ceph-mon[97033]: pgmap v1570: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 126 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 726/13572 objects degraded (5.349%); 25 MiB/s, 32 objects/s recovering 2022-09-07T22:29:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:33 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 726/13572 objects degraded (5.349%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:33 smithi059 ceph-mon[97450]: pgmap v1570: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 126 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 726/13572 objects degraded (5.349%); 25 MiB/s, 32 objects/s recovering 2022-09-07T22:29:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:33 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 726/13572 objects degraded (5.349%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:35 smithi055 ceph-mon[97033]: osdmap e185: 12 total, 11 up, 11 in 2022-09-07T22:29:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:35 smithi059 ceph-mon[97450]: osdmap e185: 12 total, 11 up, 11 in 2022-09-07T22:29:36.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:36 smithi055 ceph-mon[97033]: pgmap v1572: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 126 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.2 KiB/s rd, 4 op/s; 726/13572 objects degraded (5.349%); 13 MiB/s, 29 objects/s recovering 2022-09-07T22:29:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:36 smithi055 ceph-mon[97033]: osdmap e186: 12 total, 11 up, 11 in 2022-09-07T22:29:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:36 smithi059 ceph-mon[97450]: pgmap v1572: 129 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 126 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.2 KiB/s rd, 4 op/s; 726/13572 objects degraded (5.349%); 13 MiB/s, 29 objects/s recovering 2022-09-07T22:29:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:36 smithi059 ceph-mon[97450]: osdmap e186: 12 total, 11 up, 11 in 2022-09-07T22:29:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:38 smithi055 ceph-mon[97033]: pgmap v1574: 129 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 126 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s; 167/13572 objects degraded (1.230%); 65 MiB/s, 67 objects/s recovering 2022-09-07T22:29:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:38 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 167/13572 objects degraded (1.230%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:38.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:38 smithi055 ceph-mon[97033]: osdmap e187: 12 total, 11 up, 11 in 2022-09-07T22:29:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:38 smithi059 ceph-mon[97450]: pgmap v1574: 129 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 126 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s; 167/13572 objects degraded (1.230%); 65 MiB/s, 67 objects/s recovering 2022-09-07T22:29:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:38 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 167/13572 objects degraded (1.230%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2022-09-07T22:29:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:38 smithi059 ceph-mon[97450]: osdmap e187: 12 total, 11 up, 11 in 2022-09-07T22:29:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:40 smithi055 ceph-mon[97033]: pgmap v1576: 129 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 126 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 167/13572 objects degraded (1.230%); 71 MiB/s, 55 objects/s recovering 2022-09-07T22:29:40.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:40 smithi055 ceph-mon[97033]: osdmap e188: 12 total, 11 up, 11 in 2022-09-07T22:29:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:40 smithi059 ceph-mon[97450]: pgmap v1576: 129 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 126 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 167/13572 objects degraded (1.230%); 71 MiB/s, 55 objects/s recovering 2022-09-07T22:29:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:40 smithi059 ceph-mon[97450]: osdmap e188: 12 total, 11 up, 11 in 2022-09-07T22:29:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:42 smithi055 ceph-mon[97033]: pgmap v1578: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 5.0 KiB/s rd, 9 op/s; 70/13572 objects degraded (0.516%); 70 MiB/s, 45 objects/s recovering 2022-09-07T22:29:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:42 smithi059 ceph-mon[97450]: pgmap v1578: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 5.0 KiB/s rd, 9 op/s; 70/13572 objects degraded (0.516%); 70 MiB/s, 45 objects/s recovering 2022-09-07T22:29:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:43 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 70/13572 objects degraded (0.516%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2022-09-07T22:29:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:43 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 70/13572 objects degraded (0.516%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2022-09-07T22:29:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:44 smithi055 ceph-mon[97033]: pgmap v1579: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 3 op/s; 70/13572 objects degraded (0.516%) 2022-09-07T22:29:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:44 smithi059 ceph-mon[97450]: pgmap v1579: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 3 op/s; 70/13572 objects degraded (0.516%) 2022-09-07T22:29:46.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:29:45.969+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:29:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:46 smithi055 ceph-mon[97033]: pgmap v1580: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 70/13572 objects degraded (0.516%) 2022-09-07T22:29:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:46 smithi059 ceph-mon[97450]: pgmap v1580: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 70/13572 objects degraded (0.516%) 2022-09-07T22:29:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:48 smithi055 ceph-mon[97033]: pgmap v1581: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s; 65/13572 objects degraded (0.479%); 1.8 MiB/s, 0 objects/s recovering 2022-09-07T22:29:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:48 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 65/13572 objects degraded (0.479%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2022-09-07T22:29:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:48 smithi059 ceph-mon[97450]: pgmap v1581: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s; 65/13572 objects degraded (0.479%); 1.8 MiB/s, 0 objects/s recovering 2022-09-07T22:29:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:48 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 65/13572 objects degraded (0.479%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2022-09-07T22:29:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:50 smithi055 ceph-mon[97033]: pgmap v1582: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s; 65/13572 objects degraded (0.479%); 1.6 MiB/s, 0 objects/s recovering 2022-09-07T22:29:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:50 smithi059 ceph-mon[97450]: pgmap v1582: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s; 65/13572 objects degraded (0.479%); 1.6 MiB/s, 0 objects/s recovering 2022-09-07T22:29:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:52 smithi055 ceph-mon[97033]: pgmap v1583: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s; 41/13572 objects degraded (0.302%); 9.7 MiB/s, 4 objects/s recovering 2022-09-07T22:29:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:52 smithi059 ceph-mon[97450]: pgmap v1583: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s; 41/13572 objects degraded (0.302%); 9.7 MiB/s, 4 objects/s recovering 2022-09-07T22:29:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:53 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 41/13572 objects degraded (0.302%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2022-09-07T22:29:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:53 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 41/13572 objects degraded (0.302%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2022-09-07T22:29:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:54 smithi055 ceph-mon[97033]: pgmap v1584: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 41/13572 objects degraded (0.302%); 9.7 MiB/s, 9 objects/s recovering 2022-09-07T22:29:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:54 smithi059 ceph-mon[97450]: pgmap v1584: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 41/13572 objects degraded (0.302%); 9.7 MiB/s, 9 objects/s recovering 2022-09-07T22:29:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:56 smithi055 ceph-mon[97033]: pgmap v1585: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 41/13572 objects degraded (0.302%); 9.7 MiB/s, 9 objects/s recovering 2022-09-07T22:29:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:56 smithi059 ceph-mon[97450]: pgmap v1585: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s; 41/13572 objects degraded (0.302%); 9.7 MiB/s, 9 objects/s recovering 2022-09-07T22:29:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:58 smithi055 ceph-mon[97033]: pgmap v1586: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 12/13572 objects degraded (0.088%); 19 MiB/s, 14 objects/s recovering 2022-09-07T22:29:58.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:58 smithi055 ceph-mon[97033]: osdmap e189: 12 total, 11 up, 11 in 2022-09-07T22:29:58.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:58 smithi055 ceph-mon[97033]: Health check update: Degraded data redundancy: 12/13572 objects degraded (0.088%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2022-09-07T22:29:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:58 smithi059 ceph-mon[97450]: pgmap v1586: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 12/13572 objects degraded (0.088%); 19 MiB/s, 14 objects/s recovering 2022-09-07T22:29:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:58 smithi059 ceph-mon[97450]: osdmap e189: 12 total, 11 up, 11 in 2022-09-07T22:29:58.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:58 smithi059 ceph-mon[97450]: Health check update: Degraded data redundancy: 12/13572 objects degraded (0.088%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2022-09-07T22:29:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:29:59 smithi055 ceph-mon[97033]: osdmap e190: 12 total, 11 up, 11 in 2022-09-07T22:29:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:29:59 smithi059 ceph-mon[97450]: osdmap e190: 12 total, 11 up, 11 in 2022-09-07T22:30:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:00 smithi055 ceph-mon[97033]: pgmap v1589: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 12/13572 objects degraded (0.088%); 14 MiB/s, 7 objects/s recovering 2022-09-07T22:30:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b; Degraded data redundancy: 12/13572 objects degraded (0.088%), 1 pg degraded, 1 pg undersized 2022-09-07T22:30:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T22:30:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T22:30:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:00 smithi055 ceph-mon[97033]: [WRN] PG_DEGRADED: Degraded data redundancy: 12/13572 objects degraded (0.088%), 1 pg degraded, 1 pg undersized 2022-09-07T22:30:00.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:00 smithi055 ceph-mon[97033]: pg 4.7 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [6,8,10,NONE] 2022-09-07T22:30:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:00 smithi059 ceph-mon[97450]: pgmap v1589: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s; 12/13572 objects degraded (0.088%); 14 MiB/s, 7 objects/s recovering 2022-09-07T22:30:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b; Degraded data redundancy: 12/13572 objects degraded (0.088%), 1 pg degraded, 1 pg undersized 2022-09-07T22:30:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T22:30:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T22:30:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:00 smithi059 ceph-mon[97450]: [WRN] PG_DEGRADED: Degraded data redundancy: 12/13572 objects degraded (0.088%), 1 pg degraded, 1 pg undersized 2022-09-07T22:30:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:00 smithi059 ceph-mon[97450]: pg 4.7 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [6,8,10,NONE] 2022-09-07T22:30:01.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:30:00.974+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:30:01.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:01 smithi055 ceph-mon[97033]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/13572 objects degraded (0.088%), 1 pg degraded, 1 pg undersized) 2022-09-07T22:30:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:01 smithi059 ceph-mon[97450]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/13572 objects degraded (0.088%), 1 pg degraded, 1 pg undersized) 2022-09-07T22:30:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:02 smithi055 ceph-mon[97033]: pgmap v1590: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 20 MiB/s, 10 objects/s recovering 2022-09-07T22:30:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:02 smithi059 ceph-mon[97450]: pgmap v1590: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 20 MiB/s, 10 objects/s recovering 2022-09-07T22:30:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:04 smithi055 ceph-mon[97033]: pgmap v1591: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 20 MiB/s, 10 objects/s recovering 2022-09-07T22:30:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:04 smithi055 ceph-mon[97033]: osdmap e191: 12 total, 11 up, 11 in 2022-09-07T22:30:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:04 smithi059 ceph-mon[97450]: pgmap v1591: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s; 20 MiB/s, 10 objects/s recovering 2022-09-07T22:30:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:04 smithi059 ceph-mon[97450]: osdmap e191: 12 total, 11 up, 11 in 2022-09-07T22:30:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:30:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:30:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:30:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:30:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:06 smithi055 ceph-mon[97033]: pgmap v1593: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 3 op/s; 6.4 MiB/s, 3 objects/s recovering 2022-09-07T22:30:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:06 smithi059 ceph-mon[97450]: pgmap v1593: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 3 op/s; 6.4 MiB/s, 3 objects/s recovering 2022-09-07T22:30:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:08 smithi055 ceph-mon[97033]: pgmap v1594: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s; 5.6 MiB/s, 3 objects/s recovering 2022-09-07T22:30:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:08 smithi059 ceph-mon[97450]: pgmap v1594: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s; 5.6 MiB/s, 3 objects/s recovering 2022-09-07T22:30:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:10 smithi055 ceph-mon[97033]: pgmap v1595: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s; 4.8 MiB/s, 2 objects/s recovering 2022-09-07T22:30:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:10 smithi055 ceph-mon[97033]: osdmap e192: 12 total, 11 up, 11 in 2022-09-07T22:30:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:10 smithi059 ceph-mon[97450]: pgmap v1595: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s; 4.8 MiB/s, 2 objects/s recovering 2022-09-07T22:30:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:10 smithi059 ceph-mon[97450]: osdmap e192: 12 total, 11 up, 11 in 2022-09-07T22:30:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:12 smithi055 ceph-mon[97033]: pgmap v1597: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-07T22:30:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:12 smithi059 ceph-mon[97450]: pgmap v1597: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-07T22:30:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:14 smithi055 ceph-mon[97033]: pgmap v1598: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 56 KiB/s wr, 25 op/s 2022-09-07T22:30:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:14 smithi055 ceph-mon[97033]: osdmap e193: 12 total, 11 up, 11 in 2022-09-07T22:30:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:14 smithi059 ceph-mon[97450]: pgmap v1598: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 56 KiB/s wr, 25 op/s 2022-09-07T22:30:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:14 smithi059 ceph-mon[97450]: osdmap e193: 12 total, 11 up, 11 in 2022-09-07T22:30:16.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:30:15.990+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:30:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:16 smithi055 ceph-mon[97033]: pgmap v1600: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:30:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:16 smithi059 ceph-mon[97450]: pgmap v1600: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:30:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:18 smithi055 ceph-mon[97033]: pgmap v1601: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:30:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:18 smithi059 ceph-mon[97450]: pgmap v1601: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:30:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:20 smithi055 ceph-mon[97033]: pgmap v1602: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-07T22:30:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:20 smithi059 ceph-mon[97450]: pgmap v1602: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-07T22:30:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:22 smithi055 ceph-mon[97033]: pgmap v1603: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:30:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:22 smithi059 ceph-mon[97450]: pgmap v1603: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:30:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:30:23.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:30:23.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:30:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:30:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:30:23.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:30:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:24 smithi055 ceph-mon[97033]: pgmap v1604: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:30:24.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:30:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:24 smithi059 ceph-mon[97450]: pgmap v1604: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:30:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:30:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:26 smithi055 ceph-mon[97033]: pgmap v1605: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 88 B/s wr, 5 op/s 2022-09-07T22:30:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:26 smithi059 ceph-mon[97450]: pgmap v1605: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 88 B/s wr, 5 op/s 2022-09-07T22:30:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:28 smithi055 ceph-mon[97033]: pgmap v1606: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:30:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:28 smithi059 ceph-mon[97450]: pgmap v1606: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:30:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:30 smithi055 ceph-mon[97033]: pgmap v1607: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:30 smithi059 ceph-mon[97450]: pgmap v1607: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:31.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:30:30.990+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:30:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:32 smithi055 ceph-mon[97033]: pgmap v1608: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:30:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:32 smithi059 ceph-mon[97450]: pgmap v1608: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:30:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:34 smithi059 ceph-mon[97450]: pgmap v1609: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:34 smithi055 ceph-mon[97033]: pgmap v1609: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:36 smithi059 ceph-mon[97450]: pgmap v1610: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:36 smithi055 ceph-mon[97033]: pgmap v1610: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:38 smithi059 ceph-mon[97450]: pgmap v1611: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:30:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:38 smithi055 ceph-mon[97033]: pgmap v1611: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:30:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:40 smithi059 ceph-mon[97450]: pgmap v1612: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:40 smithi055 ceph-mon[97033]: pgmap v1612: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:42 smithi059 ceph-mon[97450]: pgmap v1613: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:30:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:42 smithi055 ceph-mon[97033]: pgmap v1613: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:30:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:44 smithi059 ceph-mon[97450]: pgmap v1614: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:44 smithi055 ceph-mon[97033]: pgmap v1614: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:46.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:30:46.006+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:30:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:46 smithi059 ceph-mon[97450]: pgmap v1615: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:46 smithi055 ceph-mon[97033]: pgmap v1615: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:48 smithi059 ceph-mon[97450]: pgmap v1616: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:30:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:48 smithi055 ceph-mon[97033]: pgmap v1616: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:30:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:50 smithi059 ceph-mon[97450]: pgmap v1617: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:50 smithi055 ceph-mon[97033]: pgmap v1617: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:52 smithi059 ceph-mon[97450]: pgmap v1618: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:30:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:52 smithi055 ceph-mon[97033]: pgmap v1618: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:30:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:54 smithi059 ceph-mon[97450]: pgmap v1619: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:54 smithi055 ceph-mon[97033]: pgmap v1619: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:56 smithi059 ceph-mon[97450]: pgmap v1620: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:56 smithi055 ceph-mon[97033]: pgmap v1620: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:30:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:30:58 smithi059 ceph-mon[97450]: pgmap v1621: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:30:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:30:58 smithi055 ceph-mon[97033]: pgmap v1621: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:31:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:00 smithi059 ceph-mon[97450]: pgmap v1622: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:00 smithi055 ceph-mon[97033]: pgmap v1622: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:01.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:31:01.017+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:31:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:02 smithi059 ceph-mon[97450]: pgmap v1623: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:31:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:02 smithi055 ceph-mon[97033]: pgmap v1623: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:31:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:04 smithi059 ceph-mon[97450]: pgmap v1624: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:04 smithi055 ceph-mon[97033]: pgmap v1624: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:31:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:31:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:31:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:31:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:06 smithi055 ceph-mon[97033]: pgmap v1625: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:06 smithi059 ceph-mon[97450]: pgmap v1625: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:08 smithi055 ceph-mon[97033]: pgmap v1626: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:31:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:08 smithi059 ceph-mon[97450]: pgmap v1626: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:31:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:10 smithi055 ceph-mon[97033]: pgmap v1627: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:31:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:10 smithi055 ceph-mon[97033]: osdmap e194: 12 total, 11 up, 11 in 2022-09-07T22:31:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:10 smithi059 ceph-mon[97450]: pgmap v1627: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:31:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:10 smithi059 ceph-mon[97450]: osdmap e194: 12 total, 11 up, 11 in 2022-09-07T22:31:12.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:12 smithi055 ceph-mon[97033]: pgmap v1629: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:31:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:12 smithi059 ceph-mon[97450]: pgmap v1629: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:31:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:14 smithi055 ceph-mon[97033]: pgmap v1630: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:31:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:14 smithi059 ceph-mon[97450]: pgmap v1630: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:31:16.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:31:16.029+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:31:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:16 smithi055 ceph-mon[97033]: pgmap v1631: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:31:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:16 smithi059 ceph-mon[97450]: pgmap v1631: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:31:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:18 smithi055 ceph-mon[97033]: pgmap v1632: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:31:18.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:18 smithi055 ceph-mon[97033]: osdmap e195: 12 total, 11 up, 11 in 2022-09-07T22:31:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:18 smithi059 ceph-mon[97450]: pgmap v1632: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:31:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:18 smithi059 ceph-mon[97450]: osdmap e195: 12 total, 11 up, 11 in 2022-09-07T22:31:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:20 smithi055 ceph-mon[97033]: pgmap v1634: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2022-09-07T22:31:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:20 smithi059 ceph-mon[97450]: pgmap v1634: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2022-09-07T22:31:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:22 smithi055 ceph-mon[97033]: pgmap v1635: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:31:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:22 smithi059 ceph-mon[97450]: pgmap v1635: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:31:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:24 smithi055 ceph-mon[97033]: pgmap v1636: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:31:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:31:24.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:31:24.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:31:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:24 smithi059 ceph-mon[97450]: pgmap v1636: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:31:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:31:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:31:24.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:31:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:26 smithi055 ceph-mon[97033]: pgmap v1637: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:31:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:26 smithi059 ceph-mon[97450]: pgmap v1637: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:31:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:28 smithi055 ceph-mon[97033]: pgmap v1638: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:31:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:28 smithi059 ceph-mon[97450]: pgmap v1638: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:31:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:30 smithi055 ceph-mon[97033]: pgmap v1639: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:31:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:30 smithi059 ceph-mon[97450]: pgmap v1639: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:31:31.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:31:31.034+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:31:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:32 smithi055 ceph-mon[97033]: pgmap v1640: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:31:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:32 smithi059 ceph-mon[97450]: pgmap v1640: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:31:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:34 smithi055 ceph-mon[97033]: pgmap v1641: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:34 smithi059 ceph-mon[97450]: pgmap v1641: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:36 smithi055 ceph-mon[97033]: pgmap v1642: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:36 smithi059 ceph-mon[97450]: pgmap v1642: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:38 smithi055 ceph-mon[97033]: pgmap v1643: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:31:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:38 smithi059 ceph-mon[97450]: pgmap v1643: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:31:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:40 smithi055 ceph-mon[97033]: pgmap v1644: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:40 smithi059 ceph-mon[97450]: pgmap v1644: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:42 smithi055 ceph-mon[97033]: pgmap v1645: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:31:42.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:42 smithi059 ceph-mon[97450]: pgmap v1645: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:31:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:44 smithi055 ceph-mon[97033]: pgmap v1646: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:44 smithi059 ceph-mon[97450]: pgmap v1646: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:46.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:31:46.046+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:31:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:46 smithi055 ceph-mon[97033]: pgmap v1647: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:46 smithi059 ceph-mon[97450]: pgmap v1647: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:48 smithi055 ceph-mon[97033]: pgmap v1648: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:31:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:48 smithi059 ceph-mon[97450]: pgmap v1648: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:31:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:50 smithi055 ceph-mon[97033]: pgmap v1649: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:50 smithi059 ceph-mon[97450]: pgmap v1649: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:52 smithi055 ceph-mon[97033]: pgmap v1650: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:31:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:52 smithi059 ceph-mon[97450]: pgmap v1650: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:31:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:54 smithi055 ceph-mon[97033]: pgmap v1651: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:54.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:54 smithi055 ceph-mon[97033]: pgmap v1652: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:31:54.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:31:54.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:54 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-07T22:31:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:54 smithi059 ceph-mon[97450]: pgmap v1651: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:31:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:54 smithi059 ceph-mon[97450]: pgmap v1652: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:31:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:31:54.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:54 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-07T22:31:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:56 smithi055 ceph-mon[97033]: pgmap v1653: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-07T22:31:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:56 smithi059 ceph-mon[97450]: pgmap v1653: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-07T22:31:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:31:58 smithi055 ceph-mon[97033]: pgmap v1654: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:31:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:31:58 smithi059 ceph-mon[97450]: pgmap v1654: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:32:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:00 smithi055 ceph-mon[97033]: pgmap v1655: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:32:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:00 smithi059 ceph-mon[97450]: pgmap v1655: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:32:01.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:32:01.058+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:32:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:02 smithi059 ceph-mon[97450]: pgmap v1656: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:32:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:02 smithi055 ceph-mon[97033]: pgmap v1656: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:32:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:04 smithi055 ceph-mon[97033]: pgmap v1657: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:32:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:04 smithi059 ceph-mon[97450]: pgmap v1657: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:32:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:32:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:32:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:32:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:32:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:06 smithi055 ceph-mon[97033]: pgmap v1658: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:32:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:06 smithi059 ceph-mon[97450]: pgmap v1658: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:32:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:07 smithi055 ceph-mon[97033]: pgmap v1659: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:32:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:07 smithi059 ceph-mon[97450]: pgmap v1659: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:32:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:10 smithi055 ceph-mon[97033]: osdmap e196: 12 total, 11 up, 11 in 2022-09-07T22:32:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:10 smithi055 ceph-mon[97033]: pgmap v1661: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:32:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:10 smithi059 ceph-mon[97450]: osdmap e196: 12 total, 11 up, 11 in 2022-09-07T22:32:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:10 smithi059 ceph-mon[97450]: pgmap v1661: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:32:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:11 smithi055 ceph-mon[97033]: pgmap v1662: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:32:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:11 smithi059 ceph-mon[97450]: pgmap v1662: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:32:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:14 smithi055 ceph-mon[97033]: pgmap v1663: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:32:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:14 smithi059 ceph-mon[97450]: pgmap v1663: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:32:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:15 smithi055 ceph-mon[97033]: pgmap v1664: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:32:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:15 smithi059 ceph-mon[97450]: pgmap v1664: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:32:16.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:32:16.073+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:32:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:17 smithi055 ceph-mon[97033]: pgmap v1665: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:32:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:17 smithi059 ceph-mon[97450]: pgmap v1665: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:32:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:19 smithi055 ceph-mon[97033]: osdmap e197: 12 total, 11 up, 11 in 2022-09-07T22:32:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:19 smithi059 ceph-mon[97450]: osdmap e197: 12 total, 11 up, 11 in 2022-09-07T22:32:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:20 smithi055 ceph-mon[97033]: pgmap v1667: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:32:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:20 smithi059 ceph-mon[97450]: pgmap v1667: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:32:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:21 smithi055 ceph-mon[97033]: pgmap v1668: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:32:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:21 smithi059 ceph-mon[97450]: pgmap v1668: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:32:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:24 smithi055 ceph-mon[97033]: pgmap v1669: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:32:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:24 smithi059 ceph-mon[97450]: pgmap v1669: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:32:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:25 smithi055 ceph-mon[97033]: pgmap v1670: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:32:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:25 smithi059 ceph-mon[97450]: pgmap v1670: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:32:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:27 smithi055 ceph-mon[97033]: pgmap v1671: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:32:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:27 smithi059 ceph-mon[97450]: pgmap v1671: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:32:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:29 smithi055 ceph-mon[97033]: pgmap v1672: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:32:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:29 smithi059 ceph-mon[97450]: pgmap v1672: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:32:31.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:32:31.089+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:32:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:31 smithi055 ceph-mon[97033]: pgmap v1673: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:32:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:31 smithi059 ceph-mon[97450]: pgmap v1673: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:32:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:34 smithi055 ceph-mon[97033]: pgmap v1674: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:32:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:34 smithi059 ceph-mon[97450]: pgmap v1674: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:32:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:35 smithi055 ceph-mon[97033]: pgmap v1675: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:32:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:35 smithi059 ceph-mon[97450]: pgmap v1675: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:32:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:37 smithi055 ceph-mon[97033]: pgmap v1676: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:32:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:37 smithi059 ceph-mon[97450]: pgmap v1676: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:32:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:39 smithi055 ceph-mon[97033]: pgmap v1677: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:32:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:39 smithi059 ceph-mon[97450]: pgmap v1677: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:32:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:41 smithi055 ceph-mon[97033]: pgmap v1678: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:32:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:41 smithi059 ceph-mon[97450]: pgmap v1678: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:32:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:44 smithi055 ceph-mon[97033]: pgmap v1679: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:32:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:44 smithi059 ceph-mon[97450]: pgmap v1679: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:32:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:45 smithi059 ceph-mon[97450]: pgmap v1680: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:32:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:45 smithi055 ceph-mon[97033]: pgmap v1680: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:32:46.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:32:46.104+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:32:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:47 smithi055 ceph-mon[97033]: pgmap v1681: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:32:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:47 smithi059 ceph-mon[97450]: pgmap v1681: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:32:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:49 smithi055 ceph-mon[97033]: pgmap v1682: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:32:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:49 smithi059 ceph-mon[97450]: pgmap v1682: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:32:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:51 smithi055 ceph-mon[97033]: pgmap v1683: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:32:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:51 smithi059 ceph-mon[97450]: pgmap v1683: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:32:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:54 smithi055 ceph-mon[97033]: pgmap v1684: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:32:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:32:54.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:32:54.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:32:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:54 smithi059 ceph-mon[97450]: pgmap v1684: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:32:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:32:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:32:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:32:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:55 smithi055 ceph-mon[97033]: pgmap v1685: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:32:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:55 smithi059 ceph-mon[97450]: pgmap v1685: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:32:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:57 smithi055 ceph-mon[97033]: pgmap v1686: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:32:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:57 smithi059 ceph-mon[97450]: pgmap v1686: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:32:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:32:59 smithi055 ceph-mon[97033]: pgmap v1687: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:32:59 smithi059 ceph-mon[97450]: pgmap v1687: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:01.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:33:01.119+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:33:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:01 smithi055 ceph-mon[97033]: pgmap v1688: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:33:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:01 smithi059 ceph-mon[97450]: pgmap v1688: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:33:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:04 smithi055 ceph-mon[97033]: pgmap v1689: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:04 smithi059 ceph-mon[97450]: pgmap v1689: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:33:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:33:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:33:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:33:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:06 smithi055 ceph-mon[97033]: pgmap v1690: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:33:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:06 smithi059 ceph-mon[97450]: pgmap v1690: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:33:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:07 smithi055 ceph-mon[97033]: pgmap v1691: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:33:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:07 smithi059 ceph-mon[97450]: pgmap v1691: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:33:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:10 smithi055 ceph-mon[97033]: osdmap e198: 12 total, 11 up, 11 in 2022-09-07T22:33:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:10 smithi055 ceph-mon[97033]: pgmap v1693: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:33:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:10 smithi059 ceph-mon[97450]: osdmap e198: 12 total, 11 up, 11 in 2022-09-07T22:33:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:10 smithi059 ceph-mon[97450]: pgmap v1693: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:33:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:11 smithi055 ceph-mon[97033]: pgmap v1694: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:33:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:11 smithi059 ceph-mon[97450]: pgmap v1694: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:33:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:14 smithi055 ceph-mon[97033]: osdmap e199: 12 total, 11 up, 11 in 2022-09-07T22:33:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:14 smithi055 ceph-mon[97033]: pgmap v1696: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:33:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:14 smithi059 ceph-mon[97450]: osdmap e199: 12 total, 11 up, 11 in 2022-09-07T22:33:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:14 smithi059 ceph-mon[97450]: pgmap v1696: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:33:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:15 smithi055 ceph-mon[97033]: pgmap v1697: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:33:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:15 smithi059 ceph-mon[97450]: pgmap v1697: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:33:16.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:33:16.134+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:33:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:17 smithi055 ceph-mon[97033]: pgmap v1698: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 244 B/s wr, 7 op/s 2022-09-07T22:33:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:17 smithi059 ceph-mon[97450]: pgmap v1698: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 244 B/s wr, 7 op/s 2022-09-07T22:33:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:19 smithi055 ceph-mon[97033]: pgmap v1699: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:33:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:19 smithi059 ceph-mon[97450]: pgmap v1699: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:33:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:21 smithi055 ceph-mon[97033]: pgmap v1700: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:33:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:21 smithi059 ceph-mon[97450]: pgmap v1700: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:33:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:24 smithi055 ceph-mon[97033]: pgmap v1701: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-07T22:33:24.513 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:33:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:24 smithi059 ceph-mon[97450]: pgmap v1701: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-07T22:33:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:33:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:25 smithi055 ceph-mon[97033]: pgmap v1702: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-07T22:33:25.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:25 smithi055 ceph-mon[97033]: pgmap v1703: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:33:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:25 smithi059 ceph-mon[97450]: pgmap v1702: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-07T22:33:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:25 smithi059 ceph-mon[97450]: pgmap v1703: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:33:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:27 smithi055 ceph-mon[97033]: pgmap v1704: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:33:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:27 smithi059 ceph-mon[97450]: pgmap v1704: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:33:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:29 smithi055 ceph-mon[97033]: pgmap v1705: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:33:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:29 smithi059 ceph-mon[97450]: pgmap v1705: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:33:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:31 smithi055 ceph-mon[97033]: pgmap v1706: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:33:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:31 smithi059 ceph-mon[97450]: pgmap v1706: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:33:31.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:33:31.149+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:33:33.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:33 smithi055 ceph-mon[97033]: pgmap v1707: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:33:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:33 smithi059 ceph-mon[97450]: pgmap v1707: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:33:35.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:35 smithi055 ceph-mon[97033]: pgmap v1708: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:33:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:35 smithi059 ceph-mon[97450]: pgmap v1708: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:33:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:37 smithi055 ceph-mon[97033]: pgmap v1709: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:33:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:37 smithi059 ceph-mon[97450]: pgmap v1709: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:33:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:39 smithi055 ceph-mon[97033]: pgmap v1710: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:39 smithi059 ceph-mon[97450]: pgmap v1710: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:41.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:41 smithi055 ceph-mon[97033]: pgmap v1711: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:41 smithi059 ceph-mon[97450]: pgmap v1711: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:43.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:43 smithi055 ceph-mon[97033]: pgmap v1712: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:33:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:43 smithi059 ceph-mon[97450]: pgmap v1712: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:33:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:45 smithi055 ceph-mon[97033]: pgmap v1713: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:45 smithi059 ceph-mon[97450]: pgmap v1713: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:46.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:33:46.155+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:33:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:47 smithi055 ceph-mon[97033]: pgmap v1714: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:33:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:47 smithi059 ceph-mon[97450]: pgmap v1714: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:33:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:49 smithi055 ceph-mon[97033]: pgmap v1715: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:49 smithi059 ceph-mon[97450]: pgmap v1715: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:51 smithi055 ceph-mon[97033]: pgmap v1716: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:51 smithi059 ceph-mon[97450]: pgmap v1716: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:53 smithi055 ceph-mon[97033]: pgmap v1717: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:33:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:53 smithi059 ceph-mon[97450]: pgmap v1717: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:33:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:55 smithi055 ceph-mon[97033]: pgmap v1718: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:55 smithi059 ceph-mon[97450]: pgmap v1718: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:57 smithi055 ceph-mon[97033]: pgmap v1719: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:33:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:57 smithi059 ceph-mon[97450]: pgmap v1719: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:33:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:33:59 smithi055 ceph-mon[97033]: pgmap v1720: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:33:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:33:59 smithi059 ceph-mon[97450]: pgmap v1720: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:34:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:01 smithi055 ceph-mon[97033]: pgmap v1721: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:34:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:01 smithi059 ceph-mon[97450]: pgmap v1721: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:34:01.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:34:01.162+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:34:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:03 smithi055 ceph-mon[97033]: pgmap v1722: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:34:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:03 smithi059 ceph-mon[97450]: pgmap v1722: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:34:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:05 smithi055 ceph-mon[97033]: pgmap v1723: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:34:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:34:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:34:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:05 smithi059 ceph-mon[97450]: pgmap v1723: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:34:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:34:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:34:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:07 smithi055 ceph-mon[97033]: pgmap v1724: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:34:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:07 smithi059 ceph-mon[97450]: pgmap v1724: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:34:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:09 smithi055 ceph-mon[97033]: pgmap v1725: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:34:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:09 smithi059 ceph-mon[97450]: pgmap v1725: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:34:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:10 smithi055 ceph-mon[97033]: osdmap e200: 12 total, 11 up, 11 in 2022-09-07T22:34:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:10 smithi059 ceph-mon[97450]: osdmap e200: 12 total, 11 up, 11 in 2022-09-07T22:34:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:11 smithi055 ceph-mon[97033]: pgmap v1727: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:34:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:11 smithi059 ceph-mon[97450]: pgmap v1727: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:34:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:13 smithi055 ceph-mon[97033]: pgmap v1728: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:34:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:13 smithi059 ceph-mon[97450]: pgmap v1728: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:34:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:15 smithi059 ceph-mon[97450]: pgmap v1729: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:34:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:15 smithi055 ceph-mon[97033]: pgmap v1729: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:34:16.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:34:16.167+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:34:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:17 smithi059 ceph-mon[97450]: pgmap v1730: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:34:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:17 smithi055 ceph-mon[97033]: pgmap v1730: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:34:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:19 smithi055 ceph-mon[97033]: pgmap v1731: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:34:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:19 smithi055 ceph-mon[97033]: osdmap e201: 12 total, 11 up, 11 in 2022-09-07T22:34:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:19 smithi059 ceph-mon[97450]: pgmap v1731: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:34:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:19 smithi059 ceph-mon[97450]: osdmap e201: 12 total, 11 up, 11 in 2022-09-07T22:34:21.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:21 smithi055 ceph-mon[97033]: pgmap v1733: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:34:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:21 smithi059 ceph-mon[97450]: pgmap v1733: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:34:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:23 smithi055 ceph-mon[97033]: pgmap v1734: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:34:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:23 smithi059 ceph-mon[97450]: pgmap v1734: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:34:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:34:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:34:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:34:24.416 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:34:24.417 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:34:24.417 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:34:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:25 smithi055 ceph-mon[97033]: pgmap v1735: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:34:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:34:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:34:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:25 smithi059 ceph-mon[97450]: pgmap v1735: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:34:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:34:25.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:34:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:27 smithi059 ceph-mon[97450]: pgmap v1736: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:34:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:34:27.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:34:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:27 smithi055 ceph-mon[97033]: pgmap v1736: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:34:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:34:27.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:34:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:34:28.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:34:28.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:34:28.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:34:28.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:28.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:34:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:29 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T22:34:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:29 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T22:34:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:29 smithi055 ceph-mon[97033]: pgmap v1737: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:34:29.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:29 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T22:34:29.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:29 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T22:34:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:29 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T22:34:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:29 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T22:34:29.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:29 smithi059 ceph-mon[97450]: pgmap v1737: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:34:29.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:29 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T22:34:29.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:29 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T22:34:31.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:34:31.181+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:34:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:31 smithi055 ceph-mon[97033]: pgmap v1738: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:34:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:31 smithi059 ceph-mon[97450]: pgmap v1738: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:34:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:33 smithi055 ceph-mon[97033]: pgmap v1739: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:34:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:33 smithi059 ceph-mon[97450]: pgmap v1739: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:34:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:35 smithi055 ceph-mon[97033]: pgmap v1740: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:34:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:35 smithi059 ceph-mon[97450]: pgmap v1740: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:34:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:37 smithi055 ceph-mon[97033]: pgmap v1741: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:34:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:37 smithi059 ceph-mon[97450]: pgmap v1741: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:34:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:39 smithi055 ceph-mon[97033]: pgmap v1742: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:34:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:39 smithi059 ceph-mon[97450]: pgmap v1742: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:34:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:41 smithi055 ceph-mon[97033]: pgmap v1743: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:34:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:41 smithi059 ceph-mon[97450]: pgmap v1743: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:34:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:43 smithi055 ceph-mon[97033]: pgmap v1744: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:34:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:43 smithi059 ceph-mon[97450]: pgmap v1744: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:34:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:45 smithi055 ceph-mon[97033]: pgmap v1745: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:34:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:45 smithi059 ceph-mon[97450]: pgmap v1745: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:34:46.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:34:46.195+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:34:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:47 smithi055 ceph-mon[97033]: pgmap v1746: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:34:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:47 smithi059 ceph-mon[97450]: pgmap v1746: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:34:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:49 smithi055 ceph-mon[97033]: pgmap v1747: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:34:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:49 smithi059 ceph-mon[97450]: pgmap v1747: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:34:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:51 smithi055 ceph-mon[97033]: pgmap v1748: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:34:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:51 smithi059 ceph-mon[97450]: pgmap v1748: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:34:53.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:53 smithi055 ceph-mon[97033]: pgmap v1749: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:34:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:53 smithi059 ceph-mon[97450]: pgmap v1749: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:34:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:55 smithi059 ceph-mon[97450]: pgmap v1750: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:34:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:55 smithi059 ceph-mon[97450]: pgmap v1751: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:34:55.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:55 smithi059 ceph-mon[97450]: pgmap v1752: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:34:55.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:34:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:55 smithi055 ceph-mon[97033]: pgmap v1750: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:34:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:55 smithi055 ceph-mon[97033]: pgmap v1751: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:34:55.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:55 smithi055 ceph-mon[97033]: pgmap v1752: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:34:55.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:34:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:57 smithi059 ceph-mon[97450]: pgmap v1753: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:34:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:57 smithi055 ceph-mon[97033]: pgmap v1753: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:34:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:34:59 smithi055 ceph-mon[97033]: pgmap v1754: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:34:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:34:59 smithi059 ceph-mon[97450]: pgmap v1754: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:35:01.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:01 smithi055 ceph-mon[97033]: pgmap v1755: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:35:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:01 smithi059 ceph-mon[97450]: pgmap v1755: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:35:01.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:35:01.210+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:35:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:03 smithi055 ceph-mon[97033]: pgmap v1756: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:35:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:03 smithi059 ceph-mon[97450]: pgmap v1756: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:35:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:05 smithi055 ceph-mon[97033]: pgmap v1757: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:35:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:35:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:35:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:05 smithi059 ceph-mon[97450]: pgmap v1757: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:35:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:35:05.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:35:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:07 smithi055 ceph-mon[97033]: pgmap v1758: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:35:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:07 smithi059 ceph-mon[97450]: pgmap v1758: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:35:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:09 smithi055 ceph-mon[97033]: pgmap v1759: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:35:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:09 smithi059 ceph-mon[97450]: pgmap v1759: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:35:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:10 smithi055 ceph-mon[97033]: osdmap e202: 12 total, 11 up, 11 in 2022-09-07T22:35:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:10 smithi059 ceph-mon[97450]: osdmap e202: 12 total, 11 up, 11 in 2022-09-07T22:35:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:11 smithi055 ceph-mon[97033]: pgmap v1761: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:35:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:11 smithi059 ceph-mon[97450]: pgmap v1761: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:35:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:13 smithi055 ceph-mon[97033]: pgmap v1762: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:35:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:13 smithi055 ceph-mon[97033]: osdmap e203: 12 total, 11 up, 11 in 2022-09-07T22:35:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:13 smithi059 ceph-mon[97450]: pgmap v1762: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:35:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:13 smithi059 ceph-mon[97450]: osdmap e203: 12 total, 11 up, 11 in 2022-09-07T22:35:15.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:15 smithi055 ceph-mon[97033]: pgmap v1764: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:35:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:15 smithi059 ceph-mon[97450]: pgmap v1764: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:35:16.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:35:16.224+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:35:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:17 smithi055 ceph-mon[97033]: pgmap v1765: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:35:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:17 smithi059 ceph-mon[97450]: pgmap v1765: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:35:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:19 smithi055 ceph-mon[97033]: pgmap v1766: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-07T22:35:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:19 smithi059 ceph-mon[97450]: pgmap v1766: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-07T22:35:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:21 smithi055 ceph-mon[97033]: pgmap v1767: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:35:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:21 smithi059 ceph-mon[97450]: pgmap v1767: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:35:23.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:23 smithi055 ceph-mon[97033]: pgmap v1768: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:35:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:23 smithi059 ceph-mon[97450]: pgmap v1768: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:35:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:25 smithi055 ceph-mon[97033]: pgmap v1769: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 94 B/s wr, 5 op/s 2022-09-07T22:35:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:25 smithi059 ceph-mon[97450]: pgmap v1769: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 94 B/s wr, 5 op/s 2022-09-07T22:35:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:27 smithi055 ceph-mon[97033]: pgmap v1770: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:35:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:27 smithi059 ceph-mon[97450]: pgmap v1770: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:35:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:29 smithi055 ceph-mon[97033]: pgmap v1771: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:29 smithi059 ceph-mon[97450]: pgmap v1771: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:31.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:31 smithi055 ceph-mon[97033]: pgmap v1772: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:35:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:31 smithi059 ceph-mon[97450]: pgmap v1772: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:35:31.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:35:31.239+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:35:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:33 smithi055 ceph-mon[97033]: pgmap v1773: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:33 smithi059 ceph-mon[97450]: pgmap v1773: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:35 smithi059 ceph-mon[97450]: pgmap v1774: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:35 smithi055 ceph-mon[97033]: pgmap v1774: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:37 smithi059 ceph-mon[97450]: pgmap v1775: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:35:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:37 smithi055 ceph-mon[97033]: pgmap v1775: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:35:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:39 smithi059 ceph-mon[97450]: pgmap v1776: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:39 smithi055 ceph-mon[97033]: pgmap v1776: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:41 smithi059 ceph-mon[97450]: pgmap v1777: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:35:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:41 smithi055 ceph-mon[97033]: pgmap v1777: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:35:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:43 smithi059 ceph-mon[97450]: pgmap v1778: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:43 smithi055 ceph-mon[97033]: pgmap v1778: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:45.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:45 smithi059 ceph-mon[97450]: pgmap v1779: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:45 smithi055 ceph-mon[97033]: pgmap v1779: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:46.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:35:46.249+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:35:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:47 smithi059 ceph-mon[97450]: pgmap v1780: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:35:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:47 smithi055 ceph-mon[97033]: pgmap v1780: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:35:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:49 smithi059 ceph-mon[97450]: pgmap v1781: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:49 smithi055 ceph-mon[97033]: pgmap v1781: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:51 smithi059 ceph-mon[97450]: pgmap v1782: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:35:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:51 smithi055 ceph-mon[97033]: pgmap v1782: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:35:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:53 smithi059 ceph-mon[97450]: pgmap v1783: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:53 smithi055 ceph-mon[97033]: pgmap v1783: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:35:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:35:54.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:35:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:35:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:35:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:35:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:55 smithi059 ceph-mon[97450]: pgmap v1784: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:55.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:35:55.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:35:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:55 smithi055 ceph-mon[97033]: pgmap v1784: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:55.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:35:55.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:35:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:57 smithi059 ceph-mon[97450]: pgmap v1785: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:35:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:57 smithi055 ceph-mon[97033]: pgmap v1785: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:35:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:35:59 smithi059 ceph-mon[97450]: pgmap v1786: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:35:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:35:59 smithi055 ceph-mon[97033]: pgmap v1786: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:01 smithi059 ceph-mon[97450]: pgmap v1787: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:36:01.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:36:01.256+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:36:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:01 smithi055 ceph-mon[97033]: pgmap v1787: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:36:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:03 smithi055 ceph-mon[97033]: pgmap v1788: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:03 smithi059 ceph-mon[97450]: pgmap v1788: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:05 smithi055 ceph-mon[97033]: pgmap v1789: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:36:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:36:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:05 smithi059 ceph-mon[97450]: pgmap v1789: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:36:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:36:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:07 smithi055 ceph-mon[97033]: pgmap v1790: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:36:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:07 smithi059 ceph-mon[97450]: pgmap v1790: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:36:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:09 smithi055 ceph-mon[97033]: pgmap v1791: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:36:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:09 smithi055 ceph-mon[97033]: osdmap e204: 12 total, 11 up, 11 in 2022-09-07T22:36:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:09 smithi059 ceph-mon[97450]: pgmap v1791: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:36:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:09 smithi059 ceph-mon[97450]: osdmap e204: 12 total, 11 up, 11 in 2022-09-07T22:36:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:11 smithi055 ceph-mon[97033]: pgmap v1793: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:36:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:11 smithi059 ceph-mon[97450]: pgmap v1793: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:36:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:13 smithi055 ceph-mon[97033]: pgmap v1794: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:36:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:13 smithi059 ceph-mon[97450]: pgmap v1794: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:36:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:15 smithi055 ceph-mon[97033]: pgmap v1795: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:36:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:15 smithi059 ceph-mon[97450]: pgmap v1795: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:36:16.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:36:16.272+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:36:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:17 smithi055 ceph-mon[97033]: pgmap v1796: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:36:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:17 smithi059 ceph-mon[97450]: pgmap v1796: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:36:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:19 smithi055 ceph-mon[97033]: pgmap v1797: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:36:19.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:19 smithi055 ceph-mon[97033]: osdmap e205: 12 total, 11 up, 11 in 2022-09-07T22:36:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:19 smithi059 ceph-mon[97450]: pgmap v1797: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:36:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:19 smithi059 ceph-mon[97450]: osdmap e205: 12 total, 11 up, 11 in 2022-09-07T22:36:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:21 smithi055 ceph-mon[97033]: pgmap v1799: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:36:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:21 smithi059 ceph-mon[97450]: pgmap v1799: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:36:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:23 smithi055 ceph-mon[97033]: pgmap v1800: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:36:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:23 smithi059 ceph-mon[97450]: pgmap v1800: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:36:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:25 smithi059 ceph-mon[97450]: pgmap v1801: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:36:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:25 smithi059 ceph-mon[97450]: pgmap v1802: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:36:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:25 smithi059 ceph-mon[97450]: pgmap v1803: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:36:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:36:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:25 smithi055 ceph-mon[97033]: pgmap v1801: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:36:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:25 smithi055 ceph-mon[97033]: pgmap v1802: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:36:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:25 smithi055 ceph-mon[97033]: pgmap v1803: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:36:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:36:27.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:27 smithi055 ceph-mon[97033]: pgmap v1804: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:36:27.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:27 smithi059 ceph-mon[97450]: pgmap v1804: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:36:29.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:29 smithi055 ceph-mon[97033]: pgmap v1805: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:36:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:29 smithi059 ceph-mon[97450]: pgmap v1805: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:36:31.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:36:31.282+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:36:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:31 smithi055 ceph-mon[97033]: pgmap v1806: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:36:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:31 smithi059 ceph-mon[97450]: pgmap v1806: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:36:33.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:33 smithi055 ceph-mon[97033]: pgmap v1807: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:36:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:33 smithi059 ceph-mon[97450]: pgmap v1807: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:36:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:35 smithi055 ceph-mon[97033]: pgmap v1808: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:36:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:35 smithi059 ceph-mon[97450]: pgmap v1808: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:36:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:37 smithi055 ceph-mon[97033]: pgmap v1809: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:36:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:37 smithi059 ceph-mon[97450]: pgmap v1809: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:36:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:39 smithi055 ceph-mon[97033]: pgmap v1810: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:39 smithi059 ceph-mon[97450]: pgmap v1810: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:41 smithi055 ceph-mon[97033]: pgmap v1811: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:36:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:41 smithi059 ceph-mon[97450]: pgmap v1811: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:36:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:43 smithi055 ceph-mon[97033]: pgmap v1812: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:43 smithi059 ceph-mon[97450]: pgmap v1812: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:45 smithi055 ceph-mon[97033]: pgmap v1813: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:45 smithi059 ceph-mon[97450]: pgmap v1813: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:46.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:36:46.297+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:36:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:47 smithi055 ceph-mon[97033]: pgmap v1814: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:36:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:47 smithi059 ceph-mon[97450]: pgmap v1814: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:36:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:49 smithi055 ceph-mon[97033]: pgmap v1815: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:49 smithi059 ceph-mon[97450]: pgmap v1815: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:51 smithi055 ceph-mon[97033]: pgmap v1816: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:36:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:51 smithi059 ceph-mon[97450]: pgmap v1816: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:36:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:53 smithi055 ceph-mon[97033]: pgmap v1817: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:53 smithi059 ceph-mon[97450]: pgmap v1817: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:55 smithi055 ceph-mon[97033]: pgmap v1818: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:55 smithi059 ceph-mon[97450]: pgmap v1818: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:57 smithi055 ceph-mon[97033]: pgmap v1819: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:36:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:57 smithi059 ceph-mon[97450]: pgmap v1819: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:36:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:36:59 smithi055 ceph-mon[97033]: pgmap v1820: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:36:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:36:59 smithi059 ceph-mon[97450]: pgmap v1820: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:01.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:37:01.313+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:37:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:01 smithi055 ceph-mon[97033]: pgmap v1821: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:37:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:01 smithi059 ceph-mon[97450]: pgmap v1821: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:37:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:03 smithi055 ceph-mon[97033]: pgmap v1822: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:03 smithi059 ceph-mon[97450]: pgmap v1822: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:37:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:37:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:37:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:37:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:05 smithi055 ceph-mon[97033]: pgmap v1823: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:05 smithi059 ceph-mon[97450]: pgmap v1823: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:07 smithi055 ceph-mon[97033]: pgmap v1824: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:37:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:07 smithi059 ceph-mon[97450]: pgmap v1824: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:37:09.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:09 smithi055 ceph-mon[97033]: pgmap v1825: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:37:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:09 smithi059 ceph-mon[97450]: pgmap v1825: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:37:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:10 smithi055 ceph-mon[97033]: osdmap e206: 12 total, 11 up, 11 in 2022-09-07T22:37:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:10 smithi059 ceph-mon[97450]: osdmap e206: 12 total, 11 up, 11 in 2022-09-07T22:37:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:11 smithi055 ceph-mon[97033]: pgmap v1827: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:37:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:11 smithi059 ceph-mon[97450]: pgmap v1827: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:37:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:13 smithi055 ceph-mon[97033]: pgmap v1828: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:37:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:13 smithi055 ceph-mon[97033]: osdmap e207: 12 total, 11 up, 11 in 2022-09-07T22:37:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:13 smithi059 ceph-mon[97450]: pgmap v1828: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:37:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:13 smithi059 ceph-mon[97450]: osdmap e207: 12 total, 11 up, 11 in 2022-09-07T22:37:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:15 smithi055 ceph-mon[97033]: pgmap v1830: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:37:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:15 smithi059 ceph-mon[97450]: pgmap v1830: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:37:16.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:37:16.322+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:37:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:17 smithi055 ceph-mon[97033]: pgmap v1831: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:37:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:17 smithi059 ceph-mon[97450]: pgmap v1831: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:37:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:19 smithi055 ceph-mon[97033]: pgmap v1832: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 225 B/s wr, 7 op/s 2022-09-07T22:37:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:19 smithi059 ceph-mon[97450]: pgmap v1832: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 225 B/s wr, 7 op/s 2022-09-07T22:37:21.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:21 smithi055 ceph-mon[97033]: pgmap v1833: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:37:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:21 smithi059 ceph-mon[97450]: pgmap v1833: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:37:23.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:23 smithi055 ceph-mon[97033]: pgmap v1834: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:37:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:23 smithi059 ceph-mon[97450]: pgmap v1834: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:37:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:25 smithi055 ceph-mon[97033]: pgmap v1835: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T22:37:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:37:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:37:25.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:37:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:25 smithi059 ceph-mon[97450]: pgmap v1835: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T22:37:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:37:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:37:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:37:27.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:27 smithi055 ceph-mon[97033]: pgmap v1836: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:37:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:27 smithi059 ceph-mon[97450]: pgmap v1836: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:37:29.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:29 smithi055 ceph-mon[97033]: pgmap v1837: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:29 smithi059 ceph-mon[97450]: pgmap v1837: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:31.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:37:31.338+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:37:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:31 smithi055 ceph-mon[97033]: pgmap v1838: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:37:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:31 smithi059 ceph-mon[97450]: pgmap v1838: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:37:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:33 smithi055 ceph-mon[97033]: pgmap v1839: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:33 smithi059 ceph-mon[97450]: pgmap v1839: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:35 smithi055 ceph-mon[97033]: pgmap v1840: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:35 smithi059 ceph-mon[97450]: pgmap v1840: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:37 smithi055 ceph-mon[97033]: pgmap v1841: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:37:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:37 smithi059 ceph-mon[97450]: pgmap v1841: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:37:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:39 smithi055 ceph-mon[97033]: pgmap v1842: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:39 smithi059 ceph-mon[97450]: pgmap v1842: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:41 smithi055 ceph-mon[97033]: pgmap v1843: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:37:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:41 smithi059 ceph-mon[97450]: pgmap v1843: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:37:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:43 smithi055 ceph-mon[97033]: pgmap v1844: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:43 smithi059 ceph-mon[97450]: pgmap v1844: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:45 smithi055 ceph-mon[97033]: pgmap v1845: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:45 smithi059 ceph-mon[97450]: pgmap v1845: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:46.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:37:46.341+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:37:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:47 smithi055 ceph-mon[97033]: pgmap v1846: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:37:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:47 smithi059 ceph-mon[97450]: pgmap v1846: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:37:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:49 smithi055 ceph-mon[97033]: pgmap v1847: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:49 smithi059 ceph-mon[97450]: pgmap v1847: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:51 smithi055 ceph-mon[97033]: pgmap v1848: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:37:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:51 smithi059 ceph-mon[97450]: pgmap v1848: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:37:53.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:53 smithi055 ceph-mon[97033]: pgmap v1849: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:53 smithi059 ceph-mon[97450]: pgmap v1849: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:55 smithi055 ceph-mon[97033]: pgmap v1850: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:55 smithi055 ceph-mon[97033]: pgmap v1851: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:37:55.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:37:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:55 smithi059 ceph-mon[97450]: pgmap v1850: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:37:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:55 smithi059 ceph-mon[97450]: pgmap v1851: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:37:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:37:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:56 smithi055 ceph-mon[97033]: pgmap v1852: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:37:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:56 smithi059 ceph-mon[97450]: pgmap v1852: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:37:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:37:59 smithi055 ceph-mon[97033]: pgmap v1853: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:37:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:37:59 smithi059 ceph-mon[97450]: pgmap v1853: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:38:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:00 smithi055 ceph-mon[97033]: pgmap v1854: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:38:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:00 smithi059 ceph-mon[97450]: pgmap v1854: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:38:01.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:38:01.348+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:38:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:02 smithi055 ceph-mon[97033]: pgmap v1855: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:38:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:02 smithi059 ceph-mon[97450]: pgmap v1855: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:38:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:38:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:38:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:38:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:38:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:05 smithi055 ceph-mon[97033]: pgmap v1856: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:38:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:05 smithi059 ceph-mon[97450]: pgmap v1856: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:38:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:06 smithi055 ceph-mon[97033]: pgmap v1857: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:38:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:06 smithi059 ceph-mon[97450]: pgmap v1857: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:38:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:09 smithi055 ceph-mon[97033]: pgmap v1858: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:38:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:09 smithi059 ceph-mon[97450]: pgmap v1858: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:38:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:10 smithi055 ceph-mon[97033]: osdmap e208: 12 total, 11 up, 11 in 2022-09-07T22:38:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:10 smithi059 ceph-mon[97450]: osdmap e208: 12 total, 11 up, 11 in 2022-09-07T22:38:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:11 smithi055 ceph-mon[97033]: pgmap v1860: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:38:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:11 smithi059 ceph-mon[97450]: pgmap v1860: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:38:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:12 smithi055 ceph-mon[97033]: pgmap v1861: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:38:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:12 smithi059 ceph-mon[97450]: pgmap v1861: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:38:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:14 smithi055 ceph-mon[97033]: osdmap e209: 12 total, 11 up, 11 in 2022-09-07T22:38:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:14 smithi059 ceph-mon[97450]: osdmap e209: 12 total, 11 up, 11 in 2022-09-07T22:38:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:15 smithi055 ceph-mon[97033]: pgmap v1863: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:38:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:15 smithi059 ceph-mon[97450]: pgmap v1863: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:38:16.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:38:16.353+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:38:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:16 smithi055 ceph-mon[97033]: pgmap v1864: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:38:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:16 smithi059 ceph-mon[97450]: pgmap v1864: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:38:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:19 smithi055 ceph-mon[97033]: pgmap v1865: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-07T22:38:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:19 smithi059 ceph-mon[97450]: pgmap v1865: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-07T22:38:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:20 smithi055 ceph-mon[97033]: pgmap v1866: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2022-09-07T22:38:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:20 smithi059 ceph-mon[97450]: pgmap v1866: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2022-09-07T22:38:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:22 smithi055 ceph-mon[97033]: pgmap v1867: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2022-09-07T22:38:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:22 smithi059 ceph-mon[97450]: pgmap v1867: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2022-09-07T22:38:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:24 smithi055 ceph-mon[97033]: pgmap v1868: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 270 B/s wr, 5 op/s 2022-09-07T22:38:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:24 smithi059 ceph-mon[97450]: pgmap v1868: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 270 B/s wr, 5 op/s 2022-09-07T22:38:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:26 smithi055 ceph-mon[97033]: pgmap v1869: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:38:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:26 smithi059 ceph-mon[97450]: pgmap v1869: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:38:29.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:29 smithi055 ceph-mon[97033]: pgmap v1870: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2022-09-07T22:38:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:29 smithi059 ceph-mon[97450]: pgmap v1870: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2022-09-07T22:38:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:30 smithi055 ceph-mon[97033]: pgmap v1871: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T22:38:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:30 smithi059 ceph-mon[97450]: pgmap v1871: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 170 B/s wr, 7 op/s 2022-09-07T22:38:31.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:38:31.364+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:38:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:34 smithi055 ceph-mon[97033]: pgmap v1872: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:34 smithi059 ceph-mon[97450]: pgmap v1872: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:36.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:36 smithi055 ceph-mon[97033]: pgmap v1873: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:36 smithi059 ceph-mon[97450]: pgmap v1873: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:38 smithi055 ceph-mon[97033]: pgmap v1874: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:38:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:38 smithi059 ceph-mon[97450]: pgmap v1874: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:38:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:40 smithi055 ceph-mon[97033]: pgmap v1875: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:40 smithi059 ceph-mon[97450]: pgmap v1875: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:42 smithi059 ceph-mon[97450]: pgmap v1876: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:38:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:42 smithi055 ceph-mon[97033]: pgmap v1876: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:38:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:44 smithi059 ceph-mon[97450]: pgmap v1877: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:44 smithi055 ceph-mon[97033]: pgmap v1877: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:46 smithi059 ceph-mon[97450]: pgmap v1878: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:46.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:38:46.370+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:38:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:46 smithi055 ceph-mon[97033]: pgmap v1878: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:48 smithi059 ceph-mon[97450]: pgmap v1879: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:38:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:48 smithi055 ceph-mon[97033]: pgmap v1879: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:38:50.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:50 smithi059 ceph-mon[97450]: pgmap v1880: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:50 smithi055 ceph-mon[97033]: pgmap v1880: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:52 smithi059 ceph-mon[97450]: pgmap v1881: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:38:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:52 smithi055 ceph-mon[97033]: pgmap v1881: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:38:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:54 smithi059 ceph-mon[97450]: pgmap v1882: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:54.723 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:54 smithi055 ceph-mon[97033]: pgmap v1882: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:38:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:38:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:38:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:38:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:38:55.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:38:56.369 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:56 smithi059 ceph-mon[97450]: pgmap v1883: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:56.433 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:56 smithi055 ceph-mon[97033]: pgmap v1883: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:38:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:58 smithi055 ceph-mon[97033]: pgmap v1884: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:38:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:58 smithi059 ceph-mon[97450]: pgmap v1884: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:38:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:59 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-07T22:38:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:38:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:38:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:59 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-07T22:38:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:38:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:38:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:38:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:59 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-07T22:38:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:38:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:38:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:59 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-07T22:38:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:38:59.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:38:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:39:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:00 smithi055 ceph-mon[97033]: pgmap v1885: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:00 smithi059 ceph-mon[97450]: pgmap v1885: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:01.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:39:01.379+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:39:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:02 smithi055 ceph-mon[97033]: pgmap v1886: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:39:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:02 smithi059 ceph-mon[97450]: pgmap v1886: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:39:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:04 smithi055 ceph-mon[97033]: pgmap v1887: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:39:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:39:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:04 smithi059 ceph-mon[97450]: pgmap v1887: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:39:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:39:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:06 smithi055 ceph-mon[97033]: pgmap v1888: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:06 smithi059 ceph-mon[97450]: pgmap v1888: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:08 smithi055 ceph-mon[97033]: pgmap v1889: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:39:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:08 smithi059 ceph-mon[97450]: pgmap v1889: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:39:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:09 smithi055 ceph-mon[97033]: osdmap e210: 12 total, 11 up, 11 in 2022-09-07T22:39:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:09 smithi059 ceph-mon[97450]: osdmap e210: 12 total, 11 up, 11 in 2022-09-07T22:39:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:10 smithi055 ceph-mon[97033]: pgmap v1891: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:39:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:10 smithi059 ceph-mon[97450]: pgmap v1891: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:39:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:12 smithi055 ceph-mon[97033]: pgmap v1892: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:39:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:12 smithi059 ceph-mon[97450]: pgmap v1892: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:39:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:14 smithi055 ceph-mon[97033]: pgmap v1893: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:39:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:14 smithi055 ceph-mon[97033]: osdmap e211: 12 total, 11 up, 11 in 2022-09-07T22:39:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:14 smithi059 ceph-mon[97450]: pgmap v1893: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:39:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:14 smithi059 ceph-mon[97450]: osdmap e211: 12 total, 11 up, 11 in 2022-09-07T22:39:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:16 smithi055 ceph-mon[97033]: pgmap v1895: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:39:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:16 smithi059 ceph-mon[97450]: pgmap v1895: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:39:16.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:39:16.395+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:39:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:18 smithi055 ceph-mon[97033]: pgmap v1896: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 251 B/s wr, 7 op/s 2022-09-07T22:39:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:18 smithi059 ceph-mon[97450]: pgmap v1896: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 251 B/s wr, 7 op/s 2022-09-07T22:39:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:20 smithi059 ceph-mon[97450]: pgmap v1897: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:39:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:20 smithi055 ceph-mon[97033]: pgmap v1897: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:39:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:22 smithi059 ceph-mon[97450]: pgmap v1898: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:39:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:22 smithi055 ceph-mon[97033]: pgmap v1898: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:39:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:24 smithi059 ceph-mon[97450]: pgmap v1899: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:39:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:24 smithi055 ceph-mon[97033]: pgmap v1899: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:39:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:25 smithi055 ceph-mon[97033]: pgmap v1900: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:39:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:25 smithi055 ceph-mon[97033]: pgmap v1901: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-07T22:39:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:25 smithi055 ceph-mon[97033]: pgmap v1902: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:39:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:39:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:25 smithi059 ceph-mon[97450]: pgmap v1900: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:39:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:25 smithi059 ceph-mon[97450]: pgmap v1901: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-07T22:39:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:25 smithi059 ceph-mon[97450]: pgmap v1902: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:39:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:39:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:27 smithi055 ceph-mon[97033]: pgmap v1903: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:39:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:27 smithi059 ceph-mon[97450]: pgmap v1903: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:39:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:30 smithi055 ceph-mon[97033]: pgmap v1904: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:39:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:30 smithi059 ceph-mon[97450]: pgmap v1904: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:39:31.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:39:31.410+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:39:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:32 smithi059 ceph-mon[97450]: pgmap v1905: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:39:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:32 smithi055 ceph-mon[97033]: pgmap v1905: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:39:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:34 smithi059 ceph-mon[97450]: pgmap v1906: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:39:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:34 smithi055 ceph-mon[97033]: pgmap v1906: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:39:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:36 smithi059 ceph-mon[97450]: pgmap v1907: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:39:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:36 smithi055 ceph-mon[97033]: pgmap v1907: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:39:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:38 smithi059 ceph-mon[97450]: pgmap v1908: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:39:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:38 smithi055 ceph-mon[97033]: pgmap v1908: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:39:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:40 smithi059 ceph-mon[97450]: pgmap v1909: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:40 smithi055 ceph-mon[97033]: pgmap v1909: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:42 smithi059 ceph-mon[97450]: pgmap v1910: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:39:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:42 smithi055 ceph-mon[97033]: pgmap v1910: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:39:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:44 smithi059 ceph-mon[97450]: pgmap v1911: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:44 smithi055 ceph-mon[97033]: pgmap v1911: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:46 smithi059 ceph-mon[97450]: pgmap v1912: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:46 smithi055 ceph-mon[97033]: pgmap v1912: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:46.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:39:46.418+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:39:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:48 smithi059 ceph-mon[97450]: pgmap v1913: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:39:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:48 smithi055 ceph-mon[97033]: pgmap v1913: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:39:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:50 smithi059 ceph-mon[97450]: pgmap v1914: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:50 smithi055 ceph-mon[97033]: pgmap v1914: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:52.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:52 smithi059 ceph-mon[97450]: pgmap v1915: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:39:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:52 smithi055 ceph-mon[97033]: pgmap v1915: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:39:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:54 smithi059 ceph-mon[97450]: pgmap v1916: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:54 smithi055 ceph-mon[97033]: pgmap v1916: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:56 smithi059 ceph-mon[97450]: pgmap v1917: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:56 smithi055 ceph-mon[97033]: pgmap v1917: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:39:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:39:58 smithi055 ceph-mon[97033]: pgmap v1918: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:39:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:39:58 smithi059 ceph-mon[97450]: pgmap v1918: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:40:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:00 smithi055 ceph-mon[97033]: pgmap v1919: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-07T22:40:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-07T22:40:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-07T22:40:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T22:40:00.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T22:40:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:00 smithi059 ceph-mon[97450]: pgmap v1919: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-07T22:40:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-07T22:40:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-07T22:40:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T22:40:00.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T22:40:01.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:40:01.434+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:40:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:02 smithi055 ceph-mon[97033]: pgmap v1920: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:40:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:02 smithi059 ceph-mon[97450]: pgmap v1920: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:40:04.547 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:04 smithi055 ceph-mon[97033]: pgmap v1921: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:04 smithi059 ceph-mon[97450]: pgmap v1921: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:40:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:40:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:40:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:40:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:06 smithi055 ceph-mon[97033]: pgmap v1922: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 21 KiB/s wr, 17 op/s 2022-09-07T22:40:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:06 smithi059 ceph-mon[97450]: pgmap v1922: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 21 KiB/s wr, 17 op/s 2022-09-07T22:40:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:08 smithi055 ceph-mon[97033]: pgmap v1923: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:40:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:08 smithi059 ceph-mon[97450]: pgmap v1923: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:40:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:09 smithi055 ceph-mon[97033]: osdmap e212: 12 total, 11 up, 11 in 2022-09-07T22:40:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:09 smithi059 ceph-mon[97450]: osdmap e212: 12 total, 11 up, 11 in 2022-09-07T22:40:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:10 smithi055 ceph-mon[97033]: pgmap v1925: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:40:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:10 smithi059 ceph-mon[97450]: pgmap v1925: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:40:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:12 smithi055 ceph-mon[97033]: pgmap v1926: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:40:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:12 smithi059 ceph-mon[97450]: pgmap v1926: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:40:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:14 smithi059 ceph-mon[97450]: pgmap v1927: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:40:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:14 smithi059 ceph-mon[97450]: osdmap e213: 12 total, 11 up, 11 in 2022-09-07T22:40:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:14 smithi055 ceph-mon[97033]: pgmap v1927: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:40:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:14 smithi055 ceph-mon[97033]: osdmap e213: 12 total, 11 up, 11 in 2022-09-07T22:40:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:16 smithi059 ceph-mon[97450]: pgmap v1929: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 127 B/s wr, 5 op/s 2022-09-07T22:40:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:16 smithi055 ceph-mon[97033]: pgmap v1929: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 127 B/s wr, 5 op/s 2022-09-07T22:40:16.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:40:16.444+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:40:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:18 smithi059 ceph-mon[97450]: pgmap v1930: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 240 B/s wr, 7 op/s 2022-09-07T22:40:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:18 smithi055 ceph-mon[97033]: pgmap v1930: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 240 B/s wr, 7 op/s 2022-09-07T22:40:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:20 smithi059 ceph-mon[97450]: pgmap v1931: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:40:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:20 smithi055 ceph-mon[97033]: pgmap v1931: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:40:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:22 smithi059 ceph-mon[97450]: pgmap v1932: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:40:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:22 smithi055 ceph-mon[97033]: pgmap v1932: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:40:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:24 smithi059 ceph-mon[97450]: pgmap v1933: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:40:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:24 smithi055 ceph-mon[97033]: pgmap v1933: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:40:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:40:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:40:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:40:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:40:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:40:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:40:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:26 smithi059 ceph-mon[97450]: pgmap v1934: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2022-09-07T22:40:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:26 smithi055 ceph-mon[97033]: pgmap v1934: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2022-09-07T22:40:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:28 smithi059 ceph-mon[97450]: pgmap v1935: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2022-09-07T22:40:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:28 smithi055 ceph-mon[97033]: pgmap v1935: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2022-09-07T22:40:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:30 smithi059 ceph-mon[97450]: pgmap v1936: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:30 smithi055 ceph-mon[97033]: pgmap v1936: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:31.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:40:31.458+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:40:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:32 smithi059 ceph-mon[97450]: pgmap v1937: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:40:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:32 smithi055 ceph-mon[97033]: pgmap v1937: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:40:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:34 smithi055 ceph-mon[97033]: pgmap v1938: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:34 smithi059 ceph-mon[97450]: pgmap v1938: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:36 smithi055 ceph-mon[97033]: pgmap v1939: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:40:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:36 smithi059 ceph-mon[97450]: pgmap v1939: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:40:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:38 smithi055 ceph-mon[97033]: pgmap v1940: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:40:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:38 smithi059 ceph-mon[97450]: pgmap v1940: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:40:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:40 smithi055 ceph-mon[97033]: pgmap v1941: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:40 smithi059 ceph-mon[97450]: pgmap v1941: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:42 smithi055 ceph-mon[97033]: pgmap v1942: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:40:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:42 smithi059 ceph-mon[97450]: pgmap v1942: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:40:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:44 smithi055 ceph-mon[97033]: pgmap v1943: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:44 smithi059 ceph-mon[97450]: pgmap v1943: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:46 smithi055 ceph-mon[97033]: pgmap v1944: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:40:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:46 smithi059 ceph-mon[97450]: pgmap v1944: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:40:46.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:40:46.463+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:40:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:48 smithi055 ceph-mon[97033]: pgmap v1945: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:40:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:48 smithi059 ceph-mon[97450]: pgmap v1945: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:40:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:50 smithi055 ceph-mon[97033]: pgmap v1946: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:50 smithi059 ceph-mon[97450]: pgmap v1946: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:52 smithi055 ceph-mon[97033]: pgmap v1947: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:40:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:52 smithi059 ceph-mon[97450]: pgmap v1947: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:40:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:54 smithi055 ceph-mon[97033]: pgmap v1948: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:54 smithi059 ceph-mon[97450]: pgmap v1948: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:40:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:56 smithi055 ceph-mon[97033]: pgmap v1949: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:40:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:56 smithi059 ceph-mon[97450]: pgmap v1949: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T22:40:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:40:58 smithi055 ceph-mon[97033]: pgmap v1950: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:40:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:40:58 smithi059 ceph-mon[97450]: pgmap v1950: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:41:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:00 smithi059 ceph-mon[97450]: pgmap v1951: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:41:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:41:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:00 smithi059 ceph-mon[97450]: pgmap v1952: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:41:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:00 smithi059 ceph-mon[97450]: pgmap v1953: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-07T22:41:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:41:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:00 smithi055 ceph-mon[97033]: pgmap v1951: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:41:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:41:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:00 smithi055 ceph-mon[97033]: pgmap v1952: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T22:41:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:00 smithi055 ceph-mon[97033]: pgmap v1953: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-07T22:41:00.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:41:01.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:41:01.470+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:41:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:02 smithi055 ceph-mon[97033]: pgmap v1954: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-07T22:41:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:02 smithi059 ceph-mon[97450]: pgmap v1954: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-07T22:41:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:04 smithi059 ceph-mon[97450]: pgmap v1955: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.7 KiB/s rd, 4 op/s 2022-09-07T22:41:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:04 smithi055 ceph-mon[97033]: pgmap v1955: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.7 KiB/s rd, 4 op/s 2022-09-07T22:41:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:41:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:41:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:41:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:41:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:06 smithi059 ceph-mon[97450]: pgmap v1956: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-07T22:41:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:06 smithi055 ceph-mon[97033]: pgmap v1956: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-07T22:41:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:08 smithi055 ceph-mon[97033]: pgmap v1957: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-07T22:41:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:08 smithi059 ceph-mon[97450]: pgmap v1957: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-07T22:41:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:09 smithi055 ceph-mon[97033]: osdmap e214: 12 total, 11 up, 11 in 2022-09-07T22:41:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:09 smithi055 ceph-mon[97033]: pgmap v1959: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:41:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:09 smithi059 ceph-mon[97450]: osdmap e214: 12 total, 11 up, 11 in 2022-09-07T22:41:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:09 smithi059 ceph-mon[97450]: pgmap v1959: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:41:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:12 smithi055 ceph-mon[97033]: pgmap v1960: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:41:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:12 smithi059 ceph-mon[97450]: pgmap v1960: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:41:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:14 smithi059 ceph-mon[97450]: osdmap e215: 12 total, 11 up, 11 in 2022-09-07T22:41:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:14 smithi059 ceph-mon[97450]: pgmap v1962: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:41:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:14 smithi055 ceph-mon[97033]: osdmap e215: 12 total, 11 up, 11 in 2022-09-07T22:41:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:14 smithi055 ceph-mon[97033]: pgmap v1962: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:41:16.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:41:16.471+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:41:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:16 smithi055 ceph-mon[97033]: pgmap v1963: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:41:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:16 smithi059 ceph-mon[97450]: pgmap v1963: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:41:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:17 smithi055 ceph-mon[97033]: pgmap v1964: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-07T22:41:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:17 smithi059 ceph-mon[97450]: pgmap v1964: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-07T22:41:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:20 smithi055 ceph-mon[97033]: pgmap v1965: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:41:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:20 smithi059 ceph-mon[97450]: pgmap v1965: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:41:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:21 smithi055 ceph-mon[97033]: pgmap v1966: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:41:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:21 smithi059 ceph-mon[97450]: pgmap v1966: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:41:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:24 smithi059 ceph-mon[97450]: pgmap v1967: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-07T22:41:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:24 smithi055 ceph-mon[97033]: pgmap v1967: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-07T22:41:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:25 smithi055 ceph-mon[97033]: pgmap v1968: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:41:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:25 smithi059 ceph-mon[97450]: pgmap v1968: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:41:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:27 smithi055 ceph-mon[97033]: pgmap v1969: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:27 smithi059 ceph-mon[97450]: pgmap v1969: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:29 smithi055 ceph-mon[97033]: pgmap v1970: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:29 smithi059 ceph-mon[97450]: pgmap v1970: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:31.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:41:31.482+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:41:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:31 smithi055 ceph-mon[97033]: pgmap v1971: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:41:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:31 smithi059 ceph-mon[97450]: pgmap v1971: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:41:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:34 smithi059 ceph-mon[97450]: pgmap v1972: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:34 smithi055 ceph-mon[97033]: pgmap v1972: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:35 smithi055 ceph-mon[97033]: pgmap v1973: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:41:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:35 smithi059 ceph-mon[97450]: pgmap v1973: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:41:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:37 smithi055 ceph-mon[97033]: pgmap v1974: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:37 smithi059 ceph-mon[97450]: pgmap v1974: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:39 smithi055 ceph-mon[97033]: pgmap v1975: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:39 smithi059 ceph-mon[97450]: pgmap v1975: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:41 smithi055 ceph-mon[97033]: pgmap v1976: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:41:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:41 smithi059 ceph-mon[97450]: pgmap v1976: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:41:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:44 smithi059 ceph-mon[97450]: pgmap v1977: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:44 smithi055 ceph-mon[97033]: pgmap v1977: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:45 smithi055 ceph-mon[97033]: pgmap v1978: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:41:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:45 smithi059 ceph-mon[97450]: pgmap v1978: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:41:46.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:41:46.487+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:41:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:47 smithi055 ceph-mon[97033]: pgmap v1979: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:47 smithi059 ceph-mon[97450]: pgmap v1979: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:49 smithi055 ceph-mon[97033]: pgmap v1980: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:49 smithi059 ceph-mon[97450]: pgmap v1980: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:51 smithi055 ceph-mon[97033]: pgmap v1981: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:41:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:51 smithi059 ceph-mon[97450]: pgmap v1981: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:41:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:54 smithi059 ceph-mon[97450]: pgmap v1982: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:54 smithi055 ceph-mon[97033]: pgmap v1982: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:41:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:55 smithi055 ceph-mon[97033]: pgmap v1983: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:41:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:55 smithi059 ceph-mon[97450]: pgmap v1983: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:41:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:57 smithi055 ceph-mon[97033]: pgmap v1984: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:41:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:57 smithi059 ceph-mon[97450]: pgmap v1984: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:41:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:41:59 smithi055 ceph-mon[97033]: pgmap v1985: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:42:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:41:59 smithi059 ceph-mon[97450]: pgmap v1985: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:42:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:42:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:42:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:42:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:42:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:42:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:42:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:42:00.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:42:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:42:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:42:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:42:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:42:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:42:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:42:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:42:01.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:42:01.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:42:01.499+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:42:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:01 smithi055 ceph-mon[97033]: pgmap v1986: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-07T22:42:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:01 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-07T22:42:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:01 smithi059 ceph-mon[97450]: pgmap v1986: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-07T22:42:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:01 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-07T22:42:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:04 smithi059 ceph-mon[97450]: pgmap v1987: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:42:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:04 smithi055 ceph-mon[97033]: pgmap v1987: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:42:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:42:05.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:42:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:42:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:42:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:06 smithi059 ceph-mon[97450]: pgmap v1988: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:42:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:06 smithi055 ceph-mon[97033]: pgmap v1988: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T22:42:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:08 smithi055 ceph-mon[97033]: pgmap v1989: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 48 KiB/s wr, 21 op/s 2022-09-07T22:42:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:08 smithi059 ceph-mon[97450]: pgmap v1989: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 48 KiB/s wr, 21 op/s 2022-09-07T22:42:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:09 smithi055 ceph-mon[97033]: osdmap e216: 12 total, 11 up, 11 in 2022-09-07T22:42:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:09 smithi059 ceph-mon[97450]: osdmap e216: 12 total, 11 up, 11 in 2022-09-07T22:42:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:10 smithi055 ceph-mon[97033]: pgmap v1991: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 59 KiB/s wr, 26 op/s 2022-09-07T22:42:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:10 smithi059 ceph-mon[97450]: pgmap v1991: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 59 KiB/s wr, 26 op/s 2022-09-07T22:42:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:12 smithi055 ceph-mon[97033]: pgmap v1992: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:42:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:12 smithi059 ceph-mon[97450]: pgmap v1992: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:42:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:14 smithi059 ceph-mon[97450]: pgmap v1993: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:42:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:14 smithi059 ceph-mon[97450]: osdmap e217: 12 total, 11 up, 11 in 2022-09-07T22:42:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:14 smithi055 ceph-mon[97033]: pgmap v1993: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:42:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:14 smithi055 ceph-mon[97033]: osdmap e217: 12 total, 11 up, 11 in 2022-09-07T22:42:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:16 smithi059 ceph-mon[97450]: pgmap v1995: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:42:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:16 smithi055 ceph-mon[97033]: pgmap v1995: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:42:16.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:42:16.505+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:42:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:18 smithi059 ceph-mon[97450]: pgmap v1996: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-07T22:42:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:18 smithi055 ceph-mon[97033]: pgmap v1996: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-07T22:42:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:20 smithi055 ceph-mon[97033]: pgmap v1997: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:42:20.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:20 smithi059 ceph-mon[97450]: pgmap v1997: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:42:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:22 smithi055 ceph-mon[97033]: pgmap v1998: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:42:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:22 smithi059 ceph-mon[97450]: pgmap v1998: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:42:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:24 smithi055 ceph-mon[97033]: pgmap v1999: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:42:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:24 smithi059 ceph-mon[97450]: pgmap v1999: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:42:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:26 smithi055 ceph-mon[97033]: pgmap v2000: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:42:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:26 smithi059 ceph-mon[97450]: pgmap v2000: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:42:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:28 smithi055 ceph-mon[97033]: pgmap v2001: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:42:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:28 smithi059 ceph-mon[97450]: pgmap v2001: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:42:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:30 smithi055 ceph-mon[97033]: pgmap v2002: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:42:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:30 smithi059 ceph-mon[97450]: pgmap v2002: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:42:31.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:42:31.520+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:42:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:32 smithi055 ceph-mon[97033]: pgmap v2003: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:42:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:32 smithi059 ceph-mon[97450]: pgmap v2003: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:42:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:34 smithi055 ceph-mon[97033]: pgmap v2004: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:42:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:34 smithi059 ceph-mon[97450]: pgmap v2004: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:42:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:36 smithi055 ceph-mon[97033]: pgmap v2005: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:42:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:36 smithi059 ceph-mon[97450]: pgmap v2005: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:42:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:38 smithi055 ceph-mon[97033]: pgmap v2006: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:42:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:38 smithi059 ceph-mon[97450]: pgmap v2006: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:42:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:40 smithi055 ceph-mon[97033]: pgmap v2007: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:42:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:40 smithi059 ceph-mon[97450]: pgmap v2007: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:42:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:42 smithi055 ceph-mon[97033]: pgmap v2008: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:42:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:42 smithi059 ceph-mon[97450]: pgmap v2008: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:42:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:44 smithi055 ceph-mon[97033]: pgmap v2009: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:42:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:44 smithi059 ceph-mon[97450]: pgmap v2009: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:42:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:46 smithi055 ceph-mon[97033]: pgmap v2010: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:42:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:46 smithi059 ceph-mon[97450]: pgmap v2010: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:42:46.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:42:46.523+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:42:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:48 smithi055 ceph-mon[97033]: pgmap v2011: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:42:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:48 smithi059 ceph-mon[97450]: pgmap v2011: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:42:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:50 smithi055 ceph-mon[97033]: pgmap v2012: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:42:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:50 smithi059 ceph-mon[97450]: pgmap v2012: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:42:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:52 smithi055 ceph-mon[97033]: pgmap v2013: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:42:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:52 smithi059 ceph-mon[97450]: pgmap v2013: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:42:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:54 smithi055 ceph-mon[97033]: pgmap v2014: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:42:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:54 smithi059 ceph-mon[97450]: pgmap v2014: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:42:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:56 smithi055 ceph-mon[97033]: pgmap v2015: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:42:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:56 smithi059 ceph-mon[97450]: pgmap v2015: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:42:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:42:58 smithi055 ceph-mon[97033]: pgmap v2016: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:42:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:42:58 smithi059 ceph-mon[97450]: pgmap v2016: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:43:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:00 smithi055 ceph-mon[97033]: pgmap v2017: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:00.763 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:00 smithi059 ceph-mon[97450]: pgmap v2017: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:43:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:43:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:43:01.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:43:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:43:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:43:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:43:01.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:43:01.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:43:01.540+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:43:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:02 smithi055 ceph-mon[97033]: pgmap v2018: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:43:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:02 smithi059 ceph-mon[97450]: pgmap v2018: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:43:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:04 smithi055 ceph-mon[97033]: pgmap v2019: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:43:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:43:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:04 smithi059 ceph-mon[97450]: pgmap v2019: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:43:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:43:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:06 smithi055 ceph-mon[97033]: pgmap v2020: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:06 smithi059 ceph-mon[97450]: pgmap v2020: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:08 smithi055 ceph-mon[97033]: pgmap v2021: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:43:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:08 smithi059 ceph-mon[97450]: pgmap v2021: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:43:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:09 smithi055 ceph-mon[97033]: osdmap e218: 12 total, 11 up, 11 in 2022-09-07T22:43:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:09 smithi059 ceph-mon[97450]: osdmap e218: 12 total, 11 up, 11 in 2022-09-07T22:43:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:10 smithi055 ceph-mon[97033]: pgmap v2023: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:43:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:10 smithi059 ceph-mon[97450]: pgmap v2023: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:43:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:12 smithi055 ceph-mon[97033]: pgmap v2024: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:43:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:12 smithi059 ceph-mon[97450]: pgmap v2024: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:43:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:14 smithi055 ceph-mon[97033]: pgmap v2025: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:43:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:14 smithi055 ceph-mon[97033]: osdmap e219: 12 total, 11 up, 11 in 2022-09-07T22:43:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:14 smithi059 ceph-mon[97450]: pgmap v2025: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:43:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:14 smithi059 ceph-mon[97450]: osdmap e219: 12 total, 11 up, 11 in 2022-09-07T22:43:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:16 smithi055 ceph-mon[97033]: pgmap v2027: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:43:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:16 smithi059 ceph-mon[97450]: pgmap v2027: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:43:16.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:43:16.542+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:43:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:18 smithi055 ceph-mon[97033]: pgmap v2028: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 247 B/s wr, 7 op/s 2022-09-07T22:43:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:18 smithi059 ceph-mon[97450]: pgmap v2028: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 247 B/s wr, 7 op/s 2022-09-07T22:43:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:20 smithi055 ceph-mon[97033]: pgmap v2029: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:43:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:20 smithi059 ceph-mon[97450]: pgmap v2029: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:43:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:22 smithi055 ceph-mon[97033]: pgmap v2030: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:43:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:22 smithi059 ceph-mon[97450]: pgmap v2030: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:43:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:24 smithi055 ceph-mon[97033]: pgmap v2031: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:43:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:24 smithi059 ceph-mon[97450]: pgmap v2031: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:43:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:26 smithi055 ceph-mon[97033]: pgmap v2032: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:43:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:26 smithi059 ceph-mon[97450]: pgmap v2032: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:43:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:28 smithi055 ceph-mon[97033]: pgmap v2033: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:43:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:28 smithi059 ceph-mon[97450]: pgmap v2033: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:43:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:30 smithi055 ceph-mon[97033]: pgmap v2034: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:30 smithi059 ceph-mon[97450]: pgmap v2034: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:31.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:43:31.551+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:43:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:32 smithi055 ceph-mon[97033]: pgmap v2035: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:43:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:32 smithi059 ceph-mon[97450]: pgmap v2035: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:43:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:34 smithi055 ceph-mon[97033]: pgmap v2036: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:34 smithi059 ceph-mon[97450]: pgmap v2036: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:36 smithi055 ceph-mon[97033]: pgmap v2037: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:36.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:36 smithi059 ceph-mon[97450]: pgmap v2037: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:38 smithi055 ceph-mon[97033]: pgmap v2038: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:43:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:38 smithi059 ceph-mon[97450]: pgmap v2038: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:43:40.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:40 smithi055 ceph-mon[97033]: pgmap v2039: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:40 smithi059 ceph-mon[97450]: pgmap v2039: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:42 smithi055 ceph-mon[97033]: pgmap v2040: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:43:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:42 smithi059 ceph-mon[97450]: pgmap v2040: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:43:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:44 smithi055 ceph-mon[97033]: pgmap v2041: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:44 smithi059 ceph-mon[97450]: pgmap v2041: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:46 smithi055 ceph-mon[97033]: pgmap v2042: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:46 smithi059 ceph-mon[97450]: pgmap v2042: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:46.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:43:46.557+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:43:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:48 smithi055 ceph-mon[97033]: pgmap v2043: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:43:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:48 smithi059 ceph-mon[97450]: pgmap v2043: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:43:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:50 smithi055 ceph-mon[97033]: pgmap v2044: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:50 smithi059 ceph-mon[97450]: pgmap v2044: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:52 smithi055 ceph-mon[97033]: pgmap v2045: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:43:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:52 smithi059 ceph-mon[97450]: pgmap v2045: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:43:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:54 smithi055 ceph-mon[97033]: pgmap v2046: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:54 smithi059 ceph-mon[97450]: pgmap v2046: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:56 smithi055 ceph-mon[97033]: pgmap v2047: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:56 smithi059 ceph-mon[97450]: pgmap v2047: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:43:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:43:58 smithi055 ceph-mon[97033]: pgmap v2048: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:43:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:43:58 smithi059 ceph-mon[97450]: pgmap v2048: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:44:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:00 smithi055 ceph-mon[97033]: pgmap v2049: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:00 smithi059 ceph-mon[97450]: pgmap v2049: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:44:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:44:01.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:44:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:44:01.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:44:01.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:44:01.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:44:01.573+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:44:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:02 smithi055 ceph-mon[97033]: pgmap v2050: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:44:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:44:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:02 smithi059 ceph-mon[97450]: pgmap v2050: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:44:02.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:44:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:04 smithi055 ceph-mon[97033]: pgmap v2051: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:44:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:44:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:04 smithi059 ceph-mon[97450]: pgmap v2051: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:44:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:44:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:06 smithi055 ceph-mon[97033]: pgmap v2052: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:06 smithi059 ceph-mon[97450]: pgmap v2052: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:08 smithi055 ceph-mon[97033]: pgmap v2053: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:44:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:08 smithi059 ceph-mon[97450]: pgmap v2053: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:44:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:09 smithi055 ceph-mon[97033]: osdmap e220: 12 total, 11 up, 11 in 2022-09-07T22:44:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:09 smithi059 ceph-mon[97450]: osdmap e220: 12 total, 11 up, 11 in 2022-09-07T22:44:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:10 smithi055 ceph-mon[97033]: pgmap v2055: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:44:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:10 smithi059 ceph-mon[97450]: pgmap v2055: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:44:12.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:12 smithi055 ceph-mon[97033]: pgmap v2056: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:44:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:12 smithi059 ceph-mon[97450]: pgmap v2056: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:44:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:14 smithi055 ceph-mon[97033]: pgmap v2057: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:44:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:14 smithi055 ceph-mon[97033]: osdmap e221: 12 total, 11 up, 11 in 2022-09-07T22:44:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:14 smithi059 ceph-mon[97450]: pgmap v2057: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:44:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:14 smithi059 ceph-mon[97450]: osdmap e221: 12 total, 11 up, 11 in 2022-09-07T22:44:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:16 smithi055 ceph-mon[97033]: pgmap v2059: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:44:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:16 smithi059 ceph-mon[97450]: pgmap v2059: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:44:16.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:44:16.575+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:44:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:18 smithi055 ceph-mon[97033]: pgmap v2060: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-07T22:44:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:18 smithi059 ceph-mon[97450]: pgmap v2060: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-07T22:44:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:20 smithi055 ceph-mon[97033]: pgmap v2061: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:44:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:20 smithi059 ceph-mon[97450]: pgmap v2061: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:44:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:22 smithi055 ceph-mon[97033]: pgmap v2062: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:44:22.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:22 smithi059 ceph-mon[97450]: pgmap v2062: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:44:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:24 smithi055 ceph-mon[97033]: pgmap v2063: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:44:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:24 smithi059 ceph-mon[97450]: pgmap v2063: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:44:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:26 smithi055 ceph-mon[97033]: pgmap v2064: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:44:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:26 smithi059 ceph-mon[97450]: pgmap v2064: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:44:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:28 smithi055 ceph-mon[97033]: pgmap v2065: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:44:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:28 smithi059 ceph-mon[97450]: pgmap v2065: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:44:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:30 smithi055 ceph-mon[97033]: pgmap v2066: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:30 smithi059 ceph-mon[97450]: pgmap v2066: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:31.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:44:31.585+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:44:32.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:32 smithi055 ceph-mon[97033]: pgmap v2067: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:44:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:32 smithi059 ceph-mon[97450]: pgmap v2067: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:44:34.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:34 smithi055 ceph-mon[97033]: pgmap v2068: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:34 smithi059 ceph-mon[97450]: pgmap v2068: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:36 smithi055 ceph-mon[97033]: pgmap v2069: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:36 smithi059 ceph-mon[97450]: pgmap v2069: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:38 smithi055 ceph-mon[97033]: pgmap v2070: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:44:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:38 smithi059 ceph-mon[97450]: pgmap v2070: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:44:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:40 smithi055 ceph-mon[97033]: pgmap v2071: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:40 smithi059 ceph-mon[97450]: pgmap v2071: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:42 smithi055 ceph-mon[97033]: pgmap v2072: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:44:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:42 smithi059 ceph-mon[97450]: pgmap v2072: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:44:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:44 smithi055 ceph-mon[97033]: pgmap v2073: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:44 smithi059 ceph-mon[97450]: pgmap v2073: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:46 smithi055 ceph-mon[97033]: pgmap v2074: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:46 smithi059 ceph-mon[97450]: pgmap v2074: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:46.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:44:46.590+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:44:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:48 smithi055 ceph-mon[97033]: pgmap v2075: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:44:48.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:48 smithi059 ceph-mon[97450]: pgmap v2075: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:44:50.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:50 smithi055 ceph-mon[97033]: pgmap v2076: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:50 smithi059 ceph-mon[97450]: pgmap v2076: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:52.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:52 smithi055 ceph-mon[97033]: pgmap v2077: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:44:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:52 smithi059 ceph-mon[97450]: pgmap v2077: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:44:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:54 smithi055 ceph-mon[97033]: pgmap v2078: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:54 smithi059 ceph-mon[97450]: pgmap v2078: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:56 smithi055 ceph-mon[97033]: pgmap v2079: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:56 smithi059 ceph-mon[97450]: pgmap v2079: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:44:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:44:58 smithi055 ceph-mon[97033]: pgmap v2080: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:44:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:44:58 smithi059 ceph-mon[97450]: pgmap v2080: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:45:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:00 smithi055 ceph-mon[97033]: pgmap v2081: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:00 smithi059 ceph-mon[97450]: pgmap v2081: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:01.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:45:01.591+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:45:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:02 smithi055 ceph-mon[97033]: pgmap v2082: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:45:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:45:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:45:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:45:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:02 smithi059 ceph-mon[97450]: pgmap v2082: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:45:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:45:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:45:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:45:03.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:45:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:45:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:45:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:45:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:04 smithi055 ceph-mon[97033]: pgmap v2083: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:45:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:45:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:04 smithi059 ceph-mon[97450]: pgmap v2083: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:45:04.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:45:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:06 smithi059 ceph-mon[97450]: pgmap v2084: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:45:06.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:45:06.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:45:06.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:45:06.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.561 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.562 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.562 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.563 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.563 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.563 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:45:06.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:06 smithi055 ceph-mon[97033]: pgmap v2084: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:06.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:45:06.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:45:06.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:45:06.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:45:06.741 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.741 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.742 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.743 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.744 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.744 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.746 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.746 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:45:06.747 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:45:07.547 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:07 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T22:45:07.548 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:07 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T22:45:07.553 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:07 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T22:45:07.557 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:07 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T22:45:07.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:07 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T22:45:07.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:07 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T22:45:07.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:07 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T22:45:07.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:07 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T22:45:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:08 smithi059 ceph-mon[97450]: pgmap v2085: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:45:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:08 smithi055 ceph-mon[97033]: pgmap v2085: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:45:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:09 smithi059 ceph-mon[97450]: osdmap e222: 12 total, 11 up, 11 in 2022-09-07T22:45:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:09 smithi055 ceph-mon[97033]: osdmap e222: 12 total, 11 up, 11 in 2022-09-07T22:45:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:10 smithi055 ceph-mon[97033]: pgmap v2087: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:45:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:10 smithi059 ceph-mon[97450]: pgmap v2087: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:45:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:12 smithi055 ceph-mon[97033]: pgmap v2088: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:45:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:12 smithi059 ceph-mon[97450]: pgmap v2088: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:45:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:14 smithi055 ceph-mon[97033]: pgmap v2089: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:45:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:14 smithi055 ceph-mon[97033]: osdmap e223: 12 total, 11 up, 11 in 2022-09-07T22:45:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:14 smithi059 ceph-mon[97450]: pgmap v2089: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:45:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:14 smithi059 ceph-mon[97450]: osdmap e223: 12 total, 11 up, 11 in 2022-09-07T22:45:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:16 smithi055 ceph-mon[97033]: pgmap v2091: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:45:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:16 smithi059 ceph-mon[97450]: pgmap v2091: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:45:16.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:45:16.592+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:45:18.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:18 smithi055 ceph-mon[97033]: pgmap v2092: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-07T22:45:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:18 smithi059 ceph-mon[97450]: pgmap v2092: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-07T22:45:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:20 smithi055 ceph-mon[97033]: pgmap v2093: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:45:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:20 smithi059 ceph-mon[97450]: pgmap v2093: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:45:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:22 smithi055 ceph-mon[97033]: pgmap v2094: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:45:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:22 smithi059 ceph-mon[97450]: pgmap v2094: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:45:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:24 smithi055 ceph-mon[97033]: pgmap v2095: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:45:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:24 smithi059 ceph-mon[97450]: pgmap v2095: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:45:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:26 smithi055 ceph-mon[97033]: pgmap v2096: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:45:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:26 smithi059 ceph-mon[97450]: pgmap v2096: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:45:28.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:28 smithi055 ceph-mon[97033]: pgmap v2097: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:45:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:28 smithi059 ceph-mon[97450]: pgmap v2097: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:45:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:30 smithi055 ceph-mon[97033]: pgmap v2098: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:30 smithi059 ceph-mon[97450]: pgmap v2098: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:31.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:45:31.599+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:45:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:32 smithi055 ceph-mon[97033]: pgmap v2099: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:45:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:32 smithi059 ceph-mon[97450]: pgmap v2099: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:45:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:34 smithi055 ceph-mon[97033]: pgmap v2100: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:34.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:34 smithi059 ceph-mon[97450]: pgmap v2100: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:36 smithi055 ceph-mon[97033]: pgmap v2101: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:36 smithi059 ceph-mon[97450]: pgmap v2101: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:38 smithi055 ceph-mon[97033]: pgmap v2102: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:45:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:38 smithi059 ceph-mon[97450]: pgmap v2102: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:45:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:40 smithi055 ceph-mon[97033]: pgmap v2103: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:40 smithi059 ceph-mon[97450]: pgmap v2103: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:42 smithi055 ceph-mon[97033]: pgmap v2104: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:45:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:42 smithi059 ceph-mon[97450]: pgmap v2104: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:45:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:44 smithi055 ceph-mon[97033]: pgmap v2105: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:44 smithi059 ceph-mon[97450]: pgmap v2105: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:46 smithi055 ceph-mon[97033]: pgmap v2106: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:46 smithi059 ceph-mon[97450]: pgmap v2106: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:46.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:45:46.604+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:45:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:48 smithi055 ceph-mon[97033]: pgmap v2107: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:45:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:48 smithi059 ceph-mon[97450]: pgmap v2107: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:45:50.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:50 smithi055 ceph-mon[97033]: pgmap v2108: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:50 smithi059 ceph-mon[97450]: pgmap v2108: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:52 smithi055 ceph-mon[97033]: pgmap v2109: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:45:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:52 smithi059 ceph-mon[97450]: pgmap v2109: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:45:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:54 smithi055 ceph-mon[97033]: pgmap v2110: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:54 smithi059 ceph-mon[97450]: pgmap v2110: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:56 smithi055 ceph-mon[97033]: pgmap v2111: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:56 smithi059 ceph-mon[97450]: pgmap v2111: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:45:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:45:58 smithi055 ceph-mon[97033]: pgmap v2112: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:45:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:45:58 smithi059 ceph-mon[97450]: pgmap v2112: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:46:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:00 smithi055 ceph-mon[97033]: pgmap v2113: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:00 smithi059 ceph-mon[97450]: pgmap v2113: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:01.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:46:01.614+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:46:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:02 smithi055 ceph-mon[97033]: pgmap v2114: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:46:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:02 smithi059 ceph-mon[97450]: pgmap v2114: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:46:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:04 smithi055 ceph-mon[97033]: pgmap v2115: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:46:04.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:46:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:04 smithi059 ceph-mon[97450]: pgmap v2115: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:46:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:46:06.690 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:06 smithi059 ceph-mon[97450]: pgmap v2116: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:06.690 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:46:06.691 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:46:06.691 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:46:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:06 smithi055 ceph-mon[97033]: pgmap v2116: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:46:06.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:46:06.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:46:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:46:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:46:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:46:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:46:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:46:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:46:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:08 smithi055 ceph-mon[97033]: pgmap v2117: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:46:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:08 smithi059 ceph-mon[97450]: pgmap v2117: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:46:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:09 smithi055 ceph-mon[97033]: osdmap e224: 12 total, 11 up, 11 in 2022-09-07T22:46:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:09 smithi055 ceph-mon[97033]: pgmap v2119: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:46:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:09 smithi059 ceph-mon[97450]: osdmap e224: 12 total, 11 up, 11 in 2022-09-07T22:46:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:09 smithi059 ceph-mon[97450]: pgmap v2119: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:46:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:11 smithi059 ceph-mon[97450]: pgmap v2120: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:46:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:11 smithi055 ceph-mon[97033]: pgmap v2120: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:46:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:14 smithi055 ceph-mon[97033]: pgmap v2121: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:46:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:14 smithi055 ceph-mon[97033]: osdmap e225: 12 total, 11 up, 11 in 2022-09-07T22:46:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:14 smithi059 ceph-mon[97450]: pgmap v2121: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:46:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:14 smithi059 ceph-mon[97450]: osdmap e225: 12 total, 11 up, 11 in 2022-09-07T22:46:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:16 smithi055 ceph-mon[97033]: pgmap v2123: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:46:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:16 smithi059 ceph-mon[97450]: pgmap v2123: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:46:16.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:46:16.615+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:46:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:18 smithi055 ceph-mon[97033]: pgmap v2124: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-07T22:46:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:18 smithi059 ceph-mon[97450]: pgmap v2124: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-07T22:46:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:20 smithi055 ceph-mon[97033]: pgmap v2125: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:46:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:20 smithi059 ceph-mon[97450]: pgmap v2125: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:46:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:22 smithi055 ceph-mon[97033]: pgmap v2126: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:46:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:22 smithi059 ceph-mon[97450]: pgmap v2126: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:46:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:24 smithi055 ceph-mon[97033]: pgmap v2127: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:46:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:24 smithi059 ceph-mon[97450]: pgmap v2127: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:46:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:26 smithi055 ceph-mon[97033]: pgmap v2128: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:46:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:26 smithi059 ceph-mon[97450]: pgmap v2128: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:46:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:28 smithi055 ceph-mon[97033]: pgmap v2129: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:46:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:28 smithi059 ceph-mon[97450]: pgmap v2129: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:46:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:30 smithi055 ceph-mon[97033]: pgmap v2130: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:30 smithi059 ceph-mon[97450]: pgmap v2130: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:31.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:46:31.631+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:46:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:32 smithi055 ceph-mon[97033]: pgmap v2131: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:46:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:32 smithi059 ceph-mon[97450]: pgmap v2131: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:46:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:34 smithi055 ceph-mon[97033]: pgmap v2132: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:34 smithi059 ceph-mon[97450]: pgmap v2132: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:36 smithi055 ceph-mon[97033]: pgmap v2133: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:36 smithi059 ceph-mon[97450]: pgmap v2133: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:38 smithi055 ceph-mon[97033]: pgmap v2134: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:46:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:38 smithi059 ceph-mon[97450]: pgmap v2134: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:46:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:40 smithi055 ceph-mon[97033]: pgmap v2135: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:40 smithi059 ceph-mon[97450]: pgmap v2135: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:42 smithi055 ceph-mon[97033]: pgmap v2136: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:46:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:42 smithi059 ceph-mon[97450]: pgmap v2136: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:46:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:44 smithi055 ceph-mon[97033]: pgmap v2137: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:44 smithi059 ceph-mon[97450]: pgmap v2137: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:46 smithi055 ceph-mon[97033]: pgmap v2138: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:46 smithi059 ceph-mon[97450]: pgmap v2138: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:46.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:46:46.639+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:46:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:48 smithi055 ceph-mon[97033]: pgmap v2139: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:46:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:48 smithi059 ceph-mon[97450]: pgmap v2139: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:46:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:50 smithi055 ceph-mon[97033]: pgmap v2140: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:50 smithi059 ceph-mon[97450]: pgmap v2140: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:52 smithi055 ceph-mon[97033]: pgmap v2141: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:46:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:52 smithi059 ceph-mon[97450]: pgmap v2141: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:46:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:54 smithi055 ceph-mon[97033]: pgmap v2142: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:54 smithi059 ceph-mon[97450]: pgmap v2142: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:56 smithi055 ceph-mon[97033]: pgmap v2143: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:56 smithi059 ceph-mon[97450]: pgmap v2143: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:46:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:46:58 smithi055 ceph-mon[97033]: pgmap v2144: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:46:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:46:58 smithi059 ceph-mon[97450]: pgmap v2144: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:47:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:00 smithi055 ceph-mon[97033]: pgmap v2145: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:00 smithi059 ceph-mon[97450]: pgmap v2145: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:01.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:47:01.641+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:47:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:02 smithi055 ceph-mon[97033]: pgmap v2146: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:47:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:02 smithi059 ceph-mon[97450]: pgmap v2146: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:47:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:04 smithi055 ceph-mon[97033]: pgmap v2147: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:47:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:47:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:04 smithi059 ceph-mon[97450]: pgmap v2147: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:47:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:47:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:06 smithi055 ceph-mon[97033]: pgmap v2148: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:06 smithi059 ceph-mon[97450]: pgmap v2148: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:47:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:47:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:47:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:47:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:47:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:47:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:08 smithi055 ceph-mon[97033]: pgmap v2149: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:47:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:47:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:08 smithi059 ceph-mon[97450]: pgmap v2149: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:47:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:47:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:09 smithi059 ceph-mon[97450]: pgmap v2150: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:47:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:09 smithi059 ceph-mon[97450]: osdmap e226: 12 total, 11 up, 11 in 2022-09-07T22:47:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:09 smithi055 ceph-mon[97033]: pgmap v2150: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:47:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:09 smithi055 ceph-mon[97033]: osdmap e226: 12 total, 11 up, 11 in 2022-09-07T22:47:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:11 smithi059 ceph-mon[97450]: pgmap v2152: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-07T22:47:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:11 smithi055 ceph-mon[97033]: pgmap v2152: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-07T22:47:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:14 smithi055 ceph-mon[97033]: pgmap v2153: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-07T22:47:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:14 smithi055 ceph-mon[97033]: osdmap e227: 12 total, 11 up, 11 in 2022-09-07T22:47:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:14 smithi059 ceph-mon[97450]: pgmap v2153: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-07T22:47:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:14 smithi059 ceph-mon[97450]: osdmap e227: 12 total, 11 up, 11 in 2022-09-07T22:47:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:16 smithi055 ceph-mon[97033]: pgmap v2155: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 5.6 KiB/s rd, 3.7 KiB/s wr, 4 op/s 2022-09-07T22:47:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:16 smithi059 ceph-mon[97450]: pgmap v2155: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 5.6 KiB/s rd, 3.7 KiB/s wr, 4 op/s 2022-09-07T22:47:16.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:47:16.646+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:47:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:18 smithi055 ceph-mon[97033]: pgmap v2156: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 7.6 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2022-09-07T22:47:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:18 smithi059 ceph-mon[97450]: pgmap v2156: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 7.6 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2022-09-07T22:47:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:20 smithi055 ceph-mon[97033]: pgmap v2157: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2022-09-07T22:47:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:20 smithi059 ceph-mon[97450]: pgmap v2157: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2022-09-07T22:47:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:22 smithi055 ceph-mon[97033]: pgmap v2158: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:47:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:22 smithi059 ceph-mon[97450]: pgmap v2158: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:47:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:24 smithi055 ceph-mon[97033]: pgmap v2159: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:47:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:24 smithi059 ceph-mon[97450]: pgmap v2159: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:47:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:26 smithi055 ceph-mon[97033]: pgmap v2160: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:47:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:26 smithi059 ceph-mon[97450]: pgmap v2160: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:47:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:28 smithi055 ceph-mon[97033]: pgmap v2161: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:47:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:28 smithi059 ceph-mon[97450]: pgmap v2161: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:47:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:30 smithi055 ceph-mon[97033]: pgmap v2162: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:30 smithi059 ceph-mon[97450]: pgmap v2162: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:32.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:47:31.660+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:47:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:32 smithi055 ceph-mon[97033]: pgmap v2163: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:47:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:32 smithi059 ceph-mon[97450]: pgmap v2163: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:47:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:34 smithi055 ceph-mon[97033]: pgmap v2164: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:34 smithi059 ceph-mon[97450]: pgmap v2164: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:36 smithi055 ceph-mon[97033]: pgmap v2165: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:36 smithi059 ceph-mon[97450]: pgmap v2165: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:38 smithi055 ceph-mon[97033]: pgmap v2166: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:47:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:38 smithi059 ceph-mon[97450]: pgmap v2166: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:47:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:40 smithi055 ceph-mon[97033]: pgmap v2167: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:40 smithi059 ceph-mon[97450]: pgmap v2167: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:42 smithi055 ceph-mon[97033]: pgmap v2168: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:47:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:42 smithi059 ceph-mon[97450]: pgmap v2168: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:47:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:44 smithi055 ceph-mon[97033]: pgmap v2169: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:44 smithi059 ceph-mon[97450]: pgmap v2169: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:46 smithi055 ceph-mon[97033]: pgmap v2170: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:46 smithi059 ceph-mon[97450]: pgmap v2170: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:47.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:47:46.670+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:47:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:48 smithi055 ceph-mon[97033]: pgmap v2171: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:47:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:48 smithi059 ceph-mon[97450]: pgmap v2171: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:47:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:50 smithi055 ceph-mon[97033]: pgmap v2172: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:50 smithi059 ceph-mon[97450]: pgmap v2172: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:52 smithi055 ceph-mon[97033]: pgmap v2173: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:47:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:52 smithi059 ceph-mon[97450]: pgmap v2173: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:47:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:54 smithi055 ceph-mon[97033]: pgmap v2174: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:54 smithi059 ceph-mon[97450]: pgmap v2174: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:56 smithi055 ceph-mon[97033]: pgmap v2175: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:56 smithi059 ceph-mon[97450]: pgmap v2175: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:47:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:47:58 smithi055 ceph-mon[97033]: pgmap v2176: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:47:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:47:58 smithi059 ceph-mon[97450]: pgmap v2176: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:48:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:00 smithi055 ceph-mon[97033]: pgmap v2177: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:00 smithi059 ceph-mon[97450]: pgmap v2177: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:02.159 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:48:01.684+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:48:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:02 smithi055 ceph-mon[97033]: pgmap v2178: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:48:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:02 smithi059 ceph-mon[97450]: pgmap v2178: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:48:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:04 smithi055 ceph-mon[97033]: pgmap v2179: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:04.743 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:48:04.743 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:48:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:04 smithi059 ceph-mon[97450]: pgmap v2179: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:48:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:48:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:06 smithi055 ceph-mon[97033]: pgmap v2180: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:06 smithi059 ceph-mon[97450]: pgmap v2180: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:08 smithi055 ceph-mon[97033]: pgmap v2181: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:48:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:48:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:48:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:48:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:48:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:08 smithi059 ceph-mon[97450]: pgmap v2181: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:48:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:48:08.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:48:08.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:48:08.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:48:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:09 smithi055 ceph-mon[97033]: osdmap e228: 12 total, 11 up, 11 in 2022-09-07T22:48:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:09 smithi059 ceph-mon[97450]: osdmap e228: 12 total, 11 up, 11 in 2022-09-07T22:48:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:10 smithi055 ceph-mon[97033]: pgmap v2183: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:48:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:10 smithi059 ceph-mon[97450]: pgmap v2183: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:48:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:12 smithi055 ceph-mon[97033]: pgmap v2184: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:48:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:12 smithi059 ceph-mon[97450]: pgmap v2184: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:48:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:14 smithi055 ceph-mon[97033]: pgmap v2185: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:48:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:14 smithi055 ceph-mon[97033]: osdmap e229: 12 total, 11 up, 11 in 2022-09-07T22:48:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:14 smithi059 ceph-mon[97450]: pgmap v2185: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:48:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:14 smithi059 ceph-mon[97450]: osdmap e229: 12 total, 11 up, 11 in 2022-09-07T22:48:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:16 smithi055 ceph-mon[97033]: pgmap v2187: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:48:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:16 smithi059 ceph-mon[97450]: pgmap v2187: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:48:17.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:48:16.696+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:48:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:18 smithi055 ceph-mon[97033]: pgmap v2188: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 246 B/s wr, 7 op/s 2022-09-07T22:48:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:18 smithi059 ceph-mon[97450]: pgmap v2188: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 246 B/s wr, 7 op/s 2022-09-07T22:48:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:20 smithi055 ceph-mon[97033]: pgmap v2189: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:48:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:20 smithi059 ceph-mon[97450]: pgmap v2189: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:48:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:22 smithi055 ceph-mon[97033]: pgmap v2190: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:48:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:22 smithi059 ceph-mon[97450]: pgmap v2190: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:48:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:24 smithi055 ceph-mon[97033]: pgmap v2191: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:48:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:24 smithi059 ceph-mon[97450]: pgmap v2191: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:48:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:26 smithi055 ceph-mon[97033]: pgmap v2192: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:48:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:26 smithi059 ceph-mon[97450]: pgmap v2192: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:48:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:28 smithi055 ceph-mon[97033]: pgmap v2193: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:48:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:28 smithi059 ceph-mon[97450]: pgmap v2193: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:48:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:30 smithi055 ceph-mon[97033]: pgmap v2194: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:30 smithi059 ceph-mon[97450]: pgmap v2194: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:32.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:48:31.704+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:48:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:32 smithi055 ceph-mon[97033]: pgmap v2195: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:48:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:32 smithi059 ceph-mon[97450]: pgmap v2195: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:48:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:34 smithi055 ceph-mon[97033]: pgmap v2196: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:34 smithi059 ceph-mon[97450]: pgmap v2196: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:36 smithi055 ceph-mon[97033]: pgmap v2197: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:36 smithi059 ceph-mon[97450]: pgmap v2197: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:38 smithi055 ceph-mon[97033]: pgmap v2198: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:48:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:38 smithi059 ceph-mon[97450]: pgmap v2198: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:48:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:40 smithi055 ceph-mon[97033]: pgmap v2199: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:40 smithi059 ceph-mon[97450]: pgmap v2199: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:42 smithi055 ceph-mon[97033]: pgmap v2200: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:48:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:42 smithi059 ceph-mon[97450]: pgmap v2200: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:48:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:44 smithi055 ceph-mon[97033]: pgmap v2201: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:44 smithi059 ceph-mon[97450]: pgmap v2201: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:46 smithi055 ceph-mon[97033]: pgmap v2202: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:46 smithi059 ceph-mon[97450]: pgmap v2202: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:47.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:48:46.717+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:48:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:48 smithi055 ceph-mon[97033]: pgmap v2203: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:48:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:48 smithi059 ceph-mon[97450]: pgmap v2203: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:48:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:50 smithi055 ceph-mon[97033]: pgmap v2204: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:50 smithi059 ceph-mon[97450]: pgmap v2204: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:52.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:52 smithi055 ceph-mon[97033]: pgmap v2205: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:48:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:52 smithi059 ceph-mon[97450]: pgmap v2205: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:48:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:54 smithi055 ceph-mon[97033]: pgmap v2206: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:54 smithi059 ceph-mon[97450]: pgmap v2206: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:56 smithi055 ceph-mon[97033]: pgmap v2207: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:56 smithi059 ceph-mon[97450]: pgmap v2207: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:48:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:48:58 smithi055 ceph-mon[97033]: pgmap v2208: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:48:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:48:58 smithi059 ceph-mon[97450]: pgmap v2208: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:00 smithi055 ceph-mon[97033]: pgmap v2209: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:00 smithi059 ceph-mon[97450]: pgmap v2209: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:01.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:49:01.724+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:49:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:02 smithi055 ceph-mon[97033]: pgmap v2210: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:02 smithi059 ceph-mon[97450]: pgmap v2210: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:04 smithi055 ceph-mon[97033]: pgmap v2211: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:49:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:49:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:49:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:04 smithi059 ceph-mon[97450]: pgmap v2211: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:49:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:49:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:49:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:06 smithi055 ceph-mon[97033]: pgmap v2212: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:49:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:06 smithi059 ceph-mon[97450]: pgmap v2212: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:49:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:08 smithi055 ceph-mon[97033]: pgmap v2213: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:49:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:08 smithi059 ceph-mon[97450]: pgmap v2213: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:49:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:09 smithi059 ceph-mon[97450]: osdmap e230: 12 total, 11 up, 11 in 2022-09-07T22:49:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:49:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:49:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:49:09.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:49:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:09 smithi055 ceph-mon[97033]: osdmap e230: 12 total, 11 up, 11 in 2022-09-07T22:49:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:49:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:49:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:49:09.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:49:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:10 smithi059 ceph-mon[97450]: pgmap v2215: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:49:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:10 smithi055 ceph-mon[97033]: pgmap v2215: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:49:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:12 smithi059 ceph-mon[97450]: pgmap v2216: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:49:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:12 smithi055 ceph-mon[97033]: pgmap v2216: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:49:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:14 smithi055 ceph-mon[97033]: pgmap v2217: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:49:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:14 smithi055 ceph-mon[97033]: osdmap e231: 12 total, 11 up, 11 in 2022-09-07T22:49:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:14 smithi059 ceph-mon[97450]: pgmap v2217: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:49:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:14 smithi059 ceph-mon[97450]: osdmap e231: 12 total, 11 up, 11 in 2022-09-07T22:49:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:16 smithi055 ceph-mon[97033]: pgmap v2219: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:49:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:16 smithi059 ceph-mon[97450]: pgmap v2219: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:49:16.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:49:16.735+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:49:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:18 smithi055 ceph-mon[97033]: pgmap v2220: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 246 B/s wr, 7 op/s 2022-09-07T22:49:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:18 smithi059 ceph-mon[97450]: pgmap v2220: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 246 B/s wr, 7 op/s 2022-09-07T22:49:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:20 smithi055 ceph-mon[97033]: pgmap v2221: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:49:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:20 smithi059 ceph-mon[97450]: pgmap v2221: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:49:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:22 smithi055 ceph-mon[97033]: pgmap v2222: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:49:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:22 smithi059 ceph-mon[97450]: pgmap v2222: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:49:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:24 smithi055 ceph-mon[97033]: pgmap v2223: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:49:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:24 smithi059 ceph-mon[97450]: pgmap v2223: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:49:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:26 smithi055 ceph-mon[97033]: pgmap v2224: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:49:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:26 smithi059 ceph-mon[97450]: pgmap v2224: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-07T22:49:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:28 smithi055 ceph-mon[97033]: pgmap v2225: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:49:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:28 smithi059 ceph-mon[97450]: pgmap v2225: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:49:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:30 smithi055 ceph-mon[97033]: pgmap v2226: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:30 smithi059 ceph-mon[97450]: pgmap v2226: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:32.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:49:31.751+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:49:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:32 smithi055 ceph-mon[97033]: pgmap v2227: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:32 smithi059 ceph-mon[97450]: pgmap v2227: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:34 smithi055 ceph-mon[97033]: pgmap v2228: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:34 smithi059 ceph-mon[97450]: pgmap v2228: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:36.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:36 smithi055 ceph-mon[97033]: pgmap v2229: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:36 smithi059 ceph-mon[97450]: pgmap v2229: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:38 smithi055 ceph-mon[97033]: pgmap v2230: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:38 smithi059 ceph-mon[97450]: pgmap v2230: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:38.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:38.945059+0000) 2022-09-07T22:49:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:38.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:38.838740+0000) 2022-09-07T22:49:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:39.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:39.945223+0000) 2022-09-07T22:49:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:39.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:39.838878+0000) 2022-09-07T22:49:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:40 smithi055 ceph-mon[97033]: pgmap v2231: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:40 smithi059 ceph-mon[97450]: pgmap v2231: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:40.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:40.839087+0000) 2022-09-07T22:49:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:40.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:40.945432+0000) 2022-09-07T22:49:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:41.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:41.945598+0000) 2022-09-07T22:49:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:41.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:41.839207+0000) 2022-09-07T22:49:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:42 smithi055 ceph-mon[97033]: pgmap v2232: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:42 smithi059 ceph-mon[97450]: pgmap v2232: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:42.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:42.945815+0000) 2022-09-07T22:49:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:42.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:42.839407+0000) 2022-09-07T22:49:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:43.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:43.946008+0000) 2022-09-07T22:49:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:43.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:43.839558+0000) 2022-09-07T22:49:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:44 smithi055 ceph-mon[97033]: pgmap v2233: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:44 smithi059 ceph-mon[97450]: pgmap v2233: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:44.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:44.946246+0000) 2022-09-07T22:49:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:44.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:44.839763+0000) 2022-09-07T22:49:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:45.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:45.946472+0000) 2022-09-07T22:49:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:45.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:45.839993+0000) 2022-09-07T22:49:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:46 smithi055 ceph-mon[97033]: pgmap v2234: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:46 smithi059 ceph-mon[97450]: pgmap v2234: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:47.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:49:46.765+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:49:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:46.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:46.946736+0000) 2022-09-07T22:49:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:46.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:46.840163+0000) 2022-09-07T22:49:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:47.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:47.946963+0000) 2022-09-07T22:49:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:47.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:47.840375+0000) 2022-09-07T22:49:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:48 smithi055 ceph-mon[97033]: pgmap v2235: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:48 smithi059 ceph-mon[97450]: pgmap v2235: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:48.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:48.947204+0000) 2022-09-07T22:49:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:48.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:48.840619+0000) 2022-09-07T22:49:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:49.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:49.947448+0000) 2022-09-07T22:49:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:49.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:49.840816+0000) 2022-09-07T22:49:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:50 smithi055 ceph-mon[97033]: pgmap v2236: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:50 smithi059 ceph-mon[97450]: pgmap v2236: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:50.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:50.947686+0000) 2022-09-07T22:49:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:50.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:50.841019+0000) 2022-09-07T22:49:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:51.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:51.947820+0000) 2022-09-07T22:49:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:51.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:51.841166+0000) 2022-09-07T22:49:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:52 smithi055 ceph-mon[97033]: pgmap v2237: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:52 smithi059 ceph-mon[97450]: pgmap v2237: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:52.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:52.947997+0000) 2022-09-07T22:49:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:52.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:52.841317+0000) 2022-09-07T22:49:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:53.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:53.948221+0000) 2022-09-07T22:49:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:53.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:53.841491+0000) 2022-09-07T22:49:54.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:54 smithi055 ceph-mon[97033]: pgmap v2238: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:54 smithi059 ceph-mon[97450]: pgmap v2238: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:54.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:54.948438+0000) 2022-09-07T22:49:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:54.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:54.841768+0000) 2022-09-07T22:49:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:55.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:55.841989+0000) 2022-09-07T22:49:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:55.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:55.948716+0000) 2022-09-07T22:49:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:56 smithi055 ceph-mon[97033]: pgmap v2239: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:56 smithi059 ceph-mon[97450]: pgmap v2239: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:49:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:56.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:56.948966+0000) 2022-09-07T22:49:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:56.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:56.842172+0000) 2022-09-07T22:49:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:57.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:57.842366+0000) 2022-09-07T22:49:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:57.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:57.949121+0000) 2022-09-07T22:49:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:49:58 smithi055 ceph-mon[97033]: pgmap v2240: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:49:58 smithi059 ceph-mon[97450]: pgmap v2240: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:49:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:58.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:58.842520+0000) 2022-09-07T22:49:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:58.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:58.949339+0000) 2022-09-07T22:50:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:49:59.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:59.949680+0000) 2022-09-07T22:50:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:49:59.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:49:59.842771+0000) 2022-09-07T22:50:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:00 smithi055 ceph-mon[97033]: pgmap v2241: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-07T22:50:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T22:50:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T22:50:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:00 smithi059 ceph-mon[97450]: pgmap v2241: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-07T22:50:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T22:50:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T22:50:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:00.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:00.949874+0000) 2022-09-07T22:50:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:00.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:00.842921+0000) 2022-09-07T22:50:02.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:50:01.777+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:50:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:01.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:01.950119+0000) 2022-09-07T22:50:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:01.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:01.843158+0000) 2022-09-07T22:50:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:02 smithi055 ceph-mon[97033]: pgmap v2242: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:50:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:02 smithi059 ceph-mon[97450]: pgmap v2242: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:50:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:02.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:02.950271+0000) 2022-09-07T22:50:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:02.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:02.843322+0000) 2022-09-07T22:50:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:03.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:03.950515+0000) 2022-09-07T22:50:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:03.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:03.843451+0000) 2022-09-07T22:50:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:04 smithi059 ceph-mon[97450]: pgmap v2243: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:50:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:50:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:04 smithi055 ceph-mon[97033]: pgmap v2243: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:50:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:50:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:04.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:04.950694+0000) 2022-09-07T22:50:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:04.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:04.843609+0000) 2022-09-07T22:50:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:05.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:05.950906+0000) 2022-09-07T22:50:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:05.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:05.843745+0000) 2022-09-07T22:50:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:06 smithi059 ceph-mon[97450]: pgmap v2244: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:06 smithi055 ceph-mon[97033]: pgmap v2244: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:07.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:06.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:06.951134+0000) 2022-09-07T22:50:07.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:06.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:06.843899+0000) 2022-09-07T22:50:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:07.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:07.844138+0000) 2022-09-07T22:50:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:07.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:07.951374+0000) 2022-09-07T22:50:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:08 smithi059 ceph-mon[97450]: pgmap v2245: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:50:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:08 smithi055 ceph-mon[97033]: pgmap v2245: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:50:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:08.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:08.951562+0000) 2022-09-07T22:50:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:08.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:08.844299+0000) 2022-09-07T22:50:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:09 smithi059 ceph-mon[97450]: osdmap e232: 12 total, 11 up, 11 in 2022-09-07T22:50:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:50:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:50:09.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:50:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:09 smithi055 ceph-mon[97033]: osdmap e232: 12 total, 11 up, 11 in 2022-09-07T22:50:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:50:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:50:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:50:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:09.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:09.951790+0000) 2022-09-07T22:50:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:09.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:09.844464+0000) 2022-09-07T22:50:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:10 smithi055 ceph-mon[97033]: pgmap v2247: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:50:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:50:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:10 smithi059 ceph-mon[97450]: pgmap v2247: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:50:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:50:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:10.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:10.952024+0000) 2022-09-07T22:50:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:10.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:10.844740+0000) 2022-09-07T22:50:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:11.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:11.952257+0000) 2022-09-07T22:50:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:11.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:11.844940+0000) 2022-09-07T22:50:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:12 smithi055 ceph-mon[97033]: pgmap v2248: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:50:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:12 smithi059 ceph-mon[97450]: pgmap v2248: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:50:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:12.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:12.952437+0000) 2022-09-07T22:50:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:12.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:12.845092+0000) 2022-09-07T22:50:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:13.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:13.952725+0000) 2022-09-07T22:50:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:13.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:13.845284+0000) 2022-09-07T22:50:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:14 smithi055 ceph-mon[97033]: pgmap v2249: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:50:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:14 smithi055 ceph-mon[97033]: osdmap e233: 12 total, 11 up, 11 in 2022-09-07T22:50:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:14 smithi059 ceph-mon[97450]: pgmap v2249: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:50:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:14 smithi059 ceph-mon[97450]: osdmap e233: 12 total, 11 up, 11 in 2022-09-07T22:50:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:14.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:14.952904+0000) 2022-09-07T22:50:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:14.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:14.845459+0000) 2022-09-07T22:50:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:15.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:15.953108+0000) 2022-09-07T22:50:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:15.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:15.845597+0000) 2022-09-07T22:50:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:16 smithi055 ceph-mon[97033]: pgmap v2251: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:50:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:16 smithi059 ceph-mon[97450]: pgmap v2251: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:50:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:16.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:16.953342+0000) 2022-09-07T22:50:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:16.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:16.845749+0000) 2022-09-07T22:50:17.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:50:16.793+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:50:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:17.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:17.953558+0000) 2022-09-07T22:50:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:17.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:17.845884+0000) 2022-09-07T22:50:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:18 smithi055 ceph-mon[97033]: pgmap v2252: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 247 B/s wr, 7 op/s 2022-09-07T22:50:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:18 smithi059 ceph-mon[97450]: pgmap v2252: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 247 B/s wr, 7 op/s 2022-09-07T22:50:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:18.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:18.953753+0000) 2022-09-07T22:50:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:18.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:18.846068+0000) 2022-09-07T22:50:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:19.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:19.953892+0000) 2022-09-07T22:50:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:19.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:19.846279+0000) 2022-09-07T22:50:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:20 smithi055 ceph-mon[97033]: pgmap v2253: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:50:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:20 smithi059 ceph-mon[97450]: pgmap v2253: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:50:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:20.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:20.954127+0000) 2022-09-07T22:50:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:20.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:20.846452+0000) 2022-09-07T22:50:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:21.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:21.954358+0000) 2022-09-07T22:50:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:21.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:21.846583+0000) 2022-09-07T22:50:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:22 smithi055 ceph-mon[97033]: pgmap v2254: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T22:50:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:22 smithi059 ceph-mon[97450]: pgmap v2254: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T22:50:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:22.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:22.954583+0000) 2022-09-07T22:50:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:22.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:22.846744+0000) 2022-09-07T22:50:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:23.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:23.954748+0000) 2022-09-07T22:50:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:23.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:23.846876+0000) 2022-09-07T22:50:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:24 smithi055 ceph-mon[97033]: pgmap v2255: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T22:50:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:24 smithi059 ceph-mon[97450]: pgmap v2255: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T22:50:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:24.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:24.954935+0000) 2022-09-07T22:50:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:24.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:24.847049+0000) 2022-09-07T22:50:25.956 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:25.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:25.847214+0000) 2022-09-07T22:50:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:25.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:25.955177+0000) 2022-09-07T22:50:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:26 smithi055 ceph-mon[97033]: pgmap v2256: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-07T22:50:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:26 smithi059 ceph-mon[97450]: pgmap v2256: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-07T22:50:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:26.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:26.955416+0000) 2022-09-07T22:50:27.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:26.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:26.847373+0000) 2022-09-07T22:50:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:27.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:27.955687+0000) 2022-09-07T22:50:28.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:27.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:27.847583+0000) 2022-09-07T22:50:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:28 smithi055 ceph-mon[97033]: pgmap v2257: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:50:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:28 smithi059 ceph-mon[97450]: pgmap v2257: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:50:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:28.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:28.955871+0000) 2022-09-07T22:50:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:28.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:28.847722+0000) 2022-09-07T22:50:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:29.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:29.847873+0000) 2022-09-07T22:50:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:29.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:29.956143+0000) 2022-09-07T22:50:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:30 smithi055 ceph-mon[97033]: pgmap v2258: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:50:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:30 smithi059 ceph-mon[97450]: pgmap v2258: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:50:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:30.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:30.848083+0000) 2022-09-07T22:50:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:30.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:30.956409+0000) 2022-09-07T22:50:32.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:50:31.803+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:50:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:31.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:31.848293+0000) 2022-09-07T22:50:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:31.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:31.956712+0000) 2022-09-07T22:50:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:32 smithi055 ceph-mon[97033]: pgmap v2259: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:50:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:32 smithi059 ceph-mon[97450]: pgmap v2259: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:50:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:32.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:32.956856+0000) 2022-09-07T22:50:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:32.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:32.848565+0000) 2022-09-07T22:50:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:33.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:33.957070+0000) 2022-09-07T22:50:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:33.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:33.848749+0000) 2022-09-07T22:50:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:34 smithi055 ceph-mon[97033]: pgmap v2260: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:34 smithi059 ceph-mon[97450]: pgmap v2260: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:34.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:34.957274+0000) 2022-09-07T22:50:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:34.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:34.848977+0000) 2022-09-07T22:50:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:35.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:35.957486+0000) 2022-09-07T22:50:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:35.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:35.849137+0000) 2022-09-07T22:50:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:36 smithi055 ceph-mon[97033]: pgmap v2261: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:36 smithi059 ceph-mon[97450]: pgmap v2261: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:36.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:36.957741+0000) 2022-09-07T22:50:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:36.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:36.849352+0000) 2022-09-07T22:50:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:37.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:37.957944+0000) 2022-09-07T22:50:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:37.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:37.849577+0000) 2022-09-07T22:50:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:38 smithi055 ceph-mon[97033]: pgmap v2262: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:50:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:38 smithi059 ceph-mon[97450]: pgmap v2262: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:50:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:38.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:38.958185+0000) 2022-09-07T22:50:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:38.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:38.849750+0000) 2022-09-07T22:50:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:39.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:39.958372+0000) 2022-09-07T22:50:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:39.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:39.849881+0000) 2022-09-07T22:50:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:40 smithi055 ceph-mon[97033]: pgmap v2263: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:40 smithi059 ceph-mon[97450]: pgmap v2263: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:40.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:40.958547+0000) 2022-09-07T22:50:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:40.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:40.850013+0000) 2022-09-07T22:50:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:41.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:41.958747+0000) 2022-09-07T22:50:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:41.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:41.850138+0000) 2022-09-07T22:50:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:42 smithi055 ceph-mon[97033]: pgmap v2264: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:50:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:42 smithi059 ceph-mon[97450]: pgmap v2264: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:50:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:42.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:42.958873+0000) 2022-09-07T22:50:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:42.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:42.850318+0000) 2022-09-07T22:50:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:43.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:43.959094+0000) 2022-09-07T22:50:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:43.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:43.850524+0000) 2022-09-07T22:50:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:44 smithi055 ceph-mon[97033]: pgmap v2265: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:44 smithi059 ceph-mon[97450]: pgmap v2265: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:44.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:44.959270+0000) 2022-09-07T22:50:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:44.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:44.850700+0000) 2022-09-07T22:50:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:45.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:45.959510+0000) 2022-09-07T22:50:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:45.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:45.850880+0000) 2022-09-07T22:50:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:46 smithi055 ceph-mon[97033]: pgmap v2266: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:46 smithi059 ceph-mon[97450]: pgmap v2266: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:47.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:50:46.811+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:50:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:46.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:46.959757+0000) 2022-09-07T22:50:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:46.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:46.851031+0000) 2022-09-07T22:50:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:47.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:47.851155+0000) 2022-09-07T22:50:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:47.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:47.959972+0000) 2022-09-07T22:50:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:48 smithi055 ceph-mon[97033]: pgmap v2267: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:50:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:48 smithi059 ceph-mon[97450]: pgmap v2267: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:50:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:48.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:48.960272+0000) 2022-09-07T22:50:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:48.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:48.851274+0000) 2022-09-07T22:50:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:49.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:49.960451+0000) 2022-09-07T22:50:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:49.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:49.851462+0000) 2022-09-07T22:50:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:50 smithi055 ceph-mon[97033]: pgmap v2268: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:50 smithi059 ceph-mon[97450]: pgmap v2268: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:50.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:50.960654+0000) 2022-09-07T22:50:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:50.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:50.851614+0000) 2022-09-07T22:50:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:51.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:51.960886+0000) 2022-09-07T22:50:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:51.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:51.851785+0000) 2022-09-07T22:50:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:52 smithi055 ceph-mon[97033]: pgmap v2269: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:50:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:52 smithi059 ceph-mon[97450]: pgmap v2269: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:50:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:52.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:52.961059+0000) 2022-09-07T22:50:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:52.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:52.851990+0000) 2022-09-07T22:50:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:53.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:53.961247+0000) 2022-09-07T22:50:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:53.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:53.852206+0000) 2022-09-07T22:50:54.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:54 smithi055 ceph-mon[97033]: pgmap v2270: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:54.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:54 smithi059 ceph-mon[97450]: pgmap v2270: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:54.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:54.852467+0000) 2022-09-07T22:50:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:54.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:54.961549+0000) 2022-09-07T22:50:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:55.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:55.961788+0000) 2022-09-07T22:50:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:55.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:55.852616+0000) 2022-09-07T22:50:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:56 smithi059 ceph-mon[97450]: pgmap v2271: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:56 smithi055 ceph-mon[97033]: pgmap v2271: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:50:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:56.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:56.961972+0000) 2022-09-07T22:50:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:56.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:56.852848+0000) 2022-09-07T22:50:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:57.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:57.962191+0000) 2022-09-07T22:50:58.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:57.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:57.852996+0000) 2022-09-07T22:50:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:50:58 smithi059 ceph-mon[97450]: pgmap v2272: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:50:58.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:50:58 smithi055 ceph-mon[97033]: pgmap v2272: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:50:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:58.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:58.962430+0000) 2022-09-07T22:50:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:58.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:58.853205+0000) 2022-09-07T22:51:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:50:59.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:59.962632+0000) 2022-09-07T22:51:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:50:59.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:50:59.853427+0000) 2022-09-07T22:51:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:00 smithi059 ceph-mon[97450]: pgmap v2273: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:00 smithi055 ceph-mon[97033]: pgmap v2273: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:00.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:00.962783+0000) 2022-09-07T22:51:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:00.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:00.853576+0000) 2022-09-07T22:51:02.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:51:01.827+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:51:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:01.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:01.962999+0000) 2022-09-07T22:51:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:01.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:01.853748+0000) 2022-09-07T22:51:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:02 smithi059 ceph-mon[97450]: pgmap v2274: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:51:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:02 smithi055 ceph-mon[97033]: pgmap v2274: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:51:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:02.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:02.963204+0000) 2022-09-07T22:51:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:02.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:02.853909+0000) 2022-09-07T22:51:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:03.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:03.963340+0000) 2022-09-07T22:51:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:03.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:03.854115+0000) 2022-09-07T22:51:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:04 smithi059 ceph-mon[97450]: pgmap v2275: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:51:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:51:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:04 smithi055 ceph-mon[97033]: pgmap v2275: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:51:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:51:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:04.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:04.963542+0000) 2022-09-07T22:51:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:04.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:04.854394+0000) 2022-09-07T22:51:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:05.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:05.963776+0000) 2022-09-07T22:51:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:05.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:05.854561+0000) 2022-09-07T22:51:06.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:06 smithi059 ceph-mon[97450]: pgmap v2276: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:06 smithi055 ceph-mon[97033]: pgmap v2276: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:06.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:06.964020+0000) 2022-09-07T22:51:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:06.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:06.854720+0000) 2022-09-07T22:51:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:07.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:07.964252+0000) 2022-09-07T22:51:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:07.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:07.854962+0000) 2022-09-07T22:51:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:08 smithi059 ceph-mon[97450]: pgmap v2277: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:51:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:08 smithi055 ceph-mon[97033]: pgmap v2277: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:51:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:08.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:08.964502+0000) 2022-09-07T22:51:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:08.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:08.855170+0000) 2022-09-07T22:51:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:09 smithi059 ceph-mon[97450]: osdmap e234: 12 total, 11 up, 11 in 2022-09-07T22:51:09.803 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:09 smithi055 ceph-mon[97033]: osdmap e234: 12 total, 11 up, 11 in 2022-09-07T22:51:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:09.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:09.964745+0000) 2022-09-07T22:51:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:09.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:09.855301+0000) 2022-09-07T22:51:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:10 smithi059 ceph-mon[97450]: pgmap v2279: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:51:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:51:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:51:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:51:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:10 smithi055 ceph-mon[97033]: pgmap v2279: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:51:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:51:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:51:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:51:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:10.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:10.964975+0000) 2022-09-07T22:51:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:10.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:10.855480+0000) 2022-09-07T22:51:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:11.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:11.855631+0000) 2022-09-07T22:51:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:11.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:11.965186+0000) 2022-09-07T22:51:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:12 smithi059 ceph-mon[97450]: pgmap v2280: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:51:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:12 smithi055 ceph-mon[97033]: pgmap v2280: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:51:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:12.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:12.965368+0000) 2022-09-07T22:51:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:12.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:12.855775+0000) 2022-09-07T22:51:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:13.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:13.965597+0000) 2022-09-07T22:51:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:13.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:13.855923+0000) 2022-09-07T22:51:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:14 smithi055 ceph-mon[97033]: pgmap v2281: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:51:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:14 smithi055 ceph-mon[97033]: osdmap e235: 12 total, 11 up, 11 in 2022-09-07T22:51:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:14 smithi059 ceph-mon[97450]: pgmap v2281: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T22:51:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:14 smithi059 ceph-mon[97450]: osdmap e235: 12 total, 11 up, 11 in 2022-09-07T22:51:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:14.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:14.965833+0000) 2022-09-07T22:51:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:14.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:14.856085+0000) 2022-09-07T22:51:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:15.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:15.966029+0000) 2022-09-07T22:51:16.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:15.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:15.856291+0000) 2022-09-07T22:51:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:16 smithi055 ceph-mon[97033]: pgmap v2283: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:51:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:16 smithi059 ceph-mon[97450]: pgmap v2283: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:51:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:16.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:16.966197+0000) 2022-09-07T22:51:17.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:51:16.843+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:51:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:16.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:16.856533+0000) 2022-09-07T22:51:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:17.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:17.966385+0000) 2022-09-07T22:51:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:17.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:17.856687+0000) 2022-09-07T22:51:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:18 smithi055 ceph-mon[97033]: pgmap v2284: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 247 B/s wr, 7 op/s 2022-09-07T22:51:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:18 smithi059 ceph-mon[97450]: pgmap v2284: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 247 B/s wr, 7 op/s 2022-09-07T22:51:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:18.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:18.966612+0000) 2022-09-07T22:51:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:18.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:18.856862+0000) 2022-09-07T22:51:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:19.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:19.966810+0000) 2022-09-07T22:51:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:19.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:19.857031+0000) 2022-09-07T22:51:20.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:20 smithi055 ceph-mon[97033]: pgmap v2285: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:51:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:20 smithi059 ceph-mon[97450]: pgmap v2285: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:51:21.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:20.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:20.966967+0000) 2022-09-07T22:51:21.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:20.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:20.857204+0000) 2022-09-07T22:51:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:21.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:21.967184+0000) 2022-09-07T22:51:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:21.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:21.857439+0000) 2022-09-07T22:51:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:22 smithi055 ceph-mon[97033]: pgmap v2286: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T22:51:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:22 smithi059 ceph-mon[97450]: pgmap v2286: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T22:51:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:22.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:22.967401+0000) 2022-09-07T22:51:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:22.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:22.857700+0000) 2022-09-07T22:51:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:23.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:23.857947+0000) 2022-09-07T22:51:24.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:23.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:23.967573+0000) 2022-09-07T22:51:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:24 smithi055 ceph-mon[97033]: pgmap v2287: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T22:51:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:24 smithi059 ceph-mon[97450]: pgmap v2287: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T22:51:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:24.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:24.967787+0000) 2022-09-07T22:51:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:24.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:24.858211+0000) 2022-09-07T22:51:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:25.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:25.967905+0000) 2022-09-07T22:51:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:25.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:25.858425+0000) 2022-09-07T22:51:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:26 smithi055 ceph-mon[97033]: pgmap v2288: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-07T22:51:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:26 smithi059 ceph-mon[97450]: pgmap v2288: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-07T22:51:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:26.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:26.968109+0000) 2022-09-07T22:51:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:26.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:26.858680+0000) 2022-09-07T22:51:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:27.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:27.968302+0000) 2022-09-07T22:51:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:27.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:27.858901+0000) 2022-09-07T22:51:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:28 smithi055 ceph-mon[97033]: pgmap v2289: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:51:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:28 smithi059 ceph-mon[97450]: pgmap v2289: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:51:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:28.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:28.968528+0000) 2022-09-07T22:51:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:28.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:28.859043+0000) 2022-09-07T22:51:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:29.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:29.968681+0000) 2022-09-07T22:51:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:29.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:29.859239+0000) 2022-09-07T22:51:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:30 smithi055 ceph-mon[97033]: pgmap v2290: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:30 smithi059 ceph-mon[97450]: pgmap v2290: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:30.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:30.859416+0000) 2022-09-07T22:51:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:30.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:30.968891+0000) 2022-09-07T22:51:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:31.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:31.859558+0000) 2022-09-07T22:51:32.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:51:31.844+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:51:32.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:31.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:31.969137+0000) 2022-09-07T22:51:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:32 smithi055 ceph-mon[97033]: pgmap v2291: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:51:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:32 smithi059 ceph-mon[97450]: pgmap v2291: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:51:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:32.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:32.969387+0000) 2022-09-07T22:51:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:32.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:32.859769+0000) 2022-09-07T22:51:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:33.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:33.969536+0000) 2022-09-07T22:51:34.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:33.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:33.859937+0000) 2022-09-07T22:51:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:34 smithi055 ceph-mon[97033]: pgmap v2292: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:34 smithi059 ceph-mon[97450]: pgmap v2292: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:34.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:34.969794+0000) 2022-09-07T22:51:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:34.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:34.860093+0000) 2022-09-07T22:51:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:35.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:35.970068+0000) 2022-09-07T22:51:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:35.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:35.860265+0000) 2022-09-07T22:51:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:36 smithi055 ceph-mon[97033]: pgmap v2293: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-07T22:51:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:36 smithi059 ceph-mon[97450]: pgmap v2293: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-07T22:51:37.162 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:36.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:36.970353+0000) 2022-09-07T22:51:37.164 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:36.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:36.860470+0000) 2022-09-07T22:51:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:37.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:37.970591+0000) 2022-09-07T22:51:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:37.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:37.860742+0000) 2022-09-07T22:51:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:38 smithi055 ceph-mon[97033]: pgmap v2294: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:51:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:38 smithi059 ceph-mon[97450]: pgmap v2294: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:51:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:38.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:38.970785+0000) 2022-09-07T22:51:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:38.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:38.860885+0000) 2022-09-07T22:51:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:39.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:39.970927+0000) 2022-09-07T22:51:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:39.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:39.861099+0000) 2022-09-07T22:51:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:40 smithi055 ceph-mon[97033]: pgmap v2295: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:51:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:40 smithi059 ceph-mon[97450]: pgmap v2295: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:51:41.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:40.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:40.971055+0000) 2022-09-07T22:51:41.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:40.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:40.861351+0000) 2022-09-07T22:51:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:41 smithi055 ceph-mon[97033]: pgmap v2296: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-07T22:51:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:41 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-07T22:51:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:41 smithi059 ceph-mon[97450]: pgmap v2296: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-07T22:51:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:41 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-07T22:51:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:41.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:41.971244+0000) 2022-09-07T22:51:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:41.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:41.861543+0000) 2022-09-07T22:51:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:42.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:42.971466+0000) 2022-09-07T22:51:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:42.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:42.861709+0000) 2022-09-07T22:51:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:43 smithi055 ceph-mon[97033]: pgmap v2297: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-07T22:51:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:43 smithi059 ceph-mon[97450]: pgmap v2297: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-07T22:51:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:43.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:43.971716+0000) 2022-09-07T22:51:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:43.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:43.861899+0000) 2022-09-07T22:51:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:44.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:44.971906+0000) 2022-09-07T22:51:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:44.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:44.862070+0000) 2022-09-07T22:51:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:45 smithi059 ceph-mon[97450]: pgmap v2298: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-07T22:51:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:45 smithi055 ceph-mon[97033]: pgmap v2298: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-07T22:51:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:45.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:45.972097+0000) 2022-09-07T22:51:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:45.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:45.862261+0000) 2022-09-07T22:51:47.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:51:46.859+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:51:47.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:46.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:46.972318+0000) 2022-09-07T22:51:47.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:46.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:46.862449+0000) 2022-09-07T22:51:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:47 smithi059 ceph-mon[97450]: pgmap v2299: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-07T22:51:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:47 smithi055 ceph-mon[97033]: pgmap v2299: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-07T22:51:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:47.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:47.972516+0000) 2022-09-07T22:51:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:47.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:47.862616+0000) 2022-09-07T22:51:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:48.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:48.862848+0000) 2022-09-07T22:51:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:48.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:48.972757+0000) 2022-09-07T22:51:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:49 smithi059 ceph-mon[97450]: pgmap v2300: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-07T22:51:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:49 smithi055 ceph-mon[97033]: pgmap v2300: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-07T22:51:50.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:49.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:49.972902+0000) 2022-09-07T22:51:50.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:49.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:49.863093+0000) 2022-09-07T22:51:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:50.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:50.973150+0000) 2022-09-07T22:51:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:50.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:50.863327+0000) 2022-09-07T22:51:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:51 smithi059 ceph-mon[97450]: pgmap v2301: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-07T22:51:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:51 smithi055 ceph-mon[97033]: pgmap v2301: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-07T22:51:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:51.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:51.973454+0000) 2022-09-07T22:51:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:51.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:51.863461+0000) 2022-09-07T22:51:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:52.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:52.973683+0000) 2022-09-07T22:51:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:52.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:52.863694+0000) 2022-09-07T22:51:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:53 smithi059 ceph-mon[97450]: pgmap v2302: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:53 smithi055 ceph-mon[97033]: pgmap v2302: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:53.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:53.973888+0000) 2022-09-07T22:51:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:53.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:53.863826+0000) 2022-09-07T22:51:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:54.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:54.974109+0000) 2022-09-07T22:51:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:54.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:54.863985+0000) 2022-09-07T22:51:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:55 smithi059 ceph-mon[97450]: pgmap v2303: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:55 smithi055 ceph-mon[97033]: pgmap v2303: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:55.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:55.974255+0000) 2022-09-07T22:51:56.172 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:55.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:55.864204+0000) 2022-09-07T22:51:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:56.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:56.974440+0000) 2022-09-07T22:51:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:56.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:56.864402+0000) 2022-09-07T22:51:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:57 smithi059 ceph-mon[97450]: pgmap v2304: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:51:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:57 smithi055 ceph-mon[97033]: pgmap v2304: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:51:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:57.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:57.974664+0000) 2022-09-07T22:51:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:57.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:57.864577+0000) 2022-09-07T22:51:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:58.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:58.974851+0000) 2022-09-07T22:51:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:58.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:58.864758+0000) 2022-09-07T22:51:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:51:59 smithi059 ceph-mon[97450]: pgmap v2305: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:51:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:51:59 smithi055 ceph-mon[97033]: pgmap v2305: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:51:59.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:59.975022+0000) 2022-09-07T22:52:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:51:59.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:51:59.865007+0000) 2022-09-07T22:52:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:00.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:00.975274+0000) 2022-09-07T22:52:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:00.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:00.865173+0000) 2022-09-07T22:52:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:01 smithi059 ceph-mon[97450]: pgmap v2306: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:01 smithi055 ceph-mon[97033]: pgmap v2306: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:01.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:01.975486+0000) 2022-09-07T22:52:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:01.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:01.865414+0000) 2022-09-07T22:52:02.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:52:01.875+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:52:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:02.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:02.975707+0000) 2022-09-07T22:52:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:02.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:02.865682+0000) 2022-09-07T22:52:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:03 smithi059 ceph-mon[97450]: pgmap v2307: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:03 smithi059 ceph-mon[97450]: osdmap e236: 12 total, 11 up, 11 in 2022-09-07T22:52:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:03 smithi055 ceph-mon[97033]: pgmap v2307: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:03 smithi055 ceph-mon[97033]: osdmap e236: 12 total, 11 up, 11 in 2022-09-07T22:52:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:03.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:03.975957+0000) 2022-09-07T22:52:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:03.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:03.865921+0000) 2022-09-07T22:52:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:52:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:52:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:04 smithi059 ceph-mon[97450]: osdmap e237: 12 total, 11 up, 11 in 2022-09-07T22:52:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:52:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:52:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:04 smithi055 ceph-mon[97033]: osdmap e237: 12 total, 11 up, 11 in 2022-09-07T22:52:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:04.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:04.976171+0000) 2022-09-07T22:52:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:04.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:04.869728+0000) 2022-09-07T22:52:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:05 smithi059 ceph-mon[97450]: pgmap v2309: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:52:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:05 smithi055 ceph-mon[97033]: pgmap v2309: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:52:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:05.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:05.976350+0000) 2022-09-07T22:52:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:05.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:05.869950+0000) 2022-09-07T22:52:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:06.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:06.976565+0000) 2022-09-07T22:52:07.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:06.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:06.870180+0000) 2022-09-07T22:52:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:07 smithi059 ceph-mon[97450]: pgmap v2311: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 66 KiB/s wr, 29 op/s 2022-09-07T22:52:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:07 smithi055 ceph-mon[97033]: pgmap v2311: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 66 KiB/s wr, 29 op/s 2022-09-07T22:52:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:07.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:07.976749+0000) 2022-09-07T22:52:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:07.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:07.870436+0000) 2022-09-07T22:52:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:08.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:08.976967+0000) 2022-09-07T22:52:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:08.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:08.870707+0000) 2022-09-07T22:52:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:09 smithi059 ceph-mon[97450]: pgmap v2312: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 66 KiB/s wr, 26 op/s 2022-09-07T22:52:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:09 smithi055 ceph-mon[97033]: pgmap v2312: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 66 KiB/s wr, 26 op/s 2022-09-07T22:52:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:09.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:09.977233+0000) 2022-09-07T22:52:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:09.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:09.870952+0000) 2022-09-07T22:52:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:10.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:10.977507+0000) 2022-09-07T22:52:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:10.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:10.871181+0000) 2022-09-07T22:52:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:11 smithi055 ceph-mon[97033]: pgmap v2313: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-07T22:52:12.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:11 smithi059 ceph-mon[97450]: pgmap v2313: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-07T22:52:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:11.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:11.977734+0000) 2022-09-07T22:52:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:11.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:11.871435+0000) 2022-09-07T22:52:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:12.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:12.977961+0000) 2022-09-07T22:52:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:12.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:12.871658+0000) 2022-09-07T22:52:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:13 smithi055 ceph-mon[97033]: pgmap v2314: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 63 KiB/s wr, 28 op/s 2022-09-07T22:52:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:13 smithi055 ceph-mon[97033]: osdmap e238: 12 total, 11 up, 11 in 2022-09-07T22:52:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:13 smithi059 ceph-mon[97450]: pgmap v2314: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 63 KiB/s wr, 28 op/s 2022-09-07T22:52:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:13 smithi059 ceph-mon[97450]: osdmap e238: 12 total, 11 up, 11 in 2022-09-07T22:52:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:13.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:13.978136+0000) 2022-09-07T22:52:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:13.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:13.871833+0000) 2022-09-07T22:52:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:14.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:14.978386+0000) 2022-09-07T22:52:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:14.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:14.872086+0000) 2022-09-07T22:52:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:15 smithi055 ceph-mon[97033]: pgmap v2316: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 56 KiB/s wr, 25 op/s 2022-09-07T22:52:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:15 smithi059 ceph-mon[97450]: pgmap v2316: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 56 KiB/s wr, 25 op/s 2022-09-07T22:52:16.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:15.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:15.872348+0000) 2022-09-07T22:52:16.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:15.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:15.978553+0000) 2022-09-07T22:52:17.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:52:16.889+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:52:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:16.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:16.978769+0000) 2022-09-07T22:52:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:16.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:16.872596+0000) 2022-09-07T22:52:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:17 smithi055 ceph-mon[97033]: pgmap v2317: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:52:18.049 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:17 smithi059 ceph-mon[97450]: pgmap v2317: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:52:18.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:17.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:17.979031+0000) 2022-09-07T22:52:18.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:17.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:17.872874+0000) 2022-09-07T22:52:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:18.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:18.979279+0000) 2022-09-07T22:52:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:18.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:18.873173+0000) 2022-09-07T22:52:19.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:19 smithi055 ceph-mon[97033]: pgmap v2318: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:52:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:19 smithi059 ceph-mon[97450]: pgmap v2318: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:52:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:19.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:19.979527+0000) 2022-09-07T22:52:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:19.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:19.873394+0000) 2022-09-07T22:52:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:20.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:20.979762+0000) 2022-09-07T22:52:21.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:20.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:20.873701+0000) 2022-09-07T22:52:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:21 smithi055 ceph-mon[97033]: pgmap v2319: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:52:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:21 smithi059 ceph-mon[97450]: pgmap v2319: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:52:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:21.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:21.979984+0000) 2022-09-07T22:52:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:21.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:21.873938+0000) 2022-09-07T22:52:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:22.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:22.980225+0000) 2022-09-07T22:52:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:22.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:22.874193+0000) 2022-09-07T22:52:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:23 smithi055 ceph-mon[97033]: pgmap v2320: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:52:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:23 smithi059 ceph-mon[97450]: pgmap v2320: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:52:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:23.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:23.980413+0000) 2022-09-07T22:52:24.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:23.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:23.874450+0000) 2022-09-07T22:52:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:24.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:24.980588+0000) 2022-09-07T22:52:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:24.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:24.874760+0000) 2022-09-07T22:52:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:25 smithi055 ceph-mon[97033]: pgmap v2321: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:52:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:25 smithi059 ceph-mon[97450]: pgmap v2321: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:52:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:25.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:25.980822+0000) 2022-09-07T22:52:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:25.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:25.874978+0000) 2022-09-07T22:52:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:26.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:26.981052+0000) 2022-09-07T22:52:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:26.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:26.875170+0000) 2022-09-07T22:52:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:27 smithi055 ceph-mon[97033]: pgmap v2322: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:27 smithi059 ceph-mon[97450]: pgmap v2322: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:27.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:27.981257+0000) 2022-09-07T22:52:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:27.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:27.875394+0000) 2022-09-07T22:52:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:28.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:28.981506+0000) 2022-09-07T22:52:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:28.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:28.875690+0000) 2022-09-07T22:52:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:29 smithi055 ceph-mon[97033]: pgmap v2323: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:29 smithi059 ceph-mon[97450]: pgmap v2323: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:29.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:29.981747+0000) 2022-09-07T22:52:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:29.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:29.875919+0000) 2022-09-07T22:52:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:30.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:30.876089+0000) 2022-09-07T22:52:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:30.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:30.981883+0000) 2022-09-07T22:52:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:31 smithi055 ceph-mon[97033]: pgmap v2324: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:31 smithi059 ceph-mon[97450]: pgmap v2324: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:32.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:52:31.905+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:52:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:31.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:31.982010+0000) 2022-09-07T22:52:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:31.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:31.876320+0000) 2022-09-07T22:52:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:32.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:32.982236+0000) 2022-09-07T22:52:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:32.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:32.876566+0000) 2022-09-07T22:52:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:33 smithi055 ceph-mon[97033]: pgmap v2325: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:33 smithi059 ceph-mon[97450]: pgmap v2325: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:33.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:33.982459+0000) 2022-09-07T22:52:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:33.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:33.876760+0000) 2022-09-07T22:52:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:34.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:34.982699+0000) 2022-09-07T22:52:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:34.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:34.876915+0000) 2022-09-07T22:52:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:35 smithi055 ceph-mon[97033]: pgmap v2326: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:35 smithi059 ceph-mon[97450]: pgmap v2326: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:35.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:35.982838+0000) 2022-09-07T22:52:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:35.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:35.877045+0000) 2022-09-07T22:52:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:36.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:36.982963+0000) 2022-09-07T22:52:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:36.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:36.877262+0000) 2022-09-07T22:52:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:37 smithi055 ceph-mon[97033]: pgmap v2327: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:37 smithi059 ceph-mon[97450]: pgmap v2327: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:37.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:37.983142+0000) 2022-09-07T22:52:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:37.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:37.877467+0000) 2022-09-07T22:52:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:38.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:38.983328+0000) 2022-09-07T22:52:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:38.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:38.877715+0000) 2022-09-07T22:52:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:39 smithi055 ceph-mon[97033]: pgmap v2328: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:39 smithi059 ceph-mon[97450]: pgmap v2328: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:39.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:39.983514+0000) 2022-09-07T22:52:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:39.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:39.877933+0000) 2022-09-07T22:52:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:52:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:52:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:52:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:52:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:52:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:52:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:40.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:40.983681+0000) 2022-09-07T22:52:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:40.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:40.878153+0000) 2022-09-07T22:52:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:41 smithi055 ceph-mon[97033]: pgmap v2329: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:41 smithi059 ceph-mon[97450]: pgmap v2329: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:41.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:41.983845+0000) 2022-09-07T22:52:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:41.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:41.878374+0000) 2022-09-07T22:52:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:42.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:42.984128+0000) 2022-09-07T22:52:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:42.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:42.878594+0000) 2022-09-07T22:52:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:43 smithi055 ceph-mon[97033]: pgmap v2330: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:43 smithi059 ceph-mon[97450]: pgmap v2330: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:43.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:43.984330+0000) 2022-09-07T22:52:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:43.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:43.878867+0000) 2022-09-07T22:52:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:44.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:44.984557+0000) 2022-09-07T22:52:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:44.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:44.879069+0000) 2022-09-07T22:52:45.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:45 smithi055 ceph-mon[97033]: pgmap v2331: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:45 smithi059 ceph-mon[97450]: pgmap v2331: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:45.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:45.984787+0000) 2022-09-07T22:52:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:45.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:45.879248+0000) 2022-09-07T22:52:47.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:52:46.921+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:52:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:46.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:46.985025+0000) 2022-09-07T22:52:47.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:46.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:46.879431+0000) 2022-09-07T22:52:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:47 smithi055 ceph-mon[97033]: pgmap v2332: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:47 smithi059 ceph-mon[97450]: pgmap v2332: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:47.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:47.985195+0000) 2022-09-07T22:52:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:47.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:47.879655+0000) 2022-09-07T22:52:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:48.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:48.985539+0000) 2022-09-07T22:52:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:48.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:48.879837+0000) 2022-09-07T22:52:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:49 smithi055 ceph-mon[97033]: pgmap v2333: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:49 smithi059 ceph-mon[97450]: pgmap v2333: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:49.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:49.880007+0000) 2022-09-07T22:52:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:49.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:49.985739+0000) 2022-09-07T22:52:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:50.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:50.985967+0000) 2022-09-07T22:52:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:50.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:50.880250+0000) 2022-09-07T22:52:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:51 smithi055 ceph-mon[97033]: pgmap v2334: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:52.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:51 smithi059 ceph-mon[97450]: pgmap v2334: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:51.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:51.986215+0000) 2022-09-07T22:52:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:51.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:51.880454+0000) 2022-09-07T22:52:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:52.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:52.986383+0000) 2022-09-07T22:52:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:52.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:52.880722+0000) 2022-09-07T22:52:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:53 smithi055 ceph-mon[97033]: pgmap v2335: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:53 smithi059 ceph-mon[97450]: pgmap v2335: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:53.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:53.986604+0000) 2022-09-07T22:52:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:53.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:53.880960+0000) 2022-09-07T22:52:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:54.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:54.881167+0000) 2022-09-07T22:52:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:54.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:54.986861+0000) 2022-09-07T22:52:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:55 smithi055 ceph-mon[97033]: pgmap v2336: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:55 smithi059 ceph-mon[97450]: pgmap v2336: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:52:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:55.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:55.987088+0000) 2022-09-07T22:52:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:55.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:55.881371+0000) 2022-09-07T22:52:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:56.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:56.881627+0000) 2022-09-07T22:52:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:56.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:56.987259+0000) 2022-09-07T22:52:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:57 smithi055 ceph-mon[97033]: pgmap v2337: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:57 smithi059 ceph-mon[97450]: pgmap v2337: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:52:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:57.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:57.881867+0000) 2022-09-07T22:52:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:57.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:57.987473+0000) 2022-09-07T22:52:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:58.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:58.882039+0000) 2022-09-07T22:52:59.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:58.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:58.987735+0000) 2022-09-07T22:52:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:52:59 smithi055 ceph-mon[97033]: pgmap v2338: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:52:59 smithi059 ceph-mon[97450]: pgmap v2338: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:52:59.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:59.987957+0000) 2022-09-07T22:53:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:52:59.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:52:59.882298+0000) 2022-09-07T22:53:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:00.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:00.988201+0000) 2022-09-07T22:53:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:00.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:00.882522+0000) 2022-09-07T22:53:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:01 smithi055 ceph-mon[97033]: pgmap v2339: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:53:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:01 smithi059 ceph-mon[97450]: pgmap v2339: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:53:02.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:53:01.935+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:53:02.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:01.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:01.882791+0000) 2022-09-07T22:53:02.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:01.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:01.988436+0000) 2022-09-07T22:53:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:02.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:02.988684+0000) 2022-09-07T22:53:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:02.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:02.882960+0000) 2022-09-07T22:53:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:03 smithi055 ceph-mon[97033]: pgmap v2340: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:03 smithi059 ceph-mon[97450]: pgmap v2340: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:03.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:03.988897+0000) 2022-09-07T22:53:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:03.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:03.883175+0000) 2022-09-07T22:53:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:53:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:53:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:53:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:53:05.136 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:04.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:04.883401+0000) 2022-09-07T22:53:05.137 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:04.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:04.989140+0000) 2022-09-07T22:53:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:05 smithi055 ceph-mon[97033]: pgmap v2341: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:05 smithi059 ceph-mon[97450]: pgmap v2341: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:05.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:05.989427+0000) 2022-09-07T22:53:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:05.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:05.883647+0000) 2022-09-07T22:53:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:06.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:06.883916+0000) 2022-09-07T22:53:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:06.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:06.989682+0000) 2022-09-07T22:53:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:07 smithi055 ceph-mon[97033]: pgmap v2342: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:53:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:07 smithi059 ceph-mon[97450]: pgmap v2342: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:53:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:07.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:07.989905+0000) 2022-09-07T22:53:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:07.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:07.884201+0000) 2022-09-07T22:53:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:08.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:08.884487+0000) 2022-09-07T22:53:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:08.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:08.990069+0000) 2022-09-07T22:53:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:09 smithi055 ceph-mon[97033]: pgmap v2343: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:53:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:09 smithi055 ceph-mon[97033]: osdmap e239: 12 total, 11 up, 11 in 2022-09-07T22:53:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:09 smithi059 ceph-mon[97450]: pgmap v2343: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:53:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:09 smithi059 ceph-mon[97450]: osdmap e239: 12 total, 11 up, 11 in 2022-09-07T22:53:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:09.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:09.990279+0000) 2022-09-07T22:53:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:09.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:09.884704+0000) 2022-09-07T22:53:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:10.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:10.990569+0000) 2022-09-07T22:53:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:10.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:10.884938+0000) 2022-09-07T22:53:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:11 smithi055 ceph-mon[97033]: pgmap v2345: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:53:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:11 smithi055 ceph-mon[97033]: pgmap v2346: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-07T22:53:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:11 smithi055 ceph-mon[97033]: pgmap v2347: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-07T22:53:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:53:11.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:11 smithi059 ceph-mon[97450]: pgmap v2345: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:53:11.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:11 smithi059 ceph-mon[97450]: pgmap v2346: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-07T22:53:11.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:11 smithi059 ceph-mon[97450]: pgmap v2347: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-07T22:53:11.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:53:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:11.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:11.990747+0000) 2022-09-07T22:53:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:11.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:11.885197+0000) 2022-09-07T22:53:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:12.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:12.990957+0000) 2022-09-07T22:53:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:12.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:12.885449+0000) 2022-09-07T22:53:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:13 smithi055 ceph-mon[97033]: pgmap v2348: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T22:53:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:13 smithi059 ceph-mon[97450]: pgmap v2348: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T22:53:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:13.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:13.991156+0000) 2022-09-07T22:53:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:13.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:13.885618+0000) 2022-09-07T22:53:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:14 smithi055 ceph-mon[97033]: osdmap e240: 12 total, 11 up, 11 in 2022-09-07T22:53:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:14 smithi059 ceph-mon[97450]: osdmap e240: 12 total, 11 up, 11 in 2022-09-07T22:53:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:14.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:14.991368+0000) 2022-09-07T22:53:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:14.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:14.885827+0000) 2022-09-07T22:53:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:15 smithi055 ceph-mon[97033]: pgmap v2350: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 185 B/s wr, 5 op/s 2022-09-07T22:53:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:15 smithi059 ceph-mon[97450]: pgmap v2350: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 185 B/s wr, 5 op/s 2022-09-07T22:53:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:15.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:15.886079+0000) 2022-09-07T22:53:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:15.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:15.991580+0000) 2022-09-07T22:53:17.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:53:16.943+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:53:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:16.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:16.991755+0000) 2022-09-07T22:53:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:16.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:16.886289+0000) 2022-09-07T22:53:17.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:17 smithi055 ceph-mon[97033]: pgmap v2351: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T22:53:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:17 smithi059 ceph-mon[97450]: pgmap v2351: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T22:53:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:17.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:17.991907+0000) 2022-09-07T22:53:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:17.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:17.886510+0000) 2022-09-07T22:53:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:18.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:18.992104+0000) 2022-09-07T22:53:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:18.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:18.886792+0000) 2022-09-07T22:53:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:19 smithi055 ceph-mon[97033]: pgmap v2352: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:53:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:19 smithi059 ceph-mon[97450]: pgmap v2352: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:53:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:19.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:19.992356+0000) 2022-09-07T22:53:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:19.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:19.887009+0000) 2022-09-07T22:53:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:20.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:20.992565+0000) 2022-09-07T22:53:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:20.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:20.887252+0000) 2022-09-07T22:53:21.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:21 smithi055 ceph-mon[97033]: pgmap v2353: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:53:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:21 smithi059 ceph-mon[97450]: pgmap v2353: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:53:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:21.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:21.992801+0000) 2022-09-07T22:53:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:21.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:21.887534+0000) 2022-09-07T22:53:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:22.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:22.992936+0000) 2022-09-07T22:53:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:22.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:22.887789+0000) 2022-09-07T22:53:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:23 smithi055 ceph-mon[97033]: pgmap v2354: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:53:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:23 smithi059 ceph-mon[97450]: pgmap v2354: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:53:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:23.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:23.993182+0000) 2022-09-07T22:53:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:23.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:23.888061+0000) 2022-09-07T22:53:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:24.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:24.993428+0000) 2022-09-07T22:53:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:24.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:24.888265+0000) 2022-09-07T22:53:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:25 smithi059 ceph-mon[97450]: pgmap v2355: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 95 B/s wr, 5 op/s 2022-09-07T22:53:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:25 smithi055 ceph-mon[97033]: pgmap v2355: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 95 B/s wr, 5 op/s 2022-09-07T22:53:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:25.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:25.993681+0000) 2022-09-07T22:53:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:25.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:25.888485+0000) 2022-09-07T22:53:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:26.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:26.993855+0000) 2022-09-07T22:53:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:26.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:26.888766+0000) 2022-09-07T22:53:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:27 smithi059 ceph-mon[97450]: pgmap v2356: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:53:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:27 smithi055 ceph-mon[97033]: pgmap v2356: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:53:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:27.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:27.889060+0000) 2022-09-07T22:53:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:27.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:27.994084+0000) 2022-09-07T22:53:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:28.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:28.889356+0000) 2022-09-07T22:53:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:28.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:28.994302+0000) 2022-09-07T22:53:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:29 smithi059 ceph-mon[97450]: pgmap v2357: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:29 smithi055 ceph-mon[97033]: pgmap v2357: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:29.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:29.994521+0000) 2022-09-07T22:53:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:29.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:29.889617+0000) 2022-09-07T22:53:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:30.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:30.889915+0000) 2022-09-07T22:53:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:30.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:30.994759+0000) 2022-09-07T22:53:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:31 smithi059 ceph-mon[97450]: pgmap v2358: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:53:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:31 smithi055 ceph-mon[97033]: pgmap v2358: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:53:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:31.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:31.890165+0000) 2022-09-07T22:53:32.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:53:31.950+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:53:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:31.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:31.994934+0000) 2022-09-07T22:53:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:32.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:32.995125+0000) 2022-09-07T22:53:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:32.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:32.890433+0000) 2022-09-07T22:53:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:33 smithi059 ceph-mon[97450]: pgmap v2359: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:33 smithi055 ceph-mon[97033]: pgmap v2359: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:33.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:33.995364+0000) 2022-09-07T22:53:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:33.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:33.890680+0000) 2022-09-07T22:53:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:34.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:34.995597+0000) 2022-09-07T22:53:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:34.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:34.890916+0000) 2022-09-07T22:53:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:35 smithi059 ceph-mon[97450]: pgmap v2360: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:35 smithi055 ceph-mon[97033]: pgmap v2360: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:35.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:35.995838+0000) 2022-09-07T22:53:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:35.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:35.891119+0000) 2022-09-07T22:53:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:36.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:36.995993+0000) 2022-09-07T22:53:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:36.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:36.891413+0000) 2022-09-07T22:53:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:37 smithi059 ceph-mon[97450]: pgmap v2361: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:53:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:37 smithi055 ceph-mon[97033]: pgmap v2361: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:53:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:37.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:37.996223+0000) 2022-09-07T22:53:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:37.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:37.891657+0000) 2022-09-07T22:53:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:38.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:38.996443+0000) 2022-09-07T22:53:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:38.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:38.891907+0000) 2022-09-07T22:53:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:39 smithi059 ceph-mon[97450]: pgmap v2362: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:39 smithi055 ceph-mon[97033]: pgmap v2362: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:39.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:39.996712+0000) 2022-09-07T22:53:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:39.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:39.892173+0000) 2022-09-07T22:53:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:40.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:40.996894+0000) 2022-09-07T22:53:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:40.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:40.892364+0000) 2022-09-07T22:53:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:41 smithi059 ceph-mon[97450]: pgmap v2363: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:53:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:41 smithi055 ceph-mon[97033]: pgmap v2363: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:53:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:41.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:41.997058+0000) 2022-09-07T22:53:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:41.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:41.892622+0000) 2022-09-07T22:53:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:42.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:42.892894+0000) 2022-09-07T22:53:43.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:42.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:42.997254+0000) 2022-09-07T22:53:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:43 smithi059 ceph-mon[97450]: pgmap v2364: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:43 smithi055 ceph-mon[97033]: pgmap v2364: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:43.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:43.893156+0000) 2022-09-07T22:53:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:43.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:43.997482+0000) 2022-09-07T22:53:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:44.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:44.997742+0000) 2022-09-07T22:53:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:44.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:44.893423+0000) 2022-09-07T22:53:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:45 smithi059 ceph-mon[97450]: pgmap v2365: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:45 smithi055 ceph-mon[97033]: pgmap v2365: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:45.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:45.997989+0000) 2022-09-07T22:53:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:45.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:45.893716+0000) 2022-09-07T22:53:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:46.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:46.998188+0000) 2022-09-07T22:53:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:46.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:46.894003+0000) 2022-09-07T22:53:47.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:53:46.954+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:53:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:47 smithi059 ceph-mon[97450]: pgmap v2366: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:53:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:47 smithi055 ceph-mon[97033]: pgmap v2366: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:53:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:47.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:47.998412+0000) 2022-09-07T22:53:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:47.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:47.894225+0000) 2022-09-07T22:53:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:48.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:48.998575+0000) 2022-09-07T22:53:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:48.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:48.894486+0000) 2022-09-07T22:53:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:49 smithi059 ceph-mon[97450]: pgmap v2367: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:49 smithi055 ceph-mon[97033]: pgmap v2367: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:49.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:49.998787+0000) 2022-09-07T22:53:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:49.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:49.894769+0000) 2022-09-07T22:53:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:50.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:50.998982+0000) 2022-09-07T22:53:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:50.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:50.894989+0000) 2022-09-07T22:53:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:51 smithi059 ceph-mon[97450]: pgmap v2368: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:53:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:51 smithi055 ceph-mon[97033]: pgmap v2368: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:53:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:51.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:51.999166+0000) 2022-09-07T22:53:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:51.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:51.895251+0000) 2022-09-07T22:53:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:52.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:52.999399+0000) 2022-09-07T22:53:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:52.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:52.895542+0000) 2022-09-07T22:53:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:53 smithi059 ceph-mon[97450]: pgmap v2369: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:53.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:53 smithi055 ceph-mon[97033]: pgmap v2369: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:53.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:53.999617+0000) 2022-09-07T22:53:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:53.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:53.895746+0000) 2022-09-07T22:53:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:54.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:54.999868+0000) 2022-09-07T22:53:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:54.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:54.896003+0000) 2022-09-07T22:53:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:55 smithi059 ceph-mon[97450]: pgmap v2370: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:55 smithi055 ceph-mon[97033]: pgmap v2370: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:53:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:55.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:56.000095+0000) 2022-09-07T22:53:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:55.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:55.896227+0000) 2022-09-07T22:53:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:56.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:57.000238+0000) 2022-09-07T22:53:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:56.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:56.896452+0000) 2022-09-07T22:53:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:57 smithi059 ceph-mon[97450]: pgmap v2371: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:53:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:57 smithi055 ceph-mon[97033]: pgmap v2371: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:53:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:57.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:58.000462+0000) 2022-09-07T22:53:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:57.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:57.896614+0000) 2022-09-07T22:53:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:58.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:59.000595+0000) 2022-09-07T22:53:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:58.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:58.896827+0000) 2022-09-07T22:53:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:53:59 smithi055 ceph-mon[97033]: pgmap v2372: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:53:59 smithi059 ceph-mon[97450]: pgmap v2372: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:53:59.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:00.000792+0000) 2022-09-07T22:54:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:53:59.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:53:59.897089+0000) 2022-09-07T22:54:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:01.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:01.000998+0000) 2022-09-07T22:54:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:00.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:00.897367+0000) 2022-09-07T22:54:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:01 smithi055 ceph-mon[97033]: pgmap v2373: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:54:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:01 smithi059 ceph-mon[97450]: pgmap v2373: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:54:02.158 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:54:01.969+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:54:02.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:02.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:02.001146+0000) 2022-09-07T22:54:02.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:01.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:01.897582+0000) 2022-09-07T22:54:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:03.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:03.001355+0000) 2022-09-07T22:54:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:02.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:02.897805+0000) 2022-09-07T22:54:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:03 smithi055 ceph-mon[97033]: pgmap v2374: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:03 smithi059 ceph-mon[97450]: pgmap v2374: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:04.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:04.001581+0000) 2022-09-07T22:54:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:03.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:03.898008+0000) 2022-09-07T22:54:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:54:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:54:05.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:54:05.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:54:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:05.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:05.001746+0000) 2022-09-07T22:54:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:04.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:04.898271+0000) 2022-09-07T22:54:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:05 smithi055 ceph-mon[97033]: pgmap v2375: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:05 smithi059 ceph-mon[97450]: pgmap v2375: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:06.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:06.001880+0000) 2022-09-07T22:54:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:05.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:05.898483+0000) 2022-09-07T22:54:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:07.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:07.002102+0000) 2022-09-07T22:54:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:06.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:06.898764+0000) 2022-09-07T22:54:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:07 smithi055 ceph-mon[97033]: pgmap v2376: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:54:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:07 smithi059 ceph-mon[97450]: pgmap v2376: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:54:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:08.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:08.002330+0000) 2022-09-07T22:54:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:07.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:07.899053+0000) 2022-09-07T22:54:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:09.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:09.002601+0000) 2022-09-07T22:54:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:08.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:08.899261+0000) 2022-09-07T22:54:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:09 smithi055 ceph-mon[97033]: pgmap v2377: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:54:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:09 smithi055 ceph-mon[97033]: osdmap e241: 12 total, 11 up, 11 in 2022-09-07T22:54:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:09 smithi059 ceph-mon[97450]: pgmap v2377: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:54:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:09 smithi059 ceph-mon[97450]: osdmap e241: 12 total, 11 up, 11 in 2022-09-07T22:54:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:10.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:10.002882+0000) 2022-09-07T22:54:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:09.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:09.899448+0000) 2022-09-07T22:54:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:54:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:54:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:54:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:54:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:54:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:54:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:10.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:10.899647+0000) 2022-09-07T22:54:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:11.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:11.003147+0000) 2022-09-07T22:54:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:11 smithi055 ceph-mon[97033]: pgmap v2379: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:54:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:11 smithi059 ceph-mon[97450]: pgmap v2379: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:54:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:12.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:12.003363+0000) 2022-09-07T22:54:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:11.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:11.899786+0000) 2022-09-07T22:54:13.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:13.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:13.003530+0000) 2022-09-07T22:54:13.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:12.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:12.899913+0000) 2022-09-07T22:54:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:13 smithi055 ceph-mon[97033]: pgmap v2380: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:54:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:13 smithi055 ceph-mon[97033]: osdmap e242: 12 total, 11 up, 11 in 2022-09-07T22:54:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:13 smithi059 ceph-mon[97450]: pgmap v2380: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:54:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:13 smithi059 ceph-mon[97450]: osdmap e242: 12 total, 11 up, 11 in 2022-09-07T22:54:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:14.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:14.003751+0000) 2022-09-07T22:54:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:13.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:13.900077+0000) 2022-09-07T22:54:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:14.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:14.900315+0000) 2022-09-07T22:54:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:15.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:15.004009+0000) 2022-09-07T22:54:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:15 smithi055 ceph-mon[97033]: pgmap v2382: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:54:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:15 smithi059 ceph-mon[97450]: pgmap v2382: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:54:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:16.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:16.004257+0000) 2022-09-07T22:54:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:15.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:15.900521+0000) 2022-09-07T22:54:17.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:54:16.980+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:54:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:17.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:17.004492+0000) 2022-09-07T22:54:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:16.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:16.900780+0000) 2022-09-07T22:54:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:17 smithi055 ceph-mon[97033]: pgmap v2383: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:54:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:17 smithi059 ceph-mon[97450]: pgmap v2383: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:54:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:18.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:18.004742+0000) 2022-09-07T22:54:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:17.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:17.900956+0000) 2022-09-07T22:54:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:18.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:18.901225+0000) 2022-09-07T22:54:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:19.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:19.005025+0000) 2022-09-07T22:54:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:19 smithi055 ceph-mon[97033]: pgmap v2384: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-07T22:54:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:19 smithi059 ceph-mon[97450]: pgmap v2384: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-07T22:54:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:19.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:19.901483+0000) 2022-09-07T22:54:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:20.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:20.005288+0000) 2022-09-07T22:54:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:20.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:20.901690+0000) 2022-09-07T22:54:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:21.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:21.005493+0000) 2022-09-07T22:54:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:21 smithi055 ceph-mon[97033]: pgmap v2385: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:54:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:21 smithi059 ceph-mon[97450]: pgmap v2385: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:54:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:21.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:21.901915+0000) 2022-09-07T22:54:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:22.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:22.005727+0000) 2022-09-07T22:54:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:23.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:23.006003+0000) 2022-09-07T22:54:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:22.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:22.902151+0000) 2022-09-07T22:54:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:23 smithi055 ceph-mon[97033]: pgmap v2386: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:54:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:23 smithi059 ceph-mon[97450]: pgmap v2386: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:54:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:24.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:24.006288+0000) 2022-09-07T22:54:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:23.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:23.902386+0000) 2022-09-07T22:54:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:25.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:25.006508+0000) 2022-09-07T22:54:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:24.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:24.902682+0000) 2022-09-07T22:54:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:25 smithi055 ceph-mon[97033]: pgmap v2387: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 95 B/s wr, 5 op/s 2022-09-07T22:54:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:25 smithi059 ceph-mon[97450]: pgmap v2387: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 95 B/s wr, 5 op/s 2022-09-07T22:54:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:25.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:25.902913+0000) 2022-09-07T22:54:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:26.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:26.006773+0000) 2022-09-07T22:54:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:27.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:27.006946+0000) 2022-09-07T22:54:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:26.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:26.903114+0000) 2022-09-07T22:54:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:27 smithi055 ceph-mon[97033]: pgmap v2388: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:54:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:27 smithi059 ceph-mon[97450]: pgmap v2388: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:54:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:28.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:28.007158+0000) 2022-09-07T22:54:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:27.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:27.903290+0000) 2022-09-07T22:54:29.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:29.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:29.007425+0000) 2022-09-07T22:54:29.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:28.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:28.903541+0000) 2022-09-07T22:54:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:29 smithi055 ceph-mon[97033]: pgmap v2389: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:29 smithi059 ceph-mon[97450]: pgmap v2389: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:30.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:30.007679+0000) 2022-09-07T22:54:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:29.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:29.903720+0000) 2022-09-07T22:54:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:31.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:31.007944+0000) 2022-09-07T22:54:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:30.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:30.903928+0000) 2022-09-07T22:54:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:31 smithi055 ceph-mon[97033]: pgmap v2390: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:54:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:31 smithi059 ceph-mon[97450]: pgmap v2390: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:54:32.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:54:31.989+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:54:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:31.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:31.904177+0000) 2022-09-07T22:54:32.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:32.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:32.008194+0000) 2022-09-07T22:54:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:33.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:33.008482+0000) 2022-09-07T22:54:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:32.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:32.904424+0000) 2022-09-07T22:54:33.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:33 smithi055 ceph-mon[97033]: pgmap v2391: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:33 smithi059 ceph-mon[97450]: pgmap v2391: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:34.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:34.008721+0000) 2022-09-07T22:54:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:33.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:33.904692+0000) 2022-09-07T22:54:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:35.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:35.008847+0000) 2022-09-07T22:54:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:34.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:34.904846+0000) 2022-09-07T22:54:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:35 smithi055 ceph-mon[97033]: pgmap v2392: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:35 smithi059 ceph-mon[97450]: pgmap v2392: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:36.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:36.009061+0000) 2022-09-07T22:54:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:35.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:35.904994+0000) 2022-09-07T22:54:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:37.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:37.009300+0000) 2022-09-07T22:54:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:36.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:36.905211+0000) 2022-09-07T22:54:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:37 smithi055 ceph-mon[97033]: pgmap v2393: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:54:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:37 smithi059 ceph-mon[97450]: pgmap v2393: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:54:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:38.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:38.009535+0000) 2022-09-07T22:54:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:37.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:37.905447+0000) 2022-09-07T22:54:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:39.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:39.009768+0000) 2022-09-07T22:54:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:38.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:38.905720+0000) 2022-09-07T22:54:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:39 smithi055 ceph-mon[97033]: pgmap v2394: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:39 smithi059 ceph-mon[97450]: pgmap v2394: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:40.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:40.010044+0000) 2022-09-07T22:54:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:39.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:39.905906+0000) 2022-09-07T22:54:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:41.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:41.010341+0000) 2022-09-07T22:54:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:40.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:40.906037+0000) 2022-09-07T22:54:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:41 smithi055 ceph-mon[97033]: pgmap v2395: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:54:41.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:41 smithi055 ceph-mon[97033]: pgmap v2396: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:54:41.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:41 smithi055 ceph-mon[97033]: pgmap v2397: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:54:41.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:54:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:41 smithi059 ceph-mon[97450]: pgmap v2395: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:54:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:41 smithi059 ceph-mon[97450]: pgmap v2396: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:54:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:41 smithi059 ceph-mon[97450]: pgmap v2397: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:54:41.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:54:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:42.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:42.010602+0000) 2022-09-07T22:54:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:41.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:41.906252+0000) 2022-09-07T22:54:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:43.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:43.010769+0000) 2022-09-07T22:54:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:42.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:42.906527+0000) 2022-09-07T22:54:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:43 smithi055 ceph-mon[97033]: pgmap v2398: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:54:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:43 smithi059 ceph-mon[97450]: pgmap v2398: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:54:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:44.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:44.011024+0000) 2022-09-07T22:54:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:43.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:43.906707+0000) 2022-09-07T22:54:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:45.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:45.011190+0000) 2022-09-07T22:54:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:44.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:44.906904+0000) 2022-09-07T22:54:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:45 smithi059 ceph-mon[97450]: pgmap v2399: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:54:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:45 smithi055 ceph-mon[97033]: pgmap v2399: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:54:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:46.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:46.011404+0000) 2022-09-07T22:54:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:45.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:45.907150+0000) 2022-09-07T22:54:47.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:54:47.004+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:54:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:47.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:47.011568+0000) 2022-09-07T22:54:47.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:46.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:46.907414+0000) 2022-09-07T22:54:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:47 smithi059 ceph-mon[97450]: pgmap v2400: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:54:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:47 smithi055 ceph-mon[97033]: pgmap v2400: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:54:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:48.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:48.011795+0000) 2022-09-07T22:54:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:47.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:47.907677+0000) 2022-09-07T22:54:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:49.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:49.011960+0000) 2022-09-07T22:54:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:48.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:48.907927+0000) 2022-09-07T22:54:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:49 smithi059 ceph-mon[97450]: pgmap v2401: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:54:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:49 smithi055 ceph-mon[97033]: pgmap v2401: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:54:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:49.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:49.908158+0000) 2022-09-07T22:54:50.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:50.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:50.012187+0000) 2022-09-07T22:54:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:51.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:51.012408+0000) 2022-09-07T22:54:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:50.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:50.908352+0000) 2022-09-07T22:54:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:51 smithi059 ceph-mon[97450]: pgmap v2402: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:54:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:51 smithi055 ceph-mon[97033]: pgmap v2402: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:54:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:52.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:52.012684+0000) 2022-09-07T22:54:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:51.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:51.908580+0000) 2022-09-07T22:54:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:53.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:53.012869+0000) 2022-09-07T22:54:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:52.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:52.908771+0000) 2022-09-07T22:54:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:53 smithi059 ceph-mon[97450]: pgmap v2403: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:53 smithi055 ceph-mon[97033]: pgmap v2403: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:54.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:54.013105+0000) 2022-09-07T22:54:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:53.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:53.908975+0000) 2022-09-07T22:54:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:55.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:55.013295+0000) 2022-09-07T22:54:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:54.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:54.909288+0000) 2022-09-07T22:54:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:55 smithi059 ceph-mon[97450]: pgmap v2404: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:55 smithi055 ceph-mon[97033]: pgmap v2404: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:56.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:56.013483+0000) 2022-09-07T22:54:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:55.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:55.909541+0000) 2022-09-07T22:54:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:57.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:57.013746+0000) 2022-09-07T22:54:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:56.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:56.909792+0000) 2022-09-07T22:54:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:57 smithi059 ceph-mon[97450]: pgmap v2405: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:54:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:57 smithi055 ceph-mon[97033]: pgmap v2405: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:54:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:58.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:58.013928+0000) 2022-09-07T22:54:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:57.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:57.909965+0000) 2022-09-07T22:54:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:54:59.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:59.014117+0000) 2022-09-07T22:54:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:58.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:58.910173+0000) 2022-09-07T22:54:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:54:59 smithi059 ceph-mon[97450]: pgmap v2406: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:54:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:54:59 smithi055 ceph-mon[97033]: pgmap v2406: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:00.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:00.014335+0000) 2022-09-07T22:55:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:54:59.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:54:59.910361+0000) 2022-09-07T22:55:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:01.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:01.014485+0000) 2022-09-07T22:55:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:00.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:00.910518+0000) 2022-09-07T22:55:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:01 smithi059 ceph-mon[97450]: pgmap v2407: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:55:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:01 smithi055 ceph-mon[97033]: pgmap v2407: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:55:02.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:55:02.020+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:55:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:02.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:02.014746+0000) 2022-09-07T22:55:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:01.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:01.910740+0000) 2022-09-07T22:55:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:02.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:02.910922+0000) 2022-09-07T22:55:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:03.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:03.014905+0000) 2022-09-07T22:55:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:03 smithi059 ceph-mon[97450]: pgmap v2408: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:03 smithi055 ceph-mon[97033]: pgmap v2408: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:03.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:03.911090+0000) 2022-09-07T22:55:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:04.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:04.015062+0000) 2022-09-07T22:55:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:55:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:55:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:55:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:55:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:05.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:05.015232+0000) 2022-09-07T22:55:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:04.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:04.911322+0000) 2022-09-07T22:55:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:05 smithi055 ceph-mon[97033]: pgmap v2409: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:05 smithi059 ceph-mon[97450]: pgmap v2409: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:06.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:06.015419+0000) 2022-09-07T22:55:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:05.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:05.911552+0000) 2022-09-07T22:55:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:07.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:07.015550+0000) 2022-09-07T22:55:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:06.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:06.911838+0000) 2022-09-07T22:55:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:07 smithi055 ceph-mon[97033]: pgmap v2410: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:55:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:07 smithi059 ceph-mon[97450]: pgmap v2410: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:55:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:08.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:08.015752+0000) 2022-09-07T22:55:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:07.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:07.912115+0000) 2022-09-07T22:55:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:09.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:09.015992+0000) 2022-09-07T22:55:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:08.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:08.912389+0000) 2022-09-07T22:55:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:09 smithi055 ceph-mon[97033]: pgmap v2411: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:55:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:09 smithi055 ceph-mon[97033]: osdmap e243: 12 total, 11 up, 11 in 2022-09-07T22:55:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:09 smithi059 ceph-mon[97450]: pgmap v2411: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:55:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:09 smithi059 ceph-mon[97450]: osdmap e243: 12 total, 11 up, 11 in 2022-09-07T22:55:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:10.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:10.016247+0000) 2022-09-07T22:55:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:09.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:09.912686+0000) 2022-09-07T22:55:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:11.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:11.016426+0000) 2022-09-07T22:55:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:10.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:10.912838+0000) 2022-09-07T22:55:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:11 smithi055 ceph-mon[97033]: pgmap v2413: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:55:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:11 smithi059 ceph-mon[97450]: pgmap v2413: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:55:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:12.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:12.016667+0000) 2022-09-07T22:55:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:11.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:11.912982+0000) 2022-09-07T22:55:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:13.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:13.016916+0000) 2022-09-07T22:55:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:12.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:12.913225+0000) 2022-09-07T22:55:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:13 smithi055 ceph-mon[97033]: pgmap v2414: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:55:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:13 smithi055 ceph-mon[97033]: osdmap e244: 12 total, 11 up, 11 in 2022-09-07T22:55:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:13 smithi059 ceph-mon[97450]: pgmap v2414: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:55:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:13 smithi059 ceph-mon[97450]: osdmap e244: 12 total, 11 up, 11 in 2022-09-07T22:55:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:14.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:14.017178+0000) 2022-09-07T22:55:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:13.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:13.913424+0000) 2022-09-07T22:55:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:15.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:15.017399+0000) 2022-09-07T22:55:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:14.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:14.913718+0000) 2022-09-07T22:55:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:15 smithi055 ceph-mon[97033]: pgmap v2416: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:55:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:15 smithi059 ceph-mon[97450]: pgmap v2416: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:55:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:16.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:16.017571+0000) 2022-09-07T22:55:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:15.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:15.913886+0000) 2022-09-07T22:55:17.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:55:17.033+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:55:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:17.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:17.017798+0000) 2022-09-07T22:55:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:16.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:16.914135+0000) 2022-09-07T22:55:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:17 smithi055 ceph-mon[97033]: pgmap v2417: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:55:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:17 smithi059 ceph-mon[97450]: pgmap v2417: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:55:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:17.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:17.914406+0000) 2022-09-07T22:55:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:18.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:18.018028+0000) 2022-09-07T22:55:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:19.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:19.018286+0000) 2022-09-07T22:55:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:18.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:18.914654+0000) 2022-09-07T22:55:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:19 smithi055 ceph-mon[97033]: pgmap v2418: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T22:55:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:19 smithi059 ceph-mon[97450]: pgmap v2418: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T22:55:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:19.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:19.914798+0000) 2022-09-07T22:55:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:20.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:20.018429+0000) 2022-09-07T22:55:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:20.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:20.915046+0000) 2022-09-07T22:55:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:21.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:21.018551+0000) 2022-09-07T22:55:21.993 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:21 smithi055 ceph-mon[97033]: pgmap v2419: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:55:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:21 smithi059 ceph-mon[97450]: pgmap v2419: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:55:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:22.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:22.018744+0000) 2022-09-07T22:55:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:21.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:21.915298+0000) 2022-09-07T22:55:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:23.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:23.018954+0000) 2022-09-07T22:55:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:22.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:22.915467+0000) 2022-09-07T22:55:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:23 smithi055 ceph-mon[97033]: pgmap v2420: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:55:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:23 smithi059 ceph-mon[97450]: pgmap v2420: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:55:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:24.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:24.019191+0000) 2022-09-07T22:55:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:23.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:23.915729+0000) 2022-09-07T22:55:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:25.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:25.019436+0000) 2022-09-07T22:55:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:24.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:24.915976+0000) 2022-09-07T22:55:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:25 smithi055 ceph-mon[97033]: pgmap v2421: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-07T22:55:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:25 smithi059 ceph-mon[97450]: pgmap v2421: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-07T22:55:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:26.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:26.019722+0000) 2022-09-07T22:55:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:25.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:25.916138+0000) 2022-09-07T22:55:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:26.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:26.916404+0000) 2022-09-07T22:55:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:27.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:27.019956+0000) 2022-09-07T22:55:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:27 smithi055 ceph-mon[97033]: pgmap v2422: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:55:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:27 smithi059 ceph-mon[97450]: pgmap v2422: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:55:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:28.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:28.020144+0000) 2022-09-07T22:55:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:27.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:27.916657+0000) 2022-09-07T22:55:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:29.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:29.020384+0000) 2022-09-07T22:55:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:28.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:28.916900+0000) 2022-09-07T22:55:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:29 smithi055 ceph-mon[97033]: pgmap v2423: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:29 smithi059 ceph-mon[97450]: pgmap v2423: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:30.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:30.020598+0000) 2022-09-07T22:55:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:29.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:29.917112+0000) 2022-09-07T22:55:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:31.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:31.020804+0000) 2022-09-07T22:55:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:30.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:30.917262+0000) 2022-09-07T22:55:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:31 smithi055 ceph-mon[97033]: pgmap v2424: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:55:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:31 smithi059 ceph-mon[97450]: pgmap v2424: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:55:32.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:55:32.035+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:55:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:31.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:31.917461+0000) 2022-09-07T22:55:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:32.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:32.020966+0000) 2022-09-07T22:55:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:32.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:32.917723+0000) 2022-09-07T22:55:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:33.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:33.021214+0000) 2022-09-07T22:55:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:33 smithi055 ceph-mon[97033]: pgmap v2425: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:33 smithi059 ceph-mon[97450]: pgmap v2425: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:33.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:33.917953+0000) 2022-09-07T22:55:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:34.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:34.021454+0000) 2022-09-07T22:55:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:35.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:35.021704+0000) 2022-09-07T22:55:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:34.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:34.918171+0000) 2022-09-07T22:55:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:35 smithi055 ceph-mon[97033]: pgmap v2426: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:35 smithi059 ceph-mon[97450]: pgmap v2426: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:35.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:35.918401+0000) 2022-09-07T22:55:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:36.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:36.021910+0000) 2022-09-07T22:55:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:37.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:37.022135+0000) 2022-09-07T22:55:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:36.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:36.918649+0000) 2022-09-07T22:55:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:37 smithi055 ceph-mon[97033]: pgmap v2427: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:55:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:37 smithi059 ceph-mon[97450]: pgmap v2427: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:55:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:38.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:38.022383+0000) 2022-09-07T22:55:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:37.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:37.918909+0000) 2022-09-07T22:55:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:39.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:39.022613+0000) 2022-09-07T22:55:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:38.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:38.919107+0000) 2022-09-07T22:55:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:39 smithi055 ceph-mon[97033]: pgmap v2428: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:39 smithi059 ceph-mon[97450]: pgmap v2428: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:40.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:40.022822+0000) 2022-09-07T22:55:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:39.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:39.919332+0000) 2022-09-07T22:55:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:55:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:55:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:55:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:55:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:55:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:55:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:41.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:41.023034+0000) 2022-09-07T22:55:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:40.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:40.919517+0000) 2022-09-07T22:55:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:41 smithi055 ceph-mon[97033]: pgmap v2429: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:55:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:55:41.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:55:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:41 smithi059 ceph-mon[97450]: pgmap v2429: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:55:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:55:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:55:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:42.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:42.023269+0000) 2022-09-07T22:55:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:41.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:41.919740+0000) 2022-09-07T22:55:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:43.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:43.023493+0000) 2022-09-07T22:55:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:42.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:42.920004+0000) 2022-09-07T22:55:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:43 smithi055 ceph-mon[97033]: pgmap v2430: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:43 smithi059 ceph-mon[97450]: pgmap v2430: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:43.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:43.920215+0000) 2022-09-07T22:55:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:44.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:44.023711+0000) 2022-09-07T22:55:44.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:55:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:55:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:55:44.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:55:44.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:44.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:44.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:44.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:44.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:44.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:44.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:44.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:55:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:55:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:55:45.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:55:45.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:45.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:45.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:45.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:45.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:45.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:45.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:45.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T22:55:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:45.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:45.023935+0000) 2022-09-07T22:55:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:44.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:44.920489+0000) 2022-09-07T22:55:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:45 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T22:55:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:45 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T22:55:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:45 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T22:55:45.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:45 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T22:55:45.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:45 smithi055 ceph-mon[97033]: pgmap v2431: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:45 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T22:55:46.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:45 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T22:55:46.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:45 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T22:55:46.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:45 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T22:55:46.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:45 smithi059 ceph-mon[97450]: pgmap v2431: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:46.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:46.024148+0000) 2022-09-07T22:55:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:45.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:45.920730+0000) 2022-09-07T22:55:47.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:55:47.036+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:55:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:47.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:47.024377+0000) 2022-09-07T22:55:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:46.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:46.921001+0000) 2022-09-07T22:55:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:47 smithi055 ceph-mon[97033]: pgmap v2432: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:55:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:47 smithi059 ceph-mon[97450]: pgmap v2432: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:55:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:47.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:47.921272+0000) 2022-09-07T22:55:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:48.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:48.024583+0000) 2022-09-07T22:55:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:49.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:49.024727+0000) 2022-09-07T22:55:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:48.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:48.921559+0000) 2022-09-07T22:55:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:49 smithi055 ceph-mon[97033]: pgmap v2433: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:49 smithi059 ceph-mon[97450]: pgmap v2433: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:49.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:49.921821+0000) 2022-09-07T22:55:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:50.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:50.024861+0000) 2022-09-07T22:55:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:51.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:51.025121+0000) 2022-09-07T22:55:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:50.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:50.922036+0000) 2022-09-07T22:55:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:51 smithi055 ceph-mon[97033]: pgmap v2434: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:55:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:51 smithi059 ceph-mon[97450]: pgmap v2434: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:55:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:52.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:52.025308+0000) 2022-09-07T22:55:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:51.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:51.922309+0000) 2022-09-07T22:55:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:53.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:53.025515+0000) 2022-09-07T22:55:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:52.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:52.922618+0000) 2022-09-07T22:55:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:53 smithi055 ceph-mon[97033]: pgmap v2435: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:53 smithi059 ceph-mon[97450]: pgmap v2435: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:54.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:54.025676+0000) 2022-09-07T22:55:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:53.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:53.922937+0000) 2022-09-07T22:55:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:55.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:55.025876+0000) 2022-09-07T22:55:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:54.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:54.923165+0000) 2022-09-07T22:55:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:55 smithi055 ceph-mon[97033]: pgmap v2436: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:55 smithi059 ceph-mon[97450]: pgmap v2436: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:55:56.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:56.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:56.026151+0000) 2022-09-07T22:55:56.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:55.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:55.923392+0000) 2022-09-07T22:55:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:57.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:57.026388+0000) 2022-09-07T22:55:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:56.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:56.923689+0000) 2022-09-07T22:55:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:57 smithi055 ceph-mon[97033]: pgmap v2437: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:55:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:57 smithi059 ceph-mon[97450]: pgmap v2437: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:55:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:58.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:58.026576+0000) 2022-09-07T22:55:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:57.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:57.923901+0000) 2022-09-07T22:55:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:55:59.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:59.026744+0000) 2022-09-07T22:55:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:58.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:58.924072+0000) 2022-09-07T22:55:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:55:59 smithi055 ceph-mon[97033]: pgmap v2438: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:55:59 smithi059 ceph-mon[97450]: pgmap v2438: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:00.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:00.026981+0000) 2022-09-07T22:56:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:55:59.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:55:59.924342+0000) 2022-09-07T22:56:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:01.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:01.027159+0000) 2022-09-07T22:56:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:00.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:00.924609+0000) 2022-09-07T22:56:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:01 smithi055 ceph-mon[97033]: pgmap v2439: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:56:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:01 smithi059 ceph-mon[97450]: pgmap v2439: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:56:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:01.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:01.924928+0000) 2022-09-07T22:56:02.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:56:02.039+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:56:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:02.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:02.027333+0000) 2022-09-07T22:56:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:03.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:03.027547+0000) 2022-09-07T22:56:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:02.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:02.925191+0000) 2022-09-07T22:56:03.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:03 smithi055 ceph-mon[97033]: pgmap v2440: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:03 smithi059 ceph-mon[97450]: pgmap v2440: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:04.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:04.027719+0000) 2022-09-07T22:56:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:03.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:03.925474+0000) 2022-09-07T22:56:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:56:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:56:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:56:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:56:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:05.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:05.027890+0000) 2022-09-07T22:56:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:04.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:04.925671+0000) 2022-09-07T22:56:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:05 smithi055 ceph-mon[97033]: pgmap v2441: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:05 smithi059 ceph-mon[97450]: pgmap v2441: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:06.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:06.028173+0000) 2022-09-07T22:56:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:05.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:05.925889+0000) 2022-09-07T22:56:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:07.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:07.028432+0000) 2022-09-07T22:56:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:06.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:06.926137+0000) 2022-09-07T22:56:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:07 smithi055 ceph-mon[97033]: pgmap v2442: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:56:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:07 smithi059 ceph-mon[97450]: pgmap v2442: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:56:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:08.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:08.028678+0000) 2022-09-07T22:56:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:07.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:07.926331+0000) 2022-09-07T22:56:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:09.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:09.028901+0000) 2022-09-07T22:56:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:08.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:08.926537+0000) 2022-09-07T22:56:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:09 smithi055 ceph-mon[97033]: pgmap v2443: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:56:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:09 smithi055 ceph-mon[97033]: osdmap e245: 12 total, 11 up, 11 in 2022-09-07T22:56:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:09 smithi059 ceph-mon[97450]: pgmap v2443: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:56:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:09 smithi059 ceph-mon[97450]: osdmap e245: 12 total, 11 up, 11 in 2022-09-07T22:56:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:10.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:10.029125+0000) 2022-09-07T22:56:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:09.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:09.926760+0000) 2022-09-07T22:56:11.030 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:10.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:10.926902+0000) 2022-09-07T22:56:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:11.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:11.029348+0000) 2022-09-07T22:56:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:11 smithi055 ceph-mon[97033]: pgmap v2445: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:56:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:11 smithi055 ceph-mon[97033]: pgmap v2446: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-07T22:56:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:11 smithi055 ceph-mon[97033]: pgmap v2447: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 22 KiB/s rd, 86 KiB/s wr, 38 op/s 2022-09-07T22:56:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:56:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:11 smithi059 ceph-mon[97450]: pgmap v2445: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:56:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:11 smithi059 ceph-mon[97450]: pgmap v2446: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-07T22:56:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:11 smithi059 ceph-mon[97450]: pgmap v2447: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 22 KiB/s rd, 86 KiB/s wr, 38 op/s 2022-09-07T22:56:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:56:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:11.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:11.927121+0000) 2022-09-07T22:56:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:12.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:12.029584+0000) 2022-09-07T22:56:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:13.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:13.029830+0000) 2022-09-07T22:56:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:12.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:12.927373+0000) 2022-09-07T22:56:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:13 smithi059 ceph-mon[97450]: pgmap v2448: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-07T22:56:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:13 smithi059 ceph-mon[97450]: osdmap e246: 12 total, 11 up, 11 in 2022-09-07T22:56:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:13 smithi055 ceph-mon[97033]: pgmap v2448: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-07T22:56:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:13 smithi055 ceph-mon[97033]: osdmap e246: 12 total, 11 up, 11 in 2022-09-07T22:56:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:14.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:14.030068+0000) 2022-09-07T22:56:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:13.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:13.927703+0000) 2022-09-07T22:56:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:15.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:15.030309+0000) 2022-09-07T22:56:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:14.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:14.927950+0000) 2022-09-07T22:56:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:15 smithi059 ceph-mon[97450]: pgmap v2450: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 169 B/s wr, 5 op/s 2022-09-07T22:56:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:15 smithi055 ceph-mon[97033]: pgmap v2450: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 169 B/s wr, 5 op/s 2022-09-07T22:56:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:16.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:16.030466+0000) 2022-09-07T22:56:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:15.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:15.928117+0000) 2022-09-07T22:56:17.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:56:17.053+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:56:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:16.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:16.928334+0000) 2022-09-07T22:56:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:17.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:17.030708+0000) 2022-09-07T22:56:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:17 smithi059 ceph-mon[97450]: pgmap v2451: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-07T22:56:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:17 smithi055 ceph-mon[97033]: pgmap v2451: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-07T22:56:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:18.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:18.030929+0000) 2022-09-07T22:56:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:17.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:17.928595+0000) 2022-09-07T22:56:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:19.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:19.031166+0000) 2022-09-07T22:56:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:18.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:18.928814+0000) 2022-09-07T22:56:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:19 smithi059 ceph-mon[97450]: pgmap v2452: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:56:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:19 smithi055 ceph-mon[97033]: pgmap v2452: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:56:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:20.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:20.031356+0000) 2022-09-07T22:56:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:19.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:19.928994+0000) 2022-09-07T22:56:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:20 smithi055 ceph-mon[97033]: pgmap v2453: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:56:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:20 smithi059 ceph-mon[97450]: pgmap v2453: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:56:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:21.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:21.031538+0000) 2022-09-07T22:56:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:20.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:20.929210+0000) 2022-09-07T22:56:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:21.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:21.929464+0000) 2022-09-07T22:56:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:22.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:22.031724+0000) 2022-09-07T22:56:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:22 smithi055 ceph-mon[97033]: pgmap v2454: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:56:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:22 smithi059 ceph-mon[97450]: pgmap v2454: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:56:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:22.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:22.929745+0000) 2022-09-07T22:56:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:23.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:23.031954+0000) 2022-09-07T22:56:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:23.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:23.929952+0000) 2022-09-07T22:56:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:24.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:24.032174+0000) 2022-09-07T22:56:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:24 smithi055 ceph-mon[97033]: pgmap v2455: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-07T22:56:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:24 smithi059 ceph-mon[97450]: pgmap v2455: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-07T22:56:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:24.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:24.930222+0000) 2022-09-07T22:56:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:25.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:25.032338+0000) 2022-09-07T22:56:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:26.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:26.032507+0000) 2022-09-07T22:56:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:25.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:25.930493+0000) 2022-09-07T22:56:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:26 smithi055 ceph-mon[97033]: pgmap v2456: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:56:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:26 smithi059 ceph-mon[97450]: pgmap v2456: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T22:56:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:27.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:27.032725+0000) 2022-09-07T22:56:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:26.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:26.930745+0000) 2022-09-07T22:56:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:28.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:28.032937+0000) 2022-09-07T22:56:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:27.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:27.930889+0000) 2022-09-07T22:56:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:29.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:29.033195+0000) 2022-09-07T22:56:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:28.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:28.931078+0000) 2022-09-07T22:56:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:29 smithi059 ceph-mon[97450]: pgmap v2457: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:29 smithi055 ceph-mon[97033]: pgmap v2457: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:30.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:30.033409+0000) 2022-09-07T22:56:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:29.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:29.931241+0000) 2022-09-07T22:56:31.034 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:30.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:30.931473+0000) 2022-09-07T22:56:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:30 smithi055 ceph-mon[97033]: pgmap v2458: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:56:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:30 smithi059 ceph-mon[97450]: pgmap v2458: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:56:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:31.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:31.033568+0000) 2022-09-07T22:56:32.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:56:32.057+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:56:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:32.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:32.033801+0000) 2022-09-07T22:56:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:31.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:31.931793+0000) 2022-09-07T22:56:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:32 smithi055 ceph-mon[97033]: pgmap v2459: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:32 smithi059 ceph-mon[97450]: pgmap v2459: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:33.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:33.033995+0000) 2022-09-07T22:56:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:32.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:32.932073+0000) 2022-09-07T22:56:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:33.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:33.932340+0000) 2022-09-07T22:56:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:34.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:34.034231+0000) 2022-09-07T22:56:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:34 smithi055 ceph-mon[97033]: pgmap v2460: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:34 smithi059 ceph-mon[97450]: pgmap v2460: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:34.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:34.932509+0000) 2022-09-07T22:56:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:35.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:35.034453+0000) 2022-09-07T22:56:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:35.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:35.932763+0000) 2022-09-07T22:56:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:36.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:36.034687+0000) 2022-09-07T22:56:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:36 smithi055 ceph-mon[97033]: pgmap v2461: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:56:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:36 smithi059 ceph-mon[97450]: pgmap v2461: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:56:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:36.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:36.933042+0000) 2022-09-07T22:56:37.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:37.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:37.034898+0000) 2022-09-07T22:56:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:38.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:38.035110+0000) 2022-09-07T22:56:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:37.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:37.933307+0000) 2022-09-07T22:56:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:39.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:39.035288+0000) 2022-09-07T22:56:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:38.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:38.933601+0000) 2022-09-07T22:56:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:39 smithi059 ceph-mon[97450]: pgmap v2462: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:39 smithi055 ceph-mon[97033]: pgmap v2462: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:40.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:40.035431+0000) 2022-09-07T22:56:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:39.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:39.933811+0000) 2022-09-07T22:56:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:40 smithi055 ceph-mon[97033]: pgmap v2463: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:56:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:40 smithi059 ceph-mon[97450]: pgmap v2463: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:56:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:41.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:41.035627+0000) 2022-09-07T22:56:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:40.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:40.934056+0000) 2022-09-07T22:56:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:42.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:42.035864+0000) 2022-09-07T22:56:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:41.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:41.934262+0000) 2022-09-07T22:56:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:42 smithi055 ceph-mon[97033]: pgmap v2464: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:42 smithi059 ceph-mon[97450]: pgmap v2464: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:43.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:43.036079+0000) 2022-09-07T22:56:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:42.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:42.934472+0000) 2022-09-07T22:56:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:44.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:44.036278+0000) 2022-09-07T22:56:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:43.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:43.934650+0000) 2022-09-07T22:56:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:44 smithi055 ceph-mon[97033]: pgmap v2465: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:44 smithi059 ceph-mon[97450]: pgmap v2465: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:45.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:45.036497+0000) 2022-09-07T22:56:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:44.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:44.934899+0000) 2022-09-07T22:56:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:46.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:46.036765+0000) 2022-09-07T22:56:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:45.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:45.935150+0000) 2022-09-07T22:56:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:46 smithi055 ceph-mon[97033]: pgmap v2466: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:56:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:46 smithi059 ceph-mon[97450]: pgmap v2466: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:56:47.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:56:47.060+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:56:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:47.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:47.036953+0000) 2022-09-07T22:56:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:46.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:46.935419+0000) 2022-09-07T22:56:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:48.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:48.037184+0000) 2022-09-07T22:56:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:47.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:47.935608+0000) 2022-09-07T22:56:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:48.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:48.935848+0000) 2022-09-07T22:56:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:49.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:49.037405+0000) 2022-09-07T22:56:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:49 smithi059 ceph-mon[97450]: pgmap v2467: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:49 smithi055 ceph-mon[97033]: pgmap v2467: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:49.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:49.936071+0000) 2022-09-07T22:56:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:50.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:50.037659+0000) 2022-09-07T22:56:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:50 smithi055 ceph-mon[97033]: pgmap v2468: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:56:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:50 smithi059 ceph-mon[97450]: pgmap v2468: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:56:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:51.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:51.037869+0000) 2022-09-07T22:56:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:50.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:50.936353+0000) 2022-09-07T22:56:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:51.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:51.936581+0000) 2022-09-07T22:56:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:52.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:52.038074+0000) 2022-09-07T22:56:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:52 smithi055 ceph-mon[97033]: pgmap v2469: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:52 smithi059 ceph-mon[97450]: pgmap v2469: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:52.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:52.936822+0000) 2022-09-07T22:56:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:53.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:53.038255+0000) 2022-09-07T22:56:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:54.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:54.038468+0000) 2022-09-07T22:56:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:53.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:53.937027+0000) 2022-09-07T22:56:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:54 smithi055 ceph-mon[97033]: pgmap v2470: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:54 smithi059 ceph-mon[97450]: pgmap v2470: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:55.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:55.038722+0000) 2022-09-07T22:56:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:54.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:54.937277+0000) 2022-09-07T22:56:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:56.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:56.038922+0000) 2022-09-07T22:56:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:55.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:55.937533+0000) 2022-09-07T22:56:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:56 smithi055 ceph-mon[97033]: pgmap v2471: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:56:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:56 smithi059 ceph-mon[97450]: pgmap v2471: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:56:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:57.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:57.039146+0000) 2022-09-07T22:56:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:56.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:56.937840+0000) 2022-09-07T22:56:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:58.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:58.039349+0000) 2022-09-07T22:56:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:57.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:57.937976+0000) 2022-09-07T22:56:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:56:59.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:59.039588+0000) 2022-09-07T22:56:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:58.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:58.938175+0000) 2022-09-07T22:56:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:56:59 smithi059 ceph-mon[97450]: pgmap v2472: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:56:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:56:59 smithi055 ceph-mon[97033]: pgmap v2472: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:57:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:00.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:00.039772+0000) 2022-09-07T22:57:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:56:59.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:56:59.938416+0000) 2022-09-07T22:57:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:00 smithi055 ceph-mon[97033]: pgmap v2473: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:57:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:00 smithi059 ceph-mon[97450]: pgmap v2473: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:57:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:00.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:00.938705+0000) 2022-09-07T22:57:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:01.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:01.039984+0000) 2022-09-07T22:57:02.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:57:02.075+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:57:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:02.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:02.040211+0000) 2022-09-07T22:57:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:01.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:01.938959+0000) 2022-09-07T22:57:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:02 smithi055 ceph-mon[97033]: pgmap v2474: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:57:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:02 smithi059 ceph-mon[97450]: pgmap v2474: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:57:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:03.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:03.040460+0000) 2022-09-07T22:57:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:02.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:02.939224+0000) 2022-09-07T22:57:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:04.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:04.040725+0000) 2022-09-07T22:57:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:03.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:03.939394+0000) 2022-09-07T22:57:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:57:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:57:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:57:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:57:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:05.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:05.040982+0000) 2022-09-07T22:57:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:04.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:04.939706+0000) 2022-09-07T22:57:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:05 smithi055 ceph-mon[97033]: pgmap v2475: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 21 KiB/s wr, 18 op/s 2022-09-07T22:57:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:05 smithi059 ceph-mon[97450]: pgmap v2475: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 21 KiB/s wr, 18 op/s 2022-09-07T22:57:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:06.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:06.041143+0000) 2022-09-07T22:57:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:05.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:05.939943+0000) 2022-09-07T22:57:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:06 smithi055 ceph-mon[97033]: pgmap v2476: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:57:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:06 smithi059 ceph-mon[97450]: pgmap v2476: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:57:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:07.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:07.041331+0000) 2022-09-07T22:57:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:06.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:06.940172+0000) 2022-09-07T22:57:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:08.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:08.041535+0000) 2022-09-07T22:57:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:07.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:07.940441+0000) 2022-09-07T22:57:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:08.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:08.940727+0000) 2022-09-07T22:57:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:09.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:09.041787+0000) 2022-09-07T22:57:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:09 smithi055 ceph-mon[97033]: osdmap e247: 12 total, 11 up, 11 in 2022-09-07T22:57:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:09 smithi055 ceph-mon[97033]: pgmap v2478: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:57:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:09 smithi059 ceph-mon[97450]: osdmap e247: 12 total, 11 up, 11 in 2022-09-07T22:57:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:09 smithi059 ceph-mon[97450]: pgmap v2478: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:57:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:09.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:09.940973+0000) 2022-09-07T22:57:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:10.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:10.041977+0000) 2022-09-07T22:57:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:11.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:11.042191+0000) 2022-09-07T22:57:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:10.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:10.941171+0000) 2022-09-07T22:57:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:10 smithi055 ceph-mon[97033]: pgmap v2479: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:57:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:10 smithi059 ceph-mon[97450]: pgmap v2479: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:57:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:57:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:57:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:57:12.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:57:12.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:57:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:57:12.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:57:12.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:57:12.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:57:12.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:57:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:12.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:12.042437+0000) 2022-09-07T22:57:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:11.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:11.941405+0000) 2022-09-07T22:57:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:12 smithi055 ceph-mon[97033]: pgmap v2480: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:57:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:12 smithi059 ceph-mon[97450]: pgmap v2480: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:57:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:13.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:13.042689+0000) 2022-09-07T22:57:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:12.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:12.941669+0000) 2022-09-07T22:57:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:14.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:14.042932+0000) 2022-09-07T22:57:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:13.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:13.941872+0000) 2022-09-07T22:57:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:14 smithi059 ceph-mon[97450]: osdmap e248: 12 total, 11 up, 11 in 2022-09-07T22:57:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:14 smithi055 ceph-mon[97033]: osdmap e248: 12 total, 11 up, 11 in 2022-09-07T22:57:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:15.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:15.043150+0000) 2022-09-07T22:57:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:14.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:14.942044+0000) 2022-09-07T22:57:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:15 smithi055 ceph-mon[97033]: pgmap v2482: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-07T22:57:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:15 smithi059 ceph-mon[97450]: pgmap v2482: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-07T22:57:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:16.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:16.043367+0000) 2022-09-07T22:57:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:15.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:15.942286+0000) 2022-09-07T22:57:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:16 smithi055 ceph-mon[97033]: pgmap v2483: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-07T22:57:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:16 smithi059 ceph-mon[97450]: pgmap v2483: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-07T22:57:17.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:57:17.083+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:57:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:17.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:17.043593+0000) 2022-09-07T22:57:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:16.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:16.942498+0000) 2022-09-07T22:57:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:18.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:18.043784+0000) 2022-09-07T22:57:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:17.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:17.942747+0000) 2022-09-07T22:57:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:19.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:19.044015+0000) 2022-09-07T22:57:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:18.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:18.943000+0000) 2022-09-07T22:57:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:19 smithi055 ceph-mon[97033]: pgmap v2484: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:57:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:19 smithi059 ceph-mon[97450]: pgmap v2484: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:57:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:20.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:20.044265+0000) 2022-09-07T22:57:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:19.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:19.943185+0000) 2022-09-07T22:57:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:20 smithi055 ceph-mon[97033]: pgmap v2485: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:57:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:20 smithi059 ceph-mon[97450]: pgmap v2485: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:57:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:21.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:21.044411+0000) 2022-09-07T22:57:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:20.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:20.943415+0000) 2022-09-07T22:57:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:21.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:21.943689+0000) 2022-09-07T22:57:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:22.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:22.044564+0000) 2022-09-07T22:57:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:22 smithi055 ceph-mon[97033]: pgmap v2486: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:57:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:22 smithi059 ceph-mon[97450]: pgmap v2486: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:57:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:23.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:23.044729+0000) 2022-09-07T22:57:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:22.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:22.943948+0000) 2022-09-07T22:57:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:24.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:24.044939+0000) 2022-09-07T22:57:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:23.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:23.944195+0000) 2022-09-07T22:57:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:24 smithi055 ceph-mon[97033]: pgmap v2487: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 91 B/s wr, 7 op/s 2022-09-07T22:57:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:24 smithi059 ceph-mon[97450]: pgmap v2487: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 91 B/s wr, 7 op/s 2022-09-07T22:57:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:25.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:25.045119+0000) 2022-09-07T22:57:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:24.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:24.944440+0000) 2022-09-07T22:57:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:25.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:25.944624+0000) 2022-09-07T22:57:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:26.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:26.045382+0000) 2022-09-07T22:57:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:26 smithi055 ceph-mon[97033]: pgmap v2488: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:57:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:26 smithi059 ceph-mon[97450]: pgmap v2488: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T22:57:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:27.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:27.045608+0000) 2022-09-07T22:57:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:26.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:26.944846+0000) 2022-09-07T22:57:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:28.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:28.045895+0000) 2022-09-07T22:57:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:27.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:27.945053+0000) 2022-09-07T22:57:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:28.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:28.945185+0000) 2022-09-07T22:57:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:29.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:29.046193+0000) 2022-09-07T22:57:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:29 smithi055 ceph-mon[97033]: pgmap v2489: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:57:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:29 smithi059 ceph-mon[97450]: pgmap v2489: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:57:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:30.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:30.046465+0000) 2022-09-07T22:57:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:29.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:29.945420+0000) 2022-09-07T22:57:31.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:30 smithi055 ceph-mon[97033]: pgmap v2490: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:57:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:30 smithi059 ceph-mon[97450]: pgmap v2490: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:57:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:31.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:31.046596+0000) 2022-09-07T22:57:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:30.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:30.945682+0000) 2022-09-07T22:57:32.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:57:32.093+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:57:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:32.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:32.046841+0000) 2022-09-07T22:57:32.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:31.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:31.945914+0000) 2022-09-07T22:57:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:32 smithi055 ceph-mon[97033]: pgmap v2491: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:57:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:32 smithi059 ceph-mon[97450]: pgmap v2491: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:57:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:33.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:33.047066+0000) 2022-09-07T22:57:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:32.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:32.946163+0000) 2022-09-07T22:57:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:34.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:34.047303+0000) 2022-09-07T22:57:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:33.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:33.946339+0000) 2022-09-07T22:57:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:34 smithi055 ceph-mon[97033]: pgmap v2492: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:57:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:34 smithi059 ceph-mon[97450]: pgmap v2492: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:57:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:35.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:35.047487+0000) 2022-09-07T22:57:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:34.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:34.946559+0000) 2022-09-07T22:57:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:36.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:36.047652+0000) 2022-09-07T22:57:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:35.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:35.946803+0000) 2022-09-07T22:57:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:36 smithi055 ceph-mon[97033]: pgmap v2493: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:57:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:36 smithi059 ceph-mon[97450]: pgmap v2493: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T22:57:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:37.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:37.047910+0000) 2022-09-07T22:57:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:36.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:36.947063+0000) 2022-09-07T22:57:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:38.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:38.048052+0000) 2022-09-07T22:57:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:37.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:37.947309+0000) 2022-09-07T22:57:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:39.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:39.048310+0000) 2022-09-07T22:57:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:38.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:38.947560+0000) 2022-09-07T22:57:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:39 smithi055 ceph-mon[97033]: pgmap v2494: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:57:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:39 smithi059 ceph-mon[97450]: pgmap v2494: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:57:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:40.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:40.048543+0000) 2022-09-07T22:57:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:39.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:39.947751+0000) 2022-09-07T22:57:41.117 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:40 smithi059 ceph-mon[97450]: pgmap v2495: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:57:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:40 smithi055 ceph-mon[97033]: pgmap v2495: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:57:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:41.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:41.048771+0000) 2022-09-07T22:57:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:40.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:40.947925+0000) 2022-09-07T22:57:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:42 smithi059 ceph-mon[97450]: pgmap v2496: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:57:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:42 smithi059 ceph-mon[97450]: pgmap v2497: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:57:42.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:57:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:42.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:42.048984+0000) 2022-09-07T22:57:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:41.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:41.948129+0000) 2022-09-07T22:57:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:42 smithi055 ceph-mon[97033]: pgmap v2496: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T22:57:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:42 smithi055 ceph-mon[97033]: pgmap v2497: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:57:42.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:57:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:43.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:43.049225+0000) 2022-09-07T22:57:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:42.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:42.948329+0000) 2022-09-07T22:57:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:44.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:44.049472+0000) 2022-09-07T22:57:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:43.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:43.948580+0000) 2022-09-07T22:57:44.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:44 smithi055 ceph-mon[97033]: pgmap v2498: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-07T22:57:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:44 smithi059 ceph-mon[97450]: pgmap v2498: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-07T22:57:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:45.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:45.049754+0000) 2022-09-07T22:57:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:44.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:44.948835+0000) 2022-09-07T22:57:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:46.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:46.050035+0000) 2022-09-07T22:57:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:45.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:45.949036+0000) 2022-09-07T22:57:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:46 smithi055 ceph-mon[97033]: pgmap v2499: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:57:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:46 smithi059 ceph-mon[97450]: pgmap v2499: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:57:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:47.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:47.050352+0000) 2022-09-07T22:57:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:46.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:46.949250+0000) 2022-09-07T22:57:47.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:57:47.097+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:57:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:48.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:48.050592+0000) 2022-09-07T22:57:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:47.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:47.949475+0000) 2022-09-07T22:57:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:48 smithi055 ceph-mon[97033]: pgmap v2500: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:57:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:48 smithi059 ceph-mon[97450]: pgmap v2500: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T22:57:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:49.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:49.050817+0000) 2022-09-07T22:57:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:48.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:48.949741+0000) 2022-09-07T22:57:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:50.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:50.050998+0000) 2022-09-07T22:57:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:49.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:49.949957+0000) 2022-09-07T22:57:50.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:50 smithi055 ceph-mon[97033]: pgmap v2501: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:57:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:50 smithi059 ceph-mon[97450]: pgmap v2501: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T22:57:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:51.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:51.051214+0000) 2022-09-07T22:57:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:50.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:50.950199+0000) 2022-09-07T22:57:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:51.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:51.950450+0000) 2022-09-07T22:57:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:52.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:52.051410+0000) 2022-09-07T22:57:52.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:52 smithi055 ceph-mon[97033]: pgmap v2502: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:57:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:52 smithi059 ceph-mon[97450]: pgmap v2502: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T22:57:53.409 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:53.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:53.051577+0000) 2022-09-07T22:57:53.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:52.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:52.950734+0000) 2022-09-07T22:57:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:54.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:54.051780+0000) 2022-09-07T22:57:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:53.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:53.950886+0000) 2022-09-07T22:57:54.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:54 smithi055 ceph-mon[97033]: pgmap v2503: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:57:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:54 smithi059 ceph-mon[97450]: pgmap v2503: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:57:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:55.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:55.051997+0000) 2022-09-07T22:57:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:54.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:54.951127+0000) 2022-09-07T22:57:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:56.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:56.052184+0000) 2022-09-07T22:57:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:55.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:55.951354+0000) 2022-09-07T22:57:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:56 smithi055 ceph-mon[97033]: pgmap v2504: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:57:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:56 smithi059 ceph-mon[97450]: pgmap v2504: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:57:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:57.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:57.052388+0000) 2022-09-07T22:57:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:56.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:56.951618+0000) 2022-09-07T22:57:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:58.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:58.052612+0000) 2022-09-07T22:57:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:57.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:57.951907+0000) 2022-09-07T22:57:58.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:57:58 smithi055 ceph-mon[97033]: pgmap v2505: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:57:59.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:57:58 smithi059 ceph-mon[97450]: pgmap v2505: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:57:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:57:59.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:59.052839+0000) 2022-09-07T22:57:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:58.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:58.952066+0000) 2022-09-07T22:58:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:00.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:00.053068+0000) 2022-09-07T22:58:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:57:59.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:57:59.952297+0000) 2022-09-07T22:58:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:00 smithi055 ceph-mon[97033]: pgmap v2506: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:00 smithi059 ceph-mon[97450]: pgmap v2506: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:01.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:01.053219+0000) 2022-09-07T22:58:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:00.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:00.952496+0000) 2022-09-07T22:58:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:02.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:02.053403+0000) 2022-09-07T22:58:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:01.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:01.952646+0000) 2022-09-07T22:58:02.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:58:02.113+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:58:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:02 smithi055 ceph-mon[97033]: pgmap v2507: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:58:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:02 smithi059 ceph-mon[97450]: pgmap v2507: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:58:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:03.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:03.053626+0000) 2022-09-07T22:58:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:02.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:02.952873+0000) 2022-09-07T22:58:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:04.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:04.053812+0000) 2022-09-07T22:58:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:03.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:03.953070+0000) 2022-09-07T22:58:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:04 smithi055 ceph-mon[97033]: pgmap v2508: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:58:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:58:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:04 smithi059 ceph-mon[97450]: pgmap v2508: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:58:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:58:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:05.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:05.053937+0000) 2022-09-07T22:58:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:04.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:04.953301+0000) 2022-09-07T22:58:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:06.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:06.054070+0000) 2022-09-07T22:58:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:05.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:05.953452+0000) 2022-09-07T22:58:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:06 smithi055 ceph-mon[97033]: pgmap v2509: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:58:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:06 smithi059 ceph-mon[97450]: pgmap v2509: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:58:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:06.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:06.953613+0000) 2022-09-07T22:58:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:07.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:07.054235+0000) 2022-09-07T22:58:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:08.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:08.054471+0000) 2022-09-07T22:58:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:07.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:07.953858+0000) 2022-09-07T22:58:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:08 smithi055 ceph-mon[97033]: pgmap v2510: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:58:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:08 smithi059 ceph-mon[97450]: pgmap v2510: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:58:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:09.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:09.054770+0000) 2022-09-07T22:58:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:08.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:08.954092+0000) 2022-09-07T22:58:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:09 smithi055 ceph-mon[97033]: osdmap e249: 12 total, 11 up, 11 in 2022-09-07T22:58:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:09 smithi059 ceph-mon[97450]: osdmap e249: 12 total, 11 up, 11 in 2022-09-07T22:58:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:10.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:10.054948+0000) 2022-09-07T22:58:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:09.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:09.954317+0000) 2022-09-07T22:58:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:10 smithi055 ceph-mon[97033]: pgmap v2512: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:58:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:10 smithi059 ceph-mon[97450]: pgmap v2512: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:58:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:11.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:11.055165+0000) 2022-09-07T22:58:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:10.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:10.954540+0000) 2022-09-07T22:58:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:12.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:12.055394+0000) 2022-09-07T22:58:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:11.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:11.954783+0000) 2022-09-07T22:58:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:12 smithi055 ceph-mon[97033]: pgmap v2513: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:58:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:12 smithi059 ceph-mon[97450]: pgmap v2513: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:58:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:13.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:13.055603+0000) 2022-09-07T22:58:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:12.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:12.954976+0000) 2022-09-07T22:58:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:14.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:14.055827+0000) 2022-09-07T22:58:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:13.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:13.955155+0000) 2022-09-07T22:58:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:14 smithi055 ceph-mon[97033]: pgmap v2514: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:58:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:14 smithi055 ceph-mon[97033]: osdmap e250: 12 total, 11 up, 11 in 2022-09-07T22:58:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:14 smithi059 ceph-mon[97450]: pgmap v2514: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:58:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:14 smithi059 ceph-mon[97450]: osdmap e250: 12 total, 11 up, 11 in 2022-09-07T22:58:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:15.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:15.056064+0000) 2022-09-07T22:58:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:14.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:14.955348+0000) 2022-09-07T22:58:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:16.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:16.056317+0000) 2022-09-07T22:58:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:15.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:15.955542+0000) 2022-09-07T22:58:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:16 smithi055 ceph-mon[97033]: pgmap v2516: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:58:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:16 smithi059 ceph-mon[97450]: pgmap v2516: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T22:58:17.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:58:17.126+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:58:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:17.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:17.056579+0000) 2022-09-07T22:58:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:16.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:16.955715+0000) 2022-09-07T22:58:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:18.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:18.056814+0000) 2022-09-07T22:58:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:17.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:17.955911+0000) 2022-09-07T22:58:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:18 smithi055 ceph-mon[97033]: pgmap v2517: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-07T22:58:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:18 smithi059 ceph-mon[97450]: pgmap v2517: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-07T22:58:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:19.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:19.057048+0000) 2022-09-07T22:58:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:18.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:18.956155+0000) 2022-09-07T22:58:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:20.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:20.057275+0000) 2022-09-07T22:58:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:19.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:19.956411+0000) 2022-09-07T22:58:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:20 smithi055 ceph-mon[97033]: pgmap v2518: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:58:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:20 smithi059 ceph-mon[97450]: pgmap v2518: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T22:58:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:21.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:21.057517+0000) 2022-09-07T22:58:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:20.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:20.956717+0000) 2022-09-07T22:58:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:22.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:22.057731+0000) 2022-09-07T22:58:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:21.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:21.956986+0000) 2022-09-07T22:58:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:22 smithi055 ceph-mon[97033]: pgmap v2519: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:58:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:22 smithi059 ceph-mon[97450]: pgmap v2519: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:58:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:23.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:23.057932+0000) 2022-09-07T22:58:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:22.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:22.957200+0000) 2022-09-07T22:58:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:23.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:23.957405+0000) 2022-09-07T22:58:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:24.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:24.058138+0000) 2022-09-07T22:58:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:24 smithi055 ceph-mon[97033]: pgmap v2520: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:58:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:24 smithi059 ceph-mon[97450]: pgmap v2520: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:58:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:25.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:25.058382+0000) 2022-09-07T22:58:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:24.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:24.957672+0000) 2022-09-07T22:58:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:26.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:26.058559+0000) 2022-09-07T22:58:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:25.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:25.957897+0000) 2022-09-07T22:58:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:26 smithi055 ceph-mon[97033]: pgmap v2521: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-07T22:58:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:26 smithi059 ceph-mon[97450]: pgmap v2521: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-07T22:58:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:27.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:27.058798+0000) 2022-09-07T22:58:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:26.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:26.958078+0000) 2022-09-07T22:58:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:28.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:28.059061+0000) 2022-09-07T22:58:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:27.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:27.958257+0000) 2022-09-07T22:58:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:28 smithi055 ceph-mon[97033]: pgmap v2522: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:28 smithi059 ceph-mon[97450]: pgmap v2522: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:29.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:29.059215+0000) 2022-09-07T22:58:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:28.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:28.958393+0000) 2022-09-07T22:58:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:30.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:30.059417+0000) 2022-09-07T22:58:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:29.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:29.958595+0000) 2022-09-07T22:58:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:30 smithi055 ceph-mon[97033]: pgmap v2523: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:30 smithi059 ceph-mon[97450]: pgmap v2523: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:31.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:31.059613+0000) 2022-09-07T22:58:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:30.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:30.958841+0000) 2022-09-07T22:58:32.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:58:32.134+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:58:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:32.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:32.059872+0000) 2022-09-07T22:58:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:31.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:31.959064+0000) 2022-09-07T22:58:32.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:32 smithi055 ceph-mon[97033]: pgmap v2524: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:58:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:32 smithi059 ceph-mon[97450]: pgmap v2524: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:58:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:33.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:33.060103+0000) 2022-09-07T22:58:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:32.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:32.959355+0000) 2022-09-07T22:58:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:34.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:34.060268+0000) 2022-09-07T22:58:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:33.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:33.959521+0000) 2022-09-07T22:58:34.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:34 smithi055 ceph-mon[97033]: pgmap v2525: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:34 smithi059 ceph-mon[97450]: pgmap v2525: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:35.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:35.060481+0000) 2022-09-07T22:58:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:34.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:34.959766+0000) 2022-09-07T22:58:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:36.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:36.060733+0000) 2022-09-07T22:58:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:35.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:35.959937+0000) 2022-09-07T22:58:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:36 smithi055 ceph-mon[97033]: pgmap v2526: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:58:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:36 smithi059 ceph-mon[97450]: pgmap v2526: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:58:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:37.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:37.060964+0000) 2022-09-07T22:58:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:36.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:36.960153+0000) 2022-09-07T22:58:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:37.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:37.960391+0000) 2022-09-07T22:58:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:38.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:38.061171+0000) 2022-09-07T22:58:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:38 smithi055 ceph-mon[97033]: pgmap v2527: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:38 smithi059 ceph-mon[97450]: pgmap v2527: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:39.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:39.061405+0000) 2022-09-07T22:58:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:38.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:38.960623+0000) 2022-09-07T22:58:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:40.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:40.061677+0000) 2022-09-07T22:58:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:39.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:39.960920+0000) 2022-09-07T22:58:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:40 smithi055 ceph-mon[97033]: pgmap v2528: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:40 smithi059 ceph-mon[97450]: pgmap v2528: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:41.221 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:41.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:41.061873+0000) 2022-09-07T22:58:41.222 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:40.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:40.961141+0000) 2022-09-07T22:58:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:58:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:58:41.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:58:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T22:58:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T22:58:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T22:58:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:42.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:42.062072+0000) 2022-09-07T22:58:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:41.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:41.961390+0000) 2022-09-07T22:58:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:42 smithi055 ceph-mon[97033]: pgmap v2529: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:58:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:42 smithi059 ceph-mon[97450]: pgmap v2529: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:58:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:43.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:43.062257+0000) 2022-09-07T22:58:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:42.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:42.961584+0000) 2022-09-07T22:58:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:44.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:44.062460+0000) 2022-09-07T22:58:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:43.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:43.961825+0000) 2022-09-07T22:58:44.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:44 smithi055 ceph-mon[97033]: pgmap v2530: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:44 smithi059 ceph-mon[97450]: pgmap v2530: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:45.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:45.062688+0000) 2022-09-07T22:58:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:44.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:44.962041+0000) 2022-09-07T22:58:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:46.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:46.062929+0000) 2022-09-07T22:58:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:45.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:45.962171+0000) 2022-09-07T22:58:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:46 smithi055 ceph-mon[97033]: pgmap v2531: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:58:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:46 smithi059 ceph-mon[97450]: pgmap v2531: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:58:47.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:58:47.148+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:58:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:47.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:47.063169+0000) 2022-09-07T22:58:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:46.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:46.962408+0000) 2022-09-07T22:58:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:48.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:48.063418+0000) 2022-09-07T22:58:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:47.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:47.962657+0000) 2022-09-07T22:58:48.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:48 smithi055 ceph-mon[97033]: pgmap v2532: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:48 smithi059 ceph-mon[97450]: pgmap v2532: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:49.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:49.063674+0000) 2022-09-07T22:58:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:48.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:48.962856+0000) 2022-09-07T22:58:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:50.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:50.063881+0000) 2022-09-07T22:58:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:49.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:49.963034+0000) 2022-09-07T22:58:50.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:50 smithi055 ceph-mon[97033]: pgmap v2533: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:50 smithi059 ceph-mon[97450]: pgmap v2533: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:58:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:51.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:51.064129+0000) 2022-09-07T22:58:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:50.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:50.963297+0000) 2022-09-07T22:58:52.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:52.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:52.064301+0000) 2022-09-07T22:58:52.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:51.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:51.963460+0000) 2022-09-07T22:58:52.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:52 smithi055 ceph-mon[97033]: pgmap v2534: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:58:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:52 smithi059 ceph-mon[97450]: pgmap v2534: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:58:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:52.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:52.963617+0000) 2022-09-07T22:58:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:53.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:53.064527+0000) 2022-09-07T22:58:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:53.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:53.963849+0000) 2022-09-07T22:58:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:54.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:54.064705+0000) 2022-09-07T22:58:54.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:54 smithi055 ceph-mon[97033]: pgmap v2535: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:58:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:54 smithi059 ceph-mon[97450]: pgmap v2535: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:58:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:55.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:55.064928+0000) 2022-09-07T22:58:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:54.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:54.964128+0000) 2022-09-07T22:58:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:56.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:56.065144+0000) 2022-09-07T22:58:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:55.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:55.964377+0000) 2022-09-07T22:58:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:56 smithi055 ceph-mon[97033]: pgmap v2536: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:58:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:56 smithi059 ceph-mon[97450]: pgmap v2536: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:58:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:57.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:57.065352+0000) 2022-09-07T22:58:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:56.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:56.964582+0000) 2022-09-07T22:58:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:58.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:58.065540+0000) 2022-09-07T22:58:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:57.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:57.964830+0000) 2022-09-07T22:58:58.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:58:58 smithi055 ceph-mon[97033]: pgmap v2537: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:58:59.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:58:58 smithi059 ceph-mon[97450]: pgmap v2537: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:58:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:58:59.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:59.065765+0000) 2022-09-07T22:58:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:58.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:58.965022+0000) 2022-09-07T22:59:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:00.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:00.065935+0000) 2022-09-07T22:59:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:58:59.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:58:59.965250+0000) 2022-09-07T22:59:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:00 smithi055 ceph-mon[97033]: pgmap v2538: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:59:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:00 smithi059 ceph-mon[97450]: pgmap v2538: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T22:59:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:01.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:01.066158+0000) 2022-09-07T22:59:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:00.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:00.965390+0000) 2022-09-07T22:59:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:01.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:01.965624+0000) 2022-09-07T22:59:02.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:59:02.165+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:59:02.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:02.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:02.066390+0000) 2022-09-07T22:59:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:02 smithi055 ceph-mon[97033]: pgmap v2539: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:59:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:02 smithi059 ceph-mon[97450]: pgmap v2539: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:59:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:03.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:03.066632+0000) 2022-09-07T22:59:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:02.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:02.965827+0000) 2022-09-07T22:59:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:03.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:03.966049+0000) 2022-09-07T22:59:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:04.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:04.066892+0000) 2022-09-07T22:59:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:04 smithi055 ceph-mon[97033]: pgmap v2540: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:59:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:59:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:04 smithi059 ceph-mon[97450]: pgmap v2540: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T22:59:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T22:59:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:05.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:05.067090+0000) 2022-09-07T22:59:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:04.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:04.966254+0000) 2022-09-07T22:59:06.247 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:06.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:06.067318+0000) 2022-09-07T22:59:06.249 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:05.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:05.966483+0000) 2022-09-07T22:59:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:06 smithi055 ceph-mon[97033]: pgmap v2541: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:59:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:06 smithi059 ceph-mon[97450]: pgmap v2541: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T22:59:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:07.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:07.067510+0000) 2022-09-07T22:59:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:06.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:06.966743+0000) 2022-09-07T22:59:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:08.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:08.067657+0000) 2022-09-07T22:59:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:07.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:07.966958+0000) 2022-09-07T22:59:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:08 smithi055 ceph-mon[97033]: pgmap v2542: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:59:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:08 smithi059 ceph-mon[97450]: pgmap v2542: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T22:59:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:09.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:09.067803+0000) 2022-09-07T22:59:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:08.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:08.967146+0000) 2022-09-07T22:59:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:09 smithi055 ceph-mon[97033]: osdmap e251: 12 total, 11 up, 11 in 2022-09-07T22:59:10.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:09 smithi059 ceph-mon[97450]: osdmap e251: 12 total, 11 up, 11 in 2022-09-07T22:59:10.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:10.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:10.068008+0000) 2022-09-07T22:59:10.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:09.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:09.967402+0000) 2022-09-07T22:59:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:10 smithi055 ceph-mon[97033]: pgmap v2544: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:59:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:10 smithi059 ceph-mon[97450]: pgmap v2544: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:59:11.301 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:11.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:11.068209+0000) 2022-09-07T22:59:11.302 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:10.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:10.967655+0000) 2022-09-07T22:59:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:12.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:12.068455+0000) 2022-09-07T22:59:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:11.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:11.967909+0000) 2022-09-07T22:59:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:12 smithi055 ceph-mon[97033]: pgmap v2545: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:59:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:12 smithi055 ceph-mon[97033]: pgmap v2546: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 29 op/s 2022-09-07T22:59:12.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:12 smithi055 ceph-mon[97033]: pgmap v2547: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T22:59:12.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:59:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:12 smithi059 ceph-mon[97450]: pgmap v2545: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T22:59:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:12 smithi059 ceph-mon[97450]: pgmap v2546: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 29 op/s 2022-09-07T22:59:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:12 smithi059 ceph-mon[97450]: pgmap v2547: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T22:59:12.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T22:59:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:13.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:13.068702+0000) 2022-09-07T22:59:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:12.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:12.968179+0000) 2022-09-07T22:59:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:14.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:14.068934+0000) 2022-09-07T22:59:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:13.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:13.968433+0000) 2022-09-07T22:59:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:14 smithi055 ceph-mon[97033]: pgmap v2548: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T22:59:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:14 smithi055 ceph-mon[97033]: osdmap e252: 12 total, 11 up, 11 in 2022-09-07T22:59:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:14 smithi059 ceph-mon[97450]: pgmap v2548: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T22:59:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:14 smithi059 ceph-mon[97450]: osdmap e252: 12 total, 11 up, 11 in 2022-09-07T22:59:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:15.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:15.069165+0000) 2022-09-07T22:59:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:14.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:14.968723+0000) 2022-09-07T22:59:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:15.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:15.968959+0000) 2022-09-07T22:59:16.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:16.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:16.069404+0000) 2022-09-07T22:59:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:16 smithi055 ceph-mon[97033]: pgmap v2550: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 5.1 KiB/s rd, 326 B/s wr, 10 op/s 2022-09-07T22:59:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:16 smithi059 ceph-mon[97450]: pgmap v2550: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 5.1 KiB/s rd, 326 B/s wr, 10 op/s 2022-09-07T22:59:17.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:59:17.179+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:59:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:17.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:17.069620+0000) 2022-09-07T22:59:17.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:16.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:16.969214+0000) 2022-09-07T22:59:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:18.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:18.069804+0000) 2022-09-07T22:59:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:17.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:17.969410+0000) 2022-09-07T22:59:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:18 smithi055 ceph-mon[97033]: pgmap v2551: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T22:59:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:18 smithi059 ceph-mon[97450]: pgmap v2551: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-07T22:59:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:19.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:19.070055+0000) 2022-09-07T22:59:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:18.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:18.969610+0000) 2022-09-07T22:59:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:20.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:20.070282+0000) 2022-09-07T22:59:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:19.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:19.969880+0000) 2022-09-07T22:59:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:20 smithi055 ceph-mon[97033]: pgmap v2552: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:59:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:20 smithi059 ceph-mon[97450]: pgmap v2552: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T22:59:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:21.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:21.070543+0000) 2022-09-07T22:59:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:20.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:20.970127+0000) 2022-09-07T22:59:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:22.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:22.070789+0000) 2022-09-07T22:59:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:21.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:21.970381+0000) 2022-09-07T22:59:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:22 smithi055 ceph-mon[97033]: pgmap v2553: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:59:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:22 smithi059 ceph-mon[97450]: pgmap v2553: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:59:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:23.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:23.071058+0000) 2022-09-07T22:59:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:22.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:22.970632+0000) 2022-09-07T22:59:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:24.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:24.071209+0000) 2022-09-07T22:59:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:23.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:23.970765+0000) 2022-09-07T22:59:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:24 smithi055 ceph-mon[97033]: pgmap v2554: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:59:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:24 smithi059 ceph-mon[97450]: pgmap v2554: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T22:59:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:25.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:25.071416+0000) 2022-09-07T22:59:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:24.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:24.970991+0000) 2022-09-07T22:59:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:26.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:26.071608+0000) 2022-09-07T22:59:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:25.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:25.971166+0000) 2022-09-07T22:59:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:26 smithi055 ceph-mon[97033]: pgmap v2555: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T22:59:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:26 smithi059 ceph-mon[97450]: pgmap v2555: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T22:59:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:27.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:27.071862+0000) 2022-09-07T22:59:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:26.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:26.971442+0000) 2022-09-07T22:59:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:28.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:28.072052+0000) 2022-09-07T22:59:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:27.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:27.971668+0000) 2022-09-07T22:59:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:28 smithi055 ceph-mon[97033]: pgmap v2556: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:28 smithi059 ceph-mon[97450]: pgmap v2556: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:29.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:29.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:29.072277+0000) 2022-09-07T22:59:29.469 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:28.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:28.971797+0000) 2022-09-07T22:59:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:30.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:30.072508+0000) 2022-09-07T22:59:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:29.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:29.972000+0000) 2022-09-07T22:59:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:30 smithi055 ceph-mon[97033]: pgmap v2557: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:30 smithi059 ceph-mon[97450]: pgmap v2557: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:31.365 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:31.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:31.072753+0000) 2022-09-07T22:59:31.366 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:30.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:30.972155+0000) 2022-09-07T22:59:32.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:59:32.180+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:59:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:31.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:31.972393+0000) 2022-09-07T22:59:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:32.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:32.072991+0000) 2022-09-07T22:59:32.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:32 smithi055 ceph-mon[97033]: pgmap v2558: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:59:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:32 smithi059 ceph-mon[97450]: pgmap v2558: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:59:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:32.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:32.972661+0000) 2022-09-07T22:59:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:33.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:33.073232+0000) 2022-09-07T22:59:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:34.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:34.073460+0000) 2022-09-07T22:59:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:33.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:33.972889+0000) 2022-09-07T22:59:34.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:34 smithi055 ceph-mon[97033]: pgmap v2559: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:35.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:34 smithi059 ceph-mon[97450]: pgmap v2559: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:35.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:35.073709+0000) 2022-09-07T22:59:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:34.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:34.973183+0000) 2022-09-07T22:59:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:36.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:36.073894+0000) 2022-09-07T22:59:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:35.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:35.973428+0000) 2022-09-07T22:59:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:36 smithi055 ceph-mon[97033]: pgmap v2560: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:59:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:36 smithi059 ceph-mon[97450]: pgmap v2560: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:59:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:37.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:37.074078+0000) 2022-09-07T22:59:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:36.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:36.973706+0000) 2022-09-07T22:59:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:38.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:38.074221+0000) 2022-09-07T22:59:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:37.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:37.973877+0000) 2022-09-07T22:59:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:38 smithi055 ceph-mon[97033]: pgmap v2561: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:38 smithi059 ceph-mon[97450]: pgmap v2561: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:39.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:39.074431+0000) 2022-09-07T22:59:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:38.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:38.974080+0000) 2022-09-07T22:59:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:40.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:40.074689+0000) 2022-09-07T22:59:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:39.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:39.974323+0000) 2022-09-07T22:59:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:40 smithi055 ceph-mon[97033]: pgmap v2562: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:40 smithi059 ceph-mon[97450]: pgmap v2562: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:41.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:41.074876+0000) 2022-09-07T22:59:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:40.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:40.974520+0000) 2022-09-07T22:59:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:42.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:42.075034+0000) 2022-09-07T22:59:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:41.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:41.974752+0000) 2022-09-07T22:59:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:42 smithi055 ceph-mon[97033]: pgmap v2563: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:59:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:42 smithi059 ceph-mon[97450]: pgmap v2563: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:59:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:43.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:43.075177+0000) 2022-09-07T22:59:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:42.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:42.974978+0000) 2022-09-07T22:59:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:44.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:44.075426+0000) 2022-09-07T22:59:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:43.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:43.975185+0000) 2022-09-07T22:59:44.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:44 smithi055 ceph-mon[97033]: pgmap v2564: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:44 smithi059 ceph-mon[97450]: pgmap v2564: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:45.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:45.075752+0000) 2022-09-07T22:59:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:44.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:44.975399+0000) 2022-09-07T22:59:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:46.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:46.075942+0000) 2022-09-07T22:59:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:45.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:45.975631+0000) 2022-09-07T22:59:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:46 smithi055 ceph-mon[97033]: pgmap v2565: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:59:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:46 smithi059 ceph-mon[97450]: pgmap v2565: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:59:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:47.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:47.076162+0000) 2022-09-07T22:59:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:46.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:46.975899+0000) 2022-09-07T22:59:47.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 22:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T22:59:47.197+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T22:59:48.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:48.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:48.076407+0000) 2022-09-07T22:59:48.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:47.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:47.976172+0000) 2022-09-07T22:59:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:48 smithi055 ceph-mon[97033]: pgmap v2566: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:48 smithi059 ceph-mon[97450]: pgmap v2566: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:48.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:48.976420+0000) 2022-09-07T22:59:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:49.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:49.076671+0000) 2022-09-07T22:59:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:50.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:50.076860+0000) 2022-09-07T22:59:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:49.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:49.976689+0000) 2022-09-07T22:59:50.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:50 smithi055 ceph-mon[97033]: pgmap v2567: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:50 smithi059 ceph-mon[97450]: pgmap v2567: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:51.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:51.077085+0000) 2022-09-07T22:59:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:50.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:50.976910+0000) 2022-09-07T22:59:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:52.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:52.077304+0000) 2022-09-07T22:59:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:51.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:51.977100+0000) 2022-09-07T22:59:52.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:52 smithi055 ceph-mon[97033]: pgmap v2568: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:59:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:52 smithi059 ceph-mon[97450]: pgmap v2568: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:59:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:53.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:53.077551+0000) 2022-09-07T22:59:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:52.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:52.977346+0000) 2022-09-07T22:59:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:54.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:54.077797+0000) 2022-09-07T22:59:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:53.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:53.977555+0000) 2022-09-07T22:59:54.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:54 smithi055 ceph-mon[97033]: pgmap v2569: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:54 smithi059 ceph-mon[97450]: pgmap v2569: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:54.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:54.977792+0000) 2022-09-07T22:59:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:55.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:55.078018+0000) 2022-09-07T22:59:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:56.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:56.078256+0000) 2022-09-07T22:59:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:55.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:55.977966+0000) 2022-09-07T22:59:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:56 smithi055 ceph-mon[97033]: pgmap v2570: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:59:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:56 smithi059 ceph-mon[97450]: pgmap v2570: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T22:59:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:57.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:57.078475+0000) 2022-09-07T22:59:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:56.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:56.978218+0000) 2022-09-07T22:59:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:58.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:58.078757+0000) 2022-09-07T22:59:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:57.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:57.978410+0000) 2022-09-07T22:59:58.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 22:59:58 smithi055 ceph-mon[97033]: pgmap v2571: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:59.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 22:59:58 smithi059 ceph-mon[97450]: pgmap v2571: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T22:59:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 22:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T22:59:59.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:59.078960+0000) 2022-09-07T22:59:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:58.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:58.978630+0000) 2022-09-07T23:00:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:00.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:00.079106+0000) 2022-09-07T23:00:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 22:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T22:59:59.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T21:59:59.978846+0000) 2022-09-07T23:00:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:00 smithi055 ceph-mon[97033]: pgmap v2572: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-07T23:00:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-07T23:00:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-07T23:00:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T23:00:00.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T23:00:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:00 smithi059 ceph-mon[97450]: pgmap v2572: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-07T23:00:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-07T23:00:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-07T23:00:01.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T23:00:01.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T23:00:01.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:01.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:01.079231+0000) 2022-09-07T23:00:01.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:00.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:00.978977+0000) 2022-09-07T23:00:02.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:00:02.207+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:00:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:02.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:02.079457+0000) 2022-09-07T23:00:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:01.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:01.979186+0000) 2022-09-07T23:00:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:02 smithi055 ceph-mon[97033]: pgmap v2573: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:00:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:02 smithi059 ceph-mon[97450]: pgmap v2573: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:00:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:03.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:03.079665+0000) 2022-09-07T23:00:03.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:02.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:02.979464+0000) 2022-09-07T23:00:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:04.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:04.079861+0000) 2022-09-07T23:00:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:03.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:03.979745+0000) 2022-09-07T23:00:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:04 smithi055 ceph-mon[97033]: pgmap v2574: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:00:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:00:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:04 smithi059 ceph-mon[97450]: pgmap v2574: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:00:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:00:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:05.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:05.080093+0000) 2022-09-07T23:00:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:04.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:04.979949+0000) 2022-09-07T23:00:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:06.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:06.080272+0000) 2022-09-07T23:00:06.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:05.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:05.980144+0000) 2022-09-07T23:00:06.921 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:06 smithi059 ceph-mon[97450]: pgmap v2575: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:00:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:06 smithi055 ceph-mon[97033]: pgmap v2575: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:00:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:07.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:07.080391+0000) 2022-09-07T23:00:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:06.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:06.980416+0000) 2022-09-07T23:00:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:07.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:07.980619+0000) 2022-09-07T23:00:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:08.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:08.080579+0000) 2022-09-07T23:00:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:08 smithi055 ceph-mon[97033]: pgmap v2576: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:00:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:08 smithi059 ceph-mon[97450]: pgmap v2576: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:00:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:09.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:09.080824+0000) 2022-09-07T23:00:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:08.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:08.980770+0000) 2022-09-07T23:00:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:09 smithi055 ceph-mon[97033]: osdmap e253: 12 total, 11 up, 11 in 2022-09-07T23:00:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:09 smithi059 ceph-mon[97450]: osdmap e253: 12 total, 11 up, 11 in 2022-09-07T23:00:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:10.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:10.081011+0000) 2022-09-07T23:00:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:09.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:09.980938+0000) 2022-09-07T23:00:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:10 smithi055 ceph-mon[97033]: pgmap v2578: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:00:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:10 smithi059 ceph-mon[97450]: pgmap v2578: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:00:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:10.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:10.981186+0000) 2022-09-07T23:00:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:11.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:11.081270+0000) 2022-09-07T23:00:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:00:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:00:11.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:00:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:00:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:00:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:00:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:12.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:12.081491+0000) 2022-09-07T23:00:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:11.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:11.981437+0000) 2022-09-07T23:00:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:12 smithi055 ceph-mon[97033]: pgmap v2579: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:00:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:12 smithi059 ceph-mon[97450]: pgmap v2579: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:00:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:13.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:13.081726+0000) 2022-09-07T23:00:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:12.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:12.981728+0000) 2022-09-07T23:00:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:13.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:13.981970+0000) 2022-09-07T23:00:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:14.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:14.081912+0000) 2022-09-07T23:00:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:14 smithi055 ceph-mon[97033]: pgmap v2580: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:00:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:14 smithi055 ceph-mon[97033]: osdmap e254: 12 total, 11 up, 11 in 2022-09-07T23:00:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:14 smithi059 ceph-mon[97450]: pgmap v2580: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:00:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:14 smithi059 ceph-mon[97450]: osdmap e254: 12 total, 11 up, 11 in 2022-09-07T23:00:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:15.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:15.082054+0000) 2022-09-07T23:00:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:14.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:14.982228+0000) 2022-09-07T23:00:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:15.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:15.982447+0000) 2022-09-07T23:00:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:16.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:16.082198+0000) 2022-09-07T23:00:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:16 smithi055 ceph-mon[97033]: pgmap v2582: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:00:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:16 smithi059 ceph-mon[97450]: pgmap v2582: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:00:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:17.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:17.082439+0000) 2022-09-07T23:00:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:16.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:16.982587+0000) 2022-09-07T23:00:17.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:00:17.215+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:00:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:18.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:18.082714+0000) 2022-09-07T23:00:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:17.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:17.982697+0000) 2022-09-07T23:00:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:18 smithi055 ceph-mon[97033]: pgmap v2583: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 236 B/s wr, 7 op/s 2022-09-07T23:00:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:18 smithi059 ceph-mon[97450]: pgmap v2583: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 236 B/s wr, 7 op/s 2022-09-07T23:00:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:19.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:19.082945+0000) 2022-09-07T23:00:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:18.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:18.982915+0000) 2022-09-07T23:00:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:20.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:20.083193+0000) 2022-09-07T23:00:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:19.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:19.983137+0000) 2022-09-07T23:00:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:20 smithi055 ceph-mon[97033]: pgmap v2584: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:00:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:20 smithi059 ceph-mon[97450]: pgmap v2584: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:00:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:20.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:20.983370+0000) 2022-09-07T23:00:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:21.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:21.083406+0000) 2022-09-07T23:00:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:22.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:22.083676+0000) 2022-09-07T23:00:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:21.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:21.983601+0000) 2022-09-07T23:00:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:22 smithi055 ceph-mon[97033]: pgmap v2585: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:00:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:22 smithi059 ceph-mon[97450]: pgmap v2585: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:00:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:23.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:23.083858+0000) 2022-09-07T23:00:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:22.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:22.983845+0000) 2022-09-07T23:00:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:24.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:24.084103+0000) 2022-09-07T23:00:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:23.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:23.984078+0000) 2022-09-07T23:00:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:24 smithi055 ceph-mon[97033]: pgmap v2586: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:00:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:24 smithi059 ceph-mon[97450]: pgmap v2586: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:00:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:25.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:25.084381+0000) 2022-09-07T23:00:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:24.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:24.984301+0000) 2022-09-07T23:00:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:26.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:26.084602+0000) 2022-09-07T23:00:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:25.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:25.984503+0000) 2022-09-07T23:00:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:26 smithi055 ceph-mon[97033]: pgmap v2587: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:00:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:26 smithi059 ceph-mon[97450]: pgmap v2587: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:00:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:27.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:27.084766+0000) 2022-09-07T23:00:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:26.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:26.984767+0000) 2022-09-07T23:00:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:28.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:28.085025+0000) 2022-09-07T23:00:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:27.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:27.985038+0000) 2022-09-07T23:00:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:28 smithi055 ceph-mon[97033]: pgmap v2588: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:28 smithi059 ceph-mon[97450]: pgmap v2588: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:28.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:28.985254+0000) 2022-09-07T23:00:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:29.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:29.085274+0000) 2022-09-07T23:00:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:30.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:30.085465+0000) 2022-09-07T23:00:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:29.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:29.985477+0000) 2022-09-07T23:00:30.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:30 smithi055 ceph-mon[97033]: pgmap v2589: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:30 smithi059 ceph-mon[97450]: pgmap v2589: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:31.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:31.085743+0000) 2022-09-07T23:00:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:30.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:30.985679+0000) 2022-09-07T23:00:32.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:00:32.219+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:00:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:32.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:32.086003+0000) 2022-09-07T23:00:32.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:31.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:31.985886+0000) 2022-09-07T23:00:32.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:32 smithi055 ceph-mon[97033]: pgmap v2590: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:00:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:32 smithi059 ceph-mon[97450]: pgmap v2590: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:00:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:33.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:33.086298+0000) 2022-09-07T23:00:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:32.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:32.986157+0000) 2022-09-07T23:00:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:34.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:34.086523+0000) 2022-09-07T23:00:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:33.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:33.986366+0000) 2022-09-07T23:00:34.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:34 smithi055 ceph-mon[97033]: pgmap v2591: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:34 smithi059 ceph-mon[97450]: pgmap v2591: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:34.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:34.986609+0000) 2022-09-07T23:00:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:35.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:35.086727+0000) 2022-09-07T23:00:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:36.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:36.086929+0000) 2022-09-07T23:00:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:35.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:35.986816+0000) 2022-09-07T23:00:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:36 smithi055 ceph-mon[97033]: pgmap v2592: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:00:37.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:36 smithi059 ceph-mon[97450]: pgmap v2592: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:00:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:37.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:37.087160+0000) 2022-09-07T23:00:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:36.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:36.986966+0000) 2022-09-07T23:00:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:38.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:38.087425+0000) 2022-09-07T23:00:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:37.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:37.987187+0000) 2022-09-07T23:00:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:38 smithi055 ceph-mon[97033]: pgmap v2593: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:38 smithi059 ceph-mon[97450]: pgmap v2593: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:39.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:39.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:39.087674+0000) 2022-09-07T23:00:39.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:38.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:38.987393+0000) 2022-09-07T23:00:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:40.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:40.087794+0000) 2022-09-07T23:00:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:39.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:39.987563+0000) 2022-09-07T23:00:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:40 smithi055 ceph-mon[97033]: pgmap v2594: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:40 smithi059 ceph-mon[97450]: pgmap v2594: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:41.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:41.088032+0000) 2022-09-07T23:00:41.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:40.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:40.987723+0000) 2022-09-07T23:00:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:42.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:42.088250+0000) 2022-09-07T23:00:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:41.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:41.987927+0000) 2022-09-07T23:00:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:42 smithi055 ceph-mon[97033]: pgmap v2595: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:00:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:42 smithi059 ceph-mon[97450]: pgmap v2595: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:00:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:42.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:42.988148+0000) 2022-09-07T23:00:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:43.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:43.088499+0000) 2022-09-07T23:00:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:43.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:43.988364+0000) 2022-09-07T23:00:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:44.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:44.088660+0000) 2022-09-07T23:00:44.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:44 smithi055 ceph-mon[97033]: pgmap v2596: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:45.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:44 smithi059 ceph-mon[97450]: pgmap v2596: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:00:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:45.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:45.088794+0000) 2022-09-07T23:00:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:44.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:44.988581+0000) 2022-09-07T23:00:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:46.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:46.088963+0000) 2022-09-07T23:00:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:45.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:45.988866+0000) 2022-09-07T23:00:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:46 smithi055 ceph-mon[97033]: pgmap v2597: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:00:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:00:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:00:46.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:00:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:46 smithi059 ceph-mon[97450]: pgmap v2597: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:00:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:00:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:00:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:00:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:47.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:47.089155+0000) 2022-09-07T23:00:47.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:00:47.226+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:00:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:46.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:46.989012+0000) 2022-09-07T23:00:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:47 smithi055 ceph-mon[97033]: pgmap v2598: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:00:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:47 smithi055 ceph-mon[97033]: pgmap v2599: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-07T23:00:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:47 smithi059 ceph-mon[97450]: pgmap v2598: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:00:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:47 smithi059 ceph-mon[97450]: pgmap v2599: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-07T23:00:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:48.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:48.089327+0000) 2022-09-07T23:00:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:47.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:47.989237+0000) 2022-09-07T23:00:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:49.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:49.089558+0000) 2022-09-07T23:00:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:48.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:48.989469+0000) 2022-09-07T23:00:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:49 smithi055 ceph-mon[97033]: pgmap v2600: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-07T23:00:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:49 smithi059 ceph-mon[97450]: pgmap v2600: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-07T23:00:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:49.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:49.989671+0000) 2022-09-07T23:00:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:50.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:50.089791+0000) 2022-09-07T23:00:51.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:50.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:50.989869+0000) 2022-09-07T23:00:51.409 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:51.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:51.090047+0000) 2022-09-07T23:00:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:51 smithi055 ceph-mon[97033]: pgmap v2601: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-07T23:00:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:51 smithi059 ceph-mon[97450]: pgmap v2601: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-07T23:00:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:52.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:52.090291+0000) 2022-09-07T23:00:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:51.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:51.990120+0000) 2022-09-07T23:00:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:53.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:53.090457+0000) 2022-09-07T23:00:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:52.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:52.990337+0000) 2022-09-07T23:00:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:53 smithi055 ceph-mon[97033]: pgmap v2602: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-07T23:00:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:53 smithi059 ceph-mon[97450]: pgmap v2602: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-07T23:00:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:54.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:54.090728+0000) 2022-09-07T23:00:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:53.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:53.990571+0000) 2022-09-07T23:00:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:55.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:55.090929+0000) 2022-09-07T23:00:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:54.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:54.990806+0000) 2022-09-07T23:00:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:55 smithi055 ceph-mon[97033]: pgmap v2603: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-07T23:00:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:55 smithi059 ceph-mon[97450]: pgmap v2603: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-07T23:00:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:56.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:56.091086+0000) 2022-09-07T23:00:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:55.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:55.991024+0000) 2022-09-07T23:00:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:57.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:57.091272+0000) 2022-09-07T23:00:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:56.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:56.991243+0000) 2022-09-07T23:00:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:57 smithi055 ceph-mon[97033]: pgmap v2604: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:00:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:57 smithi059 ceph-mon[97450]: pgmap v2604: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:00:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:58.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:58.091500+0000) 2022-09-07T23:00:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:57.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:57.991484+0000) 2022-09-07T23:00:59.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:00:59.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:59.091741+0000) 2022-09-07T23:00:59.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:58.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:58.991699+0000) 2022-09-07T23:00:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:00:59 smithi055 ceph-mon[97033]: pgmap v2605: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:00:59 smithi059 ceph-mon[97450]: pgmap v2605: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:00.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:00.091878+0000) 2022-09-07T23:01:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:00:59.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:00:59.991882+0000) 2022-09-07T23:01:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:01.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:01.092027+0000) 2022-09-07T23:01:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:00.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:00.992097+0000) 2022-09-07T23:01:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:01 smithi059 ceph-mon[97450]: pgmap v2606: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:01:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:01 smithi055 ceph-mon[97033]: pgmap v2606: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:01:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:02.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:02.092253+0000) 2022-09-07T23:01:02.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:01:02.234+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:01:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:01.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:01.992248+0000) 2022-09-07T23:01:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:02.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:02.992435+0000) 2022-09-07T23:01:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:03.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:03.092490+0000) 2022-09-07T23:01:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:03 smithi055 ceph-mon[97033]: pgmap v2607: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:03 smithi059 ceph-mon[97450]: pgmap v2607: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:04.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:04.092696+0000) 2022-09-07T23:01:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:03.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:03.992692+0000) 2022-09-07T23:01:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:01:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:01:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:01:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:01:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:04.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:04.992935+0000) 2022-09-07T23:01:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:05.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:05.092918+0000) 2022-09-07T23:01:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:05 smithi055 ceph-mon[97033]: pgmap v2608: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:05 smithi059 ceph-mon[97450]: pgmap v2608: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:05.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:05.993125+0000) 2022-09-07T23:01:06.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:06.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:06.093169+0000) 2022-09-07T23:01:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:06.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:06.993317+0000) 2022-09-07T23:01:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:07.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:07.093398+0000) 2022-09-07T23:01:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:07 smithi055 ceph-mon[97033]: pgmap v2609: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:01:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:07 smithi059 ceph-mon[97450]: pgmap v2609: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:01:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:07.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:07.993573+0000) 2022-09-07T23:01:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:08.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:08.093681+0000) 2022-09-07T23:01:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:08.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:08.993806+0000) 2022-09-07T23:01:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:09.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:09.093887+0000) 2022-09-07T23:01:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:09 smithi055 ceph-mon[97033]: pgmap v2610: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:01:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:09 smithi055 ceph-mon[97033]: osdmap e255: 12 total, 11 up, 11 in 2022-09-07T23:01:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:09 smithi059 ceph-mon[97450]: pgmap v2610: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:01:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:09 smithi059 ceph-mon[97450]: osdmap e255: 12 total, 11 up, 11 in 2022-09-07T23:01:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:10.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:10.094088+0000) 2022-09-07T23:01:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:09.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:09.994052+0000) 2022-09-07T23:01:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:10.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:10.994292+0000) 2022-09-07T23:01:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:11.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:11.094298+0000) 2022-09-07T23:01:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:11 smithi055 ceph-mon[97033]: pgmap v2612: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:01:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:11 smithi059 ceph-mon[97450]: pgmap v2612: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:01:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:12.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:12.094527+0000) 2022-09-07T23:01:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:11.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:11.994452+0000) 2022-09-07T23:01:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:12.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:12.994687+0000) 2022-09-07T23:01:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:13.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:13.094735+0000) 2022-09-07T23:01:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:13 smithi055 ceph-mon[97033]: pgmap v2613: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:01:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:13 smithi055 ceph-mon[97033]: osdmap e256: 12 total, 11 up, 11 in 2022-09-07T23:01:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:13 smithi059 ceph-mon[97450]: pgmap v2613: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:01:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:13 smithi059 ceph-mon[97450]: osdmap e256: 12 total, 11 up, 11 in 2022-09-07T23:01:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:14.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:14.094878+0000) 2022-09-07T23:01:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:13.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:13.994901+0000) 2022-09-07T23:01:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:15.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:15.095088+0000) 2022-09-07T23:01:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:14.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:14.995127+0000) 2022-09-07T23:01:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:15 smithi055 ceph-mon[97033]: pgmap v2615: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:01:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:15 smithi059 ceph-mon[97450]: pgmap v2615: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:01:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:16.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:16.095264+0000) 2022-09-07T23:01:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:15.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:15.995331+0000) 2022-09-07T23:01:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:17.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:17.095481+0000) 2022-09-07T23:01:17.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:01:17.241+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:01:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:16.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:16.995539+0000) 2022-09-07T23:01:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:17 smithi055 ceph-mon[97033]: pgmap v2616: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:01:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:17 smithi059 ceph-mon[97450]: pgmap v2616: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:01:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:18.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:18.095697+0000) 2022-09-07T23:01:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:17.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:17.995753+0000) 2022-09-07T23:01:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:19.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:19.095859+0000) 2022-09-07T23:01:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:18.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:18.995923+0000) 2022-09-07T23:01:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:19 smithi055 ceph-mon[97033]: pgmap v2617: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-07T23:01:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:19 smithi059 ceph-mon[97450]: pgmap v2617: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-07T23:01:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:20.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:20.095978+0000) 2022-09-07T23:01:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:19.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:19.996158+0000) 2022-09-07T23:01:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:20.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:20.996353+0000) 2022-09-07T23:01:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:21.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:21.096179+0000) 2022-09-07T23:01:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:21 smithi055 ceph-mon[97033]: pgmap v2618: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:01:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:21 smithi059 ceph-mon[97450]: pgmap v2618: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:01:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:22.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:22.096440+0000) 2022-09-07T23:01:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:21.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:21.996517+0000) 2022-09-07T23:01:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:23.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:23.096739+0000) 2022-09-07T23:01:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:22.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:22.996720+0000) 2022-09-07T23:01:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:23 smithi055 ceph-mon[97033]: pgmap v2619: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:01:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:23 smithi059 ceph-mon[97450]: pgmap v2619: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:01:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:24.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:24.096989+0000) 2022-09-07T23:01:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:23.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:23.996885+0000) 2022-09-07T23:01:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:25.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:25.097238+0000) 2022-09-07T23:01:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:24.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:24.997079+0000) 2022-09-07T23:01:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:25 smithi055 ceph-mon[97033]: pgmap v2620: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-07T23:01:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:25 smithi059 ceph-mon[97450]: pgmap v2620: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-07T23:01:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:26.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:26.097486+0000) 2022-09-07T23:01:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:25.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:25.997310+0000) 2022-09-07T23:01:27.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:27.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:27.097749+0000) 2022-09-07T23:01:27.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:26.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:26.997571+0000) 2022-09-07T23:01:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:27 smithi055 ceph-mon[97033]: pgmap v2621: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:01:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:27 smithi059 ceph-mon[97450]: pgmap v2621: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:01:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:28.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:28.097988+0000) 2022-09-07T23:01:28.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:27.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:27.997798+0000) 2022-09-07T23:01:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:29.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:29.098186+0000) 2022-09-07T23:01:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:28.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:28.998040+0000) 2022-09-07T23:01:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:29 smithi055 ceph-mon[97033]: pgmap v2622: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:29 smithi059 ceph-mon[97450]: pgmap v2622: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:30.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:30.098420+0000) 2022-09-07T23:01:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:29.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:29.998296+0000) 2022-09-07T23:01:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:31.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:31.098672+0000) 2022-09-07T23:01:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:30.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:30.998577+0000) 2022-09-07T23:01:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:31 smithi055 ceph-mon[97033]: pgmap v2623: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:01:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:31 smithi059 ceph-mon[97450]: pgmap v2623: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:01:32.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:01:32.244+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:01:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:32.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:32.098853+0000) 2022-09-07T23:01:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:31.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:31.998788+0000) 2022-09-07T23:01:33.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:33.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:33.099033+0000) 2022-09-07T23:01:33.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:32.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:32.998957+0000) 2022-09-07T23:01:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:33 smithi055 ceph-mon[97033]: pgmap v2624: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:33 smithi059 ceph-mon[97450]: pgmap v2624: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:34.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:34.099230+0000) 2022-09-07T23:01:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:33.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:33.999123+0000) 2022-09-07T23:01:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:35.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:35.099464+0000) 2022-09-07T23:01:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:34.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:34.999362+0000) 2022-09-07T23:01:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:35 smithi059 ceph-mon[97450]: pgmap v2625: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:35 smithi055 ceph-mon[97033]: pgmap v2625: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:36.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:36.099604+0000) 2022-09-07T23:01:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:35.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:35.999584+0000) 2022-09-07T23:01:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:37.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:37.099843+0000) 2022-09-07T23:01:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:36.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:36.999832+0000) 2022-09-07T23:01:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:37 smithi059 ceph-mon[97450]: pgmap v2626: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:01:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:37 smithi055 ceph-mon[97033]: pgmap v2626: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:01:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:37.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:38.000094+0000) 2022-09-07T23:01:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:38.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:38.100014+0000) 2022-09-07T23:01:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:39.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:39.100197+0000) 2022-09-07T23:01:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:38.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:39.000254+0000) 2022-09-07T23:01:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:39 smithi059 ceph-mon[97450]: pgmap v2627: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:39 smithi055 ceph-mon[97033]: pgmap v2627: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:40.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:40.100430+0000) 2022-09-07T23:01:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:40.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:40.000512+0000) 2022-09-07T23:01:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:40.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:41.000759+0000) 2022-09-07T23:01:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:41.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:41.100609+0000) 2022-09-07T23:01:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:41 smithi059 ceph-mon[97450]: pgmap v2628: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:01:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:41 smithi055 ceph-mon[97033]: pgmap v2628: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:01:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:41.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:42.000973+0000) 2022-09-07T23:01:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:42.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:42.100821+0000) 2022-09-07T23:01:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:43.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:43.001222+0000) 2022-09-07T23:01:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:43.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:43.101051+0000) 2022-09-07T23:01:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:43 smithi059 ceph-mon[97450]: pgmap v2629: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:43 smithi055 ceph-mon[97033]: pgmap v2629: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:44.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:44.001453+0000) 2022-09-07T23:01:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:44.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:44.101297+0000) 2022-09-07T23:01:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:45.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:45.101532+0000) 2022-09-07T23:01:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:45.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:45.001721+0000) 2022-09-07T23:01:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:45 smithi059 ceph-mon[97450]: pgmap v2630: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:46.168 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:45 smithi055 ceph-mon[97033]: pgmap v2630: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:01:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:46.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:46.101749+0000) 2022-09-07T23:01:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:46.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:46.001853+0000) 2022-09-07T23:01:46.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:01:46.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:01:46.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:01:46.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:01:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:01:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:01:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:01:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:01:47.260 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:47.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:47.101978+0000) 2022-09-07T23:01:47.261 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:47.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:47.002070+0000) 2022-09-07T23:01:47.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:01:47.258+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:01:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:47 smithi059 ceph-mon[97450]: pgmap v2631: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:01:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:47 smithi059 ceph-mon[97450]: pgmap v2632: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:01:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:47 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-07T23:01:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:47 smithi055 ceph-mon[97033]: pgmap v2631: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:01:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:47 smithi055 ceph-mon[97033]: pgmap v2632: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:01:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:47 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-07T23:01:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:48.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:48.102139+0000) 2022-09-07T23:01:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:48.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:48.002311+0000) 2022-09-07T23:01:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:49.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:49.002555+0000) 2022-09-07T23:01:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:49.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:49.102269+0000) 2022-09-07T23:01:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:49 smithi055 ceph-mon[97033]: pgmap v2633: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:01:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:49 smithi059 ceph-mon[97450]: pgmap v2633: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:01:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:50.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:50.102494+0000) 2022-09-07T23:01:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:50.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:50.002746+0000) 2022-09-07T23:01:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:51.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:51.102727+0000) 2022-09-07T23:01:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:51.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:51.002889+0000) 2022-09-07T23:01:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:51 smithi055 ceph-mon[97033]: pgmap v2634: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:01:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:51 smithi059 ceph-mon[97450]: pgmap v2634: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:01:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:52.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:52.102974+0000) 2022-09-07T23:01:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:52.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:52.003113+0000) 2022-09-07T23:01:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:53.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:53.103226+0000) 2022-09-07T23:01:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:53.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:53.003297+0000) 2022-09-07T23:01:53.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:53 smithi055 ceph-mon[97033]: pgmap v2635: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:01:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:53 smithi059 ceph-mon[97450]: pgmap v2635: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:01:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:54.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:54.103477+0000) 2022-09-07T23:01:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:54.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:54.003476+0000) 2022-09-07T23:01:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:55.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:55.003701+0000) 2022-09-07T23:01:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:55.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:55.103686+0000) 2022-09-07T23:01:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:55 smithi059 ceph-mon[97450]: pgmap v2636: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:01:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:55 smithi055 ceph-mon[97033]: pgmap v2636: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:01:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:56.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:56.003907+0000) 2022-09-07T23:01:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:56.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:56.103911+0000) 2022-09-07T23:01:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:57.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:57.104136+0000) 2022-09-07T23:01:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:57.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:57.004162+0000) 2022-09-07T23:01:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:57 smithi059 ceph-mon[97450]: pgmap v2637: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:01:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:57 smithi055 ceph-mon[97033]: pgmap v2637: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:01:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:58.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:58.104328+0000) 2022-09-07T23:01:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:58.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:58.004332+0000) 2022-09-07T23:01:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:01:59.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:59.104573+0000) 2022-09-07T23:01:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:01:59.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:01:59.004543+0000) 2022-09-07T23:01:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:01:59 smithi055 ceph-mon[97033]: pgmap v2638: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:01:59 smithi059 ceph-mon[97450]: pgmap v2638: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:00.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:00.104762+0000) 2022-09-07T23:02:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:00.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:00.004692+0000) 2022-09-07T23:02:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:01.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:01.104951+0000) 2022-09-07T23:02:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:01.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:01.004841+0000) 2022-09-07T23:02:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:01 smithi055 ceph-mon[97033]: pgmap v2639: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:02:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:01 smithi059 ceph-mon[97450]: pgmap v2639: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:02:02.274 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:02.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:02.105157+0000) 2022-09-07T23:02:02.275 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:02.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:02.005031+0000) 2022-09-07T23:02:02.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:02:02.272+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:02:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:03.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:03.105294+0000) 2022-09-07T23:02:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:03.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:03.005229+0000) 2022-09-07T23:02:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:03 smithi055 ceph-mon[97033]: pgmap v2640: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:03 smithi059 ceph-mon[97450]: pgmap v2640: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:04.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:04.105498+0000) 2022-09-07T23:02:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:04.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:04.005457+0000) 2022-09-07T23:02:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:02:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:02:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:02:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:02:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:05.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:05.105763+0000) 2022-09-07T23:02:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:05.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:05.005733+0000) 2022-09-07T23:02:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:05 smithi059 ceph-mon[97450]: pgmap v2641: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:05 smithi055 ceph-mon[97033]: pgmap v2641: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:06.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:06.106010+0000) 2022-09-07T23:02:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:06.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:06.005917+0000) 2022-09-07T23:02:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:07.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:07.106204+0000) 2022-09-07T23:02:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:07.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:07.006139+0000) 2022-09-07T23:02:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:07 smithi059 ceph-mon[97450]: pgmap v2642: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:02:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:07 smithi055 ceph-mon[97033]: pgmap v2642: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:02:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:08.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:08.106443+0000) 2022-09-07T23:02:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:08.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:08.006373+0000) 2022-09-07T23:02:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:09.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:09.106693+0000) 2022-09-07T23:02:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:09.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:09.006519+0000) 2022-09-07T23:02:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:09 smithi059 ceph-mon[97450]: pgmap v2643: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:02:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:09 smithi059 ceph-mon[97450]: osdmap e257: 12 total, 11 up, 11 in 2022-09-07T23:02:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:09 smithi055 ceph-mon[97033]: pgmap v2643: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:02:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:09 smithi055 ceph-mon[97033]: osdmap e257: 12 total, 11 up, 11 in 2022-09-07T23:02:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:10.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:10.106935+0000) 2022-09-07T23:02:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:10.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:10.006744+0000) 2022-09-07T23:02:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:11.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:11.107160+0000) 2022-09-07T23:02:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:11.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:11.006912+0000) 2022-09-07T23:02:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:11 smithi059 ceph-mon[97450]: pgmap v2645: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:02:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:11 smithi055 ceph-mon[97033]: pgmap v2645: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:02:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:12.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:12.007098+0000) 2022-09-07T23:02:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:12.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:12.107379+0000) 2022-09-07T23:02:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:13.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:13.007290+0000) 2022-09-07T23:02:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:13.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:13.107543+0000) 2022-09-07T23:02:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:13 smithi059 ceph-mon[97450]: pgmap v2646: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:02:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:13 smithi059 ceph-mon[97450]: osdmap e258: 12 total, 11 up, 11 in 2022-09-07T23:02:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:13 smithi055 ceph-mon[97033]: pgmap v2646: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:02:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:13 smithi055 ceph-mon[97033]: osdmap e258: 12 total, 11 up, 11 in 2022-09-07T23:02:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:14.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:14.107770+0000) 2022-09-07T23:02:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:14.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:14.007454+0000) 2022-09-07T23:02:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:15.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:15.007722+0000) 2022-09-07T23:02:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:15.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:15.108001+0000) 2022-09-07T23:02:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:15 smithi059 ceph-mon[97450]: pgmap v2648: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:02:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:15 smithi055 ceph-mon[97033]: pgmap v2648: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:02:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:16.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:16.108141+0000) 2022-09-07T23:02:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:16.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:16.007851+0000) 2022-09-07T23:02:17.288 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:17.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:17.108361+0000) 2022-09-07T23:02:17.288 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:17.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:17.008023+0000) 2022-09-07T23:02:17.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:02:17.285+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:02:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:17 smithi059 ceph-mon[97450]: pgmap v2649: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:02:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:17 smithi055 ceph-mon[97033]: pgmap v2649: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:02:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:18.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:18.008243+0000) 2022-09-07T23:02:18.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:18.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:18.108518+0000) 2022-09-07T23:02:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:19.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:19.008485+0000) 2022-09-07T23:02:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:19.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:19.108722+0000) 2022-09-07T23:02:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:19 smithi055 ceph-mon[97033]: pgmap v2650: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:02:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:19 smithi059 ceph-mon[97450]: pgmap v2650: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:02:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:20.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:20.108959+0000) 2022-09-07T23:02:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:20.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:20.008739+0000) 2022-09-07T23:02:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:21.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:21.109207+0000) 2022-09-07T23:02:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:21.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:21.008933+0000) 2022-09-07T23:02:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:21 smithi055 ceph-mon[97033]: pgmap v2651: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:02:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:21 smithi059 ceph-mon[97450]: pgmap v2651: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:02:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:22.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:22.109478+0000) 2022-09-07T23:02:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:22.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:22.009146+0000) 2022-09-07T23:02:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:23.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:23.109625+0000) 2022-09-07T23:02:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:23.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:23.009336+0000) 2022-09-07T23:02:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:23 smithi059 ceph-mon[97450]: pgmap v2652: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:02:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:23 smithi055 ceph-mon[97033]: pgmap v2652: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:02:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:24.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:24.109820+0000) 2022-09-07T23:02:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:24.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:24.009537+0000) 2022-09-07T23:02:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:25.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:25.110045+0000) 2022-09-07T23:02:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:25.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:25.009778+0000) 2022-09-07T23:02:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:25 smithi059 ceph-mon[97450]: pgmap v2653: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:02:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:25 smithi055 ceph-mon[97033]: pgmap v2653: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:02:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:26.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:26.110300+0000) 2022-09-07T23:02:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:26.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:26.009966+0000) 2022-09-07T23:02:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:27.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:27.110553+0000) 2022-09-07T23:02:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:27.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:27.010209+0000) 2022-09-07T23:02:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:27 smithi059 ceph-mon[97450]: pgmap v2654: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:02:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:27 smithi055 ceph-mon[97033]: pgmap v2654: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:02:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:28.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:28.110797+0000) 2022-09-07T23:02:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:28.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:28.010429+0000) 2022-09-07T23:02:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:29.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:29.111043+0000) 2022-09-07T23:02:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:29.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:29.010693+0000) 2022-09-07T23:02:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:29 smithi055 ceph-mon[97033]: pgmap v2655: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:29 smithi059 ceph-mon[97450]: pgmap v2655: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:30.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:30.111220+0000) 2022-09-07T23:02:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:30.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:30.010849+0000) 2022-09-07T23:02:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:31.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:31.011090+0000) 2022-09-07T23:02:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:31.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:31.111395+0000) 2022-09-07T23:02:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:31 smithi059 ceph-mon[97450]: pgmap v2656: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:02:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:31 smithi055 ceph-mon[97033]: pgmap v2656: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:02:32.288 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:32.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:32.111609+0000) 2022-09-07T23:02:32.289 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:32.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:32.011367+0000) 2022-09-07T23:02:32.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:02:32.285+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:02:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:33.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:33.111809+0000) 2022-09-07T23:02:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:33.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:33.011618+0000) 2022-09-07T23:02:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:33 smithi059 ceph-mon[97450]: pgmap v2657: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:33 smithi055 ceph-mon[97033]: pgmap v2657: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:34.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:34.112033+0000) 2022-09-07T23:02:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:34.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:34.011852+0000) 2022-09-07T23:02:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:35.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:35.112235+0000) 2022-09-07T23:02:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:35.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:35.012022+0000) 2022-09-07T23:02:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:35 smithi059 ceph-mon[97450]: pgmap v2658: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:36.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:35 smithi055 ceph-mon[97033]: pgmap v2658: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:36.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:36.112442+0000) 2022-09-07T23:02:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:36.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:36.012267+0000) 2022-09-07T23:02:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:37.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:37.112710+0000) 2022-09-07T23:02:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:37.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:37.012522+0000) 2022-09-07T23:02:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:37 smithi059 ceph-mon[97450]: pgmap v2659: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:02:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:37 smithi055 ceph-mon[97033]: pgmap v2659: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:02:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:38.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:38.112980+0000) 2022-09-07T23:02:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:38.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:38.012743+0000) 2022-09-07T23:02:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:39.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:39.113237+0000) 2022-09-07T23:02:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:39.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:39.012940+0000) 2022-09-07T23:02:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:39 smithi055 ceph-mon[97033]: pgmap v2660: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:39 smithi059 ceph-mon[97450]: pgmap v2660: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:40.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:40.113413+0000) 2022-09-07T23:02:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:40.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:40.013084+0000) 2022-09-07T23:02:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:41.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:41.113565+0000) 2022-09-07T23:02:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:41.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:41.013317+0000) 2022-09-07T23:02:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:41 smithi059 ceph-mon[97450]: pgmap v2661: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:02:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:41 smithi055 ceph-mon[97033]: pgmap v2661: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:02:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:42.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:42.113730+0000) 2022-09-07T23:02:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:42.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:42.013447+0000) 2022-09-07T23:02:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:43.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:43.113986+0000) 2022-09-07T23:02:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:43.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:43.013673+0000) 2022-09-07T23:02:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:43 smithi059 ceph-mon[97450]: pgmap v2662: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:43 smithi055 ceph-mon[97033]: pgmap v2662: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:44.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:44.114109+0000) 2022-09-07T23:02:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:44.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:44.013807+0000) 2022-09-07T23:02:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:45.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:45.114303+0000) 2022-09-07T23:02:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:45.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:45.014049+0000) 2022-09-07T23:02:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:45 smithi059 ceph-mon[97450]: pgmap v2663: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:45 smithi055 ceph-mon[97033]: pgmap v2663: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:46.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:46.114436+0000) 2022-09-07T23:02:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:46.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:46.014178+0000) 2022-09-07T23:02:47.304 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:47.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:47.114664+0000) 2022-09-07T23:02:47.305 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:47.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:47.014378+0000) 2022-09-07T23:02:47.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:02:47.301+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:02:48.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:47 smithi059 ceph-mon[97450]: pgmap v2664: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:02:48.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:02:48.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:02:48.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:02:48.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:02:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:47 smithi055 ceph-mon[97033]: pgmap v2664: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:02:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:02:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:02:48.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:02:48.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:02:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:48.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:48.114783+0000) 2022-09-07T23:02:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:48.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:48.014621+0000) 2022-09-07T23:02:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:49.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:49.115012+0000) 2022-09-07T23:02:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:49.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:49.014831+0000) 2022-09-07T23:02:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:49 smithi055 ceph-mon[97033]: pgmap v2665: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:49 smithi059 ceph-mon[97450]: pgmap v2665: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:50.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:50.015023+0000) 2022-09-07T23:02:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:50.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:50.115172+0000) 2022-09-07T23:02:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:51.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:51.015192+0000) 2022-09-07T23:02:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:51.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:51.115378+0000) 2022-09-07T23:02:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:51 smithi059 ceph-mon[97450]: pgmap v2666: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:02:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:51 smithi055 ceph-mon[97033]: pgmap v2666: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:02:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:52.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:52.015440+0000) 2022-09-07T23:02:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:52.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:52.115671+0000) 2022-09-07T23:02:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:53.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:53.015738+0000) 2022-09-07T23:02:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:53.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:53.115922+0000) 2022-09-07T23:02:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:53 smithi059 ceph-mon[97450]: pgmap v2667: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:53 smithi055 ceph-mon[97033]: pgmap v2667: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:54.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:54.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:54.116043+0000) 2022-09-07T23:02:54.412 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:54.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:54.015977+0000) 2022-09-07T23:02:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:55.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:55.116389+0000) 2022-09-07T23:02:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:55.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:55.016151+0000) 2022-09-07T23:02:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:55 smithi059 ceph-mon[97450]: pgmap v2668: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:55 smithi055 ceph-mon[97033]: pgmap v2668: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:02:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:56.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:56.116530+0000) 2022-09-07T23:02:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:56.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:56.016303+0000) 2022-09-07T23:02:57.109 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:57.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:57.016460+0000) 2022-09-07T23:02:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:57.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:57.116720+0000) 2022-09-07T23:02:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:57 smithi059 ceph-mon[97450]: pgmap v2669: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:02:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:57 smithi055 ceph-mon[97033]: pgmap v2669: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:02:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:58.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:58.116943+0000) 2022-09-07T23:02:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:58.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:58.016729+0000) 2022-09-07T23:02:59.118 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:02:59.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:59.017005+0000) 2022-09-07T23:02:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:02:59.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:02:59.117198+0000) 2022-09-07T23:02:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:02:59 smithi055 ceph-mon[97033]: pgmap v2670: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:02:59 smithi059 ceph-mon[97450]: pgmap v2670: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:00.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:00.117450+0000) 2022-09-07T23:03:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:00.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:00.017173+0000) 2022-09-07T23:03:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:01.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:01.117736+0000) 2022-09-07T23:03:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:01.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:01.017432+0000) 2022-09-07T23:03:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:01 smithi059 ceph-mon[97450]: pgmap v2671: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:03:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:01 smithi055 ceph-mon[97033]: pgmap v2671: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:03:02.304 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:02.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:02.117970+0000) 2022-09-07T23:03:02.305 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:02.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:02.017632+0000) 2022-09-07T23:03:02.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:03:02.302+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:03:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:03.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:03.118152+0000) 2022-09-07T23:03:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:03.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:03.017795+0000) 2022-09-07T23:03:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:03 smithi059 ceph-mon[97450]: pgmap v2672: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:03 smithi055 ceph-mon[97033]: pgmap v2672: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:04.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:04.118382+0000) 2022-09-07T23:03:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:04.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:04.018059+0000) 2022-09-07T23:03:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:03:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:03:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:03:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:03:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:05.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:05.118630+0000) 2022-09-07T23:03:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:05.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:05.018372+0000) 2022-09-07T23:03:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:05 smithi059 ceph-mon[97450]: pgmap v2673: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:05 smithi055 ceph-mon[97033]: pgmap v2673: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:06.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:06.118905+0000) 2022-09-07T23:03:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:06.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:06.018632+0000) 2022-09-07T23:03:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:07.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:07.119071+0000) 2022-09-07T23:03:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:07.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:07.018904+0000) 2022-09-07T23:03:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:07 smithi059 ceph-mon[97450]: pgmap v2674: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:03:08.120 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:08.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:08.019198+0000) 2022-09-07T23:03:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:07 smithi055 ceph-mon[97033]: pgmap v2674: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:03:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:08.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:08.119325+0000) 2022-09-07T23:03:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:09.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:09.119545+0000) 2022-09-07T23:03:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:09.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:09.019445+0000) 2022-09-07T23:03:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:09 smithi059 ceph-mon[97450]: pgmap v2675: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:03:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:09 smithi059 ceph-mon[97450]: osdmap e259: 12 total, 11 up, 11 in 2022-09-07T23:03:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:09 smithi055 ceph-mon[97033]: pgmap v2675: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:03:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:09 smithi055 ceph-mon[97033]: osdmap e259: 12 total, 11 up, 11 in 2022-09-07T23:03:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:10.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:10.119696+0000) 2022-09-07T23:03:10.419 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:10.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:10.019624+0000) 2022-09-07T23:03:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:11.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:11.119927+0000) 2022-09-07T23:03:11.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:11.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:11.019825+0000) 2022-09-07T23:03:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:11 smithi059 ceph-mon[97450]: pgmap v2677: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:03:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:11 smithi055 ceph-mon[97033]: pgmap v2677: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:03:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:12.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:12.120213+0000) 2022-09-07T23:03:12.445 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:12.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:12.020038+0000) 2022-09-07T23:03:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:13.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:13.120467+0000) 2022-09-07T23:03:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:13.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:13.020238+0000) 2022-09-07T23:03:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:13 smithi059 ceph-mon[97450]: pgmap v2678: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:03:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:13 smithi059 ceph-mon[97450]: osdmap e260: 12 total, 11 up, 11 in 2022-09-07T23:03:14.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:13 smithi055 ceph-mon[97033]: pgmap v2678: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:03:14.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:13 smithi055 ceph-mon[97033]: osdmap e260: 12 total, 11 up, 11 in 2022-09-07T23:03:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:14.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:14.120748+0000) 2022-09-07T23:03:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:14.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:14.020457+0000) 2022-09-07T23:03:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:15.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:15.121000+0000) 2022-09-07T23:03:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:15.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:15.020752+0000) 2022-09-07T23:03:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:15 smithi059 ceph-mon[97450]: pgmap v2680: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:03:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:15 smithi055 ceph-mon[97033]: pgmap v2680: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:03:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:16.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:16.121208+0000) 2022-09-07T23:03:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:16.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:16.020902+0000) 2022-09-07T23:03:17.313 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:17.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:17.121426+0000) 2022-09-07T23:03:17.314 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:17.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:17.021140+0000) 2022-09-07T23:03:17.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:03:17.310+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:03:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:17 smithi059 ceph-mon[97450]: pgmap v2681: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:03:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:17 smithi055 ceph-mon[97033]: pgmap v2681: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:03:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:18.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:18.121613+0000) 2022-09-07T23:03:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:18.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:18.021263+0000) 2022-09-07T23:03:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:19.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:19.121813+0000) 2022-09-07T23:03:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:19.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:19.021488+0000) 2022-09-07T23:03:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:19 smithi055 ceph-mon[97033]: pgmap v2682: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:03:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:19 smithi059 ceph-mon[97450]: pgmap v2682: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:03:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:20.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:20.121970+0000) 2022-09-07T23:03:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:20.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:20.021643+0000) 2022-09-07T23:03:21.123 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:21.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:21.021797+0000) 2022-09-07T23:03:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:21.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:21.122125+0000) 2022-09-07T23:03:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:21 smithi059 ceph-mon[97450]: pgmap v2683: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:03:22.123 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:22.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:22.022017+0000) 2022-09-07T23:03:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:21 smithi055 ceph-mon[97033]: pgmap v2683: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:03:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:22.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:22.122359+0000) 2022-09-07T23:03:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:23.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:23.122528+0000) 2022-09-07T23:03:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:23.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:23.022188+0000) 2022-09-07T23:03:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:23 smithi059 ceph-mon[97450]: pgmap v2684: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:03:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:24.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:24.022385+0000) 2022-09-07T23:03:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:24.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:24.122720+0000) 2022-09-07T23:03:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:23 smithi055 ceph-mon[97033]: pgmap v2684: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:03:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:25.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:25.122887+0000) 2022-09-07T23:03:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:25.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:25.022667+0000) 2022-09-07T23:03:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:25 smithi059 ceph-mon[97450]: pgmap v2685: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:03:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:25 smithi055 ceph-mon[97033]: pgmap v2685: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:03:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:26.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:26.123041+0000) 2022-09-07T23:03:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:26.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:26.022842+0000) 2022-09-07T23:03:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:27.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:27.023081+0000) 2022-09-07T23:03:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:27.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:27.123255+0000) 2022-09-07T23:03:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:27 smithi059 ceph-mon[97450]: pgmap v2686: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:03:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:27 smithi055 ceph-mon[97033]: pgmap v2686: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:03:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:28.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:28.023285+0000) 2022-09-07T23:03:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:28.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:28.123472+0000) 2022-09-07T23:03:29.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:29.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:29.123682+0000) 2022-09-07T23:03:29.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:29.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:29.023503+0000) 2022-09-07T23:03:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:29 smithi055 ceph-mon[97033]: pgmap v2687: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:29 smithi059 ceph-mon[97450]: pgmap v2687: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:30.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:30.123822+0000) 2022-09-07T23:03:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:30.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:30.023710+0000) 2022-09-07T23:03:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:31.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:31.023911+0000) 2022-09-07T23:03:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:31.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:31.123953+0000) 2022-09-07T23:03:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:31 smithi059 ceph-mon[97450]: pgmap v2688: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:03:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:31 smithi055 ceph-mon[97033]: pgmap v2688: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:03:32.322 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:32.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:32.124169+0000) 2022-09-07T23:03:32.322 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:32.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:32.024105+0000) 2022-09-07T23:03:32.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:03:32.320+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:03:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:33.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:33.124338+0000) 2022-09-07T23:03:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:33.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:33.024402+0000) 2022-09-07T23:03:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:33 smithi059 ceph-mon[97450]: pgmap v2689: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:33 smithi055 ceph-mon[97033]: pgmap v2689: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:34.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:34.124544+0000) 2022-09-07T23:03:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:34.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:34.024602+0000) 2022-09-07T23:03:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:35.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:35.124804+0000) 2022-09-07T23:03:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:35.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:35.024851+0000) 2022-09-07T23:03:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:35 smithi059 ceph-mon[97450]: pgmap v2690: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:35 smithi055 ceph-mon[97033]: pgmap v2690: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:36.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:36.124943+0000) 2022-09-07T23:03:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:36.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:36.025089+0000) 2022-09-07T23:03:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:37.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:37.125167+0000) 2022-09-07T23:03:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:37.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:37.025316+0000) 2022-09-07T23:03:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:37 smithi059 ceph-mon[97450]: pgmap v2691: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:03:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:37 smithi055 ceph-mon[97033]: pgmap v2691: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:03:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:38.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:38.125392+0000) 2022-09-07T23:03:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:38.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:38.025567+0000) 2022-09-07T23:03:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:39.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:39.125660+0000) 2022-09-07T23:03:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:39.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:39.025795+0000) 2022-09-07T23:03:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:39 smithi055 ceph-mon[97033]: pgmap v2692: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:39 smithi059 ceph-mon[97450]: pgmap v2692: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:40.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:40.125890+0000) 2022-09-07T23:03:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:40.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:40.026046+0000) 2022-09-07T23:03:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:41.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:41.026244+0000) 2022-09-07T23:03:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:41.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:41.126109+0000) 2022-09-07T23:03:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:41 smithi059 ceph-mon[97450]: pgmap v2693: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:03:42.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:41 smithi055 ceph-mon[97033]: pgmap v2693: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:03:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:42.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:42.026460+0000) 2022-09-07T23:03:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:42.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:42.126294+0000) 2022-09-07T23:03:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:43.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:43.126591+0000) 2022-09-07T23:03:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:43.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:43.026719+0000) 2022-09-07T23:03:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:43 smithi059 ceph-mon[97450]: pgmap v2694: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:43 smithi055 ceph-mon[97033]: pgmap v2694: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:44.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:44.126867+0000) 2022-09-07T23:03:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:44.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:44.026944+0000) 2022-09-07T23:03:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:45.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:45.127042+0000) 2022-09-07T23:03:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:45.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:45.027127+0000) 2022-09-07T23:03:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:45 smithi059 ceph-mon[97450]: pgmap v2695: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:45 smithi055 ceph-mon[97033]: pgmap v2695: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:46.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:46.127181+0000) 2022-09-07T23:03:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:46.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:46.027270+0000) 2022-09-07T23:03:47.336 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:47.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:47.127386+0000) 2022-09-07T23:03:47.337 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:47.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:47.027416+0000) 2022-09-07T23:03:47.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:03:47.334+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:03:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:47 smithi059 ceph-mon[97450]: pgmap v2696: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:03:48.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:03:48.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:03:48.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:03:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:47 smithi055 ceph-mon[97033]: pgmap v2696: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:03:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:03:48.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:03:48.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:03:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:48.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:48.127558+0000) 2022-09-07T23:03:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:48.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:48.027565+0000) 2022-09-07T23:03:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:03:49.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:48 smithi055 ceph-mon[97033]: pgmap v2697: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:03:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:48 smithi059 ceph-mon[97450]: pgmap v2697: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:49.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:49.127791+0000) 2022-09-07T23:03:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:49.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:49.027713+0000) 2022-09-07T23:03:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:50.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:50.027924+0000) 2022-09-07T23:03:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:50.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:50.128010+0000) 2022-09-07T23:03:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:51.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:51.028150+0000) 2022-09-07T23:03:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:51.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:51.128233+0000) 2022-09-07T23:03:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:51 smithi059 ceph-mon[97450]: pgmap v2698: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:03:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:51 smithi055 ceph-mon[97033]: pgmap v2698: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:03:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:52.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:52.028399+0000) 2022-09-07T23:03:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:52.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:52.128453+0000) 2022-09-07T23:03:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:53.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:53.128684+0000) 2022-09-07T23:03:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:53.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:53.028632+0000) 2022-09-07T23:03:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:53 smithi059 ceph-mon[97450]: pgmap v2699: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:53 smithi055 ceph-mon[97033]: pgmap v2699: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:54.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:54.128929+0000) 2022-09-07T23:03:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:54.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:54.028794+0000) 2022-09-07T23:03:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:55.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:55.129130+0000) 2022-09-07T23:03:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:55.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:55.028998+0000) 2022-09-07T23:03:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:55 smithi059 ceph-mon[97450]: pgmap v2700: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:55 smithi055 ceph-mon[97033]: pgmap v2700: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:03:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:56.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:56.129339+0000) 2022-09-07T23:03:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:56.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:56.029231+0000) 2022-09-07T23:03:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:57.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:57.129509+0000) 2022-09-07T23:03:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:57.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:57.029421+0000) 2022-09-07T23:03:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:57 smithi059 ceph-mon[97450]: pgmap v2701: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:03:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:57 smithi055 ceph-mon[97033]: pgmap v2701: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:03:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:58.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:58.129693+0000) 2022-09-07T23:03:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:58.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:58.029594+0000) 2022-09-07T23:03:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:03:59.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:59.129838+0000) 2022-09-07T23:03:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:03:59.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:03:59.029790+0000) 2022-09-07T23:03:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:03:59 smithi055 ceph-mon[97033]: pgmap v2702: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:03:59 smithi059 ceph-mon[97450]: pgmap v2702: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:00.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:00.130043+0000) 2022-09-07T23:04:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:00.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:00.030020+0000) 2022-09-07T23:04:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:01.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:01.130170+0000) 2022-09-07T23:04:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:01.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:01.030175+0000) 2022-09-07T23:04:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:01 smithi059 ceph-mon[97450]: pgmap v2703: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:04:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:01 smithi055 ceph-mon[97033]: pgmap v2703: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:04:02.354 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:02.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:02.130300+0000) 2022-09-07T23:04:02.355 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:02.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:02.030326+0000) 2022-09-07T23:04:02.635 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:04:02.349+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:04:03.352 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:03.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:03.130434+0000) 2022-09-07T23:04:03.353 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:03.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:03.030465+0000) 2022-09-07T23:04:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:03 smithi059 ceph-mon[97450]: pgmap v2704: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:03 smithi055 ceph-mon[97033]: pgmap v2704: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:04.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:04.030728+0000) 2022-09-07T23:04:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:04.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:04.130693+0000) 2022-09-07T23:04:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:04:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:04:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:04:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:04:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:05.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:05.030930+0000) 2022-09-07T23:04:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:05.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:05.130899+0000) 2022-09-07T23:04:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:05 smithi059 ceph-mon[97450]: pgmap v2705: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:05 smithi055 ceph-mon[97033]: pgmap v2705: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:06.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:06.131134+0000) 2022-09-07T23:04:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:06.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:06.031217+0000) 2022-09-07T23:04:07.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:07.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:07.131368+0000) 2022-09-07T23:04:07.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:07.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:07.031430+0000) 2022-09-07T23:04:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:07 smithi059 ceph-mon[97450]: pgmap v2706: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:04:08.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:07 smithi055 ceph-mon[97033]: pgmap v2706: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:04:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:08.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:08.031664+0000) 2022-09-07T23:04:08.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:08.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:08.131558+0000) 2022-09-07T23:04:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:09.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:09.131773+0000) 2022-09-07T23:04:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:09.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:09.031816+0000) 2022-09-07T23:04:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:09 smithi059 ceph-mon[97450]: pgmap v2707: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:04:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:09 smithi059 ceph-mon[97450]: osdmap e261: 12 total, 11 up, 11 in 2022-09-07T23:04:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:09 smithi055 ceph-mon[97033]: pgmap v2707: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:04:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:09 smithi055 ceph-mon[97033]: osdmap e261: 12 total, 11 up, 11 in 2022-09-07T23:04:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:10.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:10.131950+0000) 2022-09-07T23:04:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:10.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:10.032081+0000) 2022-09-07T23:04:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:11.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:11.032271+0000) 2022-09-07T23:04:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:11.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:11.132171+0000) 2022-09-07T23:04:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:11 smithi055 ceph-mon[97033]: pgmap v2709: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:04:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:11 smithi059 ceph-mon[97450]: pgmap v2709: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:04:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:12.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:12.132391+0000) 2022-09-07T23:04:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:12.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:12.032481+0000) 2022-09-07T23:04:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:13.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:13.132602+0000) 2022-09-07T23:04:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:13.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:13.032757+0000) 2022-09-07T23:04:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:13 smithi055 ceph-mon[97033]: pgmap v2710: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:04:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:13 smithi055 ceph-mon[97033]: osdmap e262: 12 total, 11 up, 11 in 2022-09-07T23:04:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:13 smithi059 ceph-mon[97450]: pgmap v2710: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:04:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:13 smithi059 ceph-mon[97450]: osdmap e262: 12 total, 11 up, 11 in 2022-09-07T23:04:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:14.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:14.032951+0000) 2022-09-07T23:04:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:14.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:14.132883+0000) 2022-09-07T23:04:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:15.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:15.033199+0000) 2022-09-07T23:04:15.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:15.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:15.133129+0000) 2022-09-07T23:04:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:15 smithi055 ceph-mon[97033]: pgmap v2712: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:04:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:15 smithi059 ceph-mon[97450]: pgmap v2712: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:04:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:16.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:16.033452+0000) 2022-09-07T23:04:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:16.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:16.133338+0000) 2022-09-07T23:04:17.367 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:17.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:17.033684+0000) 2022-09-07T23:04:17.369 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:17.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:17.133548+0000) 2022-09-07T23:04:17.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:04:17.364+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:04:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:17 smithi055 ceph-mon[97033]: pgmap v2713: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:04:18.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:17 smithi059 ceph-mon[97450]: pgmap v2713: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:04:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:18.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:18.133786+0000) 2022-09-07T23:04:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:18.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:18.033920+0000) 2022-09-07T23:04:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:19.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:19.034210+0000) 2022-09-07T23:04:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:19.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:19.133992+0000) 2022-09-07T23:04:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:19 smithi055 ceph-mon[97033]: pgmap v2714: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:04:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:19 smithi059 ceph-mon[97450]: pgmap v2714: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:04:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:20.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:20.034491+0000) 2022-09-07T23:04:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:20.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:20.134203+0000) 2022-09-07T23:04:21.413 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:21.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:21.034766+0000) 2022-09-07T23:04:21.414 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:21.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:21.134394+0000) 2022-09-07T23:04:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:21 smithi059 ceph-mon[97450]: pgmap v2715: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:04:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:21 smithi055 ceph-mon[97033]: pgmap v2715: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:04:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:22.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:22.134626+0000) 2022-09-07T23:04:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:22.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:22.034993+0000) 2022-09-07T23:04:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:23.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:23.134943+0000) 2022-09-07T23:04:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:23.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:23.035272+0000) 2022-09-07T23:04:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:23 smithi059 ceph-mon[97450]: pgmap v2716: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:04:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:23 smithi055 ceph-mon[97033]: pgmap v2716: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:04:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:24.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:24.135202+0000) 2022-09-07T23:04:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:24.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:24.035552+0000) 2022-09-07T23:04:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:25.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:25.135495+0000) 2022-09-07T23:04:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:25.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:25.035708+0000) 2022-09-07T23:04:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:25 smithi059 ceph-mon[97450]: pgmap v2717: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:04:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:25 smithi055 ceph-mon[97033]: pgmap v2717: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:04:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:26.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:26.135760+0000) 2022-09-07T23:04:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:26.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:26.035957+0000) 2022-09-07T23:04:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:27.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:27.136037+0000) 2022-09-07T23:04:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:27.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:27.036203+0000) 2022-09-07T23:04:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:27 smithi059 ceph-mon[97450]: pgmap v2718: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:04:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:27 smithi055 ceph-mon[97033]: pgmap v2718: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:04:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:28.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:28.136298+0000) 2022-09-07T23:04:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:28.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:28.036403+0000) 2022-09-07T23:04:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:29.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:29.136470+0000) 2022-09-07T23:04:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:29.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:29.036682+0000) 2022-09-07T23:04:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:29 smithi055 ceph-mon[97033]: pgmap v2719: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:30.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:29 smithi059 ceph-mon[97450]: pgmap v2719: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:30.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:30.136685+0000) 2022-09-07T23:04:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:30.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:30.036915+0000) 2022-09-07T23:04:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:31.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:31.136829+0000) 2022-09-07T23:04:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:31.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:31.037159+0000) 2022-09-07T23:04:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:31 smithi059 ceph-mon[97450]: pgmap v2720: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:04:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:31 smithi055 ceph-mon[97033]: pgmap v2720: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:04:32.369 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:32.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:32.137080+0000) 2022-09-07T23:04:32.370 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:32.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:32.037431+0000) 2022-09-07T23:04:32.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:04:32.367+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:04:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:33.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:33.137313+0000) 2022-09-07T23:04:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:33.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:33.037746+0000) 2022-09-07T23:04:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:33 smithi059 ceph-mon[97450]: pgmap v2721: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:33 smithi055 ceph-mon[97033]: pgmap v2721: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:34.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:34.137442+0000) 2022-09-07T23:04:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:34.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:34.038040+0000) 2022-09-07T23:04:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:35.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:35.137721+0000) 2022-09-07T23:04:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:35.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:35.038251+0000) 2022-09-07T23:04:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:35 smithi059 ceph-mon[97450]: pgmap v2722: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:35 smithi055 ceph-mon[97033]: pgmap v2722: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:36.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:36.038497+0000) 2022-09-07T23:04:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:36.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:36.138033+0000) 2022-09-07T23:04:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:37.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:37.138288+0000) 2022-09-07T23:04:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:37.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:37.038759+0000) 2022-09-07T23:04:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:37 smithi055 ceph-mon[97033]: pgmap v2723: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:04:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:37 smithi059 ceph-mon[97450]: pgmap v2723: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:04:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:38.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:38.038928+0000) 2022-09-07T23:04:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:38.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:38.138525+0000) 2022-09-07T23:04:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:39.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:39.039112+0000) 2022-09-07T23:04:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:39.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:39.138822+0000) 2022-09-07T23:04:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:39 smithi055 ceph-mon[97033]: pgmap v2724: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:39 smithi059 ceph-mon[97450]: pgmap v2724: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:40.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:40.039361+0000) 2022-09-07T23:04:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:40.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:40.139040+0000) 2022-09-07T23:04:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:41.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:41.139161+0000) 2022-09-07T23:04:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:41.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:41.039664+0000) 2022-09-07T23:04:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:41 smithi059 ceph-mon[97450]: pgmap v2725: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:04:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:41 smithi055 ceph-mon[97033]: pgmap v2725: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:04:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:42.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:42.139335+0000) 2022-09-07T23:04:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:42.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:42.039879+0000) 2022-09-07T23:04:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:43.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:43.139472+0000) 2022-09-07T23:04:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:43.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:43.040108+0000) 2022-09-07T23:04:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:43 smithi059 ceph-mon[97450]: pgmap v2726: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:43 smithi055 ceph-mon[97033]: pgmap v2726: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:44.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:44.139714+0000) 2022-09-07T23:04:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:44.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:44.040383+0000) 2022-09-07T23:04:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:45.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:45.139937+0000) 2022-09-07T23:04:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:45.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:45.040681+0000) 2022-09-07T23:04:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:45 smithi059 ceph-mon[97450]: pgmap v2727: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:45 smithi055 ceph-mon[97033]: pgmap v2727: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:46.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:46.140184+0000) 2022-09-07T23:04:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:46.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:46.040828+0000) 2022-09-07T23:04:47.373 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:47.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:47.140372+0000) 2022-09-07T23:04:47.374 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:47.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:47.041058+0000) 2022-09-07T23:04:47.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:04:47.371+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:04:48.070 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:47 smithi059 ceph-mon[97450]: pgmap v2728: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:04:48.071 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:47 smithi055 ceph-mon[97033]: pgmap v2728: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:04:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:48.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:48.140591+0000) 2022-09-07T23:04:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:48.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:48.041213+0000) 2022-09-07T23:04:48.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:04:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:04:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:04:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:04:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:04:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:04:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:04:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:04:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:49.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:49.140855+0000) 2022-09-07T23:04:49.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:49.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:49.041423+0000) 2022-09-07T23:04:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:49 smithi055 ceph-mon[97033]: pgmap v2729: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:49 smithi059 ceph-mon[97450]: pgmap v2729: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:50.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:50.141088+0000) 2022-09-07T23:04:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:50.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:50.041659+0000) 2022-09-07T23:04:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:51.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:51.141304+0000) 2022-09-07T23:04:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:51.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:51.041924+0000) 2022-09-07T23:04:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:51 smithi055 ceph-mon[97033]: pgmap v2730: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:04:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:51 smithi059 ceph-mon[97450]: pgmap v2730: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:04:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:52.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:52.141554+0000) 2022-09-07T23:04:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:52.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:52.042149+0000) 2022-09-07T23:04:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:53.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:53.141795+0000) 2022-09-07T23:04:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:53.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:53.042393+0000) 2022-09-07T23:04:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:53 smithi055 ceph-mon[97033]: pgmap v2731: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:53 smithi059 ceph-mon[97450]: pgmap v2731: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:54.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:54.141959+0000) 2022-09-07T23:04:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:54.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:54.042631+0000) 2022-09-07T23:04:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:55.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:55.142167+0000) 2022-09-07T23:04:55.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:55.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:55.042844+0000) 2022-09-07T23:04:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:55 smithi055 ceph-mon[97033]: pgmap v2732: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:55 smithi059 ceph-mon[97450]: pgmap v2732: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:04:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:56.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:56.142406+0000) 2022-09-07T23:04:56.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:56.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:56.043055+0000) 2022-09-07T23:04:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:57.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:57.142673+0000) 2022-09-07T23:04:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:57.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:57.043304+0000) 2022-09-07T23:04:58.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:57 smithi055 ceph-mon[97033]: pgmap v2733: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:04:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:57 smithi059 ceph-mon[97450]: pgmap v2733: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:04:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:58.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:58.142847+0000) 2022-09-07T23:04:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:58.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:58.043545+0000) 2022-09-07T23:04:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:04:59.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:59.143014+0000) 2022-09-07T23:04:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:04:59.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:04:59.043775+0000) 2022-09-07T23:04:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:04:59 smithi055 ceph-mon[97033]: pgmap v2734: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:04:59 smithi059 ceph-mon[97450]: pgmap v2734: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:00.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:00.143163+0000) 2022-09-07T23:05:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:00.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:00.044013+0000) 2022-09-07T23:05:01.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:01.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:01.143342+0000) 2022-09-07T23:05:01.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:01.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:01.044246+0000) 2022-09-07T23:05:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:01 smithi059 ceph-mon[97450]: pgmap v2735: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:01 smithi055 ceph-mon[97033]: pgmap v2735: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:02.382 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:02.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:02.143535+0000) 2022-09-07T23:05:02.383 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:02.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:02.044464+0000) 2022-09-07T23:05:02.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:05:02.377+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:05:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:03.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:03.143786+0000) 2022-09-07T23:05:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:03.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:03.044677+0000) 2022-09-07T23:05:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:03 smithi055 ceph-mon[97033]: pgmap v2736: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:03 smithi059 ceph-mon[97450]: pgmap v2736: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:04.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:04.044911+0000) 2022-09-07T23:05:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:04.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:04.143974+0000) 2022-09-07T23:05:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:05:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:05:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:05:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:05:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:05.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:05.144251+0000) 2022-09-07T23:05:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:05.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:05.045128+0000) 2022-09-07T23:05:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:05 smithi055 ceph-mon[97033]: pgmap v2737: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:05 smithi059 ceph-mon[97450]: pgmap v2737: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:06.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:06.045270+0000) 2022-09-07T23:05:06.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:06.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:06.144397+0000) 2022-09-07T23:05:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:07.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:07.144612+0000) 2022-09-07T23:05:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:07.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:07.045434+0000) 2022-09-07T23:05:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:07 smithi055 ceph-mon[97033]: pgmap v2738: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-07T23:05:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:07 smithi059 ceph-mon[97450]: pgmap v2738: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-07T23:05:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:08.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:08.144886+0000) 2022-09-07T23:05:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:08.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:08.045703+0000) 2022-09-07T23:05:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:09.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:09.145073+0000) 2022-09-07T23:05:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:09.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:09.045929+0000) 2022-09-07T23:05:09.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:09 smithi055 ceph-mon[97033]: pgmap v2739: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-07T23:05:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:09 smithi059 ceph-mon[97450]: pgmap v2739: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-07T23:05:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:10.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:10.046142+0000) 2022-09-07T23:05:10.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:10.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:10.145196+0000) 2022-09-07T23:05:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:11.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:11.046378+0000) 2022-09-07T23:05:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:11.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:11.145440+0000) 2022-09-07T23:05:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:11 smithi055 ceph-mon[97033]: pgmap v2740: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-07T23:05:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:11 smithi059 ceph-mon[97450]: pgmap v2740: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-07T23:05:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:12.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:12.145694+0000) 2022-09-07T23:05:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:12.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:12.046601+0000) 2022-09-07T23:05:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:13.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:13.046843+0000) 2022-09-07T23:05:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:13.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:13.145944+0000) 2022-09-07T23:05:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:13 smithi055 ceph-mon[97033]: pgmap v2741: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-07T23:05:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:13 smithi059 ceph-mon[97450]: pgmap v2741: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-07T23:05:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:14.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:14.146201+0000) 2022-09-07T23:05:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:14.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:14.047023+0000) 2022-09-07T23:05:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:15.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:15.146465+0000) 2022-09-07T23:05:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:15.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:15.047246+0000) 2022-09-07T23:05:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:15 smithi055 ceph-mon[97033]: pgmap v2742: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-07T23:05:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:15 smithi059 ceph-mon[97450]: pgmap v2742: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-07T23:05:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:16.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:16.146769+0000) 2022-09-07T23:05:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:16.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:16.047483+0000) 2022-09-07T23:05:17.387 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:17.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:17.147033+0000) 2022-09-07T23:05:17.387 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:17.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:17.047741+0000) 2022-09-07T23:05:17.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:05:17.385+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:05:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:17 smithi055 ceph-mon[97033]: pgmap v2743: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-07T23:05:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:17 smithi059 ceph-mon[97450]: pgmap v2743: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-07T23:05:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:18.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:18.147346+0000) 2022-09-07T23:05:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:18.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:18.047970+0000) 2022-09-07T23:05:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:19.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:19.147684+0000) 2022-09-07T23:05:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:19.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:19.048239+0000) 2022-09-07T23:05:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:19 smithi055 ceph-mon[97033]: pgmap v2744: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T23:05:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:19 smithi059 ceph-mon[97450]: pgmap v2744: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T23:05:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:20.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:20.147820+0000) 2022-09-07T23:05:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:20.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:20.048489+0000) 2022-09-07T23:05:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:21.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:21.147984+0000) 2022-09-07T23:05:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:21.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:21.048691+0000) 2022-09-07T23:05:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:21 smithi055 ceph-mon[97033]: pgmap v2745: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:05:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:21 smithi059 ceph-mon[97450]: pgmap v2745: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:05:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:22.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:22.148135+0000) 2022-09-07T23:05:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:22.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:22.048840+0000) 2022-09-07T23:05:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:23.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:23.148392+0000) 2022-09-07T23:05:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:23.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:23.049025+0000) 2022-09-07T23:05:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:23 smithi055 ceph-mon[97033]: pgmap v2746: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:23 smithi059 ceph-mon[97450]: pgmap v2746: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:24.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:24.148690+0000) 2022-09-07T23:05:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:24.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:24.049289+0000) 2022-09-07T23:05:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:25.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:25.148980+0000) 2022-09-07T23:05:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:25.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:25.049544+0000) 2022-09-07T23:05:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:25 smithi055 ceph-mon[97033]: pgmap v2747: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:25 smithi059 ceph-mon[97450]: pgmap v2747: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:26.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:26.149241+0000) 2022-09-07T23:05:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:26.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:26.049735+0000) 2022-09-07T23:05:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:27.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:27.149456+0000) 2022-09-07T23:05:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:27.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:27.049865+0000) 2022-09-07T23:05:28.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:27 smithi055 ceph-mon[97033]: pgmap v2748: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:28.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:27 smithi059 ceph-mon[97450]: pgmap v2748: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:28.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:28.149715+0000) 2022-09-07T23:05:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:28.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:28.050057+0000) 2022-09-07T23:05:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:29.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:29.149869+0000) 2022-09-07T23:05:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:29.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:29.050302+0000) 2022-09-07T23:05:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:29 smithi055 ceph-mon[97033]: pgmap v2749: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:29 smithi059 ceph-mon[97450]: pgmap v2749: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:30.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:30.150088+0000) 2022-09-07T23:05:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:30.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:30.050471+0000) 2022-09-07T23:05:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:31.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:31.050621+0000) 2022-09-07T23:05:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:31.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:31.150320+0000) 2022-09-07T23:05:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:31 smithi055 ceph-mon[97033]: pgmap v2750: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:31 smithi059 ceph-mon[97450]: pgmap v2750: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:32.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:32.150581+0000) 2022-09-07T23:05:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:32.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:32.050822+0000) 2022-09-07T23:05:32.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:05:32.399+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:05:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:33.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:33.150845+0000) 2022-09-07T23:05:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:33.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:33.051002+0000) 2022-09-07T23:05:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:33 smithi055 ceph-mon[97033]: pgmap v2751: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:33 smithi059 ceph-mon[97450]: pgmap v2751: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:34.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:34.151023+0000) 2022-09-07T23:05:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:34.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:34.051218+0000) 2022-09-07T23:05:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:35.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:35.151270+0000) 2022-09-07T23:05:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:35.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:35.051369+0000) 2022-09-07T23:05:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:35 smithi055 ceph-mon[97033]: pgmap v2752: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:35 smithi059 ceph-mon[97450]: pgmap v2752: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:36.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:36.151487+0000) 2022-09-07T23:05:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:36.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:36.051589+0000) 2022-09-07T23:05:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:37.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:37.151735+0000) 2022-09-07T23:05:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:37.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:37.051819+0000) 2022-09-07T23:05:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:37 smithi055 ceph-mon[97033]: pgmap v2753: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:37 smithi059 ceph-mon[97450]: pgmap v2753: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:38.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:38.151966+0000) 2022-09-07T23:05:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:38.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:38.052011+0000) 2022-09-07T23:05:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:39.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:39.152186+0000) 2022-09-07T23:05:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:39.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:39.052262+0000) 2022-09-07T23:05:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:39 smithi055 ceph-mon[97033]: pgmap v2754: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:39 smithi059 ceph-mon[97450]: pgmap v2754: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:40.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:40.152385+0000) 2022-09-07T23:05:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:40.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:40.052441+0000) 2022-09-07T23:05:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:41.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:41.052582+0000) 2022-09-07T23:05:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:41.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:41.152592+0000) 2022-09-07T23:05:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:41 smithi055 ceph-mon[97033]: pgmap v2755: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:41 smithi059 ceph-mon[97450]: pgmap v2755: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:42.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:42.052751+0000) 2022-09-07T23:05:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:42.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:42.152810+0000) 2022-09-07T23:05:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:43.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:43.153045+0000) 2022-09-07T23:05:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:43.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:43.052972+0000) 2022-09-07T23:05:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:43 smithi055 ceph-mon[97033]: pgmap v2756: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:43 smithi059 ceph-mon[97450]: pgmap v2756: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:44.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:44.153264+0000) 2022-09-07T23:05:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:44.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:44.053134+0000) 2022-09-07T23:05:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:45.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:45.153432+0000) 2022-09-07T23:05:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:45.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:45.053278+0000) 2022-09-07T23:05:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:45 smithi055 ceph-mon[97033]: pgmap v2757: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:45 smithi059 ceph-mon[97450]: pgmap v2757: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:46.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:46.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:46.053470+0000) 2022-09-07T23:05:46.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:46.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:46.153683+0000) 2022-09-07T23:05:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:47.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:47.153941+0000) 2022-09-07T23:05:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:47.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:47.053706+0000) 2022-09-07T23:05:47.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:05:47.413+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:05:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:47 smithi055 ceph-mon[97033]: pgmap v2758: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:47 smithi059 ceph-mon[97450]: pgmap v2758: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:48.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:48.154138+0000) 2022-09-07T23:05:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:48.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:48.053935+0000) 2022-09-07T23:05:48.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:05:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:05:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:05:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:05:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:05:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:05:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:49.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:49.154398+0000) 2022-09-07T23:05:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:49.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:49.054223+0000) 2022-09-07T23:05:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:49 smithi055 ceph-mon[97033]: pgmap v2759: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:05:50.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:05:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:49 smithi059 ceph-mon[97450]: pgmap v2759: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:05:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:05:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:50.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:50.154587+0000) 2022-09-07T23:05:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:50.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:50.054393+0000) 2022-09-07T23:05:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:51.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:51.154762+0000) 2022-09-07T23:05:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:51.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:51.054631+0000) 2022-09-07T23:05:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:51 smithi055 ceph-mon[97033]: pgmap v2760: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:05:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:05:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:51 smithi059 ceph-mon[97450]: pgmap v2760: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:05:52.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:05:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:52.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:52.154939+0000) 2022-09-07T23:05:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:52.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:52.054813+0000) 2022-09-07T23:05:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:05:53.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:05:53.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T23:05:53.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T23:05:53.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T23:05:53.239 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T23:05:53.240 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:05:53.240 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:52 smithi055 ceph-mon[97033]: pgmap v2761: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:05:53.328 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:05:53.329 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.329 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.330 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.331 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.331 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T23:05:53.332 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T23:05:53.332 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.332 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.333 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.333 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:05:53.335 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T23:05:53.335 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T23:05:53.336 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:05:53.336 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:52 smithi059 ceph-mon[97450]: pgmap v2761: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:53.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:53.155129+0000) 2022-09-07T23:05:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:53.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:53.055021+0000) 2022-09-07T23:05:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:54.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:54.155286+0000) 2022-09-07T23:05:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:54.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:54.055249+0000) 2022-09-07T23:05:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:55.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:55.155526+0000) 2022-09-07T23:05:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:55.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:55.055512+0000) 2022-09-07T23:05:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:55 smithi055 ceph-mon[97033]: pgmap v2762: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:55 smithi059 ceph-mon[97450]: pgmap v2762: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:05:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:56.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:56.155766+0000) 2022-09-07T23:05:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:56.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:56.055750+0000) 2022-09-07T23:05:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:57.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:57.155905+0000) 2022-09-07T23:05:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:57.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:57.055967+0000) 2022-09-07T23:05:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:57 smithi055 ceph-mon[97033]: pgmap v2763: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:57 smithi059 ceph-mon[97450]: pgmap v2763: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:05:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:58.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:58.156117+0000) 2022-09-07T23:05:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:58.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:58.056102+0000) 2022-09-07T23:05:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:05:59.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:59.156307+0000) 2022-09-07T23:05:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:05:59.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:05:59.056311+0000) 2022-09-07T23:05:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:05:59 smithi055 ceph-mon[97033]: pgmap v2764: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:05:59 smithi059 ceph-mon[97450]: pgmap v2764: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:00.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:00.156498+0000) 2022-09-07T23:06:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:00.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:00.056561+0000) 2022-09-07T23:06:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:01.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:01.156723+0000) 2022-09-07T23:06:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:01.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:01.056800+0000) 2022-09-07T23:06:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:01 smithi055 ceph-mon[97033]: pgmap v2765: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:06:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:01 smithi059 ceph-mon[97450]: pgmap v2765: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:06:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:02.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:02.156885+0000) 2022-09-07T23:06:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:02.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:02.057036+0000) 2022-09-07T23:06:02.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:06:02.429+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:06:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:03.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:03.157121+0000) 2022-09-07T23:06:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:03.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:03.057278+0000) 2022-09-07T23:06:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:03 smithi055 ceph-mon[97033]: pgmap v2766: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:03 smithi059 ceph-mon[97450]: pgmap v2766: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:04.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:04.157372+0000) 2022-09-07T23:06:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:04.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:04.057551+0000) 2022-09-07T23:06:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:06:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:06:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:06:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:06:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:05.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:05.057795+0000) 2022-09-07T23:06:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:05.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:05.157566+0000) 2022-09-07T23:06:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:05 smithi055 ceph-mon[97033]: pgmap v2767: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:05 smithi059 ceph-mon[97450]: pgmap v2767: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:06.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:06.157743+0000) 2022-09-07T23:06:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:06.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:06.057925+0000) 2022-09-07T23:06:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:07.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:07.058068+0000) 2022-09-07T23:06:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:07.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:07.157985+0000) 2022-09-07T23:06:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:07 smithi055 ceph-mon[97033]: pgmap v2768: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:06:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:07 smithi059 ceph-mon[97450]: pgmap v2768: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:06:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:08.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:08.058253+0000) 2022-09-07T23:06:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:08.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:08.158186+0000) 2022-09-07T23:06:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:09.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:09.158358+0000) 2022-09-07T23:06:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:09.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:09.058472+0000) 2022-09-07T23:06:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:09 smithi055 ceph-mon[97033]: pgmap v2769: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:06:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:09 smithi055 ceph-mon[97033]: osdmap e263: 12 total, 11 up, 11 in 2022-09-07T23:06:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:09 smithi059 ceph-mon[97450]: pgmap v2769: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:06:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:09 smithi059 ceph-mon[97450]: osdmap e263: 12 total, 11 up, 11 in 2022-09-07T23:06:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:10.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:10.158626+0000) 2022-09-07T23:06:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:10.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:10.058721+0000) 2022-09-07T23:06:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:11.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:11.158852+0000) 2022-09-07T23:06:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:11.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:11.058895+0000) 2022-09-07T23:06:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:11 smithi055 ceph-mon[97033]: pgmap v2771: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:06:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:11 smithi059 ceph-mon[97450]: pgmap v2771: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:06:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:12.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:12.159106+0000) 2022-09-07T23:06:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:12.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:12.059119+0000) 2022-09-07T23:06:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:13.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:13.159343+0000) 2022-09-07T23:06:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:13.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:13.059369+0000) 2022-09-07T23:06:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:13 smithi055 ceph-mon[97033]: pgmap v2772: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:06:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:13 smithi055 ceph-mon[97033]: osdmap e264: 12 total, 11 up, 11 in 2022-09-07T23:06:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:13 smithi059 ceph-mon[97450]: pgmap v2772: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:06:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:13 smithi059 ceph-mon[97450]: osdmap e264: 12 total, 11 up, 11 in 2022-09-07T23:06:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:14.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:14.159528+0000) 2022-09-07T23:06:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:14.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:14.059510+0000) 2022-09-07T23:06:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:15.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:15.159760+0000) 2022-09-07T23:06:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:15.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:15.059705+0000) 2022-09-07T23:06:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:15 smithi055 ceph-mon[97033]: pgmap v2774: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:06:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:15 smithi059 ceph-mon[97450]: pgmap v2774: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:06:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:16.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:16.160031+0000) 2022-09-07T23:06:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:16.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:16.059939+0000) 2022-09-07T23:06:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:17.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:17.160238+0000) 2022-09-07T23:06:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:17.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:17.060142+0000) 2022-09-07T23:06:17.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:06:17.442+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:06:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:17 smithi055 ceph-mon[97033]: pgmap v2775: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:06:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:17 smithi059 ceph-mon[97450]: pgmap v2775: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:06:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:18.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:18.160442+0000) 2022-09-07T23:06:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:18.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:18.060370+0000) 2022-09-07T23:06:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:19.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:19.060574+0000) 2022-09-07T23:06:19.462 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:19.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:19.160718+0000) 2022-09-07T23:06:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:19 smithi055 ceph-mon[97033]: pgmap v2776: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:06:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:19 smithi059 ceph-mon[97450]: pgmap v2776: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:06:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:20.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:20.160900+0000) 2022-09-07T23:06:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:20.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:20.060804+0000) 2022-09-07T23:06:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:21.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:21.161086+0000) 2022-09-07T23:06:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:21.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:21.061032+0000) 2022-09-07T23:06:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:21 smithi055 ceph-mon[97033]: pgmap v2777: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:06:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:21 smithi059 ceph-mon[97450]: pgmap v2777: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:06:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:22.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:22.161289+0000) 2022-09-07T23:06:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:22.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:22.061270+0000) 2022-09-07T23:06:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:23.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:23.161440+0000) 2022-09-07T23:06:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:23.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:23.061496+0000) 2022-09-07T23:06:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:23 smithi055 ceph-mon[97033]: pgmap v2778: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:06:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:23 smithi059 ceph-mon[97450]: pgmap v2778: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:06:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:24.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:24.161654+0000) 2022-09-07T23:06:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:24.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:24.061734+0000) 2022-09-07T23:06:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:25.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:25.161829+0000) 2022-09-07T23:06:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:25.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:25.061907+0000) 2022-09-07T23:06:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:25 smithi055 ceph-mon[97033]: pgmap v2779: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:06:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:25 smithi059 ceph-mon[97450]: pgmap v2779: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:06:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:26.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:26.062177+0000) 2022-09-07T23:06:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:26.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:26.162017+0000) 2022-09-07T23:06:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:27.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:27.162308+0000) 2022-09-07T23:06:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:27.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:27.062349+0000) 2022-09-07T23:06:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:27 smithi055 ceph-mon[97033]: pgmap v2780: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:06:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:27 smithi059 ceph-mon[97450]: pgmap v2780: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:06:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:28.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:28.062532+0000) 2022-09-07T23:06:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:28.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:28.162497+0000) 2022-09-07T23:06:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:29.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:29.162729+0000) 2022-09-07T23:06:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:29.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:29.062783+0000) 2022-09-07T23:06:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:29 smithi055 ceph-mon[97033]: pgmap v2781: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:29 smithi059 ceph-mon[97450]: pgmap v2781: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:30.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:30.162963+0000) 2022-09-07T23:06:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:30.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:30.062993+0000) 2022-09-07T23:06:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:31.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:31.063188+0000) 2022-09-07T23:06:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:31.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:31.163096+0000) 2022-09-07T23:06:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:31 smithi055 ceph-mon[97033]: pgmap v2782: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:06:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:31 smithi059 ceph-mon[97450]: pgmap v2782: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:06:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:32.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:32.063372+0000) 2022-09-07T23:06:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:32.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:32.163287+0000) 2022-09-07T23:06:32.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:06:32.457+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:06:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:33.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:33.063536+0000) 2022-09-07T23:06:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:33.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:33.163484+0000) 2022-09-07T23:06:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:33 smithi055 ceph-mon[97033]: pgmap v2783: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:33 smithi059 ceph-mon[97450]: pgmap v2783: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:34.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:34.163739+0000) 2022-09-07T23:06:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:34.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:34.063767+0000) 2022-09-07T23:06:35.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:35.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:35.163964+0000) 2022-09-07T23:06:35.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:35.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:35.064032+0000) 2022-09-07T23:06:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:35 smithi055 ceph-mon[97033]: pgmap v2784: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:35 smithi059 ceph-mon[97450]: pgmap v2784: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:36.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:36.064253+0000) 2022-09-07T23:06:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:36.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:36.164187+0000) 2022-09-07T23:06:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:37.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:37.164424+0000) 2022-09-07T23:06:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:37.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:37.064475+0000) 2022-09-07T23:06:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:37 smithi055 ceph-mon[97033]: pgmap v2785: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:06:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:37 smithi059 ceph-mon[97450]: pgmap v2785: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:06:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:38.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:38.064727+0000) 2022-09-07T23:06:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:38.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:38.164613+0000) 2022-09-07T23:06:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:39.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:39.064926+0000) 2022-09-07T23:06:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:39.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:39.164905+0000) 2022-09-07T23:06:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:39 smithi055 ceph-mon[97033]: pgmap v2786: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:39 smithi059 ceph-mon[97450]: pgmap v2786: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:40.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:40.165123+0000) 2022-09-07T23:06:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:40.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:40.065182+0000) 2022-09-07T23:06:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:41.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:41.165354+0000) 2022-09-07T23:06:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:41.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:41.065351+0000) 2022-09-07T23:06:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:41 smithi055 ceph-mon[97033]: pgmap v2787: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:06:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:41 smithi059 ceph-mon[97450]: pgmap v2787: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:06:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:42.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:42.165573+0000) 2022-09-07T23:06:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:42.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:42.065577+0000) 2022-09-07T23:06:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:43.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:43.165726+0000) 2022-09-07T23:06:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:43.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:43.065796+0000) 2022-09-07T23:06:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:43 smithi055 ceph-mon[97033]: pgmap v2788: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:43 smithi059 ceph-mon[97450]: pgmap v2788: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:44.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:44.165951+0000) 2022-09-07T23:06:44.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:44.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:44.066066+0000) 2022-09-07T23:06:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:45.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:45.166199+0000) 2022-09-07T23:06:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:45.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:45.066309+0000) 2022-09-07T23:06:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:45 smithi055 ceph-mon[97033]: pgmap v2789: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:45 smithi059 ceph-mon[97450]: pgmap v2789: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:46.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:46.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:46.166406+0000) 2022-09-07T23:06:46.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:46.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:46.066520+0000) 2022-09-07T23:06:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:47.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:47.066740+0000) 2022-09-07T23:06:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:47.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:47.166608+0000) 2022-09-07T23:06:47.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:06:47.474+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:06:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:47 smithi055 ceph-mon[97033]: pgmap v2790: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:06:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:47 smithi059 ceph-mon[97450]: pgmap v2790: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:06:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:48.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:48.166863+0000) 2022-09-07T23:06:48.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:48.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:48.066882+0000) 2022-09-07T23:06:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:49.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:49.167085+0000) 2022-09-07T23:06:49.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:49.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:49.067066+0000) 2022-09-07T23:06:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:49 smithi055 ceph-mon[97033]: pgmap v2791: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:49 smithi059 ceph-mon[97450]: pgmap v2791: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:50.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:50.167304+0000) 2022-09-07T23:06:50.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:50.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:50.067284+0000) 2022-09-07T23:06:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:51.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:51.167443+0000) 2022-09-07T23:06:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:51.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:51.067520+0000) 2022-09-07T23:06:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:51 smithi055 ceph-mon[97033]: pgmap v2792: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:06:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:51 smithi059 ceph-mon[97450]: pgmap v2792: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:06:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:52.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:52.167711+0000) 2022-09-07T23:06:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:52.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:52.067749+0000) 2022-09-07T23:06:52.824 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:06:52.824 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:06:52.825 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:06:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:06:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:06:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:06:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:53.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:53.167896+0000) 2022-09-07T23:06:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:53.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:53.067942+0000) 2022-09-07T23:06:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:53 smithi055 ceph-mon[97033]: pgmap v2793: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:06:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:53 smithi059 ceph-mon[97450]: pgmap v2793: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:06:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:54.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:54.168124+0000) 2022-09-07T23:06:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:54.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:54.068177+0000) 2022-09-07T23:06:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:55.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:55.168324+0000) 2022-09-07T23:06:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:55.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:55.068413+0000) 2022-09-07T23:06:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:55 smithi055 ceph-mon[97033]: pgmap v2794: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:55 smithi059 ceph-mon[97450]: pgmap v2794: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:06:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:56.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:56.168493+0000) 2022-09-07T23:06:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:56.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:56.068618+0000) 2022-09-07T23:06:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:57.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:57.168715+0000) 2022-09-07T23:06:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:57.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:57.068896+0000) 2022-09-07T23:06:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:57 smithi055 ceph-mon[97033]: pgmap v2795: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:06:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:57 smithi059 ceph-mon[97450]: pgmap v2795: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:06:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:58.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:58.168994+0000) 2022-09-07T23:06:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:58.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:58.069065+0000) 2022-09-07T23:06:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:06:59.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:59.169188+0000) 2022-09-07T23:06:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:06:59.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:06:59.069286+0000) 2022-09-07T23:06:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:06:59 smithi055 ceph-mon[97033]: pgmap v2796: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:06:59 smithi059 ceph-mon[97450]: pgmap v2796: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:00.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:00.169418+0000) 2022-09-07T23:07:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:00.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:00.069530+0000) 2022-09-07T23:07:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:01.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:01.169698+0000) 2022-09-07T23:07:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:01.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:01.069752+0000) 2022-09-07T23:07:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:01 smithi055 ceph-mon[97033]: pgmap v2797: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:07:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:01 smithi059 ceph-mon[97450]: pgmap v2797: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:07:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:02.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:02.169888+0000) 2022-09-07T23:07:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:02.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:02.069987+0000) 2022-09-07T23:07:02.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:07:02.490+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:07:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:03.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:03.070202+0000) 2022-09-07T23:07:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:03.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:03.170124+0000) 2022-09-07T23:07:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:03 smithi055 ceph-mon[97033]: pgmap v2798: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:07:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:03 smithi059 ceph-mon[97450]: pgmap v2798: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:07:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:04.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:04.170331+0000) 2022-09-07T23:07:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:04.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:04.070426+0000) 2022-09-07T23:07:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:07:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:07:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:07:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:07:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:05.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:05.070723+0000) 2022-09-07T23:07:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:05.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:05.170561+0000) 2022-09-07T23:07:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:05 smithi055 ceph-mon[97033]: pgmap v2799: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:07:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:05 smithi059 ceph-mon[97450]: pgmap v2799: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:07:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:06.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:06.070888+0000) 2022-09-07T23:07:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:06.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:06.170781+0000) 2022-09-07T23:07:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:07.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:07.170975+0000) 2022-09-07T23:07:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:07.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:07.071057+0000) 2022-09-07T23:07:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:07 smithi055 ceph-mon[97033]: pgmap v2800: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:07:08.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:07 smithi059 ceph-mon[97450]: pgmap v2800: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:07:08.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:08.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:08.071183+0000) 2022-09-07T23:07:08.408 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:08.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:08.171211+0000) 2022-09-07T23:07:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:09.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:09.171510+0000) 2022-09-07T23:07:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:09.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:09.071365+0000) 2022-09-07T23:07:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:09 smithi055 ceph-mon[97033]: pgmap v2801: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:07:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:09 smithi055 ceph-mon[97033]: osdmap e265: 12 total, 11 up, 11 in 2022-09-07T23:07:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:09 smithi059 ceph-mon[97450]: pgmap v2801: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:07:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:09 smithi059 ceph-mon[97450]: osdmap e265: 12 total, 11 up, 11 in 2022-09-07T23:07:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:10.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:10.171745+0000) 2022-09-07T23:07:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:10.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:10.071602+0000) 2022-09-07T23:07:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:11.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:11.171926+0000) 2022-09-07T23:07:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:11.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:11.071791+0000) 2022-09-07T23:07:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:11 smithi055 ceph-mon[97033]: pgmap v2803: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:07:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:11 smithi059 ceph-mon[97450]: pgmap v2803: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:07:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:12.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:12.172117+0000) 2022-09-07T23:07:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:12.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:12.071913+0000) 2022-09-07T23:07:13.414 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:13.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:13.172302+0000) 2022-09-07T23:07:13.415 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:13.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:13.072100+0000) 2022-09-07T23:07:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:13 smithi055 ceph-mon[97033]: pgmap v2804: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:07:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:13 smithi055 ceph-mon[97033]: osdmap e266: 12 total, 11 up, 11 in 2022-09-07T23:07:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:13 smithi059 ceph-mon[97450]: pgmap v2804: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:07:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:13 smithi059 ceph-mon[97450]: osdmap e266: 12 total, 11 up, 11 in 2022-09-07T23:07:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:14.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:14.172542+0000) 2022-09-07T23:07:14.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:14.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:14.072303+0000) 2022-09-07T23:07:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:15.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:15.172699+0000) 2022-09-07T23:07:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:15.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:15.072509+0000) 2022-09-07T23:07:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:15 smithi055 ceph-mon[97033]: pgmap v2806: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:07:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:15 smithi059 ceph-mon[97450]: pgmap v2806: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:07:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:16.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:16.072742+0000) 2022-09-07T23:07:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:16.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:16.172854+0000) 2022-09-07T23:07:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:16 smithi055 ceph-mon[97033]: pgmap v2807: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:07:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:16 smithi059 ceph-mon[97450]: pgmap v2807: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:07:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:17.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:17.173122+0000) 2022-09-07T23:07:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:17.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:17.072888+0000) 2022-09-07T23:07:17.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:07:17.501+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:07:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:18.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:18.173380+0000) 2022-09-07T23:07:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:18.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:18.073108+0000) 2022-09-07T23:07:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:19.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:19.173656+0000) 2022-09-07T23:07:19.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:19.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:19.073359+0000) 2022-09-07T23:07:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:19 smithi055 ceph-mon[97033]: pgmap v2808: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:07:20.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:19 smithi059 ceph-mon[97450]: pgmap v2808: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:07:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:20.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:20.173814+0000) 2022-09-07T23:07:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:20.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:20.073595+0000) 2022-09-07T23:07:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:21.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:21.174026+0000) 2022-09-07T23:07:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:21.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:21.073801+0000) 2022-09-07T23:07:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:21 smithi055 ceph-mon[97033]: pgmap v2809: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T23:07:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:21 smithi059 ceph-mon[97450]: pgmap v2809: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T23:07:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:22.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:22.174256+0000) 2022-09-07T23:07:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:22.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:22.074029+0000) 2022-09-07T23:07:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:23.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:23.174512+0000) 2022-09-07T23:07:23.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:23.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:23.074281+0000) 2022-09-07T23:07:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:23 smithi055 ceph-mon[97033]: pgmap v2810: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T23:07:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:23 smithi059 ceph-mon[97450]: pgmap v2810: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T23:07:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:24.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:24.174764+0000) 2022-09-07T23:07:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:24.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:24.074545+0000) 2022-09-07T23:07:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:25.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:25.174945+0000) 2022-09-07T23:07:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:25.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:25.074736+0000) 2022-09-07T23:07:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:25 smithi055 ceph-mon[97033]: pgmap v2811: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:07:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:25 smithi059 ceph-mon[97450]: pgmap v2811: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:07:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:26.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:26.175135+0000) 2022-09-07T23:07:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:26.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:26.074926+0000) 2022-09-07T23:07:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:27.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:27.175252+0000) 2022-09-07T23:07:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:27.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:27.075173+0000) 2022-09-07T23:07:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:27 smithi055 ceph-mon[97033]: pgmap v2812: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:07:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:27 smithi059 ceph-mon[97450]: pgmap v2812: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:07:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:28.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:28.175459+0000) 2022-09-07T23:07:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:28.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:28.075420+0000) 2022-09-07T23:07:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:29.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:29.175723+0000) 2022-09-07T23:07:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:29.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:29.075685+0000) 2022-09-07T23:07:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:29 smithi055 ceph-mon[97033]: pgmap v2813: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:30.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:29 smithi059 ceph-mon[97450]: pgmap v2813: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:30.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:30.175931+0000) 2022-09-07T23:07:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:30.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:30.075893+0000) 2022-09-07T23:07:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:31.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:31.176078+0000) 2022-09-07T23:07:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:31.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:31.076079+0000) 2022-09-07T23:07:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:31 smithi055 ceph-mon[97033]: pgmap v2814: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:07:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:31 smithi059 ceph-mon[97450]: pgmap v2814: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:07:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:32.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:32.176285+0000) 2022-09-07T23:07:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:32.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:32.076307+0000) 2022-09-07T23:07:32.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:07:32.514+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:07:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:33.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:33.176456+0000) 2022-09-07T23:07:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:33.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:33.076480+0000) 2022-09-07T23:07:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:33 smithi055 ceph-mon[97033]: pgmap v2815: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:33 smithi059 ceph-mon[97450]: pgmap v2815: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:34.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:34.176633+0000) 2022-09-07T23:07:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:34.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:34.076656+0000) 2022-09-07T23:07:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:35.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:35.176906+0000) 2022-09-07T23:07:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:35.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:35.076870+0000) 2022-09-07T23:07:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:35 smithi055 ceph-mon[97033]: pgmap v2816: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:35 smithi059 ceph-mon[97450]: pgmap v2816: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:36.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:36.177129+0000) 2022-09-07T23:07:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:36.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:36.077089+0000) 2022-09-07T23:07:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:37.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:37.177338+0000) 2022-09-07T23:07:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:37.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:37.077272+0000) 2022-09-07T23:07:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:37 smithi055 ceph-mon[97033]: pgmap v2817: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:07:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:37 smithi059 ceph-mon[97450]: pgmap v2817: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:07:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:38.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:38.177560+0000) 2022-09-07T23:07:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:38.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:38.077478+0000) 2022-09-07T23:07:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:39.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:39.177813+0000) 2022-09-07T23:07:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:39.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:39.077712+0000) 2022-09-07T23:07:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:39 smithi055 ceph-mon[97033]: pgmap v2818: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:39 smithi059 ceph-mon[97450]: pgmap v2818: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:40.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:40.178040+0000) 2022-09-07T23:07:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:40.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:40.077919+0000) 2022-09-07T23:07:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:41.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:41.178279+0000) 2022-09-07T23:07:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:41.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:41.078125+0000) 2022-09-07T23:07:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:41 smithi055 ceph-mon[97033]: pgmap v2819: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:07:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:41 smithi059 ceph-mon[97450]: pgmap v2819: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:07:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:42.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:42.178475+0000) 2022-09-07T23:07:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:42.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:42.078390+0000) 2022-09-07T23:07:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:43.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:43.078623+0000) 2022-09-07T23:07:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:43.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:43.178691+0000) 2022-09-07T23:07:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:43 smithi055 ceph-mon[97033]: pgmap v2820: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:43 smithi059 ceph-mon[97450]: pgmap v2820: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:44.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:44.178912+0000) 2022-09-07T23:07:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:44.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:44.078833+0000) 2022-09-07T23:07:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:45.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:45.179140+0000) 2022-09-07T23:07:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:45.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:45.079014+0000) 2022-09-07T23:07:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:45 smithi055 ceph-mon[97033]: pgmap v2821: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:45 smithi059 ceph-mon[97450]: pgmap v2821: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:46.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:46.179362+0000) 2022-09-07T23:07:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:46.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:46.079226+0000) 2022-09-07T23:07:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:47.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:47.179588+0000) 2022-09-07T23:07:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:47.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:47.079443+0000) 2022-09-07T23:07:47.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:07:47.520+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:07:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:47 smithi055 ceph-mon[97033]: pgmap v2822: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:07:48.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:47 smithi059 ceph-mon[97450]: pgmap v2822: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:07:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:48.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:48.179712+0000) 2022-09-07T23:07:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:48.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:48.079664+0000) 2022-09-07T23:07:49.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:49.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:49.179890+0000) 2022-09-07T23:07:49.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:49.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:49.079842+0000) 2022-09-07T23:07:49.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:49 smithi055 ceph-mon[97033]: pgmap v2823: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:50.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:49 smithi059 ceph-mon[97450]: pgmap v2823: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:50.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:50.180093+0000) 2022-09-07T23:07:50.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:50.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:50.079994+0000) 2022-09-07T23:07:51.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:51.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:51.180327+0000) 2022-09-07T23:07:51.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:51.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:51.080129+0000) 2022-09-07T23:07:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:51 smithi055 ceph-mon[97033]: pgmap v2824: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:07:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:51 smithi059 ceph-mon[97450]: pgmap v2824: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:07:52.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:52.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:52.180589+0000) 2022-09-07T23:07:52.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:52.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:52.080341+0000) 2022-09-07T23:07:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:53.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:53.180852+0000) 2022-09-07T23:07:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:53.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:53.080575+0000) 2022-09-07T23:07:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:53 smithi059 ceph-mon[97450]: pgmap v2825: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:07:54.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:07:54.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:07:54.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:07:54.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:07:54.152 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:53 smithi055 ceph-mon[97033]: pgmap v2825: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:54.152 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:07:54.154 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:07:54.154 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:07:54.155 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:07:54.156 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:07:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:54.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:54.181006+0000) 2022-09-07T23:07:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:54.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:54.080746+0000) 2022-09-07T23:07:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:55.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:55.181219+0000) 2022-09-07T23:07:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:55.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:55.080909+0000) 2022-09-07T23:07:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:07:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:55 smithi055 ceph-mon[97033]: pgmap v2826: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:07:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:55 smithi059 ceph-mon[97450]: pgmap v2826: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:07:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:56.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:56.181450+0000) 2022-09-07T23:07:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:56.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:56.081093+0000) 2022-09-07T23:07:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:57.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:57.181648+0000) 2022-09-07T23:07:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:57.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:57.081329+0000) 2022-09-07T23:07:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:57 smithi055 ceph-mon[97033]: pgmap v2827: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:07:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:57 smithi059 ceph-mon[97450]: pgmap v2827: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:07:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:58.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:58.181890+0000) 2022-09-07T23:07:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:58.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:58.081558+0000) 2022-09-07T23:07:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:07:59.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:59.182082+0000) 2022-09-07T23:07:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:07:59.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:07:59.081777+0000) 2022-09-07T23:07:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:07:59 smithi055 ceph-mon[97033]: pgmap v2828: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:07:59 smithi059 ceph-mon[97450]: pgmap v2828: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:00.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:00.182344+0000) 2022-09-07T23:08:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:00.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:00.081933+0000) 2022-09-07T23:08:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:01.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:01.182603+0000) 2022-09-07T23:08:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:01.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:01.082099+0000) 2022-09-07T23:08:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:01 smithi055 ceph-mon[97033]: pgmap v2829: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:08:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:01 smithi059 ceph-mon[97450]: pgmap v2829: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:08:02.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:02.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:02.182834+0000) 2022-09-07T23:08:02.409 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:02.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:02.082252+0000) 2022-09-07T23:08:02.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:08:02.533+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:08:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:03.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:03.183002+0000) 2022-09-07T23:08:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:03.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:03.082384+0000) 2022-09-07T23:08:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:03 smithi055 ceph-mon[97033]: pgmap v2830: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:03 smithi059 ceph-mon[97450]: pgmap v2830: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:04.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:04.183247+0000) 2022-09-07T23:08:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:04.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:04.082559+0000) 2022-09-07T23:08:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:08:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:08:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:08:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:08:05.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:05.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:05.183474+0000) 2022-09-07T23:08:05.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:05.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:05.082705+0000) 2022-09-07T23:08:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:05 smithi055 ceph-mon[97033]: pgmap v2831: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:05 smithi059 ceph-mon[97450]: pgmap v2831: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:06.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:06.183743+0000) 2022-09-07T23:08:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:06.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:06.082875+0000) 2022-09-07T23:08:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:07.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:07.183970+0000) 2022-09-07T23:08:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:07.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:07.083091+0000) 2022-09-07T23:08:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:07 smithi055 ceph-mon[97033]: pgmap v2832: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:08:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:07 smithi059 ceph-mon[97450]: pgmap v2832: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:08:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:08.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:08.184115+0000) 2022-09-07T23:08:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:08.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:08.083321+0000) 2022-09-07T23:08:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:09.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:09.184299+0000) 2022-09-07T23:08:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:09.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:09.083550+0000) 2022-09-07T23:08:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:09 smithi055 ceph-mon[97033]: pgmap v2833: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:08:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:09 smithi055 ceph-mon[97033]: osdmap e267: 12 total, 11 up, 11 in 2022-09-07T23:08:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:09 smithi059 ceph-mon[97450]: pgmap v2833: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:08:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:09 smithi059 ceph-mon[97450]: osdmap e267: 12 total, 11 up, 11 in 2022-09-07T23:08:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:10.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:10.184523+0000) 2022-09-07T23:08:10.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:10.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:10.083775+0000) 2022-09-07T23:08:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:11.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:11.184693+0000) 2022-09-07T23:08:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:11.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:11.083963+0000) 2022-09-07T23:08:12.186 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:12.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:12.084186+0000) 2022-09-07T23:08:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:11 smithi055 ceph-mon[97033]: pgmap v2835: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:08:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:11 smithi059 ceph-mon[97450]: pgmap v2835: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:08:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:12.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:12.184921+0000) 2022-09-07T23:08:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:13.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:13.084443+0000) 2022-09-07T23:08:13.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:13.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:13.185162+0000) 2022-09-07T23:08:14.239 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:13 smithi055 ceph-mon[97033]: pgmap v2836: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:08:14.240 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:13 smithi055 ceph-mon[97033]: osdmap e268: 12 total, 11 up, 11 in 2022-09-07T23:08:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:13 smithi059 ceph-mon[97450]: pgmap v2836: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:08:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:13 smithi059 ceph-mon[97450]: osdmap e268: 12 total, 11 up, 11 in 2022-09-07T23:08:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:14.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:14.185408+0000) 2022-09-07T23:08:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:14.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:14.084684+0000) 2022-09-07T23:08:15.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:15.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:15.084929+0000) 2022-09-07T23:08:15.413 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:15.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:15.185683+0000) 2022-09-07T23:08:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:15 smithi055 ceph-mon[97033]: pgmap v2838: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:08:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:15 smithi059 ceph-mon[97450]: pgmap v2838: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:08:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:16.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:16.085057+0000) 2022-09-07T23:08:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:16.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:16.185862+0000) 2022-09-07T23:08:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:17.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:17.085290+0000) 2022-09-07T23:08:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:17.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:17.186082+0000) 2022-09-07T23:08:17.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:08:17.545+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:08:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:17 smithi055 ceph-mon[97033]: pgmap v2839: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:08:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:17 smithi059 ceph-mon[97450]: pgmap v2839: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:08:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:18.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:18.186295+0000) 2022-09-07T23:08:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:18.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:18.085549+0000) 2022-09-07T23:08:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:19.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:19.186512+0000) 2022-09-07T23:08:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:19.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:19.085810+0000) 2022-09-07T23:08:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:19 smithi055 ceph-mon[97033]: pgmap v2840: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:08:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:19 smithi059 ceph-mon[97450]: pgmap v2840: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:08:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:20.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:20.186719+0000) 2022-09-07T23:08:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:20.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:20.085980+0000) 2022-09-07T23:08:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:21.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:21.186917+0000) 2022-09-07T23:08:21.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:21.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:21.086139+0000) 2022-09-07T23:08:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:21 smithi055 ceph-mon[97033]: pgmap v2841: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:08:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:21 smithi059 ceph-mon[97450]: pgmap v2841: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:08:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:22.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:22.187130+0000) 2022-09-07T23:08:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:22.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:22.086352+0000) 2022-09-07T23:08:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:23.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:23.086589+0000) 2022-09-07T23:08:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:23.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:23.187342+0000) 2022-09-07T23:08:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:23 smithi055 ceph-mon[97033]: pgmap v2842: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:08:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:23 smithi059 ceph-mon[97450]: pgmap v2842: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:08:24.410 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:24.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:24.187595+0000) 2022-09-07T23:08:24.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:24.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:24.086749+0000) 2022-09-07T23:08:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:25.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:25.086951+0000) 2022-09-07T23:08:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:25.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:25.187839+0000) 2022-09-07T23:08:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:25 smithi055 ceph-mon[97033]: pgmap v2843: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:08:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:25 smithi059 ceph-mon[97450]: pgmap v2843: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:08:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:26.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:26.087069+0000) 2022-09-07T23:08:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:26.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:26.188059+0000) 2022-09-07T23:08:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:27.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:27.087205+0000) 2022-09-07T23:08:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:27.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:27.188287+0000) 2022-09-07T23:08:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:27 smithi055 ceph-mon[97033]: pgmap v2844: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:08:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:27 smithi059 ceph-mon[97450]: pgmap v2844: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:08:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:28.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:28.087332+0000) 2022-09-07T23:08:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:28.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:28.188520+0000) 2022-09-07T23:08:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:29.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:29.188786+0000) 2022-09-07T23:08:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:29.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:29.087557+0000) 2022-09-07T23:08:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:29 smithi055 ceph-mon[97033]: pgmap v2845: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:29 smithi059 ceph-mon[97450]: pgmap v2845: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:30.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:30.188972+0000) 2022-09-07T23:08:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:30.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:30.087789+0000) 2022-09-07T23:08:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:31.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:31.189220+0000) 2022-09-07T23:08:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:31.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:31.087988+0000) 2022-09-07T23:08:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:31 smithi055 ceph-mon[97033]: pgmap v2846: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:08:32.294 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:31 smithi059 ceph-mon[97450]: pgmap v2846: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:08:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:32.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:32.088226+0000) 2022-09-07T23:08:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:32.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:32.189460+0000) 2022-09-07T23:08:32.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:08:32.560+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:08:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:33.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:33.088398+0000) 2022-09-07T23:08:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:33.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:33.189625+0000) 2022-09-07T23:08:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:33 smithi055 ceph-mon[97033]: pgmap v2847: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:33 smithi059 ceph-mon[97450]: pgmap v2847: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:34.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:34.189830+0000) 2022-09-07T23:08:34.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:34.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:34.088631+0000) 2022-09-07T23:08:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:35.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:35.088790+0000) 2022-09-07T23:08:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:35.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:35.190090+0000) 2022-09-07T23:08:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:35 smithi055 ceph-mon[97033]: pgmap v2848: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:35 smithi059 ceph-mon[97450]: pgmap v2848: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:36.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:36.089002+0000) 2022-09-07T23:08:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:36.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:36.190347+0000) 2022-09-07T23:08:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:36 smithi055 ceph-mon[97033]: pgmap v2849: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:08:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:36 smithi059 ceph-mon[97450]: pgmap v2849: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:08:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:37.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:37.190526+0000) 2022-09-07T23:08:37.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:37.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:37.089253+0000) 2022-09-07T23:08:38.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:38.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:38.190734+0000) 2022-09-07T23:08:38.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:38.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:38.089411+0000) 2022-09-07T23:08:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:39.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:39.190944+0000) 2022-09-07T23:08:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:39.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:39.089683+0000) 2022-09-07T23:08:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:39 smithi059 ceph-mon[97450]: pgmap v2850: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:39 smithi055 ceph-mon[97033]: pgmap v2850: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:40.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:40.089854+0000) 2022-09-07T23:08:40.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:40.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:40.191108+0000) 2022-09-07T23:08:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:41.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:41.191350+0000) 2022-09-07T23:08:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:41.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:41.090025+0000) 2022-09-07T23:08:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:41 smithi055 ceph-mon[97033]: pgmap v2851: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:08:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:41 smithi059 ceph-mon[97450]: pgmap v2851: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:08:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:42.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:42.191573+0000) 2022-09-07T23:08:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:42.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:42.090240+0000) 2022-09-07T23:08:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:43.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:43.191833+0000) 2022-09-07T23:08:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:43.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:43.090507+0000) 2022-09-07T23:08:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:43 smithi055 ceph-mon[97033]: pgmap v2852: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:43 smithi059 ceph-mon[97450]: pgmap v2852: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:44.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:44.192000+0000) 2022-09-07T23:08:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:44.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:44.090746+0000) 2022-09-07T23:08:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:45.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:45.192241+0000) 2022-09-07T23:08:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:45.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:45.090957+0000) 2022-09-07T23:08:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:45 smithi055 ceph-mon[97033]: pgmap v2853: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:45 smithi059 ceph-mon[97450]: pgmap v2853: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:46.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:46.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:46.192413+0000) 2022-09-07T23:08:46.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:46.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:46.091148+0000) 2022-09-07T23:08:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:46 smithi055 ceph-mon[97033]: pgmap v2854: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:08:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:46 smithi059 ceph-mon[97450]: pgmap v2854: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:08:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:47.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:47.192665+0000) 2022-09-07T23:08:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:47.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:47.091421+0000) 2022-09-07T23:08:47.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:08:47.565+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:08:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:48.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:48.192935+0000) 2022-09-07T23:08:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:48.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:48.091711+0000) 2022-09-07T23:08:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:49.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:49.193082+0000) 2022-09-07T23:08:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:49.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:49.091964+0000) 2022-09-07T23:08:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:49 smithi059 ceph-mon[97450]: pgmap v2855: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:49 smithi055 ceph-mon[97033]: pgmap v2855: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:50.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:50.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:50.092295+0000) 2022-09-07T23:08:50.408 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:50.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:50.193319+0000) 2022-09-07T23:08:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:51.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:51.193515+0000) 2022-09-07T23:08:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:51.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:51.092539+0000) 2022-09-07T23:08:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:51 smithi055 ceph-mon[97033]: pgmap v2856: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:08:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:51 smithi059 ceph-mon[97450]: pgmap v2856: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:08:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:52.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:52.193737+0000) 2022-09-07T23:08:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:52.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:52.092792+0000) 2022-09-07T23:08:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:53.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:53.193980+0000) 2022-09-07T23:08:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:53.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:53.093038+0000) 2022-09-07T23:08:54.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:53 smithi055 ceph-mon[97033]: pgmap v2857: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:54.258 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:53 smithi059 ceph-mon[97450]: pgmap v2857: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:54.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:54.093232+0000) 2022-09-07T23:08:54.409 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:54.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:54.194222+0000) 2022-09-07T23:08:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:08:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:08:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:08:55.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:08:55.289 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:08:55.290 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:08:55.291 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:08:55.291 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:08:55.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:55.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:55.194476+0000) 2022-09-07T23:08:55.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:55.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:55.093459+0000) 2022-09-07T23:08:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:55 smithi055 ceph-mon[97033]: pgmap v2858: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:55 smithi059 ceph-mon[97450]: pgmap v2858: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:08:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:56.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:56.194751+0000) 2022-09-07T23:08:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:56.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:56.093666+0000) 2022-09-07T23:08:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:56 smithi055 ceph-mon[97033]: pgmap v2859: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:08:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:56 smithi059 ceph-mon[97450]: pgmap v2859: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:08:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:57.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:57.194984+0000) 2022-09-07T23:08:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:57.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:57.093759+0000) 2022-09-07T23:08:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:58.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:58.195231+0000) 2022-09-07T23:08:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:58.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:58.093961+0000) 2022-09-07T23:08:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:08:59.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:59.195517+0000) 2022-09-07T23:08:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:08:59.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:08:59.094154+0000) 2022-09-07T23:09:00.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:08:59 smithi059 ceph-mon[97450]: pgmap v2860: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:00.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:08:59 smithi055 ceph-mon[97033]: pgmap v2860: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:00.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:00.195734+0000) 2022-09-07T23:09:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:00.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:00.094349+0000) 2022-09-07T23:09:01.409 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:01.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:01.195854+0000) 2022-09-07T23:09:01.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:01.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:01.094493+0000) 2022-09-07T23:09:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:01 smithi055 ceph-mon[97033]: pgmap v2861: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:09:02.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:01 smithi059 ceph-mon[97450]: pgmap v2861: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:09:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:02.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:02.196050+0000) 2022-09-07T23:09:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:02.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:02.094729+0000) 2022-09-07T23:09:02.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:09:02.576+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:09:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:03.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:03.196239+0000) 2022-09-07T23:09:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:03.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:03.094875+0000) 2022-09-07T23:09:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:03 smithi055 ceph-mon[97033]: pgmap v2862: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:03 smithi059 ceph-mon[97450]: pgmap v2862: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:04.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:04.196445+0000) 2022-09-07T23:09:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:04.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:04.095061+0000) 2022-09-07T23:09:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:09:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:09:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:04 smithi055 ceph-mon[97033]: pgmap v2863: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:09:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:09:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:04 smithi059 ceph-mon[97450]: pgmap v2863: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:05.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:05.196730+0000) 2022-09-07T23:09:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:05.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:05.095277+0000) 2022-09-07T23:09:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:06.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:06.196963+0000) 2022-09-07T23:09:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:06.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:06.095478+0000) 2022-09-07T23:09:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:07.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:07.197180+0000) 2022-09-07T23:09:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:07.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:07.095734+0000) 2022-09-07T23:09:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:07 smithi055 ceph-mon[97033]: pgmap v2864: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:09:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:07 smithi059 ceph-mon[97450]: pgmap v2864: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:09:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:08.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:08.197379+0000) 2022-09-07T23:09:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:08.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:08.095946+0000) 2022-09-07T23:09:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:09.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:09.197548+0000) 2022-09-07T23:09:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:09.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:09.096100+0000) 2022-09-07T23:09:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:09 smithi059 ceph-mon[97450]: pgmap v2865: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:09:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:09 smithi059 ceph-mon[97450]: osdmap e269: 12 total, 11 up, 11 in 2022-09-07T23:09:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:09 smithi055 ceph-mon[97033]: pgmap v2865: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:09:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:09 smithi055 ceph-mon[97033]: osdmap e269: 12 total, 11 up, 11 in 2022-09-07T23:09:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:10.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:10.197741+0000) 2022-09-07T23:09:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:10.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:10.096315+0000) 2022-09-07T23:09:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:11.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:11.197935+0000) 2022-09-07T23:09:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:11.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:11.096555+0000) 2022-09-07T23:09:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:11 smithi055 ceph-mon[97033]: pgmap v2867: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:09:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:11 smithi059 ceph-mon[97450]: pgmap v2867: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:09:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:12.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:12.198066+0000) 2022-09-07T23:09:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:12.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:12.096770+0000) 2022-09-07T23:09:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:12 smithi055 ceph-mon[97033]: pgmap v2868: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:09:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:12 smithi059 ceph-mon[97450]: pgmap v2868: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:09:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:13.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:13.198274+0000) 2022-09-07T23:09:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:13.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:13.096992+0000) 2022-09-07T23:09:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:14.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:14.198491+0000) 2022-09-07T23:09:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:14.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:14.097249+0000) 2022-09-07T23:09:15.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:14 smithi059 ceph-mon[97450]: osdmap e270: 12 total, 11 up, 11 in 2022-09-07T23:09:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:14 smithi055 ceph-mon[97033]: osdmap e270: 12 total, 11 up, 11 in 2022-09-07T23:09:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:15.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:15.198666+0000) 2022-09-07T23:09:15.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:15.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:15.097474+0000) 2022-09-07T23:09:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:15 smithi059 ceph-mon[97450]: pgmap v2870: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:09:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:15 smithi055 ceph-mon[97033]: pgmap v2870: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:09:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:16.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:16.198845+0000) 2022-09-07T23:09:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:16.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:16.097614+0000) 2022-09-07T23:09:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:17.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:17.097848+0000) 2022-09-07T23:09:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:17.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:17.199084+0000) 2022-09-07T23:09:17.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:09:17.586+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:09:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:17 smithi055 ceph-mon[97033]: pgmap v2871: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:09:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:17 smithi059 ceph-mon[97450]: pgmap v2871: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:09:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:18.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:18.199241+0000) 2022-09-07T23:09:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:18.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:18.098020+0000) 2022-09-07T23:09:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:19.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:19.199428+0000) 2022-09-07T23:09:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:19.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:19.098259+0000) 2022-09-07T23:09:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:19 smithi059 ceph-mon[97450]: pgmap v2872: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T23:09:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:19 smithi055 ceph-mon[97033]: pgmap v2872: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T23:09:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:20.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:20.199655+0000) 2022-09-07T23:09:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:20.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:20.098475+0000) 2022-09-07T23:09:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:21.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:21.098728+0000) 2022-09-07T23:09:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:21.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:21.199907+0000) 2022-09-07T23:09:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:21 smithi055 ceph-mon[97033]: pgmap v2873: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:09:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:21 smithi059 ceph-mon[97450]: pgmap v2873: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:09:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:22.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:22.098948+0000) 2022-09-07T23:09:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:22.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:22.200058+0000) 2022-09-07T23:09:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:22 smithi055 ceph-mon[97033]: pgmap v2874: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:09:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:22 smithi059 ceph-mon[97450]: pgmap v2874: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:09:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:23.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:23.099146+0000) 2022-09-07T23:09:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:23.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:23.200290+0000) 2022-09-07T23:09:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:24.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:24.200522+0000) 2022-09-07T23:09:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:24.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:24.099273+0000) 2022-09-07T23:09:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:24 smithi055 ceph-mon[97033]: pgmap v2875: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:09:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:24 smithi059 ceph-mon[97450]: pgmap v2875: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:09:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:25.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:25.200741+0000) 2022-09-07T23:09:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:25.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:25.099506+0000) 2022-09-07T23:09:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:26.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:26.200947+0000) 2022-09-07T23:09:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:26.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:26.099674+0000) 2022-09-07T23:09:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:26 smithi055 ceph-mon[97033]: pgmap v2876: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:09:27.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:26 smithi059 ceph-mon[97450]: pgmap v2876: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:09:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:27.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:27.201200+0000) 2022-09-07T23:09:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:27.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:27.099870+0000) 2022-09-07T23:09:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:28.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:28.201421+0000) 2022-09-07T23:09:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:28.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:28.100011+0000) 2022-09-07T23:09:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:29.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:29.201663+0000) 2022-09-07T23:09:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:29.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:29.100243+0000) 2022-09-07T23:09:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:29 smithi059 ceph-mon[97450]: pgmap v2877: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:29 smithi055 ceph-mon[97033]: pgmap v2877: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:30.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:30.201894+0000) 2022-09-07T23:09:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:30.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:30.100459+0000) 2022-09-07T23:09:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:30 smithi055 ceph-mon[97033]: pgmap v2878: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:09:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:30 smithi059 ceph-mon[97450]: pgmap v2878: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:09:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:31.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:31.202132+0000) 2022-09-07T23:09:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:31.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:31.100734+0000) 2022-09-07T23:09:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:32.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:32.202318+0000) 2022-09-07T23:09:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:32.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:32.100895+0000) 2022-09-07T23:09:32.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:09:32.600+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:09:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:32 smithi055 ceph-mon[97033]: pgmap v2879: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:32 smithi059 ceph-mon[97450]: pgmap v2879: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:33.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:33.202524+0000) 2022-09-07T23:09:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:33.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:33.101067+0000) 2022-09-07T23:09:34.415 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:34.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:34.101315+0000) 2022-09-07T23:09:34.416 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:34.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:34.202794+0000) 2022-09-07T23:09:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:34 smithi055 ceph-mon[97033]: pgmap v2880: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:34 smithi059 ceph-mon[97450]: pgmap v2880: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:35.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:35.203014+0000) 2022-09-07T23:09:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:35.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:35.101610+0000) 2022-09-07T23:09:36.409 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:36.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:36.203176+0000) 2022-09-07T23:09:36.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:36.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:36.101871+0000) 2022-09-07T23:09:37.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:36 smithi055 ceph-mon[97033]: pgmap v2881: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:09:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:36 smithi059 ceph-mon[97450]: pgmap v2881: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:09:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:37.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:37.203412+0000) 2022-09-07T23:09:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:37.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:37.102118+0000) 2022-09-07T23:09:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:38.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:38.102338+0000) 2022-09-07T23:09:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:38.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:38.203669+0000) 2022-09-07T23:09:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:39.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:39.203924+0000) 2022-09-07T23:09:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:39.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:39.102571+0000) 2022-09-07T23:09:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:39 smithi059 ceph-mon[97450]: pgmap v2882: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:39 smithi055 ceph-mon[97033]: pgmap v2882: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:40.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:40.204148+0000) 2022-09-07T23:09:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:40.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:40.102825+0000) 2022-09-07T23:09:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:40 smithi055 ceph-mon[97033]: pgmap v2883: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:09:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:40 smithi059 ceph-mon[97450]: pgmap v2883: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:09:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:41.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:41.103059+0000) 2022-09-07T23:09:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:41.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:41.204375+0000) 2022-09-07T23:09:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:42.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:42.204547+0000) 2022-09-07T23:09:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:42.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:42.103290+0000) 2022-09-07T23:09:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:42 smithi055 ceph-mon[97033]: pgmap v2884: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:42 smithi059 ceph-mon[97450]: pgmap v2884: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:43.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:43.204779+0000) 2022-09-07T23:09:43.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:43.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:43.103565+0000) 2022-09-07T23:09:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:44.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:44.204936+0000) 2022-09-07T23:09:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:44.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:44.103837+0000) 2022-09-07T23:09:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:44 smithi055 ceph-mon[97033]: pgmap v2885: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:44 smithi059 ceph-mon[97450]: pgmap v2885: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:45.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:45.205123+0000) 2022-09-07T23:09:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:45.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:45.104035+0000) 2022-09-07T23:09:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:46.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:46.205313+0000) 2022-09-07T23:09:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:46.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:46.104190+0000) 2022-09-07T23:09:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:46 smithi055 ceph-mon[97033]: pgmap v2886: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:09:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:46 smithi059 ceph-mon[97450]: pgmap v2886: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:09:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:47.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:47.205558+0000) 2022-09-07T23:09:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:47.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:47.104389+0000) 2022-09-07T23:09:47.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:09:47.612+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:09:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:48.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:48.205791+0000) 2022-09-07T23:09:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:48.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:48.104608+0000) 2022-09-07T23:09:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:49.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:49.205992+0000) 2022-09-07T23:09:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:49.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:49.104895+0000) 2022-09-07T23:09:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:49 smithi059 ceph-mon[97450]: pgmap v2887: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:49 smithi055 ceph-mon[97033]: pgmap v2887: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:50.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:50.206153+0000) 2022-09-07T23:09:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:50.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:50.105071+0000) 2022-09-07T23:09:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:50 smithi055 ceph-mon[97033]: pgmap v2888: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:09:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:50 smithi059 ceph-mon[97450]: pgmap v2888: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:09:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:51.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:51.206375+0000) 2022-09-07T23:09:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:51.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:51.105270+0000) 2022-09-07T23:09:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:52.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:52.206608+0000) 2022-09-07T23:09:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:52.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:52.105397+0000) 2022-09-07T23:09:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:52 smithi055 ceph-mon[97033]: pgmap v2889: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:52 smithi059 ceph-mon[97450]: pgmap v2889: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:53.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:53.105624+0000) 2022-09-07T23:09:53.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:53.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:53.206795+0000) 2022-09-07T23:09:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:54.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:54.207056+0000) 2022-09-07T23:09:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:54.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:54.105783+0000) 2022-09-07T23:09:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:54 smithi055 ceph-mon[97033]: pgmap v2890: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:09:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:09:55.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:09:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:54 smithi059 ceph-mon[97450]: pgmap v2890: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:09:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:09:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:09:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:55.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:55.207331+0000) 2022-09-07T23:09:55.447 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:55.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:55.106043+0000) 2022-09-07T23:09:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:56.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:56.207579+0000) 2022-09-07T23:09:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:56.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:56.106286+0000) 2022-09-07T23:09:57.132 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:56 smithi059 ceph-mon[97450]: pgmap v2891: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:09:57.222 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:56 smithi055 ceph-mon[97033]: pgmap v2891: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:09:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:57.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:57.207807+0000) 2022-09-07T23:09:57.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:57.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:57.106521+0000) 2022-09-07T23:09:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:58.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:58.208012+0000) 2022-09-07T23:09:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:58.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:58.106669+0000) 2022-09-07T23:09:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:09:59.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:59.208212+0000) 2022-09-07T23:09:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:09:59.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:09:59.106892+0000) 2022-09-07T23:09:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:59 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-07T23:09:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:09:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:09:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:59 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-07T23:09:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:09:59.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:09:59.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:09:59.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:09:59 smithi055 ceph-mon[97033]: pgmap v2892: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:09:59.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:59 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-07T23:09:59.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:09:59.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:09:59.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:59 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-07T23:09:59.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:09:59.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:09:59.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:09:59.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:09:59 smithi059 ceph-mon[97450]: pgmap v2892: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:00.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:00.208476+0000) 2022-09-07T23:10:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:00.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:00.107127+0000) 2022-09-07T23:10:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-07T23:10:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T23:10:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T23:10:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-07T23:10:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T23:10:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T23:10:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:01.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:01.208746+0000) 2022-09-07T23:10:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:01.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:01.107359+0000) 2022-09-07T23:10:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:01 smithi059 ceph-mon[97450]: pgmap v2893: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:10:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:01 smithi055 ceph-mon[97033]: pgmap v2893: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:10:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:02.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:02.209005+0000) 2022-09-07T23:10:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:02.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:02.107589+0000) 2022-09-07T23:10:02.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:10:02.627+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:10:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:02 smithi055 ceph-mon[97033]: pgmap v2894: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:02 smithi059 ceph-mon[97450]: pgmap v2894: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:03.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:03.209236+0000) 2022-09-07T23:10:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:03.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:03.107897+0000) 2022-09-07T23:10:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:04.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:04.209484+0000) 2022-09-07T23:10:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:04.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:04.108137+0000) 2022-09-07T23:10:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:10:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:10:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:10:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:10:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:05.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:05.209678+0000) 2022-09-07T23:10:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:05.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:05.108363+0000) 2022-09-07T23:10:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:05 smithi059 ceph-mon[97450]: pgmap v2895: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:05 smithi055 ceph-mon[97033]: pgmap v2895: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:06.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:06.209861+0000) 2022-09-07T23:10:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:06.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:06.108595+0000) 2022-09-07T23:10:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:06 smithi055 ceph-mon[97033]: pgmap v2896: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:10:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:06 smithi059 ceph-mon[97450]: pgmap v2896: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:10:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:07.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:07.210089+0000) 2022-09-07T23:10:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:07.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:07.108797+0000) 2022-09-07T23:10:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:08.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:08.210241+0000) 2022-09-07T23:10:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:08.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:08.109004+0000) 2022-09-07T23:10:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:09.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:09.210408+0000) 2022-09-07T23:10:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:09.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:09.109237+0000) 2022-09-07T23:10:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:09 smithi055 ceph-mon[97033]: osdmap e271: 12 total, 11 up, 11 in 2022-09-07T23:10:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:09 smithi055 ceph-mon[97033]: pgmap v2898: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:10:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:09 smithi059 ceph-mon[97450]: osdmap e271: 12 total, 11 up, 11 in 2022-09-07T23:10:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:09 smithi059 ceph-mon[97450]: pgmap v2898: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:10:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:10.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:10.210675+0000) 2022-09-07T23:10:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:10.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:10.109449+0000) 2022-09-07T23:10:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:10 smithi055 ceph-mon[97033]: pgmap v2899: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:10:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:10 smithi059 ceph-mon[97450]: pgmap v2899: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:10:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:11.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:11.210857+0000) 2022-09-07T23:10:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:11.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:11.109698+0000) 2022-09-07T23:10:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:12.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:12.211060+0000) 2022-09-07T23:10:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:12.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:12.109909+0000) 2022-09-07T23:10:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:12 smithi055 ceph-mon[97033]: pgmap v2900: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:10:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:12 smithi059 ceph-mon[97450]: pgmap v2900: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:10:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:13.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:13.211288+0000) 2022-09-07T23:10:13.417 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:13.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:13.110166+0000) 2022-09-07T23:10:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:14.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:14.110366+0000) 2022-09-07T23:10:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:14.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:14.211452+0000) 2022-09-07T23:10:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:14 smithi059 ceph-mon[97450]: osdmap e272: 12 total, 11 up, 11 in 2022-09-07T23:10:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:14 smithi055 ceph-mon[97033]: osdmap e272: 12 total, 11 up, 11 in 2022-09-07T23:10:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:15.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:15.211700+0000) 2022-09-07T23:10:15.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:15.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:15.110586+0000) 2022-09-07T23:10:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:15 smithi059 ceph-mon[97450]: pgmap v2902: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:10:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:15 smithi055 ceph-mon[97033]: pgmap v2902: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:10:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:16.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:16.211845+0000) 2022-09-07T23:10:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:16.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:16.110805+0000) 2022-09-07T23:10:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:16 smithi055 ceph-mon[97033]: pgmap v2903: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 240 B/s wr, 7 op/s 2022-09-07T23:10:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:16 smithi059 ceph-mon[97450]: pgmap v2903: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 240 B/s wr, 7 op/s 2022-09-07T23:10:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:17.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:17.212014+0000) 2022-09-07T23:10:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:17.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:17.111017+0000) 2022-09-07T23:10:17.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:10:17.642+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:10:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:18.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:18.212229+0000) 2022-09-07T23:10:18.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:18.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:18.111183+0000) 2022-09-07T23:10:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:19.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:19.212435+0000) 2022-09-07T23:10:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:19.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:19.111394+0000) 2022-09-07T23:10:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:19 smithi059 ceph-mon[97450]: pgmap v2904: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:10:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:19 smithi055 ceph-mon[97033]: pgmap v2904: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:10:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:20.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:20.212702+0000) 2022-09-07T23:10:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:20.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:20.111532+0000) 2022-09-07T23:10:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:20 smithi055 ceph-mon[97033]: pgmap v2905: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:10:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:20 smithi059 ceph-mon[97450]: pgmap v2905: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:10:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:21.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:21.212869+0000) 2022-09-07T23:10:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:21.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:21.111769+0000) 2022-09-07T23:10:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:22.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:22.213100+0000) 2022-09-07T23:10:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:22.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:22.111907+0000) 2022-09-07T23:10:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:22 smithi055 ceph-mon[97033]: pgmap v2906: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:10:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:22 smithi059 ceph-mon[97450]: pgmap v2906: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:10:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:23.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:23.213314+0000) 2022-09-07T23:10:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:23.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:23.112095+0000) 2022-09-07T23:10:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:24.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:24.213456+0000) 2022-09-07T23:10:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:24.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:24.112296+0000) 2022-09-07T23:10:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:24 smithi055 ceph-mon[97033]: pgmap v2907: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:10:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:24 smithi059 ceph-mon[97450]: pgmap v2907: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-07T23:10:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:25.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:25.213696+0000) 2022-09-07T23:10:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:25.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:25.112544+0000) 2022-09-07T23:10:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:26.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:26.213856+0000) 2022-09-07T23:10:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:26.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:26.112780+0000) 2022-09-07T23:10:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:26 smithi055 ceph-mon[97033]: pgmap v2908: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:10:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:26 smithi059 ceph-mon[97450]: pgmap v2908: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:10:27.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:27.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:27.214094+0000) 2022-09-07T23:10:27.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:27.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:27.113006+0000) 2022-09-07T23:10:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:28.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:28.214324+0000) 2022-09-07T23:10:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:28.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:28.113251+0000) 2022-09-07T23:10:29.410 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:29.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:29.214589+0000) 2022-09-07T23:10:29.412 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:29.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:29.113523+0000) 2022-09-07T23:10:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:29 smithi059 ceph-mon[97450]: pgmap v2909: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:29 smithi055 ceph-mon[97033]: pgmap v2909: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:30.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:30.113746+0000) 2022-09-07T23:10:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:30.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:30.214844+0000) 2022-09-07T23:10:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:30 smithi055 ceph-mon[97033]: pgmap v2910: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:10:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:30 smithi059 ceph-mon[97450]: pgmap v2910: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:10:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:31.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:31.215112+0000) 2022-09-07T23:10:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:31.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:31.113952+0000) 2022-09-07T23:10:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:32.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:32.114198+0000) 2022-09-07T23:10:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:32.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:32.215407+0000) 2022-09-07T23:10:33.115 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:10:32.654+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:10:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:32 smithi055 ceph-mon[97033]: pgmap v2911: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:32 smithi059 ceph-mon[97450]: pgmap v2911: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:33.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:33.215681+0000) 2022-09-07T23:10:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:33.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:33.114462+0000) 2022-09-07T23:10:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:34.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:34.215969+0000) 2022-09-07T23:10:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:34.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:34.114720+0000) 2022-09-07T23:10:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:34 smithi055 ceph-mon[97033]: pgmap v2912: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:34 smithi059 ceph-mon[97450]: pgmap v2912: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:35.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:35.216231+0000) 2022-09-07T23:10:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:35.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:35.114975+0000) 2022-09-07T23:10:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:36.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:36.216463+0000) 2022-09-07T23:10:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:36.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:36.115220+0000) 2022-09-07T23:10:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:36 smithi055 ceph-mon[97033]: pgmap v2913: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:10:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:36 smithi059 ceph-mon[97450]: pgmap v2913: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:10:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:37.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:37.216722+0000) 2022-09-07T23:10:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:37.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:37.115455+0000) 2022-09-07T23:10:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:38.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:38.216907+0000) 2022-09-07T23:10:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:38.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:38.115684+0000) 2022-09-07T23:10:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:39.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:39.217060+0000) 2022-09-07T23:10:39.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:39.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:39.115881+0000) 2022-09-07T23:10:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:39 smithi059 ceph-mon[97450]: pgmap v2914: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:39 smithi055 ceph-mon[97033]: pgmap v2914: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:40.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:40.217319+0000) 2022-09-07T23:10:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:40.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:40.116104+0000) 2022-09-07T23:10:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:40 smithi055 ceph-mon[97033]: pgmap v2915: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:10:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:40 smithi059 ceph-mon[97450]: pgmap v2915: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:10:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:41.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:41.217572+0000) 2022-09-07T23:10:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:41.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:41.116322+0000) 2022-09-07T23:10:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:42.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:42.217792+0000) 2022-09-07T23:10:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:42.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:42.116538+0000) 2022-09-07T23:10:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:42 smithi055 ceph-mon[97033]: pgmap v2916: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:42 smithi059 ceph-mon[97450]: pgmap v2916: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:43.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:43.217999+0000) 2022-09-07T23:10:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:43.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:43.116823+0000) 2022-09-07T23:10:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:44.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:44.218165+0000) 2022-09-07T23:10:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:44.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:44.116970+0000) 2022-09-07T23:10:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:44 smithi055 ceph-mon[97033]: pgmap v2917: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:44 smithi059 ceph-mon[97450]: pgmap v2917: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:45.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:45.218359+0000) 2022-09-07T23:10:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:45.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:45.117182+0000) 2022-09-07T23:10:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:46.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:46.218526+0000) 2022-09-07T23:10:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:46.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:46.117355+0000) 2022-09-07T23:10:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:46 smithi055 ceph-mon[97033]: pgmap v2918: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:10:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:46 smithi059 ceph-mon[97450]: pgmap v2918: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:10:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:47.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:47.218689+0000) 2022-09-07T23:10:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:47.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:47.117534+0000) 2022-09-07T23:10:48.119 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:10:47.670+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:10:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:48.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:48.218857+0000) 2022-09-07T23:10:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:48.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:48.117745+0000) 2022-09-07T23:10:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:49.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:49.218999+0000) 2022-09-07T23:10:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:49.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:49.117914+0000) 2022-09-07T23:10:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:49 smithi059 ceph-mon[97450]: pgmap v2919: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:49 smithi055 ceph-mon[97033]: pgmap v2919: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:50.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:50.219159+0000) 2022-09-07T23:10:50.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:50.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:50.118165+0000) 2022-09-07T23:10:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:50 smithi055 ceph-mon[97033]: pgmap v2920: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:10:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:50 smithi059 ceph-mon[97450]: pgmap v2920: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:10:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:51.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:51.219381+0000) 2022-09-07T23:10:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:51.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:51.118375+0000) 2022-09-07T23:10:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:52.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:52.219603+0000) 2022-09-07T23:10:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:52.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:52.118599+0000) 2022-09-07T23:10:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:52 smithi055 ceph-mon[97033]: pgmap v2921: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:52 smithi059 ceph-mon[97450]: pgmap v2921: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:53.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:53.219799+0000) 2022-09-07T23:10:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:53.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:53.118849+0000) 2022-09-07T23:10:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:54.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:54.219941+0000) 2022-09-07T23:10:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:54.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:54.119061+0000) 2022-09-07T23:10:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:54 smithi055 ceph-mon[97033]: pgmap v2922: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:54 smithi059 ceph-mon[97450]: pgmap v2922: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:10:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:55.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:55.220134+0000) 2022-09-07T23:10:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:55.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:55.119328+0000) 2022-09-07T23:10:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:56.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:56.220345+0000) 2022-09-07T23:10:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:56.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:56.119559+0000) 2022-09-07T23:10:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:56 smithi055 ceph-mon[97033]: pgmap v2923: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:10:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:56 smithi059 ceph-mon[97450]: pgmap v2923: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:10:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:57.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:57.220559+0000) 2022-09-07T23:10:57.446 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:57.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:57.119813+0000) 2022-09-07T23:10:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:58.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:58.220780+0000) 2022-09-07T23:10:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:58.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:58.119986+0000) 2022-09-07T23:10:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:10:59.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:10:59.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:10:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:10:59.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:10:59.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:10:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:10:59.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:59.221039+0000) 2022-09-07T23:10:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:10:59.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:10:59.120213+0000) 2022-09-07T23:11:00.045 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:10:59 smithi059 ceph-mon[97450]: pgmap v2924: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:11:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:10:59 smithi055 ceph-mon[97033]: pgmap v2924: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:11:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:00.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:00.120363+0000) 2022-09-07T23:11:00.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:00.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:00.221288+0000) 2022-09-07T23:11:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:00 smithi055 ceph-mon[97033]: pgmap v2925: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:11:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:00 smithi059 ceph-mon[97450]: pgmap v2925: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:11:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:01.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:01.120588+0000) 2022-09-07T23:11:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:01.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:01.221539+0000) 2022-09-07T23:11:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:02.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:02.221764+0000) 2022-09-07T23:11:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:02.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:02.120779+0000) 2022-09-07T23:11:03.122 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:11:02.679+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:11:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:02 smithi055 ceph-mon[97033]: pgmap v2926: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:11:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:02 smithi059 ceph-mon[97450]: pgmap v2926: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:11:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:03.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:03.221974+0000) 2022-09-07T23:11:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:03.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:03.121061+0000) 2022-09-07T23:11:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:04.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:04.121322+0000) 2022-09-07T23:11:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:04.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:04.222152+0000) 2022-09-07T23:11:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:11:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:11:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:11:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:11:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:05.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:05.222407+0000) 2022-09-07T23:11:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:05.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:05.121488+0000) 2022-09-07T23:11:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:05 smithi059 ceph-mon[97450]: pgmap v2927: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T23:11:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:05 smithi055 ceph-mon[97033]: pgmap v2927: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T23:11:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:06.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:06.222671+0000) 2022-09-07T23:11:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:06.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:06.121674+0000) 2022-09-07T23:11:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:06 smithi055 ceph-mon[97033]: pgmap v2928: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:11:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:06 smithi059 ceph-mon[97450]: pgmap v2928: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:11:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:07.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:07.222862+0000) 2022-09-07T23:11:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:07.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:07.121902+0000) 2022-09-07T23:11:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:08.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:08.122142+0000) 2022-09-07T23:11:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:08.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:08.223045+0000) 2022-09-07T23:11:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:09.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:09.122375+0000) 2022-09-07T23:11:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:09.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:09.223254+0000) 2022-09-07T23:11:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:09 smithi055 ceph-mon[97033]: osdmap e273: 12 total, 11 up, 11 in 2022-09-07T23:11:09.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:09 smithi055 ceph-mon[97033]: pgmap v2930: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:11:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:09 smithi059 ceph-mon[97450]: osdmap e273: 12 total, 11 up, 11 in 2022-09-07T23:11:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:09 smithi059 ceph-mon[97450]: pgmap v2930: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:11:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:10.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:10.122599+0000) 2022-09-07T23:11:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:10.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:10.223502+0000) 2022-09-07T23:11:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:10 smithi055 ceph-mon[97033]: pgmap v2931: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:11:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:10 smithi059 ceph-mon[97450]: pgmap v2931: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:11:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:11.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:11.223733+0000) 2022-09-07T23:11:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:11.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:11.122830+0000) 2022-09-07T23:11:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:12.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:12.223975+0000) 2022-09-07T23:11:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:12.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:12.123075+0000) 2022-09-07T23:11:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:12 smithi055 ceph-mon[97033]: pgmap v2932: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:11:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:12 smithi059 ceph-mon[97450]: pgmap v2932: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:11:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:13.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:13.224179+0000) 2022-09-07T23:11:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:13.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:13.123342+0000) 2022-09-07T23:11:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:14.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:14.224377+0000) 2022-09-07T23:11:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:14.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:14.123550+0000) 2022-09-07T23:11:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:14 smithi059 ceph-mon[97450]: osdmap e274: 12 total, 11 up, 11 in 2022-09-07T23:11:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:14 smithi055 ceph-mon[97033]: osdmap e274: 12 total, 11 up, 11 in 2022-09-07T23:11:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:15.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:15.123795+0000) 2022-09-07T23:11:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:15.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:15.224592+0000) 2022-09-07T23:11:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:15 smithi059 ceph-mon[97450]: pgmap v2934: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-07T23:11:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:15 smithi055 ceph-mon[97033]: pgmap v2934: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-07T23:11:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:16.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:16.224748+0000) 2022-09-07T23:11:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:16.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:16.123995+0000) 2022-09-07T23:11:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:16 smithi055 ceph-mon[97033]: pgmap v2935: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 239 B/s wr, 7 op/s 2022-09-07T23:11:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:16 smithi059 ceph-mon[97450]: pgmap v2935: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 239 B/s wr, 7 op/s 2022-09-07T23:11:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:17.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:17.224969+0000) 2022-09-07T23:11:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:17.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:17.124171+0000) 2022-09-07T23:11:18.125 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:11:17.689+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:11:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:18.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:18.225219+0000) 2022-09-07T23:11:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:18.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:18.124388+0000) 2022-09-07T23:11:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:19.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:19.225463+0000) 2022-09-07T23:11:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:19.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:19.124619+0000) 2022-09-07T23:11:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:19 smithi059 ceph-mon[97450]: pgmap v2936: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:11:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:19 smithi055 ceph-mon[97033]: pgmap v2936: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:11:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:20.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:20.124839+0000) 2022-09-07T23:11:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:20.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:20.225685+0000) 2022-09-07T23:11:21.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:20 smithi055 ceph-mon[97033]: pgmap v2937: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:11:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:20 smithi059 ceph-mon[97450]: pgmap v2937: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:11:21.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:21.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:21.125100+0000) 2022-09-07T23:11:21.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:21.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:21.225813+0000) 2022-09-07T23:11:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:22.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:22.125286+0000) 2022-09-07T23:11:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:22.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:22.226022+0000) 2022-09-07T23:11:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:22 smithi055 ceph-mon[97033]: pgmap v2938: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:11:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:22 smithi059 ceph-mon[97450]: pgmap v2938: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:11:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:23.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:23.125453+0000) 2022-09-07T23:11:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:23.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:23.226270+0000) 2022-09-07T23:11:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:24.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:24.125609+0000) 2022-09-07T23:11:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:24.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:24.226508+0000) 2022-09-07T23:11:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:24 smithi055 ceph-mon[97033]: pgmap v2939: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 92 B/s wr, 6 op/s 2022-09-07T23:11:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:24 smithi059 ceph-mon[97450]: pgmap v2939: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 92 B/s wr, 6 op/s 2022-09-07T23:11:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:25.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:25.226742+0000) 2022-09-07T23:11:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:25.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:25.125892+0000) 2022-09-07T23:11:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:26.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:26.226894+0000) 2022-09-07T23:11:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:26.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:26.126100+0000) 2022-09-07T23:11:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:26 smithi055 ceph-mon[97033]: pgmap v2940: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-07T23:11:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:26 smithi059 ceph-mon[97450]: pgmap v2940: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-07T23:11:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:27.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:27.227131+0000) 2022-09-07T23:11:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:27.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:27.126369+0000) 2022-09-07T23:11:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:28.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:28.227319+0000) 2022-09-07T23:11:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:28.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:28.126605+0000) 2022-09-07T23:11:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:29.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:29.227614+0000) 2022-09-07T23:11:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:29.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:29.126872+0000) 2022-09-07T23:11:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:29 smithi055 ceph-mon[97033]: pgmap v2941: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:11:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:11:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:29 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-07T23:11:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:29 smithi059 ceph-mon[97450]: pgmap v2941: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:11:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:11:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:29 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-07T23:11:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:30.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:30.127086+0000) 2022-09-07T23:11:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:30.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:30.227882+0000) 2022-09-07T23:11:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:31.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:31.228080+0000) 2022-09-07T23:11:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:31.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:31.127319+0000) 2022-09-07T23:11:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:31 smithi055 ceph-mon[97033]: pgmap v2942: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-07T23:11:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:31 smithi059 ceph-mon[97450]: pgmap v2942: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-07T23:11:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:32.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:32.228346+0000) 2022-09-07T23:11:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:32.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:32.127513+0000) 2022-09-07T23:11:33.129 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:11:32.702+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:11:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:33.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:33.228554+0000) 2022-09-07T23:11:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:33.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:33.127735+0000) 2022-09-07T23:11:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:33 smithi055 ceph-mon[97033]: pgmap v2943: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:11:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:33 smithi059 ceph-mon[97450]: pgmap v2943: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:11:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:34.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:34.228775+0000) 2022-09-07T23:11:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:34.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:34.127869+0000) 2022-09-07T23:11:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:35.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:35.229051+0000) 2022-09-07T23:11:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:35.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:35.128042+0000) 2022-09-07T23:11:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:35 smithi055 ceph-mon[97033]: pgmap v2944: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:11:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:35 smithi059 ceph-mon[97450]: pgmap v2944: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:11:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:36.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:36.229321+0000) 2022-09-07T23:11:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:36.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:36.128236+0000) 2022-09-07T23:11:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:37.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:37.229586+0000) 2022-09-07T23:11:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:37.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:37.128484+0000) 2022-09-07T23:11:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:37 smithi055 ceph-mon[97033]: pgmap v2945: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-07T23:11:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:37 smithi059 ceph-mon[97450]: pgmap v2945: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-07T23:11:38.408 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:38.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:38.229851+0000) 2022-09-07T23:11:38.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:38.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:38.128724+0000) 2022-09-07T23:11:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:39.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:39.230077+0000) 2022-09-07T23:11:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:39.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:39.128984+0000) 2022-09-07T23:11:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:39 smithi059 ceph-mon[97450]: pgmap v2946: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:11:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:39 smithi055 ceph-mon[97033]: pgmap v2946: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:11:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:40.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:40.230330+0000) 2022-09-07T23:11:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:40.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:40.129154+0000) 2022-09-07T23:11:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:40 smithi055 ceph-mon[97033]: pgmap v2947: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:11:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:40 smithi059 ceph-mon[97450]: pgmap v2947: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:11:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:41.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:41.230582+0000) 2022-09-07T23:11:41.448 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:41.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:41.129315+0000) 2022-09-07T23:11:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:42.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:42.230863+0000) 2022-09-07T23:11:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:42.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:42.129527+0000) 2022-09-07T23:11:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:42 smithi055 ceph-mon[97033]: pgmap v2948: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:11:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:42 smithi059 ceph-mon[97450]: pgmap v2948: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:11:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:43.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:43.231040+0000) 2022-09-07T23:11:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:43.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:43.129794+0000) 2022-09-07T23:11:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:44.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:44.231282+0000) 2022-09-07T23:11:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:44.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:44.130019+0000) 2022-09-07T23:11:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:44 smithi055 ceph-mon[97033]: pgmap v2949: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:11:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:44 smithi059 ceph-mon[97450]: pgmap v2949: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:11:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:45.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:45.231472+0000) 2022-09-07T23:11:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:45.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:45.130234+0000) 2022-09-07T23:11:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:46.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:46.231728+0000) 2022-09-07T23:11:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:46.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:46.130418+0000) 2022-09-07T23:11:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:46 smithi055 ceph-mon[97033]: pgmap v2950: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:11:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:46 smithi059 ceph-mon[97450]: pgmap v2950: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:11:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:47.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:47.231949+0000) 2022-09-07T23:11:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:47.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:47.130711+0000) 2022-09-07T23:11:48.132 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:11:47.712+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:11:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:48.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:48.232148+0000) 2022-09-07T23:11:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:48.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:48.130961+0000) 2022-09-07T23:11:49.234 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:49.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:49.131202+0000) 2022-09-07T23:11:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:49.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:49.232381+0000) 2022-09-07T23:11:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:49 smithi055 ceph-mon[97033]: pgmap v2951: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:11:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:49 smithi059 ceph-mon[97450]: pgmap v2951: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:11:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:50.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:50.232613+0000) 2022-09-07T23:11:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:50.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:50.131434+0000) 2022-09-07T23:11:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:50 smithi055 ceph-mon[97033]: pgmap v2952: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:11:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:50 smithi059 ceph-mon[97450]: pgmap v2952: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:11:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:51.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:51.232869+0000) 2022-09-07T23:11:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:51.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:51.131722+0000) 2022-09-07T23:11:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:52.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:52.233070+0000) 2022-09-07T23:11:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:52.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:52.131964+0000) 2022-09-07T23:11:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:52 smithi055 ceph-mon[97033]: pgmap v2953: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:11:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:52 smithi059 ceph-mon[97450]: pgmap v2953: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:11:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:53.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:53.233302+0000) 2022-09-07T23:11:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:53.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:53.132212+0000) 2022-09-07T23:11:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:54.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:54.233546+0000) 2022-09-07T23:11:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:54.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:54.132432+0000) 2022-09-07T23:11:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:54 smithi055 ceph-mon[97033]: pgmap v2954: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:11:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:54 smithi059 ceph-mon[97450]: pgmap v2954: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:11:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:55.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:55.233821+0000) 2022-09-07T23:11:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:55.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:55.132633+0000) 2022-09-07T23:11:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:56.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:56.234045+0000) 2022-09-07T23:11:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:56.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:56.132870+0000) 2022-09-07T23:11:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:56 smithi055 ceph-mon[97033]: pgmap v2955: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:11:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:56 smithi059 ceph-mon[97450]: pgmap v2955: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:11:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:57.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:57.234250+0000) 2022-09-07T23:11:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:57.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:57.133034+0000) 2022-09-07T23:11:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:58.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:58.234465+0000) 2022-09-07T23:11:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:58.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:58.133271+0000) 2022-09-07T23:11:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:11:59.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:59.234676+0000) 2022-09-07T23:11:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:11:59.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:11:59.133499+0000) 2022-09-07T23:12:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:11:59 smithi055 ceph-mon[97033]: pgmap v2956: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:11:59 smithi059 ceph-mon[97450]: pgmap v2956: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:00.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:00.234873+0000) 2022-09-07T23:12:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:00.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:00.133746+0000) 2022-09-07T23:12:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:00 smithi055 ceph-mon[97033]: pgmap v2957: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:00 smithi059 ceph-mon[97450]: pgmap v2957: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:01.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:01.235068+0000) 2022-09-07T23:12:01.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:01.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:01.133869+0000) 2022-09-07T23:12:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:02.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:02.235250+0000) 2022-09-07T23:12:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:02.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:02.134069+0000) 2022-09-07T23:12:03.136 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:12:02.719+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:12:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:02 smithi055 ceph-mon[97033]: pgmap v2958: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:02 smithi059 ceph-mon[97450]: pgmap v2958: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:03.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:03.235480+0000) 2022-09-07T23:12:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:03.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:03.134300+0000) 2022-09-07T23:12:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:04.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:04.235724+0000) 2022-09-07T23:12:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:04.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:04.134535+0000) 2022-09-07T23:12:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:12:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:12:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:12:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:12:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:05.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:05.235951+0000) 2022-09-07T23:12:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:05.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:05.134747+0000) 2022-09-07T23:12:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:05 smithi055 ceph-mon[97033]: pgmap v2959: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T23:12:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:05 smithi059 ceph-mon[97450]: pgmap v2959: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T23:12:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:06.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:06.236117+0000) 2022-09-07T23:12:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:06.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:06.134922+0000) 2022-09-07T23:12:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:06 smithi055 ceph-mon[97033]: pgmap v2960: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T23:12:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:06 smithi059 ceph-mon[97450]: pgmap v2960: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T23:12:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:07.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:07.236258+0000) 2022-09-07T23:12:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:07.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:07.135116+0000) 2022-09-07T23:12:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:08.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:08.236464+0000) 2022-09-07T23:12:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:08.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:08.135289+0000) 2022-09-07T23:12:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:09.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:09.236659+0000) 2022-09-07T23:12:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:09.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:09.135447+0000) 2022-09-07T23:12:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:09 smithi055 ceph-mon[97033]: osdmap e275: 12 total, 11 up, 11 in 2022-09-07T23:12:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:09 smithi055 ceph-mon[97033]: pgmap v2962: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:12:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:09 smithi059 ceph-mon[97450]: osdmap e275: 12 total, 11 up, 11 in 2022-09-07T23:12:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:09 smithi059 ceph-mon[97450]: pgmap v2962: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:12:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:10.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:10.236833+0000) 2022-09-07T23:12:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:10.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:10.135688+0000) 2022-09-07T23:12:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:10 smithi055 ceph-mon[97033]: pgmap v2963: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:12:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:10 smithi059 ceph-mon[97450]: pgmap v2963: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:12:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:11.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:11.236999+0000) 2022-09-07T23:12:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:11.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:11.135922+0000) 2022-09-07T23:12:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:12.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:12.237231+0000) 2022-09-07T23:12:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:12.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:12.136165+0000) 2022-09-07T23:12:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:12 smithi055 ceph-mon[97033]: pgmap v2964: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:12:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:12 smithi059 ceph-mon[97450]: pgmap v2964: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:12:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:13.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:13.237466+0000) 2022-09-07T23:12:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:13.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:13.136393+0000) 2022-09-07T23:12:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:14.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:14.237692+0000) 2022-09-07T23:12:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:14.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:14.136667+0000) 2022-09-07T23:12:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:14 smithi059 ceph-mon[97450]: osdmap e276: 12 total, 11 up, 11 in 2022-09-07T23:12:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:14 smithi055 ceph-mon[97033]: osdmap e276: 12 total, 11 up, 11 in 2022-09-07T23:12:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:15.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:15.237955+0000) 2022-09-07T23:12:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:15.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:15.136867+0000) 2022-09-07T23:12:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:15 smithi055 ceph-mon[97033]: pgmap v2966: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.1 KiB/s rd, 255 B/s wr, 4 op/s 2022-09-07T23:12:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:15 smithi059 ceph-mon[97450]: pgmap v2966: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.1 KiB/s rd, 255 B/s wr, 4 op/s 2022-09-07T23:12:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:16.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:16.238203+0000) 2022-09-07T23:12:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:16.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:16.137051+0000) 2022-09-07T23:12:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:16 smithi055 ceph-mon[97033]: pgmap v2967: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 239 B/s wr, 7 op/s 2022-09-07T23:12:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:16 smithi059 ceph-mon[97450]: pgmap v2967: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 239 B/s wr, 7 op/s 2022-09-07T23:12:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:17.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:17.238413+0000) 2022-09-07T23:12:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:17.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:17.137225+0000) 2022-09-07T23:12:18.139 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:12:17.736+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:12:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:18.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:18.238596+0000) 2022-09-07T23:12:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:18.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:18.137375+0000) 2022-09-07T23:12:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:19.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:19.238842+0000) 2022-09-07T23:12:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:19.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:19.137589+0000) 2022-09-07T23:12:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:19 smithi055 ceph-mon[97033]: pgmap v2968: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:12:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:19 smithi059 ceph-mon[97450]: pgmap v2968: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:12:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:20.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:20.238997+0000) 2022-09-07T23:12:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:20.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:20.137751+0000) 2022-09-07T23:12:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:20 smithi055 ceph-mon[97033]: pgmap v2969: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:12:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:20 smithi059 ceph-mon[97450]: pgmap v2969: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:12:21.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:21.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:21.239184+0000) 2022-09-07T23:12:21.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:21.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:21.137921+0000) 2022-09-07T23:12:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:22.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:22.239344+0000) 2022-09-07T23:12:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:22.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:22.138085+0000) 2022-09-07T23:12:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:22 smithi055 ceph-mon[97033]: pgmap v2970: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:12:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:22 smithi059 ceph-mon[97450]: pgmap v2970: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:12:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:23.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:23.239566+0000) 2022-09-07T23:12:23.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:23.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:23.138314+0000) 2022-09-07T23:12:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:24.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:24.138551+0000) 2022-09-07T23:12:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:24.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:24.239775+0000) 2022-09-07T23:12:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:24 smithi055 ceph-mon[97033]: pgmap v2971: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 6 op/s 2022-09-07T23:12:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:24 smithi059 ceph-mon[97450]: pgmap v2971: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 6 op/s 2022-09-07T23:12:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:25.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:25.239955+0000) 2022-09-07T23:12:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:25.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:25.138699+0000) 2022-09-07T23:12:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:26.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:26.240115+0000) 2022-09-07T23:12:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:26.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:26.138835+0000) 2022-09-07T23:12:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:26 smithi055 ceph-mon[97033]: pgmap v2972: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:26 smithi059 ceph-mon[97450]: pgmap v2972: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:27.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:27.240339+0000) 2022-09-07T23:12:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:27.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:27.139007+0000) 2022-09-07T23:12:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:28.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:28.240541+0000) 2022-09-07T23:12:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:28.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:28.139212+0000) 2022-09-07T23:12:29.242 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:29.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:29.139433+0000) 2022-09-07T23:12:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:29.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:29.240818+0000) 2022-09-07T23:12:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:29 smithi055 ceph-mon[97033]: pgmap v2973: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:12:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:12:29.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:12:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:29 smithi059 ceph-mon[97450]: pgmap v2973: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:12:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:12:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:12:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:30.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:30.241068+0000) 2022-09-07T23:12:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:30.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:30.139628+0000) 2022-09-07T23:12:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:12:31.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:12:31.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:12:31.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:12:31.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:30 smithi055 ceph-mon[97033]: pgmap v2974: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:12:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:12:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:12:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:12:31.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:30 smithi059 ceph-mon[97450]: pgmap v2974: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:31.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:31.241305+0000) 2022-09-07T23:12:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:31.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:31.139772+0000) 2022-09-07T23:12:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:32.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:32.241500+0000) 2022-09-07T23:12:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:32.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:32.139942+0000) 2022-09-07T23:12:33.141 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:12:32.751+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:12:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:32 smithi055 ceph-mon[97033]: pgmap v2975: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:32 smithi059 ceph-mon[97450]: pgmap v2975: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:33.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:33.241770+0000) 2022-09-07T23:12:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:33.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:33.140140+0000) 2022-09-07T23:12:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:34.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:34.242061+0000) 2022-09-07T23:12:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:34.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:34.140336+0000) 2022-09-07T23:12:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:34 smithi055 ceph-mon[97033]: pgmap v2976: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:12:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:34 smithi059 ceph-mon[97450]: pgmap v2976: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:12:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:35.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:35.140516+0000) 2022-09-07T23:12:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:35.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:35.242284+0000) 2022-09-07T23:12:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:36.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:36.140728+0000) 2022-09-07T23:12:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:36.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:36.242485+0000) 2022-09-07T23:12:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:36 smithi055 ceph-mon[97033]: pgmap v2977: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:36 smithi059 ceph-mon[97450]: pgmap v2977: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:37.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:37.242732+0000) 2022-09-07T23:12:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:37.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:37.140900+0000) 2022-09-07T23:12:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:38.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:38.242963+0000) 2022-09-07T23:12:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:38.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:38.141061+0000) 2022-09-07T23:12:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:39.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:39.243221+0000) 2022-09-07T23:12:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:39.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:39.141191+0000) 2022-09-07T23:12:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:39 smithi055 ceph-mon[97033]: pgmap v2978: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:39 smithi059 ceph-mon[97450]: pgmap v2978: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:40.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:40.243464+0000) 2022-09-07T23:12:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:40.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:40.141364+0000) 2022-09-07T23:12:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:40 smithi055 ceph-mon[97033]: pgmap v2979: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:40 smithi059 ceph-mon[97450]: pgmap v2979: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:41.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:41.243720+0000) 2022-09-07T23:12:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:41.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:41.141587+0000) 2022-09-07T23:12:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:42.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:42.243935+0000) 2022-09-07T23:12:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:42.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:42.141777+0000) 2022-09-07T23:12:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:42 smithi055 ceph-mon[97033]: pgmap v2980: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:43.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:42 smithi059 ceph-mon[97450]: pgmap v2980: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:43.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:43.244167+0000) 2022-09-07T23:12:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:43.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:43.141974+0000) 2022-09-07T23:12:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:44.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:44.244383+0000) 2022-09-07T23:12:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:44.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:44.142178+0000) 2022-09-07T23:12:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:44 smithi055 ceph-mon[97033]: pgmap v2981: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:44 smithi059 ceph-mon[97450]: pgmap v2981: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:45.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:45.244578+0000) 2022-09-07T23:12:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:45.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:45.142332+0000) 2022-09-07T23:12:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:46.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:46.244870+0000) 2022-09-07T23:12:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:46.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:46.142491+0000) 2022-09-07T23:12:47.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:46 smithi055 ceph-mon[97033]: pgmap v2982: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-07T23:12:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:46 smithi059 ceph-mon[97450]: pgmap v2982: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-07T23:12:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:47.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:47.245137+0000) 2022-09-07T23:12:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:47.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:47.142668+0000) 2022-09-07T23:12:48.144 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:12:47.753+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:12:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:48.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:48.245367+0000) 2022-09-07T23:12:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:48.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:48.142870+0000) 2022-09-07T23:12:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:49.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:49.143051+0000) 2022-09-07T23:12:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:49.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:49.245601+0000) 2022-09-07T23:12:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:49 smithi055 ceph-mon[97033]: pgmap v2983: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:49 smithi059 ceph-mon[97450]: pgmap v2983: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:50.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:50.245863+0000) 2022-09-07T23:12:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:50.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:50.143252+0000) 2022-09-07T23:12:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:50 smithi055 ceph-mon[97033]: pgmap v2984: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:50 smithi059 ceph-mon[97450]: pgmap v2984: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:51.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:51.246098+0000) 2022-09-07T23:12:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:51.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:51.143435+0000) 2022-09-07T23:12:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:52.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:52.246369+0000) 2022-09-07T23:12:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:52.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:52.143577+0000) 2022-09-07T23:12:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:52 smithi055 ceph-mon[97033]: pgmap v2985: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:53.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:52 smithi059 ceph-mon[97450]: pgmap v2985: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:12:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:53.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:53.143735+0000) 2022-09-07T23:12:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:53.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:53.246608+0000) 2022-09-07T23:12:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:54.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:54.246861+0000) 2022-09-07T23:12:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:54.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:54.143915+0000) 2022-09-07T23:12:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:54 smithi055 ceph-mon[97033]: pgmap v2986: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:54 smithi059 ceph-mon[97450]: pgmap v2986: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:12:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:55.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:55.247071+0000) 2022-09-07T23:12:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:55.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:55.144054+0000) 2022-09-07T23:12:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:56.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:56.247317+0000) 2022-09-07T23:12:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:56.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:56.144234+0000) 2022-09-07T23:12:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:56 smithi055 ceph-mon[97033]: pgmap v2987: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:12:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:56 smithi059 ceph-mon[97450]: pgmap v2987: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:12:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:57.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:57.247531+0000) 2022-09-07T23:12:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:57.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:57.144387+0000) 2022-09-07T23:12:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:58.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:58.247755+0000) 2022-09-07T23:12:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:58.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:58.144575+0000) 2022-09-07T23:12:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:12:59.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:59.247928+0000) 2022-09-07T23:12:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:12:59.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:12:59.144806+0000) 2022-09-07T23:13:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:59 smithi055 ceph-mon[97033]: pgmap v2988: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:12:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:13:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:59 smithi059 ceph-mon[97450]: pgmap v2988: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:12:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:13:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:00.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:00.248153+0000) 2022-09-07T23:13:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:00.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:00.144936+0000) 2022-09-07T23:13:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:00 smithi055 ceph-mon[97033]: pgmap v2989: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:13:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:00 smithi055 ceph-mon[97033]: pgmap v2990: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:13:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:00 smithi059 ceph-mon[97450]: pgmap v2989: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:13:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:00 smithi059 ceph-mon[97450]: pgmap v2990: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:13:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:01.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:01.248395+0000) 2022-09-07T23:13:01.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:01.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:01.145078+0000) 2022-09-07T23:13:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:02.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:02.248600+0000) 2022-09-07T23:13:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:02.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:02.145284+0000) 2022-09-07T23:13:03.147 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:13:02.765+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:13:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:02 smithi055 ceph-mon[97033]: pgmap v2991: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:13:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:02 smithi059 ceph-mon[97450]: pgmap v2991: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:13:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:03.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:03.248799+0000) 2022-09-07T23:13:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:03.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:03.145473+0000) 2022-09-07T23:13:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:04.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:04.248996+0000) 2022-09-07T23:13:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:04.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:04.145672+0000) 2022-09-07T23:13:05.241 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:04 smithi055 ceph-mon[97033]: pgmap v2992: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-07T23:13:05.241 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:13:05.242 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:13:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:04 smithi059 ceph-mon[97450]: pgmap v2992: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-07T23:13:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:13:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:13:05.416 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:05.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:05.249282+0000) 2022-09-07T23:13:05.418 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:05.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:05.145833+0000) 2022-09-07T23:13:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:06.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:06.249567+0000) 2022-09-07T23:13:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:06.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:06.145994+0000) 2022-09-07T23:13:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:06 smithi055 ceph-mon[97033]: pgmap v2993: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-07T23:13:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:06 smithi059 ceph-mon[97450]: pgmap v2993: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-07T23:13:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:07.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:07.249780+0000) 2022-09-07T23:13:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:07.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:07.146195+0000) 2022-09-07T23:13:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:08.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:08.249907+0000) 2022-09-07T23:13:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:08.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:08.146414+0000) 2022-09-07T23:13:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:08 smithi055 ceph-mon[97033]: pgmap v2994: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-07T23:13:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:08 smithi059 ceph-mon[97450]: pgmap v2994: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-07T23:13:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:09.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:09.250129+0000) 2022-09-07T23:13:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:09.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:09.146576+0000) 2022-09-07T23:13:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:09 smithi055 ceph-mon[97033]: osdmap e277: 12 total, 11 up, 11 in 2022-09-07T23:13:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:09 smithi059 ceph-mon[97450]: osdmap e277: 12 total, 11 up, 11 in 2022-09-07T23:13:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:10.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:10.250300+0000) 2022-09-07T23:13:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:10.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:10.146723+0000) 2022-09-07T23:13:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:10 smithi055 ceph-mon[97033]: pgmap v2996: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:13:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:10 smithi059 ceph-mon[97450]: pgmap v2996: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:13:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:11.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:11.250527+0000) 2022-09-07T23:13:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:11.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:11.146964+0000) 2022-09-07T23:13:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:12.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:12.250732+0000) 2022-09-07T23:13:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:12.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:12.147208+0000) 2022-09-07T23:13:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:12 smithi055 ceph-mon[97033]: pgmap v2997: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:13:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:12 smithi059 ceph-mon[97450]: pgmap v2997: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:13:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:13.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:13.250965+0000) 2022-09-07T23:13:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:13.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:13.147399+0000) 2022-09-07T23:13:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:14.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:14.251134+0000) 2022-09-07T23:13:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:14.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:14.147575+0000) 2022-09-07T23:13:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:14 smithi055 ceph-mon[97033]: pgmap v2998: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:13:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:14 smithi055 ceph-mon[97033]: osdmap e278: 12 total, 11 up, 11 in 2022-09-07T23:13:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:14 smithi059 ceph-mon[97450]: pgmap v2998: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:13:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:14 smithi059 ceph-mon[97450]: osdmap e278: 12 total, 11 up, 11 in 2022-09-07T23:13:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:15.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:15.251348+0000) 2022-09-07T23:13:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:15.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:15.147736+0000) 2022-09-07T23:13:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:16.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:16.251568+0000) 2022-09-07T23:13:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:16.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:16.147951+0000) 2022-09-07T23:13:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:16 smithi055 ceph-mon[97033]: pgmap v3000: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:13:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:16 smithi059 ceph-mon[97450]: pgmap v3000: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:13:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:17.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:17.251801+0000) 2022-09-07T23:13:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:17.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:17.148188+0000) 2022-09-07T23:13:18.149 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:13:17.776+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:13:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:18.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:18.251983+0000) 2022-09-07T23:13:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:18.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:18.148403+0000) 2022-09-07T23:13:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:18 smithi055 ceph-mon[97033]: pgmap v3001: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 246 B/s wr, 7 op/s 2022-09-07T23:13:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:18 smithi059 ceph-mon[97450]: pgmap v3001: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 246 B/s wr, 7 op/s 2022-09-07T23:13:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:19.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:19.252173+0000) 2022-09-07T23:13:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:19.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:19.148671+0000) 2022-09-07T23:13:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:20.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:20.252418+0000) 2022-09-07T23:13:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:20.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:20.148874+0000) 2022-09-07T23:13:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:20 smithi055 ceph-mon[97033]: pgmap v3002: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:13:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:20 smithi059 ceph-mon[97450]: pgmap v3002: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:13:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:21.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:21.252681+0000) 2022-09-07T23:13:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:21.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:21.149104+0000) 2022-09-07T23:13:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:22.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:22.149326+0000) 2022-09-07T23:13:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:22.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:22.252929+0000) 2022-09-07T23:13:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:22 smithi055 ceph-mon[97033]: pgmap v3003: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:13:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:22 smithi059 ceph-mon[97450]: pgmap v3003: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:13:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:23.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:23.253158+0000) 2022-09-07T23:13:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:23.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:23.149558+0000) 2022-09-07T23:13:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:24.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:24.149777+0000) 2022-09-07T23:13:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:24.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:24.253344+0000) 2022-09-07T23:13:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:24 smithi055 ceph-mon[97033]: pgmap v3004: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:13:25.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:24 smithi059 ceph-mon[97450]: pgmap v3004: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:13:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:25.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:25.253558+0000) 2022-09-07T23:13:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:25.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:25.149939+0000) 2022-09-07T23:13:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:26.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:26.253746+0000) 2022-09-07T23:13:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:26.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:26.150080+0000) 2022-09-07T23:13:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:26 smithi055 ceph-mon[97033]: pgmap v3005: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2022-09-07T23:13:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:26 smithi059 ceph-mon[97450]: pgmap v3005: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2022-09-07T23:13:27.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:27.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:27.253961+0000) 2022-09-07T23:13:27.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:27.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:27.150289+0000) 2022-09-07T23:13:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:28.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:28.254134+0000) 2022-09-07T23:13:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:28.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:28.150496+0000) 2022-09-07T23:13:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:28 smithi055 ceph-mon[97033]: pgmap v3006: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:28 smithi059 ceph-mon[97450]: pgmap v3006: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:29.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:29.254370+0000) 2022-09-07T23:13:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:29.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:29.150736+0000) 2022-09-07T23:13:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:30.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:30.254559+0000) 2022-09-07T23:13:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:30.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:30.150890+0000) 2022-09-07T23:13:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:30 smithi055 ceph-mon[97033]: pgmap v3007: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:30 smithi059 ceph-mon[97450]: pgmap v3007: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:31.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:31.254720+0000) 2022-09-07T23:13:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:31.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:31.151032+0000) 2022-09-07T23:13:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:32.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:32.254941+0000) 2022-09-07T23:13:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:32.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:32.151250+0000) 2022-09-07T23:13:33.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:13:32.777+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:13:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:32 smithi055 ceph-mon[97033]: pgmap v3008: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:13:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:32 smithi059 ceph-mon[97450]: pgmap v3008: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:13:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:33.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:33.151536+0000) 2022-09-07T23:13:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:33.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:33.255161+0000) 2022-09-07T23:13:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:34.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:34.151685+0000) 2022-09-07T23:13:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:34.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:34.255447+0000) 2022-09-07T23:13:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:34 smithi055 ceph-mon[97033]: pgmap v3009: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:34 smithi059 ceph-mon[97450]: pgmap v3009: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:35.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:35.151823+0000) 2022-09-07T23:13:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:35.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:35.255704+0000) 2022-09-07T23:13:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:36.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:36.255936+0000) 2022-09-07T23:13:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:36.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:36.152014+0000) 2022-09-07T23:13:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:36 smithi055 ceph-mon[97033]: pgmap v3010: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:13:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:36 smithi059 ceph-mon[97450]: pgmap v3010: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:13:37.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:37.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:37.256122+0000) 2022-09-07T23:13:37.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:37.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:37.152250+0000) 2022-09-07T23:13:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:38.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:38.256357+0000) 2022-09-07T23:13:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:38.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:38.152482+0000) 2022-09-07T23:13:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:38 smithi055 ceph-mon[97033]: pgmap v3011: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:38 smithi059 ceph-mon[97450]: pgmap v3011: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:39.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:39.256556+0000) 2022-09-07T23:13:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:39.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:39.152728+0000) 2022-09-07T23:13:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:40.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:40.256756+0000) 2022-09-07T23:13:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:40.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:40.152905+0000) 2022-09-07T23:13:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:40 smithi055 ceph-mon[97033]: pgmap v3012: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:40 smithi059 ceph-mon[97450]: pgmap v3012: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:41.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:41.256881+0000) 2022-09-07T23:13:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:41.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:41.153154+0000) 2022-09-07T23:13:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:42.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:42.257084+0000) 2022-09-07T23:13:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:42.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:42.153295+0000) 2022-09-07T23:13:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:42 smithi055 ceph-mon[97033]: pgmap v3013: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:13:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:42 smithi059 ceph-mon[97450]: pgmap v3013: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:13:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:43.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:43.257316+0000) 2022-09-07T23:13:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:43.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:43.153527+0000) 2022-09-07T23:13:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:44.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:44.257518+0000) 2022-09-07T23:13:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:44.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:44.153724+0000) 2022-09-07T23:13:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:44 smithi055 ceph-mon[97033]: pgmap v3014: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:44 smithi059 ceph-mon[97450]: pgmap v3014: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:45.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:45.257738+0000) 2022-09-07T23:13:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:45.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:45.153954+0000) 2022-09-07T23:13:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:46.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:46.257905+0000) 2022-09-07T23:13:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:46.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:46.154115+0000) 2022-09-07T23:13:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:46 smithi055 ceph-mon[97033]: pgmap v3015: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:13:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:46 smithi059 ceph-mon[97450]: pgmap v3015: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:13:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:47.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:47.258074+0000) 2022-09-07T23:13:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:47.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:47.154266+0000) 2022-09-07T23:13:48.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:13:47.781+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:13:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:48.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:48.258270+0000) 2022-09-07T23:13:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:48.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:48.154428+0000) 2022-09-07T23:13:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:48 smithi055 ceph-mon[97033]: pgmap v3016: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:48 smithi059 ceph-mon[97450]: pgmap v3016: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:49.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:49.258523+0000) 2022-09-07T23:13:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:49.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:49.154590+0000) 2022-09-07T23:13:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:50.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:50.258729+0000) 2022-09-07T23:13:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:50.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:50.154816+0000) 2022-09-07T23:13:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:50 smithi055 ceph-mon[97033]: pgmap v3017: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:50 smithi059 ceph-mon[97450]: pgmap v3017: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:51.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:51.258855+0000) 2022-09-07T23:13:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:51.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:51.155039+0000) 2022-09-07T23:13:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:52.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:52.259136+0000) 2022-09-07T23:13:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:52.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:52.155198+0000) 2022-09-07T23:13:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:52 smithi055 ceph-mon[97033]: pgmap v3018: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:13:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:52 smithi059 ceph-mon[97450]: pgmap v3018: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:13:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:53.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:53.259383+0000) 2022-09-07T23:13:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:53.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:53.155431+0000) 2022-09-07T23:13:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:54.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:54.259582+0000) 2022-09-07T23:13:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:54.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:54.155624+0000) 2022-09-07T23:13:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:54 smithi055 ceph-mon[97033]: pgmap v3019: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:54 smithi059 ceph-mon[97450]: pgmap v3019: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:55.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:55.259831+0000) 2022-09-07T23:13:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:55.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:55.155876+0000) 2022-09-07T23:13:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:56.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:56.260028+0000) 2022-09-07T23:13:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:56.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:56.156058+0000) 2022-09-07T23:13:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:56 smithi055 ceph-mon[97033]: pgmap v3020: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:13:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:56 smithi059 ceph-mon[97450]: pgmap v3020: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:13:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:57.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:57.260187+0000) 2022-09-07T23:13:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:57.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:57.156236+0000) 2022-09-07T23:13:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:58.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:58.260321+0000) 2022-09-07T23:13:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:58.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:58.156434+0000) 2022-09-07T23:13:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:58 smithi055 ceph-mon[97033]: pgmap v3021: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:58 smithi059 ceph-mon[97450]: pgmap v3021: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:13:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:13:59.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:59.260472+0000) 2022-09-07T23:13:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:13:59.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:13:59.156597+0000) 2022-09-07T23:14:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:14:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:14:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:13:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:14:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:14:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:14:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:13:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:14:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:00.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:00.260665+0000) 2022-09-07T23:14:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:00.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:00.156831+0000) 2022-09-07T23:14:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:00 smithi055 ceph-mon[97033]: pgmap v3022: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:00 smithi059 ceph-mon[97450]: pgmap v3022: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:01.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:01.260890+0000) 2022-09-07T23:14:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:01.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:01.157066+0000) 2022-09-07T23:14:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:02.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:02.157307+0000) 2022-09-07T23:14:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:02.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:02.261030+0000) 2022-09-07T23:14:03.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:14:02.788+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:14:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:02 smithi055 ceph-mon[97033]: pgmap v3023: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:14:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:02 smithi059 ceph-mon[97450]: pgmap v3023: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:14:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:03.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:03.261168+0000) 2022-09-07T23:14:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:03.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:03.157493+0000) 2022-09-07T23:14:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:04.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:04.261327+0000) 2022-09-07T23:14:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:04.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:04.157747+0000) 2022-09-07T23:14:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:04 smithi055 ceph-mon[97033]: pgmap v3024: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:14:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:14:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:04 smithi059 ceph-mon[97450]: pgmap v3024: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:05.325 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:14:05.325 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:14:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:05.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:05.261456+0000) 2022-09-07T23:14:05.685 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:05.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:05.157938+0000) 2022-09-07T23:14:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:06.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:06.261665+0000) 2022-09-07T23:14:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:06.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:06.158156+0000) 2022-09-07T23:14:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:06 smithi055 ceph-mon[97033]: pgmap v3025: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:14:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:06 smithi059 ceph-mon[97450]: pgmap v3025: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:14:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:07.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:07.261843+0000) 2022-09-07T23:14:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:07.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:07.158354+0000) 2022-09-07T23:14:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:08.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:08.261995+0000) 2022-09-07T23:14:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:08.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:08.158593+0000) 2022-09-07T23:14:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:08 smithi055 ceph-mon[97033]: pgmap v3026: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:14:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:08 smithi059 ceph-mon[97450]: pgmap v3026: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:14:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:09.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:09.262128+0000) 2022-09-07T23:14:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:09.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:09.158720+0000) 2022-09-07T23:14:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:09 smithi055 ceph-mon[97033]: osdmap e279: 12 total, 11 up, 11 in 2022-09-07T23:14:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:09 smithi059 ceph-mon[97450]: osdmap e279: 12 total, 11 up, 11 in 2022-09-07T23:14:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:10.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:10.262290+0000) 2022-09-07T23:14:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:10.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:10.158954+0000) 2022-09-07T23:14:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:10 smithi055 ceph-mon[97033]: pgmap v3028: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:14:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:10 smithi059 ceph-mon[97450]: pgmap v3028: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:14:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:11.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:11.262417+0000) 2022-09-07T23:14:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:11.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:11.159164+0000) 2022-09-07T23:14:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:12.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:12.262536+0000) 2022-09-07T23:14:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:12.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:12.159387+0000) 2022-09-07T23:14:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:12 smithi055 ceph-mon[97033]: pgmap v3029: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:14:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:12 smithi059 ceph-mon[97450]: pgmap v3029: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:14:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:13.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:13.262692+0000) 2022-09-07T23:14:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:13.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:13.159585+0000) 2022-09-07T23:14:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:14.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:14.262813+0000) 2022-09-07T23:14:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:14.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:14.159759+0000) 2022-09-07T23:14:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:14 smithi055 ceph-mon[97033]: pgmap v3030: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:14:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:14 smithi055 ceph-mon[97033]: osdmap e280: 12 total, 11 up, 11 in 2022-09-07T23:14:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:14 smithi059 ceph-mon[97450]: pgmap v3030: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:14:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:14 smithi059 ceph-mon[97450]: osdmap e280: 12 total, 11 up, 11 in 2022-09-07T23:14:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:15.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:15.262968+0000) 2022-09-07T23:14:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:15.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:15.159992+0000) 2022-09-07T23:14:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:16.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:16.263096+0000) 2022-09-07T23:14:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:16.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:16.160238+0000) 2022-09-07T23:14:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:16 smithi055 ceph-mon[97033]: pgmap v3032: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:14:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:16 smithi059 ceph-mon[97450]: pgmap v3032: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:14:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:17.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:17.263218+0000) 2022-09-07T23:14:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:17.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:17.160469+0000) 2022-09-07T23:14:18.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:14:17.797+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:14:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:18.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:18.263358+0000) 2022-09-07T23:14:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:18.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:18.160714+0000) 2022-09-07T23:14:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:18 smithi055 ceph-mon[97033]: pgmap v3033: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-07T23:14:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:18 smithi059 ceph-mon[97450]: pgmap v3033: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-07T23:14:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:19.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:19.263497+0000) 2022-09-07T23:14:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:19.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:19.160934+0000) 2022-09-07T23:14:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:20.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:20.263741+0000) 2022-09-07T23:14:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:20.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:20.161135+0000) 2022-09-07T23:14:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:20 smithi055 ceph-mon[97033]: pgmap v3034: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:14:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:20 smithi059 ceph-mon[97450]: pgmap v3034: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:14:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:21.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:21.263956+0000) 2022-09-07T23:14:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:21.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:21.161362+0000) 2022-09-07T23:14:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:22.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:22.264183+0000) 2022-09-07T23:14:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:22.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:22.161571+0000) 2022-09-07T23:14:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:22 smithi055 ceph-mon[97033]: pgmap v3035: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:14:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:22 smithi059 ceph-mon[97450]: pgmap v3035: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:14:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:23.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:23.264422+0000) 2022-09-07T23:14:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:23.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:23.161756+0000) 2022-09-07T23:14:24.436 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:24.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:24.264673+0000) 2022-09-07T23:14:24.437 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:24.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:24.161994+0000) 2022-09-07T23:14:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:24 smithi055 ceph-mon[97033]: pgmap v3036: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:14:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:24 smithi059 ceph-mon[97450]: pgmap v3036: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:14:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:25.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:25.264896+0000) 2022-09-07T23:14:25.666 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:25.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:25.162260+0000) 2022-09-07T23:14:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:26.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:26.265114+0000) 2022-09-07T23:14:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:26.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:26.162525+0000) 2022-09-07T23:14:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:26 smithi055 ceph-mon[97033]: pgmap v3037: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2022-09-07T23:14:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:26 smithi059 ceph-mon[97450]: pgmap v3037: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2022-09-07T23:14:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:27.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:27.265252+0000) 2022-09-07T23:14:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:27.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:27.162769+0000) 2022-09-07T23:14:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:28.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:28.265440+0000) 2022-09-07T23:14:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:28.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:28.163005+0000) 2022-09-07T23:14:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:28 smithi055 ceph-mon[97033]: pgmap v3038: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:28 smithi059 ceph-mon[97450]: pgmap v3038: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:29.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:29.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:29.265657+0000) 2022-09-07T23:14:29.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:29.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:29.163174+0000) 2022-09-07T23:14:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:30.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:30.265863+0000) 2022-09-07T23:14:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:30.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:30.163380+0000) 2022-09-07T23:14:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:30 smithi055 ceph-mon[97033]: pgmap v3039: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:30 smithi055 ceph-mon[97033]: pgmap v3040: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:14:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:30 smithi055 ceph-mon[97033]: pgmap v3041: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:14:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:14:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:30 smithi059 ceph-mon[97450]: pgmap v3039: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:30 smithi059 ceph-mon[97450]: pgmap v3040: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:14:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:30 smithi059 ceph-mon[97450]: pgmap v3041: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:14:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:14:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:31.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:31.266036+0000) 2022-09-07T23:14:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:31.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:31.163549+0000) 2022-09-07T23:14:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:32.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:32.266235+0000) 2022-09-07T23:14:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:32.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:32.163787+0000) 2022-09-07T23:14:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:32 smithi055 ceph-mon[97033]: pgmap v3042: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:14:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:32 smithi059 ceph-mon[97450]: pgmap v3042: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:14:33.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:14:32.806+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:14:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:33.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:33.266433+0000) 2022-09-07T23:14:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:33.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:33.163987+0000) 2022-09-07T23:14:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:34.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:34.164222+0000) 2022-09-07T23:14:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:34.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:34.266546+0000) 2022-09-07T23:14:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:34 smithi055 ceph-mon[97033]: pgmap v3043: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:14:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:34 smithi059 ceph-mon[97450]: pgmap v3043: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:14:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:35.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:35.164493+0000) 2022-09-07T23:14:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:35.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:35.266719+0000) 2022-09-07T23:14:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:36.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:36.266890+0000) 2022-09-07T23:14:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:36.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:36.164774+0000) 2022-09-07T23:14:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:36 smithi055 ceph-mon[97033]: pgmap v3044: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:14:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:36 smithi059 ceph-mon[97450]: pgmap v3044: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:14:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:37.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:37.267099+0000) 2022-09-07T23:14:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:37.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:37.164977+0000) 2022-09-07T23:14:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:38.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:38.267336+0000) 2022-09-07T23:14:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:38.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:38.165171+0000) 2022-09-07T23:14:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:38 smithi055 ceph-mon[97033]: pgmap v3045: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:14:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:38 smithi059 ceph-mon[97450]: pgmap v3045: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:14:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:39.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:39.267560+0000) 2022-09-07T23:14:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:39.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:39.165400+0000) 2022-09-07T23:14:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:40.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:40.165667+0000) 2022-09-07T23:14:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:40.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:40.267775+0000) 2022-09-07T23:14:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:40 smithi055 ceph-mon[97033]: pgmap v3046: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:14:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:40 smithi059 ceph-mon[97450]: pgmap v3046: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:14:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:41.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:41.267957+0000) 2022-09-07T23:14:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:41.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:41.165900+0000) 2022-09-07T23:14:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:42.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:42.268183+0000) 2022-09-07T23:14:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:42.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:42.166131+0000) 2022-09-07T23:14:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:42 smithi055 ceph-mon[97033]: pgmap v3047: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:14:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:42 smithi059 ceph-mon[97450]: pgmap v3047: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:14:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:43.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:43.268388+0000) 2022-09-07T23:14:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:43.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:43.166400+0000) 2022-09-07T23:14:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:44.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:44.268606+0000) 2022-09-07T23:14:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:44.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:44.166698+0000) 2022-09-07T23:14:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:44 smithi055 ceph-mon[97033]: pgmap v3048: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:44 smithi059 ceph-mon[97450]: pgmap v3048: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:45.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:45.268899+0000) 2022-09-07T23:14:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:45.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:45.166940+0000) 2022-09-07T23:14:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:46.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:46.269109+0000) 2022-09-07T23:14:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:46.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:46.167185+0000) 2022-09-07T23:14:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:46 smithi055 ceph-mon[97033]: pgmap v3049: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:14:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:46 smithi059 ceph-mon[97450]: pgmap v3049: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:14:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:47.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:47.269312+0000) 2022-09-07T23:14:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:47.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:47.167398+0000) 2022-09-07T23:14:48.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:14:47.821+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:14:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:48.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:48.269544+0000) 2022-09-07T23:14:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:48.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:48.167681+0000) 2022-09-07T23:14:49.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:48 smithi055 ceph-mon[97033]: pgmap v3050: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:49.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:48 smithi059 ceph-mon[97450]: pgmap v3050: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:49.457 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:49.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:49.269837+0000) 2022-09-07T23:14:49.458 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:49.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:49.167839+0000) 2022-09-07T23:14:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:50.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:50.270039+0000) 2022-09-07T23:14:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:50.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:50.168050+0000) 2022-09-07T23:14:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:50 smithi055 ceph-mon[97033]: pgmap v3051: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:50 smithi059 ceph-mon[97450]: pgmap v3051: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:51.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:51.270165+0000) 2022-09-07T23:14:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:51.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:51.168232+0000) 2022-09-07T23:14:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:52.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:52.270452+0000) 2022-09-07T23:14:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:52.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:52.168401+0000) 2022-09-07T23:14:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:52 smithi055 ceph-mon[97033]: pgmap v3052: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:14:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:52 smithi059 ceph-mon[97450]: pgmap v3052: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:14:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:53.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:53.270713+0000) 2022-09-07T23:14:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:53.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:53.168603+0000) 2022-09-07T23:14:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:54.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:54.270918+0000) 2022-09-07T23:14:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:54.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:54.168792+0000) 2022-09-07T23:14:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:54 smithi055 ceph-mon[97033]: pgmap v3053: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:54 smithi059 ceph-mon[97450]: pgmap v3053: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:55.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:55.169052+0000) 2022-09-07T23:14:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:55.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:55.271134+0000) 2022-09-07T23:14:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:56.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:56.271379+0000) 2022-09-07T23:14:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:56.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:56.169271+0000) 2022-09-07T23:14:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:56 smithi055 ceph-mon[97033]: pgmap v3054: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:14:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:56 smithi059 ceph-mon[97450]: pgmap v3054: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:14:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:57.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:57.271515+0000) 2022-09-07T23:14:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:57.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:57.169495+0000) 2022-09-07T23:14:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:58.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:58.271717+0000) 2022-09-07T23:14:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:58.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:58.169749+0000) 2022-09-07T23:14:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:14:58 smithi055 ceph-mon[97033]: pgmap v3055: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:14:58 smithi059 ceph-mon[97450]: pgmap v3055: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:14:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:14:59.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:59.271965+0000) 2022-09-07T23:14:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:14:59.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:14:59.169980+0000) 2022-09-07T23:15:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:00.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:00.272143+0000) 2022-09-07T23:15:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:00.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:00.170206+0000) 2022-09-07T23:15:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:00 smithi055 ceph-mon[97033]: pgmap v3056: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:00 smithi059 ceph-mon[97450]: pgmap v3056: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:01.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:01.170427+0000) 2022-09-07T23:15:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:01.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:01.272344+0000) 2022-09-07T23:15:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:02.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:02.170677+0000) 2022-09-07T23:15:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:02.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:02.272572+0000) 2022-09-07T23:15:03.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:15:02.822+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:15:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:02 smithi055 ceph-mon[97033]: pgmap v3057: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:15:03.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:02 smithi059 ceph-mon[97450]: pgmap v3057: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:15:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:03.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:03.272814+0000) 2022-09-07T23:15:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:03.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:03.170920+0000) 2022-09-07T23:15:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:04.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:04.273008+0000) 2022-09-07T23:15:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:04.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:04.171162+0000) 2022-09-07T23:15:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:04 smithi055 ceph-mon[97033]: pgmap v3058: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:15:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:15:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:04 smithi059 ceph-mon[97450]: pgmap v3058: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:15:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:15:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:05.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:05.273205+0000) 2022-09-07T23:15:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:05.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:05.171369+0000) 2022-09-07T23:15:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:06.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:06.273413+0000) 2022-09-07T23:15:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:06.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:06.171595+0000) 2022-09-07T23:15:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:06 smithi055 ceph-mon[97033]: pgmap v3059: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:15:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:06 smithi059 ceph-mon[97450]: pgmap v3059: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:15:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:07.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:07.273556+0000) 2022-09-07T23:15:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:07.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:07.171790+0000) 2022-09-07T23:15:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:08.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:08.273801+0000) 2022-09-07T23:15:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:08.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:08.172030+0000) 2022-09-07T23:15:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:08 smithi055 ceph-mon[97033]: pgmap v3060: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:15:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:08 smithi059 ceph-mon[97450]: pgmap v3060: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:15:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:09.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:09.274002+0000) 2022-09-07T23:15:09.695 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:09.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:09.172214+0000) 2022-09-07T23:15:10.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:09 smithi055 ceph-mon[97033]: osdmap e281: 12 total, 11 up, 11 in 2022-09-07T23:15:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:09 smithi059 ceph-mon[97450]: osdmap e281: 12 total, 11 up, 11 in 2022-09-07T23:15:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:10.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:10.274215+0000) 2022-09-07T23:15:10.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:10.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:10.172469+0000) 2022-09-07T23:15:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:10 smithi055 ceph-mon[97033]: pgmap v3062: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:15:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:10 smithi059 ceph-mon[97450]: pgmap v3062: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:15:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:11.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:11.274464+0000) 2022-09-07T23:15:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:11.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:11.172663+0000) 2022-09-07T23:15:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:12.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:12.274715+0000) 2022-09-07T23:15:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:12.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:12.172929+0000) 2022-09-07T23:15:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:12 smithi055 ceph-mon[97033]: pgmap v3063: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:15:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:12 smithi059 ceph-mon[97450]: pgmap v3063: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:15:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:13.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:13.274911+0000) 2022-09-07T23:15:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:13.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:13.173134+0000) 2022-09-07T23:15:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:14.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:14.275082+0000) 2022-09-07T23:15:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:14.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:14.173412+0000) 2022-09-07T23:15:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:14 smithi055 ceph-mon[97033]: pgmap v3064: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:15:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:14 smithi055 ceph-mon[97033]: osdmap e282: 12 total, 11 up, 11 in 2022-09-07T23:15:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:14 smithi059 ceph-mon[97450]: pgmap v3064: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:15:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:14 smithi059 ceph-mon[97450]: osdmap e282: 12 total, 11 up, 11 in 2022-09-07T23:15:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:15.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:15.275297+0000) 2022-09-07T23:15:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:15.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:15.173673+0000) 2022-09-07T23:15:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:16.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:16.275536+0000) 2022-09-07T23:15:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:16.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:16.173905+0000) 2022-09-07T23:15:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:16 smithi055 ceph-mon[97033]: pgmap v3066: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:15:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:16 smithi059 ceph-mon[97450]: pgmap v3066: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:15:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:17.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:17.275783+0000) 2022-09-07T23:15:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:17.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:17.174168+0000) 2022-09-07T23:15:18.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:15:17.839+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:15:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:18.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:18.275960+0000) 2022-09-07T23:15:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:18.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:18.174380+0000) 2022-09-07T23:15:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:18 smithi055 ceph-mon[97033]: pgmap v3067: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 240 B/s wr, 7 op/s 2022-09-07T23:15:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:18 smithi059 ceph-mon[97450]: pgmap v3067: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 240 B/s wr, 7 op/s 2022-09-07T23:15:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:19.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:19.276196+0000) 2022-09-07T23:15:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:19.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:19.174577+0000) 2022-09-07T23:15:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:20.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:20.276379+0000) 2022-09-07T23:15:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:20.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:20.174805+0000) 2022-09-07T23:15:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:20 smithi055 ceph-mon[97033]: pgmap v3068: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:15:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:20 smithi059 ceph-mon[97450]: pgmap v3068: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:15:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:21.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:21.276603+0000) 2022-09-07T23:15:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:21.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:21.175045+0000) 2022-09-07T23:15:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:22.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:22.276844+0000) 2022-09-07T23:15:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:22.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:22.175213+0000) 2022-09-07T23:15:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:22 smithi055 ceph-mon[97033]: pgmap v3069: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:15:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:22 smithi059 ceph-mon[97450]: pgmap v3069: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:15:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:23.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:23.277061+0000) 2022-09-07T23:15:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:23.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:23.175406+0000) 2022-09-07T23:15:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:24.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:24.277268+0000) 2022-09-07T23:15:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:24.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:24.175540+0000) 2022-09-07T23:15:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:24 smithi055 ceph-mon[97033]: pgmap v3070: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:15:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:24 smithi059 ceph-mon[97450]: pgmap v3070: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:15:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:25.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:25.277464+0000) 2022-09-07T23:15:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:25.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:25.175714+0000) 2022-09-07T23:15:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:26.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:26.277728+0000) 2022-09-07T23:15:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:26.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:26.175948+0000) 2022-09-07T23:15:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:26 smithi055 ceph-mon[97033]: pgmap v3071: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2022-09-07T23:15:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:26 smithi059 ceph-mon[97450]: pgmap v3071: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2022-09-07T23:15:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:27.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:27.277935+0000) 2022-09-07T23:15:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:27.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:27.176179+0000) 2022-09-07T23:15:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:28.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:28.278134+0000) 2022-09-07T23:15:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:28.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:28.176438+0000) 2022-09-07T23:15:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:28 smithi055 ceph-mon[97033]: pgmap v3072: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:28 smithi059 ceph-mon[97450]: pgmap v3072: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:29.557 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:29.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:29.278380+0000) 2022-09-07T23:15:29.557 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:29.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:29.176664+0000) 2022-09-07T23:15:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:15:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:15:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:15:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:15:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:15:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:15:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:30.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:30.278547+0000) 2022-09-07T23:15:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:30.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:30.176890+0000) 2022-09-07T23:15:31.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:30 smithi055 ceph-mon[97033]: pgmap v3073: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:30 smithi059 ceph-mon[97450]: pgmap v3073: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:31.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:31.278775+0000) 2022-09-07T23:15:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:31.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:31.177036+0000) 2022-09-07T23:15:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:32.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:32.279006+0000) 2022-09-07T23:15:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:32.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:32.177269+0000) 2022-09-07T23:15:33.164 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:15:32.846+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:15:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:32 smithi055 ceph-mon[97033]: pgmap v3074: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:15:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:32 smithi059 ceph-mon[97450]: pgmap v3074: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:15:33.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:33.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:33.177505+0000) 2022-09-07T23:15:33.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:33.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:33.279280+0000) 2022-09-07T23:15:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:34.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:34.279455+0000) 2022-09-07T23:15:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:34.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:34.177697+0000) 2022-09-07T23:15:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:34 smithi055 ceph-mon[97033]: pgmap v3075: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:34 smithi059 ceph-mon[97450]: pgmap v3075: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:35.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:35.177940+0000) 2022-09-07T23:15:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:35.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:35.279661+0000) 2022-09-07T23:15:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:36.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:36.178196+0000) 2022-09-07T23:15:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:36.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:36.279874+0000) 2022-09-07T23:15:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:36 smithi055 ceph-mon[97033]: pgmap v3076: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:15:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:36 smithi059 ceph-mon[97450]: pgmap v3076: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:15:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:37.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:37.280091+0000) 2022-09-07T23:15:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:37.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:37.178424+0000) 2022-09-07T23:15:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:38.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:38.178596+0000) 2022-09-07T23:15:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:38.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:38.280332+0000) 2022-09-07T23:15:39.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:38 smithi055 ceph-mon[97033]: pgmap v3077: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:38 smithi059 ceph-mon[97450]: pgmap v3077: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:39.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:39.178885+0000) 2022-09-07T23:15:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:39.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:39.280522+0000) 2022-09-07T23:15:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:40.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:40.280698+0000) 2022-09-07T23:15:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:40.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:40.179135+0000) 2022-09-07T23:15:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:40 smithi055 ceph-mon[97033]: pgmap v3078: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:40 smithi059 ceph-mon[97450]: pgmap v3078: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:41.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:41.179329+0000) 2022-09-07T23:15:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:41.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:41.280945+0000) 2022-09-07T23:15:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:42.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:42.281155+0000) 2022-09-07T23:15:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:42.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:42.179496+0000) 2022-09-07T23:15:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:42 smithi055 ceph-mon[97033]: pgmap v3079: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:15:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:42 smithi059 ceph-mon[97450]: pgmap v3079: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:15:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:43.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:43.179724+0000) 2022-09-07T23:15:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:43.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:43.281381+0000) 2022-09-07T23:15:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:44.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:44.179920+0000) 2022-09-07T23:15:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:44.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:44.281619+0000) 2022-09-07T23:15:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:44 smithi055 ceph-mon[97033]: pgmap v3080: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:44 smithi059 ceph-mon[97450]: pgmap v3080: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:45.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:45.281776+0000) 2022-09-07T23:15:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:45.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:45.180157+0000) 2022-09-07T23:15:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:46.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:46.281933+0000) 2022-09-07T23:15:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:46.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:46.180408+0000) 2022-09-07T23:15:47.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:46 smithi055 ceph-mon[97033]: pgmap v3081: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:15:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:46 smithi059 ceph-mon[97450]: pgmap v3081: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:15:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:47.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:47.282141+0000) 2022-09-07T23:15:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:47.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:47.180674+0000) 2022-09-07T23:15:48.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:15:47.850+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:15:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:48.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:48.282317+0000) 2022-09-07T23:15:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:48.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:48.180855+0000) 2022-09-07T23:15:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:48 smithi055 ceph-mon[97033]: pgmap v3082: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:48 smithi059 ceph-mon[97450]: pgmap v3082: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:49.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:49.282555+0000) 2022-09-07T23:15:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:49.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:49.181049+0000) 2022-09-07T23:15:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:50.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:50.282780+0000) 2022-09-07T23:15:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:50.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:50.181283+0000) 2022-09-07T23:15:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:50 smithi055 ceph-mon[97033]: pgmap v3083: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:50 smithi059 ceph-mon[97450]: pgmap v3083: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:51.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:51.283030+0000) 2022-09-07T23:15:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:51.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:51.181458+0000) 2022-09-07T23:15:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:52.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:52.283180+0000) 2022-09-07T23:15:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:52.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:52.181727+0000) 2022-09-07T23:15:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:52 smithi055 ceph-mon[97033]: pgmap v3084: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:15:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:52 smithi059 ceph-mon[97450]: pgmap v3084: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:15:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:53.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:53.283381+0000) 2022-09-07T23:15:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:53.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:53.181965+0000) 2022-09-07T23:15:54.619 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:54.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:54.283605+0000) 2022-09-07T23:15:54.620 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:54.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:54.182174+0000) 2022-09-07T23:15:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:54 smithi055 ceph-mon[97033]: pgmap v3085: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:54 smithi059 ceph-mon[97450]: pgmap v3085: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:55.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:55.182413+0000) 2022-09-07T23:15:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:55.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:55.283803+0000) 2022-09-07T23:15:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:56.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:56.284022+0000) 2022-09-07T23:15:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:56.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:56.182669+0000) 2022-09-07T23:15:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:56 smithi055 ceph-mon[97033]: pgmap v3086: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:15:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:56 smithi059 ceph-mon[97450]: pgmap v3086: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:15:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:57.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:57.284218+0000) 2022-09-07T23:15:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:57.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:57.182844+0000) 2022-09-07T23:15:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:58.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:58.182969+0000) 2022-09-07T23:15:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:58.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:58.284435+0000) 2022-09-07T23:15:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:15:58 smithi055 ceph-mon[97033]: pgmap v3087: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:15:58 smithi059 ceph-mon[97450]: pgmap v3087: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:15:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:15:59.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:59.284630+0000) 2022-09-07T23:15:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:15:59.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:15:59.183175+0000) 2022-09-07T23:16:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:00.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:00.284858+0000) 2022-09-07T23:16:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:00.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:00.183422+0000) 2022-09-07T23:16:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:00 smithi055 ceph-mon[97033]: pgmap v3088: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:00 smithi055 ceph-mon[97033]: pgmap v3089: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:16:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:00 smithi055 ceph-mon[97033]: pgmap v3090: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:16:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:16:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:00 smithi059 ceph-mon[97450]: pgmap v3088: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:00 smithi059 ceph-mon[97450]: pgmap v3089: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:16:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:00 smithi059 ceph-mon[97450]: pgmap v3090: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:16:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:16:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:01.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:01.285033+0000) 2022-09-07T23:16:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:01.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:01.183587+0000) 2022-09-07T23:16:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:02.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:02.285279+0000) 2022-09-07T23:16:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:02.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:02.183770+0000) 2022-09-07T23:16:02.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:02 smithi055 ceph-mon[97033]: pgmap v3091: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:16:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:02 smithi059 ceph-mon[97450]: pgmap v3091: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:16:03.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:16:02.853+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:16:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:03.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:03.285506+0000) 2022-09-07T23:16:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:03.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:03.183983+0000) 2022-09-07T23:16:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:04.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:04.285697+0000) 2022-09-07T23:16:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:04.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:04.184151+0000) 2022-09-07T23:16:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:04 smithi055 ceph-mon[97033]: pgmap v3092: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:16:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:16:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:16:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:04 smithi059 ceph-mon[97450]: pgmap v3092: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:16:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:16:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:16:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:05.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:05.285837+0000) 2022-09-07T23:16:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:05.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:05.184395+0000) 2022-09-07T23:16:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:06.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:06.286097+0000) 2022-09-07T23:16:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:06.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:06.184660+0000) 2022-09-07T23:16:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:06 smithi055 ceph-mon[97033]: pgmap v3093: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-07T23:16:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:06 smithi059 ceph-mon[97450]: pgmap v3093: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-07T23:16:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:07.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:07.286375+0000) 2022-09-07T23:16:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:07.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:07.184798+0000) 2022-09-07T23:16:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:08.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:08.286536+0000) 2022-09-07T23:16:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:08.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:08.184978+0000) 2022-09-07T23:16:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:08 smithi055 ceph-mon[97033]: pgmap v3094: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-07T23:16:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:08 smithi059 ceph-mon[97450]: pgmap v3094: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-07T23:16:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:09.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:09.185216+0000) 2022-09-07T23:16:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:09.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:09.286778+0000) 2022-09-07T23:16:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:09 smithi055 ceph-mon[97033]: osdmap e283: 12 total, 11 up, 11 in 2022-09-07T23:16:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:09 smithi059 ceph-mon[97450]: osdmap e283: 12 total, 11 up, 11 in 2022-09-07T23:16:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:10.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:10.287001+0000) 2022-09-07T23:16:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:10.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:10.185471+0000) 2022-09-07T23:16:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:10 smithi055 ceph-mon[97033]: pgmap v3096: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:16:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:10 smithi059 ceph-mon[97450]: pgmap v3096: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:16:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:11.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:11.287229+0000) 2022-09-07T23:16:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:11.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:11.185754+0000) 2022-09-07T23:16:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:12.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:12.287489+0000) 2022-09-07T23:16:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:12.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:12.185921+0000) 2022-09-07T23:16:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:12 smithi055 ceph-mon[97033]: pgmap v3097: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:16:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:12 smithi059 ceph-mon[97450]: pgmap v3097: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:16:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:13.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:13.287753+0000) 2022-09-07T23:16:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:13.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:13.186086+0000) 2022-09-07T23:16:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:14.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:14.287978+0000) 2022-09-07T23:16:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:14.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:14.186301+0000) 2022-09-07T23:16:15.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:14 smithi055 ceph-mon[97033]: pgmap v3098: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:16:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:14 smithi059 ceph-mon[97450]: pgmap v3098: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:16:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:15.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:15.186546+0000) 2022-09-07T23:16:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:15.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:15.288189+0000) 2022-09-07T23:16:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:16.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:16.186779+0000) 2022-09-07T23:16:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:16.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:16.288422+0000) 2022-09-07T23:16:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:16 smithi055 ceph-mon[97033]: pgmap v3099: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:16:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:16 smithi059 ceph-mon[97450]: pgmap v3099: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:16:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:17.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:17.288680+0000) 2022-09-07T23:16:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:17.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:17.186975+0000) 2022-09-07T23:16:18.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:16:17.868+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:16:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:18.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:18.288924+0000) 2022-09-07T23:16:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:18.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:18.187164+0000) 2022-09-07T23:16:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:18 smithi055 ceph-mon[97033]: pgmap v3100: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:16:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:18 smithi055 ceph-mon[97033]: osdmap e284: 12 total, 11 up, 11 in 2022-09-07T23:16:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:18 smithi059 ceph-mon[97450]: pgmap v3100: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:16:19.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:18 smithi059 ceph-mon[97450]: osdmap e284: 12 total, 11 up, 11 in 2022-09-07T23:16:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:19.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:19.289187+0000) 2022-09-07T23:16:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:19.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:19.187416+0000) 2022-09-07T23:16:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:20.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:20.289331+0000) 2022-09-07T23:16:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:20.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:20.187554+0000) 2022-09-07T23:16:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:20 smithi055 ceph-mon[97033]: pgmap v3102: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:16:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:20 smithi059 ceph-mon[97450]: pgmap v3102: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:16:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:21.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:21.289561+0000) 2022-09-07T23:16:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:21.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:21.187735+0000) 2022-09-07T23:16:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:22.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:22.289738+0000) 2022-09-07T23:16:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:22.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:22.187924+0000) 2022-09-07T23:16:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:22 smithi055 ceph-mon[97033]: pgmap v3103: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T23:16:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:22 smithi059 ceph-mon[97450]: pgmap v3103: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T23:16:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:23.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:23.289895+0000) 2022-09-07T23:16:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:23.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:23.188144+0000) 2022-09-07T23:16:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:24.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:24.188359+0000) 2022-09-07T23:16:24.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:24.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:24.290115+0000) 2022-09-07T23:16:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:24 smithi055 ceph-mon[97033]: pgmap v3104: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T23:16:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:24 smithi059 ceph-mon[97450]: pgmap v3104: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T23:16:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:25.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:25.290292+0000) 2022-09-07T23:16:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:25.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:25.188610+0000) 2022-09-07T23:16:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:26.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:26.290451+0000) 2022-09-07T23:16:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:26.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:26.188863+0000) 2022-09-07T23:16:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:26 smithi055 ceph-mon[97033]: pgmap v3105: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:16:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:26 smithi059 ceph-mon[97450]: pgmap v3105: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:16:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:27.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:27.290717+0000) 2022-09-07T23:16:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:27.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:27.189036+0000) 2022-09-07T23:16:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:28.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:28.290958+0000) 2022-09-07T23:16:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:28.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:28.189253+0000) 2022-09-07T23:16:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:28 smithi055 ceph-mon[97033]: pgmap v3106: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:16:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:28 smithi059 ceph-mon[97450]: pgmap v3106: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:16:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:29.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:29.291235+0000) 2022-09-07T23:16:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:29.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:29.189455+0000) 2022-09-07T23:16:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:30.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:30.291517+0000) 2022-09-07T23:16:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:30.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:30.189728+0000) 2022-09-07T23:16:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:30 smithi055 ceph-mon[97033]: pgmap v3107: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T23:16:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:30 smithi059 ceph-mon[97450]: pgmap v3107: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-07T23:16:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:31.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:31.291712+0000) 2022-09-07T23:16:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:31.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:31.189979+0000) 2022-09-07T23:16:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:32.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:32.291899+0000) 2022-09-07T23:16:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:32.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:32.190184+0000) 2022-09-07T23:16:33.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:16:32.882+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:16:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:32 smithi055 ceph-mon[97033]: pgmap v3108: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:16:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:32 smithi059 ceph-mon[97450]: pgmap v3108: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:16:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:33.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:33.292105+0000) 2022-09-07T23:16:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:33.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:33.190408+0000) 2022-09-07T23:16:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:34.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:34.292330+0000) 2022-09-07T23:16:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:34.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:34.190696+0000) 2022-09-07T23:16:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:34 smithi055 ceph-mon[97033]: pgmap v3109: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:35.291 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:34 smithi059 ceph-mon[97450]: pgmap v3109: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:35.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:35.292590+0000) 2022-09-07T23:16:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:35.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:35.190931+0000) 2022-09-07T23:16:36.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:36.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:36.292774+0000) 2022-09-07T23:16:36.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:36.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:36.191153+0000) 2022-09-07T23:16:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:36 smithi055 ceph-mon[97033]: pgmap v3110: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:16:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:36 smithi059 ceph-mon[97450]: pgmap v3110: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:16:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:37.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:37.292926+0000) 2022-09-07T23:16:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:37.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:37.191377+0000) 2022-09-07T23:16:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:38.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:38.293179+0000) 2022-09-07T23:16:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:38.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:38.191621+0000) 2022-09-07T23:16:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:38 smithi055 ceph-mon[97033]: pgmap v3111: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:38 smithi059 ceph-mon[97450]: pgmap v3111: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:39.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:39.293390+0000) 2022-09-07T23:16:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:39.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:39.191915+0000) 2022-09-07T23:16:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:40.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:40.293675+0000) 2022-09-07T23:16:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:40.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:40.192109+0000) 2022-09-07T23:16:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:40 smithi055 ceph-mon[97033]: pgmap v3112: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:40 smithi059 ceph-mon[97450]: pgmap v3112: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:41.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:41.293954+0000) 2022-09-07T23:16:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:41.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:41.192310+0000) 2022-09-07T23:16:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:42.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:42.192539+0000) 2022-09-07T23:16:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:42.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:42.294073+0000) 2022-09-07T23:16:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:42 smithi055 ceph-mon[97033]: pgmap v3113: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:16:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:42 smithi059 ceph-mon[97450]: pgmap v3113: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:16:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:43.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:43.192782+0000) 2022-09-07T23:16:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:43.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:43.294282+0000) 2022-09-07T23:16:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:44.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:44.294517+0000) 2022-09-07T23:16:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:44.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:44.193065+0000) 2022-09-07T23:16:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:44 smithi055 ceph-mon[97033]: pgmap v3114: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:45.292 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:44 smithi059 ceph-mon[97450]: pgmap v3114: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:45.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:45.294684+0000) 2022-09-07T23:16:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:45.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:45.193238+0000) 2022-09-07T23:16:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:46.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:46.294909+0000) 2022-09-07T23:16:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:46.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:46.193430+0000) 2022-09-07T23:16:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:46 smithi055 ceph-mon[97033]: pgmap v3115: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:16:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:46 smithi059 ceph-mon[97450]: pgmap v3115: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:16:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:47.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:47.295051+0000) 2022-09-07T23:16:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:47.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:47.193707+0000) 2022-09-07T23:16:48.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:16:47.886+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:16:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:48.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:48.295228+0000) 2022-09-07T23:16:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:48.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:48.193897+0000) 2022-09-07T23:16:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:48 smithi055 ceph-mon[97033]: pgmap v3116: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:49.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:48 smithi059 ceph-mon[97450]: pgmap v3116: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:49.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:49.295457+0000) 2022-09-07T23:16:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:49.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:49.194064+0000) 2022-09-07T23:16:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:50.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:50.295745+0000) 2022-09-07T23:16:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:50.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:50.194354+0000) 2022-09-07T23:16:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:50 smithi055 ceph-mon[97033]: pgmap v3117: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:50 smithi059 ceph-mon[97450]: pgmap v3117: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:51.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:51.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:51.295927+0000) 2022-09-07T23:16:51.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:51.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:51.194568+0000) 2022-09-07T23:16:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:52.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:52.296152+0000) 2022-09-07T23:16:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:52.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:52.194765+0000) 2022-09-07T23:16:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:52 smithi055 ceph-mon[97033]: pgmap v3118: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:16:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:52 smithi059 ceph-mon[97450]: pgmap v3118: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:16:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:53.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:53.194992+0000) 2022-09-07T23:16:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:53.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:53.296392+0000) 2022-09-07T23:16:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:54.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:54.296620+0000) 2022-09-07T23:16:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:54.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:54.195231+0000) 2022-09-07T23:16:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:54 smithi055 ceph-mon[97033]: pgmap v3119: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:54 smithi059 ceph-mon[97450]: pgmap v3119: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:55.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:55.296882+0000) 2022-09-07T23:16:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:55.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:55.195524+0000) 2022-09-07T23:16:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:56.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:56.297094+0000) 2022-09-07T23:16:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:56.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:56.195767+0000) 2022-09-07T23:16:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:56 smithi055 ceph-mon[97033]: pgmap v3120: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:16:57.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:56 smithi059 ceph-mon[97450]: pgmap v3120: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:16:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:57.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:57.297225+0000) 2022-09-07T23:16:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:57.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:57.195946+0000) 2022-09-07T23:16:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:58.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:58.196066+0000) 2022-09-07T23:16:58.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:58.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:58.297468+0000) 2022-09-07T23:16:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:58 smithi055 ceph-mon[97033]: pgmap v3121: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:58 smithi059 ceph-mon[97450]: pgmap v3121: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:16:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:16:59.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:59.297723+0000) 2022-09-07T23:16:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:16:59.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:16:59.196279+0000) 2022-09-07T23:17:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:59 smithi055 ceph-mon[97033]: pgmap v3122: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:17:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:17:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:16:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:17:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:59 smithi059 ceph-mon[97450]: pgmap v3122: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:00.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:17:00.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:17:00.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:16:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:17:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:00.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:00.297954+0000) 2022-09-07T23:17:00.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:00.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:00.196529+0000) 2022-09-07T23:17:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:17:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:17:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:01.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:01.298196+0000) 2022-09-07T23:17:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:01.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:01.196696+0000) 2022-09-07T23:17:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:17:01.759 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:17:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:02 smithi059 ceph-mon[97450]: pgmap v3123: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:17:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:02.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:02.196897+0000) 2022-09-07T23:17:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:02.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:02.298447+0000) 2022-09-07T23:17:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:02 smithi055 ceph-mon[97033]: pgmap v3123: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:17:03.198 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:17:02.901+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:17:03.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:03.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:03.298654+0000) 2022-09-07T23:17:03.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:03.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:03.197099+0000) 2022-09-07T23:17:03.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:17:03.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:17:03.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:17:03.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:17:03.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:03.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:03.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:03.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:03.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:03.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:03.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:03.991 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:17:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:17:04.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:17:04.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:17:04.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:04.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:04.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:04.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:04.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:04.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:04.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:04.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:17:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:04.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:04.298874+0000) 2022-09-07T23:17:04.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:04.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:04.197332+0000) 2022-09-07T23:17:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:04 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T23:17:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:04 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T23:17:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:04 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T23:17:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:04 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T23:17:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:04 smithi055 ceph-mon[97033]: pgmap v3124: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:05.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:17:05.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:17:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:04 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T23:17:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:04 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T23:17:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:04 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T23:17:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:04 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T23:17:05.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:04 smithi059 ceph-mon[97450]: pgmap v3124: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:05.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:17:05.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:17:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:05.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:05.299134+0000) 2022-09-07T23:17:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:05.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:05.197556+0000) 2022-09-07T23:17:06.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:06.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:06.299393+0000) 2022-09-07T23:17:06.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:06.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:06.197702+0000) 2022-09-07T23:17:07.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:06 smithi055 ceph-mon[97033]: pgmap v3125: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:17:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:06 smithi059 ceph-mon[97450]: pgmap v3125: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:17:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:07.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:07.299627+0000) 2022-09-07T23:17:07.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:07.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:07.197935+0000) 2022-09-07T23:17:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:08.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:08.299828+0000) 2022-09-07T23:17:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:08.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:08.198162+0000) 2022-09-07T23:17:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:08 smithi055 ceph-mon[97033]: pgmap v3126: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:17:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:08 smithi059 ceph-mon[97450]: pgmap v3126: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:17:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:09.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:09.299985+0000) 2022-09-07T23:17:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:09.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:09.198304+0000) 2022-09-07T23:17:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:09 smithi055 ceph-mon[97033]: osdmap e285: 12 total, 11 up, 11 in 2022-09-07T23:17:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:09 smithi059 ceph-mon[97450]: osdmap e285: 12 total, 11 up, 11 in 2022-09-07T23:17:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:10.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:10.300205+0000) 2022-09-07T23:17:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:10.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:10.198536+0000) 2022-09-07T23:17:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:10 smithi055 ceph-mon[97033]: pgmap v3128: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:17:11.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:10 smithi059 ceph-mon[97450]: pgmap v3128: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:17:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:11.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:11.300446+0000) 2022-09-07T23:17:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:11.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:11.198789+0000) 2022-09-07T23:17:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:12.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:12.300626+0000) 2022-09-07T23:17:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:12.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:12.199023+0000) 2022-09-07T23:17:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:12 smithi055 ceph-mon[97033]: pgmap v3129: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:17:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:12 smithi059 ceph-mon[97450]: pgmap v3129: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:17:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:13.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:13.300783+0000) 2022-09-07T23:17:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:13.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:13.199241+0000) 2022-09-07T23:17:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:14.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:14.300988+0000) 2022-09-07T23:17:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:14.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:14.199495+0000) 2022-09-07T23:17:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:14 smithi055 ceph-mon[97033]: pgmap v3130: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:17:15.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:14 smithi055 ceph-mon[97033]: osdmap e286: 12 total, 11 up, 11 in 2022-09-07T23:17:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:14 smithi059 ceph-mon[97450]: pgmap v3130: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:17:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:14 smithi059 ceph-mon[97450]: osdmap e286: 12 total, 11 up, 11 in 2022-09-07T23:17:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:15.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:15.301246+0000) 2022-09-07T23:17:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:15.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:15.199752+0000) 2022-09-07T23:17:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:16.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:16.301395+0000) 2022-09-07T23:17:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:16.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:16.200001+0000) 2022-09-07T23:17:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:16 smithi055 ceph-mon[97033]: pgmap v3132: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:17:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:16 smithi059 ceph-mon[97450]: pgmap v3132: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:17:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:17.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:17.301609+0000) 2022-09-07T23:17:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:17.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:17.200230+0000) 2022-09-07T23:17:18.202 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:17:17.903+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:17:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:18.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:18.301858+0000) 2022-09-07T23:17:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:18.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:18.200356+0000) 2022-09-07T23:17:19.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:18 smithi055 ceph-mon[97033]: pgmap v3133: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-07T23:17:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:18 smithi059 ceph-mon[97450]: pgmap v3133: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-07T23:17:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:19.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:19.302116+0000) 2022-09-07T23:17:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:19.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:19.200599+0000) 2022-09-07T23:17:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:20.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:20.302319+0000) 2022-09-07T23:17:20.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:20.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:20.200886+0000) 2022-09-07T23:17:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:20 smithi055 ceph-mon[97033]: pgmap v3134: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:17:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:20 smithi059 ceph-mon[97450]: pgmap v3134: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:17:21.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:21.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:21.201108+0000) 2022-09-07T23:17:21.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:21.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:21.302556+0000) 2022-09-07T23:17:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:22.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:22.201336+0000) 2022-09-07T23:17:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:22.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:22.302796+0000) 2022-09-07T23:17:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:22 smithi055 ceph-mon[97033]: pgmap v3135: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:17:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:22 smithi059 ceph-mon[97450]: pgmap v3135: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:17:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:23.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:23.303044+0000) 2022-09-07T23:17:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:23.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:23.201559+0000) 2022-09-07T23:17:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:24.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:24.201709+0000) 2022-09-07T23:17:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:24.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:24.303282+0000) 2022-09-07T23:17:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:24 smithi055 ceph-mon[97033]: pgmap v3136: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:17:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:24 smithi059 ceph-mon[97450]: pgmap v3136: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:17:25.661 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:25.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:25.303484+0000) 2022-09-07T23:17:25.662 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:25.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:25.201960+0000) 2022-09-07T23:17:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:26.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:26.303739+0000) 2022-09-07T23:17:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:26.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:26.202218+0000) 2022-09-07T23:17:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:26 smithi055 ceph-mon[97033]: pgmap v3137: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-07T23:17:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:26 smithi059 ceph-mon[97450]: pgmap v3137: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-07T23:17:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:27.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:27.303970+0000) 2022-09-07T23:17:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:27.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:27.202452+0000) 2022-09-07T23:17:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:28.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:28.304165+0000) 2022-09-07T23:17:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:28.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:28.202595+0000) 2022-09-07T23:17:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:28 smithi055 ceph-mon[97033]: pgmap v3138: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:28 smithi059 ceph-mon[97450]: pgmap v3138: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:29.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:29.304399+0000) 2022-09-07T23:17:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:29.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:29.202798+0000) 2022-09-07T23:17:30.306 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:30.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:30.203038+0000) 2022-09-07T23:17:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:30.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:30.304631+0000) 2022-09-07T23:17:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:30 smithi055 ceph-mon[97033]: pgmap v3139: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:31.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:17:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:30 smithi059 ceph-mon[97450]: pgmap v3139: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:17:31.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:31.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:31.304849+0000) 2022-09-07T23:17:31.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:31.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:31.203270+0000) 2022-09-07T23:17:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:31 smithi055 ceph-mon[97033]: pgmap v3140: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 8 op/s 2022-09-07T23:17:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:31 smithi055 ceph-mon[97033]: pgmap v3141: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:17:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:31 smithi059 ceph-mon[97450]: pgmap v3140: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 8 op/s 2022-09-07T23:17:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:31 smithi059 ceph-mon[97450]: pgmap v3141: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:17:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:32.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:32.305090+0000) 2022-09-07T23:17:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:32.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:32.203497+0000) 2022-09-07T23:17:33.205 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:17:32.907+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:17:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:33.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:33.305310+0000) 2022-09-07T23:17:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:33.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:33.203657+0000) 2022-09-07T23:17:34.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:33 smithi055 ceph-mon[97033]: pgmap v3142: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:17:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:33 smithi059 ceph-mon[97450]: pgmap v3142: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:17:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:34.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:34.305441+0000) 2022-09-07T23:17:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:34.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:34.203767+0000) 2022-09-07T23:17:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:35.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:35.305666+0000) 2022-09-07T23:17:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:35.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:35.203959+0000) 2022-09-07T23:17:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:35 smithi055 ceph-mon[97033]: pgmap v3143: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:17:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:35 smithi059 ceph-mon[97450]: pgmap v3143: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:17:36.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:36.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:36.305788+0000) 2022-09-07T23:17:36.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:36.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:36.204212+0000) 2022-09-07T23:17:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:37.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:37.305987+0000) 2022-09-07T23:17:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:37.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:37.204396+0000) 2022-09-07T23:17:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:37 smithi055 ceph-mon[97033]: pgmap v3144: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:17:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:37 smithi059 ceph-mon[97450]: pgmap v3144: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:17:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:38.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:38.306170+0000) 2022-09-07T23:17:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:38.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:38.204625+0000) 2022-09-07T23:17:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:39.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:39.306431+0000) 2022-09-07T23:17:39.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:39.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:39.204910+0000) 2022-09-07T23:17:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:39 smithi055 ceph-mon[97033]: pgmap v3145: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:17:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:39 smithi059 ceph-mon[97450]: pgmap v3145: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:17:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:40.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:40.306730+0000) 2022-09-07T23:17:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:40.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:40.205145+0000) 2022-09-07T23:17:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:41.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:41.306974+0000) 2022-09-07T23:17:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:41.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:41.205387+0000) 2022-09-07T23:17:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:41 smithi055 ceph-mon[97033]: pgmap v3146: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:17:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:41 smithi059 ceph-mon[97450]: pgmap v3146: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:17:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:42.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:42.307155+0000) 2022-09-07T23:17:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:42.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:42.205614+0000) 2022-09-07T23:17:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:43.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:43.307353+0000) 2022-09-07T23:17:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:43.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:43.205811+0000) 2022-09-07T23:17:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:43 smithi055 ceph-mon[97033]: pgmap v3147: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:43 smithi059 ceph-mon[97450]: pgmap v3147: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:44.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:44.307600+0000) 2022-09-07T23:17:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:44.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:44.206024+0000) 2022-09-07T23:17:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:45.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:45.307842+0000) 2022-09-07T23:17:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:45.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:45.206260+0000) 2022-09-07T23:17:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:45 smithi055 ceph-mon[97033]: pgmap v3148: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:45 smithi059 ceph-mon[97450]: pgmap v3148: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:46.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:46.308083+0000) 2022-09-07T23:17:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:46.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:46.206448+0000) 2022-09-07T23:17:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:47.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:47.308295+0000) 2022-09-07T23:17:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:47.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:47.206723+0000) 2022-09-07T23:17:48.208 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:17:47.910+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:17:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:47 smithi055 ceph-mon[97033]: pgmap v3149: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:17:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:47 smithi059 ceph-mon[97450]: pgmap v3149: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:17:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:48.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:48.308466+0000) 2022-09-07T23:17:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:48.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:48.206863+0000) 2022-09-07T23:17:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:49.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:49.207080+0000) 2022-09-07T23:17:49.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:49.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:49.308737+0000) 2022-09-07T23:17:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:49 smithi055 ceph-mon[97033]: pgmap v3150: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:49 smithi059 ceph-mon[97450]: pgmap v3150: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:50.311 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:50.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:50.207279+0000) 2022-09-07T23:17:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:50.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:50.308983+0000) 2022-09-07T23:17:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:51.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:51.309243+0000) 2022-09-07T23:17:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:51.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:51.207424+0000) 2022-09-07T23:17:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:51 smithi055 ceph-mon[97033]: pgmap v3151: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:17:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:51 smithi059 ceph-mon[97450]: pgmap v3151: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:17:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:52.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:52.309498+0000) 2022-09-07T23:17:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:52.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:52.207715+0000) 2022-09-07T23:17:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:53.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:53.309776+0000) 2022-09-07T23:17:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:53.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:53.207858+0000) 2022-09-07T23:17:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:53 smithi055 ceph-mon[97033]: pgmap v3152: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:53 smithi059 ceph-mon[97450]: pgmap v3152: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:54.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:54.310051+0000) 2022-09-07T23:17:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:54.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:54.208053+0000) 2022-09-07T23:17:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:55.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:55.310340+0000) 2022-09-07T23:17:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:55.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:55.208234+0000) 2022-09-07T23:17:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:55 smithi055 ceph-mon[97033]: pgmap v3153: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:55 smithi059 ceph-mon[97450]: pgmap v3153: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:17:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:56.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:56.208379+0000) 2022-09-07T23:17:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:56.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:56.310602+0000) 2022-09-07T23:17:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:56 smithi055 ceph-mon[97033]: pgmap v3154: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:17:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:56 smithi059 ceph-mon[97450]: pgmap v3154: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:17:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:57.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:57.208581+0000) 2022-09-07T23:17:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:57.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:57.310898+0000) 2022-09-07T23:17:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:58.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:58.311084+0000) 2022-09-07T23:17:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:58.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:58.208753+0000) 2022-09-07T23:17:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:17:59.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:59.311354+0000) 2022-09-07T23:17:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:17:59.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:17:59.208920+0000) 2022-09-07T23:18:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:17:59 smithi055 ceph-mon[97033]: pgmap v3155: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:00.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:17:59 smithi059 ceph-mon[97450]: pgmap v3155: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:00.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:00.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:00.311535+0000) 2022-09-07T23:18:00.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:00.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:00.209094+0000) 2022-09-07T23:18:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:01.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:01.209217+0000) 2022-09-07T23:18:01.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:01.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:01.311753+0000) 2022-09-07T23:18:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:01 smithi055 ceph-mon[97033]: pgmap v3156: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:18:02.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:01 smithi059 ceph-mon[97450]: pgmap v3156: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:18:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:02.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:02.311974+0000) 2022-09-07T23:18:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:02.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:02.209436+0000) 2022-09-07T23:18:03.211 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:18:02.923+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:18:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:03.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:03.312200+0000) 2022-09-07T23:18:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:03.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:03.209703+0000) 2022-09-07T23:18:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:03 smithi055 ceph-mon[97033]: pgmap v3157: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:03 smithi059 ceph-mon[97450]: pgmap v3157: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:04.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:04.312446+0000) 2022-09-07T23:18:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:04.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:04.209990+0000) 2022-09-07T23:18:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:18:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:18:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:18:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:18:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:05.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:05.312683+0000) 2022-09-07T23:18:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:05.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:05.210221+0000) 2022-09-07T23:18:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:05 smithi055 ceph-mon[97033]: pgmap v3158: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:05 smithi059 ceph-mon[97450]: pgmap v3158: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:06.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:06.312912+0000) 2022-09-07T23:18:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:06.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:06.210469+0000) 2022-09-07T23:18:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:07.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:07.210754+0000) 2022-09-07T23:18:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:07.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:07.313162+0000) 2022-09-07T23:18:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:07 smithi055 ceph-mon[97033]: pgmap v3159: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:18:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:07 smithi059 ceph-mon[97450]: pgmap v3159: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:18:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:08.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:08.313370+0000) 2022-09-07T23:18:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:08.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:08.210998+0000) 2022-09-07T23:18:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:09.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:09.313531+0000) 2022-09-07T23:18:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:09.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:09.211224+0000) 2022-09-07T23:18:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:09 smithi055 ceph-mon[97033]: pgmap v3160: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:18:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:09 smithi055 ceph-mon[97033]: osdmap e287: 12 total, 11 up, 11 in 2022-09-07T23:18:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:09 smithi059 ceph-mon[97450]: pgmap v3160: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:18:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:09 smithi059 ceph-mon[97450]: osdmap e287: 12 total, 11 up, 11 in 2022-09-07T23:18:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:10.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:10.313751+0000) 2022-09-07T23:18:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:10.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:10.211437+0000) 2022-09-07T23:18:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:11.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:11.211694+0000) 2022-09-07T23:18:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:11.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:11.313916+0000) 2022-09-07T23:18:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:11 smithi055 ceph-mon[97033]: pgmap v3162: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:18:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:11 smithi059 ceph-mon[97450]: pgmap v3162: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:18:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:12.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:12.314145+0000) 2022-09-07T23:18:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:12.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:12.211944+0000) 2022-09-07T23:18:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:13.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:13.314377+0000) 2022-09-07T23:18:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:13.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:13.212188+0000) 2022-09-07T23:18:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:13 smithi055 ceph-mon[97033]: pgmap v3163: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:18:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:13 smithi055 ceph-mon[97033]: osdmap e288: 12 total, 11 up, 11 in 2022-09-07T23:18:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:13 smithi059 ceph-mon[97450]: pgmap v3163: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:18:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:13 smithi059 ceph-mon[97450]: osdmap e288: 12 total, 11 up, 11 in 2022-09-07T23:18:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:14.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:14.212362+0000) 2022-09-07T23:18:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:14.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:14.314545+0000) 2022-09-07T23:18:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:15.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:15.314780+0000) 2022-09-07T23:18:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:15.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:15.212539+0000) 2022-09-07T23:18:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:15 smithi055 ceph-mon[97033]: pgmap v3165: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:18:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:15 smithi059 ceph-mon[97450]: pgmap v3165: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:18:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:16.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:16.212772+0000) 2022-09-07T23:18:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:16.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:16.315006+0000) 2022-09-07T23:18:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:17.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:17.315273+0000) 2022-09-07T23:18:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:17.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:17.212902+0000) 2022-09-07T23:18:18.214 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:18:17.932+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:18:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:17 smithi055 ceph-mon[97033]: pgmap v3166: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:18:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:17 smithi059 ceph-mon[97450]: pgmap v3166: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:18:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:18.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:18.315414+0000) 2022-09-07T23:18:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:18.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:18.213088+0000) 2022-09-07T23:18:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:19.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:19.213293+0000) 2022-09-07T23:18:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:19.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:19.315692+0000) 2022-09-07T23:18:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:19 smithi055 ceph-mon[97033]: pgmap v3167: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 225 B/s wr, 7 op/s 2022-09-07T23:18:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:19 smithi059 ceph-mon[97450]: pgmap v3167: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 225 B/s wr, 7 op/s 2022-09-07T23:18:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:20.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:20.315937+0000) 2022-09-07T23:18:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:20.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:20.213507+0000) 2022-09-07T23:18:21.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:21.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:21.316080+0000) 2022-09-07T23:18:21.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:21.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:21.213754+0000) 2022-09-07T23:18:22.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:21 smithi055 ceph-mon[97033]: pgmap v3168: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:18:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:21 smithi059 ceph-mon[97450]: pgmap v3168: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:18:22.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:22.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:22.316275+0000) 2022-09-07T23:18:22.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:22.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:22.213959+0000) 2022-09-07T23:18:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:23.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:23.316486+0000) 2022-09-07T23:18:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:23.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:23.214182+0000) 2022-09-07T23:18:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:23 smithi055 ceph-mon[97033]: pgmap v3169: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:18:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:23 smithi059 ceph-mon[97450]: pgmap v3169: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:18:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:24.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:24.214383+0000) 2022-09-07T23:18:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:24.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:24.316741+0000) 2022-09-07T23:18:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:25.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:25.316941+0000) 2022-09-07T23:18:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:25.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:25.214605+0000) 2022-09-07T23:18:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:25 smithi055 ceph-mon[97033]: pgmap v3170: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 101 B/s wr, 6 op/s 2022-09-07T23:18:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:25 smithi059 ceph-mon[97450]: pgmap v3170: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 101 B/s wr, 6 op/s 2022-09-07T23:18:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:26.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:26.214882+0000) 2022-09-07T23:18:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:26.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:26.317163+0000) 2022-09-07T23:18:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:27.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:27.317363+0000) 2022-09-07T23:18:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:27.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:27.215114+0000) 2022-09-07T23:18:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:27 smithi055 ceph-mon[97033]: pgmap v3171: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:18:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:27 smithi059 ceph-mon[97450]: pgmap v3171: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:18:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:28.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:28.317543+0000) 2022-09-07T23:18:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:28.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:28.215323+0000) 2022-09-07T23:18:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:29.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:29.215504+0000) 2022-09-07T23:18:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:29.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:29.317734+0000) 2022-09-07T23:18:30.196 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:29 smithi055 ceph-mon[97033]: pgmap v3172: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:30.197 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:29 smithi059 ceph-mon[97450]: pgmap v3172: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:30.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:30.317929+0000) 2022-09-07T23:18:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:30.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:30.215716+0000) 2022-09-07T23:18:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:18:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:18:31.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:18:31.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:18:31.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:18:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:18:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:18:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:18:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:18:31.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:18:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:31.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:31.318120+0000) 2022-09-07T23:18:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:31.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:31.215922+0000) 2022-09-07T23:18:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:31 smithi055 ceph-mon[97033]: pgmap v3173: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:18:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:31 smithi059 ceph-mon[97450]: pgmap v3173: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:18:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:32.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:32.318376+0000) 2022-09-07T23:18:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:32.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:32.216173+0000) 2022-09-07T23:18:33.217 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:18:32.943+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:18:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:33.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:33.216405+0000) 2022-09-07T23:18:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:33.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:33.318592+0000) 2022-09-07T23:18:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:33 smithi055 ceph-mon[97033]: pgmap v3174: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:33 smithi059 ceph-mon[97450]: pgmap v3174: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:34.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:34.318804+0000) 2022-09-07T23:18:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:34.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:34.216617+0000) 2022-09-07T23:18:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:35.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:35.318929+0000) 2022-09-07T23:18:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:35.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:35.216853+0000) 2022-09-07T23:18:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:35 smithi055 ceph-mon[97033]: pgmap v3175: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:35 smithi059 ceph-mon[97450]: pgmap v3175: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:36.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:36.319120+0000) 2022-09-07T23:18:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:36.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:36.217082+0000) 2022-09-07T23:18:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:37.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:37.319368+0000) 2022-09-07T23:18:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:37.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:37.217308+0000) 2022-09-07T23:18:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:37 smithi055 ceph-mon[97033]: pgmap v3176: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:18:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:37 smithi059 ceph-mon[97450]: pgmap v3176: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:18:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:38.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:38.319503+0000) 2022-09-07T23:18:38.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:38.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:38.217533+0000) 2022-09-07T23:18:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:39.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:39.217772+0000) 2022-09-07T23:18:39.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:39.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:39.319711+0000) 2022-09-07T23:18:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:39 smithi055 ceph-mon[97033]: pgmap v3177: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:39 smithi059 ceph-mon[97450]: pgmap v3177: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:40.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:40.217966+0000) 2022-09-07T23:18:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:40.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:40.319898+0000) 2022-09-07T23:18:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:41.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:41.320101+0000) 2022-09-07T23:18:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:41.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:41.218204+0000) 2022-09-07T23:18:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:41 smithi055 ceph-mon[97033]: pgmap v3178: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:18:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:41 smithi059 ceph-mon[97450]: pgmap v3178: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:18:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:42.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:42.320227+0000) 2022-09-07T23:18:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:42.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:42.218443+0000) 2022-09-07T23:18:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:43.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:43.320451+0000) 2022-09-07T23:18:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:43.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:43.218728+0000) 2022-09-07T23:18:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:43 smithi055 ceph-mon[97033]: pgmap v3179: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:43 smithi059 ceph-mon[97450]: pgmap v3179: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:44.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:44.320648+0000) 2022-09-07T23:18:44.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:44.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:44.218874+0000) 2022-09-07T23:18:45.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:45.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:45.320865+0000) 2022-09-07T23:18:45.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:45.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:45.219030+0000) 2022-09-07T23:18:46.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:45 smithi055 ceph-mon[97033]: pgmap v3180: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:45 smithi059 ceph-mon[97450]: pgmap v3180: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:46.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:46.321085+0000) 2022-09-07T23:18:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:46.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:46.219270+0000) 2022-09-07T23:18:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:47.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:47.321340+0000) 2022-09-07T23:18:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:47.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:47.219514+0000) 2022-09-07T23:18:48.221 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:18:47.959+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:18:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:47 smithi055 ceph-mon[97033]: pgmap v3181: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:18:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:47 smithi059 ceph-mon[97450]: pgmap v3181: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:18:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:48.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:48.219757+0000) 2022-09-07T23:18:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:48.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:48.321533+0000) 2022-09-07T23:18:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:49.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:49.321728+0000) 2022-09-07T23:18:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:49.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:49.219896+0000) 2022-09-07T23:18:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:49 smithi055 ceph-mon[97033]: pgmap v3182: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:49 smithi059 ceph-mon[97450]: pgmap v3182: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:50.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:50.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:50.321882+0000) 2022-09-07T23:18:50.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:50.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:50.220022+0000) 2022-09-07T23:18:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:51.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:51.322103+0000) 2022-09-07T23:18:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:51.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:51.220249+0000) 2022-09-07T23:18:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:51 smithi055 ceph-mon[97033]: pgmap v3183: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:18:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:51 smithi059 ceph-mon[97450]: pgmap v3183: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:18:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:52.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:52.322306+0000) 2022-09-07T23:18:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:52.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:52.220454+0000) 2022-09-07T23:18:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:53.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:53.322553+0000) 2022-09-07T23:18:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:53.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:53.220655+0000) 2022-09-07T23:18:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:53 smithi055 ceph-mon[97033]: pgmap v3184: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:53 smithi059 ceph-mon[97450]: pgmap v3184: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:54.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:54.322770+0000) 2022-09-07T23:18:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:54.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:54.220781+0000) 2022-09-07T23:18:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:55.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:55.323011+0000) 2022-09-07T23:18:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:55.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:55.220931+0000) 2022-09-07T23:18:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:55 smithi055 ceph-mon[97033]: pgmap v3185: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:55 smithi059 ceph-mon[97450]: pgmap v3185: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:18:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:56.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:56.323239+0000) 2022-09-07T23:18:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:56.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:56.221117+0000) 2022-09-07T23:18:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:57.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:57.221312+0000) 2022-09-07T23:18:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:57.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:57.323406+0000) 2022-09-07T23:18:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:57 smithi055 ceph-mon[97033]: pgmap v3186: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:18:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:57 smithi059 ceph-mon[97450]: pgmap v3186: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:18:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:58.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:58.221524+0000) 2022-09-07T23:18:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:58.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:58.323615+0000) 2022-09-07T23:18:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:18:59.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:59.323758+0000) 2022-09-07T23:18:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:18:59.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:18:59.221793+0000) 2022-09-07T23:19:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:18:59 smithi055 ceph-mon[97033]: pgmap v3187: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:18:59 smithi059 ceph-mon[97450]: pgmap v3187: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:00.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:00.323941+0000) 2022-09-07T23:19:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:00.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:00.221983+0000) 2022-09-07T23:19:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:01 smithi055 ceph-mon[97033]: pgmap v3188: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:19:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:01 smithi055 ceph-mon[97033]: pgmap v3189: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:19:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:01 smithi055 ceph-mon[97033]: pgmap v3190: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:19:01.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:19:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:01 smithi059 ceph-mon[97450]: pgmap v3188: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:19:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:01 smithi059 ceph-mon[97450]: pgmap v3189: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:19:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:01 smithi059 ceph-mon[97450]: pgmap v3190: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:19:01.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:19:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:01.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:01.324056+0000) 2022-09-07T23:19:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:01.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:01.222203+0000) 2022-09-07T23:19:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:02.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:02.324315+0000) 2022-09-07T23:19:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:02.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:02.222449+0000) 2022-09-07T23:19:03.326 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:19:02.974+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:19:03.327 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:03.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:03.222673+0000) 2022-09-07T23:19:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:03 smithi059 ceph-mon[97450]: pgmap v3191: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:19:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:03.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:03.324556+0000) 2022-09-07T23:19:03.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:03 smithi055 ceph-mon[97033]: pgmap v3191: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:19:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:04.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:04.222803+0000) 2022-09-07T23:19:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:04.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:04.324768+0000) 2022-09-07T23:19:05.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:19:05.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:19:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:19:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:19:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:05.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:05.223044+0000) 2022-09-07T23:19:05.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:05.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:05.324970+0000) 2022-09-07T23:19:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:05 smithi055 ceph-mon[97033]: pgmap v3192: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:19:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:05 smithi059 ceph-mon[97450]: pgmap v3192: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:19:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:06.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:06.325185+0000) 2022-09-07T23:19:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:06.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:06.223296+0000) 2022-09-07T23:19:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:07.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:07.325420+0000) 2022-09-07T23:19:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:07.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:07.223518+0000) 2022-09-07T23:19:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:07 smithi055 ceph-mon[97033]: pgmap v3193: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-07T23:19:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:07 smithi059 ceph-mon[97450]: pgmap v3193: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-07T23:19:08.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:08.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:08.325648+0000) 2022-09-07T23:19:08.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:08.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:08.223740+0000) 2022-09-07T23:19:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:09.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:09.325871+0000) 2022-09-07T23:19:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:09.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:09.223900+0000) 2022-09-07T23:19:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:09 smithi055 ceph-mon[97033]: pgmap v3194: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 65 KiB/s wr, 25 op/s 2022-09-07T23:19:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:09 smithi055 ceph-mon[97033]: osdmap e289: 12 total, 11 up, 11 in 2022-09-07T23:19:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:09 smithi059 ceph-mon[97450]: pgmap v3194: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 65 KiB/s wr, 25 op/s 2022-09-07T23:19:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:09 smithi059 ceph-mon[97450]: osdmap e289: 12 total, 11 up, 11 in 2022-09-07T23:19:10.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:10.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:10.326016+0000) 2022-09-07T23:19:10.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:10.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:10.224032+0000) 2022-09-07T23:19:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:11.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:11.326222+0000) 2022-09-07T23:19:11.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:11.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:11.224245+0000) 2022-09-07T23:19:12.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:11 smithi055 ceph-mon[97033]: pgmap v3196: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:19:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:11 smithi059 ceph-mon[97450]: pgmap v3196: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:19:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:12.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:12.224498+0000) 2022-09-07T23:19:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:12.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:12.326474+0000) 2022-09-07T23:19:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:13.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:13.326741+0000) 2022-09-07T23:19:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:13.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:13.224702+0000) 2022-09-07T23:19:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:13 smithi055 ceph-mon[97033]: pgmap v3197: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:19:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:13 smithi055 ceph-mon[97033]: osdmap e290: 12 total, 11 up, 11 in 2022-09-07T23:19:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:13 smithi059 ceph-mon[97450]: pgmap v3197: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:19:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:13 smithi059 ceph-mon[97450]: osdmap e290: 12 total, 11 up, 11 in 2022-09-07T23:19:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:14.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:14.326887+0000) 2022-09-07T23:19:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:14.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:14.224884+0000) 2022-09-07T23:19:15.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:15.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:15.327048+0000) 2022-09-07T23:19:15.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:15.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:15.225090+0000) 2022-09-07T23:19:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:15 smithi055 ceph-mon[97033]: pgmap v3199: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:19:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:15 smithi059 ceph-mon[97450]: pgmap v3199: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:19:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:16.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:16.327279+0000) 2022-09-07T23:19:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:16.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:16.225335+0000) 2022-09-07T23:19:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:17.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:17.327516+0000) 2022-09-07T23:19:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:17.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:17.225516+0000) 2022-09-07T23:19:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:17 smithi055 ceph-mon[97033]: pgmap v3200: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:19:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:17 smithi059 ceph-mon[97450]: pgmap v3200: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:19:18.329 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:19:17.976+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:19:18.330 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:18.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:18.225762+0000) 2022-09-07T23:19:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:18.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:18.327765+0000) 2022-09-07T23:19:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:19.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:19.327970+0000) 2022-09-07T23:19:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:19.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:19.225997+0000) 2022-09-07T23:19:20.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:19 smithi055 ceph-mon[97033]: pgmap v3201: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-07T23:19:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:19 smithi059 ceph-mon[97450]: pgmap v3201: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-07T23:19:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:20.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:20.328133+0000) 2022-09-07T23:19:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:20.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:20.226196+0000) 2022-09-07T23:19:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:21.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:21.328262+0000) 2022-09-07T23:19:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:21.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:21.226373+0000) 2022-09-07T23:19:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:21 smithi055 ceph-mon[97033]: pgmap v3202: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:19:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:21 smithi059 ceph-mon[97450]: pgmap v3202: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:19:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:22.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:22.328488+0000) 2022-09-07T23:19:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:22.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:22.226521+0000) 2022-09-07T23:19:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:23.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:23.328754+0000) 2022-09-07T23:19:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:23.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:23.226785+0000) 2022-09-07T23:19:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:23 smithi055 ceph-mon[97033]: pgmap v3203: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:19:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:23 smithi059 ceph-mon[97450]: pgmap v3203: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:19:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:24.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:24.328960+0000) 2022-09-07T23:19:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:24.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:24.226933+0000) 2022-09-07T23:19:25.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:25.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:25.329133+0000) 2022-09-07T23:19:25.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:25.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:25.227152+0000) 2022-09-07T23:19:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:25 smithi055 ceph-mon[97033]: pgmap v3204: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-07T23:19:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:25 smithi059 ceph-mon[97450]: pgmap v3204: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-07T23:19:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:26.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:26.329316+0000) 2022-09-07T23:19:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:26.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:26.227334+0000) 2022-09-07T23:19:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:27.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:27.329548+0000) 2022-09-07T23:19:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:27.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:27.227587+0000) 2022-09-07T23:19:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:27 smithi055 ceph-mon[97033]: pgmap v3205: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:19:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:27 smithi059 ceph-mon[97450]: pgmap v3205: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:19:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:28.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:28.329797+0000) 2022-09-07T23:19:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:28.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:28.227724+0000) 2022-09-07T23:19:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:29.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:29.330047+0000) 2022-09-07T23:19:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:29.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:29.227963+0000) 2022-09-07T23:19:30.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:29 smithi055 ceph-mon[97033]: pgmap v3206: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:29 smithi059 ceph-mon[97450]: pgmap v3206: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:30.666 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:30.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:30.330211+0000) 2022-09-07T23:19:30.668 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:30.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:30.228113+0000) 2022-09-07T23:19:31.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:31.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:31.330436+0000) 2022-09-07T23:19:31.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:31.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:31.228303+0000) 2022-09-07T23:19:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:31 smithi055 ceph-mon[97033]: pgmap v3207: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:19:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:31 smithi059 ceph-mon[97450]: pgmap v3207: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:19:32.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:32.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:32.330680+0000) 2022-09-07T23:19:32.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:32.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:32.228444+0000) 2022-09-07T23:19:33.332 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:19:32.991+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:19:33.333 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:33.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:33.228721+0000) 2022-09-07T23:19:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:33.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:33.330906+0000) 2022-09-07T23:19:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:33 smithi055 ceph-mon[97033]: pgmap v3208: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:33 smithi059 ceph-mon[97450]: pgmap v3208: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:34.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:34.331103+0000) 2022-09-07T23:19:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:34.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:34.228881+0000) 2022-09-07T23:19:35.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:35.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:35.331357+0000) 2022-09-07T23:19:35.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:35.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:35.229113+0000) 2022-09-07T23:19:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:35 smithi055 ceph-mon[97033]: pgmap v3209: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:35 smithi059 ceph-mon[97450]: pgmap v3209: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:36.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:36.331526+0000) 2022-09-07T23:19:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:36.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:36.229268+0000) 2022-09-07T23:19:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:37.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:37.331711+0000) 2022-09-07T23:19:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:37.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:37.229487+0000) 2022-09-07T23:19:38.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:37 smithi055 ceph-mon[97033]: pgmap v3210: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:19:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:37 smithi059 ceph-mon[97450]: pgmap v3210: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:19:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:38.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:38.331860+0000) 2022-09-07T23:19:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:38.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:38.229675+0000) 2022-09-07T23:19:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:39.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:39.332098+0000) 2022-09-07T23:19:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:39.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:39.229890+0000) 2022-09-07T23:19:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:39 smithi055 ceph-mon[97033]: pgmap v3211: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:39 smithi059 ceph-mon[97450]: pgmap v3211: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:40.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:40.332346+0000) 2022-09-07T23:19:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:40.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:40.230113+0000) 2022-09-07T23:19:41.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:41 smithi059 ceph-mon[97450]: pgmap v3212: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:19:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:41 smithi055 ceph-mon[97033]: pgmap v3212: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:19:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:41.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:41.332506+0000) 2022-09-07T23:19:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:41.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:41.230337+0000) 2022-09-07T23:19:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:42.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:42.332665+0000) 2022-09-07T23:19:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:42.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:42.230569+0000) 2022-09-07T23:19:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:43.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:43.332873+0000) 2022-09-07T23:19:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:43.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:43.230829+0000) 2022-09-07T23:19:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:43 smithi055 ceph-mon[97033]: pgmap v3213: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:43 smithi059 ceph-mon[97450]: pgmap v3213: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:44.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:44.333135+0000) 2022-09-07T23:19:44.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:44.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:44.231010+0000) 2022-09-07T23:19:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:45.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:45.333370+0000) 2022-09-07T23:19:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:45.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:45.231244+0000) 2022-09-07T23:19:46.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:45 smithi055 ceph-mon[97033]: pgmap v3214: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:45 smithi059 ceph-mon[97450]: pgmap v3214: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:46.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:46.333516+0000) 2022-09-07T23:19:46.673 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:46.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:46.231471+0000) 2022-09-07T23:19:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:47.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:47.333737+0000) 2022-09-07T23:19:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:47.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:47.231741+0000) 2022-09-07T23:19:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:47 smithi055 ceph-mon[97033]: pgmap v3215: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:19:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:47 smithi059 ceph-mon[97450]: pgmap v3215: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:19:48.337 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:19:48.004+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:19:48.338 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:48.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:48.231922+0000) 2022-09-07T23:19:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:48.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:48.333876+0000) 2022-09-07T23:19:49.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:49.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:49.334058+0000) 2022-09-07T23:19:49.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:49.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:49.232138+0000) 2022-09-07T23:19:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:49 smithi055 ceph-mon[97033]: pgmap v3216: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:49 smithi059 ceph-mon[97450]: pgmap v3216: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:50.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:50.334297+0000) 2022-09-07T23:19:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:50.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:50.232375+0000) 2022-09-07T23:19:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:51.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:51.232597+0000) 2022-09-07T23:19:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:51.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:51.334529+0000) 2022-09-07T23:19:52.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:51 smithi055 ceph-mon[97033]: pgmap v3217: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:19:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:51 smithi059 ceph-mon[97450]: pgmap v3217: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:19:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:52.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:52.334711+0000) 2022-09-07T23:19:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:52.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:52.232814+0000) 2022-09-07T23:19:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:53.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:53.334973+0000) 2022-09-07T23:19:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:53.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:53.233036+0000) 2022-09-07T23:19:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:53 smithi055 ceph-mon[97033]: pgmap v3218: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:54.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:53 smithi059 ceph-mon[97450]: pgmap v3218: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:54.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:54.335184+0000) 2022-09-07T23:19:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:54.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:54.233169+0000) 2022-09-07T23:19:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:55.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:55.335412+0000) 2022-09-07T23:19:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:55.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:55.233421+0000) 2022-09-07T23:19:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:55 smithi055 ceph-mon[97033]: pgmap v3219: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:55 smithi059 ceph-mon[97450]: pgmap v3219: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:19:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:56.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:56.335665+0000) 2022-09-07T23:19:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:56.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:56.233698+0000) 2022-09-07T23:19:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:57.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:57.335842+0000) 2022-09-07T23:19:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:57.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:57.233889+0000) 2022-09-07T23:19:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:57 smithi055 ceph-mon[97033]: pgmap v3220: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:19:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:57 smithi059 ceph-mon[97450]: pgmap v3220: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:19:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:58.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:58.336075+0000) 2022-09-07T23:19:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:58.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:58.234111+0000) 2022-09-07T23:19:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:19:59.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:59.336288+0000) 2022-09-07T23:19:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:19:59.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:19:59.234359+0000) 2022-09-07T23:20:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:19:59 smithi055 ceph-mon[97033]: pgmap v3221: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:19:59 smithi059 ceph-mon[97450]: pgmap v3221: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:00.415 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:00.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:00.336544+0000) 2022-09-07T23:20:00.416 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:00.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:00.234531+0000) 2022-09-07T23:20:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-07T23:20:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-07T23:20:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-07T23:20:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T23:20:01.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T23:20:01.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:20:01.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:20:01.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:20:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-07T23:20:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-07T23:20:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-07T23:20:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T23:20:01.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T23:20:01.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:20:01.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:20:01.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:20:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:01.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:01.336811+0000) 2022-09-07T23:20:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:01.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:01.234753+0000) 2022-09-07T23:20:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:01 smithi055 ceph-mon[97033]: pgmap v3222: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:20:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:01 smithi059 ceph-mon[97450]: pgmap v3222: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:20:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:02.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:02.337016+0000) 2022-09-07T23:20:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:02.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:02.234939+0000) 2022-09-07T23:20:03.338 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:20:03.014+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:20:03.339 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:03.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:03.235175+0000) 2022-09-07T23:20:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:03.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:03.337188+0000) 2022-09-07T23:20:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:03 smithi055 ceph-mon[97033]: pgmap v3223: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:03 smithi059 ceph-mon[97450]: pgmap v3223: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:04.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:04.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:04.337401+0000) 2022-09-07T23:20:04.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:04.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:04.235421+0000) 2022-09-07T23:20:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:20:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:20:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:20:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:20:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:05.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:05.337685+0000) 2022-09-07T23:20:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:05.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:05.235573+0000) 2022-09-07T23:20:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:05 smithi055 ceph-mon[97033]: pgmap v3224: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:05 smithi059 ceph-mon[97450]: pgmap v3224: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:06.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:06.337925+0000) 2022-09-07T23:20:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:06.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:06.235747+0000) 2022-09-07T23:20:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:07.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:07.338146+0000) 2022-09-07T23:20:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:07.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:07.235990+0000) 2022-09-07T23:20:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:07 smithi055 ceph-mon[97033]: pgmap v3225: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:20:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:07 smithi059 ceph-mon[97450]: pgmap v3225: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:20:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:08.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:08.338377+0000) 2022-09-07T23:20:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:08.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:08.236238+0000) 2022-09-07T23:20:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:09.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:09.338613+0000) 2022-09-07T23:20:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:09.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:09.236411+0000) 2022-09-07T23:20:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:09 smithi055 ceph-mon[97033]: pgmap v3226: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:20:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:09 smithi055 ceph-mon[97033]: osdmap e291: 12 total, 11 up, 11 in 2022-09-07T23:20:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:09 smithi059 ceph-mon[97450]: pgmap v3226: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:20:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:09 smithi059 ceph-mon[97450]: osdmap e291: 12 total, 11 up, 11 in 2022-09-07T23:20:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:10.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:10.338881+0000) 2022-09-07T23:20:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:10.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:10.236630+0000) 2022-09-07T23:20:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:11.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:11.339060+0000) 2022-09-07T23:20:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:11.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:11.236896+0000) 2022-09-07T23:20:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:11 smithi055 ceph-mon[97033]: pgmap v3228: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:20:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:11 smithi059 ceph-mon[97450]: pgmap v3228: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:20:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:12.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:12.339340+0000) 2022-09-07T23:20:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:12.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:12.237098+0000) 2022-09-07T23:20:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:13.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:13.339585+0000) 2022-09-07T23:20:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:13.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:13.237284+0000) 2022-09-07T23:20:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:13 smithi055 ceph-mon[97033]: pgmap v3229: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:20:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:13 smithi055 ceph-mon[97033]: osdmap e292: 12 total, 11 up, 11 in 2022-09-07T23:20:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:13 smithi059 ceph-mon[97450]: pgmap v3229: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:20:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:13 smithi059 ceph-mon[97450]: osdmap e292: 12 total, 11 up, 11 in 2022-09-07T23:20:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:14.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:14.339809+0000) 2022-09-07T23:20:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:14.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:14.237521+0000) 2022-09-07T23:20:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:15.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:15.237792+0000) 2022-09-07T23:20:15.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:15.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:15.340038+0000) 2022-09-07T23:20:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:15 smithi055 ceph-mon[97033]: pgmap v3231: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:20:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:15 smithi059 ceph-mon[97450]: pgmap v3231: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:20:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:16.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:16.340195+0000) 2022-09-07T23:20:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:16.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:16.238017+0000) 2022-09-07T23:20:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:16 smithi059 ceph-mon[97450]: pgmap v3232: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:20:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:16 smithi055 ceph-mon[97033]: pgmap v3232: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:20:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:17.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:17.340422+0000) 2022-09-07T23:20:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:17.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:17.238216+0000) 2022-09-07T23:20:18.342 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:18.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:18.238416+0000) 2022-09-07T23:20:18.342 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:20:18.023+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:20:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:18.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:18.340570+0000) 2022-09-07T23:20:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:19.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:19.340706+0000) 2022-09-07T23:20:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:19.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:19.238649+0000) 2022-09-07T23:20:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:19 smithi055 ceph-mon[97033]: pgmap v3233: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-07T23:20:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:19 smithi059 ceph-mon[97450]: pgmap v3233: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-07T23:20:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:20.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:20.340940+0000) 2022-09-07T23:20:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:20.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:20.238778+0000) 2022-09-07T23:20:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:21.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:21.341163+0000) 2022-09-07T23:20:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:21.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:21.239009+0000) 2022-09-07T23:20:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:21 smithi055 ceph-mon[97033]: pgmap v3234: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:20:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:21 smithi059 ceph-mon[97450]: pgmap v3234: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:20:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:22.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:22.341380+0000) 2022-09-07T23:20:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:22.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:22.239252+0000) 2022-09-07T23:20:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:23.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:23.341628+0000) 2022-09-07T23:20:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:23.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:23.239472+0000) 2022-09-07T23:20:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:23 smithi055 ceph-mon[97033]: pgmap v3235: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:20:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:23 smithi059 ceph-mon[97450]: pgmap v3235: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:20:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:24.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:24.341845+0000) 2022-09-07T23:20:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:24.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:24.239749+0000) 2022-09-07T23:20:25.530 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:25.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:25.239936+0000) 2022-09-07T23:20:25.530 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:25.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:25.342064+0000) 2022-09-07T23:20:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:25 smithi055 ceph-mon[97033]: pgmap v3236: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-07T23:20:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:25 smithi059 ceph-mon[97450]: pgmap v3236: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-07T23:20:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:26.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:26.342234+0000) 2022-09-07T23:20:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:26.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:26.240192+0000) 2022-09-07T23:20:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:27.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:27.342425+0000) 2022-09-07T23:20:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:27.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:27.240404+0000) 2022-09-07T23:20:28.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:27 smithi055 ceph-mon[97033]: pgmap v3237: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:20:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:27 smithi059 ceph-mon[97450]: pgmap v3237: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:20:28.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:28.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:28.342698+0000) 2022-09-07T23:20:28.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:28.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:28.240662+0000) 2022-09-07T23:20:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:29.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:29.342975+0000) 2022-09-07T23:20:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:29.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:29.240793+0000) 2022-09-07T23:20:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:29 smithi055 ceph-mon[97033]: pgmap v3238: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:29 smithi059 ceph-mon[97450]: pgmap v3238: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:30.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:30.343167+0000) 2022-09-07T23:20:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:30.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:30.241042+0000) 2022-09-07T23:20:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:31.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:31.343339+0000) 2022-09-07T23:20:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:31.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:31.241292+0000) 2022-09-07T23:20:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:31 smithi055 ceph-mon[97033]: pgmap v3239: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:20:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:31 smithi059 ceph-mon[97450]: pgmap v3239: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:20:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:32.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:32.343559+0000) 2022-09-07T23:20:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:32.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:32.241552+0000) 2022-09-07T23:20:33.345 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:20:33.034+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:20:33.346 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:33.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:33.241802+0000) 2022-09-07T23:20:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:33.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:33.343785+0000) 2022-09-07T23:20:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:33 smithi055 ceph-mon[97033]: pgmap v3240: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:33 smithi059 ceph-mon[97450]: pgmap v3240: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:34.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:34.343999+0000) 2022-09-07T23:20:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:34.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:34.241992+0000) 2022-09-07T23:20:35.345 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:35.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:35.242287+0000) 2022-09-07T23:20:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:35.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:35.344197+0000) 2022-09-07T23:20:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:35 smithi055 ceph-mon[97033]: pgmap v3241: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:36.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:20:36.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:20:36.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:20:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:35 smithi059 ceph-mon[97450]: pgmap v3241: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:20:36.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:20:36.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:20:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:36.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:36.344396+0000) 2022-09-07T23:20:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:36.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:36.242542+0000) 2022-09-07T23:20:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:36 smithi055 ceph-mon[97033]: pgmap v3242: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-07T23:20:37.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:36 smithi055 ceph-mon[97033]: pgmap v3243: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-07T23:20:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:36 smithi059 ceph-mon[97450]: pgmap v3242: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-07T23:20:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:36 smithi059 ceph-mon[97450]: pgmap v3243: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-07T23:20:37.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:37.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:37.344629+0000) 2022-09-07T23:20:37.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:37.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:37.242710+0000) 2022-09-07T23:20:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:38.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:38.344798+0000) 2022-09-07T23:20:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:38.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:38.242943+0000) 2022-09-07T23:20:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:38 smithi055 ceph-mon[97033]: pgmap v3244: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-07T23:20:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:38 smithi059 ceph-mon[97450]: pgmap v3244: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-07T23:20:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:39.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:39.345017+0000) 2022-09-07T23:20:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:39.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:39.243087+0000) 2022-09-07T23:20:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:40.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:40.345273+0000) 2022-09-07T23:20:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:40.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:40.243348+0000) 2022-09-07T23:20:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:40 smithi055 ceph-mon[97033]: pgmap v3245: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-07T23:20:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:40 smithi059 ceph-mon[97450]: pgmap v3245: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-07T23:20:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:41.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:41.243568+0000) 2022-09-07T23:20:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:41.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:41.345540+0000) 2022-09-07T23:20:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:42.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:42.345836+0000) 2022-09-07T23:20:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:42.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:42.243733+0000) 2022-09-07T23:20:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:42 smithi055 ceph-mon[97033]: pgmap v3246: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-07T23:20:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:42 smithi059 ceph-mon[97450]: pgmap v3246: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-07T23:20:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:43.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:43.346125+0000) 2022-09-07T23:20:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:43.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:43.243964+0000) 2022-09-07T23:20:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:44.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:44.346392+0000) 2022-09-07T23:20:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:44.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:44.244175+0000) 2022-09-07T23:20:45.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:44 smithi055 ceph-mon[97033]: pgmap v3247: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-07T23:20:45.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:44 smithi059 ceph-mon[97450]: pgmap v3247: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-07T23:20:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:45.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:45.346607+0000) 2022-09-07T23:20:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:45.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:45.244399+0000) 2022-09-07T23:20:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:46.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:46.346849+0000) 2022-09-07T23:20:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:46.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:46.244699+0000) 2022-09-07T23:20:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:46 smithi055 ceph-mon[97033]: pgmap v3248: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:20:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:46 smithi059 ceph-mon[97450]: pgmap v3248: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:20:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:47.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:47.347090+0000) 2022-09-07T23:20:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:47.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:47.244887+0000) 2022-09-07T23:20:48.348 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:20:48.051+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:20:48.349 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:48.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:48.245082+0000) 2022-09-07T23:20:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:48.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:48.347258+0000) 2022-09-07T23:20:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:48 smithi055 ceph-mon[97033]: pgmap v3249: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:48 smithi059 ceph-mon[97450]: pgmap v3249: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:49.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:49.347472+0000) 2022-09-07T23:20:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:49.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:49.245318+0000) 2022-09-07T23:20:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:50.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:50.347746+0000) 2022-09-07T23:20:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:50.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:50.245569+0000) 2022-09-07T23:20:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:50 smithi055 ceph-mon[97033]: pgmap v3250: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:50 smithi059 ceph-mon[97450]: pgmap v3250: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:51.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:51.348039+0000) 2022-09-07T23:20:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:51.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:51.245797+0000) 2022-09-07T23:20:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:52.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:52.348268+0000) 2022-09-07T23:20:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:52.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:52.245978+0000) 2022-09-07T23:20:53.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:52 smithi055 ceph-mon[97033]: pgmap v3251: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:20:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:52 smithi059 ceph-mon[97450]: pgmap v3251: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:20:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:53.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:53.348486+0000) 2022-09-07T23:20:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:53.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:53.246148+0000) 2022-09-07T23:20:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:54.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:54.246363+0000) 2022-09-07T23:20:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:54.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:54.348737+0000) 2022-09-07T23:20:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:54 smithi055 ceph-mon[97033]: pgmap v3252: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:54 smithi059 ceph-mon[97450]: pgmap v3252: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:55.353 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:55.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:55.246566+0000) 2022-09-07T23:20:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:55.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:55.348953+0000) 2022-09-07T23:20:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:56.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:56.349119+0000) 2022-09-07T23:20:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:56.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:56.246770+0000) 2022-09-07T23:20:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:56 smithi055 ceph-mon[97033]: pgmap v3253: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:20:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:56 smithi059 ceph-mon[97450]: pgmap v3253: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:20:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:57.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:57.349320+0000) 2022-09-07T23:20:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:57.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:57.246999+0000) 2022-09-07T23:20:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:58.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:58.349579+0000) 2022-09-07T23:20:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:58.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:58.247240+0000) 2022-09-07T23:20:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:20:58 smithi055 ceph-mon[97033]: pgmap v3254: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:20:58 smithi059 ceph-mon[97450]: pgmap v3254: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:20:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:20:59.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:59.349785+0000) 2022-09-07T23:20:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:20:59.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:20:59.247455+0000) 2022-09-07T23:21:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:00.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:00.349991+0000) 2022-09-07T23:21:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:00.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:00.247740+0000) 2022-09-07T23:21:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:00 smithi055 ceph-mon[97033]: pgmap v3255: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:01.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:00 smithi059 ceph-mon[97450]: pgmap v3255: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:01.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:01.350197+0000) 2022-09-07T23:21:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:01.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:01.247891+0000) 2022-09-07T23:21:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:02.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:02.350411+0000) 2022-09-07T23:21:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:02.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:02.248128+0000) 2022-09-07T23:21:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:02 smithi055 ceph-mon[97033]: pgmap v3256: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:21:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:02 smithi059 ceph-mon[97450]: pgmap v3256: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:21:03.352 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:21:03.067+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:21:03.353 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:03.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:03.248320+0000) 2022-09-07T23:21:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:03.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:03.350660+0000) 2022-09-07T23:21:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:04.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:04.350836+0000) 2022-09-07T23:21:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:04.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:04.248566+0000) 2022-09-07T23:21:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:04 smithi055 ceph-mon[97033]: pgmap v3257: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:21:05.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:21:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:04 smithi059 ceph-mon[97450]: pgmap v3257: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:21:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:21:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:05.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:05.351057+0000) 2022-09-07T23:21:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:05.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:05.248808+0000) 2022-09-07T23:21:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:06.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:06.351200+0000) 2022-09-07T23:21:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:06.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:06.249042+0000) 2022-09-07T23:21:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:06 smithi055 ceph-mon[97033]: pgmap v3258: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:21:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:06 smithi059 ceph-mon[97450]: pgmap v3258: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:21:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:07.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:07.351400+0000) 2022-09-07T23:21:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:07.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:07.249296+0000) 2022-09-07T23:21:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:08.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:08.351541+0000) 2022-09-07T23:21:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:08.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:08.249517+0000) 2022-09-07T23:21:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:08 smithi055 ceph-mon[97033]: pgmap v3259: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:21:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:08 smithi059 ceph-mon[97450]: pgmap v3259: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:21:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:09.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:09.249761+0000) 2022-09-07T23:21:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:09.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:09.351776+0000) 2022-09-07T23:21:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:09 smithi055 ceph-mon[97033]: osdmap e293: 12 total, 11 up, 11 in 2022-09-07T23:21:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:09 smithi059 ceph-mon[97450]: osdmap e293: 12 total, 11 up, 11 in 2022-09-07T23:21:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:10.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:10.249973+0000) 2022-09-07T23:21:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:10.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:10.352027+0000) 2022-09-07T23:21:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:10 smithi055 ceph-mon[97033]: pgmap v3261: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:21:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:10 smithi059 ceph-mon[97450]: pgmap v3261: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:21:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:11.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:11.250202+0000) 2022-09-07T23:21:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:11.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:11.352281+0000) 2022-09-07T23:21:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:12.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:12.250442+0000) 2022-09-07T23:21:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:12.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:12.352541+0000) 2022-09-07T23:21:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:12 smithi055 ceph-mon[97033]: pgmap v3262: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:21:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:12 smithi059 ceph-mon[97450]: pgmap v3262: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:21:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:13.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:13.352788+0000) 2022-09-07T23:21:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:13.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:13.250722+0000) 2022-09-07T23:21:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:14.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:14.353010+0000) 2022-09-07T23:21:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:14.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:14.250955+0000) 2022-09-07T23:21:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:14 smithi055 ceph-mon[97033]: pgmap v3263: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:21:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:14 smithi055 ceph-mon[97033]: osdmap e294: 12 total, 11 up, 11 in 2022-09-07T23:21:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:14 smithi059 ceph-mon[97450]: pgmap v3263: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:21:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:14 smithi059 ceph-mon[97450]: osdmap e294: 12 total, 11 up, 11 in 2022-09-07T23:21:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:15.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:15.353211+0000) 2022-09-07T23:21:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:15.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:15.251195+0000) 2022-09-07T23:21:16.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:16.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:16.353446+0000) 2022-09-07T23:21:16.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:16.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:16.251344+0000) 2022-09-07T23:21:17.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:16 smithi055 ceph-mon[97033]: pgmap v3265: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:21:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:16 smithi059 ceph-mon[97450]: pgmap v3265: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:21:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:17.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:17.353593+0000) 2022-09-07T23:21:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:17.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:17.251581+0000) 2022-09-07T23:21:18.355 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:21:18.081+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:21:18.356 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:18.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:18.251821+0000) 2022-09-07T23:21:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:18.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:18.353818+0000) 2022-09-07T23:21:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:18 smithi055 ceph-mon[97033]: pgmap v3266: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-07T23:21:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:18 smithi059 ceph-mon[97450]: pgmap v3266: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-07T23:21:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:19.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:19.353969+0000) 2022-09-07T23:21:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:19.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:19.252025+0000) 2022-09-07T23:21:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:20.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:20.354166+0000) 2022-09-07T23:21:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:20.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:20.252239+0000) 2022-09-07T23:21:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:20 smithi055 ceph-mon[97033]: pgmap v3267: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:21:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:20 smithi059 ceph-mon[97450]: pgmap v3267: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:21:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:21.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:21.354394+0000) 2022-09-07T23:21:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:21.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:21.252473+0000) 2022-09-07T23:21:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:22.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:22.354720+0000) 2022-09-07T23:21:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:22.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:22.252736+0000) 2022-09-07T23:21:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:22 smithi055 ceph-mon[97033]: pgmap v3268: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:21:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:22 smithi059 ceph-mon[97450]: pgmap v3268: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:21:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:23.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:23.354887+0000) 2022-09-07T23:21:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:23.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:23.252884+0000) 2022-09-07T23:21:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:24.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:24.355048+0000) 2022-09-07T23:21:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:24.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:24.253118+0000) 2022-09-07T23:21:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:24 smithi055 ceph-mon[97033]: pgmap v3269: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:21:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:24 smithi059 ceph-mon[97450]: pgmap v3269: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:21:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:25.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:25.355241+0000) 2022-09-07T23:21:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:25.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:25.253372+0000) 2022-09-07T23:21:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:26.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:26.355390+0000) 2022-09-07T23:21:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:26.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:26.253606+0000) 2022-09-07T23:21:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:26 smithi055 ceph-mon[97033]: pgmap v3270: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 92 B/s wr, 8 op/s 2022-09-07T23:21:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:26 smithi059 ceph-mon[97450]: pgmap v3270: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 92 B/s wr, 8 op/s 2022-09-07T23:21:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:27.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:27.355661+0000) 2022-09-07T23:21:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:27.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:27.253883+0000) 2022-09-07T23:21:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:28.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:28.355975+0000) 2022-09-07T23:21:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:28.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:28.254054+0000) 2022-09-07T23:21:29.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:28 smithi055 ceph-mon[97033]: pgmap v3271: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:28 smithi059 ceph-mon[97450]: pgmap v3271: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:29.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:29.356207+0000) 2022-09-07T23:21:29.668 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:29.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:29.254235+0000) 2022-09-07T23:21:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:30.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:30.356398+0000) 2022-09-07T23:21:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:30.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:30.254406+0000) 2022-09-07T23:21:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:30 smithi055 ceph-mon[97033]: pgmap v3272: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:30 smithi059 ceph-mon[97450]: pgmap v3272: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:31.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:31.356662+0000) 2022-09-07T23:21:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:31.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:31.254616+0000) 2022-09-07T23:21:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:32.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:32.356877+0000) 2022-09-07T23:21:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:32.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:32.254891+0000) 2022-09-07T23:21:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:32 smithi055 ceph-mon[97033]: pgmap v3273: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:21:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:32 smithi059 ceph-mon[97450]: pgmap v3273: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:21:33.359 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:21:33.093+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:21:33.360 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:33.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:33.255109+0000) 2022-09-07T23:21:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:33.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:33.357088+0000) 2022-09-07T23:21:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:34.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:34.357277+0000) 2022-09-07T23:21:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:34.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:34.255328+0000) 2022-09-07T23:21:35.226 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:34 smithi055 ceph-mon[97033]: pgmap v3274: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:35.227 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:34 smithi059 ceph-mon[97450]: pgmap v3274: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:35.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:35.357471+0000) 2022-09-07T23:21:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:35.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:35.255550+0000) 2022-09-07T23:21:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:21:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:21:36.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:21:36.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:21:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:21:36.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:21:36.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:21:36.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:21:36.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:36.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:36.357687+0000) 2022-09-07T23:21:36.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:36.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:36.255758+0000) 2022-09-07T23:21:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:36 smithi059 ceph-mon[97450]: pgmap v3275: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:21:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:36 smithi059 ceph-mon[97450]: pgmap v3276: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:21:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:36 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-07T23:21:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:36 smithi055 ceph-mon[97033]: pgmap v3275: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:21:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:36 smithi055 ceph-mon[97033]: pgmap v3276: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:21:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:36 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-07T23:21:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:37.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:37.357833+0000) 2022-09-07T23:21:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:37.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:37.255989+0000) 2022-09-07T23:21:38.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:38.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:38.358038+0000) 2022-09-07T23:21:38.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:38.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:38.256206+0000) 2022-09-07T23:21:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:38 smithi059 ceph-mon[97450]: pgmap v3277: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:21:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:38 smithi055 ceph-mon[97033]: pgmap v3277: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:21:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:39.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:39.358282+0000) 2022-09-07T23:21:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:39.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:39.256456+0000) 2022-09-07T23:21:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:40 smithi055 ceph-mon[97033]: pgmap v3278: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:21:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:40 smithi059 ceph-mon[97450]: pgmap v3278: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:21:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:40.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:40.358536+0000) 2022-09-07T23:21:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:40.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:40.256728+0000) 2022-09-07T23:21:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:41.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:41.358764+0000) 2022-09-07T23:21:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:41.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:41.256903+0000) 2022-09-07T23:21:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:42.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:42.359018+0000) 2022-09-07T23:21:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:42.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:42.257114+0000) 2022-09-07T23:21:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:42 smithi055 ceph-mon[97033]: pgmap v3279: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:21:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:42 smithi059 ceph-mon[97450]: pgmap v3279: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:21:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:43.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:43.359207+0000) 2022-09-07T23:21:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:43.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:43.257286+0000) 2022-09-07T23:21:44.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:44.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:44.359382+0000) 2022-09-07T23:21:44.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:44.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:44.257508+0000) 2022-09-07T23:21:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:44 smithi055 ceph-mon[97033]: pgmap v3280: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:21:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:44 smithi059 ceph-mon[97450]: pgmap v3280: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:21:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:45.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:45.359540+0000) 2022-09-07T23:21:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:45.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:45.257779+0000) 2022-09-07T23:21:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:46.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:46.359698+0000) 2022-09-07T23:21:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:46.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:46.257989+0000) 2022-09-07T23:21:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:46 smithi055 ceph-mon[97033]: pgmap v3281: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:21:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:46 smithi059 ceph-mon[97450]: pgmap v3281: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:21:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:47.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:47.359914+0000) 2022-09-07T23:21:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:47.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:47.258264+0000) 2022-09-07T23:21:48.362 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:48.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:48.258437+0000) 2022-09-07T23:21:48.364 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:21:48.108+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:21:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:48.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:48.360143+0000) 2022-09-07T23:21:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:48 smithi055 ceph-mon[97033]: pgmap v3282: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:49.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:48 smithi059 ceph-mon[97450]: pgmap v3282: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:49.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:49.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:49.360402+0000) 2022-09-07T23:21:49.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:49.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:49.258712+0000) 2022-09-07T23:21:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:50.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:50.360584+0000) 2022-09-07T23:21:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:50.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:50.258964+0000) 2022-09-07T23:21:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:50 smithi055 ceph-mon[97033]: pgmap v3283: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:50 smithi059 ceph-mon[97450]: pgmap v3283: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:51.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:51.360710+0000) 2022-09-07T23:21:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:51.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:51.259142+0000) 2022-09-07T23:21:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:52.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:52.360924+0000) 2022-09-07T23:21:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:52.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:52.259357+0000) 2022-09-07T23:21:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:52 smithi055 ceph-mon[97033]: pgmap v3284: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:21:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:52 smithi059 ceph-mon[97450]: pgmap v3284: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:21:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:53.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:53.361168+0000) 2022-09-07T23:21:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:53.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:53.259581+0000) 2022-09-07T23:21:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:54.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:54.361387+0000) 2022-09-07T23:21:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:54.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:54.259774+0000) 2022-09-07T23:21:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:54 smithi055 ceph-mon[97033]: pgmap v3285: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:54 smithi059 ceph-mon[97450]: pgmap v3285: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:55.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:55.361608+0000) 2022-09-07T23:21:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:55.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:55.260025+0000) 2022-09-07T23:21:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:56.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:56.361864+0000) 2022-09-07T23:21:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:56.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:56.260192+0000) 2022-09-07T23:21:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:56 smithi055 ceph-mon[97033]: pgmap v3286: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:21:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:56 smithi059 ceph-mon[97450]: pgmap v3286: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:21:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:57.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:57.362034+0000) 2022-09-07T23:21:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:57.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:57.260408+0000) 2022-09-07T23:21:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:58.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:58.362233+0000) 2022-09-07T23:21:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:58.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:58.260681+0000) 2022-09-07T23:21:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:21:58 smithi059 ceph-mon[97450]: pgmap v3287: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:21:58 smithi055 ceph-mon[97033]: pgmap v3287: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:21:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:21:59.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:59.362478+0000) 2022-09-07T23:21:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:21:59.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:21:59.260903+0000) 2022-09-07T23:22:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:00.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:00.362649+0000) 2022-09-07T23:22:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:00.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:00.261158+0000) 2022-09-07T23:22:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:00 smithi059 ceph-mon[97450]: pgmap v3288: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:00 smithi055 ceph-mon[97033]: pgmap v3288: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:01.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:01.362893+0000) 2022-09-07T23:22:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:01.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:01.261345+0000) 2022-09-07T23:22:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:02.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:02.363143+0000) 2022-09-07T23:22:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:02.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:02.261538+0000) 2022-09-07T23:22:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:02 smithi059 ceph-mon[97450]: pgmap v3289: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:22:03.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:22:03.119+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:22:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:03.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:03.363363+0000) 2022-09-07T23:22:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:03.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:03.261722+0000) 2022-09-07T23:22:03.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:02 smithi055 ceph-mon[97033]: pgmap v3289: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:22:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:04.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:04.363610+0000) 2022-09-07T23:22:04.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:04.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:04.261965+0000) 2022-09-07T23:22:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:04 smithi055 ceph-mon[97033]: pgmap v3290: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:05.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:22:05.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:22:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:04 smithi059 ceph-mon[97450]: pgmap v3290: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:22:05.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:22:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:05.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:05.363881+0000) 2022-09-07T23:22:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:05.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:05.262135+0000) 2022-09-07T23:22:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:06.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:06.364048+0000) 2022-09-07T23:22:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:06.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:06.262317+0000) 2022-09-07T23:22:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:06 smithi055 ceph-mon[97033]: pgmap v3291: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:22:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:06 smithi059 ceph-mon[97450]: pgmap v3291: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:22:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:07.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:07.364276+0000) 2022-09-07T23:22:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:07.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:07.262615+0000) 2022-09-07T23:22:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:08.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:08.364498+0000) 2022-09-07T23:22:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:08.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:08.262902+0000) 2022-09-07T23:22:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:08 smithi055 ceph-mon[97033]: pgmap v3292: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:22:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:08 smithi059 ceph-mon[97450]: pgmap v3292: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:22:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:09.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:09.364716+0000) 2022-09-07T23:22:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:09.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:09.263119+0000) 2022-09-07T23:22:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:09 smithi055 ceph-mon[97033]: osdmap e295: 12 total, 11 up, 11 in 2022-09-07T23:22:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:09 smithi059 ceph-mon[97450]: osdmap e295: 12 total, 11 up, 11 in 2022-09-07T23:22:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:10.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:10.364925+0000) 2022-09-07T23:22:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:10.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:10.263321+0000) 2022-09-07T23:22:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:10 smithi055 ceph-mon[97033]: pgmap v3294: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:22:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:10 smithi059 ceph-mon[97450]: pgmap v3294: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:22:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:11.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:11.365161+0000) 2022-09-07T23:22:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:11.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:11.263541+0000) 2022-09-07T23:22:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:12.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:12.365410+0000) 2022-09-07T23:22:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:12.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:12.263790+0000) 2022-09-07T23:22:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:12 smithi059 ceph-mon[97450]: pgmap v3295: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:22:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:12 smithi055 ceph-mon[97033]: pgmap v3295: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:22:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:13.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:13.365686+0000) 2022-09-07T23:22:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:13.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:13.264033+0000) 2022-09-07T23:22:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:14.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:14.365838+0000) 2022-09-07T23:22:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:14.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:14.264287+0000) 2022-09-07T23:22:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:14 smithi055 ceph-mon[97033]: pgmap v3296: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:22:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:14 smithi055 ceph-mon[97033]: osdmap e296: 12 total, 11 up, 11 in 2022-09-07T23:22:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:14 smithi059 ceph-mon[97450]: pgmap v3296: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:22:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:14 smithi059 ceph-mon[97450]: osdmap e296: 12 total, 11 up, 11 in 2022-09-07T23:22:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:15.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:15.366026+0000) 2022-09-07T23:22:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:15.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:15.264501+0000) 2022-09-07T23:22:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:16.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:16.366242+0000) 2022-09-07T23:22:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:16.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:16.264728+0000) 2022-09-07T23:22:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:16 smithi055 ceph-mon[97033]: pgmap v3298: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:22:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:16 smithi059 ceph-mon[97450]: pgmap v3298: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:22:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:17.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:17.366470+0000) 2022-09-07T23:22:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:17.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:17.264885+0000) 2022-09-07T23:22:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:18.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:18.366752+0000) 2022-09-07T23:22:18.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:22:18.135+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:22:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:18.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:18.265071+0000) 2022-09-07T23:22:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:18 smithi055 ceph-mon[97033]: pgmap v3299: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 235 B/s wr, 7 op/s 2022-09-07T23:22:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:18 smithi059 ceph-mon[97450]: pgmap v3299: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 235 B/s wr, 7 op/s 2022-09-07T23:22:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:19.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:19.366956+0000) 2022-09-07T23:22:19.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:19.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:19.265301+0000) 2022-09-07T23:22:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:20.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:20.367195+0000) 2022-09-07T23:22:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:20.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:20.265524+0000) 2022-09-07T23:22:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:20 smithi055 ceph-mon[97033]: pgmap v3300: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:22:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:20 smithi059 ceph-mon[97450]: pgmap v3300: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:22:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:21.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:21.367338+0000) 2022-09-07T23:22:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:21.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:21.265751+0000) 2022-09-07T23:22:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:22.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:22.367550+0000) 2022-09-07T23:22:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:22.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:22.265967+0000) 2022-09-07T23:22:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:22 smithi055 ceph-mon[97033]: pgmap v3301: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:22:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:22 smithi059 ceph-mon[97450]: pgmap v3301: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:22:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:23.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:23.367800+0000) 2022-09-07T23:22:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:23.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:23.266202+0000) 2022-09-07T23:22:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:24.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:24.367931+0000) 2022-09-07T23:22:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:24.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:24.266389+0000) 2022-09-07T23:22:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:24 smithi055 ceph-mon[97033]: pgmap v3302: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:22:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:24 smithi059 ceph-mon[97450]: pgmap v3302: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:22:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:25.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:25.368157+0000) 2022-09-07T23:22:25.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:25.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:25.266669+0000) 2022-09-07T23:22:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:26.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:26.368370+0000) 2022-09-07T23:22:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:26.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:26.266914+0000) 2022-09-07T23:22:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:26 smithi055 ceph-mon[97033]: pgmap v3303: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:22:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:26 smithi059 ceph-mon[97450]: pgmap v3303: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:22:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:27.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:27.368622+0000) 2022-09-07T23:22:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:27.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:27.267165+0000) 2022-09-07T23:22:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:28.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:28.368897+0000) 2022-09-07T23:22:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:28.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:28.267380+0000) 2022-09-07T23:22:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:28 smithi059 ceph-mon[97450]: pgmap v3304: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:29.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:28 smithi055 ceph-mon[97033]: pgmap v3304: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:29.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:29.369141+0000) 2022-09-07T23:22:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:29.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:29.267609+0000) 2022-09-07T23:22:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:30.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:30.369303+0000) 2022-09-07T23:22:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:30.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:30.267871+0000) 2022-09-07T23:22:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:30 smithi059 ceph-mon[97450]: pgmap v3305: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:31.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:30 smithi055 ceph-mon[97033]: pgmap v3305: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:31.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:31.369556+0000) 2022-09-07T23:22:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:31.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:31.268035+0000) 2022-09-07T23:22:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:32.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:32.369759+0000) 2022-09-07T23:22:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:32.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:32.268198+0000) 2022-09-07T23:22:33.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:32 smithi059 ceph-mon[97450]: pgmap v3306: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:22:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:33.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:33.369976+0000) 2022-09-07T23:22:33.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:22:33.146+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:22:33.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:33.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:33.268349+0000) 2022-09-07T23:22:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:32 smithi055 ceph-mon[97033]: pgmap v3306: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:22:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:34.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:34.370102+0000) 2022-09-07T23:22:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:34.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:34.268505+0000) 2022-09-07T23:22:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:34 smithi059 ceph-mon[97450]: pgmap v3307: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:34 smithi055 ceph-mon[97033]: pgmap v3307: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:35.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:35.370314+0000) 2022-09-07T23:22:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:35.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:35.268714+0000) 2022-09-07T23:22:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:22:36.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:22:36.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:22:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:22:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:22:36.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:22:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:36.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:36.370480+0000) 2022-09-07T23:22:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:36.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:36.268909+0000) 2022-09-07T23:22:37.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:36 smithi059 ceph-mon[97450]: pgmap v3308: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:22:37.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:22:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:36 smithi055 ceph-mon[97033]: pgmap v3308: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:22:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:22:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:37.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:37.370721+0000) 2022-09-07T23:22:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:37.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:37.269079+0000) 2022-09-07T23:22:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:38 smithi059 ceph-mon[97450]: pgmap v3309: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:38 smithi055 ceph-mon[97033]: pgmap v3309: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:38.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:38.370973+0000) 2022-09-07T23:22:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:38.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:38.269307+0000) 2022-09-07T23:22:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:39.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:39.371208+0000) 2022-09-07T23:22:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:39.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:39.269544+0000) 2022-09-07T23:22:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:40.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:40.371439+0000) 2022-09-07T23:22:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:40.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:40.269784+0000) 2022-09-07T23:22:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:40 smithi055 ceph-mon[97033]: pgmap v3310: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:41.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:40 smithi059 ceph-mon[97450]: pgmap v3310: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:41.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:41.371632+0000) 2022-09-07T23:22:41.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:41.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:41.270033+0000) 2022-09-07T23:22:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:42.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:42.371763+0000) 2022-09-07T23:22:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:42.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:42.270270+0000) 2022-09-07T23:22:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:42 smithi055 ceph-mon[97033]: pgmap v3311: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:22:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:42 smithi059 ceph-mon[97450]: pgmap v3311: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:22:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:43.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:43.371944+0000) 2022-09-07T23:22:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:43.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:43.270420+0000) 2022-09-07T23:22:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:44.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:44.270679+0000) 2022-09-07T23:22:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:44.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:44.372072+0000) 2022-09-07T23:22:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:44 smithi055 ceph-mon[97033]: pgmap v3312: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:44 smithi059 ceph-mon[97450]: pgmap v3312: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:45.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:45.372307+0000) 2022-09-07T23:22:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:45.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:45.270908+0000) 2022-09-07T23:22:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:46.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:46.372581+0000) 2022-09-07T23:22:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:46.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:46.271143+0000) 2022-09-07T23:22:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:46 smithi055 ceph-mon[97033]: pgmap v3313: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:22:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:46 smithi059 ceph-mon[97450]: pgmap v3313: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:22:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:47.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:47.372813+0000) 2022-09-07T23:22:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:47.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:47.271337+0000) 2022-09-07T23:22:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:48.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:48.373013+0000) 2022-09-07T23:22:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:48.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:48.271570+0000) 2022-09-07T23:22:48.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:22:48.152+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:22:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:48 smithi055 ceph-mon[97033]: pgmap v3314: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:48 smithi059 ceph-mon[97450]: pgmap v3314: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:49.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:49.271832+0000) 2022-09-07T23:22:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:49.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:49.373233+0000) 2022-09-07T23:22:50.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:50.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:50.272034+0000) 2022-09-07T23:22:50.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:50.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:50.373403+0000) 2022-09-07T23:22:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:50 smithi059 ceph-mon[97450]: pgmap v3315: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:51.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:50 smithi055 ceph-mon[97033]: pgmap v3315: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:51.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:51.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:51.373649+0000) 2022-09-07T23:22:51.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:51.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:51.272285+0000) 2022-09-07T23:22:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:52.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:52.373877+0000) 2022-09-07T23:22:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:52.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:52.272490+0000) 2022-09-07T23:22:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:52 smithi059 ceph-mon[97450]: pgmap v3316: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:22:53.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:52 smithi055 ceph-mon[97033]: pgmap v3316: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:22:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:53.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:53.373998+0000) 2022-09-07T23:22:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:53.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:53.272688+0000) 2022-09-07T23:22:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:54.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:54.374143+0000) 2022-09-07T23:22:54.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:54.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:54.272916+0000) 2022-09-07T23:22:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:54 smithi059 ceph-mon[97450]: pgmap v3317: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:54 smithi055 ceph-mon[97033]: pgmap v3317: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:55.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:55.374352+0000) 2022-09-07T23:22:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:55.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:55.273121+0000) 2022-09-07T23:22:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:56.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:56.374587+0000) 2022-09-07T23:22:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:56.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:56.273365+0000) 2022-09-07T23:22:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:56 smithi059 ceph-mon[97450]: pgmap v3318: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:22:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:56 smithi055 ceph-mon[97033]: pgmap v3318: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:22:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:57.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:57.273554+0000) 2022-09-07T23:22:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:57.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:57.374827+0000) 2022-09-07T23:22:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:58.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:58.374994+0000) 2022-09-07T23:22:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:58.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:58.273733+0000) 2022-09-07T23:22:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:22:58 smithi059 ceph-mon[97450]: pgmap v3319: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:22:58 smithi055 ceph-mon[97033]: pgmap v3319: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:22:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:22:59.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:59.375154+0000) 2022-09-07T23:22:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:22:59.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:22:59.273957+0000) 2022-09-07T23:23:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:00.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:00.375402+0000) 2022-09-07T23:23:00.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:00.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:00.274218+0000) 2022-09-07T23:23:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:00 smithi059 ceph-mon[97450]: pgmap v3320: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:00 smithi055 ceph-mon[97033]: pgmap v3320: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:01.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:01.375685+0000) 2022-09-07T23:23:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:01.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:01.274405+0000) 2022-09-07T23:23:02.380 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:02.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:02.274589+0000) 2022-09-07T23:23:02.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:02 smithi055 ceph-mon[97033]: pgmap v3321: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:23:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:02 smithi059 ceph-mon[97450]: pgmap v3321: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:23:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:02.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:02.375872+0000) 2022-09-07T23:23:03.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:23:03.159+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:23:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:03.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:03.376117+0000) 2022-09-07T23:23:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:03.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:03.274753+0000) 2022-09-07T23:23:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:04.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:04.376283+0000) 2022-09-07T23:23:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:04.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:04.274913+0000) 2022-09-07T23:23:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:04 smithi055 ceph-mon[97033]: pgmap v3322: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:23:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:23:05.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:04 smithi059 ceph-mon[97450]: pgmap v3322: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:05.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:23:05.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:23:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:05.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:05.376484+0000) 2022-09-07T23:23:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:05.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:05.275135+0000) 2022-09-07T23:23:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:06.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:06.376725+0000) 2022-09-07T23:23:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:06.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:06.275363+0000) 2022-09-07T23:23:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:06 smithi055 ceph-mon[97033]: pgmap v3323: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:23:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:06 smithi059 ceph-mon[97450]: pgmap v3323: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:23:07.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:07.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:07.376876+0000) 2022-09-07T23:23:07.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:07.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:07.275619+0000) 2022-09-07T23:23:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:08.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:08.377096+0000) 2022-09-07T23:23:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:08.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:08.275866+0000) 2022-09-07T23:23:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:08 smithi059 ceph-mon[97450]: pgmap v3324: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:23:09.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:08 smithi055 ceph-mon[97033]: pgmap v3324: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:23:09.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:09.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:09.377345+0000) 2022-09-07T23:23:09.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:09.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:09.276052+0000) 2022-09-07T23:23:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:09 smithi059 ceph-mon[97450]: osdmap e297: 12 total, 11 up, 11 in 2022-09-07T23:23:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:09 smithi055 ceph-mon[97033]: osdmap e297: 12 total, 11 up, 11 in 2022-09-07T23:23:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:10.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:10.377593+0000) 2022-09-07T23:23:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:10.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:10.276310+0000) 2022-09-07T23:23:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:10 smithi059 ceph-mon[97450]: pgmap v3326: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:23:11.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:10 smithi055 ceph-mon[97033]: pgmap v3326: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:23:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:11.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:11.377796+0000) 2022-09-07T23:23:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:11.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:11.276544+0000) 2022-09-07T23:23:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:12.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:12.378008+0000) 2022-09-07T23:23:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:12.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:12.276783+0000) 2022-09-07T23:23:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:12 smithi059 ceph-mon[97450]: pgmap v3327: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:23:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:12 smithi055 ceph-mon[97033]: pgmap v3327: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:23:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:13.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:13.378260+0000) 2022-09-07T23:23:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:13.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:13.276929+0000) 2022-09-07T23:23:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:14.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:14.378521+0000) 2022-09-07T23:23:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:14.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:14.277186+0000) 2022-09-07T23:23:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:14 smithi055 ceph-mon[97033]: pgmap v3328: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:23:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:14 smithi055 ceph-mon[97033]: osdmap e298: 12 total, 11 up, 11 in 2022-09-07T23:23:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:14 smithi059 ceph-mon[97450]: pgmap v3328: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:23:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:14 smithi059 ceph-mon[97450]: osdmap e298: 12 total, 11 up, 11 in 2022-09-07T23:23:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:15.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:15.378732+0000) 2022-09-07T23:23:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:15.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:15.277413+0000) 2022-09-07T23:23:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:16.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:16.378922+0000) 2022-09-07T23:23:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:16.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:16.277587+0000) 2022-09-07T23:23:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:16 smithi055 ceph-mon[97033]: pgmap v3330: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:23:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:16 smithi059 ceph-mon[97450]: pgmap v3330: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:23:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:17.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:17.379177+0000) 2022-09-07T23:23:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:17.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:17.277770+0000) 2022-09-07T23:23:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:18.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:18.379425+0000) 2022-09-07T23:23:18.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:23:18.165+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:23:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:18.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:18.277976+0000) 2022-09-07T23:23:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:18 smithi055 ceph-mon[97033]: pgmap v3331: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 235 B/s wr, 7 op/s 2022-09-07T23:23:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:18 smithi059 ceph-mon[97450]: pgmap v3331: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 235 B/s wr, 7 op/s 2022-09-07T23:23:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:19.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:19.379690+0000) 2022-09-07T23:23:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:19.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:19.278198+0000) 2022-09-07T23:23:20.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:20.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:20.379903+0000) 2022-09-07T23:23:20.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:20.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:20.278413+0000) 2022-09-07T23:23:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:20 smithi059 ceph-mon[97450]: pgmap v3332: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:23:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:20 smithi055 ceph-mon[97033]: pgmap v3332: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:23:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:21.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:21.380127+0000) 2022-09-07T23:23:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:21.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:21.278621+0000) 2022-09-07T23:23:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:22.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:22.380342+0000) 2022-09-07T23:23:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:22.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:22.278824+0000) 2022-09-07T23:23:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:22 smithi059 ceph-mon[97450]: pgmap v3333: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:23:23.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:22 smithi055 ceph-mon[97033]: pgmap v3333: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:23:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:23.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:23.380577+0000) 2022-09-07T23:23:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:23.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:23.279083+0000) 2022-09-07T23:23:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:24.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:24.380839+0000) 2022-09-07T23:23:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:24.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:24.279237+0000) 2022-09-07T23:23:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:24 smithi059 ceph-mon[97450]: pgmap v3334: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:23:25.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:24 smithi055 ceph-mon[97033]: pgmap v3334: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:23:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:25.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:25.381036+0000) 2022-09-07T23:23:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:25.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:25.279468+0000) 2022-09-07T23:23:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:26.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:26.381242+0000) 2022-09-07T23:23:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:26.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:26.279615+0000) 2022-09-07T23:23:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:26 smithi059 ceph-mon[97450]: pgmap v3335: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:23:27.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:26 smithi055 ceph-mon[97033]: pgmap v3335: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:23:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:27.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:27.381439+0000) 2022-09-07T23:23:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:27.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:27.279784+0000) 2022-09-07T23:23:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:28.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:28.381629+0000) 2022-09-07T23:23:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:28.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:28.279955+0000) 2022-09-07T23:23:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:28 smithi059 ceph-mon[97450]: pgmap v3336: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:29.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:28 smithi055 ceph-mon[97033]: pgmap v3336: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:29.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:29.381860+0000) 2022-09-07T23:23:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:29.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:29.280152+0000) 2022-09-07T23:23:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:30.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:30.382036+0000) 2022-09-07T23:23:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:30.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:30.280380+0000) 2022-09-07T23:23:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:30 smithi059 ceph-mon[97450]: pgmap v3337: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:31.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:31 smithi055 ceph-mon[97033]: pgmap v3337: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:31.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:31.382242+0000) 2022-09-07T23:23:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:31.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:31.280631+0000) 2022-09-07T23:23:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:32.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:32.382459+0000) 2022-09-07T23:23:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:32.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:32.280847+0000) 2022-09-07T23:23:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:33 smithi059 ceph-mon[97450]: pgmap v3338: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:23:33.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:33 smithi055 ceph-mon[97033]: pgmap v3338: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:23:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:33.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:33.281022+0000) 2022-09-07T23:23:33.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:23:33.181+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:23:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:33.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:33.382738+0000) 2022-09-07T23:23:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:34.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:34.281174+0000) 2022-09-07T23:23:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:34.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:34.382923+0000) 2022-09-07T23:23:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:34 smithi059 ceph-mon[97450]: pgmap v3339: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:34 smithi055 ceph-mon[97033]: pgmap v3339: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:35.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:35.383109+0000) 2022-09-07T23:23:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:35.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:35.281360+0000) 2022-09-07T23:23:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:36.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:36.383302+0000) 2022-09-07T23:23:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:36.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:36.281560+0000) 2022-09-07T23:23:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:36 smithi055 ceph-mon[97033]: pgmap v3340: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:23:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:23:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:23:37.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:23:37.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:23:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:36 smithi059 ceph-mon[97450]: pgmap v3340: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:23:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:23:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:23:37.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:23:37.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:23:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:37.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:37.383525+0000) 2022-09-07T23:23:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:37.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:37.281801+0000) 2022-09-07T23:23:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:38.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:38.282057+0000) 2022-09-07T23:23:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:38.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:38.383776+0000) 2022-09-07T23:23:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:38 smithi059 ceph-mon[97450]: pgmap v3341: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:38 smithi055 ceph-mon[97033]: pgmap v3341: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:39.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:39.384035+0000) 2022-09-07T23:23:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:39.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:39.282244+0000) 2022-09-07T23:23:40.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:40.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:40.384318+0000) 2022-09-07T23:23:40.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:40.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:40.282430+0000) 2022-09-07T23:23:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:40 smithi059 ceph-mon[97450]: pgmap v3342: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:41.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:40 smithi055 ceph-mon[97033]: pgmap v3342: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:41.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:41.384551+0000) 2022-09-07T23:23:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:41.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:41.282675+0000) 2022-09-07T23:23:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:42.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:42.384728+0000) 2022-09-07T23:23:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:42.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:42.282924+0000) 2022-09-07T23:23:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:42 smithi059 ceph-mon[97450]: pgmap v3343: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:23:43.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:42 smithi055 ceph-mon[97033]: pgmap v3343: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:23:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:43.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:43.384953+0000) 2022-09-07T23:23:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:43.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:43.283176+0000) 2022-09-07T23:23:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:44.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:44.385155+0000) 2022-09-07T23:23:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:44.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:44.283359+0000) 2022-09-07T23:23:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:44 smithi059 ceph-mon[97450]: pgmap v3344: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:45 smithi055 ceph-mon[97033]: pgmap v3344: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:45.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:45.385374+0000) 2022-09-07T23:23:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:45.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:45.283540+0000) 2022-09-07T23:23:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:46.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:46.385613+0000) 2022-09-07T23:23:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:46.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:46.283753+0000) 2022-09-07T23:23:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:47 smithi059 ceph-mon[97450]: pgmap v3345: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:23:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:47 smithi055 ceph-mon[97033]: pgmap v3345: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:23:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:47.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:47.385889+0000) 2022-09-07T23:23:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:47.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:47.284005+0000) 2022-09-07T23:23:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:48.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:48.386138+0000) 2022-09-07T23:23:48.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:23:48.196+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:23:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:48.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:48.284243+0000) 2022-09-07T23:23:49.291 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:49 smithi059 ceph-mon[97450]: pgmap v3346: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:49 smithi055 ceph-mon[97033]: pgmap v3346: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:49.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:49.386380+0000) 2022-09-07T23:23:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:49.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:49.284409+0000) 2022-09-07T23:23:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:50 smithi055 ceph-mon[97033]: pgmap v3347: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:50 smithi059 ceph-mon[97450]: pgmap v3347: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:23:50.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:50.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:50.386584+0000) 2022-09-07T23:23:50.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:50.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:50.284612+0000) 2022-09-07T23:23:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:51.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:51.386808+0000) 2022-09-07T23:23:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:51.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:51.284833+0000) 2022-09-07T23:23:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:52.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:52.387077+0000) 2022-09-07T23:23:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:52.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:52.285004+0000) 2022-09-07T23:23:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:52 smithi059 ceph-mon[97450]: pgmap v3348: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:23:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:52 smithi055 ceph-mon[97033]: pgmap v3348: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:23:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:53.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:53.387266+0000) 2022-09-07T23:23:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:53.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:53.285175+0000) 2022-09-07T23:23:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:54.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:54.387473+0000) 2022-09-07T23:23:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:54.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:54.285345+0000) 2022-09-07T23:23:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:54 smithi059 ceph-mon[97450]: pgmap v3349: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:23:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:54 smithi055 ceph-mon[97033]: pgmap v3349: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:23:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:55.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:55.387699+0000) 2022-09-07T23:23:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:55.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:55.285544+0000) 2022-09-07T23:23:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:56.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:56.387926+0000) 2022-09-07T23:23:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:56.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:56.285778+0000) 2022-09-07T23:23:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:56 smithi059 ceph-mon[97450]: pgmap v3350: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:23:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:56 smithi055 ceph-mon[97033]: pgmap v3350: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:23:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:57.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:57.388164+0000) 2022-09-07T23:23:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:57.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:57.285949+0000) 2022-09-07T23:23:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:58.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:58.388360+0000) 2022-09-07T23:23:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:58.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:58.286178+0000) 2022-09-07T23:23:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:23:58 smithi059 ceph-mon[97450]: pgmap v3351: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:23:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:23:58 smithi055 ceph-mon[97033]: pgmap v3351: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:23:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:23:59.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:59.388498+0000) 2022-09-07T23:23:59.746 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:23:59.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:23:59.286433+0000) 2022-09-07T23:24:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:00.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:00.388749+0000) 2022-09-07T23:24:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:00.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:00.286691+0000) 2022-09-07T23:24:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:00 smithi059 ceph-mon[97450]: pgmap v3352: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:24:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:00 smithi055 ceph-mon[97033]: pgmap v3352: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:24:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:01.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:01.388937+0000) 2022-09-07T23:24:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:01.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:01.286905+0000) 2022-09-07T23:24:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:02.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:02.287049+0000) 2022-09-07T23:24:02.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:02.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:02.389161+0000) 2022-09-07T23:24:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:02 smithi059 ceph-mon[97450]: pgmap v3353: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:24:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:02 smithi055 ceph-mon[97033]: pgmap v3353: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:24:03.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:24:03.213+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:24:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:03.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:03.389366+0000) 2022-09-07T23:24:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:03.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:03.287285+0000) 2022-09-07T23:24:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:04.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:04.389566+0000) 2022-09-07T23:24:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:04.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:04.287526+0000) 2022-09-07T23:24:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:04 smithi059 ceph-mon[97450]: pgmap v3354: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:24:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:24:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:24:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:05 smithi055 ceph-mon[97033]: pgmap v3354: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:24:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:24:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:24:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:05.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:05.389829+0000) 2022-09-07T23:24:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:05.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:05.287774+0000) 2022-09-07T23:24:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:06.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:06.390004+0000) 2022-09-07T23:24:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:06.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:06.288028+0000) 2022-09-07T23:24:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:07 smithi059 ceph-mon[97450]: pgmap v3355: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:24:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:07 smithi055 ceph-mon[97033]: pgmap v3355: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:24:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:07.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:07.390243+0000) 2022-09-07T23:24:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:07.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:07.288262+0000) 2022-09-07T23:24:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:08.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:08.390494+0000) 2022-09-07T23:24:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:08.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:08.288509+0000) 2022-09-07T23:24:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:09 smithi059 ceph-mon[97450]: pgmap v3356: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:24:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:09 smithi055 ceph-mon[97033]: pgmap v3356: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:24:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:09.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:09.390680+0000) 2022-09-07T23:24:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:09.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:09.288716+0000) 2022-09-07T23:24:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:10 smithi059 ceph-mon[97450]: osdmap e299: 12 total, 11 up, 11 in 2022-09-07T23:24:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:10 smithi059 ceph-mon[97450]: pgmap v3358: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:24:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:10 smithi055 ceph-mon[97033]: osdmap e299: 12 total, 11 up, 11 in 2022-09-07T23:24:10.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:10 smithi055 ceph-mon[97033]: pgmap v3358: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:24:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:10.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:10.390934+0000) 2022-09-07T23:24:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:10.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:10.288884+0000) 2022-09-07T23:24:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:11.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:11.391146+0000) 2022-09-07T23:24:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:11.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:11.289028+0000) 2022-09-07T23:24:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:12.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:12.391355+0000) 2022-09-07T23:24:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:12.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:12.289262+0000) 2022-09-07T23:24:13.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:12 smithi059 ceph-mon[97450]: pgmap v3359: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:24:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:12 smithi055 ceph-mon[97033]: pgmap v3359: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:24:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:13.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:13.391541+0000) 2022-09-07T23:24:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:13.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:13.289431+0000) 2022-09-07T23:24:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:14.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:14.391795+0000) 2022-09-07T23:24:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:14.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:14.289712+0000) 2022-09-07T23:24:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:14 smithi059 ceph-mon[97450]: pgmap v3360: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:24:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:14 smithi059 ceph-mon[97450]: osdmap e300: 12 total, 11 up, 11 in 2022-09-07T23:24:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:14 smithi055 ceph-mon[97033]: pgmap v3360: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:24:15.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:14 smithi055 ceph-mon[97033]: osdmap e300: 12 total, 11 up, 11 in 2022-09-07T23:24:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:15.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:15.391984+0000) 2022-09-07T23:24:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:15.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:15.289946+0000) 2022-09-07T23:24:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:16.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:16.392204+0000) 2022-09-07T23:24:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:16.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:16.290105+0000) 2022-09-07T23:24:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:16 smithi059 ceph-mon[97450]: pgmap v3362: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:24:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:16 smithi055 ceph-mon[97033]: pgmap v3362: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:24:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:17.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:17.392433+0000) 2022-09-07T23:24:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:17.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:17.290338+0000) 2022-09-07T23:24:18.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:24:18.228+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:24:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:18.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:18.392718+0000) 2022-09-07T23:24:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:18.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:18.290508+0000) 2022-09-07T23:24:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:19 smithi059 ceph-mon[97450]: pgmap v3363: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 235 B/s wr, 7 op/s 2022-09-07T23:24:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:19 smithi055 ceph-mon[97033]: pgmap v3363: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 235 B/s wr, 7 op/s 2022-09-07T23:24:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:19.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:19.392970+0000) 2022-09-07T23:24:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:19.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:19.290733+0000) 2022-09-07T23:24:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:20.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:20.393222+0000) 2022-09-07T23:24:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:20.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:20.290953+0000) 2022-09-07T23:24:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:21 smithi059 ceph-mon[97450]: pgmap v3364: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:24:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:21 smithi055 ceph-mon[97033]: pgmap v3364: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:24:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:21.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:21.393456+0000) 2022-09-07T23:24:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:21.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:21.291124+0000) 2022-09-07T23:24:22.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:22.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:22.393600+0000) 2022-09-07T23:24:22.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:22.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:22.291357+0000) 2022-09-07T23:24:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:23 smithi059 ceph-mon[97450]: pgmap v3365: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:24:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:23 smithi055 ceph-mon[97033]: pgmap v3365: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:24:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:23.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:23.393718+0000) 2022-09-07T23:24:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:23.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:23.291503+0000) 2022-09-07T23:24:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:24.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:24.393915+0000) 2022-09-07T23:24:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:24.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:24.291745+0000) 2022-09-07T23:24:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:25 smithi059 ceph-mon[97450]: pgmap v3366: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:24:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:25 smithi055 ceph-mon[97033]: pgmap v3366: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:24:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:25.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:25.394092+0000) 2022-09-07T23:24:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:25.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:25.292009+0000) 2022-09-07T23:24:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:26 smithi055 ceph-mon[97033]: pgmap v3367: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:24:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:26 smithi059 ceph-mon[97450]: pgmap v3367: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:24:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:26.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:26.394320+0000) 2022-09-07T23:24:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:26.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:26.292169+0000) 2022-09-07T23:24:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:27.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:27.394579+0000) 2022-09-07T23:24:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:27.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:27.292374+0000) 2022-09-07T23:24:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:28.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:28.394787+0000) 2022-09-07T23:24:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:28.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:28.292615+0000) 2022-09-07T23:24:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:28 smithi059 ceph-mon[97450]: pgmap v3368: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:24:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:28 smithi055 ceph-mon[97033]: pgmap v3368: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:24:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:29.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:29.395029+0000) 2022-09-07T23:24:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:29.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:29.292873+0000) 2022-09-07T23:24:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:30.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:30.395249+0000) 2022-09-07T23:24:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:30.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:30.293110+0000) 2022-09-07T23:24:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:30 smithi059 ceph-mon[97450]: pgmap v3369: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:24:31.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:30 smithi055 ceph-mon[97033]: pgmap v3369: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:24:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:31.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:31.395453+0000) 2022-09-07T23:24:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:31.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:31.293318+0000) 2022-09-07T23:24:32.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:32.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:32.395727+0000) 2022-09-07T23:24:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:32.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:32.293560+0000) 2022-09-07T23:24:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:32 smithi059 ceph-mon[97450]: pgmap v3370: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:24:33.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:32 smithi055 ceph-mon[97033]: pgmap v3370: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:24:33.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:24:33.243+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:24:33.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:33.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:33.395973+0000) 2022-09-07T23:24:33.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:33.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:33.293809+0000) 2022-09-07T23:24:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:34.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:34.396174+0000) 2022-09-07T23:24:34.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:34.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:34.294041+0000) 2022-09-07T23:24:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:34 smithi059 ceph-mon[97450]: pgmap v3371: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:24:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:34 smithi055 ceph-mon[97033]: pgmap v3371: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:24:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:35.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:35.396408+0000) 2022-09-07T23:24:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:35.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:35.294235+0000) 2022-09-07T23:24:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:36.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:36.396655+0000) 2022-09-07T23:24:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:36.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:36.294485+0000) 2022-09-07T23:24:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:37 smithi059 ceph-mon[97450]: pgmap v3372: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:24:37.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:37 smithi055 ceph-mon[97033]: pgmap v3372: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:24:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:37.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:37.396836+0000) 2022-09-07T23:24:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:37.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:37.294748+0000) 2022-09-07T23:24:38.289 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:24:38.290 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:24:38.291 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:24:38.291 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:24:38.487 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:24:38.488 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:24:38.490 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:24:38.493 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:24:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:38.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:38.397038+0000) 2022-09-07T23:24:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:38.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:38.294941+0000) 2022-09-07T23:24:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:39 smithi059 ceph-mon[97450]: pgmap v3373: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:24:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:39 smithi055 ceph-mon[97033]: pgmap v3373: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:24:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:39.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:39.397250+0000) 2022-09-07T23:24:39.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:39.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:39.295182+0000) 2022-09-07T23:24:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:40.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:40.397465+0000) 2022-09-07T23:24:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:40.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:40.295453+0000) 2022-09-07T23:24:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:41 smithi059 ceph-mon[97450]: pgmap v3374: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:24:41.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:41 smithi055 ceph-mon[97033]: pgmap v3374: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:24:41.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:41.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:41.397679+0000) 2022-09-07T23:24:41.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:41.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:41.295709+0000) 2022-09-07T23:24:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:42.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:42.397909+0000) 2022-09-07T23:24:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:42.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:42.295935+0000) 2022-09-07T23:24:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:43 smithi059 ceph-mon[97450]: pgmap v3375: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:24:43.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:43 smithi055 ceph-mon[97033]: pgmap v3375: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:24:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:43.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:43.398106+0000) 2022-09-07T23:24:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:43.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:43.296108+0000) 2022-09-07T23:24:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:44.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:44.398308+0000) 2022-09-07T23:24:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:44.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:44.296301+0000) 2022-09-07T23:24:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:45 smithi059 ceph-mon[97450]: pgmap v3376: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:24:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:45 smithi055 ceph-mon[97033]: pgmap v3376: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:24:45.661 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:45.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:45.398469+0000) 2022-09-07T23:24:45.662 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:45.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:45.296589+0000) 2022-09-07T23:24:46.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:46 smithi055 ceph-mon[97033]: pgmap v3377: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:24:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:46 smithi059 ceph-mon[97450]: pgmap v3377: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:24:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:46.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:46.398625+0000) 2022-09-07T23:24:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:46.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:46.296803+0000) 2022-09-07T23:24:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:47.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:47.398800+0000) 2022-09-07T23:24:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:47.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:47.297025+0000) 2022-09-07T23:24:48.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:24:48.257+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:24:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:48.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:48.399049+0000) 2022-09-07T23:24:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:48.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:48.297282+0000) 2022-09-07T23:24:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:48 smithi059 ceph-mon[97450]: pgmap v3378: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:24:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:48 smithi055 ceph-mon[97033]: pgmap v3378: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:24:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:49.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:49.399262+0000) 2022-09-07T23:24:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:49.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:49.297515+0000) 2022-09-07T23:24:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:50.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:50.399495+0000) 2022-09-07T23:24:50.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:50.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:50.297733+0000) 2022-09-07T23:24:51.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:50 smithi059 ceph-mon[97450]: pgmap v3379: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:24:51.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:51 smithi055 ceph-mon[97033]: pgmap v3379: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:24:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:51.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:51.399759+0000) 2022-09-07T23:24:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:51.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:51.297889+0000) 2022-09-07T23:24:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:52.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:52.399891+0000) 2022-09-07T23:24:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:52.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:52.298092+0000) 2022-09-07T23:24:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:52 smithi059 ceph-mon[97450]: pgmap v3380: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:24:53.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:53 smithi055 ceph-mon[97033]: pgmap v3380: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:24:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:53.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:53.400096+0000) 2022-09-07T23:24:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:53.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:53.298308+0000) 2022-09-07T23:24:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:54.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:54.400322+0000) 2022-09-07T23:24:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:54.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:54.298425+0000) 2022-09-07T23:24:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:55 smithi059 ceph-mon[97450]: pgmap v3381: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:24:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:55 smithi055 ceph-mon[97033]: pgmap v3381: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:24:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:55.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:55.400563+0000) 2022-09-07T23:24:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:55.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:55.298702+0000) 2022-09-07T23:24:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:56.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:56.400740+0000) 2022-09-07T23:24:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:56.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:56.298891+0000) 2022-09-07T23:24:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:57 smithi059 ceph-mon[97450]: pgmap v3382: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:24:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:57 smithi055 ceph-mon[97033]: pgmap v3382: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:24:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:57.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:57.400903+0000) 2022-09-07T23:24:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:57.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:57.299130+0000) 2022-09-07T23:24:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:58.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:58.401137+0000) 2022-09-07T23:24:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:58.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:58.299352+0000) 2022-09-07T23:24:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:24:59 smithi059 ceph-mon[97450]: pgmap v3383: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:24:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:24:59 smithi055 ceph-mon[97033]: pgmap v3383: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:24:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:24:59.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:59.401337+0000) 2022-09-07T23:24:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:24:59.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:24:59.299578+0000) 2022-09-07T23:25:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:00.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:00.401560+0000) 2022-09-07T23:25:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:00.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:00.299820+0000) 2022-09-07T23:25:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:01 smithi059 ceph-mon[97450]: pgmap v3384: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:25:01.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:01 smithi055 ceph-mon[97033]: pgmap v3384: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:25:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:01.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:01.401789+0000) 2022-09-07T23:25:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:01.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:01.300000+0000) 2022-09-07T23:25:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:02 smithi055 ceph-mon[97033]: pgmap v3385: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:25:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:02 smithi059 ceph-mon[97450]: pgmap v3385: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:25:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:02.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:02.401974+0000) 2022-09-07T23:25:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:02.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:02.300218+0000) 2022-09-07T23:25:03.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:25:03.273+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:25:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:03.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:03.402166+0000) 2022-09-07T23:25:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:03.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:03.300445+0000) 2022-09-07T23:25:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:04.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:04.402343+0000) 2022-09-07T23:25:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:04.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:04.300594+0000) 2022-09-07T23:25:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:04 smithi059 ceph-mon[97450]: pgmap v3386: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:25:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:25:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:04 smithi055 ceph-mon[97033]: pgmap v3386: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:25:05.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:25:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:05.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:05.402553+0000) 2022-09-07T23:25:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:05.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:05.300796+0000) 2022-09-07T23:25:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:06.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:06.402757+0000) 2022-09-07T23:25:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:06.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:06.301023+0000) 2022-09-07T23:25:07.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:07 smithi059 ceph-mon[97450]: pgmap v3387: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:25:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:07 smithi055 ceph-mon[97033]: pgmap v3387: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:25:07.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:07.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:07.402981+0000) 2022-09-07T23:25:07.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:07.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:07.301287+0000) 2022-09-07T23:25:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:08.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:08.403158+0000) 2022-09-07T23:25:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:08.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:08.301533+0000) 2022-09-07T23:25:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:09 smithi059 ceph-mon[97450]: pgmap v3388: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:25:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:09 smithi055 ceph-mon[97033]: pgmap v3388: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:25:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:09.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:09.403350+0000) 2022-09-07T23:25:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:09.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:09.301815+0000) 2022-09-07T23:25:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:10 smithi059 ceph-mon[97450]: osdmap e301: 12 total, 11 up, 11 in 2022-09-07T23:25:10.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:10 smithi055 ceph-mon[97033]: osdmap e301: 12 total, 11 up, 11 in 2022-09-07T23:25:10.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:10.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:10.403570+0000) 2022-09-07T23:25:10.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:10.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:10.302040+0000) 2022-09-07T23:25:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:11 smithi059 ceph-mon[97450]: pgmap v3390: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:25:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:11 smithi055 ceph-mon[97033]: pgmap v3390: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:25:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:11.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:11.403795+0000) 2022-09-07T23:25:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:11.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:11.302258+0000) 2022-09-07T23:25:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:12.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:12.403996+0000) 2022-09-07T23:25:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:12.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:12.302484+0000) 2022-09-07T23:25:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:13 smithi059 ceph-mon[97450]: pgmap v3391: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:25:13.492 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:13 smithi055 ceph-mon[97033]: pgmap v3391: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:25:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:13.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:13.404221+0000) 2022-09-07T23:25:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:13.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:13.302722+0000) 2022-09-07T23:25:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:14.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:14.404390+0000) 2022-09-07T23:25:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:14.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:14.302919+0000) 2022-09-07T23:25:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:15 smithi059 ceph-mon[97450]: pgmap v3392: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:25:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:15 smithi059 ceph-mon[97450]: osdmap e302: 12 total, 11 up, 11 in 2022-09-07T23:25:15.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:15 smithi055 ceph-mon[97033]: pgmap v3392: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:25:15.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:15 smithi055 ceph-mon[97033]: osdmap e302: 12 total, 11 up, 11 in 2022-09-07T23:25:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:15.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:15.404546+0000) 2022-09-07T23:25:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:15.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:15.303134+0000) 2022-09-07T23:25:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:16.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:16.404781+0000) 2022-09-07T23:25:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:16.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:16.303364+0000) 2022-09-07T23:25:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:17 smithi059 ceph-mon[97450]: pgmap v3394: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:25:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:17 smithi055 ceph-mon[97033]: pgmap v3394: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:25:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:17.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:17.405047+0000) 2022-09-07T23:25:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:17.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:17.303631+0000) 2022-09-07T23:25:18.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:25:18.276+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:25:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:18.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:18.405219+0000) 2022-09-07T23:25:18.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:18.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:18.303890+0000) 2022-09-07T23:25:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:19 smithi059 ceph-mon[97450]: pgmap v3395: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-07T23:25:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:19 smithi055 ceph-mon[97033]: pgmap v3395: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-07T23:25:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:19.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:19.405463+0000) 2022-09-07T23:25:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:19.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:19.304120+0000) 2022-09-07T23:25:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:20.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:20.405744+0000) 2022-09-07T23:25:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:20.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:20.304368+0000) 2022-09-07T23:25:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:21 smithi059 ceph-mon[97450]: pgmap v3396: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:25:21.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:21 smithi055 ceph-mon[97033]: pgmap v3396: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:25:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:21.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:21.405898+0000) 2022-09-07T23:25:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:21.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:21.304597+0000) 2022-09-07T23:25:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:22.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:22.406056+0000) 2022-09-07T23:25:22.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:22.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:22.304806+0000) 2022-09-07T23:25:23.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:23 smithi059 ceph-mon[97450]: pgmap v3397: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:25:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:23 smithi055 ceph-mon[97033]: pgmap v3397: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:25:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:23.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:23.406271+0000) 2022-09-07T23:25:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:23.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:23.304985+0000) 2022-09-07T23:25:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:24.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:24.406490+0000) 2022-09-07T23:25:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:24.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:24.305181+0000) 2022-09-07T23:25:25.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:25 smithi059 ceph-mon[97450]: pgmap v3398: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:25:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:25 smithi055 ceph-mon[97033]: pgmap v3398: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:25:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:25.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:25.406748+0000) 2022-09-07T23:25:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:25.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:25.305435+0000) 2022-09-07T23:25:26.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:26.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:26.406885+0000) 2022-09-07T23:25:26.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:26.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:26.305698+0000) 2022-09-07T23:25:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:27 smithi059 ceph-mon[97450]: pgmap v3399: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:25:27.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:27 smithi055 ceph-mon[97033]: pgmap v3399: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:25:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:27.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:27.407117+0000) 2022-09-07T23:25:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:27.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:27.305938+0000) 2022-09-07T23:25:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:28 smithi059 ceph-mon[97450]: pgmap v3400: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:28.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:28 smithi055 ceph-mon[97033]: pgmap v3400: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:28.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:28.407344+0000) 2022-09-07T23:25:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:28.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:28.306143+0000) 2022-09-07T23:25:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:29.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:29.407613+0000) 2022-09-07T23:25:29.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:29.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:29.306395+0000) 2022-09-07T23:25:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:30.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:30.407918+0000) 2022-09-07T23:25:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:30.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:30.306619+0000) 2022-09-07T23:25:31.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:31 smithi059 ceph-mon[97450]: pgmap v3401: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:31.494 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:31 smithi055 ceph-mon[97033]: pgmap v3401: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:31.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:31.408136+0000) 2022-09-07T23:25:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:31.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:31.306900+0000) 2022-09-07T23:25:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:32.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:32.408322+0000) 2022-09-07T23:25:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:32.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:32.307176+0000) 2022-09-07T23:25:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:33 smithi059 ceph-mon[97450]: pgmap v3402: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:25:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:33 smithi055 ceph-mon[97033]: pgmap v3402: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:25:33.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:25:33.282+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:25:33.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:33.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:33.408538+0000) 2022-09-07T23:25:33.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:33.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:33.307372+0000) 2022-09-07T23:25:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:34.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:34.408766+0000) 2022-09-07T23:25:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:34.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:34.307587+0000) 2022-09-07T23:25:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:35 smithi059 ceph-mon[97450]: pgmap v3403: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:35 smithi055 ceph-mon[97033]: pgmap v3403: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:35.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:35.408954+0000) 2022-09-07T23:25:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:35.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:35.307854+0000) 2022-09-07T23:25:36.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:36.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:36.409192+0000) 2022-09-07T23:25:36.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:36.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:36.308054+0000) 2022-09-07T23:25:37.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:37 smithi059 ceph-mon[97450]: pgmap v3404: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:25:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:37 smithi055 ceph-mon[97033]: pgmap v3404: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:25:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:37.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:37.409452+0000) 2022-09-07T23:25:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:37.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:37.308223+0000) 2022-09-07T23:25:38.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:25:38.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:25:38.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:25:38.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:25:38.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:25:38.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:25:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:38.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:38.409669+0000) 2022-09-07T23:25:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:38.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:38.308447+0000) 2022-09-07T23:25:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:39 smithi059 ceph-mon[97450]: pgmap v3405: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:25:39.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:39 smithi055 ceph-mon[97033]: pgmap v3405: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:39.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:25:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:39.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:39.409868+0000) 2022-09-07T23:25:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:39.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:39.308675+0000) 2022-09-07T23:25:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:40 smithi055 ceph-mon[97033]: pgmap v3406: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:40 smithi059 ceph-mon[97450]: pgmap v3406: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:40.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:40.410120+0000) 2022-09-07T23:25:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:40.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:40.308860+0000) 2022-09-07T23:25:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:41.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:41.410396+0000) 2022-09-07T23:25:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:41.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:41.309089+0000) 2022-09-07T23:25:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:42.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:42.309345+0000) 2022-09-07T23:25:42.666 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:42.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:42.410708+0000) 2022-09-07T23:25:43.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:42 smithi055 ceph-mon[97033]: pgmap v3407: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:25:43.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:42 smithi059 ceph-mon[97450]: pgmap v3407: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:25:43.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:43.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:43.410970+0000) 2022-09-07T23:25:43.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:43.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:43.309565+0000) 2022-09-07T23:25:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:44.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:44.411236+0000) 2022-09-07T23:25:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:44.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:44.309801+0000) 2022-09-07T23:25:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:45 smithi059 ceph-mon[97450]: pgmap v3408: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:45 smithi055 ceph-mon[97033]: pgmap v3408: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:45.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:45.411413+0000) 2022-09-07T23:25:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:45.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:45.310073+0000) 2022-09-07T23:25:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:46.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:46.411615+0000) 2022-09-07T23:25:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:46.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:46.310282+0000) 2022-09-07T23:25:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:47 smithi059 ceph-mon[97450]: pgmap v3409: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:25:47.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:47 smithi055 ceph-mon[97033]: pgmap v3409: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:25:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:47.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:47.411894+0000) 2022-09-07T23:25:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:47.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:47.310504+0000) 2022-09-07T23:25:48.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:25:48.293+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:25:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:48.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:48.412105+0000) 2022-09-07T23:25:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:48.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:48.310742+0000) 2022-09-07T23:25:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:49 smithi059 ceph-mon[97450]: pgmap v3410: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:49.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:49 smithi055 ceph-mon[97033]: pgmap v3410: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:49.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:49.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:49.412340+0000) 2022-09-07T23:25:49.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:49.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:49.310993+0000) 2022-09-07T23:25:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:50.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:50.412558+0000) 2022-09-07T23:25:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:50.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:50.311210+0000) 2022-09-07T23:25:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:51 smithi059 ceph-mon[97450]: pgmap v3411: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:51.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:51 smithi055 ceph-mon[97033]: pgmap v3411: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:51.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:51.412756+0000) 2022-09-07T23:25:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:51.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:51.311439+0000) 2022-09-07T23:25:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:52 smithi055 ceph-mon[97033]: pgmap v3412: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:25:52.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:52 smithi059 ceph-mon[97450]: pgmap v3412: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:25:52.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:52.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:52.412882+0000) 2022-09-07T23:25:52.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:52.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:52.311716+0000) 2022-09-07T23:25:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:53.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:53.413026+0000) 2022-09-07T23:25:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:53.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:53.311902+0000) 2022-09-07T23:25:54.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:54.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:54.413168+0000) 2022-09-07T23:25:54.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:54.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:54.312066+0000) 2022-09-07T23:25:55.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:55 smithi059 ceph-mon[97450]: pgmap v3413: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:55.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:55 smithi055 ceph-mon[97033]: pgmap v3413: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:55.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:55.312247+0000) 2022-09-07T23:25:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:55.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:55.413402+0000) 2022-09-07T23:25:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:56.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:56.413550+0000) 2022-09-07T23:25:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:56.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:56.312429+0000) 2022-09-07T23:25:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:57 smithi059 ceph-mon[97450]: pgmap v3414: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:25:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:57 smithi055 ceph-mon[97033]: pgmap v3414: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:25:57.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:57.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:57.413778+0000) 2022-09-07T23:25:57.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:57.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:57.312678+0000) 2022-09-07T23:25:58.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:58.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:58.414019+0000) 2022-09-07T23:25:58.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:58.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:58.312930+0000) 2022-09-07T23:25:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:25:59 smithi059 ceph-mon[97450]: pgmap v3415: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:25:59 smithi055 ceph-mon[97033]: pgmap v3415: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:25:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:25:59.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:59.414209+0000) 2022-09-07T23:25:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:25:59.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:25:59.313177+0000) 2022-09-07T23:26:00.661 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:00.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:00.414340+0000) 2022-09-07T23:26:00.662 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:00.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:00.313323+0000) 2022-09-07T23:26:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:01 smithi059 ceph-mon[97450]: pgmap v3416: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:01 smithi055 ceph-mon[97033]: pgmap v3416: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:01.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:01.414554+0000) 2022-09-07T23:26:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:01.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:01.313552+0000) 2022-09-07T23:26:02.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:02.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:02.414736+0000) 2022-09-07T23:26:02.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:02.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:02.313782+0000) 2022-09-07T23:26:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:03 smithi059 ceph-mon[97450]: pgmap v3417: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:26:03.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:03 smithi055 ceph-mon[97033]: pgmap v3417: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:26:03.658 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:26:03.298+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:26:03.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:03.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:03.414858+0000) 2022-09-07T23:26:03.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:03.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:03.313950+0000) 2022-09-07T23:26:04.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:04.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:04.415126+0000) 2022-09-07T23:26:04.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:04.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:04.314113+0000) 2022-09-07T23:26:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:05 smithi059 ceph-mon[97450]: pgmap v3418: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:26:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:26:05.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:05 smithi055 ceph-mon[97033]: pgmap v3418: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:05.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:26:05.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:26:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:05.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:05.415387+0000) 2022-09-07T23:26:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:05.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:05.314365+0000) 2022-09-07T23:26:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:06.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:06.415615+0000) 2022-09-07T23:26:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:06.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:06.314611+0000) 2022-09-07T23:26:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:07 smithi059 ceph-mon[97450]: pgmap v3419: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:26:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:07 smithi055 ceph-mon[97033]: pgmap v3419: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:26:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:07.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:07.415893+0000) 2022-09-07T23:26:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:07.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:07.314843+0000) 2022-09-07T23:26:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:08.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:08.416105+0000) 2022-09-07T23:26:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:08.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:08.314989+0000) 2022-09-07T23:26:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:09 smithi059 ceph-mon[97450]: pgmap v3420: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:26:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:09 smithi055 ceph-mon[97033]: pgmap v3420: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:26:09.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:09.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:09.416360+0000) 2022-09-07T23:26:09.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:09.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:09.315202+0000) 2022-09-07T23:26:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:10 smithi055 ceph-mon[97033]: osdmap e303: 12 total, 11 up, 11 in 2022-09-07T23:26:10.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:10 smithi055 ceph-mon[97033]: pgmap v3422: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2022-09-07T23:26:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:10 smithi059 ceph-mon[97450]: osdmap e303: 12 total, 11 up, 11 in 2022-09-07T23:26:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:10 smithi059 ceph-mon[97450]: pgmap v3422: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2022-09-07T23:26:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:10.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:10.416563+0000) 2022-09-07T23:26:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:10.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:10.315395+0000) 2022-09-07T23:26:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:11.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:11.416771+0000) 2022-09-07T23:26:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:11.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:11.315628+0000) 2022-09-07T23:26:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:12.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:12.417022+0000) 2022-09-07T23:26:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:12.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:12.315874+0000) 2022-09-07T23:26:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:12 smithi055 ceph-mon[97033]: pgmap v3423: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:26:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:12 smithi059 ceph-mon[97450]: pgmap v3423: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:26:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:13.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:13.417239+0000) 2022-09-07T23:26:13.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:13.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:13.316022+0000) 2022-09-07T23:26:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:14.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:14.417457+0000) 2022-09-07T23:26:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:14.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:14.316221+0000) 2022-09-07T23:26:15.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:15 smithi059 ceph-mon[97450]: pgmap v3424: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:26:15.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:15 smithi059 ceph-mon[97450]: osdmap e304: 12 total, 11 up, 11 in 2022-09-07T23:26:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:15 smithi055 ceph-mon[97033]: pgmap v3424: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:26:15.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:15 smithi055 ceph-mon[97033]: osdmap e304: 12 total, 11 up, 11 in 2022-09-07T23:26:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:15.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:15.316462+0000) 2022-09-07T23:26:15.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:15.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:15.417688+0000) 2022-09-07T23:26:16.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:16.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:16.417898+0000) 2022-09-07T23:26:16.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:16.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:16.316723+0000) 2022-09-07T23:26:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:17 smithi059 ceph-mon[97450]: pgmap v3426: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:26:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:17 smithi055 ceph-mon[97033]: pgmap v3426: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:26:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:17.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:17.418112+0000) 2022-09-07T23:26:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:17.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:17.316885+0000) 2022-09-07T23:26:18.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:26:18.315+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:26:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:18.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:18.418301+0000) 2022-09-07T23:26:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:18.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:18.317034+0000) 2022-09-07T23:26:19.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:19 smithi059 ceph-mon[97450]: pgmap v3427: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-07T23:26:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:19 smithi055 ceph-mon[97033]: pgmap v3427: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-07T23:26:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:19.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:19.418525+0000) 2022-09-07T23:26:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:19.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:19.317217+0000) 2022-09-07T23:26:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:20.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:20.317459+0000) 2022-09-07T23:26:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:20.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:20.418725+0000) 2022-09-07T23:26:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:21 smithi055 ceph-mon[97033]: pgmap v3428: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:26:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:21 smithi059 ceph-mon[97450]: pgmap v3428: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:26:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:21.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:21.317729+0000) 2022-09-07T23:26:21.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:21.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:21.418976+0000) 2022-09-07T23:26:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:22 smithi055 ceph-mon[97033]: pgmap v3429: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:26:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:22 smithi059 ceph-mon[97450]: pgmap v3429: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:26:22.663 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:22.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:22.419182+0000) 2022-09-07T23:26:22.664 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:22.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:22.317952+0000) 2022-09-07T23:26:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:23.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:23.318178+0000) 2022-09-07T23:26:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:23.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:23.419399+0000) 2022-09-07T23:26:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:24.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:24.419580+0000) 2022-09-07T23:26:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:24.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:24.318414+0000) 2022-09-07T23:26:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:25 smithi059 ceph-mon[97450]: pgmap v3430: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:26:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:25 smithi055 ceph-mon[97033]: pgmap v3430: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:26:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:25.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:25.318631+0000) 2022-09-07T23:26:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:25.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:25.419823+0000) 2022-09-07T23:26:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:26.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:26.318756+0000) 2022-09-07T23:26:26.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:26.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:26.420029+0000) 2022-09-07T23:26:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:27 smithi059 ceph-mon[97450]: pgmap v3431: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:26:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:27 smithi055 ceph-mon[97033]: pgmap v3431: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:26:27.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:27.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:27.420181+0000) 2022-09-07T23:26:27.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:27.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:27.318943+0000) 2022-09-07T23:26:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:28.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:28.420404+0000) 2022-09-07T23:26:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:28.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:28.319129+0000) 2022-09-07T23:26:29.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:29 smithi055 ceph-mon[97033]: pgmap v3432: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:29.539 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:29 smithi059 ceph-mon[97450]: pgmap v3432: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:29.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:29.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:29.420565+0000) 2022-09-07T23:26:29.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:29.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:29.319279+0000) 2022-09-07T23:26:30.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:30 smithi055 ceph-mon[97033]: pgmap v3433: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:26:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:30 smithi059 ceph-mon[97450]: pgmap v3433: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:26:30.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:30.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:30.420725+0000) 2022-09-07T23:26:30.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:30.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:30.319495+0000) 2022-09-07T23:26:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:31.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:31.420930+0000) 2022-09-07T23:26:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:31.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:31.319723+0000) 2022-09-07T23:26:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:32.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:32.421147+0000) 2022-09-07T23:26:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:32.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:32.319875+0000) 2022-09-07T23:26:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:32 smithi055 ceph-mon[97033]: pgmap v3434: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:26:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:32 smithi059 ceph-mon[97450]: pgmap v3434: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:26:33.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:33.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:33.421357+0000) 2022-09-07T23:26:33.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:33.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:33.320051+0000) 2022-09-07T23:26:33.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:26:33.318+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:26:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:34.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:34.421533+0000) 2022-09-07T23:26:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:34.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:34.320291+0000) 2022-09-07T23:26:35.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:35 smithi059 ceph-mon[97450]: pgmap v3435: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:35 smithi055 ceph-mon[97033]: pgmap v3435: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:35.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:35.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:35.421761+0000) 2022-09-07T23:26:35.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:35.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:35.320561+0000) 2022-09-07T23:26:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:36.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:36.421899+0000) 2022-09-07T23:26:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:36.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:36.320738+0000) 2022-09-07T23:26:37.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:37 smithi059 ceph-mon[97450]: pgmap v3436: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:26:37.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:37 smithi055 ceph-mon[97033]: pgmap v3436: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:26:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:37.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:37.422163+0000) 2022-09-07T23:26:37.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:37.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:37.320919+0000) 2022-09-07T23:26:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:38.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:38.422339+0000) 2022-09-07T23:26:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:38.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:38.321130+0000) 2022-09-07T23:26:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:39 smithi055 ceph-mon[97033]: pgmap v3437: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:26:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:26:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:26:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:26:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:39 smithi059 ceph-mon[97450]: pgmap v3437: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:39.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:26:39.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:26:39.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:26:39.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:26:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:39.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:39.422546+0000) 2022-09-07T23:26:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:39.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:39.321352+0000) 2022-09-07T23:26:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:40 smithi055 ceph-mon[97033]: pgmap v3438: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:26:40.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:40 smithi059 ceph-mon[97450]: pgmap v3438: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:26:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:40.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:40.422737+0000) 2022-09-07T23:26:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:40.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:40.321586+0000) 2022-09-07T23:26:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:41.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:41.422864+0000) 2022-09-07T23:26:41.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:41.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:41.321820+0000) 2022-09-07T23:26:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:42.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:42.423100+0000) 2022-09-07T23:26:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:42.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:42.322011+0000) 2022-09-07T23:26:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:42 smithi055 ceph-mon[97033]: pgmap v3439: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:26:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:42 smithi059 ceph-mon[97450]: pgmap v3439: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:26:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:43.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:43.423351+0000) 2022-09-07T23:26:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:43.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:43.322233+0000) 2022-09-07T23:26:44.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:44.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:44.423584+0000) 2022-09-07T23:26:44.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:44.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:44.322408+0000) 2022-09-07T23:26:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:45 smithi059 ceph-mon[97450]: pgmap v3440: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:45 smithi055 ceph-mon[97033]: pgmap v3440: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:45.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:45.423785+0000) 2022-09-07T23:26:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:45.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:45.322720+0000) 2022-09-07T23:26:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:46.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:46.423917+0000) 2022-09-07T23:26:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:46.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:46.322899+0000) 2022-09-07T23:26:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:47 smithi059 ceph-mon[97450]: pgmap v3441: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:26:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:47 smithi055 ceph-mon[97033]: pgmap v3441: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:26:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:47.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:47.424153+0000) 2022-09-07T23:26:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:47.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:47.323123+0000) 2022-09-07T23:26:48.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:26:48.333+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:26:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:48.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:48.424380+0000) 2022-09-07T23:26:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:48.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:48.323362+0000) 2022-09-07T23:26:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:49 smithi055 ceph-mon[97033]: pgmap v3442: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:49 smithi059 ceph-mon[97450]: pgmap v3442: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:49.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:49.424625+0000) 2022-09-07T23:26:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:49.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:49.323600+0000) 2022-09-07T23:26:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:50 smithi055 ceph-mon[97033]: pgmap v3443: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:26:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:50 smithi059 ceph-mon[97450]: pgmap v3443: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:26:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:50.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:50.424891+0000) 2022-09-07T23:26:50.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:50.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:50.323822+0000) 2022-09-07T23:26:51.668 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:51.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:51.425117+0000) 2022-09-07T23:26:51.669 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:51.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:51.323986+0000) 2022-09-07T23:26:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:52.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:52.425327+0000) 2022-09-07T23:26:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:52.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:52.324162+0000) 2022-09-07T23:26:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:52 smithi055 ceph-mon[97033]: pgmap v3444: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:26:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:52 smithi059 ceph-mon[97450]: pgmap v3444: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:26:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:53.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:53.425528+0000) 2022-09-07T23:26:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:53.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:53.324404+0000) 2022-09-07T23:26:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:54.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:54.425728+0000) 2022-09-07T23:26:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:54.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:54.324624+0000) 2022-09-07T23:26:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:55 smithi055 ceph-mon[97033]: pgmap v3445: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:55.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:55 smithi059 ceph-mon[97450]: pgmap v3445: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:55.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:55.425956+0000) 2022-09-07T23:26:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:55.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:55.324845+0000) 2022-09-07T23:26:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:56.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:56.426175+0000) 2022-09-07T23:26:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:56.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:56.325053+0000) 2022-09-07T23:26:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:57 smithi055 ceph-mon[97033]: pgmap v3446: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:26:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:57 smithi059 ceph-mon[97450]: pgmap v3446: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:26:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:57.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:57.426378+0000) 2022-09-07T23:26:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:57.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:57.325269+0000) 2022-09-07T23:26:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:58.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:58.426606+0000) 2022-09-07T23:26:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:58.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:58.325537+0000) 2022-09-07T23:26:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:26:59 smithi055 ceph-mon[97033]: pgmap v3447: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:26:59 smithi059 ceph-mon[97450]: pgmap v3447: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:26:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:26:59.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:59.426858+0000) 2022-09-07T23:26:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:26:59.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:26:59.325744+0000) 2022-09-07T23:27:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:00 smithi055 ceph-mon[97033]: pgmap v3448: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:27:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:00 smithi059 ceph-mon[97450]: pgmap v3448: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:27:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:00.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:00.427003+0000) 2022-09-07T23:27:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:00.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:00.325927+0000) 2022-09-07T23:27:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:01.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:01.326074+0000) 2022-09-07T23:27:01.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:01.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:01.427210+0000) 2022-09-07T23:27:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:02.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:02.427438+0000) 2022-09-07T23:27:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:02.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:02.326223+0000) 2022-09-07T23:27:03.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:02 smithi055 ceph-mon[97033]: pgmap v3449: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:27:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:02 smithi059 ceph-mon[97450]: pgmap v3449: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:27:03.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:27:03.349+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:27:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:03.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:03.427722+0000) 2022-09-07T23:27:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:03.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:03.326359+0000) 2022-09-07T23:27:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:04.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:04.326590+0000) 2022-09-07T23:27:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:04.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:04.427893+0000) 2022-09-07T23:27:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:05 smithi055 ceph-mon[97033]: pgmap v3450: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:27:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:27:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:27:05.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:05 smithi059 ceph-mon[97450]: pgmap v3450: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:27:05.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:27:05.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:27:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:05.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:05.428055+0000) 2022-09-07T23:27:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:05.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:05.326894+0000) 2022-09-07T23:27:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:06.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:06.428254+0000) 2022-09-07T23:27:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:06.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:06.327139+0000) 2022-09-07T23:27:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:07 smithi055 ceph-mon[97033]: pgmap v3451: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:27:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:07 smithi059 ceph-mon[97450]: pgmap v3451: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:27:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:07.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:07.428412+0000) 2022-09-07T23:27:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:07.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:07.327408+0000) 2022-09-07T23:27:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:08.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:08.428576+0000) 2022-09-07T23:27:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:08.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:08.327697+0000) 2022-09-07T23:27:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:09 smithi055 ceph-mon[97033]: pgmap v3452: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:27:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:09 smithi059 ceph-mon[97450]: pgmap v3452: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:27:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:09.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:09.327840+0000) 2022-09-07T23:27:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:09.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:09.428783+0000) 2022-09-07T23:27:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:10 smithi055 ceph-mon[97033]: osdmap e305: 12 total, 11 up, 11 in 2022-09-07T23:27:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:10 smithi055 ceph-mon[97033]: pgmap v3454: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:27:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:10 smithi059 ceph-mon[97450]: osdmap e305: 12 total, 11 up, 11 in 2022-09-07T23:27:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:10 smithi059 ceph-mon[97450]: pgmap v3454: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:27:10.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:10.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:10.327996+0000) 2022-09-07T23:27:10.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:10.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:10.429010+0000) 2022-09-07T23:27:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:11.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:11.429269+0000) 2022-09-07T23:27:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:11.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:11.328242+0000) 2022-09-07T23:27:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:12.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:12.429540+0000) 2022-09-07T23:27:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:12.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:12.328492+0000) 2022-09-07T23:27:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:12 smithi055 ceph-mon[97033]: pgmap v3455: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:27:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:12 smithi059 ceph-mon[97450]: pgmap v3455: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:27:13.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:13.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:13.429818+0000) 2022-09-07T23:27:13.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:13.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:13.328714+0000) 2022-09-07T23:27:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:14.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:14.430088+0000) 2022-09-07T23:27:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:14.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:14.328881+0000) 2022-09-07T23:27:15.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:15 smithi055 ceph-mon[97033]: pgmap v3456: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:27:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:15 smithi059 ceph-mon[97450]: pgmap v3456: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:27:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:15.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:15.430223+0000) 2022-09-07T23:27:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:15.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:15.329066+0000) 2022-09-07T23:27:16.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:16.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:16.430413+0000) 2022-09-07T23:27:16.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:16.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:16.329321+0000) 2022-09-07T23:27:17.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:17 smithi055 ceph-mon[97033]: pgmap v3457: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:27:17.546 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:17 smithi059 ceph-mon[97450]: pgmap v3457: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:27:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:17.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:17.430667+0000) 2022-09-07T23:27:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:17.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:17.329596+0000) 2022-09-07T23:27:18.659 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:27:18.365+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:27:18.661 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:18.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:18.430878+0000) 2022-09-07T23:27:18.662 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:18.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:18.329920+0000) 2022-09-07T23:27:19.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:19 smithi055 ceph-mon[97033]: pgmap v3458: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:27:19.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:19 smithi055 ceph-mon[97033]: osdmap e306: 12 total, 11 up, 11 in 2022-09-07T23:27:19.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:19 smithi059 ceph-mon[97450]: pgmap v3458: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:27:19.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:19 smithi059 ceph-mon[97450]: osdmap e306: 12 total, 11 up, 11 in 2022-09-07T23:27:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:19.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:19.431113+0000) 2022-09-07T23:27:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:19.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:19.330160+0000) 2022-09-07T23:27:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:20 smithi055 ceph-mon[97033]: pgmap v3460: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:27:20.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:20 smithi059 ceph-mon[97450]: pgmap v3460: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:27:20.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:20.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:20.431349+0000) 2022-09-07T23:27:20.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:20.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:20.330309+0000) 2022-09-07T23:27:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:21.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:21.431544+0000) 2022-09-07T23:27:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:21.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:21.330550+0000) 2022-09-07T23:27:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:22.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:22.431772+0000) 2022-09-07T23:27:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:22.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:22.330824+0000) 2022-09-07T23:27:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:22 smithi055 ceph-mon[97033]: pgmap v3461: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:27:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:22 smithi059 ceph-mon[97450]: pgmap v3461: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:27:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:23.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:23.432022+0000) 2022-09-07T23:27:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:23.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:23.331072+0000) 2022-09-07T23:27:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:24.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:24.432231+0000) 2022-09-07T23:27:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:24.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:24.331309+0000) 2022-09-07T23:27:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:25 smithi055 ceph-mon[97033]: pgmap v3462: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:27:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:25 smithi059 ceph-mon[97450]: pgmap v3462: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:27:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:25.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:25.432469+0000) 2022-09-07T23:27:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:25.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:25.331516+0000) 2022-09-07T23:27:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:26.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:26.432719+0000) 2022-09-07T23:27:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:26.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:26.331707+0000) 2022-09-07T23:27:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:27 smithi055 ceph-mon[97033]: pgmap v3463: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:27:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:27 smithi059 ceph-mon[97450]: pgmap v3463: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:27:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:27.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:27.432935+0000) 2022-09-07T23:27:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:27.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:27.331964+0000) 2022-09-07T23:27:28.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:28 smithi055 ceph-mon[97033]: pgmap v3464: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:27:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:28 smithi059 ceph-mon[97450]: pgmap v3464: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:27:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:28.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:28.433186+0000) 2022-09-07T23:27:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:28.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:28.332238+0000) 2022-09-07T23:27:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:29.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:29.433422+0000) 2022-09-07T23:27:29.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:29.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:29.332513+0000) 2022-09-07T23:27:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:30.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:30.332779+0000) 2022-09-07T23:27:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:30.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:30.433669+0000) 2022-09-07T23:27:31.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:31 smithi055 ceph-mon[97033]: pgmap v3465: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 7 op/s 2022-09-07T23:27:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:31 smithi059 ceph-mon[97450]: pgmap v3465: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 7 op/s 2022-09-07T23:27:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:31.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:31.433834+0000) 2022-09-07T23:27:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:31.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:31.332972+0000) 2022-09-07T23:27:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:32 smithi055 ceph-mon[97033]: pgmap v3466: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:27:32.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:32 smithi059 ceph-mon[97450]: pgmap v3466: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:27:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:32.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:32.433960+0000) 2022-09-07T23:27:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:32.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:32.333215+0000) 2022-09-07T23:27:33.658 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:27:33.374+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:27:33.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:33.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:33.434125+0000) 2022-09-07T23:27:33.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:33.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:33.333498+0000) 2022-09-07T23:27:34.662 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:34.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:34.434316+0000) 2022-09-07T23:27:34.664 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:34.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:34.333768+0000) 2022-09-07T23:27:35.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:35 smithi055 ceph-mon[97033]: pgmap v3467: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:27:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:35 smithi059 ceph-mon[97450]: pgmap v3467: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:27:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:35.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:35.434559+0000) 2022-09-07T23:27:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:35.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:35.333986+0000) 2022-09-07T23:27:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:36.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:36.434798+0000) 2022-09-07T23:27:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:36.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:36.334178+0000) 2022-09-07T23:27:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:37 smithi055 ceph-mon[97033]: pgmap v3468: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:27:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:37 smithi059 ceph-mon[97450]: pgmap v3468: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:27:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:37.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:37.434980+0000) 2022-09-07T23:27:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:37.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:37.334353+0000) 2022-09-07T23:27:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:38 smithi055 ceph-mon[97033]: pgmap v3469: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:27:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:38 smithi059 ceph-mon[97450]: pgmap v3469: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:27:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:38.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:38.435211+0000) 2022-09-07T23:27:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:38.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:38.334496+0000) 2022-09-07T23:27:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:27:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:27:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:27:39.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:27:39.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:27:39.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:27:39.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:39.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:39.435454+0000) 2022-09-07T23:27:39.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:39.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:39.334678+0000) 2022-09-07T23:27:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:40.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:40.435626+0000) 2022-09-07T23:27:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:40.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:40.334910+0000) 2022-09-07T23:27:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:27:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:27:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:40 smithi055 ceph-mon[97033]: pgmap v3470: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:27:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:27:40.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:27:40.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:40 smithi059 ceph-mon[97450]: pgmap v3470: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:27:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:41.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:41.435772+0000) 2022-09-07T23:27:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:41.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:41.335105+0000) 2022-09-07T23:27:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:42.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:42.435997+0000) 2022-09-07T23:27:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:42.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:42.335372+0000) 2022-09-07T23:27:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:42 smithi055 ceph-mon[97033]: pgmap v3471: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:27:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:27:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:27:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:27:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:27:42.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:42.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:42.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:42.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:42.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:42.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:42.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:42.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:42.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:27:43.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:42 smithi059 ceph-mon[97450]: pgmap v3471: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:27:43.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:27:43.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:27:43.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:27:43.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:27:43.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:43.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:43.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:43.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:43.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:43.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:43.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:43.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:27:43.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:27:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:43.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:43.436217+0000) 2022-09-07T23:27:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:43.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:43.335632+0000) 2022-09-07T23:27:44.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:43 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T23:27:44.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:43 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T23:27:44.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:43 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T23:27:44.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:43 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T23:27:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:43 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T23:27:44.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:43 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T23:27:44.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:43 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T23:27:44.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:43 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T23:27:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:44.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:44.436378+0000) 2022-09-07T23:27:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:44.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:44.335874+0000) 2022-09-07T23:27:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:45 smithi055 ceph-mon[97033]: pgmap v3472: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:27:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:45 smithi059 ceph-mon[97450]: pgmap v3472: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:27:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:45.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:45.436608+0000) 2022-09-07T23:27:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:45.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:45.336043+0000) 2022-09-07T23:27:46.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:46 smithi055 ceph-mon[97033]: pgmap v3473: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:27:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:46 smithi059 ceph-mon[97450]: pgmap v3473: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:27:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:46.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:46.436874+0000) 2022-09-07T23:27:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:46.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:46.336286+0000) 2022-09-07T23:27:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:47.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:47.437113+0000) 2022-09-07T23:27:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:47.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:47.336519+0000) 2022-09-07T23:27:48.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:27:48.388+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:27:48.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:48.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:48.437362+0000) 2022-09-07T23:27:48.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:48.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:48.336782+0000) 2022-09-07T23:27:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:48 smithi055 ceph-mon[97033]: pgmap v3474: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:27:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:48 smithi059 ceph-mon[97450]: pgmap v3474: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:27:49.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:49.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:49.437599+0000) 2022-09-07T23:27:49.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:49.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:49.337019+0000) 2022-09-07T23:27:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:50.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:50.437790+0000) 2022-09-07T23:27:50.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:50.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:50.337236+0000) 2022-09-07T23:27:51.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:51 smithi055 ceph-mon[97033]: pgmap v3475: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:27:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:51 smithi059 ceph-mon[97450]: pgmap v3475: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:27:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:51.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:51.437966+0000) 2022-09-07T23:27:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:51.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:51.337498+0000) 2022-09-07T23:27:52.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:52 smithi055 ceph-mon[97033]: pgmap v3476: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:27:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:52 smithi059 ceph-mon[97450]: pgmap v3476: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:27:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:52.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:52.438208+0000) 2022-09-07T23:27:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:52.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:52.337742+0000) 2022-09-07T23:27:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:53.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:53.438390+0000) 2022-09-07T23:27:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:53.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:53.337883+0000) 2022-09-07T23:27:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:54.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:54.438553+0000) 2022-09-07T23:27:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:54.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:54.338069+0000) 2022-09-07T23:27:55.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:55 smithi055 ceph-mon[97033]: pgmap v3477: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:27:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:55 smithi059 ceph-mon[97450]: pgmap v3477: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:27:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:55.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:55.438729+0000) 2022-09-07T23:27:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:55.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:55.338264+0000) 2022-09-07T23:27:56.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:56 smithi055 ceph-mon[97033]: pgmap v3478: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:27:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:56 smithi059 ceph-mon[97450]: pgmap v3478: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:27:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:56.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:56.438948+0000) 2022-09-07T23:27:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:56.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:56.338420+0000) 2022-09-07T23:27:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:57.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:57.439247+0000) 2022-09-07T23:27:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:57.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:57.338672+0000) 2022-09-07T23:27:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:58.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:58.439456+0000) 2022-09-07T23:27:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:58.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:58.338891+0000) 2022-09-07T23:27:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:27:58 smithi055 ceph-mon[97033]: pgmap v3479: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:27:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:27:58 smithi059 ceph-mon[97450]: pgmap v3479: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:27:59.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:27:59.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:59.439666+0000) 2022-09-07T23:27:59.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:27:59.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:27:59.339164+0000) 2022-09-07T23:28:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:00 smithi055 ceph-mon[97033]: pgmap v3480: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:28:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:00 smithi059 ceph-mon[97450]: pgmap v3480: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:28:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:00.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:00.439848+0000) 2022-09-07T23:28:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:00.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:00.339382+0000) 2022-09-07T23:28:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:01.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:01.440096+0000) 2022-09-07T23:28:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:01.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:01.339576+0000) 2022-09-07T23:28:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:02.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:02.440306+0000) 2022-09-07T23:28:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:02.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:02.339768+0000) 2022-09-07T23:28:03.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:02 smithi055 ceph-mon[97033]: pgmap v3481: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:28:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:02 smithi059 ceph-mon[97450]: pgmap v3481: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:28:03.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:28:03.404+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:28:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:03.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:03.440462+0000) 2022-09-07T23:28:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:03.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:03.339995+0000) 2022-09-07T23:28:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:04.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:04.440706+0000) 2022-09-07T23:28:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:04.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:04.340208+0000) 2022-09-07T23:28:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:05 smithi055 ceph-mon[97033]: pgmap v3482: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:05.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:28:05.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:28:05.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:05 smithi059 ceph-mon[97450]: pgmap v3482: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:05.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:28:05.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:28:05.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:05.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:05.440944+0000) 2022-09-07T23:28:05.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:05.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:05.340417+0000) 2022-09-07T23:28:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:06 smithi055 ceph-mon[97033]: pgmap v3483: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:28:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:06 smithi059 ceph-mon[97450]: pgmap v3483: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:28:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:06.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:06.441199+0000) 2022-09-07T23:28:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:06.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:06.340689+0000) 2022-09-07T23:28:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:07.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:07.441462+0000) 2022-09-07T23:28:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:07.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:07.340982+0000) 2022-09-07T23:28:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:08.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:08.441691+0000) 2022-09-07T23:28:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:08.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:08.341197+0000) 2022-09-07T23:28:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:08 smithi055 ceph-mon[97033]: pgmap v3484: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:28:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:08 smithi059 ceph-mon[97450]: pgmap v3484: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:28:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:09.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:09.441873+0000) 2022-09-07T23:28:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:09.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:09.341401+0000) 2022-09-07T23:28:10.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:09 smithi055 ceph-mon[97033]: osdmap e307: 12 total, 11 up, 11 in 2022-09-07T23:28:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:09 smithi059 ceph-mon[97450]: osdmap e307: 12 total, 11 up, 11 in 2022-09-07T23:28:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:10.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:10.442115+0000) 2022-09-07T23:28:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:10.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:10.341627+0000) 2022-09-07T23:28:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:10 smithi055 ceph-mon[97033]: pgmap v3486: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:28:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:10 smithi059 ceph-mon[97450]: pgmap v3486: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:28:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:11.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:11.442378+0000) 2022-09-07T23:28:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:11.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:11.341866+0000) 2022-09-07T23:28:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:12.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:12.442590+0000) 2022-09-07T23:28:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:12.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:12.342107+0000) 2022-09-07T23:28:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:12 smithi055 ceph-mon[97033]: pgmap v3487: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:28:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:12 smithi059 ceph-mon[97450]: pgmap v3487: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:28:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:13.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:13.442729+0000) 2022-09-07T23:28:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:13.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:13.342273+0000) 2022-09-07T23:28:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:14.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:14.442860+0000) 2022-09-07T23:28:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:14.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:14.342502+0000) 2022-09-07T23:28:15.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:15 smithi055 ceph-mon[97033]: pgmap v3488: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:28:15.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:15 smithi055 ceph-mon[97033]: osdmap e308: 12 total, 11 up, 11 in 2022-09-07T23:28:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:15 smithi059 ceph-mon[97450]: pgmap v3488: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:28:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:15 smithi059 ceph-mon[97450]: osdmap e308: 12 total, 11 up, 11 in 2022-09-07T23:28:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:15.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:15.443058+0000) 2022-09-07T23:28:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:15.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:15.342760+0000) 2022-09-07T23:28:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:16 smithi055 ceph-mon[97033]: pgmap v3490: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:28:16.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:16 smithi059 ceph-mon[97450]: pgmap v3490: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:28:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:16.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:16.443283+0000) 2022-09-07T23:28:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:16.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:16.343027+0000) 2022-09-07T23:28:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:17.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:17.443507+0000) 2022-09-07T23:28:17.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:17.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:17.343288+0000) 2022-09-07T23:28:18.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:28:18.414+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:28:18.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:18.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:18.443764+0000) 2022-09-07T23:28:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:18.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:18.343541+0000) 2022-09-07T23:28:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:18 smithi055 ceph-mon[97033]: pgmap v3491: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-07T23:28:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:18 smithi059 ceph-mon[97450]: pgmap v3491: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-07T23:28:19.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:19.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:19.444025+0000) 2022-09-07T23:28:19.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:19.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:19.343756+0000) 2022-09-07T23:28:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:20 smithi055 ceph-mon[97033]: pgmap v3492: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:28:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:20 smithi059 ceph-mon[97450]: pgmap v3492: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:28:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:20.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:20.444271+0000) 2022-09-07T23:28:20.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:20.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:20.343986+0000) 2022-09-07T23:28:21.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:21.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:21.444506+0000) 2022-09-07T23:28:21.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:21.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:21.344205+0000) 2022-09-07T23:28:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:22.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:22.444780+0000) 2022-09-07T23:28:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:22.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:22.344440+0000) 2022-09-07T23:28:23.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:22 smithi055 ceph-mon[97033]: pgmap v3493: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:28:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:22 smithi059 ceph-mon[97450]: pgmap v3493: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:28:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:23.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:23.444955+0000) 2022-09-07T23:28:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:23.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:23.344689+0000) 2022-09-07T23:28:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:24.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:24.445205+0000) 2022-09-07T23:28:24.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:24.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:24.344928+0000) 2022-09-07T23:28:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:24 smithi055 ceph-mon[97033]: pgmap v3494: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:28:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:24 smithi059 ceph-mon[97450]: pgmap v3494: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:28:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:25.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:25.345167+0000) 2022-09-07T23:28:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:25.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:25.445471+0000) 2022-09-07T23:28:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:26.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:26.445772+0000) 2022-09-07T23:28:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:26.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:26.345360+0000) 2022-09-07T23:28:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:26 smithi055 ceph-mon[97033]: pgmap v3495: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:28:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:26 smithi059 ceph-mon[97450]: pgmap v3495: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:28:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:27.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:27.445989+0000) 2022-09-07T23:28:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:27.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:27.345579+0000) 2022-09-07T23:28:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:28.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:28.446162+0000) 2022-09-07T23:28:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:28.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:28.345731+0000) 2022-09-07T23:28:29.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:28 smithi055 ceph-mon[97033]: pgmap v3496: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:28 smithi059 ceph-mon[97450]: pgmap v3496: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:29.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:29.446331+0000) 2022-09-07T23:28:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:29.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:29.345911+0000) 2022-09-07T23:28:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:30 smithi055 ceph-mon[97033]: pgmap v3497: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:28:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:30 smithi059 ceph-mon[97450]: pgmap v3497: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:28:30.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:30.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:30.446558+0000) 2022-09-07T23:28:30.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:30.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:30.346128+0000) 2022-09-07T23:28:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:31.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:31.446707+0000) 2022-09-07T23:28:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:31.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:31.346343+0000) 2022-09-07T23:28:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:32.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:32.446836+0000) 2022-09-07T23:28:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:32.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:32.346553+0000) 2022-09-07T23:28:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:32 smithi055 ceph-mon[97033]: pgmap v3498: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:28:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:32 smithi059 ceph-mon[97450]: pgmap v3498: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:28:33.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:28:33.419+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:28:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:33.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:33.446985+0000) 2022-09-07T23:28:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:33.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:33.346732+0000) 2022-09-07T23:28:34.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:34.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:34.447205+0000) 2022-09-07T23:28:34.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:34.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:34.346968+0000) 2022-09-07T23:28:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:34 smithi055 ceph-mon[97033]: pgmap v3499: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:35.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:34 smithi059 ceph-mon[97450]: pgmap v3499: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:35.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:35.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:35.447383+0000) 2022-09-07T23:28:35.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:35.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:35.347218+0000) 2022-09-07T23:28:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:36.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:36.447572+0000) 2022-09-07T23:28:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:36.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:36.347445+0000) 2022-09-07T23:28:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:36 smithi055 ceph-mon[97033]: pgmap v3500: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:28:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:36 smithi059 ceph-mon[97450]: pgmap v3500: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:28:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:37.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:37.347712+0000) 2022-09-07T23:28:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:37.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:37.447780+0000) 2022-09-07T23:28:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:38.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:38.347985+0000) 2022-09-07T23:28:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:38.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:38.447908+0000) 2022-09-07T23:28:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:38 smithi055 ceph-mon[97033]: pgmap v3501: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:38 smithi059 ceph-mon[97450]: pgmap v3501: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:39.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:39.448065+0000) 2022-09-07T23:28:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:39.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:39.348154+0000) 2022-09-07T23:28:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:40 smithi055 ceph-mon[97033]: pgmap v3502: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:28:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:40 smithi059 ceph-mon[97450]: pgmap v3502: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:28:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:40.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:40.348355+0000) 2022-09-07T23:28:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:40.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:40.448300+0000) 2022-09-07T23:28:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:41.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:41.448545+0000) 2022-09-07T23:28:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:41.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:41.348591+0000) 2022-09-07T23:28:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:42.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:42.348858+0000) 2022-09-07T23:28:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:42.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:42.448709+0000) 2022-09-07T23:28:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:42 smithi055 ceph-mon[97033]: pgmap v3503: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:28:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:28:43.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:28:43.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:28:43.269 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:42 smithi059 ceph-mon[97450]: pgmap v3503: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:28:43.269 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:28:43.270 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:28:43.270 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:28:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:43.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:43.448884+0000) 2022-09-07T23:28:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:43.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:43.349071+0000) 2022-09-07T23:28:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:28:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:28:44.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:28:44.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:44 smithi059 ceph-mon[97450]: pgmap v3504: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:44.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:44.349284+0000) 2022-09-07T23:28:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:44.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:44.449087+0000) 2022-09-07T23:28:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:28:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:28:44.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:28:44.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:44 smithi055 ceph-mon[97033]: pgmap v3504: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:45.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:45.349542+0000) 2022-09-07T23:28:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:45.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:45.449300+0000) 2022-09-07T23:28:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:46.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:46.349794+0000) 2022-09-07T23:28:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:46.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:46.449534+0000) 2022-09-07T23:28:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:46 smithi055 ceph-mon[97033]: pgmap v3505: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:28:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:46 smithi059 ceph-mon[97450]: pgmap v3505: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:28:47.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:47.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:47.449715+0000) 2022-09-07T23:28:47.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:47.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:47.350037+0000) 2022-09-07T23:28:48.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:48.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:48.449938+0000) 2022-09-07T23:28:48.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:48.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:48.350314+0000) 2022-09-07T23:28:48.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:28:48.429+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:28:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:48 smithi055 ceph-mon[97033]: pgmap v3506: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:48 smithi059 ceph-mon[97450]: pgmap v3506: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:49.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:49.450177+0000) 2022-09-07T23:28:49.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:49.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:49.350560+0000) 2022-09-07T23:28:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:50 smithi055 ceph-mon[97033]: pgmap v3507: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:28:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:50 smithi059 ceph-mon[97450]: pgmap v3507: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:28:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:50.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:50.450414+0000) 2022-09-07T23:28:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:50.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:50.350795+0000) 2022-09-07T23:28:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:51.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:51.450599+0000) 2022-09-07T23:28:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:51.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:51.350953+0000) 2022-09-07T23:28:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:52.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:52.450786+0000) 2022-09-07T23:28:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:52.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:52.351157+0000) 2022-09-07T23:28:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:52 smithi055 ceph-mon[97033]: pgmap v3508: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:28:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:52 smithi059 ceph-mon[97450]: pgmap v3508: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:28:53.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:53.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:53.450968+0000) 2022-09-07T23:28:53.665 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:53.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:53.351406+0000) 2022-09-07T23:28:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:54.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:54.451181+0000) 2022-09-07T23:28:54.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:54.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:54.351617+0000) 2022-09-07T23:28:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:54 smithi055 ceph-mon[97033]: pgmap v3509: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:54 smithi059 ceph-mon[97450]: pgmap v3509: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:55.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:55.451418+0000) 2022-09-07T23:28:55.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:55.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:55.351932+0000) 2022-09-07T23:28:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:56.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:56.451667+0000) 2022-09-07T23:28:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:56.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:56.352092+0000) 2022-09-07T23:28:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:56 smithi055 ceph-mon[97033]: pgmap v3510: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:28:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:56 smithi059 ceph-mon[97450]: pgmap v3510: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:28:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:57.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:57.451842+0000) 2022-09-07T23:28:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:57.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:57.352344+0000) 2022-09-07T23:28:58.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:58.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:58.352612+0000) 2022-09-07T23:28:58.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:58.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:58.452028+0000) 2022-09-07T23:28:59.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:28:58 smithi055 ceph-mon[97033]: pgmap v3511: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:59.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:28:58 smithi059 ceph-mon[97450]: pgmap v3511: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:28:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:28:59.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:59.352852+0000) 2022-09-07T23:28:59.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:28:59.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:28:59.452213+0000) 2022-09-07T23:29:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:00 smithi055 ceph-mon[97033]: pgmap v3512: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:29:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:00 smithi059 ceph-mon[97450]: pgmap v3512: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:29:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:00.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:00.353051+0000) 2022-09-07T23:29:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:00.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:00.452480+0000) 2022-09-07T23:29:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:01.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:01.452720+0000) 2022-09-07T23:29:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:01.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:01.353256+0000) 2022-09-07T23:29:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:02.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:02.353458+0000) 2022-09-07T23:29:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:02.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:02.452962+0000) 2022-09-07T23:29:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:02 smithi055 ceph-mon[97033]: pgmap v3513: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:29:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:02 smithi059 ceph-mon[97450]: pgmap v3513: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:29:03.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:29:03.443+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:29:03.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:03.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:03.453135+0000) 2022-09-07T23:29:03.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:03.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:03.353760+0000) 2022-09-07T23:29:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:04.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:04.453268+0000) 2022-09-07T23:29:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:04.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:04.353972+0000) 2022-09-07T23:29:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:04 smithi055 ceph-mon[97033]: pgmap v3514: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:29:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:29:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:04 smithi059 ceph-mon[97450]: pgmap v3514: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:29:05.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:29:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:05.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:05.453442+0000) 2022-09-07T23:29:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:05.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:05.354215+0000) 2022-09-07T23:29:06.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:06.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:06.453648+0000) 2022-09-07T23:29:06.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:06.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:06.354390+0000) 2022-09-07T23:29:07.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:06 smithi055 ceph-mon[97033]: pgmap v3515: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:29:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:06 smithi059 ceph-mon[97450]: pgmap v3515: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:29:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:07.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:07.453825+0000) 2022-09-07T23:29:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:07.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:07.354605+0000) 2022-09-07T23:29:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:08.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:08.454067+0000) 2022-09-07T23:29:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:08.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:08.354820+0000) 2022-09-07T23:29:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:08 smithi055 ceph-mon[97033]: pgmap v3516: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:29:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:08 smithi059 ceph-mon[97450]: pgmap v3516: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:29:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:09.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:09.454321+0000) 2022-09-07T23:29:09.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:09.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:09.354973+0000) 2022-09-07T23:29:10.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:09 smithi055 ceph-mon[97033]: osdmap e309: 12 total, 11 up, 11 in 2022-09-07T23:29:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:09 smithi059 ceph-mon[97450]: osdmap e309: 12 total, 11 up, 11 in 2022-09-07T23:29:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:10.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:10.355170+0000) 2022-09-07T23:29:10.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:10.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:10.454577+0000) 2022-09-07T23:29:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:10 smithi055 ceph-mon[97033]: pgmap v3518: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:29:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:10 smithi059 ceph-mon[97450]: pgmap v3518: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:29:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:11.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:11.454830+0000) 2022-09-07T23:29:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:11.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:11.355330+0000) 2022-09-07T23:29:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:12.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:12.455092+0000) 2022-09-07T23:29:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:12.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:12.355552+0000) 2022-09-07T23:29:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:12 smithi055 ceph-mon[97033]: pgmap v3519: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:29:13.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:12 smithi059 ceph-mon[97450]: pgmap v3519: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:29:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:13.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:13.455270+0000) 2022-09-07T23:29:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:13.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:13.355781+0000) 2022-09-07T23:29:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:14.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:14.455417+0000) 2022-09-07T23:29:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:14.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:14.356032+0000) 2022-09-07T23:29:15.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:14 smithi055 ceph-mon[97033]: pgmap v3520: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:29:15.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:14 smithi055 ceph-mon[97033]: osdmap e310: 12 total, 11 up, 11 in 2022-09-07T23:29:15.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:14 smithi059 ceph-mon[97450]: pgmap v3520: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:29:15.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:14 smithi059 ceph-mon[97450]: osdmap e310: 12 total, 11 up, 11 in 2022-09-07T23:29:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:15.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:15.356248+0000) 2022-09-07T23:29:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:15.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:15.455631+0000) 2022-09-07T23:29:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:16.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:16.455875+0000) 2022-09-07T23:29:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:16.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:16.356497+0000) 2022-09-07T23:29:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:16 smithi055 ceph-mon[97033]: pgmap v3522: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:29:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:16 smithi059 ceph-mon[97450]: pgmap v3522: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:29:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:17.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:17.456096+0000) 2022-09-07T23:29:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:17.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:17.356731+0000) 2022-09-07T23:29:18.664 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:29:18.451+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:29:18.667 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:18.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:18.456311+0000) 2022-09-07T23:29:18.667 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:18.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:18.356984+0000) 2022-09-07T23:29:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:18 smithi055 ceph-mon[97033]: pgmap v3523: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:29:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:18 smithi059 ceph-mon[97450]: pgmap v3523: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:29:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:19.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:19.456537+0000) 2022-09-07T23:29:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:19.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:19.357248+0000) 2022-09-07T23:29:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:20 smithi055 ceph-mon[97033]: pgmap v3524: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:29:20.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:20 smithi059 ceph-mon[97450]: pgmap v3524: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:29:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:20.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:20.456735+0000) 2022-09-07T23:29:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:20.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:20.357477+0000) 2022-09-07T23:29:21.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:21.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:21.456930+0000) 2022-09-07T23:29:21.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:21.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:21.357741+0000) 2022-09-07T23:29:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:22.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:22.457144+0000) 2022-09-07T23:29:22.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:22.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:22.357893+0000) 2022-09-07T23:29:23.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:22 smithi055 ceph-mon[97033]: pgmap v3525: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:29:23.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:22 smithi059 ceph-mon[97450]: pgmap v3525: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:29:23.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:23.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:23.457360+0000) 2022-09-07T23:29:23.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:23.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:23.358125+0000) 2022-09-07T23:29:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:24.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:24.457564+0000) 2022-09-07T23:29:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:24.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:24.358373+0000) 2022-09-07T23:29:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:24 smithi055 ceph-mon[97033]: pgmap v3526: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:29:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:24 smithi059 ceph-mon[97450]: pgmap v3526: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:29:25.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:25.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:25.457806+0000) 2022-09-07T23:29:25.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:25.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:25.358547+0000) 2022-09-07T23:29:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:26.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:26.457969+0000) 2022-09-07T23:29:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:26.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:26.358756+0000) 2022-09-07T23:29:27.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:26 smithi055 ceph-mon[97033]: pgmap v3527: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:29:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:26 smithi059 ceph-mon[97450]: pgmap v3527: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:29:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:27.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:27.458226+0000) 2022-09-07T23:29:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:27.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:27.358922+0000) 2022-09-07T23:29:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:28.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:28.458461+0000) 2022-09-07T23:29:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:28.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:28.359105+0000) 2022-09-07T23:29:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:28 smithi055 ceph-mon[97033]: pgmap v3528: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:28 smithi059 ceph-mon[97450]: pgmap v3528: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:29.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:29.458689+0000) 2022-09-07T23:29:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:29.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:29.359332+0000) 2022-09-07T23:29:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:30 smithi055 ceph-mon[97033]: pgmap v3529: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:29:30.541 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:30 smithi059 ceph-mon[97450]: pgmap v3529: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:29:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:30.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:30.458910+0000) 2022-09-07T23:29:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:30.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:30.359559+0000) 2022-09-07T23:29:31.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:31.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:31.459160+0000) 2022-09-07T23:29:31.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:31.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:31.359751+0000) 2022-09-07T23:29:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:32.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:32.459417+0000) 2022-09-07T23:29:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:32.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:32.359986+0000) 2022-09-07T23:29:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:32 smithi055 ceph-mon[97033]: pgmap v3530: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:29:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:32 smithi059 ceph-mon[97450]: pgmap v3530: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:29:33.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:29:33.458+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:29:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:33.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:33.459583+0000) 2022-09-07T23:29:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:33.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:33.360238+0000) 2022-09-07T23:29:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:34.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:34.459815+0000) 2022-09-07T23:29:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:34.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:34.360468+0000) 2022-09-07T23:29:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:34 smithi055 ceph-mon[97033]: pgmap v3531: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:34 smithi059 ceph-mon[97450]: pgmap v3531: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:35.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:35.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:35.459990+0000) 2022-09-07T23:29:35.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:35.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:35.360733+0000) 2022-09-07T23:29:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:36.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:36.460187+0000) 2022-09-07T23:29:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:36.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:36.360912+0000) 2022-09-07T23:29:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:36 smithi055 ceph-mon[97033]: pgmap v3532: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:29:37.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:36 smithi059 ceph-mon[97450]: pgmap v3532: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:29:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:37.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:37.460400+0000) 2022-09-07T23:29:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:37.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:37.361133+0000) 2022-09-07T23:29:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:38.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:38.460572+0000) 2022-09-07T23:29:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:38.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:38.361295+0000) 2022-09-07T23:29:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:38 smithi055 ceph-mon[97033]: pgmap v3533: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:38 smithi059 ceph-mon[97450]: pgmap v3533: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:39.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:39.460801+0000) 2022-09-07T23:29:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:39.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:39.361483+0000) 2022-09-07T23:29:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:40 smithi055 ceph-mon[97033]: pgmap v3534: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:29:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:40 smithi059 ceph-mon[97450]: pgmap v3534: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:29:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:40.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:40.461010+0000) 2022-09-07T23:29:40.675 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:40.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:40.361732+0000) 2022-09-07T23:29:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:41.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:41.461174+0000) 2022-09-07T23:29:41.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:41.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:41.361919+0000) 2022-09-07T23:29:42.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:42.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:42.461392+0000) 2022-09-07T23:29:42.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:42.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:42.362083+0000) 2022-09-07T23:29:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:42 smithi055 ceph-mon[97033]: pgmap v3535: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:29:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:42 smithi059 ceph-mon[97450]: pgmap v3535: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:29:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:43.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:43.461692+0000) 2022-09-07T23:29:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:43.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:43.362317+0000) 2022-09-07T23:29:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:29:44.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:29:44.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:29:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:29:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:29:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:29:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:44.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:44.461980+0000) 2022-09-07T23:29:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:44.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:44.362592+0000) 2022-09-07T23:29:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:44 smithi055 ceph-mon[97033]: pgmap v3536: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:45.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:29:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:44 smithi059 ceph-mon[97450]: pgmap v3536: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:29:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:45.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:45.462195+0000) 2022-09-07T23:29:45.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:45.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:45.362885+0000) 2022-09-07T23:29:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:46.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:46.462418+0000) 2022-09-07T23:29:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:46.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:46.363015+0000) 2022-09-07T23:29:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:46 smithi055 ceph-mon[97033]: pgmap v3537: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:29:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:46 smithi059 ceph-mon[97450]: pgmap v3537: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:29:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:47.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:47.462711+0000) 2022-09-07T23:29:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:47.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:47.363238+0000) 2022-09-07T23:29:48.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:29:48.473+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:29:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:48.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:48.462979+0000) 2022-09-07T23:29:48.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:48.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:48.363480+0000) 2022-09-07T23:29:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:48 smithi055 ceph-mon[97033]: pgmap v3538: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:48 smithi059 ceph-mon[97450]: pgmap v3538: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:49.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:49.463247+0000) 2022-09-07T23:29:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:49.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:49.363735+0000) 2022-09-07T23:29:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:50 smithi055 ceph-mon[97033]: pgmap v3539: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:29:50.546 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:50 smithi059 ceph-mon[97450]: pgmap v3539: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:29:50.667 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:50.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:50.463410+0000) 2022-09-07T23:29:50.669 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:50.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:50.364012+0000) 2022-09-07T23:29:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:51.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:51.364283+0000) 2022-09-07T23:29:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:51.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:51.463667+0000) 2022-09-07T23:29:52.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:52.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:52.364553+0000) 2022-09-07T23:29:52.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:52.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:52.463944+0000) 2022-09-07T23:29:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:52 smithi055 ceph-mon[97033]: pgmap v3540: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:29:53.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:52 smithi059 ceph-mon[97450]: pgmap v3540: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:29:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:53.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:53.464197+0000) 2022-09-07T23:29:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:53.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:53.364778+0000) 2022-09-07T23:29:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:54.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:54.464404+0000) 2022-09-07T23:29:54.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:54.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:54.365049+0000) 2022-09-07T23:29:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:54 smithi055 ceph-mon[97033]: pgmap v3541: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:54 smithi059 ceph-mon[97450]: pgmap v3541: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:55.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:55.464691+0000) 2022-09-07T23:29:55.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:55.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:55.365265+0000) 2022-09-07T23:29:56.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:56.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:56.464845+0000) 2022-09-07T23:29:56.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:56.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:56.365467+0000) 2022-09-07T23:29:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:56 smithi055 ceph-mon[97033]: pgmap v3542: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:29:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:56 smithi059 ceph-mon[97450]: pgmap v3542: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:29:57.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:57.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:57.365736+0000) 2022-09-07T23:29:57.663 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:57.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:57.465115+0000) 2022-09-07T23:29:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:58.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:58.465322+0000) 2022-09-07T23:29:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:58.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:58.365982+0000) 2022-09-07T23:29:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:29:58 smithi055 ceph-mon[97033]: pgmap v3543: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:29:58 smithi059 ceph-mon[97450]: pgmap v3543: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:29:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:29:59.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:59.366236+0000) 2022-09-07T23:29:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:29:59.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:29:59.465486+0000) 2022-09-07T23:30:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:00 smithi055 ceph-mon[97033]: pgmap v3544: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:30:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-07T23:30:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T23:30:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T23:30:00.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:00 smithi059 ceph-mon[97450]: pgmap v3544: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:30:00.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-07T23:30:00.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T23:30:00.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T23:30:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:00.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:00.366466+0000) 2022-09-07T23:30:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:00.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:00.465774+0000) 2022-09-07T23:30:01.467 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:01.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:01.366699+0000) 2022-09-07T23:30:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:01.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:01.466023+0000) 2022-09-07T23:30:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:02.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:02.466216+0000) 2022-09-07T23:30:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:02.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:02.366874+0000) 2022-09-07T23:30:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:02 smithi055 ceph-mon[97033]: pgmap v3545: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:30:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:02 smithi059 ceph-mon[97450]: pgmap v3545: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:30:03.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:30:03.485+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:30:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:03.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:03.466407+0000) 2022-09-07T23:30:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:03.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:03.367114+0000) 2022-09-07T23:30:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:04.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:04.466700+0000) 2022-09-07T23:30:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:04.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:04.367313+0000) 2022-09-07T23:30:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:04 smithi055 ceph-mon[97033]: pgmap v3546: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:30:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:30:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:30:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:04 smithi059 ceph-mon[97450]: pgmap v3546: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:30:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:30:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:30:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:05.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:05.466968+0000) 2022-09-07T23:30:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:05.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:05.367463+0000) 2022-09-07T23:30:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:06.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:06.467246+0000) 2022-09-07T23:30:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:06.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:06.367733+0000) 2022-09-07T23:30:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:06 smithi055 ceph-mon[97033]: pgmap v3547: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:30:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:06 smithi059 ceph-mon[97450]: pgmap v3547: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:30:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:07.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:07.467488+0000) 2022-09-07T23:30:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:07.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:07.367899+0000) 2022-09-07T23:30:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:08.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:08.368156+0000) 2022-09-07T23:30:08.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:08.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:08.467738+0000) 2022-09-07T23:30:09.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:08 smithi055 ceph-mon[97033]: pgmap v3548: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:30:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:08 smithi059 ceph-mon[97450]: pgmap v3548: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:30:09.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:09.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:09.467955+0000) 2022-09-07T23:30:09.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:09.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:09.368370+0000) 2022-09-07T23:30:10.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:09 smithi055 ceph-mon[97033]: osdmap e311: 12 total, 11 up, 11 in 2022-09-07T23:30:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:09 smithi059 ceph-mon[97450]: osdmap e311: 12 total, 11 up, 11 in 2022-09-07T23:30:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:10.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:10.368602+0000) 2022-09-07T23:30:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:10.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:10.468131+0000) 2022-09-07T23:30:11.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:10 smithi055 ceph-mon[97033]: pgmap v3550: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:30:11.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:10 smithi059 ceph-mon[97450]: pgmap v3550: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:30:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:11.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:11.368760+0000) 2022-09-07T23:30:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:11.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:11.468352+0000) 2022-09-07T23:30:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:12.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:12.368901+0000) 2022-09-07T23:30:12.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:12.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:12.468566+0000) 2022-09-07T23:30:13.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:12 smithi055 ceph-mon[97033]: pgmap v3551: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:30:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:12 smithi059 ceph-mon[97450]: pgmap v3551: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:30:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:13.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:13.468812+0000) 2022-09-07T23:30:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:13.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:13.369135+0000) 2022-09-07T23:30:14.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:14.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:14.469039+0000) 2022-09-07T23:30:14.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:14.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:14.369301+0000) 2022-09-07T23:30:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:14 smithi055 ceph-mon[97033]: pgmap v3552: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:30:15.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:14 smithi055 ceph-mon[97033]: osdmap e312: 12 total, 11 up, 11 in 2022-09-07T23:30:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:14 smithi059 ceph-mon[97450]: pgmap v3552: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:30:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:14 smithi059 ceph-mon[97450]: osdmap e312: 12 total, 11 up, 11 in 2022-09-07T23:30:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:15.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:15.469278+0000) 2022-09-07T23:30:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:15.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:15.369531+0000) 2022-09-07T23:30:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:16.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:16.469468+0000) 2022-09-07T23:30:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:16.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:16.369743+0000) 2022-09-07T23:30:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:16 smithi055 ceph-mon[97033]: pgmap v3554: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:30:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:16 smithi059 ceph-mon[97450]: pgmap v3554: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:30:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:17.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:17.469672+0000) 2022-09-07T23:30:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:17.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:17.369898+0000) 2022-09-07T23:30:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:18.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:18.469879+0000) 2022-09-07T23:30:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:18.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:18.370109+0000) 2022-09-07T23:30:18.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:30:18.496+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:30:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:18 smithi055 ceph-mon[97033]: pgmap v3555: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:30:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:18 smithi059 ceph-mon[97450]: pgmap v3555: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:30:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:19.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:19.470084+0000) 2022-09-07T23:30:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:19.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:19.370331+0000) 2022-09-07T23:30:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:20 smithi055 ceph-mon[97033]: pgmap v3556: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:30:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:20 smithi059 ceph-mon[97450]: pgmap v3556: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:30:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:20.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:20.470316+0000) 2022-09-07T23:30:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:20.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:20.370550+0000) 2022-09-07T23:30:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:21.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:21.470518+0000) 2022-09-07T23:30:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:21.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:21.370760+0000) 2022-09-07T23:30:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:22.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:22.370988+0000) 2022-09-07T23:30:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:22.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:22.470750+0000) 2022-09-07T23:30:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:22 smithi055 ceph-mon[97033]: pgmap v3557: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:30:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:22 smithi059 ceph-mon[97450]: pgmap v3557: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:30:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:23.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:23.371225+0000) 2022-09-07T23:30:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:23.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:23.470984+0000) 2022-09-07T23:30:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:24.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:24.371387+0000) 2022-09-07T23:30:24.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:24.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:24.471215+0000) 2022-09-07T23:30:25.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:24 smithi055 ceph-mon[97033]: pgmap v3558: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:30:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:24 smithi059 ceph-mon[97450]: pgmap v3558: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:30:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:25.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:25.371611+0000) 2022-09-07T23:30:25.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:25.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:25.471446+0000) 2022-09-07T23:30:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:26.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:26.371812+0000) 2022-09-07T23:30:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:26.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:26.471664+0000) 2022-09-07T23:30:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:26 smithi055 ceph-mon[97033]: pgmap v3559: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:30:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:26 smithi059 ceph-mon[97450]: pgmap v3559: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:30:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:27.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:27.372033+0000) 2022-09-07T23:30:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:27.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:27.471842+0000) 2022-09-07T23:30:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:28.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:28.472055+0000) 2022-09-07T23:30:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:28.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:28.372213+0000) 2022-09-07T23:30:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:28 smithi055 ceph-mon[97033]: pgmap v3560: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:30:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:28 smithi059 ceph-mon[97450]: pgmap v3560: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:30:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:29.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:29.472220+0000) 2022-09-07T23:30:29.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:29.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:29.372399+0000) 2022-09-07T23:30:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:30 smithi055 ceph-mon[97033]: pgmap v3561: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T23:30:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:30 smithi059 ceph-mon[97450]: pgmap v3561: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-07T23:30:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:30.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:30.372616+0000) 2022-09-07T23:30:30.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:30.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:30.472380+0000) 2022-09-07T23:30:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:31.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:31.472534+0000) 2022-09-07T23:30:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:31.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:31.372891+0000) 2022-09-07T23:30:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:32.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:32.472676+0000) 2022-09-07T23:30:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:32.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:32.373120+0000) 2022-09-07T23:30:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:32 smithi055 ceph-mon[97033]: pgmap v3562: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:30:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:32 smithi059 ceph-mon[97450]: pgmap v3562: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:30:33.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:30:33.505+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:30:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:33.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:33.472882+0000) 2022-09-07T23:30:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:33.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:33.373325+0000) 2022-09-07T23:30:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:34.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:34.473113+0000) 2022-09-07T23:30:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:34.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:34.373483+0000) 2022-09-07T23:30:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:34 smithi055 ceph-mon[97033]: pgmap v3563: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:30:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:34 smithi059 ceph-mon[97450]: pgmap v3563: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:30:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:35.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:35.473387+0000) 2022-09-07T23:30:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:35.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:35.373741+0000) 2022-09-07T23:30:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:36.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:36.473645+0000) 2022-09-07T23:30:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:36.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:36.373979+0000) 2022-09-07T23:30:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:36 smithi055 ceph-mon[97033]: pgmap v3564: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:30:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:36 smithi059 ceph-mon[97450]: pgmap v3564: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:30:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:37.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:37.473865+0000) 2022-09-07T23:30:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:37.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:37.374252+0000) 2022-09-07T23:30:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:38.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:38.474135+0000) 2022-09-07T23:30:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:38.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:38.374497+0000) 2022-09-07T23:30:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:38 smithi055 ceph-mon[97033]: pgmap v3565: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:30:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:38 smithi059 ceph-mon[97450]: pgmap v3565: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:30:39.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:39.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:39.474371+0000) 2022-09-07T23:30:39.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:39.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:39.374754+0000) 2022-09-07T23:30:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:40 smithi055 ceph-mon[97033]: pgmap v3566: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:30:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:40 smithi059 ceph-mon[97450]: pgmap v3566: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:30:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:40.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:40.474615+0000) 2022-09-07T23:30:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:40.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:40.374912+0000) 2022-09-07T23:30:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:41.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:41.474807+0000) 2022-09-07T23:30:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:41.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:41.375113+0000) 2022-09-07T23:30:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:42.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:42.475040+0000) 2022-09-07T23:30:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:42.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:42.375324+0000) 2022-09-07T23:30:43.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:42 smithi055 ceph-mon[97033]: pgmap v3567: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:30:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:42 smithi059 ceph-mon[97450]: pgmap v3567: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:30:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:43.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:43.475272+0000) 2022-09-07T23:30:43.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:43.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:43.375528+0000) 2022-09-07T23:30:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:44.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:44.475429+0000) 2022-09-07T23:30:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:44.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:44.375744+0000) 2022-09-07T23:30:45.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:44 smithi055 ceph-mon[97033]: pgmap v3568: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:30:45.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:30:45.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:30:45.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:30:45.297 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:44 smithi059 ceph-mon[97450]: pgmap v3568: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:30:45.298 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:30:45.298 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:30:45.299 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:30:45.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:45.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:45.475613+0000) 2022-09-07T23:30:45.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:45.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:45.375970+0000) 2022-09-07T23:30:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:46.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:46.475850+0000) 2022-09-07T23:30:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:46.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:46.376199+0000) 2022-09-07T23:30:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:46 smithi055 ceph-mon[97033]: pgmap v3569: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:30:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:46 smithi059 ceph-mon[97450]: pgmap v3569: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:30:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:47.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:47.476025+0000) 2022-09-07T23:30:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:47.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:47.376441+0000) 2022-09-07T23:30:48.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:30:48.506+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:30:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:48.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:48.476197+0000) 2022-09-07T23:30:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:48.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:48.376705+0000) 2022-09-07T23:30:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:48 smithi055 ceph-mon[97033]: pgmap v3570: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:30:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:48 smithi059 ceph-mon[97450]: pgmap v3570: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:30:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:49.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:49.476443+0000) 2022-09-07T23:30:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:49.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:49.376886+0000) 2022-09-07T23:30:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:50 smithi055 ceph-mon[97033]: pgmap v3571: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:30:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:50 smithi059 ceph-mon[97450]: pgmap v3571: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:30:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:50.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:50.476678+0000) 2022-09-07T23:30:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:50.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:50.377088+0000) 2022-09-07T23:30:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:51.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:51.476920+0000) 2022-09-07T23:30:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:51.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:51.377288+0000) 2022-09-07T23:30:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:52.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:52.477104+0000) 2022-09-07T23:30:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:52.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:52.377441+0000) 2022-09-07T23:30:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:52 smithi055 ceph-mon[97033]: pgmap v3572: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:30:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:52 smithi059 ceph-mon[97450]: pgmap v3572: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:30:53.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:53.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:53.477284+0000) 2022-09-07T23:30:53.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:53.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:53.377675+0000) 2022-09-07T23:30:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:54.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:54.477504+0000) 2022-09-07T23:30:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:54.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:54.377912+0000) 2022-09-07T23:30:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:54 smithi055 ceph-mon[97033]: pgmap v3573: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:30:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:54 smithi059 ceph-mon[97450]: pgmap v3573: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:30:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:55.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:55.477732+0000) 2022-09-07T23:30:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:55.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:55.378137+0000) 2022-09-07T23:30:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:56.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:56.477908+0000) 2022-09-07T23:30:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:56.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:56.378381+0000) 2022-09-07T23:30:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:56 smithi055 ceph-mon[97033]: pgmap v3574: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:30:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:56 smithi059 ceph-mon[97450]: pgmap v3574: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:30:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:57.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:57.478054+0000) 2022-09-07T23:30:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:57.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:57.378595+0000) 2022-09-07T23:30:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:58.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:58.478249+0000) 2022-09-07T23:30:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:58.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:58.378834+0000) 2022-09-07T23:30:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:30:58 smithi055 ceph-mon[97033]: pgmap v3575: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:30:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:30:58 smithi059 ceph-mon[97450]: pgmap v3575: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:30:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:30:59.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:59.478457+0000) 2022-09-07T23:30:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:30:59.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:30:59.379083+0000) 2022-09-07T23:31:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:00 smithi055 ceph-mon[97033]: pgmap v3576: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:31:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:00 smithi059 ceph-mon[97450]: pgmap v3576: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:31:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:00.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:00.478620+0000) 2022-09-07T23:31:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:00.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:00.379240+0000) 2022-09-07T23:31:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:01.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:01.478866+0000) 2022-09-07T23:31:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:01.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:01.379480+0000) 2022-09-07T23:31:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:02.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:02.479034+0000) 2022-09-07T23:31:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:02.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:02.379677+0000) 2022-09-07T23:31:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:02 smithi055 ceph-mon[97033]: pgmap v3577: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:31:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:02 smithi059 ceph-mon[97450]: pgmap v3577: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:31:03.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:31:03.512+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:31:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:03.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:03.479166+0000) 2022-09-07T23:31:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:03.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:03.379819+0000) 2022-09-07T23:31:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:04.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:04.479404+0000) 2022-09-07T23:31:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:04.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:04.379996+0000) 2022-09-07T23:31:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:04 smithi055 ceph-mon[97033]: pgmap v3578: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:31:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:31:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:31:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:04 smithi059 ceph-mon[97450]: pgmap v3578: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:31:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:31:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:31:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:05.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:05.479670+0000) 2022-09-07T23:31:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:05.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:05.380274+0000) 2022-09-07T23:31:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:06.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:06.479905+0000) 2022-09-07T23:31:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:06.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:06.380443+0000) 2022-09-07T23:31:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:06 smithi055 ceph-mon[97033]: pgmap v3579: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:31:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:06 smithi059 ceph-mon[97450]: pgmap v3579: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:31:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:07.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:07.480172+0000) 2022-09-07T23:31:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:07.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:07.380656+0000) 2022-09-07T23:31:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:08.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:08.480421+0000) 2022-09-07T23:31:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:08.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:08.380915+0000) 2022-09-07T23:31:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:08 smithi055 ceph-mon[97033]: pgmap v3580: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:31:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:08 smithi059 ceph-mon[97450]: pgmap v3580: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:31:09.634 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:09.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:09.480659+0000) 2022-09-07T23:31:09.635 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:09.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:09.381139+0000) 2022-09-07T23:31:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:09 smithi055 ceph-mon[97033]: osdmap e313: 12 total, 11 up, 11 in 2022-09-07T23:31:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:09 smithi059 ceph-mon[97450]: osdmap e313: 12 total, 11 up, 11 in 2022-09-07T23:31:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:10.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:10.480785+0000) 2022-09-07T23:31:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:10.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:10.381342+0000) 2022-09-07T23:31:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:10 smithi055 ceph-mon[97033]: pgmap v3582: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:31:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:10 smithi059 ceph-mon[97450]: pgmap v3582: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:31:11.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:11.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:11.481052+0000) 2022-09-07T23:31:11.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:11.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:11.381577+0000) 2022-09-07T23:31:12.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:12.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:12.481232+0000) 2022-09-07T23:31:12.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:12.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:12.381820+0000) 2022-09-07T23:31:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:12 smithi055 ceph-mon[97033]: pgmap v3583: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:31:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:12 smithi059 ceph-mon[97450]: pgmap v3583: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:31:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:13.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:13.481473+0000) 2022-09-07T23:31:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:13.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:13.381967+0000) 2022-09-07T23:31:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:14.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:14.481715+0000) 2022-09-07T23:31:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:14.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:14.382217+0000) 2022-09-07T23:31:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:14 smithi059 ceph-mon[97450]: pgmap v3584: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:31:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:14 smithi059 ceph-mon[97450]: osdmap e314: 12 total, 11 up, 11 in 2022-09-07T23:31:15.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:31:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:14 smithi055 ceph-mon[97033]: pgmap v3584: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:31:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:14 smithi055 ceph-mon[97033]: osdmap e314: 12 total, 11 up, 11 in 2022-09-07T23:31:15.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:31:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:15.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:15.481908+0000) 2022-09-07T23:31:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:15.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:15.382501+0000) 2022-09-07T23:31:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:15 smithi055 ceph-mon[97033]: pgmap v3586: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2022-09-07T23:31:16.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:15 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-07T23:31:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:15 smithi059 ceph-mon[97450]: pgmap v3586: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2022-09-07T23:31:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:15 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-07T23:31:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:16.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:16.382740+0000) 2022-09-07T23:31:16.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:16.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:16.482081+0000) 2022-09-07T23:31:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:17.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:17.482306+0000) 2022-09-07T23:31:17.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:17.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:17.382902+0000) 2022-09-07T23:31:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:17 smithi055 ceph-mon[97033]: pgmap v3587: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 266 B/s wr, 8 op/s 2022-09-07T23:31:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:17 smithi059 ceph-mon[97450]: pgmap v3587: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 266 B/s wr, 8 op/s 2022-09-07T23:31:18.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:31:18.522+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:31:18.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:18.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:18.482482+0000) 2022-09-07T23:31:18.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:18.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:18.383104+0000) 2022-09-07T23:31:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:19.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:19.482687+0000) 2022-09-07T23:31:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:19.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:19.383336+0000) 2022-09-07T23:31:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:19 smithi055 ceph-mon[97033]: pgmap v3588: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 234 B/s wr, 4 op/s 2022-09-07T23:31:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:19 smithi059 ceph-mon[97450]: pgmap v3588: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 234 B/s wr, 4 op/s 2022-09-07T23:31:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:20.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:20.383557+0000) 2022-09-07T23:31:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:20.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:20.482929+0000) 2022-09-07T23:31:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:21.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:21.383719+0000) 2022-09-07T23:31:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:21.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:21.483123+0000) 2022-09-07T23:31:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:21 smithi055 ceph-mon[97033]: pgmap v3589: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 117 B/s wr, 6 op/s 2022-09-07T23:31:22.291 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:21 smithi059 ceph-mon[97450]: pgmap v3589: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 117 B/s wr, 6 op/s 2022-09-07T23:31:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:22.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:22.383964+0000) 2022-09-07T23:31:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:22.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:22.483381+0000) 2022-09-07T23:31:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:23.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:23.483618+0000) 2022-09-07T23:31:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:23.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:23.384215+0000) 2022-09-07T23:31:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:23 smithi055 ceph-mon[97033]: pgmap v3590: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 117 B/s wr, 6 op/s 2022-09-07T23:31:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:23 smithi059 ceph-mon[97450]: pgmap v3590: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 117 B/s wr, 6 op/s 2022-09-07T23:31:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:24.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:24.483823+0000) 2022-09-07T23:31:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:24.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:24.384412+0000) 2022-09-07T23:31:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:25 smithi055 ceph-mon[97033]: pgmap v3591: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2022-09-07T23:31:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:25 smithi059 ceph-mon[97450]: pgmap v3591: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2022-09-07T23:31:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:25.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:25.483985+0000) 2022-09-07T23:31:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:25.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:25.384674+0000) 2022-09-07T23:31:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:26.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:26.484220+0000) 2022-09-07T23:31:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:26.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:26.384936+0000) 2022-09-07T23:31:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:27.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:27.484357+0000) 2022-09-07T23:31:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:27.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:27.385181+0000) 2022-09-07T23:31:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:27 smithi055 ceph-mon[97033]: pgmap v3592: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:31:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:27 smithi059 ceph-mon[97450]: pgmap v3592: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:31:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:28.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:28.484486+0000) 2022-09-07T23:31:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:28.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:28.385426+0000) 2022-09-07T23:31:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:29.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:29.484719+0000) 2022-09-07T23:31:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:29.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:29.385671+0000) 2022-09-07T23:31:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:29 smithi055 ceph-mon[97033]: pgmap v3593: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:31:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:29 smithi059 ceph-mon[97450]: pgmap v3593: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:31:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:30.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:30.484906+0000) 2022-09-07T23:31:30.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:30.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:30.385941+0000) 2022-09-07T23:31:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:31.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:31.386107+0000) 2022-09-07T23:31:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:31.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:31.485129+0000) 2022-09-07T23:31:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:31 smithi055 ceph-mon[97033]: pgmap v3594: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:31:32.047 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:31 smithi059 ceph-mon[97450]: pgmap v3594: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:31:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:32.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:32.485278+0000) 2022-09-07T23:31:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:32.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:32.386321+0000) 2022-09-07T23:31:33.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:31:33.527+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:31:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:33.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:33.485481+0000) 2022-09-07T23:31:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:33.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:33.386555+0000) 2022-09-07T23:31:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:33 smithi055 ceph-mon[97033]: pgmap v3595: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:31:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:33 smithi059 ceph-mon[97450]: pgmap v3595: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:31:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:34.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:34.485724+0000) 2022-09-07T23:31:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:34.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:34.386783+0000) 2022-09-07T23:31:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:35 smithi055 ceph-mon[97033]: pgmap v3596: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:31:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:35 smithi059 ceph-mon[97450]: pgmap v3596: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:31:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:35.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:35.485948+0000) 2022-09-07T23:31:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:35.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:35.387048+0000) 2022-09-07T23:31:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:36.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:36.486196+0000) 2022-09-07T23:31:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:36.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:36.387266+0000) 2022-09-07T23:31:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:37.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:37.486456+0000) 2022-09-07T23:31:37.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:37.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:37.387514+0000) 2022-09-07T23:31:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:37 smithi055 ceph-mon[97033]: pgmap v3597: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:31:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:37 smithi059 ceph-mon[97450]: pgmap v3597: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:31:38.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:38.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:38.486757+0000) 2022-09-07T23:31:38.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:38.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:38.387734+0000) 2022-09-07T23:31:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:39.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:39.486978+0000) 2022-09-07T23:31:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:39.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:39.387966+0000) 2022-09-07T23:31:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:39 smithi055 ceph-mon[97033]: pgmap v3598: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:31:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:39 smithi059 ceph-mon[97450]: pgmap v3598: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:31:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:40.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:40.388207+0000) 2022-09-07T23:31:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:40.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:40.487230+0000) 2022-09-07T23:31:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:41.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:41.487445+0000) 2022-09-07T23:31:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:41.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:41.388412+0000) 2022-09-07T23:31:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:41 smithi055 ceph-mon[97033]: pgmap v3599: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:31:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:41 smithi059 ceph-mon[97450]: pgmap v3599: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:31:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:42.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:42.487696+0000) 2022-09-07T23:31:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:42.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:42.388680+0000) 2022-09-07T23:31:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:43.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:43.487872+0000) 2022-09-07T23:31:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:43.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:43.388954+0000) 2022-09-07T23:31:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:43 smithi055 ceph-mon[97033]: pgmap v3600: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:31:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:43 smithi059 ceph-mon[97450]: pgmap v3600: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:31:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:44.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:44.488053+0000) 2022-09-07T23:31:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:44.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:44.389156+0000) 2022-09-07T23:31:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:45 smithi055 ceph-mon[97033]: pgmap v3601: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:31:45.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:45 smithi059 ceph-mon[97450]: pgmap v3601: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:31:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:45.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:45.488248+0000) 2022-09-07T23:31:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:45.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:45.389386+0000) 2022-09-07T23:31:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:46.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:46.488485+0000) 2022-09-07T23:31:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:46.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:46.389592+0000) 2022-09-07T23:31:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:47.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:47.389864+0000) 2022-09-07T23:31:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:47.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:47.488738+0000) 2022-09-07T23:31:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:47 smithi055 ceph-mon[97033]: pgmap v3602: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:31:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:47 smithi059 ceph-mon[97450]: pgmap v3602: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:31:48.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:31:48.544+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:31:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:48.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:48.390133+0000) 2022-09-07T23:31:48.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:48.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:48.488962+0000) 2022-09-07T23:31:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:49.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:49.489211+0000) 2022-09-07T23:31:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:49.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:49.390383+0000) 2022-09-07T23:31:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:49 smithi055 ceph-mon[97033]: pgmap v3603: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:31:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:49 smithi059 ceph-mon[97450]: pgmap v3603: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:31:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:50.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:50.489423+0000) 2022-09-07T23:31:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:50.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:50.390663+0000) 2022-09-07T23:31:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:51.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:51.489699+0000) 2022-09-07T23:31:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:51.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:51.390920+0000) 2022-09-07T23:31:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:51 smithi055 ceph-mon[97033]: pgmap v3604: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:31:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:51 smithi059 ceph-mon[97450]: pgmap v3604: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:31:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:52.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:52.489867+0000) 2022-09-07T23:31:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:52.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:52.391158+0000) 2022-09-07T23:31:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:53.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:53.490145+0000) 2022-09-07T23:31:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:53.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:53.391404+0000) 2022-09-07T23:31:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:53 smithi055 ceph-mon[97033]: pgmap v3605: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:31:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:53 smithi059 ceph-mon[97450]: pgmap v3605: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:31:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:54.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:54.490380+0000) 2022-09-07T23:31:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:54.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:54.391626+0000) 2022-09-07T23:31:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:55 smithi055 ceph-mon[97033]: pgmap v3606: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:31:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:55 smithi059 ceph-mon[97450]: pgmap v3606: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:31:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:55.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:55.490679+0000) 2022-09-07T23:31:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:55.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:55.391853+0000) 2022-09-07T23:31:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:56.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:56.490967+0000) 2022-09-07T23:31:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:56.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:56.392101+0000) 2022-09-07T23:31:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:57.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:57.491146+0000) 2022-09-07T23:31:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:57.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:57.392351+0000) 2022-09-07T23:31:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:57 smithi055 ceph-mon[97033]: pgmap v3607: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:31:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:57 smithi059 ceph-mon[97450]: pgmap v3607: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:31:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:58.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:58.491360+0000) 2022-09-07T23:31:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:58.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:58.392545+0000) 2022-09-07T23:31:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:31:59.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:59.491597+0000) 2022-09-07T23:31:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:31:59.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:31:59.392780+0000) 2022-09-07T23:31:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:31:59 smithi055 ceph-mon[97033]: pgmap v3608: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:31:59 smithi059 ceph-mon[97450]: pgmap v3608: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:00.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:00.491801+0000) 2022-09-07T23:32:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:00.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:00.393021+0000) 2022-09-07T23:32:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:01.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:01.492027+0000) 2022-09-07T23:32:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:01.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:01.393265+0000) 2022-09-07T23:32:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:01 smithi055 ceph-mon[97033]: pgmap v3609: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:32:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:01 smithi059 ceph-mon[97450]: pgmap v3609: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:32:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:02.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:02.492154+0000) 2022-09-07T23:32:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:02.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:02.393424+0000) 2022-09-07T23:32:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:03.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:03.492350+0000) 2022-09-07T23:32:03.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:32:03.559+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:32:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:03.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:03.393561+0000) 2022-09-07T23:32:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:03 smithi055 ceph-mon[97033]: pgmap v3610: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:04.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:03 smithi059 ceph-mon[97450]: pgmap v3610: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:04.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:04.492556+0000) 2022-09-07T23:32:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:04.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:04.393799+0000) 2022-09-07T23:32:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:32:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:32:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:05 smithi055 ceph-mon[97033]: pgmap v3611: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:32:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:32:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:05 smithi059 ceph-mon[97450]: pgmap v3611: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:05.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:05.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:05.492785+0000) 2022-09-07T23:32:05.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:05.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:05.394039+0000) 2022-09-07T23:32:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:06.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:06.492964+0000) 2022-09-07T23:32:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:06.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:06.394261+0000) 2022-09-07T23:32:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:07.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:07.493135+0000) 2022-09-07T23:32:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:07.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:07.394494+0000) 2022-09-07T23:32:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:07 smithi055 ceph-mon[97033]: pgmap v3612: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:32:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:07 smithi059 ceph-mon[97450]: pgmap v3612: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:32:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:08.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:08.493350+0000) 2022-09-07T23:32:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:08.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:08.394751+0000) 2022-09-07T23:32:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:09.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:09.493558+0000) 2022-09-07T23:32:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:09.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:09.394931+0000) 2022-09-07T23:32:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:09 smithi055 ceph-mon[97033]: pgmap v3613: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:32:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:09 smithi055 ceph-mon[97033]: osdmap e315: 12 total, 11 up, 11 in 2022-09-07T23:32:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:09 smithi059 ceph-mon[97450]: pgmap v3613: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:32:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:09 smithi059 ceph-mon[97450]: osdmap e315: 12 total, 11 up, 11 in 2022-09-07T23:32:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:10.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:10.493741+0000) 2022-09-07T23:32:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:10.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:10.395130+0000) 2022-09-07T23:32:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:11.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:11.493914+0000) 2022-09-07T23:32:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:11.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:11.395381+0000) 2022-09-07T23:32:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:11 smithi055 ceph-mon[97033]: pgmap v3615: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:32:12.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:11 smithi059 ceph-mon[97450]: pgmap v3615: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:32:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:12.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:12.494105+0000) 2022-09-07T23:32:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:12.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:12.395607+0000) 2022-09-07T23:32:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:13.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:13.494346+0000) 2022-09-07T23:32:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:13.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:13.395835+0000) 2022-09-07T23:32:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:13 smithi055 ceph-mon[97033]: pgmap v3616: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:32:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:13 smithi059 ceph-mon[97450]: pgmap v3616: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:32:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:14.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:14.494600+0000) 2022-09-07T23:32:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:14.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:14.396053+0000) 2022-09-07T23:32:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:15 smithi055 ceph-mon[97033]: osdmap e316: 12 total, 11 up, 11 in 2022-09-07T23:32:15.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:15 smithi055 ceph-mon[97033]: pgmap v3618: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:32:15.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:32:15.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:32:15.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:32:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:15 smithi059 ceph-mon[97450]: osdmap e316: 12 total, 11 up, 11 in 2022-09-07T23:32:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:15 smithi059 ceph-mon[97450]: pgmap v3618: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:32:15.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:32:15.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:32:15.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:32:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:15.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:15.494799+0000) 2022-09-07T23:32:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:15.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:15.396280+0000) 2022-09-07T23:32:16.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:16.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:16.495004+0000) 2022-09-07T23:32:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:16.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:16.396504+0000) 2022-09-07T23:32:17.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:17.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:17.495243+0000) 2022-09-07T23:32:17.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:17.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:17.396682+0000) 2022-09-07T23:32:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:17 smithi055 ceph-mon[97033]: pgmap v3619: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:32:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:17 smithi059 ceph-mon[97450]: pgmap v3619: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:32:18.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:32:18.574+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:32:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:18.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:18.495405+0000) 2022-09-07T23:32:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:18.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:18.396845+0000) 2022-09-07T23:32:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:19.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:19.495680+0000) 2022-09-07T23:32:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:19.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:19.397054+0000) 2022-09-07T23:32:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:19 smithi055 ceph-mon[97033]: pgmap v3620: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:32:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:19 smithi059 ceph-mon[97450]: pgmap v3620: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:32:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:20.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:20.495926+0000) 2022-09-07T23:32:20.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:20.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:20.397310+0000) 2022-09-07T23:32:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:21.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:21.496128+0000) 2022-09-07T23:32:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:21.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:21.397563+0000) 2022-09-07T23:32:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:21 smithi055 ceph-mon[97033]: pgmap v3621: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:32:22.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:21 smithi059 ceph-mon[97450]: pgmap v3621: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:32:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:22.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:22.496372+0000) 2022-09-07T23:32:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:22.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:22.397799+0000) 2022-09-07T23:32:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:23.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:23.496615+0000) 2022-09-07T23:32:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:23.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:23.398034+0000) 2022-09-07T23:32:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:23 smithi055 ceph-mon[97033]: pgmap v3622: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:32:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:23 smithi059 ceph-mon[97450]: pgmap v3622: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:32:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:24.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:24.496845+0000) 2022-09-07T23:32:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:24.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:24.398269+0000) 2022-09-07T23:32:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:25 smithi055 ceph-mon[97033]: pgmap v3623: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2022-09-07T23:32:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:25 smithi059 ceph-mon[97450]: pgmap v3623: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2022-09-07T23:32:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:25.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:25.497029+0000) 2022-09-07T23:32:25.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:25.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:25.398524+0000) 2022-09-07T23:32:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:26.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:26.497168+0000) 2022-09-07T23:32:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:26.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:26.398748+0000) 2022-09-07T23:32:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:27.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:27.497305+0000) 2022-09-07T23:32:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:27.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:27.398907+0000) 2022-09-07T23:32:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:27 smithi055 ceph-mon[97033]: pgmap v3624: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:32:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:27 smithi059 ceph-mon[97450]: pgmap v3624: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:32:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:28.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:28.497558+0000) 2022-09-07T23:32:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:28.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:28.399173+0000) 2022-09-07T23:32:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:29.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:29.497845+0000) 2022-09-07T23:32:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:29.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:29.399393+0000) 2022-09-07T23:32:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:29 smithi055 ceph-mon[97033]: pgmap v3625: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:30.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:29 smithi059 ceph-mon[97450]: pgmap v3625: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:30.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:30.497995+0000) 2022-09-07T23:32:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:30.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:30.399643+0000) 2022-09-07T23:32:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:31.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:31.498162+0000) 2022-09-07T23:32:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:31.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:31.399820+0000) 2022-09-07T23:32:31.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:31 smithi055 ceph-mon[97033]: pgmap v3626: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:32:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:31 smithi059 ceph-mon[97450]: pgmap v3626: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:32:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:32.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:32.498413+0000) 2022-09-07T23:32:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:32.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:32.400041+0000) 2022-09-07T23:32:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:33.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:33.498654+0000) 2022-09-07T23:32:33.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:32:33.576+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:32:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:33.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:33.400258+0000) 2022-09-07T23:32:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:33 smithi055 ceph-mon[97033]: pgmap v3627: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:33 smithi059 ceph-mon[97450]: pgmap v3627: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:34.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:34.498903+0000) 2022-09-07T23:32:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:34.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:34.400513+0000) 2022-09-07T23:32:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:35 smithi055 ceph-mon[97033]: pgmap v3628: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:35 smithi059 ceph-mon[97450]: pgmap v3628: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:35.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:35.499159+0000) 2022-09-07T23:32:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:35.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:35.400778+0000) 2022-09-07T23:32:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:36.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:36.499411+0000) 2022-09-07T23:32:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:36.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:36.401010+0000) 2022-09-07T23:32:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:37.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:37.499654+0000) 2022-09-07T23:32:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:37.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:37.401246+0000) 2022-09-07T23:32:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:37 smithi055 ceph-mon[97033]: pgmap v3629: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:32:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:37 smithi059 ceph-mon[97450]: pgmap v3629: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:32:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:38.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:38.499893+0000) 2022-09-07T23:32:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:38.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:38.401422+0000) 2022-09-07T23:32:39.808 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:39.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:39.500129+0000) 2022-09-07T23:32:39.809 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:39.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:39.401667+0000) 2022-09-07T23:32:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:39 smithi055 ceph-mon[97033]: pgmap v3630: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:39 smithi059 ceph-mon[97450]: pgmap v3630: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:40.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:40.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:40.500364+0000) 2022-09-07T23:32:40.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:40.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:40.401898+0000) 2022-09-07T23:32:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:41.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:41.500545+0000) 2022-09-07T23:32:41.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:41.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:41.402160+0000) 2022-09-07T23:32:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:41 smithi055 ceph-mon[97033]: pgmap v3631: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:32:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:41 smithi059 ceph-mon[97450]: pgmap v3631: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:32:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:42.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:42.500775+0000) 2022-09-07T23:32:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:42.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:42.402387+0000) 2022-09-07T23:32:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:43.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:43.501050+0000) 2022-09-07T23:32:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:43.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:43.402724+0000) 2022-09-07T23:32:43.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:43 smithi055 ceph-mon[97033]: pgmap v3632: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:43 smithi059 ceph-mon[97450]: pgmap v3632: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:44.862 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:44.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:44.501329+0000) 2022-09-07T23:32:44.863 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:44.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:44.402914+0000) 2022-09-07T23:32:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:45.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:45.501537+0000) 2022-09-07T23:32:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:45.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:45.403156+0000) 2022-09-07T23:32:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:45 smithi055 ceph-mon[97033]: pgmap v3633: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:45 smithi055 ceph-mon[97033]: pgmap v3634: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 8 op/s 2022-09-07T23:32:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:45 smithi055 ceph-mon[97033]: pgmap v3635: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T23:32:46.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:32:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:45 smithi059 ceph-mon[97450]: pgmap v3633: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:32:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:45 smithi059 ceph-mon[97450]: pgmap v3634: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 8 op/s 2022-09-07T23:32:46.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:45 smithi059 ceph-mon[97450]: pgmap v3635: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T23:32:46.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:32:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:46.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:46.403371+0000) 2022-09-07T23:32:46.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:46.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:46.501743+0000) 2022-09-07T23:32:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:47.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:47.403597+0000) 2022-09-07T23:32:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:47.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:47.501986+0000) 2022-09-07T23:32:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:47 smithi055 ceph-mon[97033]: pgmap v3636: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:32:48.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:47 smithi059 ceph-mon[97450]: pgmap v3636: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:32:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:48.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:48.502127+0000) 2022-09-07T23:32:48.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:32:48.582+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:32:48.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:48.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:48.403822+0000) 2022-09-07T23:32:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:49.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:49.502368+0000) 2022-09-07T23:32:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:49.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:49.404058+0000) 2022-09-07T23:32:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:49 smithi055 ceph-mon[97033]: pgmap v3637: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:32:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:49 smithi059 ceph-mon[97450]: pgmap v3637: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:32:50.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:50.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:50.502524+0000) 2022-09-07T23:32:50.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:50.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:50.404235+0000) 2022-09-07T23:32:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:51.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:51.502744+0000) 2022-09-07T23:32:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:51.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:51.404451+0000) 2022-09-07T23:32:52.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:51 smithi055 ceph-mon[97033]: pgmap v3638: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:32:52.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:51 smithi059 ceph-mon[97450]: pgmap v3638: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:32:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:52.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:52.404687+0000) 2022-09-07T23:32:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:52.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:52.502969+0000) 2022-09-07T23:32:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:53.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:53.503218+0000) 2022-09-07T23:32:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:53.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:53.404904+0000) 2022-09-07T23:32:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:53 smithi055 ceph-mon[97033]: pgmap v3639: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:32:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:53 smithi059 ceph-mon[97450]: pgmap v3639: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:32:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:54.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:54.503392+0000) 2022-09-07T23:32:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:54.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:54.405143+0000) 2022-09-07T23:32:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:55 smithi055 ceph-mon[97033]: pgmap v3640: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:32:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:55 smithi059 ceph-mon[97450]: pgmap v3640: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:32:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:55.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:55.503519+0000) 2022-09-07T23:32:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:55.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:55.405380+0000) 2022-09-07T23:32:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:56.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:56.503699+0000) 2022-09-07T23:32:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:56.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:56.405573+0000) 2022-09-07T23:32:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:57.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:57.503859+0000) 2022-09-07T23:32:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:57.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:57.405807+0000) 2022-09-07T23:32:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:57 smithi055 ceph-mon[97033]: pgmap v3641: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:32:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:57 smithi059 ceph-mon[97450]: pgmap v3641: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:32:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:58.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:58.405998+0000) 2022-09-07T23:32:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:58.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:58.504111+0000) 2022-09-07T23:32:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:32:59.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:59.406246+0000) 2022-09-07T23:32:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:32:59.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:32:59.504415+0000) 2022-09-07T23:33:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:32:59 smithi055 ceph-mon[97033]: pgmap v3642: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:32:59 smithi059 ceph-mon[97450]: pgmap v3642: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:00.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:00.504662+0000) 2022-09-07T23:33:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:00.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:00.406422+0000) 2022-09-07T23:33:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:01.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:01.504887+0000) 2022-09-07T23:33:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:01.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:01.406684+0000) 2022-09-07T23:33:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:01 smithi055 ceph-mon[97033]: pgmap v3643: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:33:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:01 smithi059 ceph-mon[97450]: pgmap v3643: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:33:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:02.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:02.505067+0000) 2022-09-07T23:33:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:02.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:02.406928+0000) 2022-09-07T23:33:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:03.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:03.505361+0000) 2022-09-07T23:33:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:03.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:03.407134+0000) 2022-09-07T23:33:03.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:33:03.595+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:33:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:03 smithi055 ceph-mon[97033]: pgmap v3644: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:03 smithi059 ceph-mon[97450]: pgmap v3644: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:04.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:04.505608+0000) 2022-09-07T23:33:04.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:04.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:04.407371+0000) 2022-09-07T23:33:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:33:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:33:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:05 smithi055 ceph-mon[97033]: pgmap v3645: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T23:33:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:33:05.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:33:05.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:05 smithi059 ceph-mon[97450]: pgmap v3645: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T23:33:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:05.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:05.505848+0000) 2022-09-07T23:33:05.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:05.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:05.407601+0000) 2022-09-07T23:33:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:06.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:06.506010+0000) 2022-09-07T23:33:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:06.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:06.407851+0000) 2022-09-07T23:33:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:07.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:07.506230+0000) 2022-09-07T23:33:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:07.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:07.408088+0000) 2022-09-07T23:33:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:07 smithi055 ceph-mon[97033]: pgmap v3646: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T23:33:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:07 smithi059 ceph-mon[97450]: pgmap v3646: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T23:33:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:08.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:08.506418+0000) 2022-09-07T23:33:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:08.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:08.408330+0000) 2022-09-07T23:33:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:09.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:09.506666+0000) 2022-09-07T23:33:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:09.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:09.408577+0000) 2022-09-07T23:33:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:09 smithi055 ceph-mon[97033]: pgmap v3647: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:33:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:09 smithi055 ceph-mon[97033]: osdmap e317: 12 total, 11 up, 11 in 2022-09-07T23:33:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:09 smithi059 ceph-mon[97450]: pgmap v3647: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:33:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:09 smithi059 ceph-mon[97450]: osdmap e317: 12 total, 11 up, 11 in 2022-09-07T23:33:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:10.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:10.408797+0000) 2022-09-07T23:33:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:10.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:10.506904+0000) 2022-09-07T23:33:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:11.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:11.507139+0000) 2022-09-07T23:33:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:11.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:11.409050+0000) 2022-09-07T23:33:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:11 smithi055 ceph-mon[97033]: pgmap v3649: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:33:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:11 smithi059 ceph-mon[97450]: pgmap v3649: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:33:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:12.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:12.507294+0000) 2022-09-07T23:33:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:12.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:12.409221+0000) 2022-09-07T23:33:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:13.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:13.507525+0000) 2022-09-07T23:33:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:13.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:13.409347+0000) 2022-09-07T23:33:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:13 smithi055 ceph-mon[97033]: pgmap v3650: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:33:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:13 smithi059 ceph-mon[97450]: pgmap v3650: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:33:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:14.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:14.507770+0000) 2022-09-07T23:33:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:14.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:14.409578+0000) 2022-09-07T23:33:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:15 smithi055 ceph-mon[97033]: osdmap e318: 12 total, 11 up, 11 in 2022-09-07T23:33:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:15 smithi055 ceph-mon[97033]: pgmap v3652: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-07T23:33:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:15 smithi059 ceph-mon[97450]: osdmap e318: 12 total, 11 up, 11 in 2022-09-07T23:33:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:15 smithi059 ceph-mon[97450]: pgmap v3652: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-07T23:33:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:15.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:15.508012+0000) 2022-09-07T23:33:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:15.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:15.409859+0000) 2022-09-07T23:33:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:16.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:16.508228+0000) 2022-09-07T23:33:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:16.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:16.410022+0000) 2022-09-07T23:33:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:17.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:17.508453+0000) 2022-09-07T23:33:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:17.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:17.410253+0000) 2022-09-07T23:33:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:17 smithi055 ceph-mon[97033]: pgmap v3653: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:33:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:17 smithi059 ceph-mon[97450]: pgmap v3653: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:33:18.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:33:18.609+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:33:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:18.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:18.508729+0000) 2022-09-07T23:33:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:18.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:18.410501+0000) 2022-09-07T23:33:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:19.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:19.508976+0000) 2022-09-07T23:33:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:19.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:19.410699+0000) 2022-09-07T23:33:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:19 smithi055 ceph-mon[97033]: pgmap v3654: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:33:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:19 smithi059 ceph-mon[97450]: pgmap v3654: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:33:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:20.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:20.509233+0000) 2022-09-07T23:33:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:20.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:20.410934+0000) 2022-09-07T23:33:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:21.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:21.509518+0000) 2022-09-07T23:33:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:21.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:21.411076+0000) 2022-09-07T23:33:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:21 smithi055 ceph-mon[97033]: pgmap v3655: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:33:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:21 smithi059 ceph-mon[97450]: pgmap v3655: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:33:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:22.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:22.509769+0000) 2022-09-07T23:33:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:22.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:22.411248+0000) 2022-09-07T23:33:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:23.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:23.509985+0000) 2022-09-07T23:33:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:23.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:23.411467+0000) 2022-09-07T23:33:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:23 smithi055 ceph-mon[97033]: pgmap v3656: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:33:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:23 smithi059 ceph-mon[97450]: pgmap v3656: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:33:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:24.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:24.510216+0000) 2022-09-07T23:33:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:24.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:24.411758+0000) 2022-09-07T23:33:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:25 smithi055 ceph-mon[97033]: pgmap v3657: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 95 B/s wr, 8 op/s 2022-09-07T23:33:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:25 smithi059 ceph-mon[97450]: pgmap v3657: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 95 B/s wr, 8 op/s 2022-09-07T23:33:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:25.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:25.510506+0000) 2022-09-07T23:33:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:25.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:25.412035+0000) 2022-09-07T23:33:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:26.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:26.510760+0000) 2022-09-07T23:33:26.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:26.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:26.412259+0000) 2022-09-07T23:33:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:27.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:27.412501+0000) 2022-09-07T23:33:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:27.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:27.511027+0000) 2022-09-07T23:33:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:27 smithi055 ceph-mon[97033]: pgmap v3658: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T23:33:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:27 smithi059 ceph-mon[97450]: pgmap v3658: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-07T23:33:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:28.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:28.511271+0000) 2022-09-07T23:33:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:28.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:28.412705+0000) 2022-09-07T23:33:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:29.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:29.511421+0000) 2022-09-07T23:33:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:29.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:29.412932+0000) 2022-09-07T23:33:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:29 smithi055 ceph-mon[97033]: pgmap v3659: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:29 smithi059 ceph-mon[97450]: pgmap v3659: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:30.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:30.511622+0000) 2022-09-07T23:33:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:30.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:30.413169+0000) 2022-09-07T23:33:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:31.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:31.511838+0000) 2022-09-07T23:33:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:31.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:31.413425+0000) 2022-09-07T23:33:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:31 smithi055 ceph-mon[97033]: pgmap v3660: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:33:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:31 smithi059 ceph-mon[97450]: pgmap v3660: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:33:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:32.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:32.512039+0000) 2022-09-07T23:33:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:32.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:32.413673+0000) 2022-09-07T23:33:33.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:33:33.613+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:33:33.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:33.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:33.512264+0000) 2022-09-07T23:33:33.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:33.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:33.413922+0000) 2022-09-07T23:33:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:33 smithi055 ceph-mon[97033]: pgmap v3661: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:33 smithi059 ceph-mon[97450]: pgmap v3661: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:34.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:34.512540+0000) 2022-09-07T23:33:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:34.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:34.414114+0000) 2022-09-07T23:33:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:35 smithi059 ceph-mon[97450]: pgmap v3662: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:33:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:35 smithi055 ceph-mon[97033]: pgmap v3662: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:33:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:35.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:35.512760+0000) 2022-09-07T23:33:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:35.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:35.414340+0000) 2022-09-07T23:33:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:36.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:36.512996+0000) 2022-09-07T23:33:36.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:36.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:36.414557+0000) 2022-09-07T23:33:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:37.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:37.513190+0000) 2022-09-07T23:33:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:37.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:37.414784+0000) 2022-09-07T23:33:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:37 smithi055 ceph-mon[97033]: pgmap v3663: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:33:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:37 smithi059 ceph-mon[97450]: pgmap v3663: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:33:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:38.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:38.513402+0000) 2022-09-07T23:33:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:38.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:38.415031+0000) 2022-09-07T23:33:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:39.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:39.513669+0000) 2022-09-07T23:33:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:39.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:39.415273+0000) 2022-09-07T23:33:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:39 smithi059 ceph-mon[97450]: pgmap v3664: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:39 smithi055 ceph-mon[97033]: pgmap v3664: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:40.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:40.513874+0000) 2022-09-07T23:33:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:40.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:40.415460+0000) 2022-09-07T23:33:41.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:41.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:41.514095+0000) 2022-09-07T23:33:41.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:41.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:41.415723+0000) 2022-09-07T23:33:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:41 smithi055 ceph-mon[97033]: pgmap v3665: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:33:42.246 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:41 smithi059 ceph-mon[97450]: pgmap v3665: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:33:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:42.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:42.514355+0000) 2022-09-07T23:33:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:42.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:42.415928+0000) 2022-09-07T23:33:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:43.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:43.416163+0000) 2022-09-07T23:33:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:43.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:43.514582+0000) 2022-09-07T23:33:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:43 smithi055 ceph-mon[97033]: pgmap v3666: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:43 smithi059 ceph-mon[97450]: pgmap v3666: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:44.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:44.514817+0000) 2022-09-07T23:33:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:44.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:44.416405+0000) 2022-09-07T23:33:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:45 smithi055 ceph-mon[97033]: pgmap v3667: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:33:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:33:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:33:45.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:33:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:45 smithi059 ceph-mon[97450]: pgmap v3667: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:33:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:33:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:33:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:33:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:45.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:45.515005+0000) 2022-09-07T23:33:45.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:45.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:45.416680+0000) 2022-09-07T23:33:46.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:46.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:46.515194+0000) 2022-09-07T23:33:46.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:46.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:46.416872+0000) 2022-09-07T23:33:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:47.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:47.515411+0000) 2022-09-07T23:33:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:47.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:47.417103+0000) 2022-09-07T23:33:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:47 smithi055 ceph-mon[97033]: pgmap v3668: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:33:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:47 smithi059 ceph-mon[97450]: pgmap v3668: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:33:48.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:33:48.629+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:33:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:48.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:48.515658+0000) 2022-09-07T23:33:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:48.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:48.417334+0000) 2022-09-07T23:33:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:49.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:49.515847+0000) 2022-09-07T23:33:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:49.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:49.417586+0000) 2022-09-07T23:33:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:49 smithi055 ceph-mon[97033]: pgmap v3669: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:49 smithi059 ceph-mon[97450]: pgmap v3669: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:50.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:50.417741+0000) 2022-09-07T23:33:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:50.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:50.516050+0000) 2022-09-07T23:33:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:51.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:51.516227+0000) 2022-09-07T23:33:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:51.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:51.417963+0000) 2022-09-07T23:33:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:51 smithi055 ceph-mon[97033]: pgmap v3670: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:33:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:51 smithi059 ceph-mon[97450]: pgmap v3670: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:33:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:52.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:52.516368+0000) 2022-09-07T23:33:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:52.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:52.418203+0000) 2022-09-07T23:33:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:53.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:53.516716+0000) 2022-09-07T23:33:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:53.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:53.418445+0000) 2022-09-07T23:33:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:53 smithi055 ceph-mon[97033]: pgmap v3671: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:53 smithi059 ceph-mon[97450]: pgmap v3671: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:33:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:54.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:54.516949+0000) 2022-09-07T23:33:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:54.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:54.418719+0000) 2022-09-07T23:33:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:55 smithi055 ceph-mon[97033]: pgmap v3672: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:33:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:55 smithi059 ceph-mon[97450]: pgmap v3672: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:33:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:55.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:55.517140+0000) 2022-09-07T23:33:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:55.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:55.418959+0000) 2022-09-07T23:33:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:56.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:56.419190+0000) 2022-09-07T23:33:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:56.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:56.517330+0000) 2022-09-07T23:33:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:57.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:57.517571+0000) 2022-09-07T23:33:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:57.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:57.419389+0000) 2022-09-07T23:33:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:57 smithi055 ceph-mon[97033]: pgmap v3673: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:33:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:57 smithi059 ceph-mon[97450]: pgmap v3673: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:33:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:58.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:58.517793+0000) 2022-09-07T23:33:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:58.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:58.419572+0000) 2022-09-07T23:33:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:33:59.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:59.518036+0000) 2022-09-07T23:33:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:33:59.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:33:59.419801+0000) 2022-09-07T23:34:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:33:59 smithi055 ceph-mon[97033]: pgmap v3674: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:33:59 smithi059 ceph-mon[97450]: pgmap v3674: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:00.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:00.518276+0000) 2022-09-07T23:34:00.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:00.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:00.419975+0000) 2022-09-07T23:34:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:01.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:01.420212+0000) 2022-09-07T23:34:01.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:01.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:01.518486+0000) 2022-09-07T23:34:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:01 smithi055 ceph-mon[97033]: pgmap v3675: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:34:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:01 smithi059 ceph-mon[97450]: pgmap v3675: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:34:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:02.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:02.518730+0000) 2022-09-07T23:34:02.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:02.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:02.420453+0000) 2022-09-07T23:34:03.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:34:03.631+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:34:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:03.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:03.518969+0000) 2022-09-07T23:34:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:03.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:03.420751+0000) 2022-09-07T23:34:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:03 smithi055 ceph-mon[97033]: pgmap v3676: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:03 smithi059 ceph-mon[97450]: pgmap v3676: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:04.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:04.519169+0000) 2022-09-07T23:34:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:04.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:04.420899+0000) 2022-09-07T23:34:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:34:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:34:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:05 smithi055 ceph-mon[97033]: pgmap v3677: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T23:34:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:34:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:34:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:05 smithi059 ceph-mon[97450]: pgmap v3677: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-07T23:34:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:05.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:05.519375+0000) 2022-09-07T23:34:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:05.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:05.421155+0000) 2022-09-07T23:34:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:06.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:06.519606+0000) 2022-09-07T23:34:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:06.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:06.421385+0000) 2022-09-07T23:34:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:07.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:07.519811+0000) 2022-09-07T23:34:07.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:07.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:07.421605+0000) 2022-09-07T23:34:08.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:07 smithi055 ceph-mon[97033]: pgmap v3678: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T23:34:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:07 smithi059 ceph-mon[97450]: pgmap v3678: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T23:34:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:08.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:08.520025+0000) 2022-09-07T23:34:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:08.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:08.421804+0000) 2022-09-07T23:34:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:09.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:09.520269+0000) 2022-09-07T23:34:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:09.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:09.421990+0000) 2022-09-07T23:34:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:09 smithi055 ceph-mon[97033]: pgmap v3679: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:34:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:09 smithi055 ceph-mon[97033]: osdmap e319: 12 total, 11 up, 11 in 2022-09-07T23:34:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:09 smithi059 ceph-mon[97450]: pgmap v3679: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:34:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:09 smithi059 ceph-mon[97450]: osdmap e319: 12 total, 11 up, 11 in 2022-09-07T23:34:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:10.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:10.520481+0000) 2022-09-07T23:34:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:10.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:10.422174+0000) 2022-09-07T23:34:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:11.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:11.520736+0000) 2022-09-07T23:34:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:11.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:11.422405+0000) 2022-09-07T23:34:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:11 smithi055 ceph-mon[97033]: pgmap v3681: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:34:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:11 smithi059 ceph-mon[97450]: pgmap v3681: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:34:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:12.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:12.520949+0000) 2022-09-07T23:34:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:12.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:12.422648+0000) 2022-09-07T23:34:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:13.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:13.521169+0000) 2022-09-07T23:34:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:13.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:13.422889+0000) 2022-09-07T23:34:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:13 smithi055 ceph-mon[97033]: pgmap v3682: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:34:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:13 smithi059 ceph-mon[97450]: pgmap v3682: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:34:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:14.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:14.423138+0000) 2022-09-07T23:34:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:14.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:14.521391+0000) 2022-09-07T23:34:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:15 smithi055 ceph-mon[97033]: osdmap e320: 12 total, 11 up, 11 in 2022-09-07T23:34:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:15 smithi055 ceph-mon[97033]: pgmap v3684: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-07T23:34:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:34:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:15 smithi059 ceph-mon[97450]: osdmap e320: 12 total, 11 up, 11 in 2022-09-07T23:34:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:15 smithi059 ceph-mon[97450]: pgmap v3684: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-07T23:34:15.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:34:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:15.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:15.521558+0000) 2022-09-07T23:34:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:15.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:15.423385+0000) 2022-09-07T23:34:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:16 smithi055 ceph-mon[97033]: pgmap v3685: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 5.1 KiB/s rd, 327 B/s wr, 10 op/s 2022-09-07T23:34:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:16 smithi059 ceph-mon[97450]: pgmap v3685: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 5.1 KiB/s rd, 327 B/s wr, 10 op/s 2022-09-07T23:34:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:16.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:16.423577+0000) 2022-09-07T23:34:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:16.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:16.521771+0000) 2022-09-07T23:34:17.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:17.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:17.521989+0000) 2022-09-07T23:34:17.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:17.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:17.423804+0000) 2022-09-07T23:34:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:18 smithi055 ceph-mon[97033]: pgmap v3686: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-07T23:34:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:18 smithi059 ceph-mon[97450]: pgmap v3686: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-07T23:34:18.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:34:18.648+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:34:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:18.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:18.423984+0000) 2022-09-07T23:34:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:18.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:18.522215+0000) 2022-09-07T23:34:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:19.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:19.424260+0000) 2022-09-07T23:34:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:19.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:19.522465+0000) 2022-09-07T23:34:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:20 smithi055 ceph-mon[97033]: pgmap v3687: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 124 B/s wr, 3 op/s 2022-09-07T23:34:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:20 smithi059 ceph-mon[97450]: pgmap v3687: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 124 B/s wr, 3 op/s 2022-09-07T23:34:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:20.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:20.424473+0000) 2022-09-07T23:34:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:20.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:20.522730+0000) 2022-09-07T23:34:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:21.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:21.522916+0000) 2022-09-07T23:34:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:21.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:21.424692+0000) 2022-09-07T23:34:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:22 smithi055 ceph-mon[97033]: pgmap v3688: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-07T23:34:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:22 smithi059 ceph-mon[97450]: pgmap v3688: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-07T23:34:22.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:22.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:22.424871+0000) 2022-09-07T23:34:22.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:22.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:22.523069+0000) 2022-09-07T23:34:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:23.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:23.523275+0000) 2022-09-07T23:34:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:23.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:23.425034+0000) 2022-09-07T23:34:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:24 smithi055 ceph-mon[97033]: pgmap v3689: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 113 B/s wr, 6 op/s 2022-09-07T23:34:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:24 smithi059 ceph-mon[97450]: pgmap v3689: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 113 B/s wr, 6 op/s 2022-09-07T23:34:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:24.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:24.523539+0000) 2022-09-07T23:34:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:24.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:24.425274+0000) 2022-09-07T23:34:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:25.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:25.523807+0000) 2022-09-07T23:34:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:25.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:25.425506+0000) 2022-09-07T23:34:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:26 smithi055 ceph-mon[97033]: pgmap v3690: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-07T23:34:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:26 smithi059 ceph-mon[97450]: pgmap v3690: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-07T23:34:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:26.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:26.524039+0000) 2022-09-07T23:34:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:26.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:26.425714+0000) 2022-09-07T23:34:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:27.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:27.425918+0000) 2022-09-07T23:34:27.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:27.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:27.524236+0000) 2022-09-07T23:34:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:28 smithi055 ceph-mon[97033]: pgmap v3691: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:28 smithi059 ceph-mon[97450]: pgmap v3691: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:28.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:28.524474+0000) 2022-09-07T23:34:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:28.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:28.426154+0000) 2022-09-07T23:34:29.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:29.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:29.524726+0000) 2022-09-07T23:34:29.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:29.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:29.426383+0000) 2022-09-07T23:34:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:30 smithi055 ceph-mon[97033]: pgmap v3692: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:30 smithi059 ceph-mon[97450]: pgmap v3692: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:30.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:30.524891+0000) 2022-09-07T23:34:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:30.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:30.426631+0000) 2022-09-07T23:34:31.880 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:31.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:31.525153+0000) 2022-09-07T23:34:31.881 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:31.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:31.426790+0000) 2022-09-07T23:34:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:32 smithi055 ceph-mon[97033]: pgmap v3693: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:34:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:32 smithi059 ceph-mon[97450]: pgmap v3693: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:34:32.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:32.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:32.525360+0000) 2022-09-07T23:34:32.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:32.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:32.427042+0000) 2022-09-07T23:34:33.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:34:33.650+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:34:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:33.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:33.525586+0000) 2022-09-07T23:34:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:33.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:33.427285+0000) 2022-09-07T23:34:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:34 smithi055 ceph-mon[97033]: pgmap v3694: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:34.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:34 smithi059 ceph-mon[97450]: pgmap v3694: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:34.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:34.525774+0000) 2022-09-07T23:34:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:34.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:34.427507+0000) 2022-09-07T23:34:35.527 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:35.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:35.427753+0000) 2022-09-07T23:34:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:35.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:35.525953+0000) 2022-09-07T23:34:36.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:36 smithi055 ceph-mon[97033]: pgmap v3695: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:34:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:36 smithi059 ceph-mon[97450]: pgmap v3695: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:34:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:36.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:36.526166+0000) 2022-09-07T23:34:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:36.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:36.427961+0000) 2022-09-07T23:34:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:37.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:37.526390+0000) 2022-09-07T23:34:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:37.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:37.428176+0000) 2022-09-07T23:34:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:38 smithi055 ceph-mon[97033]: pgmap v3696: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:38 smithi059 ceph-mon[97450]: pgmap v3696: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:38.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:38.526648+0000) 2022-09-07T23:34:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:38.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:38.428390+0000) 2022-09-07T23:34:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:39.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:39.526861+0000) 2022-09-07T23:34:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:39.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:39.428605+0000) 2022-09-07T23:34:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:40 smithi055 ceph-mon[97033]: pgmap v3697: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:40 smithi059 ceph-mon[97450]: pgmap v3697: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:40.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:40.428755+0000) 2022-09-07T23:34:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:40.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:40.527064+0000) 2022-09-07T23:34:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:41.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:41.428990+0000) 2022-09-07T23:34:41.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:41.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:41.527300+0000) 2022-09-07T23:34:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:42 smithi055 ceph-mon[97033]: pgmap v3698: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:34:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:42 smithi059 ceph-mon[97450]: pgmap v3698: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:34:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:42.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:42.429224+0000) 2022-09-07T23:34:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:42.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:42.527545+0000) 2022-09-07T23:34:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:43.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:43.527776+0000) 2022-09-07T23:34:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:43.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:43.429480+0000) 2022-09-07T23:34:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:44 smithi055 ceph-mon[97033]: pgmap v3699: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:44 smithi059 ceph-mon[97450]: pgmap v3699: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:44.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:44.528010+0000) 2022-09-07T23:34:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:44.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:44.429710+0000) 2022-09-07T23:34:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:45.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:45.528296+0000) 2022-09-07T23:34:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:45.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:45.429928+0000) 2022-09-07T23:34:46.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:46 smithi055 ceph-mon[97033]: pgmap v3700: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:34:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:46 smithi059 ceph-mon[97450]: pgmap v3700: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:34:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:46.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:46.528498+0000) 2022-09-07T23:34:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:46.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:46.430149+0000) 2022-09-07T23:34:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:47.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:47.430388+0000) 2022-09-07T23:34:47.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:47.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:47.528724+0000) 2022-09-07T23:34:48.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:48 smithi055 ceph-mon[97033]: pgmap v3701: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:48 smithi059 ceph-mon[97450]: pgmap v3701: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:48.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:48.528936+0000) 2022-09-07T23:34:48.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:34:48.664+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:34:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:48.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:48.430619+0000) 2022-09-07T23:34:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:49.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:49.529153+0000) 2022-09-07T23:34:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:49.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:49.430818+0000) 2022-09-07T23:34:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:50 smithi055 ceph-mon[97033]: pgmap v3702: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:50.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:50 smithi059 ceph-mon[97450]: pgmap v3702: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:50.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:50.529365+0000) 2022-09-07T23:34:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:50.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:50.431033+0000) 2022-09-07T23:34:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:51.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:51.529607+0000) 2022-09-07T23:34:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:51.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:51.431261+0000) 2022-09-07T23:34:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:52 smithi055 ceph-mon[97033]: pgmap v3703: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:34:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:52 smithi059 ceph-mon[97450]: pgmap v3703: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:34:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:52.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:52.529915+0000) 2022-09-07T23:34:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:52.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:52.431484+0000) 2022-09-07T23:34:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:53.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:53.530061+0000) 2022-09-07T23:34:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:53.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:53.431753+0000) 2022-09-07T23:34:54.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:54 smithi055 ceph-mon[97033]: pgmap v3704: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:54 smithi059 ceph-mon[97450]: pgmap v3704: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:54.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:54.530266+0000) 2022-09-07T23:34:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:54.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:54.432002+0000) 2022-09-07T23:34:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:55.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:55.530507+0000) 2022-09-07T23:34:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:55.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:55.432261+0000) 2022-09-07T23:34:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:56 smithi059 ceph-mon[97450]: pgmap v3705: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:34:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:56 smithi055 ceph-mon[97033]: pgmap v3705: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:34:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:56.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:56.432470+0000) 2022-09-07T23:34:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:56.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:56.530777+0000) 2022-09-07T23:34:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:57.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:57.432742+0000) 2022-09-07T23:34:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:57.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:57.531050+0000) 2022-09-07T23:34:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:34:58 smithi059 ceph-mon[97450]: pgmap v3706: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:34:58 smithi055 ceph-mon[97033]: pgmap v3706: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:34:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:58.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:58.531325+0000) 2022-09-07T23:34:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:58.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:58.432905+0000) 2022-09-07T23:34:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:34:59.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:59.531573+0000) 2022-09-07T23:34:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:34:59.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:34:59.433121+0000) 2022-09-07T23:35:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:00 smithi059 ceph-mon[97450]: pgmap v3707: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:00 smithi055 ceph-mon[97033]: pgmap v3707: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:00.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:00.531708+0000) 2022-09-07T23:35:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:00.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:00.433274+0000) 2022-09-07T23:35:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:01.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:01.531829+0000) 2022-09-07T23:35:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:01.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:01.433501+0000) 2022-09-07T23:35:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:02 smithi059 ceph-mon[97450]: pgmap v3708: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:35:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:02 smithi055 ceph-mon[97033]: pgmap v3708: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:35:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:02.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:02.532072+0000) 2022-09-07T23:35:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:02.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:02.433710+0000) 2022-09-07T23:35:03.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:35:03.679+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:35:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:03.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:03.532259+0000) 2022-09-07T23:35:03.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:03.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:03.433913+0000) 2022-09-07T23:35:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:04 smithi059 ceph-mon[97450]: pgmap v3709: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:04.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:04 smithi055 ceph-mon[97033]: pgmap v3709: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:04.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:04.532522+0000) 2022-09-07T23:35:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:04.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:04.434121+0000) 2022-09-07T23:35:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:35:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:35:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:35:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:35:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:05.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:05.532702+0000) 2022-09-07T23:35:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:05.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:05.434335+0000) 2022-09-07T23:35:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:06 smithi059 ceph-mon[97450]: pgmap v3710: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:35:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:06 smithi055 ceph-mon[97033]: pgmap v3710: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:35:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:06.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:06.532922+0000) 2022-09-07T23:35:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:06.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:06.434562+0000) 2022-09-07T23:35:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:07.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:07.533145+0000) 2022-09-07T23:35:07.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:07.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:07.434787+0000) 2022-09-07T23:35:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:08 smithi059 ceph-mon[97450]: pgmap v3711: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:35:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:08 smithi055 ceph-mon[97033]: pgmap v3711: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:35:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:08.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:08.533354+0000) 2022-09-07T23:35:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:08.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:08.435056+0000) 2022-09-07T23:35:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:09.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:09.533516+0000) 2022-09-07T23:35:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:09.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:09.435305+0000) 2022-09-07T23:35:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:10 smithi059 ceph-mon[97450]: pgmap v3712: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:35:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:10 smithi059 ceph-mon[97450]: osdmap e321: 12 total, 11 up, 11 in 2022-09-07T23:35:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:10 smithi055 ceph-mon[97033]: pgmap v3712: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:35:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:10 smithi055 ceph-mon[97033]: osdmap e321: 12 total, 11 up, 11 in 2022-09-07T23:35:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:10.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:10.533758+0000) 2022-09-07T23:35:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:10.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:10.435553+0000) 2022-09-07T23:35:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:11.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:11.533968+0000) 2022-09-07T23:35:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:11.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:11.435776+0000) 2022-09-07T23:35:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:12 smithi059 ceph-mon[97450]: pgmap v3714: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:35:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:12 smithi055 ceph-mon[97033]: pgmap v3714: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:35:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:12.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:12.435978+0000) 2022-09-07T23:35:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:12.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:12.534143+0000) 2022-09-07T23:35:13.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:13.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:13.534378+0000) 2022-09-07T23:35:13.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:13.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:13.436147+0000) 2022-09-07T23:35:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:14 smithi059 ceph-mon[97450]: pgmap v3715: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:35:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:14 smithi059 ceph-mon[97450]: osdmap e322: 12 total, 11 up, 11 in 2022-09-07T23:35:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:14 smithi055 ceph-mon[97033]: pgmap v3715: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:35:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:14 smithi055 ceph-mon[97033]: osdmap e322: 12 total, 11 up, 11 in 2022-09-07T23:35:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:14.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:14.534609+0000) 2022-09-07T23:35:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:14.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:14.436393+0000) 2022-09-07T23:35:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:35:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:35:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:35:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:15.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:15.436623+0000) 2022-09-07T23:35:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:15.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:15.534828+0000) 2022-09-07T23:35:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:35:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:35:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:35:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:16 smithi059 ceph-mon[97450]: pgmap v3717: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:35:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:16 smithi055 ceph-mon[97033]: pgmap v3717: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:35:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:16.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:16.436923+0000) 2022-09-07T23:35:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:16.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:16.535023+0000) 2022-09-07T23:35:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:17.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:17.437152+0000) 2022-09-07T23:35:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:17.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:17.535261+0000) 2022-09-07T23:35:18.694 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:18.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:18.535483+0000) 2022-09-07T23:35:18.695 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:18.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:18.437375+0000) 2022-09-07T23:35:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:18 smithi055 ceph-mon[97033]: pgmap v3718: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:35:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:18 smithi059 ceph-mon[97450]: pgmap v3718: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:35:19.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:35:18.692+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:35:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:19.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:19.437610+0000) 2022-09-07T23:35:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:19.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:19.535752+0000) 2022-09-07T23:35:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:20 smithi055 ceph-mon[97033]: pgmap v3719: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-07T23:35:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:20 smithi059 ceph-mon[97450]: pgmap v3719: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-07T23:35:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:20.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:20.437790+0000) 2022-09-07T23:35:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:20.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:20.535985+0000) 2022-09-07T23:35:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:21.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:21.536223+0000) 2022-09-07T23:35:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:21.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:21.437975+0000) 2022-09-07T23:35:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:22 smithi055 ceph-mon[97033]: pgmap v3720: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:35:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:22 smithi059 ceph-mon[97450]: pgmap v3720: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:35:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:22.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:22.536445+0000) 2022-09-07T23:35:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:22.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:22.438173+0000) 2022-09-07T23:35:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:23.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:23.438430+0000) 2022-09-07T23:35:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:23.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:23.536614+0000) 2022-09-07T23:35:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:24 smithi055 ceph-mon[97033]: pgmap v3721: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:35:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:24 smithi059 ceph-mon[97450]: pgmap v3721: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:35:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:24.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:24.438698+0000) 2022-09-07T23:35:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:24.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:24.536799+0000) 2022-09-07T23:35:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:25.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:25.536981+0000) 2022-09-07T23:35:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:25.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:25.438869+0000) 2022-09-07T23:35:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:26 smithi055 ceph-mon[97033]: pgmap v3722: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 93 B/s wr, 8 op/s 2022-09-07T23:35:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:26 smithi059 ceph-mon[97450]: pgmap v3722: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 93 B/s wr, 8 op/s 2022-09-07T23:35:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:26.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:26.537197+0000) 2022-09-07T23:35:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:26.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:26.439062+0000) 2022-09-07T23:35:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:27.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:27.439237+0000) 2022-09-07T23:35:27.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:27.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:27.537461+0000) 2022-09-07T23:35:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:28 smithi055 ceph-mon[97033]: pgmap v3723: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:28 smithi059 ceph-mon[97450]: pgmap v3723: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:28.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:28.537783+0000) 2022-09-07T23:35:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:28.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:28.439485+0000) 2022-09-07T23:35:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:29.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:29.538017+0000) 2022-09-07T23:35:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:29.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:29.439729+0000) 2022-09-07T23:35:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:30 smithi055 ceph-mon[97033]: pgmap v3724: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:30 smithi059 ceph-mon[97450]: pgmap v3724: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:30.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:30.439892+0000) 2022-09-07T23:35:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:30.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:30.538170+0000) 2022-09-07T23:35:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:31.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:31.440126+0000) 2022-09-07T23:35:31.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:31.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:31.538295+0000) 2022-09-07T23:35:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:32 smithi055 ceph-mon[97033]: pgmap v3725: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:35:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:32 smithi059 ceph-mon[97450]: pgmap v3725: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:35:32.808 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:32.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:32.538437+0000) 2022-09-07T23:35:32.808 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:32.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:32.440250+0000) 2022-09-07T23:35:33.696 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:33.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:33.538625+0000) 2022-09-07T23:35:33.697 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:33.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:33.440490+0000) 2022-09-07T23:35:34.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:35:33.693+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:35:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:34 smithi055 ceph-mon[97033]: pgmap v3726: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:34 smithi059 ceph-mon[97450]: pgmap v3726: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:34.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:34.538798+0000) 2022-09-07T23:35:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:34.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:34.440732+0000) 2022-09-07T23:35:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:35.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:35.440975+0000) 2022-09-07T23:35:35.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:35.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:35.539029+0000) 2022-09-07T23:35:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:36 smithi055 ceph-mon[97033]: pgmap v3727: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:35:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:36 smithi059 ceph-mon[97450]: pgmap v3727: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:35:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:36.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:36.539251+0000) 2022-09-07T23:35:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:36.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:36.441124+0000) 2022-09-07T23:35:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:37.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:37.539462+0000) 2022-09-07T23:35:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:37.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:37.441264+0000) 2022-09-07T23:35:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:38 smithi055 ceph-mon[97033]: pgmap v3728: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:38.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:38 smithi059 ceph-mon[97450]: pgmap v3728: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:38.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:38.539726+0000) 2022-09-07T23:35:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:38.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:38.441483+0000) 2022-09-07T23:35:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:39.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:39.441737+0000) 2022-09-07T23:35:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:39.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:39.539970+0000) 2022-09-07T23:35:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:40.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:40.540192+0000) 2022-09-07T23:35:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:40.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:40.441891+0000) 2022-09-07T23:35:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:40 smithi055 ceph-mon[97033]: pgmap v3729: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:40 smithi059 ceph-mon[97450]: pgmap v3729: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:41.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:41.442110+0000) 2022-09-07T23:35:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:41.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:41.540396+0000) 2022-09-07T23:35:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:42 smithi055 ceph-mon[97033]: pgmap v3730: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:35:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:42 smithi059 ceph-mon[97450]: pgmap v3730: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:35:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:42.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:42.442297+0000) 2022-09-07T23:35:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:42.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:42.540595+0000) 2022-09-07T23:35:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:43.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:43.540793+0000) 2022-09-07T23:35:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:43.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:43.442548+0000) 2022-09-07T23:35:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:44 smithi055 ceph-mon[97033]: pgmap v3731: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:44 smithi059 ceph-mon[97450]: pgmap v3731: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:44.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:44.541021+0000) 2022-09-07T23:35:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:44.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:44.442732+0000) 2022-09-07T23:35:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:45.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:45.541216+0000) 2022-09-07T23:35:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:45.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:45.442889+0000) 2022-09-07T23:35:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:46 smithi055 ceph-mon[97033]: pgmap v3732: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:35:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:46 smithi055 ceph-mon[97033]: pgmap v3733: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:35:46.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:46 smithi055 ceph-mon[97033]: pgmap v3734: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:35:46.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:35:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:46 smithi059 ceph-mon[97450]: pgmap v3732: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:35:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:46 smithi059 ceph-mon[97450]: pgmap v3733: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:35:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:46 smithi059 ceph-mon[97450]: pgmap v3734: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:35:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:35:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:46.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:46.541369+0000) 2022-09-07T23:35:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:46.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:46.443094+0000) 2022-09-07T23:35:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:47.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:47.443336+0000) 2022-09-07T23:35:47.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:47.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:47.541602+0000) 2022-09-07T23:35:48.710 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:48.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:48.541790+0000) 2022-09-07T23:35:48.711 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:48.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:48.443626+0000) 2022-09-07T23:35:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:48 smithi055 ceph-mon[97033]: pgmap v3735: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:35:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:48 smithi059 ceph-mon[97450]: pgmap v3735: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:35:49.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:35:48.708+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:35:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:49.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:49.542022+0000) 2022-09-07T23:35:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:49.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:49.443849+0000) 2022-09-07T23:35:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:50 smithi055 ceph-mon[97033]: pgmap v3736: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:35:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:50 smithi059 ceph-mon[97450]: pgmap v3736: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:35:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:50.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:50.542247+0000) 2022-09-07T23:35:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:50.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:50.444070+0000) 2022-09-07T23:35:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:51.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:51.542465+0000) 2022-09-07T23:35:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:51.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:51.444310+0000) 2022-09-07T23:35:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:52 smithi055 ceph-mon[97033]: pgmap v3737: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:35:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:52 smithi059 ceph-mon[97450]: pgmap v3737: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:35:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:52.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:52.542698+0000) 2022-09-07T23:35:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:52.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:52.444543+0000) 2022-09-07T23:35:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:53.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:53.542839+0000) 2022-09-07T23:35:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:53.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:53.444769+0000) 2022-09-07T23:35:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:54 smithi055 ceph-mon[97033]: pgmap v3738: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:35:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:54 smithi059 ceph-mon[97450]: pgmap v3738: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:35:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:54.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:54.542954+0000) 2022-09-07T23:35:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:54.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:54.444917+0000) 2022-09-07T23:35:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:55.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:55.543101+0000) 2022-09-07T23:35:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:55.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:55.445058+0000) 2022-09-07T23:35:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:56 smithi055 ceph-mon[97033]: pgmap v3739: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:35:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:56 smithi059 ceph-mon[97450]: pgmap v3739: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:35:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:56.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:56.543306+0000) 2022-09-07T23:35:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:56.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:56.445281+0000) 2022-09-07T23:35:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:57.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:57.543573+0000) 2022-09-07T23:35:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:57.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:57.445493+0000) 2022-09-07T23:35:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:35:58 smithi059 ceph-mon[97450]: pgmap v3740: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:58.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:58.543754+0000) 2022-09-07T23:35:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:58.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:58.445628+0000) 2022-09-07T23:35:58.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:35:58 smithi055 ceph-mon[97033]: pgmap v3740: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:35:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:35:59.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:59.543893+0000) 2022-09-07T23:35:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:35:59.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:35:59.445777+0000) 2022-09-07T23:36:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:00 smithi055 ceph-mon[97033]: pgmap v3741: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:00 smithi059 ceph-mon[97450]: pgmap v3741: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:00.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:00.544056+0000) 2022-09-07T23:36:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:00.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:00.445912+0000) 2022-09-07T23:36:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:01.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:01.544292+0000) 2022-09-07T23:36:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:01.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:01.446053+0000) 2022-09-07T23:36:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:02 smithi055 ceph-mon[97033]: pgmap v3742: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:36:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:02 smithi059 ceph-mon[97450]: pgmap v3742: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:36:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:02.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:02.544497+0000) 2022-09-07T23:36:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:02.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:02.446272+0000) 2022-09-07T23:36:03.712 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:03.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:03.544661+0000) 2022-09-07T23:36:03.712 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:03.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:03.446485+0000) 2022-09-07T23:36:04.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:36:03.709+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:36:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:04 smithi059 ceph-mon[97450]: pgmap v3743: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:36:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:36:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:04.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:04.544875+0000) 2022-09-07T23:36:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:04.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:04.446702+0000) 2022-09-07T23:36:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:04 smithi055 ceph-mon[97033]: pgmap v3743: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:36:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:36:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:05.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:05.545132+0000) 2022-09-07T23:36:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:05.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:05.446943+0000) 2022-09-07T23:36:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:06 smithi059 ceph-mon[97450]: pgmap v3744: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:36:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:06.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:06.545296+0000) 2022-09-07T23:36:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:06.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:06.447137+0000) 2022-09-07T23:36:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:06 smithi055 ceph-mon[97033]: pgmap v3744: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:36:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:07.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:07.545467+0000) 2022-09-07T23:36:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:07.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:07.447361+0000) 2022-09-07T23:36:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:08 smithi059 ceph-mon[97450]: pgmap v3745: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:36:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:08.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:08.545734+0000) 2022-09-07T23:36:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:08.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:08.447574+0000) 2022-09-07T23:36:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:08 smithi055 ceph-mon[97033]: pgmap v3745: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:36:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:09 smithi059 ceph-mon[97450]: osdmap e323: 12 total, 11 up, 11 in 2022-09-07T23:36:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:09.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:09.545925+0000) 2022-09-07T23:36:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:09.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:09.447732+0000) 2022-09-07T23:36:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:09 smithi055 ceph-mon[97033]: osdmap e323: 12 total, 11 up, 11 in 2022-09-07T23:36:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:10 smithi059 ceph-mon[97450]: pgmap v3747: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:36:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:10.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:10.546103+0000) 2022-09-07T23:36:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:10.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:10.447875+0000) 2022-09-07T23:36:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:10 smithi055 ceph-mon[97033]: pgmap v3747: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:36:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:11.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:11.546330+0000) 2022-09-07T23:36:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:11.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:11.448077+0000) 2022-09-07T23:36:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:12 smithi059 ceph-mon[97450]: pgmap v3748: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:36:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:12.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:12.546542+0000) 2022-09-07T23:36:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:12.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:12.448277+0000) 2022-09-07T23:36:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:12 smithi055 ceph-mon[97033]: pgmap v3748: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:36:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:13.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:13.546795+0000) 2022-09-07T23:36:13.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:13.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:13.448504+0000) 2022-09-07T23:36:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:14 smithi059 ceph-mon[97450]: pgmap v3749: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:36:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:14 smithi059 ceph-mon[97450]: osdmap e324: 12 total, 11 up, 11 in 2022-09-07T23:36:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:14.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:14.547014+0000) 2022-09-07T23:36:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:14.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:14.448747+0000) 2022-09-07T23:36:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:14 smithi055 ceph-mon[97033]: pgmap v3749: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:36:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:14 smithi055 ceph-mon[97033]: osdmap e324: 12 total, 11 up, 11 in 2022-09-07T23:36:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:15.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:15.547254+0000) 2022-09-07T23:36:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:15.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:15.449024+0000) 2022-09-07T23:36:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:16 smithi059 ceph-mon[97450]: pgmap v3751: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:36:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:16.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:16.547456+0000) 2022-09-07T23:36:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:16.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:16.449280+0000) 2022-09-07T23:36:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:16 smithi055 ceph-mon[97033]: pgmap v3751: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:36:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:17.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:17.547675+0000) 2022-09-07T23:36:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:17.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:17.449420+0000) 2022-09-07T23:36:18.724 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:18.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:18.449631+0000) 2022-09-07T23:36:18.725 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:18.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:18.547901+0000) 2022-09-07T23:36:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:18 smithi059 ceph-mon[97450]: pgmap v3752: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:36:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:18 smithi055 ceph-mon[97033]: pgmap v3752: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:36:19.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:36:18.721+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:36:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:19.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:19.548211+0000) 2022-09-07T23:36:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:19.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:19.449928+0000) 2022-09-07T23:36:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:20 smithi055 ceph-mon[97033]: pgmap v3753: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:36:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:20 smithi059 ceph-mon[97450]: pgmap v3753: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:36:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:20.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:20.548426+0000) 2022-09-07T23:36:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:20.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:20.450152+0000) 2022-09-07T23:36:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:21.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:21.548623+0000) 2022-09-07T23:36:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:21.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:21.450363+0000) 2022-09-07T23:36:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:22 smithi059 ceph-mon[97450]: pgmap v3754: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:36:22.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:22.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:22.548805+0000) 2022-09-07T23:36:22.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:22.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:22.450587+0000) 2022-09-07T23:36:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:22 smithi055 ceph-mon[97033]: pgmap v3754: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:36:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:23.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:23.549054+0000) 2022-09-07T23:36:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:23.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:23.450791+0000) 2022-09-07T23:36:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:24 smithi059 ceph-mon[97450]: pgmap v3755: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:36:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:24.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:24.549325+0000) 2022-09-07T23:36:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:24.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:24.450979+0000) 2022-09-07T23:36:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:24 smithi055 ceph-mon[97033]: pgmap v3755: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:36:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:25.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:25.549598+0000) 2022-09-07T23:36:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:25.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:25.451240+0000) 2022-09-07T23:36:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:26 smithi059 ceph-mon[97450]: pgmap v3756: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 91 B/s wr, 8 op/s 2022-09-07T23:36:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:26.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:26.549821+0000) 2022-09-07T23:36:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:26.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:26.451471+0000) 2022-09-07T23:36:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:26 smithi055 ceph-mon[97033]: pgmap v3756: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 91 B/s wr, 8 op/s 2022-09-07T23:36:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:27.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:27.550019+0000) 2022-09-07T23:36:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:27.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:27.451715+0000) 2022-09-07T23:36:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:28 smithi059 ceph-mon[97450]: pgmap v3757: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:28.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:28.451986+0000) 2022-09-07T23:36:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:28.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:28.550212+0000) 2022-09-07T23:36:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:28 smithi055 ceph-mon[97033]: pgmap v3757: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:29.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:29.550495+0000) 2022-09-07T23:36:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:29.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:29.452236+0000) 2022-09-07T23:36:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:30 smithi055 ceph-mon[97033]: pgmap v3758: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:30 smithi059 ceph-mon[97450]: pgmap v3758: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:30.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:30.550698+0000) 2022-09-07T23:36:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:30.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:30.452506+0000) 2022-09-07T23:36:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:31.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:31.452746+0000) 2022-09-07T23:36:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:31.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:31.550828+0000) 2022-09-07T23:36:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:32 smithi059 ceph-mon[97450]: pgmap v3759: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:36:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:32.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:32.550960+0000) 2022-09-07T23:36:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:32.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:32.452929+0000) 2022-09-07T23:36:32.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:32 smithi055 ceph-mon[97033]: pgmap v3759: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:36:33.728 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:33.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:33.453135+0000) 2022-09-07T23:36:33.729 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:33.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:33.551131+0000) 2022-09-07T23:36:34.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:36:33.726+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:36:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:34 smithi059 ceph-mon[97450]: pgmap v3760: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:34.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:34.551355+0000) 2022-09-07T23:36:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:34.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:34.453342+0000) 2022-09-07T23:36:34.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:34 smithi055 ceph-mon[97033]: pgmap v3760: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:35.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:35.551588+0000) 2022-09-07T23:36:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:35.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:35.453552+0000) 2022-09-07T23:36:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:36 smithi059 ceph-mon[97450]: pgmap v3761: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:36:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:36.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:36.551827+0000) 2022-09-07T23:36:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:36.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:36.453768+0000) 2022-09-07T23:36:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:36 smithi055 ceph-mon[97033]: pgmap v3761: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:36:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:37.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:37.552017+0000) 2022-09-07T23:36:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:37.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:37.454017+0000) 2022-09-07T23:36:38.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:38 smithi059 ceph-mon[97450]: pgmap v3762: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:38.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:38.552167+0000) 2022-09-07T23:36:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:38.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:38.454231+0000) 2022-09-07T23:36:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:38 smithi055 ceph-mon[97033]: pgmap v3762: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:39.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:39.552395+0000) 2022-09-07T23:36:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:39.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:39.454468+0000) 2022-09-07T23:36:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:40 smithi055 ceph-mon[97033]: pgmap v3763: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:40 smithi059 ceph-mon[97450]: pgmap v3763: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:40.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:40.552666+0000) 2022-09-07T23:36:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:40.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:40.454741+0000) 2022-09-07T23:36:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:41.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:41.552846+0000) 2022-09-07T23:36:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:41.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:41.454949+0000) 2022-09-07T23:36:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:42 smithi059 ceph-mon[97450]: pgmap v3764: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:36:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:42.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:42.553031+0000) 2022-09-07T23:36:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:42.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:42.455081+0000) 2022-09-07T23:36:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:42 smithi055 ceph-mon[97033]: pgmap v3764: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:36:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:43.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:43.553151+0000) 2022-09-07T23:36:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:43.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:43.455302+0000) 2022-09-07T23:36:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:44 smithi059 ceph-mon[97450]: pgmap v3765: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:44.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:44.553284+0000) 2022-09-07T23:36:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:44.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:44.455561+0000) 2022-09-07T23:36:44.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:44 smithi055 ceph-mon[97033]: pgmap v3765: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:45.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:45.553427+0000) 2022-09-07T23:36:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:45.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:45.455828+0000) 2022-09-07T23:36:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:46 smithi059 ceph-mon[97450]: pgmap v3766: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:36:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:36:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:36:46.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:36:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:46.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:46.456091+0000) 2022-09-07T23:36:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:46.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:46.553617+0000) 2022-09-07T23:36:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:46 smithi055 ceph-mon[97033]: pgmap v3766: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:36:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:36:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:36:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:36:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:47.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:47.456359+0000) 2022-09-07T23:36:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:47.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:47.553822+0000) 2022-09-07T23:36:48.740 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:48.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:48.554026+0000) 2022-09-07T23:36:48.741 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:48.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:48.456627+0000) 2022-09-07T23:36:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:48 smithi059 ceph-mon[97450]: pgmap v3767: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:48.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:48 smithi055 ceph-mon[97033]: pgmap v3767: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:49.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:36:48.738+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:36:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:49.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:49.554254+0000) 2022-09-07T23:36:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:49.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:49.456898+0000) 2022-09-07T23:36:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:50 smithi055 ceph-mon[97033]: pgmap v3768: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:50 smithi059 ceph-mon[97450]: pgmap v3768: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:50.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:50.554416+0000) 2022-09-07T23:36:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:50.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:50.457106+0000) 2022-09-07T23:36:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:51.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:51.457266+0000) 2022-09-07T23:36:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:51.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:51.554579+0000) 2022-09-07T23:36:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:52 smithi059 ceph-mon[97450]: pgmap v3769: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:36:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:52.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:52.554754+0000) 2022-09-07T23:36:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:52.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:52.457454+0000) 2022-09-07T23:36:52.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:52 smithi055 ceph-mon[97033]: pgmap v3769: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:36:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:53.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:53.554961+0000) 2022-09-07T23:36:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:53.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:53.457711+0000) 2022-09-07T23:36:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:54 smithi059 ceph-mon[97450]: pgmap v3770: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:54.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:54.555195+0000) 2022-09-07T23:36:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:54.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:54.457931+0000) 2022-09-07T23:36:54.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:54 smithi055 ceph-mon[97033]: pgmap v3770: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:55.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:55.555421+0000) 2022-09-07T23:36:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:55.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:55.458082+0000) 2022-09-07T23:36:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:56 smithi059 ceph-mon[97450]: pgmap v3771: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:36:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:56.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:56.555690+0000) 2022-09-07T23:36:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:56.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:56.458294+0000) 2022-09-07T23:36:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:56 smithi055 ceph-mon[97033]: pgmap v3771: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:36:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:57.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:57.555893+0000) 2022-09-07T23:36:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:57.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:57.458556+0000) 2022-09-07T23:36:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:36:58 smithi059 ceph-mon[97450]: pgmap v3772: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:58.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:58.556079+0000) 2022-09-07T23:36:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:58.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:58.458715+0000) 2022-09-07T23:36:58.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:36:58 smithi055 ceph-mon[97033]: pgmap v3772: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:36:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:36:59.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:59.556319+0000) 2022-09-07T23:36:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:36:59.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:36:59.458854+0000) 2022-09-07T23:37:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:00 smithi055 ceph-mon[97033]: pgmap v3773: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:00 smithi059 ceph-mon[97450]: pgmap v3773: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:00.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:00.556549+0000) 2022-09-07T23:37:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:00.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:00.459075+0000) 2022-09-07T23:37:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:01.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:01.556760+0000) 2022-09-07T23:37:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:01.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:01.459303+0000) 2022-09-07T23:37:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:02 smithi059 ceph-mon[97450]: pgmap v3774: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:37:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:02.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:02.556893+0000) 2022-09-07T23:37:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:02.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:02.459538+0000) 2022-09-07T23:37:02.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:02 smithi055 ceph-mon[97033]: pgmap v3774: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:37:03.752 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:03.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:03.459790+0000) 2022-09-07T23:37:03.753 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:03.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:03.557120+0000) 2022-09-07T23:37:04.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:37:03.749+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:37:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:04 smithi059 ceph-mon[97450]: pgmap v3775: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:37:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:37:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:04.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:04.557354+0000) 2022-09-07T23:37:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:04.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:04.460027+0000) 2022-09-07T23:37:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:04 smithi055 ceph-mon[97033]: pgmap v3775: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:37:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:37:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:05.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:05.557594+0000) 2022-09-07T23:37:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:05.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:05.460250+0000) 2022-09-07T23:37:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:06 smithi059 ceph-mon[97450]: pgmap v3776: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:37:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:06.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:06.557877+0000) 2022-09-07T23:37:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:06.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:06.460457+0000) 2022-09-07T23:37:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:06 smithi055 ceph-mon[97033]: pgmap v3776: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:37:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:07.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:07.558050+0000) 2022-09-07T23:37:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:07.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:07.460691+0000) 2022-09-07T23:37:08.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:08 smithi059 ceph-mon[97450]: pgmap v3777: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:37:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:08.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:08.558217+0000) 2022-09-07T23:37:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:08.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:08.460931+0000) 2022-09-07T23:37:08.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:08 smithi055 ceph-mon[97033]: pgmap v3777: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:37:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:09 smithi059 ceph-mon[97450]: osdmap e325: 12 total, 11 up, 11 in 2022-09-07T23:37:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:09.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:09.558457+0000) 2022-09-07T23:37:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:09.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:09.461207+0000) 2022-09-07T23:37:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:09 smithi055 ceph-mon[97033]: osdmap e325: 12 total, 11 up, 11 in 2022-09-07T23:37:10.719 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:10.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:10.558593+0000) 2022-09-07T23:37:10.719 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:10.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:10.461370+0000) 2022-09-07T23:37:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:10 smithi059 ceph-mon[97450]: pgmap v3779: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:37:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:10 smithi055 ceph-mon[97033]: pgmap v3779: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:37:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:11.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:11.558807+0000) 2022-09-07T23:37:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:11.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:11.461587+0000) 2022-09-07T23:37:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:12 smithi059 ceph-mon[97450]: pgmap v3780: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:37:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:12.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:12.559056+0000) 2022-09-07T23:37:12.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:12.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:12.461875+0000) 2022-09-07T23:37:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:12 smithi055 ceph-mon[97033]: pgmap v3780: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:37:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:13.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:13.462147+0000) 2022-09-07T23:37:13.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:13.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:13.559285+0000) 2022-09-07T23:37:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:14 smithi059 ceph-mon[97450]: pgmap v3781: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:37:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:14 smithi059 ceph-mon[97450]: osdmap e326: 12 total, 11 up, 11 in 2022-09-07T23:37:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:14.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:14.462396+0000) 2022-09-07T23:37:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:14.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:14.559493+0000) 2022-09-07T23:37:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:14 smithi055 ceph-mon[97033]: pgmap v3781: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:37:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:14 smithi055 ceph-mon[97033]: osdmap e326: 12 total, 11 up, 11 in 2022-09-07T23:37:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:15.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:15.559709+0000) 2022-09-07T23:37:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:15.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:15.462629+0000) 2022-09-07T23:37:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:16.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:16.559877+0000) 2022-09-07T23:37:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:16.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:16.462907+0000) 2022-09-07T23:37:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:16 smithi055 ceph-mon[97033]: pgmap v3783: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:37:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:16 smithi055 ceph-mon[97033]: pgmap v3784: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.5 KiB/s rd, 285 B/s wr, 8 op/s 2022-09-07T23:37:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:16 smithi055 ceph-mon[97033]: pgmap v3785: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 5.1 KiB/s rd, 328 B/s wr, 10 op/s 2022-09-07T23:37:16.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:37:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:16 smithi059 ceph-mon[97450]: pgmap v3783: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:37:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:16 smithi059 ceph-mon[97450]: pgmap v3784: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.5 KiB/s rd, 285 B/s wr, 8 op/s 2022-09-07T23:37:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:16 smithi059 ceph-mon[97450]: pgmap v3785: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 5.1 KiB/s rd, 328 B/s wr, 10 op/s 2022-09-07T23:37:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:37:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:17.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:17.560066+0000) 2022-09-07T23:37:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:17.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:17.463087+0000) 2022-09-07T23:37:18.767 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:18.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:18.560284+0000) 2022-09-07T23:37:18.768 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:18.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:18.463298+0000) 2022-09-07T23:37:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:18 smithi055 ceph-mon[97033]: pgmap v3786: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-07T23:37:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:18 smithi059 ceph-mon[97450]: pgmap v3786: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-07T23:37:19.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:37:18.764+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:37:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:19.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:19.560525+0000) 2022-09-07T23:37:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:19.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:19.463548+0000) 2022-09-07T23:37:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:20 smithi055 ceph-mon[97033]: pgmap v3787: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-07T23:37:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:20 smithi059 ceph-mon[97450]: pgmap v3787: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-07T23:37:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:20.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:20.560705+0000) 2022-09-07T23:37:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:20.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:20.463737+0000) 2022-09-07T23:37:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:21.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:21.560859+0000) 2022-09-07T23:37:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:21.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:21.463948+0000) 2022-09-07T23:37:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:22.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:22.561056+0000) 2022-09-07T23:37:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:22.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:22.464180+0000) 2022-09-07T23:37:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:22 smithi059 ceph-mon[97450]: pgmap v3788: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 136 B/s wr, 8 op/s 2022-09-07T23:37:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:22 smithi055 ceph-mon[97033]: pgmap v3788: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 136 B/s wr, 8 op/s 2022-09-07T23:37:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:23.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:23.561192+0000) 2022-09-07T23:37:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:23.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:23.464461+0000) 2022-09-07T23:37:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:24.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:24.561409+0000) 2022-09-07T23:37:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:24.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:24.464721+0000) 2022-09-07T23:37:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:24 smithi059 ceph-mon[97450]: pgmap v3789: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:37:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:24 smithi055 ceph-mon[97033]: pgmap v3789: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:37:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:25.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:25.561656+0000) 2022-09-07T23:37:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:25.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:25.464970+0000) 2022-09-07T23:37:26.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:26.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:26.465207+0000) 2022-09-07T23:37:26.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:26.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:26.561843+0000) 2022-09-07T23:37:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:26 smithi059 ceph-mon[97450]: pgmap v3790: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:37:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:26 smithi055 ceph-mon[97033]: pgmap v3790: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:37:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:27.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:27.562068+0000) 2022-09-07T23:37:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:27.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:27.465353+0000) 2022-09-07T23:37:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:28.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:28.562347+0000) 2022-09-07T23:37:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:28.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:28.465479+0000) 2022-09-07T23:37:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:28 smithi059 ceph-mon[97450]: pgmap v3791: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:28 smithi055 ceph-mon[97033]: pgmap v3791: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:29.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:29.562601+0000) 2022-09-07T23:37:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:29.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:29.465690+0000) 2022-09-07T23:37:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:30 smithi055 ceph-mon[97033]: pgmap v3792: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:30 smithi059 ceph-mon[97450]: pgmap v3792: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:30.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:30.562830+0000) 2022-09-07T23:37:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:30.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:30.465863+0000) 2022-09-07T23:37:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:31.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:31.563028+0000) 2022-09-07T23:37:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:31.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:31.465997+0000) 2022-09-07T23:37:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:32.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:32.563145+0000) 2022-09-07T23:37:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:32.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:32.466137+0000) 2022-09-07T23:37:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:32 smithi059 ceph-mon[97450]: pgmap v3793: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:37:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:32 smithi055 ceph-mon[97033]: pgmap v3793: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:37:33.776 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:33.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:33.466330+0000) 2022-09-07T23:37:33.777 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:33.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:33.563377+0000) 2022-09-07T23:37:34.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:37:33.774+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:37:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:34.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:34.563590+0000) 2022-09-07T23:37:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:34.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:34.466517+0000) 2022-09-07T23:37:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:34 smithi059 ceph-mon[97450]: pgmap v3794: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:34 smithi055 ceph-mon[97033]: pgmap v3794: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:35.738 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:35.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:35.563808+0000) 2022-09-07T23:37:35.738 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:35.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:35.466765+0000) 2022-09-07T23:37:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:36.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:36.563969+0000) 2022-09-07T23:37:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:36.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:36.466910+0000) 2022-09-07T23:37:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:36 smithi059 ceph-mon[97450]: pgmap v3795: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:37:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:36 smithi055 ceph-mon[97033]: pgmap v3795: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:37:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:37.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:37.564176+0000) 2022-09-07T23:37:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:37.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:37.467083+0000) 2022-09-07T23:37:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:38.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:38.564407+0000) 2022-09-07T23:37:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:38.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:38.467262+0000) 2022-09-07T23:37:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:38 smithi059 ceph-mon[97450]: pgmap v3796: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:38 smithi055 ceph-mon[97033]: pgmap v3796: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:39.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:39.564725+0000) 2022-09-07T23:37:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:39.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:39.467499+0000) 2022-09-07T23:37:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:40 smithi055 ceph-mon[97033]: pgmap v3797: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:40 smithi059 ceph-mon[97450]: pgmap v3797: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:40.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:40.467671+0000) 2022-09-07T23:37:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:40.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:40.564955+0000) 2022-09-07T23:37:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:41.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:41.565204+0000) 2022-09-07T23:37:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:41.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:41.467842+0000) 2022-09-07T23:37:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:42.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:42.565428+0000) 2022-09-07T23:37:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:42.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:42.468092+0000) 2022-09-07T23:37:43.043 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:42 smithi059 ceph-mon[97450]: pgmap v3798: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:37:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:42 smithi055 ceph-mon[97033]: pgmap v3798: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:37:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:43.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:43.565674+0000) 2022-09-07T23:37:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:43.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:43.468386+0000) 2022-09-07T23:37:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:44.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:44.565907+0000) 2022-09-07T23:37:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:44.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:44.468700+0000) 2022-09-07T23:37:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:44 smithi059 ceph-mon[97450]: pgmap v3799: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:44 smithi055 ceph-mon[97033]: pgmap v3799: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:45.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:45.566055+0000) 2022-09-07T23:37:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:45.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:45.468880+0000) 2022-09-07T23:37:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:46.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:46.469046+0000) 2022-09-07T23:37:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:46.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:46.566216+0000) 2022-09-07T23:37:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:46 smithi059 ceph-mon[97450]: pgmap v3800: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:37:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:46 smithi055 ceph-mon[97033]: pgmap v3800: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:37:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:47.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:47.469230+0000) 2022-09-07T23:37:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:47.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:47.566461+0000) 2022-09-07T23:37:48.789 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:48.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:48.566663+0000) 2022-09-07T23:37:48.790 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:48.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:48.469419+0000) 2022-09-07T23:37:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:48 smithi059 ceph-mon[97450]: pgmap v3801: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:49.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:37:48.787+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:37:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:48 smithi055 ceph-mon[97033]: pgmap v3801: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:49.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:49.566838+0000) 2022-09-07T23:37:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:49.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:49.469605+0000) 2022-09-07T23:37:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:50 smithi055 ceph-mon[97033]: pgmap v3802: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:50 smithi059 ceph-mon[97450]: pgmap v3802: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:50.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:50.566991+0000) 2022-09-07T23:37:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:50.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:50.469816+0000) 2022-09-07T23:37:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:51.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:51.567197+0000) 2022-09-07T23:37:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:51.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:51.469947+0000) 2022-09-07T23:37:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:52.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:52.567459+0000) 2022-09-07T23:37:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:52.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:52.470204+0000) 2022-09-07T23:37:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:52 smithi059 ceph-mon[97450]: pgmap v3803: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:37:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:52 smithi055 ceph-mon[97033]: pgmap v3803: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:37:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:53.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:53.567715+0000) 2022-09-07T23:37:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:53.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:53.470446+0000) 2022-09-07T23:37:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:54.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:54.470680+0000) 2022-09-07T23:37:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:54.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:54.567936+0000) 2022-09-07T23:37:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:54 smithi059 ceph-mon[97450]: pgmap v3804: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:54 smithi055 ceph-mon[97033]: pgmap v3804: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:55.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:55.568121+0000) 2022-09-07T23:37:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:55.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:55.470913+0000) 2022-09-07T23:37:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:56.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:56.568246+0000) 2022-09-07T23:37:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:56.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:56.471154+0000) 2022-09-07T23:37:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:56 smithi059 ceph-mon[97450]: pgmap v3805: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:37:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:56 smithi055 ceph-mon[97033]: pgmap v3805: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:37:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:57.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:57.568406+0000) 2022-09-07T23:37:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:57.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:57.471386+0000) 2022-09-07T23:37:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:58.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:58.568697+0000) 2022-09-07T23:37:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:58.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:58.471606+0000) 2022-09-07T23:37:59.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:37:58 smithi059 ceph-mon[97450]: pgmap v3806: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:59.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:37:58 smithi055 ceph-mon[97033]: pgmap v3806: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:37:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:37:59.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:59.568970+0000) 2022-09-07T23:37:59.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:37:59.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:37:59.471858+0000) 2022-09-07T23:38:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:00 smithi055 ceph-mon[97033]: pgmap v3807: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:00 smithi059 ceph-mon[97450]: pgmap v3807: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:00.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:00.569164+0000) 2022-09-07T23:38:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:00.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:00.472118+0000) 2022-09-07T23:38:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:01.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:01.569393+0000) 2022-09-07T23:38:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:01.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:01.472363+0000) 2022-09-07T23:38:02.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:02.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:02.569616+0000) 2022-09-07T23:38:02.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:02.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:02.472626+0000) 2022-09-07T23:38:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:02 smithi059 ceph-mon[97450]: pgmap v3808: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:38:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:02 smithi055 ceph-mon[97033]: pgmap v3808: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:38:03.795 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:03.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:03.569787+0000) 2022-09-07T23:38:03.796 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:03.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:03.472899+0000) 2022-09-07T23:38:04.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:38:03.792+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:38:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:04.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:04.570028+0000) 2022-09-07T23:38:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:04.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:04.473097+0000) 2022-09-07T23:38:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:04 smithi059 ceph-mon[97450]: pgmap v3809: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:38:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:38:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:04 smithi055 ceph-mon[97033]: pgmap v3809: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:05.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:38:05.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:38:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:05.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:05.570198+0000) 2022-09-07T23:38:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:05.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:05.473403+0000) 2022-09-07T23:38:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:06.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:06.473622+0000) 2022-09-07T23:38:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:06.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:06.570425+0000) 2022-09-07T23:38:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:06 smithi059 ceph-mon[97450]: pgmap v3810: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:38:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:06 smithi055 ceph-mon[97033]: pgmap v3810: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:38:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:07.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:07.570598+0000) 2022-09-07T23:38:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:07.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:07.473896+0000) 2022-09-07T23:38:08.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:08.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:08.570911+0000) 2022-09-07T23:38:08.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:08.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:08.474144+0000) 2022-09-07T23:38:09.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:08 smithi059 ceph-mon[97450]: pgmap v3811: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:38:09.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:08 smithi055 ceph-mon[97033]: pgmap v3811: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:38:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:09.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:09.474410+0000) 2022-09-07T23:38:09.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:09.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:09.571056+0000) 2022-09-07T23:38:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:09 smithi059 ceph-mon[97450]: osdmap e327: 12 total, 11 up, 11 in 2022-09-07T23:38:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:09 smithi055 ceph-mon[97033]: osdmap e327: 12 total, 11 up, 11 in 2022-09-07T23:38:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:10.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:10.571308+0000) 2022-09-07T23:38:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:10.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:10.474678+0000) 2022-09-07T23:38:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:10 smithi059 ceph-mon[97450]: pgmap v3813: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:38:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:10 smithi055 ceph-mon[97033]: pgmap v3813: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:38:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:11.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:11.571541+0000) 2022-09-07T23:38:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:11.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:11.474857+0000) 2022-09-07T23:38:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:12.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:12.571708+0000) 2022-09-07T23:38:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:12.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:12.475144+0000) 2022-09-07T23:38:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:12 smithi059 ceph-mon[97450]: pgmap v3814: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:38:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:12 smithi055 ceph-mon[97033]: pgmap v3814: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:38:13.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:13.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:13.571836+0000) 2022-09-07T23:38:13.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:13.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:13.475345+0000) 2022-09-07T23:38:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:14.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:14.572001+0000) 2022-09-07T23:38:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:14.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:14.475561+0000) 2022-09-07T23:38:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:14 smithi059 ceph-mon[97450]: pgmap v3815: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:38:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:14 smithi059 ceph-mon[97450]: osdmap e328: 12 total, 11 up, 11 in 2022-09-07T23:38:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:14 smithi055 ceph-mon[97033]: pgmap v3815: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:38:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:14 smithi055 ceph-mon[97033]: osdmap e328: 12 total, 11 up, 11 in 2022-09-07T23:38:15.827 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:15.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:15.475811+0000) 2022-09-07T23:38:15.828 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:15.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:15.572230+0000) 2022-09-07T23:38:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:16.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:16.572451+0000) 2022-09-07T23:38:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:16.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:16.476014+0000) 2022-09-07T23:38:17.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:16 smithi059 ceph-mon[97450]: pgmap v3817: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:38:17.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:38:17.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:38:17.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:38:17.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:38:17.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:38:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:16 smithi055 ceph-mon[97033]: pgmap v3817: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:38:17.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:38:17.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:38:17.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:38:17.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:38:17.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:38:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:17.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:17.572673+0000) 2022-09-07T23:38:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:17.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:17.476242+0000) 2022-09-07T23:38:18.809 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:18.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:18.572886+0000) 2022-09-07T23:38:18.811 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:18.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:18.476467+0000) 2022-09-07T23:38:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:18 smithi059 ceph-mon[97450]: pgmap v3818: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:38:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:38:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:38:19.116 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:18 smithi055 ceph-mon[97033]: pgmap v3818: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:38:19.117 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:38:19.118 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:38:19.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:38:18.807+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:38:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:19.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:19.573056+0000) 2022-09-07T23:38:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:19.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:19.476688+0000) 2022-09-07T23:38:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:38:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:38:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:38:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:38:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:38:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:20.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:20.476922+0000) 2022-09-07T23:38:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:20.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:20.573271+0000) 2022-09-07T23:38:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:21 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T23:38:21.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:21 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T23:38:21.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:21 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T23:38:21.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:21 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T23:38:21.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:21 smithi055 ceph-mon[97033]: pgmap v3819: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:38:21.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:21 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T23:38:21.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:21 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T23:38:21.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:21 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T23:38:21.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:21 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T23:38:21.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:21 smithi059 ceph-mon[97450]: pgmap v3819: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:38:21.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:21.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:21.477143+0000) 2022-09-07T23:38:21.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:21.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:21.573496+0000) 2022-09-07T23:38:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:22.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:22.477398+0000) 2022-09-07T23:38:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:22.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:22.573764+0000) 2022-09-07T23:38:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:23 smithi055 ceph-mon[97033]: pgmap v3820: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:38:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:23 smithi059 ceph-mon[97450]: pgmap v3820: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:38:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:23.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:23.477626+0000) 2022-09-07T23:38:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:23.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:23.573989+0000) 2022-09-07T23:38:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:24.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:24.477866+0000) 2022-09-07T23:38:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:24.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:24.574164+0000) 2022-09-07T23:38:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:25 smithi055 ceph-mon[97033]: pgmap v3821: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:38:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:25 smithi059 ceph-mon[97450]: pgmap v3821: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:38:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:25.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:25.574412+0000) 2022-09-07T23:38:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:25.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:25.478090+0000) 2022-09-07T23:38:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:26.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:26.478293+0000) 2022-09-07T23:38:26.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:26.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:26.574690+0000) 2022-09-07T23:38:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:27 smithi055 ceph-mon[97033]: pgmap v3822: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-07T23:38:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:27 smithi059 ceph-mon[97450]: pgmap v3822: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-07T23:38:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:27.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:27.574866+0000) 2022-09-07T23:38:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:27.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:27.478566+0000) 2022-09-07T23:38:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:28.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:28.575076+0000) 2022-09-07T23:38:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:28.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:28.478774+0000) 2022-09-07T23:38:29.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:29 smithi055 ceph-mon[97033]: pgmap v3823: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:29 smithi059 ceph-mon[97450]: pgmap v3823: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:29.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:29.479055+0000) 2022-09-07T23:38:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:29.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:29.575302+0000) 2022-09-07T23:38:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:30 smithi059 ceph-mon[97450]: pgmap v3824: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:30 smithi055 ceph-mon[97033]: pgmap v3824: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:30.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:30.479330+0000) 2022-09-07T23:38:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:30.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:30.575448+0000) 2022-09-07T23:38:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:31.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:31.479586+0000) 2022-09-07T23:38:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:31.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:31.575598+0000) 2022-09-07T23:38:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:32.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:32.479777+0000) 2022-09-07T23:38:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:32.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:32.575790+0000) 2022-09-07T23:38:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:32 smithi059 ceph-mon[97450]: pgmap v3825: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:38:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:32 smithi055 ceph-mon[97033]: pgmap v3825: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:38:33.812 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:33.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:33.575955+0000) 2022-09-07T23:38:33.813 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:33.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:33.480014+0000) 2022-09-07T23:38:34.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:38:33.810+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:38:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:34.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:34.480207+0000) 2022-09-07T23:38:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:34.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:34.576199+0000) 2022-09-07T23:38:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:34 smithi059 ceph-mon[97450]: pgmap v3826: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:34 smithi055 ceph-mon[97033]: pgmap v3826: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:35.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:35.576419+0000) 2022-09-07T23:38:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:35.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:35.480430+0000) 2022-09-07T23:38:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:36.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:36.576673+0000) 2022-09-07T23:38:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:36.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:36.480717+0000) 2022-09-07T23:38:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:36 smithi059 ceph-mon[97450]: pgmap v3827: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:38:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:36 smithi055 ceph-mon[97033]: pgmap v3827: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:38:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:37.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:37.576880+0000) 2022-09-07T23:38:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:37.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:37.480951+0000) 2022-09-07T23:38:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:38.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:38.577110+0000) 2022-09-07T23:38:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:38.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:38.481188+0000) 2022-09-07T23:38:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:38 smithi059 ceph-mon[97450]: pgmap v3828: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:38 smithi055 ceph-mon[97033]: pgmap v3828: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:39.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:39.577270+0000) 2022-09-07T23:38:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:39.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:39.481420+0000) 2022-09-07T23:38:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:40 smithi059 ceph-mon[97450]: pgmap v3829: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:40 smithi055 ceph-mon[97033]: pgmap v3829: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:40.879 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:40.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:40.577417+0000) 2022-09-07T23:38:40.880 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:40.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:40.481691+0000) 2022-09-07T23:38:41.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:41.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:41.577604+0000) 2022-09-07T23:38:41.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:41.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:41.481935+0000) 2022-09-07T23:38:42.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:42.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:42.577815+0000) 2022-09-07T23:38:42.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:42.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:42.482169+0000) 2022-09-07T23:38:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:42 smithi059 ceph-mon[97450]: pgmap v3830: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:38:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:42 smithi055 ceph-mon[97033]: pgmap v3830: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:38:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:43.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:43.578017+0000) 2022-09-07T23:38:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:43.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:43.482454+0000) 2022-09-07T23:38:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:44.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:44.578239+0000) 2022-09-07T23:38:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:44.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:44.482694+0000) 2022-09-07T23:38:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:44 smithi059 ceph-mon[97450]: pgmap v3831: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:45.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:44 smithi055 ceph-mon[97033]: pgmap v3831: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:45.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:45.578491+0000) 2022-09-07T23:38:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:45.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:45.482877+0000) 2022-09-07T23:38:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:46.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:46.578684+0000) 2022-09-07T23:38:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:46.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:46.483120+0000) 2022-09-07T23:38:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:46 smithi055 ceph-mon[97033]: pgmap v3832: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:38:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:46 smithi055 ceph-mon[97033]: pgmap v3833: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:38:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:46 smithi055 ceph-mon[97033]: pgmap v3834: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:38:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:38:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:46 smithi059 ceph-mon[97450]: pgmap v3832: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:38:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:46 smithi059 ceph-mon[97450]: pgmap v3833: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:38:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:46 smithi059 ceph-mon[97450]: pgmap v3834: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:38:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:38:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:47.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:47.578816+0000) 2022-09-07T23:38:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:47.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:47.483296+0000) 2022-09-07T23:38:48.818 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:48.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:48.578981+0000) 2022-09-07T23:38:48.819 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:48.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:48.483539+0000) 2022-09-07T23:38:49.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:38:48.816+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:38:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:48 smithi055 ceph-mon[97033]: pgmap v3835: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:38:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:48 smithi059 ceph-mon[97450]: pgmap v3835: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:38:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:49.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:49.579238+0000) 2022-09-07T23:38:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:49.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:49.483741+0000) 2022-09-07T23:38:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:50 smithi059 ceph-mon[97450]: pgmap v3836: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T23:38:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:50 smithi055 ceph-mon[97033]: pgmap v3836: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-07T23:38:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:50.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:50.579485+0000) 2022-09-07T23:38:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:50.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:50.483882+0000) 2022-09-07T23:38:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:51.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:51.579755+0000) 2022-09-07T23:38:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:51.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:51.484112+0000) 2022-09-07T23:38:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:52.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:52.579997+0000) 2022-09-07T23:38:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:52.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:52.484286+0000) 2022-09-07T23:38:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:52 smithi059 ceph-mon[97450]: pgmap v3837: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:38:53.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:52 smithi055 ceph-mon[97033]: pgmap v3837: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:38:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:53.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:53.580147+0000) 2022-09-07T23:38:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:53.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:53.484538+0000) 2022-09-07T23:38:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:54.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:54.580359+0000) 2022-09-07T23:38:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:54.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:54.484776+0000) 2022-09-07T23:38:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:54 smithi059 ceph-mon[97450]: pgmap v3838: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:38:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:54 smithi055 ceph-mon[97033]: pgmap v3838: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:38:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:55.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:55.580568+0000) 2022-09-07T23:38:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:55.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:55.485023+0000) 2022-09-07T23:38:56.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:56.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:56.580801+0000) 2022-09-07T23:38:56.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:56.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:56.485276+0000) 2022-09-07T23:38:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:56 smithi059 ceph-mon[97450]: pgmap v3839: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:38:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:57 smithi055 ceph-mon[97033]: pgmap v3839: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:38:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:57.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:57.580951+0000) 2022-09-07T23:38:57.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:57.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:57.485520+0000) 2022-09-07T23:38:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:58.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:58.581191+0000) 2022-09-07T23:38:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:58.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:58.485782+0000) 2022-09-07T23:38:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:38:59 smithi059 ceph-mon[97450]: pgmap v3840: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:38:59 smithi055 ceph-mon[97033]: pgmap v3840: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:38:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:38:59.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:59.581422+0000) 2022-09-07T23:38:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:38:59.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:38:59.486036+0000) 2022-09-07T23:39:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:00 smithi059 ceph-mon[97450]: pgmap v3841: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:39:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:00 smithi055 ceph-mon[97033]: pgmap v3841: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:39:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:00.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:00.581567+0000) 2022-09-07T23:39:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:00.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:00.486228+0000) 2022-09-07T23:39:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:01.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:01.581734+0000) 2022-09-07T23:39:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:01.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:01.486433+0000) 2022-09-07T23:39:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:02.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:02.581963+0000) 2022-09-07T23:39:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:02.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:02.486692+0000) 2022-09-07T23:39:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:02 smithi059 ceph-mon[97450]: pgmap v3842: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:39:03.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:02 smithi055 ceph-mon[97033]: pgmap v3842: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-07T23:39:03.834 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:03.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:03.582209+0000) 2022-09-07T23:39:03.835 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:03.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:03.486915+0000) 2022-09-07T23:39:04.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:39:03.832+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:39:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:04.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:04.582481+0000) 2022-09-07T23:39:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:04.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:04.487154+0000) 2022-09-07T23:39:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:04 smithi059 ceph-mon[97450]: pgmap v3843: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:39:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:39:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:04 smithi055 ceph-mon[97033]: pgmap v3843: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:39:05.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:39:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:05.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:05.582748+0000) 2022-09-07T23:39:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:05.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:05.487364+0000) 2022-09-07T23:39:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:06.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:06.487566+0000) 2022-09-07T23:39:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:06.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:06.582957+0000) 2022-09-07T23:39:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:07 smithi059 ceph-mon[97450]: pgmap v3844: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:39:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:07 smithi055 ceph-mon[97033]: pgmap v3844: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:39:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:07.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:07.487776+0000) 2022-09-07T23:39:07.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:07.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:07.583095+0000) 2022-09-07T23:39:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:08.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:08.583277+0000) 2022-09-07T23:39:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:08.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:08.488022+0000) 2022-09-07T23:39:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:09 smithi059 ceph-mon[97450]: pgmap v3845: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:39:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:09 smithi055 ceph-mon[97033]: pgmap v3845: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:39:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:09.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:09.488199+0000) 2022-09-07T23:39:09.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:09.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:09.583430+0000) 2022-09-07T23:39:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:10 smithi059 ceph-mon[97450]: osdmap e329: 12 total, 11 up, 11 in 2022-09-07T23:39:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:10 smithi055 ceph-mon[97033]: osdmap e329: 12 total, 11 up, 11 in 2022-09-07T23:39:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:10.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:10.583562+0000) 2022-09-07T23:39:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:10.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:10.488409+0000) 2022-09-07T23:39:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:11 smithi059 ceph-mon[97450]: pgmap v3847: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:39:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:11 smithi055 ceph-mon[97033]: pgmap v3847: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:39:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:11.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:11.583793+0000) 2022-09-07T23:39:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:11.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:11.488673+0000) 2022-09-07T23:39:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:12.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:12.584003+0000) 2022-09-07T23:39:12.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:12.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:12.488867+0000) 2022-09-07T23:39:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:13 smithi059 ceph-mon[97450]: pgmap v3848: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:39:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:13 smithi055 ceph-mon[97033]: pgmap v3848: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:39:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:13.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:13.489107+0000) 2022-09-07T23:39:13.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:13.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:13.584180+0000) 2022-09-07T23:39:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:14.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:14.584400+0000) 2022-09-07T23:39:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:14.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:14.489305+0000) 2022-09-07T23:39:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:15 smithi059 ceph-mon[97450]: pgmap v3849: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:39:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:15 smithi059 ceph-mon[97450]: osdmap e330: 12 total, 11 up, 11 in 2022-09-07T23:39:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:15 smithi055 ceph-mon[97033]: pgmap v3849: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:39:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:15 smithi055 ceph-mon[97033]: osdmap e330: 12 total, 11 up, 11 in 2022-09-07T23:39:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:15.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:15.584627+0000) 2022-09-07T23:39:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:15.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:15.489438+0000) 2022-09-07T23:39:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:16.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:16.584800+0000) 2022-09-07T23:39:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:16.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:16.489622+0000) 2022-09-07T23:39:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:17 smithi059 ceph-mon[97450]: pgmap v3851: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:39:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:17 smithi055 ceph-mon[97033]: pgmap v3851: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:39:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:17.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:17.585024+0000) 2022-09-07T23:39:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:17.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:17.489887+0000) 2022-09-07T23:39:18.842 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:18.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:18.585220+0000) 2022-09-07T23:39:18.843 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:18.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:18.490113+0000) 2022-09-07T23:39:19.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:39:18.840+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:39:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:19 smithi059 ceph-mon[97450]: pgmap v3852: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:39:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:19 smithi055 ceph-mon[97033]: pgmap v3852: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:39:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:19.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:19.585454+0000) 2022-09-07T23:39:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:19.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:19.490377+0000) 2022-09-07T23:39:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:20 smithi059 ceph-mon[97450]: pgmap v3853: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:39:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:20 smithi055 ceph-mon[97033]: pgmap v3853: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:39:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:20.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:20.585665+0000) 2022-09-07T23:39:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:20.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:20.490662+0000) 2022-09-07T23:39:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:21.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:21.585879+0000) 2022-09-07T23:39:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:21.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:21.490924+0000) 2022-09-07T23:39:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:22.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:22.586111+0000) 2022-09-07T23:39:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:22.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:22.491168+0000) 2022-09-07T23:39:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:23 smithi059 ceph-mon[97450]: pgmap v3854: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:39:23.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:23 smithi055 ceph-mon[97033]: pgmap v3854: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:39:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:23.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:23.586269+0000) 2022-09-07T23:39:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:23.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:23.491408+0000) 2022-09-07T23:39:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:24.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:24.586516+0000) 2022-09-07T23:39:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:24.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:24.491670+0000) 2022-09-07T23:39:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:25 smithi059 ceph-mon[97450]: pgmap v3855: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:39:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:25 smithi055 ceph-mon[97033]: pgmap v3855: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:39:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:25.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:25.586748+0000) 2022-09-07T23:39:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:25.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:25.491915+0000) 2022-09-07T23:39:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:26.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:26.492135+0000) 2022-09-07T23:39:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:26.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:26.586965+0000) 2022-09-07T23:39:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:27 smithi059 ceph-mon[97450]: pgmap v3856: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:39:27.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:27 smithi055 ceph-mon[97033]: pgmap v3856: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:39:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:27.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:27.587168+0000) 2022-09-07T23:39:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:27.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:27.492411+0000) 2022-09-07T23:39:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:28.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:28.587327+0000) 2022-09-07T23:39:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:28.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:28.492679+0000) 2022-09-07T23:39:29.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:29 smithi059 ceph-mon[97450]: pgmap v3857: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:39:29.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:29 smithi055 ceph-mon[97033]: pgmap v3857: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:39:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:29.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:29.587499+0000) 2022-09-07T23:39:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:29.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:29.492930+0000) 2022-09-07T23:39:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:30 smithi059 ceph-mon[97450]: pgmap v3858: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:39:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:30 smithi055 ceph-mon[97033]: pgmap v3858: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:39:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:30.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:30.587744+0000) 2022-09-07T23:39:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:30.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:30.493170+0000) 2022-09-07T23:39:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:31.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:31.588024+0000) 2022-09-07T23:39:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:31.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:31.493361+0000) 2022-09-07T23:39:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:32.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:32.588229+0000) 2022-09-07T23:39:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:32.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:32.493589+0000) 2022-09-07T23:39:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:33 smithi059 ceph-mon[97450]: pgmap v3859: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:33.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:33 smithi055 ceph-mon[97033]: pgmap v3859: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:33.851 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:33.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:33.588468+0000) 2022-09-07T23:39:33.851 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:33.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:33.493822+0000) 2022-09-07T23:39:34.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:39:33.848+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:39:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:34.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:34.588744+0000) 2022-09-07T23:39:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:34.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:34.494030+0000) 2022-09-07T23:39:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:35 smithi059 ceph-mon[97450]: pgmap v3860: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:35 smithi055 ceph-mon[97033]: pgmap v3860: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:35.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:35.494264+0000) 2022-09-07T23:39:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:35.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:35.589009+0000) 2022-09-07T23:39:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:36.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:36.494473+0000) 2022-09-07T23:39:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:36.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:36.589174+0000) 2022-09-07T23:39:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:37 smithi059 ceph-mon[97450]: pgmap v3861: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:39:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:37 smithi055 ceph-mon[97033]: pgmap v3861: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:39:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:37.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:37.589407+0000) 2022-09-07T23:39:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:37.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:37.494771+0000) 2022-09-07T23:39:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:38.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:38.494947+0000) 2022-09-07T23:39:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:38.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:38.589628+0000) 2022-09-07T23:39:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:39 smithi059 ceph-mon[97450]: pgmap v3862: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:39 smithi055 ceph-mon[97033]: pgmap v3862: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:39.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:39.589935+0000) 2022-09-07T23:39:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:39.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:39.495148+0000) 2022-09-07T23:39:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:40 smithi059 ceph-mon[97450]: pgmap v3863: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:39:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:40 smithi055 ceph-mon[97033]: pgmap v3863: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:39:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:40.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:40.590083+0000) 2022-09-07T23:39:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:40.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:40.495370+0000) 2022-09-07T23:39:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:41.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:41.590308+0000) 2022-09-07T23:39:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:41.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:41.495621+0000) 2022-09-07T23:39:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:42.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:42.590518+0000) 2022-09-07T23:39:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:42.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:42.495791+0000) 2022-09-07T23:39:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:43 smithi059 ceph-mon[97450]: pgmap v3864: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:43.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:43 smithi055 ceph-mon[97033]: pgmap v3864: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:43.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:43.590771+0000) 2022-09-07T23:39:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:43.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:43.496046+0000) 2022-09-07T23:39:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:44.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:44.590981+0000) 2022-09-07T23:39:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:44.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:44.496269+0000) 2022-09-07T23:39:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:45 smithi059 ceph-mon[97450]: pgmap v3865: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:45 smithi055 ceph-mon[97033]: pgmap v3865: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:45.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:45.591219+0000) 2022-09-07T23:39:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:45.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:45.496540+0000) 2022-09-07T23:39:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:46.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:46.591439+0000) 2022-09-07T23:39:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:46.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:46.496720+0000) 2022-09-07T23:39:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:47 smithi059 ceph-mon[97450]: pgmap v3866: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:39:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:39:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:39:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:39:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:39:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:39:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:47 smithi055 ceph-mon[97033]: pgmap v3866: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:39:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:39:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:39:47.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:39:47.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:39:47.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:39:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:47.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:47.496939+0000) 2022-09-07T23:39:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:47.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:47.591725+0000) 2022-09-07T23:39:48.858 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:48.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:48.591917+0000) 2022-09-07T23:39:48.859 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:48.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:48.497132+0000) 2022-09-07T23:39:49.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:39:48.856+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:39:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:49 smithi059 ceph-mon[97450]: pgmap v3867: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:49 smithi055 ceph-mon[97033]: pgmap v3867: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:49.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:49.497346+0000) 2022-09-07T23:39:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:49.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:49.592063+0000) 2022-09-07T23:39:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:50 smithi059 ceph-mon[97450]: pgmap v3868: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:39:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:50 smithi055 ceph-mon[97033]: pgmap v3868: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:39:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:50.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:50.592269+0000) 2022-09-07T23:39:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:50.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:50.497567+0000) 2022-09-07T23:39:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:51.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:51.592493+0000) 2022-09-07T23:39:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:51.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:51.497781+0000) 2022-09-07T23:39:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:52.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:52.498010+0000) 2022-09-07T23:39:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:52.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:52.592698+0000) 2022-09-07T23:39:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:53 smithi059 ceph-mon[97450]: pgmap v3869: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:53.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:53 smithi055 ceph-mon[97033]: pgmap v3869: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:53.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:53.592900+0000) 2022-09-07T23:39:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:53.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:53.498223+0000) 2022-09-07T23:39:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:54.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:54.498413+0000) 2022-09-07T23:39:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:54.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:54.593090+0000) 2022-09-07T23:39:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:55 smithi059 ceph-mon[97450]: pgmap v3870: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:55 smithi055 ceph-mon[97033]: pgmap v3870: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:55.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:55.593356+0000) 2022-09-07T23:39:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:55.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:55.498623+0000) 2022-09-07T23:39:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:56.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:56.498883+0000) 2022-09-07T23:39:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:56.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:56.593580+0000) 2022-09-07T23:39:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:57 smithi059 ceph-mon[97450]: pgmap v3871: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:39:57.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:57 smithi055 ceph-mon[97033]: pgmap v3871: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:39:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:57.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:57.593800+0000) 2022-09-07T23:39:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:57.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:57.499099+0000) 2022-09-07T23:39:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:58.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:58.593978+0000) 2022-09-07T23:39:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:58.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:58.499332+0000) 2022-09-07T23:39:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:39:59 smithi059 ceph-mon[97450]: pgmap v3872: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:39:59 smithi055 ceph-mon[97033]: pgmap v3872: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:39:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:39:59.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:59.594227+0000) 2022-09-07T23:39:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:39:59.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:39:59.499554+0000) 2022-09-07T23:40:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:00 smithi059 ceph-mon[97450]: pgmap v3873: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:40:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-07T23:40:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-07T23:40:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-07T23:40:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T23:40:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T23:40:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:00 smithi055 ceph-mon[97033]: pgmap v3873: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:40:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-07T23:40:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-07T23:40:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-07T23:40:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T23:40:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T23:40:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:00.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:00.594382+0000) 2022-09-07T23:40:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:00.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:00.499765+0000) 2022-09-07T23:40:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:01.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:01.594597+0000) 2022-09-07T23:40:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:01.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:01.499918+0000) 2022-09-07T23:40:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:02.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:02.594821+0000) 2022-09-07T23:40:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:02.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:02.500077+0000) 2022-09-07T23:40:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:03 smithi059 ceph-mon[97450]: pgmap v3874: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:03 smithi055 ceph-mon[97033]: pgmap v3874: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:03.874 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:03.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:03.500289+0000) 2022-09-07T23:40:03.875 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:03.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:03.595038+0000) 2022-09-07T23:40:04.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:40:03.872+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:40:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:04.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:04.500543+0000) 2022-09-07T23:40:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:04.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:04.595288+0000) 2022-09-07T23:40:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:05 smithi059 ceph-mon[97450]: pgmap v3875: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:40:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:40:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:05 smithi055 ceph-mon[97033]: pgmap v3875: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:40:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:40:05.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:05.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:05.500786+0000) 2022-09-07T23:40:05.912 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:05.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:05.595440+0000) 2022-09-07T23:40:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:06.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:06.595611+0000) 2022-09-07T23:40:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:06.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:06.501022+0000) 2022-09-07T23:40:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:07 smithi059 ceph-mon[97450]: pgmap v3876: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:40:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:07 smithi055 ceph-mon[97033]: pgmap v3876: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:40:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:07.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:07.595751+0000) 2022-09-07T23:40:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:07.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:07.501218+0000) 2022-09-07T23:40:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:08.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:08.595940+0000) 2022-09-07T23:40:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:08.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:08.501446+0000) 2022-09-07T23:40:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:09 smithi059 ceph-mon[97450]: pgmap v3877: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:40:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:09 smithi055 ceph-mon[97033]: pgmap v3877: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:40:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:09.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:09.596081+0000) 2022-09-07T23:40:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:09.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:09.501624+0000) 2022-09-07T23:40:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:10 smithi055 ceph-mon[97033]: osdmap e331: 12 total, 11 up, 11 in 2022-09-07T23:40:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:10 smithi059 ceph-mon[97450]: osdmap e331: 12 total, 11 up, 11 in 2022-09-07T23:40:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:10.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:10.596332+0000) 2022-09-07T23:40:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:10.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:10.501851+0000) 2022-09-07T23:40:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:11 smithi055 ceph-mon[97033]: pgmap v3879: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:40:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:11 smithi059 ceph-mon[97450]: pgmap v3879: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:40:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:11.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:11.596548+0000) 2022-09-07T23:40:11.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:11.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:11.502066+0000) 2022-09-07T23:40:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:12.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:12.596794+0000) 2022-09-07T23:40:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:12.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:12.502286+0000) 2022-09-07T23:40:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:13 smithi055 ceph-mon[97033]: pgmap v3880: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:40:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:13 smithi059 ceph-mon[97450]: pgmap v3880: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:40:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:13.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:13.597074+0000) 2022-09-07T23:40:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:13.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:13.502519+0000) 2022-09-07T23:40:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:14.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:14.597307+0000) 2022-09-07T23:40:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:14.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:14.502763+0000) 2022-09-07T23:40:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:15 smithi055 ceph-mon[97033]: pgmap v3881: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:40:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:15 smithi055 ceph-mon[97033]: osdmap e332: 12 total, 11 up, 11 in 2022-09-07T23:40:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:15 smithi059 ceph-mon[97450]: pgmap v3881: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:40:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:15 smithi059 ceph-mon[97450]: osdmap e332: 12 total, 11 up, 11 in 2022-09-07T23:40:15.773 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:15.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:15.597564+0000) 2022-09-07T23:40:15.773 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:15.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:15.503010+0000) 2022-09-07T23:40:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:16.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:16.597760+0000) 2022-09-07T23:40:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:16.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:16.503259+0000) 2022-09-07T23:40:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:17 smithi055 ceph-mon[97033]: pgmap v3883: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:40:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:17 smithi059 ceph-mon[97450]: pgmap v3883: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:40:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:17.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:17.597928+0000) 2022-09-07T23:40:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:17.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:17.503518+0000) 2022-09-07T23:40:18.877 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:18.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:18.598068+0000) 2022-09-07T23:40:18.879 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:18.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:18.503734+0000) 2022-09-07T23:40:19.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:40:18.874+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:40:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:19 smithi055 ceph-mon[97033]: pgmap v3884: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:40:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:19 smithi059 ceph-mon[97450]: pgmap v3884: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:40:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:19.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:19.503972+0000) 2022-09-07T23:40:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:19.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:19.598307+0000) 2022-09-07T23:40:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:20 smithi059 ceph-mon[97450]: pgmap v3885: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T23:40:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:20 smithi055 ceph-mon[97033]: pgmap v3885: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T23:40:20.760 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:20.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:20.598550+0000) 2022-09-07T23:40:20.761 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:20.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:20.504200+0000) 2022-09-07T23:40:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:21.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:21.598782+0000) 2022-09-07T23:40:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:21.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:21.504393+0000) 2022-09-07T23:40:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:40:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:40:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:21 smithi059 ceph-mon[97450]: pgmap v3886: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 116 B/s wr, 6 op/s 2022-09-07T23:40:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:21 smithi059 ceph-mon[97450]: pgmap v3887: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 150 B/s wr, 8 op/s 2022-09-07T23:40:22.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:40:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:40:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:40:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:21 smithi055 ceph-mon[97033]: pgmap v3886: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 116 B/s wr, 6 op/s 2022-09-07T23:40:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:21 smithi055 ceph-mon[97033]: pgmap v3887: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 150 B/s wr, 8 op/s 2022-09-07T23:40:22.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:40:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:22.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:22.599022+0000) 2022-09-07T23:40:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:22.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:22.504588+0000) 2022-09-07T23:40:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:23.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:23.599156+0000) 2022-09-07T23:40:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:23.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:23.504818+0000) 2022-09-07T23:40:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:23 smithi055 ceph-mon[97033]: pgmap v3888: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 120 B/s wr, 6 op/s 2022-09-07T23:40:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:23 smithi059 ceph-mon[97450]: pgmap v3888: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 120 B/s wr, 6 op/s 2022-09-07T23:40:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:24.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:24.599370+0000) 2022-09-07T23:40:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:24.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:24.505055+0000) 2022-09-07T23:40:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:25 smithi059 ceph-mon[97450]: pgmap v3889: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.6 KiB/s rd, 3 op/s 2022-09-07T23:40:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:25 smithi055 ceph-mon[97033]: pgmap v3889: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.6 KiB/s rd, 3 op/s 2022-09-07T23:40:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:25.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:25.599679+0000) 2022-09-07T23:40:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:25.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:25.505271+0000) 2022-09-07T23:40:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:26.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:26.599918+0000) 2022-09-07T23:40:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:26.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:26.505532+0000) 2022-09-07T23:40:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:27.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:27.600148+0000) 2022-09-07T23:40:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:27.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:27.505738+0000) 2022-09-07T23:40:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:27 smithi055 ceph-mon[97033]: pgmap v3890: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-07T23:40:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:27 smithi059 ceph-mon[97450]: pgmap v3890: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-07T23:40:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:28.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:28.505910+0000) 2022-09-07T23:40:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:28.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:28.600352+0000) 2022-09-07T23:40:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:29.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:29.600585+0000) 2022-09-07T23:40:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:29.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:29.506128+0000) 2022-09-07T23:40:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:29 smithi055 ceph-mon[97033]: pgmap v3891: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.6 KiB/s rd, 3 op/s 2022-09-07T23:40:30.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:29 smithi059 ceph-mon[97450]: pgmap v3891: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1.6 KiB/s rd, 3 op/s 2022-09-07T23:40:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:30.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:30.506282+0000) 2022-09-07T23:40:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:30.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:30.600763+0000) 2022-09-07T23:40:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:31.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:31.601011+0000) 2022-09-07T23:40:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:31.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:31.506489+0000) 2022-09-07T23:40:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:31 smithi055 ceph-mon[97033]: pgmap v3892: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:40:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:31 smithi059 ceph-mon[97450]: pgmap v3892: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:40:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:32.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:32.601249+0000) 2022-09-07T23:40:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:32.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:32.506686+0000) 2022-09-07T23:40:33.884 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:33.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:33.601520+0000) 2022-09-07T23:40:33.885 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:33.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:33.506868+0000) 2022-09-07T23:40:34.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:40:33.880+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:40:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:33 smithi055 ceph-mon[97033]: pgmap v3893: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:40:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:33 smithi059 ceph-mon[97450]: pgmap v3893: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:40:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:34.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:34.507035+0000) 2022-09-07T23:40:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:34.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:34.601739+0000) 2022-09-07T23:40:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:35 smithi059 ceph-mon[97450]: pgmap v3894: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:40:35.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:35 smithi055 ceph-mon[97033]: pgmap v3894: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:40:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:35.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:35.601980+0000) 2022-09-07T23:40:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:35.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:35.507248+0000) 2022-09-07T23:40:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:36.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:36.602184+0000) 2022-09-07T23:40:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:36.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:36.507506+0000) 2022-09-07T23:40:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:37.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:37.507760+0000) 2022-09-07T23:40:37.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:37.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:37.602435+0000) 2022-09-07T23:40:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:37 smithi055 ceph-mon[97033]: pgmap v3895: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:40:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:37 smithi059 ceph-mon[97450]: pgmap v3895: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:40:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:38.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:38.602734+0000) 2022-09-07T23:40:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:38.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:38.508018+0000) 2022-09-07T23:40:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:39.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:39.602927+0000) 2022-09-07T23:40:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:39.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:39.508207+0000) 2022-09-07T23:40:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:39 smithi055 ceph-mon[97033]: pgmap v3896: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:39 smithi059 ceph-mon[97450]: pgmap v3896: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:40.781 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:40.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:40.603128+0000) 2022-09-07T23:40:40.782 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:40.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:40.508432+0000) 2022-09-07T23:40:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:41.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:41.603303+0000) 2022-09-07T23:40:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:41.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:41.508725+0000) 2022-09-07T23:40:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:41 smithi055 ceph-mon[97033]: pgmap v3897: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:40:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:41 smithi059 ceph-mon[97450]: pgmap v3897: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:40:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:42.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:42.603510+0000) 2022-09-07T23:40:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:42.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:42.508859+0000) 2022-09-07T23:40:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:43.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:43.603732+0000) 2022-09-07T23:40:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:43.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:43.509076+0000) 2022-09-07T23:40:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:43 smithi055 ceph-mon[97033]: pgmap v3898: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:43 smithi059 ceph-mon[97450]: pgmap v3898: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:44.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:44.603969+0000) 2022-09-07T23:40:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:44.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:44.509285+0000) 2022-09-07T23:40:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:45 smithi059 ceph-mon[97450]: pgmap v3899: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:45 smithi055 ceph-mon[97033]: pgmap v3899: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:45.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:45.604190+0000) 2022-09-07T23:40:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:45.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:45.509530+0000) 2022-09-07T23:40:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:46.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:46.604408+0000) 2022-09-07T23:40:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:46.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:46.509766+0000) 2022-09-07T23:40:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:47.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:47.604620+0000) 2022-09-07T23:40:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:47.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:47.509929+0000) 2022-09-07T23:40:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:47 smithi055 ceph-mon[97033]: pgmap v3900: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:40:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:47 smithi059 ceph-mon[97450]: pgmap v3900: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:40:48.894 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:48.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:48.510172+0000) 2022-09-07T23:40:48.895 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:48.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:48.604822+0000) 2022-09-07T23:40:49.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:40:48.892+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:40:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:49.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:49.510400+0000) 2022-09-07T23:40:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:49.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:49.605035+0000) 2022-09-07T23:40:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:49 smithi055 ceph-mon[97033]: pgmap v3901: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:49 smithi059 ceph-mon[97450]: pgmap v3901: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:50.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:50.605262+0000) 2022-09-07T23:40:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:50.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:50.510677+0000) 2022-09-07T23:40:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:51.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:51.605461+0000) 2022-09-07T23:40:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:51.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:51.510842+0000) 2022-09-07T23:40:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:51 smithi055 ceph-mon[97033]: pgmap v3902: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:40:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:51 smithi059 ceph-mon[97450]: pgmap v3902: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:40:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:52.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:52.605659+0000) 2022-09-07T23:40:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:52.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:52.511069+0000) 2022-09-07T23:40:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:53.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:53.605784+0000) 2022-09-07T23:40:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:53.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:53.511299+0000) 2022-09-07T23:40:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:53 smithi055 ceph-mon[97033]: pgmap v3903: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:53 smithi059 ceph-mon[97450]: pgmap v3903: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:54.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:54.605981+0000) 2022-09-07T23:40:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:54.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:54.511520+0000) 2022-09-07T23:40:55.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:55 smithi059 ceph-mon[97450]: pgmap v3904: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:55 smithi055 ceph-mon[97033]: pgmap v3904: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:40:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:55.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:55.606192+0000) 2022-09-07T23:40:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:55.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:55.511774+0000) 2022-09-07T23:40:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:56.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:56.606377+0000) 2022-09-07T23:40:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:56.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:56.512020+0000) 2022-09-07T23:40:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:57.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:57.606599+0000) 2022-09-07T23:40:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:57.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:57.512213+0000) 2022-09-07T23:40:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:57 smithi055 ceph-mon[97033]: pgmap v3905: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:40:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:57 smithi059 ceph-mon[97450]: pgmap v3905: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:40:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:58.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:58.512404+0000) 2022-09-07T23:40:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:58.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:58.606806+0000) 2022-09-07T23:40:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:40:59.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:59.512614+0000) 2022-09-07T23:40:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:40:59.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:40:59.607042+0000) 2022-09-07T23:41:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:40:59 smithi055 ceph-mon[97033]: pgmap v3906: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:41:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:40:59 smithi059 ceph-mon[97450]: pgmap v3906: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:41:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:00.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:00.607289+0000) 2022-09-07T23:41:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:00.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:00.512806+0000) 2022-09-07T23:41:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:01.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:01.512968+0000) 2022-09-07T23:41:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:01.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:01.607458+0000) 2022-09-07T23:41:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:01 smithi055 ceph-mon[97033]: pgmap v3907: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:41:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:01 smithi059 ceph-mon[97450]: pgmap v3907: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:41:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:02.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:02.513184+0000) 2022-09-07T23:41:02.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:02.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:02.607690+0000) 2022-09-07T23:41:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:03.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:03.513384+0000) 2022-09-07T23:41:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:03.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:03.607921+0000) 2022-09-07T23:41:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:03 smithi055 ceph-mon[97033]: pgmap v3908: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:41:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:03 smithi059 ceph-mon[97450]: pgmap v3908: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:41:04.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:41:03.906+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:41:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:04.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:04.608181+0000) 2022-09-07T23:41:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:04.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:04.513587+0000) 2022-09-07T23:41:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:41:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:41:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:05 smithi059 ceph-mon[97450]: pgmap v3909: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 22 KiB/s wr, 19 op/s 2022-09-07T23:41:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:41:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:41:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:05 smithi055 ceph-mon[97033]: pgmap v3909: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 22 KiB/s wr, 19 op/s 2022-09-07T23:41:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:05.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:05.608403+0000) 2022-09-07T23:41:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:05.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:05.513798+0000) 2022-09-07T23:41:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:06.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:06.608683+0000) 2022-09-07T23:41:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:06.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:06.514030+0000) 2022-09-07T23:41:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:07.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:07.514178+0000) 2022-09-07T23:41:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:07.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:07.608948+0000) 2022-09-07T23:41:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:07 smithi055 ceph-mon[97033]: pgmap v3910: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:41:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:07 smithi059 ceph-mon[97450]: pgmap v3910: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:41:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:08.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:08.609105+0000) 2022-09-07T23:41:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:08.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:08.514418+0000) 2022-09-07T23:41:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:09.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:09.609319+0000) 2022-09-07T23:41:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:09.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:09.514545+0000) 2022-09-07T23:41:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:09 smithi055 ceph-mon[97033]: pgmap v3911: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:41:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:09 smithi055 ceph-mon[97033]: osdmap e333: 12 total, 11 up, 11 in 2022-09-07T23:41:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:09 smithi059 ceph-mon[97450]: pgmap v3911: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:41:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:09 smithi059 ceph-mon[97450]: osdmap e333: 12 total, 11 up, 11 in 2022-09-07T23:41:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:10.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:10.609547+0000) 2022-09-07T23:41:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:10.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:10.514748+0000) 2022-09-07T23:41:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:11.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:11.609780+0000) 2022-09-07T23:41:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:11.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:11.515008+0000) 2022-09-07T23:41:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:11 smithi055 ceph-mon[97033]: pgmap v3913: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:41:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:11 smithi059 ceph-mon[97450]: pgmap v3913: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:41:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:12.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:12.610006+0000) 2022-09-07T23:41:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:12.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:12.515195+0000) 2022-09-07T23:41:13.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:13.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:13.515422+0000) 2022-09-07T23:41:13.911 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:13.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:13.610203+0000) 2022-09-07T23:41:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:13 smithi055 ceph-mon[97033]: pgmap v3914: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:41:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:13 smithi059 ceph-mon[97450]: pgmap v3914: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:41:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:14.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:14.515664+0000) 2022-09-07T23:41:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:14.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:14.610445+0000) 2022-09-07T23:41:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:15 smithi059 ceph-mon[97450]: osdmap e334: 12 total, 11 up, 11 in 2022-09-07T23:41:15.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:15 smithi059 ceph-mon[97450]: pgmap v3916: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-07T23:41:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:15 smithi055 ceph-mon[97033]: osdmap e334: 12 total, 11 up, 11 in 2022-09-07T23:41:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:15 smithi055 ceph-mon[97033]: pgmap v3916: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-07T23:41:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:15.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:15.610616+0000) 2022-09-07T23:41:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:15.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:15.515805+0000) 2022-09-07T23:41:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:16.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:16.516033+0000) 2022-09-07T23:41:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:16.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:16.610839+0000) 2022-09-07T23:41:17.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:17.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:17.611044+0000) 2022-09-07T23:41:17.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:17.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:17.516234+0000) 2022-09-07T23:41:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:17 smithi055 ceph-mon[97033]: pgmap v3917: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:41:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:17 smithi059 ceph-mon[97450]: pgmap v3917: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:41:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:18.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:18.516459+0000) 2022-09-07T23:41:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:18.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:18.611229+0000) 2022-09-07T23:41:19.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:41:18.915+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:41:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:19.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:19.611448+0000) 2022-09-07T23:41:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:19.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:19.516718+0000) 2022-09-07T23:41:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:19 smithi055 ceph-mon[97033]: pgmap v3918: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:41:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:19 smithi059 ceph-mon[97450]: pgmap v3918: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:41:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:20.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:20.516935+0000) 2022-09-07T23:41:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:20.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:20.611721+0000) 2022-09-07T23:41:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:21.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:21.611872+0000) 2022-09-07T23:41:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:21.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:21.517076+0000) 2022-09-07T23:41:22.192 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:21 smithi059 ceph-mon[97450]: pgmap v3919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:41:22.193 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:41:22.193 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:41:22.194 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:41:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:21 smithi055 ceph-mon[97033]: pgmap v3919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:41:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:41:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:41:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:41:22.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:22.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:22.612073+0000) 2022-09-07T23:41:22.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:22.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:22.517303+0000) 2022-09-07T23:41:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:23.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:23.612261+0000) 2022-09-07T23:41:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:23.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:23.517578+0000) 2022-09-07T23:41:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:23 smithi055 ceph-mon[97033]: pgmap v3920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:41:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:23 smithi059 ceph-mon[97450]: pgmap v3920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:41:24.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:24.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:24.612456+0000) 2022-09-07T23:41:24.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:24.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:24.517797+0000) 2022-09-07T23:41:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:25 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-07T23:41:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:41:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:41:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:25 smithi055 ceph-mon[97033]: pgmap v3921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 97 B/s wr, 7 op/s 2022-09-07T23:41:25.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:25 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-07T23:41:25.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:41:25.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:41:25.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:25 smithi055 ceph-mon[97033]: pgmap v3922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:41:25.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:41:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:25 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-07T23:41:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:41:25.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:41:25.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:25 smithi059 ceph-mon[97450]: pgmap v3921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 97 B/s wr, 7 op/s 2022-09-07T23:41:25.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:25 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-07T23:41:25.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:41:25.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:41:25.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:25 smithi059 ceph-mon[97450]: pgmap v3922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:41:25.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:41:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:25.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:25.612740+0000) 2022-09-07T23:41:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:25.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:25.518020+0000) 2022-09-07T23:41:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:26 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-07T23:41:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:26 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-07T23:41:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:26.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:26.612964+0000) 2022-09-07T23:41:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:26.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:26.518195+0000) 2022-09-07T23:41:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:27 smithi055 ceph-mon[97033]: pgmap v3923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:41:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:27 smithi059 ceph-mon[97450]: pgmap v3923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:41:27.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:27.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:27.613191+0000) 2022-09-07T23:41:27.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:27.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:27.518452+0000) 2022-09-07T23:41:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:28.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:28.613430+0000) 2022-09-07T23:41:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:28.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:28.518740+0000) 2022-09-07T23:41:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:29.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:29.613741+0000) 2022-09-07T23:41:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:29.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:29.518959+0000) 2022-09-07T23:41:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:29 smithi055 ceph-mon[97033]: pgmap v3924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:41:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:29 smithi059 ceph-mon[97450]: pgmap v3924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:41:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:30.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:30.613910+0000) 2022-09-07T23:41:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:30.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:30.519225+0000) 2022-09-07T23:41:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:31.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:31.614127+0000) 2022-09-07T23:41:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:31.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:31.519448+0000) 2022-09-07T23:41:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:31 smithi055 ceph-mon[97033]: pgmap v3925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:41:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:31 smithi059 ceph-mon[97450]: pgmap v3925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:41:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:32.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:32.614358+0000) 2022-09-07T23:41:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:32.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:32.519683+0000) 2022-09-07T23:41:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:33.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:33.614501+0000) 2022-09-07T23:41:33.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:33.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:33.519956+0000) 2022-09-07T23:41:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:33 smithi055 ceph-mon[97033]: pgmap v3926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:41:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:33 smithi059 ceph-mon[97450]: pgmap v3926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:41:34.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:41:33.921+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:41:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:34.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:34.614744+0000) 2022-09-07T23:41:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:34.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:34.520142+0000) 2022-09-07T23:41:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:35 smithi059 ceph-mon[97450]: pgmap v3927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:41:35.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:35 smithi055 ceph-mon[97033]: pgmap v3927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:41:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:35.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:35.520353+0000) 2022-09-07T23:41:35.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:35.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:35.614924+0000) 2022-09-07T23:41:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:36.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:36.615139+0000) 2022-09-07T23:41:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:36.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:36.520530+0000) 2022-09-07T23:41:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:37.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:37.615281+0000) 2022-09-07T23:41:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:37.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:37.520742+0000) 2022-09-07T23:41:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:37 smithi055 ceph-mon[97033]: pgmap v3928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:41:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:37 smithi059 ceph-mon[97450]: pgmap v3928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:41:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:38.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:38.615495+0000) 2022-09-07T23:41:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:38.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:38.520955+0000) 2022-09-07T23:41:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:39.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:39.615727+0000) 2022-09-07T23:41:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:39.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:39.521183+0000) 2022-09-07T23:41:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:39 smithi055 ceph-mon[97033]: pgmap v3929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:41:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:39 smithi059 ceph-mon[97450]: pgmap v3929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:41:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:40.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:40.615880+0000) 2022-09-07T23:41:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:40.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:40.521403+0000) 2022-09-07T23:41:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:41.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:41.616080+0000) 2022-09-07T23:41:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:41.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:41.521610+0000) 2022-09-07T23:41:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:41 smithi055 ceph-mon[97033]: pgmap v3930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:41:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:41 smithi059 ceph-mon[97450]: pgmap v3930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:41:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:42.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:42.616254+0000) 2022-09-07T23:41:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:42.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:42.521834+0000) 2022-09-07T23:41:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:43.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:43.616463+0000) 2022-09-07T23:41:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:43.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:43.522040+0000) 2022-09-07T23:41:44.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:43 smithi055 ceph-mon[97033]: pgmap v3931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:41:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:43 smithi059 ceph-mon[97450]: pgmap v3931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:41:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:44.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:44.616620+0000) 2022-09-07T23:41:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:44.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:44.522260+0000) 2022-09-07T23:41:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:45 smithi055 ceph-mon[97033]: pgmap v3932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:41:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:45 smithi059 ceph-mon[97450]: pgmap v3932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:41:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:45.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:45.616853+0000) 2022-09-07T23:41:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:45.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:45.522503+0000) 2022-09-07T23:41:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:46.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:46.616996+0000) 2022-09-07T23:41:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:46.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:46.522752+0000) 2022-09-07T23:41:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:47.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:47.617192+0000) 2022-09-07T23:41:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:47.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:47.522971+0000) 2022-09-07T23:41:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:47 smithi055 ceph-mon[97033]: pgmap v3933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:41:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:47 smithi059 ceph-mon[97450]: pgmap v3933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:41:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:48.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:48.523242+0000) 2022-09-07T23:41:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:48.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:48.617343+0000) 2022-09-07T23:41:49.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:41:48.932+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:41:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:49.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:49.617477+0000) 2022-09-07T23:41:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:49.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:49.523476+0000) 2022-09-07T23:41:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:49 smithi055 ceph-mon[97033]: pgmap v3934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:41:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:49 smithi059 ceph-mon[97450]: pgmap v3934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:41:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:50.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:50.617631+0000) 2022-09-07T23:41:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:50.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:50.523777+0000) 2022-09-07T23:41:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:51.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:51.617853+0000) 2022-09-07T23:41:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:51.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:51.523991+0000) 2022-09-07T23:41:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:51 smithi055 ceph-mon[97033]: pgmap v3935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:41:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:51 smithi059 ceph-mon[97450]: pgmap v3935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:41:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:52.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:52.524211+0000) 2022-09-07T23:41:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:52.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:52.618059+0000) 2022-09-07T23:41:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:53.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:53.524442+0000) 2022-09-07T23:41:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:53.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:53.618251+0000) 2022-09-07T23:41:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:53 smithi055 ceph-mon[97033]: pgmap v3936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:41:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:53 smithi059 ceph-mon[97450]: pgmap v3936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:41:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:54.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:54.618427+0000) 2022-09-07T23:41:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:54.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:54.524711+0000) 2022-09-07T23:41:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:55 smithi055 ceph-mon[97033]: pgmap v3937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:41:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:55 smithi059 ceph-mon[97450]: pgmap v3937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:41:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:55.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:55.618692+0000) 2022-09-07T23:41:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:55.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:55.524877+0000) 2022-09-07T23:41:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:56.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:56.618899+0000) 2022-09-07T23:41:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:56.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:56.525084+0000) 2022-09-07T23:41:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:57.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:57.619127+0000) 2022-09-07T23:41:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:57.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:57.525315+0000) 2022-09-07T23:41:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:57 smithi055 ceph-mon[97033]: pgmap v3938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:41:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:57 smithi059 ceph-mon[97450]: pgmap v3938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:41:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:58.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:58.619395+0000) 2022-09-07T23:41:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:58.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:58.525616+0000) 2022-09-07T23:41:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:41:59.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:59.619655+0000) 2022-09-07T23:41:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:41:59.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:41:59.525886+0000) 2022-09-07T23:42:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:41:59 smithi055 ceph-mon[97033]: pgmap v3939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:41:59 smithi059 ceph-mon[97450]: pgmap v3939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:00.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:00.619885+0000) 2022-09-07T23:42:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:00.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:00.526155+0000) 2022-09-07T23:42:01.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:01.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:01.620125+0000) 2022-09-07T23:42:01.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:01.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:01.526410+0000) 2022-09-07T23:42:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:01 smithi055 ceph-mon[97033]: pgmap v3940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:42:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:01 smithi059 ceph-mon[97450]: pgmap v3940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:42:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:02.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:02.620334+0000) 2022-09-07T23:42:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:02.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:02.526709+0000) 2022-09-07T23:42:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:03.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:03.620591+0000) 2022-09-07T23:42:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:03.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:03.526939+0000) 2022-09-07T23:42:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:03 smithi055 ceph-mon[97033]: pgmap v3941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:03 smithi059 ceph-mon[97450]: pgmap v3941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:04.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:42:03.947+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:42:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:04.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:04.620848+0000) 2022-09-07T23:42:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:04.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:04.527116+0000) 2022-09-07T23:42:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:42:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:42:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:05 smithi055 ceph-mon[97033]: pgmap v3942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 22 KiB/s wr, 19 op/s 2022-09-07T23:42:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:42:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:42:05.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:05 smithi059 ceph-mon[97450]: pgmap v3942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 22 KiB/s wr, 19 op/s 2022-09-07T23:42:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:05.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:05.621026+0000) 2022-09-07T23:42:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:05.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:05.527326+0000) 2022-09-07T23:42:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:06.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:06.621273+0000) 2022-09-07T23:42:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:06.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:06.527536+0000) 2022-09-07T23:42:07.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:07.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:07.621517+0000) 2022-09-07T23:42:07.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:07.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:07.527778+0000) 2022-09-07T23:42:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:07 smithi055 ceph-mon[97033]: pgmap v3943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T23:42:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:07 smithi059 ceph-mon[97450]: pgmap v3943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T23:42:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:08.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:08.528026+0000) 2022-09-07T23:42:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:08.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:08.621761+0000) 2022-09-07T23:42:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:09.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:09.621981+0000) 2022-09-07T23:42:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:09.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:09.528280+0000) 2022-09-07T23:42:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:09 smithi055 ceph-mon[97033]: pgmap v3944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:42:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:09 smithi055 ceph-mon[97033]: osdmap e335: 12 total, 11 up, 11 in 2022-09-07T23:42:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:09 smithi059 ceph-mon[97450]: pgmap v3944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:42:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:09 smithi059 ceph-mon[97450]: osdmap e335: 12 total, 11 up, 11 in 2022-09-07T23:42:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:10.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:10.622137+0000) 2022-09-07T23:42:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:10.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:10.528544+0000) 2022-09-07T23:42:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:11.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:11.622338+0000) 2022-09-07T23:42:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:11.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:11.528779+0000) 2022-09-07T23:42:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:11 smithi055 ceph-mon[97033]: pgmap v3946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:42:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:11 smithi059 ceph-mon[97450]: pgmap v3946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:42:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:12.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:12.529030+0000) 2022-09-07T23:42:12.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:12.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:12.622562+0000) 2022-09-07T23:42:13.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:13.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:13.622802+0000) 2022-09-07T23:42:13.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:13.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:13.529295+0000) 2022-09-07T23:42:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:13 smithi055 ceph-mon[97033]: pgmap v3947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:42:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:13 smithi059 ceph-mon[97450]: pgmap v3947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:42:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:14.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:14.529576+0000) 2022-09-07T23:42:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:14.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:14.622955+0000) 2022-09-07T23:42:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:15 smithi055 ceph-mon[97033]: osdmap e336: 12 total, 11 up, 11 in 2022-09-07T23:42:15.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:15 smithi055 ceph-mon[97033]: pgmap v3949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-07T23:42:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:15 smithi059 ceph-mon[97450]: osdmap e336: 12 total, 11 up, 11 in 2022-09-07T23:42:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:15 smithi059 ceph-mon[97450]: pgmap v3949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-07T23:42:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:15.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:15.623158+0000) 2022-09-07T23:42:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:15.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:15.529836+0000) 2022-09-07T23:42:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:16.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:16.623418+0000) 2022-09-07T23:42:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:16.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:16.530042+0000) 2022-09-07T23:42:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:17.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:17.623666+0000) 2022-09-07T23:42:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:17.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:17.530274+0000) 2022-09-07T23:42:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:17 smithi055 ceph-mon[97033]: pgmap v3950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:42:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:17 smithi059 ceph-mon[97450]: pgmap v3950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:42:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:18.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:18.623907+0000) 2022-09-07T23:42:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:18.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:18.530512+0000) 2022-09-07T23:42:19.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:42:18.955+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:42:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:19.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:19.624151+0000) 2022-09-07T23:42:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:19.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:19.530732+0000) 2022-09-07T23:42:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:19 smithi055 ceph-mon[97033]: pgmap v3951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T23:42:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:19 smithi059 ceph-mon[97450]: pgmap v3951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T23:42:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:20.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:20.624390+0000) 2022-09-07T23:42:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:20.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:20.530978+0000) 2022-09-07T23:42:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:21.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:21.624560+0000) 2022-09-07T23:42:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:21.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:21.531232+0000) 2022-09-07T23:42:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:21 smithi055 ceph-mon[97033]: pgmap v3952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:42:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:21 smithi059 ceph-mon[97450]: pgmap v3952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:42:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:22.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:22.624718+0000) 2022-09-07T23:42:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:22.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:22.531507+0000) 2022-09-07T23:42:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:23.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:23.624859+0000) 2022-09-07T23:42:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:23.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:23.531753+0000) 2022-09-07T23:42:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:23 smithi055 ceph-mon[97033]: pgmap v3953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:42:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:23 smithi059 ceph-mon[97450]: pgmap v3953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:42:24.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:24.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:24.625049+0000) 2022-09-07T23:42:24.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:24.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:24.531974+0000) 2022-09-07T23:42:25.565 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:25 smithi059 ceph-mon[97450]: pgmap v3954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 96 B/s wr, 7 op/s 2022-09-07T23:42:25.566 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:42:25.568 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:42:25.569 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:42:25.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:25 smithi055 ceph-mon[97033]: pgmap v3954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 96 B/s wr, 7 op/s 2022-09-07T23:42:25.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:42:25.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:42:25.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:42:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:25.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:25.625221+0000) 2022-09-07T23:42:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:25.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:25.532176+0000) 2022-09-07T23:42:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:42:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:26.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:26.532390+0000) 2022-09-07T23:42:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:26.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:26.625436+0000) 2022-09-07T23:42:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:42:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:27 smithi059 ceph-mon[97450]: pgmap v3955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:42:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:27.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:27.625681+0000) 2022-09-07T23:42:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:27.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:27.532601+0000) 2022-09-07T23:42:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:27 smithi055 ceph-mon[97033]: pgmap v3955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:42:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:28.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:28.532750+0000) 2022-09-07T23:42:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:28.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:28.625912+0000) 2022-09-07T23:42:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:29.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:29.532986+0000) 2022-09-07T23:42:29.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:29.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:29.626108+0000) 2022-09-07T23:42:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:29 smithi055 ceph-mon[97033]: pgmap v3956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:29 smithi059 ceph-mon[97450]: pgmap v3956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:30.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:30.533260+0000) 2022-09-07T23:42:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:30.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:30.626368+0000) 2022-09-07T23:42:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:31.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:31.533484+0000) 2022-09-07T23:42:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:31.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:31.626532+0000) 2022-09-07T23:42:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:31 smithi055 ceph-mon[97033]: pgmap v3957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:42:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:31 smithi059 ceph-mon[97450]: pgmap v3957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:42:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:32.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:32.626759+0000) 2022-09-07T23:42:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:32.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:32.533728+0000) 2022-09-07T23:42:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:33.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:33.626960+0000) 2022-09-07T23:42:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:33.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:33.533979+0000) 2022-09-07T23:42:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:33 smithi055 ceph-mon[97033]: pgmap v3958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:33 smithi059 ceph-mon[97450]: pgmap v3958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:34.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:42:33.962+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:42:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:34.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:34.627135+0000) 2022-09-07T23:42:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:34.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:34.534169+0000) 2022-09-07T23:42:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:35 smithi055 ceph-mon[97033]: pgmap v3959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:42:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:35 smithi059 ceph-mon[97450]: pgmap v3959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:42:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:35.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:35.627331+0000) 2022-09-07T23:42:35.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:35.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:35.534384+0000) 2022-09-07T23:42:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:36.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:36.627532+0000) 2022-09-07T23:42:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:36.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:36.534585+0000) 2022-09-07T23:42:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:37.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:37.627748+0000) 2022-09-07T23:42:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:37.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:37.534741+0000) 2022-09-07T23:42:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:37 smithi055 ceph-mon[97033]: pgmap v3960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:42:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:37 smithi059 ceph-mon[97450]: pgmap v3960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:42:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:38.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:38.627929+0000) 2022-09-07T23:42:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:38.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:38.535015+0000) 2022-09-07T23:42:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:39.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:39.628173+0000) 2022-09-07T23:42:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:39.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:39.535222+0000) 2022-09-07T23:42:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:39 smithi055 ceph-mon[97033]: pgmap v3961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:39 smithi059 ceph-mon[97450]: pgmap v3961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:40.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:40.628333+0000) 2022-09-07T23:42:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:40.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:40.535445+0000) 2022-09-07T23:42:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:41.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:41.628494+0000) 2022-09-07T23:42:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:41.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:41.535723+0000) 2022-09-07T23:42:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:41 smithi055 ceph-mon[97033]: pgmap v3962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:42:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:41 smithi059 ceph-mon[97450]: pgmap v3962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:42:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:42.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:42.628748+0000) 2022-09-07T23:42:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:42.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:42.535996+0000) 2022-09-07T23:42:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:43.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:43.628877+0000) 2022-09-07T23:42:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:43.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:43.536198+0000) 2022-09-07T23:42:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:43 smithi055 ceph-mon[97033]: pgmap v3963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:44.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:43 smithi059 ceph-mon[97450]: pgmap v3963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:44.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:44.629091+0000) 2022-09-07T23:42:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:44.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:44.536379+0000) 2022-09-07T23:42:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:45 smithi055 ceph-mon[97033]: pgmap v3964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:42:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:45 smithi059 ceph-mon[97450]: pgmap v3964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:42:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:45.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:45.629346+0000) 2022-09-07T23:42:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:45.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:45.536558+0000) 2022-09-07T23:42:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:46.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:46.536773+0000) 2022-09-07T23:42:46.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:46.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:46.629491+0000) 2022-09-07T23:42:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:47.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:47.629735+0000) 2022-09-07T23:42:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:47.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:47.537013+0000) 2022-09-07T23:42:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:47 smithi055 ceph-mon[97033]: pgmap v3965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:42:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:47 smithi059 ceph-mon[97450]: pgmap v3965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:42:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:48.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:48.537237+0000) 2022-09-07T23:42:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:48.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:48.629889+0000) 2022-09-07T23:42:49.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:42:48.977+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:42:49.909 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:49.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:49.630008+0000) 2022-09-07T23:42:49.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:49.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:49.537447+0000) 2022-09-07T23:42:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:49 smithi055 ceph-mon[97033]: pgmap v3966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:49 smithi059 ceph-mon[97450]: pgmap v3966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:50.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:50.630177+0000) 2022-09-07T23:42:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:50.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:50.537686+0000) 2022-09-07T23:42:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:51.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:51.630403+0000) 2022-09-07T23:42:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:51.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:51.537817+0000) 2022-09-07T23:42:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:51 smithi055 ceph-mon[97033]: pgmap v3967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:42:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:51 smithi059 ceph-mon[97450]: pgmap v3967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:42:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:52.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:52.630674+0000) 2022-09-07T23:42:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:52.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:52.537964+0000) 2022-09-07T23:42:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:53.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:53.630891+0000) 2022-09-07T23:42:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:53.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:53.538153+0000) 2022-09-07T23:42:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:53 smithi055 ceph-mon[97033]: pgmap v3968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:53 smithi059 ceph-mon[97450]: pgmap v3968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:42:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:54.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:54.631103+0000) 2022-09-07T23:42:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:54.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:54.538310+0000) 2022-09-07T23:42:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:55 smithi055 ceph-mon[97033]: pgmap v3969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:42:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:55 smithi059 ceph-mon[97450]: pgmap v3969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:42:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:55.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:55.631284+0000) 2022-09-07T23:42:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:55.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:55.538504+0000) 2022-09-07T23:42:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:56.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:56.631414+0000) 2022-09-07T23:42:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:56.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:56.538754+0000) 2022-09-07T23:42:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:57.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:57.631593+0000) 2022-09-07T23:42:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:57.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:57.538943+0000) 2022-09-07T23:42:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:57 smithi055 ceph-mon[97033]: pgmap v3970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:42:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:57 smithi059 ceph-mon[97450]: pgmap v3970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:42:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:58.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:58.631832+0000) 2022-09-07T23:42:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:58.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:58.539174+0000) 2022-09-07T23:42:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:42:59.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:59.632078+0000) 2022-09-07T23:42:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:42:59.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:42:59.539399+0000) 2022-09-07T23:43:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:42:59 smithi055 ceph-mon[97033]: pgmap v3971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:42:59 smithi059 ceph-mon[97450]: pgmap v3971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:00.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:00.539596+0000) 2022-09-07T23:43:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:00.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:00.632267+0000) 2022-09-07T23:43:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:01.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:01.539711+0000) 2022-09-07T23:43:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:01.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:01.632478+0000) 2022-09-07T23:43:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:01 smithi055 ceph-mon[97033]: pgmap v3972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:43:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:01 smithi059 ceph-mon[97450]: pgmap v3972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:43:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:02.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:02.539871+0000) 2022-09-07T23:43:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:02.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:02.632729+0000) 2022-09-07T23:43:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:03.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:03.632943+0000) 2022-09-07T23:43:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:03.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:03.540116+0000) 2022-09-07T23:43:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:03 smithi055 ceph-mon[97033]: pgmap v3973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:03 smithi059 ceph-mon[97450]: pgmap v3973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:04.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:43:03.993+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:43:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:04.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:04.540309+0000) 2022-09-07T23:43:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:04.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:04.633101+0000) 2022-09-07T23:43:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:43:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:43:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:05 smithi055 ceph-mon[97033]: pgmap v3974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 22 KiB/s wr, 19 op/s 2022-09-07T23:43:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:43:05.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:43:05.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:05 smithi059 ceph-mon[97450]: pgmap v3974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 22 KiB/s wr, 19 op/s 2022-09-07T23:43:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:05.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:05.633303+0000) 2022-09-07T23:43:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:05.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:05.540568+0000) 2022-09-07T23:43:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:06.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:06.633621+0000) 2022-09-07T23:43:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:06.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:06.540782+0000) 2022-09-07T23:43:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:07.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:07.633962+0000) 2022-09-07T23:43:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:07.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:07.540970+0000) 2022-09-07T23:43:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:07 smithi055 ceph-mon[97033]: pgmap v3975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T23:43:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:07 smithi059 ceph-mon[97450]: pgmap v3975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-07T23:43:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:08.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:08.634171+0000) 2022-09-07T23:43:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:08.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:08.541147+0000) 2022-09-07T23:43:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:09.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:09.634437+0000) 2022-09-07T23:43:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:09.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:09.541402+0000) 2022-09-07T23:43:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:09 smithi055 ceph-mon[97033]: pgmap v3976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:43:10.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:09 smithi055 ceph-mon[97033]: osdmap e337: 12 total, 11 up, 11 in 2022-09-07T23:43:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:09 smithi059 ceph-mon[97450]: pgmap v3976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:43:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:09 smithi059 ceph-mon[97450]: osdmap e337: 12 total, 11 up, 11 in 2022-09-07T23:43:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:10.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:10.634674+0000) 2022-09-07T23:43:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:10.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:10.541566+0000) 2022-09-07T23:43:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:11.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:11.634845+0000) 2022-09-07T23:43:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:11.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:11.541793+0000) 2022-09-07T23:43:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:11 smithi055 ceph-mon[97033]: pgmap v3978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:43:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:11 smithi059 ceph-mon[97450]: pgmap v3978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:43:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:12.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:12.635081+0000) 2022-09-07T23:43:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:12.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:12.542046+0000) 2022-09-07T23:43:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:13.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:13.542228+0000) 2022-09-07T23:43:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:13.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:13.635312+0000) 2022-09-07T23:43:14.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:13 smithi055 ceph-mon[97033]: pgmap v3979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:43:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:13 smithi059 ceph-mon[97450]: pgmap v3979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:43:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:14.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:14.542453+0000) 2022-09-07T23:43:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:14.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:14.635524+0000) 2022-09-07T23:43:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:15 smithi055 ceph-mon[97033]: osdmap e338: 12 total, 11 up, 11 in 2022-09-07T23:43:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:15 smithi055 ceph-mon[97033]: pgmap v3981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-07T23:43:15.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:15 smithi059 ceph-mon[97450]: osdmap e338: 12 total, 11 up, 11 in 2022-09-07T23:43:15.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:15 smithi059 ceph-mon[97450]: pgmap v3981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-07T23:43:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:15.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:15.635776+0000) 2022-09-07T23:43:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:15.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:15.542720+0000) 2022-09-07T23:43:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:16.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:16.636006+0000) 2022-09-07T23:43:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:16.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:16.542903+0000) 2022-09-07T23:43:17.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:17.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:17.636222+0000) 2022-09-07T23:43:17.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:17.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:17.543120+0000) 2022-09-07T23:43:18.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:17 smithi055 ceph-mon[97033]: pgmap v3982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:43:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:17 smithi059 ceph-mon[97450]: pgmap v3982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:43:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:18.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:18.636389+0000) 2022-09-07T23:43:18.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:18.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:18.543352+0000) 2022-09-07T23:43:19.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:43:19.009+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:43:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:19.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:19.636615+0000) 2022-09-07T23:43:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:19.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:19.543581+0000) 2022-09-07T23:43:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:19 smithi055 ceph-mon[97033]: pgmap v3983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:43:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:19 smithi059 ceph-mon[97450]: pgmap v3983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:43:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:20.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:20.636864+0000) 2022-09-07T23:43:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:20.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:20.543715+0000) 2022-09-07T23:43:21.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:21.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:21.637055+0000) 2022-09-07T23:43:21.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:21.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:21.543880+0000) 2022-09-07T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:21 smithi055 ceph-mon[97033]: pgmap v3984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:43:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:21 smithi059 ceph-mon[97450]: pgmap v3984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:43:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:22.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:22.637292+0000) 2022-09-07T23:43:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:22.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:22.544125+0000) 2022-09-07T23:43:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:23.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:23.544361+0000) 2022-09-07T23:43:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:23.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:23.637497+0000) 2022-09-07T23:43:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:23 smithi055 ceph-mon[97033]: pgmap v3985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:43:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:23 smithi059 ceph-mon[97450]: pgmap v3985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:43:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:24.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:24.637735+0000) 2022-09-07T23:43:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:24.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:24.544522+0000) 2022-09-07T23:43:25.643 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:25 smithi055 ceph-mon[97033]: pgmap v3986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 96 B/s wr, 7 op/s 2022-09-07T23:43:25.644 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:25 smithi059 ceph-mon[97450]: pgmap v3986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 96 B/s wr, 7 op/s 2022-09-07T23:43:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:25.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:25.637886+0000) 2022-09-07T23:43:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:25.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:25.544694+0000) 2022-09-07T23:43:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:43:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:43:26.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:43:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:43:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:43:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:43:26.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:26.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:26.638055+0000) 2022-09-07T23:43:26.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:26.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:26.544874+0000) 2022-09-07T23:43:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:27.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:27.638300+0000) 2022-09-07T23:43:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:27.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:27.545061+0000) 2022-09-07T23:43:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:43:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:43:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:43:27.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:43:27.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:43:27.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:27 smithi055 ceph-mon[97033]: pgmap v3987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:43:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:43:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:43:28.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:43:28.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:43:28.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:43:28.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:27 smithi059 ceph-mon[97450]: pgmap v3987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:43:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:28.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:28.638516+0000) 2022-09-07T23:43:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:28.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:28.545229+0000) 2022-09-07T23:43:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:29.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:29.638783+0000) 2022-09-07T23:43:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:29.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:29.545421+0000) 2022-09-07T23:43:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:29 smithi055 ceph-mon[97033]: pgmap v3988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:29 smithi059 ceph-mon[97450]: pgmap v3988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:30.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:30.545662+0000) 2022-09-07T23:43:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:30.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:30.638931+0000) 2022-09-07T23:43:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:31.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:31.639139+0000) 2022-09-07T23:43:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:31.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:31.545901+0000) 2022-09-07T23:43:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:31 smithi055 ceph-mon[97033]: pgmap v3989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:43:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:31 smithi059 ceph-mon[97450]: pgmap v3989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:43:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:32.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:32.639341+0000) 2022-09-07T23:43:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:32.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:32.546128+0000) 2022-09-07T23:43:33.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:33.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:33.639548+0000) 2022-09-07T23:43:33.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:33.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:33.546351+0000) 2022-09-07T23:43:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:33 smithi055 ceph-mon[97033]: pgmap v3990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:33 smithi059 ceph-mon[97450]: pgmap v3990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:34.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:43:34.023+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:43:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:34.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:34.639724+0000) 2022-09-07T23:43:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:34.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:34.546594+0000) 2022-09-07T23:43:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:35 smithi055 ceph-mon[97033]: pgmap v3991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:43:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:35 smithi059 ceph-mon[97450]: pgmap v3991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:43:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:35.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:35.546891+0000) 2022-09-07T23:43:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:35.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:35.639882+0000) 2022-09-07T23:43:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:36.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:36.640045+0000) 2022-09-07T23:43:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:36.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:36.547133+0000) 2022-09-07T23:43:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:37.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:37.640267+0000) 2022-09-07T23:43:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:37.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:37.547338+0000) 2022-09-07T23:43:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:37 smithi055 ceph-mon[97033]: pgmap v3992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:43:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:37 smithi059 ceph-mon[97450]: pgmap v3992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:43:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:38.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:38.640504+0000) 2022-09-07T23:43:38.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:38.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:38.547544+0000) 2022-09-07T23:43:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:39.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:39.640740+0000) 2022-09-07T23:43:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:39.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:39.547766+0000) 2022-09-07T23:43:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:39 smithi055 ceph-mon[97033]: pgmap v3993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:39 smithi059 ceph-mon[97450]: pgmap v3993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:40.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:40.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:40.641019+0000) 2022-09-07T23:43:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:40.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:40.548010+0000) 2022-09-07T23:43:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:41.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:41.641301+0000) 2022-09-07T23:43:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:41.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:41.548287+0000) 2022-09-07T23:43:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:41 smithi055 ceph-mon[97033]: pgmap v3994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:43:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:41 smithi059 ceph-mon[97450]: pgmap v3994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:43:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:42.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:42.641565+0000) 2022-09-07T23:43:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:42.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:42.548504+0000) 2022-09-07T23:43:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:43.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:43.641814+0000) 2022-09-07T23:43:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:43.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:43.548745+0000) 2022-09-07T23:43:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:43 smithi055 ceph-mon[97033]: pgmap v3995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:43 smithi059 ceph-mon[97450]: pgmap v3995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:44.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:44.642054+0000) 2022-09-07T23:43:44.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:44.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:44.548919+0000) 2022-09-07T23:43:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:45 smithi055 ceph-mon[97033]: pgmap v3996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:43:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:45 smithi059 ceph-mon[97450]: pgmap v3996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:43:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:45.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:45.642252+0000) 2022-09-07T23:43:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:45.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:45.549170+0000) 2022-09-07T23:43:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:46.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:46.549390+0000) 2022-09-07T23:43:46.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:46.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:46.642482+0000) 2022-09-07T23:43:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:47.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:47.549622+0000) 2022-09-07T23:43:47.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:47.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:47.642627+0000) 2022-09-07T23:43:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:47 smithi055 ceph-mon[97033]: pgmap v3997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:43:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:47 smithi059 ceph-mon[97450]: pgmap v3997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:43:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:48.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:48.642834+0000) 2022-09-07T23:43:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:48.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:48.549864+0000) 2022-09-07T23:43:49.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:43:49.038+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:43:49.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:49.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:49.643071+0000) 2022-09-07T23:43:49.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:49.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:49.550008+0000) 2022-09-07T23:43:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:49 smithi055 ceph-mon[97033]: pgmap v3998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:49 smithi059 ceph-mon[97450]: pgmap v3998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:50.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:50.643259+0000) 2022-09-07T23:43:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:50.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:50.550235+0000) 2022-09-07T23:43:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:51.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:51.643505+0000) 2022-09-07T23:43:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:51.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:51.550485+0000) 2022-09-07T23:43:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:51 smithi055 ceph-mon[97033]: pgmap v3999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:43:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:51 smithi059 ceph-mon[97450]: pgmap v3999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:43:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:52.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:52.643734+0000) 2022-09-07T23:43:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:52.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:52.550728+0000) 2022-09-07T23:43:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:53.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:53.643958+0000) 2022-09-07T23:43:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:53.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:53.550932+0000) 2022-09-07T23:43:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:53 smithi055 ceph-mon[97033]: pgmap v4000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:54.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:53 smithi059 ceph-mon[97450]: pgmap v4000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:54.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:54.644198+0000) 2022-09-07T23:43:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:54.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:54.551070+0000) 2022-09-07T23:43:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:55 smithi055 ceph-mon[97033]: pgmap v4001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:43:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:55 smithi059 ceph-mon[97450]: pgmap v4001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:43:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:55.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:55.644481+0000) 2022-09-07T23:43:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:55.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:55.551214+0000) 2022-09-07T23:43:56.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:56.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:56.551450+0000) 2022-09-07T23:43:56.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:56.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:56.644750+0000) 2022-09-07T23:43:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:57.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:57.644917+0000) 2022-09-07T23:43:57.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:57.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:57.551679+0000) 2022-09-07T23:43:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:57 smithi055 ceph-mon[97033]: pgmap v4002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:57 smithi059 ceph-mon[97450]: pgmap v4002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:43:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:58.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:58.645044+0000) 2022-09-07T23:43:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:58.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:58.551925+0000) 2022-09-07T23:43:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:43:59.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:59.645274+0000) 2022-09-07T23:43:59.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:43:59.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:43:59.552105+0000) 2022-09-07T23:44:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:43:59 smithi055 ceph-mon[97033]: pgmap v4003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:43:59 smithi059 ceph-mon[97450]: pgmap v4003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:00.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:00.645431+0000) 2022-09-07T23:44:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:00.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:00.552250+0000) 2022-09-07T23:44:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:01.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:01.645694+0000) 2022-09-07T23:44:01.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:01.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:01.552442+0000) 2022-09-07T23:44:02.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:01 smithi055 ceph-mon[97033]: pgmap v4004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:44:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:01 smithi059 ceph-mon[97450]: pgmap v4004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:44:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:02.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:02.645904+0000) 2022-09-07T23:44:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:02.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:02.552680+0000) 2022-09-07T23:44:03.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:03.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:03.646081+0000) 2022-09-07T23:44:03.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:03.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:03.552856+0000) 2022-09-07T23:44:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:03 smithi055 ceph-mon[97033]: pgmap v4005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:03 smithi059 ceph-mon[97450]: pgmap v4005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:04.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:44:04.054+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:44:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:04.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:04.646263+0000) 2022-09-07T23:44:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:04.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:04.553049+0000) 2022-09-07T23:44:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:44:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:44:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:05 smithi055 ceph-mon[97033]: pgmap v4006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:44:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:44:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:44:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:05 smithi059 ceph-mon[97450]: pgmap v4006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:44:05.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:05.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:05.646463+0000) 2022-09-07T23:44:05.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:05.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:05.553308+0000) 2022-09-07T23:44:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:06.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:06.553500+0000) 2022-09-07T23:44:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:06.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:06.646588+0000) 2022-09-07T23:44:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:07.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:07.646778+0000) 2022-09-07T23:44:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:07.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:07.553725+0000) 2022-09-07T23:44:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:07 smithi055 ceph-mon[97033]: pgmap v4007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:44:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:07 smithi059 ceph-mon[97450]: pgmap v4007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:44:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:08.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:08.646990+0000) 2022-09-07T23:44:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:08.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:08.553954+0000) 2022-09-07T23:44:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:09.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:09.647178+0000) 2022-09-07T23:44:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:09.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:09.554135+0000) 2022-09-07T23:44:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:09 smithi055 ceph-mon[97033]: pgmap v4008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:44:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:09 smithi055 ceph-mon[97033]: osdmap e339: 12 total, 11 up, 11 in 2022-09-07T23:44:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:09 smithi059 ceph-mon[97450]: pgmap v4008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:44:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:09 smithi059 ceph-mon[97450]: osdmap e339: 12 total, 11 up, 11 in 2022-09-07T23:44:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:10.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:10.647382+0000) 2022-09-07T23:44:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:10.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:10.554358+0000) 2022-09-07T23:44:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:11.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:11.647609+0000) 2022-09-07T23:44:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:11.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:11.554594+0000) 2022-09-07T23:44:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:11 smithi055 ceph-mon[97033]: pgmap v4010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:44:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:11 smithi059 ceph-mon[97450]: pgmap v4010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:44:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:12.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:12.647808+0000) 2022-09-07T23:44:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:12.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:12.554797+0000) 2022-09-07T23:44:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:13.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:13.647942+0000) 2022-09-07T23:44:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:13.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:13.555018+0000) 2022-09-07T23:44:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:13 smithi055 ceph-mon[97033]: pgmap v4011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:44:14.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:13 smithi059 ceph-mon[97450]: pgmap v4011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:44:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:14.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:14.648144+0000) 2022-09-07T23:44:14.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:14.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:14.555185+0000) 2022-09-07T23:44:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:15 smithi055 ceph-mon[97033]: osdmap e340: 12 total, 11 up, 11 in 2022-09-07T23:44:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:15 smithi055 ceph-mon[97033]: pgmap v4013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:44:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:15 smithi059 ceph-mon[97450]: osdmap e340: 12 total, 11 up, 11 in 2022-09-07T23:44:15.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:15 smithi059 ceph-mon[97450]: pgmap v4013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:44:15.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:15.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:15.648387+0000) 2022-09-07T23:44:15.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:15.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:15.555424+0000) 2022-09-07T23:44:16.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:16.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:16.648516+0000) 2022-09-07T23:44:16.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:16.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:16.555707+0000) 2022-09-07T23:44:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:17.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:17.555907+0000) 2022-09-07T23:44:17.934 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:17.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:17.648728+0000) 2022-09-07T23:44:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:17 smithi055 ceph-mon[97033]: pgmap v4014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:44:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:17 smithi059 ceph-mon[97450]: pgmap v4014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:44:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:18.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:18.648962+0000) 2022-09-07T23:44:18.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:18.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:18.556146+0000) 2022-09-07T23:44:19.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:44:19.061+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:44:19.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:19.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:19.649216+0000) 2022-09-07T23:44:19.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:19.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:19.556329+0000) 2022-09-07T23:44:20.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:19 smithi055 ceph-mon[97033]: pgmap v4015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:44:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:19 smithi059 ceph-mon[97450]: pgmap v4015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:44:20.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:20.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:20.649411+0000) 2022-09-07T23:44:20.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:20.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:20.556560+0000) 2022-09-07T23:44:21.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:21.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:21.649632+0000) 2022-09-07T23:44:21.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:21.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:21.556738+0000) 2022-09-07T23:44:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:21 smithi055 ceph-mon[97033]: pgmap v4016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:44:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:21 smithi059 ceph-mon[97450]: pgmap v4016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:44:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:22.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:22.649868+0000) 2022-09-07T23:44:22.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:22.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:22.556929+0000) 2022-09-07T23:44:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:23.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:23.650042+0000) 2022-09-07T23:44:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:23.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:23.557083+0000) 2022-09-07T23:44:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:23 smithi055 ceph-mon[97033]: pgmap v4017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:44:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:23 smithi059 ceph-mon[97450]: pgmap v4017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:44:24.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:24.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:24.650188+0000) 2022-09-07T23:44:24.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:24.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:24.557232+0000) 2022-09-07T23:44:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:25 smithi055 ceph-mon[97033]: pgmap v4018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-07T23:44:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:25 smithi059 ceph-mon[97450]: pgmap v4018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-07T23:44:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:25.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:25.650389+0000) 2022-09-07T23:44:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:25.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:25.557471+0000) 2022-09-07T23:44:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:26.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:26.650699+0000) 2022-09-07T23:44:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:26.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:26.557751+0000) 2022-09-07T23:44:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:44:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:44:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:44:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:44:27.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:44:27.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:44:27.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:27.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:27.650920+0000) 2022-09-07T23:44:27.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:27.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:27.557940+0000) 2022-09-07T23:44:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:27 smithi059 ceph-mon[97450]: pgmap v4019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:44:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:27 smithi055 ceph-mon[97033]: pgmap v4019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:28.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:44:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:28.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:28.651136+0000) 2022-09-07T23:44:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:28.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:28.558182+0000) 2022-09-07T23:44:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:29.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:29.651371+0000) 2022-09-07T23:44:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:29.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:29.558373+0000) 2022-09-07T23:44:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:29 smithi055 ceph-mon[97033]: pgmap v4020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:29 smithi059 ceph-mon[97450]: pgmap v4020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:30.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:30.651596+0000) 2022-09-07T23:44:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:30.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:30.558627+0000) 2022-09-07T23:44:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:31.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:31.651784+0000) 2022-09-07T23:44:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:31.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:31.558932+0000) 2022-09-07T23:44:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:31 smithi055 ceph-mon[97033]: pgmap v4021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:44:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:31 smithi059 ceph-mon[97450]: pgmap v4021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:44:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:32.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:32.652007+0000) 2022-09-07T23:44:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:32.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:32.559174+0000) 2022-09-07T23:44:33.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:33.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:33.652218+0000) 2022-09-07T23:44:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:33.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:33.559413+0000) 2022-09-07T23:44:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:33 smithi055 ceph-mon[97033]: pgmap v4022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:33 smithi059 ceph-mon[97450]: pgmap v4022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:34.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:44:34.069+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:44:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:34.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:34.652444+0000) 2022-09-07T23:44:34.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:34.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:34.559575+0000) 2022-09-07T23:44:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:35 smithi055 ceph-mon[97033]: pgmap v4023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:44:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:35 smithi059 ceph-mon[97450]: pgmap v4023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:44:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:35.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:35.652695+0000) 2022-09-07T23:44:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:35.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:35.559841+0000) 2022-09-07T23:44:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:36.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:36.652933+0000) 2022-09-07T23:44:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:36.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:36.560077+0000) 2022-09-07T23:44:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:37.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:37.653099+0000) 2022-09-07T23:44:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:37.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:37.560307+0000) 2022-09-07T23:44:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:37 smithi055 ceph-mon[97033]: pgmap v4024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:37 smithi059 ceph-mon[97450]: pgmap v4024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:38.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:38.653237+0000) 2022-09-07T23:44:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:38.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:38.560539+0000) 2022-09-07T23:44:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:39.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:39.653406+0000) 2022-09-07T23:44:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:39.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:39.560775+0000) 2022-09-07T23:44:40.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:39 smithi055 ceph-mon[97033]: pgmap v4025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:39 smithi059 ceph-mon[97450]: pgmap v4025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:40.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:40.560972+0000) 2022-09-07T23:44:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:40.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:40.653680+0000) 2022-09-07T23:44:41.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:41.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:41.653913+0000) 2022-09-07T23:44:41.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:41.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:41.561208+0000) 2022-09-07T23:44:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:41 smithi055 ceph-mon[97033]: pgmap v4026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:44:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:41 smithi059 ceph-mon[97450]: pgmap v4026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:44:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:42.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:42.654073+0000) 2022-09-07T23:44:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:42.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:42.561392+0000) 2022-09-07T23:44:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:43.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:43.654311+0000) 2022-09-07T23:44:43.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:43.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:43.561630+0000) 2022-09-07T23:44:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:43 smithi055 ceph-mon[97033]: pgmap v4027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:43 smithi059 ceph-mon[97450]: pgmap v4027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:44.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:44.561856+0000) 2022-09-07T23:44:44.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:44.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:44.654546+0000) 2022-09-07T23:44:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:45 smithi055 ceph-mon[97033]: pgmap v4028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:44:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:45 smithi059 ceph-mon[97450]: pgmap v4028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:44:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:45.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:45.654741+0000) 2022-09-07T23:44:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:45.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:45.562049+0000) 2022-09-07T23:44:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:46.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:46.654909+0000) 2022-09-07T23:44:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:46.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:46.562280+0000) 2022-09-07T23:44:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:47.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:47.655129+0000) 2022-09-07T23:44:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:47.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:47.562490+0000) 2022-09-07T23:44:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:47 smithi055 ceph-mon[97033]: pgmap v4029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:47 smithi059 ceph-mon[97450]: pgmap v4029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:48.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:48.562711+0000) 2022-09-07T23:44:48.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:48.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:48.655364+0000) 2022-09-07T23:44:49.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:44:49.085+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:44:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:49.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:49.655603+0000) 2022-09-07T23:44:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:49.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:49.562925+0000) 2022-09-07T23:44:50.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:49 smithi055 ceph-mon[97033]: pgmap v4030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:49 smithi059 ceph-mon[97450]: pgmap v4030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:50.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:50.655864+0000) 2022-09-07T23:44:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:50.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:50.563192+0000) 2022-09-07T23:44:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:51.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:51.656092+0000) 2022-09-07T23:44:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:51.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:51.563392+0000) 2022-09-07T23:44:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:51 smithi055 ceph-mon[97033]: pgmap v4031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:44:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:51 smithi059 ceph-mon[97450]: pgmap v4031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:44:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:52.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:52.656328+0000) 2022-09-07T23:44:52.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:52.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:52.563589+0000) 2022-09-07T23:44:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:53.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:53.656586+0000) 2022-09-07T23:44:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:53.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:53.563792+0000) 2022-09-07T23:44:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:53 smithi055 ceph-mon[97033]: pgmap v4032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:53 smithi059 ceph-mon[97450]: pgmap v4032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:54.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:54.656746+0000) 2022-09-07T23:44:54.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:54.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:54.563939+0000) 2022-09-07T23:44:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:55 smithi055 ceph-mon[97033]: pgmap v4033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:44:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:55 smithi059 ceph-mon[97450]: pgmap v4033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:44:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:55.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:55.656956+0000) 2022-09-07T23:44:55.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:55.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:55.564118+0000) 2022-09-07T23:44:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:56.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:56.657127+0000) 2022-09-07T23:44:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:56.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:56.564310+0000) 2022-09-07T23:44:57.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:57.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:57.657334+0000) 2022-09-07T23:44:57.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:57.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:57.564534+0000) 2022-09-07T23:44:58.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:57 smithi055 ceph-mon[97033]: pgmap v4034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:58.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:57 smithi059 ceph-mon[97450]: pgmap v4034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:44:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:58.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:58.657553+0000) 2022-09-07T23:44:58.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:58.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:58.564788+0000) 2022-09-07T23:44:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:44:59.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:59.657796+0000) 2022-09-07T23:44:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:44:59.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:44:59.565043+0000) 2022-09-07T23:45:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:44:59 smithi055 ceph-mon[97033]: pgmap v4035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:44:59 smithi059 ceph-mon[97450]: pgmap v4035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:00.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:00.658026+0000) 2022-09-07T23:45:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:00.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:00.565308+0000) 2022-09-07T23:45:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:01.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:01.658285+0000) 2022-09-07T23:45:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:01.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:01.565521+0000) 2022-09-07T23:45:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:01 smithi055 ceph-mon[97033]: pgmap v4036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:45:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:01 smithi059 ceph-mon[97450]: pgmap v4036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:45:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:02.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:02.658487+0000) 2022-09-07T23:45:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:02.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:02.565766+0000) 2022-09-07T23:45:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:03.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:03.565962+0000) 2022-09-07T23:45:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:03.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:03.658738+0000) 2022-09-07T23:45:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:03 smithi055 ceph-mon[97033]: pgmap v4037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:03 smithi059 ceph-mon[97450]: pgmap v4037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:04.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:45:04.100+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:45:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:04.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:04.658953+0000) 2022-09-07T23:45:04.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:04.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:04.566139+0000) 2022-09-07T23:45:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:45:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:45:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:05 smithi055 ceph-mon[97033]: pgmap v4038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:45:05.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:45:05.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:45:05.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:05 smithi059 ceph-mon[97450]: pgmap v4038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:45:05.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:05.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:05.659165+0000) 2022-09-07T23:45:05.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:05.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:05.566370+0000) 2022-09-07T23:45:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:06.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:06.659324+0000) 2022-09-07T23:45:06.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:06.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:06.566612+0000) 2022-09-07T23:45:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:07.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:07.566789+0000) 2022-09-07T23:45:07.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:07.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:07.659520+0000) 2022-09-07T23:45:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:07 smithi055 ceph-mon[97033]: pgmap v4039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:45:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:07 smithi059 ceph-mon[97450]: pgmap v4039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:45:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:08.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:08.659750+0000) 2022-09-07T23:45:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:08.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:08.566962+0000) 2022-09-07T23:45:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:09.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:09.659942+0000) 2022-09-07T23:45:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:09.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:09.567149+0000) 2022-09-07T23:45:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:09 smithi055 ceph-mon[97033]: pgmap v4040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:45:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:09 smithi055 ceph-mon[97033]: osdmap e341: 12 total, 11 up, 11 in 2022-09-07T23:45:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:09 smithi059 ceph-mon[97450]: pgmap v4040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:45:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:09 smithi059 ceph-mon[97450]: osdmap e341: 12 total, 11 up, 11 in 2022-09-07T23:45:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:10.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:10.567383+0000) 2022-09-07T23:45:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:10.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:10.660168+0000) 2022-09-07T23:45:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:11.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:11.660320+0000) 2022-09-07T23:45:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:11.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:11.567624+0000) 2022-09-07T23:45:12.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:11 smithi055 ceph-mon[97033]: pgmap v4042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:45:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:11 smithi059 ceph-mon[97450]: pgmap v4042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:45:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:12.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:12.660549+0000) 2022-09-07T23:45:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:12.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:12.567837+0000) 2022-09-07T23:45:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:13.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:13.660787+0000) 2022-09-07T23:45:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:13.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:13.568017+0000) 2022-09-07T23:45:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:13 smithi055 ceph-mon[97033]: pgmap v4043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:45:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:13 smithi059 ceph-mon[97450]: pgmap v4043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:45:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:14.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:14.661016+0000) 2022-09-07T23:45:14.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:14.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:14.568161+0000) 2022-09-07T23:45:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:15 smithi055 ceph-mon[97033]: osdmap e342: 12 total, 11 up, 11 in 2022-09-07T23:45:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:15 smithi055 ceph-mon[97033]: pgmap v4045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:45:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:15 smithi059 ceph-mon[97450]: osdmap e342: 12 total, 11 up, 11 in 2022-09-07T23:45:15.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:15 smithi059 ceph-mon[97450]: pgmap v4045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:45:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:15.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:15.661163+0000) 2022-09-07T23:45:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:15.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:15.568396+0000) 2022-09-07T23:45:16.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:16.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:16.661400+0000) 2022-09-07T23:45:16.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:16.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:16.568670+0000) 2022-09-07T23:45:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:17.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:17.568859+0000) 2022-09-07T23:45:17.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:17.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:17.661617+0000) 2022-09-07T23:45:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:17 smithi055 ceph-mon[97033]: pgmap v4046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:45:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:17 smithi059 ceph-mon[97450]: pgmap v4046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:45:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:18.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:18.661891+0000) 2022-09-07T23:45:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:18.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:18.569036+0000) 2022-09-07T23:45:19.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:45:19.110+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:45:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:19.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:19.569266+0000) 2022-09-07T23:45:19.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:19.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:19.662063+0000) 2022-09-07T23:45:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:19 smithi055 ceph-mon[97033]: pgmap v4047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T23:45:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:19 smithi059 ceph-mon[97450]: pgmap v4047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T23:45:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:20.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:20.569456+0000) 2022-09-07T23:45:20.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:20.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:20.662253+0000) 2022-09-07T23:45:21.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:21.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:21.569687+0000) 2022-09-07T23:45:21.911 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:21.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:21.662497+0000) 2022-09-07T23:45:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:21 smithi055 ceph-mon[97033]: pgmap v4048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:45:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:21 smithi059 ceph-mon[97450]: pgmap v4048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:45:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:22.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:22.662677+0000) 2022-09-07T23:45:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:22.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:22.569847+0000) 2022-09-07T23:45:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:23.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:23.662855+0000) 2022-09-07T23:45:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:23.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:23.570103+0000) 2022-09-07T23:45:24.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:23 smithi055 ceph-mon[97033]: pgmap v4049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:45:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:23 smithi059 ceph-mon[97450]: pgmap v4049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:45:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:24.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:24.663040+0000) 2022-09-07T23:45:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:24.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:24.570346+0000) 2022-09-07T23:45:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:25 smithi055 ceph-mon[97033]: pgmap v4050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-07T23:45:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:25 smithi059 ceph-mon[97450]: pgmap v4050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-07T23:45:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:25.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:25.663265+0000) 2022-09-07T23:45:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:25.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:25.570593+0000) 2022-09-07T23:45:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:26.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:26.663457+0000) 2022-09-07T23:45:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:26.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:26.570727+0000) 2022-09-07T23:45:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:27.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:27.663655+0000) 2022-09-07T23:45:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:27.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:27.570951+0000) 2022-09-07T23:45:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:27 smithi055 ceph-mon[97033]: pgmap v4051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:28.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:45:28.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:45:28.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:45:28.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:45:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:27 smithi059 ceph-mon[97450]: pgmap v4051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:45:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:45:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:45:28.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:45:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:28.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:28.663878+0000) 2022-09-07T23:45:28.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:28.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:28.571201+0000) 2022-09-07T23:45:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:29.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:29.664093+0000) 2022-09-07T23:45:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:29.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:29.571444+0000) 2022-09-07T23:45:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:29 smithi055 ceph-mon[97033]: pgmap v4052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:29 smithi059 ceph-mon[97450]: pgmap v4052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:30.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:30.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:30.664349+0000) 2022-09-07T23:45:30.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:30.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:30.571622+0000) 2022-09-07T23:45:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:31.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:31.571866+0000) 2022-09-07T23:45:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:31.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:31.664617+0000) 2022-09-07T23:45:32.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:31 smithi055 ceph-mon[97033]: pgmap v4053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:45:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:31 smithi059 ceph-mon[97450]: pgmap v4053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:45:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:32.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:32.664902+0000) 2022-09-07T23:45:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:32.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:32.572099+0000) 2022-09-07T23:45:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:33.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:33.665074+0000) 2022-09-07T23:45:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:33.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:33.572338+0000) 2022-09-07T23:45:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:33 smithi055 ceph-mon[97033]: pgmap v4054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:33 smithi059 ceph-mon[97450]: pgmap v4054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:34.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:45:34.112+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:45:34.912 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:34.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:34.665306+0000) 2022-09-07T23:45:34.913 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:34.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:34.572578+0000) 2022-09-07T23:45:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:35 smithi055 ceph-mon[97033]: pgmap v4055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:45:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:35 smithi059 ceph-mon[97450]: pgmap v4055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:45:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:35.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:35.665530+0000) 2022-09-07T23:45:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:35.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:35.572822+0000) 2022-09-07T23:45:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:36.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:36.665764+0000) 2022-09-07T23:45:36.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:36.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:36.573036+0000) 2022-09-07T23:45:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:37.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:37.666001+0000) 2022-09-07T23:45:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:37.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:37.573269+0000) 2022-09-07T23:45:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:37 smithi055 ceph-mon[97033]: pgmap v4056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:37 smithi059 ceph-mon[97450]: pgmap v4056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:38.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:38.666249+0000) 2022-09-07T23:45:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:38.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:38.573475+0000) 2022-09-07T23:45:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:39.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:39.666489+0000) 2022-09-07T23:45:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:39.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:39.573724+0000) 2022-09-07T23:45:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:39 smithi055 ceph-mon[97033]: pgmap v4057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:39 smithi059 ceph-mon[97450]: pgmap v4057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:40.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:40.666651+0000) 2022-09-07T23:45:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:40.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:40.573912+0000) 2022-09-07T23:45:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:41.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:41.666890+0000) 2022-09-07T23:45:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:41.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:41.574085+0000) 2022-09-07T23:45:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:41 smithi055 ceph-mon[97033]: pgmap v4058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:45:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:41 smithi059 ceph-mon[97450]: pgmap v4058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:45:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:42.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:42.667119+0000) 2022-09-07T23:45:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:42.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:42.574297+0000) 2022-09-07T23:45:43.912 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:43.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:43.667348+0000) 2022-09-07T23:45:43.916 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:43.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:43.574533+0000) 2022-09-07T23:45:44.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:43 smithi055 ceph-mon[97033]: pgmap v4059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:44.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:43 smithi059 ceph-mon[97450]: pgmap v4059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:44.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:44.667541+0000) 2022-09-07T23:45:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:44.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:44.574708+0000) 2022-09-07T23:45:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:45 smithi055 ceph-mon[97033]: pgmap v4060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:45:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:45 smithi059 ceph-mon[97450]: pgmap v4060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:45:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:45.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:45.667755+0000) 2022-09-07T23:45:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:45.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:45.574893+0000) 2022-09-07T23:45:46.669 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:46.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:46.575120+0000) 2022-09-07T23:45:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:46.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:46.668002+0000) 2022-09-07T23:45:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:47.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:47.668226+0000) 2022-09-07T23:45:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:47.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:47.575368+0000) 2022-09-07T23:45:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:47 smithi055 ceph-mon[97033]: pgmap v4061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:47 smithi059 ceph-mon[97450]: pgmap v4061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:48.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:48.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:48.668478+0000) 2022-09-07T23:45:48.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:48.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:48.575555+0000) 2022-09-07T23:45:49.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:45:49.114+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:45:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:49.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:49.668741+0000) 2022-09-07T23:45:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:49.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:49.575783+0000) 2022-09-07T23:45:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:49 smithi055 ceph-mon[97033]: pgmap v4062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:49 smithi059 ceph-mon[97450]: pgmap v4062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:50.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:50.668900+0000) 2022-09-07T23:45:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:50.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:50.576051+0000) 2022-09-07T23:45:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:51.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:51.669130+0000) 2022-09-07T23:45:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:51.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:51.576301+0000) 2022-09-07T23:45:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:51 smithi055 ceph-mon[97033]: pgmap v4063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:45:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:51 smithi059 ceph-mon[97450]: pgmap v4063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:45:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:52.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:52.669268+0000) 2022-09-07T23:45:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:52.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:52.576545+0000) 2022-09-07T23:45:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:53.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:53.669458+0000) 2022-09-07T23:45:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:53.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:53.576748+0000) 2022-09-07T23:45:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:53 smithi055 ceph-mon[97033]: pgmap v4064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:53 smithi059 ceph-mon[97450]: pgmap v4064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:54.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:54.669715+0000) 2022-09-07T23:45:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:54.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:54.576994+0000) 2022-09-07T23:45:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:55 smithi055 ceph-mon[97033]: pgmap v4065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:45:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:55 smithi059 ceph-mon[97450]: pgmap v4065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:45:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:55.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:55.669958+0000) 2022-09-07T23:45:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:55.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:55.577190+0000) 2022-09-07T23:45:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:56.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:56.670224+0000) 2022-09-07T23:45:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:56.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:56.577425+0000) 2022-09-07T23:45:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:57.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:57.670423+0000) 2022-09-07T23:45:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:57.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:57.577617+0000) 2022-09-07T23:45:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:57 smithi055 ceph-mon[97033]: pgmap v4066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:57 smithi059 ceph-mon[97450]: pgmap v4066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:45:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:58.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:58.670571+0000) 2022-09-07T23:45:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:58.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:58.577897+0000) 2022-09-07T23:45:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:45:59.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:59.670799+0000) 2022-09-07T23:45:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:45:59.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:45:59.578126+0000) 2022-09-07T23:46:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:45:59 smithi055 ceph-mon[97033]: pgmap v4067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:45:59 smithi059 ceph-mon[97450]: pgmap v4067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:00.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:00.670998+0000) 2022-09-07T23:46:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:00.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:00.578371+0000) 2022-09-07T23:46:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:01.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:01.671259+0000) 2022-09-07T23:46:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:01.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:01.578627+0000) 2022-09-07T23:46:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:01 smithi055 ceph-mon[97033]: pgmap v4068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:46:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:01 smithi059 ceph-mon[97450]: pgmap v4068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:46:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:02.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:02.671512+0000) 2022-09-07T23:46:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:02.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:02.578883+0000) 2022-09-07T23:46:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:03.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:03.671733+0000) 2022-09-07T23:46:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:03.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:03.579079+0000) 2022-09-07T23:46:04.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:03 smithi055 ceph-mon[97033]: pgmap v4069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:03 smithi059 ceph-mon[97450]: pgmap v4069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:04.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:46:04.119+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:46:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:04.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:04.671897+0000) 2022-09-07T23:46:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:04.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:04.579285+0000) 2022-09-07T23:46:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:46:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:46:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:05 smithi055 ceph-mon[97033]: pgmap v4070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:46:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:46:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:46:05.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:05 smithi059 ceph-mon[97450]: pgmap v4070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:46:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:05.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:05.672116+0000) 2022-09-07T23:46:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:05.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:05.579541+0000) 2022-09-07T23:46:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:06.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:06.672349+0000) 2022-09-07T23:46:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:06.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:06.579801+0000) 2022-09-07T23:46:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:07.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:07.672574+0000) 2022-09-07T23:46:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:07.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:07.580006+0000) 2022-09-07T23:46:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:07 smithi055 ceph-mon[97033]: pgmap v4071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:46:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:07 smithi059 ceph-mon[97450]: pgmap v4071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:46:08.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:08.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:08.672700+0000) 2022-09-07T23:46:08.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:08.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:08.580261+0000) 2022-09-07T23:46:09.909 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:09.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:09.672897+0000) 2022-09-07T23:46:09.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:09.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:09.580471+0000) 2022-09-07T23:46:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:09 smithi059 ceph-mon[97450]: pgmap v4072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:46:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:09 smithi059 ceph-mon[97450]: osdmap e343: 12 total, 11 up, 11 in 2022-09-07T23:46:10.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:09 smithi055 ceph-mon[97033]: pgmap v4072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:46:10.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:09 smithi055 ceph-mon[97033]: osdmap e343: 12 total, 11 up, 11 in 2022-09-07T23:46:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:10.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:10.673126+0000) 2022-09-07T23:46:10.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:10.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:10.580710+0000) 2022-09-07T23:46:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:11.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:11.673304+0000) 2022-09-07T23:46:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:11.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:11.580916+0000) 2022-09-07T23:46:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:11 smithi059 ceph-mon[97450]: pgmap v4074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:46:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:11 smithi055 ceph-mon[97033]: pgmap v4074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:46:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:12.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:12.673502+0000) 2022-09-07T23:46:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:12.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:12.581114+0000) 2022-09-07T23:46:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:13.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:13.673739+0000) 2022-09-07T23:46:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:13.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:13.581359+0000) 2022-09-07T23:46:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:13 smithi059 ceph-mon[97450]: pgmap v4075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:46:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:13 smithi055 ceph-mon[97033]: pgmap v4075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:46:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:14.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:14.674000+0000) 2022-09-07T23:46:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:14.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:14.581618+0000) 2022-09-07T23:46:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:15 smithi055 ceph-mon[97033]: osdmap e344: 12 total, 11 up, 11 in 2022-09-07T23:46:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:15 smithi055 ceph-mon[97033]: pgmap v4077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:46:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:15 smithi059 ceph-mon[97450]: osdmap e344: 12 total, 11 up, 11 in 2022-09-07T23:46:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:15 smithi059 ceph-mon[97450]: pgmap v4077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:46:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:15.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:15.674195+0000) 2022-09-07T23:46:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:15.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:15.581878+0000) 2022-09-07T23:46:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:16.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:16.674415+0000) 2022-09-07T23:46:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:16.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:16.582103+0000) 2022-09-07T23:46:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:17.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:17.674628+0000) 2022-09-07T23:46:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:17.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:17.582356+0000) 2022-09-07T23:46:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:17 smithi055 ceph-mon[97033]: pgmap v4078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:46:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:17 smithi059 ceph-mon[97450]: pgmap v4078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:46:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:18.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:18.674779+0000) 2022-09-07T23:46:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:18.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:18.582553+0000) 2022-09-07T23:46:19.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:46:19.134+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:46:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:19.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:19.674992+0000) 2022-09-07T23:46:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:19.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:19.582775+0000) 2022-09-07T23:46:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:19 smithi059 ceph-mon[97450]: pgmap v4079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:46:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:19 smithi055 ceph-mon[97033]: pgmap v4079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:46:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:20.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:20.675225+0000) 2022-09-07T23:46:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:20.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:20.582981+0000) 2022-09-07T23:46:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:21.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:21.675462+0000) 2022-09-07T23:46:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:21.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:21.583170+0000) 2022-09-07T23:46:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:21 smithi059 ceph-mon[97450]: pgmap v4080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:46:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:21 smithi055 ceph-mon[97033]: pgmap v4080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:46:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:22.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:22.675671+0000) 2022-09-07T23:46:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:22.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:22.583377+0000) 2022-09-07T23:46:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:23.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:23.583618+0000) 2022-09-07T23:46:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:23.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:23.675898+0000) 2022-09-07T23:46:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:23 smithi059 ceph-mon[97450]: pgmap v4081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:46:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:23 smithi055 ceph-mon[97033]: pgmap v4081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:46:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:24.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:24.583860+0000) 2022-09-07T23:46:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:24.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:24.676054+0000) 2022-09-07T23:46:25.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:25 smithi055 ceph-mon[97033]: pgmap v4082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-07T23:46:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:25 smithi059 ceph-mon[97450]: pgmap v4082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-07T23:46:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:25.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:25.676264+0000) 2022-09-07T23:46:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:25.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:25.584110+0000) 2022-09-07T23:46:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:26.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:26.584261+0000) 2022-09-07T23:46:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:26.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:26.676438+0000) 2022-09-07T23:46:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:27.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:27.676654+0000) 2022-09-07T23:46:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:27.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:27.584464+0000) 2022-09-07T23:46:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:27 smithi055 ceph-mon[97033]: pgmap v4083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:46:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:46:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:46:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:27 smithi059 ceph-mon[97450]: pgmap v4083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:46:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:46:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:46:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:28.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:28.676841+0000) 2022-09-07T23:46:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:28.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:28.584779+0000) 2022-09-07T23:46:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:46:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:29 smithi055 ceph-mon[97033]: pgmap v4084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:46:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:29 smithi059 ceph-mon[97450]: pgmap v4084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:29.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:29.677044+0000) 2022-09-07T23:46:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:29.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:29.584933+0000) 2022-09-07T23:46:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:30.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:30.677226+0000) 2022-09-07T23:46:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:30.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:30.585134+0000) 2022-09-07T23:46:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:31.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:31.677440+0000) 2022-09-07T23:46:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:31.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:31.585371+0000) 2022-09-07T23:46:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:31 smithi059 ceph-mon[97450]: pgmap v4085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:46:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:31 smithi055 ceph-mon[97033]: pgmap v4085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:46:32.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:32.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:32.677695+0000) 2022-09-07T23:46:32.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:32.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:32.585604+0000) 2022-09-07T23:46:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:33.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:33.677936+0000) 2022-09-07T23:46:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:33.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:33.585842+0000) 2022-09-07T23:46:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:33 smithi059 ceph-mon[97450]: pgmap v4086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:34.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:46:34.140+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:46:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:33 smithi055 ceph-mon[97033]: pgmap v4086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:34.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:34.678149+0000) 2022-09-07T23:46:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:34.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:34.585995+0000) 2022-09-07T23:46:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:35 smithi055 ceph-mon[97033]: pgmap v4087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:46:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:35 smithi059 ceph-mon[97450]: pgmap v4087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:46:35.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:35.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:35.678372+0000) 2022-09-07T23:46:35.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:35.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:35.586232+0000) 2022-09-07T23:46:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:36.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:36.678623+0000) 2022-09-07T23:46:36.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:36.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:36.586459+0000) 2022-09-07T23:46:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:37.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:37.678903+0000) 2022-09-07T23:46:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:37.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:37.586721+0000) 2022-09-07T23:46:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:37 smithi059 ceph-mon[97450]: pgmap v4088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:37 smithi055 ceph-mon[97033]: pgmap v4088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:38.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:38.679112+0000) 2022-09-07T23:46:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:38.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:38.586926+0000) 2022-09-07T23:46:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:39.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:39.679290+0000) 2022-09-07T23:46:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:39.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:39.587166+0000) 2022-09-07T23:46:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:39 smithi059 ceph-mon[97450]: pgmap v4089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:39 smithi055 ceph-mon[97033]: pgmap v4089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:40.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:40.679515+0000) 2022-09-07T23:46:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:40.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:40.587421+0000) 2022-09-07T23:46:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:41.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:41.679770+0000) 2022-09-07T23:46:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:41.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:41.587609+0000) 2022-09-07T23:46:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:41 smithi059 ceph-mon[97450]: pgmap v4090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:46:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:41 smithi055 ceph-mon[97033]: pgmap v4090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:46:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:42.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:42.680005+0000) 2022-09-07T23:46:42.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:42.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:42.587844+0000) 2022-09-07T23:46:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:43.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:43.680290+0000) 2022-09-07T23:46:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:43.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:43.588025+0000) 2022-09-07T23:46:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:43 smithi059 ceph-mon[97450]: pgmap v4091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:43 smithi055 ceph-mon[97033]: pgmap v4091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:44.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:44.680545+0000) 2022-09-07T23:46:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:44.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:44.588297+0000) 2022-09-07T23:46:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:45 smithi055 ceph-mon[97033]: pgmap v4092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:46:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:45 smithi059 ceph-mon[97450]: pgmap v4092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:46:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:45.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:45.680767+0000) 2022-09-07T23:46:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:45.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:45.588542+0000) 2022-09-07T23:46:46.913 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:46.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:46.681013+0000) 2022-09-07T23:46:46.914 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:46.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:46.588766+0000) 2022-09-07T23:46:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:47.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:47.681217+0000) 2022-09-07T23:46:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:47.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:47.588990+0000) 2022-09-07T23:46:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:47 smithi059 ceph-mon[97450]: pgmap v4093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:47 smithi055 ceph-mon[97033]: pgmap v4093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:48.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:48.681434+0000) 2022-09-07T23:46:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:48.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:48.589225+0000) 2022-09-07T23:46:49.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:46:49.141+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:46:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:49.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:49.681684+0000) 2022-09-07T23:46:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:49.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:49.589453+0000) 2022-09-07T23:46:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:50 smithi059 ceph-mon[97450]: pgmap v4094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:50 smithi055 ceph-mon[97033]: pgmap v4094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:50.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:50.681917+0000) 2022-09-07T23:46:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:50.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:50.589719+0000) 2022-09-07T23:46:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:51.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:51.682125+0000) 2022-09-07T23:46:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:51.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:51.589976+0000) 2022-09-07T23:46:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:52 smithi059 ceph-mon[97450]: pgmap v4095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:46:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:52 smithi055 ceph-mon[97033]: pgmap v4095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:46:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:52.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:52.682407+0000) 2022-09-07T23:46:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:52.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:52.590173+0000) 2022-09-07T23:46:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:53.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:53.682733+0000) 2022-09-07T23:46:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:53.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:53.590403+0000) 2022-09-07T23:46:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:54 smithi059 ceph-mon[97450]: pgmap v4096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:54 smithi055 ceph-mon[97033]: pgmap v4096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:54.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:54.682995+0000) 2022-09-07T23:46:54.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:54.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:54.590618+0000) 2022-09-07T23:46:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:55 smithi055 ceph-mon[97033]: pgmap v4097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:46:55.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:55 smithi059 ceph-mon[97450]: pgmap v4097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:46:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:55.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:55.683201+0000) 2022-09-07T23:46:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:55.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:55.590881+0000) 2022-09-07T23:46:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:56.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:56.683411+0000) 2022-09-07T23:46:56.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:56.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:56.591139+0000) 2022-09-07T23:46:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:57.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:57.683620+0000) 2022-09-07T23:46:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:57.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:57.591380+0000) 2022-09-07T23:46:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:57 smithi059 ceph-mon[97450]: pgmap v4098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:58.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:57 smithi055 ceph-mon[97033]: pgmap v4098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:46:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:58.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:58.683895+0000) 2022-09-07T23:46:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:58.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:58.591626+0000) 2022-09-07T23:46:59.909 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:46:59.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:59.684032+0000) 2022-09-07T23:46:59.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:46:59.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:46:59.591916+0000) 2022-09-07T23:47:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:46:59 smithi059 ceph-mon[97450]: pgmap v4099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:46:59 smithi055 ceph-mon[97033]: pgmap v4099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:00.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:00.684228+0000) 2022-09-07T23:47:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:00.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:00.592124+0000) 2022-09-07T23:47:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:01.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:01.684439+0000) 2022-09-07T23:47:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:01.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:01.592356+0000) 2022-09-07T23:47:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:01 smithi059 ceph-mon[97450]: pgmap v4100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:47:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:01 smithi055 ceph-mon[97033]: pgmap v4100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:47:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:02.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:02.684592+0000) 2022-09-07T23:47:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:02.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:02.592582+0000) 2022-09-07T23:47:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:03.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:03.592780+0000) 2022-09-07T23:47:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:03.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:03.684795+0000) 2022-09-07T23:47:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:03 smithi059 ceph-mon[97450]: pgmap v4101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:03 smithi055 ceph-mon[97033]: pgmap v4101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:04.594 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:47:04.156+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:47:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:04.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:04.684975+0000) 2022-09-07T23:47:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:04.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:04.593033+0000) 2022-09-07T23:47:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:47:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:47:05.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:05 smithi055 ceph-mon[97033]: pgmap v4102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:47:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:47:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:47:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:05 smithi059 ceph-mon[97450]: pgmap v4102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:47:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:05.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:05.685217+0000) 2022-09-07T23:47:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:05.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:05.593278+0000) 2022-09-07T23:47:06.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:06.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:06.685440+0000) 2022-09-07T23:47:06.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:06.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:06.593510+0000) 2022-09-07T23:47:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:07.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:07.685708+0000) 2022-09-07T23:47:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:07.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:07.593674+0000) 2022-09-07T23:47:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:07 smithi059 ceph-mon[97450]: pgmap v4103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:47:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:07 smithi055 ceph-mon[97033]: pgmap v4103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:47:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:08.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:08.685958+0000) 2022-09-07T23:47:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:08.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:08.593793+0000) 2022-09-07T23:47:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:09.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:09.686234+0000) 2022-09-07T23:47:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:09.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:09.593993+0000) 2022-09-07T23:47:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:10 smithi059 ceph-mon[97450]: pgmap v4104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:47:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:10 smithi059 ceph-mon[97450]: osdmap e345: 12 total, 11 up, 11 in 2022-09-07T23:47:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:10 smithi055 ceph-mon[97033]: pgmap v4104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:47:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:10 smithi055 ceph-mon[97033]: osdmap e345: 12 total, 11 up, 11 in 2022-09-07T23:47:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:10.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:10.686486+0000) 2022-09-07T23:47:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:10.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:10.594167+0000) 2022-09-07T23:47:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:11.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:11.686762+0000) 2022-09-07T23:47:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:11.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:11.594391+0000) 2022-09-07T23:47:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:12 smithi059 ceph-mon[97450]: pgmap v4106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:47:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:12 smithi055 ceph-mon[97033]: pgmap v4106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:47:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:12.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:12.686997+0000) 2022-09-07T23:47:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:12.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:12.594580+0000) 2022-09-07T23:47:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:13.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:13.687238+0000) 2022-09-07T23:47:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:13.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:13.594792+0000) 2022-09-07T23:47:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:14 smithi059 ceph-mon[97450]: pgmap v4107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:47:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:14 smithi055 ceph-mon[97033]: pgmap v4107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:47:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:14.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:14.687390+0000) 2022-09-07T23:47:14.944 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:14.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:14.595030+0000) 2022-09-07T23:47:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:15 smithi055 ceph-mon[97033]: osdmap e346: 12 total, 11 up, 11 in 2022-09-07T23:47:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:15 smithi055 ceph-mon[97033]: pgmap v4109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:47:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:15 smithi059 ceph-mon[97450]: osdmap e346: 12 total, 11 up, 11 in 2022-09-07T23:47:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:15 smithi059 ceph-mon[97450]: pgmap v4109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:47:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:15.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:15.687578+0000) 2022-09-07T23:47:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:15.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:15.595266+0000) 2022-09-07T23:47:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:16.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:16.687813+0000) 2022-09-07T23:47:16.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:16.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:16.595468+0000) 2022-09-07T23:47:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:17.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:17.687980+0000) 2022-09-07T23:47:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:17.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:17.595701+0000) 2022-09-07T23:47:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:17 smithi059 ceph-mon[97450]: pgmap v4110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:47:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:17 smithi055 ceph-mon[97033]: pgmap v4110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:47:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:18.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:18.688192+0000) 2022-09-07T23:47:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:18.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:18.595828+0000) 2022-09-07T23:47:19.597 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:47:19.159+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:47:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:19.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:19.688438+0000) 2022-09-07T23:47:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:19.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:19.596033+0000) 2022-09-07T23:47:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:19 smithi059 ceph-mon[97450]: pgmap v4111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:47:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:19 smithi055 ceph-mon[97033]: pgmap v4111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:47:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:20.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:20.688645+0000) 2022-09-07T23:47:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:20.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:20.596268+0000) 2022-09-07T23:47:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:21.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:21.688868+0000) 2022-09-07T23:47:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:21.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:21.596477+0000) 2022-09-07T23:47:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:21 smithi059 ceph-mon[97450]: pgmap v4112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:47:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:21 smithi055 ceph-mon[97033]: pgmap v4112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:47:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:22.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:22.689020+0000) 2022-09-07T23:47:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:22.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:22.596729+0000) 2022-09-07T23:47:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:23.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:23.689154+0000) 2022-09-07T23:47:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:23.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:23.596906+0000) 2022-09-07T23:47:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:24 smithi059 ceph-mon[97450]: pgmap v4113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:47:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:24 smithi055 ceph-mon[97033]: pgmap v4113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:47:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:24.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:24.689335+0000) 2022-09-07T23:47:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:24.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:24.597141+0000) 2022-09-07T23:47:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:25 smithi055 ceph-mon[97033]: pgmap v4114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-07T23:47:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:25 smithi059 ceph-mon[97450]: pgmap v4114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-07T23:47:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:25.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:25.689521+0000) 2022-09-07T23:47:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:25.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:25.597403+0000) 2022-09-07T23:47:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:26.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:26.597631+0000) 2022-09-07T23:47:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:26.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:26.689690+0000) 2022-09-07T23:47:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:27.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:27.597918+0000) 2022-09-07T23:47:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:27.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:27.689891+0000) 2022-09-07T23:47:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:27 smithi059 ceph-mon[97450]: pgmap v4115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:27 smithi055 ceph-mon[97033]: pgmap v4115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:28.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:28.690104+0000) 2022-09-07T23:47:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:28.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:28.598124+0000) 2022-09-07T23:47:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:47:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:47:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:47:29.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:47:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:47:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:47:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:29.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:29.690277+0000) 2022-09-07T23:47:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:29.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:29.598294+0000) 2022-09-07T23:47:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:30 smithi055 ceph-mon[97033]: pgmap v4116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:47:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:30 smithi059 ceph-mon[97450]: pgmap v4116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:47:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:30.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:30.690461+0000) 2022-09-07T23:47:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:30.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:30.598469+0000) 2022-09-07T23:47:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:31.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:31.690725+0000) 2022-09-07T23:47:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:31.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:31.598702+0000) 2022-09-07T23:47:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:32 smithi055 ceph-mon[97033]: pgmap v4117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:47:32.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:32 smithi059 ceph-mon[97450]: pgmap v4117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:47:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:32.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:32.690923+0000) 2022-09-07T23:47:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:32.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:32.598936+0000) 2022-09-07T23:47:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:33.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:33.691168+0000) 2022-09-07T23:47:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:33.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:33.599187+0000) 2022-09-07T23:47:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:34 smithi055 ceph-mon[97033]: pgmap v4118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:34 smithi059 ceph-mon[97450]: pgmap v4118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:34.600 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:47:34.169+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:47:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:34.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:34.691328+0000) 2022-09-07T23:47:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:34.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:34.599423+0000) 2022-09-07T23:47:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:35 smithi055 ceph-mon[97033]: pgmap v4119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:47:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:35 smithi059 ceph-mon[97450]: pgmap v4119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:47:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:35.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:35.691525+0000) 2022-09-07T23:47:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:35.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:35.599701+0000) 2022-09-07T23:47:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:36.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:36.691748+0000) 2022-09-07T23:47:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:36.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:36.599934+0000) 2022-09-07T23:47:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:37.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:37.691927+0000) 2022-09-07T23:47:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:37.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:37.600185+0000) 2022-09-07T23:47:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:37 smithi059 ceph-mon[97450]: pgmap v4120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:38 smithi055 ceph-mon[97033]: pgmap v4120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:38.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:38.692159+0000) 2022-09-07T23:47:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:38.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:38.600436+0000) 2022-09-07T23:47:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:39.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:39.692364+0000) 2022-09-07T23:47:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:39.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:39.600565+0000) 2022-09-07T23:47:40.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:40 smithi059 ceph-mon[97450]: pgmap v4121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:40 smithi055 ceph-mon[97033]: pgmap v4121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:40.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:40.692661+0000) 2022-09-07T23:47:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:40.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:40.600803+0000) 2022-09-07T23:47:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:41.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:41.692897+0000) 2022-09-07T23:47:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:41.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:41.601037+0000) 2022-09-07T23:47:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:42 smithi059 ceph-mon[97450]: pgmap v4122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:47:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:42 smithi055 ceph-mon[97033]: pgmap v4122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:47:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:42.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:42.693149+0000) 2022-09-07T23:47:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:42.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:42.601257+0000) 2022-09-07T23:47:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:43.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:43.693362+0000) 2022-09-07T23:47:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:43.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:43.601509+0000) 2022-09-07T23:47:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:44 smithi059 ceph-mon[97450]: pgmap v4123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:44 smithi055 ceph-mon[97033]: pgmap v4123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:44.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:44.693584+0000) 2022-09-07T23:47:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:44.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:44.601797+0000) 2022-09-07T23:47:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:45 smithi055 ceph-mon[97033]: pgmap v4124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:47:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:45 smithi059 ceph-mon[97450]: pgmap v4124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:47:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:45.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:45.602017+0000) 2022-09-07T23:47:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:45.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:45.693808+0000) 2022-09-07T23:47:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:46.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:46.693968+0000) 2022-09-07T23:47:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:46.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:46.602225+0000) 2022-09-07T23:47:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:47.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:47.694122+0000) 2022-09-07T23:47:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:47.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:47.602462+0000) 2022-09-07T23:47:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:48 smithi059 ceph-mon[97450]: pgmap v4125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:48 smithi055 ceph-mon[97033]: pgmap v4125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:48.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:48.602707+0000) 2022-09-07T23:47:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:48.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:48.694275+0000) 2022-09-07T23:47:49.604 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:47:49.183+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:47:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:49.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:49.602917+0000) 2022-09-07T23:47:49.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:49.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:49.694439+0000) 2022-09-07T23:47:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:50 smithi059 ceph-mon[97450]: pgmap v4126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:50 smithi055 ceph-mon[97033]: pgmap v4126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:50.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:50.694713+0000) 2022-09-07T23:47:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:50.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:50.603152+0000) 2022-09-07T23:47:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:51.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:51.694945+0000) 2022-09-07T23:47:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:51.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:51.603341+0000) 2022-09-07T23:47:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:52 smithi059 ceph-mon[97450]: pgmap v4127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:47:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:52 smithi055 ceph-mon[97033]: pgmap v4127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:47:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:52.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:52.695170+0000) 2022-09-07T23:47:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:52.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:52.603558+0000) 2022-09-07T23:47:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:53.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:53.695337+0000) 2022-09-07T23:47:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:53.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:53.603764+0000) 2022-09-07T23:47:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:54 smithi059 ceph-mon[97450]: pgmap v4128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:54 smithi055 ceph-mon[97033]: pgmap v4128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:54.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:54.695559+0000) 2022-09-07T23:47:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:54.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:54.603970+0000) 2022-09-07T23:47:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:55 smithi055 ceph-mon[97033]: pgmap v4129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:47:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:55 smithi059 ceph-mon[97450]: pgmap v4129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:47:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:55.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:55.604189+0000) 2022-09-07T23:47:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:55.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:55.695784+0000) 2022-09-07T23:47:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:56.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:56.695937+0000) 2022-09-07T23:47:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:56.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:56.604330+0000) 2022-09-07T23:47:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:57.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:57.696102+0000) 2022-09-07T23:47:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:57.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:57.604578+0000) 2022-09-07T23:47:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:47:58 smithi059 ceph-mon[97450]: pgmap v4130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:58.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:47:58 smithi055 ceph-mon[97033]: pgmap v4130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:47:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:58.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:58.696264+0000) 2022-09-07T23:47:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:58.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:58.604792+0000) 2022-09-07T23:47:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:47:59.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:59.696472+0000) 2022-09-07T23:47:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:47:59.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:47:59.605027+0000) 2022-09-07T23:48:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:00 smithi059 ceph-mon[97450]: pgmap v4131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:00 smithi055 ceph-mon[97033]: pgmap v4131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:00.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:00.696697+0000) 2022-09-07T23:48:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:00.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:00.605177+0000) 2022-09-07T23:48:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:01.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:01.696840+0000) 2022-09-07T23:48:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:01.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:01.605420+0000) 2022-09-07T23:48:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:02 smithi059 ceph-mon[97450]: pgmap v4132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:48:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:02 smithi055 ceph-mon[97033]: pgmap v4132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:48:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:02.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:02.697089+0000) 2022-09-07T23:48:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:02.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:02.605620+0000) 2022-09-07T23:48:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:03.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:03.605842+0000) 2022-09-07T23:48:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:03.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:03.697303+0000) 2022-09-07T23:48:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:04 smithi059 ceph-mon[97450]: pgmap v4133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:04 smithi055 ceph-mon[97033]: pgmap v4133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:04.610 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:48:04.199+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:48:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:04.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:04.606000+0000) 2022-09-07T23:48:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:04.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:04.697536+0000) 2022-09-07T23:48:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:48:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:48:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:05 smithi055 ceph-mon[97033]: pgmap v4134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:48:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:48:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:48:05.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:05 smithi059 ceph-mon[97450]: pgmap v4134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:48:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:05.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:05.697782+0000) 2022-09-07T23:48:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:05.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:05.606239+0000) 2022-09-07T23:48:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:06.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:06.606383+0000) 2022-09-07T23:48:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:06.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:06.697955+0000) 2022-09-07T23:48:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:07.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:07.698134+0000) 2022-09-07T23:48:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:07.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:07.606595+0000) 2022-09-07T23:48:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:08 smithi059 ceph-mon[97450]: pgmap v4135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:48:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:08 smithi055 ceph-mon[97033]: pgmap v4135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:48:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:08.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:08.606831+0000) 2022-09-07T23:48:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:08.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:08.698295+0000) 2022-09-07T23:48:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:09.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:09.698508+0000) 2022-09-07T23:48:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:09.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:09.607059+0000) 2022-09-07T23:48:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:10 smithi059 ceph-mon[97450]: pgmap v4136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:48:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:10 smithi059 ceph-mon[97450]: osdmap e347: 12 total, 11 up, 11 in 2022-09-07T23:48:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:10 smithi055 ceph-mon[97033]: pgmap v4136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:48:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:10 smithi055 ceph-mon[97033]: osdmap e347: 12 total, 11 up, 11 in 2022-09-07T23:48:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:10.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:10.698735+0000) 2022-09-07T23:48:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:10.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:10.607258+0000) 2022-09-07T23:48:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:11.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:11.698918+0000) 2022-09-07T23:48:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:11.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:11.607509+0000) 2022-09-07T23:48:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:12 smithi059 ceph-mon[97450]: pgmap v4138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:48:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:12 smithi055 ceph-mon[97033]: pgmap v4138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:48:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:12.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:12.699138+0000) 2022-09-07T23:48:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:12.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:12.607681+0000) 2022-09-07T23:48:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:13.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:13.699355+0000) 2022-09-07T23:48:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:13.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:13.607831+0000) 2022-09-07T23:48:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:14 smithi059 ceph-mon[97450]: pgmap v4139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:48:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:14 smithi055 ceph-mon[97033]: pgmap v4139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:48:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:14.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:14.699576+0000) 2022-09-07T23:48:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:14.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:14.608060+0000) 2022-09-07T23:48:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:15 smithi055 ceph-mon[97033]: osdmap e348: 12 total, 11 up, 11 in 2022-09-07T23:48:15.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:15 smithi055 ceph-mon[97033]: pgmap v4141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:48:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:15 smithi059 ceph-mon[97450]: osdmap e348: 12 total, 11 up, 11 in 2022-09-07T23:48:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:15 smithi059 ceph-mon[97450]: pgmap v4141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:48:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:15.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:15.608312+0000) 2022-09-07T23:48:15.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:15.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:15.699840+0000) 2022-09-07T23:48:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:16.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:16.700060+0000) 2022-09-07T23:48:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:16.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:16.608524+0000) 2022-09-07T23:48:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:17.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:17.700327+0000) 2022-09-07T23:48:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:17.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:17.608754+0000) 2022-09-07T23:48:18.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:18 smithi059 ceph-mon[97450]: pgmap v4142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:48:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:18 smithi055 ceph-mon[97033]: pgmap v4142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:48:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:18.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:18.700546+0000) 2022-09-07T23:48:18.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:18.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:18.608936+0000) 2022-09-07T23:48:19.610 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:48:19.207+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:48:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:19.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:19.700759+0000) 2022-09-07T23:48:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:19.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:19.609185+0000) 2022-09-07T23:48:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:20 smithi059 ceph-mon[97450]: pgmap v4143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T23:48:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:20 smithi055 ceph-mon[97033]: pgmap v4143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T23:48:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:20.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:20.700939+0000) 2022-09-07T23:48:20.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:20.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:20.609401+0000) 2022-09-07T23:48:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:21.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:21.701165+0000) 2022-09-07T23:48:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:21.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:21.609709+0000) 2022-09-07T23:48:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:22 smithi059 ceph-mon[97450]: pgmap v4144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T23:48:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:22 smithi055 ceph-mon[97033]: pgmap v4144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T23:48:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:22.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:22.701398+0000) 2022-09-07T23:48:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:22.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:22.609933+0000) 2022-09-07T23:48:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:23.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:23.701612+0000) 2022-09-07T23:48:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:23.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:23.610153+0000) 2022-09-07T23:48:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:24 smithi059 ceph-mon[97450]: pgmap v4145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T23:48:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:24 smithi055 ceph-mon[97033]: pgmap v4145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-07T23:48:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:24.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:24.701733+0000) 2022-09-07T23:48:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:24.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:24.610371+0000) 2022-09-07T23:48:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:25 smithi055 ceph-mon[97033]: pgmap v4146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-07T23:48:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:25 smithi059 ceph-mon[97450]: pgmap v4146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-07T23:48:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:25.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:25.701920+0000) 2022-09-07T23:48:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:25.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:25.610625+0000) 2022-09-07T23:48:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:26.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:26.702159+0000) 2022-09-07T23:48:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:26.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:26.610918+0000) 2022-09-07T23:48:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:27.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:27.702418+0000) 2022-09-07T23:48:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:27.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:27.611063+0000) 2022-09-07T23:48:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:28 smithi059 ceph-mon[97450]: pgmap v4147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:28 smithi055 ceph-mon[97033]: pgmap v4147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:28.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:28.702688+0000) 2022-09-07T23:48:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:28.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:28.611255+0000) 2022-09-07T23:48:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:29.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:29.702897+0000) 2022-09-07T23:48:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:29.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:29.611442+0000) 2022-09-07T23:48:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:30 smithi059 ceph-mon[97450]: pgmap v4148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:48:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:48:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:48:30.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:48:30.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:48:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:30 smithi055 ceph-mon[97033]: pgmap v4148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:48:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:48:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:48:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:48:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:48:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:30.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:30.703118+0000) 2022-09-07T23:48:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:30.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:30.611589+0000) 2022-09-07T23:48:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:31.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:31.703352+0000) 2022-09-07T23:48:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:31.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:31.611778+0000) 2022-09-07T23:48:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:32 smithi059 ceph-mon[97450]: pgmap v4149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:48:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:32 smithi055 ceph-mon[97033]: pgmap v4149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:48:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:32.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:32.703535+0000) 2022-09-07T23:48:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:32.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:32.612000+0000) 2022-09-07T23:48:33.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:48:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:48:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:48:33.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:48:33.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:48:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:48:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:48:33.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:48:33.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:48:33.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:48:33.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:48:33.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:33.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:33.703692+0000) 2022-09-07T23:48:33.922 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:33.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:33.612251+0000) 2022-09-07T23:48:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:34 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T23:48:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:34 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T23:48:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:34 smithi055 ceph-mon[97033]: pgmap v4150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:34.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:34 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T23:48:34.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:34 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T23:48:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:34 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T23:48:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:34 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T23:48:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:34 smithi059 ceph-mon[97450]: pgmap v4150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:34 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T23:48:34.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:34 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T23:48:34.613 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:48:34.217+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:48:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:34.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:34.703922+0000) 2022-09-07T23:48:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:34.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:34.612453+0000) 2022-09-07T23:48:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:35 smithi055 ceph-mon[97033]: pgmap v4151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:48:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:35 smithi059 ceph-mon[97450]: pgmap v4151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:48:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:35.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:35.704179+0000) 2022-09-07T23:48:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:35.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:35.612736+0000) 2022-09-07T23:48:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:36.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:36.704388+0000) 2022-09-07T23:48:36.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:36.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:36.612991+0000) 2022-09-07T23:48:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:37.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:37.704594+0000) 2022-09-07T23:48:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:37.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:37.613187+0000) 2022-09-07T23:48:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:38 smithi059 ceph-mon[97450]: pgmap v4152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:38 smithi055 ceph-mon[97033]: pgmap v4152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:38.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:38.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:38.704815+0000) 2022-09-07T23:48:38.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:38.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:38.613366+0000) 2022-09-07T23:48:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:39.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:39.705030+0000) 2022-09-07T23:48:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:39.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:39.613540+0000) 2022-09-07T23:48:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:40 smithi059 ceph-mon[97450]: pgmap v4153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:40 smithi055 ceph-mon[97033]: pgmap v4153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:40.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:40.705220+0000) 2022-09-07T23:48:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:40.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:40.613782+0000) 2022-09-07T23:48:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:41.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:41.614038+0000) 2022-09-07T23:48:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:41.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:41.705492+0000) 2022-09-07T23:48:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:42 smithi059 ceph-mon[97450]: pgmap v4154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:48:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:42 smithi055 ceph-mon[97033]: pgmap v4154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:48:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:42.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:42.614264+0000) 2022-09-07T23:48:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:42.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:42.705779+0000) 2022-09-07T23:48:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:43.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:43.705984+0000) 2022-09-07T23:48:43.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:43.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:43.614502+0000) 2022-09-07T23:48:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:44 smithi059 ceph-mon[97450]: pgmap v4155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:44 smithi055 ceph-mon[97033]: pgmap v4155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:44.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:44.614734+0000) 2022-09-07T23:48:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:44.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:44.706145+0000) 2022-09-07T23:48:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:45 smithi055 ceph-mon[97033]: pgmap v4156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:48:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:45 smithi059 ceph-mon[97450]: pgmap v4156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:48:45.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:45.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:45.706367+0000) 2022-09-07T23:48:45.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:45.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:45.614928+0000) 2022-09-07T23:48:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:46.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:46.615172+0000) 2022-09-07T23:48:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:46.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:46.706611+0000) 2022-09-07T23:48:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:47.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:47.706844+0000) 2022-09-07T23:48:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:47.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:47.615391+0000) 2022-09-07T23:48:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:48 smithi059 ceph-mon[97450]: pgmap v4157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:48 smithi055 ceph-mon[97033]: pgmap v4157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:48.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:48.707039+0000) 2022-09-07T23:48:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:48.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:48.615544+0000) 2022-09-07T23:48:49.617 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:48:49.222+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:48:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:49.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:49.707226+0000) 2022-09-07T23:48:49.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:49.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:49.615752+0000) 2022-09-07T23:48:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:50 smithi059 ceph-mon[97450]: pgmap v4158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:50 smithi055 ceph-mon[97033]: pgmap v4158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:50.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:50.707423+0000) 2022-09-07T23:48:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:50.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:50.615984+0000) 2022-09-07T23:48:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:51.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:51.707616+0000) 2022-09-07T23:48:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:51.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:51.616223+0000) 2022-09-07T23:48:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:52 smithi059 ceph-mon[97450]: pgmap v4159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:48:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:52 smithi055 ceph-mon[97033]: pgmap v4159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:48:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:52.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:52.707814+0000) 2022-09-07T23:48:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:52.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:52.616501+0000) 2022-09-07T23:48:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:53.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:53.708024+0000) 2022-09-07T23:48:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:53.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:53.616748+0000) 2022-09-07T23:48:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:54 smithi055 ceph-mon[97033]: pgmap v4160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:54 smithi059 ceph-mon[97450]: pgmap v4160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:54.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:54.708252+0000) 2022-09-07T23:48:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:54.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:54.616985+0000) 2022-09-07T23:48:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:55 smithi055 ceph-mon[97033]: pgmap v4161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:48:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:55 smithi059 ceph-mon[97450]: pgmap v4161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:48:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:55.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:55.708485+0000) 2022-09-07T23:48:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:55.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:55.617265+0000) 2022-09-07T23:48:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:56.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:56.708753+0000) 2022-09-07T23:48:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:56.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:56.617480+0000) 2022-09-07T23:48:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:57.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:57.708920+0000) 2022-09-07T23:48:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:57.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:57.617762+0000) 2022-09-07T23:48:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:48:58 smithi059 ceph-mon[97450]: pgmap v4162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:48:58 smithi055 ceph-mon[97033]: pgmap v4162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:48:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:58.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:58.709156+0000) 2022-09-07T23:48:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:58.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:58.617986+0000) 2022-09-07T23:48:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:48:59.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:59.709439+0000) 2022-09-07T23:48:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:48:59.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:48:59.618236+0000) 2022-09-07T23:49:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:00 smithi059 ceph-mon[97450]: pgmap v4163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:00 smithi055 ceph-mon[97033]: pgmap v4163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:00.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:00.709630+0000) 2022-09-07T23:49:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:00.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:00.618437+0000) 2022-09-07T23:49:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:01.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:01.709850+0000) 2022-09-07T23:49:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:01.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:01.618677+0000) 2022-09-07T23:49:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:02 smithi055 ceph-mon[97033]: pgmap v4164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:49:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:02 smithi059 ceph-mon[97450]: pgmap v4164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:49:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:02.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:02.710051+0000) 2022-09-07T23:49:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:02.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:02.618831+0000) 2022-09-07T23:49:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:03.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:03.710230+0000) 2022-09-07T23:49:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:03.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:03.619053+0000) 2022-09-07T23:49:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:04 smithi055 ceph-mon[97033]: pgmap v4165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:04 smithi059 ceph-mon[97450]: pgmap v4165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:04.621 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:49:04.235+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:49:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:04.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:04.619262+0000) 2022-09-07T23:49:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:04.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:04.710451+0000) 2022-09-07T23:49:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:49:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:49:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:05 smithi055 ceph-mon[97033]: pgmap v4166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:49:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:49:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:49:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:05 smithi059 ceph-mon[97450]: pgmap v4166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:49:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:05.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:05.710593+0000) 2022-09-07T23:49:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:05.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:05.619501+0000) 2022-09-07T23:49:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:06.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:06.710826+0000) 2022-09-07T23:49:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:06.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:06.619673+0000) 2022-09-07T23:49:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:07.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:07.619849+0000) 2022-09-07T23:49:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:07.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:07.711072+0000) 2022-09-07T23:49:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:08 smithi059 ceph-mon[97450]: pgmap v4167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:49:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:08 smithi055 ceph-mon[97033]: pgmap v4167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:49:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:08.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:08.620095+0000) 2022-09-07T23:49:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:08.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:08.711207+0000) 2022-09-07T23:49:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:09.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:09.620290+0000) 2022-09-07T23:49:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:09.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:09.711329+0000) 2022-09-07T23:49:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:10 smithi055 ceph-mon[97033]: pgmap v4168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:49:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:10 smithi055 ceph-mon[97033]: osdmap e349: 12 total, 11 up, 11 in 2022-09-07T23:49:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:10 smithi059 ceph-mon[97450]: pgmap v4168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:49:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:10 smithi059 ceph-mon[97450]: osdmap e349: 12 total, 11 up, 11 in 2022-09-07T23:49:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:10.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:10.711532+0000) 2022-09-07T23:49:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:10.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:10.620492+0000) 2022-09-07T23:49:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:11.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:11.711769+0000) 2022-09-07T23:49:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:11.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:11.620735+0000) 2022-09-07T23:49:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:12 smithi055 ceph-mon[97033]: pgmap v4170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:49:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:12 smithi059 ceph-mon[97450]: pgmap v4170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:49:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:12.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:12.711972+0000) 2022-09-07T23:49:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:12.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:12.622758+0000) 2022-09-07T23:49:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:13.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:13.712204+0000) 2022-09-07T23:49:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:13.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:13.622939+0000) 2022-09-07T23:49:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:14 smithi055 ceph-mon[97033]: pgmap v4171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:49:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:14 smithi059 ceph-mon[97450]: pgmap v4171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:49:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:14.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:14.712436+0000) 2022-09-07T23:49:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:14.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:14.623171+0000) 2022-09-07T23:49:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:15 smithi055 ceph-mon[97033]: osdmap e350: 12 total, 11 up, 11 in 2022-09-07T23:49:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:15 smithi055 ceph-mon[97033]: pgmap v4173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:49:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:15 smithi059 ceph-mon[97450]: osdmap e350: 12 total, 11 up, 11 in 2022-09-07T23:49:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:15 smithi059 ceph-mon[97450]: pgmap v4173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:49:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:15.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:15.712691+0000) 2022-09-07T23:49:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:15.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:15.623393+0000) 2022-09-07T23:49:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:16.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:16.712946+0000) 2022-09-07T23:49:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:16.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:16.623666+0000) 2022-09-07T23:49:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:17.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:17.713135+0000) 2022-09-07T23:49:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:17.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:17.623931+0000) 2022-09-07T23:49:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:18 smithi055 ceph-mon[97033]: pgmap v4174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:49:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:18 smithi059 ceph-mon[97450]: pgmap v4174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:49:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:18.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:18.713394+0000) 2022-09-07T23:49:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:18.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:18.624106+0000) 2022-09-07T23:49:19.625 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:49:19.238+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:49:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:19.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:19.713682+0000) 2022-09-07T23:49:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:19.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:19.624352+0000) 2022-09-07T23:49:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:20 smithi055 ceph-mon[97033]: pgmap v4175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T23:49:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:20 smithi059 ceph-mon[97450]: pgmap v4175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T23:49:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:20.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:20.624597+0000) 2022-09-07T23:49:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:20.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:20.713936+0000) 2022-09-07T23:49:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:21.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:21.714146+0000) 2022-09-07T23:49:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:21.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:21.624815+0000) 2022-09-07T23:49:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:22 smithi055 ceph-mon[97033]: pgmap v4176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:49:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:22 smithi059 ceph-mon[97450]: pgmap v4176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:49:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:22.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:22.714346+0000) 2022-09-07T23:49:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:22.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:22.625023+0000) 2022-09-07T23:49:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:23.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:23.714551+0000) 2022-09-07T23:49:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:23.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:23.625243+0000) 2022-09-07T23:49:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:24 smithi055 ceph-mon[97033]: pgmap v4177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:49:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:24 smithi059 ceph-mon[97450]: pgmap v4177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:49:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:24.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:24.714754+0000) 2022-09-07T23:49:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:24.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:24.625406+0000) 2022-09-07T23:49:25.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:25 smithi055 ceph-mon[97033]: pgmap v4178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-07T23:49:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:25 smithi059 ceph-mon[97450]: pgmap v4178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-07T23:49:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:25.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:25.714958+0000) 2022-09-07T23:49:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:25.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:25.625654+0000) 2022-09-07T23:49:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:26.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:26.715110+0000) 2022-09-07T23:49:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:26.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:26.625872+0000) 2022-09-07T23:49:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:27.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:27.626108+0000) 2022-09-07T23:49:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:27.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:27.715330+0000) 2022-09-07T23:49:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:28 smithi055 ceph-mon[97033]: pgmap v4179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:49:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:28 smithi059 ceph-mon[97450]: pgmap v4179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:49:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:28.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:28.715538+0000) 2022-09-07T23:49:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:28.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:28.626324+0000) 2022-09-07T23:49:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:29.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:29.715761+0000) 2022-09-07T23:49:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:29.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:29.626566+0000) 2022-09-07T23:49:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:30 smithi055 ceph-mon[97033]: pgmap v4180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:49:30.538 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:30 smithi059 ceph-mon[97450]: pgmap v4180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:49:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:30.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:30.715992+0000) 2022-09-07T23:49:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:30.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:30.626796+0000) 2022-09-07T23:49:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:31.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:31.716164+0000) 2022-09-07T23:49:31.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:31.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:31.627046+0000) 2022-09-07T23:49:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:32 smithi055 ceph-mon[97033]: pgmap v4181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:49:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:32 smithi059 ceph-mon[97450]: pgmap v4181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:49:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:32.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:32.716364+0000) 2022-09-07T23:49:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:32.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:32.627273+0000) 2022-09-07T23:49:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:33.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:33.716547+0000) 2022-09-07T23:49:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:33.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:33.627515+0000) 2022-09-07T23:49:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:34 smithi055 ceph-mon[97033]: pgmap v4182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:49:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:49:34.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:49:34.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:49:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:34 smithi059 ceph-mon[97450]: pgmap v4182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:49:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:49:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:49:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:49:34.629 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:49:34.250+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:49:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:34.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:34.716799+0000) 2022-09-07T23:49:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:34.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:34.627732+0000) 2022-09-07T23:49:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:35 smithi055 ceph-mon[97033]: pgmap v4183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:49:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:35 smithi059 ceph-mon[97450]: pgmap v4183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:49:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:35.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:35.717029+0000) 2022-09-07T23:49:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:35.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:35.628002+0000) 2022-09-07T23:49:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:36.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:36.717243+0000) 2022-09-07T23:49:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:36.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:36.628238+0000) 2022-09-07T23:49:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:37.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:37.717453+0000) 2022-09-07T23:49:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:37.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:37.628507+0000) 2022-09-07T23:49:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:38 smithi055 ceph-mon[97033]: pgmap v4184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:38 smithi059 ceph-mon[97450]: pgmap v4184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:38.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:38.717741+0000) 2022-09-07T23:49:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:38.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:38.628676+0000) 2022-09-07T23:49:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:39.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:39.717965+0000) 2022-09-07T23:49:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:39.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:39.628894+0000) 2022-09-07T23:49:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:40 smithi055 ceph-mon[97033]: pgmap v4185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:40 smithi059 ceph-mon[97450]: pgmap v4185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:40.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:40.718197+0000) 2022-09-07T23:49:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:40.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:40.629170+0000) 2022-09-07T23:49:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:41.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:41.718396+0000) 2022-09-07T23:49:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:41.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:41.629336+0000) 2022-09-07T23:49:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:42 smithi055 ceph-mon[97033]: pgmap v4186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:49:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:42 smithi059 ceph-mon[97450]: pgmap v4186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:49:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:42.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:42.718608+0000) 2022-09-07T23:49:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:42.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:42.629558+0000) 2022-09-07T23:49:43.912 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:43.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:43.718852+0000) 2022-09-07T23:49:43.913 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:43.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:43.629808+0000) 2022-09-07T23:49:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:44 smithi055 ceph-mon[97033]: pgmap v4187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:44 smithi059 ceph-mon[97450]: pgmap v4187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:44.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:44.719106+0000) 2022-09-07T23:49:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:44.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:44.629949+0000) 2022-09-07T23:49:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:45 smithi055 ceph-mon[97033]: pgmap v4188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:49:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:45 smithi059 ceph-mon[97450]: pgmap v4188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:49:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:45.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:45.719319+0000) 2022-09-07T23:49:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:45.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:45.630182+0000) 2022-09-07T23:49:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:46.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:46.719542+0000) 2022-09-07T23:49:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:46.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:46.630410+0000) 2022-09-07T23:49:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:47.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:47.719732+0000) 2022-09-07T23:49:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:47.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:47.630564+0000) 2022-09-07T23:49:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:48 smithi055 ceph-mon[97033]: pgmap v4189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:48 smithi059 ceph-mon[97450]: pgmap v4189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:48.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:48.719891+0000) 2022-09-07T23:49:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:48.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:48.630779+0000) 2022-09-07T23:49:49.632 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:49:49.260+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:49:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:49.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:49.720094+0000) 2022-09-07T23:49:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:49.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:49.630972+0000) 2022-09-07T23:49:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:50 smithi055 ceph-mon[97033]: pgmap v4190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:50 smithi059 ceph-mon[97450]: pgmap v4190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:50.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:50.720301+0000) 2022-09-07T23:49:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:50.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:50.631201+0000) 2022-09-07T23:49:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:51.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:51.720568+0000) 2022-09-07T23:49:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:51.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:51.631417+0000) 2022-09-07T23:49:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:52 smithi055 ceph-mon[97033]: pgmap v4191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:49:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:52 smithi059 ceph-mon[97450]: pgmap v4191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:49:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:52.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:52.720795+0000) 2022-09-07T23:49:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:52.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:52.631615+0000) 2022-09-07T23:49:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:53.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:53.721032+0000) 2022-09-07T23:49:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:53.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:53.631815+0000) 2022-09-07T23:49:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:54 smithi055 ceph-mon[97033]: pgmap v4192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:54 smithi059 ceph-mon[97450]: pgmap v4192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:54.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:54.721194+0000) 2022-09-07T23:49:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:54.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:54.632022+0000) 2022-09-07T23:49:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:55 smithi055 ceph-mon[97033]: pgmap v4193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:49:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:55 smithi059 ceph-mon[97450]: pgmap v4193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:49:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:55.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:55.721410+0000) 2022-09-07T23:49:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:55.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:55.632243+0000) 2022-09-07T23:49:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:56.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:56.721686+0000) 2022-09-07T23:49:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:56.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:56.632443+0000) 2022-09-07T23:49:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:57.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:57.721944+0000) 2022-09-07T23:49:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:57.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:57.632757+0000) 2022-09-07T23:49:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:49:58 smithi055 ceph-mon[97033]: pgmap v4194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:49:58 smithi059 ceph-mon[97450]: pgmap v4194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:49:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:58.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:58.632983+0000) 2022-09-07T23:49:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:58.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:58.722138+0000) 2022-09-07T23:49:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:49:59.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:59.722419+0000) 2022-09-07T23:49:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:49:59.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:49:59.633123+0000) 2022-09-07T23:50:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:00 smithi055 ceph-mon[97033]: pgmap v4195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-07T23:50:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T23:50:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T23:50:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:00 smithi059 ceph-mon[97450]: pgmap v4195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-07T23:50:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-07T23:50:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-07T23:50:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:00.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:00.722686+0000) 2022-09-07T23:50:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:00.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:00.633309+0000) 2022-09-07T23:50:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:01.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:01.722895+0000) 2022-09-07T23:50:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:01.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:01.633476+0000) 2022-09-07T23:50:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:02 smithi055 ceph-mon[97033]: pgmap v4196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:50:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:02 smithi059 ceph-mon[97450]: pgmap v4196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:50:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:02.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:02.633678+0000) 2022-09-07T23:50:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:02.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:02.723139+0000) 2022-09-07T23:50:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:03.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:03.723364+0000) 2022-09-07T23:50:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:03.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:03.633908+0000) 2022-09-07T23:50:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:04 smithi055 ceph-mon[97033]: pgmap v4197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:04 smithi059 ceph-mon[97450]: pgmap v4197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:04.635 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:50:04.267+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:50:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:04.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:04.634100+0000) 2022-09-07T23:50:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:04.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:04.723618+0000) 2022-09-07T23:50:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:50:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:50:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:05 smithi055 ceph-mon[97033]: pgmap v4198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:50:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:50:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:50:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:05 smithi059 ceph-mon[97450]: pgmap v4198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:50:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:05.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:05.723891+0000) 2022-09-07T23:50:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:05.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:05.634337+0000) 2022-09-07T23:50:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:06.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:06.724151+0000) 2022-09-07T23:50:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:06.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:06.634588+0000) 2022-09-07T23:50:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:07.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:07.724402+0000) 2022-09-07T23:50:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:07.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:07.634812+0000) 2022-09-07T23:50:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:08 smithi055 ceph-mon[97033]: pgmap v4199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:50:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:08 smithi059 ceph-mon[97450]: pgmap v4199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:50:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:08.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:08.724626+0000) 2022-09-07T23:50:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:08.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:08.635044+0000) 2022-09-07T23:50:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:09.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:09.724903+0000) 2022-09-07T23:50:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:09.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:09.635200+0000) 2022-09-07T23:50:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:10 smithi055 ceph-mon[97033]: pgmap v4200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:50:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:10 smithi055 ceph-mon[97033]: osdmap e351: 12 total, 11 up, 11 in 2022-09-07T23:50:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:10 smithi059 ceph-mon[97450]: pgmap v4200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:50:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:10 smithi059 ceph-mon[97450]: osdmap e351: 12 total, 11 up, 11 in 2022-09-07T23:50:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:10.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:10.725112+0000) 2022-09-07T23:50:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:10.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:10.635372+0000) 2022-09-07T23:50:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:11.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:11.725343+0000) 2022-09-07T23:50:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:11.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:11.635553+0000) 2022-09-07T23:50:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:12 smithi055 ceph-mon[97033]: pgmap v4202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:50:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:12 smithi059 ceph-mon[97450]: pgmap v4202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:50:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:12.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:12.635729+0000) 2022-09-07T23:50:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:12.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:12.725620+0000) 2022-09-07T23:50:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:13.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:13.725914+0000) 2022-09-07T23:50:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:13.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:13.635944+0000) 2022-09-07T23:50:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:14 smithi055 ceph-mon[97033]: pgmap v4203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:50:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:14 smithi059 ceph-mon[97450]: pgmap v4203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:50:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:14.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:14.726163+0000) 2022-09-07T23:50:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:14.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:14.636170+0000) 2022-09-07T23:50:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:15 smithi055 ceph-mon[97033]: osdmap e352: 12 total, 11 up, 11 in 2022-09-07T23:50:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:15 smithi055 ceph-mon[97033]: pgmap v4205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:50:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:15 smithi059 ceph-mon[97450]: osdmap e352: 12 total, 11 up, 11 in 2022-09-07T23:50:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:15 smithi059 ceph-mon[97450]: pgmap v4205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:50:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:15.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:15.726379+0000) 2022-09-07T23:50:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:15.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:15.636401+0000) 2022-09-07T23:50:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:16.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:16.726585+0000) 2022-09-07T23:50:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:16.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:16.636629+0000) 2022-09-07T23:50:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:17.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:17.636886+0000) 2022-09-07T23:50:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:17.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:17.726822+0000) 2022-09-07T23:50:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:18 smithi055 ceph-mon[97033]: pgmap v4206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:50:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:18 smithi059 ceph-mon[97450]: pgmap v4206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:50:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:18.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:18.726990+0000) 2022-09-07T23:50:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:18.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:18.637073+0000) 2022-09-07T23:50:19.638 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:50:19.271+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:50:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:19.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:19.727204+0000) 2022-09-07T23:50:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:19.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:19.637312+0000) 2022-09-07T23:50:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:20 smithi055 ceph-mon[97033]: pgmap v4207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T23:50:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:20 smithi059 ceph-mon[97450]: pgmap v4207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T23:50:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:20.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:20.727403+0000) 2022-09-07T23:50:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:20.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:20.637553+0000) 2022-09-07T23:50:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:21.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:21.727685+0000) 2022-09-07T23:50:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:21.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:21.637763+0000) 2022-09-07T23:50:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:22 smithi055 ceph-mon[97033]: pgmap v4208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:50:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:22 smithi059 ceph-mon[97450]: pgmap v4208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:50:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:22.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:22.727895+0000) 2022-09-07T23:50:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:22.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:22.637984+0000) 2022-09-07T23:50:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:23.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:23.728076+0000) 2022-09-07T23:50:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:23.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:23.638165+0000) 2022-09-07T23:50:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:24 smithi055 ceph-mon[97033]: pgmap v4209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:50:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:24 smithi059 ceph-mon[97450]: pgmap v4209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:50:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:24.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:24.728294+0000) 2022-09-07T23:50:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:24.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:24.638392+0000) 2022-09-07T23:50:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:25 smithi055 ceph-mon[97033]: pgmap v4210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-07T23:50:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:25 smithi059 ceph-mon[97450]: pgmap v4210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-07T23:50:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:25.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:25.728541+0000) 2022-09-07T23:50:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:25.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:25.638615+0000) 2022-09-07T23:50:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:26.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:26.728775+0000) 2022-09-07T23:50:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:26.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:26.638846+0000) 2022-09-07T23:50:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:27.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:27.729049+0000) 2022-09-07T23:50:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:27.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:27.639044+0000) 2022-09-07T23:50:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:28 smithi055 ceph-mon[97033]: pgmap v4211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:28 smithi059 ceph-mon[97450]: pgmap v4211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:28.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:28.729268+0000) 2022-09-07T23:50:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:28.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:28.639244+0000) 2022-09-07T23:50:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:29.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:29.729485+0000) 2022-09-07T23:50:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:29.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:29.639494+0000) 2022-09-07T23:50:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:30 smithi055 ceph-mon[97033]: pgmap v4212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:30 smithi059 ceph-mon[97450]: pgmap v4212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:30.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:30.639740+0000) 2022-09-07T23:50:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:30.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:30.729736+0000) 2022-09-07T23:50:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:31.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:31.729893+0000) 2022-09-07T23:50:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:31.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:31.639971+0000) 2022-09-07T23:50:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:32 smithi055 ceph-mon[97033]: pgmap v4213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:50:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:32 smithi059 ceph-mon[97450]: pgmap v4213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:50:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:32.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:32.640146+0000) 2022-09-07T23:50:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:32.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:32.730129+0000) 2022-09-07T23:50:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:33.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:33.730360+0000) 2022-09-07T23:50:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:33.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:33.640392+0000) 2022-09-07T23:50:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:34 smithi055 ceph-mon[97033]: pgmap v4214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:50:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:50:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:50:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:34 smithi059 ceph-mon[97450]: pgmap v4214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:50:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:50:34.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:50:34.641 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:50:34.286+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:50:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:34.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:34.640605+0000) 2022-09-07T23:50:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:34.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:34.730527+0000) 2022-09-07T23:50:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:50:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:50:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:35 smithi059 ceph-mon[97450]: pgmap v4215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:50:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:50:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:50:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:35 smithi055 ceph-mon[97033]: pgmap v4215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:50:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:35.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:35.730730+0000) 2022-09-07T23:50:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:35.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:35.640854+0000) 2022-09-07T23:50:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:36.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:36.731056+0000) 2022-09-07T23:50:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:36.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:36.641071+0000) 2022-09-07T23:50:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:37.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:37.731345+0000) 2022-09-07T23:50:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:37.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:37.641310+0000) 2022-09-07T23:50:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:38 smithi055 ceph-mon[97033]: pgmap v4216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:38 smithi059 ceph-mon[97450]: pgmap v4216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:38.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:38.641537+0000) 2022-09-07T23:50:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:38.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:38.731549+0000) 2022-09-07T23:50:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:39.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:39.641763+0000) 2022-09-07T23:50:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:39.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:39.731787+0000) 2022-09-07T23:50:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:40 smithi055 ceph-mon[97033]: pgmap v4217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:40 smithi059 ceph-mon[97450]: pgmap v4217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:40.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:40.731942+0000) 2022-09-07T23:50:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:40.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:40.641936+0000) 2022-09-07T23:50:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:41.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:41.732151+0000) 2022-09-07T23:50:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:41.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:41.642169+0000) 2022-09-07T23:50:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:42 smithi055 ceph-mon[97033]: pgmap v4218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:50:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:42 smithi059 ceph-mon[97450]: pgmap v4218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:50:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:42.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:42.732348+0000) 2022-09-07T23:50:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:42.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:42.642383+0000) 2022-09-07T23:50:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:43.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:43.732568+0000) 2022-09-07T23:50:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:43.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:43.642590+0000) 2022-09-07T23:50:44.491 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:44 smithi055 ceph-mon[97033]: pgmap v4219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:44 smithi059 ceph-mon[97450]: pgmap v4219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:44.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:44.732758+0000) 2022-09-07T23:50:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:44.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:44.642899+0000) 2022-09-07T23:50:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:45 smithi055 ceph-mon[97033]: pgmap v4220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:50:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:45 smithi059 ceph-mon[97450]: pgmap v4220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:50:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:45.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:45.732907+0000) 2022-09-07T23:50:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:45.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:45.643146+0000) 2022-09-07T23:50:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:46.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:46.733086+0000) 2022-09-07T23:50:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:46.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:46.643342+0000) 2022-09-07T23:50:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:47.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:47.733337+0000) 2022-09-07T23:50:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:47.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:47.643583+0000) 2022-09-07T23:50:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:48 smithi055 ceph-mon[97033]: pgmap v4221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:48 smithi059 ceph-mon[97450]: pgmap v4221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:48.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:48.733590+0000) 2022-09-07T23:50:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:48.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:48.643775+0000) 2022-09-07T23:50:49.645 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:50:49.289+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:50:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:49.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:49.733818+0000) 2022-09-07T23:50:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:49.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:49.643953+0000) 2022-09-07T23:50:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:50 smithi055 ceph-mon[97033]: pgmap v4222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:50 smithi059 ceph-mon[97450]: pgmap v4222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:50.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:50.734101+0000) 2022-09-07T23:50:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:50.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:50.644201+0000) 2022-09-07T23:50:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:51.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:51.734368+0000) 2022-09-07T23:50:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:51.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:51.644448+0000) 2022-09-07T23:50:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:52 smithi055 ceph-mon[97033]: pgmap v4223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:50:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:52 smithi059 ceph-mon[97450]: pgmap v4223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:50:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:52.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:52.734563+0000) 2022-09-07T23:50:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:52.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:52.644612+0000) 2022-09-07T23:50:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:53.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:53.734782+0000) 2022-09-07T23:50:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:53.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:53.644811+0000) 2022-09-07T23:50:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:54 smithi055 ceph-mon[97033]: pgmap v4224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:54.538 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:54 smithi059 ceph-mon[97450]: pgmap v4224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:54.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:54.734928+0000) 2022-09-07T23:50:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:54.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:54.645048+0000) 2022-09-07T23:50:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:55 smithi055 ceph-mon[97033]: pgmap v4225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:50:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:55 smithi059 ceph-mon[97450]: pgmap v4225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:50:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:55.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:55.735089+0000) 2022-09-07T23:50:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:55.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:55.645290+0000) 2022-09-07T23:50:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:56.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:56.735325+0000) 2022-09-07T23:50:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:56.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:56.645529+0000) 2022-09-07T23:50:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:57.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:57.735578+0000) 2022-09-07T23:50:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:57.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:57.645769+0000) 2022-09-07T23:50:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:50:58 smithi055 ceph-mon[97033]: pgmap v4226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:50:58 smithi059 ceph-mon[97450]: pgmap v4226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:50:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:58.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:58.735785+0000) 2022-09-07T23:50:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:58.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:58.645913+0000) 2022-09-07T23:50:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:50:59.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:59.735990+0000) 2022-09-07T23:50:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:50:59.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:50:59.646137+0000) 2022-09-07T23:51:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:00 smithi055 ceph-mon[97033]: pgmap v4227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:00 smithi059 ceph-mon[97450]: pgmap v4227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:00.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:00.736191+0000) 2022-09-07T23:51:00.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:00.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:00.646376+0000) 2022-09-07T23:51:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:01.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:01.736438+0000) 2022-09-07T23:51:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:01.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:01.646581+0000) 2022-09-07T23:51:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:02 smithi055 ceph-mon[97033]: pgmap v4228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:51:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:02 smithi059 ceph-mon[97450]: pgmap v4228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:51:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:02.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:02.736754+0000) 2022-09-07T23:51:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:02.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:02.646803+0000) 2022-09-07T23:51:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:03.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:03.736941+0000) 2022-09-07T23:51:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:03.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:03.647056+0000) 2022-09-07T23:51:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:04 smithi055 ceph-mon[97033]: pgmap v4229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:51:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:04 smithi059 ceph-mon[97450]: pgmap v4229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:51:04.648 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:51:04.298+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:51:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:04.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:04.737087+0000) 2022-09-07T23:51:04.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:04.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:04.647298+0000) 2022-09-07T23:51:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:05 smithi055 ceph-mon[97033]: pgmap v4230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:51:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:05 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-07T23:51:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:51:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:51:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:05 smithi059 ceph-mon[97450]: pgmap v4230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-07T23:51:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:05 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-07T23:51:05.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:51:05.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:51:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:05.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:05.737277+0000) 2022-09-07T23:51:05.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:05.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:05.647523+0000) 2022-09-07T23:51:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:06.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:06.647799+0000) 2022-09-07T23:51:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:06.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:06.737524+0000) 2022-09-07T23:51:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:07 smithi055 ceph-mon[97033]: pgmap v4231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 49 KiB/s wr, 21 op/s 2022-09-07T23:51:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:07 smithi059 ceph-mon[97450]: pgmap v4231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 49 KiB/s wr, 21 op/s 2022-09-07T23:51:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:07.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:07.737766+0000) 2022-09-07T23:51:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:07.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:07.647950+0000) 2022-09-07T23:51:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:08.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:08.737943+0000) 2022-09-07T23:51:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:08.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:08.648115+0000) 2022-09-07T23:51:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:09 smithi055 ceph-mon[97033]: pgmap v4232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 49 KiB/s wr, 21 op/s 2022-09-07T23:51:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:09 smithi059 ceph-mon[97450]: pgmap v4232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 49 KiB/s wr, 21 op/s 2022-09-07T23:51:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:09.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:09.738079+0000) 2022-09-07T23:51:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:09.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:09.648269+0000) 2022-09-07T23:51:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:10 smithi055 ceph-mon[97033]: osdmap e353: 12 total, 11 up, 11 in 2022-09-07T23:51:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:10 smithi059 ceph-mon[97450]: osdmap e353: 12 total, 11 up, 11 in 2022-09-07T23:51:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:10.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:10.738313+0000) 2022-09-07T23:51:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:10.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:10.648503+0000) 2022-09-07T23:51:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:11 smithi055 ceph-mon[97033]: pgmap v4234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 60 KiB/s wr, 25 op/s 2022-09-07T23:51:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:11 smithi059 ceph-mon[97450]: pgmap v4234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 60 KiB/s wr, 25 op/s 2022-09-07T23:51:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:11.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:11.738591+0000) 2022-09-07T23:51:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:11.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:11.648734+0000) 2022-09-07T23:51:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:12.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:12.738842+0000) 2022-09-07T23:51:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:12.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:12.648889+0000) 2022-09-07T23:51:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:13 smithi055 ceph-mon[97033]: pgmap v4235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 60 KiB/s wr, 27 op/s 2022-09-07T23:51:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:13 smithi059 ceph-mon[97450]: pgmap v4235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 60 KiB/s wr, 27 op/s 2022-09-07T23:51:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:13.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:13.739005+0000) 2022-09-07T23:51:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:13.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:13.649069+0000) 2022-09-07T23:51:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:14.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:14.739134+0000) 2022-09-07T23:51:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:14.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:14.649273+0000) 2022-09-07T23:51:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:15 smithi055 ceph-mon[97033]: pgmap v4236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:51:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:15 smithi055 ceph-mon[97033]: osdmap e354: 12 total, 11 up, 11 in 2022-09-07T23:51:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:15 smithi059 ceph-mon[97450]: pgmap v4236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:51:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:15 smithi059 ceph-mon[97450]: osdmap e354: 12 total, 11 up, 11 in 2022-09-07T23:51:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:15.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:15.739354+0000) 2022-09-07T23:51:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:15.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:15.649532+0000) 2022-09-07T23:51:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:16.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:16.739597+0000) 2022-09-07T23:51:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:16.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:16.649798+0000) 2022-09-07T23:51:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:17 smithi055 ceph-mon[97033]: pgmap v4238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:51:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:17 smithi059 ceph-mon[97450]: pgmap v4238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:51:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:17.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:17.739804+0000) 2022-09-07T23:51:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:17.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:17.650063+0000) 2022-09-07T23:51:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:18.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:18.740029+0000) 2022-09-07T23:51:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:18.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:18.650210+0000) 2022-09-07T23:51:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:19 smithi055 ceph-mon[97033]: pgmap v4239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-07T23:51:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:19 smithi059 ceph-mon[97450]: pgmap v4239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-07T23:51:19.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:51:19.300+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:51:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:19.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:19.740287+0000) 2022-09-07T23:51:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:19.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:19.650435+0000) 2022-09-07T23:51:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:20 smithi055 ceph-mon[97033]: pgmap v4240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:51:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:20 smithi059 ceph-mon[97450]: pgmap v4240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:51:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:20.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:20.740429+0000) 2022-09-07T23:51:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:20.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:20.650677+0000) 2022-09-07T23:51:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:21.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:21.740630+0000) 2022-09-07T23:51:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:21.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:21.650897+0000) 2022-09-07T23:51:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:22.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:22.740877+0000) 2022-09-07T23:51:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:22.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:22.651148+0000) 2022-09-07T23:51:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:22 smithi055 ceph-mon[97033]: pgmap v4241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:51:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:22 smithi059 ceph-mon[97450]: pgmap v4241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:51:23.977 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:23.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:23.741102+0000) 2022-09-07T23:51:23.978 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:23.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:23.651333+0000) 2022-09-07T23:51:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:24.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:24.741323+0000) 2022-09-07T23:51:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:24.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:24.651506+0000) 2022-09-07T23:51:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:24 smithi059 ceph-mon[97450]: pgmap v4242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:51:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:24 smithi055 ceph-mon[97033]: pgmap v4242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:51:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:25.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:25.651730+0000) 2022-09-07T23:51:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:25.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:25.741534+0000) 2022-09-07T23:51:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:26.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:26.741735+0000) 2022-09-07T23:51:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:26.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:26.651899+0000) 2022-09-07T23:51:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:26 smithi059 ceph-mon[97450]: pgmap v4243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-07T23:51:27.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:26 smithi055 ceph-mon[97033]: pgmap v4243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-07T23:51:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:27.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:27.741972+0000) 2022-09-07T23:51:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:27.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:27.652132+0000) 2022-09-07T23:51:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:28.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:28.742184+0000) 2022-09-07T23:51:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:28.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:28.652315+0000) 2022-09-07T23:51:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:28 smithi059 ceph-mon[97450]: pgmap v4244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:29.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:28 smithi055 ceph-mon[97033]: pgmap v4244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:29.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:29.742412+0000) 2022-09-07T23:51:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:29.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:29.652545+0000) 2022-09-07T23:51:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:30 smithi055 ceph-mon[97033]: pgmap v4245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:51:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:30 smithi059 ceph-mon[97450]: pgmap v4245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:51:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:30.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:30.742733+0000) 2022-09-07T23:51:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:30.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:30.652775+0000) 2022-09-07T23:51:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:31.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:31.742965+0000) 2022-09-07T23:51:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:31.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:31.652957+0000) 2022-09-07T23:51:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:32.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:32.743175+0000) 2022-09-07T23:51:33.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:32.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:32.653182+0000) 2022-09-07T23:51:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:32 smithi055 ceph-mon[97033]: pgmap v4246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:51:33.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:32 smithi059 ceph-mon[97450]: pgmap v4246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:51:34.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:33.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:33.743405+0000) 2022-09-07T23:51:34.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:33.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:33.653369+0000) 2022-09-07T23:51:34.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:51:34.302+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:51:35.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:34.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:34.743677+0000) 2022-09-07T23:51:35.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:34.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:34.653535+0000) 2022-09-07T23:51:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:34 smithi059 ceph-mon[97450]: pgmap v4247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:34 smithi055 ceph-mon[97033]: pgmap v4247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:35.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:35.743925+0000) 2022-09-07T23:51:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:35.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:35.653745+0000) 2022-09-07T23:51:37.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:36.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:36.744169+0000) 2022-09-07T23:51:37.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:36.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:36.653910+0000) 2022-09-07T23:51:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:36 smithi059 ceph-mon[97450]: pgmap v4248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:51:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:36 smithi055 ceph-mon[97033]: pgmap v4248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:51:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:37.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:37.744394+0000) 2022-09-07T23:51:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:37.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:37.654137+0000) 2022-09-07T23:51:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:38.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:38.744601+0000) 2022-09-07T23:51:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:38.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:38.654368+0000) 2022-09-07T23:51:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:38 smithi059 ceph-mon[97450]: pgmap v4249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:38 smithi055 ceph-mon[97033]: pgmap v4249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:39.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:39.744834+0000) 2022-09-07T23:51:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:39.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:39.654593+0000) 2022-09-07T23:51:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:40 smithi055 ceph-mon[97033]: pgmap v4250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:51:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:40 smithi059 ceph-mon[97450]: pgmap v4250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:51:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:40.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:40.745024+0000) 2022-09-07T23:51:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:40.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:40.654832+0000) 2022-09-07T23:51:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:41.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:41.745255+0000) 2022-09-07T23:51:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:41.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:41.655058+0000) 2022-09-07T23:51:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:42.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:42.745449+0000) 2022-09-07T23:51:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:42.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:42.655248+0000) 2022-09-07T23:51:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:42 smithi059 ceph-mon[97450]: pgmap v4251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:51:43.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:42 smithi055 ceph-mon[97033]: pgmap v4251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:51:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:43.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:43.745715+0000) 2022-09-07T23:51:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:43.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:43.655480+0000) 2022-09-07T23:51:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:44.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:44.745849+0000) 2022-09-07T23:51:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:44.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:44.655716+0000) 2022-09-07T23:51:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:44 smithi059 ceph-mon[97450]: pgmap v4252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:44 smithi055 ceph-mon[97033]: pgmap v4252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:45.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:45.746015+0000) 2022-09-07T23:51:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:45.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:45.655952+0000) 2022-09-07T23:51:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:46.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:46.746238+0000) 2022-09-07T23:51:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:46.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:46.656116+0000) 2022-09-07T23:51:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:46 smithi059 ceph-mon[97450]: pgmap v4253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:51:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:46 smithi055 ceph-mon[97033]: pgmap v4253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:51:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:47.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:47.746450+0000) 2022-09-07T23:51:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:47.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:47.656310+0000) 2022-09-07T23:51:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:48.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:48.656496+0000) 2022-09-07T23:51:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:48.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:48.746696+0000) 2022-09-07T23:51:49.289 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:48 smithi059 ceph-mon[97450]: pgmap v4254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:48 smithi055 ceph-mon[97033]: pgmap v4254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:49.667 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:51:49.305+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:51:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:49.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:49.656729+0000) 2022-09-07T23:51:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:49.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:49.746932+0000) 2022-09-07T23:51:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:50 smithi055 ceph-mon[97033]: pgmap v4255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:51:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:50 smithi059 ceph-mon[97450]: pgmap v4255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:51:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:50.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:50.656923+0000) 2022-09-07T23:51:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:50.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:50.747155+0000) 2022-09-07T23:51:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:51.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:51.747399+0000) 2022-09-07T23:51:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:51.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:51.657148+0000) 2022-09-07T23:51:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:52.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:52.747632+0000) 2022-09-07T23:51:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:52.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:52.657409+0000) 2022-09-07T23:51:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:52 smithi059 ceph-mon[97450]: pgmap v4256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:51:53.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:52 smithi055 ceph-mon[97033]: pgmap v4256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:51:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:53.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:53.747857+0000) 2022-09-07T23:51:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:53.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:53.657648+0000) 2022-09-07T23:51:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:54.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:54.748002+0000) 2022-09-07T23:51:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:54.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:54.657824+0000) 2022-09-07T23:51:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:54 smithi059 ceph-mon[97450]: pgmap v4257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:54 smithi055 ceph-mon[97033]: pgmap v4257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:55.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:55.748220+0000) 2022-09-07T23:51:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:55.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:55.658078+0000) 2022-09-07T23:51:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:56.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:56.748458+0000) 2022-09-07T23:51:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:56.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:56.658274+0000) 2022-09-07T23:51:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:56 smithi059 ceph-mon[97450]: pgmap v4258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:51:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:56 smithi055 ceph-mon[97033]: pgmap v4258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:51:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:57.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:57.748716+0000) 2022-09-07T23:51:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:57.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:57.658456+0000) 2022-09-07T23:51:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:58.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:58.748944+0000) 2022-09-07T23:51:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:58.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:58.658738+0000) 2022-09-07T23:51:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:51:58 smithi059 ceph-mon[97450]: pgmap v4259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:51:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:51:58 smithi055 ceph-mon[97033]: pgmap v4259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:51:59.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:59.749086+0000) 2022-09-07T23:52:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:51:59.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:51:59.658948+0000) 2022-09-07T23:52:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:00 smithi055 ceph-mon[97033]: pgmap v4260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:52:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:00 smithi059 ceph-mon[97450]: pgmap v4260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:52:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:00.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:00.749281+0000) 2022-09-07T23:52:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:00.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:00.659082+0000) 2022-09-07T23:52:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:01.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:01.659351+0000) 2022-09-07T23:52:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:01.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:01.749420+0000) 2022-09-07T23:52:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:02.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:02.749674+0000) 2022-09-07T23:52:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:02.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:02.659607+0000) 2022-09-07T23:52:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:02 smithi059 ceph-mon[97450]: pgmap v4261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:02 smithi055 ceph-mon[97033]: pgmap v4261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:04.067 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:03.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:03.749881+0000) 2022-09-07T23:52:04.068 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:03.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:03.659837+0000) 2022-09-07T23:52:04.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:52:04.320+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:52:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:04.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:04.750116+0000) 2022-09-07T23:52:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:04.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:04.660013+0000) 2022-09-07T23:52:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:04 smithi059 ceph-mon[97450]: pgmap v4262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:52:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:52:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:52:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:52:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:52:05.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:04 smithi055 ceph-mon[97033]: pgmap v4262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:05.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:52:05.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:52:05.487 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:52:05.488 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:52:05.488 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:52:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:05.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:05.750332+0000) 2022-09-07T23:52:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:05.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:05.660188+0000) 2022-09-07T23:52:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:06.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:06.750491+0000) 2022-09-07T23:52:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:06.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:06.660366+0000) 2022-09-07T23:52:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:06 smithi059 ceph-mon[97450]: pgmap v4263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:52:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:06 smithi055 ceph-mon[97033]: pgmap v4263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:52:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:07.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:07.750741+0000) 2022-09-07T23:52:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:07.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:07.660668+0000) 2022-09-07T23:52:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:08.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:08.660904+0000) 2022-09-07T23:52:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:08.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:08.750923+0000) 2022-09-07T23:52:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:09 smithi059 ceph-mon[97450]: pgmap v4264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:52:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:09 smithi055 ceph-mon[97033]: pgmap v4264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:52:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:09.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:09.751116+0000) 2022-09-07T23:52:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:09.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:09.661044+0000) 2022-09-07T23:52:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:10 smithi059 ceph-mon[97450]: osdmap e355: 12 total, 11 up, 11 in 2022-09-07T23:52:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:10 smithi055 ceph-mon[97033]: osdmap e355: 12 total, 11 up, 11 in 2022-09-07T23:52:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:10.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:10.751374+0000) 2022-09-07T23:52:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:10.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:10.661292+0000) 2022-09-07T23:52:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:11 smithi059 ceph-mon[97450]: pgmap v4266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:52:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:11 smithi055 ceph-mon[97033]: pgmap v4266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:52:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:11.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:11.751567+0000) 2022-09-07T23:52:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:11.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:11.661529+0000) 2022-09-07T23:52:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:12.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:12.751755+0000) 2022-09-07T23:52:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:12.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:12.661758+0000) 2022-09-07T23:52:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:13 smithi059 ceph-mon[97450]: pgmap v4267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:52:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:13 smithi055 ceph-mon[97033]: pgmap v4267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:52:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:13.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:13.751906+0000) 2022-09-07T23:52:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:13.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:13.661963+0000) 2022-09-07T23:52:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:14.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:14.752136+0000) 2022-09-07T23:52:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:14.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:14.662151+0000) 2022-09-07T23:52:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:15 smithi059 ceph-mon[97450]: pgmap v4268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:52:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:15 smithi059 ceph-mon[97450]: osdmap e356: 12 total, 11 up, 11 in 2022-09-07T23:52:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:15 smithi055 ceph-mon[97033]: pgmap v4268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:52:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:15 smithi055 ceph-mon[97033]: osdmap e356: 12 total, 11 up, 11 in 2022-09-07T23:52:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:15.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:15.752356+0000) 2022-09-07T23:52:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:15.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:15.662381+0000) 2022-09-07T23:52:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:16.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:16.752490+0000) 2022-09-07T23:52:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:16.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:16.662554+0000) 2022-09-07T23:52:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:17 smithi059 ceph-mon[97450]: pgmap v4270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:52:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:17 smithi055 ceph-mon[97033]: pgmap v4270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:52:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:17.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:17.752697+0000) 2022-09-07T23:52:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:17.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:17.662781+0000) 2022-09-07T23:52:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:18.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:18.752877+0000) 2022-09-07T23:52:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:18.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:18.662973+0000) 2022-09-07T23:52:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:19 smithi059 ceph-mon[97450]: pgmap v4271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:52:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:19 smithi055 ceph-mon[97033]: pgmap v4271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:52:19.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:52:19.332+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:52:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:19.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:19.753118+0000) 2022-09-07T23:52:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:19.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:19.663108+0000) 2022-09-07T23:52:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:20 smithi055 ceph-mon[97033]: pgmap v4272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:52:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:20 smithi059 ceph-mon[97450]: pgmap v4272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:52:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:20.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:20.753375+0000) 2022-09-07T23:52:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:20.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:20.663280+0000) 2022-09-07T23:52:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:21.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:21.753627+0000) 2022-09-07T23:52:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:21.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:21.663411+0000) 2022-09-07T23:52:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:22.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:22.753853+0000) 2022-09-07T23:52:23.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:22.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:22.663628+0000) 2022-09-07T23:52:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:22 smithi059 ceph-mon[97450]: pgmap v4273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:52:23.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:22 smithi055 ceph-mon[97033]: pgmap v4273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:52:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:23.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:23.753977+0000) 2022-09-07T23:52:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:23.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:23.663882+0000) 2022-09-07T23:52:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:24.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:24.754216+0000) 2022-09-07T23:52:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:24.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:24.664076+0000) 2022-09-07T23:52:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:24 smithi059 ceph-mon[97450]: pgmap v4274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:52:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:24 smithi055 ceph-mon[97033]: pgmap v4274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:52:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:25.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:25.754479+0000) 2022-09-07T23:52:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:25.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:25.664272+0000) 2022-09-07T23:52:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:26.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:26.754743+0000) 2022-09-07T23:52:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:26.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:26.664455+0000) 2022-09-07T23:52:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:27 smithi059 ceph-mon[97450]: pgmap v4275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-07T23:52:27.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:27 smithi055 ceph-mon[97033]: pgmap v4275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-07T23:52:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:27.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:27.754965+0000) 2022-09-07T23:52:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:27.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:27.664658+0000) 2022-09-07T23:52:29.121 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:28.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:28.755206+0000) 2022-09-07T23:52:29.122 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:28.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:28.664909+0000) 2022-09-07T23:52:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:29 smithi059 ceph-mon[97450]: pgmap v4276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:29.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:29 smithi055 ceph-mon[97033]: pgmap v4276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:29.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:29.665078+0000) 2022-09-07T23:52:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:29.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:29.755415+0000) 2022-09-07T23:52:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:30 smithi055 ceph-mon[97033]: pgmap v4277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:52:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:30 smithi059 ceph-mon[97450]: pgmap v4277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:52:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:30.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:30.755624+0000) 2022-09-07T23:52:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:30.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:30.665287+0000) 2022-09-07T23:52:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:31.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:31.755816+0000) 2022-09-07T23:52:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:31.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:31.665521+0000) 2022-09-07T23:52:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:32.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:32.756047+0000) 2022-09-07T23:52:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:32.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:32.665736+0000) 2022-09-07T23:52:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:32 smithi059 ceph-mon[97450]: pgmap v4278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:33.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:33 smithi055 ceph-mon[97033]: pgmap v4278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:33.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:33.756277+0000) 2022-09-07T23:52:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:33.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:33.665975+0000) 2022-09-07T23:52:34.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:52:34.348+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:52:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:34.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:34.756506+0000) 2022-09-07T23:52:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:34.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:34.666147+0000) 2022-09-07T23:52:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:35 smithi055 ceph-mon[97033]: pgmap v4279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:35 smithi055 ceph-mon[97033]: pgmap v4280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:52:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:35 smithi055 ceph-mon[97033]: pgmap v4281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:52:35.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:52:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:35 smithi059 ceph-mon[97450]: pgmap v4279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:35 smithi059 ceph-mon[97450]: pgmap v4280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:52:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:35 smithi059 ceph-mon[97450]: pgmap v4281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:52:35.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:52:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:35.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:35.756667+0000) 2022-09-07T23:52:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:35.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:35.666372+0000) 2022-09-07T23:52:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:36.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:36.666522+0000) 2022-09-07T23:52:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:36.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:36.756830+0000) 2022-09-07T23:52:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:37 smithi055 ceph-mon[97033]: pgmap v4282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:52:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:37 smithi059 ceph-mon[97450]: pgmap v4282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:52:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:37.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:37.757048+0000) 2022-09-07T23:52:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:37.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:37.666693+0000) 2022-09-07T23:52:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:38.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:38.666857+0000) 2022-09-07T23:52:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:38.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:38.757290+0000) 2022-09-07T23:52:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:39 smithi055 ceph-mon[97033]: pgmap v4283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:52:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:39 smithi059 ceph-mon[97450]: pgmap v4283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:52:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:39.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:39.757494+0000) 2022-09-07T23:52:40.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:39.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:39.667069+0000) 2022-09-07T23:52:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:40 smithi055 ceph-mon[97033]: pgmap v4284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:52:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:40 smithi059 ceph-mon[97450]: pgmap v4284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:52:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:40.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:40.757742+0000) 2022-09-07T23:52:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:40.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:40.667321+0000) 2022-09-07T23:52:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:41.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:41.667548+0000) 2022-09-07T23:52:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:41.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:41.757881+0000) 2022-09-07T23:52:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:42.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:42.667793+0000) 2022-09-07T23:52:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:42.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:42.758105+0000) 2022-09-07T23:52:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:43 smithi059 ceph-mon[97450]: pgmap v4285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:52:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:43 smithi055 ceph-mon[97033]: pgmap v4285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:52:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:43.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:43.668035+0000) 2022-09-07T23:52:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:43.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:43.758274+0000) 2022-09-07T23:52:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:44.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:44.758483+0000) 2022-09-07T23:52:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:44.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:44.668257+0000) 2022-09-07T23:52:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:45 smithi059 ceph-mon[97450]: pgmap v4286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:52:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:45 smithi055 ceph-mon[97033]: pgmap v4286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:52:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:45.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:45.758616+0000) 2022-09-07T23:52:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:45.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:45.668510+0000) 2022-09-07T23:52:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:46.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:46.668752+0000) 2022-09-07T23:52:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:46.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:46.758840+0000) 2022-09-07T23:52:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:47 smithi059 ceph-mon[97450]: pgmap v4287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:52:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:47 smithi055 ceph-mon[97033]: pgmap v4287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:52:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:47.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:47.759037+0000) 2022-09-07T23:52:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:47.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:47.668998+0000) 2022-09-07T23:52:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:48.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:48.759273+0000) 2022-09-07T23:52:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:48.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:48.669236+0000) 2022-09-07T23:52:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:49 smithi059 ceph-mon[97450]: pgmap v4288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:49.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:52:49.350+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:52:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:49 smithi055 ceph-mon[97033]: pgmap v4288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:49.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:49.759490+0000) 2022-09-07T23:52:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:49.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:49.669462+0000) 2022-09-07T23:52:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:50 smithi055 ceph-mon[97033]: pgmap v4289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:52:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:50 smithi059 ceph-mon[97450]: pgmap v4289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:52:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:50.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:50.759714+0000) 2022-09-07T23:52:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:50.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:50.669603+0000) 2022-09-07T23:52:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:51.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:51.759933+0000) 2022-09-07T23:52:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:51.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:51.669795+0000) 2022-09-07T23:52:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:52.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:52.760090+0000) 2022-09-07T23:52:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:52.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:52.669977+0000) 2022-09-07T23:52:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:53 smithi059 ceph-mon[97450]: pgmap v4290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:53 smithi055 ceph-mon[97033]: pgmap v4290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:53.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:53.760319+0000) 2022-09-07T23:52:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:53.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:53.670153+0000) 2022-09-07T23:52:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:54.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:54.760528+0000) 2022-09-07T23:52:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:54.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:54.670321+0000) 2022-09-07T23:52:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:55 smithi059 ceph-mon[97450]: pgmap v4291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:55 smithi055 ceph-mon[97033]: pgmap v4291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:55.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:55.760750+0000) 2022-09-07T23:52:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:55.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:55.670549+0000) 2022-09-07T23:52:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:56.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:56.760975+0000) 2022-09-07T23:52:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:56.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:56.670781+0000) 2022-09-07T23:52:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:57 smithi059 ceph-mon[97450]: pgmap v4292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:52:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:57 smithi055 ceph-mon[97033]: pgmap v4292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:52:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:57.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:57.761223+0000) 2022-09-07T23:52:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:57.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:57.671037+0000) 2022-09-07T23:52:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:58.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:58.761508+0000) 2022-09-07T23:52:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:58.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:58.671284+0000) 2022-09-07T23:52:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:52:59 smithi059 ceph-mon[97450]: pgmap v4293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:52:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:52:59 smithi055 ceph-mon[97033]: pgmap v4293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:52:59.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:59.761778+0000) 2022-09-07T23:53:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:52:59.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:52:59.671518+0000) 2022-09-07T23:53:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:00 smithi055 ceph-mon[97033]: pgmap v4294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:53:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:00 smithi059 ceph-mon[97450]: pgmap v4294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:53:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:00.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:00.761933+0000) 2022-09-07T23:53:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:00.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:00.671688+0000) 2022-09-07T23:53:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:01.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:01.762207+0000) 2022-09-07T23:53:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:01.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:01.671923+0000) 2022-09-07T23:53:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:02.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:02.762464+0000) 2022-09-07T23:53:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:02.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:02.672175+0000) 2022-09-07T23:53:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:03 smithi059 ceph-mon[97450]: pgmap v4295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:03.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:03 smithi055 ceph-mon[97033]: pgmap v4295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:03.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:03.762716+0000) 2022-09-07T23:53:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:03.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:03.672403+0000) 2022-09-07T23:53:04.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:53:04.361+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:53:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:04.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:04.762877+0000) 2022-09-07T23:53:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:04.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:04.672607+0000) 2022-09-07T23:53:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:05 smithi059 ceph-mon[97450]: pgmap v4296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:53:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:53:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:05 smithi055 ceph-mon[97033]: pgmap v4296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:53:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:53:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:05.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:05.763073+0000) 2022-09-07T23:53:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:05.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:05.672872+0000) 2022-09-07T23:53:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:06.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:06.763244+0000) 2022-09-07T23:53:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:06.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:06.673097+0000) 2022-09-07T23:53:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:07 smithi059 ceph-mon[97450]: pgmap v4297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:53:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:07 smithi055 ceph-mon[97033]: pgmap v4297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:53:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:07.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:07.763491+0000) 2022-09-07T23:53:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:07.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:07.673339+0000) 2022-09-07T23:53:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:08.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:08.673541+0000) 2022-09-07T23:53:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:08.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:08.763718+0000) 2022-09-07T23:53:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:09 smithi059 ceph-mon[97450]: pgmap v4298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:53:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:09 smithi055 ceph-mon[97033]: pgmap v4298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:53:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:09.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:09.763849+0000) 2022-09-07T23:53:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:09.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:09.673730+0000) 2022-09-07T23:53:10.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:10 smithi059 ceph-mon[97450]: osdmap e357: 12 total, 11 up, 11 in 2022-09-07T23:53:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:10 smithi055 ceph-mon[97033]: osdmap e357: 12 total, 11 up, 11 in 2022-09-07T23:53:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:10.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:10.764057+0000) 2022-09-07T23:53:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:10.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:10.673953+0000) 2022-09-07T23:53:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:11 smithi059 ceph-mon[97450]: pgmap v4300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:53:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:11 smithi055 ceph-mon[97033]: pgmap v4300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:53:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:11.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:11.764277+0000) 2022-09-07T23:53:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:11.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:11.674207+0000) 2022-09-07T23:53:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:12.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:12.764566+0000) 2022-09-07T23:53:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:12.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:12.674470+0000) 2022-09-07T23:53:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:13 smithi059 ceph-mon[97450]: pgmap v4301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:53:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:13 smithi055 ceph-mon[97033]: pgmap v4301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:53:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:13.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:13.764820+0000) 2022-09-07T23:53:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:13.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:13.674618+0000) 2022-09-07T23:53:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:14.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:14.765068+0000) 2022-09-07T23:53:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:14.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:14.674834+0000) 2022-09-07T23:53:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:15 smithi059 ceph-mon[97450]: pgmap v4302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:53:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:15 smithi059 ceph-mon[97450]: osdmap e358: 12 total, 11 up, 11 in 2022-09-07T23:53:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:15 smithi055 ceph-mon[97033]: pgmap v4302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:53:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:15 smithi055 ceph-mon[97033]: osdmap e358: 12 total, 11 up, 11 in 2022-09-07T23:53:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:15.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:15.765367+0000) 2022-09-07T23:53:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:15.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:15.675020+0000) 2022-09-07T23:53:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:16.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:16.765604+0000) 2022-09-07T23:53:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:16.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:16.675194+0000) 2022-09-07T23:53:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:17 smithi059 ceph-mon[97450]: pgmap v4304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:53:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:17 smithi055 ceph-mon[97033]: pgmap v4304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:53:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:17.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:17.765901+0000) 2022-09-07T23:53:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:17.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:17.675444+0000) 2022-09-07T23:53:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:18.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:18.675736+0000) 2022-09-07T23:53:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:18.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:18.766131+0000) 2022-09-07T23:53:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:19 smithi059 ceph-mon[97450]: pgmap v4305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:53:19.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:53:19.373+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:53:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:19 smithi055 ceph-mon[97033]: pgmap v4305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:53:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:19.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:19.675967+0000) 2022-09-07T23:53:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:19.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:19.766408+0000) 2022-09-07T23:53:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:20 smithi059 ceph-mon[97450]: pgmap v4306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:53:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:20 smithi055 ceph-mon[97033]: pgmap v4306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:53:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:20.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:20.676212+0000) 2022-09-07T23:53:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:20.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:20.766652+0000) 2022-09-07T23:53:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:21.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:21.766874+0000) 2022-09-07T23:53:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:21.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:21.676425+0000) 2022-09-07T23:53:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:22.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:22.767120+0000) 2022-09-07T23:53:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:22.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:22.676619+0000) 2022-09-07T23:53:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:23 smithi059 ceph-mon[97450]: pgmap v4307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:53:23.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:23 smithi055 ceph-mon[97033]: pgmap v4307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:53:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:23.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:23.767348+0000) 2022-09-07T23:53:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:23.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:23.676856+0000) 2022-09-07T23:53:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:24.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:24.767544+0000) 2022-09-07T23:53:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:24.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:24.677067+0000) 2022-09-07T23:53:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:25 smithi059 ceph-mon[97450]: pgmap v4308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:53:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:25 smithi055 ceph-mon[97033]: pgmap v4308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:53:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:25.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:25.767807+0000) 2022-09-07T23:53:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:25.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:25.677261+0000) 2022-09-07T23:53:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:26.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:26.768067+0000) 2022-09-07T23:53:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:26.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:26.677478+0000) 2022-09-07T23:53:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:27 smithi059 ceph-mon[97450]: pgmap v4309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:53:27.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:27 smithi055 ceph-mon[97033]: pgmap v4309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-07T23:53:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:27.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:27.768303+0000) 2022-09-07T23:53:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:27.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:27.677747+0000) 2022-09-07T23:53:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:28.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:28.768513+0000) 2022-09-07T23:53:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:28.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:28.677996+0000) 2022-09-07T23:53:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:29 smithi059 ceph-mon[97450]: pgmap v4310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:29 smithi055 ceph-mon[97033]: pgmap v4310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:29.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:29.768660+0000) 2022-09-07T23:53:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:29.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:29.678213+0000) 2022-09-07T23:53:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:30 smithi059 ceph-mon[97450]: pgmap v4311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:53:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:30 smithi055 ceph-mon[97033]: pgmap v4311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:53:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:30.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:30.768791+0000) 2022-09-07T23:53:31.170 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:30.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:30.678483+0000) 2022-09-07T23:53:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:31.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:31.768993+0000) 2022-09-07T23:53:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:31.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:31.678729+0000) 2022-09-07T23:53:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:32.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:32.769194+0000) 2022-09-07T23:53:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:32.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:32.678986+0000) 2022-09-07T23:53:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:33 smithi059 ceph-mon[97450]: pgmap v4312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:33 smithi055 ceph-mon[97033]: pgmap v4312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:33.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:33.769423+0000) 2022-09-07T23:53:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:33.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:33.679175+0000) 2022-09-07T23:53:34.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:53:34.388+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:53:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:34.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:34.769701+0000) 2022-09-07T23:53:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:34.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:34.679389+0000) 2022-09-07T23:53:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:35 smithi059 ceph-mon[97450]: pgmap v4313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:53:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:53:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:53:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:35 smithi055 ceph-mon[97033]: pgmap v4313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:53:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:53:35.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:53:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:35.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:35.769953+0000) 2022-09-07T23:53:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:35.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:35.679717+0000) 2022-09-07T23:53:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:36.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:36.770183+0000) 2022-09-07T23:53:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:36.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:36.679885+0000) 2022-09-07T23:53:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:37 smithi059 ceph-mon[97450]: pgmap v4314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:53:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:37 smithi055 ceph-mon[97033]: pgmap v4314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:53:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:37.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:37.770373+0000) 2022-09-07T23:53:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:37.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:37.680110+0000) 2022-09-07T23:53:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:38.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:38.770542+0000) 2022-09-07T23:53:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:38.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:38.680346+0000) 2022-09-07T23:53:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:39 smithi059 ceph-mon[97450]: pgmap v4315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:39 smithi055 ceph-mon[97033]: pgmap v4315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:39.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:39.680593+0000) 2022-09-07T23:53:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:39.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:39.770699+0000) 2022-09-07T23:53:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:40 smithi055 ceph-mon[97033]: pgmap v4316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:53:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:40 smithi059 ceph-mon[97450]: pgmap v4316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:53:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:40.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:40.680822+0000) 2022-09-07T23:53:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:40.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:40.770913+0000) 2022-09-07T23:53:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:41.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:41.771077+0000) 2022-09-07T23:53:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:41.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:41.681066+0000) 2022-09-07T23:53:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:42.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:42.681272+0000) 2022-09-07T23:53:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:42.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:42.771290+0000) 2022-09-07T23:53:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:43 smithi059 ceph-mon[97450]: pgmap v4317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:43 smithi055 ceph-mon[97033]: pgmap v4317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:43.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:43.681519+0000) 2022-09-07T23:53:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:43.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:43.771485+0000) 2022-09-07T23:53:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:44.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:44.771716+0000) 2022-09-07T23:53:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:44.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:44.681678+0000) 2022-09-07T23:53:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:45 smithi059 ceph-mon[97450]: pgmap v4318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:45 smithi055 ceph-mon[97033]: pgmap v4318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:45.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:45.771954+0000) 2022-09-07T23:53:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:45.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:45.681899+0000) 2022-09-07T23:53:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:46.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:46.772093+0000) 2022-09-07T23:53:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:46.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:46.682092+0000) 2022-09-07T23:53:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:47 smithi059 ceph-mon[97450]: pgmap v4319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:53:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:47 smithi055 ceph-mon[97033]: pgmap v4319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:53:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:47.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:47.772318+0000) 2022-09-07T23:53:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:47.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:47.682334+0000) 2022-09-07T23:53:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:48.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:48.772494+0000) 2022-09-07T23:53:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:48.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:48.682584+0000) 2022-09-07T23:53:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:49 smithi059 ceph-mon[97450]: pgmap v4320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:49.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:53:49.388+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:53:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:49 smithi055 ceph-mon[97033]: pgmap v4320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:49.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:49.772759+0000) 2022-09-07T23:53:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:49.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:49.682799+0000) 2022-09-07T23:53:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:50 smithi059 ceph-mon[97450]: pgmap v4321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:53:50.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:50 smithi055 ceph-mon[97033]: pgmap v4321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:53:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:50.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:50.773008+0000) 2022-09-07T23:53:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:50.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:50.683033+0000) 2022-09-07T23:53:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:51.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:51.773201+0000) 2022-09-07T23:53:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:51.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:51.683211+0000) 2022-09-07T23:53:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:52.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:52.773422+0000) 2022-09-07T23:53:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:52.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:52.683440+0000) 2022-09-07T23:53:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:53 smithi059 ceph-mon[97450]: pgmap v4322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:53 smithi055 ceph-mon[97033]: pgmap v4322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:53.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:53.773698+0000) 2022-09-07T23:53:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:53.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:53.683670+0000) 2022-09-07T23:53:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:54.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:54.773917+0000) 2022-09-07T23:53:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:54.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:54.683903+0000) 2022-09-07T23:53:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:55 smithi059 ceph-mon[97450]: pgmap v4323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:55 smithi055 ceph-mon[97033]: pgmap v4323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:55.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:55.774114+0000) 2022-09-07T23:53:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:55.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:55.684183+0000) 2022-09-07T23:53:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:56.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:56.774300+0000) 2022-09-07T23:53:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:56.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:56.684361+0000) 2022-09-07T23:53:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:57 smithi059 ceph-mon[97450]: pgmap v4324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:53:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:57 smithi055 ceph-mon[97033]: pgmap v4324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:53:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:57.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:57.774566+0000) 2022-09-07T23:53:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:57.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:57.684586+0000) 2022-09-07T23:53:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:58.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:58.774855+0000) 2022-09-07T23:53:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:58.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:58.684784+0000) 2022-09-07T23:53:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:53:59 smithi059 ceph-mon[97450]: pgmap v4325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:53:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:53:59 smithi055 ceph-mon[97033]: pgmap v4325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:53:59.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:59.775090+0000) 2022-09-07T23:54:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:53:59.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:53:59.685014+0000) 2022-09-07T23:54:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:00 smithi059 ceph-mon[97450]: pgmap v4326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:54:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:00 smithi055 ceph-mon[97033]: pgmap v4326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:54:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:00.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:00.775316+0000) 2022-09-07T23:54:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:00.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:00.685270+0000) 2022-09-07T23:54:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:01.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:01.775563+0000) 2022-09-07T23:54:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:01.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:01.685535+0000) 2022-09-07T23:54:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:02.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:02.775834+0000) 2022-09-07T23:54:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:02.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:02.685778+0000) 2022-09-07T23:54:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:03 smithi059 ceph-mon[97450]: pgmap v4327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:03.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:03 smithi055 ceph-mon[97033]: pgmap v4327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:03.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:03.776037+0000) 2022-09-07T23:54:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:03.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:03.685921+0000) 2022-09-07T23:54:04.452 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:54:04.395+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:54:04.777 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:04.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:04.686126+0000) 2022-09-07T23:54:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:04.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:04.776239+0000) 2022-09-07T23:54:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:05 smithi059 ceph-mon[97450]: pgmap v4328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:54:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:54:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:54:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:05 smithi055 ceph-mon[97033]: pgmap v4328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:54:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:54:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:54:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:05.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:05.776473+0000) 2022-09-07T23:54:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:05.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:05.686332+0000) 2022-09-07T23:54:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:06 smithi059 ceph-mon[97450]: pgmap v4329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:54:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:06 smithi059 ceph-mon[97450]: pgmap v4330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:54:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:06 smithi055 ceph-mon[97033]: pgmap v4329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:54:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:06 smithi055 ceph-mon[97033]: pgmap v4330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:54:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:06.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:06.776728+0000) 2022-09-07T23:54:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:06.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:06.686565+0000) 2022-09-07T23:54:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:07.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:07.776980+0000) 2022-09-07T23:54:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:07.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:07.686756+0000) 2022-09-07T23:54:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:08 smithi059 ceph-mon[97450]: pgmap v4331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-07T23:54:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:08 smithi055 ceph-mon[97033]: pgmap v4331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-07T23:54:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:08.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:08.777167+0000) 2022-09-07T23:54:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:08.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:08.686916+0000) 2022-09-07T23:54:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:09.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:09.777370+0000) 2022-09-07T23:54:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:09.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:09.687080+0000) 2022-09-07T23:54:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:10 smithi055 ceph-mon[97033]: pgmap v4332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-07T23:54:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:10 smithi055 ceph-mon[97033]: osdmap e359: 12 total, 11 up, 11 in 2022-09-07T23:54:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:10 smithi059 ceph-mon[97450]: pgmap v4332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-07T23:54:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:10 smithi059 ceph-mon[97450]: osdmap e359: 12 total, 11 up, 11 in 2022-09-07T23:54:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:10.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:10.777589+0000) 2022-09-07T23:54:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:10.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:10.687256+0000) 2022-09-07T23:54:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:11.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:11.777805+0000) 2022-09-07T23:54:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:11.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:11.687425+0000) 2022-09-07T23:54:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:12 smithi055 ceph-mon[97033]: pgmap v4334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-07T23:54:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:12 smithi059 ceph-mon[97450]: pgmap v4334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-07T23:54:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:12.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:12.778010+0000) 2022-09-07T23:54:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:12.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:12.687593+0000) 2022-09-07T23:54:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:13.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:13.778152+0000) 2022-09-07T23:54:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:13.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:13.687766+0000) 2022-09-07T23:54:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:14 smithi055 ceph-mon[97033]: pgmap v4335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-07T23:54:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:14 smithi059 ceph-mon[97450]: pgmap v4335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-07T23:54:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:14.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:14.778411+0000) 2022-09-07T23:54:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:14.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:14.687944+0000) 2022-09-07T23:54:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:15 smithi059 ceph-mon[97450]: pgmap v4336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:54:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:15 smithi059 ceph-mon[97450]: osdmap e360: 12 total, 11 up, 11 in 2022-09-07T23:54:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:15 smithi055 ceph-mon[97033]: pgmap v4336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:54:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:15 smithi055 ceph-mon[97033]: osdmap e360: 12 total, 11 up, 11 in 2022-09-07T23:54:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:15.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:15.778729+0000) 2022-09-07T23:54:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:15.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:15.688136+0000) 2022-09-07T23:54:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:16.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:16.778908+0000) 2022-09-07T23:54:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:16.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:16.688334+0000) 2022-09-07T23:54:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:17 smithi059 ceph-mon[97450]: pgmap v4338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:54:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:17 smithi055 ceph-mon[97033]: pgmap v4338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:54:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:17.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:17.779063+0000) 2022-09-07T23:54:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:17.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:17.688503+0000) 2022-09-07T23:54:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:18.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:18.779189+0000) 2022-09-07T23:54:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:18.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:18.688684+0000) 2022-09-07T23:54:19.689 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:54:19.409+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:54:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:19 smithi059 ceph-mon[97450]: pgmap v4339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2022-09-07T23:54:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:19 smithi055 ceph-mon[97033]: pgmap v4339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2022-09-07T23:54:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:19.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:19.779404+0000) 2022-09-07T23:54:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:19.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:19.688830+0000) 2022-09-07T23:54:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:20.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:20.688988+0000) 2022-09-07T23:54:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:20.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:20.779677+0000) 2022-09-07T23:54:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:21 smithi059 ceph-mon[97450]: pgmap v4340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:54:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:21 smithi055 ceph-mon[97033]: pgmap v4340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:54:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:21.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:21.689131+0000) 2022-09-07T23:54:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:21.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:21.779923+0000) 2022-09-07T23:54:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:22.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:22.780084+0000) 2022-09-07T23:54:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:22.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:22.689302+0000) 2022-09-07T23:54:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:23 smithi059 ceph-mon[97450]: pgmap v4341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:54:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:23 smithi055 ceph-mon[97033]: pgmap v4341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:54:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:23.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:23.780312+0000) 2022-09-07T23:54:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:23.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:23.689513+0000) 2022-09-07T23:54:24.781 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:24.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:24.689769+0000) 2022-09-07T23:54:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:24.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:24.780503+0000) 2022-09-07T23:54:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:25 smithi059 ceph-mon[97450]: pgmap v4342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:54:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:25 smithi055 ceph-mon[97033]: pgmap v4342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:54:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:25.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:25.780739+0000) 2022-09-07T23:54:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:25.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:25.689927+0000) 2022-09-07T23:54:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:26.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:26.780933+0000) 2022-09-07T23:54:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:26.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:26.690087+0000) 2022-09-07T23:54:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:27 smithi059 ceph-mon[97450]: pgmap v4343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:54:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:27 smithi055 ceph-mon[97033]: pgmap v4343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:54:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:27.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:27.781162+0000) 2022-09-07T23:54:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:27.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:27.690292+0000) 2022-09-07T23:54:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:28.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:28.781386+0000) 2022-09-07T23:54:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:28.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:28.690548+0000) 2022-09-07T23:54:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:29 smithi059 ceph-mon[97450]: pgmap v4344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:29 smithi055 ceph-mon[97033]: pgmap v4344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:29.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:29.781663+0000) 2022-09-07T23:54:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:29.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:29.690764+0000) 2022-09-07T23:54:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:30.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:30.781914+0000) 2022-09-07T23:54:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:30.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:30.690963+0000) 2022-09-07T23:54:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:31 smithi059 ceph-mon[97450]: pgmap v4345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:54:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:31 smithi055 ceph-mon[97033]: pgmap v4345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:54:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:31.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:31.782140+0000) 2022-09-07T23:54:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:31.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:31.691164+0000) 2022-09-07T23:54:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:32.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:32.782362+0000) 2022-09-07T23:54:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:32.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:32.691329+0000) 2022-09-07T23:54:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:33 smithi059 ceph-mon[97450]: pgmap v4346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:33 smithi055 ceph-mon[97033]: pgmap v4346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:33.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:33.782556+0000) 2022-09-07T23:54:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:33.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:33.691551+0000) 2022-09-07T23:54:34.692 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:54:34.412+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:54:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:34.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:34.782773+0000) 2022-09-07T23:54:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:34.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:34.691697+0000) 2022-09-07T23:54:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:35 smithi059 ceph-mon[97450]: pgmap v4347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:35 smithi055 ceph-mon[97033]: pgmap v4347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:35.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:35.782915+0000) 2022-09-07T23:54:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:35.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:35.691833+0000) 2022-09-07T23:54:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:36.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:36.783140+0000) 2022-09-07T23:54:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:36.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:36.691975+0000) 2022-09-07T23:54:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:37 smithi059 ceph-mon[97450]: pgmap v4348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:54:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:37 smithi055 ceph-mon[97033]: pgmap v4348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:54:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:37.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:37.783316+0000) 2022-09-07T23:54:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:37.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:37.692117+0000) 2022-09-07T23:54:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:38.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:38.783528+0000) 2022-09-07T23:54:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:38.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:38.692247+0000) 2022-09-07T23:54:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:39 smithi059 ceph-mon[97450]: pgmap v4349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:39 smithi055 ceph-mon[97033]: pgmap v4349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:39.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:39.783748+0000) 2022-09-07T23:54:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:39.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:39.692366+0000) 2022-09-07T23:54:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:40.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:40.783884+0000) 2022-09-07T23:54:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:40.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:40.692535+0000) 2022-09-07T23:54:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:41 smithi059 ceph-mon[97450]: pgmap v4350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:54:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:41 smithi055 ceph-mon[97033]: pgmap v4350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:54:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:41.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:41.784093+0000) 2022-09-07T23:54:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:41.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:41.692767+0000) 2022-09-07T23:54:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:42.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:42.784349+0000) 2022-09-07T23:54:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:42.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:42.692984+0000) 2022-09-07T23:54:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:43 smithi055 ceph-mon[97033]: pgmap v4351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:43 smithi059 ceph-mon[97450]: pgmap v4351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:43.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:43.784543+0000) 2022-09-07T23:54:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:43.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:43.693211+0000) 2022-09-07T23:54:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:44.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:44.784762+0000) 2022-09-07T23:54:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:44.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:44.693431+0000) 2022-09-07T23:54:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:45 smithi055 ceph-mon[97033]: pgmap v4352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:45 smithi059 ceph-mon[97450]: pgmap v4352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:45.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:45.784976+0000) 2022-09-07T23:54:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:45.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:45.693694+0000) 2022-09-07T23:54:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:46.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:46.785190+0000) 2022-09-07T23:54:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:46.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:46.693883+0000) 2022-09-07T23:54:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:47 smithi055 ceph-mon[97033]: pgmap v4353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:54:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:47 smithi059 ceph-mon[97450]: pgmap v4353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:54:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:47.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:47.785361+0000) 2022-09-07T23:54:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:47.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:47.694052+0000) 2022-09-07T23:54:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:48.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:48.785585+0000) 2022-09-07T23:54:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:48.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:48.694271+0000) 2022-09-07T23:54:49.695 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:54:49.428+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:54:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:49 smithi055 ceph-mon[97033]: pgmap v4354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:49 smithi059 ceph-mon[97450]: pgmap v4354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:49.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:49.785729+0000) 2022-09-07T23:54:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:49.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:49.694482+0000) 2022-09-07T23:54:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:50.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:50.785957+0000) 2022-09-07T23:54:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:50.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:50.694702+0000) 2022-09-07T23:54:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:51 smithi055 ceph-mon[97033]: pgmap v4355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:54:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:51 smithi059 ceph-mon[97450]: pgmap v4355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:54:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:51.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:51.786192+0000) 2022-09-07T23:54:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:51.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:51.694904+0000) 2022-09-07T23:54:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:52.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:52.786408+0000) 2022-09-07T23:54:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:52.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:52.695140+0000) 2022-09-07T23:54:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:53 smithi055 ceph-mon[97033]: pgmap v4356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:53 smithi059 ceph-mon[97450]: pgmap v4356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:53.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:53.786581+0000) 2022-09-07T23:54:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:53.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:53.695289+0000) 2022-09-07T23:54:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:54.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:54.786717+0000) 2022-09-07T23:54:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:54.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:54.695458+0000) 2022-09-07T23:54:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:55 smithi055 ceph-mon[97033]: pgmap v4357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:55 smithi059 ceph-mon[97450]: pgmap v4357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:54:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:55.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:55.786851+0000) 2022-09-07T23:54:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:55.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:55.695613+0000) 2022-09-07T23:54:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:56.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:56.787034+0000) 2022-09-07T23:54:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:56.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:56.695741+0000) 2022-09-07T23:54:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:57 smithi055 ceph-mon[97033]: pgmap v4358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:54:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:57 smithi059 ceph-mon[97450]: pgmap v4358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:54:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:57.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:57.787223+0000) 2022-09-07T23:54:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:57.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:57.695962+0000) 2022-09-07T23:54:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:58.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:58.787445+0000) 2022-09-07T23:54:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:58.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:58.696224+0000) 2022-09-07T23:54:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:54:59 smithi055 ceph-mon[97033]: pgmap v4359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:54:59 smithi059 ceph-mon[97450]: pgmap v4359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:54:59.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:59.787703+0000) 2022-09-07T23:55:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:54:59.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:54:59.696366+0000) 2022-09-07T23:55:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:00.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:00.787942+0000) 2022-09-07T23:55:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:00.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:00.696522+0000) 2022-09-07T23:55:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:01 smithi055 ceph-mon[97033]: pgmap v4360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:55:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:01 smithi059 ceph-mon[97450]: pgmap v4360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:55:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:01.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:01.788204+0000) 2022-09-07T23:55:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:01.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:01.696738+0000) 2022-09-07T23:55:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:02.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:02.788430+0000) 2022-09-07T23:55:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:02.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:02.696976+0000) 2022-09-07T23:55:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:03 smithi055 ceph-mon[97033]: pgmap v4361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:03 smithi059 ceph-mon[97450]: pgmap v4361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:03.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:03.788656+0000) 2022-09-07T23:55:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:03.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:03.697156+0000) 2022-09-07T23:55:04.698 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:55:04.441+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:55:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:04.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:04.788777+0000) 2022-09-07T23:55:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:04.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:04.697328+0000) 2022-09-07T23:55:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:05 smithi059 ceph-mon[97450]: pgmap v4362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:55:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:55:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:55:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:55:05.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:55:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:05 smithi055 ceph-mon[97033]: pgmap v4362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:55:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:55:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:55:05.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:55:05.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:55:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:05.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:05.788950+0000) 2022-09-07T23:55:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:05.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:05.697472+0000) 2022-09-07T23:55:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:06.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:06.789143+0000) 2022-09-07T23:55:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:06.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:06.697734+0000) 2022-09-07T23:55:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:07 smithi059 ceph-mon[97450]: pgmap v4363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:55:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:07 smithi055 ceph-mon[97033]: pgmap v4363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:55:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:07.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:07.789336+0000) 2022-09-07T23:55:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:07.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:07.697973+0000) 2022-09-07T23:55:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:08.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:08.789484+0000) 2022-09-07T23:55:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:08.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:08.698166+0000) 2022-09-07T23:55:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:09 smithi059 ceph-mon[97450]: pgmap v4364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:55:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:09 smithi059 ceph-mon[97450]: osdmap e361: 12 total, 11 up, 11 in 2022-09-07T23:55:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:09 smithi055 ceph-mon[97033]: pgmap v4364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:55:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:09 smithi055 ceph-mon[97033]: osdmap e361: 12 total, 11 up, 11 in 2022-09-07T23:55:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:09.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:09.789730+0000) 2022-09-07T23:55:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:09.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:09.698336+0000) 2022-09-07T23:55:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:10.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:10.789901+0000) 2022-09-07T23:55:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:10.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:10.698467+0000) 2022-09-07T23:55:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:11 smithi059 ceph-mon[97450]: pgmap v4366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:55:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:11 smithi055 ceph-mon[97033]: pgmap v4366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:55:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:11.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:11.790083+0000) 2022-09-07T23:55:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:11.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:11.698723+0000) 2022-09-07T23:55:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:12.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:12.790311+0000) 2022-09-07T23:55:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:12.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:12.698882+0000) 2022-09-07T23:55:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:13 smithi059 ceph-mon[97450]: pgmap v4367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:55:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:13 smithi055 ceph-mon[97033]: pgmap v4367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:55:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:13.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:13.790540+0000) 2022-09-07T23:55:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:13.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:13.699046+0000) 2022-09-07T23:55:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:14.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:14.790748+0000) 2022-09-07T23:55:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:14.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:14.699230+0000) 2022-09-07T23:55:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:15 smithi059 ceph-mon[97450]: pgmap v4368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:55:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:15 smithi059 ceph-mon[97450]: osdmap e362: 12 total, 11 up, 11 in 2022-09-07T23:55:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:15 smithi055 ceph-mon[97033]: pgmap v4368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:55:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:15 smithi055 ceph-mon[97033]: osdmap e362: 12 total, 11 up, 11 in 2022-09-07T23:55:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:15.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:15.790909+0000) 2022-09-07T23:55:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:15.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:15.699453+0000) 2022-09-07T23:55:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:16.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:16.791104+0000) 2022-09-07T23:55:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:16.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:16.699624+0000) 2022-09-07T23:55:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:17 smithi059 ceph-mon[97450]: pgmap v4370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:55:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:17 smithi055 ceph-mon[97033]: pgmap v4370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:55:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:17.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:17.791353+0000) 2022-09-07T23:55:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:17.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:17.699858+0000) 2022-09-07T23:55:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:18.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:18.791569+0000) 2022-09-07T23:55:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:18.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:18.700133+0000) 2022-09-07T23:55:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:19 smithi059 ceph-mon[97450]: pgmap v4371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T23:55:19.792 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:55:19.454+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:55:19.793 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:19.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:19.700333+0000) 2022-09-07T23:55:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:19 smithi055 ceph-mon[97033]: pgmap v4371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-07T23:55:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:19.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:19.791817+0000) 2022-09-07T23:55:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:20.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:20.700569+0000) 2022-09-07T23:55:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:20.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:20.791994+0000) 2022-09-07T23:55:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:21 smithi059 ceph-mon[97450]: pgmap v4372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:55:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:21 smithi055 ceph-mon[97033]: pgmap v4372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:55:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:21.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:21.792186+0000) 2022-09-07T23:55:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:21.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:21.700774+0000) 2022-09-07T23:55:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:22.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:22.792404+0000) 2022-09-07T23:55:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:22.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:22.700980+0000) 2022-09-07T23:55:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:23 smithi059 ceph-mon[97450]: pgmap v4373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:55:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:23 smithi055 ceph-mon[97033]: pgmap v4373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:55:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:23.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:23.792664+0000) 2022-09-07T23:55:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:23.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:23.701201+0000) 2022-09-07T23:55:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:24.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:24.792836+0000) 2022-09-07T23:55:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:24.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:24.701448+0000) 2022-09-07T23:55:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:25 smithi059 ceph-mon[97450]: pgmap v4374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:55:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:25 smithi055 ceph-mon[97033]: pgmap v4374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:55:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:25.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:25.793055+0000) 2022-09-07T23:55:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:25.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:25.701698+0000) 2022-09-07T23:55:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:26.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:26.793284+0000) 2022-09-07T23:55:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:26.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:26.701908+0000) 2022-09-07T23:55:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:27 smithi059 ceph-mon[97450]: pgmap v4375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:55:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:27 smithi055 ceph-mon[97033]: pgmap v4375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:55:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:27.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:27.793559+0000) 2022-09-07T23:55:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:27.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:27.702176+0000) 2022-09-07T23:55:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:28.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:28.793792+0000) 2022-09-07T23:55:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:28.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:28.702374+0000) 2022-09-07T23:55:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:29 smithi059 ceph-mon[97450]: pgmap v4376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:29 smithi055 ceph-mon[97033]: pgmap v4376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:29.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:29.793995+0000) 2022-09-07T23:55:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:29.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:29.702507+0000) 2022-09-07T23:55:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:30.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:30.794195+0000) 2022-09-07T23:55:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:30.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:30.702746+0000) 2022-09-07T23:55:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:31 smithi059 ceph-mon[97450]: pgmap v4377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:55:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:31 smithi055 ceph-mon[97033]: pgmap v4377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:55:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:31.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:31.794346+0000) 2022-09-07T23:55:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:31.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:31.702973+0000) 2022-09-07T23:55:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:32.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:32.794545+0000) 2022-09-07T23:55:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:32.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:32.703197+0000) 2022-09-07T23:55:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:33 smithi059 ceph-mon[97450]: pgmap v4378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:33 smithi055 ceph-mon[97033]: pgmap v4378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:33.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:33.794761+0000) 2022-09-07T23:55:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:33.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:33.703404+0000) 2022-09-07T23:55:34.721 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:55:34.464+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:55:34.721 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:34.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:34.703707+0000) 2022-09-07T23:55:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:34.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:34.794898+0000) 2022-09-07T23:55:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:35 smithi059 ceph-mon[97450]: pgmap v4379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:35 smithi059 ceph-mon[97450]: pgmap v4380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:55:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:35 smithi059 ceph-mon[97450]: pgmap v4381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:55:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:55:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:35.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:35.795023+0000) 2022-09-07T23:55:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:35.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:35.703945+0000) 2022-09-07T23:55:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:35 smithi055 ceph-mon[97033]: pgmap v4379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:35 smithi055 ceph-mon[97033]: pgmap v4380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:55:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:35 smithi055 ceph-mon[97033]: pgmap v4381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:55:36.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:55:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:36.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:36.795230+0000) 2022-09-07T23:55:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:36.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:36.704195+0000) 2022-09-07T23:55:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:37 smithi059 ceph-mon[97450]: pgmap v4382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:55:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:37.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:37.795462+0000) 2022-09-07T23:55:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:37.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:37.704444+0000) 2022-09-07T23:55:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:37 smithi055 ceph-mon[97033]: pgmap v4382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:55:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:38.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:38.795687+0000) 2022-09-07T23:55:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:38.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:38.704677+0000) 2022-09-07T23:55:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:39 smithi059 ceph-mon[97450]: pgmap v4383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:55:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:39.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:39.795840+0000) 2022-09-07T23:55:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:39.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:39.704876+0000) 2022-09-07T23:55:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:39 smithi055 ceph-mon[97033]: pgmap v4383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:55:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:40.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:40.795978+0000) 2022-09-07T23:55:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:40.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:40.705021+0000) 2022-09-07T23:55:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:41 smithi059 ceph-mon[97450]: pgmap v4384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:55:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:41.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:41.705239+0000) 2022-09-07T23:55:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:41.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:41.796218+0000) 2022-09-07T23:55:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:41 smithi055 ceph-mon[97033]: pgmap v4384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:55:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:42.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:42.796476+0000) 2022-09-07T23:55:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:42.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:42.705494+0000) 2022-09-07T23:55:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:43 smithi059 ceph-mon[97450]: pgmap v4385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:55:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:43.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:43.796770+0000) 2022-09-07T23:55:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:43.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:43.705694+0000) 2022-09-07T23:55:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:43 smithi055 ceph-mon[97033]: pgmap v4385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:55:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:44.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:44.796928+0000) 2022-09-07T23:55:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:44.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:44.705939+0000) 2022-09-07T23:55:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:45 smithi059 ceph-mon[97450]: pgmap v4386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:55:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:45 smithi055 ceph-mon[97033]: pgmap v4386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:55:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:45.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:45.797149+0000) 2022-09-07T23:55:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:45.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:45.706218+0000) 2022-09-07T23:55:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:46.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:46.797382+0000) 2022-09-07T23:55:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:46.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:46.706439+0000) 2022-09-07T23:55:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:47 smithi059 ceph-mon[97450]: pgmap v4387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:55:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:47.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:47.706576+0000) 2022-09-07T23:55:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:47.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:47.797624+0000) 2022-09-07T23:55:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:47 smithi055 ceph-mon[97033]: pgmap v4387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:55:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:48.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:48.797878+0000) 2022-09-07T23:55:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:48.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:48.706824+0000) 2022-09-07T23:55:49.799 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:55:49.471+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:55:49.799 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:49.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:49.707020+0000) 2022-09-07T23:55:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:49.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:49.798039+0000) 2022-09-07T23:55:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:49 smithi055 ceph-mon[97033]: pgmap v4388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:49 smithi059 ceph-mon[97450]: pgmap v4388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:50.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:50.798205+0000) 2022-09-07T23:55:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:50.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:50.707249+0000) 2022-09-07T23:55:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:51.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:51.798426+0000) 2022-09-07T23:55:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:51.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:51.707506+0000) 2022-09-07T23:55:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:51 smithi055 ceph-mon[97033]: pgmap v4389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:55:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:51 smithi059 ceph-mon[97450]: pgmap v4389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:55:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:52.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:52.707753+0000) 2022-09-07T23:55:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:52.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:52.798649+0000) 2022-09-07T23:55:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:53.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:53.708007+0000) 2022-09-07T23:55:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:53.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:53.798875+0000) 2022-09-07T23:55:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:53 smithi055 ceph-mon[97033]: pgmap v4390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:53 smithi059 ceph-mon[97450]: pgmap v4390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:54.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:54.799012+0000) 2022-09-07T23:55:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:54.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:54.708232+0000) 2022-09-07T23:55:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:55 smithi059 ceph-mon[97450]: pgmap v4391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:55 smithi055 ceph-mon[97033]: pgmap v4391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:55:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:55.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:55.799218+0000) 2022-09-07T23:55:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:55.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:55.708507+0000) 2022-09-07T23:55:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:56.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:56.708759+0000) 2022-09-07T23:55:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:56.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:56.799451+0000) 2022-09-07T23:55:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:57.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:57.799725+0000) 2022-09-07T23:55:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:57.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:57.709009+0000) 2022-09-07T23:55:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:57 smithi055 ceph-mon[97033]: pgmap v4392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:55:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:57 smithi059 ceph-mon[97450]: pgmap v4392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:55:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:58.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:58.799872+0000) 2022-09-07T23:55:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:58.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:58.709172+0000) 2022-09-07T23:56:00.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:55:59.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:59.800049+0000) 2022-09-07T23:56:00.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:55:59.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:55:59.709354+0000) 2022-09-07T23:56:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:55:59 smithi055 ceph-mon[97033]: pgmap v4393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:56:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:55:59 smithi059 ceph-mon[97450]: pgmap v4393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:56:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:00.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:00.800255+0000) 2022-09-07T23:56:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:00.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:00.709502+0000) 2022-09-07T23:56:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:01.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:01.800487+0000) 2022-09-07T23:56:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:01.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:01.709681+0000) 2022-09-07T23:56:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:01 smithi055 ceph-mon[97033]: pgmap v4394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:56:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:01 smithi059 ceph-mon[97450]: pgmap v4394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:56:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:02.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:02.709832+0000) 2022-09-07T23:56:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:02.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:02.800764+0000) 2022-09-07T23:56:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:03.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:03.801022+0000) 2022-09-07T23:56:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:03.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:03.709972+0000) 2022-09-07T23:56:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:03 smithi055 ceph-mon[97033]: pgmap v4395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:56:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:03 smithi059 ceph-mon[97450]: pgmap v4395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:56:04.711 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:56:04.484+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:56:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:04.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:04.801177+0000) 2022-09-07T23:56:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:04.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:04.710217+0000) 2022-09-07T23:56:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:56:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:56:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:05 smithi059 ceph-mon[97450]: pgmap v4396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-07T23:56:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:56:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:56:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:05 smithi055 ceph-mon[97033]: pgmap v4396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-07T23:56:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:05.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:05.801337+0000) 2022-09-07T23:56:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:05.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:05.710403+0000) 2022-09-07T23:56:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:06.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:06.801570+0000) 2022-09-07T23:56:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:06.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:06.710679+0000) 2022-09-07T23:56:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:07.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:07.801822+0000) 2022-09-07T23:56:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:07.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:07.710900+0000) 2022-09-07T23:56:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:07 smithi055 ceph-mon[97033]: pgmap v4397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:56:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:07 smithi059 ceph-mon[97450]: pgmap v4397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:56:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:08.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:08.801986+0000) 2022-09-07T23:56:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:08.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:08.711034+0000) 2022-09-07T23:56:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:09.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:09.802210+0000) 2022-09-07T23:56:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:09.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:09.711255+0000) 2022-09-07T23:56:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:09 smithi055 ceph-mon[97033]: pgmap v4398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:56:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:09 smithi055 ceph-mon[97033]: osdmap e363: 12 total, 11 up, 11 in 2022-09-07T23:56:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:09 smithi059 ceph-mon[97450]: pgmap v4398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:56:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:09 smithi059 ceph-mon[97450]: osdmap e363: 12 total, 11 up, 11 in 2022-09-07T23:56:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:10.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:10.802460+0000) 2022-09-07T23:56:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:10.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:10.711523+0000) 2022-09-07T23:56:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:11.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:11.711797+0000) 2022-09-07T23:56:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:11.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:11.802715+0000) 2022-09-07T23:56:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:11 smithi055 ceph-mon[97033]: pgmap v4400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:56:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:11 smithi059 ceph-mon[97450]: pgmap v4400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:56:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:12.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:12.802963+0000) 2022-09-07T23:56:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:12.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:12.712070+0000) 2022-09-07T23:56:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:13.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:13.803270+0000) 2022-09-07T23:56:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:13.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:13.712344+0000) 2022-09-07T23:56:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:13 smithi055 ceph-mon[97033]: pgmap v4401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:56:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:13 smithi059 ceph-mon[97450]: pgmap v4401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:56:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:14.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:14.804302+0000) 2022-09-07T23:56:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:14.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:14.712501+0000) 2022-09-07T23:56:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:15 smithi059 ceph-mon[97450]: osdmap e364: 12 total, 11 up, 11 in 2022-09-07T23:56:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:15 smithi059 ceph-mon[97450]: pgmap v4403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:56:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:15 smithi055 ceph-mon[97033]: osdmap e364: 12 total, 11 up, 11 in 2022-09-07T23:56:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:15 smithi055 ceph-mon[97033]: pgmap v4403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:56:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:15.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:15.804536+0000) 2022-09-07T23:56:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:15.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:15.712719+0000) 2022-09-07T23:56:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:16.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:16.804709+0000) 2022-09-07T23:56:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:16.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:16.712956+0000) 2022-09-07T23:56:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:17.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:17.804835+0000) 2022-09-07T23:56:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:17.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:17.713231+0000) 2022-09-07T23:56:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:17 smithi055 ceph-mon[97033]: pgmap v4404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:56:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:17 smithi059 ceph-mon[97450]: pgmap v4404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:56:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:18.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:18.804960+0000) 2022-09-07T23:56:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:18.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:18.713450+0000) 2022-09-07T23:56:19.806 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:56:19.498+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:56:19.807 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:19.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:19.713667+0000) 2022-09-07T23:56:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:19.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:19.805148+0000) 2022-09-07T23:56:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:19 smithi055 ceph-mon[97033]: pgmap v4405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:56:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:19 smithi059 ceph-mon[97450]: pgmap v4405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:56:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:20.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:20.713892+0000) 2022-09-07T23:56:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:20.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:20.805355+0000) 2022-09-07T23:56:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:21.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:21.805574+0000) 2022-09-07T23:56:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:21.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:21.714094+0000) 2022-09-07T23:56:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:21 smithi055 ceph-mon[97033]: pgmap v4406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:56:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:21 smithi059 ceph-mon[97450]: pgmap v4406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:56:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:22.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:22.805703+0000) 2022-09-07T23:56:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:22.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:22.714347+0000) 2022-09-07T23:56:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:23.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:23.805870+0000) 2022-09-07T23:56:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:23.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:23.714611+0000) 2022-09-07T23:56:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:23 smithi055 ceph-mon[97033]: pgmap v4407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:56:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:23 smithi059 ceph-mon[97450]: pgmap v4407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:56:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:24.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:24.805990+0000) 2022-09-07T23:56:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:24.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:24.714840+0000) 2022-09-07T23:56:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:25 smithi059 ceph-mon[97450]: pgmap v4408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 100 B/s wr, 7 op/s 2022-09-07T23:56:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:25 smithi055 ceph-mon[97033]: pgmap v4408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 100 B/s wr, 7 op/s 2022-09-07T23:56:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:25.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:25.715090+0000) 2022-09-07T23:56:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:25.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:25.806180+0000) 2022-09-07T23:56:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:26.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:26.715239+0000) 2022-09-07T23:56:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:26.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:26.806343+0000) 2022-09-07T23:56:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:27.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:27.715449+0000) 2022-09-07T23:56:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:27.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:27.806513+0000) 2022-09-07T23:56:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:27 smithi055 ceph-mon[97033]: pgmap v4409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:56:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:27 smithi059 ceph-mon[97450]: pgmap v4409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-07T23:56:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:28.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:28.715750+0000) 2022-09-07T23:56:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:28.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:28.806699+0000) 2022-09-07T23:56:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:29.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:29.715964+0000) 2022-09-07T23:56:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:29.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:29.806845+0000) 2022-09-07T23:56:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:29 smithi055 ceph-mon[97033]: pgmap v4410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:56:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:29 smithi059 ceph-mon[97450]: pgmap v4410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:56:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:30.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:30.807057+0000) 2022-09-07T23:56:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:30.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:30.716167+0000) 2022-09-07T23:56:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:31.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:31.807262+0000) 2022-09-07T23:56:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:31.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:31.716321+0000) 2022-09-07T23:56:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:31 smithi055 ceph-mon[97033]: pgmap v4411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:56:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:31 smithi059 ceph-mon[97450]: pgmap v4411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:56:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:32.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:32.807433+0000) 2022-09-07T23:56:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:32.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:32.716538+0000) 2022-09-07T23:56:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:33.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:33.807667+0000) 2022-09-07T23:56:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:33.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:33.716755+0000) 2022-09-07T23:56:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:33 smithi055 ceph-mon[97033]: pgmap v4412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:56:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:33 smithi059 ceph-mon[97450]: pgmap v4412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:56:34.809 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:56:34.508+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:56:34.809 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:34.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:34.716980+0000) 2022-09-07T23:56:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:34.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:34.807945+0000) 2022-09-07T23:56:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:35 smithi059 ceph-mon[97450]: pgmap v4413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:56:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:56:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:56:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:56:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:35 smithi055 ceph-mon[97033]: pgmap v4413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:56:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:56:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:56:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:56:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:35.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:35.808136+0000) 2022-09-07T23:56:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:35.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:35.717323+0000) 2022-09-07T23:56:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:36.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:36.808301+0000) 2022-09-07T23:56:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:36.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:36.717539+0000) 2022-09-07T23:56:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:37.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:37.717775+0000) 2022-09-07T23:56:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:37.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:37.808538+0000) 2022-09-07T23:56:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:37 smithi055 ceph-mon[97033]: pgmap v4414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:56:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:37 smithi059 ceph-mon[97450]: pgmap v4414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:56:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:38.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:38.717940+0000) 2022-09-07T23:56:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:38.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:38.808761+0000) 2022-09-07T23:56:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:39.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:39.808993+0000) 2022-09-07T23:56:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:39.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:39.718153+0000) 2022-09-07T23:56:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:39 smithi055 ceph-mon[97033]: pgmap v4415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:56:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:39 smithi059 ceph-mon[97450]: pgmap v4415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:56:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:40.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:40.718318+0000) 2022-09-07T23:56:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:40.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:40.809144+0000) 2022-09-07T23:56:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:41.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:41.809363+0000) 2022-09-07T23:56:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:41.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:41.718475+0000) 2022-09-07T23:56:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:41 smithi055 ceph-mon[97033]: pgmap v4416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:56:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:41 smithi059 ceph-mon[97450]: pgmap v4416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:56:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:42.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:42.718758+0000) 2022-09-07T23:56:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:42.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:42.809592+0000) 2022-09-07T23:56:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:43.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:43.719068+0000) 2022-09-07T23:56:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:43.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:43.809834+0000) 2022-09-07T23:56:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:43 smithi055 ceph-mon[97033]: pgmap v4417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:56:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:43 smithi059 ceph-mon[97450]: pgmap v4417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:56:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:44.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:44.719281+0000) 2022-09-07T23:56:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:44.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:44.810065+0000) 2022-09-07T23:56:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:45 smithi059 ceph-mon[97450]: pgmap v4418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 5 op/s 2022-09-07T23:56:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:45 smithi055 ceph-mon[97033]: pgmap v4418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 5 op/s 2022-09-07T23:56:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:45.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:45.719551+0000) 2022-09-07T23:56:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:45.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:45.810221+0000) 2022-09-07T23:56:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:46.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:46.810441+0000) 2022-09-07T23:56:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:46.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:46.719832+0000) 2022-09-07T23:56:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:47.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:47.810726+0000) 2022-09-07T23:56:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:47.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:47.720082+0000) 2022-09-07T23:56:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:47 smithi055 ceph-mon[97033]: pgmap v4419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 6 op/s 2022-09-07T23:56:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:47 smithi059 ceph-mon[97450]: pgmap v4419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 6 op/s 2022-09-07T23:56:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:48.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:48.810901+0000) 2022-09-07T23:56:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:48.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:48.720230+0000) 2022-09-07T23:56:49.812 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:56:49.523+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:56:49.813 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:49.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:49.720470+0000) 2022-09-07T23:56:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:49.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:49.811135+0000) 2022-09-07T23:56:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:49 smithi055 ceph-mon[97033]: pgmap v4420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:56:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:49 smithi059 ceph-mon[97450]: pgmap v4420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-07T23:56:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:50.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:50.720717+0000) 2022-09-07T23:56:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:50.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:50.811381+0000) 2022-09-07T23:56:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:51.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:51.811562+0000) 2022-09-07T23:56:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:51.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:51.720912+0000) 2022-09-07T23:56:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:51 smithi055 ceph-mon[97033]: pgmap v4421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:56:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:51 smithi059 ceph-mon[97450]: pgmap v4421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:56:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:52.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:52.811791+0000) 2022-09-07T23:56:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:52.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:52.721209+0000) 2022-09-07T23:56:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:53.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:53.811972+0000) 2022-09-07T23:56:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:53.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:53.721480+0000) 2022-09-07T23:56:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:53 smithi055 ceph-mon[97033]: pgmap v4422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:56:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:53 smithi059 ceph-mon[97450]: pgmap v4422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:56:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:54.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:54.812174+0000) 2022-09-07T23:56:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:54.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:54.721760+0000) 2022-09-07T23:56:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:55 smithi055 ceph-mon[97033]: pgmap v4423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:56:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:55 smithi059 ceph-mon[97450]: pgmap v4423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-07T23:56:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:55.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:55.812413+0000) 2022-09-07T23:56:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:55.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:55.722030+0000) 2022-09-07T23:56:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:56.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:56.812622+0000) 2022-09-07T23:56:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:56.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:56.722285+0000) 2022-09-07T23:56:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:57.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:57.812894+0000) 2022-09-07T23:56:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:57.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:57.722534+0000) 2022-09-07T23:56:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:57 smithi055 ceph-mon[97033]: pgmap v4424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:56:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:57 smithi059 ceph-mon[97450]: pgmap v4424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-07T23:56:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:58.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:58.722744+0000) 2022-09-07T23:56:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:58.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:58.813091+0000) 2022-09-07T23:57:00.054 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:56:59.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:59.722998+0000) 2022-09-07T23:57:00.055 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:56:59.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:56:59.813300+0000) 2022-09-07T23:57:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:56:59 smithi055 ceph-mon[97033]: pgmap v4425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:56:59 smithi059 ceph-mon[97450]: pgmap v4425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:00.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:00.723161+0000) 2022-09-07T23:57:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:00.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:00.813471+0000) 2022-09-07T23:57:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:01.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:01.813692+0000) 2022-09-07T23:57:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:01.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:01.723395+0000) 2022-09-07T23:57:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:01 smithi055 ceph-mon[97033]: pgmap v4426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:57:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:01 smithi059 ceph-mon[97450]: pgmap v4426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:57:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:02.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:02.813905+0000) 2022-09-07T23:57:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:02.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:02.723569+0000) 2022-09-07T23:57:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:03.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:03.814142+0000) 2022-09-07T23:57:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:03.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:03.723763+0000) 2022-09-07T23:57:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:03 smithi055 ceph-mon[97033]: pgmap v4427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:03 smithi059 ceph-mon[97450]: pgmap v4427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:04.815 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:57:04.533+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:57:04.816 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:04.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:04.723964+0000) 2022-09-07T23:57:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:04.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:04.814377+0000) 2022-09-07T23:57:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:57:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:57:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:05 smithi055 ceph-mon[97033]: pgmap v4428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 8 op/s 2022-09-07T23:57:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:05 smithi055 ceph-mon[97033]: pgmap v4429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 52 KiB/s wr, 25 op/s 2022-09-07T23:57:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:05 smithi055 ceph-mon[97033]: pgmap v4430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-07T23:57:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:57:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:57:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:57:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:05 smithi059 ceph-mon[97450]: pgmap v4428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 8 op/s 2022-09-07T23:57:06.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:05 smithi059 ceph-mon[97450]: pgmap v4429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 52 KiB/s wr, 25 op/s 2022-09-07T23:57:06.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:05 smithi059 ceph-mon[97450]: pgmap v4430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-07T23:57:06.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:57:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:05.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:05.814546+0000) 2022-09-07T23:57:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:05.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:05.724180+0000) 2022-09-07T23:57:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:06.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:06.814740+0000) 2022-09-07T23:57:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:06.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:06.724401+0000) 2022-09-07T23:57:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:07.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:07.815005+0000) 2022-09-07T23:57:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:07.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:07.724689+0000) 2022-09-07T23:57:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:08 smithi055 ceph-mon[97033]: pgmap v4431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-07T23:57:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:08 smithi059 ceph-mon[97450]: pgmap v4431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-07T23:57:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:08.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:08.815282+0000) 2022-09-07T23:57:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:08.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:08.724936+0000) 2022-09-07T23:57:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:09.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:09.815500+0000) 2022-09-07T23:57:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:09.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:09.725188+0000) 2022-09-07T23:57:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:10 smithi055 ceph-mon[97033]: pgmap v4432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-07T23:57:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:10 smithi055 ceph-mon[97033]: osdmap e365: 12 total, 11 up, 11 in 2022-09-07T23:57:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:10 smithi059 ceph-mon[97450]: pgmap v4432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-07T23:57:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:10 smithi059 ceph-mon[97450]: osdmap e365: 12 total, 11 up, 11 in 2022-09-07T23:57:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:10.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:10.815748+0000) 2022-09-07T23:57:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:10.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:10.725409+0000) 2022-09-07T23:57:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:11.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:11.815960+0000) 2022-09-07T23:57:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:11.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:11.725630+0000) 2022-09-07T23:57:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:12 smithi055 ceph-mon[97033]: pgmap v4434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 19 KiB/s rd, 42 KiB/s wr, 32 op/s 2022-09-07T23:57:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:12 smithi059 ceph-mon[97450]: pgmap v4434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 19 KiB/s rd, 42 KiB/s wr, 32 op/s 2022-09-07T23:57:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:12.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:12.816193+0000) 2022-09-07T23:57:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:12.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:12.725911+0000) 2022-09-07T23:57:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:13.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:13.816411+0000) 2022-09-07T23:57:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:13.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:13.726132+0000) 2022-09-07T23:57:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:14 smithi055 ceph-mon[97033]: pgmap v4435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:57:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:14 smithi059 ceph-mon[97450]: pgmap v4435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-07T23:57:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:14.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:14.726368+0000) 2022-09-07T23:57:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:14.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:14.816667+0000) 2022-09-07T23:57:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:15 smithi055 ceph-mon[97033]: pgmap v4436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:57:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:15 smithi059 ceph-mon[97450]: pgmap v4436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:57:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:15.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:15.726597+0000) 2022-09-07T23:57:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:15.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:15.816899+0000) 2022-09-07T23:57:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:16.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:16.817139+0000) 2022-09-07T23:57:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:16.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:16.726779+0000) 2022-09-07T23:57:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:17.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:17.817378+0000) 2022-09-07T23:57:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:17.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:17.726960+0000) 2022-09-07T23:57:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:18 smithi055 ceph-mon[97033]: pgmap v4437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:57:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:18 smithi059 ceph-mon[97450]: pgmap v4437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:57:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:18.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:18.817633+0000) 2022-09-07T23:57:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:18.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:18.727182+0000) 2022-09-07T23:57:19.818 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:57:19.550+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:57:19.819 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:19.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:19.727423+0000) 2022-09-07T23:57:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:19.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:19.817839+0000) 2022-09-07T23:57:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:20 smithi055 ceph-mon[97033]: pgmap v4438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:57:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:20 smithi055 ceph-mon[97033]: osdmap e366: 12 total, 11 up, 11 in 2022-09-07T23:57:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:20 smithi059 ceph-mon[97450]: pgmap v4438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:57:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:20 smithi059 ceph-mon[97450]: osdmap e366: 12 total, 11 up, 11 in 2022-09-07T23:57:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:20.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:20.818032+0000) 2022-09-07T23:57:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:20.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:20.727619+0000) 2022-09-07T23:57:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:21.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:21.818227+0000) 2022-09-07T23:57:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:21.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:21.727840+0000) 2022-09-07T23:57:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:22 smithi055 ceph-mon[97033]: pgmap v4440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:57:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:22 smithi059 ceph-mon[97450]: pgmap v4440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:57:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:22.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:22.818461+0000) 2022-09-07T23:57:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:22.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:22.728083+0000) 2022-09-07T23:57:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:23.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:23.818651+0000) 2022-09-07T23:57:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:23.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:23.728254+0000) 2022-09-07T23:57:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:24 smithi055 ceph-mon[97033]: pgmap v4441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:57:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:24 smithi059 ceph-mon[97450]: pgmap v4441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:57:25.081 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:24.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:24.818890+0000) 2022-09-07T23:57:25.082 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:24.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:24.728394+0000) 2022-09-07T23:57:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:25 smithi055 ceph-mon[97033]: pgmap v4442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:57:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:25 smithi059 ceph-mon[97450]: pgmap v4442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:57:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:25.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:25.819112+0000) 2022-09-07T23:57:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:25.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:25.728627+0000) 2022-09-07T23:57:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:26.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:26.819336+0000) 2022-09-07T23:57:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:26.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:26.728875+0000) 2022-09-07T23:57:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:27.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:27.819568+0000) 2022-09-07T23:57:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:27.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:27.729097+0000) 2022-09-07T23:57:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:28 smithi055 ceph-mon[97033]: pgmap v4443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:57:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:28 smithi059 ceph-mon[97450]: pgmap v4443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:57:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:28.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:28.819764+0000) 2022-09-07T23:57:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:28.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:28.729324+0000) 2022-09-07T23:57:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:29.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:29.819886+0000) 2022-09-07T23:57:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:29.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:29.729565+0000) 2022-09-07T23:57:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:30 smithi055 ceph-mon[97033]: pgmap v4444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:57:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:30 smithi059 ceph-mon[97450]: pgmap v4444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:57:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:30.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:30.820055+0000) 2022-09-07T23:57:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:30.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:30.729777+0000) 2022-09-07T23:57:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:31.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:31.820318+0000) 2022-09-07T23:57:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:31.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:31.729940+0000) 2022-09-07T23:57:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:32 smithi055 ceph-mon[97033]: pgmap v4445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-07T23:57:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:32 smithi059 ceph-mon[97450]: pgmap v4445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-07T23:57:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:32.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:32.820522+0000) 2022-09-07T23:57:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:32.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:32.730121+0000) 2022-09-07T23:57:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:33.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:33.820774+0000) 2022-09-07T23:57:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:33.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:33.730335+0000) 2022-09-07T23:57:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:34 smithi055 ceph-mon[97033]: pgmap v4446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:34 smithi059 ceph-mon[97450]: pgmap v4446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:34.822 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:57:34.557+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:57:34.823 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:34.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:34.730476+0000) 2022-09-07T23:57:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:34.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:34.820993+0000) 2022-09-07T23:57:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:35 smithi055 ceph-mon[97033]: pgmap v4447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:57:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:35 smithi059 ceph-mon[97450]: pgmap v4447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:57:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:35.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:35.730695+0000) 2022-09-07T23:57:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:35.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:35.821214+0000) 2022-09-07T23:57:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:36.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:36.730903+0000) 2022-09-07T23:57:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:36.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:36.821428+0000) 2022-09-07T23:57:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:37.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:37.821646+0000) 2022-09-07T23:57:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:37.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:37.731079+0000) 2022-09-07T23:57:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:38 smithi055 ceph-mon[97033]: pgmap v4448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:38 smithi059 ceph-mon[97450]: pgmap v4448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:38.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:38.821875+0000) 2022-09-07T23:57:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:38.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:38.731302+0000) 2022-09-07T23:57:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:39.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:39.822052+0000) 2022-09-07T23:57:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:39.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:39.731541+0000) 2022-09-07T23:57:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:40 smithi055 ceph-mon[97033]: pgmap v4449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:40 smithi059 ceph-mon[97450]: pgmap v4449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:40.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:40.822235+0000) 2022-09-07T23:57:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:40.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:40.731764+0000) 2022-09-07T23:57:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:41.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:41.822354+0000) 2022-09-07T23:57:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:41.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:41.731973+0000) 2022-09-07T23:57:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:42 smithi055 ceph-mon[97033]: pgmap v4450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:57:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:42 smithi059 ceph-mon[97450]: pgmap v4450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:57:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:42.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:42.732237+0000) 2022-09-07T23:57:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:42.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:42.822545+0000) 2022-09-07T23:57:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:43.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:43.822776+0000) 2022-09-07T23:57:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:43.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:43.732512+0000) 2022-09-07T23:57:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:44 smithi055 ceph-mon[97033]: pgmap v4451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:44 smithi059 ceph-mon[97450]: pgmap v4451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:44.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:44.732736+0000) 2022-09-07T23:57:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:44.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:44.823028+0000) 2022-09-07T23:57:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:45 smithi055 ceph-mon[97033]: pgmap v4452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:57:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:45 smithi059 ceph-mon[97450]: pgmap v4452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:57:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:45.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:45.823263+0000) 2022-09-07T23:57:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:45.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:45.732897+0000) 2022-09-07T23:57:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:46.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:46.823442+0000) 2022-09-07T23:57:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:46.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:46.733016+0000) 2022-09-07T23:57:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:47.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:47.733222+0000) 2022-09-07T23:57:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:47.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:47.823690+0000) 2022-09-07T23:57:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:48 smithi055 ceph-mon[97033]: pgmap v4453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:48 smithi059 ceph-mon[97450]: pgmap v4453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:48.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:48.823859+0000) 2022-09-07T23:57:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:48.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:48.733463+0000) 2022-09-07T23:57:49.825 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:57:49.557+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:57:49.826 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:49.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:49.733705+0000) 2022-09-07T23:57:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:49.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:49.824101+0000) 2022-09-07T23:57:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:50 smithi055 ceph-mon[97033]: pgmap v4454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:50 smithi059 ceph-mon[97450]: pgmap v4454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:50.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:50.824285+0000) 2022-09-07T23:57:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:50.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:50.733918+0000) 2022-09-07T23:57:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:51.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:51.824507+0000) 2022-09-07T23:57:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:51.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:51.734129+0000) 2022-09-07T23:57:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:52 smithi055 ceph-mon[97033]: pgmap v4455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:57:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:52 smithi059 ceph-mon[97450]: pgmap v4455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:57:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:52.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:52.824735+0000) 2022-09-07T23:57:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:52.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:52.734301+0000) 2022-09-07T23:57:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:53.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:53.824973+0000) 2022-09-07T23:57:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:53.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:53.734556+0000) 2022-09-07T23:57:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:54 smithi055 ceph-mon[97033]: pgmap v4456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:54 smithi059 ceph-mon[97450]: pgmap v4456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:54.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:54.825207+0000) 2022-09-07T23:57:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:54.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:54.734784+0000) 2022-09-07T23:57:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:55 smithi055 ceph-mon[97033]: pgmap v4457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:57:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:55 smithi059 ceph-mon[97450]: pgmap v4457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:57:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:55.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:55.825428+0000) 2022-09-07T23:57:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:55.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:55.735050+0000) 2022-09-07T23:57:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:56.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:56.825721+0000) 2022-09-07T23:57:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:56.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:56.735283+0000) 2022-09-07T23:57:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:57.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:57.735476+0000) 2022-09-07T23:57:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:57.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:57.825880+0000) 2022-09-07T23:57:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:57:58 smithi055 ceph-mon[97033]: pgmap v4458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:57:58 smithi059 ceph-mon[97450]: pgmap v4458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:57:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:58.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:58.826071+0000) 2022-09-07T23:57:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:58.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:58.735741+0000) 2022-09-07T23:58:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:57:59.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:59.826322+0000) 2022-09-07T23:58:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:57:59.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:57:59.735905+0000) 2022-09-07T23:58:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:00 smithi055 ceph-mon[97033]: pgmap v4459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:00 smithi059 ceph-mon[97450]: pgmap v4459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:00.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:00.826548+0000) 2022-09-07T23:58:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:00.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:00.736141+0000) 2022-09-07T23:58:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:01.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:01.826698+0000) 2022-09-07T23:58:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:01.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:01.736365+0000) 2022-09-07T23:58:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:02 smithi055 ceph-mon[97033]: pgmap v4460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:58:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:02 smithi059 ceph-mon[97450]: pgmap v4460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:58:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:02.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:02.826927+0000) 2022-09-07T23:58:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:02.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:02.736599+0000) 2022-09-07T23:58:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:03.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:03.827177+0000) 2022-09-07T23:58:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:03.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:03.736831+0000) 2022-09-07T23:58:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:04 smithi055 ceph-mon[97033]: pgmap v4461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:04 smithi059 ceph-mon[97450]: pgmap v4461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:04.828 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:58:04.566+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:58:04.829 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:04.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:04.737000+0000) 2022-09-07T23:58:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:04.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:04.827395+0000) 2022-09-07T23:58:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:58:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:58:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:05 smithi055 ceph-mon[97033]: pgmap v4462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:58:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:58:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:58:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:58:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:58:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:58:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:05 smithi059 ceph-mon[97450]: pgmap v4462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:58:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:58:06.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:58:06.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:58:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:05.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:05.827683+0000) 2022-09-07T23:58:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:05.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:05.737217+0000) 2022-09-07T23:58:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:06.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:06.827890+0000) 2022-09-07T23:58:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:06.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:06.737461+0000) 2022-09-07T23:58:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:07.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:07.828114+0000) 2022-09-07T23:58:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:07.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:07.737700+0000) 2022-09-07T23:58:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:08 smithi055 ceph-mon[97033]: pgmap v4463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:58:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:08 smithi059 ceph-mon[97450]: pgmap v4463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:58:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:08.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:08.737841+0000) 2022-09-07T23:58:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:08.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:08.828312+0000) 2022-09-07T23:58:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:09.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:09.828500+0000) 2022-09-07T23:58:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:09.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:09.738013+0000) 2022-09-07T23:58:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:10 smithi055 ceph-mon[97033]: pgmap v4464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:58:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:10 smithi055 ceph-mon[97033]: osdmap e367: 12 total, 11 up, 11 in 2022-09-07T23:58:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:10 smithi059 ceph-mon[97450]: pgmap v4464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:58:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:10 smithi059 ceph-mon[97450]: osdmap e367: 12 total, 11 up, 11 in 2022-09-07T23:58:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:10.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:10.828740+0000) 2022-09-07T23:58:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:10.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:10.738255+0000) 2022-09-07T23:58:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:11.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:11.828972+0000) 2022-09-07T23:58:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:11.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:11.738503+0000) 2022-09-07T23:58:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:12 smithi055 ceph-mon[97033]: pgmap v4466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:58:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:12 smithi059 ceph-mon[97450]: pgmap v4466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:58:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:12.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:12.829201+0000) 2022-09-07T23:58:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:12.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:12.738747+0000) 2022-09-07T23:58:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:13.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:13.829435+0000) 2022-09-07T23:58:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:13.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:13.738917+0000) 2022-09-07T23:58:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:14 smithi055 ceph-mon[97033]: pgmap v4467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:58:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:14 smithi059 ceph-mon[97450]: pgmap v4467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:58:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:14.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:14.829658+0000) 2022-09-07T23:58:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:14.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:14.739168+0000) 2022-09-07T23:58:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:15 smithi055 ceph-mon[97033]: osdmap e368: 12 total, 11 up, 11 in 2022-09-07T23:58:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:15 smithi055 ceph-mon[97033]: pgmap v4469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:58:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:15 smithi059 ceph-mon[97450]: osdmap e368: 12 total, 11 up, 11 in 2022-09-07T23:58:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:15 smithi059 ceph-mon[97450]: pgmap v4469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:58:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:15.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:15.829883+0000) 2022-09-07T23:58:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:15.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:15.739439+0000) 2022-09-07T23:58:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:16.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:16.830146+0000) 2022-09-07T23:58:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:16.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:16.739680+0000) 2022-09-07T23:58:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:17.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:17.830366+0000) 2022-09-07T23:58:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:17.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:17.739851+0000) 2022-09-07T23:58:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:18 smithi055 ceph-mon[97033]: pgmap v4470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:58:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:18 smithi059 ceph-mon[97450]: pgmap v4470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:58:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:18.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:18.740068+0000) 2022-09-07T23:58:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:18.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:18.830580+0000) 2022-09-07T23:58:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:19.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:19.830799+0000) 2022-09-07T23:58:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:19.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:19.740284+0000) 2022-09-07T23:58:19.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:58:19.581+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:58:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:20 smithi055 ceph-mon[97033]: pgmap v4471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:58:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:20 smithi059 ceph-mon[97450]: pgmap v4471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-07T23:58:21.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:20.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:20.830992+0000) 2022-09-07T23:58:21.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:20.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:20.740507+0000) 2022-09-07T23:58:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:21.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:21.831217+0000) 2022-09-07T23:58:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:21.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:21.740739+0000) 2022-09-07T23:58:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:22 smithi055 ceph-mon[97033]: pgmap v4472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:58:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:22 smithi059 ceph-mon[97450]: pgmap v4472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:58:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:22.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:22.831453+0000) 2022-09-07T23:58:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:22.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:22.740984+0000) 2022-09-07T23:58:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:23.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:23.741214+0000) 2022-09-07T23:58:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:23.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:23.831680+0000) 2022-09-07T23:58:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:24 smithi055 ceph-mon[97033]: pgmap v4473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:58:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:24 smithi059 ceph-mon[97450]: pgmap v4473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:58:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:24.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:24.741460+0000) 2022-09-07T23:58:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:24.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:24.831924+0000) 2022-09-07T23:58:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:25 smithi055 ceph-mon[97033]: pgmap v4474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 97 B/s wr, 8 op/s 2022-09-07T23:58:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:25 smithi059 ceph-mon[97450]: pgmap v4474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 97 B/s wr, 8 op/s 2022-09-07T23:58:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:25.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:25.832057+0000) 2022-09-07T23:58:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:25.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:25.741718+0000) 2022-09-07T23:58:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:26.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:26.741952+0000) 2022-09-07T23:58:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:26.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:26.832262+0000) 2022-09-07T23:58:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:27.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:27.832517+0000) 2022-09-07T23:58:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:27.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:27.742194+0000) 2022-09-07T23:58:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:28 smithi055 ceph-mon[97033]: pgmap v4475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:28 smithi059 ceph-mon[97450]: pgmap v4475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:28.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:28.742332+0000) 2022-09-07T23:58:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:28.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:28.832734+0000) 2022-09-07T23:58:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:29.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:29.832975+0000) 2022-09-07T23:58:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:29.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:29.742451+0000) 2022-09-07T23:58:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:30 smithi055 ceph-mon[97033]: pgmap v4476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:30 smithi059 ceph-mon[97450]: pgmap v4476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:30.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:30.833186+0000) 2022-09-07T23:58:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:30.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:30.742717+0000) 2022-09-07T23:58:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:31.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:31.833344+0000) 2022-09-07T23:58:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:31.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:31.742959+0000) 2022-09-07T23:58:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:32 smithi055 ceph-mon[97033]: pgmap v4477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:58:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:32 smithi059 ceph-mon[97450]: pgmap v4477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:58:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:32.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:32.833525+0000) 2022-09-07T23:58:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:32.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:32.743174+0000) 2022-09-07T23:58:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:33.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:33.833764+0000) 2022-09-07T23:58:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:33.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:33.743404+0000) 2022-09-07T23:58:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:34 smithi055 ceph-mon[97033]: pgmap v4478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:34 smithi059 ceph-mon[97450]: pgmap v4478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:34.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:58:34.591+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:58:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:34.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:34.833932+0000) 2022-09-07T23:58:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:34.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:34.743703+0000) 2022-09-07T23:58:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:35.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:35.834145+0000) 2022-09-07T23:58:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:35.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:35.743929+0000) 2022-09-07T23:58:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:36 smithi055 ceph-mon[97033]: pgmap v4479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:58:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:36 smithi055 ceph-mon[97033]: pgmap v4480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:58:36.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:36 smithi055 ceph-mon[97033]: pgmap v4481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:58:36.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:58:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:36 smithi059 ceph-mon[97450]: pgmap v4479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:58:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:36 smithi059 ceph-mon[97450]: pgmap v4480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-07T23:58:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:36 smithi059 ceph-mon[97450]: pgmap v4481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:58:36.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:58:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:36.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:36.834282+0000) 2022-09-07T23:58:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:36.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:36.744160+0000) 2022-09-07T23:58:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:37.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:37.744336+0000) 2022-09-07T23:58:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:37.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:37.834497+0000) 2022-09-07T23:58:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:38 smithi055 ceph-mon[97033]: pgmap v4482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:58:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:38 smithi059 ceph-mon[97450]: pgmap v4482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:58:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:38.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:38.834703+0000) 2022-09-07T23:58:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:38.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:38.744568+0000) 2022-09-07T23:58:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:39.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:39.744803+0000) 2022-09-07T23:58:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:39.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:39.834917+0000) 2022-09-07T23:58:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:40 smithi055 ceph-mon[97033]: pgmap v4483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:58:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:40 smithi059 ceph-mon[97450]: pgmap v4483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:58:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:40.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:40.745042+0000) 2022-09-07T23:58:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:40.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:40.835090+0000) 2022-09-07T23:58:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:41.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:41.835294+0000) 2022-09-07T23:58:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:41.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:41.745238+0000) 2022-09-07T23:58:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:42 smithi055 ceph-mon[97033]: pgmap v4484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:58:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:42 smithi059 ceph-mon[97450]: pgmap v4484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-07T23:58:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:42.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:42.745452+0000) 2022-09-07T23:58:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:42.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:42.835458+0000) 2022-09-07T23:58:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:43.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:43.835658+0000) 2022-09-07T23:58:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:43.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:43.745742+0000) 2022-09-07T23:58:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:44 smithi055 ceph-mon[97033]: pgmap v4485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:58:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:44 smithi059 ceph-mon[97450]: pgmap v4485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-07T23:58:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:44.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:44.835896+0000) 2022-09-07T23:58:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:44.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:44.745931+0000) 2022-09-07T23:58:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:45 smithi055 ceph-mon[97033]: pgmap v4486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:58:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:45 smithi059 ceph-mon[97450]: pgmap v4486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-07T23:58:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:45.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:45.836118+0000) 2022-09-07T23:58:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:45.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:45.746140+0000) 2022-09-07T23:58:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:46.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:46.836320+0000) 2022-09-07T23:58:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:46.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:46.746371+0000) 2022-09-07T23:58:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:47 smithi055 ceph-mon[97033]: pgmap v4487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:47 smithi059 ceph-mon[97450]: pgmap v4487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:47.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:47.836550+0000) 2022-09-07T23:58:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:47.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:47.746655+0000) 2022-09-07T23:58:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:48.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:48.836762+0000) 2022-09-07T23:58:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:48.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:48.746844+0000) 2022-09-07T23:58:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:49 smithi055 ceph-mon[97033]: pgmap v4488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:49 smithi059 ceph-mon[97450]: pgmap v4488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:49.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:58:49.603+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:58:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:49.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:49.836977+0000) 2022-09-07T23:58:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:49.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:49.746994+0000) 2022-09-07T23:58:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:50.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:50.837205+0000) 2022-09-07T23:58:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:50.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:50.747167+0000) 2022-09-07T23:58:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:51 smithi055 ceph-mon[97033]: pgmap v4489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:58:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:51 smithi059 ceph-mon[97450]: pgmap v4489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:58:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:51.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:51.837442+0000) 2022-09-07T23:58:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:51.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:51.747367+0000) 2022-09-07T23:58:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:52.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:52.837669+0000) 2022-09-07T23:58:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:52.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:52.747605+0000) 2022-09-07T23:58:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:53 smithi055 ceph-mon[97033]: pgmap v4490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:53 smithi059 ceph-mon[97450]: pgmap v4490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:53.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:53.837882+0000) 2022-09-07T23:58:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:53.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:53.747879+0000) 2022-09-07T23:58:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:54.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:54.838121+0000) 2022-09-07T23:58:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:54.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:54.748026+0000) 2022-09-07T23:58:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:55 smithi055 ceph-mon[97033]: pgmap v4491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:58:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:55 smithi059 ceph-mon[97450]: pgmap v4491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:58:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:55.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:55.748260+0000) 2022-09-07T23:58:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:55.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:55.838319+0000) 2022-09-07T23:58:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:56.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:56.748499+0000) 2022-09-07T23:58:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:56.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:56.838507+0000) 2022-09-07T23:58:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:57 smithi055 ceph-mon[97033]: pgmap v4492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:57 smithi059 ceph-mon[97450]: pgmap v4492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:57.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:57.748738+0000) 2022-09-07T23:58:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:57.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:57.838691+0000) 2022-09-07T23:58:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:58.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:58.838901+0000) 2022-09-07T23:58:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:58.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:58.748960+0000) 2022-09-07T23:58:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:58:59 smithi055 ceph-mon[97033]: pgmap v4493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:58:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:58:59 smithi059 ceph-mon[97450]: pgmap v4493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:58:59.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:59.749108+0000) 2022-09-07T23:59:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:58:59.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:58:59.839154+0000) 2022-09-07T23:59:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:00.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:00.839393+0000) 2022-09-07T23:59:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:00.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:00.749298+0000) 2022-09-07T23:59:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:01 smithi055 ceph-mon[97033]: pgmap v4494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:59:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:01 smithi059 ceph-mon[97450]: pgmap v4494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:59:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:01.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:01.839606+0000) 2022-09-07T23:59:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:01.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:01.749452+0000) 2022-09-07T23:59:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:02.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:02.749684+0000) 2022-09-07T23:59:03.165 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:02.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:02.839766+0000) 2022-09-07T23:59:03.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:03 smithi055 ceph-mon[97033]: pgmap v4495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:03 smithi059 ceph-mon[97450]: pgmap v4495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:03.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:03.840006+0000) 2022-09-07T23:59:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:03.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:03.749890+0000) 2022-09-07T23:59:04.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:59:04.609+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:59:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:04.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:04.840240+0000) 2022-09-07T23:59:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:04.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:04.750035+0000) 2022-09-07T23:59:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:59:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:59:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:05 smithi055 ceph-mon[97033]: pgmap v4496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:59:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-07T23:59:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-07T23:59:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:05 smithi059 ceph-mon[97450]: pgmap v4496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-07T23:59:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:05.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:05.840508+0000) 2022-09-07T23:59:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:05.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:05.750252+0000) 2022-09-07T23:59:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:06.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:06.840776+0000) 2022-09-07T23:59:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:06.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:06.750513+0000) 2022-09-07T23:59:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:07 smithi055 ceph-mon[97033]: pgmap v4497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:59:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:07 smithi059 ceph-mon[97450]: pgmap v4497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-07T23:59:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:07.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:07.840911+0000) 2022-09-07T23:59:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:07.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:07.750740+0000) 2022-09-07T23:59:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:08.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:08.841324+0000) 2022-09-07T23:59:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:08.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:08.750968+0000) 2022-09-07T23:59:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:09 smithi055 ceph-mon[97033]: osdmap e369: 12 total, 11 up, 11 in 2022-09-07T23:59:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:09 smithi055 ceph-mon[97033]: pgmap v4499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:59:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:09 smithi059 ceph-mon[97450]: osdmap e369: 12 total, 11 up, 11 in 2022-09-07T23:59:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:09 smithi059 ceph-mon[97450]: pgmap v4499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-07T23:59:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:09.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:09.841565+0000) 2022-09-07T23:59:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:09.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:09.751176+0000) 2022-09-07T23:59:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:10.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:10.841810+0000) 2022-09-07T23:59:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:10.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:10.751396+0000) 2022-09-07T23:59:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:11 smithi055 ceph-mon[97033]: pgmap v4500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:59:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:11 smithi059 ceph-mon[97450]: pgmap v4500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:59:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:11.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:11.842055+0000) 2022-09-07T23:59:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:11.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:11.751673+0000) 2022-09-07T23:59:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:12.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:12.842280+0000) 2022-09-07T23:59:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:12.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:12.751954+0000) 2022-09-07T23:59:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:13 smithi055 ceph-mon[97033]: pgmap v4501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:59:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:13 smithi059 ceph-mon[97450]: pgmap v4501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-07T23:59:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:13.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:13.842443+0000) 2022-09-07T23:59:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:13.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:13.752149+0000) 2022-09-07T23:59:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:14.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:14.842721+0000) 2022-09-07T23:59:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:14.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:14.752348+0000) 2022-09-07T23:59:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:15 smithi055 ceph-mon[97033]: osdmap e370: 12 total, 11 up, 11 in 2022-09-07T23:59:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:15 smithi055 ceph-mon[97033]: pgmap v4503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:59:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:15 smithi059 ceph-mon[97450]: osdmap e370: 12 total, 11 up, 11 in 2022-09-07T23:59:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:15 smithi059 ceph-mon[97450]: pgmap v4503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-07T23:59:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:15.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:15.842882+0000) 2022-09-07T23:59:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:15.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:15.752508+0000) 2022-09-07T23:59:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:16.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:16.843029+0000) 2022-09-07T23:59:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:16.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:16.752751+0000) 2022-09-07T23:59:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:17 smithi055 ceph-mon[97033]: pgmap v4504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:59:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:17 smithi059 ceph-mon[97450]: pgmap v4504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-07T23:59:18.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:17.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:17.843184+0000) 2022-09-07T23:59:18.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:17.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:17.752939+0000) 2022-09-07T23:59:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:18.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:18.753189+0000) 2022-09-07T23:59:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:18.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:18.843378+0000) 2022-09-07T23:59:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:19 smithi055 ceph-mon[97033]: pgmap v4505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:59:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:19 smithi059 ceph-mon[97450]: pgmap v4505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-07T23:59:19.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:59:19.616+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:59:19.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:19.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:19.843611+0000) 2022-09-07T23:59:19.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:19.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:19.753389+0000) 2022-09-07T23:59:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:20.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:20.843862+0000) 2022-09-07T23:59:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:20.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:20.753563+0000) 2022-09-07T23:59:21.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:21 smithi055 ceph-mon[97033]: pgmap v4506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:59:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:21 smithi059 ceph-mon[97450]: pgmap v4506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:59:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:21.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:21.844066+0000) 2022-09-07T23:59:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:21.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:21.753793+0000) 2022-09-07T23:59:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:22.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:22.844250+0000) 2022-09-07T23:59:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:22.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:22.754045+0000) 2022-09-07T23:59:23.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:23 smithi055 ceph-mon[97033]: pgmap v4507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:59:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:23 smithi059 ceph-mon[97450]: pgmap v4507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-07T23:59:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:23.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:23.844484+0000) 2022-09-07T23:59:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:23.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:23.754286+0000) 2022-09-07T23:59:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:24.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:24.844673+0000) 2022-09-07T23:59:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:24.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:24.754467+0000) 2022-09-07T23:59:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:25 smithi055 ceph-mon[97033]: pgmap v4508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 95 B/s wr, 8 op/s 2022-09-07T23:59:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:25 smithi059 ceph-mon[97450]: pgmap v4508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 95 B/s wr, 8 op/s 2022-09-07T23:59:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:25.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:25.844762+0000) 2022-09-07T23:59:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:25.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:25.754733+0000) 2022-09-07T23:59:27.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:26.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:26.754962+0000) 2022-09-07T23:59:27.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:26.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:26.844902+0000) 2022-09-07T23:59:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:27 smithi055 ceph-mon[97033]: pgmap v4509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:27 smithi059 ceph-mon[97450]: pgmap v4509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:27.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:27.845092+0000) 2022-09-07T23:59:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:27.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:27.755113+0000) 2022-09-07T23:59:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:28.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:28.845239+0000) 2022-09-07T23:59:29.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:28.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:28.755260+0000) 2022-09-07T23:59:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:29 smithi055 ceph-mon[97033]: pgmap v4510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:29 smithi059 ceph-mon[97450]: pgmap v4510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:29.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:29.755433+0000) 2022-09-07T23:59:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:29.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:29.845461+0000) 2022-09-07T23:59:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:30.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:30.755668+0000) 2022-09-07T23:59:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:30.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:30.845729+0000) 2022-09-07T23:59:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:31 smithi055 ceph-mon[97033]: pgmap v4511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:59:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:31 smithi059 ceph-mon[97450]: pgmap v4511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:59:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:31.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:31.845991+0000) 2022-09-07T23:59:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:31.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:31.755896+0000) 2022-09-07T23:59:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:32.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:32.846159+0000) 2022-09-07T23:59:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:32.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:32.756107+0000) 2022-09-07T23:59:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:33 smithi055 ceph-mon[97033]: pgmap v4512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:33 smithi059 ceph-mon[97450]: pgmap v4512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:33.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:33.846346+0000) 2022-09-07T23:59:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:33.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:33.756322+0000) 2022-09-07T23:59:34.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:59:34.622+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:59:34.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:34.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:34.846519+0000) 2022-09-07T23:59:34.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:34.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:34.756448+0000) 2022-09-07T23:59:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:35 smithi055 ceph-mon[97033]: pgmap v4513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:59:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:59:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:59:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:59:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:35 smithi059 ceph-mon[97450]: pgmap v4513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:59:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-07T23:59:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-07T23:59:35.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-07T23:59:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:35.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:35.846784+0000) 2022-09-07T23:59:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:35.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:35.756718+0000) 2022-09-07T23:59:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:36.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:36.847011+0000) 2022-09-07T23:59:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:36.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:36.756908+0000) 2022-09-07T23:59:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:59:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:59:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:59:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:59:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:37.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:37.847231+0000) 2022-09-07T23:59:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:37.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:37.757061+0000) 2022-09-07T23:59:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:37 smithi055 ceph-mon[97033]: pgmap v4514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:38.264 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:37 smithi059 ceph-mon[97450]: pgmap v4514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:38.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:38.847450+0000) 2022-09-07T23:59:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:38.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:38.757298+0000) 2022-09-07T23:59:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:59:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:59:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:59:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:59:39.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:59:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:59:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:59:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-07T23:59:39.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:39.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-07T23:59:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:39.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:39.847708+0000) 2022-09-07T23:59:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:39.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:39.757519+0000) 2022-09-07T23:59:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:40 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T23:59:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:40 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T23:59:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:40 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T23:59:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:40 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T23:59:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:40 smithi059 ceph-mon[97450]: pgmap v4515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:40 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-07T23:59:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:40 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-07T23:59:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:40 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-07T23:59:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:40 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-07T23:59:40.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:40 smithi055 ceph-mon[97033]: pgmap v4515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:40.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:40.847893+0000) 2022-09-07T23:59:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:40.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:40.757748+0000) 2022-09-07T23:59:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:41 smithi055 ceph-mon[97033]: pgmap v4516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:59:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:41 smithi059 ceph-mon[97450]: pgmap v4516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:59:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:41.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:41.848120+0000) 2022-09-07T23:59:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:41.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:41.757948+0000) 2022-09-07T23:59:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:42.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:42.758192+0000) 2022-09-07T23:59:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:42.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:42.848330+0000) 2022-09-07T23:59:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:43 smithi055 ceph-mon[97033]: pgmap v4517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:43.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:43 smithi059 ceph-mon[97450]: pgmap v4517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:43.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:43.848571+0000) 2022-09-07T23:59:44.221 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:43.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:43.758427+0000) 2022-09-07T23:59:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:44.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:44.848816+0000) 2022-09-07T23:59:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:44.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:44.758601+0000) 2022-09-07T23:59:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:45 smithi055 ceph-mon[97033]: pgmap v4518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:59:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:45 smithi059 ceph-mon[97450]: pgmap v4518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:59:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:45.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:45.848996+0000) 2022-09-07T23:59:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:45.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:45.758778+0000) 2022-09-07T23:59:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:46.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:46.849218+0000) 2022-09-07T23:59:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:46.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:46.758989+0000) 2022-09-07T23:59:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:47 smithi055 ceph-mon[97033]: pgmap v4519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:47 smithi059 ceph-mon[97450]: pgmap v4519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:47.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:47.759194+0000) 2022-09-07T23:59:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:47.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:47.849350+0000) 2022-09-07T23:59:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:48.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:48.849524+0000) 2022-09-07T23:59:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:48.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:48.759445+0000) 2022-09-07T23:59:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:49 smithi055 ceph-mon[97033]: pgmap v4520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:49 smithi059 ceph-mon[97450]: pgmap v4520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:49.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 07 23:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-07T23:59:49.638+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-07T23:59:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:49.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:49.849780+0000) 2022-09-07T23:59:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:49.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:49.759709+0000) 2022-09-07T23:59:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:50.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:50.850048+0000) 2022-09-07T23:59:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:50.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:50.759942+0000) 2022-09-07T23:59:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:51 smithi055 ceph-mon[97033]: pgmap v4521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:59:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:51 smithi059 ceph-mon[97450]: pgmap v4521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:59:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:51.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:51.850310+0000) 2022-09-07T23:59:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:51.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:51.760167+0000) 2022-09-07T23:59:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:52.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:52.850574+0000) 2022-09-07T23:59:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:52.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:52.760340+0000) 2022-09-07T23:59:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:53 smithi055 ceph-mon[97033]: pgmap v4522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:53 smithi059 ceph-mon[97450]: pgmap v4522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:53.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:53.850841+0000) 2022-09-07T23:59:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:53.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:53.760526+0000) 2022-09-07T23:59:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:54.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:54.851080+0000) 2022-09-07T23:59:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:54.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:54.760701+0000) 2022-09-07T23:59:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:55 smithi055 ceph-mon[97033]: pgmap v4523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:59:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:55 smithi059 ceph-mon[97450]: pgmap v4523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-07T23:59:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:55.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:55.851290+0000) 2022-09-07T23:59:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:55.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:55.760907+0000) 2022-09-07T23:59:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:56.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:56.851438+0000) 2022-09-07T23:59:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:56.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:56.761110+0000) 2022-09-07T23:59:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:57 smithi055 ceph-mon[97033]: pgmap v4524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:57 smithi059 ceph-mon[97450]: pgmap v4524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:57.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:57.851687+0000) 2022-09-07T23:59:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:57.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:57.761324+0000) 2022-09-07T23:59:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:58.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:58.851870+0000) 2022-09-07T23:59:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:58.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:58.761561+0000) 2022-09-07T23:59:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 07 23:59:59 smithi055 ceph-mon[97033]: pgmap v4525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-07T23:59:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 07 23:59:59 smithi059 ceph-mon[97450]: pgmap v4525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:00.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 07 23:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-07T23:59:59.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:59.852016+0000) 2022-09-08T00:00:00.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 07 23:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-07T23:59:59.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T22:59:59.761798+0000) 2022-09-08T00:00:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T00:00:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T00:00:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T00:00:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T00:00:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T00:00:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T00:00:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T00:00:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T00:00:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T00:00:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T00:00:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:00.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:00.852180+0000) 2022-09-08T00:00:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:00.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:00.761968+0000) 2022-09-08T00:00:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:01 smithi055 ceph-mon[97033]: pgmap v4526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:00:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:01 smithi059 ceph-mon[97450]: pgmap v4526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:00:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:01.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:01.852402+0000) 2022-09-08T00:00:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:01.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:01.762080+0000) 2022-09-08T00:00:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:02.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:02.852677+0000) 2022-09-08T00:00:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:02.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:02.762232+0000) 2022-09-08T00:00:03.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:03 smithi055 ceph-mon[97033]: pgmap v4527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:03 smithi059 ceph-mon[97450]: pgmap v4527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:03.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:03.852869+0000) 2022-09-08T00:00:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:03.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:03.762442+0000) 2022-09-08T00:00:04.914 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:00:04.653+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:00:04.915 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:04.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:04.762655+0000) 2022-09-08T00:00:04.916 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:04.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:04.853056+0000) 2022-09-08T00:00:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:00:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:00:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:00:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:00:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:05 smithi055 ceph-mon[97033]: pgmap v4528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:00:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:05 smithi059 ceph-mon[97450]: pgmap v4528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:00:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:05.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:05.853231+0000) 2022-09-08T00:00:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:05.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:05.762852+0000) 2022-09-08T00:00:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:06.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:06.763088+0000) 2022-09-08T00:00:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:06.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:06.853476+0000) 2022-09-08T00:00:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:07 smithi055 ceph-mon[97033]: pgmap v4529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:00:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:07 smithi059 ceph-mon[97450]: pgmap v4529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:00:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:07.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:07.853702+0000) 2022-09-08T00:00:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:07.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:07.763325+0000) 2022-09-08T00:00:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:08.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:08.853854+0000) 2022-09-08T00:00:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:08.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:08.763551+0000) 2022-09-08T00:00:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:09 smithi055 ceph-mon[97033]: osdmap e371: 12 total, 11 up, 11 in 2022-09-08T00:00:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:09 smithi055 ceph-mon[97033]: pgmap v4531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:00:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:09 smithi059 ceph-mon[97450]: osdmap e371: 12 total, 11 up, 11 in 2022-09-08T00:00:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:09 smithi059 ceph-mon[97450]: pgmap v4531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:00:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:09.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:09.853995+0000) 2022-09-08T00:00:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:09.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:09.763737+0000) 2022-09-08T00:00:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:10.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:10.854212+0000) 2022-09-08T00:00:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:10.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:10.763929+0000) 2022-09-08T00:00:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:00:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:00:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:10 smithi059 ceph-mon[97450]: pgmap v4532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T00:00:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:10 smithi059 ceph-mon[97450]: pgmap v4533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 20 KiB/s rd, 80 KiB/s wr, 35 op/s 2022-09-08T00:00:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:00:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:00:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:00:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:11 smithi055 ceph-mon[97033]: pgmap v4532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T00:00:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:11 smithi055 ceph-mon[97033]: pgmap v4533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 20 KiB/s rd, 80 KiB/s wr, 35 op/s 2022-09-08T00:00:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:00:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:11.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:11.854422+0000) 2022-09-08T00:00:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:11.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:11.764100+0000) 2022-09-08T00:00:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:12.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:12.854709+0000) 2022-09-08T00:00:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:12.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:12.764350+0000) 2022-09-08T00:00:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:13 smithi055 ceph-mon[97033]: pgmap v4534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 153 B/s wr, 4 op/s 2022-09-08T00:00:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:13 smithi059 ceph-mon[97450]: pgmap v4534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 153 B/s wr, 4 op/s 2022-09-08T00:00:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:13.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:13.854968+0000) 2022-09-08T00:00:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:13.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:13.764612+0000) 2022-09-08T00:00:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:14.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:14.855145+0000) 2022-09-08T00:00:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:14.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:14.764814+0000) 2022-09-08T00:00:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:15 smithi055 ceph-mon[97033]: pgmap v4535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 153 B/s wr, 4 op/s 2022-09-08T00:00:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:15 smithi055 ceph-mon[97033]: osdmap e372: 12 total, 11 up, 11 in 2022-09-08T00:00:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:15 smithi059 ceph-mon[97450]: pgmap v4535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 153 B/s wr, 4 op/s 2022-09-08T00:00:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:15 smithi059 ceph-mon[97450]: osdmap e372: 12 total, 11 up, 11 in 2022-09-08T00:00:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:15.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:15.855381+0000) 2022-09-08T00:00:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:15.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:15.764988+0000) 2022-09-08T00:00:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:16.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:16.855595+0000) 2022-09-08T00:00:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:16.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:16.765222+0000) 2022-09-08T00:00:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:17 smithi055 ceph-mon[97033]: pgmap v4537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.8 KiB/s rd, 306 B/s wr, 9 op/s 2022-09-08T00:00:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:17 smithi059 ceph-mon[97450]: pgmap v4537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.8 KiB/s rd, 306 B/s wr, 9 op/s 2022-09-08T00:00:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:17.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:17.855847+0000) 2022-09-08T00:00:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:17.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:17.765382+0000) 2022-09-08T00:00:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:18.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:18.856117+0000) 2022-09-08T00:00:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:18.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:18.765614+0000) 2022-09-08T00:00:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:19 smithi055 ceph-mon[97033]: pgmap v4538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:00:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:19 smithi059 ceph-mon[97450]: pgmap v4538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:00:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:19.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:19.856368+0000) 2022-09-08T00:00:20.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:00:19.668+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:00:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:19.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:19.765877+0000) 2022-09-08T00:00:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:20 smithi055 ceph-mon[97033]: pgmap v4539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:00:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:20 smithi059 ceph-mon[97450]: pgmap v4539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:00:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:20.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:20.856618+0000) 2022-09-08T00:00:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:20.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:20.766054+0000) 2022-09-08T00:00:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:21.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:21.856840+0000) 2022-09-08T00:00:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:21.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:21.766226+0000) 2022-09-08T00:00:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:22.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:22.857053+0000) 2022-09-08T00:00:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:22.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:22.766393+0000) 2022-09-08T00:00:23.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:23 smithi055 ceph-mon[97033]: pgmap v4540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:00:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:23 smithi059 ceph-mon[97450]: pgmap v4540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:00:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:23.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:23.857283+0000) 2022-09-08T00:00:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:23.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:23.766574+0000) 2022-09-08T00:00:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:24.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:24.857451+0000) 2022-09-08T00:00:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:24.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:24.766785+0000) 2022-09-08T00:00:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:25 smithi055 ceph-mon[97033]: pgmap v4541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:00:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:25 smithi059 ceph-mon[97450]: pgmap v4541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:00:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:25.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:25.766996+0000) 2022-09-08T00:00:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:25.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:25.857668+0000) 2022-09-08T00:00:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:26.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:26.857881+0000) 2022-09-08T00:00:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:26.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:26.767245+0000) 2022-09-08T00:00:27.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:27 smithi055 ceph-mon[97033]: pgmap v4542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-08T00:00:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:27 smithi059 ceph-mon[97450]: pgmap v4542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-08T00:00:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:27.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:27.858083+0000) 2022-09-08T00:00:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:27.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:27.767485+0000) 2022-09-08T00:00:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:28.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:28.858328+0000) 2022-09-08T00:00:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:28.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:28.767749+0000) 2022-09-08T00:00:29.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:29 smithi055 ceph-mon[97033]: pgmap v4543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:29 smithi059 ceph-mon[97450]: pgmap v4543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:30.021 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:29.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:29.858508+0000) 2022-09-08T00:00:30.022 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:29.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:29.767953+0000) 2022-09-08T00:00:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:30 smithi055 ceph-mon[97033]: pgmap v4544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:00:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:30 smithi059 ceph-mon[97450]: pgmap v4544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:00:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:30.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:30.858737+0000) 2022-09-08T00:00:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:30.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:30.768226+0000) 2022-09-08T00:00:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:31.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:31.858851+0000) 2022-09-08T00:00:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:31.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:31.768454+0000) 2022-09-08T00:00:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:32.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:32.859071+0000) 2022-09-08T00:00:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:32.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:32.768720+0000) 2022-09-08T00:00:33.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:33 smithi055 ceph-mon[97033]: pgmap v4545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:33 smithi059 ceph-mon[97450]: pgmap v4545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:33.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:33.859310+0000) 2022-09-08T00:00:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:33.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:33.768925+0000) 2022-09-08T00:00:35.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:00:34.675+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:00:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:34.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:34.769083+0000) 2022-09-08T00:00:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:34.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:34.859529+0000) 2022-09-08T00:00:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:35 smithi055 ceph-mon[97033]: pgmap v4546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:35 smithi059 ceph-mon[97450]: pgmap v4546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:35.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:35.769219+0000) 2022-09-08T00:00:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:35.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:35.859759+0000) 2022-09-08T00:00:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:36.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:36.859983+0000) 2022-09-08T00:00:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:36.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:36.769469+0000) 2022-09-08T00:00:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:37 smithi055 ceph-mon[97033]: pgmap v4547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:00:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:37 smithi059 ceph-mon[97450]: pgmap v4547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:00:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:37.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:37.860190+0000) 2022-09-08T00:00:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:37.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:37.769715+0000) 2022-09-08T00:00:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:38.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:38.860424+0000) 2022-09-08T00:00:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:38.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:38.769945+0000) 2022-09-08T00:00:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:39 smithi055 ceph-mon[97033]: pgmap v4548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:39 smithi059 ceph-mon[97450]: pgmap v4548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:39.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:39.860683+0000) 2022-09-08T00:00:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:39.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:39.770104+0000) 2022-09-08T00:00:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:40 smithi055 ceph-mon[97033]: pgmap v4549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:00:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:40 smithi059 ceph-mon[97450]: pgmap v4549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:00:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:40.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:40.860900+0000) 2022-09-08T00:00:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:40.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:40.770329+0000) 2022-09-08T00:00:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:41.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:41.861139+0000) 2022-09-08T00:00:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:41.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:41.770535+0000) 2022-09-08T00:00:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:42.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:42.861307+0000) 2022-09-08T00:00:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:42.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:42.770782+0000) 2022-09-08T00:00:43.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:43 smithi055 ceph-mon[97033]: pgmap v4550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:43 smithi059 ceph-mon[97450]: pgmap v4550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:43.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:43.771013+0000) 2022-09-08T00:00:44.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:43.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:43.861533+0000) 2022-09-08T00:00:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:44.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:44.861725+0000) 2022-09-08T00:00:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:44.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:44.771216+0000) 2022-09-08T00:00:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:45 smithi055 ceph-mon[97033]: pgmap v4551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:45 smithi059 ceph-mon[97450]: pgmap v4551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:45.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:45.861912+0000) 2022-09-08T00:00:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:45.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:45.771418+0000) 2022-09-08T00:00:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:46.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:46.862128+0000) 2022-09-08T00:00:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:46.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:46.771650+0000) 2022-09-08T00:00:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:47 smithi055 ceph-mon[97033]: pgmap v4552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:00:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:47 smithi059 ceph-mon[97450]: pgmap v4552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:00:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:47.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:47.862334+0000) 2022-09-08T00:00:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:47.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:47.771826+0000) 2022-09-08T00:00:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:48.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:48.862554+0000) 2022-09-08T00:00:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:48.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:48.772085+0000) 2022-09-08T00:00:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:49 smithi055 ceph-mon[97033]: pgmap v4553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:49.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:49 smithi059 ceph-mon[97450]: pgmap v4553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:50.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:00:49.682+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:00:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:49.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:49.862812+0000) 2022-09-08T00:00:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:49.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:49.772315+0000) 2022-09-08T00:00:50.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:50 smithi055 ceph-mon[97033]: pgmap v4554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:00:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:50 smithi059 ceph-mon[97450]: pgmap v4554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:00:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:50.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:50.863058+0000) 2022-09-08T00:00:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:50.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:50.772549+0000) 2022-09-08T00:00:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:51.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:51.863195+0000) 2022-09-08T00:00:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:51.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:51.772805+0000) 2022-09-08T00:00:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:52.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:52.863414+0000) 2022-09-08T00:00:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:52.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:52.773024+0000) 2022-09-08T00:00:53.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:53 smithi055 ceph-mon[97033]: pgmap v4555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:53 smithi059 ceph-mon[97450]: pgmap v4555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:53.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:53.773231+0000) 2022-09-08T00:00:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:53.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:53.863679+0000) 2022-09-08T00:00:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:54.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:54.773463+0000) 2022-09-08T00:00:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:54.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:54.863899+0000) 2022-09-08T00:00:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:55 smithi055 ceph-mon[97033]: pgmap v4556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:55 smithi059 ceph-mon[97450]: pgmap v4556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:55.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:55.864124+0000) 2022-09-08T00:00:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:55.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:55.773625+0000) 2022-09-08T00:00:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:56.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:56.864293+0000) 2022-09-08T00:00:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:56.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:56.773896+0000) 2022-09-08T00:00:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:57 smithi055 ceph-mon[97033]: pgmap v4557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:00:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:57 smithi059 ceph-mon[97450]: pgmap v4557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:00:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:57.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:57.864512+0000) 2022-09-08T00:00:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:57.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:57.774144+0000) 2022-09-08T00:00:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:58.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:58.864728+0000) 2022-09-08T00:00:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:58.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:58.774355+0000) 2022-09-08T00:00:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:00:59 smithi055 ceph-mon[97033]: pgmap v4558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:00:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:00:59 smithi059 ceph-mon[97450]: pgmap v4558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:00:59.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:59.864933+0000) 2022-09-08T00:01:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:00:59.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:00:59.774584+0000) 2022-09-08T00:01:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:00 smithi055 ceph-mon[97033]: pgmap v4559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:01:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:00 smithi059 ceph-mon[97450]: pgmap v4559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:01:01.097 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:00.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:00.865087+0000) 2022-09-08T00:01:01.098 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:00.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:00.774700+0000) 2022-09-08T00:01:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:01.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:01.865300+0000) 2022-09-08T00:01:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:01.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:01.774869+0000) 2022-09-08T00:01:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:02.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:02.775115+0000) 2022-09-08T00:01:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:02.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:02.865469+0000) 2022-09-08T00:01:03.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:03 smithi055 ceph-mon[97033]: pgmap v4560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:03 smithi059 ceph-mon[97450]: pgmap v4560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:03.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:03.865746+0000) 2022-09-08T00:01:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:03.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:03.775314+0000) 2022-09-08T00:01:05.082 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 08 00:01:04 smithi055 sudo[152457]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-08T00:01:05.083 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 08 00:01:04 smithi055 sudo[152457]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:05.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 08 00:01:04 smithi055 sudo[152457]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:05.084 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 08 00:01:04 smithi055 sudo[152457]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:04.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:04.865970+0000) 2022-09-08T00:01:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:04.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:04.775575+0000) 2022-09-08T00:01:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:01:04.696+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:01:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:05 smithi055 ceph-mon[97033]: pgmap v4561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:01:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:01:05.480 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 08 00:01:05 smithi055 sudo[152460]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-08T00:01:05.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 08 00:01:05 smithi055 sudo[152460]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:05.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 08 00:01:05 smithi055 sudo[152460]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:05.482 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 08 00:01:05 smithi055 sudo[152460]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:05 smithi059 ceph-mon[97450]: pgmap v4561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:01:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:01:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:05.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:05.866216+0000) 2022-09-08T00:01:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:05.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:05.775794+0000) 2022-09-08T00:01:06.462 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 08 00:01:06 smithi059 sudo[153805]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-08T00:01:06.463 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 08 00:01:06 smithi059 sudo[153805]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:06.464 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 08 00:01:06 smithi059 sudo[153805]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:06.465 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 08 00:01:06 smithi059 sudo[153805]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:01:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:06 smithi055 ceph-mon[97033]: pgmap v4562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:01:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:01:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:06 smithi059 ceph-mon[97450]: pgmap v4562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:01:06.782 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 08 00:01:06 smithi059 sudo[153808]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-08T00:01:06.782 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 08 00:01:06 smithi059 sudo[153808]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:06.782 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 08 00:01:06 smithi059 sudo[153808]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:06.784 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 08 00:01:06 smithi059 sudo[153808]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:06.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:06.866367+0000) 2022-09-08T00:01:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:06.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:06.775983+0000) 2022-09-08T00:01:07.777 INFO:journalctl@ceph.osd.3.smithi047.stdout:Sep 08 00:01:07 smithi047 sudo[150559]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-08T00:01:07.778 INFO:journalctl@ceph.osd.3.smithi047.stdout:Sep 08 00:01:07 smithi047 sudo[150559]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:07.778 INFO:journalctl@ceph.osd.3.smithi047.stdout:Sep 08 00:01:07 smithi047 sudo[150559]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:07.779 INFO:journalctl@ceph.osd.3.smithi047.stdout:Sep 08 00:01:07 smithi047 sudo[150559]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:01:08.044 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:07.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:07.866495+0000) 2022-09-08T00:01:08.045 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:07.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:07.776127+0000) 2022-09-08T00:01:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:01:08.402 INFO:journalctl@ceph.osd.3.smithi047.stdout:Sep 08 00:01:08 smithi047 sudo[150562]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-08T00:01:08.403 INFO:journalctl@ceph.osd.3.smithi047.stdout:Sep 08 00:01:08 smithi047 sudo[150562]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:08.404 INFO:journalctl@ceph.osd.3.smithi047.stdout:Sep 08 00:01:08 smithi047 sudo[150562]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:08.404 INFO:journalctl@ceph.osd.3.smithi047.stdout:Sep 08 00:01:08 smithi047 sudo[150562]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:08.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:08.776260+0000) 2022-09-08T00:01:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:08.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:08.866734+0000) 2022-09-08T00:01:09.202 INFO:journalctl@ceph.osd.4.smithi055.stdout:Sep 08 00:01:08 smithi055 sudo[152463]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-08T00:01:09.203 INFO:journalctl@ceph.osd.4.smithi055.stdout:Sep 08 00:01:08 smithi055 sudo[152463]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:09.203 INFO:journalctl@ceph.osd.4.smithi055.stdout:Sep 08 00:01:08 smithi055 sudo[152463]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:09.204 INFO:journalctl@ceph.osd.4.smithi055.stdout:Sep 08 00:01:08 smithi055 sudo[152463]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:09 smithi055 ceph-mon[97033]: pgmap v4563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:01:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:01:09.480 INFO:journalctl@ceph.osd.4.smithi055.stdout:Sep 08 00:01:09 smithi055 sudo[152466]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-08T00:01:09.480 INFO:journalctl@ceph.osd.4.smithi055.stdout:Sep 08 00:01:09 smithi055 sudo[152466]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:09.481 INFO:journalctl@ceph.osd.4.smithi055.stdout:Sep 08 00:01:09 smithi055 sudo[152466]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:09.481 INFO:journalctl@ceph.osd.4.smithi055.stdout:Sep 08 00:01:09 smithi055 sudo[152466]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:09 smithi059 ceph-mon[97450]: pgmap v4563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:01:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:01:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:09.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:09.866888+0000) 2022-09-08T00:01:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:09.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:09.776463+0000) 2022-09-08T00:01:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:10 smithi059 ceph-mon[97450]: osdmap e373: 12 total, 11 up, 11 in 2022-09-08T00:01:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:01:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:01:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:01:10.284 INFO:journalctl@ceph.osd.5.smithi059.stdout:Sep 08 00:01:09 smithi059 sudo[153819]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-08T00:01:10.284 INFO:journalctl@ceph.osd.5.smithi059.stdout:Sep 08 00:01:09 smithi059 sudo[153819]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:10.285 INFO:journalctl@ceph.osd.5.smithi059.stdout:Sep 08 00:01:09 smithi059 sudo[153819]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:10.286 INFO:journalctl@ceph.osd.5.smithi059.stdout:Sep 08 00:01:09 smithi059 sudo[153819]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:10 smithi055 ceph-mon[97033]: osdmap e373: 12 total, 11 up, 11 in 2022-09-08T00:01:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:01:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:01:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:01:11.030 INFO:journalctl@ceph.osd.5.smithi059.stdout:Sep 08 00:01:10 smithi059 sudo[153906]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-08T00:01:11.031 INFO:journalctl@ceph.osd.5.smithi059.stdout:Sep 08 00:01:10 smithi059 sudo[153906]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:11.031 INFO:journalctl@ceph.osd.5.smithi059.stdout:Sep 08 00:01:10 smithi059 sudo[153906]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:11.032 INFO:journalctl@ceph.osd.5.smithi059.stdout:Sep 08 00:01:10 smithi059 sudo[153906]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:10.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:10.867033+0000) 2022-09-08T00:01:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:10.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:10.776648+0000) 2022-09-08T00:01:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:11 smithi055 ceph-mon[97033]: pgmap v4565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 37 KiB/s rd, 171 KiB/s wr, 34 op/s 2022-09-08T00:01:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:01:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:01:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:01:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:11 smithi059 ceph-mon[97450]: pgmap v4565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 37 KiB/s rd, 171 KiB/s wr, 34 op/s 2022-09-08T00:01:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:01:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:01:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:01:11.652 INFO:journalctl@ceph.osd.6.smithi047.stdout:Sep 08 00:01:11 smithi047 sudo[150566]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-08T00:01:11.653 INFO:journalctl@ceph.osd.6.smithi047.stdout:Sep 08 00:01:11 smithi047 sudo[150566]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:11.654 INFO:journalctl@ceph.osd.6.smithi047.stdout:Sep 08 00:01:11 smithi047 sudo[150566]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:11.654 INFO:journalctl@ceph.osd.6.smithi047.stdout:Sep 08 00:01:11 smithi047 sudo[150566]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:11.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:11.867231+0000) 2022-09-08T00:01:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:11.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:11.776782+0000) 2022-09-08T00:01:12.155 INFO:journalctl@ceph.osd.6.smithi047.stdout:Sep 08 00:01:11 smithi047 sudo[150569]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-08T00:01:12.155 INFO:journalctl@ceph.osd.6.smithi047.stdout:Sep 08 00:01:11 smithi047 sudo[150569]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:12.155 INFO:journalctl@ceph.osd.6.smithi047.stdout:Sep 08 00:01:11 smithi047 sudo[150569]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:12.156 INFO:journalctl@ceph.osd.6.smithi047.stdout:Sep 08 00:01:11 smithi047 sudo[150569]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:12.479 INFO:journalctl@ceph.osd.7.smithi055.stdout:Sep 08 00:01:12 smithi055 sudo[152607]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-08T00:01:12.480 INFO:journalctl@ceph.osd.7.smithi055.stdout:Sep 08 00:01:12 smithi055 sudo[152607]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:12.480 INFO:journalctl@ceph.osd.7.smithi055.stdout:Sep 08 00:01:12 smithi055 sudo[152607]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:12.481 INFO:journalctl@ceph.osd.7.smithi055.stdout:Sep 08 00:01:12 smithi055 sudo[152607]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:12.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:12 smithi055 ceph-mon[97033]: pgmap v4566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 40 KiB/s rd, 188 KiB/s wr, 37 op/s 2022-09-08T00:01:12.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:12 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T00:01:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:12 smithi059 ceph-mon[97450]: pgmap v4566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 40 KiB/s rd, 188 KiB/s wr, 37 op/s 2022-09-08T00:01:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:12 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T00:01:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:12.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:12.776951+0000) 2022-09-08T00:01:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:12.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:12.867453+0000) 2022-09-08T00:01:13.229 INFO:journalctl@ceph.osd.7.smithi055.stdout:Sep 08 00:01:12 smithi055 sudo[152610]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-08T00:01:13.229 INFO:journalctl@ceph.osd.7.smithi055.stdout:Sep 08 00:01:12 smithi055 sudo[152610]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:13.230 INFO:journalctl@ceph.osd.7.smithi055.stdout:Sep 08 00:01:12 smithi055 sudo[152610]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:13.230 INFO:journalctl@ceph.osd.7.smithi055.stdout:Sep 08 00:01:12 smithi055 sudo[152610]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:14.030 INFO:journalctl@ceph.osd.8.smithi059.stdout:Sep 08 00:01:13 smithi059 sudo[154044]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-08T00:01:14.031 INFO:journalctl@ceph.osd.8.smithi059.stdout:Sep 08 00:01:13 smithi059 sudo[154044]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:14.032 INFO:journalctl@ceph.osd.8.smithi059.stdout:Sep 08 00:01:13 smithi059 sudo[154044]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:14.032 INFO:journalctl@ceph.osd.8.smithi059.stdout:Sep 08 00:01:13 smithi059 sudo[154044]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:13.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:13.867632+0000) 2022-09-08T00:01:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:13.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:13.777105+0000) 2022-09-08T00:01:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:14 smithi055 ceph-mon[97033]: pgmap v4567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 40 KiB/s rd, 188 KiB/s wr, 37 op/s 2022-09-08T00:01:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:14 smithi059 ceph-mon[97450]: pgmap v4567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 40 KiB/s rd, 188 KiB/s wr, 37 op/s 2022-09-08T00:01:14.532 INFO:journalctl@ceph.osd.8.smithi059.stdout:Sep 08 00:01:14 smithi059 sudo[154047]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-08T00:01:14.532 INFO:journalctl@ceph.osd.8.smithi059.stdout:Sep 08 00:01:14 smithi059 sudo[154047]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:14.532 INFO:journalctl@ceph.osd.8.smithi059.stdout:Sep 08 00:01:14 smithi059 sudo[154047]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:14.533 INFO:journalctl@ceph.osd.8.smithi059.stdout:Sep 08 00:01:14 smithi059 sudo[154047]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:14.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:14.867768+0000) 2022-09-08T00:01:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:14.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:14.777228+0000) 2022-09-08T00:01:14.904 INFO:journalctl@ceph.osd.9.smithi047.stdout:Sep 08 00:01:14 smithi047 sudo[150573]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-08T00:01:14.905 INFO:journalctl@ceph.osd.9.smithi047.stdout:Sep 08 00:01:14 smithi047 sudo[150573]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:14.905 INFO:journalctl@ceph.osd.9.smithi047.stdout:Sep 08 00:01:14 smithi047 sudo[150573]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:14.906 INFO:journalctl@ceph.osd.9.smithi047.stdout:Sep 08 00:01:14 smithi047 sudo[150573]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:15.652 INFO:journalctl@ceph.osd.9.smithi047.stdout:Sep 08 00:01:15 smithi047 sudo[150576]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-08T00:01:15.653 INFO:journalctl@ceph.osd.9.smithi047.stdout:Sep 08 00:01:15 smithi047 sudo[150576]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:15.653 INFO:journalctl@ceph.osd.9.smithi047.stdout:Sep 08 00:01:15 smithi047 sudo[150576]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:15.654 INFO:journalctl@ceph.osd.9.smithi047.stdout:Sep 08 00:01:15 smithi047 sudo[150576]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:15 smithi055 ceph-mon[97033]: osdmap e374: 12 total, 11 up, 11 in 2022-09-08T00:01:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:15 smithi055 ceph-mon[97033]: pgmap v4569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 37 KiB/s rd, 166 KiB/s wr, 23 op/s 2022-09-08T00:01:15.980 INFO:journalctl@ceph.osd.10.smithi055.stdout:Sep 08 00:01:15 smithi055 sudo[152613]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-08T00:01:15.981 INFO:journalctl@ceph.osd.10.smithi055.stdout:Sep 08 00:01:15 smithi055 sudo[152613]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:15.981 INFO:journalctl@ceph.osd.10.smithi055.stdout:Sep 08 00:01:15 smithi055 sudo[152613]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:15.982 INFO:journalctl@ceph.osd.10.smithi055.stdout:Sep 08 00:01:15 smithi055 sudo[152613]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:15 smithi059 ceph-mon[97450]: osdmap e374: 12 total, 11 up, 11 in 2022-09-08T00:01:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:15 smithi059 ceph-mon[97450]: pgmap v4569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 37 KiB/s rd, 166 KiB/s wr, 23 op/s 2022-09-08T00:01:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:15.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:15.777453+0000) 2022-09-08T00:01:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:15.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:15.867975+0000) 2022-09-08T00:01:16.729 INFO:journalctl@ceph.osd.10.smithi055.stdout:Sep 08 00:01:16 smithi055 sudo[152616]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-08T00:01:16.729 INFO:journalctl@ceph.osd.10.smithi055.stdout:Sep 08 00:01:16 smithi055 sudo[152616]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:16.730 INFO:journalctl@ceph.osd.10.smithi055.stdout:Sep 08 00:01:16 smithi055 sudo[152616]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:16.730 INFO:journalctl@ceph.osd.10.smithi055.stdout:Sep 08 00:01:16 smithi055 sudo[152616]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:16.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:16.868171+0000) 2022-09-08T00:01:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:16.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:16.777746+0000) 2022-09-08T00:01:17.281 INFO:journalctl@ceph.osd.11.smithi059.stdout:Sep 08 00:01:16 smithi059 sudo[154050]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-08T00:01:17.281 INFO:journalctl@ceph.osd.11.smithi059.stdout:Sep 08 00:01:16 smithi059 sudo[154050]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:17.282 INFO:journalctl@ceph.osd.11.smithi059.stdout:Sep 08 00:01:16 smithi059 sudo[154050]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:17.282 INFO:journalctl@ceph.osd.11.smithi059.stdout:Sep 08 00:01:16 smithi059 sudo[154050]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:17.780 INFO:journalctl@ceph.osd.11.smithi059.stdout:Sep 08 00:01:17 smithi059 sudo[154053]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-08T00:01:17.781 INFO:journalctl@ceph.osd.11.smithi059.stdout:Sep 08 00:01:17 smithi059 sudo[154053]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:17.782 INFO:journalctl@ceph.osd.11.smithi059.stdout:Sep 08 00:01:17 smithi059 sudo[154053]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:17.783 INFO:journalctl@ceph.osd.11.smithi059.stdout:Sep 08 00:01:17 smithi059 sudo[154053]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:17.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:17.868394+0000) 2022-09-08T00:01:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:17.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:17.777952+0000) 2022-09-08T00:01:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:18 smithi055 ceph-mon[97033]: pgmap v4570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 33 KiB/s rd, 149 KiB/s wr, 20 op/s 2022-09-08T00:01:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:18 smithi059 ceph-mon[97450]: pgmap v4570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 33 KiB/s rd, 149 KiB/s wr, 20 op/s 2022-09-08T00:01:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:18.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:18.868608+0000) 2022-09-08T00:01:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:18.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:18.778127+0000) 2022-09-08T00:01:20.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:01:19.698+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:01:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:19.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:19.868851+0000) 2022-09-08T00:01:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:19.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:19.778345+0000) 2022-09-08T00:01:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:20 smithi055 ceph-mon[97033]: pgmap v4571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 112 B/s wr, 3 op/s 2022-09-08T00:01:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:20 smithi059 ceph-mon[97450]: pgmap v4571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 112 B/s wr, 3 op/s 2022-09-08T00:01:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:20.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:20.778588+0000) 2022-09-08T00:01:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:20.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:20.869141+0000) 2022-09-08T00:01:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:21.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:21.869331+0000) 2022-09-08T00:01:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:21.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:21.778726+0000) 2022-09-08T00:01:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:22 smithi055 ceph-mon[97033]: pgmap v4572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:01:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:22 smithi059 ceph-mon[97450]: pgmap v4572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:01:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:22.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:22.778950+0000) 2022-09-08T00:01:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:22.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:22.869570+0000) 2022-09-08T00:01:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:23.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:23.869826+0000) 2022-09-08T00:01:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:23.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:23.779231+0000) 2022-09-08T00:01:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:24 smithi055 ceph-mon[97033]: pgmap v4573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:01:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:24 smithi059 ceph-mon[97450]: pgmap v4573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:01:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:24.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:24.870022+0000) 2022-09-08T00:01:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:24.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:24.779431+0000) 2022-09-08T00:01:25.265 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:24 smithi059 sudo[154056]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-08T00:01:25.266 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:24 smithi059 sudo[154056]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:25.266 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:24 smithi059 sudo[154056]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:25.267 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:24 smithi059 sudo[154056]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:25 smithi059 sudo[154059]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-08T00:01:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:25 smithi059 sudo[154059]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:25 smithi059 sudo[154059]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:25 smithi059 sudo[154059]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:25 smithi055 sudo[152619]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-08T00:01:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:25 smithi055 sudo[152619]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:25 smithi055 sudo[152619]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:25 smithi055 sudo[152619]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:25.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:25.870234+0000) 2022-09-08T00:01:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:25.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:25.779714+0000) 2022-09-08T00:01:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:26 smithi055 sudo[152622]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-08T00:01:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:26 smithi055 sudo[152622]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-08T00:01:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:26 smithi055 sudo[152622]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-08T00:01:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:26 smithi055 sudo[152622]: pam_unix(sudo:session): session closed for user root 2022-09-08T00:01:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:26 smithi055 ceph-mon[97033]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-08T00:01:26.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:26 smithi055 ceph-mon[97033]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-08T00:01:26.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-08T00:01:26.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-08T00:01:26.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-08T00:01:26.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-08T00:01:26.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-08T00:01:26.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-08T00:01:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:26 smithi059 ceph-mon[97450]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-08T00:01:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:26 smithi059 ceph-mon[97450]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-08T00:01:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-08T00:01:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-08T00:01:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-08T00:01:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-08T00:01:26.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-08T00:01:26.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-08T00:01:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:26.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:26.870384+0000) 2022-09-08T00:01:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:26.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:26.779858+0000) 2022-09-08T00:01:27.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:27 smithi055 ceph-mon[97033]: pgmap v4574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 97 B/s wr, 8 op/s 2022-09-08T00:01:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:27 smithi055 ceph-mon[97033]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-08T00:01:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:27 smithi055 ceph-mon[97033]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-08T00:01:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:27 smithi059 ceph-mon[97450]: pgmap v4574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 97 B/s wr, 8 op/s 2022-09-08T00:01:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:27 smithi059 ceph-mon[97450]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-08T00:01:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:27 smithi059 ceph-mon[97450]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-08T00:01:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:27.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:27.780039+0000) 2022-09-08T00:01:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:27.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:27.870532+0000) 2022-09-08T00:01:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:28 smithi055 ceph-mon[97033]: pgmap v4575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:28 smithi059 ceph-mon[97450]: pgmap v4575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:28.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:28.870684+0000) 2022-09-08T00:01:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:28.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:28.780267+0000) 2022-09-08T00:01:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:29.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:29.780497+0000) 2022-09-08T00:01:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:29.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:29.870835+0000) 2022-09-08T00:01:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:30 smithi055 ceph-mon[97033]: pgmap v4576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:30 smithi059 ceph-mon[97450]: pgmap v4576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:30.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:30.870993+0000) 2022-09-08T00:01:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:30.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:30.780769+0000) 2022-09-08T00:01:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:31.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:31.871177+0000) 2022-09-08T00:01:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:31.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:31.780913+0000) 2022-09-08T00:01:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:32 smithi059 ceph-mon[97450]: pgmap v4577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 9 op/s 2022-09-08T00:01:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:32 smithi055 ceph-mon[97033]: pgmap v4577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 9 op/s 2022-09-08T00:01:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:32.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:32.781158+0000) 2022-09-08T00:01:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:32.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:32.871327+0000) 2022-09-08T00:01:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:33.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:33.871454+0000) 2022-09-08T00:01:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:33.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:33.781418+0000) 2022-09-08T00:01:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:34 smithi055 ceph-mon[97033]: pgmap v4578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-08T00:01:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:34 smithi059 ceph-mon[97450]: pgmap v4578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-08T00:01:35.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:01:34.699+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:01:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:34.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:34.871678+0000) 2022-09-08T00:01:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:34.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:34.781671+0000) 2022-09-08T00:01:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:35.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:35.871848+0000) 2022-09-08T00:01:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:35.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:35.781874+0000) 2022-09-08T00:01:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:36 smithi055 ceph-mon[97033]: pgmap v4579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 9 op/s 2022-09-08T00:01:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:36 smithi059 ceph-mon[97450]: pgmap v4579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 9 op/s 2022-09-08T00:01:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:36.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:36.872028+0000) 2022-09-08T00:01:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:36.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:36.782099+0000) 2022-09-08T00:01:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:37.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:37.872226+0000) 2022-09-08T00:01:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:37.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:37.782323+0000) 2022-09-08T00:01:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:38 smithi055 ceph-mon[97033]: pgmap v4580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-08T00:01:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:38 smithi059 ceph-mon[97450]: pgmap v4580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-08T00:01:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:38.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:38.872452+0000) 2022-09-08T00:01:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:38.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:38.782520+0000) 2022-09-08T00:01:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:39.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:39.872706+0000) 2022-09-08T00:01:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:39.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:39.782751+0000) 2022-09-08T00:01:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:40 smithi055 ceph-mon[97033]: pgmap v4581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-08T00:01:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:40 smithi059 ceph-mon[97450]: pgmap v4581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-08T00:01:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:40.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:40.872846+0000) 2022-09-08T00:01:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:40.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:40.782994+0000) 2022-09-08T00:01:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:41.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:41.873063+0000) 2022-09-08T00:01:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:41.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:41.783159+0000) 2022-09-08T00:01:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:42 smithi055 ceph-mon[97033]: pgmap v4582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 9 op/s 2022-09-08T00:01:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:42 smithi059 ceph-mon[97450]: pgmap v4582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 9 op/s 2022-09-08T00:01:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:42.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:42.783392+0000) 2022-09-08T00:01:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:42.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:42.873229+0000) 2022-09-08T00:01:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:43.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:43.873458+0000) 2022-09-08T00:01:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:43.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:43.783594+0000) 2022-09-08T00:01:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:44 smithi055 ceph-mon[97033]: pgmap v4583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:44 smithi059 ceph-mon[97450]: pgmap v4583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:45.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:44.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:44.873729+0000) 2022-09-08T00:01:45.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:44.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:44.783706+0000) 2022-09-08T00:01:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:45.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:45.873935+0000) 2022-09-08T00:01:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:45.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:45.783881+0000) 2022-09-08T00:01:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:46 smithi055 ceph-mon[97033]: pgmap v4584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:01:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:46 smithi059 ceph-mon[97450]: pgmap v4584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:01:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:46.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:46.874143+0000) 2022-09-08T00:01:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:46.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:46.784081+0000) 2022-09-08T00:01:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:47.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:47.874358+0000) 2022-09-08T00:01:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:47.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:47.784282+0000) 2022-09-08T00:01:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:48 smithi055 ceph-mon[97033]: pgmap v4585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:48 smithi059 ceph-mon[97450]: pgmap v4585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:48.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:48.874588+0000) 2022-09-08T00:01:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:48.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:48.784511+0000) 2022-09-08T00:01:50.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:01:49.699+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:01:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:49.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:49.874807+0000) 2022-09-08T00:01:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:49.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:49.784753+0000) 2022-09-08T00:01:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:50 smithi055 ceph-mon[97033]: pgmap v4586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:50 smithi059 ceph-mon[97450]: pgmap v4586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:50.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:50.874982+0000) 2022-09-08T00:01:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:50.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:50.784966+0000) 2022-09-08T00:01:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:51.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:51.785207+0000) 2022-09-08T00:01:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:51.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:51.875221+0000) 2022-09-08T00:01:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:52 smithi055 ceph-mon[97033]: pgmap v4587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:01:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:52 smithi059 ceph-mon[97450]: pgmap v4587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:01:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:52.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:52.875463+0000) 2022-09-08T00:01:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:52.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:52.785462+0000) 2022-09-08T00:01:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:53.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:53.875746+0000) 2022-09-08T00:01:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:53.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:53.785618+0000) 2022-09-08T00:01:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:54 smithi055 ceph-mon[97033]: pgmap v4588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:54 smithi059 ceph-mon[97450]: pgmap v4588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:54.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:54.875934+0000) 2022-09-08T00:01:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:54.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:54.785792+0000) 2022-09-08T00:01:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:55.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:55.876098+0000) 2022-09-08T00:01:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:55.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:55.786015+0000) 2022-09-08T00:01:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:56 smithi055 ceph-mon[97033]: pgmap v4589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:01:56.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:56 smithi059 ceph-mon[97450]: pgmap v4589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:01:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:56.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:56.876282+0000) 2022-09-08T00:01:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:56.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:56.786238+0000) 2022-09-08T00:01:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:57.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:57.876445+0000) 2022-09-08T00:01:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:57.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:57.786496+0000) 2022-09-08T00:01:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:01:58 smithi055 ceph-mon[97033]: pgmap v4590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:01:58 smithi059 ceph-mon[97450]: pgmap v4590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:01:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:58.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:58.876706+0000) 2022-09-08T00:01:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:58.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:58.786760+0000) 2022-09-08T00:02:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:01:59.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:59.876912+0000) 2022-09-08T00:02:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:01:59.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:01:59.786912+0000) 2022-09-08T00:02:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:00 smithi055 ceph-mon[97033]: pgmap v4591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:00 smithi059 ceph-mon[97450]: pgmap v4591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:00.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:00.877087+0000) 2022-09-08T00:02:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:00.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:00.787151+0000) 2022-09-08T00:02:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:01.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:01.877263+0000) 2022-09-08T00:02:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:01.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:01.787257+0000) 2022-09-08T00:02:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:02 smithi055 ceph-mon[97033]: pgmap v4592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:02:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:02 smithi059 ceph-mon[97450]: pgmap v4592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:02:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:02.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:02.877478+0000) 2022-09-08T00:02:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:02.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:02.787493+0000) 2022-09-08T00:02:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:03.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:03.877674+0000) 2022-09-08T00:02:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:03.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:03.787778+0000) 2022-09-08T00:02:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:04 smithi055 ceph-mon[97033]: pgmap v4593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:04 smithi059 ceph-mon[97450]: pgmap v4593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:04.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:04.877909+0000) 2022-09-08T00:02:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:04.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:04.787952+0000) 2022-09-08T00:02:05.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:02:04.716+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:02:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:02:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:02:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:02:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:02:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:05.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:05.878108+0000) 2022-09-08T00:02:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:05.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:05.788158+0000) 2022-09-08T00:02:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:06 smithi055 ceph-mon[97033]: pgmap v4594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:02:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:06 smithi059 ceph-mon[97450]: pgmap v4594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:02:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:06.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:06.878328+0000) 2022-09-08T00:02:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:06.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:06.788422+0000) 2022-09-08T00:02:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:07.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:07.878528+0000) 2022-09-08T00:02:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:07.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:07.788721+0000) 2022-09-08T00:02:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:08 smithi055 ceph-mon[97033]: pgmap v4595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:02:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:08 smithi059 ceph-mon[97450]: pgmap v4595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:02:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:08.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:08.878768+0000) 2022-09-08T00:02:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:08.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:08.788914+0000) 2022-09-08T00:02:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:09 smithi055 ceph-mon[97033]: osdmap e375: 12 total, 11 up, 11 in 2022-09-08T00:02:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:09 smithi059 ceph-mon[97450]: osdmap e375: 12 total, 11 up, 11 in 2022-09-08T00:02:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:09.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:09.879032+0000) 2022-09-08T00:02:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:09.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:09.789138+0000) 2022-09-08T00:02:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:10 smithi055 ceph-mon[97033]: pgmap v4597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:02:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:10 smithi059 ceph-mon[97450]: pgmap v4597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:02:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:10.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:10.879251+0000) 2022-09-08T00:02:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:10.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:10.789368+0000) 2022-09-08T00:02:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:11 smithi059 ceph-mon[97450]: pgmap v4598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:02:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:02:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:02:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:02:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:11 smithi055 ceph-mon[97033]: pgmap v4598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:02:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:02:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:02:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:02:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:11.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:11.879510+0000) 2022-09-08T00:02:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:11.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:11.789612+0000) 2022-09-08T00:02:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:02:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:12.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:12.879748+0000) 2022-09-08T00:02:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:12.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:12.789850+0000) 2022-09-08T00:02:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:02:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:13 smithi059 ceph-mon[97450]: pgmap v4599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:02:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:13.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:13.790074+0000) 2022-09-08T00:02:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:13.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:13.879960+0000) 2022-09-08T00:02:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:13 smithi055 ceph-mon[97033]: pgmap v4599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:02:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:14.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:14.880201+0000) 2022-09-08T00:02:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:14.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:14.790380+0000) 2022-09-08T00:02:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:15.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:15.880421+0000) 2022-09-08T00:02:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:15.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:15.790615+0000) 2022-09-08T00:02:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:16 smithi055 ceph-mon[97033]: pgmap v4600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:02:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:16 smithi055 ceph-mon[97033]: osdmap e376: 12 total, 11 up, 11 in 2022-09-08T00:02:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:16 smithi059 ceph-mon[97450]: pgmap v4600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:02:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:16 smithi059 ceph-mon[97450]: osdmap e376: 12 total, 11 up, 11 in 2022-09-08T00:02:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:16.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:16.880666+0000) 2022-09-08T00:02:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:16.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:16.790785+0000) 2022-09-08T00:02:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:17.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:17.880855+0000) 2022-09-08T00:02:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:17.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:17.791007+0000) 2022-09-08T00:02:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:18 smithi055 ceph-mon[97033]: pgmap v4602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 233 B/s wr, 7 op/s 2022-09-08T00:02:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:18 smithi059 ceph-mon[97450]: pgmap v4602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 233 B/s wr, 7 op/s 2022-09-08T00:02:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:18.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:18.881028+0000) 2022-09-08T00:02:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:18.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:18.791231+0000) 2022-09-08T00:02:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:19.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:19.881264+0000) 2022-09-08T00:02:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:19.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:19.791468+0000) 2022-09-08T00:02:20.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:02:19.731+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:02:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:20 smithi055 ceph-mon[97033]: pgmap v4603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:02:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:20 smithi059 ceph-mon[97450]: pgmap v4603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:02:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:20.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:20.881494+0000) 2022-09-08T00:02:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:20.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:20.791693+0000) 2022-09-08T00:02:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:21.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:21.881692+0000) 2022-09-08T00:02:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:21.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:21.791860+0000) 2022-09-08T00:02:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:22 smithi055 ceph-mon[97033]: pgmap v4604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:02:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:22 smithi059 ceph-mon[97450]: pgmap v4604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:02:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:22.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:22.881890+0000) 2022-09-08T00:02:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:22.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:22.792071+0000) 2022-09-08T00:02:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:23.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:23.882112+0000) 2022-09-08T00:02:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:23.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:23.792301+0000) 2022-09-08T00:02:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:24 smithi055 ceph-mon[97033]: pgmap v4605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:02:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:24 smithi059 ceph-mon[97450]: pgmap v4605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:02:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:24.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:24.882324+0000) 2022-09-08T00:02:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:24.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:24.792506+0000) 2022-09-08T00:02:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:25.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:25.882507+0000) 2022-09-08T00:02:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:25.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:25.792705+0000) 2022-09-08T00:02:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:26 smithi055 ceph-mon[97033]: pgmap v4606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:02:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:26 smithi059 ceph-mon[97450]: pgmap v4606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:02:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:26.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:26.882709+0000) 2022-09-08T00:02:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:26.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:26.792924+0000) 2022-09-08T00:02:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:27.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:27.882915+0000) 2022-09-08T00:02:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:27.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:27.793153+0000) 2022-09-08T00:02:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:28 smithi055 ceph-mon[97033]: pgmap v4607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:02:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:28 smithi059 ceph-mon[97450]: pgmap v4607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:02:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:28.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:28.883156+0000) 2022-09-08T00:02:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:28.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:28.793417+0000) 2022-09-08T00:02:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:29.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:29.793653+0000) 2022-09-08T00:02:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:29.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:29.883351+0000) 2022-09-08T00:02:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:30 smithi055 ceph-mon[97033]: pgmap v4608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:30 smithi059 ceph-mon[97450]: pgmap v4608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:30.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:30.883584+0000) 2022-09-08T00:02:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:30.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:30.793901+0000) 2022-09-08T00:02:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:31.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:31.883814+0000) 2022-09-08T00:02:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:31.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:31.794140+0000) 2022-09-08T00:02:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:32 smithi055 ceph-mon[97033]: pgmap v4609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:02:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:32 smithi059 ceph-mon[97450]: pgmap v4609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:02:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:32.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:32.883979+0000) 2022-09-08T00:02:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:32.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:32.794416+0000) 2022-09-08T00:02:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:33.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:33.884126+0000) 2022-09-08T00:02:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:33.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:33.794633+0000) 2022-09-08T00:02:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:34 smithi055 ceph-mon[97033]: pgmap v4610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:34 smithi059 ceph-mon[97450]: pgmap v4610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:35.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:02:34.736+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:02:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:34.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:34.884311+0000) 2022-09-08T00:02:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:34.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:34.794863+0000) 2022-09-08T00:02:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:35.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:35.884480+0000) 2022-09-08T00:02:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:35.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:35.795104+0000) 2022-09-08T00:02:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:36 smithi055 ceph-mon[97033]: pgmap v4611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:02:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:36 smithi059 ceph-mon[97450]: pgmap v4611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:02:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:36.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:36.884686+0000) 2022-09-08T00:02:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:36.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:36.795319+0000) 2022-09-08T00:02:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:37.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:37.884815+0000) 2022-09-08T00:02:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:37.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:37.795538+0000) 2022-09-08T00:02:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:38 smithi055 ceph-mon[97033]: pgmap v4612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:38 smithi059 ceph-mon[97450]: pgmap v4612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:38.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:38.885064+0000) 2022-09-08T00:02:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:38.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:38.795807+0000) 2022-09-08T00:02:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:39.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:39.885376+0000) 2022-09-08T00:02:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:39.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:39.795937+0000) 2022-09-08T00:02:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:40 smithi055 ceph-mon[97033]: pgmap v4613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:40 smithi059 ceph-mon[97450]: pgmap v4613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:40.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:40.885529+0000) 2022-09-08T00:02:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:40.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:40.796189+0000) 2022-09-08T00:02:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:41.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:41.796385+0000) 2022-09-08T00:02:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:41.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:41.885731+0000) 2022-09-08T00:02:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:42 smithi055 ceph-mon[97033]: pgmap v4614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:02:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:42 smithi059 ceph-mon[97450]: pgmap v4614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:02:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:42.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:42.796570+0000) 2022-09-08T00:02:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:42.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:42.885911+0000) 2022-09-08T00:02:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:43.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:43.886083+0000) 2022-09-08T00:02:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:43.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:43.796816+0000) 2022-09-08T00:02:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:44 smithi055 ceph-mon[97033]: pgmap v4615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:44 smithi059 ceph-mon[97450]: pgmap v4615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:44.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:44.886280+0000) 2022-09-08T00:02:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:44.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:44.796990+0000) 2022-09-08T00:02:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:45.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:45.886451+0000) 2022-09-08T00:02:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:45.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:45.797183+0000) 2022-09-08T00:02:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:46 smithi055 ceph-mon[97033]: pgmap v4616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:02:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:46 smithi059 ceph-mon[97450]: pgmap v4616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:02:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:46.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:46.886614+0000) 2022-09-08T00:02:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:46.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:46.797420+0000) 2022-09-08T00:02:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:47.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:47.886744+0000) 2022-09-08T00:02:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:47.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:47.797682+0000) 2022-09-08T00:02:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:48 smithi055 ceph-mon[97033]: pgmap v4617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:48 smithi059 ceph-mon[97450]: pgmap v4617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:48.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:48.886890+0000) 2022-09-08T00:02:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:48.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:48.797910+0000) 2022-09-08T00:02:50.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:02:49.750+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:02:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:49.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:49.887063+0000) 2022-09-08T00:02:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:49.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:49.798161+0000) 2022-09-08T00:02:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:50 smithi055 ceph-mon[97033]: pgmap v4618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:50 smithi059 ceph-mon[97450]: pgmap v4618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:50.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:50.887267+0000) 2022-09-08T00:02:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:50.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:50.798393+0000) 2022-09-08T00:02:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:51.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:51.887515+0000) 2022-09-08T00:02:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:51.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:51.798594+0000) 2022-09-08T00:02:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:52 smithi055 ceph-mon[97033]: pgmap v4619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:02:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:52 smithi059 ceph-mon[97450]: pgmap v4619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:02:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:52.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:52.887694+0000) 2022-09-08T00:02:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:52.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:52.798757+0000) 2022-09-08T00:02:54.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:53.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:53.887869+0000) 2022-09-08T00:02:54.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:53.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:53.798956+0000) 2022-09-08T00:02:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:54 smithi055 ceph-mon[97033]: pgmap v4620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:54 smithi059 ceph-mon[97450]: pgmap v4620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:54.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:54.888009+0000) 2022-09-08T00:02:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:54.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:54.799187+0000) 2022-09-08T00:02:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:55.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:55.888212+0000) 2022-09-08T00:02:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:55.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:55.799448+0000) 2022-09-08T00:02:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:56 smithi055 ceph-mon[97033]: pgmap v4621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:02:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:56 smithi059 ceph-mon[97450]: pgmap v4621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:02:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:56.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:56.888427+0000) 2022-09-08T00:02:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:56.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:56.799711+0000) 2022-09-08T00:02:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:57.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:57.888670+0000) 2022-09-08T00:02:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:57.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:57.799944+0000) 2022-09-08T00:02:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:02:58 smithi055 ceph-mon[97033]: pgmap v4622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:02:58 smithi059 ceph-mon[97450]: pgmap v4622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:02:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:58.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:58.888887+0000) 2022-09-08T00:02:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:58.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:58.800250+0000) 2022-09-08T00:03:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:02:59.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:59.889042+0000) 2022-09-08T00:03:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:02:59.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:02:59.800499+0000) 2022-09-08T00:03:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:00 smithi055 ceph-mon[97033]: pgmap v4623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:03:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:00 smithi059 ceph-mon[97450]: pgmap v4623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:03:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:00.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:00.889286+0000) 2022-09-08T00:03:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:00.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:00.800739+0000) 2022-09-08T00:03:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:01.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:01.889547+0000) 2022-09-08T00:03:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:01.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:01.800940+0000) 2022-09-08T00:03:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:02 smithi055 ceph-mon[97033]: pgmap v4624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:03:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:02 smithi059 ceph-mon[97450]: pgmap v4624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:03:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:02.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:02.889775+0000) 2022-09-08T00:03:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:02.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:02.801164+0000) 2022-09-08T00:03:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:03.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:03.890034+0000) 2022-09-08T00:03:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:03.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:03.801376+0000) 2022-09-08T00:03:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:04 smithi055 ceph-mon[97033]: pgmap v4625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:03:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:04 smithi059 ceph-mon[97450]: pgmap v4625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:03:05.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:03:04.766+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:03:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:04.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:04.890259+0000) 2022-09-08T00:03:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:04.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:04.801595+0000) 2022-09-08T00:03:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:03:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:03:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:03:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:03:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:05.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:05.890487+0000) 2022-09-08T00:03:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:05.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:05.801851+0000) 2022-09-08T00:03:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:06 smithi055 ceph-mon[97033]: pgmap v4626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:03:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:06 smithi059 ceph-mon[97450]: pgmap v4626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:03:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:06.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:06.890756+0000) 2022-09-08T00:03:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:06.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:06.802091+0000) 2022-09-08T00:03:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:07.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:07.890894+0000) 2022-09-08T00:03:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:07.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:07.802319+0000) 2022-09-08T00:03:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:08 smithi055 ceph-mon[97033]: pgmap v4627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:03:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:08 smithi059 ceph-mon[97450]: pgmap v4627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:03:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:08.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:08.891095+0000) 2022-09-08T00:03:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:08.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:08.802569+0000) 2022-09-08T00:03:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:09.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:09.802704+0000) 2022-09-08T00:03:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:09.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:09.891300+0000) 2022-09-08T00:03:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:10 smithi055 ceph-mon[97033]: pgmap v4628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:03:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:10 smithi055 ceph-mon[97033]: osdmap e377: 12 total, 11 up, 11 in 2022-09-08T00:03:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:10 smithi059 ceph-mon[97450]: pgmap v4628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:03:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:10 smithi059 ceph-mon[97450]: osdmap e377: 12 total, 11 up, 11 in 2022-09-08T00:03:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:10.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:10.891505+0000) 2022-09-08T00:03:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:10.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:10.802890+0000) 2022-09-08T00:03:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:11.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:11.891742+0000) 2022-09-08T00:03:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:11.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:11.803127+0000) 2022-09-08T00:03:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:12 smithi055 ceph-mon[97033]: pgmap v4630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:03:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:03:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:03:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:03:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:12 smithi059 ceph-mon[97450]: pgmap v4630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:03:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:03:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:03:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:03:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:12.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:12.891919+0000) 2022-09-08T00:03:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:12.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:12.803378+0000) 2022-09-08T00:03:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:03:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:03:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:13.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:13.892123+0000) 2022-09-08T00:03:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:13.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:13.803614+0000) 2022-09-08T00:03:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:14 smithi055 ceph-mon[97033]: pgmap v4631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:03:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:14 smithi059 ceph-mon[97450]: pgmap v4631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:03:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:14.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:14.892297+0000) 2022-09-08T00:03:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:14.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:14.803875+0000) 2022-09-08T00:03:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:15.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:15.892526+0000) 2022-09-08T00:03:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:15.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:15.804118+0000) 2022-09-08T00:03:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:16 smithi055 ceph-mon[97033]: pgmap v4632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:03:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:16 smithi059 ceph-mon[97450]: pgmap v4632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:03:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:16.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:16.892768+0000) 2022-09-08T00:03:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:16.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:16.804352+0000) 2022-09-08T00:03:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:17.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:17.892899+0000) 2022-09-08T00:03:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:17.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:17.804599+0000) 2022-09-08T00:03:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:18 smithi055 ceph-mon[97033]: pgmap v4633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:03:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:18 smithi059 ceph-mon[97450]: pgmap v4633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:03:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:18.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:18.893097+0000) 2022-09-08T00:03:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:18.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:18.804829+0000) 2022-09-08T00:03:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:19.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:19.893343+0000) 2022-09-08T00:03:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:19.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:19.805065+0000) 2022-09-08T00:03:20.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:03:19.773+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:03:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:20 smithi055 ceph-mon[97033]: pgmap v4634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:03:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:20 smithi055 ceph-mon[97033]: osdmap e378: 12 total, 11 up, 11 in 2022-09-08T00:03:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:20 smithi059 ceph-mon[97450]: pgmap v4634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:03:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:20 smithi059 ceph-mon[97450]: osdmap e378: 12 total, 11 up, 11 in 2022-09-08T00:03:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:20.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:20.893562+0000) 2022-09-08T00:03:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:20.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:20.805241+0000) 2022-09-08T00:03:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:21.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:21.893782+0000) 2022-09-08T00:03:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:21.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:21.805463+0000) 2022-09-08T00:03:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:22 smithi055 ceph-mon[97033]: pgmap v4636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:03:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:22 smithi059 ceph-mon[97450]: pgmap v4636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:03:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:22.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:22.893997+0000) 2022-09-08T00:03:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:22.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:22.805743+0000) 2022-09-08T00:03:23.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:23 smithi055 ceph-mon[97033]: pgmap v4637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:03:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:23 smithi059 ceph-mon[97450]: pgmap v4637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:03:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:23.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:23.894225+0000) 2022-09-08T00:03:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:23.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:23.805997+0000) 2022-09-08T00:03:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:24.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:24.894358+0000) 2022-09-08T00:03:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:24.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:24.806276+0000) 2022-09-08T00:03:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:25.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:25.894562+0000) 2022-09-08T00:03:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:25.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:25.806453+0000) 2022-09-08T00:03:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:26 smithi055 ceph-mon[97033]: pgmap v4638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:03:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:26 smithi059 ceph-mon[97450]: pgmap v4638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:03:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:26.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:26.894802+0000) 2022-09-08T00:03:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:26.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:26.806664+0000) 2022-09-08T00:03:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:27.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:27.895013+0000) 2022-09-08T00:03:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:27.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:27.806828+0000) 2022-09-08T00:03:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:28 smithi055 ceph-mon[97033]: pgmap v4639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:03:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:28 smithi059 ceph-mon[97450]: pgmap v4639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:03:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:28.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:28.895154+0000) 2022-09-08T00:03:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:28.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:28.807038+0000) 2022-09-08T00:03:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:29.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:29.895356+0000) 2022-09-08T00:03:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:29.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:29.807217+0000) 2022-09-08T00:03:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:30 smithi055 ceph-mon[97033]: pgmap v4640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:03:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:30 smithi059 ceph-mon[97450]: pgmap v4640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:03:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:30.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:30.895588+0000) 2022-09-08T00:03:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:30.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:30.807475+0000) 2022-09-08T00:03:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:31.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:31.895724+0000) 2022-09-08T00:03:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:31.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:31.807712+0000) 2022-09-08T00:03:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:32 smithi055 ceph-mon[97033]: pgmap v4641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:03:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:32 smithi059 ceph-mon[97450]: pgmap v4641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:03:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:32.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:32.895958+0000) 2022-09-08T00:03:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:32.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:32.807953+0000) 2022-09-08T00:03:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:33.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:33.896190+0000) 2022-09-08T00:03:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:33.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:33.808198+0000) 2022-09-08T00:03:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:34 smithi055 ceph-mon[97033]: pgmap v4642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:03:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:34 smithi059 ceph-mon[97450]: pgmap v4642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:03:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:34.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:34.896421+0000) 2022-09-08T00:03:35.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:03:34.789+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:03:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:34.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:34.808452+0000) 2022-09-08T00:03:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:35.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:35.896704+0000) 2022-09-08T00:03:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:35.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:35.808680+0000) 2022-09-08T00:03:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:36 smithi055 ceph-mon[97033]: pgmap v4643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:03:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:36 smithi059 ceph-mon[97450]: pgmap v4643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:03:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:36.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:36.808832+0000) 2022-09-08T00:03:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:36.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:36.896876+0000) 2022-09-08T00:03:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:37.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:37.809002+0000) 2022-09-08T00:03:38.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:37.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:37.897120+0000) 2022-09-08T00:03:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:38 smithi055 ceph-mon[97033]: pgmap v4644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:03:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:38 smithi059 ceph-mon[97450]: pgmap v4644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:03:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:38.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:38.897289+0000) 2022-09-08T00:03:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:38.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:38.809165+0000) 2022-09-08T00:03:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:39.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:39.897495+0000) 2022-09-08T00:03:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:39.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:39.809346+0000) 2022-09-08T00:03:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:40 smithi055 ceph-mon[97033]: pgmap v4645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:03:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:40 smithi059 ceph-mon[97450]: pgmap v4645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:03:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:40.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:40.897741+0000) 2022-09-08T00:03:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:40.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:40.809568+0000) 2022-09-08T00:03:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:41.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:41.897962+0000) 2022-09-08T00:03:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:41.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:41.809784+0000) 2022-09-08T00:03:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:42 smithi055 ceph-mon[97033]: pgmap v4646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:03:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:42 smithi059 ceph-mon[97450]: pgmap v4646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:03:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:42.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:42.898166+0000) 2022-09-08T00:03:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:42.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:42.810033+0000) 2022-09-08T00:03:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:43.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:43.898373+0000) 2022-09-08T00:03:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:43.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:43.810211+0000) 2022-09-08T00:03:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:44 smithi055 ceph-mon[97033]: pgmap v4647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:03:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:44 smithi059 ceph-mon[97450]: pgmap v4647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:03:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:44.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:44.898588+0000) 2022-09-08T00:03:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:44.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:44.810344+0000) 2022-09-08T00:03:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:45.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:45.898810+0000) 2022-09-08T00:03:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:45.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:45.810552+0000) 2022-09-08T00:03:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:46 smithi055 ceph-mon[97033]: pgmap v4648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:03:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:46 smithi059 ceph-mon[97450]: pgmap v4648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:03:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:46.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:46.899059+0000) 2022-09-08T00:03:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:46.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:46.810813+0000) 2022-09-08T00:03:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:47.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:47.899309+0000) 2022-09-08T00:03:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:47.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:47.811020+0000) 2022-09-08T00:03:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:48 smithi055 ceph-mon[97033]: pgmap v4649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:03:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:48 smithi059 ceph-mon[97450]: pgmap v4649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:03:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:48.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:48.811231+0000) 2022-09-08T00:03:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:48.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:48.899480+0000) 2022-09-08T00:03:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:49.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:49.899761+0000) 2022-09-08T00:03:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:49.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:49.811453+0000) 2022-09-08T00:03:50.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:03:49.792+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:03:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:50 smithi055 ceph-mon[97033]: pgmap v4650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:03:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:50 smithi059 ceph-mon[97450]: pgmap v4650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:03:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:50.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:50.811720+0000) 2022-09-08T00:03:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:50.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:50.899952+0000) 2022-09-08T00:03:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:51.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:51.900192+0000) 2022-09-08T00:03:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:51.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:51.811880+0000) 2022-09-08T00:03:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:52 smithi055 ceph-mon[97033]: pgmap v4651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:03:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:52 smithi059 ceph-mon[97450]: pgmap v4651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:03:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:52.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:52.900403+0000) 2022-09-08T00:03:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:52.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:52.812115+0000) 2022-09-08T00:03:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:53.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:53.900622+0000) 2022-09-08T00:03:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:53.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:53.812405+0000) 2022-09-08T00:03:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:54 smithi055 ceph-mon[97033]: pgmap v4652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:03:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:54 smithi059 ceph-mon[97450]: pgmap v4652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:03:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:54.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:54.812599+0000) 2022-09-08T00:03:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:54.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:54.900829+0000) 2022-09-08T00:03:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:55.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:55.812800+0000) 2022-09-08T00:03:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:55.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:55.901047+0000) 2022-09-08T00:03:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:56 smithi055 ceph-mon[97033]: pgmap v4653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:03:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:56 smithi059 ceph-mon[97450]: pgmap v4653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:03:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:56.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:56.901287+0000) 2022-09-08T00:03:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:56.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:56.813037+0000) 2022-09-08T00:03:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:57.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:57.901515+0000) 2022-09-08T00:03:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:57.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:57.813262+0000) 2022-09-08T00:03:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:03:58 smithi055 ceph-mon[97033]: pgmap v4654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:03:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:03:58 smithi059 ceph-mon[97450]: pgmap v4654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:03:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:58.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:58.901707+0000) 2022-09-08T00:03:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:58.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:58.813534+0000) 2022-09-08T00:04:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:03:59.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:59.901928+0000) 2022-09-08T00:04:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:03:59.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:03:59.813733+0000) 2022-09-08T00:04:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:00 smithi055 ceph-mon[97033]: pgmap v4655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:00 smithi059 ceph-mon[97450]: pgmap v4655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:00.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:00.902084+0000) 2022-09-08T00:04:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:00.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:00.813903+0000) 2022-09-08T00:04:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:01.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:01.902301+0000) 2022-09-08T00:04:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:01.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:01.814100+0000) 2022-09-08T00:04:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:02 smithi055 ceph-mon[97033]: pgmap v4656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:04:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:02 smithi059 ceph-mon[97450]: pgmap v4656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:04:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:02.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:02.902547+0000) 2022-09-08T00:04:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:02.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:02.814351+0000) 2022-09-08T00:04:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:03.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:03.902776+0000) 2022-09-08T00:04:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:03.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:03.814589+0000) 2022-09-08T00:04:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:04 smithi055 ceph-mon[97033]: pgmap v4657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:04 smithi059 ceph-mon[97450]: pgmap v4657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:04.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:04.814788+0000) 2022-09-08T00:04:05.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:04:04.794+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:04:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:04.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:04.902953+0000) 2022-09-08T00:04:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:04:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:04:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:04:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:04:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:05.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:05.903199+0000) 2022-09-08T00:04:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:05.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:05.814917+0000) 2022-09-08T00:04:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:06 smithi055 ceph-mon[97033]: pgmap v4658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:04:06.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:06 smithi059 ceph-mon[97450]: pgmap v4658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:04:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:06.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:06.903466+0000) 2022-09-08T00:04:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:06.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:06.815117+0000) 2022-09-08T00:04:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:07 smithi055 ceph-mon[97033]: pgmap v4659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:04:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:07 smithi059 ceph-mon[97450]: pgmap v4659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:04:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:07.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:07.903713+0000) 2022-09-08T00:04:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:07.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:07.815368+0000) 2022-09-08T00:04:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:08.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:08.903963+0000) 2022-09-08T00:04:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:08.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:08.815599+0000) 2022-09-08T00:04:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:09 smithi055 ceph-mon[97033]: osdmap e379: 12 total, 11 up, 11 in 2022-09-08T00:04:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:09 smithi055 ceph-mon[97033]: pgmap v4661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:04:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:09 smithi059 ceph-mon[97450]: osdmap e379: 12 total, 11 up, 11 in 2022-09-08T00:04:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:09 smithi059 ceph-mon[97450]: pgmap v4661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:04:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:09.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:09.904236+0000) 2022-09-08T00:04:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:09.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:09.815787+0000) 2022-09-08T00:04:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:10.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:10.904472+0000) 2022-09-08T00:04:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:10.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:10.815973+0000) 2022-09-08T00:04:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:11.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:11.904658+0000) 2022-09-08T00:04:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:11.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:11.816192+0000) 2022-09-08T00:04:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:12 smithi055 ceph-mon[97033]: pgmap v4662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:04:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:12 smithi059 ceph-mon[97450]: pgmap v4662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:04:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:12.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:12.904908+0000) 2022-09-08T00:04:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:12.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:12.816388+0000) 2022-09-08T00:04:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:04:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:04:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:04:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:04:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:04:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:04:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:04:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:04:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:13.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:13.816533+0000) 2022-09-08T00:04:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:13.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:13.905139+0000) 2022-09-08T00:04:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:14 smithi055 ceph-mon[97033]: pgmap v4663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:04:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:14 smithi059 ceph-mon[97450]: pgmap v4663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:04:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:14.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:14.905363+0000) 2022-09-08T00:04:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:14.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:14.816678+0000) 2022-09-08T00:04:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:15.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:15.905611+0000) 2022-09-08T00:04:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:15.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:15.816862+0000) 2022-09-08T00:04:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:16 smithi055 ceph-mon[97033]: pgmap v4664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:04:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:16 smithi055 ceph-mon[97033]: osdmap e380: 12 total, 11 up, 11 in 2022-09-08T00:04:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:16 smithi059 ceph-mon[97450]: pgmap v4664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:04:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:16 smithi059 ceph-mon[97450]: osdmap e380: 12 total, 11 up, 11 in 2022-09-08T00:04:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:16.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:16.905810+0000) 2022-09-08T00:04:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:16.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:16.817101+0000) 2022-09-08T00:04:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:17.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:17.906024+0000) 2022-09-08T00:04:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:17.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:17.817343+0000) 2022-09-08T00:04:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:18 smithi055 ceph-mon[97033]: pgmap v4666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 233 B/s wr, 7 op/s 2022-09-08T00:04:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:18 smithi059 ceph-mon[97450]: pgmap v4666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 233 B/s wr, 7 op/s 2022-09-08T00:04:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:18.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:18.906202+0000) 2022-09-08T00:04:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:18.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:18.817492+0000) 2022-09-08T00:04:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:19.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:19.817713+0000) 2022-09-08T00:04:20.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:04:19.796+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:04:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:19.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:19.906460+0000) 2022-09-08T00:04:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:20 smithi055 ceph-mon[97033]: pgmap v4667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:04:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:20 smithi059 ceph-mon[97450]: pgmap v4667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:04:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:20.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:20.817952+0000) 2022-09-08T00:04:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:20.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:20.906773+0000) 2022-09-08T00:04:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:21.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:21.906994+0000) 2022-09-08T00:04:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:21.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:21.818190+0000) 2022-09-08T00:04:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:22 smithi055 ceph-mon[97033]: pgmap v4668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:04:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:22 smithi059 ceph-mon[97450]: pgmap v4668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:04:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:22.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:22.907230+0000) 2022-09-08T00:04:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:22.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:22.818455+0000) 2022-09-08T00:04:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:23.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:23.907420+0000) 2022-09-08T00:04:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:23.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:23.818688+0000) 2022-09-08T00:04:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:24 smithi055 ceph-mon[97033]: pgmap v4669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:04:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:24 smithi059 ceph-mon[97450]: pgmap v4669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:04:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:24.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:24.907684+0000) 2022-09-08T00:04:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:24.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:24.818856+0000) 2022-09-08T00:04:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:25.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:25.907924+0000) 2022-09-08T00:04:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:25.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:25.818974+0000) 2022-09-08T00:04:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:26 smithi055 ceph-mon[97033]: pgmap v4670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:04:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:26 smithi059 ceph-mon[97450]: pgmap v4670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:04:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:26.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:26.908195+0000) 2022-09-08T00:04:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:26.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:26.819113+0000) 2022-09-08T00:04:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:27.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:27.908398+0000) 2022-09-08T00:04:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:27.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:27.819325+0000) 2022-09-08T00:04:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:28 smithi055 ceph-mon[97033]: pgmap v4671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:04:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:28 smithi059 ceph-mon[97450]: pgmap v4671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:04:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:28.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:28.819574+0000) 2022-09-08T00:04:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:28.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:28.908612+0000) 2022-09-08T00:04:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:29.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:29.908803+0000) 2022-09-08T00:04:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:29.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:29.819709+0000) 2022-09-08T00:04:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:30 smithi055 ceph-mon[97033]: pgmap v4672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:30 smithi059 ceph-mon[97450]: pgmap v4672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:30.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:30.908971+0000) 2022-09-08T00:04:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:30.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:30.819862+0000) 2022-09-08T00:04:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:31.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:31.909185+0000) 2022-09-08T00:04:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:31.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:31.820048+0000) 2022-09-08T00:04:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:32 smithi055 ceph-mon[97033]: pgmap v4673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:04:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:32 smithi059 ceph-mon[97450]: pgmap v4673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:04:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:32.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:32.820291+0000) 2022-09-08T00:04:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:32.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:32.909410+0000) 2022-09-08T00:04:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:33.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:33.909669+0000) 2022-09-08T00:04:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:33.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:33.820593+0000) 2022-09-08T00:04:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:34 smithi055 ceph-mon[97033]: pgmap v4674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:04:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:34 smithi059 ceph-mon[97450]: pgmap v4674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:04:35.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:04:34.799+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:04:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:34.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:34.909840+0000) 2022-09-08T00:04:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:34.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:34.820765+0000) 2022-09-08T00:04:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:35.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:35.820916+0000) 2022-09-08T00:04:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:35.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:35.910077+0000) 2022-09-08T00:04:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:36 smithi055 ceph-mon[97033]: pgmap v4675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:04:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:36 smithi059 ceph-mon[97450]: pgmap v4675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:04:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:36.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:36.910280+0000) 2022-09-08T00:04:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:36.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:36.821140+0000) 2022-09-08T00:04:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:37.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:37.910463+0000) 2022-09-08T00:04:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:37.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:37.821319+0000) 2022-09-08T00:04:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:38 smithi055 ceph-mon[97033]: pgmap v4676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:04:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:38 smithi059 ceph-mon[97450]: pgmap v4676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:04:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:38.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:38.821540+0000) 2022-09-08T00:04:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:38.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:38.910626+0000) 2022-09-08T00:04:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:39.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:39.821700+0000) 2022-09-08T00:04:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:39.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:39.910785+0000) 2022-09-08T00:04:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:40 smithi055 ceph-mon[97033]: pgmap v4677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:04:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:40 smithi059 ceph-mon[97450]: pgmap v4677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:04:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:40.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:40.910957+0000) 2022-09-08T00:04:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:40.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:40.821921+0000) 2022-09-08T00:04:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:41.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:41.911123+0000) 2022-09-08T00:04:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:41.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:41.822147+0000) 2022-09-08T00:04:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:42 smithi055 ceph-mon[97033]: pgmap v4678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:04:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:42 smithi059 ceph-mon[97450]: pgmap v4678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:04:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:42.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:42.911311+0000) 2022-09-08T00:04:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:42.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:42.822408+0000) 2022-09-08T00:04:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:43.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:43.911513+0000) 2022-09-08T00:04:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:43.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:43.822680+0000) 2022-09-08T00:04:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:44 smithi055 ceph-mon[97033]: pgmap v4679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:44 smithi059 ceph-mon[97450]: pgmap v4679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:44.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:44.911667+0000) 2022-09-08T00:04:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:44.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:44.822851+0000) 2022-09-08T00:04:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:45.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:45.911898+0000) 2022-09-08T00:04:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:45.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:45.823079+0000) 2022-09-08T00:04:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:46 smithi055 ceph-mon[97033]: pgmap v4680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:04:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:46 smithi059 ceph-mon[97450]: pgmap v4680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:04:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:46.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:46.912071+0000) 2022-09-08T00:04:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:46.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:46.823329+0000) 2022-09-08T00:04:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:47.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:47.912302+0000) 2022-09-08T00:04:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:47.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:47.823507+0000) 2022-09-08T00:04:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:48 smithi055 ceph-mon[97033]: pgmap v4681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:48 smithi059 ceph-mon[97450]: pgmap v4681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:48.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:48.912525+0000) 2022-09-08T00:04:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:48.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:48.823743+0000) 2022-09-08T00:04:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:49 smithi055 ceph-mon[97033]: pgmap v4682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:49 smithi059 ceph-mon[97450]: pgmap v4682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:49.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:49.912727+0000) 2022-09-08T00:04:50.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:04:49.814+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:04:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:49.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:49.823890+0000) 2022-09-08T00:04:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:50.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:50.912880+0000) 2022-09-08T00:04:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:50.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:50.824081+0000) 2022-09-08T00:04:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:51.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:51.913080+0000) 2022-09-08T00:04:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:51.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:51.824296+0000) 2022-09-08T00:04:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:52 smithi055 ceph-mon[97033]: pgmap v4683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:04:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:52 smithi059 ceph-mon[97450]: pgmap v4683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:04:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:52.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:52.913300+0000) 2022-09-08T00:04:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:52.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:52.824530+0000) 2022-09-08T00:04:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:53.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:53.913526+0000) 2022-09-08T00:04:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:53.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:53.824760+0000) 2022-09-08T00:04:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:54 smithi055 ceph-mon[97033]: pgmap v4684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:54 smithi059 ceph-mon[97450]: pgmap v4684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:54.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:54.913740+0000) 2022-09-08T00:04:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:54.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:54.824944+0000) 2022-09-08T00:04:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:55.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:55.914006+0000) 2022-09-08T00:04:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:55.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:55.825154+0000) 2022-09-08T00:04:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:56 smithi055 ceph-mon[97033]: pgmap v4685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:04:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:56 smithi059 ceph-mon[97450]: pgmap v4685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:04:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:56.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:56.914292+0000) 2022-09-08T00:04:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:56.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:56.825358+0000) 2022-09-08T00:04:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:57.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:57.914568+0000) 2022-09-08T00:04:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:57.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:57.825612+0000) 2022-09-08T00:04:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:04:58 smithi055 ceph-mon[97033]: pgmap v4686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:58.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:04:58 smithi059 ceph-mon[97450]: pgmap v4686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:04:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:58.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:58.914788+0000) 2022-09-08T00:04:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:58.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:58.825887+0000) 2022-09-08T00:05:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:04:59.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:59.914989+0000) 2022-09-08T00:05:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:04:59.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:04:59.826072+0000) 2022-09-08T00:05:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:00 smithi055 ceph-mon[97033]: pgmap v4687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:00 smithi059 ceph-mon[97450]: pgmap v4687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:00.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:00.826193+0000) 2022-09-08T00:05:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:00.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:00.915179+0000) 2022-09-08T00:05:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:01.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:01.915409+0000) 2022-09-08T00:05:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:01.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:01.826389+0000) 2022-09-08T00:05:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:02 smithi055 ceph-mon[97033]: pgmap v4688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:02 smithi059 ceph-mon[97450]: pgmap v4688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:02.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:02.915616+0000) 2022-09-08T00:05:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:02.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:02.826599+0000) 2022-09-08T00:05:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:03.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:03.915779+0000) 2022-09-08T00:05:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:03.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:03.826827+0000) 2022-09-08T00:05:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:04 smithi055 ceph-mon[97033]: pgmap v4689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:04 smithi059 ceph-mon[97450]: pgmap v4689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:05.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:05:04.829+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:05:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:04.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:04.915904+0000) 2022-09-08T00:05:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:04.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:04.826992+0000) 2022-09-08T00:05:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:05:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:05:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:05:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:05:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:05.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:05.916076+0000) 2022-09-08T00:05:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:05.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:05.827214+0000) 2022-09-08T00:05:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:06 smithi055 ceph-mon[97033]: pgmap v4690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-08T00:05:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:06 smithi059 ceph-mon[97450]: pgmap v4690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-08T00:05:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:06.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:06.916283+0000) 2022-09-08T00:05:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:06.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:06.827488+0000) 2022-09-08T00:05:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:07.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:07.827669+0000) 2022-09-08T00:05:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:07.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:07.916442+0000) 2022-09-08T00:05:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:08 smithi055 ceph-mon[97033]: pgmap v4691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-08T00:05:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:08 smithi059 ceph-mon[97450]: pgmap v4691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-08T00:05:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:08.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:08.916662+0000) 2022-09-08T00:05:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:08.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:08.827869+0000) 2022-09-08T00:05:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:09.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:09.916852+0000) 2022-09-08T00:05:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:09.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:09.828064+0000) 2022-09-08T00:05:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:10 smithi055 ceph-mon[97033]: pgmap v4692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-08T00:05:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:10 smithi059 ceph-mon[97450]: pgmap v4692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-08T00:05:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:10.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:10.917089+0000) 2022-09-08T00:05:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:10.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:10.828318+0000) 2022-09-08T00:05:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:11.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:11.917307+0000) 2022-09-08T00:05:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:11.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:11.828539+0000) 2022-09-08T00:05:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:12 smithi055 ceph-mon[97033]: pgmap v4693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-08T00:05:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:12 smithi059 ceph-mon[97450]: pgmap v4693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-08T00:05:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:12.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:12.917511+0000) 2022-09-08T00:05:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:12.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:12.828771+0000) 2022-09-08T00:05:13.663 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:05:13.664 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:05:13.665 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:05:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:05:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:05:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:05:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:13.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:13.917737+0000) 2022-09-08T00:05:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:13.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:13.828944+0000) 2022-09-08T00:05:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:14 smithi055 ceph-mon[97033]: pgmap v4694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-08T00:05:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:05:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:14 smithi059 ceph-mon[97450]: pgmap v4694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2022-09-08T00:05:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:05:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:14.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:14.917873+0000) 2022-09-08T00:05:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:14.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:14.829177+0000) 2022-09-08T00:05:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:15.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:15.918085+0000) 2022-09-08T00:05:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:15.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:15.829384+0000) 2022-09-08T00:05:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:16 smithi055 ceph-mon[97033]: pgmap v4695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-08T00:05:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:16 smithi059 ceph-mon[97450]: pgmap v4695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-08T00:05:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:16.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:16.918304+0000) 2022-09-08T00:05:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:16.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:16.829599+0000) 2022-09-08T00:05:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:17 smithi055 ceph-mon[97033]: pgmap v4696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:05:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:17 smithi059 ceph-mon[97450]: pgmap v4696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:05:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:17.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:17.918527+0000) 2022-09-08T00:05:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:17.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:17.829832+0000) 2022-09-08T00:05:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:18.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:18.918747+0000) 2022-09-08T00:05:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:18.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:18.830074+0000) 2022-09-08T00:05:20.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:05:19.846+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:05:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:19.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:19.918903+0000) 2022-09-08T00:05:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:19.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:19.830321+0000) 2022-09-08T00:05:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:20 smithi059 ceph-mon[97450]: pgmap v4697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:05:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:20 smithi055 ceph-mon[97033]: pgmap v4697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:05:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:20.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:20.830524+0000) 2022-09-08T00:05:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:20.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:20.919120+0000) 2022-09-08T00:05:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:21.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:21.830698+0000) 2022-09-08T00:05:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:21.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:21.919347+0000) 2022-09-08T00:05:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:22 smithi055 ceph-mon[97033]: pgmap v4698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:05:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:22 smithi059 ceph-mon[97450]: pgmap v4698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:05:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:22.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:22.919561+0000) 2022-09-08T00:05:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:22.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:22.830936+0000) 2022-09-08T00:05:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:23.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:23.831174+0000) 2022-09-08T00:05:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:23.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:23.919757+0000) 2022-09-08T00:05:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:24 smithi055 ceph-mon[97033]: pgmap v4699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:24 smithi059 ceph-mon[97450]: pgmap v4699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:24.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:24.919903+0000) 2022-09-08T00:05:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:24.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:24.831424+0000) 2022-09-08T00:05:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:25.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:25.920043+0000) 2022-09-08T00:05:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:25.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:25.831710+0000) 2022-09-08T00:05:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:26 smithi055 ceph-mon[97033]: pgmap v4700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:26 smithi059 ceph-mon[97450]: pgmap v4700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:26.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:26.920256+0000) 2022-09-08T00:05:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:26.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:26.831950+0000) 2022-09-08T00:05:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:27.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:27.920477+0000) 2022-09-08T00:05:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:27.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:27.832126+0000) 2022-09-08T00:05:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:28 smithi055 ceph-mon[97033]: pgmap v4701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:28 smithi059 ceph-mon[97450]: pgmap v4701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:28.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:28.920730+0000) 2022-09-08T00:05:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:28.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:28.832354+0000) 2022-09-08T00:05:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:29.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:29.920919+0000) 2022-09-08T00:05:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:29.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:29.832568+0000) 2022-09-08T00:05:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:30 smithi055 ceph-mon[97033]: pgmap v4702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:30 smithi059 ceph-mon[97450]: pgmap v4702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:30.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:30.921079+0000) 2022-09-08T00:05:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:30.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:30.832810+0000) 2022-09-08T00:05:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:31.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:31.921325+0000) 2022-09-08T00:05:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:31.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:31.832982+0000) 2022-09-08T00:05:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:32 smithi055 ceph-mon[97033]: pgmap v4703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:32 smithi059 ceph-mon[97450]: pgmap v4703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:32.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:32.921534+0000) 2022-09-08T00:05:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:32.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:32.833219+0000) 2022-09-08T00:05:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:33.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:33.921691+0000) 2022-09-08T00:05:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:33.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:33.833465+0000) 2022-09-08T00:05:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:34 smithi055 ceph-mon[97033]: pgmap v4704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:34 smithi059 ceph-mon[97450]: pgmap v4704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:35.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:05:34.858+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:05:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:34.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:34.921826+0000) 2022-09-08T00:05:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:34.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:34.833651+0000) 2022-09-08T00:05:35.922 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:35.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:35.833898+0000) 2022-09-08T00:05:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:35.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:35.922046+0000) 2022-09-08T00:05:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:36 smithi055 ceph-mon[97033]: pgmap v4705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:36 smithi059 ceph-mon[97450]: pgmap v4705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:36.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:36.834142+0000) 2022-09-08T00:05:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:36.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:36.922276+0000) 2022-09-08T00:05:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:37.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:37.834311+0000) 2022-09-08T00:05:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:37.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:37.922487+0000) 2022-09-08T00:05:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:38 smithi055 ceph-mon[97033]: pgmap v4706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:05:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:38 smithi059 ceph-mon[97450]: pgmap v4706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:05:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:38.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:38.922724+0000) 2022-09-08T00:05:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:38.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:38.834496+0000) 2022-09-08T00:05:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:39.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:39.922889+0000) 2022-09-08T00:05:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:39.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:39.834731+0000) 2022-09-08T00:05:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:40 smithi055 ceph-mon[97033]: pgmap v4707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:05:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:40 smithi059 ceph-mon[97450]: pgmap v4707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:05:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:40.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:40.923052+0000) 2022-09-08T00:05:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:40.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:40.834968+0000) 2022-09-08T00:05:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:41.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:41.923285+0000) 2022-09-08T00:05:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:41.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:41.835094+0000) 2022-09-08T00:05:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:42 smithi055 ceph-mon[97033]: pgmap v4708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:42 smithi059 ceph-mon[97450]: pgmap v4708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:42.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:42.923443+0000) 2022-09-08T00:05:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:42.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:42.835303+0000) 2022-09-08T00:05:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:43.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:43.923676+0000) 2022-09-08T00:05:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:43.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:43.835548+0000) 2022-09-08T00:05:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:44 smithi055 ceph-mon[97033]: pgmap v4709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:05:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:44 smithi059 ceph-mon[97450]: pgmap v4709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:05:45.161 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:44.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:44.923918+0000) 2022-09-08T00:05:45.162 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:44.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:44.835757+0000) 2022-09-08T00:05:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:45.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:45.924135+0000) 2022-09-08T00:05:46.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:45.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:45.835875+0000) 2022-09-08T00:05:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:46 smithi055 ceph-mon[97033]: pgmap v4710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:46 smithi059 ceph-mon[97450]: pgmap v4710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:46.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:46.836033+0000) 2022-09-08T00:05:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:46.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:46.924349+0000) 2022-09-08T00:05:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:47.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:47.924571+0000) 2022-09-08T00:05:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:47.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:47.836305+0000) 2022-09-08T00:05:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:48 smithi055 ceph-mon[97033]: pgmap v4711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:48 smithi059 ceph-mon[97450]: pgmap v4711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:48.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:48.924812+0000) 2022-09-08T00:05:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:48.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:48.836552+0000) 2022-09-08T00:05:50.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:05:49.872+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:05:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:49.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:49.925044+0000) 2022-09-08T00:05:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:49.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:49.836780+0000) 2022-09-08T00:05:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:50 smithi055 ceph-mon[97033]: pgmap v4712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:50 smithi059 ceph-mon[97450]: pgmap v4712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:50.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:50.837006+0000) 2022-09-08T00:05:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:50.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:50.925309+0000) 2022-09-08T00:05:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:51.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:51.925562+0000) 2022-09-08T00:05:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:51.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:51.837236+0000) 2022-09-08T00:05:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:52 smithi055 ceph-mon[97033]: pgmap v4713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:52 smithi059 ceph-mon[97450]: pgmap v4713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:52.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:52.925752+0000) 2022-09-08T00:05:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:52.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:52.837507+0000) 2022-09-08T00:05:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:53.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:53.837744+0000) 2022-09-08T00:05:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:53.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:53.925990+0000) 2022-09-08T00:05:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:54 smithi055 ceph-mon[97033]: pgmap v4714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:54 smithi059 ceph-mon[97450]: pgmap v4714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:54.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:54.926238+0000) 2022-09-08T00:05:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:54.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:54.837953+0000) 2022-09-08T00:05:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:55.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:55.926489+0000) 2022-09-08T00:05:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:55.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:55.838195+0000) 2022-09-08T00:05:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:56 smithi055 ceph-mon[97033]: pgmap v4715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:56 smithi059 ceph-mon[97450]: pgmap v4715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:05:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:56.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:56.926737+0000) 2022-09-08T00:05:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:56.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:56.838406+0000) 2022-09-08T00:05:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:57.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:57.926894+0000) 2022-09-08T00:05:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:57.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:57.838622+0000) 2022-09-08T00:05:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:05:58 smithi055 ceph-mon[97033]: pgmap v4716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:05:58 smithi059 ceph-mon[97450]: pgmap v4716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:05:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:58.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:58.927116+0000) 2022-09-08T00:05:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:58.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:58.838809+0000) 2022-09-08T00:06:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:05:59.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:59.927314+0000) 2022-09-08T00:06:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:05:59.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:05:59.838937+0000) 2022-09-08T00:06:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:00 smithi055 ceph-mon[97033]: pgmap v4717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:00 smithi059 ceph-mon[97450]: pgmap v4717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:01.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:00.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:00.927492+0000) 2022-09-08T00:06:01.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:00.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:00.839163+0000) 2022-09-08T00:06:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:01.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:01.927745+0000) 2022-09-08T00:06:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:01.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:01.839328+0000) 2022-09-08T00:06:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:02 smithi055 ceph-mon[97033]: pgmap v4718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:06:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:02 smithi059 ceph-mon[97450]: pgmap v4718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:06:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:02.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:02.927983+0000) 2022-09-08T00:06:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:02.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:02.839554+0000) 2022-09-08T00:06:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:03.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:03.928133+0000) 2022-09-08T00:06:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:03.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:03.839798+0000) 2022-09-08T00:06:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:04 smithi055 ceph-mon[97033]: pgmap v4719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:04 smithi059 ceph-mon[97450]: pgmap v4719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:05.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:06:04.888+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:06:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:04.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:04.928368+0000) 2022-09-08T00:06:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:04.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:04.840063+0000) 2022-09-08T00:06:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:06:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:06:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:06:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:06:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:05.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:05.840306+0000) 2022-09-08T00:06:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:05.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:05.928592+0000) 2022-09-08T00:06:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:06 smithi055 ceph-mon[97033]: pgmap v4720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:06:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:06 smithi059 ceph-mon[97450]: pgmap v4720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:06:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:06.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:06.928823+0000) 2022-09-08T00:06:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:06.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:06.840596+0000) 2022-09-08T00:06:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:07 smithi055 ceph-mon[97033]: pgmap v4721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:06:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:07 smithi059 ceph-mon[97450]: pgmap v4721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:06:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:07.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:07.929046+0000) 2022-09-08T00:06:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:07.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:07.840807+0000) 2022-09-08T00:06:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:08.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:08.929276+0000) 2022-09-08T00:06:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:08.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:08.841062+0000) 2022-09-08T00:06:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:09 smithi055 ceph-mon[97033]: osdmap e381: 12 total, 11 up, 11 in 2022-09-08T00:06:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:09 smithi055 ceph-mon[97033]: pgmap v4723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:06:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:09 smithi059 ceph-mon[97450]: osdmap e381: 12 total, 11 up, 11 in 2022-09-08T00:06:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:09 smithi059 ceph-mon[97450]: pgmap v4723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:06:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:09.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:09.929506+0000) 2022-09-08T00:06:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:09.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:09.841269+0000) 2022-09-08T00:06:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:10.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:10.929751+0000) 2022-09-08T00:06:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:10.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:10.841464+0000) 2022-09-08T00:06:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:11.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:11.929995+0000) 2022-09-08T00:06:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:11.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:11.841712+0000) 2022-09-08T00:06:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:12 smithi055 ceph-mon[97033]: pgmap v4724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:06:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:12 smithi059 ceph-mon[97450]: pgmap v4724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:06:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:12.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:12.930230+0000) 2022-09-08T00:06:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:12.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:12.841949+0000) 2022-09-08T00:06:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:13.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:13.930423+0000) 2022-09-08T00:06:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:13.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:13.842182+0000) 2022-09-08T00:06:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:14 smithi059 ceph-mon[97450]: pgmap v4725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:06:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:06:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:06:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:06:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:06:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:14 smithi055 ceph-mon[97033]: pgmap v4725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:06:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:06:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:06:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:06:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:06:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:14.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:14.930662+0000) 2022-09-08T00:06:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:14.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:14.842441+0000) 2022-09-08T00:06:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:15.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:15.930950+0000) 2022-09-08T00:06:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:15.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:15.842659+0000) 2022-09-08T00:06:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:16 smithi055 ceph-mon[97033]: pgmap v4726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:06:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:16 smithi055 ceph-mon[97033]: osdmap e382: 12 total, 11 up, 11 in 2022-09-08T00:06:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:16 smithi059 ceph-mon[97450]: pgmap v4726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:06:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:16 smithi059 ceph-mon[97450]: osdmap e382: 12 total, 11 up, 11 in 2022-09-08T00:06:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:16.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:16.931123+0000) 2022-09-08T00:06:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:16.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:16.842834+0000) 2022-09-08T00:06:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:17.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:17.931330+0000) 2022-09-08T00:06:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:17.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:17.842975+0000) 2022-09-08T00:06:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:18 smithi055 ceph-mon[97033]: pgmap v4728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 233 B/s wr, 7 op/s 2022-09-08T00:06:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:18 smithi059 ceph-mon[97450]: pgmap v4728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 233 B/s wr, 7 op/s 2022-09-08T00:06:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:18.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:18.931499+0000) 2022-09-08T00:06:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:18.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:18.843144+0000) 2022-09-08T00:06:20.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:06:19.900+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:06:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:19.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:19.843355+0000) 2022-09-08T00:06:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:19.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:19.931744+0000) 2022-09-08T00:06:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:20 smithi055 ceph-mon[97033]: pgmap v4729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:06:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:20 smithi059 ceph-mon[97450]: pgmap v4729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:06:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:20.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:20.931996+0000) 2022-09-08T00:06:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:20.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:20.843587+0000) 2022-09-08T00:06:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:21.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:21.932239+0000) 2022-09-08T00:06:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:21.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:21.843796+0000) 2022-09-08T00:06:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:22 smithi055 ceph-mon[97033]: pgmap v4730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:06:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:22 smithi059 ceph-mon[97450]: pgmap v4730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:06:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:22.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:22.844011+0000) 2022-09-08T00:06:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:22.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:22.932435+0000) 2022-09-08T00:06:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:23.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:23.932684+0000) 2022-09-08T00:06:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:23.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:23.844218+0000) 2022-09-08T00:06:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:24 smithi055 ceph-mon[97033]: pgmap v4731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:06:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:24 smithi059 ceph-mon[97450]: pgmap v4731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:06:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:24.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:24.932831+0000) 2022-09-08T00:06:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:24.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:24.844391+0000) 2022-09-08T00:06:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:25.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:25.933019+0000) 2022-09-08T00:06:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:25.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:25.844547+0000) 2022-09-08T00:06:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:26 smithi055 ceph-mon[97033]: pgmap v4732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:06:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:26 smithi059 ceph-mon[97450]: pgmap v4732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:06:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:26.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:26.933250+0000) 2022-09-08T00:06:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:26.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:26.844747+0000) 2022-09-08T00:06:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:27.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:27.933480+0000) 2022-09-08T00:06:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:27.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:27.844988+0000) 2022-09-08T00:06:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:28 smithi055 ceph-mon[97033]: pgmap v4733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:06:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:28 smithi059 ceph-mon[97450]: pgmap v4733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:06:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:28.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:28.933743+0000) 2022-09-08T00:06:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:28.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:28.845155+0000) 2022-09-08T00:06:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:29.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:29.845364+0000) 2022-09-08T00:06:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:29.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:29.933933+0000) 2022-09-08T00:06:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:30 smithi055 ceph-mon[97033]: pgmap v4734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:30 smithi059 ceph-mon[97450]: pgmap v4734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:30.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:30.934111+0000) 2022-09-08T00:06:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:30.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:30.845588+0000) 2022-09-08T00:06:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:31.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:31.934260+0000) 2022-09-08T00:06:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:31.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:31.845800+0000) 2022-09-08T00:06:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:32 smithi055 ceph-mon[97033]: pgmap v4735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:06:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:32 smithi059 ceph-mon[97450]: pgmap v4735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:06:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:32.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:32.934504+0000) 2022-09-08T00:06:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:32.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:32.846015+0000) 2022-09-08T00:06:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:33.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:33.846245+0000) 2022-09-08T00:06:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:33.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:33.934690+0000) 2022-09-08T00:06:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:34 smithi055 ceph-mon[97033]: pgmap v4736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:34 smithi059 ceph-mon[97450]: pgmap v4736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:34.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:34.934833+0000) 2022-09-08T00:06:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:34.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:34.846499+0000) 2022-09-08T00:06:35.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:06:34.903+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:06:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:35.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:35.935108+0000) 2022-09-08T00:06:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:35.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:35.846704+0000) 2022-09-08T00:06:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:36 smithi055 ceph-mon[97033]: pgmap v4737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:06:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:36 smithi059 ceph-mon[97450]: pgmap v4737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:06:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:36.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:36.935312+0000) 2022-09-08T00:06:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:36.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:36.846913+0000) 2022-09-08T00:06:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:37.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:37.935537+0000) 2022-09-08T00:06:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:37.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:37.847153+0000) 2022-09-08T00:06:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:38 smithi055 ceph-mon[97033]: pgmap v4738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:38 smithi059 ceph-mon[97450]: pgmap v4738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:38.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:38.935757+0000) 2022-09-08T00:06:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:38.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:38.847402+0000) 2022-09-08T00:06:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:39.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:39.935943+0000) 2022-09-08T00:06:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:39.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:39.847668+0000) 2022-09-08T00:06:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:40 smithi055 ceph-mon[97033]: pgmap v4739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:40 smithi059 ceph-mon[97450]: pgmap v4739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:40.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:40.936184+0000) 2022-09-08T00:06:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:40.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:40.847910+0000) 2022-09-08T00:06:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:41 smithi055 ceph-mon[97033]: pgmap v4740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:06:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:41 smithi059 ceph-mon[97450]: pgmap v4740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:06:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:41.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:41.936407+0000) 2022-09-08T00:06:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:41.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:41.848027+0000) 2022-09-08T00:06:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:42.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:42.936659+0000) 2022-09-08T00:06:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:42.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:42.848257+0000) 2022-09-08T00:06:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:43.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:43.936807+0000) 2022-09-08T00:06:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:43.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:43.848496+0000) 2022-09-08T00:06:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:44 smithi059 ceph-mon[97450]: pgmap v4741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:44 smithi055 ceph-mon[97033]: pgmap v4741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:44.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:44.936939+0000) 2022-09-08T00:06:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:44.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:44.848759+0000) 2022-09-08T00:06:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:45.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:45.937145+0000) 2022-09-08T00:06:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:45.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:45.849008+0000) 2022-09-08T00:06:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:46 smithi055 ceph-mon[97033]: pgmap v4742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:06:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:46 smithi059 ceph-mon[97450]: pgmap v4742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:06:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:46.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:46.937305+0000) 2022-09-08T00:06:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:46.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:46.849236+0000) 2022-09-08T00:06:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:47.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:47.937524+0000) 2022-09-08T00:06:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:47.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:47.849428+0000) 2022-09-08T00:06:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:48 smithi055 ceph-mon[97033]: pgmap v4743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:48 smithi059 ceph-mon[97450]: pgmap v4743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:48.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:48.937757+0000) 2022-09-08T00:06:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:48.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:48.849672+0000) 2022-09-08T00:06:50.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:06:49.916+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:06:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:49.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:49.937945+0000) 2022-09-08T00:06:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:49.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:49.849914+0000) 2022-09-08T00:06:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:50 smithi055 ceph-mon[97033]: pgmap v4744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:50 smithi059 ceph-mon[97450]: pgmap v4744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:50.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:50.850194+0000) 2022-09-08T00:06:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:50.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:50.938176+0000) 2022-09-08T00:06:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:51.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:51.850413+0000) 2022-09-08T00:06:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:51.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:51.938397+0000) 2022-09-08T00:06:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:52 smithi055 ceph-mon[97033]: pgmap v4745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:06:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:52 smithi059 ceph-mon[97450]: pgmap v4745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:06:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:52.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:52.938686+0000) 2022-09-08T00:06:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:52.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:52.850686+0000) 2022-09-08T00:06:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:53.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:53.938857+0000) 2022-09-08T00:06:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:53.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:53.850881+0000) 2022-09-08T00:06:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:54 smithi055 ceph-mon[97033]: pgmap v4746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:54 smithi059 ceph-mon[97450]: pgmap v4746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:54.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:54.939035+0000) 2022-09-08T00:06:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:54.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:54.851138+0000) 2022-09-08T00:06:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:55.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:55.851400+0000) 2022-09-08T00:06:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:55.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:55.939211+0000) 2022-09-08T00:06:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:56 smithi055 ceph-mon[97033]: pgmap v4747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:06:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:56 smithi059 ceph-mon[97450]: pgmap v4747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:06:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:56.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:56.851610+0000) 2022-09-08T00:06:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:56.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:56.939445+0000) 2022-09-08T00:06:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:57.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:57.939732+0000) 2022-09-08T00:06:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:57.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:57.851797+0000) 2022-09-08T00:06:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:06:58 smithi055 ceph-mon[97033]: pgmap v4748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:06:58 smithi059 ceph-mon[97450]: pgmap v4748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:06:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:58.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:58.939926+0000) 2022-09-08T00:06:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:58.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:58.851957+0000) 2022-09-08T00:07:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:06:59.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:59.852156+0000) 2022-09-08T00:07:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:06:59.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:06:59.940207+0000) 2022-09-08T00:07:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:00 smithi055 ceph-mon[97033]: pgmap v4749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:00 smithi059 ceph-mon[97450]: pgmap v4749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:00.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:00.852343+0000) 2022-09-08T00:07:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:00.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:00.940376+0000) 2022-09-08T00:07:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:01.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:01.940532+0000) 2022-09-08T00:07:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:01.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:01.852581+0000) 2022-09-08T00:07:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:02 smithi055 ceph-mon[97033]: pgmap v4750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:07:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:02 smithi059 ceph-mon[97450]: pgmap v4750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:07:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:02.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:02.940760+0000) 2022-09-08T00:07:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:02.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:02.852747+0000) 2022-09-08T00:07:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:03.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:03.852916+0000) 2022-09-08T00:07:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:03.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:03.941010+0000) 2022-09-08T00:07:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:04 smithi055 ceph-mon[97033]: pgmap v4751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:04 smithi059 ceph-mon[97450]: pgmap v4751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:04.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:04.941173+0000) 2022-09-08T00:07:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:04.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:04.853169+0000) 2022-09-08T00:07:05.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:07:04.932+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:07:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:07:05.770 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:07:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:07:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:07:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:05.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:05.853413+0000) 2022-09-08T00:07:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:05.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:05.941386+0000) 2022-09-08T00:07:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:06 smithi055 ceph-mon[97033]: pgmap v4752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:07:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:06 smithi059 ceph-mon[97450]: pgmap v4752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:07:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:06.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:06.941616+0000) 2022-09-08T00:07:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:06.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:06.853705+0000) 2022-09-08T00:07:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:07 smithi055 ceph-mon[97033]: pgmap v4753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:07:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:07 smithi059 ceph-mon[97450]: pgmap v4753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:07:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:07.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:07.853827+0000) 2022-09-08T00:07:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:07.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:07.941872+0000) 2022-09-08T00:07:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:08.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:08.942017+0000) 2022-09-08T00:07:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:08.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:08.854010+0000) 2022-09-08T00:07:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:09 smithi055 ceph-mon[97033]: osdmap e383: 12 total, 11 up, 11 in 2022-09-08T00:07:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:09 smithi055 ceph-mon[97033]: pgmap v4755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:07:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:09 smithi059 ceph-mon[97450]: osdmap e383: 12 total, 11 up, 11 in 2022-09-08T00:07:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:09 smithi059 ceph-mon[97450]: pgmap v4755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:07:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:09.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:09.942226+0000) 2022-09-08T00:07:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:09.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:09.854217+0000) 2022-09-08T00:07:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:10.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:10.942468+0000) 2022-09-08T00:07:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:10.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:10.854474+0000) 2022-09-08T00:07:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:11.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:11.942735+0000) 2022-09-08T00:07:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:11.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:11.854741+0000) 2022-09-08T00:07:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:12 smithi055 ceph-mon[97033]: pgmap v4756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:07:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:12 smithi059 ceph-mon[97450]: pgmap v4756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:07:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:12.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:12.942933+0000) 2022-09-08T00:07:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:12.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:12.854970+0000) 2022-09-08T00:07:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:13.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:13.943160+0000) 2022-09-08T00:07:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:13.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:13.855089+0000) 2022-09-08T00:07:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:14 smithi055 ceph-mon[97033]: pgmap v4757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:07:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:07:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:07:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:07:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:14 smithi059 ceph-mon[97450]: pgmap v4757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:07:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:07:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:07:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:07:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:14.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:14.943330+0000) 2022-09-08T00:07:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:14.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:14.855291+0000) 2022-09-08T00:07:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:15.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:15.943538+0000) 2022-09-08T00:07:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:15.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:15.855517+0000) 2022-09-08T00:07:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:07:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:15 smithi055 ceph-mon[97033]: pgmap v4758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:07:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:15 smithi055 ceph-mon[97033]: osdmap e384: 12 total, 11 up, 11 in 2022-09-08T00:07:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:07:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:15 smithi059 ceph-mon[97450]: pgmap v4758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:07:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:15 smithi059 ceph-mon[97450]: osdmap e384: 12 total, 11 up, 11 in 2022-09-08T00:07:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:16.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:16.943702+0000) 2022-09-08T00:07:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:16.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:16.855719+0000) 2022-09-08T00:07:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:17.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:17.943819+0000) 2022-09-08T00:07:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:17.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:17.855954+0000) 2022-09-08T00:07:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:18 smithi055 ceph-mon[97033]: pgmap v4760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-08T00:07:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:18 smithi059 ceph-mon[97450]: pgmap v4760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-08T00:07:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:18.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:18.943938+0000) 2022-09-08T00:07:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:18.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:18.856167+0000) 2022-09-08T00:07:20.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:07:19.947+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:07:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:19.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:19.944127+0000) 2022-09-08T00:07:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:19.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:19.856375+0000) 2022-09-08T00:07:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:20 smithi055 ceph-mon[97033]: pgmap v4761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:07:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:20 smithi059 ceph-mon[97450]: pgmap v4761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:07:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:20.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:20.944345+0000) 2022-09-08T00:07:21.185 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:20.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:20.856612+0000) 2022-09-08T00:07:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:21.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:21.944572+0000) 2022-09-08T00:07:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:21.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:21.856793+0000) 2022-09-08T00:07:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:22 smithi055 ceph-mon[97033]: pgmap v4762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:07:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:22 smithi059 ceph-mon[97450]: pgmap v4762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:07:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:22.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:22.944703+0000) 2022-09-08T00:07:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:22.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:22.857030+0000) 2022-09-08T00:07:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:23.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:23.944864+0000) 2022-09-08T00:07:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:23.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:23.857279+0000) 2022-09-08T00:07:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:24 smithi055 ceph-mon[97033]: pgmap v4763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:07:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:24 smithi059 ceph-mon[97450]: pgmap v4763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:07:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:24.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:24.945062+0000) 2022-09-08T00:07:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:24.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:24.857488+0000) 2022-09-08T00:07:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:25.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:25.945304+0000) 2022-09-08T00:07:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:25.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:25.857709+0000) 2022-09-08T00:07:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:26 smithi055 ceph-mon[97033]: pgmap v4764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:07:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:26 smithi059 ceph-mon[97450]: pgmap v4764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:07:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:26.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:26.945572+0000) 2022-09-08T00:07:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:26.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:26.857928+0000) 2022-09-08T00:07:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:27.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:27.945742+0000) 2022-09-08T00:07:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:27.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:27.858162+0000) 2022-09-08T00:07:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:28 smithi055 ceph-mon[97033]: pgmap v4765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:07:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:28 smithi059 ceph-mon[97450]: pgmap v4765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:07:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:28.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:28.945959+0000) 2022-09-08T00:07:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:28.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:28.858311+0000) 2022-09-08T00:07:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:29.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:29.946173+0000) 2022-09-08T00:07:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:29.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:29.858460+0000) 2022-09-08T00:07:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:30 smithi055 ceph-mon[97033]: pgmap v4766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:30 smithi059 ceph-mon[97450]: pgmap v4766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:30.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:30.858672+0000) 2022-09-08T00:07:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:30.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:30.946414+0000) 2022-09-08T00:07:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:31.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:31.946688+0000) 2022-09-08T00:07:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:31.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:31.858798+0000) 2022-09-08T00:07:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:32 smithi055 ceph-mon[97033]: pgmap v4767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:07:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:32 smithi059 ceph-mon[97450]: pgmap v4767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:07:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:32.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:32.946918+0000) 2022-09-08T00:07:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:32.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:32.858977+0000) 2022-09-08T00:07:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:33.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:33.859193+0000) 2022-09-08T00:07:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:33.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:33.947164+0000) 2022-09-08T00:07:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:34 smithi055 ceph-mon[97033]: pgmap v4768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:34 smithi059 ceph-mon[97450]: pgmap v4768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:35.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:07:34.950+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:07:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:34.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:34.859425+0000) 2022-09-08T00:07:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:34.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:34.947415+0000) 2022-09-08T00:07:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:35.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:35.947721+0000) 2022-09-08T00:07:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:35.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:35.859583+0000) 2022-09-08T00:07:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:36 smithi055 ceph-mon[97033]: pgmap v4769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:07:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:36 smithi059 ceph-mon[97450]: pgmap v4769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:07:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:36.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:36.947882+0000) 2022-09-08T00:07:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:36.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:36.859740+0000) 2022-09-08T00:07:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:37 smithi055 ceph-mon[97033]: pgmap v4770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:37 smithi059 ceph-mon[97450]: pgmap v4770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:37.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:37.948092+0000) 2022-09-08T00:07:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:37.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:37.859939+0000) 2022-09-08T00:07:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:38.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:38.948288+0000) 2022-09-08T00:07:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:38.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:38.860199+0000) 2022-09-08T00:07:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:39.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:39.948413+0000) 2022-09-08T00:07:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:39.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:39.860419+0000) 2022-09-08T00:07:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:40 smithi055 ceph-mon[97033]: pgmap v4771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:40 smithi059 ceph-mon[97450]: pgmap v4771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:40.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:40.948632+0000) 2022-09-08T00:07:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:40.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:40.860711+0000) 2022-09-08T00:07:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:41.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:41.948804+0000) 2022-09-08T00:07:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:41.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:41.860958+0000) 2022-09-08T00:07:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:42 smithi055 ceph-mon[97033]: pgmap v4772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:07:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:42 smithi059 ceph-mon[97450]: pgmap v4772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:07:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:42.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:42.861174+0000) 2022-09-08T00:07:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:42.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:42.948965+0000) 2022-09-08T00:07:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:43.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:43.949148+0000) 2022-09-08T00:07:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:43.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:43.861340+0000) 2022-09-08T00:07:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:44 smithi055 ceph-mon[97033]: pgmap v4773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:44 smithi059 ceph-mon[97450]: pgmap v4773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:44.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:44.949334+0000) 2022-09-08T00:07:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:44.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:44.861573+0000) 2022-09-08T00:07:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:45.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:45.949544+0000) 2022-09-08T00:07:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:45.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:45.861760+0000) 2022-09-08T00:07:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:46 smithi055 ceph-mon[97033]: pgmap v4774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:07:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:46 smithi059 ceph-mon[97450]: pgmap v4774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:07:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:46.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:46.949783+0000) 2022-09-08T00:07:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:46.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:46.861974+0000) 2022-09-08T00:07:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:47.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:47.949979+0000) 2022-09-08T00:07:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:47.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:47.862112+0000) 2022-09-08T00:07:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:48 smithi055 ceph-mon[97033]: pgmap v4775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:48 smithi059 ceph-mon[97450]: pgmap v4775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:48.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:48.950205+0000) 2022-09-08T00:07:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:48.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:48.862333+0000) 2022-09-08T00:07:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:49.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:49.950451+0000) 2022-09-08T00:07:50.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:07:49.958+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:07:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:49.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:49.862508+0000) 2022-09-08T00:07:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:50 smithi055 ceph-mon[97033]: pgmap v4776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:50 smithi059 ceph-mon[97450]: pgmap v4776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:50.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:50.950669+0000) 2022-09-08T00:07:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:50.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:50.862729+0000) 2022-09-08T00:07:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:51.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:51.950797+0000) 2022-09-08T00:07:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:51.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:51.862932+0000) 2022-09-08T00:07:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:52 smithi055 ceph-mon[97033]: pgmap v4777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:07:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:52 smithi059 ceph-mon[97450]: pgmap v4777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:07:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:52.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:52.951035+0000) 2022-09-08T00:07:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:52.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:52.863165+0000) 2022-09-08T00:07:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:53.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:53.951199+0000) 2022-09-08T00:07:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:53.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:53.863301+0000) 2022-09-08T00:07:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:54 smithi055 ceph-mon[97033]: pgmap v4778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:54 smithi059 ceph-mon[97450]: pgmap v4778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:54.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:54.951409+0000) 2022-09-08T00:07:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:54.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:54.863527+0000) 2022-09-08T00:07:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:55.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:55.951733+0000) 2022-09-08T00:07:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:55.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:55.863774+0000) 2022-09-08T00:07:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:56 smithi055 ceph-mon[97033]: pgmap v4779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:07:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:56 smithi059 ceph-mon[97450]: pgmap v4779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:07:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:56.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:56.951996+0000) 2022-09-08T00:07:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:56.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:56.863975+0000) 2022-09-08T00:07:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:57.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:57.952237+0000) 2022-09-08T00:07:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:57.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:57.864153+0000) 2022-09-08T00:07:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:58 smithi055 ceph-mon[97033]: pgmap v4780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:58 smithi059 ceph-mon[97450]: pgmap v4780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:58.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:58.952479+0000) 2022-09-08T00:07:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:58.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:58.864397+0000) 2022-09-08T00:07:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:07:59 smithi055 ceph-mon[97033]: pgmap v4781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:07:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:07:59 smithi059 ceph-mon[97450]: pgmap v4781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:07:59.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:59.952717+0000) 2022-09-08T00:08:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:07:59.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:07:59.864666+0000) 2022-09-08T00:08:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:00.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:00.952939+0000) 2022-09-08T00:08:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:00.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:00.864879+0000) 2022-09-08T00:08:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:01.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:01.953186+0000) 2022-09-08T00:08:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:01.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:01.865074+0000) 2022-09-08T00:08:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:02 smithi055 ceph-mon[97033]: pgmap v4782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:08:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:02 smithi059 ceph-mon[97450]: pgmap v4782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:08:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:02.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:02.953357+0000) 2022-09-08T00:08:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:02.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:02.865293+0000) 2022-09-08T00:08:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:03.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:03.953583+0000) 2022-09-08T00:08:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:03.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:03.865546+0000) 2022-09-08T00:08:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:04 smithi055 ceph-mon[97033]: pgmap v4783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:04 smithi059 ceph-mon[97450]: pgmap v4783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:05.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:08:04.966+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:08:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:04.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:04.953752+0000) 2022-09-08T00:08:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:04.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:04.865757+0000) 2022-09-08T00:08:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:08:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:08:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:08:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:08:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:05.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:05.953985+0000) 2022-09-08T00:08:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:05.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:05.865916+0000) 2022-09-08T00:08:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:06 smithi055 ceph-mon[97033]: pgmap v4784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:08:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:06 smithi059 ceph-mon[97450]: pgmap v4784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:08:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:06.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:06.954234+0000) 2022-09-08T00:08:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:06.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:06.866172+0000) 2022-09-08T00:08:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:07.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:07.954453+0000) 2022-09-08T00:08:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:07.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:07.866432+0000) 2022-09-08T00:08:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:08 smithi055 ceph-mon[97033]: pgmap v4785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:08:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:08 smithi059 ceph-mon[97450]: pgmap v4785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:08:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:08.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:08.954658+0000) 2022-09-08T00:08:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:08.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:08.866615+0000) 2022-09-08T00:08:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:09 smithi055 ceph-mon[97033]: osdmap e385: 12 total, 11 up, 11 in 2022-09-08T00:08:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:09 smithi059 ceph-mon[97450]: osdmap e385: 12 total, 11 up, 11 in 2022-09-08T00:08:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:09.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:09.954904+0000) 2022-09-08T00:08:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:09.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:09.866867+0000) 2022-09-08T00:08:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:10 smithi055 ceph-mon[97033]: pgmap v4787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:08:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:10 smithi059 ceph-mon[97450]: pgmap v4787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:08:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:10.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:10.955143+0000) 2022-09-08T00:08:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:10.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:10.867130+0000) 2022-09-08T00:08:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:11 smithi055 ceph-mon[97033]: pgmap v4788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:08:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:11 smithi059 ceph-mon[97450]: pgmap v4788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:08:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:11.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:11.955381+0000) 2022-09-08T00:08:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:11.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:11.867334+0000) 2022-09-08T00:08:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:12.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:12.955603+0000) 2022-09-08T00:08:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:12.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:12.867529+0000) 2022-09-08T00:08:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:13.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:13.955875+0000) 2022-09-08T00:08:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:13.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:13.867760+0000) 2022-09-08T00:08:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:14 smithi055 ceph-mon[97033]: pgmap v4789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:08:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:14 smithi059 ceph-mon[97450]: pgmap v4789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:08:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:14.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:14.956018+0000) 2022-09-08T00:08:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:14.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:14.867941+0000) 2022-09-08T00:08:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:08:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:08:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:08:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:08:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:08:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:08:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:15.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:15.956197+0000) 2022-09-08T00:08:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:15.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:15.868190+0000) 2022-09-08T00:08:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:16 smithi055 ceph-mon[97033]: pgmap v4790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:08:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:16 smithi055 ceph-mon[97033]: osdmap e386: 12 total, 11 up, 11 in 2022-09-08T00:08:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:08:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:16 smithi059 ceph-mon[97450]: pgmap v4790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:08:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:16 smithi059 ceph-mon[97450]: osdmap e386: 12 total, 11 up, 11 in 2022-09-08T00:08:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:08:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:16.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:16.868410+0000) 2022-09-08T00:08:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:16.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:16.956369+0000) 2022-09-08T00:08:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:17.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:17.868686+0000) 2022-09-08T00:08:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:17.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:17.956575+0000) 2022-09-08T00:08:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:18 smithi055 ceph-mon[97033]: pgmap v4792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-08T00:08:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:18 smithi059 ceph-mon[97450]: pgmap v4792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-08T00:08:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:18.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:18.956764+0000) 2022-09-08T00:08:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:18.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:18.868901+0000) 2022-09-08T00:08:20.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:08:19.978+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:08:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:19.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:19.956988+0000) 2022-09-08T00:08:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:19.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:19.869087+0000) 2022-09-08T00:08:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:20 smithi055 ceph-mon[97033]: pgmap v4793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:08:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:20 smithi059 ceph-mon[97450]: pgmap v4793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:08:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:20.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:20.957228+0000) 2022-09-08T00:08:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:20.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:20.869309+0000) 2022-09-08T00:08:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:21.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:21.957397+0000) 2022-09-08T00:08:22.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:21.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:21.869442+0000) 2022-09-08T00:08:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:22 smithi055 ceph-mon[97033]: pgmap v4794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:08:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:22 smithi059 ceph-mon[97450]: pgmap v4794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:08:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:22.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:22.957662+0000) 2022-09-08T00:08:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:22.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:22.869607+0000) 2022-09-08T00:08:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:23.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:23.957801+0000) 2022-09-08T00:08:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:23.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:23.869742+0000) 2022-09-08T00:08:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:24 smithi055 ceph-mon[97033]: pgmap v4795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:08:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:24 smithi059 ceph-mon[97450]: pgmap v4795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:08:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:24.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:24.957990+0000) 2022-09-08T00:08:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:24.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:24.869960+0000) 2022-09-08T00:08:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:25.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:25.958202+0000) 2022-09-08T00:08:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:25.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:25.870231+0000) 2022-09-08T00:08:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:26 smithi055 ceph-mon[97033]: pgmap v4796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:08:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:26 smithi059 ceph-mon[97450]: pgmap v4796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:08:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:26.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:26.958446+0000) 2022-09-08T00:08:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:26.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:26.870499+0000) 2022-09-08T00:08:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:27.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:27.958703+0000) 2022-09-08T00:08:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:27.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:27.870705+0000) 2022-09-08T00:08:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:28 smithi055 ceph-mon[97033]: pgmap v4797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:08:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:28 smithi059 ceph-mon[97450]: pgmap v4797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:08:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:28.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:28.958883+0000) 2022-09-08T00:08:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:28.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:28.870930+0000) 2022-09-08T00:08:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:29.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:29.959173+0000) 2022-09-08T00:08:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:29.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:29.871049+0000) 2022-09-08T00:08:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:30 smithi055 ceph-mon[97033]: pgmap v4798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:30 smithi059 ceph-mon[97450]: pgmap v4798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:30.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:30.959452+0000) 2022-09-08T00:08:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:30.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:30.871254+0000) 2022-09-08T00:08:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:31.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:31.959717+0000) 2022-09-08T00:08:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:31.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:31.871470+0000) 2022-09-08T00:08:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:32 smithi055 ceph-mon[97033]: pgmap v4799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:08:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:32 smithi059 ceph-mon[97450]: pgmap v4799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:08:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:32.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:32.959841+0000) 2022-09-08T00:08:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:32.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:32.871674+0000) 2022-09-08T00:08:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:33.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:33.960060+0000) 2022-09-08T00:08:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:33.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:33.871841+0000) 2022-09-08T00:08:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:34 smithi055 ceph-mon[97033]: pgmap v4800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:34 smithi059 ceph-mon[97450]: pgmap v4800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:35.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:08:34.993+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:08:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:34.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:34.872050+0000) 2022-09-08T00:08:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:34.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:34.960262+0000) 2022-09-08T00:08:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:35.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:35.960462+0000) 2022-09-08T00:08:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:35.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:35.872273+0000) 2022-09-08T00:08:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:36 smithi055 ceph-mon[97033]: pgmap v4801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:08:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:36 smithi059 ceph-mon[97450]: pgmap v4801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:08:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:36.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:36.960712+0000) 2022-09-08T00:08:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:36.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:36.872511+0000) 2022-09-08T00:08:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:37.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:37.872751+0000) 2022-09-08T00:08:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:37.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:37.960945+0000) 2022-09-08T00:08:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:38 smithi055 ceph-mon[97033]: pgmap v4802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:38 smithi059 ceph-mon[97450]: pgmap v4802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:38.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:38.961182+0000) 2022-09-08T00:08:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:38.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:38.872978+0000) 2022-09-08T00:08:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:39.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:39.961425+0000) 2022-09-08T00:08:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:39.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:39.873189+0000) 2022-09-08T00:08:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:40 smithi055 ceph-mon[97033]: pgmap v4803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:40 smithi059 ceph-mon[97450]: pgmap v4803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:40.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:40.873385+0000) 2022-09-08T00:08:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:40.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:40.961683+0000) 2022-09-08T00:08:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:41.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:41.961971+0000) 2022-09-08T00:08:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:41.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:41.873602+0000) 2022-09-08T00:08:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:42 smithi055 ceph-mon[97033]: pgmap v4804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:08:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:42 smithi059 ceph-mon[97450]: pgmap v4804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:08:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:42.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:42.962253+0000) 2022-09-08T00:08:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:42.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:42.873813+0000) 2022-09-08T00:08:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:43.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:43.962531+0000) 2022-09-08T00:08:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:43.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:43.874040+0000) 2022-09-08T00:08:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:44 smithi055 ceph-mon[97033]: pgmap v4805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:44 smithi059 ceph-mon[97450]: pgmap v4805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:44.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:44.962746+0000) 2022-09-08T00:08:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:44.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:44.874291+0000) 2022-09-08T00:08:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:45.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:45.962886+0000) 2022-09-08T00:08:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:45.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:45.874504+0000) 2022-09-08T00:08:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:46 smithi055 ceph-mon[97033]: pgmap v4806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:08:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:46 smithi059 ceph-mon[97450]: pgmap v4806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:08:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:46.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:46.963048+0000) 2022-09-08T00:08:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:46.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:46.874774+0000) 2022-09-08T00:08:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:47.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:47.963185+0000) 2022-09-08T00:08:48.180 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:47.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:47.874976+0000) 2022-09-08T00:08:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:48 smithi055 ceph-mon[97033]: pgmap v4807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:48 smithi059 ceph-mon[97450]: pgmap v4807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:48.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:48.963420+0000) 2022-09-08T00:08:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:48.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:48.875214+0000) 2022-09-08T00:08:50.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:08:50.002+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:08:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:49.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:49.963595+0000) 2022-09-08T00:08:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:49.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:49.875450+0000) 2022-09-08T00:08:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:50 smithi055 ceph-mon[97033]: pgmap v4808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:50 smithi059 ceph-mon[97450]: pgmap v4808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:50.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:50.963804+0000) 2022-09-08T00:08:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:50.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:50.875726+0000) 2022-09-08T00:08:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:51 smithi055 ceph-mon[97033]: pgmap v4809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:08:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:51 smithi059 ceph-mon[97450]: pgmap v4809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:08:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:51.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:51.964063+0000) 2022-09-08T00:08:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:51.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:51.875871+0000) 2022-09-08T00:08:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:52.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:52.964253+0000) 2022-09-08T00:08:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:52.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:52.876057+0000) 2022-09-08T00:08:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:53.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:53.964397+0000) 2022-09-08T00:08:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:53.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:53.876283+0000) 2022-09-08T00:08:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:54 smithi055 ceph-mon[97033]: pgmap v4810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:54 smithi059 ceph-mon[97450]: pgmap v4810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:54.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:54.964568+0000) 2022-09-08T00:08:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:54.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:54.876519+0000) 2022-09-08T00:08:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:55.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:55.964766+0000) 2022-09-08T00:08:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:55.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:55.876705+0000) 2022-09-08T00:08:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:56 smithi055 ceph-mon[97033]: pgmap v4811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:08:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:56 smithi059 ceph-mon[97450]: pgmap v4811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:08:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:56.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:56.876912+0000) 2022-09-08T00:08:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:56.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:56.964979+0000) 2022-09-08T00:08:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:57.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:57.965154+0000) 2022-09-08T00:08:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:57.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:57.877135+0000) 2022-09-08T00:08:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:08:58 smithi055 ceph-mon[97033]: pgmap v4812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:08:58 smithi059 ceph-mon[97450]: pgmap v4812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:08:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:58.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:58.965402+0000) 2022-09-08T00:08:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:58.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:58.877275+0000) 2022-09-08T00:09:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:08:59.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:59.965687+0000) 2022-09-08T00:09:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:08:59.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:08:59.877462+0000) 2022-09-08T00:09:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:00 smithi055 ceph-mon[97033]: pgmap v4813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:00.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:00 smithi059 ceph-mon[97450]: pgmap v4813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:00.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:00.965838+0000) 2022-09-08T00:09:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:00.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:00.877614+0000) 2022-09-08T00:09:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:01.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:01.965970+0000) 2022-09-08T00:09:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:01.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:01.877785+0000) 2022-09-08T00:09:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:02 smithi055 ceph-mon[97033]: pgmap v4814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:09:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:02 smithi059 ceph-mon[97450]: pgmap v4814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:09:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:02.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:02.878050+0000) 2022-09-08T00:09:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:02.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:02.966171+0000) 2022-09-08T00:09:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:03.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:03.966365+0000) 2022-09-08T00:09:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:03.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:03.878279+0000) 2022-09-08T00:09:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:04 smithi055 ceph-mon[97033]: pgmap v4815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:04 smithi059 ceph-mon[97450]: pgmap v4815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:05.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:09:05.013+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:09:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:04.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:04.966592+0000) 2022-09-08T00:09:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:04.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:04.878472+0000) 2022-09-08T00:09:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:09:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:09:05.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:09:05.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:09:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:05.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:05.966818+0000) 2022-09-08T00:09:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:05.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:05.878699+0000) 2022-09-08T00:09:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:06 smithi055 ceph-mon[97033]: pgmap v4816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:09:06.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:06 smithi059 ceph-mon[97450]: pgmap v4816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:09:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:06.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:06.967040+0000) 2022-09-08T00:09:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:06.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:06.878932+0000) 2022-09-08T00:09:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:07.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:07.967235+0000) 2022-09-08T00:09:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:07.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:07.879205+0000) 2022-09-08T00:09:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:08 smithi055 ceph-mon[97033]: pgmap v4817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:09:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:08 smithi059 ceph-mon[97450]: pgmap v4817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:09:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:08.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:08.967469+0000) 2022-09-08T00:09:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:08.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:08.879424+0000) 2022-09-08T00:09:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:09 smithi055 ceph-mon[97033]: osdmap e387: 12 total, 11 up, 11 in 2022-09-08T00:09:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:09 smithi055 ceph-mon[97033]: pgmap v4819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:09:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:09 smithi059 ceph-mon[97450]: osdmap e387: 12 total, 11 up, 11 in 2022-09-08T00:09:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:09 smithi059 ceph-mon[97450]: pgmap v4819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:09:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:09.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:09.967683+0000) 2022-09-08T00:09:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:09.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:09.879672+0000) 2022-09-08T00:09:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:10.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:10.967915+0000) 2022-09-08T00:09:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:10.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:10.879946+0000) 2022-09-08T00:09:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:11.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:11.968111+0000) 2022-09-08T00:09:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:11.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:11.880155+0000) 2022-09-08T00:09:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:12 smithi055 ceph-mon[97033]: pgmap v4820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:09:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:12 smithi059 ceph-mon[97450]: pgmap v4820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:09:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:12.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:12.968317+0000) 2022-09-08T00:09:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:12.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:12.880397+0000) 2022-09-08T00:09:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:13.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:13.968444+0000) 2022-09-08T00:09:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:13.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:13.880602+0000) 2022-09-08T00:09:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:14 smithi055 ceph-mon[97033]: pgmap v4821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:09:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:14 smithi059 ceph-mon[97450]: pgmap v4821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:09:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:14.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:14.968696+0000) 2022-09-08T00:09:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:14.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:14.880869+0000) 2022-09-08T00:09:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:15.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:15.881203+0000) 2022-09-08T00:09:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:15.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:15.968855+0000) 2022-09-08T00:09:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:16 smithi055 ceph-mon[97033]: pgmap v4822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:09:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:16 smithi055 ceph-mon[97033]: osdmap e388: 12 total, 11 up, 11 in 2022-09-08T00:09:16.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:09:16.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:09:16.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:09:16.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:09:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:16 smithi059 ceph-mon[97450]: pgmap v4822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:09:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:16 smithi059 ceph-mon[97450]: osdmap e388: 12 total, 11 up, 11 in 2022-09-08T00:09:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:09:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:09:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:09:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:09:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:16.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:16.969116+0000) 2022-09-08T00:09:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:16.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:16.881447+0000) 2022-09-08T00:09:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:17.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:17.969369+0000) 2022-09-08T00:09:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:17.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:17.881626+0000) 2022-09-08T00:09:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:18 smithi055 ceph-mon[97033]: pgmap v4824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-08T00:09:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:18 smithi059 ceph-mon[97450]: pgmap v4824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-08T00:09:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:18.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:18.881830+0000) 2022-09-08T00:09:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:18.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:18.969656+0000) 2022-09-08T00:09:20.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:09:20.017+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:09:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:19.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:19.882091+0000) 2022-09-08T00:09:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:19.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:19.969885+0000) 2022-09-08T00:09:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:20 smithi055 ceph-mon[97033]: pgmap v4825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:09:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:20 smithi059 ceph-mon[97450]: pgmap v4825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:09:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:20.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:20.882298+0000) 2022-09-08T00:09:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:20.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:20.970136+0000) 2022-09-08T00:09:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:21.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:21.970297+0000) 2022-09-08T00:09:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:21.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:21.882530+0000) 2022-09-08T00:09:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:22 smithi055 ceph-mon[97033]: pgmap v4826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:09:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:22 smithi059 ceph-mon[97450]: pgmap v4826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:09:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:22.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:22.970422+0000) 2022-09-08T00:09:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:22.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:22.882790+0000) 2022-09-08T00:09:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:23.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:23.882980+0000) 2022-09-08T00:09:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:23.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:23.970662+0000) 2022-09-08T00:09:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:24 smithi055 ceph-mon[97033]: pgmap v4827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:09:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:24 smithi059 ceph-mon[97450]: pgmap v4827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:09:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:24.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:24.970805+0000) 2022-09-08T00:09:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:24.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:24.883166+0000) 2022-09-08T00:09:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:25.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:25.971024+0000) 2022-09-08T00:09:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:25.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:25.883367+0000) 2022-09-08T00:09:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:26 smithi055 ceph-mon[97033]: pgmap v4828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:09:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:26 smithi059 ceph-mon[97450]: pgmap v4828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:09:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:26.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:26.971277+0000) 2022-09-08T00:09:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:26.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:26.883616+0000) 2022-09-08T00:09:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:27.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:27.971504+0000) 2022-09-08T00:09:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:27.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:27.883883+0000) 2022-09-08T00:09:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:28 smithi055 ceph-mon[97033]: pgmap v4829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:09:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:28 smithi059 ceph-mon[97450]: pgmap v4829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:09:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:28.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:28.971688+0000) 2022-09-08T00:09:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:28.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:28.884159+0000) 2022-09-08T00:09:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:29.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:29.884424+0000) 2022-09-08T00:09:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:29.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:29.971842+0000) 2022-09-08T00:09:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:30 smithi055 ceph-mon[97033]: pgmap v4830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:30 smithi059 ceph-mon[97450]: pgmap v4830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:30.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:30.884672+0000) 2022-09-08T00:09:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:30.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:30.971993+0000) 2022-09-08T00:09:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:31.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:31.972224+0000) 2022-09-08T00:09:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:31.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:31.884898+0000) 2022-09-08T00:09:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:32 smithi055 ceph-mon[97033]: pgmap v4831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:09:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:32 smithi059 ceph-mon[97450]: pgmap v4831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:09:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:32.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:32.885127+0000) 2022-09-08T00:09:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:32.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:32.972447+0000) 2022-09-08T00:09:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:33.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:33.972724+0000) 2022-09-08T00:09:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:33.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:33.885346+0000) 2022-09-08T00:09:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:34 smithi055 ceph-mon[97033]: pgmap v4832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:34 smithi059 ceph-mon[97450]: pgmap v4832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:34.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:34.972871+0000) 2022-09-08T00:09:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:34.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:34.885593+0000) 2022-09-08T00:09:35.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:09:35.018+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:09:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:35.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:35.973085+0000) 2022-09-08T00:09:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:35.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:35.885876+0000) 2022-09-08T00:09:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:36 smithi055 ceph-mon[97033]: pgmap v4833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:09:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:36 smithi059 ceph-mon[97450]: pgmap v4833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:09:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:36.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:36.973314+0000) 2022-09-08T00:09:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:36.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:36.886139+0000) 2022-09-08T00:09:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:37 smithi059 ceph-mon[97450]: pgmap v4834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:37 smithi055 ceph-mon[97033]: pgmap v4834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:37.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:37.973477+0000) 2022-09-08T00:09:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:37.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:37.886376+0000) 2022-09-08T00:09:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:38.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:38.973720+0000) 2022-09-08T00:09:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:38.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:38.886616+0000) 2022-09-08T00:09:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:39.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:39.886852+0000) 2022-09-08T00:09:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:39.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:39.973908+0000) 2022-09-08T00:09:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:40 smithi055 ceph-mon[97033]: pgmap v4835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:40 smithi059 ceph-mon[97450]: pgmap v4835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:40.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:40.974133+0000) 2022-09-08T00:09:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:40.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:40.887057+0000) 2022-09-08T00:09:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:41.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:41.974390+0000) 2022-09-08T00:09:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:41.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:41.887200+0000) 2022-09-08T00:09:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:42 smithi055 ceph-mon[97033]: pgmap v4836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:09:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:42 smithi059 ceph-mon[97450]: pgmap v4836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:09:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:42.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:42.974618+0000) 2022-09-08T00:09:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:42.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:42.887433+0000) 2022-09-08T00:09:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:43.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:43.887719+0000) 2022-09-08T00:09:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:43.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:43.974919+0000) 2022-09-08T00:09:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:44 smithi055 ceph-mon[97033]: pgmap v4837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:44 smithi059 ceph-mon[97450]: pgmap v4837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:44.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:44.887954+0000) 2022-09-08T00:09:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:44.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:44.975064+0000) 2022-09-08T00:09:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:45.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:45.975218+0000) 2022-09-08T00:09:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:45.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:45.888137+0000) 2022-09-08T00:09:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:46 smithi055 ceph-mon[97033]: pgmap v4838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:09:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:46 smithi059 ceph-mon[97450]: pgmap v4838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:09:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:46.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:46.975427+0000) 2022-09-08T00:09:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:46.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:46.888372+0000) 2022-09-08T00:09:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:47.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:47.888516+0000) 2022-09-08T00:09:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:47.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:47.975672+0000) 2022-09-08T00:09:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:48 smithi055 ceph-mon[97033]: pgmap v4839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:48 smithi059 ceph-mon[97450]: pgmap v4839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:48.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:48.888693+0000) 2022-09-08T00:09:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:48.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:48.975882+0000) 2022-09-08T00:09:50.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:09:50.033+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:09:50.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:49.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:49.976098+0000) 2022-09-08T00:09:50.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:49.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:49.888850+0000) 2022-09-08T00:09:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:50 smithi055 ceph-mon[97033]: pgmap v4840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:50 smithi059 ceph-mon[97450]: pgmap v4840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:50.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:50.976296+0000) 2022-09-08T00:09:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:50.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:50.889100+0000) 2022-09-08T00:09:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:51 smithi059 ceph-mon[97450]: pgmap v4841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:09:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:51 smithi055 ceph-mon[97033]: pgmap v4841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:09:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:51.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:51.889326+0000) 2022-09-08T00:09:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:51.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:51.976510+0000) 2022-09-08T00:09:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:52.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:52.889545+0000) 2022-09-08T00:09:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:52.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:52.976776+0000) 2022-09-08T00:09:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:53.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:53.889759+0000) 2022-09-08T00:09:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:53.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:53.976936+0000) 2022-09-08T00:09:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:54 smithi055 ceph-mon[97033]: pgmap v4842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:54 smithi059 ceph-mon[97450]: pgmap v4842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:54.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:54.890001+0000) 2022-09-08T00:09:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:54.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:54.977161+0000) 2022-09-08T00:09:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:55.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:55.977436+0000) 2022-09-08T00:09:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:55.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:55.890256+0000) 2022-09-08T00:09:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:56 smithi055 ceph-mon[97033]: pgmap v4843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:09:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:56 smithi059 ceph-mon[97450]: pgmap v4843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:09:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:56.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:56.890504+0000) 2022-09-08T00:09:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:56.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:56.977707+0000) 2022-09-08T00:09:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:57.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:57.890700+0000) 2022-09-08T00:09:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:57.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:57.977969+0000) 2022-09-08T00:09:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:09:58 smithi055 ceph-mon[97033]: pgmap v4844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:09:58 smithi059 ceph-mon[97450]: pgmap v4844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:09:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:58.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:58.978200+0000) 2022-09-08T00:09:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:58.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:58.890895+0000) 2022-09-08T00:10:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:09:59.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:59.891142+0000) 2022-09-08T00:10:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:09:59.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:09:59.978380+0000) 2022-09-08T00:10:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:00 smithi055 ceph-mon[97033]: pgmap v4845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T00:10:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T00:10:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T00:10:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:00 smithi059 ceph-mon[97450]: pgmap v4845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T00:10:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T00:10:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T00:10:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:00.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:00.891408+0000) 2022-09-08T00:10:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:00.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:00.978569+0000) 2022-09-08T00:10:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:01 smithi055 ceph-mon[97033]: pgmap v4846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:10:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:01 smithi059 ceph-mon[97450]: pgmap v4846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:10:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:01.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:01.891665+0000) 2022-09-08T00:10:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:01.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:01.978790+0000) 2022-09-08T00:10:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:02.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:02.891925+0000) 2022-09-08T00:10:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:02.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:02.978980+0000) 2022-09-08T00:10:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:03.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:03.979135+0000) 2022-09-08T00:10:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:03.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:03.892195+0000) 2022-09-08T00:10:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:04 smithi055 ceph-mon[97033]: pgmap v4847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:04 smithi059 ceph-mon[97450]: pgmap v4847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:05.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:10:05.034+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:10:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:04.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:04.979340+0000) 2022-09-08T00:10:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:04.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:04.892460+0000) 2022-09-08T00:10:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:10:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:10:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:10:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:10:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:05.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:05.979610+0000) 2022-09-08T00:10:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:05.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:05.892693+0000) 2022-09-08T00:10:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:06 smithi055 ceph-mon[97033]: pgmap v4848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:10:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:06 smithi059 ceph-mon[97450]: pgmap v4848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:10:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:06.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:06.979890+0000) 2022-09-08T00:10:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:06.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:06.892931+0000) 2022-09-08T00:10:08.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:07.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:07.980125+0000) 2022-09-08T00:10:08.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:07.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:07.893115+0000) 2022-09-08T00:10:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:08 smithi055 ceph-mon[97033]: pgmap v4849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:10:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:08 smithi059 ceph-mon[97450]: pgmap v4849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:10:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:08.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:08.980347+0000) 2022-09-08T00:10:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:08.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:08.893333+0000) 2022-09-08T00:10:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:09 smithi055 ceph-mon[97033]: osdmap e389: 12 total, 11 up, 11 in 2022-09-08T00:10:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:09 smithi055 ceph-mon[97033]: pgmap v4851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:10:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:09 smithi059 ceph-mon[97450]: osdmap e389: 12 total, 11 up, 11 in 2022-09-08T00:10:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:09 smithi059 ceph-mon[97450]: pgmap v4851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:10:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:09.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:09.980574+0000) 2022-09-08T00:10:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:09.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:09.893549+0000) 2022-09-08T00:10:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:10.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:10.980743+0000) 2022-09-08T00:10:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:10.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:10.893793+0000) 2022-09-08T00:10:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:11.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:11.980961+0000) 2022-09-08T00:10:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:11.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:11.894016+0000) 2022-09-08T00:10:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:12 smithi055 ceph-mon[97033]: pgmap v4852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:10:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:12 smithi059 ceph-mon[97450]: pgmap v4852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:10:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:12.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:12.981202+0000) 2022-09-08T00:10:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:12.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:12.894212+0000) 2022-09-08T00:10:14.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:13.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:13.894412+0000) 2022-09-08T00:10:14.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:13.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:13.981412+0000) 2022-09-08T00:10:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:14 smithi055 ceph-mon[97033]: pgmap v4853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:10:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:14 smithi059 ceph-mon[97450]: pgmap v4853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:10:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:14.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:14.981616+0000) 2022-09-08T00:10:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:14.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:14.894684+0000) 2022-09-08T00:10:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:15.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:15.981890+0000) 2022-09-08T00:10:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:15.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:15.894937+0000) 2022-09-08T00:10:16.542 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:16 smithi059 ceph-mon[97450]: pgmap v4854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:10:16.543 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:16 smithi059 ceph-mon[97450]: osdmap e390: 12 total, 11 up, 11 in 2022-09-08T00:10:16.543 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:10:16.544 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:10:16.545 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:10:16.692 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:16 smithi055 ceph-mon[97033]: pgmap v4854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:10:16.692 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:16 smithi055 ceph-mon[97033]: osdmap e390: 12 total, 11 up, 11 in 2022-09-08T00:10:16.693 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:10:16.693 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:10:16.693 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:10:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:16.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:16.982108+0000) 2022-09-08T00:10:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:16.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:16.895175+0000) 2022-09-08T00:10:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:10:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:10:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:17 smithi055 ceph-mon[97033]: pgmap v4856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-08T00:10:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:10:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:10:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:17 smithi059 ceph-mon[97450]: pgmap v4856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-08T00:10:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:17.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:17.982307+0000) 2022-09-08T00:10:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:17.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:17.895414+0000) 2022-09-08T00:10:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:18.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:18.895599+0000) 2022-09-08T00:10:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:18.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:18.982509+0000) 2022-09-08T00:10:20.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:10:20.042+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:10:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:19.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:19.895868+0000) 2022-09-08T00:10:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:19.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:19.982747+0000) 2022-09-08T00:10:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:10:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:10:20.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:20 smithi055 ceph-mon[97033]: pgmap v4857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:10:20.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:10:20.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:10:20.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.487 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:10:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:10:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:20 smithi059 ceph-mon[97450]: pgmap v4857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:10:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:10:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:10:20.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:20.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:10:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:20.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:20.982935+0000) 2022-09-08T00:10:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:20.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:20.896091+0000) 2022-09-08T00:10:21.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:21 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T00:10:21.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:21 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T00:10:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:21 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T00:10:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:21 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T00:10:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:21 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T00:10:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:21 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T00:10:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:21 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T00:10:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:21 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T00:10:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:21.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:21.983148+0000) 2022-09-08T00:10:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:21.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:21.896316+0000) 2022-09-08T00:10:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:22 smithi055 ceph-mon[97033]: pgmap v4858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:10:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:22 smithi059 ceph-mon[97450]: pgmap v4858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:10:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:22.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:22.983306+0000) 2022-09-08T00:10:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:22.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:22.896565+0000) 2022-09-08T00:10:23.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:23 smithi055 ceph-mon[97033]: pgmap v4859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:10:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:23 smithi059 ceph-mon[97450]: pgmap v4859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:10:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:23.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:23.896808+0000) 2022-09-08T00:10:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:23.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:23.983518+0000) 2022-09-08T00:10:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:24.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:24.897039+0000) 2022-09-08T00:10:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:24.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:24.983728+0000) 2022-09-08T00:10:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:25.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:25.897183+0000) 2022-09-08T00:10:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:25.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:25.983939+0000) 2022-09-08T00:10:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:26 smithi055 ceph-mon[97033]: pgmap v4860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:10:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:26 smithi059 ceph-mon[97450]: pgmap v4860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:10:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:26.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:26.897360+0000) 2022-09-08T00:10:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:26.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:26.984138+0000) 2022-09-08T00:10:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:27.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:27.984333+0000) 2022-09-08T00:10:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:27.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:27.897605+0000) 2022-09-08T00:10:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:28 smithi055 ceph-mon[97033]: pgmap v4861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:10:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:28 smithi059 ceph-mon[97450]: pgmap v4861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:10:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:28.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:28.897864+0000) 2022-09-08T00:10:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:28.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:28.984545+0000) 2022-09-08T00:10:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:29.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:29.984746+0000) 2022-09-08T00:10:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:29.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:29.898047+0000) 2022-09-08T00:10:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:30 smithi055 ceph-mon[97033]: pgmap v4862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:30 smithi059 ceph-mon[97450]: pgmap v4862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:30.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:30.984859+0000) 2022-09-08T00:10:31.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:30.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:30.898243+0000) 2022-09-08T00:10:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:31.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:31.985094+0000) 2022-09-08T00:10:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:31.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:31.898431+0000) 2022-09-08T00:10:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:32 smithi055 ceph-mon[97033]: pgmap v4863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:10:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:32 smithi059 ceph-mon[97450]: pgmap v4863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:10:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:32.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:32.985278+0000) 2022-09-08T00:10:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:32.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:32.898622+0000) 2022-09-08T00:10:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:33 smithi055 ceph-mon[97033]: pgmap v4864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:33 smithi059 ceph-mon[97450]: pgmap v4864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:33.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:33.985531+0000) 2022-09-08T00:10:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:33.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:33.898872+0000) 2022-09-08T00:10:35.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:10:35.051+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:10:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:34.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:34.985774+0000) 2022-09-08T00:10:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:34.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:34.899143+0000) 2022-09-08T00:10:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:35.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:35.985976+0000) 2022-09-08T00:10:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:35.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:35.899402+0000) 2022-09-08T00:10:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:36 smithi055 ceph-mon[97033]: pgmap v4865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:10:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:36 smithi059 ceph-mon[97450]: pgmap v4865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:10:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:36.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:36.986208+0000) 2022-09-08T00:10:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:36.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:36.899715+0000) 2022-09-08T00:10:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:37.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:37.986438+0000) 2022-09-08T00:10:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:37.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:37.899966+0000) 2022-09-08T00:10:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:38 smithi055 ceph-mon[97033]: pgmap v4866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:38 smithi059 ceph-mon[97450]: pgmap v4866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:38.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:38.986651+0000) 2022-09-08T00:10:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:38.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:38.900148+0000) 2022-09-08T00:10:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:39.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:39.986789+0000) 2022-09-08T00:10:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:39.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:39.900321+0000) 2022-09-08T00:10:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:40 smithi055 ceph-mon[97033]: pgmap v4867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:40 smithi059 ceph-mon[97450]: pgmap v4867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:40.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:40.987032+0000) 2022-09-08T00:10:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:40.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:40.900558+0000) 2022-09-08T00:10:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:41 smithi059 ceph-mon[97450]: pgmap v4868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:10:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:41 smithi055 ceph-mon[97033]: pgmap v4868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:10:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:41.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:41.900745+0000) 2022-09-08T00:10:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:41.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:41.987268+0000) 2022-09-08T00:10:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:42.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:42.987500+0000) 2022-09-08T00:10:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:42.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:42.900939+0000) 2022-09-08T00:10:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:43.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:43.987707+0000) 2022-09-08T00:10:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:43.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:43.901189+0000) 2022-09-08T00:10:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:44 smithi055 ceph-mon[97033]: pgmap v4869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:44 smithi059 ceph-mon[97450]: pgmap v4869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:44.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:44.987942+0000) 2022-09-08T00:10:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:44.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:44.901430+0000) 2022-09-08T00:10:46.328 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:45.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:45.988184+0000) 2022-09-08T00:10:46.328 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:45.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:45.901697+0000) 2022-09-08T00:10:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:46 smithi055 ceph-mon[97033]: pgmap v4870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:10:46.741 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:10:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:46 smithi059 ceph-mon[97450]: pgmap v4870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:10:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:10:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:46.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:46.988455+0000) 2022-09-08T00:10:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:46.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:46.901880+0000) 2022-09-08T00:10:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:47 smithi055 ceph-mon[97033]: pgmap v4871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:10:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:47 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T00:10:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:47 smithi059 ceph-mon[97450]: pgmap v4871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:10:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:47 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T00:10:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:47.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:47.988705+0000) 2022-09-08T00:10:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:47.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:47.902020+0000) 2022-09-08T00:10:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:48 smithi055 ceph-mon[97033]: pgmap v4872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:10:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:48 smithi059 ceph-mon[97450]: pgmap v4872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:10:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:48.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:48.988917+0000) 2022-09-08T00:10:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:48.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:48.902265+0000) 2022-09-08T00:10:50.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:10:50.063+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:10:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:49.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:49.989177+0000) 2022-09-08T00:10:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:49.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:49.902536+0000) 2022-09-08T00:10:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:50.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:50.902738+0000) 2022-09-08T00:10:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:50.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:50.989392+0000) 2022-09-08T00:10:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:51 smithi055 ceph-mon[97033]: pgmap v4873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T00:10:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:51 smithi059 ceph-mon[97450]: pgmap v4873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T00:10:52.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:51.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:51.902881+0000) 2022-09-08T00:10:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:51.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:51.989615+0000) 2022-09-08T00:10:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:52.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:52.989786+0000) 2022-09-08T00:10:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:52.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:52.903117+0000) 2022-09-08T00:10:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:53 smithi055 ceph-mon[97033]: pgmap v4874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:10:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:53 smithi059 ceph-mon[97450]: pgmap v4874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:10:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:53.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:53.990009+0000) 2022-09-08T00:10:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:53.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:53.903381+0000) 2022-09-08T00:10:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:54 smithi059 ceph-mon[97450]: pgmap v4875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:10:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:54 smithi055 ceph-mon[97033]: pgmap v4875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:10:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:54.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:54.990233+0000) 2022-09-08T00:10:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:54.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:54.903680+0000) 2022-09-08T00:10:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:55.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:55.903911+0000) 2022-09-08T00:10:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:55.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:55.990467+0000) 2022-09-08T00:10:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:56.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:56.904133+0000) 2022-09-08T00:10:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:56.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:56.990590+0000) 2022-09-08T00:10:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:57 smithi055 ceph-mon[97033]: pgmap v4876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:10:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:57 smithi059 ceph-mon[97450]: pgmap v4876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:10:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:57.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:57.990758+0000) 2022-09-08T00:10:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:57.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:57.904331+0000) 2022-09-08T00:10:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:58.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:58.990929+0000) 2022-09-08T00:10:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:58.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:58.904489+0000) 2022-09-08T00:10:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:10:59 smithi055 ceph-mon[97033]: pgmap v4877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:10:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:10:59 smithi059 ceph-mon[97450]: pgmap v4877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:10:59.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:59.991210+0000) 2022-09-08T00:11:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:10:59.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:10:59.904760+0000) 2022-09-08T00:11:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:00.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:00.991433+0000) 2022-09-08T00:11:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:00.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:00.904965+0000) 2022-09-08T00:11:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:01 smithi055 ceph-mon[97033]: pgmap v4878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:11:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:01 smithi059 ceph-mon[97450]: pgmap v4878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:11:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:01.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:01.991722+0000) 2022-09-08T00:11:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:01.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:01.905108+0000) 2022-09-08T00:11:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:02.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:02.991914+0000) 2022-09-08T00:11:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:02.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:02.905294+0000) 2022-09-08T00:11:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:03 smithi055 ceph-mon[97033]: pgmap v4879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:03 smithi059 ceph-mon[97450]: pgmap v4879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:03.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:03.905513+0000) 2022-09-08T00:11:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:03.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:03.992165+0000) 2022-09-08T00:11:05.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:11:05.072+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:11:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:04.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:04.905708+0000) 2022-09-08T00:11:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:04.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:04.992406+0000) 2022-09-08T00:11:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:05 smithi055 ceph-mon[97033]: pgmap v4880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:11:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:11:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:05 smithi059 ceph-mon[97450]: pgmap v4880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:11:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:11:06.382 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:05.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:05.992659+0000) 2022-09-08T00:11:06.383 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:05.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:05.905843+0000) 2022-09-08T00:11:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:06 smithi055 ceph-mon[97033]: pgmap v4881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:11:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:06 smithi059 ceph-mon[97450]: pgmap v4881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:11:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:06.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:06.906049+0000) 2022-09-08T00:11:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:06.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:06.992911+0000) 2022-09-08T00:11:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:07.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:07.906256+0000) 2022-09-08T00:11:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:07.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:07.993121+0000) 2022-09-08T00:11:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:08.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:08.993287+0000) 2022-09-08T00:11:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:08.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:08.906475+0000) 2022-09-08T00:11:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:09 smithi055 ceph-mon[97033]: pgmap v4882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:11:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:09 smithi055 ceph-mon[97033]: osdmap e391: 12 total, 11 up, 11 in 2022-09-08T00:11:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:09 smithi059 ceph-mon[97450]: pgmap v4882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:11:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:09 smithi059 ceph-mon[97450]: osdmap e391: 12 total, 11 up, 11 in 2022-09-08T00:11:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:09.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:09.993535+0000) 2022-09-08T00:11:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:09.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:09.906717+0000) 2022-09-08T00:11:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:10.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:10.993783+0000) 2022-09-08T00:11:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:10.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:10.906967+0000) 2022-09-08T00:11:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:11 smithi055 ceph-mon[97033]: pgmap v4884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:11:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:11 smithi055 ceph-mon[97033]: osdmap e392: 12 total, 11 up, 11 in 2022-09-08T00:11:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:11 smithi059 ceph-mon[97450]: pgmap v4884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:11:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:11 smithi059 ceph-mon[97450]: osdmap e392: 12 total, 11 up, 11 in 2022-09-08T00:11:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:11.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:11.994018+0000) 2022-09-08T00:11:12.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:11.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:11.907213+0000) 2022-09-08T00:11:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:12.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:12.994255+0000) 2022-09-08T00:11:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:12.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:12.907429+0000) 2022-09-08T00:11:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:13 smithi055 ceph-mon[97033]: pgmap v4886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:11:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:13 smithi059 ceph-mon[97450]: pgmap v4886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:11:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:13.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:13.994522+0000) 2022-09-08T00:11:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:13.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:13.907687+0000) 2022-09-08T00:11:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:14 smithi059 ceph-mon[97450]: pgmap v4887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:11:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:14 smithi055 ceph-mon[97033]: pgmap v4887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:11:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:14.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:14.994718+0000) 2022-09-08T00:11:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:14.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:14.907937+0000) 2022-09-08T00:11:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:15.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:15.994948+0000) 2022-09-08T00:11:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:15.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:15.908204+0000) 2022-09-08T00:11:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:16.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:16.995192+0000) 2022-09-08T00:11:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:16.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:16.908417+0000) 2022-09-08T00:11:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:17 smithi055 ceph-mon[97033]: pgmap v4888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:11:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:17 smithi059 ceph-mon[97450]: pgmap v4888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:11:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:17.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:17.995372+0000) 2022-09-08T00:11:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:17.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:17.908676+0000) 2022-09-08T00:11:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:18 smithi059 ceph-mon[97450]: pgmap v4889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:11:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:18 smithi055 ceph-mon[97033]: pgmap v4889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:11:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:18.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:18.995616+0000) 2022-09-08T00:11:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:18.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:18.908913+0000) 2022-09-08T00:11:20.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:11:20.076+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:11:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:19.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:19.995871+0000) 2022-09-08T00:11:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:19.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:19.909146+0000) 2022-09-08T00:11:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:20.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:20.996100+0000) 2022-09-08T00:11:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:20.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:20.909312+0000) 2022-09-08T00:11:21.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:21 smithi055 ceph-mon[97033]: pgmap v4890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:11:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:21 smithi059 ceph-mon[97450]: pgmap v4890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:11:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:21.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:21.996264+0000) 2022-09-08T00:11:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:21.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:21.909533+0000) 2022-09-08T00:11:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:22 smithi059 ceph-mon[97450]: pgmap v4891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:11:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:22 smithi055 ceph-mon[97033]: pgmap v4891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:11:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:22.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:22.996399+0000) 2022-09-08T00:11:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:22.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:22.909764+0000) 2022-09-08T00:11:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:23.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:23.996618+0000) 2022-09-08T00:11:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:23.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:23.909925+0000) 2022-09-08T00:11:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:24.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:24.996925+0000) 2022-09-08T00:11:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:24.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:24.910078+0000) 2022-09-08T00:11:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:25 smithi055 ceph-mon[97033]: pgmap v4892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:11:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:25 smithi059 ceph-mon[97450]: pgmap v4892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:11:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:25.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:25.997163+0000) 2022-09-08T00:11:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:25.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:25.910302+0000) 2022-09-08T00:11:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:26.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:26.910556+0000) 2022-09-08T00:11:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:26.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:26.997401+0000) 2022-09-08T00:11:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:27 smithi055 ceph-mon[97033]: pgmap v4893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:11:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:27 smithi059 ceph-mon[97450]: pgmap v4893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:11:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:27.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:27.997548+0000) 2022-09-08T00:11:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:27.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:27.910748+0000) 2022-09-08T00:11:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:28 smithi059 ceph-mon[97450]: pgmap v4894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:28 smithi055 ceph-mon[97033]: pgmap v4894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:28.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:28.997777+0000) 2022-09-08T00:11:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:28.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:28.910933+0000) 2022-09-08T00:11:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:29.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:29.997964+0000) 2022-09-08T00:11:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:29.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:29.911157+0000) 2022-09-08T00:11:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:30.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:30.998182+0000) 2022-09-08T00:11:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:30.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:30.911347+0000) 2022-09-08T00:11:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:31 smithi055 ceph-mon[97033]: pgmap v4895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:11:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:31 smithi059 ceph-mon[97450]: pgmap v4895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:11:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:31.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:31.998414+0000) 2022-09-08T00:11:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:31.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:31.911592+0000) 2022-09-08T00:11:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:32 smithi059 ceph-mon[97450]: pgmap v4896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:32.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:32 smithi055 ceph-mon[97033]: pgmap v4896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:32.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:32.998615+0000) 2022-09-08T00:11:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:32.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:32.911807+0000) 2022-09-08T00:11:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:33.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:33.998843+0000) 2022-09-08T00:11:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:33.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:33.912044+0000) 2022-09-08T00:11:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:34.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:34.999061+0000) 2022-09-08T00:11:35.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:11:35.088+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:11:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:34.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:34.912279+0000) 2022-09-08T00:11:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:35 smithi055 ceph-mon[97033]: pgmap v4897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:35 smithi059 ceph-mon[97450]: pgmap v4897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:35.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:35.999213+0000) 2022-09-08T00:11:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:35.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:35.912575+0000) 2022-09-08T00:11:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:36 smithi059 ceph-mon[97450]: pgmap v4898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:11:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:36 smithi055 ceph-mon[97033]: pgmap v4898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:11:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:36.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:36.912795+0000) 2022-09-08T00:11:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:36.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:36.999459+0000) 2022-09-08T00:11:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:37.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:37.999723+0000) 2022-09-08T00:11:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:37.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:37.913026+0000) 2022-09-08T00:11:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:38.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:38.999943+0000) 2022-09-08T00:11:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:38.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:38.913200+0000) 2022-09-08T00:11:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:39 smithi055 ceph-mon[97033]: pgmap v4899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:39 smithi059 ceph-mon[97450]: pgmap v4899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:39.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:40.000200+0000) 2022-09-08T00:11:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:39.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:39.913329+0000) 2022-09-08T00:11:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:40.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:41.000366+0000) 2022-09-08T00:11:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:40.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:40.913554+0000) 2022-09-08T00:11:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:41 smithi055 ceph-mon[97033]: pgmap v4900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:11:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:41 smithi059 ceph-mon[97450]: pgmap v4900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:11:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:42.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:42.000622+0000) 2022-09-08T00:11:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:41.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:41.913770+0000) 2022-09-08T00:11:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:42 smithi059 ceph-mon[97450]: pgmap v4901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:11:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:42 smithi055 ceph-mon[97033]: pgmap v4901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:11:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:43.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:43.000943+0000) 2022-09-08T00:11:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:42.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:42.914006+0000) 2022-09-08T00:11:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:44.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:44.001099+0000) 2022-09-08T00:11:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:43.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:43.914175+0000) 2022-09-08T00:11:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:45.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:45.001266+0000) 2022-09-08T00:11:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:44.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:44.914323+0000) 2022-09-08T00:11:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:45 smithi055 ceph-mon[97033]: pgmap v4902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:11:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:45 smithi059 ceph-mon[97450]: pgmap v4902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:11:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:46.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:46.001504+0000) 2022-09-08T00:11:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:45.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:45.914519+0000) 2022-09-08T00:11:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:11:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:11:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:47.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:47.001783+0000) 2022-09-08T00:11:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:46.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:46.914728+0000) 2022-09-08T00:11:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:47 smithi055 ceph-mon[97033]: pgmap v4903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:11:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:11:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:11:47.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:11:47.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:11:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:47 smithi059 ceph-mon[97450]: pgmap v4903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:11:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:11:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:11:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:11:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:11:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:48.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:48.002022+0000) 2022-09-08T00:11:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:47.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:47.914911+0000) 2022-09-08T00:11:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:48 smithi059 ceph-mon[97450]: pgmap v4904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:11:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:48 smithi055 ceph-mon[97033]: pgmap v4904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:11:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:49.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:49.002160+0000) 2022-09-08T00:11:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:48.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:48.915047+0000) 2022-09-08T00:11:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:50.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:50.002383+0000) 2022-09-08T00:11:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:49.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:49.915283+0000) 2022-09-08T00:11:50.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:11:50.103+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:11:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:50.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:50.915511+0000) 2022-09-08T00:11:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:51.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:51.002562+0000) 2022-09-08T00:11:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:51 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T00:11:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:11:51.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:11:51.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:51 smithi055 ceph-mon[97033]: pgmap v4905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:11:51.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:11:51.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:11:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:51 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T00:11:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:11:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:11:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:51 smithi059 ceph-mon[97450]: pgmap v4905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:11:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:11:51.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:11:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:51.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:51.915764+0000) 2022-09-08T00:11:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:52.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:52.002777+0000) 2022-09-08T00:11:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:52 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T00:11:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:52 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T00:11:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:53.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:53.003080+0000) 2022-09-08T00:11:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:52.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:52.915997+0000) 2022-09-08T00:11:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:53 smithi055 ceph-mon[97033]: pgmap v4906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:53 smithi059 ceph-mon[97450]: pgmap v4906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:54.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:54.003337+0000) 2022-09-08T00:11:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:53.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:53.916214+0000) 2022-09-08T00:11:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:54.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:54.916474+0000) 2022-09-08T00:11:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:55.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:55.003566+0000) 2022-09-08T00:11:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:55 smithi055 ceph-mon[97033]: pgmap v4907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:55 smithi059 ceph-mon[97450]: pgmap v4907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:56.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:56.003751+0000) 2022-09-08T00:11:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:55.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:55.916745+0000) 2022-09-08T00:11:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:56 smithi059 ceph-mon[97450]: pgmap v4908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:11:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:56 smithi055 ceph-mon[97033]: pgmap v4908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:11:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:57.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:57.003949+0000) 2022-09-08T00:11:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:56.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:56.916923+0000) 2022-09-08T00:11:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:58.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:58.004150+0000) 2022-09-08T00:11:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:57.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:57.917128+0000) 2022-09-08T00:11:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:11:59.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:59.004336+0000) 2022-09-08T00:11:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:58.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:58.917339+0000) 2022-09-08T00:11:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:11:59 smithi055 ceph-mon[97033]: pgmap v4909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:11:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:11:59 smithi059 ceph-mon[97450]: pgmap v4909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:00.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:00.004520+0000) 2022-09-08T00:12:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:11:59.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:11:59.917574+0000) 2022-09-08T00:12:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:01.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:01.004689+0000) 2022-09-08T00:12:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:00.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:00.917798+0000) 2022-09-08T00:12:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:01 smithi055 ceph-mon[97033]: pgmap v4910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:12:01.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:01 smithi059 ceph-mon[97450]: pgmap v4910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:12:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:02.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:02.004865+0000) 2022-09-08T00:12:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:01.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:01.917998+0000) 2022-09-08T00:12:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:03.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:03.005092+0000) 2022-09-08T00:12:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:02.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:02.918208+0000) 2022-09-08T00:12:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:03 smithi055 ceph-mon[97033]: pgmap v4911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:03 smithi059 ceph-mon[97450]: pgmap v4911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:04.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:04.005311+0000) 2022-09-08T00:12:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:03.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:03.918424+0000) 2022-09-08T00:12:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:05.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:05.005554+0000) 2022-09-08T00:12:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:04.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:04.918617+0000) 2022-09-08T00:12:05.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:12:05.110+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:12:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:05 smithi055 ceph-mon[97033]: pgmap v4912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:12:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:12:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:05 smithi059 ceph-mon[97450]: pgmap v4912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:12:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:12:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:06.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:06.005799+0000) 2022-09-08T00:12:06.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:05.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:05.918861+0000) 2022-09-08T00:12:06.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:06 smithi055 ceph-mon[97033]: pgmap v4913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:12:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:06 smithi059 ceph-mon[97450]: pgmap v4913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:12:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:06.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:06.919107+0000) 2022-09-08T00:12:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:07.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:07.005962+0000) 2022-09-08T00:12:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:08.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:08.006099+0000) 2022-09-08T00:12:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:07.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:07.919330+0000) 2022-09-08T00:12:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:09.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:09.006293+0000) 2022-09-08T00:12:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:08.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:08.919566+0000) 2022-09-08T00:12:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:09 smithi055 ceph-mon[97033]: pgmap v4914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:12:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:09 smithi055 ceph-mon[97033]: osdmap e393: 12 total, 11 up, 11 in 2022-09-08T00:12:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:09 smithi059 ceph-mon[97450]: pgmap v4914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:12:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:09 smithi059 ceph-mon[97450]: osdmap e393: 12 total, 11 up, 11 in 2022-09-08T00:12:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:10.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:10.006494+0000) 2022-09-08T00:12:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:09.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:09.919736+0000) 2022-09-08T00:12:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:11.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:11.006743+0000) 2022-09-08T00:12:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:10.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:10.919990+0000) 2022-09-08T00:12:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:11 smithi055 ceph-mon[97033]: pgmap v4916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:12:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:11 smithi055 ceph-mon[97033]: osdmap e394: 12 total, 11 up, 11 in 2022-09-08T00:12:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:11 smithi059 ceph-mon[97450]: pgmap v4916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:12:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:11 smithi059 ceph-mon[97450]: osdmap e394: 12 total, 11 up, 11 in 2022-09-08T00:12:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:12.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:12.006981+0000) 2022-09-08T00:12:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:11.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:11.920213+0000) 2022-09-08T00:12:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:13.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:13.007190+0000) 2022-09-08T00:12:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:12.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:12.920510+0000) 2022-09-08T00:12:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:13 smithi055 ceph-mon[97033]: pgmap v4918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:12:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:13 smithi059 ceph-mon[97450]: pgmap v4918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:12:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:14.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:14.007407+0000) 2022-09-08T00:12:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:13.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:13.920763+0000) 2022-09-08T00:12:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:15.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:15.007589+0000) 2022-09-08T00:12:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:14.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:14.920931+0000) 2022-09-08T00:12:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:15 smithi055 ceph-mon[97033]: pgmap v4919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:12:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:15 smithi059 ceph-mon[97450]: pgmap v4919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:12:16.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:15.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:15.921174+0000) 2022-09-08T00:12:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:16.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:16.007801+0000) 2022-09-08T00:12:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:16 smithi055 ceph-mon[97033]: pgmap v4920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:12:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:16 smithi059 ceph-mon[97450]: pgmap v4920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:12:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:17.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:17.008021+0000) 2022-09-08T00:12:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:16.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:16.921404+0000) 2022-09-08T00:12:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:17 smithi055 ceph-mon[97033]: pgmap v4921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 249 B/s wr, 7 op/s 2022-09-08T00:12:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:17 smithi055 ceph-mon[97033]: pgmap v4922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T00:12:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:12:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:17 smithi059 ceph-mon[97450]: pgmap v4921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 249 B/s wr, 7 op/s 2022-09-08T00:12:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:17 smithi059 ceph-mon[97450]: pgmap v4922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T00:12:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:12:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:18.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:18.008261+0000) 2022-09-08T00:12:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:17.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:17.921577+0000) 2022-09-08T00:12:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:19.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:19.008431+0000) 2022-09-08T00:12:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:18.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:18.921800+0000) 2022-09-08T00:12:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:19 smithi055 ceph-mon[97033]: pgmap v4923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 125 B/s wr, 3 op/s 2022-09-08T00:12:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:19 smithi059 ceph-mon[97450]: pgmap v4923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 125 B/s wr, 3 op/s 2022-09-08T00:12:20.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:12:20.117+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:12:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:20.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:20.008668+0000) 2022-09-08T00:12:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:19.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:19.921998+0000) 2022-09-08T00:12:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:21.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:21.008890+0000) 2022-09-08T00:12:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:20.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:20.922209+0000) 2022-09-08T00:12:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:21 smithi055 ceph-mon[97033]: pgmap v4924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T00:12:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:21 smithi059 ceph-mon[97450]: pgmap v4924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T00:12:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:22.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:22.009053+0000) 2022-09-08T00:12:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:21.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:21.922424+0000) 2022-09-08T00:12:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:23.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:23.009241+0000) 2022-09-08T00:12:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:22.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:22.922589+0000) 2022-09-08T00:12:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:23 smithi055 ceph-mon[97033]: pgmap v4925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T00:12:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:23 smithi059 ceph-mon[97450]: pgmap v4925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T00:12:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:24.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:24.009427+0000) 2022-09-08T00:12:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:23.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:23.922837+0000) 2022-09-08T00:12:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:25.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:25.009552+0000) 2022-09-08T00:12:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:24.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:24.923113+0000) 2022-09-08T00:12:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:25 smithi055 ceph-mon[97033]: pgmap v4926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:12:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:25 smithi059 ceph-mon[97450]: pgmap v4926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:12:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:26.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:26.009757+0000) 2022-09-08T00:12:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:25.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:25.923361+0000) 2022-09-08T00:12:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:27.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:27.009957+0000) 2022-09-08T00:12:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:26.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:26.923629+0000) 2022-09-08T00:12:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:27 smithi055 ceph-mon[97033]: pgmap v4927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:12:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:27 smithi059 ceph-mon[97450]: pgmap v4927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:12:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:28.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:28.010119+0000) 2022-09-08T00:12:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:27.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:27.923916+0000) 2022-09-08T00:12:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:29.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:29.010334+0000) 2022-09-08T00:12:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:28.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:28.924117+0000) 2022-09-08T00:12:29.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:29 smithi055 ceph-mon[97033]: pgmap v4928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:30.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:29 smithi059 ceph-mon[97450]: pgmap v4928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:30.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:30.010533+0000) 2022-09-08T00:12:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:29.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:29.924287+0000) 2022-09-08T00:12:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:31.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:31.010743+0000) 2022-09-08T00:12:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:30.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:30.924485+0000) 2022-09-08T00:12:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:31 smithi055 ceph-mon[97033]: pgmap v4929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:12:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:31 smithi059 ceph-mon[97450]: pgmap v4929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:12:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:32.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:32.010993+0000) 2022-09-08T00:12:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:31.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:31.924726+0000) 2022-09-08T00:12:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:33.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:33.011155+0000) 2022-09-08T00:12:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:32.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:32.924897+0000) 2022-09-08T00:12:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:33 smithi055 ceph-mon[97033]: pgmap v4930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:33 smithi059 ceph-mon[97450]: pgmap v4930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:34.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:34.011401+0000) 2022-09-08T00:12:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:33.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:33.925013+0000) 2022-09-08T00:12:35.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:12:35.132+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:12:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:35.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:35.011627+0000) 2022-09-08T00:12:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:34.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:34.925220+0000) 2022-09-08T00:12:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:35 smithi055 ceph-mon[97033]: pgmap v4931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:35 smithi059 ceph-mon[97450]: pgmap v4931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:36.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:36.011834+0000) 2022-09-08T00:12:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:35.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:35.925433+0000) 2022-09-08T00:12:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:36.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:36.925711+0000) 2022-09-08T00:12:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:37.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:37.012062+0000) 2022-09-08T00:12:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:37 smithi055 ceph-mon[97033]: pgmap v4932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:12:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:37 smithi059 ceph-mon[97450]: pgmap v4932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:12:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:38.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:38.012272+0000) 2022-09-08T00:12:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:37.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:37.925938+0000) 2022-09-08T00:12:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:39.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:39.012493+0000) 2022-09-08T00:12:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:38.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:38.926163+0000) 2022-09-08T00:12:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:39 smithi055 ceph-mon[97033]: pgmap v4933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:12:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:39 smithi059 ceph-mon[97450]: pgmap v4933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:12:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:40.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:40.012696+0000) 2022-09-08T00:12:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:39.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:39.926381+0000) 2022-09-08T00:12:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:41.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:41.012871+0000) 2022-09-08T00:12:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:40.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:40.926678+0000) 2022-09-08T00:12:41.791 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:41 smithi059 ceph-mon[97450]: pgmap v4934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:12:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:41 smithi055 ceph-mon[97033]: pgmap v4934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:12:42.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:42.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:42.013065+0000) 2022-09-08T00:12:42.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:41.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:41.926915+0000) 2022-09-08T00:12:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:43.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:43.013238+0000) 2022-09-08T00:12:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:42.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:42.927139+0000) 2022-09-08T00:12:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:43 smithi055 ceph-mon[97033]: pgmap v4935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T00:12:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:43 smithi059 ceph-mon[97450]: pgmap v4935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T00:12:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:44.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:44.013412+0000) 2022-09-08T00:12:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:43.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:43.927367+0000) 2022-09-08T00:12:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:45.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:45.013603+0000) 2022-09-08T00:12:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:44.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:44.927596+0000) 2022-09-08T00:12:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:45 smithi055 ceph-mon[97033]: pgmap v4936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T00:12:46.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:45 smithi059 ceph-mon[97450]: pgmap v4936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T00:12:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:46.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:46.013856+0000) 2022-09-08T00:12:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:45.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:45.927844+0000) 2022-09-08T00:12:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:47.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:47.014088+0000) 2022-09-08T00:12:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:46.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:46.928001+0000) 2022-09-08T00:12:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:47 smithi055 ceph-mon[97033]: pgmap v4937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:12:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:47 smithi059 ceph-mon[97450]: pgmap v4937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:12:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:48.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:48.014273+0000) 2022-09-08T00:12:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:47.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:47.928215+0000) 2022-09-08T00:12:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:49.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:49.014392+0000) 2022-09-08T00:12:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:48.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:48.928458+0000) 2022-09-08T00:12:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:49 smithi055 ceph-mon[97033]: pgmap v4938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:12:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:49 smithi059 ceph-mon[97450]: pgmap v4938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:12:50.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:12:50.148+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:12:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:50.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:50.014681+0000) 2022-09-08T00:12:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:49.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:49.928630+0000) 2022-09-08T00:12:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:51.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:51.014841+0000) 2022-09-08T00:12:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:50.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:50.928832+0000) 2022-09-08T00:12:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:51 smithi055 ceph-mon[97033]: pgmap v4939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:12:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:51 smithi059 ceph-mon[97450]: pgmap v4939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:12:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:52.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:52.015040+0000) 2022-09-08T00:12:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:51.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:51.929071+0000) 2022-09-08T00:12:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:53.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:53.015207+0000) 2022-09-08T00:12:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:52.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:52.929299+0000) 2022-09-08T00:12:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:53 smithi055 ceph-mon[97033]: pgmap v4940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:53 smithi059 ceph-mon[97450]: pgmap v4940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:54.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:54.015431+0000) 2022-09-08T00:12:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:53.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:53.929541+0000) 2022-09-08T00:12:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:55.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:55.015606+0000) 2022-09-08T00:12:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:54.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:54.929756+0000) 2022-09-08T00:12:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:55 smithi055 ceph-mon[97033]: pgmap v4941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:55 smithi059 ceph-mon[97450]: pgmap v4941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:12:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:56.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:56.015846+0000) 2022-09-08T00:12:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:55.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:55.930010+0000) 2022-09-08T00:12:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:57.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:57.016036+0000) 2022-09-08T00:12:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:56.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:56.930226+0000) 2022-09-08T00:12:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:57 smithi055 ceph-mon[97033]: pgmap v4942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:12:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:57 smithi059 ceph-mon[97450]: pgmap v4942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:12:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:57.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:57.930485+0000) 2022-09-08T00:12:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:58.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:58.016264+0000) 2022-09-08T00:12:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:12:59.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:59.016477+0000) 2022-09-08T00:12:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:58.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:58.930760+0000) 2022-09-08T00:12:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:12:59 smithi055 ceph-mon[97033]: pgmap v4943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:12:59 smithi059 ceph-mon[97450]: pgmap v4943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:00.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:00.016710+0000) 2022-09-08T00:13:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:12:59.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:12:59.930980+0000) 2022-09-08T00:13:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:00.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:00.931135+0000) 2022-09-08T00:13:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:01.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:01.016934+0000) 2022-09-08T00:13:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:01 smithi059 ceph-mon[97450]: pgmap v4944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:13:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:01 smithi055 ceph-mon[97033]: pgmap v4944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:13:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:01.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:01.931347+0000) 2022-09-08T00:13:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:02.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:02.017155+0000) 2022-09-08T00:13:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:03.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:03.017335+0000) 2022-09-08T00:13:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:02.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:02.931577+0000) 2022-09-08T00:13:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:03 smithi055 ceph-mon[97033]: pgmap v4945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:03 smithi059 ceph-mon[97450]: pgmap v4945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:04.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:04.017558+0000) 2022-09-08T00:13:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:03.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:03.931819+0000) 2022-09-08T00:13:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:13:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:13:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:13:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:13:05.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:13:05.157+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:13:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:05.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:05.017779+0000) 2022-09-08T00:13:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:04.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:04.932007+0000) 2022-09-08T00:13:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:05 smithi055 ceph-mon[97033]: pgmap v4946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:05 smithi059 ceph-mon[97450]: pgmap v4946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:05.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:05.932248+0000) 2022-09-08T00:13:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:06.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:06.018078+0000) 2022-09-08T00:13:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:06.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:06.932474+0000) 2022-09-08T00:13:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:07.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:07.018280+0000) 2022-09-08T00:13:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:07 smithi055 ceph-mon[97033]: pgmap v4947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:13:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:07 smithi059 ceph-mon[97450]: pgmap v4947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:13:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:08.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:08.018523+0000) 2022-09-08T00:13:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:07.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:07.932659+0000) 2022-09-08T00:13:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:09.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:09.018800+0000) 2022-09-08T00:13:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:08.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:08.932888+0000) 2022-09-08T00:13:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:09 smithi055 ceph-mon[97033]: pgmap v4948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:13:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:09 smithi055 ceph-mon[97033]: osdmap e395: 12 total, 11 up, 11 in 2022-09-08T00:13:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:09 smithi059 ceph-mon[97450]: pgmap v4948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:13:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:09 smithi059 ceph-mon[97450]: osdmap e395: 12 total, 11 up, 11 in 2022-09-08T00:13:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:09.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:09.933120+0000) 2022-09-08T00:13:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:10.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:10.018994+0000) 2022-09-08T00:13:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:10.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:10.933356+0000) 2022-09-08T00:13:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:11.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:11.019205+0000) 2022-09-08T00:13:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:11 smithi059 ceph-mon[97450]: pgmap v4950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:13:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:11 smithi055 ceph-mon[97033]: pgmap v4950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:13:12.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:12.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:12.019406+0000) 2022-09-08T00:13:12.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:11.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:11.933666+0000) 2022-09-08T00:13:13.417 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:13.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:13.019584+0000) 2022-09-08T00:13:13.421 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:12.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:12.933945+0000) 2022-09-08T00:13:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:13 smithi055 ceph-mon[97033]: pgmap v4951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:13:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:13 smithi059 ceph-mon[97450]: pgmap v4951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:13:14.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:13.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:13.934211+0000) 2022-09-08T00:13:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:14.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:14.019805+0000) 2022-09-08T00:13:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:15.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:15.019981+0000) 2022-09-08T00:13:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:14.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:14.934510+0000) 2022-09-08T00:13:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:15 smithi055 ceph-mon[97033]: pgmap v4952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:13:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:15 smithi055 ceph-mon[97033]: osdmap e396: 12 total, 11 up, 11 in 2022-09-08T00:13:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:15 smithi059 ceph-mon[97450]: pgmap v4952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:13:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:15 smithi059 ceph-mon[97450]: osdmap e396: 12 total, 11 up, 11 in 2022-09-08T00:13:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:16.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:16.020213+0000) 2022-09-08T00:13:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:15.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:15.934771+0000) 2022-09-08T00:13:17.021 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:16.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:16.935037+0000) 2022-09-08T00:13:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:17.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:17.020335+0000) 2022-09-08T00:13:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:17 smithi055 ceph-mon[97033]: pgmap v4954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:13:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:13:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:13:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:13:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:17 smithi059 ceph-mon[97450]: pgmap v4954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:13:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:13:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:13:18.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:13:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:18.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:18.020562+0000) 2022-09-08T00:13:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:17.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:17.935249+0000) 2022-09-08T00:13:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:18.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:18.935465+0000) 2022-09-08T00:13:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:19.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:19.020723+0000) 2022-09-08T00:13:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:19 smithi055 ceph-mon[97033]: pgmap v4955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:13:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:19 smithi059 ceph-mon[97450]: pgmap v4955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:13:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:20.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:20.020895+0000) 2022-09-08T00:13:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:19.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:19.935670+0000) 2022-09-08T00:13:20.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:13:20.164+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:13:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:21.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:21.021059+0000) 2022-09-08T00:13:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:20.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:20.935934+0000) 2022-09-08T00:13:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:21 smithi059 ceph-mon[97450]: pgmap v4956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:13:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:21 smithi055 ceph-mon[97033]: pgmap v4956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:13:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:22.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:22.021282+0000) 2022-09-08T00:13:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:21.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:21.936098+0000) 2022-09-08T00:13:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:23.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:23.021589+0000) 2022-09-08T00:13:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:22.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:22.936307+0000) 2022-09-08T00:13:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:23 smithi055 ceph-mon[97033]: pgmap v4957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:13:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:23 smithi059 ceph-mon[97450]: pgmap v4957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:13:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:24.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:24.021883+0000) 2022-09-08T00:13:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:23.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:23.936550+0000) 2022-09-08T00:13:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:25.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:25.022068+0000) 2022-09-08T00:13:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:24.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:24.936778+0000) 2022-09-08T00:13:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:25 smithi055 ceph-mon[97033]: pgmap v4958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:13:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:25 smithi059 ceph-mon[97450]: pgmap v4958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:13:26.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:25.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:25.937014+0000) 2022-09-08T00:13:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:26.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:26.022319+0000) 2022-09-08T00:13:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:27.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:27.022486+0000) 2022-09-08T00:13:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:26.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:26.937243+0000) 2022-09-08T00:13:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:27 smithi055 ceph-mon[97033]: pgmap v4959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:13:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:27 smithi059 ceph-mon[97450]: pgmap v4959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:13:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:27.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:27.937471+0000) 2022-09-08T00:13:28.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:28.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:28.022700+0000) 2022-09-08T00:13:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:28.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:28.937742+0000) 2022-09-08T00:13:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:29.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:29.022821+0000) 2022-09-08T00:13:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:29 smithi055 ceph-mon[97033]: pgmap v4960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:29 smithi059 ceph-mon[97450]: pgmap v4960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:30.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:30.022993+0000) 2022-09-08T00:13:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:29.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:29.937985+0000) 2022-09-08T00:13:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:31.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:31.023225+0000) 2022-09-08T00:13:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:30.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:30.938216+0000) 2022-09-08T00:13:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:31 smithi059 ceph-mon[97450]: pgmap v4961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:13:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:31 smithi055 ceph-mon[97033]: pgmap v4961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:13:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:32.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:32.023447+0000) 2022-09-08T00:13:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:31.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:31.938453+0000) 2022-09-08T00:13:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:33.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:33.023689+0000) 2022-09-08T00:13:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:32.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:32.938705+0000) 2022-09-08T00:13:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:33 smithi055 ceph-mon[97033]: pgmap v4962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:33 smithi059 ceph-mon[97450]: pgmap v4962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:34.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:34.023915+0000) 2022-09-08T00:13:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:33.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:33.938937+0000) 2022-09-08T00:13:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:35.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:35.024158+0000) 2022-09-08T00:13:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:34.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:34.939147+0000) 2022-09-08T00:13:35.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:13:35.178+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:13:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:35 smithi055 ceph-mon[97033]: pgmap v4963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:35 smithi059 ceph-mon[97450]: pgmap v4963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:36.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:36.024395+0000) 2022-09-08T00:13:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:35.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:35.939396+0000) 2022-09-08T00:13:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:37.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:37.024628+0000) 2022-09-08T00:13:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:36.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:36.939655+0000) 2022-09-08T00:13:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:37 smithi055 ceph-mon[97033]: pgmap v4964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:13:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:37 smithi059 ceph-mon[97450]: pgmap v4964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:13:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:38.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:38.024799+0000) 2022-09-08T00:13:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:37.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:37.939881+0000) 2022-09-08T00:13:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:39.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:39.024976+0000) 2022-09-08T00:13:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:38.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:38.940049+0000) 2022-09-08T00:13:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:39 smithi055 ceph-mon[97033]: pgmap v4965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:39 smithi059 ceph-mon[97450]: pgmap v4965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:40.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:40.025199+0000) 2022-09-08T00:13:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:39.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:39.940263+0000) 2022-09-08T00:13:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:41.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:41.025427+0000) 2022-09-08T00:13:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:40.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:40.940460+0000) 2022-09-08T00:13:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:41 smithi059 ceph-mon[97450]: pgmap v4966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:13:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:41 smithi055 ceph-mon[97033]: pgmap v4966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:13:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:42.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:42.025613+0000) 2022-09-08T00:13:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:41.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:41.940718+0000) 2022-09-08T00:13:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:43.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:43.025821+0000) 2022-09-08T00:13:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:42.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:42.940961+0000) 2022-09-08T00:13:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:43 smithi055 ceph-mon[97033]: pgmap v4967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:43 smithi059 ceph-mon[97450]: pgmap v4967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:43.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:43.941182+0000) 2022-09-08T00:13:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:44.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:44.026059+0000) 2022-09-08T00:13:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:44.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:44.941425+0000) 2022-09-08T00:13:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:45.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:45.026253+0000) 2022-09-08T00:13:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:45 smithi055 ceph-mon[97033]: pgmap v4968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:45 smithi059 ceph-mon[97450]: pgmap v4968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:13:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:45.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:45.941710+0000) 2022-09-08T00:13:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:46.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:46.026451+0000) 2022-09-08T00:13:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:47.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:47.026716+0000) 2022-09-08T00:13:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:46.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:46.941928+0000) 2022-09-08T00:13:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:47 smithi055 ceph-mon[97033]: pgmap v4969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:13:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:47 smithi055 ceph-mon[97033]: pgmap v4970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:13:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:47 smithi055 ceph-mon[97033]: pgmap v4971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:13:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:13:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:47 smithi059 ceph-mon[97450]: pgmap v4969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:13:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:47 smithi059 ceph-mon[97450]: pgmap v4970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:13:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:47 smithi059 ceph-mon[97450]: pgmap v4971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:13:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:13:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:48.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:48.026974+0000) 2022-09-08T00:13:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:47.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:47.942148+0000) 2022-09-08T00:13:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:49.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:49.027168+0000) 2022-09-08T00:13:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:48.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:48.942386+0000) 2022-09-08T00:13:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:49 smithi055 ceph-mon[97033]: pgmap v4972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:13:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:49 smithi059 ceph-mon[97450]: pgmap v4972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:13:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:50.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:50.027379+0000) 2022-09-08T00:13:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:49.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:49.942601+0000) 2022-09-08T00:13:50.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:13:50.189+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:13:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:51.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:51.027629+0000) 2022-09-08T00:13:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:50.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:50.942831+0000) 2022-09-08T00:13:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:51 smithi059 ceph-mon[97450]: pgmap v4973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:13:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:51 smithi055 ceph-mon[97033]: pgmap v4973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:13:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:52.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:52.027878+0000) 2022-09-08T00:13:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:51.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:51.943039+0000) 2022-09-08T00:13:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:53.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:53.028132+0000) 2022-09-08T00:13:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:52.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:52.943259+0000) 2022-09-08T00:13:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:53 smithi055 ceph-mon[97033]: pgmap v4974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:13:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:53 smithi059 ceph-mon[97450]: pgmap v4974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:13:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:54.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:54.028278+0000) 2022-09-08T00:13:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:53.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:53.943444+0000) 2022-09-08T00:13:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:55.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:55.028508+0000) 2022-09-08T00:13:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:54.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:54.943680+0000) 2022-09-08T00:13:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:55 smithi055 ceph-mon[97033]: pgmap v4975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:13:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:55 smithi059 ceph-mon[97450]: pgmap v4975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:13:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:56.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:56.028714+0000) 2022-09-08T00:13:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:55.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:55.943927+0000) 2022-09-08T00:13:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:57.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:57.028857+0000) 2022-09-08T00:13:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:56.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:56.944132+0000) 2022-09-08T00:13:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:57 smithi055 ceph-mon[97033]: pgmap v4976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:13:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:57 smithi059 ceph-mon[97450]: pgmap v4976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:13:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:58.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:58.029005+0000) 2022-09-08T00:13:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:57.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:57.944330+0000) 2022-09-08T00:13:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:13:59.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:59.029159+0000) 2022-09-08T00:13:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:58.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:58.944576+0000) 2022-09-08T00:14:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:13:59 smithi055 ceph-mon[97033]: pgmap v4977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:13:59 smithi059 ceph-mon[97450]: pgmap v4977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:00.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:00.029396+0000) 2022-09-08T00:14:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:13:59.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:13:59.944803+0000) 2022-09-08T00:14:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:01.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:01.029585+0000) 2022-09-08T00:14:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:00.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:00.945058+0000) 2022-09-08T00:14:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:01 smithi059 ceph-mon[97450]: pgmap v4978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:14:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:01 smithi055 ceph-mon[97033]: pgmap v4978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:14:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:02.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:02.029801+0000) 2022-09-08T00:14:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:01.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:01.945291+0000) 2022-09-08T00:14:03.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:02.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:02.945461+0000) 2022-09-08T00:14:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:03.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:03.030003+0000) 2022-09-08T00:14:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:03 smithi055 ceph-mon[97033]: pgmap v4979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:03 smithi059 ceph-mon[97450]: pgmap v4979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:03.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:03.945712+0000) 2022-09-08T00:14:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:04.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:04.030187+0000) 2022-09-08T00:14:05.201 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:05.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:05.030412+0000) 2022-09-08T00:14:05.202 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:04.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:04.945911+0000) 2022-09-08T00:14:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:14:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:14:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:14:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:14:05.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:14:05.200+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:14:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:05 smithi055 ceph-mon[97033]: pgmap v4980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:14:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:05 smithi059 ceph-mon[97450]: pgmap v4980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:14:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:06.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:06.030664+0000) 2022-09-08T00:14:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:05.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:05.946119+0000) 2022-09-08T00:14:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:07.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:07.030901+0000) 2022-09-08T00:14:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:06.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:06.946371+0000) 2022-09-08T00:14:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:07 smithi055 ceph-mon[97033]: pgmap v4981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:14:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:07 smithi059 ceph-mon[97450]: pgmap v4981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:14:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:07.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:07.946579+0000) 2022-09-08T00:14:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:08.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:08.031150+0000) 2022-09-08T00:14:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:09.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:09.031360+0000) 2022-09-08T00:14:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:08.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:08.946807+0000) 2022-09-08T00:14:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:09 smithi055 ceph-mon[97033]: pgmap v4982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:14:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:09 smithi055 ceph-mon[97033]: osdmap e397: 12 total, 11 up, 11 in 2022-09-08T00:14:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:09 smithi059 ceph-mon[97450]: pgmap v4982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:14:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:09 smithi059 ceph-mon[97450]: osdmap e397: 12 total, 11 up, 11 in 2022-09-08T00:14:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:10.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:10.031533+0000) 2022-09-08T00:14:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:09.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:09.947026+0000) 2022-09-08T00:14:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:11.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:11.031749+0000) 2022-09-08T00:14:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:10.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:10.947242+0000) 2022-09-08T00:14:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:11 smithi059 ceph-mon[97450]: pgmap v4984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:14:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:11 smithi055 ceph-mon[97033]: pgmap v4984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:14:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:12.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:12.031986+0000) 2022-09-08T00:14:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:11.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:11.947469+0000) 2022-09-08T00:14:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:13.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:13.032175+0000) 2022-09-08T00:14:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:12.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:12.947751+0000) 2022-09-08T00:14:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:13 smithi055 ceph-mon[97033]: pgmap v4985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:14:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:13 smithi059 ceph-mon[97450]: pgmap v4985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:14:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:14.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:14.032381+0000) 2022-09-08T00:14:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:13.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:13.947909+0000) 2022-09-08T00:14:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:15.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:15.032666+0000) 2022-09-08T00:14:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:14.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:14.948077+0000) 2022-09-08T00:14:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:15 smithi055 ceph-mon[97033]: pgmap v4986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:14:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:15 smithi055 ceph-mon[97033]: osdmap e398: 12 total, 11 up, 11 in 2022-09-08T00:14:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:15 smithi059 ceph-mon[97450]: pgmap v4986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:14:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:15 smithi059 ceph-mon[97450]: osdmap e398: 12 total, 11 up, 11 in 2022-09-08T00:14:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:16.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:16.032900+0000) 2022-09-08T00:14:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:15.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:15.948332+0000) 2022-09-08T00:14:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:16.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:16.948586+0000) 2022-09-08T00:14:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:17.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:17.033124+0000) 2022-09-08T00:14:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:17 smithi055 ceph-mon[97033]: pgmap v4988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:14:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:17 smithi059 ceph-mon[97450]: pgmap v4988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:14:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:18.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:18.033320+0000) 2022-09-08T00:14:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:17.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:17.948763+0000) 2022-09-08T00:14:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:19.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:19.033521+0000) 2022-09-08T00:14:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:18.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:18.949004+0000) 2022-09-08T00:14:20.213 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:20.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:20.033754+0000) 2022-09-08T00:14:20.214 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:19.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:19.949229+0000) 2022-09-08T00:14:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:19 smithi055 ceph-mon[97033]: pgmap v4989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:14:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:19 smithi059 ceph-mon[97450]: pgmap v4989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:14:20.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:14:20.211+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:14:21.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:20.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:20.949456+0000) 2022-09-08T00:14:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:21.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:21.033943+0000) 2022-09-08T00:14:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:21 smithi059 ceph-mon[97450]: pgmap v4990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:14:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:21 smithi055 ceph-mon[97033]: pgmap v4990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:14:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:22.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:22.034131+0000) 2022-09-08T00:14:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:21.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:21.949720+0000) 2022-09-08T00:14:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:23.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:23.034372+0000) 2022-09-08T00:14:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:22.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:22.949977+0000) 2022-09-08T00:14:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:23 smithi055 ceph-mon[97033]: pgmap v4991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:14:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:23 smithi059 ceph-mon[97450]: pgmap v4991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:14:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:24.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:24.034598+0000) 2022-09-08T00:14:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:23.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:23.950221+0000) 2022-09-08T00:14:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:25.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:25.034767+0000) 2022-09-08T00:14:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:24.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:24.950419+0000) 2022-09-08T00:14:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:25 smithi055 ceph-mon[97033]: pgmap v4992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:14:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:25 smithi059 ceph-mon[97450]: pgmap v4992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:14:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:26.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:26.034955+0000) 2022-09-08T00:14:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:25.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:25.950630+0000) 2022-09-08T00:14:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:27.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:27.035163+0000) 2022-09-08T00:14:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:26.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:26.950875+0000) 2022-09-08T00:14:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:27 smithi055 ceph-mon[97033]: pgmap v4993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:14:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:27 smithi059 ceph-mon[97450]: pgmap v4993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:14:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:28.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:28.035353+0000) 2022-09-08T00:14:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:27.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:27.951101+0000) 2022-09-08T00:14:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:29.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:29.035557+0000) 2022-09-08T00:14:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:28.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:28.951338+0000) 2022-09-08T00:14:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:29 smithi055 ceph-mon[97033]: pgmap v4994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:29 smithi059 ceph-mon[97450]: pgmap v4994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:29.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:29.951498+0000) 2022-09-08T00:14:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:30.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:30.035760+0000) 2022-09-08T00:14:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:31.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:31.035964+0000) 2022-09-08T00:14:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:30.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:30.951718+0000) 2022-09-08T00:14:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:31 smithi059 ceph-mon[97450]: pgmap v4995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:14:31.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:31 smithi055 ceph-mon[97033]: pgmap v4995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:14:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:32.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:32.036251+0000) 2022-09-08T00:14:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:31.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:31.951937+0000) 2022-09-08T00:14:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:33.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:33.036497+0000) 2022-09-08T00:14:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:32.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:32.952164+0000) 2022-09-08T00:14:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:33 smithi055 ceph-mon[97033]: pgmap v4996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:33 smithi059 ceph-mon[97450]: pgmap v4996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:34.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:34.036732+0000) 2022-09-08T00:14:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:33.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:33.952344+0000) 2022-09-08T00:14:35.223 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:34.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:34.952471+0000) 2022-09-08T00:14:35.224 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:35.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:35.036962+0000) 2022-09-08T00:14:35.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:14:35.221+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:14:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:35 smithi055 ceph-mon[97033]: pgmap v4997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:14:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:35 smithi059 ceph-mon[97450]: pgmap v4997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:14:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:36.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:36.037187+0000) 2022-09-08T00:14:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:35.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:35.952717+0000) 2022-09-08T00:14:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:37.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:37.037430+0000) 2022-09-08T00:14:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:36.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:36.952930+0000) 2022-09-08T00:14:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:37 smithi055 ceph-mon[97033]: pgmap v4998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:37 smithi059 ceph-mon[97450]: pgmap v4998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:37.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:37.953150+0000) 2022-09-08T00:14:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:38.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:38.037696+0000) 2022-09-08T00:14:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:39.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:39.037975+0000) 2022-09-08T00:14:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:38.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:38.953363+0000) 2022-09-08T00:14:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:39 smithi055 ceph-mon[97033]: pgmap v4999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:39 smithi059 ceph-mon[97450]: pgmap v4999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:40.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:40.038150+0000) 2022-09-08T00:14:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:39.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:39.953592+0000) 2022-09-08T00:14:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:41.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:41.038343+0000) 2022-09-08T00:14:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:40.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:40.953804+0000) 2022-09-08T00:14:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:41 smithi059 ceph-mon[97450]: pgmap v5000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:14:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:41 smithi055 ceph-mon[97033]: pgmap v5000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:14:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:42.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:42.038572+0000) 2022-09-08T00:14:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:41.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:41.954024+0000) 2022-09-08T00:14:43.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:42.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:42.954251+0000) 2022-09-08T00:14:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:43.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:43.038798+0000) 2022-09-08T00:14:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:43 smithi055 ceph-mon[97033]: pgmap v5001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:43 smithi059 ceph-mon[97450]: pgmap v5001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:44.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:44.038980+0000) 2022-09-08T00:14:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:43.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:43.954473+0000) 2022-09-08T00:14:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:44.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:44.954726+0000) 2022-09-08T00:14:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:45.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:45.039242+0000) 2022-09-08T00:14:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:45 smithi055 ceph-mon[97033]: pgmap v5002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:14:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:45 smithi059 ceph-mon[97450]: pgmap v5002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:14:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:46.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:46.039418+0000) 2022-09-08T00:14:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:45.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:45.954951+0000) 2022-09-08T00:14:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:47.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:47.039584+0000) 2022-09-08T00:14:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:46.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:46.955189+0000) 2022-09-08T00:14:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:47 smithi055 ceph-mon[97033]: pgmap v5003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:14:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:14:48.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:14:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:47 smithi059 ceph-mon[97450]: pgmap v5003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:14:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:14:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:14:48.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:48.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:48.039746+0000) 2022-09-08T00:14:48.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:47.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:47.955371+0000) 2022-09-08T00:14:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:14:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:14:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:14:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:14:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:49.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:49.039867+0000) 2022-09-08T00:14:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:48.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:48.955614+0000) 2022-09-08T00:14:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:14:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:14:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:14:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:14:50.239 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:50.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:50.040039+0000) 2022-09-08T00:14:50.240 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:49.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:49.955905+0000) 2022-09-08T00:14:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:50 smithi059 ceph-mon[97450]: pgmap v5004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:50 smithi055 ceph-mon[97033]: pgmap v5004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:50.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:14:50.237+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:14:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:51.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:51.040167+0000) 2022-09-08T00:14:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:50.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:50.956180+0000) 2022-09-08T00:14:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:51 smithi059 ceph-mon[97450]: pgmap v5005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:14:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:51 smithi055 ceph-mon[97033]: pgmap v5005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:14:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:52.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:52.040369+0000) 2022-09-08T00:14:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:51.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:51.956387+0000) 2022-09-08T00:14:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:53.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:53.040612+0000) 2022-09-08T00:14:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:52.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:52.956616+0000) 2022-09-08T00:14:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:53 smithi055 ceph-mon[97033]: pgmap v5006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:53 smithi059 ceph-mon[97450]: pgmap v5006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:54.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:54.040818+0000) 2022-09-08T00:14:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:53.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:53.956750+0000) 2022-09-08T00:14:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:55.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:55.040954+0000) 2022-09-08T00:14:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:54.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:54.956983+0000) 2022-09-08T00:14:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:55 smithi055 ceph-mon[97033]: pgmap v5007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:14:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:55 smithi059 ceph-mon[97450]: pgmap v5007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:14:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:56.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:56.041156+0000) 2022-09-08T00:14:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:55.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:55.957220+0000) 2022-09-08T00:14:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:57.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:57.041366+0000) 2022-09-08T00:14:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:56.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:56.957468+0000) 2022-09-08T00:14:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:57 smithi055 ceph-mon[97033]: pgmap v5008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:57 smithi059 ceph-mon[97450]: pgmap v5008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:14:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:58.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:58.041570+0000) 2022-09-08T00:14:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:57.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:57.957689+0000) 2022-09-08T00:14:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:14:59.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:59.041796+0000) 2022-09-08T00:14:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:58.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:58.957828+0000) 2022-09-08T00:15:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:14:59 smithi059 ceph-mon[97450]: pgmap v5009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:14:59.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:14:59.958075+0000) 2022-09-08T00:15:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:00.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:00.041994+0000) 2022-09-08T00:15:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:14:59 smithi055 ceph-mon[97033]: pgmap v5009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:00.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:00.958325+0000) 2022-09-08T00:15:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:01.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:01.042210+0000) 2022-09-08T00:15:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:01 smithi059 ceph-mon[97450]: pgmap v5010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:15:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:01 smithi055 ceph-mon[97033]: pgmap v5010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:15:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:02.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:02.042503+0000) 2022-09-08T00:15:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:01.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:01.958567+0000) 2022-09-08T00:15:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:02.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:02.958742+0000) 2022-09-08T00:15:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:03.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:03.042756+0000) 2022-09-08T00:15:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:03 smithi055 ceph-mon[97033]: pgmap v5011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:03 smithi059 ceph-mon[97450]: pgmap v5011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:04.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:04.042946+0000) 2022-09-08T00:15:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:03.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:03.958880+0000) 2022-09-08T00:15:05.250 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:05.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:05.043172+0000) 2022-09-08T00:15:05.251 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:04.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:04.959064+0000) 2022-09-08T00:15:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:15:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:15:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:15:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:15:05.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:15:05.246+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:15:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:05 smithi055 ceph-mon[97033]: pgmap v5012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:15:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:05 smithi059 ceph-mon[97450]: pgmap v5012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:15:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:05.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:05.959201+0000) 2022-09-08T00:15:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:06.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:06.043311+0000) 2022-09-08T00:15:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:07.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:07.043523+0000) 2022-09-08T00:15:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:06.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:06.959451+0000) 2022-09-08T00:15:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:07 smithi059 ceph-mon[97450]: pgmap v5013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:15:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:07.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:07.959679+0000) 2022-09-08T00:15:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:08.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:08.043767+0000) 2022-09-08T00:15:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:07 smithi055 ceph-mon[97033]: pgmap v5013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:15:09.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:08.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:08.959861+0000) 2022-09-08T00:15:09.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:09.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:09.043932+0000) 2022-09-08T00:15:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:09 smithi059 ceph-mon[97450]: pgmap v5014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:15:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:09 smithi059 ceph-mon[97450]: osdmap e399: 12 total, 11 up, 11 in 2022-09-08T00:15:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:10.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:10.044151+0000) 2022-09-08T00:15:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:09.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:09.960086+0000) 2022-09-08T00:15:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:09 smithi055 ceph-mon[97033]: pgmap v5014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:15:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:09 smithi055 ceph-mon[97033]: osdmap e399: 12 total, 11 up, 11 in 2022-09-08T00:15:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:11.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:11.044402+0000) 2022-09-08T00:15:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:10.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:10.960368+0000) 2022-09-08T00:15:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:11 smithi059 ceph-mon[97450]: pgmap v5016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:15:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:11 smithi055 ceph-mon[97033]: pgmap v5016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:15:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:12.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:12.044658+0000) 2022-09-08T00:15:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:11.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:11.960591+0000) 2022-09-08T00:15:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:13.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:13.044916+0000) 2022-09-08T00:15:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:12.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:12.960797+0000) 2022-09-08T00:15:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:13 smithi055 ceph-mon[97033]: pgmap v5017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:15:14.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:13 smithi059 ceph-mon[97450]: pgmap v5017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:15:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:14.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:14.045090+0000) 2022-09-08T00:15:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:13.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:13.960974+0000) 2022-09-08T00:15:15.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:14.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:14.961248+0000) 2022-09-08T00:15:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:15.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:15.045298+0000) 2022-09-08T00:15:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:15 smithi055 ceph-mon[97033]: pgmap v5018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:15:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:15 smithi055 ceph-mon[97033]: osdmap e400: 12 total, 11 up, 11 in 2022-09-08T00:15:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:15 smithi059 ceph-mon[97450]: pgmap v5018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:15:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:15 smithi059 ceph-mon[97450]: osdmap e400: 12 total, 11 up, 11 in 2022-09-08T00:15:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:15.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:15.961510+0000) 2022-09-08T00:15:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:16.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:16.045524+0000) 2022-09-08T00:15:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:16.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:16.961756+0000) 2022-09-08T00:15:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:17.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:17.045750+0000) 2022-09-08T00:15:18.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:17.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:17.961990+0000) 2022-09-08T00:15:18.413 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:18.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:18.045874+0000) 2022-09-08T00:15:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:18 smithi055 ceph-mon[97033]: pgmap v5020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:15:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:18 smithi055 ceph-mon[97033]: pgmap v5021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T00:15:18.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:18 smithi055 ceph-mon[97033]: pgmap v5022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T00:15:18.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:15:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:18 smithi059 ceph-mon[97450]: pgmap v5020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:15:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:18 smithi059 ceph-mon[97450]: pgmap v5021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T00:15:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:18 smithi059 ceph-mon[97450]: pgmap v5022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T00:15:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:15:19.409 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:19.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:19.046066+0000) 2022-09-08T00:15:19.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:18.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:18.962257+0000) 2022-09-08T00:15:20.261 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:20.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:20.046299+0000) 2022-09-08T00:15:20.261 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:19.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:19.962457+0000) 2022-09-08T00:15:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:20 smithi055 ceph-mon[97033]: pgmap v5023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T00:15:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:20 smithi059 ceph-mon[97450]: pgmap v5023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T00:15:20.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:15:20.258+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:15:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:21.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:21.046537+0000) 2022-09-08T00:15:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:20.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:20.962710+0000) 2022-09-08T00:15:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:21 smithi059 ceph-mon[97450]: pgmap v5024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:15:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:21 smithi055 ceph-mon[97033]: pgmap v5024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:15:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:22.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:22.046783+0000) 2022-09-08T00:15:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:21.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:21.962914+0000) 2022-09-08T00:15:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:23.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:23.046920+0000) 2022-09-08T00:15:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:22.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:22.963165+0000) 2022-09-08T00:15:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:24.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:24.047126+0000) 2022-09-08T00:15:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:23.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:23.963368+0000) 2022-09-08T00:15:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:24 smithi055 ceph-mon[97033]: pgmap v5025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 3 op/s 2022-09-08T00:15:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:24 smithi059 ceph-mon[97450]: pgmap v5025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 3 op/s 2022-09-08T00:15:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:25.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:25.047240+0000) 2022-09-08T00:15:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:24.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:24.963522+0000) 2022-09-08T00:15:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:26.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:26.047482+0000) 2022-09-08T00:15:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:25.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:25.963747+0000) 2022-09-08T00:15:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:26 smithi055 ceph-mon[97033]: pgmap v5026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:15:26.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:26 smithi059 ceph-mon[97450]: pgmap v5026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:15:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:27.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:27.047780+0000) 2022-09-08T00:15:27.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:26.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:26.963977+0000) 2022-09-08T00:15:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:27.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:27.964194+0000) 2022-09-08T00:15:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:28.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:28.048011+0000) 2022-09-08T00:15:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:28 smithi055 ceph-mon[97033]: pgmap v5027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:15:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:28 smithi059 ceph-mon[97450]: pgmap v5027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:15:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:29.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:29.048222+0000) 2022-09-08T00:15:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:28.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:28.964399+0000) 2022-09-08T00:15:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:30.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:30.048454+0000) 2022-09-08T00:15:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:29.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:29.964606+0000) 2022-09-08T00:15:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:30 smithi055 ceph-mon[97033]: pgmap v5028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:30 smithi059 ceph-mon[97450]: pgmap v5028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:31.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:31.048717+0000) 2022-09-08T00:15:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:30.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:30.964881+0000) 2022-09-08T00:15:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:31 smithi059 ceph-mon[97450]: pgmap v5029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:15:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:31 smithi055 ceph-mon[97033]: pgmap v5029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:15:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:32.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:32.048971+0000) 2022-09-08T00:15:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:31.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:31.965040+0000) 2022-09-08T00:15:33.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:33.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:33.049175+0000) 2022-09-08T00:15:33.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:32.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:32.965238+0000) 2022-09-08T00:15:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:34.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:34.049423+0000) 2022-09-08T00:15:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:33.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:33.965436+0000) 2022-09-08T00:15:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:34 smithi055 ceph-mon[97033]: pgmap v5030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:34 smithi059 ceph-mon[97450]: pgmap v5030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:35.270 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:35.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:35.049664+0000) 2022-09-08T00:15:35.271 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:34.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:34.965697+0000) 2022-09-08T00:15:35.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:15:35.268+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:15:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:35.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:35.965929+0000) 2022-09-08T00:15:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:36.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:36.049943+0000) 2022-09-08T00:15:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:36 smithi055 ceph-mon[97033]: pgmap v5031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:15:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:36 smithi059 ceph-mon[97450]: pgmap v5031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:15:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:36.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:36.966204+0000) 2022-09-08T00:15:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:37.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:37.050196+0000) 2022-09-08T00:15:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:38.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:38.050445+0000) 2022-09-08T00:15:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:37.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:37.966451+0000) 2022-09-08T00:15:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:38 smithi055 ceph-mon[97033]: pgmap v5032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:38.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:38 smithi059 ceph-mon[97450]: pgmap v5032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:39.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:39.050722+0000) 2022-09-08T00:15:39.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:38.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:38.966702+0000) 2022-09-08T00:15:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:40.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:40.050927+0000) 2022-09-08T00:15:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:39.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:39.966912+0000) 2022-09-08T00:15:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:40 smithi055 ceph-mon[97033]: pgmap v5033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:40.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:40 smithi059 ceph-mon[97450]: pgmap v5033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:41.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:41.051150+0000) 2022-09-08T00:15:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:40.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:40.967052+0000) 2022-09-08T00:15:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:41 smithi059 ceph-mon[97450]: pgmap v5034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:15:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:41 smithi055 ceph-mon[97033]: pgmap v5034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:15:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:42.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:42.051362+0000) 2022-09-08T00:15:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:41.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:41.967256+0000) 2022-09-08T00:15:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:42.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:42.967484+0000) 2022-09-08T00:15:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:43.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:43.051570+0000) 2022-09-08T00:15:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:43.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:43.967736+0000) 2022-09-08T00:15:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:44.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:44.051746+0000) 2022-09-08T00:15:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:44 smithi055 ceph-mon[97033]: pgmap v5035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:44 smithi059 ceph-mon[97450]: pgmap v5035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:45.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:44.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:44.967956+0000) 2022-09-08T00:15:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:45.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:45.051988+0000) 2022-09-08T00:15:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:46.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:46.052249+0000) 2022-09-08T00:15:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:45.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:45.968164+0000) 2022-09-08T00:15:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:46 smithi055 ceph-mon[97033]: pgmap v5036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:15:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:46 smithi059 ceph-mon[97450]: pgmap v5036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:15:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:47.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:47.052462+0000) 2022-09-08T00:15:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:46.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:46.968408+0000) 2022-09-08T00:15:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:48.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:48.052702+0000) 2022-09-08T00:15:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:47.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:47.968670+0000) 2022-09-08T00:15:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:48 smithi059 ceph-mon[97450]: pgmap v5037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:48 smithi055 ceph-mon[97033]: pgmap v5037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:49.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:49.052905+0000) 2022-09-08T00:15:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:48.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:48.968929+0000) 2022-09-08T00:15:50.276 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:50.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:50.053064+0000) 2022-09-08T00:15:50.277 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:49.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:49.969130+0000) 2022-09-08T00:15:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:50 smithi055 ceph-mon[97033]: pgmap v5038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:50 smithi059 ceph-mon[97450]: pgmap v5038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:50.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:15:50.274+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:15:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:51.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:51.053282+0000) 2022-09-08T00:15:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:50.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:50.969369+0000) 2022-09-08T00:15:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:51 smithi059 ceph-mon[97450]: pgmap v5039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:15:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:51 smithi055 ceph-mon[97033]: pgmap v5039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:15:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:52.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:52.053584+0000) 2022-09-08T00:15:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:51.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:51.969589+0000) 2022-09-08T00:15:53.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:52.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:52.969839+0000) 2022-09-08T00:15:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:53.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:53.053867+0000) 2022-09-08T00:15:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:54.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:54.054116+0000) 2022-09-08T00:15:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:53.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:53.970011+0000) 2022-09-08T00:15:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:54 smithi055 ceph-mon[97033]: pgmap v5040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:54 smithi059 ceph-mon[97450]: pgmap v5040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:55.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:55.054266+0000) 2022-09-08T00:15:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:54.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:54.970291+0000) 2022-09-08T00:15:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:56.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:56.054502+0000) 2022-09-08T00:15:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:55.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:55.970546+0000) 2022-09-08T00:15:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:56 smithi055 ceph-mon[97033]: pgmap v5041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:15:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:56 smithi059 ceph-mon[97450]: pgmap v5041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:15:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:57.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:57.054694+0000) 2022-09-08T00:15:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:56.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:56.970771+0000) 2022-09-08T00:15:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:57.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:57.971026+0000) 2022-09-08T00:15:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:58.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:58.054891+0000) 2022-09-08T00:15:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:15:58 smithi059 ceph-mon[97450]: pgmap v5042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:15:58 smithi055 ceph-mon[97033]: pgmap v5042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:15:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:15:59.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:59.055091+0000) 2022-09-08T00:15:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:58.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:58.971274+0000) 2022-09-08T00:16:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:00.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:00.055294+0000) 2022-09-08T00:16:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:15:59.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:15:59.971559+0000) 2022-09-08T00:16:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:00 smithi059 ceph-mon[97450]: pgmap v5043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:00 smithi055 ceph-mon[97033]: pgmap v5043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:01.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:01.055481+0000) 2022-09-08T00:16:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:00.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:00.971800+0000) 2022-09-08T00:16:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:01 smithi055 ceph-mon[97033]: pgmap v5044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:16:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:01 smithi059 ceph-mon[97450]: pgmap v5044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:16:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:02.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:02.055722+0000) 2022-09-08T00:16:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:01.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:01.972029+0000) 2022-09-08T00:16:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:02.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:02.972299+0000) 2022-09-08T00:16:03.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:03.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:03.055902+0000) 2022-09-08T00:16:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:04.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:04.056116+0000) 2022-09-08T00:16:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:03.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:03.972510+0000) 2022-09-08T00:16:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:04 smithi055 ceph-mon[97033]: pgmap v5045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:04 smithi059 ceph-mon[97450]: pgmap v5045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:05.286 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:05.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:05.056353+0000) 2022-09-08T00:16:05.287 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:04.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:04.972741+0000) 2022-09-08T00:16:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:16:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:16:05.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:16:05.284+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:16:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:16:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:16:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:06.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:06.056566+0000) 2022-09-08T00:16:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:05.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:05.972963+0000) 2022-09-08T00:16:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:06 smithi059 ceph-mon[97450]: pgmap v5046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:16:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:06 smithi055 ceph-mon[97033]: pgmap v5046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:16:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:07.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:07.056707+0000) 2022-09-08T00:16:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:06.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:06.973205+0000) 2022-09-08T00:16:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:08.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:08.056885+0000) 2022-09-08T00:16:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:07.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:07.973456+0000) 2022-09-08T00:16:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:08 smithi059 ceph-mon[97450]: pgmap v5047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:16:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:08 smithi055 ceph-mon[97033]: pgmap v5047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:16:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:09.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:09.057085+0000) 2022-09-08T00:16:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:08.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:08.973728+0000) 2022-09-08T00:16:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:10.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:10.057303+0000) 2022-09-08T00:16:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:09.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:09.973891+0000) 2022-09-08T00:16:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:10 smithi059 ceph-mon[97450]: pgmap v5048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:16:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:10 smithi059 ceph-mon[97450]: osdmap e401: 12 total, 11 up, 11 in 2022-09-08T00:16:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:10 smithi055 ceph-mon[97033]: pgmap v5048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:16:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:10 smithi055 ceph-mon[97033]: osdmap e401: 12 total, 11 up, 11 in 2022-09-08T00:16:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:11.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:11.057518+0000) 2022-09-08T00:16:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:10.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:10.974104+0000) 2022-09-08T00:16:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:11 smithi059 ceph-mon[97450]: pgmap v5050: 129 pgs: 18 active+clean+snaptrim_wait, 111 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:16:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:11 smithi055 ceph-mon[97033]: pgmap v5050: 129 pgs: 18 active+clean+snaptrim_wait, 111 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:16:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:12.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:12.057744+0000) 2022-09-08T00:16:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:11.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:11.974310+0000) 2022-09-08T00:16:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:13.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:13.057959+0000) 2022-09-08T00:16:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:12.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:12.974581+0000) 2022-09-08T00:16:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:14.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:14.058128+0000) 2022-09-08T00:16:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:13.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:13.974799+0000) 2022-09-08T00:16:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:14 smithi059 ceph-mon[97450]: pgmap v5051: 129 pgs: 18 active+clean+snaptrim_wait, 111 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:16:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:14 smithi055 ceph-mon[97033]: pgmap v5051: 129 pgs: 18 active+clean+snaptrim_wait, 111 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:16:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:15.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:15.058319+0000) 2022-09-08T00:16:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:14.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:14.974980+0000) 2022-09-08T00:16:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:16.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:16.058531+0000) 2022-09-08T00:16:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:15.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:15.975189+0000) 2022-09-08T00:16:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:16 smithi059 ceph-mon[97450]: pgmap v5052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:16:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:16 smithi059 ceph-mon[97450]: osdmap e402: 12 total, 11 up, 11 in 2022-09-08T00:16:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:16 smithi055 ceph-mon[97033]: pgmap v5052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:16:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:16 smithi055 ceph-mon[97033]: osdmap e402: 12 total, 11 up, 11 in 2022-09-08T00:16:17.314 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:16.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:16.975335+0000) 2022-09-08T00:16:17.315 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:17.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:17.058762+0000) 2022-09-08T00:16:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:18.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:18.058989+0000) 2022-09-08T00:16:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:17.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:17.975549+0000) 2022-09-08T00:16:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:18 smithi059 ceph-mon[97450]: pgmap v5054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:16:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:16:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:16:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:16:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:18 smithi055 ceph-mon[97033]: pgmap v5054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:16:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:16:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:16:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:16:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:19.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:19.059203+0000) 2022-09-08T00:16:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:18.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:18.975764+0000) 2022-09-08T00:16:20.290 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:20.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:20.059401+0000) 2022-09-08T00:16:20.291 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:19.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:19.975955+0000) 2022-09-08T00:16:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:20 smithi059 ceph-mon[97450]: pgmap v5055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:16:20.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:16:20.288+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:16:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:20 smithi055 ceph-mon[97033]: pgmap v5055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:16:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:21.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:21.059619+0000) 2022-09-08T00:16:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:20.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:20.976097+0000) 2022-09-08T00:16:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:21 smithi055 ceph-mon[97033]: pgmap v5056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:16:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:21 smithi059 ceph-mon[97450]: pgmap v5056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:16:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:22.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:22.059851+0000) 2022-09-08T00:16:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:21.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:21.976316+0000) 2022-09-08T00:16:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:23.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:23.060026+0000) 2022-09-08T00:16:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:22.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:22.976497+0000) 2022-09-08T00:16:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:24.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:24.060209+0000) 2022-09-08T00:16:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:23.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:23.976735+0000) 2022-09-08T00:16:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:24 smithi059 ceph-mon[97450]: pgmap v5057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:16:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:24 smithi055 ceph-mon[97033]: pgmap v5057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:16:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:25.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:25.060329+0000) 2022-09-08T00:16:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:24.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:24.976987+0000) 2022-09-08T00:16:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:26.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:26.060554+0000) 2022-09-08T00:16:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:25.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:25.977163+0000) 2022-09-08T00:16:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:26 smithi059 ceph-mon[97450]: pgmap v5058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:16:26.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:26 smithi055 ceph-mon[97033]: pgmap v5058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:16:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:27.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:27.060785+0000) 2022-09-08T00:16:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:26.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:26.977349+0000) 2022-09-08T00:16:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:28.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:28.060994+0000) 2022-09-08T00:16:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:27.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:27.977650+0000) 2022-09-08T00:16:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:28 smithi059 ceph-mon[97450]: pgmap v5059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:16:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:28 smithi055 ceph-mon[97033]: pgmap v5059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:16:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:29.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:29.061267+0000) 2022-09-08T00:16:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:28.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:28.977838+0000) 2022-09-08T00:16:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:29.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:29.978049+0000) 2022-09-08T00:16:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:30.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:30.061513+0000) 2022-09-08T00:16:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:30 smithi059 ceph-mon[97450]: pgmap v5060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:30 smithi055 ceph-mon[97033]: pgmap v5060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:31.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:31.061732+0000) 2022-09-08T00:16:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:30.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:30.978268+0000) 2022-09-08T00:16:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:31 smithi055 ceph-mon[97033]: pgmap v5061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:16:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:31 smithi059 ceph-mon[97450]: pgmap v5061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:16:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:31.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:31.978437+0000) 2022-09-08T00:16:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:32.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:32.061944+0000) 2022-09-08T00:16:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:32.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:32.978715+0000) 2022-09-08T00:16:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:33.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:33.062144+0000) 2022-09-08T00:16:34.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:33.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:33.978955+0000) 2022-09-08T00:16:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:34.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:34.062333+0000) 2022-09-08T00:16:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:34 smithi059 ceph-mon[97450]: pgmap v5062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:34 smithi055 ceph-mon[97033]: pgmap v5062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:35.295 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:35.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:35.062573+0000) 2022-09-08T00:16:35.296 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:34.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:34.979192+0000) 2022-09-08T00:16:35.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:16:35.294+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:16:36.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:35.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:35.979418+0000) 2022-09-08T00:16:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:36.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:36.062786+0000) 2022-09-08T00:16:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:36 smithi059 ceph-mon[97450]: pgmap v5063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:16:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:36 smithi055 ceph-mon[97033]: pgmap v5063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:16:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:37.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:37.063022+0000) 2022-09-08T00:16:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:36.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:36.979609+0000) 2022-09-08T00:16:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:38.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:38.063208+0000) 2022-09-08T00:16:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:37.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:37.979755+0000) 2022-09-08T00:16:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:38 smithi059 ceph-mon[97450]: pgmap v5064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:38 smithi055 ceph-mon[97033]: pgmap v5064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:38.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:38.979960+0000) 2022-09-08T00:16:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:39.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:39.063366+0000) 2022-09-08T00:16:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:40.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:40.063558+0000) 2022-09-08T00:16:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:39.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:39.980189+0000) 2022-09-08T00:16:40.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:40 smithi059 ceph-mon[97450]: pgmap v5065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:40 smithi055 ceph-mon[97033]: pgmap v5065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:41.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:41.063711+0000) 2022-09-08T00:16:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:40.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:40.980433+0000) 2022-09-08T00:16:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:41 smithi055 ceph-mon[97033]: pgmap v5066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:16:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:41 smithi059 ceph-mon[97450]: pgmap v5066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:16:42.387 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:42.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:42.063849+0000) 2022-09-08T00:16:42.388 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:41.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:41.980716+0000) 2022-09-08T00:16:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:43.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:43.064014+0000) 2022-09-08T00:16:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:42.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:42.980892+0000) 2022-09-08T00:16:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:44.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:44.064232+0000) 2022-09-08T00:16:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:43.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:43.981057+0000) 2022-09-08T00:16:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:44 smithi059 ceph-mon[97450]: pgmap v5067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:44 smithi055 ceph-mon[97033]: pgmap v5067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:45.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:44.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:44.981310+0000) 2022-09-08T00:16:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:45.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:45.064464+0000) 2022-09-08T00:16:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:46.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:46.064725+0000) 2022-09-08T00:16:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:45.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:45.981586+0000) 2022-09-08T00:16:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:46 smithi059 ceph-mon[97450]: pgmap v5068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:16:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:46 smithi055 ceph-mon[97033]: pgmap v5068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:16:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:46.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:46.981859+0000) 2022-09-08T00:16:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:47.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:47.064938+0000) 2022-09-08T00:16:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:47.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:47.982163+0000) 2022-09-08T00:16:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:48.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:48.065179+0000) 2022-09-08T00:16:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:48 smithi055 ceph-mon[97033]: pgmap v5069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:48 smithi055 ceph-mon[97033]: pgmap v5070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:16:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:48 smithi055 ceph-mon[97033]: pgmap v5071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:16:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:16:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:48 smithi059 ceph-mon[97450]: pgmap v5069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:16:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:48 smithi059 ceph-mon[97450]: pgmap v5070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:16:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:48 smithi059 ceph-mon[97450]: pgmap v5071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:16:48.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:16:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:49.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:49.065397+0000) 2022-09-08T00:16:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:48.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:48.982469+0000) 2022-09-08T00:16:50.310 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:50.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:50.065669+0000) 2022-09-08T00:16:50.311 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:49.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:49.982727+0000) 2022-09-08T00:16:50.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:16:50.309+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:16:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:50 smithi055 ceph-mon[97033]: pgmap v5072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:16:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:50 smithi059 ceph-mon[97450]: pgmap v5072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:16:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:51.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:51.065873+0000) 2022-09-08T00:16:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:50.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:50.982970+0000) 2022-09-08T00:16:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:51 smithi055 ceph-mon[97033]: pgmap v5073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:16:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:51 smithi059 ceph-mon[97450]: pgmap v5073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:16:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:51.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:51.983157+0000) 2022-09-08T00:16:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:52.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:52.066082+0000) 2022-09-08T00:16:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:52.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:52.983414+0000) 2022-09-08T00:16:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:53.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:53.066332+0000) 2022-09-08T00:16:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:54.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:54.066496+0000) 2022-09-08T00:16:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:53.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:53.983690+0000) 2022-09-08T00:16:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:54 smithi059 ceph-mon[97450]: pgmap v5074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:16:54.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:54 smithi055 ceph-mon[97033]: pgmap v5074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:16:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:55.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:55.066742+0000) 2022-09-08T00:16:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:54.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:54.983978+0000) 2022-09-08T00:16:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:56.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:56.066983+0000) 2022-09-08T00:16:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:55.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:55.984184+0000) 2022-09-08T00:16:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:56 smithi059 ceph-mon[97450]: pgmap v5075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:16:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:56 smithi055 ceph-mon[97033]: pgmap v5075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:16:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:57.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:57.067169+0000) 2022-09-08T00:16:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:56.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:56.984450+0000) 2022-09-08T00:16:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:58.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:58.067359+0000) 2022-09-08T00:16:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:57.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:57.984754+0000) 2022-09-08T00:16:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:16:58 smithi059 ceph-mon[97450]: pgmap v5076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:16:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:16:58 smithi055 ceph-mon[97033]: pgmap v5076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:16:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:58.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:58.985028+0000) 2022-09-08T00:16:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:16:59.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:59.067528+0000) 2022-09-08T00:17:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:16:59.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:16:59.985310+0000) 2022-09-08T00:17:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:00.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:00.067725+0000) 2022-09-08T00:17:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:00 smithi059 ceph-mon[97450]: pgmap v5077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:00 smithi055 ceph-mon[97033]: pgmap v5077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:01.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:01.067861+0000) 2022-09-08T00:17:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:00.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:00.985476+0000) 2022-09-08T00:17:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:01 smithi055 ceph-mon[97033]: pgmap v5078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:17:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:01 smithi059 ceph-mon[97450]: pgmap v5078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:17:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:02.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:02.067993+0000) 2022-09-08T00:17:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:01.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:01.985723+0000) 2022-09-08T00:17:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:03.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:03.068230+0000) 2022-09-08T00:17:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:02.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:02.985994+0000) 2022-09-08T00:17:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:04.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:04.068512+0000) 2022-09-08T00:17:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:03.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:03.986243+0000) 2022-09-08T00:17:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:04 smithi059 ceph-mon[97450]: pgmap v5079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:04 smithi055 ceph-mon[97033]: pgmap v5079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:05.324 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:05.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:05.068734+0000) 2022-09-08T00:17:05.325 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:04.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:04.986419+0000) 2022-09-08T00:17:05.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:17:05.322+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:17:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:17:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:17:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:17:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:17:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:05.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:05.986592+0000) 2022-09-08T00:17:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:06.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:06.068949+0000) 2022-09-08T00:17:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:06 smithi059 ceph-mon[97450]: pgmap v5080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:17:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:06 smithi055 ceph-mon[97033]: pgmap v5080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:17:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:07.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:07.069147+0000) 2022-09-08T00:17:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:06.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:06.986737+0000) 2022-09-08T00:17:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:07 smithi059 ceph-mon[97450]: pgmap v5081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:17:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:07 smithi055 ceph-mon[97033]: pgmap v5081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:17:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:08.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:08.069344+0000) 2022-09-08T00:17:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:07.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:07.986974+0000) 2022-09-08T00:17:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:09.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:09.069562+0000) 2022-09-08T00:17:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:08.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:08.987121+0000) 2022-09-08T00:17:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:09 smithi059 ceph-mon[97450]: osdmap e403: 12 total, 11 up, 11 in 2022-09-08T00:17:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:09 smithi059 ceph-mon[97450]: pgmap v5083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:17:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:09 smithi055 ceph-mon[97033]: osdmap e403: 12 total, 11 up, 11 in 2022-09-08T00:17:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:09 smithi055 ceph-mon[97033]: pgmap v5083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:17:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:10.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:10.069751+0000) 2022-09-08T00:17:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:09.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:09.987246+0000) 2022-09-08T00:17:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:11.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:11.069992+0000) 2022-09-08T00:17:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:10.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:10.987434+0000) 2022-09-08T00:17:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:11 smithi055 ceph-mon[97033]: pgmap v5084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:17:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:11 smithi059 ceph-mon[97450]: pgmap v5084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:17:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:12.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:12.070192+0000) 2022-09-08T00:17:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:11.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:11.987732+0000) 2022-09-08T00:17:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:13.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:13.070405+0000) 2022-09-08T00:17:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:12.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:12.987982+0000) 2022-09-08T00:17:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:14.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:14.070624+0000) 2022-09-08T00:17:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:13.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:13.988171+0000) 2022-09-08T00:17:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:14 smithi059 ceph-mon[97450]: pgmap v5085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:17:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:14 smithi055 ceph-mon[97033]: pgmap v5085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:17:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:15.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:15.070863+0000) 2022-09-08T00:17:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:14.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:14.988411+0000) 2022-09-08T00:17:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:16.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:16.071017+0000) 2022-09-08T00:17:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:15.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:15.988732+0000) 2022-09-08T00:17:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:16 smithi059 ceph-mon[97450]: pgmap v5086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:17:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:16 smithi059 ceph-mon[97450]: osdmap e404: 12 total, 11 up, 11 in 2022-09-08T00:17:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:16 smithi055 ceph-mon[97033]: pgmap v5086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:17:16.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:16 smithi055 ceph-mon[97033]: osdmap e404: 12 total, 11 up, 11 in 2022-09-08T00:17:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:17.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:17.071184+0000) 2022-09-08T00:17:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:16.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:16.988883+0000) 2022-09-08T00:17:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:18.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:18.071374+0000) 2022-09-08T00:17:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:17.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:17.989010+0000) 2022-09-08T00:17:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:18 smithi059 ceph-mon[97450]: pgmap v5088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T00:17:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:18 smithi055 ceph-mon[97033]: pgmap v5088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T00:17:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:18.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:18.989206+0000) 2022-09-08T00:17:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:19.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:19.071546+0000) 2022-09-08T00:17:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:19 smithi059 ceph-mon[97450]: pgmap v5089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:17:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:19 smithi055 ceph-mon[97033]: pgmap v5089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:17:20.339 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:20.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:20.071761+0000) 2022-09-08T00:17:20.340 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:19.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:19.989481+0000) 2022-09-08T00:17:20.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:17:20.338+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:17:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:21.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:21.071947+0000) 2022-09-08T00:17:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:20.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:20.989734+0000) 2022-09-08T00:17:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:21 smithi055 ceph-mon[97033]: pgmap v5090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:17:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:21 smithi059 ceph-mon[97450]: pgmap v5090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:17:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:22.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:22.072131+0000) 2022-09-08T00:17:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:21.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:21.989916+0000) 2022-09-08T00:17:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:23.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:23.072265+0000) 2022-09-08T00:17:23.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:22.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:22.990076+0000) 2022-09-08T00:17:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:24.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:24.072491+0000) 2022-09-08T00:17:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:23.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:23.990263+0000) 2022-09-08T00:17:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:24 smithi059 ceph-mon[97450]: pgmap v5091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:17:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:24 smithi055 ceph-mon[97033]: pgmap v5091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:17:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:25.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:25.072664+0000) 2022-09-08T00:17:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:24.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:24.990420+0000) 2022-09-08T00:17:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:26.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:26.072887+0000) 2022-09-08T00:17:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:25.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:25.990612+0000) 2022-09-08T00:17:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:26 smithi059 ceph-mon[97450]: pgmap v5092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:17:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:26 smithi055 ceph-mon[97033]: pgmap v5092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:17:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:27.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:27.073093+0000) 2022-09-08T00:17:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:26.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:26.990877+0000) 2022-09-08T00:17:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:27 smithi055 ceph-mon[97033]: pgmap v5093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:17:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:27 smithi059 ceph-mon[97450]: pgmap v5093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:17:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:28.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:28.073328+0000) 2022-09-08T00:17:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:27.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:27.991054+0000) 2022-09-08T00:17:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:29.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:29.073532+0000) 2022-09-08T00:17:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:28.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:28.991280+0000) 2022-09-08T00:17:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:30.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:30.073702+0000) 2022-09-08T00:17:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:29.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:29.991507+0000) 2022-09-08T00:17:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:30 smithi059 ceph-mon[97450]: pgmap v5094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:30 smithi055 ceph-mon[97033]: pgmap v5094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:31.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:31.073814+0000) 2022-09-08T00:17:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:30.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:30.991776+0000) 2022-09-08T00:17:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:31 smithi055 ceph-mon[97033]: pgmap v5095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:17:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:31 smithi059 ceph-mon[97450]: pgmap v5095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:17:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:32.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:32.073968+0000) 2022-09-08T00:17:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:31.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:31.992007+0000) 2022-09-08T00:17:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:33.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:33.074147+0000) 2022-09-08T00:17:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:32.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:32.992239+0000) 2022-09-08T00:17:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:34.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:34.074387+0000) 2022-09-08T00:17:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:33.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:33.992398+0000) 2022-09-08T00:17:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:34 smithi059 ceph-mon[97450]: pgmap v5096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:34.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:34 smithi055 ceph-mon[97033]: pgmap v5096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:35.353 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:35.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:35.074562+0000) 2022-09-08T00:17:35.354 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:34.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:34.992713+0000) 2022-09-08T00:17:35.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:17:35.352+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:17:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:35 smithi055 ceph-mon[97033]: pgmap v5097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:17:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:35 smithi059 ceph-mon[97450]: pgmap v5097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:17:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:36.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:36.074788+0000) 2022-09-08T00:17:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:35.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:35.992864+0000) 2022-09-08T00:17:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:37.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:37.075016+0000) 2022-09-08T00:17:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:36.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:36.993033+0000) 2022-09-08T00:17:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:38.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:38.075233+0000) 2022-09-08T00:17:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:37.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:37.993242+0000) 2022-09-08T00:17:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:38 smithi059 ceph-mon[97450]: pgmap v5098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:38 smithi055 ceph-mon[97033]: pgmap v5098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:39.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:39.075453+0000) 2022-09-08T00:17:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:38.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:38.993462+0000) 2022-09-08T00:17:39.831 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:39 smithi055 ceph-mon[97033]: pgmap v5099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:39 smithi059 ceph-mon[97450]: pgmap v5099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:40.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:40.075735+0000) 2022-09-08T00:17:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:39.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:39.993662+0000) 2022-09-08T00:17:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:41.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:41.075973+0000) 2022-09-08T00:17:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:40.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:40.993850+0000) 2022-09-08T00:17:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:41 smithi055 ceph-mon[97033]: pgmap v5100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:17:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:41 smithi059 ceph-mon[97450]: pgmap v5100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:17:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:42.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:42.076197+0000) 2022-09-08T00:17:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:41.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:41.993995+0000) 2022-09-08T00:17:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:43.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:43.076424+0000) 2022-09-08T00:17:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:42.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:42.994178+0000) 2022-09-08T00:17:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:44.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:44.076671+0000) 2022-09-08T00:17:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:43.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:43.994375+0000) 2022-09-08T00:17:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:44 smithi059 ceph-mon[97450]: pgmap v5101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:44.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:44 smithi055 ceph-mon[97033]: pgmap v5101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:45.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:45.076917+0000) 2022-09-08T00:17:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:44.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:44.994608+0000) 2022-09-08T00:17:45.992 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:45 smithi055 ceph-mon[97033]: pgmap v5102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:17:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:45 smithi059 ceph-mon[97450]: pgmap v5102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:17:46.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:46.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:46.077134+0000) 2022-09-08T00:17:46.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:45.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:45.994779+0000) 2022-09-08T00:17:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:47.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:47.077353+0000) 2022-09-08T00:17:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:46.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:46.995042+0000) 2022-09-08T00:17:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:48.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:48.077503+0000) 2022-09-08T00:17:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:47.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:47.995282+0000) 2022-09-08T00:17:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:48 smithi059 ceph-mon[97450]: pgmap v5103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:17:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:17:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:17:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:48 smithi055 ceph-mon[97033]: pgmap v5103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:48.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:17:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:17:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:17:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:49.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:49.077681+0000) 2022-09-08T00:17:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:48.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:48.995535+0000) 2022-09-08T00:17:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:49 smithi059 ceph-mon[97450]: pgmap v5104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:49 smithi055 ceph-mon[97033]: pgmap v5104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:50.361 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:50.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:50.077890+0000) 2022-09-08T00:17:50.361 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:49.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:49.995783+0000) 2022-09-08T00:17:50.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:17:50.358+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:17:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:51.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:51.078046+0000) 2022-09-08T00:17:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:50.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:50.996038+0000) 2022-09-08T00:17:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:51 smithi055 ceph-mon[97033]: pgmap v5105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:17:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:51 smithi059 ceph-mon[97450]: pgmap v5105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:17:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:52.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:52.078246+0000) 2022-09-08T00:17:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:51.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:51.996318+0000) 2022-09-08T00:17:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:53.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:53.078466+0000) 2022-09-08T00:17:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:52.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:52.996592+0000) 2022-09-08T00:17:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:53 smithi055 ceph-mon[97033]: pgmap v5106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:53 smithi059 ceph-mon[97450]: pgmap v5106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:54.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:54.078713+0000) 2022-09-08T00:17:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:53.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:53.996843+0000) 2022-09-08T00:17:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:55.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:55.078953+0000) 2022-09-08T00:17:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:54.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:54.997006+0000) 2022-09-08T00:17:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:55 smithi055 ceph-mon[97033]: pgmap v5107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:17:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:55 smithi059 ceph-mon[97450]: pgmap v5107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:17:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:56.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:56.079141+0000) 2022-09-08T00:17:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:55.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:55.997245+0000) 2022-09-08T00:17:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:56.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:56.997492+0000) 2022-09-08T00:17:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:57.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:57.079328+0000) 2022-09-08T00:17:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:57 smithi055 ceph-mon[97033]: pgmap v5108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:57 smithi059 ceph-mon[97450]: pgmap v5108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:17:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:57.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:57.997720+0000) 2022-09-08T00:17:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:58.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:58.079545+0000) 2022-09-08T00:17:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:17:59.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:59.079767+0000) 2022-09-08T00:17:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:58.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:58.997921+0000) 2022-09-08T00:17:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:17:59 smithi055 ceph-mon[97033]: pgmap v5109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:17:59 smithi059 ceph-mon[97450]: pgmap v5109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:00.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:00.079946+0000) 2022-09-08T00:18:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:17:59.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:17:59.998073+0000) 2022-09-08T00:18:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:00.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:00.998302+0000) 2022-09-08T00:18:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:01.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:01.080187+0000) 2022-09-08T00:18:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:01 smithi055 ceph-mon[97033]: pgmap v5110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:18:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:01 smithi059 ceph-mon[97450]: pgmap v5110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:18:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:01.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:01.998482+0000) 2022-09-08T00:18:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:02.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:02.080394+0000) 2022-09-08T00:18:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:03.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:03.080590+0000) 2022-09-08T00:18:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:02.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:02.998697+0000) 2022-09-08T00:18:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:03 smithi055 ceph-mon[97033]: pgmap v5111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:03 smithi059 ceph-mon[97450]: pgmap v5111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:03.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:03.998858+0000) 2022-09-08T00:18:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:04.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:04.080714+0000) 2022-09-08T00:18:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:18:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:18:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:18:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:18:05.364 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:05.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:05.080947+0000) 2022-09-08T00:18:05.365 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:04.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:04.999020+0000) 2022-09-08T00:18:05.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:18:05.363+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:18:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:05 smithi059 ceph-mon[97450]: pgmap v5112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:18:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:05 smithi055 ceph-mon[97033]: pgmap v5112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:18:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:06.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:06.081137+0000) 2022-09-08T00:18:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:05.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:05.999199+0000) 2022-09-08T00:18:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:07.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:07.081372+0000) 2022-09-08T00:18:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:06.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:06.999414+0000) 2022-09-08T00:18:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:07 smithi055 ceph-mon[97033]: pgmap v5113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:18:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:07 smithi059 ceph-mon[97450]: pgmap v5113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:18:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:07.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:07.999617+0000) 2022-09-08T00:18:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:08.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:08.081512+0000) 2022-09-08T00:18:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:09.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:09.081702+0000) 2022-09-08T00:18:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:08.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:08.999795+0000) 2022-09-08T00:18:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:09 smithi055 ceph-mon[97033]: osdmap e405: 12 total, 11 up, 11 in 2022-09-08T00:18:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:09 smithi055 ceph-mon[97033]: pgmap v5115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:18:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:09 smithi059 ceph-mon[97450]: osdmap e405: 12 total, 11 up, 11 in 2022-09-08T00:18:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:09 smithi059 ceph-mon[97450]: pgmap v5115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:18:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:10.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:10.081938+0000) 2022-09-08T00:18:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:09.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:09.999975+0000) 2022-09-08T00:18:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:11.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:11.082153+0000) 2022-09-08T00:18:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:10.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:11.000197+0000) 2022-09-08T00:18:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:11 smithi055 ceph-mon[97033]: pgmap v5116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:18:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:11 smithi059 ceph-mon[97450]: pgmap v5116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:18:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:12.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:12.082375+0000) 2022-09-08T00:18:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:12.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:12.000426+0000) 2022-09-08T00:18:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:13.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:13.082768+0000) 2022-09-08T00:18:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:13.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:13.000715+0000) 2022-09-08T00:18:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:13 smithi055 ceph-mon[97033]: pgmap v5117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:18:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:13 smithi059 ceph-mon[97450]: pgmap v5117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:18:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:14.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:14.082999+0000) 2022-09-08T00:18:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:14.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:14.000949+0000) 2022-09-08T00:18:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:15.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:15.083162+0000) 2022-09-08T00:18:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:15.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:15.001141+0000) 2022-09-08T00:18:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:15 smithi055 ceph-mon[97033]: pgmap v5118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:18:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:15 smithi055 ceph-mon[97033]: osdmap e406: 12 total, 11 up, 11 in 2022-09-08T00:18:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:15 smithi059 ceph-mon[97450]: pgmap v5118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:18:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:15 smithi059 ceph-mon[97450]: osdmap e406: 12 total, 11 up, 11 in 2022-09-08T00:18:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:16.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:16.083379+0000) 2022-09-08T00:18:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:16.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:16.001407+0000) 2022-09-08T00:18:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:17.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:17.083517+0000) 2022-09-08T00:18:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:17.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:17.001628+0000) 2022-09-08T00:18:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:17 smithi055 ceph-mon[97033]: pgmap v5120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T00:18:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:17 smithi059 ceph-mon[97450]: pgmap v5120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T00:18:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:18.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:18.083719+0000) 2022-09-08T00:18:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:18.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:18.001847+0000) 2022-09-08T00:18:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:18 smithi059 ceph-mon[97450]: pgmap v5121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T00:18:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:18 smithi059 ceph-mon[97450]: pgmap v5122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T00:18:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:18:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:18 smithi055 ceph-mon[97033]: pgmap v5121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T00:18:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:18 smithi055 ceph-mon[97033]: pgmap v5122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T00:18:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:18:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:19.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:19.083937+0000) 2022-09-08T00:18:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:19.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:19.002071+0000) 2022-09-08T00:18:20.372 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:20.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:20.084154+0000) 2022-09-08T00:18:20.373 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:20.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:20.002319+0000) 2022-09-08T00:18:20.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:18:20.371+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:18:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:20 smithi059 ceph-mon[97450]: pgmap v5123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T00:18:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:20 smithi055 ceph-mon[97033]: pgmap v5123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T00:18:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:21.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:21.002592+0000) 2022-09-08T00:18:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:21.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:21.084330+0000) 2022-09-08T00:18:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:22.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:22.084540+0000) 2022-09-08T00:18:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:22.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:22.002888+0000) 2022-09-08T00:18:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:22 smithi059 ceph-mon[97450]: pgmap v5124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:18:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:22 smithi055 ceph-mon[97033]: pgmap v5124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:18:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:23.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:23.084757+0000) 2022-09-08T00:18:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:23.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:23.003105+0000) 2022-09-08T00:18:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:24.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:24.084971+0000) 2022-09-08T00:18:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:24.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:24.003342+0000) 2022-09-08T00:18:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:24 smithi059 ceph-mon[97450]: pgmap v5125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:18:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:24 smithi055 ceph-mon[97033]: pgmap v5125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:18:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:25.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:25.085206+0000) 2022-09-08T00:18:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:25.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:25.003599+0000) 2022-09-08T00:18:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:26.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:26.085435+0000) 2022-09-08T00:18:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:26.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:26.003844+0000) 2022-09-08T00:18:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:26 smithi055 ceph-mon[97033]: pgmap v5126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:18:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:26 smithi059 ceph-mon[97450]: pgmap v5126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:18:27.412 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:27.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:27.085708+0000) 2022-09-08T00:18:27.413 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:27.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:27.004151+0000) 2022-09-08T00:18:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:28.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:28.085981+0000) 2022-09-08T00:18:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:28.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:28.004383+0000) 2022-09-08T00:18:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:28 smithi055 ceph-mon[97033]: pgmap v5127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:18:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:28 smithi059 ceph-mon[97450]: pgmap v5127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:18:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:29.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:29.086200+0000) 2022-09-08T00:18:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:29.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:29.004615+0000) 2022-09-08T00:18:30.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:30.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:30.086417+0000) 2022-09-08T00:18:30.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:30.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:30.004863+0000) 2022-09-08T00:18:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:30 smithi055 ceph-mon[97033]: pgmap v5128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:30 smithi059 ceph-mon[97450]: pgmap v5128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:31.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:31.005040+0000) 2022-09-08T00:18:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:31.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:31.086650+0000) 2022-09-08T00:18:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:32.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:32.086936+0000) 2022-09-08T00:18:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:32.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:32.005280+0000) 2022-09-08T00:18:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:32 smithi055 ceph-mon[97033]: pgmap v5129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:18:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:32 smithi059 ceph-mon[97450]: pgmap v5129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:18:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:33.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:33.087163+0000) 2022-09-08T00:18:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:33.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:33.005475+0000) 2022-09-08T00:18:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:34.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:34.087395+0000) 2022-09-08T00:18:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:34.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:34.005726+0000) 2022-09-08T00:18:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:34 smithi055 ceph-mon[97033]: pgmap v5130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:34 smithi059 ceph-mon[97450]: pgmap v5130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:35.381 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:35.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:35.087661+0000) 2022-09-08T00:18:35.382 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:35.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:35.005911+0000) 2022-09-08T00:18:35.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:18:35.379+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:18:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:36.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:36.087830+0000) 2022-09-08T00:18:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:36.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:36.006133+0000) 2022-09-08T00:18:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:36 smithi055 ceph-mon[97033]: pgmap v5131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:18:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:36 smithi059 ceph-mon[97450]: pgmap v5131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:18:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:37.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:37.088039+0000) 2022-09-08T00:18:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:37.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:37.006353+0000) 2022-09-08T00:18:38.088 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:38.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:38.006549+0000) 2022-09-08T00:18:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:38.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:38.088285+0000) 2022-09-08T00:18:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:38 smithi055 ceph-mon[97033]: pgmap v5132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:38 smithi059 ceph-mon[97450]: pgmap v5132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:39.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:39.088493+0000) 2022-09-08T00:18:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:39.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:39.006763+0000) 2022-09-08T00:18:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:40.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:40.088741+0000) 2022-09-08T00:18:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:40.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:40.006944+0000) 2022-09-08T00:18:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:40 smithi055 ceph-mon[97033]: pgmap v5133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:40 smithi059 ceph-mon[97450]: pgmap v5133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:41.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:41.088914+0000) 2022-09-08T00:18:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:41.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:41.007212+0000) 2022-09-08T00:18:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:42.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:42.089159+0000) 2022-09-08T00:18:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:42.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:42.007411+0000) 2022-09-08T00:18:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:42 smithi055 ceph-mon[97033]: pgmap v5134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:18:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:42 smithi059 ceph-mon[97450]: pgmap v5134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:18:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:43.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:43.089352+0000) 2022-09-08T00:18:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:43.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:43.007625+0000) 2022-09-08T00:18:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:44.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:44.089548+0000) 2022-09-08T00:18:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:44.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:44.007794+0000) 2022-09-08T00:18:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:44 smithi055 ceph-mon[97033]: pgmap v5135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:44 smithi059 ceph-mon[97450]: pgmap v5135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:45.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:45.089762+0000) 2022-09-08T00:18:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:45.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:45.008011+0000) 2022-09-08T00:18:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:46.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:46.089966+0000) 2022-09-08T00:18:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:46.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:46.008184+0000) 2022-09-08T00:18:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:46 smithi055 ceph-mon[97033]: pgmap v5136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:18:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:46 smithi059 ceph-mon[97450]: pgmap v5136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:18:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:47.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:47.090192+0000) 2022-09-08T00:18:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:47.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:47.008380+0000) 2022-09-08T00:18:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:48.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:48.090444+0000) 2022-09-08T00:18:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:48.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:48.008594+0000) 2022-09-08T00:18:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:48 smithi055 ceph-mon[97033]: pgmap v5137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:48 smithi059 ceph-mon[97450]: pgmap v5137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:49.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:49.090713+0000) 2022-09-08T00:18:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:49.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:49.008829+0000) 2022-09-08T00:18:50.397 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:50.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:50.090948+0000) 2022-09-08T00:18:50.398 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:50.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:50.009012+0000) 2022-09-08T00:18:50.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:18:50.395+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:18:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:50 smithi055 ceph-mon[97033]: pgmap v5138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:50 smithi059 ceph-mon[97450]: pgmap v5138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:51.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:51.091188+0000) 2022-09-08T00:18:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:51.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:51.009193+0000) 2022-09-08T00:18:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:52.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:52.091437+0000) 2022-09-08T00:18:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:52.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:52.009406+0000) 2022-09-08T00:18:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:52 smithi055 ceph-mon[97033]: pgmap v5139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:18:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:52 smithi059 ceph-mon[97450]: pgmap v5139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:18:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:53.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:53.091619+0000) 2022-09-08T00:18:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:53.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:53.009660+0000) 2022-09-08T00:18:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:54.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:54.091831+0000) 2022-09-08T00:18:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:54.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:54.009942+0000) 2022-09-08T00:18:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:54 smithi055 ceph-mon[97033]: pgmap v5140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:54 smithi059 ceph-mon[97450]: pgmap v5140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:55.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:55.092038+0000) 2022-09-08T00:18:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:55.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:55.010105+0000) 2022-09-08T00:18:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:56.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:56.092221+0000) 2022-09-08T00:18:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:56.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:56.010283+0000) 2022-09-08T00:18:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:56 smithi055 ceph-mon[97033]: pgmap v5141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:18:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:56 smithi059 ceph-mon[97450]: pgmap v5141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:18:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:57.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:57.092454+0000) 2022-09-08T00:18:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:57.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:57.010505+0000) 2022-09-08T00:18:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:58.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:58.092721+0000) 2022-09-08T00:18:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:58.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:58.010733+0000) 2022-09-08T00:18:59.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:18:58 smithi055 ceph-mon[97033]: pgmap v5142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:18:58 smithi059 ceph-mon[97450]: pgmap v5142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:18:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:18:59.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:59.092967+0000) 2022-09-08T00:18:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:18:59.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:18:59.010940+0000) 2022-09-08T00:19:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:00.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:00.093170+0000) 2022-09-08T00:19:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:00.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:00.011181+0000) 2022-09-08T00:19:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:00 smithi055 ceph-mon[97033]: pgmap v5143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:00 smithi059 ceph-mon[97450]: pgmap v5143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:01.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:01.093338+0000) 2022-09-08T00:19:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:01.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:01.011400+0000) 2022-09-08T00:19:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:02.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:02.093480+0000) 2022-09-08T00:19:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:02.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:02.011689+0000) 2022-09-08T00:19:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:02 smithi055 ceph-mon[97033]: pgmap v5144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:19:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:02 smithi059 ceph-mon[97450]: pgmap v5144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:19:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:03.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:03.093715+0000) 2022-09-08T00:19:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:03.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:03.011931+0000) 2022-09-08T00:19:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:04.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:04.093909+0000) 2022-09-08T00:19:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:04.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:04.012186+0000) 2022-09-08T00:19:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:04 smithi055 ceph-mon[97033]: pgmap v5145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:19:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:19:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:04 smithi059 ceph-mon[97450]: pgmap v5145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:19:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:19:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:05.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:05.094154+0000) 2022-09-08T00:19:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:05.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:05.012428+0000) 2022-09-08T00:19:05.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:19:05.410+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:19:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:06.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:06.094374+0000) 2022-09-08T00:19:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:06.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:06.012571+0000) 2022-09-08T00:19:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:06 smithi055 ceph-mon[97033]: pgmap v5146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:19:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:06 smithi059 ceph-mon[97450]: pgmap v5146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:19:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:07.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:07.094613+0000) 2022-09-08T00:19:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:07.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:07.012827+0000) 2022-09-08T00:19:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:08.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:08.094863+0000) 2022-09-08T00:19:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:08.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:08.013039+0000) 2022-09-08T00:19:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:08 smithi055 ceph-mon[97033]: pgmap v5147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:19:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:08 smithi059 ceph-mon[97450]: pgmap v5147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:19:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:09.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:09.095114+0000) 2022-09-08T00:19:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:09.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:09.013240+0000) 2022-09-08T00:19:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:09 smithi055 ceph-mon[97033]: osdmap e407: 12 total, 11 up, 11 in 2022-09-08T00:19:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:09 smithi059 ceph-mon[97450]: osdmap e407: 12 total, 11 up, 11 in 2022-09-08T00:19:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:10.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:10.095351+0000) 2022-09-08T00:19:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:10.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:10.013439+0000) 2022-09-08T00:19:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:10 smithi055 ceph-mon[97033]: pgmap v5149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:19:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:10 smithi059 ceph-mon[97450]: pgmap v5149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:19:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:11.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:11.095593+0000) 2022-09-08T00:19:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:11.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:11.013653+0000) 2022-09-08T00:19:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:12.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:12.095847+0000) 2022-09-08T00:19:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:12.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:12.013915+0000) 2022-09-08T00:19:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:12 smithi055 ceph-mon[97033]: pgmap v5150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:19:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:12 smithi059 ceph-mon[97450]: pgmap v5150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:19:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:13.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:13.096124+0000) 2022-09-08T00:19:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:13.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:13.014167+0000) 2022-09-08T00:19:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:14.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:14.096373+0000) 2022-09-08T00:19:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:14.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:14.014337+0000) 2022-09-08T00:19:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:14 smithi055 ceph-mon[97033]: pgmap v5151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:19:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:14 smithi059 ceph-mon[97450]: pgmap v5151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:19:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:15.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:15.096700+0000) 2022-09-08T00:19:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:15.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:15.014557+0000) 2022-09-08T00:19:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:16.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:16.096986+0000) 2022-09-08T00:19:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:16.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:16.014727+0000) 2022-09-08T00:19:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:16 smithi055 ceph-mon[97033]: osdmap e408: 12 total, 11 up, 11 in 2022-09-08T00:19:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:16 smithi055 ceph-mon[97033]: pgmap v5153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:19:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:16 smithi059 ceph-mon[97450]: osdmap e408: 12 total, 11 up, 11 in 2022-09-08T00:19:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:16 smithi059 ceph-mon[97450]: pgmap v5153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:19:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:17.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:17.097193+0000) 2022-09-08T00:19:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:17.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:17.014893+0000) 2022-09-08T00:19:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:18.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:18.097421+0000) 2022-09-08T00:19:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:18.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:18.015037+0000) 2022-09-08T00:19:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:18 smithi055 ceph-mon[97033]: pgmap v5154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T00:19:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:19:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:19:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:19:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:18 smithi059 ceph-mon[97450]: pgmap v5154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T00:19:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:19:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:19:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:19:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:19.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:19.097692+0000) 2022-09-08T00:19:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:19.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:19.015265+0000) 2022-09-08T00:19:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:20.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:20.097906+0000) 2022-09-08T00:19:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:20.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:20.015507+0000) 2022-09-08T00:19:20.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:19:20.410+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:19:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:20 smithi055 ceph-mon[97033]: pgmap v5155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:19:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:20 smithi059 ceph-mon[97450]: pgmap v5155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:19:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:21.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:21.098117+0000) 2022-09-08T00:19:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:21.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:21.015759+0000) 2022-09-08T00:19:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:22.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:22.098380+0000) 2022-09-08T00:19:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:22.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:22.015991+0000) 2022-09-08T00:19:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:22 smithi055 ceph-mon[97033]: pgmap v5156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:19:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:22 smithi059 ceph-mon[97450]: pgmap v5156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:19:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:23.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:23.098503+0000) 2022-09-08T00:19:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:23.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:23.016182+0000) 2022-09-08T00:19:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:24.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:24.016379+0000) 2022-09-08T00:19:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:24.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:24.098728+0000) 2022-09-08T00:19:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:24 smithi055 ceph-mon[97033]: pgmap v5157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:19:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:24 smithi059 ceph-mon[97450]: pgmap v5157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:19:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:25.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:25.099002+0000) 2022-09-08T00:19:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:25.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:25.016591+0000) 2022-09-08T00:19:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:26.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:26.099267+0000) 2022-09-08T00:19:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:26.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:26.016854+0000) 2022-09-08T00:19:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:26 smithi055 ceph-mon[97033]: pgmap v5158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:19:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:26 smithi059 ceph-mon[97450]: pgmap v5158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:19:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:27.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:27.017099+0000) 2022-09-08T00:19:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:27.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:27.099450+0000) 2022-09-08T00:19:28.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:28.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:28.017363+0000) 2022-09-08T00:19:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:28.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:28.099592+0000) 2022-09-08T00:19:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:28 smithi055 ceph-mon[97033]: pgmap v5159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:28 smithi059 ceph-mon[97450]: pgmap v5159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:29.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:29.099864+0000) 2022-09-08T00:19:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:29.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:29.017548+0000) 2022-09-08T00:19:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:30.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:30.100109+0000) 2022-09-08T00:19:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:30.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:30.017696+0000) 2022-09-08T00:19:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:30 smithi055 ceph-mon[97033]: pgmap v5160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:30 smithi059 ceph-mon[97450]: pgmap v5160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:31.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:31.100325+0000) 2022-09-08T00:19:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:31.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:31.017815+0000) 2022-09-08T00:19:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:32.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:32.100495+0000) 2022-09-08T00:19:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:32.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:32.018074+0000) 2022-09-08T00:19:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:32 smithi055 ceph-mon[97033]: pgmap v5161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:19:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:32 smithi059 ceph-mon[97450]: pgmap v5161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:19:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:33.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:33.100780+0000) 2022-09-08T00:19:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:33.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:33.018306+0000) 2022-09-08T00:19:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:34.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:34.101069+0000) 2022-09-08T00:19:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:34.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:34.018527+0000) 2022-09-08T00:19:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:34 smithi055 ceph-mon[97033]: pgmap v5162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:34 smithi059 ceph-mon[97450]: pgmap v5162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:35.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:35.101224+0000) 2022-09-08T00:19:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:35.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:35.018723+0000) 2022-09-08T00:19:35.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:19:35.427+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:19:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:36.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:36.101470+0000) 2022-09-08T00:19:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:36.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:36.018862+0000) 2022-09-08T00:19:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:36 smithi055 ceph-mon[97033]: pgmap v5163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:19:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:36 smithi059 ceph-mon[97450]: pgmap v5163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:19:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:37.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:37.101705+0000) 2022-09-08T00:19:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:37.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:37.019038+0000) 2022-09-08T00:19:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:38.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:38.101915+0000) 2022-09-08T00:19:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:38.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:38.019151+0000) 2022-09-08T00:19:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:38 smithi055 ceph-mon[97033]: pgmap v5164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:38 smithi059 ceph-mon[97450]: pgmap v5164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:39.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:39.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:39.019352+0000) 2022-09-08T00:19:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:39.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:39.102140+0000) 2022-09-08T00:19:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:40.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:40.102279+0000) 2022-09-08T00:19:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:40.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:40.019510+0000) 2022-09-08T00:19:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:40 smithi055 ceph-mon[97033]: pgmap v5165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:40 smithi059 ceph-mon[97450]: pgmap v5165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:41.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:41.102523+0000) 2022-09-08T00:19:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:41.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:41.019749+0000) 2022-09-08T00:19:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:42.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:42.019969+0000) 2022-09-08T00:19:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:42.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:42.102769+0000) 2022-09-08T00:19:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:42 smithi055 ceph-mon[97033]: pgmap v5166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:19:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:42 smithi059 ceph-mon[97450]: pgmap v5166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:19:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:43.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:43.103022+0000) 2022-09-08T00:19:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:43.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:43.020130+0000) 2022-09-08T00:19:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:44.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:44.020304+0000) 2022-09-08T00:19:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:44.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:44.103169+0000) 2022-09-08T00:19:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:44 smithi055 ceph-mon[97033]: pgmap v5167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:44 smithi059 ceph-mon[97450]: pgmap v5167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:45.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:45.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:45.020494+0000) 2022-09-08T00:19:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:45.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:45.103360+0000) 2022-09-08T00:19:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:46.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:46.020728+0000) 2022-09-08T00:19:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:46.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:46.103525+0000) 2022-09-08T00:19:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:46 smithi055 ceph-mon[97033]: pgmap v5168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:19:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:46 smithi059 ceph-mon[97450]: pgmap v5168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:19:47.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:47.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:47.103748+0000) 2022-09-08T00:19:47.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:47.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:47.020969+0000) 2022-09-08T00:19:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:48.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:48.104001+0000) 2022-09-08T00:19:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:48.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:48.021246+0000) 2022-09-08T00:19:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:48 smithi055 ceph-mon[97033]: pgmap v5169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:48 smithi059 ceph-mon[97450]: pgmap v5169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:49.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:49.104198+0000) 2022-09-08T00:19:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:49.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:49.021435+0000) 2022-09-08T00:19:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:50.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:50.104400+0000) 2022-09-08T00:19:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:50.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:50.021708+0000) 2022-09-08T00:19:50.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:19:50.431+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:19:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:50 smithi055 ceph-mon[97033]: pgmap v5170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:50 smithi059 ceph-mon[97450]: pgmap v5170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:19:51.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:51.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:51.021952+0000) 2022-09-08T00:19:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:51.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:51.104598+0000) 2022-09-08T00:19:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:52.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:52.104709+0000) 2022-09-08T00:19:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:52.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:52.022155+0000) 2022-09-08T00:19:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:53.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:53.104931+0000) 2022-09-08T00:19:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:53.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:53.022369+0000) 2022-09-08T00:19:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:53 smithi059 ceph-mon[97450]: pgmap v5171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:19:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:19:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:19:53.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:53 smithi059 ceph-mon[97450]: pgmap v5172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:19:53.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:53 smithi059 ceph-mon[97450]: pgmap v5173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:19:53.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:19:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:53 smithi055 ceph-mon[97033]: pgmap v5171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:19:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:19:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:19:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:53 smithi055 ceph-mon[97033]: pgmap v5172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:19:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:53 smithi055 ceph-mon[97033]: pgmap v5173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:19:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:19:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:54.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:54.105133+0000) 2022-09-08T00:19:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:54.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:54.022603+0000) 2022-09-08T00:19:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:54 smithi055 ceph-mon[97033]: pgmap v5174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T00:19:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:54 smithi059 ceph-mon[97450]: pgmap v5174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T00:19:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:55.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:55.105322+0000) 2022-09-08T00:19:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:55.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:55.022823+0000) 2022-09-08T00:19:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:56.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:56.105575+0000) 2022-09-08T00:19:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:56.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:56.023002+0000) 2022-09-08T00:19:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:56 smithi055 ceph-mon[97033]: pgmap v5175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:19:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:56 smithi059 ceph-mon[97450]: pgmap v5175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:19:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:57.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:57.105812+0000) 2022-09-08T00:19:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:57.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:57.023239+0000) 2022-09-08T00:19:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:58.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:58.105956+0000) 2022-09-08T00:19:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:58.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:58.023487+0000) 2022-09-08T00:19:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:19:58 smithi055 ceph-mon[97033]: pgmap v5176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:19:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:19:58 smithi059 ceph-mon[97450]: pgmap v5176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:19:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:19:59.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:59.106145+0000) 2022-09-08T00:19:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:19:59.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:19:59.023666+0000) 2022-09-08T00:20:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:00.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:00.106385+0000) 2022-09-08T00:20:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:00.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:00.023863+0000) 2022-09-08T00:20:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T00:20:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T00:20:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: ['Unable to reach remote host smithi047. '] 2022-09-08T00:20:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T00:20:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T00:20:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T00:20:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T00:20:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: ['Unable to reach remote host smithi047. '] 2022-09-08T00:20:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T00:20:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T00:20:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:01.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:01.106571+0000) 2022-09-08T00:20:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:01.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:01.024020+0000) 2022-09-08T00:20:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:01 smithi055 ceph-mon[97033]: pgmap v5177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:20:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:01 smithi059 ceph-mon[97450]: pgmap v5177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:20:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:02.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:02.106713+0000) 2022-09-08T00:20:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:02.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:02.024201+0000) 2022-09-08T00:20:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:02 smithi055 ceph-mon[97033]: pgmap v5178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:20:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:02 smithi059 ceph-mon[97450]: pgmap v5178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:20:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:03.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:03.106931+0000) 2022-09-08T00:20:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:03.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:03.024425+0000) 2022-09-08T00:20:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:04.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:04.107137+0000) 2022-09-08T00:20:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:04.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:04.024688+0000) 2022-09-08T00:20:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:04 smithi055 ceph-mon[97033]: pgmap v5179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:04 smithi059 ceph-mon[97450]: pgmap v5179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:05.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:05.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:05.024926+0000) 2022-09-08T00:20:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:05.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:05.107370+0000) 2022-09-08T00:20:05.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:20:05.437+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:20:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:20:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:20:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:20:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:20:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:06.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:06.025139+0000) 2022-09-08T00:20:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:06.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:06.107585+0000) 2022-09-08T00:20:06.863 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:06 smithi059 ceph-mon[97450]: pgmap v5180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:20:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:06 smithi055 ceph-mon[97033]: pgmap v5180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:20:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:07.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:07.107763+0000) 2022-09-08T00:20:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:07.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:07.025398+0000) 2022-09-08T00:20:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:08.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:08.107996+0000) 2022-09-08T00:20:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:08.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:08.025613+0000) 2022-09-08T00:20:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:09.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:09.108212+0000) 2022-09-08T00:20:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:09.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:09.025801+0000) 2022-09-08T00:20:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:09 smithi055 ceph-mon[97033]: osdmap e409: 12 total, 11 up, 11 in 2022-09-08T00:20:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:09 smithi055 ceph-mon[97033]: pgmap v5182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:20:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:09 smithi059 ceph-mon[97450]: osdmap e409: 12 total, 11 up, 11 in 2022-09-08T00:20:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:09 smithi059 ceph-mon[97450]: pgmap v5182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:20:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:10.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:10.108370+0000) 2022-09-08T00:20:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:10.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:10.025968+0000) 2022-09-08T00:20:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:10 smithi055 ceph-mon[97033]: pgmap v5183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:20:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:10 smithi059 ceph-mon[97450]: pgmap v5183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:20:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:11.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:11.108526+0000) 2022-09-08T00:20:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:11.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:11.026196+0000) 2022-09-08T00:20:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:11 smithi055 ceph-mon[97033]: osdmap e410: 12 total, 11 up, 11 in 2022-09-08T00:20:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:11 smithi059 ceph-mon[97450]: osdmap e410: 12 total, 11 up, 11 in 2022-09-08T00:20:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:12.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:12.026481+0000) 2022-09-08T00:20:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:12.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:12.108665+0000) 2022-09-08T00:20:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:12 smithi055 ceph-mon[97033]: pgmap v5185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:20:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:12 smithi059 ceph-mon[97450]: pgmap v5185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:20:13.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:13.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:13.108878+0000) 2022-09-08T00:20:13.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:13.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:13.026694+0000) 2022-09-08T00:20:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:14.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:14.109111+0000) 2022-09-08T00:20:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:14.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:14.026864+0000) 2022-09-08T00:20:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:14 smithi055 ceph-mon[97033]: pgmap v5186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:20:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:14 smithi059 ceph-mon[97450]: pgmap v5186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:20:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:15.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:15.109305+0000) 2022-09-08T00:20:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:15.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:15.027098+0000) 2022-09-08T00:20:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:16.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:16.109449+0000) 2022-09-08T00:20:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:16.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:16.027276+0000) 2022-09-08T00:20:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:16 smithi055 ceph-mon[97033]: pgmap v5187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T00:20:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:16 smithi059 ceph-mon[97450]: pgmap v5187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T00:20:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:17.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:17.109669+0000) 2022-09-08T00:20:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:17.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:17.027505+0000) 2022-09-08T00:20:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:18.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:18.109865+0000) 2022-09-08T00:20:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:18.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:18.027734+0000) 2022-09-08T00:20:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:18 smithi055 ceph-mon[97033]: pgmap v5188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:20:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:18 smithi059 ceph-mon[97450]: pgmap v5188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:20:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:19.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:19.110053+0000) 2022-09-08T00:20:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:19.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:19.027962+0000) 2022-09-08T00:20:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:20.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:20.110298+0000) 2022-09-08T00:20:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:20.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:20.028173+0000) 2022-09-08T00:20:20.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:20:20.441+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:20:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:20 smithi055 ceph-mon[97033]: pgmap v5189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:20:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:20 smithi059 ceph-mon[97450]: pgmap v5189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:20:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:21.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:21.110499+0000) 2022-09-08T00:20:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:21.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:21.028368+0000) 2022-09-08T00:20:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:22.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:22.110710+0000) 2022-09-08T00:20:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:22.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:22.028603+0000) 2022-09-08T00:20:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:22 smithi055 ceph-mon[97033]: pgmap v5190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:20:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:22 smithi059 ceph-mon[97450]: pgmap v5190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:20:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:23.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:23.110949+0000) 2022-09-08T00:20:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:23.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:23.028818+0000) 2022-09-08T00:20:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:24.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:24.111138+0000) 2022-09-08T00:20:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:24.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:24.029047+0000) 2022-09-08T00:20:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:24 smithi055 ceph-mon[97033]: pgmap v5191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:20:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:24 smithi059 ceph-mon[97450]: pgmap v5191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:20:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:25.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:25.111363+0000) 2022-09-08T00:20:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:25.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:25.029274+0000) 2022-09-08T00:20:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:26.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:26.111550+0000) 2022-09-08T00:20:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:26.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:26.029510+0000) 2022-09-08T00:20:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:26 smithi055 ceph-mon[97033]: pgmap v5192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:20:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:26 smithi059 ceph-mon[97450]: pgmap v5192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:20:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:27.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:27.111768+0000) 2022-09-08T00:20:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:27.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:27.029774+0000) 2022-09-08T00:20:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:28.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:28.112008+0000) 2022-09-08T00:20:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:28.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:28.030042+0000) 2022-09-08T00:20:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:28 smithi055 ceph-mon[97033]: pgmap v5193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:28 smithi059 ceph-mon[97450]: pgmap v5193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:29.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:29.112238+0000) 2022-09-08T00:20:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:29.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:29.030308+0000) 2022-09-08T00:20:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:30.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:30.112525+0000) 2022-09-08T00:20:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:30.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:30.030547+0000) 2022-09-08T00:20:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:30 smithi055 ceph-mon[97033]: pgmap v5194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:20:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:30 smithi059 ceph-mon[97450]: pgmap v5194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:20:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:31.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:31.112764+0000) 2022-09-08T00:20:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:31.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:31.030772+0000) 2022-09-08T00:20:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:32.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:32.113018+0000) 2022-09-08T00:20:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:32.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:32.030913+0000) 2022-09-08T00:20:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:32 smithi055 ceph-mon[97033]: pgmap v5195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:32 smithi059 ceph-mon[97450]: pgmap v5195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:33.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:33.113249+0000) 2022-09-08T00:20:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:33.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:33.031087+0000) 2022-09-08T00:20:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:34.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:34.113397+0000) 2022-09-08T00:20:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:34.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:34.031209+0000) 2022-09-08T00:20:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:34 smithi055 ceph-mon[97033]: pgmap v5196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:34 smithi059 ceph-mon[97450]: pgmap v5196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:35.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:35.113605+0000) 2022-09-08T00:20:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:35.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:35.031450+0000) 2022-09-08T00:20:35.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:20:35.453+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:20:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:36.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:36.113870+0000) 2022-09-08T00:20:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:36.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:36.031754+0000) 2022-09-08T00:20:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:36 smithi055 ceph-mon[97033]: pgmap v5197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:20:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:36 smithi059 ceph-mon[97450]: pgmap v5197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:20:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:37.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:37.114068+0000) 2022-09-08T00:20:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:37.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:37.031914+0000) 2022-09-08T00:20:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:38.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:38.114284+0000) 2022-09-08T00:20:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:38.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:38.032036+0000) 2022-09-08T00:20:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:38 smithi055 ceph-mon[97033]: pgmap v5198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:38 smithi059 ceph-mon[97450]: pgmap v5198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:39.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:39.114486+0000) 2022-09-08T00:20:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:39.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:39.032299+0000) 2022-09-08T00:20:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:40.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:40.114751+0000) 2022-09-08T00:20:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:40.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:40.032562+0000) 2022-09-08T00:20:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:40 smithi055 ceph-mon[97033]: pgmap v5199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:20:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:40 smithi059 ceph-mon[97450]: pgmap v5199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:20:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:41.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:41.114942+0000) 2022-09-08T00:20:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:41.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:41.032764+0000) 2022-09-08T00:20:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:42.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:42.115146+0000) 2022-09-08T00:20:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:42.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:42.032888+0000) 2022-09-08T00:20:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:42 smithi055 ceph-mon[97033]: pgmap v5200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:42 smithi059 ceph-mon[97450]: pgmap v5200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:43.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:43.115371+0000) 2022-09-08T00:20:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:43.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:43.033021+0000) 2022-09-08T00:20:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:44.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:44.115651+0000) 2022-09-08T00:20:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:44.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:44.033258+0000) 2022-09-08T00:20:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:44 smithi055 ceph-mon[97033]: pgmap v5201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:44 smithi059 ceph-mon[97450]: pgmap v5201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:45.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:45.115869+0000) 2022-09-08T00:20:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:45.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:45.033447+0000) 2022-09-08T00:20:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:46.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:46.116039+0000) 2022-09-08T00:20:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:46.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:46.033651+0000) 2022-09-08T00:20:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:46 smithi055 ceph-mon[97033]: pgmap v5202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:20:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:46 smithi059 ceph-mon[97450]: pgmap v5202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:20:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:47.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:47.116251+0000) 2022-09-08T00:20:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:47.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:47.033784+0000) 2022-09-08T00:20:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:48.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:48.116499+0000) 2022-09-08T00:20:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:48.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:48.033971+0000) 2022-09-08T00:20:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:48 smithi055 ceph-mon[97033]: pgmap v5203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:48 smithi059 ceph-mon[97450]: pgmap v5203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:49.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:49.116763+0000) 2022-09-08T00:20:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:49.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:49.034169+0000) 2022-09-08T00:20:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:50.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:50.116988+0000) 2022-09-08T00:20:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:50.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:50.034387+0000) 2022-09-08T00:20:50.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:20:50.460+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:20:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:50 smithi055 ceph-mon[97033]: pgmap v5204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:20:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:50 smithi059 ceph-mon[97450]: pgmap v5204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:20:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:51.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:51.034630+0000) 2022-09-08T00:20:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:51.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:51.117228+0000) 2022-09-08T00:20:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:52.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:52.117446+0000) 2022-09-08T00:20:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:52.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:52.034933+0000) 2022-09-08T00:20:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:52 smithi055 ceph-mon[97033]: pgmap v5205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:52 smithi059 ceph-mon[97450]: pgmap v5205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:53.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:53.117591+0000) 2022-09-08T00:20:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:53.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:53.035206+0000) 2022-09-08T00:20:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:20:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:20:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:20:53.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:20:53.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:20:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:20:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:20:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:20:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:20:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:20:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:54.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:54.117807+0000) 2022-09-08T00:20:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:54.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:54.035465+0000) 2022-09-08T00:20:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:54 smithi055 ceph-mon[97033]: pgmap v5206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:54 smithi059 ceph-mon[97450]: pgmap v5206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:20:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:55.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:55.118042+0000) 2022-09-08T00:20:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:55.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:55.035715+0000) 2022-09-08T00:20:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:56.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:56.118246+0000) 2022-09-08T00:20:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:56.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:56.035976+0000) 2022-09-08T00:20:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:20:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:20:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:20:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:20:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:20:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:20:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:20:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:20:56.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:20:56.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:20:56.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:20:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:57.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:57.118415+0000) 2022-09-08T00:20:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:57.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:57.036179+0000) 2022-09-08T00:20:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:57 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T00:20:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:57 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T00:20:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:57 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T00:20:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:57 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T00:20:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:57 smithi055 ceph-mon[97033]: pgmap v5207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T00:20:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:57 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T00:20:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:57 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T00:20:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:57 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T00:20:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:57 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T00:20:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:57 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T00:20:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:57 smithi059 ceph-mon[97450]: pgmap v5207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T00:20:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:57 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T00:20:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:58.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:58.118689+0000) 2022-09-08T00:20:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:58.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:58.036424+0000) 2022-09-08T00:20:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:20:59.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:59.118889+0000) 2022-09-08T00:20:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:20:59.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:20:59.036700+0000) 2022-09-08T00:20:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:20:59 smithi055 ceph-mon[97033]: pgmap v5208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:20:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:20:59 smithi059 ceph-mon[97450]: pgmap v5208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:21:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:00.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:00.119140+0000) 2022-09-08T00:21:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:00.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:00.036942+0000) 2022-09-08T00:21:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:01.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:01.037102+0000) 2022-09-08T00:21:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:01.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:01.119282+0000) 2022-09-08T00:21:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:01 smithi055 ceph-mon[97033]: pgmap v5209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T00:21:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:01 smithi059 ceph-mon[97450]: pgmap v5209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T00:21:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:02.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:02.119448+0000) 2022-09-08T00:21:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:02.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:02.037297+0000) 2022-09-08T00:21:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:03.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:03.119721+0000) 2022-09-08T00:21:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:03.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:03.037535+0000) 2022-09-08T00:21:03.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:03 smithi055 ceph-mon[97033]: pgmap v5210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:21:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:03 smithi059 ceph-mon[97450]: pgmap v5210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:21:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:04.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:04.119939+0000) 2022-09-08T00:21:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:04.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:04.037723+0000) 2022-09-08T00:21:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:05.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:05.120106+0000) 2022-09-08T00:21:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:05.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:05.037959+0000) 2022-09-08T00:21:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:05 smithi055 ceph-mon[97033]: pgmap v5211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:21:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:21:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:21:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:05 smithi059 ceph-mon[97450]: pgmap v5211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:21:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:21:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:21:05.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:21:05.476+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:21:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:06.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:06.120339+0000) 2022-09-08T00:21:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:06.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:06.038186+0000) 2022-09-08T00:21:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:06 smithi055 ceph-mon[97033]: pgmap v5212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2022-09-08T00:21:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:06 smithi059 ceph-mon[97450]: pgmap v5212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2022-09-08T00:21:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:07.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:07.120593+0000) 2022-09-08T00:21:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:07.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:07.038487+0000) 2022-09-08T00:21:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:08.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:08.120852+0000) 2022-09-08T00:21:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:08.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:08.038675+0000) 2022-09-08T00:21:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:09.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:09.121067+0000) 2022-09-08T00:21:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:09.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:09.038868+0000) 2022-09-08T00:21:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:09 smithi055 ceph-mon[97033]: pgmap v5213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:21:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:09 smithi055 ceph-mon[97033]: osdmap e411: 12 total, 11 up, 11 in 2022-09-08T00:21:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:09 smithi059 ceph-mon[97450]: pgmap v5213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:21:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:09 smithi059 ceph-mon[97450]: osdmap e411: 12 total, 11 up, 11 in 2022-09-08T00:21:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:10.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:10.039110+0000) 2022-09-08T00:21:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:10.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:10.121272+0000) 2022-09-08T00:21:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:11.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:11.121473+0000) 2022-09-08T00:21:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:11.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:11.039328+0000) 2022-09-08T00:21:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:11 smithi055 ceph-mon[97033]: pgmap v5215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:21:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:11 smithi055 ceph-mon[97033]: osdmap e412: 12 total, 11 up, 11 in 2022-09-08T00:21:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:11 smithi059 ceph-mon[97450]: pgmap v5215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:21:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:11 smithi059 ceph-mon[97450]: osdmap e412: 12 total, 11 up, 11 in 2022-09-08T00:21:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:12.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:12.121746+0000) 2022-09-08T00:21:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:12.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:12.039575+0000) 2022-09-08T00:21:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:13.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:13.121992+0000) 2022-09-08T00:21:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:13.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:13.039753+0000) 2022-09-08T00:21:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:13 smithi055 ceph-mon[97033]: pgmap v5217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T00:21:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:13 smithi059 ceph-mon[97450]: pgmap v5217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T00:21:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:14.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:14.122271+0000) 2022-09-08T00:21:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:14.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:14.039996+0000) 2022-09-08T00:21:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:15.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:15.040244+0000) 2022-09-08T00:21:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:15.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:15.122489+0000) 2022-09-08T00:21:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:15 smithi055 ceph-mon[97033]: pgmap v5218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:21:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:15 smithi059 ceph-mon[97450]: pgmap v5218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:21:16.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:16.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:16.040449+0000) 2022-09-08T00:21:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:16.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:16.122626+0000) 2022-09-08T00:21:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:16 smithi055 ceph-mon[97033]: pgmap v5219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:21:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:16 smithi059 ceph-mon[97450]: pgmap v5219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:21:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:17.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:17.040654+0000) 2022-09-08T00:21:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:17.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:17.122819+0000) 2022-09-08T00:21:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:18.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:18.122979+0000) 2022-09-08T00:21:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:18.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:18.040781+0000) 2022-09-08T00:21:19.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:19.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:19.040996+0000) 2022-09-08T00:21:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:19.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:19.123156+0000) 2022-09-08T00:21:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:19 smithi055 ceph-mon[97033]: pgmap v5220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T00:21:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:19 smithi059 ceph-mon[97450]: pgmap v5220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T00:21:20.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:20.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:20.041222+0000) 2022-09-08T00:21:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:20.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:20.123406+0000) 2022-09-08T00:21:20.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:21:20.482+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:21:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:21.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:21.123624+0000) 2022-09-08T00:21:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:21.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:21.041448+0000) 2022-09-08T00:21:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:21 smithi055 ceph-mon[97033]: pgmap v5221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T00:21:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:21 smithi059 ceph-mon[97450]: pgmap v5221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T00:21:22.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:22.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:22.123881+0000) 2022-09-08T00:21:22.409 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:22.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:22.041718+0000) 2022-09-08T00:21:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:23.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:23.124128+0000) 2022-09-08T00:21:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:23.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:23.041997+0000) 2022-09-08T00:21:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:23 smithi055 ceph-mon[97033]: pgmap v5222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 86 B/s wr, 4 op/s 2022-09-08T00:21:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:23 smithi059 ceph-mon[97450]: pgmap v5222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 86 B/s wr, 4 op/s 2022-09-08T00:21:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:24.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:24.124329+0000) 2022-09-08T00:21:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:24.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:24.042207+0000) 2022-09-08T00:21:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:25.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:25.124522+0000) 2022-09-08T00:21:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:25.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:25.042454+0000) 2022-09-08T00:21:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:25 smithi055 ceph-mon[97033]: pgmap v5223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 85 B/s wr, 4 op/s 2022-09-08T00:21:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:25 smithi059 ceph-mon[97450]: pgmap v5223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 85 B/s wr, 4 op/s 2022-09-08T00:21:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:26.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:26.124767+0000) 2022-09-08T00:21:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:26.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:26.042733+0000) 2022-09-08T00:21:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:26 smithi055 ceph-mon[97033]: pgmap v5224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:21:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:26 smithi059 ceph-mon[97450]: pgmap v5224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:21:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:27.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:27.125003+0000) 2022-09-08T00:21:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:27.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:27.042948+0000) 2022-09-08T00:21:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:28.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:28.125231+0000) 2022-09-08T00:21:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:28.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:28.043200+0000) 2022-09-08T00:21:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:29.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:29.125491+0000) 2022-09-08T00:21:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:29.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:29.043479+0000) 2022-09-08T00:21:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:29 smithi055 ceph-mon[97033]: pgmap v5225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:21:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:29 smithi059 ceph-mon[97450]: pgmap v5225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:21:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:30.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:30.125741+0000) 2022-09-08T00:21:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:30.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:30.043707+0000) 2022-09-08T00:21:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:31.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:31.126006+0000) 2022-09-08T00:21:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:31.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:31.043963+0000) 2022-09-08T00:21:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:31 smithi055 ceph-mon[97033]: pgmap v5226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:21:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:31 smithi059 ceph-mon[97450]: pgmap v5226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:21:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:32.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:32.126230+0000) 2022-09-08T00:21:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:32.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:32.044144+0000) 2022-09-08T00:21:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:33.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:33.126415+0000) 2022-09-08T00:21:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:33.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:33.044375+0000) 2022-09-08T00:21:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:33 smithi055 ceph-mon[97033]: pgmap v5227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:33 smithi059 ceph-mon[97450]: pgmap v5227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:34.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:34.126616+0000) 2022-09-08T00:21:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:34.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:34.044563+0000) 2022-09-08T00:21:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:35.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:35.126775+0000) 2022-09-08T00:21:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:35.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:35.044816+0000) 2022-09-08T00:21:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:35 smithi055 ceph-mon[97033]: pgmap v5228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:35 smithi059 ceph-mon[97450]: pgmap v5228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:35.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:21:35.496+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:21:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:36.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:36.126913+0000) 2022-09-08T00:21:36.445 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:36.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:36.045109+0000) 2022-09-08T00:21:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:36 smithi055 ceph-mon[97033]: pgmap v5229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:21:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:36 smithi059 ceph-mon[97450]: pgmap v5229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:21:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:37.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:37.127106+0000) 2022-09-08T00:21:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:37.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:37.045367+0000) 2022-09-08T00:21:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:38.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:38.127325+0000) 2022-09-08T00:21:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:38.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:38.045568+0000) 2022-09-08T00:21:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:39.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:39.127525+0000) 2022-09-08T00:21:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:39.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:39.045803+0000) 2022-09-08T00:21:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:39 smithi055 ceph-mon[97033]: pgmap v5230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:39 smithi059 ceph-mon[97450]: pgmap v5230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:40.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:40.127693+0000) 2022-09-08T00:21:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:40.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:40.046077+0000) 2022-09-08T00:21:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:41.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:41.127923+0000) 2022-09-08T00:21:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:41.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:41.046306+0000) 2022-09-08T00:21:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:41 smithi055 ceph-mon[97033]: pgmap v5231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:21:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:41 smithi059 ceph-mon[97450]: pgmap v5231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:21:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:42.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:42.128043+0000) 2022-09-08T00:21:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:42.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:42.046531+0000) 2022-09-08T00:21:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:43.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:43.128282+0000) 2022-09-08T00:21:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:43.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:43.046727+0000) 2022-09-08T00:21:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:43 smithi055 ceph-mon[97033]: pgmap v5232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:43 smithi059 ceph-mon[97450]: pgmap v5232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:44.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:44.128518+0000) 2022-09-08T00:21:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:44.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:44.046947+0000) 2022-09-08T00:21:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:45.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:45.128736+0000) 2022-09-08T00:21:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:45.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:45.047128+0000) 2022-09-08T00:21:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:45 smithi055 ceph-mon[97033]: pgmap v5233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:45 smithi059 ceph-mon[97450]: pgmap v5233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:46.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:46.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:46.047380+0000) 2022-09-08T00:21:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:46.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:46.128954+0000) 2022-09-08T00:21:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:46 smithi055 ceph-mon[97033]: pgmap v5234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:21:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:46 smithi059 ceph-mon[97450]: pgmap v5234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:21:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:47.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:47.047750+0000) 2022-09-08T00:21:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:47.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:47.129145+0000) 2022-09-08T00:21:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:48.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:48.129346+0000) 2022-09-08T00:21:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:48.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:48.047959+0000) 2022-09-08T00:21:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:49.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:49.129577+0000) 2022-09-08T00:21:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:49.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:49.048195+0000) 2022-09-08T00:21:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:49 smithi055 ceph-mon[97033]: pgmap v5235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:49 smithi059 ceph-mon[97450]: pgmap v5235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:50.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:50.129832+0000) 2022-09-08T00:21:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:50.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:50.048435+0000) 2022-09-08T00:21:50.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:21:50.509+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:21:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:51.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:51.048679+0000) 2022-09-08T00:21:51.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:51.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:51.130043+0000) 2022-09-08T00:21:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:51 smithi055 ceph-mon[97033]: pgmap v5236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:21:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:51 smithi059 ceph-mon[97450]: pgmap v5236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:21:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:52.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:52.048887+0000) 2022-09-08T00:21:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:52.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:52.130182+0000) 2022-09-08T00:21:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:53.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:53.130381+0000) 2022-09-08T00:21:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:53.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:53.049083+0000) 2022-09-08T00:21:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:53 smithi055 ceph-mon[97033]: pgmap v5237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:53 smithi059 ceph-mon[97450]: pgmap v5237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:54.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:54.130611+0000) 2022-09-08T00:21:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:54.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:54.049295+0000) 2022-09-08T00:21:55.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:55.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:55.049469+0000) 2022-09-08T00:21:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:55.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:55.130840+0000) 2022-09-08T00:21:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:55 smithi055 ceph-mon[97033]: pgmap v5238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:55 smithi059 ceph-mon[97450]: pgmap v5238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:56.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:56.131069+0000) 2022-09-08T00:21:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:56.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:56.049743+0000) 2022-09-08T00:21:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:56 smithi059 ceph-mon[97450]: pgmap v5239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:21:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:21:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:21:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:21:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:56 smithi055 ceph-mon[97033]: pgmap v5239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:21:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:21:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:21:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:21:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:57.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:57.131210+0000) 2022-09-08T00:21:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:57.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:57.049978+0000) 2022-09-08T00:21:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:21:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:21:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:21:58.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:21:58.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:21:58.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:21:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:58.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:58.131447+0000) 2022-09-08T00:21:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:58.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:58.050175+0000) 2022-09-08T00:21:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:21:58 smithi055 ceph-mon[97033]: pgmap v5240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:21:58 smithi059 ceph-mon[97450]: pgmap v5240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:21:59.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:21:59.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:59.050346+0000) 2022-09-08T00:21:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:21:59.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:21:59.131625+0000) 2022-09-08T00:22:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:00.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:00.131917+0000) 2022-09-08T00:22:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:00.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:00.050565+0000) 2022-09-08T00:22:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:01.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:01.132116+0000) 2022-09-08T00:22:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:01.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:01.050778+0000) 2022-09-08T00:22:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:01 smithi055 ceph-mon[97033]: pgmap v5241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:22:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:01 smithi059 ceph-mon[97450]: pgmap v5241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:22:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:02.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:02.132280+0000) 2022-09-08T00:22:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:02.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:02.050966+0000) 2022-09-08T00:22:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:03.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:03.132480+0000) 2022-09-08T00:22:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:03.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:03.051159+0000) 2022-09-08T00:22:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:03 smithi055 ceph-mon[97033]: pgmap v5242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:03 smithi059 ceph-mon[97450]: pgmap v5242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:04.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:04.132620+0000) 2022-09-08T00:22:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:04.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:04.051346+0000) 2022-09-08T00:22:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:05.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:05.132866+0000) 2022-09-08T00:22:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:05.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:05.051560+0000) 2022-09-08T00:22:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:05 smithi055 ceph-mon[97033]: pgmap v5243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:22:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:22:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:05 smithi059 ceph-mon[97450]: pgmap v5243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:22:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:22:05.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:22:05.526+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:22:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:06.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:06.133116+0000) 2022-09-08T00:22:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:06.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:06.051705+0000) 2022-09-08T00:22:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:06 smithi055 ceph-mon[97033]: pgmap v5244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:22:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:06 smithi059 ceph-mon[97450]: pgmap v5244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:22:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:07.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:07.133361+0000) 2022-09-08T00:22:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:07.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:07.051942+0000) 2022-09-08T00:22:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:08.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:08.133507+0000) 2022-09-08T00:22:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:08.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:08.052087+0000) 2022-09-08T00:22:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:09.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:09.133640+0000) 2022-09-08T00:22:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:09.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:09.052277+0000) 2022-09-08T00:22:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:09 smithi055 ceph-mon[97033]: pgmap v5245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:22:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:09 smithi055 ceph-mon[97033]: osdmap e413: 12 total, 11 up, 11 in 2022-09-08T00:22:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:09 smithi059 ceph-mon[97450]: pgmap v5245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:22:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:09 smithi059 ceph-mon[97450]: osdmap e413: 12 total, 11 up, 11 in 2022-09-08T00:22:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:10.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:10.133851+0000) 2022-09-08T00:22:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:10.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:10.052508+0000) 2022-09-08T00:22:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:11.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:11.134097+0000) 2022-09-08T00:22:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:11.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:11.052753+0000) 2022-09-08T00:22:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:11 smithi055 ceph-mon[97033]: pgmap v5247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:22:11.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:11 smithi055 ceph-mon[97033]: osdmap e414: 12 total, 11 up, 11 in 2022-09-08T00:22:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:11 smithi059 ceph-mon[97450]: pgmap v5247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:22:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:11 smithi059 ceph-mon[97450]: osdmap e414: 12 total, 11 up, 11 in 2022-09-08T00:22:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:12.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:12.134325+0000) 2022-09-08T00:22:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:12.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:12.053000+0000) 2022-09-08T00:22:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:13.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:13.134586+0000) 2022-09-08T00:22:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:13.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:13.053252+0000) 2022-09-08T00:22:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:13 smithi055 ceph-mon[97033]: pgmap v5249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:22:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:13 smithi059 ceph-mon[97450]: pgmap v5249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:22:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:14.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:14.134855+0000) 2022-09-08T00:22:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:14.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:14.053459+0000) 2022-09-08T00:22:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:15.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:15.135058+0000) 2022-09-08T00:22:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:15.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:15.053599+0000) 2022-09-08T00:22:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:15 smithi055 ceph-mon[97033]: pgmap v5250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:22:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:15 smithi059 ceph-mon[97450]: pgmap v5250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:22:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:16.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:16.135328+0000) 2022-09-08T00:22:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:16.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:16.053815+0000) 2022-09-08T00:22:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:16 smithi055 ceph-mon[97033]: pgmap v5251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:22:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:16 smithi059 ceph-mon[97450]: pgmap v5251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:22:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:17.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:17.135551+0000) 2022-09-08T00:22:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:17.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:17.054038+0000) 2022-09-08T00:22:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:18.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:18.135790+0000) 2022-09-08T00:22:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:18.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:18.054273+0000) 2022-09-08T00:22:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:19.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:19.135958+0000) 2022-09-08T00:22:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:19.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:19.054551+0000) 2022-09-08T00:22:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:19 smithi055 ceph-mon[97033]: pgmap v5252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:22:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:19 smithi059 ceph-mon[97450]: pgmap v5252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:22:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:20.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:20.136198+0000) 2022-09-08T00:22:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:20.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:20.054801+0000) 2022-09-08T00:22:20.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:22:20.528+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:22:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:21.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:21.136417+0000) 2022-09-08T00:22:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:21.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:21.055003+0000) 2022-09-08T00:22:21.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:21 smithi055 ceph-mon[97033]: pgmap v5253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:22:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:21 smithi059 ceph-mon[97450]: pgmap v5253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:22:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:22.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:22.136692+0000) 2022-09-08T00:22:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:22.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:22.055198+0000) 2022-09-08T00:22:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:23.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:23.136839+0000) 2022-09-08T00:22:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:23.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:23.055358+0000) 2022-09-08T00:22:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:23 smithi055 ceph-mon[97033]: pgmap v5254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:22:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:23 smithi059 ceph-mon[97450]: pgmap v5254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:22:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:24.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:24.055519+0000) 2022-09-08T00:22:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:24.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:24.136982+0000) 2022-09-08T00:22:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:25.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:25.137146+0000) 2022-09-08T00:22:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:25.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:25.055748+0000) 2022-09-08T00:22:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:25 smithi059 ceph-mon[97450]: pgmap v5255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:22:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:25 smithi055 ceph-mon[97033]: pgmap v5255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:22:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:26.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:26.137323+0000) 2022-09-08T00:22:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:26.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:26.055992+0000) 2022-09-08T00:22:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:26 smithi055 ceph-mon[97033]: pgmap v5256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:22:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:26 smithi059 ceph-mon[97450]: pgmap v5256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:22:27.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:27.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:27.137542+0000) 2022-09-08T00:22:27.412 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:27.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:27.056187+0000) 2022-09-08T00:22:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:28.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:28.137793+0000) 2022-09-08T00:22:28.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:28.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:28.056373+0000) 2022-09-08T00:22:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:29.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:29.138024+0000) 2022-09-08T00:22:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:29.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:29.056535+0000) 2022-09-08T00:22:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:29 smithi055 ceph-mon[97033]: pgmap v5257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:29 smithi059 ceph-mon[97450]: pgmap v5257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:30.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:30.138268+0000) 2022-09-08T00:22:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:30.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:30.056726+0000) 2022-09-08T00:22:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:31.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:31.056945+0000) 2022-09-08T00:22:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:31.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:31.138517+0000) 2022-09-08T00:22:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:31 smithi055 ceph-mon[97033]: pgmap v5258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:22:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:31 smithi059 ceph-mon[97450]: pgmap v5258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:22:32.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:32.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:32.057135+0000) 2022-09-08T00:22:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:32.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:32.138758+0000) 2022-09-08T00:22:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:33.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:33.139000+0000) 2022-09-08T00:22:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:33.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:33.057352+0000) 2022-09-08T00:22:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:33 smithi055 ceph-mon[97033]: pgmap v5259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:33 smithi059 ceph-mon[97450]: pgmap v5259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:34.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:34.139178+0000) 2022-09-08T00:22:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:34.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:34.057546+0000) 2022-09-08T00:22:35.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:35.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:35.057810+0000) 2022-09-08T00:22:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:35.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:35.139296+0000) 2022-09-08T00:22:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:35 smithi059 ceph-mon[97450]: pgmap v5260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:35.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:22:35.541+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:22:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:35 smithi055 ceph-mon[97033]: pgmap v5260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:36.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:36.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:36.058094+0000) 2022-09-08T00:22:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:36.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:36.139514+0000) 2022-09-08T00:22:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:36 smithi055 ceph-mon[97033]: pgmap v5261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:22:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:36 smithi059 ceph-mon[97450]: pgmap v5261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:22:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:37.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:37.058363+0000) 2022-09-08T00:22:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:37.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:37.139757+0000) 2022-09-08T00:22:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:38.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:38.139878+0000) 2022-09-08T00:22:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:38.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:38.058677+0000) 2022-09-08T00:22:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:39.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:39.140083+0000) 2022-09-08T00:22:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:39.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:39.058940+0000) 2022-09-08T00:22:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:39 smithi055 ceph-mon[97033]: pgmap v5262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:39 smithi059 ceph-mon[97450]: pgmap v5262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:40.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:40.140359+0000) 2022-09-08T00:22:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:40.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:40.059207+0000) 2022-09-08T00:22:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:41.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:41.140620+0000) 2022-09-08T00:22:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:41.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:41.059474+0000) 2022-09-08T00:22:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:41 smithi055 ceph-mon[97033]: pgmap v5263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:22:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:41 smithi059 ceph-mon[97450]: pgmap v5263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:22:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:42.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:42.140898+0000) 2022-09-08T00:22:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:42.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:42.059705+0000) 2022-09-08T00:22:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:43.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:43.141180+0000) 2022-09-08T00:22:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:43.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:43.059966+0000) 2022-09-08T00:22:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:43 smithi059 ceph-mon[97450]: pgmap v5264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:43 smithi055 ceph-mon[97033]: pgmap v5264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:44.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:44.141371+0000) 2022-09-08T00:22:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:44.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:44.060215+0000) 2022-09-08T00:22:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:45.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:45.141632+0000) 2022-09-08T00:22:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:45.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:45.060390+0000) 2022-09-08T00:22:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:45 smithi059 ceph-mon[97450]: pgmap v5265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:45 smithi055 ceph-mon[97033]: pgmap v5265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:46.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:46.141926+0000) 2022-09-08T00:22:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:46.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:46.060620+0000) 2022-09-08T00:22:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:46 smithi055 ceph-mon[97033]: pgmap v5266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:22:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:46 smithi059 ceph-mon[97450]: pgmap v5266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:22:47.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:47.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:47.060823+0000) 2022-09-08T00:22:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:47.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:47.142154+0000) 2022-09-08T00:22:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:48.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:48.142368+0000) 2022-09-08T00:22:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:48.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:48.061059+0000) 2022-09-08T00:22:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:49.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:49.142586+0000) 2022-09-08T00:22:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:49.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:49.061221+0000) 2022-09-08T00:22:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:49 smithi055 ceph-mon[97033]: pgmap v5267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:49 smithi059 ceph-mon[97450]: pgmap v5267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:50.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:50.061379+0000) 2022-09-08T00:22:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:50.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:50.142829+0000) 2022-09-08T00:22:50.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:22:50.545+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:22:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:51.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:51.061604+0000) 2022-09-08T00:22:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:51.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:51.143058+0000) 2022-09-08T00:22:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:51 smithi059 ceph-mon[97450]: pgmap v5268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:22:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:51 smithi055 ceph-mon[97033]: pgmap v5268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:22:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:52.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:52.143298+0000) 2022-09-08T00:22:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:52.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:52.061806+0000) 2022-09-08T00:22:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:53.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:53.143519+0000) 2022-09-08T00:22:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:53.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:53.062064+0000) 2022-09-08T00:22:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:53 smithi059 ceph-mon[97450]: pgmap v5269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:53 smithi055 ceph-mon[97033]: pgmap v5269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:54.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:54.143767+0000) 2022-09-08T00:22:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:54.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:54.062280+0000) 2022-09-08T00:22:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:55.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:55.144029+0000) 2022-09-08T00:22:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:55.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:55.062525+0000) 2022-09-08T00:22:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:55 smithi059 ceph-mon[97450]: pgmap v5270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:55 smithi055 ceph-mon[97033]: pgmap v5270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:56.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:56.144190+0000) 2022-09-08T00:22:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:56.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:56.062732+0000) 2022-09-08T00:22:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:56 smithi055 ceph-mon[97033]: pgmap v5271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:22:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:56 smithi059 ceph-mon[97450]: pgmap v5271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:22:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:57.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:57.144388+0000) 2022-09-08T00:22:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:57.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:57.062915+0000) 2022-09-08T00:22:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:22:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:22:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:22:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:22:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:22:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:22:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:58.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:58.144535+0000) 2022-09-08T00:22:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:58.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:58.063159+0000) 2022-09-08T00:22:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:22:59.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:59.144743+0000) 2022-09-08T00:22:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:22:59.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:22:59.063364+0000) 2022-09-08T00:22:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:22:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:22:59 smithi055 ceph-mon[97033]: pgmap v5272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:22:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:22:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:22:59 smithi059 ceph-mon[97450]: pgmap v5272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:00.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:00.144972+0000) 2022-09-08T00:23:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:00.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:00.063590+0000) 2022-09-08T00:23:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:01.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:01.145112+0000) 2022-09-08T00:23:01.415 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:01.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:01.063800+0000) 2022-09-08T00:23:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:01 smithi059 ceph-mon[97450]: pgmap v5273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:23:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:01 smithi055 ceph-mon[97033]: pgmap v5273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:23:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:02.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:02.145280+0000) 2022-09-08T00:23:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:02.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:02.063958+0000) 2022-09-08T00:23:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:03.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:03.145517+0000) 2022-09-08T00:23:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:03.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:03.064161+0000) 2022-09-08T00:23:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:03 smithi059 ceph-mon[97450]: pgmap v5274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:03 smithi055 ceph-mon[97033]: pgmap v5274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:04.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:04.145727+0000) 2022-09-08T00:23:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:04.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:04.064381+0000) 2022-09-08T00:23:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:05.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:05.145861+0000) 2022-09-08T00:23:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:05.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:05.064663+0000) 2022-09-08T00:23:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:05 smithi059 ceph-mon[97450]: pgmap v5275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:23:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:23:05.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:23:05.553+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:23:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:05 smithi055 ceph-mon[97033]: pgmap v5275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:23:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:23:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:06.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:06.146064+0000) 2022-09-08T00:23:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:06.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:06.064851+0000) 2022-09-08T00:23:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:06 smithi055 ceph-mon[97033]: pgmap v5276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:23:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:06 smithi059 ceph-mon[97450]: pgmap v5276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:23:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:07.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:07.146274+0000) 2022-09-08T00:23:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:07.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:07.065077+0000) 2022-09-08T00:23:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:08.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:08.146394+0000) 2022-09-08T00:23:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:08.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:08.065232+0000) 2022-09-08T00:23:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:09.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:09.146574+0000) 2022-09-08T00:23:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:09.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:09.065464+0000) 2022-09-08T00:23:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:09 smithi055 ceph-mon[97033]: osdmap e415: 12 total, 11 up, 11 in 2022-09-08T00:23:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:09 smithi055 ceph-mon[97033]: pgmap v5278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:23:09.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:09 smithi059 ceph-mon[97450]: osdmap e415: 12 total, 11 up, 11 in 2022-09-08T00:23:09.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:09 smithi059 ceph-mon[97450]: pgmap v5278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:23:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:10.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:10.146831+0000) 2022-09-08T00:23:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:10.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:10.065665+0000) 2022-09-08T00:23:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:11.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:11.147124+0000) 2022-09-08T00:23:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:11.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:11.065899+0000) 2022-09-08T00:23:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:11 smithi059 ceph-mon[97450]: pgmap v5279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:23:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:11 smithi059 ceph-mon[97450]: osdmap e416: 12 total, 11 up, 11 in 2022-09-08T00:23:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:11 smithi055 ceph-mon[97033]: pgmap v5279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:23:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:11 smithi055 ceph-mon[97033]: osdmap e416: 12 total, 11 up, 11 in 2022-09-08T00:23:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:12.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:12.147257+0000) 2022-09-08T00:23:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:12.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:12.066077+0000) 2022-09-08T00:23:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:13.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:13.066330+0000) 2022-09-08T00:23:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:13.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:13.147475+0000) 2022-09-08T00:23:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:13 smithi059 ceph-mon[97450]: pgmap v5281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:23:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:13 smithi055 ceph-mon[97033]: pgmap v5281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:23:14.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:14.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:14.066583+0000) 2022-09-08T00:23:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:14.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:14.147614+0000) 2022-09-08T00:23:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:15.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:15.147827+0000) 2022-09-08T00:23:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:15.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:15.066826+0000) 2022-09-08T00:23:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:15 smithi059 ceph-mon[97450]: pgmap v5282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:23:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:15 smithi055 ceph-mon[97033]: pgmap v5282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:23:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:16.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:16.066977+0000) 2022-09-08T00:23:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:16.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:16.148054+0000) 2022-09-08T00:23:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:16 smithi055 ceph-mon[97033]: pgmap v5283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:23:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:16 smithi059 ceph-mon[97450]: pgmap v5283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:23:17.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:17.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:17.067205+0000) 2022-09-08T00:23:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:17.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:17.148284+0000) 2022-09-08T00:23:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:18.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:18.148538+0000) 2022-09-08T00:23:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:18.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:18.067448+0000) 2022-09-08T00:23:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:19.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:19.148766+0000) 2022-09-08T00:23:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:19.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:19.067722+0000) 2022-09-08T00:23:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:19 smithi059 ceph-mon[97450]: pgmap v5284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:23:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:19 smithi055 ceph-mon[97033]: pgmap v5284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:23:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:20.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:20.149002+0000) 2022-09-08T00:23:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:20.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:20.067980+0000) 2022-09-08T00:23:20.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:23:20.568+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:23:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:21.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:21.068162+0000) 2022-09-08T00:23:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:21.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:21.149225+0000) 2022-09-08T00:23:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:21 smithi059 ceph-mon[97450]: pgmap v5285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:23:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:21 smithi055 ceph-mon[97033]: pgmap v5285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:23:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:22.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:22.149452+0000) 2022-09-08T00:23:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:22.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:22.068354+0000) 2022-09-08T00:23:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:23.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:23.149613+0000) 2022-09-08T00:23:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:23.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:23.068569+0000) 2022-09-08T00:23:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:23 smithi059 ceph-mon[97450]: pgmap v5286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:23:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:23 smithi055 ceph-mon[97033]: pgmap v5286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:23:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:24.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:24.149838+0000) 2022-09-08T00:23:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:24.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:24.068803+0000) 2022-09-08T00:23:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:25.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:25.150064+0000) 2022-09-08T00:23:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:25.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:25.069026+0000) 2022-09-08T00:23:25.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:25 smithi059 ceph-mon[97450]: pgmap v5287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:23:25.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:25 smithi055 ceph-mon[97033]: pgmap v5287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:23:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:26.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:26.150344+0000) 2022-09-08T00:23:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:26.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:26.069272+0000) 2022-09-08T00:23:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:26 smithi055 ceph-mon[97033]: pgmap v5288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:23:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:26 smithi059 ceph-mon[97450]: pgmap v5288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:23:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:27.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:27.150579+0000) 2022-09-08T00:23:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:27.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:27.069547+0000) 2022-09-08T00:23:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:28.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:28.150755+0000) 2022-09-08T00:23:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:28.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:28.069756+0000) 2022-09-08T00:23:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:29.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:29.150919+0000) 2022-09-08T00:23:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:29.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:29.069993+0000) 2022-09-08T00:23:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:29 smithi059 ceph-mon[97450]: pgmap v5289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:29 smithi055 ceph-mon[97033]: pgmap v5289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:30.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:30.151158+0000) 2022-09-08T00:23:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:30.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:30.070250+0000) 2022-09-08T00:23:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:31.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:31.151372+0000) 2022-09-08T00:23:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:31.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:31.070413+0000) 2022-09-08T00:23:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:31 smithi059 ceph-mon[97450]: pgmap v5290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:23:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:31 smithi055 ceph-mon[97033]: pgmap v5290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:23:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:32.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:32.151594+0000) 2022-09-08T00:23:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:32.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:32.070594+0000) 2022-09-08T00:23:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:33.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:33.070796+0000) 2022-09-08T00:23:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:33.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:33.151696+0000) 2022-09-08T00:23:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:33 smithi059 ceph-mon[97450]: pgmap v5291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:33 smithi055 ceph-mon[97033]: pgmap v5291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:34.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:34.151841+0000) 2022-09-08T00:23:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:34.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:34.070995+0000) 2022-09-08T00:23:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:35.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:35.152077+0000) 2022-09-08T00:23:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:35.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:35.071206+0000) 2022-09-08T00:23:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:35 smithi059 ceph-mon[97450]: pgmap v5292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:35.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:23:35.577+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:23:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:35 smithi055 ceph-mon[97033]: pgmap v5292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:36.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:36.152315+0000) 2022-09-08T00:23:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:36.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:36.071422+0000) 2022-09-08T00:23:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:36 smithi055 ceph-mon[97033]: pgmap v5293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:23:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:36 smithi059 ceph-mon[97450]: pgmap v5293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:23:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:37.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:37.152550+0000) 2022-09-08T00:23:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:37.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:37.071709+0000) 2022-09-08T00:23:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:38.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:38.152822+0000) 2022-09-08T00:23:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:38.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:38.071869+0000) 2022-09-08T00:23:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:39.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:39.153115+0000) 2022-09-08T00:23:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:39.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:39.072135+0000) 2022-09-08T00:23:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:39 smithi059 ceph-mon[97450]: pgmap v5294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:39 smithi055 ceph-mon[97033]: pgmap v5294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:40.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:40.153379+0000) 2022-09-08T00:23:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:40.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:40.072300+0000) 2022-09-08T00:23:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:41.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:41.153582+0000) 2022-09-08T00:23:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:41.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:41.072477+0000) 2022-09-08T00:23:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:41 smithi059 ceph-mon[97450]: pgmap v5295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:23:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:41 smithi055 ceph-mon[97033]: pgmap v5295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:23:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:42.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:42.153792+0000) 2022-09-08T00:23:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:42.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:42.072697+0000) 2022-09-08T00:23:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:43.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:43.072909+0000) 2022-09-08T00:23:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:43.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:43.154018+0000) 2022-09-08T00:23:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:43 smithi059 ceph-mon[97450]: pgmap v5296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:43 smithi055 ceph-mon[97033]: pgmap v5296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:44.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:44.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:44.073116+0000) 2022-09-08T00:23:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:44.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:44.154191+0000) 2022-09-08T00:23:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:45.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:45.154424+0000) 2022-09-08T00:23:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:45.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:45.073364+0000) 2022-09-08T00:23:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:45 smithi059 ceph-mon[97450]: pgmap v5297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:45 smithi055 ceph-mon[97033]: pgmap v5297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:46.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:46.154736+0000) 2022-09-08T00:23:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:46.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:46.073715+0000) 2022-09-08T00:23:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:46 smithi055 ceph-mon[97033]: pgmap v5298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:23:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:46 smithi059 ceph-mon[97450]: pgmap v5298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:23:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:47.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:47.155001+0000) 2022-09-08T00:23:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:47.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:47.073963+0000) 2022-09-08T00:23:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:48.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:48.155247+0000) 2022-09-08T00:23:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:48.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:48.074255+0000) 2022-09-08T00:23:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:49.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:49.155501+0000) 2022-09-08T00:23:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:49.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:49.074461+0000) 2022-09-08T00:23:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:49 smithi059 ceph-mon[97450]: pgmap v5299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:49 smithi055 ceph-mon[97033]: pgmap v5299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:50.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:50.155773+0000) 2022-09-08T00:23:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:50.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:50.074719+0000) 2022-09-08T00:23:50.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:23:50.580+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:23:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:51.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:51.155922+0000) 2022-09-08T00:23:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:51.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:51.074962+0000) 2022-09-08T00:23:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:51 smithi059 ceph-mon[97450]: pgmap v5300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:23:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:51 smithi055 ceph-mon[97033]: pgmap v5300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:23:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:52.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:52.156151+0000) 2022-09-08T00:23:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:52.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:52.075185+0000) 2022-09-08T00:23:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:53.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:53.156335+0000) 2022-09-08T00:23:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:53.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:53.075396+0000) 2022-09-08T00:23:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:53 smithi059 ceph-mon[97450]: pgmap v5301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:53 smithi055 ceph-mon[97033]: pgmap v5301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:54.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:54.156495+0000) 2022-09-08T00:23:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:54.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:54.075699+0000) 2022-09-08T00:23:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:55.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:55.156747+0000) 2022-09-08T00:23:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:55.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:55.075829+0000) 2022-09-08T00:23:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:55 smithi059 ceph-mon[97450]: pgmap v5302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:55 smithi055 ceph-mon[97033]: pgmap v5302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:56.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:56.156991+0000) 2022-09-08T00:23:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:56.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:56.076082+0000) 2022-09-08T00:23:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:56 smithi055 ceph-mon[97033]: pgmap v5303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:23:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:56 smithi059 ceph-mon[97450]: pgmap v5303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:23:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:57.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:57.157195+0000) 2022-09-08T00:23:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:57.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:57.076269+0000) 2022-09-08T00:23:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:58.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:58.157360+0000) 2022-09-08T00:23:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:58.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:58.076491+0000) 2022-09-08T00:23:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:23:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:23:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:23:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:23:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:23:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:23:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:23:59.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:59.076690+0000) 2022-09-08T00:23:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:23:59.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:23:59.157574+0000) 2022-09-08T00:23:59.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:59 smithi055 ceph-mon[97033]: pgmap v5304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:59.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:23:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:23:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:59 smithi059 ceph-mon[97450]: pgmap v5304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:23:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:23:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:24:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:00.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:00.157769+0000) 2022-09-08T00:24:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:00.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:00.076872+0000) 2022-09-08T00:24:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:01.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:01.157980+0000) 2022-09-08T00:24:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:01.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:01.077053+0000) 2022-09-08T00:24:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:01 smithi059 ceph-mon[97450]: pgmap v5305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:24:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:01 smithi055 ceph-mon[97033]: pgmap v5305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:24:02.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:02.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:02.077265+0000) 2022-09-08T00:24:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:02.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:02.158148+0000) 2022-09-08T00:24:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:03.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:03.158286+0000) 2022-09-08T00:24:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:03.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:03.077503+0000) 2022-09-08T00:24:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:03 smithi059 ceph-mon[97450]: pgmap v5306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:03 smithi055 ceph-mon[97033]: pgmap v5306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:04.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:04.158496+0000) 2022-09-08T00:24:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:04.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:04.077735+0000) 2022-09-08T00:24:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:05.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:05.158750+0000) 2022-09-08T00:24:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:05.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:05.078011+0000) 2022-09-08T00:24:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:05 smithi059 ceph-mon[97450]: pgmap v5307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:24:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:24:05.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:24:05.595+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:24:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:05 smithi055 ceph-mon[97033]: pgmap v5307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:24:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:24:06.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:06.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:06.078313+0000) 2022-09-08T00:24:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:06.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:06.158909+0000) 2022-09-08T00:24:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:06 smithi055 ceph-mon[97033]: pgmap v5308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:24:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:06 smithi059 ceph-mon[97450]: pgmap v5308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:24:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:07.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:07.159110+0000) 2022-09-08T00:24:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:07.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:07.078539+0000) 2022-09-08T00:24:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:08.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:08.159341+0000) 2022-09-08T00:24:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:08.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:08.078779+0000) 2022-09-08T00:24:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:09.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:09.159513+0000) 2022-09-08T00:24:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:09.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:09.078996+0000) 2022-09-08T00:24:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:09 smithi055 ceph-mon[97033]: osdmap e417: 12 total, 11 up, 11 in 2022-09-08T00:24:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:09 smithi055 ceph-mon[97033]: pgmap v5310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:24:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:09 smithi059 ceph-mon[97450]: osdmap e417: 12 total, 11 up, 11 in 2022-09-08T00:24:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:09 smithi059 ceph-mon[97450]: pgmap v5310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:24:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:10.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:10.159740+0000) 2022-09-08T00:24:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:10.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:10.079157+0000) 2022-09-08T00:24:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:11.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:11.159886+0000) 2022-09-08T00:24:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:11.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:11.079418+0000) 2022-09-08T00:24:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:11 smithi059 ceph-mon[97450]: pgmap v5311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:24:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:11 smithi059 ceph-mon[97450]: osdmap e418: 12 total, 11 up, 11 in 2022-09-08T00:24:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:11 smithi055 ceph-mon[97033]: pgmap v5311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:24:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:11 smithi055 ceph-mon[97033]: osdmap e418: 12 total, 11 up, 11 in 2022-09-08T00:24:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:12.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:12.160083+0000) 2022-09-08T00:24:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:12.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:12.079658+0000) 2022-09-08T00:24:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:13.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:13.160269+0000) 2022-09-08T00:24:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:13.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:13.079953+0000) 2022-09-08T00:24:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:13 smithi059 ceph-mon[97450]: pgmap v5313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:24:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:13 smithi055 ceph-mon[97033]: pgmap v5313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:24:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:14.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:14.160490+0000) 2022-09-08T00:24:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:14.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:14.080211+0000) 2022-09-08T00:24:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:15.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:15.160698+0000) 2022-09-08T00:24:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:15.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:15.080454+0000) 2022-09-08T00:24:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:15 smithi059 ceph-mon[97450]: pgmap v5314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:24:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:15 smithi055 ceph-mon[97033]: pgmap v5314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:24:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:16.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:16.160909+0000) 2022-09-08T00:24:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:16.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:16.080752+0000) 2022-09-08T00:24:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:16 smithi055 ceph-mon[97033]: pgmap v5315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-08T00:24:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:16 smithi059 ceph-mon[97450]: pgmap v5315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-08T00:24:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:17.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:17.161115+0000) 2022-09-08T00:24:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:17.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:17.081040+0000) 2022-09-08T00:24:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:18.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:18.081346+0000) 2022-09-08T00:24:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:18.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:18.161325+0000) 2022-09-08T00:24:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:19.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:19.161576+0000) 2022-09-08T00:24:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:19.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:19.081603+0000) 2022-09-08T00:24:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:19 smithi059 ceph-mon[97450]: pgmap v5316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:24:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:19 smithi055 ceph-mon[97033]: pgmap v5316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:24:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:20.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:20.161780+0000) 2022-09-08T00:24:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:20.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:20.081717+0000) 2022-09-08T00:24:20.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:24:20.598+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:24:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:21.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:21.162025+0000) 2022-09-08T00:24:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:21.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:21.081932+0000) 2022-09-08T00:24:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:21 smithi059 ceph-mon[97450]: pgmap v5317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:24:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:21 smithi055 ceph-mon[97033]: pgmap v5317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:24:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:22.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:22.162265+0000) 2022-09-08T00:24:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:22.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:22.082187+0000) 2022-09-08T00:24:23.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:23.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:23.082424+0000) 2022-09-08T00:24:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:23.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:23.162463+0000) 2022-09-08T00:24:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:23 smithi059 ceph-mon[97450]: pgmap v5318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:24:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:23 smithi055 ceph-mon[97033]: pgmap v5318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:24:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:24.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:24.162675+0000) 2022-09-08T00:24:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:24.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:24.082699+0000) 2022-09-08T00:24:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:25.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:25.162902+0000) 2022-09-08T00:24:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:25.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:25.082867+0000) 2022-09-08T00:24:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:25 smithi059 ceph-mon[97450]: pgmap v5319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:24:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:25 smithi055 ceph-mon[97033]: pgmap v5319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:24:26.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:26.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:26.163120+0000) 2022-09-08T00:24:26.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:26.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:26.083019+0000) 2022-09-08T00:24:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:26 smithi055 ceph-mon[97033]: pgmap v5320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:24:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:26 smithi059 ceph-mon[97450]: pgmap v5320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:24:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:27.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:27.163298+0000) 2022-09-08T00:24:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:27.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:27.083284+0000) 2022-09-08T00:24:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:28.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:28.163547+0000) 2022-09-08T00:24:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:28.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:28.083585+0000) 2022-09-08T00:24:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:29.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:29.163826+0000) 2022-09-08T00:24:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:29.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:29.083908+0000) 2022-09-08T00:24:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:29 smithi059 ceph-mon[97450]: pgmap v5321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:29 smithi055 ceph-mon[97033]: pgmap v5321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:30.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:30.164071+0000) 2022-09-08T00:24:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:30.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:30.084152+0000) 2022-09-08T00:24:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:31.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:31.164310+0000) 2022-09-08T00:24:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:31.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:31.084363+0000) 2022-09-08T00:24:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:31 smithi059 ceph-mon[97450]: pgmap v5322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:24:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:31 smithi055 ceph-mon[97033]: pgmap v5322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:24:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:32.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:32.164570+0000) 2022-09-08T00:24:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:32.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:32.084546+0000) 2022-09-08T00:24:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:33.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:33.164790+0000) 2022-09-08T00:24:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:33.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:33.084810+0000) 2022-09-08T00:24:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:33 smithi059 ceph-mon[97450]: pgmap v5323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:33 smithi055 ceph-mon[97033]: pgmap v5323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:34.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:34.164953+0000) 2022-09-08T00:24:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:34.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:34.085063+0000) 2022-09-08T00:24:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:35.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:35.165189+0000) 2022-09-08T00:24:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:35.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:35.085292+0000) 2022-09-08T00:24:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:35 smithi059 ceph-mon[97450]: pgmap v5324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:35.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:24:35.607+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:24:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:35 smithi055 ceph-mon[97033]: pgmap v5324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:36.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:36.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:36.085544+0000) 2022-09-08T00:24:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:36.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:36.165431+0000) 2022-09-08T00:24:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:36 smithi055 ceph-mon[97033]: pgmap v5325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:24:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:36 smithi059 ceph-mon[97450]: pgmap v5325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:24:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:37.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:37.165735+0000) 2022-09-08T00:24:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:37.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:37.085710+0000) 2022-09-08T00:24:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:38.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:38.165910+0000) 2022-09-08T00:24:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:38.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:38.085963+0000) 2022-09-08T00:24:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:39.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:39.166125+0000) 2022-09-08T00:24:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:39.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:39.086173+0000) 2022-09-08T00:24:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:39 smithi059 ceph-mon[97450]: pgmap v5326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:39 smithi055 ceph-mon[97033]: pgmap v5326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:40.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:40.166309+0000) 2022-09-08T00:24:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:40.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:40.086411+0000) 2022-09-08T00:24:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:41.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:41.166526+0000) 2022-09-08T00:24:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:41.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:41.086591+0000) 2022-09-08T00:24:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:41 smithi059 ceph-mon[97450]: pgmap v5327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:24:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:41 smithi055 ceph-mon[97033]: pgmap v5327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:24:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:42.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:42.166702+0000) 2022-09-08T00:24:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:42.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:42.086771+0000) 2022-09-08T00:24:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:43.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:43.166848+0000) 2022-09-08T00:24:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:43.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:43.087033+0000) 2022-09-08T00:24:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:43 smithi059 ceph-mon[97450]: pgmap v5328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:43 smithi055 ceph-mon[97033]: pgmap v5328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:44.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:44.167067+0000) 2022-09-08T00:24:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:44.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:44.087321+0000) 2022-09-08T00:24:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:45.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:45.167238+0000) 2022-09-08T00:24:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:45.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:45.087581+0000) 2022-09-08T00:24:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:45 smithi059 ceph-mon[97450]: pgmap v5329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:45 smithi055 ceph-mon[97033]: pgmap v5329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:46.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:46.167464+0000) 2022-09-08T00:24:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:46.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:46.087904+0000) 2022-09-08T00:24:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:46 smithi055 ceph-mon[97033]: pgmap v5330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:24:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:46 smithi059 ceph-mon[97450]: pgmap v5330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:24:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:47.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:47.167728+0000) 2022-09-08T00:24:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:47.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:47.088129+0000) 2022-09-08T00:24:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:48.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:48.088405+0000) 2022-09-08T00:24:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:48.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:48.167914+0000) 2022-09-08T00:24:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:49.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:49.088580+0000) 2022-09-08T00:24:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:49.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:49.168144+0000) 2022-09-08T00:24:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:49 smithi059 ceph-mon[97450]: pgmap v5331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:49 smithi055 ceph-mon[97033]: pgmap v5331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:50.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:50.088810+0000) 2022-09-08T00:24:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:50.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:50.168377+0000) 2022-09-08T00:24:50.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:24:50.608+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:24:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:51.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:51.168610+0000) 2022-09-08T00:24:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:51.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:51.089083+0000) 2022-09-08T00:24:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:51 smithi059 ceph-mon[97450]: pgmap v5332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:24:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:51 smithi055 ceph-mon[97033]: pgmap v5332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:24:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:52.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:52.168871+0000) 2022-09-08T00:24:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:52.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:52.089259+0000) 2022-09-08T00:24:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:53.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:53.169118+0000) 2022-09-08T00:24:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:53.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:53.089501+0000) 2022-09-08T00:24:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:53 smithi059 ceph-mon[97450]: pgmap v5333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:53 smithi055 ceph-mon[97033]: pgmap v5333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:54.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:54.169331+0000) 2022-09-08T00:24:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:54.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:54.089743+0000) 2022-09-08T00:24:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:55.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:55.169567+0000) 2022-09-08T00:24:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:55.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:55.089996+0000) 2022-09-08T00:24:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:55 smithi059 ceph-mon[97450]: pgmap v5334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:55 smithi055 ceph-mon[97033]: pgmap v5334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:56.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:56.169743+0000) 2022-09-08T00:24:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:56.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:56.090209+0000) 2022-09-08T00:24:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:56 smithi055 ceph-mon[97033]: pgmap v5335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:24:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:56 smithi059 ceph-mon[97450]: pgmap v5335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:24:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:57.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:57.090413+0000) 2022-09-08T00:24:57.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:57.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:57.169928+0000) 2022-09-08T00:24:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:58.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:58.090662+0000) 2022-09-08T00:24:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:58.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:58.170122+0000) 2022-09-08T00:24:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:24:59.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:59.090816+0000) 2022-09-08T00:24:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:24:59.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:24:59.170312+0000) 2022-09-08T00:24:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:59 smithi055 ceph-mon[97033]: pgmap v5336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:24:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:24:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:24:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:24:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:24:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:59 smithi059 ceph-mon[97450]: pgmap v5336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:24:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:24:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:24:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:24:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:24:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:25:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:00.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:00.170571+0000) 2022-09-08T00:25:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:00.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:00.091032+0000) 2022-09-08T00:25:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:01.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:01.170709+0000) 2022-09-08T00:25:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:01.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:01.091193+0000) 2022-09-08T00:25:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:01 smithi059 ceph-mon[97450]: pgmap v5337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:25:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:01 smithi055 ceph-mon[97033]: pgmap v5337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:25:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:02.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:02.091391+0000) 2022-09-08T00:25:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:02.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:02.170876+0000) 2022-09-08T00:25:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:03.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:03.091614+0000) 2022-09-08T00:25:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:03.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:03.171092+0000) 2022-09-08T00:25:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:03 smithi059 ceph-mon[97450]: pgmap v5338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:03 smithi055 ceph-mon[97033]: pgmap v5338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:04.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:04.171322+0000) 2022-09-08T00:25:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:04.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:04.091899+0000) 2022-09-08T00:25:05.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:05.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:05.092100+0000) 2022-09-08T00:25:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:05.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:05.171610+0000) 2022-09-08T00:25:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:05 smithi059 ceph-mon[97450]: pgmap v5339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:25:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:25:05.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:25:05.623+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:25:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:05 smithi055 ceph-mon[97033]: pgmap v5339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:25:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:25:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:06.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:06.171832+0000) 2022-09-08T00:25:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:06.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:06.092352+0000) 2022-09-08T00:25:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:06 smithi055 ceph-mon[97033]: pgmap v5340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:25:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:06 smithi059 ceph-mon[97450]: pgmap v5340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:25:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:07.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:07.171959+0000) 2022-09-08T00:25:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:07.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:07.092622+0000) 2022-09-08T00:25:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:08.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:08.172122+0000) 2022-09-08T00:25:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:08.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:08.092901+0000) 2022-09-08T00:25:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:09.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:09.172417+0000) 2022-09-08T00:25:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:09.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:09.093087+0000) 2022-09-08T00:25:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:09 smithi055 ceph-mon[97033]: osdmap e419: 12 total, 11 up, 11 in 2022-09-08T00:25:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:09 smithi055 ceph-mon[97033]: pgmap v5342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:25:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:09 smithi059 ceph-mon[97450]: osdmap e419: 12 total, 11 up, 11 in 2022-09-08T00:25:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:09 smithi059 ceph-mon[97450]: pgmap v5342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:25:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:10.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:10.093329+0000) 2022-09-08T00:25:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:10.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:10.172701+0000) 2022-09-08T00:25:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:11.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:11.172872+0000) 2022-09-08T00:25:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:11.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:11.093501+0000) 2022-09-08T00:25:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:11 smithi059 ceph-mon[97450]: pgmap v5343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:25:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:11 smithi059 ceph-mon[97450]: osdmap e420: 12 total, 11 up, 11 in 2022-09-08T00:25:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:11 smithi055 ceph-mon[97033]: pgmap v5343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:25:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:11 smithi055 ceph-mon[97033]: osdmap e420: 12 total, 11 up, 11 in 2022-09-08T00:25:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:12.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:12.173079+0000) 2022-09-08T00:25:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:12.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:12.093737+0000) 2022-09-08T00:25:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:13.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:13.173311+0000) 2022-09-08T00:25:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:13.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:13.093956+0000) 2022-09-08T00:25:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:13 smithi059 ceph-mon[97450]: pgmap v5345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:25:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:13 smithi055 ceph-mon[97033]: pgmap v5345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:25:14.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:14.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:14.094138+0000) 2022-09-08T00:25:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:14.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:14.173588+0000) 2022-09-08T00:25:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:15.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:15.094374+0000) 2022-09-08T00:25:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:15.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:15.173818+0000) 2022-09-08T00:25:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:15 smithi055 ceph-mon[97033]: pgmap v5346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:25:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:15 smithi059 ceph-mon[97450]: pgmap v5346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:25:16.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:16.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:16.094585+0000) 2022-09-08T00:25:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:16.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:16.174008+0000) 2022-09-08T00:25:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:16 smithi055 ceph-mon[97033]: pgmap v5347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-08T00:25:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:16 smithi059 ceph-mon[97450]: pgmap v5347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-08T00:25:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:17.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:17.174221+0000) 2022-09-08T00:25:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:17.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:17.094814+0000) 2022-09-08T00:25:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:18.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:18.174431+0000) 2022-09-08T00:25:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:18.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:18.094972+0000) 2022-09-08T00:25:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:19.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:19.174657+0000) 2022-09-08T00:25:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:19.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:19.095186+0000) 2022-09-08T00:25:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:19 smithi059 ceph-mon[97450]: pgmap v5348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:25:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:19 smithi055 ceph-mon[97033]: pgmap v5348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:25:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:20.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:20.174836+0000) 2022-09-08T00:25:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:20.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:20.095405+0000) 2022-09-08T00:25:20.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:25:20.639+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:25:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:21.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:21.175023+0000) 2022-09-08T00:25:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:21.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:21.095629+0000) 2022-09-08T00:25:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:21 smithi059 ceph-mon[97450]: pgmap v5349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:25:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:21 smithi055 ceph-mon[97033]: pgmap v5349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:25:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:22.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:22.175186+0000) 2022-09-08T00:25:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:22.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:22.095870+0000) 2022-09-08T00:25:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:23.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:23.175385+0000) 2022-09-08T00:25:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:23.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:23.096112+0000) 2022-09-08T00:25:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:23 smithi059 ceph-mon[97450]: pgmap v5350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:25:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:23 smithi055 ceph-mon[97033]: pgmap v5350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:25:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:24.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:24.175592+0000) 2022-09-08T00:25:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:24.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:24.096319+0000) 2022-09-08T00:25:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:25.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:25.175806+0000) 2022-09-08T00:25:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:25.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:25.096570+0000) 2022-09-08T00:25:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:25 smithi055 ceph-mon[97033]: pgmap v5351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:25:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:25 smithi059 ceph-mon[97450]: pgmap v5351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:25:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:26.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:26.175981+0000) 2022-09-08T00:25:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:26.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:26.096804+0000) 2022-09-08T00:25:26.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:26 smithi055 ceph-mon[97033]: pgmap v5352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:25:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:26 smithi059 ceph-mon[97450]: pgmap v5352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:25:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:27.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:27.176156+0000) 2022-09-08T00:25:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:27.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:27.097002+0000) 2022-09-08T00:25:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:28.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:28.176394+0000) 2022-09-08T00:25:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:28.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:28.097134+0000) 2022-09-08T00:25:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:29.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:29.176618+0000) 2022-09-08T00:25:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:29.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:29.097399+0000) 2022-09-08T00:25:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:29 smithi059 ceph-mon[97450]: pgmap v5353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:29 smithi055 ceph-mon[97033]: pgmap v5353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:30.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:30.176905+0000) 2022-09-08T00:25:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:30.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:30.097714+0000) 2022-09-08T00:25:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:31.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:31.177125+0000) 2022-09-08T00:25:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:31.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:31.097913+0000) 2022-09-08T00:25:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:31 smithi059 ceph-mon[97450]: pgmap v5354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:25:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:31 smithi055 ceph-mon[97033]: pgmap v5354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:25:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:32.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:32.177369+0000) 2022-09-08T00:25:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:32.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:32.098137+0000) 2022-09-08T00:25:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:33.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:33.177589+0000) 2022-09-08T00:25:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:33.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:33.098375+0000) 2022-09-08T00:25:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:33 smithi055 ceph-mon[97033]: pgmap v5355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:33 smithi059 ceph-mon[97450]: pgmap v5355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:34.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:34.177820+0000) 2022-09-08T00:25:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:34.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:34.098627+0000) 2022-09-08T00:25:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:35.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:35.177969+0000) 2022-09-08T00:25:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:35.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:35.098844+0000) 2022-09-08T00:25:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:35 smithi055 ceph-mon[97033]: pgmap v5356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:35 smithi059 ceph-mon[97450]: pgmap v5356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:36.099 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:25:35.652+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:25:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:36.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:36.178191+0000) 2022-09-08T00:25:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:36.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:36.099093+0000) 2022-09-08T00:25:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:36 smithi055 ceph-mon[97033]: pgmap v5357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:25:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:36 smithi059 ceph-mon[97450]: pgmap v5357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:25:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:37.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:37.099381+0000) 2022-09-08T00:25:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:37.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:37.178378+0000) 2022-09-08T00:25:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:38.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:38.178664+0000) 2022-09-08T00:25:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:38.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:38.099546+0000) 2022-09-08T00:25:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:39.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:39.178890+0000) 2022-09-08T00:25:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:39.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:39.099685+0000) 2022-09-08T00:25:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:39 smithi059 ceph-mon[97450]: pgmap v5358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:39 smithi055 ceph-mon[97033]: pgmap v5358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:40.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:40.179110+0000) 2022-09-08T00:25:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:40.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:40.099920+0000) 2022-09-08T00:25:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:41.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:41.179335+0000) 2022-09-08T00:25:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:41.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:41.100144+0000) 2022-09-08T00:25:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:41 smithi055 ceph-mon[97033]: pgmap v5359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:25:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:41 smithi059 ceph-mon[97450]: pgmap v5359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:25:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:42.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:42.179553+0000) 2022-09-08T00:25:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:42.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:42.100359+0000) 2022-09-08T00:25:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:43.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:43.179776+0000) 2022-09-08T00:25:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:43.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:43.100589+0000) 2022-09-08T00:25:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:43 smithi055 ceph-mon[97033]: pgmap v5360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:43 smithi059 ceph-mon[97450]: pgmap v5360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:44.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:44.179993+0000) 2022-09-08T00:25:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:44.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:44.100811+0000) 2022-09-08T00:25:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:45.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:45.180184+0000) 2022-09-08T00:25:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:45.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:45.101047+0000) 2022-09-08T00:25:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:45 smithi055 ceph-mon[97033]: pgmap v5361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:45 smithi059 ceph-mon[97450]: pgmap v5361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:46.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:46.180363+0000) 2022-09-08T00:25:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:46.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:46.101270+0000) 2022-09-08T00:25:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:46 smithi055 ceph-mon[97033]: pgmap v5362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:25:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:46 smithi059 ceph-mon[97450]: pgmap v5362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:25:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:47.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:47.180589+0000) 2022-09-08T00:25:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:47.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:47.101484+0000) 2022-09-08T00:25:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:48.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:48.101728+0000) 2022-09-08T00:25:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:48.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:48.180777+0000) 2022-09-08T00:25:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:49.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:49.181010+0000) 2022-09-08T00:25:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:49.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:49.101913+0000) 2022-09-08T00:25:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:49 smithi055 ceph-mon[97033]: pgmap v5363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:49 smithi059 ceph-mon[97450]: pgmap v5363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:50.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:50.181185+0000) 2022-09-08T00:25:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:50.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:50.102137+0000) 2022-09-08T00:25:51.104 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:25:50.669+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:25:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:51.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:51.181365+0000) 2022-09-08T00:25:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:51.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:51.102371+0000) 2022-09-08T00:25:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:51 smithi055 ceph-mon[97033]: pgmap v5364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:25:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:51 smithi059 ceph-mon[97450]: pgmap v5364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:25:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:52.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:52.181553+0000) 2022-09-08T00:25:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:52.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:52.102604+0000) 2022-09-08T00:25:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:53.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:53.181800+0000) 2022-09-08T00:25:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:53.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:53.102782+0000) 2022-09-08T00:25:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:53 smithi055 ceph-mon[97033]: pgmap v5365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:53 smithi059 ceph-mon[97450]: pgmap v5365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:54.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:54.182031+0000) 2022-09-08T00:25:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:54.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:54.102999+0000) 2022-09-08T00:25:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:55.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:55.182210+0000) 2022-09-08T00:25:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:55.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:55.103235+0000) 2022-09-08T00:25:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:55 smithi055 ceph-mon[97033]: pgmap v5366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:55 smithi059 ceph-mon[97450]: pgmap v5366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:56.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:56.182489+0000) 2022-09-08T00:25:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:56.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:56.103468+0000) 2022-09-08T00:25:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:56 smithi055 ceph-mon[97033]: pgmap v5367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:25:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:56 smithi059 ceph-mon[97450]: pgmap v5367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:25:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:57.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:57.182695+0000) 2022-09-08T00:25:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:57.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:57.103730+0000) 2022-09-08T00:25:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:58.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:58.182922+0000) 2022-09-08T00:25:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:58.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:58.103977+0000) 2022-09-08T00:25:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:25:59.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:59.183176+0000) 2022-09-08T00:25:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:25:59.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:25:59.104246+0000) 2022-09-08T00:25:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:59 smithi055 ceph-mon[97033]: pgmap v5368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:25:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:25:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:25:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:25:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:59 smithi059 ceph-mon[97450]: pgmap v5368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:25:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:25:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:25:59.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:25:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:26:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:00.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:00.183359+0000) 2022-09-08T00:26:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:00.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:00.104465+0000) 2022-09-08T00:26:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:26:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:00 smithi055 ceph-mon[97033]: pgmap v5369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:26:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:26:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:00 smithi059 ceph-mon[97450]: pgmap v5369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:26:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:01.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:01.183586+0000) 2022-09-08T00:26:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:01.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:01.104612+0000) 2022-09-08T00:26:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:02.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:02.183788+0000) 2022-09-08T00:26:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:02.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:02.104793+0000) 2022-09-08T00:26:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:03.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:03.184019+0000) 2022-09-08T00:26:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:03.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:03.105035+0000) 2022-09-08T00:26:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:03 smithi055 ceph-mon[97033]: pgmap v5370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:03 smithi059 ceph-mon[97450]: pgmap v5370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:04.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:04.184259+0000) 2022-09-08T00:26:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:04.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:04.105277+0000) 2022-09-08T00:26:05.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:05.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:05.184457+0000) 2022-09-08T00:26:05.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:05.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:05.105493+0000) 2022-09-08T00:26:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:05 smithi055 ceph-mon[97033]: pgmap v5371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:26:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:26:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:05 smithi059 ceph-mon[97450]: pgmap v5371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:26:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:26:06.106 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:26:05.682+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:26:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:06.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:06.105679+0000) 2022-09-08T00:26:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:06.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:06.184613+0000) 2022-09-08T00:26:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:06 smithi055 ceph-mon[97033]: pgmap v5372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:26:07.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:06 smithi059 ceph-mon[97450]: pgmap v5372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:26:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:07.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:07.184767+0000) 2022-09-08T00:26:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:07.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:07.105908+0000) 2022-09-08T00:26:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:08.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:08.106138+0000) 2022-09-08T00:26:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:08.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:08.184991+0000) 2022-09-08T00:26:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:09.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:09.106344+0000) 2022-09-08T00:26:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:09.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:09.185204+0000) 2022-09-08T00:26:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:09 smithi055 ceph-mon[97033]: osdmap e421: 12 total, 11 up, 11 in 2022-09-08T00:26:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:09 smithi055 ceph-mon[97033]: pgmap v5374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:26:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:09 smithi059 ceph-mon[97450]: osdmap e421: 12 total, 11 up, 11 in 2022-09-08T00:26:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:09 smithi059 ceph-mon[97450]: pgmap v5374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:26:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:10.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:10.185447+0000) 2022-09-08T00:26:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:10.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:10.106561+0000) 2022-09-08T00:26:11.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:11.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:11.185613+0000) 2022-09-08T00:26:11.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:11.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:11.106734+0000) 2022-09-08T00:26:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:11 smithi055 ceph-mon[97033]: pgmap v5375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:26:11.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:11 smithi055 ceph-mon[97033]: osdmap e422: 12 total, 11 up, 11 in 2022-09-08T00:26:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:11 smithi059 ceph-mon[97450]: pgmap v5375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:26:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:11 smithi059 ceph-mon[97450]: osdmap e422: 12 total, 11 up, 11 in 2022-09-08T00:26:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:12.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:12.106947+0000) 2022-09-08T00:26:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:12.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:12.185792+0000) 2022-09-08T00:26:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:13.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:13.107172+0000) 2022-09-08T00:26:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:13.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:13.185923+0000) 2022-09-08T00:26:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:13 smithi055 ceph-mon[97033]: pgmap v5377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:26:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:13 smithi059 ceph-mon[97450]: pgmap v5377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:26:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:14.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:14.107467+0000) 2022-09-08T00:26:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:14.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:14.186139+0000) 2022-09-08T00:26:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:15.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:15.107681+0000) 2022-09-08T00:26:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:15.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:15.186361+0000) 2022-09-08T00:26:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:15 smithi055 ceph-mon[97033]: pgmap v5378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:26:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:15 smithi059 ceph-mon[97450]: pgmap v5378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:26:16.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:16.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:16.186578+0000) 2022-09-08T00:26:16.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:16.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:16.107849+0000) 2022-09-08T00:26:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:16 smithi055 ceph-mon[97033]: pgmap v5379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T00:26:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:16 smithi059 ceph-mon[97450]: pgmap v5379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T00:26:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:17.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:17.186836+0000) 2022-09-08T00:26:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:17.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:17.108089+0000) 2022-09-08T00:26:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:18.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:18.187090+0000) 2022-09-08T00:26:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:18.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:18.108323+0000) 2022-09-08T00:26:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:19.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:19.187369+0000) 2022-09-08T00:26:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:19.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:19.108578+0000) 2022-09-08T00:26:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:19 smithi055 ceph-mon[97033]: pgmap v5380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:26:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:19 smithi059 ceph-mon[97450]: pgmap v5380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:26:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:20.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:20.108808+0000) 2022-09-08T00:26:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:20.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:20.187693+0000) 2022-09-08T00:26:21.110 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:26:20.683+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:26:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:21.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:21.187837+0000) 2022-09-08T00:26:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:21.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:21.109030+0000) 2022-09-08T00:26:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:21 smithi055 ceph-mon[97033]: pgmap v5381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:26:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:21 smithi059 ceph-mon[97450]: pgmap v5381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:26:22.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:22.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:22.188042+0000) 2022-09-08T00:26:22.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:22.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:22.109245+0000) 2022-09-08T00:26:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:23.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:23.109448+0000) 2022-09-08T00:26:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:23.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:23.188222+0000) 2022-09-08T00:26:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:23 smithi055 ceph-mon[97033]: pgmap v5382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:26:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:23 smithi059 ceph-mon[97450]: pgmap v5382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:26:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:24.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:24.188464+0000) 2022-09-08T00:26:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:24.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:24.109620+0000) 2022-09-08T00:26:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:25.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:25.188733+0000) 2022-09-08T00:26:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:25.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:25.109857+0000) 2022-09-08T00:26:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:25 smithi055 ceph-mon[97033]: pgmap v5383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:26:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:25 smithi059 ceph-mon[97450]: pgmap v5383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:26:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:26.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:26.188963+0000) 2022-09-08T00:26:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:26.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:26.110100+0000) 2022-09-08T00:26:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:26 smithi055 ceph-mon[97033]: pgmap v5384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:26:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:26 smithi059 ceph-mon[97450]: pgmap v5384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:26:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:27.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:27.189119+0000) 2022-09-08T00:26:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:27.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:27.110295+0000) 2022-09-08T00:26:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:28.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:28.189330+0000) 2022-09-08T00:26:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:28.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:28.110458+0000) 2022-09-08T00:26:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:29.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:29.189575+0000) 2022-09-08T00:26:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:29.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:29.110721+0000) 2022-09-08T00:26:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:29 smithi055 ceph-mon[97033]: pgmap v5385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:29 smithi059 ceph-mon[97450]: pgmap v5385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:30.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:30.110942+0000) 2022-09-08T00:26:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:30.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:30.189815+0000) 2022-09-08T00:26:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:31.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:31.111114+0000) 2022-09-08T00:26:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:31.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:31.190005+0000) 2022-09-08T00:26:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:31 smithi055 ceph-mon[97033]: pgmap v5386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:26:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:31 smithi059 ceph-mon[97450]: pgmap v5386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:26:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:32.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:32.190177+0000) 2022-09-08T00:26:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:32.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:32.111308+0000) 2022-09-08T00:26:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:33.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:33.190399+0000) 2022-09-08T00:26:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:33.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:33.111568+0000) 2022-09-08T00:26:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:33 smithi055 ceph-mon[97033]: pgmap v5387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:33 smithi059 ceph-mon[97450]: pgmap v5387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:34.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:34.111782+0000) 2022-09-08T00:26:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:34.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:34.190585+0000) 2022-09-08T00:26:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:35.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:35.190813+0000) 2022-09-08T00:26:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:35.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:35.112025+0000) 2022-09-08T00:26:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:35 smithi055 ceph-mon[97033]: pgmap v5388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:35 smithi059 ceph-mon[97450]: pgmap v5388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:36.113 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:26:35.699+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:26:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:36.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:36.112261+0000) 2022-09-08T00:26:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:36.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:36.190973+0000) 2022-09-08T00:26:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:36 smithi055 ceph-mon[97033]: pgmap v5389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:26:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:36 smithi059 ceph-mon[97450]: pgmap v5389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:26:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:37.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:37.191236+0000) 2022-09-08T00:26:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:37.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:37.112511+0000) 2022-09-08T00:26:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:38.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:38.191448+0000) 2022-09-08T00:26:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:38.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:38.112744+0000) 2022-09-08T00:26:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:39.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:39.191677+0000) 2022-09-08T00:26:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:39.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:39.112969+0000) 2022-09-08T00:26:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:39 smithi055 ceph-mon[97033]: pgmap v5390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:39 smithi059 ceph-mon[97450]: pgmap v5390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:40.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:40.191804+0000) 2022-09-08T00:26:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:40.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:40.113204+0000) 2022-09-08T00:26:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:41.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:41.192022+0000) 2022-09-08T00:26:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:41.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:41.113404+0000) 2022-09-08T00:26:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:41 smithi055 ceph-mon[97033]: pgmap v5391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:26:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:41 smithi059 ceph-mon[97450]: pgmap v5391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:26:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:42.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:42.192235+0000) 2022-09-08T00:26:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:42.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:42.113597+0000) 2022-09-08T00:26:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:43.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:43.192412+0000) 2022-09-08T00:26:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:43.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:43.113811+0000) 2022-09-08T00:26:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:43 smithi055 ceph-mon[97033]: pgmap v5392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:43 smithi059 ceph-mon[97450]: pgmap v5392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:44.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:44.192690+0000) 2022-09-08T00:26:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:44.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:44.114088+0000) 2022-09-08T00:26:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:45.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:45.114316+0000) 2022-09-08T00:26:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:45.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:45.192837+0000) 2022-09-08T00:26:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:45 smithi055 ceph-mon[97033]: pgmap v5393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:45 smithi059 ceph-mon[97450]: pgmap v5393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:46.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:46.193076+0000) 2022-09-08T00:26:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:46.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:46.114556+0000) 2022-09-08T00:26:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:46 smithi055 ceph-mon[97033]: pgmap v5394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:26:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:46 smithi059 ceph-mon[97450]: pgmap v5394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:26:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:47.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:47.193306+0000) 2022-09-08T00:26:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:47.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:47.114808+0000) 2022-09-08T00:26:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:48.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:48.193536+0000) 2022-09-08T00:26:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:48.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:48.115046+0000) 2022-09-08T00:26:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:49.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:49.193773+0000) 2022-09-08T00:26:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:49.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:49.115302+0000) 2022-09-08T00:26:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:49 smithi055 ceph-mon[97033]: pgmap v5395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:49 smithi059 ceph-mon[97450]: pgmap v5395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:50.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:50.194018+0000) 2022-09-08T00:26:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:50.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:50.115551+0000) 2022-09-08T00:26:51.117 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:26:50.714+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:26:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:51.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:51.194210+0000) 2022-09-08T00:26:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:51.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:51.115783+0000) 2022-09-08T00:26:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:51 smithi055 ceph-mon[97033]: pgmap v5396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:26:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:51 smithi059 ceph-mon[97450]: pgmap v5396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:26:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:52.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:52.194427+0000) 2022-09-08T00:26:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:52.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:52.115937+0000) 2022-09-08T00:26:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:53.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:53.194660+0000) 2022-09-08T00:26:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:53.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:53.116099+0000) 2022-09-08T00:26:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:53 smithi055 ceph-mon[97033]: pgmap v5397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:53 smithi059 ceph-mon[97450]: pgmap v5397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:54.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:54.194948+0000) 2022-09-08T00:26:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:54.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:54.116262+0000) 2022-09-08T00:26:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:55.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:55.195180+0000) 2022-09-08T00:26:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:55.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:55.116475+0000) 2022-09-08T00:26:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:55 smithi055 ceph-mon[97033]: pgmap v5398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:55 smithi059 ceph-mon[97450]: pgmap v5398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:26:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:56.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:56.195383+0000) 2022-09-08T00:26:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:56.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:56.116685+0000) 2022-09-08T00:26:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:56 smithi055 ceph-mon[97033]: pgmap v5399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:26:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:56 smithi059 ceph-mon[97450]: pgmap v5399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:26:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:57.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:57.195597+0000) 2022-09-08T00:26:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:57.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:57.116930+0000) 2022-09-08T00:26:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:58.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:58.195756+0000) 2022-09-08T00:26:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:58.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:58.117154+0000) 2022-09-08T00:26:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:26:59.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:59.196002+0000) 2022-09-08T00:26:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:26:59.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:26:59.117373+0000) 2022-09-08T00:26:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:26:59 smithi055 ceph-mon[97033]: pgmap v5400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:26:59 smithi059 ceph-mon[97450]: pgmap v5400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:00.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:00.196188+0000) 2022-09-08T00:27:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:00.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:00.117568+0000) 2022-09-08T00:27:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:27:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:27:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:27:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:27:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:27:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:27:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:27:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:27:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:01.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:01.117784+0000) 2022-09-08T00:27:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:01.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:01.196419+0000) 2022-09-08T00:27:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:01 smithi055 ceph-mon[97033]: pgmap v5401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:27:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:01 smithi059 ceph-mon[97450]: pgmap v5401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:27:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:02.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:02.196675+0000) 2022-09-08T00:27:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:02.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:02.117984+0000) 2022-09-08T00:27:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:03.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:03.196884+0000) 2022-09-08T00:27:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:03.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:03.118204+0000) 2022-09-08T00:27:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:03 smithi055 ceph-mon[97033]: pgmap v5402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:03 smithi059 ceph-mon[97450]: pgmap v5402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:04.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:04.197080+0000) 2022-09-08T00:27:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:04.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:04.118457+0000) 2022-09-08T00:27:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:05.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:05.197307+0000) 2022-09-08T00:27:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:05.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:05.118728+0000) 2022-09-08T00:27:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:05 smithi055 ceph-mon[97033]: pgmap v5403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:27:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:27:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:05 smithi059 ceph-mon[97450]: pgmap v5403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:27:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:27:06.119 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:27:05.730+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:27:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:06.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:06.197565+0000) 2022-09-08T00:27:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:06.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:06.118985+0000) 2022-09-08T00:27:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:06 smithi055 ceph-mon[97033]: pgmap v5404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:27:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:06 smithi059 ceph-mon[97450]: pgmap v5404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:27:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:07.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:07.197825+0000) 2022-09-08T00:27:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:07.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:07.119200+0000) 2022-09-08T00:27:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:08.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:08.198084+0000) 2022-09-08T00:27:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:08.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:08.119420+0000) 2022-09-08T00:27:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:09.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:09.198264+0000) 2022-09-08T00:27:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:09.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:09.119646+0000) 2022-09-08T00:27:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:09 smithi055 ceph-mon[97033]: osdmap e423: 12 total, 11 up, 11 in 2022-09-08T00:27:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:09 smithi055 ceph-mon[97033]: pgmap v5406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:27:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:09 smithi059 ceph-mon[97450]: osdmap e423: 12 total, 11 up, 11 in 2022-09-08T00:27:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:09 smithi059 ceph-mon[97450]: pgmap v5406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:27:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:10.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:10.198469+0000) 2022-09-08T00:27:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:10.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:10.119868+0000) 2022-09-08T00:27:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:11.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:11.198698+0000) 2022-09-08T00:27:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:11.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:11.120098+0000) 2022-09-08T00:27:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:11 smithi055 ceph-mon[97033]: pgmap v5407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:27:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:11 smithi055 ceph-mon[97033]: osdmap e424: 12 total, 11 up, 11 in 2022-09-08T00:27:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:11 smithi059 ceph-mon[97450]: pgmap v5407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:27:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:11 smithi059 ceph-mon[97450]: osdmap e424: 12 total, 11 up, 11 in 2022-09-08T00:27:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:12.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:12.198949+0000) 2022-09-08T00:27:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:12.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:12.120333+0000) 2022-09-08T00:27:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:13.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:13.199176+0000) 2022-09-08T00:27:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:13.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:13.120516+0000) 2022-09-08T00:27:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:13 smithi055 ceph-mon[97033]: pgmap v5409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T00:27:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:13 smithi059 ceph-mon[97450]: pgmap v5409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T00:27:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:14.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:14.199403+0000) 2022-09-08T00:27:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:14.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:14.120752+0000) 2022-09-08T00:27:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:15.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:15.199663+0000) 2022-09-08T00:27:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:15.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:15.121004+0000) 2022-09-08T00:27:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:15 smithi055 ceph-mon[97033]: pgmap v5410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:27:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:15 smithi059 ceph-mon[97450]: pgmap v5410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:27:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:16.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:16.199860+0000) 2022-09-08T00:27:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:16.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:16.121263+0000) 2022-09-08T00:27:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:16 smithi055 ceph-mon[97033]: pgmap v5411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T00:27:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:16 smithi059 ceph-mon[97450]: pgmap v5411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T00:27:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:17.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:17.200093+0000) 2022-09-08T00:27:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:17.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:17.121488+0000) 2022-09-08T00:27:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:18.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:18.200323+0000) 2022-09-08T00:27:18.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:18.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:18.121719+0000) 2022-09-08T00:27:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:19.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:19.121953+0000) 2022-09-08T00:27:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:19.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:19.200554+0000) 2022-09-08T00:27:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:19 smithi055 ceph-mon[97033]: pgmap v5412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:27:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:19 smithi059 ceph-mon[97450]: pgmap v5412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:27:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:20.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:20.200809+0000) 2022-09-08T00:27:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:20.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:20.122149+0000) 2022-09-08T00:27:21.123 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:27:20.737+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:27:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:21.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:21.122359+0000) 2022-09-08T00:27:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:21.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:21.200981+0000) 2022-09-08T00:27:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:21 smithi055 ceph-mon[97033]: pgmap v5413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:27:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:21 smithi059 ceph-mon[97450]: pgmap v5413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:27:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:22.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:22.201160+0000) 2022-09-08T00:27:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:22.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:22.122576+0000) 2022-09-08T00:27:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:23.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:23.201398+0000) 2022-09-08T00:27:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:23.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:23.122793+0000) 2022-09-08T00:27:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:23 smithi055 ceph-mon[97033]: pgmap v5414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:27:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:23 smithi059 ceph-mon[97450]: pgmap v5414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:27:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:24.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:24.201676+0000) 2022-09-08T00:27:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:24.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:24.123029+0000) 2022-09-08T00:27:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:25.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:25.201860+0000) 2022-09-08T00:27:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:25.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:25.123270+0000) 2022-09-08T00:27:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:25 smithi055 ceph-mon[97033]: pgmap v5415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:27:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:25 smithi059 ceph-mon[97450]: pgmap v5415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:27:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:26.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:26.202052+0000) 2022-09-08T00:27:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:26.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:26.123493+0000) 2022-09-08T00:27:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:26 smithi055 ceph-mon[97033]: pgmap v5416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:27:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:26 smithi059 ceph-mon[97450]: pgmap v5416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:27:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:27.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:27.202286+0000) 2022-09-08T00:27:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:27.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:27.123730+0000) 2022-09-08T00:27:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:28.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:28.202485+0000) 2022-09-08T00:27:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:28.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:28.123968+0000) 2022-09-08T00:27:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:29.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:29.202765+0000) 2022-09-08T00:27:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:29.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:29.124234+0000) 2022-09-08T00:27:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:29 smithi055 ceph-mon[97033]: pgmap v5417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:29 smithi059 ceph-mon[97450]: pgmap v5417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:30.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:30.124437+0000) 2022-09-08T00:27:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:30.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:30.202941+0000) 2022-09-08T00:27:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:31.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:31.124735+0000) 2022-09-08T00:27:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:31.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:31.203128+0000) 2022-09-08T00:27:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:31 smithi055 ceph-mon[97033]: pgmap v5418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:27:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:31 smithi059 ceph-mon[97450]: pgmap v5418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:27:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:32.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:32.124981+0000) 2022-09-08T00:27:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:32.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:32.203361+0000) 2022-09-08T00:27:33.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:33.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:33.125252+0000) 2022-09-08T00:27:33.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:33.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:33.203587+0000) 2022-09-08T00:27:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:33 smithi055 ceph-mon[97033]: pgmap v5419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:33 smithi059 ceph-mon[97450]: pgmap v5419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:34.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:34.125454+0000) 2022-09-08T00:27:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:34.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:34.203796+0000) 2022-09-08T00:27:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:35.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:35.125727+0000) 2022-09-08T00:27:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:35.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:35.203919+0000) 2022-09-08T00:27:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:35 smithi055 ceph-mon[97033]: pgmap v5420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:35 smithi059 ceph-mon[97450]: pgmap v5420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:36.126 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:27:35.741+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:27:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:36.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:36.204078+0000) 2022-09-08T00:27:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:36.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:36.125904+0000) 2022-09-08T00:27:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:36 smithi055 ceph-mon[97033]: pgmap v5421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:27:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:36 smithi059 ceph-mon[97450]: pgmap v5421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:27:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:37.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:37.204288+0000) 2022-09-08T00:27:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:37.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:37.126106+0000) 2022-09-08T00:27:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:38.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:38.204474+0000) 2022-09-08T00:27:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:38.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:38.126303+0000) 2022-09-08T00:27:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:39.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:39.204752+0000) 2022-09-08T00:27:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:39.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:39.126497+0000) 2022-09-08T00:27:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:39 smithi055 ceph-mon[97033]: pgmap v5422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:39 smithi059 ceph-mon[97450]: pgmap v5422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:40.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:40.205038+0000) 2022-09-08T00:27:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:40.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:40.126709+0000) 2022-09-08T00:27:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:41.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:41.205223+0000) 2022-09-08T00:27:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:41.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:41.126869+0000) 2022-09-08T00:27:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:41 smithi055 ceph-mon[97033]: pgmap v5423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:27:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:41 smithi059 ceph-mon[97450]: pgmap v5423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:27:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:42.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:42.205430+0000) 2022-09-08T00:27:42.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:42.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:42.127079+0000) 2022-09-08T00:27:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:43.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:43.127233+0000) 2022-09-08T00:27:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:43.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:43.205720+0000) 2022-09-08T00:27:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:43 smithi055 ceph-mon[97033]: pgmap v5424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:43 smithi059 ceph-mon[97450]: pgmap v5424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:44.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:44.127475+0000) 2022-09-08T00:27:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:44.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:44.205942+0000) 2022-09-08T00:27:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:45.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:45.127687+0000) 2022-09-08T00:27:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:45.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:45.206175+0000) 2022-09-08T00:27:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:45 smithi055 ceph-mon[97033]: pgmap v5425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:45 smithi059 ceph-mon[97450]: pgmap v5425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:46.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:46.127857+0000) 2022-09-08T00:27:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:46.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:46.206397+0000) 2022-09-08T00:27:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:46 smithi055 ceph-mon[97033]: pgmap v5426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:27:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:46 smithi059 ceph-mon[97450]: pgmap v5426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:27:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:47.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:47.128079+0000) 2022-09-08T00:27:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:47.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:47.206656+0000) 2022-09-08T00:27:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:48.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:48.128343+0000) 2022-09-08T00:27:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:48.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:48.206824+0000) 2022-09-08T00:27:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:49.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:49.206987+0000) 2022-09-08T00:27:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:49.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:49.128584+0000) 2022-09-08T00:27:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:49 smithi055 ceph-mon[97033]: pgmap v5427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:49 smithi059 ceph-mon[97450]: pgmap v5427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:50.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:50.207114+0000) 2022-09-08T00:27:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:50.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:50.128771+0000) 2022-09-08T00:27:51.130 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:27:50.751+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:27:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:51.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:51.207268+0000) 2022-09-08T00:27:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:51.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:51.128992+0000) 2022-09-08T00:27:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:51 smithi055 ceph-mon[97033]: pgmap v5428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:27:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:51 smithi059 ceph-mon[97450]: pgmap v5428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:27:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:52.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:52.207466+0000) 2022-09-08T00:27:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:52.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:52.129201+0000) 2022-09-08T00:27:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:53.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:53.129450+0000) 2022-09-08T00:27:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:53.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:53.207729+0000) 2022-09-08T00:27:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:53 smithi055 ceph-mon[97033]: pgmap v5429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:53 smithi059 ceph-mon[97450]: pgmap v5429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:54.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:54.207961+0000) 2022-09-08T00:27:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:54.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:54.129763+0000) 2022-09-08T00:27:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:55.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:55.208203+0000) 2022-09-08T00:27:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:55.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:55.129896+0000) 2022-09-08T00:27:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:55 smithi055 ceph-mon[97033]: pgmap v5430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:55 smithi059 ceph-mon[97450]: pgmap v5430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:27:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:56.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:56.208394+0000) 2022-09-08T00:27:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:56.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:56.130111+0000) 2022-09-08T00:27:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:56 smithi059 ceph-mon[97450]: pgmap v5431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:27:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:56 smithi055 ceph-mon[97033]: pgmap v5431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:27:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:57.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:57.208630+0000) 2022-09-08T00:27:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:57.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:57.130325+0000) 2022-09-08T00:27:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:58.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:58.208820+0000) 2022-09-08T00:27:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:58.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:58.130588+0000) 2022-09-08T00:27:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:27:59.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:59.209053+0000) 2022-09-08T00:27:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:27:59.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:27:59.130741+0000) 2022-09-08T00:27:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:27:59 smithi055 ceph-mon[97033]: pgmap v5432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:27:59 smithi059 ceph-mon[97450]: pgmap v5432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:00.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:00.209273+0000) 2022-09-08T00:28:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:00.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:00.130898+0000) 2022-09-08T00:28:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:01.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:01.209480+0000) 2022-09-08T00:28:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:01.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:01.131036+0000) 2022-09-08T00:28:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:01 smithi055 ceph-mon[97033]: pgmap v5433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:28:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:28:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:28:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:28:01.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:28:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:01 smithi059 ceph-mon[97450]: pgmap v5433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:28:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:28:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:28:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:28:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:28:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:02.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:02.209693+0000) 2022-09-08T00:28:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:02.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:02.131184+0000) 2022-09-08T00:28:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:03.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:03.209814+0000) 2022-09-08T00:28:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:03.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:03.131435+0000) 2022-09-08T00:28:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:03 smithi055 ceph-mon[97033]: pgmap v5434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:03 smithi059 ceph-mon[97450]: pgmap v5434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:04.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:04.210053+0000) 2022-09-08T00:28:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:04.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:04.131691+0000) 2022-09-08T00:28:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:05.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:05.210310+0000) 2022-09-08T00:28:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:05.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:05.131896+0000) 2022-09-08T00:28:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:05 smithi055 ceph-mon[97033]: pgmap v5435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:28:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:28:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:05 smithi059 ceph-mon[97450]: pgmap v5435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:28:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:28:06.132 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:28:05.765+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:28:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:06.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:06.210561+0000) 2022-09-08T00:28:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:06.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:06.132103+0000) 2022-09-08T00:28:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:06 smithi059 ceph-mon[97450]: pgmap v5436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:28:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:06 smithi055 ceph-mon[97033]: pgmap v5436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:28:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:07.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:07.210806+0000) 2022-09-08T00:28:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:07.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:07.132249+0000) 2022-09-08T00:28:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:08.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:08.210943+0000) 2022-09-08T00:28:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:08.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:08.132473+0000) 2022-09-08T00:28:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:09.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:09.211137+0000) 2022-09-08T00:28:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:09.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:09.132739+0000) 2022-09-08T00:28:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:09 smithi055 ceph-mon[97033]: osdmap e425: 12 total, 11 up, 11 in 2022-09-08T00:28:09.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:09 smithi055 ceph-mon[97033]: pgmap v5438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:28:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:09 smithi059 ceph-mon[97450]: osdmap e425: 12 total, 11 up, 11 in 2022-09-08T00:28:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:09 smithi059 ceph-mon[97450]: pgmap v5438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:28:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:10.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:10.211367+0000) 2022-09-08T00:28:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:10.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:10.132874+0000) 2022-09-08T00:28:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:11.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:11.211612+0000) 2022-09-08T00:28:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:11.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:11.133057+0000) 2022-09-08T00:28:11.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:11 smithi055 ceph-mon[97033]: pgmap v5439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:28:11.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:11 smithi055 ceph-mon[97033]: osdmap e426: 12 total, 11 up, 11 in 2022-09-08T00:28:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:11 smithi059 ceph-mon[97450]: pgmap v5439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:28:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:11 smithi059 ceph-mon[97450]: osdmap e426: 12 total, 11 up, 11 in 2022-09-08T00:28:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:12.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:12.211817+0000) 2022-09-08T00:28:12.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:12.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:12.133313+0000) 2022-09-08T00:28:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:13.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:13.212066+0000) 2022-09-08T00:28:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:13.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:13.133558+0000) 2022-09-08T00:28:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:13 smithi055 ceph-mon[97033]: pgmap v5441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T00:28:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:13 smithi059 ceph-mon[97450]: pgmap v5441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T00:28:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:14.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:14.212227+0000) 2022-09-08T00:28:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:14.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:14.133824+0000) 2022-09-08T00:28:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:15.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:15.212408+0000) 2022-09-08T00:28:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:15.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:15.134098+0000) 2022-09-08T00:28:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:15 smithi055 ceph-mon[97033]: pgmap v5442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:28:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:15 smithi059 ceph-mon[97450]: pgmap v5442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:28:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:16.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:16.212589+0000) 2022-09-08T00:28:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:16.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:16.134354+0000) 2022-09-08T00:28:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:16 smithi059 ceph-mon[97450]: pgmap v5443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T00:28:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:16 smithi055 ceph-mon[97033]: pgmap v5443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T00:28:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:17.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:17.134624+0000) 2022-09-08T00:28:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:17.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:17.212692+0000) 2022-09-08T00:28:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:18.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:18.212851+0000) 2022-09-08T00:28:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:18.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:18.134784+0000) 2022-09-08T00:28:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:19.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:19.134985+0000) 2022-09-08T00:28:19.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:19.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:19.213067+0000) 2022-09-08T00:28:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:19 smithi055 ceph-mon[97033]: pgmap v5444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:28:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:19 smithi059 ceph-mon[97450]: pgmap v5444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:28:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:20.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:20.135168+0000) 2022-09-08T00:28:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:20.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:20.213264+0000) 2022-09-08T00:28:21.136 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:28:20.779+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:28:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:21.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:21.213495+0000) 2022-09-08T00:28:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:21.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:21.135401+0000) 2022-09-08T00:28:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:21 smithi055 ceph-mon[97033]: pgmap v5445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:28:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:21 smithi059 ceph-mon[97450]: pgmap v5445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:28:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:22.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:22.213676+0000) 2022-09-08T00:28:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:22.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:22.135620+0000) 2022-09-08T00:28:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:23.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:23.213866+0000) 2022-09-08T00:28:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:23.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:23.135886+0000) 2022-09-08T00:28:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:23 smithi055 ceph-mon[97033]: pgmap v5446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:28:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:23 smithi059 ceph-mon[97450]: pgmap v5446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:28:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:24.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:24.214032+0000) 2022-09-08T00:28:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:24.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:24.136119+0000) 2022-09-08T00:28:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:25.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:25.214190+0000) 2022-09-08T00:28:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:25.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:25.136360+0000) 2022-09-08T00:28:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:25 smithi055 ceph-mon[97033]: pgmap v5447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:28:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:25 smithi059 ceph-mon[97450]: pgmap v5447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:28:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:26.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:26.214417+0000) 2022-09-08T00:28:26.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:26.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:26.136572+0000) 2022-09-08T00:28:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:26 smithi059 ceph-mon[97450]: pgmap v5448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:28:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:26 smithi055 ceph-mon[97033]: pgmap v5448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:28:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:27.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:27.214670+0000) 2022-09-08T00:28:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:27.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:27.136782+0000) 2022-09-08T00:28:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:28.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:28.214899+0000) 2022-09-08T00:28:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:28.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:28.136934+0000) 2022-09-08T00:28:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:29.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:29.215111+0000) 2022-09-08T00:28:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:29.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:29.137208+0000) 2022-09-08T00:28:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:29 smithi055 ceph-mon[97033]: pgmap v5449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:29 smithi059 ceph-mon[97450]: pgmap v5449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:30.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:30.215337+0000) 2022-09-08T00:28:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:30.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:30.137439+0000) 2022-09-08T00:28:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:31.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:31.137730+0000) 2022-09-08T00:28:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:31.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:31.215530+0000) 2022-09-08T00:28:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:31 smithi055 ceph-mon[97033]: pgmap v5450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:28:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:31 smithi059 ceph-mon[97450]: pgmap v5450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:28:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:32.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:32.215773+0000) 2022-09-08T00:28:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:32.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:32.137966+0000) 2022-09-08T00:28:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:33.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:33.216012+0000) 2022-09-08T00:28:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:33.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:33.138161+0000) 2022-09-08T00:28:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:33 smithi055 ceph-mon[97033]: pgmap v5451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:33 smithi059 ceph-mon[97450]: pgmap v5451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:34.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:34.138361+0000) 2022-09-08T00:28:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:34.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:34.216148+0000) 2022-09-08T00:28:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:35.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:35.216287+0000) 2022-09-08T00:28:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:35.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:35.138612+0000) 2022-09-08T00:28:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:35 smithi055 ceph-mon[97033]: pgmap v5452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:35 smithi059 ceph-mon[97450]: pgmap v5452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:36.141 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:28:35.794+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:28:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:36.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:36.138863+0000) 2022-09-08T00:28:36.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:36.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:36.216499+0000) 2022-09-08T00:28:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:36 smithi059 ceph-mon[97450]: pgmap v5453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:28:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:36 smithi055 ceph-mon[97033]: pgmap v5453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:28:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:37.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:37.139089+0000) 2022-09-08T00:28:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:37.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:37.216720+0000) 2022-09-08T00:28:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:38.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:38.216963+0000) 2022-09-08T00:28:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:38.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:38.139228+0000) 2022-09-08T00:28:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:39.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:39.217180+0000) 2022-09-08T00:28:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:39.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:39.139383+0000) 2022-09-08T00:28:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:39 smithi055 ceph-mon[97033]: pgmap v5454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:39 smithi059 ceph-mon[97450]: pgmap v5454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:40.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:40.217385+0000) 2022-09-08T00:28:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:40.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:40.139606+0000) 2022-09-08T00:28:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:41.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:41.217580+0000) 2022-09-08T00:28:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:41.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:41.139873+0000) 2022-09-08T00:28:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:41 smithi055 ceph-mon[97033]: pgmap v5455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:28:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:41 smithi059 ceph-mon[97450]: pgmap v5455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:28:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:42.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:42.140071+0000) 2022-09-08T00:28:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:42.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:42.217836+0000) 2022-09-08T00:28:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:43.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:43.218097+0000) 2022-09-08T00:28:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:43.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:43.140253+0000) 2022-09-08T00:28:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:43 smithi055 ceph-mon[97033]: pgmap v5456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:28:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:43 smithi059 ceph-mon[97450]: pgmap v5456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:28:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:44.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:44.218287+0000) 2022-09-08T00:28:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:44.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:44.140444+0000) 2022-09-08T00:28:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:45.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:45.218528+0000) 2022-09-08T00:28:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:45.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:45.140704+0000) 2022-09-08T00:28:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:45 smithi055 ceph-mon[97033]: pgmap v5457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:28:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:45 smithi059 ceph-mon[97450]: pgmap v5457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:28:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:46.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:46.140945+0000) 2022-09-08T00:28:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:46.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:46.218681+0000) 2022-09-08T00:28:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:46 smithi059 ceph-mon[97450]: pgmap v5458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:28:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:46 smithi055 ceph-mon[97033]: pgmap v5458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:28:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:47.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:47.218902+0000) 2022-09-08T00:28:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:47.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:47.141190+0000) 2022-09-08T00:28:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:48.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:48.219131+0000) 2022-09-08T00:28:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:48.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:48.141364+0000) 2022-09-08T00:28:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:49.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:49.219310+0000) 2022-09-08T00:28:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:49.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:49.141525+0000) 2022-09-08T00:28:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:49 smithi055 ceph-mon[97033]: pgmap v5459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:28:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:49 smithi059 ceph-mon[97450]: pgmap v5459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:28:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:50.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:50.219547+0000) 2022-09-08T00:28:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:50.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:50.141728+0000) 2022-09-08T00:28:51.143 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:28:50.804+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:28:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:51.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:51.219782+0000) 2022-09-08T00:28:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:51.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:51.141989+0000) 2022-09-08T00:28:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:51 smithi055 ceph-mon[97033]: pgmap v5460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:28:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:51 smithi059 ceph-mon[97450]: pgmap v5460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:28:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:52.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:52.219997+0000) 2022-09-08T00:28:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:52.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:52.142230+0000) 2022-09-08T00:28:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:53.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:53.220177+0000) 2022-09-08T00:28:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:53.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:53.142502+0000) 2022-09-08T00:28:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:53 smithi055 ceph-mon[97033]: pgmap v5461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:53 smithi059 ceph-mon[97450]: pgmap v5461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:54.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:54.220398+0000) 2022-09-08T00:28:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:54.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:54.142701+0000) 2022-09-08T00:28:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:55.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:55.220673+0000) 2022-09-08T00:28:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:55.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:55.142902+0000) 2022-09-08T00:28:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:55 smithi055 ceph-mon[97033]: pgmap v5462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:55 smithi059 ceph-mon[97450]: pgmap v5462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:28:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:56.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:56.220944+0000) 2022-09-08T00:28:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:56.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:56.143134+0000) 2022-09-08T00:28:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:56 smithi059 ceph-mon[97450]: pgmap v5463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:28:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:56 smithi055 ceph-mon[97033]: pgmap v5463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:28:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:57.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:57.221130+0000) 2022-09-08T00:28:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:57.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:57.143368+0000) 2022-09-08T00:28:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:58.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:58.221314+0000) 2022-09-08T00:28:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:58.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:58.143575+0000) 2022-09-08T00:28:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:28:59.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:59.221562+0000) 2022-09-08T00:28:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:28:59.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:28:59.143862+0000) 2022-09-08T00:28:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:28:59 smithi055 ceph-mon[97033]: pgmap v5464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:28:59 smithi059 ceph-mon[97450]: pgmap v5464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:00.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:00.221780+0000) 2022-09-08T00:29:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:00.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:00.144051+0000) 2022-09-08T00:29:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:01.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:01.221925+0000) 2022-09-08T00:29:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:01.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:01.144186+0000) 2022-09-08T00:29:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:01 smithi055 ceph-mon[97033]: pgmap v5465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:29:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:29:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:29:01.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:29:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:01 smithi059 ceph-mon[97450]: pgmap v5465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:29:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:29:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:29:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:29:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:02.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:02.144377+0000) 2022-09-08T00:29:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:02.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:02.222157+0000) 2022-09-08T00:29:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:29:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:02 smithi059 ceph-mon[97450]: pgmap v5466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:29:03.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:02 smithi055 ceph-mon[97033]: pgmap v5466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:03.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:03.144564+0000) 2022-09-08T00:29:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:03.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:03.222339+0000) 2022-09-08T00:29:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:04.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:04.144739+0000) 2022-09-08T00:29:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:04.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:04.222569+0000) 2022-09-08T00:29:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:05.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:05.144963+0000) 2022-09-08T00:29:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:05.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:05.222834+0000) 2022-09-08T00:29:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:05 smithi055 ceph-mon[97033]: pgmap v5467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:29:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:29:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:05 smithi059 ceph-mon[97450]: pgmap v5467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:29:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:29:06.147 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:29:05.812+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:29:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:06.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:06.223031+0000) 2022-09-08T00:29:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:06.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:06.145173+0000) 2022-09-08T00:29:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:06 smithi059 ceph-mon[97450]: pgmap v5468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:29:07.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:06 smithi055 ceph-mon[97033]: pgmap v5468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:29:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:07.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:07.223254+0000) 2022-09-08T00:29:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:07.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:07.145400+0000) 2022-09-08T00:29:08.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:08.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:08.223496+0000) 2022-09-08T00:29:08.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:08.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:08.145674+0000) 2022-09-08T00:29:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:09.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:09.223784+0000) 2022-09-08T00:29:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:09.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:09.145921+0000) 2022-09-08T00:29:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:09 smithi055 ceph-mon[97033]: osdmap e427: 12 total, 11 up, 11 in 2022-09-08T00:29:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:09 smithi055 ceph-mon[97033]: pgmap v5470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:29:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:09 smithi059 ceph-mon[97450]: osdmap e427: 12 total, 11 up, 11 in 2022-09-08T00:29:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:09 smithi059 ceph-mon[97450]: pgmap v5470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:29:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:10.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:10.223936+0000) 2022-09-08T00:29:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:10.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:10.146066+0000) 2022-09-08T00:29:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:11.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:11.224129+0000) 2022-09-08T00:29:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:11.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:11.146287+0000) 2022-09-08T00:29:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:11 smithi055 ceph-mon[97033]: pgmap v5471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:29:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:11 smithi055 ceph-mon[97033]: osdmap e428: 12 total, 11 up, 11 in 2022-09-08T00:29:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:11 smithi059 ceph-mon[97450]: pgmap v5471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:29:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:11 smithi059 ceph-mon[97450]: osdmap e428: 12 total, 11 up, 11 in 2022-09-08T00:29:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:12.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:12.224404+0000) 2022-09-08T00:29:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:12.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:12.146494+0000) 2022-09-08T00:29:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:13.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:13.224669+0000) 2022-09-08T00:29:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:13.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:13.146725+0000) 2022-09-08T00:29:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:13 smithi055 ceph-mon[97033]: pgmap v5473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:29:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:13 smithi059 ceph-mon[97450]: pgmap v5473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:29:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:14.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:14.224938+0000) 2022-09-08T00:29:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:14.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:14.146915+0000) 2022-09-08T00:29:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:15.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:15.225183+0000) 2022-09-08T00:29:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:15.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:15.147155+0000) 2022-09-08T00:29:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:15 smithi055 ceph-mon[97033]: pgmap v5474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:29:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:15 smithi059 ceph-mon[97450]: pgmap v5474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:29:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:16.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:16.225405+0000) 2022-09-08T00:29:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:16.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:16.147357+0000) 2022-09-08T00:29:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:16 smithi059 ceph-mon[97450]: pgmap v5475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T00:29:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:16 smithi055 ceph-mon[97033]: pgmap v5475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T00:29:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:17.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:17.225621+0000) 2022-09-08T00:29:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:17.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:17.147556+0000) 2022-09-08T00:29:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:18.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:18.225877+0000) 2022-09-08T00:29:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:18.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:18.147754+0000) 2022-09-08T00:29:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:19.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:19.226101+0000) 2022-09-08T00:29:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:19.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:19.147932+0000) 2022-09-08T00:29:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:19 smithi055 ceph-mon[97033]: pgmap v5476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:29:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:19 smithi059 ceph-mon[97450]: pgmap v5476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:29:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:20.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:20.226352+0000) 2022-09-08T00:29:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:20.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:20.148173+0000) 2022-09-08T00:29:21.149 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:29:20.824+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:29:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:21.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:21.226585+0000) 2022-09-08T00:29:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:21.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:21.148378+0000) 2022-09-08T00:29:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:21 smithi055 ceph-mon[97033]: pgmap v5477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T00:29:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:21 smithi059 ceph-mon[97450]: pgmap v5477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T00:29:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:22.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:22.226825+0000) 2022-09-08T00:29:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:22.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:22.148609+0000) 2022-09-08T00:29:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:23.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:23.148890+0000) 2022-09-08T00:29:23.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:23.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:23.227007+0000) 2022-09-08T00:29:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:23 smithi055 ceph-mon[97033]: pgmap v5478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:29:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:23 smithi059 ceph-mon[97450]: pgmap v5478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:29:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:24.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:24.149068+0000) 2022-09-08T00:29:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:24.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:24.227191+0000) 2022-09-08T00:29:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:25.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:25.149267+0000) 2022-09-08T00:29:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:25.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:25.227357+0000) 2022-09-08T00:29:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:25 smithi055 ceph-mon[97033]: pgmap v5479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2022-09-08T00:29:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:25 smithi059 ceph-mon[97450]: pgmap v5479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2022-09-08T00:29:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:26.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:26.227572+0000) 2022-09-08T00:29:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:26.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:26.149472+0000) 2022-09-08T00:29:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:26 smithi059 ceph-mon[97450]: pgmap v5480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:29:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:26 smithi055 ceph-mon[97033]: pgmap v5480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:29:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:27.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:27.227828+0000) 2022-09-08T00:29:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:27.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:27.149677+0000) 2022-09-08T00:29:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:28.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:28.228031+0000) 2022-09-08T00:29:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:28.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:28.149904+0000) 2022-09-08T00:29:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:29.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:29.228260+0000) 2022-09-08T00:29:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:29.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:29.150112+0000) 2022-09-08T00:29:29.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:29 smithi055 ceph-mon[97033]: pgmap v5481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:29 smithi059 ceph-mon[97450]: pgmap v5481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:30.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:30.228498+0000) 2022-09-08T00:29:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:30.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:30.150345+0000) 2022-09-08T00:29:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:31.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:31.228736+0000) 2022-09-08T00:29:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:31.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:31.150575+0000) 2022-09-08T00:29:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:31 smithi055 ceph-mon[97033]: pgmap v5482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:29:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:31 smithi059 ceph-mon[97450]: pgmap v5482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:29:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:32.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:32.228865+0000) 2022-09-08T00:29:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:32.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:32.150811+0000) 2022-09-08T00:29:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:33.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:33.229069+0000) 2022-09-08T00:29:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:33.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:33.150983+0000) 2022-09-08T00:29:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:33 smithi055 ceph-mon[97033]: pgmap v5483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:33 smithi059 ceph-mon[97450]: pgmap v5483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:34.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:34.229248+0000) 2022-09-08T00:29:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:34.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:34.151242+0000) 2022-09-08T00:29:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:35.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:35.229467+0000) 2022-09-08T00:29:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:35.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:35.151486+0000) 2022-09-08T00:29:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:35 smithi055 ceph-mon[97033]: pgmap v5484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:35 smithi059 ceph-mon[97450]: pgmap v5484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:36.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:29:35.839+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:29:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:36.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:36.229723+0000) 2022-09-08T00:29:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:36.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:36.151871+0000) 2022-09-08T00:29:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:36 smithi059 ceph-mon[97450]: pgmap v5485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:29:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:36 smithi055 ceph-mon[97033]: pgmap v5485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:29:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:37.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:37.229940+0000) 2022-09-08T00:29:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:37.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:37.152105+0000) 2022-09-08T00:29:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:38.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:38.230160+0000) 2022-09-08T00:29:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:38.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:38.152345+0000) 2022-09-08T00:29:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:39.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:39.230386+0000) 2022-09-08T00:29:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:39.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:39.152618+0000) 2022-09-08T00:29:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:39 smithi055 ceph-mon[97033]: pgmap v5486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:39 smithi059 ceph-mon[97450]: pgmap v5486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:40.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:40.230619+0000) 2022-09-08T00:29:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:40.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:40.152893+0000) 2022-09-08T00:29:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:41.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:41.230874+0000) 2022-09-08T00:29:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:41.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:41.153121+0000) 2022-09-08T00:29:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:41 smithi055 ceph-mon[97033]: pgmap v5487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:29:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:41 smithi059 ceph-mon[97450]: pgmap v5487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:29:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:42.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:42.231103+0000) 2022-09-08T00:29:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:42.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:42.153347+0000) 2022-09-08T00:29:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:43.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:43.231332+0000) 2022-09-08T00:29:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:43.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:43.153595+0000) 2022-09-08T00:29:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:43 smithi055 ceph-mon[97033]: pgmap v5488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:43 smithi059 ceph-mon[97450]: pgmap v5488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:44.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:44.231604+0000) 2022-09-08T00:29:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:44.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:44.153845+0000) 2022-09-08T00:29:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:45.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:45.231794+0000) 2022-09-08T00:29:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:45.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:45.154071+0000) 2022-09-08T00:29:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:45 smithi055 ceph-mon[97033]: pgmap v5489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:45 smithi059 ceph-mon[97450]: pgmap v5489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:46.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:46.232015+0000) 2022-09-08T00:29:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:46.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:46.154272+0000) 2022-09-08T00:29:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:46 smithi059 ceph-mon[97450]: pgmap v5490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:29:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:46 smithi055 ceph-mon[97033]: pgmap v5490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:29:47.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:47.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:47.232240+0000) 2022-09-08T00:29:47.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:47.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:47.154469+0000) 2022-09-08T00:29:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:48.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:48.232445+0000) 2022-09-08T00:29:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:48.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:48.154719+0000) 2022-09-08T00:29:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:49.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:49.232707+0000) 2022-09-08T00:29:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:49.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:49.154965+0000) 2022-09-08T00:29:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:49 smithi055 ceph-mon[97033]: pgmap v5491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:49 smithi059 ceph-mon[97450]: pgmap v5491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:50.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:50.232840+0000) 2022-09-08T00:29:50.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:50.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:50.155194+0000) 2022-09-08T00:29:51.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:29:50.854+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:29:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:51.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:51.233063+0000) 2022-09-08T00:29:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:51.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:51.155448+0000) 2022-09-08T00:29:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:51 smithi055 ceph-mon[97033]: pgmap v5492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:29:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:51 smithi059 ceph-mon[97450]: pgmap v5492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:29:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:52.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:52.233323+0000) 2022-09-08T00:29:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:52.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:52.155613+0000) 2022-09-08T00:29:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:53.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:53.233565+0000) 2022-09-08T00:29:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:53.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:53.155801+0000) 2022-09-08T00:29:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:53 smithi055 ceph-mon[97033]: pgmap v5493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:53 smithi059 ceph-mon[97450]: pgmap v5493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:54.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:54.233828+0000) 2022-09-08T00:29:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:54.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:54.156055+0000) 2022-09-08T00:29:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:55.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:55.234044+0000) 2022-09-08T00:29:55.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:55.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:55.156290+0000) 2022-09-08T00:29:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:55 smithi055 ceph-mon[97033]: pgmap v5494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:55 smithi059 ceph-mon[97450]: pgmap v5494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:29:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:56.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:56.234249+0000) 2022-09-08T00:29:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:56.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:56.156549+0000) 2022-09-08T00:29:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:56 smithi059 ceph-mon[97450]: pgmap v5495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:29:57.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:56 smithi055 ceph-mon[97033]: pgmap v5495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:29:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:57.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:57.234501+0000) 2022-09-08T00:29:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:57.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:57.156700+0000) 2022-09-08T00:29:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:58.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:58.156891+0000) 2022-09-08T00:29:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:58.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:58.234726+0000) 2022-09-08T00:29:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:29:59.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:59.234867+0000) 2022-09-08T00:29:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:29:59.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:29:59.157071+0000) 2022-09-08T00:29:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:29:59 smithi055 ceph-mon[97033]: pgmap v5496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:30:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:29:59 smithi059 ceph-mon[97450]: pgmap v5496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:30:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:00.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:00.235022+0000) 2022-09-08T00:30:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:00.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:00.157249+0000) 2022-09-08T00:30:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T00:30:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T00:30:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T00:30:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T00:30:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T00:30:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T00:30:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:01.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:01.235198+0000) 2022-09-08T00:30:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:01.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:01.157408+0000) 2022-09-08T00:30:01.893 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:01 smithi055 ceph-mon[97033]: pgmap v5497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:30:01.895 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:01 smithi059 ceph-mon[97450]: pgmap v5497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:30:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:02.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:02.235449+0000) 2022-09-08T00:30:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:02.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:02.157557+0000) 2022-09-08T00:30:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:30:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:30:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:30:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:30:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:30:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:30:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:03.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:03.235665+0000) 2022-09-08T00:30:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:03.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:03.157784+0000) 2022-09-08T00:30:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:03 smithi055 ceph-mon[97033]: pgmap v5498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:30:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:03 smithi059 ceph-mon[97450]: pgmap v5498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:30:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:04.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:04.235888+0000) 2022-09-08T00:30:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:04.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:04.158031+0000) 2022-09-08T00:30:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:05.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:05.236083+0000) 2022-09-08T00:30:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:05.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:05.158267+0000) 2022-09-08T00:30:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:05 smithi055 ceph-mon[97033]: pgmap v5499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:30:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:30:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:30:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:05 smithi059 ceph-mon[97450]: pgmap v5499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:30:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:30:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:30:06.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:30:05.866+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:30:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:06.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:06.236302+0000) 2022-09-08T00:30:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:06.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:06.158510+0000) 2022-09-08T00:30:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:06 smithi059 ceph-mon[97450]: pgmap v5500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 49 KiB/s wr, 22 op/s 2022-09-08T00:30:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:06 smithi055 ceph-mon[97033]: pgmap v5500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 49 KiB/s wr, 22 op/s 2022-09-08T00:30:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:07.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:07.236475+0000) 2022-09-08T00:30:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:07.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:07.158743+0000) 2022-09-08T00:30:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:08.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:08.236725+0000) 2022-09-08T00:30:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:08.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:08.158909+0000) 2022-09-08T00:30:09.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:09.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:09.236888+0000) 2022-09-08T00:30:09.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:09.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:09.159094+0000) 2022-09-08T00:30:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:09 smithi055 ceph-mon[97033]: osdmap e429: 12 total, 11 up, 11 in 2022-09-08T00:30:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:09 smithi055 ceph-mon[97033]: pgmap v5502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 19 KiB/s rd, 59 KiB/s wr, 24 op/s 2022-09-08T00:30:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:09 smithi059 ceph-mon[97450]: osdmap e429: 12 total, 11 up, 11 in 2022-09-08T00:30:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:09 smithi059 ceph-mon[97450]: pgmap v5502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 19 KiB/s rd, 59 KiB/s wr, 24 op/s 2022-09-08T00:30:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:10.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:10.237090+0000) 2022-09-08T00:30:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:10.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:10.159322+0000) 2022-09-08T00:30:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:11.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:11.237246+0000) 2022-09-08T00:30:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:11.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:11.159525+0000) 2022-09-08T00:30:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:11 smithi055 ceph-mon[97033]: pgmap v5503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 19 KiB/s rd, 59 KiB/s wr, 25 op/s 2022-09-08T00:30:11.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:11 smithi055 ceph-mon[97033]: osdmap e430: 12 total, 11 up, 11 in 2022-09-08T00:30:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:11 smithi059 ceph-mon[97450]: pgmap v5503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 19 KiB/s rd, 59 KiB/s wr, 25 op/s 2022-09-08T00:30:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:11 smithi059 ceph-mon[97450]: osdmap e430: 12 total, 11 up, 11 in 2022-09-08T00:30:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:12.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:12.237417+0000) 2022-09-08T00:30:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:12.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:12.159744+0000) 2022-09-08T00:30:13.665 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:13.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:13.237692+0000) 2022-09-08T00:30:13.666 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:13.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:13.160011+0000) 2022-09-08T00:30:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:13 smithi055 ceph-mon[97033]: pgmap v5505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 24 KiB/s rd, 74 KiB/s wr, 31 op/s 2022-09-08T00:30:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:13 smithi059 ceph-mon[97450]: pgmap v5505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 24 KiB/s rd, 74 KiB/s wr, 31 op/s 2022-09-08T00:30:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:14.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:14.237879+0000) 2022-09-08T00:30:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:14.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:14.160225+0000) 2022-09-08T00:30:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:15.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:15.238112+0000) 2022-09-08T00:30:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:15.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:15.160436+0000) 2022-09-08T00:30:15.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:15 smithi055 ceph-mon[97033]: pgmap v5506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 4 op/s 2022-09-08T00:30:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:15 smithi059 ceph-mon[97450]: pgmap v5506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 4 op/s 2022-09-08T00:30:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:16.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:16.238238+0000) 2022-09-08T00:30:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:16.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:16.160683+0000) 2022-09-08T00:30:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:16 smithi059 ceph-mon[97450]: pgmap v5507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 251 B/s wr, 9 op/s 2022-09-08T00:30:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:16 smithi055 ceph-mon[97033]: pgmap v5507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 251 B/s wr, 9 op/s 2022-09-08T00:30:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:17.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:17.238478+0000) 2022-09-08T00:30:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:17.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:17.160824+0000) 2022-09-08T00:30:18.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:18.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:18.238699+0000) 2022-09-08T00:30:18.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:18.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:18.160986+0000) 2022-09-08T00:30:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:19.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:19.238856+0000) 2022-09-08T00:30:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:19.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:19.161155+0000) 2022-09-08T00:30:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:19 smithi055 ceph-mon[97033]: pgmap v5508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2022-09-08T00:30:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:19 smithi059 ceph-mon[97450]: pgmap v5508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2022-09-08T00:30:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:20.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:20.239086+0000) 2022-09-08T00:30:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:20.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:20.161371+0000) 2022-09-08T00:30:21.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:30:20.875+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:30:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:21.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:21.239319+0000) 2022-09-08T00:30:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:21.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:21.161585+0000) 2022-09-08T00:30:21.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:21 smithi055 ceph-mon[97033]: pgmap v5509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 7 op/s 2022-09-08T00:30:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:21 smithi059 ceph-mon[97450]: pgmap v5509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 7 op/s 2022-09-08T00:30:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:22.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:22.239532+0000) 2022-09-08T00:30:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:22.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:22.161867+0000) 2022-09-08T00:30:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:23.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:23.239766+0000) 2022-09-08T00:30:23.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:23.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:23.162082+0000) 2022-09-08T00:30:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:23 smithi055 ceph-mon[97033]: pgmap v5510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 86 B/s wr, 6 op/s 2022-09-08T00:30:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:23 smithi059 ceph-mon[97450]: pgmap v5510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 86 B/s wr, 6 op/s 2022-09-08T00:30:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:24.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:24.240018+0000) 2022-09-08T00:30:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:24.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:24.162316+0000) 2022-09-08T00:30:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:25.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:25.240208+0000) 2022-09-08T00:30:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:25.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:25.162553+0000) 2022-09-08T00:30:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:25 smithi055 ceph-mon[97033]: pgmap v5511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:30:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:25 smithi059 ceph-mon[97450]: pgmap v5511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:30:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:26.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:26.240443+0000) 2022-09-08T00:30:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:26.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:26.162782+0000) 2022-09-08T00:30:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:26 smithi059 ceph-mon[97450]: pgmap v5512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-08T00:30:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:26 smithi055 ceph-mon[97033]: pgmap v5512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-08T00:30:27.241 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:27.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:27.163059+0000) 2022-09-08T00:30:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:27.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:27.240722+0000) 2022-09-08T00:30:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:28.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:28.240862+0000) 2022-09-08T00:30:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:28.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:28.163287+0000) 2022-09-08T00:30:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:29.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:29.240987+0000) 2022-09-08T00:30:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:29.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:29.163459+0000) 2022-09-08T00:30:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:29 smithi055 ceph-mon[97033]: pgmap v5513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-08T00:30:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:29 smithi059 ceph-mon[97450]: pgmap v5513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-08T00:30:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:30.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:30.241213+0000) 2022-09-08T00:30:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:30.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:30.163591+0000) 2022-09-08T00:30:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:31.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:31.241440+0000) 2022-09-08T00:30:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:31.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:31.163774+0000) 2022-09-08T00:30:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:31 smithi055 ceph-mon[97033]: pgmap v5514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-08T00:30:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:31 smithi059 ceph-mon[97450]: pgmap v5514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-08T00:30:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:32.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:32.241674+0000) 2022-09-08T00:30:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:32.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:32.163937+0000) 2022-09-08T00:30:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:33 smithi059 ceph-mon[97450]: pgmap v5515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 6 op/s 2022-09-08T00:30:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:30:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:33 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T00:30:33.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:33 smithi055 ceph-mon[97033]: pgmap v5515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 6 op/s 2022-09-08T00:30:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:30:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:33 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T00:30:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:33.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:33.164114+0000) 2022-09-08T00:30:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:33.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:33.241841+0000) 2022-09-08T00:30:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:34.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:34.242011+0000) 2022-09-08T00:30:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:34.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:34.164369+0000) 2022-09-08T00:30:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:35 smithi059 ceph-mon[97450]: pgmap v5516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 6 op/s 2022-09-08T00:30:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:35 smithi055 ceph-mon[97033]: pgmap v5516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 6 op/s 2022-09-08T00:30:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:35.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:35.242218+0000) 2022-09-08T00:30:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:35.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:35.164562+0000) 2022-09-08T00:30:36.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:30:35.884+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:30:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:36.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:36.242443+0000) 2022-09-08T00:30:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:36.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:36.164795+0000) 2022-09-08T00:30:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:36 smithi059 ceph-mon[97450]: pgmap v5517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 9 op/s 2022-09-08T00:30:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:36 smithi055 ceph-mon[97033]: pgmap v5517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 9 op/s 2022-09-08T00:30:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:37.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:37.242719+0000) 2022-09-08T00:30:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:37.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:37.165029+0000) 2022-09-08T00:30:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:38.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:38.242971+0000) 2022-09-08T00:30:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:38.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:38.165246+0000) 2022-09-08T00:30:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:39 smithi055 ceph-mon[97033]: pgmap v5518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 6 op/s 2022-09-08T00:30:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:39 smithi059 ceph-mon[97450]: pgmap v5518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 6 op/s 2022-09-08T00:30:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:39.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:39.243270+0000) 2022-09-08T00:30:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:39.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:39.165446+0000) 2022-09-08T00:30:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:40.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:40.243492+0000) 2022-09-08T00:30:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:40.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:40.165576+0000) 2022-09-08T00:30:41.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:41 smithi055 ceph-mon[97033]: pgmap v5519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 9 op/s 2022-09-08T00:30:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:41 smithi059 ceph-mon[97450]: pgmap v5519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 9 op/s 2022-09-08T00:30:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:41.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:41.243735+0000) 2022-09-08T00:30:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:41.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:41.165782+0000) 2022-09-08T00:30:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:42.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:42.244022+0000) 2022-09-08T00:30:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:42.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:42.165977+0000) 2022-09-08T00:30:43.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:43 smithi055 ceph-mon[97033]: pgmap v5520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 6 op/s 2022-09-08T00:30:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:43 smithi059 ceph-mon[97450]: pgmap v5520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 6 op/s 2022-09-08T00:30:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:43.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:43.244264+0000) 2022-09-08T00:30:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:43.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:43.166178+0000) 2022-09-08T00:30:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:44.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:44.244524+0000) 2022-09-08T00:30:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:44.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:44.166435+0000) 2022-09-08T00:30:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:45 smithi055 ceph-mon[97033]: pgmap v5521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:30:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:45 smithi059 ceph-mon[97450]: pgmap v5521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:30:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:45.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:45.244766+0000) 2022-09-08T00:30:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:45.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:45.166729+0000) 2022-09-08T00:30:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:46.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:46.244902+0000) 2022-09-08T00:30:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:46.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:46.166959+0000) 2022-09-08T00:30:47.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:46 smithi059 ceph-mon[97450]: pgmap v5522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-08T00:30:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:46 smithi055 ceph-mon[97033]: pgmap v5522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-08T00:30:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:47.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:47.245153+0000) 2022-09-08T00:30:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:47.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:47.167223+0000) 2022-09-08T00:30:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:48.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:48.245381+0000) 2022-09-08T00:30:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:48.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:48.167418+0000) 2022-09-08T00:30:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:49 smithi055 ceph-mon[97033]: pgmap v5523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:30:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:49 smithi059 ceph-mon[97450]: pgmap v5523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:30:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:49.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:49.245594+0000) 2022-09-08T00:30:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:49.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:49.167676+0000) 2022-09-08T00:30:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:50.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:50.245793+0000) 2022-09-08T00:30:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:50.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:50.167948+0000) 2022-09-08T00:30:51.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:30:50.895+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:30:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:51 smithi055 ceph-mon[97033]: pgmap v5524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-08T00:30:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:51 smithi059 ceph-mon[97450]: pgmap v5524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-08T00:30:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:51.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:51.245970+0000) 2022-09-08T00:30:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:51.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:51.168164+0000) 2022-09-08T00:30:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:52.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:52.246177+0000) 2022-09-08T00:30:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:52.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:52.168399+0000) 2022-09-08T00:30:53.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:53 smithi055 ceph-mon[97033]: pgmap v5525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:30:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:53 smithi059 ceph-mon[97450]: pgmap v5525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:30:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:53.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:53.246383+0000) 2022-09-08T00:30:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:53.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:53.168683+0000) 2022-09-08T00:30:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:54.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:54.246617+0000) 2022-09-08T00:30:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:54.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:54.168944+0000) 2022-09-08T00:30:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:55 smithi055 ceph-mon[97033]: pgmap v5526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:30:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:55 smithi059 ceph-mon[97450]: pgmap v5526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:30:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:55.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:55.246890+0000) 2022-09-08T00:30:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:55.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:55.169176+0000) 2022-09-08T00:30:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:56.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:56.247109+0000) 2022-09-08T00:30:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:56.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:56.169380+0000) 2022-09-08T00:30:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:56 smithi059 ceph-mon[97450]: pgmap v5527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-08T00:30:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:56 smithi055 ceph-mon[97033]: pgmap v5527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-08T00:30:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:57.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:57.247319+0000) 2022-09-08T00:30:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:57.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:57.169606+0000) 2022-09-08T00:30:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:58.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:58.247590+0000) 2022-09-08T00:30:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:58.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:58.169857+0000) 2022-09-08T00:30:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:30:59 smithi055 ceph-mon[97033]: pgmap v5528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:30:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:30:59 smithi059 ceph-mon[97450]: pgmap v5528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:30:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:30:59.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:59.247920+0000) 2022-09-08T00:30:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:30:59.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:30:59.170103+0000) 2022-09-08T00:31:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:00.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:00.248091+0000) 2022-09-08T00:31:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:00.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:00.170321+0000) 2022-09-08T00:31:01.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:01 smithi055 ceph-mon[97033]: pgmap v5529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-08T00:31:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:01 smithi059 ceph-mon[97450]: pgmap v5529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-08T00:31:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:01.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:01.248273+0000) 2022-09-08T00:31:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:01.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:01.170545+0000) 2022-09-08T00:31:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:02.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:02.248479+0000) 2022-09-08T00:31:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:02.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:02.170769+0000) 2022-09-08T00:31:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:03 smithi055 ceph-mon[97033]: pgmap v5530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:31:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:03 smithi059 ceph-mon[97450]: pgmap v5530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:31:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:03.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:03.248672+0000) 2022-09-08T00:31:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:03.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:03.171020+0000) 2022-09-08T00:31:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:04.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:04.248806+0000) 2022-09-08T00:31:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:04.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:04.171264+0000) 2022-09-08T00:31:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:05 smithi055 ceph-mon[97033]: pgmap v5531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:31:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:31:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:31:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:05 smithi059 ceph-mon[97450]: pgmap v5531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:31:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:31:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:31:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:05.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:05.248984+0000) 2022-09-08T00:31:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:05.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:05.171469+0000) 2022-09-08T00:31:06.172 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:31:05.906+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:31:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:06.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:06.249213+0000) 2022-09-08T00:31:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:06.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:06.171709+0000) 2022-09-08T00:31:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:06 smithi059 ceph-mon[97450]: pgmap v5532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-08T00:31:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:06 smithi055 ceph-mon[97033]: pgmap v5532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-08T00:31:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:07.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:07.249407+0000) 2022-09-08T00:31:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:07.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:07.171923+0000) 2022-09-08T00:31:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:08.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:08.249593+0000) 2022-09-08T00:31:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:08.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:08.172144+0000) 2022-09-08T00:31:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:09 smithi055 ceph-mon[97033]: pgmap v5533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T00:31:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:09 smithi059 ceph-mon[97450]: pgmap v5533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T00:31:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:09.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:09.249707+0000) 2022-09-08T00:31:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:09.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:09.172295+0000) 2022-09-08T00:31:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:10 smithi055 ceph-mon[97033]: osdmap e431: 12 total, 11 up, 11 in 2022-09-08T00:31:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:10 smithi059 ceph-mon[97450]: osdmap e431: 12 total, 11 up, 11 in 2022-09-08T00:31:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:10.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:10.249901+0000) 2022-09-08T00:31:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:10.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:10.172526+0000) 2022-09-08T00:31:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:11 smithi055 ceph-mon[97033]: pgmap v5535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:31:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:11 smithi055 ceph-mon[97033]: osdmap e432: 12 total, 11 up, 11 in 2022-09-08T00:31:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:11 smithi059 ceph-mon[97450]: pgmap v5535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:31:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:11 smithi059 ceph-mon[97450]: osdmap e432: 12 total, 11 up, 11 in 2022-09-08T00:31:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:11.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:11.250075+0000) 2022-09-08T00:31:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:11.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:11.172751+0000) 2022-09-08T00:31:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:12.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:12.250250+0000) 2022-09-08T00:31:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:12.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:12.172927+0000) 2022-09-08T00:31:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:13 smithi055 ceph-mon[97033]: pgmap v5537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T00:31:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:13 smithi059 ceph-mon[97450]: pgmap v5537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T00:31:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:13.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:13.250410+0000) 2022-09-08T00:31:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:13.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:13.173101+0000) 2022-09-08T00:31:14.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:14.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:14.250568+0000) 2022-09-08T00:31:14.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:14.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:14.173322+0000) 2022-09-08T00:31:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:15 smithi055 ceph-mon[97033]: pgmap v5538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:31:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:15 smithi059 ceph-mon[97450]: pgmap v5538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:31:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:15.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:15.250768+0000) 2022-09-08T00:31:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:15.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:15.173570+0000) 2022-09-08T00:31:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:16.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:16.250913+0000) 2022-09-08T00:31:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:16.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:16.173871+0000) 2022-09-08T00:31:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:16 smithi055 ceph-mon[97033]: pgmap v5539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:31:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:16 smithi059 ceph-mon[97450]: pgmap v5539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:31:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:17.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:17.251104+0000) 2022-09-08T00:31:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:17.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:17.174049+0000) 2022-09-08T00:31:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:18.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:18.251340+0000) 2022-09-08T00:31:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:18.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:18.174301+0000) 2022-09-08T00:31:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:19 smithi055 ceph-mon[97033]: pgmap v5540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T00:31:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:19 smithi059 ceph-mon[97450]: pgmap v5540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T00:31:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:19.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:19.251498+0000) 2022-09-08T00:31:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:19.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:19.174565+0000) 2022-09-08T00:31:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:20.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:20.251749+0000) 2022-09-08T00:31:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:20.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:20.174736+0000) 2022-09-08T00:31:21.175 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:31:20.913+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:31:21.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:21 smithi055 ceph-mon[97033]: pgmap v5541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:31:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:21 smithi059 ceph-mon[97450]: pgmap v5541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:31:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:21.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:21.251940+0000) 2022-09-08T00:31:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:21.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:21.175009+0000) 2022-09-08T00:31:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:22.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:22.252164+0000) 2022-09-08T00:31:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:22.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:22.175260+0000) 2022-09-08T00:31:23.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:23 smithi055 ceph-mon[97033]: pgmap v5542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T00:31:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:23 smithi059 ceph-mon[97450]: pgmap v5542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T00:31:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:23.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:23.252406+0000) 2022-09-08T00:31:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:23.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:23.175529+0000) 2022-09-08T00:31:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:24.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:24.252699+0000) 2022-09-08T00:31:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:24.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:24.175752+0000) 2022-09-08T00:31:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:25 smithi055 ceph-mon[97033]: pgmap v5543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:31:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:25 smithi059 ceph-mon[97450]: pgmap v5543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:31:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:25.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:25.252898+0000) 2022-09-08T00:31:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:25.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:25.175956+0000) 2022-09-08T00:31:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:26.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:26.253040+0000) 2022-09-08T00:31:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:26.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:26.176239+0000) 2022-09-08T00:31:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:26 smithi055 ceph-mon[97033]: pgmap v5544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:31:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:26 smithi059 ceph-mon[97450]: pgmap v5544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:31:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:27.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:27.253238+0000) 2022-09-08T00:31:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:27.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:27.176430+0000) 2022-09-08T00:31:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:28.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:28.253459+0000) 2022-09-08T00:31:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:28.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:28.176679+0000) 2022-09-08T00:31:29.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:29 smithi055 ceph-mon[97033]: pgmap v5545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:29 smithi059 ceph-mon[97450]: pgmap v5545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:29.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:29.253715+0000) 2022-09-08T00:31:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:29.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:29.176895+0000) 2022-09-08T00:31:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:30.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:30.253896+0000) 2022-09-08T00:31:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:30.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:30.177125+0000) 2022-09-08T00:31:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:31 smithi055 ceph-mon[97033]: pgmap v5546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:31:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:31 smithi059 ceph-mon[97450]: pgmap v5546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:31:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:31.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:31.254035+0000) 2022-09-08T00:31:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:31.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:31.177346+0000) 2022-09-08T00:31:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:32.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:32.254164+0000) 2022-09-08T00:31:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:32.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:32.177517+0000) 2022-09-08T00:31:33.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:33 smithi055 ceph-mon[97033]: pgmap v5547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:33.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:31:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:31:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:31:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:31:33.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:31:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:33 smithi059 ceph-mon[97450]: pgmap v5547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:31:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:31:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:31:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:31:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:31:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:33.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:33.254302+0000) 2022-09-08T00:31:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:33.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:33.177662+0000) 2022-09-08T00:31:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:34.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:34.254459+0000) 2022-09-08T00:31:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:34.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:34.177789+0000) 2022-09-08T00:31:35.425 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:35 smithi055 ceph-mon[97033]: pgmap v5548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:35 smithi059 ceph-mon[97450]: pgmap v5548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:35.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:35.254702+0000) 2022-09-08T00:31:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:35.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:35.178036+0000) 2022-09-08T00:31:36.178 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:31:35.915+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:31:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:31:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:31:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:31:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:31:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:36.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:36.254954+0000) 2022-09-08T00:31:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:36.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:36.178216+0000) 2022-09-08T00:31:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:31:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:31:36.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:31:36.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:31:36.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.739 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.739 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:36.740 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:31:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:37 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T00:31:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:37 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T00:31:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:37 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T00:31:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:37 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T00:31:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:37 smithi059 ceph-mon[97450]: pgmap v5549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:31:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:37.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:37.255195+0000) 2022-09-08T00:31:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:37.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:37.178443+0000) 2022-09-08T00:31:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:37 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T00:31:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:37 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T00:31:37.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:37 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T00:31:37.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:37 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T00:31:37.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:37 smithi055 ceph-mon[97033]: pgmap v5549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:31:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:38.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:38.255425+0000) 2022-09-08T00:31:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:38.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:38.178735+0000) 2022-09-08T00:31:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:39 smithi059 ceph-mon[97450]: pgmap v5550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:39.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:39.255659+0000) 2022-09-08T00:31:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:39.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:39.178948+0000) 2022-09-08T00:31:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:39 smithi055 ceph-mon[97033]: pgmap v5550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:40.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:40.255892+0000) 2022-09-08T00:31:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:40.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:40.179140+0000) 2022-09-08T00:31:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:41 smithi059 ceph-mon[97450]: pgmap v5551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:31:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:41.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:41.256059+0000) 2022-09-08T00:31:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:41.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:41.179311+0000) 2022-09-08T00:31:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:41 smithi055 ceph-mon[97033]: pgmap v5551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:31:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:42.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:42.256227+0000) 2022-09-08T00:31:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:42.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:42.179550+0000) 2022-09-08T00:31:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:43 smithi059 ceph-mon[97450]: pgmap v5552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:43.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:43.179787+0000) 2022-09-08T00:31:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:43.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:43.256434+0000) 2022-09-08T00:31:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:43 smithi055 ceph-mon[97033]: pgmap v5552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:44.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:44.180060+0000) 2022-09-08T00:31:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:44.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:44.256682+0000) 2022-09-08T00:31:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:45 smithi059 ceph-mon[97450]: pgmap v5553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:45.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:45.256814+0000) 2022-09-08T00:31:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:45.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:45.180300+0000) 2022-09-08T00:31:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:45 smithi055 ceph-mon[97033]: pgmap v5553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:46.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:46.257028+0000) 2022-09-08T00:31:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:46.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:46.180537+0000) 2022-09-08T00:31:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:46 smithi055 ceph-mon[97033]: pgmap v5554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:31:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:46 smithi059 ceph-mon[97450]: pgmap v5554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:31:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:47.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:47.257227+0000) 2022-09-08T00:31:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:47.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:47.180782+0000) 2022-09-08T00:31:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:48.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:48.181025+0000) 2022-09-08T00:31:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:48.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:48.257421+0000) 2022-09-08T00:31:49.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:49 smithi055 ceph-mon[97033]: pgmap v5555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:49 smithi059 ceph-mon[97450]: pgmap v5555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:49.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:49.257685+0000) 2022-09-08T00:31:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:49.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:49.181275+0000) 2022-09-08T00:31:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:50.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:50.257893+0000) 2022-09-08T00:31:50.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:50.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:50.181547+0000) 2022-09-08T00:31:51.183 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:31:50.930+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:31:51.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:51 smithi055 ceph-mon[97033]: pgmap v5556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:31:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:51 smithi059 ceph-mon[97450]: pgmap v5556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:31:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:51.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:51.258127+0000) 2022-09-08T00:31:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:51.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:51.181757+0000) 2022-09-08T00:31:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:52.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:52.258319+0000) 2022-09-08T00:31:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:52.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:52.181970+0000) 2022-09-08T00:31:53.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:53 smithi055 ceph-mon[97033]: pgmap v5557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:53 smithi059 ceph-mon[97450]: pgmap v5557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:53.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:53.258503+0000) 2022-09-08T00:31:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:53.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:53.182215+0000) 2022-09-08T00:31:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:54.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:54.258749+0000) 2022-09-08T00:31:54.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:54.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:54.182417+0000) 2022-09-08T00:31:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:55 smithi055 ceph-mon[97033]: pgmap v5558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:55 smithi059 ceph-mon[97450]: pgmap v5558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:55.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:55.258939+0000) 2022-09-08T00:31:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:55.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:55.182715+0000) 2022-09-08T00:31:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:56.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:56.259105+0000) 2022-09-08T00:31:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:56.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:56.182899+0000) 2022-09-08T00:31:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:56 smithi055 ceph-mon[97033]: pgmap v5559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:31:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:56 smithi059 ceph-mon[97450]: pgmap v5559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:31:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:57.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:57.259263+0000) 2022-09-08T00:31:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:57.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:57.183106+0000) 2022-09-08T00:31:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:58.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:58.259516+0000) 2022-09-08T00:31:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:58.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:58.183297+0000) 2022-09-08T00:31:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:31:59 smithi055 ceph-mon[97033]: pgmap v5560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:31:59 smithi059 ceph-mon[97450]: pgmap v5560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:31:59.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:31:59.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:59.259728+0000) 2022-09-08T00:31:59.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:31:59.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:31:59.183459+0000) 2022-09-08T00:32:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:00.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:00.183721+0000) 2022-09-08T00:32:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:00.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:00.259876+0000) 2022-09-08T00:32:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:01 smithi055 ceph-mon[97033]: pgmap v5561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:32:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:01 smithi059 ceph-mon[97450]: pgmap v5561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:32:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:01.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:01.183854+0000) 2022-09-08T00:32:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:01.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:01.260083+0000) 2022-09-08T00:32:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:02.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:02.260250+0000) 2022-09-08T00:32:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:02.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:02.184049+0000) 2022-09-08T00:32:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:03.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:03.260475+0000) 2022-09-08T00:32:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:03.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:03.184227+0000) 2022-09-08T00:32:03.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:03 smithi055 ceph-mon[97033]: pgmap v5562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:03 smithi055 ceph-mon[97033]: pgmap v5563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:32:03.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:03 smithi055 ceph-mon[97033]: pgmap v5564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:32:03.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:32:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:03 smithi059 ceph-mon[97450]: pgmap v5562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:03 smithi059 ceph-mon[97450]: pgmap v5563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:32:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:03 smithi059 ceph-mon[97450]: pgmap v5564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:32:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:32:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:04.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:04.260727+0000) 2022-09-08T00:32:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:04.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:04.184460+0000) 2022-09-08T00:32:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:05.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:05.260956+0000) 2022-09-08T00:32:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:05.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:05.184693+0000) 2022-09-08T00:32:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:05 smithi055 ceph-mon[97033]: pgmap v5565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:32:05.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:32:05.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:32:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:05 smithi059 ceph-mon[97450]: pgmap v5565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:32:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:32:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:32:06.263 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:32:05.946+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:32:06.265 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:06.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:06.184943+0000) 2022-09-08T00:32:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:06.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:06.261172+0000) 2022-09-08T00:32:07.239 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:06 smithi055 ceph-mon[97033]: pgmap v5566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T00:32:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:06 smithi059 ceph-mon[97450]: pgmap v5566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T00:32:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:07.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:07.261345+0000) 2022-09-08T00:32:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:07.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:07.185156+0000) 2022-09-08T00:32:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:08.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:08.261493+0000) 2022-09-08T00:32:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:08.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:08.185310+0000) 2022-09-08T00:32:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:09.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:09.261656+0000) 2022-09-08T00:32:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:09.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:09.185542+0000) 2022-09-08T00:32:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:09 smithi055 ceph-mon[97033]: pgmap v5567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T00:32:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:09 smithi059 ceph-mon[97450]: pgmap v5567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T00:32:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:10.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:10.261859+0000) 2022-09-08T00:32:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:10.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:10.185748+0000) 2022-09-08T00:32:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:10 smithi055 ceph-mon[97033]: osdmap e433: 12 total, 11 up, 11 in 2022-09-08T00:32:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:10 smithi059 ceph-mon[97450]: osdmap e433: 12 total, 11 up, 11 in 2022-09-08T00:32:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:11.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:11.262077+0000) 2022-09-08T00:32:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:11.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:11.185926+0000) 2022-09-08T00:32:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:11 smithi055 ceph-mon[97033]: pgmap v5569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:32:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:11 smithi059 ceph-mon[97450]: pgmap v5569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:32:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:12.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:12.262252+0000) 2022-09-08T00:32:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:12.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:12.186126+0000) 2022-09-08T00:32:13.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:13.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:13.262476+0000) 2022-09-08T00:32:13.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:13.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:13.186338+0000) 2022-09-08T00:32:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:13 smithi055 ceph-mon[97033]: pgmap v5570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:32:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:13 smithi059 ceph-mon[97450]: pgmap v5570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:32:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:14.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:14.262737+0000) 2022-09-08T00:32:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:14.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:14.186602+0000) 2022-09-08T00:32:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:15.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:15.263011+0000) 2022-09-08T00:32:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:15.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:15.186774+0000) 2022-09-08T00:32:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:15 smithi055 ceph-mon[97033]: pgmap v5571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:32:15.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:15 smithi059 ceph-mon[97450]: pgmap v5571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:32:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:16.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:16.263159+0000) 2022-09-08T00:32:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:16.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:16.186960+0000) 2022-09-08T00:32:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:16 smithi055 ceph-mon[97033]: pgmap v5572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:32:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:16 smithi059 ceph-mon[97450]: pgmap v5572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:32:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:17.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:17.187163+0000) 2022-09-08T00:32:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:17.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:17.263362+0000) 2022-09-08T00:32:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:18.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:18.187382+0000) 2022-09-08T00:32:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:18.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:18.263576+0000) 2022-09-08T00:32:19.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:19.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:19.263816+0000) 2022-09-08T00:32:19.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:19.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:19.187597+0000) 2022-09-08T00:32:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:19 smithi055 ceph-mon[97033]: pgmap v5573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:32:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:19 smithi059 ceph-mon[97450]: pgmap v5573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:32:20.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:20.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:20.264045+0000) 2022-09-08T00:32:20.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:20.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:20.187790+0000) 2022-09-08T00:32:21.265 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:32:20.961+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:32:21.266 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:21.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:21.187999+0000) 2022-09-08T00:32:21.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:21.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:21.264252+0000) 2022-09-08T00:32:21.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:21 smithi055 ceph-mon[97033]: pgmap v5574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T00:32:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:21 smithi055 ceph-mon[97033]: osdmap e434: 12 total, 11 up, 11 in 2022-09-08T00:32:21.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:21 smithi059 ceph-mon[97450]: pgmap v5574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T00:32:21.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:21 smithi059 ceph-mon[97450]: osdmap e434: 12 total, 11 up, 11 in 2022-09-08T00:32:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:22.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:22.264483+0000) 2022-09-08T00:32:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:22.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:22.188171+0000) 2022-09-08T00:32:23.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:23.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:23.264719+0000) 2022-09-08T00:32:23.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:23.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:23.188368+0000) 2022-09-08T00:32:23.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:23 smithi055 ceph-mon[97033]: pgmap v5576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:32:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:23 smithi059 ceph-mon[97450]: pgmap v5576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:32:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:24.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:24.264891+0000) 2022-09-08T00:32:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:24.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:24.188589+0000) 2022-09-08T00:32:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:25.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:25.265122+0000) 2022-09-08T00:32:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:25.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:25.188781+0000) 2022-09-08T00:32:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:25 smithi055 ceph-mon[97033]: pgmap v5577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:32:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:25 smithi059 ceph-mon[97450]: pgmap v5577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:32:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:26.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:26.188936+0000) 2022-09-08T00:32:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:26.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:26.265282+0000) 2022-09-08T00:32:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:26 smithi055 ceph-mon[97033]: pgmap v5578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:32:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:26 smithi059 ceph-mon[97450]: pgmap v5578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:32:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:27.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:27.265542+0000) 2022-09-08T00:32:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:27.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:27.189144+0000) 2022-09-08T00:32:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:28.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:28.265760+0000) 2022-09-08T00:32:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:28.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:28.189364+0000) 2022-09-08T00:32:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:29.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:29.265946+0000) 2022-09-08T00:32:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:29.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:29.189526+0000) 2022-09-08T00:32:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:29 smithi055 ceph-mon[97033]: pgmap v5579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:32:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:29 smithi059 ceph-mon[97450]: pgmap v5579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:32:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:30.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:30.266096+0000) 2022-09-08T00:32:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:30.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:30.189771+0000) 2022-09-08T00:32:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:31.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:31.266322+0000) 2022-09-08T00:32:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:31.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:31.189958+0000) 2022-09-08T00:32:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:31 smithi055 ceph-mon[97033]: pgmap v5580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:32:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:31 smithi059 ceph-mon[97450]: pgmap v5580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:32:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:32.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:32.266553+0000) 2022-09-08T00:32:32.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:32.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:32.190144+0000) 2022-09-08T00:32:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:33.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:33.266786+0000) 2022-09-08T00:32:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:33.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:33.190338+0000) 2022-09-08T00:32:33.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:33 smithi055 ceph-mon[97033]: pgmap v5581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:32:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:33 smithi059 ceph-mon[97450]: pgmap v5581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:32:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:34.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:34.267027+0000) 2022-09-08T00:32:34.675 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:34.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:34.190567+0000) 2022-09-08T00:32:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:35.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:35.267168+0000) 2022-09-08T00:32:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:35.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:35.190748+0000) 2022-09-08T00:32:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:35 smithi055 ceph-mon[97033]: pgmap v5582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:35.795 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:35 smithi059 ceph-mon[97450]: pgmap v5582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:36.270 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:32:35.972+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:32:36.272 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:36.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:36.190962+0000) 2022-09-08T00:32:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:36.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:36.267371+0000) 2022-09-08T00:32:37.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:36 smithi055 ceph-mon[97033]: pgmap v5583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:32:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:36 smithi059 ceph-mon[97450]: pgmap v5583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:32:37.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:37.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:37.267558+0000) 2022-09-08T00:32:37.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:37.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:37.191215+0000) 2022-09-08T00:32:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:38.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:38.267803+0000) 2022-09-08T00:32:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:38.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:38.191488+0000) 2022-09-08T00:32:39.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:39.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:39.268035+0000) 2022-09-08T00:32:39.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:39.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:39.191763+0000) 2022-09-08T00:32:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:39 smithi055 ceph-mon[97033]: pgmap v5584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:39.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:39 smithi059 ceph-mon[97450]: pgmap v5584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:40.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:40.268234+0000) 2022-09-08T00:32:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:40.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:40.191937+0000) 2022-09-08T00:32:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:41.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:41.192187+0000) 2022-09-08T00:32:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:41.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:41.268480+0000) 2022-09-08T00:32:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:41 smithi055 ceph-mon[97033]: pgmap v5585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:32:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:41 smithi059 ceph-mon[97450]: pgmap v5585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:32:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:42.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:42.268738+0000) 2022-09-08T00:32:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:42.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:42.192439+0000) 2022-09-08T00:32:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:43.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:43.268980+0000) 2022-09-08T00:32:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:43.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:43.192692+0000) 2022-09-08T00:32:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:43 smithi055 ceph-mon[97033]: pgmap v5586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:43.797 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:43 smithi059 ceph-mon[97450]: pgmap v5586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:44.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:44.269186+0000) 2022-09-08T00:32:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:44.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:44.192941+0000) 2022-09-08T00:32:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:45.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:45.269339+0000) 2022-09-08T00:32:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:45.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:45.193226+0000) 2022-09-08T00:32:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:45 smithi055 ceph-mon[97033]: pgmap v5587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:45 smithi059 ceph-mon[97450]: pgmap v5587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:46.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:46.269553+0000) 2022-09-08T00:32:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:46.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:46.193485+0000) 2022-09-08T00:32:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:46 smithi059 ceph-mon[97450]: pgmap v5588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:32:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:46 smithi055 ceph-mon[97033]: pgmap v5588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:32:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:47.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:47.269765+0000) 2022-09-08T00:32:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:47.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:47.193754+0000) 2022-09-08T00:32:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:48.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:48.269923+0000) 2022-09-08T00:32:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:48.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:48.194017+0000) 2022-09-08T00:32:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:49.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:49.194270+0000) 2022-09-08T00:32:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:49.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:49.270095+0000) 2022-09-08T00:32:49.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:49 smithi055 ceph-mon[97033]: pgmap v5589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:49 smithi059 ceph-mon[97450]: pgmap v5589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:50.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:50.270362+0000) 2022-09-08T00:32:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:50.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:50.194492+0000) 2022-09-08T00:32:51.271 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:32:50.984+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:32:51.272 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:51.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:51.194749+0000) 2022-09-08T00:32:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:51.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:51.270591+0000) 2022-09-08T00:32:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:51 smithi055 ceph-mon[97033]: pgmap v5590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:32:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:51 smithi059 ceph-mon[97450]: pgmap v5590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:32:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:52.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:52.270772+0000) 2022-09-08T00:32:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:52.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:52.194993+0000) 2022-09-08T00:32:53.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:53.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:53.271019+0000) 2022-09-08T00:32:53.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:53.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:53.195247+0000) 2022-09-08T00:32:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:53 smithi055 ceph-mon[97033]: pgmap v5591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:53 smithi059 ceph-mon[97450]: pgmap v5591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:54.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:54.271140+0000) 2022-09-08T00:32:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:54.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:54.195472+0000) 2022-09-08T00:32:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:55.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:55.271276+0000) 2022-09-08T00:32:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:55.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:55.195699+0000) 2022-09-08T00:32:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:55 smithi055 ceph-mon[97033]: pgmap v5592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:55.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:55 smithi059 ceph-mon[97450]: pgmap v5592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:56.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:56.271426+0000) 2022-09-08T00:32:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:56.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:56.195895+0000) 2022-09-08T00:32:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:56 smithi055 ceph-mon[97033]: pgmap v5593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:32:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:56 smithi059 ceph-mon[97450]: pgmap v5593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:32:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:57.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:57.271689+0000) 2022-09-08T00:32:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:57.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:57.196070+0000) 2022-09-08T00:32:58.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:58.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:58.271810+0000) 2022-09-08T00:32:58.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:58.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:58.196283+0000) 2022-09-08T00:32:59.662 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:32:59.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:59.272055+0000) 2022-09-08T00:32:59.663 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:32:59.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:32:59.196541+0000) 2022-09-08T00:32:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:32:59 smithi055 ceph-mon[97033]: pgmap v5594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:32:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:32:59 smithi059 ceph-mon[97450]: pgmap v5594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:33:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:00.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:00.196705+0000) 2022-09-08T00:33:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:00.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:00.272281+0000) 2022-09-08T00:33:01.662 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:01.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:01.272429+0000) 2022-09-08T00:33:01.664 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:01.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:01.196835+0000) 2022-09-08T00:33:01.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:01 smithi055 ceph-mon[97033]: pgmap v5595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:33:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:01 smithi059 ceph-mon[97450]: pgmap v5595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:33:02.453 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:02.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:02.197038+0000) 2022-09-08T00:33:02.454 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:02.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:02.272706+0000) 2022-09-08T00:33:03.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:03.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:03.197287+0000) 2022-09-08T00:33:03.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:03.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:03.272934+0000) 2022-09-08T00:33:03.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:03 smithi055 ceph-mon[97033]: pgmap v5596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:33:03.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:33:03.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:33:03.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:33:03.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:33:03.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:33:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:03 smithi059 ceph-mon[97450]: pgmap v5596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:33:03.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:33:03.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:33:03.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:33:03.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:33:03.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:33:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:04.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:04.273168+0000) 2022-09-08T00:33:04.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:04.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:04.197515+0000) 2022-09-08T00:33:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:05.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:05.273400+0000) 2022-09-08T00:33:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:05.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:05.197693+0000) 2022-09-08T00:33:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:05 smithi055 ceph-mon[97033]: pgmap v5597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:33:05.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:33:05.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:33:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:05 smithi059 ceph-mon[97450]: pgmap v5597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:33:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:33:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:33:06.274 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:06.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:06.197865+0000) 2022-09-08T00:33:06.275 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:33:05.984+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:33:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:06.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:06.273609+0000) 2022-09-08T00:33:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:06 smithi055 ceph-mon[97033]: pgmap v5598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:33:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:06 smithi059 ceph-mon[97450]: pgmap v5598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:33:07.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:07.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:07.273850+0000) 2022-09-08T00:33:07.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:07.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:07.198136+0000) 2022-09-08T00:33:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:08.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:08.274050+0000) 2022-09-08T00:33:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:08.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:08.198324+0000) 2022-09-08T00:33:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:09.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:09.274193+0000) 2022-09-08T00:33:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:09.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:09.198505+0000) 2022-09-08T00:33:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:09 smithi055 ceph-mon[97033]: pgmap v5599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:33:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:09 smithi059 ceph-mon[97450]: pgmap v5599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:33:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:10.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:10.274364+0000) 2022-09-08T00:33:10.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:10.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:10.198733+0000) 2022-09-08T00:33:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:10 smithi055 ceph-mon[97033]: osdmap e435: 12 total, 11 up, 11 in 2022-09-08T00:33:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:10 smithi059 ceph-mon[97450]: osdmap e435: 12 total, 11 up, 11 in 2022-09-08T00:33:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:11.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:11.274585+0000) 2022-09-08T00:33:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:11.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:11.198871+0000) 2022-09-08T00:33:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:11 smithi055 ceph-mon[97033]: pgmap v5601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:33:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:11 smithi059 ceph-mon[97450]: pgmap v5601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:33:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:12.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:12.199094+0000) 2022-09-08T00:33:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:12.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:12.274857+0000) 2022-09-08T00:33:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:13.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:13.199319+0000) 2022-09-08T00:33:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:13.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:13.275007+0000) 2022-09-08T00:33:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:13 smithi055 ceph-mon[97033]: pgmap v5602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:33:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:13 smithi059 ceph-mon[97450]: pgmap v5602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:33:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:14.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:14.199538+0000) 2022-09-08T00:33:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:14.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:14.275234+0000) 2022-09-08T00:33:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:15.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:15.199757+0000) 2022-09-08T00:33:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:15.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:15.275477+0000) 2022-09-08T00:33:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:15 smithi055 ceph-mon[97033]: pgmap v5603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:33:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:15 smithi059 ceph-mon[97450]: pgmap v5603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:33:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:16.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:16.199963+0000) 2022-09-08T00:33:16.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:16.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:16.275749+0000) 2022-09-08T00:33:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:16 smithi055 ceph-mon[97033]: pgmap v5604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:33:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:16 smithi059 ceph-mon[97450]: pgmap v5604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:33:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:17.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:17.275921+0000) 2022-09-08T00:33:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:17.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:17.200153+0000) 2022-09-08T00:33:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:18.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:18.276150+0000) 2022-09-08T00:33:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:18.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:18.200359+0000) 2022-09-08T00:33:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:19.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:19.200609+0000) 2022-09-08T00:33:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:19.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:19.276403+0000) 2022-09-08T00:33:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:19 smithi055 ceph-mon[97033]: pgmap v5605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:33:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:19 smithi059 ceph-mon[97450]: pgmap v5605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:33:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:20.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:20.200822+0000) 2022-09-08T00:33:20.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:20.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:20.276685+0000) 2022-09-08T00:33:21.277 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:33:20.993+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:33:21.278 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:21.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:21.201037+0000) 2022-09-08T00:33:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:21.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:21.276900+0000) 2022-09-08T00:33:21.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:21 smithi055 ceph-mon[97033]: pgmap v5606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T00:33:21.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:21 smithi055 ceph-mon[97033]: osdmap e436: 12 total, 11 up, 11 in 2022-09-08T00:33:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:21 smithi059 ceph-mon[97450]: pgmap v5606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T00:33:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:21 smithi059 ceph-mon[97450]: osdmap e436: 12 total, 11 up, 11 in 2022-09-08T00:33:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:22.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:22.277022+0000) 2022-09-08T00:33:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:22.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:22.201258+0000) 2022-09-08T00:33:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:23.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:23.277146+0000) 2022-09-08T00:33:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:23.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:23.201532+0000) 2022-09-08T00:33:23.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:23 smithi055 ceph-mon[97033]: pgmap v5608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:33:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:23 smithi059 ceph-mon[97450]: pgmap v5608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:33:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:24.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:24.277348+0000) 2022-09-08T00:33:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:24.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:24.201762+0000) 2022-09-08T00:33:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:25.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:25.277604+0000) 2022-09-08T00:33:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:25.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:25.201978+0000) 2022-09-08T00:33:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:25 smithi055 ceph-mon[97033]: pgmap v5609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:33:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:25 smithi059 ceph-mon[97450]: pgmap v5609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:33:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:26.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:26.277802+0000) 2022-09-08T00:33:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:26.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:26.202210+0000) 2022-09-08T00:33:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:26 smithi055 ceph-mon[97033]: pgmap v5610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:33:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:26 smithi059 ceph-mon[97450]: pgmap v5610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:33:27.522 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:27.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:27.277986+0000) 2022-09-08T00:33:27.523 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:27.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:27.202382+0000) 2022-09-08T00:33:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:28.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:28.202573+0000) 2022-09-08T00:33:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:28.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:28.278219+0000) 2022-09-08T00:33:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:29.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:29.278438+0000) 2022-09-08T00:33:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:29.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:29.202794+0000) 2022-09-08T00:33:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:29 smithi055 ceph-mon[97033]: pgmap v5611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:33:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:29 smithi059 ceph-mon[97450]: pgmap v5611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:33:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:30.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:30.203019+0000) 2022-09-08T00:33:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:30.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:30.278677+0000) 2022-09-08T00:33:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:31.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:31.203297+0000) 2022-09-08T00:33:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:31.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:31.278893+0000) 2022-09-08T00:33:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:31 smithi055 ceph-mon[97033]: pgmap v5612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:33:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:31 smithi059 ceph-mon[97450]: pgmap v5612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:33:32.577 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:32.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:32.279110+0000) 2022-09-08T00:33:32.578 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:32.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:32.203543+0000) 2022-09-08T00:33:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:33.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:33.279323+0000) 2022-09-08T00:33:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:33.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:33.203784+0000) 2022-09-08T00:33:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:33 smithi055 ceph-mon[97033]: pgmap v5613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:33:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:33 smithi055 ceph-mon[97033]: pgmap v5614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:33:33.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:33 smithi055 ceph-mon[97033]: pgmap v5615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:33:33.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:33:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:33 smithi059 ceph-mon[97450]: pgmap v5613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:33:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:33 smithi059 ceph-mon[97450]: pgmap v5614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:33:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:33 smithi059 ceph-mon[97450]: pgmap v5615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:33:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:33:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:34.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:34.279563+0000) 2022-09-08T00:33:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:34.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:34.203938+0000) 2022-09-08T00:33:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:35.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:35.279788+0000) 2022-09-08T00:33:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:35.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:35.204145+0000) 2022-09-08T00:33:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:35 smithi055 ceph-mon[97033]: pgmap v5616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:33:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:35 smithi059 ceph-mon[97450]: pgmap v5616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:33:36.280 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:33:36.007+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:33:36.281 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:36.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:36.204404+0000) 2022-09-08T00:33:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:36.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:36.279912+0000) 2022-09-08T00:33:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:36 smithi055 ceph-mon[97033]: pgmap v5617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:33:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:36 smithi059 ceph-mon[97450]: pgmap v5617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:33:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:37.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:37.280129+0000) 2022-09-08T00:33:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:37.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:37.204661+0000) 2022-09-08T00:33:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:38.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:38.280272+0000) 2022-09-08T00:33:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:38.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:38.204886+0000) 2022-09-08T00:33:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:39.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:39.280394+0000) 2022-09-08T00:33:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:39.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:39.205118+0000) 2022-09-08T00:33:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:39 smithi055 ceph-mon[97033]: pgmap v5618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:33:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:39 smithi059 ceph-mon[97450]: pgmap v5618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:33:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:40.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:40.280577+0000) 2022-09-08T00:33:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:40.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:40.205395+0000) 2022-09-08T00:33:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:41.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:41.280775+0000) 2022-09-08T00:33:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:41.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:41.205659+0000) 2022-09-08T00:33:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:41 smithi055 ceph-mon[97033]: pgmap v5619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:33:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:41 smithi059 ceph-mon[97450]: pgmap v5619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:33:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:42.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:42.205811+0000) 2022-09-08T00:33:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:42.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:42.280908+0000) 2022-09-08T00:33:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:43.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:43.206014+0000) 2022-09-08T00:33:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:43.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:43.281135+0000) 2022-09-08T00:33:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:43 smithi055 ceph-mon[97033]: pgmap v5620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:33:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:43 smithi059 ceph-mon[97450]: pgmap v5620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:33:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:44.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:44.281386+0000) 2022-09-08T00:33:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:44.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:44.206274+0000) 2022-09-08T00:33:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:45.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:45.281615+0000) 2022-09-08T00:33:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:45.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:45.206524+0000) 2022-09-08T00:33:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:45 smithi055 ceph-mon[97033]: pgmap v5621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:33:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:45 smithi059 ceph-mon[97450]: pgmap v5621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:33:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:46.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:46.281849+0000) 2022-09-08T00:33:46.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:46.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:46.206763+0000) 2022-09-08T00:33:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:46 smithi055 ceph-mon[97033]: pgmap v5622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:33:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:46 smithi059 ceph-mon[97450]: pgmap v5622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:33:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:47.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:47.282061+0000) 2022-09-08T00:33:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:47.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:47.207042+0000) 2022-09-08T00:33:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:48.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:48.282231+0000) 2022-09-08T00:33:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:48.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:48.207237+0000) 2022-09-08T00:33:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:49.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:49.207481+0000) 2022-09-08T00:33:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:49.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:49.282414+0000) 2022-09-08T00:33:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:49 smithi055 ceph-mon[97033]: pgmap v5623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:33:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:49 smithi059 ceph-mon[97450]: pgmap v5623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:33:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:50.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:50.282631+0000) 2022-09-08T00:33:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:50.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:50.207751+0000) 2022-09-08T00:33:51.289 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:33:51.008+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:33:51.290 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:51.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:51.208011+0000) 2022-09-08T00:33:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:51.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:51.282890+0000) 2022-09-08T00:33:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:51 smithi055 ceph-mon[97033]: pgmap v5624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:33:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:51 smithi059 ceph-mon[97450]: pgmap v5624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:33:52.631 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:52.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:52.283079+0000) 2022-09-08T00:33:52.632 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:52.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:52.208194+0000) 2022-09-08T00:33:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:53.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:53.283257+0000) 2022-09-08T00:33:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:53.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:53.208440+0000) 2022-09-08T00:33:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:53 smithi055 ceph-mon[97033]: pgmap v5625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:33:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:53 smithi059 ceph-mon[97450]: pgmap v5625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:33:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:54.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:54.283468+0000) 2022-09-08T00:33:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:54.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:54.208632+0000) 2022-09-08T00:33:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:55.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:55.283715+0000) 2022-09-08T00:33:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:55.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:55.208899+0000) 2022-09-08T00:33:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:55 smithi055 ceph-mon[97033]: pgmap v5626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:33:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:55 smithi059 ceph-mon[97450]: pgmap v5626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:33:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:56.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:56.283861+0000) 2022-09-08T00:33:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:56.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:56.209085+0000) 2022-09-08T00:33:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:56 smithi055 ceph-mon[97033]: pgmap v5627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:33:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:56 smithi059 ceph-mon[97450]: pgmap v5627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:33:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:57.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:57.284070+0000) 2022-09-08T00:33:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:57.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:57.209323+0000) 2022-09-08T00:33:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:58.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:58.284291+0000) 2022-09-08T00:33:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:58.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:58.209522+0000) 2022-09-08T00:33:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:33:59.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:59.284513+0000) 2022-09-08T00:33:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:33:59.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:33:59.209759+0000) 2022-09-08T00:33:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:33:59 smithi055 ceph-mon[97033]: pgmap v5628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:33:59 smithi059 ceph-mon[97450]: pgmap v5628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:00.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:00.284688+0000) 2022-09-08T00:34:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:00.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:00.209934+0000) 2022-09-08T00:34:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:01.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:01.284814+0000) 2022-09-08T00:34:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:01.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:01.210122+0000) 2022-09-08T00:34:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:01 smithi055 ceph-mon[97033]: pgmap v5629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:34:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:01 smithi059 ceph-mon[97450]: pgmap v5629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:34:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:02.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:02.285004+0000) 2022-09-08T00:34:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:02.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:02.210352+0000) 2022-09-08T00:34:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:03.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:03.285224+0000) 2022-09-08T00:34:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:03.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:03.210628+0000) 2022-09-08T00:34:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:03 smithi055 ceph-mon[97033]: pgmap v5630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:03 smithi059 ceph-mon[97450]: pgmap v5630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:04.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:04.285456+0000) 2022-09-08T00:34:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:04.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:04.210907+0000) 2022-09-08T00:34:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:05.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:05.211109+0000) 2022-09-08T00:34:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:05.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:05.285742+0000) 2022-09-08T00:34:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:05 smithi055 ceph-mon[97033]: pgmap v5631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:34:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:34:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:05 smithi059 ceph-mon[97450]: pgmap v5631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:34:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:34:06.286 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:34:06.009+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:34:06.287 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:06.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:06.211370+0000) 2022-09-08T00:34:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:06.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:06.285965+0000) 2022-09-08T00:34:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:06 smithi055 ceph-mon[97033]: pgmap v5632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:34:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:06 smithi059 ceph-mon[97450]: pgmap v5632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:34:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:07.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:07.286206+0000) 2022-09-08T00:34:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:07.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:07.211669+0000) 2022-09-08T00:34:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:08.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:08.286416+0000) 2022-09-08T00:34:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:08.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:08.211935+0000) 2022-09-08T00:34:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:09.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:09.286677+0000) 2022-09-08T00:34:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:09.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:09.212160+0000) 2022-09-08T00:34:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:09 smithi055 ceph-mon[97033]: osdmap e437: 12 total, 11 up, 11 in 2022-09-08T00:34:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:09 smithi055 ceph-mon[97033]: pgmap v5634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:34:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:09 smithi059 ceph-mon[97450]: osdmap e437: 12 total, 11 up, 11 in 2022-09-08T00:34:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:09 smithi059 ceph-mon[97450]: pgmap v5634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:34:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:10.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:10.286944+0000) 2022-09-08T00:34:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:10.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:10.212388+0000) 2022-09-08T00:34:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:11.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:11.287108+0000) 2022-09-08T00:34:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:11.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:11.212690+0000) 2022-09-08T00:34:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:11 smithi055 ceph-mon[97033]: pgmap v5635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:34:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:11 smithi055 ceph-mon[97033]: osdmap e438: 12 total, 11 up, 11 in 2022-09-08T00:34:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:11 smithi059 ceph-mon[97450]: pgmap v5635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:34:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:11 smithi059 ceph-mon[97450]: osdmap e438: 12 total, 11 up, 11 in 2022-09-08T00:34:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:12.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:12.212951+0000) 2022-09-08T00:34:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:12.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:12.287332+0000) 2022-09-08T00:34:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:13.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:13.287549+0000) 2022-09-08T00:34:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:13.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:13.213120+0000) 2022-09-08T00:34:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:13 smithi055 ceph-mon[97033]: pgmap v5637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:34:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:13 smithi059 ceph-mon[97450]: pgmap v5637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:34:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:14.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:14.287812+0000) 2022-09-08T00:34:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:14.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:14.213242+0000) 2022-09-08T00:34:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:15.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:15.287963+0000) 2022-09-08T00:34:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:15.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:15.213454+0000) 2022-09-08T00:34:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:15 smithi055 ceph-mon[97033]: pgmap v5638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:34:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:15 smithi059 ceph-mon[97450]: pgmap v5638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:34:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:16.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:16.213719+0000) 2022-09-08T00:34:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:16.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:16.288177+0000) 2022-09-08T00:34:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:16 smithi055 ceph-mon[97033]: pgmap v5639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 251 B/s wr, 7 op/s 2022-09-08T00:34:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:16 smithi059 ceph-mon[97450]: pgmap v5639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 251 B/s wr, 7 op/s 2022-09-08T00:34:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:17.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:17.213869+0000) 2022-09-08T00:34:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:17.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:17.288396+0000) 2022-09-08T00:34:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:18.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:18.288661+0000) 2022-09-08T00:34:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:18.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:18.214050+0000) 2022-09-08T00:34:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:19.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:19.288842+0000) 2022-09-08T00:34:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:19.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:19.214292+0000) 2022-09-08T00:34:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:19 smithi055 ceph-mon[97033]: pgmap v5640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:34:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:19 smithi059 ceph-mon[97450]: pgmap v5640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:34:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:20.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:20.289035+0000) 2022-09-08T00:34:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:20.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:20.214526+0000) 2022-09-08T00:34:21.290 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:34:21.012+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:34:21.290 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:21.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:21.214751+0000) 2022-09-08T00:34:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:21.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:21.289264+0000) 2022-09-08T00:34:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:21 smithi055 ceph-mon[97033]: pgmap v5641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:34:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:21 smithi059 ceph-mon[97450]: pgmap v5641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:34:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:22.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:22.214987+0000) 2022-09-08T00:34:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:22.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:22.289479+0000) 2022-09-08T00:34:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:23.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:23.289736+0000) 2022-09-08T00:34:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:23.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:23.215253+0000) 2022-09-08T00:34:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:23 smithi055 ceph-mon[97033]: pgmap v5642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:34:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:23 smithi059 ceph-mon[97450]: pgmap v5642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T00:34:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:24.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:24.289952+0000) 2022-09-08T00:34:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:24.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:24.215500+0000) 2022-09-08T00:34:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:25.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:25.290177+0000) 2022-09-08T00:34:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:25.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:25.215748+0000) 2022-09-08T00:34:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:25 smithi055 ceph-mon[97033]: pgmap v5643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:34:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:25 smithi059 ceph-mon[97450]: pgmap v5643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:34:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:26.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:26.290339+0000) 2022-09-08T00:34:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:26.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:26.215950+0000) 2022-09-08T00:34:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:26 smithi055 ceph-mon[97033]: pgmap v5644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:34:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:26 smithi059 ceph-mon[97450]: pgmap v5644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:34:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:27.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:27.290562+0000) 2022-09-08T00:34:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:27.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:27.216118+0000) 2022-09-08T00:34:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:28.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:28.290787+0000) 2022-09-08T00:34:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:28.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:28.216344+0000) 2022-09-08T00:34:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:29.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:29.216509+0000) 2022-09-08T00:34:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:29.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:29.291009+0000) 2022-09-08T00:34:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:29 smithi055 ceph-mon[97033]: pgmap v5645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:29 smithi059 ceph-mon[97450]: pgmap v5645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:30.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:30.291237+0000) 2022-09-08T00:34:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:30.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:30.216780+0000) 2022-09-08T00:34:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:31.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:31.216983+0000) 2022-09-08T00:34:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:31.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:31.291446+0000) 2022-09-08T00:34:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:31 smithi055 ceph-mon[97033]: pgmap v5646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:34:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:31 smithi059 ceph-mon[97450]: pgmap v5646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:34:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:32.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:32.291594+0000) 2022-09-08T00:34:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:32.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:32.217226+0000) 2022-09-08T00:34:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:33.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:33.291818+0000) 2022-09-08T00:34:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:33.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:33.217417+0000) 2022-09-08T00:34:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:33 smithi055 ceph-mon[97033]: pgmap v5647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:34:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:34:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:34:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:33 smithi059 ceph-mon[97450]: pgmap v5647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:34:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:34:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:34:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:34.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:34.292058+0000) 2022-09-08T00:34:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:34.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:34.217613+0000) 2022-09-08T00:34:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:35.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:35.292300+0000) 2022-09-08T00:34:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:35.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:35.217840+0000) 2022-09-08T00:34:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:35 smithi055 ceph-mon[97033]: pgmap v5648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:35 smithi059 ceph-mon[97450]: pgmap v5648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:36.293 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:34:36.022+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:34:36.294 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:36.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:36.218028+0000) 2022-09-08T00:34:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:36.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:36.292476+0000) 2022-09-08T00:34:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:36 smithi055 ceph-mon[97033]: pgmap v5649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:34:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:36 smithi059 ceph-mon[97450]: pgmap v5649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:34:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:37.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:37.292677+0000) 2022-09-08T00:34:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:37.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:37.218263+0000) 2022-09-08T00:34:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:38.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:38.292899+0000) 2022-09-08T00:34:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:38.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:38.218500+0000) 2022-09-08T00:34:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:39.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:39.293052+0000) 2022-09-08T00:34:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:39.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:39.218786+0000) 2022-09-08T00:34:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:39 smithi055 ceph-mon[97033]: pgmap v5650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:39 smithi059 ceph-mon[97450]: pgmap v5650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:40.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:40.293289+0000) 2022-09-08T00:34:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:40.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:40.218938+0000) 2022-09-08T00:34:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:41.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:41.293518+0000) 2022-09-08T00:34:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:41.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:41.219113+0000) 2022-09-08T00:34:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:41 smithi055 ceph-mon[97033]: pgmap v5651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:34:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:41 smithi059 ceph-mon[97450]: pgmap v5651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:34:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:42.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:42.293764+0000) 2022-09-08T00:34:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:42.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:42.219313+0000) 2022-09-08T00:34:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:43.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:43.293997+0000) 2022-09-08T00:34:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:43.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:43.219544+0000) 2022-09-08T00:34:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:43 smithi055 ceph-mon[97033]: pgmap v5652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:43 smithi059 ceph-mon[97450]: pgmap v5652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:44.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:44.294231+0000) 2022-09-08T00:34:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:44.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:44.219750+0000) 2022-09-08T00:34:45.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:45.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:45.294459+0000) 2022-09-08T00:34:45.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:45.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:45.219948+0000) 2022-09-08T00:34:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:45 smithi055 ceph-mon[97033]: pgmap v5653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:45 smithi059 ceph-mon[97450]: pgmap v5653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:46.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:46.294627+0000) 2022-09-08T00:34:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:46.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:46.220138+0000) 2022-09-08T00:34:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:46 smithi055 ceph-mon[97033]: pgmap v5654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:34:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:46 smithi059 ceph-mon[97450]: pgmap v5654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:34:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:47.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:47.294797+0000) 2022-09-08T00:34:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:47.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:47.220330+0000) 2022-09-08T00:34:48.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:48.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:48.220532+0000) 2022-09-08T00:34:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:48.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:48.295024+0000) 2022-09-08T00:34:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:49.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:49.220758+0000) 2022-09-08T00:34:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:49.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:49.295252+0000) 2022-09-08T00:34:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:49 smithi055 ceph-mon[97033]: pgmap v5655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:49 smithi059 ceph-mon[97450]: pgmap v5655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:50.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:50.295434+0000) 2022-09-08T00:34:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:50.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:50.220990+0000) 2022-09-08T00:34:51.296 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:51.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:51.221251+0000) 2022-09-08T00:34:51.297 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:34:51.035+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:34:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:51.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:51.295623+0000) 2022-09-08T00:34:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:51 smithi055 ceph-mon[97033]: pgmap v5656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:34:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:51 smithi059 ceph-mon[97450]: pgmap v5656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:34:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:52.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:52.295880+0000) 2022-09-08T00:34:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:52.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:52.221489+0000) 2022-09-08T00:34:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:53.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:53.296054+0000) 2022-09-08T00:34:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:53.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:53.221739+0000) 2022-09-08T00:34:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:53 smithi055 ceph-mon[97033]: pgmap v5657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:53 smithi059 ceph-mon[97450]: pgmap v5657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:54.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:54.296311+0000) 2022-09-08T00:34:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:54.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:54.221949+0000) 2022-09-08T00:34:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:55.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:55.296552+0000) 2022-09-08T00:34:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:55.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:55.222201+0000) 2022-09-08T00:34:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:55 smithi055 ceph-mon[97033]: pgmap v5658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:55 smithi059 ceph-mon[97450]: pgmap v5658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:34:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:56.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:56.296802+0000) 2022-09-08T00:34:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:56.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:56.222456+0000) 2022-09-08T00:34:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:56 smithi055 ceph-mon[97033]: pgmap v5659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:34:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:56 smithi059 ceph-mon[97450]: pgmap v5659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:34:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:57.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:57.297092+0000) 2022-09-08T00:34:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:57.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:57.222694+0000) 2022-09-08T00:34:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:58.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:58.222881+0000) 2022-09-08T00:34:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:58.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:58.297285+0000) 2022-09-08T00:34:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:34:59.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:59.223111+0000) 2022-09-08T00:34:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:34:59.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:34:59.297552+0000) 2022-09-08T00:34:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:34:59 smithi055 ceph-mon[97033]: pgmap v5660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:34:59 smithi059 ceph-mon[97450]: pgmap v5660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:00.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:00.223333+0000) 2022-09-08T00:35:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:00.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:00.297794+0000) 2022-09-08T00:35:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:01.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:01.298002+0000) 2022-09-08T00:35:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:01.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:01.223546+0000) 2022-09-08T00:35:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:01 smithi055 ceph-mon[97033]: pgmap v5661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:35:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:01 smithi059 ceph-mon[97450]: pgmap v5661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:35:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:02.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:02.298264+0000) 2022-09-08T00:35:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:02.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:02.223744+0000) 2022-09-08T00:35:03.663 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:03.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:03.298502+0000) 2022-09-08T00:35:03.664 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:03.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:03.224052+0000) 2022-09-08T00:35:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:03 smithi055 ceph-mon[97033]: pgmap v5662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:35:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:03 smithi059 ceph-mon[97450]: pgmap v5662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:35:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:04.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:04.298685+0000) 2022-09-08T00:35:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:04.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:04.224295+0000) 2022-09-08T00:35:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:04 smithi055 ceph-mon[97033]: pgmap v5663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:35:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:04 smithi055 ceph-mon[97033]: pgmap v5664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:35:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:04 smithi059 ceph-mon[97450]: pgmap v5663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:35:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:04 smithi059 ceph-mon[97450]: pgmap v5664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:35:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:05.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:05.298919+0000) 2022-09-08T00:35:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:05.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:05.224517+0000) 2022-09-08T00:35:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:35:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:35:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:35:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:35:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:06.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:06.299150+0000) 2022-09-08T00:35:06.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:35:06.050+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:35:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:06.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:06.224781+0000) 2022-09-08T00:35:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:06 smithi055 ceph-mon[97033]: pgmap v5665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 27 op/s 2022-09-08T00:35:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:06 smithi059 ceph-mon[97450]: pgmap v5665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 27 op/s 2022-09-08T00:35:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:07.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:07.299358+0000) 2022-09-08T00:35:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:07.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:07.225018+0000) 2022-09-08T00:35:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:07 smithi055 ceph-mon[97033]: pgmap v5666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T00:35:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:07 smithi059 ceph-mon[97450]: pgmap v5666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T00:35:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:08.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:08.299586+0000) 2022-09-08T00:35:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:08.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:08.225242+0000) 2022-09-08T00:35:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:09.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:09.299851+0000) 2022-09-08T00:35:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:09.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:09.225492+0000) 2022-09-08T00:35:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:09 smithi055 ceph-mon[97033]: osdmap e439: 12 total, 11 up, 11 in 2022-09-08T00:35:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:09 smithi055 ceph-mon[97033]: pgmap v5668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 19 KiB/s rd, 85 KiB/s wr, 33 op/s 2022-09-08T00:35:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:09 smithi059 ceph-mon[97450]: osdmap e439: 12 total, 11 up, 11 in 2022-09-08T00:35:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:09 smithi059 ceph-mon[97450]: pgmap v5668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 19 KiB/s rd, 85 KiB/s wr, 33 op/s 2022-09-08T00:35:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:10.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:10.225709+0000) 2022-09-08T00:35:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:10.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:10.300090+0000) 2022-09-08T00:35:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:11.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:11.300313+0000) 2022-09-08T00:35:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:11.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:11.225924+0000) 2022-09-08T00:35:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:11 smithi055 ceph-mon[97033]: pgmap v5669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T00:35:12.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:11 smithi059 ceph-mon[97450]: pgmap v5669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T00:35:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:12.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:12.300598+0000) 2022-09-08T00:35:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:12.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:12.226137+0000) 2022-09-08T00:35:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:13.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:13.300847+0000) 2022-09-08T00:35:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:13.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:13.226370+0000) 2022-09-08T00:35:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:13 smithi055 ceph-mon[97033]: pgmap v5670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:35:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:13 smithi059 ceph-mon[97450]: pgmap v5670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:35:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:14.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:14.301085+0000) 2022-09-08T00:35:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:14.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:14.226618+0000) 2022-09-08T00:35:15.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:15.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:15.301322+0000) 2022-09-08T00:35:15.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:15.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:15.226844+0000) 2022-09-08T00:35:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:15 smithi055 ceph-mon[97033]: pgmap v5671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2022-09-08T00:35:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:15 smithi055 ceph-mon[97033]: osdmap e440: 12 total, 11 up, 11 in 2022-09-08T00:35:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:15 smithi059 ceph-mon[97450]: pgmap v5671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2022-09-08T00:35:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:15 smithi059 ceph-mon[97450]: osdmap e440: 12 total, 11 up, 11 in 2022-09-08T00:35:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:16.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:16.227043+0000) 2022-09-08T00:35:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:16.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:16.301556+0000) 2022-09-08T00:35:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:17.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:17.301761+0000) 2022-09-08T00:35:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:17.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:17.227247+0000) 2022-09-08T00:35:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:17 smithi055 ceph-mon[97033]: pgmap v5673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 251 B/s wr, 7 op/s 2022-09-08T00:35:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:17 smithi059 ceph-mon[97450]: pgmap v5673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 251 B/s wr, 7 op/s 2022-09-08T00:35:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:18.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:18.301951+0000) 2022-09-08T00:35:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:18.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:18.227453+0000) 2022-09-08T00:35:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:19.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:19.227731+0000) 2022-09-08T00:35:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:19.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:19.302166+0000) 2022-09-08T00:35:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:19 smithi055 ceph-mon[97033]: pgmap v5674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:35:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:19 smithi059 ceph-mon[97450]: pgmap v5674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:35:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:20.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:20.302375+0000) 2022-09-08T00:35:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:20.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:20.227919+0000) 2022-09-08T00:35:21.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:35:21.059+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:35:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:21.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:21.302612+0000) 2022-09-08T00:35:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:21.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:21.228152+0000) 2022-09-08T00:35:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:21 smithi055 ceph-mon[97033]: pgmap v5675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:35:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:21 smithi059 ceph-mon[97450]: pgmap v5675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:35:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:22.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:22.302840+0000) 2022-09-08T00:35:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:22.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:22.228367+0000) 2022-09-08T00:35:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:23.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:23.228678+0000) 2022-09-08T00:35:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:23.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:23.303012+0000) 2022-09-08T00:35:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:23 smithi055 ceph-mon[97033]: pgmap v5676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:35:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:23 smithi059 ceph-mon[97450]: pgmap v5676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:35:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:24.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:24.228898+0000) 2022-09-08T00:35:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:24.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:24.303251+0000) 2022-09-08T00:35:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:25.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:25.303483+0000) 2022-09-08T00:35:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:25.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:25.229151+0000) 2022-09-08T00:35:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:25 smithi055 ceph-mon[97033]: pgmap v5677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:35:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:25 smithi059 ceph-mon[97450]: pgmap v5677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:35:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:26.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:26.229364+0000) 2022-09-08T00:35:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:26.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:26.303683+0000) 2022-09-08T00:35:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:27.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:27.303894+0000) 2022-09-08T00:35:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:27.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:27.229490+0000) 2022-09-08T00:35:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:27 smithi055 ceph-mon[97033]: pgmap v5678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:35:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:27 smithi059 ceph-mon[97450]: pgmap v5678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:35:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:28.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:28.304104+0000) 2022-09-08T00:35:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:28.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:28.229692+0000) 2022-09-08T00:35:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:29.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:29.304319+0000) 2022-09-08T00:35:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:29.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:29.229928+0000) 2022-09-08T00:35:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:29 smithi055 ceph-mon[97033]: pgmap v5679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:29 smithi059 ceph-mon[97450]: pgmap v5679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:30.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:30.304580+0000) 2022-09-08T00:35:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:30.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:30.230142+0000) 2022-09-08T00:35:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:31.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:31.304755+0000) 2022-09-08T00:35:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:31.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:31.230392+0000) 2022-09-08T00:35:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:31 smithi055 ceph-mon[97033]: pgmap v5680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:35:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:31 smithi059 ceph-mon[97450]: pgmap v5680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:35:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:32.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:32.230629+0000) 2022-09-08T00:35:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:32.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:32.304980+0000) 2022-09-08T00:35:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:33.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:33.230894+0000) 2022-09-08T00:35:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:33.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:33.305236+0000) 2022-09-08T00:35:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:33 smithi055 ceph-mon[97033]: pgmap v5681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:33 smithi059 ceph-mon[97450]: pgmap v5681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:34.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:34.231138+0000) 2022-09-08T00:35:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:34.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:34.305472+0000) 2022-09-08T00:35:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:35.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:35.305721+0000) 2022-09-08T00:35:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:35.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:35.231401+0000) 2022-09-08T00:35:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:35 smithi055 ceph-mon[97033]: pgmap v5682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T00:35:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:35 smithi059 ceph-mon[97450]: pgmap v5682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T00:35:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:36.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:36.305978+0000) 2022-09-08T00:35:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:36.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:36.231616+0000) 2022-09-08T00:35:36.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:35:36.067+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:35:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:37.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:37.306228+0000) 2022-09-08T00:35:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:37.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:37.231852+0000) 2022-09-08T00:35:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:37 smithi055 ceph-mon[97033]: pgmap v5683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T00:35:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:37 smithi059 ceph-mon[97450]: pgmap v5683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T00:35:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:38.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:38.306505+0000) 2022-09-08T00:35:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:38.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:38.232096+0000) 2022-09-08T00:35:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:39.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:39.306755+0000) 2022-09-08T00:35:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:39.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:39.232354+0000) 2022-09-08T00:35:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:39 smithi055 ceph-mon[97033]: pgmap v5684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:39 smithi059 ceph-mon[97450]: pgmap v5684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:40.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:40.306992+0000) 2022-09-08T00:35:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:40.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:40.232588+0000) 2022-09-08T00:35:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:41.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:41.232819+0000) 2022-09-08T00:35:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:41.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:41.307253+0000) 2022-09-08T00:35:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:41 smithi055 ceph-mon[97033]: pgmap v5685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:35:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:41 smithi059 ceph-mon[97450]: pgmap v5685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:35:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:42.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:42.307504+0000) 2022-09-08T00:35:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:42.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:42.233044+0000) 2022-09-08T00:35:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:43.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:43.307778+0000) 2022-09-08T00:35:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:43.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:43.233252+0000) 2022-09-08T00:35:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:43 smithi055 ceph-mon[97033]: pgmap v5686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:43 smithi059 ceph-mon[97450]: pgmap v5686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:44.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:44.308026+0000) 2022-09-08T00:35:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:44.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:44.233437+0000) 2022-09-08T00:35:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:45.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:45.233667+0000) 2022-09-08T00:35:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:45.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:45.308303+0000) 2022-09-08T00:35:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:45 smithi055 ceph-mon[97033]: pgmap v5687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T00:35:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:45 smithi059 ceph-mon[97450]: pgmap v5687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T00:35:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:46.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:46.233896+0000) 2022-09-08T00:35:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:46.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:46.308524+0000) 2022-09-08T00:35:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:47.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:47.308751+0000) 2022-09-08T00:35:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:47.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:47.234087+0000) 2022-09-08T00:35:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:47 smithi055 ceph-mon[97033]: pgmap v5688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T00:35:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:47 smithi059 ceph-mon[97450]: pgmap v5688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T00:35:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:48.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:48.308979+0000) 2022-09-08T00:35:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:48.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:48.234323+0000) 2022-09-08T00:35:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:49.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:49.309188+0000) 2022-09-08T00:35:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:49.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:49.234542+0000) 2022-09-08T00:35:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:49 smithi055 ceph-mon[97033]: pgmap v5689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:49 smithi059 ceph-mon[97450]: pgmap v5689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:50.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:50.309355+0000) 2022-09-08T00:35:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:50.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:50.234796+0000) 2022-09-08T00:35:51.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:35:51.082+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:35:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:51.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:51.309566+0000) 2022-09-08T00:35:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:51.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:51.234959+0000) 2022-09-08T00:35:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:51 smithi055 ceph-mon[97033]: pgmap v5690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:35:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:51 smithi059 ceph-mon[97450]: pgmap v5690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:35:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:52.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:52.309709+0000) 2022-09-08T00:35:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:52.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:52.235190+0000) 2022-09-08T00:35:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:53.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:53.309875+0000) 2022-09-08T00:35:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:53.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:53.235441+0000) 2022-09-08T00:35:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:53 smithi055 ceph-mon[97033]: pgmap v5691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:53 smithi059 ceph-mon[97450]: pgmap v5691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:35:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:54.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:54.310058+0000) 2022-09-08T00:35:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:54.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:54.235655+0000) 2022-09-08T00:35:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:55.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:55.310270+0000) 2022-09-08T00:35:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:55.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:55.235853+0000) 2022-09-08T00:35:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:55 smithi055 ceph-mon[97033]: pgmap v5692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T00:35:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:55 smithi059 ceph-mon[97450]: pgmap v5692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T00:35:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:56.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:56.310490+0000) 2022-09-08T00:35:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:56.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:56.236093+0000) 2022-09-08T00:35:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:57.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:57.310732+0000) 2022-09-08T00:35:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:57.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:57.236251+0000) 2022-09-08T00:35:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:57 smithi055 ceph-mon[97033]: pgmap v5693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T00:35:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:57 smithi059 ceph-mon[97450]: pgmap v5693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T00:35:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:58.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:58.310967+0000) 2022-09-08T00:35:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:58.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:58.236451+0000) 2022-09-08T00:35:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:35:59.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:59.311221+0000) 2022-09-08T00:35:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:35:59.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:35:59.236736+0000) 2022-09-08T00:36:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:35:59 smithi055 ceph-mon[97033]: pgmap v5694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:35:59 smithi059 ceph-mon[97450]: pgmap v5694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:00.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:00.311540+0000) 2022-09-08T00:36:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:00.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:00.236950+0000) 2022-09-08T00:36:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:01.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:01.311771+0000) 2022-09-08T00:36:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:01.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:01.237146+0000) 2022-09-08T00:36:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:01 smithi055 ceph-mon[97033]: pgmap v5695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:36:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:01 smithi059 ceph-mon[97450]: pgmap v5695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:36:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:02.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:02.311971+0000) 2022-09-08T00:36:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:02.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:02.237370+0000) 2022-09-08T00:36:03.312 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:03.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:03.237652+0000) 2022-09-08T00:36:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:03.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:03.312212+0000) 2022-09-08T00:36:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:03 smithi055 ceph-mon[97033]: pgmap v5696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:36:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:36:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:36:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:03 smithi059 ceph-mon[97450]: pgmap v5696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:36:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:36:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:36:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:04.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:04.312433+0000) 2022-09-08T00:36:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:04.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:04.237953+0000) 2022-09-08T00:36:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:36:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:36:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:36:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:36:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:05.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:05.312723+0000) 2022-09-08T00:36:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:05.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:05.238109+0000) 2022-09-08T00:36:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:05 smithi055 ceph-mon[97033]: pgmap v5697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T00:36:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:05 smithi059 ceph-mon[97450]: pgmap v5697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T00:36:06.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:36:06.096+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:36:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:06.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:06.312909+0000) 2022-09-08T00:36:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:06.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:06.238304+0000) 2022-09-08T00:36:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:07.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:07.313093+0000) 2022-09-08T00:36:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:07.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:07.238497+0000) 2022-09-08T00:36:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:07 smithi055 ceph-mon[97033]: pgmap v5698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T00:36:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:07 smithi059 ceph-mon[97450]: pgmap v5698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T00:36:08.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:08.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:08.313310+0000) 2022-09-08T00:36:08.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:08.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:08.238676+0000) 2022-09-08T00:36:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:09.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:09.313527+0000) 2022-09-08T00:36:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:09.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:09.238858+0000) 2022-09-08T00:36:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:09 smithi055 ceph-mon[97033]: pgmap v5699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:36:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:09 smithi055 ceph-mon[97033]: osdmap e441: 12 total, 11 up, 11 in 2022-09-08T00:36:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:09 smithi059 ceph-mon[97450]: pgmap v5699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:36:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:09 smithi059 ceph-mon[97450]: osdmap e441: 12 total, 11 up, 11 in 2022-09-08T00:36:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:10.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:10.313730+0000) 2022-09-08T00:36:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:10.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:10.239042+0000) 2022-09-08T00:36:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:11.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:11.313943+0000) 2022-09-08T00:36:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:11.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:11.239188+0000) 2022-09-08T00:36:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:11 smithi055 ceph-mon[97033]: pgmap v5701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:36:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:11 smithi059 ceph-mon[97450]: pgmap v5701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:36:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:12.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:12.314155+0000) 2022-09-08T00:36:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:12.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:12.239350+0000) 2022-09-08T00:36:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:13.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:13.314382+0000) 2022-09-08T00:36:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:13.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:13.239615+0000) 2022-09-08T00:36:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:13 smithi055 ceph-mon[97033]: pgmap v5702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:36:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:13 smithi059 ceph-mon[97450]: pgmap v5702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:36:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:14.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:14.239848+0000) 2022-09-08T00:36:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:14.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:14.314608+0000) 2022-09-08T00:36:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:15.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:15.240046+0000) 2022-09-08T00:36:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:15.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:15.314823+0000) 2022-09-08T00:36:16.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:15 smithi055 ceph-mon[97033]: pgmap v5703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2022-09-08T00:36:16.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:15 smithi055 ceph-mon[97033]: osdmap e442: 12 total, 11 up, 11 in 2022-09-08T00:36:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:15 smithi059 ceph-mon[97450]: pgmap v5703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2022-09-08T00:36:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:15 smithi059 ceph-mon[97450]: osdmap e442: 12 total, 11 up, 11 in 2022-09-08T00:36:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:16.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:16.315035+0000) 2022-09-08T00:36:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:16.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:16.240221+0000) 2022-09-08T00:36:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:17.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:17.240468+0000) 2022-09-08T00:36:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:17.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:17.315264+0000) 2022-09-08T00:36:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:17 smithi055 ceph-mon[97033]: pgmap v5705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:36:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:17 smithi059 ceph-mon[97450]: pgmap v5705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:36:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:18.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:18.315472+0000) 2022-09-08T00:36:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:18.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:18.240721+0000) 2022-09-08T00:36:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:19.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:19.315709+0000) 2022-09-08T00:36:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:19.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:19.240938+0000) 2022-09-08T00:36:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:19 smithi055 ceph-mon[97033]: pgmap v5706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:36:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:19 smithi059 ceph-mon[97450]: pgmap v5706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:36:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:20.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:20.315914+0000) 2022-09-08T00:36:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:20.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:20.241118+0000) 2022-09-08T00:36:21.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:36:21.108+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:36:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:21.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:21.316072+0000) 2022-09-08T00:36:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:21.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:21.241274+0000) 2022-09-08T00:36:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:21 smithi055 ceph-mon[97033]: pgmap v5707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:36:22.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:21 smithi059 ceph-mon[97450]: pgmap v5707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:36:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:22.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:22.316199+0000) 2022-09-08T00:36:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:22.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:22.241464+0000) 2022-09-08T00:36:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:23.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:23.316415+0000) 2022-09-08T00:36:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:23.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:23.241719+0000) 2022-09-08T00:36:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:23 smithi055 ceph-mon[97033]: pgmap v5708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:36:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:23 smithi059 ceph-mon[97450]: pgmap v5708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:36:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:24.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:24.316604+0000) 2022-09-08T00:36:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:24.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:24.241931+0000) 2022-09-08T00:36:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:25.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:25.316704+0000) 2022-09-08T00:36:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:25.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:25.242188+0000) 2022-09-08T00:36:26.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:25 smithi055 ceph-mon[97033]: pgmap v5709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:36:26.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:25 smithi059 ceph-mon[97450]: pgmap v5709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:36:26.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:26.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:26.316877+0000) 2022-09-08T00:36:26.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:26.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:26.242417+0000) 2022-09-08T00:36:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:27.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:27.317027+0000) 2022-09-08T00:36:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:27.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:27.242615+0000) 2022-09-08T00:36:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:27 smithi055 ceph-mon[97033]: pgmap v5710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:36:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:27 smithi059 ceph-mon[97450]: pgmap v5710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:36:28.322 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:28.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:28.242846+0000) 2022-09-08T00:36:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:28.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:28.317263+0000) 2022-09-08T00:36:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:29.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:29.317428+0000) 2022-09-08T00:36:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:29.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:29.243072+0000) 2022-09-08T00:36:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:29 smithi055 ceph-mon[97033]: pgmap v5711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:29 smithi059 ceph-mon[97450]: pgmap v5711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:30.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:30.317662+0000) 2022-09-08T00:36:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:30.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:30.243305+0000) 2022-09-08T00:36:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:31.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:31.317815+0000) 2022-09-08T00:36:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:31.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:31.243557+0000) 2022-09-08T00:36:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:31 smithi055 ceph-mon[97033]: pgmap v5712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:36:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:31 smithi059 ceph-mon[97450]: pgmap v5712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:36:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:32.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:32.318020+0000) 2022-09-08T00:36:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:32.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:32.243773+0000) 2022-09-08T00:36:33.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:33.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:33.318198+0000) 2022-09-08T00:36:33.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:33.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:33.243978+0000) 2022-09-08T00:36:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:34 smithi055 ceph-mon[97033]: pgmap v5713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:34 smithi055 ceph-mon[97033]: pgmap v5714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:36:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:34 smithi055 ceph-mon[97033]: pgmap v5715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 4 op/s 2022-09-08T00:36:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:36:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:34 smithi059 ceph-mon[97450]: pgmap v5713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:34 smithi059 ceph-mon[97450]: pgmap v5714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:36:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:34 smithi059 ceph-mon[97450]: pgmap v5715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 4 op/s 2022-09-08T00:36:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:36:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:34.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:34.318408+0000) 2022-09-08T00:36:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:34.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:34.244198+0000) 2022-09-08T00:36:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:35.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:35.318666+0000) 2022-09-08T00:36:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:35.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:35.244404+0000) 2022-09-08T00:36:36.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:36:36.117+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:36:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:36.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:36.318886+0000) 2022-09-08T00:36:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:36.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:36.244623+0000) 2022-09-08T00:36:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:36 smithi055 ceph-mon[97033]: pgmap v5716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:36:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:36 smithi059 ceph-mon[97450]: pgmap v5716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:36:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:37.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:37.319085+0000) 2022-09-08T00:36:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:37.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:37.244852+0000) 2022-09-08T00:36:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:38 smithi055 ceph-mon[97033]: pgmap v5717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:36:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:38 smithi059 ceph-mon[97450]: pgmap v5717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:36:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:38.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:38.319292+0000) 2022-09-08T00:36:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:38.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:38.245030+0000) 2022-09-08T00:36:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:39.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:39.319518+0000) 2022-09-08T00:36:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:39.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:39.245245+0000) 2022-09-08T00:36:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:40 smithi055 ceph-mon[97033]: pgmap v5718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:36:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:40 smithi059 ceph-mon[97450]: pgmap v5718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:36:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:40.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:40.319784+0000) 2022-09-08T00:36:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:40.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:40.245510+0000) 2022-09-08T00:36:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:41.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:41.319978+0000) 2022-09-08T00:36:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:41.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:41.245669+0000) 2022-09-08T00:36:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:41 smithi055 ceph-mon[97033]: pgmap v5719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:36:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:41 smithi059 ceph-mon[97450]: pgmap v5719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:36:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:42.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:42.245863+0000) 2022-09-08T00:36:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:42.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:42.320175+0000) 2022-09-08T00:36:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:43.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:43.246082+0000) 2022-09-08T00:36:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:43.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:43.320405+0000) 2022-09-08T00:36:44.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:44 smithi055 ceph-mon[97033]: pgmap v5720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:36:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:44 smithi059 ceph-mon[97450]: pgmap v5720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:36:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:44.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:44.246279+0000) 2022-09-08T00:36:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:44.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:44.320615+0000) 2022-09-08T00:36:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:45.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:45.320768+0000) 2022-09-08T00:36:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:45.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:45.246507+0000) 2022-09-08T00:36:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:46 smithi055 ceph-mon[97033]: pgmap v5721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:36:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:46 smithi059 ceph-mon[97450]: pgmap v5721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:36:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:46.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:46.320927+0000) 2022-09-08T00:36:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:46.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:46.246753+0000) 2022-09-08T00:36:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:47.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:47.321126+0000) 2022-09-08T00:36:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:47.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:47.246903+0000) 2022-09-08T00:36:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:48 smithi055 ceph-mon[97033]: pgmap v5722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:48 smithi059 ceph-mon[97450]: pgmap v5722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:48.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:48.321364+0000) 2022-09-08T00:36:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:48.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:48.247110+0000) 2022-09-08T00:36:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:49.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:49.321582+0000) 2022-09-08T00:36:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:49.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:49.247361+0000) 2022-09-08T00:36:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:50 smithi055 ceph-mon[97033]: pgmap v5723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:50.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:50 smithi059 ceph-mon[97450]: pgmap v5723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:50.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:50.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:50.321749+0000) 2022-09-08T00:36:50.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:50.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:50.247601+0000) 2022-09-08T00:36:51.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:36:51.124+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:36:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:51.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:51.321882+0000) 2022-09-08T00:36:51.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:51.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:51.247858+0000) 2022-09-08T00:36:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:51 smithi055 ceph-mon[97033]: pgmap v5724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:36:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:51 smithi059 ceph-mon[97450]: pgmap v5724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:36:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:52.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:52.322025+0000) 2022-09-08T00:36:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:52.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:52.248087+0000) 2022-09-08T00:36:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:53.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:53.322263+0000) 2022-09-08T00:36:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:53.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:53.248288+0000) 2022-09-08T00:36:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:54 smithi055 ceph-mon[97033]: pgmap v5725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:54 smithi059 ceph-mon[97450]: pgmap v5725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:54.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:54.322468+0000) 2022-09-08T00:36:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:54.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:54.248511+0000) 2022-09-08T00:36:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:55.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:55.322728+0000) 2022-09-08T00:36:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:55.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:55.248743+0000) 2022-09-08T00:36:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:56 smithi055 ceph-mon[97033]: pgmap v5726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:36:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:56 smithi059 ceph-mon[97450]: pgmap v5726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:36:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:56.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:56.322920+0000) 2022-09-08T00:36:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:56.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:56.248908+0000) 2022-09-08T00:36:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:57.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:57.323139+0000) 2022-09-08T00:36:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:57.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:57.249137+0000) 2022-09-08T00:36:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:36:58 smithi055 ceph-mon[97033]: pgmap v5727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:36:58 smithi059 ceph-mon[97450]: pgmap v5727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:36:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:58.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:58.323357+0000) 2022-09-08T00:36:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:58.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:58.249364+0000) 2022-09-08T00:36:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:36:59.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:59.249606+0000) 2022-09-08T00:36:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:36:59.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:36:59.323531+0000) 2022-09-08T00:37:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:00 smithi055 ceph-mon[97033]: pgmap v5728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:00 smithi059 ceph-mon[97450]: pgmap v5728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:00.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:00.323725+0000) 2022-09-08T00:37:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:00.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:00.249874+0000) 2022-09-08T00:37:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:01.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:01.323867+0000) 2022-09-08T00:37:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:01.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:01.250009+0000) 2022-09-08T00:37:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:01 smithi055 ceph-mon[97033]: pgmap v5729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:37:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:01 smithi059 ceph-mon[97450]: pgmap v5729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:37:02.664 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:02.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:02.324080+0000) 2022-09-08T00:37:02.665 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:02.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:02.250182+0000) 2022-09-08T00:37:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:03.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:03.324294+0000) 2022-09-08T00:37:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:03.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:03.250418+0000) 2022-09-08T00:37:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:04 smithi055 ceph-mon[97033]: pgmap v5730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:04.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:04 smithi059 ceph-mon[97450]: pgmap v5730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:04.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:04.324458+0000) 2022-09-08T00:37:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:04.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:04.250713+0000) 2022-09-08T00:37:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:37:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:37:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:37:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:37:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:05.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:05.324725+0000) 2022-09-08T00:37:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:05.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:05.250955+0000) 2022-09-08T00:37:06.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:37:06.137+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:37:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:06.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:06.324905+0000) 2022-09-08T00:37:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:06.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:06.251189+0000) 2022-09-08T00:37:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:06 smithi055 ceph-mon[97033]: pgmap v5731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:37:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:06 smithi059 ceph-mon[97450]: pgmap v5731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:37:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:07.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:07.325137+0000) 2022-09-08T00:37:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:07.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:07.251421+0000) 2022-09-08T00:37:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:08 smithi055 ceph-mon[97033]: pgmap v5732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:37:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:08 smithi059 ceph-mon[97450]: pgmap v5732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:37:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:08.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:08.325382+0000) 2022-09-08T00:37:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:08.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:08.251605+0000) 2022-09-08T00:37:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:09.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:09.325610+0000) 2022-09-08T00:37:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:09.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:09.251851+0000) 2022-09-08T00:37:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:10 smithi055 ceph-mon[97033]: pgmap v5733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:37:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:10 smithi055 ceph-mon[97033]: osdmap e443: 12 total, 11 up, 11 in 2022-09-08T00:37:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:10 smithi059 ceph-mon[97450]: pgmap v5733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:37:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:10 smithi059 ceph-mon[97450]: osdmap e443: 12 total, 11 up, 11 in 2022-09-08T00:37:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:10.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:10.325834+0000) 2022-09-08T00:37:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:10.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:10.252105+0000) 2022-09-08T00:37:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:11.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:11.326090+0000) 2022-09-08T00:37:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:11.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:11.252277+0000) 2022-09-08T00:37:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:11 smithi055 ceph-mon[97033]: pgmap v5735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:37:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:11 smithi059 ceph-mon[97450]: pgmap v5735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:37:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:12.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:12.326304+0000) 2022-09-08T00:37:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:12.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:12.252508+0000) 2022-09-08T00:37:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:13.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:13.326519+0000) 2022-09-08T00:37:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:13.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:13.252652+0000) 2022-09-08T00:37:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:14 smithi055 ceph-mon[97033]: pgmap v5736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:37:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:14 smithi059 ceph-mon[97450]: pgmap v5736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:37:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:14.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:14.326786+0000) 2022-09-08T00:37:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:14.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:14.252863+0000) 2022-09-08T00:37:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:15.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:15.326951+0000) 2022-09-08T00:37:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:15.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:15.253093+0000) 2022-09-08T00:37:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:16 smithi055 ceph-mon[97033]: pgmap v5737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:37:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:16 smithi055 ceph-mon[97033]: osdmap e444: 12 total, 11 up, 11 in 2022-09-08T00:37:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:16 smithi059 ceph-mon[97450]: pgmap v5737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:37:16.587 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:16 smithi059 ceph-mon[97450]: osdmap e444: 12 total, 11 up, 11 in 2022-09-08T00:37:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:16.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:16.327119+0000) 2022-09-08T00:37:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:16.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:16.253257+0000) 2022-09-08T00:37:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:17.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:17.327347+0000) 2022-09-08T00:37:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:17.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:17.253476+0000) 2022-09-08T00:37:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:18 smithi055 ceph-mon[97033]: pgmap v5739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:37:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:18 smithi059 ceph-mon[97450]: pgmap v5739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:37:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:18.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:18.327600+0000) 2022-09-08T00:37:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:18.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:18.253740+0000) 2022-09-08T00:37:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:19.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:19.327856+0000) 2022-09-08T00:37:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:19.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:19.253939+0000) 2022-09-08T00:37:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:20 smithi055 ceph-mon[97033]: pgmap v5740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:37:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:20 smithi059 ceph-mon[97450]: pgmap v5740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:37:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:20.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:20.328138+0000) 2022-09-08T00:37:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:20.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:20.254181+0000) 2022-09-08T00:37:21.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:37:21.154+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:37:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:21.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:21.328414+0000) 2022-09-08T00:37:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:21.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:21.254432+0000) 2022-09-08T00:37:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:21 smithi055 ceph-mon[97033]: pgmap v5741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:37:22.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:21 smithi059 ceph-mon[97450]: pgmap v5741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:37:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:22.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:22.328625+0000) 2022-09-08T00:37:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:22.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:22.254676+0000) 2022-09-08T00:37:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:23.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:23.328852+0000) 2022-09-08T00:37:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:23.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:23.254862+0000) 2022-09-08T00:37:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:24 smithi055 ceph-mon[97033]: pgmap v5742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:37:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:24 smithi059 ceph-mon[97450]: pgmap v5742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:37:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:24.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:24.329069+0000) 2022-09-08T00:37:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:24.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:24.255014+0000) 2022-09-08T00:37:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:25.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:25.329312+0000) 2022-09-08T00:37:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:25.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:25.255237+0000) 2022-09-08T00:37:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:26 smithi055 ceph-mon[97033]: pgmap v5743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:37:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:26 smithi059 ceph-mon[97450]: pgmap v5743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:37:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:26.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:26.329561+0000) 2022-09-08T00:37:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:26.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:26.255501+0000) 2022-09-08T00:37:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:27.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:27.329782+0000) 2022-09-08T00:37:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:27.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:27.255742+0000) 2022-09-08T00:37:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:28 smithi055 ceph-mon[97033]: pgmap v5744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:37:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:28 smithi059 ceph-mon[97450]: pgmap v5744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:37:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:28.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:28.330013+0000) 2022-09-08T00:37:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:28.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:28.255932+0000) 2022-09-08T00:37:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:29.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:29.330251+0000) 2022-09-08T00:37:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:29.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:29.256163+0000) 2022-09-08T00:37:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:30 smithi055 ceph-mon[97033]: pgmap v5745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:30 smithi059 ceph-mon[97450]: pgmap v5745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:30.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:30.330501+0000) 2022-09-08T00:37:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:30.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:30.256386+0000) 2022-09-08T00:37:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:31.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:31.330754+0000) 2022-09-08T00:37:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:31.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:31.256696+0000) 2022-09-08T00:37:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:31 smithi055 ceph-mon[97033]: pgmap v5746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:37:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:31 smithi059 ceph-mon[97450]: pgmap v5746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:37:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:32.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:32.330988+0000) 2022-09-08T00:37:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:32.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:32.256870+0000) 2022-09-08T00:37:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:33.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:33.331190+0000) 2022-09-08T00:37:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:33.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:33.256998+0000) 2022-09-08T00:37:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:34 smithi055 ceph-mon[97033]: pgmap v5747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:37:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:37:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:37:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:34 smithi059 ceph-mon[97450]: pgmap v5747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:37:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:37:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:37:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:34.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:34.331402+0000) 2022-09-08T00:37:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:34.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:34.257220+0000) 2022-09-08T00:37:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:35.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:35.331594+0000) 2022-09-08T00:37:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:35.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:35.257446+0000) 2022-09-08T00:37:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:36 smithi055 ceph-mon[97033]: pgmap v5748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:37:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:36 smithi059 ceph-mon[97450]: pgmap v5748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:37:36.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:37:36.167+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:37:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:36.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:36.331818+0000) 2022-09-08T00:37:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:36.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:36.257689+0000) 2022-09-08T00:37:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:37.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:37.332051+0000) 2022-09-08T00:37:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:37.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:37.257912+0000) 2022-09-08T00:37:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:38 smithi055 ceph-mon[97033]: pgmap v5749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:38 smithi059 ceph-mon[97450]: pgmap v5749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:38.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:38.258080+0000) 2022-09-08T00:37:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:38.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:38.332255+0000) 2022-09-08T00:37:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:39.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:39.258289+0000) 2022-09-08T00:37:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:39.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:39.332490+0000) 2022-09-08T00:37:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:40 smithi055 ceph-mon[97033]: pgmap v5750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:40 smithi059 ceph-mon[97450]: pgmap v5750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:40.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:40.332741+0000) 2022-09-08T00:37:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:40.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:40.258554+0000) 2022-09-08T00:37:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:41.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:41.332889+0000) 2022-09-08T00:37:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:41.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:41.258743+0000) 2022-09-08T00:37:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:41 smithi055 ceph-mon[97033]: pgmap v5751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:37:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:41 smithi059 ceph-mon[97450]: pgmap v5751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:37:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:42.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:42.333054+0000) 2022-09-08T00:37:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:42.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:42.258977+0000) 2022-09-08T00:37:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:43.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:43.259203+0000) 2022-09-08T00:37:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:43.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:43.333279+0000) 2022-09-08T00:37:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:44 smithi055 ceph-mon[97033]: pgmap v5752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:44 smithi059 ceph-mon[97450]: pgmap v5752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:44.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:44.333411+0000) 2022-09-08T00:37:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:44.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:44.259429+0000) 2022-09-08T00:37:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:45.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:45.333674+0000) 2022-09-08T00:37:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:45.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:45.259678+0000) 2022-09-08T00:37:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:46 smithi055 ceph-mon[97033]: pgmap v5753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:37:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:46 smithi059 ceph-mon[97450]: pgmap v5753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:37:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:46.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:46.333930+0000) 2022-09-08T00:37:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:46.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:46.259947+0000) 2022-09-08T00:37:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:47.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:47.334182+0000) 2022-09-08T00:37:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:47.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:47.260132+0000) 2022-09-08T00:37:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:48 smithi055 ceph-mon[97033]: pgmap v5754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:48 smithi059 ceph-mon[97450]: pgmap v5754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:48.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:48.334446+0000) 2022-09-08T00:37:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:48.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:48.260377+0000) 2022-09-08T00:37:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:49.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:49.334739+0000) 2022-09-08T00:37:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:49.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:49.260603+0000) 2022-09-08T00:37:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:50 smithi055 ceph-mon[97033]: pgmap v5755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:50 smithi059 ceph-mon[97450]: pgmap v5755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:50.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:50.334983+0000) 2022-09-08T00:37:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:50.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:50.260853+0000) 2022-09-08T00:37:51.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:37:51.181+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:37:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:51.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:51.335175+0000) 2022-09-08T00:37:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:51.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:51.261109+0000) 2022-09-08T00:37:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:51 smithi055 ceph-mon[97033]: pgmap v5756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:37:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:51 smithi059 ceph-mon[97450]: pgmap v5756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:37:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:52.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:52.335388+0000) 2022-09-08T00:37:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:52.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:52.261335+0000) 2022-09-08T00:37:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:53.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:53.335599+0000) 2022-09-08T00:37:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:53.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:53.261565+0000) 2022-09-08T00:37:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:54 smithi055 ceph-mon[97033]: pgmap v5757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:54 smithi059 ceph-mon[97450]: pgmap v5757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:54.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:54.261790+0000) 2022-09-08T00:37:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:54.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:54.335886+0000) 2022-09-08T00:37:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:55.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:55.262056+0000) 2022-09-08T00:37:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:55.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:55.336021+0000) 2022-09-08T00:37:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:56 smithi055 ceph-mon[97033]: pgmap v5758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:37:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:56 smithi059 ceph-mon[97450]: pgmap v5758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:37:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:56.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:56.336261+0000) 2022-09-08T00:37:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:56.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:56.262313+0000) 2022-09-08T00:37:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:57.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:57.336478+0000) 2022-09-08T00:37:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:57.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:57.262485+0000) 2022-09-08T00:37:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:37:58 smithi055 ceph-mon[97033]: pgmap v5759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:37:58 smithi059 ceph-mon[97450]: pgmap v5759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:37:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:58.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:58.336720+0000) 2022-09-08T00:37:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:58.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:58.262740+0000) 2022-09-08T00:37:59.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:37:59.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:59.336947+0000) 2022-09-08T00:37:59.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:37:59.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:37:59.262913+0000) 2022-09-08T00:38:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:00 smithi055 ceph-mon[97033]: pgmap v5760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:00 smithi059 ceph-mon[97450]: pgmap v5760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:00.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:00.337113+0000) 2022-09-08T00:38:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:00.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:00.263045+0000) 2022-09-08T00:38:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:01.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:01.337321+0000) 2022-09-08T00:38:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:01.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:01.263254+0000) 2022-09-08T00:38:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:01 smithi055 ceph-mon[97033]: pgmap v5761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:38:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:01 smithi059 ceph-mon[97450]: pgmap v5761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:38:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:02.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:02.337568+0000) 2022-09-08T00:38:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:02.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:02.263475+0000) 2022-09-08T00:38:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:03.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:03.337812+0000) 2022-09-08T00:38:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:03.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:03.263746+0000) 2022-09-08T00:38:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:04.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:04.337961+0000) 2022-09-08T00:38:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:04.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:04.263920+0000) 2022-09-08T00:38:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:04 smithi055 ceph-mon[97033]: pgmap v5762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:04 smithi055 ceph-mon[97033]: pgmap v5763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:38:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:04 smithi055 ceph-mon[97033]: pgmap v5764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:38:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:38:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:04 smithi059 ceph-mon[97450]: pgmap v5762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:04 smithi059 ceph-mon[97450]: pgmap v5763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:38:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:04 smithi059 ceph-mon[97450]: pgmap v5764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:38:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:38:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:05.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:05.338200+0000) 2022-09-08T00:38:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:05.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:05.264110+0000) 2022-09-08T00:38:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:38:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:38:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:38:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:38:06.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:38:06.197+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:38:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:06.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:06.338406+0000) 2022-09-08T00:38:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:06.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:06.264382+0000) 2022-09-08T00:38:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:06 smithi055 ceph-mon[97033]: pgmap v5765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T00:38:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:06 smithi059 ceph-mon[97450]: pgmap v5765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T00:38:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:07.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:07.264608+0000) 2022-09-08T00:38:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:07.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:07.338620+0000) 2022-09-08T00:38:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:08.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:08.264909+0000) 2022-09-08T00:38:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:08.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:08.338850+0000) 2022-09-08T00:38:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:08 smithi055 ceph-mon[97033]: pgmap v5766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T00:38:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:08 smithi059 ceph-mon[97450]: pgmap v5766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T00:38:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:09.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:09.339013+0000) 2022-09-08T00:38:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:09.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:09.265104+0000) 2022-09-08T00:38:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:09 smithi055 ceph-mon[97033]: osdmap e445: 12 total, 11 up, 11 in 2022-09-08T00:38:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:09 smithi059 ceph-mon[97450]: osdmap e445: 12 total, 11 up, 11 in 2022-09-08T00:38:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:10.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:10.339224+0000) 2022-09-08T00:38:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:10.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:10.265342+0000) 2022-09-08T00:38:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:10 smithi055 ceph-mon[97033]: pgmap v5768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 19 KiB/s rd, 85 KiB/s wr, 33 op/s 2022-09-08T00:38:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:10 smithi059 ceph-mon[97450]: pgmap v5768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 19 KiB/s rd, 85 KiB/s wr, 33 op/s 2022-09-08T00:38:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:11.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:11.339387+0000) 2022-09-08T00:38:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:11.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:11.265598+0000) 2022-09-08T00:38:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:11 smithi055 ceph-mon[97033]: pgmap v5769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:38:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:11 smithi059 ceph-mon[97450]: pgmap v5769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:38:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:12.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:12.265832+0000) 2022-09-08T00:38:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:12.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:12.339615+0000) 2022-09-08T00:38:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:13.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:13.339830+0000) 2022-09-08T00:38:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:13.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:13.266035+0000) 2022-09-08T00:38:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:14.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:14.339989+0000) 2022-09-08T00:38:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:14.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:14.266261+0000) 2022-09-08T00:38:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:14 smithi055 ceph-mon[97033]: pgmap v5770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:38:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:14 smithi059 ceph-mon[97450]: pgmap v5770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:38:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:15.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:15.266494+0000) 2022-09-08T00:38:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:15.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:15.340105+0000) 2022-09-08T00:38:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:16.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:16.266760+0000) 2022-09-08T00:38:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:16.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:16.340319+0000) 2022-09-08T00:38:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:16 smithi055 ceph-mon[97033]: pgmap v5771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:38:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:16 smithi055 ceph-mon[97033]: osdmap e446: 12 total, 11 up, 11 in 2022-09-08T00:38:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:16 smithi059 ceph-mon[97450]: pgmap v5771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:38:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:16 smithi059 ceph-mon[97450]: osdmap e446: 12 total, 11 up, 11 in 2022-09-08T00:38:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:17.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:17.266907+0000) 2022-09-08T00:38:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:17.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:17.340496+0000) 2022-09-08T00:38:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:18.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:18.340736+0000) 2022-09-08T00:38:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:18.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:18.267109+0000) 2022-09-08T00:38:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:18 smithi055 ceph-mon[97033]: pgmap v5773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T00:38:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:18 smithi059 ceph-mon[97450]: pgmap v5773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T00:38:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:19.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:19.340965+0000) 2022-09-08T00:38:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:19.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:19.267351+0000) 2022-09-08T00:38:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:20.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:20.341191+0000) 2022-09-08T00:38:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:20.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:20.267554+0000) 2022-09-08T00:38:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:20 smithi055 ceph-mon[97033]: pgmap v5774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:38:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:20 smithi059 ceph-mon[97450]: pgmap v5774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:38:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:21.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:21.341408+0000) 2022-09-08T00:38:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:21.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:21.267800+0000) 2022-09-08T00:38:21.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:38:21.213+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:38:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:21 smithi055 ceph-mon[97033]: pgmap v5775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T00:38:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:21 smithi059 ceph-mon[97450]: pgmap v5775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T00:38:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:22.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:22.341673+0000) 2022-09-08T00:38:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:22.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:22.268026+0000) 2022-09-08T00:38:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:23.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:23.341845+0000) 2022-09-08T00:38:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:23.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:23.268218+0000) 2022-09-08T00:38:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:24.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:24.342034+0000) 2022-09-08T00:38:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:24.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:24.268396+0000) 2022-09-08T00:38:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:24 smithi055 ceph-mon[97033]: pgmap v5776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T00:38:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:24 smithi059 ceph-mon[97450]: pgmap v5776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T00:38:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:25.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:25.342235+0000) 2022-09-08T00:38:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:25.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:25.268655+0000) 2022-09-08T00:38:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:26.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:26.342475+0000) 2022-09-08T00:38:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:26.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:26.268873+0000) 2022-09-08T00:38:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:26 smithi055 ceph-mon[97033]: pgmap v5777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:38:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:26 smithi059 ceph-mon[97450]: pgmap v5777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:38:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:27.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:27.342726+0000) 2022-09-08T00:38:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:27.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:27.269053+0000) 2022-09-08T00:38:28.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:28.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:28.269284+0000) 2022-09-08T00:38:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:28.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:28.342924+0000) 2022-09-08T00:38:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:28 smithi055 ceph-mon[97033]: pgmap v5778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:38:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:28 smithi059 ceph-mon[97450]: pgmap v5778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:38:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:29.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:29.269524+0000) 2022-09-08T00:38:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:29.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:29.343120+0000) 2022-09-08T00:38:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:30.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:30.269754+0000) 2022-09-08T00:38:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:30.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:30.343365+0000) 2022-09-08T00:38:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:30 smithi055 ceph-mon[97033]: pgmap v5779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T00:38:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:30 smithi059 ceph-mon[97450]: pgmap v5779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T00:38:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:31.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:31.269984+0000) 2022-09-08T00:38:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:31.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:31.343600+0000) 2022-09-08T00:38:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:31 smithi055 ceph-mon[97033]: pgmap v5780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:38:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:31 smithi059 ceph-mon[97450]: pgmap v5780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:38:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:32.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:32.343808+0000) 2022-09-08T00:38:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:32.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:32.270145+0000) 2022-09-08T00:38:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:33.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:33.344032+0000) 2022-09-08T00:38:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:33.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:33.270372+0000) 2022-09-08T00:38:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:34.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:34.344277+0000) 2022-09-08T00:38:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:34.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:34.270629+0000) 2022-09-08T00:38:34.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:34 smithi055 ceph-mon[97033]: pgmap v5781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:34 smithi059 ceph-mon[97450]: pgmap v5781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:35.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:35.344487+0000) 2022-09-08T00:38:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:35.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:35.270942+0000) 2022-09-08T00:38:36.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:38:36.226+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:38:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:36.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:36.344627+0000) 2022-09-08T00:38:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:36.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:36.271201+0000) 2022-09-08T00:38:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:36 smithi055 ceph-mon[97033]: pgmap v5782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:38:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:36 smithi059 ceph-mon[97450]: pgmap v5782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:38:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:37.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:37.344810+0000) 2022-09-08T00:38:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:37.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:37.271419+0000) 2022-09-08T00:38:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:38.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:38.344972+0000) 2022-09-08T00:38:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:38.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:38.271629+0000) 2022-09-08T00:38:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:38 smithi055 ceph-mon[97033]: pgmap v5783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:38 smithi059 ceph-mon[97450]: pgmap v5783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:39.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:39.345150+0000) 2022-09-08T00:38:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:39.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:39.271926+0000) 2022-09-08T00:38:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:40.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:40.345376+0000) 2022-09-08T00:38:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:40.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:40.272144+0000) 2022-09-08T00:38:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:40 smithi055 ceph-mon[97033]: pgmap v5784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:40 smithi059 ceph-mon[97450]: pgmap v5784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:41.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:41.345611+0000) 2022-09-08T00:38:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:41.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:41.272373+0000) 2022-09-08T00:38:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:41 smithi055 ceph-mon[97033]: pgmap v5785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:38:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:41 smithi059 ceph-mon[97450]: pgmap v5785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:38:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:42.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:42.345882+0000) 2022-09-08T00:38:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:42.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:42.272622+0000) 2022-09-08T00:38:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:43.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:43.272754+0000) 2022-09-08T00:38:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:43.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:43.346091+0000) 2022-09-08T00:38:44.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:44.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:44.272990+0000) 2022-09-08T00:38:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:44.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:44.346328+0000) 2022-09-08T00:38:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:44 smithi055 ceph-mon[97033]: pgmap v5786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:44 smithi059 ceph-mon[97450]: pgmap v5786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:45.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:45.346567+0000) 2022-09-08T00:38:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:45.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:45.273226+0000) 2022-09-08T00:38:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:46.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:46.346785+0000) 2022-09-08T00:38:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:46.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:46.273445+0000) 2022-09-08T00:38:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:46 smithi055 ceph-mon[97033]: pgmap v5787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:38:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:46 smithi059 ceph-mon[97450]: pgmap v5787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:38:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:47.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:47.347032+0000) 2022-09-08T00:38:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:47.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:47.273711+0000) 2022-09-08T00:38:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:48.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:48.347257+0000) 2022-09-08T00:38:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:48.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:48.273848+0000) 2022-09-08T00:38:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:48 smithi055 ceph-mon[97033]: pgmap v5788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:48 smithi059 ceph-mon[97450]: pgmap v5788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:49.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:49.347494+0000) 2022-09-08T00:38:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:49.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:49.273989+0000) 2022-09-08T00:38:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:50.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:50.347745+0000) 2022-09-08T00:38:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:50.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:50.274243+0000) 2022-09-08T00:38:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:50 smithi055 ceph-mon[97033]: pgmap v5789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:50 smithi059 ceph-mon[97450]: pgmap v5789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:51.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:38:51.241+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:38:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:51.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:51.347985+0000) 2022-09-08T00:38:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:51.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:51.274462+0000) 2022-09-08T00:38:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:51 smithi055 ceph-mon[97033]: pgmap v5790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:38:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:51 smithi059 ceph-mon[97450]: pgmap v5790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:38:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:52.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:52.348172+0000) 2022-09-08T00:38:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:52.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:52.274720+0000) 2022-09-08T00:38:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:53.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:53.348356+0000) 2022-09-08T00:38:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:53.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:53.274953+0000) 2022-09-08T00:38:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:54.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:54.348555+0000) 2022-09-08T00:38:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:54.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:54.275153+0000) 2022-09-08T00:38:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:54 smithi055 ceph-mon[97033]: pgmap v5791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:54 smithi059 ceph-mon[97450]: pgmap v5791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:55.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:55.348776+0000) 2022-09-08T00:38:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:55.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:55.275410+0000) 2022-09-08T00:38:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:56.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:56.348951+0000) 2022-09-08T00:38:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:56.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:56.275692+0000) 2022-09-08T00:38:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:56 smithi055 ceph-mon[97033]: pgmap v5792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:38:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:56 smithi059 ceph-mon[97450]: pgmap v5792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:38:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:57.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:57.349182+0000) 2022-09-08T00:38:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:57.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:57.275891+0000) 2022-09-08T00:38:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:58.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:58.349417+0000) 2022-09-08T00:38:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:58.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:58.276092+0000) 2022-09-08T00:38:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:38:58 smithi055 ceph-mon[97033]: pgmap v5793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:38:58 smithi059 ceph-mon[97450]: pgmap v5793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:38:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:38:59.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:59.349669+0000) 2022-09-08T00:38:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:38:59.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:38:59.276374+0000) 2022-09-08T00:39:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:00.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:00.349897+0000) 2022-09-08T00:39:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:00.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:00.276587+0000) 2022-09-08T00:39:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:00 smithi059 ceph-mon[97450]: pgmap v5794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:00 smithi055 ceph-mon[97033]: pgmap v5794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:01.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:01.350087+0000) 2022-09-08T00:39:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:01.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:01.276772+0000) 2022-09-08T00:39:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:01 smithi055 ceph-mon[97033]: pgmap v5795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:39:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:01 smithi059 ceph-mon[97450]: pgmap v5795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:39:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:02.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:02.350314+0000) 2022-09-08T00:39:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:02.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:02.276941+0000) 2022-09-08T00:39:03.549 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:03.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:03.350501+0000) 2022-09-08T00:39:03.550 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:03.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:03.277204+0000) 2022-09-08T00:39:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:04.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:04.350629+0000) 2022-09-08T00:39:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:04.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:04.277436+0000) 2022-09-08T00:39:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:04 smithi055 ceph-mon[97033]: pgmap v5796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:39:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:39:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:39:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:04 smithi059 ceph-mon[97450]: pgmap v5796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:39:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:39:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:39:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:05.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:05.350877+0000) 2022-09-08T00:39:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:05.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:05.277658+0000) 2022-09-08T00:39:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:39:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:39:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:39:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:39:06.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:39:06.246+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:39:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:06.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:06.277901+0000) 2022-09-08T00:39:06.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:06.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:06.351098+0000) 2022-09-08T00:39:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:06 smithi059 ceph-mon[97450]: pgmap v5797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:39:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:06 smithi055 ceph-mon[97033]: pgmap v5797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:39:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:07.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:07.351313+0000) 2022-09-08T00:39:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:07.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:07.278192+0000) 2022-09-08T00:39:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:08.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:08.351515+0000) 2022-09-08T00:39:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:08.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:08.278367+0000) 2022-09-08T00:39:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:08 smithi059 ceph-mon[97450]: pgmap v5798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:39:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:08 smithi055 ceph-mon[97033]: pgmap v5798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:39:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:09.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:09.351746+0000) 2022-09-08T00:39:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:09.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:09.278577+0000) 2022-09-08T00:39:09.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:09 smithi059 ceph-mon[97450]: osdmap e447: 12 total, 11 up, 11 in 2022-09-08T00:39:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:09 smithi055 ceph-mon[97033]: osdmap e447: 12 total, 11 up, 11 in 2022-09-08T00:39:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:10.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:10.351893+0000) 2022-09-08T00:39:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:10.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:10.278830+0000) 2022-09-08T00:39:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:10 smithi059 ceph-mon[97450]: pgmap v5800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:39:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:10 smithi055 ceph-mon[97033]: pgmap v5800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:39:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:11.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:11.352100+0000) 2022-09-08T00:39:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:11.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:11.279071+0000) 2022-09-08T00:39:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:11 smithi055 ceph-mon[97033]: pgmap v5801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:39:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:11 smithi059 ceph-mon[97450]: pgmap v5801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:39:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:12.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:12.279244+0000) 2022-09-08T00:39:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:12.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:12.352346+0000) 2022-09-08T00:39:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:13.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:13.279468+0000) 2022-09-08T00:39:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:13.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:13.352523+0000) 2022-09-08T00:39:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:14.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:14.352729+0000) 2022-09-08T00:39:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:14.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:14.279721+0000) 2022-09-08T00:39:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:14 smithi059 ceph-mon[97450]: pgmap v5802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:39:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:14 smithi055 ceph-mon[97033]: pgmap v5802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:39:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:15.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:15.352852+0000) 2022-09-08T00:39:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:15.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:15.279976+0000) 2022-09-08T00:39:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:16.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:16.280232+0000) 2022-09-08T00:39:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:16.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:16.353093+0000) 2022-09-08T00:39:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:16 smithi055 ceph-mon[97033]: pgmap v5803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:39:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:16 smithi055 ceph-mon[97033]: osdmap e448: 12 total, 11 up, 11 in 2022-09-08T00:39:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:16 smithi059 ceph-mon[97450]: pgmap v5803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:39:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:16 smithi059 ceph-mon[97450]: osdmap e448: 12 total, 11 up, 11 in 2022-09-08T00:39:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:17.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:17.353289+0000) 2022-09-08T00:39:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:17.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:17.280364+0000) 2022-09-08T00:39:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:18.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:18.353512+0000) 2022-09-08T00:39:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:18.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:18.280595+0000) 2022-09-08T00:39:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:18 smithi059 ceph-mon[97450]: pgmap v5805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T00:39:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:18 smithi055 ceph-mon[97033]: pgmap v5805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T00:39:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:19.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:19.280845+0000) 2022-09-08T00:39:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:19.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:19.353731+0000) 2022-09-08T00:39:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:20.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:20.354005+0000) 2022-09-08T00:39:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:20.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:20.281016+0000) 2022-09-08T00:39:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:20 smithi059 ceph-mon[97450]: pgmap v5806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:39:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:20 smithi055 ceph-mon[97033]: pgmap v5806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:39:21.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:39:21.258+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:39:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:21.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:21.354172+0000) 2022-09-08T00:39:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:21.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:21.281237+0000) 2022-09-08T00:39:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:21 smithi055 ceph-mon[97033]: pgmap v5807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:39:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:21 smithi059 ceph-mon[97450]: pgmap v5807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:39:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:22.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:22.354383+0000) 2022-09-08T00:39:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:22.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:22.281431+0000) 2022-09-08T00:39:23.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:23.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:23.281700+0000) 2022-09-08T00:39:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:23.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:23.354613+0000) 2022-09-08T00:39:24.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:24.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:24.281928+0000) 2022-09-08T00:39:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:24.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:24.354909+0000) 2022-09-08T00:39:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:24 smithi059 ceph-mon[97450]: pgmap v5808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:39:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:24 smithi055 ceph-mon[97033]: pgmap v5808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:39:25.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:25.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:25.282116+0000) 2022-09-08T00:39:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:25.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:25.355141+0000) 2022-09-08T00:39:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:26.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:26.355363+0000) 2022-09-08T00:39:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:26.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:26.282347+0000) 2022-09-08T00:39:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:26 smithi059 ceph-mon[97450]: pgmap v5809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:39:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:26 smithi055 ceph-mon[97033]: pgmap v5809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:39:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:27.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:27.282614+0000) 2022-09-08T00:39:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:27.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:27.355611+0000) 2022-09-08T00:39:28.621 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:28.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:28.355756+0000) 2022-09-08T00:39:28.623 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:28.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:28.282867+0000) 2022-09-08T00:39:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:28 smithi059 ceph-mon[97450]: pgmap v5810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:39:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:28 smithi055 ceph-mon[97033]: pgmap v5810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:39:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:29.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:29.356003+0000) 2022-09-08T00:39:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:29.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:29.283050+0000) 2022-09-08T00:39:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:30.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:30.356292+0000) 2022-09-08T00:39:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:30.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:30.283254+0000) 2022-09-08T00:39:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:30 smithi059 ceph-mon[97450]: pgmap v5811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:30 smithi055 ceph-mon[97033]: pgmap v5811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:31.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:31.356438+0000) 2022-09-08T00:39:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:31.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:31.283482+0000) 2022-09-08T00:39:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:31 smithi055 ceph-mon[97033]: pgmap v5812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:39:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:31 smithi059 ceph-mon[97450]: pgmap v5812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:39:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:32.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:32.356678+0000) 2022-09-08T00:39:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:32.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:32.283746+0000) 2022-09-08T00:39:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:33.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:33.283886+0000) 2022-09-08T00:39:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:33.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:33.356857+0000) 2022-09-08T00:39:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:34.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:34.357089+0000) 2022-09-08T00:39:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:34.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:34.284051+0000) 2022-09-08T00:39:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:34 smithi059 ceph-mon[97450]: pgmap v5813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:34 smithi055 ceph-mon[97033]: pgmap v5813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:35.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:35.284231+0000) 2022-09-08T00:39:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:35.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:35.357297+0000) 2022-09-08T00:39:36.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:39:36.273+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:39:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:36.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:36.284434+0000) 2022-09-08T00:39:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:36.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:36.357535+0000) 2022-09-08T00:39:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:36 smithi059 ceph-mon[97450]: pgmap v5814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:39:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:36 smithi055 ceph-mon[97033]: pgmap v5814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:39:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:37.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:37.284723+0000) 2022-09-08T00:39:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:37.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:37.357788+0000) 2022-09-08T00:39:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:38.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:38.357959+0000) 2022-09-08T00:39:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:38.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:38.284874+0000) 2022-09-08T00:39:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:38 smithi055 ceph-mon[97033]: pgmap v5815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:39:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:39:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:39:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:38 smithi059 ceph-mon[97450]: pgmap v5815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:39:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:39:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:39:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:39.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:39.358102+0000) 2022-09-08T00:39:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:39.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:39.285121+0000) 2022-09-08T00:39:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:39 smithi059 ceph-mon[97450]: pgmap v5816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:39:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:39 smithi059 ceph-mon[97450]: pgmap v5817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:39:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:39 smithi055 ceph-mon[97033]: pgmap v5816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:39:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:39 smithi055 ceph-mon[97033]: pgmap v5817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:39:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:40.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:40.358356+0000) 2022-09-08T00:39:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:40.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:40.285370+0000) 2022-09-08T00:39:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:41.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:41.358631+0000) 2022-09-08T00:39:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:41.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:41.285590+0000) 2022-09-08T00:39:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:41 smithi059 ceph-mon[97450]: pgmap v5818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:39:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:41 smithi055 ceph-mon[97033]: pgmap v5818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:39:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:42.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:42.358798+0000) 2022-09-08T00:39:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:42.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:42.285739+0000) 2022-09-08T00:39:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:43.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:43.359022+0000) 2022-09-08T00:39:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:43.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:43.285958+0000) 2022-09-08T00:39:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:43 smithi059 ceph-mon[97450]: pgmap v5819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:39:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:43 smithi055 ceph-mon[97033]: pgmap v5819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:39:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:44.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:44.359234+0000) 2022-09-08T00:39:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:44.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:44.286099+0000) 2022-09-08T00:39:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:45.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:45.359487+0000) 2022-09-08T00:39:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:45.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:45.286331+0000) 2022-09-08T00:39:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:45 smithi059 ceph-mon[97450]: pgmap v5820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T00:39:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:45 smithi055 ceph-mon[97033]: pgmap v5820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T00:39:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:46.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:46.359731+0000) 2022-09-08T00:39:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:46.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:46.286551+0000) 2022-09-08T00:39:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:46 smithi055 ceph-mon[97033]: pgmap v5821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:39:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:46 smithi059 ceph-mon[97450]: pgmap v5821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T00:39:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:47.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:47.359938+0000) 2022-09-08T00:39:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:47.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:47.286803+0000) 2022-09-08T00:39:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:48.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:48.287035+0000) 2022-09-08T00:39:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:48.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:48.360110+0000) 2022-09-08T00:39:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:49.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:49.360321+0000) 2022-09-08T00:39:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:49.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:49.287290+0000) 2022-09-08T00:39:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:49 smithi055 ceph-mon[97033]: pgmap v5822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:39:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:49 smithi059 ceph-mon[97450]: pgmap v5822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:39:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:50.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:50.360583+0000) 2022-09-08T00:39:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:50.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:50.287483+0000) 2022-09-08T00:39:51.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:39:51.288+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:39:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:51.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:51.360799+0000) 2022-09-08T00:39:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:51.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:51.287765+0000) 2022-09-08T00:39:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:51 smithi055 ceph-mon[97033]: pgmap v5823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:39:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:51 smithi059 ceph-mon[97450]: pgmap v5823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:39:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:52.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:52.287973+0000) 2022-09-08T00:39:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:52.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:52.361037+0000) 2022-09-08T00:39:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:53.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:53.361270+0000) 2022-09-08T00:39:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:53.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:53.288210+0000) 2022-09-08T00:39:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:53 smithi055 ceph-mon[97033]: pgmap v5824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:53 smithi059 ceph-mon[97450]: pgmap v5824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:54.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:54.288425+0000) 2022-09-08T00:39:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:54.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:54.361467+0000) 2022-09-08T00:39:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:55.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:55.361720+0000) 2022-09-08T00:39:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:55.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:55.288676+0000) 2022-09-08T00:39:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:55 smithi055 ceph-mon[97033]: pgmap v5825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:55 smithi059 ceph-mon[97450]: pgmap v5825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:56.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:56.361953+0000) 2022-09-08T00:39:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:56.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:56.288890+0000) 2022-09-08T00:39:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:56 smithi055 ceph-mon[97033]: pgmap v5826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:39:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:56 smithi059 ceph-mon[97450]: pgmap v5826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:39:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:57.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:57.362138+0000) 2022-09-08T00:39:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:57.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:57.289131+0000) 2022-09-08T00:39:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:58.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:58.362358+0000) 2022-09-08T00:39:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:58.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:58.289367+0000) 2022-09-08T00:39:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:39:59.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:59.362501+0000) 2022-09-08T00:39:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:39:59.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:39:59.289595+0000) 2022-09-08T00:39:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:39:59 smithi055 ceph-mon[97033]: pgmap v5827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:39:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:39:59 smithi059 ceph-mon[97450]: pgmap v5827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:40:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:00.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:00.362733+0000) 2022-09-08T00:40:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:00.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:00.289873+0000) 2022-09-08T00:40:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T00:40:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T00:40:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: ['Unable to reach remote host smithi047. '] 2022-09-08T00:40:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T00:40:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T00:40:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T00:40:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T00:40:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: ['Unable to reach remote host smithi047. '] 2022-09-08T00:40:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T00:40:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T00:40:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:01.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:01.362971+0000) 2022-09-08T00:40:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:01.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:01.290122+0000) 2022-09-08T00:40:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:01 smithi055 ceph-mon[97033]: pgmap v5828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:40:01.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:01 smithi059 ceph-mon[97450]: pgmap v5828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:40:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:02.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:02.290321+0000) 2022-09-08T00:40:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:02.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:02.363213+0000) 2022-09-08T00:40:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:03.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:03.363384+0000) 2022-09-08T00:40:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:03.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:03.290524+0000) 2022-09-08T00:40:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:03 smithi055 ceph-mon[97033]: pgmap v5829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:40:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:03 smithi059 ceph-mon[97450]: pgmap v5829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:40:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:04.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:04.363611+0000) 2022-09-08T00:40:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:04.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:04.290770+0000) 2022-09-08T00:40:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:05.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:05.363801+0000) 2022-09-08T00:40:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:05.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:05.291013+0000) 2022-09-08T00:40:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:05 smithi055 ceph-mon[97033]: pgmap v5830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:40:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:40:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:40:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:05 smithi059 ceph-mon[97450]: pgmap v5830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:40:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:40:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:40:06.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:40:06.290+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:40:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:06.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:06.364043+0000) 2022-09-08T00:40:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:06.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:06.291210+0000) 2022-09-08T00:40:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:06 smithi055 ceph-mon[97033]: pgmap v5831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:40:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:06 smithi059 ceph-mon[97450]: pgmap v5831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:40:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:07.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:07.364342+0000) 2022-09-08T00:40:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:07.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:07.291461+0000) 2022-09-08T00:40:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:08.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:08.291678+0000) 2022-09-08T00:40:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:08.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:08.364619+0000) 2022-09-08T00:40:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:09.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:09.291874+0000) 2022-09-08T00:40:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:09.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:09.364838+0000) 2022-09-08T00:40:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:09 smithi055 ceph-mon[97033]: pgmap v5832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:40:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:09 smithi059 ceph-mon[97450]: pgmap v5832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:40:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:10.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:10.365009+0000) 2022-09-08T00:40:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:10.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:10.292070+0000) 2022-09-08T00:40:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:10 smithi055 ceph-mon[97033]: osdmap e449: 12 total, 11 up, 11 in 2022-09-08T00:40:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:10 smithi059 ceph-mon[97450]: osdmap e449: 12 total, 11 up, 11 in 2022-09-08T00:40:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:11.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:11.365221+0000) 2022-09-08T00:40:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:11.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:11.292318+0000) 2022-09-08T00:40:11.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:11 smithi055 ceph-mon[97033]: pgmap v5834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:40:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:11 smithi059 ceph-mon[97450]: pgmap v5834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:40:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:12.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:12.292543+0000) 2022-09-08T00:40:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:12.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:12.365451+0000) 2022-09-08T00:40:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:13.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:13.365604+0000) 2022-09-08T00:40:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:13.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:13.292766+0000) 2022-09-08T00:40:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:13 smithi055 ceph-mon[97033]: pgmap v5835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:40:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:13 smithi059 ceph-mon[97450]: pgmap v5835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:40:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:14.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:14.365857+0000) 2022-09-08T00:40:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:14.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:14.292964+0000) 2022-09-08T00:40:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:15.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:15.366057+0000) 2022-09-08T00:40:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:15.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:15.293236+0000) 2022-09-08T00:40:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:15 smithi055 ceph-mon[97033]: pgmap v5836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:40:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:15 smithi059 ceph-mon[97450]: pgmap v5836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:40:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:16.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:16.366287+0000) 2022-09-08T00:40:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:16.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:16.293464+0000) 2022-09-08T00:40:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:16 smithi055 ceph-mon[97033]: pgmap v5837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:40:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:16 smithi059 ceph-mon[97450]: pgmap v5837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:40:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:17.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:17.366501+0000) 2022-09-08T00:40:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:17.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:17.293723+0000) 2022-09-08T00:40:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:18.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:18.293862+0000) 2022-09-08T00:40:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:18.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:18.366754+0000) 2022-09-08T00:40:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:19.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:19.366919+0000) 2022-09-08T00:40:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:19.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:19.293984+0000) 2022-09-08T00:40:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:19 smithi055 ceph-mon[97033]: pgmap v5838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:40:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:19 smithi059 ceph-mon[97450]: pgmap v5838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:40:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:20.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:20.367089+0000) 2022-09-08T00:40:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:20.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:20.294136+0000) 2022-09-08T00:40:21.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:40:21.293+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:40:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:21.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:21.294366+0000) 2022-09-08T00:40:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:21.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:21.367305+0000) 2022-09-08T00:40:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:21 smithi055 ceph-mon[97033]: pgmap v5839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T00:40:21.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:21 smithi055 ceph-mon[97033]: osdmap e450: 12 total, 11 up, 11 in 2022-09-08T00:40:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:21 smithi059 ceph-mon[97450]: pgmap v5839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T00:40:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:21 smithi059 ceph-mon[97450]: osdmap e450: 12 total, 11 up, 11 in 2022-09-08T00:40:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:22.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:22.367484+0000) 2022-09-08T00:40:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:22.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:22.294606+0000) 2022-09-08T00:40:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:23.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:23.367738+0000) 2022-09-08T00:40:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:23.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:23.294866+0000) 2022-09-08T00:40:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:23 smithi055 ceph-mon[97033]: pgmap v5841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:40:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:23 smithi059 ceph-mon[97450]: pgmap v5841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:40:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:24.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:24.367908+0000) 2022-09-08T00:40:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:24.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:24.295101+0000) 2022-09-08T00:40:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:25.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:25.368132+0000) 2022-09-08T00:40:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:25.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:25.295296+0000) 2022-09-08T00:40:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:25 smithi055 ceph-mon[97033]: pgmap v5842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:40:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:25 smithi059 ceph-mon[97450]: pgmap v5842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:40:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:26.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:26.368268+0000) 2022-09-08T00:40:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:26.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:26.295530+0000) 2022-09-08T00:40:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:26 smithi055 ceph-mon[97033]: pgmap v5843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:40:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:26 smithi059 ceph-mon[97450]: pgmap v5843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:40:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:27.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:27.368443+0000) 2022-09-08T00:40:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:27.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:27.295800+0000) 2022-09-08T00:40:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:28.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:28.368688+0000) 2022-09-08T00:40:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:28.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:28.296051+0000) 2022-09-08T00:40:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:29.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:29.368833+0000) 2022-09-08T00:40:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:29.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:29.296217+0000) 2022-09-08T00:40:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:29 smithi055 ceph-mon[97033]: pgmap v5844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:40:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:29 smithi059 ceph-mon[97450]: pgmap v5844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:40:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:30.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:30.369024+0000) 2022-09-08T00:40:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:30.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:30.296429+0000) 2022-09-08T00:40:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:31.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:31.369229+0000) 2022-09-08T00:40:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:31.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:31.296748+0000) 2022-09-08T00:40:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:31 smithi055 ceph-mon[97033]: pgmap v5845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:40:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:31 smithi059 ceph-mon[97450]: pgmap v5845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:40:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:32.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:32.369477+0000) 2022-09-08T00:40:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:32.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:32.296972+0000) 2022-09-08T00:40:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:33.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:33.369717+0000) 2022-09-08T00:40:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:33.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:33.297209+0000) 2022-09-08T00:40:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:33 smithi055 ceph-mon[97033]: pgmap v5846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:40:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:33 smithi059 ceph-mon[97450]: pgmap v5846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:40:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:34.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:34.369927+0000) 2022-09-08T00:40:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:34.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:34.297370+0000) 2022-09-08T00:40:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:35.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:35.370162+0000) 2022-09-08T00:40:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:35.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:35.297500+0000) 2022-09-08T00:40:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:35 smithi055 ceph-mon[97033]: pgmap v5847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:40:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:35 smithi059 ceph-mon[97450]: pgmap v5847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:40:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:36.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:36.370392+0000) 2022-09-08T00:40:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:36.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:36.297735+0000) 2022-09-08T00:40:36.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:40:36.295+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:40:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:36 smithi055 ceph-mon[97033]: pgmap v5848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:40:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:36 smithi059 ceph-mon[97450]: pgmap v5848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:40:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:37.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:37.370677+0000) 2022-09-08T00:40:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:37.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:37.297982+0000) 2022-09-08T00:40:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:38.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:38.370867+0000) 2022-09-08T00:40:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:38.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:38.298174+0000) 2022-09-08T00:40:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:39.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:39.371077+0000) 2022-09-08T00:40:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:39.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:39.298415+0000) 2022-09-08T00:40:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:39 smithi055 ceph-mon[97033]: pgmap v5849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:40:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:40:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:40:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:40:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:40:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:39 smithi059 ceph-mon[97450]: pgmap v5849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:40:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:40:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:40:39.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:40:39.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:40:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:40.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:40.371325+0000) 2022-09-08T00:40:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:40.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:40.298627+0000) 2022-09-08T00:40:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:40 smithi055 ceph-mon[97033]: pgmap v5850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:40:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:40 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T00:40:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:40 smithi059 ceph-mon[97450]: pgmap v5850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:40:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:40 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T00:40:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:41.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:41.371564+0000) 2022-09-08T00:40:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:41.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:41.298887+0000) 2022-09-08T00:40:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:42.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:42.371755+0000) 2022-09-08T00:40:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:42.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:42.299135+0000) 2022-09-08T00:40:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:42 smithi055 ceph-mon[97033]: pgmap v5851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:40:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:42 smithi059 ceph-mon[97450]: pgmap v5851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:40:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:43.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:43.372014+0000) 2022-09-08T00:40:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:43.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:43.299354+0000) 2022-09-08T00:40:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:44.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:44.299620+0000) 2022-09-08T00:40:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:44.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:44.372294+0000) 2022-09-08T00:40:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:44 smithi055 ceph-mon[97033]: pgmap v5852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:40:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:44 smithi059 ceph-mon[97450]: pgmap v5852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:40:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:45.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:45.372548+0000) 2022-09-08T00:40:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:45.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:45.299882+0000) 2022-09-08T00:40:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:46.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:46.372839+0000) 2022-09-08T00:40:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:46.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:46.300219+0000) 2022-09-08T00:40:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:46 smithi055 ceph-mon[97033]: pgmap v5853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 7 op/s 2022-09-08T00:40:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:46 smithi059 ceph-mon[97450]: pgmap v5853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 7 op/s 2022-09-08T00:40:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:47.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:47.373126+0000) 2022-09-08T00:40:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:47.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:47.300472+0000) 2022-09-08T00:40:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:48.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:48.373385+0000) 2022-09-08T00:40:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:48.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:48.300701+0000) 2022-09-08T00:40:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:48 smithi055 ceph-mon[97033]: pgmap v5854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:40:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:48 smithi059 ceph-mon[97450]: pgmap v5854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:40:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:49.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:49.373627+0000) 2022-09-08T00:40:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:49.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:49.300841+0000) 2022-09-08T00:40:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:50.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:50.373886+0000) 2022-09-08T00:40:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:50.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:50.301082+0000) 2022-09-08T00:40:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:50 smithi055 ceph-mon[97033]: pgmap v5855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:40:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:50 smithi059 ceph-mon[97450]: pgmap v5855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:40:51.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:40:51.311+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:40:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:51.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:51.374109+0000) 2022-09-08T00:40:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:51.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:51.301252+0000) 2022-09-08T00:40:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:52.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:52.374355+0000) 2022-09-08T00:40:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:52.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:52.301491+0000) 2022-09-08T00:40:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:52 smithi055 ceph-mon[97033]: pgmap v5856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:40:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:52 smithi059 ceph-mon[97450]: pgmap v5856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:40:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:53.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:53.374590+0000) 2022-09-08T00:40:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:53.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:53.301772+0000) 2022-09-08T00:40:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:54.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:54.374800+0000) 2022-09-08T00:40:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:54.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:54.301988+0000) 2022-09-08T00:40:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:54 smithi055 ceph-mon[97033]: pgmap v5857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:40:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:54 smithi059 ceph-mon[97450]: pgmap v5857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:40:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:55.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:55.375008+0000) 2022-09-08T00:40:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:55.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:55.302201+0000) 2022-09-08T00:40:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:56.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:56.375275+0000) 2022-09-08T00:40:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:56.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:56.302436+0000) 2022-09-08T00:40:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:56 smithi055 ceph-mon[97033]: pgmap v5858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T00:40:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:56 smithi059 ceph-mon[97450]: pgmap v5858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T00:40:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:57.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:57.375519+0000) 2022-09-08T00:40:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:57.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:57.302731+0000) 2022-09-08T00:40:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:58.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:58.302936+0000) 2022-09-08T00:40:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:58.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:58.375734+0000) 2022-09-08T00:40:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:40:58 smithi055 ceph-mon[97033]: pgmap v5859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T00:40:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:40:58 smithi059 ceph-mon[97450]: pgmap v5859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T00:40:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:40:59.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:59.375861+0000) 2022-09-08T00:40:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:40:59.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:40:59.303209+0000) 2022-09-08T00:41:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:00.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:00.303347+0000) 2022-09-08T00:41:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:00.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:00.375985+0000) 2022-09-08T00:41:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:00 smithi055 ceph-mon[97033]: pgmap v5860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:00 smithi059 ceph-mon[97450]: pgmap v5860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:01.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:01.303577+0000) 2022-09-08T00:41:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:01.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:01.376177+0000) 2022-09-08T00:41:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:02.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:02.303796+0000) 2022-09-08T00:41:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:02.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:02.376389+0000) 2022-09-08T00:41:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:02 smithi055 ceph-mon[97033]: pgmap v5861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:41:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:02 smithi059 ceph-mon[97450]: pgmap v5861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:41:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:03.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:03.376538+0000) 2022-09-08T00:41:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:03.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:03.303934+0000) 2022-09-08T00:41:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:04.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:04.376801+0000) 2022-09-08T00:41:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:04.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:04.304133+0000) 2022-09-08T00:41:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:04 smithi055 ceph-mon[97033]: pgmap v5862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:04 smithi059 ceph-mon[97450]: pgmap v5862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:05.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:05.304307+0000) 2022-09-08T00:41:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:05.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:05.377023+0000) 2022-09-08T00:41:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:41:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:41:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:41:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:41:06.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:41:06.312+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:41:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:06.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:06.304558+0000) 2022-09-08T00:41:06.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:06.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:06.377240+0000) 2022-09-08T00:41:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:06 smithi055 ceph-mon[97033]: pgmap v5863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:41:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:06 smithi059 ceph-mon[97450]: pgmap v5863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:41:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:07.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:07.304889+0000) 2022-09-08T00:41:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:07.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:07.377491+0000) 2022-09-08T00:41:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:08.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:08.377733+0000) 2022-09-08T00:41:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:08.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:08.305066+0000) 2022-09-08T00:41:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:08 smithi055 ceph-mon[97033]: pgmap v5864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T00:41:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:08 smithi059 ceph-mon[97450]: pgmap v5864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T00:41:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:09.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:09.377965+0000) 2022-09-08T00:41:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:09.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:09.305280+0000) 2022-09-08T00:41:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:09 smithi055 ceph-mon[97033]: osdmap e451: 12 total, 11 up, 11 in 2022-09-08T00:41:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:09 smithi059 ceph-mon[97450]: osdmap e451: 12 total, 11 up, 11 in 2022-09-08T00:41:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:10.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:10.378155+0000) 2022-09-08T00:41:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:10.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:10.305463+0000) 2022-09-08T00:41:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:10 smithi055 ceph-mon[97033]: pgmap v5866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:41:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:10 smithi059 ceph-mon[97450]: pgmap v5866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:41:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:11.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:11.378374+0000) 2022-09-08T00:41:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:11.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:11.305688+0000) 2022-09-08T00:41:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:12.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:12.378619+0000) 2022-09-08T00:41:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:12.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:12.305880+0000) 2022-09-08T00:41:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:12 smithi055 ceph-mon[97033]: pgmap v5867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:41:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:12 smithi059 ceph-mon[97450]: pgmap v5867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:41:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:13.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:13.378937+0000) 2022-09-08T00:41:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:13.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:13.306103+0000) 2022-09-08T00:41:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:14.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:14.379188+0000) 2022-09-08T00:41:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:14.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:14.306363+0000) 2022-09-08T00:41:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:14 smithi055 ceph-mon[97033]: pgmap v5868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:41:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:14 smithi059 ceph-mon[97450]: pgmap v5868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:41:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:15.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:15.379446+0000) 2022-09-08T00:41:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:15.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:15.306615+0000) 2022-09-08T00:41:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:16.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:16.379711+0000) 2022-09-08T00:41:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:16.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:16.306927+0000) 2022-09-08T00:41:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:16 smithi055 ceph-mon[97033]: pgmap v5869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:41:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:16 smithi055 ceph-mon[97033]: osdmap e452: 12 total, 11 up, 11 in 2022-09-08T00:41:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:16 smithi059 ceph-mon[97450]: pgmap v5869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:41:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:16 smithi059 ceph-mon[97450]: osdmap e452: 12 total, 11 up, 11 in 2022-09-08T00:41:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:17.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:17.379874+0000) 2022-09-08T00:41:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:17.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:17.307111+0000) 2022-09-08T00:41:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:18.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:18.380034+0000) 2022-09-08T00:41:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:18.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:18.307298+0000) 2022-09-08T00:41:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:18 smithi055 ceph-mon[97033]: pgmap v5871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 240 B/s wr, 7 op/s 2022-09-08T00:41:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:18 smithi059 ceph-mon[97450]: pgmap v5871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 240 B/s wr, 7 op/s 2022-09-08T00:41:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:19.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:19.307513+0000) 2022-09-08T00:41:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:19.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:19.380279+0000) 2022-09-08T00:41:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:20.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:20.307753+0000) 2022-09-08T00:41:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:20.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:20.380469+0000) 2022-09-08T00:41:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:20 smithi055 ceph-mon[97033]: pgmap v5872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:41:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:20 smithi059 ceph-mon[97450]: pgmap v5872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:41:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:21.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:21.380659+0000) 2022-09-08T00:41:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:21.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:21.307983+0000) 2022-09-08T00:41:21.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:41:21.329+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:41:22.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:22.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:22.308212+0000) 2022-09-08T00:41:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:22.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:22.380870+0000) 2022-09-08T00:41:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:22 smithi055 ceph-mon[97033]: pgmap v5873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:41:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:22 smithi059 ceph-mon[97450]: pgmap v5873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:41:23.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:23.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:23.308436+0000) 2022-09-08T00:41:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:23.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:23.381103+0000) 2022-09-08T00:41:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:24.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:24.308695+0000) 2022-09-08T00:41:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:24.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:24.381328+0000) 2022-09-08T00:41:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:24 smithi055 ceph-mon[97033]: pgmap v5874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:41:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:24 smithi059 ceph-mon[97450]: pgmap v5874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:41:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:25.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:25.381581+0000) 2022-09-08T00:41:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:25.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:25.308848+0000) 2022-09-08T00:41:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:26.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:26.381812+0000) 2022-09-08T00:41:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:26.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:26.309124+0000) 2022-09-08T00:41:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:26 smithi055 ceph-mon[97033]: pgmap v5875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:41:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:26 smithi059 ceph-mon[97450]: pgmap v5875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:41:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:27.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:27.309349+0000) 2022-09-08T00:41:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:27.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:27.382046+0000) 2022-09-08T00:41:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:28.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:28.382237+0000) 2022-09-08T00:41:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:28.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:28.309556+0000) 2022-09-08T00:41:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:28 smithi055 ceph-mon[97033]: pgmap v5876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:41:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:28 smithi059 ceph-mon[97450]: pgmap v5876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:41:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:29.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:29.382391+0000) 2022-09-08T00:41:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:29.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:29.309702+0000) 2022-09-08T00:41:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:30.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:30.309927+0000) 2022-09-08T00:41:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:30.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:30.382576+0000) 2022-09-08T00:41:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:30 smithi055 ceph-mon[97033]: pgmap v5877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:30 smithi059 ceph-mon[97450]: pgmap v5877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:31.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:31.310223+0000) 2022-09-08T00:41:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:31.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:31.382807+0000) 2022-09-08T00:41:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:32.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:32.383019+0000) 2022-09-08T00:41:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:32.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:32.310441+0000) 2022-09-08T00:41:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:32 smithi055 ceph-mon[97033]: pgmap v5878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:41:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:32 smithi059 ceph-mon[97450]: pgmap v5878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:41:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:33.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:33.383233+0000) 2022-09-08T00:41:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:33.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:33.310698+0000) 2022-09-08T00:41:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:34.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:34.383489+0000) 2022-09-08T00:41:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:34.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:34.310911+0000) 2022-09-08T00:41:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:34 smithi055 ceph-mon[97033]: pgmap v5879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:34 smithi059 ceph-mon[97450]: pgmap v5879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:35.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:35.383723+0000) 2022-09-08T00:41:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:35.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:35.311121+0000) 2022-09-08T00:41:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:36.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:36.311301+0000) 2022-09-08T00:41:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:36.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:36.383849+0000) 2022-09-08T00:41:36.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:41:36.331+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:41:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:36 smithi059 ceph-mon[97450]: pgmap v5880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:41:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:36 smithi055 ceph-mon[97033]: pgmap v5880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:41:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:37.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:37.384052+0000) 2022-09-08T00:41:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:37.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:37.311438+0000) 2022-09-08T00:41:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:38.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:38.384282+0000) 2022-09-08T00:41:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:38.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:38.311674+0000) 2022-09-08T00:41:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:38 smithi059 ceph-mon[97450]: pgmap v5881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:38 smithi055 ceph-mon[97033]: pgmap v5881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:39.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:39.311924+0000) 2022-09-08T00:41:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:39.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:39.384507+0000) 2022-09-08T00:41:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:41:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:41:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:41:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:41:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:41:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:41:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:41:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:41:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:41:39.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:41:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:40.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:40.384761+0000) 2022-09-08T00:41:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:40.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:40.312184+0000) 2022-09-08T00:41:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:40 smithi059 ceph-mon[97450]: pgmap v5882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:40.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:40 smithi055 ceph-mon[97033]: pgmap v5882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:41.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:41.384941+0000) 2022-09-08T00:41:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:41.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:41.312447+0000) 2022-09-08T00:41:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:42.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:42.312782+0000) 2022-09-08T00:41:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:42.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:42.385179+0000) 2022-09-08T00:41:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:42 smithi055 ceph-mon[97033]: pgmap v5883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:41:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:41:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:41:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:41:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:41:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:42.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:42 smithi059 ceph-mon[97450]: pgmap v5883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:41:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:41:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:41:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:41:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:41:42.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:42.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:42.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:42.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:43.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:43.385404+0000) 2022-09-08T00:41:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:43.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:43.312955+0000) 2022-09-08T00:41:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:43 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T00:41:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:43 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T00:41:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:43.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:43 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T00:41:43.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:43 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T00:41:43.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:41:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:43 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T00:41:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:43 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T00:41:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:43.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:41:43.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:43 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T00:41:43.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:43 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T00:41:43.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:41:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:44.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:44.385577+0000) 2022-09-08T00:41:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:44.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:44.313165+0000) 2022-09-08T00:41:44.740 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:44 smithi059 ceph-mon[97450]: pgmap v5884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:44 smithi055 ceph-mon[97033]: pgmap v5884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:45.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:45.385783+0000) 2022-09-08T00:41:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:45.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:45.313372+0000) 2022-09-08T00:41:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:46.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:46.386005+0000) 2022-09-08T00:41:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:46.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:46.313622+0000) 2022-09-08T00:41:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:46 smithi059 ceph-mon[97450]: pgmap v5885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:41:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:46 smithi055 ceph-mon[97033]: pgmap v5885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:41:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:47.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:47.386233+0000) 2022-09-08T00:41:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:47.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:47.313803+0000) 2022-09-08T00:41:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:48.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:48.313931+0000) 2022-09-08T00:41:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:48.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:48.386449+0000) 2022-09-08T00:41:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:48 smithi059 ceph-mon[97450]: pgmap v5886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:48 smithi055 ceph-mon[97033]: pgmap v5886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:49.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:49.314077+0000) 2022-09-08T00:41:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:49.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:49.386629+0000) 2022-09-08T00:41:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:50.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:50.386791+0000) 2022-09-08T00:41:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:50.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:50.314294+0000) 2022-09-08T00:41:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:50 smithi059 ceph-mon[97450]: pgmap v5887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:50 smithi055 ceph-mon[97033]: pgmap v5887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:51.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:41:51.342+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:41:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:51.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:51.386981+0000) 2022-09-08T00:41:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:51.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:51.314557+0000) 2022-09-08T00:41:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:51 smithi055 ceph-mon[97033]: pgmap v5888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:41:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:51 smithi059 ceph-mon[97450]: pgmap v5888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:41:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:52.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:52.387152+0000) 2022-09-08T00:41:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:52.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:52.314797+0000) 2022-09-08T00:41:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:53.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:53.387399+0000) 2022-09-08T00:41:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:53.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:53.314984+0000) 2022-09-08T00:41:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:53 smithi055 ceph-mon[97033]: pgmap v5889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:53 smithi059 ceph-mon[97450]: pgmap v5889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:54.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:54.387711+0000) 2022-09-08T00:41:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:54.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:54.315180+0000) 2022-09-08T00:41:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:55.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:55.387939+0000) 2022-09-08T00:41:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:55.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:55.315400+0000) 2022-09-08T00:41:55.959 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:55 smithi055 ceph-mon[97033]: pgmap v5890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:41:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:55 smithi059 ceph-mon[97450]: pgmap v5890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:41:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:56.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:56.315672+0000) 2022-09-08T00:41:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:56.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:56.388187+0000) 2022-09-08T00:41:57.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:57.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:57.315934+0000) 2022-09-08T00:41:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:57.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:57.388478+0000) 2022-09-08T00:41:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:57 smithi055 ceph-mon[97033]: pgmap v5891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:57 smithi059 ceph-mon[97450]: pgmap v5891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:41:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:58.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:58.388686+0000) 2022-09-08T00:41:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:58.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:58.316161+0000) 2022-09-08T00:41:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:41:59.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:59.316430+0000) 2022-09-08T00:41:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:41:59.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:41:59.388894+0000) 2022-09-08T00:42:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:41:59 smithi055 ceph-mon[97033]: pgmap v5892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:41:59 smithi059 ceph-mon[97450]: pgmap v5892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:00.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:00.389047+0000) 2022-09-08T00:42:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:00.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:00.316574+0000) 2022-09-08T00:42:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:01.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:01.389292+0000) 2022-09-08T00:42:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:01.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:01.316797+0000) 2022-09-08T00:42:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:01 smithi055 ceph-mon[97033]: pgmap v5893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:42:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:01 smithi059 ceph-mon[97450]: pgmap v5893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:42:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:02.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:02.389532+0000) 2022-09-08T00:42:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:02.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:02.317062+0000) 2022-09-08T00:42:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:03.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:03.389773+0000) 2022-09-08T00:42:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:03.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:03.317310+0000) 2022-09-08T00:42:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:03 smithi055 ceph-mon[97033]: pgmap v5894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:03 smithi059 ceph-mon[97450]: pgmap v5894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:04.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:04.317543+0000) 2022-09-08T00:42:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:04.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:04.389939+0000) 2022-09-08T00:42:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:42:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:42:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:42:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:42:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:05.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:05.390153+0000) 2022-09-08T00:42:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:05.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:05.317760+0000) 2022-09-08T00:42:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:05 smithi059 ceph-mon[97450]: pgmap v5895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:42:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:05 smithi055 ceph-mon[97033]: pgmap v5895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:42:06.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:42:06.344+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:42:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:06.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:06.390321+0000) 2022-09-08T00:42:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:06.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:06.317971+0000) 2022-09-08T00:42:07.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:07.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:07.390564+0000) 2022-09-08T00:42:07.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:07.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:07.318262+0000) 2022-09-08T00:42:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:07 smithi059 ceph-mon[97450]: pgmap v5896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:42:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:07 smithi055 ceph-mon[97033]: pgmap v5896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:42:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:08.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:08.390722+0000) 2022-09-08T00:42:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:08.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:08.318444+0000) 2022-09-08T00:42:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:09.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:09.390960+0000) 2022-09-08T00:42:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:09.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:09.318603+0000) 2022-09-08T00:42:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:09 smithi059 ceph-mon[97450]: pgmap v5897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:42:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:09 smithi059 ceph-mon[97450]: osdmap e453: 12 total, 11 up, 11 in 2022-09-08T00:42:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:09 smithi055 ceph-mon[97033]: pgmap v5897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:42:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:09 smithi055 ceph-mon[97033]: osdmap e453: 12 total, 11 up, 11 in 2022-09-08T00:42:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:10.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:10.391194+0000) 2022-09-08T00:42:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:10.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:10.318697+0000) 2022-09-08T00:42:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:11.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:11.391414+0000) 2022-09-08T00:42:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:11.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:11.318912+0000) 2022-09-08T00:42:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:11 smithi059 ceph-mon[97450]: pgmap v5899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:42:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:11 smithi055 ceph-mon[97033]: pgmap v5899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:42:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:12.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:12.391544+0000) 2022-09-08T00:42:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:12.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:12.319147+0000) 2022-09-08T00:42:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:13.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:13.391695+0000) 2022-09-08T00:42:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:13.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:13.319356+0000) 2022-09-08T00:42:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:13 smithi059 ceph-mon[97450]: pgmap v5900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:42:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:14 smithi055 ceph-mon[97033]: pgmap v5900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:42:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:14.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:14.391923+0000) 2022-09-08T00:42:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:14.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:14.319578+0000) 2022-09-08T00:42:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:15.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:15.392128+0000) 2022-09-08T00:42:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:15.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:15.319807+0000) 2022-09-08T00:42:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:16 smithi059 ceph-mon[97450]: pgmap v5901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:42:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:16 smithi059 ceph-mon[97450]: osdmap e454: 12 total, 11 up, 11 in 2022-09-08T00:42:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:16 smithi055 ceph-mon[97033]: pgmap v5901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:42:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:16 smithi055 ceph-mon[97033]: osdmap e454: 12 total, 11 up, 11 in 2022-09-08T00:42:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:16.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:16.392348+0000) 2022-09-08T00:42:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:16.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:16.320017+0000) 2022-09-08T00:42:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:17.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:17.392610+0000) 2022-09-08T00:42:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:17.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:17.320187+0000) 2022-09-08T00:42:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:18 smithi059 ceph-mon[97450]: pgmap v5903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:42:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:18 smithi055 ceph-mon[97033]: pgmap v5903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:42:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:18.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:18.392766+0000) 2022-09-08T00:42:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:18.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:18.320409+0000) 2022-09-08T00:42:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:19.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:19.393014+0000) 2022-09-08T00:42:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:19.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:19.320680+0000) 2022-09-08T00:42:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:20 smithi059 ceph-mon[97450]: pgmap v5904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T00:42:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:20 smithi055 ceph-mon[97033]: pgmap v5904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T00:42:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:20.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:20.393245+0000) 2022-09-08T00:42:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:20.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:20.320886+0000) 2022-09-08T00:42:21.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:42:21.356+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:42:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:21.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:21.393489+0000) 2022-09-08T00:42:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:21.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:21.321146+0000) 2022-09-08T00:42:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:22 smithi059 ceph-mon[97450]: pgmap v5905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:42:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:22 smithi055 ceph-mon[97033]: pgmap v5905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:42:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:22.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:22.393699+0000) 2022-09-08T00:42:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:22.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:22.321379+0000) 2022-09-08T00:42:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:23.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:23.321632+0000) 2022-09-08T00:42:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:23.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:23.393864+0000) 2022-09-08T00:42:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:24 smithi059 ceph-mon[97450]: pgmap v5906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:42:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:24 smithi055 ceph-mon[97033]: pgmap v5906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:42:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:24.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:24.394010+0000) 2022-09-08T00:42:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:24.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:24.321728+0000) 2022-09-08T00:42:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:25.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:25.394133+0000) 2022-09-08T00:42:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:25.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:25.321934+0000) 2022-09-08T00:42:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:26 smithi059 ceph-mon[97450]: pgmap v5907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:42:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:26 smithi055 ceph-mon[97033]: pgmap v5907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:42:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:26.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:26.394364+0000) 2022-09-08T00:42:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:26.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:26.322158+0000) 2022-09-08T00:42:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:27.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:27.394605+0000) 2022-09-08T00:42:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:27.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:27.322446+0000) 2022-09-08T00:42:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:28 smithi059 ceph-mon[97450]: pgmap v5908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:42:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:28 smithi055 ceph-mon[97033]: pgmap v5908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:42:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:28.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:28.394780+0000) 2022-09-08T00:42:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:28.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:28.322664+0000) 2022-09-08T00:42:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:29.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:29.322902+0000) 2022-09-08T00:42:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:29.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:29.394916+0000) 2022-09-08T00:42:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:30 smithi059 ceph-mon[97450]: pgmap v5909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:30 smithi055 ceph-mon[97033]: pgmap v5909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:30.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:30.395126+0000) 2022-09-08T00:42:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:30.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:30.323114+0000) 2022-09-08T00:42:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:31.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:31.395364+0000) 2022-09-08T00:42:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:31.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:31.323334+0000) 2022-09-08T00:42:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:32 smithi055 ceph-mon[97033]: pgmap v5910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:42:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:32 smithi059 ceph-mon[97450]: pgmap v5910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:42:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:32.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:32.395613+0000) 2022-09-08T00:42:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:32.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:32.323550+0000) 2022-09-08T00:42:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:33.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:33.395815+0000) 2022-09-08T00:42:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:33.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:33.323747+0000) 2022-09-08T00:42:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:34 smithi055 ceph-mon[97033]: pgmap v5911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:34 smithi059 ceph-mon[97450]: pgmap v5911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:34.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:34.396057+0000) 2022-09-08T00:42:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:34.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:34.323909+0000) 2022-09-08T00:42:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:35.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:35.396249+0000) 2022-09-08T00:42:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:35.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:35.324129+0000) 2022-09-08T00:42:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:36 smithi055 ceph-mon[97033]: pgmap v5912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:42:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:36 smithi059 ceph-mon[97450]: pgmap v5912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:42:36.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:42:36.365+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:42:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:36.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:36.396473+0000) 2022-09-08T00:42:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:36.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:36.324350+0000) 2022-09-08T00:42:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:37.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:37.396694+0000) 2022-09-08T00:42:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:37.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:37.324574+0000) 2022-09-08T00:42:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:38 smithi055 ceph-mon[97033]: pgmap v5913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:38 smithi059 ceph-mon[97450]: pgmap v5913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:38.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:38.324750+0000) 2022-09-08T00:42:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:38.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:38.396890+0000) 2022-09-08T00:42:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:39.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:39.397146+0000) 2022-09-08T00:42:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:39.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:39.324978+0000) 2022-09-08T00:42:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:40 smithi055 ceph-mon[97033]: pgmap v5914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:40 smithi059 ceph-mon[97450]: pgmap v5914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:40.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:40.397368+0000) 2022-09-08T00:42:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:40.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:40.325227+0000) 2022-09-08T00:42:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:41.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:41.397549+0000) 2022-09-08T00:42:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:41.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:41.325477+0000) 2022-09-08T00:42:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:42 smithi055 ceph-mon[97033]: pgmap v5915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:42:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:42 smithi059 ceph-mon[97450]: pgmap v5915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:42:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:42.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:42.397715+0000) 2022-09-08T00:42:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:42.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:42.325682+0000) 2022-09-08T00:42:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:42:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:42:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:42:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:42:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:42:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:42:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:43.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:43.397867+0000) 2022-09-08T00:42:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:43.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:43.325842+0000) 2022-09-08T00:42:44.439 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:44 smithi055 ceph-mon[97033]: pgmap v5916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:44.508 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:44 smithi059 ceph-mon[97450]: pgmap v5916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:44.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:44.326082+0000) 2022-09-08T00:42:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:44.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:44.398094+0000) 2022-09-08T00:42:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:45.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:45.398346+0000) 2022-09-08T00:42:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:45.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:45.326321+0000) 2022-09-08T00:42:46.354 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:46 smithi055 ceph-mon[97033]: pgmap v5917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:42:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:46 smithi059 ceph-mon[97450]: pgmap v5917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:42:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:46.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:46.398669+0000) 2022-09-08T00:42:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:46.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:46.326479+0000) 2022-09-08T00:42:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:47.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:47.398945+0000) 2022-09-08T00:42:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:47.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:47.326748+0000) 2022-09-08T00:42:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:47 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T00:42:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:42:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:42:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:47 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T00:42:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:42:47.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:42:47.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:42:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:47 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T00:42:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:42:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:42:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:47 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T00:42:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:42:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:42:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:42:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:48.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:48.399084+0000) 2022-09-08T00:42:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:48.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:48.327143+0000) 2022-09-08T00:42:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:48 smithi055 ceph-mon[97033]: pgmap v5918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:48 smithi059 ceph-mon[97450]: pgmap v5918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:49.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:49.399301+0000) 2022-09-08T00:42:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:49.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:49.327362+0000) 2022-09-08T00:42:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:50.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:50.399579+0000) 2022-09-08T00:42:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:50.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:50.327544+0000) 2022-09-08T00:42:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:50 smithi055 ceph-mon[97033]: pgmap v5919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:50 smithi059 ceph-mon[97450]: pgmap v5919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:51.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:51.327790+0000) 2022-09-08T00:42:51.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:42:51.381+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:42:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:51.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:51.399840+0000) 2022-09-08T00:42:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:52.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:52.400061+0000) 2022-09-08T00:42:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:52.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:52.328014+0000) 2022-09-08T00:42:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:52 smithi055 ceph-mon[97033]: pgmap v5920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:42:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:52 smithi059 ceph-mon[97450]: pgmap v5920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:42:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:53.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:53.400297+0000) 2022-09-08T00:42:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:53.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:53.328249+0000) 2022-09-08T00:42:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:54.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:54.400513+0000) 2022-09-08T00:42:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:54.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:54.328478+0000) 2022-09-08T00:42:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:54 smithi055 ceph-mon[97033]: pgmap v5921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:54 smithi059 ceph-mon[97450]: pgmap v5921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:55.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:55.400672+0000) 2022-09-08T00:42:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:55.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:55.328747+0000) 2022-09-08T00:42:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:56.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:56.329013+0000) 2022-09-08T00:42:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:56.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:56.400905+0000) 2022-09-08T00:42:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:56 smithi055 ceph-mon[97033]: pgmap v5922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:42:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:56 smithi059 ceph-mon[97450]: pgmap v5922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:42:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:57.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:57.401097+0000) 2022-09-08T00:42:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:57.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:57.329187+0000) 2022-09-08T00:42:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:58.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:58.401312+0000) 2022-09-08T00:42:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:58.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:58.329351+0000) 2022-09-08T00:42:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:42:58 smithi055 ceph-mon[97033]: pgmap v5923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:42:58 smithi059 ceph-mon[97450]: pgmap v5923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:42:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:42:59.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:59.401429+0000) 2022-09-08T00:42:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:42:59.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:42:59.329565+0000) 2022-09-08T00:43:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:00.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:00.329786+0000) 2022-09-08T00:43:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:00.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:00.401676+0000) 2022-09-08T00:43:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:00 smithi055 ceph-mon[97033]: pgmap v5924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:00 smithi059 ceph-mon[97450]: pgmap v5924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:01.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:01.401849+0000) 2022-09-08T00:43:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:01.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:01.330035+0000) 2022-09-08T00:43:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:02.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:02.401978+0000) 2022-09-08T00:43:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:02.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:02.330185+0000) 2022-09-08T00:43:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:02 smithi055 ceph-mon[97033]: pgmap v5925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:43:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:02 smithi059 ceph-mon[97450]: pgmap v5925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:43:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:03.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:03.402202+0000) 2022-09-08T00:43:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:03.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:03.330389+0000) 2022-09-08T00:43:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:04.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:04.402448+0000) 2022-09-08T00:43:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:04.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:04.330671+0000) 2022-09-08T00:43:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:04 smithi055 ceph-mon[97033]: pgmap v5926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:04 smithi059 ceph-mon[97450]: pgmap v5926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:05.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:05.402722+0000) 2022-09-08T00:43:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:05.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:05.330958+0000) 2022-09-08T00:43:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:43:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:43:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:43:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:43:06.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:43:06.393+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:43:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:06.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:06.402896+0000) 2022-09-08T00:43:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:06.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:06.331155+0000) 2022-09-08T00:43:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:06 smithi055 ceph-mon[97033]: pgmap v5927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:43:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:06 smithi059 ceph-mon[97450]: pgmap v5927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:43:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:07.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:07.331402+0000) 2022-09-08T00:43:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:07.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:07.403105+0000) 2022-09-08T00:43:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:08.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:08.403335+0000) 2022-09-08T00:43:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:08.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:08.331534+0000) 2022-09-08T00:43:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:08 smithi055 ceph-mon[97033]: pgmap v5928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:43:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:08 smithi059 ceph-mon[97450]: pgmap v5928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:43:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:09.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:09.403578+0000) 2022-09-08T00:43:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:09.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:09.331717+0000) 2022-09-08T00:43:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:09 smithi055 ceph-mon[97033]: osdmap e455: 12 total, 11 up, 11 in 2022-09-08T00:43:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:09 smithi059 ceph-mon[97450]: osdmap e455: 12 total, 11 up, 11 in 2022-09-08T00:43:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:10.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:10.403883+0000) 2022-09-08T00:43:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:10.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:10.331936+0000) 2022-09-08T00:43:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:10 smithi055 ceph-mon[97033]: pgmap v5930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:43:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:10 smithi059 ceph-mon[97450]: pgmap v5930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:43:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:11.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:11.404112+0000) 2022-09-08T00:43:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:11.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:11.332154+0000) 2022-09-08T00:43:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:12.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:12.404322+0000) 2022-09-08T00:43:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:12.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:12.332394+0000) 2022-09-08T00:43:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:12 smithi059 ceph-mon[97450]: pgmap v5931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:43:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:12 smithi055 ceph-mon[97033]: pgmap v5931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:43:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:13.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:13.404538+0000) 2022-09-08T00:43:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:13.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:13.332559+0000) 2022-09-08T00:43:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:14.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:14.404770+0000) 2022-09-08T00:43:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:14.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:14.332750+0000) 2022-09-08T00:43:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:14 smithi059 ceph-mon[97450]: pgmap v5932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:43:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:14 smithi055 ceph-mon[97033]: pgmap v5932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:43:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:15.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:15.405016+0000) 2022-09-08T00:43:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:15.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:15.332976+0000) 2022-09-08T00:43:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:16.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:16.405184+0000) 2022-09-08T00:43:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:16.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:16.333251+0000) 2022-09-08T00:43:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:16 smithi059 ceph-mon[97450]: pgmap v5933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:43:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:16 smithi059 ceph-mon[97450]: osdmap e456: 12 total, 11 up, 11 in 2022-09-08T00:43:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:16 smithi055 ceph-mon[97033]: pgmap v5933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:43:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:16 smithi055 ceph-mon[97033]: osdmap e456: 12 total, 11 up, 11 in 2022-09-08T00:43:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:17.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:17.405420+0000) 2022-09-08T00:43:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:17.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:17.333516+0000) 2022-09-08T00:43:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:18.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:18.405663+0000) 2022-09-08T00:43:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:18.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:18.333756+0000) 2022-09-08T00:43:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:18 smithi059 ceph-mon[97450]: pgmap v5935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T00:43:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:18 smithi055 ceph-mon[97033]: pgmap v5935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T00:43:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:19.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:19.405891+0000) 2022-09-08T00:43:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:19.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:19.334034+0000) 2022-09-08T00:43:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:20.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:20.406144+0000) 2022-09-08T00:43:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:20.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:20.334289+0000) 2022-09-08T00:43:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:20 smithi059 ceph-mon[97450]: pgmap v5936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:43:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:20 smithi055 ceph-mon[97033]: pgmap v5936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:43:21.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:43:21.408+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:43:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:21.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:21.406380+0000) 2022-09-08T00:43:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:21.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:21.334519+0000) 2022-09-08T00:43:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:22.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:22.406626+0000) 2022-09-08T00:43:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:22.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:22.334708+0000) 2022-09-08T00:43:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:22 smithi059 ceph-mon[97450]: pgmap v5937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:43:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:22 smithi055 ceph-mon[97033]: pgmap v5937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:43:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:23.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:23.406829+0000) 2022-09-08T00:43:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:23.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:23.334856+0000) 2022-09-08T00:43:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:24.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:24.407074+0000) 2022-09-08T00:43:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:24.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:24.335079+0000) 2022-09-08T00:43:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:24 smithi059 ceph-mon[97450]: pgmap v5938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:43:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:24 smithi055 ceph-mon[97033]: pgmap v5938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:43:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:25.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:25.407326+0000) 2022-09-08T00:43:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:25.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:25.335310+0000) 2022-09-08T00:43:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:26.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:26.407508+0000) 2022-09-08T00:43:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:26.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:26.335560+0000) 2022-09-08T00:43:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:26 smithi059 ceph-mon[97450]: pgmap v5939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:43:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:26 smithi055 ceph-mon[97033]: pgmap v5939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:43:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:27.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:27.407716+0000) 2022-09-08T00:43:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:27.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:27.335765+0000) 2022-09-08T00:43:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:28.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:28.407929+0000) 2022-09-08T00:43:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:28.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:28.335937+0000) 2022-09-08T00:43:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:28 smithi059 ceph-mon[97450]: pgmap v5940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:43:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:28 smithi055 ceph-mon[97033]: pgmap v5940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:43:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:29.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:29.408144+0000) 2022-09-08T00:43:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:29.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:29.336154+0000) 2022-09-08T00:43:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:30.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:30.408358+0000) 2022-09-08T00:43:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:30.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:30.336312+0000) 2022-09-08T00:43:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:30 smithi059 ceph-mon[97450]: pgmap v5941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:30 smithi055 ceph-mon[97033]: pgmap v5941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:31.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:31.408576+0000) 2022-09-08T00:43:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:31.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:31.336515+0000) 2022-09-08T00:43:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:32.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:32.408761+0000) 2022-09-08T00:43:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:32.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:32.336775+0000) 2022-09-08T00:43:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:32 smithi059 ceph-mon[97450]: pgmap v5942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:43:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:32 smithi055 ceph-mon[97033]: pgmap v5942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:43:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:33.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:33.408952+0000) 2022-09-08T00:43:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:33.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:33.336987+0000) 2022-09-08T00:43:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:34.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:34.409199+0000) 2022-09-08T00:43:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:34.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:34.337176+0000) 2022-09-08T00:43:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:34 smithi059 ceph-mon[97450]: pgmap v5943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:34 smithi055 ceph-mon[97033]: pgmap v5943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:35.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:35.409359+0000) 2022-09-08T00:43:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:35.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:35.337385+0000) 2022-09-08T00:43:36.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:43:36.424+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:43:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:36.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:36.337595+0000) 2022-09-08T00:43:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:36.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:36.409586+0000) 2022-09-08T00:43:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:36 smithi059 ceph-mon[97450]: pgmap v5944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:43:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:36 smithi055 ceph-mon[97033]: pgmap v5944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:43:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:37.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:37.409827+0000) 2022-09-08T00:43:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:37.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:37.337841+0000) 2022-09-08T00:43:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:37 smithi055 ceph-mon[97033]: pgmap v5945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:37 smithi059 ceph-mon[97450]: pgmap v5945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:38.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:38.410095+0000) 2022-09-08T00:43:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:38.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:38.337989+0000) 2022-09-08T00:43:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:39.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:39.338165+0000) 2022-09-08T00:43:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:39.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:39.410375+0000) 2022-09-08T00:43:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:40 smithi059 ceph-mon[97450]: pgmap v5946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:40 smithi055 ceph-mon[97033]: pgmap v5946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:40.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:40.410716+0000) 2022-09-08T00:43:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:40.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:40.338492+0000) 2022-09-08T00:43:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:41.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:41.410965+0000) 2022-09-08T00:43:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:41.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:41.338742+0000) 2022-09-08T00:43:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:42 smithi059 ceph-mon[97450]: pgmap v5947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:43:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:42 smithi055 ceph-mon[97033]: pgmap v5947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:43:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:42.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:42.411182+0000) 2022-09-08T00:43:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:42.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:42.338905+0000) 2022-09-08T00:43:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:43.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:43.411418+0000) 2022-09-08T00:43:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:43.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:43.339083+0000) 2022-09-08T00:43:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:44 smithi059 ceph-mon[97450]: pgmap v5948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:44 smithi055 ceph-mon[97033]: pgmap v5948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:44.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:44.339284+0000) 2022-09-08T00:43:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:44.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:44.411721+0000) 2022-09-08T00:43:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:45.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:45.339460+0000) 2022-09-08T00:43:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:45.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:45.411945+0000) 2022-09-08T00:43:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:46 smithi059 ceph-mon[97450]: pgmap v5949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:43:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:46 smithi055 ceph-mon[97033]: pgmap v5949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:43:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:46.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:46.339683+0000) 2022-09-08T00:43:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:46.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:46.412072+0000) 2022-09-08T00:43:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:43:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:43:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:43:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:43:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:43:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:43:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:47.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:47.412254+0000) 2022-09-08T00:43:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:47.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:47.339817+0000) 2022-09-08T00:43:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:48 smithi059 ceph-mon[97450]: pgmap v5950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:43:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:43:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:43:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:48.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:48.412436+0000) 2022-09-08T00:43:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:48.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:48.339996+0000) 2022-09-08T00:43:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:48 smithi055 ceph-mon[97033]: pgmap v5950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:43:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:43:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:43:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:49.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:49.340234+0000) 2022-09-08T00:43:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:49.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:49.412682+0000) 2022-09-08T00:43:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:50 smithi059 ceph-mon[97450]: pgmap v5951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:50.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:50.412912+0000) 2022-09-08T00:43:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:50.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:50.340471+0000) 2022-09-08T00:43:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:50 smithi055 ceph-mon[97033]: pgmap v5951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:51.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:43:51.435+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:43:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:51.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:51.413121+0000) 2022-09-08T00:43:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:51.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:51.340719+0000) 2022-09-08T00:43:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:52 smithi059 ceph-mon[97450]: pgmap v5952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:43:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:52.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:52.413319+0000) 2022-09-08T00:43:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:52.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:52.340944+0000) 2022-09-08T00:43:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:52 smithi055 ceph-mon[97033]: pgmap v5952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:43:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:53.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:53.413528+0000) 2022-09-08T00:43:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:53.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:53.341174+0000) 2022-09-08T00:43:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:54 smithi059 ceph-mon[97450]: pgmap v5953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:54.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:54.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:54.413737+0000) 2022-09-08T00:43:54.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:54.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:54.341415+0000) 2022-09-08T00:43:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:54 smithi055 ceph-mon[97033]: pgmap v5953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:55.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:55.413946+0000) 2022-09-08T00:43:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:55.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:55.341695+0000) 2022-09-08T00:43:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:56 smithi059 ceph-mon[97450]: pgmap v5954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:43:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:56.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:56.414152+0000) 2022-09-08T00:43:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:56.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:56.341931+0000) 2022-09-08T00:43:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:56 smithi055 ceph-mon[97033]: pgmap v5954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:43:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:57.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:57.414373+0000) 2022-09-08T00:43:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:57.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:57.342183+0000) 2022-09-08T00:43:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:43:58 smithi059 ceph-mon[97450]: pgmap v5955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:58.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:58.414576+0000) 2022-09-08T00:43:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:58.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:58.342418+0000) 2022-09-08T00:43:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:43:58 smithi055 ceph-mon[97033]: pgmap v5955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:43:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:43:59.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:59.342692+0000) 2022-09-08T00:43:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:43:59.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:43:59.414774+0000) 2022-09-08T00:44:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:00 smithi059 ceph-mon[97450]: pgmap v5956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:00.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:00.415023+0000) 2022-09-08T00:44:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:00.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:00.342909+0000) 2022-09-08T00:44:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:00 smithi055 ceph-mon[97033]: pgmap v5956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:01.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:01.343144+0000) 2022-09-08T00:44:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:01.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:01.415225+0000) 2022-09-08T00:44:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:02 smithi059 ceph-mon[97450]: pgmap v5957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:44:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:02.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:02.415491+0000) 2022-09-08T00:44:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:02.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:02.343378+0000) 2022-09-08T00:44:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:02 smithi055 ceph-mon[97033]: pgmap v5957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:44:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:03.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:03.415717+0000) 2022-09-08T00:44:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:03.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:03.343662+0000) 2022-09-08T00:44:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:04.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:04.415920+0000) 2022-09-08T00:44:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:04.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:04.343863+0000) 2022-09-08T00:44:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:04 smithi055 ceph-mon[97033]: pgmap v5958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:04 smithi059 ceph-mon[97450]: pgmap v5958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:05.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:05.344078+0000) 2022-09-08T00:44:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:05.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:05.416173+0000) 2022-09-08T00:44:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:44:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:44:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:44:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:44:06.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:44:06.449+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:44:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:06.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:06.344305+0000) 2022-09-08T00:44:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:06.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:06.416374+0000) 2022-09-08T00:44:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:06 smithi055 ceph-mon[97033]: pgmap v5959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:44:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:06 smithi059 ceph-mon[97450]: pgmap v5959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:44:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:07.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:07.416586+0000) 2022-09-08T00:44:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:07.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:07.344522+0000) 2022-09-08T00:44:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:08.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:08.416745+0000) 2022-09-08T00:44:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:08.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:08.344742+0000) 2022-09-08T00:44:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:08 smithi055 ceph-mon[97033]: pgmap v5960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:44:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:08 smithi059 ceph-mon[97450]: pgmap v5960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:44:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:09.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:09.416873+0000) 2022-09-08T00:44:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:09.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:09.344932+0000) 2022-09-08T00:44:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:10.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:10.417069+0000) 2022-09-08T00:44:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:10.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:10.345181+0000) 2022-09-08T00:44:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:10 smithi055 ceph-mon[97033]: pgmap v5961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:44:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:10 smithi055 ceph-mon[97033]: osdmap e457: 12 total, 11 up, 11 in 2022-09-08T00:44:10.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:10 smithi059 ceph-mon[97450]: pgmap v5961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:44:10.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:10 smithi059 ceph-mon[97450]: osdmap e457: 12 total, 11 up, 11 in 2022-09-08T00:44:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:11.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:11.417211+0000) 2022-09-08T00:44:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:11.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:11.345385+0000) 2022-09-08T00:44:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:12.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:12.417461+0000) 2022-09-08T00:44:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:12.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:12.345605+0000) 2022-09-08T00:44:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:12 smithi055 ceph-mon[97033]: pgmap v5963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:44:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:12 smithi059 ceph-mon[97450]: pgmap v5963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:44:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:13.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:13.417677+0000) 2022-09-08T00:44:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:13.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:13.345812+0000) 2022-09-08T00:44:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:14.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:14.346032+0000) 2022-09-08T00:44:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:14.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:14.417942+0000) 2022-09-08T00:44:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:14 smithi055 ceph-mon[97033]: pgmap v5964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:44:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:14 smithi059 ceph-mon[97450]: pgmap v5964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:44:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:15.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:15.346223+0000) 2022-09-08T00:44:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:15.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:15.418179+0000) 2022-09-08T00:44:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:16.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:16.418438+0000) 2022-09-08T00:44:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:16.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:16.346472+0000) 2022-09-08T00:44:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:16 smithi055 ceph-mon[97033]: pgmap v5965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:44:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:16 smithi055 ceph-mon[97033]: osdmap e458: 12 total, 11 up, 11 in 2022-09-08T00:44:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:16 smithi059 ceph-mon[97450]: pgmap v5965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:44:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:16 smithi059 ceph-mon[97450]: osdmap e458: 12 total, 11 up, 11 in 2022-09-08T00:44:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:17.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:17.418683+0000) 2022-09-08T00:44:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:17.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:17.346701+0000) 2022-09-08T00:44:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:18.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:18.418872+0000) 2022-09-08T00:44:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:18.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:18.346857+0000) 2022-09-08T00:44:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:18 smithi055 ceph-mon[97033]: pgmap v5967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:44:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:18 smithi059 ceph-mon[97450]: pgmap v5967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:44:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:19.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:19.419110+0000) 2022-09-08T00:44:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:19.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:19.347100+0000) 2022-09-08T00:44:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:20.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:20.419363+0000) 2022-09-08T00:44:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:20.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:20.347358+0000) 2022-09-08T00:44:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:20 smithi055 ceph-mon[97033]: pgmap v5968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 211 B/s wr, 6 op/s 2022-09-08T00:44:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:20 smithi059 ceph-mon[97450]: pgmap v5968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 211 B/s wr, 6 op/s 2022-09-08T00:44:21.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:44:21.461+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:44:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:21.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:21.347618+0000) 2022-09-08T00:44:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:21.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:21.419588+0000) 2022-09-08T00:44:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:22.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:22.419829+0000) 2022-09-08T00:44:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:22.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:22.347839+0000) 2022-09-08T00:44:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:22 smithi055 ceph-mon[97033]: pgmap v5969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:44:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:22 smithi059 ceph-mon[97450]: pgmap v5969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:44:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:23.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:23.420009+0000) 2022-09-08T00:44:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:23.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:23.348109+0000) 2022-09-08T00:44:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:24.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:24.420230+0000) 2022-09-08T00:44:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:24.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:24.348315+0000) 2022-09-08T00:44:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:24 smithi055 ceph-mon[97033]: pgmap v5970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:44:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:24 smithi059 ceph-mon[97450]: pgmap v5970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:44:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:25.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:25.420469+0000) 2022-09-08T00:44:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:25.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:25.348579+0000) 2022-09-08T00:44:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:26.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:26.420690+0000) 2022-09-08T00:44:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:26.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:26.348739+0000) 2022-09-08T00:44:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:26 smithi055 ceph-mon[97033]: pgmap v5971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:44:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:26 smithi059 ceph-mon[97450]: pgmap v5971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:44:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:27.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:27.420920+0000) 2022-09-08T00:44:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:27.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:27.348995+0000) 2022-09-08T00:44:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:28.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:28.421151+0000) 2022-09-08T00:44:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:28.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:28.349220+0000) 2022-09-08T00:44:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:28 smithi055 ceph-mon[97033]: pgmap v5972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:44:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:28 smithi059 ceph-mon[97450]: pgmap v5972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:44:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:29.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:29.421401+0000) 2022-09-08T00:44:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:29.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:29.349449+0000) 2022-09-08T00:44:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:30.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:30.421631+0000) 2022-09-08T00:44:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:30.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:30.349724+0000) 2022-09-08T00:44:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:30 smithi055 ceph-mon[97033]: pgmap v5973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:30 smithi059 ceph-mon[97450]: pgmap v5973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:31.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:31.421914+0000) 2022-09-08T00:44:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:31.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:31.349991+0000) 2022-09-08T00:44:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:32.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:32.422109+0000) 2022-09-08T00:44:32.664 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:32.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:32.350264+0000) 2022-09-08T00:44:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:32 smithi055 ceph-mon[97033]: pgmap v5974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:44:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:32 smithi059 ceph-mon[97450]: pgmap v5974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:44:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:33.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:33.422378+0000) 2022-09-08T00:44:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:33.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:33.350515+0000) 2022-09-08T00:44:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:34.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:34.422604+0000) 2022-09-08T00:44:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:34.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:34.350764+0000) 2022-09-08T00:44:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:34 smithi055 ceph-mon[97033]: pgmap v5975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:34 smithi059 ceph-mon[97450]: pgmap v5975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:35.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:35.422771+0000) 2022-09-08T00:44:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:35.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:35.351006+0000) 2022-09-08T00:44:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:36.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:36.422969+0000) 2022-09-08T00:44:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:36.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:36.351274+0000) 2022-09-08T00:44:36.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:44:36.474+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:44:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:36 smithi055 ceph-mon[97033]: pgmap v5976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:44:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:36 smithi059 ceph-mon[97450]: pgmap v5976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:44:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:37.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:37.423148+0000) 2022-09-08T00:44:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:37.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:37.351521+0000) 2022-09-08T00:44:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:38.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:38.423363+0000) 2022-09-08T00:44:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:38.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:38.351776+0000) 2022-09-08T00:44:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:38 smithi055 ceph-mon[97033]: pgmap v5977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:38 smithi059 ceph-mon[97450]: pgmap v5977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:39.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:39.423592+0000) 2022-09-08T00:44:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:39.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:39.352037+0000) 2022-09-08T00:44:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:40.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:40.423800+0000) 2022-09-08T00:44:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:40.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:40.352242+0000) 2022-09-08T00:44:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:40 smithi055 ceph-mon[97033]: pgmap v5978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:40 smithi059 ceph-mon[97450]: pgmap v5978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:41.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:41.352464+0000) 2022-09-08T00:44:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:41.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:41.424029+0000) 2022-09-08T00:44:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:42.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:42.424235+0000) 2022-09-08T00:44:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:42.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:42.352723+0000) 2022-09-08T00:44:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:42 smithi055 ceph-mon[97033]: pgmap v5979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:44:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:42 smithi059 ceph-mon[97450]: pgmap v5979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:44:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:43.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:43.424454+0000) 2022-09-08T00:44:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:43.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:43.352875+0000) 2022-09-08T00:44:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:44.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:44.424725+0000) 2022-09-08T00:44:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:44.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:44.353056+0000) 2022-09-08T00:44:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:44 smithi055 ceph-mon[97033]: pgmap v5980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:44 smithi059 ceph-mon[97450]: pgmap v5980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:45.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:45.424894+0000) 2022-09-08T00:44:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:45.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:45.353273+0000) 2022-09-08T00:44:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:46.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:46.425114+0000) 2022-09-08T00:44:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:46.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:46.353548+0000) 2022-09-08T00:44:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:46 smithi055 ceph-mon[97033]: pgmap v5981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:44:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:46 smithi059 ceph-mon[97450]: pgmap v5981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:44:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:47.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:47.425313+0000) 2022-09-08T00:44:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:47.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:47.353775+0000) 2022-09-08T00:44:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:48.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:48.353947+0000) 2022-09-08T00:44:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:48.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:48.425465+0000) 2022-09-08T00:44:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:48 smithi055 ceph-mon[97033]: pgmap v5982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:44:48.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:44:48.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:44:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:48 smithi059 ceph-mon[97450]: pgmap v5982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:44:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:44:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:44:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:49.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:49.354087+0000) 2022-09-08T00:44:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:49.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:49.425650+0000) 2022-09-08T00:44:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:44:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:44:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:44:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:44:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:44:50.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:49 smithi055 ceph-mon[97033]: pgmap v5983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:44:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:44:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:44:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:44:50.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:44:50.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:49 smithi059 ceph-mon[97450]: pgmap v5983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:50.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:50.354299+0000) 2022-09-08T00:44:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:50.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:50.425885+0000) 2022-09-08T00:44:51.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:44:51.490+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:44:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:51.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:51.426126+0000) 2022-09-08T00:44:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:51.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:51.354492+0000) 2022-09-08T00:44:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:52 smithi055 ceph-mon[97033]: pgmap v5984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:44:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:52 smithi059 ceph-mon[97450]: pgmap v5984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:44:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:52.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:52.426305+0000) 2022-09-08T00:44:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:52.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:52.354747+0000) 2022-09-08T00:44:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:53.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:53.426465+0000) 2022-09-08T00:44:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:53.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:53.354895+0000) 2022-09-08T00:44:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:54 smithi055 ceph-mon[97033]: pgmap v5985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:54 smithi059 ceph-mon[97450]: pgmap v5985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:54.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:54.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:54.426711+0000) 2022-09-08T00:44:54.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:54.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:54.355095+0000) 2022-09-08T00:44:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:55.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:55.355371+0000) 2022-09-08T00:44:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:55.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:55.426936+0000) 2022-09-08T00:44:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:56 smithi055 ceph-mon[97033]: pgmap v5986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:44:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:56 smithi059 ceph-mon[97450]: pgmap v5986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:44:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:56.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:56.427076+0000) 2022-09-08T00:44:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:56.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:56.355625+0000) 2022-09-08T00:44:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:57.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:57.427301+0000) 2022-09-08T00:44:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:57.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:57.355890+0000) 2022-09-08T00:44:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:44:58 smithi055 ceph-mon[97033]: pgmap v5987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:44:58 smithi059 ceph-mon[97450]: pgmap v5987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:44:58.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:58.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:58.356103+0000) 2022-09-08T00:44:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:58.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:58.427439+0000) 2022-09-08T00:44:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:44:59.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:59.427666+0000) 2022-09-08T00:44:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:44:59.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:44:59.356357+0000) 2022-09-08T00:45:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:00 smithi055 ceph-mon[97033]: pgmap v5988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:00 smithi059 ceph-mon[97450]: pgmap v5988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:00.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:00.427927+0000) 2022-09-08T00:45:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:00.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:00.356492+0000) 2022-09-08T00:45:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:01.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:01.428178+0000) 2022-09-08T00:45:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:01.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:01.356730+0000) 2022-09-08T00:45:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:02 smithi055 ceph-mon[97033]: pgmap v5989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:45:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:02 smithi059 ceph-mon[97450]: pgmap v5989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:45:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:02.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:02.428356+0000) 2022-09-08T00:45:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:02.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:02.356857+0000) 2022-09-08T00:45:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:03.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:03.428607+0000) 2022-09-08T00:45:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:03.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:03.357070+0000) 2022-09-08T00:45:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:04 smithi055 ceph-mon[97033]: pgmap v5990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:04 smithi059 ceph-mon[97450]: pgmap v5990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:04.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:04.357274+0000) 2022-09-08T00:45:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:04.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:04.428897+0000) 2022-09-08T00:45:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:45:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:45:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:45:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:45:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:05.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:05.357530+0000) 2022-09-08T00:45:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:05.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:05.429113+0000) 2022-09-08T00:45:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:06 smithi055 ceph-mon[97033]: pgmap v5991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:45:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:06 smithi059 ceph-mon[97450]: pgmap v5991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:45:06.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:45:06.505+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:45:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:06.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:06.357717+0000) 2022-09-08T00:45:06.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:06.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:06.429364+0000) 2022-09-08T00:45:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:07.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:07.429599+0000) 2022-09-08T00:45:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:07.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:07.357957+0000) 2022-09-08T00:45:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:08 smithi055 ceph-mon[97033]: pgmap v5992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:45:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:08 smithi059 ceph-mon[97450]: pgmap v5992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:45:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:08.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:08.358199+0000) 2022-09-08T00:45:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:08.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:08.429832+0000) 2022-09-08T00:45:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:09.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:09.430011+0000) 2022-09-08T00:45:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:09.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:09.358391+0000) 2022-09-08T00:45:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:10 smithi055 ceph-mon[97033]: pgmap v5993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:45:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:10 smithi055 ceph-mon[97033]: osdmap e459: 12 total, 11 up, 11 in 2022-09-08T00:45:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:10 smithi059 ceph-mon[97450]: pgmap v5993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:45:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:10 smithi059 ceph-mon[97450]: osdmap e459: 12 total, 11 up, 11 in 2022-09-08T00:45:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:10.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:10.358592+0000) 2022-09-08T00:45:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:10.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:10.430242+0000) 2022-09-08T00:45:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:11.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:11.430488+0000) 2022-09-08T00:45:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:11.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:11.358861+0000) 2022-09-08T00:45:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:12 smithi055 ceph-mon[97033]: pgmap v5995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:45:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:12 smithi059 ceph-mon[97450]: pgmap v5995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:45:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:12.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:12.430671+0000) 2022-09-08T00:45:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:12.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:12.359111+0000) 2022-09-08T00:45:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:13.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:13.430882+0000) 2022-09-08T00:45:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:13.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:13.359310+0000) 2022-09-08T00:45:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:14 smithi055 ceph-mon[97033]: pgmap v5996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:45:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:14 smithi059 ceph-mon[97450]: pgmap v5996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:45:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:14.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:14.431077+0000) 2022-09-08T00:45:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:14.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:14.359551+0000) 2022-09-08T00:45:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:15.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:15.431274+0000) 2022-09-08T00:45:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:15.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:15.359817+0000) 2022-09-08T00:45:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:16 smithi055 ceph-mon[97033]: pgmap v5997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:45:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:16 smithi055 ceph-mon[97033]: osdmap e460: 12 total, 11 up, 11 in 2022-09-08T00:45:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:16 smithi059 ceph-mon[97450]: pgmap v5997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:45:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:16 smithi059 ceph-mon[97450]: osdmap e460: 12 total, 11 up, 11 in 2022-09-08T00:45:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:16.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:16.431522+0000) 2022-09-08T00:45:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:16.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:16.360026+0000) 2022-09-08T00:45:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:17.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:17.431736+0000) 2022-09-08T00:45:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:17.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:17.360263+0000) 2022-09-08T00:45:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:18 smithi055 ceph-mon[97033]: pgmap v5999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:45:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:18 smithi059 ceph-mon[97450]: pgmap v5999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:45:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:18.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:18.431970+0000) 2022-09-08T00:45:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:18.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:18.360547+0000) 2022-09-08T00:45:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:19.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:19.432194+0000) 2022-09-08T00:45:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:19.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:19.360786+0000) 2022-09-08T00:45:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:20 smithi055 ceph-mon[97033]: pgmap v6000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:45:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:20 smithi059 ceph-mon[97450]: pgmap v6000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:45:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:20.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:20.432432+0000) 2022-09-08T00:45:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:20.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:20.361041+0000) 2022-09-08T00:45:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:21.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:21.432612+0000) 2022-09-08T00:45:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:21.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:21.361262+0000) 2022-09-08T00:45:21.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:45:21.520+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:45:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:22 smithi055 ceph-mon[97033]: pgmap v6001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:45:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:22 smithi059 ceph-mon[97450]: pgmap v6001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:45:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:22.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:22.432852+0000) 2022-09-08T00:45:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:22.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:22.361429+0000) 2022-09-08T00:45:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:23.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:23.433087+0000) 2022-09-08T00:45:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:23.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:23.361678+0000) 2022-09-08T00:45:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:24 smithi055 ceph-mon[97033]: pgmap v6002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:45:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:24 smithi059 ceph-mon[97450]: pgmap v6002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:45:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:24.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:24.433332+0000) 2022-09-08T00:45:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:24.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:24.361929+0000) 2022-09-08T00:45:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:25.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:25.433548+0000) 2022-09-08T00:45:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:25.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:25.362138+0000) 2022-09-08T00:45:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:26 smithi055 ceph-mon[97033]: pgmap v6003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:45:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:26 smithi059 ceph-mon[97450]: pgmap v6003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:45:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:26.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:26.433788+0000) 2022-09-08T00:45:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:26.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:26.362325+0000) 2022-09-08T00:45:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:27.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:27.434036+0000) 2022-09-08T00:45:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:27.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:27.362537+0000) 2022-09-08T00:45:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:28 smithi055 ceph-mon[97033]: pgmap v6004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:45:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:28 smithi059 ceph-mon[97450]: pgmap v6004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:45:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:28.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:28.434272+0000) 2022-09-08T00:45:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:28.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:28.362749+0000) 2022-09-08T00:45:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:29.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:29.434542+0000) 2022-09-08T00:45:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:29.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:29.362894+0000) 2022-09-08T00:45:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:30 smithi055 ceph-mon[97033]: pgmap v6005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:30 smithi059 ceph-mon[97450]: pgmap v6005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:30.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:30.434883+0000) 2022-09-08T00:45:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:30.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:30.363013+0000) 2022-09-08T00:45:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:31.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:31.435152+0000) 2022-09-08T00:45:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:31.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:31.363203+0000) 2022-09-08T00:45:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:32 smithi055 ceph-mon[97033]: pgmap v6006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:45:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:32 smithi059 ceph-mon[97450]: pgmap v6006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:45:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:32.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:32.435363+0000) 2022-09-08T00:45:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:32.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:32.363441+0000) 2022-09-08T00:45:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:33.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:33.435575+0000) 2022-09-08T00:45:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:33.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:33.363685+0000) 2022-09-08T00:45:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:34 smithi055 ceph-mon[97033]: pgmap v6007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:34 smithi059 ceph-mon[97450]: pgmap v6007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:34.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:34.435812+0000) 2022-09-08T00:45:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:34.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:34.363820+0000) 2022-09-08T00:45:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:35.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:35.436042+0000) 2022-09-08T00:45:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:35.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:35.363965+0000) 2022-09-08T00:45:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:36 smithi055 ceph-mon[97033]: pgmap v6008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:45:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:36 smithi059 ceph-mon[97450]: pgmap v6008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:45:36.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:45:36.530+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:45:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:36.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:36.436273+0000) 2022-09-08T00:45:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:36.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:36.364113+0000) 2022-09-08T00:45:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:37.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:37.436502+0000) 2022-09-08T00:45:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:37.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:37.364342+0000) 2022-09-08T00:45:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:38 smithi055 ceph-mon[97033]: pgmap v6009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:38 smithi059 ceph-mon[97450]: pgmap v6009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:38.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:38.436749+0000) 2022-09-08T00:45:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:38.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:38.364573+0000) 2022-09-08T00:45:39.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:39.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:39.364820+0000) 2022-09-08T00:45:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:39.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:39.436980+0000) 2022-09-08T00:45:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:40 smithi055 ceph-mon[97033]: pgmap v6010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:40 smithi059 ceph-mon[97450]: pgmap v6010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:40.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:40.365073+0000) 2022-09-08T00:45:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:40.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:40.437216+0000) 2022-09-08T00:45:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:41.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:41.365304+0000) 2022-09-08T00:45:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:41.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:41.437414+0000) 2022-09-08T00:45:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:42 smithi055 ceph-mon[97033]: pgmap v6011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:45:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:42 smithi059 ceph-mon[97450]: pgmap v6011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:45:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:42.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:42.437632+0000) 2022-09-08T00:45:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:42.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:42.365539+0000) 2022-09-08T00:45:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:43.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:43.437854+0000) 2022-09-08T00:45:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:43.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:43.365763+0000) 2022-09-08T00:45:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:44 smithi055 ceph-mon[97033]: pgmap v6012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:44 smithi059 ceph-mon[97450]: pgmap v6012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:44.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:44.365956+0000) 2022-09-08T00:45:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:44.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:44.438018+0000) 2022-09-08T00:45:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:45.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:45.438238+0000) 2022-09-08T00:45:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:45.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:45.366144+0000) 2022-09-08T00:45:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:46 smithi055 ceph-mon[97033]: pgmap v6013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:45:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:46 smithi059 ceph-mon[97450]: pgmap v6013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:45:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:46.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:46.366328+0000) 2022-09-08T00:45:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:46.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:46.438484+0000) 2022-09-08T00:45:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:47.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:47.438721+0000) 2022-09-08T00:45:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:47.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:47.366522+0000) 2022-09-08T00:45:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:48 smithi055 ceph-mon[97033]: pgmap v6014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:48 smithi059 ceph-mon[97450]: pgmap v6014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:48.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:48.438933+0000) 2022-09-08T00:45:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:48.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:48.366757+0000) 2022-09-08T00:45:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:45:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:45:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:45:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:45:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:45:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:45:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:49.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:49.439104+0000) 2022-09-08T00:45:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:49.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:49.367006+0000) 2022-09-08T00:45:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:50.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:50.439333+0000) 2022-09-08T00:45:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:50.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:50.367233+0000) 2022-09-08T00:45:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:50 smithi055 ceph-mon[97033]: pgmap v6015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:45:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:50 smithi059 ceph-mon[97450]: pgmap v6015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:45:51.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:45:51.545+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:45:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:51.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:51.367433+0000) 2022-09-08T00:45:51.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:51.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:51.439570+0000) 2022-09-08T00:45:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:52.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:52.439761+0000) 2022-09-08T00:45:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:52.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:52.367609+0000) 2022-09-08T00:45:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:52 smithi055 ceph-mon[97033]: pgmap v6016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:45:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:52 smithi059 ceph-mon[97450]: pgmap v6016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:45:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:53.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:53.440015+0000) 2022-09-08T00:45:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:53.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:53.367886+0000) 2022-09-08T00:45:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:54.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:54.440274+0000) 2022-09-08T00:45:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:54.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:54.368088+0000) 2022-09-08T00:45:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:54 smithi055 ceph-mon[97033]: pgmap v6017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:54 smithi059 ceph-mon[97450]: pgmap v6017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:55.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:55.440496+0000) 2022-09-08T00:45:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:55.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:55.368297+0000) 2022-09-08T00:45:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:56.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:56.440737+0000) 2022-09-08T00:45:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:56.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:56.368467+0000) 2022-09-08T00:45:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:56 smithi055 ceph-mon[97033]: pgmap v6018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:45:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:56 smithi059 ceph-mon[97450]: pgmap v6018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:45:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:57.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:57.368722+0000) 2022-09-08T00:45:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:57.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:57.440961+0000) 2022-09-08T00:45:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:58.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:58.441197+0000) 2022-09-08T00:45:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:58.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:58.368949+0000) 2022-09-08T00:45:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:45:58 smithi055 ceph-mon[97033]: pgmap v6019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:45:58 smithi059 ceph-mon[97450]: pgmap v6019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:45:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:45:59.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:59.441406+0000) 2022-09-08T00:45:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:45:59.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:45:59.369191+0000) 2022-09-08T00:46:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:00.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:00.369362+0000) 2022-09-08T00:46:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:00.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:00.441581+0000) 2022-09-08T00:46:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:00 smithi055 ceph-mon[97033]: pgmap v6020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:00 smithi059 ceph-mon[97450]: pgmap v6020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:01.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:01.441748+0000) 2022-09-08T00:46:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:01.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:01.369491+0000) 2022-09-08T00:46:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:02.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:02.441956+0000) 2022-09-08T00:46:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:02.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:02.369747+0000) 2022-09-08T00:46:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:02 smithi055 ceph-mon[97033]: pgmap v6021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:46:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:02 smithi059 ceph-mon[97450]: pgmap v6021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:46:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:03.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:03.442153+0000) 2022-09-08T00:46:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:03.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:03.370001+0000) 2022-09-08T00:46:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:04.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:04.442391+0000) 2022-09-08T00:46:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:04.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:04.370251+0000) 2022-09-08T00:46:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:04 smithi055 ceph-mon[97033]: pgmap v6022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:04 smithi059 ceph-mon[97450]: pgmap v6022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:05.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:05.442630+0000) 2022-09-08T00:46:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:05.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:05.370480+0000) 2022-09-08T00:46:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:46:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:46:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:46:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:46:06.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:46:06.551+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:46:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:06.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:06.442833+0000) 2022-09-08T00:46:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:06.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:06.370667+0000) 2022-09-08T00:46:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:06 smithi055 ceph-mon[97033]: pgmap v6023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:46:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:06 smithi059 ceph-mon[97450]: pgmap v6023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:46:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:07.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:07.443066+0000) 2022-09-08T00:46:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:07.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:07.370867+0000) 2022-09-08T00:46:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:08.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:08.443285+0000) 2022-09-08T00:46:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:08.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:08.371052+0000) 2022-09-08T00:46:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:08 smithi055 ceph-mon[97033]: pgmap v6024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:46:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:08 smithi059 ceph-mon[97450]: pgmap v6024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:46:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:09.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:09.443552+0000) 2022-09-08T00:46:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:09.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:09.371293+0000) 2022-09-08T00:46:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:09 smithi055 ceph-mon[97033]: osdmap e461: 12 total, 11 up, 11 in 2022-09-08T00:46:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:09 smithi059 ceph-mon[97450]: osdmap e461: 12 total, 11 up, 11 in 2022-09-08T00:46:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:10.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:10.443781+0000) 2022-09-08T00:46:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:10.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:10.371504+0000) 2022-09-08T00:46:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:10 smithi055 ceph-mon[97033]: pgmap v6026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:46:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:10 smithi059 ceph-mon[97450]: pgmap v6026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:46:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:11.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:11.443985+0000) 2022-09-08T00:46:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:11.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:11.371758+0000) 2022-09-08T00:46:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:12.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:12.444265+0000) 2022-09-08T00:46:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:12.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:12.371924+0000) 2022-09-08T00:46:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:12 smithi059 ceph-mon[97450]: pgmap v6027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:46:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:12 smithi055 ceph-mon[97033]: pgmap v6027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:46:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:13.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:13.372060+0000) 2022-09-08T00:46:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:13.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:13.444552+0000) 2022-09-08T00:46:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:14.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:14.372222+0000) 2022-09-08T00:46:14.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:14.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:14.444791+0000) 2022-09-08T00:46:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:14 smithi059 ceph-mon[97450]: pgmap v6028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:46:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:14 smithi055 ceph-mon[97033]: pgmap v6028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:46:15.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:15.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:15.372352+0000) 2022-09-08T00:46:15.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:15.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:15.445026+0000) 2022-09-08T00:46:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:16.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:16.372493+0000) 2022-09-08T00:46:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:16.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:16.445238+0000) 2022-09-08T00:46:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:16 smithi059 ceph-mon[97450]: pgmap v6029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 204 B/s wr, 5 op/s 2022-09-08T00:46:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:16 smithi059 ceph-mon[97450]: osdmap e462: 12 total, 11 up, 11 in 2022-09-08T00:46:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:16 smithi055 ceph-mon[97033]: pgmap v6029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 204 B/s wr, 5 op/s 2022-09-08T00:46:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:16 smithi055 ceph-mon[97033]: osdmap e462: 12 total, 11 up, 11 in 2022-09-08T00:46:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:17.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:17.372800+0000) 2022-09-08T00:46:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:17.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:17.445459+0000) 2022-09-08T00:46:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:18.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:18.445703+0000) 2022-09-08T00:46:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:18.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:18.372983+0000) 2022-09-08T00:46:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:18 smithi059 ceph-mon[97450]: pgmap v6031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 239 B/s wr, 6 op/s 2022-09-08T00:46:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:18 smithi055 ceph-mon[97033]: pgmap v6031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 239 B/s wr, 6 op/s 2022-09-08T00:46:19.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:19.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:19.373189+0000) 2022-09-08T00:46:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:19.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:19.445883+0000) 2022-09-08T00:46:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:20.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:20.446070+0000) 2022-09-08T00:46:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:20.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:20.373377+0000) 2022-09-08T00:46:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:20 smithi059 ceph-mon[97450]: pgmap v6032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 204 B/s wr, 5 op/s 2022-09-08T00:46:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:20 smithi055 ceph-mon[97033]: pgmap v6032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 204 B/s wr, 5 op/s 2022-09-08T00:46:21.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:46:21.566+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:46:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:21.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:21.446292+0000) 2022-09-08T00:46:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:21.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:21.373610+0000) 2022-09-08T00:46:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:22.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:22.446465+0000) 2022-09-08T00:46:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:22.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:22.373870+0000) 2022-09-08T00:46:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:22 smithi059 ceph-mon[97450]: pgmap v6033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:46:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:22 smithi055 ceph-mon[97033]: pgmap v6033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:46:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:23.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:23.446721+0000) 2022-09-08T00:46:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:23.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:23.374090+0000) 2022-09-08T00:46:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:24.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:24.446871+0000) 2022-09-08T00:46:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:24.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:24.374240+0000) 2022-09-08T00:46:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:24 smithi059 ceph-mon[97450]: pgmap v6034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:46:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:24 smithi055 ceph-mon[97033]: pgmap v6034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:46:25.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:25.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:25.447090+0000) 2022-09-08T00:46:25.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:25.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:25.374412+0000) 2022-09-08T00:46:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:26.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:26.447295+0000) 2022-09-08T00:46:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:26.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:26.374679+0000) 2022-09-08T00:46:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:26 smithi059 ceph-mon[97450]: pgmap v6035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:46:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:26 smithi055 ceph-mon[97033]: pgmap v6035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:46:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:27.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:27.447535+0000) 2022-09-08T00:46:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:27.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:27.374868+0000) 2022-09-08T00:46:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:28.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:28.447801+0000) 2022-09-08T00:46:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:28.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:28.375027+0000) 2022-09-08T00:46:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:28 smithi059 ceph-mon[97450]: pgmap v6036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:46:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:28 smithi055 ceph-mon[97033]: pgmap v6036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:46:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:29.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:29.448037+0000) 2022-09-08T00:46:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:29.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:29.375202+0000) 2022-09-08T00:46:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:30.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:30.448295+0000) 2022-09-08T00:46:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:30.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:30.375342+0000) 2022-09-08T00:46:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:30 smithi059 ceph-mon[97450]: pgmap v6037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:30 smithi055 ceph-mon[97033]: pgmap v6037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:31.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:31.448564+0000) 2022-09-08T00:46:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:31.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:31.375568+0000) 2022-09-08T00:46:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:32.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:32.448778+0000) 2022-09-08T00:46:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:32.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:32.375785+0000) 2022-09-08T00:46:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:32 smithi059 ceph-mon[97450]: pgmap v6038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:46:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:32 smithi055 ceph-mon[97033]: pgmap v6038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:46:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:33.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:33.449028+0000) 2022-09-08T00:46:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:33.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:33.375943+0000) 2022-09-08T00:46:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:34.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:34.449263+0000) 2022-09-08T00:46:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:34.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:34.376190+0000) 2022-09-08T00:46:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:34 smithi059 ceph-mon[97450]: pgmap v6039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:34 smithi055 ceph-mon[97033]: pgmap v6039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:35.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:35.449512+0000) 2022-09-08T00:46:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:35.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:35.376419+0000) 2022-09-08T00:46:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:36.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:36.376576+0000) 2022-09-08T00:46:36.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:46:36.581+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:46:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:36.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:36.449665+0000) 2022-09-08T00:46:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:36 smithi059 ceph-mon[97450]: pgmap v6040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:46:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:36 smithi055 ceph-mon[97033]: pgmap v6040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:46:37.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:37.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:37.376793+0000) 2022-09-08T00:46:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:37.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:37.449789+0000) 2022-09-08T00:46:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:38.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:38.450017+0000) 2022-09-08T00:46:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:38.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:38.377072+0000) 2022-09-08T00:46:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:38 smithi059 ceph-mon[97450]: pgmap v6041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:38 smithi055 ceph-mon[97033]: pgmap v6041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:39.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:39.377306+0000) 2022-09-08T00:46:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:39.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:39.450223+0000) 2022-09-08T00:46:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:40.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:40.377539+0000) 2022-09-08T00:46:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:40.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:40.450479+0000) 2022-09-08T00:46:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:40 smithi059 ceph-mon[97450]: pgmap v6042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:40 smithi055 ceph-mon[97033]: pgmap v6042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:41.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:41.450761+0000) 2022-09-08T00:46:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:41.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:41.377759+0000) 2022-09-08T00:46:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:42.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:42.377995+0000) 2022-09-08T00:46:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:42.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:42.451033+0000) 2022-09-08T00:46:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:42 smithi055 ceph-mon[97033]: pgmap v6043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:46:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:42 smithi059 ceph-mon[97450]: pgmap v6043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:46:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:43.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:43.451194+0000) 2022-09-08T00:46:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:43.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:43.378170+0000) 2022-09-08T00:46:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:44.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:44.451406+0000) 2022-09-08T00:46:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:44.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:44.378327+0000) 2022-09-08T00:46:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:44 smithi055 ceph-mon[97033]: pgmap v6044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:44 smithi059 ceph-mon[97450]: pgmap v6044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:45.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:45.451612+0000) 2022-09-08T00:46:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:45.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:45.378527+0000) 2022-09-08T00:46:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:46.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:46.451821+0000) 2022-09-08T00:46:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:46.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:46.378769+0000) 2022-09-08T00:46:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:46 smithi055 ceph-mon[97033]: pgmap v6045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:46:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:46 smithi059 ceph-mon[97450]: pgmap v6045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:46:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:47.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:47.451960+0000) 2022-09-08T00:46:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:47.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:47.379025+0000) 2022-09-08T00:46:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:48.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:48.452213+0000) 2022-09-08T00:46:48.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:48.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:48.379185+0000) 2022-09-08T00:46:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:48 smithi055 ceph-mon[97033]: pgmap v6046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:48 smithi059 ceph-mon[97450]: pgmap v6046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:49.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:49.452482+0000) 2022-09-08T00:46:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:49.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:49.379405+0000) 2022-09-08T00:46:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:46:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:46:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:46:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:46:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:46:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:46:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:50.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:50.452706+0000) 2022-09-08T00:46:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:50.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:50.379579+0000) 2022-09-08T00:46:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:50 smithi055 ceph-mon[97033]: pgmap v6047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:50.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:46:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:50 smithi059 ceph-mon[97450]: pgmap v6047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:46:51.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:46:51.590+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:46:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:51.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:51.452911+0000) 2022-09-08T00:46:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:51.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:51.379747+0000) 2022-09-08T00:46:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:52.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:52.453173+0000) 2022-09-08T00:46:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:52.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:52.379960+0000) 2022-09-08T00:46:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:52 smithi055 ceph-mon[97033]: pgmap v6048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:46:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:52 smithi059 ceph-mon[97450]: pgmap v6048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:46:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:53.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:53.453454+0000) 2022-09-08T00:46:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:53.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:53.380195+0000) 2022-09-08T00:46:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:54.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:54.453662+0000) 2022-09-08T00:46:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:54.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:54.380428+0000) 2022-09-08T00:46:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:54 smithi055 ceph-mon[97033]: pgmap v6049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:54 smithi059 ceph-mon[97450]: pgmap v6049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:55.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:55.453882+0000) 2022-09-08T00:46:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:55.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:55.380693+0000) 2022-09-08T00:46:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:56.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:56.454103+0000) 2022-09-08T00:46:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:56.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:56.380846+0000) 2022-09-08T00:46:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:56 smithi055 ceph-mon[97033]: pgmap v6050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:46:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:56 smithi059 ceph-mon[97450]: pgmap v6050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:46:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:57.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:57.454354+0000) 2022-09-08T00:46:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:57.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:57.381097+0000) 2022-09-08T00:46:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:58.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:58.381260+0000) 2022-09-08T00:46:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:58.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:58.454541+0000) 2022-09-08T00:46:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:46:58 smithi055 ceph-mon[97033]: pgmap v6051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:46:58 smithi059 ceph-mon[97450]: pgmap v6051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:46:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:46:59.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:59.454732+0000) 2022-09-08T00:46:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:46:59.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:46:59.381447+0000) 2022-09-08T00:47:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:00.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:00.454910+0000) 2022-09-08T00:47:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:00.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:00.381723+0000) 2022-09-08T00:47:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:00 smithi055 ceph-mon[97033]: pgmap v6052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:00 smithi059 ceph-mon[97450]: pgmap v6052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:01.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:01.455131+0000) 2022-09-08T00:47:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:01.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:01.381860+0000) 2022-09-08T00:47:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:02.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:02.455425+0000) 2022-09-08T00:47:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:02.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:02.382052+0000) 2022-09-08T00:47:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:02 smithi055 ceph-mon[97033]: pgmap v6053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:47:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:02 smithi059 ceph-mon[97450]: pgmap v6053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:47:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:03.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:03.455699+0000) 2022-09-08T00:47:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:03.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:03.382270+0000) 2022-09-08T00:47:04.459 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:04.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:04.382490+0000) 2022-09-08T00:47:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:04.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:04.455927+0000) 2022-09-08T00:47:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:04 smithi055 ceph-mon[97033]: pgmap v6054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:04 smithi059 ceph-mon[97450]: pgmap v6054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:05.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:05.456164+0000) 2022-09-08T00:47:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:05.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:05.382706+0000) 2022-09-08T00:47:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:47:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:47:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:47:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:47:06.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:47:06.606+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:47:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:06.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:06.456312+0000) 2022-09-08T00:47:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:06.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:06.382912+0000) 2022-09-08T00:47:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:06 smithi055 ceph-mon[97033]: pgmap v6055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:47:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:06 smithi059 ceph-mon[97450]: pgmap v6055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:47:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:07.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:07.456477+0000) 2022-09-08T00:47:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:07.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:07.383160+0000) 2022-09-08T00:47:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:08.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:08.456731+0000) 2022-09-08T00:47:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:08.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:08.383382+0000) 2022-09-08T00:47:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:08 smithi055 ceph-mon[97033]: pgmap v6056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:47:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:08 smithi059 ceph-mon[97450]: pgmap v6056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:47:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:09.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:09.456977+0000) 2022-09-08T00:47:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:09.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:09.383589+0000) 2022-09-08T00:47:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:09 smithi055 ceph-mon[97033]: osdmap e463: 12 total, 11 up, 11 in 2022-09-08T00:47:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:09 smithi059 ceph-mon[97450]: osdmap e463: 12 total, 11 up, 11 in 2022-09-08T00:47:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:10.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:10.457152+0000) 2022-09-08T00:47:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:10.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:10.383743+0000) 2022-09-08T00:47:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:10 smithi055 ceph-mon[97033]: pgmap v6058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:47:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:10 smithi059 ceph-mon[97450]: pgmap v6058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:47:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:11.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:11.457325+0000) 2022-09-08T00:47:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:11.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:11.383991+0000) 2022-09-08T00:47:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:12.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:12.457548+0000) 2022-09-08T00:47:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:12.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:12.384191+0000) 2022-09-08T00:47:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:12 smithi055 ceph-mon[97033]: pgmap v6059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:47:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:12 smithi059 ceph-mon[97450]: pgmap v6059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:47:13.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:13.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:13.384324+0000) 2022-09-08T00:47:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:13.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:13.457779+0000) 2022-09-08T00:47:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:14.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:14.458006+0000) 2022-09-08T00:47:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:14.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:14.384519+0000) 2022-09-08T00:47:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:14 smithi055 ceph-mon[97033]: pgmap v6060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:47:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:14 smithi059 ceph-mon[97450]: pgmap v6060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:47:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:15.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:15.458191+0000) 2022-09-08T00:47:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:15.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:15.384777+0000) 2022-09-08T00:47:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:16.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:16.384988+0000) 2022-09-08T00:47:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:16.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:16.458434+0000) 2022-09-08T00:47:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:16 smithi055 ceph-mon[97033]: pgmap v6061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:47:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:16 smithi055 ceph-mon[97033]: osdmap e464: 12 total, 11 up, 11 in 2022-09-08T00:47:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:16 smithi059 ceph-mon[97450]: pgmap v6061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:47:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:16 smithi059 ceph-mon[97450]: osdmap e464: 12 total, 11 up, 11 in 2022-09-08T00:47:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:17.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:17.385171+0000) 2022-09-08T00:47:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:17.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:17.458714+0000) 2022-09-08T00:47:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:18.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:18.458950+0000) 2022-09-08T00:47:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:18.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:18.385444+0000) 2022-09-08T00:47:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:18 smithi055 ceph-mon[97033]: pgmap v6063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-08T00:47:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:18 smithi059 ceph-mon[97450]: pgmap v6063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-08T00:47:19.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:19.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:19.385717+0000) 2022-09-08T00:47:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:19.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:19.459125+0000) 2022-09-08T00:47:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:20.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:20.459406+0000) 2022-09-08T00:47:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:20.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:20.385947+0000) 2022-09-08T00:47:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:20 smithi055 ceph-mon[97033]: pgmap v6064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:47:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:20 smithi059 ceph-mon[97450]: pgmap v6064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:47:21.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:47:21.614+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:47:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:21.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:21.459550+0000) 2022-09-08T00:47:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:21.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:21.386144+0000) 2022-09-08T00:47:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:22.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:22.459713+0000) 2022-09-08T00:47:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:22.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:22.386389+0000) 2022-09-08T00:47:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:22 smithi055 ceph-mon[97033]: pgmap v6065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:47:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:22 smithi059 ceph-mon[97450]: pgmap v6065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:47:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:23.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:23.459885+0000) 2022-09-08T00:47:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:23.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:23.386615+0000) 2022-09-08T00:47:24.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:24.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:24.386842+0000) 2022-09-08T00:47:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:24.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:24.460112+0000) 2022-09-08T00:47:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:24 smithi055 ceph-mon[97033]: pgmap v6066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:47:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:24 smithi059 ceph-mon[97450]: pgmap v6066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:47:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:25.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:25.460354+0000) 2022-09-08T00:47:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:25.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:25.387074+0000) 2022-09-08T00:47:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:26.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:26.460589+0000) 2022-09-08T00:47:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:26.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:26.387310+0000) 2022-09-08T00:47:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:26 smithi055 ceph-mon[97033]: pgmap v6067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:47:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:26 smithi059 ceph-mon[97450]: pgmap v6067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:47:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:27.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:27.460818+0000) 2022-09-08T00:47:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:27.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:27.387489+0000) 2022-09-08T00:47:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:28.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:28.461041+0000) 2022-09-08T00:47:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:28.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:28.387725+0000) 2022-09-08T00:47:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:28 smithi055 ceph-mon[97033]: pgmap v6068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:47:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:28 smithi059 ceph-mon[97450]: pgmap v6068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:47:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:29.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:29.461284+0000) 2022-09-08T00:47:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:29.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:29.387971+0000) 2022-09-08T00:47:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:30.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:30.461456+0000) 2022-09-08T00:47:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:30.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:30.388127+0000) 2022-09-08T00:47:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:30 smithi055 ceph-mon[97033]: pgmap v6069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:30 smithi059 ceph-mon[97450]: pgmap v6069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:31.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:31.461678+0000) 2022-09-08T00:47:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:31.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:31.388338+0000) 2022-09-08T00:47:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:32.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:32.461887+0000) 2022-09-08T00:47:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:32.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:32.388591+0000) 2022-09-08T00:47:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:32 smithi055 ceph-mon[97033]: pgmap v6070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:47:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:32 smithi059 ceph-mon[97450]: pgmap v6070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:47:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:33.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:33.462062+0000) 2022-09-08T00:47:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:33.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:33.388839+0000) 2022-09-08T00:47:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:34.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:34.389026+0000) 2022-09-08T00:47:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:34.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:34.462247+0000) 2022-09-08T00:47:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:34 smithi055 ceph-mon[97033]: pgmap v6071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:34 smithi059 ceph-mon[97450]: pgmap v6071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:35.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:35.389308+0000) 2022-09-08T00:47:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:35.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:35.462456+0000) 2022-09-08T00:47:36.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:47:36.623+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:47:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:36.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:36.462676+0000) 2022-09-08T00:47:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:36.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:36.389531+0000) 2022-09-08T00:47:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:36 smithi055 ceph-mon[97033]: pgmap v6072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:47:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:36 smithi059 ceph-mon[97450]: pgmap v6072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:47:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:37.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:37.462936+0000) 2022-09-08T00:47:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:37.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:37.389776+0000) 2022-09-08T00:47:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:38.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:38.463214+0000) 2022-09-08T00:47:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:38.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:38.390023+0000) 2022-09-08T00:47:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:38 smithi055 ceph-mon[97033]: pgmap v6073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:38 smithi059 ceph-mon[97450]: pgmap v6073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:39.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:39.463480+0000) 2022-09-08T00:47:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:39.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:39.390229+0000) 2022-09-08T00:47:40.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:40.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:40.463739+0000) 2022-09-08T00:47:40.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:40.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:40.390459+0000) 2022-09-08T00:47:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:40 smithi055 ceph-mon[97033]: pgmap v6074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:40 smithi059 ceph-mon[97450]: pgmap v6074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:41.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:41.463960+0000) 2022-09-08T00:47:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:41.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:41.390675+0000) 2022-09-08T00:47:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:42.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:42.464165+0000) 2022-09-08T00:47:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:42.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:42.390907+0000) 2022-09-08T00:47:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:42 smithi055 ceph-mon[97033]: pgmap v6075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:47:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:42 smithi059 ceph-mon[97450]: pgmap v6075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:47:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:43.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:43.464412+0000) 2022-09-08T00:47:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:43.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:43.391078+0000) 2022-09-08T00:47:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:44.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:44.464611+0000) 2022-09-08T00:47:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:44.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:44.391274+0000) 2022-09-08T00:47:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:44 smithi055 ceph-mon[97033]: pgmap v6076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:44 smithi059 ceph-mon[97450]: pgmap v6076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:45.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:45.464827+0000) 2022-09-08T00:47:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:45.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:45.391552+0000) 2022-09-08T00:47:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:46.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:46.465028+0000) 2022-09-08T00:47:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:46.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:46.391815+0000) 2022-09-08T00:47:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:46 smithi055 ceph-mon[97033]: pgmap v6077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:47:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:46 smithi059 ceph-mon[97450]: pgmap v6077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:47:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:47.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:47.465196+0000) 2022-09-08T00:47:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:47.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:47.392037+0000) 2022-09-08T00:47:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:48.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:48.465402+0000) 2022-09-08T00:47:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:48.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:48.392309+0000) 2022-09-08T00:47:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:48 smithi055 ceph-mon[97033]: pgmap v6078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:48 smithi059 ceph-mon[97450]: pgmap v6078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:49.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:49.465565+0000) 2022-09-08T00:47:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:49.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:49.392545+0000) 2022-09-08T00:47:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:50.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:50.465707+0000) 2022-09-08T00:47:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:50.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:50.392746+0000) 2022-09-08T00:47:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:50 smithi055 ceph-mon[97033]: pgmap v6079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:50.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:47:50.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:47:50.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:47:50.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:47:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:50 smithi059 ceph-mon[97450]: pgmap v6079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:47:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:47:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:47:51.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:47:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:51.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:51.465858+0000) 2022-09-08T00:47:51.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:47:51.638+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:47:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:51.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:51.392960+0000) 2022-09-08T00:47:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:52.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:52.466042+0000) 2022-09-08T00:47:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:52.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:52.393167+0000) 2022-09-08T00:47:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:52 smithi055 ceph-mon[97033]: pgmap v6080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:47:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:52 smithi059 ceph-mon[97450]: pgmap v6080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:47:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:53.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:53.466191+0000) 2022-09-08T00:47:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:53.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:53.393420+0000) 2022-09-08T00:47:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:54.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:54.466373+0000) 2022-09-08T00:47:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:54.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:54.393650+0000) 2022-09-08T00:47:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:54 smithi055 ceph-mon[97033]: pgmap v6081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:54 smithi059 ceph-mon[97450]: pgmap v6081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:55.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:55.466531+0000) 2022-09-08T00:47:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:55.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:55.393900+0000) 2022-09-08T00:47:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:56.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:56.466727+0000) 2022-09-08T00:47:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:56.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:56.394149+0000) 2022-09-08T00:47:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:56 smithi055 ceph-mon[97033]: pgmap v6082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:47:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:56 smithi059 ceph-mon[97450]: pgmap v6082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:47:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:57.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:57.466859+0000) 2022-09-08T00:47:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:57.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:57.394473+0000) 2022-09-08T00:47:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:58.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:58.467069+0000) 2022-09-08T00:47:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:58.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:58.394777+0000) 2022-09-08T00:47:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:47:58 smithi055 ceph-mon[97033]: pgmap v6083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:47:58 smithi059 ceph-mon[97450]: pgmap v6083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:47:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:47:59.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:59.394980+0000) 2022-09-08T00:47:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:47:59.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:47:59.467265+0000) 2022-09-08T00:48:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:00.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:00.467490+0000) 2022-09-08T00:48:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:00.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:00.395219+0000) 2022-09-08T00:48:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:00 smithi055 ceph-mon[97033]: pgmap v6084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:00 smithi059 ceph-mon[97450]: pgmap v6084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:01.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:01.467725+0000) 2022-09-08T00:48:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:01.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:01.395412+0000) 2022-09-08T00:48:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:02.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:02.467884+0000) 2022-09-08T00:48:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:02.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:02.395625+0000) 2022-09-08T00:48:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:02 smithi055 ceph-mon[97033]: pgmap v6085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:48:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:02 smithi059 ceph-mon[97450]: pgmap v6085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:48:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:03.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:03.468037+0000) 2022-09-08T00:48:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:03.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:03.395917+0000) 2022-09-08T00:48:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:04.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:04.468218+0000) 2022-09-08T00:48:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:04.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:04.396169+0000) 2022-09-08T00:48:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:04 smithi055 ceph-mon[97033]: pgmap v6086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:04 smithi059 ceph-mon[97450]: pgmap v6086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:05.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:05.468364+0000) 2022-09-08T00:48:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:05.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:05.396350+0000) 2022-09-08T00:48:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:48:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:48:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:48:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:48:06.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:06.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:06.468509+0000) 2022-09-08T00:48:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:06.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:06.396583+0000) 2022-09-08T00:48:06.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:48:06.646+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:48:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:06 smithi055 ceph-mon[97033]: pgmap v6087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:48:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:06 smithi059 ceph-mon[97450]: pgmap v6087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:48:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:07.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:07.468754+0000) 2022-09-08T00:48:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:07.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:07.396798+0000) 2022-09-08T00:48:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:08.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:08.468929+0000) 2022-09-08T00:48:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:08.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:08.397038+0000) 2022-09-08T00:48:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:08 smithi055 ceph-mon[97033]: pgmap v6088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:48:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:08 smithi059 ceph-mon[97450]: pgmap v6088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:48:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:09.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:09.469137+0000) 2022-09-08T00:48:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:09.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:09.397241+0000) 2022-09-08T00:48:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:09 smithi055 ceph-mon[97033]: osdmap e465: 12 total, 11 up, 11 in 2022-09-08T00:48:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:09 smithi059 ceph-mon[97450]: osdmap e465: 12 total, 11 up, 11 in 2022-09-08T00:48:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:10.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:10.469363+0000) 2022-09-08T00:48:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:10.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:10.397477+0000) 2022-09-08T00:48:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:10 smithi055 ceph-mon[97033]: pgmap v6090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:48:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:10 smithi059 ceph-mon[97450]: pgmap v6090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:48:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:11.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:11.469537+0000) 2022-09-08T00:48:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:11.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:11.397747+0000) 2022-09-08T00:48:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:12.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:12.469710+0000) 2022-09-08T00:48:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:12.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:12.397932+0000) 2022-09-08T00:48:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:12 smithi055 ceph-mon[97033]: pgmap v6091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:48:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:12 smithi059 ceph-mon[97450]: pgmap v6091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:48:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:13.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:13.469880+0000) 2022-09-08T00:48:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:13.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:13.398148+0000) 2022-09-08T00:48:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:14.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:14.470098+0000) 2022-09-08T00:48:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:14.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:14.398362+0000) 2022-09-08T00:48:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:14 smithi059 ceph-mon[97450]: pgmap v6092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:48:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:14 smithi055 ceph-mon[97033]: pgmap v6092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:48:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:15.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:15.470260+0000) 2022-09-08T00:48:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:15.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:15.398576+0000) 2022-09-08T00:48:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:16.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:16.470414+0000) 2022-09-08T00:48:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:16.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:16.398802+0000) 2022-09-08T00:48:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:16 smithi059 ceph-mon[97450]: pgmap v6093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:48:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:16 smithi059 ceph-mon[97450]: osdmap e466: 12 total, 11 up, 11 in 2022-09-08T00:48:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:16 smithi055 ceph-mon[97033]: pgmap v6093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:48:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:16 smithi055 ceph-mon[97033]: osdmap e466: 12 total, 11 up, 11 in 2022-09-08T00:48:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:17.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:17.470675+0000) 2022-09-08T00:48:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:17.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:17.399021+0000) 2022-09-08T00:48:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:18.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:18.470933+0000) 2022-09-08T00:48:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:18.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:18.399202+0000) 2022-09-08T00:48:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:18 smithi059 ceph-mon[97450]: pgmap v6095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T00:48:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:18 smithi055 ceph-mon[97033]: pgmap v6095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T00:48:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:19.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:19.471175+0000) 2022-09-08T00:48:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:19.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:19.399448+0000) 2022-09-08T00:48:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:20.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:20.471400+0000) 2022-09-08T00:48:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:20.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:20.399721+0000) 2022-09-08T00:48:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:20 smithi059 ceph-mon[97450]: pgmap v6096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:48:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:20 smithi055 ceph-mon[97033]: pgmap v6096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:48:21.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:21.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:21.399960+0000) 2022-09-08T00:48:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:21.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:21.471559+0000) 2022-09-08T00:48:22.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:48:21.662+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:48:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:22.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:22.471755+0000) 2022-09-08T00:48:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:22.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:22.400175+0000) 2022-09-08T00:48:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:22 smithi059 ceph-mon[97450]: pgmap v6097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:48:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:22 smithi055 ceph-mon[97033]: pgmap v6097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:48:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:23.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:23.400399+0000) 2022-09-08T00:48:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:23.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:23.471907+0000) 2022-09-08T00:48:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:24.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:24.400711+0000) 2022-09-08T00:48:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:24.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:24.472120+0000) 2022-09-08T00:48:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:24 smithi059 ceph-mon[97450]: pgmap v6098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:48:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:24 smithi055 ceph-mon[97033]: pgmap v6098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:48:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:25.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:25.400974+0000) 2022-09-08T00:48:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:25.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:25.472353+0000) 2022-09-08T00:48:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:26.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:26.401263+0000) 2022-09-08T00:48:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:26.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:26.472617+0000) 2022-09-08T00:48:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:26 smithi059 ceph-mon[97450]: pgmap v6099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:48:27.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:26 smithi055 ceph-mon[97033]: pgmap v6099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:48:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:27.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:27.472894+0000) 2022-09-08T00:48:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:27.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:27.401513+0000) 2022-09-08T00:48:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:28.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:28.473148+0000) 2022-09-08T00:48:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:28.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:28.401701+0000) 2022-09-08T00:48:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:28 smithi059 ceph-mon[97450]: pgmap v6100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:48:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:28 smithi055 ceph-mon[97033]: pgmap v6100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:48:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:29.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:29.473381+0000) 2022-09-08T00:48:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:29.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:29.401875+0000) 2022-09-08T00:48:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:30.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:30.473579+0000) 2022-09-08T00:48:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:30.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:30.402075+0000) 2022-09-08T00:48:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:30 smithi059 ceph-mon[97450]: pgmap v6101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:30 smithi055 ceph-mon[97033]: pgmap v6101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:31.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:31.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:31.473746+0000) 2022-09-08T00:48:31.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:31.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:31.402288+0000) 2022-09-08T00:48:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:32.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:32.402548+0000) 2022-09-08T00:48:32.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:32.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:32.474013+0000) 2022-09-08T00:48:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:32 smithi059 ceph-mon[97450]: pgmap v6102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:48:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:32 smithi055 ceph-mon[97033]: pgmap v6102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:48:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:33.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:33.474221+0000) 2022-09-08T00:48:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:33.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:33.402788+0000) 2022-09-08T00:48:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:34.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:34.474435+0000) 2022-09-08T00:48:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:34.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:34.403030+0000) 2022-09-08T00:48:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:34 smithi059 ceph-mon[97450]: pgmap v6103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:34 smithi055 ceph-mon[97033]: pgmap v6103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:35.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:35.474576+0000) 2022-09-08T00:48:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:35.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:35.403286+0000) 2022-09-08T00:48:36.672 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:36.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:36.474843+0000) 2022-09-08T00:48:36.672 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:36.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:36.403525+0000) 2022-09-08T00:48:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:36 smithi059 ceph-mon[97450]: pgmap v6104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:48:37.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:48:36.671+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:48:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:36 smithi055 ceph-mon[97033]: pgmap v6104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:48:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:37.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:37.475037+0000) 2022-09-08T00:48:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:37.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:37.403782+0000) 2022-09-08T00:48:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:38.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:38.475300+0000) 2022-09-08T00:48:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:38.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:38.403935+0000) 2022-09-08T00:48:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:38 smithi059 ceph-mon[97450]: pgmap v6105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:38 smithi055 ceph-mon[97033]: pgmap v6105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:39.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:39.475599+0000) 2022-09-08T00:48:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:39.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:39.404158+0000) 2022-09-08T00:48:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:40.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:40.475818+0000) 2022-09-08T00:48:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:40.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:40.404373+0000) 2022-09-08T00:48:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:40 smithi059 ceph-mon[97450]: pgmap v6106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:40 smithi055 ceph-mon[97033]: pgmap v6106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:41.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:41.404545+0000) 2022-09-08T00:48:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:41.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:41.476038+0000) 2022-09-08T00:48:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:42.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:42.476227+0000) 2022-09-08T00:48:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:42.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:42.404747+0000) 2022-09-08T00:48:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:42 smithi059 ceph-mon[97450]: pgmap v6107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:48:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:42 smithi055 ceph-mon[97033]: pgmap v6107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:48:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:43.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:43.476449+0000) 2022-09-08T00:48:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:43.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:43.404943+0000) 2022-09-08T00:48:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:44.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:44.476755+0000) 2022-09-08T00:48:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:44.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:44.405120+0000) 2022-09-08T00:48:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:44 smithi059 ceph-mon[97450]: pgmap v6108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:44 smithi055 ceph-mon[97033]: pgmap v6108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:45.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:45.405358+0000) 2022-09-08T00:48:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:45.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:45.477030+0000) 2022-09-08T00:48:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:46.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:46.477268+0000) 2022-09-08T00:48:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:46.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:46.405674+0000) 2022-09-08T00:48:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:46 smithi055 ceph-mon[97033]: pgmap v6109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:48:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:46 smithi059 ceph-mon[97450]: pgmap v6109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:48:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:47.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:47.405950+0000) 2022-09-08T00:48:47.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:47.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:47.477455+0000) 2022-09-08T00:48:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:48.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:48.477731+0000) 2022-09-08T00:48:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:48.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:48.406229+0000) 2022-09-08T00:48:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:48 smithi055 ceph-mon[97033]: pgmap v6110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:48 smithi059 ceph-mon[97450]: pgmap v6110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:49.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:49.478018+0000) 2022-09-08T00:48:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:49.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:49.406457+0000) 2022-09-08T00:48:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:50.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:50.478242+0000) 2022-09-08T00:48:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:50.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:50.406750+0000) 2022-09-08T00:48:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:50 smithi055 ceph-mon[97033]: pgmap v6111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:48:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:48:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:48:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:50 smithi059 ceph-mon[97450]: pgmap v6111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:48:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:48:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:48:51.679 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:51.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:51.478476+0000) 2022-09-08T00:48:51.680 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:51.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:51.406971+0000) 2022-09-08T00:48:52.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:48:51.678+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:48:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:52 smithi059 ceph-mon[97450]: pgmap v6112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:48:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:48:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:52 smithi055 ceph-mon[97033]: pgmap v6112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:48:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:48:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:52.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:52.478739+0000) 2022-09-08T00:48:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:52.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:52.407232+0000) 2022-09-08T00:48:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:53.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:53.478933+0000) 2022-09-08T00:48:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:53.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:53.407529+0000) 2022-09-08T00:48:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:54 smithi059 ceph-mon[97450]: pgmap v6113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:54 smithi055 ceph-mon[97033]: pgmap v6113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:54.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:54.479067+0000) 2022-09-08T00:48:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:54.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:54.407760+0000) 2022-09-08T00:48:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:55.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:55.479272+0000) 2022-09-08T00:48:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:55.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:55.407957+0000) 2022-09-08T00:48:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:56 smithi055 ceph-mon[97033]: pgmap v6114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:48:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:56 smithi059 ceph-mon[97450]: pgmap v6114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:48:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:56.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:56.479522+0000) 2022-09-08T00:48:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:56.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:56.408163+0000) 2022-09-08T00:48:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:57.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:57.408420+0000) 2022-09-08T00:48:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:57.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:57.479748+0000) 2022-09-08T00:48:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:48:58 smithi055 ceph-mon[97033]: pgmap v6115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:48:58 smithi059 ceph-mon[97450]: pgmap v6115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:48:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:58.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:58.479965+0000) 2022-09-08T00:48:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:58.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:58.408671+0000) 2022-09-08T00:48:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:48:59.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:59.480170+0000) 2022-09-08T00:48:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:48:59.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:48:59.408873+0000) 2022-09-08T00:49:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:00 smithi055 ceph-mon[97033]: pgmap v6116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:00 smithi059 ceph-mon[97450]: pgmap v6116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:00.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:00.480344+0000) 2022-09-08T00:49:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:00.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:00.408999+0000) 2022-09-08T00:49:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:01.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:01.480546+0000) 2022-09-08T00:49:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:01.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:01.409221+0000) 2022-09-08T00:49:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:02 smithi055 ceph-mon[97033]: pgmap v6117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:49:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:02 smithi059 ceph-mon[97450]: pgmap v6117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:49:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:02.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:02.480815+0000) 2022-09-08T00:49:02.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:02.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:02.409444+0000) 2022-09-08T00:49:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:03.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:03.409661+0000) 2022-09-08T00:49:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:03.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:03.481069+0000) 2022-09-08T00:49:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:04 smithi055 ceph-mon[97033]: pgmap v6118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:04 smithi059 ceph-mon[97450]: pgmap v6118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:04.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:04.481327+0000) 2022-09-08T00:49:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:04.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:04.409801+0000) 2022-09-08T00:49:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:49:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:49:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:49:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:49:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:05.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:05.481511+0000) 2022-09-08T00:49:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:05.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:05.410018+0000) 2022-09-08T00:49:06.681 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:06.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:06.481715+0000) 2022-09-08T00:49:06.682 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:06.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:06.410269+0000) 2022-09-08T00:49:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:06 smithi055 ceph-mon[97033]: pgmap v6119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:49:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:06 smithi059 ceph-mon[97450]: pgmap v6119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:49:07.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:49:06.680+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:49:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:07.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:07.481925+0000) 2022-09-08T00:49:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:07.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:07.410536+0000) 2022-09-08T00:49:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:08 smithi055 ceph-mon[97033]: pgmap v6120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:49:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:08 smithi059 ceph-mon[97450]: pgmap v6120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:49:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:08.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:08.482187+0000) 2022-09-08T00:49:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:08.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:08.410767+0000) 2022-09-08T00:49:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:09.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:09.482482+0000) 2022-09-08T00:49:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:09.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:09.410940+0000) 2022-09-08T00:49:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:10 smithi055 ceph-mon[97033]: pgmap v6121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:49:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:10 smithi055 ceph-mon[97033]: osdmap e467: 12 total, 11 up, 11 in 2022-09-08T00:49:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:10 smithi059 ceph-mon[97450]: pgmap v6121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:49:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:10 smithi059 ceph-mon[97450]: osdmap e467: 12 total, 11 up, 11 in 2022-09-08T00:49:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:10.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:10.482667+0000) 2022-09-08T00:49:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:10.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:10.411106+0000) 2022-09-08T00:49:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:11.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:11.482848+0000) 2022-09-08T00:49:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:11.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:11.411336+0000) 2022-09-08T00:49:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:12 smithi055 ceph-mon[97033]: pgmap v6123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:49:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:12 smithi059 ceph-mon[97450]: pgmap v6123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:49:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:12.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:12.483059+0000) 2022-09-08T00:49:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:12.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:12.411554+0000) 2022-09-08T00:49:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:13.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:13.483247+0000) 2022-09-08T00:49:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:13.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:13.411804+0000) 2022-09-08T00:49:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:14 smithi055 ceph-mon[97033]: pgmap v6124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:49:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:14 smithi059 ceph-mon[97450]: pgmap v6124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:49:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:14.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:14.483498+0000) 2022-09-08T00:49:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:14.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:14.411978+0000) 2022-09-08T00:49:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:15.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:15.483765+0000) 2022-09-08T00:49:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:15.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:15.412225+0000) 2022-09-08T00:49:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:16 smithi055 ceph-mon[97033]: pgmap v6125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:49:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:16 smithi055 ceph-mon[97033]: osdmap e468: 12 total, 11 up, 11 in 2022-09-08T00:49:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:16 smithi059 ceph-mon[97450]: pgmap v6125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:49:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:16 smithi059 ceph-mon[97450]: osdmap e468: 12 total, 11 up, 11 in 2022-09-08T00:49:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:16.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:16.484002+0000) 2022-09-08T00:49:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:16.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:16.412536+0000) 2022-09-08T00:49:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:17.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:17.484269+0000) 2022-09-08T00:49:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:17.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:17.412707+0000) 2022-09-08T00:49:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:18 smithi055 ceph-mon[97033]: pgmap v6127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:49:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:18 smithi059 ceph-mon[97450]: pgmap v6127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:49:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:18.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:18.484534+0000) 2022-09-08T00:49:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:18.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:18.412862+0000) 2022-09-08T00:49:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:19.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:19.484795+0000) 2022-09-08T00:49:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:19.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:19.413006+0000) 2022-09-08T00:49:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:20 smithi055 ceph-mon[97033]: pgmap v6128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 210 B/s wr, 6 op/s 2022-09-08T00:49:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:20 smithi059 ceph-mon[97450]: pgmap v6128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 210 B/s wr, 6 op/s 2022-09-08T00:49:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:20.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:20.485020+0000) 2022-09-08T00:49:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:20.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:20.413251+0000) 2022-09-08T00:49:21.692 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:21.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:21.485289+0000) 2022-09-08T00:49:21.692 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:21.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:21.413457+0000) 2022-09-08T00:49:22.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:49:21.691+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:49:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:22 smithi055 ceph-mon[97033]: pgmap v6129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:49:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:22 smithi059 ceph-mon[97450]: pgmap v6129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:49:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:22.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:22.485550+0000) 2022-09-08T00:49:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:22.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:22.413631+0000) 2022-09-08T00:49:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:23.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:23.485805+0000) 2022-09-08T00:49:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:23.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:23.413918+0000) 2022-09-08T00:49:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:24 smithi055 ceph-mon[97033]: pgmap v6130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:49:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:24 smithi059 ceph-mon[97450]: pgmap v6130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:49:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:24.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:24.485987+0000) 2022-09-08T00:49:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:24.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:24.414141+0000) 2022-09-08T00:49:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:25.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:25.486255+0000) 2022-09-08T00:49:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:25.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:25.414357+0000) 2022-09-08T00:49:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:26 smithi055 ceph-mon[97033]: pgmap v6131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:49:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:26 smithi059 ceph-mon[97450]: pgmap v6131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:49:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:26.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:26.486521+0000) 2022-09-08T00:49:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:26.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:26.414574+0000) 2022-09-08T00:49:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:27.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:27.414788+0000) 2022-09-08T00:49:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:27.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:27.486768+0000) 2022-09-08T00:49:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:28 smithi055 ceph-mon[97033]: pgmap v6132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:49:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:28 smithi059 ceph-mon[97450]: pgmap v6132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T00:49:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:28.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:28.415012+0000) 2022-09-08T00:49:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:28.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:28.487015+0000) 2022-09-08T00:49:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:29.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:29.487232+0000) 2022-09-08T00:49:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:29.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:29.415218+0000) 2022-09-08T00:49:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:30 smithi055 ceph-mon[97033]: pgmap v6133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:30 smithi059 ceph-mon[97450]: pgmap v6133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:30.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:30.487469+0000) 2022-09-08T00:49:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:30.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:30.415410+0000) 2022-09-08T00:49:31.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:31.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:31.487713+0000) 2022-09-08T00:49:31.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:31.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:31.415678+0000) 2022-09-08T00:49:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:32 smithi055 ceph-mon[97033]: pgmap v6134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:49:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:32 smithi059 ceph-mon[97450]: pgmap v6134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:49:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:32.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:32.487913+0000) 2022-09-08T00:49:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:32.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:32.415919+0000) 2022-09-08T00:49:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:33.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:33.488165+0000) 2022-09-08T00:49:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:33.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:33.416175+0000) 2022-09-08T00:49:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:34 smithi055 ceph-mon[97033]: pgmap v6135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:34 smithi059 ceph-mon[97450]: pgmap v6135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:34.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:34.488419+0000) 2022-09-08T00:49:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:34.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:34.416360+0000) 2022-09-08T00:49:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:35.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:35.488715+0000) 2022-09-08T00:49:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:35.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:35.416619+0000) 2022-09-08T00:49:36.708 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:36.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:36.488919+0000) 2022-09-08T00:49:36.708 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:36.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:36.416870+0000) 2022-09-08T00:49:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:36 smithi055 ceph-mon[97033]: pgmap v6136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:49:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:36 smithi059 ceph-mon[97450]: pgmap v6136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:49:37.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:49:36.706+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:49:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:37.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:37.489137+0000) 2022-09-08T00:49:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:37.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:37.417084+0000) 2022-09-08T00:49:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:38 smithi055 ceph-mon[97033]: pgmap v6137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:38 smithi059 ceph-mon[97450]: pgmap v6137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:38.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:38.489353+0000) 2022-09-08T00:49:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:38.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:38.417215+0000) 2022-09-08T00:49:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:39.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:39.489547+0000) 2022-09-08T00:49:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:39.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:39.417434+0000) 2022-09-08T00:49:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:40 smithi055 ceph-mon[97033]: pgmap v6138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:40 smithi059 ceph-mon[97450]: pgmap v6138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:40.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:40.489824+0000) 2022-09-08T00:49:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:40.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:40.417622+0000) 2022-09-08T00:49:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:41.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:41.490099+0000) 2022-09-08T00:49:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:41.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:41.417792+0000) 2022-09-08T00:49:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:42 smithi055 ceph-mon[97033]: pgmap v6139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:49:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:42 smithi059 ceph-mon[97450]: pgmap v6139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:49:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:42.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:42.490326+0000) 2022-09-08T00:49:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:42.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:42.417975+0000) 2022-09-08T00:49:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:43.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:43.490513+0000) 2022-09-08T00:49:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:43.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:43.418205+0000) 2022-09-08T00:49:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:44 smithi055 ceph-mon[97033]: pgmap v6140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:44 smithi059 ceph-mon[97450]: pgmap v6140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:44.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:44.490740+0000) 2022-09-08T00:49:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:44.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:44.418437+0000) 2022-09-08T00:49:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:45.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:45.490977+0000) 2022-09-08T00:49:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:45.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:45.418647+0000) 2022-09-08T00:49:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:46 smithi055 ceph-mon[97033]: pgmap v6141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:49:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:46 smithi059 ceph-mon[97450]: pgmap v6141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:49:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:46.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:46.491225+0000) 2022-09-08T00:49:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:46.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:46.418876+0000) 2022-09-08T00:49:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:47.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:47.419054+0000) 2022-09-08T00:49:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:47.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:47.491455+0000) 2022-09-08T00:49:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:48 smithi055 ceph-mon[97033]: pgmap v6142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:48 smithi059 ceph-mon[97450]: pgmap v6142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:48.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:48.491717+0000) 2022-09-08T00:49:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:48.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:48.419249+0000) 2022-09-08T00:49:49.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:49.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:49.419491+0000) 2022-09-08T00:49:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:49.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:49.491967+0000) 2022-09-08T00:49:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:50 smithi055 ceph-mon[97033]: pgmap v6143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:50 smithi059 ceph-mon[97450]: pgmap v6143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:50.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:50.492226+0000) 2022-09-08T00:49:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:50.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:50.419737+0000) 2022-09-08T00:49:51.718 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:51.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:51.492399+0000) 2022-09-08T00:49:51.719 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:51.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:51.419940+0000) 2022-09-08T00:49:52.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:49:51.715+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:49:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:52 smithi055 ceph-mon[97033]: pgmap v6144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:49:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:49:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:49:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:49:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:52 smithi059 ceph-mon[97450]: pgmap v6144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:49:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:49:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:49:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:49:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:52.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:52.492630+0000) 2022-09-08T00:49:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:52.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:52.420115+0000) 2022-09-08T00:49:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:53.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:53.492903+0000) 2022-09-08T00:49:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:53.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:53.420305+0000) 2022-09-08T00:49:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:54 smithi055 ceph-mon[97033]: pgmap v6145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:54 smithi059 ceph-mon[97450]: pgmap v6145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:54.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:54.493136+0000) 2022-09-08T00:49:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:54.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:54.420511+0000) 2022-09-08T00:49:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:55.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:55.493363+0000) 2022-09-08T00:49:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:55.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:55.420736+0000) 2022-09-08T00:49:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:56 smithi055 ceph-mon[97033]: pgmap v6146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:49:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:56 smithi059 ceph-mon[97450]: pgmap v6146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:49:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:56.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:56.493591+0000) 2022-09-08T00:49:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:56.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:56.420929+0000) 2022-09-08T00:49:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:57.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:57.493811+0000) 2022-09-08T00:49:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:57.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:57.421168+0000) 2022-09-08T00:49:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:49:58 smithi055 ceph-mon[97033]: pgmap v6147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:49:58 smithi059 ceph-mon[97450]: pgmap v6147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:49:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:58.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:58.494057+0000) 2022-09-08T00:49:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:58.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:58.421311+0000) 2022-09-08T00:49:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:49:59.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:59.494284+0000) 2022-09-08T00:49:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:49:59.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:49:59.421567+0000) 2022-09-08T00:50:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:00 smithi055 ceph-mon[97033]: pgmap v6148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T00:50:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T00:50:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T00:50:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:00 smithi059 ceph-mon[97450]: pgmap v6148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T00:50:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T00:50:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T00:50:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:00.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:00.494450+0000) 2022-09-08T00:50:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:00.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:00.421718+0000) 2022-09-08T00:50:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:01.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:01.494580+0000) 2022-09-08T00:50:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:01.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:01.421893+0000) 2022-09-08T00:50:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:02 smithi055 ceph-mon[97033]: pgmap v6149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:50:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:02 smithi059 ceph-mon[97450]: pgmap v6149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:50:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:02.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:02.494727+0000) 2022-09-08T00:50:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:02.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:02.422108+0000) 2022-09-08T00:50:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:03.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:03.494904+0000) 2022-09-08T00:50:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:03.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:03.422356+0000) 2022-09-08T00:50:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:04 smithi055 ceph-mon[97033]: pgmap v6150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:04.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:04 smithi059 ceph-mon[97450]: pgmap v6150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:04.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:04.495046+0000) 2022-09-08T00:50:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:04.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:04.422614+0000) 2022-09-08T00:50:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:50:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:50:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:50:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:50:05.913 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:05.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:05.495255+0000) 2022-09-08T00:50:05.914 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:05.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:05.422822+0000) 2022-09-08T00:50:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:06 smithi055 ceph-mon[97033]: pgmap v6151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:50:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:06 smithi059 ceph-mon[97450]: pgmap v6151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:50:06.729 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:06.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:06.495452+0000) 2022-09-08T00:50:06.730 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:06.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:06.423041+0000) 2022-09-08T00:50:07.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:50:06.728+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:50:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:07.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:07.495717+0000) 2022-09-08T00:50:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:07.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:07.423199+0000) 2022-09-08T00:50:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:08 smithi055 ceph-mon[97033]: pgmap v6152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:50:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:08 smithi059 ceph-mon[97450]: pgmap v6152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:50:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:08.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:08.495881+0000) 2022-09-08T00:50:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:08.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:08.423356+0000) 2022-09-08T00:50:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:09.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:09.496060+0000) 2022-09-08T00:50:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:09.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:09.423507+0000) 2022-09-08T00:50:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:10 smithi055 ceph-mon[97033]: pgmap v6153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:50:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:10 smithi055 ceph-mon[97033]: osdmap e469: 12 total, 11 up, 11 in 2022-09-08T00:50:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:10 smithi059 ceph-mon[97450]: pgmap v6153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:50:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:10 smithi059 ceph-mon[97450]: osdmap e469: 12 total, 11 up, 11 in 2022-09-08T00:50:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:10.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:10.496210+0000) 2022-09-08T00:50:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:10.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:10.423743+0000) 2022-09-08T00:50:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:11.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:11.496356+0000) 2022-09-08T00:50:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:11.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:11.423968+0000) 2022-09-08T00:50:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:12 smithi055 ceph-mon[97033]: pgmap v6155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:50:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:12 smithi059 ceph-mon[97450]: pgmap v6155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:50:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:12.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:12.496557+0000) 2022-09-08T00:50:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:12.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:12.424225+0000) 2022-09-08T00:50:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:13.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:13.496744+0000) 2022-09-08T00:50:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:13.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:13.424417+0000) 2022-09-08T00:50:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:14 smithi055 ceph-mon[97033]: pgmap v6156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:50:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:14 smithi059 ceph-mon[97450]: pgmap v6156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:50:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:14.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:14.496931+0000) 2022-09-08T00:50:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:14.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:14.424662+0000) 2022-09-08T00:50:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:15.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:15.497181+0000) 2022-09-08T00:50:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:15.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:15.424896+0000) 2022-09-08T00:50:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:16 smithi055 ceph-mon[97033]: pgmap v6157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:50:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:16 smithi055 ceph-mon[97033]: osdmap e470: 12 total, 11 up, 11 in 2022-09-08T00:50:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:16 smithi059 ceph-mon[97450]: pgmap v6157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:50:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:16 smithi059 ceph-mon[97450]: osdmap e470: 12 total, 11 up, 11 in 2022-09-08T00:50:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:16.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:16.497417+0000) 2022-09-08T00:50:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:16.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:16.425124+0000) 2022-09-08T00:50:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:17.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:17.497680+0000) 2022-09-08T00:50:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:17.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:17.425363+0000) 2022-09-08T00:50:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:18 smithi055 ceph-mon[97033]: pgmap v6159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:50:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:18 smithi059 ceph-mon[97450]: pgmap v6159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:50:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:18.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:18.497826+0000) 2022-09-08T00:50:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:18.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:18.425586+0000) 2022-09-08T00:50:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:19.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:19.498061+0000) 2022-09-08T00:50:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:19.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:19.425843+0000) 2022-09-08T00:50:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:20 smithi055 ceph-mon[97033]: pgmap v6160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T00:50:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:20 smithi059 ceph-mon[97450]: pgmap v6160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T00:50:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:20.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:20.498296+0000) 2022-09-08T00:50:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:20.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:20.426111+0000) 2022-09-08T00:50:21.741 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:21.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:21.498464+0000) 2022-09-08T00:50:21.742 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:21.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:21.426341+0000) 2022-09-08T00:50:22.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:50:21.740+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:50:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:22.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:22.498687+0000) 2022-09-08T00:50:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:22.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:22.426594+0000) 2022-09-08T00:50:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:22 smithi055 ceph-mon[97033]: pgmap v6161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:50:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:22 smithi055 ceph-mon[97033]: pgmap v6162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 121 B/s wr, 7 op/s 2022-09-08T00:50:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:50:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:22 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T00:50:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:22 smithi059 ceph-mon[97450]: pgmap v6161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:50:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:22 smithi059 ceph-mon[97450]: pgmap v6162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 121 B/s wr, 7 op/s 2022-09-08T00:50:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:50:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:22 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T00:50:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:23.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:23.498901+0000) 2022-09-08T00:50:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:23.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:23.426824+0000) 2022-09-08T00:50:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:24.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:24.499175+0000) 2022-09-08T00:50:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:24.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:24.427095+0000) 2022-09-08T00:50:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:24 smithi055 ceph-mon[97033]: pgmap v6163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:50:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:24 smithi059 ceph-mon[97450]: pgmap v6163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:50:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:25.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:25.499414+0000) 2022-09-08T00:50:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:25.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:25.427341+0000) 2022-09-08T00:50:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:26.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:26.499611+0000) 2022-09-08T00:50:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:26.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:26.427528+0000) 2022-09-08T00:50:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:26 smithi055 ceph-mon[97033]: pgmap v6164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-08T00:50:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:26 smithi059 ceph-mon[97450]: pgmap v6164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-08T00:50:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:27.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:27.499770+0000) 2022-09-08T00:50:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:27.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:27.427758+0000) 2022-09-08T00:50:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:28.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:28.499981+0000) 2022-09-08T00:50:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:28.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:28.427906+0000) 2022-09-08T00:50:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:28 smithi055 ceph-mon[97033]: pgmap v6165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:50:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:28 smithi059 ceph-mon[97450]: pgmap v6165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:50:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:29.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:29.500205+0000) 2022-09-08T00:50:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:29.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:29.428088+0000) 2022-09-08T00:50:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:30.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:30.500489+0000) 2022-09-08T00:50:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:30.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:30.428309+0000) 2022-09-08T00:50:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:30 smithi055 ceph-mon[97033]: pgmap v6166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:50:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:30 smithi059 ceph-mon[97450]: pgmap v6166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:50:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:31.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:31.500735+0000) 2022-09-08T00:50:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:31.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:31.428577+0000) 2022-09-08T00:50:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:32 smithi055 ceph-mon[97033]: pgmap v6167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:50:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:32 smithi059 ceph-mon[97450]: pgmap v6167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:50:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:32.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:32.500920+0000) 2022-09-08T00:50:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:32.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:32.428778+0000) 2022-09-08T00:50:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:33.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:33.501145+0000) 2022-09-08T00:50:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:33.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:33.428972+0000) 2022-09-08T00:50:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:34.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:34.501389+0000) 2022-09-08T00:50:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:34.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:34.429215+0000) 2022-09-08T00:50:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:34 smithi055 ceph-mon[97033]: pgmap v6168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:34 smithi059 ceph-mon[97450]: pgmap v6168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:35.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:35.501664+0000) 2022-09-08T00:50:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:35.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:35.429356+0000) 2022-09-08T00:50:36.754 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:36.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:36.501861+0000) 2022-09-08T00:50:36.756 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:36.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:36.429608+0000) 2022-09-08T00:50:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:36 smithi055 ceph-mon[97033]: pgmap v6169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:50:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:36 smithi059 ceph-mon[97450]: pgmap v6169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:50:37.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:50:36.751+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:50:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:37.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:37.502047+0000) 2022-09-08T00:50:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:37.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:37.429898+0000) 2022-09-08T00:50:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:38.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:38.502204+0000) 2022-09-08T00:50:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:38.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:38.430157+0000) 2022-09-08T00:50:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:38 smithi055 ceph-mon[97033]: pgmap v6170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:38 smithi059 ceph-mon[97450]: pgmap v6170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:39.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:39.502424+0000) 2022-09-08T00:50:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:39.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:39.430318+0000) 2022-09-08T00:50:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:40.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:40.502720+0000) 2022-09-08T00:50:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:40.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:40.430454+0000) 2022-09-08T00:50:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:40 smithi055 ceph-mon[97033]: pgmap v6171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:40 smithi059 ceph-mon[97450]: pgmap v6171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:41.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:41.502910+0000) 2022-09-08T00:50:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:41.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:41.430714+0000) 2022-09-08T00:50:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:42 smithi055 ceph-mon[97033]: pgmap v6172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:50:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:42 smithi059 ceph-mon[97450]: pgmap v6172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:50:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:42.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:42.430949+0000) 2022-09-08T00:50:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:42.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:42.503048+0000) 2022-09-08T00:50:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:43.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:43.503246+0000) 2022-09-08T00:50:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:43.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:43.431217+0000) 2022-09-08T00:50:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:44.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:44.503477+0000) 2022-09-08T00:50:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:44.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:44.431433+0000) 2022-09-08T00:50:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:44 smithi055 ceph-mon[97033]: pgmap v6173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:44 smithi059 ceph-mon[97450]: pgmap v6173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:45.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:45.503767+0000) 2022-09-08T00:50:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:45.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:45.431716+0000) 2022-09-08T00:50:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:46.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:46.503960+0000) 2022-09-08T00:50:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:46.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:46.431987+0000) 2022-09-08T00:50:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:46 smithi055 ceph-mon[97033]: pgmap v6174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:50:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:46 smithi059 ceph-mon[97450]: pgmap v6174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:50:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:47.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:47.504246+0000) 2022-09-08T00:50:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:47.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:47.432233+0000) 2022-09-08T00:50:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:48.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:48.504460+0000) 2022-09-08T00:50:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:48.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:48.432478+0000) 2022-09-08T00:50:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:48 smithi055 ceph-mon[97033]: pgmap v6175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:48 smithi059 ceph-mon[97450]: pgmap v6175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:49.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:49.504624+0000) 2022-09-08T00:50:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:49.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:49.432718+0000) 2022-09-08T00:50:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:50.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:50.504910+0000) 2022-09-08T00:50:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:50.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:50.432984+0000) 2022-09-08T00:50:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:50 smithi055 ceph-mon[97033]: pgmap v6176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:50 smithi059 ceph-mon[97450]: pgmap v6176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:51.759 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:51.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:51.505065+0000) 2022-09-08T00:50:51.759 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:51.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:51.433262+0000) 2022-09-08T00:50:52.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:50:51.757+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:50:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:52 smithi055 ceph-mon[97033]: pgmap v6177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:50:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:52 smithi059 ceph-mon[97450]: pgmap v6177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:50:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:52.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:52.505273+0000) 2022-09-08T00:50:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:52.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:52.433533+0000) 2022-09-08T00:50:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:53.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:53.505503+0000) 2022-09-08T00:50:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:53.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:53.433766+0000) 2022-09-08T00:50:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:54.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:54.505759+0000) 2022-09-08T00:50:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:54.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:54.434005+0000) 2022-09-08T00:50:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:54 smithi055 ceph-mon[97033]: pgmap v6178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:54 smithi059 ceph-mon[97450]: pgmap v6178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:55.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:55.506053+0000) 2022-09-08T00:50:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:55.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:55.434246+0000) 2022-09-08T00:50:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:56.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:56.506319+0000) 2022-09-08T00:50:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:56.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:56.434483+0000) 2022-09-08T00:50:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:56 smithi055 ceph-mon[97033]: pgmap v6179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:50:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:56 smithi059 ceph-mon[97450]: pgmap v6179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:50:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:57.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:57.506476+0000) 2022-09-08T00:50:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:57.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:57.434743+0000) 2022-09-08T00:50:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:58.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:58.506618+0000) 2022-09-08T00:50:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:58.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:58.434945+0000) 2022-09-08T00:50:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:50:58 smithi055 ceph-mon[97033]: pgmap v6180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:50:58 smithi059 ceph-mon[97450]: pgmap v6180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:50:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:50:59.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:59.506843+0000) 2022-09-08T00:50:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:50:59.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:50:59.435158+0000) 2022-09-08T00:51:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:00.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:00.507078+0000) 2022-09-08T00:51:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:00.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:00.435379+0000) 2022-09-08T00:51:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:00 smithi055 ceph-mon[97033]: pgmap v6181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:00 smithi059 ceph-mon[97450]: pgmap v6181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:01.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:01.507332+0000) 2022-09-08T00:51:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:01.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:01.435565+0000) 2022-09-08T00:51:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:02 smithi055 ceph-mon[97033]: pgmap v6182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:51:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:02 smithi059 ceph-mon[97450]: pgmap v6182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:51:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:02.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:02.507581+0000) 2022-09-08T00:51:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:02.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:02.435767+0000) 2022-09-08T00:51:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:03.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:03.507783+0000) 2022-09-08T00:51:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:03.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:03.436012+0000) 2022-09-08T00:51:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:04.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:04.508013+0000) 2022-09-08T00:51:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:04.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:04.436282+0000) 2022-09-08T00:51:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:04 smithi055 ceph-mon[97033]: pgmap v6183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:04 smithi059 ceph-mon[97450]: pgmap v6183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:05.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:05.508243+0000) 2022-09-08T00:51:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:05.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:05.436530+0000) 2022-09-08T00:51:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:51:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:51:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:51:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:51:06.767 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:06.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:06.508466+0000) 2022-09-08T00:51:06.768 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:06.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:06.436764+0000) 2022-09-08T00:51:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:06 smithi055 ceph-mon[97033]: pgmap v6184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:51:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:06 smithi059 ceph-mon[97450]: pgmap v6184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:51:07.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:51:06.766+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:51:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:07.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:07.508730+0000) 2022-09-08T00:51:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:07.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:07.436912+0000) 2022-09-08T00:51:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:08.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:08.508960+0000) 2022-09-08T00:51:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:08.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:08.437123+0000) 2022-09-08T00:51:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:08 smithi055 ceph-mon[97033]: pgmap v6185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:51:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:08 smithi059 ceph-mon[97450]: pgmap v6185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:51:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:09.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:09.509172+0000) 2022-09-08T00:51:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:09.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:09.437373+0000) 2022-09-08T00:51:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:09 smithi055 ceph-mon[97033]: osdmap e471: 12 total, 11 up, 11 in 2022-09-08T00:51:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:09 smithi059 ceph-mon[97450]: osdmap e471: 12 total, 11 up, 11 in 2022-09-08T00:51:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:10.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:10.509411+0000) 2022-09-08T00:51:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:10.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:10.437542+0000) 2022-09-08T00:51:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:10 smithi055 ceph-mon[97033]: pgmap v6187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:51:11.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:10 smithi059 ceph-mon[97450]: pgmap v6187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:51:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:11.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:11.509603+0000) 2022-09-08T00:51:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:11.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:11.437716+0000) 2022-09-08T00:51:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:12 smithi055 ceph-mon[97033]: pgmap v6188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:51:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:12 smithi059 ceph-mon[97450]: pgmap v6188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:51:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:12.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:12.509717+0000) 2022-09-08T00:51:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:12.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:12.437905+0000) 2022-09-08T00:51:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:13.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:13.509916+0000) 2022-09-08T00:51:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:13.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:13.438192+0000) 2022-09-08T00:51:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:14.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:14.510072+0000) 2022-09-08T00:51:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:14.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:14.438404+0000) 2022-09-08T00:51:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:14 smithi055 ceph-mon[97033]: pgmap v6189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:51:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:14 smithi059 ceph-mon[97450]: pgmap v6189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:51:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:15.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:15.510235+0000) 2022-09-08T00:51:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:15.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:15.438665+0000) 2022-09-08T00:51:16.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:16.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:16.510450+0000) 2022-09-08T00:51:16.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:16.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:16.438825+0000) 2022-09-08T00:51:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:16 smithi055 ceph-mon[97033]: pgmap v6190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:51:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:16 smithi055 ceph-mon[97033]: osdmap e472: 12 total, 11 up, 11 in 2022-09-08T00:51:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:16 smithi059 ceph-mon[97450]: pgmap v6190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:51:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:16 smithi059 ceph-mon[97450]: osdmap e472: 12 total, 11 up, 11 in 2022-09-08T00:51:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:17.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:17.510712+0000) 2022-09-08T00:51:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:17.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:17.439059+0000) 2022-09-08T00:51:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:18.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:18.439215+0000) 2022-09-08T00:51:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:18.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:18.510967+0000) 2022-09-08T00:51:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:18 smithi055 ceph-mon[97033]: pgmap v6192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T00:51:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:18 smithi059 ceph-mon[97450]: pgmap v6192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T00:51:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:19.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:19.511245+0000) 2022-09-08T00:51:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:19.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:19.439457+0000) 2022-09-08T00:51:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:20.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:20.511475+0000) 2022-09-08T00:51:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:20.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:20.439702+0000) 2022-09-08T00:51:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:20 smithi055 ceph-mon[97033]: pgmap v6193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:51:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:20 smithi059 ceph-mon[97450]: pgmap v6193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:51:21.730 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:21.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:21.511722+0000) 2022-09-08T00:51:21.731 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:21.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:21.439869+0000) 2022-09-08T00:51:22.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:51:21.773+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:51:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:22 smithi055 ceph-mon[97033]: pgmap v6194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:51:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:51:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:51:22.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:51:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:22 smithi059 ceph-mon[97450]: pgmap v6194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:51:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:51:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:51:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:51:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:22.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:22.511945+0000) 2022-09-08T00:51:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:22.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:22.440083+0000) 2022-09-08T00:51:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:23.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:23.512170+0000) 2022-09-08T00:51:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:23.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:23.440312+0000) 2022-09-08T00:51:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:24.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:24.512305+0000) 2022-09-08T00:51:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:24.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:24.440484+0000) 2022-09-08T00:51:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:24 smithi055 ceph-mon[97033]: pgmap v6195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:51:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:24 smithi059 ceph-mon[97450]: pgmap v6195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:51:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:25.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:25.512534+0000) 2022-09-08T00:51:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:25.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:25.440693+0000) 2022-09-08T00:51:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:26.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:26.512755+0000) 2022-09-08T00:51:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:26.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:26.440832+0000) 2022-09-08T00:51:26.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:26 smithi055 ceph-mon[97033]: pgmap v6196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:51:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:26 smithi059 ceph-mon[97450]: pgmap v6196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:51:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:27.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:27.512954+0000) 2022-09-08T00:51:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:27.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:27.441071+0000) 2022-09-08T00:51:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:28.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:28.513102+0000) 2022-09-08T00:51:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:28.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:28.441321+0000) 2022-09-08T00:51:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:28 smithi055 ceph-mon[97033]: pgmap v6197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:51:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:28 smithi059 ceph-mon[97450]: pgmap v6197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:51:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:29.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:29.441563+0000) 2022-09-08T00:51:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:29.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:29.513327+0000) 2022-09-08T00:51:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:30.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:30.513537+0000) 2022-09-08T00:51:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:30.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:30.441794+0000) 2022-09-08T00:51:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:30 smithi055 ceph-mon[97033]: pgmap v6198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:30 smithi059 ceph-mon[97450]: pgmap v6198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:31.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:31.513746+0000) 2022-09-08T00:51:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:31.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:31.442048+0000) 2022-09-08T00:51:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:32 smithi055 ceph-mon[97033]: pgmap v6199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:51:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:32 smithi059 ceph-mon[97450]: pgmap v6199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:51:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:32.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:32.513971+0000) 2022-09-08T00:51:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:32.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:32.442265+0000) 2022-09-08T00:51:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:33.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:33.514185+0000) 2022-09-08T00:51:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:33.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:33.442469+0000) 2022-09-08T00:51:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:34.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:34.514400+0000) 2022-09-08T00:51:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:34.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:34.442742+0000) 2022-09-08T00:51:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:34 smithi055 ceph-mon[97033]: pgmap v6200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:34 smithi059 ceph-mon[97450]: pgmap v6200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:35.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:35.514681+0000) 2022-09-08T00:51:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:35.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:35.442977+0000) 2022-09-08T00:51:36.788 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:36.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:36.514881+0000) 2022-09-08T00:51:36.789 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:36.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:36.443169+0000) 2022-09-08T00:51:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:36 smithi055 ceph-mon[97033]: pgmap v6201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:51:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:36 smithi059 ceph-mon[97450]: pgmap v6201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:51:37.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:51:36.787+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:51:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:37.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:37.515089+0000) 2022-09-08T00:51:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:37.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:37.443438+0000) 2022-09-08T00:51:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:38.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:38.515279+0000) 2022-09-08T00:51:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:38.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:38.443681+0000) 2022-09-08T00:51:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:38 smithi055 ceph-mon[97033]: pgmap v6202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:38 smithi059 ceph-mon[97450]: pgmap v6202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:39.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:39.515493+0000) 2022-09-08T00:51:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:39.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:39.443904+0000) 2022-09-08T00:51:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:40.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:40.515729+0000) 2022-09-08T00:51:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:40.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:40.444170+0000) 2022-09-08T00:51:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:40 smithi055 ceph-mon[97033]: pgmap v6203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:40 smithi059 ceph-mon[97450]: pgmap v6203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:41.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:41.515889+0000) 2022-09-08T00:51:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:41.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:41.444385+0000) 2022-09-08T00:51:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:42 smithi055 ceph-mon[97033]: pgmap v6204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:51:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:42 smithi059 ceph-mon[97450]: pgmap v6204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:51:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:42.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:42.516079+0000) 2022-09-08T00:51:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:42.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:42.444596+0000) 2022-09-08T00:51:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:43.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:43.516312+0000) 2022-09-08T00:51:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:43.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:43.444880+0000) 2022-09-08T00:51:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:44.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:44.516527+0000) 2022-09-08T00:51:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:44.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:44.445117+0000) 2022-09-08T00:51:44.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:44 smithi055 ceph-mon[97033]: pgmap v6205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:44 smithi059 ceph-mon[97450]: pgmap v6205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:45.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:45.516731+0000) 2022-09-08T00:51:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:45.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:45.445340+0000) 2022-09-08T00:51:46.759 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:46.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:46.516856+0000) 2022-09-08T00:51:46.760 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:46.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:46.445584+0000) 2022-09-08T00:51:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:46 smithi055 ceph-mon[97033]: pgmap v6206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:51:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:46 smithi059 ceph-mon[97450]: pgmap v6206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:51:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:47.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:47.517066+0000) 2022-09-08T00:51:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:47.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:47.445818+0000) 2022-09-08T00:51:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:48.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:48.517316+0000) 2022-09-08T00:51:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:48.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:48.445997+0000) 2022-09-08T00:51:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:48 smithi055 ceph-mon[97033]: pgmap v6207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:48 smithi059 ceph-mon[97450]: pgmap v6207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:49.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:49.517551+0000) 2022-09-08T00:51:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:49.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:49.446252+0000) 2022-09-08T00:51:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:50.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:50.517799+0000) 2022-09-08T00:51:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:50.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:50.446485+0000) 2022-09-08T00:51:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:50 smithi055 ceph-mon[97033]: pgmap v6208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:50 smithi059 ceph-mon[97450]: pgmap v6208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:51:51.794 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:51.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:51.517953+0000) 2022-09-08T00:51:51.795 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:51.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:51.446773+0000) 2022-09-08T00:51:52.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:51:51.793+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:51:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:52.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:52.518104+0000) 2022-09-08T00:51:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:52.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:52.446988+0000) 2022-09-08T00:51:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:52 smithi055 ceph-mon[97033]: pgmap v6209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:51:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:52 smithi055 ceph-mon[97033]: pgmap v6210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:51:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:52 smithi055 ceph-mon[97033]: pgmap v6211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:51:53.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:51:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:52 smithi059 ceph-mon[97450]: pgmap v6209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:51:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:52 smithi059 ceph-mon[97450]: pgmap v6210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:51:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:52 smithi059 ceph-mon[97450]: pgmap v6211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:51:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:51:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:53.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:53.518231+0000) 2022-09-08T00:51:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:53.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:53.447223+0000) 2022-09-08T00:51:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:54.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:54.518441+0000) 2022-09-08T00:51:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:54.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:54.447427+0000) 2022-09-08T00:51:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:54 smithi055 ceph-mon[97033]: pgmap v6212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:51:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:54 smithi059 ceph-mon[97450]: pgmap v6212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:51:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:55.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:55.518689+0000) 2022-09-08T00:51:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:55.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:55.447695+0000) 2022-09-08T00:51:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:56.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:56.447945+0000) 2022-09-08T00:51:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:56.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:56.518933+0000) 2022-09-08T00:51:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:56 smithi055 ceph-mon[97033]: pgmap v6213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:51:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:56 smithi059 ceph-mon[97450]: pgmap v6213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:51:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:57.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:57.519079+0000) 2022-09-08T00:51:57.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:57.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:57.448160+0000) 2022-09-08T00:51:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:58.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:58.448400+0000) 2022-09-08T00:51:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:58.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:58.519293+0000) 2022-09-08T00:51:59.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:51:58 smithi055 ceph-mon[97033]: pgmap v6214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:51:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:51:58 smithi059 ceph-mon[97450]: pgmap v6214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:51:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:51:59.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:59.448621+0000) 2022-09-08T00:51:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:51:59.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:51:59.519539+0000) 2022-09-08T00:52:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:00.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:00.519757+0000) 2022-09-08T00:52:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:00.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:00.448896+0000) 2022-09-08T00:52:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:00 smithi055 ceph-mon[97033]: pgmap v6215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:52:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:00 smithi059 ceph-mon[97450]: pgmap v6215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:52:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:01.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:01.519954+0000) 2022-09-08T00:52:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:01.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:01.449098+0000) 2022-09-08T00:52:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:02 smithi055 ceph-mon[97033]: pgmap v6216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:52:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:02 smithi059 ceph-mon[97450]: pgmap v6216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:52:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:02.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:02.449269+0000) 2022-09-08T00:52:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:02.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:02.520175+0000) 2022-09-08T00:52:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:03.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:03.449518+0000) 2022-09-08T00:52:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:03.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:03.520397+0000) 2022-09-08T00:52:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:03 smithi055 ceph-mon[97033]: pgmap v6217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:52:04.381 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:03 smithi059 ceph-mon[97450]: pgmap v6217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:52:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:04.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:04.520560+0000) 2022-09-08T00:52:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:04.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:04.449760+0000) 2022-09-08T00:52:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:52:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:52:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:52:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:52:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:05.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:05.520812+0000) 2022-09-08T00:52:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:05.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:05.449980+0000) 2022-09-08T00:52:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:05 smithi055 ceph-mon[97033]: pgmap v6218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:52:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:05 smithi059 ceph-mon[97450]: pgmap v6218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:52:06.805 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:06.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:06.521020+0000) 2022-09-08T00:52:06.805 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:06.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:06.450150+0000) 2022-09-08T00:52:07.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:52:06.802+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:52:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:07.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:07.521208+0000) 2022-09-08T00:52:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:07.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:07.450322+0000) 2022-09-08T00:52:08.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:07 smithi055 ceph-mon[97033]: pgmap v6219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:52:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:07 smithi059 ceph-mon[97450]: pgmap v6219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:52:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:08.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:08.521432+0000) 2022-09-08T00:52:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:08.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:08.450583+0000) 2022-09-08T00:52:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:09.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:09.521597+0000) 2022-09-08T00:52:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:09.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:09.450786+0000) 2022-09-08T00:52:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:09 smithi055 ceph-mon[97033]: osdmap e473: 12 total, 11 up, 11 in 2022-09-08T00:52:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:09 smithi055 ceph-mon[97033]: pgmap v6221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:52:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:09 smithi059 ceph-mon[97450]: osdmap e473: 12 total, 11 up, 11 in 2022-09-08T00:52:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:09 smithi059 ceph-mon[97450]: pgmap v6221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:52:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:10.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:10.521822+0000) 2022-09-08T00:52:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:10.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:10.451021+0000) 2022-09-08T00:52:11.862 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:11.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:11.521997+0000) 2022-09-08T00:52:11.863 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:11.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:11.451251+0000) 2022-09-08T00:52:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:12 smithi055 ceph-mon[97033]: pgmap v6222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:52:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:12 smithi059 ceph-mon[97450]: pgmap v6222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:52:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:12.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:12.522120+0000) 2022-09-08T00:52:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:12.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:12.451479+0000) 2022-09-08T00:52:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:13.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:13.522332+0000) 2022-09-08T00:52:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:13.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:13.451719+0000) 2022-09-08T00:52:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:13 smithi055 ceph-mon[97033]: pgmap v6223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:52:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:13 smithi059 ceph-mon[97450]: pgmap v6223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:52:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:14.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:14.522491+0000) 2022-09-08T00:52:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:14.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:14.451949+0000) 2022-09-08T00:52:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:15.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:15.522734+0000) 2022-09-08T00:52:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:15.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:15.452187+0000) 2022-09-08T00:52:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:15 smithi055 ceph-mon[97033]: pgmap v6224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:52:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:15 smithi059 ceph-mon[97450]: pgmap v6224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:52:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:16.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:16.522988+0000) 2022-09-08T00:52:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:16.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:16.452453+0000) 2022-09-08T00:52:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:17 smithi055 ceph-mon[97033]: osdmap e474: 12 total, 11 up, 11 in 2022-09-08T00:52:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:17 smithi059 ceph-mon[97450]: osdmap e474: 12 total, 11 up, 11 in 2022-09-08T00:52:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:17.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:17.523121+0000) 2022-09-08T00:52:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:17.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:17.452679+0000) 2022-09-08T00:52:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:18 smithi055 ceph-mon[97033]: pgmap v6226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T00:52:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:18 smithi059 ceph-mon[97450]: pgmap v6226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T00:52:18.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:18.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:18.452914+0000) 2022-09-08T00:52:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:18.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:18.523350+0000) 2022-09-08T00:52:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:19.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:19.523558+0000) 2022-09-08T00:52:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:19.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:19.453142+0000) 2022-09-08T00:52:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:19 smithi055 ceph-mon[97033]: pgmap v6227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:52:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:19 smithi059 ceph-mon[97450]: pgmap v6227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:52:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:20.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:20.523704+0000) 2022-09-08T00:52:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:20.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:20.453364+0000) 2022-09-08T00:52:21.813 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:21.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:21.453605+0000) 2022-09-08T00:52:21.813 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:21.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:21.523913+0000) 2022-09-08T00:52:22.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:52:21.812+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:52:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:22 smithi055 ceph-mon[97033]: pgmap v6228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:52:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:22 smithi059 ceph-mon[97450]: pgmap v6228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:52:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:22.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:22.524161+0000) 2022-09-08T00:52:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:22.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:22.453787+0000) 2022-09-08T00:52:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:23.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:23.524364+0000) 2022-09-08T00:52:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:23.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:23.454008+0000) 2022-09-08T00:52:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:23 smithi055 ceph-mon[97033]: pgmap v6229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:52:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:23 smithi059 ceph-mon[97450]: pgmap v6229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:52:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:24.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:24.454272+0000) 2022-09-08T00:52:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:24.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:24.524672+0000) 2022-09-08T00:52:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:25.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:25.454516+0000) 2022-09-08T00:52:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:25.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:25.524873+0000) 2022-09-08T00:52:26.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:25 smithi055 ceph-mon[97033]: pgmap v6230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:52:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:25 smithi059 ceph-mon[97450]: pgmap v6230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:52:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:26.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:26.525095+0000) 2022-09-08T00:52:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:26.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:26.454681+0000) 2022-09-08T00:52:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:27.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:27.525332+0000) 2022-09-08T00:52:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:27.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:27.454845+0000) 2022-09-08T00:52:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:27 smithi055 ceph-mon[97033]: pgmap v6231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:52:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:27 smithi059 ceph-mon[97450]: pgmap v6231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T00:52:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:28.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:28.525561+0000) 2022-09-08T00:52:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:28.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:28.455074+0000) 2022-09-08T00:52:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:29.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:29.525803+0000) 2022-09-08T00:52:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:29.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:29.455278+0000) 2022-09-08T00:52:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:29 smithi055 ceph-mon[97033]: pgmap v6232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T00:52:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:29 smithi059 ceph-mon[97450]: pgmap v6232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T00:52:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:30.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:30.455503+0000) 2022-09-08T00:52:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:30.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:30.526106+0000) 2022-09-08T00:52:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:31.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:31.526419+0000) 2022-09-08T00:52:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:31.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:31.455746+0000) 2022-09-08T00:52:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:32 smithi055 ceph-mon[97033]: pgmap v6233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:52:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:32 smithi059 ceph-mon[97450]: pgmap v6233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:52:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:32.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:32.526692+0000) 2022-09-08T00:52:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:32.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:32.455978+0000) 2022-09-08T00:52:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:33.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:33.526950+0000) 2022-09-08T00:52:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:33.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:33.456198+0000) 2022-09-08T00:52:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:33 smithi055 ceph-mon[97033]: pgmap v6234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:52:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:33 smithi059 ceph-mon[97450]: pgmap v6234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:52:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:34.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:34.456351+0000) 2022-09-08T00:52:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:34.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:34.527128+0000) 2022-09-08T00:52:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:35.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:35.527296+0000) 2022-09-08T00:52:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:35.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:35.456581+0000) 2022-09-08T00:52:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:35 smithi055 ceph-mon[97033]: pgmap v6235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:52:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:35 smithi059 ceph-mon[97450]: pgmap v6235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:52:36.820 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:36.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:36.527496+0000) 2022-09-08T00:52:36.821 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:36.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:36.456816+0000) 2022-09-08T00:52:37.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:52:36.819+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:52:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:37.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:37.527745+0000) 2022-09-08T00:52:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:37.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:37.456984+0000) 2022-09-08T00:52:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:37 smithi055 ceph-mon[97033]: pgmap v6236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:52:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:37 smithi059 ceph-mon[97450]: pgmap v6236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:52:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:38.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:38.527955+0000) 2022-09-08T00:52:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:38.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:38.457198+0000) 2022-09-08T00:52:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:39.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:39.528118+0000) 2022-09-08T00:52:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:39.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:39.457423+0000) 2022-09-08T00:52:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:39 smithi055 ceph-mon[97033]: pgmap v6237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:52:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:39 smithi059 ceph-mon[97450]: pgmap v6237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:52:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:40.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:40.528334+0000) 2022-09-08T00:52:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:40.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:40.457624+0000) 2022-09-08T00:52:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:41.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:41.528577+0000) 2022-09-08T00:52:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:41.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:41.457786+0000) 2022-09-08T00:52:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:42 smithi055 ceph-mon[97033]: pgmap v6238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T00:52:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:42 smithi059 ceph-mon[97450]: pgmap v6238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T00:52:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:42.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:42.528777+0000) 2022-09-08T00:52:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:42.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:42.458007+0000) 2022-09-08T00:52:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:43.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:43.528954+0000) 2022-09-08T00:52:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:43.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:43.458194+0000) 2022-09-08T00:52:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:43 smithi055 ceph-mon[97033]: pgmap v6239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:52:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:43 smithi059 ceph-mon[97450]: pgmap v6239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:52:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:44.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:44.529084+0000) 2022-09-08T00:52:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:44.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:44.458406+0000) 2022-09-08T00:52:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:45.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:45.529287+0000) 2022-09-08T00:52:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:45.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:45.458602+0000) 2022-09-08T00:52:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:45 smithi055 ceph-mon[97033]: pgmap v6240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:52:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:45 smithi059 ceph-mon[97450]: pgmap v6240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:52:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:46.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:46.529537+0000) 2022-09-08T00:52:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:46.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:46.458841+0000) 2022-09-08T00:52:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:47.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:47.529740+0000) 2022-09-08T00:52:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:47.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:47.459093+0000) 2022-09-08T00:52:48.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:47 smithi055 ceph-mon[97033]: pgmap v6241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:52:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:47 smithi059 ceph-mon[97450]: pgmap v6241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:52:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:48.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:48.530011+0000) 2022-09-08T00:52:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:48.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:48.459286+0000) 2022-09-08T00:52:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:49.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:49.530220+0000) 2022-09-08T00:52:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:49.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:49.459561+0000) 2022-09-08T00:52:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:49 smithi055 ceph-mon[97033]: pgmap v6242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:52:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:49 smithi059 ceph-mon[97450]: pgmap v6242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:52:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:50.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:50.530454+0000) 2022-09-08T00:52:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:50.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:50.459796+0000) 2022-09-08T00:52:51.821 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:51.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:51.530615+0000) 2022-09-08T00:52:51.822 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:51.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:51.460070+0000) 2022-09-08T00:52:52.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:52:51.819+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:52:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:52 smithi059 ceph-mon[97450]: pgmap v6243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:52:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:52:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:52:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:52:52.458 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:52 smithi055 ceph-mon[97033]: pgmap v6243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:52:52.459 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:52:52.459 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:52:52.459 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:52:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:52.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:52.530707+0000) 2022-09-08T00:52:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:52.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:52.460296+0000) 2022-09-08T00:52:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:52:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:52:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:52:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:52:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:53.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:53.530879+0000) 2022-09-08T00:52:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:53.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:53.460542+0000) 2022-09-08T00:52:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:54 smithi055 ceph-mon[97033]: pgmap v6244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:52:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:54 smithi059 ceph-mon[97450]: pgmap v6244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:52:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:54.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:54.531016+0000) 2022-09-08T00:52:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:54.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:54.460755+0000) 2022-09-08T00:52:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:55.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:55.531193+0000) 2022-09-08T00:52:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:55.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:55.460885+0000) 2022-09-08T00:52:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:52:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:52:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:52:56.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:52:56.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:52:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:52:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:52:56.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:52:56.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T00:52:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:56.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:56.531399+0000) 2022-09-08T00:52:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:56.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:56.461141+0000) 2022-09-08T00:52:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:56 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T00:52:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:56 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T00:52:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:56 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T00:52:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:56 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T00:52:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:56 smithi055 ceph-mon[97033]: pgmap v6245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:52:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:56 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T00:52:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:56 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T00:52:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:56 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T00:52:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:56 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T00:52:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:56 smithi059 ceph-mon[97450]: pgmap v6245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:52:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:57.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:57.531674+0000) 2022-09-08T00:52:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:57.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:57.461375+0000) 2022-09-08T00:52:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:57 smithi055 ceph-mon[97033]: pgmap v6246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:52:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:57 smithi059 ceph-mon[97450]: pgmap v6246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:52:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:58.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:58.461592+0000) 2022-09-08T00:52:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:58.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:58.531836+0000) 2022-09-08T00:52:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:52:59.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:59.532080+0000) 2022-09-08T00:52:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:52:59.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:52:59.461785+0000) 2022-09-08T00:53:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:52:59 smithi055 ceph-mon[97033]: pgmap v6247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:53:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:52:59 smithi059 ceph-mon[97450]: pgmap v6247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:53:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:00.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:00.461903+0000) 2022-09-08T00:53:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:00.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:00.532307+0000) 2022-09-08T00:53:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:01.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:01.462109+0000) 2022-09-08T00:53:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:01.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:01.532481+0000) 2022-09-08T00:53:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:02 smithi055 ceph-mon[97033]: pgmap v6248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:53:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:02 smithi059 ceph-mon[97450]: pgmap v6248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T00:53:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:02.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:02.532765+0000) 2022-09-08T00:53:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:02.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:02.462331+0000) 2022-09-08T00:53:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:03.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:03.532979+0000) 2022-09-08T00:53:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:03.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:03.462521+0000) 2022-09-08T00:53:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:03 smithi055 ceph-mon[97033]: pgmap v6249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:03 smithi059 ceph-mon[97450]: pgmap v6249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:04.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:04.533215+0000) 2022-09-08T00:53:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:04.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:04.462748+0000) 2022-09-08T00:53:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:53:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:53:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:53:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:53:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:05.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:05.533383+0000) 2022-09-08T00:53:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:05.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:05.462969+0000) 2022-09-08T00:53:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:05 smithi055 ceph-mon[97033]: pgmap v6250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:53:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:05 smithi059 ceph-mon[97450]: pgmap v6250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:53:06.829 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:06.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:06.463121+0000) 2022-09-08T00:53:06.830 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:06.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:06.533623+0000) 2022-09-08T00:53:07.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:53:06.828+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:53:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:07.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:07.533775+0000) 2022-09-08T00:53:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:07.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:07.463347+0000) 2022-09-08T00:53:08.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:07 smithi055 ceph-mon[97033]: pgmap v6251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:53:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:07 smithi059 ceph-mon[97450]: pgmap v6251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:53:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:08.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:08.533995+0000) 2022-09-08T00:53:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:08.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:08.463504+0000) 2022-09-08T00:53:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:09.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:09.534173+0000) 2022-09-08T00:53:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:09.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:09.463715+0000) 2022-09-08T00:53:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:09 smithi055 ceph-mon[97033]: osdmap e475: 12 total, 11 up, 11 in 2022-09-08T00:53:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:09 smithi055 ceph-mon[97033]: pgmap v6253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:53:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:09 smithi059 ceph-mon[97450]: osdmap e475: 12 total, 11 up, 11 in 2022-09-08T00:53:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:09 smithi059 ceph-mon[97450]: pgmap v6253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:53:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:10.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:10.534439+0000) 2022-09-08T00:53:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:10.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:10.463909+0000) 2022-09-08T00:53:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:11.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:11.534733+0000) 2022-09-08T00:53:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:11.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:11.464092+0000) 2022-09-08T00:53:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:12 smithi055 ceph-mon[97033]: pgmap v6254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:53:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:12 smithi059 ceph-mon[97450]: pgmap v6254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:53:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:12.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:12.534997+0000) 2022-09-08T00:53:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:12.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:12.464360+0000) 2022-09-08T00:53:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:13.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:13.535249+0000) 2022-09-08T00:53:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:13.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:13.464614+0000) 2022-09-08T00:53:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:13 smithi055 ceph-mon[97033]: pgmap v6255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:53:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:13 smithi059 ceph-mon[97450]: pgmap v6255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:53:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:14.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:14.535517+0000) 2022-09-08T00:53:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:14.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:14.464845+0000) 2022-09-08T00:53:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:15.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:15.535760+0000) 2022-09-08T00:53:15.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:15.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:15.465074+0000) 2022-09-08T00:53:16.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:15 smithi055 ceph-mon[97033]: pgmap v6256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:53:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:15 smithi059 ceph-mon[97450]: pgmap v6256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:53:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:16.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:16.535919+0000) 2022-09-08T00:53:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:16.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:16.465302+0000) 2022-09-08T00:53:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:17 smithi055 ceph-mon[97033]: osdmap e476: 12 total, 11 up, 11 in 2022-09-08T00:53:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:17 smithi059 ceph-mon[97450]: osdmap e476: 12 total, 11 up, 11 in 2022-09-08T00:53:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:17.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:17.536152+0000) 2022-09-08T00:53:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:17.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:17.465538+0000) 2022-09-08T00:53:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:18 smithi055 ceph-mon[97033]: pgmap v6258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T00:53:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:18 smithi059 ceph-mon[97450]: pgmap v6258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T00:53:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:18.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:18.536335+0000) 2022-09-08T00:53:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:18.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:18.465724+0000) 2022-09-08T00:53:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:19.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:19.536545+0000) 2022-09-08T00:53:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:19.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:19.465968+0000) 2022-09-08T00:53:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:19 smithi055 ceph-mon[97033]: pgmap v6259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:53:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:19 smithi059 ceph-mon[97450]: pgmap v6259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:53:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:20.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:20.536782+0000) 2022-09-08T00:53:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:20.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:20.466218+0000) 2022-09-08T00:53:21.845 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:21.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:21.536991+0000) 2022-09-08T00:53:21.846 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:21.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:21.466465+0000) 2022-09-08T00:53:22.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:53:21.844+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:53:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:22 smithi055 ceph-mon[97033]: pgmap v6260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:53:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:53:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:22 smithi059 ceph-mon[97450]: pgmap v6260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:53:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:53:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:22.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:22.537203+0000) 2022-09-08T00:53:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:22.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:22.466729+0000) 2022-09-08T00:53:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:23 smithi055 ceph-mon[97033]: pgmap v6261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T00:53:23.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:23 smithi055 ceph-mon[97033]: pgmap v6262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:53:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:23 smithi059 ceph-mon[97450]: pgmap v6261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T00:53:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:23 smithi059 ceph-mon[97450]: pgmap v6262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:53:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:23.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:23.537436+0000) 2022-09-08T00:53:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:23.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:23.466889+0000) 2022-09-08T00:53:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:24.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:24.467126+0000) 2022-09-08T00:53:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:24.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:24.537696+0000) 2022-09-08T00:53:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:25 smithi055 ceph-mon[97033]: pgmap v6263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s 2022-09-08T00:53:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:25 smithi059 ceph-mon[97450]: pgmap v6263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s 2022-09-08T00:53:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:25.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:25.537905+0000) 2022-09-08T00:53:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:25.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:25.467358+0000) 2022-09-08T00:53:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:26.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:26.538117+0000) 2022-09-08T00:53:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:26.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:26.467602+0000) 2022-09-08T00:53:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:27 smithi055 ceph-mon[97033]: pgmap v6264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:53:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:27 smithi059 ceph-mon[97450]: pgmap v6264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:53:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:27.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:27.538290+0000) 2022-09-08T00:53:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:27.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:27.467877+0000) 2022-09-08T00:53:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:28.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:28.538503+0000) 2022-09-08T00:53:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:28.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:28.468073+0000) 2022-09-08T00:53:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:29 smithi055 ceph-mon[97033]: pgmap v6265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-08T00:53:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:29 smithi059 ceph-mon[97450]: pgmap v6265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-08T00:53:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:29.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:29.538685+0000) 2022-09-08T00:53:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:29.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:29.468262+0000) 2022-09-08T00:53:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:30.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:30.538811+0000) 2022-09-08T00:53:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:30.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:30.468438+0000) 2022-09-08T00:53:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:31 smithi055 ceph-mon[97033]: pgmap v6266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:53:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:31 smithi059 ceph-mon[97450]: pgmap v6266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:53:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:31.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:31.539032+0000) 2022-09-08T00:53:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:31.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:31.468623+0000) 2022-09-08T00:53:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:32.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:32.468862+0000) 2022-09-08T00:53:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:32.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:32.539249+0000) 2022-09-08T00:53:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:33 smithi055 ceph-mon[97033]: pgmap v6267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:53:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:33 smithi059 ceph-mon[97450]: pgmap v6267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:53:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:33.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:33.539491+0000) 2022-09-08T00:53:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:33.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:33.469118+0000) 2022-09-08T00:53:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:34.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:34.539761+0000) 2022-09-08T00:53:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:34.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:34.469379+0000) 2022-09-08T00:53:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:35 smithi055 ceph-mon[97033]: pgmap v6268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:35 smithi059 ceph-mon[97450]: pgmap v6268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:35.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:35.469619+0000) 2022-09-08T00:53:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:35.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:35.539929+0000) 2022-09-08T00:53:36.860 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:36.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:36.540141+0000) 2022-09-08T00:53:36.861 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:36.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:36.469848+0000) 2022-09-08T00:53:37.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:53:36.858+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:53:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:37 smithi055 ceph-mon[97033]: pgmap v6269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:53:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:37 smithi059 ceph-mon[97450]: pgmap v6269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:53:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:37.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:37.540385+0000) 2022-09-08T00:53:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:37.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:37.470048+0000) 2022-09-08T00:53:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:38.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:38.540605+0000) 2022-09-08T00:53:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:38.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:38.470285+0000) 2022-09-08T00:53:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:39 smithi055 ceph-mon[97033]: pgmap v6270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:39 smithi059 ceph-mon[97450]: pgmap v6270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:39.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:39.540815+0000) 2022-09-08T00:53:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:39.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:39.470521+0000) 2022-09-08T00:53:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:40.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:40.540961+0000) 2022-09-08T00:53:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:40.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:40.470730+0000) 2022-09-08T00:53:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:41 smithi055 ceph-mon[97033]: pgmap v6271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:53:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:41 smithi059 ceph-mon[97450]: pgmap v6271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:53:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:41.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:41.541210+0000) 2022-09-08T00:53:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:41.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:41.470901+0000) 2022-09-08T00:53:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:42.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:42.471084+0000) 2022-09-08T00:53:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:42.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:42.541415+0000) 2022-09-08T00:53:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:43 smithi055 ceph-mon[97033]: pgmap v6272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:43 smithi059 ceph-mon[97450]: pgmap v6272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:43.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:43.541680+0000) 2022-09-08T00:53:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:43.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:43.471296+0000) 2022-09-08T00:53:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:44.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:44.541867+0000) 2022-09-08T00:53:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:44.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:44.471538+0000) 2022-09-08T00:53:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:45 smithi055 ceph-mon[97033]: pgmap v6273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:45 smithi059 ceph-mon[97450]: pgmap v6273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:45.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:45.542089+0000) 2022-09-08T00:53:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:45.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:45.471770+0000) 2022-09-08T00:53:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:46.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:46.542332+0000) 2022-09-08T00:53:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:46.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:46.471998+0000) 2022-09-08T00:53:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:47 smithi055 ceph-mon[97033]: pgmap v6274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:53:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:47 smithi059 ceph-mon[97450]: pgmap v6274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:53:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:47.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:47.542549+0000) 2022-09-08T00:53:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:47.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:47.472218+0000) 2022-09-08T00:53:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:48.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:48.542728+0000) 2022-09-08T00:53:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:48.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:48.472471+0000) 2022-09-08T00:53:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:49 smithi055 ceph-mon[97033]: pgmap v6275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:49 smithi059 ceph-mon[97450]: pgmap v6275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:49.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:49.542873+0000) 2022-09-08T00:53:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:49.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:49.472750+0000) 2022-09-08T00:53:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:50.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:50.543077+0000) 2022-09-08T00:53:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:50.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:50.472958+0000) 2022-09-08T00:53:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:51 smithi055 ceph-mon[97033]: pgmap v6276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:53:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:51 smithi059 ceph-mon[97450]: pgmap v6276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:53:51.867 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:51.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:51.543235+0000) 2022-09-08T00:53:51.868 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:51.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:51.473129+0000) 2022-09-08T00:53:52.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:53:51.865+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:53:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:52.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:52.543394+0000) 2022-09-08T00:53:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:52.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:52.473309+0000) 2022-09-08T00:53:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:53 smithi055 ceph-mon[97033]: pgmap v6277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:53 smithi059 ceph-mon[97450]: pgmap v6277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:53.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:53.543615+0000) 2022-09-08T00:53:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:53.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:53.473539+0000) 2022-09-08T00:53:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:54.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:54.543748+0000) 2022-09-08T00:53:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:54.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:54.473766+0000) 2022-09-08T00:53:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:55 smithi055 ceph-mon[97033]: pgmap v6278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:55 smithi059 ceph-mon[97450]: pgmap v6278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:55.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:55.543914+0000) 2022-09-08T00:53:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:55.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:55.473971+0000) 2022-09-08T00:53:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:56.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:56.544132+0000) 2022-09-08T00:53:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:56.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:56.474155+0000) 2022-09-08T00:53:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:57 smithi055 ceph-mon[97033]: pgmap v6279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:53:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:57 smithi059 ceph-mon[97450]: pgmap v6279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:53:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:57.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:57.544361+0000) 2022-09-08T00:53:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:57.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:57.474363+0000) 2022-09-08T00:53:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:58.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:58.474590+0000) 2022-09-08T00:53:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:58.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:58.544620+0000) 2022-09-08T00:53:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:53:59 smithi055 ceph-mon[97033]: pgmap v6280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:53:59 smithi059 ceph-mon[97450]: pgmap v6280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:53:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:53:59.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:59.544870+0000) 2022-09-08T00:53:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:53:59.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:53:59.474831+0000) 2022-09-08T00:54:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:00.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:00.545078+0000) 2022-09-08T00:54:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:00.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:00.474982+0000) 2022-09-08T00:54:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:01 smithi055 ceph-mon[97033]: pgmap v6281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:54:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:01 smithi059 ceph-mon[97450]: pgmap v6281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:54:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:01.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:01.545260+0000) 2022-09-08T00:54:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:01.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:01.475198+0000) 2022-09-08T00:54:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:02.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:02.545419+0000) 2022-09-08T00:54:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:02.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:02.475413+0000) 2022-09-08T00:54:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:03 smithi055 ceph-mon[97033]: pgmap v6282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:03 smithi059 ceph-mon[97450]: pgmap v6282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:03.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:03.545610+0000) 2022-09-08T00:54:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:03.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:03.475650+0000) 2022-09-08T00:54:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:04.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:04.545852+0000) 2022-09-08T00:54:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:04.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:04.475851+0000) 2022-09-08T00:54:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:05 smithi055 ceph-mon[97033]: pgmap v6283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:54:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:54:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:05 smithi059 ceph-mon[97450]: pgmap v6283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:54:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:54:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:05.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:05.546062+0000) 2022-09-08T00:54:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:05.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:05.476087+0000) 2022-09-08T00:54:06.882 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:06.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:06.546288+0000) 2022-09-08T00:54:06.883 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:06.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:06.476296+0000) 2022-09-08T00:54:07.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:54:06.881+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:54:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:07 smithi059 ceph-mon[97450]: pgmap v6284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:54:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:07 smithi055 ceph-mon[97033]: pgmap v6284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:54:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:07.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:07.476528+0000) 2022-09-08T00:54:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:07.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:07.546555+0000) 2022-09-08T00:54:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:08.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:08.476790+0000) 2022-09-08T00:54:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:08.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:08.546799+0000) 2022-09-08T00:54:09.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:09 smithi059 ceph-mon[97450]: pgmap v6285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:54:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:09 smithi055 ceph-mon[97033]: pgmap v6285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:54:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:09.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:09.546929+0000) 2022-09-08T00:54:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:09.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:09.476935+0000) 2022-09-08T00:54:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:10 smithi059 ceph-mon[97450]: osdmap e477: 12 total, 11 up, 11 in 2022-09-08T00:54:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:10 smithi055 ceph-mon[97033]: osdmap e477: 12 total, 11 up, 11 in 2022-09-08T00:54:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:10.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:10.547103+0000) 2022-09-08T00:54:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:10.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:10.477142+0000) 2022-09-08T00:54:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:11 smithi059 ceph-mon[97450]: pgmap v6287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:54:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:11 smithi059 ceph-mon[97450]: osdmap e478: 12 total, 11 up, 11 in 2022-09-08T00:54:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:11 smithi055 ceph-mon[97033]: pgmap v6287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T00:54:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:11 smithi055 ceph-mon[97033]: osdmap e478: 12 total, 11 up, 11 in 2022-09-08T00:54:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:11.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:11.547342+0000) 2022-09-08T00:54:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:11.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:11.477388+0000) 2022-09-08T00:54:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:12.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:12.547573+0000) 2022-09-08T00:54:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:12.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:12.477609+0000) 2022-09-08T00:54:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:13 smithi059 ceph-mon[97450]: pgmap v6289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T00:54:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:13 smithi055 ceph-mon[97033]: pgmap v6289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T00:54:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:13.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:13.547801+0000) 2022-09-08T00:54:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:13.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:13.477813+0000) 2022-09-08T00:54:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:14.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:14.548023+0000) 2022-09-08T00:54:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:14.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:14.478063+0000) 2022-09-08T00:54:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:15 smithi059 ceph-mon[97450]: pgmap v6290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:54:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:15 smithi055 ceph-mon[97033]: pgmap v6290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:54:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:15.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:15.548243+0000) 2022-09-08T00:54:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:15.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:15.478273+0000) 2022-09-08T00:54:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:16.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:16.548486+0000) 2022-09-08T00:54:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:16.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:16.478441+0000) 2022-09-08T00:54:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:17 smithi059 ceph-mon[97450]: pgmap v6291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:54:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:17 smithi055 ceph-mon[97033]: pgmap v6291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:54:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:17.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:17.548751+0000) 2022-09-08T00:54:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:17.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:17.478625+0000) 2022-09-08T00:54:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:18.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:18.548976+0000) 2022-09-08T00:54:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:18.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:18.478771+0000) 2022-09-08T00:54:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:19 smithi059 ceph-mon[97450]: pgmap v6292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T00:54:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:19 smithi055 ceph-mon[97033]: pgmap v6292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T00:54:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:19.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:19.549251+0000) 2022-09-08T00:54:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:19.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:19.478919+0000) 2022-09-08T00:54:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:20.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:20.549529+0000) 2022-09-08T00:54:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:20.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:20.479097+0000) 2022-09-08T00:54:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:21 smithi059 ceph-mon[97450]: pgmap v6293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:54:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:21 smithi055 ceph-mon[97033]: pgmap v6293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:54:21.894 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:21.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:21.549772+0000) 2022-09-08T00:54:21.895 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:21.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:21.479333+0000) 2022-09-08T00:54:22.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:54:21.892+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:54:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:54:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:54:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:54:22.550 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:22.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:22.479605+0000) 2022-09-08T00:54:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:54:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:54:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:54:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:22.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:22.550023+0000) 2022-09-08T00:54:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:23.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:23.550172+0000) 2022-09-08T00:54:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:23.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:23.479859+0000) 2022-09-08T00:54:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:23 smithi055 ceph-mon[97033]: pgmap v6294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T00:54:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:54:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:54:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:23 smithi059 ceph-mon[97450]: pgmap v6294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T00:54:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:54:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:54:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:24.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:24.550322+0000) 2022-09-08T00:54:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:24.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:24.480039+0000) 2022-09-08T00:54:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:25.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:25.550489+0000) 2022-09-08T00:54:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:25.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:25.480221+0000) 2022-09-08T00:54:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:25 smithi055 ceph-mon[97033]: pgmap v6295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:54:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:25 smithi059 ceph-mon[97450]: pgmap v6295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:54:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:26.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:26.550744+0000) 2022-09-08T00:54:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:26.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:26.480452+0000) 2022-09-08T00:54:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:27.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:27.550963+0000) 2022-09-08T00:54:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:27.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:27.480717+0000) 2022-09-08T00:54:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:27 smithi055 ceph-mon[97033]: pgmap v6296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:54:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:27 smithi059 ceph-mon[97450]: pgmap v6296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:54:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:28.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:28.551199+0000) 2022-09-08T00:54:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:28.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:28.480940+0000) 2022-09-08T00:54:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:29.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:29.551446+0000) 2022-09-08T00:54:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:29.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:29.481155+0000) 2022-09-08T00:54:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:29 smithi055 ceph-mon[97033]: pgmap v6297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:29 smithi059 ceph-mon[97450]: pgmap v6297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:30.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:30.551675+0000) 2022-09-08T00:54:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:30.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:30.481374+0000) 2022-09-08T00:54:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:31.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:31.551885+0000) 2022-09-08T00:54:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:31.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:31.481586+0000) 2022-09-08T00:54:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:31 smithi055 ceph-mon[97033]: pgmap v6298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:54:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:31 smithi059 ceph-mon[97450]: pgmap v6298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:54:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:32.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:32.552018+0000) 2022-09-08T00:54:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:32.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:32.481823+0000) 2022-09-08T00:54:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:33.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:33.552219+0000) 2022-09-08T00:54:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:33.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:33.481948+0000) 2022-09-08T00:54:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:33 smithi059 ceph-mon[97450]: pgmap v6299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:33 smithi055 ceph-mon[97033]: pgmap v6299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:34.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:34.552360+0000) 2022-09-08T00:54:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:34.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:34.482185+0000) 2022-09-08T00:54:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:35.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:35.552606+0000) 2022-09-08T00:54:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:35.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:35.482390+0000) 2022-09-08T00:54:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:35 smithi059 ceph-mon[97450]: pgmap v6300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:35 smithi055 ceph-mon[97033]: pgmap v6300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:36.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:36.552882+0000) 2022-09-08T00:54:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:36.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:36.482583+0000) 2022-09-08T00:54:37.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:54:36.906+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:54:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:37.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:37.482831+0000) 2022-09-08T00:54:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:37.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:37.553127+0000) 2022-09-08T00:54:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:37 smithi059 ceph-mon[97450]: pgmap v6301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:54:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:37 smithi055 ceph-mon[97033]: pgmap v6301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:54:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:38.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:38.482994+0000) 2022-09-08T00:54:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:38.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:38.553377+0000) 2022-09-08T00:54:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:39.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:39.553573+0000) 2022-09-08T00:54:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:39.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:39.483199+0000) 2022-09-08T00:54:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:39 smithi059 ceph-mon[97450]: pgmap v6302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:39 smithi055 ceph-mon[97033]: pgmap v6302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:40.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:40.553819+0000) 2022-09-08T00:54:40.914 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:40.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:40.483375+0000) 2022-09-08T00:54:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:41.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:41.554040+0000) 2022-09-08T00:54:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:41.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:41.483578+0000) 2022-09-08T00:54:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:41 smithi059 ceph-mon[97450]: pgmap v6303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:54:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:41 smithi055 ceph-mon[97033]: pgmap v6303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:54:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:42.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:42.554179+0000) 2022-09-08T00:54:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:42.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:42.483800+0000) 2022-09-08T00:54:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:43.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:43.554395+0000) 2022-09-08T00:54:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:43.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:43.484018+0000) 2022-09-08T00:54:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:43 smithi059 ceph-mon[97450]: pgmap v6304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:43 smithi055 ceph-mon[97033]: pgmap v6304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:44.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:44.554631+0000) 2022-09-08T00:54:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:44.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:44.484198+0000) 2022-09-08T00:54:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:45.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:45.554829+0000) 2022-09-08T00:54:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:45.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:45.484414+0000) 2022-09-08T00:54:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:45 smithi059 ceph-mon[97450]: pgmap v6305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:45 smithi055 ceph-mon[97033]: pgmap v6305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:54:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:46.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:46.555057+0000) 2022-09-08T00:54:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:46.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:46.484676+0000) 2022-09-08T00:54:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:47.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:47.555282+0000) 2022-09-08T00:54:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:47.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:47.484911+0000) 2022-09-08T00:54:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:47 smithi059 ceph-mon[97450]: pgmap v6306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:54:48.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:47 smithi055 ceph-mon[97033]: pgmap v6306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:54:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:48.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:48.555503+0000) 2022-09-08T00:54:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:48.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:48.485162+0000) 2022-09-08T00:54:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:49.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:49.555755+0000) 2022-09-08T00:54:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:49.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:49.485408+0000) 2022-09-08T00:54:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:49 smithi059 ceph-mon[97450]: pgmap v6307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:54:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:49 smithi055 ceph-mon[97033]: pgmap v6307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:54:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:50.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:50.555979+0000) 2022-09-08T00:54:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:50.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:50.485677+0000) 2022-09-08T00:54:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:51.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:51.556121+0000) 2022-09-08T00:54:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:51.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:51.485848+0000) 2022-09-08T00:54:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:51 smithi059 ceph-mon[97450]: pgmap v6308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:54:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:51 smithi055 ceph-mon[97033]: pgmap v6308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:54:52.364 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:54:51.911+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:54:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:52.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:52.556365+0000) 2022-09-08T00:54:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:52.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:52.486039+0000) 2022-09-08T00:54:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:53 smithi055 ceph-mon[97033]: pgmap v6309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:54:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:53 smithi055 ceph-mon[97033]: pgmap v6310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:54:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:53 smithi055 ceph-mon[97033]: pgmap v6311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-09-08T00:54:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:54:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:53 smithi059 ceph-mon[97450]: pgmap v6309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:54:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:53 smithi059 ceph-mon[97450]: pgmap v6310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:54:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:53 smithi059 ceph-mon[97450]: pgmap v6311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-09-08T00:54:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:54:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:53.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:53.556630+0000) 2022-09-08T00:54:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:53.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:53.486226+0000) 2022-09-08T00:54:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:54.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:54.556898+0000) 2022-09-08T00:54:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:54.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:54.486443+0000) 2022-09-08T00:54:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:55 smithi055 ceph-mon[97033]: pgmap v6312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:54:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:55 smithi059 ceph-mon[97450]: pgmap v6312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:54:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:55.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:55.486726+0000) 2022-09-08T00:54:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:55.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:55.557143+0000) 2022-09-08T00:54:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:56.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:56.557408+0000) 2022-09-08T00:54:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:56.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:56.486877+0000) 2022-09-08T00:54:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:57 smithi055 ceph-mon[97033]: pgmap v6313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:54:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:57 smithi059 ceph-mon[97450]: pgmap v6313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:54:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:57.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:57.557609+0000) 2022-09-08T00:54:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:57.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:57.487053+0000) 2022-09-08T00:54:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:58.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:58.557810+0000) 2022-09-08T00:54:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:58.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:58.487273+0000) 2022-09-08T00:54:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:54:59 smithi055 ceph-mon[97033]: pgmap v6314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:54:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:54:59 smithi059 ceph-mon[97450]: pgmap v6314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:54:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:54:59.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:59.558048+0000) 2022-09-08T00:54:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:54:59.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:54:59.487531+0000) 2022-09-08T00:55:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:00.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:00.558283+0000) 2022-09-08T00:55:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:00.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:00.487745+0000) 2022-09-08T00:55:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:01 smithi055 ceph-mon[97033]: pgmap v6315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:55:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:01 smithi059 ceph-mon[97450]: pgmap v6315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:55:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:01.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:01.558428+0000) 2022-09-08T00:55:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:01.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:01.488000+0000) 2022-09-08T00:55:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:02.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:02.488193+0000) 2022-09-08T00:55:02.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:02.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:02.558665+0000) 2022-09-08T00:55:03.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:03 smithi055 ceph-mon[97033]: pgmap v6316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:55:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:03 smithi059 ceph-mon[97450]: pgmap v6316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:55:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:03.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:03.558831+0000) 2022-09-08T00:55:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:03.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:03.488403+0000) 2022-09-08T00:55:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:04.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:04.559017+0000) 2022-09-08T00:55:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:04.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:04.488691+0000) 2022-09-08T00:55:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:05 smithi055 ceph-mon[97033]: pgmap v6317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:55:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:55:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:05 smithi059 ceph-mon[97450]: pgmap v6317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:55:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:55:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:05.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:05.559212+0000) 2022-09-08T00:55:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:05.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:05.488949+0000) 2022-09-08T00:55:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:06.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:06.559390+0000) 2022-09-08T00:55:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:06.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:06.489163+0000) 2022-09-08T00:55:07.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:55:06.918+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:55:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:07 smithi055 ceph-mon[97033]: pgmap v6318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:55:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:07 smithi059 ceph-mon[97450]: pgmap v6318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:55:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:07.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:07.559501+0000) 2022-09-08T00:55:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:07.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:07.489375+0000) 2022-09-08T00:55:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:08.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:08.489578+0000) 2022-09-08T00:55:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:08.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:08.559696+0000) 2022-09-08T00:55:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:09 smithi055 ceph-mon[97033]: pgmap v6319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:55:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:09 smithi055 ceph-mon[97033]: osdmap e479: 12 total, 11 up, 11 in 2022-09-08T00:55:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:09 smithi059 ceph-mon[97450]: pgmap v6319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:55:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:09 smithi059 ceph-mon[97450]: osdmap e479: 12 total, 11 up, 11 in 2022-09-08T00:55:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:09.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:09.559943+0000) 2022-09-08T00:55:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:09.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:09.489789+0000) 2022-09-08T00:55:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:10.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:10.560127+0000) 2022-09-08T00:55:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:10.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:10.490031+0000) 2022-09-08T00:55:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:11 smithi055 ceph-mon[97033]: pgmap v6321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:55:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:11 smithi055 ceph-mon[97033]: osdmap e480: 12 total, 11 up, 11 in 2022-09-08T00:55:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:11 smithi059 ceph-mon[97450]: pgmap v6321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:55:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:11 smithi059 ceph-mon[97450]: osdmap e480: 12 total, 11 up, 11 in 2022-09-08T00:55:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:11.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:11.560306+0000) 2022-09-08T00:55:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:11.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:11.490294+0000) 2022-09-08T00:55:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:12.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:12.560515+0000) 2022-09-08T00:55:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:12.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:12.490500+0000) 2022-09-08T00:55:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:13 smithi055 ceph-mon[97033]: pgmap v6323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:55:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:13 smithi059 ceph-mon[97450]: pgmap v6323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:55:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:13.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:13.560729+0000) 2022-09-08T00:55:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:13.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:13.490747+0000) 2022-09-08T00:55:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:14.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:14.560935+0000) 2022-09-08T00:55:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:14.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:14.490921+0000) 2022-09-08T00:55:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:15 smithi055 ceph-mon[97033]: pgmap v6324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:55:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:15 smithi059 ceph-mon[97450]: pgmap v6324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:55:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:15.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:15.561235+0000) 2022-09-08T00:55:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:15.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:15.491087+0000) 2022-09-08T00:55:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:16.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:16.561519+0000) 2022-09-08T00:55:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:16.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:16.491291+0000) 2022-09-08T00:55:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:17 smithi055 ceph-mon[97033]: pgmap v6325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:55:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:17 smithi059 ceph-mon[97450]: pgmap v6325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:55:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:17.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:17.561766+0000) 2022-09-08T00:55:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:17.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:17.491498+0000) 2022-09-08T00:55:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:18.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:18.561957+0000) 2022-09-08T00:55:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:18.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:18.491725+0000) 2022-09-08T00:55:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:19 smithi055 ceph-mon[97033]: pgmap v6326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:55:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:19 smithi059 ceph-mon[97450]: pgmap v6326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:55:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:19.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:19.562161+0000) 2022-09-08T00:55:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:19.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:19.491997+0000) 2022-09-08T00:55:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:20.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:20.562324+0000) 2022-09-08T00:55:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:20.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:20.492229+0000) 2022-09-08T00:55:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:21 smithi055 ceph-mon[97033]: pgmap v6327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:55:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:21 smithi059 ceph-mon[97450]: pgmap v6327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:55:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:21.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:21.562543+0000) 2022-09-08T00:55:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:21.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:21.492458+0000) 2022-09-08T00:55:22.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:55:21.922+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:55:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:22.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:22.562797+0000) 2022-09-08T00:55:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:22.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:22.492719+0000) 2022-09-08T00:55:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:23 smithi055 ceph-mon[97033]: pgmap v6328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T00:55:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:23 smithi059 ceph-mon[97450]: pgmap v6328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T00:55:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:23.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:23.563002+0000) 2022-09-08T00:55:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:23.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:23.492891+0000) 2022-09-08T00:55:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:24.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:24.563188+0000) 2022-09-08T00:55:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:24.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:24.493108+0000) 2022-09-08T00:55:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:25 smithi055 ceph-mon[97033]: pgmap v6329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:55:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:25 smithi059 ceph-mon[97450]: pgmap v6329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:55:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:25.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:25.563413+0000) 2022-09-08T00:55:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:25.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:25.493321+0000) 2022-09-08T00:55:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:26.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:26.563692+0000) 2022-09-08T00:55:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:26.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:26.493548+0000) 2022-09-08T00:55:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:27 smithi055 ceph-mon[97033]: pgmap v6330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:55:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:27 smithi059 ceph-mon[97450]: pgmap v6330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:55:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:27.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:27.563866+0000) 2022-09-08T00:55:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:27.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:27.493790+0000) 2022-09-08T00:55:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:28.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:28.564079+0000) 2022-09-08T00:55:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:28.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:28.493930+0000) 2022-09-08T00:55:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:29 smithi055 ceph-mon[97033]: pgmap v6331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:55:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:29 smithi059 ceph-mon[97450]: pgmap v6331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:55:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:29.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:29.564224+0000) 2022-09-08T00:55:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:29.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:29.494150+0000) 2022-09-08T00:55:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:30.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:30.564457+0000) 2022-09-08T00:55:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:30.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:30.494333+0000) 2022-09-08T00:55:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:31 smithi055 ceph-mon[97033]: pgmap v6332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:55:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:31 smithi059 ceph-mon[97450]: pgmap v6332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:55:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:31.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:31.494545+0000) 2022-09-08T00:55:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:31.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:31.564669+0000) 2022-09-08T00:55:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:32.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:32.494752+0000) 2022-09-08T00:55:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:32.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:32.564847+0000) 2022-09-08T00:55:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:33 smithi055 ceph-mon[97033]: pgmap v6333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:55:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:33 smithi059 ceph-mon[97450]: pgmap v6333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:55:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:33.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:33.565012+0000) 2022-09-08T00:55:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:33.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:33.494933+0000) 2022-09-08T00:55:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:34.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:34.565257+0000) 2022-09-08T00:55:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:34.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:34.495177+0000) 2022-09-08T00:55:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:35 smithi055 ceph-mon[97033]: pgmap v6334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:55:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:35 smithi059 ceph-mon[97450]: pgmap v6334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T00:55:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:35.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:35.565523+0000) 2022-09-08T00:55:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:35.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:35.495438+0000) 2022-09-08T00:55:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:36.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:36.565755+0000) 2022-09-08T00:55:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:36.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:36.495651+0000) 2022-09-08T00:55:37.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:55:36.939+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:55:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:37 smithi055 ceph-mon[97033]: pgmap v6335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:55:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:37 smithi059 ceph-mon[97450]: pgmap v6335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:55:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:37.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:37.565894+0000) 2022-09-08T00:55:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:37.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:37.495832+0000) 2022-09-08T00:55:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:38.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:38.566132+0000) 2022-09-08T00:55:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:38.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:38.496065+0000) 2022-09-08T00:55:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:39 smithi055 ceph-mon[97033]: pgmap v6336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:39 smithi059 ceph-mon[97450]: pgmap v6336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:39.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:39.566381+0000) 2022-09-08T00:55:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:39.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:39.496326+0000) 2022-09-08T00:55:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:40.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:40.566598+0000) 2022-09-08T00:55:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:40.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:40.496472+0000) 2022-09-08T00:55:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:41 smithi055 ceph-mon[97033]: pgmap v6337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:55:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:41 smithi059 ceph-mon[97450]: pgmap v6337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:55:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:41.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:41.496730+0000) 2022-09-08T00:55:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:41.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:41.566884+0000) 2022-09-08T00:55:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:42.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:42.567195+0000) 2022-09-08T00:55:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:42.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:42.496964+0000) 2022-09-08T00:55:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:43 smithi055 ceph-mon[97033]: pgmap v6338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:43 smithi059 ceph-mon[97450]: pgmap v6338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:43.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:43.567457+0000) 2022-09-08T00:55:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:43.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:43.497199+0000) 2022-09-08T00:55:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:44.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:44.567719+0000) 2022-09-08T00:55:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:44.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:44.497384+0000) 2022-09-08T00:55:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:45 smithi055 ceph-mon[97033]: pgmap v6339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:45 smithi059 ceph-mon[97450]: pgmap v6339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:45.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:45.567933+0000) 2022-09-08T00:55:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:45.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:45.497581+0000) 2022-09-08T00:55:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:46.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:46.568165+0000) 2022-09-08T00:55:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:46.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:46.497811+0000) 2022-09-08T00:55:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:47 smithi055 ceph-mon[97033]: pgmap v6340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:55:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:47 smithi059 ceph-mon[97450]: pgmap v6340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:55:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:47.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:47.568388+0000) 2022-09-08T00:55:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:47.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:47.498101+0000) 2022-09-08T00:55:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:48.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:48.568580+0000) 2022-09-08T00:55:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:48.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:48.498338+0000) 2022-09-08T00:55:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:49 smithi055 ceph-mon[97033]: pgmap v6341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:49 smithi059 ceph-mon[97450]: pgmap v6341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:49.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:49.568795+0000) 2022-09-08T00:55:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:49.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:49.498578+0000) 2022-09-08T00:55:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:50.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:50.569064+0000) 2022-09-08T00:55:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:50.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:50.498771+0000) 2022-09-08T00:55:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:51 smithi055 ceph-mon[97033]: pgmap v6342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:55:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:51 smithi059 ceph-mon[97450]: pgmap v6342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:55:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:51.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:51.569359+0000) 2022-09-08T00:55:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:51.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:51.498943+0000) 2022-09-08T00:55:52.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:55:51.953+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:55:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:52.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:52.569553+0000) 2022-09-08T00:55:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:52.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:52.499107+0000) 2022-09-08T00:55:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:53 smithi055 ceph-mon[97033]: pgmap v6343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:55:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:55:53.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:55:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:53 smithi059 ceph-mon[97450]: pgmap v6343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:55:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:55:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:55:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:53.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:53.569729+0000) 2022-09-08T00:55:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:53.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:53.499339+0000) 2022-09-08T00:55:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:54.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:54.569977+0000) 2022-09-08T00:55:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:54.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:54.499520+0000) 2022-09-08T00:55:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:55 smithi055 ceph-mon[97033]: pgmap v6344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:55 smithi059 ceph-mon[97450]: pgmap v6344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:55.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:55.570259+0000) 2022-09-08T00:55:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:55.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:55.499760+0000) 2022-09-08T00:55:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:56.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:56.570504+0000) 2022-09-08T00:55:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:56.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:56.499947+0000) 2022-09-08T00:55:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:57 smithi055 ceph-mon[97033]: pgmap v6345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:55:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:57 smithi059 ceph-mon[97450]: pgmap v6345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:55:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:57.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:57.570759+0000) 2022-09-08T00:55:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:57.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:57.500158+0000) 2022-09-08T00:55:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:58.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:58.570993+0000) 2022-09-08T00:55:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:58.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:58.500297+0000) 2022-09-08T00:55:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:55:59 smithi055 ceph-mon[97033]: pgmap v6346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:55:59 smithi059 ceph-mon[97450]: pgmap v6346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:55:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:55:59.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:59.571202+0000) 2022-09-08T00:55:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:55:59.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:55:59.500500+0000) 2022-09-08T00:56:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:00.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:00.571486+0000) 2022-09-08T00:56:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:00.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:00.500748+0000) 2022-09-08T00:56:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:01 smithi055 ceph-mon[97033]: pgmap v6347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:56:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:01 smithi059 ceph-mon[97450]: pgmap v6347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:56:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:01.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:01.571752+0000) 2022-09-08T00:56:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:01.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:01.500925+0000) 2022-09-08T00:56:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:02.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:02.571937+0000) 2022-09-08T00:56:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:02.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:02.501142+0000) 2022-09-08T00:56:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:03 smithi055 ceph-mon[97033]: pgmap v6348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:03 smithi059 ceph-mon[97450]: pgmap v6348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:03.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:03.572128+0000) 2022-09-08T00:56:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:03.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:03.501444+0000) 2022-09-08T00:56:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:04.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:04.572331+0000) 2022-09-08T00:56:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:04.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:04.501694+0000) 2022-09-08T00:56:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:05 smithi055 ceph-mon[97033]: pgmap v6349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:56:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:56:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:05 smithi059 ceph-mon[97450]: pgmap v6349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:56:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:56:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:05.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:05.572485+0000) 2022-09-08T00:56:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:05.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:05.501917+0000) 2022-09-08T00:56:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:06.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:06.572734+0000) 2022-09-08T00:56:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:06.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:06.502116+0000) 2022-09-08T00:56:07.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:56:06.968+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:56:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:07 smithi055 ceph-mon[97033]: pgmap v6350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:56:07.539 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:07 smithi059 ceph-mon[97450]: pgmap v6350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:56:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:07.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:07.572952+0000) 2022-09-08T00:56:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:07.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:07.502311+0000) 2022-09-08T00:56:08.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:08.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:08.573204+0000) 2022-09-08T00:56:08.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:08.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:08.502547+0000) 2022-09-08T00:56:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:09 smithi055 ceph-mon[97033]: pgmap v6351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:56:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:09 smithi055 ceph-mon[97033]: osdmap e481: 12 total, 11 up, 11 in 2022-09-08T00:56:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:09 smithi059 ceph-mon[97450]: pgmap v6351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:56:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:09 smithi059 ceph-mon[97450]: osdmap e481: 12 total, 11 up, 11 in 2022-09-08T00:56:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:09.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:09.573439+0000) 2022-09-08T00:56:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:09.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:09.502742+0000) 2022-09-08T00:56:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:10.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:10.573691+0000) 2022-09-08T00:56:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:10.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:10.502902+0000) 2022-09-08T00:56:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:11 smithi055 ceph-mon[97033]: pgmap v6353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:56:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:11 smithi055 ceph-mon[97033]: osdmap e482: 12 total, 11 up, 11 in 2022-09-08T00:56:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:11 smithi059 ceph-mon[97450]: pgmap v6353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:56:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:11 smithi059 ceph-mon[97450]: osdmap e482: 12 total, 11 up, 11 in 2022-09-08T00:56:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:11.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:11.573932+0000) 2022-09-08T00:56:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:11.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:11.503058+0000) 2022-09-08T00:56:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:12.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:12.574129+0000) 2022-09-08T00:56:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:12.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:12.503293+0000) 2022-09-08T00:56:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:13 smithi055 ceph-mon[97033]: pgmap v6355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:56:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:13 smithi059 ceph-mon[97450]: pgmap v6355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:56:13.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:13.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:13.574395+0000) 2022-09-08T00:56:13.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:13.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:13.503534+0000) 2022-09-08T00:56:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:14.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:14.503779+0000) 2022-09-08T00:56:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:14.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:14.574602+0000) 2022-09-08T00:56:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:15 smithi055 ceph-mon[97033]: pgmap v6356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:56:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:15 smithi059 ceph-mon[97450]: pgmap v6356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:56:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:15.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:15.574822+0000) 2022-09-08T00:56:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:15.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:15.503913+0000) 2022-09-08T00:56:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:16.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:16.575015+0000) 2022-09-08T00:56:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:16.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:16.504131+0000) 2022-09-08T00:56:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:17 smithi055 ceph-mon[97033]: pgmap v6357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:56:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:17 smithi059 ceph-mon[97450]: pgmap v6357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:56:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:17.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:17.575240+0000) 2022-09-08T00:56:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:17.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:17.504341+0000) 2022-09-08T00:56:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:18.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:18.575470+0000) 2022-09-08T00:56:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:18.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:18.504590+0000) 2022-09-08T00:56:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:19 smithi055 ceph-mon[97033]: pgmap v6358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:56:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:19 smithi059 ceph-mon[97450]: pgmap v6358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T00:56:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:19.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:19.575731+0000) 2022-09-08T00:56:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:19.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:19.504811+0000) 2022-09-08T00:56:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:20.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:20.575920+0000) 2022-09-08T00:56:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:20.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:20.505045+0000) 2022-09-08T00:56:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:21 smithi055 ceph-mon[97033]: pgmap v6359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:56:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:21 smithi059 ceph-mon[97450]: pgmap v6359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:56:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:21.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:21.576136+0000) 2022-09-08T00:56:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:21.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:21.505238+0000) 2022-09-08T00:56:22.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:56:21.973+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:56:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:22.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:22.576299+0000) 2022-09-08T00:56:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:22.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:22.505474+0000) 2022-09-08T00:56:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:23.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:23.576527+0000) 2022-09-08T00:56:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:23.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:23.505746+0000) 2022-09-08T00:56:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:23 smithi055 ceph-mon[97033]: pgmap v6360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T00:56:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:23 smithi055 ceph-mon[97033]: pgmap v6361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T00:56:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:23 smithi055 ceph-mon[97033]: pgmap v6362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T00:56:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:56:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:23 smithi059 ceph-mon[97450]: pgmap v6360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T00:56:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:23 smithi059 ceph-mon[97450]: pgmap v6361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T00:56:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:23 smithi059 ceph-mon[97450]: pgmap v6362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T00:56:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:56:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:24.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:24.576765+0000) 2022-09-08T00:56:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:24.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:24.505928+0000) 2022-09-08T00:56:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:25.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:25.576978+0000) 2022-09-08T00:56:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:25.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:25.506140+0000) 2022-09-08T00:56:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:25 smithi055 ceph-mon[97033]: pgmap v6363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:56:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:25 smithi059 ceph-mon[97450]: pgmap v6363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:56:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:26.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:26.506345+0000) 2022-09-08T00:56:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:26.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:26.577154+0000) 2022-09-08T00:56:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:27 smithi055 ceph-mon[97033]: pgmap v6364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:56:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:27 smithi059 ceph-mon[97450]: pgmap v6364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:56:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:27.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:27.577281+0000) 2022-09-08T00:56:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:27.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:27.506575+0000) 2022-09-08T00:56:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:28.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:28.577500+0000) 2022-09-08T00:56:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:28.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:28.506804+0000) 2022-09-08T00:56:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:29.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:29.577725+0000) 2022-09-08T00:56:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:29.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:29.507029+0000) 2022-09-08T00:56:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:29 smithi055 ceph-mon[97033]: pgmap v6365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:56:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:29 smithi059 ceph-mon[97450]: pgmap v6365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:56:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:30.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:30.577940+0000) 2022-09-08T00:56:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:30.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:30.507239+0000) 2022-09-08T00:56:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:31.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:31.507414+0000) 2022-09-08T00:56:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:31.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:31.578156+0000) 2022-09-08T00:56:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:31 smithi055 ceph-mon[97033]: pgmap v6366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:56:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:31 smithi059 ceph-mon[97450]: pgmap v6366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:56:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:32.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:32.578380+0000) 2022-09-08T00:56:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:32.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:32.507669+0000) 2022-09-08T00:56:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:33.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:33.578601+0000) 2022-09-08T00:56:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:33.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:33.507891+0000) 2022-09-08T00:56:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:33 smithi055 ceph-mon[97033]: pgmap v6367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:56:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:33 smithi059 ceph-mon[97450]: pgmap v6367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:56:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:34.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:34.578794+0000) 2022-09-08T00:56:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:34.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:34.508162+0000) 2022-09-08T00:56:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:35.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:35.578994+0000) 2022-09-08T00:56:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:35.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:35.508382+0000) 2022-09-08T00:56:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:35 smithi055 ceph-mon[97033]: pgmap v6368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:35 smithi059 ceph-mon[97450]: pgmap v6368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:36.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:36.508658+0000) 2022-09-08T00:56:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:36.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:36.579178+0000) 2022-09-08T00:56:37.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:56:36.988+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:56:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:37 smithi055 ceph-mon[97033]: pgmap v6369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:56:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:37 smithi059 ceph-mon[97450]: pgmap v6369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:56:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:37.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:37.579398+0000) 2022-09-08T00:56:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:37.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:37.508860+0000) 2022-09-08T00:56:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:38.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:38.579681+0000) 2022-09-08T00:56:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:38.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:38.509048+0000) 2022-09-08T00:56:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:39.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:39.579923+0000) 2022-09-08T00:56:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:39.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:39.509280+0000) 2022-09-08T00:56:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:39 smithi055 ceph-mon[97033]: pgmap v6370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:39 smithi059 ceph-mon[97450]: pgmap v6370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:40.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:40.580123+0000) 2022-09-08T00:56:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:40.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:40.509468+0000) 2022-09-08T00:56:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:41.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:41.580325+0000) 2022-09-08T00:56:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:41.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:41.509679+0000) 2022-09-08T00:56:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:41 smithi055 ceph-mon[97033]: pgmap v6371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:56:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:41 smithi059 ceph-mon[97450]: pgmap v6371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:56:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:42.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:42.509898+0000) 2022-09-08T00:56:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:42.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:42.580560+0000) 2022-09-08T00:56:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:43.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:43.510122+0000) 2022-09-08T00:56:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:43.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:43.580801+0000) 2022-09-08T00:56:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:43 smithi055 ceph-mon[97033]: pgmap v6372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:43 smithi059 ceph-mon[97450]: pgmap v6372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:44.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:44.581015+0000) 2022-09-08T00:56:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:44.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:44.510351+0000) 2022-09-08T00:56:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:45.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:45.581206+0000) 2022-09-08T00:56:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:45.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:45.510588+0000) 2022-09-08T00:56:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:45 smithi055 ceph-mon[97033]: pgmap v6373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:45 smithi059 ceph-mon[97450]: pgmap v6373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:46.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:46.581377+0000) 2022-09-08T00:56:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:46.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:46.510806+0000) 2022-09-08T00:56:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:47 smithi055 ceph-mon[97033]: pgmap v6374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:56:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:47 smithi059 ceph-mon[97450]: pgmap v6374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:56:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:47.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:47.581620+0000) 2022-09-08T00:56:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:47.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:47.510988+0000) 2022-09-08T00:56:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:48.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:48.581827+0000) 2022-09-08T00:56:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:48.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:48.511155+0000) 2022-09-08T00:56:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:49.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:49.582021+0000) 2022-09-08T00:56:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:49.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:49.511330+0000) 2022-09-08T00:56:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:49 smithi055 ceph-mon[97033]: pgmap v6375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:49 smithi059 ceph-mon[97450]: pgmap v6375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:50.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:50.582244+0000) 2022-09-08T00:56:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:50.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:50.511560+0000) 2022-09-08T00:56:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:51.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:51.582470+0000) 2022-09-08T00:56:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:51.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:51.511723+0000) 2022-09-08T00:56:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:51 smithi055 ceph-mon[97033]: pgmap v6376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:56:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:51 smithi059 ceph-mon[97450]: pgmap v6376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:56:52.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:56:51.996+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:56:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:52.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:52.582743+0000) 2022-09-08T00:56:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:52.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:52.511950+0000) 2022-09-08T00:56:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:53.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:53.582974+0000) 2022-09-08T00:56:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:53.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:53.512128+0000) 2022-09-08T00:56:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:53 smithi055 ceph-mon[97033]: pgmap v6377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:53 smithi059 ceph-mon[97450]: pgmap v6377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:54.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:54.583168+0000) 2022-09-08T00:56:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:54.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:54.512381+0000) 2022-09-08T00:56:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:55.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:55.583420+0000) 2022-09-08T00:56:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:55.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:55.512577+0000) 2022-09-08T00:56:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:55 smithi055 ceph-mon[97033]: pgmap v6378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:55 smithi059 ceph-mon[97450]: pgmap v6378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:56:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:56.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:56.583692+0000) 2022-09-08T00:56:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:56.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:56.512820+0000) 2022-09-08T00:56:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:57 smithi055 ceph-mon[97033]: pgmap v6379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:56:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:57 smithi059 ceph-mon[97450]: pgmap v6379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:56:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:57.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:57.583901+0000) 2022-09-08T00:56:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:57.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:57.513063+0000) 2022-09-08T00:56:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:58.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:58.584130+0000) 2022-09-08T00:56:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:58.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:58.513298+0000) 2022-09-08T00:56:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:56:59.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:59.584320+0000) 2022-09-08T00:56:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:56:59.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:56:59.513466+0000) 2022-09-08T00:56:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:56:59 smithi055 ceph-mon[97033]: pgmap v6380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:00.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:56:59 smithi059 ceph-mon[97450]: pgmap v6380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:00.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:00.513581+0000) 2022-09-08T00:57:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:00.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:00.584472+0000) 2022-09-08T00:57:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:01.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:01.513809+0000) 2022-09-08T00:57:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:01.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:01.584597+0000) 2022-09-08T00:57:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:01 smithi055 ceph-mon[97033]: pgmap v6381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:57:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:01 smithi059 ceph-mon[97450]: pgmap v6381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:57:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:02.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:02.584850+0000) 2022-09-08T00:57:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:02.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:02.514056+0000) 2022-09-08T00:57:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:03.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:03.585138+0000) 2022-09-08T00:57:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:03.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:03.514253+0000) 2022-09-08T00:57:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:03 smithi055 ceph-mon[97033]: pgmap v6382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:03 smithi059 ceph-mon[97450]: pgmap v6382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:04.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:04.585296+0000) 2022-09-08T00:57:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:04.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:04.514503+0000) 2022-09-08T00:57:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:05.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:05.585492+0000) 2022-09-08T00:57:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:05.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:05.514703+0000) 2022-09-08T00:57:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:05 smithi055 ceph-mon[97033]: pgmap v6383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:57:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:57:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:05 smithi059 ceph-mon[97450]: pgmap v6383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:57:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:57:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:06.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:06.585676+0000) 2022-09-08T00:57:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:06.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:06.514861+0000) 2022-09-08T00:57:07.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:57:07.010+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:57:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:07 smithi055 ceph-mon[97033]: pgmap v6384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:57:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:07 smithi059 ceph-mon[97450]: pgmap v6384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:57:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:07.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:07.585948+0000) 2022-09-08T00:57:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:07.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:07.515034+0000) 2022-09-08T00:57:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:08.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:08.586097+0000) 2022-09-08T00:57:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:08.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:08.515161+0000) 2022-09-08T00:57:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:09 smithi055 ceph-mon[97033]: osdmap e483: 12 total, 11 up, 11 in 2022-09-08T00:57:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:09 smithi055 ceph-mon[97033]: pgmap v6386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:57:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:09 smithi059 ceph-mon[97450]: osdmap e483: 12 total, 11 up, 11 in 2022-09-08T00:57:09.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:09 smithi059 ceph-mon[97450]: pgmap v6386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:57:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:09.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:09.515317+0000) 2022-09-08T00:57:09.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:09.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:09.586246+0000) 2022-09-08T00:57:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:10.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:10.515555+0000) 2022-09-08T00:57:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:10.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:10.586448+0000) 2022-09-08T00:57:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:11.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:11.515721+0000) 2022-09-08T00:57:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:11.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:11.586717+0000) 2022-09-08T00:57:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:11 smithi055 ceph-mon[97033]: pgmap v6387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:57:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:11 smithi055 ceph-mon[97033]: osdmap e484: 12 total, 11 up, 11 in 2022-09-08T00:57:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:11 smithi059 ceph-mon[97450]: pgmap v6387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:57:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:11 smithi059 ceph-mon[97450]: osdmap e484: 12 total, 11 up, 11 in 2022-09-08T00:57:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:12.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:12.586991+0000) 2022-09-08T00:57:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:12.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:12.515968+0000) 2022-09-08T00:57:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:13.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:13.516123+0000) 2022-09-08T00:57:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:13.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:13.587126+0000) 2022-09-08T00:57:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:13 smithi055 ceph-mon[97033]: pgmap v6389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:57:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:13 smithi059 ceph-mon[97450]: pgmap v6389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T00:57:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:14.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:14.587304+0000) 2022-09-08T00:57:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:14.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:14.516358+0000) 2022-09-08T00:57:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:15.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:15.587560+0000) 2022-09-08T00:57:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:15.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:15.516575+0000) 2022-09-08T00:57:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:15 smithi055 ceph-mon[97033]: pgmap v6390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:57:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:15 smithi059 ceph-mon[97450]: pgmap v6390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:57:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:16.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:16.587748+0000) 2022-09-08T00:57:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:16.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:16.516774+0000) 2022-09-08T00:57:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:17 smithi055 ceph-mon[97033]: pgmap v6391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T00:57:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:17 smithi059 ceph-mon[97450]: pgmap v6391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T00:57:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:17.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:17.587969+0000) 2022-09-08T00:57:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:17.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:17.516992+0000) 2022-09-08T00:57:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:18.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:18.588178+0000) 2022-09-08T00:57:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:18.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:18.517150+0000) 2022-09-08T00:57:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:19.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:19.588451+0000) 2022-09-08T00:57:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:19.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:19.517336+0000) 2022-09-08T00:57:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:19 smithi055 ceph-mon[97033]: pgmap v6392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:57:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:19 smithi059 ceph-mon[97450]: pgmap v6392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:57:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:20.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:20.588596+0000) 2022-09-08T00:57:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:20.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:20.517564+0000) 2022-09-08T00:57:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:21.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:21.588743+0000) 2022-09-08T00:57:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:21.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:21.517782+0000) 2022-09-08T00:57:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:21 smithi055 ceph-mon[97033]: pgmap v6393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:57:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:21 smithi059 ceph-mon[97450]: pgmap v6393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:57:22.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:57:22.014+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:57:22.770 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:22.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:22.588906+0000) 2022-09-08T00:57:22.771 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:22.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:22.518011+0000) 2022-09-08T00:57:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:23.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:23.589150+0000) 2022-09-08T00:57:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:23.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:23.518240+0000) 2022-09-08T00:57:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:23 smithi055 ceph-mon[97033]: pgmap v6394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T00:57:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:57:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:57:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:57:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:23 smithi059 ceph-mon[97450]: pgmap v6394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T00:57:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:57:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:57:24.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:57:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:24.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:24.589356+0000) 2022-09-08T00:57:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:24.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:24.518457+0000) 2022-09-08T00:57:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:25.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:25.589521+0000) 2022-09-08T00:57:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:25.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:25.518616+0000) 2022-09-08T00:57:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:25 smithi055 ceph-mon[97033]: pgmap v6395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:57:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:25 smithi059 ceph-mon[97450]: pgmap v6395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T00:57:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:26.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:26.589755+0000) 2022-09-08T00:57:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:26.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:26.518862+0000) 2022-09-08T00:57:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:27 smithi055 ceph-mon[97033]: pgmap v6396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:57:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:27 smithi059 ceph-mon[97450]: pgmap v6396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:57:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:27.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:27.589974+0000) 2022-09-08T00:57:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:27.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:27.519110+0000) 2022-09-08T00:57:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:28.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:28.590220+0000) 2022-09-08T00:57:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:28.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:28.519324+0000) 2022-09-08T00:57:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:29.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:29.590386+0000) 2022-09-08T00:57:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:29.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:29.519606+0000) 2022-09-08T00:57:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:29 smithi055 ceph-mon[97033]: pgmap v6397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:29 smithi059 ceph-mon[97450]: pgmap v6397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:30.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:30.590570+0000) 2022-09-08T00:57:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:30.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:30.519881+0000) 2022-09-08T00:57:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:31.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:31.590755+0000) 2022-09-08T00:57:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:31.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:31.520142+0000) 2022-09-08T00:57:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:31 smithi055 ceph-mon[97033]: pgmap v6398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:57:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:31 smithi059 ceph-mon[97450]: pgmap v6398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:57:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:32.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:32.520371+0000) 2022-09-08T00:57:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:32.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:32.590990+0000) 2022-09-08T00:57:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:33.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:33.520682+0000) 2022-09-08T00:57:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:33.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:33.591206+0000) 2022-09-08T00:57:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:33 smithi055 ceph-mon[97033]: pgmap v6399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:33 smithi059 ceph-mon[97450]: pgmap v6399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:34.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:34.591419+0000) 2022-09-08T00:57:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:34.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:34.520831+0000) 2022-09-08T00:57:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:35.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:35.591710+0000) 2022-09-08T00:57:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:35.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:35.521022+0000) 2022-09-08T00:57:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:35 smithi055 ceph-mon[97033]: pgmap v6400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:35 smithi059 ceph-mon[97450]: pgmap v6400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:36.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:36.591915+0000) 2022-09-08T00:57:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:36.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:36.521219+0000) 2022-09-08T00:57:37.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:57:37.022+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:57:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:37 smithi055 ceph-mon[97033]: pgmap v6401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:57:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:37 smithi059 ceph-mon[97450]: pgmap v6401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:57:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:37.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:37.521393+0000) 2022-09-08T00:57:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:37.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:37.592091+0000) 2022-09-08T00:57:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:38.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:38.592285+0000) 2022-09-08T00:57:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:38.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:38.521657+0000) 2022-09-08T00:57:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:39.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:39.592449+0000) 2022-09-08T00:57:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:39.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:39.521833+0000) 2022-09-08T00:57:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:39 smithi055 ceph-mon[97033]: pgmap v6402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:39 smithi059 ceph-mon[97450]: pgmap v6402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:40.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:40.592748+0000) 2022-09-08T00:57:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:40.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:40.522005+0000) 2022-09-08T00:57:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:41.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:41.592971+0000) 2022-09-08T00:57:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:41.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:41.522177+0000) 2022-09-08T00:57:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:41 smithi055 ceph-mon[97033]: pgmap v6403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:57:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:41 smithi059 ceph-mon[97450]: pgmap v6403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:57:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:42.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:42.593185+0000) 2022-09-08T00:57:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:42.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:42.522410+0000) 2022-09-08T00:57:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:43.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:43.593348+0000) 2022-09-08T00:57:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:43.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:43.522655+0000) 2022-09-08T00:57:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:43 smithi055 ceph-mon[97033]: pgmap v6404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:44.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:43 smithi059 ceph-mon[97450]: pgmap v6404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:44.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:44.593527+0000) 2022-09-08T00:57:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:44.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:44.522886+0000) 2022-09-08T00:57:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:45.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:45.593761+0000) 2022-09-08T00:57:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:45.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:45.523050+0000) 2022-09-08T00:57:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:45 smithi055 ceph-mon[97033]: pgmap v6405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:45 smithi059 ceph-mon[97450]: pgmap v6405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:46.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:46.594026+0000) 2022-09-08T00:57:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:46.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:46.523250+0000) 2022-09-08T00:57:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:47 smithi059 ceph-mon[97450]: pgmap v6406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:57:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:47 smithi055 ceph-mon[97033]: pgmap v6406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:57:47.840 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:47.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:47.594270+0000) 2022-09-08T00:57:47.840 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:47.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:47.523431+0000) 2022-09-08T00:57:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:48.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:48.594575+0000) 2022-09-08T00:57:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:48.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:48.523560+0000) 2022-09-08T00:57:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:49.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:49.594858+0000) 2022-09-08T00:57:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:49.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:49.523778+0000) 2022-09-08T00:57:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:49 smithi055 ceph-mon[97033]: pgmap v6407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:49 smithi059 ceph-mon[97450]: pgmap v6407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:50.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:50.595115+0000) 2022-09-08T00:57:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:50.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:50.524004+0000) 2022-09-08T00:57:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:51.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:51.595317+0000) 2022-09-08T00:57:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:51.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:51.524194+0000) 2022-09-08T00:57:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:51 smithi055 ceph-mon[97033]: pgmap v6408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:57:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:51 smithi059 ceph-mon[97450]: pgmap v6408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:57:52.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:57:52.025+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:57:52.894 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:52.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:52.595524+0000) 2022-09-08T00:57:52.895 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:52.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:52.524375+0000) 2022-09-08T00:57:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:53.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:53.595762+0000) 2022-09-08T00:57:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:53.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:53.524588+0000) 2022-09-08T00:57:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:53 smithi055 ceph-mon[97033]: pgmap v6409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:53 smithi055 ceph-mon[97033]: pgmap v6410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:57:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:53 smithi055 ceph-mon[97033]: pgmap v6411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:57:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:57:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:53 smithi059 ceph-mon[97450]: pgmap v6409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:57:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:53 smithi059 ceph-mon[97450]: pgmap v6410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T00:57:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:53 smithi059 ceph-mon[97450]: pgmap v6411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:57:54.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:57:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:54.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:54.596001+0000) 2022-09-08T00:57:54.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:54.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:54.524817+0000) 2022-09-08T00:57:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:55.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:55.596192+0000) 2022-09-08T00:57:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:55.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:55.525019+0000) 2022-09-08T00:57:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:55 smithi055 ceph-mon[97033]: pgmap v6412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:57:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:55 smithi059 ceph-mon[97450]: pgmap v6412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:57:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:56.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:56.596417+0000) 2022-09-08T00:57:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:56.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:56.525240+0000) 2022-09-08T00:57:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:57 smithi055 ceph-mon[97033]: pgmap v6413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:57:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:57 smithi059 ceph-mon[97450]: pgmap v6413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:57:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:57.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:57.596586+0000) 2022-09-08T00:57:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:57.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:57.525487+0000) 2022-09-08T00:57:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:58.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:58.596789+0000) 2022-09-08T00:57:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:58.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:58.525710+0000) 2022-09-08T00:57:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:57:58 smithi055 ceph-mon[97033]: pgmap v6414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:57:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:57:58 smithi059 ceph-mon[97450]: pgmap v6414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T00:57:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:57:59.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:59.596944+0000) 2022-09-08T00:57:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:57:59.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:57:59.525922+0000) 2022-09-08T00:58:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:00.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:00.526068+0000) 2022-09-08T00:58:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:00.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:00.597097+0000) 2022-09-08T00:58:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:00 smithi055 ceph-mon[97033]: pgmap v6415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:58:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:00 smithi059 ceph-mon[97450]: pgmap v6415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T00:58:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:01.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:01.597263+0000) 2022-09-08T00:58:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:01.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:01.526290+0000) 2022-09-08T00:58:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:02.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:02.597490+0000) 2022-09-08T00:58:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:02.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:02.526544+0000) 2022-09-08T00:58:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:02 smithi055 ceph-mon[97033]: pgmap v6416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:58:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:02 smithi059 ceph-mon[97450]: pgmap v6416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T00:58:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:03.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:03.597787+0000) 2022-09-08T00:58:03.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:03.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:03.526749+0000) 2022-09-08T00:58:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:04.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:04.598030+0000) 2022-09-08T00:58:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:04.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:04.526952+0000) 2022-09-08T00:58:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:04 smithi055 ceph-mon[97033]: pgmap v6417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:58:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:58:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:58:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:04 smithi059 ceph-mon[97450]: pgmap v6417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:58:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:58:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:58:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:05.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:05.527202+0000) 2022-09-08T00:58:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:05.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:05.598294+0000) 2022-09-08T00:58:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:06.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:06.598588+0000) 2022-09-08T00:58:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:06.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:06.527489+0000) 2022-09-08T00:58:07.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:58:07.036+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:58:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:07 smithi055 ceph-mon[97033]: pgmap v6418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:58:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:07 smithi059 ceph-mon[97450]: pgmap v6418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:58:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:07.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:07.598896+0000) 2022-09-08T00:58:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:07.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:07.527706+0000) 2022-09-08T00:58:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:08.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:08.599147+0000) 2022-09-08T00:58:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:08.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:08.527852+0000) 2022-09-08T00:58:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:09 smithi055 ceph-mon[97033]: osdmap e485: 12 total, 11 up, 11 in 2022-09-08T00:58:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:09 smithi055 ceph-mon[97033]: pgmap v6420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:58:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:09 smithi059 ceph-mon[97450]: osdmap e485: 12 total, 11 up, 11 in 2022-09-08T00:58:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:09 smithi059 ceph-mon[97450]: pgmap v6420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:58:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:09.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:09.599329+0000) 2022-09-08T00:58:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:09.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:09.528122+0000) 2022-09-08T00:58:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:10.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:10.599573+0000) 2022-09-08T00:58:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:10.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:10.528401+0000) 2022-09-08T00:58:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:10 smithi055 ceph-mon[97033]: pgmap v6421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:58:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:10 smithi059 ceph-mon[97450]: pgmap v6421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:58:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:11.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:11.599792+0000) 2022-09-08T00:58:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:11.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:11.528583+0000) 2022-09-08T00:58:12.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:12 smithi055 ceph-mon[97033]: osdmap e486: 12 total, 11 up, 11 in 2022-09-08T00:58:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:12 smithi059 ceph-mon[97450]: osdmap e486: 12 total, 11 up, 11 in 2022-09-08T00:58:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:12.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:12.599984+0000) 2022-09-08T00:58:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:12.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:12.528805+0000) 2022-09-08T00:58:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:13 smithi059 ceph-mon[97450]: pgmap v6423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:58:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:13 smithi055 ceph-mon[97033]: pgmap v6423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:58:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:13.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:13.600202+0000) 2022-09-08T00:58:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:13.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:13.529013+0000) 2022-09-08T00:58:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:14.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:14.600392+0000) 2022-09-08T00:58:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:14.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:14.529299+0000) 2022-09-08T00:58:15.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:14 smithi055 ceph-mon[97033]: pgmap v6424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:58:15.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:14 smithi059 ceph-mon[97450]: pgmap v6424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:58:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:15.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:15.600656+0000) 2022-09-08T00:58:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:15.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:15.529538+0000) 2022-09-08T00:58:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:16.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:16.600809+0000) 2022-09-08T00:58:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:16.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:16.529739+0000) 2022-09-08T00:58:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:17 smithi059 ceph-mon[97450]: pgmap v6425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-08T00:58:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:17 smithi055 ceph-mon[97033]: pgmap v6425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-08T00:58:17.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:17.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:17.601039+0000) 2022-09-08T00:58:17.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:17.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:17.529885+0000) 2022-09-08T00:58:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:18.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:18.530101+0000) 2022-09-08T00:58:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:18.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:18.601243+0000) 2022-09-08T00:58:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:18 smithi055 ceph-mon[97033]: pgmap v6426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:58:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:18 smithi059 ceph-mon[97450]: pgmap v6426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:58:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:19.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:19.601484+0000) 2022-09-08T00:58:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:19.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:19.530369+0000) 2022-09-08T00:58:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:20.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:20.601785+0000) 2022-09-08T00:58:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:20.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:20.530592+0000) 2022-09-08T00:58:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:20 smithi055 ceph-mon[97033]: pgmap v6427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:58:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:20 smithi059 ceph-mon[97450]: pgmap v6427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:58:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:21.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:21.530828+0000) 2022-09-08T00:58:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:21.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:21.602046+0000) 2022-09-08T00:58:22.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:58:22.051+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:58:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:22.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:22.602290+0000) 2022-09-08T00:58:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:22.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:22.531035+0000) 2022-09-08T00:58:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:22 smithi055 ceph-mon[97033]: pgmap v6428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 87 B/s wr, 5 op/s 2022-09-08T00:58:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:22 smithi059 ceph-mon[97450]: pgmap v6428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 87 B/s wr, 5 op/s 2022-09-08T00:58:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:23.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:23.602490+0000) 2022-09-08T00:58:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:23.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:23.531272+0000) 2022-09-08T00:58:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:24.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:24.602737+0000) 2022-09-08T00:58:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:24.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:24.531483+0000) 2022-09-08T00:58:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:24 smithi055 ceph-mon[97033]: pgmap v6429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:58:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:24 smithi059 ceph-mon[97450]: pgmap v6429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:58:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:25.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:25.602974+0000) 2022-09-08T00:58:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:25.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:25.531719+0000) 2022-09-08T00:58:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:26.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:26.603193+0000) 2022-09-08T00:58:26.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:26.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:26.531862+0000) 2022-09-08T00:58:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:27 smithi059 ceph-mon[97450]: pgmap v6430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:27 smithi055 ceph-mon[97033]: pgmap v6430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:27.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:27.603427+0000) 2022-09-08T00:58:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:27.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:27.532088+0000) 2022-09-08T00:58:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:28.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:28.603656+0000) 2022-09-08T00:58:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:28.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:28.532242+0000) 2022-09-08T00:58:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:28 smithi055 ceph-mon[97033]: pgmap v6431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:28 smithi059 ceph-mon[97450]: pgmap v6431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:29.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:29.603891+0000) 2022-09-08T00:58:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:29.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:29.532462+0000) 2022-09-08T00:58:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:30.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:30.604128+0000) 2022-09-08T00:58:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:30.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:30.532716+0000) 2022-09-08T00:58:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:30 smithi055 ceph-mon[97033]: pgmap v6432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:58:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:30 smithi059 ceph-mon[97450]: pgmap v6432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:58:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:31.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:31.604354+0000) 2022-09-08T00:58:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:31.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:31.532929+0000) 2022-09-08T00:58:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:32.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:32.604584+0000) 2022-09-08T00:58:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:32.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:32.533172+0000) 2022-09-08T00:58:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:32 smithi055 ceph-mon[97033]: pgmap v6433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:32 smithi059 ceph-mon[97450]: pgmap v6433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:33.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:33.604835+0000) 2022-09-08T00:58:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:33.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:33.533415+0000) 2022-09-08T00:58:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:34.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:34.605097+0000) 2022-09-08T00:58:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:34.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:34.533648+0000) 2022-09-08T00:58:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:34 smithi055 ceph-mon[97033]: pgmap v6434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:58:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:34 smithi059 ceph-mon[97450]: pgmap v6434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:58:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:35.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:35.605337+0000) 2022-09-08T00:58:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:35.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:35.533818+0000) 2022-09-08T00:58:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:36.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:36.605597+0000) 2022-09-08T00:58:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:36.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:36.534074+0000) 2022-09-08T00:58:37.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:58:37.067+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:58:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:37 smithi059 ceph-mon[97450]: pgmap v6435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:37 smithi055 ceph-mon[97033]: pgmap v6435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:37.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:37.605828+0000) 2022-09-08T00:58:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:37.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:37.534246+0000) 2022-09-08T00:58:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:38.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:38.606063+0000) 2022-09-08T00:58:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:38.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:38.534416+0000) 2022-09-08T00:58:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:38 smithi055 ceph-mon[97033]: pgmap v6436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:38 smithi059 ceph-mon[97450]: pgmap v6436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:39.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:39.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:39.606284+0000) 2022-09-08T00:58:39.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:39.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:39.534594+0000) 2022-09-08T00:58:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:40.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:40.606532+0000) 2022-09-08T00:58:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:40.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:40.534822+0000) 2022-09-08T00:58:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:40 smithi055 ceph-mon[97033]: pgmap v6437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:58:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:40 smithi059 ceph-mon[97450]: pgmap v6437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:58:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:41.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:41.606693+0000) 2022-09-08T00:58:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:41.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:41.535091+0000) 2022-09-08T00:58:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:42.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:42.606899+0000) 2022-09-08T00:58:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:42.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:42.535270+0000) 2022-09-08T00:58:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:42 smithi055 ceph-mon[97033]: pgmap v6438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:42 smithi059 ceph-mon[97450]: pgmap v6438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:43.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:43.607084+0000) 2022-09-08T00:58:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:43.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:43.535512+0000) 2022-09-08T00:58:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:44.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:44.535757+0000) 2022-09-08T00:58:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:44.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:44.607276+0000) 2022-09-08T00:58:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:44 smithi055 ceph-mon[97033]: pgmap v6439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:58:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:44 smithi059 ceph-mon[97450]: pgmap v6439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:58:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:45.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:45.607443+0000) 2022-09-08T00:58:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:45.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:45.535932+0000) 2022-09-08T00:58:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:46.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:46.607706+0000) 2022-09-08T00:58:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:46.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:46.536187+0000) 2022-09-08T00:58:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:47 smithi059 ceph-mon[97450]: pgmap v6440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:47 smithi055 ceph-mon[97033]: pgmap v6440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:47.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:47.607914+0000) 2022-09-08T00:58:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:47.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:47.536391+0000) 2022-09-08T00:58:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:48.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:48.608115+0000) 2022-09-08T00:58:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:48.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:48.536609+0000) 2022-09-08T00:58:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:48 smithi059 ceph-mon[97450]: pgmap v6441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:48 smithi055 ceph-mon[97033]: pgmap v6441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:49.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:49.608347+0000) 2022-09-08T00:58:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:49.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:49.536885+0000) 2022-09-08T00:58:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:50.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:50.608582+0000) 2022-09-08T00:58:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:50.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:50.537124+0000) 2022-09-08T00:58:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:50 smithi055 ceph-mon[97033]: pgmap v6442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:58:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:50 smithi059 ceph-mon[97450]: pgmap v6442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:58:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:51.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:51.608817+0000) 2022-09-08T00:58:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:51.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:51.537285+0000) 2022-09-08T00:58:52.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:58:52.076+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:58:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:52.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:52.609080+0000) 2022-09-08T00:58:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:52.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:52.537540+0000) 2022-09-08T00:58:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:52 smithi055 ceph-mon[97033]: pgmap v6443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:52 smithi059 ceph-mon[97450]: pgmap v6443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:53.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:53.609299+0000) 2022-09-08T00:58:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:53.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:53.537765+0000) 2022-09-08T00:58:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:58:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:58:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:58:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T00:58:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T00:58:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T00:58:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:54.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:54.609508+0000) 2022-09-08T00:58:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:54.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:54.538019+0000) 2022-09-08T00:58:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:54 smithi059 ceph-mon[97450]: pgmap v6444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:58:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:54 smithi055 ceph-mon[97033]: pgmap v6444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:58:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:55.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:55.609743+0000) 2022-09-08T00:58:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:55.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:55.538258+0000) 2022-09-08T00:58:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:56.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:56.538464+0000) 2022-09-08T00:58:56.909 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:56.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:56.609963+0000) 2022-09-08T00:58:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:57 smithi059 ceph-mon[97450]: pgmap v6445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:57 smithi055 ceph-mon[97033]: pgmap v6445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:57.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:57.610211+0000) 2022-09-08T00:58:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:57.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:57.538709+0000) 2022-09-08T00:58:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:58.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:58.610373+0000) 2022-09-08T00:58:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:58.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:58.538946+0000) 2022-09-08T00:58:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:58:58 smithi055 ceph-mon[97033]: pgmap v6446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:58:58 smithi059 ceph-mon[97450]: pgmap v6446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:58:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:58:59.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:59.610604+0000) 2022-09-08T00:58:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:58:59.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:58:59.539124+0000) 2022-09-08T00:59:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:00.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:00.610884+0000) 2022-09-08T00:59:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:00.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:00.539298+0000) 2022-09-08T00:59:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:00 smithi055 ceph-mon[97033]: pgmap v6447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:59:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:00 smithi059 ceph-mon[97450]: pgmap v6447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:59:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:01.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:01.611120+0000) 2022-09-08T00:59:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:01.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:01.539520+0000) 2022-09-08T00:59:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:02.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:02.611343+0000) 2022-09-08T00:59:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:02.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:02.539688+0000) 2022-09-08T00:59:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:02 smithi055 ceph-mon[97033]: pgmap v6448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:59:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:02 smithi059 ceph-mon[97450]: pgmap v6448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:59:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:03.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:03.611561+0000) 2022-09-08T00:59:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:03.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:03.539940+0000) 2022-09-08T00:59:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:04.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:04.611707+0000) 2022-09-08T00:59:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:04.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:04.540162+0000) 2022-09-08T00:59:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:04 smithi055 ceph-mon[97033]: pgmap v6449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:59:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:59:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:59:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:04 smithi059 ceph-mon[97450]: pgmap v6449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T00:59:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T00:59:05.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T00:59:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:05.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:05.611841+0000) 2022-09-08T00:59:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:05.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:05.540429+0000) 2022-09-08T00:59:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:06.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:06.612051+0000) 2022-09-08T00:59:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:06.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:06.540669+0000) 2022-09-08T00:59:07.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:59:07.086+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:59:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:07 smithi059 ceph-mon[97450]: pgmap v6450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:59:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:07 smithi055 ceph-mon[97033]: pgmap v6450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T00:59:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:07.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:07.612245+0000) 2022-09-08T00:59:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:07.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:07.540886+0000) 2022-09-08T00:59:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:08.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:08.612433+0000) 2022-09-08T00:59:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:08.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:08.541054+0000) 2022-09-08T00:59:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:09 smithi055 ceph-mon[97033]: osdmap e487: 12 total, 11 up, 11 in 2022-09-08T00:59:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:09 smithi055 ceph-mon[97033]: pgmap v6452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:59:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:09 smithi059 ceph-mon[97450]: osdmap e487: 12 total, 11 up, 11 in 2022-09-08T00:59:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:09 smithi059 ceph-mon[97450]: pgmap v6452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:59:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:09.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:09.612567+0000) 2022-09-08T00:59:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:09.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:09.541230+0000) 2022-09-08T00:59:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:10.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:10.612825+0000) 2022-09-08T00:59:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:10.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:10.541428+0000) 2022-09-08T00:59:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:10 smithi059 ceph-mon[97450]: pgmap v6453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:59:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:10 smithi055 ceph-mon[97033]: pgmap v6453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T00:59:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:11.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:11.613089+0000) 2022-09-08T00:59:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:11.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:11.541673+0000) 2022-09-08T00:59:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:12 smithi059 ceph-mon[97450]: osdmap e488: 12 total, 11 up, 11 in 2022-09-08T00:59:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:12 smithi055 ceph-mon[97033]: osdmap e488: 12 total, 11 up, 11 in 2022-09-08T00:59:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:12.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:12.541886+0000) 2022-09-08T00:59:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:12.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:12.613323+0000) 2022-09-08T00:59:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:13 smithi059 ceph-mon[97450]: pgmap v6455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:59:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:13 smithi055 ceph-mon[97033]: pgmap v6455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T00:59:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:13.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:13.542074+0000) 2022-09-08T00:59:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:13.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:13.613473+0000) 2022-09-08T00:59:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:14.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:14.613760+0000) 2022-09-08T00:59:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:14.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:14.542296+0000) 2022-09-08T00:59:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:14 smithi059 ceph-mon[97450]: pgmap v6456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:59:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:14 smithi055 ceph-mon[97033]: pgmap v6456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T00:59:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:15.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:15.614029+0000) 2022-09-08T00:59:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:15.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:15.542471+0000) 2022-09-08T00:59:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:16.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:16.614245+0000) 2022-09-08T00:59:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:16.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:16.542660+0000) 2022-09-08T00:59:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:17 smithi059 ceph-mon[97450]: pgmap v6457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-08T00:59:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:17 smithi055 ceph-mon[97033]: pgmap v6457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-08T00:59:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:17.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:17.614457+0000) 2022-09-08T00:59:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:17.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:17.542880+0000) 2022-09-08T00:59:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:18.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:18.614672+0000) 2022-09-08T00:59:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:18.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:18.543143+0000) 2022-09-08T00:59:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:18 smithi059 ceph-mon[97450]: pgmap v6458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:59:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:18 smithi055 ceph-mon[97033]: pgmap v6458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T00:59:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:19.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:19.614877+0000) 2022-09-08T00:59:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:19.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:19.543375+0000) 2022-09-08T00:59:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:20.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:20.543598+0000) 2022-09-08T00:59:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:20.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:20.615045+0000) 2022-09-08T00:59:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:20 smithi059 ceph-mon[97450]: pgmap v6459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:59:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:20 smithi055 ceph-mon[97033]: pgmap v6459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T00:59:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:21.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:21.615258+0000) 2022-09-08T00:59:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:21.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:21.543844+0000) 2022-09-08T00:59:22.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:59:22.101+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:59:22.799 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:22.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:22.615449+0000) 2022-09-08T00:59:22.800 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:22.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:22.543975+0000) 2022-09-08T00:59:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:22 smithi059 ceph-mon[97450]: pgmap v6460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 87 B/s wr, 5 op/s 2022-09-08T00:59:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:22 smithi055 ceph-mon[97033]: pgmap v6460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 87 B/s wr, 5 op/s 2022-09-08T00:59:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:23.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:23.615656+0000) 2022-09-08T00:59:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:23.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:23.544174+0000) 2022-09-08T00:59:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:24.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:24.615803+0000) 2022-09-08T00:59:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:24.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:24.544383+0000) 2022-09-08T00:59:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:24 smithi059 ceph-mon[97450]: pgmap v6461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:59:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:24 smithi055 ceph-mon[97033]: pgmap v6461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T00:59:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:25.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:25.615969+0000) 2022-09-08T00:59:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:25.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:25.544665+0000) 2022-09-08T00:59:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:26.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:26.616179+0000) 2022-09-08T00:59:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:26.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:26.544871+0000) 2022-09-08T00:59:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:27 smithi059 ceph-mon[97450]: pgmap v6462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:59:27.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:27 smithi055 ceph-mon[97033]: pgmap v6462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:59:27.911 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:27.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:27.616409+0000) 2022-09-08T00:59:27.912 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:27.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:27.545088+0000) 2022-09-08T00:59:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:28.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:28.616678+0000) 2022-09-08T00:59:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:28.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:28.545285+0000) 2022-09-08T00:59:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:59:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:59:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:28 smithi059 ceph-mon[97450]: pgmap v6463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:59:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:59:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:59:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:59:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:28 smithi055 ceph-mon[97033]: pgmap v6463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:59:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T00:59:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:29.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:29.616934+0000) 2022-09-08T00:59:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:29.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:29.545486+0000) 2022-09-08T00:59:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:30.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:30.617190+0000) 2022-09-08T00:59:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:30.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:30.545664+0000) 2022-09-08T00:59:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:30 smithi059 ceph-mon[97450]: pgmap v6464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:59:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:30 smithi055 ceph-mon[97033]: pgmap v6464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:59:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:31.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:31.617423+0000) 2022-09-08T00:59:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:31.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:31.545875+0000) 2022-09-08T00:59:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:32 smithi059 ceph-mon[97450]: pgmap v6465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:59:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:32 smithi055 ceph-mon[97033]: pgmap v6465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:59:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:32.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:32.617669+0000) 2022-09-08T00:59:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:32.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:32.546013+0000) 2022-09-08T00:59:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:33.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:33.546228+0000) 2022-09-08T00:59:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:33.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:33.617840+0000) 2022-09-08T00:59:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:34.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:34.618057+0000) 2022-09-08T00:59:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:34.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:34.546480+0000) 2022-09-08T00:59:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:34 smithi055 ceph-mon[97033]: pgmap v6466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:59:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:34 smithi059 ceph-mon[97450]: pgmap v6466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:59:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:35.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:35.618224+0000) 2022-09-08T00:59:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:35.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:35.546744+0000) 2022-09-08T00:59:36.914 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:36.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:36.618410+0000) 2022-09-08T00:59:36.915 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:36.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:36.546996+0000) 2022-09-08T00:59:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:36 smithi055 ceph-mon[97033]: pgmap v6467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:59:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:36 smithi059 ceph-mon[97450]: pgmap v6467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:59:37.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:59:37.117+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:59:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:37.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:37.618670+0000) 2022-09-08T00:59:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:37.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:37.547212+0000) 2022-09-08T00:59:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:38.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:38.618810+0000) 2022-09-08T00:59:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:38.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:38.547476+0000) 2022-09-08T00:59:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:38 smithi055 ceph-mon[97033]: pgmap v6468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:59:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:38 smithi059 ceph-mon[97450]: pgmap v6468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T00:59:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:39.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:39.618958+0000) 2022-09-08T00:59:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:39.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:39.547733+0000) 2022-09-08T00:59:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:40.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:40.619188+0000) 2022-09-08T00:59:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:40.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:40.547903+0000) 2022-09-08T00:59:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:40 smithi055 ceph-mon[97033]: pgmap v6469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:59:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:40 smithi059 ceph-mon[97450]: pgmap v6469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:59:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:41.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:41.619400+0000) 2022-09-08T00:59:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:41.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:41.548070+0000) 2022-09-08T00:59:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:42 smithi059 ceph-mon[97450]: pgmap v6470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:59:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:42 smithi055 ceph-mon[97033]: pgmap v6470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:59:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:42.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:42.619595+0000) 2022-09-08T00:59:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:42.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:42.548206+0000) 2022-09-08T00:59:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:43.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:43.619854+0000) 2022-09-08T00:59:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:43.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:43.548405+0000) 2022-09-08T00:59:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:44.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:44.620098+0000) 2022-09-08T00:59:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:44.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:44.548665+0000) 2022-09-08T00:59:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:44 smithi055 ceph-mon[97033]: pgmap v6471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:59:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:44 smithi059 ceph-mon[97450]: pgmap v6471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:59:45.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:45.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:45.620327+0000) 2022-09-08T00:59:45.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:45.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:45.548786+0000) 2022-09-08T00:59:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:46.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:46.620596+0000) 2022-09-08T00:59:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:46.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:46.549020+0000) 2022-09-08T00:59:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:46 smithi055 ceph-mon[97033]: pgmap v6472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:59:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:46 smithi059 ceph-mon[97450]: pgmap v6472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:59:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:47.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:47.620781+0000) 2022-09-08T00:59:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:47.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:47.549256+0000) 2022-09-08T00:59:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:48.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:48.621017+0000) 2022-09-08T00:59:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:48.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:48.549485+0000) 2022-09-08T00:59:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:48 smithi055 ceph-mon[97033]: pgmap v6473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:59:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:48 smithi059 ceph-mon[97450]: pgmap v6473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:59:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:49.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:49.621296+0000) 2022-09-08T00:59:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:49.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:49.549733+0000) 2022-09-08T00:59:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:50.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:50.621604+0000) 2022-09-08T00:59:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:50.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:50.549981+0000) 2022-09-08T00:59:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:50 smithi055 ceph-mon[97033]: pgmap v6474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T00:59:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:50 smithi059 ceph-mon[97450]: pgmap v6474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T00:59:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:51.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:51.621864+0000) 2022-09-08T00:59:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:51.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:51.550290+0000) 2022-09-08T00:59:52.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 00:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T00:59:52.118+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T00:59:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:52 smithi059 ceph-mon[97450]: pgmap v6475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:59:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:52 smithi055 ceph-mon[97033]: pgmap v6475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:59:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:52.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:52.622060+0000) 2022-09-08T00:59:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:52.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:52.550558+0000) 2022-09-08T00:59:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:53.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:53.622328+0000) 2022-09-08T00:59:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:53.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:53.550774+0000) 2022-09-08T00:59:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:54.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:54.622586+0000) 2022-09-08T00:59:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:54.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:54.550946+0000) 2022-09-08T00:59:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:54 smithi055 ceph-mon[97033]: pgmap v6476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:59:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:54 smithi059 ceph-mon[97450]: pgmap v6476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:59:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:55.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:55.622821+0000) 2022-09-08T00:59:55.945 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:55.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:55.551184+0000) 2022-09-08T00:59:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:56.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:56.622999+0000) 2022-09-08T00:59:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:56.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:56.551419+0000) 2022-09-08T00:59:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:56 smithi055 ceph-mon[97033]: pgmap v6477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:59:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:56 smithi059 ceph-mon[97450]: pgmap v6477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T00:59:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:57.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:57.623238+0000) 2022-09-08T00:59:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:57.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:57.551647+0000) 2022-09-08T00:59:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:58.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:58.623451+0000) 2022-09-08T00:59:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:58.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:58.551832+0000) 2022-09-08T00:59:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 00:59:58 smithi055 ceph-mon[97033]: pgmap v6478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:59:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 00:59:58 smithi059 ceph-mon[97450]: pgmap v6478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T00:59:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 00:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T00:59:59.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:59.623731+0000) 2022-09-08T00:59:59.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 00:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T00:59:59.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-07T23:59:59.552072+0000) 2022-09-08T01:00:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:00.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:00.624022+0000) 2022-09-08T01:00:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:00.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:00.552350+0000) 2022-09-08T01:00:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:00 smithi055 ceph-mon[97033]: pgmap v6479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:00:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T01:00:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T01:00:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: ['Unable to reach remote host smithi047. '] 2022-09-08T01:00:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T01:00:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T01:00:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:00 smithi059 ceph-mon[97450]: pgmap v6479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:00:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T01:00:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T01:00:01.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: ['Unable to reach remote host smithi047. '] 2022-09-08T01:00:01.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T01:00:01.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T01:00:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:01.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:01.624259+0000) 2022-09-08T01:00:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:01.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:01.552568+0000) 2022-09-08T01:00:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:02 smithi059 ceph-mon[97450]: pgmap v6480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T01:00:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:02 smithi055 ceph-mon[97033]: pgmap v6480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T01:00:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:02.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:02.624483+0000) 2022-09-08T01:00:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:02.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:02.552771+0000) 2022-09-08T01:00:03.912 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:03.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:03.624694+0000) 2022-09-08T01:00:03.914 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:03.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:03.552980+0000) 2022-09-08T01:00:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:04.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:04.624932+0000) 2022-09-08T01:00:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:04.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:04.553235+0000) 2022-09-08T01:00:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:04 smithi055 ceph-mon[97033]: pgmap v6481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:00:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:04 smithi059 ceph-mon[97450]: pgmap v6481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:00:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:05.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:05.553452+0000) 2022-09-08T01:00:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:05.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:05.625150+0000) 2022-09-08T01:00:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:00:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:00:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:00:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:00:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:06.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:06.625409+0000) 2022-09-08T01:00:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:06.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:06.553750+0000) 2022-09-08T01:00:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:06 smithi055 ceph-mon[97033]: pgmap v6482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:00:07.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:06 smithi059 ceph-mon[97450]: pgmap v6482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:00:07.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:00:07.133+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:00:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:07.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:07.625681+0000) 2022-09-08T01:00:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:07.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:07.553917+0000) 2022-09-08T01:00:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:08.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:08.625900+0000) 2022-09-08T01:00:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:08.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:08.554102+0000) 2022-09-08T01:00:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:08 smithi055 ceph-mon[97033]: pgmap v6483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:00:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:08 smithi059 ceph-mon[97450]: pgmap v6483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:00:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:09.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:09.626135+0000) 2022-09-08T01:00:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:09.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:09.554329+0000) 2022-09-08T01:00:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:09 smithi055 ceph-mon[97033]: osdmap e489: 12 total, 11 up, 11 in 2022-09-08T01:00:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:09 smithi059 ceph-mon[97450]: osdmap e489: 12 total, 11 up, 11 in 2022-09-08T01:00:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:10.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:10.626366+0000) 2022-09-08T01:00:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:10.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:10.554523+0000) 2022-09-08T01:00:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:10 smithi055 ceph-mon[97033]: pgmap v6485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:00:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:10 smithi059 ceph-mon[97450]: pgmap v6485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:00:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:11.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:11.626585+0000) 2022-09-08T01:00:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:11.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:11.554753+0000) 2022-09-08T01:00:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:12 smithi059 ceph-mon[97450]: pgmap v6486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:00:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:12 smithi055 ceph-mon[97033]: pgmap v6486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:00:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:12.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:12.626804+0000) 2022-09-08T01:00:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:12.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:12.554971+0000) 2022-09-08T01:00:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:13.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:13.627022+0000) 2022-09-08T01:00:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:13.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:13.555136+0000) 2022-09-08T01:00:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:14.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:14.627255+0000) 2022-09-08T01:00:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:14.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:14.555356+0000) 2022-09-08T01:00:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:14 smithi055 ceph-mon[97033]: pgmap v6487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:00:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:14 smithi059 ceph-mon[97450]: pgmap v6487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:00:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:15.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:15.627504+0000) 2022-09-08T01:00:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:15.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:15.555596+0000) 2022-09-08T01:00:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:16.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:16.627766+0000) 2022-09-08T01:00:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:16.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:16.555869+0000) 2022-09-08T01:00:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:16 smithi055 ceph-mon[97033]: pgmap v6488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:00:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:16 smithi055 ceph-mon[97033]: osdmap e490: 12 total, 11 up, 11 in 2022-09-08T01:00:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:16 smithi059 ceph-mon[97450]: pgmap v6488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:00:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:16 smithi059 ceph-mon[97450]: osdmap e490: 12 total, 11 up, 11 in 2022-09-08T01:00:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:17.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:17.627973+0000) 2022-09-08T01:00:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:17.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:17.556085+0000) 2022-09-08T01:00:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:18.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:18.628103+0000) 2022-09-08T01:00:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:18.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:18.556273+0000) 2022-09-08T01:00:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:18 smithi055 ceph-mon[97033]: pgmap v6490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:00:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:18 smithi059 ceph-mon[97450]: pgmap v6490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:00:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:19.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:19.628343+0000) 2022-09-08T01:00:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:19.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:19.556486+0000) 2022-09-08T01:00:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:20.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:20.628600+0000) 2022-09-08T01:00:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:20.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:20.556740+0000) 2022-09-08T01:00:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:20 smithi055 ceph-mon[97033]: pgmap v6491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:00:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:20 smithi059 ceph-mon[97450]: pgmap v6491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:00:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:21.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:21.628831+0000) 2022-09-08T01:00:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:21.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:21.556957+0000) 2022-09-08T01:00:22.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:00:22.149+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:00:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:22 smithi059 ceph-mon[97450]: pgmap v6492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:00:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:22 smithi055 ceph-mon[97033]: pgmap v6492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:00:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:22.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:22.629032+0000) 2022-09-08T01:00:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:22.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:22.557199+0000) 2022-09-08T01:00:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:23.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:23.629197+0000) 2022-09-08T01:00:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:23.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:23.557366+0000) 2022-09-08T01:00:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:24.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:24.629429+0000) 2022-09-08T01:00:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:24.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:24.557547+0000) 2022-09-08T01:00:25.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:24 smithi055 ceph-mon[97033]: pgmap v6493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:00:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:24 smithi059 ceph-mon[97450]: pgmap v6493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:00:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:25.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:25.629670+0000) 2022-09-08T01:00:25.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:25.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:25.557787+0000) 2022-09-08T01:00:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:26.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:26.558040+0000) 2022-09-08T01:00:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:26.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:26.629926+0000) 2022-09-08T01:00:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:26 smithi055 ceph-mon[97033]: pgmap v6494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:00:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:26 smithi059 ceph-mon[97450]: pgmap v6494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:00:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:27.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:27.630181+0000) 2022-09-08T01:00:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:27.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:27.558252+0000) 2022-09-08T01:00:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:28.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:28.558531+0000) 2022-09-08T01:00:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:28.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:28.630432+0000) 2022-09-08T01:00:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:28 smithi055 ceph-mon[97033]: pgmap v6495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T01:00:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:00:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:00:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:00:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:00:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:28 smithi059 ceph-mon[97450]: pgmap v6495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T01:00:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:00:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:00:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:00:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:00:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:29.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:29.630733+0000) 2022-09-08T01:00:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:29.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:29.558773+0000) 2022-09-08T01:00:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:29 smithi055 ceph-mon[97033]: pgmap v6496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:00:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:29 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T01:00:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:29 smithi059 ceph-mon[97450]: pgmap v6496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:00:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:29 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T01:00:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:30.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:30.630951+0000) 2022-09-08T01:00:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:30.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:30.558964+0000) 2022-09-08T01:00:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:31.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:31.631220+0000) 2022-09-08T01:00:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:31.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:31.559135+0000) 2022-09-08T01:00:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:31 smithi055 ceph-mon[97033]: pgmap v6497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:00:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:31 smithi059 ceph-mon[97450]: pgmap v6497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:00:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:32.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:32.559352+0000) 2022-09-08T01:00:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:32.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:32.631440+0000) 2022-09-08T01:00:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:33.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:33.631677+0000) 2022-09-08T01:00:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:33.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:33.559519+0000) 2022-09-08T01:00:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:33 smithi055 ceph-mon[97033]: pgmap v6498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:00:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:33 smithi059 ceph-mon[97450]: pgmap v6498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:00:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:34.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:34.631919+0000) 2022-09-08T01:00:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:34.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:34.559758+0000) 2022-09-08T01:00:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:35.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:35.632142+0000) 2022-09-08T01:00:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:35.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:35.559961+0000) 2022-09-08T01:00:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:35 smithi055 ceph-mon[97033]: pgmap v6499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:00:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:35 smithi059 ceph-mon[97450]: pgmap v6499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:00:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:36.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:36.632314+0000) 2022-09-08T01:00:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:36.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:36.560102+0000) 2022-09-08T01:00:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:37 smithi059 ceph-mon[97450]: pgmap v6500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:00:37.560 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:00:37.164+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:00:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:37 smithi055 ceph-mon[97033]: pgmap v6500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:00:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:37.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:37.632503+0000) 2022-09-08T01:00:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:37.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:37.560310+0000) 2022-09-08T01:00:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:38.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:38.632731+0000) 2022-09-08T01:00:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:38.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:38.560557+0000) 2022-09-08T01:00:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:39 smithi055 ceph-mon[97033]: pgmap v6501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:00:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:39 smithi059 ceph-mon[97450]: pgmap v6501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:00:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:39.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:39.632880+0000) 2022-09-08T01:00:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:39.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:39.560747+0000) 2022-09-08T01:00:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:40.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:40.633046+0000) 2022-09-08T01:00:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:40.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:40.560899+0000) 2022-09-08T01:00:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:41 smithi055 ceph-mon[97033]: pgmap v6502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:00:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:41 smithi059 ceph-mon[97450]: pgmap v6502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:00:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:41.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:41.633245+0000) 2022-09-08T01:00:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:41.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:41.561180+0000) 2022-09-08T01:00:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:42.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:42.633450+0000) 2022-09-08T01:00:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:42.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:42.561368+0000) 2022-09-08T01:00:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:43 smithi055 ceph-mon[97033]: pgmap v6503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:00:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:43 smithi059 ceph-mon[97450]: pgmap v6503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:00:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:43.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:43.633691+0000) 2022-09-08T01:00:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:43.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:43.561595+0000) 2022-09-08T01:00:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:44.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:44.633920+0000) 2022-09-08T01:00:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:44.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:44.561816+0000) 2022-09-08T01:00:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:45 smithi059 ceph-mon[97450]: pgmap v6504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:00:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:45.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:45.634159+0000) 2022-09-08T01:00:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:45.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:45.561994+0000) 2022-09-08T01:00:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:45 smithi055 ceph-mon[97033]: pgmap v6504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:00:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:46.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:46.634368+0000) 2022-09-08T01:00:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:46.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:46.562187+0000) 2022-09-08T01:00:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:47 smithi059 ceph-mon[97450]: pgmap v6505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:00:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:47 smithi055 ceph-mon[97033]: pgmap v6505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:00:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:47.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:47.634606+0000) 2022-09-08T01:00:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:47.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:47.562400+0000) 2022-09-08T01:00:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:48.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:48.634799+0000) 2022-09-08T01:00:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:48.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:48.562620+0000) 2022-09-08T01:00:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:49 smithi055 ceph-mon[97033]: pgmap v6506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:00:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:49 smithi059 ceph-mon[97450]: pgmap v6506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:00:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:49.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:49.635026+0000) 2022-09-08T01:00:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:49.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:49.562852+0000) 2022-09-08T01:00:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:50.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:50.635316+0000) 2022-09-08T01:00:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:50.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:50.563071+0000) 2022-09-08T01:00:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:51 smithi055 ceph-mon[97033]: pgmap v6507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:00:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:51 smithi059 ceph-mon[97450]: pgmap v6507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:00:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:51.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:51.635580+0000) 2022-09-08T01:00:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:51.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:51.563318+0000) 2022-09-08T01:00:52.566 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:00:52.179+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:00:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:52.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:52.635781+0000) 2022-09-08T01:00:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:52.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:52.563511+0000) 2022-09-08T01:00:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:53 smithi059 ceph-mon[97450]: pgmap v6508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:00:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:53.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:53.636018+0000) 2022-09-08T01:00:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:53.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:53.563745+0000) 2022-09-08T01:00:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:53 smithi055 ceph-mon[97033]: pgmap v6508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:00:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:54.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:54.636248+0000) 2022-09-08T01:00:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:54.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:54.563974+0000) 2022-09-08T01:00:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:55 smithi059 ceph-mon[97450]: pgmap v6509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:00:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:55.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:55.636371+0000) 2022-09-08T01:00:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:55.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:55.564222+0000) 2022-09-08T01:00:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:55 smithi055 ceph-mon[97033]: pgmap v6509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:00:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:56.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:56.636664+0000) 2022-09-08T01:00:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:56.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:56.564466+0000) 2022-09-08T01:00:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:57 smithi059 ceph-mon[97450]: pgmap v6510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:00:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:57 smithi055 ceph-mon[97033]: pgmap v6510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:00:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:57.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:57.636909+0000) 2022-09-08T01:00:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:57.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:57.564670+0000) 2022-09-08T01:00:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:58.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:58.637101+0000) 2022-09-08T01:00:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:58.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:58.564820+0000) 2022-09-08T01:00:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:00:59 smithi055 ceph-mon[97033]: pgmap v6511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:00:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:00:59 smithi059 ceph-mon[97450]: pgmap v6511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:00:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:00:59.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:59.637292+0000) 2022-09-08T01:00:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:00:59.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:00:59.565049+0000) 2022-09-08T01:01:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:00.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:00.637447+0000) 2022-09-08T01:01:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:00.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:00.565218+0000) 2022-09-08T01:01:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:01 smithi055 ceph-mon[97033]: pgmap v6512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:01:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:01 smithi059 ceph-mon[97450]: pgmap v6512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:01:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:01.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:01.637672+0000) 2022-09-08T01:01:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:01.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:01.565460+0000) 2022-09-08T01:01:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:02.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:02.637944+0000) 2022-09-08T01:01:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:02.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:02.565681+0000) 2022-09-08T01:01:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:03 smithi059 ceph-mon[97450]: pgmap v6513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:03.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:03.565836+0000) 2022-09-08T01:01:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:03.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:03.638165+0000) 2022-09-08T01:01:03.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:03 smithi055 ceph-mon[97033]: pgmap v6513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:04.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:04.638429+0000) 2022-09-08T01:01:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:04.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:04.565996+0000) 2022-09-08T01:01:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:05 smithi059 ceph-mon[97450]: pgmap v6514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:01:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:01:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:05.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:05.566240+0000) 2022-09-08T01:01:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:05.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:05.638718+0000) 2022-09-08T01:01:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:05 smithi055 ceph-mon[97033]: pgmap v6514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:01:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:01:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:06.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:06.638910+0000) 2022-09-08T01:01:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:06.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:06.566489+0000) 2022-09-08T01:01:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:07 smithi059 ceph-mon[97450]: pgmap v6515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:01:07.567 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:01:07.195+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:01:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:07 smithi055 ceph-mon[97033]: pgmap v6515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:01:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:07.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:07.566732+0000) 2022-09-08T01:01:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:07.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:07.639088+0000) 2022-09-08T01:01:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:08.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:08.639254+0000) 2022-09-08T01:01:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:08.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:08.566931+0000) 2022-09-08T01:01:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:09 smithi055 ceph-mon[97033]: pgmap v6516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:01:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:09 smithi055 ceph-mon[97033]: osdmap e491: 12 total, 11 up, 11 in 2022-09-08T01:01:09.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:09 smithi059 ceph-mon[97450]: pgmap v6516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:01:09.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:09 smithi059 ceph-mon[97450]: osdmap e491: 12 total, 11 up, 11 in 2022-09-08T01:01:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:09.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:09.639470+0000) 2022-09-08T01:01:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:09.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:09.567183+0000) 2022-09-08T01:01:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:10.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:10.639734+0000) 2022-09-08T01:01:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:10.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:10.567402+0000) 2022-09-08T01:01:11.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:11 smithi055 ceph-mon[97033]: pgmap v6518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:01:11.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:11 smithi055 ceph-mon[97033]: osdmap e492: 12 total, 11 up, 11 in 2022-09-08T01:01:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:11 smithi059 ceph-mon[97450]: pgmap v6518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:01:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:11 smithi059 ceph-mon[97450]: osdmap e492: 12 total, 11 up, 11 in 2022-09-08T01:01:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:11.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:11.639877+0000) 2022-09-08T01:01:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:11.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:11.567540+0000) 2022-09-08T01:01:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:12.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:12.640113+0000) 2022-09-08T01:01:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:12.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:12.567724+0000) 2022-09-08T01:01:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:13 smithi059 ceph-mon[97450]: pgmap v6520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T01:01:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:13.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:13.640311+0000) 2022-09-08T01:01:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:13.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:13.567903+0000) 2022-09-08T01:01:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:13 smithi055 ceph-mon[97033]: pgmap v6520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T01:01:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:14.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:14.640557+0000) 2022-09-08T01:01:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:14.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:14.568106+0000) 2022-09-08T01:01:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:15 smithi059 ceph-mon[97450]: pgmap v6521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:01:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:15.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:15.640729+0000) 2022-09-08T01:01:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:15.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:15.568301+0000) 2022-09-08T01:01:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:15 smithi055 ceph-mon[97033]: pgmap v6521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:01:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:16.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:16.640942+0000) 2022-09-08T01:01:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:16.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:16.568510+0000) 2022-09-08T01:01:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:17 smithi055 ceph-mon[97033]: pgmap v6522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:01:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:17 smithi059 ceph-mon[97450]: pgmap v6522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:01:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:17.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:17.568741+0000) 2022-09-08T01:01:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:17.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:17.641192+0000) 2022-09-08T01:01:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:18.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:18.641375+0000) 2022-09-08T01:01:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:18.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:18.568949+0000) 2022-09-08T01:01:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:19 smithi059 ceph-mon[97450]: pgmap v6523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T01:01:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:19.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:19.641584+0000) 2022-09-08T01:01:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:19.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:19.569191+0000) 2022-09-08T01:01:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:19 smithi055 ceph-mon[97033]: pgmap v6523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T01:01:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:20.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:20.641811+0000) 2022-09-08T01:01:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:20.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:20.569486+0000) 2022-09-08T01:01:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:21 smithi059 ceph-mon[97450]: pgmap v6524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:01:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:21.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:21.641944+0000) 2022-09-08T01:01:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:21.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:21.569708+0000) 2022-09-08T01:01:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:21 smithi055 ceph-mon[97033]: pgmap v6524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:01:22.570 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:01:22.210+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:01:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:22.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:22.642163+0000) 2022-09-08T01:01:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:22.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:22.569910+0000) 2022-09-08T01:01:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:23 smithi059 ceph-mon[97450]: pgmap v6525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:01:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:23.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:23.642406+0000) 2022-09-08T01:01:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:23.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:23.570172+0000) 2022-09-08T01:01:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:23 smithi055 ceph-mon[97033]: pgmap v6525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:01:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:24.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:24.642708+0000) 2022-09-08T01:01:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:24.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:24.570378+0000) 2022-09-08T01:01:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:25 smithi059 ceph-mon[97450]: pgmap v6526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:01:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:25.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:25.642917+0000) 2022-09-08T01:01:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:25.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:25.570617+0000) 2022-09-08T01:01:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:25 smithi055 ceph-mon[97033]: pgmap v6526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:01:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:26.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:26.643097+0000) 2022-09-08T01:01:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:26.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:26.570882+0000) 2022-09-08T01:01:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:27 smithi055 ceph-mon[97033]: pgmap v6527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:01:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:27 smithi059 ceph-mon[97450]: pgmap v6527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:01:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:27.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:27.643364+0000) 2022-09-08T01:01:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:27.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:27.571144+0000) 2022-09-08T01:01:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:28.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:28.643596+0000) 2022-09-08T01:01:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:28.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:28.571416+0000) 2022-09-08T01:01:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:29 smithi059 ceph-mon[97450]: pgmap v6528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:01:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:01:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:01:29.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:01:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:29.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:29.643825+0000) 2022-09-08T01:01:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:29.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:29.571591+0000) 2022-09-08T01:01:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:29 smithi055 ceph-mon[97033]: pgmap v6528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:01:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:01:29.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:01:29.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:01:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:30.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:30.644018+0000) 2022-09-08T01:01:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:30.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:30.571715+0000) 2022-09-08T01:01:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:31 smithi059 ceph-mon[97450]: pgmap v6529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:01:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:31.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:31.644162+0000) 2022-09-08T01:01:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:31.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:31.571915+0000) 2022-09-08T01:01:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:31 smithi055 ceph-mon[97033]: pgmap v6529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:01:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:32.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:32.644355+0000) 2022-09-08T01:01:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:32.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:32.572096+0000) 2022-09-08T01:01:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:33 smithi059 ceph-mon[97450]: pgmap v6530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:33.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:33.644585+0000) 2022-09-08T01:01:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:33.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:33.572307+0000) 2022-09-08T01:01:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:33 smithi055 ceph-mon[97033]: pgmap v6530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:34.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:34.644821+0000) 2022-09-08T01:01:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:34.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:34.572531+0000) 2022-09-08T01:01:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:35 smithi059 ceph-mon[97450]: pgmap v6531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:35.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:35.645056+0000) 2022-09-08T01:01:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:35.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:35.572750+0000) 2022-09-08T01:01:35.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:35 smithi055 ceph-mon[97033]: pgmap v6531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:36.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:36.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:36.645248+0000) 2022-09-08T01:01:36.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:36.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:36.572996+0000) 2022-09-08T01:01:37.574 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:01:37.213+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:01:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:37 smithi055 ceph-mon[97033]: pgmap v6532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:01:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:37 smithi059 ceph-mon[97450]: pgmap v6532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:01:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:37.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:37.645422+0000) 2022-09-08T01:01:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:37.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:37.573212+0000) 2022-09-08T01:01:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:38.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:38.645569+0000) 2022-09-08T01:01:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:38.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:38.573423+0000) 2022-09-08T01:01:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:39 smithi059 ceph-mon[97450]: pgmap v6533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:39.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:39.645712+0000) 2022-09-08T01:01:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:39.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:39.573697+0000) 2022-09-08T01:01:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:39 smithi055 ceph-mon[97033]: pgmap v6533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:40.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:40.573931+0000) 2022-09-08T01:01:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:40.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:40.645883+0000) 2022-09-08T01:01:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:41 smithi059 ceph-mon[97450]: pgmap v6534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:01:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:41.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:41.646129+0000) 2022-09-08T01:01:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:41.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:41.574175+0000) 2022-09-08T01:01:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:41 smithi055 ceph-mon[97033]: pgmap v6534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:01:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:42.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:42.574395+0000) 2022-09-08T01:01:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:42.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:42.646315+0000) 2022-09-08T01:01:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:43 smithi059 ceph-mon[97450]: pgmap v6535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:43.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:43.574741+0000) 2022-09-08T01:01:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:43.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:43.646482+0000) 2022-09-08T01:01:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:43 smithi055 ceph-mon[97033]: pgmap v6535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:44.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:44.575006+0000) 2022-09-08T01:01:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:44.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:44.646656+0000) 2022-09-08T01:01:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:45 smithi059 ceph-mon[97450]: pgmap v6536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:45.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:45.646785+0000) 2022-09-08T01:01:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:45.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:45.575284+0000) 2022-09-08T01:01:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:45 smithi055 ceph-mon[97033]: pgmap v6536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:46.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:46.647019+0000) 2022-09-08T01:01:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:46.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:46.575534+0000) 2022-09-08T01:01:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:47 smithi055 ceph-mon[97033]: pgmap v6537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:01:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:47 smithi059 ceph-mon[97450]: pgmap v6537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:01:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:47.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:47.575735+0000) 2022-09-08T01:01:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:47.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:47.647228+0000) 2022-09-08T01:01:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:48.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:48.647472+0000) 2022-09-08T01:01:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:48.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:48.575944+0000) 2022-09-08T01:01:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:49 smithi059 ceph-mon[97450]: pgmap v6538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:49.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:49.576104+0000) 2022-09-08T01:01:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:49.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:49.647720+0000) 2022-09-08T01:01:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:49 smithi055 ceph-mon[97033]: pgmap v6538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:50.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:50.647927+0000) 2022-09-08T01:01:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:50.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:50.576286+0000) 2022-09-08T01:01:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:51 smithi059 ceph-mon[97450]: pgmap v6539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:01:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:51.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:51.648138+0000) 2022-09-08T01:01:51.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:51.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:51.576488+0000) 2022-09-08T01:01:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:51 smithi055 ceph-mon[97033]: pgmap v6539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:01:52.577 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:01:52.229+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:01:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:52.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:52.648361+0000) 2022-09-08T01:01:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:52.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:52.576735+0000) 2022-09-08T01:01:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:53 smithi059 ceph-mon[97450]: pgmap v6540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:53.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:53.648556+0000) 2022-09-08T01:01:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:53.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:53.576975+0000) 2022-09-08T01:01:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:53 smithi055 ceph-mon[97033]: pgmap v6540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:54.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:54.648734+0000) 2022-09-08T01:01:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:54.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:54.577159+0000) 2022-09-08T01:01:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:55 smithi059 ceph-mon[97450]: pgmap v6541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:55.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:55.648957+0000) 2022-09-08T01:01:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:55.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:55.577335+0000) 2022-09-08T01:01:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:55 smithi055 ceph-mon[97033]: pgmap v6541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:56.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:56.649178+0000) 2022-09-08T01:01:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:56.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:56.577578+0000) 2022-09-08T01:01:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:57 smithi055 ceph-mon[97033]: pgmap v6542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:01:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:57 smithi059 ceph-mon[97450]: pgmap v6542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:01:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:57.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:57.649392+0000) 2022-09-08T01:01:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:57.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:57.577756+0000) 2022-09-08T01:01:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:58.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:58.649608+0000) 2022-09-08T01:01:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:58.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:58.577998+0000) 2022-09-08T01:01:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:01:59 smithi059 ceph-mon[97450]: pgmap v6543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:01:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:01:59.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:59.649858+0000) 2022-09-08T01:01:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:01:59.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:01:59.578235+0000) 2022-09-08T01:01:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:01:59 smithi055 ceph-mon[97033]: pgmap v6543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:00.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:00.650027+0000) 2022-09-08T01:02:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:00.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:00.578459+0000) 2022-09-08T01:02:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:01 smithi059 ceph-mon[97450]: pgmap v6544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:02:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:01.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:01.650246+0000) 2022-09-08T01:02:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:01.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:01.578620+0000) 2022-09-08T01:02:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:01 smithi055 ceph-mon[97033]: pgmap v6544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:02:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:02.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:02.650378+0000) 2022-09-08T01:02:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:02.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:02.578775+0000) 2022-09-08T01:02:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:03 smithi059 ceph-mon[97450]: pgmap v6545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:03.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:03.650615+0000) 2022-09-08T01:02:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:03.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:03.579021+0000) 2022-09-08T01:02:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:03 smithi055 ceph-mon[97033]: pgmap v6545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:04.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:04.650880+0000) 2022-09-08T01:02:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:04.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:04.579214+0000) 2022-09-08T01:02:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:05 smithi059 ceph-mon[97450]: pgmap v6546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:02:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:02:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:05.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:05.651073+0000) 2022-09-08T01:02:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:05.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:05.579440+0000) 2022-09-08T01:02:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:05 smithi055 ceph-mon[97033]: pgmap v6546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:02:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:02:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:06.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:06.651258+0000) 2022-09-08T01:02:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:06.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:06.579612+0000) 2022-09-08T01:02:07.582 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:02:07.240+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:02:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:07 smithi055 ceph-mon[97033]: pgmap v6547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:02:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:07 smithi059 ceph-mon[97450]: pgmap v6547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:02:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:07.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:07.651465+0000) 2022-09-08T01:02:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:07.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:07.579870+0000) 2022-09-08T01:02:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:08.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:08.651697+0000) 2022-09-08T01:02:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:08.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:08.580056+0000) 2022-09-08T01:02:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:09 smithi055 ceph-mon[97033]: pgmap v6548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:02:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:09 smithi055 ceph-mon[97033]: osdmap e493: 12 total, 11 up, 11 in 2022-09-08T01:02:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:09 smithi059 ceph-mon[97450]: pgmap v6548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:02:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:09 smithi059 ceph-mon[97450]: osdmap e493: 12 total, 11 up, 11 in 2022-09-08T01:02:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:09.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:09.651953+0000) 2022-09-08T01:02:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:09.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:09.580279+0000) 2022-09-08T01:02:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:10.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:10.652148+0000) 2022-09-08T01:02:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:10.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:10.580515+0000) 2022-09-08T01:02:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:11 smithi055 ceph-mon[97033]: pgmap v6550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:02:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:11 smithi055 ceph-mon[97033]: osdmap e494: 12 total, 11 up, 11 in 2022-09-08T01:02:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:11 smithi059 ceph-mon[97450]: pgmap v6550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:02:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:11 smithi059 ceph-mon[97450]: osdmap e494: 12 total, 11 up, 11 in 2022-09-08T01:02:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:11.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:11.652356+0000) 2022-09-08T01:02:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:11.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:11.580785+0000) 2022-09-08T01:02:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:12.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:12.652597+0000) 2022-09-08T01:02:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:12.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:12.581023+0000) 2022-09-08T01:02:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:13 smithi059 ceph-mon[97450]: pgmap v6552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:02:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:13.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:13.652790+0000) 2022-09-08T01:02:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:13.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:13.581225+0000) 2022-09-08T01:02:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:13 smithi055 ceph-mon[97033]: pgmap v6552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:02:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:14.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:14.653000+0000) 2022-09-08T01:02:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:14.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:14.581426+0000) 2022-09-08T01:02:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:15 smithi059 ceph-mon[97450]: pgmap v6553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:02:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:15.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:15.653238+0000) 2022-09-08T01:02:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:15.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:15.581663+0000) 2022-09-08T01:02:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:15 smithi055 ceph-mon[97033]: pgmap v6553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:02:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:16.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:16.653444+0000) 2022-09-08T01:02:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:16.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:16.581929+0000) 2022-09-08T01:02:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:17 smithi055 ceph-mon[97033]: pgmap v6554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:02:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:17 smithi059 ceph-mon[97450]: pgmap v6554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:02:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:17.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:17.582153+0000) 2022-09-08T01:02:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:17.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:17.653676+0000) 2022-09-08T01:02:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:18.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:18.582378+0000) 2022-09-08T01:02:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:18.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:18.653827+0000) 2022-09-08T01:02:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:19 smithi059 ceph-mon[97450]: pgmap v6555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T01:02:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:19.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:19.654037+0000) 2022-09-08T01:02:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:19.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:19.582622+0000) 2022-09-08T01:02:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:19 smithi055 ceph-mon[97033]: pgmap v6555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T01:02:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:20.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:20.654338+0000) 2022-09-08T01:02:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:20.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:20.582828+0000) 2022-09-08T01:02:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:21 smithi059 ceph-mon[97450]: pgmap v6556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:02:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:21.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:21.654590+0000) 2022-09-08T01:02:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:21.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:21.583034+0000) 2022-09-08T01:02:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:21 smithi055 ceph-mon[97033]: pgmap v6556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:02:22.583 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:02:22.253+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:02:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:22.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:22.654782+0000) 2022-09-08T01:02:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:22.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:22.583279+0000) 2022-09-08T01:02:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:23 smithi059 ceph-mon[97450]: pgmap v6557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:02:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:23.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:23.583533+0000) 2022-09-08T01:02:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:23.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:23.654944+0000) 2022-09-08T01:02:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:23 smithi055 ceph-mon[97033]: pgmap v6557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:02:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:24.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:24.655167+0000) 2022-09-08T01:02:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:24.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:24.583754+0000) 2022-09-08T01:02:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:25 smithi059 ceph-mon[97450]: pgmap v6558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:02:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:25.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:25.655407+0000) 2022-09-08T01:02:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:25.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:25.583984+0000) 2022-09-08T01:02:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:25 smithi055 ceph-mon[97033]: pgmap v6558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:02:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:26.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:26.655671+0000) 2022-09-08T01:02:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:26.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:26.584201+0000) 2022-09-08T01:02:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:27 smithi055 ceph-mon[97033]: pgmap v6559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:02:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:27 smithi059 ceph-mon[97450]: pgmap v6559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:02:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:27.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:27.584468+0000) 2022-09-08T01:02:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:27.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:27.655951+0000) 2022-09-08T01:02:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:28.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:28.584723+0000) 2022-09-08T01:02:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:28.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:28.656134+0000) 2022-09-08T01:02:29.752 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:29 smithi059 ceph-mon[97450]: pgmap v6560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:29.752 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:02:29.753 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:02:29.753 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:02:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:29.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:29.656373+0000) 2022-09-08T01:02:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:29.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:29.584929+0000) 2022-09-08T01:02:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:29 smithi055 ceph-mon[97033]: pgmap v6560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:02:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:02:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:02:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:30.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:30.585134+0000) 2022-09-08T01:02:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:30.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:30.656686+0000) 2022-09-08T01:02:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:02:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:02:31.916 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:31.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:31.585333+0000) 2022-09-08T01:02:31.917 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:31.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:31.656960+0000) 2022-09-08T01:02:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:31 smithi055 ceph-mon[97033]: pgmap v6561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:02:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:31 smithi059 ceph-mon[97450]: pgmap v6561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:02:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:32.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:32.657215+0000) 2022-09-08T01:02:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:32.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:32.585564+0000) 2022-09-08T01:02:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:33.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:33.585808+0000) 2022-09-08T01:02:33.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:33.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:33.657438+0000) 2022-09-08T01:02:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:33 smithi055 ceph-mon[97033]: pgmap v6562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:02:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:33 smithi059 ceph-mon[97450]: pgmap v6562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:02:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:34.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:34.657609+0000) 2022-09-08T01:02:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:34.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:34.586039+0000) 2022-09-08T01:02:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:35.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:35.657849+0000) 2022-09-08T01:02:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:35.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:35.586260+0000) 2022-09-08T01:02:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:35 smithi055 ceph-mon[97033]: pgmap v6563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:02:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:35 smithi059 ceph-mon[97450]: pgmap v6563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:02:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:36.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:36.658064+0000) 2022-09-08T01:02:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:36.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:36.586480+0000) 2022-09-08T01:02:37.588 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:02:37.261+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:02:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:37 smithi055 ceph-mon[97033]: pgmap v6564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:02:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:37 smithi059 ceph-mon[97450]: pgmap v6564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:02:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:37.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:37.658312+0000) 2022-09-08T01:02:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:37.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:37.586604+0000) 2022-09-08T01:02:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:38.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:38.658566+0000) 2022-09-08T01:02:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:38.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:38.586849+0000) 2022-09-08T01:02:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:39 smithi059 ceph-mon[97450]: pgmap v6565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:02:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:39.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:39.658740+0000) 2022-09-08T01:02:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:39.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:39.587062+0000) 2022-09-08T01:02:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:39 smithi055 ceph-mon[97033]: pgmap v6565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:02:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:40.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:40.587312+0000) 2022-09-08T01:02:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:40.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:40.659003+0000) 2022-09-08T01:02:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:41 smithi059 ceph-mon[97450]: pgmap v6566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:02:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:41.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:41.659255+0000) 2022-09-08T01:02:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:41.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:41.587561+0000) 2022-09-08T01:02:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:41 smithi055 ceph-mon[97033]: pgmap v6566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:02:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:42.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:42.659532+0000) 2022-09-08T01:02:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:42.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:42.587788+0000) 2022-09-08T01:02:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:43 smithi059 ceph-mon[97450]: pgmap v6567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:43.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:43.587967+0000) 2022-09-08T01:02:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:43.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:43.659758+0000) 2022-09-08T01:02:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:43 smithi055 ceph-mon[97033]: pgmap v6567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:44.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:44.659922+0000) 2022-09-08T01:02:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:44.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:44.588190+0000) 2022-09-08T01:02:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:45 smithi059 ceph-mon[97450]: pgmap v6568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:45.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:45.660134+0000) 2022-09-08T01:02:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:45.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:45.588369+0000) 2022-09-08T01:02:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:45 smithi055 ceph-mon[97033]: pgmap v6568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:46.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:46.660308+0000) 2022-09-08T01:02:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:46.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:46.588559+0000) 2022-09-08T01:02:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:47 smithi055 ceph-mon[97033]: pgmap v6569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:02:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:47 smithi059 ceph-mon[97450]: pgmap v6569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:02:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:47.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:47.660553+0000) 2022-09-08T01:02:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:47.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:47.588801+0000) 2022-09-08T01:02:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:48.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:48.660789+0000) 2022-09-08T01:02:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:48.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:48.589044+0000) 2022-09-08T01:02:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:49 smithi059 ceph-mon[97450]: pgmap v6570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:49.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:49.660940+0000) 2022-09-08T01:02:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:49.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:49.589313+0000) 2022-09-08T01:02:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:49 smithi055 ceph-mon[97033]: pgmap v6570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:50.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:50.661121+0000) 2022-09-08T01:02:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:50.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:50.589542+0000) 2022-09-08T01:02:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:51 smithi059 ceph-mon[97450]: pgmap v6571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:02:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:51.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:51.589776+0000) 2022-09-08T01:02:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:51.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:51.661303+0000) 2022-09-08T01:02:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:51 smithi055 ceph-mon[97033]: pgmap v6571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:02:52.590 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:02:52.277+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:02:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:52.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:52.661539+0000) 2022-09-08T01:02:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:52.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:52.590024+0000) 2022-09-08T01:02:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:53 smithi059 ceph-mon[97450]: pgmap v6572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:53.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:53.661779+0000) 2022-09-08T01:02:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:53.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:53.590229+0000) 2022-09-08T01:02:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:53 smithi055 ceph-mon[97033]: pgmap v6572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:54.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:54.661920+0000) 2022-09-08T01:02:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:54.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:54.590412+0000) 2022-09-08T01:02:55.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:55 smithi059 ceph-mon[97450]: pgmap v6573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:55.862 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:55.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:55.662043+0000) 2022-09-08T01:02:55.863 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:55.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:55.590555+0000) 2022-09-08T01:02:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:55 smithi055 ceph-mon[97033]: pgmap v6573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:56.845 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:56.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:56.662182+0000) 2022-09-08T01:02:56.845 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:56.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:56.590708+0000) 2022-09-08T01:02:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:57 smithi055 ceph-mon[97033]: pgmap v6574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:02:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:57 smithi059 ceph-mon[97450]: pgmap v6574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:02:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:57.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:57.662365+0000) 2022-09-08T01:02:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:57.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:57.590830+0000) 2022-09-08T01:02:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:58.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:58.591033+0000) 2022-09-08T01:02:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:58.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:58.662568+0000) 2022-09-08T01:02:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:02:59 smithi059 ceph-mon[97450]: pgmap v6575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:02:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:02:59.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:59.662821+0000) 2022-09-08T01:02:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:02:59.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:02:59.591297+0000) 2022-09-08T01:02:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:02:59 smithi055 ceph-mon[97033]: pgmap v6575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:00.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:00.663016+0000) 2022-09-08T01:03:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:00.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:00.591522+0000) 2022-09-08T01:03:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:01 smithi059 ceph-mon[97450]: pgmap v6576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:03:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:01.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:01.663240+0000) 2022-09-08T01:03:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:01.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:01.591746+0000) 2022-09-08T01:03:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:01 smithi055 ceph-mon[97033]: pgmap v6576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:03:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:02.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:02.663469+0000) 2022-09-08T01:03:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:02.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:02.591955+0000) 2022-09-08T01:03:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:03 smithi059 ceph-mon[97450]: pgmap v6577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:03.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:03.663661+0000) 2022-09-08T01:03:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:03.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:03.592199+0000) 2022-09-08T01:03:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:03 smithi055 ceph-mon[97033]: pgmap v6577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:04.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:04.663903+0000) 2022-09-08T01:03:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:04.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:04.592432+0000) 2022-09-08T01:03:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:05 smithi059 ceph-mon[97450]: pgmap v6578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:03:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:03:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:05.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:05.664124+0000) 2022-09-08T01:03:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:05.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:05.592578+0000) 2022-09-08T01:03:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:05 smithi055 ceph-mon[97033]: pgmap v6578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:03:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:03:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:06.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:06.664335+0000) 2022-09-08T01:03:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:06.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:06.592831+0000) 2022-09-08T01:03:07.593 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:03:07.289+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:03:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:07 smithi055 ceph-mon[97033]: pgmap v6579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:03:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:07 smithi059 ceph-mon[97450]: pgmap v6579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:03:07.912 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:07.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:07.592990+0000) 2022-09-08T01:03:07.913 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:07.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:07.664571+0000) 2022-09-08T01:03:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:08.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:08.664705+0000) 2022-09-08T01:03:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:08.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:08.593143+0000) 2022-09-08T01:03:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:09 smithi055 ceph-mon[97033]: pgmap v6580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:03:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:09 smithi055 ceph-mon[97033]: osdmap e495: 12 total, 11 up, 11 in 2022-09-08T01:03:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:09 smithi059 ceph-mon[97450]: pgmap v6580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:03:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:09 smithi059 ceph-mon[97450]: osdmap e495: 12 total, 11 up, 11 in 2022-09-08T01:03:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:09.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:09.664902+0000) 2022-09-08T01:03:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:09.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:09.593383+0000) 2022-09-08T01:03:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:10.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:10.665088+0000) 2022-09-08T01:03:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:10.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:10.593665+0000) 2022-09-08T01:03:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:11 smithi059 ceph-mon[97450]: pgmap v6582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:03:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:11 smithi059 ceph-mon[97450]: osdmap e496: 12 total, 11 up, 11 in 2022-09-08T01:03:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:11.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:11.665274+0000) 2022-09-08T01:03:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:11.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:11.593829+0000) 2022-09-08T01:03:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:11 smithi055 ceph-mon[97033]: pgmap v6582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:03:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:11 smithi055 ceph-mon[97033]: osdmap e496: 12 total, 11 up, 11 in 2022-09-08T01:03:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:12.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:12.665491+0000) 2022-09-08T01:03:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:12.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:12.594036+0000) 2022-09-08T01:03:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:13 smithi059 ceph-mon[97450]: pgmap v6584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:03:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:13.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:13.594194+0000) 2022-09-08T01:03:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:13.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:13.665752+0000) 2022-09-08T01:03:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:13 smithi055 ceph-mon[97033]: pgmap v6584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:03:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:14.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:14.665966+0000) 2022-09-08T01:03:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:14.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:14.594392+0000) 2022-09-08T01:03:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:15 smithi059 ceph-mon[97450]: pgmap v6585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:03:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:15.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:15.666153+0000) 2022-09-08T01:03:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:15.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:15.594620+0000) 2022-09-08T01:03:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:15 smithi055 ceph-mon[97033]: pgmap v6585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:03:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:16.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:16.666373+0000) 2022-09-08T01:03:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:16.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:16.594871+0000) 2022-09-08T01:03:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:17 smithi055 ceph-mon[97033]: pgmap v6586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:03:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:17 smithi059 ceph-mon[97450]: pgmap v6586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:03:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:17.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:17.666592+0000) 2022-09-08T01:03:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:17.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:17.595039+0000) 2022-09-08T01:03:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:18.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:18.666855+0000) 2022-09-08T01:03:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:18.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:18.595281+0000) 2022-09-08T01:03:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:19 smithi059 ceph-mon[97450]: pgmap v6587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T01:03:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:19.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:19.667020+0000) 2022-09-08T01:03:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:19.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:19.595595+0000) 2022-09-08T01:03:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:19 smithi055 ceph-mon[97033]: pgmap v6587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T01:03:20.865 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:20.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:20.667203+0000) 2022-09-08T01:03:20.866 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:20.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:20.595828+0000) 2022-09-08T01:03:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:21 smithi059 ceph-mon[97450]: pgmap v6588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:03:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:21.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:21.667342+0000) 2022-09-08T01:03:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:21.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:21.595983+0000) 2022-09-08T01:03:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:21 smithi055 ceph-mon[97033]: pgmap v6588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:03:22.596 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:03:22.306+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:03:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:22.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:22.667590+0000) 2022-09-08T01:03:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:22.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:22.596180+0000) 2022-09-08T01:03:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:23 smithi059 ceph-mon[97450]: pgmap v6589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:03:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:23.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:23.667836+0000) 2022-09-08T01:03:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:23.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:23.596406+0000) 2022-09-08T01:03:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:23 smithi055 ceph-mon[97033]: pgmap v6589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:03:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:24.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:24.668086+0000) 2022-09-08T01:03:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:24.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:24.596631+0000) 2022-09-08T01:03:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:25 smithi059 ceph-mon[97450]: pgmap v6590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:03:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:25.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:25.668299+0000) 2022-09-08T01:03:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:25.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:25.596842+0000) 2022-09-08T01:03:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:25 smithi055 ceph-mon[97033]: pgmap v6590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:03:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:26.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:26.668468+0000) 2022-09-08T01:03:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:26.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:26.597061+0000) 2022-09-08T01:03:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:27 smithi055 ceph-mon[97033]: pgmap v6591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:03:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:27 smithi059 ceph-mon[97450]: pgmap v6591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:03:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:27.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:27.668742+0000) 2022-09-08T01:03:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:27.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:27.597329+0000) 2022-09-08T01:03:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:28.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:28.668927+0000) 2022-09-08T01:03:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:28.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:28.597573+0000) 2022-09-08T01:03:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:29 smithi059 ceph-mon[97450]: pgmap v6592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:29.814 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:29 smithi055 ceph-mon[97033]: pgmap v6592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:29.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:29.669159+0000) 2022-09-08T01:03:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:29.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:29.597811+0000) 2022-09-08T01:03:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:03:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:03:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:03:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:03:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:03:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:03:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:03:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:03:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:03:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:03:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:30.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:30.669394+0000) 2022-09-08T01:03:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:30.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:30.598036+0000) 2022-09-08T01:03:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:31 smithi059 ceph-mon[97450]: pgmap v6593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:03:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:31.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:31.669678+0000) 2022-09-08T01:03:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:31.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:31.598286+0000) 2022-09-08T01:03:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:31 smithi055 ceph-mon[97033]: pgmap v6593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:03:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:32.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:32.669846+0000) 2022-09-08T01:03:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:32.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:32.598452+0000) 2022-09-08T01:03:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:33.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:33.669989+0000) 2022-09-08T01:03:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:33.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:33.598596+0000) 2022-09-08T01:03:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:33 smithi055 ceph-mon[97033]: pgmap v6594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:03:33.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:03:33.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:03:33.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:03:33.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:33.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:33.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:33.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:33.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:33.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:33.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:33.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:33.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:03:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:33 smithi059 ceph-mon[97450]: pgmap v6594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:03:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:03:34.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:03:34.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:03:34.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:34.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:34.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:34.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:34.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:34.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:34.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:34.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:03:34.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:03:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:34.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:34.598829+0000) 2022-09-08T01:03:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:34.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:34.670186+0000) 2022-09-08T01:03:34.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:34 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T01:03:34.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:34 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T01:03:34.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:34 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T01:03:34.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:34 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T01:03:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:34 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T01:03:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:34 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T01:03:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:34 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T01:03:35.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:34 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T01:03:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:35.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:35.599066+0000) 2022-09-08T01:03:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:35.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:35.670401+0000) 2022-09-08T01:03:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:35 smithi055 ceph-mon[97033]: pgmap v6595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:35 smithi059 ceph-mon[97450]: pgmap v6595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:36.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:36.670665+0000) 2022-09-08T01:03:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:36.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:36.599287+0000) 2022-09-08T01:03:37.599 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:03:37.321+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:03:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:37 smithi055 ceph-mon[97033]: pgmap v6596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:03:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:37 smithi059 ceph-mon[97450]: pgmap v6596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:03:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:37.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:37.670901+0000) 2022-09-08T01:03:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:37.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:37.599533+0000) 2022-09-08T01:03:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:38.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:38.671124+0000) 2022-09-08T01:03:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:38.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:38.599786+0000) 2022-09-08T01:03:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:39 smithi059 ceph-mon[97450]: pgmap v6597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:39.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:39.671307+0000) 2022-09-08T01:03:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:39.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:39.599964+0000) 2022-09-08T01:03:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:39 smithi055 ceph-mon[97033]: pgmap v6597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:40.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:40.671529+0000) 2022-09-08T01:03:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:40.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:40.600130+0000) 2022-09-08T01:03:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:41 smithi059 ceph-mon[97450]: pgmap v6598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:03:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:41.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:41.671759+0000) 2022-09-08T01:03:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:41.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:41.600354+0000) 2022-09-08T01:03:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:41 smithi055 ceph-mon[97033]: pgmap v6598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:03:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:42.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:42.672015+0000) 2022-09-08T01:03:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:42.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:42.600561+0000) 2022-09-08T01:03:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:43.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:43.672192+0000) 2022-09-08T01:03:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:43.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:43.600712+0000) 2022-09-08T01:03:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:43 smithi055 ceph-mon[97033]: pgmap v6599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:43 smithi059 ceph-mon[97450]: pgmap v6599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:44.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:44.672428+0000) 2022-09-08T01:03:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:44.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:44.600920+0000) 2022-09-08T01:03:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:45.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:45.601086+0000) 2022-09-08T01:03:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:45.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:45.672585+0000) 2022-09-08T01:03:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:45 smithi055 ceph-mon[97033]: pgmap v6600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:45 smithi059 ceph-mon[97450]: pgmap v6600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:46.673 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:46.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:46.601230+0000) 2022-09-08T01:03:46.997 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:46.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:46.672715+0000) 2022-09-08T01:03:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:47 smithi055 ceph-mon[97033]: pgmap v6601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:03:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:47 smithi059 ceph-mon[97450]: pgmap v6601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:03:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:47.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:47.672948+0000) 2022-09-08T01:03:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:47.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:47.601431+0000) 2022-09-08T01:03:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:48.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:48.673182+0000) 2022-09-08T01:03:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:48.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:48.601647+0000) 2022-09-08T01:03:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:49 smithi059 ceph-mon[97450]: pgmap v6602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:49.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:49.673404+0000) 2022-09-08T01:03:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:49.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:49.601793+0000) 2022-09-08T01:03:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:49 smithi055 ceph-mon[97033]: pgmap v6602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:50.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:50.673633+0000) 2022-09-08T01:03:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:50.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:50.602024+0000) 2022-09-08T01:03:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:51.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:51.673867+0000) 2022-09-08T01:03:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:51.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:51.602202+0000) 2022-09-08T01:03:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:51 smithi055 ceph-mon[97033]: pgmap v6603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:03:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:51 smithi059 ceph-mon[97450]: pgmap v6603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:03:52.603 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:03:52.325+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:03:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:52.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:52.674120+0000) 2022-09-08T01:03:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:52.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:52.602445+0000) 2022-09-08T01:03:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:53.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:53.674351+0000) 2022-09-08T01:03:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:53.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:53.602649+0000) 2022-09-08T01:03:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:53 smithi055 ceph-mon[97033]: pgmap v6604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:53 smithi059 ceph-mon[97450]: pgmap v6604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:54.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:54.674519+0000) 2022-09-08T01:03:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:54.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:54.602879+0000) 2022-09-08T01:03:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:55.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:55.674757+0000) 2022-09-08T01:03:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:55.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:55.603158+0000) 2022-09-08T01:03:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:55 smithi055 ceph-mon[97033]: pgmap v6605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:55 smithi059 ceph-mon[97450]: pgmap v6605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:56.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:56.674945+0000) 2022-09-08T01:03:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:56.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:56.603334+0000) 2022-09-08T01:03:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:57 smithi055 ceph-mon[97033]: pgmap v6606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:03:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:57 smithi059 ceph-mon[97450]: pgmap v6606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:03:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:57.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:57.675171+0000) 2022-09-08T01:03:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:57.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:57.603530+0000) 2022-09-08T01:03:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:58.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:58.603672+0000) 2022-09-08T01:03:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:58.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:58.675417+0000) 2022-09-08T01:03:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:03:59 smithi059 ceph-mon[97450]: pgmap v6607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:03:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:03:59.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:59.675661+0000) 2022-09-08T01:03:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:03:59.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:03:59.603817+0000) 2022-09-08T01:03:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:03:59 smithi055 ceph-mon[97033]: pgmap v6607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:00.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:00.675859+0000) 2022-09-08T01:04:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:00.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:00.603981+0000) 2022-09-08T01:04:01.676 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:01.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:01.604136+0000) 2022-09-08T01:04:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:01 smithi055 ceph-mon[97033]: pgmap v6608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:04:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:01 smithi059 ceph-mon[97450]: pgmap v6608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:04:02.141 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:01.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:01.676023+0000) 2022-09-08T01:04:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:02.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:02.604280+0000) 2022-09-08T01:04:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:02.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:02.676164+0000) 2022-09-08T01:04:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:03.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:03.676377+0000) 2022-09-08T01:04:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:03.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:03.604507+0000) 2022-09-08T01:04:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:03 smithi055 ceph-mon[97033]: pgmap v6609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:03 smithi059 ceph-mon[97450]: pgmap v6609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:04.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:04.676713+0000) 2022-09-08T01:04:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:04.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:04.604738+0000) 2022-09-08T01:04:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:05.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:05.676951+0000) 2022-09-08T01:04:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:05.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:05.605002+0000) 2022-09-08T01:04:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:05 smithi055 ceph-mon[97033]: pgmap v6610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:04:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:04:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:05 smithi059 ceph-mon[97450]: pgmap v6610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:04:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:04:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:06.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:06.677154+0000) 2022-09-08T01:04:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:06.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:06.605248+0000) 2022-09-08T01:04:07.606 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:04:07.338+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:04:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:07 smithi055 ceph-mon[97033]: pgmap v6611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:04:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:07 smithi059 ceph-mon[97450]: pgmap v6611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:04:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:07.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:07.677352+0000) 2022-09-08T01:04:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:07.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:07.605479+0000) 2022-09-08T01:04:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:08.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:08.605743+0000) 2022-09-08T01:04:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:08.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:08.677578+0000) 2022-09-08T01:04:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:09 smithi059 ceph-mon[97450]: osdmap e497: 12 total, 11 up, 11 in 2022-09-08T01:04:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:09 smithi059 ceph-mon[97450]: pgmap v6613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:04:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:09.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:09.677799+0000) 2022-09-08T01:04:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:09.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:09.605890+0000) 2022-09-08T01:04:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:09 smithi055 ceph-mon[97033]: osdmap e497: 12 total, 11 up, 11 in 2022-09-08T01:04:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:09 smithi055 ceph-mon[97033]: pgmap v6613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:04:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:10.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:10.678053+0000) 2022-09-08T01:04:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:10.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:10.606061+0000) 2022-09-08T01:04:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:11.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:11.678276+0000) 2022-09-08T01:04:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:11.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:11.606298+0000) 2022-09-08T01:04:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:11 smithi055 ceph-mon[97033]: pgmap v6614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:04:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:11 smithi055 ceph-mon[97033]: osdmap e498: 12 total, 11 up, 11 in 2022-09-08T01:04:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:11 smithi059 ceph-mon[97450]: pgmap v6614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:04:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:11 smithi059 ceph-mon[97450]: osdmap e498: 12 total, 11 up, 11 in 2022-09-08T01:04:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:12.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:12.678428+0000) 2022-09-08T01:04:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:12.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:12.606530+0000) 2022-09-08T01:04:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:13.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:13.606778+0000) 2022-09-08T01:04:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:13.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:13.678746+0000) 2022-09-08T01:04:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:13 smithi055 ceph-mon[97033]: pgmap v6616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:04:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:13 smithi059 ceph-mon[97450]: pgmap v6616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:04:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:14.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:14.678903+0000) 2022-09-08T01:04:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:14.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:14.606954+0000) 2022-09-08T01:04:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:15.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:15.679076+0000) 2022-09-08T01:04:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:15.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:15.607209+0000) 2022-09-08T01:04:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:15 smithi055 ceph-mon[97033]: pgmap v6617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:04:16.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:15 smithi059 ceph-mon[97450]: pgmap v6617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:04:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:16.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:16.607405+0000) 2022-09-08T01:04:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:16.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:16.679244+0000) 2022-09-08T01:04:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:17 smithi055 ceph-mon[97033]: pgmap v6618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:04:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:17 smithi059 ceph-mon[97450]: pgmap v6618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:04:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:17.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:17.679455+0000) 2022-09-08T01:04:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:17.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:17.607571+0000) 2022-09-08T01:04:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:18.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:18.679703+0000) 2022-09-08T01:04:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:18.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:18.607806+0000) 2022-09-08T01:04:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:19.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:19.679901+0000) 2022-09-08T01:04:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:19.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:19.608074+0000) 2022-09-08T01:04:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:19 smithi055 ceph-mon[97033]: pgmap v6619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:04:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:19 smithi059 ceph-mon[97450]: pgmap v6619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:04:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:20.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:20.680068+0000) 2022-09-08T01:04:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:20.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:20.608257+0000) 2022-09-08T01:04:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:21.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:21.680262+0000) 2022-09-08T01:04:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:21.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:21.608523+0000) 2022-09-08T01:04:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:21 smithi055 ceph-mon[97033]: pgmap v6620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:04:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:21 smithi059 ceph-mon[97450]: pgmap v6620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:04:22.609 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:04:22.354+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:04:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:22.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:22.680470+0000) 2022-09-08T01:04:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:22.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:22.608765+0000) 2022-09-08T01:04:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:23.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:23.680610+0000) 2022-09-08T01:04:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:23.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:23.608933+0000) 2022-09-08T01:04:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:23 smithi055 ceph-mon[97033]: pgmap v6621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:04:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:23 smithi059 ceph-mon[97450]: pgmap v6621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:04:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:24.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:24.680816+0000) 2022-09-08T01:04:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:24.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:24.609092+0000) 2022-09-08T01:04:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:25.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:25.681007+0000) 2022-09-08T01:04:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:25.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:25.609328+0000) 2022-09-08T01:04:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:25 smithi055 ceph-mon[97033]: pgmap v6622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:04:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:25 smithi059 ceph-mon[97450]: pgmap v6622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:04:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:26.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:26.681254+0000) 2022-09-08T01:04:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:26.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:26.609508+0000) 2022-09-08T01:04:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:27 smithi055 ceph-mon[97033]: pgmap v6623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:04:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:27 smithi059 ceph-mon[97450]: pgmap v6623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:04:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:27.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:27.681471+0000) 2022-09-08T01:04:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:27.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:27.609697+0000) 2022-09-08T01:04:28.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:28.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:28.681602+0000) 2022-09-08T01:04:28.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:28.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:28.609844+0000) 2022-09-08T01:04:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:29.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:29.681805+0000) 2022-09-08T01:04:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:29.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:29.610075+0000) 2022-09-08T01:04:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:29 smithi055 ceph-mon[97033]: pgmap v6624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:29 smithi059 ceph-mon[97450]: pgmap v6624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:30.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:30.682058+0000) 2022-09-08T01:04:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:30.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:30.610328+0000) 2022-09-08T01:04:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:31.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:31.682279+0000) 2022-09-08T01:04:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:31.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:31.610523+0000) 2022-09-08T01:04:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:31 smithi055 ceph-mon[97033]: pgmap v6625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:04:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:31 smithi059 ceph-mon[97450]: pgmap v6625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:04:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:32.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:32.682491+0000) 2022-09-08T01:04:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:32.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:32.610738+0000) 2022-09-08T01:04:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:33.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:33.682771+0000) 2022-09-08T01:04:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:33.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:33.610910+0000) 2022-09-08T01:04:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:33 smithi055 ceph-mon[97033]: pgmap v6626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:33 smithi059 ceph-mon[97450]: pgmap v6626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:34.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:34.611121+0000) 2022-09-08T01:04:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:34.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:34.682999+0000) 2022-09-08T01:04:34.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:04:34.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:04:34.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:04:34.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:04:34.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:04:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:04:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:04:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:04:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:04:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:04:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:35.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:35.611356+0000) 2022-09-08T01:04:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:35.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:35.683214+0000) 2022-09-08T01:04:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:35 smithi055 ceph-mon[97033]: pgmap v6627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:04:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:35 smithi059 ceph-mon[97450]: pgmap v6627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:04:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:36.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:36.611593+0000) 2022-09-08T01:04:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:36.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:36.683427+0000) 2022-09-08T01:04:37.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:04:37.370+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:04:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:37.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:37.611819+0000) 2022-09-08T01:04:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:37 smithi055 ceph-mon[97033]: pgmap v6628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:04:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:37 smithi059 ceph-mon[97450]: pgmap v6628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:04:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:37.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:37.683622+0000) 2022-09-08T01:04:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:38.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:38.683859+0000) 2022-09-08T01:04:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:38.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:38.612023+0000) 2022-09-08T01:04:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:39.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:39.684050+0000) 2022-09-08T01:04:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:39.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:39.612232+0000) 2022-09-08T01:04:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:39 smithi055 ceph-mon[97033]: pgmap v6629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:39 smithi059 ceph-mon[97450]: pgmap v6629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:40.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:40.684256+0000) 2022-09-08T01:04:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:40.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:40.612499+0000) 2022-09-08T01:04:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:41.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:41.684456+0000) 2022-09-08T01:04:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:41.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:41.612757+0000) 2022-09-08T01:04:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:41 smithi055 ceph-mon[97033]: pgmap v6630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:04:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:41 smithi059 ceph-mon[97450]: pgmap v6630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:04:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:42.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:42.684714+0000) 2022-09-08T01:04:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:42.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:42.612939+0000) 2022-09-08T01:04:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:43.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:43.684936+0000) 2022-09-08T01:04:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:43.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:43.613166+0000) 2022-09-08T01:04:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:43 smithi055 ceph-mon[97033]: pgmap v6631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:43 smithi059 ceph-mon[97450]: pgmap v6631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:44.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:44.685113+0000) 2022-09-08T01:04:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:44.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:44.613393+0000) 2022-09-08T01:04:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:45.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:45.685325+0000) 2022-09-08T01:04:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:45.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:45.613709+0000) 2022-09-08T01:04:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:45 smithi055 ceph-mon[97033]: pgmap v6632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:45 smithi059 ceph-mon[97450]: pgmap v6632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:46.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:46.685537+0000) 2022-09-08T01:04:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:46.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:46.613898+0000) 2022-09-08T01:04:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:47 smithi055 ceph-mon[97033]: pgmap v6633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:04:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:47 smithi059 ceph-mon[97450]: pgmap v6633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:04:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:47.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:47.685725+0000) 2022-09-08T01:04:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:47.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:47.614110+0000) 2022-09-08T01:04:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:48.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:48.685881+0000) 2022-09-08T01:04:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:48.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:48.614241+0000) 2022-09-08T01:04:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:49.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:49.686096+0000) 2022-09-08T01:04:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:49.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:49.614417+0000) 2022-09-08T01:04:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:49 smithi055 ceph-mon[97033]: pgmap v6634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:49 smithi059 ceph-mon[97450]: pgmap v6634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:50.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:50.686335+0000) 2022-09-08T01:04:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:50.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:50.614555+0000) 2022-09-08T01:04:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:51.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:51.686608+0000) 2022-09-08T01:04:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:51.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:51.614784+0000) 2022-09-08T01:04:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:51 smithi055 ceph-mon[97033]: pgmap v6635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:04:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:51 smithi059 ceph-mon[97450]: pgmap v6635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:04:52.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:04:52.373+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:04:52.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:52.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:52.615031+0000) 2022-09-08T01:04:53.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:52.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:52.686889+0000) 2022-09-08T01:04:53.687 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:53.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:53.615276+0000) 2022-09-08T01:04:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:53 smithi055 ceph-mon[97033]: pgmap v6636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:53 smithi059 ceph-mon[97450]: pgmap v6636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:53.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:53.687091+0000) 2022-09-08T01:04:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:54.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:54.615534+0000) 2022-09-08T01:04:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:54.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:54.687293+0000) 2022-09-08T01:04:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:55.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:55.687505+0000) 2022-09-08T01:04:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:55.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:55.615803+0000) 2022-09-08T01:04:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:55 smithi055 ceph-mon[97033]: pgmap v6637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:55 smithi059 ceph-mon[97450]: pgmap v6637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:04:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:56.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:56.687749+0000) 2022-09-08T01:04:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:56.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:56.616044+0000) 2022-09-08T01:04:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:57 smithi055 ceph-mon[97033]: pgmap v6638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:04:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:57 smithi059 ceph-mon[97450]: pgmap v6638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:04:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:57.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:57.687944+0000) 2022-09-08T01:04:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:57.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:57.616284+0000) 2022-09-08T01:04:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:58.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:58.688189+0000) 2022-09-08T01:04:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:58.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:58.616507+0000) 2022-09-08T01:04:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:04:59.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:59.688388+0000) 2022-09-08T01:04:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:04:59.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:04:59.616677+0000) 2022-09-08T01:04:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:04:59 smithi055 ceph-mon[97033]: pgmap v6639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:04:59 smithi059 ceph-mon[97450]: pgmap v6639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:00.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:00.688530+0000) 2022-09-08T01:05:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:00.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:00.616872+0000) 2022-09-08T01:05:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:01.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:01.688771+0000) 2022-09-08T01:05:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:01.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:01.617093+0000) 2022-09-08T01:05:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:01 smithi055 ceph-mon[97033]: pgmap v6640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:05:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:01 smithi059 ceph-mon[97450]: pgmap v6640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:05:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:02.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:02.688998+0000) 2022-09-08T01:05:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:02.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:02.617288+0000) 2022-09-08T01:05:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:03.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:03.689218+0000) 2022-09-08T01:05:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:03.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:03.617531+0000) 2022-09-08T01:05:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:03 smithi055 ceph-mon[97033]: pgmap v6641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:03 smithi059 ceph-mon[97450]: pgmap v6641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:04.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:04.689429+0000) 2022-09-08T01:05:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:04.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:04.617830+0000) 2022-09-08T01:05:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:05.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:05.689713+0000) 2022-09-08T01:05:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:05.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:05.618014+0000) 2022-09-08T01:05:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:05 smithi055 ceph-mon[97033]: pgmap v6642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:05:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:05:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:05 smithi059 ceph-mon[97450]: pgmap v6642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:05:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:05:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:06.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:06.689954+0000) 2022-09-08T01:05:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:06.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:06.618261+0000) 2022-09-08T01:05:07.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:05:07.377+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:05:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:07.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:07.618511+0000) 2022-09-08T01:05:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:07 smithi055 ceph-mon[97033]: pgmap v6643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:05:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:07 smithi059 ceph-mon[97450]: pgmap v6643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:05:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:07.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:07.690192+0000) 2022-09-08T01:05:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:08.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:08.618782+0000) 2022-09-08T01:05:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:08.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:08.690426+0000) 2022-09-08T01:05:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:09 smithi059 ceph-mon[97450]: osdmap e499: 12 total, 11 up, 11 in 2022-09-08T01:05:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:09 smithi059 ceph-mon[97450]: pgmap v6645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:05:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:09.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:09.690607+0000) 2022-09-08T01:05:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:09.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:09.618936+0000) 2022-09-08T01:05:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:09 smithi055 ceph-mon[97033]: osdmap e499: 12 total, 11 up, 11 in 2022-09-08T01:05:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:09 smithi055 ceph-mon[97033]: pgmap v6645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:05:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:10.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:10.690902+0000) 2022-09-08T01:05:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:10.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:10.619161+0000) 2022-09-08T01:05:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:11.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:11.691074+0000) 2022-09-08T01:05:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:11.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:11.619328+0000) 2022-09-08T01:05:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:11 smithi055 ceph-mon[97033]: pgmap v6646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:05:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:11 smithi055 ceph-mon[97033]: osdmap e500: 12 total, 11 up, 11 in 2022-09-08T01:05:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:11 smithi059 ceph-mon[97450]: pgmap v6646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:05:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:11 smithi059 ceph-mon[97450]: osdmap e500: 12 total, 11 up, 11 in 2022-09-08T01:05:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:12.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:12.691192+0000) 2022-09-08T01:05:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:12.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:12.619555+0000) 2022-09-08T01:05:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:13.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:13.691393+0000) 2022-09-08T01:05:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:13.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:13.619766+0000) 2022-09-08T01:05:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:13 smithi055 ceph-mon[97033]: pgmap v6648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:05:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:13 smithi059 ceph-mon[97450]: pgmap v6648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:05:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:14.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:14.691677+0000) 2022-09-08T01:05:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:14.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:14.619971+0000) 2022-09-08T01:05:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:15.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:15.691896+0000) 2022-09-08T01:05:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:15.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:15.620215+0000) 2022-09-08T01:05:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:15 smithi055 ceph-mon[97033]: pgmap v6649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:05:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:15 smithi059 ceph-mon[97450]: pgmap v6649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:05:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:16.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:16.620413+0000) 2022-09-08T01:05:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:16.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:16.692118+0000) 2022-09-08T01:05:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:17 smithi055 ceph-mon[97033]: pgmap v6650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:05:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:17 smithi059 ceph-mon[97450]: pgmap v6650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:05:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:17.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:17.620653+0000) 2022-09-08T01:05:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:17.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:17.692346+0000) 2022-09-08T01:05:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:18.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:18.692480+0000) 2022-09-08T01:05:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:18.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:18.620854+0000) 2022-09-08T01:05:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:19.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:19.692730+0000) 2022-09-08T01:05:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:19.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:19.621061+0000) 2022-09-08T01:05:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:19 smithi055 ceph-mon[97033]: pgmap v6651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:05:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:19 smithi059 ceph-mon[97450]: pgmap v6651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:05:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:20.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:20.693012+0000) 2022-09-08T01:05:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:20.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:20.621318+0000) 2022-09-08T01:05:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:21.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:21.621563+0000) 2022-09-08T01:05:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:21.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:21.693258+0000) 2022-09-08T01:05:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:21 smithi055 ceph-mon[97033]: pgmap v6652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:05:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:21 smithi059 ceph-mon[97450]: pgmap v6652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:05:22.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:05:22.393+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:05:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:22.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:22.621834+0000) 2022-09-08T01:05:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:22.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:22.693503+0000) 2022-09-08T01:05:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:23.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:23.693734+0000) 2022-09-08T01:05:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:23.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:23.622066+0000) 2022-09-08T01:05:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:23 smithi055 ceph-mon[97033]: pgmap v6653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:05:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:23 smithi059 ceph-mon[97450]: pgmap v6653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:05:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:24.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:24.693970+0000) 2022-09-08T01:05:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:24.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:24.622331+0000) 2022-09-08T01:05:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:25.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:25.694206+0000) 2022-09-08T01:05:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:25.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:25.622547+0000) 2022-09-08T01:05:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:25 smithi055 ceph-mon[97033]: pgmap v6654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:05:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:25 smithi059 ceph-mon[97450]: pgmap v6654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:05:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:26.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:26.694416+0000) 2022-09-08T01:05:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:26.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:26.622705+0000) 2022-09-08T01:05:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:27 smithi055 ceph-mon[97033]: pgmap v6655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:05:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:27 smithi059 ceph-mon[97450]: pgmap v6655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:05:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:27.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:27.694594+0000) 2022-09-08T01:05:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:27.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:27.622928+0000) 2022-09-08T01:05:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:28.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:28.694860+0000) 2022-09-08T01:05:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:28.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:28.623148+0000) 2022-09-08T01:05:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:29.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:29.695057+0000) 2022-09-08T01:05:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:29.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:29.623425+0000) 2022-09-08T01:05:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:29 smithi055 ceph-mon[97033]: pgmap v6656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:29 smithi059 ceph-mon[97450]: pgmap v6656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:30.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:30.695279+0000) 2022-09-08T01:05:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:30.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:30.623725+0000) 2022-09-08T01:05:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:31.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:31.695478+0000) 2022-09-08T01:05:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:31.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:31.623969+0000) 2022-09-08T01:05:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:31 smithi055 ceph-mon[97033]: pgmap v6657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:05:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:31 smithi059 ceph-mon[97450]: pgmap v6657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:05:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:32.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:32.624104+0000) 2022-09-08T01:05:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:32.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:32.695724+0000) 2022-09-08T01:05:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:33.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:33.695900+0000) 2022-09-08T01:05:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:33.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:33.624319+0000) 2022-09-08T01:05:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:33 smithi055 ceph-mon[97033]: pgmap v6658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:33 smithi059 ceph-mon[97450]: pgmap v6658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:34.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:34.696068+0000) 2022-09-08T01:05:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:34.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:34.624480+0000) 2022-09-08T01:05:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:35.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:35.696302+0000) 2022-09-08T01:05:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:35.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:35.624744+0000) 2022-09-08T01:05:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:35 smithi055 ceph-mon[97033]: pgmap v6659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:05:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:05:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:05:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:05:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:35 smithi059 ceph-mon[97450]: pgmap v6659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:05:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:05:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:05:36.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:05:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:36.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:36.696551+0000) 2022-09-08T01:05:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:36.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:36.624892+0000) 2022-09-08T01:05:37.697 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:05:37.406+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:05:37.698 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:37.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:37.625093+0000) 2022-09-08T01:05:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:37 smithi055 ceph-mon[97033]: pgmap v6660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:05:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:37 smithi059 ceph-mon[97450]: pgmap v6660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:05:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:37.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:37.696836+0000) 2022-09-08T01:05:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:38.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:38.697061+0000) 2022-09-08T01:05:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:38.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:38.625263+0000) 2022-09-08T01:05:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:39.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:39.697322+0000) 2022-09-08T01:05:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:39.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:39.625451+0000) 2022-09-08T01:05:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:39 smithi055 ceph-mon[97033]: pgmap v6661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:39 smithi059 ceph-mon[97450]: pgmap v6661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:40.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:40.697560+0000) 2022-09-08T01:05:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:40.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:40.625727+0000) 2022-09-08T01:05:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:41.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:41.697801+0000) 2022-09-08T01:05:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:41.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:41.625963+0000) 2022-09-08T01:05:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:41 smithi055 ceph-mon[97033]: pgmap v6662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:05:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:41 smithi059 ceph-mon[97450]: pgmap v6662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:05:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:42.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:42.698047+0000) 2022-09-08T01:05:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:42.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:42.626142+0000) 2022-09-08T01:05:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:43.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:43.698283+0000) 2022-09-08T01:05:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:43.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:43.626387+0000) 2022-09-08T01:05:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:43 smithi055 ceph-mon[97033]: pgmap v6663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:43 smithi059 ceph-mon[97450]: pgmap v6663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:44.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:44.698548+0000) 2022-09-08T01:05:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:44.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:44.626628+0000) 2022-09-08T01:05:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:45.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:45.698744+0000) 2022-09-08T01:05:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:45.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:45.626873+0000) 2022-09-08T01:05:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:45 smithi055 ceph-mon[97033]: pgmap v6664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:45 smithi059 ceph-mon[97450]: pgmap v6664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:46.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:46.698913+0000) 2022-09-08T01:05:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:46.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:46.627066+0000) 2022-09-08T01:05:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:47 smithi055 ceph-mon[97033]: pgmap v6665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:05:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:47 smithi059 ceph-mon[97450]: pgmap v6665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:05:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:47.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:47.699143+0000) 2022-09-08T01:05:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:47.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:47.627247+0000) 2022-09-08T01:05:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:48.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:48.627474+0000) 2022-09-08T01:05:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:48.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:48.699376+0000) 2022-09-08T01:05:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:49.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:49.699590+0000) 2022-09-08T01:05:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:49.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:49.627670+0000) 2022-09-08T01:05:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:49 smithi055 ceph-mon[97033]: pgmap v6666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:49 smithi059 ceph-mon[97450]: pgmap v6666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:50.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:50.699922+0000) 2022-09-08T01:05:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:50.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:50.627905+0000) 2022-09-08T01:05:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:51.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:51.628132+0000) 2022-09-08T01:05:51.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:51.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:51.700125+0000) 2022-09-08T01:05:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:51 smithi055 ceph-mon[97033]: pgmap v6667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:05:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:51 smithi059 ceph-mon[97450]: pgmap v6667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:05:52.700 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:05:52.414+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:05:52.701 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:52.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:52.628341+0000) 2022-09-08T01:05:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:52.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:52.700382+0000) 2022-09-08T01:05:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:53.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:53.700603+0000) 2022-09-08T01:05:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:53.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:53.628532+0000) 2022-09-08T01:05:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:53 smithi055 ceph-mon[97033]: pgmap v6668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:53 smithi059 ceph-mon[97450]: pgmap v6668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:54.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:54.628677+0000) 2022-09-08T01:05:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:54.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:54.700756+0000) 2022-09-08T01:05:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:55.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:55.700920+0000) 2022-09-08T01:05:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:55.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:55.628805+0000) 2022-09-08T01:05:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:55 smithi055 ceph-mon[97033]: pgmap v6669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:55 smithi059 ceph-mon[97450]: pgmap v6669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:05:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:56.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:56.629056+0000) 2022-09-08T01:05:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:56.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:56.701165+0000) 2022-09-08T01:05:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:57 smithi055 ceph-mon[97033]: pgmap v6670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:05:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:57 smithi059 ceph-mon[97450]: pgmap v6670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:05:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:57.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:57.701400+0000) 2022-09-08T01:05:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:57.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:57.629250+0000) 2022-09-08T01:05:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:58.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:58.701617+0000) 2022-09-08T01:05:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:58.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:58.629482+0000) 2022-09-08T01:05:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:05:59.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:59.701866+0000) 2022-09-08T01:05:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:05:59.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:05:59.629690+0000) 2022-09-08T01:05:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:05:59 smithi055 ceph-mon[97033]: pgmap v6671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:05:59 smithi059 ceph-mon[97450]: pgmap v6671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:00.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:00.702012+0000) 2022-09-08T01:06:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:00.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:00.629865+0000) 2022-09-08T01:06:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:01.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:01.702208+0000) 2022-09-08T01:06:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:01.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:01.630102+0000) 2022-09-08T01:06:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:01 smithi055 ceph-mon[97033]: pgmap v6672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:06:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:01 smithi059 ceph-mon[97450]: pgmap v6672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:06:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:02.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:02.702401+0000) 2022-09-08T01:06:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:02.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:02.630330+0000) 2022-09-08T01:06:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:03.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:03.702665+0000) 2022-09-08T01:06:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:03.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:03.630553+0000) 2022-09-08T01:06:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:03 smithi055 ceph-mon[97033]: pgmap v6673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:03 smithi059 ceph-mon[97450]: pgmap v6673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:04.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:04.702947+0000) 2022-09-08T01:06:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:04.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:04.630739+0000) 2022-09-08T01:06:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:05.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:05.703212+0000) 2022-09-08T01:06:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:05.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:05.630925+0000) 2022-09-08T01:06:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:05 smithi055 ceph-mon[97033]: pgmap v6674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:06:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:06:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:05 smithi059 ceph-mon[97450]: pgmap v6674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:06:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:06:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:06.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:06.631103+0000) 2022-09-08T01:06:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:06.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:06.703433+0000) 2022-09-08T01:06:07.704 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:06:07.418+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:06:07.704 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:07.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:07.631328+0000) 2022-09-08T01:06:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:07 smithi055 ceph-mon[97033]: pgmap v6675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:06:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:07 smithi059 ceph-mon[97450]: pgmap v6675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:06:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:07.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:07.703610+0000) 2022-09-08T01:06:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:08.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:08.703755+0000) 2022-09-08T01:06:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:08.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:08.631740+0000) 2022-09-08T01:06:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:09 smithi059 ceph-mon[97450]: osdmap e501: 12 total, 11 up, 11 in 2022-09-08T01:06:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:09 smithi059 ceph-mon[97450]: pgmap v6677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:06:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:09.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:09.631963+0000) 2022-09-08T01:06:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:09.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:09.703912+0000) 2022-09-08T01:06:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:09 smithi055 ceph-mon[97033]: osdmap e501: 12 total, 11 up, 11 in 2022-09-08T01:06:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:09 smithi055 ceph-mon[97033]: pgmap v6677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:06:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:10.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:10.704148+0000) 2022-09-08T01:06:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:10.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:10.632150+0000) 2022-09-08T01:06:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:11.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:11.704417+0000) 2022-09-08T01:06:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:11.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:11.632323+0000) 2022-09-08T01:06:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:11 smithi055 ceph-mon[97033]: pgmap v6678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:06:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:11 smithi055 ceph-mon[97033]: osdmap e502: 12 total, 11 up, 11 in 2022-09-08T01:06:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:11 smithi059 ceph-mon[97450]: pgmap v6678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:06:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:11 smithi059 ceph-mon[97450]: osdmap e502: 12 total, 11 up, 11 in 2022-09-08T01:06:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:12.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:12.632554+0000) 2022-09-08T01:06:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:12.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:12.704727+0000) 2022-09-08T01:06:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:13.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:13.632802+0000) 2022-09-08T01:06:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:13.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:13.704937+0000) 2022-09-08T01:06:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:13 smithi055 ceph-mon[97033]: pgmap v6680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:06:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:13 smithi059 ceph-mon[97450]: pgmap v6680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:06:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:14.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:14.633037+0000) 2022-09-08T01:06:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:14.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:14.705164+0000) 2022-09-08T01:06:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:15.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:15.633247+0000) 2022-09-08T01:06:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:15.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:15.705370+0000) 2022-09-08T01:06:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:15 smithi055 ceph-mon[97033]: pgmap v6681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:06:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:15 smithi059 ceph-mon[97450]: pgmap v6681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:06:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:16.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:16.705609+0000) 2022-09-08T01:06:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:16.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:16.633462+0000) 2022-09-08T01:06:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:17 smithi055 ceph-mon[97033]: pgmap v6682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-08T01:06:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:17 smithi059 ceph-mon[97450]: pgmap v6682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-08T01:06:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:17.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:17.705879+0000) 2022-09-08T01:06:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:17.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:17.633607+0000) 2022-09-08T01:06:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:18.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:18.706113+0000) 2022-09-08T01:06:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:18.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:18.633759+0000) 2022-09-08T01:06:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:19.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:19.706355+0000) 2022-09-08T01:06:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:19.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:19.633940+0000) 2022-09-08T01:06:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:19 smithi055 ceph-mon[97033]: pgmap v6683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:06:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:19 smithi059 ceph-mon[97450]: pgmap v6683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:06:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:20.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:20.706577+0000) 2022-09-08T01:06:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:20.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:20.634148+0000) 2022-09-08T01:06:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:21.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:21.706792+0000) 2022-09-08T01:06:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:21.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:21.634381+0000) 2022-09-08T01:06:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:21 smithi055 ceph-mon[97033]: pgmap v6684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:06:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:21 smithi059 ceph-mon[97450]: pgmap v6684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:06:22.707 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:06:22.432+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:06:22.708 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:22.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:22.634558+0000) 2022-09-08T01:06:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:22.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:22.707072+0000) 2022-09-08T01:06:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:23.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:23.707338+0000) 2022-09-08T01:06:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:23.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:23.634785+0000) 2022-09-08T01:06:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:23 smithi055 ceph-mon[97033]: pgmap v6685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:06:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:23 smithi059 ceph-mon[97450]: pgmap v6685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:06:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:24.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:24.707545+0000) 2022-09-08T01:06:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:24.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:24.634941+0000) 2022-09-08T01:06:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:25.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:25.707766+0000) 2022-09-08T01:06:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:25.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:25.635183+0000) 2022-09-08T01:06:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:25 smithi055 ceph-mon[97033]: pgmap v6686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:06:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:25 smithi059 ceph-mon[97450]: pgmap v6686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:06:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:26.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:26.707973+0000) 2022-09-08T01:06:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:26.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:26.635435+0000) 2022-09-08T01:06:27.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:27 smithi055 ceph-mon[97033]: pgmap v6687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:06:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:27 smithi059 ceph-mon[97450]: pgmap v6687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:06:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:27.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:27.708185+0000) 2022-09-08T01:06:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:27.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:27.635593+0000) 2022-09-08T01:06:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:28.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:28.708319+0000) 2022-09-08T01:06:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:28.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:28.635809+0000) 2022-09-08T01:06:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:29.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:29.708507+0000) 2022-09-08T01:06:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:29.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:29.636029+0000) 2022-09-08T01:06:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:29 smithi055 ceph-mon[97033]: pgmap v6688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:29 smithi059 ceph-mon[97450]: pgmap v6688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:30.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:30.708664+0000) 2022-09-08T01:06:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:30.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:30.636288+0000) 2022-09-08T01:06:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:31.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:31.708853+0000) 2022-09-08T01:06:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:31.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:31.636554+0000) 2022-09-08T01:06:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:31 smithi055 ceph-mon[97033]: pgmap v6689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:06:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:31 smithi059 ceph-mon[97450]: pgmap v6689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:06:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:32.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:32.636752+0000) 2022-09-08T01:06:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:32.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:32.709057+0000) 2022-09-08T01:06:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:33.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:33.709252+0000) 2022-09-08T01:06:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:33.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:33.636928+0000) 2022-09-08T01:06:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:33 smithi055 ceph-mon[97033]: pgmap v6690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:33 smithi059 ceph-mon[97450]: pgmap v6690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:34.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:34.709481+0000) 2022-09-08T01:06:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:34.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:34.637072+0000) 2022-09-08T01:06:35.855 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:35 smithi055 ceph-mon[97033]: pgmap v6691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:35.856 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:06:35.856 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:06:35.857 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:06:35.900 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:35 smithi059 ceph-mon[97450]: pgmap v6691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:35.900 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:06:35.900 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:06:35.901 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:06:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:35.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:35.709740+0000) 2022-09-08T01:06:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:35.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:35.637322+0000) 2022-09-08T01:06:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:36.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:36.709994+0000) 2022-09-08T01:06:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:36.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:36.637569+0000) 2022-09-08T01:06:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:06:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:06:37.711 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:06:37.440+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:06:37.712 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:37.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:37.637788+0000) 2022-09-08T01:06:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:37.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:37.710232+0000) 2022-09-08T01:06:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:37 smithi055 ceph-mon[97033]: pgmap v6692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:06:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:37 smithi059 ceph-mon[97450]: pgmap v6692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:06:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:38.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:38.710462+0000) 2022-09-08T01:06:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:38.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:38.637967+0000) 2022-09-08T01:06:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:39.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:39.710694+0000) 2022-09-08T01:06:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:39.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:39.638135+0000) 2022-09-08T01:06:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:39 smithi055 ceph-mon[97033]: pgmap v6693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:39 smithi059 ceph-mon[97450]: pgmap v6693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:40.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:40.710917+0000) 2022-09-08T01:06:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:40.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:40.638407+0000) 2022-09-08T01:06:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:41.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:41.711144+0000) 2022-09-08T01:06:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:41.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:41.638684+0000) 2022-09-08T01:06:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:41 smithi055 ceph-mon[97033]: pgmap v6694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:06:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:41 smithi059 ceph-mon[97450]: pgmap v6694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:06:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:42.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:42.711386+0000) 2022-09-08T01:06:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:42.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:42.638821+0000) 2022-09-08T01:06:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:43.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:43.711551+0000) 2022-09-08T01:06:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:43.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:43.639053+0000) 2022-09-08T01:06:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:43 smithi055 ceph-mon[97033]: pgmap v6695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:43 smithi059 ceph-mon[97450]: pgmap v6695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:44.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:44.711740+0000) 2022-09-08T01:06:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:44.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:44.639277+0000) 2022-09-08T01:06:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:45.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:45.711976+0000) 2022-09-08T01:06:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:45.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:45.639537+0000) 2022-09-08T01:06:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:45 smithi055 ceph-mon[97033]: pgmap v6696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:45 smithi059 ceph-mon[97450]: pgmap v6696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:46.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:46.712183+0000) 2022-09-08T01:06:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:46.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:46.639797+0000) 2022-09-08T01:06:47.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:47 smithi055 ceph-mon[97033]: pgmap v6697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:06:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:47 smithi059 ceph-mon[97450]: pgmap v6697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:06:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:47.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:47.712353+0000) 2022-09-08T01:06:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:47.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:47.640037+0000) 2022-09-08T01:06:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:48.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:48.712590+0000) 2022-09-08T01:06:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:48.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:48.640208+0000) 2022-09-08T01:06:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:49.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:49.712802+0000) 2022-09-08T01:06:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:49.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:49.640433+0000) 2022-09-08T01:06:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:49 smithi055 ceph-mon[97033]: pgmap v6698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:49 smithi059 ceph-mon[97450]: pgmap v6698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:50.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:50.713037+0000) 2022-09-08T01:06:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:50.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:50.640679+0000) 2022-09-08T01:06:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:51.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:51.713266+0000) 2022-09-08T01:06:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:51.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:51.640908+0000) 2022-09-08T01:06:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:51 smithi055 ceph-mon[97033]: pgmap v6699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:06:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:51 smithi059 ceph-mon[97450]: pgmap v6699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:06:52.714 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:06:52.442+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:06:52.715 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:52.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:52.641151+0000) 2022-09-08T01:06:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:52.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:52.713473+0000) 2022-09-08T01:06:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:53.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:53.713731+0000) 2022-09-08T01:06:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:53.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:53.641378+0000) 2022-09-08T01:06:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:53 smithi055 ceph-mon[97033]: pgmap v6700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:53 smithi059 ceph-mon[97450]: pgmap v6700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:54.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:54.713931+0000) 2022-09-08T01:06:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:54.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:54.641629+0000) 2022-09-08T01:06:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:55.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:55.714088+0000) 2022-09-08T01:06:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:55.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:55.641927+0000) 2022-09-08T01:06:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:55 smithi055 ceph-mon[97033]: pgmap v6701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:55 smithi059 ceph-mon[97450]: pgmap v6701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:06:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:56.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:56.714316+0000) 2022-09-08T01:06:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:56.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:56.642170+0000) 2022-09-08T01:06:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:57 smithi055 ceph-mon[97033]: pgmap v6702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:06:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:57 smithi059 ceph-mon[97450]: pgmap v6702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:06:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:57.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:57.642397+0000) 2022-09-08T01:06:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:57.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:57.714567+0000) 2022-09-08T01:06:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:58.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:58.642671+0000) 2022-09-08T01:06:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:58.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:58.714812+0000) 2022-09-08T01:06:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:06:59.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:59.714986+0000) 2022-09-08T01:06:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:06:59.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:06:59.642923+0000) 2022-09-08T01:06:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:06:59 smithi055 ceph-mon[97033]: pgmap v6703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:06:59 smithi059 ceph-mon[97450]: pgmap v6703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:00.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:00.643133+0000) 2022-09-08T01:07:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:00.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:00.715163+0000) 2022-09-08T01:07:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:01.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:01.715380+0000) 2022-09-08T01:07:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:01.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:01.643335+0000) 2022-09-08T01:07:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:01 smithi055 ceph-mon[97033]: pgmap v6704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:07:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:01 smithi059 ceph-mon[97450]: pgmap v6704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:07:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:02.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:02.715606+0000) 2022-09-08T01:07:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:02.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:02.643572+0000) 2022-09-08T01:07:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:03.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:03.643803+0000) 2022-09-08T01:07:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:03.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:03.715868+0000) 2022-09-08T01:07:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:03 smithi055 ceph-mon[97033]: pgmap v6705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:03 smithi059 ceph-mon[97450]: pgmap v6705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:04.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:04.644056+0000) 2022-09-08T01:07:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:04.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:04.716070+0000) 2022-09-08T01:07:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:05.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:05.716276+0000) 2022-09-08T01:07:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:05.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:05.644322+0000) 2022-09-08T01:07:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:05 smithi055 ceph-mon[97033]: pgmap v6706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:07:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:07:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:05 smithi059 ceph-mon[97450]: pgmap v6706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:07:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:07:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:06.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:06.716465+0000) 2022-09-08T01:07:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:06.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:06.644584+0000) 2022-09-08T01:07:07.717 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:07:07.451+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:07:07.717 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:07.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:07.644776+0000) 2022-09-08T01:07:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:07 smithi055 ceph-mon[97033]: pgmap v6707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:07:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:07 smithi059 ceph-mon[97450]: pgmap v6707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:07:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:07.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:07.716621+0000) 2022-09-08T01:07:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:08.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:08.644911+0000) 2022-09-08T01:07:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:08.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:08.716811+0000) 2022-09-08T01:07:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:09 smithi059 ceph-mon[97450]: osdmap e503: 12 total, 11 up, 11 in 2022-09-08T01:07:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:09 smithi059 ceph-mon[97450]: pgmap v6709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:07:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:09.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:09.717013+0000) 2022-09-08T01:07:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:09.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:09.645152+0000) 2022-09-08T01:07:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:09 smithi055 ceph-mon[97033]: osdmap e503: 12 total, 11 up, 11 in 2022-09-08T01:07:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:09 smithi055 ceph-mon[97033]: pgmap v6709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:07:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:10.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:10.645349+0000) 2022-09-08T01:07:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:10.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:10.717196+0000) 2022-09-08T01:07:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:11.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:11.645585+0000) 2022-09-08T01:07:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:11.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:11.717429+0000) 2022-09-08T01:07:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:11 smithi055 ceph-mon[97033]: pgmap v6710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:07:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:11 smithi055 ceph-mon[97033]: osdmap e504: 12 total, 11 up, 11 in 2022-09-08T01:07:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:11 smithi059 ceph-mon[97450]: pgmap v6710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:07:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:11 smithi059 ceph-mon[97450]: osdmap e504: 12 total, 11 up, 11 in 2022-09-08T01:07:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:12.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:12.645793+0000) 2022-09-08T01:07:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:12.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:12.717570+0000) 2022-09-08T01:07:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:13.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:13.646015+0000) 2022-09-08T01:07:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:13.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:13.717768+0000) 2022-09-08T01:07:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:13 smithi055 ceph-mon[97033]: pgmap v6712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T01:07:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:13 smithi059 ceph-mon[97450]: pgmap v6712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T01:07:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:14.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:14.717967+0000) 2022-09-08T01:07:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:14.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:14.646194+0000) 2022-09-08T01:07:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:15.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:15.718127+0000) 2022-09-08T01:07:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:15.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:15.646363+0000) 2022-09-08T01:07:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:15 smithi055 ceph-mon[97033]: pgmap v6713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:07:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:15 smithi059 ceph-mon[97450]: pgmap v6713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:07:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:16.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:16.718297+0000) 2022-09-08T01:07:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:16.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:16.646609+0000) 2022-09-08T01:07:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:17 smithi055 ceph-mon[97033]: pgmap v6714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-08T01:07:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:17 smithi059 ceph-mon[97450]: pgmap v6714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-08T01:07:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:17.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:17.718508+0000) 2022-09-08T01:07:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:17.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:17.646821+0000) 2022-09-08T01:07:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:18.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:18.647011+0000) 2022-09-08T01:07:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:18.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:18.718772+0000) 2022-09-08T01:07:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:19.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:19.719044+0000) 2022-09-08T01:07:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:19.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:19.647250+0000) 2022-09-08T01:07:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:19 smithi055 ceph-mon[97033]: pgmap v6715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:07:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:19 smithi059 ceph-mon[97450]: pgmap v6715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:07:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:20.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:20.719308+0000) 2022-09-08T01:07:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:20.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:20.647504+0000) 2022-09-08T01:07:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:21.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:21.719560+0000) 2022-09-08T01:07:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:21.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:21.647731+0000) 2022-09-08T01:07:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:21 smithi055 ceph-mon[97033]: pgmap v6716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:07:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:21 smithi059 ceph-mon[97450]: pgmap v6716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:07:22.720 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:22.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:22.647971+0000) 2022-09-08T01:07:22.722 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:07:22.460+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:07:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:22.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:22.719772+0000) 2022-09-08T01:07:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:23.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:23.719989+0000) 2022-09-08T01:07:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:23.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:23.648216+0000) 2022-09-08T01:07:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:23 smithi055 ceph-mon[97033]: pgmap v6717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:07:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:23 smithi059 ceph-mon[97450]: pgmap v6717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:07:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:24.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:24.720188+0000) 2022-09-08T01:07:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:24.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:24.648448+0000) 2022-09-08T01:07:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:25.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:25.720435+0000) 2022-09-08T01:07:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:25.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:25.648665+0000) 2022-09-08T01:07:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:25 smithi055 ceph-mon[97033]: pgmap v6718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:07:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:25 smithi059 ceph-mon[97450]: pgmap v6718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:07:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:26.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:26.720668+0000) 2022-09-08T01:07:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:26.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:26.648855+0000) 2022-09-08T01:07:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:27 smithi055 ceph-mon[97033]: pgmap v6719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:07:27.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:27 smithi059 ceph-mon[97450]: pgmap v6719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:07:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:27.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:27.720859+0000) 2022-09-08T01:07:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:27.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:27.649091+0000) 2022-09-08T01:07:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:28.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:28.721077+0000) 2022-09-08T01:07:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:28.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:28.649285+0000) 2022-09-08T01:07:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:29.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:29.721236+0000) 2022-09-08T01:07:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:29.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:29.649502+0000) 2022-09-08T01:07:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:29 smithi055 ceph-mon[97033]: pgmap v6720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:29 smithi059 ceph-mon[97450]: pgmap v6720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:30.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:30.721455+0000) 2022-09-08T01:07:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:30.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:30.649703+0000) 2022-09-08T01:07:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:31.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:31.721650+0000) 2022-09-08T01:07:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:31.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:31.649878+0000) 2022-09-08T01:07:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:31 smithi055 ceph-mon[97033]: pgmap v6721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:07:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:31 smithi059 ceph-mon[97450]: pgmap v6721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:07:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:32.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:32.721787+0000) 2022-09-08T01:07:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:32.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:32.650115+0000) 2022-09-08T01:07:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:33.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:33.721955+0000) 2022-09-08T01:07:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:33.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:33.650304+0000) 2022-09-08T01:07:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:33 smithi055 ceph-mon[97033]: pgmap v6722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:33 smithi059 ceph-mon[97450]: pgmap v6722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:34.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:34.722160+0000) 2022-09-08T01:07:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:34.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:34.650533+0000) 2022-09-08T01:07:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:35.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:35.722314+0000) 2022-09-08T01:07:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:35.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:35.650725+0000) 2022-09-08T01:07:35.963 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:35 smithi055 ceph-mon[97033]: pgmap v6723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:35.975 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:35 smithi059 ceph-mon[97450]: pgmap v6723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:07:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:07:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:07:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:07:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:07:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:07:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:07:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:07:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:36.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:36.722540+0000) 2022-09-08T01:07:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:36.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:36.651009+0000) 2022-09-08T01:07:37.723 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:37.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:37.651246+0000) 2022-09-08T01:07:37.724 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:07:37.469+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:07:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:37 smithi055 ceph-mon[97033]: pgmap v6724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:07:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:37 smithi059 ceph-mon[97450]: pgmap v6724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:07:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:37.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:37.722797+0000) 2022-09-08T01:07:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:38.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:38.722959+0000) 2022-09-08T01:07:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:38.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:38.651451+0000) 2022-09-08T01:07:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:39 smithi055 ceph-mon[97033]: pgmap v6725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:39 smithi059 ceph-mon[97450]: pgmap v6725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:39.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:39.651615+0000) 2022-09-08T01:07:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:39.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:39.723195+0000) 2022-09-08T01:07:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:40.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:40.651778+0000) 2022-09-08T01:07:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:40.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:40.723428+0000) 2022-09-08T01:07:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:41 smithi055 ceph-mon[97033]: pgmap v6726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:07:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:41 smithi059 ceph-mon[97450]: pgmap v6726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:07:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:41.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:41.652021+0000) 2022-09-08T01:07:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:41.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:41.723693+0000) 2022-09-08T01:07:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:42.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:42.723878+0000) 2022-09-08T01:07:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:42.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:42.652236+0000) 2022-09-08T01:07:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:43 smithi055 ceph-mon[97033]: pgmap v6727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:43 smithi059 ceph-mon[97450]: pgmap v6727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:43.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:43.724100+0000) 2022-09-08T01:07:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:43.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:43.652445+0000) 2022-09-08T01:07:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:44.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:44.652670+0000) 2022-09-08T01:07:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:44.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:44.724352+0000) 2022-09-08T01:07:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:45 smithi055 ceph-mon[97033]: pgmap v6728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:45 smithi059 ceph-mon[97450]: pgmap v6728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:45.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:45.724572+0000) 2022-09-08T01:07:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:45.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:45.652885+0000) 2022-09-08T01:07:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:46.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:46.653171+0000) 2022-09-08T01:07:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:46.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:46.724795+0000) 2022-09-08T01:07:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:47 smithi055 ceph-mon[97033]: pgmap v6729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:07:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:47 smithi059 ceph-mon[97450]: pgmap v6729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:07:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:47.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:47.724970+0000) 2022-09-08T01:07:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:47.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:47.653455+0000) 2022-09-08T01:07:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:48.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:48.725190+0000) 2022-09-08T01:07:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:48.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:48.653716+0000) 2022-09-08T01:07:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:49 smithi055 ceph-mon[97033]: pgmap v6730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:49 smithi059 ceph-mon[97450]: pgmap v6730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:49.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:49.725403+0000) 2022-09-08T01:07:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:49.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:49.653933+0000) 2022-09-08T01:07:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:50.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:50.725627+0000) 2022-09-08T01:07:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:50.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:50.654117+0000) 2022-09-08T01:07:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:51 smithi055 ceph-mon[97033]: pgmap v6731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:07:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:51 smithi059 ceph-mon[97450]: pgmap v6731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:07:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:51.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:51.725864+0000) 2022-09-08T01:07:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:51.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:51.654259+0000) 2022-09-08T01:07:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:52.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:52.654422+0000) 2022-09-08T01:07:52.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:07:52.476+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:07:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:52.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:52.726064+0000) 2022-09-08T01:07:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:53 smithi055 ceph-mon[97033]: pgmap v6732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:53 smithi059 ceph-mon[97450]: pgmap v6732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:53.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:53.726233+0000) 2022-09-08T01:07:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:53.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:53.654682+0000) 2022-09-08T01:07:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:54.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:54.726421+0000) 2022-09-08T01:07:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:54.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:54.654941+0000) 2022-09-08T01:07:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:55 smithi055 ceph-mon[97033]: pgmap v6733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:55 smithi059 ceph-mon[97450]: pgmap v6733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:07:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:55.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:55.726620+0000) 2022-09-08T01:07:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:55.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:55.655195+0000) 2022-09-08T01:07:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:56.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:56.726856+0000) 2022-09-08T01:07:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:56.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:56.655425+0000) 2022-09-08T01:07:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:57 smithi055 ceph-mon[97033]: pgmap v6734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:07:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:57 smithi059 ceph-mon[97450]: pgmap v6734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:07:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:57.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:57.727084+0000) 2022-09-08T01:07:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:57.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:57.655678+0000) 2022-09-08T01:07:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:58.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:58.727251+0000) 2022-09-08T01:07:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:58.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:58.655904+0000) 2022-09-08T01:07:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:07:59 smithi055 ceph-mon[97033]: pgmap v6735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:07:59 smithi059 ceph-mon[97450]: pgmap v6735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:07:59.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:59.727466+0000) 2022-09-08T01:08:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:07:59.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:07:59.656127+0000) 2022-09-08T01:08:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:00.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:00.727590+0000) 2022-09-08T01:08:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:00.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:00.656381+0000) 2022-09-08T01:08:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:01 smithi055 ceph-mon[97033]: pgmap v6736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:08:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:01 smithi059 ceph-mon[97450]: pgmap v6736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:08:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:01.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:01.727804+0000) 2022-09-08T01:08:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:01.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:01.656622+0000) 2022-09-08T01:08:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:02.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:02.728040+0000) 2022-09-08T01:08:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:02.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:02.656867+0000) 2022-09-08T01:08:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:03 smithi055 ceph-mon[97033]: pgmap v6737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:03 smithi059 ceph-mon[97450]: pgmap v6737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:03.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:03.728233+0000) 2022-09-08T01:08:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:03.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:03.657108+0000) 2022-09-08T01:08:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:04.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:04.728362+0000) 2022-09-08T01:08:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:04.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:04.657318+0000) 2022-09-08T01:08:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:05 smithi055 ceph-mon[97033]: pgmap v6738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:08:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:08:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:05 smithi059 ceph-mon[97450]: pgmap v6738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:08:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:08:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:05.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:05.728571+0000) 2022-09-08T01:08:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:05.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:05.657586+0000) 2022-09-08T01:08:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:06.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:06.728796+0000) 2022-09-08T01:08:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:06.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:06.657903+0000) 2022-09-08T01:08:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:07 smithi055 ceph-mon[97033]: pgmap v6739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:08:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:07 smithi059 ceph-mon[97450]: pgmap v6739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:08:07.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:08:07.493+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:08:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:07.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:07.728994+0000) 2022-09-08T01:08:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:07.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:07.658152+0000) 2022-09-08T01:08:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:08.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:08.729173+0000) 2022-09-08T01:08:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:08.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:08.658338+0000) 2022-09-08T01:08:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:09 smithi059 ceph-mon[97450]: osdmap e505: 12 total, 11 up, 11 in 2022-09-08T01:08:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:09 smithi059 ceph-mon[97450]: pgmap v6741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:08:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:09 smithi055 ceph-mon[97033]: osdmap e505: 12 total, 11 up, 11 in 2022-09-08T01:08:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:09 smithi055 ceph-mon[97033]: pgmap v6741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:08:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:09.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:09.729388+0000) 2022-09-08T01:08:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:09.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:09.658475+0000) 2022-09-08T01:08:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:10.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:10.658693+0000) 2022-09-08T01:08:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:10.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:10.729595+0000) 2022-09-08T01:08:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:11 smithi055 ceph-mon[97033]: pgmap v6742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:08:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:11 smithi055 ceph-mon[97033]: osdmap e506: 12 total, 11 up, 11 in 2022-09-08T01:08:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:11 smithi059 ceph-mon[97450]: pgmap v6742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:08:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:11 smithi059 ceph-mon[97450]: osdmap e506: 12 total, 11 up, 11 in 2022-09-08T01:08:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:11.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:11.729853+0000) 2022-09-08T01:08:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:11.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:11.658898+0000) 2022-09-08T01:08:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:12.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:12.730087+0000) 2022-09-08T01:08:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:12.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:12.659069+0000) 2022-09-08T01:08:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:13 smithi055 ceph-mon[97033]: pgmap v6744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T01:08:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:13 smithi059 ceph-mon[97450]: pgmap v6744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T01:08:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:13.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:13.730325+0000) 2022-09-08T01:08:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:13.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:13.659286+0000) 2022-09-08T01:08:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:14.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:14.730567+0000) 2022-09-08T01:08:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:14.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:14.659506+0000) 2022-09-08T01:08:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:15 smithi055 ceph-mon[97033]: pgmap v6745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:08:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:15 smithi059 ceph-mon[97450]: pgmap v6745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:08:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:15.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:15.730760+0000) 2022-09-08T01:08:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:15.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:15.659702+0000) 2022-09-08T01:08:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:16.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:16.731009+0000) 2022-09-08T01:08:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:16.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:16.659945+0000) 2022-09-08T01:08:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:17 smithi055 ceph-mon[97033]: pgmap v6746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T01:08:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:17 smithi059 ceph-mon[97450]: pgmap v6746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T01:08:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:17.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:17.660176+0000) 2022-09-08T01:08:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:17.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:17.731224+0000) 2022-09-08T01:08:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:18.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:18.731428+0000) 2022-09-08T01:08:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:18.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:18.660351+0000) 2022-09-08T01:08:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:19 smithi055 ceph-mon[97033]: pgmap v6747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:08:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:19 smithi059 ceph-mon[97450]: pgmap v6747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:08:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:19.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:19.731688+0000) 2022-09-08T01:08:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:19.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:19.660596+0000) 2022-09-08T01:08:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:20.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:20.731921+0000) 2022-09-08T01:08:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:20.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:20.660874+0000) 2022-09-08T01:08:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:21 smithi055 ceph-mon[97033]: pgmap v6748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:08:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:21 smithi059 ceph-mon[97450]: pgmap v6748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:08:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:21.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:21.732109+0000) 2022-09-08T01:08:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:21.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:21.661093+0000) 2022-09-08T01:08:22.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:08:22.496+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:08:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:22.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:22.732286+0000) 2022-09-08T01:08:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:22.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:22.661225+0000) 2022-09-08T01:08:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:23 smithi055 ceph-mon[97033]: pgmap v6749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:08:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:23 smithi059 ceph-mon[97450]: pgmap v6749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:08:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:23.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:23.732441+0000) 2022-09-08T01:08:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:23.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:23.661389+0000) 2022-09-08T01:08:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:24.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:24.732661+0000) 2022-09-08T01:08:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:24.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:24.661552+0000) 2022-09-08T01:08:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:25 smithi055 ceph-mon[97033]: pgmap v6750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:08:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:25 smithi059 ceph-mon[97450]: pgmap v6750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:08:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:25.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:25.732926+0000) 2022-09-08T01:08:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:25.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:25.661802+0000) 2022-09-08T01:08:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:26.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:26.733120+0000) 2022-09-08T01:08:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:26.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:26.662036+0000) 2022-09-08T01:08:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:27 smithi055 ceph-mon[97033]: pgmap v6751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:08:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:27 smithi059 ceph-mon[97450]: pgmap v6751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:08:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:27.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:27.733338+0000) 2022-09-08T01:08:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:27.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:27.662286+0000) 2022-09-08T01:08:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:28.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:28.733555+0000) 2022-09-08T01:08:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:28.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:28.662491+0000) 2022-09-08T01:08:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:29 smithi055 ceph-mon[97033]: pgmap v6752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:29 smithi059 ceph-mon[97450]: pgmap v6752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:29.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:29.733798+0000) 2022-09-08T01:08:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:29.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:29.662730+0000) 2022-09-08T01:08:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:30.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:30.733983+0000) 2022-09-08T01:08:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:30.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:30.662926+0000) 2022-09-08T01:08:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:31 smithi055 ceph-mon[97033]: pgmap v6753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:08:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:31 smithi059 ceph-mon[97450]: pgmap v6753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:08:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:31.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:31.734172+0000) 2022-09-08T01:08:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:31.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:31.663173+0000) 2022-09-08T01:08:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:32.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:32.734312+0000) 2022-09-08T01:08:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:32.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:32.663452+0000) 2022-09-08T01:08:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:33 smithi055 ceph-mon[97033]: pgmap v6754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:33 smithi059 ceph-mon[97450]: pgmap v6754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:33.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:33.734459+0000) 2022-09-08T01:08:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:33.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:33.663590+0000) 2022-09-08T01:08:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:34.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:34.734688+0000) 2022-09-08T01:08:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:34.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:34.663815+0000) 2022-09-08T01:08:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:35 smithi055 ceph-mon[97033]: pgmap v6755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:35 smithi059 ceph-mon[97450]: pgmap v6755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:35.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:35.734898+0000) 2022-09-08T01:08:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:35.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:35.664037+0000) 2022-09-08T01:08:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:36.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:36.664223+0000) 2022-09-08T01:08:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:36.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:36.735066+0000) 2022-09-08T01:08:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:08:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:08:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:08:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:37 smithi059 ceph-mon[97450]: pgmap v6756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:08:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:08:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:08:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:08:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:08:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:37 smithi055 ceph-mon[97033]: pgmap v6756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:08:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:08:37.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:08:37.496+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:08:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:37.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:37.735282+0000) 2022-09-08T01:08:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:37.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:37.664453+0000) 2022-09-08T01:08:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:38.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:38.735502+0000) 2022-09-08T01:08:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:38.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:38.664734+0000) 2022-09-08T01:08:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:39 smithi055 ceph-mon[97033]: pgmap v6757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:39 smithi059 ceph-mon[97450]: pgmap v6757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:39.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:39.735736+0000) 2022-09-08T01:08:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:39.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:39.664974+0000) 2022-09-08T01:08:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:40.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:40.735951+0000) 2022-09-08T01:08:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:40.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:40.665153+0000) 2022-09-08T01:08:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:41 smithi055 ceph-mon[97033]: pgmap v6758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:08:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:41 smithi059 ceph-mon[97450]: pgmap v6758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:08:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:41.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:41.736194+0000) 2022-09-08T01:08:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:41.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:41.665387+0000) 2022-09-08T01:08:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:42.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:42.736400+0000) 2022-09-08T01:08:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:42.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:42.665534+0000) 2022-09-08T01:08:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:43 smithi055 ceph-mon[97033]: pgmap v6759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:43 smithi059 ceph-mon[97450]: pgmap v6759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:43.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:43.736570+0000) 2022-09-08T01:08:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:43.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:43.665774+0000) 2022-09-08T01:08:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:44.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:44.736769+0000) 2022-09-08T01:08:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:44.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:44.666035+0000) 2022-09-08T01:08:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:45 smithi055 ceph-mon[97033]: pgmap v6760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:45 smithi059 ceph-mon[97450]: pgmap v6760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:45.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:45.736991+0000) 2022-09-08T01:08:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:45.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:45.666329+0000) 2022-09-08T01:08:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:46.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:46.737181+0000) 2022-09-08T01:08:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:46.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:46.666498+0000) 2022-09-08T01:08:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:47 smithi055 ceph-mon[97033]: pgmap v6761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:08:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:47 smithi059 ceph-mon[97450]: pgmap v6761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:08:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:47.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:47.737354+0000) 2022-09-08T01:08:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:47.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:47.666737+0000) 2022-09-08T01:08:49.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:48.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:48.737593+0000) 2022-09-08T01:08:49.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:48.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:48.666985+0000) 2022-09-08T01:08:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:49 smithi055 ceph-mon[97033]: pgmap v6762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:49 smithi059 ceph-mon[97450]: pgmap v6762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:49.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:49.667203+0000) 2022-09-08T01:08:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:49.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:49.737878+0000) 2022-09-08T01:08:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:50.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:50.738045+0000) 2022-09-08T01:08:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:50.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:50.667418+0000) 2022-09-08T01:08:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:51 smithi055 ceph-mon[97033]: pgmap v6763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:08:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:51 smithi059 ceph-mon[97450]: pgmap v6763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:08:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:51.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:51.738277+0000) 2022-09-08T01:08:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:51.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:51.667600+0000) 2022-09-08T01:08:52.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:08:52.512+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:08:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:52.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:52.738486+0000) 2022-09-08T01:08:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:52.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:52.667777+0000) 2022-09-08T01:08:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:53 smithi055 ceph-mon[97033]: pgmap v6764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:53 smithi059 ceph-mon[97450]: pgmap v6764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:53.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:53.738732+0000) 2022-09-08T01:08:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:53.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:53.668025+0000) 2022-09-08T01:08:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:54.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:54.738965+0000) 2022-09-08T01:08:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:54.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:54.668224+0000) 2022-09-08T01:08:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:55 smithi055 ceph-mon[97033]: pgmap v6765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:55 smithi059 ceph-mon[97450]: pgmap v6765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:08:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:55.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:55.739150+0000) 2022-09-08T01:08:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:55.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:55.668457+0000) 2022-09-08T01:08:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:56.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:56.739361+0000) 2022-09-08T01:08:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:56.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:56.668701+0000) 2022-09-08T01:08:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:57 smithi055 ceph-mon[97033]: pgmap v6766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:08:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:57 smithi059 ceph-mon[97450]: pgmap v6766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:08:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:57.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:57.739600+0000) 2022-09-08T01:08:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:57.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:57.668921+0000) 2022-09-08T01:08:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:58.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:58.669100+0000) 2022-09-08T01:08:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:58.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:58.739767+0000) 2022-09-08T01:08:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:08:59 smithi055 ceph-mon[97033]: pgmap v6767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:08:59 smithi059 ceph-mon[97450]: pgmap v6767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:08:59.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:59.739998+0000) 2022-09-08T01:09:00.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:08:59.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:08:59.669324+0000) 2022-09-08T01:09:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:00.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:00.740146+0000) 2022-09-08T01:09:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:00.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:00.669497+0000) 2022-09-08T01:09:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:01 smithi055 ceph-mon[97033]: pgmap v6768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:09:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:01 smithi059 ceph-mon[97450]: pgmap v6768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:09:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:01.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:01.740265+0000) 2022-09-08T01:09:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:01.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:01.669744+0000) 2022-09-08T01:09:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:02.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:02.740484+0000) 2022-09-08T01:09:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:02.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:02.669928+0000) 2022-09-08T01:09:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:03 smithi055 ceph-mon[97033]: pgmap v6769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:03 smithi059 ceph-mon[97450]: pgmap v6769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:03.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:03.740661+0000) 2022-09-08T01:09:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:03.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:03.670114+0000) 2022-09-08T01:09:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:04.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:04.740787+0000) 2022-09-08T01:09:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:04.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:04.670363+0000) 2022-09-08T01:09:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:05 smithi055 ceph-mon[97033]: pgmap v6770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:09:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:09:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:05 smithi059 ceph-mon[97450]: pgmap v6770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:09:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:09:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:05.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:05.740990+0000) 2022-09-08T01:09:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:05.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:05.670536+0000) 2022-09-08T01:09:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:06.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:06.741229+0000) 2022-09-08T01:09:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:06.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:06.670750+0000) 2022-09-08T01:09:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:07 smithi055 ceph-mon[97033]: pgmap v6771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:09:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:07 smithi059 ceph-mon[97450]: pgmap v6771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:09:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:07.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:07.741474+0000) 2022-09-08T01:09:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:07.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:07.670900+0000) 2022-09-08T01:09:07.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:09:07.524+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:09:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:08.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:08.741657+0000) 2022-09-08T01:09:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:08.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:08.671058+0000) 2022-09-08T01:09:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:09 smithi059 ceph-mon[97450]: osdmap e507: 12 total, 11 up, 11 in 2022-09-08T01:09:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:09 smithi059 ceph-mon[97450]: pgmap v6773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:09:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:09 smithi055 ceph-mon[97033]: osdmap e507: 12 total, 11 up, 11 in 2022-09-08T01:09:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:09 smithi055 ceph-mon[97033]: pgmap v6773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:09:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:09.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:09.741863+0000) 2022-09-08T01:09:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:09.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:09.671226+0000) 2022-09-08T01:09:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:10.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:10.742050+0000) 2022-09-08T01:09:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:10.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:10.671389+0000) 2022-09-08T01:09:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:11 smithi055 ceph-mon[97033]: pgmap v6774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:09:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:11 smithi055 ceph-mon[97033]: osdmap e508: 12 total, 11 up, 11 in 2022-09-08T01:09:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:11 smithi059 ceph-mon[97450]: pgmap v6774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:09:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:11 smithi059 ceph-mon[97450]: osdmap e508: 12 total, 11 up, 11 in 2022-09-08T01:09:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:11.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:11.671613+0000) 2022-09-08T01:09:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:11.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:11.742197+0000) 2022-09-08T01:09:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:12.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:12.742429+0000) 2022-09-08T01:09:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:12.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:12.671876+0000) 2022-09-08T01:09:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:13 smithi055 ceph-mon[97033]: pgmap v6776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:09:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:13 smithi059 ceph-mon[97450]: pgmap v6776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:09:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:13.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:13.742681+0000) 2022-09-08T01:09:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:13.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:13.672130+0000) 2022-09-08T01:09:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:14.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:14.742840+0000) 2022-09-08T01:09:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:14.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:14.672341+0000) 2022-09-08T01:09:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:15 smithi055 ceph-mon[97033]: pgmap v6777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:09:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:15 smithi059 ceph-mon[97450]: pgmap v6777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:09:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:15.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:15.672619+0000) 2022-09-08T01:09:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:15.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:15.743042+0000) 2022-09-08T01:09:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:16.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:16.672830+0000) 2022-09-08T01:09:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:16.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:16.743280+0000) 2022-09-08T01:09:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:17 smithi055 ceph-mon[97033]: pgmap v6778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T01:09:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:17 smithi059 ceph-mon[97450]: pgmap v6778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T01:09:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:17.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:17.743489+0000) 2022-09-08T01:09:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:17.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:17.673059+0000) 2022-09-08T01:09:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:18.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:18.743730+0000) 2022-09-08T01:09:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:18.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:18.673270+0000) 2022-09-08T01:09:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:19 smithi055 ceph-mon[97033]: pgmap v6779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:09:20.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:19 smithi059 ceph-mon[97450]: pgmap v6779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:09:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:19.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:19.743946+0000) 2022-09-08T01:09:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:19.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:19.673476+0000) 2022-09-08T01:09:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:20.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:20.744130+0000) 2022-09-08T01:09:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:20.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:20.673720+0000) 2022-09-08T01:09:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:21 smithi055 ceph-mon[97033]: pgmap v6780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:09:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:21 smithi059 ceph-mon[97450]: pgmap v6780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:09:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:21.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:21.744322+0000) 2022-09-08T01:09:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:21.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:21.673898+0000) 2022-09-08T01:09:22.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:09:22.539+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:09:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:22.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:22.744550+0000) 2022-09-08T01:09:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:22.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:22.674132+0000) 2022-09-08T01:09:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:23 smithi055 ceph-mon[97033]: pgmap v6781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:09:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:23 smithi059 ceph-mon[97450]: pgmap v6781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:09:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:23.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:23.744761+0000) 2022-09-08T01:09:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:23.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:23.674343+0000) 2022-09-08T01:09:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:24.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:24.674534+0000) 2022-09-08T01:09:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:24.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:24.744891+0000) 2022-09-08T01:09:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:25 smithi055 ceph-mon[97033]: pgmap v6782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:09:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:25 smithi059 ceph-mon[97450]: pgmap v6782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:09:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:25.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:25.745142+0000) 2022-09-08T01:09:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:25.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:25.674787+0000) 2022-09-08T01:09:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:26.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:26.745384+0000) 2022-09-08T01:09:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:26.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:26.674982+0000) 2022-09-08T01:09:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:27 smithi055 ceph-mon[97033]: pgmap v6783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:09:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:27 smithi059 ceph-mon[97450]: pgmap v6783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:09:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:27.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:27.745599+0000) 2022-09-08T01:09:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:27.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:27.675221+0000) 2022-09-08T01:09:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:28.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:28.745754+0000) 2022-09-08T01:09:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:28.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:28.675420+0000) 2022-09-08T01:09:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:29 smithi055 ceph-mon[97033]: pgmap v6784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:29 smithi059 ceph-mon[97450]: pgmap v6784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:29.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:29.745940+0000) 2022-09-08T01:09:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:29.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:29.675693+0000) 2022-09-08T01:09:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:30.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:30.746183+0000) 2022-09-08T01:09:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:30.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:30.675944+0000) 2022-09-08T01:09:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:31 smithi055 ceph-mon[97033]: pgmap v6785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:09:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:31 smithi059 ceph-mon[97450]: pgmap v6785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:09:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:31.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:31.746384+0000) 2022-09-08T01:09:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:31.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:31.676187+0000) 2022-09-08T01:09:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:32.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:32.746575+0000) 2022-09-08T01:09:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:32.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:32.676425+0000) 2022-09-08T01:09:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:33 smithi055 ceph-mon[97033]: pgmap v6786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:33 smithi059 ceph-mon[97450]: pgmap v6786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:33.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:33.746811+0000) 2022-09-08T01:09:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:33.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:33.676581+0000) 2022-09-08T01:09:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:34.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:34.746975+0000) 2022-09-08T01:09:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:34.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:34.676764+0000) 2022-09-08T01:09:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:35 smithi055 ceph-mon[97033]: pgmap v6787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:35 smithi059 ceph-mon[97450]: pgmap v6787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:35.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:35.747132+0000) 2022-09-08T01:09:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:35.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:35.677014+0000) 2022-09-08T01:09:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:36.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:36.747364+0000) 2022-09-08T01:09:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:36.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:36.677294+0000) 2022-09-08T01:09:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:37 smithi059 ceph-mon[97450]: pgmap v6788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:09:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:09:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:09:37.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:09:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:37 smithi055 ceph-mon[97033]: pgmap v6788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:09:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:09:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:09:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:09:37.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:09:37.552+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:09:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:37.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:37.747616+0000) 2022-09-08T01:09:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:37.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:37.677494+0000) 2022-09-08T01:09:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:38.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:38.747858+0000) 2022-09-08T01:09:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:38.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:38.677716+0000) 2022-09-08T01:09:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:39 smithi055 ceph-mon[97033]: pgmap v6789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:39 smithi059 ceph-mon[97450]: pgmap v6789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:39.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:39.748017+0000) 2022-09-08T01:09:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:39.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:39.677894+0000) 2022-09-08T01:09:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:40.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:40.748232+0000) 2022-09-08T01:09:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:40.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:40.678096+0000) 2022-09-08T01:09:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:41 smithi055 ceph-mon[97033]: pgmap v6790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:09:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:41 smithi059 ceph-mon[97450]: pgmap v6790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:09:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:41.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:41.678319+0000) 2022-09-08T01:09:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:41.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:41.748408+0000) 2022-09-08T01:09:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:42.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:42.748631+0000) 2022-09-08T01:09:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:42.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:42.678551+0000) 2022-09-08T01:09:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:43 smithi055 ceph-mon[97033]: pgmap v6791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:43 smithi059 ceph-mon[97450]: pgmap v6791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:43.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:43.748913+0000) 2022-09-08T01:09:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:43.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:43.678790+0000) 2022-09-08T01:09:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:44.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:44.749123+0000) 2022-09-08T01:09:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:44.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:44.679006+0000) 2022-09-08T01:09:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:45 smithi055 ceph-mon[97033]: pgmap v6792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:45 smithi059 ceph-mon[97450]: pgmap v6792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:45.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:45.749289+0000) 2022-09-08T01:09:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:45.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:45.679299+0000) 2022-09-08T01:09:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:46.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:46.749502+0000) 2022-09-08T01:09:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:46.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:46.679564+0000) 2022-09-08T01:09:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:47 smithi055 ceph-mon[97033]: pgmap v6793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:09:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:47 smithi059 ceph-mon[97450]: pgmap v6793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:09:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:47.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:47.749749+0000) 2022-09-08T01:09:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:47.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:47.679811+0000) 2022-09-08T01:09:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:48.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:48.749961+0000) 2022-09-08T01:09:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:48.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:48.680057+0000) 2022-09-08T01:09:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:49 smithi055 ceph-mon[97033]: pgmap v6794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:49 smithi059 ceph-mon[97450]: pgmap v6794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:49.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:49.750095+0000) 2022-09-08T01:09:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:49.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:49.680220+0000) 2022-09-08T01:09:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:50.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:50.750312+0000) 2022-09-08T01:09:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:50.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:50.680416+0000) 2022-09-08T01:09:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:51 smithi055 ceph-mon[97033]: pgmap v6795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:09:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:51 smithi059 ceph-mon[97450]: pgmap v6795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:09:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:51.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:51.750505+0000) 2022-09-08T01:09:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:51.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:51.680652+0000) 2022-09-08T01:09:52.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:09:52.567+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:09:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:52.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:52.750739+0000) 2022-09-08T01:09:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:52.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:52.680872+0000) 2022-09-08T01:09:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:53 smithi055 ceph-mon[97033]: pgmap v6796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:53 smithi059 ceph-mon[97450]: pgmap v6796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:53.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:53.681119+0000) 2022-09-08T01:09:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:53.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:53.750969+0000) 2022-09-08T01:09:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:54.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:54.751178+0000) 2022-09-08T01:09:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:54.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:54.681294+0000) 2022-09-08T01:09:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:55 smithi055 ceph-mon[97033]: pgmap v6797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:55 smithi059 ceph-mon[97450]: pgmap v6797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:09:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:55.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:55.681539+0000) 2022-09-08T01:09:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:55.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:55.751412+0000) 2022-09-08T01:09:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:56.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:56.751592+0000) 2022-09-08T01:09:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:56.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:56.681818+0000) 2022-09-08T01:09:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:57 smithi055 ceph-mon[97033]: pgmap v6798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:09:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:57 smithi059 ceph-mon[97450]: pgmap v6798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:09:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:57.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:57.682096+0000) 2022-09-08T01:09:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:57.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:57.751784+0000) 2022-09-08T01:09:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:58.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:58.682300+0000) 2022-09-08T01:09:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:58.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:58.751942+0000) 2022-09-08T01:09:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:09:59 smithi055 ceph-mon[97033]: pgmap v6799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:09:59 smithi059 ceph-mon[97450]: pgmap v6799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:09:59.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:59.752167+0000) 2022-09-08T01:10:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:09:59.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:09:59.682537+0000) 2022-09-08T01:10:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T01:10:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T01:10:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T01:10:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T01:10:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T01:10:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T01:10:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:00.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:00.752353+0000) 2022-09-08T01:10:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:00.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:00.682708+0000) 2022-09-08T01:10:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:01 smithi055 ceph-mon[97033]: pgmap v6800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:10:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:01 smithi059 ceph-mon[97450]: pgmap v6800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:10:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:01.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:01.752543+0000) 2022-09-08T01:10:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:01.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:01.682887+0000) 2022-09-08T01:10:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:02.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:02.752740+0000) 2022-09-08T01:10:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:02.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:02.683157+0000) 2022-09-08T01:10:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:03 smithi055 ceph-mon[97033]: pgmap v6801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:03 smithi059 ceph-mon[97450]: pgmap v6801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:03.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:03.752949+0000) 2022-09-08T01:10:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:03.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:03.683423+0000) 2022-09-08T01:10:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:04.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:04.753127+0000) 2022-09-08T01:10:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:04.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:04.683696+0000) 2022-09-08T01:10:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:05 smithi055 ceph-mon[97033]: pgmap v6802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:10:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:10:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:05 smithi059 ceph-mon[97450]: pgmap v6802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:10:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:10:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:05.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:05.753303+0000) 2022-09-08T01:10:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:05.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:05.683881+0000) 2022-09-08T01:10:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:06.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:06.753524+0000) 2022-09-08T01:10:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:06.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:06.684147+0000) 2022-09-08T01:10:07.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:10:07.583+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:10:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:07 smithi055 ceph-mon[97033]: pgmap v6803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:10:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:10:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:07 smithi059 ceph-mon[97450]: pgmap v6803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:10:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:10:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:07.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:07.753733+0000) 2022-09-08T01:10:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:07.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:07.684292+0000) 2022-09-08T01:10:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:08 smithi055 ceph-mon[97033]: pgmap v6804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 49 KiB/s wr, 22 op/s 2022-09-08T01:10:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:08 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T01:10:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:08 smithi059 ceph-mon[97450]: pgmap v6804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 49 KiB/s wr, 22 op/s 2022-09-08T01:10:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:08 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T01:10:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:08.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:08.753927+0000) 2022-09-08T01:10:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:08.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:08.684513+0000) 2022-09-08T01:10:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:09 smithi059 ceph-mon[97450]: osdmap e509: 12 total, 11 up, 11 in 2022-09-08T01:10:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:09 smithi055 ceph-mon[97033]: osdmap e509: 12 total, 11 up, 11 in 2022-09-08T01:10:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:09.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:09.754168+0000) 2022-09-08T01:10:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:09.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:09.684746+0000) 2022-09-08T01:10:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:10 smithi059 ceph-mon[97450]: pgmap v6806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 61 KiB/s wr, 23 op/s 2022-09-08T01:10:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:10 smithi055 ceph-mon[97033]: pgmap v6806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 61 KiB/s wr, 23 op/s 2022-09-08T01:10:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:10.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:10.754407+0000) 2022-09-08T01:10:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:10.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:10.684967+0000) 2022-09-08T01:10:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:11.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:11.754683+0000) 2022-09-08T01:10:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:11.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:11.685176+0000) 2022-09-08T01:10:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:12 smithi059 ceph-mon[97450]: pgmap v6807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T01:10:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:12 smithi055 ceph-mon[97033]: pgmap v6807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T01:10:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:12.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:12.754841+0000) 2022-09-08T01:10:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:12.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:12.685370+0000) 2022-09-08T01:10:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:13.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:13.755017+0000) 2022-09-08T01:10:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:13.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:13.685562+0000) 2022-09-08T01:10:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:14 smithi059 ceph-mon[97450]: pgmap v6808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T01:10:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:14 smithi055 ceph-mon[97033]: pgmap v6808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T01:10:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:14.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:14.755166+0000) 2022-09-08T01:10:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:14.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:14.685763+0000) 2022-09-08T01:10:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:15.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:15.755373+0000) 2022-09-08T01:10:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:15.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:15.685935+0000) 2022-09-08T01:10:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:16 smithi059 ceph-mon[97450]: pgmap v6809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 233 B/s wr, 7 op/s 2022-09-08T01:10:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:16 smithi059 ceph-mon[97450]: osdmap e510: 12 total, 11 up, 11 in 2022-09-08T01:10:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:16 smithi055 ceph-mon[97033]: pgmap v6809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 233 B/s wr, 7 op/s 2022-09-08T01:10:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:16 smithi055 ceph-mon[97033]: osdmap e510: 12 total, 11 up, 11 in 2022-09-08T01:10:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:16.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:16.755493+0000) 2022-09-08T01:10:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:16.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:16.686165+0000) 2022-09-08T01:10:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:17.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:17.755732+0000) 2022-09-08T01:10:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:17.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:17.686403+0000) 2022-09-08T01:10:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:18 smithi059 ceph-mon[97450]: pgmap v6811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T01:10:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:18 smithi055 ceph-mon[97033]: pgmap v6811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T01:10:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:18.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:18.755924+0000) 2022-09-08T01:10:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:18.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:18.686557+0000) 2022-09-08T01:10:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:19.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:19.756151+0000) 2022-09-08T01:10:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:19.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:19.686726+0000) 2022-09-08T01:10:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:20 smithi059 ceph-mon[97450]: pgmap v6812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:10:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:20 smithi055 ceph-mon[97033]: pgmap v6812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:10:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:20.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:20.756384+0000) 2022-09-08T01:10:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:20.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:20.686958+0000) 2022-09-08T01:10:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:21.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:21.756601+0000) 2022-09-08T01:10:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:21.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:21.687221+0000) 2022-09-08T01:10:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:22 smithi059 ceph-mon[97450]: pgmap v6813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:10:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:22.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:22.687491+0000) 2022-09-08T01:10:22.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:10:22.590+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:10:22.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:22.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:22.756857+0000) 2022-09-08T01:10:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:22 smithi055 ceph-mon[97033]: pgmap v6813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:10:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:23.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:23.757063+0000) 2022-09-08T01:10:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:23.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:23.687736+0000) 2022-09-08T01:10:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:24 smithi059 ceph-mon[97450]: pgmap v6814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:10:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:24 smithi055 ceph-mon[97033]: pgmap v6814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:10:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:24.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:24.757232+0000) 2022-09-08T01:10:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:24.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:24.687926+0000) 2022-09-08T01:10:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:25.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:25.757456+0000) 2022-09-08T01:10:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:25.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:25.688119+0000) 2022-09-08T01:10:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:26 smithi059 ceph-mon[97450]: pgmap v6815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:10:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:26 smithi055 ceph-mon[97033]: pgmap v6815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:10:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:26.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:26.757663+0000) 2022-09-08T01:10:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:26.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:26.688352+0000) 2022-09-08T01:10:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:27.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:27.757892+0000) 2022-09-08T01:10:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:27.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:27.688528+0000) 2022-09-08T01:10:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:28 smithi059 ceph-mon[97450]: pgmap v6816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:10:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:28 smithi055 ceph-mon[97033]: pgmap v6816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:10:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:28.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:28.758069+0000) 2022-09-08T01:10:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:28.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:28.688752+0000) 2022-09-08T01:10:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:29.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:29.758295+0000) 2022-09-08T01:10:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:29.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:29.688956+0000) 2022-09-08T01:10:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:30 smithi059 ceph-mon[97450]: pgmap v6817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:30 smithi055 ceph-mon[97033]: pgmap v6817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:30.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:30.758546+0000) 2022-09-08T01:10:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:30.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:30.689120+0000) 2022-09-08T01:10:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:31.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:31.758794+0000) 2022-09-08T01:10:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:31.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:31.689340+0000) 2022-09-08T01:10:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:32 smithi059 ceph-mon[97450]: pgmap v6818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:10:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:32 smithi055 ceph-mon[97033]: pgmap v6818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:10:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:32.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:32.759008+0000) 2022-09-08T01:10:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:32.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:32.689551+0000) 2022-09-08T01:10:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:33.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:33.759250+0000) 2022-09-08T01:10:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:33.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:33.689705+0000) 2022-09-08T01:10:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:34 smithi055 ceph-mon[97033]: pgmap v6819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:34 smithi059 ceph-mon[97450]: pgmap v6819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:34.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:34.759472+0000) 2022-09-08T01:10:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:34.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:34.689910+0000) 2022-09-08T01:10:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:35.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:35.759751+0000) 2022-09-08T01:10:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:35.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:35.690095+0000) 2022-09-08T01:10:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:36 smithi055 ceph-mon[97033]: pgmap v6820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:10:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:36 smithi059 ceph-mon[97450]: pgmap v6820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:10:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:36.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:36.760029+0000) 2022-09-08T01:10:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:36.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:36.690343+0000) 2022-09-08T01:10:37.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:10:37.606+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:10:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:37.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:37.760263+0000) 2022-09-08T01:10:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:37.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:37.690560+0000) 2022-09-08T01:10:38.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:38 smithi055 ceph-mon[97033]: pgmap v6821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:38 smithi059 ceph-mon[97450]: pgmap v6821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:38.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:38.690799+0000) 2022-09-08T01:10:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:38.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:38.760511+0000) 2022-09-08T01:10:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:39.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:39.760659+0000) 2022-09-08T01:10:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:39.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:39.690940+0000) 2022-09-08T01:10:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:40 smithi055 ceph-mon[97033]: pgmap v6822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:40 smithi059 ceph-mon[97450]: pgmap v6822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:40.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:40.760790+0000) 2022-09-08T01:10:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:40.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:40.691127+0000) 2022-09-08T01:10:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:41.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:41.760911+0000) 2022-09-08T01:10:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:41.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:41.691280+0000) 2022-09-08T01:10:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:42 smithi055 ceph-mon[97033]: pgmap v6823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:10:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:42 smithi059 ceph-mon[97450]: pgmap v6823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:10:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:42.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:42.761134+0000) 2022-09-08T01:10:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:42.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:42.691499+0000) 2022-09-08T01:10:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:43.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:43.761325+0000) 2022-09-08T01:10:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:43.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:43.691817+0000) 2022-09-08T01:10:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:44 smithi055 ceph-mon[97033]: pgmap v6824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:44 smithi059 ceph-mon[97450]: pgmap v6824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:44.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:44.761578+0000) 2022-09-08T01:10:45.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:44.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:44.692023+0000) 2022-09-08T01:10:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:45.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:45.761789+0000) 2022-09-08T01:10:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:45.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:45.692266+0000) 2022-09-08T01:10:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:46 smithi055 ceph-mon[97033]: pgmap v6825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:10:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:46 smithi059 ceph-mon[97450]: pgmap v6825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:10:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:46.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:46.762018+0000) 2022-09-08T01:10:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:46.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:46.692535+0000) 2022-09-08T01:10:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:47.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:47.762196+0000) 2022-09-08T01:10:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:47.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:47.692780+0000) 2022-09-08T01:10:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:48 smithi055 ceph-mon[97033]: pgmap v6826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:48 smithi059 ceph-mon[97450]: pgmap v6826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:48.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:48.762389+0000) 2022-09-08T01:10:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:48.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:48.693006+0000) 2022-09-08T01:10:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:49.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:49.762601+0000) 2022-09-08T01:10:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:49.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:49.693198+0000) 2022-09-08T01:10:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:50 smithi055 ceph-mon[97033]: pgmap v6827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:50 smithi059 ceph-mon[97450]: pgmap v6827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:50.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:50.762803+0000) 2022-09-08T01:10:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:50.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:50.693426+0000) 2022-09-08T01:10:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:51.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:51.763008+0000) 2022-09-08T01:10:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:51.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:51.693733+0000) 2022-09-08T01:10:52.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:10:52.611+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:10:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:52.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:52.763202+0000) 2022-09-08T01:10:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:52.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:52.693983+0000) 2022-09-08T01:10:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:52 smithi055 ceph-mon[97033]: pgmap v6828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:10:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:52 smithi059 ceph-mon[97450]: pgmap v6828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:10:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:53.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:53.763433+0000) 2022-09-08T01:10:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:53.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:53.694211+0000) 2022-09-08T01:10:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:54 smithi055 ceph-mon[97033]: pgmap v6829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:54 smithi059 ceph-mon[97450]: pgmap v6829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:54.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:54.763692+0000) 2022-09-08T01:10:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:54.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:54.694454+0000) 2022-09-08T01:10:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:55.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:55.763921+0000) 2022-09-08T01:10:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:55.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:55.694752+0000) 2022-09-08T01:10:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:56 smithi055 ceph-mon[97033]: pgmap v6830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:10:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:56 smithi059 ceph-mon[97450]: pgmap v6830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:10:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:56.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:56.764140+0000) 2022-09-08T01:10:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:56.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:56.694962+0000) 2022-09-08T01:10:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:57.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:57.764368+0000) 2022-09-08T01:10:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:57.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:57.695190+0000) 2022-09-08T01:10:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:10:58 smithi055 ceph-mon[97033]: pgmap v6831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:10:58 smithi059 ceph-mon[97450]: pgmap v6831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:10:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:58.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:58.764572+0000) 2022-09-08T01:10:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:58.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:58.695405+0000) 2022-09-08T01:11:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:10:59.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:59.695720+0000) 2022-09-08T01:11:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:10:59.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:10:59.764792+0000) 2022-09-08T01:11:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:00 smithi055 ceph-mon[97033]: pgmap v6832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:00 smithi059 ceph-mon[97450]: pgmap v6832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:00.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:00.695915+0000) 2022-09-08T01:11:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:00.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:00.765019+0000) 2022-09-08T01:11:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:01.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:01.765234+0000) 2022-09-08T01:11:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:01.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:01.696157+0000) 2022-09-08T01:11:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:02 smithi055 ceph-mon[97033]: pgmap v6833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:11:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:02 smithi059 ceph-mon[97450]: pgmap v6833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:11:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:02.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:02.765490+0000) 2022-09-08T01:11:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:02.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:02.696396+0000) 2022-09-08T01:11:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:03.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:03.765693+0000) 2022-09-08T01:11:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:03.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:03.696593+0000) 2022-09-08T01:11:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:04 smithi055 ceph-mon[97033]: pgmap v6834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:04 smithi059 ceph-mon[97450]: pgmap v6834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:04.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:04.765843+0000) 2022-09-08T01:11:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:04.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:04.696823+0000) 2022-09-08T01:11:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:11:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:11:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:11:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:11:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:05.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:05.766056+0000) 2022-09-08T01:11:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:05.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:05.697072+0000) 2022-09-08T01:11:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:06 smithi055 ceph-mon[97033]: pgmap v6835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:11:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:06 smithi059 ceph-mon[97450]: pgmap v6835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:11:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:06.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:06.766269+0000) 2022-09-08T01:11:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:06.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:06.697274+0000) 2022-09-08T01:11:07.698 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:11:07.622+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:11:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:11:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:11:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:11:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:11:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:11:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:11:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:07.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:07.697518+0000) 2022-09-08T01:11:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:07.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:07.766522+0000) 2022-09-08T01:11:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:08 smithi055 ceph-mon[97033]: pgmap v6836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:11:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:08 smithi059 ceph-mon[97450]: pgmap v6836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:11:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:08.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:08.697759+0000) 2022-09-08T01:11:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:08.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:08.766719+0000) 2022-09-08T01:11:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:09 smithi055 ceph-mon[97033]: osdmap e511: 12 total, 11 up, 11 in 2022-09-08T01:11:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:09 smithi059 ceph-mon[97450]: osdmap e511: 12 total, 11 up, 11 in 2022-09-08T01:11:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:09.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:09.697974+0000) 2022-09-08T01:11:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:09.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:09.766951+0000) 2022-09-08T01:11:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:10 smithi055 ceph-mon[97033]: pgmap v6838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:11:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:10 smithi059 ceph-mon[97450]: pgmap v6838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:11:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:10.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:10.767187+0000) 2022-09-08T01:11:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:10.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:10.698191+0000) 2022-09-08T01:11:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:11.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:11.767418+0000) 2022-09-08T01:11:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:11.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:11.698369+0000) 2022-09-08T01:11:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:12 smithi055 ceph-mon[97033]: pgmap v6839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:11:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:12 smithi059 ceph-mon[97450]: pgmap v6839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:11:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:12.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:12.767679+0000) 2022-09-08T01:11:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:12.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:12.698578+0000) 2022-09-08T01:11:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:13.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:13.767827+0000) 2022-09-08T01:11:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:13.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:13.698934+0000) 2022-09-08T01:11:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:14 smithi055 ceph-mon[97033]: pgmap v6840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:11:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:14 smithi059 ceph-mon[97450]: pgmap v6840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:11:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:14.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:14.768053+0000) 2022-09-08T01:11:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:14.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:14.699132+0000) 2022-09-08T01:11:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:15.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:15.768281+0000) 2022-09-08T01:11:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:15.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:15.699383+0000) 2022-09-08T01:11:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:16 smithi055 ceph-mon[97033]: pgmap v6841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:11:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:16 smithi055 ceph-mon[97033]: osdmap e512: 12 total, 11 up, 11 in 2022-09-08T01:11:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:16 smithi059 ceph-mon[97450]: pgmap v6841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:11:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:16 smithi059 ceph-mon[97450]: osdmap e512: 12 total, 11 up, 11 in 2022-09-08T01:11:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:16.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:16.768464+0000) 2022-09-08T01:11:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:16.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:16.699544+0000) 2022-09-08T01:11:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:17.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:17.768628+0000) 2022-09-08T01:11:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:17.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:17.699791+0000) 2022-09-08T01:11:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:18 smithi055 ceph-mon[97033]: pgmap v6843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 233 B/s wr, 7 op/s 2022-09-08T01:11:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:18 smithi059 ceph-mon[97450]: pgmap v6843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 233 B/s wr, 7 op/s 2022-09-08T01:11:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:18.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:18.768812+0000) 2022-09-08T01:11:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:18.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:18.700032+0000) 2022-09-08T01:11:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:19.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:19.769015+0000) 2022-09-08T01:11:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:19.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:19.700248+0000) 2022-09-08T01:11:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:20 smithi055 ceph-mon[97033]: pgmap v6844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:11:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:20 smithi059 ceph-mon[97450]: pgmap v6844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:11:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:20.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:20.769251+0000) 2022-09-08T01:11:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:20.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:20.700449+0000) 2022-09-08T01:11:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:21.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:21.700700+0000) 2022-09-08T01:11:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:21.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:21.769478+0000) 2022-09-08T01:11:22.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:11:22.638+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:11:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:22.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:22.700912+0000) 2022-09-08T01:11:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:22.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:22.769737+0000) 2022-09-08T01:11:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:22 smithi055 ceph-mon[97033]: pgmap v6845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:11:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:22 smithi059 ceph-mon[97450]: pgmap v6845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:11:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:23.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:23.769968+0000) 2022-09-08T01:11:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:23.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:23.701160+0000) 2022-09-08T01:11:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:24 smithi055 ceph-mon[97033]: pgmap v6846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:11:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:24 smithi059 ceph-mon[97450]: pgmap v6846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:11:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:24.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:24.770234+0000) 2022-09-08T01:11:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:24.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:24.701476+0000) 2022-09-08T01:11:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:25.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:25.770412+0000) 2022-09-08T01:11:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:25.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:25.701703+0000) 2022-09-08T01:11:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:26 smithi055 ceph-mon[97033]: pgmap v6847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:11:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:26 smithi059 ceph-mon[97450]: pgmap v6847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:11:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:26.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:26.701925+0000) 2022-09-08T01:11:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:26.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:26.770607+0000) 2022-09-08T01:11:28.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:27.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:27.770776+0000) 2022-09-08T01:11:28.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:27.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:27.702159+0000) 2022-09-08T01:11:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:28 smithi055 ceph-mon[97033]: pgmap v6848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:11:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:28 smithi059 ceph-mon[97450]: pgmap v6848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:11:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:28.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:28.770899+0000) 2022-09-08T01:11:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:28.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:28.702360+0000) 2022-09-08T01:11:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:29.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:29.771051+0000) 2022-09-08T01:11:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:29.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:29.702591+0000) 2022-09-08T01:11:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:30 smithi055 ceph-mon[97033]: pgmap v6849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:30 smithi059 ceph-mon[97450]: pgmap v6849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:30.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:30.771232+0000) 2022-09-08T01:11:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:30.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:30.702842+0000) 2022-09-08T01:11:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:31.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:31.771455+0000) 2022-09-08T01:11:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:31.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:31.703096+0000) 2022-09-08T01:11:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:32.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:32.771707+0000) 2022-09-08T01:11:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:32.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:32.703337+0000) 2022-09-08T01:11:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:32 smithi055 ceph-mon[97033]: pgmap v6850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:11:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:32 smithi059 ceph-mon[97450]: pgmap v6850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:11:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:33.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:33.771893+0000) 2022-09-08T01:11:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:33.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:33.703580+0000) 2022-09-08T01:11:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:34 smithi055 ceph-mon[97033]: pgmap v6851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:34 smithi059 ceph-mon[97450]: pgmap v6851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:34.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:34.772069+0000) 2022-09-08T01:11:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:34.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:34.703820+0000) 2022-09-08T01:11:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:35.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:35.772278+0000) 2022-09-08T01:11:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:35.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:35.704007+0000) 2022-09-08T01:11:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:36 smithi055 ceph-mon[97033]: pgmap v6852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:11:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:36 smithi059 ceph-mon[97450]: pgmap v6852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:11:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:36.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:36.772515+0000) 2022-09-08T01:11:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:36.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:36.704158+0000) 2022-09-08T01:11:37.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:11:37.653+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:11:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:37.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:37.772773+0000) 2022-09-08T01:11:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:37.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:37.704385+0000) 2022-09-08T01:11:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:38 smithi055 ceph-mon[97033]: pgmap v6853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:38 smithi055 ceph-mon[97033]: pgmap v6854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:11:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:38 smithi055 ceph-mon[97033]: pgmap v6855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:11:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:11:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:38 smithi059 ceph-mon[97450]: pgmap v6853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:38 smithi059 ceph-mon[97450]: pgmap v6854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:11:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:38 smithi059 ceph-mon[97450]: pgmap v6855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:11:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:11:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:38.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:38.772919+0000) 2022-09-08T01:11:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:38.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:38.704621+0000) 2022-09-08T01:11:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:39.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:39.773120+0000) 2022-09-08T01:11:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:39.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:39.704860+0000) 2022-09-08T01:11:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:40 smithi055 ceph-mon[97033]: pgmap v6856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:11:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:40 smithi059 ceph-mon[97450]: pgmap v6856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:11:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:40.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:40.773430+0000) 2022-09-08T01:11:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:40.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:40.705100+0000) 2022-09-08T01:11:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:41.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:41.773751+0000) 2022-09-08T01:11:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:41.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:41.705306+0000) 2022-09-08T01:11:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:42 smithi055 ceph-mon[97033]: pgmap v6857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:11:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:42 smithi059 ceph-mon[97450]: pgmap v6857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:11:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:42.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:42.773965+0000) 2022-09-08T01:11:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:42.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:42.705545+0000) 2022-09-08T01:11:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:43.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:43.774232+0000) 2022-09-08T01:11:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:43.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:43.705775+0000) 2022-09-08T01:11:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:44 smithi055 ceph-mon[97033]: pgmap v6858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:11:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:44 smithi059 ceph-mon[97450]: pgmap v6858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:11:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:44.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:44.774473+0000) 2022-09-08T01:11:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:44.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:44.705907+0000) 2022-09-08T01:11:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:45.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:45.774664+0000) 2022-09-08T01:11:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:45.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:45.706118+0000) 2022-09-08T01:11:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:46 smithi055 ceph-mon[97033]: pgmap v6859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:11:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:46 smithi059 ceph-mon[97450]: pgmap v6859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:11:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:46.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:46.774881+0000) 2022-09-08T01:11:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:46.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:46.706351+0000) 2022-09-08T01:11:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:47.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:47.775090+0000) 2022-09-08T01:11:48.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:47.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:47.706598+0000) 2022-09-08T01:11:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:48 smithi055 ceph-mon[97033]: pgmap v6860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:11:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:48 smithi059 ceph-mon[97450]: pgmap v6860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:11:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:48.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:48.775267+0000) 2022-09-08T01:11:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:48.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:48.706812+0000) 2022-09-08T01:11:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:49.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:49.707036+0000) 2022-09-08T01:11:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:49.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:49.775557+0000) 2022-09-08T01:11:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:50 smithi055 ceph-mon[97033]: pgmap v6861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:50 smithi059 ceph-mon[97450]: pgmap v6861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:50.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:50.707198+0000) 2022-09-08T01:11:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:50.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:50.775714+0000) 2022-09-08T01:11:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:51.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:51.775858+0000) 2022-09-08T01:11:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:51.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:51.707432+0000) 2022-09-08T01:11:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:52 smithi055 ceph-mon[97033]: pgmap v6862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:11:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:52 smithi059 ceph-mon[97450]: pgmap v6862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:11:53.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:11:52.655+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:11:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:52.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:52.776097+0000) 2022-09-08T01:11:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:52.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:52.707716+0000) 2022-09-08T01:11:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:53.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:53.707956+0000) 2022-09-08T01:11:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:53.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:53.776342+0000) 2022-09-08T01:11:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:54 smithi055 ceph-mon[97033]: pgmap v6863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:54 smithi059 ceph-mon[97450]: pgmap v6863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:54.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:54.776626+0000) 2022-09-08T01:11:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:54.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:54.708170+0000) 2022-09-08T01:11:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:55.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:55.776924+0000) 2022-09-08T01:11:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:55.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:55.708409+0000) 2022-09-08T01:11:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:56 smithi055 ceph-mon[97033]: pgmap v6864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:11:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:56 smithi059 ceph-mon[97450]: pgmap v6864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:11:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:56.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:56.777178+0000) 2022-09-08T01:11:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:56.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:56.708562+0000) 2022-09-08T01:11:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:57.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:57.777412+0000) 2022-09-08T01:11:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:57.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:57.708797+0000) 2022-09-08T01:11:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:11:58 smithi055 ceph-mon[97033]: pgmap v6865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:11:58 smithi059 ceph-mon[97450]: pgmap v6865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:11:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:58.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:58.777567+0000) 2022-09-08T01:11:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:58.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:58.709037+0000) 2022-09-08T01:12:00.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:11:59.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:59.777796+0000) 2022-09-08T01:12:00.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:11:59.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:11:59.709245+0000) 2022-09-08T01:12:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:00 smithi055 ceph-mon[97033]: pgmap v6866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:00 smithi059 ceph-mon[97450]: pgmap v6866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:00.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:00.778026+0000) 2022-09-08T01:12:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:00.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:00.709463+0000) 2022-09-08T01:12:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:01.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:01.778265+0000) 2022-09-08T01:12:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:01.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:01.709741+0000) 2022-09-08T01:12:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:02 smithi055 ceph-mon[97033]: pgmap v6867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:12:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:02 smithi059 ceph-mon[97450]: pgmap v6867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:12:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:02.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:02.778501+0000) 2022-09-08T01:12:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:02.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:02.709945+0000) 2022-09-08T01:12:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:03.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:03.778699+0000) 2022-09-08T01:12:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:03.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:03.710196+0000) 2022-09-08T01:12:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:04 smithi055 ceph-mon[97033]: pgmap v6868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:04 smithi059 ceph-mon[97450]: pgmap v6868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:04.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:04.778823+0000) 2022-09-08T01:12:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:04.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:04.710419+0000) 2022-09-08T01:12:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:12:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:12:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:12:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:12:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:05.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:05.778978+0000) 2022-09-08T01:12:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:05.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:05.710716+0000) 2022-09-08T01:12:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:06 smithi055 ceph-mon[97033]: pgmap v6869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:12:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:06 smithi059 ceph-mon[97450]: pgmap v6869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:12:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:06.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:06.779188+0000) 2022-09-08T01:12:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:06.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:06.710968+0000) 2022-09-08T01:12:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:07.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:07.779419+0000) 2022-09-08T01:12:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:07.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:07.711189+0000) 2022-09-08T01:12:08.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:12:07.670+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:12:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:08 smithi055 ceph-mon[97033]: pgmap v6870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:12:09.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:08 smithi059 ceph-mon[97450]: pgmap v6870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:12:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:08.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:08.779650+0000) 2022-09-08T01:12:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:08.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:08.711431+0000) 2022-09-08T01:12:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:09 smithi055 ceph-mon[97033]: osdmap e513: 12 total, 11 up, 11 in 2022-09-08T01:12:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:09 smithi059 ceph-mon[97450]: osdmap e513: 12 total, 11 up, 11 in 2022-09-08T01:12:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:09.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:09.779779+0000) 2022-09-08T01:12:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:09.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:09.711679+0000) 2022-09-08T01:12:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:10 smithi059 ceph-mon[97450]: pgmap v6872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:12:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:10.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:10.780012+0000) 2022-09-08T01:12:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:10.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:10.711917+0000) 2022-09-08T01:12:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:10 smithi055 ceph-mon[97033]: pgmap v6872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:12:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:11.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:11.780210+0000) 2022-09-08T01:12:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:11.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:11.712118+0000) 2022-09-08T01:12:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:12 smithi055 ceph-mon[97033]: pgmap v6873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:12:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:12 smithi059 ceph-mon[97450]: pgmap v6873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:12:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:12.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:12.780378+0000) 2022-09-08T01:12:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:12.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:12.712355+0000) 2022-09-08T01:12:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:13.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:13.780595+0000) 2022-09-08T01:12:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:13.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:13.712607+0000) 2022-09-08T01:12:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:14 smithi055 ceph-mon[97033]: pgmap v6874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:12:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:14 smithi059 ceph-mon[97450]: pgmap v6874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:12:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:14.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:14.780776+0000) 2022-09-08T01:12:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:14.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:14.712801+0000) 2022-09-08T01:12:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:15.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:15.780916+0000) 2022-09-08T01:12:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:15.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:15.713005+0000) 2022-09-08T01:12:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:16 smithi055 ceph-mon[97033]: pgmap v6875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:12:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:16 smithi055 ceph-mon[97033]: osdmap e514: 12 total, 11 up, 11 in 2022-09-08T01:12:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:16 smithi059 ceph-mon[97450]: pgmap v6875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:12:17.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:16 smithi059 ceph-mon[97450]: osdmap e514: 12 total, 11 up, 11 in 2022-09-08T01:12:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:16.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:16.781127+0000) 2022-09-08T01:12:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:16.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:16.713210+0000) 2022-09-08T01:12:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:17.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:17.781320+0000) 2022-09-08T01:12:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:17.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:17.713399+0000) 2022-09-08T01:12:18.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:18 smithi055 ceph-mon[97033]: pgmap v6877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:12:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:18 smithi059 ceph-mon[97450]: pgmap v6877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:12:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:18.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:18.781516+0000) 2022-09-08T01:12:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:18.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:18.713562+0000) 2022-09-08T01:12:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:19.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:19.781746+0000) 2022-09-08T01:12:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:19.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:19.713707+0000) 2022-09-08T01:12:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:20 smithi055 ceph-mon[97033]: pgmap v6878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:12:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:20 smithi059 ceph-mon[97450]: pgmap v6878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:12:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:20.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:20.781955+0000) 2022-09-08T01:12:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:20.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:20.713934+0000) 2022-09-08T01:12:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:21.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:21.782135+0000) 2022-09-08T01:12:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:21.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:21.714159+0000) 2022-09-08T01:12:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:22 smithi055 ceph-mon[97033]: pgmap v6879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:12:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:22 smithi059 ceph-mon[97450]: pgmap v6879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:12:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:22.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:22.782324+0000) 2022-09-08T01:12:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:22.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:22.714398+0000) 2022-09-08T01:12:23.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:12:22.684+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:12:24.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:23.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:23.782572+0000) 2022-09-08T01:12:24.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:23.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:23.714667+0000) 2022-09-08T01:12:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:24 smithi055 ceph-mon[97033]: pgmap v6880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:12:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:24 smithi059 ceph-mon[97450]: pgmap v6880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:12:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:24.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:24.782855+0000) 2022-09-08T01:12:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:24.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:24.714875+0000) 2022-09-08T01:12:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:25.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:25.715093+0000) 2022-09-08T01:12:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:25.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:25.783146+0000) 2022-09-08T01:12:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:26 smithi055 ceph-mon[97033]: pgmap v6881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:12:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:26 smithi059 ceph-mon[97450]: pgmap v6881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:12:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:26.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:26.783402+0000) 2022-09-08T01:12:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:26.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:26.715318+0000) 2022-09-08T01:12:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:27.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:27.715476+0000) 2022-09-08T01:12:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:27.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:27.783657+0000) 2022-09-08T01:12:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:28 smithi055 ceph-mon[97033]: pgmap v6882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:12:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:28 smithi059 ceph-mon[97450]: pgmap v6882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:12:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:28.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:28.715664+0000) 2022-09-08T01:12:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:28.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:28.783918+0000) 2022-09-08T01:12:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:29.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:29.715874+0000) 2022-09-08T01:12:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:29.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:29.784178+0000) 2022-09-08T01:12:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:30 smithi055 ceph-mon[97033]: pgmap v6883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:30 smithi059 ceph-mon[97450]: pgmap v6883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:30.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:30.784421+0000) 2022-09-08T01:12:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:30.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:30.716099+0000) 2022-09-08T01:12:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:31.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:31.784780+0000) 2022-09-08T01:12:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:31.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:31.716273+0000) 2022-09-08T01:12:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:32 smithi055 ceph-mon[97033]: pgmap v6884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:12:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:32 smithi059 ceph-mon[97450]: pgmap v6884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:12:33.162 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:32.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:32.784920+0000) 2022-09-08T01:12:33.163 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:32.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:32.716439+0000) 2022-09-08T01:12:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:33.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:33.785159+0000) 2022-09-08T01:12:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:33.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:33.716650+0000) 2022-09-08T01:12:34.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:34 smithi055 ceph-mon[97033]: pgmap v6885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:34 smithi059 ceph-mon[97450]: pgmap v6885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:34.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:34.785401+0000) 2022-09-08T01:12:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:34.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:34.716767+0000) 2022-09-08T01:12:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:35.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:35.785553+0000) 2022-09-08T01:12:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:35.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:35.716935+0000) 2022-09-08T01:12:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:36 smithi055 ceph-mon[97033]: pgmap v6886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:12:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:36 smithi059 ceph-mon[97450]: pgmap v6886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:12:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:36.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:36.785764+0000) 2022-09-08T01:12:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:36.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:36.717073+0000) 2022-09-08T01:12:38.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:12:37.688+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:12:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:37.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:37.785936+0000) 2022-09-08T01:12:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:37.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:37.717263+0000) 2022-09-08T01:12:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:38 smithi055 ceph-mon[97033]: pgmap v6887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:12:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:12:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:12:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:38 smithi059 ceph-mon[97450]: pgmap v6887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:12:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:12:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:12:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:38.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:38.786125+0000) 2022-09-08T01:12:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:38.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:38.717493+0000) 2022-09-08T01:12:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:39.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:39.786306+0000) 2022-09-08T01:12:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:39.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:39.717751+0000) 2022-09-08T01:12:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:40 smithi055 ceph-mon[97033]: pgmap v6888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:40 smithi059 ceph-mon[97450]: pgmap v6888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:40.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:40.786583+0000) 2022-09-08T01:12:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:40.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:40.717975+0000) 2022-09-08T01:12:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:41.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:41.786813+0000) 2022-09-08T01:12:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:41.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:41.718136+0000) 2022-09-08T01:12:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:42 smithi055 ceph-mon[97033]: pgmap v6889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:12:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:42 smithi059 ceph-mon[97450]: pgmap v6889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:12:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:42.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:42.787019+0000) 2022-09-08T01:12:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:42.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:42.718374+0000) 2022-09-08T01:12:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:43.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:43.787222+0000) 2022-09-08T01:12:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:43.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:43.718628+0000) 2022-09-08T01:12:44.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:44 smithi055 ceph-mon[97033]: pgmap v6890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:44 smithi059 ceph-mon[97450]: pgmap v6890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:44.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:44.787442+0000) 2022-09-08T01:12:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:44.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:44.718844+0000) 2022-09-08T01:12:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:45.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:45.787709+0000) 2022-09-08T01:12:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:45.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:45.719119+0000) 2022-09-08T01:12:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:46 smithi055 ceph-mon[97033]: pgmap v6891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:12:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:46 smithi059 ceph-mon[97450]: pgmap v6891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:12:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:46.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:46.787926+0000) 2022-09-08T01:12:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:46.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:46.719377+0000) 2022-09-08T01:12:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:47.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:47.719581+0000) 2022-09-08T01:12:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:47.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:47.788170+0000) 2022-09-08T01:12:48.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:48 smithi055 ceph-mon[97033]: pgmap v6892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:48 smithi059 ceph-mon[97450]: pgmap v6892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:48.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:48.788385+0000) 2022-09-08T01:12:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:48.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:48.719794+0000) 2022-09-08T01:12:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:49.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:49.788569+0000) 2022-09-08T01:12:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:49.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:49.719997+0000) 2022-09-08T01:12:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:50 smithi055 ceph-mon[97033]: pgmap v6893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:50 smithi059 ceph-mon[97450]: pgmap v6893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:50.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:50.788767+0000) 2022-09-08T01:12:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:50.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:50.720234+0000) 2022-09-08T01:12:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:51.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:51.788997+0000) 2022-09-08T01:12:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:51.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:51.720483+0000) 2022-09-08T01:12:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:52 smithi055 ceph-mon[97033]: pgmap v6894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:12:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:52 smithi059 ceph-mon[97450]: pgmap v6894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:12:53.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:12:52.705+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:12:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:52.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:52.789216+0000) 2022-09-08T01:12:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:52.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:52.720762+0000) 2022-09-08T01:12:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:53.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:53.789409+0000) 2022-09-08T01:12:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:53.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:53.720962+0000) 2022-09-08T01:12:54.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:54 smithi055 ceph-mon[97033]: pgmap v6895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:54 smithi059 ceph-mon[97450]: pgmap v6895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:54.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:54.789619+0000) 2022-09-08T01:12:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:54.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:54.721128+0000) 2022-09-08T01:12:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:55.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:55.789861+0000) 2022-09-08T01:12:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:55.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:55.721288+0000) 2022-09-08T01:12:56.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:56 smithi055 ceph-mon[97033]: pgmap v6896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:12:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:56 smithi059 ceph-mon[97450]: pgmap v6896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:12:57.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:56.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:56.790026+0000) 2022-09-08T01:12:57.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:56.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:56.721492+0000) 2022-09-08T01:12:58.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:57.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:57.790212+0000) 2022-09-08T01:12:58.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:57.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:57.721738+0000) 2022-09-08T01:12:58.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:12:58 smithi055 ceph-mon[97033]: pgmap v6897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:12:58 smithi059 ceph-mon[97450]: pgmap v6897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:12:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:58.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:58.790448+0000) 2022-09-08T01:12:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:58.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:58.721915+0000) 2022-09-08T01:13:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:12:59.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:59.790713+0000) 2022-09-08T01:13:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:12:59.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:12:59.722030+0000) 2022-09-08T01:13:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:00 smithi055 ceph-mon[97033]: pgmap v6898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:00 smithi059 ceph-mon[97450]: pgmap v6898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:00.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:00.722183+0000) 2022-09-08T01:13:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:00.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:00.790877+0000) 2022-09-08T01:13:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:01.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:01.722338+0000) 2022-09-08T01:13:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:01.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:01.791054+0000) 2022-09-08T01:13:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:02 smithi055 ceph-mon[97033]: pgmap v6899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:13:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:02 smithi059 ceph-mon[97450]: pgmap v6899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:13:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:02.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:02.791257+0000) 2022-09-08T01:13:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:02.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:02.722555+0000) 2022-09-08T01:13:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:03.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:03.791457+0000) 2022-09-08T01:13:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:03.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:03.722759+0000) 2022-09-08T01:13:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:04 smithi055 ceph-mon[97033]: pgmap v6900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:04 smithi059 ceph-mon[97450]: pgmap v6900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:05.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:04.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:04.791659+0000) 2022-09-08T01:13:05.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:04.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:04.722971+0000) 2022-09-08T01:13:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:13:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:13:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:13:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:13:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:05.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:05.791817+0000) 2022-09-08T01:13:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:05.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:05.723236+0000) 2022-09-08T01:13:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:06 smithi059 ceph-mon[97450]: pgmap v6901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:13:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:06.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:06.723453+0000) 2022-09-08T01:13:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:06.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:06.792024+0000) 2022-09-08T01:13:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:06 smithi055 ceph-mon[97033]: pgmap v6901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:13:07.909 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:13:07.720+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:13:07.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:07.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:07.723691+0000) 2022-09-08T01:13:07.911 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:07.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:07.792268+0000) 2022-09-08T01:13:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:08.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:08.792538+0000) 2022-09-08T01:13:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:08.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:08.723929+0000) 2022-09-08T01:13:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:08 smithi055 ceph-mon[97033]: pgmap v6902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:13:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:08 smithi055 ceph-mon[97033]: pgmap v6903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T01:13:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:08 smithi055 ceph-mon[97033]: pgmap v6904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T01:13:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:13:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:08 smithi059 ceph-mon[97450]: pgmap v6902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:13:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:08 smithi059 ceph-mon[97450]: pgmap v6903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T01:13:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:08 smithi059 ceph-mon[97450]: pgmap v6904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T01:13:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:13:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:09.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:09.792755+0000) 2022-09-08T01:13:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:09.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:09.724089+0000) 2022-09-08T01:13:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:09 smithi055 ceph-mon[97033]: osdmap e515: 12 total, 11 up, 11 in 2022-09-08T01:13:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:09 smithi059 ceph-mon[97450]: osdmap e515: 12 total, 11 up, 11 in 2022-09-08T01:13:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:10.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:10.724250+0000) 2022-09-08T01:13:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:10.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:10.793057+0000) 2022-09-08T01:13:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:10 smithi055 ceph-mon[97033]: pgmap v6906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T01:13:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:10 smithi059 ceph-mon[97450]: pgmap v6906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T01:13:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:11.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:11.793207+0000) 2022-09-08T01:13:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:11.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:11.724400+0000) 2022-09-08T01:13:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:12 smithi055 ceph-mon[97033]: osdmap e516: 12 total, 11 up, 11 in 2022-09-08T01:13:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:12 smithi055 ceph-mon[97033]: pgmap v6908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 241 B/s wr, 7 op/s 2022-09-08T01:13:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:12 smithi059 ceph-mon[97450]: osdmap e516: 12 total, 11 up, 11 in 2022-09-08T01:13:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:12 smithi059 ceph-mon[97450]: pgmap v6908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 241 B/s wr, 7 op/s 2022-09-08T01:13:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:12.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:12.793376+0000) 2022-09-08T01:13:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:12.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:12.724598+0000) 2022-09-08T01:13:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:13.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:13.793625+0000) 2022-09-08T01:13:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:13.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:13.724848+0000) 2022-09-08T01:13:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:14.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:14.793847+0000) 2022-09-08T01:13:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:14.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:14.725025+0000) 2022-09-08T01:13:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:14 smithi055 ceph-mon[97033]: pgmap v6909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2022-09-08T01:13:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:14 smithi059 ceph-mon[97450]: pgmap v6909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2022-09-08T01:13:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:15.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:15.794021+0000) 2022-09-08T01:13:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:15.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:15.725272+0000) 2022-09-08T01:13:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:16.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:16.794175+0000) 2022-09-08T01:13:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:16.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:16.725521+0000) 2022-09-08T01:13:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:16 smithi055 ceph-mon[97033]: pgmap v6910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:13:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:16 smithi059 ceph-mon[97450]: pgmap v6910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:13:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:17.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:17.794391+0000) 2022-09-08T01:13:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:17.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:17.725704+0000) 2022-09-08T01:13:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:18.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:18.794604+0000) 2022-09-08T01:13:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:18.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:18.725926+0000) 2022-09-08T01:13:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:18 smithi055 ceph-mon[97033]: pgmap v6911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T01:13:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:18 smithi059 ceph-mon[97450]: pgmap v6911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T01:13:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:19.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:19.794833+0000) 2022-09-08T01:13:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:19.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:19.726141+0000) 2022-09-08T01:13:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:20.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:20.726380+0000) 2022-09-08T01:13:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:20.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:20.795126+0000) 2022-09-08T01:13:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:20 smithi055 ceph-mon[97033]: pgmap v6912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:13:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:20 smithi059 ceph-mon[97450]: pgmap v6912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:13:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:21.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:21.726712+0000) 2022-09-08T01:13:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:21.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:21.795320+0000) 2022-09-08T01:13:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:22 smithi059 ceph-mon[97450]: pgmap v6913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T01:13:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:22 smithi055 ceph-mon[97033]: pgmap v6913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T01:13:23.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:13:22.725+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:13:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:22.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:22.726941+0000) 2022-09-08T01:13:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:22.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:22.795553+0000) 2022-09-08T01:13:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:23.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:23.727096+0000) 2022-09-08T01:13:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:23.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:23.795804+0000) 2022-09-08T01:13:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:24.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:24.727287+0000) 2022-09-08T01:13:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:24.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:24.796033+0000) 2022-09-08T01:13:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:24 smithi055 ceph-mon[97033]: pgmap v6914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:13:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:24 smithi059 ceph-mon[97450]: pgmap v6914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:13:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:25.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:25.796272+0000) 2022-09-08T01:13:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:25.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:25.727532+0000) 2022-09-08T01:13:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:26.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:26.796509+0000) 2022-09-08T01:13:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:26.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:26.727748+0000) 2022-09-08T01:13:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:26 smithi055 ceph-mon[97033]: pgmap v6915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:13:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:26 smithi059 ceph-mon[97450]: pgmap v6915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:13:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:27.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:27.796730+0000) 2022-09-08T01:13:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:27.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:27.727919+0000) 2022-09-08T01:13:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:28.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:28.796854+0000) 2022-09-08T01:13:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:28.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:28.728149+0000) 2022-09-08T01:13:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:28 smithi059 ceph-mon[97450]: pgmap v6916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:28 smithi055 ceph-mon[97033]: pgmap v6916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:29.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:29.797055+0000) 2022-09-08T01:13:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:29.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:29.728362+0000) 2022-09-08T01:13:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:30.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:30.797256+0000) 2022-09-08T01:13:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:30.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:30.728586+0000) 2022-09-08T01:13:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:30 smithi059 ceph-mon[97450]: pgmap v6917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:13:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:30 smithi055 ceph-mon[97033]: pgmap v6917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:13:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:31.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:31.728745+0000) 2022-09-08T01:13:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:31.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:31.797447+0000) 2022-09-08T01:13:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:32 smithi059 ceph-mon[97450]: pgmap v6918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:32.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:32 smithi055 ceph-mon[97033]: pgmap v6918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:32.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:32.797665+0000) 2022-09-08T01:13:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:32.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:32.728981+0000) 2022-09-08T01:13:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:33.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:33.797811+0000) 2022-09-08T01:13:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:33.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:33.729225+0000) 2022-09-08T01:13:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:34.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:34.798042+0000) 2022-09-08T01:13:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:34.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:34.729444+0000) 2022-09-08T01:13:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:34 smithi055 ceph-mon[97033]: pgmap v6919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:34 smithi059 ceph-mon[97450]: pgmap v6919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:35.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:35.798264+0000) 2022-09-08T01:13:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:35.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:35.729626+0000) 2022-09-08T01:13:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:36.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:36.798524+0000) 2022-09-08T01:13:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:36.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:36.729772+0000) 2022-09-08T01:13:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:36 smithi059 ceph-mon[97450]: pgmap v6920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:13:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:36 smithi055 ceph-mon[97033]: pgmap v6920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:13:38.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:13:37.742+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:13:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:37.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:37.798772+0000) 2022-09-08T01:13:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:37.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:37.729989+0000) 2022-09-08T01:13:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:38.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:38.799002+0000) 2022-09-08T01:13:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:38.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:38.730223+0000) 2022-09-08T01:13:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:38 smithi059 ceph-mon[97450]: pgmap v6921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:38 smithi055 ceph-mon[97033]: pgmap v6921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:39.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:39.799234+0000) 2022-09-08T01:13:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:39.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:39.730456+0000) 2022-09-08T01:13:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:40.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:40.799510+0000) 2022-09-08T01:13:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:40.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:40.730741+0000) 2022-09-08T01:13:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:40 smithi059 ceph-mon[97450]: pgmap v6922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:13:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:40 smithi055 ceph-mon[97033]: pgmap v6922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:13:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:41.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:41.731002+0000) 2022-09-08T01:13:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:41.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:41.799749+0000) 2022-09-08T01:13:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:42 smithi059 ceph-mon[97450]: pgmap v6923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:42 smithi055 ceph-mon[97033]: pgmap v6923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:42.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:42.731192+0000) 2022-09-08T01:13:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:42.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:42.800008+0000) 2022-09-08T01:13:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:43.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:43.800194+0000) 2022-09-08T01:13:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:43.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:43.731374+0000) 2022-09-08T01:13:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:44.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:44.731522+0000) 2022-09-08T01:13:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:44.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:44.800327+0000) 2022-09-08T01:13:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:44 smithi059 ceph-mon[97450]: pgmap v6924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:44 smithi055 ceph-mon[97033]: pgmap v6924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:45.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:45.800478+0000) 2022-09-08T01:13:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:45.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:45.731752+0000) 2022-09-08T01:13:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:46.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:46.800740+0000) 2022-09-08T01:13:47.171 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:46.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:46.732007+0000) 2022-09-08T01:13:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:46 smithi059 ceph-mon[97450]: pgmap v6925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:13:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:46 smithi055 ceph-mon[97033]: pgmap v6925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:13:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:47.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:47.800883+0000) 2022-09-08T01:13:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:47.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:47.732285+0000) 2022-09-08T01:13:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:48.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:48.801131+0000) 2022-09-08T01:13:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:48.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:48.732461+0000) 2022-09-08T01:13:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:48 smithi059 ceph-mon[97450]: pgmap v6926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:49.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:48 smithi055 ceph-mon[97033]: pgmap v6926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:49.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:49.801372+0000) 2022-09-08T01:13:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:49.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:49.732682+0000) 2022-09-08T01:13:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:50.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:50.732805+0000) 2022-09-08T01:13:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:50.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:50.801606+0000) 2022-09-08T01:13:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:50 smithi059 ceph-mon[97450]: pgmap v6927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:13:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:50 smithi055 ceph-mon[97033]: pgmap v6927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:13:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:51.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:51.801814+0000) 2022-09-08T01:13:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:51.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:51.732944+0000) 2022-09-08T01:13:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:52 smithi059 ceph-mon[97450]: pgmap v6928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:52 smithi055 ceph-mon[97033]: pgmap v6928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:52.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:52.733148+0000) 2022-09-08T01:13:53.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:13:52.758+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:13:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:52.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:52.802007+0000) 2022-09-08T01:13:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:53.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:53.802238+0000) 2022-09-08T01:13:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:53.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:53.733374+0000) 2022-09-08T01:13:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:54.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:54.802430+0000) 2022-09-08T01:13:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:54.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:54.733607+0000) 2022-09-08T01:13:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:54 smithi059 ceph-mon[97450]: pgmap v6929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:54 smithi055 ceph-mon[97033]: pgmap v6929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:55.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:55.802657+0000) 2022-09-08T01:13:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:55.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:55.733844+0000) 2022-09-08T01:13:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:56.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:56.802873+0000) 2022-09-08T01:13:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:56.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:56.734129+0000) 2022-09-08T01:13:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:56 smithi059 ceph-mon[97450]: pgmap v6930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:13:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:56 smithi055 ceph-mon[97033]: pgmap v6930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:13:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:57.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:57.803018+0000) 2022-09-08T01:13:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:57.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:57.734415+0000) 2022-09-08T01:13:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:58.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:58.803230+0000) 2022-09-08T01:13:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:58.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:58.734610+0000) 2022-09-08T01:13:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:13:58 smithi059 ceph-mon[97450]: pgmap v6931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:13:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:13:58 smithi055 ceph-mon[97033]: pgmap v6931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:13:59.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:59.803436+0000) 2022-09-08T01:14:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:13:59.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:13:59.734834+0000) 2022-09-08T01:14:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:00.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:00.803579+0000) 2022-09-08T01:14:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:00.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:00.735027+0000) 2022-09-08T01:14:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:00 smithi059 ceph-mon[97450]: pgmap v6932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:14:01.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:00 smithi055 ceph-mon[97033]: pgmap v6932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:14:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:01.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:01.803801+0000) 2022-09-08T01:14:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:01.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:01.735253+0000) 2022-09-08T01:14:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:02 smithi059 ceph-mon[97450]: pgmap v6933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:02 smithi055 ceph-mon[97033]: pgmap v6933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:02.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:02.804021+0000) 2022-09-08T01:14:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:02.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:02.735470+0000) 2022-09-08T01:14:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:03.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:03.804274+0000) 2022-09-08T01:14:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:03.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:03.735736+0000) 2022-09-08T01:14:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:04.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:04.804497+0000) 2022-09-08T01:14:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:04.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:04.735980+0000) 2022-09-08T01:14:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:04 smithi059 ceph-mon[97450]: pgmap v6934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:04 smithi055 ceph-mon[97033]: pgmap v6934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:05.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:05.804742+0000) 2022-09-08T01:14:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:05.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:05.736237+0000) 2022-09-08T01:14:06.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:14:06.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:14:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:14:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:14:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:06.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:06.804904+0000) 2022-09-08T01:14:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:06.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:06.736447+0000) 2022-09-08T01:14:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:06 smithi059 ceph-mon[97450]: pgmap v6935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:14:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:06 smithi055 ceph-mon[97033]: pgmap v6935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:14:08.061 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:14:07.772+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:14:08.062 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:07.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:07.805154+0000) 2022-09-08T01:14:08.063 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:07.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:07.736721+0000) 2022-09-08T01:14:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:08.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:08.805390+0000) 2022-09-08T01:14:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:08.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:08.736923+0000) 2022-09-08T01:14:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:08 smithi059 ceph-mon[97450]: pgmap v6936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:14:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:14:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:14:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:14:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:08 smithi055 ceph-mon[97033]: pgmap v6936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:14:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:14:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:14:09.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:14:09.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:09.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:09.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:09.805623+0000) 2022-09-08T01:14:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:09.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:09.737132+0000) 2022-09-08T01:14:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:10 smithi059 ceph-mon[97450]: osdmap e517: 12 total, 11 up, 11 in 2022-09-08T01:14:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:10 smithi055 ceph-mon[97033]: osdmap e517: 12 total, 11 up, 11 in 2022-09-08T01:14:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:10.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:10.805897+0000) 2022-09-08T01:14:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:10.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:10.737299+0000) 2022-09-08T01:14:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:11 smithi059 ceph-mon[97450]: pgmap v6938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:14:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:11.360 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:11 smithi055 ceph-mon[97033]: pgmap v6938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:14:11.361 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:11.362 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:11.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:11.806127+0000) 2022-09-08T01:14:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:11.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:11.737499+0000) 2022-09-08T01:14:12.510 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:12.592 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:12.593 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:12 smithi059 ceph-mon[97450]: pgmap v6939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:14:12.690 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:12.691 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:12.692 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:12 smithi055 ceph-mon[97033]: pgmap v6939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:14:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:12.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:12.737673+0000) 2022-09-08T01:14:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:12.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:12.806358+0000) 2022-09-08T01:14:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:13.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:13.806600+0000) 2022-09-08T01:14:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:13.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:13.737889+0000) 2022-09-08T01:14:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:14.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:14.806770+0000) 2022-09-08T01:14:15.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:14.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:14.738068+0000) 2022-09-08T01:14:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:14 smithi055 ceph-mon[97033]: pgmap v6940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:14:15.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:15.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:15.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:15.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:15.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:14 smithi059 ceph-mon[97450]: pgmap v6940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:14:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:15.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:15.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:15.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:15.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:14:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:15.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:15.806914+0000) 2022-09-08T01:14:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:15.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:15.738259+0000) 2022-09-08T01:14:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:15 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T01:14:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:15 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T01:14:16.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:15 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T01:14:16.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:15 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T01:14:16.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:15 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T01:14:16.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:15 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T01:14:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:15 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T01:14:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:15 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T01:14:16.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:15 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T01:14:16.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:15 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T01:14:16.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:15 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T01:14:16.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:15 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T01:14:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:16.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:16.807062+0000) 2022-09-08T01:14:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:16.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:16.738530+0000) 2022-09-08T01:14:17.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:16 smithi059 ceph-mon[97450]: pgmap v6941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:14:17.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:16 smithi059 ceph-mon[97450]: osdmap e518: 12 total, 11 up, 11 in 2022-09-08T01:14:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:16 smithi055 ceph-mon[97033]: pgmap v6941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:14:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:16 smithi055 ceph-mon[97033]: osdmap e518: 12 total, 11 up, 11 in 2022-09-08T01:14:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:17.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:17.807277+0000) 2022-09-08T01:14:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:17.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:17.738716+0000) 2022-09-08T01:14:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:18.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:18.807504+0000) 2022-09-08T01:14:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:18.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:18.738948+0000) 2022-09-08T01:14:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:18 smithi059 ceph-mon[97450]: pgmap v6943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:14:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:19 smithi055 ceph-mon[97033]: pgmap v6943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:14:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:19.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:19.807724+0000) 2022-09-08T01:14:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:19.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:19.739193+0000) 2022-09-08T01:14:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:20.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:20.807957+0000) 2022-09-08T01:14:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:20.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:20.739403+0000) 2022-09-08T01:14:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:21 smithi059 ceph-mon[97450]: pgmap v6944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:14:21.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:21 smithi055 ceph-mon[97033]: pgmap v6944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:14:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:21.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:21.808191+0000) 2022-09-08T01:14:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:21.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:21.739621+0000) 2022-09-08T01:14:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:22 smithi059 ceph-mon[97450]: pgmap v6945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:14:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:22 smithi055 ceph-mon[97033]: pgmap v6945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:14:23.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:14:22.781+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:14:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:22.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:22.808380+0000) 2022-09-08T01:14:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:22.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:22.739856+0000) 2022-09-08T01:14:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:23.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:23.808564+0000) 2022-09-08T01:14:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:23.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:23.740023+0000) 2022-09-08T01:14:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:24.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:24.808778+0000) 2022-09-08T01:14:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:24.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:24.740139+0000) 2022-09-08T01:14:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:24 smithi059 ceph-mon[97450]: pgmap v6946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:14:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:24 smithi055 ceph-mon[97033]: pgmap v6946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:14:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:25.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:25.809001+0000) 2022-09-08T01:14:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:25.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:25.740368+0000) 2022-09-08T01:14:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:26.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:26.809215+0000) 2022-09-08T01:14:27.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:26.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:26.740519+0000) 2022-09-08T01:14:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:26 smithi059 ceph-mon[97450]: pgmap v6947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:14:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:26 smithi055 ceph-mon[97033]: pgmap v6947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:14:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:27.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:27.809365+0000) 2022-09-08T01:14:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:27.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:27.740707+0000) 2022-09-08T01:14:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:28.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:28.809547+0000) 2022-09-08T01:14:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:28.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:28.740866+0000) 2022-09-08T01:14:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:28 smithi059 ceph-mon[97450]: pgmap v6948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T01:14:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:28 smithi055 ceph-mon[97033]: pgmap v6948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T01:14:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:29.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:29.809770+0000) 2022-09-08T01:14:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:29.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:29.741056+0000) 2022-09-08T01:14:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:30.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:30.810005+0000) 2022-09-08T01:14:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:30.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:30.741234+0000) 2022-09-08T01:14:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:30 smithi059 ceph-mon[97450]: pgmap v6949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:14:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:30 smithi055 ceph-mon[97033]: pgmap v6949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:14:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:31.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:31.741474+0000) 2022-09-08T01:14:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:31.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:31.810250+0000) 2022-09-08T01:14:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:32 smithi059 ceph-mon[97450]: pgmap v6950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:32 smithi055 ceph-mon[97033]: pgmap v6950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:33.121 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:32.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:32.810467+0000) 2022-09-08T01:14:33.122 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:32.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:32.741661+0000) 2022-09-08T01:14:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:33.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:33.810710+0000) 2022-09-08T01:14:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:33.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:33.741801+0000) 2022-09-08T01:14:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:34.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:34.810889+0000) 2022-09-08T01:14:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:34.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:34.741963+0000) 2022-09-08T01:14:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:34 smithi059 ceph-mon[97450]: pgmap v6951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:34 smithi055 ceph-mon[97033]: pgmap v6951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:35.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:35.811124+0000) 2022-09-08T01:14:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:35.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:35.742131+0000) 2022-09-08T01:14:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:36.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:36.811344+0000) 2022-09-08T01:14:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:36.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:36.742291+0000) 2022-09-08T01:14:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:36 smithi059 ceph-mon[97450]: pgmap v6952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:14:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:36 smithi055 ceph-mon[97033]: pgmap v6952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:14:38.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:14:37.793+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:14:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:37.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:37.811568+0000) 2022-09-08T01:14:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:37.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:37.742502+0000) 2022-09-08T01:14:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:38.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:38.811765+0000) 2022-09-08T01:14:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:38.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:38.742739+0000) 2022-09-08T01:14:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:39 smithi059 ceph-mon[97450]: pgmap v6953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:39 smithi055 ceph-mon[97033]: pgmap v6953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:14:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:39.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:39.811988+0000) 2022-09-08T01:14:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:39.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:39.742925+0000) 2022-09-08T01:14:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:40 smithi059 ceph-mon[97450]: pgmap v6954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:14:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:40 smithi059 ceph-mon[97450]: pgmap v6955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:14:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:40 smithi055 ceph-mon[97033]: pgmap v6954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:14:40.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:40 smithi055 ceph-mon[97033]: pgmap v6955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:14:41.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:40.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:40.812210+0000) 2022-09-08T01:14:41.321 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:40.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:40.743067+0000) 2022-09-08T01:14:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:41.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:41.812432+0000) 2022-09-08T01:14:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:41.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:41.743285+0000) 2022-09-08T01:14:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:42 smithi059 ceph-mon[97450]: pgmap v6956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:14:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:42 smithi055 ceph-mon[97033]: pgmap v6956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:14:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:42.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:42.743513+0000) 2022-09-08T01:14:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:42.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:42.812712+0000) 2022-09-08T01:14:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:43.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:43.812901+0000) 2022-09-08T01:14:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:43.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:43.743742+0000) 2022-09-08T01:14:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:44 smithi059 ceph-mon[97450]: pgmap v6957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:14:44.493 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:44 smithi055 ceph-mon[97033]: pgmap v6957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:14:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:44.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:44.813090+0000) 2022-09-08T01:14:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:44.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:44.743979+0000) 2022-09-08T01:14:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:45.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:45.813326+0000) 2022-09-08T01:14:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:45.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:45.744186+0000) 2022-09-08T01:14:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:46 smithi059 ceph-mon[97450]: pgmap v6958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:14:46.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:46 smithi055 ceph-mon[97033]: pgmap v6958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:14:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:46.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:46.813548+0000) 2022-09-08T01:14:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:46.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:46.744365+0000) 2022-09-08T01:14:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:47.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:47.813807+0000) 2022-09-08T01:14:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:47.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:47.744590+0000) 2022-09-08T01:14:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:48 smithi059 ceph-mon[97450]: pgmap v6959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:14:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:48 smithi055 ceph-mon[97033]: pgmap v6959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:14:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:48.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:48.813991+0000) 2022-09-08T01:14:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:48.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:48.744791+0000) 2022-09-08T01:14:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:49.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:49.814239+0000) 2022-09-08T01:14:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:49.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:49.744987+0000) 2022-09-08T01:14:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:50 smithi059 ceph-mon[97450]: pgmap v6960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:14:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:50 smithi055 ceph-mon[97033]: pgmap v6960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:14:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:50.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:50.814487+0000) 2022-09-08T01:14:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:50.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:50.745219+0000) 2022-09-08T01:14:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:51.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:51.814617+0000) 2022-09-08T01:14:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:51.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:51.745378+0000) 2022-09-08T01:14:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:52 smithi059 ceph-mon[97450]: pgmap v6961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:14:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:52 smithi055 ceph-mon[97033]: pgmap v6961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:14:53.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:14:52.796+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:14:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:52.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:52.814861+0000) 2022-09-08T01:14:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:52.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:52.745586+0000) 2022-09-08T01:14:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:53.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:53.815009+0000) 2022-09-08T01:14:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:53.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:53.745809+0000) 2022-09-08T01:14:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:54 smithi059 ceph-mon[97450]: pgmap v6962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:54 smithi055 ceph-mon[97033]: pgmap v6962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:54.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:54.746022+0000) 2022-09-08T01:14:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:54.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:54.815193+0000) 2022-09-08T01:14:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:55.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:55.815417+0000) 2022-09-08T01:14:56.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:55.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:55.746265+0000) 2022-09-08T01:14:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:56 smithi055 ceph-mon[97033]: pgmap v6963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:56 smithi059 ceph-mon[97450]: pgmap v6963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:14:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:56.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:56.815612+0000) 2022-09-08T01:14:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:56.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:56.746545+0000) 2022-09-08T01:14:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:57.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:57.815872+0000) 2022-09-08T01:14:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:57.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:57.746775+0000) 2022-09-08T01:14:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:14:58 smithi055 ceph-mon[97033]: pgmap v6964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:14:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:14:58 smithi059 ceph-mon[97450]: pgmap v6964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:14:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:58.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:58.747033+0000) 2022-09-08T01:14:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:58.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:58.816095+0000) 2022-09-08T01:15:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:14:59.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:59.747308+0000) 2022-09-08T01:15:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:14:59.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:14:59.816215+0000) 2022-09-08T01:15:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:00 smithi055 ceph-mon[97033]: pgmap v6965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:00 smithi059 ceph-mon[97450]: pgmap v6965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:00.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:00.816413+0000) 2022-09-08T01:15:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:00.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:00.747536+0000) 2022-09-08T01:15:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:01.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:01.747780+0000) 2022-09-08T01:15:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:01.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:01.816674+0000) 2022-09-08T01:15:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:02 smithi055 ceph-mon[97033]: pgmap v6966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:15:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:02 smithi059 ceph-mon[97450]: pgmap v6966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:15:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:02.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:02.747982+0000) 2022-09-08T01:15:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:02.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:02.816941+0000) 2022-09-08T01:15:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:03.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:03.748240+0000) 2022-09-08T01:15:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:03.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:03.817188+0000) 2022-09-08T01:15:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:04 smithi055 ceph-mon[97033]: pgmap v6967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:04 smithi059 ceph-mon[97450]: pgmap v6967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:04.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:04.748380+0000) 2022-09-08T01:15:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:04.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:04.817319+0000) 2022-09-08T01:15:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:15:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:15:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:15:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:15:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:05.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:05.748581+0000) 2022-09-08T01:15:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:05.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:05.817537+0000) 2022-09-08T01:15:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:06 smithi055 ceph-mon[97033]: pgmap v6968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:06 smithi059 ceph-mon[97450]: pgmap v6968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:06.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:06.817757+0000) 2022-09-08T01:15:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:06.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:06.748924+0000) 2022-09-08T01:15:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:07.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:07.817971+0000) 2022-09-08T01:15:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:07.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:07.749150+0000) 2022-09-08T01:15:08.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:15:07.799+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:15:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:08 smithi055 ceph-mon[97033]: pgmap v6969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:15:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:08 smithi059 ceph-mon[97450]: pgmap v6969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:15:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:08.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:08.818223+0000) 2022-09-08T01:15:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:08.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:08.749440+0000) 2022-09-08T01:15:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:09.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:09.818471+0000) 2022-09-08T01:15:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:09.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:09.749701+0000) 2022-09-08T01:15:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:10 smithi055 ceph-mon[97033]: pgmap v6970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:15:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:10 smithi055 ceph-mon[97033]: osdmap e519: 12 total, 11 up, 11 in 2022-09-08T01:15:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:10 smithi059 ceph-mon[97450]: pgmap v6970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:15:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:10 smithi059 ceph-mon[97450]: osdmap e519: 12 total, 11 up, 11 in 2022-09-08T01:15:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:10.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:10.818693+0000) 2022-09-08T01:15:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:10.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:10.749989+0000) 2022-09-08T01:15:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:11.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:11.818938+0000) 2022-09-08T01:15:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:11.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:11.750220+0000) 2022-09-08T01:15:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:12 smithi055 ceph-mon[97033]: pgmap v6972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:15:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:12 smithi055 ceph-mon[97033]: osdmap e520: 12 total, 11 up, 11 in 2022-09-08T01:15:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:12 smithi059 ceph-mon[97450]: pgmap v6972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:15:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:12 smithi059 ceph-mon[97450]: osdmap e520: 12 total, 11 up, 11 in 2022-09-08T01:15:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:12.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:12.819218+0000) 2022-09-08T01:15:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:12.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:12.750453+0000) 2022-09-08T01:15:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:13.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:13.819460+0000) 2022-09-08T01:15:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:13.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:13.750621+0000) 2022-09-08T01:15:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:14 smithi055 ceph-mon[97033]: pgmap v6974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:15:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:14 smithi059 ceph-mon[97450]: pgmap v6974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:15:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:14.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:14.750885+0000) 2022-09-08T01:15:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:14.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:14.819615+0000) 2022-09-08T01:15:16.160 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:15.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:15.819799+0000) 2022-09-08T01:15:16.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:15.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:15.751179+0000) 2022-09-08T01:15:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:16 smithi055 ceph-mon[97033]: pgmap v6975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:15:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:16 smithi059 ceph-mon[97450]: pgmap v6975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:15:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:16.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:16.820008+0000) 2022-09-08T01:15:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:16.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:16.751470+0000) 2022-09-08T01:15:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:17.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:17.820175+0000) 2022-09-08T01:15:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:17.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:17.751769+0000) 2022-09-08T01:15:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:18 smithi055 ceph-mon[97033]: pgmap v6976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:15:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:18 smithi059 ceph-mon[97450]: pgmap v6976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:15:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:18.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:18.820384+0000) 2022-09-08T01:15:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:18.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:18.752019+0000) 2022-09-08T01:15:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:19.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:19.752284+0000) 2022-09-08T01:15:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:19.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:19.820521+0000) 2022-09-08T01:15:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:20 smithi055 ceph-mon[97033]: pgmap v6977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 224 B/s wr, 7 op/s 2022-09-08T01:15:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:20 smithi059 ceph-mon[97450]: pgmap v6977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 224 B/s wr, 7 op/s 2022-09-08T01:15:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:20.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:20.820725+0000) 2022-09-08T01:15:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:20.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:20.752546+0000) 2022-09-08T01:15:22.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:21.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:21.820936+0000) 2022-09-08T01:15:22.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:21.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:21.752830+0000) 2022-09-08T01:15:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:22 smithi055 ceph-mon[97033]: pgmap v6978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:15:22.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:22 smithi059 ceph-mon[97450]: pgmap v6978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:15:23.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:15:22.810+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:15:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:22.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:22.821179+0000) 2022-09-08T01:15:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:22.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:22.753062+0000) 2022-09-08T01:15:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:23.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:23.821411+0000) 2022-09-08T01:15:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:23.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:23.753287+0000) 2022-09-08T01:15:24.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:24 smithi055 ceph-mon[97033]: pgmap v6979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 95 B/s wr, 5 op/s 2022-09-08T01:15:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:24 smithi059 ceph-mon[97450]: pgmap v6979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 95 B/s wr, 5 op/s 2022-09-08T01:15:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:24.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:24.821604+0000) 2022-09-08T01:15:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:24.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:24.753526+0000) 2022-09-08T01:15:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:25.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:25.821806+0000) 2022-09-08T01:15:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:25.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:25.753799+0000) 2022-09-08T01:15:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:26 smithi055 ceph-mon[97033]: pgmap v6980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:15:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:26 smithi059 ceph-mon[97450]: pgmap v6980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:15:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:26.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:26.821938+0000) 2022-09-08T01:15:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:26.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:26.754042+0000) 2022-09-08T01:15:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:27.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:27.822134+0000) 2022-09-08T01:15:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:27.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:27.754219+0000) 2022-09-08T01:15:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:28 smithi055 ceph-mon[97033]: pgmap v6981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:15:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:28 smithi059 ceph-mon[97450]: pgmap v6981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:15:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:28.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:28.822284+0000) 2022-09-08T01:15:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:28.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:28.754419+0000) 2022-09-08T01:15:29.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:29 smithi055 ceph-mon[97033]: pgmap v6982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:29 smithi059 ceph-mon[97450]: pgmap v6982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:29.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:29.822415+0000) 2022-09-08T01:15:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:29.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:29.754699+0000) 2022-09-08T01:15:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:30.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:30.822599+0000) 2022-09-08T01:15:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:30.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:30.754949+0000) 2022-09-08T01:15:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:31 smithi059 ceph-mon[97450]: pgmap v6983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:15:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:31 smithi055 ceph-mon[97033]: pgmap v6983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:15:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:31.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:31.822839+0000) 2022-09-08T01:15:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:31.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:31.755188+0000) 2022-09-08T01:15:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:32.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:32.823085+0000) 2022-09-08T01:15:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:32.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:32.755486+0000) 2022-09-08T01:15:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:33 smithi059 ceph-mon[97450]: pgmap v6984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:33 smithi055 ceph-mon[97033]: pgmap v6984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:34.161 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:33.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:33.823258+0000) 2022-09-08T01:15:34.198 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:33.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:33.755787+0000) 2022-09-08T01:15:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:34.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:34.823388+0000) 2022-09-08T01:15:35.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:34.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:34.756034+0000) 2022-09-08T01:15:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:35 smithi059 ceph-mon[97450]: pgmap v6985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:35 smithi055 ceph-mon[97033]: pgmap v6985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:35.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:35.823593+0000) 2022-09-08T01:15:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:35.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:35.756178+0000) 2022-09-08T01:15:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:36.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:36.823831+0000) 2022-09-08T01:15:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:36.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:36.756424+0000) 2022-09-08T01:15:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:37 smithi059 ceph-mon[97450]: pgmap v6986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:15:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:37 smithi055 ceph-mon[97033]: pgmap v6986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:15:38.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:15:37.825+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:15:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:37.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:37.824029+0000) 2022-09-08T01:15:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:37.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:37.756599+0000) 2022-09-08T01:15:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:15:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:15:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:15:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:15:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:15:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:15:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:38.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:38.824234+0000) 2022-09-08T01:15:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:38.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:38.756820+0000) 2022-09-08T01:15:40.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:39.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:39.824449+0000) 2022-09-08T01:15:40.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:39.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:39.757033+0000) 2022-09-08T01:15:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:39 smithi055 ceph-mon[97033]: pgmap v6987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:15:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:15:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:15:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:15:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:15:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:15:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:39 smithi059 ceph-mon[97450]: pgmap v6987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:15:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:15:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:15:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:15:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:15:40.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:15:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:40.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:40.824678+0000) 2022-09-08T01:15:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:40.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:40.757231+0000) 2022-09-08T01:15:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:41.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:41.824909+0000) 2022-09-08T01:15:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:41.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:41.757518+0000) 2022-09-08T01:15:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:41 smithi055 ceph-mon[97033]: pgmap v6988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:15:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:41 smithi059 ceph-mon[97450]: pgmap v6988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:15:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:42.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:42.825108+0000) 2022-09-08T01:15:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:42.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:42.757810+0000) 2022-09-08T01:15:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:43.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:43.758011+0000) 2022-09-08T01:15:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:43.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:43.825325+0000) 2022-09-08T01:15:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:43 smithi055 ceph-mon[97033]: pgmap v6989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:43 smithi059 ceph-mon[97450]: pgmap v6989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:44.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:44.825555+0000) 2022-09-08T01:15:45.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:44.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:44.758194+0000) 2022-09-08T01:15:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:45.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:45.825719+0000) 2022-09-08T01:15:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:45.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:45.758430+0000) 2022-09-08T01:15:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:45 smithi055 ceph-mon[97033]: pgmap v6990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:45 smithi059 ceph-mon[97450]: pgmap v6990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:46.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:46.825937+0000) 2022-09-08T01:15:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:46.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:46.758737+0000) 2022-09-08T01:15:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:47.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:47.826173+0000) 2022-09-08T01:15:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:47.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:47.758931+0000) 2022-09-08T01:15:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:47 smithi055 ceph-mon[97033]: pgmap v6991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:15:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:47 smithi059 ceph-mon[97450]: pgmap v6991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:15:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:48.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:48.759156+0000) 2022-09-08T01:15:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:48.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:48.826385+0000) 2022-09-08T01:15:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:49.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:49.826682+0000) 2022-09-08T01:15:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:49.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:49.759376+0000) 2022-09-08T01:15:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:49 smithi055 ceph-mon[97033]: pgmap v6992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:49 smithi059 ceph-mon[97450]: pgmap v6992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:50.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:50.826931+0000) 2022-09-08T01:15:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:50.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:50.759681+0000) 2022-09-08T01:15:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:51.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:51.827059+0000) 2022-09-08T01:15:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:51.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:51.759973+0000) 2022-09-08T01:15:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:51 smithi055 ceph-mon[97033]: pgmap v6993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:15:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:51 smithi059 ceph-mon[97450]: pgmap v6993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:15:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:52.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:52.827183+0000) 2022-09-08T01:15:53.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:15:52.839+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:15:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:52.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:52.760173+0000) 2022-09-08T01:15:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:53.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:53.760320+0000) 2022-09-08T01:15:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:53.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:53.827373+0000) 2022-09-08T01:15:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:53 smithi055 ceph-mon[97033]: pgmap v6994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:53 smithi059 ceph-mon[97450]: pgmap v6994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:54.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:54.827601+0000) 2022-09-08T01:15:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:54.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:54.760539+0000) 2022-09-08T01:15:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:55.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:55.827823+0000) 2022-09-08T01:15:56.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:55.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:55.760782+0000) 2022-09-08T01:15:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:55 smithi055 ceph-mon[97033]: pgmap v6995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:55 smithi059 ceph-mon[97450]: pgmap v6995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:15:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:56.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:56.828025+0000) 2022-09-08T01:15:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:56.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:56.761057+0000) 2022-09-08T01:15:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:57.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:57.828179+0000) 2022-09-08T01:15:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:57.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:57.761391+0000) 2022-09-08T01:15:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:57 smithi055 ceph-mon[97033]: pgmap v6996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:15:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:57 smithi059 ceph-mon[97450]: pgmap v6996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:15:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:58.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:58.761701+0000) 2022-09-08T01:15:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:58.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:58.828394+0000) 2022-09-08T01:16:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:15:59.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:59.828623+0000) 2022-09-08T01:16:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:15:59.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:15:59.761871+0000) 2022-09-08T01:16:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:15:59 smithi055 ceph-mon[97033]: pgmap v6997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:15:59 smithi059 ceph-mon[97450]: pgmap v6997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:00.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:00.828784+0000) 2022-09-08T01:16:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:00.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:00.762058+0000) 2022-09-08T01:16:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:01.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:01.829055+0000) 2022-09-08T01:16:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:01.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:01.762243+0000) 2022-09-08T01:16:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:01 smithi055 ceph-mon[97033]: pgmap v6998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:16:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:01 smithi059 ceph-mon[97450]: pgmap v6998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:16:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:02.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:02.829299+0000) 2022-09-08T01:16:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:02.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:02.762473+0000) 2022-09-08T01:16:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:03.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:03.829539+0000) 2022-09-08T01:16:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:03.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:03.762677+0000) 2022-09-08T01:16:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:03 smithi055 ceph-mon[97033]: pgmap v6999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:03 smithi059 ceph-mon[97450]: pgmap v6999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:04.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:04.829766+0000) 2022-09-08T01:16:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:04.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:04.762901+0000) 2022-09-08T01:16:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:05.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:05.829973+0000) 2022-09-08T01:16:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:05.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:05.763151+0000) 2022-09-08T01:16:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:05 smithi055 ceph-mon[97033]: pgmap v7000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:16:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:16:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:05 smithi059 ceph-mon[97450]: pgmap v7000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:16:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:16:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:06.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:06.830175+0000) 2022-09-08T01:16:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:06.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:06.763304+0000) 2022-09-08T01:16:08.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:16:07.841+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:16:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:07.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:07.830400+0000) 2022-09-08T01:16:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:07.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:07.763486+0000) 2022-09-08T01:16:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:07 smithi055 ceph-mon[97033]: pgmap v7001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:16:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:07 smithi059 ceph-mon[97450]: pgmap v7001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:16:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:08.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:08.830679+0000) 2022-09-08T01:16:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:08.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:08.763711+0000) 2022-09-08T01:16:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:09 smithi055 ceph-mon[97033]: pgmap v7002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:16:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:09 smithi055 ceph-mon[97033]: pgmap v7003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T01:16:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:09 smithi055 ceph-mon[97033]: pgmap v7004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T01:16:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:16:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:09 smithi055 ceph-mon[97033]: osdmap e521: 12 total, 11 up, 11 in 2022-09-08T01:16:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:09 smithi059 ceph-mon[97450]: pgmap v7002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:16:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:09 smithi059 ceph-mon[97450]: pgmap v7003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T01:16:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:09 smithi059 ceph-mon[97450]: pgmap v7004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T01:16:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:16:09.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:09 smithi059 ceph-mon[97450]: osdmap e521: 12 total, 11 up, 11 in 2022-09-08T01:16:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:09.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:09.830925+0000) 2022-09-08T01:16:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:09.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:09.763942+0000) 2022-09-08T01:16:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:10.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:10.831186+0000) 2022-09-08T01:16:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:10.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:10.764183+0000) 2022-09-08T01:16:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:11 smithi059 ceph-mon[97450]: pgmap v7006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T01:16:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:11 smithi055 ceph-mon[97033]: pgmap v7006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T01:16:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:11.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:11.831483+0000) 2022-09-08T01:16:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:11.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:11.764443+0000) 2022-09-08T01:16:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:12 smithi059 ceph-mon[97450]: osdmap e522: 12 total, 11 up, 11 in 2022-09-08T01:16:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:12 smithi055 ceph-mon[97033]: osdmap e522: 12 total, 11 up, 11 in 2022-09-08T01:16:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:12.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:12.831750+0000) 2022-09-08T01:16:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:12.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:12.764685+0000) 2022-09-08T01:16:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:13 smithi059 ceph-mon[97450]: pgmap v7008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 239 B/s wr, 7 op/s 2022-09-08T01:16:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:13 smithi055 ceph-mon[97033]: pgmap v7008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 239 B/s wr, 7 op/s 2022-09-08T01:16:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:13.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:13.832005+0000) 2022-09-08T01:16:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:13.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:13.764824+0000) 2022-09-08T01:16:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:14.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:14.832210+0000) 2022-09-08T01:16:15.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:14.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:14.764942+0000) 2022-09-08T01:16:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:15 smithi055 ceph-mon[97033]: pgmap v7009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2022-09-08T01:16:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:15 smithi059 ceph-mon[97450]: pgmap v7009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2022-09-08T01:16:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:15.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:15.832449+0000) 2022-09-08T01:16:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:15.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:15.765186+0000) 2022-09-08T01:16:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:16.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:16.765432+0000) 2022-09-08T01:16:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:16.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:16.832756+0000) 2022-09-08T01:16:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:17 smithi055 ceph-mon[97033]: pgmap v7010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:16:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:17 smithi059 ceph-mon[97450]: pgmap v7010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:16:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:17.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:17.832952+0000) 2022-09-08T01:16:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:17.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:17.765729+0000) 2022-09-08T01:16:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:18.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:18.833252+0000) 2022-09-08T01:16:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:18.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:18.765953+0000) 2022-09-08T01:16:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:19 smithi055 ceph-mon[97033]: pgmap v7011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2022-09-08T01:16:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:19 smithi059 ceph-mon[97450]: pgmap v7011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2022-09-08T01:16:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:19.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:19.833455+0000) 2022-09-08T01:16:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:19.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:19.766125+0000) 2022-09-08T01:16:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:20.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:20.833672+0000) 2022-09-08T01:16:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:20.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:20.766341+0000) 2022-09-08T01:16:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:21 smithi055 ceph-mon[97033]: pgmap v7012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:16:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:21 smithi059 ceph-mon[97450]: pgmap v7012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:16:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:21.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:21.833897+0000) 2022-09-08T01:16:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:21.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:21.766511+0000) 2022-09-08T01:16:23.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:16:22.851+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:16:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:22.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:22.834124+0000) 2022-09-08T01:16:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:22.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:22.766744+0000) 2022-09-08T01:16:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:23 smithi055 ceph-mon[97033]: pgmap v7013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T01:16:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:23 smithi059 ceph-mon[97450]: pgmap v7013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T01:16:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:23.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:23.834373+0000) 2022-09-08T01:16:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:23.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:23.767014+0000) 2022-09-08T01:16:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:24.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:24.834604+0000) 2022-09-08T01:16:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:24.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:24.767164+0000) 2022-09-08T01:16:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:25 smithi055 ceph-mon[97033]: pgmap v7014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:16:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:25 smithi059 ceph-mon[97450]: pgmap v7014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:16:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:25.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:25.834839+0000) 2022-09-08T01:16:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:25.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:25.767381+0000) 2022-09-08T01:16:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:26.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:26.835078+0000) 2022-09-08T01:16:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:26.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:26.767589+0000) 2022-09-08T01:16:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:27 smithi055 ceph-mon[97033]: pgmap v7015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:16:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:27 smithi059 ceph-mon[97450]: pgmap v7015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:16:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:27.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:27.835293+0000) 2022-09-08T01:16:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:27.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:27.767847+0000) 2022-09-08T01:16:28.836 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:28.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:28.768083+0000) 2022-09-08T01:16:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:28.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:28.835502+0000) 2022-09-08T01:16:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:29 smithi055 ceph-mon[97033]: pgmap v7016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:29 smithi059 ceph-mon[97450]: pgmap v7016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:29.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:29.768249+0000) 2022-09-08T01:16:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:29.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:29.835679+0000) 2022-09-08T01:16:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:30.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:30.835803+0000) 2022-09-08T01:16:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:30.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:30.768516+0000) 2022-09-08T01:16:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:31 smithi055 ceph-mon[97033]: pgmap v7017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:16:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:31 smithi059 ceph-mon[97450]: pgmap v7017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:16:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:31.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:31.768758+0000) 2022-09-08T01:16:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:31.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:31.836015+0000) 2022-09-08T01:16:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:32.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:32.836193+0000) 2022-09-08T01:16:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:32.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:32.768891+0000) 2022-09-08T01:16:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:33 smithi055 ceph-mon[97033]: pgmap v7018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:33 smithi059 ceph-mon[97450]: pgmap v7018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:33.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:33.836351+0000) 2022-09-08T01:16:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:33.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:33.769121+0000) 2022-09-08T01:16:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:34.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:34.836510+0000) 2022-09-08T01:16:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:34.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:34.769307+0000) 2022-09-08T01:16:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:35 smithi055 ceph-mon[97033]: pgmap v7019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:35 smithi059 ceph-mon[97450]: pgmap v7019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:35.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:35.836668+0000) 2022-09-08T01:16:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:35.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:35.769541+0000) 2022-09-08T01:16:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:36.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:36.836863+0000) 2022-09-08T01:16:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:36.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:36.769780+0000) 2022-09-08T01:16:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:37 smithi055 ceph-mon[97033]: pgmap v7020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:16:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:37 smithi059 ceph-mon[97450]: pgmap v7020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:16:38.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:16:37.858+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:16:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:37.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:37.837020+0000) 2022-09-08T01:16:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:37.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:37.770024+0000) 2022-09-08T01:16:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:38.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:38.770186+0000) 2022-09-08T01:16:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:38.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:38.837258+0000) 2022-09-08T01:16:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:39 smithi055 ceph-mon[97033]: pgmap v7021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:39 smithi059 ceph-mon[97450]: pgmap v7021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:39.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:39.837521+0000) 2022-09-08T01:16:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:39.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:39.770351+0000) 2022-09-08T01:16:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:40.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:40.837677+0000) 2022-09-08T01:16:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:40.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:40.770588+0000) 2022-09-08T01:16:41.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:41 smithi055 ceph-mon[97033]: pgmap v7022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:16:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:41 smithi059 ceph-mon[97450]: pgmap v7022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:16:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:41.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:41.770813+0000) 2022-09-08T01:16:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:41.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:41.837803+0000) 2022-09-08T01:16:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:42.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:42.838017+0000) 2022-09-08T01:16:43.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:42.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:42.771092+0000) 2022-09-08T01:16:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:43 smithi055 ceph-mon[97033]: pgmap v7023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:43 smithi059 ceph-mon[97450]: pgmap v7023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:43.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:43.838250+0000) 2022-09-08T01:16:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:43.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:43.771360+0000) 2022-09-08T01:16:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:44.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:44.838448+0000) 2022-09-08T01:16:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:44.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:44.771562+0000) 2022-09-08T01:16:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:45 smithi055 ceph-mon[97033]: pgmap v7024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:45 smithi059 ceph-mon[97450]: pgmap v7024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:45.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:45.838698+0000) 2022-09-08T01:16:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:45.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:45.771752+0000) 2022-09-08T01:16:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:46.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:46.838877+0000) 2022-09-08T01:16:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:46.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:46.771936+0000) 2022-09-08T01:16:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:47 smithi055 ceph-mon[97033]: pgmap v7025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:16:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:47 smithi059 ceph-mon[97450]: pgmap v7025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:16:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:47.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:47.839115+0000) 2022-09-08T01:16:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:47.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:47.772164+0000) 2022-09-08T01:16:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:48.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:48.839417+0000) 2022-09-08T01:16:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:48.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:48.772390+0000) 2022-09-08T01:16:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:49 smithi055 ceph-mon[97033]: pgmap v7026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:49 smithi059 ceph-mon[97450]: pgmap v7026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:49.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:49.839663+0000) 2022-09-08T01:16:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:49.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:49.772695+0000) 2022-09-08T01:16:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:50.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:50.839899+0000) 2022-09-08T01:16:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:50.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:50.772952+0000) 2022-09-08T01:16:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:51 smithi055 ceph-mon[97033]: pgmap v7027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:16:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:51 smithi059 ceph-mon[97450]: pgmap v7027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:16:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:51.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:51.840037+0000) 2022-09-08T01:16:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:51.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:51.773178+0000) 2022-09-08T01:16:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:52.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:52.840272+0000) 2022-09-08T01:16:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:52.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:52.773426+0000) 2022-09-08T01:16:53.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:16:52.874+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:16:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:53 smithi055 ceph-mon[97033]: pgmap v7028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:53 smithi059 ceph-mon[97450]: pgmap v7028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:53.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:53.840506+0000) 2022-09-08T01:16:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:53.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:53.773735+0000) 2022-09-08T01:16:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:54.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:54.840663+0000) 2022-09-08T01:16:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:54.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:54.773952+0000) 2022-09-08T01:16:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:55 smithi055 ceph-mon[97033]: pgmap v7029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:55 smithi059 ceph-mon[97450]: pgmap v7029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:16:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:55.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:55.840877+0000) 2022-09-08T01:16:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:55.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:55.774171+0000) 2022-09-08T01:16:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:56.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:56.841035+0000) 2022-09-08T01:16:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:56.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:56.774385+0000) 2022-09-08T01:16:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:57 smithi055 ceph-mon[97033]: pgmap v7030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:16:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:57 smithi059 ceph-mon[97450]: pgmap v7030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:16:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:57.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:57.841286+0000) 2022-09-08T01:16:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:57.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:57.774601+0000) 2022-09-08T01:16:59.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:58.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:58.841543+0000) 2022-09-08T01:16:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:58.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:58.774719+0000) 2022-09-08T01:16:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:16:59 smithi055 ceph-mon[97033]: pgmap v7031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:16:59 smithi059 ceph-mon[97450]: pgmap v7031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:16:59.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:59.841769+0000) 2022-09-08T01:17:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:16:59.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:16:59.774906+0000) 2022-09-08T01:17:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:00.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:00.841916+0000) 2022-09-08T01:17:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:00.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:00.775071+0000) 2022-09-08T01:17:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:01 smithi055 ceph-mon[97033]: pgmap v7032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:17:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:01 smithi059 ceph-mon[97450]: pgmap v7032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:17:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:01.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:01.775334+0000) 2022-09-08T01:17:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:01.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:01.842139+0000) 2022-09-08T01:17:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:02.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:02.842332+0000) 2022-09-08T01:17:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:02.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:02.775619+0000) 2022-09-08T01:17:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:03 smithi055 ceph-mon[97033]: pgmap v7033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:03 smithi059 ceph-mon[97450]: pgmap v7033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:03.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:03.842612+0000) 2022-09-08T01:17:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:03.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:03.775953+0000) 2022-09-08T01:17:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:04.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:04.842832+0000) 2022-09-08T01:17:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:04.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:04.776182+0000) 2022-09-08T01:17:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:05 smithi055 ceph-mon[97033]: pgmap v7034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:17:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:17:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:05 smithi059 ceph-mon[97450]: pgmap v7034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:17:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:17:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:05.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:05.843023+0000) 2022-09-08T01:17:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:05.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:05.776444+0000) 2022-09-08T01:17:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:06.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:06.843279+0000) 2022-09-08T01:17:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:06.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:06.776738+0000) 2022-09-08T01:17:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:07 smithi055 ceph-mon[97033]: pgmap v7035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:17:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:07 smithi059 ceph-mon[97450]: pgmap v7035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:17:08.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:17:07.890+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:17:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:07.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:07.843540+0000) 2022-09-08T01:17:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:07.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:07.776912+0000) 2022-09-08T01:17:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:08.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:08.843752+0000) 2022-09-08T01:17:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:08.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:08.777107+0000) 2022-09-08T01:17:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:17:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:17:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:17:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:17:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:17:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:17:09.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:09 smithi055 ceph-mon[97033]: pgmap v7036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:17:09.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:09 smithi055 ceph-mon[97033]: osdmap e523: 12 total, 11 up, 11 in 2022-09-08T01:17:10.043 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:09 smithi059 ceph-mon[97450]: pgmap v7036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:17:10.044 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:09 smithi059 ceph-mon[97450]: osdmap e523: 12 total, 11 up, 11 in 2022-09-08T01:17:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:09.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:09.777245+0000) 2022-09-08T01:17:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:09.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:09.843917+0000) 2022-09-08T01:17:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:10.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:10.844130+0000) 2022-09-08T01:17:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:10.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:10.777512+0000) 2022-09-08T01:17:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:11 smithi055 ceph-mon[97033]: pgmap v7038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:17:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:11 smithi055 ceph-mon[97033]: osdmap e524: 12 total, 11 up, 11 in 2022-09-08T01:17:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:11 smithi059 ceph-mon[97450]: pgmap v7038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:17:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:11 smithi059 ceph-mon[97450]: osdmap e524: 12 total, 11 up, 11 in 2022-09-08T01:17:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:11.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:11.844329+0000) 2022-09-08T01:17:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:11.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:11.777805+0000) 2022-09-08T01:17:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:12.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:12.844574+0000) 2022-09-08T01:17:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:12.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:12.778105+0000) 2022-09-08T01:17:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:13 smithi055 ceph-mon[97033]: pgmap v7040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:17:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:13 smithi059 ceph-mon[97450]: pgmap v7040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:17:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:13.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:13.844805+0000) 2022-09-08T01:17:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:13.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:13.778334+0000) 2022-09-08T01:17:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:14.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:14.844968+0000) 2022-09-08T01:17:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:14.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:14.778513+0000) 2022-09-08T01:17:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:15 smithi055 ceph-mon[97033]: pgmap v7041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:17:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:15 smithi059 ceph-mon[97450]: pgmap v7041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:17:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:15.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:15.845172+0000) 2022-09-08T01:17:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:15.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:15.778762+0000) 2022-09-08T01:17:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:16.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:16.845386+0000) 2022-09-08T01:17:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:16.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:16.779068+0000) 2022-09-08T01:17:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:17 smithi055 ceph-mon[97033]: pgmap v7042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:17:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:17 smithi059 ceph-mon[97450]: pgmap v7042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:17:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:17.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:17.845612+0000) 2022-09-08T01:17:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:17.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:17.779355+0000) 2022-09-08T01:17:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:18.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:18.845875+0000) 2022-09-08T01:17:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:18.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:18.779612+0000) 2022-09-08T01:17:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:19 smithi055 ceph-mon[97033]: pgmap v7043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T01:17:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:19 smithi059 ceph-mon[97450]: pgmap v7043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T01:17:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:19.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:19.846081+0000) 2022-09-08T01:17:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:19.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:19.779806+0000) 2022-09-08T01:17:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:20.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:20.846216+0000) 2022-09-08T01:17:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:20.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:20.780020+0000) 2022-09-08T01:17:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:21 smithi055 ceph-mon[97033]: pgmap v7044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:17:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:21 smithi059 ceph-mon[97450]: pgmap v7044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:17:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:21.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:21.846362+0000) 2022-09-08T01:17:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:21.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:21.780239+0000) 2022-09-08T01:17:23.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:17:22.900+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:17:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:22.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:22.846564+0000) 2022-09-08T01:17:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:22.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:22.780373+0000) 2022-09-08T01:17:23.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:23 smithi055 ceph-mon[97033]: pgmap v7045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T01:17:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:23 smithi059 ceph-mon[97450]: pgmap v7045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T01:17:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:23.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:23.780609+0000) 2022-09-08T01:17:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:23.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:23.846794+0000) 2022-09-08T01:17:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:24.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:24.846984+0000) 2022-09-08T01:17:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:24.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:24.780881+0000) 2022-09-08T01:17:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:25 smithi055 ceph-mon[97033]: pgmap v7046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:17:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:25 smithi059 ceph-mon[97450]: pgmap v7046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:17:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:25.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:25.847103+0000) 2022-09-08T01:17:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:25.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:25.781117+0000) 2022-09-08T01:17:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:26.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:26.847328+0000) 2022-09-08T01:17:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:26.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:26.781307+0000) 2022-09-08T01:17:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:27 smithi055 ceph-mon[97033]: pgmap v7047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:17:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:27 smithi059 ceph-mon[97450]: pgmap v7047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:17:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:27.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:27.781527+0000) 2022-09-08T01:17:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:27.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:27.847545+0000) 2022-09-08T01:17:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:28.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:28.847757+0000) 2022-09-08T01:17:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:28.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:28.781709+0000) 2022-09-08T01:17:29.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:29 smithi055 ceph-mon[97033]: pgmap v7048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:29 smithi059 ceph-mon[97450]: pgmap v7048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:29.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:29.847986+0000) 2022-09-08T01:17:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:29.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:29.781916+0000) 2022-09-08T01:17:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:30.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:30.848221+0000) 2022-09-08T01:17:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:30.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:30.782080+0000) 2022-09-08T01:17:31.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:31 smithi055 ceph-mon[97033]: pgmap v7049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:17:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:31 smithi059 ceph-mon[97450]: pgmap v7049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:17:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:31.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:31.848422+0000) 2022-09-08T01:17:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:31.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:31.782262+0000) 2022-09-08T01:17:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:32.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:32.848672+0000) 2022-09-08T01:17:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:32.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:32.782466+0000) 2022-09-08T01:17:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:33 smithi055 ceph-mon[97033]: pgmap v7050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:33 smithi059 ceph-mon[97450]: pgmap v7050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:33.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:33.848892+0000) 2022-09-08T01:17:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:33.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:33.782689+0000) 2022-09-08T01:17:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:34.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:34.849131+0000) 2022-09-08T01:17:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:34.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:34.782866+0000) 2022-09-08T01:17:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:35 smithi055 ceph-mon[97033]: pgmap v7051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:35 smithi059 ceph-mon[97450]: pgmap v7051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:35.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:35.849397+0000) 2022-09-08T01:17:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:35.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:35.783102+0000) 2022-09-08T01:17:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:36.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:36.849572+0000) 2022-09-08T01:17:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:36.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:36.783352+0000) 2022-09-08T01:17:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:37 smithi055 ceph-mon[97033]: pgmap v7052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:17:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:37 smithi059 ceph-mon[97450]: pgmap v7052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:17:38.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:17:37.910+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:17:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:37.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:37.783602+0000) 2022-09-08T01:17:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:37.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:37.849807+0000) 2022-09-08T01:17:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:38.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:38.783779+0000) 2022-09-08T01:17:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:38.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:38.850045+0000) 2022-09-08T01:17:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:39 smithi059 ceph-mon[97450]: pgmap v7053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:39 smithi059 ceph-mon[97450]: pgmap v7054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:17:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:39 smithi059 ceph-mon[97450]: pgmap v7055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:17:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:17:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:39.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:39.850268+0000) 2022-09-08T01:17:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:39.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:39.783948+0000) 2022-09-08T01:17:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:39 smithi055 ceph-mon[97033]: pgmap v7053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:39 smithi055 ceph-mon[97033]: pgmap v7054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:17:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:39 smithi055 ceph-mon[97033]: pgmap v7055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:17:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:17:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:40.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:40.850485+0000) 2022-09-08T01:17:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:40.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:40.784140+0000) 2022-09-08T01:17:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:41 smithi059 ceph-mon[97450]: pgmap v7056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:17:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:41.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:41.850738+0000) 2022-09-08T01:17:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:41.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:41.784324+0000) 2022-09-08T01:17:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:41 smithi055 ceph-mon[97033]: pgmap v7056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:17:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:42.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:42.850918+0000) 2022-09-08T01:17:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:42.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:42.784573+0000) 2022-09-08T01:17:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:43 smithi059 ceph-mon[97450]: pgmap v7057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:17:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:43.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:43.851170+0000) 2022-09-08T01:17:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:43.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:43.784834+0000) 2022-09-08T01:17:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:43 smithi055 ceph-mon[97033]: pgmap v7057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:17:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:44.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:44.851388+0000) 2022-09-08T01:17:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:44.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:44.785000+0000) 2022-09-08T01:17:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:45 smithi059 ceph-mon[97450]: pgmap v7058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:17:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:45.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:45.851577+0000) 2022-09-08T01:17:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:45.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:45.785244+0000) 2022-09-08T01:17:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:45 smithi055 ceph-mon[97033]: pgmap v7058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:17:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:46.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:46.851773+0000) 2022-09-08T01:17:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:46.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:46.785423+0000) 2022-09-08T01:17:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:47 smithi055 ceph-mon[97033]: pgmap v7059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:17:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:47 smithi059 ceph-mon[97450]: pgmap v7059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:17:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:47.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:47.851945+0000) 2022-09-08T01:17:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:47.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:47.785701+0000) 2022-09-08T01:17:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:48.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:48.852166+0000) 2022-09-08T01:17:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:48.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:48.785963+0000) 2022-09-08T01:17:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:49.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:49.852424+0000) 2022-09-08T01:17:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:49.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:49.786206+0000) 2022-09-08T01:17:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:49 smithi055 ceph-mon[97033]: pgmap v7060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:17:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:49 smithi059 ceph-mon[97450]: pgmap v7060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:17:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:50.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:50.852698+0000) 2022-09-08T01:17:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:50.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:50.786459+0000) 2022-09-08T01:17:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:51.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:51.852895+0000) 2022-09-08T01:17:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:51.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:51.786700+0000) 2022-09-08T01:17:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:51 smithi055 ceph-mon[97033]: pgmap v7061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:17:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:51 smithi059 ceph-mon[97450]: pgmap v7061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:17:53.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:17:52.925+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:17:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:52.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:52.853071+0000) 2022-09-08T01:17:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:52.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:52.786895+0000) 2022-09-08T01:17:54.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:53.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:53.853340+0000) 2022-09-08T01:17:54.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:53.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:53.787114+0000) 2022-09-08T01:17:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:53 smithi055 ceph-mon[97033]: pgmap v7062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:54.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:53 smithi059 ceph-mon[97450]: pgmap v7062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:54.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:54.853619+0000) 2022-09-08T01:17:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:54.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:54.787258+0000) 2022-09-08T01:17:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:55.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:55.853820+0000) 2022-09-08T01:17:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:55.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:55.787474+0000) 2022-09-08T01:17:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:55 smithi055 ceph-mon[97033]: pgmap v7063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:55 smithi059 ceph-mon[97450]: pgmap v7063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:17:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:56.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:56.787738+0000) 2022-09-08T01:17:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:56.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:56.853983+0000) 2022-09-08T01:17:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:57 smithi055 ceph-mon[97033]: pgmap v7064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:17:58.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:57 smithi059 ceph-mon[97450]: pgmap v7064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:17:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:57.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:57.788008+0000) 2022-09-08T01:17:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:57.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:57.854139+0000) 2022-09-08T01:17:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:58.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:58.788220+0000) 2022-09-08T01:17:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:58.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:58.854366+0000) 2022-09-08T01:18:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:17:59.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:59.788386+0000) 2022-09-08T01:18:00.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:17:59.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:17:59.854666+0000) 2022-09-08T01:18:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:17:59 smithi055 ceph-mon[97033]: pgmap v7065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:17:59 smithi059 ceph-mon[97450]: pgmap v7065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:00.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:00.854859+0000) 2022-09-08T01:18:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:00.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:00.788549+0000) 2022-09-08T01:18:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:01.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:01.855033+0000) 2022-09-08T01:18:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:01.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:01.788785+0000) 2022-09-08T01:18:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:01 smithi055 ceph-mon[97033]: pgmap v7066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:18:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:01 smithi059 ceph-mon[97450]: pgmap v7066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:18:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:02.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:02.855209+0000) 2022-09-08T01:18:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:02.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:02.789042+0000) 2022-09-08T01:18:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:03.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:03.855423+0000) 2022-09-08T01:18:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:03.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:03.789274+0000) 2022-09-08T01:18:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:03 smithi055 ceph-mon[97033]: pgmap v7067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:03 smithi059 ceph-mon[97450]: pgmap v7067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:04.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:04.855677+0000) 2022-09-08T01:18:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:04.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:04.789440+0000) 2022-09-08T01:18:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:05.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:05.855825+0000) 2022-09-08T01:18:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:05.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:05.789715+0000) 2022-09-08T01:18:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:05 smithi055 ceph-mon[97033]: pgmap v7068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:18:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:18:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:05 smithi059 ceph-mon[97450]: pgmap v7068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:18:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:18:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:06.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:06.856024+0000) 2022-09-08T01:18:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:06.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:06.789912+0000) 2022-09-08T01:18:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:07 smithi055 ceph-mon[97033]: pgmap v7069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:18:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:07 smithi059 ceph-mon[97450]: pgmap v7069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:18:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:07.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:07.790083+0000) 2022-09-08T01:18:08.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:18:07.940+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:18:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:07.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:07.856270+0000) 2022-09-08T01:18:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:08.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:08.790224+0000) 2022-09-08T01:18:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:08.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:08.856427+0000) 2022-09-08T01:18:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:09 smithi055 ceph-mon[97033]: osdmap e525: 12 total, 11 up, 11 in 2022-09-08T01:18:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:09 smithi055 ceph-mon[97033]: pgmap v7071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:18:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:09 smithi059 ceph-mon[97450]: osdmap e525: 12 total, 11 up, 11 in 2022-09-08T01:18:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:09 smithi059 ceph-mon[97450]: pgmap v7071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:18:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:09.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:09.856559+0000) 2022-09-08T01:18:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:09.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:09.790390+0000) 2022-09-08T01:18:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:10.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:10.856755+0000) 2022-09-08T01:18:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:10.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:10.790655+0000) 2022-09-08T01:18:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:11.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:11.856980+0000) 2022-09-08T01:18:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:11.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:11.790812+0000) 2022-09-08T01:18:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:11 smithi055 ceph-mon[97033]: pgmap v7072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:18:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:11 smithi055 ceph-mon[97033]: osdmap e526: 12 total, 11 up, 11 in 2022-09-08T01:18:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:11 smithi059 ceph-mon[97450]: pgmap v7072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:18:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:11 smithi059 ceph-mon[97450]: osdmap e526: 12 total, 11 up, 11 in 2022-09-08T01:18:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:12.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:12.857211+0000) 2022-09-08T01:18:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:12.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:12.791013+0000) 2022-09-08T01:18:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:13.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:13.857446+0000) 2022-09-08T01:18:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:13.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:13.791268+0000) 2022-09-08T01:18:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:13 smithi055 ceph-mon[97033]: pgmap v7074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:18:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:13 smithi059 ceph-mon[97450]: pgmap v7074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:18:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:14.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:14.857704+0000) 2022-09-08T01:18:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:14.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:14.791505+0000) 2022-09-08T01:18:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:15.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:15.857919+0000) 2022-09-08T01:18:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:15.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:15.791750+0000) 2022-09-08T01:18:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:15 smithi055 ceph-mon[97033]: pgmap v7075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:18:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:15 smithi059 ceph-mon[97450]: pgmap v7075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:18:17.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:16.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:16.791992+0000) 2022-09-08T01:18:17.161 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:16.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:16.858139+0000) 2022-09-08T01:18:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:17 smithi055 ceph-mon[97033]: pgmap v7076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T01:18:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:17 smithi059 ceph-mon[97450]: pgmap v7076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T01:18:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:17.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:17.858392+0000) 2022-09-08T01:18:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:17.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:17.792276+0000) 2022-09-08T01:18:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:18.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:18.858686+0000) 2022-09-08T01:18:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:18.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:18.792576+0000) 2022-09-08T01:18:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:19.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:19.858860+0000) 2022-09-08T01:18:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:19.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:19.792712+0000) 2022-09-08T01:18:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:19 smithi055 ceph-mon[97033]: pgmap v7077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:18:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:19 smithi059 ceph-mon[97450]: pgmap v7077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:18:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:20.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:20.859076+0000) 2022-09-08T01:18:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:20.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:20.792906+0000) 2022-09-08T01:18:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:21.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:21.859298+0000) 2022-09-08T01:18:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:21.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:21.793087+0000) 2022-09-08T01:18:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:21 smithi055 ceph-mon[97033]: pgmap v7078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:18:22.290 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:21 smithi059 ceph-mon[97450]: pgmap v7078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:18:23.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:18:22.956+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:18:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:22.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:22.859537+0000) 2022-09-08T01:18:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:22.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:22.793281+0000) 2022-09-08T01:18:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:23.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:23.859751+0000) 2022-09-08T01:18:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:23.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:23.793566+0000) 2022-09-08T01:18:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:23 smithi055 ceph-mon[97033]: pgmap v7079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:18:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:23 smithi059 ceph-mon[97450]: pgmap v7079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T01:18:25.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:24.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:24.859892+0000) 2022-09-08T01:18:25.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:24.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:24.793703+0000) 2022-09-08T01:18:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:25.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:25.860061+0000) 2022-09-08T01:18:26.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:25.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:25.793877+0000) 2022-09-08T01:18:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:25 smithi055 ceph-mon[97033]: pgmap v7080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:18:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:25 smithi059 ceph-mon[97450]: pgmap v7080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:18:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:26.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:26.860239+0000) 2022-09-08T01:18:27.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:26.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:26.794134+0000) 2022-09-08T01:18:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:27 smithi055 ceph-mon[97033]: pgmap v7081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:18:28.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:27 smithi059 ceph-mon[97450]: pgmap v7081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:18:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:27.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:27.860464+0000) 2022-09-08T01:18:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:27.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:27.794336+0000) 2022-09-08T01:18:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:28.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:28.860712+0000) 2022-09-08T01:18:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:28.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:28.794567+0000) 2022-09-08T01:18:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:29.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:29.860863+0000) 2022-09-08T01:18:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:29.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:29.794732+0000) 2022-09-08T01:18:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:29 smithi055 ceph-mon[97033]: pgmap v7082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:29 smithi059 ceph-mon[97450]: pgmap v7082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:30.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:30.861013+0000) 2022-09-08T01:18:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:30.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:30.794921+0000) 2022-09-08T01:18:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:31.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:31.861191+0000) 2022-09-08T01:18:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:31.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:31.795063+0000) 2022-09-08T01:18:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:31 smithi055 ceph-mon[97033]: pgmap v7083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:18:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:31 smithi059 ceph-mon[97450]: pgmap v7083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:18:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:32.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:32.861434+0000) 2022-09-08T01:18:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:32.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:32.795263+0000) 2022-09-08T01:18:34.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:33.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:33.861720+0000) 2022-09-08T01:18:34.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:33.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:33.795540+0000) 2022-09-08T01:18:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:33 smithi055 ceph-mon[97033]: pgmap v7084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:33 smithi059 ceph-mon[97450]: pgmap v7084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:34.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:34.862008+0000) 2022-09-08T01:18:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:34.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:34.795772+0000) 2022-09-08T01:18:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:35.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:35.862235+0000) 2022-09-08T01:18:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:35.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:35.795956+0000) 2022-09-08T01:18:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:35 smithi055 ceph-mon[97033]: pgmap v7085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:35 smithi059 ceph-mon[97450]: pgmap v7085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:36.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:36.862479+0000) 2022-09-08T01:18:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:36.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:36.796203+0000) 2022-09-08T01:18:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:37 smithi055 ceph-mon[97033]: pgmap v7086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:18:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:37 smithi059 ceph-mon[97450]: pgmap v7086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:18:38.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:18:37.963+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:18:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:37.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:37.862712+0000) 2022-09-08T01:18:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:37.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:37.796439+0000) 2022-09-08T01:18:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:38.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:38.862856+0000) 2022-09-08T01:18:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:38.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:38.796750+0000) 2022-09-08T01:18:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:39.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:39.863082+0000) 2022-09-08T01:18:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:39.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:39.796983+0000) 2022-09-08T01:18:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:39 smithi055 ceph-mon[97033]: pgmap v7087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:18:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:18:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:18:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:39 smithi059 ceph-mon[97450]: pgmap v7087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:18:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:18:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:18:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:40.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:40.863322+0000) 2022-09-08T01:18:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:40.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:40.797232+0000) 2022-09-08T01:18:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:41.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:41.863502+0000) 2022-09-08T01:18:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:41.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:41.797360+0000) 2022-09-08T01:18:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:41 smithi055 ceph-mon[97033]: pgmap v7088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:18:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:41 smithi059 ceph-mon[97450]: pgmap v7088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:18:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:42.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:42.863749+0000) 2022-09-08T01:18:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:42.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:42.797508+0000) 2022-09-08T01:18:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:43.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:43.797668+0000) 2022-09-08T01:18:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:43.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:43.863985+0000) 2022-09-08T01:18:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:43 smithi055 ceph-mon[97033]: pgmap v7089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:43 smithi059 ceph-mon[97450]: pgmap v7089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:44.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:44.864205+0000) 2022-09-08T01:18:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:44.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:44.797870+0000) 2022-09-08T01:18:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:45.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:45.864346+0000) 2022-09-08T01:18:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:45.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:45.798060+0000) 2022-09-08T01:18:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:45 smithi055 ceph-mon[97033]: pgmap v7090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:45 smithi059 ceph-mon[97450]: pgmap v7090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:46.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:46.864567+0000) 2022-09-08T01:18:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:46.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:46.798242+0000) 2022-09-08T01:18:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:47 smithi055 ceph-mon[97033]: pgmap v7091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:18:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:47 smithi059 ceph-mon[97450]: pgmap v7091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:18:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:47.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:47.864759+0000) 2022-09-08T01:18:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:47.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:47.798488+0000) 2022-09-08T01:18:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:48.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:48.865003+0000) 2022-09-08T01:18:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:48.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:48.798758+0000) 2022-09-08T01:18:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:49.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:49.865142+0000) 2022-09-08T01:18:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:49.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:49.799004+0000) 2022-09-08T01:18:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:49 smithi055 ceph-mon[97033]: pgmap v7092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:49 smithi059 ceph-mon[97450]: pgmap v7092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:50.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:50.865373+0000) 2022-09-08T01:18:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:50.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:50.799190+0000) 2022-09-08T01:18:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:51.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:51.865608+0000) 2022-09-08T01:18:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:51.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:51.799445+0000) 2022-09-08T01:18:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:51 smithi055 ceph-mon[97033]: pgmap v7093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:18:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:51 smithi059 ceph-mon[97450]: pgmap v7093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:18:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:52.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:52.865814+0000) 2022-09-08T01:18:53.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:18:52.968+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:18:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:52.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:52.799711+0000) 2022-09-08T01:18:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:53.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:53.865939+0000) 2022-09-08T01:18:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:53.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:53.799959+0000) 2022-09-08T01:18:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:53 smithi055 ceph-mon[97033]: pgmap v7094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:53 smithi059 ceph-mon[97450]: pgmap v7094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:54.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:54.866146+0000) 2022-09-08T01:18:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:54.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:54.800152+0000) 2022-09-08T01:18:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:55.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:55.866360+0000) 2022-09-08T01:18:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:55.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:55.800371+0000) 2022-09-08T01:18:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:55 smithi055 ceph-mon[97033]: pgmap v7095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:55 smithi059 ceph-mon[97450]: pgmap v7095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:18:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:56.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:56.866543+0000) 2022-09-08T01:18:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:56.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:56.800604+0000) 2022-09-08T01:18:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:57 smithi055 ceph-mon[97033]: pgmap v7096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:18:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:57 smithi059 ceph-mon[97450]: pgmap v7096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:18:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:57.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:57.866743+0000) 2022-09-08T01:18:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:57.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:57.800914+0000) 2022-09-08T01:18:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:58.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:58.866943+0000) 2022-09-08T01:18:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:58.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:58.801056+0000) 2022-09-08T01:19:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:18:59.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:59.867150+0000) 2022-09-08T01:19:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:18:59.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:18:59.801264+0000) 2022-09-08T01:19:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:18:59 smithi055 ceph-mon[97033]: pgmap v7097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:18:59 smithi059 ceph-mon[97450]: pgmap v7097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:00.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:00.867319+0000) 2022-09-08T01:19:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:00.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:00.801454+0000) 2022-09-08T01:19:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:01.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:01.867529+0000) 2022-09-08T01:19:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:01.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:01.801698+0000) 2022-09-08T01:19:02.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:01 smithi055 ceph-mon[97033]: pgmap v7098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:19:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:01 smithi059 ceph-mon[97450]: pgmap v7098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:19:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:02.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:02.867738+0000) 2022-09-08T01:19:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:02.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:02.801911+0000) 2022-09-08T01:19:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:03.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:03.867911+0000) 2022-09-08T01:19:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:03.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:03.802137+0000) 2022-09-08T01:19:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:03 smithi055 ceph-mon[97033]: pgmap v7099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:03 smithi059 ceph-mon[97450]: pgmap v7099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:04.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:04.802281+0000) 2022-09-08T01:19:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:04.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:04.868128+0000) 2022-09-08T01:19:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:05.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:05.868358+0000) 2022-09-08T01:19:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:05.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:05.802500+0000) 2022-09-08T01:19:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:05 smithi055 ceph-mon[97033]: pgmap v7100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:19:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:19:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:05 smithi059 ceph-mon[97450]: pgmap v7100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:19:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:19:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:06.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:06.868620+0000) 2022-09-08T01:19:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:06.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:06.802730+0000) 2022-09-08T01:19:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:07 smithi055 ceph-mon[97033]: pgmap v7101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:19:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:07 smithi059 ceph-mon[97450]: pgmap v7101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:19:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:07.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:07.868794+0000) 2022-09-08T01:19:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:07.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:07.802981+0000) 2022-09-08T01:19:08.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:19:07.982+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:19:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:08.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:08.868933+0000) 2022-09-08T01:19:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:08.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:08.803156+0000) 2022-09-08T01:19:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:09 smithi055 ceph-mon[97033]: osdmap e527: 12 total, 11 up, 11 in 2022-09-08T01:19:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:09 smithi055 ceph-mon[97033]: pgmap v7103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:19:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:09 smithi059 ceph-mon[97450]: osdmap e527: 12 total, 11 up, 11 in 2022-09-08T01:19:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:09 smithi059 ceph-mon[97450]: pgmap v7103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:19:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:09.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:09.869099+0000) 2022-09-08T01:19:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:09.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:09.803293+0000) 2022-09-08T01:19:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:10.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:10.869283+0000) 2022-09-08T01:19:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:10.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:10.803512+0000) 2022-09-08T01:19:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:11.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:11.869534+0000) 2022-09-08T01:19:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:11.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:11.803734+0000) 2022-09-08T01:19:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:11 smithi055 ceph-mon[97033]: pgmap v7104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:19:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:11 smithi055 ceph-mon[97033]: osdmap e528: 12 total, 11 up, 11 in 2022-09-08T01:19:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:11 smithi059 ceph-mon[97450]: pgmap v7104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:19:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:11 smithi059 ceph-mon[97450]: osdmap e528: 12 total, 11 up, 11 in 2022-09-08T01:19:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:12.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:12.869747+0000) 2022-09-08T01:19:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:12.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:12.803973+0000) 2022-09-08T01:19:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:13 smithi059 ceph-mon[97450]: pgmap v7106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:19:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:19:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:19:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:19:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:13.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:13.869951+0000) 2022-09-08T01:19:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:13.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:13.804232+0000) 2022-09-08T01:19:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:13 smithi055 ceph-mon[97033]: pgmap v7106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:19:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:19:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:19:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:19:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:14.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:14.870118+0000) 2022-09-08T01:19:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:14.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:14.804471+0000) 2022-09-08T01:19:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:14 smithi055 ceph-mon[97033]: pgmap v7107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2022-09-08T01:19:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:14 smithi055 ceph-mon[97033]: pgmap v7108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T01:19:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:14 smithi059 ceph-mon[97450]: pgmap v7107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2022-09-08T01:19:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:14 smithi059 ceph-mon[97450]: pgmap v7108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T01:19:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:15.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:15.870375+0000) 2022-09-08T01:19:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:15.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:15.804678+0000) 2022-09-08T01:19:17.162 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:16.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:16.870641+0000) 2022-09-08T01:19:17.163 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:16.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:16.804915+0000) 2022-09-08T01:19:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:16 smithi055 ceph-mon[97033]: pgmap v7109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.7 KiB/s rd, 303 B/s wr, 9 op/s 2022-09-08T01:19:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:16 smithi059 ceph-mon[97450]: pgmap v7109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.7 KiB/s rd, 303 B/s wr, 9 op/s 2022-09-08T01:19:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:17.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:17.870804+0000) 2022-09-08T01:19:18.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:17.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:17.805108+0000) 2022-09-08T01:19:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:18.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:18.871034+0000) 2022-09-08T01:19:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:18.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:18.805343+0000) 2022-09-08T01:19:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:18 smithi055 ceph-mon[97033]: pgmap v7110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2022-09-08T01:19:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:18 smithi059 ceph-mon[97450]: pgmap v7110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2022-09-08T01:19:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:19.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:19.871287+0000) 2022-09-08T01:19:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:19.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:19.805486+0000) 2022-09-08T01:19:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:20.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:20.871453+0000) 2022-09-08T01:19:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:20.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:20.805745+0000) 2022-09-08T01:19:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:20 smithi055 ceph-mon[97033]: pgmap v7111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 128 B/s wr, 4 op/s 2022-09-08T01:19:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:20 smithi059 ceph-mon[97450]: pgmap v7111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 128 B/s wr, 4 op/s 2022-09-08T01:19:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:21.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:21.871714+0000) 2022-09-08T01:19:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:21.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:21.805994+0000) 2022-09-08T01:19:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:22.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:22.871931+0000) 2022-09-08T01:19:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:22.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:22.806206+0000) 2022-09-08T01:19:23.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:19:22.996+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:19:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:22 smithi055 ceph-mon[97033]: pgmap v7112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 116 B/s wr, 7 op/s 2022-09-08T01:19:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:22 smithi059 ceph-mon[97450]: pgmap v7112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 116 B/s wr, 7 op/s 2022-09-08T01:19:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:23.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:23.872165+0000) 2022-09-08T01:19:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:23.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:23.806466+0000) 2022-09-08T01:19:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:24.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:24.872407+0000) 2022-09-08T01:19:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:24.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:24.806753+0000) 2022-09-08T01:19:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:24 smithi055 ceph-mon[97033]: pgmap v7113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:19:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:24 smithi059 ceph-mon[97450]: pgmap v7113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:19:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:25.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:25.872579+0000) 2022-09-08T01:19:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:25.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:25.806957+0000) 2022-09-08T01:19:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:26.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:26.872749+0000) 2022-09-08T01:19:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:26.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:26.807145+0000) 2022-09-08T01:19:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:26 smithi055 ceph-mon[97033]: pgmap v7114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:19:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:26 smithi059 ceph-mon[97450]: pgmap v7114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:19:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:27.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:27.872878+0000) 2022-09-08T01:19:28.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:27.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:27.807375+0000) 2022-09-08T01:19:29.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:28.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:28.873004+0000) 2022-09-08T01:19:29.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:28.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:28.807589+0000) 2022-09-08T01:19:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:28 smithi055 ceph-mon[97033]: pgmap v7115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:28 smithi059 ceph-mon[97450]: pgmap v7115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:29.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:29.873138+0000) 2022-09-08T01:19:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:29.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:29.807743+0000) 2022-09-08T01:19:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:30.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:30.873297+0000) 2022-09-08T01:19:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:30.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:30.807970+0000) 2022-09-08T01:19:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:30 smithi055 ceph-mon[97033]: pgmap v7116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:30 smithi059 ceph-mon[97450]: pgmap v7116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:31.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:31.873448+0000) 2022-09-08T01:19:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:31.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:31.808134+0000) 2022-09-08T01:19:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:32.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:32.873582+0000) 2022-09-08T01:19:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:32.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:32.808355+0000) 2022-09-08T01:19:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:32 smithi055 ceph-mon[97033]: pgmap v7117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:19:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:32 smithi059 ceph-mon[97450]: pgmap v7117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:19:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:33.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:33.873816+0000) 2022-09-08T01:19:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:33.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:33.808619+0000) 2022-09-08T01:19:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:34.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:34.874060+0000) 2022-09-08T01:19:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:34.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:34.808790+0000) 2022-09-08T01:19:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:34 smithi055 ceph-mon[97033]: pgmap v7118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:34 smithi059 ceph-mon[97450]: pgmap v7118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:35.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:35.874278+0000) 2022-09-08T01:19:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:35.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:35.809023+0000) 2022-09-08T01:19:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:36.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:36.874528+0000) 2022-09-08T01:19:37.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:36.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:36.809262+0000) 2022-09-08T01:19:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:36 smithi055 ceph-mon[97033]: pgmap v7119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:19:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:36 smithi059 ceph-mon[97450]: pgmap v7119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:19:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:37.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:37.874777+0000) 2022-09-08T01:19:38.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:19:38.011+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:19:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:37.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:37.809477+0000) 2022-09-08T01:19:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:38.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:38.874990+0000) 2022-09-08T01:19:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:38.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:38.809747+0000) 2022-09-08T01:19:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:38 smithi055 ceph-mon[97033]: pgmap v7120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:38 smithi059 ceph-mon[97450]: pgmap v7120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:39.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:39.875224+0000) 2022-09-08T01:19:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:39.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:39.809894+0000) 2022-09-08T01:19:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:40.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:40.810037+0000) 2022-09-08T01:19:41.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:40.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:40.875374+0000) 2022-09-08T01:19:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:40 smithi055 ceph-mon[97033]: pgmap v7121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:40 smithi059 ceph-mon[97450]: pgmap v7121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:19:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:41.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:41.875545+0000) 2022-09-08T01:19:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:41.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:41.810267+0000) 2022-09-08T01:19:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:42.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:42.875797+0000) 2022-09-08T01:19:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:42.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:42.810519+0000) 2022-09-08T01:19:43.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:42 smithi055 ceph-mon[97033]: pgmap v7122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:19:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:42 smithi059 ceph-mon[97450]: pgmap v7122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:19:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:43.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:43.875944+0000) 2022-09-08T01:19:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:43.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:43.810771+0000) 2022-09-08T01:19:45.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:44.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:44.876071+0000) 2022-09-08T01:19:45.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:44.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:44.810911+0000) 2022-09-08T01:19:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:44 smithi055 ceph-mon[97033]: pgmap v7123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:19:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:44 smithi059 ceph-mon[97450]: pgmap v7123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:19:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:45.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:45.876234+0000) 2022-09-08T01:19:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:45.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:45.811054+0000) 2022-09-08T01:19:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:46.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:46.876386+0000) 2022-09-08T01:19:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:46.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:46.811305+0000) 2022-09-08T01:19:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:46 smithi055 ceph-mon[97033]: pgmap v7124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:19:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:46 smithi059 ceph-mon[97450]: pgmap v7124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:19:48.161 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:47.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:47.876588+0000) 2022-09-08T01:19:48.162 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:47.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:47.811482+0000) 2022-09-08T01:19:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:48.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:48.876828+0000) 2022-09-08T01:19:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:48.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:48.811733+0000) 2022-09-08T01:19:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:48 smithi055 ceph-mon[97033]: pgmap v7125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:19:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:48 smithi059 ceph-mon[97450]: pgmap v7125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:19:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:49.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:49.877084+0000) 2022-09-08T01:19:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:49.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:49.811901+0000) 2022-09-08T01:19:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:50.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:50.812091+0000) 2022-09-08T01:19:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:50.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:50.877363+0000) 2022-09-08T01:19:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:50 smithi055 ceph-mon[97033]: pgmap v7126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:19:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:50 smithi059 ceph-mon[97450]: pgmap v7126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:19:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:51.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:51.877606+0000) 2022-09-08T01:19:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:51.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:51.812299+0000) 2022-09-08T01:19:53.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:19:53.018+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:19:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:52.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:52.812580+0000) 2022-09-08T01:19:53.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:52.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:52.877813+0000) 2022-09-08T01:19:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:52 smithi055 ceph-mon[97033]: pgmap v7127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:19:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:52 smithi059 ceph-mon[97450]: pgmap v7127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:19:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:53.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:53.878105+0000) 2022-09-08T01:19:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:53.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:53.812869+0000) 2022-09-08T01:19:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:54.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:54.878368+0000) 2022-09-08T01:19:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:54.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:54.813041+0000) 2022-09-08T01:19:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:54 smithi055 ceph-mon[97033]: pgmap v7128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:19:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:54 smithi059 ceph-mon[97450]: pgmap v7128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:19:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:55.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:55.878549+0000) 2022-09-08T01:19:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:55.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:55.813221+0000) 2022-09-08T01:19:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:56.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:56.878780+0000) 2022-09-08T01:19:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:56.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:56.813432+0000) 2022-09-08T01:19:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:56 smithi055 ceph-mon[97033]: pgmap v7129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:19:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:56 smithi059 ceph-mon[97450]: pgmap v7129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:19:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:57.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:57.879050+0000) 2022-09-08T01:19:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:57.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:57.813603+0000) 2022-09-08T01:19:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:58.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:58.879272+0000) 2022-09-08T01:19:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:58.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:58.813847+0000) 2022-09-08T01:19:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:19:58 smithi055 ceph-mon[97033]: pgmap v7130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:19:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:19:58 smithi059 ceph-mon[97450]: pgmap v7130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:20:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:19:59.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:59.879481+0000) 2022-09-08T01:20:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:19:59.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:19:59.814049+0000) 2022-09-08T01:20:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:00.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:00.879708+0000) 2022-09-08T01:20:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:00.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:00.814290+0000) 2022-09-08T01:20:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:00 smithi055 ceph-mon[97033]: pgmap v7131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:20:01.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T01:20:01.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T01:20:01.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: ['Unable to reach remote host smithi047. '] 2022-09-08T01:20:01.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T01:20:01.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T01:20:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:00 smithi059 ceph-mon[97450]: pgmap v7131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:20:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T01:20:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T01:20:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: ['Unable to reach remote host smithi047. '] 2022-09-08T01:20:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T01:20:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T01:20:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:01.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:01.879955+0000) 2022-09-08T01:20:02.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:01.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:01.814534+0000) 2022-09-08T01:20:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:02.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:02.880182+0000) 2022-09-08T01:20:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:02.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:02.814795+0000) 2022-09-08T01:20:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:02 smithi055 ceph-mon[97033]: pgmap v7132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:20:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:02 smithi059 ceph-mon[97450]: pgmap v7132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:20:04.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:03.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:03.815091+0000) 2022-09-08T01:20:04.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:03.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:03.880422+0000) 2022-09-08T01:20:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:04.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:04.880563+0000) 2022-09-08T01:20:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:04.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:04.815355+0000) 2022-09-08T01:20:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:04 smithi055 ceph-mon[97033]: pgmap v7133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:04 smithi059 ceph-mon[97450]: pgmap v7133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:05.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:05.815687+0000) 2022-09-08T01:20:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:05.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:05.880816+0000) 2022-09-08T01:20:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:20:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:20:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:20:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:20:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:06.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:06.815983+0000) 2022-09-08T01:20:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:06.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:06.880980+0000) 2022-09-08T01:20:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:06 smithi055 ceph-mon[97033]: pgmap v7134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:20:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:06 smithi059 ceph-mon[97450]: pgmap v7134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:20:08.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:20:08.034+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:20:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:07.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:07.881176+0000) 2022-09-08T01:20:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:07.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:07.816210+0000) 2022-09-08T01:20:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:08.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:08.816478+0000) 2022-09-08T01:20:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:08.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:08.881466+0000) 2022-09-08T01:20:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:08 smithi055 ceph-mon[97033]: pgmap v7135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:20:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:08 smithi059 ceph-mon[97450]: pgmap v7135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:20:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:09.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:09.881711+0000) 2022-09-08T01:20:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:09.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:09.816657+0000) 2022-09-08T01:20:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:09 smithi055 ceph-mon[97033]: osdmap e529: 12 total, 11 up, 11 in 2022-09-08T01:20:10.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:09 smithi059 ceph-mon[97450]: osdmap e529: 12 total, 11 up, 11 in 2022-09-08T01:20:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:10.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:10.881953+0000) 2022-09-08T01:20:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:10.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:10.816820+0000) 2022-09-08T01:20:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:10 smithi055 ceph-mon[97033]: pgmap v7137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:20:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:10 smithi059 ceph-mon[97450]: pgmap v7137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:20:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:11.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:11.882100+0000) 2022-09-08T01:20:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:11.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:11.817046+0000) 2022-09-08T01:20:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:12.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:12.817276+0000) 2022-09-08T01:20:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:12.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:12.882330+0000) 2022-09-08T01:20:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:12 smithi055 ceph-mon[97033]: pgmap v7138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:20:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:12 smithi059 ceph-mon[97450]: pgmap v7138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:20:13.947 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:20:13.947 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:20:13.948 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:20:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:13.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:13.817474+0000) 2022-09-08T01:20:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:13.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:13.882549+0000) 2022-09-08T01:20:14.248 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:20:14.248 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:20:14.249 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:20:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:14.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:14.882731+0000) 2022-09-08T01:20:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:14.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:14.817745+0000) 2022-09-08T01:20:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:14 smithi055 ceph-mon[97033]: pgmap v7139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:20:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:20:15.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:14 smithi059 ceph-mon[97450]: pgmap v7139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:20:15.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:20:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:15.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:15.882908+0000) 2022-09-08T01:20:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:15.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:15.817934+0000) 2022-09-08T01:20:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:15 smithi055 ceph-mon[97033]: pgmap v7140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 118 B/s wr, 3 op/s 2022-09-08T01:20:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:15 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T01:20:16.292 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:15 smithi059 ceph-mon[97450]: pgmap v7140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.9 KiB/s rd, 118 B/s wr, 3 op/s 2022-09-08T01:20:16.293 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:15 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T01:20:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:16.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:16.883099+0000) 2022-09-08T01:20:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:16.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:16.818145+0000) 2022-09-08T01:20:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:17 smithi055 ceph-mon[97033]: pgmap v7141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2022-09-08T01:20:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:17 smithi055 ceph-mon[97033]: osdmap e530: 12 total, 11 up, 11 in 2022-09-08T01:20:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:17 smithi059 ceph-mon[97450]: pgmap v7141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2022-09-08T01:20:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:17 smithi059 ceph-mon[97450]: osdmap e530: 12 total, 11 up, 11 in 2022-09-08T01:20:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:17.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:17.883346+0000) 2022-09-08T01:20:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:17.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:17.818351+0000) 2022-09-08T01:20:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:18.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:18.883552+0000) 2022-09-08T01:20:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:18.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:18.818540+0000) 2022-09-08T01:20:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:19 smithi055 ceph-mon[97033]: pgmap v7143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2022-09-08T01:20:20.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:19 smithi059 ceph-mon[97450]: pgmap v7143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2022-09-08T01:20:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:19.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:19.883785+0000) 2022-09-08T01:20:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:19.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:19.818762+0000) 2022-09-08T01:20:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:20.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:20.884045+0000) 2022-09-08T01:20:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:20.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:20.818941+0000) 2022-09-08T01:20:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:21 smithi055 ceph-mon[97033]: pgmap v7144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T01:20:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:21 smithi059 ceph-mon[97450]: pgmap v7144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T01:20:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:21.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:21.884217+0000) 2022-09-08T01:20:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:21.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:21.819204+0000) 2022-09-08T01:20:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:22.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:22.884429+0000) 2022-09-08T01:20:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:22.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:22.819418+0000) 2022-09-08T01:20:23.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:20:23.045+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:20:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:23 smithi055 ceph-mon[97033]: pgmap v7145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T01:20:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:23 smithi059 ceph-mon[97450]: pgmap v7145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T01:20:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:23.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:23.884572+0000) 2022-09-08T01:20:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:23.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:23.819652+0000) 2022-09-08T01:20:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:24.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:24.819820+0000) 2022-09-08T01:20:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:24.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:24.884809+0000) 2022-09-08T01:20:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:25 smithi055 ceph-mon[97033]: pgmap v7146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:20:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:25 smithi059 ceph-mon[97450]: pgmap v7146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:20:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:25.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:25.819984+0000) 2022-09-08T01:20:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:25.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:25.885021+0000) 2022-09-08T01:20:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:26.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:26.820254+0000) 2022-09-08T01:20:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:26.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:26.885182+0000) 2022-09-08T01:20:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:27 smithi055 ceph-mon[97033]: pgmap v7147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:20:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:27 smithi059 ceph-mon[97450]: pgmap v7147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:20:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:27.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:27.885392+0000) 2022-09-08T01:20:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:27.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:27.820523+0000) 2022-09-08T01:20:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:28.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:28.885569+0000) 2022-09-08T01:20:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:28.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:28.820808+0000) 2022-09-08T01:20:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:29 smithi055 ceph-mon[97033]: pgmap v7148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T01:20:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:29 smithi059 ceph-mon[97450]: pgmap v7148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T01:20:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:29.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:29.885805+0000) 2022-09-08T01:20:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:29.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:29.821002+0000) 2022-09-08T01:20:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:30.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:30.886056+0000) 2022-09-08T01:20:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:30.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:30.821193+0000) 2022-09-08T01:20:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:31 smithi055 ceph-mon[97033]: pgmap v7149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:20:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:31 smithi059 ceph-mon[97450]: pgmap v7149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:20:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:31.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:31.821398+0000) 2022-09-08T01:20:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:31.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:31.886264+0000) 2022-09-08T01:20:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:32.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:32.886484+0000) 2022-09-08T01:20:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:32.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:32.821610+0000) 2022-09-08T01:20:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:33 smithi055 ceph-mon[97033]: pgmap v7150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:33 smithi059 ceph-mon[97450]: pgmap v7150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:33.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:33.886719+0000) 2022-09-08T01:20:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:33.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:33.821831+0000) 2022-09-08T01:20:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:34.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:34.821979+0000) 2022-09-08T01:20:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:34.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:34.886886+0000) 2022-09-08T01:20:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:35 smithi055 ceph-mon[97033]: pgmap v7151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:35 smithi059 ceph-mon[97450]: pgmap v7151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:35.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:35.887113+0000) 2022-09-08T01:20:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:35.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:35.822151+0000) 2022-09-08T01:20:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:36.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:36.822422+0000) 2022-09-08T01:20:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:36.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:36.887283+0000) 2022-09-08T01:20:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:37 smithi055 ceph-mon[97033]: pgmap v7152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:20:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:37 smithi059 ceph-mon[97450]: pgmap v7152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:20:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:37.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:37.887525+0000) 2022-09-08T01:20:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:37.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:37.822681+0000) 2022-09-08T01:20:38.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:20:38.055+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:20:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:38.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:38.887791+0000) 2022-09-08T01:20:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:38.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:38.822955+0000) 2022-09-08T01:20:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:39 smithi055 ceph-mon[97033]: pgmap v7153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:39 smithi059 ceph-mon[97450]: pgmap v7153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:40.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:39.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:39.887948+0000) 2022-09-08T01:20:40.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:39.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:39.823215+0000) 2022-09-08T01:20:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:40.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:40.888098+0000) 2022-09-08T01:20:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:40.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:40.823412+0000) 2022-09-08T01:20:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:41 smithi055 ceph-mon[97033]: pgmap v7154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:20:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:41 smithi059 ceph-mon[97450]: pgmap v7154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:20:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:41.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:41.888288+0000) 2022-09-08T01:20:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:41.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:41.823718+0000) 2022-09-08T01:20:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:42.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:42.888511+0000) 2022-09-08T01:20:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:42.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:42.823958+0000) 2022-09-08T01:20:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:43 smithi055 ceph-mon[97033]: pgmap v7155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:43 smithi059 ceph-mon[97450]: pgmap v7155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:43.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:43.888758+0000) 2022-09-08T01:20:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:43.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:43.824193+0000) 2022-09-08T01:20:45.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:44.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:44.889007+0000) 2022-09-08T01:20:45.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:44.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:44.824472+0000) 2022-09-08T01:20:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:45 smithi055 ceph-mon[97033]: pgmap v7156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:45 smithi059 ceph-mon[97450]: pgmap v7156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:45.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:45.889246+0000) 2022-09-08T01:20:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:45.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:45.824715+0000) 2022-09-08T01:20:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:46.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:46.889404+0000) 2022-09-08T01:20:47.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:46.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:46.824869+0000) 2022-09-08T01:20:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:47 smithi055 ceph-mon[97033]: pgmap v7157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:20:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:47 smithi059 ceph-mon[97450]: pgmap v7157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:20:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:47.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:47.889621+0000) 2022-09-08T01:20:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:47.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:47.825079+0000) 2022-09-08T01:20:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:48.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:48.889822+0000) 2022-09-08T01:20:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:48.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:48.825331+0000) 2022-09-08T01:20:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:49 smithi055 ceph-mon[97033]: pgmap v7158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:49 smithi059 ceph-mon[97450]: pgmap v7158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:49.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:49.890058+0000) 2022-09-08T01:20:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:49.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:49.825579+0000) 2022-09-08T01:20:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:50.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:50.890312+0000) 2022-09-08T01:20:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:50.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:50.825813+0000) 2022-09-08T01:20:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:51 smithi055 ceph-mon[97033]: pgmap v7159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:20:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:51 smithi059 ceph-mon[97450]: pgmap v7159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:20:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:51.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:51.890524+0000) 2022-09-08T01:20:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:51.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:51.826045+0000) 2022-09-08T01:20:53.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:20:53.067+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:20:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:52.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:52.890789+0000) 2022-09-08T01:20:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:52.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:52.826269+0000) 2022-09-08T01:20:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:53 smithi055 ceph-mon[97033]: pgmap v7160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:53 smithi059 ceph-mon[97450]: pgmap v7160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:53.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:53.891051+0000) 2022-09-08T01:20:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:53.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:53.826492+0000) 2022-09-08T01:20:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:54.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:54.826710+0000) 2022-09-08T01:20:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:54.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:54.891281+0000) 2022-09-08T01:20:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:55 smithi055 ceph-mon[97033]: pgmap v7161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:55 smithi059 ceph-mon[97450]: pgmap v7161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:20:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:55.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:55.891517+0000) 2022-09-08T01:20:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:55.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:55.826857+0000) 2022-09-08T01:20:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:56.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:56.891741+0000) 2022-09-08T01:20:57.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:56.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:56.827128+0000) 2022-09-08T01:20:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:57 smithi055 ceph-mon[97033]: pgmap v7162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:20:58.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:57 smithi059 ceph-mon[97450]: pgmap v7162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:20:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:57.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:57.891975+0000) 2022-09-08T01:20:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:57.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:57.827428+0000) 2022-09-08T01:20:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:58.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:58.892216+0000) 2022-09-08T01:20:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:58.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:58.827692+0000) 2022-09-08T01:20:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:20:59 smithi055 ceph-mon[97033]: pgmap v7163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:20:59 smithi059 ceph-mon[97450]: pgmap v7163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:20:59.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:59.892440+0000) 2022-09-08T01:21:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:20:59.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:20:59.827955+0000) 2022-09-08T01:21:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:00.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:00.892625+0000) 2022-09-08T01:21:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:00.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:00.828125+0000) 2022-09-08T01:21:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:01 smithi055 ceph-mon[97033]: pgmap v7164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:21:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:01 smithi059 ceph-mon[97450]: pgmap v7164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:21:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:01.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:01.892890+0000) 2022-09-08T01:21:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:01.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:01.828361+0000) 2022-09-08T01:21:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:02.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:02.893121+0000) 2022-09-08T01:21:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:02.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:02.828667+0000) 2022-09-08T01:21:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:03 smithi055 ceph-mon[97033]: pgmap v7165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:03 smithi059 ceph-mon[97450]: pgmap v7165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:03.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:03.828924+0000) 2022-09-08T01:21:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:03.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:03.893375+0000) 2022-09-08T01:21:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:04.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:04.893623+0000) 2022-09-08T01:21:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:04.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:04.829191+0000) 2022-09-08T01:21:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:05 smithi055 ceph-mon[97033]: pgmap v7166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:21:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:21:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:05 smithi059 ceph-mon[97450]: pgmap v7166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:21:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:21:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:05.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:05.893907+0000) 2022-09-08T01:21:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:05.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:05.829429+0000) 2022-09-08T01:21:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:06.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:06.829609+0000) 2022-09-08T01:21:07.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:06.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:06.894182+0000) 2022-09-08T01:21:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:07 smithi055 ceph-mon[97033]: pgmap v7167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:21:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:07 smithi059 ceph-mon[97450]: pgmap v7167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:21:08.085 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:07.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:07.894431+0000) 2022-09-08T01:21:08.086 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:07.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:07.829754+0000) 2022-09-08T01:21:08.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:21:08.082+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:21:09.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:08.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:08.894667+0000) 2022-09-08T01:21:09.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:08.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:08.829968+0000) 2022-09-08T01:21:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:09 smithi055 ceph-mon[97033]: pgmap v7168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:21:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:09 smithi055 ceph-mon[97033]: osdmap e531: 12 total, 11 up, 11 in 2022-09-08T01:21:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:09 smithi059 ceph-mon[97450]: pgmap v7168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:21:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:09 smithi059 ceph-mon[97450]: osdmap e531: 12 total, 11 up, 11 in 2022-09-08T01:21:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:09.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:09.894843+0000) 2022-09-08T01:21:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:09.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:09.830168+0000) 2022-09-08T01:21:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:10.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:10.895020+0000) 2022-09-08T01:21:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:10.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:10.830460+0000) 2022-09-08T01:21:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:11 smithi055 ceph-mon[97033]: pgmap v7170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:21:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:11 smithi055 ceph-mon[97033]: osdmap e532: 12 total, 11 up, 11 in 2022-09-08T01:21:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:11 smithi059 ceph-mon[97450]: pgmap v7170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:21:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:11 smithi059 ceph-mon[97450]: osdmap e532: 12 total, 11 up, 11 in 2022-09-08T01:21:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:11.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:11.895199+0000) 2022-09-08T01:21:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:11.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:11.830703+0000) 2022-09-08T01:21:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:12.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:12.895426+0000) 2022-09-08T01:21:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:12.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:12.830950+0000) 2022-09-08T01:21:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:13 smithi055 ceph-mon[97033]: pgmap v7172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:21:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:13 smithi059 ceph-mon[97450]: pgmap v7172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:21:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:13.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:13.895654+0000) 2022-09-08T01:21:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:13.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:13.831235+0000) 2022-09-08T01:21:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:21:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:21:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:21:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:21:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:21:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:21:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:14.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:14.895794+0000) 2022-09-08T01:21:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:14.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:14.831492+0000) 2022-09-08T01:21:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:15.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:15.896015+0000) 2022-09-08T01:21:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:15.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:15.831757+0000) 2022-09-08T01:21:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:15 smithi055 ceph-mon[97033]: pgmap v7173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:21:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:21:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:15 smithi059 ceph-mon[97450]: pgmap v7173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:21:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:21:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:16.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:16.896169+0000) 2022-09-08T01:21:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:16.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:16.832038+0000) 2022-09-08T01:21:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:17.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:17.896403+0000) 2022-09-08T01:21:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:17.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:17.832289+0000) 2022-09-08T01:21:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:17 smithi055 ceph-mon[97033]: pgmap v7174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:21:18.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:17 smithi059 ceph-mon[97450]: pgmap v7174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:21:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:18.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:18.832540+0000) 2022-09-08T01:21:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:18.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:18.896678+0000) 2022-09-08T01:21:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:19.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:19.896836+0000) 2022-09-08T01:21:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:19.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:19.832792+0000) 2022-09-08T01:21:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:19 smithi055 ceph-mon[97033]: pgmap v7175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2022-09-08T01:21:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:19 smithi059 ceph-mon[97450]: pgmap v7175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2022-09-08T01:21:21.166 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:20.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:20.897062+0000) 2022-09-08T01:21:21.167 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:20.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:20.833028+0000) 2022-09-08T01:21:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:21.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:21.897276+0000) 2022-09-08T01:21:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:21.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:21.833233+0000) 2022-09-08T01:21:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:21 smithi055 ceph-mon[97033]: pgmap v7176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:21:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:21 smithi059 ceph-mon[97450]: pgmap v7176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:21:23.101 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:22.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:22.897523+0000) 2022-09-08T01:21:23.102 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:22.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:22.833443+0000) 2022-09-08T01:21:23.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:21:23.097+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:21:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:23.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:23.897748+0000) 2022-09-08T01:21:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:23.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:23.833688+0000) 2022-09-08T01:21:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:23 smithi055 ceph-mon[97033]: pgmap v7177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:21:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:23 smithi059 ceph-mon[97450]: pgmap v7177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:21:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:24.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:24.897977+0000) 2022-09-08T01:21:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:24.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:24.834035+0000) 2022-09-08T01:21:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:25.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:25.898147+0000) 2022-09-08T01:21:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:25.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:25.834182+0000) 2022-09-08T01:21:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:25 smithi055 ceph-mon[97033]: pgmap v7178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:21:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:25 smithi059 ceph-mon[97450]: pgmap v7178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:21:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:26.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:26.898356+0000) 2022-09-08T01:21:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:26.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:26.834400+0000) 2022-09-08T01:21:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:27.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:27.898595+0000) 2022-09-08T01:21:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:27.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:27.834686+0000) 2022-09-08T01:21:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:27 smithi055 ceph-mon[97033]: pgmap v7179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:21:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:27 smithi059 ceph-mon[97450]: pgmap v7179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:21:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:28.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:28.898861+0000) 2022-09-08T01:21:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:28.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:28.834875+0000) 2022-09-08T01:21:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:29.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:29.899084+0000) 2022-09-08T01:21:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:29.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:29.835081+0000) 2022-09-08T01:21:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:29 smithi055 ceph-mon[97033]: pgmap v7180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:29 smithi059 ceph-mon[97450]: pgmap v7180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:30.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:30.899273+0000) 2022-09-08T01:21:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:30.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:30.835298+0000) 2022-09-08T01:21:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:31.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:31.835524+0000) 2022-09-08T01:21:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:31.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:31.899389+0000) 2022-09-08T01:21:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:31 smithi055 ceph-mon[97033]: pgmap v7181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:21:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:31 smithi059 ceph-mon[97450]: pgmap v7181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:21:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:32.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:32.835668+0000) 2022-09-08T01:21:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:32.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:32.899622+0000) 2022-09-08T01:21:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:33.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:33.899832+0000) 2022-09-08T01:21:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:33.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:33.835883+0000) 2022-09-08T01:21:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:33 smithi055 ceph-mon[97033]: pgmap v7182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:33 smithi059 ceph-mon[97450]: pgmap v7182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:34.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:34.900087+0000) 2022-09-08T01:21:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:34.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:34.836126+0000) 2022-09-08T01:21:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:35.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:35.900301+0000) 2022-09-08T01:21:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:35.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:35.836409+0000) 2022-09-08T01:21:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:35 smithi055 ceph-mon[97033]: pgmap v7183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:35 smithi059 ceph-mon[97450]: pgmap v7183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:36.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:36.836730+0000) 2022-09-08T01:21:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:36.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:36.900469+0000) 2022-09-08T01:21:38.117 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:37.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:37.900689+0000) 2022-09-08T01:21:38.119 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:37.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:37.837003+0000) 2022-09-08T01:21:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:37 smithi055 ceph-mon[97033]: pgmap v7184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:21:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:37 smithi059 ceph-mon[97450]: pgmap v7184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:21:38.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:21:38.113+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:21:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:38.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:38.900896+0000) 2022-09-08T01:21:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:38.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:38.837257+0000) 2022-09-08T01:21:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:39.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:39.901161+0000) 2022-09-08T01:21:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:39.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:39.837549+0000) 2022-09-08T01:21:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:39 smithi055 ceph-mon[97033]: pgmap v7185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:40.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:39 smithi059 ceph-mon[97450]: pgmap v7185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:41.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:40.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:40.901391+0000) 2022-09-08T01:21:41.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:40.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:40.837814+0000) 2022-09-08T01:21:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:41.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:41.901573+0000) 2022-09-08T01:21:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:41.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:41.838068+0000) 2022-09-08T01:21:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:41 smithi055 ceph-mon[97033]: pgmap v7186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:21:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:41 smithi059 ceph-mon[97450]: pgmap v7186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:21:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:42.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:42.901796+0000) 2022-09-08T01:21:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:42.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:42.838348+0000) 2022-09-08T01:21:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:43.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:43.901988+0000) 2022-09-08T01:21:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:43.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:43.838582+0000) 2022-09-08T01:21:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:43 smithi055 ceph-mon[97033]: pgmap v7187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:43 smithi059 ceph-mon[97450]: pgmap v7187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:44.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:44.902236+0000) 2022-09-08T01:21:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:44.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:44.838830+0000) 2022-09-08T01:21:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:45.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:45.902453+0000) 2022-09-08T01:21:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:45.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:45.839019+0000) 2022-09-08T01:21:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:45 smithi055 ceph-mon[97033]: pgmap v7188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:45 smithi059 ceph-mon[97450]: pgmap v7188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:46.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:46.902720+0000) 2022-09-08T01:21:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:46.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:46.839224+0000) 2022-09-08T01:21:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:47.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:47.902947+0000) 2022-09-08T01:21:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:47.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:47.839435+0000) 2022-09-08T01:21:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:47 smithi055 ceph-mon[97033]: pgmap v7189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:21:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:47 smithi059 ceph-mon[97450]: pgmap v7189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:21:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:48.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:48.839775+0000) 2022-09-08T01:21:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:48.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:48.903095+0000) 2022-09-08T01:21:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:49.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:49.903274+0000) 2022-09-08T01:21:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:49.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:49.839982+0000) 2022-09-08T01:21:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:49 smithi055 ceph-mon[97033]: pgmap v7190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:49 smithi059 ceph-mon[97450]: pgmap v7190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:50.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:50.903520+0000) 2022-09-08T01:21:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:50.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:50.840144+0000) 2022-09-08T01:21:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:51.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:51.903747+0000) 2022-09-08T01:21:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:51.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:51.840378+0000) 2022-09-08T01:21:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:51 smithi055 ceph-mon[97033]: pgmap v7191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:21:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:51 smithi059 ceph-mon[97450]: pgmap v7191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:21:53.124 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:52.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:52.903980+0000) 2022-09-08T01:21:53.125 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:52.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:52.840625+0000) 2022-09-08T01:21:53.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:21:53.122+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:21:54.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:53.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:53.904235+0000) 2022-09-08T01:21:54.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:53.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:53.840842+0000) 2022-09-08T01:21:54.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:53 smithi055 ceph-mon[97033]: pgmap v7192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:53 smithi059 ceph-mon[97450]: pgmap v7192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:54.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:54.904529+0000) 2022-09-08T01:21:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:54.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:54.841143+0000) 2022-09-08T01:21:56.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:55.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:55.904758+0000) 2022-09-08T01:21:56.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:55.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:55.841418+0000) 2022-09-08T01:21:56.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:55 smithi055 ceph-mon[97033]: pgmap v7193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:55 smithi059 ceph-mon[97450]: pgmap v7193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:21:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:56.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:56.905009+0000) 2022-09-08T01:21:57.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:56.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:56.841683+0000) 2022-09-08T01:21:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:57.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:57.905246+0000) 2022-09-08T01:21:58.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:57.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:57.841948+0000) 2022-09-08T01:21:58.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:57 smithi055 ceph-mon[97033]: pgmap v7194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:21:58.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:57 smithi059 ceph-mon[97450]: pgmap v7194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:21:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:58.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:58.842171+0000) 2022-09-08T01:21:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:58.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:58.905439+0000) 2022-09-08T01:22:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:21:59.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:59.905657+0000) 2022-09-08T01:22:00.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:21:59.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:21:59.842416+0000) 2022-09-08T01:22:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:21:59 smithi055 ceph-mon[97033]: pgmap v7195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:21:59 smithi059 ceph-mon[97450]: pgmap v7195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:00.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:00.905832+0000) 2022-09-08T01:22:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:00.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:00.842685+0000) 2022-09-08T01:22:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:01.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:01.906032+0000) 2022-09-08T01:22:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:01.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:01.842859+0000) 2022-09-08T01:22:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:01 smithi055 ceph-mon[97033]: pgmap v7196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:22:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:01 smithi059 ceph-mon[97450]: pgmap v7196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:22:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:02.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:02.906254+0000) 2022-09-08T01:22:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:02.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:02.843068+0000) 2022-09-08T01:22:04.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:03.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:03.906462+0000) 2022-09-08T01:22:04.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:03.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:03.843269+0000) 2022-09-08T01:22:04.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:03 smithi055 ceph-mon[97033]: pgmap v7197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:04.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:03 smithi059 ceph-mon[97450]: pgmap v7197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:04.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:04.906733+0000) 2022-09-08T01:22:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:04.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:04.843493+0000) 2022-09-08T01:22:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:05.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:05.906978+0000) 2022-09-08T01:22:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:05.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:05.843783+0000) 2022-09-08T01:22:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:05 smithi055 ceph-mon[97033]: pgmap v7198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:22:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:22:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:05 smithi059 ceph-mon[97450]: pgmap v7198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:22:06.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:22:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:06.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:06.907206+0000) 2022-09-08T01:22:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:06.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:06.844034+0000) 2022-09-08T01:22:08.135 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:07.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:07.907423+0000) 2022-09-08T01:22:08.135 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:07.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:07.844293+0000) 2022-09-08T01:22:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:07 smithi055 ceph-mon[97033]: pgmap v7199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:22:08.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:07 smithi059 ceph-mon[97450]: pgmap v7199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:22:08.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:22:08.132+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:22:09.160 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:08.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:08.907620+0000) 2022-09-08T01:22:09.169 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:08.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:08.844475+0000) 2022-09-08T01:22:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:09.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:09.844756+0000) 2022-09-08T01:22:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:09.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:09.907899+0000) 2022-09-08T01:22:10.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:09 smithi055 ceph-mon[97033]: pgmap v7200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:22:10.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:09 smithi055 ceph-mon[97033]: osdmap e533: 12 total, 11 up, 11 in 2022-09-08T01:22:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:09 smithi059 ceph-mon[97450]: pgmap v7200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:22:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:09 smithi059 ceph-mon[97450]: osdmap e533: 12 total, 11 up, 11 in 2022-09-08T01:22:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:10.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:10.908131+0000) 2022-09-08T01:22:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:10.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:10.844986+0000) 2022-09-08T01:22:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:11.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:11.908348+0000) 2022-09-08T01:22:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:11.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:11.845168+0000) 2022-09-08T01:22:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:11 smithi055 ceph-mon[97033]: pgmap v7202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:22:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:11 smithi055 ceph-mon[97033]: osdmap e534: 12 total, 11 up, 11 in 2022-09-08T01:22:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:11 smithi059 ceph-mon[97450]: pgmap v7202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:22:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:11 smithi059 ceph-mon[97450]: osdmap e534: 12 total, 11 up, 11 in 2022-09-08T01:22:13.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:12.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:12.908601+0000) 2022-09-08T01:22:13.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:12.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:12.845392+0000) 2022-09-08T01:22:14.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:13.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:13.908830+0000) 2022-09-08T01:22:14.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:13.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:13.845705+0000) 2022-09-08T01:22:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:13 smithi055 ceph-mon[97033]: pgmap v7204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:22:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:13 smithi059 ceph-mon[97450]: pgmap v7204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:22:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:14.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:14.909000+0000) 2022-09-08T01:22:15.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:14.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:14.845885+0000) 2022-09-08T01:22:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:22:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:22:15.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:22:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:22:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:22:15.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:22:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:15.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:15.909135+0000) 2022-09-08T01:22:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:15.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:15.846031+0000) 2022-09-08T01:22:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:15 smithi055 ceph-mon[97033]: pgmap v7205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:22:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:22:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:15 smithi059 ceph-mon[97450]: pgmap v7205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:22:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:22:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:16.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:16.909350+0000) 2022-09-08T01:22:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:16.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:16.846204+0000) 2022-09-08T01:22:18.167 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:17.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:17.909538+0000) 2022-09-08T01:22:18.168 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:17.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:17.846439+0000) 2022-09-08T01:22:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:17 smithi055 ceph-mon[97033]: pgmap v7206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:22:18.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:17 smithi059 ceph-mon[97450]: pgmap v7206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:22:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:18.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:18.909798+0000) 2022-09-08T01:22:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:18.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:18.846620+0000) 2022-09-08T01:22:20.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:19.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:19.910089+0000) 2022-09-08T01:22:20.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:19.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:19.846764+0000) 2022-09-08T01:22:20.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:19 smithi055 ceph-mon[97033]: pgmap v7207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-08T01:22:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:19 smithi059 ceph-mon[97450]: pgmap v7207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-08T01:22:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:20.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:20.910346+0000) 2022-09-08T01:22:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:20.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:20.846992+0000) 2022-09-08T01:22:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:21.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:21.910603+0000) 2022-09-08T01:22:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:21.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:21.847217+0000) 2022-09-08T01:22:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:21 smithi055 ceph-mon[97033]: pgmap v7208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:22:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:21 smithi059 ceph-mon[97450]: pgmap v7208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:22:23.144 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:22.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:22.910757+0000) 2022-09-08T01:22:23.145 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:22.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:22.847426+0000) 2022-09-08T01:22:23.416 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:22:23.143+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:22:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:23.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:23.910988+0000) 2022-09-08T01:22:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:23.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:23.847648+0000) 2022-09-08T01:22:24.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:23 smithi055 ceph-mon[97033]: pgmap v7209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:22:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:23 smithi059 ceph-mon[97450]: pgmap v7209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:22:25.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:24.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:24.911219+0000) 2022-09-08T01:22:25.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:24.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:24.847834+0000) 2022-09-08T01:22:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:25.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:25.911451+0000) 2022-09-08T01:22:26.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:25.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:25.847986+0000) 2022-09-08T01:22:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:25 smithi055 ceph-mon[97033]: pgmap v7210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:22:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:25 smithi059 ceph-mon[97450]: pgmap v7210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:22:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:26.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:26.911723+0000) 2022-09-08T01:22:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:26.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:26.848221+0000) 2022-09-08T01:22:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:27.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:27.911943+0000) 2022-09-08T01:22:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:27.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:27.848459+0000) 2022-09-08T01:22:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:27 smithi055 ceph-mon[97033]: pgmap v7211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:22:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:27 smithi059 ceph-mon[97450]: pgmap v7211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:22:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:28.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:28.912182+0000) 2022-09-08T01:22:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:28.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:28.848743+0000) 2022-09-08T01:22:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:29.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:29.848966+0000) 2022-09-08T01:22:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:29.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:29.912392+0000) 2022-09-08T01:22:30.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:29 smithi055 ceph-mon[97033]: pgmap v7212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:29 smithi059 ceph-mon[97450]: pgmap v7212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:30.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:30.912587+0000) 2022-09-08T01:22:31.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:30.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:30.849177+0000) 2022-09-08T01:22:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:31.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:31.912810+0000) 2022-09-08T01:22:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:31.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:31.849390+0000) 2022-09-08T01:22:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:31 smithi055 ceph-mon[97033]: pgmap v7213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:22:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:31 smithi059 ceph-mon[97450]: pgmap v7213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:22:33.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:32.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:32.849607+0000) 2022-09-08T01:22:33.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:32.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:32.912984+0000) 2022-09-08T01:22:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:33.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:33.849909+0000) 2022-09-08T01:22:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:33.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:33.913227+0000) 2022-09-08T01:22:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:33 smithi055 ceph-mon[97033]: pgmap v7214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:33 smithi059 ceph-mon[97450]: pgmap v7214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:34.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:34.913490+0000) 2022-09-08T01:22:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:34.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:34.850080+0000) 2022-09-08T01:22:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:35.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:35.913735+0000) 2022-09-08T01:22:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:35.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:35.850320+0000) 2022-09-08T01:22:36.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:35 smithi055 ceph-mon[97033]: pgmap v7215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:36.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:35 smithi059 ceph-mon[97450]: pgmap v7215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:36.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:36.914003+0000) 2022-09-08T01:22:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:36.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:36.850556+0000) 2022-09-08T01:22:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:37.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:37.914188+0000) 2022-09-08T01:22:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:37.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:37.850713+0000) 2022-09-08T01:22:38.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:37 smithi055 ceph-mon[97033]: pgmap v7216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:22:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:37 smithi059 ceph-mon[97450]: pgmap v7216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:22:38.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:22:38.158+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:22:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:38.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:38.914320+0000) 2022-09-08T01:22:39.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:38.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:38.850843+0000) 2022-09-08T01:22:40.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:39.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:39.914530+0000) 2022-09-08T01:22:40.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:39.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:39.851018+0000) 2022-09-08T01:22:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:39 smithi055 ceph-mon[97033]: pgmap v7217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:39 smithi059 ceph-mon[97450]: pgmap v7217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:40.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:40.914756+0000) 2022-09-08T01:22:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:40.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:40.851275+0000) 2022-09-08T01:22:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:41.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:41.914908+0000) 2022-09-08T01:22:42.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:41.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:41.851494+0000) 2022-09-08T01:22:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:41 smithi055 ceph-mon[97033]: pgmap v7218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:22:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:41 smithi059 ceph-mon[97450]: pgmap v7218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:22:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:42.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:42.915123+0000) 2022-09-08T01:22:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:42.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:42.851724+0000) 2022-09-08T01:22:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:43.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:43.915363+0000) 2022-09-08T01:22:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:43.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:43.851919+0000) 2022-09-08T01:22:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:43 smithi055 ceph-mon[97033]: pgmap v7219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:43 smithi059 ceph-mon[97450]: pgmap v7219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:44.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:44.915618+0000) 2022-09-08T01:22:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:44.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:44.852069+0000) 2022-09-08T01:22:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:45.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:45.915912+0000) 2022-09-08T01:22:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:45.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:45.852280+0000) 2022-09-08T01:22:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:45 smithi055 ceph-mon[97033]: pgmap v7220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:45 smithi059 ceph-mon[97450]: pgmap v7220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:46.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:46.916159+0000) 2022-09-08T01:22:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:46.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:46.852452+0000) 2022-09-08T01:22:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:47.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:47.916373+0000) 2022-09-08T01:22:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:47.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:47.852736+0000) 2022-09-08T01:22:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:47 smithi055 ceph-mon[97033]: pgmap v7221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:22:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:47 smithi059 ceph-mon[97450]: pgmap v7221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:22:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:48.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:48.916593+0000) 2022-09-08T01:22:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:48.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:48.852975+0000) 2022-09-08T01:22:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:49.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:49.916868+0000) 2022-09-08T01:22:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:49.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:49.853201+0000) 2022-09-08T01:22:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:49 smithi055 ceph-mon[97033]: pgmap v7222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:49 smithi059 ceph-mon[97450]: pgmap v7222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:50.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:50.917041+0000) 2022-09-08T01:22:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:50.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:50.853438+0000) 2022-09-08T01:22:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:51.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:51.917277+0000) 2022-09-08T01:22:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:51.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:51.853708+0000) 2022-09-08T01:22:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:51 smithi055 ceph-mon[97033]: pgmap v7223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:22:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:51 smithi059 ceph-mon[97450]: pgmap v7223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:22:53.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:52.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:52.917497+0000) 2022-09-08T01:22:53.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:52.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:52.853947+0000) 2022-09-08T01:22:53.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:22:53.170+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:22:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:53.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:53.917748+0000) 2022-09-08T01:22:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:53.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:53.854174+0000) 2022-09-08T01:22:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:53 smithi059 ceph-mon[97450]: pgmap v7224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:54.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:53 smithi055 ceph-mon[97033]: pgmap v7224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:55.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:54.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:54.917951+0000) 2022-09-08T01:22:55.162 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:54.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:54.854386+0000) 2022-09-08T01:22:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:55.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:55.918177+0000) 2022-09-08T01:22:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:55.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:55.854585+0000) 2022-09-08T01:22:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:55 smithi059 ceph-mon[97450]: pgmap v7225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:55 smithi055 ceph-mon[97033]: pgmap v7225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:22:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:56.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:56.918458+0000) 2022-09-08T01:22:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:56.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:56.854803+0000) 2022-09-08T01:22:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:57.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:57.918695+0000) 2022-09-08T01:22:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:57.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:57.855064+0000) 2022-09-08T01:22:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:57 smithi059 ceph-mon[97450]: pgmap v7226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:22:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:57 smithi055 ceph-mon[97033]: pgmap v7226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:22:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:58.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:58.918861+0000) 2022-09-08T01:22:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:58.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:58.855345+0000) 2022-09-08T01:23:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:22:59.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:59.919044+0000) 2022-09-08T01:23:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:22:59.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:22:59.855607+0000) 2022-09-08T01:23:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:22:59 smithi059 ceph-mon[97450]: pgmap v7227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:23:00.492 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:22:59 smithi055 ceph-mon[97033]: pgmap v7227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:23:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:00.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:00.919297+0000) 2022-09-08T01:23:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:00.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:00.855781+0000) 2022-09-08T01:23:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:01.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:01.919536+0000) 2022-09-08T01:23:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:01.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:01.855961+0000) 2022-09-08T01:23:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:01 smithi059 ceph-mon[97450]: pgmap v7228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:23:02.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:01 smithi055 ceph-mon[97033]: pgmap v7228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:23:03.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:02.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:02.856193+0000) 2022-09-08T01:23:03.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:02.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:02.919692+0000) 2022-09-08T01:23:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:03.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:03.856396+0000) 2022-09-08T01:23:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:03.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:03.919886+0000) 2022-09-08T01:23:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:04 smithi059 ceph-mon[97450]: pgmap v7229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:23:04.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:04 smithi055 ceph-mon[97033]: pgmap v7229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:23:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:04.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:04.856576+0000) 2022-09-08T01:23:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:04.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:04.920056+0000) 2022-09-08T01:23:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:05.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:05.856700+0000) 2022-09-08T01:23:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:05.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:05.920246+0000) 2022-09-08T01:23:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:06 smithi059 ceph-mon[97450]: pgmap v7230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:23:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:23:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:23:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:06 smithi055 ceph-mon[97033]: pgmap v7230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:23:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:23:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:23:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:06.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:06.920471+0000) 2022-09-08T01:23:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:06.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:06.856816+0000) 2022-09-08T01:23:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:07.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:07.920724+0000) 2022-09-08T01:23:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:07.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:07.856979+0000) 2022-09-08T01:23:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:08 smithi059 ceph-mon[97450]: pgmap v7231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:23:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:08 smithi055 ceph-mon[97033]: pgmap v7231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:23:08.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:23:08.186+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:23:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:08.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:08.857160+0000) 2022-09-08T01:23:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:08.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:08.920951+0000) 2022-09-08T01:23:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:09.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:09.921180+0000) 2022-09-08T01:23:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:09.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:09.857420+0000) 2022-09-08T01:23:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:10 smithi059 ceph-mon[97450]: pgmap v7232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:23:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:10 smithi059 ceph-mon[97450]: osdmap e535: 12 total, 11 up, 11 in 2022-09-08T01:23:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:10 smithi055 ceph-mon[97033]: pgmap v7232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:23:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:10 smithi055 ceph-mon[97033]: osdmap e535: 12 total, 11 up, 11 in 2022-09-08T01:23:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:10.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:10.921427+0000) 2022-09-08T01:23:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:10.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:10.857701+0000) 2022-09-08T01:23:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:11.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:11.921677+0000) 2022-09-08T01:23:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:11.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:11.857927+0000) 2022-09-08T01:23:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:12 smithi059 ceph-mon[97450]: pgmap v7234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:23:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:12 smithi059 ceph-mon[97450]: osdmap e536: 12 total, 11 up, 11 in 2022-09-08T01:23:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:12 smithi055 ceph-mon[97033]: pgmap v7234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:23:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:12 smithi055 ceph-mon[97033]: osdmap e536: 12 total, 11 up, 11 in 2022-09-08T01:23:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:12.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:12.921961+0000) 2022-09-08T01:23:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:12.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:12.858132+0000) 2022-09-08T01:23:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:13.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:13.922231+0000) 2022-09-08T01:23:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:13.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:13.858379+0000) 2022-09-08T01:23:14.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:14 smithi059 ceph-mon[97450]: pgmap v7236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:23:14.495 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:14 smithi055 ceph-mon[97033]: pgmap v7236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:23:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:14.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:14.922470+0000) 2022-09-08T01:23:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:14.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:14.858646+0000) 2022-09-08T01:23:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:15.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:15.858867+0000) 2022-09-08T01:23:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:15.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:15.922736+0000) 2022-09-08T01:23:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:16 smithi059 ceph-mon[97450]: pgmap v7237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:23:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:23:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:23:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:23:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:16 smithi055 ceph-mon[97033]: pgmap v7237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:23:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:23:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:23:16.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:23:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:16.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:16.922918+0000) 2022-09-08T01:23:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:16.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:16.859072+0000) 2022-09-08T01:23:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:23:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:23:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:17.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:17.923127+0000) 2022-09-08T01:23:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:17.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:17.859310+0000) 2022-09-08T01:23:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:18 smithi055 ceph-mon[97033]: pgmap v7238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:23:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:18 smithi059 ceph-mon[97450]: pgmap v7238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:23:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:18.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:18.923339+0000) 2022-09-08T01:23:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:18.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:18.859610+0000) 2022-09-08T01:23:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:19.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:19.923527+0000) 2022-09-08T01:23:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:19.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:19.859892+0000) 2022-09-08T01:23:20.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:20 smithi055 ceph-mon[97033]: pgmap v7239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2022-09-08T01:23:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:20 smithi059 ceph-mon[97450]: pgmap v7239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2022-09-08T01:23:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:20.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:20.860157+0000) 2022-09-08T01:23:21.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:20.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:20.923715+0000) 2022-09-08T01:23:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:21.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:21.923881+0000) 2022-09-08T01:23:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:21.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:21.860406+0000) 2022-09-08T01:23:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:22 smithi055 ceph-mon[97033]: pgmap v7240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:23:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:22 smithi059 ceph-mon[97450]: pgmap v7240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:23:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:22.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:22.924134+0000) 2022-09-08T01:23:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:22.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:22.860694+0000) 2022-09-08T01:23:23.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:23:23.196+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:23:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:23.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:23.924334+0000) 2022-09-08T01:23:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:23.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:23.860846+0000) 2022-09-08T01:23:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:24 smithi055 ceph-mon[97033]: pgmap v7241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:23:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:24 smithi059 ceph-mon[97450]: pgmap v7241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:23:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:24.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:24.924619+0000) 2022-09-08T01:23:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:24.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:24.861077+0000) 2022-09-08T01:23:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:25.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:25.924880+0000) 2022-09-08T01:23:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:25.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:25.861334+0000) 2022-09-08T01:23:26.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:26 smithi055 ceph-mon[97033]: pgmap v7242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:23:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:26 smithi059 ceph-mon[97450]: pgmap v7242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:23:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:26.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:26.925102+0000) 2022-09-08T01:23:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:26.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:26.861578+0000) 2022-09-08T01:23:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:27.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:27.861811+0000) 2022-09-08T01:23:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:27.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:27.925306+0000) 2022-09-08T01:23:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:28 smithi055 ceph-mon[97033]: pgmap v7243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:23:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:28 smithi059 ceph-mon[97450]: pgmap v7243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:23:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:28.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:28.925527+0000) 2022-09-08T01:23:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:28.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:28.861961+0000) 2022-09-08T01:23:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:29.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:29.925752+0000) 2022-09-08T01:23:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:29.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:29.862220+0000) 2022-09-08T01:23:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:30 smithi055 ceph-mon[97033]: pgmap v7244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:30.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:30 smithi059 ceph-mon[97450]: pgmap v7244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:31.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:30.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:30.925956+0000) 2022-09-08T01:23:31.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:30.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:30.862449+0000) 2022-09-08T01:23:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:31.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:31.926171+0000) 2022-09-08T01:23:32.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:31.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:31.862757+0000) 2022-09-08T01:23:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:32 smithi055 ceph-mon[97033]: pgmap v7245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:23:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:32 smithi059 ceph-mon[97450]: pgmap v7245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:23:33.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:32.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:32.863035+0000) 2022-09-08T01:23:33.160 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:32.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:32.926401+0000) 2022-09-08T01:23:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:33.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:33.863233+0000) 2022-09-08T01:23:34.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:33.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:33.926702+0000) 2022-09-08T01:23:34.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:34 smithi055 ceph-mon[97033]: pgmap v7246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:34.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:34 smithi059 ceph-mon[97450]: pgmap v7246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:34.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:34.863408+0000) 2022-09-08T01:23:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:34.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:34.926854+0000) 2022-09-08T01:23:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:35.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:35.863606+0000) 2022-09-08T01:23:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:35.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:35.927016+0000) 2022-09-08T01:23:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:36 smithi055 ceph-mon[97033]: pgmap v7247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:36 smithi059 ceph-mon[97450]: pgmap v7247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:36.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:36.927202+0000) 2022-09-08T01:23:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:36.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:36.863852+0000) 2022-09-08T01:23:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:37.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:37.864091+0000) 2022-09-08T01:23:38.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:37.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:37.927412+0000) 2022-09-08T01:23:38.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:38 smithi055 ceph-mon[97033]: pgmap v7248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:23:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:38 smithi059 ceph-mon[97450]: pgmap v7248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:23:38.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:23:38.210+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:23:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:38.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:38.927670+0000) 2022-09-08T01:23:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:38.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:38.864333+0000) 2022-09-08T01:23:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:39.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:39.927922+0000) 2022-09-08T01:23:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:39.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:39.864597+0000) 2022-09-08T01:23:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:40 smithi055 ceph-mon[97033]: pgmap v7249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:40 smithi059 ceph-mon[97450]: pgmap v7249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:40.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:40.864859+0000) 2022-09-08T01:23:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:40.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:40.928200+0000) 2022-09-08T01:23:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:41.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:41.865046+0000) 2022-09-08T01:23:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:41.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:41.928421+0000) 2022-09-08T01:23:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:42 smithi055 ceph-mon[97033]: pgmap v7250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:23:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:42 smithi059 ceph-mon[97450]: pgmap v7250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:23:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:42.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:42.928597+0000) 2022-09-08T01:23:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:42.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:42.865229+0000) 2022-09-08T01:23:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:43.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:43.928706+0000) 2022-09-08T01:23:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:43.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:43.865507+0000) 2022-09-08T01:23:44.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:44 smithi055 ceph-mon[97033]: pgmap v7251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:44 smithi059 ceph-mon[97450]: pgmap v7251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:44.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:44.928862+0000) 2022-09-08T01:23:45.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:44.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:44.865790+0000) 2022-09-08T01:23:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:45.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:45.929062+0000) 2022-09-08T01:23:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:45.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:45.866082+0000) 2022-09-08T01:23:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:46 smithi055 ceph-mon[97033]: pgmap v7252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:46 smithi059 ceph-mon[97450]: pgmap v7252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:46.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:46.929238+0000) 2022-09-08T01:23:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:46.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:46.866363+0000) 2022-09-08T01:23:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:47.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:47.929446+0000) 2022-09-08T01:23:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:47.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:47.866675+0000) 2022-09-08T01:23:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:48 smithi055 ceph-mon[97033]: pgmap v7253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:23:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:48 smithi059 ceph-mon[97450]: pgmap v7253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:23:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:48.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:48.929682+0000) 2022-09-08T01:23:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:48.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:48.866943+0000) 2022-09-08T01:23:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:49.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:49.929867+0000) 2022-09-08T01:23:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:49.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:49.867161+0000) 2022-09-08T01:23:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:50 smithi055 ceph-mon[97033]: pgmap v7254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:50 smithi059 ceph-mon[97450]: pgmap v7254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:50.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:50.930118+0000) 2022-09-08T01:23:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:50.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:50.867384+0000) 2022-09-08T01:23:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:51.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:51.867545+0000) 2022-09-08T01:23:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:51.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:51.930342+0000) 2022-09-08T01:23:52.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:52 smithi055 ceph-mon[97033]: pgmap v7255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:23:52.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:52 smithi059 ceph-mon[97450]: pgmap v7255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:23:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:52.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:52.867809+0000) 2022-09-08T01:23:53.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:52.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:52.930533+0000) 2022-09-08T01:23:53.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:23:53.215+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:23:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:53.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:53.930738+0000) 2022-09-08T01:23:54.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:53.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:53.867991+0000) 2022-09-08T01:23:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:54 smithi055 ceph-mon[97033]: pgmap v7256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:54 smithi059 ceph-mon[97450]: pgmap v7256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:54.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:54.930890+0000) 2022-09-08T01:23:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:54.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:54.868229+0000) 2022-09-08T01:23:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:55.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:55.931052+0000) 2022-09-08T01:23:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:55.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:55.868440+0000) 2022-09-08T01:23:56.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:56 smithi055 ceph-mon[97033]: pgmap v7257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:56.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:56 smithi059 ceph-mon[97450]: pgmap v7257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:23:57.162 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:56.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:56.931301+0000) 2022-09-08T01:23:57.163 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:56.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:56.868708+0000) 2022-09-08T01:23:58.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:57.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:57.931534+0000) 2022-09-08T01:23:58.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:57.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:57.868935+0000) 2022-09-08T01:23:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:23:58 smithi055 ceph-mon[97033]: pgmap v7258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:23:58.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:23:58 smithi059 ceph-mon[97450]: pgmap v7258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:23:59.160 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:58.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:58.931760+0000) 2022-09-08T01:23:59.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:58.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:58.869197+0000) 2022-09-08T01:24:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:23:59.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:59.932007+0000) 2022-09-08T01:24:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:23:59.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:23:59.869356+0000) 2022-09-08T01:24:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:00 smithi055 ceph-mon[97033]: pgmap v7259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:00 smithi059 ceph-mon[97450]: pgmap v7259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:00.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:00.932251+0000) 2022-09-08T01:24:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:00.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:00.869575+0000) 2022-09-08T01:24:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:01.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:01.932472+0000) 2022-09-08T01:24:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:01.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:01.869764+0000) 2022-09-08T01:24:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:02 smithi055 ceph-mon[97033]: pgmap v7260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:24:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:02 smithi059 ceph-mon[97450]: pgmap v7260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:24:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:02.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:02.932603+0000) 2022-09-08T01:24:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:02.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:02.869936+0000) 2022-09-08T01:24:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:03.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:03.932831+0000) 2022-09-08T01:24:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:03.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:03.870167+0000) 2022-09-08T01:24:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:04 smithi055 ceph-mon[97033]: pgmap v7261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:04 smithi059 ceph-mon[97450]: pgmap v7261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:04.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:04.933093+0000) 2022-09-08T01:24:05.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:04.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:04.870398+0000) 2022-09-08T01:24:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:24:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:24:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:24:05.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:24:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:05.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:05.933346+0000) 2022-09-08T01:24:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:05.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:05.870661+0000) 2022-09-08T01:24:06.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:06 smithi055 ceph-mon[97033]: pgmap v7262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:06 smithi059 ceph-mon[97450]: pgmap v7262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:06.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:06.933559+0000) 2022-09-08T01:24:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:06.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:06.870926+0000) 2022-09-08T01:24:08.163 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:07.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:07.933829+0000) 2022-09-08T01:24:08.164 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:07.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:07.871206+0000) 2022-09-08T01:24:08.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:08 smithi055 ceph-mon[97033]: pgmap v7263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:24:08.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:08 smithi059 ceph-mon[97450]: pgmap v7263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:24:08.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:24:08.229+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:24:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:08.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:08.871394+0000) 2022-09-08T01:24:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:08.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:08.934047+0000) 2022-09-08T01:24:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:09.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:09.871536+0000) 2022-09-08T01:24:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:09.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:09.934204+0000) 2022-09-08T01:24:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:10 smithi055 ceph-mon[97033]: pgmap v7264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:24:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:10 smithi055 ceph-mon[97033]: osdmap e537: 12 total, 11 up, 11 in 2022-09-08T01:24:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:10 smithi059 ceph-mon[97450]: pgmap v7264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:24:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:10 smithi059 ceph-mon[97450]: osdmap e537: 12 total, 11 up, 11 in 2022-09-08T01:24:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:10.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:10.934406+0000) 2022-09-08T01:24:11.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:10.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:10.871770+0000) 2022-09-08T01:24:12.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:11.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:11.934704+0000) 2022-09-08T01:24:12.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:11.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:11.872004+0000) 2022-09-08T01:24:12.487 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:12 smithi055 ceph-mon[97033]: pgmap v7266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:24:12.488 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:12 smithi055 ceph-mon[97033]: osdmap e538: 12 total, 11 up, 11 in 2022-09-08T01:24:12.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:12 smithi059 ceph-mon[97450]: pgmap v7266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:24:12.543 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:12 smithi059 ceph-mon[97450]: osdmap e538: 12 total, 11 up, 11 in 2022-09-08T01:24:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:12.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:12.934928+0000) 2022-09-08T01:24:13.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:12.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:12.872255+0000) 2022-09-08T01:24:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:13.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:13.935137+0000) 2022-09-08T01:24:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:13.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:13.872488+0000) 2022-09-08T01:24:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:14 smithi055 ceph-mon[97033]: pgmap v7268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:24:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:14 smithi059 ceph-mon[97450]: pgmap v7268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:24:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:14.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:14.935405+0000) 2022-09-08T01:24:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:14.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:14.872631+0000) 2022-09-08T01:24:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:15.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:15.935668+0000) 2022-09-08T01:24:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:15.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:15.872855+0000) 2022-09-08T01:24:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:16 smithi055 ceph-mon[97033]: pgmap v7269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:24:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:24:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:24:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:24:16.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:16 smithi059 ceph-mon[97450]: pgmap v7269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:24:16.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:24:16.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:24:16.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:24:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:16.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:16.873074+0000) 2022-09-08T01:24:17.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:16.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:16.935894+0000) 2022-09-08T01:24:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:17 smithi055 ceph-mon[97033]: pgmap v7270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:24:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:24:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:24:18.041 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:17 smithi059 ceph-mon[97450]: pgmap v7270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:24:18.042 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:24:18.043 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:24:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:17.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:17.936071+0000) 2022-09-08T01:24:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:17.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:17.873310+0000) 2022-09-08T01:24:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:18.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:18.936255+0000) 2022-09-08T01:24:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:18.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:18.873568+0000) 2022-09-08T01:24:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:19.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:19.936389+0000) 2022-09-08T01:24:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:19.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:19.873762+0000) 2022-09-08T01:24:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:19 smithi055 ceph-mon[97033]: pgmap v7271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 225 B/s wr, 7 op/s 2022-09-08T01:24:20.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:24:20.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:24:20.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:24:20.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:24:20.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.239 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.239 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.240 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:19 smithi059 ceph-mon[97450]: pgmap v7271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 225 B/s wr, 7 op/s 2022-09-08T01:24:20.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:24:20.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:24:20.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:24:20.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:24:20.303 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.304 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.304 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:20.305 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:24:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:20.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:20.936565+0000) 2022-09-08T01:24:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:20.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:20.873941+0000) 2022-09-08T01:24:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:20 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T01:24:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:20 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T01:24:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:20 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T01:24:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:20 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T01:24:21.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:24:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:20 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T01:24:21.293 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:20 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T01:24:21.293 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:20 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T01:24:21.294 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:20 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T01:24:21.295 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:24:22.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:21.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:21.936781+0000) 2022-09-08T01:24:22.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:21.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:21.874077+0000) 2022-09-08T01:24:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:21 smithi055 ceph-mon[97033]: pgmap v7272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:24:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:21 smithi059 ceph-mon[97450]: pgmap v7272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:24:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:22.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:22.936967+0000) 2022-09-08T01:24:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:22.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:22.874276+0000) 2022-09-08T01:24:23.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:24:23.243+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:24:24.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:23.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:23.937238+0000) 2022-09-08T01:24:24.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:23.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:23.874505+0000) 2022-09-08T01:24:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:23 smithi055 ceph-mon[97033]: pgmap v7273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:24:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:23 smithi059 ceph-mon[97450]: pgmap v7273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:24:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:24.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:24.937396+0000) 2022-09-08T01:24:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:24.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:24.874776+0000) 2022-09-08T01:24:26.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:25.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:25.874952+0000) 2022-09-08T01:24:26.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:25.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:25.937600+0000) 2022-09-08T01:24:26.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:25 smithi055 ceph-mon[97033]: pgmap v7274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:24:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:25 smithi059 ceph-mon[97450]: pgmap v7274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:24:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:26.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:26.875136+0000) 2022-09-08T01:24:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:26.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:26.937835+0000) 2022-09-08T01:24:28.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:27.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:27.938034+0000) 2022-09-08T01:24:28.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:27.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:27.875400+0000) 2022-09-08T01:24:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:27 smithi055 ceph-mon[97033]: pgmap v7275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:24:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:27 smithi059 ceph-mon[97450]: pgmap v7275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:24:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:28.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:28.938276+0000) 2022-09-08T01:24:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:28.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:28.875619+0000) 2022-09-08T01:24:30.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:29.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:29.938518+0000) 2022-09-08T01:24:30.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:29.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:29.875921+0000) 2022-09-08T01:24:30.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:29 smithi055 ceph-mon[97033]: pgmap v7276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:30.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:29 smithi059 ceph-mon[97450]: pgmap v7276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:30.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:30.938753+0000) 2022-09-08T01:24:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:30.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:30.876099+0000) 2022-09-08T01:24:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:31.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:31.938887+0000) 2022-09-08T01:24:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:31.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:31.876317+0000) 2022-09-08T01:24:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:31 smithi055 ceph-mon[97033]: pgmap v7277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:24:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:31 smithi059 ceph-mon[97450]: pgmap v7277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:24:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:32.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:32.939112+0000) 2022-09-08T01:24:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:32.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:32.876564+0000) 2022-09-08T01:24:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:33.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:33.939357+0000) 2022-09-08T01:24:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:33.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:33.876785+0000) 2022-09-08T01:24:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:33 smithi059 ceph-mon[97450]: pgmap v7278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:33 smithi055 ceph-mon[97033]: pgmap v7278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:34.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:34.939549+0000) 2022-09-08T01:24:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:34.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:34.877005+0000) 2022-09-08T01:24:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:35.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:35.939730+0000) 2022-09-08T01:24:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:35.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:35.877244+0000) 2022-09-08T01:24:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:35 smithi059 ceph-mon[97450]: pgmap v7279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:36.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:35 smithi055 ceph-mon[97033]: pgmap v7279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:36.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:36.939857+0000) 2022-09-08T01:24:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:36.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:36.877474+0000) 2022-09-08T01:24:38.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:37.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:37.940060+0000) 2022-09-08T01:24:38.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:37.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:37.877649+0000) 2022-09-08T01:24:38.300 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:37 smithi059 ceph-mon[97450]: pgmap v7280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:24:38.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:37 smithi055 ceph-mon[97033]: pgmap v7280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:24:38.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:24:38.246+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:24:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:38.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:38.940297+0000) 2022-09-08T01:24:39.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:38.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:38.877784+0000) 2022-09-08T01:24:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:39.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:39.940542+0000) 2022-09-08T01:24:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:39.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:39.877973+0000) 2022-09-08T01:24:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:39 smithi059 ceph-mon[97450]: pgmap v7281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:39 smithi055 ceph-mon[97033]: pgmap v7281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:40.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:40.940696+0000) 2022-09-08T01:24:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:40.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:40.878228+0000) 2022-09-08T01:24:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:41.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:41.940897+0000) 2022-09-08T01:24:42.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:41.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:41.878422+0000) 2022-09-08T01:24:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:41 smithi059 ceph-mon[97450]: pgmap v7282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:24:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:41 smithi055 ceph-mon[97033]: pgmap v7282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:24:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:42.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:42.941141+0000) 2022-09-08T01:24:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:42.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:42.878615+0000) 2022-09-08T01:24:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:43.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:43.941386+0000) 2022-09-08T01:24:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:43.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:43.878777+0000) 2022-09-08T01:24:44.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:43 smithi059 ceph-mon[97450]: pgmap v7283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:43 smithi055 ceph-mon[97033]: pgmap v7283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:45.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:44.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:44.941660+0000) 2022-09-08T01:24:45.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:44.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:44.879049+0000) 2022-09-08T01:24:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:45.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:45.941916+0000) 2022-09-08T01:24:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:45.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:45.879231+0000) 2022-09-08T01:24:46.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:45 smithi059 ceph-mon[97450]: pgmap v7284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:46.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:45 smithi055 ceph-mon[97033]: pgmap v7284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:46.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:46.879423+0000) 2022-09-08T01:24:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:46.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:46.942176+0000) 2022-09-08T01:24:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:47.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:47.942426+0000) 2022-09-08T01:24:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:47.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:47.879723+0000) 2022-09-08T01:24:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:47 smithi059 ceph-mon[97450]: pgmap v7285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:24:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:48 smithi055 ceph-mon[97033]: pgmap v7285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:24:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:48.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:48.942680+0000) 2022-09-08T01:24:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:48.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:48.879867+0000) 2022-09-08T01:24:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:49.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:49.942884+0000) 2022-09-08T01:24:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:49.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:49.879988+0000) 2022-09-08T01:24:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:50 smithi059 ceph-mon[97450]: pgmap v7286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:50.489 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:50 smithi055 ceph-mon[97033]: pgmap v7286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:51.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:50.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:50.943118+0000) 2022-09-08T01:24:51.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:50.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:50.880180+0000) 2022-09-08T01:24:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:51.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:51.943304+0000) 2022-09-08T01:24:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:51.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:51.880405+0000) 2022-09-08T01:24:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:52 smithi059 ceph-mon[97450]: pgmap v7287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:24:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:52 smithi055 ceph-mon[97033]: pgmap v7287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:24:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:52.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:52.943560+0000) 2022-09-08T01:24:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:52.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:52.880633+0000) 2022-09-08T01:24:53.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:24:53.262+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:24:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:53.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:53.943826+0000) 2022-09-08T01:24:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:53.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:53.880904+0000) 2022-09-08T01:24:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:54 smithi059 ceph-mon[97450]: pgmap v7288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:54.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:54 smithi055 ceph-mon[97033]: pgmap v7288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:54.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:54.944022+0000) 2022-09-08T01:24:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:54.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:54.881139+0000) 2022-09-08T01:24:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:55.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:55.881349+0000) 2022-09-08T01:24:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:55.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:55.944207+0000) 2022-09-08T01:24:56.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:56 smithi059 ceph-mon[97450]: pgmap v7289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:56.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:56 smithi055 ceph-mon[97033]: pgmap v7289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:24:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:56.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:56.944396+0000) 2022-09-08T01:24:57.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:56.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:56.881587+0000) 2022-09-08T01:24:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:57.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:57.944608+0000) 2022-09-08T01:24:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:57.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:57.881816+0000) 2022-09-08T01:24:58.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:24:58 smithi059 ceph-mon[97450]: pgmap v7290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:24:58.488 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:24:58 smithi055 ceph-mon[97033]: pgmap v7290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:24:59.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:58.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:58.944697+0000) 2022-09-08T01:24:59.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:58.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:58.882087+0000) 2022-09-08T01:25:00.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:24:59.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:59.944945+0000) 2022-09-08T01:25:00.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:24:59.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:24:59.882256+0000) 2022-09-08T01:25:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:00 smithi059 ceph-mon[97450]: pgmap v7291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:00 smithi055 ceph-mon[97033]: pgmap v7291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:00.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:00.945141+0000) 2022-09-08T01:25:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:00.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:00.882389+0000) 2022-09-08T01:25:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:01.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:01.945313+0000) 2022-09-08T01:25:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:01.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:01.882550+0000) 2022-09-08T01:25:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:02 smithi059 ceph-mon[97450]: pgmap v7292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:25:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:02 smithi055 ceph-mon[97033]: pgmap v7292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:25:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:02.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:02.945527+0000) 2022-09-08T01:25:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:02.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:02.882750+0000) 2022-09-08T01:25:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:03.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:03.882984+0000) 2022-09-08T01:25:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:03.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:03.945794+0000) 2022-09-08T01:25:04.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:04 smithi059 ceph-mon[97450]: pgmap v7293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:04 smithi055 ceph-mon[97033]: pgmap v7293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:05.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:04.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:04.883176+0000) 2022-09-08T01:25:05.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:04.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:04.946061+0000) 2022-09-08T01:25:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:05.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:05.946379+0000) 2022-09-08T01:25:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:05.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:05.883376+0000) 2022-09-08T01:25:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:06 smithi059 ceph-mon[97450]: pgmap v7294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:25:06.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:25:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:06 smithi055 ceph-mon[97033]: pgmap v7294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:06.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:25:06.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:25:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:06.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:06.883627+0000) 2022-09-08T01:25:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:06.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:06.946662+0000) 2022-09-08T01:25:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:07.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:07.946820+0000) 2022-09-08T01:25:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:07.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:07.883878+0000) 2022-09-08T01:25:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:08 smithi055 ceph-mon[97033]: pgmap v7295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:25:08.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:08 smithi059 ceph-mon[97450]: pgmap v7295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:25:08.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:25:08.269+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:25:09.161 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:08.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:08.947039+0000) 2022-09-08T01:25:09.162 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:08.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:08.884101+0000) 2022-09-08T01:25:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:09.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:09.947307+0000) 2022-09-08T01:25:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:09.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:09.884311+0000) 2022-09-08T01:25:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:10 smithi055 ceph-mon[97033]: pgmap v7296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:25:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:10 smithi055 ceph-mon[97033]: osdmap e539: 12 total, 11 up, 11 in 2022-09-08T01:25:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:10 smithi059 ceph-mon[97450]: pgmap v7296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:25:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:10 smithi059 ceph-mon[97450]: osdmap e539: 12 total, 11 up, 11 in 2022-09-08T01:25:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:10.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:10.947589+0000) 2022-09-08T01:25:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:10.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:10.884552+0000) 2022-09-08T01:25:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:11.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:11.884779+0000) 2022-09-08T01:25:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:11.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:11.947839+0000) 2022-09-08T01:25:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:12 smithi055 ceph-mon[97033]: pgmap v7298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:25:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:12 smithi055 ceph-mon[97033]: osdmap e540: 12 total, 11 up, 11 in 2022-09-08T01:25:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:12 smithi059 ceph-mon[97450]: pgmap v7298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:25:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:12 smithi059 ceph-mon[97450]: osdmap e540: 12 total, 11 up, 11 in 2022-09-08T01:25:13.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:12.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:12.948059+0000) 2022-09-08T01:25:13.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:12.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:12.884905+0000) 2022-09-08T01:25:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:13.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:13.948214+0000) 2022-09-08T01:25:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:13.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:13.885149+0000) 2022-09-08T01:25:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:14 smithi055 ceph-mon[97033]: pgmap v7300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:25:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:14 smithi059 ceph-mon[97450]: pgmap v7300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:25:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:14.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:14.948467+0000) 2022-09-08T01:25:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:14.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:14.885383+0000) 2022-09-08T01:25:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:15.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:15.885601+0000) 2022-09-08T01:25:16.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:15.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:15.948766+0000) 2022-09-08T01:25:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:16 smithi055 ceph-mon[97033]: pgmap v7301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:25:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:16 smithi059 ceph-mon[97450]: pgmap v7301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:25:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:16.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:16.949054+0000) 2022-09-08T01:25:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:16.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:16.885853+0000) 2022-09-08T01:25:18.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:17.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:17.949283+0000) 2022-09-08T01:25:18.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:17.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:17.886080+0000) 2022-09-08T01:25:18.490 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:18 smithi055 ceph-mon[97033]: pgmap v7302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:25:18.539 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:18 smithi059 ceph-mon[97450]: pgmap v7302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:25:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:18.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:18.886296+0000) 2022-09-08T01:25:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:18.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:18.949491+0000) 2022-09-08T01:25:20.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:19.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:19.949706+0000) 2022-09-08T01:25:20.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:19.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:19.886570+0000) 2022-09-08T01:25:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:20 smithi055 ceph-mon[97033]: pgmap v7303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s 2022-09-08T01:25:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:25:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:25:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:25:20.539 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:20 smithi059 ceph-mon[97450]: pgmap v7303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s 2022-09-08T01:25:20.540 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:25:20.540 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:25:20.541 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:25:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:20.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:20.949947+0000) 2022-09-08T01:25:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:20.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:20.886817+0000) 2022-09-08T01:25:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:21 smithi055 ceph-mon[97033]: pgmap v7304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:25:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:25:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:21 smithi059 ceph-mon[97450]: pgmap v7304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:25:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:25:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:21.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:21.950131+0000) 2022-09-08T01:25:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:21.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:21.887064+0000) 2022-09-08T01:25:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:22.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:22.887216+0000) 2022-09-08T01:25:23.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:22.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:22.950333+0000) 2022-09-08T01:25:23.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:25:23.279+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:25:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:23 smithi055 ceph-mon[97033]: pgmap v7305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:25:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:23 smithi059 ceph-mon[97450]: pgmap v7305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:25:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:23.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:23.950578+0000) 2022-09-08T01:25:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:23.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:23.887385+0000) 2022-09-08T01:25:25.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:24.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:24.950822+0000) 2022-09-08T01:25:25.164 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:24.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:24.887599+0000) 2022-09-08T01:25:25.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:25 smithi055 ceph-mon[97033]: pgmap v7306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:25:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:25 smithi059 ceph-mon[97450]: pgmap v7306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:25:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:25.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:25.950953+0000) 2022-09-08T01:25:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:25.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:25.887789+0000) 2022-09-08T01:25:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:26.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:26.951162+0000) 2022-09-08T01:25:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:26.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:26.888015+0000) 2022-09-08T01:25:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:27 smithi055 ceph-mon[97033]: pgmap v7307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:25:28.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:27 smithi059 ceph-mon[97450]: pgmap v7307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:25:28.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:27.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:27.888201+0000) 2022-09-08T01:25:28.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:27.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:27.951433+0000) 2022-09-08T01:25:29.163 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:28.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:28.951710+0000) 2022-09-08T01:25:29.164 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:28.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:28.888333+0000) 2022-09-08T01:25:29.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:29 smithi055 ceph-mon[97033]: pgmap v7308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:29 smithi059 ceph-mon[97450]: pgmap v7308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:29.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:29.951853+0000) 2022-09-08T01:25:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:29.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:29.888560+0000) 2022-09-08T01:25:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:30.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:30.888789+0000) 2022-09-08T01:25:31.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:30.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:30.952042+0000) 2022-09-08T01:25:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:31 smithi055 ceph-mon[97033]: pgmap v7309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:25:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:31 smithi059 ceph-mon[97450]: pgmap v7309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:25:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:31.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:31.952286+0000) 2022-09-08T01:25:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:31.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:31.888952+0000) 2022-09-08T01:25:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:32.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:32.889189+0000) 2022-09-08T01:25:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:32.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:32.952527+0000) 2022-09-08T01:25:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:33 smithi055 ceph-mon[97033]: pgmap v7310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:34.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:33 smithi059 ceph-mon[97450]: pgmap v7310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:33.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:33.952713+0000) 2022-09-08T01:25:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:33.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:33.889465+0000) 2022-09-08T01:25:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:34.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:34.952928+0000) 2022-09-08T01:25:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:34.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:34.889725+0000) 2022-09-08T01:25:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:35 smithi055 ceph-mon[97033]: pgmap v7311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:35 smithi059 ceph-mon[97450]: pgmap v7311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:35.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:35.953144+0000) 2022-09-08T01:25:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:35.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:35.889914+0000) 2022-09-08T01:25:37.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:36.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:36.953390+0000) 2022-09-08T01:25:37.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:36.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:36.890121+0000) 2022-09-08T01:25:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:37 smithi055 ceph-mon[97033]: pgmap v7312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:25:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:37 smithi059 ceph-mon[97450]: pgmap v7312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:25:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:37.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:37.890263+0000) 2022-09-08T01:25:38.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:37.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:37.953620+0000) 2022-09-08T01:25:38.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:25:38.294+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:25:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:38.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:38.953852+0000) 2022-09-08T01:25:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:38.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:38.890446+0000) 2022-09-08T01:25:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:39 smithi055 ceph-mon[97033]: pgmap v7313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:39 smithi059 ceph-mon[97450]: pgmap v7313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:39.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:39.954019+0000) 2022-09-08T01:25:40.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:39.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:39.890672+0000) 2022-09-08T01:25:41.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:40.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:40.954179+0000) 2022-09-08T01:25:41.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:40.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:40.890884+0000) 2022-09-08T01:25:41.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:41 smithi055 ceph-mon[97033]: pgmap v7314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:25:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:41 smithi059 ceph-mon[97450]: pgmap v7314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:25:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:41.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:41.954400+0000) 2022-09-08T01:25:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:41.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:41.891080+0000) 2022-09-08T01:25:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:42.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:42.891228+0000) 2022-09-08T01:25:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:42.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:42.954605+0000) 2022-09-08T01:25:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:43 smithi055 ceph-mon[97033]: pgmap v7315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:43 smithi059 ceph-mon[97450]: pgmap v7315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:43.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:43.954774+0000) 2022-09-08T01:25:44.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:43.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:43.891437+0000) 2022-09-08T01:25:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:44.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:44.954947+0000) 2022-09-08T01:25:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:44.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:44.891653+0000) 2022-09-08T01:25:45.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:45 smithi055 ceph-mon[97033]: pgmap v7316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:46.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:45 smithi059 ceph-mon[97450]: pgmap v7316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:45.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:45.955140+0000) 2022-09-08T01:25:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:45.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:45.891859+0000) 2022-09-08T01:25:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:46.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:46.955353+0000) 2022-09-08T01:25:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:46.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:46.892085+0000) 2022-09-08T01:25:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:47 smithi059 ceph-mon[97450]: pgmap v7317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:25:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:47.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:47.955574+0000) 2022-09-08T01:25:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:47.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:47.892316+0000) 2022-09-08T01:25:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:47 smithi055 ceph-mon[97033]: pgmap v7317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:25:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:48.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:48.955805+0000) 2022-09-08T01:25:49.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:48.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:48.892551+0000) 2022-09-08T01:25:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:49 smithi059 ceph-mon[97450]: pgmap v7318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:49.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:49.955945+0000) 2022-09-08T01:25:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:49.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:49.892696+0000) 2022-09-08T01:25:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:49 smithi055 ceph-mon[97033]: pgmap v7318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:50.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:50.956099+0000) 2022-09-08T01:25:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:50.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:50.892924+0000) 2022-09-08T01:25:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:51 smithi059 ceph-mon[97450]: pgmap v7319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:25:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:51.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:51.893095+0000) 2022-09-08T01:25:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:51.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:51.956307+0000) 2022-09-08T01:25:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:51 smithi055 ceph-mon[97033]: pgmap v7319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:25:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:52.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:52.893286+0000) 2022-09-08T01:25:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:52.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:52.956497+0000) 2022-09-08T01:25:53.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:25:53.306+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:25:54.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:53 smithi059 ceph-mon[97450]: pgmap v7320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:53.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:53.956697+0000) 2022-09-08T01:25:54.162 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:53.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:53.893464+0000) 2022-09-08T01:25:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:53 smithi055 ceph-mon[97033]: pgmap v7320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:55.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:54.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:54.956823+0000) 2022-09-08T01:25:55.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:54.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:54.893691+0000) 2022-09-08T01:25:56.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:55 smithi059 ceph-mon[97450]: pgmap v7321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:55.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:55.956992+0000) 2022-09-08T01:25:56.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:55.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:55.893933+0000) 2022-09-08T01:25:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:55 smithi055 ceph-mon[97033]: pgmap v7321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:25:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:56.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:56.957211+0000) 2022-09-08T01:25:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:56.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:56.894171+0000) 2022-09-08T01:25:58.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:57 smithi059 ceph-mon[97450]: pgmap v7322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:25:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:57.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:57.957497+0000) 2022-09-08T01:25:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:57.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:57.894404+0000) 2022-09-08T01:25:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:57 smithi055 ceph-mon[97033]: pgmap v7322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:25:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:58.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:58.957753+0000) 2022-09-08T01:25:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:58.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:58.894623+0000) 2022-09-08T01:26:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:25:59 smithi059 ceph-mon[97450]: pgmap v7323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:00.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:25:59.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:59.958035+0000) 2022-09-08T01:26:00.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:25:59.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:25:59.894811+0000) 2022-09-08T01:26:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:25:59 smithi055 ceph-mon[97033]: pgmap v7323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:00.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:00.958244+0000) 2022-09-08T01:26:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:00.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:00.894941+0000) 2022-09-08T01:26:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:01 smithi059 ceph-mon[97450]: pgmap v7324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:26:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:01.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:01.958444+0000) 2022-09-08T01:26:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:01.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:01.895180+0000) 2022-09-08T01:26:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:01 smithi055 ceph-mon[97033]: pgmap v7324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:26:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:02.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:02.958684+0000) 2022-09-08T01:26:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:02.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:02.895428+0000) 2022-09-08T01:26:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:03 smithi059 ceph-mon[97450]: pgmap v7325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:03.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:03.958894+0000) 2022-09-08T01:26:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:03.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:03.895607+0000) 2022-09-08T01:26:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:03 smithi055 ceph-mon[97033]: pgmap v7325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:04.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:04.959027+0000) 2022-09-08T01:26:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:04.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:04.895840+0000) 2022-09-08T01:26:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:05 smithi059 ceph-mon[97450]: pgmap v7326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:26:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:26:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:05.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:05.959257+0000) 2022-09-08T01:26:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:05.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:05.896060+0000) 2022-09-08T01:26:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:05 smithi055 ceph-mon[97033]: pgmap v7326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:26:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:26:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:06.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:06.959431+0000) 2022-09-08T01:26:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:06.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:06.896273+0000) 2022-09-08T01:26:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:07 smithi059 ceph-mon[97450]: pgmap v7327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:26:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:07.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:07.959714+0000) 2022-09-08T01:26:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:07.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:07.896411+0000) 2022-09-08T01:26:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:07 smithi055 ceph-mon[97033]: pgmap v7327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:26:08.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:26:08.313+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:26:09.163 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:08.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:08.959909+0000) 2022-09-08T01:26:09.164 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:08.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:08.896560+0000) 2022-09-08T01:26:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:09.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:09.960113+0000) 2022-09-08T01:26:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:09.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:09.896800+0000) 2022-09-08T01:26:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:09 smithi055 ceph-mon[97033]: pgmap v7328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:26:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:09 smithi055 ceph-mon[97033]: osdmap e541: 12 total, 11 up, 11 in 2022-09-08T01:26:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:09 smithi059 ceph-mon[97450]: pgmap v7328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:26:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:09 smithi059 ceph-mon[97450]: osdmap e541: 12 total, 11 up, 11 in 2022-09-08T01:26:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:10.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:10.960333+0000) 2022-09-08T01:26:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:10.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:10.896981+0000) 2022-09-08T01:26:12.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:11.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:11.960506+0000) 2022-09-08T01:26:12.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:11.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:11.897127+0000) 2022-09-08T01:26:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:11 smithi055 ceph-mon[97033]: pgmap v7330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:26:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:11 smithi059 ceph-mon[97450]: pgmap v7330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:26:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:12.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:12.960743+0000) 2022-09-08T01:26:13.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:12.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:12.897388+0000) 2022-09-08T01:26:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:13.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:13.960947+0000) 2022-09-08T01:26:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:13.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:13.897674+0000) 2022-09-08T01:26:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:13 smithi055 ceph-mon[97033]: pgmap v7331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:26:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:13 smithi059 ceph-mon[97450]: pgmap v7331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:26:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:14.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:14.961153+0000) 2022-09-08T01:26:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:14.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:14.897822+0000) 2022-09-08T01:26:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:15.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:15.898085+0000) 2022-09-08T01:26:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:15.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:15.961396+0000) 2022-09-08T01:26:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:15 smithi055 ceph-mon[97033]: pgmap v7332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:26:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:15 smithi059 ceph-mon[97450]: pgmap v7332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:26:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:16.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:16.961632+0000) 2022-09-08T01:26:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:16.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:16.898301+0000) 2022-09-08T01:26:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:17.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:17.961870+0000) 2022-09-08T01:26:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:17.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:17.898507+0000) 2022-09-08T01:26:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:17 smithi055 ceph-mon[97033]: pgmap v7333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:26:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:17 smithi059 ceph-mon[97450]: pgmap v7333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:26:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:18.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:18.962006+0000) 2022-09-08T01:26:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:18.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:18.898767+0000) 2022-09-08T01:26:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:19.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:19.962179+0000) 2022-09-08T01:26:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:19.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:19.898960+0000) 2022-09-08T01:26:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:19 smithi055 ceph-mon[97033]: pgmap v7334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:26:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:19 smithi059 ceph-mon[97450]: pgmap v7334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:26:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:26:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:26:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:26:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:26:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:26:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:26:21.168 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:20.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:20.962389+0000) 2022-09-08T01:26:21.169 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:20.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:20.899172+0000) 2022-09-08T01:26:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:21.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:21.962601+0000) 2022-09-08T01:26:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:21.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:21.899391+0000) 2022-09-08T01:26:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:22 smithi055 ceph-mon[97033]: pgmap v7335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 177 B/s wr, 8 op/s 2022-09-08T01:26:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:26:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:26:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:26:22.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:22 smithi055 ceph-mon[97033]: osdmap e542: 12 total, 11 up, 11 in 2022-09-08T01:26:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:22 smithi059 ceph-mon[97450]: pgmap v7335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.1 KiB/s rd, 177 B/s wr, 8 op/s 2022-09-08T01:26:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:26:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:26:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:26:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:22 smithi059 ceph-mon[97450]: osdmap e542: 12 total, 11 up, 11 in 2022-09-08T01:26:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:22.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:22.962807+0000) 2022-09-08T01:26:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:22.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:22.899592+0000) 2022-09-08T01:26:23.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:26:23.321+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:26:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:23.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:23.962969+0000) 2022-09-08T01:26:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:23.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:23.899825+0000) 2022-09-08T01:26:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:24 smithi055 ceph-mon[97033]: pgmap v7337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:26:24.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:24 smithi059 ceph-mon[97450]: pgmap v7337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:26:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:24.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:24.963170+0000) 2022-09-08T01:26:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:24.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:24.899999+0000) 2022-09-08T01:26:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:25.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:25.963339+0000) 2022-09-08T01:26:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:25.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:25.900178+0000) 2022-09-08T01:26:26.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:26 smithi055 ceph-mon[97033]: pgmap v7338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:26:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:26 smithi059 ceph-mon[97450]: pgmap v7338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:26:27.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:26.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:26.963562+0000) 2022-09-08T01:26:27.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:26.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:26.900353+0000) 2022-09-08T01:26:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:27.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:27.900499+0000) 2022-09-08T01:26:28.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:27.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:27.963853+0000) 2022-09-08T01:26:28.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:28 smithi055 ceph-mon[97033]: pgmap v7339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:26:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:28 smithi059 ceph-mon[97450]: pgmap v7339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:26:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:28.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:28.900659+0000) 2022-09-08T01:26:29.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:28.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:28.964121+0000) 2022-09-08T01:26:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:29.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:29.900844+0000) 2022-09-08T01:26:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:29.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:29.964387+0000) 2022-09-08T01:26:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:30 smithi055 ceph-mon[97033]: pgmap v7340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:26:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:30 smithi059 ceph-mon[97450]: pgmap v7340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:26:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:30.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:30.964624+0000) 2022-09-08T01:26:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:30.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:30.901013+0000) 2022-09-08T01:26:32.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:31.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:31.964872+0000) 2022-09-08T01:26:32.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:31.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:31.901177+0000) 2022-09-08T01:26:32.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:32 smithi055 ceph-mon[97033]: pgmap v7341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:26:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:32 smithi059 ceph-mon[97450]: pgmap v7341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:26:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:32.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:32.965147+0000) 2022-09-08T01:26:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:32.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:32.901379+0000) 2022-09-08T01:26:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:33.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:33.965344+0000) 2022-09-08T01:26:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:33.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:33.901586+0000) 2022-09-08T01:26:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:34 smithi055 ceph-mon[97033]: pgmap v7342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:26:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:34 smithi059 ceph-mon[97450]: pgmap v7342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:26:35.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:34.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:34.965594+0000) 2022-09-08T01:26:35.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:34.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:34.901837+0000) 2022-09-08T01:26:36.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:35.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:35.965834+0000) 2022-09-08T01:26:36.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:35.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:35.902102+0000) 2022-09-08T01:26:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:36 smithi055 ceph-mon[97033]: pgmap v7343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:26:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:36 smithi059 ceph-mon[97450]: pgmap v7343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:26:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:36.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:36.966100+0000) 2022-09-08T01:26:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:36.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:36.902382+0000) 2022-09-08T01:26:38.329 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:37.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:37.966343+0000) 2022-09-08T01:26:38.333 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:37.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:37.902585+0000) 2022-09-08T01:26:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:38 smithi055 ceph-mon[97033]: pgmap v7344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:26:38.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:38 smithi059 ceph-mon[97450]: pgmap v7344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:26:38.658 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:26:38.326+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:26:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:38.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:38.966547+0000) 2022-09-08T01:26:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:38.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:38.902820+0000) 2022-09-08T01:26:40.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:39.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:39.966814+0000) 2022-09-08T01:26:40.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:39.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:39.902992+0000) 2022-09-08T01:26:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:40 smithi055 ceph-mon[97033]: pgmap v7345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:26:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:40 smithi059 ceph-mon[97450]: pgmap v7345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:26:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:40.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:40.967100+0000) 2022-09-08T01:26:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:40.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:40.903256+0000) 2022-09-08T01:26:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:41.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:41.967332+0000) 2022-09-08T01:26:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:41.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:41.903493+0000) 2022-09-08T01:26:42.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:42 smithi055 ceph-mon[97033]: pgmap v7346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:26:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:42 smithi059 ceph-mon[97450]: pgmap v7346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:26:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:42.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:42.967544+0000) 2022-09-08T01:26:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:42.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:42.903699+0000) 2022-09-08T01:26:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:43.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:43.967782+0000) 2022-09-08T01:26:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:43.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:43.903882+0000) 2022-09-08T01:26:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:44 smithi055 ceph-mon[97033]: pgmap v7347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:44 smithi059 ceph-mon[97450]: pgmap v7347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:44.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:44.967982+0000) 2022-09-08T01:26:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:44.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:44.904133+0000) 2022-09-08T01:26:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:45.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:45.968242+0000) 2022-09-08T01:26:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:45.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:45.904345+0000) 2022-09-08T01:26:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:46 smithi055 ceph-mon[97033]: pgmap v7348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:46 smithi059 ceph-mon[97450]: pgmap v7348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:46.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:46.968436+0000) 2022-09-08T01:26:47.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:46.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:46.904522+0000) 2022-09-08T01:26:48.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:47.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:47.968586+0000) 2022-09-08T01:26:48.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:47.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:47.904719+0000) 2022-09-08T01:26:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:48 smithi055 ceph-mon[97033]: pgmap v7349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:26:48.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:48 smithi059 ceph-mon[97450]: pgmap v7349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:26:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:48.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:48.968791+0000) 2022-09-08T01:26:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:48.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:48.904877+0000) 2022-09-08T01:26:50.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:49.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:49.968933+0000) 2022-09-08T01:26:50.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:49.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:49.905099+0000) 2022-09-08T01:26:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:50 smithi055 ceph-mon[97033]: pgmap v7350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:50 smithi059 ceph-mon[97450]: pgmap v7350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:50.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:50.969176+0000) 2022-09-08T01:26:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:50.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:50.905298+0000) 2022-09-08T01:26:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:51.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:51.969366+0000) 2022-09-08T01:26:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:51.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:51.905527+0000) 2022-09-08T01:26:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:52 smithi055 ceph-mon[97033]: pgmap v7351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:26:52.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:52 smithi059 ceph-mon[97450]: pgmap v7351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:26:53.331 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:52.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:52.969548+0000) 2022-09-08T01:26:53.332 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:52.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:52.905770+0000) 2022-09-08T01:26:53.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:26:53.330+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:26:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:53.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:53.969792+0000) 2022-09-08T01:26:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:53.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:53.905929+0000) 2022-09-08T01:26:54.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:54 smithi055 ceph-mon[97033]: pgmap v7352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:54 smithi059 ceph-mon[97450]: pgmap v7352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:54.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:54.906167+0000) 2022-09-08T01:26:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:54.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:54.970024+0000) 2022-09-08T01:26:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:55.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:55.970174+0000) 2022-09-08T01:26:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:55.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:55.906381+0000) 2022-09-08T01:26:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:56 smithi055 ceph-mon[97033]: pgmap v7353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:56 smithi059 ceph-mon[97450]: pgmap v7353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:26:57.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:56.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:56.906616+0000) 2022-09-08T01:26:57.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:56.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:56.970332+0000) 2022-09-08T01:26:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:57.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:57.970525+0000) 2022-09-08T01:26:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:57.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:57.906834+0000) 2022-09-08T01:26:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:26:58 smithi055 ceph-mon[97033]: pgmap v7354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:26:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:26:58 smithi059 ceph-mon[97450]: pgmap v7354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:26:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:58.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:58.970754+0000) 2022-09-08T01:26:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:58.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:58.906991+0000) 2022-09-08T01:27:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:26:59.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:59.970977+0000) 2022-09-08T01:27:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:26:59.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:26:59.907248+0000) 2022-09-08T01:27:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:00 smithi055 ceph-mon[97033]: pgmap v7355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:00 smithi059 ceph-mon[97450]: pgmap v7355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:00.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:00.907513+0000) 2022-09-08T01:27:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:00.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:00.971167+0000) 2022-09-08T01:27:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:01.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:01.907768+0000) 2022-09-08T01:27:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:01.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:01.971408+0000) 2022-09-08T01:27:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:02 smithi055 ceph-mon[97033]: pgmap v7356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:27:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:02 smithi059 ceph-mon[97450]: pgmap v7356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:27:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:02.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:02.971593+0000) 2022-09-08T01:27:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:02.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:02.907966+0000) 2022-09-08T01:27:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:03.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:03.908238+0000) 2022-09-08T01:27:04.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:03.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:03.971763+0000) 2022-09-08T01:27:04.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:04 smithi055 ceph-mon[97033]: pgmap v7357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:04.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:04 smithi059 ceph-mon[97450]: pgmap v7357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:05.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:04.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:04.908465+0000) 2022-09-08T01:27:05.408 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:04.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:04.971967+0000) 2022-09-08T01:27:05.490 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:27:05.490 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:27:05.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:27:05.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:27:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:05.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:05.908755+0000) 2022-09-08T01:27:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:05.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:05.972146+0000) 2022-09-08T01:27:06.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:06 smithi055 ceph-mon[97033]: pgmap v7358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:06 smithi059 ceph-mon[97450]: pgmap v7358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:06.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:06.972352+0000) 2022-09-08T01:27:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:06.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:06.909012+0000) 2022-09-08T01:27:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:07 smithi055 ceph-mon[97033]: pgmap v7359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:27:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:07 smithi059 ceph-mon[97450]: pgmap v7359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:27:08.342 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:07.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:07.972531+0000) 2022-09-08T01:27:08.343 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:07.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:07.909224+0000) 2022-09-08T01:27:08.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:27:08.341+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:27:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:08.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:08.972710+0000) 2022-09-08T01:27:09.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:08.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:08.909410+0000) 2022-09-08T01:27:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:09 smithi055 ceph-mon[97033]: pgmap v7360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:27:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:09 smithi055 ceph-mon[97033]: osdmap e543: 12 total, 11 up, 11 in 2022-09-08T01:27:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:09 smithi059 ceph-mon[97450]: pgmap v7360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:27:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:09 smithi059 ceph-mon[97450]: osdmap e543: 12 total, 11 up, 11 in 2022-09-08T01:27:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:09.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:09.972926+0000) 2022-09-08T01:27:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:09.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:09.909701+0000) 2022-09-08T01:27:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:10.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:10.973168+0000) 2022-09-08T01:27:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:10.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:10.909962+0000) 2022-09-08T01:27:11.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:11 smithi055 ceph-mon[97033]: pgmap v7362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:27:11.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:11 smithi055 ceph-mon[97033]: osdmap e544: 12 total, 11 up, 11 in 2022-09-08T01:27:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:11 smithi059 ceph-mon[97450]: pgmap v7362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:27:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:11 smithi059 ceph-mon[97450]: osdmap e544: 12 total, 11 up, 11 in 2022-09-08T01:27:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:11.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:11.973407+0000) 2022-09-08T01:27:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:11.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:11.910224+0000) 2022-09-08T01:27:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:12.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:12.973679+0000) 2022-09-08T01:27:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:12.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:12.910495+0000) 2022-09-08T01:27:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:13 smithi055 ceph-mon[97033]: pgmap v7364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T01:27:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:13 smithi059 ceph-mon[97450]: pgmap v7364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T01:27:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:13.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:13.973869+0000) 2022-09-08T01:27:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:13.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:13.910771+0000) 2022-09-08T01:27:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:14.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:14.974045+0000) 2022-09-08T01:27:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:14.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:14.910994+0000) 2022-09-08T01:27:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:15 smithi055 ceph-mon[97033]: pgmap v7365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:27:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:15 smithi059 ceph-mon[97450]: pgmap v7365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:27:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:15.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:15.974227+0000) 2022-09-08T01:27:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:15.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:15.911248+0000) 2022-09-08T01:27:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:16.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:16.974407+0000) 2022-09-08T01:27:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:16.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:16.911517+0000) 2022-09-08T01:27:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:17 smithi059 ceph-mon[97450]: pgmap v7366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:27:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:17 smithi055 ceph-mon[97033]: pgmap v7366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:27:18.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:17.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:17.974596+0000) 2022-09-08T01:27:18.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:17.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:17.911701+0000) 2022-09-08T01:27:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:18.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:18.974866+0000) 2022-09-08T01:27:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:18.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:18.911948+0000) 2022-09-08T01:27:20.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:19 smithi059 ceph-mon[97450]: pgmap v7367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2022-09-08T01:27:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:19 smithi055 ceph-mon[97033]: pgmap v7367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2022-09-08T01:27:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:19.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:19.975175+0000) 2022-09-08T01:27:20.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:19.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:19.912233+0000) 2022-09-08T01:27:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:20.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:20.975491+0000) 2022-09-08T01:27:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:20.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:20.912503+0000) 2022-09-08T01:27:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:21 smithi059 ceph-mon[97450]: pgmap v7368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:27:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:27:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:21 smithi055 ceph-mon[97033]: pgmap v7368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:27:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:27:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:21.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:21.975738+0000) 2022-09-08T01:27:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:21.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:21.912796+0000) 2022-09-08T01:27:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:27:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:27:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:27:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:27:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:27:23.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:27:23.358 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:22.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:22.975969+0000) 2022-09-08T01:27:23.359 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:22.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:22.912967+0000) 2022-09-08T01:27:23.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:27:23.356+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:27:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:23 smithi059 ceph-mon[97450]: pgmap v7369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:27:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:23 smithi055 ceph-mon[97033]: pgmap v7369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:27:24.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:23.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:23.976179+0000) 2022-09-08T01:27:24.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:23.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:23.913232+0000) 2022-09-08T01:27:25.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:24.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:24.976471+0000) 2022-09-08T01:27:25.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:24.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:24.913451+0000) 2022-09-08T01:27:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:25 smithi059 ceph-mon[97450]: pgmap v7370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:27:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:25 smithi055 ceph-mon[97033]: pgmap v7370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:27:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:25.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:25.976745+0000) 2022-09-08T01:27:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:25.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:25.913695+0000) 2022-09-08T01:27:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:26.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:26.976904+0000) 2022-09-08T01:27:27.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:26.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:26.913917+0000) 2022-09-08T01:27:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:27 smithi059 ceph-mon[97450]: pgmap v7371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:27:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:27 smithi055 ceph-mon[97033]: pgmap v7371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:27:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:27.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:27.977063+0000) 2022-09-08T01:27:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:27.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:27.914147+0000) 2022-09-08T01:27:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:28.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:28.977215+0000) 2022-09-08T01:27:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:28.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:28.914354+0000) 2022-09-08T01:27:30.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:29 smithi059 ceph-mon[97450]: pgmap v7372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:29 smithi055 ceph-mon[97033]: pgmap v7372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:29.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:29.977448+0000) 2022-09-08T01:27:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:29.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:29.914590+0000) 2022-09-08T01:27:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:30.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:30.977616+0000) 2022-09-08T01:27:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:30.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:30.914771+0000) 2022-09-08T01:27:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:31 smithi059 ceph-mon[97450]: pgmap v7373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:27:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:31 smithi055 ceph-mon[97033]: pgmap v7373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:27:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:31.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:31.977876+0000) 2022-09-08T01:27:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:31.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:31.915012+0000) 2022-09-08T01:27:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:32.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:32.978037+0000) 2022-09-08T01:27:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:32.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:32.915263+0000) 2022-09-08T01:27:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:33 smithi059 ceph-mon[97450]: pgmap v7374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:33 smithi055 ceph-mon[97033]: pgmap v7374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:33.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:33.978288+0000) 2022-09-08T01:27:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:33.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:33.915501+0000) 2022-09-08T01:27:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:34.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:34.978482+0000) 2022-09-08T01:27:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:34.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:34.915765+0000) 2022-09-08T01:27:36.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:35 smithi059 ceph-mon[97450]: pgmap v7375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:35 smithi055 ceph-mon[97033]: pgmap v7375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:35.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:35.978739+0000) 2022-09-08T01:27:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:35.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:35.915994+0000) 2022-09-08T01:27:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:36.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:36.978987+0000) 2022-09-08T01:27:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:36.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:36.916179+0000) 2022-09-08T01:27:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:37 smithi059 ceph-mon[97450]: pgmap v7376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:27:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:37 smithi055 ceph-mon[97033]: pgmap v7376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:27:38.363 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:37.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:37.979135+0000) 2022-09-08T01:27:38.364 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:37.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:37.916364+0000) 2022-09-08T01:27:38.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:27:38.361+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:27:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:38.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:38.979335+0000) 2022-09-08T01:27:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:38.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:38.916506+0000) 2022-09-08T01:27:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:39 smithi059 ceph-mon[97450]: pgmap v7377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:39 smithi055 ceph-mon[97033]: pgmap v7377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:39.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:39.979585+0000) 2022-09-08T01:27:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:39.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:39.916702+0000) 2022-09-08T01:27:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:40.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:40.979787+0000) 2022-09-08T01:27:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:40.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:40.916908+0000) 2022-09-08T01:27:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:41 smithi059 ceph-mon[97450]: pgmap v7378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:27:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:41 smithi055 ceph-mon[97033]: pgmap v7378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:27:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:41.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:41.980031+0000) 2022-09-08T01:27:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:41.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:41.917132+0000) 2022-09-08T01:27:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:42.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:42.980256+0000) 2022-09-08T01:27:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:42.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:42.917417+0000) 2022-09-08T01:27:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:43 smithi059 ceph-mon[97450]: pgmap v7379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:43 smithi055 ceph-mon[97033]: pgmap v7379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:43.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:43.980400+0000) 2022-09-08T01:27:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:43.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:43.917664+0000) 2022-09-08T01:27:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:44.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:44.980528+0000) 2022-09-08T01:27:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:44.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:44.917899+0000) 2022-09-08T01:27:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:45 smithi059 ceph-mon[97450]: pgmap v7380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:45 smithi055 ceph-mon[97033]: pgmap v7380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:45.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:45.980704+0000) 2022-09-08T01:27:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:45.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:45.918143+0000) 2022-09-08T01:27:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:46.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:46.980903+0000) 2022-09-08T01:27:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:46.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:46.918410+0000) 2022-09-08T01:27:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:47 smithi055 ceph-mon[97033]: pgmap v7381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:27:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:47 smithi059 ceph-mon[97450]: pgmap v7381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:27:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:47.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:47.981074+0000) 2022-09-08T01:27:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:47.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:47.918599+0000) 2022-09-08T01:27:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:48.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:48.981259+0000) 2022-09-08T01:27:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:48.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:48.918790+0000) 2022-09-08T01:27:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:49 smithi055 ceph-mon[97033]: pgmap v7382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:50.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:49 smithi059 ceph-mon[97450]: pgmap v7382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:49.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:49.919035+0000) 2022-09-08T01:27:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:49.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:49.981386+0000) 2022-09-08T01:27:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:50.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:50.981591+0000) 2022-09-08T01:27:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:50.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:50.919311+0000) 2022-09-08T01:27:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:51 smithi055 ceph-mon[97033]: pgmap v7383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:27:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:51 smithi059 ceph-mon[97450]: pgmap v7383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:27:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:51.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:51.981822+0000) 2022-09-08T01:27:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:51.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:51.919585+0000) 2022-09-08T01:27:53.369 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:52.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:52.981980+0000) 2022-09-08T01:27:53.370 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:52.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:52.919889+0000) 2022-09-08T01:27:53.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:27:53.367+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:27:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:53 smithi055 ceph-mon[97033]: pgmap v7384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:53 smithi059 ceph-mon[97450]: pgmap v7384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:53.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:53.982195+0000) 2022-09-08T01:27:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:53.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:53.920121+0000) 2022-09-08T01:27:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:54.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:54.920366+0000) 2022-09-08T01:27:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:54.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:54.982408+0000) 2022-09-08T01:27:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:55 smithi055 ceph-mon[97033]: pgmap v7385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:55 smithi059 ceph-mon[97450]: pgmap v7385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:27:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:55.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:55.920628+0000) 2022-09-08T01:27:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:55.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:55.982613+0000) 2022-09-08T01:27:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:56.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:56.982857+0000) 2022-09-08T01:27:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:56.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:56.920931+0000) 2022-09-08T01:27:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:57 smithi055 ceph-mon[97033]: pgmap v7386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:27:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:57 smithi059 ceph-mon[97450]: pgmap v7386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:27:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:57.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:57.983023+0000) 2022-09-08T01:27:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:57.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:57.921070+0000) 2022-09-08T01:27:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:58.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:58.983210+0000) 2022-09-08T01:27:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:58.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:58.921291+0000) 2022-09-08T01:28:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:27:59 smithi055 ceph-mon[97033]: pgmap v7387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:28:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:27:59 smithi059 ceph-mon[97450]: pgmap v7387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:28:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:27:59.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:59.983444+0000) 2022-09-08T01:28:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:27:59.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:27:59.921587+0000) 2022-09-08T01:28:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:00.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:00.921906+0000) 2022-09-08T01:28:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:00.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:00.983678+0000) 2022-09-08T01:28:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:01 smithi055 ceph-mon[97033]: pgmap v7388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:28:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:01 smithi059 ceph-mon[97450]: pgmap v7388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:28:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:01.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:01.983919+0000) 2022-09-08T01:28:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:01.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:01.922192+0000) 2022-09-08T01:28:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:02.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:02.984099+0000) 2022-09-08T01:28:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:02.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:02.922451+0000) 2022-09-08T01:28:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:03 smithi055 ceph-mon[97033]: pgmap v7389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:28:04.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:03 smithi059 ceph-mon[97450]: pgmap v7389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:28:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:03.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:03.984314+0000) 2022-09-08T01:28:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:03.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:03.922664+0000) 2022-09-08T01:28:05.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:04.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:04.984540+0000) 2022-09-08T01:28:05.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:04.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:04.922864+0000) 2022-09-08T01:28:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:05 smithi055 ceph-mon[97033]: pgmap v7390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:28:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:28:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:28:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:05 smithi059 ceph-mon[97450]: pgmap v7390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:28:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:28:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:28:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:05.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:05.984775+0000) 2022-09-08T01:28:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:05.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:05.923173+0000) 2022-09-08T01:28:07.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:06.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:06.984978+0000) 2022-09-08T01:28:07.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:06.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:06.923422+0000) 2022-09-08T01:28:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:07 smithi055 ceph-mon[97033]: pgmap v7391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:28:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:07 smithi059 ceph-mon[97450]: pgmap v7391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:28:08.388 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:07.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:07.985196+0000) 2022-09-08T01:28:08.390 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:07.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:07.923684+0000) 2022-09-08T01:28:08.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:28:08.383+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:28:09.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:08.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:08.985354+0000) 2022-09-08T01:28:09.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:08.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:08.923943+0000) 2022-09-08T01:28:10.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:09 smithi055 ceph-mon[97033]: pgmap v7392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:28:10.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:09 smithi055 ceph-mon[97033]: osdmap e545: 12 total, 11 up, 11 in 2022-09-08T01:28:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:09 smithi059 ceph-mon[97450]: pgmap v7392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:28:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:09 smithi059 ceph-mon[97450]: osdmap e545: 12 total, 11 up, 11 in 2022-09-08T01:28:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:09.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:09.924197+0000) 2022-09-08T01:28:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:09.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:09.985589+0000) 2022-09-08T01:28:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:10.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:10.924391+0000) 2022-09-08T01:28:11.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:10.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:10.985784+0000) 2022-09-08T01:28:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:11 smithi055 ceph-mon[97033]: pgmap v7394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:28:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:11 smithi055 ceph-mon[97033]: osdmap e546: 12 total, 11 up, 11 in 2022-09-08T01:28:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:11 smithi059 ceph-mon[97450]: pgmap v7394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:28:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:11 smithi059 ceph-mon[97450]: osdmap e546: 12 total, 11 up, 11 in 2022-09-08T01:28:12.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:11.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:11.924700+0000) 2022-09-08T01:28:12.408 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:11.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:11.986030+0000) 2022-09-08T01:28:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:12.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:12.986220+0000) 2022-09-08T01:28:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:12.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:12.924855+0000) 2022-09-08T01:28:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:13 smithi055 ceph-mon[97033]: pgmap v7396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:28:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:13 smithi059 ceph-mon[97450]: pgmap v7396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:28:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:13.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:13.986420+0000) 2022-09-08T01:28:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:13.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:13.925087+0000) 2022-09-08T01:28:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:14.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:14.986648+0000) 2022-09-08T01:28:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:14.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:14.925362+0000) 2022-09-08T01:28:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:15 smithi055 ceph-mon[97033]: pgmap v7397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:28:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:15 smithi059 ceph-mon[97450]: pgmap v7397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:28:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:15.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:15.925604+0000) 2022-09-08T01:28:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:15.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:15.986801+0000) 2022-09-08T01:28:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:16.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:16.987019+0000) 2022-09-08T01:28:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:16.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:16.925869+0000) 2022-09-08T01:28:18.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:17 smithi055 ceph-mon[97033]: pgmap v7398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:28:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:17 smithi059 ceph-mon[97450]: pgmap v7398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:28:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:17.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:17.987192+0000) 2022-09-08T01:28:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:17.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:17.926011+0000) 2022-09-08T01:28:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:18.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:18.926233+0000) 2022-09-08T01:28:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:18.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:18.987376+0000) 2022-09-08T01:28:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:19 smithi055 ceph-mon[97033]: pgmap v7399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2022-09-08T01:28:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:19 smithi059 ceph-mon[97450]: pgmap v7399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2022-09-08T01:28:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:19.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:19.926433+0000) 2022-09-08T01:28:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:19.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:19.987622+0000) 2022-09-08T01:28:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:20.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:20.926679+0000) 2022-09-08T01:28:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:20.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:20.987854+0000) 2022-09-08T01:28:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:21 smithi055 ceph-mon[97033]: pgmap v7400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:28:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:21 smithi059 ceph-mon[97450]: pgmap v7400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:28:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:21.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:21.988127+0000) 2022-09-08T01:28:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:21.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:21.926861+0000) 2022-09-08T01:28:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:28:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:28:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:28:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:28:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:28:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:28:23.399 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:22.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:22.988362+0000) 2022-09-08T01:28:23.400 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:22.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:22.927024+0000) 2022-09-08T01:28:23.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:28:23.397+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:28:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:23 smithi055 ceph-mon[97033]: pgmap v7401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:28:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:28:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:23 smithi059 ceph-mon[97450]: pgmap v7401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:28:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:28:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:23.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:23.927264+0000) 2022-09-08T01:28:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:23.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:23.988537+0000) 2022-09-08T01:28:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:24.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:24.988772+0000) 2022-09-08T01:28:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:24.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:24.927500+0000) 2022-09-08T01:28:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:25 smithi055 ceph-mon[97033]: pgmap v7402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:28:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:25 smithi059 ceph-mon[97450]: pgmap v7402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:28:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:25.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:25.927746+0000) 2022-09-08T01:28:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:25.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:25.989013+0000) 2022-09-08T01:28:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:26.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:26.927999+0000) 2022-09-08T01:28:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:26.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:26.989178+0000) 2022-09-08T01:28:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:27 smithi055 ceph-mon[97033]: pgmap v7403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:28:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:27 smithi059 ceph-mon[97450]: pgmap v7403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:28:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:27.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:27.928152+0000) 2022-09-08T01:28:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:27.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:27.989367+0000) 2022-09-08T01:28:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:28.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:28.989607+0000) 2022-09-08T01:28:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:28.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:28.928366+0000) 2022-09-08T01:28:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:29 smithi055 ceph-mon[97033]: pgmap v7404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:29 smithi059 ceph-mon[97450]: pgmap v7404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:29.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:29.989888+0000) 2022-09-08T01:28:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:29.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:29.928608+0000) 2022-09-08T01:28:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:30.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:30.990124+0000) 2022-09-08T01:28:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:30.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:30.928872+0000) 2022-09-08T01:28:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:31 smithi055 ceph-mon[97033]: pgmap v7405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:28:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:31 smithi059 ceph-mon[97450]: pgmap v7405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:28:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:31.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:31.990346+0000) 2022-09-08T01:28:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:31.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:31.929121+0000) 2022-09-08T01:28:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:32.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:32.990588+0000) 2022-09-08T01:28:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:32.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:32.929303+0000) 2022-09-08T01:28:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:33 smithi055 ceph-mon[97033]: pgmap v7406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:33 smithi059 ceph-mon[97450]: pgmap v7406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:33.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:33.990834+0000) 2022-09-08T01:28:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:33.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:33.929533+0000) 2022-09-08T01:28:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:34.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:34.991001+0000) 2022-09-08T01:28:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:34.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:34.929700+0000) 2022-09-08T01:28:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:35 smithi055 ceph-mon[97033]: pgmap v7407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:35 smithi059 ceph-mon[97450]: pgmap v7407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:35.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:35.991168+0000) 2022-09-08T01:28:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:35.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:35.929949+0000) 2022-09-08T01:28:37.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:36.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:36.991392+0000) 2022-09-08T01:28:37.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:36.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:36.930190+0000) 2022-09-08T01:28:38.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:37 smithi055 ceph-mon[97033]: pgmap v7408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:28:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:37 smithi059 ceph-mon[97450]: pgmap v7408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:28:38.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:37.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:37.991536+0000) 2022-09-08T01:28:38.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:37.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:37.930387+0000) 2022-09-08T01:28:38.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:28:38.413+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:28:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:38.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:38.991743+0000) 2022-09-08T01:28:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:38.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:38.930665+0000) 2022-09-08T01:28:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:39 smithi055 ceph-mon[97033]: pgmap v7409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:39 smithi059 ceph-mon[97450]: pgmap v7409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:39.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:39.991980+0000) 2022-09-08T01:28:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:39.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:39.930908+0000) 2022-09-08T01:28:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:40.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:40.992241+0000) 2022-09-08T01:28:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:40.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:40.931042+0000) 2022-09-08T01:28:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:41 smithi055 ceph-mon[97033]: pgmap v7410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:28:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:41 smithi059 ceph-mon[97450]: pgmap v7410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:28:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:41.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:41.992487+0000) 2022-09-08T01:28:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:41.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:41.931201+0000) 2022-09-08T01:28:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:42.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:42.992754+0000) 2022-09-08T01:28:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:42.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:42.931413+0000) 2022-09-08T01:28:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:43 smithi055 ceph-mon[97033]: pgmap v7411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:43 smithi059 ceph-mon[97450]: pgmap v7411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:43.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:43.992995+0000) 2022-09-08T01:28:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:43.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:43.931682+0000) 2022-09-08T01:28:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:44.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:44.993175+0000) 2022-09-08T01:28:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:44.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:44.931886+0000) 2022-09-08T01:28:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:45 smithi055 ceph-mon[97033]: pgmap v7412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:45 smithi059 ceph-mon[97450]: pgmap v7412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:45.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:45.993403+0000) 2022-09-08T01:28:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:45.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:45.932058+0000) 2022-09-08T01:28:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:46.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:46.993577+0000) 2022-09-08T01:28:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:46.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:46.932266+0000) 2022-09-08T01:28:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:47 smithi055 ceph-mon[97033]: pgmap v7413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:28:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:47 smithi059 ceph-mon[97450]: pgmap v7413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:28:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:47.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:47.993798+0000) 2022-09-08T01:28:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:47.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:47.932482+0000) 2022-09-08T01:28:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:48.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:48.994032+0000) 2022-09-08T01:28:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:48.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:48.932740+0000) 2022-09-08T01:28:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:49 smithi055 ceph-mon[97033]: pgmap v7414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:49 smithi059 ceph-mon[97450]: pgmap v7414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:49.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:49.994250+0000) 2022-09-08T01:28:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:49.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:49.932924+0000) 2022-09-08T01:28:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:50.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:50.994491+0000) 2022-09-08T01:28:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:50.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:50.933166+0000) 2022-09-08T01:28:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:51 smithi055 ceph-mon[97033]: pgmap v7415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:28:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:51 smithi059 ceph-mon[97450]: pgmap v7415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:28:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:51.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:51.933395+0000) 2022-09-08T01:28:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:51.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:51.994747+0000) 2022-09-08T01:28:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:52.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:52.994982+0000) 2022-09-08T01:28:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:52.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:52.933584+0000) 2022-09-08T01:28:53.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:28:53.416+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:28:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:53 smithi055 ceph-mon[97033]: pgmap v7416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:53 smithi059 ceph-mon[97450]: pgmap v7416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:53.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:53.995209+0000) 2022-09-08T01:28:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:53.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:53.933821+0000) 2022-09-08T01:28:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:54.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:54.995415+0000) 2022-09-08T01:28:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:54.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:54.934043+0000) 2022-09-08T01:28:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:55 smithi055 ceph-mon[97033]: pgmap v7417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:55 smithi059 ceph-mon[97450]: pgmap v7417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:28:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:55.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:55.995674+0000) 2022-09-08T01:28:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:55.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:55.934249+0000) 2022-09-08T01:28:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:56.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:56.995920+0000) 2022-09-08T01:28:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:56.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:56.934432+0000) 2022-09-08T01:28:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:57 smithi055 ceph-mon[97033]: pgmap v7418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:28:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:57 smithi059 ceph-mon[97450]: pgmap v7418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:28:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:57.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:57.996059+0000) 2022-09-08T01:28:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:57.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:57.934711+0000) 2022-09-08T01:28:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:58.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:58.996283+0000) 2022-09-08T01:28:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:58.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:58.934928+0000) 2022-09-08T01:29:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:28:59 smithi055 ceph-mon[97033]: pgmap v7419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:28:59 smithi059 ceph-mon[97450]: pgmap v7419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:28:59.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:59.996426+0000) 2022-09-08T01:29:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:28:59.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:28:59.935160+0000) 2022-09-08T01:29:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:00.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:00.996573+0000) 2022-09-08T01:29:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:00.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:00.935282+0000) 2022-09-08T01:29:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:01 smithi055 ceph-mon[97033]: pgmap v7420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:29:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:01 smithi059 ceph-mon[97450]: pgmap v7420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:29:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:01.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:01.996782+0000) 2022-09-08T01:29:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:01.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:01.935498+0000) 2022-09-08T01:29:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:02.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:02.996977+0000) 2022-09-08T01:29:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:02.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:02.935748+0000) 2022-09-08T01:29:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:03 smithi055 ceph-mon[97033]: pgmap v7421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:03 smithi059 ceph-mon[97450]: pgmap v7421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:03.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:03.997257+0000) 2022-09-08T01:29:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:03.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:03.935918+0000) 2022-09-08T01:29:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:04.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:04.997489+0000) 2022-09-08T01:29:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:04.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:04.936089+0000) 2022-09-08T01:29:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:05 smithi055 ceph-mon[97033]: pgmap v7422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:29:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:29:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:05 smithi059 ceph-mon[97450]: pgmap v7422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:29:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:29:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:05.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:05.997749+0000) 2022-09-08T01:29:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:05.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:05.936316+0000) 2022-09-08T01:29:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:06.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:06.997989+0000) 2022-09-08T01:29:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:06.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:06.936529+0000) 2022-09-08T01:29:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:07 smithi055 ceph-mon[97033]: pgmap v7423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:29:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:07 smithi059 ceph-mon[97450]: pgmap v7423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:29:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:07.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:07.998129+0000) 2022-09-08T01:29:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:07.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:07.936730+0000) 2022-09-08T01:29:08.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:29:08.430+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:29:09.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:08.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:08.998274+0000) 2022-09-08T01:29:09.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:08.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:08.936907+0000) 2022-09-08T01:29:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:09 smithi055 ceph-mon[97033]: pgmap v7424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:29:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:09 smithi055 ceph-mon[97033]: osdmap e547: 12 total, 11 up, 11 in 2022-09-08T01:29:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:09 smithi059 ceph-mon[97450]: pgmap v7424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:29:10.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:09 smithi059 ceph-mon[97450]: osdmap e547: 12 total, 11 up, 11 in 2022-09-08T01:29:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:09.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:09.998535+0000) 2022-09-08T01:29:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:09.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:09.937110+0000) 2022-09-08T01:29:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:10.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:10.998738+0000) 2022-09-08T01:29:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:10.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:10.937275+0000) 2022-09-08T01:29:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:11 smithi055 ceph-mon[97033]: pgmap v7426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:29:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:11 smithi055 ceph-mon[97033]: osdmap e548: 12 total, 11 up, 11 in 2022-09-08T01:29:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:11 smithi059 ceph-mon[97450]: pgmap v7426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:29:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:11 smithi059 ceph-mon[97450]: osdmap e548: 12 total, 11 up, 11 in 2022-09-08T01:29:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:11.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:11.937435+0000) 2022-09-08T01:29:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:11.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:11.998918+0000) 2022-09-08T01:29:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:12.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:12.999127+0000) 2022-09-08T01:29:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:12.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:12.937710+0000) 2022-09-08T01:29:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:13 smithi055 ceph-mon[97033]: pgmap v7428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:29:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:13 smithi059 ceph-mon[97450]: pgmap v7428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:29:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:13.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:13.999356+0000) 2022-09-08T01:29:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:13.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:13.937956+0000) 2022-09-08T01:29:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:14.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:14.999567+0000) 2022-09-08T01:29:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:14.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:14.938118+0000) 2022-09-08T01:29:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:15 smithi059 ceph-mon[97450]: pgmap v7429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:29:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:15.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:15.938341+0000) 2022-09-08T01:29:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:15.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:15.999816+0000) 2022-09-08T01:29:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:15 smithi055 ceph-mon[97033]: pgmap v7429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:29:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:16.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:16.938549+0000) 2022-09-08T01:29:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:16.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:17.000000+0000) 2022-09-08T01:29:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:17 smithi059 ceph-mon[97450]: pgmap v7430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:29:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:17.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:17.938800+0000) 2022-09-08T01:29:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:17.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:18.000199+0000) 2022-09-08T01:29:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:17 smithi055 ceph-mon[97033]: pgmap v7430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:29:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:18.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:18.939018+0000) 2022-09-08T01:29:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:18.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:19.000440+0000) 2022-09-08T01:29:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:19 smithi059 ceph-mon[97450]: pgmap v7431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 218 B/s wr, 6 op/s 2022-09-08T01:29:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:20.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:20.000686+0000) 2022-09-08T01:29:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:19.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:19.939272+0000) 2022-09-08T01:29:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:19 smithi055 ceph-mon[97033]: pgmap v7431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 218 B/s wr, 6 op/s 2022-09-08T01:29:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:20.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:20.939515+0000) 2022-09-08T01:29:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:21.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:21.000825+0000) 2022-09-08T01:29:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:21 smithi059 ceph-mon[97450]: pgmap v7432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:29:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:21.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:21.939750+0000) 2022-09-08T01:29:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:22.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:22.001050+0000) 2022-09-08T01:29:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:22 smithi055 ceph-mon[97033]: pgmap v7432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:29:23.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:29:23.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:29:23.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:29:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:23.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:23.001205+0000) 2022-09-08T01:29:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:22.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:22.939958+0000) 2022-09-08T01:29:23.490 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:29:23.491 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:29:23.491 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:29:23.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:29:23.434+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:29:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:24 smithi059 ceph-mon[97450]: pgmap v7433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:29:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:24.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:24.001403+0000) 2022-09-08T01:29:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:23.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:23.940193+0000) 2022-09-08T01:29:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:24 smithi055 ceph-mon[97033]: pgmap v7433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:29:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:25.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:25.001686+0000) 2022-09-08T01:29:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:24.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:24.940400+0000) 2022-09-08T01:29:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:26 smithi059 ceph-mon[97450]: pgmap v7434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:29:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:26.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:26.001884+0000) 2022-09-08T01:29:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:25.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:25.940546+0000) 2022-09-08T01:29:26.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:26 smithi055 ceph-mon[97033]: pgmap v7434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:29:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:26.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:26.940763+0000) 2022-09-08T01:29:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:27.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:27.002137+0000) 2022-09-08T01:29:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:28 smithi059 ceph-mon[97450]: pgmap v7435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:29:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:27.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:27.940937+0000) 2022-09-08T01:29:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:28.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:28.002350+0000) 2022-09-08T01:29:28.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:28 smithi055 ceph-mon[97033]: pgmap v7435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:29:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:29.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:29.002532+0000) 2022-09-08T01:29:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:28.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:28.941170+0000) 2022-09-08T01:29:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:30 smithi059 ceph-mon[97450]: pgmap v7436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:30.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:30.002754+0000) 2022-09-08T01:29:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:29.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:29.941397+0000) 2022-09-08T01:29:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:30 smithi055 ceph-mon[97033]: pgmap v7436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:31.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:31.003058+0000) 2022-09-08T01:29:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:30.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:30.941521+0000) 2022-09-08T01:29:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:32 smithi059 ceph-mon[97450]: pgmap v7437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:29:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:32.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:32.003227+0000) 2022-09-08T01:29:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:31.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:31.941747+0000) 2022-09-08T01:29:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:32 smithi055 ceph-mon[97033]: pgmap v7437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:29:33.409 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:33.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:33.003456+0000) 2022-09-08T01:29:33.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:32.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:32.941984+0000) 2022-09-08T01:29:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:34 smithi059 ceph-mon[97450]: pgmap v7438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:34.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:34.003727+0000) 2022-09-08T01:29:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:33.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:33.942230+0000) 2022-09-08T01:29:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:34 smithi055 ceph-mon[97033]: pgmap v7438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:35.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:35.003976+0000) 2022-09-08T01:29:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:34.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:34.942501+0000) 2022-09-08T01:29:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:36 smithi059 ceph-mon[97450]: pgmap v7439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:36.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:36.004277+0000) 2022-09-08T01:29:36.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:35.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:35.942729+0000) 2022-09-08T01:29:36.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:36 smithi055 ceph-mon[97033]: pgmap v7439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:37.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:37.004514+0000) 2022-09-08T01:29:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:36.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:36.942957+0000) 2022-09-08T01:29:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:38 smithi059 ceph-mon[97450]: pgmap v7440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:29:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:38.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:38.004765+0000) 2022-09-08T01:29:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:37.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:37.943207+0000) 2022-09-08T01:29:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:38 smithi055 ceph-mon[97033]: pgmap v7440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:29:38.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:29:38.447+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:29:39.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:39.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:39.004991+0000) 2022-09-08T01:29:39.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:38.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:38.943429+0000) 2022-09-08T01:29:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:40 smithi059 ceph-mon[97450]: pgmap v7441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:40.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:40.005187+0000) 2022-09-08T01:29:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:39.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:39.943738+0000) 2022-09-08T01:29:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:40 smithi055 ceph-mon[97033]: pgmap v7441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:41.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:41.005377+0000) 2022-09-08T01:29:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:40.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:40.943941+0000) 2022-09-08T01:29:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:41.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:41.944180+0000) 2022-09-08T01:29:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:42.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:42.005618+0000) 2022-09-08T01:29:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:42 smithi055 ceph-mon[97033]: pgmap v7442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:29:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:42 smithi059 ceph-mon[97450]: pgmap v7442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:29:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:42.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:42.944380+0000) 2022-09-08T01:29:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:43.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:43.005759+0000) 2022-09-08T01:29:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:44.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:44.005984+0000) 2022-09-08T01:29:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:43.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:43.944607+0000) 2022-09-08T01:29:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:44 smithi055 ceph-mon[97033]: pgmap v7443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:44 smithi059 ceph-mon[97450]: pgmap v7443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:45.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:45.006262+0000) 2022-09-08T01:29:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:44.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:44.944793+0000) 2022-09-08T01:29:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:46.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:46.006526+0000) 2022-09-08T01:29:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:45.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:45.944982+0000) 2022-09-08T01:29:46.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:46 smithi055 ceph-mon[97033]: pgmap v7444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:46 smithi059 ceph-mon[97450]: pgmap v7444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:47.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:47.006724+0000) 2022-09-08T01:29:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:46.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:46.945180+0000) 2022-09-08T01:29:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:47.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:47.945400+0000) 2022-09-08T01:29:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:48.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:48.006906+0000) 2022-09-08T01:29:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:48 smithi055 ceph-mon[97033]: pgmap v7445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:29:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:48 smithi059 ceph-mon[97450]: pgmap v7445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:29:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:48.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:48.945685+0000) 2022-09-08T01:29:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:49.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:49.007188+0000) 2022-09-08T01:29:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:50.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:50.007433+0000) 2022-09-08T01:29:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:49.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:49.945909+0000) 2022-09-08T01:29:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:50 smithi055 ceph-mon[97033]: pgmap v7446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:50 smithi059 ceph-mon[97450]: pgmap v7446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:50.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:50.946152+0000) 2022-09-08T01:29:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:51.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:51.007728+0000) 2022-09-08T01:29:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:52.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:52.007988+0000) 2022-09-08T01:29:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:51.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:51.946311+0000) 2022-09-08T01:29:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:52 smithi055 ceph-mon[97033]: pgmap v7447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:29:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:52 smithi059 ceph-mon[97450]: pgmap v7447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:29:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:53.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:53.008208+0000) 2022-09-08T01:29:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:52.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:52.946564+0000) 2022-09-08T01:29:53.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:29:53.460+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:29:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:54.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:54.008424+0000) 2022-09-08T01:29:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:53.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:53.946785+0000) 2022-09-08T01:29:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:54 smithi055 ceph-mon[97033]: pgmap v7448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:54 smithi055 ceph-mon[97033]: pgmap v7449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:29:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:29:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:54 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T01:29:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:54 smithi059 ceph-mon[97450]: pgmap v7448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:29:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:54 smithi059 ceph-mon[97450]: pgmap v7449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:29:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:29:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:54 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T01:29:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:55.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:55.008709+0000) 2022-09-08T01:29:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:54.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:54.946936+0000) 2022-09-08T01:29:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:56.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:56.008930+0000) 2022-09-08T01:29:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:55.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:55.947155+0000) 2022-09-08T01:29:56.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:56 smithi055 ceph-mon[97033]: pgmap v7450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T01:29:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:56 smithi059 ceph-mon[97450]: pgmap v7450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T01:29:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:57.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:57.009097+0000) 2022-09-08T01:29:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:56.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:56.947415+0000) 2022-09-08T01:29:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:29:57 smithi059 ceph-mon[97450]: pgmap v7451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:29:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:29:57 smithi055 ceph-mon[97033]: pgmap v7451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:29:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:58.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:58.009236+0000) 2022-09-08T01:29:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:57.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:57.947608+0000) 2022-09-08T01:29:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:29:59.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:59.009419+0000) 2022-09-08T01:29:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:58.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:58.947866+0000) 2022-09-08T01:30:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:29:59.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:29:59.948058+0000) 2022-09-08T01:30:00.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:00.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:00.009651+0000) 2022-09-08T01:30:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:00 smithi055 ceph-mon[97033]: pgmap v7452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:30:00.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:00 smithi055 ceph-mon[97033]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T01:30:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:00 smithi059 ceph-mon[97450]: pgmap v7452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:30:00.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:00 smithi059 ceph-mon[97450]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T01:30:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:01.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:01.009800+0000) 2022-09-08T01:30:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:00.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:00.948292+0000) 2022-09-08T01:30:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:02.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:02.010044+0000) 2022-09-08T01:30:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:01.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:01.948519+0000) 2022-09-08T01:30:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:02 smithi055 ceph-mon[97033]: pgmap v7453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:30:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:02 smithi059 ceph-mon[97450]: pgmap v7453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:30:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:03.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:03.010281+0000) 2022-09-08T01:30:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:02.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:02.948733+0000) 2022-09-08T01:30:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:04.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:04.010509+0000) 2022-09-08T01:30:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:03.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:03.948904+0000) 2022-09-08T01:30:04.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:04 smithi055 ceph-mon[97033]: pgmap v7454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:30:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:04 smithi059 ceph-mon[97450]: pgmap v7454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:30:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:05.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:05.010759+0000) 2022-09-08T01:30:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:04.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:04.949115+0000) 2022-09-08T01:30:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:30:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:30:05.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:30:05.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:30:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:06.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:06.010951+0000) 2022-09-08T01:30:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:05.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:05.949304+0000) 2022-09-08T01:30:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:06 smithi055 ceph-mon[97033]: pgmap v7455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:30:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:06 smithi059 ceph-mon[97450]: pgmap v7455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:30:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:07.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:07.011177+0000) 2022-09-08T01:30:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:06.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:06.949456+0000) 2022-09-08T01:30:08.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:07 smithi059 ceph-mon[97450]: pgmap v7456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:30:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:07 smithi055 ceph-mon[97033]: pgmap v7456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:30:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:08.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:08.011426+0000) 2022-09-08T01:30:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:07.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:07.949710+0000) 2022-09-08T01:30:08.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:30:08.477+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:30:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:09.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:09.011659+0000) 2022-09-08T01:30:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:08.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:08.949947+0000) 2022-09-08T01:30:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:09 smithi059 ceph-mon[97450]: osdmap e549: 12 total, 11 up, 11 in 2022-09-08T01:30:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:09 smithi059 ceph-mon[97450]: pgmap v7458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:30:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:09 smithi055 ceph-mon[97033]: osdmap e549: 12 total, 11 up, 11 in 2022-09-08T01:30:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:09 smithi055 ceph-mon[97033]: pgmap v7458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:30:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:10.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:10.011873+0000) 2022-09-08T01:30:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:09.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:09.950110+0000) 2022-09-08T01:30:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:11.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:11.012007+0000) 2022-09-08T01:30:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:10.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:10.950315+0000) 2022-09-08T01:30:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:12.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:12.012237+0000) 2022-09-08T01:30:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:11.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:11.950591+0000) 2022-09-08T01:30:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:12 smithi055 ceph-mon[97033]: pgmap v7459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:30:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:12 smithi055 ceph-mon[97033]: osdmap e550: 12 total, 11 up, 11 in 2022-09-08T01:30:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:12 smithi059 ceph-mon[97450]: pgmap v7459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:30:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:12 smithi059 ceph-mon[97450]: osdmap e550: 12 total, 11 up, 11 in 2022-09-08T01:30:13.214 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:13.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:13.012488+0000) 2022-09-08T01:30:13.215 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:12.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:12.950765+0000) 2022-09-08T01:30:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:14.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:14.012772+0000) 2022-09-08T01:30:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:13.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:13.950918+0000) 2022-09-08T01:30:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:14 smithi055 ceph-mon[97033]: pgmap v7461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:30:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:14 smithi059 ceph-mon[97450]: pgmap v7461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:30:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:15.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:15.013015+0000) 2022-09-08T01:30:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:14.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:14.951108+0000) 2022-09-08T01:30:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:16.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:16.013242+0000) 2022-09-08T01:30:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:15.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:15.951333+0000) 2022-09-08T01:30:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:16 smithi055 ceph-mon[97033]: pgmap v7462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:30:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:16 smithi059 ceph-mon[97450]: pgmap v7462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:30:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:17.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:17.013437+0000) 2022-09-08T01:30:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:16.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:16.951563+0000) 2022-09-08T01:30:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:17 smithi059 ceph-mon[97450]: pgmap v7463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 244 B/s wr, 7 op/s 2022-09-08T01:30:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:17 smithi055 ceph-mon[97033]: pgmap v7463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 244 B/s wr, 7 op/s 2022-09-08T01:30:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:18.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:18.013689+0000) 2022-09-08T01:30:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:17.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:17.951814+0000) 2022-09-08T01:30:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:19.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:19.013912+0000) 2022-09-08T01:30:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:18.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:18.952006+0000) 2022-09-08T01:30:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:19.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:19.952183+0000) 2022-09-08T01:30:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:20.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:20.014152+0000) 2022-09-08T01:30:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:20 smithi055 ceph-mon[97033]: pgmap v7464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:30:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:20 smithi059 ceph-mon[97450]: pgmap v7464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:30:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:21.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:21.014405+0000) 2022-09-08T01:30:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:20.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:20.952364+0000) 2022-09-08T01:30:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:22.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:22.014675+0000) 2022-09-08T01:30:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:21.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:21.952590+0000) 2022-09-08T01:30:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:22 smithi055 ceph-mon[97033]: pgmap v7465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:30:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:22 smithi059 ceph-mon[97450]: pgmap v7465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:30:23.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:22.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:22.952807+0000) 2022-09-08T01:30:23.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:23.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:23.014821+0000) 2022-09-08T01:30:23.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:30:23.478+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:30:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:24.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:24.015063+0000) 2022-09-08T01:30:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:23.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:23.952946+0000) 2022-09-08T01:30:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:24 smithi055 ceph-mon[97033]: pgmap v7466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T01:30:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:24 smithi059 ceph-mon[97450]: pgmap v7466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T01:30:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:25.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:25.015239+0000) 2022-09-08T01:30:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:24.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:24.953169+0000) 2022-09-08T01:30:26.409 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:26.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:26.015450+0000) 2022-09-08T01:30:26.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:25.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:25.953431+0000) 2022-09-08T01:30:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:26 smithi055 ceph-mon[97033]: pgmap v7467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:30:26.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:26 smithi059 ceph-mon[97450]: pgmap v7467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:30:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:27.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:27.015737+0000) 2022-09-08T01:30:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:26.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:26.953725+0000) 2022-09-08T01:30:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:27 smithi059 ceph-mon[97450]: pgmap v7468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:27 smithi055 ceph-mon[97033]: pgmap v7468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:28.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:28.015886+0000) 2022-09-08T01:30:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:27.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:27.953905+0000) 2022-09-08T01:30:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:28.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:28.954135+0000) 2022-09-08T01:30:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:29.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:29.016119+0000) 2022-09-08T01:30:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:30.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:30.016425+0000) 2022-09-08T01:30:30.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:29.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:29.954377+0000) 2022-09-08T01:30:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:30 smithi055 ceph-mon[97033]: pgmap v7469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:30 smithi059 ceph-mon[97450]: pgmap v7469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:31.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:31.016655+0000) 2022-09-08T01:30:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:30.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:30.954594+0000) 2022-09-08T01:30:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:31.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:31.954735+0000) 2022-09-08T01:30:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:32.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:32.016861+0000) 2022-09-08T01:30:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:32 smithi055 ceph-mon[97033]: pgmap v7470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:30:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:32 smithi059 ceph-mon[97450]: pgmap v7470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:30:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:33.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:33.017078+0000) 2022-09-08T01:30:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:32.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:32.954916+0000) 2022-09-08T01:30:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:34.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:34.017331+0000) 2022-09-08T01:30:34.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:33.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:33.955135+0000) 2022-09-08T01:30:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:34 smithi055 ceph-mon[97033]: pgmap v7471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:34 smithi059 ceph-mon[97450]: pgmap v7471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:35.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:35.017519+0000) 2022-09-08T01:30:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:34.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:34.955387+0000) 2022-09-08T01:30:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:36.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:36.017736+0000) 2022-09-08T01:30:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:35.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:35.955542+0000) 2022-09-08T01:30:36.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:36 smithi055 ceph-mon[97033]: pgmap v7472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:30:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:36 smithi059 ceph-mon[97450]: pgmap v7472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:30:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:37.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:37.017996+0000) 2022-09-08T01:30:37.415 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:36.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:36.955752+0000) 2022-09-08T01:30:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:37 smithi059 ceph-mon[97450]: pgmap v7473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:37 smithi055 ceph-mon[97033]: pgmap v7473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:38.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:38.018216+0000) 2022-09-08T01:30:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:37.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:37.955950+0000) 2022-09-08T01:30:38.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:30:38.482+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:30:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:39.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:39.018405+0000) 2022-09-08T01:30:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:38.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:38.956184+0000) 2022-09-08T01:30:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:40.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:40.018539+0000) 2022-09-08T01:30:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:39.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:39.956362+0000) 2022-09-08T01:30:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:40 smithi055 ceph-mon[97033]: pgmap v7474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:40 smithi059 ceph-mon[97450]: pgmap v7474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:41.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:41.018784+0000) 2022-09-08T01:30:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:40.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:40.956612+0000) 2022-09-08T01:30:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:42.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:42.019010+0000) 2022-09-08T01:30:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:41.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:41.956899+0000) 2022-09-08T01:30:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:42 smithi055 ceph-mon[97033]: pgmap v7475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:30:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:42 smithi059 ceph-mon[97450]: pgmap v7475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:30:43.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:43.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:43.019244+0000) 2022-09-08T01:30:43.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:42.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:42.957094+0000) 2022-09-08T01:30:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:44.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:44.019488+0000) 2022-09-08T01:30:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:43.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:43.957317+0000) 2022-09-08T01:30:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:44 smithi055 ceph-mon[97033]: pgmap v7476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:44 smithi059 ceph-mon[97450]: pgmap v7476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:45.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:45.019773+0000) 2022-09-08T01:30:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:44.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:44.957544+0000) 2022-09-08T01:30:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:46.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:46.020026+0000) 2022-09-08T01:30:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:45.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:45.957780+0000) 2022-09-08T01:30:46.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:46 smithi055 ceph-mon[97033]: pgmap v7477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:30:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:46 smithi059 ceph-mon[97450]: pgmap v7477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:30:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:47.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:47.020259+0000) 2022-09-08T01:30:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:46.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:46.958048+0000) 2022-09-08T01:30:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:47 smithi059 ceph-mon[97450]: pgmap v7478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:47 smithi055 ceph-mon[97033]: pgmap v7478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:48.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:48.020464+0000) 2022-09-08T01:30:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:47.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:47.958278+0000) 2022-09-08T01:30:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:49.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:49.020685+0000) 2022-09-08T01:30:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:48.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:48.958524+0000) 2022-09-08T01:30:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:50.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:50.020943+0000) 2022-09-08T01:30:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:49.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:49.958792+0000) 2022-09-08T01:30:50.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:50 smithi055 ceph-mon[97033]: pgmap v7479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:50 smithi059 ceph-mon[97450]: pgmap v7479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:51.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:51.021151+0000) 2022-09-08T01:30:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:50.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:50.959039+0000) 2022-09-08T01:30:52.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:52.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:52.021370+0000) 2022-09-08T01:30:52.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:51.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:51.959264+0000) 2022-09-08T01:30:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:52 smithi055 ceph-mon[97033]: pgmap v7480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:30:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:52 smithi059 ceph-mon[97450]: pgmap v7480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:30:53.290 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:53.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:53.021594+0000) 2022-09-08T01:30:53.291 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:52.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:52.959504+0000) 2022-09-08T01:30:53.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:30:53.498+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:30:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:54.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:54.021866+0000) 2022-09-08T01:30:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:53.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:53.959732+0000) 2022-09-08T01:30:54.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:54 smithi055 ceph-mon[97033]: pgmap v7481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:54.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:30:54.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:30:54.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:30:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:54 smithi059 ceph-mon[97450]: pgmap v7481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:30:54.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:30:54.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:30:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:55.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:55.022081+0000) 2022-09-08T01:30:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:54.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:54.959980+0000) 2022-09-08T01:30:56.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:56.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:56.022264+0000) 2022-09-08T01:30:56.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:55.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:55.960210+0000) 2022-09-08T01:30:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:56 smithi059 ceph-mon[97450]: pgmap v7482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:30:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:56 smithi055 ceph-mon[97033]: pgmap v7482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:30:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:57.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:57.022480+0000) 2022-09-08T01:30:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:56.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:56.960448+0000) 2022-09-08T01:30:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:30:57 smithi055 ceph-mon[97033]: pgmap v7483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:30:57 smithi059 ceph-mon[97450]: pgmap v7483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:30:58.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:58.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:58.022725+0000) 2022-09-08T01:30:58.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:57.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:57.960605+0000) 2022-09-08T01:30:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:30:59.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:59.022880+0000) 2022-09-08T01:30:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:58.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:58.960864+0000) 2022-09-08T01:31:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:00.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:00.023113+0000) 2022-09-08T01:31:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:30:59.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:30:59.961084+0000) 2022-09-08T01:31:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:00 smithi055 ceph-mon[97033]: pgmap v7484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:31:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:00 smithi059 ceph-mon[97450]: pgmap v7484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:31:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:01.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:01.023326+0000) 2022-09-08T01:31:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:00.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:00.961291+0000) 2022-09-08T01:31:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:02.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:02.023558+0000) 2022-09-08T01:31:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:01.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:01.961493+0000) 2022-09-08T01:31:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:02 smithi055 ceph-mon[97033]: pgmap v7485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:31:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:02 smithi059 ceph-mon[97450]: pgmap v7485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:31:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:03.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:03.023757+0000) 2022-09-08T01:31:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:02.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:02.961738+0000) 2022-09-08T01:31:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:04.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:04.024008+0000) 2022-09-08T01:31:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:03.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:03.961986+0000) 2022-09-08T01:31:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:04 smithi055 ceph-mon[97033]: pgmap v7486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:31:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:04 smithi059 ceph-mon[97450]: pgmap v7486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:31:05.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:05.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:05.024231+0000) 2022-09-08T01:31:05.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:04.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:04.962199+0000) 2022-09-08T01:31:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:31:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:31:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:31:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:31:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:06.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:06.024489+0000) 2022-09-08T01:31:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:05.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:05.962444+0000) 2022-09-08T01:31:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:06 smithi059 ceph-mon[97450]: pgmap v7487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:31:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:06 smithi055 ceph-mon[97033]: pgmap v7487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:31:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:07.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:07.024690+0000) 2022-09-08T01:31:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:06.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:06.962708+0000) 2022-09-08T01:31:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:07 smithi055 ceph-mon[97033]: pgmap v7488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:31:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:07 smithi059 ceph-mon[97450]: pgmap v7488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:31:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:08.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:08.024921+0000) 2022-09-08T01:31:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:07.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:07.962935+0000) 2022-09-08T01:31:08.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:31:08.511+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:31:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:09.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:09.025093+0000) 2022-09-08T01:31:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:08.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:08.963141+0000) 2022-09-08T01:31:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:09 smithi055 ceph-mon[97033]: osdmap e551: 12 total, 11 up, 11 in 2022-09-08T01:31:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:09 smithi055 ceph-mon[97033]: pgmap v7490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:31:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:09 smithi059 ceph-mon[97450]: osdmap e551: 12 total, 11 up, 11 in 2022-09-08T01:31:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:09 smithi059 ceph-mon[97450]: pgmap v7490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:31:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:10.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:10.025297+0000) 2022-09-08T01:31:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:09.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:09.963389+0000) 2022-09-08T01:31:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:11.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:11.025535+0000) 2022-09-08T01:31:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:10.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:10.963669+0000) 2022-09-08T01:31:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:12.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:12.025718+0000) 2022-09-08T01:31:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:11.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:11.963826+0000) 2022-09-08T01:31:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:12 smithi055 ceph-mon[97033]: pgmap v7491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:31:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:12 smithi055 ceph-mon[97033]: osdmap e552: 12 total, 11 up, 11 in 2022-09-08T01:31:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:12 smithi059 ceph-mon[97450]: pgmap v7491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:31:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:12 smithi059 ceph-mon[97450]: osdmap e552: 12 total, 11 up, 11 in 2022-09-08T01:31:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:13.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:13.025933+0000) 2022-09-08T01:31:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:12.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:12.964064+0000) 2022-09-08T01:31:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:14.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:14.026145+0000) 2022-09-08T01:31:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:13.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:13.964209+0000) 2022-09-08T01:31:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:14 smithi055 ceph-mon[97033]: pgmap v7493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:31:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:14 smithi059 ceph-mon[97450]: pgmap v7493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:31:15.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:15.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:15.026368+0000) 2022-09-08T01:31:15.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:14.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:14.964461+0000) 2022-09-08T01:31:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:15.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:15.964723+0000) 2022-09-08T01:31:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:16.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:16.026593+0000) 2022-09-08T01:31:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:16 smithi059 ceph-mon[97450]: pgmap v7494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:31:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:16 smithi055 ceph-mon[97033]: pgmap v7494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:31:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:16.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:16.964971+0000) 2022-09-08T01:31:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:17.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:17.026781+0000) 2022-09-08T01:31:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:17 smithi055 ceph-mon[97033]: pgmap v7495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 244 B/s wr, 7 op/s 2022-09-08T01:31:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:17 smithi059 ceph-mon[97450]: pgmap v7495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 244 B/s wr, 7 op/s 2022-09-08T01:31:18.357 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:18.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:18.026921+0000) 2022-09-08T01:31:18.359 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:17.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:17.965216+0000) 2022-09-08T01:31:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:18.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:18.965463+0000) 2022-09-08T01:31:19.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:19.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:19.027175+0000) 2022-09-08T01:31:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:20.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:20.027366+0000) 2022-09-08T01:31:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:19.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:19.965723+0000) 2022-09-08T01:31:20.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:20 smithi055 ceph-mon[97033]: pgmap v7496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:31:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:20 smithi059 ceph-mon[97450]: pgmap v7496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:31:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:21.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:21.027589+0000) 2022-09-08T01:31:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:20.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:20.965957+0000) 2022-09-08T01:31:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:22.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:22.027787+0000) 2022-09-08T01:31:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:21.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:21.966119+0000) 2022-09-08T01:31:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:22 smithi059 ceph-mon[97450]: pgmap v7497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:31:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:22 smithi055 ceph-mon[97033]: pgmap v7497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:31:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:23.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:23.027983+0000) 2022-09-08T01:31:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:22.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:22.966341+0000) 2022-09-08T01:31:23.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:31:23.513+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:31:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:24.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:24.028175+0000) 2022-09-08T01:31:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:23.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:23.966562+0000) 2022-09-08T01:31:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:24 smithi059 ceph-mon[97450]: pgmap v7498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T01:31:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:31:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:24 smithi055 ceph-mon[97033]: pgmap v7498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T01:31:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:31:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:25.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:25.028388+0000) 2022-09-08T01:31:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:24.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:24.966752+0000) 2022-09-08T01:31:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:25 smithi059 ceph-mon[97450]: pgmap v7499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-08T01:31:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:25 smithi059 ceph-mon[97450]: pgmap v7500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:31:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:25 smithi055 ceph-mon[97033]: pgmap v7499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-08T01:31:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:25 smithi055 ceph-mon[97033]: pgmap v7500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:31:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:26.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:26.028600+0000) 2022-09-08T01:31:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:25.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:25.966975+0000) 2022-09-08T01:31:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:27.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:27.028703+0000) 2022-09-08T01:31:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:26.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:26.967219+0000) 2022-09-08T01:31:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:27 smithi059 ceph-mon[97450]: pgmap v7501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:31:27.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:27 smithi055 ceph-mon[97033]: pgmap v7501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:31:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:28.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:28.028868+0000) 2022-09-08T01:31:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:27.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:27.967423+0000) 2022-09-08T01:31:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:29.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:29.029094+0000) 2022-09-08T01:31:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:28.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:28.967676+0000) 2022-09-08T01:31:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:29 smithi059 ceph-mon[97450]: pgmap v7502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:31:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:29 smithi055 ceph-mon[97033]: pgmap v7502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:31:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:30.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:30.029341+0000) 2022-09-08T01:31:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:29.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:29.967892+0000) 2022-09-08T01:31:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:30.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:30.968143+0000) 2022-09-08T01:31:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:31.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:31.029618+0000) 2022-09-08T01:31:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:31 smithi059 ceph-mon[97450]: pgmap v7503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:31:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:31 smithi055 ceph-mon[97033]: pgmap v7503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:31:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:31.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:31.968340+0000) 2022-09-08T01:31:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:32.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:32.029930+0000) 2022-09-08T01:31:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:32.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:32.968480+0000) 2022-09-08T01:31:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:33.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:33.030182+0000) 2022-09-08T01:31:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:33 smithi059 ceph-mon[97450]: pgmap v7504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:31:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:33 smithi055 ceph-mon[97033]: pgmap v7504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:31:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:33.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:33.968662+0000) 2022-09-08T01:31:34.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:34.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:34.030467+0000) 2022-09-08T01:31:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:35.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:35.030757+0000) 2022-09-08T01:31:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:34.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:34.968895+0000) 2022-09-08T01:31:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:35 smithi059 ceph-mon[97450]: pgmap v7505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:31:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:35 smithi055 ceph-mon[97033]: pgmap v7505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:31:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:35.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:35.969124+0000) 2022-09-08T01:31:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:36.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:36.030905+0000) 2022-09-08T01:31:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:36.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:36.969429+0000) 2022-09-08T01:31:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:37.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:37.031140+0000) 2022-09-08T01:31:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:37 smithi059 ceph-mon[97450]: pgmap v7506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:31:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:37 smithi055 ceph-mon[97033]: pgmap v7506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:31:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:38.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:38.031328+0000) 2022-09-08T01:31:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:37.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:37.969578+0000) 2022-09-08T01:31:38.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:31:38.530+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:31:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:39.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:39.031526+0000) 2022-09-08T01:31:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:38.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:38.969782+0000) 2022-09-08T01:31:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:39 smithi059 ceph-mon[97450]: pgmap v7507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:31:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:39 smithi055 ceph-mon[97033]: pgmap v7507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:31:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:40.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:40.031764+0000) 2022-09-08T01:31:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:39.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:39.969969+0000) 2022-09-08T01:31:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:41.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:41.032002+0000) 2022-09-08T01:31:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:40.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:40.970117+0000) 2022-09-08T01:31:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:41 smithi059 ceph-mon[97450]: pgmap v7508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:31:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:41 smithi055 ceph-mon[97033]: pgmap v7508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:31:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:42.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:42.032257+0000) 2022-09-08T01:31:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:41.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:41.970307+0000) 2022-09-08T01:31:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:42 smithi055 ceph-mon[97033]: pgmap v7509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:31:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:42 smithi059 ceph-mon[97450]: pgmap v7509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:31:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:43.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:43.032432+0000) 2022-09-08T01:31:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:42.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:42.970489+0000) 2022-09-08T01:31:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:44.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:44.032620+0000) 2022-09-08T01:31:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:43.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:43.970745+0000) 2022-09-08T01:31:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:44 smithi055 ceph-mon[97033]: pgmap v7510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:31:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:44 smithi059 ceph-mon[97450]: pgmap v7510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:31:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:45.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:45.032796+0000) 2022-09-08T01:31:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:44.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:44.970895+0000) 2022-09-08T01:31:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:45.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:45.971101+0000) 2022-09-08T01:31:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:46.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:46.032964+0000) 2022-09-08T01:31:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:46 smithi055 ceph-mon[97033]: pgmap v7511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:31:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:46 smithi059 ceph-mon[97450]: pgmap v7511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:31:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:46.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:46.971344+0000) 2022-09-08T01:31:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:47.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:47.033186+0000) 2022-09-08T01:31:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:47.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:47.971491+0000) 2022-09-08T01:31:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:48.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:48.033432+0000) 2022-09-08T01:31:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:48 smithi055 ceph-mon[97033]: pgmap v7512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:31:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:48 smithi059 ceph-mon[97450]: pgmap v7512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:31:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:48.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:48.971625+0000) 2022-09-08T01:31:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:49.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:49.033696+0000) 2022-09-08T01:31:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:49.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:49.971801+0000) 2022-09-08T01:31:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:50.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:50.033901+0000) 2022-09-08T01:31:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:50 smithi055 ceph-mon[97033]: pgmap v7513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:31:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:50 smithi059 ceph-mon[97450]: pgmap v7513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:31:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:50.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:50.972009+0000) 2022-09-08T01:31:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:51.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:51.034197+0000) 2022-09-08T01:31:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:52.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:52.034523+0000) 2022-09-08T01:31:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:51.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:51.972200+0000) 2022-09-08T01:31:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:52 smithi055 ceph-mon[97033]: pgmap v7514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:31:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:52 smithi059 ceph-mon[97450]: pgmap v7514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:31:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:53.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:53.034715+0000) 2022-09-08T01:31:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:52.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:52.972349+0000) 2022-09-08T01:31:53.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:31:53.543+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:31:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:54.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:54.034951+0000) 2022-09-08T01:31:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:53.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:53.972515+0000) 2022-09-08T01:31:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:54 smithi055 ceph-mon[97033]: pgmap v7515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:31:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:54 smithi059 ceph-mon[97450]: pgmap v7515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:31:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:55.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:55.035076+0000) 2022-09-08T01:31:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:54.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:54.972700+0000) 2022-09-08T01:31:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:56.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:56.035359+0000) 2022-09-08T01:31:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:55.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:55.972852+0000) 2022-09-08T01:31:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:56 smithi055 ceph-mon[97033]: pgmap v7516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:31:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:56 smithi059 ceph-mon[97450]: pgmap v7516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:31:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:57.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:57.035575+0000) 2022-09-08T01:31:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:56.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:56.973060+0000) 2022-09-08T01:31:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:58.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:58.035737+0000) 2022-09-08T01:31:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:57.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:57.973264+0000) 2022-09-08T01:31:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:31:58 smithi055 ceph-mon[97033]: pgmap v7517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:31:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:31:58 smithi059 ceph-mon[97450]: pgmap v7517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:31:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:31:59.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:59.035943+0000) 2022-09-08T01:31:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:58.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:58.973438+0000) 2022-09-08T01:32:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:00.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:00.036106+0000) 2022-09-08T01:32:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:31:59.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:31:59.973661+0000) 2022-09-08T01:32:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:00 smithi055 ceph-mon[97033]: pgmap v7518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:32:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:00 smithi059 ceph-mon[97450]: pgmap v7518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:32:01.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:01.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:01.036318+0000) 2022-09-08T01:32:01.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:00.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:00.973837+0000) 2022-09-08T01:32:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:02.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:02.036571+0000) 2022-09-08T01:32:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:01.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:01.974041+0000) 2022-09-08T01:32:03.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:02 smithi055 ceph-mon[97033]: pgmap v7519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:32:03.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:02 smithi059 ceph-mon[97450]: pgmap v7519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:32:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:03.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:03.036794+0000) 2022-09-08T01:32:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:02.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:02.974221+0000) 2022-09-08T01:32:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:04.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:04.037041+0000) 2022-09-08T01:32:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:03.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:03.974359+0000) 2022-09-08T01:32:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:04 smithi055 ceph-mon[97033]: pgmap v7520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:04 smithi059 ceph-mon[97450]: pgmap v7520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:05.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:05.037251+0000) 2022-09-08T01:32:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:04.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:04.974565+0000) 2022-09-08T01:32:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:32:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:32:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:32:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:32:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:06.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:06.037476+0000) 2022-09-08T01:32:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:05.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:05.974739+0000) 2022-09-08T01:32:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:06 smithi055 ceph-mon[97033]: pgmap v7521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:32:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:06 smithi059 ceph-mon[97450]: pgmap v7521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:32:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:06.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:06.974893+0000) 2022-09-08T01:32:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:07.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:07.037691+0000) 2022-09-08T01:32:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:07.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:07.975083+0000) 2022-09-08T01:32:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:08.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:08.037815+0000) 2022-09-08T01:32:08.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:32:08.553+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:32:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:08 smithi055 ceph-mon[97033]: pgmap v7522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:32:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:08 smithi059 ceph-mon[97450]: pgmap v7522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:32:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:09.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:09.037980+0000) 2022-09-08T01:32:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:08.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:08.975238+0000) 2022-09-08T01:32:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:09 smithi055 ceph-mon[97033]: osdmap e553: 12 total, 11 up, 11 in 2022-09-08T01:32:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:09 smithi059 ceph-mon[97450]: osdmap e553: 12 total, 11 up, 11 in 2022-09-08T01:32:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:10.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:10.038202+0000) 2022-09-08T01:32:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:09.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:09.975406+0000) 2022-09-08T01:32:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:10 smithi055 ceph-mon[97033]: pgmap v7524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:32:11.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:10 smithi059 ceph-mon[97450]: pgmap v7524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:32:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:11.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:11.038447+0000) 2022-09-08T01:32:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:10.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:10.975618+0000) 2022-09-08T01:32:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:12.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:12.038681+0000) 2022-09-08T01:32:12.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:11.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:11.975827+0000) 2022-09-08T01:32:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:12 smithi055 ceph-mon[97033]: pgmap v7525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:32:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:12 smithi059 ceph-mon[97450]: pgmap v7525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:32:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:13.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:13.038890+0000) 2022-09-08T01:32:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:12.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:12.976014+0000) 2022-09-08T01:32:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:14.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:14.039029+0000) 2022-09-08T01:32:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:13.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:13.976225+0000) 2022-09-08T01:32:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:14 smithi055 ceph-mon[97033]: pgmap v7526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:32:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:14 smithi059 ceph-mon[97450]: pgmap v7526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:32:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:14.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:14.976460+0000) 2022-09-08T01:32:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:15.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:15.039210+0000) 2022-09-08T01:32:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:16.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:16.039457+0000) 2022-09-08T01:32:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:15.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:15.976736+0000) 2022-09-08T01:32:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:16 smithi055 ceph-mon[97033]: pgmap v7527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:32:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:16 smithi055 ceph-mon[97033]: osdmap e554: 12 total, 11 up, 11 in 2022-09-08T01:32:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:16 smithi059 ceph-mon[97450]: pgmap v7527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:32:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:16 smithi059 ceph-mon[97450]: osdmap e554: 12 total, 11 up, 11 in 2022-09-08T01:32:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:16.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:16.976966+0000) 2022-09-08T01:32:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:17.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:17.039721+0000) 2022-09-08T01:32:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:17.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:17.977201+0000) 2022-09-08T01:32:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:18.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:18.039930+0000) 2022-09-08T01:32:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:18 smithi055 ceph-mon[97033]: pgmap v7529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 238 B/s wr, 7 op/s 2022-09-08T01:32:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:18 smithi059 ceph-mon[97450]: pgmap v7529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 238 B/s wr, 7 op/s 2022-09-08T01:32:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:19.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:19.040134+0000) 2022-09-08T01:32:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:18.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:18.977407+0000) 2022-09-08T01:32:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:20.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:20.040351+0000) 2022-09-08T01:32:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:19.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:19.977676+0000) 2022-09-08T01:32:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:20 smithi055 ceph-mon[97033]: pgmap v7530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:32:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:20 smithi059 ceph-mon[97450]: pgmap v7530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:32:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:21.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:21.040522+0000) 2022-09-08T01:32:21.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:20.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:20.977859+0000) 2022-09-08T01:32:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:21.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:21.978110+0000) 2022-09-08T01:32:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:22.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:22.040695+0000) 2022-09-08T01:32:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:22 smithi055 ceph-mon[97033]: pgmap v7531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:32:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:22 smithi059 ceph-mon[97450]: pgmap v7531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:32:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:23.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:23.040847+0000) 2022-09-08T01:32:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:22.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:22.978349+0000) 2022-09-08T01:32:23.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:32:23.554+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:32:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:32:24.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:32:24.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:32:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:32:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:32:24.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:32:24.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:24.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:24.041064+0000) 2022-09-08T01:32:24.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:23.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:23.978618+0000) 2022-09-08T01:32:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:24 smithi055 ceph-mon[97033]: pgmap v7532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:32:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:24 smithi059 ceph-mon[97450]: pgmap v7532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:32:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:25.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:25.041237+0000) 2022-09-08T01:32:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:24.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:24.978836+0000) 2022-09-08T01:32:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:26.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:26.041455+0000) 2022-09-08T01:32:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:25.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:25.979053+0000) 2022-09-08T01:32:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:26 smithi055 ceph-mon[97033]: pgmap v7533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:32:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:26 smithi059 ceph-mon[97450]: pgmap v7533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:32:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:27.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:27.041697+0000) 2022-09-08T01:32:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:26.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:26.979296+0000) 2022-09-08T01:32:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:28.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:28.041912+0000) 2022-09-08T01:32:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:27.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:27.979516+0000) 2022-09-08T01:32:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:28 smithi055 ceph-mon[97033]: pgmap v7534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:32:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:28 smithi059 ceph-mon[97450]: pgmap v7534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:32:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:29.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:29.042120+0000) 2022-09-08T01:32:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:28.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:28.979745+0000) 2022-09-08T01:32:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:30.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:30.042346+0000) 2022-09-08T01:32:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:29.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:29.980000+0000) 2022-09-08T01:32:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:30 smithi055 ceph-mon[97033]: pgmap v7535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:30 smithi059 ceph-mon[97450]: pgmap v7535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:31.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:31.042572+0000) 2022-09-08T01:32:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:30.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:30.980228+0000) 2022-09-08T01:32:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:32.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:32.042701+0000) 2022-09-08T01:32:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:31.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:31.980428+0000) 2022-09-08T01:32:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:32 smithi055 ceph-mon[97033]: pgmap v7536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:32:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:32 smithi059 ceph-mon[97450]: pgmap v7536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:32:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:33.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:33.042902+0000) 2022-09-08T01:32:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:32.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:32.980696+0000) 2022-09-08T01:32:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:34.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:34.043074+0000) 2022-09-08T01:32:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:33.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:33.980946+0000) 2022-09-08T01:32:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:34 smithi055 ceph-mon[97033]: pgmap v7537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:34 smithi059 ceph-mon[97450]: pgmap v7537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:35.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:35.043257+0000) 2022-09-08T01:32:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:34.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:34.981186+0000) 2022-09-08T01:32:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:36.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:36.043489+0000) 2022-09-08T01:32:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:35.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:35.981436+0000) 2022-09-08T01:32:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:36 smithi055 ceph-mon[97033]: pgmap v7538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:32:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:36 smithi059 ceph-mon[97450]: pgmap v7538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:32:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:37.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:37.043748+0000) 2022-09-08T01:32:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:36.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:36.981724+0000) 2022-09-08T01:32:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:38.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:38.043970+0000) 2022-09-08T01:32:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:37.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:37.981865+0000) 2022-09-08T01:32:38.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:32:38.570+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:32:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:38 smithi055 ceph-mon[97033]: pgmap v7539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:38 smithi059 ceph-mon[97450]: pgmap v7539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:39.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:39.044137+0000) 2022-09-08T01:32:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:38.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:38.982012+0000) 2022-09-08T01:32:40.408 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:40.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:40.044354+0000) 2022-09-08T01:32:40.409 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:39.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:39.982220+0000) 2022-09-08T01:32:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:40 smithi055 ceph-mon[97033]: pgmap v7540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:40 smithi059 ceph-mon[97450]: pgmap v7540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:41.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:41.044616+0000) 2022-09-08T01:32:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:40.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:40.982487+0000) 2022-09-08T01:32:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:42.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:42.044870+0000) 2022-09-08T01:32:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:41.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:41.982748+0000) 2022-09-08T01:32:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:42 smithi055 ceph-mon[97033]: pgmap v7541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:32:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:42 smithi059 ceph-mon[97450]: pgmap v7541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:32:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:43.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:43.045015+0000) 2022-09-08T01:32:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:42.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:42.982987+0000) 2022-09-08T01:32:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:43.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:43.983257+0000) 2022-09-08T01:32:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:44.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:44.045178+0000) 2022-09-08T01:32:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:44 smithi055 ceph-mon[97033]: pgmap v7542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:44 smithi059 ceph-mon[97450]: pgmap v7542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:45.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:45.045386+0000) 2022-09-08T01:32:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:44.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:44.983474+0000) 2022-09-08T01:32:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:46.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:46.045509+0000) 2022-09-08T01:32:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:45.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:45.983723+0000) 2022-09-08T01:32:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:46 smithi055 ceph-mon[97033]: pgmap v7543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:32:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:46 smithi059 ceph-mon[97450]: pgmap v7543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:32:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:47.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:47.045717+0000) 2022-09-08T01:32:47.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:46.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:46.983974+0000) 2022-09-08T01:32:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:48.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:48.045921+0000) 2022-09-08T01:32:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:47.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:47.984162+0000) 2022-09-08T01:32:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:48 smithi055 ceph-mon[97033]: pgmap v7544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:48 smithi059 ceph-mon[97450]: pgmap v7544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:49.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:49.046087+0000) 2022-09-08T01:32:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:48.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:48.984392+0000) 2022-09-08T01:32:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:50.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:50.046305+0000) 2022-09-08T01:32:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:49.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:49.984615+0000) 2022-09-08T01:32:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:50 smithi055 ceph-mon[97033]: pgmap v7545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:50 smithi059 ceph-mon[97450]: pgmap v7545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:51.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:51.046546+0000) 2022-09-08T01:32:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:50.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:50.984843+0000) 2022-09-08T01:32:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:51.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:51.985062+0000) 2022-09-08T01:32:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:52.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:52.046777+0000) 2022-09-08T01:32:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:52 smithi055 ceph-mon[97033]: pgmap v7546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:32:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:52 smithi059 ceph-mon[97450]: pgmap v7546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:32:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:53.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:53.046992+0000) 2022-09-08T01:32:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:52.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:52.985288+0000) 2022-09-08T01:32:53.682 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:32:53.570+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:32:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:54.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:54.047165+0000) 2022-09-08T01:32:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:53.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:53.985500+0000) 2022-09-08T01:32:54.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:54 smithi055 ceph-mon[97033]: pgmap v7547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:54 smithi055 ceph-mon[97033]: pgmap v7548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:32:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:54 smithi055 ceph-mon[97033]: pgmap v7549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:32:54.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:32:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:54 smithi059 ceph-mon[97450]: pgmap v7547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:32:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:54 smithi059 ceph-mon[97450]: pgmap v7548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:32:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:54 smithi059 ceph-mon[97450]: pgmap v7549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:32:55.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:32:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:55.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:55.047351+0000) 2022-09-08T01:32:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:54.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:54.985736+0000) 2022-09-08T01:32:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:55.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:55.985894+0000) 2022-09-08T01:32:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:56.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:56.047555+0000) 2022-09-08T01:32:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:56 smithi055 ceph-mon[97033]: pgmap v7550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:32:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:56 smithi059 ceph-mon[97450]: pgmap v7550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:32:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:56.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:56.986033+0000) 2022-09-08T01:32:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:57.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:57.047789+0000) 2022-09-08T01:32:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:58.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:58.047998+0000) 2022-09-08T01:32:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:57.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:57.986264+0000) 2022-09-08T01:32:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:32:58 smithi055 ceph-mon[97033]: pgmap v7551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:32:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:32:58 smithi059 ceph-mon[97450]: pgmap v7551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:32:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:58.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:58.986511+0000) 2022-09-08T01:32:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:32:59.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:59.048219+0000) 2022-09-08T01:33:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:32:59.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:32:59.986713+0000) 2022-09-08T01:33:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:00.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:00.048462+0000) 2022-09-08T01:33:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:00 smithi055 ceph-mon[97033]: pgmap v7552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:33:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:00 smithi059 ceph-mon[97450]: pgmap v7552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:33:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:01.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:01.048609+0000) 2022-09-08T01:33:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:00.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:00.986834+0000) 2022-09-08T01:33:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:01.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:01.987055+0000) 2022-09-08T01:33:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:02.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:02.048759+0000) 2022-09-08T01:33:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:02 smithi055 ceph-mon[97033]: pgmap v7553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:33:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:02 smithi059 ceph-mon[97450]: pgmap v7553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:33:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:02.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:02.987293+0000) 2022-09-08T01:33:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:03.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:03.048920+0000) 2022-09-08T01:33:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:04.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:04.049156+0000) 2022-09-08T01:33:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:03.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:03.987522+0000) 2022-09-08T01:33:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:04 smithi055 ceph-mon[97033]: pgmap v7554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:33:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:04 smithi059 ceph-mon[97450]: pgmap v7554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:33:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:04.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:04.987744+0000) 2022-09-08T01:33:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:05.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:05.049329+0000) 2022-09-08T01:33:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:33:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:33:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:33:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:33:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:06.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:06.049494+0000) 2022-09-08T01:33:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:05.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:05.987904+0000) 2022-09-08T01:33:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:06 smithi055 ceph-mon[97033]: pgmap v7555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:33:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:06 smithi059 ceph-mon[97450]: pgmap v7555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:33:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:07.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:07.049743+0000) 2022-09-08T01:33:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:06.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:06.988122+0000) 2022-09-08T01:33:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:08.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:08.049932+0000) 2022-09-08T01:33:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:07.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:07.988300+0000) 2022-09-08T01:33:08.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:33:08.583+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:33:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:08 smithi055 ceph-mon[97033]: pgmap v7556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:33:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:08 smithi059 ceph-mon[97450]: pgmap v7556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:33:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:08.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:08.988523+0000) 2022-09-08T01:33:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:09.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:09.050135+0000) 2022-09-08T01:33:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:09 smithi055 ceph-mon[97033]: osdmap e555: 12 total, 11 up, 11 in 2022-09-08T01:33:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:09 smithi059 ceph-mon[97450]: osdmap e555: 12 total, 11 up, 11 in 2022-09-08T01:33:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:10.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:10.050313+0000) 2022-09-08T01:33:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:09.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:09.988746+0000) 2022-09-08T01:33:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:10 smithi055 ceph-mon[97033]: pgmap v7558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:33:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:10 smithi059 ceph-mon[97450]: pgmap v7558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:33:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:10.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:10.988921+0000) 2022-09-08T01:33:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:11.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:11.050549+0000) 2022-09-08T01:33:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:12.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:12.050803+0000) 2022-09-08T01:33:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:11.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:11.989121+0000) 2022-09-08T01:33:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:12 smithi055 ceph-mon[97033]: pgmap v7559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:33:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:12 smithi059 ceph-mon[97450]: pgmap v7559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:33:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:13.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:13.051052+0000) 2022-09-08T01:33:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:12.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:12.989362+0000) 2022-09-08T01:33:14.052 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:13.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:13.989630+0000) 2022-09-08T01:33:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:14.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:14.051277+0000) 2022-09-08T01:33:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:14 smithi055 ceph-mon[97033]: pgmap v7560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:33:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:14 smithi059 ceph-mon[97450]: pgmap v7560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:33:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:15.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:15.051508+0000) 2022-09-08T01:33:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:14.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:14.989832+0000) 2022-09-08T01:33:16.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:16.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:16.051799+0000) 2022-09-08T01:33:16.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:15.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:15.989982+0000) 2022-09-08T01:33:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:16 smithi055 ceph-mon[97033]: pgmap v7561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:33:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:16 smithi055 ceph-mon[97033]: osdmap e556: 12 total, 11 up, 11 in 2022-09-08T01:33:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:16 smithi059 ceph-mon[97450]: pgmap v7561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:33:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:16 smithi059 ceph-mon[97450]: osdmap e556: 12 total, 11 up, 11 in 2022-09-08T01:33:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:17.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:17.052018+0000) 2022-09-08T01:33:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:16.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:16.990149+0000) 2022-09-08T01:33:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:18.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:18.052184+0000) 2022-09-08T01:33:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:17.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:17.990388+0000) 2022-09-08T01:33:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:18 smithi055 ceph-mon[97033]: pgmap v7563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-08T01:33:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:18 smithi059 ceph-mon[97450]: pgmap v7563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-08T01:33:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:19.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:19.052413+0000) 2022-09-08T01:33:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:18.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:18.990573+0000) 2022-09-08T01:33:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:20.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:20.052566+0000) 2022-09-08T01:33:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:19.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:19.990801+0000) 2022-09-08T01:33:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:20 smithi055 ceph-mon[97033]: pgmap v7564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:33:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:20 smithi059 ceph-mon[97450]: pgmap v7564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:33:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:21.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:21.052728+0000) 2022-09-08T01:33:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:20.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:20.991042+0000) 2022-09-08T01:33:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:22.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:22.052894+0000) 2022-09-08T01:33:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:21.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:21.991286+0000) 2022-09-08T01:33:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:22 smithi055 ceph-mon[97033]: pgmap v7565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:33:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:22 smithi059 ceph-mon[97450]: pgmap v7565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:33:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:23.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:23.053131+0000) 2022-09-08T01:33:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:22.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:22.991492+0000) 2022-09-08T01:33:23.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:33:23.593+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:33:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:24.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:24.053421+0000) 2022-09-08T01:33:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:23.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:23.991766+0000) 2022-09-08T01:33:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:24 smithi055 ceph-mon[97033]: pgmap v7566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:33:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:24 smithi059 ceph-mon[97450]: pgmap v7566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:33:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:25.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:25.053732+0000) 2022-09-08T01:33:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:24.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:24.992012+0000) 2022-09-08T01:33:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:26.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:26.054016+0000) 2022-09-08T01:33:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:25.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:25.992264+0000) 2022-09-08T01:33:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:26 smithi055 ceph-mon[97033]: pgmap v7567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:33:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:26 smithi059 ceph-mon[97450]: pgmap v7567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:33:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:27.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:27.054188+0000) 2022-09-08T01:33:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:26.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:26.992480+0000) 2022-09-08T01:33:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:27.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:27.992740+0000) 2022-09-08T01:33:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:28.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:28.054402+0000) 2022-09-08T01:33:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:28 smithi055 ceph-mon[97033]: pgmap v7568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:33:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:28 smithi059 ceph-mon[97450]: pgmap v7568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:33:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:29.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:29.054620+0000) 2022-09-08T01:33:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:28.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:28.992884+0000) 2022-09-08T01:33:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:29.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:29.993040+0000) 2022-09-08T01:33:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:30.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:30.054892+0000) 2022-09-08T01:33:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:30 smithi055 ceph-mon[97033]: pgmap v7569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:30 smithi059 ceph-mon[97450]: pgmap v7569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:31.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:31.055126+0000) 2022-09-08T01:33:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:30.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:30.993245+0000) 2022-09-08T01:33:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:32.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:32.055377+0000) 2022-09-08T01:33:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:31.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:31.993454+0000) 2022-09-08T01:33:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:32 smithi055 ceph-mon[97033]: pgmap v7570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:33:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:32 smithi059 ceph-mon[97450]: pgmap v7570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:33:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:33.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:33.055604+0000) 2022-09-08T01:33:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:32.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:32.993716+0000) 2022-09-08T01:33:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:34.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:34.055865+0000) 2022-09-08T01:33:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:33.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:33.993926+0000) 2022-09-08T01:33:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:34 smithi055 ceph-mon[97033]: pgmap v7571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:34 smithi059 ceph-mon[97450]: pgmap v7571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:35.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:35.056063+0000) 2022-09-08T01:33:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:34.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:34.994155+0000) 2022-09-08T01:33:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:36.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:36.056251+0000) 2022-09-08T01:33:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:35.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:35.994346+0000) 2022-09-08T01:33:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:36 smithi055 ceph-mon[97033]: pgmap v7572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:33:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:36 smithi059 ceph-mon[97450]: pgmap v7572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:33:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:37.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:37.056486+0000) 2022-09-08T01:33:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:36.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:36.994603+0000) 2022-09-08T01:33:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:38.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:38.056746+0000) 2022-09-08T01:33:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:37.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:37.994858+0000) 2022-09-08T01:33:38.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:33:38.608+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:33:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:38 smithi055 ceph-mon[97033]: pgmap v7573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:38 smithi059 ceph-mon[97450]: pgmap v7573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:39.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:39.056962+0000) 2022-09-08T01:33:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:38.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:38.995085+0000) 2022-09-08T01:33:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:40.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:40.057172+0000) 2022-09-08T01:33:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:39.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:39.995363+0000) 2022-09-08T01:33:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:40 smithi055 ceph-mon[97033]: pgmap v7574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:40 smithi059 ceph-mon[97450]: pgmap v7574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:41.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:41.057418+0000) 2022-09-08T01:33:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:40.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:40.995630+0000) 2022-09-08T01:33:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:42.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:42.057616+0000) 2022-09-08T01:33:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:41.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:41.995846+0000) 2022-09-08T01:33:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:42 smithi055 ceph-mon[97033]: pgmap v7575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:33:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:42 smithi059 ceph-mon[97450]: pgmap v7575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:33:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:43.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:43.057789+0000) 2022-09-08T01:33:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:42.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:42.996012+0000) 2022-09-08T01:33:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:44.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:44.057983+0000) 2022-09-08T01:33:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:43.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:43.996155+0000) 2022-09-08T01:33:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:44 smithi055 ceph-mon[97033]: pgmap v7576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:44 smithi059 ceph-mon[97450]: pgmap v7576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:45.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:45.058225+0000) 2022-09-08T01:33:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:44.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:44.996373+0000) 2022-09-08T01:33:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:46.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:46.058440+0000) 2022-09-08T01:33:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:45.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:45.996617+0000) 2022-09-08T01:33:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:46 smithi055 ceph-mon[97033]: pgmap v7577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:33:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:46 smithi059 ceph-mon[97450]: pgmap v7577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:33:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:47.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:47.058700+0000) 2022-09-08T01:33:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:46.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:46.996880+0000) 2022-09-08T01:33:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:48.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:48.058926+0000) 2022-09-08T01:33:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:47.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:47.997109+0000) 2022-09-08T01:33:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:48 smithi055 ceph-mon[97033]: pgmap v7578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:48 smithi059 ceph-mon[97450]: pgmap v7578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:49.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:49.059137+0000) 2022-09-08T01:33:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:48.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:48.997292+0000) 2022-09-08T01:33:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:50.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:50.059391+0000) 2022-09-08T01:33:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:49.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:49.997511+0000) 2022-09-08T01:33:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:50 smithi055 ceph-mon[97033]: pgmap v7579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:50 smithi059 ceph-mon[97450]: pgmap v7579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:51.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:51.059665+0000) 2022-09-08T01:33:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:50.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:50.997741+0000) 2022-09-08T01:33:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:51.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:51.997984+0000) 2022-09-08T01:33:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:52.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:52.059886+0000) 2022-09-08T01:33:53.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:52 smithi055 ceph-mon[97033]: pgmap v7580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:33:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:52 smithi059 ceph-mon[97450]: pgmap v7580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:33:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:53.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:53.060068+0000) 2022-09-08T01:33:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:52.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:52.998232+0000) 2022-09-08T01:33:53.876 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:33:53.623+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:33:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:54.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:54.060282+0000) 2022-09-08T01:33:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:53.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:53.998454+0000) 2022-09-08T01:33:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:33:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:33:54.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:33:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:33:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:33:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:33:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:54 smithi055 ceph-mon[97033]: pgmap v7581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:54 smithi059 ceph-mon[97450]: pgmap v7581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:55.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:55.060541+0000) 2022-09-08T01:33:55.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:54.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:54.998682+0000) 2022-09-08T01:33:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:55.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:55.998905+0000) 2022-09-08T01:33:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:56.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:56.060799+0000) 2022-09-08T01:33:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:56 smithi055 ceph-mon[97033]: pgmap v7582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:33:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:56 smithi059 ceph-mon[97450]: pgmap v7582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:33:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:57.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:57.061078+0000) 2022-09-08T01:33:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:56.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:56.999170+0000) 2022-09-08T01:33:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:58.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:58.061223+0000) 2022-09-08T01:33:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:57.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:57.999367+0000) 2022-09-08T01:33:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:33:58 smithi055 ceph-mon[97033]: pgmap v7583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:33:58 smithi059 ceph-mon[97450]: pgmap v7583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:33:59.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:33:59.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:59.061448+0000) 2022-09-08T01:33:59.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:58.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:58.999580+0000) 2022-09-08T01:34:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:00.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:00.061621+0000) 2022-09-08T01:34:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:33:59.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:33:59.999738+0000) 2022-09-08T01:34:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:00 smithi055 ceph-mon[97033]: pgmap v7584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:00 smithi059 ceph-mon[97450]: pgmap v7584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:01.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:01.061820+0000) 2022-09-08T01:34:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:00.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:00.999974+0000) 2022-09-08T01:34:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:02.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:02.062034+0000) 2022-09-08T01:34:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:01.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:02.000226+0000) 2022-09-08T01:34:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:02 smithi055 ceph-mon[97033]: pgmap v7585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:34:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:02 smithi059 ceph-mon[97450]: pgmap v7585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:34:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:03.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:03.062244+0000) 2022-09-08T01:34:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:02.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:03.000474+0000) 2022-09-08T01:34:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:04.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:04.000746+0000) 2022-09-08T01:34:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:04.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:04.062452+0000) 2022-09-08T01:34:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:04 smithi055 ceph-mon[97033]: pgmap v7586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:04 smithi059 ceph-mon[97450]: pgmap v7586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:05.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:05.062678+0000) 2022-09-08T01:34:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:05.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:05.000888+0000) 2022-09-08T01:34:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:34:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:34:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:34:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:34:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:06.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:06.001093+0000) 2022-09-08T01:34:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:06.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:06.062819+0000) 2022-09-08T01:34:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:06 smithi055 ceph-mon[97033]: pgmap v7587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:34:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:06 smithi059 ceph-mon[97450]: pgmap v7587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:34:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:07.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:07.062966+0000) 2022-09-08T01:34:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:07.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:07.001249+0000) 2022-09-08T01:34:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:08.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:08.001467+0000) 2022-09-08T01:34:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:08.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:08.063201+0000) 2022-09-08T01:34:08.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:34:08.638+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:34:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:08 smithi055 ceph-mon[97033]: pgmap v7588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:34:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:08 smithi059 ceph-mon[97450]: pgmap v7588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:34:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:09.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:09.063390+0000) 2022-09-08T01:34:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:09.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:09.001729+0000) 2022-09-08T01:34:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:09 smithi059 ceph-mon[97450]: osdmap e557: 12 total, 11 up, 11 in 2022-09-08T01:34:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:10.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:10.063589+0000) 2022-09-08T01:34:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:10.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:10.001984+0000) 2022-09-08T01:34:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:09 smithi055 ceph-mon[97033]: osdmap e557: 12 total, 11 up, 11 in 2022-09-08T01:34:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:10 smithi059 ceph-mon[97450]: pgmap v7590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:34:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:11.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:11.063808+0000) 2022-09-08T01:34:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:11.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:11.002198+0000) 2022-09-08T01:34:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:10 smithi055 ceph-mon[97033]: pgmap v7590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:34:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:12.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:12.063982+0000) 2022-09-08T01:34:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:12.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:12.002421+0000) 2022-09-08T01:34:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:12 smithi059 ceph-mon[97450]: pgmap v7591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:34:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:13.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:13.002627+0000) 2022-09-08T01:34:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:13.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:13.064213+0000) 2022-09-08T01:34:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:12 smithi055 ceph-mon[97033]: pgmap v7591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:34:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:14.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:14.064346+0000) 2022-09-08T01:34:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:14.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:14.002820+0000) 2022-09-08T01:34:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:14 smithi059 ceph-mon[97450]: pgmap v7592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:34:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:15.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:15.064529+0000) 2022-09-08T01:34:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:15.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:15.003025+0000) 2022-09-08T01:34:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:14 smithi055 ceph-mon[97033]: pgmap v7592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:34:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:16.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:16.003218+0000) 2022-09-08T01:34:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:16.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:16.064764+0000) 2022-09-08T01:34:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:16 smithi059 ceph-mon[97450]: pgmap v7593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:34:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:16 smithi059 ceph-mon[97450]: osdmap e558: 12 total, 11 up, 11 in 2022-09-08T01:34:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:17.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:17.065004+0000) 2022-09-08T01:34:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:17.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:17.003472+0000) 2022-09-08T01:34:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:16 smithi055 ceph-mon[97033]: pgmap v7593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:34:17.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:16 smithi055 ceph-mon[97033]: osdmap e558: 12 total, 11 up, 11 in 2022-09-08T01:34:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:18.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:18.003703+0000) 2022-09-08T01:34:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:18.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:18.065272+0000) 2022-09-08T01:34:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:19.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:19.065511+0000) 2022-09-08T01:34:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:19.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:19.003874+0000) 2022-09-08T01:34:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:18 smithi059 ceph-mon[97450]: pgmap v7595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-08T01:34:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:18 smithi055 ceph-mon[97033]: pgmap v7595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-08T01:34:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:20.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:20.065767+0000) 2022-09-08T01:34:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:20.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:20.004092+0000) 2022-09-08T01:34:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:20 smithi059 ceph-mon[97450]: pgmap v7596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:34:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:21.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:21.065976+0000) 2022-09-08T01:34:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:21.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:21.004340+0000) 2022-09-08T01:34:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:20 smithi055 ceph-mon[97033]: pgmap v7596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:34:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:22.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:22.066182+0000) 2022-09-08T01:34:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:22.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:22.004569+0000) 2022-09-08T01:34:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:23 smithi059 ceph-mon[97450]: pgmap v7597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:34:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:23.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:23.004791+0000) 2022-09-08T01:34:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:23.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:23.066391+0000) 2022-09-08T01:34:23.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:23 smithi055 ceph-mon[97033]: pgmap v7597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:34:23.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:34:23.649+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:34:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:24.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:24.066587+0000) 2022-09-08T01:34:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:24.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:24.004966+0000) 2022-09-08T01:34:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:24 smithi055 ceph-mon[97033]: pgmap v7598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:34:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:24 smithi055 ceph-mon[97033]: pgmap v7599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:34:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:24 smithi055 ceph-mon[97033]: pgmap v7600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.1 KiB/s rd, 4 op/s 2022-09-08T01:34:25.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:34:25.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:24 smithi059 ceph-mon[97450]: pgmap v7598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:34:25.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:24 smithi059 ceph-mon[97450]: pgmap v7599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:34:25.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:24 smithi059 ceph-mon[97450]: pgmap v7600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.1 KiB/s rd, 4 op/s 2022-09-08T01:34:25.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:34:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:25.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:25.066694+0000) 2022-09-08T01:34:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:25.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:25.005198+0000) 2022-09-08T01:34:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:26.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:26.066951+0000) 2022-09-08T01:34:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:26.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:26.005391+0000) 2022-09-08T01:34:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:26 smithi055 ceph-mon[97033]: pgmap v7601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:34:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:26 smithi059 ceph-mon[97450]: pgmap v7601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:34:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:27.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:27.067190+0000) 2022-09-08T01:34:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:27.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:27.005603+0000) 2022-09-08T01:34:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:28.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:28.067350+0000) 2022-09-08T01:34:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:28.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:28.005837+0000) 2022-09-08T01:34:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:28 smithi055 ceph-mon[97033]: pgmap v7602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:34:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:28 smithi059 ceph-mon[97450]: pgmap v7602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:34:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:29.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:29.067589+0000) 2022-09-08T01:34:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:29.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:29.006083+0000) 2022-09-08T01:34:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:30.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:30.067781+0000) 2022-09-08T01:34:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:30.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:30.006285+0000) 2022-09-08T01:34:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:30 smithi055 ceph-mon[97033]: pgmap v7603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:34:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:30 smithi059 ceph-mon[97450]: pgmap v7603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:34:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:31.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:31.068051+0000) 2022-09-08T01:34:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:31.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:31.006507+0000) 2022-09-08T01:34:32.410 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:32.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:32.068258+0000) 2022-09-08T01:34:32.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:32.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:32.006726+0000) 2022-09-08T01:34:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:32 smithi055 ceph-mon[97033]: pgmap v7604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:34:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:32 smithi059 ceph-mon[97450]: pgmap v7604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:34:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:33.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:33.068498+0000) 2022-09-08T01:34:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:33.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:33.006927+0000) 2022-09-08T01:34:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:34.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:34.068749+0000) 2022-09-08T01:34:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:34.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:34.007151+0000) 2022-09-08T01:34:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:35 smithi059 ceph-mon[97450]: pgmap v7605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:34:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:35.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:35.007324+0000) 2022-09-08T01:34:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:35.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:35.068987+0000) 2022-09-08T01:34:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:35 smithi055 ceph-mon[97033]: pgmap v7605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:34:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:36.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:36.069163+0000) 2022-09-08T01:34:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:36.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:36.007535+0000) 2022-09-08T01:34:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:37 smithi059 ceph-mon[97450]: pgmap v7606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:34:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:37.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:37.069382+0000) 2022-09-08T01:34:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:37.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:37.007701+0000) 2022-09-08T01:34:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:37 smithi055 ceph-mon[97033]: pgmap v7606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:34:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:38.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:38.069563+0000) 2022-09-08T01:34:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:38.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:38.007909+0000) 2022-09-08T01:34:39.009 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:34:38.664+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:34:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:39 smithi059 ceph-mon[97450]: pgmap v7607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:39.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:39.069762+0000) 2022-09-08T01:34:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:39.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:39.008142+0000) 2022-09-08T01:34:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:39 smithi055 ceph-mon[97033]: pgmap v7607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:40.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:40.069902+0000) 2022-09-08T01:34:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:40.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:40.008380+0000) 2022-09-08T01:34:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:41 smithi059 ceph-mon[97450]: pgmap v7608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:34:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:41.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:41.070152+0000) 2022-09-08T01:34:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:41.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:41.008613+0000) 2022-09-08T01:34:41.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:41 smithi055 ceph-mon[97033]: pgmap v7608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:34:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:42.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:42.070435+0000) 2022-09-08T01:34:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:42.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:42.008834+0000) 2022-09-08T01:34:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:42 smithi055 ceph-mon[97033]: pgmap v7609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:42 smithi059 ceph-mon[97450]: pgmap v7609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:43.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:43.070759+0000) 2022-09-08T01:34:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:43.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:43.009042+0000) 2022-09-08T01:34:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:44.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:44.071003+0000) 2022-09-08T01:34:44.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:44.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:44.009254+0000) 2022-09-08T01:34:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:45 smithi059 ceph-mon[97450]: pgmap v7610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:45.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:45.071253+0000) 2022-09-08T01:34:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:45.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:45.009496+0000) 2022-09-08T01:34:45.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:45 smithi055 ceph-mon[97033]: pgmap v7610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:46.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:46.071481+0000) 2022-09-08T01:34:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:46.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:46.009752+0000) 2022-09-08T01:34:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:47 smithi059 ceph-mon[97450]: pgmap v7611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:34:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:47.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:47.071750+0000) 2022-09-08T01:34:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:47.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:47.009996+0000) 2022-09-08T01:34:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:47 smithi055 ceph-mon[97033]: pgmap v7611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:34:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:48.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:48.071961+0000) 2022-09-08T01:34:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:48.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:48.010224+0000) 2022-09-08T01:34:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:49 smithi059 ceph-mon[97450]: pgmap v7612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:49.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:49.010376+0000) 2022-09-08T01:34:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:49.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:49.072185+0000) 2022-09-08T01:34:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:49 smithi055 ceph-mon[97033]: pgmap v7612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:50.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:50.072393+0000) 2022-09-08T01:34:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:50.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:50.010587+0000) 2022-09-08T01:34:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:51 smithi059 ceph-mon[97450]: pgmap v7613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:34:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:51.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:51.072571+0000) 2022-09-08T01:34:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:51.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:51.010819+0000) 2022-09-08T01:34:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:51 smithi055 ceph-mon[97033]: pgmap v7613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:34:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:52.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:52.072775+0000) 2022-09-08T01:34:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:52.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:52.011074+0000) 2022-09-08T01:34:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:52 smithi055 ceph-mon[97033]: pgmap v7614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:52 smithi059 ceph-mon[97450]: pgmap v7614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:53.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:53.072999+0000) 2022-09-08T01:34:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:53.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:53.011307+0000) 2022-09-08T01:34:54.012 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:34:53.677+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:34:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:54.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:54.073237+0000) 2022-09-08T01:34:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:54.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:54.011538+0000) 2022-09-08T01:34:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:55 smithi059 ceph-mon[97450]: pgmap v7615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:55.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:55.073498+0000) 2022-09-08T01:34:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:55.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:55.011761+0000) 2022-09-08T01:34:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:55 smithi055 ceph-mon[97033]: pgmap v7615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:56.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:56.073772+0000) 2022-09-08T01:34:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:56.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:56.012020+0000) 2022-09-08T01:34:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:57 smithi059 ceph-mon[97450]: pgmap v7616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:34:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:57.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:57.073981+0000) 2022-09-08T01:34:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:57.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:57.012283+0000) 2022-09-08T01:34:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:57 smithi055 ceph-mon[97033]: pgmap v7616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:34:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:58.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:58.074205+0000) 2022-09-08T01:34:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:58.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:58.012517+0000) 2022-09-08T01:34:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:34:59 smithi059 ceph-mon[97450]: pgmap v7617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:34:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:34:59.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:59.074410+0000) 2022-09-08T01:34:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:34:59.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:34:59.012702+0000) 2022-09-08T01:34:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:34:59 smithi055 ceph-mon[97033]: pgmap v7617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:00.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:00.074622+0000) 2022-09-08T01:35:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:00.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:00.012945+0000) 2022-09-08T01:35:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:01.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:01.074896+0000) 2022-09-08T01:35:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:01.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:01.013127+0000) 2022-09-08T01:35:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:01 smithi055 ceph-mon[97033]: pgmap v7618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:35:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:01 smithi059 ceph-mon[97450]: pgmap v7618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:35:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:02.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:02.075141+0000) 2022-09-08T01:35:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:02.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:02.013321+0000) 2022-09-08T01:35:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:02 smithi055 ceph-mon[97033]: pgmap v7619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:02 smithi059 ceph-mon[97450]: pgmap v7619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:03.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:03.013533+0000) 2022-09-08T01:35:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:03.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:03.075352+0000) 2022-09-08T01:35:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:04.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:04.013693+0000) 2022-09-08T01:35:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:04.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:04.075508+0000) 2022-09-08T01:35:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:05 smithi059 ceph-mon[97450]: pgmap v7620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:05.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:05.075775+0000) 2022-09-08T01:35:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:05.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:05.013947+0000) 2022-09-08T01:35:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:05 smithi055 ceph-mon[97033]: pgmap v7620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:06.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:06.075987+0000) 2022-09-08T01:35:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:06.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:06.014117+0000) 2022-09-08T01:35:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:35:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:35:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:35:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:35:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:07.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:07.076183+0000) 2022-09-08T01:35:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:07.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:07.014380+0000) 2022-09-08T01:35:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:07 smithi055 ceph-mon[97033]: pgmap v7621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:35:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:07 smithi059 ceph-mon[97450]: pgmap v7621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:35:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:08.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:08.076429+0000) 2022-09-08T01:35:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:08.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:08.014593+0000) 2022-09-08T01:35:09.015 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:35:08.691+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:35:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:09.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:09.076711+0000) 2022-09-08T01:35:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:09.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:09.014816+0000) 2022-09-08T01:35:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:09 smithi055 ceph-mon[97033]: pgmap v7622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:35:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:09 smithi059 ceph-mon[97450]: pgmap v7622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:35:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:10.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:10.076950+0000) 2022-09-08T01:35:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:10.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:10.015034+0000) 2022-09-08T01:35:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:10 smithi055 ceph-mon[97033]: osdmap e559: 12 total, 11 up, 11 in 2022-09-08T01:35:10.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:10 smithi059 ceph-mon[97450]: osdmap e559: 12 total, 11 up, 11 in 2022-09-08T01:35:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:11.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:11.077187+0000) 2022-09-08T01:35:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:11.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:11.015263+0000) 2022-09-08T01:35:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:11 smithi055 ceph-mon[97033]: pgmap v7624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:35:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:11 smithi059 ceph-mon[97450]: pgmap v7624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:35:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:12.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:12.077362+0000) 2022-09-08T01:35:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:12.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:12.015440+0000) 2022-09-08T01:35:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:12 smithi055 ceph-mon[97033]: osdmap e560: 12 total, 11 up, 11 in 2022-09-08T01:35:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:12 smithi055 ceph-mon[97033]: pgmap v7626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:35:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:12 smithi059 ceph-mon[97450]: osdmap e560: 12 total, 11 up, 11 in 2022-09-08T01:35:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:12 smithi059 ceph-mon[97450]: pgmap v7626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:35:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:13.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:13.077567+0000) 2022-09-08T01:35:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:13.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:13.015721+0000) 2022-09-08T01:35:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:14.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:14.077754+0000) 2022-09-08T01:35:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:14.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:14.015940+0000) 2022-09-08T01:35:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:15 smithi059 ceph-mon[97450]: pgmap v7627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:35:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:15.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:15.077949+0000) 2022-09-08T01:35:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:15.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:15.016169+0000) 2022-09-08T01:35:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:15 smithi055 ceph-mon[97033]: pgmap v7627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:35:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:16.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:16.078170+0000) 2022-09-08T01:35:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:16.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:16.016317+0000) 2022-09-08T01:35:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:17.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:17.078369+0000) 2022-09-08T01:35:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:17.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:17.016456+0000) 2022-09-08T01:35:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:17 smithi055 ceph-mon[97033]: pgmap v7628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:35:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:17 smithi059 ceph-mon[97450]: pgmap v7628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:35:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:18.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:18.016716+0000) 2022-09-08T01:35:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:18.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:18.078617+0000) 2022-09-08T01:35:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:19.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:19.078809+0000) 2022-09-08T01:35:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:19.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:19.016941+0000) 2022-09-08T01:35:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:19 smithi055 ceph-mon[97033]: pgmap v7629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:35:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:19 smithi059 ceph-mon[97450]: pgmap v7629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:35:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:20.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:20.079025+0000) 2022-09-08T01:35:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:20.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:20.017211+0000) 2022-09-08T01:35:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:21.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:21.017492+0000) 2022-09-08T01:35:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:21.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:21.079342+0000) 2022-09-08T01:35:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:21 smithi055 ceph-mon[97033]: pgmap v7630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:35:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:21 smithi059 ceph-mon[97450]: pgmap v7630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:35:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:22.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:22.079505+0000) 2022-09-08T01:35:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:22.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:22.017713+0000) 2022-09-08T01:35:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:22 smithi055 ceph-mon[97033]: pgmap v7631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 101 B/s wr, 6 op/s 2022-09-08T01:35:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:22 smithi059 ceph-mon[97450]: pgmap v7631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 101 B/s wr, 6 op/s 2022-09-08T01:35:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:23.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:23.079747+0000) 2022-09-08T01:35:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:23.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:23.017946+0000) 2022-09-08T01:35:24.018 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:35:23.702+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:35:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:24.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:24.079934+0000) 2022-09-08T01:35:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:24.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:24.018123+0000) 2022-09-08T01:35:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:25 smithi059 ceph-mon[97450]: pgmap v7632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:35:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:35:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:35:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:35:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:35:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:35:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:25.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:25.080105+0000) 2022-09-08T01:35:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:25.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:25.018351+0000) 2022-09-08T01:35:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:25 smithi055 ceph-mon[97033]: pgmap v7632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:35:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:35:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:35:25.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:35:25.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:35:25.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:35:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:26.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:26.080284+0000) 2022-09-08T01:35:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:26.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:26.018524+0000) 2022-09-08T01:35:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:27 smithi059 ceph-mon[97450]: pgmap v7633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:35:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:35:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:35:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:27.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:27.080441+0000) 2022-09-08T01:35:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:27.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:27.018786+0000) 2022-09-08T01:35:27.421 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:27 smithi055 ceph-mon[97033]: pgmap v7633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:35:27.422 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:35:27.422 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:35:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:28.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:28.080686+0000) 2022-09-08T01:35:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:28.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:28.018918+0000) 2022-09-08T01:35:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:35:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:35:28.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T01:35:28.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T01:35:28.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T01:35:28.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T01:35:28.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:28 smithi055 ceph-mon[97033]: pgmap v7634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:35:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:35:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T01:35:28.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T01:35:28.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:35:28.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T01:35:28.790 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T01:35:28.790 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:28 smithi059 ceph-mon[97450]: pgmap v7634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:29.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:29.080842+0000) 2022-09-08T01:35:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:29.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:29.019151+0000) 2022-09-08T01:35:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:30.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:30.081049+0000) 2022-09-08T01:35:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:30.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:30.019358+0000) 2022-09-08T01:35:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:31.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:31.081281+0000) 2022-09-08T01:35:31.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:31.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:31.019569+0000) 2022-09-08T01:35:31.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:31 smithi055 ceph-mon[97033]: pgmap v7635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:35:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:31 smithi059 ceph-mon[97450]: pgmap v7635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:35:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:32.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:32.081464+0000) 2022-09-08T01:35:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:32.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:32.019817+0000) 2022-09-08T01:35:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:32 smithi055 ceph-mon[97033]: pgmap v7636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:32 smithi059 ceph-mon[97450]: pgmap v7636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:33.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:33.081618+0000) 2022-09-08T01:35:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:33.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:33.019949+0000) 2022-09-08T01:35:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:34.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:34.081855+0000) 2022-09-08T01:35:34.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:34.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:34.020084+0000) 2022-09-08T01:35:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:35.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:35.082114+0000) 2022-09-08T01:35:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:35.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:35.020328+0000) 2022-09-08T01:35:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:35 smithi055 ceph-mon[97033]: pgmap v7637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:35 smithi059 ceph-mon[97450]: pgmap v7637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:36.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:36.082379+0000) 2022-09-08T01:35:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:36.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:36.020509+0000) 2022-09-08T01:35:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:37.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:37.082486+0000) 2022-09-08T01:35:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:37.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:37.020761+0000) 2022-09-08T01:35:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:37 smithi055 ceph-mon[97033]: pgmap v7638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:35:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:37 smithi059 ceph-mon[97450]: pgmap v7638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:35:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:38.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:38.082709+0000) 2022-09-08T01:35:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:38.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:38.020978+0000) 2022-09-08T01:35:39.022 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:35:38.707+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:35:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:39.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:39.082872+0000) 2022-09-08T01:35:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:39.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:39.021210+0000) 2022-09-08T01:35:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:39 smithi055 ceph-mon[97033]: pgmap v7639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:39 smithi059 ceph-mon[97450]: pgmap v7639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:40.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:40.083104+0000) 2022-09-08T01:35:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:40.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:40.021466+0000) 2022-09-08T01:35:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:41.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:41.083321+0000) 2022-09-08T01:35:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:41.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:41.021679+0000) 2022-09-08T01:35:41.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:41 smithi055 ceph-mon[97033]: pgmap v7640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:35:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:41 smithi059 ceph-mon[97450]: pgmap v7640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:35:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:42.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:42.021857+0000) 2022-09-08T01:35:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:42.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:42.083480+0000) 2022-09-08T01:35:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:42 smithi055 ceph-mon[97033]: pgmap v7641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:42 smithi059 ceph-mon[97450]: pgmap v7641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:43.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:43.083734+0000) 2022-09-08T01:35:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:43.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:43.022090+0000) 2022-09-08T01:35:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:44.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:44.022307+0000) 2022-09-08T01:35:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:44.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:44.083979+0000) 2022-09-08T01:35:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:45.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:45.084241+0000) 2022-09-08T01:35:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:45.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:45.022546+0000) 2022-09-08T01:35:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:45 smithi055 ceph-mon[97033]: pgmap v7642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:45 smithi059 ceph-mon[97450]: pgmap v7642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:46.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:46.084485+0000) 2022-09-08T01:35:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:46.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:46.022808+0000) 2022-09-08T01:35:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:47.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:47.084699+0000) 2022-09-08T01:35:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:47.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:47.022943+0000) 2022-09-08T01:35:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:47 smithi055 ceph-mon[97033]: pgmap v7643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:35:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:47 smithi059 ceph-mon[97450]: pgmap v7643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:35:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:48.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:48.084938+0000) 2022-09-08T01:35:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:48.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:48.023158+0000) 2022-09-08T01:35:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:49.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:49.085165+0000) 2022-09-08T01:35:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:49.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:49.023396+0000) 2022-09-08T01:35:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:49 smithi055 ceph-mon[97033]: pgmap v7644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:49 smithi059 ceph-mon[97450]: pgmap v7644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:50.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:50.085406+0000) 2022-09-08T01:35:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:50.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:50.023621+0000) 2022-09-08T01:35:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:51.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:51.085677+0000) 2022-09-08T01:35:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:51.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:51.023848+0000) 2022-09-08T01:35:51.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:51 smithi055 ceph-mon[97033]: pgmap v7645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:35:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:51 smithi059 ceph-mon[97450]: pgmap v7645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:35:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:52.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:52.085913+0000) 2022-09-08T01:35:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:52.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:52.024039+0000) 2022-09-08T01:35:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:52 smithi055 ceph-mon[97033]: pgmap v7646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:52 smithi059 ceph-mon[97450]: pgmap v7646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:53.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:53.086091+0000) 2022-09-08T01:35:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:53.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:53.024227+0000) 2022-09-08T01:35:54.025 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:35:53.722+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:35:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:54.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:54.086234+0000) 2022-09-08T01:35:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:54.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:54.024457+0000) 2022-09-08T01:35:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:55.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:55.024692+0000) 2022-09-08T01:35:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:55.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:55.086450+0000) 2022-09-08T01:35:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:55 smithi059 ceph-mon[97450]: pgmap v7647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:55 smithi059 ceph-mon[97450]: pgmap v7648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:35:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:55 smithi059 ceph-mon[97450]: pgmap v7649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:35:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:35:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:55 smithi055 ceph-mon[97033]: pgmap v7647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:35:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:55 smithi055 ceph-mon[97033]: pgmap v7648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:35:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:55 smithi055 ceph-mon[97033]: pgmap v7649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:35:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:35:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:56.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:56.024856+0000) 2022-09-08T01:35:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:56.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:56.086711+0000) 2022-09-08T01:35:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:57.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:57.025015+0000) 2022-09-08T01:35:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:57.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:57.086960+0000) 2022-09-08T01:35:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:57 smithi059 ceph-mon[97450]: pgmap v7650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:35:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:57 smithi055 ceph-mon[97033]: pgmap v7650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:35:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:58.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:58.087192+0000) 2022-09-08T01:35:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:58.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:58.025167+0000) 2022-09-08T01:35:59.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:35:59.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:59.087389+0000) 2022-09-08T01:35:59.412 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:35:59.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:35:59.025368+0000) 2022-09-08T01:35:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:35:59 smithi059 ceph-mon[97450]: pgmap v7651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:35:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:35:59 smithi055 ceph-mon[97033]: pgmap v7651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:36:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:00.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:00.087535+0000) 2022-09-08T01:36:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:00.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:00.025587+0000) 2022-09-08T01:36:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:01.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:01.087673+0000) 2022-09-08T01:36:01.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:01.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:01.025797+0000) 2022-09-08T01:36:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:01 smithi059 ceph-mon[97450]: pgmap v7652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:36:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:01 smithi055 ceph-mon[97033]: pgmap v7652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:36:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:02.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:02.087873+0000) 2022-09-08T01:36:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:02.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:02.025946+0000) 2022-09-08T01:36:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:02 smithi055 ceph-mon[97033]: pgmap v7653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:36:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:02 smithi059 ceph-mon[97450]: pgmap v7653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:36:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:03.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:03.088119+0000) 2022-09-08T01:36:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:03.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:03.026206+0000) 2022-09-08T01:36:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:04.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:04.088329+0000) 2022-09-08T01:36:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:04.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:04.026449+0000) 2022-09-08T01:36:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:05.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:05.088535+0000) 2022-09-08T01:36:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:05.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:05.026728+0000) 2022-09-08T01:36:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:05 smithi055 ceph-mon[97033]: pgmap v7654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:36:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:36:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:36:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:05 smithi059 ceph-mon[97450]: pgmap v7654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:36:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:36:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:36:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:06.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:06.088734+0000) 2022-09-08T01:36:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:06.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:06.026899+0000) 2022-09-08T01:36:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:07.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:07.088905+0000) 2022-09-08T01:36:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:07.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:07.027135+0000) 2022-09-08T01:36:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:07 smithi055 ceph-mon[97033]: pgmap v7655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:36:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:07 smithi059 ceph-mon[97450]: pgmap v7655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:36:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:08.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:08.089125+0000) 2022-09-08T01:36:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:08.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:08.027405+0000) 2022-09-08T01:36:09.028 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:36:08.733+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:36:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:09.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:09.089382+0000) 2022-09-08T01:36:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:09.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:09.027613+0000) 2022-09-08T01:36:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:09 smithi055 ceph-mon[97033]: pgmap v7656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:36:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:09 smithi059 ceph-mon[97450]: pgmap v7656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:36:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:10.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:10.089597+0000) 2022-09-08T01:36:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:10.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:10.027748+0000) 2022-09-08T01:36:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:10 smithi055 ceph-mon[97033]: osdmap e561: 12 total, 11 up, 11 in 2022-09-08T01:36:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:10 smithi059 ceph-mon[97450]: osdmap e561: 12 total, 11 up, 11 in 2022-09-08T01:36:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:11.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:11.089869+0000) 2022-09-08T01:36:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:11.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:11.027965+0000) 2022-09-08T01:36:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:11 smithi055 ceph-mon[97033]: pgmap v7658: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:36:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:11 smithi059 ceph-mon[97450]: pgmap v7658: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:36:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:12.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:12.090054+0000) 2022-09-08T01:36:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:12.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:12.028198+0000) 2022-09-08T01:36:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:12 smithi055 ceph-mon[97033]: pgmap v7659: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:36:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:12 smithi059 ceph-mon[97450]: pgmap v7659: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:36:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:13.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:13.090301+0000) 2022-09-08T01:36:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:13.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:13.028435+0000) 2022-09-08T01:36:14.292 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:14.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:14.090518+0000) 2022-09-08T01:36:14.292 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:14.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:14.028578+0000) 2022-09-08T01:36:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:15.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:15.028808+0000) 2022-09-08T01:36:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:15.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:15.090747+0000) 2022-09-08T01:36:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:15 smithi055 ceph-mon[97033]: pgmap v7660: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:36:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:15 smithi059 ceph-mon[97450]: pgmap v7660: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:36:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:16.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:16.090983+0000) 2022-09-08T01:36:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:16.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:16.029014+0000) 2022-09-08T01:36:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:17.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:17.091170+0000) 2022-09-08T01:36:17.415 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:17.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:17.029270+0000) 2022-09-08T01:36:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:17 smithi055 ceph-mon[97033]: pgmap v7661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:36:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:17 smithi055 ceph-mon[97033]: osdmap e562: 12 total, 11 up, 11 in 2022-09-08T01:36:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:17 smithi059 ceph-mon[97450]: pgmap v7661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:36:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:17 smithi059 ceph-mon[97450]: osdmap e562: 12 total, 11 up, 11 in 2022-09-08T01:36:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:18.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:18.091350+0000) 2022-09-08T01:36:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:18.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:18.029510+0000) 2022-09-08T01:36:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:19.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:19.091514+0000) 2022-09-08T01:36:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:19.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:19.029769+0000) 2022-09-08T01:36:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:19 smithi055 ceph-mon[97033]: pgmap v7663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:36:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:19 smithi059 ceph-mon[97450]: pgmap v7663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:36:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:20.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:20.091696+0000) 2022-09-08T01:36:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:20.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:20.030022+0000) 2022-09-08T01:36:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:21.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:21.030239+0000) 2022-09-08T01:36:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:21.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:21.091896+0000) 2022-09-08T01:36:21.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:21 smithi055 ceph-mon[97033]: pgmap v7664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:36:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:21 smithi059 ceph-mon[97450]: pgmap v7664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:36:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:22.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:22.092054+0000) 2022-09-08T01:36:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:22.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:22.030467+0000) 2022-09-08T01:36:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:22 smithi055 ceph-mon[97033]: pgmap v7665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:36:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:22 smithi059 ceph-mon[97450]: pgmap v7665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:36:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:23.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:23.030718+0000) 2022-09-08T01:36:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:23.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:23.092269+0000) 2022-09-08T01:36:24.031 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:36:23.745+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:36:24.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:24.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:24.092405+0000) 2022-09-08T01:36:24.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:24.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:24.030857+0000) 2022-09-08T01:36:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:25.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:25.092578+0000) 2022-09-08T01:36:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:25.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:25.031102+0000) 2022-09-08T01:36:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:25 smithi055 ceph-mon[97033]: pgmap v7666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:36:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:25 smithi059 ceph-mon[97450]: pgmap v7666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:36:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:26.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:26.031268+0000) 2022-09-08T01:36:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:26.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:26.092778+0000) 2022-09-08T01:36:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:27.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:27.092988+0000) 2022-09-08T01:36:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:27.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:27.031449+0000) 2022-09-08T01:36:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:27 smithi055 ceph-mon[97033]: pgmap v7667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:36:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:27 smithi059 ceph-mon[97450]: pgmap v7667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:36:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:28.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:28.093161+0000) 2022-09-08T01:36:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:28.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:28.031661+0000) 2022-09-08T01:36:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:29.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:29.093365+0000) 2022-09-08T01:36:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:29.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:29.031780+0000) 2022-09-08T01:36:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:29 smithi055 ceph-mon[97033]: pgmap v7668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T01:36:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:29 smithi059 ceph-mon[97450]: pgmap v7668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T01:36:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:30.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:30.093596+0000) 2022-09-08T01:36:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:30.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:30.031981+0000) 2022-09-08T01:36:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:31.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:31.032201+0000) 2022-09-08T01:36:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:31.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:31.093869+0000) 2022-09-08T01:36:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:31 smithi055 ceph-mon[97033]: pgmap v7669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:36:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:31 smithi059 ceph-mon[97450]: pgmap v7669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:36:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:32.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:32.032385+0000) 2022-09-08T01:36:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:32.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:32.094086+0000) 2022-09-08T01:36:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:32 smithi055 ceph-mon[97033]: pgmap v7670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:36:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:32 smithi059 ceph-mon[97450]: pgmap v7670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:36:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:33.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:33.032647+0000) 2022-09-08T01:36:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:33.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:33.094305+0000) 2022-09-08T01:36:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:34.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:34.032809+0000) 2022-09-08T01:36:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:34.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:34.094447+0000) 2022-09-08T01:36:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:35.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:35.033060+0000) 2022-09-08T01:36:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:35.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:35.094696+0000) 2022-09-08T01:36:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:35 smithi055 ceph-mon[97033]: pgmap v7671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:36:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:35 smithi059 ceph-mon[97450]: pgmap v7671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:36:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:36.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:36.095078+0000) 2022-09-08T01:36:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:36.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:36.033220+0000) 2022-09-08T01:36:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:37.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:37.095275+0000) 2022-09-08T01:36:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:37.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:37.033390+0000) 2022-09-08T01:36:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:37 smithi055 ceph-mon[97033]: pgmap v7672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:36:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:37 smithi059 ceph-mon[97450]: pgmap v7672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:36:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:38.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:38.095504+0000) 2022-09-08T01:36:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:38.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:38.033608+0000) 2022-09-08T01:36:39.034 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:36:38.759+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:36:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:39.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:39.033879+0000) 2022-09-08T01:36:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:39.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:39.095749+0000) 2022-09-08T01:36:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:39 smithi055 ceph-mon[97033]: pgmap v7673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:36:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:39 smithi059 ceph-mon[97450]: pgmap v7673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:36:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:40.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:40.034057+0000) 2022-09-08T01:36:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:40.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:40.095982+0000) 2022-09-08T01:36:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:41.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:41.034202+0000) 2022-09-08T01:36:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:41.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:41.096245+0000) 2022-09-08T01:36:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:41 smithi055 ceph-mon[97033]: pgmap v7674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:36:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:41 smithi059 ceph-mon[97450]: pgmap v7674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:36:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:42.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:42.034370+0000) 2022-09-08T01:36:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:42.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:42.096443+0000) 2022-09-08T01:36:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:42 smithi055 ceph-mon[97033]: pgmap v7675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:36:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:42 smithi059 ceph-mon[97450]: pgmap v7675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:36:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:43.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:43.034535+0000) 2022-09-08T01:36:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:43.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:43.096617+0000) 2022-09-08T01:36:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:44.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:44.034757+0000) 2022-09-08T01:36:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:44.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:44.096844+0000) 2022-09-08T01:36:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:45.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:45.097054+0000) 2022-09-08T01:36:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:45.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:45.035013+0000) 2022-09-08T01:36:45.739 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:45 smithi055 ceph-mon[97033]: pgmap v7676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:36:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:45 smithi059 ceph-mon[97450]: pgmap v7676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:36:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:46.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:46.097270+0000) 2022-09-08T01:36:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:46.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:46.035231+0000) 2022-09-08T01:36:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:47.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:47.097458+0000) 2022-09-08T01:36:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:47.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:47.035513+0000) 2022-09-08T01:36:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:47 smithi055 ceph-mon[97033]: pgmap v7677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:36:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:47 smithi059 ceph-mon[97450]: pgmap v7677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:36:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:48.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:48.097692+0000) 2022-09-08T01:36:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:48.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:48.035716+0000) 2022-09-08T01:36:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:49.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:49.097925+0000) 2022-09-08T01:36:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:49.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:49.035935+0000) 2022-09-08T01:36:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:49 smithi055 ceph-mon[97033]: pgmap v7678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:36:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:49 smithi059 ceph-mon[97450]: pgmap v7678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:36:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:50.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:50.098186+0000) 2022-09-08T01:36:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:50.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:50.036142+0000) 2022-09-08T01:36:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:51.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:51.098461+0000) 2022-09-08T01:36:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:51.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:51.036350+0000) 2022-09-08T01:36:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:51 smithi055 ceph-mon[97033]: pgmap v7679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:36:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:51 smithi059 ceph-mon[97450]: pgmap v7679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:36:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:52.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:52.036601+0000) 2022-09-08T01:36:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:52.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:52.098709+0000) 2022-09-08T01:36:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:52 smithi055 ceph-mon[97033]: pgmap v7680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:36:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:52 smithi059 ceph-mon[97450]: pgmap v7680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:36:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:53.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:53.036850+0000) 2022-09-08T01:36:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:53.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:53.098901+0000) 2022-09-08T01:36:54.037 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:36:53.762+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:36:54.393 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:54.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:54.099050+0000) 2022-09-08T01:36:54.394 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:54.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:54.037052+0000) 2022-09-08T01:36:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:55.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:55.099295+0000) 2022-09-08T01:36:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:55.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:55.037200+0000) 2022-09-08T01:36:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:55 smithi055 ceph-mon[97033]: pgmap v7681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:36:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:36:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:36:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:36:55.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:36:55.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:36:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:55 smithi059 ceph-mon[97450]: pgmap v7681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:36:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:36:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:36:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:36:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:36:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:36:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:56.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:56.099511+0000) 2022-09-08T01:36:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:56.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:56.037403+0000) 2022-09-08T01:36:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:57.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:57.099726+0000) 2022-09-08T01:36:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:57.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:57.037629+0000) 2022-09-08T01:36:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:57 smithi055 ceph-mon[97033]: pgmap v7682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:36:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:57 smithi059 ceph-mon[97450]: pgmap v7682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:36:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:58.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:58.099986+0000) 2022-09-08T01:36:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:58.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:58.037762+0000) 2022-09-08T01:36:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:36:59.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:59.100269+0000) 2022-09-08T01:36:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:36:59.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:36:59.037990+0000) 2022-09-08T01:36:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:36:59 smithi055 ceph-mon[97033]: pgmap v7683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:36:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:36:59 smithi059 ceph-mon[97450]: pgmap v7683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:37:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:00.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:00.100562+0000) 2022-09-08T01:37:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:00.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:00.038251+0000) 2022-09-08T01:37:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:01.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:01.100695+0000) 2022-09-08T01:37:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:01.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:01.038405+0000) 2022-09-08T01:37:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:01 smithi055 ceph-mon[97033]: pgmap v7684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:37:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:01 smithi059 ceph-mon[97450]: pgmap v7684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:37:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:02.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:02.100906+0000) 2022-09-08T01:37:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:02.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:02.038591+0000) 2022-09-08T01:37:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:02 smithi055 ceph-mon[97033]: pgmap v7685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:02 smithi059 ceph-mon[97450]: pgmap v7685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:03.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:03.101101+0000) 2022-09-08T01:37:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:03.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:03.038749+0000) 2022-09-08T01:37:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:04.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:04.101295+0000) 2022-09-08T01:37:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:04.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:04.038957+0000) 2022-09-08T01:37:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:05.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:05.101502+0000) 2022-09-08T01:37:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:05.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:05.039196+0000) 2022-09-08T01:37:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:05 smithi055 ceph-mon[97033]: pgmap v7686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:37:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:37:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:05 smithi059 ceph-mon[97450]: pgmap v7686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:37:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:37:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:06.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:06.101655+0000) 2022-09-08T01:37:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:06.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:06.039412+0000) 2022-09-08T01:37:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:07.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:07.101825+0000) 2022-09-08T01:37:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:07.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:07.039704+0000) 2022-09-08T01:37:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:07 smithi055 ceph-mon[97033]: pgmap v7687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:37:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:07 smithi059 ceph-mon[97450]: pgmap v7687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:37:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:08.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:08.102005+0000) 2022-09-08T01:37:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:08.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:08.039890+0000) 2022-09-08T01:37:09.040 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:37:08.770+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:37:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:09.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:09.102236+0000) 2022-09-08T01:37:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:09.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:09.040131+0000) 2022-09-08T01:37:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:09 smithi055 ceph-mon[97033]: pgmap v7688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:37:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:09 smithi059 ceph-mon[97450]: pgmap v7688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:37:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:10.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:10.040357+0000) 2022-09-08T01:37:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:10.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:10.102413+0000) 2022-09-08T01:37:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:10 smithi055 ceph-mon[97033]: osdmap e563: 12 total, 11 up, 11 in 2022-09-08T01:37:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:10 smithi055 ceph-mon[97033]: pgmap v7690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:37:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:10 smithi059 ceph-mon[97450]: osdmap e563: 12 total, 11 up, 11 in 2022-09-08T01:37:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:10 smithi059 ceph-mon[97450]: pgmap v7690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:37:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:11.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:11.102617+0000) 2022-09-08T01:37:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:11.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:11.040592+0000) 2022-09-08T01:37:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:12.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:12.102733+0000) 2022-09-08T01:37:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:12.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:12.040785+0000) 2022-09-08T01:37:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:12 smithi055 ceph-mon[97033]: osdmap e564: 12 total, 11 up, 11 in 2022-09-08T01:37:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:12 smithi055 ceph-mon[97033]: pgmap v7692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:37:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:12 smithi059 ceph-mon[97450]: osdmap e564: 12 total, 11 up, 11 in 2022-09-08T01:37:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:12 smithi059 ceph-mon[97450]: pgmap v7692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:37:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:13.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:13.102947+0000) 2022-09-08T01:37:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:13.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:13.041065+0000) 2022-09-08T01:37:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:14.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:14.103126+0000) 2022-09-08T01:37:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:14.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:14.041310+0000) 2022-09-08T01:37:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:15.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:15.103386+0000) 2022-09-08T01:37:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:15.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:15.041561+0000) 2022-09-08T01:37:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:15 smithi055 ceph-mon[97033]: pgmap v7693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:37:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:15 smithi059 ceph-mon[97450]: pgmap v7693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:37:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:16.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:16.103623+0000) 2022-09-08T01:37:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:16.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:16.041808+0000) 2022-09-08T01:37:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:17.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:17.103880+0000) 2022-09-08T01:37:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:17.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:17.042015+0000) 2022-09-08T01:37:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:17 smithi055 ceph-mon[97033]: pgmap v7694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:37:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:17 smithi059 ceph-mon[97450]: pgmap v7694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:37:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:18.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:18.104169+0000) 2022-09-08T01:37:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:18.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:18.042249+0000) 2022-09-08T01:37:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:18 smithi055 ceph-mon[97033]: pgmap v7695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:37:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:18 smithi059 ceph-mon[97450]: pgmap v7695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:37:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:19.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:19.104432+0000) 2022-09-08T01:37:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:19.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:19.042508+0000) 2022-09-08T01:37:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:20.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:20.104686+0000) 2022-09-08T01:37:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:20.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:20.042750+0000) 2022-09-08T01:37:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:21.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:21.104825+0000) 2022-09-08T01:37:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:21.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:21.042964+0000) 2022-09-08T01:37:21.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:21 smithi055 ceph-mon[97033]: pgmap v7696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:37:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:21 smithi059 ceph-mon[97450]: pgmap v7696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:37:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:22.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:22.105095+0000) 2022-09-08T01:37:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:22.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:22.043160+0000) 2022-09-08T01:37:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:22 smithi055 ceph-mon[97033]: pgmap v7697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T01:37:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:22 smithi059 ceph-mon[97450]: pgmap v7697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T01:37:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:23.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:23.105317+0000) 2022-09-08T01:37:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:23.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:23.043435+0000) 2022-09-08T01:37:24.044 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:37:23.775+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:37:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:24.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:24.105471+0000) 2022-09-08T01:37:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:24.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:24.043674+0000) 2022-09-08T01:37:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:25.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:25.105626+0000) 2022-09-08T01:37:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:25.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:25.043843+0000) 2022-09-08T01:37:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:25 smithi055 ceph-mon[97033]: pgmap v7698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:37:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:37:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:25 smithi059 ceph-mon[97450]: pgmap v7698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:37:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:37:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:26.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:26.105843+0000) 2022-09-08T01:37:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:26.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:26.044061+0000) 2022-09-08T01:37:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:26 smithi055 ceph-mon[97033]: pgmap v7699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T01:37:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:26 smithi059 ceph-mon[97450]: pgmap v7699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T01:37:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:27.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:27.106086+0000) 2022-09-08T01:37:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:27.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:27.044283+0000) 2022-09-08T01:37:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:27 smithi055 ceph-mon[97033]: pgmap v7700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:37:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:27 smithi059 ceph-mon[97450]: pgmap v7700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:37:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:28.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:28.106239+0000) 2022-09-08T01:37:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:28.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:28.044492+0000) 2022-09-08T01:37:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:29.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:29.106426+0000) 2022-09-08T01:37:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:29.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:29.044742+0000) 2022-09-08T01:37:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:29 smithi055 ceph-mon[97033]: pgmap v7701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:37:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:29 smithi059 ceph-mon[97450]: pgmap v7701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:37:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:30.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:30.044891+0000) 2022-09-08T01:37:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:30.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:30.106712+0000) 2022-09-08T01:37:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:31.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:31.106910+0000) 2022-09-08T01:37:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:31.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:31.045104+0000) 2022-09-08T01:37:32.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:31 smithi055 ceph-mon[97033]: pgmap v7702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:37:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:31 smithi059 ceph-mon[97450]: pgmap v7702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:37:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:32.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:32.107120+0000) 2022-09-08T01:37:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:32.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:32.045346+0000) 2022-09-08T01:37:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:33.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:33.107342+0000) 2022-09-08T01:37:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:33.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:33.045566+0000) 2022-09-08T01:37:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:33 smithi055 ceph-mon[97033]: pgmap v7703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:37:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:33 smithi059 ceph-mon[97450]: pgmap v7703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:37:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:34.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:34.107566+0000) 2022-09-08T01:37:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:34.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:34.045780+0000) 2022-09-08T01:37:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:35.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:35.107769+0000) 2022-09-08T01:37:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:35.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:35.046039+0000) 2022-09-08T01:37:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:35 smithi055 ceph-mon[97033]: pgmap v7704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:37:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:35 smithi059 ceph-mon[97450]: pgmap v7704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:37:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:36.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:36.046279+0000) 2022-09-08T01:37:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:36.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:36.107991+0000) 2022-09-08T01:37:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:37.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:37.046441+0000) 2022-09-08T01:37:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:37.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:37.108178+0000) 2022-09-08T01:37:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:37 smithi055 ceph-mon[97033]: pgmap v7705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:37:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:37 smithi059 ceph-mon[97450]: pgmap v7705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:37:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:38.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:38.108387+0000) 2022-09-08T01:37:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:38.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:38.046660+0000) 2022-09-08T01:37:39.047 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:37:38.779+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:37:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:39.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:39.046851+0000) 2022-09-08T01:37:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:39.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:39.108600+0000) 2022-09-08T01:37:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:39 smithi055 ceph-mon[97033]: pgmap v7706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:39 smithi059 ceph-mon[97450]: pgmap v7706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:40.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:40.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:40.108839+0000) 2022-09-08T01:37:40.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:40.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:40.047044+0000) 2022-09-08T01:37:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:41.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:41.109040+0000) 2022-09-08T01:37:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:41.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:41.047289+0000) 2022-09-08T01:37:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:41 smithi055 ceph-mon[97033]: pgmap v7707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:37:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:41 smithi059 ceph-mon[97450]: pgmap v7707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:37:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:42.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:42.109179+0000) 2022-09-08T01:37:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:42.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:42.047549+0000) 2022-09-08T01:37:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:43.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:43.109384+0000) 2022-09-08T01:37:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:43.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:43.047807+0000) 2022-09-08T01:37:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:43 smithi055 ceph-mon[97033]: pgmap v7708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:43 smithi059 ceph-mon[97450]: pgmap v7708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:44.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:44.109633+0000) 2022-09-08T01:37:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:44.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:44.048055+0000) 2022-09-08T01:37:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:45.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:45.109912+0000) 2022-09-08T01:37:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:45.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:45.048255+0000) 2022-09-08T01:37:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:45 smithi055 ceph-mon[97033]: pgmap v7709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:45 smithi059 ceph-mon[97450]: pgmap v7709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:46.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:46.110075+0000) 2022-09-08T01:37:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:46.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:46.048475+0000) 2022-09-08T01:37:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:47.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:47.110315+0000) 2022-09-08T01:37:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:47.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:47.048804+0000) 2022-09-08T01:37:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:47 smithi055 ceph-mon[97033]: pgmap v7710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:37:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:47 smithi059 ceph-mon[97450]: pgmap v7710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:37:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:48.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:48.110480+0000) 2022-09-08T01:37:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:48.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:48.049057+0000) 2022-09-08T01:37:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:49.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:49.049311+0000) 2022-09-08T01:37:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:49.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:49.110731+0000) 2022-09-08T01:37:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:49 smithi055 ceph-mon[97033]: pgmap v7711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:49 smithi059 ceph-mon[97450]: pgmap v7711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:50.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:50.110853+0000) 2022-09-08T01:37:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:50.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:50.049548+0000) 2022-09-08T01:37:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:51.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:51.111029+0000) 2022-09-08T01:37:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:51.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:51.049808+0000) 2022-09-08T01:37:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:51 smithi055 ceph-mon[97033]: pgmap v7712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:37:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:51 smithi059 ceph-mon[97450]: pgmap v7712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:37:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:52.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:52.111184+0000) 2022-09-08T01:37:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:52.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:52.050068+0000) 2022-09-08T01:37:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:53.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:53.111409+0000) 2022-09-08T01:37:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:53.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:53.050279+0000) 2022-09-08T01:37:54.051 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:37:53.787+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:37:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:53 smithi055 ceph-mon[97033]: pgmap v7713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:53 smithi059 ceph-mon[97450]: pgmap v7713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:54.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:54.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:54.111669+0000) 2022-09-08T01:37:54.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:54.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:54.050482+0000) 2022-09-08T01:37:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:55.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:55.111868+0000) 2022-09-08T01:37:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:55.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:55.050700+0000) 2022-09-08T01:37:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:55 smithi055 ceph-mon[97033]: pgmap v7714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:55 smithi059 ceph-mon[97450]: pgmap v7714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:37:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:56.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:56.112071+0000) 2022-09-08T01:37:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:56.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:56.050944+0000) 2022-09-08T01:37:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:57.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:57.112350+0000) 2022-09-08T01:37:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:57.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:57.051181+0000) 2022-09-08T01:37:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:57 smithi055 ceph-mon[97033]: pgmap v7715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:37:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:57 smithi059 ceph-mon[97450]: pgmap v7715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:37:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:58.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:58.112584+0000) 2022-09-08T01:37:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:58.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:58.051421+0000) 2022-09-08T01:37:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:37:59.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:59.112755+0000) 2022-09-08T01:37:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:37:59.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:37:59.051706+0000) 2022-09-08T01:38:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:37:59 smithi055 ceph-mon[97033]: pgmap v7716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:37:59 smithi059 ceph-mon[97450]: pgmap v7716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:00.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:00.112962+0000) 2022-09-08T01:38:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:00.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:00.051958+0000) 2022-09-08T01:38:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:01.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:01.052186+0000) 2022-09-08T01:38:01.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:01.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:01.113209+0000) 2022-09-08T01:38:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:01 smithi055 ceph-mon[97033]: pgmap v7717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:38:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:01 smithi059 ceph-mon[97450]: pgmap v7717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:38:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:02.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:02.052437+0000) 2022-09-08T01:38:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:02.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:02.113484+0000) 2022-09-08T01:38:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:03.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:03.052730+0000) 2022-09-08T01:38:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:03.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:03.113690+0000) 2022-09-08T01:38:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:03 smithi055 ceph-mon[97033]: pgmap v7718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:03 smithi059 ceph-mon[97450]: pgmap v7718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:04.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:04.113814+0000) 2022-09-08T01:38:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:04.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:04.052976+0000) 2022-09-08T01:38:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:05.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:05.114046+0000) 2022-09-08T01:38:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:05.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:05.053162+0000) 2022-09-08T01:38:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:05 smithi055 ceph-mon[97033]: pgmap v7719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:06.251 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:38:06.251 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:38:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:05 smithi059 ceph-mon[97450]: pgmap v7719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:38:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:38:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:06.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:06.114250+0000) 2022-09-08T01:38:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:06.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:06.053386+0000) 2022-09-08T01:38:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:07.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:07.114448+0000) 2022-09-08T01:38:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:07.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:07.053667+0000) 2022-09-08T01:38:08.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:07 smithi055 ceph-mon[97033]: pgmap v7720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:38:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:07 smithi059 ceph-mon[97450]: pgmap v7720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:38:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:08.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:08.114575+0000) 2022-09-08T01:38:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:08.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:08.053828+0000) 2022-09-08T01:38:09.054 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:38:08.799+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:38:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:09.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:09.114809+0000) 2022-09-08T01:38:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:09.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:09.053987+0000) 2022-09-08T01:38:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:09 smithi055 ceph-mon[97033]: pgmap v7721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:38:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:09 smithi055 ceph-mon[97033]: osdmap e565: 12 total, 11 up, 11 in 2022-09-08T01:38:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:09 smithi059 ceph-mon[97450]: pgmap v7721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:38:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:09 smithi059 ceph-mon[97450]: osdmap e565: 12 total, 11 up, 11 in 2022-09-08T01:38:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:10.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:10.115030+0000) 2022-09-08T01:38:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:10.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:10.054142+0000) 2022-09-08T01:38:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:11.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:11.115224+0000) 2022-09-08T01:38:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:11.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:11.054390+0000) 2022-09-08T01:38:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:11 smithi055 ceph-mon[97033]: pgmap v7723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:38:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:11 smithi055 ceph-mon[97033]: osdmap e566: 12 total, 11 up, 11 in 2022-09-08T01:38:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:11 smithi059 ceph-mon[97450]: pgmap v7723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:38:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:11 smithi059 ceph-mon[97450]: osdmap e566: 12 total, 11 up, 11 in 2022-09-08T01:38:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:12.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:12.115439+0000) 2022-09-08T01:38:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:12.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:12.054540+0000) 2022-09-08T01:38:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:13.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:13.115710+0000) 2022-09-08T01:38:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:13.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:13.054699+0000) 2022-09-08T01:38:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:13 smithi055 ceph-mon[97033]: pgmap v7725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T01:38:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:13 smithi059 ceph-mon[97450]: pgmap v7725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T01:38:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:14.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:14.115898+0000) 2022-09-08T01:38:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:14.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:14.054809+0000) 2022-09-08T01:38:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:15.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:15.116139+0000) 2022-09-08T01:38:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:15.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:15.055018+0000) 2022-09-08T01:38:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:15 smithi055 ceph-mon[97033]: pgmap v7726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:38:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:15 smithi059 ceph-mon[97450]: pgmap v7726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:38:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:16.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:16.116416+0000) 2022-09-08T01:38:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:16.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:16.055198+0000) 2022-09-08T01:38:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:17.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:17.116734+0000) 2022-09-08T01:38:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:17.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:17.055443+0000) 2022-09-08T01:38:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:17 smithi059 ceph-mon[97450]: pgmap v7727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:38:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:18.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:18.116976+0000) 2022-09-08T01:38:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:18.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:18.055583+0000) 2022-09-08T01:38:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:17 smithi055 ceph-mon[97033]: pgmap v7727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:38:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:19.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:19.055772+0000) 2022-09-08T01:38:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:19.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:19.117220+0000) 2022-09-08T01:38:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:19 smithi059 ceph-mon[97450]: pgmap v7728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T01:38:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:20.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:20.117460+0000) 2022-09-08T01:38:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:20.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:20.056063+0000) 2022-09-08T01:38:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:19 smithi055 ceph-mon[97033]: pgmap v7728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T01:38:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:21.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:21.117654+0000) 2022-09-08T01:38:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:21.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:21.056203+0000) 2022-09-08T01:38:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:21 smithi059 ceph-mon[97450]: pgmap v7729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:38:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:22.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:22.117895+0000) 2022-09-08T01:38:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:22.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:22.056387+0000) 2022-09-08T01:38:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:21 smithi055 ceph-mon[97033]: pgmap v7729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:38:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:23.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:23.118119+0000) 2022-09-08T01:38:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:23.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:23.056578+0000) 2022-09-08T01:38:24.119 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:38:23.806+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:38:24.119 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:24.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:24.056820+0000) 2022-09-08T01:38:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:23 smithi059 ceph-mon[97450]: pgmap v7730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:38:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:24.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:24.118343+0000) 2022-09-08T01:38:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:23 smithi055 ceph-mon[97033]: pgmap v7730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T01:38:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:38:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:38:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:38:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:25.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:25.118607+0000) 2022-09-08T01:38:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:25.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:25.056991+0000) 2022-09-08T01:38:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:38:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:38:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:38:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:25 smithi059 ceph-mon[97450]: pgmap v7731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:38:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:26.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:26.118864+0000) 2022-09-08T01:38:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:26.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:26.057174+0000) 2022-09-08T01:38:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:25 smithi055 ceph-mon[97033]: pgmap v7731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:38:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:27.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:27.119054+0000) 2022-09-08T01:38:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:27.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:27.057373+0000) 2022-09-08T01:38:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:27 smithi059 ceph-mon[97450]: pgmap v7732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:38:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:28.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:28.119277+0000) 2022-09-08T01:38:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:28.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:28.057610+0000) 2022-09-08T01:38:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:28 smithi055 ceph-mon[97033]: pgmap v7732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:38:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:29.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:29.119502+0000) 2022-09-08T01:38:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:29.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:29.057887+0000) 2022-09-08T01:38:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:30 smithi059 ceph-mon[97450]: pgmap v7733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:30.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:30.119728+0000) 2022-09-08T01:38:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:30.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:30.058072+0000) 2022-09-08T01:38:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:30 smithi055 ceph-mon[97033]: pgmap v7733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:31.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:31.119924+0000) 2022-09-08T01:38:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:31.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:31.058307+0000) 2022-09-08T01:38:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:32 smithi059 ceph-mon[97450]: pgmap v7734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:38:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:32.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:32.120109+0000) 2022-09-08T01:38:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:32.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:32.058556+0000) 2022-09-08T01:38:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:32 smithi055 ceph-mon[97033]: pgmap v7734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:38:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:33.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:33.120330+0000) 2022-09-08T01:38:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:33.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:33.058798+0000) 2022-09-08T01:38:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:34 smithi059 ceph-mon[97450]: pgmap v7735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:34.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:34.120568+0000) 2022-09-08T01:38:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:34.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:34.058998+0000) 2022-09-08T01:38:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:34 smithi055 ceph-mon[97033]: pgmap v7735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:35.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:35.120794+0000) 2022-09-08T01:38:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:35.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:35.059234+0000) 2022-09-08T01:38:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:36 smithi059 ceph-mon[97450]: pgmap v7736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:36.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:36.120999+0000) 2022-09-08T01:38:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:36.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:36.059380+0000) 2022-09-08T01:38:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:36 smithi055 ceph-mon[97033]: pgmap v7736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:37.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:37.121194+0000) 2022-09-08T01:38:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:37.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:37.059604+0000) 2022-09-08T01:38:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:38 smithi059 ceph-mon[97450]: pgmap v7737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:38:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:38.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:38.121459+0000) 2022-09-08T01:38:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:38.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:38.059775+0000) 2022-09-08T01:38:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:38 smithi055 ceph-mon[97033]: pgmap v7737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:38:39.122 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:38:38.820+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:38:39.123 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:39.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:39.059973+0000) 2022-09-08T01:38:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:39.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:39.121731+0000) 2022-09-08T01:38:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:40 smithi059 ceph-mon[97450]: pgmap v7738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:40.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:40.121969+0000) 2022-09-08T01:38:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:40.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:40.060215+0000) 2022-09-08T01:38:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:40 smithi055 ceph-mon[97033]: pgmap v7738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:41.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:41.122204+0000) 2022-09-08T01:38:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:41.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:41.060432+0000) 2022-09-08T01:38:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:42 smithi059 ceph-mon[97450]: pgmap v7739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:38:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:42.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:42.122403+0000) 2022-09-08T01:38:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:42.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:42.060699+0000) 2022-09-08T01:38:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:42 smithi055 ceph-mon[97033]: pgmap v7739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:38:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:43.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:43.122622+0000) 2022-09-08T01:38:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:43.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:43.060945+0000) 2022-09-08T01:38:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:44 smithi059 ceph-mon[97450]: pgmap v7740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:44.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:44.122787+0000) 2022-09-08T01:38:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:44.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:44.061194+0000) 2022-09-08T01:38:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:44 smithi055 ceph-mon[97033]: pgmap v7740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:45.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:45.122970+0000) 2022-09-08T01:38:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:45.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:45.061449+0000) 2022-09-08T01:38:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:46 smithi059 ceph-mon[97450]: pgmap v7741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:46.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:46.123198+0000) 2022-09-08T01:38:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:46.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:46.061715+0000) 2022-09-08T01:38:46.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:46 smithi055 ceph-mon[97033]: pgmap v7741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:47.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:47.123447+0000) 2022-09-08T01:38:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:47.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:47.061953+0000) 2022-09-08T01:38:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:48.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:48.123631+0000) 2022-09-08T01:38:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:48.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:48.062182+0000) 2022-09-08T01:38:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:48 smithi055 ceph-mon[97033]: pgmap v7742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:38:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:48 smithi059 ceph-mon[97450]: pgmap v7742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:38:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:49.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:49.062434+0000) 2022-09-08T01:38:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:49.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:49.123807+0000) 2022-09-08T01:38:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:50.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:50.123977+0000) 2022-09-08T01:38:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:50.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:50.062688+0000) 2022-09-08T01:38:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:50 smithi055 ceph-mon[97033]: pgmap v7743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:50 smithi059 ceph-mon[97450]: pgmap v7743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:51.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:51.062892+0000) 2022-09-08T01:38:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:51.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:51.124202+0000) 2022-09-08T01:38:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:52.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:52.124425+0000) 2022-09-08T01:38:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:52.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:52.063131+0000) 2022-09-08T01:38:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:52 smithi055 ceph-mon[97033]: pgmap v7744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:38:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:52 smithi059 ceph-mon[97450]: pgmap v7744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:38:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:53.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:53.124679+0000) 2022-09-08T01:38:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:53.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:53.063393+0000) 2022-09-08T01:38:54.125 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:38:53.836+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:38:54.126 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:54.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:54.063527+0000) 2022-09-08T01:38:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:54.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:54.124877+0000) 2022-09-08T01:38:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:54 smithi055 ceph-mon[97033]: pgmap v7745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:54 smithi059 ceph-mon[97450]: pgmap v7745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:55.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:55.125006+0000) 2022-09-08T01:38:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:55.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:55.063744+0000) 2022-09-08T01:38:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:56.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:56.125270+0000) 2022-09-08T01:38:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:56.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:56.063942+0000) 2022-09-08T01:38:56.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:56 smithi055 ceph-mon[97033]: pgmap v7746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:56 smithi059 ceph-mon[97450]: pgmap v7746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:38:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:57.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:57.125531+0000) 2022-09-08T01:38:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:57.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:57.064172+0000) 2022-09-08T01:38:58.126 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:58.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:58.064400+0000) 2022-09-08T01:38:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:58.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:58.125765+0000) 2022-09-08T01:38:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:38:58 smithi055 ceph-mon[97033]: pgmap v7747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:38:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:38:58 smithi059 ceph-mon[97450]: pgmap v7747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:38:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:38:59.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:59.126016+0000) 2022-09-08T01:38:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:38:59.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:38:59.064618+0000) 2022-09-08T01:39:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:00.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:00.126214+0000) 2022-09-08T01:39:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:00.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:00.064903+0000) 2022-09-08T01:39:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:00 smithi059 ceph-mon[97450]: pgmap v7748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:39:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:39:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:00 smithi059 ceph-mon[97450]: pgmap v7749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:39:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:00 smithi059 ceph-mon[97450]: pgmap v7750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T01:39:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:39:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:00 smithi055 ceph-mon[97033]: pgmap v7748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:39:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:39:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:00 smithi055 ceph-mon[97033]: pgmap v7749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:39:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:00 smithi055 ceph-mon[97033]: pgmap v7750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T01:39:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:39:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:01.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:01.065137+0000) 2022-09-08T01:39:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:01.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:01.126466+0000) 2022-09-08T01:39:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:02.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:02.126732+0000) 2022-09-08T01:39:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:02.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:02.065379+0000) 2022-09-08T01:39:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:02 smithi055 ceph-mon[97033]: pgmap v7751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T01:39:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:02 smithi059 ceph-mon[97450]: pgmap v7751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T01:39:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:03.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:03.126863+0000) 2022-09-08T01:39:03.459 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:03.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:03.065680+0000) 2022-09-08T01:39:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:04.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:04.127027+0000) 2022-09-08T01:39:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:04.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:04.065872+0000) 2022-09-08T01:39:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:04 smithi055 ceph-mon[97033]: pgmap v7752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T01:39:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:04 smithi059 ceph-mon[97450]: pgmap v7752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T01:39:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:05.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:05.127240+0000) 2022-09-08T01:39:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:05.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:05.066003+0000) 2022-09-08T01:39:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:39:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:39:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:39:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:39:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:06.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:06.127453+0000) 2022-09-08T01:39:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:06.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:06.066136+0000) 2022-09-08T01:39:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:06 smithi055 ceph-mon[97033]: pgmap v7753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T01:39:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:06 smithi059 ceph-mon[97450]: pgmap v7753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T01:39:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:07.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:07.127728+0000) 2022-09-08T01:39:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:07.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:07.066316+0000) 2022-09-08T01:39:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:07 smithi055 ceph-mon[97033]: pgmap v7754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T01:39:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:07 smithi059 ceph-mon[97450]: pgmap v7754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T01:39:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:08.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:08.127962+0000) 2022-09-08T01:39:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:08.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:08.066531+0000) 2022-09-08T01:39:09.130 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:09.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:09.066769+0000) 2022-09-08T01:39:09.132 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:39:08.848+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:39:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:09.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:09.128329+0000) 2022-09-08T01:39:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:09 smithi055 ceph-mon[97033]: osdmap e567: 12 total, 11 up, 11 in 2022-09-08T01:39:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:09 smithi055 ceph-mon[97033]: pgmap v7756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:39:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:09 smithi059 ceph-mon[97450]: osdmap e567: 12 total, 11 up, 11 in 2022-09-08T01:39:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:09 smithi059 ceph-mon[97450]: pgmap v7756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:39:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:10.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:10.128550+0000) 2022-09-08T01:39:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:10.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:10.066966+0000) 2022-09-08T01:39:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:11.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:11.067228+0000) 2022-09-08T01:39:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:11.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:11.128773+0000) 2022-09-08T01:39:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:12.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:12.128912+0000) 2022-09-08T01:39:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:12.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:12.067413+0000) 2022-09-08T01:39:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:12 smithi055 ceph-mon[97033]: pgmap v7757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:39:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:12 smithi055 ceph-mon[97033]: osdmap e568: 12 total, 11 up, 11 in 2022-09-08T01:39:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:12 smithi059 ceph-mon[97450]: pgmap v7757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:39:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:12 smithi059 ceph-mon[97450]: osdmap e568: 12 total, 11 up, 11 in 2022-09-08T01:39:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:13.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:13.129157+0000) 2022-09-08T01:39:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:13.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:13.067679+0000) 2022-09-08T01:39:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:14.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:14.129385+0000) 2022-09-08T01:39:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:14.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:14.067810+0000) 2022-09-08T01:39:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:14 smithi055 ceph-mon[97033]: pgmap v7759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:39:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:14 smithi059 ceph-mon[97450]: pgmap v7759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:39:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:15.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:15.129618+0000) 2022-09-08T01:39:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:15.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:15.068024+0000) 2022-09-08T01:39:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:16.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:16.129849+0000) 2022-09-08T01:39:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:16.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:16.068159+0000) 2022-09-08T01:39:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:16 smithi055 ceph-mon[97033]: pgmap v7760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:39:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:16 smithi059 ceph-mon[97450]: pgmap v7760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:39:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:17.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:17.068395+0000) 2022-09-08T01:39:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:17.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:17.130046+0000) 2022-09-08T01:39:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:17 smithi055 ceph-mon[97033]: pgmap v7761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T01:39:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:17 smithi059 ceph-mon[97450]: pgmap v7761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T01:39:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:18.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:18.130276+0000) 2022-09-08T01:39:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:18.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:18.068610+0000) 2022-09-08T01:39:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:19.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:19.130440+0000) 2022-09-08T01:39:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:19.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:19.068906+0000) 2022-09-08T01:39:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:20.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:20.130686+0000) 2022-09-08T01:39:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:20.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:20.069174+0000) 2022-09-08T01:39:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:20 smithi055 ceph-mon[97033]: pgmap v7762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:39:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:20 smithi059 ceph-mon[97450]: pgmap v7762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:39:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:21.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:21.130844+0000) 2022-09-08T01:39:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:21.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:21.069358+0000) 2022-09-08T01:39:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:22.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:22.131082+0000) 2022-09-08T01:39:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:22.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:22.069557+0000) 2022-09-08T01:39:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:22 smithi055 ceph-mon[97033]: pgmap v7763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:39:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:22 smithi059 ceph-mon[97450]: pgmap v7763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:39:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:23.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:23.131318+0000) 2022-09-08T01:39:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:23.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:23.069743+0000) 2022-09-08T01:39:24.132 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:39:23.862+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:39:24.133 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:24.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:24.069943+0000) 2022-09-08T01:39:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:24.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:24.131545+0000) 2022-09-08T01:39:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:24 smithi055 ceph-mon[97033]: pgmap v7764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-08T01:39:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:24 smithi059 ceph-mon[97450]: pgmap v7764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-08T01:39:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:25.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:25.131780+0000) 2022-09-08T01:39:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:25.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:25.070107+0000) 2022-09-08T01:39:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:26.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:26.131948+0000) 2022-09-08T01:39:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:26.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:26.070311+0000) 2022-09-08T01:39:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:26 smithi055 ceph-mon[97033]: pgmap v7765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:39:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:26 smithi059 ceph-mon[97450]: pgmap v7765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:39:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:27.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:27.132164+0000) 2022-09-08T01:39:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:27.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:27.070506+0000) 2022-09-08T01:39:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:27 smithi055 ceph-mon[97033]: pgmap v7766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:27 smithi059 ceph-mon[97450]: pgmap v7766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:28.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:28.070751+0000) 2022-09-08T01:39:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:28.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:28.132325+0000) 2022-09-08T01:39:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:29.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:29.132478+0000) 2022-09-08T01:39:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:29.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:29.070979+0000) 2022-09-08T01:39:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:30.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:30.132607+0000) 2022-09-08T01:39:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:30.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:30.071129+0000) 2022-09-08T01:39:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:30 smithi055 ceph-mon[97033]: pgmap v7767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:30 smithi059 ceph-mon[97450]: pgmap v7767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:31.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:31.132782+0000) 2022-09-08T01:39:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:31.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:31.071326+0000) 2022-09-08T01:39:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:32.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:32.132997+0000) 2022-09-08T01:39:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:32.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:32.071529+0000) 2022-09-08T01:39:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:32 smithi055 ceph-mon[97033]: pgmap v7768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:39:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:32 smithi059 ceph-mon[97450]: pgmap v7768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:39:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:33.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:33.071705+0000) 2022-09-08T01:39:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:33.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:33.133183+0000) 2022-09-08T01:39:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:34.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:34.071897+0000) 2022-09-08T01:39:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:34.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:34.133375+0000) 2022-09-08T01:39:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:34 smithi055 ceph-mon[97033]: pgmap v7769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:34 smithi059 ceph-mon[97450]: pgmap v7769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:35.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:35.133545+0000) 2022-09-08T01:39:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:35.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:35.072105+0000) 2022-09-08T01:39:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:36.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:36.072357+0000) 2022-09-08T01:39:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:36.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:36.133700+0000) 2022-09-08T01:39:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:36 smithi055 ceph-mon[97033]: pgmap v7770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:39:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:36 smithi059 ceph-mon[97450]: pgmap v7770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:39:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:37.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:37.133886+0000) 2022-09-08T01:39:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:37.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:37.072604+0000) 2022-09-08T01:39:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:37 smithi055 ceph-mon[97033]: pgmap v7771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:37 smithi059 ceph-mon[97450]: pgmap v7771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:38.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:38.072863+0000) 2022-09-08T01:39:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:38.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:38.134099+0000) 2022-09-08T01:39:39.135 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:39:38.876+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:39:39.135 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:39.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:39.073132+0000) 2022-09-08T01:39:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:39.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:39.134329+0000) 2022-09-08T01:39:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:40.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:40.134517+0000) 2022-09-08T01:39:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:40.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:40.073401+0000) 2022-09-08T01:39:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:40 smithi055 ceph-mon[97033]: pgmap v7772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:40 smithi059 ceph-mon[97450]: pgmap v7772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:41.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:41.134717+0000) 2022-09-08T01:39:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:41.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:41.073611+0000) 2022-09-08T01:39:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:42.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:42.134925+0000) 2022-09-08T01:39:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:42.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:42.073833+0000) 2022-09-08T01:39:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:42 smithi055 ceph-mon[97033]: pgmap v7773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:39:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:42 smithi059 ceph-mon[97450]: pgmap v7773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:39:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:43.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:43.135076+0000) 2022-09-08T01:39:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:43.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:43.074072+0000) 2022-09-08T01:39:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:44.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:44.135294+0000) 2022-09-08T01:39:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:44.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:44.074320+0000) 2022-09-08T01:39:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:44 smithi055 ceph-mon[97033]: pgmap v7774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:44 smithi059 ceph-mon[97450]: pgmap v7774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:45.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:45.074561+0000) 2022-09-08T01:39:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:45.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:45.135462+0000) 2022-09-08T01:39:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:46.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:46.135722+0000) 2022-09-08T01:39:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:46.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:46.074761+0000) 2022-09-08T01:39:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:46 smithi055 ceph-mon[97033]: pgmap v7775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:39:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:46 smithi059 ceph-mon[97450]: pgmap v7775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:39:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:47.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:47.075012+0000) 2022-09-08T01:39:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:47.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:47.135931+0000) 2022-09-08T01:39:48.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:47 smithi055 ceph-mon[97033]: pgmap v7776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:47 smithi059 ceph-mon[97450]: pgmap v7776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:48.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:48.136177+0000) 2022-09-08T01:39:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:48.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:48.075221+0000) 2022-09-08T01:39:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:49.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:49.075458+0000) 2022-09-08T01:39:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:49.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:49.136448+0000) 2022-09-08T01:39:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:50.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:50.136699+0000) 2022-09-08T01:39:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:50.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:50.075697+0000) 2022-09-08T01:39:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:50 smithi055 ceph-mon[97033]: pgmap v7777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:50 smithi059 ceph-mon[97450]: pgmap v7777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:51.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:51.075885+0000) 2022-09-08T01:39:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:51.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:51.136978+0000) 2022-09-08T01:39:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:52.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:52.137198+0000) 2022-09-08T01:39:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:52.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:52.076056+0000) 2022-09-08T01:39:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:52 smithi055 ceph-mon[97033]: pgmap v7778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:39:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:52 smithi059 ceph-mon[97450]: pgmap v7778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:39:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:53.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:53.137435+0000) 2022-09-08T01:39:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:53.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:53.076313+0000) 2022-09-08T01:39:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:54.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:54.137670+0000) 2022-09-08T01:39:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:54.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:54.076527+0000) 2022-09-08T01:39:54.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:39:53.890+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:39:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:54 smithi055 ceph-mon[97033]: pgmap v7779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:54 smithi059 ceph-mon[97450]: pgmap v7779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:55.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:55.137895+0000) 2022-09-08T01:39:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:55.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:55.076794+0000) 2022-09-08T01:39:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:56.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:56.138095+0000) 2022-09-08T01:39:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:56.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:56.077047+0000) 2022-09-08T01:39:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:56 smithi055 ceph-mon[97033]: pgmap v7780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:39:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:56 smithi059 ceph-mon[97450]: pgmap v7780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:39:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:57.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:57.077251+0000) 2022-09-08T01:39:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:57.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:57.138302+0000) 2022-09-08T01:39:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:39:57 smithi055 ceph-mon[97033]: pgmap v7781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:39:57 smithi059 ceph-mon[97450]: pgmap v7781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:39:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:58.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:58.077473+0000) 2022-09-08T01:39:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:58.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:58.138489+0000) 2022-09-08T01:39:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:39:59.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:59.138661+0000) 2022-09-08T01:39:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:39:59.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:39:59.077742+0000) 2022-09-08T01:40:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:00.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:00.138909+0000) 2022-09-08T01:40:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:00.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:00.077981+0000) 2022-09-08T01:40:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:00 smithi055 ceph-mon[97033]: pgmap v7782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:40:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:40:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:40:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:40:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:00 smithi055 ceph-mon[97033]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T01:40:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:00 smithi059 ceph-mon[97450]: pgmap v7782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:40:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:40:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:40:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:40:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:00 smithi059 ceph-mon[97450]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T01:40:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:01.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:01.139095+0000) 2022-09-08T01:40:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:01.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:01.078102+0000) 2022-09-08T01:40:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:01 smithi055 ceph-mon[97033]: pgmap v7783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T01:40:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:01 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T01:40:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:01 smithi059 ceph-mon[97450]: pgmap v7783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T01:40:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:01 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T01:40:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:02.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:02.139346+0000) 2022-09-08T01:40:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:02.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:02.078284+0000) 2022-09-08T01:40:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:03.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:03.139556+0000) 2022-09-08T01:40:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:03.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:03.078426+0000) 2022-09-08T01:40:03.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:03 smithi055 ceph-mon[97033]: pgmap v7784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:40:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:03 smithi059 ceph-mon[97450]: pgmap v7784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:40:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:04.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:04.139788+0000) 2022-09-08T01:40:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:04.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:04.078657+0000) 2022-09-08T01:40:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:05.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:05.140012+0000) 2022-09-08T01:40:05.473 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:05.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:05.078802+0000) 2022-09-08T01:40:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:05 smithi055 ceph-mon[97033]: pgmap v7785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:40:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:40:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:40:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:05 smithi059 ceph-mon[97450]: pgmap v7785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:40:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:40:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:40:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:06.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:06.140224+0000) 2022-09-08T01:40:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:06.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:06.079025+0000) 2022-09-08T01:40:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:06 smithi055 ceph-mon[97033]: pgmap v7786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T01:40:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:06 smithi059 ceph-mon[97450]: pgmap v7786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T01:40:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:07.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:07.140469+0000) 2022-09-08T01:40:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:07.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:07.079333+0000) 2022-09-08T01:40:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:08.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:08.140765+0000) 2022-09-08T01:40:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:08.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:08.079539+0000) 2022-09-08T01:40:09.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:40:08.899+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:40:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:09.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:09.140914+0000) 2022-09-08T01:40:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:09.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:09.079759+0000) 2022-09-08T01:40:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:08 smithi055 ceph-mon[97033]: pgmap v7787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T01:40:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:08 smithi059 ceph-mon[97450]: pgmap v7787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T01:40:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:09 smithi055 ceph-mon[97033]: osdmap e569: 12 total, 11 up, 11 in 2022-09-08T01:40:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:09 smithi059 ceph-mon[97450]: osdmap e569: 12 total, 11 up, 11 in 2022-09-08T01:40:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:10.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:10.141103+0000) 2022-09-08T01:40:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:10.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:10.079946+0000) 2022-09-08T01:40:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:10 smithi055 ceph-mon[97033]: pgmap v7789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:40:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:10 smithi059 ceph-mon[97450]: pgmap v7789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:40:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:11.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:11.141333+0000) 2022-09-08T01:40:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:11.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:11.080165+0000) 2022-09-08T01:40:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:12.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:12.141517+0000) 2022-09-08T01:40:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:12.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:12.080377+0000) 2022-09-08T01:40:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:12 smithi055 ceph-mon[97033]: pgmap v7790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:40:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:12 smithi055 ceph-mon[97033]: osdmap e570: 12 total, 11 up, 11 in 2022-09-08T01:40:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:12 smithi059 ceph-mon[97450]: pgmap v7790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:40:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:12 smithi059 ceph-mon[97450]: osdmap e570: 12 total, 11 up, 11 in 2022-09-08T01:40:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:13.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:13.141704+0000) 2022-09-08T01:40:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:13.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:13.080581+0000) 2022-09-08T01:40:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:14.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:14.141923+0000) 2022-09-08T01:40:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:14.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:14.080717+0000) 2022-09-08T01:40:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:14 smithi055 ceph-mon[97033]: pgmap v7792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:40:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:14 smithi059 ceph-mon[97450]: pgmap v7792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:40:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:15.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:15.142166+0000) 2022-09-08T01:40:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:15.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:15.080936+0000) 2022-09-08T01:40:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:16.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:16.142394+0000) 2022-09-08T01:40:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:16.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:16.081114+0000) 2022-09-08T01:40:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:16 smithi055 ceph-mon[97033]: pgmap v7793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:40:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:16 smithi059 ceph-mon[97450]: pgmap v7793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:40:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:17.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:17.142678+0000) 2022-09-08T01:40:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:17.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:17.081305+0000) 2022-09-08T01:40:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:18.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:18.142920+0000) 2022-09-08T01:40:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:18.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:18.081507+0000) 2022-09-08T01:40:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:18 smithi055 ceph-mon[97033]: pgmap v7794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T01:40:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:18 smithi059 ceph-mon[97450]: pgmap v7794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T01:40:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:19.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:19.143166+0000) 2022-09-08T01:40:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:19.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:19.081753+0000) 2022-09-08T01:40:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:20.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:20.143416+0000) 2022-09-08T01:40:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:20.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:20.081972+0000) 2022-09-08T01:40:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:20 smithi055 ceph-mon[97033]: pgmap v7795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:40:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:20 smithi059 ceph-mon[97450]: pgmap v7795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:40:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:21.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:21.143721+0000) 2022-09-08T01:40:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:21.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:21.082137+0000) 2022-09-08T01:40:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:22.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:22.143986+0000) 2022-09-08T01:40:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:22.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:22.082312+0000) 2022-09-08T01:40:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:22 smithi055 ceph-mon[97033]: pgmap v7796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:40:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:22 smithi059 ceph-mon[97450]: pgmap v7796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:40:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:23.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:23.144283+0000) 2022-09-08T01:40:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:23.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:23.082564+0000) 2022-09-08T01:40:24.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:40:23.901+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:40:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:24.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:24.144526+0000) 2022-09-08T01:40:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:24.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:24.082815+0000) 2022-09-08T01:40:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:24 smithi055 ceph-mon[97033]: pgmap v7797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:40:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:24 smithi059 ceph-mon[97450]: pgmap v7797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:40:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:25.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:25.144781+0000) 2022-09-08T01:40:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:25.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:25.083045+0000) 2022-09-08T01:40:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:26.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:26.144971+0000) 2022-09-08T01:40:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:26.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:26.083177+0000) 2022-09-08T01:40:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:26 smithi055 ceph-mon[97033]: pgmap v7798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:40:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:26 smithi059 ceph-mon[97450]: pgmap v7798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:40:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:27.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:27.145226+0000) 2022-09-08T01:40:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:27.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:27.083420+0000) 2022-09-08T01:40:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:28.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:28.145502+0000) 2022-09-08T01:40:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:28.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:28.083663+0000) 2022-09-08T01:40:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:28 smithi055 ceph-mon[97033]: pgmap v7799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:28 smithi059 ceph-mon[97450]: pgmap v7799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:29.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:29.145751+0000) 2022-09-08T01:40:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:29.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:29.083878+0000) 2022-09-08T01:40:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:30.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:30.145917+0000) 2022-09-08T01:40:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:30.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:30.084092+0000) 2022-09-08T01:40:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:30 smithi055 ceph-mon[97033]: pgmap v7800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:40:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:30 smithi059 ceph-mon[97450]: pgmap v7800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:40:31.413 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:31.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:31.146040+0000) 2022-09-08T01:40:31.414 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:31.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:31.084296+0000) 2022-09-08T01:40:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:32.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:32.146233+0000) 2022-09-08T01:40:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:32.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:32.084499+0000) 2022-09-08T01:40:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:32 smithi055 ceph-mon[97033]: pgmap v7801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:32 smithi059 ceph-mon[97450]: pgmap v7801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:33.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:33.146459+0000) 2022-09-08T01:40:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:33.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:33.084731+0000) 2022-09-08T01:40:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:34.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:34.146697+0000) 2022-09-08T01:40:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:34.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:34.084974+0000) 2022-09-08T01:40:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:34 smithi055 ceph-mon[97033]: pgmap v7802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:34 smithi059 ceph-mon[97450]: pgmap v7802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:35.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:35.146914+0000) 2022-09-08T01:40:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:35.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:35.085255+0000) 2022-09-08T01:40:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:36.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:36.147137+0000) 2022-09-08T01:40:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:36.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:36.085490+0000) 2022-09-08T01:40:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:36 smithi055 ceph-mon[97033]: pgmap v7803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:40:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:36 smithi059 ceph-mon[97450]: pgmap v7803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:40:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:37.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:37.147310+0000) 2022-09-08T01:40:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:37.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:37.085743+0000) 2022-09-08T01:40:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:38.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:38.085958+0000) 2022-09-08T01:40:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:38.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:38.147523+0000) 2022-09-08T01:40:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:38 smithi055 ceph-mon[97033]: pgmap v7804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:38 smithi059 ceph-mon[97450]: pgmap v7804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:39.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:40:38.911+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:40:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:39.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:39.086185+0000) 2022-09-08T01:40:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:39.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:39.147719+0000) 2022-09-08T01:40:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:40.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:40.147945+0000) 2022-09-08T01:40:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:40.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:40.086388+0000) 2022-09-08T01:40:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:40 smithi055 ceph-mon[97033]: pgmap v7805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:40:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:40 smithi059 ceph-mon[97450]: pgmap v7805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:40:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:41.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:41.148272+0000) 2022-09-08T01:40:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:41.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:41.086529+0000) 2022-09-08T01:40:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:42.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:42.148501+0000) 2022-09-08T01:40:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:42.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:42.086739+0000) 2022-09-08T01:40:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:42 smithi059 ceph-mon[97450]: pgmap v7806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:43.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:43.148757+0000) 2022-09-08T01:40:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:43.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:43.086961+0000) 2022-09-08T01:40:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:42 smithi055 ceph-mon[97033]: pgmap v7806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:44.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:44.148947+0000) 2022-09-08T01:40:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:44.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:44.087144+0000) 2022-09-08T01:40:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:44 smithi059 ceph-mon[97450]: pgmap v7807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:45.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:45.149144+0000) 2022-09-08T01:40:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:45.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:45.087341+0000) 2022-09-08T01:40:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:44 smithi055 ceph-mon[97033]: pgmap v7807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:46.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:46.149394+0000) 2022-09-08T01:40:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:46.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:46.087569+0000) 2022-09-08T01:40:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:46 smithi059 ceph-mon[97450]: pgmap v7808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:40:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:47.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:47.149671+0000) 2022-09-08T01:40:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:47.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:47.087810+0000) 2022-09-08T01:40:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:46 smithi055 ceph-mon[97033]: pgmap v7808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:40:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:48.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:48.087997+0000) 2022-09-08T01:40:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:48.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:48.149913+0000) 2022-09-08T01:40:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:48 smithi059 ceph-mon[97450]: pgmap v7809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:49.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:49.150150+0000) 2022-09-08T01:40:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:49.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:49.088283+0000) 2022-09-08T01:40:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:48 smithi055 ceph-mon[97033]: pgmap v7809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:50.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:50.150386+0000) 2022-09-08T01:40:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:50.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:50.088451+0000) 2022-09-08T01:40:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:50 smithi059 ceph-mon[97450]: pgmap v7810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:40:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:51.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:51.150605+0000) 2022-09-08T01:40:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:51.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:51.088697+0000) 2022-09-08T01:40:51.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:50 smithi055 ceph-mon[97033]: pgmap v7810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:40:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:52.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:52.150845+0000) 2022-09-08T01:40:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:52.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:52.088872+0000) 2022-09-08T01:40:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:52 smithi059 ceph-mon[97450]: pgmap v7811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:53.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:53.151088+0000) 2022-09-08T01:40:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:53.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:53.089096+0000) 2022-09-08T01:40:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:52 smithi055 ceph-mon[97033]: pgmap v7811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:54.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:40:53.921+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:40:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:54.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:54.151273+0000) 2022-09-08T01:40:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:54.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:54.089312+0000) 2022-09-08T01:40:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:55 smithi059 ceph-mon[97450]: pgmap v7812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:55.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:55.151524+0000) 2022-09-08T01:40:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:55.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:55.089576+0000) 2022-09-08T01:40:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:55 smithi055 ceph-mon[97033]: pgmap v7812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:56.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:56.151704+0000) 2022-09-08T01:40:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:56.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:56.089790+0000) 2022-09-08T01:40:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:57 smithi059 ceph-mon[97450]: pgmap v7813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:40:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:57.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:57.151874+0000) 2022-09-08T01:40:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:57.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:57.090025+0000) 2022-09-08T01:40:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:57 smithi055 ceph-mon[97033]: pgmap v7813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:40:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:58.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:58.152094+0000) 2022-09-08T01:40:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:58.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:58.090247+0000) 2022-09-08T01:40:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:40:59 smithi059 ceph-mon[97450]: pgmap v7814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:40:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:40:59.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:59.152330+0000) 2022-09-08T01:40:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:40:59.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:40:59.090446+0000) 2022-09-08T01:40:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:40:59 smithi055 ceph-mon[97033]: pgmap v7814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:00.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:00.152481+0000) 2022-09-08T01:41:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:00.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:00.090595+0000) 2022-09-08T01:41:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:01 smithi059 ceph-mon[97450]: pgmap v7815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:41:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:41:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:41:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:41:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:41:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:01.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:01.152753+0000) 2022-09-08T01:41:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:01.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:01.090799+0000) 2022-09-08T01:41:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:01 smithi055 ceph-mon[97033]: pgmap v7815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:41:01.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:41:01.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:41:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:41:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:41:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:02.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:02.152999+0000) 2022-09-08T01:41:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:02.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:02.090972+0000) 2022-09-08T01:41:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:03 smithi059 ceph-mon[97450]: pgmap v7816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:03.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:03.153278+0000) 2022-09-08T01:41:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:03.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:03.091243+0000) 2022-09-08T01:41:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:03 smithi055 ceph-mon[97033]: pgmap v7816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:04.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:04.153536+0000) 2022-09-08T01:41:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:04.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:04.091396+0000) 2022-09-08T01:41:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:05 smithi059 ceph-mon[97450]: pgmap v7817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:05.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:05.153749+0000) 2022-09-08T01:41:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:05.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:05.091682+0000) 2022-09-08T01:41:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:05 smithi055 ceph-mon[97033]: pgmap v7817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:41:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:41:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:06.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:06.153896+0000) 2022-09-08T01:41:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:06.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:06.091911+0000) 2022-09-08T01:41:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:41:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:41:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:07 smithi059 ceph-mon[97450]: pgmap v7818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:41:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:07.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:07.092152+0000) 2022-09-08T01:41:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:07.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:07.154095+0000) 2022-09-08T01:41:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:07 smithi055 ceph-mon[97033]: pgmap v7818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:41:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:08.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:08.154263+0000) 2022-09-08T01:41:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:08.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:08.092329+0000) 2022-09-08T01:41:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:09 smithi059 ceph-mon[97450]: pgmap v7819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:41:09.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:41:08.934+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:41:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:09.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:09.092609+0000) 2022-09-08T01:41:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:09.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:09.154488+0000) 2022-09-08T01:41:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:09 smithi055 ceph-mon[97033]: pgmap v7819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:41:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:10.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:10.154692+0000) 2022-09-08T01:41:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:10.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:10.092757+0000) 2022-09-08T01:41:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:10 smithi055 ceph-mon[97033]: osdmap e571: 12 total, 11 up, 11 in 2022-09-08T01:41:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:10 smithi059 ceph-mon[97450]: osdmap e571: 12 total, 11 up, 11 in 2022-09-08T01:41:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:11.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:11.092981+0000) 2022-09-08T01:41:11.522 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:11.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:11.154899+0000) 2022-09-08T01:41:11.523 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:11 smithi055 ceph-mon[97033]: pgmap v7821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:41:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:11 smithi059 ceph-mon[97450]: pgmap v7821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:41:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:12.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:12.093211+0000) 2022-09-08T01:41:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:12.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:12.155065+0000) 2022-09-08T01:41:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:12 smithi055 ceph-mon[97033]: pgmap v7822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:41:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:12 smithi055 ceph-mon[97033]: osdmap e572: 12 total, 11 up, 11 in 2022-09-08T01:41:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:12 smithi059 ceph-mon[97450]: pgmap v7822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:41:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:12 smithi059 ceph-mon[97450]: osdmap e572: 12 total, 11 up, 11 in 2022-09-08T01:41:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:13.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:13.155234+0000) 2022-09-08T01:41:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:13.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:13.093452+0000) 2022-09-08T01:41:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:14.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:14.155411+0000) 2022-09-08T01:41:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:14.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:14.093720+0000) 2022-09-08T01:41:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:14 smithi055 ceph-mon[97033]: pgmap v7824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:41:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:14 smithi059 ceph-mon[97450]: pgmap v7824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:41:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:15.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:15.155654+0000) 2022-09-08T01:41:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:15.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:15.093979+0000) 2022-09-08T01:41:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:16.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:16.155814+0000) 2022-09-08T01:41:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:16.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:16.094182+0000) 2022-09-08T01:41:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:16 smithi055 ceph-mon[97033]: pgmap v7825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:41:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:16 smithi059 ceph-mon[97450]: pgmap v7825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:41:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:17.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:17.155986+0000) 2022-09-08T01:41:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:17.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:17.094355+0000) 2022-09-08T01:41:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:18.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:18.156218+0000) 2022-09-08T01:41:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:18.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:18.094558+0000) 2022-09-08T01:41:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:18 smithi055 ceph-mon[97033]: pgmap v7826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T01:41:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:18 smithi059 ceph-mon[97450]: pgmap v7826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T01:41:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:19.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:19.156491+0000) 2022-09-08T01:41:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:19.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:19.094842+0000) 2022-09-08T01:41:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:20.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:20.095072+0000) 2022-09-08T01:41:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:20.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:20.156759+0000) 2022-09-08T01:41:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:20 smithi055 ceph-mon[97033]: pgmap v7827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:41:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:20 smithi059 ceph-mon[97450]: pgmap v7827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:41:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:21.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:21.156971+0000) 2022-09-08T01:41:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:21.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:21.095246+0000) 2022-09-08T01:41:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:22.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:22.157158+0000) 2022-09-08T01:41:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:22.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:22.095446+0000) 2022-09-08T01:41:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:22 smithi055 ceph-mon[97033]: pgmap v7828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:41:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:22 smithi059 ceph-mon[97450]: pgmap v7828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:41:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:23.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:23.157383+0000) 2022-09-08T01:41:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:23.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:23.095672+0000) 2022-09-08T01:41:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:24.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:24.157599+0000) 2022-09-08T01:41:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:24.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:24.095906+0000) 2022-09-08T01:41:24.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:41:23.950+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:41:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:24 smithi055 ceph-mon[97033]: pgmap v7829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:41:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:24 smithi059 ceph-mon[97450]: pgmap v7829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:41:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:25.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:25.157820+0000) 2022-09-08T01:41:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:25.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:25.096083+0000) 2022-09-08T01:41:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:26.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:26.158029+0000) 2022-09-08T01:41:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:26.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:26.096305+0000) 2022-09-08T01:41:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:26 smithi055 ceph-mon[97033]: pgmap v7830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:41:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:26 smithi059 ceph-mon[97450]: pgmap v7830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:41:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:27.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:27.158241+0000) 2022-09-08T01:41:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:27.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:27.096590+0000) 2022-09-08T01:41:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:28.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:28.158448+0000) 2022-09-08T01:41:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:28.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:28.096839+0000) 2022-09-08T01:41:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:28 smithi059 ceph-mon[97450]: pgmap v7831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:29.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:29.097062+0000) 2022-09-08T01:41:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:29.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:29.158722+0000) 2022-09-08T01:41:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:28 smithi055 ceph-mon[97033]: pgmap v7831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:30.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:30.097256+0000) 2022-09-08T01:41:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:30.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:30.158988+0000) 2022-09-08T01:41:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:30 smithi059 ceph-mon[97450]: pgmap v7832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:41:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:31.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:31.159234+0000) 2022-09-08T01:41:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:31.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:31.097468+0000) 2022-09-08T01:41:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:30 smithi055 ceph-mon[97033]: pgmap v7832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:41:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:32.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:32.159444+0000) 2022-09-08T01:41:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:32.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:32.097726+0000) 2022-09-08T01:41:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:32 smithi059 ceph-mon[97450]: pgmap v7833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:33.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:33.159614+0000) 2022-09-08T01:41:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:33.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:33.097980+0000) 2022-09-08T01:41:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:32 smithi055 ceph-mon[97033]: pgmap v7833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:34.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:34.159719+0000) 2022-09-08T01:41:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:34.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:34.098138+0000) 2022-09-08T01:41:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:34 smithi059 ceph-mon[97450]: pgmap v7834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:35.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:35.098382+0000) 2022-09-08T01:41:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:35.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:35.159912+0000) 2022-09-08T01:41:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:34 smithi055 ceph-mon[97033]: pgmap v7834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:36.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:36.160121+0000) 2022-09-08T01:41:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:36.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:36.098601+0000) 2022-09-08T01:41:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:36 smithi059 ceph-mon[97450]: pgmap v7835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:41:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:37.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:37.160352+0000) 2022-09-08T01:41:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:37.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:37.098797+0000) 2022-09-08T01:41:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:36 smithi055 ceph-mon[97033]: pgmap v7835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:41:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:38.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:38.160603+0000) 2022-09-08T01:41:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:38.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:38.098950+0000) 2022-09-08T01:41:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:38 smithi059 ceph-mon[97450]: pgmap v7836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:39.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:41:38.957+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:41:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:39.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:39.160871+0000) 2022-09-08T01:41:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:39.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:39.099176+0000) 2022-09-08T01:41:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:38 smithi055 ceph-mon[97033]: pgmap v7836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:40.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:40.161106+0000) 2022-09-08T01:41:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:40.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:40.099429+0000) 2022-09-08T01:41:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:41 smithi059 ceph-mon[97450]: pgmap v7837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:41:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:41.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:41.161361+0000) 2022-09-08T01:41:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:41.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:41.099590+0000) 2022-09-08T01:41:41.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:41 smithi055 ceph-mon[97033]: pgmap v7837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:41:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:42.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:42.161579+0000) 2022-09-08T01:41:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:42.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:42.099768+0000) 2022-09-08T01:41:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:43 smithi059 ceph-mon[97450]: pgmap v7838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:41:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:43.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:43.161773+0000) 2022-09-08T01:41:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:43.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:43.100017+0000) 2022-09-08T01:41:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:43 smithi055 ceph-mon[97033]: pgmap v7838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:41:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:44.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:44.162003+0000) 2022-09-08T01:41:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:44.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:44.100269+0000) 2022-09-08T01:41:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:45 smithi059 ceph-mon[97450]: pgmap v7839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:41:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:45.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:45.162236+0000) 2022-09-08T01:41:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:45.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:45.100511+0000) 2022-09-08T01:41:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:45 smithi055 ceph-mon[97033]: pgmap v7839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:41:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:46.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:46.162463+0000) 2022-09-08T01:41:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:46.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:46.100738+0000) 2022-09-08T01:41:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:47 smithi059 ceph-mon[97450]: pgmap v7840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:41:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:47.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:47.162685+0000) 2022-09-08T01:41:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:47.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:47.100890+0000) 2022-09-08T01:41:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:47 smithi055 ceph-mon[97033]: pgmap v7840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:41:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:48.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:48.162881+0000) 2022-09-08T01:41:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:48.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:48.101103+0000) 2022-09-08T01:41:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:49 smithi059 ceph-mon[97450]: pgmap v7841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:41:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:49.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:49.163071+0000) 2022-09-08T01:41:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:49.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:49.101329+0000) 2022-09-08T01:41:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:49 smithi055 ceph-mon[97033]: pgmap v7841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:41:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:50.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:50.163328+0000) 2022-09-08T01:41:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:50.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:50.101514+0000) 2022-09-08T01:41:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:51 smithi059 ceph-mon[97450]: pgmap v7842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:41:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:51.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:51.163576+0000) 2022-09-08T01:41:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:51.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:51.101726+0000) 2022-09-08T01:41:51.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:51 smithi055 ceph-mon[97033]: pgmap v7842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:41:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:52.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:52.163822+0000) 2022-09-08T01:41:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:52.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:52.101886+0000) 2022-09-08T01:41:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:53 smithi059 ceph-mon[97450]: pgmap v7843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:53.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:53.164052+0000) 2022-09-08T01:41:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:53.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:53.102098+0000) 2022-09-08T01:41:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:53 smithi055 ceph-mon[97033]: pgmap v7843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:54.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:41:53.963+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:41:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:54.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:54.164297+0000) 2022-09-08T01:41:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:54.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:54.102255+0000) 2022-09-08T01:41:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:55 smithi059 ceph-mon[97450]: pgmap v7844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:55.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:55.164521+0000) 2022-09-08T01:41:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:55.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:55.102382+0000) 2022-09-08T01:41:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:55 smithi055 ceph-mon[97033]: pgmap v7844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:56.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:56.102625+0000) 2022-09-08T01:41:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:56.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:56.164757+0000) 2022-09-08T01:41:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:57 smithi059 ceph-mon[97450]: pgmap v7845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:41:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:57.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:57.164945+0000) 2022-09-08T01:41:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:57.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:57.102926+0000) 2022-09-08T01:41:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:57 smithi055 ceph-mon[97033]: pgmap v7845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:41:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:58.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:58.165180+0000) 2022-09-08T01:41:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:58.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:58.103185+0000) 2022-09-08T01:41:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:41:59.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:59.165387+0000) 2022-09-08T01:41:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:41:59.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:41:59.103384+0000) 2022-09-08T01:41:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:41:59 smithi055 ceph-mon[97033]: pgmap v7846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:41:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:41:59 smithi059 ceph-mon[97450]: pgmap v7846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:00.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:00.165605+0000) 2022-09-08T01:42:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:00.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:00.103602+0000) 2022-09-08T01:42:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:01.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:01.165783+0000) 2022-09-08T01:42:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:01.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:01.103844+0000) 2022-09-08T01:42:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:01 smithi055 ceph-mon[97033]: pgmap v7847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:42:01.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:42:01.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:42:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:42:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:01 smithi059 ceph-mon[97450]: pgmap v7847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:42:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:42:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:42:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:42:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:02.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:02.165910+0000) 2022-09-08T01:42:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:02.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:02.104045+0000) 2022-09-08T01:42:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:42:02.538 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:42:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:03.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:03.166122+0000) 2022-09-08T01:42:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:03.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:03.104274+0000) 2022-09-08T01:42:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:03 smithi055 ceph-mon[97033]: pgmap v7848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:03 smithi059 ceph-mon[97450]: pgmap v7848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:04.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:04.166368+0000) 2022-09-08T01:42:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:04.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:04.104496+0000) 2022-09-08T01:42:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:05.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:05.166610+0000) 2022-09-08T01:42:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:05.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:05.104696+0000) 2022-09-08T01:42:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:05 smithi055 ceph-mon[97033]: pgmap v7849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:05 smithi059 ceph-mon[97450]: pgmap v7849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:06.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:06.166858+0000) 2022-09-08T01:42:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:06.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:06.104886+0000) 2022-09-08T01:42:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:42:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:42:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:42:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:42:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:07.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:07.166990+0000) 2022-09-08T01:42:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:07.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:07.105014+0000) 2022-09-08T01:42:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:07 smithi055 ceph-mon[97033]: pgmap v7850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:42:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:07 smithi059 ceph-mon[97450]: pgmap v7850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:42:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:08.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:08.167216+0000) 2022-09-08T01:42:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:08.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:08.105228+0000) 2022-09-08T01:42:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:09.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:09.167363+0000) 2022-09-08T01:42:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:09.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:09.105456+0000) 2022-09-08T01:42:09.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:42:08.977+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:42:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:09 smithi055 ceph-mon[97033]: pgmap v7851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:42:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:09 smithi059 ceph-mon[97450]: pgmap v7851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:42:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:10.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:10.167516+0000) 2022-09-08T01:42:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:10.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:10.105678+0000) 2022-09-08T01:42:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:10 smithi055 ceph-mon[97033]: osdmap e573: 12 total, 11 up, 11 in 2022-09-08T01:42:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:10 smithi059 ceph-mon[97450]: osdmap e573: 12 total, 11 up, 11 in 2022-09-08T01:42:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:11.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:11.105862+0000) 2022-09-08T01:42:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:11.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:11.167764+0000) 2022-09-08T01:42:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:11 smithi055 ceph-mon[97033]: pgmap v7853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:42:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:11 smithi059 ceph-mon[97450]: pgmap v7853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:42:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:12.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:12.167943+0000) 2022-09-08T01:42:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:12.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:12.106104+0000) 2022-09-08T01:42:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:13.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:13.168147+0000) 2022-09-08T01:42:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:13.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:13.106339+0000) 2022-09-08T01:42:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:13 smithi055 ceph-mon[97033]: pgmap v7854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:42:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:13 smithi059 ceph-mon[97450]: pgmap v7854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:42:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:14.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:14.168361+0000) 2022-09-08T01:42:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:14.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:14.106571+0000) 2022-09-08T01:42:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:15.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:15.168567+0000) 2022-09-08T01:42:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:15.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:15.106815+0000) 2022-09-08T01:42:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:15 smithi055 ceph-mon[97033]: pgmap v7855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:42:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:15 smithi059 ceph-mon[97450]: pgmap v7855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:42:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:16.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:16.168737+0000) 2022-09-08T01:42:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:16.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:16.107034+0000) 2022-09-08T01:42:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:17.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:17.169009+0000) 2022-09-08T01:42:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:17.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:17.107284+0000) 2022-09-08T01:42:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:17 smithi055 ceph-mon[97033]: pgmap v7856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:42:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:17 smithi055 ceph-mon[97033]: osdmap e574: 12 total, 11 up, 11 in 2022-09-08T01:42:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:17 smithi059 ceph-mon[97450]: pgmap v7856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:42:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:17 smithi059 ceph-mon[97450]: osdmap e574: 12 total, 11 up, 11 in 2022-09-08T01:42:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:18.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:18.169279+0000) 2022-09-08T01:42:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:18.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:18.107504+0000) 2022-09-08T01:42:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:19.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:19.169523+0000) 2022-09-08T01:42:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:19.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:19.107747+0000) 2022-09-08T01:42:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:19 smithi059 ceph-mon[97450]: pgmap v7858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-08T01:42:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:19 smithi055 ceph-mon[97033]: pgmap v7858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-08T01:42:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:20.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:20.169757+0000) 2022-09-08T01:42:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:20.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:20.107931+0000) 2022-09-08T01:42:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:21.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:21.169932+0000) 2022-09-08T01:42:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:21.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:21.108163+0000) 2022-09-08T01:42:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:21 smithi059 ceph-mon[97450]: pgmap v7859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:42:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:21 smithi055 ceph-mon[97033]: pgmap v7859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:42:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:22.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:22.170137+0000) 2022-09-08T01:42:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:22.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:22.108402+0000) 2022-09-08T01:42:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:23.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:23.170406+0000) 2022-09-08T01:42:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:23.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:23.108683+0000) 2022-09-08T01:42:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:23 smithi059 ceph-mon[97450]: pgmap v7860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:42:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:23 smithi055 ceph-mon[97033]: pgmap v7860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:42:24.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:42:23.992+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:42:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:24.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:24.170632+0000) 2022-09-08T01:42:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:24.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:24.108869+0000) 2022-09-08T01:42:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:25.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:25.170891+0000) 2022-09-08T01:42:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:25.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:25.109115+0000) 2022-09-08T01:42:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:25 smithi059 ceph-mon[97450]: pgmap v7861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:42:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:25 smithi055 ceph-mon[97033]: pgmap v7861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:42:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:26.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:26.109347+0000) 2022-09-08T01:42:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:26.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:26.171046+0000) 2022-09-08T01:42:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:27.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:27.109606+0000) 2022-09-08T01:42:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:27.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:27.171280+0000) 2022-09-08T01:42:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:27 smithi059 ceph-mon[97450]: pgmap v7862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:42:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:27 smithi055 ceph-mon[97033]: pgmap v7862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:42:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:28.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:28.171494+0000) 2022-09-08T01:42:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:28.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:28.109827+0000) 2022-09-08T01:42:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:29.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:29.110038+0000) 2022-09-08T01:42:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:29.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:29.171745+0000) 2022-09-08T01:42:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:29 smithi059 ceph-mon[97450]: pgmap v7863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:42:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:29 smithi055 ceph-mon[97033]: pgmap v7863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:42:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:30.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:30.171982+0000) 2022-09-08T01:42:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:30.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:30.110296+0000) 2022-09-08T01:42:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:31.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:31.110477+0000) 2022-09-08T01:42:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:31.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:31.172221+0000) 2022-09-08T01:42:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:31 smithi059 ceph-mon[97450]: pgmap v7864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:42:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:31 smithi055 ceph-mon[97033]: pgmap v7864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:42:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:32.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:32.110721+0000) 2022-09-08T01:42:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:32.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:32.172460+0000) 2022-09-08T01:42:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:33.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:33.172740+0000) 2022-09-08T01:42:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:33.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:33.110910+0000) 2022-09-08T01:42:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:33 smithi059 ceph-mon[97450]: pgmap v7865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:33 smithi055 ceph-mon[97033]: pgmap v7865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:34.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:34.172955+0000) 2022-09-08T01:42:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:34.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:34.111104+0000) 2022-09-08T01:42:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:35.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:35.173178+0000) 2022-09-08T01:42:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:35.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:35.111311+0000) 2022-09-08T01:42:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:35 smithi059 ceph-mon[97450]: pgmap v7866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:35 smithi055 ceph-mon[97033]: pgmap v7866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:36.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:36.173425+0000) 2022-09-08T01:42:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:36.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:36.111529+0000) 2022-09-08T01:42:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:37.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:37.173710+0000) 2022-09-08T01:42:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:37.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:37.111795+0000) 2022-09-08T01:42:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:37 smithi059 ceph-mon[97450]: pgmap v7867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:42:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:37 smithi055 ceph-mon[97033]: pgmap v7867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:42:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:38.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:38.173856+0000) 2022-09-08T01:42:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:38.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:38.112066+0000) 2022-09-08T01:42:39.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:42:39.007+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:42:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:39.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:39.174065+0000) 2022-09-08T01:42:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:39.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:39.112351+0000) 2022-09-08T01:42:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:39 smithi059 ceph-mon[97450]: pgmap v7868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:39 smithi055 ceph-mon[97033]: pgmap v7868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:40.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:40.174286+0000) 2022-09-08T01:42:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:40.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:40.112632+0000) 2022-09-08T01:42:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:41.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:41.174535+0000) 2022-09-08T01:42:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:41.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:41.112912+0000) 2022-09-08T01:42:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:41 smithi059 ceph-mon[97450]: pgmap v7869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:42:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:41 smithi055 ceph-mon[97033]: pgmap v7869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:42:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:42.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:42.174729+0000) 2022-09-08T01:42:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:42.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:42.113122+0000) 2022-09-08T01:42:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:43.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:43.174959+0000) 2022-09-08T01:42:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:43.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:43.113325+0000) 2022-09-08T01:42:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:43 smithi059 ceph-mon[97450]: pgmap v7870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:43 smithi055 ceph-mon[97033]: pgmap v7870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:44.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:44.175189+0000) 2022-09-08T01:42:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:44.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:44.113546+0000) 2022-09-08T01:42:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:45.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:45.113768+0000) 2022-09-08T01:42:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:45.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:45.175385+0000) 2022-09-08T01:42:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:45 smithi059 ceph-mon[97450]: pgmap v7871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:45 smithi055 ceph-mon[97033]: pgmap v7871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:46.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:46.113989+0000) 2022-09-08T01:42:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:46.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:46.175529+0000) 2022-09-08T01:42:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:47.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:47.114203+0000) 2022-09-08T01:42:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:47.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:47.175715+0000) 2022-09-08T01:42:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:47 smithi055 ceph-mon[97033]: pgmap v7872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:42:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:47 smithi059 ceph-mon[97450]: pgmap v7872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:42:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:48.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:48.114448+0000) 2022-09-08T01:42:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:48.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:48.175963+0000) 2022-09-08T01:42:49.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:49.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:49.114764+0000) 2022-09-08T01:42:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:49.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:49.176195+0000) 2022-09-08T01:42:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:49 smithi055 ceph-mon[97033]: pgmap v7873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:42:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:49 smithi059 ceph-mon[97450]: pgmap v7873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:42:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:50.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:50.176427+0000) 2022-09-08T01:42:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:50.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:50.115040+0000) 2022-09-08T01:42:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:51.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:51.176575+0000) 2022-09-08T01:42:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:51.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:51.115222+0000) 2022-09-08T01:42:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:51 smithi055 ceph-mon[97033]: pgmap v7874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:42:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:51 smithi059 ceph-mon[97450]: pgmap v7874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:42:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:52.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:52.115491+0000) 2022-09-08T01:42:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:52.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:52.176790+0000) 2022-09-08T01:42:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:53.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:53.115738+0000) 2022-09-08T01:42:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:53.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:53.177036+0000) 2022-09-08T01:42:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:53 smithi055 ceph-mon[97033]: pgmap v7875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:42:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:53 smithi059 ceph-mon[97450]: pgmap v7875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:42:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:54.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:54.177247+0000) 2022-09-08T01:42:54.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:42:54.012+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:42:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:54.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:54.115959+0000) 2022-09-08T01:42:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:55.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:55.116126+0000) 2022-09-08T01:42:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:55.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:55.177402+0000) 2022-09-08T01:42:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:55 smithi055 ceph-mon[97033]: pgmap v7876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:42:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:55 smithi059 ceph-mon[97450]: pgmap v7876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:42:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:56.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:56.177591+0000) 2022-09-08T01:42:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:56.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:56.116360+0000) 2022-09-08T01:42:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:57.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:57.177777+0000) 2022-09-08T01:42:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:57.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:57.116683+0000) 2022-09-08T01:42:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:57 smithi055 ceph-mon[97033]: pgmap v7877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:42:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:57 smithi059 ceph-mon[97450]: pgmap v7877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:42:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:58.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:58.178014+0000) 2022-09-08T01:42:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:58.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:58.116922+0000) 2022-09-08T01:42:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:42:59.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:59.178292+0000) 2022-09-08T01:42:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:42:59.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:42:59.117135+0000) 2022-09-08T01:42:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:42:59 smithi055 ceph-mon[97033]: pgmap v7878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:42:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:42:59 smithi059 ceph-mon[97450]: pgmap v7878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:00.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:00.117337+0000) 2022-09-08T01:43:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:00.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:00.178543+0000) 2022-09-08T01:43:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:01.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:01.117587+0000) 2022-09-08T01:43:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:01.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:01.178789+0000) 2022-09-08T01:43:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:01 smithi055 ceph-mon[97033]: pgmap v7879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:43:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:43:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:43:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:43:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:01 smithi059 ceph-mon[97450]: pgmap v7879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:43:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:43:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:43:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:43:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:02.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:02.178975+0000) 2022-09-08T01:43:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:02.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:02.117895+0000) 2022-09-08T01:43:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:43:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:02 smithi059 ceph-mon[97450]: pgmap v7880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:43:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:02 smithi055 ceph-mon[97033]: pgmap v7880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:03.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:03.179193+0000) 2022-09-08T01:43:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:03.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:03.118189+0000) 2022-09-08T01:43:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:04.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:04.179373+0000) 2022-09-08T01:43:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:04.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:04.118422+0000) 2022-09-08T01:43:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:05 smithi059 ceph-mon[97450]: pgmap v7881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:05.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:05.179516+0000) 2022-09-08T01:43:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:05.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:05.118701+0000) 2022-09-08T01:43:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:05 smithi055 ceph-mon[97033]: pgmap v7881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:43:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:43:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:06.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:06.179681+0000) 2022-09-08T01:43:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:06.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:06.118889+0000) 2022-09-08T01:43:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:43:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:43:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:07.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:07.179879+0000) 2022-09-08T01:43:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:07.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:07.119158+0000) 2022-09-08T01:43:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:07 smithi055 ceph-mon[97033]: pgmap v7882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:43:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:07 smithi059 ceph-mon[97450]: pgmap v7882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:43:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:08.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:08.180117+0000) 2022-09-08T01:43:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:08.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:08.119426+0000) 2022-09-08T01:43:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:09.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:09.180296+0000) 2022-09-08T01:43:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:09.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:09.119694+0000) 2022-09-08T01:43:09.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:43:09.018+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:43:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:09 smithi055 ceph-mon[97033]: pgmap v7883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:43:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:09 smithi059 ceph-mon[97450]: pgmap v7883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:43:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:10.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:10.180526+0000) 2022-09-08T01:43:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:10.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:10.119821+0000) 2022-09-08T01:43:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:10 smithi055 ceph-mon[97033]: osdmap e575: 12 total, 11 up, 11 in 2022-09-08T01:43:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:10 smithi059 ceph-mon[97450]: osdmap e575: 12 total, 11 up, 11 in 2022-09-08T01:43:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:11.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:11.180795+0000) 2022-09-08T01:43:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:11.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:11.120003+0000) 2022-09-08T01:43:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:11 smithi055 ceph-mon[97033]: pgmap v7885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:43:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:11 smithi059 ceph-mon[97450]: pgmap v7885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:43:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:12.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:12.180959+0000) 2022-09-08T01:43:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:12.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:12.120240+0000) 2022-09-08T01:43:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:13.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:13.181196+0000) 2022-09-08T01:43:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:13.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:13.120505+0000) 2022-09-08T01:43:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:13 smithi055 ceph-mon[97033]: pgmap v7886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:43:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:13 smithi059 ceph-mon[97450]: pgmap v7886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:43:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:14.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:14.181442+0000) 2022-09-08T01:43:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:14.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:14.120722+0000) 2022-09-08T01:43:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:15.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:15.181689+0000) 2022-09-08T01:43:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:15.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:15.120950+0000) 2022-09-08T01:43:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:15 smithi055 ceph-mon[97033]: pgmap v7887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:43:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:15 smithi059 ceph-mon[97450]: pgmap v7887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:43:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:16.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:16.181910+0000) 2022-09-08T01:43:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:16.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:16.121173+0000) 2022-09-08T01:43:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:17.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:17.182108+0000) 2022-09-08T01:43:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:17.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:17.121354+0000) 2022-09-08T01:43:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:17 smithi055 ceph-mon[97033]: pgmap v7888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:43:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:17 smithi055 ceph-mon[97033]: osdmap e576: 12 total, 11 up, 11 in 2022-09-08T01:43:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:17 smithi059 ceph-mon[97450]: pgmap v7888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:43:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:17 smithi059 ceph-mon[97450]: osdmap e576: 12 total, 11 up, 11 in 2022-09-08T01:43:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:18.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:18.182316+0000) 2022-09-08T01:43:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:18.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:18.121560+0000) 2022-09-08T01:43:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:19.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:19.182562+0000) 2022-09-08T01:43:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:19.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:19.121758+0000) 2022-09-08T01:43:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:19 smithi055 ceph-mon[97033]: pgmap v7890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:43:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:19 smithi059 ceph-mon[97450]: pgmap v7890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:43:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:20.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:20.182765+0000) 2022-09-08T01:43:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:20.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:20.121942+0000) 2022-09-08T01:43:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:21.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:21.182991+0000) 2022-09-08T01:43:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:21.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:21.122187+0000) 2022-09-08T01:43:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:21 smithi055 ceph-mon[97033]: pgmap v7891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:43:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:21 smithi059 ceph-mon[97450]: pgmap v7891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:43:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:22.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:22.183201+0000) 2022-09-08T01:43:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:22.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:22.122403+0000) 2022-09-08T01:43:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:23.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:23.183382+0000) 2022-09-08T01:43:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:23.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:23.122725+0000) 2022-09-08T01:43:23.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:23 smithi055 ceph-mon[97033]: pgmap v7892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:43:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:23 smithi059 ceph-mon[97450]: pgmap v7892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:43:24.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:43:24.025+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:43:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:24.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:24.122902+0000) 2022-09-08T01:43:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:24.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:24.183624+0000) 2022-09-08T01:43:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:25.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:25.183832+0000) 2022-09-08T01:43:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:25.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:25.123046+0000) 2022-09-08T01:43:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:25 smithi055 ceph-mon[97033]: pgmap v7893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:43:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:25 smithi059 ceph-mon[97450]: pgmap v7893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:43:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:26.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:26.184069+0000) 2022-09-08T01:43:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:26.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:26.123274+0000) 2022-09-08T01:43:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:27.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:27.184294+0000) 2022-09-08T01:43:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:27.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:27.123528+0000) 2022-09-08T01:43:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:27 smithi055 ceph-mon[97033]: pgmap v7894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:43:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:27 smithi059 ceph-mon[97450]: pgmap v7894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:43:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:28.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:28.123774+0000) 2022-09-08T01:43:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:28.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:28.184449+0000) 2022-09-08T01:43:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:29.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:29.184715+0000) 2022-09-08T01:43:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:29.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:29.124001+0000) 2022-09-08T01:43:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:29 smithi055 ceph-mon[97033]: pgmap v7895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:43:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:29 smithi059 ceph-mon[97450]: pgmap v7895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:43:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:30.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:30.184945+0000) 2022-09-08T01:43:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:30.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:30.124228+0000) 2022-09-08T01:43:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:31.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:31.185158+0000) 2022-09-08T01:43:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:31.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:31.124416+0000) 2022-09-08T01:43:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:31 smithi055 ceph-mon[97033]: pgmap v7896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:43:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:31 smithi059 ceph-mon[97450]: pgmap v7896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:43:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:32.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:32.185390+0000) 2022-09-08T01:43:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:32.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:32.124667+0000) 2022-09-08T01:43:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:33.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:33.124918+0000) 2022-09-08T01:43:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:33.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:33.185553+0000) 2022-09-08T01:43:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:33 smithi055 ceph-mon[97033]: pgmap v7897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:33 smithi059 ceph-mon[97450]: pgmap v7897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:34.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:34.185735+0000) 2022-09-08T01:43:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:34.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:34.125153+0000) 2022-09-08T01:43:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:35.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:35.125371+0000) 2022-09-08T01:43:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:35.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:35.185905+0000) 2022-09-08T01:43:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:35 smithi055 ceph-mon[97033]: pgmap v7898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:35 smithi059 ceph-mon[97450]: pgmap v7898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:36.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:36.186108+0000) 2022-09-08T01:43:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:36.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:36.125542+0000) 2022-09-08T01:43:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:37.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:37.125746+0000) 2022-09-08T01:43:37.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:37.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:37.186332+0000) 2022-09-08T01:43:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:37 smithi055 ceph-mon[97033]: pgmap v7899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:43:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:37 smithi059 ceph-mon[97450]: pgmap v7899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:43:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:38.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:38.186526+0000) 2022-09-08T01:43:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:38.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:38.125882+0000) 2022-09-08T01:43:39.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:43:39.036+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:43:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:39.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:39.186686+0000) 2022-09-08T01:43:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:39.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:39.126101+0000) 2022-09-08T01:43:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:39 smithi055 ceph-mon[97033]: pgmap v7900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:39 smithi059 ceph-mon[97450]: pgmap v7900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:40.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:40.186808+0000) 2022-09-08T01:43:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:40.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:40.126318+0000) 2022-09-08T01:43:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:41.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:41.126532+0000) 2022-09-08T01:43:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:41.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:41.186974+0000) 2022-09-08T01:43:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:41 smithi055 ceph-mon[97033]: pgmap v7901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:43:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:41 smithi059 ceph-mon[97450]: pgmap v7901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:43:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:42.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:42.187119+0000) 2022-09-08T01:43:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:42.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:42.126756+0000) 2022-09-08T01:43:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:43.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:43.187254+0000) 2022-09-08T01:43:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:43.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:43.127004+0000) 2022-09-08T01:43:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:43 smithi055 ceph-mon[97033]: pgmap v7902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:43 smithi059 ceph-mon[97450]: pgmap v7902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:44.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:44.187391+0000) 2022-09-08T01:43:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:44.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:44.127174+0000) 2022-09-08T01:43:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:45.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:45.187535+0000) 2022-09-08T01:43:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:45.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:45.127386+0000) 2022-09-08T01:43:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:45 smithi055 ceph-mon[97033]: pgmap v7903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:45 smithi059 ceph-mon[97450]: pgmap v7903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:46.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:46.187758+0000) 2022-09-08T01:43:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:46.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:46.127585+0000) 2022-09-08T01:43:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:47.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:47.187968+0000) 2022-09-08T01:43:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:47.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:47.127833+0000) 2022-09-08T01:43:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:47 smithi055 ceph-mon[97033]: pgmap v7904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:43:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:47 smithi059 ceph-mon[97450]: pgmap v7904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:43:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:48.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:48.188142+0000) 2022-09-08T01:43:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:48.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:48.128082+0000) 2022-09-08T01:43:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:49.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:49.188313+0000) 2022-09-08T01:43:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:49.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:49.128284+0000) 2022-09-08T01:43:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:49 smithi055 ceph-mon[97033]: pgmap v7905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:49 smithi059 ceph-mon[97450]: pgmap v7905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:50.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:50.128485+0000) 2022-09-08T01:43:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:50.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:50.188469+0000) 2022-09-08T01:43:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:51.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:51.188618+0000) 2022-09-08T01:43:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:51.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:51.128736+0000) 2022-09-08T01:43:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:51 smithi055 ceph-mon[97033]: pgmap v7906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:43:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:51 smithi059 ceph-mon[97450]: pgmap v7906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:43:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:52.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:52.188768+0000) 2022-09-08T01:43:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:52.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:52.128959+0000) 2022-09-08T01:43:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:53.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:53.129167+0000) 2022-09-08T01:43:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:53.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:53.188884+0000) 2022-09-08T01:43:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:53 smithi055 ceph-mon[97033]: pgmap v7907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:53 smithi059 ceph-mon[97450]: pgmap v7907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:54.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:54.189086+0000) 2022-09-08T01:43:54.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:43:54.051+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:43:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:54.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:54.129397+0000) 2022-09-08T01:43:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:55.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:55.189302+0000) 2022-09-08T01:43:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:55.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:55.129573+0000) 2022-09-08T01:43:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:55 smithi055 ceph-mon[97033]: pgmap v7908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:55 smithi059 ceph-mon[97450]: pgmap v7908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:56.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:56.189520+0000) 2022-09-08T01:43:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:56.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:56.129780+0000) 2022-09-08T01:43:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:57.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:57.189730+0000) 2022-09-08T01:43:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:57.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:57.130012+0000) 2022-09-08T01:43:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:57 smithi055 ceph-mon[97033]: pgmap v7909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:43:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:57 smithi059 ceph-mon[97450]: pgmap v7909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:43:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:58.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:58.189871+0000) 2022-09-08T01:43:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:58.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:58.130298+0000) 2022-09-08T01:43:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:43:59.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:59.130549+0000) 2022-09-08T01:43:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:43:59.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:43:59.190023+0000) 2022-09-08T01:43:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:43:59 smithi055 ceph-mon[97033]: pgmap v7910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:43:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:43:59 smithi059 ceph-mon[97450]: pgmap v7910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:00.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:00.190232+0000) 2022-09-08T01:44:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:00.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:00.130778+0000) 2022-09-08T01:44:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:01.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:01.130911+0000) 2022-09-08T01:44:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:01.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:01.190388+0000) 2022-09-08T01:44:01.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:01 smithi055 ceph-mon[97033]: pgmap v7911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:44:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:01 smithi059 ceph-mon[97450]: pgmap v7911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:44:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:44:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:44:02.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:44:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:44:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:44:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:44:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:02.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:02.190604+0000) 2022-09-08T01:44:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:02.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:02.131032+0000) 2022-09-08T01:44:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:03.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:03.190775+0000) 2022-09-08T01:44:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:03.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:03.131171+0000) 2022-09-08T01:44:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:03 smithi055 ceph-mon[97033]: pgmap v7912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:03.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:44:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:03 smithi059 ceph-mon[97450]: pgmap v7912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:44:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:04.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:04.190946+0000) 2022-09-08T01:44:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:04.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:04.131392+0000) 2022-09-08T01:44:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:05.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:05.191153+0000) 2022-09-08T01:44:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:05.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:05.131553+0000) 2022-09-08T01:44:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:05 smithi055 ceph-mon[97033]: pgmap v7913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:05 smithi059 ceph-mon[97450]: pgmap v7913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:06.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:06.191377+0000) 2022-09-08T01:44:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:06.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:06.131802+0000) 2022-09-08T01:44:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:44:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:44:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:44:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:44:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:07.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:07.191490+0000) 2022-09-08T01:44:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:07.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:07.132026+0000) 2022-09-08T01:44:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:07 smithi055 ceph-mon[97033]: pgmap v7914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:44:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:07 smithi059 ceph-mon[97450]: pgmap v7914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:44:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:08.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:08.191665+0000) 2022-09-08T01:44:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:08.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:08.132250+0000) 2022-09-08T01:44:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:09.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:09.191798+0000) 2022-09-08T01:44:09.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:44:09.064+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:44:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:09.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:09.132456+0000) 2022-09-08T01:44:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:09 smithi055 ceph-mon[97033]: pgmap v7915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:44:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:09 smithi059 ceph-mon[97450]: pgmap v7915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:44:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:10.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:10.191962+0000) 2022-09-08T01:44:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:10.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:10.132701+0000) 2022-09-08T01:44:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:10 smithi055 ceph-mon[97033]: osdmap e577: 12 total, 11 up, 11 in 2022-09-08T01:44:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:10 smithi059 ceph-mon[97450]: osdmap e577: 12 total, 11 up, 11 in 2022-09-08T01:44:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:11.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:11.192151+0000) 2022-09-08T01:44:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:11.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:11.132923+0000) 2022-09-08T01:44:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:11 smithi055 ceph-mon[97033]: pgmap v7917: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:44:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:11 smithi059 ceph-mon[97450]: pgmap v7917: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:44:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:12.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:12.192361+0000) 2022-09-08T01:44:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:12.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:12.133138+0000) 2022-09-08T01:44:13.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:13.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:13.133370+0000) 2022-09-08T01:44:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:13.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:13.192604+0000) 2022-09-08T01:44:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:13 smithi055 ceph-mon[97033]: pgmap v7918: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:44:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:13 smithi059 ceph-mon[97450]: pgmap v7918: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:44:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:14.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:14.192829+0000) 2022-09-08T01:44:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:14.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:14.133597+0000) 2022-09-08T01:44:15.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:15.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:15.133746+0000) 2022-09-08T01:44:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:15.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:15.193067+0000) 2022-09-08T01:44:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:15 smithi055 ceph-mon[97033]: pgmap v7919: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:44:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:15 smithi059 ceph-mon[97450]: pgmap v7919: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:44:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:16.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:16.133949+0000) 2022-09-08T01:44:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:16.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:16.193288+0000) 2022-09-08T01:44:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:17.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:17.193566+0000) 2022-09-08T01:44:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:17.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:17.134149+0000) 2022-09-08T01:44:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:17 smithi055 ceph-mon[97033]: pgmap v7920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:44:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:17 smithi055 ceph-mon[97033]: osdmap e578: 12 total, 11 up, 11 in 2022-09-08T01:44:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:17 smithi059 ceph-mon[97450]: pgmap v7920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:44:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:17 smithi059 ceph-mon[97450]: osdmap e578: 12 total, 11 up, 11 in 2022-09-08T01:44:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:18.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:18.134381+0000) 2022-09-08T01:44:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:18.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:18.193854+0000) 2022-09-08T01:44:19.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:19.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:19.134620+0000) 2022-09-08T01:44:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:19.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:19.194067+0000) 2022-09-08T01:44:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:19 smithi055 ceph-mon[97033]: pgmap v7922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-08T01:44:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:19 smithi059 ceph-mon[97450]: pgmap v7922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-08T01:44:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:20.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:20.194282+0000) 2022-09-08T01:44:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:20.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:20.134832+0000) 2022-09-08T01:44:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:21.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:21.194551+0000) 2022-09-08T01:44:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:21.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:21.134967+0000) 2022-09-08T01:44:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:21 smithi055 ceph-mon[97033]: pgmap v7923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:44:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:21 smithi059 ceph-mon[97450]: pgmap v7923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:44:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:22.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:22.194756+0000) 2022-09-08T01:44:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:22.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:22.135187+0000) 2022-09-08T01:44:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:23.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:23.195020+0000) 2022-09-08T01:44:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:23.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:23.135367+0000) 2022-09-08T01:44:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:23 smithi055 ceph-mon[97033]: pgmap v7924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:44:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:23 smithi059 ceph-mon[97450]: pgmap v7924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:44:24.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:44:24.069+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:44:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:24.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:24.195293+0000) 2022-09-08T01:44:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:24.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:24.135610+0000) 2022-09-08T01:44:25.196 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:25.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:25.135782+0000) 2022-09-08T01:44:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:25 smithi055 ceph-mon[97033]: pgmap v7925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:44:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:25 smithi059 ceph-mon[97450]: pgmap v7925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:44:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:25.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:25.195529+0000) 2022-09-08T01:44:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:26.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:26.195760+0000) 2022-09-08T01:44:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:26.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:26.136014+0000) 2022-09-08T01:44:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:27.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:27.195966+0000) 2022-09-08T01:44:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:27.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:27.136188+0000) 2022-09-08T01:44:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:27 smithi055 ceph-mon[97033]: pgmap v7926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:44:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:27 smithi059 ceph-mon[97450]: pgmap v7926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:44:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:28.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:28.196201+0000) 2022-09-08T01:44:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:28.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:28.136424+0000) 2022-09-08T01:44:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:29.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:29.196442+0000) 2022-09-08T01:44:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:29.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:29.136674+0000) 2022-09-08T01:44:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:29 smithi055 ceph-mon[97033]: pgmap v7927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:44:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:29 smithi059 ceph-mon[97450]: pgmap v7927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:44:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:30.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:30.196731+0000) 2022-09-08T01:44:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:30.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:30.136896+0000) 2022-09-08T01:44:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:31.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:31.196933+0000) 2022-09-08T01:44:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:31.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:31.137113+0000) 2022-09-08T01:44:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:31 smithi055 ceph-mon[97033]: pgmap v7928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:44:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:31 smithi059 ceph-mon[97450]: pgmap v7928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:44:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:32.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:32.137381+0000) 2022-09-08T01:44:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:32.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:32.197188+0000) 2022-09-08T01:44:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:33.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:33.197426+0000) 2022-09-08T01:44:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:33.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:33.137606+0000) 2022-09-08T01:44:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:33 smithi055 ceph-mon[97033]: pgmap v7929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:33 smithi059 ceph-mon[97450]: pgmap v7929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:34.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:34.197632+0000) 2022-09-08T01:44:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:34.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:34.137802+0000) 2022-09-08T01:44:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:35.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:35.197886+0000) 2022-09-08T01:44:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:35.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:35.138002+0000) 2022-09-08T01:44:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:35 smithi055 ceph-mon[97033]: pgmap v7930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:35 smithi059 ceph-mon[97450]: pgmap v7930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:36.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:36.198097+0000) 2022-09-08T01:44:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:36.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:36.138219+0000) 2022-09-08T01:44:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:37.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:37.198270+0000) 2022-09-08T01:44:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:37.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:37.138458+0000) 2022-09-08T01:44:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:37 smithi055 ceph-mon[97033]: pgmap v7931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:44:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:37 smithi059 ceph-mon[97450]: pgmap v7931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:44:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:38.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:38.198539+0000) 2022-09-08T01:44:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:38.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:38.138740+0000) 2022-09-08T01:44:39.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:44:39.075+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:44:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:39.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:39.198818+0000) 2022-09-08T01:44:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:39.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:39.138916+0000) 2022-09-08T01:44:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:39 smithi055 ceph-mon[97033]: pgmap v7932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:39 smithi059 ceph-mon[97450]: pgmap v7932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:40.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:40.198986+0000) 2022-09-08T01:44:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:40.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:40.139128+0000) 2022-09-08T01:44:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:41.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:41.199200+0000) 2022-09-08T01:44:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:41.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:41.139310+0000) 2022-09-08T01:44:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:41 smithi055 ceph-mon[97033]: pgmap v7933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:44:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:41 smithi059 ceph-mon[97450]: pgmap v7933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:44:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:42.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:42.199356+0000) 2022-09-08T01:44:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:42.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:42.139497+0000) 2022-09-08T01:44:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:43.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:43.199508+0000) 2022-09-08T01:44:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:43.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:43.139755+0000) 2022-09-08T01:44:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:43 smithi059 ceph-mon[97450]: pgmap v7934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:43 smithi055 ceph-mon[97033]: pgmap v7934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:44.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:44.199769+0000) 2022-09-08T01:44:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:44.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:44.139923+0000) 2022-09-08T01:44:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:45.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:45.200013+0000) 2022-09-08T01:44:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:45.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:45.140159+0000) 2022-09-08T01:44:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:45 smithi059 ceph-mon[97450]: pgmap v7935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:45 smithi055 ceph-mon[97033]: pgmap v7935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:46.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:46.200207+0000) 2022-09-08T01:44:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:46.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:46.140284+0000) 2022-09-08T01:44:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:47.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:47.200429+0000) 2022-09-08T01:44:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:47.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:47.140506+0000) 2022-09-08T01:44:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:47 smithi059 ceph-mon[97450]: pgmap v7936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:44:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:47 smithi055 ceph-mon[97033]: pgmap v7936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:44:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:48.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:48.200706+0000) 2022-09-08T01:44:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:48.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:48.140718+0000) 2022-09-08T01:44:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:49.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:49.200884+0000) 2022-09-08T01:44:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:49.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:49.140865+0000) 2022-09-08T01:44:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:49 smithi059 ceph-mon[97450]: pgmap v7937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:49 smithi055 ceph-mon[97033]: pgmap v7937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:50.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:50.201125+0000) 2022-09-08T01:44:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:50.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:50.141005+0000) 2022-09-08T01:44:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:51.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:51.141236+0000) 2022-09-08T01:44:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:51.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:51.201320+0000) 2022-09-08T01:44:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:51 smithi059 ceph-mon[97450]: pgmap v7938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:44:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:51 smithi055 ceph-mon[97033]: pgmap v7938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:44:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:52.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:52.141460+0000) 2022-09-08T01:44:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:52.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:52.201582+0000) 2022-09-08T01:44:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:53.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:53.141710+0000) 2022-09-08T01:44:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:53.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:53.201802+0000) 2022-09-08T01:44:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:53 smithi059 ceph-mon[97450]: pgmap v7939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:53 smithi055 ceph-mon[97033]: pgmap v7939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:54.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:44:54.075+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:44:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:54.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:54.201937+0000) 2022-09-08T01:44:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:54.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:54.141870+0000) 2022-09-08T01:44:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:55.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:55.202139+0000) 2022-09-08T01:44:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:55.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:55.142135+0000) 2022-09-08T01:44:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:55 smithi059 ceph-mon[97450]: pgmap v7940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:55 smithi055 ceph-mon[97033]: pgmap v7940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:56.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:56.202390+0000) 2022-09-08T01:44:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:56.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:56.142348+0000) 2022-09-08T01:44:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:57.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:57.202609+0000) 2022-09-08T01:44:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:57.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:57.142532+0000) 2022-09-08T01:44:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:57 smithi059 ceph-mon[97450]: pgmap v7941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:44:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:57 smithi055 ceph-mon[97033]: pgmap v7941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:44:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:58.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:58.202888+0000) 2022-09-08T01:44:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:58.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:58.142753+0000) 2022-09-08T01:44:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:44:59.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:59.203108+0000) 2022-09-08T01:44:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:44:59.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:44:59.143023+0000) 2022-09-08T01:44:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:44:59 smithi059 ceph-mon[97450]: pgmap v7942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:44:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:44:59 smithi055 ceph-mon[97033]: pgmap v7942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:00.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:00.203292+0000) 2022-09-08T01:45:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:00.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:00.143193+0000) 2022-09-08T01:45:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:01.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:01.203513+0000) 2022-09-08T01:45:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:01.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:01.143407+0000) 2022-09-08T01:45:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:01 smithi059 ceph-mon[97450]: pgmap v7943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:45:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:01 smithi055 ceph-mon[97033]: pgmap v7943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:45:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:02.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:02.203792+0000) 2022-09-08T01:45:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:02.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:02.143570+0000) 2022-09-08T01:45:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:03.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:03.143855+0000) 2022-09-08T01:45:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:03.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:03.203956+0000) 2022-09-08T01:45:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:03 smithi055 ceph-mon[97033]: pgmap v7944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:03.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:45:03.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:45:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:45:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:03 smithi059 ceph-mon[97450]: pgmap v7944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:45:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:45:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:45:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:04.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:04.204103+0000) 2022-09-08T01:45:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:04.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:04.144037+0000) 2022-09-08T01:45:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:05.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:05.204324+0000) 2022-09-08T01:45:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:05.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:05.144275+0000) 2022-09-08T01:45:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:05 smithi059 ceph-mon[97450]: pgmap v7945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:45:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:45:05.666 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:05 smithi055 ceph-mon[97033]: pgmap v7945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:05.667 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:45:05.667 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:45:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:06.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:06.204548+0000) 2022-09-08T01:45:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:06.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:06.144510+0000) 2022-09-08T01:45:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:07.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:07.204782+0000) 2022-09-08T01:45:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:07.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:07.144780+0000) 2022-09-08T01:45:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:07 smithi055 ceph-mon[97033]: pgmap v7946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:45:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:07 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T01:45:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:45:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:45:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:07 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T01:45:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:45:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:45:07.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:45:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:07 smithi059 ceph-mon[97450]: pgmap v7946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:45:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:07 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T01:45:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:45:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:45:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:07 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T01:45:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:45:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:45:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:45:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:08.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:08.205029+0000) 2022-09-08T01:45:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:08.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:08.144991+0000) 2022-09-08T01:45:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:09.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:09.205253+0000) 2022-09-08T01:45:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:09.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:09.145218+0000) 2022-09-08T01:45:09.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:45:09.092+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:45:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:09 smithi055 ceph-mon[97033]: pgmap v7947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:45:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:09 smithi059 ceph-mon[97450]: pgmap v7947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:45:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:10.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:10.205481+0000) 2022-09-08T01:45:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:10.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:10.145454+0000) 2022-09-08T01:45:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:10 smithi055 ceph-mon[97033]: osdmap e579: 12 total, 11 up, 11 in 2022-09-08T01:45:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:10 smithi059 ceph-mon[97450]: osdmap e579: 12 total, 11 up, 11 in 2022-09-08T01:45:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:11.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:11.205669+0000) 2022-09-08T01:45:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:11.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:11.145690+0000) 2022-09-08T01:45:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:11 smithi055 ceph-mon[97033]: pgmap v7949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:45:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:11 smithi059 ceph-mon[97450]: pgmap v7949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:45:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:12.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:12.145838+0000) 2022-09-08T01:45:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:12.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:12.205844+0000) 2022-09-08T01:45:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:13 smithi059 ceph-mon[97450]: pgmap v7950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:45:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:13 smithi059 ceph-mon[97450]: osdmap e580: 12 total, 11 up, 11 in 2022-09-08T01:45:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:13.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:13.206125+0000) 2022-09-08T01:45:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:13.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:13.146054+0000) 2022-09-08T01:45:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:13 smithi055 ceph-mon[97033]: pgmap v7950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:45:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:13 smithi055 ceph-mon[97033]: osdmap e580: 12 total, 11 up, 11 in 2022-09-08T01:45:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:14.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:14.206395+0000) 2022-09-08T01:45:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:14.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:14.146229+0000) 2022-09-08T01:45:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:15 smithi059 ceph-mon[97450]: pgmap v7952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:45:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:15.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:15.206671+0000) 2022-09-08T01:45:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:15.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:15.146394+0000) 2022-09-08T01:45:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:15 smithi055 ceph-mon[97033]: pgmap v7952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:45:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:16.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:16.206816+0000) 2022-09-08T01:45:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:16.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:16.146570+0000) 2022-09-08T01:45:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:17.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:17.207041+0000) 2022-09-08T01:45:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:17.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:17.146885+0000) 2022-09-08T01:45:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:17 smithi055 ceph-mon[97033]: pgmap v7953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:45:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:17 smithi059 ceph-mon[97450]: pgmap v7953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:45:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:18.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:18.207296+0000) 2022-09-08T01:45:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:18.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:18.147132+0000) 2022-09-08T01:45:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:19.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:19.207563+0000) 2022-09-08T01:45:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:19.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:19.147377+0000) 2022-09-08T01:45:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:19 smithi055 ceph-mon[97033]: pgmap v7954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-08T01:45:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:19 smithi059 ceph-mon[97450]: pgmap v7954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.5 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-08T01:45:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:20.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:20.207834+0000) 2022-09-08T01:45:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:20.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:20.147586+0000) 2022-09-08T01:45:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:21.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:21.208100+0000) 2022-09-08T01:45:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:21.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:21.147754+0000) 2022-09-08T01:45:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:21 smithi055 ceph-mon[97033]: pgmap v7955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:45:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:21 smithi059 ceph-mon[97450]: pgmap v7955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:45:22.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:22.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:22.147894+0000) 2022-09-08T01:45:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:22.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:22.208339+0000) 2022-09-08T01:45:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:23.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:23.208605+0000) 2022-09-08T01:45:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:23.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:23.148114+0000) 2022-09-08T01:45:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:23 smithi055 ceph-mon[97033]: pgmap v7956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:45:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:23 smithi059 ceph-mon[97450]: pgmap v7956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:45:24.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:45:24.107+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:45:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:24.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:24.208839+0000) 2022-09-08T01:45:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:24.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:24.148344+0000) 2022-09-08T01:45:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:25.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:25.148580+0000) 2022-09-08T01:45:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:25.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:25.209080+0000) 2022-09-08T01:45:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:25 smithi055 ceph-mon[97033]: pgmap v7957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:45:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:25 smithi059 ceph-mon[97450]: pgmap v7957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:45:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:26.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:26.209353+0000) 2022-09-08T01:45:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:26.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:26.148805+0000) 2022-09-08T01:45:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:27.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:27.209584+0000) 2022-09-08T01:45:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:27.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:27.148999+0000) 2022-09-08T01:45:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:27 smithi055 ceph-mon[97033]: pgmap v7958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:45:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:27 smithi059 ceph-mon[97450]: pgmap v7958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:45:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:28.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:28.209832+0000) 2022-09-08T01:45:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:28.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:28.149169+0000) 2022-09-08T01:45:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:29.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:29.210013+0000) 2022-09-08T01:45:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:29.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:29.149404+0000) 2022-09-08T01:45:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:29 smithi055 ceph-mon[97033]: pgmap v7959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:29 smithi059 ceph-mon[97450]: pgmap v7959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:30.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:30.210236+0000) 2022-09-08T01:45:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:30.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:30.149628+0000) 2022-09-08T01:45:31.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:31.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:31.149808+0000) 2022-09-08T01:45:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:31.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:31.210449+0000) 2022-09-08T01:45:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:31 smithi055 ceph-mon[97033]: pgmap v7960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:45:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:31 smithi059 ceph-mon[97450]: pgmap v7960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:45:32.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:32.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:32.150006+0000) 2022-09-08T01:45:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:32.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:32.210718+0000) 2022-09-08T01:45:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:33.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:33.210964+0000) 2022-09-08T01:45:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:33.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:33.150298+0000) 2022-09-08T01:45:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:33 smithi055 ceph-mon[97033]: pgmap v7961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:33 smithi059 ceph-mon[97450]: pgmap v7961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:34.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:34.211222+0000) 2022-09-08T01:45:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:34.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:34.150422+0000) 2022-09-08T01:45:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:35.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:35.211412+0000) 2022-09-08T01:45:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:35.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:35.150616+0000) 2022-09-08T01:45:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:35 smithi055 ceph-mon[97033]: pgmap v7962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:35 smithi059 ceph-mon[97450]: pgmap v7962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:36.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:36.211653+0000) 2022-09-08T01:45:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:36.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:36.150825+0000) 2022-09-08T01:45:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:37.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:37.211938+0000) 2022-09-08T01:45:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:37.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:37.151003+0000) 2022-09-08T01:45:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:37 smithi055 ceph-mon[97033]: pgmap v7963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:45:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:37 smithi059 ceph-mon[97450]: pgmap v7963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:45:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:38.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:38.212188+0000) 2022-09-08T01:45:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:38.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:38.151136+0000) 2022-09-08T01:45:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:39.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:39.212428+0000) 2022-09-08T01:45:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:39.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:39.151349+0000) 2022-09-08T01:45:39.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:45:39.109+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:45:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:39 smithi055 ceph-mon[97033]: pgmap v7964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:39 smithi059 ceph-mon[97450]: pgmap v7964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:40.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:40.212605+0000) 2022-09-08T01:45:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:40.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:40.151575+0000) 2022-09-08T01:45:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:41 smithi055 ceph-mon[97033]: pgmap v7965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:45:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:41 smithi059 ceph-mon[97450]: pgmap v7965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:45:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:41.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:41.212731+0000) 2022-09-08T01:45:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:41.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:41.151793+0000) 2022-09-08T01:45:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:42.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:42.212973+0000) 2022-09-08T01:45:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:42.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:42.151934+0000) 2022-09-08T01:45:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:43 smithi055 ceph-mon[97033]: pgmap v7966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:43 smithi059 ceph-mon[97450]: pgmap v7966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:43.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:43.213213+0000) 2022-09-08T01:45:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:43.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:43.152060+0000) 2022-09-08T01:45:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:44.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:44.213455+0000) 2022-09-08T01:45:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:44.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:44.152288+0000) 2022-09-08T01:45:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:45 smithi055 ceph-mon[97033]: pgmap v7967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:45 smithi059 ceph-mon[97450]: pgmap v7967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:45.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:45.213767+0000) 2022-09-08T01:45:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:45.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:45.152507+0000) 2022-09-08T01:45:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:46.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:46.214012+0000) 2022-09-08T01:45:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:46.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:46.152738+0000) 2022-09-08T01:45:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:47 smithi055 ceph-mon[97033]: pgmap v7968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:45:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:47 smithi059 ceph-mon[97450]: pgmap v7968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:45:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:47.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:47.214233+0000) 2022-09-08T01:45:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:47.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:47.152993+0000) 2022-09-08T01:45:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:48.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:48.214459+0000) 2022-09-08T01:45:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:48.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:48.153230+0000) 2022-09-08T01:45:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:49 smithi055 ceph-mon[97033]: pgmap v7969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:49 smithi059 ceph-mon[97450]: pgmap v7969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:49.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:49.214651+0000) 2022-09-08T01:45:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:49.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:49.153430+0000) 2022-09-08T01:45:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:50.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:50.214869+0000) 2022-09-08T01:45:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:50.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:50.153697+0000) 2022-09-08T01:45:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:51 smithi055 ceph-mon[97033]: pgmap v7970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:45:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:51 smithi059 ceph-mon[97450]: pgmap v7970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:45:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:51.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:51.215086+0000) 2022-09-08T01:45:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:51.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:51.153831+0000) 2022-09-08T01:45:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:52.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:52.215295+0000) 2022-09-08T01:45:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:52.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:52.154008+0000) 2022-09-08T01:45:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:53 smithi055 ceph-mon[97033]: pgmap v7971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:53 smithi059 ceph-mon[97450]: pgmap v7971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:53.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:53.215431+0000) 2022-09-08T01:45:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:53.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:53.154275+0000) 2022-09-08T01:45:54.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:45:54.111+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:45:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:54.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:54.215698+0000) 2022-09-08T01:45:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:54.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:54.154508+0000) 2022-09-08T01:45:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:55 smithi055 ceph-mon[97033]: pgmap v7972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:55 smithi059 ceph-mon[97450]: pgmap v7972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:55.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:55.215910+0000) 2022-09-08T01:45:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:55.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:55.154692+0000) 2022-09-08T01:45:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:56.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:56.216147+0000) 2022-09-08T01:45:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:56.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:56.154893+0000) 2022-09-08T01:45:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:57 smithi055 ceph-mon[97033]: pgmap v7973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:45:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:57 smithi059 ceph-mon[97450]: pgmap v7973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:45:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:57.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:57.216338+0000) 2022-09-08T01:45:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:57.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:57.155101+0000) 2022-09-08T01:45:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:58.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:58.216557+0000) 2022-09-08T01:45:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:58.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:58.155321+0000) 2022-09-08T01:45:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:45:59 smithi055 ceph-mon[97033]: pgmap v7974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:45:59 smithi059 ceph-mon[97450]: pgmap v7974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:45:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:45:59.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:59.216809+0000) 2022-09-08T01:45:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:45:59.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:45:59.155600+0000) 2022-09-08T01:46:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:00.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:00.216948+0000) 2022-09-08T01:46:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:00.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:00.155801+0000) 2022-09-08T01:46:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:01 smithi055 ceph-mon[97033]: pgmap v7975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:46:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:01 smithi059 ceph-mon[97450]: pgmap v7975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:46:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:01.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:01.217117+0000) 2022-09-08T01:46:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:01.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:01.155973+0000) 2022-09-08T01:46:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:02.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:02.156212+0000) 2022-09-08T01:46:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:02.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:02.217306+0000) 2022-09-08T01:46:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:03 smithi055 ceph-mon[97033]: pgmap v7976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:03 smithi059 ceph-mon[97450]: pgmap v7976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:03.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:03.217506+0000) 2022-09-08T01:46:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:03.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:03.156468+0000) 2022-09-08T01:46:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:04.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:04.156739+0000) 2022-09-08T01:46:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:04.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:04.217756+0000) 2022-09-08T01:46:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:05 smithi055 ceph-mon[97033]: pgmap v7977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:05 smithi059 ceph-mon[97450]: pgmap v7977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:05.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:05.217980+0000) 2022-09-08T01:46:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:05.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:05.156957+0000) 2022-09-08T01:46:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:46:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:46:06.520 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:46:06.520 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:46:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:06.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:06.218207+0000) 2022-09-08T01:46:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:06.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:06.157116+0000) 2022-09-08T01:46:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:07 smithi055 ceph-mon[97033]: pgmap v7978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:46:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:46:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:46:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:46:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:07 smithi059 ceph-mon[97450]: pgmap v7978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:46:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:46:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:46:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:46:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:07.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:07.218445+0000) 2022-09-08T01:46:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:07.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:07.157399+0000) 2022-09-08T01:46:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:08.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:08.218724+0000) 2022-09-08T01:46:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:08.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:08.157678+0000) 2022-09-08T01:46:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:09.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:09.218900+0000) 2022-09-08T01:46:09.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:46:09.116+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:46:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:09.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:09.157877+0000) 2022-09-08T01:46:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:09 smithi055 ceph-mon[97033]: pgmap v7979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:46:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:09 smithi059 ceph-mon[97450]: pgmap v7979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:46:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:10 smithi055 ceph-mon[97033]: osdmap e581: 12 total, 11 up, 11 in 2022-09-08T01:46:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:10 smithi059 ceph-mon[97450]: osdmap e581: 12 total, 11 up, 11 in 2022-09-08T01:46:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:10.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:10.219088+0000) 2022-09-08T01:46:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:10.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:10.158031+0000) 2022-09-08T01:46:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:11 smithi055 ceph-mon[97033]: pgmap v7981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:46:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:11.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:11.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:11 smithi059 ceph-mon[97450]: pgmap v7981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:46:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:11.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:11.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:46:11.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:46:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:11.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:11.219263+0000) 2022-09-08T01:46:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:11.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:11.158237+0000) 2022-09-08T01:46:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:12 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T01:46:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:12 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T01:46:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:12 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T01:46:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:12 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T01:46:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:12 smithi055 ceph-mon[97033]: osdmap e582: 12 total, 11 up, 11 in 2022-09-08T01:46:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:12 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T01:46:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:12 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T01:46:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:12 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T01:46:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:12 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T01:46:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:12 smithi059 ceph-mon[97450]: osdmap e582: 12 total, 11 up, 11 in 2022-09-08T01:46:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:12.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:12.219417+0000) 2022-09-08T01:46:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:12.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:12.158453+0000) 2022-09-08T01:46:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:13 smithi055 ceph-mon[97033]: pgmap v7982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:46:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:13 smithi059 ceph-mon[97450]: pgmap v7982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:46:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:13.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:13.219616+0000) 2022-09-08T01:46:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:13.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:13.158734+0000) 2022-09-08T01:46:14.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:14.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:14.219860+0000) 2022-09-08T01:46:14.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:14.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:14.158914+0000) 2022-09-08T01:46:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:15 smithi055 ceph-mon[97033]: pgmap v7984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:46:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:15 smithi059 ceph-mon[97450]: pgmap v7984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:46:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:15.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:15.220146+0000) 2022-09-08T01:46:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:15.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:15.159152+0000) 2022-09-08T01:46:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:16.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:16.220327+0000) 2022-09-08T01:46:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:16.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:16.159386+0000) 2022-09-08T01:46:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:17 smithi055 ceph-mon[97033]: pgmap v7985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:46:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:17 smithi059 ceph-mon[97450]: pgmap v7985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:46:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:17.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:17.220533+0000) 2022-09-08T01:46:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:17.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:17.159697+0000) 2022-09-08T01:46:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:18.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:18.159936+0000) 2022-09-08T01:46:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:18.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:18.220701+0000) 2022-09-08T01:46:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:19 smithi055 ceph-mon[97033]: pgmap v7986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T01:46:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:19 smithi059 ceph-mon[97450]: pgmap v7986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T01:46:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:19.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:19.220932+0000) 2022-09-08T01:46:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:19.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:19.160190+0000) 2022-09-08T01:46:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:20.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:20.221122+0000) 2022-09-08T01:46:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:20.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:20.160429+0000) 2022-09-08T01:46:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:21 smithi055 ceph-mon[97033]: pgmap v7987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:46:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:21 smithi059 ceph-mon[97450]: pgmap v7987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:46:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:21.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:21.160597+0000) 2022-09-08T01:46:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:21.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:21.221391+0000) 2022-09-08T01:46:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:22.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:22.221605+0000) 2022-09-08T01:46:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:22.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:22.160813+0000) 2022-09-08T01:46:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:23 smithi055 ceph-mon[97033]: pgmap v7988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:46:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:23 smithi059 ceph-mon[97450]: pgmap v7988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:46:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:23.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:23.221798+0000) 2022-09-08T01:46:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:23.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:23.161060+0000) 2022-09-08T01:46:24.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:46:24.119+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:46:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:24.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:24.221969+0000) 2022-09-08T01:46:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:24.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:24.161221+0000) 2022-09-08T01:46:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:25 smithi055 ceph-mon[97033]: pgmap v7989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:46:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:25 smithi059 ceph-mon[97450]: pgmap v7989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:46:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:25.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:25.222143+0000) 2022-09-08T01:46:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:25.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:25.161412+0000) 2022-09-08T01:46:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:26.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:26.222351+0000) 2022-09-08T01:46:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:26.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:26.161596+0000) 2022-09-08T01:46:27.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:27 smithi055 ceph-mon[97033]: pgmap v7990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:46:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:27 smithi059 ceph-mon[97450]: pgmap v7990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:46:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:27.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:27.222493+0000) 2022-09-08T01:46:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:27.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:27.161834+0000) 2022-09-08T01:46:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:28.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:28.222769+0000) 2022-09-08T01:46:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:28.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:28.162108+0000) 2022-09-08T01:46:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:29 smithi055 ceph-mon[97033]: pgmap v7991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:29 smithi059 ceph-mon[97450]: pgmap v7991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:29.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:29.222964+0000) 2022-09-08T01:46:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:29.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:29.162353+0000) 2022-09-08T01:46:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:30.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:30.223140+0000) 2022-09-08T01:46:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:30.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:30.162568+0000) 2022-09-08T01:46:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:31 smithi055 ceph-mon[97033]: pgmap v7992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:46:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:31 smithi059 ceph-mon[97450]: pgmap v7992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:46:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:31.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:31.223336+0000) 2022-09-08T01:46:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:31.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:31.162792+0000) 2022-09-08T01:46:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:32.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:32.223530+0000) 2022-09-08T01:46:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:32.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:32.162978+0000) 2022-09-08T01:46:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:33 smithi055 ceph-mon[97033]: pgmap v7993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:33 smithi059 ceph-mon[97450]: pgmap v7993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:33.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:33.223727+0000) 2022-09-08T01:46:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:33.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:33.163215+0000) 2022-09-08T01:46:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:34.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:34.223961+0000) 2022-09-08T01:46:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:34.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:34.163450+0000) 2022-09-08T01:46:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:35 smithi055 ceph-mon[97033]: pgmap v7994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:35 smithi059 ceph-mon[97450]: pgmap v7994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:35.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:35.224176+0000) 2022-09-08T01:46:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:35.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:35.163663+0000) 2022-09-08T01:46:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:36.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:36.224396+0000) 2022-09-08T01:46:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:36.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:36.163841+0000) 2022-09-08T01:46:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:37 smithi055 ceph-mon[97033]: pgmap v7995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:46:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:37 smithi059 ceph-mon[97450]: pgmap v7995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:46:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:37.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:37.224553+0000) 2022-09-08T01:46:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:37.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:37.164034+0000) 2022-09-08T01:46:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:38.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:38.224807+0000) 2022-09-08T01:46:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:38.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:38.164224+0000) 2022-09-08T01:46:39.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:46:39.136+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:46:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:39.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:39.225012+0000) 2022-09-08T01:46:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:39.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:39.164401+0000) 2022-09-08T01:46:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:39 smithi055 ceph-mon[97033]: pgmap v7996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:39 smithi059 ceph-mon[97450]: pgmap v7996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:40.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:40.225252+0000) 2022-09-08T01:46:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:40.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:40.164550+0000) 2022-09-08T01:46:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:41 smithi055 ceph-mon[97033]: pgmap v7997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:46:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:41 smithi059 ceph-mon[97450]: pgmap v7997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:46:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:41.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:41.225501+0000) 2022-09-08T01:46:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:41.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:41.164784+0000) 2022-09-08T01:46:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:42.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:42.225752+0000) 2022-09-08T01:46:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:42.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:42.164929+0000) 2022-09-08T01:46:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:43 smithi055 ceph-mon[97033]: pgmap v7998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:43 smithi059 ceph-mon[97450]: pgmap v7998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:43.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:43.225970+0000) 2022-09-08T01:46:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:43.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:43.165090+0000) 2022-09-08T01:46:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:44.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:44.226116+0000) 2022-09-08T01:46:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:44.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:44.165291+0000) 2022-09-08T01:46:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:45 smithi055 ceph-mon[97033]: pgmap v7999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:45 smithi059 ceph-mon[97450]: pgmap v7999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:45.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:45.226321+0000) 2022-09-08T01:46:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:45.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:45.165484+0000) 2022-09-08T01:46:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:46.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:46.226558+0000) 2022-09-08T01:46:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:46.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:46.165716+0000) 2022-09-08T01:46:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:47 smithi055 ceph-mon[97033]: pgmap v8000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:46:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:47 smithi059 ceph-mon[97450]: pgmap v8000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:46:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:47.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:47.226795+0000) 2022-09-08T01:46:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:47.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:47.165954+0000) 2022-09-08T01:46:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:48.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:48.226992+0000) 2022-09-08T01:46:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:48.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:48.166206+0000) 2022-09-08T01:46:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:49 smithi055 ceph-mon[97033]: pgmap v8001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:49 smithi059 ceph-mon[97450]: pgmap v8001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:49.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:49.227240+0000) 2022-09-08T01:46:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:49.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:49.166430+0000) 2022-09-08T01:46:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:50.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:50.227482+0000) 2022-09-08T01:46:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:50.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:50.166689+0000) 2022-09-08T01:46:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:51 smithi055 ceph-mon[97033]: pgmap v8002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:46:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:51 smithi059 ceph-mon[97450]: pgmap v8002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:46:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:51.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:51.227669+0000) 2022-09-08T01:46:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:51.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:51.166940+0000) 2022-09-08T01:46:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:52.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:52.227829+0000) 2022-09-08T01:46:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:52.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:52.167059+0000) 2022-09-08T01:46:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:53 smithi055 ceph-mon[97033]: pgmap v8003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:53 smithi059 ceph-mon[97450]: pgmap v8003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:53.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:53.228059+0000) 2022-09-08T01:46:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:53.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:53.167191+0000) 2022-09-08T01:46:54.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:46:54.150+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:46:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:54.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:54.228292+0000) 2022-09-08T01:46:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:54.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:54.167412+0000) 2022-09-08T01:46:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:55 smithi055 ceph-mon[97033]: pgmap v8004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:55 smithi059 ceph-mon[97450]: pgmap v8004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:55.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:55.228516+0000) 2022-09-08T01:46:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:55.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:55.167618+0000) 2022-09-08T01:46:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:56.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:56.228703+0000) 2022-09-08T01:46:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:56.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:56.167822+0000) 2022-09-08T01:46:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:57 smithi055 ceph-mon[97033]: pgmap v8005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:46:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:57 smithi059 ceph-mon[97450]: pgmap v8005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:46:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:57.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:57.228959+0000) 2022-09-08T01:46:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:57.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:57.168021+0000) 2022-09-08T01:46:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:58.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:58.229231+0000) 2022-09-08T01:46:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:58.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:58.168233+0000) 2022-09-08T01:46:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:46:59 smithi059 ceph-mon[97450]: pgmap v8006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:46:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:46:59.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:59.229505+0000) 2022-09-08T01:46:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:46:59.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:46:59.168431+0000) 2022-09-08T01:46:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:46:59 smithi055 ceph-mon[97033]: pgmap v8006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:00.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:00.229666+0000) 2022-09-08T01:47:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:00.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:00.168571+0000) 2022-09-08T01:47:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:01 smithi059 ceph-mon[97450]: pgmap v8007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:47:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:01.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:01.229828+0000) 2022-09-08T01:47:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:01.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:01.168753+0000) 2022-09-08T01:47:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:01 smithi055 ceph-mon[97033]: pgmap v8007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:47:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:02.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:02.230015+0000) 2022-09-08T01:47:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:02.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:02.168957+0000) 2022-09-08T01:47:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:03 smithi059 ceph-mon[97450]: pgmap v8008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:03.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:03.230177+0000) 2022-09-08T01:47:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:03.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:03.169245+0000) 2022-09-08T01:47:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:03 smithi055 ceph-mon[97033]: pgmap v8008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:04.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:04.230404+0000) 2022-09-08T01:47:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:04.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:04.169485+0000) 2022-09-08T01:47:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:05 smithi059 ceph-mon[97450]: pgmap v8009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:47:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:47:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:05.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:05.230563+0000) 2022-09-08T01:47:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:05.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:05.169745+0000) 2022-09-08T01:47:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:05 smithi055 ceph-mon[97033]: pgmap v8009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:47:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:47:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:06.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:06.230728+0000) 2022-09-08T01:47:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:06.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:06.169863+0000) 2022-09-08T01:47:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:07 smithi059 ceph-mon[97450]: pgmap v8010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:47:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:07.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:07.230885+0000) 2022-09-08T01:47:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:07.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:07.170067+0000) 2022-09-08T01:47:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:07 smithi055 ceph-mon[97033]: pgmap v8010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:47:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:08.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:08.231008+0000) 2022-09-08T01:47:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:08.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:08.170305+0000) 2022-09-08T01:47:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:09 smithi059 ceph-mon[97450]: pgmap v8011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:47:09.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:47:09.153+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:47:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:09.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:09.231159+0000) 2022-09-08T01:47:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:09.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:09.170546+0000) 2022-09-08T01:47:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:09 smithi055 ceph-mon[97033]: pgmap v8011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:47:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:10 smithi059 ceph-mon[97450]: osdmap e583: 12 total, 11 up, 11 in 2022-09-08T01:47:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:10.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:10.231350+0000) 2022-09-08T01:47:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:10.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:10.170696+0000) 2022-09-08T01:47:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:10 smithi055 ceph-mon[97033]: osdmap e583: 12 total, 11 up, 11 in 2022-09-08T01:47:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:11 smithi059 ceph-mon[97450]: pgmap v8013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:47:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:47:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:47:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:47:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:11.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:11.231555+0000) 2022-09-08T01:47:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:11.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:11.170936+0000) 2022-09-08T01:47:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:11 smithi055 ceph-mon[97033]: pgmap v8013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:47:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:47:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:47:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:47:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:12.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:12.231745+0000) 2022-09-08T01:47:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:12.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:12.171152+0000) 2022-09-08T01:47:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:47:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:47:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:47:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:12 smithi055 ceph-mon[97033]: osdmap e584: 12 total, 11 up, 11 in 2022-09-08T01:47:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:47:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:47:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:47:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:12 smithi059 ceph-mon[97450]: osdmap e584: 12 total, 11 up, 11 in 2022-09-08T01:47:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:13.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:13.231898+0000) 2022-09-08T01:47:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:13.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:13.171378+0000) 2022-09-08T01:47:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:13 smithi055 ceph-mon[97033]: pgmap v8014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:47:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:13 smithi059 ceph-mon[97450]: pgmap v8014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:47:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:14.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:14.232053+0000) 2022-09-08T01:47:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:14.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:14.171571+0000) 2022-09-08T01:47:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:15.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:15.232250+0000) 2022-09-08T01:47:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:15.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:15.171829+0000) 2022-09-08T01:47:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:15 smithi055 ceph-mon[97033]: pgmap v8016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:47:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:15 smithi059 ceph-mon[97450]: pgmap v8016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:47:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:16.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:16.232450+0000) 2022-09-08T01:47:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:16.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:16.171983+0000) 2022-09-08T01:47:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:17.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:17.232622+0000) 2022-09-08T01:47:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:17.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:17.172253+0000) 2022-09-08T01:47:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:17 smithi055 ceph-mon[97033]: pgmap v8017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:47:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:17 smithi059 ceph-mon[97450]: pgmap v8017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:47:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:18.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:18.232796+0000) 2022-09-08T01:47:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:18.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:18.172370+0000) 2022-09-08T01:47:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:19.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:19.232929+0000) 2022-09-08T01:47:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:19.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:19.172608+0000) 2022-09-08T01:47:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:19 smithi055 ceph-mon[97033]: pgmap v8018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-08T01:47:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:19 smithi059 ceph-mon[97450]: pgmap v8018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-08T01:47:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:20.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:20.233140+0000) 2022-09-08T01:47:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:20.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:20.172795+0000) 2022-09-08T01:47:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:21.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:21.233349+0000) 2022-09-08T01:47:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:21.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:21.173002+0000) 2022-09-08T01:47:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:21 smithi055 ceph-mon[97033]: pgmap v8019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:47:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:21 smithi059 ceph-mon[97450]: pgmap v8019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:47:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:22.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:22.233506+0000) 2022-09-08T01:47:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:22.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:22.173146+0000) 2022-09-08T01:47:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:23.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:23.233694+0000) 2022-09-08T01:47:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:23.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:23.173339+0000) 2022-09-08T01:47:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:23 smithi055 ceph-mon[97033]: pgmap v8020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:47:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:23 smithi059 ceph-mon[97450]: pgmap v8020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:47:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:24.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:24.233863+0000) 2022-09-08T01:47:24.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:47:24.164+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:47:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:24.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:24.173576+0000) 2022-09-08T01:47:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:25.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:25.234029+0000) 2022-09-08T01:47:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:25.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:25.173763+0000) 2022-09-08T01:47:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:25 smithi055 ceph-mon[97033]: pgmap v8021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:47:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:25 smithi059 ceph-mon[97450]: pgmap v8021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:47:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:26.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:26.234199+0000) 2022-09-08T01:47:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:26.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:26.173929+0000) 2022-09-08T01:47:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:27.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:27.234414+0000) 2022-09-08T01:47:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:27.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:27.174124+0000) 2022-09-08T01:47:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:27 smithi055 ceph-mon[97033]: pgmap v8022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:47:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:27 smithi059 ceph-mon[97450]: pgmap v8022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:47:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:28.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:28.234683+0000) 2022-09-08T01:47:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:28.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:28.174358+0000) 2022-09-08T01:47:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:29.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:29.234924+0000) 2022-09-08T01:47:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:29.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:29.174535+0000) 2022-09-08T01:47:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:29 smithi055 ceph-mon[97033]: pgmap v8023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:29 smithi059 ceph-mon[97450]: pgmap v8023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:30.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:30.235093+0000) 2022-09-08T01:47:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:30.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:30.174730+0000) 2022-09-08T01:47:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:31.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:31.235341+0000) 2022-09-08T01:47:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:31.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:31.174882+0000) 2022-09-08T01:47:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:31 smithi055 ceph-mon[97033]: pgmap v8024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:47:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:31 smithi059 ceph-mon[97450]: pgmap v8024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:47:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:32.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:32.235568+0000) 2022-09-08T01:47:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:32.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:32.175140+0000) 2022-09-08T01:47:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:33.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:33.235792+0000) 2022-09-08T01:47:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:33.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:33.175378+0000) 2022-09-08T01:47:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:33 smithi055 ceph-mon[97033]: pgmap v8025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:33 smithi059 ceph-mon[97450]: pgmap v8025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:34.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:34.235982+0000) 2022-09-08T01:47:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:34.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:34.175673+0000) 2022-09-08T01:47:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:35.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:35.236192+0000) 2022-09-08T01:47:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:35.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:35.175915+0000) 2022-09-08T01:47:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:35 smithi055 ceph-mon[97033]: pgmap v8026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:35 smithi059 ceph-mon[97450]: pgmap v8026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:36.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:36.236411+0000) 2022-09-08T01:47:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:36.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:36.176113+0000) 2022-09-08T01:47:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:37.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:37.236662+0000) 2022-09-08T01:47:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:37.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:37.176337+0000) 2022-09-08T01:47:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:37 smithi055 ceph-mon[97033]: pgmap v8027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:47:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:37 smithi059 ceph-mon[97450]: pgmap v8027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:47:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:38.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:38.236852+0000) 2022-09-08T01:47:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:38.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:38.176589+0000) 2022-09-08T01:47:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:39.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:39.237045+0000) 2022-09-08T01:47:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:39.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:39.176750+0000) 2022-09-08T01:47:39.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:47:39.170+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:47:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:39 smithi055 ceph-mon[97033]: pgmap v8028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:39 smithi059 ceph-mon[97450]: pgmap v8028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:40.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:40.237195+0000) 2022-09-08T01:47:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:40.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:40.176964+0000) 2022-09-08T01:47:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:41.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:41.237452+0000) 2022-09-08T01:47:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:41.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:41.177170+0000) 2022-09-08T01:47:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:41 smithi055 ceph-mon[97033]: pgmap v8029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:47:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:41 smithi059 ceph-mon[97450]: pgmap v8029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:47:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:42.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:42.237706+0000) 2022-09-08T01:47:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:42.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:42.177367+0000) 2022-09-08T01:47:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:43 smithi055 ceph-mon[97033]: pgmap v8030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:43 smithi059 ceph-mon[97450]: pgmap v8030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:43.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:43.237873+0000) 2022-09-08T01:47:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:43.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:43.177619+0000) 2022-09-08T01:47:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:44.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:44.238121+0000) 2022-09-08T01:47:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:44.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:44.177898+0000) 2022-09-08T01:47:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:45 smithi055 ceph-mon[97033]: pgmap v8031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:45 smithi059 ceph-mon[97450]: pgmap v8031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:45.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:45.238336+0000) 2022-09-08T01:47:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:45.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:45.178080+0000) 2022-09-08T01:47:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:46.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:46.238544+0000) 2022-09-08T01:47:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:46.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:46.178231+0000) 2022-09-08T01:47:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:47 smithi055 ceph-mon[97033]: pgmap v8032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:47:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:47 smithi059 ceph-mon[97450]: pgmap v8032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:47:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:47.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:47.238755+0000) 2022-09-08T01:47:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:47.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:47.178423+0000) 2022-09-08T01:47:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:48.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:48.238961+0000) 2022-09-08T01:47:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:48.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:48.178579+0000) 2022-09-08T01:47:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:49 smithi055 ceph-mon[97033]: pgmap v8033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:49 smithi059 ceph-mon[97450]: pgmap v8033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:49.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:49.239194+0000) 2022-09-08T01:47:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:49.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:49.178842+0000) 2022-09-08T01:47:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:50.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:50.239391+0000) 2022-09-08T01:47:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:50.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:50.179085+0000) 2022-09-08T01:47:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:51 smithi055 ceph-mon[97033]: pgmap v8034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:47:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:51 smithi059 ceph-mon[97450]: pgmap v8034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:47:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:51.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:51.239580+0000) 2022-09-08T01:47:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:51.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:51.179297+0000) 2022-09-08T01:47:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:52.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:52.239813+0000) 2022-09-08T01:47:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:52.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:52.179539+0000) 2022-09-08T01:47:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:53 smithi055 ceph-mon[97033]: pgmap v8035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:53 smithi059 ceph-mon[97450]: pgmap v8035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:53.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:53.240071+0000) 2022-09-08T01:47:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:53.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:53.179748+0000) 2022-09-08T01:47:54.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:47:54.171+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:47:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:54.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:54.240251+0000) 2022-09-08T01:47:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:54.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:54.179935+0000) 2022-09-08T01:47:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:55 smithi055 ceph-mon[97033]: pgmap v8036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:55 smithi059 ceph-mon[97450]: pgmap v8036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:55.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:55.240456+0000) 2022-09-08T01:47:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:55.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:55.180087+0000) 2022-09-08T01:47:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:56.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:56.240604+0000) 2022-09-08T01:47:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:56.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:56.180307+0000) 2022-09-08T01:47:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:57 smithi055 ceph-mon[97033]: pgmap v8037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:47:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:57 smithi059 ceph-mon[97450]: pgmap v8037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:47:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:57.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:57.240746+0000) 2022-09-08T01:47:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:57.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:57.180494+0000) 2022-09-08T01:47:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:58.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:58.240878+0000) 2022-09-08T01:47:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:58.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:58.180730+0000) 2022-09-08T01:47:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:47:59 smithi055 ceph-mon[97033]: pgmap v8038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:47:59 smithi059 ceph-mon[97450]: pgmap v8038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:47:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:47:59.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:59.241055+0000) 2022-09-08T01:47:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:47:59.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:47:59.180937+0000) 2022-09-08T01:48:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:00.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:00.241296+0000) 2022-09-08T01:48:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:00.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:00.181100+0000) 2022-09-08T01:48:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:01 smithi055 ceph-mon[97033]: pgmap v8039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:48:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:01 smithi059 ceph-mon[97450]: pgmap v8039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:48:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:01.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:01.241430+0000) 2022-09-08T01:48:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:01.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:01.181229+0000) 2022-09-08T01:48:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:02.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:02.181449+0000) 2022-09-08T01:48:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:02.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:02.241664+0000) 2022-09-08T01:48:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:03 smithi055 ceph-mon[97033]: pgmap v8040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:03 smithi059 ceph-mon[97450]: pgmap v8040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:03.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:03.241918+0000) 2022-09-08T01:48:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:03.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:03.181703+0000) 2022-09-08T01:48:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:04.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:04.242149+0000) 2022-09-08T01:48:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:04.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:04.181924+0000) 2022-09-08T01:48:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:05 smithi055 ceph-mon[97033]: pgmap v8041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:05 smithi059 ceph-mon[97450]: pgmap v8041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:05.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:05.182157+0000) 2022-09-08T01:48:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:05.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:05.242383+0000) 2022-09-08T01:48:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:48:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:48:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:48:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:48:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:06.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:06.242596+0000) 2022-09-08T01:48:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:06.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:06.182364+0000) 2022-09-08T01:48:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:07 smithi055 ceph-mon[97033]: pgmap v8042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:48:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:07 smithi059 ceph-mon[97450]: pgmap v8042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:48:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:07.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:07.242786+0000) 2022-09-08T01:48:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:07.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:07.182616+0000) 2022-09-08T01:48:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:08.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:08.182854+0000) 2022-09-08T01:48:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:08.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:08.242991+0000) 2022-09-08T01:48:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:09 smithi055 ceph-mon[97033]: pgmap v8043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:48:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:09 smithi059 ceph-mon[97450]: pgmap v8043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:48:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:09.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:09.243171+0000) 2022-09-08T01:48:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:09.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:09.183098+0000) 2022-09-08T01:48:09.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:48:09.174+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:48:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:10 smithi055 ceph-mon[97033]: osdmap e585: 12 total, 11 up, 11 in 2022-09-08T01:48:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:10 smithi059 ceph-mon[97450]: osdmap e585: 12 total, 11 up, 11 in 2022-09-08T01:48:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:10.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:10.243386+0000) 2022-09-08T01:48:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:10.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:10.183347+0000) 2022-09-08T01:48:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:11 smithi055 ceph-mon[97033]: pgmap v8045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:48:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:11 smithi059 ceph-mon[97450]: pgmap v8045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:48:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:11.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:11.243633+0000) 2022-09-08T01:48:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:11.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:11.183594+0000) 2022-09-08T01:48:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:48:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:48:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:48:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:12 smithi059 ceph-mon[97450]: osdmap e586: 12 total, 11 up, 11 in 2022-09-08T01:48:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:48:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:48:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:48:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:12 smithi055 ceph-mon[97033]: osdmap e586: 12 total, 11 up, 11 in 2022-09-08T01:48:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:12.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:12.243823+0000) 2022-09-08T01:48:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:12.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:12.183845+0000) 2022-09-08T01:48:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:13 smithi055 ceph-mon[97033]: pgmap v8047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:48:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:48:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:13 smithi059 ceph-mon[97450]: pgmap v8047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:48:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:48:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:13.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:13.244032+0000) 2022-09-08T01:48:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:13.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:13.184059+0000) 2022-09-08T01:48:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:14.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:14.244227+0000) 2022-09-08T01:48:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:14.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:14.184256+0000) 2022-09-08T01:48:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:15 smithi055 ceph-mon[97033]: pgmap v8048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:48:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:15 smithi059 ceph-mon[97450]: pgmap v8048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:48:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:15.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:15.244430+0000) 2022-09-08T01:48:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:15.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:15.184474+0000) 2022-09-08T01:48:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:16.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:16.244599+0000) 2022-09-08T01:48:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:16.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:16.184654+0000) 2022-09-08T01:48:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:17 smithi055 ceph-mon[97033]: pgmap v8049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:48:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:17 smithi059 ceph-mon[97450]: pgmap v8049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:48:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:17.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:17.184900+0000) 2022-09-08T01:48:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:17.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:17.244856+0000) 2022-09-08T01:48:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:18.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:18.245092+0000) 2022-09-08T01:48:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:18.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:18.185030+0000) 2022-09-08T01:48:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:19 smithi055 ceph-mon[97033]: pgmap v8050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T01:48:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:19 smithi059 ceph-mon[97450]: pgmap v8050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T01:48:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:19.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:19.245303+0000) 2022-09-08T01:48:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:19.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:19.185277+0000) 2022-09-08T01:48:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:20.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:20.245461+0000) 2022-09-08T01:48:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:20.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:20.185527+0000) 2022-09-08T01:48:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:21 smithi055 ceph-mon[97033]: pgmap v8051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:48:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:21 smithi059 ceph-mon[97450]: pgmap v8051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:48:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:21.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:21.245606+0000) 2022-09-08T01:48:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:21.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:21.185749+0000) 2022-09-08T01:48:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:22.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:22.245838+0000) 2022-09-08T01:48:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:22.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:22.186017+0000) 2022-09-08T01:48:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:23 smithi055 ceph-mon[97033]: pgmap v8052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:48:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:23 smithi059 ceph-mon[97450]: pgmap v8052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:48:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:23.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:23.245990+0000) 2022-09-08T01:48:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:23.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:23.186281+0000) 2022-09-08T01:48:24.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:48:24.185+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:48:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:24.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:24.246197+0000) 2022-09-08T01:48:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:24.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:24.186463+0000) 2022-09-08T01:48:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:25 smithi055 ceph-mon[97033]: pgmap v8053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:48:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:25 smithi059 ceph-mon[97450]: pgmap v8053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:48:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:25.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:25.246409+0000) 2022-09-08T01:48:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:25.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:25.186666+0000) 2022-09-08T01:48:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:26.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:26.246674+0000) 2022-09-08T01:48:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:26.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:26.186846+0000) 2022-09-08T01:48:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:27 smithi055 ceph-mon[97033]: pgmap v8054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:48:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:27 smithi059 ceph-mon[97450]: pgmap v8054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:48:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:27.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:27.246924+0000) 2022-09-08T01:48:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:27.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:27.187042+0000) 2022-09-08T01:48:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:28.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:28.247182+0000) 2022-09-08T01:48:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:28.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:28.187273+0000) 2022-09-08T01:48:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:29 smithi055 ceph-mon[97033]: pgmap v8055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:29 smithi059 ceph-mon[97450]: pgmap v8055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:29.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:29.247401+0000) 2022-09-08T01:48:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:29.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:29.187496+0000) 2022-09-08T01:48:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:30.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:30.247677+0000) 2022-09-08T01:48:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:30.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:30.187687+0000) 2022-09-08T01:48:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:31 smithi055 ceph-mon[97033]: pgmap v8056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:48:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:31 smithi059 ceph-mon[97450]: pgmap v8056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:48:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:31.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:31.247943+0000) 2022-09-08T01:48:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:31.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:31.187904+0000) 2022-09-08T01:48:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:32.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:32.248130+0000) 2022-09-08T01:48:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:32.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:32.188125+0000) 2022-09-08T01:48:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:33 smithi055 ceph-mon[97033]: pgmap v8057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:33 smithi059 ceph-mon[97450]: pgmap v8057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:33.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:33.248355+0000) 2022-09-08T01:48:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:33.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:33.188395+0000) 2022-09-08T01:48:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:34.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:34.248663+0000) 2022-09-08T01:48:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:34.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:34.188571+0000) 2022-09-08T01:48:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:35 smithi055 ceph-mon[97033]: pgmap v8058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:35 smithi059 ceph-mon[97450]: pgmap v8058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:35.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:35.248823+0000) 2022-09-08T01:48:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:35.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:35.188774+0000) 2022-09-08T01:48:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:36.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:36.248974+0000) 2022-09-08T01:48:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:36.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:36.188977+0000) 2022-09-08T01:48:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:37 smithi055 ceph-mon[97033]: pgmap v8059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:48:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:37 smithi059 ceph-mon[97450]: pgmap v8059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:48:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:37.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:37.249183+0000) 2022-09-08T01:48:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:37.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:37.189247+0000) 2022-09-08T01:48:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:38.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:38.249410+0000) 2022-09-08T01:48:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:38.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:38.189450+0000) 2022-09-08T01:48:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:39 smithi055 ceph-mon[97033]: pgmap v8060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:39 smithi059 ceph-mon[97450]: pgmap v8060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:39.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:48:39.188+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:48:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:39.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:39.249552+0000) 2022-09-08T01:48:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:39.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:39.189617+0000) 2022-09-08T01:48:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:40.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:40.249773+0000) 2022-09-08T01:48:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:40.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:40.189832+0000) 2022-09-08T01:48:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:41 smithi055 ceph-mon[97033]: pgmap v8061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:48:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:41 smithi059 ceph-mon[97450]: pgmap v8061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:48:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:41.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:41.250024+0000) 2022-09-08T01:48:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:41.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:41.190045+0000) 2022-09-08T01:48:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:42.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:42.250215+0000) 2022-09-08T01:48:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:42.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:42.190262+0000) 2022-09-08T01:48:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:43 smithi055 ceph-mon[97033]: pgmap v8062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:43 smithi059 ceph-mon[97450]: pgmap v8062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:43.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:43.190523+0000) 2022-09-08T01:48:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:43.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:43.250408+0000) 2022-09-08T01:48:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:44.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:44.250590+0000) 2022-09-08T01:48:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:44.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:44.190794+0000) 2022-09-08T01:48:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:45 smithi055 ceph-mon[97033]: pgmap v8063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:45 smithi059 ceph-mon[97450]: pgmap v8063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:45.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:45.250812+0000) 2022-09-08T01:48:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:45.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:45.191001+0000) 2022-09-08T01:48:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:46.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:46.251049+0000) 2022-09-08T01:48:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:46.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:46.191214+0000) 2022-09-08T01:48:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:47 smithi055 ceph-mon[97033]: pgmap v8064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:48:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:47 smithi059 ceph-mon[97450]: pgmap v8064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:48:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:47.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:47.251261+0000) 2022-09-08T01:48:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:47.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:47.191396+0000) 2022-09-08T01:48:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:48.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:48.251473+0000) 2022-09-08T01:48:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:48.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:48.191579+0000) 2022-09-08T01:48:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:49 smithi055 ceph-mon[97033]: pgmap v8065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:49 smithi059 ceph-mon[97450]: pgmap v8065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:49.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:49.251713+0000) 2022-09-08T01:48:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:49.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:49.191831+0000) 2022-09-08T01:48:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:50.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:50.192094+0000) 2022-09-08T01:48:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:50.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:50.251889+0000) 2022-09-08T01:48:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:51 smithi055 ceph-mon[97033]: pgmap v8066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:48:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:51 smithi059 ceph-mon[97450]: pgmap v8066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:48:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:51.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:51.252109+0000) 2022-09-08T01:48:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:51.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:51.192346+0000) 2022-09-08T01:48:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:52.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:52.252262+0000) 2022-09-08T01:48:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:52.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:52.192586+0000) 2022-09-08T01:48:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:53 smithi055 ceph-mon[97033]: pgmap v8067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:53 smithi059 ceph-mon[97450]: pgmap v8067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:53.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:53.252437+0000) 2022-09-08T01:48:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:53.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:53.192799+0000) 2022-09-08T01:48:54.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:48:54.202+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:48:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:54.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:54.252603+0000) 2022-09-08T01:48:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:54.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:54.192915+0000) 2022-09-08T01:48:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:55 smithi055 ceph-mon[97033]: pgmap v8068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:55 smithi059 ceph-mon[97450]: pgmap v8068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:55.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:55.252754+0000) 2022-09-08T01:48:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:55.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:55.193136+0000) 2022-09-08T01:48:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:56.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:56.252898+0000) 2022-09-08T01:48:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:56.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:56.193362+0000) 2022-09-08T01:48:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:57 smithi055 ceph-mon[97033]: pgmap v8069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:48:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:57 smithi059 ceph-mon[97450]: pgmap v8069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:48:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:57.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:57.253088+0000) 2022-09-08T01:48:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:57.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:57.193671+0000) 2022-09-08T01:48:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:58.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:58.253308+0000) 2022-09-08T01:48:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:58.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:58.193886+0000) 2022-09-08T01:48:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:48:59 smithi055 ceph-mon[97033]: pgmap v8070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:48:59 smithi059 ceph-mon[97450]: pgmap v8070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:48:59.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:48:59.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:59.194140+0000) 2022-09-08T01:48:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:48:59.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:48:59.253474+0000) 2022-09-08T01:49:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:00.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:00.253603+0000) 2022-09-08T01:49:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:00.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:00.194396+0000) 2022-09-08T01:49:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:01 smithi055 ceph-mon[97033]: pgmap v8071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:49:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:01 smithi059 ceph-mon[97450]: pgmap v8071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:49:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:01.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:01.253754+0000) 2022-09-08T01:49:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:01.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:01.194669+0000) 2022-09-08T01:49:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:02.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:02.253941+0000) 2022-09-08T01:49:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:02.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:02.194865+0000) 2022-09-08T01:49:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:03 smithi055 ceph-mon[97033]: pgmap v8072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:03 smithi059 ceph-mon[97450]: pgmap v8072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:03.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:03.254093+0000) 2022-09-08T01:49:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:03.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:03.195072+0000) 2022-09-08T01:49:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:04.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:04.254327+0000) 2022-09-08T01:49:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:04.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:04.195327+0000) 2022-09-08T01:49:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:05 smithi055 ceph-mon[97033]: pgmap v8073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:05 smithi059 ceph-mon[97450]: pgmap v8073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:05.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:05.195562+0000) 2022-09-08T01:49:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:05.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:05.254556+0000) 2022-09-08T01:49:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:49:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:49:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:49:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:49:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:06.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:06.254784+0000) 2022-09-08T01:49:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:06.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:06.195771+0000) 2022-09-08T01:49:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:07 smithi055 ceph-mon[97033]: pgmap v8074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:49:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:07 smithi059 ceph-mon[97450]: pgmap v8074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:49:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:07.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:07.255041+0000) 2022-09-08T01:49:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:07.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:07.196065+0000) 2022-09-08T01:49:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:08.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:08.255291+0000) 2022-09-08T01:49:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:08.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:08.196272+0000) 2022-09-08T01:49:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:09 smithi055 ceph-mon[97033]: pgmap v8075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:49:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:09 smithi059 ceph-mon[97450]: pgmap v8075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:49:09.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:49:09.204+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:49:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:09.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:09.255445+0000) 2022-09-08T01:49:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:09.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:09.196553+0000) 2022-09-08T01:49:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:10 smithi055 ceph-mon[97033]: osdmap e587: 12 total, 11 up, 11 in 2022-09-08T01:49:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:10 smithi059 ceph-mon[97450]: osdmap e587: 12 total, 11 up, 11 in 2022-09-08T01:49:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:10.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:10.255652+0000) 2022-09-08T01:49:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:10.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:10.196776+0000) 2022-09-08T01:49:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:11 smithi055 ceph-mon[97033]: pgmap v8077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:49:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:11 smithi059 ceph-mon[97450]: pgmap v8077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:49:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:11.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:11.255772+0000) 2022-09-08T01:49:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:11.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:11.196996+0000) 2022-09-08T01:49:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:12.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:12.255992+0000) 2022-09-08T01:49:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:12.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:12.197253+0000) 2022-09-08T01:49:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:13 smithi055 ceph-mon[97033]: osdmap e588: 12 total, 11 up, 11 in 2022-09-08T01:49:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:13 smithi055 ceph-mon[97033]: pgmap v8079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:49:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:49:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:49:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:49:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:13 smithi059 ceph-mon[97450]: osdmap e588: 12 total, 11 up, 11 in 2022-09-08T01:49:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:13 smithi059 ceph-mon[97450]: pgmap v8079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:49:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:49:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:49:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:49:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:13.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:13.256204+0000) 2022-09-08T01:49:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:13.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:13.197425+0000) 2022-09-08T01:49:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:14.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:14.256396+0000) 2022-09-08T01:49:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:14.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:14.197615+0000) 2022-09-08T01:49:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:15 smithi055 ceph-mon[97033]: pgmap v8080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:49:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:15 smithi059 ceph-mon[97450]: pgmap v8080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:49:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:15.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:15.256609+0000) 2022-09-08T01:49:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:15.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:15.197838+0000) 2022-09-08T01:49:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:16.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:16.198082+0000) 2022-09-08T01:49:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:16.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:16.256876+0000) 2022-09-08T01:49:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:17 smithi055 ceph-mon[97033]: pgmap v8081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:49:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:17 smithi059 ceph-mon[97450]: pgmap v8081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:49:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:17.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:17.257060+0000) 2022-09-08T01:49:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:17.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:17.198293+0000) 2022-09-08T01:49:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:18.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:18.257316+0000) 2022-09-08T01:49:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:18.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:18.198489+0000) 2022-09-08T01:49:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:19 smithi055 ceph-mon[97033]: pgmap v8082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T01:49:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:19 smithi059 ceph-mon[97450]: pgmap v8082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T01:49:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:19.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:19.257453+0000) 2022-09-08T01:49:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:19.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:19.198701+0000) 2022-09-08T01:49:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:20.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:20.257631+0000) 2022-09-08T01:49:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:20.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:20.198925+0000) 2022-09-08T01:49:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:21 smithi055 ceph-mon[97033]: pgmap v8083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:49:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:21 smithi059 ceph-mon[97450]: pgmap v8083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:49:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:21.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:21.257861+0000) 2022-09-08T01:49:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:21.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:21.199151+0000) 2022-09-08T01:49:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:22.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:22.258026+0000) 2022-09-08T01:49:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:22.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:22.199339+0000) 2022-09-08T01:49:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:23 smithi055 ceph-mon[97033]: pgmap v8084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:49:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:23 smithi059 ceph-mon[97450]: pgmap v8084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:49:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:23.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:23.258249+0000) 2022-09-08T01:49:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:23.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:23.199599+0000) 2022-09-08T01:49:24.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:49:24.219+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:49:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:24.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:24.258462+0000) 2022-09-08T01:49:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:24.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:24.199830+0000) 2022-09-08T01:49:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:25 smithi055 ceph-mon[97033]: pgmap v8085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:49:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:25 smithi059 ceph-mon[97450]: pgmap v8085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:49:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:25.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:25.258676+0000) 2022-09-08T01:49:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:25.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:25.200087+0000) 2022-09-08T01:49:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:26.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:26.258877+0000) 2022-09-08T01:49:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:26.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:26.200260+0000) 2022-09-08T01:49:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:27 smithi055 ceph-mon[97033]: pgmap v8086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:49:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:27 smithi059 ceph-mon[97450]: pgmap v8086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:49:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:27.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:27.259094+0000) 2022-09-08T01:49:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:27.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:27.200460+0000) 2022-09-08T01:49:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:28.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:28.259251+0000) 2022-09-08T01:49:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:28.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:28.200707+0000) 2022-09-08T01:49:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:29 smithi055 ceph-mon[97033]: pgmap v8087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:29 smithi059 ceph-mon[97450]: pgmap v8087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:29.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:29.259469+0000) 2022-09-08T01:49:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:29.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:29.200888+0000) 2022-09-08T01:49:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:30.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:30.259679+0000) 2022-09-08T01:49:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:30.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:30.201128+0000) 2022-09-08T01:49:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:31 smithi055 ceph-mon[97033]: pgmap v8088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:49:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:31 smithi059 ceph-mon[97450]: pgmap v8088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:49:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:31.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:31.259855+0000) 2022-09-08T01:49:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:31.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:31.201270+0000) 2022-09-08T01:49:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:32.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:32.260045+0000) 2022-09-08T01:49:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:32.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:32.201448+0000) 2022-09-08T01:49:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:33 smithi055 ceph-mon[97033]: pgmap v8089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:33 smithi059 ceph-mon[97450]: pgmap v8089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:33.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:33.260240+0000) 2022-09-08T01:49:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:33.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:33.201707+0000) 2022-09-08T01:49:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:34.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:34.260442+0000) 2022-09-08T01:49:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:34.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:34.201933+0000) 2022-09-08T01:49:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:35 smithi055 ceph-mon[97033]: pgmap v8090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:35 smithi059 ceph-mon[97450]: pgmap v8090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:35.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:35.260599+0000) 2022-09-08T01:49:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:35.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:35.202134+0000) 2022-09-08T01:49:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:36.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:36.260875+0000) 2022-09-08T01:49:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:36.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:36.202351+0000) 2022-09-08T01:49:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:37 smithi055 ceph-mon[97033]: pgmap v8091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:49:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:37 smithi059 ceph-mon[97450]: pgmap v8091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:49:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:37.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:37.261080+0000) 2022-09-08T01:49:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:37.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:37.202598+0000) 2022-09-08T01:49:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:38.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:38.261281+0000) 2022-09-08T01:49:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:38.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:38.202781+0000) 2022-09-08T01:49:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:39 smithi055 ceph-mon[97033]: pgmap v8092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:39 smithi059 ceph-mon[97450]: pgmap v8092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:39.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:49:39.226+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:49:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:39.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:39.261436+0000) 2022-09-08T01:49:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:39.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:39.202997+0000) 2022-09-08T01:49:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:40.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:40.261566+0000) 2022-09-08T01:49:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:40.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:40.203180+0000) 2022-09-08T01:49:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:41 smithi055 ceph-mon[97033]: pgmap v8093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:49:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:41 smithi059 ceph-mon[97450]: pgmap v8093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:49:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:41.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:41.261757+0000) 2022-09-08T01:49:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:41.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:41.203450+0000) 2022-09-08T01:49:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:42.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:42.261886+0000) 2022-09-08T01:49:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:42.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:42.203718+0000) 2022-09-08T01:49:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:43 smithi059 ceph-mon[97450]: pgmap v8094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:49:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:43 smithi055 ceph-mon[97033]: pgmap v8094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:43.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:49:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:43.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:43.262121+0000) 2022-09-08T01:49:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:43.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:43.203975+0000) 2022-09-08T01:49:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:44 smithi055 ceph-mon[97033]: pgmap v8095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:49:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:44 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T01:49:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:44 smithi059 ceph-mon[97450]: pgmap v8095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:49:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:44 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T01:49:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:44.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:44.262272+0000) 2022-09-08T01:49:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:44.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:44.204179+0000) 2022-09-08T01:49:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:45.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:45.262492+0000) 2022-09-08T01:49:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:45.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:45.204372+0000) 2022-09-08T01:49:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:46 smithi055 ceph-mon[97033]: pgmap v8096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:49:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:46 smithi059 ceph-mon[97450]: pgmap v8096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:49:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:46.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:46.262713+0000) 2022-09-08T01:49:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:46.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:46.204565+0000) 2022-09-08T01:49:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:47.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:47.262939+0000) 2022-09-08T01:49:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:47.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:47.204783+0000) 2022-09-08T01:49:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:48 smithi055 ceph-mon[97033]: pgmap v8097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:49:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:48 smithi059 ceph-mon[97450]: pgmap v8097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:49:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:48.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:48.263145+0000) 2022-09-08T01:49:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:48.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:48.204997+0000) 2022-09-08T01:49:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:49.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:49.263380+0000) 2022-09-08T01:49:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:49.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:49.205252+0000) 2022-09-08T01:49:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:50 smithi055 ceph-mon[97033]: pgmap v8098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:49:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:50 smithi059 ceph-mon[97450]: pgmap v8098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:49:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:50.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:50.263590+0000) 2022-09-08T01:49:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:50.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:50.205450+0000) 2022-09-08T01:49:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:51.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:51.263790+0000) 2022-09-08T01:49:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:51.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:51.205696+0000) 2022-09-08T01:49:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:52 smithi055 ceph-mon[97033]: pgmap v8099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:49:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:52 smithi059 ceph-mon[97450]: pgmap v8099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:49:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:52.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:52.263970+0000) 2022-09-08T01:49:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:52.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:52.205834+0000) 2022-09-08T01:49:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:53.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:53.264164+0000) 2022-09-08T01:49:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:53.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:53.206040+0000) 2022-09-08T01:49:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:54 smithi055 ceph-mon[97033]: pgmap v8100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:49:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:54 smithi059 ceph-mon[97450]: pgmap v8100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:49:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:54.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:54.264317+0000) 2022-09-08T01:49:54.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:49:54.243+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:49:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:54.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:54.206194+0000) 2022-09-08T01:49:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:55.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:55.264435+0000) 2022-09-08T01:49:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:55.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:55.206403+0000) 2022-09-08T01:49:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:56 smithi055 ceph-mon[97033]: pgmap v8101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:56 smithi059 ceph-mon[97450]: pgmap v8101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:49:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:56.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:56.264627+0000) 2022-09-08T01:49:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:56.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:56.206591+0000) 2022-09-08T01:49:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:57.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:57.264791+0000) 2022-09-08T01:49:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:57.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:57.206846+0000) 2022-09-08T01:49:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:49:58 smithi055 ceph-mon[97033]: pgmap v8102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:49:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:49:58 smithi059 ceph-mon[97450]: pgmap v8102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:49:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:58.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:58.265001+0000) 2022-09-08T01:49:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:58.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:58.207014+0000) 2022-09-08T01:49:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:49:59.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:59.265200+0000) 2022-09-08T01:49:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:49:59.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:49:59.207294+0000) 2022-09-08T01:50:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:00 smithi055 ceph-mon[97033]: pgmap v8103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:00.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:00 smithi055 ceph-mon[97033]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T01:50:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:00 smithi059 ceph-mon[97450]: pgmap v8103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:00 smithi059 ceph-mon[97450]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T01:50:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:00.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:00.265325+0000) 2022-09-08T01:50:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:00.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:00.207491+0000) 2022-09-08T01:50:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:01.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:01.265509+0000) 2022-09-08T01:50:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:01.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:01.207676+0000) 2022-09-08T01:50:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:02 smithi055 ceph-mon[97033]: pgmap v8104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:50:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:02 smithi059 ceph-mon[97450]: pgmap v8104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:50:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:02.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:02.265759+0000) 2022-09-08T01:50:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:02.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:02.207887+0000) 2022-09-08T01:50:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:03.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:03.265951+0000) 2022-09-08T01:50:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:03.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:03.208205+0000) 2022-09-08T01:50:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:04 smithi055 ceph-mon[97033]: pgmap v8105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:04 smithi059 ceph-mon[97450]: pgmap v8105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:04.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:04.266130+0000) 2022-09-08T01:50:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:04.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:04.208456+0000) 2022-09-08T01:50:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:05.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:05.266260+0000) 2022-09-08T01:50:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:05.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:05.208731+0000) 2022-09-08T01:50:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:06 smithi055 ceph-mon[97033]: pgmap v8106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:50:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:50:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:06 smithi059 ceph-mon[97450]: pgmap v8106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:50:06.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:50:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:06.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:06.266477+0000) 2022-09-08T01:50:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:06.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:06.208945+0000) 2022-09-08T01:50:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:07.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:07.266759+0000) 2022-09-08T01:50:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:07.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:07.209195+0000) 2022-09-08T01:50:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:08 smithi055 ceph-mon[97033]: pgmap v8107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:50:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:08 smithi059 ceph-mon[97450]: pgmap v8107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:50:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:08.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:08.267021+0000) 2022-09-08T01:50:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:08.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:08.209409+0000) 2022-09-08T01:50:09.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:50:09.244+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:50:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:09.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:09.267271+0000) 2022-09-08T01:50:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:09.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:09.209702+0000) 2022-09-08T01:50:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:10 smithi055 ceph-mon[97033]: pgmap v8108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:50:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:10 smithi055 ceph-mon[97033]: osdmap e589: 12 total, 11 up, 11 in 2022-09-08T01:50:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:10 smithi059 ceph-mon[97450]: pgmap v8108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:50:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:10 smithi059 ceph-mon[97450]: osdmap e589: 12 total, 11 up, 11 in 2022-09-08T01:50:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:10.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:10.267502+0000) 2022-09-08T01:50:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:10.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:10.209918+0000) 2022-09-08T01:50:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:11.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:11.267735+0000) 2022-09-08T01:50:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:11.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:11.210135+0000) 2022-09-08T01:50:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:12 smithi055 ceph-mon[97033]: pgmap v8110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:50:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:12 smithi055 ceph-mon[97033]: osdmap e590: 12 total, 11 up, 11 in 2022-09-08T01:50:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:12 smithi059 ceph-mon[97450]: pgmap v8110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:50:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:12 smithi059 ceph-mon[97450]: osdmap e590: 12 total, 11 up, 11 in 2022-09-08T01:50:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:12.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:12.267896+0000) 2022-09-08T01:50:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:12.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:12.210334+0000) 2022-09-08T01:50:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:13.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:13.268095+0000) 2022-09-08T01:50:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:13.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:13.210521+0000) 2022-09-08T01:50:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:14 smithi055 ceph-mon[97033]: pgmap v8112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:50:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:14 smithi059 ceph-mon[97450]: pgmap v8112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T01:50:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:14.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:14.268317+0000) 2022-09-08T01:50:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:14.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:14.210739+0000) 2022-09-08T01:50:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:15.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:15.210945+0000) 2022-09-08T01:50:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:15.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:15.268506+0000) 2022-09-08T01:50:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:16 smithi055 ceph-mon[97033]: pgmap v8113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:50:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:16 smithi059 ceph-mon[97450]: pgmap v8113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:50:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:16.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:16.268755+0000) 2022-09-08T01:50:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:16.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:16.211103+0000) 2022-09-08T01:50:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:17.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:17.268917+0000) 2022-09-08T01:50:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:17.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:17.211382+0000) 2022-09-08T01:50:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:18 smithi055 ceph-mon[97033]: pgmap v8114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:50:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:18 smithi059 ceph-mon[97450]: pgmap v8114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:50:18.662 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:18.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:18.269140+0000) 2022-09-08T01:50:18.663 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:18.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:18.211585+0000) 2022-09-08T01:50:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:19.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:19.269381+0000) 2022-09-08T01:50:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:19.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:19.211794+0000) 2022-09-08T01:50:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:20 smithi055 ceph-mon[97033]: pgmap v8115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T01:50:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:20 smithi059 ceph-mon[97450]: pgmap v8115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T01:50:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:20.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:20.269619+0000) 2022-09-08T01:50:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:20.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:20.211995+0000) 2022-09-08T01:50:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:21.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:21.269889+0000) 2022-09-08T01:50:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:21.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:21.212225+0000) 2022-09-08T01:50:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:22 smithi055 ceph-mon[97033]: pgmap v8116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:50:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:22 smithi059 ceph-mon[97450]: pgmap v8116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:50:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:22.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:22.270137+0000) 2022-09-08T01:50:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:22.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:22.212457+0000) 2022-09-08T01:50:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:23.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:23.270404+0000) 2022-09-08T01:50:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:23.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:23.212725+0000) 2022-09-08T01:50:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:24 smithi055 ceph-mon[97033]: pgmap v8117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 95 B/s wr, 5 op/s 2022-09-08T01:50:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:24 smithi059 ceph-mon[97450]: pgmap v8117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 95 B/s wr, 5 op/s 2022-09-08T01:50:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:24.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:24.270687+0000) 2022-09-08T01:50:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:24.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:24.212884+0000) 2022-09-08T01:50:24.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:50:24.259+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:50:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:25.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:25.270907+0000) 2022-09-08T01:50:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:25.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:25.213063+0000) 2022-09-08T01:50:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:26 smithi055 ceph-mon[97033]: pgmap v8118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:50:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:26 smithi059 ceph-mon[97450]: pgmap v8118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:50:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:26.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:26.271096+0000) 2022-09-08T01:50:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:26.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:26.213288+0000) 2022-09-08T01:50:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:27.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:27.271327+0000) 2022-09-08T01:50:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:27.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:27.213466+0000) 2022-09-08T01:50:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:28 smithi055 ceph-mon[97033]: pgmap v8119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:50:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:28 smithi059 ceph-mon[97450]: pgmap v8119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:50:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:28.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:28.271462+0000) 2022-09-08T01:50:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:28.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:28.213669+0000) 2022-09-08T01:50:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:29.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:29.271646+0000) 2022-09-08T01:50:29.668 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:29.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:29.213946+0000) 2022-09-08T01:50:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:30 smithi055 ceph-mon[97033]: pgmap v8120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:30 smithi059 ceph-mon[97450]: pgmap v8120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:30.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:30.271866+0000) 2022-09-08T01:50:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:30.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:30.214171+0000) 2022-09-08T01:50:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:31.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:31.272083+0000) 2022-09-08T01:50:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:31.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:31.214434+0000) 2022-09-08T01:50:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:32 smithi055 ceph-mon[97033]: pgmap v8121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:50:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:32 smithi059 ceph-mon[97450]: pgmap v8121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:50:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:32.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:32.272300+0000) 2022-09-08T01:50:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:32.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:32.214605+0000) 2022-09-08T01:50:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:33.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:33.272493+0000) 2022-09-08T01:50:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:33.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:33.214834+0000) 2022-09-08T01:50:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:34 smithi055 ceph-mon[97033]: pgmap v8122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:34 smithi059 ceph-mon[97450]: pgmap v8122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:34.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:34.272739+0000) 2022-09-08T01:50:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:34.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:34.214995+0000) 2022-09-08T01:50:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:35.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:35.272985+0000) 2022-09-08T01:50:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:35.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:35.215166+0000) 2022-09-08T01:50:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:36 smithi055 ceph-mon[97033]: pgmap v8123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:36 smithi059 ceph-mon[97450]: pgmap v8123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:36.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:36.273200+0000) 2022-09-08T01:50:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:36.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:36.215382+0000) 2022-09-08T01:50:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:37.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:37.273469+0000) 2022-09-08T01:50:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:37.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:37.215600+0000) 2022-09-08T01:50:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:38 smithi055 ceph-mon[97033]: pgmap v8124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:50:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:38 smithi059 ceph-mon[97450]: pgmap v8124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:50:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:38.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:38.273679+0000) 2022-09-08T01:50:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:38.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:38.215919+0000) 2022-09-08T01:50:39.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:50:39.265+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:50:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:39.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:39.273829+0000) 2022-09-08T01:50:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:39.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:39.216136+0000) 2022-09-08T01:50:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:40 smithi055 ceph-mon[97033]: pgmap v8125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:40 smithi059 ceph-mon[97450]: pgmap v8125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:40.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:40.273992+0000) 2022-09-08T01:50:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:40.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:40.216357+0000) 2022-09-08T01:50:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:41.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:41.274194+0000) 2022-09-08T01:50:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:41.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:41.216606+0000) 2022-09-08T01:50:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:42 smithi055 ceph-mon[97033]: pgmap v8126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:50:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:42 smithi059 ceph-mon[97450]: pgmap v8126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:50:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:42.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:42.274428+0000) 2022-09-08T01:50:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:42.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:42.216852+0000) 2022-09-08T01:50:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:50:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:50:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:50:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:50:43.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:50:43.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:50:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:43.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:43.274555+0000) 2022-09-08T01:50:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:43.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:43.217030+0000) 2022-09-08T01:50:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:44 smithi055 ceph-mon[97033]: pgmap v8127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:44 smithi059 ceph-mon[97450]: pgmap v8127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:44.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:44.274795+0000) 2022-09-08T01:50:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:44.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:44.217213+0000) 2022-09-08T01:50:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:45.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:45.275002+0000) 2022-09-08T01:50:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:45.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:45.217451+0000) 2022-09-08T01:50:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:46 smithi055 ceph-mon[97033]: pgmap v8128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:46 smithi059 ceph-mon[97450]: pgmap v8128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:46.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:46.275224+0000) 2022-09-08T01:50:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:46.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:46.217628+0000) 2022-09-08T01:50:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:47.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:47.275429+0000) 2022-09-08T01:50:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:47.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:47.217849+0000) 2022-09-08T01:50:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:48 smithi055 ceph-mon[97033]: pgmap v8129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:50:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:48 smithi059 ceph-mon[97450]: pgmap v8129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:50:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:48.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:48.275660+0000) 2022-09-08T01:50:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:48.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:48.218033+0000) 2022-09-08T01:50:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:49.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:49.275827+0000) 2022-09-08T01:50:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:49.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:49.218284+0000) 2022-09-08T01:50:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:50 smithi055 ceph-mon[97033]: pgmap v8130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:50 smithi059 ceph-mon[97450]: pgmap v8130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:50.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:50.276089+0000) 2022-09-08T01:50:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:50.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:50.218505+0000) 2022-09-08T01:50:51.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:51.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:51.218742+0000) 2022-09-08T01:50:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:51.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:51.276302+0000) 2022-09-08T01:50:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:52 smithi059 ceph-mon[97450]: pgmap v8131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:50:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:52.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:52.276526+0000) 2022-09-08T01:50:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:52.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:52.218978+0000) 2022-09-08T01:50:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:52 smithi055 ceph-mon[97033]: pgmap v8131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:50:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:53.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:53.276772+0000) 2022-09-08T01:50:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:53.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:53.219167+0000) 2022-09-08T01:50:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:54 smithi059 ceph-mon[97450]: pgmap v8132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:54.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:50:54.280+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:50:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:54.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:54.276896+0000) 2022-09-08T01:50:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:54.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:54.219377+0000) 2022-09-08T01:50:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:54 smithi055 ceph-mon[97033]: pgmap v8132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:55.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:55.219667+0000) 2022-09-08T01:50:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:55.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:55.277074+0000) 2022-09-08T01:50:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:56 smithi059 ceph-mon[97450]: pgmap v8133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:56.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:56.277188+0000) 2022-09-08T01:50:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:56.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:56.219885+0000) 2022-09-08T01:50:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:56 smithi055 ceph-mon[97033]: pgmap v8133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:50:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:57.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:57.277410+0000) 2022-09-08T01:50:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:57.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:57.220130+0000) 2022-09-08T01:50:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:50:58 smithi059 ceph-mon[97450]: pgmap v8134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:50:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:58.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:58.277631+0000) 2022-09-08T01:50:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:58.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:58.220263+0000) 2022-09-08T01:50:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:50:58 smithi055 ceph-mon[97033]: pgmap v8134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:50:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:50:59.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:59.277925+0000) 2022-09-08T01:50:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:50:59.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:50:59.220478+0000) 2022-09-08T01:51:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:00 smithi059 ceph-mon[97450]: pgmap v8135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:51:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:00.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:00.278174+0000) 2022-09-08T01:51:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:00.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:00.220753+0000) 2022-09-08T01:51:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:00 smithi055 ceph-mon[97033]: pgmap v8135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:51:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:01.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:01.278420+0000) 2022-09-08T01:51:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:01.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:01.220974+0000) 2022-09-08T01:51:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:02 smithi059 ceph-mon[97450]: pgmap v8136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:51:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:02.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:02.278678+0000) 2022-09-08T01:51:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:02.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:02.221189+0000) 2022-09-08T01:51:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:02 smithi055 ceph-mon[97033]: pgmap v8136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:51:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:03.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:03.278841+0000) 2022-09-08T01:51:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:03.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:03.221445+0000) 2022-09-08T01:51:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:04 smithi059 ceph-mon[97450]: pgmap v8137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:51:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:04.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:04.278966+0000) 2022-09-08T01:51:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:04.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:04.221706+0000) 2022-09-08T01:51:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:04 smithi055 ceph-mon[97033]: pgmap v8137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:51:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:51:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:51:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:05.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:05.279124+0000) 2022-09-08T01:51:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:05.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:05.221936+0000) 2022-09-08T01:51:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:51:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:51:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:06 smithi059 ceph-mon[97450]: pgmap v8138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-08T01:51:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:06.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:06.279328+0000) 2022-09-08T01:51:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:06.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:06.222171+0000) 2022-09-08T01:51:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:06 smithi055 ceph-mon[97033]: pgmap v8138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-08T01:51:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:07.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:07.279512+0000) 2022-09-08T01:51:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:07.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:07.222344+0000) 2022-09-08T01:51:08.280 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:08.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:08.222523+0000) 2022-09-08T01:51:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:08 smithi059 ceph-mon[97450]: pgmap v8139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:51:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:08.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:08.279749+0000) 2022-09-08T01:51:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:08 smithi055 ceph-mon[97033]: pgmap v8139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:51:09.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:51:09.280+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:51:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:09.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:09.279878+0000) 2022-09-08T01:51:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:09.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:09.222768+0000) 2022-09-08T01:51:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:10 smithi059 ceph-mon[97450]: pgmap v8140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:51:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:10 smithi059 ceph-mon[97450]: osdmap e591: 12 total, 11 up, 11 in 2022-09-08T01:51:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:10.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:10.280031+0000) 2022-09-08T01:51:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:10.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:10.222997+0000) 2022-09-08T01:51:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:10 smithi055 ceph-mon[97033]: pgmap v8140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:51:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:10 smithi055 ceph-mon[97033]: osdmap e591: 12 total, 11 up, 11 in 2022-09-08T01:51:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:11.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:11.280280+0000) 2022-09-08T01:51:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:11.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:11.223222+0000) 2022-09-08T01:51:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:12 smithi059 ceph-mon[97450]: pgmap v8142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:51:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:12 smithi059 ceph-mon[97450]: osdmap e592: 12 total, 11 up, 11 in 2022-09-08T01:51:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:12.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:12.280456+0000) 2022-09-08T01:51:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:12.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:12.223461+0000) 2022-09-08T01:51:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:12 smithi055 ceph-mon[97033]: pgmap v8142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:51:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:12 smithi055 ceph-mon[97033]: osdmap e592: 12 total, 11 up, 11 in 2022-09-08T01:51:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:13.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:13.280709+0000) 2022-09-08T01:51:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:13.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:13.223686+0000) 2022-09-08T01:51:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:14 smithi055 ceph-mon[97033]: pgmap v8144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 33 KiB/s wr, 26 op/s 2022-09-08T01:51:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:14 smithi055 ceph-mon[97033]: pgmap v8145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T01:51:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:14 smithi055 ceph-mon[97033]: pgmap v8146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 240 B/s wr, 7 op/s 2022-09-08T01:51:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:51:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:14 smithi059 ceph-mon[97450]: pgmap v8144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 16 KiB/s rd, 33 KiB/s wr, 26 op/s 2022-09-08T01:51:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:14 smithi059 ceph-mon[97450]: pgmap v8145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T01:51:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:14 smithi059 ceph-mon[97450]: pgmap v8146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 240 B/s wr, 7 op/s 2022-09-08T01:51:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:51:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:14.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:14.280960+0000) 2022-09-08T01:51:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:14.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:14.223855+0000) 2022-09-08T01:51:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:15.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:15.281164+0000) 2022-09-08T01:51:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:15.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:15.224097+0000) 2022-09-08T01:51:16.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:16 smithi055 ceph-mon[97033]: pgmap v8147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 5.5 KiB/s rd, 352 B/s wr, 10 op/s 2022-09-08T01:51:16.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:16 smithi059 ceph-mon[97450]: pgmap v8147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 5.5 KiB/s rd, 352 B/s wr, 10 op/s 2022-09-08T01:51:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:16.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:16.281357+0000) 2022-09-08T01:51:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:16.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:16.224305+0000) 2022-09-08T01:51:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:17.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:17.281603+0000) 2022-09-08T01:51:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:17.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:17.224577+0000) 2022-09-08T01:51:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:18 smithi055 ceph-mon[97033]: pgmap v8148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 4 op/s 2022-09-08T01:51:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:18 smithi059 ceph-mon[97450]: pgmap v8148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 4 op/s 2022-09-08T01:51:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:18.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:18.281873+0000) 2022-09-08T01:51:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:18.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:18.224781+0000) 2022-09-08T01:51:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:19.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:19.282115+0000) 2022-09-08T01:51:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:19.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:19.224958+0000) 2022-09-08T01:51:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:20 smithi055 ceph-mon[97033]: pgmap v8149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.3 KiB/s rd, 147 B/s wr, 4 op/s 2022-09-08T01:51:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:20 smithi059 ceph-mon[97450]: pgmap v8149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.3 KiB/s rd, 147 B/s wr, 4 op/s 2022-09-08T01:51:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:20.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:20.282316+0000) 2022-09-08T01:51:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:20.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:20.225155+0000) 2022-09-08T01:51:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:21.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:21.282502+0000) 2022-09-08T01:51:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:21.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:21.225370+0000) 2022-09-08T01:51:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:22 smithi055 ceph-mon[97033]: pgmap v8150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T01:51:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:22 smithi059 ceph-mon[97450]: pgmap v8150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T01:51:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:22.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:22.282716+0000) 2022-09-08T01:51:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:22.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:22.225588+0000) 2022-09-08T01:51:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:23.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:23.282881+0000) 2022-09-08T01:51:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:23.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:23.225898+0000) 2022-09-08T01:51:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:24 smithi055 ceph-mon[97033]: pgmap v8151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:51:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:24 smithi059 ceph-mon[97450]: pgmap v8151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:51:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:24.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:24.283062+0000) 2022-09-08T01:51:24.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:51:24.296+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:51:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:24.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:24.226153+0000) 2022-09-08T01:51:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:25.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:25.226375+0000) 2022-09-08T01:51:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:25.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:25.283228+0000) 2022-09-08T01:51:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:26 smithi055 ceph-mon[97033]: pgmap v8152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:51:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:26 smithi059 ceph-mon[97450]: pgmap v8152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:51:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:26.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:26.283459+0000) 2022-09-08T01:51:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:26.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:26.226593+0000) 2022-09-08T01:51:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:27.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:27.283739+0000) 2022-09-08T01:51:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:27.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:27.226869+0000) 2022-09-08T01:51:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:28 smithi055 ceph-mon[97033]: pgmap v8153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:51:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:28 smithi059 ceph-mon[97450]: pgmap v8153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:51:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:28.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:28.283967+0000) 2022-09-08T01:51:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:28.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:28.227043+0000) 2022-09-08T01:51:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:29.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:29.284207+0000) 2022-09-08T01:51:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:29.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:29.227239+0000) 2022-09-08T01:51:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:30 smithi055 ceph-mon[97033]: pgmap v8154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:51:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:30 smithi059 ceph-mon[97450]: pgmap v8154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:51:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:30.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:30.284435+0000) 2022-09-08T01:51:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:30.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:30.227409+0000) 2022-09-08T01:51:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:31.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:31.284689+0000) 2022-09-08T01:51:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:31.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:31.227656+0000) 2022-09-08T01:51:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:32 smithi055 ceph-mon[97033]: pgmap v8155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:51:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:32 smithi059 ceph-mon[97450]: pgmap v8155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:51:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:32.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:32.284947+0000) 2022-09-08T01:51:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:32.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:32.227832+0000) 2022-09-08T01:51:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:33.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:33.285213+0000) 2022-09-08T01:51:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:33.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:33.228007+0000) 2022-09-08T01:51:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:34 smithi055 ceph-mon[97033]: pgmap v8156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T01:51:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:34 smithi059 ceph-mon[97450]: pgmap v8156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T01:51:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:34.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:34.285428+0000) 2022-09-08T01:51:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:34.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:34.228152+0000) 2022-09-08T01:51:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:35.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:35.285658+0000) 2022-09-08T01:51:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:35.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:35.228357+0000) 2022-09-08T01:51:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:36 smithi055 ceph-mon[97033]: pgmap v8157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:51:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:36 smithi059 ceph-mon[97450]: pgmap v8157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:51:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:36.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:36.228536+0000) 2022-09-08T01:51:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:36.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:36.285851+0000) 2022-09-08T01:51:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:37.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:37.286077+0000) 2022-09-08T01:51:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:37.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:37.228734+0000) 2022-09-08T01:51:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:38 smithi055 ceph-mon[97033]: pgmap v8158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:51:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:38 smithi059 ceph-mon[97450]: pgmap v8158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:51:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:38.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:38.286315+0000) 2022-09-08T01:51:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:38.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:38.228927+0000) 2022-09-08T01:51:39.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:51:39.311+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:51:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:39.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:39.286552+0000) 2022-09-08T01:51:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:39.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:39.229187+0000) 2022-09-08T01:51:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:40 smithi055 ceph-mon[97033]: pgmap v8159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:51:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:40 smithi059 ceph-mon[97450]: pgmap v8159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:51:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:40.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:40.286711+0000) 2022-09-08T01:51:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:40.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:40.229429+0000) 2022-09-08T01:51:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:41.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:41.229714+0000) 2022-09-08T01:51:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:41.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:41.286953+0000) 2022-09-08T01:51:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:42 smithi055 ceph-mon[97033]: pgmap v8160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:51:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:42 smithi059 ceph-mon[97450]: pgmap v8160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:51:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:42.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:42.287053+0000) 2022-09-08T01:51:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:42.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:42.229926+0000) 2022-09-08T01:51:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:43.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:43.230200+0000) 2022-09-08T01:51:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:43.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:43.287266+0000) 2022-09-08T01:51:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:44 smithi055 ceph-mon[97033]: pgmap v8161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:51:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:44 smithi059 ceph-mon[97450]: pgmap v8161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:51:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:44.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:44.287476+0000) 2022-09-08T01:51:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:44.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:44.230414+0000) 2022-09-08T01:51:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:45.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:45.287765+0000) 2022-09-08T01:51:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:45.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:45.230617+0000) 2022-09-08T01:51:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:46 smithi055 ceph-mon[97033]: pgmap v8162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:51:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:46 smithi059 ceph-mon[97450]: pgmap v8162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:51:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:46.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:46.287911+0000) 2022-09-08T01:51:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:46.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:46.230853+0000) 2022-09-08T01:51:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:47.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:47.231057+0000) 2022-09-08T01:51:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:47.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:47.288111+0000) 2022-09-08T01:51:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:48 smithi055 ceph-mon[97033]: pgmap v8163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:51:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:48 smithi059 ceph-mon[97450]: pgmap v8163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:51:48.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:48.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:48.231223+0000) 2022-09-08T01:51:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:48.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:48.288296+0000) 2022-09-08T01:51:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:49.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:49.288577+0000) 2022-09-08T01:51:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:49.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:49.231451+0000) 2022-09-08T01:51:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:50 smithi055 ceph-mon[97033]: pgmap v8164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:51:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:50 smithi059 ceph-mon[97450]: pgmap v8164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:51:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:50.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:50.288823+0000) 2022-09-08T01:51:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:50.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:50.231670+0000) 2022-09-08T01:51:51.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:51.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:51.231941+0000) 2022-09-08T01:51:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:51.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:51.288989+0000) 2022-09-08T01:51:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:52 smithi055 ceph-mon[97033]: pgmap v8165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:51:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:52 smithi059 ceph-mon[97450]: pgmap v8165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:51:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:52.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:52.232154+0000) 2022-09-08T01:51:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:52.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:52.289128+0000) 2022-09-08T01:51:53.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:53.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:53.232375+0000) 2022-09-08T01:51:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:53.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:53.289389+0000) 2022-09-08T01:51:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:54 smithi055 ceph-mon[97033]: pgmap v8166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:51:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:54 smithi059 ceph-mon[97450]: pgmap v8166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:51:54.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:51:54.326+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:51:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:54.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:54.289683+0000) 2022-09-08T01:51:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:54.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:54.232538+0000) 2022-09-08T01:51:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:55.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:55.289973+0000) 2022-09-08T01:51:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:55.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:55.232762+0000) 2022-09-08T01:51:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:56 smithi055 ceph-mon[97033]: pgmap v8167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:51:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:56 smithi059 ceph-mon[97450]: pgmap v8167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:51:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:56.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:56.232994+0000) 2022-09-08T01:51:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:56.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:56.290181+0000) 2022-09-08T01:51:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:57.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:57.290394+0000) 2022-09-08T01:51:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:57.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:57.233227+0000) 2022-09-08T01:51:58.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:51:58 smithi055 ceph-mon[97033]: pgmap v8168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:51:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:51:58 smithi059 ceph-mon[97450]: pgmap v8168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:51:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:58.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:58.290609+0000) 2022-09-08T01:51:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:58.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:58.233453+0000) 2022-09-08T01:51:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:51:59.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:59.290833+0000) 2022-09-08T01:51:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:51:59.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:51:59.233724+0000) 2022-09-08T01:52:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:00 smithi055 ceph-mon[97033]: pgmap v8169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:00 smithi059 ceph-mon[97450]: pgmap v8169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:00.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:00.291052+0000) 2022-09-08T01:52:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:00.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:00.233916+0000) 2022-09-08T01:52:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:01.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:01.291215+0000) 2022-09-08T01:52:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:01.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:01.234048+0000) 2022-09-08T01:52:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:02 smithi055 ceph-mon[97033]: pgmap v8170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:52:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:02 smithi059 ceph-mon[97450]: pgmap v8170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:52:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:02.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:02.291450+0000) 2022-09-08T01:52:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:02.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:02.234251+0000) 2022-09-08T01:52:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:03.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:03.291712+0000) 2022-09-08T01:52:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:03.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:03.234384+0000) 2022-09-08T01:52:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:04 smithi055 ceph-mon[97033]: pgmap v8171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:04 smithi059 ceph-mon[97450]: pgmap v8171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:04.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:04.291957+0000) 2022-09-08T01:52:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:04.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:04.234613+0000) 2022-09-08T01:52:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:52:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:52:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:05.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:05.292202+0000) 2022-09-08T01:52:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:05.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:05.234829+0000) 2022-09-08T01:52:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:06 smithi055 ceph-mon[97033]: pgmap v8172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:52:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:52:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:06 smithi059 ceph-mon[97450]: pgmap v8172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:52:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:52:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:06.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:06.235018+0000) 2022-09-08T01:52:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:06.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:06.292406+0000) 2022-09-08T01:52:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:07.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:07.235196+0000) 2022-09-08T01:52:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:07.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:07.292562+0000) 2022-09-08T01:52:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:08 smithi055 ceph-mon[97033]: pgmap v8173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:52:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:08 smithi059 ceph-mon[97450]: pgmap v8173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:52:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:08.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:08.235367+0000) 2022-09-08T01:52:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:08.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:08.292720+0000) 2022-09-08T01:52:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:09.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:09.292986+0000) 2022-09-08T01:52:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:09.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:09.235571+0000) 2022-09-08T01:52:09.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:52:09.341+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:52:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:10 smithi059 ceph-mon[97450]: pgmap v8174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:52:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:10 smithi059 ceph-mon[97450]: osdmap e593: 12 total, 11 up, 11 in 2022-09-08T01:52:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:10.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:10.293267+0000) 2022-09-08T01:52:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:10.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:10.235797+0000) 2022-09-08T01:52:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:10 smithi055 ceph-mon[97033]: pgmap v8174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:52:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:10 smithi055 ceph-mon[97033]: osdmap e593: 12 total, 11 up, 11 in 2022-09-08T01:52:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:11.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:11.293484+0000) 2022-09-08T01:52:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:11.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:11.235953+0000) 2022-09-08T01:52:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:12 smithi059 ceph-mon[97450]: pgmap v8176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:52:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:12 smithi059 ceph-mon[97450]: osdmap e594: 12 total, 11 up, 11 in 2022-09-08T01:52:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:12.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:12.293703+0000) 2022-09-08T01:52:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:12.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:12.236183+0000) 2022-09-08T01:52:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:12 smithi055 ceph-mon[97033]: pgmap v8176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:52:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:12 smithi055 ceph-mon[97033]: osdmap e594: 12 total, 11 up, 11 in 2022-09-08T01:52:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:52:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:52:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:52:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:13.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:13.293839+0000) 2022-09-08T01:52:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:13.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:13.236335+0000) 2022-09-08T01:52:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:52:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:52:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:52:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:14 smithi059 ceph-mon[97450]: pgmap v8178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:52:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:14.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:14.294073+0000) 2022-09-08T01:52:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:14.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:14.236554+0000) 2022-09-08T01:52:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:14 smithi055 ceph-mon[97033]: pgmap v8178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:52:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:15.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:15.294338+0000) 2022-09-08T01:52:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:15.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:15.236819+0000) 2022-09-08T01:52:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:16 smithi059 ceph-mon[97450]: pgmap v8179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:52:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:16.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:16.294529+0000) 2022-09-08T01:52:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:16.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:16.237070+0000) 2022-09-08T01:52:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:16 smithi055 ceph-mon[97033]: pgmap v8179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:52:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:17.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:17.237357+0000) 2022-09-08T01:52:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:17.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:17.294735+0000) 2022-09-08T01:52:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:18 smithi059 ceph-mon[97450]: pgmap v8180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:52:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:18.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:18.294952+0000) 2022-09-08T01:52:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:18.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:18.237620+0000) 2022-09-08T01:52:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:18 smithi055 ceph-mon[97033]: pgmap v8180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:52:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:19.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:19.295178+0000) 2022-09-08T01:52:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:19.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:19.237919+0000) 2022-09-08T01:52:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:20 smithi059 ceph-mon[97450]: pgmap v8181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T01:52:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:20.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:20.295414+0000) 2022-09-08T01:52:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:20.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:20.238092+0000) 2022-09-08T01:52:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:20 smithi055 ceph-mon[97033]: pgmap v8181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T01:52:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:21.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:21.295662+0000) 2022-09-08T01:52:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:21.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:21.238230+0000) 2022-09-08T01:52:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:22 smithi059 ceph-mon[97450]: pgmap v8182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:52:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:22.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:22.295882+0000) 2022-09-08T01:52:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:22.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:22.238457+0000) 2022-09-08T01:52:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:22 smithi055 ceph-mon[97033]: pgmap v8182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:52:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:23.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:23.296084+0000) 2022-09-08T01:52:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:23.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:23.238719+0000) 2022-09-08T01:52:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:24 smithi059 ceph-mon[97450]: pgmap v8183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T01:52:24.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:52:24.344+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:52:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:24.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:24.296333+0000) 2022-09-08T01:52:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:24.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:24.238948+0000) 2022-09-08T01:52:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:24 smithi055 ceph-mon[97033]: pgmap v8183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T01:52:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:25.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:25.296552+0000) 2022-09-08T01:52:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:25.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:25.239200+0000) 2022-09-08T01:52:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:26 smithi059 ceph-mon[97450]: pgmap v8184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:52:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:26.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:26.296764+0000) 2022-09-08T01:52:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:26.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:26.239436+0000) 2022-09-08T01:52:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:26 smithi055 ceph-mon[97033]: pgmap v8184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:52:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:27.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:27.296980+0000) 2022-09-08T01:52:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:27.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:27.239601+0000) 2022-09-08T01:52:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:28 smithi059 ceph-mon[97450]: pgmap v8185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:28.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:28.297134+0000) 2022-09-08T01:52:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:28.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:28.239812+0000) 2022-09-08T01:52:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:28 smithi055 ceph-mon[97033]: pgmap v8185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:29.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:29.297288+0000) 2022-09-08T01:52:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:29.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:29.240058+0000) 2022-09-08T01:52:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:30 smithi059 ceph-mon[97450]: pgmap v8186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:30.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:30.240322+0000) 2022-09-08T01:52:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:30.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:30.297517+0000) 2022-09-08T01:52:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:30 smithi055 ceph-mon[97033]: pgmap v8186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:31.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:31.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:31.240517+0000) 2022-09-08T01:52:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:31.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:31.297753+0000) 2022-09-08T01:52:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:32 smithi059 ceph-mon[97450]: pgmap v8187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:52:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:32.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:32.297986+0000) 2022-09-08T01:52:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:32.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:32.240730+0000) 2022-09-08T01:52:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:32 smithi055 ceph-mon[97033]: pgmap v8187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:52:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:33.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:33.298213+0000) 2022-09-08T01:52:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:33.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:33.240970+0000) 2022-09-08T01:52:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:34 smithi059 ceph-mon[97450]: pgmap v8188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:34.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:34.241181+0000) 2022-09-08T01:52:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:34.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:34.298473+0000) 2022-09-08T01:52:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:34 smithi055 ceph-mon[97033]: pgmap v8188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:35.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:35.298763+0000) 2022-09-08T01:52:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:35.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:35.241443+0000) 2022-09-08T01:52:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:36.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:36.299023+0000) 2022-09-08T01:52:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:36.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:36.241691+0000) 2022-09-08T01:52:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:36 smithi055 ceph-mon[97033]: pgmap v8189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:52:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:36 smithi059 ceph-mon[97450]: pgmap v8189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:52:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:37.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:37.299223+0000) 2022-09-08T01:52:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:37.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:37.241933+0000) 2022-09-08T01:52:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:38.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:38.299437+0000) 2022-09-08T01:52:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:38.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:38.242112+0000) 2022-09-08T01:52:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:38 smithi055 ceph-mon[97033]: pgmap v8190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:38 smithi059 ceph-mon[97450]: pgmap v8190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:39.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:52:39.353+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:52:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:39.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:39.299705+0000) 2022-09-08T01:52:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:39.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:39.242366+0000) 2022-09-08T01:52:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:40.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:40.299916+0000) 2022-09-08T01:52:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:40.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:40.242603+0000) 2022-09-08T01:52:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:40 smithi055 ceph-mon[97033]: pgmap v8191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:40 smithi059 ceph-mon[97450]: pgmap v8191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:41.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:41.300161+0000) 2022-09-08T01:52:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:41.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:41.242829+0000) 2022-09-08T01:52:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:42.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:42.300395+0000) 2022-09-08T01:52:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:42.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:42.243002+0000) 2022-09-08T01:52:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:42 smithi055 ceph-mon[97033]: pgmap v8192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:52:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:42 smithi059 ceph-mon[97450]: pgmap v8192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:52:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:43.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:43.300573+0000) 2022-09-08T01:52:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:43.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:43.243214+0000) 2022-09-08T01:52:44.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:44.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:44.243455+0000) 2022-09-08T01:52:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:44.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:44.300752+0000) 2022-09-08T01:52:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:44 smithi055 ceph-mon[97033]: pgmap v8193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:44 smithi055 ceph-mon[97033]: pgmap v8194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:52:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:44 smithi055 ceph-mon[97033]: pgmap v8195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:52:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:52:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:44 smithi059 ceph-mon[97450]: pgmap v8193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:44 smithi059 ceph-mon[97450]: pgmap v8194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:52:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:44 smithi059 ceph-mon[97450]: pgmap v8195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:52:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:52:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:45.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:45.243694+0000) 2022-09-08T01:52:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:45.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:45.300966+0000) 2022-09-08T01:52:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:46.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:46.301136+0000) 2022-09-08T01:52:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:46.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:46.243919+0000) 2022-09-08T01:52:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:46 smithi055 ceph-mon[97033]: pgmap v8196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:52:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:46 smithi059 ceph-mon[97450]: pgmap v8196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:52:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:47.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:47.301334+0000) 2022-09-08T01:52:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:47.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:47.244113+0000) 2022-09-08T01:52:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:48 smithi055 ceph-mon[97033]: pgmap v8197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:52:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:48 smithi059 ceph-mon[97450]: pgmap v8197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:52:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:48.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:48.301456+0000) 2022-09-08T01:52:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:48.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:48.244298+0000) 2022-09-08T01:52:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:49.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:49.244500+0000) 2022-09-08T01:52:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:49.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:49.301710+0000) 2022-09-08T01:52:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:50.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:50.244729+0000) 2022-09-08T01:52:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:50.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:50.301860+0000) 2022-09-08T01:52:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:50 smithi055 ceph-mon[97033]: pgmap v8198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:52:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:50 smithi059 ceph-mon[97450]: pgmap v8198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:52:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:51.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:51.244971+0000) 2022-09-08T01:52:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:51.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:51.302065+0000) 2022-09-08T01:52:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:52.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:52.245186+0000) 2022-09-08T01:52:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:52.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:52.302234+0000) 2022-09-08T01:52:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:52 smithi055 ceph-mon[97033]: pgmap v8199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:52:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:52 smithi059 ceph-mon[97450]: pgmap v8199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:52:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:53.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:53.302438+0000) 2022-09-08T01:52:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:53.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:53.245432+0000) 2022-09-08T01:52:54.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:52:54.367+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:52:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:54.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:54.302701+0000) 2022-09-08T01:52:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:54.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:54.245598+0000) 2022-09-08T01:52:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:54 smithi055 ceph-mon[97033]: pgmap v8200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:52:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:54 smithi059 ceph-mon[97450]: pgmap v8200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:52:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:55.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:55.302943+0000) 2022-09-08T01:52:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:55.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:55.245829+0000) 2022-09-08T01:52:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:56.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:56.303180+0000) 2022-09-08T01:52:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:56.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:56.246074+0000) 2022-09-08T01:52:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:56 smithi055 ceph-mon[97033]: pgmap v8201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:52:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:56 smithi059 ceph-mon[97450]: pgmap v8201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:52:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:57.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:57.303382+0000) 2022-09-08T01:52:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:57.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:57.246328+0000) 2022-09-08T01:52:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:52:58 smithi055 ceph-mon[97033]: pgmap v8202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:52:58 smithi059 ceph-mon[97450]: pgmap v8202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:52:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:58.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:58.303582+0000) 2022-09-08T01:52:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:58.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:58.246555+0000) 2022-09-08T01:52:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:52:59.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:59.303774+0000) 2022-09-08T01:52:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:52:59.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:52:59.246759+0000) 2022-09-08T01:53:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:00.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:00.303944+0000) 2022-09-08T01:53:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:00.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:00.246993+0000) 2022-09-08T01:53:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:00 smithi055 ceph-mon[97033]: pgmap v8203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:00 smithi059 ceph-mon[97450]: pgmap v8203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:01.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:01.304197+0000) 2022-09-08T01:53:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:01.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:01.247174+0000) 2022-09-08T01:53:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:02.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:02.304404+0000) 2022-09-08T01:53:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:02.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:02.247390+0000) 2022-09-08T01:53:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:02 smithi055 ceph-mon[97033]: pgmap v8204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:53:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:02 smithi059 ceph-mon[97450]: pgmap v8204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:53:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:03.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:03.304593+0000) 2022-09-08T01:53:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:03.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:03.247612+0000) 2022-09-08T01:53:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:04.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:04.304758+0000) 2022-09-08T01:53:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:04.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:04.247846+0000) 2022-09-08T01:53:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:04 smithi055 ceph-mon[97033]: pgmap v8205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:04 smithi059 ceph-mon[97450]: pgmap v8205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:05.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:05.304982+0000) 2022-09-08T01:53:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:05.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:05.247979+0000) 2022-09-08T01:53:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:53:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:53:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:53:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:53:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:06.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:06.305157+0000) 2022-09-08T01:53:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:06.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:06.248148+0000) 2022-09-08T01:53:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:06 smithi055 ceph-mon[97033]: pgmap v8206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:53:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:06 smithi059 ceph-mon[97450]: pgmap v8206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:53:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:07.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:07.305354+0000) 2022-09-08T01:53:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:07.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:07.248367+0000) 2022-09-08T01:53:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:08 smithi055 ceph-mon[97033]: pgmap v8207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:53:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:08 smithi059 ceph-mon[97450]: pgmap v8207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:53:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:08.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:08.305497+0000) 2022-09-08T01:53:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:08.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:08.248675+0000) 2022-09-08T01:53:09.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:53:09.382+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:53:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:09.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:09.305646+0000) 2022-09-08T01:53:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:09.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:09.248827+0000) 2022-09-08T01:53:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:10 smithi055 ceph-mon[97033]: osdmap e595: 12 total, 11 up, 11 in 2022-09-08T01:53:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:10 smithi055 ceph-mon[97033]: pgmap v8209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:53:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:10 smithi059 ceph-mon[97450]: osdmap e595: 12 total, 11 up, 11 in 2022-09-08T01:53:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:10 smithi059 ceph-mon[97450]: pgmap v8209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:53:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:10.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:10.249033+0000) 2022-09-08T01:53:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:10.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:10.305843+0000) 2022-09-08T01:53:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:11.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:11.249274+0000) 2022-09-08T01:53:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:11.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:11.306099+0000) 2022-09-08T01:53:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:12.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:12.306333+0000) 2022-09-08T01:53:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:12.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:12.249489+0000) 2022-09-08T01:53:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:12 smithi055 ceph-mon[97033]: pgmap v8210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:53:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:12 smithi059 ceph-mon[97450]: pgmap v8210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:53:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:13.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:13.306600+0000) 2022-09-08T01:53:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:13.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:13.249740+0000) 2022-09-08T01:53:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:14.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:14.306910+0000) 2022-09-08T01:53:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:14.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:14.249976+0000) 2022-09-08T01:53:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:14 smithi055 ceph-mon[97033]: pgmap v8211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:53:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:14 smithi059 ceph-mon[97450]: pgmap v8211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:53:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:15.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:15.307140+0000) 2022-09-08T01:53:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:15.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:15.250159+0000) 2022-09-08T01:53:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:16.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:16.307297+0000) 2022-09-08T01:53:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:16.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:16.250395+0000) 2022-09-08T01:53:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:16 smithi055 ceph-mon[97033]: pgmap v8212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:53:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:16 smithi059 ceph-mon[97450]: pgmap v8212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:53:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:17.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:17.307521+0000) 2022-09-08T01:53:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:17.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:17.250611+0000) 2022-09-08T01:53:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:18 smithi055 ceph-mon[97033]: osdmap e596: 12 total, 11 up, 11 in 2022-09-08T01:53:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:18 smithi055 ceph-mon[97033]: pgmap v8214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 249 B/s wr, 7 op/s 2022-09-08T01:53:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:18 smithi059 ceph-mon[97450]: osdmap e596: 12 total, 11 up, 11 in 2022-09-08T01:53:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:18 smithi059 ceph-mon[97450]: pgmap v8214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 249 B/s wr, 7 op/s 2022-09-08T01:53:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:18.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:18.307741+0000) 2022-09-08T01:53:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:18.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:18.250844+0000) 2022-09-08T01:53:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:19.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:19.307985+0000) 2022-09-08T01:53:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:19.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:19.251053+0000) 2022-09-08T01:53:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:20.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:20.308223+0000) 2022-09-08T01:53:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:20.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:20.251197+0000) 2022-09-08T01:53:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:20 smithi055 ceph-mon[97033]: pgmap v8215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:53:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:20 smithi059 ceph-mon[97450]: pgmap v8215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:53:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:21.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:21.308477+0000) 2022-09-08T01:53:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:21.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:21.251398+0000) 2022-09-08T01:53:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:22.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:22.308756+0000) 2022-09-08T01:53:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:22.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:22.251605+0000) 2022-09-08T01:53:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:22 smithi055 ceph-mon[97033]: pgmap v8216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:53:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:22 smithi059 ceph-mon[97450]: pgmap v8216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:53:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:23.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:23.308995+0000) 2022-09-08T01:53:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:23.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:23.251737+0000) 2022-09-08T01:53:24.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:53:24.398+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:53:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:24.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:24.309229+0000) 2022-09-08T01:53:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:24.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:24.251937+0000) 2022-09-08T01:53:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:24 smithi055 ceph-mon[97033]: pgmap v8217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:53:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:24 smithi059 ceph-mon[97450]: pgmap v8217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:53:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:25.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:25.309411+0000) 2022-09-08T01:53:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:25.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:25.252204+0000) 2022-09-08T01:53:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:26.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:26.309665+0000) 2022-09-08T01:53:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:26.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:26.252435+0000) 2022-09-08T01:53:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:26 smithi055 ceph-mon[97033]: pgmap v8218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:53:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:26 smithi059 ceph-mon[97450]: pgmap v8218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:53:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:27.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:27.309846+0000) 2022-09-08T01:53:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:27.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:27.252711+0000) 2022-09-08T01:53:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:28 smithi055 ceph-mon[97033]: pgmap v8219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:53:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:28 smithi059 ceph-mon[97450]: pgmap v8219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:53:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:28.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:28.310060+0000) 2022-09-08T01:53:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:28.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:28.252874+0000) 2022-09-08T01:53:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:29.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:29.310282+0000) 2022-09-08T01:53:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:29.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:29.253097+0000) 2022-09-08T01:53:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:30.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:30.310530+0000) 2022-09-08T01:53:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:30.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:30.253338+0000) 2022-09-08T01:53:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:30 smithi055 ceph-mon[97033]: pgmap v8220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:30 smithi059 ceph-mon[97450]: pgmap v8220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:31.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:31.310782+0000) 2022-09-08T01:53:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:31.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:31.253585+0000) 2022-09-08T01:53:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:32.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:32.310938+0000) 2022-09-08T01:53:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:32.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:32.253857+0000) 2022-09-08T01:53:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:32 smithi055 ceph-mon[97033]: pgmap v8221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:53:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:32 smithi059 ceph-mon[97450]: pgmap v8221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:53:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:33.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:33.311105+0000) 2022-09-08T01:53:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:33.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:33.254051+0000) 2022-09-08T01:53:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:34.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:34.311340+0000) 2022-09-08T01:53:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:34.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:34.254311+0000) 2022-09-08T01:53:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:34 smithi055 ceph-mon[97033]: pgmap v8222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:34 smithi059 ceph-mon[97450]: pgmap v8222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:35.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:35.311567+0000) 2022-09-08T01:53:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:35.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:35.254585+0000) 2022-09-08T01:53:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:36.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:36.311792+0000) 2022-09-08T01:53:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:36.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:36.254816+0000) 2022-09-08T01:53:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:36 smithi055 ceph-mon[97033]: pgmap v8223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:53:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:36 smithi059 ceph-mon[97450]: pgmap v8223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:53:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:37.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:37.311999+0000) 2022-09-08T01:53:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:37.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:37.255088+0000) 2022-09-08T01:53:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:38 smithi055 ceph-mon[97033]: pgmap v8224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:38 smithi059 ceph-mon[97450]: pgmap v8224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:38.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:38.312209+0000) 2022-09-08T01:53:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:38.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:38.255291+0000) 2022-09-08T01:53:39.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:53:39.412+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:53:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:39.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:39.312451+0000) 2022-09-08T01:53:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:39.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:39.255485+0000) 2022-09-08T01:53:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:40.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:40.312668+0000) 2022-09-08T01:53:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:40.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:40.255732+0000) 2022-09-08T01:53:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:40 smithi055 ceph-mon[97033]: pgmap v8225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:40 smithi059 ceph-mon[97450]: pgmap v8225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:41.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:41.312911+0000) 2022-09-08T01:53:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:41.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:41.255888+0000) 2022-09-08T01:53:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:42.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:42.313075+0000) 2022-09-08T01:53:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:42.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:42.256020+0000) 2022-09-08T01:53:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:42 smithi055 ceph-mon[97033]: pgmap v8226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:53:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:42 smithi059 ceph-mon[97450]: pgmap v8226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:53:43.515 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:43.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:43.313298+0000) 2022-09-08T01:53:43.515 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:43.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:43.256140+0000) 2022-09-08T01:53:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:44.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:44.313510+0000) 2022-09-08T01:53:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:44.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:44.256359+0000) 2022-09-08T01:53:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:44 smithi055 ceph-mon[97033]: pgmap v8227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:53:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:53:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:53:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:44 smithi059 ceph-mon[97450]: pgmap v8227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:53:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:53:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:53:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:45.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:45.313690+0000) 2022-09-08T01:53:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:45.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:45.256530+0000) 2022-09-08T01:53:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:46.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:46.313893+0000) 2022-09-08T01:53:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:46.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:46.256734+0000) 2022-09-08T01:53:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:46 smithi055 ceph-mon[97033]: pgmap v8228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:53:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:46 smithi059 ceph-mon[97450]: pgmap v8228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:53:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:47.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:47.314105+0000) 2022-09-08T01:53:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:47.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:47.256856+0000) 2022-09-08T01:53:48.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:48 smithi055 ceph-mon[97033]: pgmap v8229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:48 smithi059 ceph-mon[97450]: pgmap v8229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:48.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:48.314263+0000) 2022-09-08T01:53:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:48.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:48.257031+0000) 2022-09-08T01:53:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:49.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:49.314470+0000) 2022-09-08T01:53:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:49.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:49.257260+0000) 2022-09-08T01:53:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:50.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:50.314743+0000) 2022-09-08T01:53:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:50.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:50.257439+0000) 2022-09-08T01:53:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:50 smithi055 ceph-mon[97033]: pgmap v8230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:50 smithi059 ceph-mon[97450]: pgmap v8230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:51.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:51.314950+0000) 2022-09-08T01:53:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:51.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:51.257618+0000) 2022-09-08T01:53:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:52.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:52.315103+0000) 2022-09-08T01:53:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:52.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:52.257816+0000) 2022-09-08T01:53:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:52 smithi055 ceph-mon[97033]: pgmap v8231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:53:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:52 smithi059 ceph-mon[97450]: pgmap v8231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:53:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:53.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:53.315320+0000) 2022-09-08T01:53:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:53.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:53.258000+0000) 2022-09-08T01:53:54.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:53:54.426+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:53:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:54.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:54.315537+0000) 2022-09-08T01:53:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:54.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:54.258210+0000) 2022-09-08T01:53:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:54 smithi055 ceph-mon[97033]: pgmap v8232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:54 smithi059 ceph-mon[97450]: pgmap v8232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:53:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:55.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:55.315760+0000) 2022-09-08T01:53:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:55.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:55.258422+0000) 2022-09-08T01:53:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:56.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:56.315954+0000) 2022-09-08T01:53:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:56.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:56.258623+0000) 2022-09-08T01:53:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:56 smithi055 ceph-mon[97033]: pgmap v8233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:53:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:56 smithi059 ceph-mon[97450]: pgmap v8233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:53:57.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:57.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:57.316175+0000) 2022-09-08T01:53:57.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:57.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:57.258851+0000) 2022-09-08T01:53:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:53:58 smithi055 ceph-mon[97033]: pgmap v8234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:53:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:53:58 smithi059 ceph-mon[97450]: pgmap v8234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:53:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:58.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:58.316359+0000) 2022-09-08T01:53:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:58.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:58.259060+0000) 2022-09-08T01:53:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:53:59.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:59.316617+0000) 2022-09-08T01:53:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:53:59.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:53:59.259305+0000) 2022-09-08T01:54:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:00.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:00.316859+0000) 2022-09-08T01:54:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:00.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:00.259551+0000) 2022-09-08T01:54:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:00 smithi055 ceph-mon[97033]: pgmap v8235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:54:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:00 smithi059 ceph-mon[97450]: pgmap v8235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:54:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:01.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:01.317141+0000) 2022-09-08T01:54:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:01.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:01.259787+0000) 2022-09-08T01:54:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:02.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:02.317338+0000) 2022-09-08T01:54:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:02.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:02.260019+0000) 2022-09-08T01:54:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:02 smithi055 ceph-mon[97033]: pgmap v8236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:54:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:02 smithi059 ceph-mon[97450]: pgmap v8236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:54:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:03.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:03.317500+0000) 2022-09-08T01:54:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:03.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:03.260257+0000) 2022-09-08T01:54:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:04.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:04.260518+0000) 2022-09-08T01:54:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:04.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:04.317750+0000) 2022-09-08T01:54:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:04 smithi055 ceph-mon[97033]: pgmap v8237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:54:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:04 smithi059 ceph-mon[97450]: pgmap v8237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:54:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:05.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:05.317924+0000) 2022-09-08T01:54:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:05.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:05.260756+0000) 2022-09-08T01:54:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:54:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:54:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:54:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:54:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:06.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:06.318092+0000) 2022-09-08T01:54:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:06.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:06.260978+0000) 2022-09-08T01:54:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:06 smithi055 ceph-mon[97033]: pgmap v8238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:54:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:06 smithi059 ceph-mon[97450]: pgmap v8238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:54:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:07.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:07.318259+0000) 2022-09-08T01:54:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:07.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:07.261258+0000) 2022-09-08T01:54:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:08 smithi055 ceph-mon[97033]: pgmap v8239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:54:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:08 smithi059 ceph-mon[97450]: pgmap v8239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:54:08.581 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:08.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:08.318481+0000) 2022-09-08T01:54:08.582 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:08.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:08.261434+0000) 2022-09-08T01:54:09.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:54:09.438+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:54:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:09.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:09.318694+0000) 2022-09-08T01:54:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:09.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:09.261622+0000) 2022-09-08T01:54:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:10 smithi055 ceph-mon[97033]: osdmap e597: 12 total, 11 up, 11 in 2022-09-08T01:54:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:10 smithi055 ceph-mon[97033]: pgmap v8241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:54:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:10 smithi059 ceph-mon[97450]: osdmap e597: 12 total, 11 up, 11 in 2022-09-08T01:54:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:10 smithi059 ceph-mon[97450]: pgmap v8241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:54:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:10.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:10.318838+0000) 2022-09-08T01:54:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:10.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:10.261850+0000) 2022-09-08T01:54:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:11.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:11.319004+0000) 2022-09-08T01:54:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:11.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:11.262113+0000) 2022-09-08T01:54:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:12.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:12.319162+0000) 2022-09-08T01:54:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:12.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:12.262364+0000) 2022-09-08T01:54:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:12 smithi055 ceph-mon[97033]: pgmap v8242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:54:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:12 smithi055 ceph-mon[97033]: osdmap e598: 12 total, 11 up, 11 in 2022-09-08T01:54:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:12 smithi059 ceph-mon[97450]: pgmap v8242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:54:12.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:12 smithi059 ceph-mon[97450]: osdmap e598: 12 total, 11 up, 11 in 2022-09-08T01:54:13.634 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:13.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:13.319359+0000) 2022-09-08T01:54:13.635 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:13.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:13.262559+0000) 2022-09-08T01:54:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:14.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:14.319578+0000) 2022-09-08T01:54:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:14.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:14.262782+0000) 2022-09-08T01:54:14.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:14 smithi055 ceph-mon[97033]: pgmap v8244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:54:14.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:54:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:14 smithi059 ceph-mon[97450]: pgmap v8244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:54:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:54:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:15.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:15.319804+0000) 2022-09-08T01:54:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:15.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:15.263066+0000) 2022-09-08T01:54:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:15 smithi055 ceph-mon[97033]: pgmap v8245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T01:54:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:15 smithi055 ceph-mon[97033]: pgmap v8246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:54:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:15 smithi059 ceph-mon[97450]: pgmap v8245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T01:54:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:15 smithi059 ceph-mon[97450]: pgmap v8246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T01:54:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:16.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:16.319929+0000) 2022-09-08T01:54:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:16.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:16.263290+0000) 2022-09-08T01:54:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:17.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:17.320087+0000) 2022-09-08T01:54:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:17.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:17.263555+0000) 2022-09-08T01:54:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:17 smithi055 ceph-mon[97033]: pgmap v8247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 5.1 KiB/s rd, 328 B/s wr, 10 op/s 2022-09-08T01:54:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:17 smithi059 ceph-mon[97450]: pgmap v8247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 5.1 KiB/s rd, 328 B/s wr, 10 op/s 2022-09-08T01:54:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:18.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:18.320228+0000) 2022-09-08T01:54:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:18.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:18.263811+0000) 2022-09-08T01:54:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:19.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:19.320451+0000) 2022-09-08T01:54:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:19.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:19.264049+0000) 2022-09-08T01:54:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:19 smithi055 ceph-mon[97033]: pgmap v8248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T01:54:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:19 smithi059 ceph-mon[97450]: pgmap v8248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T01:54:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:20.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:20.320615+0000) 2022-09-08T01:54:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:20.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:20.264257+0000) 2022-09-08T01:54:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:21.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:21.320822+0000) 2022-09-08T01:54:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:21.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:21.264475+0000) 2022-09-08T01:54:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:21 smithi055 ceph-mon[97033]: pgmap v8249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.1 KiB/s rd, 137 B/s wr, 4 op/s 2022-09-08T01:54:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:21 smithi059 ceph-mon[97450]: pgmap v8249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.1 KiB/s rd, 137 B/s wr, 4 op/s 2022-09-08T01:54:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:22.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:22.321053+0000) 2022-09-08T01:54:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:22.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:22.264706+0000) 2022-09-08T01:54:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:23.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:23.321265+0000) 2022-09-08T01:54:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:23.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:23.264927+0000) 2022-09-08T01:54:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:23 smithi059 ceph-mon[97450]: pgmap v8250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T01:54:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:23 smithi055 ceph-mon[97033]: pgmap v8250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T01:54:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:24.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:24.321434+0000) 2022-09-08T01:54:24.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:54:24.442+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:54:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:24.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:24.265125+0000) 2022-09-08T01:54:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:25.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:25.321658+0000) 2022-09-08T01:54:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:25.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:25.265360+0000) 2022-09-08T01:54:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:25 smithi059 ceph-mon[97450]: pgmap v8251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:54:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:25 smithi055 ceph-mon[97033]: pgmap v8251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:54:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:26.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:26.321797+0000) 2022-09-08T01:54:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:26.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:26.265570+0000) 2022-09-08T01:54:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:27.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:27.321993+0000) 2022-09-08T01:54:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:27.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:27.265786+0000) 2022-09-08T01:54:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:27 smithi059 ceph-mon[97450]: pgmap v8252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:54:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:27 smithi055 ceph-mon[97033]: pgmap v8252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:54:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:28.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:28.265918+0000) 2022-09-08T01:54:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:28.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:28.322206+0000) 2022-09-08T01:54:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:29.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:29.266156+0000) 2022-09-08T01:54:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:29.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:29.322391+0000) 2022-09-08T01:54:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:29 smithi059 ceph-mon[97450]: pgmap v8253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:29 smithi055 ceph-mon[97033]: pgmap v8253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:30.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:30.266398+0000) 2022-09-08T01:54:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:30.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:30.322597+0000) 2022-09-08T01:54:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:31.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:31.266681+0000) 2022-09-08T01:54:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:31.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:31.322841+0000) 2022-09-08T01:54:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:31 smithi059 ceph-mon[97450]: pgmap v8254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:31 smithi055 ceph-mon[97033]: pgmap v8254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:32.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:32.266945+0000) 2022-09-08T01:54:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:32.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:32.323076+0000) 2022-09-08T01:54:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:33.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:33.323317+0000) 2022-09-08T01:54:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:33.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:33.267208+0000) 2022-09-08T01:54:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:33 smithi059 ceph-mon[97450]: pgmap v8255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:54:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:33 smithi055 ceph-mon[97033]: pgmap v8255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:54:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:34.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:34.323533+0000) 2022-09-08T01:54:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:34.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:34.267448+0000) 2022-09-08T01:54:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:35.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:35.267708+0000) 2022-09-08T01:54:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:35.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:35.323684+0000) 2022-09-08T01:54:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:35 smithi059 ceph-mon[97450]: pgmap v8256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:35 smithi055 ceph-mon[97033]: pgmap v8256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:36.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:36.323919+0000) 2022-09-08T01:54:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:36.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:36.267875+0000) 2022-09-08T01:54:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:37.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:37.324161+0000) 2022-09-08T01:54:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:37.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:37.268060+0000) 2022-09-08T01:54:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:37 smithi059 ceph-mon[97450]: pgmap v8257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:54:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:37 smithi055 ceph-mon[97033]: pgmap v8257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:54:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:38.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:38.324374+0000) 2022-09-08T01:54:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:38.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:38.268278+0000) 2022-09-08T01:54:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:38 smithi055 ceph-mon[97033]: pgmap v8258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:38 smithi059 ceph-mon[97450]: pgmap v8258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:39.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:39.324601+0000) 2022-09-08T01:54:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:39.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:39.268516+0000) 2022-09-08T01:54:39.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:54:39.448+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:54:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:40.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:40.324793+0000) 2022-09-08T01:54:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:40.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:40.268737+0000) 2022-09-08T01:54:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:40 smithi055 ceph-mon[97033]: pgmap v8259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:40 smithi059 ceph-mon[97450]: pgmap v8259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:41.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:41.325035+0000) 2022-09-08T01:54:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:41.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:41.268979+0000) 2022-09-08T01:54:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:42.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:42.325247+0000) 2022-09-08T01:54:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:42.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:42.269237+0000) 2022-09-08T01:54:43.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:43 smithi055 ceph-mon[97033]: pgmap v8260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:54:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:43 smithi059 ceph-mon[97450]: pgmap v8260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:54:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:43.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:43.325484+0000) 2022-09-08T01:54:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:43.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:43.269446+0000) 2022-09-08T01:54:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:44.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:44.325743+0000) 2022-09-08T01:54:44.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:44.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:44.269719+0000) 2022-09-08T01:54:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:45 smithi055 ceph-mon[97033]: pgmap v8261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:45 smithi059 ceph-mon[97450]: pgmap v8261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:45.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:45.325922+0000) 2022-09-08T01:54:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:45.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:45.269996+0000) 2022-09-08T01:54:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:46.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:46.326101+0000) 2022-09-08T01:54:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:46.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:46.270233+0000) 2022-09-08T01:54:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:47 smithi055 ceph-mon[97033]: pgmap v8262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:54:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:47 smithi059 ceph-mon[97450]: pgmap v8262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:54:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:47.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:47.326308+0000) 2022-09-08T01:54:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:47.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:47.270497+0000) 2022-09-08T01:54:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:48.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:48.326533+0000) 2022-09-08T01:54:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:48.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:48.270687+0000) 2022-09-08T01:54:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:49 smithi055 ceph-mon[97033]: pgmap v8263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:49 smithi059 ceph-mon[97450]: pgmap v8263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:49.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:49.326754+0000) 2022-09-08T01:54:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:49.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:49.270867+0000) 2022-09-08T01:54:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:50.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:50.326948+0000) 2022-09-08T01:54:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:50.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:50.271082+0000) 2022-09-08T01:54:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:51 smithi055 ceph-mon[97033]: pgmap v8264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:51 smithi059 ceph-mon[97450]: pgmap v8264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:51.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:51.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:51.271324+0000) 2022-09-08T01:54:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:51.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:51.327153+0000) 2022-09-08T01:54:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:52.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:52.327313+0000) 2022-09-08T01:54:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:52.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:52.271523+0000) 2022-09-08T01:54:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:53 smithi055 ceph-mon[97033]: pgmap v8265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:54:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:53 smithi059 ceph-mon[97450]: pgmap v8265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:54:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:53.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:53.327466+0000) 2022-09-08T01:54:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:53.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:53.271703+0000) 2022-09-08T01:54:54.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:54:54.457+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:54:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:54.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:54.327726+0000) 2022-09-08T01:54:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:54.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:54.271919+0000) 2022-09-08T01:54:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:55 smithi055 ceph-mon[97033]: pgmap v8266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:55 smithi059 ceph-mon[97450]: pgmap v8266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:55.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:55.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:55.272138+0000) 2022-09-08T01:54:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:55.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:55.327967+0000) 2022-09-08T01:54:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:56.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:56.328209+0000) 2022-09-08T01:54:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:56.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:56.272369+0000) 2022-09-08T01:54:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:57 smithi055 ceph-mon[97033]: pgmap v8267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:54:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:57 smithi059 ceph-mon[97450]: pgmap v8267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:54:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:57.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:57.328418+0000) 2022-09-08T01:54:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:57.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:57.272608+0000) 2022-09-08T01:54:58.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:58.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:58.272850+0000) 2022-09-08T01:54:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:58.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:58.328591+0000) 2022-09-08T01:54:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:54:59 smithi055 ceph-mon[97033]: pgmap v8268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:54:59 smithi059 ceph-mon[97450]: pgmap v8268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:54:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:54:59.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:59.328768+0000) 2022-09-08T01:54:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:54:59.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:54:59.272994+0000) 2022-09-08T01:55:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:00.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:00.273187+0000) 2022-09-08T01:55:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:00.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:00.328949+0000) 2022-09-08T01:55:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:01 smithi055 ceph-mon[97033]: pgmap v8269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:01 smithi059 ceph-mon[97450]: pgmap v8269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:01.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:01.329130+0000) 2022-09-08T01:55:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:01.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:01.273369+0000) 2022-09-08T01:55:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:02.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:02.329329+0000) 2022-09-08T01:55:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:02.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:02.273579+0000) 2022-09-08T01:55:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:03 smithi055 ceph-mon[97033]: pgmap v8270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:55:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:03 smithi059 ceph-mon[97450]: pgmap v8270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:55:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:03.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:03.329584+0000) 2022-09-08T01:55:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:03.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:03.273803+0000) 2022-09-08T01:55:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:04.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:04.329810+0000) 2022-09-08T01:55:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:04.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:04.273999+0000) 2022-09-08T01:55:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:05 smithi055 ceph-mon[97033]: pgmap v8271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:55:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:05 smithi059 ceph-mon[97450]: pgmap v8271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:55:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:05.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:05.329982+0000) 2022-09-08T01:55:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:05.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:05.274231+0000) 2022-09-08T01:55:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:55:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:06.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:06.330213+0000) 2022-09-08T01:55:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:06.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:06.274417+0000) 2022-09-08T01:55:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:55:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:07 smithi059 ceph-mon[97450]: pgmap v8272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:55:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:07.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:07.330370+0000) 2022-09-08T01:55:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:07.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:07.274682+0000) 2022-09-08T01:55:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:07 smithi055 ceph-mon[97033]: pgmap v8272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:55:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:08.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:08.330605+0000) 2022-09-08T01:55:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:08.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:08.274871+0000) 2022-09-08T01:55:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:09 smithi059 ceph-mon[97450]: pgmap v8273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:55:09.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:55:09.469+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:55:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:09.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:09.330823+0000) 2022-09-08T01:55:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:09.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:09.275022+0000) 2022-09-08T01:55:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:09 smithi055 ceph-mon[97033]: pgmap v8273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:55:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:10 smithi059 ceph-mon[97450]: osdmap e599: 12 total, 11 up, 11 in 2022-09-08T01:55:10.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:10.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:10.275234+0000) 2022-09-08T01:55:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:10.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:10.330976+0000) 2022-09-08T01:55:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:10 smithi055 ceph-mon[97033]: osdmap e599: 12 total, 11 up, 11 in 2022-09-08T01:55:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:11 smithi059 ceph-mon[97450]: pgmap v8275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:55:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:11.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:11.331130+0000) 2022-09-08T01:55:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:11.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:11.275407+0000) 2022-09-08T01:55:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:11 smithi055 ceph-mon[97033]: pgmap v8275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:55:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:12.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:12.275617+0000) 2022-09-08T01:55:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:12.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:12.331322+0000) 2022-09-08T01:55:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:13 smithi059 ceph-mon[97450]: pgmap v8276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:55:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:13.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:13.331553+0000) 2022-09-08T01:55:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:13.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:13.275936+0000) 2022-09-08T01:55:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:13 smithi055 ceph-mon[97033]: pgmap v8276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:55:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:55:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:55:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:55:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:14.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:14.331786+0000) 2022-09-08T01:55:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:14.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:14.276188+0000) 2022-09-08T01:55:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:55:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:55:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:55:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:15 smithi059 ceph-mon[97450]: pgmap v8277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:55:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:15.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:15.331932+0000) 2022-09-08T01:55:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:15.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:15.276414+0000) 2022-09-08T01:55:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:15 smithi055 ceph-mon[97033]: pgmap v8277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:55:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:16.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:16.332158+0000) 2022-09-08T01:55:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:16.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:16.276610+0000) 2022-09-08T01:55:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:17 smithi059 ceph-mon[97450]: pgmap v8278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:55:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:17 smithi059 ceph-mon[97450]: osdmap e600: 12 total, 11 up, 11 in 2022-09-08T01:55:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:17.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:17.332385+0000) 2022-09-08T01:55:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:17.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:17.276896+0000) 2022-09-08T01:55:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:17 smithi055 ceph-mon[97033]: pgmap v8278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:55:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:17 smithi055 ceph-mon[97033]: osdmap e600: 12 total, 11 up, 11 in 2022-09-08T01:55:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:18.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:18.332609+0000) 2022-09-08T01:55:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:18.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:18.277128+0000) 2022-09-08T01:55:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:19 smithi059 ceph-mon[97450]: pgmap v8280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 243 B/s wr, 7 op/s 2022-09-08T01:55:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:19.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:19.332753+0000) 2022-09-08T01:55:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:19.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:19.277408+0000) 2022-09-08T01:55:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:19 smithi055 ceph-mon[97033]: pgmap v8280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 243 B/s wr, 7 op/s 2022-09-08T01:55:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:20.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:20.332913+0000) 2022-09-08T01:55:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:20.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:20.277724+0000) 2022-09-08T01:55:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:21 smithi059 ceph-mon[97450]: pgmap v8281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:55:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:21.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:21.333144+0000) 2022-09-08T01:55:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:21.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:21.277960+0000) 2022-09-08T01:55:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:21 smithi055 ceph-mon[97033]: pgmap v8281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:55:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:22.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:22.333361+0000) 2022-09-08T01:55:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:22.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:22.278157+0000) 2022-09-08T01:55:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:23 smithi059 ceph-mon[97450]: pgmap v8282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:55:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:23.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:23.333607+0000) 2022-09-08T01:55:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:23.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:23.278406+0000) 2022-09-08T01:55:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:23 smithi055 ceph-mon[97033]: pgmap v8282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:55:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:24.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:24.333855+0000) 2022-09-08T01:55:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:24.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:24.278719+0000) 2022-09-08T01:55:24.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:55:24.483+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:55:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:25 smithi059 ceph-mon[97450]: pgmap v8283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:55:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:25.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:25.334069+0000) 2022-09-08T01:55:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:25.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:25.278906+0000) 2022-09-08T01:55:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:25 smithi055 ceph-mon[97033]: pgmap v8283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:55:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:26.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:26.334264+0000) 2022-09-08T01:55:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:26.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:26.279087+0000) 2022-09-08T01:55:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:27.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:27.334488+0000) 2022-09-08T01:55:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:27.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:27.279335+0000) 2022-09-08T01:55:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:27 smithi055 ceph-mon[97033]: pgmap v8284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:55:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:27 smithi059 ceph-mon[97450]: pgmap v8284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:55:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:28.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:28.334673+0000) 2022-09-08T01:55:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:28.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:28.279562+0000) 2022-09-08T01:55:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:29.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:29.334910+0000) 2022-09-08T01:55:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:29.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:29.279734+0000) 2022-09-08T01:55:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:29 smithi055 ceph-mon[97033]: pgmap v8285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:55:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:29 smithi059 ceph-mon[97450]: pgmap v8285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:55:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:30.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:30.335124+0000) 2022-09-08T01:55:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:30.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:30.279900+0000) 2022-09-08T01:55:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:31.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:31.335311+0000) 2022-09-08T01:55:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:31.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:31.280160+0000) 2022-09-08T01:55:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:31 smithi055 ceph-mon[97033]: pgmap v8286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:31 smithi059 ceph-mon[97450]: pgmap v8286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:32.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:32.335552+0000) 2022-09-08T01:55:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:32.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:32.280454+0000) 2022-09-08T01:55:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:33.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:33.335779+0000) 2022-09-08T01:55:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:33.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:33.280752+0000) 2022-09-08T01:55:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:33 smithi055 ceph-mon[97033]: pgmap v8287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:55:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:33 smithi059 ceph-mon[97450]: pgmap v8287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:55:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:34.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:34.335992+0000) 2022-09-08T01:55:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:34.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:34.280920+0000) 2022-09-08T01:55:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:35.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:35.336212+0000) 2022-09-08T01:55:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:35.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:35.281172+0000) 2022-09-08T01:55:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:35 smithi055 ceph-mon[97033]: pgmap v8288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:35 smithi059 ceph-mon[97450]: pgmap v8288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:36.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:36.336411+0000) 2022-09-08T01:55:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:36.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:36.281473+0000) 2022-09-08T01:55:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:37.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:37.336613+0000) 2022-09-08T01:55:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:37.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:37.281719+0000) 2022-09-08T01:55:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:37 smithi055 ceph-mon[97033]: pgmap v8289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:55:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:37 smithi059 ceph-mon[97450]: pgmap v8289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:55:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:38.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:38.336883+0000) 2022-09-08T01:55:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:38.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:38.281900+0000) 2022-09-08T01:55:39.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:55:39.493+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:55:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:39.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:39.337120+0000) 2022-09-08T01:55:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:39.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:39.282178+0000) 2022-09-08T01:55:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:39 smithi055 ceph-mon[97033]: pgmap v8290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:39 smithi059 ceph-mon[97450]: pgmap v8290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:40.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:40.337355+0000) 2022-09-08T01:55:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:40.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:40.282474+0000) 2022-09-08T01:55:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:41.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:41.337625+0000) 2022-09-08T01:55:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:41.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:41.282767+0000) 2022-09-08T01:55:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:41 smithi055 ceph-mon[97033]: pgmap v8291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:41 smithi059 ceph-mon[97450]: pgmap v8291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:42.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:42.337897+0000) 2022-09-08T01:55:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:42.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:42.283012+0000) 2022-09-08T01:55:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:43.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:43.338182+0000) 2022-09-08T01:55:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:43.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:43.283213+0000) 2022-09-08T01:55:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:43 smithi055 ceph-mon[97033]: pgmap v8292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:55:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:43 smithi059 ceph-mon[97450]: pgmap v8292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:55:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:44.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:44.338417+0000) 2022-09-08T01:55:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:44.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:44.283449+0000) 2022-09-08T01:55:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:44 smithi055 ceph-mon[97033]: pgmap v8293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:44 smithi055 ceph-mon[97033]: pgmap v8294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:55:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:44 smithi055 ceph-mon[97033]: pgmap v8295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:55:45.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:55:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:44 smithi059 ceph-mon[97450]: pgmap v8293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:44 smithi059 ceph-mon[97450]: pgmap v8294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T01:55:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:44 smithi059 ceph-mon[97450]: pgmap v8295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T01:55:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:55:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:45.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:45.338651+0000) 2022-09-08T01:55:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:45.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:45.283768+0000) 2022-09-08T01:55:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:46.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:46.338764+0000) 2022-09-08T01:55:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:46.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:46.283928+0000) 2022-09-08T01:55:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:46 smithi055 ceph-mon[97033]: pgmap v8296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:55:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:46 smithi059 ceph-mon[97450]: pgmap v8296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:55:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:47.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:47.338925+0000) 2022-09-08T01:55:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:47.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:47.284137+0000) 2022-09-08T01:55:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:48.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:48.339102+0000) 2022-09-08T01:55:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:48.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:48.284370+0000) 2022-09-08T01:55:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:49 smithi055 ceph-mon[97033]: pgmap v8297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:55:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:49 smithi059 ceph-mon[97450]: pgmap v8297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:55:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:49.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:49.339351+0000) 2022-09-08T01:55:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:49.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:49.284654+0000) 2022-09-08T01:55:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:50.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:50.339534+0000) 2022-09-08T01:55:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:50.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:50.284856+0000) 2022-09-08T01:55:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:51 smithi055 ceph-mon[97033]: pgmap v8298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-08T01:55:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:51 smithi059 ceph-mon[97450]: pgmap v8298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-08T01:55:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:51.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:51.285041+0000) 2022-09-08T01:55:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:51.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:51.339766+0000) 2022-09-08T01:55:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:52.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:52.285270+0000) 2022-09-08T01:55:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:52.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:52.339997+0000) 2022-09-08T01:55:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:53 smithi055 ceph-mon[97033]: pgmap v8299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:55:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:53 smithi059 ceph-mon[97450]: pgmap v8299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T01:55:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:53.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:53.340186+0000) 2022-09-08T01:55:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:53.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:53.285574+0000) 2022-09-08T01:55:54.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:55:54.503+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:55:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:54.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:54.285812+0000) 2022-09-08T01:55:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:54.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:54.340442+0000) 2022-09-08T01:55:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:55 smithi055 ceph-mon[97033]: pgmap v8300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:55:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:55 smithi059 ceph-mon[97450]: pgmap v8300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:55:55.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:55.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:55.286063+0000) 2022-09-08T01:55:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:55.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:55.340718+0000) 2022-09-08T01:55:56.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:56.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:56.286248+0000) 2022-09-08T01:55:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:56.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:56.340997+0000) 2022-09-08T01:55:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:57 smithi055 ceph-mon[97033]: pgmap v8301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:55:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:57 smithi059 ceph-mon[97450]: pgmap v8301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:55:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:57.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:57.341127+0000) 2022-09-08T01:55:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:57.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:57.286505+0000) 2022-09-08T01:55:58.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:58.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:58.286765+0000) 2022-09-08T01:55:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:58.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:58.341399+0000) 2022-09-08T01:55:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:55:59 smithi055 ceph-mon[97033]: pgmap v8302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:55:59 smithi059 ceph-mon[97450]: pgmap v8302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:55:59.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:55:59.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:59.286922+0000) 2022-09-08T01:55:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:55:59.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:55:59.341627+0000) 2022-09-08T01:56:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:00.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:00.287074+0000) 2022-09-08T01:56:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:00.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:00.341793+0000) 2022-09-08T01:56:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:01 smithi055 ceph-mon[97033]: pgmap v8303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T01:56:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:01 smithi059 ceph-mon[97450]: pgmap v8303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T01:56:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:01.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:01.287275+0000) 2022-09-08T01:56:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:01.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:01.341973+0000) 2022-09-08T01:56:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:02.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:02.287502+0000) 2022-09-08T01:56:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:02.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:02.342202+0000) 2022-09-08T01:56:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:03 smithi055 ceph-mon[97033]: pgmap v8304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T01:56:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:03 smithi059 ceph-mon[97450]: pgmap v8304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T01:56:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:03.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:03.287624+0000) 2022-09-08T01:56:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:03.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:03.342440+0000) 2022-09-08T01:56:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:04.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:04.287829+0000) 2022-09-08T01:56:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:04.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:04.342744+0000) 2022-09-08T01:56:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:05 smithi055 ceph-mon[97033]: pgmap v8305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:56:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:05 smithi059 ceph-mon[97450]: pgmap v8305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:56:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:05.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:05.342973+0000) 2022-09-08T01:56:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:05.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:05.287956+0000) 2022-09-08T01:56:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:56:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:06.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:06.288172+0000) 2022-09-08T01:56:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:06.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:06.343245+0000) 2022-09-08T01:56:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:56:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:07 smithi059 ceph-mon[97450]: pgmap v8306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:56:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:07.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:07.288397+0000) 2022-09-08T01:56:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:07.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:07.343514+0000) 2022-09-08T01:56:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:07 smithi055 ceph-mon[97033]: pgmap v8306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:56:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:08.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:08.343745+0000) 2022-09-08T01:56:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:08.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:08.288583+0000) 2022-09-08T01:56:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:09 smithi059 ceph-mon[97450]: pgmap v8307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:56:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:09.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:09.343883+0000) 2022-09-08T01:56:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:09.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:09.288780+0000) 2022-09-08T01:56:09.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:56:09.519+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:56:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:09 smithi055 ceph-mon[97033]: pgmap v8307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:56:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:10 smithi059 ceph-mon[97450]: osdmap e601: 12 total, 11 up, 11 in 2022-09-08T01:56:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:10.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:10.344042+0000) 2022-09-08T01:56:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:10.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:10.289021+0000) 2022-09-08T01:56:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:10 smithi055 ceph-mon[97033]: osdmap e601: 12 total, 11 up, 11 in 2022-09-08T01:56:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:11 smithi059 ceph-mon[97450]: pgmap v8309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 53 KiB/s wr, 25 op/s 2022-09-08T01:56:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:11.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:11.344298+0000) 2022-09-08T01:56:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:11.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:11.289213+0000) 2022-09-08T01:56:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:11 smithi055 ceph-mon[97033]: pgmap v8309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 15 KiB/s rd, 53 KiB/s wr, 25 op/s 2022-09-08T01:56:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:12.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:12.289444+0000) 2022-09-08T01:56:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:12.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:12.344541+0000) 2022-09-08T01:56:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:13 smithi055 ceph-mon[97033]: pgmap v8310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:56:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:13 smithi055 ceph-mon[97033]: osdmap e602: 12 total, 11 up, 11 in 2022-09-08T01:56:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:13 smithi059 ceph-mon[97450]: pgmap v8310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:56:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:13 smithi059 ceph-mon[97450]: osdmap e602: 12 total, 11 up, 11 in 2022-09-08T01:56:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:13.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:13.344754+0000) 2022-09-08T01:56:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:13.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:13.289719+0000) 2022-09-08T01:56:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:14.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:14.344907+0000) 2022-09-08T01:56:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:14.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:14.289945+0000) 2022-09-08T01:56:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:15 smithi055 ceph-mon[97033]: pgmap v8312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:56:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:15 smithi059 ceph-mon[97450]: pgmap v8312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:56:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:15.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:15.345030+0000) 2022-09-08T01:56:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:15.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:15.290148+0000) 2022-09-08T01:56:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:16.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:16.290351+0000) 2022-09-08T01:56:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:16.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:16.345223+0000) 2022-09-08T01:56:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:17 smithi055 ceph-mon[97033]: pgmap v8313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:56:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:17 smithi059 ceph-mon[97450]: pgmap v8313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:56:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:17.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:17.290544+0000) 2022-09-08T01:56:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:17.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:17.345413+0000) 2022-09-08T01:56:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:18.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:18.290705+0000) 2022-09-08T01:56:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:18.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:18.345547+0000) 2022-09-08T01:56:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:19 smithi055 ceph-mon[97033]: pgmap v8314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 235 B/s wr, 7 op/s 2022-09-08T01:56:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:19 smithi059 ceph-mon[97450]: pgmap v8314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 235 B/s wr, 7 op/s 2022-09-08T01:56:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:19.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:19.345733+0000) 2022-09-08T01:56:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:19.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:19.290947+0000) 2022-09-08T01:56:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:20.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:20.291146+0000) 2022-09-08T01:56:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:20.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:20.345983+0000) 2022-09-08T01:56:21.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:21 smithi055 ceph-mon[97033]: pgmap v8315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:56:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:21 smithi059 ceph-mon[97450]: pgmap v8315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:56:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:21.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:21.346185+0000) 2022-09-08T01:56:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:21.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:21.291277+0000) 2022-09-08T01:56:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:22.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:22.346333+0000) 2022-09-08T01:56:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:22.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:22.291493+0000) 2022-09-08T01:56:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:23 smithi055 ceph-mon[97033]: pgmap v8316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:56:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:23 smithi059 ceph-mon[97450]: pgmap v8316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:56:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:23.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:23.346554+0000) 2022-09-08T01:56:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:23.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:23.291748+0000) 2022-09-08T01:56:24.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:56:24.529+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:56:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:24.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:24.291916+0000) 2022-09-08T01:56:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:24.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:24.346778+0000) 2022-09-08T01:56:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:25 smithi055 ceph-mon[97033]: pgmap v8317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 87 B/s wr, 5 op/s 2022-09-08T01:56:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:25 smithi059 ceph-mon[97450]: pgmap v8317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 87 B/s wr, 5 op/s 2022-09-08T01:56:25.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:25.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:25.292164+0000) 2022-09-08T01:56:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:25.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:25.347004+0000) 2022-09-08T01:56:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:26.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:26.347169+0000) 2022-09-08T01:56:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:26.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:26.292376+0000) 2022-09-08T01:56:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:27 smithi055 ceph-mon[97033]: pgmap v8318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:56:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:27 smithi059 ceph-mon[97450]: pgmap v8318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:56:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:27.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:27.347331+0000) 2022-09-08T01:56:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:27.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:27.292513+0000) 2022-09-08T01:56:28.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:28.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:28.292666+0000) 2022-09-08T01:56:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:28.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:28.347478+0000) 2022-09-08T01:56:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:29 smithi059 ceph-mon[97450]: pgmap v8319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:29.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:29.347611+0000) 2022-09-08T01:56:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:29.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:29.292808+0000) 2022-09-08T01:56:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:29 smithi055 ceph-mon[97033]: pgmap v8319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:30.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:30.292938+0000) 2022-09-08T01:56:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:30.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:30.347708+0000) 2022-09-08T01:56:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:31 smithi059 ceph-mon[97450]: pgmap v8320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:56:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:31.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:31.347836+0000) 2022-09-08T01:56:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:31.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:31.293061+0000) 2022-09-08T01:56:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:31 smithi055 ceph-mon[97033]: pgmap v8320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:56:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:32.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:32.347955+0000) 2022-09-08T01:56:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:32.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:32.293185+0000) 2022-09-08T01:56:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:33 smithi059 ceph-mon[97450]: pgmap v8321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:33.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:33.348093+0000) 2022-09-08T01:56:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:33.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:33.293326+0000) 2022-09-08T01:56:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:33 smithi055 ceph-mon[97033]: pgmap v8321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:34.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:34.348226+0000) 2022-09-08T01:56:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:34.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:34.293477+0000) 2022-09-08T01:56:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:35 smithi059 ceph-mon[97450]: pgmap v8322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:35.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:35.348362+0000) 2022-09-08T01:56:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:35.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:35.293626+0000) 2022-09-08T01:56:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:35 smithi055 ceph-mon[97033]: pgmap v8322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:36.620 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:36.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:36.348505+0000) 2022-09-08T01:56:36.621 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:36.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:36.293786+0000) 2022-09-08T01:56:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:37 smithi059 ceph-mon[97450]: pgmap v8323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:56:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:37.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:37.348646+0000) 2022-09-08T01:56:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:37.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:37.293931+0000) 2022-09-08T01:56:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:37 smithi055 ceph-mon[97033]: pgmap v8323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:56:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:38.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:38.348768+0000) 2022-09-08T01:56:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:38.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:38.294066+0000) 2022-09-08T01:56:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:39.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:39.348901+0000) 2022-09-08T01:56:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:39.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:39.294207+0000) 2022-09-08T01:56:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:39 smithi059 ceph-mon[97450]: pgmap v8324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:39 smithi055 ceph-mon[97033]: pgmap v8324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:39.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:56:39.544+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:56:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:40.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:40.349031+0000) 2022-09-08T01:56:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:40.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:40.294347+0000) 2022-09-08T01:56:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:41 smithi059 ceph-mon[97450]: pgmap v8325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:56:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:41.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:41.349155+0000) 2022-09-08T01:56:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:41.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:41.294488+0000) 2022-09-08T01:56:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:41 smithi055 ceph-mon[97033]: pgmap v8325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:56:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:42.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:42.349394+0000) 2022-09-08T01:56:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:42.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:42.294695+0000) 2022-09-08T01:56:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:43 smithi059 ceph-mon[97450]: pgmap v8326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:43.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:43.349707+0000) 2022-09-08T01:56:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:43.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:43.294907+0000) 2022-09-08T01:56:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:43 smithi055 ceph-mon[97033]: pgmap v8326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:44.350 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:44.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:44.295102+0000) 2022-09-08T01:56:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:56:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:56:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:56:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:56:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:56:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:56:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:44.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:44.349865+0000) 2022-09-08T01:56:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:45 smithi059 ceph-mon[97450]: pgmap v8327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:56:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:56:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:45.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:45.350121+0000) 2022-09-08T01:56:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:45.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:45.295327+0000) 2022-09-08T01:56:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:45 smithi055 ceph-mon[97033]: pgmap v8327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:56:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:56:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:46.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:46.350387+0000) 2022-09-08T01:56:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:46.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:46.295517+0000) 2022-09-08T01:56:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:47 smithi059 ceph-mon[97450]: pgmap v8328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:56:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:56:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:56:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:47.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:47.350544+0000) 2022-09-08T01:56:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:47.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:47.295715+0000) 2022-09-08T01:56:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:47 smithi055 ceph-mon[97033]: pgmap v8328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:56:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:56:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:56:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:48.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:48.350769+0000) 2022-09-08T01:56:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:48.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:48.295888+0000) 2022-09-08T01:56:48.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:56:48.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:56:48.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:48 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T01:56:48.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:48 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T01:56:48.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.739 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.739 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:56:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:56:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:48 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T01:56:48.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:48 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T01:56:48.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:48.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T01:56:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:49.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:49.350914+0000) 2022-09-08T01:56:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:49.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:49.296088+0000) 2022-09-08T01:56:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:49 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T01:56:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:49 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T01:56:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:49 smithi055 ceph-mon[97033]: pgmap v8329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:49 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T01:56:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:49 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T01:56:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:49 smithi059 ceph-mon[97450]: pgmap v8329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:50.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:50.296268+0000) 2022-09-08T01:56:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:50.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:50.351082+0000) 2022-09-08T01:56:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:51.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:51.351339+0000) 2022-09-08T01:56:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:51.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:51.296472+0000) 2022-09-08T01:56:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:51 smithi055 ceph-mon[97033]: pgmap v8330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:56:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:51 smithi059 ceph-mon[97450]: pgmap v8330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:56:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:52.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:52.351561+0000) 2022-09-08T01:56:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:52.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:52.296680+0000) 2022-09-08T01:56:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:53.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:53.351796+0000) 2022-09-08T01:56:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:53.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:53.296892+0000) 2022-09-08T01:56:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:53 smithi055 ceph-mon[97033]: pgmap v8331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:53 smithi059 ceph-mon[97450]: pgmap v8331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:54.553 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:54.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:54.351918+0000) 2022-09-08T01:56:54.554 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:54.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:54.297123+0000) 2022-09-08T01:56:54.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:56:54.552+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:56:55.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:55.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:55.297374+0000) 2022-09-08T01:56:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:55.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:55.352136+0000) 2022-09-08T01:56:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:55 smithi055 ceph-mon[97033]: pgmap v8332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:55 smithi059 ceph-mon[97450]: pgmap v8332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:56.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:56.352395+0000) 2022-09-08T01:56:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:56.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:56.297560+0000) 2022-09-08T01:56:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:57.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:57.352617+0000) 2022-09-08T01:56:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:57.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:57.297772+0000) 2022-09-08T01:56:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:57 smithi055 ceph-mon[97033]: pgmap v8333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:56:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:57 smithi059 ceph-mon[97450]: pgmap v8333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:56:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:58.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:58.298007+0000) 2022-09-08T01:56:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:58.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:58.352759+0000) 2022-09-08T01:56:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:56:59.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:59.352930+0000) 2022-09-08T01:56:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:56:59.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:56:59.298243+0000) 2022-09-08T01:56:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:56:59 smithi055 ceph-mon[97033]: pgmap v8334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:56:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:56:59 smithi059 ceph-mon[97450]: pgmap v8334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:00.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:00.353132+0000) 2022-09-08T01:57:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:00.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:00.298415+0000) 2022-09-08T01:57:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:01.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:01.298622+0000) 2022-09-08T01:57:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:01.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:01.353313+0000) 2022-09-08T01:57:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:01 smithi055 ceph-mon[97033]: pgmap v8335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:57:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:01 smithi059 ceph-mon[97450]: pgmap v8335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:57:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:02.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:02.353484+0000) 2022-09-08T01:57:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:02.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:02.298840+0000) 2022-09-08T01:57:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:03.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:03.353731+0000) 2022-09-08T01:57:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:03.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:03.299037+0000) 2022-09-08T01:57:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:03 smithi055 ceph-mon[97033]: pgmap v8336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:03 smithi059 ceph-mon[97450]: pgmap v8336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:04.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:04.353946+0000) 2022-09-08T01:57:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:04.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:04.299251+0000) 2022-09-08T01:57:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:05.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:05.354119+0000) 2022-09-08T01:57:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:05.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:05.299458+0000) 2022-09-08T01:57:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:05 smithi055 ceph-mon[97033]: pgmap v8337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:57:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:57:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:05 smithi059 ceph-mon[97450]: pgmap v8337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:57:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:57:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:06.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:06.354321+0000) 2022-09-08T01:57:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:06.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:06.299666+0000) 2022-09-08T01:57:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:07.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:07.354545+0000) 2022-09-08T01:57:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:07.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:07.299821+0000) 2022-09-08T01:57:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:07 smithi055 ceph-mon[97033]: pgmap v8338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:57:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:07 smithi059 ceph-mon[97450]: pgmap v8338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:57:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:08.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:08.354690+0000) 2022-09-08T01:57:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:08.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:08.299959+0000) 2022-09-08T01:57:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:09.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:09.354815+0000) 2022-09-08T01:57:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:09.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:09.300161+0000) 2022-09-08T01:57:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:09 smithi055 ceph-mon[97033]: pgmap v8339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:57:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:09 smithi059 ceph-mon[97450]: pgmap v8339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:57:09.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:57:09.568+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:57:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:10.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:10.354990+0000) 2022-09-08T01:57:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:10.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:10.300386+0000) 2022-09-08T01:57:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:10 smithi055 ceph-mon[97033]: osdmap e603: 12 total, 11 up, 11 in 2022-09-08T01:57:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:10 smithi059 ceph-mon[97450]: osdmap e603: 12 total, 11 up, 11 in 2022-09-08T01:57:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:11.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:11.355200+0000) 2022-09-08T01:57:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:11.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:11.300663+0000) 2022-09-08T01:57:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:11 smithi055 ceph-mon[97033]: pgmap v8341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:57:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:11 smithi059 ceph-mon[97450]: pgmap v8341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:57:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:12.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:12.355416+0000) 2022-09-08T01:57:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:12.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:12.300852+0000) 2022-09-08T01:57:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:13 smithi055 ceph-mon[97033]: pgmap v8342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:57:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:13 smithi055 ceph-mon[97033]: osdmap e604: 12 total, 11 up, 11 in 2022-09-08T01:57:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:13 smithi059 ceph-mon[97450]: pgmap v8342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T01:57:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:13 smithi059 ceph-mon[97450]: osdmap e604: 12 total, 11 up, 11 in 2022-09-08T01:57:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:13.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:13.355629+0000) 2022-09-08T01:57:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:13.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:13.301040+0000) 2022-09-08T01:57:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:14.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:14.355853+0000) 2022-09-08T01:57:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:14.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:14.301216+0000) 2022-09-08T01:57:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:15 smithi055 ceph-mon[97033]: pgmap v8344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:57:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:15 smithi055 ceph-mon[97033]: pgmap v8345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T01:57:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:15 smithi055 ceph-mon[97033]: pgmap v8346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2022-09-08T01:57:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:57:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:15 smithi059 ceph-mon[97450]: pgmap v8344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:57:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:15 smithi059 ceph-mon[97450]: pgmap v8345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T01:57:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:15 smithi059 ceph-mon[97450]: pgmap v8346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2022-09-08T01:57:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:57:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:15.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:15.356097+0000) 2022-09-08T01:57:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:15.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:15.301421+0000) 2022-09-08T01:57:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:16.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:16.356375+0000) 2022-09-08T01:57:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:16.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:16.301567+0000) 2022-09-08T01:57:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:17 smithi055 ceph-mon[97033]: pgmap v8347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T01:57:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:17 smithi059 ceph-mon[97450]: pgmap v8347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T01:57:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:17.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:17.356620+0000) 2022-09-08T01:57:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:17.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:17.301776+0000) 2022-09-08T01:57:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:18.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:18.356850+0000) 2022-09-08T01:57:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:18.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:18.301919+0000) 2022-09-08T01:57:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:19 smithi055 ceph-mon[97033]: pgmap v8348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T01:57:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:19 smithi059 ceph-mon[97450]: pgmap v8348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T01:57:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:19.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:19.357018+0000) 2022-09-08T01:57:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:19.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:19.302092+0000) 2022-09-08T01:57:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:20.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:20.357159+0000) 2022-09-08T01:57:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:20.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:20.302304+0000) 2022-09-08T01:57:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:21 smithi055 ceph-mon[97033]: pgmap v8349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 128 B/s wr, 7 op/s 2022-09-08T01:57:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:21 smithi059 ceph-mon[97450]: pgmap v8349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.9 KiB/s rd, 128 B/s wr, 7 op/s 2022-09-08T01:57:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:21.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:21.357351+0000) 2022-09-08T01:57:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:21.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:21.302517+0000) 2022-09-08T01:57:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:22.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:22.357530+0000) 2022-09-08T01:57:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:22.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:22.302680+0000) 2022-09-08T01:57:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:23 smithi059 ceph-mon[97450]: pgmap v8350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T01:57:23.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:23.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:23.302825+0000) 2022-09-08T01:57:23.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:23.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:23.357736+0000) 2022-09-08T01:57:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:23 smithi055 ceph-mon[97033]: pgmap v8350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T01:57:24.585 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:24.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:24.357968+0000) 2022-09-08T01:57:24.586 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:24.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:24.303030+0000) 2022-09-08T01:57:24.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:57:24.583+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:57:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:25 smithi059 ceph-mon[97450]: pgmap v8351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:57:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:25.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:25.358201+0000) 2022-09-08T01:57:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:25.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:25.303238+0000) 2022-09-08T01:57:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:25 smithi055 ceph-mon[97033]: pgmap v8351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:57:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:26.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:26.358373+0000) 2022-09-08T01:57:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:26.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:26.303438+0000) 2022-09-08T01:57:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:27 smithi059 ceph-mon[97450]: pgmap v8352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:57:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:27.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:27.358531+0000) 2022-09-08T01:57:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:27.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:27.303694+0000) 2022-09-08T01:57:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:27 smithi055 ceph-mon[97033]: pgmap v8352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:57:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:28.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:28.358757+0000) 2022-09-08T01:57:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:28.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:28.303928+0000) 2022-09-08T01:57:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:29 smithi059 ceph-mon[97450]: pgmap v8353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:29.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:29.358908+0000) 2022-09-08T01:57:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:29.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:29.304145+0000) 2022-09-08T01:57:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:29 smithi055 ceph-mon[97033]: pgmap v8353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:30.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:30.359089+0000) 2022-09-08T01:57:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:30.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:30.304294+0000) 2022-09-08T01:57:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:31 smithi059 ceph-mon[97450]: pgmap v8354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:57:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:31.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:31.359331+0000) 2022-09-08T01:57:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:31.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:31.304528+0000) 2022-09-08T01:57:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:31 smithi055 ceph-mon[97033]: pgmap v8354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:57:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:32.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:32.359456+0000) 2022-09-08T01:57:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:32.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:32.304732+0000) 2022-09-08T01:57:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:33 smithi059 ceph-mon[97450]: pgmap v8355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:33.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:33.359570+0000) 2022-09-08T01:57:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:33.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:33.304965+0000) 2022-09-08T01:57:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:33 smithi055 ceph-mon[97033]: pgmap v8355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:34.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:34.359701+0000) 2022-09-08T01:57:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:34.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:34.305222+0000) 2022-09-08T01:57:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:35 smithi059 ceph-mon[97450]: pgmap v8356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:35.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:35.359875+0000) 2022-09-08T01:57:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:35.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:35.305460+0000) 2022-09-08T01:57:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:35 smithi055 ceph-mon[97033]: pgmap v8356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:36.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:36.360063+0000) 2022-09-08T01:57:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:36.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:36.305720+0000) 2022-09-08T01:57:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:37 smithi059 ceph-mon[97450]: pgmap v8357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:57:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:37.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:37.360207+0000) 2022-09-08T01:57:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:37.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:37.305930+0000) 2022-09-08T01:57:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:37 smithi055 ceph-mon[97033]: pgmap v8357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:57:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:38.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:38.360407+0000) 2022-09-08T01:57:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:38.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:38.306153+0000) 2022-09-08T01:57:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:39 smithi059 ceph-mon[97450]: pgmap v8358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:39.599 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:39.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:39.360587+0000) 2022-09-08T01:57:39.600 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:39.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:39.306384+0000) 2022-09-08T01:57:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:39 smithi055 ceph-mon[97033]: pgmap v8358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:39.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:57:39.598+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:57:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:40.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:40.360753+0000) 2022-09-08T01:57:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:40.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:40.306530+0000) 2022-09-08T01:57:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:41 smithi059 ceph-mon[97450]: pgmap v8359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:57:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:41.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:41.360924+0000) 2022-09-08T01:57:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:41.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:41.306760+0000) 2022-09-08T01:57:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:41 smithi055 ceph-mon[97033]: pgmap v8359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:57:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:42.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:42.361153+0000) 2022-09-08T01:57:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:42.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:42.306900+0000) 2022-09-08T01:57:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:43 smithi059 ceph-mon[97450]: pgmap v8360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:43.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:43.361306+0000) 2022-09-08T01:57:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:43.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:43.307094+0000) 2022-09-08T01:57:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:43 smithi055 ceph-mon[97033]: pgmap v8360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:44.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:44.361479+0000) 2022-09-08T01:57:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:44.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:44.307336+0000) 2022-09-08T01:57:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:45 smithi059 ceph-mon[97450]: pgmap v8361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:45.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:45.307582+0000) 2022-09-08T01:57:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:45.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:45.361723+0000) 2022-09-08T01:57:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:45 smithi055 ceph-mon[97033]: pgmap v8361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:46.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:46.361953+0000) 2022-09-08T01:57:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:46.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:46.307825+0000) 2022-09-08T01:57:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:47 smithi059 ceph-mon[97450]: pgmap v8362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:57:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:47.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:47.362165+0000) 2022-09-08T01:57:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:47.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:47.308011+0000) 2022-09-08T01:57:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:47 smithi055 ceph-mon[97033]: pgmap v8362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:57:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:48.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:48.362332+0000) 2022-09-08T01:57:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:48.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:48.308210+0000) 2022-09-08T01:57:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:49 smithi059 ceph-mon[97450]: pgmap v8363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:49.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:49.362463+0000) 2022-09-08T01:57:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:49.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:49.308447+0000) 2022-09-08T01:57:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:49 smithi055 ceph-mon[97033]: pgmap v8363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:50.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:50.362616+0000) 2022-09-08T01:57:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:50.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:50.308679+0000) 2022-09-08T01:57:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:51 smithi059 ceph-mon[97450]: pgmap v8364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:57:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:51.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:51.362861+0000) 2022-09-08T01:57:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:51.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:51.308893+0000) 2022-09-08T01:57:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:51 smithi055 ceph-mon[97033]: pgmap v8364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:57:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:52.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:52.309137+0000) 2022-09-08T01:57:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:52.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:52.363068+0000) 2022-09-08T01:57:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:53 smithi059 ceph-mon[97450]: pgmap v8365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:53.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:53.363276+0000) 2022-09-08T01:57:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:53.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:53.309377+0000) 2022-09-08T01:57:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:53 smithi055 ceph-mon[97033]: pgmap v8365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:54.615 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:54.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:54.363490+0000) 2022-09-08T01:57:54.616 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:54.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:54.309594+0000) 2022-09-08T01:57:54.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:57:54.614+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:57:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:55.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:55.363714+0000) 2022-09-08T01:57:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:55.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:55.309847+0000) 2022-09-08T01:57:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:55 smithi055 ceph-mon[97033]: pgmap v8366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:55 smithi059 ceph-mon[97450]: pgmap v8366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:56.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:56.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:56.310023+0000) 2022-09-08T01:57:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:56.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:56.363883+0000) 2022-09-08T01:57:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:57.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:57.364054+0000) 2022-09-08T01:57:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:57.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:57.310201+0000) 2022-09-08T01:57:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:57 smithi055 ceph-mon[97033]: pgmap v8367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:57:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:57 smithi059 ceph-mon[97450]: pgmap v8367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:57:58.662 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:58.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:58.364284+0000) 2022-09-08T01:57:58.663 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:58.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:58.310417+0000) 2022-09-08T01:57:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:57:59.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:59.364492+0000) 2022-09-08T01:57:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:57:59.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:57:59.310717+0000) 2022-09-08T01:57:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:57:59 smithi055 ceph-mon[97033]: pgmap v8368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:57:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:57:59 smithi059 ceph-mon[97450]: pgmap v8368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:00.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:00.310949+0000) 2022-09-08T01:58:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:00.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:00.364739+0000) 2022-09-08T01:58:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:01.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:01.364937+0000) 2022-09-08T01:58:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:01.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:01.311179+0000) 2022-09-08T01:58:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:01 smithi055 ceph-mon[97033]: pgmap v8369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:58:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:01 smithi059 ceph-mon[97450]: pgmap v8369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:58:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:02.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:02.365170+0000) 2022-09-08T01:58:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:02.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:02.311393+0000) 2022-09-08T01:58:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:03.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:03.365313+0000) 2022-09-08T01:58:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:03.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:03.311682+0000) 2022-09-08T01:58:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:03 smithi055 ceph-mon[97033]: pgmap v8370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:03 smithi059 ceph-mon[97450]: pgmap v8370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:04.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:04.365479+0000) 2022-09-08T01:58:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:04.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:04.311941+0000) 2022-09-08T01:58:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:05.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:05.365744+0000) 2022-09-08T01:58:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:05.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:05.312166+0000) 2022-09-08T01:58:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:05 smithi055 ceph-mon[97033]: pgmap v8371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:58:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:58:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:05 smithi059 ceph-mon[97450]: pgmap v8371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:58:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:58:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:06.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:06.365923+0000) 2022-09-08T01:58:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:06.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:06.312390+0000) 2022-09-08T01:58:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:07.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:07.366172+0000) 2022-09-08T01:58:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:07.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:07.312679+0000) 2022-09-08T01:58:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:07 smithi055 ceph-mon[97033]: pgmap v8372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:58:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:07 smithi059 ceph-mon[97450]: pgmap v8372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:58:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:08.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:08.312933+0000) 2022-09-08T01:58:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:08.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:08.366355+0000) 2022-09-08T01:58:09.631 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:09.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:09.366527+0000) 2022-09-08T01:58:09.632 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:09.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:09.313080+0000) 2022-09-08T01:58:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:09 smithi055 ceph-mon[97033]: pgmap v8373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:58:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:09 smithi059 ceph-mon[97450]: pgmap v8373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:58:09.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:58:09.630+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:58:10.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:10.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:10.313354+0000) 2022-09-08T01:58:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:10.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:10.366715+0000) 2022-09-08T01:58:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:10 smithi055 ceph-mon[97033]: osdmap e605: 12 total, 11 up, 11 in 2022-09-08T01:58:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:10 smithi059 ceph-mon[97450]: osdmap e605: 12 total, 11 up, 11 in 2022-09-08T01:58:11.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:11.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:11.313614+0000) 2022-09-08T01:58:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:11.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:11.366875+0000) 2022-09-08T01:58:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:11 smithi055 ceph-mon[97033]: pgmap v8375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:58:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:11 smithi059 ceph-mon[97450]: pgmap v8375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:58:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:12.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:12.367049+0000) 2022-09-08T01:58:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:12.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:12.313701+0000) 2022-09-08T01:58:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:13 smithi059 ceph-mon[97450]: pgmap v8376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:58:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:13 smithi059 ceph-mon[97450]: osdmap e606: 12 total, 11 up, 11 in 2022-09-08T01:58:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:13.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:13.367229+0000) 2022-09-08T01:58:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:13.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:13.313839+0000) 2022-09-08T01:58:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:13 smithi055 ceph-mon[97033]: pgmap v8376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:58:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:13 smithi055 ceph-mon[97033]: osdmap e606: 12 total, 11 up, 11 in 2022-09-08T01:58:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:14.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:14.367367+0000) 2022-09-08T01:58:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:14.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:14.313981+0000) 2022-09-08T01:58:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:15 smithi059 ceph-mon[97450]: pgmap v8378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:58:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:58:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:58:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:58:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:58:15.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:58:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:15.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:15.367586+0000) 2022-09-08T01:58:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:15.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:15.314188+0000) 2022-09-08T01:58:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:15 smithi055 ceph-mon[97033]: pgmap v8378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T01:58:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:58:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:58:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:58:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:58:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:58:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:16.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:16.367810+0000) 2022-09-08T01:58:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:16.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:16.314405+0000) 2022-09-08T01:58:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:17 smithi059 ceph-mon[97450]: pgmap v8379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:58:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:17.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:17.367989+0000) 2022-09-08T01:58:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:17.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:17.314683+0000) 2022-09-08T01:58:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:17 smithi055 ceph-mon[97033]: pgmap v8379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T01:58:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:18.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:18.368205+0000) 2022-09-08T01:58:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:18.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:18.314914+0000) 2022-09-08T01:58:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:19 smithi059 ceph-mon[97450]: pgmap v8380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T01:58:19.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:19.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:19.315114+0000) 2022-09-08T01:58:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:19.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:19.368390+0000) 2022-09-08T01:58:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:19 smithi055 ceph-mon[97033]: pgmap v8380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T01:58:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:20.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:20.315339+0000) 2022-09-08T01:58:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:20.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:20.368667+0000) 2022-09-08T01:58:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:21 smithi059 ceph-mon[97450]: pgmap v8381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:58:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:21.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:21.368956+0000) 2022-09-08T01:58:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:21.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:21.315594+0000) 2022-09-08T01:58:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:21 smithi055 ceph-mon[97033]: pgmap v8381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:58:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:22.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:22.369201+0000) 2022-09-08T01:58:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:22.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:22.315815+0000) 2022-09-08T01:58:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:23 smithi059 ceph-mon[97450]: pgmap v8382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:58:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:23.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:23.369420+0000) 2022-09-08T01:58:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:23.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:23.315968+0000) 2022-09-08T01:58:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:23 smithi055 ceph-mon[97033]: pgmap v8382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:58:24.640 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:24.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:24.369667+0000) 2022-09-08T01:58:24.641 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:24.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:24.316191+0000) 2022-09-08T01:58:24.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:58:24.638+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:58:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:25 smithi059 ceph-mon[97450]: pgmap v8383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:58:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:25.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:25.369907+0000) 2022-09-08T01:58:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:25.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:25.316372+0000) 2022-09-08T01:58:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:25 smithi055 ceph-mon[97033]: pgmap v8383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T01:58:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:26.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:26.370135+0000) 2022-09-08T01:58:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:26.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:26.316597+0000) 2022-09-08T01:58:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:27 smithi059 ceph-mon[97450]: pgmap v8384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:58:27.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:27.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:27.316849+0000) 2022-09-08T01:58:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:27.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:27.370327+0000) 2022-09-08T01:58:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:27 smithi055 ceph-mon[97033]: pgmap v8384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T01:58:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:28.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:28.370555+0000) 2022-09-08T01:58:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:28.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:28.316986+0000) 2022-09-08T01:58:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:29 smithi059 ceph-mon[97450]: pgmap v8385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:29.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:29.370785+0000) 2022-09-08T01:58:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:29.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:29.317239+0000) 2022-09-08T01:58:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:29 smithi055 ceph-mon[97033]: pgmap v8385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:30.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:30.317481+0000) 2022-09-08T01:58:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:30.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:30.370998+0000) 2022-09-08T01:58:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:31 smithi059 ceph-mon[97450]: pgmap v8386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:58:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:31.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:31.371236+0000) 2022-09-08T01:58:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:31.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:31.317669+0000) 2022-09-08T01:58:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:31 smithi055 ceph-mon[97033]: pgmap v8386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:58:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:32.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:32.371436+0000) 2022-09-08T01:58:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:32.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:32.317879+0000) 2022-09-08T01:58:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:33 smithi059 ceph-mon[97450]: pgmap v8387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:33.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:33.371708+0000) 2022-09-08T01:58:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:33.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:33.318130+0000) 2022-09-08T01:58:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:33 smithi055 ceph-mon[97033]: pgmap v8387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:34.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:34.371864+0000) 2022-09-08T01:58:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:34.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:34.318339+0000) 2022-09-08T01:58:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:35 smithi059 ceph-mon[97450]: pgmap v8388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:35.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:35.372094+0000) 2022-09-08T01:58:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:35.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:35.318526+0000) 2022-09-08T01:58:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:35 smithi055 ceph-mon[97033]: pgmap v8388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:36.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:36.318753+0000) 2022-09-08T01:58:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:36.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:36.372362+0000) 2022-09-08T01:58:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:37.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:37.372559+0000) 2022-09-08T01:58:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:37.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:37.319013+0000) 2022-09-08T01:58:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:37 smithi055 ceph-mon[97033]: pgmap v8389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:58:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:37 smithi059 ceph-mon[97450]: pgmap v8389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:58:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:38.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:38.372726+0000) 2022-09-08T01:58:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:38.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:38.319183+0000) 2022-09-08T01:58:39.647 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:39.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:39.372859+0000) 2022-09-08T01:58:39.647 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:39.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:39.319418+0000) 2022-09-08T01:58:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:39 smithi055 ceph-mon[97033]: pgmap v8390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:39 smithi059 ceph-mon[97450]: pgmap v8390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:39.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:58:39.645+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:58:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:40.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:40.373086+0000) 2022-09-08T01:58:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:40.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:40.319616+0000) 2022-09-08T01:58:41.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:41.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:41.319858+0000) 2022-09-08T01:58:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:41.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:41.373353+0000) 2022-09-08T01:58:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:41 smithi055 ceph-mon[97033]: pgmap v8391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:58:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:41 smithi059 ceph-mon[97450]: pgmap v8391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:58:42.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:42.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:42.320066+0000) 2022-09-08T01:58:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:42.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:42.373549+0000) 2022-09-08T01:58:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:43.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:43.373780+0000) 2022-09-08T01:58:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:43.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:43.320261+0000) 2022-09-08T01:58:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:43 smithi055 ceph-mon[97033]: pgmap v8392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:43 smithi059 ceph-mon[97450]: pgmap v8392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:44.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:44.374001+0000) 2022-09-08T01:58:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:44.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:44.320457+0000) 2022-09-08T01:58:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:45.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:45.374204+0000) 2022-09-08T01:58:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:45.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:45.320686+0000) 2022-09-08T01:58:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:45 smithi055 ceph-mon[97033]: pgmap v8393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:45 smithi059 ceph-mon[97450]: pgmap v8393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:58:46.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:46.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:46.320839+0000) 2022-09-08T01:58:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:46.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:46.374431+0000) 2022-09-08T01:58:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:47.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:47.374653+0000) 2022-09-08T01:58:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:47.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:47.321107+0000) 2022-09-08T01:58:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:47 smithi055 ceph-mon[97033]: pgmap v8394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:58:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:47 smithi059 ceph-mon[97450]: pgmap v8394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:58:48.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:48.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:48.321232+0000) 2022-09-08T01:58:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:48.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:48.374869+0000) 2022-09-08T01:58:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:49 smithi055 ceph-mon[97033]: pgmap v8395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:58:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:58:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:58:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:58:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:49 smithi059 ceph-mon[97450]: pgmap v8395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:58:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:58:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:58:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:58:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:49.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:49.321450+0000) 2022-09-08T01:58:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:49.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:49.375071+0000) 2022-09-08T01:58:50.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:50.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:50.321714+0000) 2022-09-08T01:58:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:50.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:50.375292+0000) 2022-09-08T01:58:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:50 smithi055 ceph-mon[97033]: pgmap v8396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:58:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:50 smithi055 ceph-mon[97033]: pgmap v8397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.6 KiB/s rd, 3 op/s 2022-09-08T01:58:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:50 smithi059 ceph-mon[97450]: pgmap v8396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T01:58:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:50 smithi059 ceph-mon[97450]: pgmap v8397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 1.6 KiB/s rd, 3 op/s 2022-09-08T01:58:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:51.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:51.375487+0000) 2022-09-08T01:58:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:51.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:51.321924+0000) 2022-09-08T01:58:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:52.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:52.375679+0000) 2022-09-08T01:58:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:52.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:52.322173+0000) 2022-09-08T01:58:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:52 smithi055 ceph-mon[97033]: pgmap v8398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T01:58:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:52 smithi059 ceph-mon[97450]: pgmap v8398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T01:58:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:53.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:53.375903+0000) 2022-09-08T01:58:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:53.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:53.322421+0000) 2022-09-08T01:58:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:54.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:54.376117+0000) 2022-09-08T01:58:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:54.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:54.322614+0000) 2022-09-08T01:58:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:54 smithi055 ceph-mon[97033]: pgmap v8399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T01:58:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:54 smithi059 ceph-mon[97450]: pgmap v8399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T01:58:54.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:58:54.650+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:58:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:55.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:55.376318+0000) 2022-09-08T01:58:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:55.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:55.322794+0000) 2022-09-08T01:58:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:56.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:56.376505+0000) 2022-09-08T01:58:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:56.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:56.322997+0000) 2022-09-08T01:58:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:56 smithi055 ceph-mon[97033]: pgmap v8400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T01:58:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:56 smithi059 ceph-mon[97450]: pgmap v8400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T01:58:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:57.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:57.376756+0000) 2022-09-08T01:58:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:57.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:57.323234+0000) 2022-09-08T01:58:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:58.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:58.376922+0000) 2022-09-08T01:58:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:58.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:58.323485+0000) 2022-09-08T01:58:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:58:58 smithi055 ceph-mon[97033]: pgmap v8401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T01:58:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:58:58 smithi059 ceph-mon[97450]: pgmap v8401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T01:58:59.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:58:59.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:59.323697+0000) 2022-09-08T01:58:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:58:59.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:58:59.377161+0000) 2022-09-08T01:59:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:00.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:00.377401+0000) 2022-09-08T01:59:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:00.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:00.323865+0000) 2022-09-08T01:59:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:00 smithi055 ceph-mon[97033]: pgmap v8402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:59:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:00 smithi059 ceph-mon[97450]: pgmap v8402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:59:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:01.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:01.324055+0000) 2022-09-08T01:59:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:01.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:01.377568+0000) 2022-09-08T01:59:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:02.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:02.324256+0000) 2022-09-08T01:59:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:02.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:02.377778+0000) 2022-09-08T01:59:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:02 smithi055 ceph-mon[97033]: pgmap v8403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:59:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:02 smithi059 ceph-mon[97450]: pgmap v8403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:59:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:03.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:03.378015+0000) 2022-09-08T01:59:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:03.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:03.324527+0000) 2022-09-08T01:59:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:04.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:04.378179+0000) 2022-09-08T01:59:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:04.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:04.324775+0000) 2022-09-08T01:59:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:04 smithi055 ceph-mon[97033]: pgmap v8404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:59:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:04 smithi059 ceph-mon[97450]: pgmap v8404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T01:59:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:05.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:05.325012+0000) 2022-09-08T01:59:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:05.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:05.378420+0000) 2022-09-08T01:59:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:59:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:59:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T01:59:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T01:59:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:06.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:06.325165+0000) 2022-09-08T01:59:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:06.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:06.378683+0000) 2022-09-08T01:59:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:06 smithi055 ceph-mon[97033]: pgmap v8405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:59:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:06 smithi059 ceph-mon[97450]: pgmap v8405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T01:59:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:07.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:07.378904+0000) 2022-09-08T01:59:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:07.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:07.325380+0000) 2022-09-08T01:59:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:08.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:08.379135+0000) 2022-09-08T01:59:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:08.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:08.325593+0000) 2022-09-08T01:59:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:08 smithi055 ceph-mon[97033]: pgmap v8406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:59:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:08 smithi059 ceph-mon[97450]: pgmap v8406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:59:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:09.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:09.379276+0000) 2022-09-08T01:59:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:09.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:09.325782+0000) 2022-09-08T01:59:10.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:59:09.665+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:59:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:10.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:10.379482+0000) 2022-09-08T01:59:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:10.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:10.326013+0000) 2022-09-08T01:59:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:10 smithi055 ceph-mon[97033]: pgmap v8407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:59:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:10 smithi055 ceph-mon[97033]: osdmap e607: 12 total, 11 up, 11 in 2022-09-08T01:59:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:10 smithi059 ceph-mon[97450]: pgmap v8407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T01:59:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:10 smithi059 ceph-mon[97450]: osdmap e607: 12 total, 11 up, 11 in 2022-09-08T01:59:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:11.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:11.379688+0000) 2022-09-08T01:59:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:11.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:11.326170+0000) 2022-09-08T01:59:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:12.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:12.379917+0000) 2022-09-08T01:59:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:12.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:12.326384+0000) 2022-09-08T01:59:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:12 smithi055 ceph-mon[97033]: pgmap v8409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:59:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:12 smithi059 ceph-mon[97450]: pgmap v8409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:59:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:13.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:13.380139+0000) 2022-09-08T01:59:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:13.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:13.326528+0000) 2022-09-08T01:59:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:14.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:14.380377+0000) 2022-09-08T01:59:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:14.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:14.326697+0000) 2022-09-08T01:59:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:14 smithi055 ceph-mon[97033]: pgmap v8410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:59:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:14 smithi059 ceph-mon[97450]: pgmap v8410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T01:59:15.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:15.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:15.326946+0000) 2022-09-08T01:59:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:15.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:15.380599+0000) 2022-09-08T01:59:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:16.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:16.380793+0000) 2022-09-08T01:59:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:16.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:16.327195+0000) 2022-09-08T01:59:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:16 smithi055 ceph-mon[97033]: pgmap v8411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:59:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:16 smithi059 ceph-mon[97450]: pgmap v8411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:59:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:17.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:17.380982+0000) 2022-09-08T01:59:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:17.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:17.327450+0000) 2022-09-08T01:59:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:18 smithi059 ceph-mon[97450]: pgmap v8412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:59:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:18 smithi059 ceph-mon[97450]: osdmap e608: 12 total, 11 up, 11 in 2022-09-08T01:59:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:18.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:18.381165+0000) 2022-09-08T01:59:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:18.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:18.327597+0000) 2022-09-08T01:59:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:18 smithi055 ceph-mon[97033]: pgmap v8412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T01:59:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:18 smithi055 ceph-mon[97033]: osdmap e608: 12 total, 11 up, 11 in 2022-09-08T01:59:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:19.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:19.381363+0000) 2022-09-08T01:59:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:19.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:19.327700+0000) 2022-09-08T01:59:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:20 smithi059 ceph-mon[97450]: pgmap v8414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 211 B/s wr, 6 op/s 2022-09-08T01:59:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:20.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:20.381556+0000) 2022-09-08T01:59:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:20.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:20.327921+0000) 2022-09-08T01:59:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:20 smithi055 ceph-mon[97033]: pgmap v8414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.3 KiB/s rd, 211 B/s wr, 6 op/s 2022-09-08T01:59:21.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:21.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:21.328144+0000) 2022-09-08T01:59:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:21.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:21.381728+0000) 2022-09-08T01:59:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:22 smithi059 ceph-mon[97450]: pgmap v8415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:59:22.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:22.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:22.328374+0000) 2022-09-08T01:59:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:22.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:22.381875+0000) 2022-09-08T01:59:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:22 smithi055 ceph-mon[97033]: pgmap v8415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:59:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:23.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:23.382155+0000) 2022-09-08T01:59:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:23.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:23.328568+0000) 2022-09-08T01:59:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:24 smithi059 ceph-mon[97450]: pgmap v8416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:59:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:24.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:24.328769+0000) 2022-09-08T01:59:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:24.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:24.382439+0000) 2022-09-08T01:59:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:24 smithi055 ceph-mon[97033]: pgmap v8416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T01:59:25.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:59:24.673+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:59:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:25.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:25.329002+0000) 2022-09-08T01:59:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:25.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:25.382631+0000) 2022-09-08T01:59:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:26 smithi059 ceph-mon[97450]: pgmap v8417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:59:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:26.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:26.382906+0000) 2022-09-08T01:59:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:26.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:26.329207+0000) 2022-09-08T01:59:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:26 smithi055 ceph-mon[97033]: pgmap v8417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:59:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:27.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:27.383144+0000) 2022-09-08T01:59:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:27.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:27.329410+0000) 2022-09-08T01:59:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:28 smithi059 ceph-mon[97450]: pgmap v8418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:59:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:28.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:28.383354+0000) 2022-09-08T01:59:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:28.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:28.329624+0000) 2022-09-08T01:59:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:28 smithi055 ceph-mon[97033]: pgmap v8418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T01:59:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:29.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:29.383496+0000) 2022-09-08T01:59:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:29.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:29.329894+0000) 2022-09-08T01:59:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:30 smithi059 ceph-mon[97450]: pgmap v8419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T01:59:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:30.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:30.330072+0000) 2022-09-08T01:59:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:30.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:30.383732+0000) 2022-09-08T01:59:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:30 smithi055 ceph-mon[97033]: pgmap v8419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T01:59:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:31.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:31.383861+0000) 2022-09-08T01:59:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:31.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:31.330274+0000) 2022-09-08T01:59:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:32.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:32.384080+0000) 2022-09-08T01:59:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:32.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:32.330521+0000) 2022-09-08T01:59:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:32 smithi055 ceph-mon[97033]: pgmap v8420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:59:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:32 smithi059 ceph-mon[97450]: pgmap v8420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:59:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:33.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:33.384327+0000) 2022-09-08T01:59:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:33.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:33.330723+0000) 2022-09-08T01:59:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:34.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:34.384576+0000) 2022-09-08T01:59:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:34.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:34.330835+0000) 2022-09-08T01:59:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:34 smithi055 ceph-mon[97033]: pgmap v8421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:59:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:34 smithi059 ceph-mon[97450]: pgmap v8421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:59:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:35.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:35.384822+0000) 2022-09-08T01:59:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:35.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:35.331067+0000) 2022-09-08T01:59:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:36.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:36.384948+0000) 2022-09-08T01:59:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:36.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:36.331309+0000) 2022-09-08T01:59:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:36 smithi055 ceph-mon[97033]: pgmap v8422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:59:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:36 smithi059 ceph-mon[97450]: pgmap v8422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:59:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:37.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:37.385114+0000) 2022-09-08T01:59:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:37.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:37.331545+0000) 2022-09-08T01:59:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:38.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:38.385265+0000) 2022-09-08T01:59:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:38.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:38.331679+0000) 2022-09-08T01:59:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:38 smithi055 ceph-mon[97033]: pgmap v8423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:59:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:38 smithi059 ceph-mon[97450]: pgmap v8423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:59:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:39.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:39.385513+0000) 2022-09-08T01:59:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:39.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:39.331885+0000) 2022-09-08T01:59:40.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:59:39.689+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:59:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:40.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:40.332031+0000) 2022-09-08T01:59:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:40.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:40.385771+0000) 2022-09-08T01:59:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:40 smithi055 ceph-mon[97033]: pgmap v8424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:59:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:40 smithi059 ceph-mon[97450]: pgmap v8424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:59:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:41.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:41.385919+0000) 2022-09-08T01:59:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:41.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:41.332237+0000) 2022-09-08T01:59:42.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:42.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:42.332463+0000) 2022-09-08T01:59:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:42.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:42.386107+0000) 2022-09-08T01:59:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:42 smithi055 ceph-mon[97033]: pgmap v8425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:59:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:42 smithi059 ceph-mon[97450]: pgmap v8425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:59:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:43.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:43.386347+0000) 2022-09-08T01:59:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:43.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:43.332724+0000) 2022-09-08T01:59:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:44.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:44.386589+0000) 2022-09-08T01:59:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:44.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:44.332963+0000) 2022-09-08T01:59:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:44 smithi055 ceph-mon[97033]: pgmap v8426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:59:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:44 smithi059 ceph-mon[97450]: pgmap v8426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:59:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:45.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:45.386808+0000) 2022-09-08T01:59:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:45.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:45.333223+0000) 2022-09-08T01:59:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:46.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:46.387008+0000) 2022-09-08T01:59:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:46.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:46.333470+0000) 2022-09-08T01:59:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:46 smithi055 ceph-mon[97033]: pgmap v8427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:59:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:46 smithi059 ceph-mon[97450]: pgmap v8427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T01:59:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:47.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:47.387200+0000) 2022-09-08T01:59:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:47.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:47.333735+0000) 2022-09-08T01:59:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:48.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:48.387443+0000) 2022-09-08T01:59:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:48.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:48.333971+0000) 2022-09-08T01:59:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:48 smithi055 ceph-mon[97033]: pgmap v8428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:59:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:48 smithi059 ceph-mon[97450]: pgmap v8428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:59:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:59:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:59:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:59:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T01:59:49.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T01:59:49.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T01:59:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:49.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:49.387710+0000) 2022-09-08T01:59:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:49.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:49.334193+0000) 2022-09-08T01:59:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:50.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:50.387961+0000) 2022-09-08T01:59:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:50.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:50.334430+0000) 2022-09-08T01:59:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:50 smithi055 ceph-mon[97033]: pgmap v8429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:59:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:59:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:50 smithi059 ceph-mon[97450]: pgmap v8429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T01:59:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T01:59:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:51.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:51.388165+0000) 2022-09-08T01:59:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:51.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:51.334583+0000) 2022-09-08T01:59:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:51 smithi055 ceph-mon[97033]: pgmap v8430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:59:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:51 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T01:59:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:51 smithi059 ceph-mon[97450]: pgmap v8430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:59:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:51 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T01:59:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:52.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:52.388388+0000) 2022-09-08T01:59:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:52.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:52.334724+0000) 2022-09-08T01:59:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:53.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:53.388628+0000) 2022-09-08T01:59:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:53.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:53.334966+0000) 2022-09-08T01:59:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:53 smithi055 ceph-mon[97033]: pgmap v8431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:59:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:53 smithi059 ceph-mon[97450]: pgmap v8431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:59:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:54.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:54.388842+0000) 2022-09-08T01:59:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:54.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:54.335098+0000) 2022-09-08T01:59:55.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 01:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T01:59:54.695+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T01:59:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:55.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:55.388990+0000) 2022-09-08T01:59:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:55.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:55.335302+0000) 2022-09-08T01:59:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:55 smithi055 ceph-mon[97033]: pgmap v8432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:59:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:55 smithi059 ceph-mon[97450]: pgmap v8432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:59:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:56.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:56.389238+0000) 2022-09-08T01:59:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:56.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:56.335534+0000) 2022-09-08T01:59:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:57.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:57.389512+0000) 2022-09-08T01:59:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:57.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:57.335694+0000) 2022-09-08T01:59:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:57 smithi055 ceph-mon[97033]: pgmap v8433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:59:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:57 smithi059 ceph-mon[97450]: pgmap v8433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:59:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:58.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:58.389758+0000) 2022-09-08T01:59:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:58.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:58.335834+0000) 2022-09-08T01:59:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 01:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T01:59:59.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:59.389998+0000) 2022-09-08T01:59:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 01:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T01:59:59.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T00:59:59.336008+0000) 2022-09-08T01:59:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 01:59:59 smithi055 ceph-mon[97033]: pgmap v8434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T01:59:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 01:59:59 smithi059 ceph-mon[97450]: pgmap v8434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T02:00:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:00.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:00.390151+0000) 2022-09-08T02:00:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:00.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:00.336154+0000) 2022-09-08T02:00:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T02:00:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T02:00:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T02:00:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T02:00:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T02:00:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T02:00:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:01.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:01.390308+0000) 2022-09-08T02:00:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:01.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:01.336305+0000) 2022-09-08T02:00:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:01 smithi055 ceph-mon[97033]: pgmap v8435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T02:00:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:01 smithi059 ceph-mon[97450]: pgmap v8435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T02:00:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:02.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:02.390516+0000) 2022-09-08T02:00:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:02.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:02.336524+0000) 2022-09-08T02:00:03.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:03.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:03.336764+0000) 2022-09-08T02:00:03.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:03.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:03.390747+0000) 2022-09-08T02:00:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:03 smithi055 ceph-mon[97033]: pgmap v8436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:00:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:03 smithi059 ceph-mon[97450]: pgmap v8436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:00:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:04.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:04.390882+0000) 2022-09-08T02:00:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:04.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:04.337040+0000) 2022-09-08T02:00:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:05.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:05.391068+0000) 2022-09-08T02:00:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:05.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:05.337269+0000) 2022-09-08T02:00:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:05 smithi055 ceph-mon[97033]: pgmap v8437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:00:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:00:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:00:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:05 smithi059 ceph-mon[97450]: pgmap v8437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:00:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:00:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:00:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:06.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:06.337506+0000) 2022-09-08T02:00:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:06.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:06.391272+0000) 2022-09-08T02:00:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:07.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:07.391516+0000) 2022-09-08T02:00:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:07.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:07.337809+0000) 2022-09-08T02:00:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:07 smithi055 ceph-mon[97033]: pgmap v8438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:00:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:07 smithi059 ceph-mon[97450]: pgmap v8438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:00:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:08.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:08.337948+0000) 2022-09-08T02:00:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:08.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:08.391734+0000) 2022-09-08T02:00:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:09.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:09.391865+0000) 2022-09-08T02:00:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:09.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:09.338160+0000) 2022-09-08T02:00:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:09 smithi055 ceph-mon[97033]: pgmap v8439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:00:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:09 smithi059 ceph-mon[97450]: pgmap v8439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:00:10.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:00:09.711+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:00:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:10.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:10.392048+0000) 2022-09-08T02:00:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:10.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:10.338321+0000) 2022-09-08T02:00:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:10 smithi055 ceph-mon[97033]: osdmap e609: 12 total, 11 up, 11 in 2022-09-08T02:00:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:10 smithi059 ceph-mon[97450]: osdmap e609: 12 total, 11 up, 11 in 2022-09-08T02:00:11.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:11.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:11.338511+0000) 2022-09-08T02:00:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:11.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:11.392298+0000) 2022-09-08T02:00:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:11 smithi055 ceph-mon[97033]: pgmap v8441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:00:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:11 smithi059 ceph-mon[97450]: pgmap v8441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:00:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:12.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:12.392528+0000) 2022-09-08T02:00:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:12.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:12.338743+0000) 2022-09-08T02:00:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:13.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:13.392784+0000) 2022-09-08T02:00:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:13.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:13.338919+0000) 2022-09-08T02:00:13.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:13 smithi055 ceph-mon[97033]: pgmap v8442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:00:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:13 smithi059 ceph-mon[97450]: pgmap v8442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:00:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:14.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:14.392972+0000) 2022-09-08T02:00:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:14.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:14.339078+0000) 2022-09-08T02:00:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:15.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:15.393096+0000) 2022-09-08T02:00:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:15.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:15.339364+0000) 2022-09-08T02:00:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:15 smithi055 ceph-mon[97033]: pgmap v8443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:00:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:15 smithi059 ceph-mon[97450]: pgmap v8443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:00:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:16.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:16.393322+0000) 2022-09-08T02:00:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:16.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:16.339598+0000) 2022-09-08T02:00:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:17.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:17.339859+0000) 2022-09-08T02:00:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:17.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:17.393594+0000) 2022-09-08T02:00:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:17 smithi055 ceph-mon[97033]: pgmap v8444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:00:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:17 smithi055 ceph-mon[97033]: osdmap e610: 12 total, 11 up, 11 in 2022-09-08T02:00:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:17 smithi059 ceph-mon[97450]: pgmap v8444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:00:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:17 smithi059 ceph-mon[97450]: osdmap e610: 12 total, 11 up, 11 in 2022-09-08T02:00:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:18.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:18.393818+0000) 2022-09-08T02:00:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:18.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:18.340115+0000) 2022-09-08T02:00:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:19.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:19.394058+0000) 2022-09-08T02:00:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:19.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:19.340255+0000) 2022-09-08T02:00:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:19 smithi055 ceph-mon[97033]: pgmap v8446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T02:00:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:19 smithi059 ceph-mon[97450]: pgmap v8446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T02:00:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:20.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:20.394306+0000) 2022-09-08T02:00:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:20.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:20.340458+0000) 2022-09-08T02:00:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:21.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:21.394493+0000) 2022-09-08T02:00:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:21.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:21.340665+0000) 2022-09-08T02:00:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:21 smithi055 ceph-mon[97033]: pgmap v8447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:00:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:21 smithi059 ceph-mon[97450]: pgmap v8447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:00:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:22.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:22.394680+0000) 2022-09-08T02:00:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:22.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:22.340842+0000) 2022-09-08T02:00:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:23.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:23.394909+0000) 2022-09-08T02:00:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:23.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:23.341076+0000) 2022-09-08T02:00:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:23 smithi055 ceph-mon[97033]: pgmap v8448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:00:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:23 smithi059 ceph-mon[97450]: pgmap v8448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:00:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:24.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:24.395097+0000) 2022-09-08T02:00:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:24.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:24.341305+0000) 2022-09-08T02:00:25.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:00:24.714+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:00:25.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:25.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:25.341529+0000) 2022-09-08T02:00:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:25.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:25.395304+0000) 2022-09-08T02:00:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:25 smithi055 ceph-mon[97033]: pgmap v8449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:00:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:25 smithi059 ceph-mon[97450]: pgmap v8449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:00:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:26.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:26.395526+0000) 2022-09-08T02:00:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:26.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:26.341693+0000) 2022-09-08T02:00:27.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:27.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:27.341911+0000) 2022-09-08T02:00:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:27.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:27.395677+0000) 2022-09-08T02:00:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:27 smithi055 ceph-mon[97033]: pgmap v8450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:00:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:27 smithi059 ceph-mon[97450]: pgmap v8450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:00:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:28.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:28.395836+0000) 2022-09-08T02:00:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:28.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:28.342095+0000) 2022-09-08T02:00:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:29.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:29.342334+0000) 2022-09-08T02:00:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:29.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:29.396077+0000) 2022-09-08T02:00:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:29 smithi055 ceph-mon[97033]: pgmap v8451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:00:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:29 smithi059 ceph-mon[97450]: pgmap v8451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:00:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:30.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:30.396286+0000) 2022-09-08T02:00:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:30.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:30.342523+0000) 2022-09-08T02:00:31.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:31.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:31.342741+0000) 2022-09-08T02:00:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:31.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:31.396528+0000) 2022-09-08T02:00:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:31 smithi055 ceph-mon[97033]: pgmap v8452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:00:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:31 smithi059 ceph-mon[97450]: pgmap v8452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:00:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:32.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:32.396697+0000) 2022-09-08T02:00:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:32.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:32.342983+0000) 2022-09-08T02:00:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:33.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:33.343174+0000) 2022-09-08T02:00:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:33.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:33.396915+0000) 2022-09-08T02:00:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:33 smithi055 ceph-mon[97033]: pgmap v8453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:00:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:33 smithi059 ceph-mon[97450]: pgmap v8453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:00:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:34.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:34.343417+0000) 2022-09-08T02:00:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:34.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:34.397147+0000) 2022-09-08T02:00:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:35.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:35.397356+0000) 2022-09-08T02:00:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:35.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:35.343654+0000) 2022-09-08T02:00:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:35 smithi055 ceph-mon[97033]: pgmap v8454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:00:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:35 smithi059 ceph-mon[97450]: pgmap v8454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:00:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:36.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:36.397603+0000) 2022-09-08T02:00:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:36.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:36.343880+0000) 2022-09-08T02:00:37.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:37.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:37.344041+0000) 2022-09-08T02:00:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:37.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:37.397821+0000) 2022-09-08T02:00:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:37 smithi055 ceph-mon[97033]: pgmap v8455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:00:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:37 smithi059 ceph-mon[97450]: pgmap v8455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:00:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:38.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:38.397972+0000) 2022-09-08T02:00:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:38.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:38.344263+0000) 2022-09-08T02:00:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:39.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:39.398183+0000) 2022-09-08T02:00:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:39.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:39.344513+0000) 2022-09-08T02:00:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:39 smithi055 ceph-mon[97033]: pgmap v8456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:00:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:39 smithi059 ceph-mon[97450]: pgmap v8456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:00:40.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:00:39.719+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:00:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:40.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:40.398348+0000) 2022-09-08T02:00:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:40.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:40.344675+0000) 2022-09-08T02:00:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:41.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:41.398508+0000) 2022-09-08T02:00:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:41.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:41.344916+0000) 2022-09-08T02:00:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:41 smithi055 ceph-mon[97033]: pgmap v8457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:00:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:41 smithi059 ceph-mon[97450]: pgmap v8457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:00:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:42.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:42.398759+0000) 2022-09-08T02:00:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:42.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:42.345156+0000) 2022-09-08T02:00:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:43.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:43.398999+0000) 2022-09-08T02:00:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:43.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:43.345378+0000) 2022-09-08T02:00:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:43 smithi055 ceph-mon[97033]: pgmap v8458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:00:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:43 smithi059 ceph-mon[97450]: pgmap v8458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:00:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:44.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:44.399207+0000) 2022-09-08T02:00:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:44.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:44.345674+0000) 2022-09-08T02:00:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:45.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:45.399434+0000) 2022-09-08T02:00:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:45.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:45.345823+0000) 2022-09-08T02:00:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:45 smithi055 ceph-mon[97033]: pgmap v8459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:00:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:45 smithi059 ceph-mon[97450]: pgmap v8459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:00:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:46.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:46.399699+0000) 2022-09-08T02:00:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:46.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:46.345995+0000) 2022-09-08T02:00:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:47.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:47.399865+0000) 2022-09-08T02:00:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:47.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:47.346265+0000) 2022-09-08T02:00:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:47 smithi055 ceph-mon[97033]: pgmap v8460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:00:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:47 smithi059 ceph-mon[97450]: pgmap v8460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:00:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:48.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:48.400092+0000) 2022-09-08T02:00:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:48.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:48.346438+0000) 2022-09-08T02:00:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:49.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:49.400326+0000) 2022-09-08T02:00:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:49.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:49.346605+0000) 2022-09-08T02:00:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:49 smithi055 ceph-mon[97033]: pgmap v8461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:00:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:49 smithi059 ceph-mon[97450]: pgmap v8461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:00:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:50.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:50.400567+0000) 2022-09-08T02:00:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:50.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:50.346842+0000) 2022-09-08T02:00:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:00:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:00:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:00:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:00:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:00:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:00:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:00:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:00:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:51.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:51.400806+0000) 2022-09-08T02:00:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:51.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:51.347067+0000) 2022-09-08T02:00:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:51 smithi055 ceph-mon[97033]: pgmap v8462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:00:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:51 smithi059 ceph-mon[97450]: pgmap v8462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:00:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:52.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:52.401037+0000) 2022-09-08T02:00:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:52.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:52.347305+0000) 2022-09-08T02:00:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:53.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:53.401266+0000) 2022-09-08T02:00:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:53.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:53.347552+0000) 2022-09-08T02:00:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:53 smithi055 ceph-mon[97033]: pgmap v8463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:00:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:53 smithi059 ceph-mon[97450]: pgmap v8463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:00:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:54.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:54.401489+0000) 2022-09-08T02:00:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:54.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:54.347788+0000) 2022-09-08T02:00:55.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:00:54.734+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:00:55.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:55.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:55.348060+0000) 2022-09-08T02:00:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:55.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:55.401674+0000) 2022-09-08T02:00:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:55 smithi055 ceph-mon[97033]: pgmap v8464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:00:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:55 smithi059 ceph-mon[97450]: pgmap v8464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:00:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:56.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:56.401826+0000) 2022-09-08T02:00:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:56.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:56.348265+0000) 2022-09-08T02:00:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:57.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:57.402047+0000) 2022-09-08T02:00:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:57.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:57.348493+0000) 2022-09-08T02:00:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:57 smithi055 ceph-mon[97033]: pgmap v8465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:00:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:57 smithi059 ceph-mon[97450]: pgmap v8465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:00:58.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:58.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:58.348739+0000) 2022-09-08T02:00:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:58.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:58.402300+0000) 2022-09-08T02:00:59.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:00:59.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:59.348982+0000) 2022-09-08T02:00:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:00:59.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:00:59.402499+0000) 2022-09-08T02:00:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:00:59 smithi055 ceph-mon[97033]: pgmap v8466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:00:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:00:59 smithi059 ceph-mon[97450]: pgmap v8466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:00.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:00.349221+0000) 2022-09-08T02:01:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:00.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:00.402732+0000) 2022-09-08T02:01:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:01.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:01.349445+0000) 2022-09-08T02:01:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:01.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:01.402951+0000) 2022-09-08T02:01:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:01 smithi055 ceph-mon[97033]: pgmap v8467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:01 smithi059 ceph-mon[97450]: pgmap v8467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:02.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:02.403173+0000) 2022-09-08T02:01:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:02.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:02.349697+0000) 2022-09-08T02:01:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:03.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:03.403373+0000) 2022-09-08T02:01:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:03.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:03.349942+0000) 2022-09-08T02:01:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:03 smithi055 ceph-mon[97033]: pgmap v8468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:01:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:03 smithi059 ceph-mon[97450]: pgmap v8468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:01:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:04.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:04.403513+0000) 2022-09-08T02:01:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:04.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:04.350164+0000) 2022-09-08T02:01:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:05.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:05.403684+0000) 2022-09-08T02:01:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:05.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:05.350393+0000) 2022-09-08T02:01:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:05 smithi055 ceph-mon[97033]: pgmap v8469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:01:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:01:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:05 smithi059 ceph-mon[97450]: pgmap v8469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:01:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:01:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:06.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:06.403902+0000) 2022-09-08T02:01:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:06.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:06.350604+0000) 2022-09-08T02:01:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:07.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:07.404104+0000) 2022-09-08T02:01:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:07.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:07.350874+0000) 2022-09-08T02:01:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:07 smithi055 ceph-mon[97033]: pgmap v8470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:01:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:07 smithi059 ceph-mon[97450]: pgmap v8470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:01:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:08.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:08.404351+0000) 2022-09-08T02:01:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:08.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:08.351056+0000) 2022-09-08T02:01:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:09.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:09.404583+0000) 2022-09-08T02:01:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:09.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:09.351310+0000) 2022-09-08T02:01:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:09 smithi055 ceph-mon[97033]: pgmap v8471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:01:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:09 smithi059 ceph-mon[97450]: pgmap v8471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:01:10.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:01:09.749+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:01:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:10.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:10.404860+0000) 2022-09-08T02:01:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:10.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:10.351533+0000) 2022-09-08T02:01:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:10 smithi059 ceph-mon[97450]: osdmap e611: 12 total, 11 up, 11 in 2022-09-08T02:01:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:10 smithi055 ceph-mon[97033]: osdmap e611: 12 total, 11 up, 11 in 2022-09-08T02:01:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:11.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:11.405127+0000) 2022-09-08T02:01:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:11.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:11.351691+0000) 2022-09-08T02:01:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:11 smithi059 ceph-mon[97450]: pgmap v8473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:01:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:11 smithi055 ceph-mon[97033]: pgmap v8473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:01:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:12.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:12.405343+0000) 2022-09-08T02:01:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:12.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:12.351863+0000) 2022-09-08T02:01:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:13.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:13.405561+0000) 2022-09-08T02:01:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:13.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:13.352107+0000) 2022-09-08T02:01:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:13 smithi059 ceph-mon[97450]: pgmap v8474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:01:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:13 smithi055 ceph-mon[97033]: pgmap v8474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:01:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:14.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:14.405769+0000) 2022-09-08T02:01:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:14.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:14.352333+0000) 2022-09-08T02:01:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:15.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:15.405967+0000) 2022-09-08T02:01:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:15.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:15.352592+0000) 2022-09-08T02:01:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:15 smithi059 ceph-mon[97450]: pgmap v8475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:01:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:15 smithi055 ceph-mon[97033]: pgmap v8475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:01:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:16.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:16.406202+0000) 2022-09-08T02:01:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:16.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:16.352764+0000) 2022-09-08T02:01:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:17.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:17.406428+0000) 2022-09-08T02:01:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:17.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:17.352936+0000) 2022-09-08T02:01:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:17 smithi059 ceph-mon[97450]: pgmap v8476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:01:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:17 smithi059 ceph-mon[97450]: osdmap e612: 12 total, 11 up, 11 in 2022-09-08T02:01:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:17 smithi055 ceph-mon[97033]: pgmap v8476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:01:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:17 smithi055 ceph-mon[97033]: osdmap e612: 12 total, 11 up, 11 in 2022-09-08T02:01:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:18.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:18.406723+0000) 2022-09-08T02:01:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:18.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:18.353075+0000) 2022-09-08T02:01:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:19.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:19.406891+0000) 2022-09-08T02:01:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:19.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:19.353274+0000) 2022-09-08T02:01:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:19 smithi059 ceph-mon[97450]: pgmap v8478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T02:01:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:19 smithi055 ceph-mon[97033]: pgmap v8478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T02:01:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:20.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:20.407030+0000) 2022-09-08T02:01:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:20.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:20.353451+0000) 2022-09-08T02:01:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:21.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:21.407252+0000) 2022-09-08T02:01:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:21.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:21.353733+0000) 2022-09-08T02:01:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:21 smithi059 ceph-mon[97450]: pgmap v8479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:01:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:21 smithi055 ceph-mon[97033]: pgmap v8479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:01:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:22.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:22.407489+0000) 2022-09-08T02:01:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:22.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:22.353921+0000) 2022-09-08T02:01:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:23.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:23.407670+0000) 2022-09-08T02:01:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:23.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:23.354112+0000) 2022-09-08T02:01:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:23 smithi059 ceph-mon[97450]: pgmap v8480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:01:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:23 smithi055 ceph-mon[97033]: pgmap v8480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:01:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:24.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:24.407841+0000) 2022-09-08T02:01:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:24.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:24.354319+0000) 2022-09-08T02:01:25.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:01:24.755+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:01:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:25.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:25.408074+0000) 2022-09-08T02:01:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:25.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:25.354579+0000) 2022-09-08T02:01:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:25 smithi059 ceph-mon[97450]: pgmap v8481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:01:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:25 smithi055 ceph-mon[97033]: pgmap v8481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:01:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:26.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:26.354802+0000) 2022-09-08T02:01:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:26.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:26.408260+0000) 2022-09-08T02:01:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:27.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:27.408492+0000) 2022-09-08T02:01:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:27.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:27.355003+0000) 2022-09-08T02:01:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:27 smithi059 ceph-mon[97450]: pgmap v8482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:01:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:27 smithi055 ceph-mon[97033]: pgmap v8482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:01:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:28.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:28.408726+0000) 2022-09-08T02:01:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:28.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:28.355187+0000) 2022-09-08T02:01:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:29.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:29.355347+0000) 2022-09-08T02:01:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:29.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:29.408932+0000) 2022-09-08T02:01:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:29 smithi059 ceph-mon[97450]: pgmap v8483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:01:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:29 smithi055 ceph-mon[97033]: pgmap v8483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:01:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:30.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:30.355553+0000) 2022-09-08T02:01:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:30.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:30.409132+0000) 2022-09-08T02:01:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:31.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:31.355721+0000) 2022-09-08T02:01:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:31.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:31.409361+0000) 2022-09-08T02:01:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:31 smithi059 ceph-mon[97450]: pgmap v8484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:31 smithi055 ceph-mon[97033]: pgmap v8484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:32.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:32.355892+0000) 2022-09-08T02:01:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:32.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:32.409599+0000) 2022-09-08T02:01:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:33.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:33.356073+0000) 2022-09-08T02:01:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:33.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:33.409840+0000) 2022-09-08T02:01:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:33 smithi059 ceph-mon[97450]: pgmap v8485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:01:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:33 smithi055 ceph-mon[97033]: pgmap v8485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:01:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:34.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:34.356275+0000) 2022-09-08T02:01:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:34.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:34.409980+0000) 2022-09-08T02:01:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:35.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:35.410224+0000) 2022-09-08T02:01:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:35.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:35.356523+0000) 2022-09-08T02:01:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:35 smithi055 ceph-mon[97033]: pgmap v8486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:35 smithi059 ceph-mon[97450]: pgmap v8486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:36.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:36.410432+0000) 2022-09-08T02:01:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:36.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:36.356762+0000) 2022-09-08T02:01:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:37.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:37.410677+0000) 2022-09-08T02:01:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:37.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:37.356990+0000) 2022-09-08T02:01:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:37 smithi055 ceph-mon[97033]: pgmap v8487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:01:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:37 smithi059 ceph-mon[97450]: pgmap v8487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:01:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:38.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:38.410817+0000) 2022-09-08T02:01:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:38.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:38.357135+0000) 2022-09-08T02:01:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:39.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:39.410970+0000) 2022-09-08T02:01:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:39.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:39.357334+0000) 2022-09-08T02:01:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:39 smithi055 ceph-mon[97033]: pgmap v8488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:39 smithi059 ceph-mon[97450]: pgmap v8488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:40.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:01:39.771+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:01:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:40.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:40.411187+0000) 2022-09-08T02:01:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:40.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:40.357543+0000) 2022-09-08T02:01:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:41.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:41.411448+0000) 2022-09-08T02:01:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:41.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:41.357793+0000) 2022-09-08T02:01:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:41 smithi055 ceph-mon[97033]: pgmap v8489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:41 smithi059 ceph-mon[97450]: pgmap v8489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:42.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:42.411676+0000) 2022-09-08T02:01:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:42.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:42.358037+0000) 2022-09-08T02:01:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:43.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:43.411899+0000) 2022-09-08T02:01:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:43.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:43.358306+0000) 2022-09-08T02:01:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:43 smithi055 ceph-mon[97033]: pgmap v8490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:01:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:43 smithi059 ceph-mon[97450]: pgmap v8490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:01:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:44.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:44.412031+0000) 2022-09-08T02:01:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:44.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:44.358520+0000) 2022-09-08T02:01:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:45.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:45.412197+0000) 2022-09-08T02:01:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:45.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:45.358686+0000) 2022-09-08T02:01:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:45 smithi055 ceph-mon[97033]: pgmap v8491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:45 smithi059 ceph-mon[97450]: pgmap v8491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:46.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:46.412370+0000) 2022-09-08T02:01:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:46.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:46.358847+0000) 2022-09-08T02:01:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:47.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:47.412573+0000) 2022-09-08T02:01:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:47.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:47.359013+0000) 2022-09-08T02:01:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:47 smithi055 ceph-mon[97033]: pgmap v8492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:01:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:47 smithi059 ceph-mon[97450]: pgmap v8492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:01:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:48.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:48.412701+0000) 2022-09-08T02:01:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:48.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:48.359140+0000) 2022-09-08T02:01:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:49.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:49.412924+0000) 2022-09-08T02:01:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:49.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:49.359289+0000) 2022-09-08T02:01:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:49 smithi055 ceph-mon[97033]: pgmap v8493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:50.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:49 smithi059 ceph-mon[97450]: pgmap v8493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:50.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:50.413169+0000) 2022-09-08T02:01:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:50.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:50.359502+0000) 2022-09-08T02:01:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:01:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:01:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:01:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:01:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:01:50.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:01:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:51.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:51.413389+0000) 2022-09-08T02:01:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:51.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:51.359649+0000) 2022-09-08T02:01:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:51 smithi055 ceph-mon[97033]: pgmap v8494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:01:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:51 smithi059 ceph-mon[97450]: pgmap v8494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:01:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:52.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:52.413616+0000) 2022-09-08T02:01:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:52.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:52.359786+0000) 2022-09-08T02:01:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:53.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:53.413873+0000) 2022-09-08T02:01:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:53.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:53.359944+0000) 2022-09-08T02:01:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:53 smithi055 ceph-mon[97033]: pgmap v8495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:01:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:53 smithi059 ceph-mon[97450]: pgmap v8495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:01:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:54.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:54.414012+0000) 2022-09-08T02:01:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:54.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:54.360155+0000) 2022-09-08T02:01:55.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:01:54.782+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:01:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:55.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:55.414200+0000) 2022-09-08T02:01:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:55.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:55.360334+0000) 2022-09-08T02:01:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:55 smithi055 ceph-mon[97033]: pgmap v8496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:55 smithi059 ceph-mon[97450]: pgmap v8496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:01:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:56.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:56.414425+0000) 2022-09-08T02:01:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:56.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:56.360519+0000) 2022-09-08T02:01:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:57.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:57.414780+0000) 2022-09-08T02:01:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:57.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:57.360675+0000) 2022-09-08T02:01:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:57 smithi055 ceph-mon[97033]: pgmap v8497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:01:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:57 smithi059 ceph-mon[97450]: pgmap v8497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:01:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:58.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:58.414944+0000) 2022-09-08T02:01:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:58.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:58.360787+0000) 2022-09-08T02:01:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:01:59.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:59.415194+0000) 2022-09-08T02:01:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:01:59.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:01:59.360917+0000) 2022-09-08T02:01:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:01:59 smithi055 ceph-mon[97033]: pgmap v8498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:01:59 smithi059 ceph-mon[97450]: pgmap v8498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:00.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:00.415485+0000) 2022-09-08T02:02:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:00.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:00.361072+0000) 2022-09-08T02:02:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:00 smithi055 ceph-mon[97033]: pgmap v8499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:00 smithi059 ceph-mon[97450]: pgmap v8499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:01.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:01.415708+0000) 2022-09-08T02:02:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:01.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:01.361215+0000) 2022-09-08T02:02:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:02.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:02.415921+0000) 2022-09-08T02:02:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:02.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:02.361416+0000) 2022-09-08T02:02:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:03.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:03.416129+0000) 2022-09-08T02:02:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:03.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:03.361582+0000) 2022-09-08T02:02:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:03 smithi055 ceph-mon[97033]: pgmap v8500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:02:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:03 smithi059 ceph-mon[97450]: pgmap v8500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:02:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:04.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:04.416378+0000) 2022-09-08T02:02:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:04.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:04.361723+0000) 2022-09-08T02:02:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:05.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:05.361852+0000) 2022-09-08T02:02:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:05.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:05.416602+0000) 2022-09-08T02:02:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:05 smithi055 ceph-mon[97033]: pgmap v8501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:02:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:02:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:05 smithi059 ceph-mon[97450]: pgmap v8501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:02:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:02:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:06.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:06.416758+0000) 2022-09-08T02:02:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:06.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:06.362032+0000) 2022-09-08T02:02:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:07.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:07.416985+0000) 2022-09-08T02:02:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:07.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:07.362210+0000) 2022-09-08T02:02:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:07 smithi055 ceph-mon[97033]: pgmap v8502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:02:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:07 smithi059 ceph-mon[97450]: pgmap v8502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:02:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:08.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:08.417167+0000) 2022-09-08T02:02:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:08.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:08.362350+0000) 2022-09-08T02:02:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:09.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:09.417419+0000) 2022-09-08T02:02:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:09.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:09.362491+0000) 2022-09-08T02:02:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:09 smithi055 ceph-mon[97033]: pgmap v8503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:02:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:09 smithi059 ceph-mon[97450]: pgmap v8503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:02:10.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:02:09.793+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:02:10.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:10.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:10.417632+0000) 2022-09-08T02:02:10.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:10.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:10.362632+0000) 2022-09-08T02:02:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:10 smithi055 ceph-mon[97033]: osdmap e613: 12 total, 11 up, 11 in 2022-09-08T02:02:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:10 smithi059 ceph-mon[97450]: osdmap e613: 12 total, 11 up, 11 in 2022-09-08T02:02:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:11.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:11.417877+0000) 2022-09-08T02:02:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:11.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:11.362810+0000) 2022-09-08T02:02:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:11 smithi055 ceph-mon[97033]: pgmap v8505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:02:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:11 smithi059 ceph-mon[97450]: pgmap v8505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:02:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:12.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:12.418117+0000) 2022-09-08T02:02:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:12.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:12.362971+0000) 2022-09-08T02:02:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:13.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:13.418306+0000) 2022-09-08T02:02:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:13.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:13.363087+0000) 2022-09-08T02:02:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:13 smithi055 ceph-mon[97033]: pgmap v8506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:02:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:13 smithi059 ceph-mon[97450]: pgmap v8506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:02:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:14.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:14.418470+0000) 2022-09-08T02:02:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:14.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:14.363244+0000) 2022-09-08T02:02:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:15.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:15.418732+0000) 2022-09-08T02:02:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:15.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:15.363390+0000) 2022-09-08T02:02:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:15 smithi055 ceph-mon[97033]: pgmap v8507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:02:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:15 smithi059 ceph-mon[97450]: pgmap v8507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:02:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:16.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:16.418921+0000) 2022-09-08T02:02:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:16.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:16.363590+0000) 2022-09-08T02:02:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:17.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:17.363707+0000) 2022-09-08T02:02:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:17.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:17.419104+0000) 2022-09-08T02:02:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:17 smithi055 ceph-mon[97033]: pgmap v8508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:02:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:17 smithi055 ceph-mon[97033]: osdmap e614: 12 total, 11 up, 11 in 2022-09-08T02:02:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:17 smithi059 ceph-mon[97450]: pgmap v8508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:02:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:17 smithi059 ceph-mon[97450]: osdmap e614: 12 total, 11 up, 11 in 2022-09-08T02:02:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:18.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:18.363888+0000) 2022-09-08T02:02:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:18.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:18.419335+0000) 2022-09-08T02:02:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:19.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:19.419568+0000) 2022-09-08T02:02:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:19.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:19.364062+0000) 2022-09-08T02:02:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:19 smithi055 ceph-mon[97033]: pgmap v8510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 243 B/s wr, 7 op/s 2022-09-08T02:02:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:19 smithi059 ceph-mon[97450]: pgmap v8510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 243 B/s wr, 7 op/s 2022-09-08T02:02:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:20.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:20.419821+0000) 2022-09-08T02:02:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:20.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:20.364228+0000) 2022-09-08T02:02:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:21.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:21.420045+0000) 2022-09-08T02:02:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:21.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:21.364385+0000) 2022-09-08T02:02:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:21 smithi055 ceph-mon[97033]: pgmap v8511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:02:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:21 smithi059 ceph-mon[97450]: pgmap v8511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:02:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:22.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:22.420235+0000) 2022-09-08T02:02:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:22.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:22.364629+0000) 2022-09-08T02:02:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:23.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:23.420435+0000) 2022-09-08T02:02:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:23.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:23.364809+0000) 2022-09-08T02:02:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:23 smithi055 ceph-mon[97033]: pgmap v8512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:02:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:23 smithi059 ceph-mon[97450]: pgmap v8512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:02:24.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:24.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:24.364955+0000) 2022-09-08T02:02:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:24.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:24.420679+0000) 2022-09-08T02:02:25.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:02:24.795+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:02:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:25.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:25.365085+0000) 2022-09-08T02:02:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:25.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:25.420909+0000) 2022-09-08T02:02:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:25 smithi055 ceph-mon[97033]: pgmap v8513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:02:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:25 smithi059 ceph-mon[97450]: pgmap v8513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:02:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:26.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:26.421125+0000) 2022-09-08T02:02:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:26.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:26.365215+0000) 2022-09-08T02:02:27.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:27.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:27.365419+0000) 2022-09-08T02:02:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:27.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:27.421336+0000) 2022-09-08T02:02:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:27 smithi055 ceph-mon[97033]: pgmap v8514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:02:27.811 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:27 smithi059 ceph-mon[97450]: pgmap v8514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:02:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:28.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:28.421559+0000) 2022-09-08T02:02:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:28.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:28.365577+0000) 2022-09-08T02:02:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:29.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:29.421797+0000) 2022-09-08T02:02:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:29.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:29.365707+0000) 2022-09-08T02:02:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:29 smithi055 ceph-mon[97033]: pgmap v8515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:02:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:29 smithi059 ceph-mon[97450]: pgmap v8515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:02:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:30.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:30.422035+0000) 2022-09-08T02:02:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:30.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:30.365834+0000) 2022-09-08T02:02:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:31.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:31.422207+0000) 2022-09-08T02:02:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:31.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:31.366002+0000) 2022-09-08T02:02:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:31 smithi055 ceph-mon[97033]: pgmap v8516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:31 smithi059 ceph-mon[97450]: pgmap v8516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:32.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:32.422400+0000) 2022-09-08T02:02:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:32.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:32.366159+0000) 2022-09-08T02:02:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:33.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:33.366322+0000) 2022-09-08T02:02:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:33.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:33.422595+0000) 2022-09-08T02:02:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:33 smithi055 ceph-mon[97033]: pgmap v8517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:02:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:33 smithi059 ceph-mon[97450]: pgmap v8517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:02:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:34.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:34.366484+0000) 2022-09-08T02:02:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:34.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:34.422811+0000) 2022-09-08T02:02:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:35.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:35.423027+0000) 2022-09-08T02:02:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:35.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:35.366695+0000) 2022-09-08T02:02:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:35 smithi055 ceph-mon[97033]: pgmap v8518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:35 smithi059 ceph-mon[97450]: pgmap v8518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:36.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:36.423166+0000) 2022-09-08T02:02:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:36.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:36.366883+0000) 2022-09-08T02:02:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:37.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:37.423324+0000) 2022-09-08T02:02:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:37.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:37.367059+0000) 2022-09-08T02:02:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:37 smithi055 ceph-mon[97033]: pgmap v8519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:02:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:37 smithi059 ceph-mon[97450]: pgmap v8519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:02:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:38.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:38.423549+0000) 2022-09-08T02:02:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:38.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:38.367234+0000) 2022-09-08T02:02:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:39.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:39.423781+0000) 2022-09-08T02:02:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:39.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:39.367435+0000) 2022-09-08T02:02:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:39 smithi055 ceph-mon[97033]: pgmap v8520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:39 smithi059 ceph-mon[97450]: pgmap v8520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:40.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:02:39.810+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:02:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:40.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:40.424024+0000) 2022-09-08T02:02:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:40.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:40.367618+0000) 2022-09-08T02:02:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:41.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:41.424246+0000) 2022-09-08T02:02:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:41.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:41.367866+0000) 2022-09-08T02:02:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:41 smithi055 ceph-mon[97033]: pgmap v8521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:41 smithi059 ceph-mon[97450]: pgmap v8521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:42.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:42.424475+0000) 2022-09-08T02:02:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:42.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:42.368034+0000) 2022-09-08T02:02:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:43.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:43.424710+0000) 2022-09-08T02:02:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:43.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:43.368264+0000) 2022-09-08T02:02:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:43 smithi055 ceph-mon[97033]: pgmap v8522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:02:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:43 smithi059 ceph-mon[97450]: pgmap v8522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:02:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:44.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:44.424952+0000) 2022-09-08T02:02:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:44.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:44.368514+0000) 2022-09-08T02:02:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:45.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:45.425165+0000) 2022-09-08T02:02:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:45.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:45.368736+0000) 2022-09-08T02:02:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:45 smithi055 ceph-mon[97033]: pgmap v8523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:45 smithi059 ceph-mon[97450]: pgmap v8523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:46.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:46.425414+0000) 2022-09-08T02:02:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:46.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:46.368895+0000) 2022-09-08T02:02:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:47.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:47.425583+0000) 2022-09-08T02:02:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:47.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:47.369088+0000) 2022-09-08T02:02:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:47 smithi055 ceph-mon[97033]: pgmap v8524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:02:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:47 smithi059 ceph-mon[97450]: pgmap v8524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:02:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:48.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:48.425794+0000) 2022-09-08T02:02:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:48.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:48.369325+0000) 2022-09-08T02:02:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:49.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:49.369484+0000) 2022-09-08T02:02:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:49.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:49.426021+0000) 2022-09-08T02:02:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:49 smithi055 ceph-mon[97033]: pgmap v8525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:49 smithi059 ceph-mon[97450]: pgmap v8525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:50.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:50.426195+0000) 2022-09-08T02:02:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:50.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:50.369747+0000) 2022-09-08T02:02:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:51.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:51.426446+0000) 2022-09-08T02:02:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:51.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:51.369942+0000) 2022-09-08T02:02:51.677 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:51 smithi059 ceph-mon[97450]: pgmap v8526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:51.678 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:02:51.678 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:02:51.679 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:02:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:51 smithi055 ceph-mon[97033]: pgmap v8526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:02:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:02:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:02:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:52.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:52.370174+0000) 2022-09-08T02:02:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:52.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:52.426748+0000) 2022-09-08T02:02:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:02:52.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:52 smithi055 ceph-mon[97033]: pgmap v8527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:02:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:02:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:52 smithi059 ceph-mon[97450]: pgmap v8527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:02:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:53.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:53.426992+0000) 2022-09-08T02:02:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:53.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:53.370418+0000) 2022-09-08T02:02:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:54.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:54.427218+0000) 2022-09-08T02:02:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:54.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:54.370693+0000) 2022-09-08T02:02:55.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:02:54.815+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:02:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:54 smithi055 ceph-mon[97033]: pgmap v8528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:54 smithi059 ceph-mon[97450]: pgmap v8528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:55.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:55.427376+0000) 2022-09-08T02:02:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:55.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:55.370856+0000) 2022-09-08T02:02:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:56.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:56.427616+0000) 2022-09-08T02:02:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:56.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:56.371011+0000) 2022-09-08T02:02:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:56 smithi055 ceph-mon[97033]: pgmap v8529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:02:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:56 smithi059 ceph-mon[97450]: pgmap v8529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:02:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:57.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:57.427887+0000) 2022-09-08T02:02:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:57.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:57.371233+0000) 2022-09-08T02:02:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:58.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:58.428091+0000) 2022-09-08T02:02:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:58.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:58.371464+0000) 2022-09-08T02:02:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:02:59.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:59.428293+0000) 2022-09-08T02:02:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:02:59.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:02:59.371692+0000) 2022-09-08T02:02:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:02:59 smithi055 ceph-mon[97033]: pgmap v8530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:02:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:02:59 smithi059 ceph-mon[97450]: pgmap v8530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:00.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:00.428516+0000) 2022-09-08T02:03:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:00.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:00.371828+0000) 2022-09-08T02:03:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:01.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:01.428751+0000) 2022-09-08T02:03:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:01.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:01.372045+0000) 2022-09-08T02:03:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:01 smithi055 ceph-mon[97033]: pgmap v8531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:01 smithi059 ceph-mon[97450]: pgmap v8531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:02.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:02.428948+0000) 2022-09-08T02:03:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:02.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:02.372265+0000) 2022-09-08T02:03:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:03.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:03.429070+0000) 2022-09-08T02:03:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:03.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:03.372515+0000) 2022-09-08T02:03:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:03 smithi055 ceph-mon[97033]: pgmap v8532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:03:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:03 smithi059 ceph-mon[97450]: pgmap v8532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:03:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:04.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:04.429309+0000) 2022-09-08T02:03:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:04.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:04.372814+0000) 2022-09-08T02:03:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:05.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:05.429585+0000) 2022-09-08T02:03:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:05.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:05.372999+0000) 2022-09-08T02:03:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:05 smithi055 ceph-mon[97033]: pgmap v8533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:03:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:03:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:05 smithi059 ceph-mon[97450]: pgmap v8533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:03:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:03:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:06.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:06.429776+0000) 2022-09-08T02:03:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:06.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:06.373227+0000) 2022-09-08T02:03:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:07.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:07.429948+0000) 2022-09-08T02:03:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:07.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:07.373407+0000) 2022-09-08T02:03:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:07 smithi055 ceph-mon[97033]: pgmap v8534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:03:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:07 smithi059 ceph-mon[97450]: pgmap v8534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:03:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:08.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:08.430156+0000) 2022-09-08T02:03:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:08.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:08.373570+0000) 2022-09-08T02:03:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:09.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:09.430380+0000) 2022-09-08T02:03:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:09.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:09.373744+0000) 2022-09-08T02:03:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:09 smithi055 ceph-mon[97033]: pgmap v8535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:03:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:09 smithi059 ceph-mon[97450]: pgmap v8535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:03:10.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:03:09.829+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:03:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:10.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:10.430615+0000) 2022-09-08T02:03:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:10.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:10.373910+0000) 2022-09-08T02:03:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:10 smithi055 ceph-mon[97033]: osdmap e615: 12 total, 11 up, 11 in 2022-09-08T02:03:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:10 smithi059 ceph-mon[97450]: osdmap e615: 12 total, 11 up, 11 in 2022-09-08T02:03:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:11.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:11.430833+0000) 2022-09-08T02:03:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:11.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:11.374146+0000) 2022-09-08T02:03:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:11 smithi055 ceph-mon[97033]: pgmap v8537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:03:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:11 smithi059 ceph-mon[97450]: pgmap v8537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:03:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:12.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:12.430966+0000) 2022-09-08T02:03:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:12.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:12.374349+0000) 2022-09-08T02:03:13.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:13.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:13.374546+0000) 2022-09-08T02:03:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:13.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:13.431123+0000) 2022-09-08T02:03:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:13 smithi055 ceph-mon[97033]: pgmap v8538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:03:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:13 smithi059 ceph-mon[97450]: pgmap v8538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:03:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:14.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:14.431338+0000) 2022-09-08T02:03:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:14.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:14.374762+0000) 2022-09-08T02:03:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:15.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:15.431530+0000) 2022-09-08T02:03:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:15.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:15.374943+0000) 2022-09-08T02:03:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:15 smithi055 ceph-mon[97033]: pgmap v8539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:03:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:15 smithi059 ceph-mon[97450]: pgmap v8539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:03:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:16.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:16.431735+0000) 2022-09-08T02:03:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:16.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:16.375223+0000) 2022-09-08T02:03:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:17.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:17.431907+0000) 2022-09-08T02:03:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:17.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:17.375455+0000) 2022-09-08T02:03:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:17 smithi055 ceph-mon[97033]: pgmap v8540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:03:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:17 smithi055 ceph-mon[97033]: osdmap e616: 12 total, 11 up, 11 in 2022-09-08T02:03:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:17 smithi059 ceph-mon[97450]: pgmap v8540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:03:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:17 smithi059 ceph-mon[97450]: osdmap e616: 12 total, 11 up, 11 in 2022-09-08T02:03:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:18.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:18.375686+0000) 2022-09-08T02:03:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:18.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:18.432049+0000) 2022-09-08T02:03:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:19.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:19.432225+0000) 2022-09-08T02:03:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:19.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:19.375869+0000) 2022-09-08T02:03:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:19 smithi055 ceph-mon[97033]: pgmap v8542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 243 B/s wr, 7 op/s 2022-09-08T02:03:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:19 smithi059 ceph-mon[97450]: pgmap v8542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 243 B/s wr, 7 op/s 2022-09-08T02:03:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:20.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:20.432486+0000) 2022-09-08T02:03:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:20.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:20.376095+0000) 2022-09-08T02:03:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:21.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:21.432724+0000) 2022-09-08T02:03:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:21.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:21.376285+0000) 2022-09-08T02:03:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:21 smithi055 ceph-mon[97033]: pgmap v8543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:03:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:21 smithi059 ceph-mon[97450]: pgmap v8543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:03:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:22.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:22.432942+0000) 2022-09-08T02:03:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:22.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:22.376521+0000) 2022-09-08T02:03:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:23.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:23.433173+0000) 2022-09-08T02:03:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:23.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:23.376767+0000) 2022-09-08T02:03:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:23 smithi055 ceph-mon[97033]: pgmap v8544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:03:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:23 smithi059 ceph-mon[97450]: pgmap v8544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:03:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:24.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:24.433380+0000) 2022-09-08T02:03:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:24.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:24.376965+0000) 2022-09-08T02:03:25.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:03:24.845+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:03:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:25.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:25.433609+0000) 2022-09-08T02:03:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:25.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:25.377184+0000) 2022-09-08T02:03:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:25 smithi055 ceph-mon[97033]: pgmap v8545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:03:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:25 smithi059 ceph-mon[97450]: pgmap v8545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:03:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:26.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:26.433834+0000) 2022-09-08T02:03:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:26.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:26.377452+0000) 2022-09-08T02:03:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:27.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:27.433996+0000) 2022-09-08T02:03:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:27.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:27.377724+0000) 2022-09-08T02:03:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:27 smithi055 ceph-mon[97033]: pgmap v8546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:03:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:27 smithi059 ceph-mon[97450]: pgmap v8546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:03:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:28.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:28.434152+0000) 2022-09-08T02:03:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:28.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:28.377912+0000) 2022-09-08T02:03:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:29.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:29.434359+0000) 2022-09-08T02:03:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:29.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:29.378166+0000) 2022-09-08T02:03:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:29 smithi055 ceph-mon[97033]: pgmap v8547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:03:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:29 smithi059 ceph-mon[97450]: pgmap v8547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:03:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:30.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:30.378396+0000) 2022-09-08T02:03:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:30.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:30.434675+0000) 2022-09-08T02:03:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:31.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:31.434837+0000) 2022-09-08T02:03:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:31.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:31.378592+0000) 2022-09-08T02:03:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:31 smithi055 ceph-mon[97033]: pgmap v8548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:31 smithi059 ceph-mon[97450]: pgmap v8548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:32.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:32.378741+0000) 2022-09-08T02:03:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:32.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:32.435032+0000) 2022-09-08T02:03:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:33.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:33.378960+0000) 2022-09-08T02:03:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:33.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:33.435220+0000) 2022-09-08T02:03:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:33 smithi055 ceph-mon[97033]: pgmap v8549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:03:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:33 smithi059 ceph-mon[97450]: pgmap v8549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:03:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:34.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:34.379111+0000) 2022-09-08T02:03:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:34.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:34.435475+0000) 2022-09-08T02:03:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:35.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:35.435749+0000) 2022-09-08T02:03:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:35.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:35.379343+0000) 2022-09-08T02:03:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:35 smithi055 ceph-mon[97033]: pgmap v8550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:35 smithi059 ceph-mon[97450]: pgmap v8550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:36.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:36.379544+0000) 2022-09-08T02:03:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:36.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:36.436006+0000) 2022-09-08T02:03:37.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:37.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:37.379790+0000) 2022-09-08T02:03:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:37.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:37.436234+0000) 2022-09-08T02:03:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:37 smithi055 ceph-mon[97033]: pgmap v8551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:03:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:37 smithi059 ceph-mon[97450]: pgmap v8551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:03:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:38.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:38.436475+0000) 2022-09-08T02:03:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:38.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:38.380007+0000) 2022-09-08T02:03:39.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:39.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:39.380265+0000) 2022-09-08T02:03:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:39.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:39.436709+0000) 2022-09-08T02:03:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:39 smithi055 ceph-mon[97033]: pgmap v8552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:39 smithi059 ceph-mon[97450]: pgmap v8552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:40.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:03:39.847+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:03:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:40.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:40.436939+0000) 2022-09-08T02:03:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:40.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:40.380519+0000) 2022-09-08T02:03:41.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:41.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:41.380768+0000) 2022-09-08T02:03:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:41.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:41.437104+0000) 2022-09-08T02:03:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:41 smithi055 ceph-mon[97033]: pgmap v8553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:41 smithi059 ceph-mon[97450]: pgmap v8553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:42.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:42.437319+0000) 2022-09-08T02:03:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:42.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:42.380943+0000) 2022-09-08T02:03:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:43.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:43.437545+0000) 2022-09-08T02:03:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:43.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:43.381179+0000) 2022-09-08T02:03:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:43 smithi055 ceph-mon[97033]: pgmap v8554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:03:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:43 smithi059 ceph-mon[97450]: pgmap v8554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:03:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:44.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:44.437769+0000) 2022-09-08T02:03:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:44.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:44.381378+0000) 2022-09-08T02:03:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:45.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:45.438025+0000) 2022-09-08T02:03:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:45.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:45.381610+0000) 2022-09-08T02:03:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:45 smithi055 ceph-mon[97033]: pgmap v8555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:45 smithi059 ceph-mon[97450]: pgmap v8555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:46.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:46.438286+0000) 2022-09-08T02:03:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:46.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:46.381824+0000) 2022-09-08T02:03:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:47.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:47.438503+0000) 2022-09-08T02:03:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:47.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:47.381953+0000) 2022-09-08T02:03:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:47 smithi055 ceph-mon[97033]: pgmap v8556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:03:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:47 smithi059 ceph-mon[97450]: pgmap v8556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:03:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:48.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:48.438771+0000) 2022-09-08T02:03:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:48.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:48.382186+0000) 2022-09-08T02:03:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:49.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:49.438981+0000) 2022-09-08T02:03:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:49.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:49.382424+0000) 2022-09-08T02:03:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:49 smithi055 ceph-mon[97033]: pgmap v8557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:49 smithi059 ceph-mon[97450]: pgmap v8557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:50.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:50.439138+0000) 2022-09-08T02:03:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:50.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:50.382631+0000) 2022-09-08T02:03:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:51.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:51.439332+0000) 2022-09-08T02:03:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:51.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:51.382867+0000) 2022-09-08T02:03:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:51 smithi055 ceph-mon[97033]: pgmap v8558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:51.752 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:51 smithi059 ceph-mon[97450]: pgmap v8558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:52.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:52.439590+0000) 2022-09-08T02:03:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:52.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:52.383114+0000) 2022-09-08T02:03:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:03:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:03:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:03:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:03:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:03:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:03:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:03:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:03:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:53.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:53.439879+0000) 2022-09-08T02:03:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:53.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:53.383293+0000) 2022-09-08T02:03:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:53 smithi055 ceph-mon[97033]: pgmap v8559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:03:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:53 smithi059 ceph-mon[97450]: pgmap v8559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:03:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:54.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:54.440038+0000) 2022-09-08T02:03:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:54.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:54.383528+0000) 2022-09-08T02:03:55.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:03:54.847+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:03:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:55.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:55.440191+0000) 2022-09-08T02:03:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:55.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:55.383720+0000) 2022-09-08T02:03:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:55 smithi055 ceph-mon[97033]: pgmap v8560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:55 smithi059 ceph-mon[97450]: pgmap v8560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:56.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:56.440399+0000) 2022-09-08T02:03:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:56.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:56.383924+0000) 2022-09-08T02:03:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:57.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:57.440581+0000) 2022-09-08T02:03:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:57.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:57.384112+0000) 2022-09-08T02:03:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:57 smithi055 ceph-mon[97033]: pgmap v8561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:03:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:57 smithi059 ceph-mon[97450]: pgmap v8561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:03:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:58.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:58.440805+0000) 2022-09-08T02:03:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:58.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:58.384265+0000) 2022-09-08T02:03:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:03:59.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:59.440987+0000) 2022-09-08T02:03:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:03:59.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:03:59.384480+0000) 2022-09-08T02:03:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:03:59 smithi055 ceph-mon[97033]: pgmap v8562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:03:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:03:59 smithi059 ceph-mon[97450]: pgmap v8562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:00.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:00.441147+0000) 2022-09-08T02:04:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:00.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:00.384712+0000) 2022-09-08T02:04:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:01.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:01.441387+0000) 2022-09-08T02:04:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:01.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:01.384943+0000) 2022-09-08T02:04:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:01 smithi055 ceph-mon[97033]: pgmap v8563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:01 smithi059 ceph-mon[97450]: pgmap v8563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:02.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:02.385160+0000) 2022-09-08T02:04:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:02.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:02.441530+0000) 2022-09-08T02:04:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:03.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:03.441763+0000) 2022-09-08T02:04:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:03.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:03.385326+0000) 2022-09-08T02:04:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:03 smithi055 ceph-mon[97033]: pgmap v8564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:04:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:03 smithi059 ceph-mon[97450]: pgmap v8564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:04:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:04.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:04.441886+0000) 2022-09-08T02:04:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:04.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:04.385558+0000) 2022-09-08T02:04:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:05.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:05.442128+0000) 2022-09-08T02:04:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:05.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:05.385810+0000) 2022-09-08T02:04:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:05 smithi055 ceph-mon[97033]: pgmap v8565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:04:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:04:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:05 smithi059 ceph-mon[97450]: pgmap v8565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:04:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:04:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:06.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:06.442369+0000) 2022-09-08T02:04:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:06.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:06.386038+0000) 2022-09-08T02:04:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:07.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:07.442667+0000) 2022-09-08T02:04:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:07.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:07.386219+0000) 2022-09-08T02:04:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:07 smithi055 ceph-mon[97033]: pgmap v8566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:04:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:07 smithi059 ceph-mon[97450]: pgmap v8566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:04:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:08.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:08.442940+0000) 2022-09-08T02:04:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:08.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:08.386351+0000) 2022-09-08T02:04:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:09.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:09.443146+0000) 2022-09-08T02:04:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:09.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:09.386584+0000) 2022-09-08T02:04:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:09 smithi055 ceph-mon[97033]: pgmap v8567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:04:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:09 smithi059 ceph-mon[97450]: pgmap v8567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:04:10.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:04:09.863+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:04:10.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:10.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:10.386810+0000) 2022-09-08T02:04:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:10.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:10.443360+0000) 2022-09-08T02:04:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:10 smithi059 ceph-mon[97450]: osdmap e617: 12 total, 11 up, 11 in 2022-09-08T02:04:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:10 smithi055 ceph-mon[97033]: osdmap e617: 12 total, 11 up, 11 in 2022-09-08T02:04:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:11.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:11.443495+0000) 2022-09-08T02:04:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:11.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:11.387043+0000) 2022-09-08T02:04:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:11 smithi059 ceph-mon[97450]: pgmap v8569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:04:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:11 smithi055 ceph-mon[97033]: pgmap v8569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:04:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:12.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:12.443725+0000) 2022-09-08T02:04:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:12.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:12.387290+0000) 2022-09-08T02:04:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:13.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:13.443852+0000) 2022-09-08T02:04:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:13.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:13.387537+0000) 2022-09-08T02:04:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:13 smithi059 ceph-mon[97450]: pgmap v8570: 129 pgs: 1 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 109 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:04:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:13 smithi055 ceph-mon[97033]: pgmap v8570: 129 pgs: 1 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 109 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:04:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:14.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:14.444086+0000) 2022-09-08T02:04:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:14.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:14.387769+0000) 2022-09-08T02:04:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:15.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:15.444338+0000) 2022-09-08T02:04:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:15.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:15.388022+0000) 2022-09-08T02:04:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:15 smithi059 ceph-mon[97450]: pgmap v8571: 129 pgs: 1 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 109 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:04:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:15 smithi055 ceph-mon[97033]: pgmap v8571: 129 pgs: 1 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 109 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:04:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:16.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:16.444551+0000) 2022-09-08T02:04:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:16.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:16.388238+0000) 2022-09-08T02:04:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:17.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:17.444717+0000) 2022-09-08T02:04:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:17.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:17.388372+0000) 2022-09-08T02:04:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:17 smithi059 ceph-mon[97450]: pgmap v8572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:04:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:17 smithi059 ceph-mon[97450]: osdmap e618: 12 total, 11 up, 11 in 2022-09-08T02:04:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:17 smithi055 ceph-mon[97033]: pgmap v8572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:04:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:17 smithi055 ceph-mon[97033]: osdmap e618: 12 total, 11 up, 11 in 2022-09-08T02:04:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:18.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:18.444913+0000) 2022-09-08T02:04:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:18.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:18.388518+0000) 2022-09-08T02:04:19.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:19.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:19.388648+0000) 2022-09-08T02:04:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:19.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:19.445108+0000) 2022-09-08T02:04:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:19 smithi059 ceph-mon[97450]: pgmap v8574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 247 B/s wr, 7 op/s 2022-09-08T02:04:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:19 smithi055 ceph-mon[97033]: pgmap v8574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 247 B/s wr, 7 op/s 2022-09-08T02:04:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:20.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:20.388862+0000) 2022-09-08T02:04:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:20.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:20.445320+0000) 2022-09-08T02:04:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:21.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:21.445574+0000) 2022-09-08T02:04:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:21.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:21.389027+0000) 2022-09-08T02:04:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:21 smithi059 ceph-mon[97450]: pgmap v8575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:04:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:21 smithi055 ceph-mon[97033]: pgmap v8575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:04:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:22.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:22.445819+0000) 2022-09-08T02:04:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:22.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:22.389147+0000) 2022-09-08T02:04:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:23.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:23.445979+0000) 2022-09-08T02:04:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:23.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:23.389336+0000) 2022-09-08T02:04:23.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:23 smithi059 ceph-mon[97450]: pgmap v8576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:04:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:23 smithi055 ceph-mon[97033]: pgmap v8576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:04:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:24.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:24.446198+0000) 2022-09-08T02:04:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:24.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:24.389477+0000) 2022-09-08T02:04:25.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:04:24.872+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:04:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:25.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:25.446392+0000) 2022-09-08T02:04:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:25.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:25.389684+0000) 2022-09-08T02:04:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:25 smithi059 ceph-mon[97450]: pgmap v8577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:04:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:25 smithi055 ceph-mon[97033]: pgmap v8577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:04:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:26.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:26.446619+0000) 2022-09-08T02:04:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:26.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:26.389806+0000) 2022-09-08T02:04:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:27.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:27.446789+0000) 2022-09-08T02:04:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:27.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:27.390014+0000) 2022-09-08T02:04:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:27 smithi059 ceph-mon[97450]: pgmap v8578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:04:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:27 smithi055 ceph-mon[97033]: pgmap v8578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:04:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:28.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:28.446992+0000) 2022-09-08T02:04:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:28.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:28.390170+0000) 2022-09-08T02:04:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:29.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:29.447212+0000) 2022-09-08T02:04:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:29.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:29.390322+0000) 2022-09-08T02:04:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:29 smithi055 ceph-mon[97033]: pgmap v8579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:04:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:29 smithi059 ceph-mon[97450]: pgmap v8579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:04:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:30.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:30.447460+0000) 2022-09-08T02:04:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:30.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:30.390481+0000) 2022-09-08T02:04:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:31.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:31.447759+0000) 2022-09-08T02:04:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:31.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:31.390683+0000) 2022-09-08T02:04:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:31 smithi055 ceph-mon[97033]: pgmap v8580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:31 smithi059 ceph-mon[97450]: pgmap v8580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:32.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:32.448029+0000) 2022-09-08T02:04:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:32.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:32.390851+0000) 2022-09-08T02:04:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:33.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:33.448179+0000) 2022-09-08T02:04:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:33.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:33.391030+0000) 2022-09-08T02:04:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:33 smithi055 ceph-mon[97033]: pgmap v8581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:04:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:33 smithi059 ceph-mon[97450]: pgmap v8581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:04:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:34.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:34.448306+0000) 2022-09-08T02:04:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:34.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:34.391152+0000) 2022-09-08T02:04:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:35.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:35.448548+0000) 2022-09-08T02:04:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:35.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:35.391325+0000) 2022-09-08T02:04:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:35 smithi055 ceph-mon[97033]: pgmap v8582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:35 smithi059 ceph-mon[97450]: pgmap v8582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:36.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:36.448816+0000) 2022-09-08T02:04:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:36.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:36.391562+0000) 2022-09-08T02:04:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:37.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:37.449048+0000) 2022-09-08T02:04:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:37.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:37.391723+0000) 2022-09-08T02:04:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:37 smithi055 ceph-mon[97033]: pgmap v8583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:04:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:37 smithi059 ceph-mon[97450]: pgmap v8583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:04:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:38.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:38.449248+0000) 2022-09-08T02:04:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:38.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:38.391918+0000) 2022-09-08T02:04:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:39.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:39.449491+0000) 2022-09-08T02:04:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:39.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:39.392152+0000) 2022-09-08T02:04:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:39 smithi055 ceph-mon[97033]: pgmap v8584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:39 smithi059 ceph-mon[97450]: pgmap v8584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:40.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:04:39.887+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:04:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:40.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:40.449727+0000) 2022-09-08T02:04:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:40.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:40.392350+0000) 2022-09-08T02:04:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:41.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:41.450006+0000) 2022-09-08T02:04:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:41.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:41.392555+0000) 2022-09-08T02:04:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:41 smithi055 ceph-mon[97033]: pgmap v8585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:41 smithi059 ceph-mon[97450]: pgmap v8585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:42.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:42.450247+0000) 2022-09-08T02:04:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:42.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:42.392760+0000) 2022-09-08T02:04:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:43.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:43.450496+0000) 2022-09-08T02:04:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:43.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:43.392974+0000) 2022-09-08T02:04:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:43 smithi055 ceph-mon[97033]: pgmap v8586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:04:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:43 smithi059 ceph-mon[97450]: pgmap v8586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:04:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:44.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:44.450727+0000) 2022-09-08T02:04:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:44.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:44.393212+0000) 2022-09-08T02:04:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:45.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:45.393456+0000) 2022-09-08T02:04:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:45.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:45.450908+0000) 2022-09-08T02:04:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:45 smithi055 ceph-mon[97033]: pgmap v8587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:45 smithi059 ceph-mon[97450]: pgmap v8587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:46.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:46.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:46.393724+0000) 2022-09-08T02:04:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:46.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:46.451072+0000) 2022-09-08T02:04:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:47.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:47.393952+0000) 2022-09-08T02:04:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:47.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:47.451308+0000) 2022-09-08T02:04:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:47 smithi055 ceph-mon[97033]: pgmap v8588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:04:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:47 smithi059 ceph-mon[97450]: pgmap v8588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:04:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:48.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:48.451539+0000) 2022-09-08T02:04:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:48.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:48.394132+0000) 2022-09-08T02:04:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:49.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:49.451747+0000) 2022-09-08T02:04:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:49.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:49.394367+0000) 2022-09-08T02:04:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:49 smithi055 ceph-mon[97033]: pgmap v8589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:49 smithi059 ceph-mon[97450]: pgmap v8589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:50.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:50.451928+0000) 2022-09-08T02:04:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:50.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:50.394591+0000) 2022-09-08T02:04:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:51.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:51.452186+0000) 2022-09-08T02:04:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:51.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:51.394838+0000) 2022-09-08T02:04:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:51 smithi055 ceph-mon[97033]: pgmap v8590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:51 smithi059 ceph-mon[97450]: pgmap v8590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:52.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:52.452396+0000) 2022-09-08T02:04:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:52.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:52.394982+0000) 2022-09-08T02:04:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:04:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:04:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:04:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:04:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:04:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:04:53.661 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:53.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:53.452609+0000) 2022-09-08T02:04:53.662 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:53.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:53.395136+0000) 2022-09-08T02:04:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:53 smithi055 ceph-mon[97033]: pgmap v8591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:04:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:04:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:53 smithi059 ceph-mon[97450]: pgmap v8591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:04:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:04:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:54.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:54.452739+0000) 2022-09-08T02:04:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:54.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:54.395291+0000) 2022-09-08T02:04:55.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:04:54.890+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:04:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:55.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:55.452971+0000) 2022-09-08T02:04:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:55.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:55.395536+0000) 2022-09-08T02:04:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:55 smithi055 ceph-mon[97033]: pgmap v8592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:55 smithi059 ceph-mon[97450]: pgmap v8592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:04:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:56.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:56.453208+0000) 2022-09-08T02:04:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:56.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:56.395706+0000) 2022-09-08T02:04:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:57.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:57.453468+0000) 2022-09-08T02:04:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:57.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:57.395880+0000) 2022-09-08T02:04:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:57 smithi055 ceph-mon[97033]: pgmap v8593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:04:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:57 smithi059 ceph-mon[97450]: pgmap v8593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:04:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:58.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:58.453698+0000) 2022-09-08T02:04:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:58.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:58.396024+0000) 2022-09-08T02:04:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:04:59.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:59.453844+0000) 2022-09-08T02:04:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:04:59.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:04:59.396207+0000) 2022-09-08T02:04:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:04:59 smithi055 ceph-mon[97033]: pgmap v8594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:04:59 smithi059 ceph-mon[97450]: pgmap v8594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:00.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:00.454016+0000) 2022-09-08T02:05:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:00.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:00.396459+0000) 2022-09-08T02:05:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:01.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:01.454227+0000) 2022-09-08T02:05:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:01.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:01.396722+0000) 2022-09-08T02:05:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:01 smithi055 ceph-mon[97033]: pgmap v8595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:01 smithi059 ceph-mon[97450]: pgmap v8595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:02.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:02.454365+0000) 2022-09-08T02:05:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:02.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:02.396947+0000) 2022-09-08T02:05:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:03.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:03.454555+0000) 2022-09-08T02:05:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:03.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:03.397149+0000) 2022-09-08T02:05:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:03 smithi055 ceph-mon[97033]: pgmap v8596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:05:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:03 smithi059 ceph-mon[97450]: pgmap v8596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:05:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:04.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:04.454791+0000) 2022-09-08T02:05:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:04.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:04.397365+0000) 2022-09-08T02:05:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:05.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:05.397584+0000) 2022-09-08T02:05:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:05.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:05.454973+0000) 2022-09-08T02:05:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:05 smithi055 ceph-mon[97033]: pgmap v8597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:05:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:05:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:05 smithi059 ceph-mon[97450]: pgmap v8597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:05:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:05:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:06.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:06.397746+0000) 2022-09-08T02:05:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:06.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:06.455214+0000) 2022-09-08T02:05:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:07.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:07.455421+0000) 2022-09-08T02:05:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:07.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:07.397982+0000) 2022-09-08T02:05:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:07 smithi055 ceph-mon[97033]: pgmap v8598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:05:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:07 smithi059 ceph-mon[97450]: pgmap v8598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:05:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:08.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:08.455684+0000) 2022-09-08T02:05:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:08.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:08.398204+0000) 2022-09-08T02:05:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:09.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:09.398435+0000) 2022-09-08T02:05:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:09.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:09.455924+0000) 2022-09-08T02:05:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:09 smithi055 ceph-mon[97033]: pgmap v8599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:05:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:09 smithi055 ceph-mon[97033]: osdmap e619: 12 total, 11 up, 11 in 2022-09-08T02:05:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:09 smithi059 ceph-mon[97450]: pgmap v8599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:05:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:09 smithi059 ceph-mon[97450]: osdmap e619: 12 total, 11 up, 11 in 2022-09-08T02:05:10.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:05:09.899+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:05:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:10.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:10.456174+0000) 2022-09-08T02:05:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:10.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:10.398719+0000) 2022-09-08T02:05:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:11.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:11.456415+0000) 2022-09-08T02:05:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:11.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:11.398970+0000) 2022-09-08T02:05:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:11 smithi055 ceph-mon[97033]: pgmap v8601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:05:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:11 smithi059 ceph-mon[97450]: pgmap v8601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:05:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:12.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:12.456623+0000) 2022-09-08T02:05:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:12.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:12.399219+0000) 2022-09-08T02:05:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:13.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:13.456874+0000) 2022-09-08T02:05:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:13.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:13.399365+0000) 2022-09-08T02:05:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:13 smithi055 ceph-mon[97033]: pgmap v8602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:05:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:13 smithi059 ceph-mon[97450]: pgmap v8602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:05:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:14.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:14.457087+0000) 2022-09-08T02:05:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:14.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:14.399582+0000) 2022-09-08T02:05:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:15.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:15.457253+0000) 2022-09-08T02:05:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:15.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:15.399882+0000) 2022-09-08T02:05:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:15 smithi055 ceph-mon[97033]: pgmap v8603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:05:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:15 smithi059 ceph-mon[97450]: pgmap v8603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:05:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:16.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:16.457375+0000) 2022-09-08T02:05:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:16.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:16.400073+0000) 2022-09-08T02:05:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:17.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:17.457522+0000) 2022-09-08T02:05:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:17.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:17.400300+0000) 2022-09-08T02:05:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:17 smithi055 ceph-mon[97033]: pgmap v8604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:05:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:17 smithi055 ceph-mon[97033]: osdmap e620: 12 total, 11 up, 11 in 2022-09-08T02:05:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:17 smithi059 ceph-mon[97450]: pgmap v8604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:05:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:17 smithi059 ceph-mon[97450]: osdmap e620: 12 total, 11 up, 11 in 2022-09-08T02:05:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:18.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:18.457756+0000) 2022-09-08T02:05:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:18.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:18.400494+0000) 2022-09-08T02:05:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:19.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:19.457956+0000) 2022-09-08T02:05:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:19.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:19.400739+0000) 2022-09-08T02:05:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:19 smithi055 ceph-mon[97033]: pgmap v8606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2022-09-08T02:05:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:19 smithi059 ceph-mon[97450]: pgmap v8606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2022-09-08T02:05:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:20.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:20.458186+0000) 2022-09-08T02:05:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:20.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:20.400981+0000) 2022-09-08T02:05:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:21 smithi055 ceph-mon[97033]: pgmap v8607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:05:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:21 smithi059 ceph-mon[97450]: pgmap v8607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:05:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:21.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:21.458350+0000) 2022-09-08T02:05:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:21.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:21.401216+0000) 2022-09-08T02:05:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:22.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:22.458477+0000) 2022-09-08T02:05:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:22.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:22.401457+0000) 2022-09-08T02:05:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:23 smithi055 ceph-mon[97033]: pgmap v8608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:05:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:23 smithi059 ceph-mon[97450]: pgmap v8608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:05:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:23.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:23.458680+0000) 2022-09-08T02:05:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:23.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:23.401737+0000) 2022-09-08T02:05:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:24.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:24.458892+0000) 2022-09-08T02:05:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:24.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:24.401930+0000) 2022-09-08T02:05:25.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:05:24.900+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:05:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:25 smithi055 ceph-mon[97033]: pgmap v8609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:05:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:25 smithi059 ceph-mon[97450]: pgmap v8609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:05:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:25.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:25.402091+0000) 2022-09-08T02:05:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:25.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:25.459026+0000) 2022-09-08T02:05:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:26.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:26.459232+0000) 2022-09-08T02:05:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:26.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:26.402244+0000) 2022-09-08T02:05:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:27 smithi055 ceph-mon[97033]: pgmap v8610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:05:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:27 smithi059 ceph-mon[97450]: pgmap v8610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:05:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:27.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:27.459446+0000) 2022-09-08T02:05:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:27.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:27.402488+0000) 2022-09-08T02:05:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:28.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:28.459705+0000) 2022-09-08T02:05:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:28.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:28.402737+0000) 2022-09-08T02:05:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:29 smithi055 ceph-mon[97033]: pgmap v8611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:05:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:29 smithi059 ceph-mon[97450]: pgmap v8611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:05:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:29.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:29.459945+0000) 2022-09-08T02:05:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:29.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:29.402904+0000) 2022-09-08T02:05:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:30.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:30.460212+0000) 2022-09-08T02:05:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:30.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:30.403134+0000) 2022-09-08T02:05:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:31 smithi055 ceph-mon[97033]: pgmap v8612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:31 smithi059 ceph-mon[97450]: pgmap v8612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:31.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:31.460431+0000) 2022-09-08T02:05:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:31.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:31.403405+0000) 2022-09-08T02:05:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:32.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:32.460616+0000) 2022-09-08T02:05:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:32.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:32.403672+0000) 2022-09-08T02:05:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:33 smithi055 ceph-mon[97033]: pgmap v8613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:05:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:33 smithi059 ceph-mon[97450]: pgmap v8613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:05:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:33.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:33.460885+0000) 2022-09-08T02:05:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:33.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:33.403853+0000) 2022-09-08T02:05:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:34.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:34.461032+0000) 2022-09-08T02:05:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:34.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:34.404046+0000) 2022-09-08T02:05:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:35 smithi055 ceph-mon[97033]: pgmap v8614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:35 smithi059 ceph-mon[97450]: pgmap v8614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:35.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:35.461286+0000) 2022-09-08T02:05:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:35.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:35.404254+0000) 2022-09-08T02:05:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:36.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:36.404473+0000) 2022-09-08T02:05:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:36.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:36.461557+0000) 2022-09-08T02:05:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:37 smithi055 ceph-mon[97033]: pgmap v8615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:05:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:37 smithi059 ceph-mon[97450]: pgmap v8615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:05:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:37.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:37.404747+0000) 2022-09-08T02:05:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:37.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:37.461807+0000) 2022-09-08T02:05:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:38.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:38.404960+0000) 2022-09-08T02:05:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:38.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:38.462036+0000) 2022-09-08T02:05:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:39 smithi055 ceph-mon[97033]: pgmap v8616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:39 smithi059 ceph-mon[97450]: pgmap v8616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:39.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:39.405173+0000) 2022-09-08T02:05:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:39.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:39.462257+0000) 2022-09-08T02:05:40.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:05:39.916+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:05:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:40.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:40.405371+0000) 2022-09-08T02:05:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:40.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:40.462494+0000) 2022-09-08T02:05:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:41 smithi055 ceph-mon[97033]: pgmap v8617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:41 smithi059 ceph-mon[97450]: pgmap v8617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:41.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:41.462745+0000) 2022-09-08T02:05:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:41.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:41.405608+0000) 2022-09-08T02:05:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:42.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:42.463006+0000) 2022-09-08T02:05:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:42.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:42.405853+0000) 2022-09-08T02:05:43.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:43 smithi055 ceph-mon[97033]: pgmap v8618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:05:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:43 smithi059 ceph-mon[97450]: pgmap v8618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:05:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:43.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:43.463271+0000) 2022-09-08T02:05:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:43.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:43.406155+0000) 2022-09-08T02:05:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:44.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:44.463467+0000) 2022-09-08T02:05:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:44.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:44.406402+0000) 2022-09-08T02:05:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:45 smithi055 ceph-mon[97033]: pgmap v8619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:45 smithi059 ceph-mon[97450]: pgmap v8619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:45.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:45.463710+0000) 2022-09-08T02:05:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:45.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:45.406660+0000) 2022-09-08T02:05:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:46.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:46.463865+0000) 2022-09-08T02:05:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:46.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:46.406779+0000) 2022-09-08T02:05:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:47 smithi055 ceph-mon[97033]: pgmap v8620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:05:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:47 smithi059 ceph-mon[97450]: pgmap v8620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:05:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:47.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:47.406899+0000) 2022-09-08T02:05:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:47.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:47.464121+0000) 2022-09-08T02:05:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:48.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:48.464369+0000) 2022-09-08T02:05:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:48.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:48.407112+0000) 2022-09-08T02:05:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:49 smithi055 ceph-mon[97033]: pgmap v8621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:49 smithi059 ceph-mon[97450]: pgmap v8621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:49.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:49.464536+0000) 2022-09-08T02:05:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:49.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:49.407363+0000) 2022-09-08T02:05:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:50.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:50.464774+0000) 2022-09-08T02:05:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:50.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:50.407581+0000) 2022-09-08T02:05:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:51 smithi055 ceph-mon[97033]: pgmap v8622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:51 smithi059 ceph-mon[97450]: pgmap v8622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:51.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:51.464932+0000) 2022-09-08T02:05:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:51.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:51.407811+0000) 2022-09-08T02:05:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:52.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:52.465159+0000) 2022-09-08T02:05:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:52.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:52.408044+0000) 2022-09-08T02:05:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:53 smithi055 ceph-mon[97033]: pgmap v8623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:05:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:05:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:05:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:05:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:53 smithi059 ceph-mon[97450]: pgmap v8623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:05:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:05:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:05:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:05:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:53.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:53.465336+0000) 2022-09-08T02:05:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:53.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:53.408280+0000) 2022-09-08T02:05:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:05:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:05:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:54.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:54.408501+0000) 2022-09-08T02:05:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:54.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:54.465498+0000) 2022-09-08T02:05:55.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:05:54.932+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:05:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:55 smithi059 ceph-mon[97450]: pgmap v8624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:55.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:55.465734+0000) 2022-09-08T02:05:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:55.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:55.408706+0000) 2022-09-08T02:05:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:55 smithi055 ceph-mon[97033]: pgmap v8624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:56.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:56.465900+0000) 2022-09-08T02:05:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:56.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:56.408932+0000) 2022-09-08T02:05:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:57 smithi059 ceph-mon[97450]: pgmap v8625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:05:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:57.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:57.466077+0000) 2022-09-08T02:05:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:57.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:57.409222+0000) 2022-09-08T02:05:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:57 smithi055 ceph-mon[97033]: pgmap v8625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:05:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:58.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:58.466330+0000) 2022-09-08T02:05:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:58.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:58.409409+0000) 2022-09-08T02:05:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:05:59 smithi059 ceph-mon[97450]: pgmap v8626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:05:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:05:59.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:59.466581+0000) 2022-09-08T02:05:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:05:59.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:05:59.409624+0000) 2022-09-08T02:05:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:05:59 smithi055 ceph-mon[97033]: pgmap v8626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:00.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:00.409870+0000) 2022-09-08T02:06:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:00.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:00.466844+0000) 2022-09-08T02:06:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:01 smithi059 ceph-mon[97450]: pgmap v8627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:01.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:01.467055+0000) 2022-09-08T02:06:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:01.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:01.410004+0000) 2022-09-08T02:06:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:01 smithi055 ceph-mon[97033]: pgmap v8627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:02.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:02.467260+0000) 2022-09-08T02:06:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:02.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:02.410155+0000) 2022-09-08T02:06:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:03 smithi059 ceph-mon[97450]: pgmap v8628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:06:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:03.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:03.467482+0000) 2022-09-08T02:06:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:03.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:03.410366+0000) 2022-09-08T02:06:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:03 smithi055 ceph-mon[97033]: pgmap v8628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:06:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:04.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:04.467731+0000) 2022-09-08T02:06:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:04.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:04.410532+0000) 2022-09-08T02:06:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:05 smithi059 ceph-mon[97450]: pgmap v8629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:06:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:06:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:05.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:05.467937+0000) 2022-09-08T02:06:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:05.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:05.410773+0000) 2022-09-08T02:06:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:05 smithi055 ceph-mon[97033]: pgmap v8629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:06:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:06:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:06.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:06.468140+0000) 2022-09-08T02:06:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:06.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:06.410974+0000) 2022-09-08T02:06:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:07 smithi059 ceph-mon[97450]: pgmap v8630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:06:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:07.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:07.468309+0000) 2022-09-08T02:06:07.983 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:07.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:07.411132+0000) 2022-09-08T02:06:07.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:07 smithi055 ceph-mon[97033]: pgmap v8630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:06:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:08.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:08.468460+0000) 2022-09-08T02:06:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:08.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:08.411314+0000) 2022-09-08T02:06:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:09 smithi059 ceph-mon[97450]: pgmap v8631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:06:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:09.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:09.468692+0000) 2022-09-08T02:06:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:09.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:09.411568+0000) 2022-09-08T02:06:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:09 smithi055 ceph-mon[97033]: pgmap v8631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:06:10.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:06:09.932+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:06:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:10 smithi059 ceph-mon[97450]: osdmap e621: 12 total, 11 up, 11 in 2022-09-08T02:06:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:10.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:10.468853+0000) 2022-09-08T02:06:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:10.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:10.411758+0000) 2022-09-08T02:06:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:10 smithi055 ceph-mon[97033]: osdmap e621: 12 total, 11 up, 11 in 2022-09-08T02:06:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:11 smithi059 ceph-mon[97450]: pgmap v8633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:06:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:11.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:11.469033+0000) 2022-09-08T02:06:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:11.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:11.412001+0000) 2022-09-08T02:06:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:11 smithi055 ceph-mon[97033]: pgmap v8633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:06:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:12.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:12.469269+0000) 2022-09-08T02:06:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:12.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:12.412271+0000) 2022-09-08T02:06:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:13 smithi059 ceph-mon[97450]: pgmap v8634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:06:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:13.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:13.469485+0000) 2022-09-08T02:06:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:13.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:13.412533+0000) 2022-09-08T02:06:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:13 smithi055 ceph-mon[97033]: pgmap v8634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:06:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:14.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:14.469725+0000) 2022-09-08T02:06:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:14.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:14.412733+0000) 2022-09-08T02:06:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:15.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:15.469952+0000) 2022-09-08T02:06:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:15.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:15.412878+0000) 2022-09-08T02:06:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:15 smithi055 ceph-mon[97033]: pgmap v8635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:06:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:15 smithi059 ceph-mon[97450]: pgmap v8635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:06:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:16.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:16.470140+0000) 2022-09-08T02:06:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:16.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:16.413077+0000) 2022-09-08T02:06:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:17.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:17.470354+0000) 2022-09-08T02:06:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:17.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:17.413240+0000) 2022-09-08T02:06:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:17 smithi055 ceph-mon[97033]: pgmap v8636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:06:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:17 smithi055 ceph-mon[97033]: osdmap e622: 12 total, 11 up, 11 in 2022-09-08T02:06:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:17 smithi059 ceph-mon[97450]: pgmap v8636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:06:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:17 smithi059 ceph-mon[97450]: osdmap e622: 12 total, 11 up, 11 in 2022-09-08T02:06:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:18.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:18.470575+0000) 2022-09-08T02:06:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:18.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:18.413381+0000) 2022-09-08T02:06:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:19.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:19.470803+0000) 2022-09-08T02:06:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:19.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:19.413612+0000) 2022-09-08T02:06:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:19 smithi055 ceph-mon[97033]: pgmap v8638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 246 B/s wr, 7 op/s 2022-09-08T02:06:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:19 smithi059 ceph-mon[97450]: pgmap v8638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 246 B/s wr, 7 op/s 2022-09-08T02:06:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:20.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:20.471018+0000) 2022-09-08T02:06:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:20.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:20.413890+0000) 2022-09-08T02:06:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:21.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:21.471208+0000) 2022-09-08T02:06:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:21.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:21.414029+0000) 2022-09-08T02:06:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:21 smithi055 ceph-mon[97033]: pgmap v8639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:06:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:21 smithi059 ceph-mon[97450]: pgmap v8639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:06:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:22.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:22.471402+0000) 2022-09-08T02:06:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:22.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:22.414157+0000) 2022-09-08T02:06:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:23.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:23.471660+0000) 2022-09-08T02:06:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:23.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:23.414401+0000) 2022-09-08T02:06:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:23 smithi055 ceph-mon[97033]: pgmap v8640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:06:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:23 smithi059 ceph-mon[97450]: pgmap v8640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:06:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:24.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:24.471927+0000) 2022-09-08T02:06:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:24.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:24.414623+0000) 2022-09-08T02:06:25.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:06:24.939+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:06:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:25.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:25.472107+0000) 2022-09-08T02:06:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:25.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:25.414795+0000) 2022-09-08T02:06:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:25 smithi055 ceph-mon[97033]: pgmap v8641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:06:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:25 smithi059 ceph-mon[97450]: pgmap v8641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:06:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:26.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:26.472338+0000) 2022-09-08T02:06:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:26.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:26.414937+0000) 2022-09-08T02:06:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:27.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:27.472577+0000) 2022-09-08T02:06:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:27.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:27.415089+0000) 2022-09-08T02:06:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:27 smithi055 ceph-mon[97033]: pgmap v8642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:06:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:27 smithi059 ceph-mon[97450]: pgmap v8642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:06:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:28.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:28.472703+0000) 2022-09-08T02:06:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:28.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:28.415325+0000) 2022-09-08T02:06:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:29.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:29.472852+0000) 2022-09-08T02:06:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:29.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:29.415570+0000) 2022-09-08T02:06:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:29 smithi055 ceph-mon[97033]: pgmap v8643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:06:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:29 smithi059 ceph-mon[97450]: pgmap v8643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:06:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:30.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:30.415765+0000) 2022-09-08T02:06:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:30.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:30.473081+0000) 2022-09-08T02:06:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:31.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:31.473311+0000) 2022-09-08T02:06:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:31.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:31.415971+0000) 2022-09-08T02:06:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:31 smithi055 ceph-mon[97033]: pgmap v8644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:31 smithi059 ceph-mon[97450]: pgmap v8644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:32.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:32.473504+0000) 2022-09-08T02:06:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:32.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:32.416194+0000) 2022-09-08T02:06:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:33.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:33.473753+0000) 2022-09-08T02:06:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:33.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:33.416449+0000) 2022-09-08T02:06:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:33 smithi055 ceph-mon[97033]: pgmap v8645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:06:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:33 smithi059 ceph-mon[97450]: pgmap v8645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:06:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:34.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:34.473943+0000) 2022-09-08T02:06:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:34.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:34.416627+0000) 2022-09-08T02:06:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:35.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:35.474152+0000) 2022-09-08T02:06:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:35.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:35.416885+0000) 2022-09-08T02:06:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:35 smithi055 ceph-mon[97033]: pgmap v8646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:35 smithi059 ceph-mon[97450]: pgmap v8646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:36.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:36.474333+0000) 2022-09-08T02:06:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:36.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:36.417105+0000) 2022-09-08T02:06:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:37.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:37.474547+0000) 2022-09-08T02:06:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:37.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:37.417314+0000) 2022-09-08T02:06:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:37 smithi055 ceph-mon[97033]: pgmap v8647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:06:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:37 smithi059 ceph-mon[97450]: pgmap v8647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:06:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:38.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:38.417471+0000) 2022-09-08T02:06:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:38.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:38.474773+0000) 2022-09-08T02:06:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:39.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:39.474994+0000) 2022-09-08T02:06:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:39.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:39.417666+0000) 2022-09-08T02:06:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:39 smithi055 ceph-mon[97033]: pgmap v8648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:39 smithi059 ceph-mon[97450]: pgmap v8648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:40.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:06:39.952+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:06:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:40.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:40.475257+0000) 2022-09-08T02:06:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:40.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:40.417871+0000) 2022-09-08T02:06:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:41.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:41.475485+0000) 2022-09-08T02:06:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:41.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:41.418040+0000) 2022-09-08T02:06:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:41 smithi055 ceph-mon[97033]: pgmap v8649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:41 smithi059 ceph-mon[97450]: pgmap v8649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:42.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:42.475734+0000) 2022-09-08T02:06:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:42.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:42.418210+0000) 2022-09-08T02:06:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:43.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:43.475956+0000) 2022-09-08T02:06:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:43.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:43.418441+0000) 2022-09-08T02:06:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:43 smithi055 ceph-mon[97033]: pgmap v8650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:06:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:43 smithi059 ceph-mon[97450]: pgmap v8650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:06:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:44.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:44.476126+0000) 2022-09-08T02:06:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:44.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:44.418665+0000) 2022-09-08T02:06:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:45.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:45.476327+0000) 2022-09-08T02:06:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:45.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:45.418872+0000) 2022-09-08T02:06:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:45 smithi055 ceph-mon[97033]: pgmap v8651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:45 smithi059 ceph-mon[97450]: pgmap v8651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:46.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:46.476534+0000) 2022-09-08T02:06:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:46.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:46.419044+0000) 2022-09-08T02:06:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:47.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:47.476781+0000) 2022-09-08T02:06:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:47.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:47.419222+0000) 2022-09-08T02:06:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:47 smithi055 ceph-mon[97033]: pgmap v8652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:06:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:47 smithi059 ceph-mon[97450]: pgmap v8652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:06:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:48.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:48.477031+0000) 2022-09-08T02:06:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:48.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:48.419452+0000) 2022-09-08T02:06:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:49.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:49.477314+0000) 2022-09-08T02:06:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:49.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:49.419695+0000) 2022-09-08T02:06:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:49 smithi055 ceph-mon[97033]: pgmap v8653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:49 smithi059 ceph-mon[97450]: pgmap v8653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:50.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:50.477575+0000) 2022-09-08T02:06:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:50.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:50.419895+0000) 2022-09-08T02:06:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:51.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:51.477858+0000) 2022-09-08T02:06:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:51.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:51.420116+0000) 2022-09-08T02:06:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:51 smithi055 ceph-mon[97033]: pgmap v8654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:51 smithi059 ceph-mon[97450]: pgmap v8654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:52.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:52.478077+0000) 2022-09-08T02:06:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:52.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:52.420344+0000) 2022-09-08T02:06:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:53.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:53.478264+0000) 2022-09-08T02:06:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:53.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:53.420621+0000) 2022-09-08T02:06:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:53 smithi055 ceph-mon[97033]: pgmap v8655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:06:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:06:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:06:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:06:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:53 smithi059 ceph-mon[97450]: pgmap v8655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:06:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:06:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:06:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:06:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:54.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:54.478466+0000) 2022-09-08T02:06:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:54.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:54.420829+0000) 2022-09-08T02:06:55.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:06:54.966+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:06:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:55 smithi055 ceph-mon[97033]: pgmap v8656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:06:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:06:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:55 smithi059 ceph-mon[97450]: pgmap v8656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:06:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:06:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:55.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:55.478729+0000) 2022-09-08T02:06:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:55.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:55.421016+0000) 2022-09-08T02:06:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:56.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:56.478899+0000) 2022-09-08T02:06:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:56.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:56.421203+0000) 2022-09-08T02:06:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:57 smithi055 ceph-mon[97033]: pgmap v8657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:06:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:06:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:06:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:06:57.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:06:57.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:57 smithi059 ceph-mon[97450]: pgmap v8657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:06:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:06:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:06:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:06:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:06:57.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:06:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:57.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:57.479145+0000) 2022-09-08T02:06:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:57.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:57.421432+0000) 2022-09-08T02:06:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:58 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T02:06:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:58 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T02:06:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:58 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T02:06:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:58 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T02:06:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:06:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:58 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T02:06:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:58 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T02:06:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:58 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T02:06:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:58 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T02:06:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:06:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:58.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:58.479383+0000) 2022-09-08T02:06:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:58.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:58.421652+0000) 2022-09-08T02:06:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:06:59 smithi055 ceph-mon[97033]: pgmap v8658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:06:59 smithi059 ceph-mon[97450]: pgmap v8658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:06:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:06:59.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:59.479566+0000) 2022-09-08T02:06:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:06:59.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:06:59.421876+0000) 2022-09-08T02:07:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:00.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:00.479694+0000) 2022-09-08T02:07:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:00.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:00.422044+0000) 2022-09-08T02:07:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:01 smithi055 ceph-mon[97033]: pgmap v8659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T02:07:01.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:01 smithi059 ceph-mon[97450]: pgmap v8659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T02:07:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:01.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:01.479852+0000) 2022-09-08T02:07:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:01.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:01.422261+0000) 2022-09-08T02:07:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:02.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:02.479994+0000) 2022-09-08T02:07:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:02.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:02.422473+0000) 2022-09-08T02:07:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:03 smithi055 ceph-mon[97033]: pgmap v8660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:07:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:03 smithi059 ceph-mon[97450]: pgmap v8660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:07:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:03.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:03.480212+0000) 2022-09-08T02:07:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:03.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:03.422703+0000) 2022-09-08T02:07:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:04.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:04.480450+0000) 2022-09-08T02:07:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:04.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:04.422907+0000) 2022-09-08T02:07:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:05 smithi055 ceph-mon[97033]: pgmap v8661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:07:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:07:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:07:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:05 smithi059 ceph-mon[97450]: pgmap v8661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:07:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:07:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:07:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:05.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:05.480717+0000) 2022-09-08T02:07:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:05.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:05.423168+0000) 2022-09-08T02:07:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:06.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:06.480920+0000) 2022-09-08T02:07:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:06.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:06.423389+0000) 2022-09-08T02:07:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:07 smithi055 ceph-mon[97033]: pgmap v8662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 47 KiB/s wr, 22 op/s 2022-09-08T02:07:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:07 smithi059 ceph-mon[97450]: pgmap v8662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 47 KiB/s wr, 22 op/s 2022-09-08T02:07:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:07.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:07.481094+0000) 2022-09-08T02:07:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:07.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:07.423603+0000) 2022-09-08T02:07:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:08.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:08.481235+0000) 2022-09-08T02:07:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:08.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:08.423806+0000) 2022-09-08T02:07:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:09 smithi055 ceph-mon[97033]: pgmap v8663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 47 KiB/s wr, 19 op/s 2022-09-08T02:07:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:09 smithi059 ceph-mon[97450]: pgmap v8663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 47 KiB/s wr, 19 op/s 2022-09-08T02:07:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:09.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:09.481401+0000) 2022-09-08T02:07:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:09.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:09.423943+0000) 2022-09-08T02:07:10.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:07:09.979+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:07:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:10 smithi055 ceph-mon[97033]: osdmap e623: 12 total, 11 up, 11 in 2022-09-08T02:07:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:10 smithi059 ceph-mon[97450]: osdmap e623: 12 total, 11 up, 11 in 2022-09-08T02:07:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:10.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:10.481622+0000) 2022-09-08T02:07:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:10.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:10.424120+0000) 2022-09-08T02:07:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:11 smithi055 ceph-mon[97033]: pgmap v8665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 56 KiB/s wr, 23 op/s 2022-09-08T02:07:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:11 smithi059 ceph-mon[97450]: pgmap v8665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 56 KiB/s wr, 23 op/s 2022-09-08T02:07:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:11.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:11.481929+0000) 2022-09-08T02:07:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:11.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:11.424351+0000) 2022-09-08T02:07:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:12.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:12.482191+0000) 2022-09-08T02:07:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:12.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:12.424623+0000) 2022-09-08T02:07:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:13 smithi055 ceph-mon[97033]: pgmap v8666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-08T02:07:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:13 smithi059 ceph-mon[97450]: pgmap v8666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-08T02:07:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:13.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:13.482441+0000) 2022-09-08T02:07:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:13.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:13.424868+0000) 2022-09-08T02:07:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:14.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:14.482714+0000) 2022-09-08T02:07:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:14.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:14.425106+0000) 2022-09-08T02:07:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:15 smithi055 ceph-mon[97033]: pgmap v8667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-08T02:07:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:15 smithi059 ceph-mon[97450]: pgmap v8667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-08T02:07:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:15.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:15.482952+0000) 2022-09-08T02:07:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:15.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:15.425297+0000) 2022-09-08T02:07:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:16.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:16.483217+0000) 2022-09-08T02:07:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:16.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:16.425511+0000) 2022-09-08T02:07:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:17 smithi055 ceph-mon[97033]: pgmap v8668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:07:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:17 smithi055 ceph-mon[97033]: osdmap e624: 12 total, 11 up, 11 in 2022-09-08T02:07:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:17 smithi059 ceph-mon[97450]: pgmap v8668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:07:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:17 smithi059 ceph-mon[97450]: osdmap e624: 12 total, 11 up, 11 in 2022-09-08T02:07:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:17.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:17.483466+0000) 2022-09-08T02:07:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:17.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:17.425747+0000) 2022-09-08T02:07:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:18.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:18.483754+0000) 2022-09-08T02:07:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:18.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:18.425973+0000) 2022-09-08T02:07:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:19 smithi055 ceph-mon[97033]: pgmap v8670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 243 B/s wr, 7 op/s 2022-09-08T02:07:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:19 smithi059 ceph-mon[97450]: pgmap v8670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 243 B/s wr, 7 op/s 2022-09-08T02:07:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:19.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:19.483995+0000) 2022-09-08T02:07:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:19.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:19.426237+0000) 2022-09-08T02:07:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:20.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:20.484274+0000) 2022-09-08T02:07:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:20.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:20.426508+0000) 2022-09-08T02:07:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:21 smithi055 ceph-mon[97033]: pgmap v8671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:07:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:21 smithi059 ceph-mon[97450]: pgmap v8671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:07:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:21.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:21.484558+0000) 2022-09-08T02:07:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:21.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:21.426706+0000) 2022-09-08T02:07:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:22.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:22.484812+0000) 2022-09-08T02:07:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:22.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:22.426946+0000) 2022-09-08T02:07:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:23 smithi055 ceph-mon[97033]: pgmap v8672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:07:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:23 smithi059 ceph-mon[97450]: pgmap v8672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:07:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:23.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:23.484947+0000) 2022-09-08T02:07:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:23.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:23.427207+0000) 2022-09-08T02:07:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:24.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:24.485164+0000) 2022-09-08T02:07:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:24.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:24.427462+0000) 2022-09-08T02:07:25.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:07:24.994+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:07:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:25 smithi055 ceph-mon[97033]: pgmap v8673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:07:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:25 smithi059 ceph-mon[97450]: pgmap v8673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:07:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:25.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:25.485369+0000) 2022-09-08T02:07:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:25.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:25.427765+0000) 2022-09-08T02:07:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:26.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:26.485547+0000) 2022-09-08T02:07:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:26.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:26.427933+0000) 2022-09-08T02:07:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:27 smithi055 ceph-mon[97033]: pgmap v8674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:07:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:27 smithi059 ceph-mon[97450]: pgmap v8674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:07:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:27.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:27.485721+0000) 2022-09-08T02:07:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:27.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:27.428227+0000) 2022-09-08T02:07:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:28.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:28.485917+0000) 2022-09-08T02:07:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:28.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:28.428501+0000) 2022-09-08T02:07:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:29 smithi055 ceph-mon[97033]: pgmap v8675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:07:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:29 smithi059 ceph-mon[97450]: pgmap v8675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:07:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:29.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:29.486143+0000) 2022-09-08T02:07:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:29.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:29.428760+0000) 2022-09-08T02:07:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:30.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:30.486380+0000) 2022-09-08T02:07:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:30.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:30.428985+0000) 2022-09-08T02:07:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:31 smithi055 ceph-mon[97033]: pgmap v8676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T02:07:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:31 smithi059 ceph-mon[97450]: pgmap v8676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T02:07:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:31.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:31.486628+0000) 2022-09-08T02:07:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:31.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:31.429222+0000) 2022-09-08T02:07:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:32.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:32.486799+0000) 2022-09-08T02:07:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:32.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:32.429476+0000) 2022-09-08T02:07:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:33 smithi055 ceph-mon[97033]: pgmap v8677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:07:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:33 smithi059 ceph-mon[97450]: pgmap v8677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:07:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:33.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:33.429713+0000) 2022-09-08T02:07:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:33.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:33.486996+0000) 2022-09-08T02:07:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:34.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:34.429995+0000) 2022-09-08T02:07:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:34.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:34.487179+0000) 2022-09-08T02:07:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:35 smithi059 ceph-mon[97450]: pgmap v8678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:07:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:35.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:35.487403+0000) 2022-09-08T02:07:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:35.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:35.430251+0000) 2022-09-08T02:07:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:35 smithi055 ceph-mon[97033]: pgmap v8678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:07:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:36.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:36.487623+0000) 2022-09-08T02:07:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:36.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:36.430483+0000) 2022-09-08T02:07:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:37 smithi059 ceph-mon[97450]: pgmap v8679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:07:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:37.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:37.487921+0000) 2022-09-08T02:07:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:37.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:37.430741+0000) 2022-09-08T02:07:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:37 smithi055 ceph-mon[97033]: pgmap v8679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:07:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:38.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:38.488049+0000) 2022-09-08T02:07:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:38.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:38.430986+0000) 2022-09-08T02:07:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:39 smithi059 ceph-mon[97450]: pgmap v8680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:07:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:39.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:39.431108+0000) 2022-09-08T02:07:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:39.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:39.488274+0000) 2022-09-08T02:07:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:39 smithi055 ceph-mon[97033]: pgmap v8680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:07:40.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:07:39.999+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:07:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:40.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:40.431354+0000) 2022-09-08T02:07:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:40.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:40.488466+0000) 2022-09-08T02:07:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:41 smithi059 ceph-mon[97450]: pgmap v8681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T02:07:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:41.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:41.488699+0000) 2022-09-08T02:07:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:41.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:41.431577+0000) 2022-09-08T02:07:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:41 smithi055 ceph-mon[97033]: pgmap v8681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T02:07:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:42.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:42.488943+0000) 2022-09-08T02:07:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:42.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:42.431787+0000) 2022-09-08T02:07:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:43 smithi059 ceph-mon[97450]: pgmap v8682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:07:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:43.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:43.489139+0000) 2022-09-08T02:07:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:43.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:43.432039+0000) 2022-09-08T02:07:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:43 smithi055 ceph-mon[97033]: pgmap v8682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:07:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:44.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:44.489305+0000) 2022-09-08T02:07:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:44.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:44.432210+0000) 2022-09-08T02:07:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:45 smithi059 ceph-mon[97450]: pgmap v8683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:07:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:45.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:45.489503+0000) 2022-09-08T02:07:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:45.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:45.432458+0000) 2022-09-08T02:07:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:45 smithi055 ceph-mon[97033]: pgmap v8683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:07:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:46.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:46.489726+0000) 2022-09-08T02:07:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:46.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:46.432715+0000) 2022-09-08T02:07:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:47 smithi059 ceph-mon[97450]: pgmap v8684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:07:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:47.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:47.489941+0000) 2022-09-08T02:07:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:47.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:47.432869+0000) 2022-09-08T02:07:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:47 smithi055 ceph-mon[97033]: pgmap v8684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:07:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:48.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:48.490173+0000) 2022-09-08T02:07:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:48.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:48.433089+0000) 2022-09-08T02:07:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:49 smithi059 ceph-mon[97450]: pgmap v8685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:07:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:49.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:49.490420+0000) 2022-09-08T02:07:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:49.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:49.433365+0000) 2022-09-08T02:07:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:49 smithi055 ceph-mon[97033]: pgmap v8685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:07:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:50.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:50.490607+0000) 2022-09-08T02:07:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:50.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:50.433588+0000) 2022-09-08T02:07:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:51 smithi059 ceph-mon[97450]: pgmap v8686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 5 op/s 2022-09-08T02:07:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:51.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:51.490843+0000) 2022-09-08T02:07:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:51.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:51.433804+0000) 2022-09-08T02:07:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:51 smithi055 ceph-mon[97033]: pgmap v8686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 5 op/s 2022-09-08T02:07:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:52.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:52.491134+0000) 2022-09-08T02:07:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:52.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:52.434006+0000) 2022-09-08T02:07:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:53 smithi059 ceph-mon[97450]: pgmap v8687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 6 op/s 2022-09-08T02:07:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:53.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:53.491373+0000) 2022-09-08T02:07:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:53.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:53.434220+0000) 2022-09-08T02:07:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:53 smithi055 ceph-mon[97033]: pgmap v8687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 6 op/s 2022-09-08T02:07:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:54.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:54.434438+0000) 2022-09-08T02:07:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:54.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:54.491664+0000) 2022-09-08T02:07:55.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:07:55.012+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:07:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:55 smithi059 ceph-mon[97450]: pgmap v8688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:07:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:55.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:55.491938+0000) 2022-09-08T02:07:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:55.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:55.434693+0000) 2022-09-08T02:07:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:55 smithi055 ceph-mon[97033]: pgmap v8688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:07:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:56.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:56.492171+0000) 2022-09-08T02:07:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:56.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:56.434871+0000) 2022-09-08T02:07:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:57 smithi059 ceph-mon[97450]: pgmap v8689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:07:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:07:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:07:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:07:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:57.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:57.435144+0000) 2022-09-08T02:07:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:57.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:57.492351+0000) 2022-09-08T02:07:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:57 smithi055 ceph-mon[97033]: pgmap v8689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:07:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:07:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:07:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:07:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:58.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:58.435395+0000) 2022-09-08T02:07:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:58.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:58.492512+0000) 2022-09-08T02:07:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:58 smithi059 ceph-mon[97450]: pgmap v8690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:07:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:07:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:07:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:58 smithi055 ceph-mon[97033]: pgmap v8690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:07:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:07:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:07:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:07:59.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:59.435580+0000) 2022-09-08T02:07:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:07:59.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:07:59.492763+0000) 2022-09-08T02:08:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:00.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:00.492958+0000) 2022-09-08T02:08:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:00.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:00.435743+0000) 2022-09-08T02:08:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:00 smithi059 ceph-mon[97450]: pgmap v8691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T02:08:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:00 smithi055 ceph-mon[97033]: pgmap v8691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T02:08:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:01.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:01.493151+0000) 2022-09-08T02:08:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:01.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:01.435902+0000) 2022-09-08T02:08:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:02.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:02.493412+0000) 2022-09-08T02:08:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:02.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:02.436070+0000) 2022-09-08T02:08:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:03 smithi055 ceph-mon[97033]: pgmap v8692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:08:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:03 smithi059 ceph-mon[97450]: pgmap v8692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:08:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:03.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:03.493724+0000) 2022-09-08T02:08:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:03.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:03.436233+0000) 2022-09-08T02:08:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:04.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:04.493987+0000) 2022-09-08T02:08:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:04.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:04.436444+0000) 2022-09-08T02:08:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:05 smithi055 ceph-mon[97033]: pgmap v8693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:08:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:08:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:08:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:05 smithi059 ceph-mon[97450]: pgmap v8693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:08:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:08:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:08:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:05.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:05.494216+0000) 2022-09-08T02:08:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:05.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:05.436662+0000) 2022-09-08T02:08:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:06.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:06.494440+0000) 2022-09-08T02:08:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:06.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:06.436776+0000) 2022-09-08T02:08:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:07 smithi055 ceph-mon[97033]: pgmap v8694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:08:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:07 smithi059 ceph-mon[97450]: pgmap v8694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:08:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:07.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:07.494685+0000) 2022-09-08T02:08:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:07.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:07.436932+0000) 2022-09-08T02:08:08.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:08.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:08.494945+0000) 2022-09-08T02:08:08.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:08.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:08.437137+0000) 2022-09-08T02:08:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:09 smithi055 ceph-mon[97033]: pgmap v8695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:08:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:09 smithi059 ceph-mon[97450]: pgmap v8695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:08:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:09.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:09.495158+0000) 2022-09-08T02:08:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:09.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:09.437302+0000) 2022-09-08T02:08:10.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:08:10.027+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:08:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:10 smithi055 ceph-mon[97033]: osdmap e625: 12 total, 11 up, 11 in 2022-09-08T02:08:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:10 smithi059 ceph-mon[97450]: osdmap e625: 12 total, 11 up, 11 in 2022-09-08T02:08:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:10.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:10.437488+0000) 2022-09-08T02:08:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:10.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:10.495298+0000) 2022-09-08T02:08:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:11 smithi055 ceph-mon[97033]: pgmap v8697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:08:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:11 smithi059 ceph-mon[97450]: pgmap v8697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:08:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:11.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:11.495440+0000) 2022-09-08T02:08:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:11.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:11.437623+0000) 2022-09-08T02:08:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:12.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:12.495608+0000) 2022-09-08T02:08:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:12.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:12.437803+0000) 2022-09-08T02:08:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:13 smithi055 ceph-mon[97033]: pgmap v8698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:08:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:13 smithi059 ceph-mon[97450]: pgmap v8698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:08:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:13.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:13.495738+0000) 2022-09-08T02:08:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:13.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:13.437992+0000) 2022-09-08T02:08:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:14.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:14.495915+0000) 2022-09-08T02:08:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:14.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:14.438219+0000) 2022-09-08T02:08:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:15 smithi055 ceph-mon[97033]: pgmap v8699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:08:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:15 smithi059 ceph-mon[97450]: pgmap v8699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:08:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:15.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:15.496066+0000) 2022-09-08T02:08:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:15.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:15.438432+0000) 2022-09-08T02:08:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:16.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:16.496264+0000) 2022-09-08T02:08:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:16.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:16.438660+0000) 2022-09-08T02:08:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:17 smithi055 ceph-mon[97033]: pgmap v8700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:08:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:17 smithi055 ceph-mon[97033]: osdmap e626: 12 total, 11 up, 11 in 2022-09-08T02:08:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:17 smithi059 ceph-mon[97450]: pgmap v8700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:08:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:17 smithi059 ceph-mon[97450]: osdmap e626: 12 total, 11 up, 11 in 2022-09-08T02:08:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:17.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:17.438907+0000) 2022-09-08T02:08:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:17.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:17.496490+0000) 2022-09-08T02:08:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:18.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:18.496648+0000) 2022-09-08T02:08:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:18.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:18.439140+0000) 2022-09-08T02:08:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:19 smithi055 ceph-mon[97033]: pgmap v8702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 243 B/s wr, 7 op/s 2022-09-08T02:08:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:19 smithi059 ceph-mon[97450]: pgmap v8702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 243 B/s wr, 7 op/s 2022-09-08T02:08:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:19.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:19.439352+0000) 2022-09-08T02:08:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:19.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:19.496808+0000) 2022-09-08T02:08:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:20.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:20.496953+0000) 2022-09-08T02:08:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:20.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:20.439542+0000) 2022-09-08T02:08:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:21 smithi055 ceph-mon[97033]: pgmap v8703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:08:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:21 smithi059 ceph-mon[97450]: pgmap v8703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:08:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:21.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:21.497143+0000) 2022-09-08T02:08:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:21.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:21.439759+0000) 2022-09-08T02:08:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:22.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:22.439944+0000) 2022-09-08T02:08:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:22.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:22.497292+0000) 2022-09-08T02:08:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:23 smithi055 ceph-mon[97033]: pgmap v8704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:08:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:23 smithi059 ceph-mon[97450]: pgmap v8704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:08:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:23.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:23.440125+0000) 2022-09-08T02:08:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:23.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:23.497466+0000) 2022-09-08T02:08:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:24.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:24.497624+0000) 2022-09-08T02:08:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:24.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:24.440335+0000) 2022-09-08T02:08:25.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:08:25.035+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:08:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:25 smithi055 ceph-mon[97033]: pgmap v8705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:08:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:25 smithi059 ceph-mon[97450]: pgmap v8705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:08:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:25.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:25.497751+0000) 2022-09-08T02:08:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:25.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:25.440559+0000) 2022-09-08T02:08:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:26.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:26.497926+0000) 2022-09-08T02:08:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:26.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:26.440776+0000) 2022-09-08T02:08:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:27 smithi055 ceph-mon[97033]: pgmap v8706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:08:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:27 smithi059 ceph-mon[97450]: pgmap v8706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:08:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:27.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:27.498091+0000) 2022-09-08T02:08:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:27.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:27.441029+0000) 2022-09-08T02:08:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:28.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:28.498303+0000) 2022-09-08T02:08:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:28.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:28.441283+0000) 2022-09-08T02:08:29.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:29 smithi055 ceph-mon[97033]: pgmap v8707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:08:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:29 smithi059 ceph-mon[97450]: pgmap v8707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:08:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:29.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:29.498554+0000) 2022-09-08T02:08:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:29.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:29.441514+0000) 2022-09-08T02:08:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:30.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:30.498788+0000) 2022-09-08T02:08:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:30.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:30.441738+0000) 2022-09-08T02:08:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:31 smithi055 ceph-mon[97033]: pgmap v8708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T02:08:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:31 smithi059 ceph-mon[97450]: pgmap v8708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T02:08:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:31.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:31.498966+0000) 2022-09-08T02:08:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:31.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:31.441941+0000) 2022-09-08T02:08:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:32.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:32.499139+0000) 2022-09-08T02:08:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:32.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:32.442084+0000) 2022-09-08T02:08:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:33 smithi055 ceph-mon[97033]: pgmap v8709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:08:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:33 smithi059 ceph-mon[97450]: pgmap v8709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:08:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:33.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:33.499388+0000) 2022-09-08T02:08:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:33.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:33.442253+0000) 2022-09-08T02:08:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:34.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:34.499594+0000) 2022-09-08T02:08:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:34.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:34.442472+0000) 2022-09-08T02:08:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:35 smithi055 ceph-mon[97033]: pgmap v8710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:08:35.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:35 smithi059 ceph-mon[97450]: pgmap v8710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:08:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:35.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:35.499813+0000) 2022-09-08T02:08:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:35.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:35.442735+0000) 2022-09-08T02:08:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:36.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:36.499983+0000) 2022-09-08T02:08:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:36.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:36.442913+0000) 2022-09-08T02:08:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:37 smithi055 ceph-mon[97033]: pgmap v8711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:08:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:37 smithi059 ceph-mon[97450]: pgmap v8711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:08:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:37.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:37.500239+0000) 2022-09-08T02:08:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:37.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:37.443155+0000) 2022-09-08T02:08:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:38.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:38.443380+0000) 2022-09-08T02:08:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:38.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:38.500468+0000) 2022-09-08T02:08:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:39 smithi059 ceph-mon[97450]: pgmap v8712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:08:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:39.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:39.500720+0000) 2022-09-08T02:08:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:39.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:39.443613+0000) 2022-09-08T02:08:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:39 smithi055 ceph-mon[97033]: pgmap v8712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:08:40.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:08:40.049+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:08:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:40.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:40.500958+0000) 2022-09-08T02:08:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:40.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:40.443877+0000) 2022-09-08T02:08:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:41 smithi059 ceph-mon[97450]: pgmap v8713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T02:08:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:41.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:41.501085+0000) 2022-09-08T02:08:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:41.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:41.444042+0000) 2022-09-08T02:08:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:41 smithi055 ceph-mon[97033]: pgmap v8713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T02:08:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:42.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:42.501251+0000) 2022-09-08T02:08:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:42.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:42.444245+0000) 2022-09-08T02:08:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:43 smithi059 ceph-mon[97450]: pgmap v8714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:08:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:43.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:43.501484+0000) 2022-09-08T02:08:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:43.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:43.444422+0000) 2022-09-08T02:08:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:43 smithi055 ceph-mon[97033]: pgmap v8714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:08:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:44.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:44.501740+0000) 2022-09-08T02:08:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:44.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:44.444651+0000) 2022-09-08T02:08:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:45 smithi059 ceph-mon[97450]: pgmap v8715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:08:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:45.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:45.501926+0000) 2022-09-08T02:08:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:45.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:45.444793+0000) 2022-09-08T02:08:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:45 smithi055 ceph-mon[97033]: pgmap v8715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:08:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:46.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:46.444998+0000) 2022-09-08T02:08:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:46.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:46.502090+0000) 2022-09-08T02:08:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:47 smithi059 ceph-mon[97450]: pgmap v8716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:08:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:47.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:47.502315+0000) 2022-09-08T02:08:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:47.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:47.445245+0000) 2022-09-08T02:08:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:47 smithi055 ceph-mon[97033]: pgmap v8716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:08:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:48.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:48.502530+0000) 2022-09-08T02:08:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:48.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:48.445474+0000) 2022-09-08T02:08:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:49 smithi059 ceph-mon[97450]: pgmap v8717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:08:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:49.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:49.502740+0000) 2022-09-08T02:08:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:49.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:49.445744+0000) 2022-09-08T02:08:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:49 smithi055 ceph-mon[97033]: pgmap v8717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:08:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:50.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:50.502970+0000) 2022-09-08T02:08:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:50.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:50.445954+0000) 2022-09-08T02:08:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:51 smithi059 ceph-mon[97450]: pgmap v8718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T02:08:51.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:51.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:51.503184+0000) 2022-09-08T02:08:51.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:51.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:51.446193+0000) 2022-09-08T02:08:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:51 smithi055 ceph-mon[97033]: pgmap v8718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T02:08:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:52.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:52.503415+0000) 2022-09-08T02:08:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:52.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:52.446444+0000) 2022-09-08T02:08:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:53 smithi059 ceph-mon[97450]: pgmap v8719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:08:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:53.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:53.503589+0000) 2022-09-08T02:08:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:53.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:53.446680+0000) 2022-09-08T02:08:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:53 smithi055 ceph-mon[97033]: pgmap v8719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:08:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:54.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:54.503757+0000) 2022-09-08T02:08:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:54.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:54.446900+0000) 2022-09-08T02:08:55.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:08:55.066+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:08:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:55 smithi059 ceph-mon[97450]: pgmap v8720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:08:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:55.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:55.503991+0000) 2022-09-08T02:08:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:55.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:55.447041+0000) 2022-09-08T02:08:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:55 smithi055 ceph-mon[97033]: pgmap v8720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:08:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:56.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:56.504241+0000) 2022-09-08T02:08:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:56.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:56.447289+0000) 2022-09-08T02:08:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:57 smithi059 ceph-mon[97450]: pgmap v8721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:08:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:57.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:57.504423+0000) 2022-09-08T02:08:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:57.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:57.447498+0000) 2022-09-08T02:08:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:57 smithi055 ceph-mon[97033]: pgmap v8721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:08:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:08:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:08:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:08:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:58.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:58.504700+0000) 2022-09-08T02:08:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:58.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:58.447696+0000) 2022-09-08T02:08:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:08:58.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:08:58.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:08:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:08:59.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:59.447913+0000) 2022-09-08T02:08:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:08:59.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:08:59.504963+0000) 2022-09-08T02:08:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:59 smithi055 ceph-mon[97033]: pgmap v8722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:08:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:08:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:08:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:09:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:59 smithi059 ceph-mon[97450]: pgmap v8722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:09:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:09:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:08:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:09:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:00.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:00.505247+0000) 2022-09-08T02:09:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:00.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:00.448102+0000) 2022-09-08T02:09:01.909 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:01.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:01.505537+0000) 2022-09-08T02:09:01.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:01.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:01.448311+0000) 2022-09-08T02:09:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:01 smithi055 ceph-mon[97033]: pgmap v8723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-08T02:09:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:01 smithi059 ceph-mon[97450]: pgmap v8723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-08T02:09:02.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:02.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:02.505836+0000) 2022-09-08T02:09:02.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:02.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:02.448482+0000) 2022-09-08T02:09:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:03.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:03.506004+0000) 2022-09-08T02:09:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:03.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:03.448758+0000) 2022-09-08T02:09:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:03 smithi055 ceph-mon[97033]: pgmap v8724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 6 op/s 2022-09-08T02:09:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:03 smithi059 ceph-mon[97450]: pgmap v8724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 6 op/s 2022-09-08T02:09:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:04.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:04.506227+0000) 2022-09-08T02:09:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:04.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:04.448951+0000) 2022-09-08T02:09:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:05.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:05.506455+0000) 2022-09-08T02:09:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:05.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:05.449150+0000) 2022-09-08T02:09:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:05 smithi055 ceph-mon[97033]: pgmap v8725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T02:09:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:09:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:09:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:05 smithi059 ceph-mon[97450]: pgmap v8725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T02:09:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:09:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:09:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:06.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:06.506651+0000) 2022-09-08T02:09:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:06.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:06.449366+0000) 2022-09-08T02:09:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:07.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:07.506826+0000) 2022-09-08T02:09:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:07.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:07.449705+0000) 2022-09-08T02:09:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:07 smithi055 ceph-mon[97033]: pgmap v8726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:09:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:07 smithi059 ceph-mon[97450]: pgmap v8726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:09:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:08.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:08.507061+0000) 2022-09-08T02:09:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:08.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:08.449903+0000) 2022-09-08T02:09:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:09.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:09.450144+0000) 2022-09-08T02:09:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:09.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:09.507306+0000) 2022-09-08T02:09:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:09 smithi055 ceph-mon[97033]: pgmap v8727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:09:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:09 smithi055 ceph-mon[97033]: osdmap e627: 12 total, 11 up, 11 in 2022-09-08T02:09:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:09 smithi059 ceph-mon[97450]: pgmap v8727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:09:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:09 smithi059 ceph-mon[97450]: osdmap e627: 12 total, 11 up, 11 in 2022-09-08T02:09:10.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:09:10.071+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:09:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:10.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:10.507546+0000) 2022-09-08T02:09:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:10.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:10.450371+0000) 2022-09-08T02:09:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:11.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:11.507778+0000) 2022-09-08T02:09:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:11.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:11.450531+0000) 2022-09-08T02:09:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:11 smithi055 ceph-mon[97033]: pgmap v8729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:09:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:11 smithi059 ceph-mon[97450]: pgmap v8729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:09:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:12.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:12.507962+0000) 2022-09-08T02:09:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:12.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:12.450749+0000) 2022-09-08T02:09:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:13.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:13.508132+0000) 2022-09-08T02:09:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:13.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:13.450924+0000) 2022-09-08T02:09:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:13 smithi055 ceph-mon[97033]: pgmap v8730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:09:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:13 smithi059 ceph-mon[97450]: pgmap v8730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:09:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:14.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:14.451189+0000) 2022-09-08T02:09:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:14.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:14.508305+0000) 2022-09-08T02:09:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:15.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:15.451443+0000) 2022-09-08T02:09:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:15.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:15.508522+0000) 2022-09-08T02:09:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:15 smithi055 ceph-mon[97033]: pgmap v8731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:09:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:15 smithi059 ceph-mon[97450]: pgmap v8731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:09:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:16.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:16.508746+0000) 2022-09-08T02:09:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:16.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:16.451725+0000) 2022-09-08T02:09:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:17.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:17.451954+0000) 2022-09-08T02:09:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:17.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:17.508965+0000) 2022-09-08T02:09:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:17 smithi055 ceph-mon[97033]: pgmap v8732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:09:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:17 smithi055 ceph-mon[97033]: osdmap e628: 12 total, 11 up, 11 in 2022-09-08T02:09:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:17 smithi059 ceph-mon[97450]: pgmap v8732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:09:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:17 smithi059 ceph-mon[97450]: osdmap e628: 12 total, 11 up, 11 in 2022-09-08T02:09:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:18.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:18.509206+0000) 2022-09-08T02:09:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:18.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:18.452161+0000) 2022-09-08T02:09:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:19.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:19.509424+0000) 2022-09-08T02:09:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:19.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:19.452397+0000) 2022-09-08T02:09:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:19 smithi055 ceph-mon[97033]: pgmap v8734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-08T02:09:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:19 smithi059 ceph-mon[97450]: pgmap v8734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-08T02:09:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:20.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:20.509561+0000) 2022-09-08T02:09:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:20.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:20.452564+0000) 2022-09-08T02:09:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:21.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:21.509802+0000) 2022-09-08T02:09:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:21.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:21.452784+0000) 2022-09-08T02:09:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:21 smithi055 ceph-mon[97033]: pgmap v8735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:09:22.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:21 smithi059 ceph-mon[97450]: pgmap v8735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:09:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:22.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:22.509959+0000) 2022-09-08T02:09:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:22.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:22.452915+0000) 2022-09-08T02:09:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:23.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:23.510163+0000) 2022-09-08T02:09:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:23.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:23.453050+0000) 2022-09-08T02:09:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:23 smithi055 ceph-mon[97033]: pgmap v8736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:09:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:23 smithi059 ceph-mon[97450]: pgmap v8736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:09:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:24.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:24.510381+0000) 2022-09-08T02:09:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:24.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:24.453248+0000) 2022-09-08T02:09:25.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:09:25.086+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:09:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:25.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:25.510543+0000) 2022-09-08T02:09:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:25.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:25.453438+0000) 2022-09-08T02:09:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:25 smithi055 ceph-mon[97033]: pgmap v8737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:09:26.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:25 smithi059 ceph-mon[97450]: pgmap v8737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:09:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:26.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:26.510743+0000) 2022-09-08T02:09:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:26.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:26.453673+0000) 2022-09-08T02:09:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:27.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:27.510972+0000) 2022-09-08T02:09:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:27.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:27.453973+0000) 2022-09-08T02:09:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:27 smithi055 ceph-mon[97033]: pgmap v8738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:09:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:27 smithi059 ceph-mon[97450]: pgmap v8738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:09:28.511 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:28.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:28.454221+0000) 2022-09-08T02:09:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:28.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:28.511200+0000) 2022-09-08T02:09:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:29 smithi059 ceph-mon[97450]: pgmap v8739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:09:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:29 smithi059 ceph-mon[97450]: pgmap v8740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:09:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:09:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:29 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T02:09:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:29 smithi055 ceph-mon[97033]: pgmap v8739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:09:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:29 smithi055 ceph-mon[97033]: pgmap v8740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:09:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:09:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:29 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T02:09:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:29.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:29.511440+0000) 2022-09-08T02:09:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:29.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:29.454394+0000) 2022-09-08T02:09:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:30.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:30.511723+0000) 2022-09-08T02:09:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:30.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:30.454677+0000) 2022-09-08T02:09:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:31 smithi059 ceph-mon[97450]: pgmap v8741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 7 op/s 2022-09-08T02:09:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:31 smithi055 ceph-mon[97033]: pgmap v8741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 7 op/s 2022-09-08T02:09:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:31.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:31.511947+0000) 2022-09-08T02:09:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:31.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:31.454893+0000) 2022-09-08T02:09:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:32.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:32.512134+0000) 2022-09-08T02:09:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:32.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:32.455055+0000) 2022-09-08T02:09:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:33 smithi055 ceph-mon[97033]: pgmap v8742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:09:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:33 smithi059 ceph-mon[97450]: pgmap v8742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:09:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:33.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:33.512328+0000) 2022-09-08T02:09:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:33.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:33.455218+0000) 2022-09-08T02:09:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:34.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:34.512531+0000) 2022-09-08T02:09:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:34.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:34.455427+0000) 2022-09-08T02:09:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:35 smithi055 ceph-mon[97033]: pgmap v8743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:09:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:35 smithi059 ceph-mon[97450]: pgmap v8743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:09:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:35.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:35.512760+0000) 2022-09-08T02:09:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:35.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:35.455732+0000) 2022-09-08T02:09:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:36.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:36.512951+0000) 2022-09-08T02:09:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:36.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:36.455947+0000) 2022-09-08T02:09:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:37 smithi055 ceph-mon[97033]: pgmap v8744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:09:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:37 smithi059 ceph-mon[97450]: pgmap v8744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:09:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:37.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:37.513171+0000) 2022-09-08T02:09:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:37.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:37.456150+0000) 2022-09-08T02:09:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:38.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:38.513410+0000) 2022-09-08T02:09:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:38.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:38.456401+0000) 2022-09-08T02:09:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:39 smithi055 ceph-mon[97033]: pgmap v8745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:09:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:39 smithi059 ceph-mon[97450]: pgmap v8745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:09:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:39.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:39.513624+0000) 2022-09-08T02:09:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:39.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:39.456693+0000) 2022-09-08T02:09:40.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:09:40.088+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:09:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:40.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:40.513875+0000) 2022-09-08T02:09:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:40.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:40.456921+0000) 2022-09-08T02:09:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:41 smithi055 ceph-mon[97033]: pgmap v8746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:09:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:41 smithi059 ceph-mon[97450]: pgmap v8746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:09:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:41.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:41.514025+0000) 2022-09-08T02:09:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:41.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:41.457058+0000) 2022-09-08T02:09:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:42.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:42.514240+0000) 2022-09-08T02:09:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:42.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:42.457252+0000) 2022-09-08T02:09:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:43 smithi055 ceph-mon[97033]: pgmap v8747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:09:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:43 smithi059 ceph-mon[97450]: pgmap v8747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:09:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:43.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:43.514404+0000) 2022-09-08T02:09:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:43.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:43.457515+0000) 2022-09-08T02:09:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:44.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:44.514620+0000) 2022-09-08T02:09:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:44.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:44.457731+0000) 2022-09-08T02:09:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:45 smithi055 ceph-mon[97033]: pgmap v8748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:09:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:45 smithi059 ceph-mon[97450]: pgmap v8748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:09:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:45.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:45.514870+0000) 2022-09-08T02:09:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:45.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:45.457959+0000) 2022-09-08T02:09:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:46.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:46.515026+0000) 2022-09-08T02:09:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:46.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:46.458127+0000) 2022-09-08T02:09:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:47 smithi055 ceph-mon[97033]: pgmap v8749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:09:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:47 smithi059 ceph-mon[97450]: pgmap v8749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:09:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:47.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:47.515234+0000) 2022-09-08T02:09:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:47.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:47.458349+0000) 2022-09-08T02:09:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:48.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:48.515480+0000) 2022-09-08T02:09:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:48.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:48.458632+0000) 2022-09-08T02:09:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:49 smithi055 ceph-mon[97033]: pgmap v8750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:09:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:49 smithi059 ceph-mon[97450]: pgmap v8750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:09:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:49.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:49.515759+0000) 2022-09-08T02:09:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:49.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:49.458882+0000) 2022-09-08T02:09:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:50.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:50.515978+0000) 2022-09-08T02:09:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:50.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:50.459090+0000) 2022-09-08T02:09:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:51 smithi055 ceph-mon[97033]: pgmap v8751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:09:51.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:51 smithi059 ceph-mon[97450]: pgmap v8751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:09:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:51.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:51.516249+0000) 2022-09-08T02:09:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:51.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:51.459361+0000) 2022-09-08T02:09:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:52.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:52.516465+0000) 2022-09-08T02:09:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:52.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:52.459524+0000) 2022-09-08T02:09:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:53 smithi055 ceph-mon[97033]: pgmap v8752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:09:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:53 smithi059 ceph-mon[97450]: pgmap v8752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:09:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:53.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:53.516725+0000) 2022-09-08T02:09:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:53.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:53.459782+0000) 2022-09-08T02:09:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:54.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:54.516940+0000) 2022-09-08T02:09:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:54.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:54.460001+0000) 2022-09-08T02:09:55.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:09:55.097+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:09:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:55 smithi055 ceph-mon[97033]: pgmap v8753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:09:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:55 smithi059 ceph-mon[97450]: pgmap v8753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:09:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:55.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:55.460249+0000) 2022-09-08T02:09:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:55.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:55.517140+0000) 2022-09-08T02:09:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:56.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:56.460493+0000) 2022-09-08T02:09:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:56.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:56.517355+0000) 2022-09-08T02:09:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:57 smithi055 ceph-mon[97033]: pgmap v8754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:09:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:57 smithi059 ceph-mon[97450]: pgmap v8754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:09:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:57.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:57.517577+0000) 2022-09-08T02:09:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:57.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:57.460759+0000) 2022-09-08T02:09:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:58.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:58.517710+0000) 2022-09-08T02:09:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:58.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:58.461020+0000) 2022-09-08T02:09:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:09:59 smithi055 ceph-mon[97033]: pgmap v8755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:09:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:09:59 smithi059 ceph-mon[97450]: pgmap v8755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:09:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:09:59.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:59.517934+0000) 2022-09-08T02:09:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:09:59.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:09:59.461260+0000) 2022-09-08T02:10:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T02:10:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T02:10:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T02:10:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T02:10:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T02:10:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:00.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:00.518145+0000) 2022-09-08T02:10:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:00.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:00.461562+0000) 2022-09-08T02:10:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T02:10:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T02:10:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T02:10:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T02:10:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T02:10:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:01 smithi059 ceph-mon[97450]: pgmap v8756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:10:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:01.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:01.518356+0000) 2022-09-08T02:10:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:01.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:01.461794+0000) 2022-09-08T02:10:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:01 smithi055 ceph-mon[97033]: pgmap v8756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:10:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:02.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:02.518520+0000) 2022-09-08T02:10:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:02.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:02.461980+0000) 2022-09-08T02:10:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:03 smithi059 ceph-mon[97450]: pgmap v8757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:03.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:03.518701+0000) 2022-09-08T02:10:03.937 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:03.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:03.462203+0000) 2022-09-08T02:10:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:03 smithi055 ceph-mon[97033]: pgmap v8757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:04.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:04.518919+0000) 2022-09-08T02:10:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:04.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:04.462455+0000) 2022-09-08T02:10:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:05 smithi059 ceph-mon[97450]: pgmap v8758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:10:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:10:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:05.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:05.519098+0000) 2022-09-08T02:10:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:05.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:05.462664+0000) 2022-09-08T02:10:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:05 smithi055 ceph-mon[97033]: pgmap v8758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:10:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:10:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:06.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:06.519327+0000) 2022-09-08T02:10:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:06.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:06.462894+0000) 2022-09-08T02:10:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:07 smithi059 ceph-mon[97450]: pgmap v8759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:10:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:07.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:07.519548+0000) 2022-09-08T02:10:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:07.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:07.463118+0000) 2022-09-08T02:10:07.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:07 smithi055 ceph-mon[97033]: pgmap v8759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:10:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:08.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:08.519702+0000) 2022-09-08T02:10:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:08.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:08.463336+0000) 2022-09-08T02:10:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:09 smithi059 ceph-mon[97450]: pgmap v8760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:10:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:09.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:09.519835+0000) 2022-09-08T02:10:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:09.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:09.463580+0000) 2022-09-08T02:10:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:09 smithi055 ceph-mon[97033]: pgmap v8760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:10:10.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:10:10.105+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:10:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:10 smithi059 ceph-mon[97450]: osdmap e629: 12 total, 11 up, 11 in 2022-09-08T02:10:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:10.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:10.520029+0000) 2022-09-08T02:10:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:10.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:10.463739+0000) 2022-09-08T02:10:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:10 smithi055 ceph-mon[97033]: osdmap e629: 12 total, 11 up, 11 in 2022-09-08T02:10:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:11 smithi059 ceph-mon[97450]: pgmap v8762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:10:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:11.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:11.520177+0000) 2022-09-08T02:10:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:11.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:11.463962+0000) 2022-09-08T02:10:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:11 smithi055 ceph-mon[97033]: pgmap v8762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:10:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:12.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:12.520406+0000) 2022-09-08T02:10:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:12.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:12.464181+0000) 2022-09-08T02:10:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:13 smithi059 ceph-mon[97450]: pgmap v8763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:10:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:13.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:13.520541+0000) 2022-09-08T02:10:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:13.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:13.464361+0000) 2022-09-08T02:10:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:13 smithi055 ceph-mon[97033]: pgmap v8763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:10:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:14.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:14.520689+0000) 2022-09-08T02:10:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:14.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:14.464526+0000) 2022-09-08T02:10:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:15 smithi059 ceph-mon[97450]: pgmap v8764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:10:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:15.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:15.520841+0000) 2022-09-08T02:10:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:15.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:15.464800+0000) 2022-09-08T02:10:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:15 smithi055 ceph-mon[97033]: pgmap v8764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:10:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:16.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:16.521011+0000) 2022-09-08T02:10:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:16.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:16.465050+0000) 2022-09-08T02:10:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:17 smithi059 ceph-mon[97450]: pgmap v8765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:10:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:17 smithi059 ceph-mon[97450]: osdmap e630: 12 total, 11 up, 11 in 2022-09-08T02:10:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:17.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:17.521223+0000) 2022-09-08T02:10:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:17.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:17.465275+0000) 2022-09-08T02:10:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:17 smithi055 ceph-mon[97033]: pgmap v8765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:10:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:17 smithi055 ceph-mon[97033]: osdmap e630: 12 total, 11 up, 11 in 2022-09-08T02:10:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:18.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:18.521380+0000) 2022-09-08T02:10:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:18.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:18.465486+0000) 2022-09-08T02:10:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:19 smithi059 ceph-mon[97450]: pgmap v8767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 233 B/s wr, 7 op/s 2022-09-08T02:10:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:19.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:19.521577+0000) 2022-09-08T02:10:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:19.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:19.465757+0000) 2022-09-08T02:10:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:19 smithi055 ceph-mon[97033]: pgmap v8767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 233 B/s wr, 7 op/s 2022-09-08T02:10:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:20.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:20.521836+0000) 2022-09-08T02:10:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:20.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:20.466003+0000) 2022-09-08T02:10:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:21 smithi059 ceph-mon[97450]: pgmap v8768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:10:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:21.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:21.522091+0000) 2022-09-08T02:10:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:21.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:21.466164+0000) 2022-09-08T02:10:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:21 smithi055 ceph-mon[97033]: pgmap v8768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:10:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:22.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:22.522274+0000) 2022-09-08T02:10:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:22.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:22.466391+0000) 2022-09-08T02:10:23.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:23 smithi059 ceph-mon[97450]: pgmap v8769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:10:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:23.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:23.522497+0000) 2022-09-08T02:10:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:23.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:23.466665+0000) 2022-09-08T02:10:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:23 smithi055 ceph-mon[97033]: pgmap v8769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:10:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:24.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:24.522689+0000) 2022-09-08T02:10:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:24.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:24.466901+0000) 2022-09-08T02:10:25.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:10:25.118+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:10:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:25.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:25.522838+0000) 2022-09-08T02:10:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:25.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:25.467132+0000) 2022-09-08T02:10:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:25 smithi055 ceph-mon[97033]: pgmap v8770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:10:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:25 smithi059 ceph-mon[97450]: pgmap v8770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:10:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:26.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:26.523060+0000) 2022-09-08T02:10:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:26.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:26.467319+0000) 2022-09-08T02:10:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:27.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:27.523299+0000) 2022-09-08T02:10:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:27.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:27.467539+0000) 2022-09-08T02:10:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:27 smithi055 ceph-mon[97033]: pgmap v8771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:10:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:27 smithi059 ceph-mon[97450]: pgmap v8771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:10:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:28.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:28.523532+0000) 2022-09-08T02:10:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:28.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:28.467772+0000) 2022-09-08T02:10:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:10:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:10:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:10:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:10:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:10:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:10:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:29.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:29.523742+0000) 2022-09-08T02:10:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:29.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:29.468024+0000) 2022-09-08T02:10:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:29 smithi055 ceph-mon[97033]: pgmap v8772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T02:10:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:29 smithi059 ceph-mon[97450]: pgmap v8772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T02:10:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:30.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:30.523965+0000) 2022-09-08T02:10:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:30.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:30.468258+0000) 2022-09-08T02:10:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:31.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:31.524219+0000) 2022-09-08T02:10:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:31.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:31.468488+0000) 2022-09-08T02:10:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:31 smithi055 ceph-mon[97033]: pgmap v8773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:10:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:31 smithi059 ceph-mon[97450]: pgmap v8773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:10:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:32.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:32.524397+0000) 2022-09-08T02:10:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:32.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:32.468726+0000) 2022-09-08T02:10:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:33.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:33.524604+0000) 2022-09-08T02:10:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:33.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:33.468872+0000) 2022-09-08T02:10:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:33 smithi055 ceph-mon[97033]: pgmap v8774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:33 smithi059 ceph-mon[97450]: pgmap v8774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:34.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:34.524832+0000) 2022-09-08T02:10:34.931 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:34.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:34.469100+0000) 2022-09-08T02:10:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:35.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:35.524999+0000) 2022-09-08T02:10:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:35.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:35.469346+0000) 2022-09-08T02:10:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:35 smithi055 ceph-mon[97033]: pgmap v8775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:35 smithi059 ceph-mon[97450]: pgmap v8775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:36.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:36.525256+0000) 2022-09-08T02:10:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:36.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:36.469507+0000) 2022-09-08T02:10:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:37.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:37.525449+0000) 2022-09-08T02:10:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:37.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:37.469677+0000) 2022-09-08T02:10:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:37 smithi055 ceph-mon[97033]: pgmap v8776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:10:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:37 smithi059 ceph-mon[97450]: pgmap v8776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:10:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:38.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:38.525718+0000) 2022-09-08T02:10:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:38.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:38.469877+0000) 2022-09-08T02:10:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:39.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:39.525931+0000) 2022-09-08T02:10:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:39.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:39.470148+0000) 2022-09-08T02:10:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:39 smithi055 ceph-mon[97033]: pgmap v8777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:39 smithi059 ceph-mon[97450]: pgmap v8777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:40.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:10:40.131+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:10:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:40.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:40.526080+0000) 2022-09-08T02:10:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:40.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:40.470380+0000) 2022-09-08T02:10:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:41.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:41.470672+0000) 2022-09-08T02:10:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:41.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:41.526312+0000) 2022-09-08T02:10:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:41 smithi055 ceph-mon[97033]: pgmap v8778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:10:42.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:41 smithi059 ceph-mon[97450]: pgmap v8778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:10:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:42.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:42.470934+0000) 2022-09-08T02:10:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:42.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:42.526557+0000) 2022-09-08T02:10:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:43.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:43.526769+0000) 2022-09-08T02:10:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:43.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:43.471202+0000) 2022-09-08T02:10:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:43 smithi055 ceph-mon[97033]: pgmap v8779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:43 smithi059 ceph-mon[97450]: pgmap v8779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:44.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:44.526964+0000) 2022-09-08T02:10:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:44.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:44.471440+0000) 2022-09-08T02:10:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:45.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:45.527162+0000) 2022-09-08T02:10:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:45.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:45.471722+0000) 2022-09-08T02:10:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:45 smithi055 ceph-mon[97033]: pgmap v8780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:45 smithi059 ceph-mon[97450]: pgmap v8780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:46.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:46.527396+0000) 2022-09-08T02:10:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:46.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:46.471982+0000) 2022-09-08T02:10:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:47.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:47.527658+0000) 2022-09-08T02:10:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:47.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:47.472259+0000) 2022-09-08T02:10:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:47 smithi055 ceph-mon[97033]: pgmap v8781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:10:48.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:47 smithi059 ceph-mon[97450]: pgmap v8781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:10:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:48.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:48.527911+0000) 2022-09-08T02:10:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:48.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:48.472488+0000) 2022-09-08T02:10:49.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:49.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:49.472752+0000) 2022-09-08T02:10:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:49.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:49.528100+0000) 2022-09-08T02:10:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:49 smithi055 ceph-mon[97033]: pgmap v8782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:49 smithi059 ceph-mon[97450]: pgmap v8782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:50.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:50.528280+0000) 2022-09-08T02:10:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:50.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:50.472943+0000) 2022-09-08T02:10:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:51.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:51.528499+0000) 2022-09-08T02:10:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:51.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:51.473138+0000) 2022-09-08T02:10:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:51 smithi055 ceph-mon[97033]: pgmap v8783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:10:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:51 smithi059 ceph-mon[97450]: pgmap v8783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:10:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:52.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:52.528759+0000) 2022-09-08T02:10:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:52.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:52.473357+0000) 2022-09-08T02:10:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:53.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:53.528926+0000) 2022-09-08T02:10:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:53.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:53.473528+0000) 2022-09-08T02:10:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:53 smithi055 ceph-mon[97033]: pgmap v8784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:53 smithi059 ceph-mon[97450]: pgmap v8784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:54.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:54.529084+0000) 2022-09-08T02:10:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:54.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:54.473686+0000) 2022-09-08T02:10:55.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:10:55.146+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:10:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:55.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:55.473873+0000) 2022-09-08T02:10:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:55.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:55.529299+0000) 2022-09-08T02:10:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:55 smithi055 ceph-mon[97033]: pgmap v8785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:55 smithi059 ceph-mon[97450]: pgmap v8785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:56.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:56.529478+0000) 2022-09-08T02:10:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:56.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:56.474129+0000) 2022-09-08T02:10:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:57.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:57.529784+0000) 2022-09-08T02:10:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:57.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:57.474360+0000) 2022-09-08T02:10:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:57 smithi055 ceph-mon[97033]: pgmap v8786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:10:58.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:57 smithi059 ceph-mon[97450]: pgmap v8786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:10:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:58.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:58.474594+0000) 2022-09-08T02:10:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:58.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:58.529952+0000) 2022-09-08T02:10:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:59 smithi059 ceph-mon[97450]: pgmap v8787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:59 smithi059 ceph-mon[97450]: pgmap v8788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:10:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:10:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:10:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:10:59.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:59.530078+0000) 2022-09-08T02:10:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:10:59.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:10:59.474899+0000) 2022-09-08T02:10:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:59 smithi055 ceph-mon[97033]: pgmap v8787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:10:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:59 smithi055 ceph-mon[97033]: pgmap v8788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:10:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:10:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:11:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:00.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:00.530221+0000) 2022-09-08T02:11:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:00.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:00.475049+0000) 2022-09-08T02:11:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:01.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:01.530449+0000) 2022-09-08T02:11:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:01.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:01.475241+0000) 2022-09-08T02:11:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:01 smithi055 ceph-mon[97033]: pgmap v8789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:11:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:01 smithi059 ceph-mon[97450]: pgmap v8789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:11:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:02.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:02.530692+0000) 2022-09-08T02:11:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:02.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:02.475463+0000) 2022-09-08T02:11:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:03 smithi055 ceph-mon[97033]: pgmap v8790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:11:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:03 smithi059 ceph-mon[97450]: pgmap v8790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:11:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:03.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:03.530840+0000) 2022-09-08T02:11:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:03.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:03.475748+0000) 2022-09-08T02:11:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:04.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:04.531031+0000) 2022-09-08T02:11:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:04.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:04.475997+0000) 2022-09-08T02:11:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:05.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:05.531234+0000) 2022-09-08T02:11:05.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:05.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:05.476210+0000) 2022-09-08T02:11:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:05 smithi055 ceph-mon[97033]: pgmap v8791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:11:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:11:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:11:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:05 smithi059 ceph-mon[97450]: pgmap v8791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:11:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:11:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:11:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:06.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:06.531417+0000) 2022-09-08T02:11:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:06.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:06.476430+0000) 2022-09-08T02:11:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:07.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:07.531603+0000) 2022-09-08T02:11:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:07.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:07.476601+0000) 2022-09-08T02:11:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:07 smithi055 ceph-mon[97033]: pgmap v8792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T02:11:08.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:07 smithi059 ceph-mon[97450]: pgmap v8792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T02:11:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:08.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:08.531821+0000) 2022-09-08T02:11:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:08.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:08.476759+0000) 2022-09-08T02:11:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:09.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:09.531964+0000) 2022-09-08T02:11:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:09.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:09.476901+0000) 2022-09-08T02:11:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:09 smithi055 ceph-mon[97033]: pgmap v8793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T02:11:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:09 smithi055 ceph-mon[97033]: osdmap e631: 12 total, 11 up, 11 in 2022-09-08T02:11:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:09 smithi059 ceph-mon[97450]: pgmap v8793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T02:11:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:09 smithi059 ceph-mon[97450]: osdmap e631: 12 total, 11 up, 11 in 2022-09-08T02:11:10.477 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:11:10.150+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:11:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:10.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:10.532145+0000) 2022-09-08T02:11:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:10.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:10.477030+0000) 2022-09-08T02:11:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:11.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:11.532355+0000) 2022-09-08T02:11:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:11.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:11.477237+0000) 2022-09-08T02:11:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:11 smithi055 ceph-mon[97033]: pgmap v8795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:11:12.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:11 smithi059 ceph-mon[97450]: pgmap v8795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:11:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:12.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:12.532587+0000) 2022-09-08T02:11:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:12.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:12.477387+0000) 2022-09-08T02:11:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:13 smithi055 ceph-mon[97033]: osdmap e632: 12 total, 11 up, 11 in 2022-09-08T02:11:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:13 smithi055 ceph-mon[97033]: pgmap v8797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T02:11:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:13 smithi059 ceph-mon[97450]: osdmap e632: 12 total, 11 up, 11 in 2022-09-08T02:11:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:13 smithi059 ceph-mon[97450]: pgmap v8797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T02:11:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:13.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:13.532783+0000) 2022-09-08T02:11:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:13.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:13.477520+0000) 2022-09-08T02:11:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:14.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:14.477727+0000) 2022-09-08T02:11:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:14.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:14.533002+0000) 2022-09-08T02:11:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:15.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:15.477903+0000) 2022-09-08T02:11:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:15.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:15.533221+0000) 2022-09-08T02:11:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:15 smithi055 ceph-mon[97033]: pgmap v8798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:11:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:15 smithi059 ceph-mon[97450]: pgmap v8798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:11:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:16.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:16.533357+0000) 2022-09-08T02:11:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:16.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:16.478055+0000) 2022-09-08T02:11:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:17.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:17.533590+0000) 2022-09-08T02:11:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:17.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:17.478250+0000) 2022-09-08T02:11:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:17 smithi055 ceph-mon[97033]: pgmap v8799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:11:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:17 smithi059 ceph-mon[97450]: pgmap v8799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:11:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:18.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:18.533723+0000) 2022-09-08T02:11:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:18.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:18.478475+0000) 2022-09-08T02:11:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:19.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:19.533932+0000) 2022-09-08T02:11:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:19.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:19.478693+0000) 2022-09-08T02:11:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:19 smithi055 ceph-mon[97033]: pgmap v8800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T02:11:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:19 smithi059 ceph-mon[97450]: pgmap v8800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T02:11:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:20.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:20.534094+0000) 2022-09-08T02:11:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:20.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:20.478826+0000) 2022-09-08T02:11:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:21.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:21.534322+0000) 2022-09-08T02:11:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:21.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:21.478990+0000) 2022-09-08T02:11:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:21 smithi055 ceph-mon[97033]: pgmap v8801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:11:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:21 smithi059 ceph-mon[97450]: pgmap v8801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:11:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:22.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:22.534487+0000) 2022-09-08T02:11:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:22.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:22.479160+0000) 2022-09-08T02:11:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:23 smithi055 ceph-mon[97033]: pgmap v8802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 101 B/s wr, 6 op/s 2022-09-08T02:11:23.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:23 smithi059 ceph-mon[97450]: pgmap v8802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 101 B/s wr, 6 op/s 2022-09-08T02:11:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:23.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:23.534744+0000) 2022-09-08T02:11:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:23.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:23.479378+0000) 2022-09-08T02:11:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:24.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:24.534868+0000) 2022-09-08T02:11:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:24.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:24.479579+0000) 2022-09-08T02:11:25.480 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:11:25.159+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:11:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:25.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:25.535036+0000) 2022-09-08T02:11:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:25.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:25.479767+0000) 2022-09-08T02:11:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:25 smithi055 ceph-mon[97033]: pgmap v8803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T02:11:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:25 smithi059 ceph-mon[97450]: pgmap v8803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T02:11:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:26.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:26.535304+0000) 2022-09-08T02:11:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:26.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:26.480007+0000) 2022-09-08T02:11:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:27.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:27.480268+0000) 2022-09-08T02:11:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:27.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:27.535531+0000) 2022-09-08T02:11:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:27 smithi055 ceph-mon[97033]: pgmap v8804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:11:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:27 smithi059 ceph-mon[97450]: pgmap v8804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:11:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:28.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:28.480521+0000) 2022-09-08T02:11:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:28.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:28.535699+0000) 2022-09-08T02:11:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:29.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:29.480771+0000) 2022-09-08T02:11:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:29.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:29.535953+0000) 2022-09-08T02:11:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:29 smithi055 ceph-mon[97033]: pgmap v8805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:29 smithi059 ceph-mon[97450]: pgmap v8805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:30.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:30.536148+0000) 2022-09-08T02:11:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:30.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:30.480982+0000) 2022-09-08T02:11:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:31.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:31.536359+0000) 2022-09-08T02:11:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:31.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:31.481157+0000) 2022-09-08T02:11:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:31 smithi055 ceph-mon[97033]: pgmap v8806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:11:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:31 smithi059 ceph-mon[97450]: pgmap v8806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:11:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:32.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:32.536532+0000) 2022-09-08T02:11:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:32.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:32.481358+0000) 2022-09-08T02:11:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:33 smithi055 ceph-mon[97033]: pgmap v8807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:33.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:33 smithi059 ceph-mon[97450]: pgmap v8807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:33.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:33.536740+0000) 2022-09-08T02:11:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:33.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:33.481627+0000) 2022-09-08T02:11:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:34.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:34.536908+0000) 2022-09-08T02:11:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:34.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:34.481895+0000) 2022-09-08T02:11:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:35.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:35.537108+0000) 2022-09-08T02:11:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:35.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:35.482134+0000) 2022-09-08T02:11:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:35 smithi055 ceph-mon[97033]: pgmap v8808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:35 smithi059 ceph-mon[97450]: pgmap v8808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:36.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:36.537265+0000) 2022-09-08T02:11:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:36.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:36.482267+0000) 2022-09-08T02:11:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:37.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:37.537426+0000) 2022-09-08T02:11:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:37.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:37.482486+0000) 2022-09-08T02:11:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:37 smithi055 ceph-mon[97033]: pgmap v8809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:11:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:37 smithi059 ceph-mon[97450]: pgmap v8809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:11:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:38.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:38.537608+0000) 2022-09-08T02:11:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:38.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:38.482727+0000) 2022-09-08T02:11:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:39.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:39.537799+0000) 2022-09-08T02:11:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:39.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:39.482914+0000) 2022-09-08T02:11:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:39 smithi055 ceph-mon[97033]: pgmap v8810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:39 smithi059 ceph-mon[97450]: pgmap v8810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:40.483 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:11:40.168+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:11:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:40.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:40.537998+0000) 2022-09-08T02:11:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:40.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:40.483143+0000) 2022-09-08T02:11:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:41.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:41.538165+0000) 2022-09-08T02:11:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:41.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:41.483340+0000) 2022-09-08T02:11:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:41 smithi055 ceph-mon[97033]: pgmap v8811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:11:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:41 smithi059 ceph-mon[97450]: pgmap v8811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:11:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:42.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:42.538368+0000) 2022-09-08T02:11:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:42.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:42.483560+0000) 2022-09-08T02:11:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:43 smithi055 ceph-mon[97033]: pgmap v8812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:43 smithi059 ceph-mon[97450]: pgmap v8812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:43.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:43.538604+0000) 2022-09-08T02:11:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:43.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:43.483841+0000) 2022-09-08T02:11:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:44.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:44.538884+0000) 2022-09-08T02:11:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:44.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:44.484059+0000) 2022-09-08T02:11:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:45.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:45.539115+0000) 2022-09-08T02:11:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:45.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:45.484258+0000) 2022-09-08T02:11:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:45 smithi055 ceph-mon[97033]: pgmap v8813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:45 smithi059 ceph-mon[97450]: pgmap v8813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:46.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:46.539329+0000) 2022-09-08T02:11:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:46.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:46.484429+0000) 2022-09-08T02:11:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:47.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:47.539561+0000) 2022-09-08T02:11:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:47.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:47.484571+0000) 2022-09-08T02:11:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:47 smithi055 ceph-mon[97033]: pgmap v8814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:11:48.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:47 smithi059 ceph-mon[97450]: pgmap v8814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:11:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:48.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:48.539781+0000) 2022-09-08T02:11:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:48.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:48.484764+0000) 2022-09-08T02:11:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:49.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:49.540019+0000) 2022-09-08T02:11:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:49.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:49.485013+0000) 2022-09-08T02:11:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:49 smithi055 ceph-mon[97033]: pgmap v8815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:49 smithi059 ceph-mon[97450]: pgmap v8815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:50.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:50.540189+0000) 2022-09-08T02:11:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:50.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:50.485244+0000) 2022-09-08T02:11:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:51.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:51.540435+0000) 2022-09-08T02:11:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:51.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:51.485482+0000) 2022-09-08T02:11:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:51 smithi055 ceph-mon[97033]: pgmap v8816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:11:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:51 smithi059 ceph-mon[97450]: pgmap v8816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:11:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:52.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:52.540592+0000) 2022-09-08T02:11:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:52.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:52.485682+0000) 2022-09-08T02:11:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:53 smithi059 ceph-mon[97450]: pgmap v8817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:53.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:53.540835+0000) 2022-09-08T02:11:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:53.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:53.485863+0000) 2022-09-08T02:11:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:53 smithi055 ceph-mon[97033]: pgmap v8817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:54.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:54.541055+0000) 2022-09-08T02:11:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:54.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:54.486005+0000) 2022-09-08T02:11:55.494 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:11:55.184+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:11:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:55.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:55.541294+0000) 2022-09-08T02:11:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:55.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:55.486248+0000) 2022-09-08T02:11:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:55 smithi055 ceph-mon[97033]: pgmap v8818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:55 smithi059 ceph-mon[97450]: pgmap v8818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:56.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:56.541515+0000) 2022-09-08T02:11:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:56.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:56.486427+0000) 2022-09-08T02:11:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:57.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:57.541750+0000) 2022-09-08T02:11:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:57.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:57.486711+0000) 2022-09-08T02:11:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:57 smithi055 ceph-mon[97033]: pgmap v8819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:11:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:57 smithi059 ceph-mon[97450]: pgmap v8819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:11:58.743 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:58.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:58.541994+0000) 2022-09-08T02:11:58.744 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:58.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:58.486855+0000) 2022-09-08T02:11:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:11:59.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:59.542132+0000) 2022-09-08T02:11:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:11:59.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:11:59.487039+0000) 2022-09-08T02:11:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:59 smithi055 ceph-mon[97033]: pgmap v8820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:11:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:11:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:11:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:11:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:12:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:59 smithi059 ceph-mon[97450]: pgmap v8820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:12:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:12:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:11:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:12:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:00.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:00.542358+0000) 2022-09-08T02:12:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:00.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:00.487238+0000) 2022-09-08T02:12:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:01.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:01.542588+0000) 2022-09-08T02:12:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:01.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:01.487465+0000) 2022-09-08T02:12:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:01 smithi055 ceph-mon[97033]: pgmap v8821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:12:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:01 smithi059 ceph-mon[97450]: pgmap v8821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:12:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:02.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:02.487673+0000) 2022-09-08T02:12:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:02.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:02.542726+0000) 2022-09-08T02:12:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:03 smithi059 ceph-mon[97450]: pgmap v8822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:03.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:03.542933+0000) 2022-09-08T02:12:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:03.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:03.487906+0000) 2022-09-08T02:12:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:03 smithi055 ceph-mon[97033]: pgmap v8822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:04.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:04.543148+0000) 2022-09-08T02:12:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:04.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:04.488155+0000) 2022-09-08T02:12:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:05.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:05.543310+0000) 2022-09-08T02:12:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:05.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:05.488343+0000) 2022-09-08T02:12:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:05 smithi055 ceph-mon[97033]: pgmap v8823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:12:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:12:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:05 smithi059 ceph-mon[97450]: pgmap v8823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:12:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:12:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:06.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:06.543485+0000) 2022-09-08T02:12:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:06.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:06.488579+0000) 2022-09-08T02:12:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:07.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:07.543731+0000) 2022-09-08T02:12:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:07.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:07.488833+0000) 2022-09-08T02:12:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:07 smithi055 ceph-mon[97033]: pgmap v8824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:12:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:07 smithi059 ceph-mon[97450]: pgmap v8824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:12:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:08.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:08.489034+0000) 2022-09-08T02:12:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:08.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:08.543941+0000) 2022-09-08T02:12:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:09.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:09.544099+0000) 2022-09-08T02:12:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:09.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:09.489290+0000) 2022-09-08T02:12:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:09 smithi055 ceph-mon[97033]: pgmap v8825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:12:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:09 smithi055 ceph-mon[97033]: osdmap e633: 12 total, 11 up, 11 in 2022-09-08T02:12:10.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:09 smithi059 ceph-mon[97450]: pgmap v8825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:12:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:09 smithi059 ceph-mon[97450]: osdmap e633: 12 total, 11 up, 11 in 2022-09-08T02:12:10.489 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:12:10.192+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:12:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:10.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:10.544299+0000) 2022-09-08T02:12:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:10.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:10.489427+0000) 2022-09-08T02:12:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:11.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:11.544514+0000) 2022-09-08T02:12:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:11.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:11.489676+0000) 2022-09-08T02:12:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:11 smithi055 ceph-mon[97033]: pgmap v8827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:12:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:11 smithi059 ceph-mon[97450]: pgmap v8827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:12:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:12.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:12.489921+0000) 2022-09-08T02:12:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:12.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:12.544735+0000) 2022-09-08T02:12:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:13 smithi055 ceph-mon[97033]: osdmap e634: 12 total, 11 up, 11 in 2022-09-08T02:12:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:13 smithi055 ceph-mon[97033]: pgmap v8829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T02:12:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:13 smithi059 ceph-mon[97450]: osdmap e634: 12 total, 11 up, 11 in 2022-09-08T02:12:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:13 smithi059 ceph-mon[97450]: pgmap v8829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T02:12:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:13.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:13.544961+0000) 2022-09-08T02:12:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:13.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:13.490169+0000) 2022-09-08T02:12:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:14.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:14.545201+0000) 2022-09-08T02:12:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:14.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:14.490372+0000) 2022-09-08T02:12:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:15.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:15.545436+0000) 2022-09-08T02:12:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:15.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:15.490610+0000) 2022-09-08T02:12:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:15 smithi055 ceph-mon[97033]: pgmap v8830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:12:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:15 smithi059 ceph-mon[97450]: pgmap v8830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:12:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:16.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:16.545602+0000) 2022-09-08T02:12:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:16.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:16.490784+0000) 2022-09-08T02:12:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:17.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:17.545859+0000) 2022-09-08T02:12:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:17.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:17.490941+0000) 2022-09-08T02:12:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:17 smithi055 ceph-mon[97033]: pgmap v8831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:12:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:17 smithi059 ceph-mon[97450]: pgmap v8831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:12:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:18.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:18.546077+0000) 2022-09-08T02:12:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:18.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:18.491184+0000) 2022-09-08T02:12:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:19.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:19.546311+0000) 2022-09-08T02:12:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:19.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:19.491418+0000) 2022-09-08T02:12:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:19 smithi055 ceph-mon[97033]: pgmap v8832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T02:12:20.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:19 smithi059 ceph-mon[97450]: pgmap v8832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T02:12:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:20.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:20.546501+0000) 2022-09-08T02:12:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:20.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:20.491617+0000) 2022-09-08T02:12:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:21.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:21.546687+0000) 2022-09-08T02:12:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:21.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:21.491782+0000) 2022-09-08T02:12:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:21 smithi055 ceph-mon[97033]: pgmap v8833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:12:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:21 smithi059 ceph-mon[97450]: pgmap v8833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:12:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:22.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:22.546905+0000) 2022-09-08T02:12:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:22.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:22.491988+0000) 2022-09-08T02:12:23.762 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:23.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:23.547157+0000) 2022-09-08T02:12:23.763 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:23.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:23.492246+0000) 2022-09-08T02:12:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:23 smithi059 ceph-mon[97450]: pgmap v8834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 101 B/s wr, 6 op/s 2022-09-08T02:12:23.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:23 smithi055 ceph-mon[97033]: pgmap v8834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 101 B/s wr, 6 op/s 2022-09-08T02:12:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:24.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:24.547316+0000) 2022-09-08T02:12:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:24.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:24.492469+0000) 2022-09-08T02:12:25.493 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:12:25.202+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:12:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:25.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:25.547507+0000) 2022-09-08T02:12:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:25.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:25.492721+0000) 2022-09-08T02:12:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:25 smithi055 ceph-mon[97033]: pgmap v8835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T02:12:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:25 smithi059 ceph-mon[97450]: pgmap v8835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T02:12:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:26.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:26.547731+0000) 2022-09-08T02:12:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:26.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:26.492929+0000) 2022-09-08T02:12:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:27.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:27.547968+0000) 2022-09-08T02:12:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:27.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:27.493133+0000) 2022-09-08T02:12:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:27 smithi055 ceph-mon[97033]: pgmap v8836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:12:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:27 smithi059 ceph-mon[97450]: pgmap v8836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:12:28.814 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:28.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:28.548227+0000) 2022-09-08T02:12:28.815 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:28.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:28.493311+0000) 2022-09-08T02:12:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:29.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:29.548400+0000) 2022-09-08T02:12:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:29.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:29.493547+0000) 2022-09-08T02:12:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:29 smithi055 ceph-mon[97033]: pgmap v8837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:12:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:29 smithi059 ceph-mon[97450]: pgmap v8837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:12:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:30.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:30.548579+0000) 2022-09-08T02:12:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:30.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:30.493708+0000) 2022-09-08T02:12:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:30 smithi055 ceph-mon[97033]: pgmap v8838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:12:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:30 smithi055 ceph-mon[97033]: pgmap v8839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:12:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:30 smithi059 ceph-mon[97450]: pgmap v8838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:12:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:30 smithi059 ceph-mon[97450]: pgmap v8839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:12:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:31.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:31.493878+0000) 2022-09-08T02:12:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:31.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:31.548787+0000) 2022-09-08T02:12:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:32.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:32.548970+0000) 2022-09-08T02:12:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:32.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:32.494087+0000) 2022-09-08T02:12:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:32 smithi055 ceph-mon[97033]: pgmap v8840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:12:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:32 smithi059 ceph-mon[97450]: pgmap v8840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:12:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:33.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:33.549190+0000) 2022-09-08T02:12:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:33.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:33.494355+0000) 2022-09-08T02:12:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:34.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:34.549416+0000) 2022-09-08T02:12:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:34.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:34.494607+0000) 2022-09-08T02:12:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:34 smithi055 ceph-mon[97033]: pgmap v8841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:12:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:34 smithi059 ceph-mon[97450]: pgmap v8841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:12:35.909 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:35.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:35.549676+0000) 2022-09-08T02:12:35.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:35.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:35.494864+0000) 2022-09-08T02:12:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:36.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:36.495041+0000) 2022-09-08T02:12:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:36.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:36.549877+0000) 2022-09-08T02:12:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:36 smithi055 ceph-mon[97033]: pgmap v8842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:12:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:36 smithi059 ceph-mon[97450]: pgmap v8842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:12:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:37.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:37.550087+0000) 2022-09-08T02:12:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:37.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:37.495259+0000) 2022-09-08T02:12:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:38.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:38.550259+0000) 2022-09-08T02:12:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:38.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:38.495463+0000) 2022-09-08T02:12:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:38 smithi055 ceph-mon[97033]: pgmap v8843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:12:39.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:38 smithi059 ceph-mon[97450]: pgmap v8843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:12:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:39.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:39.550486+0000) 2022-09-08T02:12:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:39.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:39.495756+0000) 2022-09-08T02:12:40.496 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:12:40.216+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:12:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:40.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:40.550694+0000) 2022-09-08T02:12:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:40.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:40.495918+0000) 2022-09-08T02:12:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:40 smithi055 ceph-mon[97033]: pgmap v8844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:12:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:40 smithi059 ceph-mon[97450]: pgmap v8844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:12:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:41.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:41.550913+0000) 2022-09-08T02:12:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:41.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:41.496130+0000) 2022-09-08T02:12:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:42.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:42.551124+0000) 2022-09-08T02:12:42.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:42.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:42.496359+0000) 2022-09-08T02:12:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:42 smithi055 ceph-mon[97033]: pgmap v8845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:12:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:42 smithi059 ceph-mon[97450]: pgmap v8845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:12:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:43.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:43.551323+0000) 2022-09-08T02:12:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:43.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:43.496626+0000) 2022-09-08T02:12:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:44.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:44.551537+0000) 2022-09-08T02:12:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:44.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:44.496777+0000) 2022-09-08T02:12:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:44 smithi055 ceph-mon[97033]: pgmap v8846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:44 smithi059 ceph-mon[97450]: pgmap v8846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:45.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:45.551731+0000) 2022-09-08T02:12:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:45.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:45.496992+0000) 2022-09-08T02:12:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:46.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:46.551913+0000) 2022-09-08T02:12:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:46.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:46.497237+0000) 2022-09-08T02:12:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:46 smithi055 ceph-mon[97033]: pgmap v8847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:46 smithi059 ceph-mon[97450]: pgmap v8847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:47.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:47.552140+0000) 2022-09-08T02:12:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:47.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:47.497446+0000) 2022-09-08T02:12:48.895 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:48.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:48.552284+0000) 2022-09-08T02:12:48.896 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:48.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:48.497688+0000) 2022-09-08T02:12:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:48 smithi055 ceph-mon[97033]: pgmap v8848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:12:49.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:48 smithi059 ceph-mon[97450]: pgmap v8848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:12:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:49.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:49.552417+0000) 2022-09-08T02:12:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:49.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:49.497855+0000) 2022-09-08T02:12:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:50.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:50.552683+0000) 2022-09-08T02:12:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:50.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:50.498015+0000) 2022-09-08T02:12:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:50 smithi055 ceph-mon[97033]: pgmap v8849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:50 smithi059 ceph-mon[97450]: pgmap v8849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:51.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:51.552920+0000) 2022-09-08T02:12:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:51.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:51.498281+0000) 2022-09-08T02:12:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:52.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:52.553035+0000) 2022-09-08T02:12:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:52.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:52.498505+0000) 2022-09-08T02:12:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:52 smithi055 ceph-mon[97033]: pgmap v8850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:12:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:52 smithi059 ceph-mon[97450]: pgmap v8850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:12:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:53.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:53.553237+0000) 2022-09-08T02:12:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:53.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:53.498717+0000) 2022-09-08T02:12:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:54.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:54.553443+0000) 2022-09-08T02:12:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:54.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:54.498942+0000) 2022-09-08T02:12:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:54 smithi055 ceph-mon[97033]: pgmap v8851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:54 smithi059 ceph-mon[97450]: pgmap v8851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:55.499 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:12:55.222+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:12:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:55.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:55.499174+0000) 2022-09-08T02:12:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:55.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:55.553709+0000) 2022-09-08T02:12:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:56.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:56.553954+0000) 2022-09-08T02:12:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:56.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:56.499319+0000) 2022-09-08T02:12:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:56 smithi055 ceph-mon[97033]: pgmap v8852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:56 smithi059 ceph-mon[97450]: pgmap v8852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:12:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:57.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:57.554122+0000) 2022-09-08T02:12:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:57.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:57.499531+0000) 2022-09-08T02:12:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:58.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:58.554338+0000) 2022-09-08T02:12:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:58.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:58.499776+0000) 2022-09-08T02:12:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:12:58 smithi055 ceph-mon[97033]: pgmap v8853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:12:59.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:12:58 smithi059 ceph-mon[97450]: pgmap v8853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:12:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:12:59.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:59.554557+0000) 2022-09-08T02:12:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:12:59.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:12:59.499959+0000) 2022-09-08T02:13:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:00.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:00.554794+0000) 2022-09-08T02:13:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:00.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:00.500194+0000) 2022-09-08T02:13:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:00 smithi055 ceph-mon[97033]: pgmap v8854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:00 smithi059 ceph-mon[97450]: pgmap v8854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:01.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:01.500436+0000) 2022-09-08T02:13:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:01.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:01.554976+0000) 2022-09-08T02:13:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:02.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:02.555182+0000) 2022-09-08T02:13:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:02.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:02.500620+0000) 2022-09-08T02:13:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:02 smithi055 ceph-mon[97033]: pgmap v8855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:13:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:02 smithi059 ceph-mon[97450]: pgmap v8855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:13:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:03.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:03.500871+0000) 2022-09-08T02:13:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:03.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:03.555347+0000) 2022-09-08T02:13:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:04.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:04.501114+0000) 2022-09-08T02:13:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:04.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:04.555586+0000) 2022-09-08T02:13:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:04 smithi055 ceph-mon[97033]: pgmap v8856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:04 smithi059 ceph-mon[97450]: pgmap v8856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:05.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:05.555809+0000) 2022-09-08T02:13:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:05.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:05.501372+0000) 2022-09-08T02:13:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:13:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:13:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:13:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:13:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:06.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:06.556034+0000) 2022-09-08T02:13:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:06.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:06.501608+0000) 2022-09-08T02:13:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:06 smithi055 ceph-mon[97033]: pgmap v8857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-08T02:13:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:06 smithi059 ceph-mon[97450]: pgmap v8857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-08T02:13:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:07.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:07.556241+0000) 2022-09-08T02:13:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:07.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:07.501843+0000) 2022-09-08T02:13:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:08.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:08.556429+0000) 2022-09-08T02:13:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:08.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:08.502036+0000) 2022-09-08T02:13:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:08 smithi055 ceph-mon[97033]: pgmap v8858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:13:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:08 smithi059 ceph-mon[97450]: pgmap v8858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:13:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:09.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:09.502261+0000) 2022-09-08T02:13:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:09.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:09.556650+0000) 2022-09-08T02:13:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:09 smithi055 ceph-mon[97033]: osdmap e635: 12 total, 11 up, 11 in 2022-09-08T02:13:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:09 smithi055 ceph-mon[97033]: pgmap v8860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:13:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:09 smithi059 ceph-mon[97450]: osdmap e635: 12 total, 11 up, 11 in 2022-09-08T02:13:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:09 smithi059 ceph-mon[97450]: pgmap v8860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:13:10.502 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:13:10.232+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:13:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:10.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:10.556835+0000) 2022-09-08T02:13:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:10.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:10.502461+0000) 2022-09-08T02:13:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:11.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:11.557025+0000) 2022-09-08T02:13:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:11.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:11.502716+0000) 2022-09-08T02:13:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:11 smithi055 ceph-mon[97033]: pgmap v8861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:13:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:11 smithi059 ceph-mon[97450]: pgmap v8861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:13:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:12.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:12.557169+0000) 2022-09-08T02:13:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:12.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:12.502843+0000) 2022-09-08T02:13:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:13 smithi059 ceph-mon[97450]: osdmap e636: 12 total, 11 up, 11 in 2022-09-08T02:13:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:13.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:13.557302+0000) 2022-09-08T02:13:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:13.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:13.503073+0000) 2022-09-08T02:13:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:13 smithi055 ceph-mon[97033]: osdmap e636: 12 total, 11 up, 11 in 2022-09-08T02:13:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:14 smithi059 ceph-mon[97450]: pgmap v8863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 33 KiB/s wr, 26 op/s 2022-09-08T02:13:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:14.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:14.557526+0000) 2022-09-08T02:13:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:14.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:14.503273+0000) 2022-09-08T02:13:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:14 smithi055 ceph-mon[97033]: pgmap v8863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 33 KiB/s wr, 26 op/s 2022-09-08T02:13:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:15.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:15.557770+0000) 2022-09-08T02:13:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:15.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:15.503521+0000) 2022-09-08T02:13:16.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:16 smithi059 ceph-mon[97450]: pgmap v8864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:13:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:16.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:16.557948+0000) 2022-09-08T02:13:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:16.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:16.503761+0000) 2022-09-08T02:13:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:16 smithi055 ceph-mon[97033]: pgmap v8864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:13:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:17.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:17.503958+0000) 2022-09-08T02:13:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:17.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:17.558156+0000) 2022-09-08T02:13:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:18 smithi059 ceph-mon[97450]: pgmap v8865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 251 B/s wr, 7 op/s 2022-09-08T02:13:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:18.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:18.558401+0000) 2022-09-08T02:13:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:18.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:18.504162+0000) 2022-09-08T02:13:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:18 smithi055 ceph-mon[97033]: pgmap v8865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 251 B/s wr, 7 op/s 2022-09-08T02:13:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:19.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:19.558677+0000) 2022-09-08T02:13:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:19.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:19.504395+0000) 2022-09-08T02:13:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:20 smithi059 ceph-mon[97450]: pgmap v8866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:13:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:20.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:20.504630+0000) 2022-09-08T02:13:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:20.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:20.558913+0000) 2022-09-08T02:13:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:20 smithi055 ceph-mon[97033]: pgmap v8866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:13:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:21.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:21.504847+0000) 2022-09-08T02:13:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:21.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:21.559116+0000) 2022-09-08T02:13:22.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:22 smithi059 ceph-mon[97450]: pgmap v8867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:13:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:22.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:22.559301+0000) 2022-09-08T02:13:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:22.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:22.505113+0000) 2022-09-08T02:13:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:22 smithi055 ceph-mon[97033]: pgmap v8867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:13:23.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:23.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:23.559556+0000) 2022-09-08T02:13:23.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:23.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:23.505329+0000) 2022-09-08T02:13:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:24 smithi059 ceph-mon[97450]: pgmap v8868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 5 op/s 2022-09-08T02:13:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:24.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:24.559842+0000) 2022-09-08T02:13:24.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:24.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:24.505545+0000) 2022-09-08T02:13:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:24 smithi055 ceph-mon[97033]: pgmap v8868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 5 op/s 2022-09-08T02:13:25.506 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:13:25.233+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:13:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:25.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:25.560108+0000) 2022-09-08T02:13:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:25.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:25.505792+0000) 2022-09-08T02:13:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:26 smithi059 ceph-mon[97450]: pgmap v8869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T02:13:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:26.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:26.560389+0000) 2022-09-08T02:13:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:26.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:26.506040+0000) 2022-09-08T02:13:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:26 smithi055 ceph-mon[97033]: pgmap v8869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T02:13:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:27.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:27.560667+0000) 2022-09-08T02:13:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:27.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:27.506294+0000) 2022-09-08T02:13:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:28 smithi059 ceph-mon[97450]: pgmap v8870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:13:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:28.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:28.560922+0000) 2022-09-08T02:13:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:28.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:28.506548+0000) 2022-09-08T02:13:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:28 smithi055 ceph-mon[97033]: pgmap v8870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:13:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:29.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:29.561169+0000) 2022-09-08T02:13:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:29.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:29.506789+0000) 2022-09-08T02:13:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:13:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:13:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:13:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:13:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:13:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:13:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:30.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:30.507014+0000) 2022-09-08T02:13:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:30.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:30.561403+0000) 2022-09-08T02:13:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:30 smithi055 ceph-mon[97033]: pgmap v8871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:31.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:30 smithi059 ceph-mon[97450]: pgmap v8871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:31.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:31.507246+0000) 2022-09-08T02:13:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:31.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:31.561740+0000) 2022-09-08T02:13:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:32.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:32.561954+0000) 2022-09-08T02:13:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:32.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:32.507484+0000) 2022-09-08T02:13:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:32 smithi055 ceph-mon[97033]: pgmap v8872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:13:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:32 smithi059 ceph-mon[97450]: pgmap v8872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:13:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:33.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:33.562177+0000) 2022-09-08T02:13:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:33.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:33.507762+0000) 2022-09-08T02:13:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:34.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:34.562403+0000) 2022-09-08T02:13:34.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:34.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:34.507980+0000) 2022-09-08T02:13:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:34 smithi055 ceph-mon[97033]: pgmap v8873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:34 smithi059 ceph-mon[97450]: pgmap v8873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:35.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:35.562668+0000) 2022-09-08T02:13:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:35.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:35.508235+0000) 2022-09-08T02:13:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:36.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:36.562901+0000) 2022-09-08T02:13:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:36.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:36.508459+0000) 2022-09-08T02:13:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:36 smithi055 ceph-mon[97033]: pgmap v8874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:36 smithi059 ceph-mon[97450]: pgmap v8874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:37.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:37.563121+0000) 2022-09-08T02:13:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:37.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:37.508718+0000) 2022-09-08T02:13:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:38.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:38.563333+0000) 2022-09-08T02:13:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:38.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:38.508926+0000) 2022-09-08T02:13:38.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:38 smithi055 ceph-mon[97033]: pgmap v8875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:13:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:38 smithi059 ceph-mon[97450]: pgmap v8875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:13:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:39.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:39.563535+0000) 2022-09-08T02:13:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:39.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:39.509104+0000) 2022-09-08T02:13:40.511 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:13:40.249+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:13:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:40.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:40.563757+0000) 2022-09-08T02:13:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:40.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:40.509340+0000) 2022-09-08T02:13:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:40 smithi055 ceph-mon[97033]: pgmap v8876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:40 smithi059 ceph-mon[97450]: pgmap v8876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:41.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:41.564006+0000) 2022-09-08T02:13:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:41.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:41.509557+0000) 2022-09-08T02:13:42.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:42.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:42.564246+0000) 2022-09-08T02:13:42.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:42.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:42.509777+0000) 2022-09-08T02:13:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:42 smithi055 ceph-mon[97033]: pgmap v8877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:13:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:42 smithi059 ceph-mon[97450]: pgmap v8877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:13:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:43.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:43.564467+0000) 2022-09-08T02:13:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:43.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:43.509922+0000) 2022-09-08T02:13:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:44.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:44.564740+0000) 2022-09-08T02:13:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:44.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:44.510100+0000) 2022-09-08T02:13:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:44 smithi055 ceph-mon[97033]: pgmap v8878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:44 smithi059 ceph-mon[97450]: pgmap v8878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:45.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:45.565032+0000) 2022-09-08T02:13:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:45.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:45.510330+0000) 2022-09-08T02:13:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:46.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:46.565280+0000) 2022-09-08T02:13:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:46.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:46.510507+0000) 2022-09-08T02:13:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:46 smithi055 ceph-mon[97033]: pgmap v8879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:46 smithi059 ceph-mon[97450]: pgmap v8879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:47.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:47.565493+0000) 2022-09-08T02:13:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:47.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:47.510719+0000) 2022-09-08T02:13:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:48.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:48.565730+0000) 2022-09-08T02:13:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:48.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:48.510915+0000) 2022-09-08T02:13:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:48 smithi055 ceph-mon[97033]: pgmap v8880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:13:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:48 smithi059 ceph-mon[97450]: pgmap v8880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:13:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:49.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:49.565955+0000) 2022-09-08T02:13:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:49.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:49.511106+0000) 2022-09-08T02:13:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:50.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:50.566166+0000) 2022-09-08T02:13:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:50.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:50.511287+0000) 2022-09-08T02:13:50.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:50 smithi055 ceph-mon[97033]: pgmap v8881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:50 smithi059 ceph-mon[97450]: pgmap v8881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:51.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:51.566365+0000) 2022-09-08T02:13:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:51.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:51.511542+0000) 2022-09-08T02:13:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:52.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:52.566589+0000) 2022-09-08T02:13:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:52.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:52.511784+0000) 2022-09-08T02:13:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:52 smithi055 ceph-mon[97033]: pgmap v8882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:13:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:52 smithi059 ceph-mon[97450]: pgmap v8882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:13:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:53.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:53.566817+0000) 2022-09-08T02:13:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:53.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:53.512033+0000) 2022-09-08T02:13:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:54.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:54.567056+0000) 2022-09-08T02:13:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:54.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:54.512245+0000) 2022-09-08T02:13:54.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:54 smithi055 ceph-mon[97033]: pgmap v8883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:54 smithi059 ceph-mon[97450]: pgmap v8883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:55.567 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:13:55.265+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:13:55.569 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:55.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:55.512444+0000) 2022-09-08T02:13:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:55.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:55.567287+0000) 2022-09-08T02:13:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:56.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:56.512581+0000) 2022-09-08T02:13:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:56.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:56.567529+0000) 2022-09-08T02:13:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:56 smithi055 ceph-mon[97033]: pgmap v8884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:56 smithi059 ceph-mon[97450]: pgmap v8884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:13:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:57.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:57.567792+0000) 2022-09-08T02:13:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:57.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:57.512805+0000) 2022-09-08T02:13:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:58.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:58.513035+0000) 2022-09-08T02:13:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:58.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:58.568032+0000) 2022-09-08T02:13:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:13:58 smithi055 ceph-mon[97033]: pgmap v8885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:13:59.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:13:58 smithi059 ceph-mon[97450]: pgmap v8885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:13:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:13:59.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:59.568277+0000) 2022-09-08T02:13:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:13:59.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:13:59.513311+0000) 2022-09-08T02:14:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:00 smithi055 ceph-mon[97033]: pgmap v8886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:00 smithi055 ceph-mon[97033]: pgmap v8887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:14:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:00 smithi055 ceph-mon[97033]: pgmap v8888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:14:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:14:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:00 smithi059 ceph-mon[97450]: pgmap v8886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:00 smithi059 ceph-mon[97450]: pgmap v8887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:14:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:00 smithi059 ceph-mon[97450]: pgmap v8888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:14:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:14:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:00.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:00.568494+0000) 2022-09-08T02:14:00.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:00.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:00.513476+0000) 2022-09-08T02:14:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:01.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:01.568753+0000) 2022-09-08T02:14:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:01.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:01.513723+0000) 2022-09-08T02:14:02.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:02 smithi055 ceph-mon[97033]: pgmap v8889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:14:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:02 smithi059 ceph-mon[97450]: pgmap v8889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:14:02.913 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:02.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:02.568971+0000) 2022-09-08T02:14:02.914 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:02.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:02.513932+0000) 2022-09-08T02:14:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:03.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:03.569171+0000) 2022-09-08T02:14:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:03.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:03.514142+0000) 2022-09-08T02:14:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:04 smithi059 ceph-mon[97450]: pgmap v8890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:14:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:04.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:04.569348+0000) 2022-09-08T02:14:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:04.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:04.514310+0000) 2022-09-08T02:14:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:04 smithi055 ceph-mon[97033]: pgmap v8890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:14:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:14:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:14:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:05.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:05.569563+0000) 2022-09-08T02:14:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:05.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:05.514537+0000) 2022-09-08T02:14:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:14:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:14:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:06 smithi059 ceph-mon[97450]: pgmap v8891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T02:14:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:06.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:06.569786+0000) 2022-09-08T02:14:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:06.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:06.514726+0000) 2022-09-08T02:14:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:06 smithi055 ceph-mon[97033]: pgmap v8891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T02:14:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:07.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:07.569963+0000) 2022-09-08T02:14:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:07.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:07.514927+0000) 2022-09-08T02:14:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:08 smithi059 ceph-mon[97450]: pgmap v8892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T02:14:08.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:08.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:08.570192+0000) 2022-09-08T02:14:08.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:08.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:08.515126+0000) 2022-09-08T02:14:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:08 smithi055 ceph-mon[97033]: pgmap v8892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T02:14:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:09.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:09.570349+0000) 2022-09-08T02:14:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:09.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:09.515346+0000) 2022-09-08T02:14:10.573 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:14:10.270+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:14:10.574 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:10.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:10.515599+0000) 2022-09-08T02:14:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:10 smithi059 ceph-mon[97450]: pgmap v8893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:14:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:10 smithi059 ceph-mon[97450]: osdmap e637: 12 total, 11 up, 11 in 2022-09-08T02:14:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:10.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:10.570578+0000) 2022-09-08T02:14:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:10 smithi055 ceph-mon[97033]: pgmap v8893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:14:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:10 smithi055 ceph-mon[97033]: osdmap e637: 12 total, 11 up, 11 in 2022-09-08T02:14:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:11.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:11.570818+0000) 2022-09-08T02:14:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:11.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:11.515826+0000) 2022-09-08T02:14:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:12.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:12.571102+0000) 2022-09-08T02:14:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:12.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:12.515994+0000) 2022-09-08T02:14:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:12 smithi055 ceph-mon[97033]: pgmap v8895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:14:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:12 smithi059 ceph-mon[97450]: pgmap v8895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:14:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:13.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:13.571317+0000) 2022-09-08T02:14:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:13.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:13.516126+0000) 2022-09-08T02:14:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:14.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:14.571482+0000) 2022-09-08T02:14:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:14.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:14.516267+0000) 2022-09-08T02:14:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:14 smithi055 ceph-mon[97033]: pgmap v8896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:14:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:14 smithi059 ceph-mon[97450]: pgmap v8896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:14:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:15.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:15.571630+0000) 2022-09-08T02:14:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:15.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:15.516503+0000) 2022-09-08T02:14:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:16.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:16.571838+0000) 2022-09-08T02:14:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:16.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:16.516733+0000) 2022-09-08T02:14:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:16 smithi055 ceph-mon[97033]: pgmap v8897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:14:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:16 smithi059 ceph-mon[97450]: pgmap v8897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:14:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:17.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:17.572014+0000) 2022-09-08T02:14:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:17.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:17.516977+0000) 2022-09-08T02:14:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:18 smithi059 ceph-mon[97450]: pgmap v8898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:14:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:18 smithi059 ceph-mon[97450]: osdmap e638: 12 total, 11 up, 11 in 2022-09-08T02:14:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:18.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:18.572256+0000) 2022-09-08T02:14:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:18.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:18.517183+0000) 2022-09-08T02:14:18.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:18 smithi055 ceph-mon[97033]: pgmap v8898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:14:18.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:18 smithi055 ceph-mon[97033]: osdmap e638: 12 total, 11 up, 11 in 2022-09-08T02:14:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:19.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:19.517438+0000) 2022-09-08T02:14:19.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:19.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:19.572464+0000) 2022-09-08T02:14:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:20 smithi059 ceph-mon[97450]: pgmap v8900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2022-09-08T02:14:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:20.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:20.572715+0000) 2022-09-08T02:14:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:20.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:20.517590+0000) 2022-09-08T02:14:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:20 smithi055 ceph-mon[97033]: pgmap v8900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2022-09-08T02:14:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:21.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:21.573018+0000) 2022-09-08T02:14:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:21.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:21.517794+0000) 2022-09-08T02:14:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:22 smithi059 ceph-mon[97450]: pgmap v8901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:14:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:22.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:22.573295+0000) 2022-09-08T02:14:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:22.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:22.517973+0000) 2022-09-08T02:14:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:22 smithi055 ceph-mon[97033]: pgmap v8901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:14:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:23.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:23.573570+0000) 2022-09-08T02:14:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:23.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:23.518234+0000) 2022-09-08T02:14:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:24 smithi059 ceph-mon[97450]: pgmap v8902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:14:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:24.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:24.573839+0000) 2022-09-08T02:14:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:24.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:24.518459+0000) 2022-09-08T02:14:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:24 smithi055 ceph-mon[97033]: pgmap v8902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:14:25.574 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:14:25.281+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:14:25.575 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:25.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:25.518692+0000) 2022-09-08T02:14:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:25.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:25.574030+0000) 2022-09-08T02:14:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:26 smithi059 ceph-mon[97450]: pgmap v8903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:14:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:26.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:26.518905+0000) 2022-09-08T02:14:26.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:26.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:26.574270+0000) 2022-09-08T02:14:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:26 smithi055 ceph-mon[97033]: pgmap v8903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:14:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:27.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:27.574444+0000) 2022-09-08T02:14:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:27.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:27.519156+0000) 2022-09-08T02:14:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:28 smithi059 ceph-mon[97450]: pgmap v8904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:14:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:28.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:28.574680+0000) 2022-09-08T02:14:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:28.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:28.519332+0000) 2022-09-08T02:14:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:28 smithi055 ceph-mon[97033]: pgmap v8904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:14:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:29.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:29.519557+0000) 2022-09-08T02:14:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:29.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:29.574944+0000) 2022-09-08T02:14:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:30.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:30.575143+0000) 2022-09-08T02:14:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:30.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:30.519748+0000) 2022-09-08T02:14:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:30 smithi055 ceph-mon[97033]: pgmap v8905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T02:14:31.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:30 smithi059 ceph-mon[97450]: pgmap v8905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T02:14:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:31.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:31.575395+0000) 2022-09-08T02:14:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:31.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:31.519969+0000) 2022-09-08T02:14:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:32.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:32.575588+0000) 2022-09-08T02:14:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:32.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:32.520136+0000) 2022-09-08T02:14:32.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:32 smithi055 ceph-mon[97033]: pgmap v8906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:14:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:32 smithi059 ceph-mon[97450]: pgmap v8906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:14:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:33.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:33.520390+0000) 2022-09-08T02:14:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:33.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:33.575793+0000) 2022-09-08T02:14:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:34.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:34.520604+0000) 2022-09-08T02:14:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:34.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:34.575950+0000) 2022-09-08T02:14:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:34 smithi055 ceph-mon[97033]: pgmap v8907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:34 smithi059 ceph-mon[97450]: pgmap v8907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:35.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:35.520795+0000) 2022-09-08T02:14:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:35.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:35.576192+0000) 2022-09-08T02:14:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:36.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:36.576412+0000) 2022-09-08T02:14:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:36.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:36.521006+0000) 2022-09-08T02:14:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:36 smithi055 ceph-mon[97033]: pgmap v8908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:14:37.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:36 smithi059 ceph-mon[97450]: pgmap v8908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:14:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:37.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:37.576558+0000) 2022-09-08T02:14:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:37.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:37.521251+0000) 2022-09-08T02:14:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:38.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:38.521455+0000) 2022-09-08T02:14:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:38.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:38.576735+0000) 2022-09-08T02:14:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:38 smithi055 ceph-mon[97033]: pgmap v8909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:38 smithi059 ceph-mon[97450]: pgmap v8909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:39.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:39.521743+0000) 2022-09-08T02:14:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:39.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:39.576950+0000) 2022-09-08T02:14:40.577 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:14:40.289+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:14:40.578 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:40.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:40.521929+0000) 2022-09-08T02:14:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:40.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:40.577179+0000) 2022-09-08T02:14:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:40 smithi055 ceph-mon[97033]: pgmap v8910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:40 smithi059 ceph-mon[97450]: pgmap v8910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:41.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:41.577364+0000) 2022-09-08T02:14:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:41.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:41.522110+0000) 2022-09-08T02:14:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:42.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:42.577579+0000) 2022-09-08T02:14:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:42.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:42.522330+0000) 2022-09-08T02:14:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:42 smithi055 ceph-mon[97033]: pgmap v8911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:14:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:42 smithi059 ceph-mon[97450]: pgmap v8911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:14:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:43.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:43.577845+0000) 2022-09-08T02:14:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:43.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:43.522607+0000) 2022-09-08T02:14:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:44.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:44.522866+0000) 2022-09-08T02:14:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:44.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:44.578087+0000) 2022-09-08T02:14:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:44 smithi055 ceph-mon[97033]: pgmap v8912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:44 smithi059 ceph-mon[97450]: pgmap v8912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:45.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:45.578299+0000) 2022-09-08T02:14:45.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:45.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:45.523117+0000) 2022-09-08T02:14:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:46.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:46.578548+0000) 2022-09-08T02:14:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:46.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:46.523353+0000) 2022-09-08T02:14:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:46 smithi055 ceph-mon[97033]: pgmap v8913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:14:47.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:46 smithi059 ceph-mon[97450]: pgmap v8913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:14:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:47.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:47.578788+0000) 2022-09-08T02:14:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:47.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:47.523564+0000) 2022-09-08T02:14:48.911 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:48.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:48.578999+0000) 2022-09-08T02:14:48.912 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:48.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:48.523776+0000) 2022-09-08T02:14:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:48 smithi055 ceph-mon[97033]: pgmap v8914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:48 smithi059 ceph-mon[97450]: pgmap v8914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:49.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:49.579171+0000) 2022-09-08T02:14:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:49.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:49.524022+0000) 2022-09-08T02:14:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:50.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:50.579423+0000) 2022-09-08T02:14:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:50.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:50.524192+0000) 2022-09-08T02:14:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:50 smithi055 ceph-mon[97033]: pgmap v8915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:50 smithi059 ceph-mon[97450]: pgmap v8915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:51.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:51.579650+0000) 2022-09-08T02:14:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:51.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:51.524427+0000) 2022-09-08T02:14:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:52.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:52.579822+0000) 2022-09-08T02:14:52.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:52.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:52.524717+0000) 2022-09-08T02:14:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:52 smithi055 ceph-mon[97033]: pgmap v8916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:14:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:52 smithi059 ceph-mon[97450]: pgmap v8916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:14:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:53.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:53.580051+0000) 2022-09-08T02:14:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:53.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:53.525001+0000) 2022-09-08T02:14:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:54.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:54.580255+0000) 2022-09-08T02:14:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:54.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:54.525203+0000) 2022-09-08T02:14:54.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:54 smithi055 ceph-mon[97033]: pgmap v8917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:54 smithi059 ceph-mon[97450]: pgmap v8917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:55.581 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:14:55.293+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:14:55.581 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:55.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:55.525451+0000) 2022-09-08T02:14:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:55.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:55.580464+0000) 2022-09-08T02:14:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:56.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:56.580685+0000) 2022-09-08T02:14:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:56.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:56.525677+0000) 2022-09-08T02:14:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:56 smithi055 ceph-mon[97033]: pgmap v8918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:14:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:56 smithi059 ceph-mon[97450]: pgmap v8918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:14:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:57.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:57.580913+0000) 2022-09-08T02:14:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:57.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:57.525904+0000) 2022-09-08T02:14:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:58.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:58.581131+0000) 2022-09-08T02:14:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:58.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:58.526120+0000) 2022-09-08T02:14:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:58 smithi055 ceph-mon[97033]: pgmap v8919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:58 smithi059 ceph-mon[97450]: pgmap v8919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:14:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:14:59.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:59.581359+0000) 2022-09-08T02:14:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:14:59.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:14:59.526340+0000) 2022-09-08T02:14:59.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:14:59.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:14:59.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:14:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:15:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:15:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:15:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:14:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:15:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:00.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:00.581524+0000) 2022-09-08T02:15:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:00.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:00.526494+0000) 2022-09-08T02:15:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:00 smithi055 ceph-mon[97033]: pgmap v8920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:15:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:00 smithi059 ceph-mon[97450]: pgmap v8920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:15:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:01.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:01.581746+0000) 2022-09-08T02:15:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:01.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:01.526737+0000) 2022-09-08T02:15:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:02.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:02.581988+0000) 2022-09-08T02:15:02.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:02.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:02.526937+0000) 2022-09-08T02:15:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:02 smithi055 ceph-mon[97033]: pgmap v8921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:15:03.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:02 smithi059 ceph-mon[97450]: pgmap v8921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:15:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:03.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:03.582209+0000) 2022-09-08T02:15:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:03.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:03.527073+0000) 2022-09-08T02:15:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:04.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:04.582438+0000) 2022-09-08T02:15:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:04.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:04.527299+0000) 2022-09-08T02:15:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:04 smithi055 ceph-mon[97033]: pgmap v8922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:15:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:04 smithi059 ceph-mon[97450]: pgmap v8922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:15:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:05.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:05.582727+0000) 2022-09-08T02:15:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:05.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:05.527479+0000) 2022-09-08T02:15:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:15:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:15:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:15:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:15:06.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:06.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:06.582932+0000) 2022-09-08T02:15:06.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:06.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:06.527631+0000) 2022-09-08T02:15:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:06 smithi055 ceph-mon[97033]: pgmap v8923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:15:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:06 smithi059 ceph-mon[97450]: pgmap v8923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:15:07.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:07.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:07.583146+0000) 2022-09-08T02:15:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:07.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:07.527797+0000) 2022-09-08T02:15:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:08.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:08.583381+0000) 2022-09-08T02:15:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:08.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:08.527944+0000) 2022-09-08T02:15:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:08 smithi055 ceph-mon[97033]: pgmap v8924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:15:09.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:08 smithi059 ceph-mon[97450]: pgmap v8924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:15:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:09.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:09.583613+0000) 2022-09-08T02:15:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:09.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:09.528167+0000) 2022-09-08T02:15:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:09 smithi055 ceph-mon[97033]: osdmap e639: 12 total, 11 up, 11 in 2022-09-08T02:15:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:09 smithi059 ceph-mon[97450]: osdmap e639: 12 total, 11 up, 11 in 2022-09-08T02:15:10.584 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:15:10.297+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:15:10.585 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:10.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:10.528384+0000) 2022-09-08T02:15:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:10.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:10.583787+0000) 2022-09-08T02:15:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:10 smithi055 ceph-mon[97033]: pgmap v8926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:15:11.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:10 smithi059 ceph-mon[97450]: pgmap v8926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:15:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:11.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:11.584043+0000) 2022-09-08T02:15:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:11.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:11.528611+0000) 2022-09-08T02:15:12.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:12.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:12.584226+0000) 2022-09-08T02:15:12.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:12.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:12.528762+0000) 2022-09-08T02:15:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:12 smithi055 ceph-mon[97033]: pgmap v8927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:15:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:12 smithi055 ceph-mon[97033]: osdmap e640: 12 total, 11 up, 11 in 2022-09-08T02:15:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:12 smithi059 ceph-mon[97450]: pgmap v8927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:15:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:12 smithi059 ceph-mon[97450]: osdmap e640: 12 total, 11 up, 11 in 2022-09-08T02:15:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:13.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:13.584442+0000) 2022-09-08T02:15:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:13.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:13.528918+0000) 2022-09-08T02:15:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:14.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:14.584674+0000) 2022-09-08T02:15:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:14.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:14.529116+0000) 2022-09-08T02:15:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:14 smithi055 ceph-mon[97033]: pgmap v8929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:15:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:14 smithi059 ceph-mon[97450]: pgmap v8929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:15:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:15.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:15.584858+0000) 2022-09-08T02:15:15.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:15.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:15.529296+0000) 2022-09-08T02:15:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:16.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:16.585100+0000) 2022-09-08T02:15:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:16.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:16.529539+0000) 2022-09-08T02:15:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:16 smithi055 ceph-mon[97033]: pgmap v8930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:15:17.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:16 smithi059 ceph-mon[97450]: pgmap v8930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:15:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:17.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:17.585362+0000) 2022-09-08T02:15:17.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:17.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:17.529760+0000) 2022-09-08T02:15:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:18.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:18.585627+0000) 2022-09-08T02:15:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:18.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:18.530024+0000) 2022-09-08T02:15:18.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:18 smithi055 ceph-mon[97033]: pgmap v8931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 240 B/s wr, 7 op/s 2022-09-08T02:15:19.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:18 smithi059 ceph-mon[97450]: pgmap v8931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 240 B/s wr, 7 op/s 2022-09-08T02:15:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:19.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:19.585851+0000) 2022-09-08T02:15:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:19.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:19.530255+0000) 2022-09-08T02:15:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:20.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:20.586009+0000) 2022-09-08T02:15:20.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:20.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:20.530483+0000) 2022-09-08T02:15:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:20 smithi055 ceph-mon[97033]: pgmap v8932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:15:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:20 smithi059 ceph-mon[97450]: pgmap v8932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:15:21.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:21.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:21.586248+0000) 2022-09-08T02:15:21.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:21.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:21.530737+0000) 2022-09-08T02:15:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:22.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:22.586431+0000) 2022-09-08T02:15:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:22.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:22.530944+0000) 2022-09-08T02:15:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:22 smithi055 ceph-mon[97033]: pgmap v8933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:15:23.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:22 smithi059 ceph-mon[97450]: pgmap v8933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:15:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:23.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:23.586696+0000) 2022-09-08T02:15:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:23.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:23.531208+0000) 2022-09-08T02:15:24.587 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:24.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:24.531456+0000) 2022-09-08T02:15:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:24.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:24.586872+0000) 2022-09-08T02:15:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:24 smithi055 ceph-mon[97033]: pgmap v8934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T02:15:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:24 smithi059 ceph-mon[97450]: pgmap v8934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T02:15:25.588 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:15:25.301+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:15:25.589 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:25.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:25.531689+0000) 2022-09-08T02:15:25.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:25.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:25.587046+0000) 2022-09-08T02:15:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:26.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:26.587198+0000) 2022-09-08T02:15:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:26.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:26.531913+0000) 2022-09-08T02:15:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:26 smithi055 ceph-mon[97033]: pgmap v8935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:15:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:26 smithi059 ceph-mon[97450]: pgmap v8935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:15:27.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:27.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:27.587418+0000) 2022-09-08T02:15:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:27.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:27.532122+0000) 2022-09-08T02:15:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:28.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:28.587704+0000) 2022-09-08T02:15:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:28.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:28.532374+0000) 2022-09-08T02:15:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:28 smithi055 ceph-mon[97033]: pgmap v8936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:15:29.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:28 smithi059 ceph-mon[97450]: pgmap v8936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:15:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:29.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:29.587873+0000) 2022-09-08T02:15:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:29.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:29.532564+0000) 2022-09-08T02:15:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:30 smithi055 ceph-mon[97033]: pgmap v8937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:15:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:30 smithi055 ceph-mon[97033]: pgmap v8938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:15:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:30 smithi055 ceph-mon[97033]: pgmap v8939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:15:30.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:15:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:30 smithi059 ceph-mon[97450]: pgmap v8937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:15:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:30 smithi059 ceph-mon[97450]: pgmap v8938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:15:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:30 smithi059 ceph-mon[97450]: pgmap v8939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:15:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:15:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:30.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:30.588103+0000) 2022-09-08T02:15:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:30.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:30.532766+0000) 2022-09-08T02:15:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:31.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:31.588340+0000) 2022-09-08T02:15:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:31.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:31.533015+0000) 2022-09-08T02:15:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:32 smithi055 ceph-mon[97033]: pgmap v8940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:15:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:32 smithi059 ceph-mon[97450]: pgmap v8940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:15:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:32.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:32.588571+0000) 2022-09-08T02:15:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:32.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:32.533243+0000) 2022-09-08T02:15:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:33.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:33.588748+0000) 2022-09-08T02:15:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:33.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:33.533449+0000) 2022-09-08T02:15:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:34.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:34.588880+0000) 2022-09-08T02:15:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:34.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:34.533696+0000) 2022-09-08T02:15:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:34 smithi055 ceph-mon[97033]: pgmap v8941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:15:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:34 smithi059 ceph-mon[97450]: pgmap v8941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:15:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:35.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:35.589067+0000) 2022-09-08T02:15:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:35.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:35.533886+0000) 2022-09-08T02:15:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:36.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:36.589185+0000) 2022-09-08T02:15:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:36.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:36.534126+0000) 2022-09-08T02:15:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:36 smithi055 ceph-mon[97033]: pgmap v8942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:15:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:36 smithi059 ceph-mon[97450]: pgmap v8942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:15:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:37.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:37.589419+0000) 2022-09-08T02:15:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:37.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:37.534390+0000) 2022-09-08T02:15:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:38.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:38.589703+0000) 2022-09-08T02:15:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:38.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:38.534623+0000) 2022-09-08T02:15:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:38 smithi055 ceph-mon[97033]: pgmap v8943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:15:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:38 smithi059 ceph-mon[97450]: pgmap v8943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:15:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:39.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:39.589895+0000) 2022-09-08T02:15:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:39.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:39.534840+0000) 2022-09-08T02:15:40.590 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:15:40.308+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:15:40.591 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:40.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:40.535014+0000) 2022-09-08T02:15:40.854 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:40 smithi055 ceph-mon[97033]: pgmap v8944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:15:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:40.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:40.590030+0000) 2022-09-08T02:15:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:40 smithi059 ceph-mon[97450]: pgmap v8944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:15:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:41.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:41.590212+0000) 2022-09-08T02:15:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:41.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:41.535211+0000) 2022-09-08T02:15:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:42.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:42.590456+0000) 2022-09-08T02:15:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:42.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:42.535433+0000) 2022-09-08T02:15:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:42 smithi055 ceph-mon[97033]: pgmap v8945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:15:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:42 smithi059 ceph-mon[97450]: pgmap v8945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:15:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:43.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:43.590741+0000) 2022-09-08T02:15:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:43.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:43.535727+0000) 2022-09-08T02:15:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:44 smithi055 ceph-mon[97033]: pgmap v8946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:15:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:44.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:44.590911+0000) 2022-09-08T02:15:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:44.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:44.535969+0000) 2022-09-08T02:15:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:44 smithi059 ceph-mon[97450]: pgmap v8946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:15:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:45.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:45.591105+0000) 2022-09-08T02:15:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:45.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:45.536184+0000) 2022-09-08T02:15:46.882 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:46 smithi055 ceph-mon[97033]: pgmap v8947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:15:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:46.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:46.591282+0000) 2022-09-08T02:15:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:46.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:46.536401+0000) 2022-09-08T02:15:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:46 smithi059 ceph-mon[97450]: pgmap v8947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:15:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:47.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:47.591530+0000) 2022-09-08T02:15:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:47.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:47.536552+0000) 2022-09-08T02:15:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:48.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:48.591778+0000) 2022-09-08T02:15:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:48.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:48.536795+0000) 2022-09-08T02:15:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:48 smithi055 ceph-mon[97033]: pgmap v8948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:15:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:48 smithi059 ceph-mon[97450]: pgmap v8948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:15:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:49.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:49.592026+0000) 2022-09-08T02:15:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:49.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:49.536970+0000) 2022-09-08T02:15:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:50.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:50.592262+0000) 2022-09-08T02:15:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:50.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:50.537122+0000) 2022-09-08T02:15:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:50 smithi055 ceph-mon[97033]: pgmap v8949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:15:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:50 smithi059 ceph-mon[97450]: pgmap v8949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:15:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:51.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:51.592509+0000) 2022-09-08T02:15:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:51.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:51.537348+0000) 2022-09-08T02:15:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:52.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:52.592682+0000) 2022-09-08T02:15:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:52.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:52.537576+0000) 2022-09-08T02:15:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:52 smithi055 ceph-mon[97033]: pgmap v8950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:15:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:52 smithi059 ceph-mon[97450]: pgmap v8950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:15:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:53.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:53.592875+0000) 2022-09-08T02:15:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:53.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:53.537880+0000) 2022-09-08T02:15:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:54 smithi055 ceph-mon[97033]: pgmap v8951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:15:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:54.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:54.593064+0000) 2022-09-08T02:15:54.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:54.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:54.538134+0000) 2022-09-08T02:15:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:54 smithi059 ceph-mon[97450]: pgmap v8951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:15:55.593 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:15:55.313+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:15:55.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:55.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:55.538408+0000) 2022-09-08T02:15:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:55.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:55.593257+0000) 2022-09-08T02:15:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:56.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:56.593425+0000) 2022-09-08T02:15:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:56.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:56.538661+0000) 2022-09-08T02:15:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:56 smithi055 ceph-mon[97033]: pgmap v8952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:15:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:56 smithi059 ceph-mon[97450]: pgmap v8952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:15:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:57.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:57.593680+0000) 2022-09-08T02:15:57.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:57.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:57.538884+0000) 2022-09-08T02:15:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:58.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:58.593932+0000) 2022-09-08T02:15:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:58.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:58.539113+0000) 2022-09-08T02:15:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:15:58 smithi055 ceph-mon[97033]: pgmap v8953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:15:59.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:15:58 smithi059 ceph-mon[97450]: pgmap v8953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:15:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:15:59.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:59.594101+0000) 2022-09-08T02:15:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:15:59.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:15:59.539324+0000) 2022-09-08T02:16:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:00.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:00.594329+0000) 2022-09-08T02:16:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:00.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:00.539549+0000) 2022-09-08T02:16:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:00 smithi055 ceph-mon[97033]: pgmap v8954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:16:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:00 smithi059 ceph-mon[97450]: pgmap v8954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:16:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:01.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:01.594592+0000) 2022-09-08T02:16:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:01.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:01.539733+0000) 2022-09-08T02:16:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:02.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:02.594837+0000) 2022-09-08T02:16:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:02.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:02.539915+0000) 2022-09-08T02:16:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:02 smithi055 ceph-mon[97033]: pgmap v8955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:16:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:02 smithi059 ceph-mon[97450]: pgmap v8955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:16:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:03.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:03.594978+0000) 2022-09-08T02:16:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:03.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:03.540104+0000) 2022-09-08T02:16:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:04.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:04.595098+0000) 2022-09-08T02:16:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:04.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:04.540292+0000) 2022-09-08T02:16:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:04 smithi055 ceph-mon[97033]: pgmap v8956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:04 smithi059 ceph-mon[97450]: pgmap v8956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:05.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:05.595366+0000) 2022-09-08T02:16:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:05.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:05.540526+0000) 2022-09-08T02:16:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:16:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:16:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:16:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:16:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:06.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:06.595602+0000) 2022-09-08T02:16:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:06.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:06.540730+0000) 2022-09-08T02:16:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:06 smithi055 ceph-mon[97033]: pgmap v8957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:16:07.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:06 smithi059 ceph-mon[97450]: pgmap v8957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:16:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:07.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:07.595777+0000) 2022-09-08T02:16:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:07.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:07.540919+0000) 2022-09-08T02:16:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:08.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:08.595952+0000) 2022-09-08T02:16:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:08.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:08.541114+0000) 2022-09-08T02:16:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:08 smithi055 ceph-mon[97033]: pgmap v8958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:16:09.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:08 smithi059 ceph-mon[97450]: pgmap v8958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:16:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:09.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:09.596170+0000) 2022-09-08T02:16:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:09.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:09.541282+0000) 2022-09-08T02:16:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:09 smithi055 ceph-mon[97033]: osdmap e641: 12 total, 11 up, 11 in 2022-09-08T02:16:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:09 smithi059 ceph-mon[97450]: osdmap e641: 12 total, 11 up, 11 in 2022-09-08T02:16:10.596 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:10.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:10.541534+0000) 2022-09-08T02:16:10.598 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:16:10.328+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:16:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:10.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:10.596388+0000) 2022-09-08T02:16:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:10 smithi055 ceph-mon[97033]: pgmap v8960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:16:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:10 smithi059 ceph-mon[97450]: pgmap v8960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:16:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:11.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:11.541757+0000) 2022-09-08T02:16:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:11.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:11.596664+0000) 2022-09-08T02:16:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:12.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:12.596900+0000) 2022-09-08T02:16:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:12.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:12.541945+0000) 2022-09-08T02:16:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:12 smithi055 ceph-mon[97033]: pgmap v8961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:16:12.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:12 smithi055 ceph-mon[97033]: osdmap e642: 12 total, 11 up, 11 in 2022-09-08T02:16:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:12 smithi059 ceph-mon[97450]: pgmap v8961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:16:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:12 smithi059 ceph-mon[97450]: osdmap e642: 12 total, 11 up, 11 in 2022-09-08T02:16:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:13.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:13.597142+0000) 2022-09-08T02:16:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:13.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:13.542197+0000) 2022-09-08T02:16:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:14.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:14.542406+0000) 2022-09-08T02:16:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:14.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:14.597362+0000) 2022-09-08T02:16:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:14 smithi055 ceph-mon[97033]: pgmap v8963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:16:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:14 smithi059 ceph-mon[97450]: pgmap v8963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:16:15.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:15.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:15.597616+0000) 2022-09-08T02:16:15.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:15.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:15.542674+0000) 2022-09-08T02:16:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:16.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:16.597843+0000) 2022-09-08T02:16:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:16.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:16.542928+0000) 2022-09-08T02:16:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:16 smithi055 ceph-mon[97033]: pgmap v8964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:16:17.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:16 smithi059 ceph-mon[97450]: pgmap v8964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:16:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:17.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:17.598101+0000) 2022-09-08T02:16:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:17.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:17.543156+0000) 2022-09-08T02:16:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:18.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:18.543298+0000) 2022-09-08T02:16:18.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:18.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:18.598359+0000) 2022-09-08T02:16:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:18 smithi055 ceph-mon[97033]: pgmap v8965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-08T02:16:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:18 smithi059 ceph-mon[97450]: pgmap v8965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-08T02:16:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:19.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:19.598589+0000) 2022-09-08T02:16:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:19.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:19.543445+0000) 2022-09-08T02:16:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:20.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:20.543579+0000) 2022-09-08T02:16:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:20.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:20.598715+0000) 2022-09-08T02:16:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:20 smithi055 ceph-mon[97033]: pgmap v8966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:16:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:20 smithi059 ceph-mon[97450]: pgmap v8966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:16:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:21.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:21.598925+0000) 2022-09-08T02:16:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:21.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:21.543800+0000) 2022-09-08T02:16:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:22.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:22.599114+0000) 2022-09-08T02:16:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:22.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:22.544032+0000) 2022-09-08T02:16:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:22 smithi055 ceph-mon[97033]: pgmap v8967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:16:23.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:22 smithi059 ceph-mon[97450]: pgmap v8967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:16:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:23.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:23.599317+0000) 2022-09-08T02:16:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:23.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:23.544266+0000) 2022-09-08T02:16:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:24.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:24.599573+0000) 2022-09-08T02:16:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:24.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:24.544467+0000) 2022-09-08T02:16:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:24 smithi055 ceph-mon[97033]: pgmap v8968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 94 B/s wr, 5 op/s 2022-09-08T02:16:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:24 smithi059 ceph-mon[97450]: pgmap v8968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 94 B/s wr, 5 op/s 2022-09-08T02:16:25.600 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:16:25.343+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:16:25.602 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:25.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:25.544715+0000) 2022-09-08T02:16:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:25.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:25.599835+0000) 2022-09-08T02:16:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:26.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:26.599981+0000) 2022-09-08T02:16:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:26.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:26.544912+0000) 2022-09-08T02:16:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:26 smithi055 ceph-mon[97033]: pgmap v8969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:16:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:26 smithi059 ceph-mon[97450]: pgmap v8969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:16:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:27.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:27.600192+0000) 2022-09-08T02:16:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:27.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:27.545118+0000) 2022-09-08T02:16:28.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:28.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:28.600450+0000) 2022-09-08T02:16:28.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:28.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:28.545313+0000) 2022-09-08T02:16:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:28 smithi055 ceph-mon[97033]: pgmap v8970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:28 smithi059 ceph-mon[97450]: pgmap v8970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:29.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:29.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:29.600624+0000) 2022-09-08T02:16:29.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:29.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:29.545483+0000) 2022-09-08T02:16:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:16:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:16:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:16:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:16:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:16:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:16:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:30 smithi059 ceph-mon[97450]: pgmap v8971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:30.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:30.600840+0000) 2022-09-08T02:16:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:30.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:30.545651+0000) 2022-09-08T02:16:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:30 smithi055 ceph-mon[97033]: pgmap v8971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:31.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:31.601001+0000) 2022-09-08T02:16:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:31.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:31.545808+0000) 2022-09-08T02:16:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:32.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:32.601213+0000) 2022-09-08T02:16:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:32.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:32.545994+0000) 2022-09-08T02:16:32.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:32 smithi055 ceph-mon[97033]: pgmap v8972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:16:33.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:32 smithi059 ceph-mon[97450]: pgmap v8972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:16:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:33.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:33.601458+0000) 2022-09-08T02:16:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:33.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:33.546217+0000) 2022-09-08T02:16:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:34 smithi059 ceph-mon[97450]: pgmap v8973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:34 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T02:16:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:34.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:34.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:34.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:34.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:34.601717+0000) 2022-09-08T02:16:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:34.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:34.546453+0000) 2022-09-08T02:16:34.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:34 smithi055 ceph-mon[97033]: pgmap v8973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:34.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:34 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T02:16:34.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:34.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:34.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:34.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:16:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:35 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T02:16:35.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:35.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:35.601935+0000) 2022-09-08T02:16:35.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:35.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:35.546582+0000) 2022-09-08T02:16:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:35 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T02:16:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:36 smithi059 ceph-mon[97450]: pgmap v8974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:16:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:36.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:36.602175+0000) 2022-09-08T02:16:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:36.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:36.546818+0000) 2022-09-08T02:16:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:36 smithi055 ceph-mon[97033]: pgmap v8974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:16:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:37.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:37.602425+0000) 2022-09-08T02:16:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:37.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:37.547083+0000) 2022-09-08T02:16:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:38.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:38.602658+0000) 2022-09-08T02:16:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:38.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:38.547288+0000) 2022-09-08T02:16:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:38 smithi055 ceph-mon[97033]: pgmap v8975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:38 smithi059 ceph-mon[97450]: pgmap v8975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:39.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:39.602888+0000) 2022-09-08T02:16:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:39.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:39.547537+0000) 2022-09-08T02:16:40.604 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:16:40.351+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:16:40.605 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:40.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:40.547773+0000) 2022-09-08T02:16:40.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:40.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:40.603061+0000) 2022-09-08T02:16:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:40 smithi055 ceph-mon[97033]: pgmap v8976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:40 smithi059 ceph-mon[97450]: pgmap v8976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:41.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:41.603287+0000) 2022-09-08T02:16:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:41.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:41.547979+0000) 2022-09-08T02:16:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:42.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:42.548185+0000) 2022-09-08T02:16:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:42.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:42.603473+0000) 2022-09-08T02:16:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:42 smithi055 ceph-mon[97033]: pgmap v8977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:16:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:42 smithi059 ceph-mon[97450]: pgmap v8977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:16:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:43.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:43.603687+0000) 2022-09-08T02:16:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:43.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:43.548451+0000) 2022-09-08T02:16:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:44.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:44.603921+0000) 2022-09-08T02:16:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:44.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:44.548684+0000) 2022-09-08T02:16:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:44 smithi055 ceph-mon[97033]: pgmap v8978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:45.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:44 smithi059 ceph-mon[97450]: pgmap v8978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:45.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:45.604139+0000) 2022-09-08T02:16:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:45.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:45.548900+0000) 2022-09-08T02:16:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:46.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:46.604255+0000) 2022-09-08T02:16:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:46.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:46.549097+0000) 2022-09-08T02:16:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:46 smithi055 ceph-mon[97033]: pgmap v8979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:16:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:46 smithi059 ceph-mon[97450]: pgmap v8979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:16:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:47.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:47.604448+0000) 2022-09-08T02:16:47.915 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:47.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:47.549299+0000) 2022-09-08T02:16:48.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:48.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:48.604651+0000) 2022-09-08T02:16:48.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:48.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:48.549499+0000) 2022-09-08T02:16:48.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:48 smithi055 ceph-mon[97033]: pgmap v8980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:48 smithi059 ceph-mon[97450]: pgmap v8980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:49.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:49.604894+0000) 2022-09-08T02:16:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:49.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:49.549692+0000) 2022-09-08T02:16:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:50.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:50.605130+0000) 2022-09-08T02:16:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:50.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:50.549811+0000) 2022-09-08T02:16:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:50 smithi055 ceph-mon[97033]: pgmap v8981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:50 smithi059 ceph-mon[97450]: pgmap v8981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:51.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:51.605342+0000) 2022-09-08T02:16:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:51.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:51.549942+0000) 2022-09-08T02:16:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:52.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:52.605529+0000) 2022-09-08T02:16:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:52.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:52.550081+0000) 2022-09-08T02:16:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:52 smithi055 ceph-mon[97033]: pgmap v8982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:16:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:52 smithi059 ceph-mon[97450]: pgmap v8982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:16:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:53.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:53.605809+0000) 2022-09-08T02:16:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:53.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:53.550249+0000) 2022-09-08T02:16:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:54.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:54.605969+0000) 2022-09-08T02:16:54.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:54.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:54.550451+0000) 2022-09-08T02:16:54.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:54 smithi055 ceph-mon[97033]: pgmap v8983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:54 smithi059 ceph-mon[97450]: pgmap v8983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:55.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:16:55.366+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:16:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:55.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:55.606084+0000) 2022-09-08T02:16:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:55.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:55.550726+0000) 2022-09-08T02:16:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:56.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:56.606300+0000) 2022-09-08T02:16:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:56.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:56.550890+0000) 2022-09-08T02:16:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:56 smithi055 ceph-mon[97033]: pgmap v8984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:16:57.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:56 smithi059 ceph-mon[97450]: pgmap v8984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:16:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:57.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:57.551075+0000) 2022-09-08T02:16:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:57.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:57.606526+0000) 2022-09-08T02:16:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:58.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:58.606777+0000) 2022-09-08T02:16:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:58.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:58.551236+0000) 2022-09-08T02:16:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:16:58 smithi055 ceph-mon[97033]: pgmap v8985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:16:58 smithi059 ceph-mon[97450]: pgmap v8985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:16:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:16:59.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:59.607024+0000) 2022-09-08T02:16:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:16:59.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:16:59.551483+0000) 2022-09-08T02:17:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:00.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:00.607181+0000) 2022-09-08T02:17:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:00.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:00.551683+0000) 2022-09-08T02:17:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:00 smithi055 ceph-mon[97033]: pgmap v8986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:00 smithi055 ceph-mon[97033]: pgmap v8987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:17:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:00 smithi055 ceph-mon[97033]: pgmap v8988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:17:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:17:01.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:00 smithi059 ceph-mon[97450]: pgmap v8986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:00 smithi059 ceph-mon[97450]: pgmap v8987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:17:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:00 smithi059 ceph-mon[97450]: pgmap v8988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:17:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:17:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:01.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:01.607406+0000) 2022-09-08T02:17:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:01.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:01.551894+0000) 2022-09-08T02:17:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:02.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:02.607552+0000) 2022-09-08T02:17:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:02.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:02.552088+0000) 2022-09-08T02:17:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:02 smithi055 ceph-mon[97033]: pgmap v8989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:17:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:02 smithi059 ceph-mon[97450]: pgmap v8989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:17:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:03.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:03.607788+0000) 2022-09-08T02:17:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:03.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:03.552324+0000) 2022-09-08T02:17:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:04.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:04.607953+0000) 2022-09-08T02:17:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:04.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:04.552560+0000) 2022-09-08T02:17:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:04 smithi055 ceph-mon[97033]: pgmap v8990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:17:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:04 smithi059 ceph-mon[97450]: pgmap v8990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:17:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:05.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:05.608148+0000) 2022-09-08T02:17:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:05.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:05.552814+0000) 2022-09-08T02:17:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:17:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:17:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:17:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:17:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:06.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:06.608362+0000) 2022-09-08T02:17:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:06.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:06.553028+0000) 2022-09-08T02:17:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:06 smithi055 ceph-mon[97033]: pgmap v8991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T02:17:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:06 smithi059 ceph-mon[97450]: pgmap v8991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T02:17:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:07.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:07.608655+0000) 2022-09-08T02:17:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:07.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:07.553192+0000) 2022-09-08T02:17:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:08.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:08.553435+0000) 2022-09-08T02:17:08.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:08.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:08.608868+0000) 2022-09-08T02:17:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:08 smithi055 ceph-mon[97033]: pgmap v8992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T02:17:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:08 smithi059 ceph-mon[97450]: pgmap v8992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T02:17:09.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:09.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:09.609094+0000) 2022-09-08T02:17:09.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:09.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:09.553699+0000) 2022-09-08T02:17:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:09 smithi055 ceph-mon[97033]: osdmap e643: 12 total, 11 up, 11 in 2022-09-08T02:17:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:09 smithi059 ceph-mon[97450]: osdmap e643: 12 total, 11 up, 11 in 2022-09-08T02:17:10.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:17:10.372+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:17:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:10.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:10.609348+0000) 2022-09-08T02:17:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:10.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:10.553918+0000) 2022-09-08T02:17:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:10 smithi055 ceph-mon[97033]: pgmap v8994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:17:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:10 smithi059 ceph-mon[97450]: pgmap v8994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:17:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:11.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:11.609555+0000) 2022-09-08T02:17:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:11.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:11.554161+0000) 2022-09-08T02:17:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:12.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:12.609716+0000) 2022-09-08T02:17:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:12.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:12.554441+0000) 2022-09-08T02:17:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:12 smithi055 ceph-mon[97033]: pgmap v8995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:17:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:12 smithi055 ceph-mon[97033]: osdmap e644: 12 total, 11 up, 11 in 2022-09-08T02:17:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:12 smithi059 ceph-mon[97450]: pgmap v8995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:17:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:12 smithi059 ceph-mon[97450]: osdmap e644: 12 total, 11 up, 11 in 2022-09-08T02:17:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:13.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:13.609908+0000) 2022-09-08T02:17:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:13.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:13.554749+0000) 2022-09-08T02:17:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:14.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:14.554970+0000) 2022-09-08T02:17:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:14.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:14.610128+0000) 2022-09-08T02:17:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:14 smithi055 ceph-mon[97033]: pgmap v8997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:17:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:14 smithi059 ceph-mon[97450]: pgmap v8997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:17:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:15.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:15.555159+0000) 2022-09-08T02:17:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:15.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:15.610287+0000) 2022-09-08T02:17:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:16.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:16.555393+0000) 2022-09-08T02:17:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:16.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:16.610536+0000) 2022-09-08T02:17:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:16 smithi055 ceph-mon[97033]: pgmap v8998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:17:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:16 smithi059 ceph-mon[97450]: pgmap v8998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:17:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:17.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:17.555706+0000) 2022-09-08T02:17:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:17.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:17.610816+0000) 2022-09-08T02:17:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:18.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:18.611044+0000) 2022-09-08T02:17:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:18.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:18.555941+0000) 2022-09-08T02:17:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:18 smithi055 ceph-mon[97033]: pgmap v8999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 225 B/s wr, 7 op/s 2022-09-08T02:17:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:18 smithi059 ceph-mon[97450]: pgmap v8999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 225 B/s wr, 7 op/s 2022-09-08T02:17:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:19.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:19.611260+0000) 2022-09-08T02:17:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:19.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:19.556086+0000) 2022-09-08T02:17:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:20.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:20.611488+0000) 2022-09-08T02:17:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:20.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:20.556304+0000) 2022-09-08T02:17:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:20 smithi055 ceph-mon[97033]: pgmap v9000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:17:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:20 smithi059 ceph-mon[97450]: pgmap v9000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:17:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:21.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:21.611743+0000) 2022-09-08T02:17:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:21.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:21.556501+0000) 2022-09-08T02:17:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:22.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:22.611978+0000) 2022-09-08T02:17:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:22.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:22.556723+0000) 2022-09-08T02:17:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:22 smithi055 ceph-mon[97033]: pgmap v9001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:17:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:22 smithi059 ceph-mon[97450]: pgmap v9001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:17:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:23.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:23.612239+0000) 2022-09-08T02:17:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:23.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:23.556918+0000) 2022-09-08T02:17:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:24.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:24.612453+0000) 2022-09-08T02:17:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:24.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:24.557092+0000) 2022-09-08T02:17:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:24 smithi055 ceph-mon[97033]: pgmap v9002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:17:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:24 smithi059 ceph-mon[97450]: pgmap v9002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:17:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:25.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:25.612718+0000) 2022-09-08T02:17:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:25.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:25.557290+0000) 2022-09-08T02:17:25.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:17:25.380+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:17:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:26.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:26.612969+0000) 2022-09-08T02:17:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:26.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:26.557426+0000) 2022-09-08T02:17:27.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:26 smithi059 ceph-mon[97450]: pgmap v9003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:17:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:26 smithi055 ceph-mon[97033]: pgmap v9003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:17:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:27.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:27.613185+0000) 2022-09-08T02:17:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:27.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:27.557633+0000) 2022-09-08T02:17:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:28.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:28.613411+0000) 2022-09-08T02:17:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:28.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:28.557906+0000) 2022-09-08T02:17:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:28 smithi055 ceph-mon[97033]: pgmap v9004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:28 smithi059 ceph-mon[97450]: pgmap v9004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:29.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:29.613567+0000) 2022-09-08T02:17:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:29.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:29.558138+0000) 2022-09-08T02:17:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:30.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:30.613789+0000) 2022-09-08T02:17:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:30.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:30.558315+0000) 2022-09-08T02:17:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:30 smithi055 ceph-mon[97033]: pgmap v9005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:31.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:30 smithi059 ceph-mon[97450]: pgmap v9005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:31.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:31.614060+0000) 2022-09-08T02:17:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:31.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:31.558568+0000) 2022-09-08T02:17:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:32.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:32.614271+0000) 2022-09-08T02:17:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:32.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:32.558757+0000) 2022-09-08T02:17:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:32 smithi055 ceph-mon[97033]: pgmap v9006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:17:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:32 smithi059 ceph-mon[97450]: pgmap v9006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:17:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:33.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:33.558990+0000) 2022-09-08T02:17:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:33.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:33.614464+0000) 2022-09-08T02:17:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:34.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:34.614663+0000) 2022-09-08T02:17:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:34.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:34.559169+0000) 2022-09-08T02:17:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:34 smithi059 ceph-mon[97450]: pgmap v9007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:34 smithi055 ceph-mon[97033]: pgmap v9007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:35.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:35.614856+0000) 2022-09-08T02:17:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:35.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:35.559404+0000) 2022-09-08T02:17:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:36.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:36.615089+0000) 2022-09-08T02:17:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:36.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:36.559583+0000) 2022-09-08T02:17:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:36 smithi059 ceph-mon[97450]: pgmap v9008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:17:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:36 smithi055 ceph-mon[97033]: pgmap v9008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:17:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:37.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:37.615346+0000) 2022-09-08T02:17:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:37.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:37.559838+0000) 2022-09-08T02:17:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:38.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:38.615475+0000) 2022-09-08T02:17:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:38.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:38.560021+0000) 2022-09-08T02:17:38.915 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:38 smithi055 ceph-mon[97033]: pgmap v9009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:38 smithi059 ceph-mon[97450]: pgmap v9009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:39.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:39.615739+0000) 2022-09-08T02:17:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:39.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:39.560160+0000) 2022-09-08T02:17:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:40.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:40.615972+0000) 2022-09-08T02:17:40.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:17:40.396+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:17:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:40.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:40.560369+0000) 2022-09-08T02:17:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:40 smithi055 ceph-mon[97033]: pgmap v9010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:40 smithi059 ceph-mon[97450]: pgmap v9010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:41.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:41.616261+0000) 2022-09-08T02:17:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:41.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:41.560622+0000) 2022-09-08T02:17:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:42.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:42.616550+0000) 2022-09-08T02:17:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:42.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:42.560865+0000) 2022-09-08T02:17:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:42 smithi059 ceph-mon[97450]: pgmap v9011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:17:43.058 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:42 smithi055 ceph-mon[97033]: pgmap v9011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:17:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:43.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:43.616774+0000) 2022-09-08T02:17:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:43.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:43.561001+0000) 2022-09-08T02:17:44.909 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:44.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:44.617074+0000) 2022-09-08T02:17:44.912 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:44.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:44.561219+0000) 2022-09-08T02:17:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:44 smithi059 ceph-mon[97450]: pgmap v9012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:44 smithi055 ceph-mon[97033]: pgmap v9012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:45.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:45.617305+0000) 2022-09-08T02:17:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:45.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:45.561456+0000) 2022-09-08T02:17:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:46.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:46.561672+0000) 2022-09-08T02:17:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:46.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:46.617497+0000) 2022-09-08T02:17:47.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:46 smithi059 ceph-mon[97450]: pgmap v9013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:17:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:46 smithi055 ceph-mon[97033]: pgmap v9013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:17:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:47.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:47.561915+0000) 2022-09-08T02:17:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:47.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:47.617733+0000) 2022-09-08T02:17:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:48.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:48.617998+0000) 2022-09-08T02:17:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:48.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:48.562142+0000) 2022-09-08T02:17:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:48 smithi055 ceph-mon[97033]: pgmap v9014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:49.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:48 smithi059 ceph-mon[97450]: pgmap v9014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:49.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:49.618253+0000) 2022-09-08T02:17:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:49.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:49.562346+0000) 2022-09-08T02:17:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:50.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:50.562555+0000) 2022-09-08T02:17:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:50.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:50.618494+0000) 2022-09-08T02:17:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:50 smithi059 ceph-mon[97450]: pgmap v9015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:50 smithi055 ceph-mon[97033]: pgmap v9015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:51.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:51.618762+0000) 2022-09-08T02:17:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:51.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:51.562752+0000) 2022-09-08T02:17:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:52.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:52.562973+0000) 2022-09-08T02:17:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:52.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:52.618998+0000) 2022-09-08T02:17:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:52 smithi059 ceph-mon[97450]: pgmap v9016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:17:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:52 smithi055 ceph-mon[97033]: pgmap v9016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:17:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:53.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:53.563199+0000) 2022-09-08T02:17:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:53.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:53.619209+0000) 2022-09-08T02:17:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:54.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:54.619435+0000) 2022-09-08T02:17:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:54.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:54.563393+0000) 2022-09-08T02:17:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:54 smithi059 ceph-mon[97450]: pgmap v9017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:54 smithi055 ceph-mon[97033]: pgmap v9017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:55.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:55.619723+0000) 2022-09-08T02:17:55.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:17:55.403+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:17:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:55.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:55.563620+0000) 2022-09-08T02:17:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:56.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:56.619921+0000) 2022-09-08T02:17:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:56.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:56.563848+0000) 2022-09-08T02:17:57.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:56 smithi059 ceph-mon[97450]: pgmap v9018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:17:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:56 smithi055 ceph-mon[97033]: pgmap v9018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:17:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:57.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:57.620130+0000) 2022-09-08T02:17:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:57.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:57.564083+0000) 2022-09-08T02:17:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:58.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:58.620412+0000) 2022-09-08T02:17:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:58.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:58.564283+0000) 2022-09-08T02:17:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:17:58 smithi055 ceph-mon[97033]: pgmap v9019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:17:58 smithi059 ceph-mon[97450]: pgmap v9019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:17:59.829 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:17:59.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:59.620736+0000) 2022-09-08T02:17:59.830 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:17:59.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:17:59.564584+0000) 2022-09-08T02:18:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:00.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:00.621005+0000) 2022-09-08T02:18:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:00.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:00.564773+0000) 2022-09-08T02:18:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:00 smithi059 ceph-mon[97450]: pgmap v9020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:18:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:18:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:18:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:00 smithi055 ceph-mon[97033]: pgmap v9020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:18:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:18:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:18:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:01.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:01.621265+0000) 2022-09-08T02:18:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:01.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:01.564993+0000) 2022-09-08T02:18:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:02.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:02.621496+0000) 2022-09-08T02:18:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:02.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:02.565194+0000) 2022-09-08T02:18:03.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:02 smithi059 ceph-mon[97450]: pgmap v9021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:18:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:03.105 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:02 smithi055 ceph-mon[97033]: pgmap v9021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:18:03.106 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:03.106 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:03.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:03.621756+0000) 2022-09-08T02:18:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:03.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:03.565424+0000) 2022-09-08T02:18:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:04.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T02:18:04.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T02:18:04.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T02:18:04.487 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T02:18:04.487 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:04 smithi055 ceph-mon[97033]: pgmap v9022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T02:18:04.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T02:18:04.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:18:04.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T02:18:04.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T02:18:04.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:04 smithi059 ceph-mon[97450]: pgmap v9022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:04.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:04.621999+0000) 2022-09-08T02:18:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:04.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:04.565727+0000) 2022-09-08T02:18:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:18:05.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:18:05.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:05.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:05.622160+0000) 2022-09-08T02:18:05.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:05.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:05.565970+0000) 2022-09-08T02:18:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:18:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:06 smithi055 ceph-mon[97033]: pgmap v9023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:18:06.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:18:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:06 smithi059 ceph-mon[97450]: pgmap v9023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:18:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:06.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:06.622392+0000) 2022-09-08T02:18:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:06.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:06.566158+0000) 2022-09-08T02:18:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:07.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:07.622626+0000) 2022-09-08T02:18:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:07.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:07.566398+0000) 2022-09-08T02:18:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:08.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:08.622844+0000) 2022-09-08T02:18:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:08.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:08.566642+0000) 2022-09-08T02:18:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:08 smithi055 ceph-mon[97033]: pgmap v9024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:18:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:08 smithi059 ceph-mon[97450]: pgmap v9024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:18:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:09.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:09.623024+0000) 2022-09-08T02:18:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:09.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:09.566815+0000) 2022-09-08T02:18:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:09 smithi055 ceph-mon[97033]: osdmap e645: 12 total, 11 up, 11 in 2022-09-08T02:18:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:09 smithi059 ceph-mon[97450]: osdmap e645: 12 total, 11 up, 11 in 2022-09-08T02:18:10.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:18:10.405+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:18:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:10.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:10.623236+0000) 2022-09-08T02:18:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:10.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:10.567045+0000) 2022-09-08T02:18:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:10 smithi055 ceph-mon[97033]: pgmap v9026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:18:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:10 smithi059 ceph-mon[97450]: pgmap v9026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:18:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:11.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:11.623434+0000) 2022-09-08T02:18:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:11.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:11.567290+0000) 2022-09-08T02:18:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:12.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:12.623695+0000) 2022-09-08T02:18:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:12.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:12.567536+0000) 2022-09-08T02:18:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:12 smithi059 ceph-mon[97450]: pgmap v9027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:18:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:12 smithi059 ceph-mon[97450]: osdmap e646: 12 total, 11 up, 11 in 2022-09-08T02:18:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:12 smithi055 ceph-mon[97033]: pgmap v9027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:18:13.302 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:12 smithi055 ceph-mon[97033]: osdmap e646: 12 total, 11 up, 11 in 2022-09-08T02:18:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:13.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:13.623907+0000) 2022-09-08T02:18:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:13.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:13.567704+0000) 2022-09-08T02:18:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:14.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:14.624098+0000) 2022-09-08T02:18:14.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:14.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:14.567901+0000) 2022-09-08T02:18:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:14 smithi059 ceph-mon[97450]: pgmap v9029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:18:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:14 smithi055 ceph-mon[97033]: pgmap v9029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:18:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:15.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:15.624339+0000) 2022-09-08T02:18:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:15.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:15.568083+0000) 2022-09-08T02:18:16.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:16.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:16.624522+0000) 2022-09-08T02:18:16.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:16.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:16.568276+0000) 2022-09-08T02:18:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:16 smithi059 ceph-mon[97450]: pgmap v9030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:18:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:16 smithi055 ceph-mon[97033]: pgmap v9030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:18:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:17.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:17.624744+0000) 2022-09-08T02:18:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:17.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:17.568521+0000) 2022-09-08T02:18:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:18.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:18.624891+0000) 2022-09-08T02:18:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:18.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:18.568762+0000) 2022-09-08T02:18:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:18 smithi055 ceph-mon[97033]: pgmap v9031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 225 B/s wr, 6 op/s 2022-09-08T02:18:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:18 smithi059 ceph-mon[97450]: pgmap v9031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 225 B/s wr, 6 op/s 2022-09-08T02:18:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:19.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:19.625089+0000) 2022-09-08T02:18:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:19.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:19.569000+0000) 2022-09-08T02:18:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:20.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:20.625335+0000) 2022-09-08T02:18:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:20.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:20.569251+0000) 2022-09-08T02:18:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:20 smithi059 ceph-mon[97450]: pgmap v9032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:18:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:20 smithi055 ceph-mon[97033]: pgmap v9032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:18:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:21.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:21.625582+0000) 2022-09-08T02:18:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:21.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:21.569466+0000) 2022-09-08T02:18:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:22.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:22.625692+0000) 2022-09-08T02:18:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:22.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:22.569725+0000) 2022-09-08T02:18:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:22 smithi059 ceph-mon[97450]: pgmap v9033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T02:18:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:22 smithi055 ceph-mon[97033]: pgmap v9033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T02:18:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:23.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:23.625868+0000) 2022-09-08T02:18:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:23.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:23.569844+0000) 2022-09-08T02:18:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:23 smithi055 ceph-mon[97033]: pgmap v9034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:18:24.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:23 smithi059 ceph-mon[97450]: pgmap v9034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:18:24.895 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:24.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:24.626106+0000) 2022-09-08T02:18:24.895 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:24.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:24.570083+0000) 2022-09-08T02:18:25.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:18:25.420+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:18:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:25.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:25.626331+0000) 2022-09-08T02:18:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:25.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:25.570313+0000) 2022-09-08T02:18:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:26.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:26.570529+0000) 2022-09-08T02:18:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:26.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:26.626557+0000) 2022-09-08T02:18:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:26 smithi059 ceph-mon[97450]: pgmap v9035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:18:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:26 smithi055 ceph-mon[97033]: pgmap v9035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:18:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:27.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:27.626723+0000) 2022-09-08T02:18:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:27.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:27.570786+0000) 2022-09-08T02:18:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:28.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:28.626926+0000) 2022-09-08T02:18:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:28.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:28.570990+0000) 2022-09-08T02:18:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:28 smithi055 ceph-mon[97033]: pgmap v9036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:28 smithi059 ceph-mon[97450]: pgmap v9036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:29.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:29.627127+0000) 2022-09-08T02:18:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:29.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:29.571324+0000) 2022-09-08T02:18:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:30.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:30.571563+0000) 2022-09-08T02:18:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:30.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:30.627347+0000) 2022-09-08T02:18:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:30 smithi059 ceph-mon[97450]: pgmap v9037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:30 smithi055 ceph-mon[97033]: pgmap v9037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:31.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:31.627570+0000) 2022-09-08T02:18:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:31.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:31.571708+0000) 2022-09-08T02:18:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:32.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:32.627791+0000) 2022-09-08T02:18:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:32.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:32.571924+0000) 2022-09-08T02:18:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:32 smithi059 ceph-mon[97450]: pgmap v9038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:18:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:32 smithi055 ceph-mon[97033]: pgmap v9038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:18:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:33.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:33.572108+0000) 2022-09-08T02:18:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:33.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:33.627930+0000) 2022-09-08T02:18:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:33 smithi059 ceph-mon[97450]: pgmap v9039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:33 smithi055 ceph-mon[97033]: pgmap v9039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:34.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:34.628138+0000) 2022-09-08T02:18:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:34.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:34.572336+0000) 2022-09-08T02:18:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:35 smithi059 ceph-mon[97450]: pgmap v9040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T02:18:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:35 smithi059 ceph-mon[97450]: pgmap v9041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T02:18:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:35.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:35.572593+0000) 2022-09-08T02:18:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:35.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:35.628365+0000) 2022-09-08T02:18:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:35 smithi055 ceph-mon[97033]: pgmap v9040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T02:18:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:35 smithi055 ceph-mon[97033]: pgmap v9041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T02:18:35.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:18:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:36.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:36.628588+0000) 2022-09-08T02:18:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:36.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:36.572852+0000) 2022-09-08T02:18:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:37.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:37.628763+0000) 2022-09-08T02:18:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:37.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:37.573043+0000) 2022-09-08T02:18:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:37 smithi055 ceph-mon[97033]: pgmap v9042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T02:18:38.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:37 smithi059 ceph-mon[97450]: pgmap v9042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T02:18:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:38.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:38.629011+0000) 2022-09-08T02:18:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:38.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:38.573256+0000) 2022-09-08T02:18:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:39.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:39.629252+0000) 2022-09-08T02:18:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:39.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:39.573458+0000) 2022-09-08T02:18:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:39 smithi055 ceph-mon[97033]: pgmap v9043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T02:18:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:39 smithi059 ceph-mon[97450]: pgmap v9043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T02:18:40.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:18:40.435+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:18:40.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:40.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:40.629474+0000) 2022-09-08T02:18:40.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:40.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:40.573682+0000) 2022-09-08T02:18:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:41.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:41.629744+0000) 2022-09-08T02:18:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:41.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:41.573924+0000) 2022-09-08T02:18:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:41 smithi055 ceph-mon[97033]: pgmap v9044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T02:18:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:41 smithi059 ceph-mon[97450]: pgmap v9044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T02:18:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:42.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:42.574175+0000) 2022-09-08T02:18:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:42.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:42.629942+0000) 2022-09-08T02:18:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:43.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:43.630146+0000) 2022-09-08T02:18:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:43.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:43.574416+0000) 2022-09-08T02:18:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:43 smithi055 ceph-mon[97033]: pgmap v9045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T02:18:44.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:43 smithi059 ceph-mon[97450]: pgmap v9045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T02:18:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:44.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:44.630414+0000) 2022-09-08T02:18:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:44.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:44.574739+0000) 2022-09-08T02:18:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:45.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:45.630665+0000) 2022-09-08T02:18:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:45.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:45.574907+0000) 2022-09-08T02:18:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:45 smithi055 ceph-mon[97033]: pgmap v9046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:18:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:45 smithi059 ceph-mon[97450]: pgmap v9046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:18:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:46.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:46.630875+0000) 2022-09-08T02:18:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:46.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:46.575119+0000) 2022-09-08T02:18:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:47.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:47.631054+0000) 2022-09-08T02:18:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:47.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:47.575344+0000) 2022-09-08T02:18:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:47 smithi055 ceph-mon[97033]: pgmap v9047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:18:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:47 smithi059 ceph-mon[97450]: pgmap v9047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:18:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:48.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:48.631237+0000) 2022-09-08T02:18:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:48.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:48.575577+0000) 2022-09-08T02:18:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:49.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:49.631486+0000) 2022-09-08T02:18:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:49.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:49.575760+0000) 2022-09-08T02:18:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:49 smithi055 ceph-mon[97033]: pgmap v9048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:49 smithi059 ceph-mon[97450]: pgmap v9048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:50.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:50.631728+0000) 2022-09-08T02:18:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:50.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:50.575970+0000) 2022-09-08T02:18:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:51.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:51.631957+0000) 2022-09-08T02:18:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:51.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:51.576213+0000) 2022-09-08T02:18:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:51 smithi055 ceph-mon[97033]: pgmap v9049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:18:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:51 smithi059 ceph-mon[97450]: pgmap v9049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:18:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:52.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:52.632169+0000) 2022-09-08T02:18:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:52.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:52.576438+0000) 2022-09-08T02:18:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:53.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:53.632426+0000) 2022-09-08T02:18:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:53.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:53.576593+0000) 2022-09-08T02:18:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:53 smithi055 ceph-mon[97033]: pgmap v9050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:53 smithi059 ceph-mon[97450]: pgmap v9050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:54.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:54.632675+0000) 2022-09-08T02:18:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:54.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:54.576755+0000) 2022-09-08T02:18:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:55.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:55.632900+0000) 2022-09-08T02:18:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:55.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:55.576938+0000) 2022-09-08T02:18:55.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:18:55.450+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:18:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:55 smithi055 ceph-mon[97033]: pgmap v9051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:55 smithi059 ceph-mon[97450]: pgmap v9051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:18:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:56.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:56.633083+0000) 2022-09-08T02:18:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:56.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:56.577139+0000) 2022-09-08T02:18:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:57.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:57.633318+0000) 2022-09-08T02:18:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:57.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:57.577376+0000) 2022-09-08T02:18:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:57 smithi055 ceph-mon[97033]: pgmap v9052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:18:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:57 smithi059 ceph-mon[97450]: pgmap v9052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:18:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:58.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:58.633560+0000) 2022-09-08T02:18:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:58.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:58.577624+0000) 2022-09-08T02:18:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:18:59.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:59.633771+0000) 2022-09-08T02:18:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:18:59.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:18:59.577892+0000) 2022-09-08T02:18:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:18:59 smithi055 ceph-mon[97033]: pgmap v9053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:18:59 smithi059 ceph-mon[97450]: pgmap v9053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:00.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:00.633936+0000) 2022-09-08T02:19:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:00.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:00.578035+0000) 2022-09-08T02:19:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:01.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:01.634091+0000) 2022-09-08T02:19:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:01.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:01.578275+0000) 2022-09-08T02:19:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:01 smithi055 ceph-mon[97033]: pgmap v9054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:19:02.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:01 smithi059 ceph-mon[97450]: pgmap v9054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:19:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:02.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:02.634304+0000) 2022-09-08T02:19:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:02.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:02.578494+0000) 2022-09-08T02:19:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:03.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:03.578703+0000) 2022-09-08T02:19:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:03.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:03.634531+0000) 2022-09-08T02:19:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:03 smithi055 ceph-mon[97033]: pgmap v9055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:03 smithi059 ceph-mon[97450]: pgmap v9055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:04.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:04.634747+0000) 2022-09-08T02:19:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:04.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:04.578838+0000) 2022-09-08T02:19:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:05.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:05.579079+0000) 2022-09-08T02:19:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:05.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:05.634963+0000) 2022-09-08T02:19:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:05 smithi055 ceph-mon[97033]: pgmap v9056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:19:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:19:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:05 smithi059 ceph-mon[97450]: pgmap v9056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:19:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:19:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:06.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:06.635139+0000) 2022-09-08T02:19:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:06.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:06.579329+0000) 2022-09-08T02:19:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:07.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:07.635351+0000) 2022-09-08T02:19:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:07.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:07.579573+0000) 2022-09-08T02:19:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:07 smithi055 ceph-mon[97033]: pgmap v9057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:19:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:07 smithi059 ceph-mon[97450]: pgmap v9057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:19:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:08.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:08.635593+0000) 2022-09-08T02:19:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:08.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:08.579821+0000) 2022-09-08T02:19:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:09.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:09.635835+0000) 2022-09-08T02:19:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:09.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:09.579992+0000) 2022-09-08T02:19:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:09 smithi055 ceph-mon[97033]: pgmap v9058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:19:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:09 smithi055 ceph-mon[97033]: osdmap e647: 12 total, 11 up, 11 in 2022-09-08T02:19:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:09 smithi059 ceph-mon[97450]: pgmap v9058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:19:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:09 smithi059 ceph-mon[97450]: osdmap e647: 12 total, 11 up, 11 in 2022-09-08T02:19:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:10.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:10.636096+0000) 2022-09-08T02:19:10.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:19:10.455+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:19:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:10.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:10.580249+0000) 2022-09-08T02:19:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:11.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:11.636340+0000) 2022-09-08T02:19:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:11.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:11.580515+0000) 2022-09-08T02:19:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:11 smithi055 ceph-mon[97033]: pgmap v9060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:19:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:11 smithi059 ceph-mon[97450]: pgmap v9060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:19:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:12.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:12.636576+0000) 2022-09-08T02:19:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:12.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:12.580777+0000) 2022-09-08T02:19:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:13.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:13.636767+0000) 2022-09-08T02:19:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:13.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:13.581003+0000) 2022-09-08T02:19:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:13 smithi055 ceph-mon[97033]: pgmap v9061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:19:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:13 smithi055 ceph-mon[97033]: osdmap e648: 12 total, 11 up, 11 in 2022-09-08T02:19:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:13 smithi059 ceph-mon[97450]: pgmap v9061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:19:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:13 smithi059 ceph-mon[97450]: osdmap e648: 12 total, 11 up, 11 in 2022-09-08T02:19:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:14.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:14.636969+0000) 2022-09-08T02:19:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:14.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:14.581209+0000) 2022-09-08T02:19:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:15.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:15.637186+0000) 2022-09-08T02:19:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:15.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:15.581491+0000) 2022-09-08T02:19:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:15 smithi055 ceph-mon[97033]: pgmap v9063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:19:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:15 smithi059 ceph-mon[97450]: pgmap v9063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:19:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:16.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:16.637424+0000) 2022-09-08T02:19:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:16.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:16.581762+0000) 2022-09-08T02:19:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:17.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:17.637727+0000) 2022-09-08T02:19:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:17.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:17.582022+0000) 2022-09-08T02:19:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:17 smithi055 ceph-mon[97033]: pgmap v9064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:19:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:17 smithi059 ceph-mon[97450]: pgmap v9064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:19:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:18.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:18.637959+0000) 2022-09-08T02:19:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:18.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:18.582203+0000) 2022-09-08T02:19:19.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:19.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:19.638154+0000) 2022-09-08T02:19:19.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:19.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:19.582452+0000) 2022-09-08T02:19:19.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:19 smithi055 ceph-mon[97033]: pgmap v9065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T02:19:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:19 smithi059 ceph-mon[97450]: pgmap v9065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T02:19:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:20.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:20.638301+0000) 2022-09-08T02:19:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:20.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:20.582777+0000) 2022-09-08T02:19:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:21.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:21.638502+0000) 2022-09-08T02:19:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:21.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:21.583042+0000) 2022-09-08T02:19:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:21 smithi055 ceph-mon[97033]: pgmap v9066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:19:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:21 smithi059 ceph-mon[97450]: pgmap v9066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:19:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:22.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:22.583266+0000) 2022-09-08T02:19:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:22.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:22.638715+0000) 2022-09-08T02:19:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:23.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:23.583499+0000) 2022-09-08T02:19:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:23.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:23.638933+0000) 2022-09-08T02:19:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:23 smithi055 ceph-mon[97033]: pgmap v9067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:19:24.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:23 smithi059 ceph-mon[97450]: pgmap v9067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:19:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:24.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:24.583693+0000) 2022-09-08T02:19:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:24.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:24.639157+0000) 2022-09-08T02:19:25.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:19:25.457+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:19:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:25.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:25.583822+0000) 2022-09-08T02:19:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:25.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:25.639368+0000) 2022-09-08T02:19:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:25 smithi055 ceph-mon[97033]: pgmap v9068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T02:19:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:25 smithi059 ceph-mon[97450]: pgmap v9068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T02:19:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:26.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:26.639582+0000) 2022-09-08T02:19:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:26.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:26.584031+0000) 2022-09-08T02:19:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:27.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:27.639815+0000) 2022-09-08T02:19:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:27.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:27.584281+0000) 2022-09-08T02:19:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:27 smithi055 ceph-mon[97033]: pgmap v9069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:19:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:27 smithi059 ceph-mon[97450]: pgmap v9069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:19:28.909 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:28.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:28.639947+0000) 2022-09-08T02:19:28.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:28.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:28.584472+0000) 2022-09-08T02:19:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:29.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:29.584742+0000) 2022-09-08T02:19:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:29.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:29.640214+0000) 2022-09-08T02:19:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:29 smithi055 ceph-mon[97033]: pgmap v9070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:29 smithi059 ceph-mon[97450]: pgmap v9070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:30.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:30.640439+0000) 2022-09-08T02:19:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:30.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:30.584980+0000) 2022-09-08T02:19:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:31.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:31.640718+0000) 2022-09-08T02:19:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:31.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:31.585183+0000) 2022-09-08T02:19:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:31 smithi055 ceph-mon[97033]: pgmap v9071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:19:32.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:31 smithi059 ceph-mon[97450]: pgmap v9071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:19:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:32.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:32.640903+0000) 2022-09-08T02:19:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:32.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:32.585397+0000) 2022-09-08T02:19:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:33.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:33.641201+0000) 2022-09-08T02:19:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:33.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:33.585623+0000) 2022-09-08T02:19:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:33 smithi055 ceph-mon[97033]: pgmap v9072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:34.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:33 smithi059 ceph-mon[97450]: pgmap v9072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:34.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:34.641466+0000) 2022-09-08T02:19:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:34.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:34.585869+0000) 2022-09-08T02:19:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:35.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:35.641624+0000) 2022-09-08T02:19:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:35.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:35.586040+0000) 2022-09-08T02:19:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:35 smithi055 ceph-mon[97033]: pgmap v9073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:19:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:19:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:19:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:19:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:19:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:19:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:35 smithi059 ceph-mon[97450]: pgmap v9073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:19:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:19:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:19:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:19:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:19:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:19:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:36.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:36.641868+0000) 2022-09-08T02:19:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:36.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:36.586280+0000) 2022-09-08T02:19:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:36 smithi055 ceph-mon[97033]: pgmap v9074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T02:19:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:36 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T02:19:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:36 smithi059 ceph-mon[97450]: pgmap v9074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T02:19:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:36 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T02:19:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:37.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:37.586545+0000) 2022-09-08T02:19:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:37.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:37.642062+0000) 2022-09-08T02:19:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:38.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:38.642259+0000) 2022-09-08T02:19:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:38.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:38.586688+0000) 2022-09-08T02:19:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:38 smithi055 ceph-mon[97033]: pgmap v9075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:19:39.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:38 smithi059 ceph-mon[97450]: pgmap v9075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:19:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:39.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:39.642451+0000) 2022-09-08T02:19:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:39.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:39.586870+0000) 2022-09-08T02:19:40.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:19:40.470+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:19:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:40.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:40.642711+0000) 2022-09-08T02:19:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:40.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:40.587063+0000) 2022-09-08T02:19:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:40 smithi055 ceph-mon[97033]: pgmap v9076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:19:41.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:40 smithi059 ceph-mon[97450]: pgmap v9076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:19:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:41.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:41.642944+0000) 2022-09-08T02:19:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:41.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:41.587274+0000) 2022-09-08T02:19:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:42.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:42.643174+0000) 2022-09-08T02:19:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:42.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:42.587515+0000) 2022-09-08T02:19:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:42 smithi055 ceph-mon[97033]: pgmap v9077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:19:43.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:42 smithi059 ceph-mon[97450]: pgmap v9077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:19:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:43.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:43.643391+0000) 2022-09-08T02:19:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:43.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:43.587753+0000) 2022-09-08T02:19:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:44.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:44.643631+0000) 2022-09-08T02:19:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:44.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:44.587907+0000) 2022-09-08T02:19:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:44 smithi055 ceph-mon[97033]: pgmap v9078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:19:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:44 smithi059 ceph-mon[97450]: pgmap v9078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:19:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:45.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:45.643864+0000) 2022-09-08T02:19:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:45.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:45.588131+0000) 2022-09-08T02:19:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:46.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:46.644029+0000) 2022-09-08T02:19:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:46.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:46.588370+0000) 2022-09-08T02:19:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:46 smithi055 ceph-mon[97033]: pgmap v9079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T02:19:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:46 smithi059 ceph-mon[97450]: pgmap v9079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T02:19:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:47.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:47.644150+0000) 2022-09-08T02:19:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:47.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:47.588670+0000) 2022-09-08T02:19:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:48.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:48.644323+0000) 2022-09-08T02:19:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:48.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:48.588903+0000) 2022-09-08T02:19:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:48 smithi055 ceph-mon[97033]: pgmap v9080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:48 smithi059 ceph-mon[97450]: pgmap v9080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:49.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:49.644569+0000) 2022-09-08T02:19:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:49.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:49.589144+0000) 2022-09-08T02:19:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:50.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:50.644823+0000) 2022-09-08T02:19:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:50.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:50.589394+0000) 2022-09-08T02:19:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:50 smithi055 ceph-mon[97033]: pgmap v9081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:51.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:50 smithi059 ceph-mon[97450]: pgmap v9081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:51.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:51.645142+0000) 2022-09-08T02:19:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:51.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:51.589611+0000) 2022-09-08T02:19:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:52.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:52.645363+0000) 2022-09-08T02:19:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:52.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:52.589792+0000) 2022-09-08T02:19:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:52 smithi055 ceph-mon[97033]: pgmap v9082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:19:53.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:52 smithi059 ceph-mon[97450]: pgmap v9082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:19:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:53.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:53.645674+0000) 2022-09-08T02:19:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:53.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:53.590056+0000) 2022-09-08T02:19:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:54.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:54.645909+0000) 2022-09-08T02:19:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:54.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:54.590327+0000) 2022-09-08T02:19:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:54 smithi055 ceph-mon[97033]: pgmap v9083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:54 smithi059 ceph-mon[97450]: pgmap v9083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:55.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:19:55.481+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:19:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:55.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:55.646146+0000) 2022-09-08T02:19:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:55.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:55.590595+0000) 2022-09-08T02:19:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:56.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:56.646305+0000) 2022-09-08T02:19:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:56.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:56.590813+0000) 2022-09-08T02:19:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:56 smithi055 ceph-mon[97033]: pgmap v9084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:19:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:56 smithi059 ceph-mon[97450]: pgmap v9084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:19:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:57.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:57.646530+0000) 2022-09-08T02:19:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:57.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:57.591100+0000) 2022-09-08T02:19:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:58.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:58.646699+0000) 2022-09-08T02:19:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:58.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:58.591271+0000) 2022-09-08T02:19:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:19:58 smithi055 ceph-mon[97033]: pgmap v9085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:19:58 smithi059 ceph-mon[97450]: pgmap v9085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:19:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:19:59.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:59.646904+0000) 2022-09-08T02:19:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:19:59.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:19:59.591499+0000) 2022-09-08T02:20:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:00.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:00.647106+0000) 2022-09-08T02:20:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:00.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:00.591699+0000) 2022-09-08T02:20:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:00 smithi055 ceph-mon[97033]: pgmap v9086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T02:20:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T02:20:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T02:20:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:00 smithi059 ceph-mon[97450]: pgmap v9086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T02:20:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T02:20:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T02:20:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:01.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:01.591897+0000) 2022-09-08T02:20:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:01.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:01.647285+0000) 2022-09-08T02:20:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:02.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:02.647504+0000) 2022-09-08T02:20:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:02.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:02.592116+0000) 2022-09-08T02:20:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:02 smithi055 ceph-mon[97033]: pgmap v9087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:20:03.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:02 smithi059 ceph-mon[97450]: pgmap v9087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:20:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:03.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:03.647692+0000) 2022-09-08T02:20:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:03.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:03.592287+0000) 2022-09-08T02:20:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:04.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:04.647910+0000) 2022-09-08T02:20:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:04.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:04.592492+0000) 2022-09-08T02:20:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:04 smithi055 ceph-mon[97033]: pgmap v9088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:04 smithi059 ceph-mon[97450]: pgmap v9088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:05.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:05.648161+0000) 2022-09-08T02:20:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:05.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:05.592742+0000) 2022-09-08T02:20:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:20:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:20:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:20:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:20:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:06.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:06.648345+0000) 2022-09-08T02:20:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:06.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:06.593005+0000) 2022-09-08T02:20:06.919 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:06 smithi059 ceph-mon[97450]: pgmap v9089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:20:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:06 smithi055 ceph-mon[97033]: pgmap v9089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:20:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:07.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:07.648539+0000) 2022-09-08T02:20:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:07.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:07.593264+0000) 2022-09-08T02:20:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:08.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:08.648788+0000) 2022-09-08T02:20:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:08.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:08.593482+0000) 2022-09-08T02:20:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:08 smithi055 ceph-mon[97033]: pgmap v9090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:20:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:08 smithi059 ceph-mon[97450]: pgmap v9090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:20:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:09.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:09.648961+0000) 2022-09-08T02:20:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:09.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:09.593760+0000) 2022-09-08T02:20:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:09 smithi055 ceph-mon[97033]: osdmap e649: 12 total, 11 up, 11 in 2022-09-08T02:20:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:09 smithi059 ceph-mon[97450]: osdmap e649: 12 total, 11 up, 11 in 2022-09-08T02:20:10.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:20:10.496+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:20:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:10.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:10.649187+0000) 2022-09-08T02:20:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:10.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:10.593894+0000) 2022-09-08T02:20:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:10 smithi055 ceph-mon[97033]: pgmap v9092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:20:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:10 smithi059 ceph-mon[97450]: pgmap v9092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:20:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:11.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:11.649468+0000) 2022-09-08T02:20:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:11.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:11.594109+0000) 2022-09-08T02:20:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:12.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:12.649744+0000) 2022-09-08T02:20:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:12.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:12.594340+0000) 2022-09-08T02:20:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:12 smithi055 ceph-mon[97033]: pgmap v9093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:20:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:12 smithi055 ceph-mon[97033]: osdmap e650: 12 total, 11 up, 11 in 2022-09-08T02:20:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:12 smithi059 ceph-mon[97450]: pgmap v9093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:20:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:12 smithi059 ceph-mon[97450]: osdmap e650: 12 total, 11 up, 11 in 2022-09-08T02:20:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:13.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:13.594529+0000) 2022-09-08T02:20:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:13.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:13.649962+0000) 2022-09-08T02:20:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:14.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:14.650172+0000) 2022-09-08T02:20:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:14.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:14.594756+0000) 2022-09-08T02:20:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:14 smithi055 ceph-mon[97033]: pgmap v9095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:20:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:14 smithi059 ceph-mon[97450]: pgmap v9095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:20:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:15.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:15.650403+0000) 2022-09-08T02:20:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:15.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:15.594941+0000) 2022-09-08T02:20:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:16.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:16.650602+0000) 2022-09-08T02:20:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:16.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:16.595097+0000) 2022-09-08T02:20:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:16 smithi055 ceph-mon[97033]: pgmap v9096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:20:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:16 smithi059 ceph-mon[97450]: pgmap v9096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:20:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:17.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:17.650792+0000) 2022-09-08T02:20:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:17.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:17.595272+0000) 2022-09-08T02:20:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:18.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:18.651017+0000) 2022-09-08T02:20:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:18.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:18.595481+0000) 2022-09-08T02:20:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:18 smithi055 ceph-mon[97033]: pgmap v9097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:20:19.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:18 smithi059 ceph-mon[97450]: pgmap v9097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:20:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:19.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:19.651220+0000) 2022-09-08T02:20:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:19.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:19.595750+0000) 2022-09-08T02:20:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:20.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:20.651476+0000) 2022-09-08T02:20:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:20.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:20.595917+0000) 2022-09-08T02:20:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:20 smithi055 ceph-mon[97033]: pgmap v9098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:20:21.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:20 smithi059 ceph-mon[97450]: pgmap v9098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:20:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:21.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:21.596033+0000) 2022-09-08T02:20:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:21.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:21.651670+0000) 2022-09-08T02:20:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:22.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:22.651835+0000) 2022-09-08T02:20:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:22.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:22.596262+0000) 2022-09-08T02:20:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:22 smithi055 ceph-mon[97033]: pgmap v9099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:20:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:22 smithi059 ceph-mon[97450]: pgmap v9099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:20:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:23.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:23.652036+0000) 2022-09-08T02:20:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:23.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:23.596544+0000) 2022-09-08T02:20:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:24.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:24.596823+0000) 2022-09-08T02:20:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:24.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:24.652273+0000) 2022-09-08T02:20:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:24 smithi055 ceph-mon[97033]: pgmap v9100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T02:20:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:24 smithi059 ceph-mon[97450]: pgmap v9100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T02:20:25.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:20:25.510+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:20:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:25.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:25.652497+0000) 2022-09-08T02:20:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:25.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:25.597092+0000) 2022-09-08T02:20:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:26.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:26.652749+0000) 2022-09-08T02:20:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:26.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:26.597312+0000) 2022-09-08T02:20:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:26 smithi055 ceph-mon[97033]: pgmap v9101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:20:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:26 smithi059 ceph-mon[97450]: pgmap v9101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:20:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:27.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:27.652957+0000) 2022-09-08T02:20:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:27.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:27.597453+0000) 2022-09-08T02:20:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:28.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:28.653146+0000) 2022-09-08T02:20:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:28.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:28.597662+0000) 2022-09-08T02:20:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:28 smithi055 ceph-mon[97033]: pgmap v9102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:28 smithi059 ceph-mon[97450]: pgmap v9102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:29.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:29.653374+0000) 2022-09-08T02:20:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:29.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:29.597904+0000) 2022-09-08T02:20:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:30.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:30.653591+0000) 2022-09-08T02:20:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:30.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:30.598179+0000) 2022-09-08T02:20:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:30 smithi055 ceph-mon[97033]: pgmap v9103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:30 smithi059 ceph-mon[97450]: pgmap v9103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:31.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:31.653856+0000) 2022-09-08T02:20:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:31.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:31.598417+0000) 2022-09-08T02:20:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:32.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:32.654076+0000) 2022-09-08T02:20:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:32.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:32.598665+0000) 2022-09-08T02:20:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:32 smithi055 ceph-mon[97033]: pgmap v9104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:20:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:32 smithi059 ceph-mon[97450]: pgmap v9104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:20:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:33.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:33.598801+0000) 2022-09-08T02:20:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:33.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:33.654285+0000) 2022-09-08T02:20:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:34.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:34.654515+0000) 2022-09-08T02:20:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:34.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:34.599005+0000) 2022-09-08T02:20:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:34 smithi055 ceph-mon[97033]: pgmap v9105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:34 smithi059 ceph-mon[97450]: pgmap v9105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:35.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:35.654739+0000) 2022-09-08T02:20:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:35.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:35.599227+0000) 2022-09-08T02:20:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:36.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:36.654970+0000) 2022-09-08T02:20:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:36.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:36.599443+0000) 2022-09-08T02:20:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:36 smithi055 ceph-mon[97033]: pgmap v9106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:20:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:20:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:20:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:20:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:20:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:36 smithi059 ceph-mon[97450]: pgmap v9106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:20:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:20:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:20:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:20:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:20:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:37.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:37.655146+0000) 2022-09-08T02:20:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:37.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:37.599718+0000) 2022-09-08T02:20:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:38.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:38.599851+0000) 2022-09-08T02:20:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:38.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:38.655318+0000) 2022-09-08T02:20:38.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:38 smithi055 ceph-mon[97033]: pgmap v9107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:38 smithi059 ceph-mon[97450]: pgmap v9107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:39.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:39.600089+0000) 2022-09-08T02:20:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:39.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:39.655563+0000) 2022-09-08T02:20:40.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:20:40.519+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:20:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:40.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:40.600297+0000) 2022-09-08T02:20:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:40.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:40.655715+0000) 2022-09-08T02:20:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:40 smithi055 ceph-mon[97033]: pgmap v9108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:41.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:40 smithi059 ceph-mon[97450]: pgmap v9108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:41.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:41.655844+0000) 2022-09-08T02:20:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:41.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:41.600540+0000) 2022-09-08T02:20:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:42.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:42.600722+0000) 2022-09-08T02:20:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:42.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:42.655998+0000) 2022-09-08T02:20:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:42 smithi055 ceph-mon[97033]: pgmap v9109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:20:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:42 smithi059 ceph-mon[97450]: pgmap v9109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:20:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:43.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:43.656250+0000) 2022-09-08T02:20:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:43.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:43.600945+0000) 2022-09-08T02:20:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:44.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:44.656512+0000) 2022-09-08T02:20:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:44.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:44.601169+0000) 2022-09-08T02:20:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:44 smithi055 ceph-mon[97033]: pgmap v9110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:44 smithi059 ceph-mon[97450]: pgmap v9110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:45.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:45.656729+0000) 2022-09-08T02:20:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:45.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:45.601401+0000) 2022-09-08T02:20:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:46.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:46.656875+0000) 2022-09-08T02:20:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:46.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:46.601534+0000) 2022-09-08T02:20:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:46 smithi055 ceph-mon[97033]: pgmap v9111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:20:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:46 smithi059 ceph-mon[97450]: pgmap v9111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:20:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:47.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:47.657137+0000) 2022-09-08T02:20:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:47.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:47.601750+0000) 2022-09-08T02:20:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:48.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:48.657379+0000) 2022-09-08T02:20:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:48.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:48.601984+0000) 2022-09-08T02:20:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:48 smithi055 ceph-mon[97033]: pgmap v9112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:49.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:48 smithi059 ceph-mon[97450]: pgmap v9112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:49.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:49.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:49.602246+0000) 2022-09-08T02:20:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:49.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:49.657554+0000) 2022-09-08T02:20:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:50.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:50.602461+0000) 2022-09-08T02:20:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:50.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:50.657773+0000) 2022-09-08T02:20:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:50 smithi055 ceph-mon[97033]: pgmap v9113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:50 smithi059 ceph-mon[97450]: pgmap v9113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:51.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:51.657930+0000) 2022-09-08T02:20:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:51.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:51.602726+0000) 2022-09-08T02:20:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:52.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:52.602902+0000) 2022-09-08T02:20:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:52.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:52.658103+0000) 2022-09-08T02:20:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:52 smithi055 ceph-mon[97033]: pgmap v9114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:20:53.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:52 smithi059 ceph-mon[97450]: pgmap v9114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:20:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:53.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:53.658374+0000) 2022-09-08T02:20:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:53.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:53.603180+0000) 2022-09-08T02:20:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:54.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:54.658600+0000) 2022-09-08T02:20:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:54.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:54.603364+0000) 2022-09-08T02:20:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:54 smithi055 ceph-mon[97033]: pgmap v9115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:54 smithi059 ceph-mon[97450]: pgmap v9115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:55.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:20:55.526+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:20:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:55.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:55.658824+0000) 2022-09-08T02:20:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:55.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:55.603596+0000) 2022-09-08T02:20:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:56.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:56.658985+0000) 2022-09-08T02:20:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:56.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:56.603761+0000) 2022-09-08T02:20:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:56 smithi055 ceph-mon[97033]: pgmap v9116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:20:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:56 smithi059 ceph-mon[97450]: pgmap v9116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:20:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:57.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:57.659137+0000) 2022-09-08T02:20:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:57.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:57.603899+0000) 2022-09-08T02:20:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:58.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:58.659290+0000) 2022-09-08T02:20:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:58.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:58.604110+0000) 2022-09-08T02:20:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:20:58 smithi055 ceph-mon[97033]: pgmap v9117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:59.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:20:58 smithi059 ceph-mon[97450]: pgmap v9117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:20:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:20:59.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:59.659509+0000) 2022-09-08T02:20:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:20:59.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:20:59.604332+0000) 2022-09-08T02:21:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:00.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:00.659677+0000) 2022-09-08T02:21:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:00.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:00.604450+0000) 2022-09-08T02:21:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:00 smithi055 ceph-mon[97033]: pgmap v9118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:00 smithi059 ceph-mon[97450]: pgmap v9118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:01.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:01.659897+0000) 2022-09-08T02:21:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:01.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:01.604691+0000) 2022-09-08T02:21:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:02.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:02.660068+0000) 2022-09-08T02:21:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:02.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:02.604808+0000) 2022-09-08T02:21:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:02 smithi055 ceph-mon[97033]: pgmap v9119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:21:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:02 smithi059 ceph-mon[97450]: pgmap v9119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:21:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:03.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:03.660296+0000) 2022-09-08T02:21:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:03.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:03.604989+0000) 2022-09-08T02:21:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:04.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:04.660527+0000) 2022-09-08T02:21:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:04.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:04.605219+0000) 2022-09-08T02:21:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:04 smithi055 ceph-mon[97033]: pgmap v9120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:05.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:04 smithi059 ceph-mon[97450]: pgmap v9120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:05.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:05.605496+0000) 2022-09-08T02:21:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:05.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:05.660749+0000) 2022-09-08T02:21:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:21:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:21:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:21:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:21:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:06.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:06.660960+0000) 2022-09-08T02:21:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:06.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:06.605744+0000) 2022-09-08T02:21:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:06 smithi055 ceph-mon[97033]: pgmap v9121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:21:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:06 smithi059 ceph-mon[97450]: pgmap v9121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:21:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:07.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:07.661179+0000) 2022-09-08T02:21:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:07.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:07.606024+0000) 2022-09-08T02:21:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:08.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:08.606282+0000) 2022-09-08T02:21:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:08.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:08.661316+0000) 2022-09-08T02:21:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:08 smithi055 ceph-mon[97033]: pgmap v9122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:21:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:08 smithi059 ceph-mon[97450]: pgmap v9122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:21:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:09.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:09.661617+0000) 2022-09-08T02:21:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:09.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:09.606527+0000) 2022-09-08T02:21:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:09 smithi055 ceph-mon[97033]: osdmap e651: 12 total, 11 up, 11 in 2022-09-08T02:21:10.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:09 smithi059 ceph-mon[97450]: osdmap e651: 12 total, 11 up, 11 in 2022-09-08T02:21:10.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:21:10.528+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:21:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:10.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:10.661786+0000) 2022-09-08T02:21:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:10.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:10.606786+0000) 2022-09-08T02:21:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:10 smithi055 ceph-mon[97033]: pgmap v9124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:21:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:10 smithi059 ceph-mon[97450]: pgmap v9124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:21:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:11.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:11.661959+0000) 2022-09-08T02:21:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:11.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:11.607034+0000) 2022-09-08T02:21:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:12.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:12.662208+0000) 2022-09-08T02:21:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:12.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:12.607256+0000) 2022-09-08T02:21:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:12 smithi055 ceph-mon[97033]: pgmap v9125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:21:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:12 smithi055 ceph-mon[97033]: osdmap e652: 12 total, 11 up, 11 in 2022-09-08T02:21:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:12 smithi059 ceph-mon[97450]: pgmap v9125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:21:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:12 smithi059 ceph-mon[97450]: osdmap e652: 12 total, 11 up, 11 in 2022-09-08T02:21:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:13.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:13.662494+0000) 2022-09-08T02:21:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:13.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:13.607520+0000) 2022-09-08T02:21:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:14.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:14.662773+0000) 2022-09-08T02:21:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:14.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:14.607743+0000) 2022-09-08T02:21:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:14 smithi055 ceph-mon[97033]: pgmap v9127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:21:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:14 smithi059 ceph-mon[97450]: pgmap v9127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:21:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:15.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:15.663019+0000) 2022-09-08T02:21:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:15.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:15.607968+0000) 2022-09-08T02:21:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:16.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:16.663178+0000) 2022-09-08T02:21:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:16.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:16.608200+0000) 2022-09-08T02:21:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:16 smithi055 ceph-mon[97033]: pgmap v9128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:21:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:16 smithi059 ceph-mon[97450]: pgmap v9128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:21:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:17.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:17.663419+0000) 2022-09-08T02:21:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:17.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:17.608436+0000) 2022-09-08T02:21:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:18.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:18.663738+0000) 2022-09-08T02:21:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:18.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:18.608735+0000) 2022-09-08T02:21:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:18 smithi055 ceph-mon[97033]: pgmap v9129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:21:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:18 smithi059 ceph-mon[97450]: pgmap v9129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:21:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:19.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:19.663957+0000) 2022-09-08T02:21:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:19.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:19.608970+0000) 2022-09-08T02:21:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:20.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:20.664152+0000) 2022-09-08T02:21:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:20.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:20.609232+0000) 2022-09-08T02:21:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:20 smithi055 ceph-mon[97033]: pgmap v9130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:21:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:20 smithi059 ceph-mon[97450]: pgmap v9130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:21:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:21.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:21.664376+0000) 2022-09-08T02:21:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:21.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:21.609472+0000) 2022-09-08T02:21:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:22.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:22.664523+0000) 2022-09-08T02:21:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:22.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:22.609711+0000) 2022-09-08T02:21:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:22 smithi055 ceph-mon[97033]: pgmap v9131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:21:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:22 smithi059 ceph-mon[97450]: pgmap v9131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:21:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:23.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:23.664731+0000) 2022-09-08T02:21:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:23.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:23.609964+0000) 2022-09-08T02:21:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:24.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:24.664962+0000) 2022-09-08T02:21:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:24.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:24.610129+0000) 2022-09-08T02:21:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:24 smithi055 ceph-mon[97033]: pgmap v9132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T02:21:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:24 smithi059 ceph-mon[97450]: pgmap v9132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T02:21:25.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:21:25.544+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:21:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:25.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:25.665210+0000) 2022-09-08T02:21:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:25.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:25.610332+0000) 2022-09-08T02:21:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:26.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:26.665451+0000) 2022-09-08T02:21:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:26.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:26.610565+0000) 2022-09-08T02:21:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:26 smithi055 ceph-mon[97033]: pgmap v9133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:21:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:26 smithi059 ceph-mon[97450]: pgmap v9133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:21:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:27.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:27.665616+0000) 2022-09-08T02:21:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:27.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:27.610806+0000) 2022-09-08T02:21:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:28.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:28.665869+0000) 2022-09-08T02:21:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:28.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:28.611021+0000) 2022-09-08T02:21:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:28 smithi055 ceph-mon[97033]: pgmap v9134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:28 smithi059 ceph-mon[97450]: pgmap v9134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:29.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:29.666098+0000) 2022-09-08T02:21:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:29.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:29.611248+0000) 2022-09-08T02:21:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:30.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:30.666315+0000) 2022-09-08T02:21:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:30.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:30.611513+0000) 2022-09-08T02:21:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:30 smithi055 ceph-mon[97033]: pgmap v9135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:30 smithi059 ceph-mon[97450]: pgmap v9135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:31.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:31.666500+0000) 2022-09-08T02:21:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:31.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:31.611754+0000) 2022-09-08T02:21:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:32.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:32.666661+0000) 2022-09-08T02:21:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:32.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:32.612014+0000) 2022-09-08T02:21:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:32 smithi055 ceph-mon[97033]: pgmap v9136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:21:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:32 smithi059 ceph-mon[97450]: pgmap v9136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:21:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:33.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:33.666781+0000) 2022-09-08T02:21:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:33.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:33.612274+0000) 2022-09-08T02:21:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:34.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:34.667012+0000) 2022-09-08T02:21:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:34.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:34.612480+0000) 2022-09-08T02:21:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:34 smithi055 ceph-mon[97033]: pgmap v9137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:34 smithi059 ceph-mon[97450]: pgmap v9137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:35.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:35.667257+0000) 2022-09-08T02:21:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:35.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:35.612696+0000) 2022-09-08T02:21:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:36.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:36.667487+0000) 2022-09-08T02:21:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:36.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:36.612891+0000) 2022-09-08T02:21:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:36 smithi055 ceph-mon[97033]: pgmap v9138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:21:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:21:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:21:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:21:36.983 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:36 smithi059 ceph-mon[97450]: pgmap v9138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:21:36.983 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:21:36.983 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:21:36.984 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:21:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:37.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:37.613077+0000) 2022-09-08T02:21:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:37.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:37.667732+0000) 2022-09-08T02:21:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:21:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:37 smithi055 ceph-mon[97033]: pgmap v9139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:38.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:21:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:37 smithi059 ceph-mon[97450]: pgmap v9139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:38.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:38.613216+0000) 2022-09-08T02:21:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:38.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:38.667962+0000) 2022-09-08T02:21:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:39.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:39.668210+0000) 2022-09-08T02:21:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:39.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:39.613426+0000) 2022-09-08T02:21:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:40.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:40.668439+0000) 2022-09-08T02:21:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:40.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:40.613662+0000) 2022-09-08T02:21:40.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:21:40.552+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:21:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:40 smithi055 ceph-mon[97033]: pgmap v9140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:40 smithi059 ceph-mon[97450]: pgmap v9140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:41.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:41.668723+0000) 2022-09-08T02:21:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:41.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:41.613812+0000) 2022-09-08T02:21:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:42.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:42.668864+0000) 2022-09-08T02:21:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:42.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:42.614012+0000) 2022-09-08T02:21:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:42 smithi055 ceph-mon[97033]: pgmap v9141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:21:43.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:42 smithi059 ceph-mon[97450]: pgmap v9141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:21:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:43.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:43.669029+0000) 2022-09-08T02:21:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:43.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:43.614148+0000) 2022-09-08T02:21:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:44.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:44.669252+0000) 2022-09-08T02:21:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:44.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:44.614368+0000) 2022-09-08T02:21:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:44 smithi055 ceph-mon[97033]: pgmap v9142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:44 smithi059 ceph-mon[97450]: pgmap v9142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:45.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:45.669466+0000) 2022-09-08T02:21:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:45.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:45.614568+0000) 2022-09-08T02:21:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:46.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:46.669710+0000) 2022-09-08T02:21:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:46.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:46.614735+0000) 2022-09-08T02:21:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:46 smithi055 ceph-mon[97033]: pgmap v9143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:21:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:46 smithi059 ceph-mon[97450]: pgmap v9143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:21:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:47.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:47.669941+0000) 2022-09-08T02:21:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:47.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:47.614873+0000) 2022-09-08T02:21:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:48.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:48.670165+0000) 2022-09-08T02:21:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:48.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:48.615091+0000) 2022-09-08T02:21:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:48 smithi055 ceph-mon[97033]: pgmap v9144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:48 smithi059 ceph-mon[97450]: pgmap v9144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:49.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:49.670381+0000) 2022-09-08T02:21:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:49.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:49.615307+0000) 2022-09-08T02:21:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:50.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:50.670628+0000) 2022-09-08T02:21:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:50.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:50.615460+0000) 2022-09-08T02:21:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:50 smithi055 ceph-mon[97033]: pgmap v9145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:50 smithi059 ceph-mon[97450]: pgmap v9145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:51.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:51.670906+0000) 2022-09-08T02:21:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:51.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:51.615665+0000) 2022-09-08T02:21:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:52.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:52.671123+0000) 2022-09-08T02:21:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:52.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:52.615788+0000) 2022-09-08T02:21:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:52 smithi055 ceph-mon[97033]: pgmap v9146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:21:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:52 smithi059 ceph-mon[97450]: pgmap v9146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:21:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:53.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:53.671341+0000) 2022-09-08T02:21:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:53.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:53.615957+0000) 2022-09-08T02:21:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:54.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:54.671559+0000) 2022-09-08T02:21:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:54.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:54.616094+0000) 2022-09-08T02:21:54.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:54 smithi055 ceph-mon[97033]: pgmap v9147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:54 smithi059 ceph-mon[97450]: pgmap v9147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:55.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:21:55.560+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:21:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:55.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:55.671851+0000) 2022-09-08T02:21:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:55.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:55.616302+0000) 2022-09-08T02:21:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:56.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:56.672048+0000) 2022-09-08T02:21:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:56.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:56.616458+0000) 2022-09-08T02:21:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:56 smithi055 ceph-mon[97033]: pgmap v9148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:21:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:56 smithi059 ceph-mon[97450]: pgmap v9148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:21:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:57.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:57.672228+0000) 2022-09-08T02:21:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:57.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:57.616678+0000) 2022-09-08T02:21:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:58.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:58.672461+0000) 2022-09-08T02:21:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:58.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:58.616893+0000) 2022-09-08T02:21:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:21:58 smithi055 ceph-mon[97033]: pgmap v9149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:21:58 smithi059 ceph-mon[97450]: pgmap v9149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:21:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:21:59.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:59.672739+0000) 2022-09-08T02:21:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:21:59.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:21:59.617132+0000) 2022-09-08T02:22:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:00.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:00.672889+0000) 2022-09-08T02:22:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:00.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:00.617301+0000) 2022-09-08T02:22:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:00 smithi055 ceph-mon[97033]: pgmap v9150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:00 smithi059 ceph-mon[97450]: pgmap v9150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:01.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:01.673153+0000) 2022-09-08T02:22:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:01.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:01.617531+0000) 2022-09-08T02:22:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:02.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:02.673430+0000) 2022-09-08T02:22:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:02.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:02.617755+0000) 2022-09-08T02:22:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:02 smithi055 ceph-mon[97033]: pgmap v9151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:22:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:02 smithi059 ceph-mon[97450]: pgmap v9151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:22:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:03.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:03.673688+0000) 2022-09-08T02:22:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:03.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:03.617896+0000) 2022-09-08T02:22:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:04.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:04.673937+0000) 2022-09-08T02:22:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:04.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:04.618051+0000) 2022-09-08T02:22:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:04 smithi055 ceph-mon[97033]: pgmap v9152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:05.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:04 smithi059 ceph-mon[97450]: pgmap v9152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:05.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:05.674120+0000) 2022-09-08T02:22:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:05.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:05.618295+0000) 2022-09-08T02:22:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:22:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:22:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:22:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:22:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:06.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:06.674278+0000) 2022-09-08T02:22:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:06.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:06.618425+0000) 2022-09-08T02:22:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:06 smithi055 ceph-mon[97033]: pgmap v9153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:22:07.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:06 smithi059 ceph-mon[97450]: pgmap v9153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:22:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:07.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:07.674428+0000) 2022-09-08T02:22:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:07.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:07.618627+0000) 2022-09-08T02:22:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:08.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:08.674666+0000) 2022-09-08T02:22:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:08.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:08.618826+0000) 2022-09-08T02:22:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:08 smithi055 ceph-mon[97033]: pgmap v9154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:22:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:08 smithi059 ceph-mon[97450]: pgmap v9154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:22:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:09.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:09.619059+0000) 2022-09-08T02:22:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:09.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:09.674914+0000) 2022-09-08T02:22:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:09 smithi055 ceph-mon[97033]: osdmap e653: 12 total, 11 up, 11 in 2022-09-08T02:22:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:09 smithi059 ceph-mon[97450]: osdmap e653: 12 total, 11 up, 11 in 2022-09-08T02:22:10.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:22:10.573+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:22:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:10.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:10.675165+0000) 2022-09-08T02:22:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:10.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:10.619195+0000) 2022-09-08T02:22:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:10 smithi055 ceph-mon[97033]: pgmap v9156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:22:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:10 smithi059 ceph-mon[97450]: pgmap v9156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:22:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:11.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:11.675356+0000) 2022-09-08T02:22:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:11.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:11.619371+0000) 2022-09-08T02:22:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:12.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:12.675518+0000) 2022-09-08T02:22:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:12.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:12.619604+0000) 2022-09-08T02:22:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:12 smithi055 ceph-mon[97033]: pgmap v9157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:22:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:12 smithi055 ceph-mon[97033]: osdmap e654: 12 total, 11 up, 11 in 2022-09-08T02:22:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:12 smithi059 ceph-mon[97450]: pgmap v9157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:22:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:12 smithi059 ceph-mon[97450]: osdmap e654: 12 total, 11 up, 11 in 2022-09-08T02:22:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:13.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:13.675773+0000) 2022-09-08T02:22:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:13.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:13.619774+0000) 2022-09-08T02:22:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:14.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:14.675927+0000) 2022-09-08T02:22:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:14.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:14.620025+0000) 2022-09-08T02:22:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:14 smithi055 ceph-mon[97033]: pgmap v9159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:22:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:14 smithi059 ceph-mon[97450]: pgmap v9159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:22:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:15.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:15.676074+0000) 2022-09-08T02:22:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:15.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:15.620261+0000) 2022-09-08T02:22:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:16.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:16.676216+0000) 2022-09-08T02:22:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:16.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:16.620467+0000) 2022-09-08T02:22:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:16 smithi055 ceph-mon[97033]: pgmap v9160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:22:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:16 smithi059 ceph-mon[97450]: pgmap v9160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:22:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:17.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:17.676386+0000) 2022-09-08T02:22:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:17.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:17.620688+0000) 2022-09-08T02:22:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:18.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:18.676613+0000) 2022-09-08T02:22:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:18.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:18.620883+0000) 2022-09-08T02:22:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:18 smithi055 ceph-mon[97033]: pgmap v9161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:22:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:18 smithi059 ceph-mon[97450]: pgmap v9161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:22:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:19.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:19.676858+0000) 2022-09-08T02:22:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:19.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:19.621043+0000) 2022-09-08T02:22:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:20.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:20.677010+0000) 2022-09-08T02:22:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:20.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:20.621199+0000) 2022-09-08T02:22:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:20 smithi055 ceph-mon[97033]: pgmap v9162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:22:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:20 smithi059 ceph-mon[97450]: pgmap v9162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:22:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:21.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:21.677212+0000) 2022-09-08T02:22:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:21.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:21.621437+0000) 2022-09-08T02:22:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:22.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:22.677363+0000) 2022-09-08T02:22:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:22.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:22.621704+0000) 2022-09-08T02:22:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:22 smithi055 ceph-mon[97033]: pgmap v9163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:22:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:22 smithi059 ceph-mon[97450]: pgmap v9163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:22:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:23.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:23.677570+0000) 2022-09-08T02:22:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:23.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:23.621930+0000) 2022-09-08T02:22:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:24.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:24.677744+0000) 2022-09-08T02:22:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:24.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:24.622139+0000) 2022-09-08T02:22:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:24 smithi055 ceph-mon[97033]: pgmap v9164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:22:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:24 smithi059 ceph-mon[97450]: pgmap v9164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:22:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:25.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:25.677973+0000) 2022-09-08T02:22:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:25.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:25.622371+0000) 2022-09-08T02:22:25.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:22:25.589+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:22:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:26.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:26.678230+0000) 2022-09-08T02:22:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:26.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:26.622536+0000) 2022-09-08T02:22:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:26 smithi055 ceph-mon[97033]: pgmap v9165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:22:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:26 smithi059 ceph-mon[97450]: pgmap v9165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:22:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:27.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:27.678447+0000) 2022-09-08T02:22:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:27.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:27.622750+0000) 2022-09-08T02:22:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:28.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:28.678578+0000) 2022-09-08T02:22:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:28.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:28.622959+0000) 2022-09-08T02:22:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:28 smithi055 ceph-mon[97033]: pgmap v9166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:28 smithi059 ceph-mon[97450]: pgmap v9166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:29.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:29.678772+0000) 2022-09-08T02:22:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:29.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:29.623126+0000) 2022-09-08T02:22:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:30.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:30.678965+0000) 2022-09-08T02:22:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:30.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:30.623326+0000) 2022-09-08T02:22:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:30 smithi055 ceph-mon[97033]: pgmap v9167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:31.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:30 smithi059 ceph-mon[97450]: pgmap v9167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:31.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:31.623545+0000) 2022-09-08T02:22:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:31.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:31.679262+0000) 2022-09-08T02:22:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:32.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:32.623756+0000) 2022-09-08T02:22:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:32.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:32.679498+0000) 2022-09-08T02:22:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:32 smithi055 ceph-mon[97033]: pgmap v9168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:22:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:32 smithi059 ceph-mon[97450]: pgmap v9168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:22:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:33.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:33.679681+0000) 2022-09-08T02:22:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:33.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:33.623961+0000) 2022-09-08T02:22:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:34.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:34.624174+0000) 2022-09-08T02:22:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:34.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:34.679890+0000) 2022-09-08T02:22:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:34 smithi055 ceph-mon[97033]: pgmap v9169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:34 smithi059 ceph-mon[97450]: pgmap v9169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:35.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:35.680118+0000) 2022-09-08T02:22:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:35.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:35.624417+0000) 2022-09-08T02:22:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:36.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:36.624678+0000) 2022-09-08T02:22:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:36.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:36.680345+0000) 2022-09-08T02:22:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:36 smithi055 ceph-mon[97033]: pgmap v9170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:22:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:36 smithi059 ceph-mon[97450]: pgmap v9170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:22:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:37.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:37.680571+0000) 2022-09-08T02:22:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:37.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:37.624895+0000) 2022-09-08T02:22:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:22:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:22:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:22:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:22:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:22:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:22:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:22:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:22:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:38.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:38.625083+0000) 2022-09-08T02:22:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:38.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:38.680798+0000) 2022-09-08T02:22:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:38 smithi059 ceph-mon[97450]: pgmap v9171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:38 smithi055 ceph-mon[97033]: pgmap v9171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:39.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:39.681032+0000) 2022-09-08T02:22:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:39.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:39.625336+0000) 2022-09-08T02:22:40.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:22:40.590+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:22:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:40.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:40.681262+0000) 2022-09-08T02:22:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:40.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:40.625572+0000) 2022-09-08T02:22:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:40 smithi059 ceph-mon[97450]: pgmap v9172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:40 smithi055 ceph-mon[97033]: pgmap v9172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:41.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:41.681528+0000) 2022-09-08T02:22:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:41.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:41.625765+0000) 2022-09-08T02:22:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:42.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:42.681754+0000) 2022-09-08T02:22:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:42.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:42.625998+0000) 2022-09-08T02:22:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:42 smithi059 ceph-mon[97450]: pgmap v9173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:22:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:42 smithi055 ceph-mon[97033]: pgmap v9173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:22:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:43.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:43.681983+0000) 2022-09-08T02:22:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:43.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:43.626232+0000) 2022-09-08T02:22:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:44.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:44.682196+0000) 2022-09-08T02:22:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:44.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:44.626489+0000) 2022-09-08T02:22:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:44 smithi059 ceph-mon[97450]: pgmap v9174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:44 smithi055 ceph-mon[97033]: pgmap v9174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:45.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:45.682409+0000) 2022-09-08T02:22:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:45.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:45.626732+0000) 2022-09-08T02:22:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:46.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:46.682625+0000) 2022-09-08T02:22:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:46.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:46.626969+0000) 2022-09-08T02:22:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:46 smithi059 ceph-mon[97450]: pgmap v9175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:22:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:46 smithi055 ceph-mon[97033]: pgmap v9175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:22:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:47.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:47.682816+0000) 2022-09-08T02:22:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:47.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:47.627227+0000) 2022-09-08T02:22:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:48.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:48.683053+0000) 2022-09-08T02:22:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:48.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:48.627472+0000) 2022-09-08T02:22:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:48 smithi055 ceph-mon[97033]: pgmap v9176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:49.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:48 smithi059 ceph-mon[97450]: pgmap v9176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:49.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:49.683294+0000) 2022-09-08T02:22:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:49.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:49.627753+0000) 2022-09-08T02:22:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:50.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:50.683534+0000) 2022-09-08T02:22:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:50.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:50.627992+0000) 2022-09-08T02:22:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:50 smithi055 ceph-mon[97033]: pgmap v9177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:50 smithi059 ceph-mon[97450]: pgmap v9177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:51.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:51.683788+0000) 2022-09-08T02:22:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:51.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:51.628298+0000) 2022-09-08T02:22:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:52.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:52.683932+0000) 2022-09-08T02:22:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:52.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:52.628541+0000) 2022-09-08T02:22:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:52 smithi055 ceph-mon[97033]: pgmap v9178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:22:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:52 smithi059 ceph-mon[97450]: pgmap v9178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:22:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:53.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:53.684112+0000) 2022-09-08T02:22:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:53.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:53.628780+0000) 2022-09-08T02:22:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:54.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:54.684291+0000) 2022-09-08T02:22:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:54.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:54.628998+0000) 2022-09-08T02:22:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:54 smithi059 ceph-mon[97450]: pgmap v9179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:54 smithi055 ceph-mon[97033]: pgmap v9179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:55.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:22:55.603+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:22:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:55.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:55.684485+0000) 2022-09-08T02:22:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:55.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:55.629218+0000) 2022-09-08T02:22:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:56.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:56.684694+0000) 2022-09-08T02:22:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:56.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:56.629423+0000) 2022-09-08T02:22:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:56 smithi059 ceph-mon[97450]: pgmap v9180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:22:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:56 smithi055 ceph-mon[97033]: pgmap v9180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:22:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:57.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:57.684920+0000) 2022-09-08T02:22:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:57.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:57.629689+0000) 2022-09-08T02:22:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:58.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:58.629932+0000) 2022-09-08T02:22:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:58.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:58.685096+0000) 2022-09-08T02:22:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:22:58 smithi055 ceph-mon[97033]: pgmap v9181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:22:58 smithi059 ceph-mon[97450]: pgmap v9181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:22:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:22:59.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:59.685241+0000) 2022-09-08T02:22:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:22:59.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:22:59.630175+0000) 2022-09-08T02:23:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:00.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:00.685395+0000) 2022-09-08T02:23:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:00.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:00.630419+0000) 2022-09-08T02:23:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:00 smithi055 ceph-mon[97033]: pgmap v9182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:00 smithi059 ceph-mon[97450]: pgmap v9182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:01.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:01.685665+0000) 2022-09-08T02:23:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:01.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:01.630710+0000) 2022-09-08T02:23:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:02.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:02.685862+0000) 2022-09-08T02:23:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:02.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:02.630912+0000) 2022-09-08T02:23:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:02 smithi059 ceph-mon[97450]: pgmap v9183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:23:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:02 smithi055 ceph-mon[97033]: pgmap v9183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:23:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:03.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:03.686034+0000) 2022-09-08T02:23:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:03.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:03.631084+0000) 2022-09-08T02:23:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:04.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:04.686262+0000) 2022-09-08T02:23:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:04.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:04.631301+0000) 2022-09-08T02:23:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:04 smithi059 ceph-mon[97450]: pgmap v9184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:04 smithi055 ceph-mon[97033]: pgmap v9184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:05.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:05.686509+0000) 2022-09-08T02:23:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:05.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:05.631521+0000) 2022-09-08T02:23:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:23:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:23:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:23:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:23:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:06.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:06.631711+0000) 2022-09-08T02:23:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:06.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:06.686738+0000) 2022-09-08T02:23:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:06 smithi059 ceph-mon[97450]: pgmap v9185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:23:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:06 smithi055 ceph-mon[97033]: pgmap v9185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:23:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:07.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:07.686919+0000) 2022-09-08T02:23:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:07.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:07.631942+0000) 2022-09-08T02:23:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:08.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:08.687148+0000) 2022-09-08T02:23:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:08.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:08.632189+0000) 2022-09-08T02:23:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:08 smithi055 ceph-mon[97033]: pgmap v9186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:23:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:08 smithi059 ceph-mon[97450]: pgmap v9186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:23:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:09.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:09.687392+0000) 2022-09-08T02:23:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:09.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:09.632429+0000) 2022-09-08T02:23:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:09 smithi055 ceph-mon[97033]: osdmap e655: 12 total, 11 up, 11 in 2022-09-08T02:23:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:09 smithi059 ceph-mon[97450]: osdmap e655: 12 total, 11 up, 11 in 2022-09-08T02:23:10.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:23:10.605+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:23:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:10.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:10.687616+0000) 2022-09-08T02:23:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:10.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:10.632677+0000) 2022-09-08T02:23:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:10 smithi055 ceph-mon[97033]: pgmap v9188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:23:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:10 smithi059 ceph-mon[97450]: pgmap v9188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:23:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:11.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:11.687928+0000) 2022-09-08T02:23:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:11.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:11.632853+0000) 2022-09-08T02:23:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:12.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:12.688112+0000) 2022-09-08T02:23:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:12.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:12.632976+0000) 2022-09-08T02:23:13.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:12 smithi059 ceph-mon[97450]: pgmap v9189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:23:13.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:12 smithi059 ceph-mon[97450]: osdmap e656: 12 total, 11 up, 11 in 2022-09-08T02:23:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:12 smithi055 ceph-mon[97033]: pgmap v9189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:23:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:12 smithi055 ceph-mon[97033]: osdmap e656: 12 total, 11 up, 11 in 2022-09-08T02:23:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:13.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:13.688334+0000) 2022-09-08T02:23:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:13.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:13.633231+0000) 2022-09-08T02:23:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:14.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:14.688588+0000) 2022-09-08T02:23:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:14.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:14.633430+0000) 2022-09-08T02:23:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:14 smithi059 ceph-mon[97450]: pgmap v9191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:23:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:14 smithi055 ceph-mon[97033]: pgmap v9191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:23:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:15.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:15.688878+0000) 2022-09-08T02:23:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:15.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:15.633687+0000) 2022-09-08T02:23:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:16.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:16.689127+0000) 2022-09-08T02:23:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:16.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:16.633908+0000) 2022-09-08T02:23:17.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:16 smithi059 ceph-mon[97450]: pgmap v9192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:23:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:16 smithi055 ceph-mon[97033]: pgmap v9192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:23:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:17.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:17.689312+0000) 2022-09-08T02:23:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:17.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:17.634145+0000) 2022-09-08T02:23:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:18.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:18.689504+0000) 2022-09-08T02:23:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:18.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:18.634374+0000) 2022-09-08T02:23:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:18 smithi055 ceph-mon[97033]: pgmap v9193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:23:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:18 smithi059 ceph-mon[97450]: pgmap v9193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:23:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:19.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:19.689736+0000) 2022-09-08T02:23:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:19.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:19.634668+0000) 2022-09-08T02:23:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:20.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:20.634892+0000) 2022-09-08T02:23:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:20.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:20.689894+0000) 2022-09-08T02:23:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:20 smithi059 ceph-mon[97450]: pgmap v9194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:23:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:20 smithi055 ceph-mon[97033]: pgmap v9194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:23:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:21.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:21.690081+0000) 2022-09-08T02:23:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:21.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:21.635036+0000) 2022-09-08T02:23:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:22.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:22.690282+0000) 2022-09-08T02:23:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:22.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:22.635178+0000) 2022-09-08T02:23:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:22 smithi059 ceph-mon[97450]: pgmap v9195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:23:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:22 smithi055 ceph-mon[97033]: pgmap v9195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:23:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:23.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:23.690477+0000) 2022-09-08T02:23:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:23.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:23.635493+0000) 2022-09-08T02:23:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:24.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:24.690687+0000) 2022-09-08T02:23:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:24.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:24.635766+0000) 2022-09-08T02:23:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:24 smithi059 ceph-mon[97450]: pgmap v9196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:23:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:24 smithi055 ceph-mon[97033]: pgmap v9196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:23:25.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:23:25.621+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:23:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:25.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:25.690893+0000) 2022-09-08T02:23:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:25.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:25.635915+0000) 2022-09-08T02:23:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:26.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:26.691091+0000) 2022-09-08T02:23:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:26.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:26.636155+0000) 2022-09-08T02:23:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:26 smithi059 ceph-mon[97450]: pgmap v9197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:23:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:26 smithi055 ceph-mon[97033]: pgmap v9197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:23:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:27.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:27.691293+0000) 2022-09-08T02:23:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:27.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:27.636432+0000) 2022-09-08T02:23:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:28.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:28.691518+0000) 2022-09-08T02:23:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:28.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:28.636735+0000) 2022-09-08T02:23:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:28 smithi055 ceph-mon[97033]: pgmap v9198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:28 smithi059 ceph-mon[97450]: pgmap v9198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:29.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:29.691704+0000) 2022-09-08T02:23:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:29.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:29.636981+0000) 2022-09-08T02:23:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:30.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:30.691913+0000) 2022-09-08T02:23:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:30.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:30.637248+0000) 2022-09-08T02:23:31.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:30 smithi059 ceph-mon[97450]: pgmap v9199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:30 smithi055 ceph-mon[97033]: pgmap v9199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:31.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:31.692142+0000) 2022-09-08T02:23:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:31.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:31.637519+0000) 2022-09-08T02:23:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:32.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:32.637801+0000) 2022-09-08T02:23:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:32.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:32.692364+0000) 2022-09-08T02:23:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:32 smithi059 ceph-mon[97450]: pgmap v9200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:23:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:32 smithi055 ceph-mon[97033]: pgmap v9200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:23:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:33.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:33.692553+0000) 2022-09-08T02:23:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:33.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:33.637984+0000) 2022-09-08T02:23:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:34.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:34.638165+0000) 2022-09-08T02:23:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:34.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:34.692780+0000) 2022-09-08T02:23:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:34 smithi059 ceph-mon[97450]: pgmap v9201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:34 smithi055 ceph-mon[97033]: pgmap v9201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:35.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:35.692949+0000) 2022-09-08T02:23:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:35.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:35.638380+0000) 2022-09-08T02:23:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:36.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:36.693181+0000) 2022-09-08T02:23:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:36.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:36.638550+0000) 2022-09-08T02:23:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:36 smithi059 ceph-mon[97450]: pgmap v9202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:23:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:36 smithi055 ceph-mon[97033]: pgmap v9202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:23:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:37.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:37.693352+0000) 2022-09-08T02:23:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:37.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:37.638718+0000) 2022-09-08T02:23:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:23:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:23:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:23:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:23:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:23:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:23:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:38.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:38.693560+0000) 2022-09-08T02:23:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:38.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:38.638891+0000) 2022-09-08T02:23:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:38 smithi059 ceph-mon[97450]: pgmap v9203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:23:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:38 smithi055 ceph-mon[97033]: pgmap v9203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:23:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:39.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:39.639131+0000) 2022-09-08T02:23:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:39.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:39.693791+0000) 2022-09-08T02:23:40.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:23:40.622+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:23:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:40.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:40.639363+0000) 2022-09-08T02:23:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:40.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:40.693928+0000) 2022-09-08T02:23:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:40 smithi059 ceph-mon[97450]: pgmap v9204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:40 smithi055 ceph-mon[97033]: pgmap v9204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:41.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:41.639558+0000) 2022-09-08T02:23:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:41.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:41.694166+0000) 2022-09-08T02:23:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:42.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:42.639757+0000) 2022-09-08T02:23:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:42.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:42.694397+0000) 2022-09-08T02:23:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:42 smithi059 ceph-mon[97450]: pgmap v9205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:23:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:42 smithi055 ceph-mon[97033]: pgmap v9205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:23:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:43.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:43.694684+0000) 2022-09-08T02:23:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:43.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:43.639991+0000) 2022-09-08T02:23:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:44.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:44.694880+0000) 2022-09-08T02:23:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:44.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:44.640232+0000) 2022-09-08T02:23:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:44 smithi059 ceph-mon[97450]: pgmap v9206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:44 smithi055 ceph-mon[97033]: pgmap v9206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:45.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:45.695061+0000) 2022-09-08T02:23:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:45.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:45.640486+0000) 2022-09-08T02:23:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:46.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:46.695297+0000) 2022-09-08T02:23:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:46.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:46.640706+0000) 2022-09-08T02:23:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:46 smithi059 ceph-mon[97450]: pgmap v9207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:23:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:46 smithi055 ceph-mon[97033]: pgmap v9207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:23:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:47.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:47.695574+0000) 2022-09-08T02:23:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:47.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:47.640939+0000) 2022-09-08T02:23:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:48.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:48.695817+0000) 2022-09-08T02:23:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:48.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:48.641169+0000) 2022-09-08T02:23:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:48 smithi055 ceph-mon[97033]: pgmap v9208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:48 smithi059 ceph-mon[97450]: pgmap v9208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:49.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:49.696051+0000) 2022-09-08T02:23:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:49.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:49.641410+0000) 2022-09-08T02:23:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:50.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:50.696286+0000) 2022-09-08T02:23:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:50.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:50.641677+0000) 2022-09-08T02:23:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:50 smithi059 ceph-mon[97450]: pgmap v9209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:50 smithi055 ceph-mon[97033]: pgmap v9209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:51.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:51.696562+0000) 2022-09-08T02:23:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:51.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:51.641915+0000) 2022-09-08T02:23:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:52.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:52.696852+0000) 2022-09-08T02:23:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:52.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:52.642136+0000) 2022-09-08T02:23:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:52 smithi059 ceph-mon[97450]: pgmap v9210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:23:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:52 smithi055 ceph-mon[97033]: pgmap v9210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:23:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:53.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:53.697067+0000) 2022-09-08T02:23:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:53.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:53.642379+0000) 2022-09-08T02:23:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:54.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:54.697252+0000) 2022-09-08T02:23:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:54.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:54.642585+0000) 2022-09-08T02:23:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:54 smithi059 ceph-mon[97450]: pgmap v9211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:54 smithi055 ceph-mon[97033]: pgmap v9211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:55.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:55.697482+0000) 2022-09-08T02:23:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:55.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:55.642813+0000) 2022-09-08T02:23:55.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:23:55.633+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:23:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:56.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:56.697734+0000) 2022-09-08T02:23:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:56.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:56.643048+0000) 2022-09-08T02:23:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:56 smithi059 ceph-mon[97450]: pgmap v9212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:23:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:56 smithi055 ceph-mon[97033]: pgmap v9212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:23:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:57.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:57.697883+0000) 2022-09-08T02:23:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:57.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:57.643268+0000) 2022-09-08T02:23:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:58.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:58.698052+0000) 2022-09-08T02:23:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:58.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:58.643452+0000) 2022-09-08T02:23:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:23:58 smithi055 ceph-mon[97033]: pgmap v9213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:23:58 smithi059 ceph-mon[97450]: pgmap v9213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:23:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:23:59.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:59.698241+0000) 2022-09-08T02:23:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:23:59.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:23:59.643661+0000) 2022-09-08T02:24:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:00.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:00.698458+0000) 2022-09-08T02:24:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:00.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:00.643887+0000) 2022-09-08T02:24:01.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:00 smithi059 ceph-mon[97450]: pgmap v9214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:00 smithi055 ceph-mon[97033]: pgmap v9214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:01.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:01.698751+0000) 2022-09-08T02:24:01.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:01.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:01.644020+0000) 2022-09-08T02:24:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:02.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:02.698938+0000) 2022-09-08T02:24:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:02.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:02.644219+0000) 2022-09-08T02:24:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:02 smithi059 ceph-mon[97450]: pgmap v9215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:24:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:02 smithi055 ceph-mon[97033]: pgmap v9215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:24:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:03.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:03.644403+0000) 2022-09-08T02:24:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:03.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:03.699174+0000) 2022-09-08T02:24:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:04.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:04.699424+0000) 2022-09-08T02:24:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:04.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:04.644678+0000) 2022-09-08T02:24:05.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:04 smithi059 ceph-mon[97450]: pgmap v9216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:04 smithi055 ceph-mon[97033]: pgmap v9216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:05.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:05.699706+0000) 2022-09-08T02:24:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:05.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:05.644906+0000) 2022-09-08T02:24:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:24:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:24:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:24:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:24:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:06.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:06.699912+0000) 2022-09-08T02:24:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:06.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:06.645162+0000) 2022-09-08T02:24:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:06 smithi059 ceph-mon[97450]: pgmap v9217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:24:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:06 smithi055 ceph-mon[97033]: pgmap v9217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:24:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:07.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:07.700122+0000) 2022-09-08T02:24:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:07.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:07.645373+0000) 2022-09-08T02:24:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:08.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:08.700311+0000) 2022-09-08T02:24:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:08.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:08.645541+0000) 2022-09-08T02:24:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:08 smithi055 ceph-mon[97033]: pgmap v9218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:24:09.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:08 smithi059 ceph-mon[97450]: pgmap v9218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:24:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:09.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:09.700453+0000) 2022-09-08T02:24:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:09.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:09.645775+0000) 2022-09-08T02:24:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:09 smithi055 ceph-mon[97033]: osdmap e657: 12 total, 11 up, 11 in 2022-09-08T02:24:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:09 smithi059 ceph-mon[97450]: osdmap e657: 12 total, 11 up, 11 in 2022-09-08T02:24:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:10.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:10.700721+0000) 2022-09-08T02:24:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:10.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:10.645997+0000) 2022-09-08T02:24:10.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:24:10.648+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:24:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:10 smithi055 ceph-mon[97033]: pgmap v9220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:24:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:10 smithi059 ceph-mon[97450]: pgmap v9220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:24:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:11.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:11.701020+0000) 2022-09-08T02:24:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:11.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:11.646200+0000) 2022-09-08T02:24:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:12.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:12.646389+0000) 2022-09-08T02:24:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:12.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:12.701240+0000) 2022-09-08T02:24:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:12 smithi059 ceph-mon[97450]: pgmap v9221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:24:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:12 smithi059 ceph-mon[97450]: osdmap e658: 12 total, 11 up, 11 in 2022-09-08T02:24:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:12 smithi055 ceph-mon[97033]: pgmap v9221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:24:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:12 smithi055 ceph-mon[97033]: osdmap e658: 12 total, 11 up, 11 in 2022-09-08T02:24:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:13.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:13.701459+0000) 2022-09-08T02:24:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:13.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:13.646576+0000) 2022-09-08T02:24:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:14.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:14.701666+0000) 2022-09-08T02:24:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:14.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:14.646719+0000) 2022-09-08T02:24:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:14 smithi059 ceph-mon[97450]: pgmap v9223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:24:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:14 smithi055 ceph-mon[97033]: pgmap v9223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:24:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:15.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:15.701873+0000) 2022-09-08T02:24:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:15.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:15.646980+0000) 2022-09-08T02:24:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:16.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:16.702052+0000) 2022-09-08T02:24:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:16.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:16.647195+0000) 2022-09-08T02:24:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:16 smithi059 ceph-mon[97450]: pgmap v9224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:24:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:16 smithi055 ceph-mon[97033]: pgmap v9224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:24:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:17.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:17.702275+0000) 2022-09-08T02:24:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:17.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:17.647450+0000) 2022-09-08T02:24:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:18.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:18.702499+0000) 2022-09-08T02:24:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:18.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:18.647732+0000) 2022-09-08T02:24:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:18 smithi055 ceph-mon[97033]: pgmap v9225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:24:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:18 smithi059 ceph-mon[97450]: pgmap v9225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:24:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:19.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:19.702630+0000) 2022-09-08T02:24:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:19.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:19.647855+0000) 2022-09-08T02:24:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:20.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:20.702900+0000) 2022-09-08T02:24:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:20.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:20.648072+0000) 2022-09-08T02:24:21.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:20 smithi059 ceph-mon[97450]: pgmap v9226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:24:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:20 smithi055 ceph-mon[97033]: pgmap v9226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:24:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:21.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:21.648218+0000) 2022-09-08T02:24:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:21.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:21.703092+0000) 2022-09-08T02:24:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:22.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:22.703268+0000) 2022-09-08T02:24:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:22.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:22.648426+0000) 2022-09-08T02:24:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:22 smithi059 ceph-mon[97450]: pgmap v9227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:24:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:22 smithi055 ceph-mon[97033]: pgmap v9227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:24:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:23.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:23.648600+0000) 2022-09-08T02:24:23.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:23.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:23.703475+0000) 2022-09-08T02:24:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:24.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:24.648699+0000) 2022-09-08T02:24:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:24.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:24.703660+0000) 2022-09-08T02:24:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:24 smithi059 ceph-mon[97450]: pgmap v9228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:24:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:24 smithi055 ceph-mon[97033]: pgmap v9228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:24:25.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:24:25.661+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:24:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:25.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:25.703829+0000) 2022-09-08T02:24:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:25.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:25.648888+0000) 2022-09-08T02:24:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:26.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:26.649122+0000) 2022-09-08T02:24:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:26.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:26.704000+0000) 2022-09-08T02:24:27.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:26 smithi059 ceph-mon[97450]: pgmap v9229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:24:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:26 smithi055 ceph-mon[97033]: pgmap v9229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:24:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:27.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:27.704173+0000) 2022-09-08T02:24:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:27.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:27.649371+0000) 2022-09-08T02:24:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:28.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:28.649542+0000) 2022-09-08T02:24:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:28.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:28.704397+0000) 2022-09-08T02:24:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:28 smithi055 ceph-mon[97033]: pgmap v9230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:29.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:28 smithi059 ceph-mon[97450]: pgmap v9230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:29.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:29.704689+0000) 2022-09-08T02:24:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:29.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:29.649726+0000) 2022-09-08T02:24:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:30.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:30.649944+0000) 2022-09-08T02:24:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:30.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:30.704964+0000) 2022-09-08T02:24:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:30 smithi059 ceph-mon[97450]: pgmap v9231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:30 smithi055 ceph-mon[97033]: pgmap v9231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:31.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:31.705182+0000) 2022-09-08T02:24:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:31.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:31.650113+0000) 2022-09-08T02:24:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:32.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:32.705415+0000) 2022-09-08T02:24:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:32.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:32.650315+0000) 2022-09-08T02:24:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:32 smithi059 ceph-mon[97450]: pgmap v9232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:24:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:32 smithi055 ceph-mon[97033]: pgmap v9232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:24:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:33.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:33.705679+0000) 2022-09-08T02:24:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:33.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:33.650511+0000) 2022-09-08T02:24:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:34.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:34.705926+0000) 2022-09-08T02:24:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:34.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:34.650750+0000) 2022-09-08T02:24:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:34 smithi059 ceph-mon[97450]: pgmap v9233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:34 smithi055 ceph-mon[97033]: pgmap v9233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:35.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:35.706064+0000) 2022-09-08T02:24:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:35.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:35.650968+0000) 2022-09-08T02:24:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:36.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:36.706261+0000) 2022-09-08T02:24:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:36.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:36.651165+0000) 2022-09-08T02:24:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:36 smithi059 ceph-mon[97450]: pgmap v9234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:24:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:36 smithi055 ceph-mon[97033]: pgmap v9234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:24:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:37.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:37.706498+0000) 2022-09-08T02:24:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:37.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:37.651393+0000) 2022-09-08T02:24:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:38 smithi055 ceph-mon[97033]: pgmap v9235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:24:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:24:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:24:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:24:39.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:38 smithi059 ceph-mon[97450]: pgmap v9235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:24:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:24:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:24:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:24:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:38.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:38.651531+0000) 2022-09-08T02:24:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:38.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:38.706688+0000) 2022-09-08T02:24:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:39.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:39.706914+0000) 2022-09-08T02:24:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:39.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:39.651707+0000) 2022-09-08T02:24:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:24:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:39 smithi055 ceph-mon[97033]: pgmap v9236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:24:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:24:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:39 smithi059 ceph-mon[97450]: pgmap v9236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:24:41.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:24:40.678+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:24:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:40.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:40.707122+0000) 2022-09-08T02:24:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:40.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:40.651861+0000) 2022-09-08T02:24:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:41.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:41.651996+0000) 2022-09-08T02:24:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:41.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:41.707377+0000) 2022-09-08T02:24:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:42 smithi059 ceph-mon[97450]: pgmap v9237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:24:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:42.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:42.707595+0000) 2022-09-08T02:24:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:42.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:42.652158+0000) 2022-09-08T02:24:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:42 smithi055 ceph-mon[97033]: pgmap v9237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:24:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:43.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:43.652355+0000) 2022-09-08T02:24:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:43.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:43.707718+0000) 2022-09-08T02:24:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:44 smithi059 ceph-mon[97450]: pgmap v9238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:24:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:44.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:44.707872+0000) 2022-09-08T02:24:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:44.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:44.652576+0000) 2022-09-08T02:24:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:44 smithi055 ceph-mon[97033]: pgmap v9238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:24:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:45.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:45.708105+0000) 2022-09-08T02:24:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:45.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:45.652784+0000) 2022-09-08T02:24:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:46 smithi059 ceph-mon[97450]: pgmap v9239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:24:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:46.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:46.708295+0000) 2022-09-08T02:24:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:46.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:46.652976+0000) 2022-09-08T02:24:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:46 smithi055 ceph-mon[97033]: pgmap v9239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:24:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:47.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:47.708475+0000) 2022-09-08T02:24:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:47.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:47.653179+0000) 2022-09-08T02:24:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:48 smithi055 ceph-mon[97033]: pgmap v9240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:48 smithi059 ceph-mon[97450]: pgmap v9240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:48.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:48.708632+0000) 2022-09-08T02:24:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:48.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:48.653410+0000) 2022-09-08T02:24:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:49.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:49.708802+0000) 2022-09-08T02:24:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:49.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:49.653618+0000) 2022-09-08T02:24:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:50 smithi059 ceph-mon[97450]: pgmap v9241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:50.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:50.708995+0000) 2022-09-08T02:24:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:50.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:50.653906+0000) 2022-09-08T02:24:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:50 smithi055 ceph-mon[97033]: pgmap v9241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:51.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:51.709259+0000) 2022-09-08T02:24:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:51.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:51.654201+0000) 2022-09-08T02:24:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:52 smithi059 ceph-mon[97450]: pgmap v9242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:24:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:52.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:52.709488+0000) 2022-09-08T02:24:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:52.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:52.654463+0000) 2022-09-08T02:24:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:52 smithi055 ceph-mon[97033]: pgmap v9242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:24:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:53.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:53.709734+0000) 2022-09-08T02:24:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:53.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:53.654764+0000) 2022-09-08T02:24:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:54 smithi059 ceph-mon[97450]: pgmap v9243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:54.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:54.710004+0000) 2022-09-08T02:24:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:54.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:54.654922+0000) 2022-09-08T02:24:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:54 smithi055 ceph-mon[97033]: pgmap v9243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:55.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:55.710207+0000) 2022-09-08T02:24:56.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:24:55.693+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:24:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:55.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:55.655138+0000) 2022-09-08T02:24:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:56 smithi059 ceph-mon[97450]: pgmap v9244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:24:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:56.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:56.655322+0000) 2022-09-08T02:24:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:56.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:56.710434+0000) 2022-09-08T02:24:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:56 smithi055 ceph-mon[97033]: pgmap v9244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:24:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:57.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:57.655578+0000) 2022-09-08T02:24:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:57.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:57.710592+0000) 2022-09-08T02:24:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:24:58 smithi055 ceph-mon[97033]: pgmap v9245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:59.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:24:58 smithi059 ceph-mon[97450]: pgmap v9245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:24:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:58.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:58.655802+0000) 2022-09-08T02:24:59.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:58.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:58.710835+0000) 2022-09-08T02:25:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:24:59.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:59.710993+0000) 2022-09-08T02:25:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:24:59.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:24:59.655987+0000) 2022-09-08T02:25:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:00 smithi059 ceph-mon[97450]: pgmap v9246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:00.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:00.656150+0000) 2022-09-08T02:25:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:00.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:00.711202+0000) 2022-09-08T02:25:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:00 smithi055 ceph-mon[97033]: pgmap v9246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:01.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:01.711371+0000) 2022-09-08T02:25:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:01.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:01.656378+0000) 2022-09-08T02:25:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:02 smithi059 ceph-mon[97450]: pgmap v9247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:25:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:02.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:02.711611+0000) 2022-09-08T02:25:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:02.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:02.656612+0000) 2022-09-08T02:25:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:02 smithi055 ceph-mon[97033]: pgmap v9247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:25:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:03.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:03.711850+0000) 2022-09-08T02:25:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:03.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:03.656889+0000) 2022-09-08T02:25:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:04 smithi059 ceph-mon[97450]: pgmap v9248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:04.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:04.712090+0000) 2022-09-08T02:25:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:04.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:04.657109+0000) 2022-09-08T02:25:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:04 smithi055 ceph-mon[97033]: pgmap v9248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:25:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:25:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:05.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:05.712310+0000) 2022-09-08T02:25:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:05.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:05.657346+0000) 2022-09-08T02:25:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:25:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:25:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:06.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:06.657578+0000) 2022-09-08T02:25:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:06.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:06.712496+0000) 2022-09-08T02:25:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:06 smithi055 ceph-mon[97033]: pgmap v9249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:25:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:06 smithi059 ceph-mon[97450]: pgmap v9249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:25:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:07.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:07.712691+0000) 2022-09-08T02:25:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:07.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:07.657760+0000) 2022-09-08T02:25:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:08 smithi055 ceph-mon[97033]: pgmap v9250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:25:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:08 smithi059 ceph-mon[97450]: pgmap v9250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:25:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:08.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:08.657932+0000) 2022-09-08T02:25:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:08.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:08.712846+0000) 2022-09-08T02:25:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:09 smithi055 ceph-mon[97033]: osdmap e659: 12 total, 11 up, 11 in 2022-09-08T02:25:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:09 smithi059 ceph-mon[97450]: osdmap e659: 12 total, 11 up, 11 in 2022-09-08T02:25:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:09.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:09.713023+0000) 2022-09-08T02:25:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:09.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:09.658087+0000) 2022-09-08T02:25:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:10 smithi055 ceph-mon[97033]: pgmap v9252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:25:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:10 smithi059 ceph-mon[97450]: pgmap v9252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:25:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:10.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:10.713266+0000) 2022-09-08T02:25:11.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:25:10.706+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:25:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:10.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:10.658307+0000) 2022-09-08T02:25:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:11.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:11.713512+0000) 2022-09-08T02:25:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:11.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:11.658438+0000) 2022-09-08T02:25:13.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:12 smithi059 ceph-mon[97450]: pgmap v9253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:25:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:12 smithi059 ceph-mon[97450]: osdmap e660: 12 total, 11 up, 11 in 2022-09-08T02:25:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:12.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:12.713701+0000) 2022-09-08T02:25:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:12.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:12.658677+0000) 2022-09-08T02:25:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:12 smithi055 ceph-mon[97033]: pgmap v9253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:25:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:12 smithi055 ceph-mon[97033]: osdmap e660: 12 total, 11 up, 11 in 2022-09-08T02:25:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:13.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:13.658917+0000) 2022-09-08T02:25:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:13.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:13.713832+0000) 2022-09-08T02:25:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:14 smithi059 ceph-mon[97450]: pgmap v9255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:25:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:14.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:14.714062+0000) 2022-09-08T02:25:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:14.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:14.659098+0000) 2022-09-08T02:25:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:14 smithi055 ceph-mon[97033]: pgmap v9255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:25:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:15.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:15.714290+0000) 2022-09-08T02:25:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:15.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:15.659286+0000) 2022-09-08T02:25:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:16 smithi059 ceph-mon[97450]: pgmap v9256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:25:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:16.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:16.714518+0000) 2022-09-08T02:25:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:16.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:16.659511+0000) 2022-09-08T02:25:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:16 smithi055 ceph-mon[97033]: pgmap v9256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:25:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:17.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:17.714818+0000) 2022-09-08T02:25:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:17.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:17.659803+0000) 2022-09-08T02:25:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:18 smithi055 ceph-mon[97033]: pgmap v9257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:25:19.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:18 smithi059 ceph-mon[97450]: pgmap v9257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:25:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:18.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:18.715084+0000) 2022-09-08T02:25:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:18.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:18.659960+0000) 2022-09-08T02:25:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:19.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:19.660169+0000) 2022-09-08T02:25:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:19.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:19.715278+0000) 2022-09-08T02:25:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:20 smithi059 ceph-mon[97450]: pgmap v9258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:25:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:20.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:20.715500+0000) 2022-09-08T02:25:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:20.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:20.660380+0000) 2022-09-08T02:25:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:20 smithi055 ceph-mon[97033]: pgmap v9258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:25:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:21.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:21.715777+0000) 2022-09-08T02:25:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:21.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:21.660584+0000) 2022-09-08T02:25:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:22 smithi059 ceph-mon[97450]: pgmap v9259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:25:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:22.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:22.716041+0000) 2022-09-08T02:25:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:22.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:22.660705+0000) 2022-09-08T02:25:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:22 smithi055 ceph-mon[97033]: pgmap v9259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:25:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:23.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:23.716309+0000) 2022-09-08T02:25:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:23.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:23.660855+0000) 2022-09-08T02:25:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:24 smithi059 ceph-mon[97450]: pgmap v9260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:25:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:24.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:24.716561+0000) 2022-09-08T02:25:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:24.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:24.660993+0000) 2022-09-08T02:25:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:24 smithi055 ceph-mon[97033]: pgmap v9260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:25:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:25.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:25.716769+0000) 2022-09-08T02:25:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:25.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:25.661172+0000) 2022-09-08T02:25:26.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:25:25.721+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:25:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:26.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:26.716971+0000) 2022-09-08T02:25:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:26.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:26.661334+0000) 2022-09-08T02:25:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:26 smithi055 ceph-mon[97033]: pgmap v9261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:25:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:26 smithi059 ceph-mon[97450]: pgmap v9261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:25:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:27.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:27.717213+0000) 2022-09-08T02:25:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:27.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:27.661545+0000) 2022-09-08T02:25:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:28 smithi055 ceph-mon[97033]: pgmap v9262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:28 smithi059 ceph-mon[97450]: pgmap v9262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:28.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:28.661719+0000) 2022-09-08T02:25:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:28.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:28.717438+0000) 2022-09-08T02:25:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:29.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:29.717715+0000) 2022-09-08T02:25:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:29.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:29.661845+0000) 2022-09-08T02:25:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:30 smithi059 ceph-mon[97450]: pgmap v9263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:30.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:30.717935+0000) 2022-09-08T02:25:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:30.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:30.661992+0000) 2022-09-08T02:25:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:30 smithi055 ceph-mon[97033]: pgmap v9263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:31.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:31.718179+0000) 2022-09-08T02:25:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:31.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:31.662216+0000) 2022-09-08T02:25:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:32 smithi059 ceph-mon[97450]: pgmap v9264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:25:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:32.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:32.662387+0000) 2022-09-08T02:25:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:32.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:32.718423+0000) 2022-09-08T02:25:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:32 smithi055 ceph-mon[97033]: pgmap v9264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:25:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:33.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:33.718713+0000) 2022-09-08T02:25:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:33.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:33.662614+0000) 2022-09-08T02:25:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:34.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:34.718907+0000) 2022-09-08T02:25:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:34.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:34.662813+0000) 2022-09-08T02:25:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:34 smithi055 ceph-mon[97033]: pgmap v9265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:34 smithi059 ceph-mon[97450]: pgmap v9265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:35.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:35.719129+0000) 2022-09-08T02:25:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:35.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:35.663034+0000) 2022-09-08T02:25:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:36.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:36.719388+0000) 2022-09-08T02:25:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:36.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:36.663256+0000) 2022-09-08T02:25:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:36 smithi055 ceph-mon[97033]: pgmap v9266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:25:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:36 smithi059 ceph-mon[97450]: pgmap v9266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:25:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:37.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:37.719606+0000) 2022-09-08T02:25:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:37.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:37.663505+0000) 2022-09-08T02:25:38.968 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:38 smithi059 ceph-mon[97450]: pgmap v9267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:38.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:38 smithi055 ceph-mon[97033]: pgmap v9267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:38.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:38.719693+0000) 2022-09-08T02:25:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:38.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:38.663727+0000) 2022-09-08T02:25:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:25:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:25:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:25:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:25:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:25:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:25:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:25:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:25:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:39.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:39.719835+0000) 2022-09-08T02:25:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:39.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:39.663938+0000) 2022-09-08T02:25:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:40 smithi055 ceph-mon[97033]: pgmap v9268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:40 smithi059 ceph-mon[97450]: pgmap v9268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:40.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:40.719962+0000) 2022-09-08T02:25:41.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:25:40.732+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:25:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:40.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:40.664158+0000) 2022-09-08T02:25:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:41.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:41.664399+0000) 2022-09-08T02:25:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:41.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:41.720182+0000) 2022-09-08T02:25:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:42 smithi059 ceph-mon[97450]: pgmap v9269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:25:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:42.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:42.720388+0000) 2022-09-08T02:25:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:42.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:42.664668+0000) 2022-09-08T02:25:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:42 smithi055 ceph-mon[97033]: pgmap v9269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:25:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:43.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:43.720610+0000) 2022-09-08T02:25:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:43.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:43.664848+0000) 2022-09-08T02:25:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:44.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:44.720795+0000) 2022-09-08T02:25:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:44.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:44.665072+0000) 2022-09-08T02:25:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:44 smithi055 ceph-mon[97033]: pgmap v9270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:44 smithi059 ceph-mon[97450]: pgmap v9270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:45.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:45.721013+0000) 2022-09-08T02:25:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:45.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:45.665346+0000) 2022-09-08T02:25:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:46.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:46.721184+0000) 2022-09-08T02:25:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:46.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:46.665571+0000) 2022-09-08T02:25:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:46 smithi055 ceph-mon[97033]: pgmap v9271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:25:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:46 smithi059 ceph-mon[97450]: pgmap v9271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:25:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:47.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:47.665786+0000) 2022-09-08T02:25:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:47.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:47.721413+0000) 2022-09-08T02:25:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:48 smithi055 ceph-mon[97033]: pgmap v9272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:48 smithi059 ceph-mon[97450]: pgmap v9272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:48.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:48.721710+0000) 2022-09-08T02:25:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:48.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:48.665985+0000) 2022-09-08T02:25:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:49.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:49.721874+0000) 2022-09-08T02:25:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:49.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:49.666201+0000) 2022-09-08T02:25:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:50.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:50.722112+0000) 2022-09-08T02:25:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:50.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:50.666464+0000) 2022-09-08T02:25:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:50 smithi055 ceph-mon[97033]: pgmap v9273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:50 smithi059 ceph-mon[97450]: pgmap v9273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:51.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:51.722387+0000) 2022-09-08T02:25:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:51.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:51.666610+0000) 2022-09-08T02:25:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:52.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:52.722629+0000) 2022-09-08T02:25:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:52.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:52.666803+0000) 2022-09-08T02:25:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:52 smithi055 ceph-mon[97033]: pgmap v9274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:25:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:52 smithi059 ceph-mon[97450]: pgmap v9274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:25:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:53.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:53.722781+0000) 2022-09-08T02:25:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:53.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:53.667060+0000) 2022-09-08T02:25:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:54.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:54.722930+0000) 2022-09-08T02:25:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:54.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:54.667277+0000) 2022-09-08T02:25:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:54 smithi055 ceph-mon[97033]: pgmap v9275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:54 smithi059 ceph-mon[97450]: pgmap v9275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:56.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:25:55.735+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:25:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:55.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:55.667510+0000) 2022-09-08T02:25:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:55.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:55.723106+0000) 2022-09-08T02:25:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:56.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:56.723320+0000) 2022-09-08T02:25:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:56.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:56.667761+0000) 2022-09-08T02:25:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:56 smithi055 ceph-mon[97033]: pgmap v9276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:25:57.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:56 smithi059 ceph-mon[97450]: pgmap v9276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:25:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:57.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:57.723510+0000) 2022-09-08T02:25:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:57.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:57.667939+0000) 2022-09-08T02:25:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:25:58 smithi055 ceph-mon[97033]: pgmap v9277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:25:58 smithi059 ceph-mon[97450]: pgmap v9277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:25:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:58.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:58.723747+0000) 2022-09-08T02:25:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:58.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:58.668112+0000) 2022-09-08T02:26:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:25:59.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:59.723947+0000) 2022-09-08T02:26:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:25:59.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:25:59.668293+0000) 2022-09-08T02:26:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:00.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:00.724088+0000) 2022-09-08T02:26:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:00.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:00.668452+0000) 2022-09-08T02:26:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:00 smithi055 ceph-mon[97033]: pgmap v9278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:00 smithi059 ceph-mon[97450]: pgmap v9278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:01.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:01.724291+0000) 2022-09-08T02:26:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:01.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:01.668703+0000) 2022-09-08T02:26:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:02.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:02.668861+0000) 2022-09-08T02:26:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:02.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:02.724534+0000) 2022-09-08T02:26:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:02 smithi055 ceph-mon[97033]: pgmap v9279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:26:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:02 smithi059 ceph-mon[97450]: pgmap v9279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:26:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:03.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:03.669112+0000) 2022-09-08T02:26:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:03.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:03.724784+0000) 2022-09-08T02:26:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:04.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:04.669286+0000) 2022-09-08T02:26:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:04.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:04.724960+0000) 2022-09-08T02:26:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:04 smithi055 ceph-mon[97033]: pgmap v9280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:04 smithi059 ceph-mon[97450]: pgmap v9280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:05.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:05.725204+0000) 2022-09-08T02:26:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:05.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:05.669457+0000) 2022-09-08T02:26:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:26:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:26:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:26:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:26:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:06.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:06.669608+0000) 2022-09-08T02:26:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:06.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:06.725412+0000) 2022-09-08T02:26:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:06 smithi055 ceph-mon[97033]: pgmap v9281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:26:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:06 smithi059 ceph-mon[97450]: pgmap v9281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:26:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:07.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:07.725610+0000) 2022-09-08T02:26:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:07.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:07.669857+0000) 2022-09-08T02:26:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:08 smithi055 ceph-mon[97033]: pgmap v9282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:26:09.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:08 smithi059 ceph-mon[97450]: pgmap v9282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:26:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:08.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:08.670109+0000) 2022-09-08T02:26:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:08.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:08.725830+0000) 2022-09-08T02:26:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:09 smithi055 ceph-mon[97033]: osdmap e661: 12 total, 11 up, 11 in 2022-09-08T02:26:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:09 smithi059 ceph-mon[97450]: osdmap e661: 12 total, 11 up, 11 in 2022-09-08T02:26:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:09.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:09.726079+0000) 2022-09-08T02:26:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:09.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:09.670252+0000) 2022-09-08T02:26:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:10 smithi059 ceph-mon[97450]: pgmap v9284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:26:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:10.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:10.670481+0000) 2022-09-08T02:26:11.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:26:10.739+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:26:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:10.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:10.726300+0000) 2022-09-08T02:26:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:10 smithi055 ceph-mon[97033]: pgmap v9284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:26:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:11.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:11.726560+0000) 2022-09-08T02:26:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:11.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:11.670676+0000) 2022-09-08T02:26:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:12.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:12.726718+0000) 2022-09-08T02:26:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:12.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:12.670833+0000) 2022-09-08T02:26:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:12 smithi055 ceph-mon[97033]: pgmap v9285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:26:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:12 smithi055 ceph-mon[97033]: osdmap e662: 12 total, 11 up, 11 in 2022-09-08T02:26:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:12 smithi059 ceph-mon[97450]: pgmap v9285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:26:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:12 smithi059 ceph-mon[97450]: osdmap e662: 12 total, 11 up, 11 in 2022-09-08T02:26:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:13.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:13.726862+0000) 2022-09-08T02:26:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:13.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:13.671000+0000) 2022-09-08T02:26:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:14.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:14.727077+0000) 2022-09-08T02:26:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:14.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:14.671240+0000) 2022-09-08T02:26:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:14 smithi055 ceph-mon[97033]: pgmap v9287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:26:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:14 smithi059 ceph-mon[97450]: pgmap v9287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:26:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:15.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:15.727323+0000) 2022-09-08T02:26:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:15.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:15.671492+0000) 2022-09-08T02:26:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:16.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:16.727552+0000) 2022-09-08T02:26:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:16.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:16.671653+0000) 2022-09-08T02:26:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:16 smithi055 ceph-mon[97033]: pgmap v9288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:26:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:16 smithi059 ceph-mon[97450]: pgmap v9288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:26:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:17.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:17.727779+0000) 2022-09-08T02:26:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:17.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:17.671828+0000) 2022-09-08T02:26:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:18 smithi055 ceph-mon[97033]: pgmap v9289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:26:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:18 smithi059 ceph-mon[97450]: pgmap v9289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:26:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:18.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:18.728032+0000) 2022-09-08T02:26:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:18.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:18.672055+0000) 2022-09-08T02:26:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:19.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:19.728318+0000) 2022-09-08T02:26:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:19.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:19.672233+0000) 2022-09-08T02:26:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:20.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:20.728572+0000) 2022-09-08T02:26:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:20.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:20.672432+0000) 2022-09-08T02:26:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:20 smithi055 ceph-mon[97033]: pgmap v9290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:26:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:20 smithi059 ceph-mon[97450]: pgmap v9290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:26:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:21.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:21.728828+0000) 2022-09-08T02:26:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:21.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:21.672704+0000) 2022-09-08T02:26:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:22.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:22.729043+0000) 2022-09-08T02:26:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:22.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:22.672921+0000) 2022-09-08T02:26:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:22 smithi055 ceph-mon[97033]: pgmap v9291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:26:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:22 smithi059 ceph-mon[97450]: pgmap v9291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:26:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:23.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:23.729300+0000) 2022-09-08T02:26:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:23.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:23.673157+0000) 2022-09-08T02:26:25.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:24.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:24.729534+0000) 2022-09-08T02:26:25.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:24.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:24.673328+0000) 2022-09-08T02:26:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:24 smithi055 ceph-mon[97033]: pgmap v9292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:26:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:24 smithi059 ceph-mon[97450]: pgmap v9292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:26:26.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:26:25.744+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:26:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:25.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:25.729758+0000) 2022-09-08T02:26:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:25.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:25.673562+0000) 2022-09-08T02:26:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:26.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:26.730012+0000) 2022-09-08T02:26:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:26.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:26.673714+0000) 2022-09-08T02:26:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:26 smithi055 ceph-mon[97033]: pgmap v9293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:26:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:26 smithi059 ceph-mon[97450]: pgmap v9293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:26:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:27.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:27.673852+0000) 2022-09-08T02:26:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:27.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:27.730246+0000) 2022-09-08T02:26:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:28 smithi055 ceph-mon[97033]: pgmap v9294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:28 smithi059 ceph-mon[97450]: pgmap v9294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:28.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:28.730469+0000) 2022-09-08T02:26:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:28.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:28.673998+0000) 2022-09-08T02:26:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:29.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:29.730722+0000) 2022-09-08T02:26:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:29.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:29.674202+0000) 2022-09-08T02:26:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:30.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:30.730862+0000) 2022-09-08T02:26:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:30.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:30.674395+0000) 2022-09-08T02:26:31.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:30 smithi055 ceph-mon[97033]: pgmap v9295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:30 smithi059 ceph-mon[97450]: pgmap v9295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:31.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:31.731091+0000) 2022-09-08T02:26:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:31.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:31.674607+0000) 2022-09-08T02:26:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:32.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:32.731305+0000) 2022-09-08T02:26:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:32.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:32.674821+0000) 2022-09-08T02:26:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:32 smithi055 ceph-mon[97033]: pgmap v9296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:26:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:32 smithi059 ceph-mon[97450]: pgmap v9296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:26:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:33.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:33.675050+0000) 2022-09-08T02:26:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:33.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:33.731516+0000) 2022-09-08T02:26:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:34.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:34.731759+0000) 2022-09-08T02:26:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:34.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:34.675305+0000) 2022-09-08T02:26:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:34 smithi055 ceph-mon[97033]: pgmap v9297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:34 smithi059 ceph-mon[97450]: pgmap v9297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:35.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:35.731942+0000) 2022-09-08T02:26:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:35.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:35.675491+0000) 2022-09-08T02:26:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:36.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:36.732181+0000) 2022-09-08T02:26:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:36.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:36.675722+0000) 2022-09-08T02:26:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:36 smithi055 ceph-mon[97033]: pgmap v9298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:26:37.291 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:36 smithi059 ceph-mon[97450]: pgmap v9298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:26:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:37.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:37.675939+0000) 2022-09-08T02:26:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:37.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:37.732333+0000) 2022-09-08T02:26:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:38 smithi055 ceph-mon[97033]: pgmap v9299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:38 smithi059 ceph-mon[97450]: pgmap v9299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:38.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:38.732451+0000) 2022-09-08T02:26:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:38.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:38.676127+0000) 2022-09-08T02:26:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:26:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:26:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:26:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:26:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:26:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:26:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:39.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:39.732724+0000) 2022-09-08T02:26:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:39.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:39.676308+0000) 2022-09-08T02:26:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:40 smithi059 ceph-mon[97450]: pgmap v9300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:26:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:40.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:40.676542+0000) 2022-09-08T02:26:41.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:26:40.758+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:26:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:40.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:40.732939+0000) 2022-09-08T02:26:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:40 smithi055 ceph-mon[97033]: pgmap v9300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:26:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:41.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:41.733170+0000) 2022-09-08T02:26:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:41.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:41.676793+0000) 2022-09-08T02:26:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:42.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:42.733400+0000) 2022-09-08T02:26:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:42.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:42.677005+0000) 2022-09-08T02:26:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:42 smithi055 ceph-mon[97033]: pgmap v9301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:26:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:42 smithi059 ceph-mon[97450]: pgmap v9301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:26:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:43.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:43.733624+0000) 2022-09-08T02:26:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:43.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:43.677255+0000) 2022-09-08T02:26:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:44.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:44.733775+0000) 2022-09-08T02:26:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:44.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:44.677501+0000) 2022-09-08T02:26:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:44 smithi055 ceph-mon[97033]: pgmap v9302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:44 smithi059 ceph-mon[97450]: pgmap v9302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:45.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:45.734015+0000) 2022-09-08T02:26:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:45.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:45.677780+0000) 2022-09-08T02:26:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:46.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:46.734258+0000) 2022-09-08T02:26:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:46.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:46.677980+0000) 2022-09-08T02:26:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:46 smithi055 ceph-mon[97033]: pgmap v9303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:26:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:46 smithi059 ceph-mon[97450]: pgmap v9303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:26:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:47.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:47.734480+0000) 2022-09-08T02:26:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:47.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:47.678224+0000) 2022-09-08T02:26:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:48 smithi059 ceph-mon[97450]: pgmap v9304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:48.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:48.734684+0000) 2022-09-08T02:26:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:48.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:48.678419+0000) 2022-09-08T02:26:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:48 smithi055 ceph-mon[97033]: pgmap v9304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:49.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:49.734884+0000) 2022-09-08T02:26:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:49.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:49.678680+0000) 2022-09-08T02:26:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:50.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:50.735102+0000) 2022-09-08T02:26:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:50.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:50.678926+0000) 2022-09-08T02:26:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:50 smithi055 ceph-mon[97033]: pgmap v9305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:50 smithi059 ceph-mon[97450]: pgmap v9305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:51.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:51.735344+0000) 2022-09-08T02:26:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:51.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:51.679176+0000) 2022-09-08T02:26:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:52.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:52.735549+0000) 2022-09-08T02:26:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:52.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:52.679421+0000) 2022-09-08T02:26:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:52 smithi055 ceph-mon[97033]: pgmap v9306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:26:53.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:52 smithi059 ceph-mon[97450]: pgmap v9306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:26:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:53.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:53.735794+0000) 2022-09-08T02:26:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:53.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:53.679572+0000) 2022-09-08T02:26:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:54.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:54.736037+0000) 2022-09-08T02:26:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:54.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:54.679770+0000) 2022-09-08T02:26:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:54 smithi055 ceph-mon[97033]: pgmap v9307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:54 smithi059 ceph-mon[97450]: pgmap v9307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:56.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:26:55.772+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:26:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:55.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:55.736249+0000) 2022-09-08T02:26:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:55.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:55.679932+0000) 2022-09-08T02:26:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:56.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:56.736422+0000) 2022-09-08T02:26:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:56.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:56.680098+0000) 2022-09-08T02:26:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:56 smithi055 ceph-mon[97033]: pgmap v9308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:26:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:56 smithi059 ceph-mon[97450]: pgmap v9308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:26:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:57.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:57.736601+0000) 2022-09-08T02:26:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:57.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:57.680291+0000) 2022-09-08T02:26:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:26:58 smithi059 ceph-mon[97450]: pgmap v9309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:26:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:58.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:58.736799+0000) 2022-09-08T02:26:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:58.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:58.680515+0000) 2022-09-08T02:26:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:26:58 smithi055 ceph-mon[97033]: pgmap v9309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:26:59.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:59.737023+0000) 2022-09-08T02:27:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:26:59.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:26:59.680695+0000) 2022-09-08T02:27:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:00.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:00.737228+0000) 2022-09-08T02:27:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:00.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:00.680909+0000) 2022-09-08T02:27:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:00 smithi055 ceph-mon[97033]: pgmap v9310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:00 smithi059 ceph-mon[97450]: pgmap v9310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:01.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:01.737484+0000) 2022-09-08T02:27:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:01.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:01.681118+0000) 2022-09-08T02:27:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:02.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:02.681310+0000) 2022-09-08T02:27:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:02.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:02.737727+0000) 2022-09-08T02:27:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:02 smithi055 ceph-mon[97033]: pgmap v9311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:27:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:02 smithi059 ceph-mon[97450]: pgmap v9311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:27:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:03.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:03.681539+0000) 2022-09-08T02:27:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:03.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:03.737958+0000) 2022-09-08T02:27:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:04.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:04.738159+0000) 2022-09-08T02:27:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:04.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:04.681750+0000) 2022-09-08T02:27:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:04 smithi055 ceph-mon[97033]: pgmap v9312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:05.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:04 smithi059 ceph-mon[97450]: pgmap v9312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:05.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:05.738378+0000) 2022-09-08T02:27:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:05.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:05.681968+0000) 2022-09-08T02:27:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:27:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:27:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:27:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:27:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:06.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:06.738631+0000) 2022-09-08T02:27:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:06.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:06.682146+0000) 2022-09-08T02:27:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:06 smithi055 ceph-mon[97033]: pgmap v9313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:27:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:06 smithi059 ceph-mon[97450]: pgmap v9313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:27:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:07.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:07.738879+0000) 2022-09-08T02:27:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:07.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:07.682387+0000) 2022-09-08T02:27:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:08 smithi059 ceph-mon[97450]: pgmap v9314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:27:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:08.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:08.739086+0000) 2022-09-08T02:27:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:08.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:08.682583+0000) 2022-09-08T02:27:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:08 smithi055 ceph-mon[97033]: pgmap v9314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:27:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:09 smithi059 ceph-mon[97450]: osdmap e663: 12 total, 11 up, 11 in 2022-09-08T02:27:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:09.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:09.682811+0000) 2022-09-08T02:27:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:09.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:09.739296+0000) 2022-09-08T02:27:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:09 smithi055 ceph-mon[97033]: osdmap e663: 12 total, 11 up, 11 in 2022-09-08T02:27:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:10 smithi059 ceph-mon[97450]: pgmap v9316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:27:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:10.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:10.739514+0000) 2022-09-08T02:27:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:10.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:10.683057+0000) 2022-09-08T02:27:11.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:27:10.786+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:27:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:10 smithi055 ceph-mon[97033]: pgmap v9316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:27:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:11.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:11.739788+0000) 2022-09-08T02:27:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:11.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:11.683280+0000) 2022-09-08T02:27:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:12.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:12.739957+0000) 2022-09-08T02:27:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:12.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:12.683512+0000) 2022-09-08T02:27:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:12 smithi055 ceph-mon[97033]: pgmap v9317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:27:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:12 smithi055 ceph-mon[97033]: osdmap e664: 12 total, 11 up, 11 in 2022-09-08T02:27:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:12 smithi059 ceph-mon[97450]: pgmap v9317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:27:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:12 smithi059 ceph-mon[97450]: osdmap e664: 12 total, 11 up, 11 in 2022-09-08T02:27:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:13.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:13.683700+0000) 2022-09-08T02:27:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:13.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:13.740161+0000) 2022-09-08T02:27:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:14.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:14.740317+0000) 2022-09-08T02:27:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:14.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:14.683913+0000) 2022-09-08T02:27:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:14 smithi055 ceph-mon[97033]: pgmap v9319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:27:15.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:14 smithi059 ceph-mon[97450]: pgmap v9319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:27:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:15.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:15.740593+0000) 2022-09-08T02:27:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:15.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:15.684175+0000) 2022-09-08T02:27:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:16.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:16.740871+0000) 2022-09-08T02:27:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:16.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:16.684368+0000) 2022-09-08T02:27:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:16 smithi055 ceph-mon[97033]: pgmap v9320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:27:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:16 smithi059 ceph-mon[97450]: pgmap v9320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:27:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:17.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:17.741136+0000) 2022-09-08T02:27:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:17.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:17.684604+0000) 2022-09-08T02:27:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:18 smithi059 ceph-mon[97450]: pgmap v9321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:27:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:18.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:18.741376+0000) 2022-09-08T02:27:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:18.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:18.684864+0000) 2022-09-08T02:27:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:18 smithi055 ceph-mon[97033]: pgmap v9321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T02:27:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:19.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:19.741612+0000) 2022-09-08T02:27:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:19.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:19.685033+0000) 2022-09-08T02:27:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:20.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:20.741906+0000) 2022-09-08T02:27:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:20.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:20.685235+0000) 2022-09-08T02:27:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:20 smithi055 ceph-mon[97033]: pgmap v9322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 204 B/s wr, 7 op/s 2022-09-08T02:27:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:20 smithi059 ceph-mon[97450]: pgmap v9322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 204 B/s wr, 7 op/s 2022-09-08T02:27:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:21.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:21.685471+0000) 2022-09-08T02:27:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:21.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:21.742055+0000) 2022-09-08T02:27:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:22.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:22.685672+0000) 2022-09-08T02:27:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:22.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:22.742282+0000) 2022-09-08T02:27:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:22 smithi055 ceph-mon[97033]: pgmap v9323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:27:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:22 smithi059 ceph-mon[97450]: pgmap v9323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:27:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:23.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:23.685838+0000) 2022-09-08T02:27:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:23.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:23.742492+0000) 2022-09-08T02:27:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:24.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:24.742731+0000) 2022-09-08T02:27:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:24.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:24.686016+0000) 2022-09-08T02:27:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:24 smithi055 ceph-mon[97033]: pgmap v9324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:27:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:24 smithi059 ceph-mon[97450]: pgmap v9324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:27:26.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:27:25.797+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:27:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:25.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:25.686158+0000) 2022-09-08T02:27:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:25.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:25.742973+0000) 2022-09-08T02:27:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:26.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:26.743201+0000) 2022-09-08T02:27:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:26.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:26.686381+0000) 2022-09-08T02:27:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:26 smithi055 ceph-mon[97033]: pgmap v9325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:27:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:26 smithi059 ceph-mon[97450]: pgmap v9325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:27:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:27.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:27.743414+0000) 2022-09-08T02:27:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:27.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:27.686673+0000) 2022-09-08T02:27:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:28 smithi059 ceph-mon[97450]: pgmap v9326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:28.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:28.743688+0000) 2022-09-08T02:27:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:28.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:28.686861+0000) 2022-09-08T02:27:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:28 smithi055 ceph-mon[97033]: pgmap v9326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:29.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:29.743811+0000) 2022-09-08T02:27:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:29.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:29.687007+0000) 2022-09-08T02:27:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:30.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:30.687247+0000) 2022-09-08T02:27:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:30.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:30.744057+0000) 2022-09-08T02:27:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:30 smithi055 ceph-mon[97033]: pgmap v9327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T02:27:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:30 smithi059 ceph-mon[97450]: pgmap v9327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T02:27:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:31.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:31.687463+0000) 2022-09-08T02:27:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:31.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:31.744293+0000) 2022-09-08T02:27:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:32.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:32.687725+0000) 2022-09-08T02:27:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:32.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:32.744520+0000) 2022-09-08T02:27:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:32 smithi055 ceph-mon[97033]: pgmap v9328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:27:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:32 smithi059 ceph-mon[97450]: pgmap v9328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:27:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:33.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:33.687969+0000) 2022-09-08T02:27:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:33.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:33.744704+0000) 2022-09-08T02:27:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:34.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:34.744889+0000) 2022-09-08T02:27:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:34.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:34.688229+0000) 2022-09-08T02:27:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:34 smithi055 ceph-mon[97033]: pgmap v9329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:34 smithi059 ceph-mon[97450]: pgmap v9329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:35.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:35.745050+0000) 2022-09-08T02:27:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:35.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:35.688463+0000) 2022-09-08T02:27:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:36.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:36.745231+0000) 2022-09-08T02:27:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:36.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:36.688686+0000) 2022-09-08T02:27:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:36 smithi055 ceph-mon[97033]: pgmap v9330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:27:37.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:36 smithi059 ceph-mon[97450]: pgmap v9330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:27:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:37.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:37.745440+0000) 2022-09-08T02:27:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:37.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:37.688849+0000) 2022-09-08T02:27:39.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:38 smithi059 ceph-mon[97450]: pgmap v9331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:38.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:38.745673+0000) 2022-09-08T02:27:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:38.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:38.689019+0000) 2022-09-08T02:27:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:38 smithi055 ceph-mon[97033]: pgmap v9331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:39.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:39.745800+0000) 2022-09-08T02:27:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:39.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:39.689228+0000) 2022-09-08T02:27:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:40 smithi055 ceph-mon[97033]: pgmap v9332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T02:27:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:27:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:27:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:27:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:27:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:40 smithi059 ceph-mon[97450]: pgmap v9332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T02:27:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:27:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:27:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:27:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:27:41.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:27:40.809+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:27:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:40.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:40.746012+0000) 2022-09-08T02:27:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:40.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:40.689392+0000) 2022-09-08T02:27:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:41.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:41.746247+0000) 2022-09-08T02:27:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:41.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:41.689580+0000) 2022-09-08T02:27:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:42.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:42.746394+0000) 2022-09-08T02:27:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:42.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:42.689794+0000) 2022-09-08T02:27:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:42 smithi055 ceph-mon[97033]: pgmap v9333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:27:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:42 smithi059 ceph-mon[97450]: pgmap v9333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T02:27:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:43.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:43.746612+0000) 2022-09-08T02:27:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:43.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:43.689995+0000) 2022-09-08T02:27:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:44.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:44.746883+0000) 2022-09-08T02:27:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:44.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:44.690232+0000) 2022-09-08T02:27:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:44 smithi055 ceph-mon[97033]: pgmap v9334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:44 smithi059 ceph-mon[97450]: pgmap v9334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:45.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:45.747061+0000) 2022-09-08T02:27:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:45.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:45.690472+0000) 2022-09-08T02:27:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:46.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:46.747264+0000) 2022-09-08T02:27:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:46.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:46.690734+0000) 2022-09-08T02:27:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:46 smithi055 ceph-mon[97033]: pgmap v9335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:27:47.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:46 smithi059 ceph-mon[97450]: pgmap v9335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:27:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:47.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:47.747480+0000) 2022-09-08T02:27:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:47.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:47.690972+0000) 2022-09-08T02:27:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:48 smithi059 ceph-mon[97450]: pgmap v9336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:48.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:48.747734+0000) 2022-09-08T02:27:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:48.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:48.691175+0000) 2022-09-08T02:27:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:48 smithi055 ceph-mon[97033]: pgmap v9336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:49.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:49.747886+0000) 2022-09-08T02:27:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:49.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:49.691384+0000) 2022-09-08T02:27:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:50.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:50.748112+0000) 2022-09-08T02:27:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:50.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:50.691532+0000) 2022-09-08T02:27:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:50 smithi055 ceph-mon[97033]: pgmap v9337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:27:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:50 smithi059 ceph-mon[97450]: pgmap v9337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:27:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:51.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:51.691712+0000) 2022-09-08T02:27:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:51.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:51.748409+0000) 2022-09-08T02:27:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:52.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:52.691888+0000) 2022-09-08T02:27:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:52.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:52.748724+0000) 2022-09-08T02:27:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:52 smithi055 ceph-mon[97033]: pgmap v9338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:27:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:52 smithi059 ceph-mon[97450]: pgmap v9338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:27:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:53.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:53.749023+0000) 2022-09-08T02:27:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:53.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:53.692075+0000) 2022-09-08T02:27:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:54.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:54.749197+0000) 2022-09-08T02:27:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:54.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:54.692235+0000) 2022-09-08T02:27:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:54 smithi055 ceph-mon[97033]: pgmap v9339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:54 smithi059 ceph-mon[97450]: pgmap v9339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:55.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:55.749363+0000) 2022-09-08T02:27:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:55.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:55.692488+0000) 2022-09-08T02:27:56.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:27:55.821+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:27:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:56.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:56.749572+0000) 2022-09-08T02:27:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:56.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:56.692741+0000) 2022-09-08T02:27:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:56 smithi055 ceph-mon[97033]: pgmap v9340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:27:57.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:56 smithi059 ceph-mon[97450]: pgmap v9340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:27:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:57.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:57.749715+0000) 2022-09-08T02:27:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:57.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:57.693012+0000) 2022-09-08T02:27:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:27:58 smithi059 ceph-mon[97450]: pgmap v9341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:27:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:58.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:58.749840+0000) 2022-09-08T02:27:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:58.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:58.693146+0000) 2022-09-08T02:27:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:27:58 smithi055 ceph-mon[97033]: pgmap v9341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:27:59.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:59.749984+0000) 2022-09-08T02:28:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:27:59.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:27:59.693391+0000) 2022-09-08T02:28:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:00.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:00.750179+0000) 2022-09-08T02:28:01.195 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:00.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:00.693629+0000) 2022-09-08T02:28:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:00 smithi055 ceph-mon[97033]: pgmap v9342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:28:01.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:00 smithi059 ceph-mon[97450]: pgmap v9342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:28:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:01.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:01.750424+0000) 2022-09-08T02:28:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:01.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:01.693865+0000) 2022-09-08T02:28:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:02.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:02.750578+0000) 2022-09-08T02:28:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:02.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:02.694072+0000) 2022-09-08T02:28:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:02 smithi055 ceph-mon[97033]: pgmap v9343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:02 smithi059 ceph-mon[97450]: pgmap v9343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:03.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:03.750763+0000) 2022-09-08T02:28:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:03.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:03.694322+0000) 2022-09-08T02:28:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:04.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:04.750891+0000) 2022-09-08T02:28:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:04.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:04.694556+0000) 2022-09-08T02:28:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:04 smithi055 ceph-mon[97033]: pgmap v9344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:05.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:04 smithi059 ceph-mon[97450]: pgmap v9344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:05.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:05.751052+0000) 2022-09-08T02:28:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:05.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:05.694797+0000) 2022-09-08T02:28:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:28:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:28:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:28:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:28:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:06.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:06.751278+0000) 2022-09-08T02:28:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:06.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:06.694958+0000) 2022-09-08T02:28:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:06 smithi055 ceph-mon[97033]: pgmap v9345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:28:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:06 smithi059 ceph-mon[97450]: pgmap v9345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:28:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:07.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:07.751550+0000) 2022-09-08T02:28:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:07.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:07.695168+0000) 2022-09-08T02:28:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:08 smithi059 ceph-mon[97450]: pgmap v9346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:28:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:08.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:08.751763+0000) 2022-09-08T02:28:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:08.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:08.695365+0000) 2022-09-08T02:28:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:08 smithi055 ceph-mon[97033]: pgmap v9346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:28:10.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:09 smithi059 ceph-mon[97450]: osdmap e665: 12 total, 11 up, 11 in 2022-09-08T02:28:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:09.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:09.695586+0000) 2022-09-08T02:28:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:09.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:09.751896+0000) 2022-09-08T02:28:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:09 smithi055 ceph-mon[97033]: osdmap e665: 12 total, 11 up, 11 in 2022-09-08T02:28:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:10 smithi059 ceph-mon[97450]: pgmap v9348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:28:11.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:28:10.835+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:28:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:10.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:10.752082+0000) 2022-09-08T02:28:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:10.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:10.695812+0000) 2022-09-08T02:28:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:10 smithi055 ceph-mon[97033]: pgmap v9348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:28:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:11.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:11.752231+0000) 2022-09-08T02:28:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:11.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:11.695991+0000) 2022-09-08T02:28:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:12.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:12.752442+0000) 2022-09-08T02:28:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:12.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:12.696253+0000) 2022-09-08T02:28:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:12 smithi055 ceph-mon[97033]: pgmap v9349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:28:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:12 smithi055 ceph-mon[97033]: osdmap e666: 12 total, 11 up, 11 in 2022-09-08T02:28:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:12 smithi059 ceph-mon[97450]: pgmap v9349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:28:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:12 smithi059 ceph-mon[97450]: osdmap e666: 12 total, 11 up, 11 in 2022-09-08T02:28:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:13.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:13.752678+0000) 2022-09-08T02:28:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:13.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:13.696428+0000) 2022-09-08T02:28:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:14.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:14.752876+0000) 2022-09-08T02:28:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:14.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:14.696618+0000) 2022-09-08T02:28:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:14 smithi055 ceph-mon[97033]: pgmap v9351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:28:15.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:14 smithi059 ceph-mon[97450]: pgmap v9351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:28:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:15.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:15.753100+0000) 2022-09-08T02:28:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:15.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:15.696864+0000) 2022-09-08T02:28:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:16.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:16.697022+0000) 2022-09-08T02:28:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:16.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:16.753295+0000) 2022-09-08T02:28:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:16 smithi055 ceph-mon[97033]: pgmap v9352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:28:17.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:16 smithi059 ceph-mon[97450]: pgmap v9352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:28:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:17.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:17.753497+0000) 2022-09-08T02:28:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:17.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:17.697268+0000) 2022-09-08T02:28:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:18 smithi059 ceph-mon[97450]: pgmap v9353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 226 B/s wr, 7 op/s 2022-09-08T02:28:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:18.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:18.697473+0000) 2022-09-08T02:28:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:18.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:18.753666+0000) 2022-09-08T02:28:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:18 smithi055 ceph-mon[97033]: pgmap v9353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 226 B/s wr, 7 op/s 2022-09-08T02:28:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:19.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:19.697686+0000) 2022-09-08T02:28:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:19.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:19.753842+0000) 2022-09-08T02:28:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:20.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:20.754089+0000) 2022-09-08T02:28:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:20.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:20.697904+0000) 2022-09-08T02:28:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:20 smithi055 ceph-mon[97033]: pgmap v9354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:28:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:20 smithi059 ceph-mon[97450]: pgmap v9354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:28:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:21.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:21.754362+0000) 2022-09-08T02:28:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:21.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:21.698131+0000) 2022-09-08T02:28:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:22.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:22.754574+0000) 2022-09-08T02:28:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:22.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:22.698267+0000) 2022-09-08T02:28:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:22 smithi055 ceph-mon[97033]: pgmap v9355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:28:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:22 smithi059 ceph-mon[97450]: pgmap v9355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:28:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:23.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:23.754775+0000) 2022-09-08T02:28:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:23.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:23.698468+0000) 2022-09-08T02:28:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:24.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:24.754902+0000) 2022-09-08T02:28:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:24.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:24.698721+0000) 2022-09-08T02:28:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:24 smithi055 ceph-mon[97033]: pgmap v9356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:28:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:24 smithi059 ceph-mon[97450]: pgmap v9356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T02:28:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:25.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:25.755072+0000) 2022-09-08T02:28:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:25.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:25.698962+0000) 2022-09-08T02:28:26.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:28:25.847+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:28:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:26.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:26.755290+0000) 2022-09-08T02:28:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:26.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:26.699079+0000) 2022-09-08T02:28:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:26 smithi055 ceph-mon[97033]: pgmap v9357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:28:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:26 smithi059 ceph-mon[97450]: pgmap v9357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:28:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:27.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:27.755509+0000) 2022-09-08T02:28:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:27.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:27.699348+0000) 2022-09-08T02:28:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:28 smithi059 ceph-mon[97450]: pgmap v9358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:28.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:28.755755+0000) 2022-09-08T02:28:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:28.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:28.699512+0000) 2022-09-08T02:28:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:28 smithi055 ceph-mon[97033]: pgmap v9358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:29.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:29.755915+0000) 2022-09-08T02:28:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:29.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:29.699726+0000) 2022-09-08T02:28:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:30.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:30.756186+0000) 2022-09-08T02:28:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:30.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:30.699975+0000) 2022-09-08T02:28:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:30 smithi055 ceph-mon[97033]: pgmap v9359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:28:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:30 smithi059 ceph-mon[97450]: pgmap v9359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:28:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:31.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:31.756465+0000) 2022-09-08T02:28:32.182 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:31.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:31.700138+0000) 2022-09-08T02:28:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:32.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:32.756685+0000) 2022-09-08T02:28:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:32.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:32.700357+0000) 2022-09-08T02:28:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:32 smithi055 ceph-mon[97033]: pgmap v9360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:32 smithi059 ceph-mon[97450]: pgmap v9360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:33.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:33.756876+0000) 2022-09-08T02:28:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:33.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:33.700581+0000) 2022-09-08T02:28:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:34.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:34.757102+0000) 2022-09-08T02:28:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:34.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:34.700751+0000) 2022-09-08T02:28:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:34 smithi055 ceph-mon[97033]: pgmap v9361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:34 smithi059 ceph-mon[97450]: pgmap v9361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:35.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:35.757377+0000) 2022-09-08T02:28:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:35.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:35.700927+0000) 2022-09-08T02:28:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:36.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:36.757668+0000) 2022-09-08T02:28:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:36.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:36.701174+0000) 2022-09-08T02:28:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:36 smithi055 ceph-mon[97033]: pgmap v9362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:28:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:36 smithi059 ceph-mon[97450]: pgmap v9362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:28:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:37.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:37.757823+0000) 2022-09-08T02:28:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:37.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:37.701386+0000) 2022-09-08T02:28:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:38 smithi059 ceph-mon[97450]: pgmap v9363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:38.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:38.758010+0000) 2022-09-08T02:28:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:38.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:38.701600+0000) 2022-09-08T02:28:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:38 smithi055 ceph-mon[97033]: pgmap v9363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:39.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:39.758184+0000) 2022-09-08T02:28:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:39.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:39.701899+0000) 2022-09-08T02:28:41.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:28:40.862+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:28:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:40.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:40.702044+0000) 2022-09-08T02:28:41.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:40.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:40.758387+0000) 2022-09-08T02:28:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:40 smithi055 ceph-mon[97033]: pgmap v9364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:28:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:28:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:28:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:28:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:40 smithi059 ceph-mon[97450]: pgmap v9364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:28:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:28:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:28:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:28:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:41.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:41.758645+0000) 2022-09-08T02:28:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:41.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:41.702260+0000) 2022-09-08T02:28:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:28:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:28:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:42 smithi059 ceph-mon[97450]: pgmap v9365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:28:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:28:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:42 smithi055 ceph-mon[97033]: pgmap v9365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:42.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:42.758749+0000) 2022-09-08T02:28:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:42.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:42.702407+0000) 2022-09-08T02:28:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:43.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:43.758891+0000) 2022-09-08T02:28:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:43.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:43.702667+0000) 2022-09-08T02:28:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:28:44.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:28:44.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:44 smithi055 ceph-mon[97033]: pgmap v9366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:44.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:44.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:44.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:28:44.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:28:44.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:44.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:44.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:44.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:28:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:28:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:44 smithi059 ceph-mon[97450]: pgmap v9366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:28:45.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:28:45.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:45.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:45.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:45.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:28:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:44.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:44.759051+0000) 2022-09-08T02:28:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:44.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:44.702848+0000) 2022-09-08T02:28:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:45 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T02:28:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:45 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T02:28:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:45 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T02:28:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:45 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T02:28:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:45 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T02:28:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:45 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T02:28:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:45 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T02:28:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:45 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T02:28:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:45.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:45.759290+0000) 2022-09-08T02:28:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:45.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:45.703069+0000) 2022-09-08T02:28:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:46 smithi055 ceph-mon[97033]: pgmap v9367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:28:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:46 smithi059 ceph-mon[97450]: pgmap v9367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:28:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:46.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:46.759525+0000) 2022-09-08T02:28:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:46.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:46.703260+0000) 2022-09-08T02:28:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:47.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:47.759696+0000) 2022-09-08T02:28:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:47.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:47.703401+0000) 2022-09-08T02:28:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:48 smithi059 ceph-mon[97450]: pgmap v9368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:48.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:48.759906+0000) 2022-09-08T02:28:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:48.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:48.703613+0000) 2022-09-08T02:28:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:48 smithi055 ceph-mon[97033]: pgmap v9368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:49.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:49.703850+0000) 2022-09-08T02:28:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:49.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:49.760064+0000) 2022-09-08T02:28:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:50.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:50.760264+0000) 2022-09-08T02:28:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:50.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:50.704017+0000) 2022-09-08T02:28:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:50 smithi055 ceph-mon[97033]: pgmap v9369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:28:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:50 smithi059 ceph-mon[97450]: pgmap v9369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:28:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:51.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:51.760500+0000) 2022-09-08T02:28:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:51.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:51.704252+0000) 2022-09-08T02:28:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:52.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:52.760742+0000) 2022-09-08T02:28:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:52.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:52.704438+0000) 2022-09-08T02:28:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:52 smithi055 ceph-mon[97033]: pgmap v9370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:52 smithi059 ceph-mon[97450]: pgmap v9370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:53.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:53.760940+0000) 2022-09-08T02:28:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:53.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:53.704689+0000) 2022-09-08T02:28:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:54.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:54.761070+0000) 2022-09-08T02:28:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:54.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:54.704887+0000) 2022-09-08T02:28:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:54 smithi055 ceph-mon[97033]: pgmap v9371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:54 smithi059 ceph-mon[97450]: pgmap v9371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:55.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:55.761248+0000) 2022-09-08T02:28:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:55.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:55.705081+0000) 2022-09-08T02:28:56.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:28:55.871+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:28:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:55 smithi055 ceph-mon[97033]: pgmap v9372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:28:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:55 smithi059 ceph-mon[97450]: pgmap v9372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:28:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:56.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:56.761459+0000) 2022-09-08T02:28:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:56.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:56.705290+0000) 2022-09-08T02:28:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:57.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:57.761708+0000) 2022-09-08T02:28:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:57.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:57.705485+0000) 2022-09-08T02:28:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:28:58 smithi059 ceph-mon[97450]: pgmap v9373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:28:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:58.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:58.761898+0000) 2022-09-08T02:28:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:58.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:58.705699+0000) 2022-09-08T02:28:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:28:58 smithi055 ceph-mon[97033]: pgmap v9373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:28:59.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:59.762047+0000) 2022-09-08T02:29:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:28:59.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:28:59.705957+0000) 2022-09-08T02:29:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:00.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:00.762172+0000) 2022-09-08T02:29:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:00.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:00.706239+0000) 2022-09-08T02:29:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:00 smithi055 ceph-mon[97033]: pgmap v9374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:29:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:00 smithi059 ceph-mon[97450]: pgmap v9374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:29:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:01.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:01.706473+0000) 2022-09-08T02:29:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:01.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:01.762404+0000) 2022-09-08T02:29:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:02.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:02.762624+0000) 2022-09-08T02:29:03.177 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:02.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:02.706758+0000) 2022-09-08T02:29:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:02 smithi055 ceph-mon[97033]: pgmap v9375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:02 smithi059 ceph-mon[97450]: pgmap v9375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:03.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:03.762854+0000) 2022-09-08T02:29:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:03.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:03.706998+0000) 2022-09-08T02:29:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:04.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:04.763080+0000) 2022-09-08T02:29:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:04.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:04.707243+0000) 2022-09-08T02:29:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:04 smithi055 ceph-mon[97033]: pgmap v9376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:04 smithi059 ceph-mon[97450]: pgmap v9376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:06.002 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:05.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:05.763348+0000) 2022-09-08T02:29:06.003 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:05.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:05.707407+0000) 2022-09-08T02:29:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:29:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:29:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:29:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:29:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:06.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:06.707621+0000) 2022-09-08T02:29:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:06.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:06.763621+0000) 2022-09-08T02:29:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:06 smithi055 ceph-mon[97033]: pgmap v9377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:29:07.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:06 smithi059 ceph-mon[97450]: pgmap v9377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:29:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:07.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:07.763808+0000) 2022-09-08T02:29:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:07.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:07.707918+0000) 2022-09-08T02:29:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:08 smithi059 ceph-mon[97450]: pgmap v9378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:29:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:08.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:08.764015+0000) 2022-09-08T02:29:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:08.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:08.708162+0000) 2022-09-08T02:29:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:08 smithi055 ceph-mon[97033]: pgmap v9378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:29:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:09 smithi059 ceph-mon[97450]: osdmap e667: 12 total, 11 up, 11 in 2022-09-08T02:29:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:09.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:09.764162+0000) 2022-09-08T02:29:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:09.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:09.708393+0000) 2022-09-08T02:29:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:09 smithi055 ceph-mon[97033]: osdmap e667: 12 total, 11 up, 11 in 2022-09-08T02:29:11.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:10 smithi059 ceph-mon[97450]: pgmap v9380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:29:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:10.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:10.764386+0000) 2022-09-08T02:29:11.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:29:10.886+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:29:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:10.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:10.708632+0000) 2022-09-08T02:29:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:10 smithi055 ceph-mon[97033]: pgmap v9380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:29:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:11.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:11.764681+0000) 2022-09-08T02:29:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:11.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:11.708844+0000) 2022-09-08T02:29:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:11 smithi055 ceph-mon[97033]: pgmap v9381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 58 KiB/s wr, 26 op/s 2022-09-08T02:29:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:29:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:11 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T02:29:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:11 smithi059 ceph-mon[97450]: pgmap v9381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 58 KiB/s wr, 26 op/s 2022-09-08T02:29:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:29:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:11 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T02:29:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:12.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:12.709069+0000) 2022-09-08T02:29:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:12.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:12.764829+0000) 2022-09-08T02:29:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:13 smithi059 ceph-mon[97450]: osdmap e668: 12 total, 11 up, 11 in 2022-09-08T02:29:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:13.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:13.709291+0000) 2022-09-08T02:29:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:13.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:13.765025+0000) 2022-09-08T02:29:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:13 smithi055 ceph-mon[97033]: osdmap e668: 12 total, 11 up, 11 in 2022-09-08T02:29:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:14 smithi059 ceph-mon[97450]: pgmap v9383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 143 B/s wr, 4 op/s 2022-09-08T02:29:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:14.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:14.709536+0000) 2022-09-08T02:29:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:14.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:14.765172+0000) 2022-09-08T02:29:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:14 smithi055 ceph-mon[97033]: pgmap v9383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 143 B/s wr, 4 op/s 2022-09-08T02:29:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:15.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:15.709809+0000) 2022-09-08T02:29:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:15.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:15.765414+0000) 2022-09-08T02:29:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:16.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:16.710060+0000) 2022-09-08T02:29:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:16.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:16.765593+0000) 2022-09-08T02:29:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:16 smithi055 ceph-mon[97033]: pgmap v9384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.5 KiB/s rd, 286 B/s wr, 8 op/s 2022-09-08T02:29:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:16 smithi059 ceph-mon[97450]: pgmap v9384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.5 KiB/s rd, 286 B/s wr, 8 op/s 2022-09-08T02:29:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:17.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:17.710277+0000) 2022-09-08T02:29:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:17.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:17.765872+0000) 2022-09-08T02:29:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:18.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:18.710475+0000) 2022-09-08T02:29:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:18.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:18.766081+0000) 2022-09-08T02:29:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:18 smithi055 ceph-mon[97033]: pgmap v9385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T02:29:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:18 smithi059 ceph-mon[97450]: pgmap v9385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T02:29:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:19.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:19.710711+0000) 2022-09-08T02:29:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:19.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:19.766224+0000) 2022-09-08T02:29:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:20.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:20.766423+0000) 2022-09-08T02:29:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:20.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:20.710865+0000) 2022-09-08T02:29:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:20 smithi055 ceph-mon[97033]: pgmap v9386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 111 B/s wr, 3 op/s 2022-09-08T02:29:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:20 smithi059 ceph-mon[97450]: pgmap v9386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 111 B/s wr, 3 op/s 2022-09-08T02:29:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:21.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:21.711040+0000) 2022-09-08T02:29:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:21.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:21.766697+0000) 2022-09-08T02:29:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:22.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:22.766913+0000) 2022-09-08T02:29:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:22.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:22.711256+0000) 2022-09-08T02:29:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:22 smithi055 ceph-mon[97033]: pgmap v9387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:29:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:22 smithi059 ceph-mon[97450]: pgmap v9387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:29:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:23.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:23.767041+0000) 2022-09-08T02:29:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:23.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:23.711470+0000) 2022-09-08T02:29:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:24.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:24.767191+0000) 2022-09-08T02:29:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:24.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:24.711720+0000) 2022-09-08T02:29:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:24 smithi055 ceph-mon[97033]: pgmap v9388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T02:29:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:24 smithi059 ceph-mon[97450]: pgmap v9388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T02:29:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:25.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:25.767380+0000) 2022-09-08T02:29:26.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:29:25.897+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:29:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:25.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:25.711891+0000) 2022-09-08T02:29:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:26.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:26.767555+0000) 2022-09-08T02:29:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:26.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:26.712018+0000) 2022-09-08T02:29:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:26 smithi055 ceph-mon[97033]: pgmap v9389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:29:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:26 smithi059 ceph-mon[97450]: pgmap v9389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:29:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:27.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:27.767773+0000) 2022-09-08T02:29:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:27.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:27.712221+0000) 2022-09-08T02:29:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:28.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:28.767980+0000) 2022-09-08T02:29:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:28.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:28.712352+0000) 2022-09-08T02:29:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:28 smithi055 ceph-mon[97033]: pgmap v9390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:28 smithi059 ceph-mon[97450]: pgmap v9390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:29.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:29.768209+0000) 2022-09-08T02:29:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:29.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:29.712559+0000) 2022-09-08T02:29:31.020 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:30.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:30.768455+0000) 2022-09-08T02:29:31.021 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:30.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:30.712777+0000) 2022-09-08T02:29:31.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:30 smithi055 ceph-mon[97033]: pgmap v9391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:30 smithi059 ceph-mon[97450]: pgmap v9391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:31.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:31.712969+0000) 2022-09-08T02:29:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:31.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:31.768719+0000) 2022-09-08T02:29:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:32.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:32.768948+0000) 2022-09-08T02:29:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:32.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:32.713188+0000) 2022-09-08T02:29:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:32 smithi055 ceph-mon[97033]: pgmap v9392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:29:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:32 smithi059 ceph-mon[97450]: pgmap v9392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:29:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:33.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:33.769160+0000) 2022-09-08T02:29:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:33.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:33.713364+0000) 2022-09-08T02:29:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:34.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:34.769388+0000) 2022-09-08T02:29:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:34.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:34.713574+0000) 2022-09-08T02:29:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:34 smithi055 ceph-mon[97033]: pgmap v9393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:34 smithi059 ceph-mon[97450]: pgmap v9393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:35.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:35.769615+0000) 2022-09-08T02:29:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:35.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:35.713898+0000) 2022-09-08T02:29:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:36.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:36.714147+0000) 2022-09-08T02:29:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:36.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:36.769888+0000) 2022-09-08T02:29:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:36 smithi055 ceph-mon[97033]: pgmap v9394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:29:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:36 smithi059 ceph-mon[97450]: pgmap v9394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:29:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:37.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:37.714361+0000) 2022-09-08T02:29:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:37.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:37.770121+0000) 2022-09-08T02:29:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:38.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:38.714547+0000) 2022-09-08T02:29:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:38.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:38.770342+0000) 2022-09-08T02:29:39.240 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:38 smithi055 ceph-mon[97033]: pgmap v9395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:38 smithi059 ceph-mon[97450]: pgmap v9395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:39.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:39.770541+0000) 2022-09-08T02:29:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:39.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:39.714782+0000) 2022-09-08T02:29:41.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:29:40.912+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:29:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:40.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:40.770714+0000) 2022-09-08T02:29:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:40.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:40.715004+0000) 2022-09-08T02:29:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:40 smithi055 ceph-mon[97033]: pgmap v9396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:40 smithi059 ceph-mon[97450]: pgmap v9396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:41.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:41.770931+0000) 2022-09-08T02:29:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:41.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:41.715245+0000) 2022-09-08T02:29:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:42.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:42.771101+0000) 2022-09-08T02:29:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:42.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:42.715479+0000) 2022-09-08T02:29:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:42 smithi055 ceph-mon[97033]: pgmap v9397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:29:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:42 smithi059 ceph-mon[97450]: pgmap v9397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:29:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:43.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:43.771307+0000) 2022-09-08T02:29:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:43.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:43.715660+0000) 2022-09-08T02:29:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:44.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:44.771452+0000) 2022-09-08T02:29:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:44.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:44.715830+0000) 2022-09-08T02:29:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:44 smithi055 ceph-mon[97033]: pgmap v9398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:44 smithi059 ceph-mon[97450]: pgmap v9398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:45.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:45.771737+0000) 2022-09-08T02:29:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:45.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:45.716100+0000) 2022-09-08T02:29:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:46.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:46.772020+0000) 2022-09-08T02:29:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:46.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:46.716316+0000) 2022-09-08T02:29:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:46 smithi055 ceph-mon[97033]: pgmap v9399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:29:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:46 smithi059 ceph-mon[97450]: pgmap v9399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:29:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:47.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:47.772277+0000) 2022-09-08T02:29:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:47.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:47.716508+0000) 2022-09-08T02:29:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:48.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:48.772554+0000) 2022-09-08T02:29:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:48.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:48.716732+0000) 2022-09-08T02:29:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:48 smithi055 ceph-mon[97033]: pgmap v9400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:48 smithi059 ceph-mon[97450]: pgmap v9400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:49.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:49.772832+0000) 2022-09-08T02:29:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:49.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:49.716977+0000) 2022-09-08T02:29:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:50.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:50.773057+0000) 2022-09-08T02:29:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:50.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:50.717150+0000) 2022-09-08T02:29:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:50 smithi055 ceph-mon[97033]: pgmap v9401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:50 smithi059 ceph-mon[97450]: pgmap v9401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:51.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:51.773270+0000) 2022-09-08T02:29:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:51.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:51.717362+0000) 2022-09-08T02:29:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:52.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:52.773513+0000) 2022-09-08T02:29:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:52.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:52.717580+0000) 2022-09-08T02:29:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:52 smithi055 ceph-mon[97033]: pgmap v9402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:29:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:52 smithi059 ceph-mon[97450]: pgmap v9402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:29:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:53.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:53.773741+0000) 2022-09-08T02:29:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:53.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:53.717707+0000) 2022-09-08T02:29:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:54.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:54.773883+0000) 2022-09-08T02:29:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:54.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:54.717877+0000) 2022-09-08T02:29:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:54 smithi055 ceph-mon[97033]: pgmap v9403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:54 smithi059 ceph-mon[97450]: pgmap v9403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:56.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:29:55.917+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:29:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:55.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:55.774112+0000) 2022-09-08T02:29:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:55.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:55.718026+0000) 2022-09-08T02:29:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:55 smithi055 ceph-mon[97033]: pgmap v9404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:29:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:55 smithi059 ceph-mon[97450]: pgmap v9404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:29:57.160 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:56.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:56.774380+0000) 2022-09-08T02:29:57.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:56.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:56.718229+0000) 2022-09-08T02:29:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:57.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:57.774680+0000) 2022-09-08T02:29:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:57.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:57.718450+0000) 2022-09-08T02:29:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:29:58 smithi059 ceph-mon[97450]: pgmap v9405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:29:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:58.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:58.774962+0000) 2022-09-08T02:29:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:58.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:58.718607+0000) 2022-09-08T02:29:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:29:58 smithi055 ceph-mon[97033]: pgmap v9405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:29:59.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:59.775143+0000) 2022-09-08T02:30:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:29:59.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:29:59.718842+0000) 2022-09-08T02:30:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:00 smithi059 ceph-mon[97450]: pgmap v9406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T02:30:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T02:30:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T02:30:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T02:30:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T02:30:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:00.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:00.775292+0000) 2022-09-08T02:30:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:00.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:00.719006+0000) 2022-09-08T02:30:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:00 smithi055 ceph-mon[97033]: pgmap v9406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T02:30:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T02:30:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T02:30:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T02:30:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T02:30:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:01.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:01.775521+0000) 2022-09-08T02:30:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:01.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:01.719152+0000) 2022-09-08T02:30:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:02.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:02.775798+0000) 2022-09-08T02:30:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:02.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:02.719369+0000) 2022-09-08T02:30:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:02 smithi055 ceph-mon[97033]: pgmap v9407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:30:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:02 smithi059 ceph-mon[97450]: pgmap v9407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:30:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:03.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:03.775936+0000) 2022-09-08T02:30:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:03.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:03.719534+0000) 2022-09-08T02:30:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:04 smithi055 ceph-mon[97033]: pgmap v9408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:04.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:04.776079+0000) 2022-09-08T02:30:05.176 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:04.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:04.719771+0000) 2022-09-08T02:30:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:04 smithi059 ceph-mon[97450]: pgmap v9408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:05.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:05.776277+0000) 2022-09-08T02:30:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:05.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:05.719974+0000) 2022-09-08T02:30:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:30:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:30:06.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:30:06.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:30:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:06.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:06.776488+0000) 2022-09-08T02:30:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:06.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:06.720202+0000) 2022-09-08T02:30:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:06 smithi055 ceph-mon[97033]: pgmap v9409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:30:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:06 smithi059 ceph-mon[97450]: pgmap v9409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:30:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:07.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:07.776699+0000) 2022-09-08T02:30:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:07.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:07.720491+0000) 2022-09-08T02:30:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:08.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:08.720750+0000) 2022-09-08T02:30:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:08.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:08.776914+0000) 2022-09-08T02:30:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:08 smithi055 ceph-mon[97033]: pgmap v9410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:30:09.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:08 smithi059 ceph-mon[97450]: pgmap v9410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:30:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:09.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:09.720986+0000) 2022-09-08T02:30:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:09.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:09.777091+0000) 2022-09-08T02:30:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:09 smithi055 ceph-mon[97033]: osdmap e669: 12 total, 11 up, 11 in 2022-09-08T02:30:10.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:09 smithi059 ceph-mon[97450]: osdmap e669: 12 total, 11 up, 11 in 2022-09-08T02:30:11.101 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:30:10.926+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:30:11.101 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:10.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:10.777269+0000) 2022-09-08T02:30:11.102 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:10.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:10.721207+0000) 2022-09-08T02:30:11.122 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:10 smithi055 ceph-mon[97033]: pgmap v9412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:30:11.124 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:10 smithi059 ceph-mon[97450]: pgmap v9412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:30:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:30:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:30:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:30:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:30:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:30:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:11.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:11.777487+0000) 2022-09-08T02:30:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:11.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:11.721469+0000) 2022-09-08T02:30:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:30:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:30:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:30:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:30:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:30:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:12.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:12.777716+0000) 2022-09-08T02:30:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:12.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:12.721616+0000) 2022-09-08T02:30:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:12 smithi055 ceph-mon[97033]: pgmap v9413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:30:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:12 smithi055 ceph-mon[97033]: osdmap e670: 12 total, 11 up, 11 in 2022-09-08T02:30:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:12 smithi059 ceph-mon[97450]: pgmap v9413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:30:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:12 smithi059 ceph-mon[97450]: osdmap e670: 12 total, 11 up, 11 in 2022-09-08T02:30:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:13.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:13.777903+0000) 2022-09-08T02:30:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:13.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:13.721852+0000) 2022-09-08T02:30:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:14.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:14.778057+0000) 2022-09-08T02:30:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:14.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:14.722011+0000) 2022-09-08T02:30:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:14 smithi055 ceph-mon[97033]: pgmap v9415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:30:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:14 smithi059 ceph-mon[97450]: pgmap v9415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:30:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:15.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:15.778253+0000) 2022-09-08T02:30:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:15.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:15.722193+0000) 2022-09-08T02:30:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:16.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:16.778455+0000) 2022-09-08T02:30:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:16.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:16.722409+0000) 2022-09-08T02:30:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:16 smithi055 ceph-mon[97033]: pgmap v9416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:30:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:16 smithi059 ceph-mon[97450]: pgmap v9416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:30:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:17.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:17.722657+0000) 2022-09-08T02:30:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:17.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:17.778716+0000) 2022-09-08T02:30:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:18.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:18.778849+0000) 2022-09-08T02:30:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:18.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:18.722868+0000) 2022-09-08T02:30:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:18 smithi055 ceph-mon[97033]: pgmap v9417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 251 B/s wr, 7 op/s 2022-09-08T02:30:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:18 smithi059 ceph-mon[97450]: pgmap v9417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 251 B/s wr, 7 op/s 2022-09-08T02:30:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:19.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:19.778983+0000) 2022-09-08T02:30:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:19.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:19.723084+0000) 2022-09-08T02:30:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:20.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:20.779208+0000) 2022-09-08T02:30:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:20.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:20.723327+0000) 2022-09-08T02:30:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:20 smithi055 ceph-mon[97033]: pgmap v9418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:30:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:20 smithi059 ceph-mon[97450]: pgmap v9418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:30:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:21.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:21.779434+0000) 2022-09-08T02:30:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:21.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:21.723570+0000) 2022-09-08T02:30:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:22.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:22.779660+0000) 2022-09-08T02:30:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:22.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:22.723748+0000) 2022-09-08T02:30:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:22 smithi055 ceph-mon[97033]: pgmap v9419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:30:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:22 smithi059 ceph-mon[97450]: pgmap v9419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:30:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:23.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:23.779886+0000) 2022-09-08T02:30:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:23.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:23.723977+0000) 2022-09-08T02:30:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:24.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:24.780045+0000) 2022-09-08T02:30:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:24.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:24.724189+0000) 2022-09-08T02:30:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:24 smithi055 ceph-mon[97033]: pgmap v9420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T02:30:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:24 smithi059 ceph-mon[97450]: pgmap v9420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T02:30:26.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:30:25.932+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:30:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:25.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:25.780256+0000) 2022-09-08T02:30:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:25.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:25.724370+0000) 2022-09-08T02:30:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:26.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:26.780484+0000) 2022-09-08T02:30:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:26.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:26.724538+0000) 2022-09-08T02:30:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:26 smithi055 ceph-mon[97033]: pgmap v9421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:30:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:26 smithi059 ceph-mon[97450]: pgmap v9421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:30:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:27.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:27.780707+0000) 2022-09-08T02:30:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:27.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:27.724759+0000) 2022-09-08T02:30:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:28.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:28.780913+0000) 2022-09-08T02:30:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:28.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:28.725008+0000) 2022-09-08T02:30:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:28 smithi055 ceph-mon[97033]: pgmap v9422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:28 smithi059 ceph-mon[97450]: pgmap v9422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:29.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:29.781058+0000) 2022-09-08T02:30:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:29.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:29.725171+0000) 2022-09-08T02:30:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:30.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:30.725386+0000) 2022-09-08T02:30:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:30.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:30.781289+0000) 2022-09-08T02:30:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:30 smithi055 ceph-mon[97033]: pgmap v9423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:30 smithi059 ceph-mon[97450]: pgmap v9423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:31.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:31.725630+0000) 2022-09-08T02:30:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:31.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:31.781509+0000) 2022-09-08T02:30:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:31 smithi055 ceph-mon[97033]: pgmap v9424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:30:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:31 smithi059 ceph-mon[97450]: pgmap v9424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:30:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:32.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:32.781755+0000) 2022-09-08T02:30:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:32.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:32.725860+0000) 2022-09-08T02:30:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:33.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:33.781962+0000) 2022-09-08T02:30:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:33.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:33.726077+0000) 2022-09-08T02:30:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:34.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:34.782119+0000) 2022-09-08T02:30:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:34.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:34.726312+0000) 2022-09-08T02:30:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:34 smithi055 ceph-mon[97033]: pgmap v9425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:35.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:34 smithi059 ceph-mon[97450]: pgmap v9425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:35.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:35.782335+0000) 2022-09-08T02:30:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:35.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:35.726575+0000) 2022-09-08T02:30:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:36.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:36.782809+0000) 2022-09-08T02:30:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:36.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:36.726828+0000) 2022-09-08T02:30:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:36 smithi055 ceph-mon[97033]: pgmap v9426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:30:37.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:36 smithi059 ceph-mon[97450]: pgmap v9426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:30:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:37.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:37.782986+0000) 2022-09-08T02:30:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:37.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:37.727033+0000) 2022-09-08T02:30:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:38.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:38.783163+0000) 2022-09-08T02:30:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:38.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:38.727249+0000) 2022-09-08T02:30:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:38 smithi055 ceph-mon[97033]: pgmap v9427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:38 smithi059 ceph-mon[97450]: pgmap v9427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:39.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:39.783377+0000) 2022-09-08T02:30:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:39.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:39.727458+0000) 2022-09-08T02:30:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:40.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:40.783580+0000) 2022-09-08T02:30:41.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:30:40.943+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:30:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:40.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:40.727699+0000) 2022-09-08T02:30:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:40 smithi055 ceph-mon[97033]: pgmap v9428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:40 smithi059 ceph-mon[97450]: pgmap v9428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:41.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:41.783789+0000) 2022-09-08T02:30:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:41.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:41.727967+0000) 2022-09-08T02:30:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:42 smithi055 ceph-mon[97033]: pgmap v9429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:30:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:42 smithi055 ceph-mon[97033]: pgmap v9430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:30:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:42 smithi055 ceph-mon[97033]: pgmap v9431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:30:42.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:30:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:42 smithi059 ceph-mon[97450]: pgmap v9429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:30:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:42 smithi059 ceph-mon[97450]: pgmap v9430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:30:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:42 smithi059 ceph-mon[97450]: pgmap v9431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:30:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:30:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:42.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:42.784063+0000) 2022-09-08T02:30:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:42.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:42.728188+0000) 2022-09-08T02:30:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:43.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:43.784284+0000) 2022-09-08T02:30:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:43.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:43.728431+0000) 2022-09-08T02:30:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:44.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:44.784487+0000) 2022-09-08T02:30:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:44.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:44.728694+0000) 2022-09-08T02:30:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:44 smithi055 ceph-mon[97033]: pgmap v9432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:30:45.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:44 smithi059 ceph-mon[97450]: pgmap v9432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:30:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:45.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:45.784755+0000) 2022-09-08T02:30:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:45.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:45.728839+0000) 2022-09-08T02:30:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:46.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:46.784982+0000) 2022-09-08T02:30:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:46.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:46.728988+0000) 2022-09-08T02:30:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:46 smithi055 ceph-mon[97033]: pgmap v9433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:30:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:46 smithi059 ceph-mon[97450]: pgmap v9433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:30:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:47.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:47.785225+0000) 2022-09-08T02:30:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:47.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:47.729254+0000) 2022-09-08T02:30:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:48.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:48.785449+0000) 2022-09-08T02:30:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:48.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:48.729538+0000) 2022-09-08T02:30:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:48 smithi055 ceph-mon[97033]: pgmap v9434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:30:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:48 smithi059 ceph-mon[97450]: pgmap v9434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:30:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:49.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:49.785671+0000) 2022-09-08T02:30:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:49.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:49.729742+0000) 2022-09-08T02:30:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:50.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:50.785884+0000) 2022-09-08T02:30:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:50.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:50.729914+0000) 2022-09-08T02:30:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:50 smithi055 ceph-mon[97033]: pgmap v9435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:30:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:50 smithi059 ceph-mon[97450]: pgmap v9435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:30:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:51.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:51.730118+0000) 2022-09-08T02:30:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:51.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:51.786100+0000) 2022-09-08T02:30:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:52.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:52.786302+0000) 2022-09-08T02:30:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:52.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:52.730308+0000) 2022-09-08T02:30:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:52 smithi055 ceph-mon[97033]: pgmap v9436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:30:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:52 smithi059 ceph-mon[97450]: pgmap v9436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:30:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:53.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:53.786546+0000) 2022-09-08T02:30:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:53.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:53.730523+0000) 2022-09-08T02:30:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:54.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:54.786754+0000) 2022-09-08T02:30:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:54.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:54.730735+0000) 2022-09-08T02:30:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:54 smithi055 ceph-mon[97033]: pgmap v9437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:54 smithi059 ceph-mon[97450]: pgmap v9437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:56.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:30:55.945+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:30:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:55.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:55.786986+0000) 2022-09-08T02:30:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:55.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:55.730980+0000) 2022-09-08T02:30:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:56.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:56.787188+0000) 2022-09-08T02:30:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:56.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:56.731222+0000) 2022-09-08T02:30:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:56 smithi055 ceph-mon[97033]: pgmap v9438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:30:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:56 smithi059 ceph-mon[97450]: pgmap v9438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:30:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:57.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:57.787439+0000) 2022-09-08T02:30:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:57.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:57.731424+0000) 2022-09-08T02:30:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:58.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:58.787712+0000) 2022-09-08T02:30:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:58.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:58.731621+0000) 2022-09-08T02:30:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:30:58 smithi055 ceph-mon[97033]: pgmap v9439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:30:59.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:30:58 smithi059 ceph-mon[97450]: pgmap v9439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:30:59.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:59.787944+0000) 2022-09-08T02:31:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:30:59.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:30:59.731888+0000) 2022-09-08T02:31:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:00.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:00.788204+0000) 2022-09-08T02:31:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:00.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:00.732154+0000) 2022-09-08T02:31:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:00 smithi055 ceph-mon[97033]: pgmap v9440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:00 smithi059 ceph-mon[97450]: pgmap v9440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:01.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:01.788486+0000) 2022-09-08T02:31:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:01.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:01.732416+0000) 2022-09-08T02:31:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:02.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:02.788774+0000) 2022-09-08T02:31:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:02.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:02.732572+0000) 2022-09-08T02:31:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:02 smithi055 ceph-mon[97033]: pgmap v9441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:31:03.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:02 smithi059 ceph-mon[97450]: pgmap v9441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:31:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:03.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:03.788935+0000) 2022-09-08T02:31:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:03.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:03.732757+0000) 2022-09-08T02:31:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:04.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:04.789078+0000) 2022-09-08T02:31:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:04.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:04.732943+0000) 2022-09-08T02:31:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:04 smithi055 ceph-mon[97033]: pgmap v9442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:05.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:04 smithi059 ceph-mon[97450]: pgmap v9442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:05.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:05.733184+0000) 2022-09-08T02:31:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:05.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:05.789262+0000) 2022-09-08T02:31:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:31:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:31:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:31:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:31:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:06.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:06.789477+0000) 2022-09-08T02:31:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:06.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:06.733410+0000) 2022-09-08T02:31:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:06 smithi055 ceph-mon[97033]: pgmap v9443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:31:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:06 smithi059 ceph-mon[97450]: pgmap v9443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:31:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:07.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:07.789737+0000) 2022-09-08T02:31:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:07.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:07.733677+0000) 2022-09-08T02:31:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:08.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:08.789976+0000) 2022-09-08T02:31:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:08.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:08.733949+0000) 2022-09-08T02:31:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:08 smithi055 ceph-mon[97033]: pgmap v9444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:31:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:08 smithi059 ceph-mon[97450]: pgmap v9444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:31:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:09.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:09.790140+0000) 2022-09-08T02:31:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:09.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:09.734152+0000) 2022-09-08T02:31:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:09 smithi055 ceph-mon[97033]: osdmap e671: 12 total, 11 up, 11 in 2022-09-08T02:31:10.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:09 smithi059 ceph-mon[97450]: osdmap e671: 12 total, 11 up, 11 in 2022-09-08T02:31:11.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:31:10.961+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:31:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:10.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:10.790291+0000) 2022-09-08T02:31:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:10.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:10.734381+0000) 2022-09-08T02:31:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:10 smithi055 ceph-mon[97033]: pgmap v9446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:31:11.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:10 smithi059 ceph-mon[97450]: pgmap v9446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:31:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:11.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:11.790563+0000) 2022-09-08T02:31:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:11.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:11.734603+0000) 2022-09-08T02:31:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:12.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:12.790808+0000) 2022-09-08T02:31:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:12.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:12.734846+0000) 2022-09-08T02:31:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:12 smithi055 ceph-mon[97033]: pgmap v9447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:31:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:12 smithi055 ceph-mon[97033]: osdmap e672: 12 total, 11 up, 11 in 2022-09-08T02:31:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:12 smithi059 ceph-mon[97450]: pgmap v9447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:31:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:12 smithi059 ceph-mon[97450]: osdmap e672: 12 total, 11 up, 11 in 2022-09-08T02:31:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:13.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:13.791053+0000) 2022-09-08T02:31:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:13.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:13.735064+0000) 2022-09-08T02:31:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:14.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:14.791303+0000) 2022-09-08T02:31:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:14.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:14.735322+0000) 2022-09-08T02:31:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:14 smithi055 ceph-mon[97033]: pgmap v9449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:31:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:14 smithi059 ceph-mon[97450]: pgmap v9449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:31:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:15.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:15.791534+0000) 2022-09-08T02:31:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:15.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:15.735546+0000) 2022-09-08T02:31:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:16.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:16.791807+0000) 2022-09-08T02:31:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:16.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:16.735811+0000) 2022-09-08T02:31:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:16 smithi055 ceph-mon[97033]: pgmap v9450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:31:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:16 smithi059 ceph-mon[97450]: pgmap v9450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:31:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:17.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:17.791970+0000) 2022-09-08T02:31:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:17.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:17.736054+0000) 2022-09-08T02:31:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:18.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:18.792191+0000) 2022-09-08T02:31:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:18.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:18.736251+0000) 2022-09-08T02:31:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:18 smithi055 ceph-mon[97033]: pgmap v9451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 244 B/s wr, 7 op/s 2022-09-08T02:31:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:18 smithi059 ceph-mon[97450]: pgmap v9451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 244 B/s wr, 7 op/s 2022-09-08T02:31:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:19.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:19.792422+0000) 2022-09-08T02:31:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:19.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:19.736464+0000) 2022-09-08T02:31:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:20.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:20.792698+0000) 2022-09-08T02:31:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:20.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:20.736687+0000) 2022-09-08T02:31:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:20 smithi055 ceph-mon[97033]: pgmap v9452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:31:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:20 smithi059 ceph-mon[97450]: pgmap v9452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:31:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:21.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:21.792861+0000) 2022-09-08T02:31:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:21.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:21.736944+0000) 2022-09-08T02:31:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:22.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:22.793070+0000) 2022-09-08T02:31:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:22.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:22.737166+0000) 2022-09-08T02:31:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:22 smithi055 ceph-mon[97033]: pgmap v9453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:31:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:22 smithi059 ceph-mon[97450]: pgmap v9453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:31:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:23.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:23.793277+0000) 2022-09-08T02:31:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:23.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:23.737362+0000) 2022-09-08T02:31:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:23 smithi055 ceph-mon[97033]: pgmap v9454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T02:31:24.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:23 smithi059 ceph-mon[97450]: pgmap v9454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T02:31:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:24.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:24.793504+0000) 2022-09-08T02:31:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:24.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:24.737587+0000) 2022-09-08T02:31:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:25.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:25.793740+0000) 2022-09-08T02:31:26.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:31:25.970+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:31:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:25.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:25.737781+0000) 2022-09-08T02:31:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:26 smithi059 ceph-mon[97450]: pgmap v9455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:31:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:26 smithi055 ceph-mon[97033]: pgmap v9455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:31:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:26.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:26.793947+0000) 2022-09-08T02:31:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:26.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:26.738019+0000) 2022-09-08T02:31:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:27.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:27.794151+0000) 2022-09-08T02:31:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:27.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:27.738287+0000) 2022-09-08T02:31:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:28.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:28.794377+0000) 2022-09-08T02:31:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:28.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:28.738505+0000) 2022-09-08T02:31:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:28 smithi055 ceph-mon[97033]: pgmap v9456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:29.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:28 smithi059 ceph-mon[97450]: pgmap v9456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:29.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:29.794590+0000) 2022-09-08T02:31:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:29.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:29.738756+0000) 2022-09-08T02:31:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:30.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:30.794821+0000) 2022-09-08T02:31:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:30.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:30.738964+0000) 2022-09-08T02:31:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:30 smithi055 ceph-mon[97033]: pgmap v9457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:30 smithi059 ceph-mon[97450]: pgmap v9457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:31.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:31.795048+0000) 2022-09-08T02:31:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:31.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:31.739219+0000) 2022-09-08T02:31:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:32.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:32.795228+0000) 2022-09-08T02:31:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:32.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:32.739406+0000) 2022-09-08T02:31:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:32 smithi055 ceph-mon[97033]: pgmap v9458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:31:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:32 smithi059 ceph-mon[97450]: pgmap v9458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:31:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:33.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:33.795437+0000) 2022-09-08T02:31:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:33.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:33.739551+0000) 2022-09-08T02:31:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:34.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:34.795593+0000) 2022-09-08T02:31:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:34.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:34.739705+0000) 2022-09-08T02:31:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:34 smithi055 ceph-mon[97033]: pgmap v9459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:34 smithi059 ceph-mon[97450]: pgmap v9459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:35.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:35.795727+0000) 2022-09-08T02:31:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:35.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:35.739874+0000) 2022-09-08T02:31:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:36.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:36.795951+0000) 2022-09-08T02:31:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:36.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:36.740109+0000) 2022-09-08T02:31:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:36 smithi055 ceph-mon[97033]: pgmap v9460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:31:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:36 smithi059 ceph-mon[97450]: pgmap v9460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:31:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:37.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:37.796190+0000) 2022-09-08T02:31:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:37.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:37.740291+0000) 2022-09-08T02:31:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:38.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:38.796390+0000) 2022-09-08T02:31:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:38.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:38.740485+0000) 2022-09-08T02:31:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:38 smithi055 ceph-mon[97033]: pgmap v9461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:31:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:38 smithi059 ceph-mon[97450]: pgmap v9461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:31:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:39.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:39.796626+0000) 2022-09-08T02:31:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:39.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:39.740747+0000) 2022-09-08T02:31:41.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:31:40.987+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:31:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:40.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:40.796831+0000) 2022-09-08T02:31:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:40.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:40.740988+0000) 2022-09-08T02:31:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:40 smithi055 ceph-mon[97033]: pgmap v9462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:31:41.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:40 smithi059 ceph-mon[97450]: pgmap v9462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:31:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:41.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:41.796977+0000) 2022-09-08T02:31:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:41.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:41.741240+0000) 2022-09-08T02:31:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:31:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:31:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:31:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:31:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:31:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:31:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:42.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:42.797214+0000) 2022-09-08T02:31:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:42.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:42.741482+0000) 2022-09-08T02:31:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:42 smithi055 ceph-mon[97033]: pgmap v9463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:31:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:42 smithi059 ceph-mon[97450]: pgmap v9463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:31:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:43.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:43.797462+0000) 2022-09-08T02:31:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:43.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:43.741751+0000) 2022-09-08T02:31:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:44.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:44.797774+0000) 2022-09-08T02:31:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:44.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:44.741877+0000) 2022-09-08T02:31:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:44 smithi055 ceph-mon[97033]: pgmap v9464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:31:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:44 smithi059 ceph-mon[97450]: pgmap v9464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:31:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:45.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:45.742127+0000) 2022-09-08T02:31:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:45.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:45.798057+0000) 2022-09-08T02:31:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:46.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:46.798351+0000) 2022-09-08T02:31:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:46.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:46.742293+0000) 2022-09-08T02:31:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:46 smithi055 ceph-mon[97033]: pgmap v9465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:31:47.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:46 smithi059 ceph-mon[97450]: pgmap v9465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:31:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:47.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:47.798601+0000) 2022-09-08T02:31:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:47.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:47.742540+0000) 2022-09-08T02:31:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:48.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:48.798941+0000) 2022-09-08T02:31:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:48.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:48.742756+0000) 2022-09-08T02:31:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:48 smithi055 ceph-mon[97033]: pgmap v9466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:48 smithi059 ceph-mon[97450]: pgmap v9466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:49.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:49.799221+0000) 2022-09-08T02:31:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:49.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:49.742996+0000) 2022-09-08T02:31:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:50.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:50.743199+0000) 2022-09-08T02:31:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:50.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:50.799450+0000) 2022-09-08T02:31:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:50 smithi055 ceph-mon[97033]: pgmap v9467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:50 smithi059 ceph-mon[97450]: pgmap v9467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:51.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:51.743427+0000) 2022-09-08T02:31:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:51.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:51.799697+0000) 2022-09-08T02:31:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:52.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:52.799868+0000) 2022-09-08T02:31:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:52.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:52.743611+0000) 2022-09-08T02:31:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:52 smithi055 ceph-mon[97033]: pgmap v9468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:31:53.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:52 smithi059 ceph-mon[97450]: pgmap v9468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:31:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:53.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:53.800088+0000) 2022-09-08T02:31:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:53.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:53.743800+0000) 2022-09-08T02:31:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:54.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:54.800473+0000) 2022-09-08T02:31:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:54.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:54.743996+0000) 2022-09-08T02:31:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:54 smithi055 ceph-mon[97033]: pgmap v9469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:55.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:54 smithi059 ceph-mon[97450]: pgmap v9469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:55.997 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:55.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:55.800758+0000) 2022-09-08T02:31:55.998 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:55.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:55.744196+0000) 2022-09-08T02:31:56.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:31:55.996+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:31:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:56.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:56.800999+0000) 2022-09-08T02:31:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:56.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:56.744424+0000) 2022-09-08T02:31:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:56 smithi055 ceph-mon[97033]: pgmap v9470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:31:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:56 smithi059 ceph-mon[97450]: pgmap v9470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:31:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:57.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:57.801233+0000) 2022-09-08T02:31:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:57.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:57.744720+0000) 2022-09-08T02:31:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:58.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:58.801498+0000) 2022-09-08T02:31:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:58.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:58.744965+0000) 2022-09-08T02:31:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:31:58 smithi055 ceph-mon[97033]: pgmap v9471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:31:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:31:58 smithi059 ceph-mon[97450]: pgmap v9471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:31:59.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:59.745193+0000) 2022-09-08T02:32:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:31:59.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:31:59.801724+0000) 2022-09-08T02:32:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:00.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:00.801951+0000) 2022-09-08T02:32:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:00.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:00.745433+0000) 2022-09-08T02:32:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:00 smithi055 ceph-mon[97033]: pgmap v9472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:00 smithi059 ceph-mon[97450]: pgmap v9472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:01.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:01.802181+0000) 2022-09-08T02:32:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:01.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:01.745714+0000) 2022-09-08T02:32:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:02.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:02.802366+0000) 2022-09-08T02:32:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:02.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:02.745991+0000) 2022-09-08T02:32:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:02 smithi055 ceph-mon[97033]: pgmap v9473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:32:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:02 smithi059 ceph-mon[97450]: pgmap v9473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:32:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:03.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:03.802568+0000) 2022-09-08T02:32:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:03.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:03.746200+0000) 2022-09-08T02:32:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:04.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:04.802809+0000) 2022-09-08T02:32:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:04.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:04.746453+0000) 2022-09-08T02:32:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:04 smithi055 ceph-mon[97033]: pgmap v9474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:04 smithi059 ceph-mon[97450]: pgmap v9474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:05.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:05.803007+0000) 2022-09-08T02:32:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:05.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:05.746677+0000) 2022-09-08T02:32:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:32:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:32:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:32:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:32:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:06.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:06.803212+0000) 2022-09-08T02:32:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:06.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:06.746878+0000) 2022-09-08T02:32:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:06 smithi055 ceph-mon[97033]: pgmap v9475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:32:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:06 smithi059 ceph-mon[97450]: pgmap v9475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:32:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:07.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:07.803462+0000) 2022-09-08T02:32:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:07.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:07.747006+0000) 2022-09-08T02:32:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:08.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:08.803718+0000) 2022-09-08T02:32:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:08.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:08.747211+0000) 2022-09-08T02:32:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:08 smithi055 ceph-mon[97033]: pgmap v9476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:32:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:08 smithi059 ceph-mon[97450]: pgmap v9476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:32:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:09.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:09.803888+0000) 2022-09-08T02:32:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:09.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:09.747378+0000) 2022-09-08T02:32:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:09 smithi055 ceph-mon[97033]: osdmap e673: 12 total, 11 up, 11 in 2022-09-08T02:32:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:09 smithi055 ceph-mon[97033]: pgmap v9478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:32:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:09 smithi059 ceph-mon[97450]: osdmap e673: 12 total, 11 up, 11 in 2022-09-08T02:32:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:09 smithi059 ceph-mon[97450]: pgmap v9478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:32:11.011 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:10.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:10.804090+0000) 2022-09-08T02:32:11.012 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:10.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:10.747604+0000) 2022-09-08T02:32:11.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:32:11.009+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:32:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:11.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:11.804334+0000) 2022-09-08T02:32:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:11.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:11.747852+0000) 2022-09-08T02:32:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:12 smithi059 ceph-mon[97450]: pgmap v9479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:32:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:12 smithi059 ceph-mon[97450]: pgmap v9480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T02:32:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:32:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:12 smithi055 ceph-mon[97033]: pgmap v9479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:32:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:12 smithi055 ceph-mon[97033]: pgmap v9480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T02:32:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:32:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:12.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:12.804560+0000) 2022-09-08T02:32:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:12.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:12.748055+0000) 2022-09-08T02:32:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:13.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:13.804799+0000) 2022-09-08T02:32:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:13.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:13.748296+0000) 2022-09-08T02:32:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:14.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:14.804986+0000) 2022-09-08T02:32:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:14.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:14.748425+0000) 2022-09-08T02:32:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:14 smithi055 ceph-mon[97033]: pgmap v9481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 123 B/s wr, 3 op/s 2022-09-08T02:32:15.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:14 smithi059 ceph-mon[97450]: pgmap v9481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 123 B/s wr, 3 op/s 2022-09-08T02:32:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:15.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:15.748717+0000) 2022-09-08T02:32:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:15.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:15.805239+0000) 2022-09-08T02:32:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:16.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:16.805489+0000) 2022-09-08T02:32:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:16.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:16.749005+0000) 2022-09-08T02:32:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:16 smithi055 ceph-mon[97033]: pgmap v9482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 247 B/s wr, 7 op/s 2022-09-08T02:32:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:16 smithi059 ceph-mon[97450]: pgmap v9482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 247 B/s wr, 7 op/s 2022-09-08T02:32:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:17.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:17.749249+0000) 2022-09-08T02:32:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:17.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:17.805747+0000) 2022-09-08T02:32:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:18.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:18.805923+0000) 2022-09-08T02:32:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:18.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:18.749472+0000) 2022-09-08T02:32:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:18 smithi055 ceph-mon[97033]: pgmap v9483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 238 B/s wr, 7 op/s 2022-09-08T02:32:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:18 smithi055 ceph-mon[97033]: osdmap e674: 12 total, 11 up, 11 in 2022-09-08T02:32:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:18 smithi059 ceph-mon[97450]: pgmap v9483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 238 B/s wr, 7 op/s 2022-09-08T02:32:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:18 smithi059 ceph-mon[97450]: osdmap e674: 12 total, 11 up, 11 in 2022-09-08T02:32:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:19.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:19.806091+0000) 2022-09-08T02:32:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:19.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:19.749763+0000) 2022-09-08T02:32:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:20.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:20.806307+0000) 2022-09-08T02:32:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:20.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:20.749981+0000) 2022-09-08T02:32:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:20 smithi055 ceph-mon[97033]: pgmap v9485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 123 B/s wr, 3 op/s 2022-09-08T02:32:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:20 smithi059 ceph-mon[97450]: pgmap v9485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 123 B/s wr, 3 op/s 2022-09-08T02:32:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:21.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:21.806567+0000) 2022-09-08T02:32:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:21.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:21.750180+0000) 2022-09-08T02:32:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:22.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:22.806795+0000) 2022-09-08T02:32:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:22.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:22.750396+0000) 2022-09-08T02:32:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:22 smithi055 ceph-mon[97033]: pgmap v9486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:32:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:22 smithi059 ceph-mon[97450]: pgmap v9486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:32:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:23.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:23.807035+0000) 2022-09-08T02:32:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:23.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:23.750685+0000) 2022-09-08T02:32:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:24.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:24.750950+0000) 2022-09-08T02:32:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:24.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:24.807274+0000) 2022-09-08T02:32:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:24 smithi055 ceph-mon[97033]: pgmap v9487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:32:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:24 smithi059 ceph-mon[97450]: pgmap v9487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:32:26.020 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:25.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:25.751224+0000) 2022-09-08T02:32:26.021 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:25.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:25.807506+0000) 2022-09-08T02:32:26.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:32:26.020+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:32:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:26.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:26.807725+0000) 2022-09-08T02:32:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:26.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:26.751463+0000) 2022-09-08T02:32:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:26 smithi055 ceph-mon[97033]: pgmap v9488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:32:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:26 smithi059 ceph-mon[97450]: pgmap v9488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:32:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:27.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:27.807851+0000) 2022-09-08T02:32:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:27.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:27.751689+0000) 2022-09-08T02:32:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:28.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:28.808051+0000) 2022-09-08T02:32:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:28.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:28.751948+0000) 2022-09-08T02:32:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:28 smithi055 ceph-mon[97033]: pgmap v9489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:32:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:28 smithi059 ceph-mon[97450]: pgmap v9489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:32:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:29.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:29.808336+0000) 2022-09-08T02:32:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:29.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:29.752134+0000) 2022-09-08T02:32:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:30.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:30.808553+0000) 2022-09-08T02:32:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:30.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:30.752364+0000) 2022-09-08T02:32:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:30 smithi055 ceph-mon[97033]: pgmap v9490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T02:32:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:30 smithi059 ceph-mon[97450]: pgmap v9490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T02:32:31.808 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:31.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:31.752714+0000) 2022-09-08T02:32:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:31.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:31.808736+0000) 2022-09-08T02:32:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:32.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:32.808938+0000) 2022-09-08T02:32:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:32.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:32.752980+0000) 2022-09-08T02:32:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:32 smithi055 ceph-mon[97033]: pgmap v9491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:32:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:32 smithi059 ceph-mon[97450]: pgmap v9491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:32:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:33.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:33.809125+0000) 2022-09-08T02:32:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:33.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:33.753197+0000) 2022-09-08T02:32:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:34.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:34.809343+0000) 2022-09-08T02:32:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:34.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:34.753381+0000) 2022-09-08T02:32:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:34 smithi055 ceph-mon[97033]: pgmap v9492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:34 smithi059 ceph-mon[97450]: pgmap v9492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:35.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:35.809563+0000) 2022-09-08T02:32:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:35.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:35.753574+0000) 2022-09-08T02:32:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:36.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:36.809696+0000) 2022-09-08T02:32:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:36.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:36.753831+0000) 2022-09-08T02:32:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:36 smithi055 ceph-mon[97033]: pgmap v9493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:32:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:36 smithi059 ceph-mon[97450]: pgmap v9493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:32:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:37.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:37.809901+0000) 2022-09-08T02:32:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:37.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:37.754040+0000) 2022-09-08T02:32:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:38.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:38.810059+0000) 2022-09-08T02:32:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:38.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:38.754281+0000) 2022-09-08T02:32:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:38 smithi055 ceph-mon[97033]: pgmap v9494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:38 smithi059 ceph-mon[97450]: pgmap v9494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:39.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:39.810272+0000) 2022-09-08T02:32:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:39.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:39.754529+0000) 2022-09-08T02:32:41.030 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:40.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:40.810418+0000) 2022-09-08T02:32:41.031 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:40.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:40.754828+0000) 2022-09-08T02:32:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:40 smithi055 ceph-mon[97033]: pgmap v9495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:40 smithi059 ceph-mon[97450]: pgmap v9495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:41.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:32:41.030+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:32:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:41.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:41.810563+0000) 2022-09-08T02:32:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:41.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:41.755100+0000) 2022-09-08T02:32:43.149 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:42 smithi059 ceph-mon[97450]: pgmap v9496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:32:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:42.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:42.755314+0000) 2022-09-08T02:32:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:42.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:42.810808+0000) 2022-09-08T02:32:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:42 smithi055 ceph-mon[97033]: pgmap v9496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:32:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:43.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:43.811001+0000) 2022-09-08T02:32:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:43.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:43.755572+0000) 2022-09-08T02:32:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:44.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:44.811245+0000) 2022-09-08T02:32:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:44.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:44.755826+0000) 2022-09-08T02:32:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:44 smithi055 ceph-mon[97033]: pgmap v9497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:44 smithi059 ceph-mon[97450]: pgmap v9497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:45.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:45.811497+0000) 2022-09-08T02:32:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:45.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:45.756024+0000) 2022-09-08T02:32:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:46.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:46.811694+0000) 2022-09-08T02:32:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:46.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:46.756209+0000) 2022-09-08T02:32:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:46 smithi055 ceph-mon[97033]: pgmap v9498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:32:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:46 smithi059 ceph-mon[97450]: pgmap v9498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:32:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:47.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:47.811834+0000) 2022-09-08T02:32:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:47.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:47.756411+0000) 2022-09-08T02:32:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:48.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:48.811985+0000) 2022-09-08T02:32:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:48.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:48.756577+0000) 2022-09-08T02:32:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:48 smithi055 ceph-mon[97033]: pgmap v9499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:48 smithi059 ceph-mon[97450]: pgmap v9499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:49.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:49.812141+0000) 2022-09-08T02:32:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:49.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:49.756825+0000) 2022-09-08T02:32:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:50.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:50.812343+0000) 2022-09-08T02:32:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:50.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:50.756988+0000) 2022-09-08T02:32:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:50 smithi055 ceph-mon[97033]: pgmap v9500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:50 smithi059 ceph-mon[97450]: pgmap v9500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:51.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:51.812568+0000) 2022-09-08T02:32:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:51.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:51.757221+0000) 2022-09-08T02:32:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:52.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:52.812758+0000) 2022-09-08T02:32:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:52.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:52.757491+0000) 2022-09-08T02:32:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:52 smithi055 ceph-mon[97033]: pgmap v9501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:32:53.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:52 smithi059 ceph-mon[97450]: pgmap v9501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:32:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:53.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:53.813012+0000) 2022-09-08T02:32:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:53.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:53.757772+0000) 2022-09-08T02:32:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:54.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:54.813193+0000) 2022-09-08T02:32:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:54.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:54.757934+0000) 2022-09-08T02:32:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:54 smithi055 ceph-mon[97033]: pgmap v9502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:54 smithi059 ceph-mon[97450]: pgmap v9502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:56.032 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:55.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:55.813418+0000) 2022-09-08T02:32:56.034 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:55.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:55.758105+0000) 2022-09-08T02:32:56.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:32:56.030+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:32:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:56 smithi059 ceph-mon[97450]: pgmap v9503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:32:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:56.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:56.813690+0000) 2022-09-08T02:32:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:56.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:56.758358+0000) 2022-09-08T02:32:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:56 smithi055 ceph-mon[97033]: pgmap v9503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:32:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:57.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:57.758561+0000) 2022-09-08T02:32:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:57.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:57.813878+0000) 2022-09-08T02:32:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:58.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:58.814034+0000) 2022-09-08T02:32:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:58.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:58.758746+0000) 2022-09-08T02:32:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:32:58 smithi055 ceph-mon[97033]: pgmap v9504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:32:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:32:58 smithi059 ceph-mon[97450]: pgmap v9504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:32:59.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:59.814215+0000) 2022-09-08T02:33:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:32:59.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:32:59.758882+0000) 2022-09-08T02:33:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:00.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:00.814359+0000) 2022-09-08T02:33:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:00.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:00.759035+0000) 2022-09-08T02:33:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:00 smithi055 ceph-mon[97033]: pgmap v9505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:01.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:00 smithi059 ceph-mon[97450]: pgmap v9505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:01.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:01.814607+0000) 2022-09-08T02:33:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:01.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:01.759286+0000) 2022-09-08T02:33:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:02.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:02.814806+0000) 2022-09-08T02:33:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:02.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:02.759523+0000) 2022-09-08T02:33:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:02 smithi055 ceph-mon[97033]: pgmap v9506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:33:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:02 smithi059 ceph-mon[97450]: pgmap v9506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:33:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:03.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:03.815001+0000) 2022-09-08T02:33:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:03.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:03.759687+0000) 2022-09-08T02:33:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:03 smithi055 ceph-mon[97033]: pgmap v9507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:04.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:03 smithi059 ceph-mon[97450]: pgmap v9507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:04.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:04.815216+0000) 2022-09-08T02:33:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:04.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:04.759903+0000) 2022-09-08T02:33:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:33:05.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:33:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:05.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:05.815382+0000) 2022-09-08T02:33:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:05.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:05.760137+0000) 2022-09-08T02:33:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:33:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:06 smithi055 ceph-mon[97033]: pgmap v9508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:33:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:33:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:06 smithi059 ceph-mon[97450]: pgmap v9508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:33:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:06.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:06.815603+0000) 2022-09-08T02:33:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:06.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:06.760373+0000) 2022-09-08T02:33:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:07.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:07.815818+0000) 2022-09-08T02:33:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:07.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:07.760687+0000) 2022-09-08T02:33:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:08 smithi059 ceph-mon[97450]: pgmap v9509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:33:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:08.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:08.815995+0000) 2022-09-08T02:33:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:08.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:08.760888+0000) 2022-09-08T02:33:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:08 smithi055 ceph-mon[97033]: pgmap v9509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:33:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:09.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:09.816169+0000) 2022-09-08T02:33:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:09.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:09.761126+0000) 2022-09-08T02:33:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:09 smithi055 ceph-mon[97033]: osdmap e675: 12 total, 11 up, 11 in 2022-09-08T02:33:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:09 smithi059 ceph-mon[97450]: osdmap e675: 12 total, 11 up, 11 in 2022-09-08T02:33:11.046 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:10.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:10.816379+0000) 2022-09-08T02:33:11.047 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:10.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:10.761321+0000) 2022-09-08T02:33:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:10 smithi055 ceph-mon[97033]: pgmap v9511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:33:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:10 smithi059 ceph-mon[97450]: pgmap v9511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:33:11.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:33:11.045+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:33:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:11.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:11.816578+0000) 2022-09-08T02:33:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:11.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:11.761566+0000) 2022-09-08T02:33:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:33:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:33:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:33:12.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:33:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:33:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:33:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:12.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:12.816787+0000) 2022-09-08T02:33:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:12.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:12.761813+0000) 2022-09-08T02:33:13.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:12 smithi055 ceph-mon[97033]: pgmap v9512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:33:13.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:12 smithi055 ceph-mon[97033]: osdmap e676: 12 total, 11 up, 11 in 2022-09-08T02:33:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:12 smithi059 ceph-mon[97450]: pgmap v9512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:33:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:12 smithi059 ceph-mon[97450]: osdmap e676: 12 total, 11 up, 11 in 2022-09-08T02:33:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:13.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:13.816955+0000) 2022-09-08T02:33:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:13.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:13.762069+0000) 2022-09-08T02:33:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:14.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:14.817165+0000) 2022-09-08T02:33:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:14.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:14.762289+0000) 2022-09-08T02:33:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:14 smithi055 ceph-mon[97033]: pgmap v9514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:33:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:14 smithi059 ceph-mon[97450]: pgmap v9514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:33:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:15.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:15.817411+0000) 2022-09-08T02:33:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:15.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:15.762543+0000) 2022-09-08T02:33:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:16.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:16.817654+0000) 2022-09-08T02:33:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:16.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:16.762816+0000) 2022-09-08T02:33:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:16 smithi055 ceph-mon[97033]: pgmap v9515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:33:17.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:16 smithi059 ceph-mon[97450]: pgmap v9515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:33:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:17.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:17.817887+0000) 2022-09-08T02:33:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:17.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:17.763050+0000) 2022-09-08T02:33:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:18.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:18.818040+0000) 2022-09-08T02:33:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:18.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:18.763272+0000) 2022-09-08T02:33:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:18 smithi055 ceph-mon[97033]: pgmap v9516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 235 B/s wr, 7 op/s 2022-09-08T02:33:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:18 smithi059 ceph-mon[97450]: pgmap v9516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 235 B/s wr, 7 op/s 2022-09-08T02:33:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:19.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:19.818229+0000) 2022-09-08T02:33:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:19.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:19.763472+0000) 2022-09-08T02:33:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:20.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:20.818400+0000) 2022-09-08T02:33:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:20.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:20.763721+0000) 2022-09-08T02:33:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:20 smithi055 ceph-mon[97033]: pgmap v9517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:33:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:20 smithi059 ceph-mon[97450]: pgmap v9517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:33:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:21.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:21.818567+0000) 2022-09-08T02:33:22.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:21.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:21.763941+0000) 2022-09-08T02:33:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:22.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:22.818788+0000) 2022-09-08T02:33:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:22.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:22.764128+0000) 2022-09-08T02:33:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:22 smithi055 ceph-mon[97033]: pgmap v9518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:33:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:22 smithi059 ceph-mon[97450]: pgmap v9518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:33:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:23.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:23.818908+0000) 2022-09-08T02:33:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:23.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:23.764379+0000) 2022-09-08T02:33:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:24.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:24.819086+0000) 2022-09-08T02:33:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:24.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:24.764631+0000) 2022-09-08T02:33:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:24 smithi055 ceph-mon[97033]: pgmap v9519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 95 B/s wr, 5 op/s 2022-09-08T02:33:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:24 smithi059 ceph-mon[97450]: pgmap v9519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 95 B/s wr, 5 op/s 2022-09-08T02:33:26.060 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:25.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:25.819305+0000) 2022-09-08T02:33:26.061 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:25.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:25.764924+0000) 2022-09-08T02:33:26.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:33:26.059+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:33:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:26.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:26.819494+0000) 2022-09-08T02:33:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:26.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:26.765168+0000) 2022-09-08T02:33:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:26 smithi055 ceph-mon[97033]: pgmap v9520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:33:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:26 smithi059 ceph-mon[97450]: pgmap v9520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:33:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:27.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:27.765366+0000) 2022-09-08T02:33:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:27.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:27.819757+0000) 2022-09-08T02:33:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:28.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:28.819911+0000) 2022-09-08T02:33:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:28.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:28.765599+0000) 2022-09-08T02:33:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:28 smithi055 ceph-mon[97033]: pgmap v9521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:28 smithi059 ceph-mon[97450]: pgmap v9521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:29.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:29.820162+0000) 2022-09-08T02:33:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:29.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:29.765832+0000) 2022-09-08T02:33:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:30.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:30.820374+0000) 2022-09-08T02:33:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:30.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:30.766066+0000) 2022-09-08T02:33:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:30 smithi055 ceph-mon[97033]: pgmap v9522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:30 smithi059 ceph-mon[97450]: pgmap v9522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:31.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:31.820620+0000) 2022-09-08T02:33:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:31.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:31.766231+0000) 2022-09-08T02:33:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:32.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:32.820900+0000) 2022-09-08T02:33:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:32.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:32.766425+0000) 2022-09-08T02:33:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:32 smithi055 ceph-mon[97033]: pgmap v9523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:33:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:32 smithi059 ceph-mon[97450]: pgmap v9523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:33:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:33.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:33.766722+0000) 2022-09-08T02:33:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:33.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:33.821093+0000) 2022-09-08T02:33:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:34.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:34.766902+0000) 2022-09-08T02:33:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:34.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:34.821289+0000) 2022-09-08T02:33:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:34 smithi055 ceph-mon[97033]: pgmap v9524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:34 smithi059 ceph-mon[97450]: pgmap v9524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:35.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:35.767148+0000) 2022-09-08T02:33:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:35.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:35.821403+0000) 2022-09-08T02:33:37.162 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:36.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:36.821596+0000) 2022-09-08T02:33:37.163 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:36.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:36.767345+0000) 2022-09-08T02:33:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:36 smithi055 ceph-mon[97033]: pgmap v9525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:33:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:36 smithi059 ceph-mon[97450]: pgmap v9525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:33:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:37.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:37.767576+0000) 2022-09-08T02:33:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:37.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:37.821832+0000) 2022-09-08T02:33:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:38.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:38.822009+0000) 2022-09-08T02:33:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:38.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:38.767805+0000) 2022-09-08T02:33:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:38 smithi055 ceph-mon[97033]: pgmap v9526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:38 smithi059 ceph-mon[97450]: pgmap v9526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:39.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:39.822194+0000) 2022-09-08T02:33:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:39.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:39.768040+0000) 2022-09-08T02:33:41.072 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:40.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:40.822416+0000) 2022-09-08T02:33:41.073 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:40.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:40.768249+0000) 2022-09-08T02:33:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:40 smithi055 ceph-mon[97033]: pgmap v9527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:41.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:40 smithi059 ceph-mon[97450]: pgmap v9527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:41.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:33:41.072+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:33:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:41.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:41.768425+0000) 2022-09-08T02:33:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:41.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:41.822689+0000) 2022-09-08T02:33:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:42 smithi059 ceph-mon[97450]: pgmap v9528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:33:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:42 smithi059 ceph-mon[97450]: pgmap v9529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:33:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:42 smithi059 ceph-mon[97450]: pgmap v9530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:33:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:33:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:42.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:42.768707+0000) 2022-09-08T02:33:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:42.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:42.822875+0000) 2022-09-08T02:33:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:42 smithi055 ceph-mon[97033]: pgmap v9528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:33:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:42 smithi055 ceph-mon[97033]: pgmap v9529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:33:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:42 smithi055 ceph-mon[97033]: pgmap v9530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:33:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:33:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:43.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:43.823055+0000) 2022-09-08T02:33:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:43.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:43.768857+0000) 2022-09-08T02:33:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:44.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:44.823271+0000) 2022-09-08T02:33:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:44.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:44.769077+0000) 2022-09-08T02:33:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:44 smithi055 ceph-mon[97033]: pgmap v9531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:33:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:44 smithi059 ceph-mon[97450]: pgmap v9531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:33:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:45.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:45.769251+0000) 2022-09-08T02:33:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:45.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:45.823486+0000) 2022-09-08T02:33:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:46.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:46.823758+0000) 2022-09-08T02:33:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:46.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:46.769491+0000) 2022-09-08T02:33:47.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:46 smithi055 ceph-mon[97033]: pgmap v9532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:33:47.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:46 smithi059 ceph-mon[97450]: pgmap v9532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:33:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:47.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:47.823996+0000) 2022-09-08T02:33:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:47.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:47.769768+0000) 2022-09-08T02:33:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:48.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:48.824146+0000) 2022-09-08T02:33:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:48.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:48.770016+0000) 2022-09-08T02:33:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:48 smithi055 ceph-mon[97033]: pgmap v9533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:33:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:48 smithi059 ceph-mon[97450]: pgmap v9533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:33:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:49.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:49.824330+0000) 2022-09-08T02:33:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:49.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:49.770260+0000) 2022-09-08T02:33:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:50.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:50.824560+0000) 2022-09-08T02:33:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:50.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:50.770469+0000) 2022-09-08T02:33:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:50 smithi055 ceph-mon[97033]: pgmap v9534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:33:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:50 smithi059 ceph-mon[97450]: pgmap v9534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:33:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:51.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:51.824774+0000) 2022-09-08T02:33:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:51.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:51.770726+0000) 2022-09-08T02:33:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:52.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:52.770923+0000) 2022-09-08T02:33:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:52.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:52.824998+0000) 2022-09-08T02:33:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:52 smithi055 ceph-mon[97033]: pgmap v9535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:33:53.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:52 smithi059 ceph-mon[97450]: pgmap v9535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:33:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:53.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:53.771214+0000) 2022-09-08T02:33:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:53.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:53.825182+0000) 2022-09-08T02:33:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:54.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:54.825448+0000) 2022-09-08T02:33:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:54.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:54.771475+0000) 2022-09-08T02:33:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:54 smithi055 ceph-mon[97033]: pgmap v9536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:54 smithi059 ceph-mon[97450]: pgmap v9536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:56.087 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:55.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:55.825673+0000) 2022-09-08T02:33:56.088 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:55.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:55.771628+0000) 2022-09-08T02:33:56.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:33:56.085+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:33:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:56.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:56.825933+0000) 2022-09-08T02:33:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:56.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:56.771866+0000) 2022-09-08T02:33:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:56 smithi055 ceph-mon[97033]: pgmap v9537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:33:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:56 smithi059 ceph-mon[97450]: pgmap v9537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:33:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:57.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:57.772130+0000) 2022-09-08T02:33:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:57.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:57.826128+0000) 2022-09-08T02:33:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:58.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:58.826292+0000) 2022-09-08T02:33:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:58.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:58.772344+0000) 2022-09-08T02:33:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:33:58 smithi055 ceph-mon[97033]: pgmap v9538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:33:59.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:33:58 smithi059 ceph-mon[97450]: pgmap v9538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:33:59.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:59.826478+0000) 2022-09-08T02:34:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:33:59.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:33:59.772572+0000) 2022-09-08T02:34:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:00.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:00.826605+0000) 2022-09-08T02:34:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:00.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:00.772730+0000) 2022-09-08T02:34:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:00 smithi055 ceph-mon[97033]: pgmap v9539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:01.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:00 smithi059 ceph-mon[97450]: pgmap v9539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:01.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:01.826817+0000) 2022-09-08T02:34:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:01.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:01.772873+0000) 2022-09-08T02:34:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:02.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:02.827069+0000) 2022-09-08T02:34:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:02.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:02.773079+0000) 2022-09-08T02:34:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:02 smithi055 ceph-mon[97033]: pgmap v9540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:34:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:02 smithi059 ceph-mon[97450]: pgmap v9540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:34:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:03.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:03.773249+0000) 2022-09-08T02:34:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:03.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:03.827322+0000) 2022-09-08T02:34:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:04.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:04.827530+0000) 2022-09-08T02:34:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:04.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:04.773472+0000) 2022-09-08T02:34:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:04 smithi055 ceph-mon[97033]: pgmap v9541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:05.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:04 smithi059 ceph-mon[97450]: pgmap v9541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:05.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:05.773749+0000) 2022-09-08T02:34:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:05.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:05.827761+0000) 2022-09-08T02:34:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:34:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:34:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:34:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:34:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:06.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:06.827895+0000) 2022-09-08T02:34:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:06.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:06.773901+0000) 2022-09-08T02:34:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:06 smithi055 ceph-mon[97033]: pgmap v9542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:34:07.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:06 smithi059 ceph-mon[97450]: pgmap v9542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:34:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:07.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:07.828085+0000) 2022-09-08T02:34:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:07.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:07.774068+0000) 2022-09-08T02:34:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:08.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:08.828232+0000) 2022-09-08T02:34:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:08.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:08.774281+0000) 2022-09-08T02:34:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:08 smithi055 ceph-mon[97033]: pgmap v9543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:34:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:08 smithi059 ceph-mon[97450]: pgmap v9543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:34:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:09.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:09.774537+0000) 2022-09-08T02:34:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:09.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:09.828398+0000) 2022-09-08T02:34:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:09 smithi055 ceph-mon[97033]: osdmap e677: 12 total, 11 up, 11 in 2022-09-08T02:34:10.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:09 smithi059 ceph-mon[97450]: osdmap e677: 12 total, 11 up, 11 in 2022-09-08T02:34:11.101 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:10.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:10.828528+0000) 2022-09-08T02:34:11.102 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:10.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:10.774754+0000) 2022-09-08T02:34:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:10 smithi055 ceph-mon[97033]: pgmap v9545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:34:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:10 smithi059 ceph-mon[97450]: pgmap v9545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:34:11.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:34:11.101+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:34:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:11.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:11.828772+0000) 2022-09-08T02:34:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:11.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:11.774943+0000) 2022-09-08T02:34:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:11 smithi055 ceph-mon[97033]: pgmap v9546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:34:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:11 smithi059 ceph-mon[97450]: pgmap v9546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:34:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:12.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:12.828998+0000) 2022-09-08T02:34:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:12.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:12.775170+0000) 2022-09-08T02:34:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:13.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:13.829254+0000) 2022-09-08T02:34:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:13.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:13.775368+0000) 2022-09-08T02:34:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:13 smithi055 ceph-mon[97033]: osdmap e678: 12 total, 11 up, 11 in 2022-09-08T02:34:14.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:13 smithi059 ceph-mon[97450]: osdmap e678: 12 total, 11 up, 11 in 2022-09-08T02:34:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:14.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:14.829409+0000) 2022-09-08T02:34:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:14.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:14.775570+0000) 2022-09-08T02:34:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:14 smithi055 ceph-mon[97033]: pgmap v9548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:34:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:14 smithi059 ceph-mon[97450]: pgmap v9548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:34:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:15.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:15.775802+0000) 2022-09-08T02:34:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:15.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:15.829603+0000) 2022-09-08T02:34:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:16.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:16.829772+0000) 2022-09-08T02:34:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:16.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:16.776007+0000) 2022-09-08T02:34:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:16 smithi055 ceph-mon[97033]: pgmap v9549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:34:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:16 smithi059 ceph-mon[97450]: pgmap v9549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:34:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:17.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:17.829972+0000) 2022-09-08T02:34:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:17.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:17.776218+0000) 2022-09-08T02:34:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:18.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:18.830166+0000) 2022-09-08T02:34:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:18.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:18.776467+0000) 2022-09-08T02:34:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:18 smithi055 ceph-mon[97033]: pgmap v9550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T02:34:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:18 smithi059 ceph-mon[97450]: pgmap v9550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T02:34:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:19.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:19.830387+0000) 2022-09-08T02:34:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:19.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:19.776694+0000) 2022-09-08T02:34:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:20.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:20.830592+0000) 2022-09-08T02:34:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:20.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:20.776918+0000) 2022-09-08T02:34:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:20 smithi055 ceph-mon[97033]: pgmap v9551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:34:21.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:20 smithi059 ceph-mon[97450]: pgmap v9551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:34:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:21.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:21.830815+0000) 2022-09-08T02:34:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:21.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:21.777106+0000) 2022-09-08T02:34:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:22.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:22.831050+0000) 2022-09-08T02:34:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:22.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:22.777327+0000) 2022-09-08T02:34:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:22 smithi055 ceph-mon[97033]: pgmap v9552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:34:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:22 smithi059 ceph-mon[97450]: pgmap v9552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:34:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:23.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:23.831289+0000) 2022-09-08T02:34:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:23.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:23.777587+0000) 2022-09-08T02:34:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:24.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:24.831453+0000) 2022-09-08T02:34:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:24.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:24.777867+0000) 2022-09-08T02:34:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:24 smithi055 ceph-mon[97033]: pgmap v9553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T02:34:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:24 smithi059 ceph-mon[97450]: pgmap v9553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T02:34:26.107 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:25.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:25.831769+0000) 2022-09-08T02:34:26.108 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:25.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:25.777995+0000) 2022-09-08T02:34:26.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:34:26.106+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:34:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:26.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:26.831947+0000) 2022-09-08T02:34:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:26.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:26.778164+0000) 2022-09-08T02:34:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:26 smithi055 ceph-mon[97033]: pgmap v9554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:34:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:26 smithi059 ceph-mon[97450]: pgmap v9554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:34:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:27.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:27.832197+0000) 2022-09-08T02:34:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:27.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:27.778377+0000) 2022-09-08T02:34:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:28.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:28.832422+0000) 2022-09-08T02:34:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:28.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:28.778569+0000) 2022-09-08T02:34:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:28 smithi055 ceph-mon[97033]: pgmap v9555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:28 smithi059 ceph-mon[97450]: pgmap v9555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:29.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:29.832686+0000) 2022-09-08T02:34:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:29.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:29.778773+0000) 2022-09-08T02:34:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:30.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:30.832858+0000) 2022-09-08T02:34:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:30.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:30.778934+0000) 2022-09-08T02:34:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:30 smithi055 ceph-mon[97033]: pgmap v9556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:30 smithi059 ceph-mon[97450]: pgmap v9556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:31.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:31.779160+0000) 2022-09-08T02:34:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:31.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:31.833066+0000) 2022-09-08T02:34:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:32.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:32.833266+0000) 2022-09-08T02:34:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:32.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:32.779399+0000) 2022-09-08T02:34:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:32 smithi055 ceph-mon[97033]: pgmap v9557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:34:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:32 smithi059 ceph-mon[97450]: pgmap v9557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:34:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:33.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:33.833513+0000) 2022-09-08T02:34:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:33.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:33.779659+0000) 2022-09-08T02:34:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:34.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:34.833754+0000) 2022-09-08T02:34:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:34.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:34.779906+0000) 2022-09-08T02:34:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:34 smithi055 ceph-mon[97033]: pgmap v9558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:34:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:34 smithi059 ceph-mon[97450]: pgmap v9558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:34:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:35.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:35.833934+0000) 2022-09-08T02:34:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:35.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:35.780147+0000) 2022-09-08T02:34:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:36.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:36.834157+0000) 2022-09-08T02:34:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:36.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:36.780297+0000) 2022-09-08T02:34:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:36 smithi055 ceph-mon[97033]: pgmap v9559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:34:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:36 smithi059 ceph-mon[97450]: pgmap v9559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:34:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:37.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:37.834324+0000) 2022-09-08T02:34:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:37.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:37.780531+0000) 2022-09-08T02:34:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:38.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:38.834624+0000) 2022-09-08T02:34:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:38.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:38.780776+0000) 2022-09-08T02:34:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:38 smithi055 ceph-mon[97033]: pgmap v9560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:34:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:38 smithi059 ceph-mon[97450]: pgmap v9560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:34:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:39.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:39.834794+0000) 2022-09-08T02:34:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:39.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:39.780916+0000) 2022-09-08T02:34:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:39 smithi055 ceph-mon[97033]: pgmap v9561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:34:40.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:39 smithi059 ceph-mon[97450]: pgmap v9561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:34:41.112 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:40.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:40.834957+0000) 2022-09-08T02:34:41.113 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:40.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:40.781103+0000) 2022-09-08T02:34:41.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:34:41.111+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:34:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:41.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:41.835145+0000) 2022-09-08T02:34:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:41.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:41.781340+0000) 2022-09-08T02:34:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:42.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:42.835378+0000) 2022-09-08T02:34:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:42.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:42.781616+0000) 2022-09-08T02:34:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:42 smithi055 ceph-mon[97033]: pgmap v9562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:34:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:34:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:34:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:34:43.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:42 smithi059 ceph-mon[97450]: pgmap v9562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:34:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:34:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:34:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:34:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:43.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:43.835620+0000) 2022-09-08T02:34:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:43.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:43.781908+0000) 2022-09-08T02:34:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:44.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:44.835842+0000) 2022-09-08T02:34:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:44.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:44.782087+0000) 2022-09-08T02:34:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:44 smithi055 ceph-mon[97033]: pgmap v9563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:44 smithi059 ceph-mon[97450]: pgmap v9563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:45.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:45.836037+0000) 2022-09-08T02:34:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:45.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:45.782347+0000) 2022-09-08T02:34:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:46.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:46.836250+0000) 2022-09-08T02:34:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:46.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:46.782587+0000) 2022-09-08T02:34:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:46 smithi055 ceph-mon[97033]: pgmap v9564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:34:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:46 smithi059 ceph-mon[97450]: pgmap v9564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:34:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:47.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:47.836446+0000) 2022-09-08T02:34:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:47.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:47.782845+0000) 2022-09-08T02:34:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:48.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:48.836729+0000) 2022-09-08T02:34:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:48.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:48.783112+0000) 2022-09-08T02:34:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:48 smithi055 ceph-mon[97033]: pgmap v9565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:49.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:48 smithi059 ceph-mon[97450]: pgmap v9565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:49.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:49.836893+0000) 2022-09-08T02:34:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:49.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:49.783324+0000) 2022-09-08T02:34:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:50.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:50.837101+0000) 2022-09-08T02:34:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:50.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:50.783554+0000) 2022-09-08T02:34:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:50 smithi055 ceph-mon[97033]: pgmap v9566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:50 smithi059 ceph-mon[97450]: pgmap v9566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:34:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:51.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:51.837319+0000) 2022-09-08T02:34:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:51.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:51.783792+0000) 2022-09-08T02:34:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:52.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:52.837519+0000) 2022-09-08T02:34:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:52.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:52.784045+0000) 2022-09-08T02:34:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:52 smithi055 ceph-mon[97033]: pgmap v9567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:34:53.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:52 smithi059 ceph-mon[97450]: pgmap v9567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:34:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:53.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:53.837772+0000) 2022-09-08T02:34:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:53.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:53.784323+0000) 2022-09-08T02:34:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:54.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:54.837976+0000) 2022-09-08T02:34:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:54.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:54.784499+0000) 2022-09-08T02:34:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:54 smithi055 ceph-mon[97033]: pgmap v9568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:34:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:54 smithi059 ceph-mon[97450]: pgmap v9568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:34:56.126 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:55.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:55.838132+0000) 2022-09-08T02:34:56.127 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:55.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:55.784753+0000) 2022-09-08T02:34:56.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:34:56.126+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:34:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:56.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:56.838270+0000) 2022-09-08T02:34:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:56.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:56.785004+0000) 2022-09-08T02:34:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:56 smithi055 ceph-mon[97033]: pgmap v9569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:34:57.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:56 smithi059 ceph-mon[97450]: pgmap v9569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:34:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:57.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:57.838467+0000) 2022-09-08T02:34:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:57.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:57.785256+0000) 2022-09-08T02:34:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:58.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:58.838698+0000) 2022-09-08T02:34:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:58.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:58.785504+0000) 2022-09-08T02:34:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:34:58 smithi055 ceph-mon[97033]: pgmap v9570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:34:59.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:34:58 smithi059 ceph-mon[97450]: pgmap v9570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:35:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:34:59.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:59.838912+0000) 2022-09-08T02:35:00.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:34:59.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:34:59.785702+0000) 2022-09-08T02:35:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:00.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:00.839044+0000) 2022-09-08T02:35:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:00.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:00.785911+0000) 2022-09-08T02:35:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:00 smithi055 ceph-mon[97033]: pgmap v9571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:35:01.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:00 smithi059 ceph-mon[97450]: pgmap v9571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:35:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:01.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:01.839255+0000) 2022-09-08T02:35:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:01.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:01.786175+0000) 2022-09-08T02:35:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:02.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:02.839434+0000) 2022-09-08T02:35:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:02.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:02.786407+0000) 2022-09-08T02:35:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:02 smithi055 ceph-mon[97033]: pgmap v9572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:35:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:02 smithi059 ceph-mon[97450]: pgmap v9572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:35:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:03.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:03.839567+0000) 2022-09-08T02:35:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:03.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:03.786590+0000) 2022-09-08T02:35:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:04.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:04.839736+0000) 2022-09-08T02:35:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:04.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:04.786820+0000) 2022-09-08T02:35:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:04 smithi055 ceph-mon[97033]: pgmap v9573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:05.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:04 smithi059 ceph-mon[97450]: pgmap v9573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:05.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:05.839855+0000) 2022-09-08T02:35:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:05.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:05.787014+0000) 2022-09-08T02:35:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:35:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:35:06.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:35:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:35:07.084 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:06.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:06.787146+0000) 2022-09-08T02:35:07.085 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:06.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:06.839983+0000) 2022-09-08T02:35:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:06 smithi055 ceph-mon[97033]: pgmap v9574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:35:07.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:06 smithi059 ceph-mon[97450]: pgmap v9574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:35:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:07.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:07.787352+0000) 2022-09-08T02:35:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:07.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:07.840172+0000) 2022-09-08T02:35:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:08.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:08.787612+0000) 2022-09-08T02:35:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:08.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:08.840339+0000) 2022-09-08T02:35:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:08 smithi055 ceph-mon[97033]: pgmap v9575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:35:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:08 smithi059 ceph-mon[97450]: pgmap v9575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:35:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:09.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:09.840479+0000) 2022-09-08T02:35:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:09.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:09.787705+0000) 2022-09-08T02:35:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:09 smithi055 ceph-mon[97033]: osdmap e679: 12 total, 11 up, 11 in 2022-09-08T02:35:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:09 smithi055 ceph-mon[97033]: pgmap v9577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:35:10.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:09 smithi059 ceph-mon[97450]: osdmap e679: 12 total, 11 up, 11 in 2022-09-08T02:35:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:09 smithi059 ceph-mon[97450]: pgmap v9577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:35:11.135 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:10.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:10.840718+0000) 2022-09-08T02:35:11.136 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:10.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:10.787906+0000) 2022-09-08T02:35:11.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:35:11.134+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:35:12.043 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:11.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:11.840907+0000) 2022-09-08T02:35:12.044 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:11.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:11.788065+0000) 2022-09-08T02:35:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:12.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:12.841031+0000) 2022-09-08T02:35:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:12.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:12.788293+0000) 2022-09-08T02:35:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:12 smithi055 ceph-mon[97033]: pgmap v9578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:35:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:35:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:12 smithi059 ceph-mon[97450]: pgmap v9578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:35:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:35:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:13.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:13.841164+0000) 2022-09-08T02:35:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:13.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:13.788501+0000) 2022-09-08T02:35:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:13 smithi055 ceph-mon[97033]: pgmap v9579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T02:35:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:13 smithi055 ceph-mon[97033]: pgmap v9580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T02:35:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:13 smithi055 ceph-mon[97033]: osdmap e680: 12 total, 11 up, 11 in 2022-09-08T02:35:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:13 smithi059 ceph-mon[97450]: pgmap v9579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T02:35:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:13 smithi059 ceph-mon[97450]: pgmap v9580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T02:35:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:13 smithi059 ceph-mon[97450]: osdmap e680: 12 total, 11 up, 11 in 2022-09-08T02:35:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:14.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:14.788651+0000) 2022-09-08T02:35:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:14.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:14.841341+0000) 2022-09-08T02:35:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:15.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:15.788864+0000) 2022-09-08T02:35:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:15.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:15.841567+0000) 2022-09-08T02:35:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:15 smithi055 ceph-mon[97033]: pgmap v9582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 199 B/s wr, 6 op/s 2022-09-08T02:35:16.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:15 smithi059 ceph-mon[97450]: pgmap v9582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 199 B/s wr, 6 op/s 2022-09-08T02:35:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:16.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:16.789052+0000) 2022-09-08T02:35:17.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:16.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:16.841754+0000) 2022-09-08T02:35:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:17.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:17.841920+0000) 2022-09-08T02:35:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:17.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:17.789353+0000) 2022-09-08T02:35:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:17 smithi055 ceph-mon[97033]: pgmap v9583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.1 KiB/s rd, 327 B/s wr, 10 op/s 2022-09-08T02:35:18.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:17 smithi059 ceph-mon[97450]: pgmap v9583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.1 KiB/s rd, 327 B/s wr, 10 op/s 2022-09-08T02:35:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:18.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:18.842052+0000) 2022-09-08T02:35:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:18.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:18.789629+0000) 2022-09-08T02:35:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:19.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:19.842251+0000) 2022-09-08T02:35:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:19.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:19.789838+0000) 2022-09-08T02:35:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:19 smithi055 ceph-mon[97033]: pgmap v9584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T02:35:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:19 smithi059 ceph-mon[97450]: pgmap v9584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T02:35:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:20.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:20.842470+0000) 2022-09-08T02:35:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:20.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:20.790105+0000) 2022-09-08T02:35:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:21.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:21.842732+0000) 2022-09-08T02:35:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:21.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:21.790337+0000) 2022-09-08T02:35:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:21 smithi055 ceph-mon[97033]: pgmap v9585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T02:35:22.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:21 smithi059 ceph-mon[97450]: pgmap v9585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T02:35:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:22.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:22.842943+0000) 2022-09-08T02:35:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:22.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:22.790579+0000) 2022-09-08T02:35:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:23.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:23.843173+0000) 2022-09-08T02:35:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:23.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:23.790921+0000) 2022-09-08T02:35:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:23 smithi055 ceph-mon[97033]: pgmap v9586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:35:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:23 smithi059 ceph-mon[97450]: pgmap v9586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:35:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:24.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:24.843444+0000) 2022-09-08T02:35:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:24.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:24.791168+0000) 2022-09-08T02:35:26.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:25.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:25.843681+0000) 2022-09-08T02:35:26.149 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:25.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:25.791399+0000) 2022-09-08T02:35:26.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:25 smithi055 ceph-mon[97033]: pgmap v9587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T02:35:26.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:25 smithi059 ceph-mon[97450]: pgmap v9587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T02:35:26.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:35:26.146+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:35:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:26.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:26.843895+0000) 2022-09-08T02:35:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:26.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:26.791594+0000) 2022-09-08T02:35:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:27.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:27.844174+0000) 2022-09-08T02:35:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:27.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:27.791773+0000) 2022-09-08T02:35:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:27 smithi055 ceph-mon[97033]: pgmap v9588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:35:28.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:27 smithi059 ceph-mon[97450]: pgmap v9588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:35:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:28.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:28.844389+0000) 2022-09-08T02:35:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:28.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:28.791977+0000) 2022-09-08T02:35:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:29.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:29.844524+0000) 2022-09-08T02:35:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:29.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:29.792256+0000) 2022-09-08T02:35:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:29 smithi055 ceph-mon[97033]: pgmap v9589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:30.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:29 smithi059 ceph-mon[97450]: pgmap v9589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:30.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:30.844679+0000) 2022-09-08T02:35:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:30.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:30.792535+0000) 2022-09-08T02:35:32.099 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:31.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:31.844939+0000) 2022-09-08T02:35:32.099 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:31.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:31.792791+0000) 2022-09-08T02:35:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:31 smithi055 ceph-mon[97033]: pgmap v9590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:35:32.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:31 smithi059 ceph-mon[97450]: pgmap v9590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:35:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:32.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:32.792985+0000) 2022-09-08T02:35:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:32.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:32.845146+0000) 2022-09-08T02:35:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:33.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:33.845406+0000) 2022-09-08T02:35:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:33.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:33.793210+0000) 2022-09-08T02:35:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:33 smithi055 ceph-mon[97033]: pgmap v9591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:34.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:33 smithi059 ceph-mon[97450]: pgmap v9591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:34.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:34.845633+0000) 2022-09-08T02:35:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:34.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:34.793410+0000) 2022-09-08T02:35:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:35.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:35.845766+0000) 2022-09-08T02:35:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:35.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:35.793624+0000) 2022-09-08T02:35:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:35 smithi055 ceph-mon[97033]: pgmap v9592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:36.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:35 smithi059 ceph-mon[97450]: pgmap v9592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:36.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:36.845960+0000) 2022-09-08T02:35:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:36.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:36.793907+0000) 2022-09-08T02:35:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:37.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:37.846155+0000) 2022-09-08T02:35:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:37.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:37.794170+0000) 2022-09-08T02:35:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:37 smithi055 ceph-mon[97033]: pgmap v9593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:35:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:37 smithi059 ceph-mon[97450]: pgmap v9593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:35:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:38.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:38.846404+0000) 2022-09-08T02:35:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:38.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:38.794438+0000) 2022-09-08T02:35:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:39.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:39.846596+0000) 2022-09-08T02:35:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:39.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:39.794679+0000) 2022-09-08T02:35:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:39 smithi055 ceph-mon[97033]: pgmap v9594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:39 smithi059 ceph-mon[97450]: pgmap v9594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:40.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:40.846766+0000) 2022-09-08T02:35:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:40.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:40.794899+0000) 2022-09-08T02:35:41.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:35:41.163+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:35:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:41.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:41.846937+0000) 2022-09-08T02:35:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:41.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:41.795062+0000) 2022-09-08T02:35:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:41 smithi055 ceph-mon[97033]: pgmap v9595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:35:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:41 smithi059 ceph-mon[97450]: pgmap v9595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:35:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:42.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:42.847171+0000) 2022-09-08T02:35:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:42.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:42.795246+0000) 2022-09-08T02:35:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:43.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:43.847400+0000) 2022-09-08T02:35:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:43.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:43.795457+0000) 2022-09-08T02:35:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:43 smithi055 ceph-mon[97033]: pgmap v9596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:44.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:43 smithi059 ceph-mon[97450]: pgmap v9596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:44.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:44.847615+0000) 2022-09-08T02:35:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:44.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:44.795724+0000) 2022-09-08T02:35:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:44 smithi055 ceph-mon[97033]: pgmap v9597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:44 smithi059 ceph-mon[97450]: pgmap v9597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:45.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:45.847820+0000) 2022-09-08T02:35:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:45.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:45.796016+0000) 2022-09-08T02:35:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:46.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:46.847999+0000) 2022-09-08T02:35:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:46.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:46.796276+0000) 2022-09-08T02:35:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:47 smithi055 ceph-mon[97033]: pgmap v9598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:35:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:47 smithi059 ceph-mon[97450]: pgmap v9598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:35:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:47.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:47.848235+0000) 2022-09-08T02:35:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:47.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:47.796586+0000) 2022-09-08T02:35:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:48.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:48.848490+0000) 2022-09-08T02:35:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:48.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:48.796883+0000) 2022-09-08T02:35:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:49.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:49.848740+0000) 2022-09-08T02:35:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:49.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:49.797046+0000) 2022-09-08T02:35:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:49 smithi055 ceph-mon[97033]: pgmap v9599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:50.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:49 smithi059 ceph-mon[97450]: pgmap v9599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:50.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:50.848876+0000) 2022-09-08T02:35:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:50.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:50.797279+0000) 2022-09-08T02:35:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:51.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:51.849079+0000) 2022-09-08T02:35:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:51.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:51.797480+0000) 2022-09-08T02:35:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:51 smithi055 ceph-mon[97033]: pgmap v9600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:35:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:51 smithi059 ceph-mon[97450]: pgmap v9600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:35:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:52.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:52.849217+0000) 2022-09-08T02:35:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:52.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:52.797756+0000) 2022-09-08T02:35:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:53.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:53.849429+0000) 2022-09-08T02:35:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:53.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:53.797983+0000) 2022-09-08T02:35:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:53 smithi055 ceph-mon[97033]: pgmap v9601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:53 smithi059 ceph-mon[97450]: pgmap v9601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:54.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:54.849610+0000) 2022-09-08T02:35:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:54.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:54.798235+0000) 2022-09-08T02:35:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:55.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:55.849835+0000) 2022-09-08T02:35:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:55.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:55.798486+0000) 2022-09-08T02:35:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:55 smithi055 ceph-mon[97033]: pgmap v9602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:55 smithi059 ceph-mon[97450]: pgmap v9602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:35:56.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:35:56.175+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:35:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:56.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:56.850073+0000) 2022-09-08T02:35:57.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:56.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:56.798659+0000) 2022-09-08T02:35:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:57.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:57.850281+0000) 2022-09-08T02:35:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:57.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:57.798855+0000) 2022-09-08T02:35:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:57 smithi055 ceph-mon[97033]: pgmap v9603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:35:58.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:57 smithi059 ceph-mon[97450]: pgmap v9603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:35:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:58.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:58.850547+0000) 2022-09-08T02:35:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:58.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:58.799129+0000) 2022-09-08T02:36:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:35:59.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:59.850797+0000) 2022-09-08T02:36:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:35:59.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:35:59.799346+0000) 2022-09-08T02:36:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:35:59 smithi055 ceph-mon[97033]: pgmap v9604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:00.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:35:59 smithi059 ceph-mon[97450]: pgmap v9604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:00.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:00.851032+0000) 2022-09-08T02:36:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:00.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:00.799544+0000) 2022-09-08T02:36:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:01.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:01.851291+0000) 2022-09-08T02:36:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:01.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:01.799780+0000) 2022-09-08T02:36:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:01 smithi055 ceph-mon[97033]: pgmap v9605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:36:02.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:01 smithi059 ceph-mon[97450]: pgmap v9605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:36:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:02.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:02.851553+0000) 2022-09-08T02:36:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:02.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:02.799973+0000) 2022-09-08T02:36:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:03.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:03.851793+0000) 2022-09-08T02:36:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:03.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:03.800209+0000) 2022-09-08T02:36:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:03 smithi055 ceph-mon[97033]: pgmap v9606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:03 smithi059 ceph-mon[97450]: pgmap v9606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:04.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:04.851987+0000) 2022-09-08T02:36:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:04.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:04.800451+0000) 2022-09-08T02:36:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:05.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:05.852207+0000) 2022-09-08T02:36:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:05.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:05.800730+0000) 2022-09-08T02:36:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:05 smithi055 ceph-mon[97033]: pgmap v9607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:36:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:36:06.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:05 smithi059 ceph-mon[97450]: pgmap v9607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:36:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:36:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:06.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:06.852377+0000) 2022-09-08T02:36:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:06.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:06.800936+0000) 2022-09-08T02:36:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:07.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:07.852630+0000) 2022-09-08T02:36:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:07.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:07.801189+0000) 2022-09-08T02:36:08.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:07 smithi055 ceph-mon[97033]: pgmap v9608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:36:08.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:07 smithi059 ceph-mon[97450]: pgmap v9608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:36:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:08.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:08.852879+0000) 2022-09-08T02:36:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:08.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:08.801421+0000) 2022-09-08T02:36:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:09.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:09.853040+0000) 2022-09-08T02:36:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:09.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:09.801680+0000) 2022-09-08T02:36:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:09 smithi055 ceph-mon[97033]: pgmap v9609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:36:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:09 smithi055 ceph-mon[97033]: osdmap e681: 12 total, 11 up, 11 in 2022-09-08T02:36:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:09 smithi059 ceph-mon[97450]: pgmap v9609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:36:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:09 smithi059 ceph-mon[97450]: osdmap e681: 12 total, 11 up, 11 in 2022-09-08T02:36:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:10.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:10.853255+0000) 2022-09-08T02:36:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:10.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:10.801916+0000) 2022-09-08T02:36:11.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:36:11.190+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:36:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:11.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:11.853449+0000) 2022-09-08T02:36:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:11.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:11.802108+0000) 2022-09-08T02:36:12.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:11 smithi055 ceph-mon[97033]: pgmap v9611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:36:12.224 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:11 smithi059 ceph-mon[97450]: pgmap v9611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:36:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:12.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:12.853694+0000) 2022-09-08T02:36:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:12.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:12.802323+0000) 2022-09-08T02:36:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:36:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:36:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:36:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:12 smithi055 ceph-mon[97033]: osdmap e682: 12 total, 11 up, 11 in 2022-09-08T02:36:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:36:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:36:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:36:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:12 smithi059 ceph-mon[97450]: osdmap e682: 12 total, 11 up, 11 in 2022-09-08T02:36:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:13.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:13.853905+0000) 2022-09-08T02:36:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:13.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:13.802537+0000) 2022-09-08T02:36:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:13 smithi055 ceph-mon[97033]: pgmap v9612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:36:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:13 smithi059 ceph-mon[97450]: pgmap v9612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:36:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:14.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:14.854098+0000) 2022-09-08T02:36:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:14.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:14.802766+0000) 2022-09-08T02:36:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:14 smithi055 ceph-mon[97033]: pgmap v9614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:36:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:14 smithi059 ceph-mon[97450]: pgmap v9614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:36:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:15.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:15.854271+0000) 2022-09-08T02:36:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:15.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:15.802984+0000) 2022-09-08T02:36:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:16.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:16.854504+0000) 2022-09-08T02:36:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:16.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:16.803138+0000) 2022-09-08T02:36:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:17 smithi055 ceph-mon[97033]: pgmap v9615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:36:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:17 smithi059 ceph-mon[97450]: pgmap v9615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:36:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:17.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:17.854749+0000) 2022-09-08T02:36:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:17.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:17.803383+0000) 2022-09-08T02:36:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:18.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:18.854954+0000) 2022-09-08T02:36:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:18.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:18.803588+0000) 2022-09-08T02:36:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:19.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:19.855198+0000) 2022-09-08T02:36:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:19.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:19.803807+0000) 2022-09-08T02:36:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:19 smithi055 ceph-mon[97033]: pgmap v9616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2022-09-08T02:36:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:19 smithi059 ceph-mon[97450]: pgmap v9616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2022-09-08T02:36:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:20.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:20.855422+0000) 2022-09-08T02:36:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:20.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:20.803995+0000) 2022-09-08T02:36:21.858 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:21.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:21.804264+0000) 2022-09-08T02:36:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:21 smithi055 ceph-mon[97033]: pgmap v9617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:36:22.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:21 smithi059 ceph-mon[97450]: pgmap v9617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:36:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:22.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:22.804488+0000) 2022-09-08T02:36:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:21.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:21.855691+0000) 2022-09-08T02:36:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:22.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:22.855948+0000) 2022-09-08T02:36:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:23.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:23.856189+0000) 2022-09-08T02:36:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:23.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:23.804733+0000) 2022-09-08T02:36:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:23 smithi055 ceph-mon[97033]: pgmap v9618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:36:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:23 smithi059 ceph-mon[97450]: pgmap v9618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:36:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:24.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:24.856408+0000) 2022-09-08T02:36:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:24.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:24.804903+0000) 2022-09-08T02:36:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:25.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:25.856666+0000) 2022-09-08T02:36:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:25.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:25.805145+0000) 2022-09-08T02:36:26.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:25 smithi055 ceph-mon[97033]: pgmap v9619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T02:36:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:25 smithi059 ceph-mon[97450]: pgmap v9619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T02:36:26.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:36:26.195+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:36:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:26.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:26.856913+0000) 2022-09-08T02:36:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:26.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:26.805376+0000) 2022-09-08T02:36:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:27.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:27.857161+0000) 2022-09-08T02:36:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:27.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:27.805657+0000) 2022-09-08T02:36:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:27 smithi055 ceph-mon[97033]: pgmap v9620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:36:28.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:27 smithi059 ceph-mon[97450]: pgmap v9620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:36:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:28.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:28.857412+0000) 2022-09-08T02:36:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:28.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:28.805897+0000) 2022-09-08T02:36:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:29.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:29.857623+0000) 2022-09-08T02:36:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:29.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:29.806138+0000) 2022-09-08T02:36:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:29 smithi055 ceph-mon[97033]: pgmap v9621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:30.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:29 smithi059 ceph-mon[97450]: pgmap v9621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:30.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:30.857860+0000) 2022-09-08T02:36:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:30.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:30.806363+0000) 2022-09-08T02:36:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:31.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:31.858077+0000) 2022-09-08T02:36:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:31.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:31.806611+0000) 2022-09-08T02:36:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:31 smithi055 ceph-mon[97033]: pgmap v9622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:36:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:31 smithi059 ceph-mon[97450]: pgmap v9622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:36:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:32.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:32.858317+0000) 2022-09-08T02:36:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:32.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:32.806816+0000) 2022-09-08T02:36:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:33.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:33.858584+0000) 2022-09-08T02:36:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:33.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:33.806982+0000) 2022-09-08T02:36:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:33 smithi055 ceph-mon[97033]: pgmap v9623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:33 smithi059 ceph-mon[97450]: pgmap v9623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:34.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:34.858830+0000) 2022-09-08T02:36:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:34.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:34.807180+0000) 2022-09-08T02:36:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:35.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:35.807358+0000) 2022-09-08T02:36:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:35.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:35.858992+0000) 2022-09-08T02:36:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:35 smithi055 ceph-mon[97033]: pgmap v9624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:36.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:35 smithi059 ceph-mon[97450]: pgmap v9624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:36.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:36.859190+0000) 2022-09-08T02:36:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:36.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:36.807577+0000) 2022-09-08T02:36:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:37.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:37.859406+0000) 2022-09-08T02:36:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:37.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:37.807816+0000) 2022-09-08T02:36:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:37 smithi055 ceph-mon[97033]: pgmap v9625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:36:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:37 smithi059 ceph-mon[97450]: pgmap v9625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:36:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:38.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:38.808082+0000) 2022-09-08T02:36:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:38.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:38.859678+0000) 2022-09-08T02:36:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:38 smithi055 ceph-mon[97033]: pgmap v9626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:38 smithi059 ceph-mon[97450]: pgmap v9626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:39.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:39.808332+0000) 2022-09-08T02:36:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:39.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:39.859913+0000) 2022-09-08T02:36:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:40.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:40.860101+0000) 2022-09-08T02:36:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:40.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:40.808581+0000) 2022-09-08T02:36:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:41 smithi055 ceph-mon[97033]: pgmap v9627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:36:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:41 smithi059 ceph-mon[97450]: pgmap v9627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:36:41.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:36:41.206+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:36:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:41.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:41.860287+0000) 2022-09-08T02:36:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:41.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:41.808832+0000) 2022-09-08T02:36:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:42.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:42.860513+0000) 2022-09-08T02:36:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:42.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:42.809073+0000) 2022-09-08T02:36:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:43 smithi055 ceph-mon[97033]: pgmap v9628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:43 smithi055 ceph-mon[97033]: pgmap v9629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:36:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:43 smithi055 ceph-mon[97033]: pgmap v9630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:36:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:36:43.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:43 smithi059 ceph-mon[97450]: pgmap v9628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:43 smithi059 ceph-mon[97450]: pgmap v9629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:36:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:43 smithi059 ceph-mon[97450]: pgmap v9630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:36:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:36:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:43.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:43.809309+0000) 2022-09-08T02:36:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:43.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:43.860730+0000) 2022-09-08T02:36:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:44.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:44.860945+0000) 2022-09-08T02:36:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:44.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:44.809555+0000) 2022-09-08T02:36:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:45 smithi055 ceph-mon[97033]: pgmap v9631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:36:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:45 smithi059 ceph-mon[97450]: pgmap v9631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:36:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:45.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:45.861170+0000) 2022-09-08T02:36:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:45.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:45.809782+0000) 2022-09-08T02:36:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:46.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:46.809991+0000) 2022-09-08T02:36:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:46.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:46.861352+0000) 2022-09-08T02:36:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:47 smithi055 ceph-mon[97033]: pgmap v9632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:36:47.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:47 smithi059 ceph-mon[97450]: pgmap v9632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:36:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:47.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:47.861520+0000) 2022-09-08T02:36:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:47.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:47.810229+0000) 2022-09-08T02:36:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:48.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:48.861688+0000) 2022-09-08T02:36:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:48.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:48.810492+0000) 2022-09-08T02:36:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:49.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:49.861885+0000) 2022-09-08T02:36:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:49.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:49.810741+0000) 2022-09-08T02:36:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:49 smithi055 ceph-mon[97033]: pgmap v9633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:36:50.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:49 smithi059 ceph-mon[97450]: pgmap v9633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:36:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:50.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:50.862009+0000) 2022-09-08T02:36:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:50.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:50.810961+0000) 2022-09-08T02:36:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:51.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:51.862218+0000) 2022-09-08T02:36:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:51.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:51.811249+0000) 2022-09-08T02:36:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:51 smithi055 ceph-mon[97033]: pgmap v9634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:36:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:51 smithi059 ceph-mon[97450]: pgmap v9634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:36:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:52.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:52.862441+0000) 2022-09-08T02:36:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:52.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:52.811416+0000) 2022-09-08T02:36:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:53.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:53.811670+0000) 2022-09-08T02:36:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:53.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:53.862712+0000) 2022-09-08T02:36:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:53 smithi055 ceph-mon[97033]: pgmap v9635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:36:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:53 smithi059 ceph-mon[97450]: pgmap v9635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:36:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:54.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:54.811916+0000) 2022-09-08T02:36:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:54.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:54.862877+0000) 2022-09-08T02:36:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:55.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:55.863107+0000) 2022-09-08T02:36:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:55.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:55.812113+0000) 2022-09-08T02:36:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:55 smithi055 ceph-mon[97033]: pgmap v9636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:55 smithi059 ceph-mon[97450]: pgmap v9636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:36:56.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:36:56.206+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:36:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:56.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:56.863331+0000) 2022-09-08T02:36:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:56.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:56.812296+0000) 2022-09-08T02:36:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:57.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:57.863574+0000) 2022-09-08T02:36:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:57.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:57.812535+0000) 2022-09-08T02:36:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:57 smithi055 ceph-mon[97033]: pgmap v9637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:36:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:57 smithi059 ceph-mon[97450]: pgmap v9637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:36:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:58.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:58.863820+0000) 2022-09-08T02:36:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:58.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:58.812800+0000) 2022-09-08T02:37:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:36:59.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:59.863959+0000) 2022-09-08T02:37:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:36:59.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:36:59.813008+0000) 2022-09-08T02:37:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:36:59 smithi055 ceph-mon[97033]: pgmap v9638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:00.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:36:59 smithi059 ceph-mon[97450]: pgmap v9638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:00.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:00.864067+0000) 2022-09-08T02:37:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:00.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:00.813149+0000) 2022-09-08T02:37:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:01.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:01.864286+0000) 2022-09-08T02:37:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:01.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:01.813348+0000) 2022-09-08T02:37:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:01 smithi055 ceph-mon[97033]: pgmap v9639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:37:02.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:01 smithi059 ceph-mon[97450]: pgmap v9639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:37:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:02.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:02.864536+0000) 2022-09-08T02:37:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:02.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:02.813592+0000) 2022-09-08T02:37:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:03.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:03.864801+0000) 2022-09-08T02:37:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:03.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:03.813826+0000) 2022-09-08T02:37:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:03 smithi055 ceph-mon[97033]: pgmap v9640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:04.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:03 smithi059 ceph-mon[97450]: pgmap v9640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:04.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:04.865023+0000) 2022-09-08T02:37:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:04.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:04.814049+0000) 2022-09-08T02:37:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:05.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:05.865259+0000) 2022-09-08T02:37:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:05.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:05.814290+0000) 2022-09-08T02:37:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:05 smithi055 ceph-mon[97033]: pgmap v9641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:37:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:37:06.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:05 smithi059 ceph-mon[97450]: pgmap v9641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:37:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:37:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:06.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:06.865489+0000) 2022-09-08T02:37:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:06.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:06.814504+0000) 2022-09-08T02:37:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:07.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:07.865737+0000) 2022-09-08T02:37:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:07.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:07.814796+0000) 2022-09-08T02:37:08.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:07 smithi055 ceph-mon[97033]: pgmap v9642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:37:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:07 smithi059 ceph-mon[97450]: pgmap v9642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:37:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:08.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:08.865959+0000) 2022-09-08T02:37:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:08.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:08.815030+0000) 2022-09-08T02:37:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:09.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:09.866174+0000) 2022-09-08T02:37:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:09.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:09.815234+0000) 2022-09-08T02:37:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:09 smithi055 ceph-mon[97033]: pgmap v9643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:37:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:09 smithi055 ceph-mon[97033]: osdmap e683: 12 total, 11 up, 11 in 2022-09-08T02:37:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:09 smithi059 ceph-mon[97450]: pgmap v9643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:37:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:09 smithi059 ceph-mon[97450]: osdmap e683: 12 total, 11 up, 11 in 2022-09-08T02:37:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:10.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:10.866370+0000) 2022-09-08T02:37:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:10.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:10.815451+0000) 2022-09-08T02:37:11.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:37:11.223+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:37:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:11.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:11.866524+0000) 2022-09-08T02:37:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:11.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:11.815699+0000) 2022-09-08T02:37:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:11 smithi055 ceph-mon[97033]: pgmap v9645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:37:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:11 smithi059 ceph-mon[97450]: pgmap v9645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:37:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:12.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:12.815861+0000) 2022-09-08T02:37:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:12.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:12.866695+0000) 2022-09-08T02:37:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:13.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:13.866910+0000) 2022-09-08T02:37:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:13.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:13.816017+0000) 2022-09-08T02:37:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:13 smithi055 ceph-mon[97033]: pgmap v9646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:37:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:13 smithi055 ceph-mon[97033]: osdmap e684: 12 total, 11 up, 11 in 2022-09-08T02:37:14.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:13 smithi059 ceph-mon[97450]: pgmap v9646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:37:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:13 smithi059 ceph-mon[97450]: osdmap e684: 12 total, 11 up, 11 in 2022-09-08T02:37:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:14.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:14.816174+0000) 2022-09-08T02:37:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:14.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:14.867147+0000) 2022-09-08T02:37:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:15.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:15.867375+0000) 2022-09-08T02:37:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:15.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:15.816391+0000) 2022-09-08T02:37:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:15 smithi055 ceph-mon[97033]: pgmap v9648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:37:16.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:15 smithi059 ceph-mon[97450]: pgmap v9648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:37:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:16.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:16.867601+0000) 2022-09-08T02:37:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:16.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:16.816668+0000) 2022-09-08T02:37:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:17.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:17.867814+0000) 2022-09-08T02:37:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:17.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:17.816939+0000) 2022-09-08T02:37:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:17 smithi055 ceph-mon[97033]: pgmap v9649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:37:18.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:17 smithi059 ceph-mon[97450]: pgmap v9649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:37:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:18.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:18.817202+0000) 2022-09-08T02:37:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:18.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:18.868038+0000) 2022-09-08T02:37:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:19.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:19.868294+0000) 2022-09-08T02:37:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:19.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:19.817444+0000) 2022-09-08T02:37:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:19 smithi055 ceph-mon[97033]: pgmap v9650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 217 B/s wr, 6 op/s 2022-09-08T02:37:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:19 smithi059 ceph-mon[97450]: pgmap v9650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 217 B/s wr, 6 op/s 2022-09-08T02:37:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:20.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:20.868583+0000) 2022-09-08T02:37:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:20.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:20.817735+0000) 2022-09-08T02:37:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:21.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:21.868791+0000) 2022-09-08T02:37:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:21.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:21.817904+0000) 2022-09-08T02:37:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:21 smithi055 ceph-mon[97033]: pgmap v9651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:37:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:21 smithi059 ceph-mon[97450]: pgmap v9651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:37:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:22.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:22.869018+0000) 2022-09-08T02:37:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:22.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:22.818135+0000) 2022-09-08T02:37:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:23.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:23.869281+0000) 2022-09-08T02:37:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:23.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:23.818343+0000) 2022-09-08T02:37:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:23 smithi055 ceph-mon[97033]: pgmap v9652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:37:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:23 smithi059 ceph-mon[97450]: pgmap v9652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:37:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:24.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:24.869544+0000) 2022-09-08T02:37:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:24.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:24.818552+0000) 2022-09-08T02:37:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:25.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:25.818781+0000) 2022-09-08T02:37:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:25.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:25.869803+0000) 2022-09-08T02:37:26.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:25 smithi055 ceph-mon[97033]: pgmap v9653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T02:37:26.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:25 smithi059 ceph-mon[97450]: pgmap v9653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T02:37:26.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:37:26.226+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:37:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:26.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:26.870057+0000) 2022-09-08T02:37:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:26.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:26.818941+0000) 2022-09-08T02:37:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:27.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:27.870285+0000) 2022-09-08T02:37:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:27.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:27.819187+0000) 2022-09-08T02:37:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:27 smithi055 ceph-mon[97033]: pgmap v9654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:37:28.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:27 smithi059 ceph-mon[97450]: pgmap v9654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:37:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:28.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:28.870496+0000) 2022-09-08T02:37:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:28.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:28.819417+0000) 2022-09-08T02:37:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:29.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:29.870712+0000) 2022-09-08T02:37:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:29.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:29.819669+0000) 2022-09-08T02:37:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:29 smithi055 ceph-mon[97033]: pgmap v9655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:29 smithi059 ceph-mon[97450]: pgmap v9655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:30.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:30.870925+0000) 2022-09-08T02:37:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:30.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:30.819893+0000) 2022-09-08T02:37:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:31.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:31.871068+0000) 2022-09-08T02:37:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:31.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:31.820090+0000) 2022-09-08T02:37:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:31 smithi055 ceph-mon[97033]: pgmap v9656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:37:32.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:31 smithi059 ceph-mon[97450]: pgmap v9656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:37:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:32.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:32.820285+0000) 2022-09-08T02:37:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:32.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:32.871251+0000) 2022-09-08T02:37:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:33.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:33.871500+0000) 2022-09-08T02:37:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:33.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:33.820423+0000) 2022-09-08T02:37:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:33 smithi055 ceph-mon[97033]: pgmap v9657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:34.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:33 smithi059 ceph-mon[97450]: pgmap v9657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:34.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:34.871754+0000) 2022-09-08T02:37:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:34.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:34.820630+0000) 2022-09-08T02:37:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:35.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:35.872006+0000) 2022-09-08T02:37:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:35.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:35.820859+0000) 2022-09-08T02:37:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:35 smithi055 ceph-mon[97033]: pgmap v9658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:35 smithi059 ceph-mon[97450]: pgmap v9658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:36.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:36.872197+0000) 2022-09-08T02:37:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:36.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:36.821048+0000) 2022-09-08T02:37:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:37.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:37.872423+0000) 2022-09-08T02:37:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:37.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:37.821282+0000) 2022-09-08T02:37:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:37 smithi055 ceph-mon[97033]: pgmap v9659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:37:38.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:37 smithi059 ceph-mon[97450]: pgmap v9659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:37:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:38.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:38.872679+0000) 2022-09-08T02:37:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:38.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:38.821513+0000) 2022-09-08T02:37:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:39.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:39.872848+0000) 2022-09-08T02:37:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:39.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:39.821736+0000) 2022-09-08T02:37:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:39 smithi055 ceph-mon[97033]: pgmap v9660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:40.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:39 smithi059 ceph-mon[97450]: pgmap v9660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:40.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:40.873049+0000) 2022-09-08T02:37:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:40.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:40.821947+0000) 2022-09-08T02:37:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:40 smithi055 ceph-mon[97033]: pgmap v9661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:37:41.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:40 smithi059 ceph-mon[97450]: pgmap v9661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:37:41.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:37:41.236+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:37:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:41.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:41.873305+0000) 2022-09-08T02:37:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:41.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:41.822195+0000) 2022-09-08T02:37:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:42.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:42.873532+0000) 2022-09-08T02:37:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:42.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:42.822458+0000) 2022-09-08T02:37:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:43 smithi055 ceph-mon[97033]: pgmap v9662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:37:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:37:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:37:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:43 smithi059 ceph-mon[97450]: pgmap v9662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:37:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:37:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:37:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:43.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:43.873792+0000) 2022-09-08T02:37:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:43.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:43.822672+0000) 2022-09-08T02:37:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:44.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:44.874040+0000) 2022-09-08T02:37:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:44.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:44.822797+0000) 2022-09-08T02:37:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:45 smithi055 ceph-mon[97033]: pgmap v9663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:45 smithi059 ceph-mon[97450]: pgmap v9663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:45.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:45.874218+0000) 2022-09-08T02:37:46.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:45.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:45.822963+0000) 2022-09-08T02:37:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:46.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:46.874447+0000) 2022-09-08T02:37:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:46.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:46.823104+0000) 2022-09-08T02:37:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:47 smithi055 ceph-mon[97033]: pgmap v9664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:37:47.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:47 smithi059 ceph-mon[97450]: pgmap v9664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:37:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:47.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:47.823310+0000) 2022-09-08T02:37:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:47.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:47.874714+0000) 2022-09-08T02:37:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:48.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:48.874839+0000) 2022-09-08T02:37:49.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:48.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:48.823521+0000) 2022-09-08T02:37:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:49.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:49.874986+0000) 2022-09-08T02:37:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:49.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:49.823728+0000) 2022-09-08T02:37:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:49 smithi055 ceph-mon[97033]: pgmap v9665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:50.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:49 smithi059 ceph-mon[97450]: pgmap v9665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:50.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:50.823902+0000) 2022-09-08T02:37:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:50.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:50.875130+0000) 2022-09-08T02:37:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:51.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:51.875275+0000) 2022-09-08T02:37:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:51.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:51.824162+0000) 2022-09-08T02:37:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:51 smithi055 ceph-mon[97033]: pgmap v9666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:37:52.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:51 smithi059 ceph-mon[97450]: pgmap v9666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:37:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:52.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:52.875485+0000) 2022-09-08T02:37:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:52.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:52.824333+0000) 2022-09-08T02:37:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:53.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:53.875767+0000) 2022-09-08T02:37:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:53.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:53.824475+0000) 2022-09-08T02:37:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:53 smithi055 ceph-mon[97033]: pgmap v9667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:54.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:53 smithi059 ceph-mon[97450]: pgmap v9667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:54.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:54.876045+0000) 2022-09-08T02:37:55.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:54.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:54.824730+0000) 2022-09-08T02:37:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:55.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:55.824982+0000) 2022-09-08T02:37:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:55.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:55.876202+0000) 2022-09-08T02:37:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:55 smithi055 ceph-mon[97033]: pgmap v9668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:56.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:55 smithi059 ceph-mon[97450]: pgmap v9668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:37:56.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:37:56.248+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:37:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:56.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:56.876382+0000) 2022-09-08T02:37:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:56.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:56.825236+0000) 2022-09-08T02:37:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:57.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:57.876620+0000) 2022-09-08T02:37:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:57.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:57.825468+0000) 2022-09-08T02:37:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:57 smithi055 ceph-mon[97033]: pgmap v9669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:37:58.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:57 smithi059 ceph-mon[97450]: pgmap v9669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:37:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:58.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:58.876844+0000) 2022-09-08T02:37:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:58.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:58.825758+0000) 2022-09-08T02:38:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:37:59.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:59.877123+0000) 2022-09-08T02:38:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:37:59.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:37:59.825989+0000) 2022-09-08T02:38:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:37:59 smithi055 ceph-mon[97033]: pgmap v9670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:00.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:37:59 smithi059 ceph-mon[97450]: pgmap v9670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:00.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:00.877323+0000) 2022-09-08T02:38:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:00.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:00.826209+0000) 2022-09-08T02:38:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:01.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:01.877446+0000) 2022-09-08T02:38:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:01.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:01.826439+0000) 2022-09-08T02:38:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:01 smithi055 ceph-mon[97033]: pgmap v9671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:38:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:01 smithi059 ceph-mon[97450]: pgmap v9671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:38:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:02.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:02.877700+0000) 2022-09-08T02:38:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:02.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:02.826713+0000) 2022-09-08T02:38:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:03.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:03.877972+0000) 2022-09-08T02:38:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:03.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:03.826963+0000) 2022-09-08T02:38:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:03 smithi055 ceph-mon[97033]: pgmap v9672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:04.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:03 smithi059 ceph-mon[97450]: pgmap v9672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:04.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:04.878161+0000) 2022-09-08T02:38:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:04.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:04.827197+0000) 2022-09-08T02:38:05.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:04 smithi059 ceph-mon[97450]: pgmap v9673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:04 smithi055 ceph-mon[97033]: pgmap v9673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:05.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:05.878396+0000) 2022-09-08T02:38:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:05.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:05.827400+0000) 2022-09-08T02:38:06.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:38:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:38:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:38:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:38:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:06.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:06.878660+0000) 2022-09-08T02:38:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:06.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:06.827576+0000) 2022-09-08T02:38:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:07 smithi059 ceph-mon[97450]: pgmap v9674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:38:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:07 smithi055 ceph-mon[97033]: pgmap v9674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:38:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:07.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:07.879026+0000) 2022-09-08T02:38:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:07.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:07.827807+0000) 2022-09-08T02:38:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:08.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:08.879201+0000) 2022-09-08T02:38:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:08.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:08.828044+0000) 2022-09-08T02:38:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:09.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:09.879422+0000) 2022-09-08T02:38:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:09.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:09.828298+0000) 2022-09-08T02:38:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:09 smithi055 ceph-mon[97033]: pgmap v9675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:38:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:09 smithi055 ceph-mon[97033]: osdmap e685: 12 total, 11 up, 11 in 2022-09-08T02:38:10.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:09 smithi059 ceph-mon[97450]: pgmap v9675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:38:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:09 smithi059 ceph-mon[97450]: osdmap e685: 12 total, 11 up, 11 in 2022-09-08T02:38:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:10.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:10.879676+0000) 2022-09-08T02:38:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:10.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:10.828516+0000) 2022-09-08T02:38:11.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:38:11.250+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:38:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:11.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:11.879917+0000) 2022-09-08T02:38:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:11.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:11.828743+0000) 2022-09-08T02:38:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:11 smithi055 ceph-mon[97033]: pgmap v9677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:38:12.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:11 smithi059 ceph-mon[97450]: pgmap v9677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:38:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:12.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:12.880161+0000) 2022-09-08T02:38:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:12.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:12.828891+0000) 2022-09-08T02:38:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:13.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:13.880374+0000) 2022-09-08T02:38:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:13.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:13.829114+0000) 2022-09-08T02:38:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:13 smithi055 ceph-mon[97033]: pgmap v9678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:38:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:13 smithi055 ceph-mon[97033]: osdmap e686: 12 total, 11 up, 11 in 2022-09-08T02:38:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:13 smithi059 ceph-mon[97450]: pgmap v9678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:38:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:13 smithi059 ceph-mon[97450]: osdmap e686: 12 total, 11 up, 11 in 2022-09-08T02:38:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:14.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:14.880593+0000) 2022-09-08T02:38:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:14.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:14.829363+0000) 2022-09-08T02:38:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:15.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:15.880770+0000) 2022-09-08T02:38:16.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:15.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:15.829555+0000) 2022-09-08T02:38:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:15 smithi055 ceph-mon[97033]: pgmap v9680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:38:16.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:15 smithi059 ceph-mon[97450]: pgmap v9680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:38:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:16.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:16.880991+0000) 2022-09-08T02:38:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:16.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:16.829778+0000) 2022-09-08T02:38:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:17.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:17.881218+0000) 2022-09-08T02:38:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:17.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:17.829973+0000) 2022-09-08T02:38:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:17 smithi055 ceph-mon[97033]: pgmap v9681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:38:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:38:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:38:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:38:18.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:17 smithi059 ceph-mon[97450]: pgmap v9681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:38:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:38:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:38:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:38:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:18.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:18.881376+0000) 2022-09-08T02:38:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:18.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:18.830150+0000) 2022-09-08T02:38:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:18 smithi055 ceph-mon[97033]: pgmap v9682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T02:38:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:18 smithi055 ceph-mon[97033]: pgmap v9683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 153 B/s wr, 4 op/s 2022-09-08T02:38:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:18 smithi059 ceph-mon[97450]: pgmap v9682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T02:38:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:18 smithi059 ceph-mon[97450]: pgmap v9683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 153 B/s wr, 4 op/s 2022-09-08T02:38:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:19.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:19.881587+0000) 2022-09-08T02:38:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:19.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:19.830352+0000) 2022-09-08T02:38:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:20.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:20.881819+0000) 2022-09-08T02:38:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:20.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:20.830541+0000) 2022-09-08T02:38:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:20 smithi055 ceph-mon[97033]: pgmap v9684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 153 B/s wr, 4 op/s 2022-09-08T02:38:21.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:20 smithi059 ceph-mon[97450]: pgmap v9684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 153 B/s wr, 4 op/s 2022-09-08T02:38:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:21.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:21.882073+0000) 2022-09-08T02:38:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:21.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:21.830757+0000) 2022-09-08T02:38:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:22.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:22.882269+0000) 2022-09-08T02:38:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:22.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:22.830964+0000) 2022-09-08T02:38:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:22 smithi055 ceph-mon[97033]: pgmap v9685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 126 B/s wr, 7 op/s 2022-09-08T02:38:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:22 smithi059 ceph-mon[97450]: pgmap v9685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 126 B/s wr, 7 op/s 2022-09-08T02:38:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:23.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:23.882527+0000) 2022-09-08T02:38:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:23.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:23.831174+0000) 2022-09-08T02:38:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:24.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:24.882792+0000) 2022-09-08T02:38:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:24.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:24.831383+0000) 2022-09-08T02:38:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:24 smithi055 ceph-mon[97033]: pgmap v9686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T02:38:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:24 smithi059 ceph-mon[97450]: pgmap v9686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T02:38:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:25.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:25.882946+0000) 2022-09-08T02:38:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:25.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:25.831625+0000) 2022-09-08T02:38:26.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:38:26.253+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:38:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:26.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:26.831908+0000) 2022-09-08T02:38:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:26.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:26.883143+0000) 2022-09-08T02:38:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:26 smithi055 ceph-mon[97033]: pgmap v9687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T02:38:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:26 smithi059 ceph-mon[97450]: pgmap v9687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T02:38:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:27.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:27.832168+0000) 2022-09-08T02:38:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:27.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:27.883364+0000) 2022-09-08T02:38:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:28.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:28.883565+0000) 2022-09-08T02:38:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:28.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:28.832422+0000) 2022-09-08T02:38:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:28 smithi055 ceph-mon[97033]: pgmap v9688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:38:29.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:28 smithi059 ceph-mon[97450]: pgmap v9688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:38:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:29.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:29.883716+0000) 2022-09-08T02:38:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:29.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:29.832649+0000) 2022-09-08T02:38:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:30.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:30.883913+0000) 2022-09-08T02:38:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:30.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:30.832862+0000) 2022-09-08T02:38:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:30 smithi055 ceph-mon[97033]: pgmap v9689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:30 smithi059 ceph-mon[97450]: pgmap v9689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:31.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:31.833102+0000) 2022-09-08T02:38:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:31.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:31.884153+0000) 2022-09-08T02:38:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:32.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:32.884416+0000) 2022-09-08T02:38:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:32.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:32.833290+0000) 2022-09-08T02:38:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:32 smithi055 ceph-mon[97033]: pgmap v9690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:38:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:32 smithi059 ceph-mon[97450]: pgmap v9690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:38:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:33.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:33.833506+0000) 2022-09-08T02:38:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:33.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:33.884657+0000) 2022-09-08T02:38:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:33 smithi055 ceph-mon[97033]: pgmap v9691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:34.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:33 smithi059 ceph-mon[97450]: pgmap v9691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:34.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:34.833718+0000) 2022-09-08T02:38:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:34.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:34.884913+0000) 2022-09-08T02:38:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:35.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:35.833974+0000) 2022-09-08T02:38:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:35.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:35.885217+0000) 2022-09-08T02:38:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:36 smithi055 ceph-mon[97033]: pgmap v9692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:38:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:36 smithi059 ceph-mon[97450]: pgmap v9692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:38:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:36.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:36.885460+0000) 2022-09-08T02:38:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:36.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:36.834226+0000) 2022-09-08T02:38:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:37.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:37.885650+0000) 2022-09-08T02:38:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:37.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:37.834433+0000) 2022-09-08T02:38:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:38.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:38.885873+0000) 2022-09-08T02:38:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:38.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:38.834692+0000) 2022-09-08T02:38:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:38 smithi055 ceph-mon[97033]: pgmap v9693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:38 smithi059 ceph-mon[97450]: pgmap v9693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:39.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:39.886023+0000) 2022-09-08T02:38:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:39.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:39.834911+0000) 2022-09-08T02:38:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:40.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:40.886223+0000) 2022-09-08T02:38:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:40.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:40.835116+0000) 2022-09-08T02:38:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:40 smithi055 ceph-mon[97033]: pgmap v9694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:41.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:40 smithi059 ceph-mon[97450]: pgmap v9694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:41.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:38:41.261+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:38:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:41.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:41.886526+0000) 2022-09-08T02:38:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:41.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:41.835335+0000) 2022-09-08T02:38:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:42.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:42.886778+0000) 2022-09-08T02:38:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:42.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:42.835534+0000) 2022-09-08T02:38:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:42 smithi055 ceph-mon[97033]: pgmap v9695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:38:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:42 smithi059 ceph-mon[97450]: pgmap v9695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:38:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:43.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:43.887044+0000) 2022-09-08T02:38:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:43.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:43.835708+0000) 2022-09-08T02:38:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:44.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:44.887310+0000) 2022-09-08T02:38:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:44.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:44.835916+0000) 2022-09-08T02:38:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:44 smithi055 ceph-mon[97033]: pgmap v9696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:44 smithi059 ceph-mon[97450]: pgmap v9696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:45.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:45.887531+0000) 2022-09-08T02:38:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:45.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:45.836152+0000) 2022-09-08T02:38:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:46.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:46.887774+0000) 2022-09-08T02:38:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:46.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:46.836384+0000) 2022-09-08T02:38:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:46 smithi055 ceph-mon[97033]: pgmap v9697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:38:47.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:46 smithi059 ceph-mon[97450]: pgmap v9697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:38:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:47.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:47.888016+0000) 2022-09-08T02:38:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:47.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:47.836565+0000) 2022-09-08T02:38:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:48.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:48.888268+0000) 2022-09-08T02:38:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:48.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:48.836790+0000) 2022-09-08T02:38:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:48 smithi055 ceph-mon[97033]: pgmap v9698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:48 smithi059 ceph-mon[97450]: pgmap v9698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:49.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:49.888503+0000) 2022-09-08T02:38:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:49.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:49.836967+0000) 2022-09-08T02:38:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:50.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:50.888757+0000) 2022-09-08T02:38:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:50.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:50.837208+0000) 2022-09-08T02:38:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:50 smithi055 ceph-mon[97033]: pgmap v9699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:50 smithi059 ceph-mon[97450]: pgmap v9699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:51.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:51.888965+0000) 2022-09-08T02:38:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:51.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:51.837420+0000) 2022-09-08T02:38:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:52.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:52.889163+0000) 2022-09-08T02:38:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:52.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:52.837701+0000) 2022-09-08T02:38:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:52 smithi055 ceph-mon[97033]: pgmap v9700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:38:53.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:52 smithi059 ceph-mon[97450]: pgmap v9700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:38:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:53.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:53.889404+0000) 2022-09-08T02:38:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:53.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:53.837846+0000) 2022-09-08T02:38:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:54.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:54.889573+0000) 2022-09-08T02:38:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:54.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:54.837972+0000) 2022-09-08T02:38:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:54 smithi055 ceph-mon[97033]: pgmap v9701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:54 smithi059 ceph-mon[97450]: pgmap v9701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:55.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:55.889735+0000) 2022-09-08T02:38:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:55.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:55.838204+0000) 2022-09-08T02:38:56.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:56 smithi059 ceph-mon[97450]: pgmap v9702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:38:56.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:56 smithi055 ceph-mon[97033]: pgmap v9702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:38:56.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:38:56.276+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:38:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:56.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:56.889961+0000) 2022-09-08T02:38:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:56.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:56.838448+0000) 2022-09-08T02:38:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:57.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:57.890115+0000) 2022-09-08T02:38:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:57.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:57.838735+0000) 2022-09-08T02:38:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:58.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:58.890244+0000) 2022-09-08T02:38:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:58.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:58.838910+0000) 2022-09-08T02:38:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:38:58 smithi055 ceph-mon[97033]: pgmap v9703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:38:59.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:38:58 smithi059 ceph-mon[97450]: pgmap v9703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:38:59.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:59.890476+0000) 2022-09-08T02:39:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:38:59.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:38:59.839083+0000) 2022-09-08T02:39:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:00.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:00.890674+0000) 2022-09-08T02:39:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:00.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:00.839257+0000) 2022-09-08T02:39:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:00 smithi055 ceph-mon[97033]: pgmap v9704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:01.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:00 smithi059 ceph-mon[97450]: pgmap v9704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:01.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:01.890874+0000) 2022-09-08T02:39:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:01.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:01.839487+0000) 2022-09-08T02:39:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:02.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:02.891104+0000) 2022-09-08T02:39:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:02.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:02.839735+0000) 2022-09-08T02:39:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:02 smithi055 ceph-mon[97033]: pgmap v9705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:39:03.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:02 smithi059 ceph-mon[97450]: pgmap v9705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:39:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:03.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:03.891349+0000) 2022-09-08T02:39:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:03.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:03.839979+0000) 2022-09-08T02:39:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:04.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:04.891551+0000) 2022-09-08T02:39:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:04.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:04.840202+0000) 2022-09-08T02:39:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:04 smithi055 ceph-mon[97033]: pgmap v9706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:04 smithi059 ceph-mon[97450]: pgmap v9706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:05.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:05.891767+0000) 2022-09-08T02:39:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:05.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:05.840444+0000) 2022-09-08T02:39:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:39:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:39:06.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:39:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:39:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:06.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:06.891999+0000) 2022-09-08T02:39:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:06.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:06.840654+0000) 2022-09-08T02:39:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:06 smithi055 ceph-mon[97033]: pgmap v9707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:39:07.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:06 smithi059 ceph-mon[97450]: pgmap v9707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:39:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:07.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:07.892225+0000) 2022-09-08T02:39:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:07.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:07.840945+0000) 2022-09-08T02:39:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:08.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:08.892425+0000) 2022-09-08T02:39:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:08.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:08.841202+0000) 2022-09-08T02:39:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:08 smithi055 ceph-mon[97033]: pgmap v9708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:39:09.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:08 smithi059 ceph-mon[97450]: pgmap v9708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:39:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:09.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:09.841386+0000) 2022-09-08T02:39:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:09.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:09.892630+0000) 2022-09-08T02:39:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:09 smithi055 ceph-mon[97033]: osdmap e687: 12 total, 11 up, 11 in 2022-09-08T02:39:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:09 smithi055 ceph-mon[97033]: pgmap v9710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:39:10.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:09 smithi059 ceph-mon[97450]: osdmap e687: 12 total, 11 up, 11 in 2022-09-08T02:39:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:09 smithi059 ceph-mon[97450]: pgmap v9710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:39:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:10.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:10.892845+0000) 2022-09-08T02:39:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:10.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:10.841559+0000) 2022-09-08T02:39:11.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:39:11.292+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:39:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:11.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:11.893091+0000) 2022-09-08T02:39:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:11.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:11.841731+0000) 2022-09-08T02:39:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:12 smithi055 ceph-mon[97033]: pgmap v9711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:39:12.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:12 smithi059 ceph-mon[97450]: pgmap v9711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:39:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:12.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:12.893329+0000) 2022-09-08T02:39:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:12.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:12.841917+0000) 2022-09-08T02:39:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:13.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:13.893549+0000) 2022-09-08T02:39:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:13.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:13.842100+0000) 2022-09-08T02:39:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:13 smithi055 ceph-mon[97033]: osdmap e688: 12 total, 11 up, 11 in 2022-09-08T02:39:14.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:13 smithi059 ceph-mon[97450]: osdmap e688: 12 total, 11 up, 11 in 2022-09-08T02:39:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:14.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:14.893769+0000) 2022-09-08T02:39:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:14.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:14.842272+0000) 2022-09-08T02:39:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:14 smithi055 ceph-mon[97033]: pgmap v9713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:39:15.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:14 smithi059 ceph-mon[97450]: pgmap v9713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:39:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:15.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:15.893977+0000) 2022-09-08T02:39:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:15.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:15.842428+0000) 2022-09-08T02:39:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:16.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:16.894215+0000) 2022-09-08T02:39:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:16.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:16.842665+0000) 2022-09-08T02:39:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:16 smithi055 ceph-mon[97033]: pgmap v9714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:39:17.241 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:16 smithi059 ceph-mon[97450]: pgmap v9714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:39:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:17.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:17.894466+0000) 2022-09-08T02:39:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:17.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:17.842877+0000) 2022-09-08T02:39:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:39:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:39:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:39:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:39:18.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:39:18.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:39:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:39:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:39:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:39:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:39:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:18.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:18.894761+0000) 2022-09-08T02:39:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:18.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:18.843104+0000) 2022-09-08T02:39:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:18 smithi055 ceph-mon[97033]: pgmap v9715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T02:39:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:18 smithi059 ceph-mon[97450]: pgmap v9715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T02:39:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:19.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:19.894980+0000) 2022-09-08T02:39:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:19.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:19.843250+0000) 2022-09-08T02:39:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:20.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:20.895136+0000) 2022-09-08T02:39:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:20.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:20.843430+0000) 2022-09-08T02:39:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:20 smithi055 ceph-mon[97033]: pgmap v9716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:39:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:39:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:39:21.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:39:21.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:39:21.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:39:21.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:20 smithi059 ceph-mon[97450]: pgmap v9716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:39:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:39:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:39:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:39:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:39:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:39:21.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:39:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:21.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:21.895306+0000) 2022-09-08T02:39:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:21.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:21.843688+0000) 2022-09-08T02:39:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:21 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T02:39:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:21 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T02:39:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:21 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T02:39:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:21 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T02:39:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:21 smithi055 ceph-mon[97033]: pgmap v9717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 103 B/s wr, 6 op/s 2022-09-08T02:39:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:21 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T02:39:22.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:21 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T02:39:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:21 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T02:39:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:21 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T02:39:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:21 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T02:39:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:21 smithi059 ceph-mon[97450]: pgmap v9717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 103 B/s wr, 6 op/s 2022-09-08T02:39:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:21 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T02:39:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:22.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:22.895513+0000) 2022-09-08T02:39:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:22.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:22.843857+0000) 2022-09-08T02:39:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:23.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:23.895695+0000) 2022-09-08T02:39:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:23.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:23.844094+0000) 2022-09-08T02:39:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:23 smithi055 ceph-mon[97033]: pgmap v9718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:39:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:23 smithi059 ceph-mon[97450]: pgmap v9718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:39:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:24.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:24.895813+0000) 2022-09-08T02:39:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:24.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:24.844236+0000) 2022-09-08T02:39:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:25.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:25.896034+0000) 2022-09-08T02:39:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:25.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:25.844437+0000) 2022-09-08T02:39:26.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:25 smithi055 ceph-mon[97033]: pgmap v9719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T02:39:26.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:25 smithi059 ceph-mon[97450]: pgmap v9719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T02:39:26.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:39:26.296+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:39:27.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:26.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:26.896208+0000) 2022-09-08T02:39:27.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:26.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:26.844622+0000) 2022-09-08T02:39:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:27.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:27.896429+0000) 2022-09-08T02:39:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:27.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:27.844898+0000) 2022-09-08T02:39:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:27 smithi055 ceph-mon[97033]: pgmap v9720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T02:39:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:27 smithi059 ceph-mon[97450]: pgmap v9720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T02:39:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:28.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:28.896609+0000) 2022-09-08T02:39:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:28.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:28.845053+0000) 2022-09-08T02:39:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:29.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:29.896865+0000) 2022-09-08T02:39:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:29.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:29.845217+0000) 2022-09-08T02:39:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:29 smithi055 ceph-mon[97033]: pgmap v9721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T02:39:30.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:29 smithi059 ceph-mon[97450]: pgmap v9721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T02:39:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:30.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:30.897033+0000) 2022-09-08T02:39:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:30.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:30.845396+0000) 2022-09-08T02:39:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:30 smithi055 ceph-mon[97033]: pgmap v9722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T02:39:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:30 smithi059 ceph-mon[97450]: pgmap v9722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T02:39:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:31.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:31.897277+0000) 2022-09-08T02:39:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:31.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:31.845618+0000) 2022-09-08T02:39:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:32.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:32.897527+0000) 2022-09-08T02:39:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:32.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:32.845836+0000) 2022-09-08T02:39:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:33.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:33.897675+0000) 2022-09-08T02:39:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:33.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:33.845988+0000) 2022-09-08T02:39:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:33 smithi055 ceph-mon[97033]: pgmap v9723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:34.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:33 smithi059 ceph-mon[97450]: pgmap v9723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:34.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:34.897826+0000) 2022-09-08T02:39:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:34.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:34.846194+0000) 2022-09-08T02:39:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:35.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:35.898012+0000) 2022-09-08T02:39:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:35.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:35.846400+0000) 2022-09-08T02:39:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:35 smithi055 ceph-mon[97033]: pgmap v9724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:39:36.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:35 smithi059 ceph-mon[97450]: pgmap v9724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:39:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:36.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:36.898236+0000) 2022-09-08T02:39:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:36.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:36.846596+0000) 2022-09-08T02:39:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:37.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:37.898424+0000) 2022-09-08T02:39:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:37.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:37.846879+0000) 2022-09-08T02:39:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:37 smithi055 ceph-mon[97033]: pgmap v9725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:37 smithi059 ceph-mon[97450]: pgmap v9725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:38.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:38.898684+0000) 2022-09-08T02:39:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:38.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:38.847051+0000) 2022-09-08T02:39:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:39.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:39.898909+0000) 2022-09-08T02:39:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:39.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:39.847227+0000) 2022-09-08T02:39:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:39 smithi055 ceph-mon[97033]: pgmap v9726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:39 smithi059 ceph-mon[97450]: pgmap v9726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:40.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:40.899105+0000) 2022-09-08T02:39:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:40.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:40.847391+0000) 2022-09-08T02:39:41.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:39:41.304+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:39:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:41.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:41.847571+0000) 2022-09-08T02:39:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:41.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:41.899345+0000) 2022-09-08T02:39:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:41 smithi055 ceph-mon[97033]: pgmap v9727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:39:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:41 smithi059 ceph-mon[97450]: pgmap v9727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:39:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:42.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:42.899574+0000) 2022-09-08T02:39:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:42.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:42.847813+0000) 2022-09-08T02:39:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:43.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:43.899712+0000) 2022-09-08T02:39:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:43.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:43.848032+0000) 2022-09-08T02:39:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:43 smithi055 ceph-mon[97033]: pgmap v9728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:43 smithi059 ceph-mon[97450]: pgmap v9728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:44.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:44.899921+0000) 2022-09-08T02:39:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:44.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:44.848170+0000) 2022-09-08T02:39:45.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:45 smithi059 ceph-mon[97450]: pgmap v9729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:39:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:45 smithi055 ceph-mon[97033]: pgmap v9729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:39:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:45.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:45.848383+0000) 2022-09-08T02:39:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:45.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:45.900128+0000) 2022-09-08T02:39:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:46.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:46.900265+0000) 2022-09-08T02:39:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:46.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:46.848607+0000) 2022-09-08T02:39:47.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:47 smithi059 ceph-mon[97450]: pgmap v9730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:39:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:47 smithi055 ceph-mon[97033]: pgmap v9730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:39:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:47.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:47.900438+0000) 2022-09-08T02:39:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:47.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:47.848775+0000) 2022-09-08T02:39:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:48.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:48.849013+0000) 2022-09-08T02:39:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:48.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:48.900704+0000) 2022-09-08T02:39:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:49.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:49.900940+0000) 2022-09-08T02:39:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:49.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:49.849153+0000) 2022-09-08T02:39:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:49 smithi055 ceph-mon[97033]: pgmap v9731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:39:50.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:49 smithi059 ceph-mon[97450]: pgmap v9731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:39:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:50.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:50.901125+0000) 2022-09-08T02:39:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:50.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:50.849356+0000) 2022-09-08T02:39:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:51.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:51.901362+0000) 2022-09-08T02:39:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:51.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:51.849594+0000) 2022-09-08T02:39:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:51 smithi055 ceph-mon[97033]: pgmap v9732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:39:52.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:51 smithi059 ceph-mon[97450]: pgmap v9732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:39:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:52.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:52.901591+0000) 2022-09-08T02:39:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:52.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:52.849796+0000) 2022-09-08T02:39:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:53.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:53.901763+0000) 2022-09-08T02:39:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:53.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:53.849981+0000) 2022-09-08T02:39:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:53 smithi055 ceph-mon[97033]: pgmap v9733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:39:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:53 smithi059 ceph-mon[97450]: pgmap v9733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:39:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:54.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:54.901941+0000) 2022-09-08T02:39:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:54.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:54.850133+0000) 2022-09-08T02:39:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:55.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:55.902154+0000) 2022-09-08T02:39:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:55.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:55.850313+0000) 2022-09-08T02:39:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:55 smithi055 ceph-mon[97033]: pgmap v9734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:39:56.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:55 smithi059 ceph-mon[97450]: pgmap v9734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:39:56.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:39:56.306+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:39:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:56.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:56.902383+0000) 2022-09-08T02:39:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:56.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:56.850536+0000) 2022-09-08T02:39:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:57.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:57.902620+0000) 2022-09-08T02:39:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:57.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:57.850784+0000) 2022-09-08T02:39:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:57 smithi055 ceph-mon[97033]: pgmap v9735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:57 smithi059 ceph-mon[97450]: pgmap v9735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:58.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:58.902745+0000) 2022-09-08T02:39:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:58.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:58.850928+0000) 2022-09-08T02:39:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:39:59 smithi059 ceph-mon[97450]: pgmap v9736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:39:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:39:59 smithi055 ceph-mon[97033]: pgmap v9736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:39:59.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:59.902924+0000) 2022-09-08T02:40:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:39:59.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:39:59.851108+0000) 2022-09-08T02:40:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T02:40:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T02:40:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T02:40:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T02:40:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T02:40:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T02:40:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:00.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:00.903121+0000) 2022-09-08T02:40:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:00.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:00.851261+0000) 2022-09-08T02:40:01.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:01 smithi059 ceph-mon[97450]: pgmap v9737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:40:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:01 smithi055 ceph-mon[97033]: pgmap v9737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:40:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:01.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:01.903299+0000) 2022-09-08T02:40:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:01.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:01.851498+0000) 2022-09-08T02:40:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:02.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:02.903530+0000) 2022-09-08T02:40:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:02.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:02.851772+0000) 2022-09-08T02:40:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:03.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:03.903743+0000) 2022-09-08T02:40:04.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:03.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:03.851930+0000) 2022-09-08T02:40:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:03 smithi055 ceph-mon[97033]: pgmap v9738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:03 smithi059 ceph-mon[97450]: pgmap v9738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:05.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:04.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:04.903923+0000) 2022-09-08T02:40:05.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:04.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:04.852131+0000) 2022-09-08T02:40:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:05.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:05.904143+0000) 2022-09-08T02:40:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:05.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:05.852409+0000) 2022-09-08T02:40:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:05 smithi055 ceph-mon[97033]: pgmap v9739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-08T02:40:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:40:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:40:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:05 smithi059 ceph-mon[97450]: pgmap v9739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-08T02:40:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:40:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:40:07.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:06.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:06.904299+0000) 2022-09-08T02:40:07.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:06.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:06.852615+0000) 2022-09-08T02:40:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:07.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:07.904536+0000) 2022-09-08T02:40:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:07.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:07.852856+0000) 2022-09-08T02:40:08.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:07 smithi055 ceph-mon[97033]: pgmap v9740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:40:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:07 smithi059 ceph-mon[97450]: pgmap v9740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:40:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:08.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:08.853096+0000) 2022-09-08T02:40:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:08.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:08.904741+0000) 2022-09-08T02:40:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:09.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:09.853229+0000) 2022-09-08T02:40:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:09.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:09.904925+0000) 2022-09-08T02:40:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:09 smithi055 ceph-mon[97033]: pgmap v9741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:40:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:09 smithi055 ceph-mon[97033]: osdmap e689: 12 total, 11 up, 11 in 2022-09-08T02:40:10.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:09 smithi059 ceph-mon[97450]: pgmap v9741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:40:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:09 smithi059 ceph-mon[97450]: osdmap e689: 12 total, 11 up, 11 in 2022-09-08T02:40:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:10.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:10.905085+0000) 2022-09-08T02:40:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:10.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:10.853443+0000) 2022-09-08T02:40:11.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:11 smithi059 ceph-mon[97450]: pgmap v9743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:40:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:11 smithi055 ceph-mon[97033]: pgmap v9743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:40:11.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:40:11.320+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:40:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:11.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:11.905284+0000) 2022-09-08T02:40:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:11.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:11.853744+0000) 2022-09-08T02:40:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:12.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:12.905522+0000) 2022-09-08T02:40:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:12.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:12.854024+0000) 2022-09-08T02:40:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:13.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:13.854239+0000) 2022-09-08T02:40:14.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:13.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:13.905767+0000) 2022-09-08T02:40:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:13 smithi055 ceph-mon[97033]: pgmap v9744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:40:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:13 smithi055 ceph-mon[97033]: osdmap e690: 12 total, 11 up, 11 in 2022-09-08T02:40:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:13 smithi059 ceph-mon[97450]: pgmap v9744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:40:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:13 smithi059 ceph-mon[97450]: osdmap e690: 12 total, 11 up, 11 in 2022-09-08T02:40:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:14.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:14.905927+0000) 2022-09-08T02:40:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:14.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:14.854430+0000) 2022-09-08T02:40:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:15.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:15.906144+0000) 2022-09-08T02:40:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:15.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:15.854697+0000) 2022-09-08T02:40:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:15 smithi055 ceph-mon[97033]: pgmap v9746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:40:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:15 smithi059 ceph-mon[97450]: pgmap v9746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:40:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:16.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:16.906377+0000) 2022-09-08T02:40:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:16.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:16.854966+0000) 2022-09-08T02:40:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:17.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:17.906578+0000) 2022-09-08T02:40:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:17.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:17.855301+0000) 2022-09-08T02:40:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:17 smithi055 ceph-mon[97033]: pgmap v9747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:40:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:17 smithi059 ceph-mon[97450]: pgmap v9747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:40:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:18.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:18.906810+0000) 2022-09-08T02:40:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:18.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:18.855545+0000) 2022-09-08T02:40:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:19.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:19.907058+0000) 2022-09-08T02:40:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:19.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:19.855700+0000) 2022-09-08T02:40:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:19 smithi055 ceph-mon[97033]: pgmap v9748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:40:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:19 smithi059 ceph-mon[97450]: pgmap v9748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:40:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:20.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:20.907289+0000) 2022-09-08T02:40:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:20.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:20.855892+0000) 2022-09-08T02:40:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:21.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:21.907545+0000) 2022-09-08T02:40:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:21.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:21.856140+0000) 2022-09-08T02:40:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:21 smithi055 ceph-mon[97033]: pgmap v9749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:40:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:40:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:40:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:40:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:40:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:40:22.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:21 smithi059 ceph-mon[97450]: pgmap v9749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:40:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:40:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:40:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:40:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:40:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:40:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:22.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:22.907717+0000) 2022-09-08T02:40:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:22.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:22.856384+0000) 2022-09-08T02:40:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:40:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:23 smithi059 ceph-mon[97450]: pgmap v9750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:40:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:40:23.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:23 smithi055 ceph-mon[97033]: pgmap v9750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:40:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:23.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:23.907928+0000) 2022-09-08T02:40:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:23.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:23.856670+0000) 2022-09-08T02:40:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:24.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:24.908146+0000) 2022-09-08T02:40:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:24.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:24.856853+0000) 2022-09-08T02:40:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:25 smithi059 ceph-mon[97450]: pgmap v9751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:40:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:25 smithi055 ceph-mon[97033]: pgmap v9751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:40:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:25.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:25.908366+0000) 2022-09-08T02:40:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:25.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:25.857011+0000) 2022-09-08T02:40:26.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:40:26.337+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:40:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:26.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:26.908499+0000) 2022-09-08T02:40:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:26.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:26.857254+0000) 2022-09-08T02:40:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:27 smithi059 ceph-mon[97450]: pgmap v9752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:27 smithi055 ceph-mon[97033]: pgmap v9752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:27.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:27.908742+0000) 2022-09-08T02:40:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:27.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:27.857535+0000) 2022-09-08T02:40:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:28.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:28.908932+0000) 2022-09-08T02:40:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:28.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:28.857711+0000) 2022-09-08T02:40:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:29.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:29.909142+0000) 2022-09-08T02:40:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:29.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:29.857908+0000) 2022-09-08T02:40:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:29 smithi055 ceph-mon[97033]: pgmap v9753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:29 smithi059 ceph-mon[97450]: pgmap v9753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:30.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:30.909367+0000) 2022-09-08T02:40:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:30.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:30.858153+0000) 2022-09-08T02:40:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:31.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:31.909659+0000) 2022-09-08T02:40:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:31.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:31.858423+0000) 2022-09-08T02:40:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:31 smithi055 ceph-mon[97033]: pgmap v9754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:40:32.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:31 smithi059 ceph-mon[97450]: pgmap v9754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:40:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:32.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:32.909897+0000) 2022-09-08T02:40:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:32.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:32.858676+0000) 2022-09-08T02:40:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:33.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:33.910070+0000) 2022-09-08T02:40:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:33.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:33.858890+0000) 2022-09-08T02:40:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:33 smithi055 ceph-mon[97033]: pgmap v9755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:34.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:33 smithi059 ceph-mon[97450]: pgmap v9755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:34.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:34.910274+0000) 2022-09-08T02:40:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:34.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:34.859048+0000) 2022-09-08T02:40:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:35.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:35.910474+0000) 2022-09-08T02:40:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:35.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:35.859185+0000) 2022-09-08T02:40:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:35 smithi055 ceph-mon[97033]: pgmap v9756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:40:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:35 smithi059 ceph-mon[97450]: pgmap v9756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:40:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:36.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:36.910736+0000) 2022-09-08T02:40:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:36.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:36.859421+0000) 2022-09-08T02:40:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:37.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:37.859630+0000) 2022-09-08T02:40:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:37.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:37.910949+0000) 2022-09-08T02:40:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:37 smithi055 ceph-mon[97033]: pgmap v9757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:38.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:37 smithi059 ceph-mon[97450]: pgmap v9757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:38.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:38.911197+0000) 2022-09-08T02:40:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:38.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:38.859913+0000) 2022-09-08T02:40:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:39 smithi059 ceph-mon[97450]: pgmap v9758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:39 smithi055 ceph-mon[97033]: pgmap v9758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:39.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:39.911428+0000) 2022-09-08T02:40:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:39.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:39.860113+0000) 2022-09-08T02:40:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:40.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:40.911575+0000) 2022-09-08T02:40:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:40.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:40.860331+0000) 2022-09-08T02:40:41.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:41 smithi059 ceph-mon[97450]: pgmap v9759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:40:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:41 smithi055 ceph-mon[97033]: pgmap v9759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:40:41.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:40:41.349+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:40:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:41.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:41.911710+0000) 2022-09-08T02:40:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:41.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:41.860548+0000) 2022-09-08T02:40:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:42.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:42.911892+0000) 2022-09-08T02:40:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:42.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:42.860782+0000) 2022-09-08T02:40:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:43.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:43.912127+0000) 2022-09-08T02:40:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:43.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:43.861019+0000) 2022-09-08T02:40:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:43 smithi055 ceph-mon[97033]: pgmap v9760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:43 smithi059 ceph-mon[97450]: pgmap v9760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:44.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:44.912334+0000) 2022-09-08T02:40:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:44.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:44.861261+0000) 2022-09-08T02:40:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:45.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:45.912500+0000) 2022-09-08T02:40:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:45.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:45.861466+0000) 2022-09-08T02:40:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:45 smithi055 ceph-mon[97033]: pgmap v9761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:40:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:45 smithi059 ceph-mon[97450]: pgmap v9761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:40:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:46.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:46.912711+0000) 2022-09-08T02:40:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:46.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:46.861678+0000) 2022-09-08T02:40:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:47.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:47.912897+0000) 2022-09-08T02:40:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:47.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:47.861932+0000) 2022-09-08T02:40:48.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:47 smithi055 ceph-mon[97033]: pgmap v9762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:47 smithi059 ceph-mon[97450]: pgmap v9762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:48.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:48.913133+0000) 2022-09-08T02:40:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:48.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:48.862185+0000) 2022-09-08T02:40:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:49.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:49.913388+0000) 2022-09-08T02:40:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:49.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:49.862411+0000) 2022-09-08T02:40:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:49 smithi055 ceph-mon[97033]: pgmap v9763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:49 smithi059 ceph-mon[97450]: pgmap v9763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:50.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:50.913624+0000) 2022-09-08T02:40:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:50.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:50.862626+0000) 2022-09-08T02:40:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:51.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:51.913871+0000) 2022-09-08T02:40:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:51.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:51.862828+0000) 2022-09-08T02:40:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:51 smithi055 ceph-mon[97033]: pgmap v9764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:40:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:51 smithi059 ceph-mon[97450]: pgmap v9764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:40:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:52.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:52.914047+0000) 2022-09-08T02:40:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:52.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:52.863015+0000) 2022-09-08T02:40:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:53.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:53.914260+0000) 2022-09-08T02:40:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:53.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:53.863248+0000) 2022-09-08T02:40:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:53 smithi055 ceph-mon[97033]: pgmap v9765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:53 smithi059 ceph-mon[97450]: pgmap v9765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:54.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:54.914463+0000) 2022-09-08T02:40:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:54.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:54.863468+0000) 2022-09-08T02:40:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:55.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:55.914622+0000) 2022-09-08T02:40:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:55.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:55.863747+0000) 2022-09-08T02:40:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:55 smithi055 ceph-mon[97033]: pgmap v9766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:40:56.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:55 smithi059 ceph-mon[97450]: pgmap v9766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:40:56.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:40:56.357+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:40:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:56.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:56.914792+0000) 2022-09-08T02:40:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:56.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:56.863974+0000) 2022-09-08T02:40:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:56 smithi059 ceph-mon[97450]: pgmap v9767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:56 smithi055 ceph-mon[97033]: pgmap v9767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:40:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:57.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:57.915028+0000) 2022-09-08T02:40:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:57.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:57.864258+0000) 2022-09-08T02:40:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:58.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:58.915300+0000) 2022-09-08T02:40:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:58.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:58.864498+0000) 2022-09-08T02:41:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:40:59.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:59.915474+0000) 2022-09-08T02:41:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:40:59.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:40:59.864736+0000) 2022-09-08T02:41:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:40:59 smithi055 ceph-mon[97033]: pgmap v9768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:00.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:40:59 smithi059 ceph-mon[97450]: pgmap v9768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:00.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:00.915606+0000) 2022-09-08T02:41:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:00.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:00.864921+0000) 2022-09-08T02:41:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:01.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:01.915853+0000) 2022-09-08T02:41:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:01.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:01.865089+0000) 2022-09-08T02:41:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:01 smithi055 ceph-mon[97033]: pgmap v9769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:41:02.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:01 smithi059 ceph-mon[97450]: pgmap v9769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:41:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:02.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:02.916069+0000) 2022-09-08T02:41:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:02.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:02.865259+0000) 2022-09-08T02:41:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:03.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:03.916252+0000) 2022-09-08T02:41:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:03.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:03.865423+0000) 2022-09-08T02:41:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:03 smithi055 ceph-mon[97033]: pgmap v9770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:03 smithi059 ceph-mon[97450]: pgmap v9770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:04.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:04.916471+0000) 2022-09-08T02:41:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:04.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:04.865587+0000) 2022-09-08T02:41:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:05.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:05.916718+0000) 2022-09-08T02:41:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:05.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:05.865843+0000) 2022-09-08T02:41:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:05 smithi055 ceph-mon[97033]: pgmap v9771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:41:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:41:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:41:06.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:05 smithi059 ceph-mon[97450]: pgmap v9771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:41:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:41:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:41:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:06.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:06.916972+0000) 2022-09-08T02:41:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:06.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:06.866075+0000) 2022-09-08T02:41:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:07.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:07.917182+0000) 2022-09-08T02:41:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:07.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:07.866317+0000) 2022-09-08T02:41:08.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:07 smithi059 ceph-mon[97450]: pgmap v9772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:41:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:07 smithi055 ceph-mon[97033]: pgmap v9772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:41:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:08.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:08.917353+0000) 2022-09-08T02:41:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:08.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:08.866520+0000) 2022-09-08T02:41:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:08 smithi059 ceph-mon[97450]: pgmap v9773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:41:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:08 smithi055 ceph-mon[97033]: pgmap v9773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:41:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:09.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:09.866754+0000) 2022-09-08T02:41:10.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:09.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:09.917542+0000) 2022-09-08T02:41:10.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:09 smithi059 ceph-mon[97450]: osdmap e691: 12 total, 11 up, 11 in 2022-09-08T02:41:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:09 smithi055 ceph-mon[97033]: osdmap e691: 12 total, 11 up, 11 in 2022-09-08T02:41:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:10.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:10.917722+0000) 2022-09-08T02:41:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:10.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:10.866961+0000) 2022-09-08T02:41:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:11 smithi059 ceph-mon[97450]: pgmap v9775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:41:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:11 smithi055 ceph-mon[97033]: pgmap v9775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:41:11.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:41:11.374+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:41:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:11.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:11.917885+0000) 2022-09-08T02:41:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:11.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:11.867140+0000) 2022-09-08T02:41:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:12.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:12.918056+0000) 2022-09-08T02:41:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:12.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:12.867333+0000) 2022-09-08T02:41:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:13.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:13.918267+0000) 2022-09-08T02:41:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:13.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:13.867566+0000) 2022-09-08T02:41:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:13 smithi055 ceph-mon[97033]: pgmap v9776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:41:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:13 smithi055 ceph-mon[97033]: osdmap e692: 12 total, 11 up, 11 in 2022-09-08T02:41:14.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:13 smithi059 ceph-mon[97450]: pgmap v9776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:41:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:13 smithi059 ceph-mon[97450]: osdmap e692: 12 total, 11 up, 11 in 2022-09-08T02:41:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:14.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:14.867755+0000) 2022-09-08T02:41:15.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:14.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:14.918508+0000) 2022-09-08T02:41:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:15.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:15.918740+0000) 2022-09-08T02:41:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:15.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:15.867947+0000) 2022-09-08T02:41:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:15 smithi055 ceph-mon[97033]: pgmap v9778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:41:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:15 smithi059 ceph-mon[97450]: pgmap v9778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:41:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:16.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:16.918913+0000) 2022-09-08T02:41:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:16.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:16.868189+0000) 2022-09-08T02:41:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:17.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:17.919076+0000) 2022-09-08T02:41:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:17.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:17.868429+0000) 2022-09-08T02:41:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:17 smithi055 ceph-mon[97033]: pgmap v9779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:41:18.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:17 smithi059 ceph-mon[97450]: pgmap v9779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:41:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:18.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:18.868684+0000) 2022-09-08T02:41:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:18.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:18.919318+0000) 2022-09-08T02:41:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:18 smithi059 ceph-mon[97450]: pgmap v9780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:41:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:18 smithi055 ceph-mon[97033]: pgmap v9780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:41:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:19.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:19.919561+0000) 2022-09-08T02:41:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:19.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:19.868836+0000) 2022-09-08T02:41:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:20.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:20.869074+0000) 2022-09-08T02:41:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:20.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:20.919790+0000) 2022-09-08T02:41:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:21 smithi059 ceph-mon[97450]: pgmap v9781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:41:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:21 smithi055 ceph-mon[97033]: pgmap v9781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:41:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:21.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:21.920000+0000) 2022-09-08T02:41:22.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:21.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:21.869314+0000) 2022-09-08T02:41:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:41:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:41:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:41:22.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:41:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:41:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:41:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:22.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:22.920236+0000) 2022-09-08T02:41:23.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:22.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:22.869564+0000) 2022-09-08T02:41:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:41:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:23 smithi055 ceph-mon[97033]: pgmap v9782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:41:24.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:41:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:23 smithi059 ceph-mon[97450]: pgmap v9782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:41:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:23.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:23.920477+0000) 2022-09-08T02:41:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:23.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:23.869805+0000) 2022-09-08T02:41:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:24.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:24.920732+0000) 2022-09-08T02:41:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:24.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:24.870051+0000) 2022-09-08T02:41:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:25 smithi059 ceph-mon[97450]: pgmap v9783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:41:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:25 smithi055 ceph-mon[97033]: pgmap v9783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:41:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:25.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:25.870262+0000) 2022-09-08T02:41:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:25.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:25.920961+0000) 2022-09-08T02:41:26.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:41:26.384+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:41:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:26.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:26.921123+0000) 2022-09-08T02:41:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:26.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:26.870488+0000) 2022-09-08T02:41:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:27 smithi059 ceph-mon[97450]: pgmap v9784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:27 smithi055 ceph-mon[97033]: pgmap v9784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:27.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:27.921322+0000) 2022-09-08T02:41:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:27.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:27.870714+0000) 2022-09-08T02:41:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:28.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:28.921544+0000) 2022-09-08T02:41:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:28.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:28.870899+0000) 2022-09-08T02:41:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:29.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:29.921811+0000) 2022-09-08T02:41:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:29.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:29.871031+0000) 2022-09-08T02:41:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:29 smithi055 ceph-mon[97033]: pgmap v9785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:29 smithi059 ceph-mon[97450]: pgmap v9785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:30.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:30.921982+0000) 2022-09-08T02:41:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:30.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:30.871236+0000) 2022-09-08T02:41:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:31.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:31.922207+0000) 2022-09-08T02:41:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:31.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:31.871508+0000) 2022-09-08T02:41:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:31 smithi059 ceph-mon[97450]: pgmap v9786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:41:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:31 smithi055 ceph-mon[97033]: pgmap v9786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:41:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:32.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:32.871695+0000) 2022-09-08T02:41:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:32.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:32.922343+0000) 2022-09-08T02:41:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:33.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:33.922591+0000) 2022-09-08T02:41:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:33.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:33.871968+0000) 2022-09-08T02:41:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:33 smithi055 ceph-mon[97033]: pgmap v9787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:34.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:33 smithi059 ceph-mon[97450]: pgmap v9787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:34.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:34.922753+0000) 2022-09-08T02:41:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:34.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:34.872227+0000) 2022-09-08T02:41:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:35.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:35.922959+0000) 2022-09-08T02:41:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:35.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:35.872448+0000) 2022-09-08T02:41:36.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:35 smithi059 ceph-mon[97450]: pgmap v9788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:41:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:35 smithi055 ceph-mon[97033]: pgmap v9788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:41:37.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:36.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:36.872717+0000) 2022-09-08T02:41:37.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:36.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:36.923200+0000) 2022-09-08T02:41:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:37 smithi055 ceph-mon[97033]: pgmap v9789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:37 smithi059 ceph-mon[97450]: pgmap v9789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:37.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:37.923442+0000) 2022-09-08T02:41:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:37.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:37.873011+0000) 2022-09-08T02:41:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:38.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:38.923710+0000) 2022-09-08T02:41:39.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:38.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:38.873210+0000) 2022-09-08T02:41:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:39.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:39.923909+0000) 2022-09-08T02:41:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:39.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:39.873387+0000) 2022-09-08T02:41:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:39 smithi055 ceph-mon[97033]: pgmap v9790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:39 smithi059 ceph-mon[97450]: pgmap v9790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:40.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:40.924107+0000) 2022-09-08T02:41:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:40.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:40.873612+0000) 2022-09-08T02:41:41.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:41:41.398+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:41:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:41.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:41.924377+0000) 2022-09-08T02:41:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:41.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:41.873933+0000) 2022-09-08T02:41:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:41 smithi059 ceph-mon[97450]: pgmap v9791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:41:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:41 smithi055 ceph-mon[97033]: pgmap v9791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:41:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:42.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:42.874091+0000) 2022-09-08T02:41:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:42.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:42.924617+0000) 2022-09-08T02:41:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:43.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:43.874334+0000) 2022-09-08T02:41:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:43.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:43.924927+0000) 2022-09-08T02:41:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:43 smithi059 ceph-mon[97450]: pgmap v9792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:43 smithi055 ceph-mon[97033]: pgmap v9792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:44.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:44.874584+0000) 2022-09-08T02:41:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:44.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:44.925225+0000) 2022-09-08T02:41:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:45.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:45.925432+0000) 2022-09-08T02:41:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:45.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:45.874809+0000) 2022-09-08T02:41:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:45 smithi059 ceph-mon[97450]: pgmap v9793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:41:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:45 smithi055 ceph-mon[97033]: pgmap v9793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:41:47.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:46.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:46.925694+0000) 2022-09-08T02:41:47.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:46.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:46.874937+0000) 2022-09-08T02:41:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:47 smithi055 ceph-mon[97033]: pgmap v9794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:47 smithi059 ceph-mon[97450]: pgmap v9794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:47.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:47.925811+0000) 2022-09-08T02:41:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:47.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:47.875161+0000) 2022-09-08T02:41:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:48.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:48.926048+0000) 2022-09-08T02:41:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:48.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:48.875424+0000) 2022-09-08T02:41:49.290 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:48 smithi059 ceph-mon[97450]: pgmap v9795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:49 smithi055 ceph-mon[97033]: pgmap v9795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:49.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:49.926260+0000) 2022-09-08T02:41:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:49.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:49.875648+0000) 2022-09-08T02:41:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:50.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:50.926469+0000) 2022-09-08T02:41:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:50.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:50.875850+0000) 2022-09-08T02:41:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:51 smithi055 ceph-mon[97033]: pgmap v9796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:41:51.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:51 smithi059 ceph-mon[97450]: pgmap v9796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:41:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:51.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:51.926744+0000) 2022-09-08T02:41:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:51.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:51.876012+0000) 2022-09-08T02:41:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:52.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:52.876221+0000) 2022-09-08T02:41:53.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:52.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:52.926950+0000) 2022-09-08T02:41:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:53.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:53.927129+0000) 2022-09-08T02:41:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:53.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:53.876460+0000) 2022-09-08T02:41:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:53 smithi059 ceph-mon[97450]: pgmap v9797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:53 smithi055 ceph-mon[97033]: pgmap v9797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:54.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:54.927332+0000) 2022-09-08T02:41:55.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:54.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:54.876716+0000) 2022-09-08T02:41:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:55 smithi055 ceph-mon[97033]: pgmap v9798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:41:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:55 smithi059 ceph-mon[97450]: pgmap v9798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:41:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:55.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:55.876992+0000) 2022-09-08T02:41:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:55.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:55.927553+0000) 2022-09-08T02:41:56.877 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:41:56.415+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:41:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:56.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:56.927740+0000) 2022-09-08T02:41:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:56.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:56.877208+0000) 2022-09-08T02:41:57.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:57 smithi055 ceph-mon[97033]: pgmap v9799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:57 smithi059 ceph-mon[97450]: pgmap v9799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:41:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:57.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:57.927857+0000) 2022-09-08T02:41:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:57.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:57.877459+0000) 2022-09-08T02:41:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:58.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:58.928089+0000) 2022-09-08T02:41:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:58.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:58.877738+0000) 2022-09-08T02:42:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:41:59.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:59.928269+0000) 2022-09-08T02:42:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:41:59.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:41:59.878006+0000) 2022-09-08T02:42:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:41:59 smithi059 ceph-mon[97450]: pgmap v9800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:41:59 smithi055 ceph-mon[97033]: pgmap v9800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:00.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:00.878251+0000) 2022-09-08T02:42:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:00.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:00.928488+0000) 2022-09-08T02:42:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:00 smithi059 ceph-mon[97450]: pgmap v9801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:42:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:00 smithi055 ceph-mon[97033]: pgmap v9801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:42:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:01.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:01.928710+0000) 2022-09-08T02:42:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:01.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:01.878509+0000) 2022-09-08T02:42:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:02.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:02.928944+0000) 2022-09-08T02:42:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:02.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:02.878790+0000) 2022-09-08T02:42:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:03.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:03.929202+0000) 2022-09-08T02:42:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:03.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:03.878963+0000) 2022-09-08T02:42:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:03 smithi059 ceph-mon[97450]: pgmap v9802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:03 smithi055 ceph-mon[97033]: pgmap v9802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:04.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:04.929442+0000) 2022-09-08T02:42:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:04.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:04.879200+0000) 2022-09-08T02:42:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:05 smithi055 ceph-mon[97033]: pgmap v9803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:42:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:05 smithi059 ceph-mon[97450]: pgmap v9803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:42:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:05.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:05.929651+0000) 2022-09-08T02:42:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:05.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:05.879481+0000) 2022-09-08T02:42:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:42:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:42:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:42:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:42:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:06.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:06.929817+0000) 2022-09-08T02:42:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:06.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:06.879683+0000) 2022-09-08T02:42:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:07 smithi055 ceph-mon[97033]: pgmap v9804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:42:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:07 smithi059 ceph-mon[97450]: pgmap v9804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:42:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:07.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:07.929984+0000) 2022-09-08T02:42:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:07.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:07.879941+0000) 2022-09-08T02:42:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:08.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:08.930155+0000) 2022-09-08T02:42:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:08.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:08.880211+0000) 2022-09-08T02:42:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:09 smithi055 ceph-mon[97033]: pgmap v9805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:42:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:09 smithi055 ceph-mon[97033]: osdmap e693: 12 total, 11 up, 11 in 2022-09-08T02:42:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:09 smithi059 ceph-mon[97450]: pgmap v9805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:42:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:09 smithi059 ceph-mon[97450]: osdmap e693: 12 total, 11 up, 11 in 2022-09-08T02:42:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:09.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:09.930325+0000) 2022-09-08T02:42:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:09.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:09.880467+0000) 2022-09-08T02:42:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:10.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:10.930510+0000) 2022-09-08T02:42:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:10.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:10.880767+0000) 2022-09-08T02:42:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:11 smithi055 ceph-mon[97033]: pgmap v9807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:42:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:11 smithi059 ceph-mon[97450]: pgmap v9807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:42:11.881 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:42:11.430+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:42:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:11.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:11.881061+0000) 2022-09-08T02:42:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:11.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:11.930737+0000) 2022-09-08T02:42:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:12.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:12.930942+0000) 2022-09-08T02:42:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:12.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:12.881330+0000) 2022-09-08T02:42:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:13.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:13.931146+0000) 2022-09-08T02:42:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:13.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:13.881599+0000) 2022-09-08T02:42:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:13 smithi059 ceph-mon[97450]: pgmap v9808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:42:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:13 smithi059 ceph-mon[97450]: osdmap e694: 12 total, 11 up, 11 in 2022-09-08T02:42:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:13 smithi055 ceph-mon[97033]: pgmap v9808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:42:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:13 smithi055 ceph-mon[97033]: osdmap e694: 12 total, 11 up, 11 in 2022-09-08T02:42:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:14.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:14.931289+0000) 2022-09-08T02:42:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:14.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:14.881840+0000) 2022-09-08T02:42:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:15 smithi055 ceph-mon[97033]: pgmap v9810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:42:15.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:15 smithi059 ceph-mon[97450]: pgmap v9810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:42:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:15.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:15.931514+0000) 2022-09-08T02:42:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:15.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:15.882091+0000) 2022-09-08T02:42:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:16.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:16.931734+0000) 2022-09-08T02:42:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:16.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:16.882280+0000) 2022-09-08T02:42:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:17 smithi055 ceph-mon[97033]: pgmap v9811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:42:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:17 smithi059 ceph-mon[97450]: pgmap v9811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:42:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:17.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:17.931976+0000) 2022-09-08T02:42:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:17.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:17.882501+0000) 2022-09-08T02:42:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:18.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:18.932173+0000) 2022-09-08T02:42:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:18.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:18.882754+0000) 2022-09-08T02:42:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:19 smithi055 ceph-mon[97033]: pgmap v9812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:42:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:19 smithi059 ceph-mon[97450]: pgmap v9812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:42:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:19.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:19.932369+0000) 2022-09-08T02:42:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:19.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:19.882953+0000) 2022-09-08T02:42:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:20.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:20.932558+0000) 2022-09-08T02:42:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:20.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:20.883198+0000) 2022-09-08T02:42:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:21 smithi055 ceph-mon[97033]: pgmap v9813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:42:21.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:21 smithi059 ceph-mon[97450]: pgmap v9813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:42:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:21.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:21.932776+0000) 2022-09-08T02:42:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:21.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:21.883428+0000) 2022-09-08T02:42:23.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:42:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:42:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:42:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:22.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:22.932981+0000) 2022-09-08T02:42:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:22.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:22.883648+0000) 2022-09-08T02:42:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:42:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:42:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:42:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:23.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:23.883929+0000) 2022-09-08T02:42:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:23.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:23.933170+0000) 2022-09-08T02:42:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:23 smithi059 ceph-mon[97450]: pgmap v9814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:42:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:42:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:23 smithi055 ceph-mon[97033]: pgmap v9814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:42:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:42:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:24.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:24.933395+0000) 2022-09-08T02:42:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:24.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:24.884194+0000) 2022-09-08T02:42:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:24 smithi059 ceph-mon[97450]: pgmap v9815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:42:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:25 smithi055 ceph-mon[97033]: pgmap v9815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:42:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:25.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:25.933600+0000) 2022-09-08T02:42:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:25.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:25.884481+0000) 2022-09-08T02:42:26.884 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:42:26.445+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:42:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:26.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:26.933827+0000) 2022-09-08T02:42:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:26.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:26.884727+0000) 2022-09-08T02:42:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:27 smithi055 ceph-mon[97033]: pgmap v9816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:27 smithi059 ceph-mon[97450]: pgmap v9816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:27.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:27.885010+0000) 2022-09-08T02:42:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:27.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:27.934039+0000) 2022-09-08T02:42:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:28.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:28.934282+0000) 2022-09-08T02:42:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:28.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:28.885225+0000) 2022-09-08T02:42:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:29 smithi055 ceph-mon[97033]: pgmap v9817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:29.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:29 smithi059 ceph-mon[97450]: pgmap v9817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:29.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:29.934501+0000) 2022-09-08T02:42:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:29.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:29.885470+0000) 2022-09-08T02:42:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:30.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:30.934742+0000) 2022-09-08T02:42:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:30.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:30.885700+0000) 2022-09-08T02:42:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:31 smithi055 ceph-mon[97033]: pgmap v9818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:42:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:31 smithi059 ceph-mon[97450]: pgmap v9818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:42:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:31.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:31.885931+0000) 2022-09-08T02:42:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:31.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:31.934943+0000) 2022-09-08T02:42:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:32.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:32.935085+0000) 2022-09-08T02:42:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:32.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:32.886115+0000) 2022-09-08T02:42:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:33 smithi055 ceph-mon[97033]: pgmap v9819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:33 smithi059 ceph-mon[97450]: pgmap v9819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:33.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:33.935298+0000) 2022-09-08T02:42:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:33.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:33.886363+0000) 2022-09-08T02:42:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:34.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:34.935530+0000) 2022-09-08T02:42:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:34.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:34.886567+0000) 2022-09-08T02:42:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:35 smithi055 ceph-mon[97033]: pgmap v9820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:42:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:35 smithi059 ceph-mon[97450]: pgmap v9820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:42:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:35.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:35.935772+0000) 2022-09-08T02:42:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:35.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:35.886806+0000) 2022-09-08T02:42:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:36.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:36.935932+0000) 2022-09-08T02:42:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:36.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:36.886988+0000) 2022-09-08T02:42:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:37 smithi055 ceph-mon[97033]: pgmap v9821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:37 smithi059 ceph-mon[97450]: pgmap v9821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:37.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:37.936159+0000) 2022-09-08T02:42:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:37.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:37.887167+0000) 2022-09-08T02:42:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:38.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:38.936392+0000) 2022-09-08T02:42:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:38.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:38.887381+0000) 2022-09-08T02:42:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:38 smithi059 ceph-mon[97450]: pgmap v9822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:39 smithi055 ceph-mon[97033]: pgmap v9822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:39.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:39.936617+0000) 2022-09-08T02:42:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:39.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:39.887682+0000) 2022-09-08T02:42:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:40.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:40.936762+0000) 2022-09-08T02:42:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:40.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:40.887925+0000) 2022-09-08T02:42:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:41 smithi055 ceph-mon[97033]: pgmap v9823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:42:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:41 smithi059 ceph-mon[97450]: pgmap v9823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:42:41.888 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:42:41.447+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:42:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:41.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:41.936983+0000) 2022-09-08T02:42:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:41.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:41.888128+0000) 2022-09-08T02:42:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:42.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:42.937171+0000) 2022-09-08T02:42:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:42.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:42.888286+0000) 2022-09-08T02:42:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:43 smithi055 ceph-mon[97033]: pgmap v9824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:43.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:43 smithi059 ceph-mon[97450]: pgmap v9824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:43.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:43.937359+0000) 2022-09-08T02:42:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:43.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:43.888499+0000) 2022-09-08T02:42:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:44.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:44.888750+0000) 2022-09-08T02:42:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:44.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:44.937592+0000) 2022-09-08T02:42:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:45 smithi055 ceph-mon[97033]: pgmap v9825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:42:45.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:45 smithi059 ceph-mon[97450]: pgmap v9825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:42:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:45.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:45.937830+0000) 2022-09-08T02:42:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:45.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:45.889040+0000) 2022-09-08T02:42:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:46.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:46.938033+0000) 2022-09-08T02:42:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:46.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:46.889272+0000) 2022-09-08T02:42:47.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:47 smithi055 ceph-mon[97033]: pgmap v9826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:47 smithi059 ceph-mon[97450]: pgmap v9826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:47.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:47.938265+0000) 2022-09-08T02:42:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:47.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:47.889550+0000) 2022-09-08T02:42:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:48.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:48.938510+0000) 2022-09-08T02:42:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:48.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:48.889766+0000) 2022-09-08T02:42:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:48 smithi059 ceph-mon[97450]: pgmap v9827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:49 smithi055 ceph-mon[97033]: pgmap v9827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:42:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:49.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:49.938785+0000) 2022-09-08T02:42:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:49.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:49.890072+0000) 2022-09-08T02:42:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:50.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:50.939074+0000) 2022-09-08T02:42:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:50.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:50.890281+0000) 2022-09-08T02:42:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:51 smithi055 ceph-mon[97033]: pgmap v9828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:42:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:51 smithi059 ceph-mon[97450]: pgmap v9828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:42:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:51.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:51.939339+0000) 2022-09-08T02:42:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:51.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:51.890558+0000) 2022-09-08T02:42:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:52.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:52.939558+0000) 2022-09-08T02:42:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:52.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:52.890801+0000) 2022-09-08T02:42:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:53 smithi055 ceph-mon[97033]: pgmap v9829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:42:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:53 smithi059 ceph-mon[97450]: pgmap v9829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:42:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:53.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:53.939747+0000) 2022-09-08T02:42:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:53.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:53.891030+0000) 2022-09-08T02:42:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:54.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:54.940009+0000) 2022-09-08T02:42:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:54.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:54.891259+0000) 2022-09-08T02:42:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:55 smithi055 ceph-mon[97033]: pgmap v9830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:42:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:55 smithi059 ceph-mon[97450]: pgmap v9830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:42:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:55.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:55.940237+0000) 2022-09-08T02:42:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:55.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:55.891497+0000) 2022-09-08T02:42:56.891 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:42:56.450+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:42:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:56.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:56.891703+0000) 2022-09-08T02:42:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:56.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:56.940477+0000) 2022-09-08T02:42:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:57 smithi055 ceph-mon[97033]: pgmap v9831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:42:57.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:57 smithi059 ceph-mon[97450]: pgmap v9831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:42:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:57.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:57.940729+0000) 2022-09-08T02:42:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:57.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:57.891943+0000) 2022-09-08T02:42:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:58.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:58.940830+0000) 2022-09-08T02:42:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:58.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:58.892198+0000) 2022-09-08T02:42:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:42:59 smithi059 ceph-mon[97450]: pgmap v9832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:42:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:42:59 smithi055 ceph-mon[97033]: pgmap v9832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:43:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:42:59.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:59.941035+0000) 2022-09-08T02:43:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:42:59.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:42:59.892373+0000) 2022-09-08T02:43:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:00.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:00.892587+0000) 2022-09-08T02:43:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:00.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:00.941236+0000) 2022-09-08T02:43:01.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:01 smithi055 ceph-mon[97033]: pgmap v9833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:43:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:01 smithi059 ceph-mon[97450]: pgmap v9833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:43:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:01.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:01.892856+0000) 2022-09-08T02:43:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:01.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:01.941513+0000) 2022-09-08T02:43:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:02.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:02.941772+0000) 2022-09-08T02:43:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:02.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:02.893077+0000) 2022-09-08T02:43:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:03 smithi055 ceph-mon[97033]: pgmap v9834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:43:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:03 smithi059 ceph-mon[97450]: pgmap v9834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:43:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:03.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:03.941918+0000) 2022-09-08T02:43:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:03.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:03.893310+0000) 2022-09-08T02:43:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:04.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:04.942090+0000) 2022-09-08T02:43:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:04.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:04.893469+0000) 2022-09-08T02:43:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:05 smithi055 ceph-mon[97033]: pgmap v9835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:43:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:05 smithi059 ceph-mon[97450]: pgmap v9835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:43:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:05.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:05.942291+0000) 2022-09-08T02:43:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:05.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:05.893723+0000) 2022-09-08T02:43:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:43:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:43:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:43:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:43:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:06.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:06.942413+0000) 2022-09-08T02:43:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:06.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:06.893898+0000) 2022-09-08T02:43:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:07.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:07.894203+0000) 2022-09-08T02:43:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:07.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:07.942570+0000) 2022-09-08T02:43:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:08 smithi055 ceph-mon[97033]: pgmap v9836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:43:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:08 smithi059 ceph-mon[97450]: pgmap v9836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:43:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:08.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:08.942745+0000) 2022-09-08T02:43:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:08.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:08.894431+0000) 2022-09-08T02:43:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:09.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:09.942986+0000) 2022-09-08T02:43:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:09.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:09.894677+0000) 2022-09-08T02:43:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:10 smithi055 ceph-mon[97033]: pgmap v9837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:43:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:10 smithi055 ceph-mon[97033]: osdmap e695: 12 total, 11 up, 11 in 2022-09-08T02:43:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:10 smithi059 ceph-mon[97450]: pgmap v9837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:43:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:10 smithi059 ceph-mon[97450]: osdmap e695: 12 total, 11 up, 11 in 2022-09-08T02:43:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:10.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:10.943176+0000) 2022-09-08T02:43:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:10.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:10.894859+0000) 2022-09-08T02:43:11.895 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:43:11.461+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:43:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:11.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:11.943435+0000) 2022-09-08T02:43:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:11.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:11.895048+0000) 2022-09-08T02:43:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:12 smithi055 ceph-mon[97033]: pgmap v9839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:43:12.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:12 smithi059 ceph-mon[97450]: pgmap v9839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:43:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:12.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:12.943710+0000) 2022-09-08T02:43:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:12.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:12.895215+0000) 2022-09-08T02:43:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:13.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:13.943954+0000) 2022-09-08T02:43:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:13.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:13.895364+0000) 2022-09-08T02:43:14.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:14 smithi059 ceph-mon[97450]: pgmap v9840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:43:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:14 smithi059 ceph-mon[97450]: osdmap e696: 12 total, 11 up, 11 in 2022-09-08T02:43:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:14 smithi055 ceph-mon[97033]: pgmap v9840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:43:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:14 smithi055 ceph-mon[97033]: osdmap e696: 12 total, 11 up, 11 in 2022-09-08T02:43:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:14.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:14.944132+0000) 2022-09-08T02:43:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:14.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:14.895600+0000) 2022-09-08T02:43:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:15.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:15.944372+0000) 2022-09-08T02:43:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:15.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:15.895844+0000) 2022-09-08T02:43:16.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:16 smithi059 ceph-mon[97450]: pgmap v9842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:43:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:16 smithi055 ceph-mon[97033]: pgmap v9842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:43:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:16.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:16.944534+0000) 2022-09-08T02:43:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:16.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:16.896033+0000) 2022-09-08T02:43:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:17.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:17.944746+0000) 2022-09-08T02:43:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:17.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:17.896264+0000) 2022-09-08T02:43:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:18 smithi059 ceph-mon[97450]: pgmap v9843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:43:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:18 smithi055 ceph-mon[97033]: pgmap v9843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:43:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:18.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:18.944938+0000) 2022-09-08T02:43:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:18.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:18.896468+0000) 2022-09-08T02:43:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:19.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:19.945185+0000) 2022-09-08T02:43:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:19.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:19.896670+0000) 2022-09-08T02:43:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:20 smithi059 ceph-mon[97450]: pgmap v9844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T02:43:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:20 smithi055 ceph-mon[97033]: pgmap v9844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T02:43:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:20.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:20.945460+0000) 2022-09-08T02:43:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:20.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:20.896851+0000) 2022-09-08T02:43:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:21.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:21.945696+0000) 2022-09-08T02:43:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:21.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:21.896971+0000) 2022-09-08T02:43:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:22 smithi059 ceph-mon[97450]: pgmap v9845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:43:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:22 smithi055 ceph-mon[97033]: pgmap v9845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:43:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:22.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:22.897136+0000) 2022-09-08T02:43:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:22.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:22.945921+0000) 2022-09-08T02:43:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:23.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:23.897289+0000) 2022-09-08T02:43:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:23.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:23.946088+0000) 2022-09-08T02:43:24.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:24 smithi055 ceph-mon[97033]: pgmap v9846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:43:24.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:43:24.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:43:24.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:43:24.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:43:24.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:24 smithi059 ceph-mon[97450]: pgmap v9846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:43:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:43:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:43:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:43:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:43:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:24.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:24.946316+0000) 2022-09-08T02:43:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:24.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:24.897444+0000) 2022-09-08T02:43:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:25.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:25.946562+0000) 2022-09-08T02:43:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:25.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:25.897628+0000) 2022-09-08T02:43:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:26 smithi059 ceph-mon[97450]: pgmap v9847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:43:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:26 smithi055 ceph-mon[97033]: pgmap v9847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:43:26.898 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:43:26.478+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:43:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:26.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:26.946800+0000) 2022-09-08T02:43:27.173 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:26.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:26.897882+0000) 2022-09-08T02:43:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:27.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:27.947014+0000) 2022-09-08T02:43:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:27.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:27.898137+0000) 2022-09-08T02:43:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:28 smithi059 ceph-mon[97450]: pgmap v9848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:28 smithi055 ceph-mon[97033]: pgmap v9848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:28.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:28.898281+0000) 2022-09-08T02:43:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:28.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:28.947158+0000) 2022-09-08T02:43:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:29.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:29.947374+0000) 2022-09-08T02:43:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:29.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:29.898518+0000) 2022-09-08T02:43:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:30 smithi059 ceph-mon[97450]: pgmap v9849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:30 smithi055 ceph-mon[97033]: pgmap v9849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:30.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:30.898688+0000) 2022-09-08T02:43:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:30.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:30.947569+0000) 2022-09-08T02:43:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:31.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:31.898889+0000) 2022-09-08T02:43:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:31.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:31.947757+0000) 2022-09-08T02:43:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:32 smithi055 ceph-mon[97033]: pgmap v9850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:43:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:32 smithi059 ceph-mon[97450]: pgmap v9850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:43:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:32.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:32.947995+0000) 2022-09-08T02:43:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:32.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:32.899099+0000) 2022-09-08T02:43:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:33.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:33.899276+0000) 2022-09-08T02:43:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:33.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:33.948213+0000) 2022-09-08T02:43:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:34 smithi055 ceph-mon[97033]: pgmap v9851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:34 smithi059 ceph-mon[97450]: pgmap v9851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:34.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:34.899489+0000) 2022-09-08T02:43:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:34.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:34.948444+0000) 2022-09-08T02:43:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:35.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:35.948694+0000) 2022-09-08T02:43:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:35.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:35.899742+0000) 2022-09-08T02:43:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:36 smithi055 ceph-mon[97033]: pgmap v9852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:43:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:36 smithi059 ceph-mon[97450]: pgmap v9852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:43:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:36.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:36.948923+0000) 2022-09-08T02:43:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:36.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:36.899980+0000) 2022-09-08T02:43:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:37.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:37.949121+0000) 2022-09-08T02:43:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:37.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:37.900284+0000) 2022-09-08T02:43:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:38 smithi055 ceph-mon[97033]: pgmap v9853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:38.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:38 smithi059 ceph-mon[97450]: pgmap v9853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:38.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:38.949317+0000) 2022-09-08T02:43:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:38.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:38.900581+0000) 2022-09-08T02:43:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:39.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:39.900832+0000) 2022-09-08T02:43:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:39.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:39.949461+0000) 2022-09-08T02:43:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:40 smithi055 ceph-mon[97033]: pgmap v9854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:40 smithi059 ceph-mon[97450]: pgmap v9854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:40.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:40.949709+0000) 2022-09-08T02:43:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:40.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:40.901093+0000) 2022-09-08T02:43:41.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:43:41.492+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:43:42.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:41.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:41.949895+0000) 2022-09-08T02:43:42.412 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:41.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:41.901338+0000) 2022-09-08T02:43:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:42 smithi055 ceph-mon[97033]: pgmap v9855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:43:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:42 smithi059 ceph-mon[97450]: pgmap v9855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:43:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:42.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:42.950034+0000) 2022-09-08T02:43:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:42.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:42.901538+0000) 2022-09-08T02:43:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:43.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:43.950252+0000) 2022-09-08T02:43:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:43.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:43.901776+0000) 2022-09-08T02:43:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:44 smithi055 ceph-mon[97033]: pgmap v9856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:44.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:44 smithi059 ceph-mon[97450]: pgmap v9856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:44.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:44.950488+0000) 2022-09-08T02:43:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:44.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:44.901956+0000) 2022-09-08T02:43:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:45.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:45.950738+0000) 2022-09-08T02:43:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:45.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:45.902189+0000) 2022-09-08T02:43:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:46 smithi055 ceph-mon[97033]: pgmap v9857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:43:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:46 smithi059 ceph-mon[97450]: pgmap v9857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:43:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:46.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:46.950940+0000) 2022-09-08T02:43:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:46.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:46.902447+0000) 2022-09-08T02:43:48.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:47.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:47.902727+0000) 2022-09-08T02:43:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:47.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:47.951107+0000) 2022-09-08T02:43:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:48 smithi055 ceph-mon[97033]: pgmap v9858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:48 smithi059 ceph-mon[97450]: pgmap v9858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:49.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:48.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:48.902959+0000) 2022-09-08T02:43:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:48.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:48.951318+0000) 2022-09-08T02:43:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:49.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:49.951559+0000) 2022-09-08T02:43:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:49.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:49.903225+0000) 2022-09-08T02:43:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:50 smithi055 ceph-mon[97033]: pgmap v9859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:50 smithi059 ceph-mon[97450]: pgmap v9859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:50.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:50.951695+0000) 2022-09-08T02:43:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:50.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:50.903425+0000) 2022-09-08T02:43:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:51.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:51.951903+0000) 2022-09-08T02:43:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:51.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:51.903562+0000) 2022-09-08T02:43:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:52 smithi055 ceph-mon[97033]: pgmap v9860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:43:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:52 smithi059 ceph-mon[97450]: pgmap v9860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:43:53.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:52.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:52.903781+0000) 2022-09-08T02:43:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:52.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:52.952133+0000) 2022-09-08T02:43:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:53.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:53.952331+0000) 2022-09-08T02:43:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:53.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:53.903983+0000) 2022-09-08T02:43:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:54 smithi055 ceph-mon[97033]: pgmap v9861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:54 smithi059 ceph-mon[97450]: pgmap v9861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:54.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:54.952563+0000) 2022-09-08T02:43:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:54.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:54.904201+0000) 2022-09-08T02:43:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:55.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:55.952816+0000) 2022-09-08T02:43:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:55.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:55.904383+0000) 2022-09-08T02:43:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:56 smithi055 ceph-mon[97033]: pgmap v9862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:43:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:56 smithi059 ceph-mon[97450]: pgmap v9862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:43:56.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:43:56.498+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:43:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:56.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:56.953050+0000) 2022-09-08T02:43:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:56.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:56.904565+0000) 2022-09-08T02:43:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:57.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:57.953292+0000) 2022-09-08T02:43:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:57.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:57.904698+0000) 2022-09-08T02:43:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:43:58 smithi055 ceph-mon[97033]: pgmap v9863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:58.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:43:58 smithi059 ceph-mon[97450]: pgmap v9863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:43:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:58.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:58.953537+0000) 2022-09-08T02:43:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:58.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:58.904950+0000) 2022-09-08T02:44:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:43:59.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:59.953765+0000) 2022-09-08T02:44:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:43:59.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:43:59.905215+0000) 2022-09-08T02:44:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:00 smithi055 ceph-mon[97033]: pgmap v9864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:00.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:00 smithi059 ceph-mon[97450]: pgmap v9864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:00.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:00.953887+0000) 2022-09-08T02:44:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:00.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:00.905370+0000) 2022-09-08T02:44:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:01.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:01.954101+0000) 2022-09-08T02:44:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:01.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:01.905551+0000) 2022-09-08T02:44:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:02 smithi055 ceph-mon[97033]: pgmap v9865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:44:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:02 smithi059 ceph-mon[97450]: pgmap v9865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:44:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:02.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:02.954324+0000) 2022-09-08T02:44:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:02.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:02.905774+0000) 2022-09-08T02:44:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:03.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:03.954559+0000) 2022-09-08T02:44:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:03.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:03.905990+0000) 2022-09-08T02:44:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:04 smithi055 ceph-mon[97033]: pgmap v9866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:04 smithi059 ceph-mon[97450]: pgmap v9866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:04.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:04.954751+0000) 2022-09-08T02:44:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:04.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:04.906244+0000) 2022-09-08T02:44:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:05.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:05.954973+0000) 2022-09-08T02:44:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:05.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:05.906476+0000) 2022-09-08T02:44:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:06 smithi055 ceph-mon[97033]: pgmap v9867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:44:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:44:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:44:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:06 smithi059 ceph-mon[97450]: pgmap v9867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:44:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:44:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:44:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:06.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:06.955163+0000) 2022-09-08T02:44:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:06.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:06.906732+0000) 2022-09-08T02:44:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:07.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:07.955335+0000) 2022-09-08T02:44:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:07.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:07.906965+0000) 2022-09-08T02:44:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:08 smithi055 ceph-mon[97033]: pgmap v9868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:44:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:08 smithi059 ceph-mon[97450]: pgmap v9868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:44:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:08.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:08.955582+0000) 2022-09-08T02:44:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:08.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:08.907181+0000) 2022-09-08T02:44:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:09.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:09.955814+0000) 2022-09-08T02:44:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:09.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:09.907435+0000) 2022-09-08T02:44:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:10 smithi055 ceph-mon[97033]: pgmap v9869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:44:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:10 smithi055 ceph-mon[97033]: osdmap e697: 12 total, 11 up, 11 in 2022-09-08T02:44:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:10 smithi059 ceph-mon[97450]: pgmap v9869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:44:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:10 smithi059 ceph-mon[97450]: osdmap e697: 12 total, 11 up, 11 in 2022-09-08T02:44:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:10.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:10.955989+0000) 2022-09-08T02:44:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:10.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:10.907663+0000) 2022-09-08T02:44:11.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:44:11.512+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:44:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:11.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:11.956217+0000) 2022-09-08T02:44:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:11.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:11.907828+0000) 2022-09-08T02:44:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:12 smithi055 ceph-mon[97033]: pgmap v9871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:44:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:12 smithi059 ceph-mon[97450]: pgmap v9871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:44:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:12.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:12.956470+0000) 2022-09-08T02:44:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:12.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:12.908005+0000) 2022-09-08T02:44:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:14 smithi059 ceph-mon[97450]: pgmap v9872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:44:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:14 smithi059 ceph-mon[97450]: osdmap e698: 12 total, 11 up, 11 in 2022-09-08T02:44:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:13.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:13.956719+0000) 2022-09-08T02:44:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:13.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:13.908238+0000) 2022-09-08T02:44:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:14 smithi055 ceph-mon[97033]: pgmap v9872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:44:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:14 smithi055 ceph-mon[97033]: osdmap e698: 12 total, 11 up, 11 in 2022-09-08T02:44:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:14.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:14.956940+0000) 2022-09-08T02:44:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:14.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:14.908515+0000) 2022-09-08T02:44:16.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:16 smithi059 ceph-mon[97450]: pgmap v9874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:44:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:15.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:15.957144+0000) 2022-09-08T02:44:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:15.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:15.908704+0000) 2022-09-08T02:44:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:16 smithi055 ceph-mon[97033]: pgmap v9874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:44:17.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:16.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:16.908932+0000) 2022-09-08T02:44:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:16.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:16.957298+0000) 2022-09-08T02:44:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:18 smithi059 ceph-mon[97450]: pgmap v9875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:44:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:17.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:17.957521+0000) 2022-09-08T02:44:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:17.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:17.909148+0000) 2022-09-08T02:44:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:18 smithi055 ceph-mon[97033]: pgmap v9875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:44:19.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:18.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:18.909314+0000) 2022-09-08T02:44:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:18.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:18.957751+0000) 2022-09-08T02:44:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:20 smithi059 ceph-mon[97450]: pgmap v9876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T02:44:20.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:19.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:19.909443+0000) 2022-09-08T02:44:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:19.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:19.957911+0000) 2022-09-08T02:44:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:20 smithi055 ceph-mon[97033]: pgmap v9876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T02:44:21.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:20.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:20.909735+0000) 2022-09-08T02:44:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:20.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:20.958054+0000) 2022-09-08T02:44:22.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:21.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:21.909941+0000) 2022-09-08T02:44:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:21.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:21.958306+0000) 2022-09-08T02:44:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:22 smithi055 ceph-mon[97033]: pgmap v9877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:44:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:22 smithi059 ceph-mon[97450]: pgmap v9877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:44:23.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:22.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:22.958547+0000) 2022-09-08T02:44:23.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:22.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:22.910071+0000) 2022-09-08T02:44:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:24 smithi059 ceph-mon[97450]: pgmap v9878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:44:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:44:24.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:44:24.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:44:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:23.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:23.958733+0000) 2022-09-08T02:44:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:23.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:23.910284+0000) 2022-09-08T02:44:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:24 smithi055 ceph-mon[97033]: pgmap v9878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:44:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:44:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:44:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:44:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:24.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:24.958853+0000) 2022-09-08T02:44:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:24.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:24.910524+0000) 2022-09-08T02:44:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:44:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:44:26.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:25.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:25.910748+0000) 2022-09-08T02:44:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:25.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:25.959062+0000) 2022-09-08T02:44:26.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:44:26.524+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:44:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:26 smithi055 ceph-mon[97033]: pgmap v9879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:44:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:26 smithi059 ceph-mon[97450]: pgmap v9879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:44:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:26.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:26.959234+0000) 2022-09-08T02:44:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:26.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:26.910887+0000) 2022-09-08T02:44:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:27.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:27.959458+0000) 2022-09-08T02:44:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:27.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:27.911033+0000) 2022-09-08T02:44:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:28 smithi055 ceph-mon[97033]: pgmap v9880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:28 smithi059 ceph-mon[97450]: pgmap v9880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:29.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:28.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:28.911246+0000) 2022-09-08T02:44:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:28.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:28.959617+0000) 2022-09-08T02:44:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:29.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:29.959790+0000) 2022-09-08T02:44:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:29.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:29.911441+0000) 2022-09-08T02:44:30.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:30 smithi055 ceph-mon[97033]: pgmap v9881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:30 smithi059 ceph-mon[97450]: pgmap v9881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:30.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:30.960021+0000) 2022-09-08T02:44:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:30.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:30.911716+0000) 2022-09-08T02:44:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:31.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:31.960264+0000) 2022-09-08T02:44:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:31.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:31.911906+0000) 2022-09-08T02:44:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:32 smithi055 ceph-mon[97033]: pgmap v9882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:44:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:32 smithi059 ceph-mon[97450]: pgmap v9882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:44:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:32.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:32.960526+0000) 2022-09-08T02:44:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:32.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:32.912066+0000) 2022-09-08T02:44:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:33.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:33.960718+0000) 2022-09-08T02:44:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:33.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:33.912270+0000) 2022-09-08T02:44:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:34 smithi055 ceph-mon[97033]: pgmap v9883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:34 smithi059 ceph-mon[97450]: pgmap v9883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:34.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:34.960935+0000) 2022-09-08T02:44:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:34.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:34.912508+0000) 2022-09-08T02:44:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:35.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:35.961087+0000) 2022-09-08T02:44:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:35.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:35.912722+0000) 2022-09-08T02:44:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:36 smithi055 ceph-mon[97033]: pgmap v9884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:44:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:36 smithi059 ceph-mon[97450]: pgmap v9884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:44:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:36.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:36.961340+0000) 2022-09-08T02:44:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:36.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:36.912924+0000) 2022-09-08T02:44:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:37.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:37.961505+0000) 2022-09-08T02:44:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:37.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:37.913085+0000) 2022-09-08T02:44:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:38 smithi055 ceph-mon[97033]: pgmap v9885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:38 smithi059 ceph-mon[97450]: pgmap v9885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:38.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:38.961629+0000) 2022-09-08T02:44:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:38.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:38.913328+0000) 2022-09-08T02:44:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:39.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:39.961840+0000) 2022-09-08T02:44:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:39.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:39.913570+0000) 2022-09-08T02:44:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:40 smithi055 ceph-mon[97033]: pgmap v9886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:40 smithi059 ceph-mon[97450]: pgmap v9886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:40.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:40.961985+0000) 2022-09-08T02:44:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:40.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:40.913711+0000) 2022-09-08T02:44:41.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:44:41.527+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:44:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:41.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:41.962224+0000) 2022-09-08T02:44:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:41.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:41.913919+0000) 2022-09-08T02:44:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:42 smithi055 ceph-mon[97033]: pgmap v9887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:44:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:42 smithi059 ceph-mon[97450]: pgmap v9887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:44:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:42.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:42.962461+0000) 2022-09-08T02:44:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:42.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:42.914138+0000) 2022-09-08T02:44:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:43.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:43.962739+0000) 2022-09-08T02:44:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:43.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:43.914399+0000) 2022-09-08T02:44:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:44 smithi055 ceph-mon[97033]: pgmap v9888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:44.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:44 smithi059 ceph-mon[97450]: pgmap v9888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:44.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:44.962874+0000) 2022-09-08T02:44:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:44.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:44.914583+0000) 2022-09-08T02:44:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:45.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:45.963092+0000) 2022-09-08T02:44:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:45.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:45.914806+0000) 2022-09-08T02:44:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:46 smithi055 ceph-mon[97033]: pgmap v9889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:44:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:46 smithi059 ceph-mon[97450]: pgmap v9889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:44:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:46.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:46.963303+0000) 2022-09-08T02:44:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:46.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:46.915048+0000) 2022-09-08T02:44:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:47.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:47.963544+0000) 2022-09-08T02:44:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:47.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:47.915331+0000) 2022-09-08T02:44:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:48 smithi055 ceph-mon[97033]: pgmap v9890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:48 smithi059 ceph-mon[97450]: pgmap v9890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:48.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:48.963726+0000) 2022-09-08T02:44:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:48.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:48.915462+0000) 2022-09-08T02:44:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:49.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:49.963848+0000) 2022-09-08T02:44:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:49.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:49.915738+0000) 2022-09-08T02:44:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:50 smithi055 ceph-mon[97033]: pgmap v9891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:50 smithi059 ceph-mon[97450]: pgmap v9891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:50.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:50.963968+0000) 2022-09-08T02:44:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:50.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:50.915969+0000) 2022-09-08T02:44:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:51.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:51.964093+0000) 2022-09-08T02:44:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:51.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:51.916227+0000) 2022-09-08T02:44:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:52 smithi055 ceph-mon[97033]: pgmap v9892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:44:52.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:52 smithi059 ceph-mon[97450]: pgmap v9892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:44:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:52.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:52.964211+0000) 2022-09-08T02:44:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:52.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:52.916431+0000) 2022-09-08T02:44:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:53.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:53.964358+0000) 2022-09-08T02:44:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:53.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:53.916672+0000) 2022-09-08T02:44:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:54 smithi055 ceph-mon[97033]: pgmap v9893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:54 smithi059 ceph-mon[97450]: pgmap v9893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:54.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:54.964567+0000) 2022-09-08T02:44:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:54.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:54.916919+0000) 2022-09-08T02:44:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:55.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:55.964806+0000) 2022-09-08T02:44:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:55.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:55.917122+0000) 2022-09-08T02:44:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:56 smithi055 ceph-mon[97033]: pgmap v9894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:44:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:56 smithi059 ceph-mon[97450]: pgmap v9894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:44:56.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:44:56.533+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:44:57.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:56.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:56.917313+0000) 2022-09-08T02:44:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:56.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:56.965038+0000) 2022-09-08T02:44:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:57.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:57.965200+0000) 2022-09-08T02:44:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:57.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:57.917545+0000) 2022-09-08T02:44:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:44:58 smithi055 ceph-mon[97033]: pgmap v9895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:44:58 smithi059 ceph-mon[97450]: pgmap v9895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:44:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:58.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:58.965429+0000) 2022-09-08T02:44:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:58.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:58.917791+0000) 2022-09-08T02:45:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:44:59.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:59.965677+0000) 2022-09-08T02:45:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:44:59.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:44:59.918006+0000) 2022-09-08T02:45:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:00 smithi055 ceph-mon[97033]: pgmap v9896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:00 smithi059 ceph-mon[97450]: pgmap v9896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:00.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:00.965808+0000) 2022-09-08T02:45:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:00.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:00.918160+0000) 2022-09-08T02:45:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:01.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:01.965961+0000) 2022-09-08T02:45:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:01.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:01.918359+0000) 2022-09-08T02:45:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:02 smithi055 ceph-mon[97033]: pgmap v9897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:45:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:02 smithi059 ceph-mon[97450]: pgmap v9897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:45:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:02.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:02.966186+0000) 2022-09-08T02:45:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:02.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:02.918496+0000) 2022-09-08T02:45:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:03.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:03.966434+0000) 2022-09-08T02:45:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:03.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:03.918723+0000) 2022-09-08T02:45:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:04 smithi055 ceph-mon[97033]: pgmap v9898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:04 smithi059 ceph-mon[97450]: pgmap v9898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:04.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:04.966711+0000) 2022-09-08T02:45:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:04.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:04.918993+0000) 2022-09-08T02:45:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:05.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:05.966847+0000) 2022-09-08T02:45:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:05.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:05.919196+0000) 2022-09-08T02:45:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:06 smithi055 ceph-mon[97033]: pgmap v9899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:45:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:45:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:45:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:06 smithi059 ceph-mon[97450]: pgmap v9899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:45:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:45:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:45:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:06.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:06.966998+0000) 2022-09-08T02:45:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:06.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:06.919414+0000) 2022-09-08T02:45:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:07.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:07.967193+0000) 2022-09-08T02:45:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:07.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:07.919601+0000) 2022-09-08T02:45:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:08 smithi055 ceph-mon[97033]: pgmap v9900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:45:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:08 smithi059 ceph-mon[97450]: pgmap v9900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:45:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:08.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:08.967396+0000) 2022-09-08T02:45:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:08.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:08.919835+0000) 2022-09-08T02:45:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:09.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:09.967678+0000) 2022-09-08T02:45:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:09.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:09.920066+0000) 2022-09-08T02:45:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:10 smithi055 ceph-mon[97033]: pgmap v9901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:45:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:10 smithi055 ceph-mon[97033]: osdmap e699: 12 total, 11 up, 11 in 2022-09-08T02:45:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:10 smithi059 ceph-mon[97450]: pgmap v9901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:45:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:10 smithi059 ceph-mon[97450]: osdmap e699: 12 total, 11 up, 11 in 2022-09-08T02:45:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:10.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:10.967858+0000) 2022-09-08T02:45:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:10.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:10.920285+0000) 2022-09-08T02:45:11.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:45:11.546+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:45:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:11.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:11.967987+0000) 2022-09-08T02:45:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:11.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:11.920534+0000) 2022-09-08T02:45:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:12 smithi055 ceph-mon[97033]: pgmap v9903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:45:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:12 smithi059 ceph-mon[97450]: pgmap v9903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:45:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:12.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:12.968150+0000) 2022-09-08T02:45:13.417 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:12.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:12.920758+0000) 2022-09-08T02:45:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:13.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:13.968357+0000) 2022-09-08T02:45:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:13.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:13.921037+0000) 2022-09-08T02:45:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:14 smithi055 ceph-mon[97033]: pgmap v9904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:45:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:14 smithi059 ceph-mon[97450]: pgmap v9904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:45:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:14.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:14.968514+0000) 2022-09-08T02:45:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:14.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:14.921252+0000) 2022-09-08T02:45:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:15.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:15.968741+0000) 2022-09-08T02:45:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:15.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:15.921462+0000) 2022-09-08T02:45:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:16 smithi055 ceph-mon[97033]: pgmap v9905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:45:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:16 smithi059 ceph-mon[97450]: pgmap v9905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:45:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:16.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:16.968882+0000) 2022-09-08T02:45:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:16.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:16.921708+0000) 2022-09-08T02:45:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:17.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:17.969045+0000) 2022-09-08T02:45:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:17.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:17.921895+0000) 2022-09-08T02:45:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:18 smithi055 ceph-mon[97033]: pgmap v9906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:45:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:18 smithi055 ceph-mon[97033]: osdmap e700: 12 total, 11 up, 11 in 2022-09-08T02:45:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:18 smithi059 ceph-mon[97450]: pgmap v9906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:45:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:18 smithi059 ceph-mon[97450]: osdmap e700: 12 total, 11 up, 11 in 2022-09-08T02:45:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:18.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:18.969231+0000) 2022-09-08T02:45:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:18.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:18.922158+0000) 2022-09-08T02:45:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:19.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:19.969469+0000) 2022-09-08T02:45:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:19.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:19.922327+0000) 2022-09-08T02:45:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:20 smithi055 ceph-mon[97033]: pgmap v9908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:45:20.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:20 smithi059 ceph-mon[97450]: pgmap v9908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:45:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:20.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:20.969666+0000) 2022-09-08T02:45:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:20.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:20.922555+0000) 2022-09-08T02:45:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:21.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:21.969898+0000) 2022-09-08T02:45:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:21.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:21.922790+0000) 2022-09-08T02:45:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:22 smithi055 ceph-mon[97033]: pgmap v9909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:45:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:22 smithi059 ceph-mon[97450]: pgmap v9909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:45:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:22.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:22.970110+0000) 2022-09-08T02:45:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:22.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:22.923011+0000) 2022-09-08T02:45:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:23.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:23.970301+0000) 2022-09-08T02:45:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:23.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:23.923261+0000) 2022-09-08T02:45:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:24 smithi055 ceph-mon[97033]: pgmap v9910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:45:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:24 smithi059 ceph-mon[97450]: pgmap v9910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:45:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:24.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:24.970524+0000) 2022-09-08T02:45:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:24.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:24.923502+0000) 2022-09-08T02:45:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:45:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:45:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:45:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:45:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:45:25.542 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:45:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:25.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:25.970751+0000) 2022-09-08T02:45:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:25.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:25.923678+0000) 2022-09-08T02:45:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:26 smithi055 ceph-mon[97033]: pgmap v9911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:45:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:45:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:26 smithi059 ceph-mon[97450]: pgmap v9911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:45:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:45:26.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:45:26.555+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:45:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:26.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:26.970992+0000) 2022-09-08T02:45:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:26.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:26.923849+0000) 2022-09-08T02:45:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:27.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:27.971156+0000) 2022-09-08T02:45:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:27.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:27.924067+0000) 2022-09-08T02:45:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:28 smithi055 ceph-mon[97033]: pgmap v9912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:45:28.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:28 smithi059 ceph-mon[97450]: pgmap v9912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:45:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:28.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:28.971397+0000) 2022-09-08T02:45:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:28.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:28.924196+0000) 2022-09-08T02:45:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:29.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:29.971585+0000) 2022-09-08T02:45:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:29.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:29.924411+0000) 2022-09-08T02:45:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:30 smithi055 ceph-mon[97033]: pgmap v9913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:45:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:30 smithi059 ceph-mon[97450]: pgmap v9913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:45:31.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:30.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:30.924616+0000) 2022-09-08T02:45:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:30.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:30.971766+0000) 2022-09-08T02:45:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:31.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:31.971989+0000) 2022-09-08T02:45:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:31.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:31.924748+0000) 2022-09-08T02:45:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:32 smithi055 ceph-mon[97033]: pgmap v9914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:45:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:32 smithi059 ceph-mon[97450]: pgmap v9914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:45:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:32.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:32.972154+0000) 2022-09-08T02:45:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:32.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:32.924974+0000) 2022-09-08T02:45:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:33.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:33.972342+0000) 2022-09-08T02:45:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:33.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:33.925245+0000) 2022-09-08T02:45:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:34 smithi055 ceph-mon[97033]: pgmap v9915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:34 smithi059 ceph-mon[97450]: pgmap v9915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:34.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:34.972516+0000) 2022-09-08T02:45:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:34.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:34.925489+0000) 2022-09-08T02:45:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:35.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:35.972738+0000) 2022-09-08T02:45:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:35.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:35.925746+0000) 2022-09-08T02:45:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:36 smithi055 ceph-mon[97033]: pgmap v9916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:45:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:36 smithi059 ceph-mon[97450]: pgmap v9916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:45:37.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:36.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:36.925905+0000) 2022-09-08T02:45:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:36.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:36.972934+0000) 2022-09-08T02:45:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:37.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:37.973166+0000) 2022-09-08T02:45:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:37.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:37.926161+0000) 2022-09-08T02:45:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:38 smithi055 ceph-mon[97033]: pgmap v9917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:38 smithi059 ceph-mon[97450]: pgmap v9917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:38.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:38.973407+0000) 2022-09-08T02:45:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:38.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:38.926347+0000) 2022-09-08T02:45:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:39.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:39.973625+0000) 2022-09-08T02:45:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:39.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:39.926480+0000) 2022-09-08T02:45:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:40 smithi055 ceph-mon[97033]: pgmap v9918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:40 smithi059 ceph-mon[97450]: pgmap v9918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:40.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:40.973851+0000) 2022-09-08T02:45:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:40.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:40.926747+0000) 2022-09-08T02:45:41.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:45:41.561+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:45:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:41.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:41.974135+0000) 2022-09-08T02:45:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:41.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:41.926967+0000) 2022-09-08T02:45:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:42 smithi055 ceph-mon[97033]: pgmap v9919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:45:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:42 smithi059 ceph-mon[97450]: pgmap v9919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:45:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:42.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:42.974352+0000) 2022-09-08T02:45:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:42.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:42.927201+0000) 2022-09-08T02:45:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:43.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:43.927442+0000) 2022-09-08T02:45:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:43.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:43.974581+0000) 2022-09-08T02:45:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:44 smithi055 ceph-mon[97033]: pgmap v9920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:44 smithi059 ceph-mon[97450]: pgmap v9920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:44.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:44.974805+0000) 2022-09-08T02:45:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:44.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:44.927613+0000) 2022-09-08T02:45:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:45.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:45.975038+0000) 2022-09-08T02:45:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:45.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:45.927788+0000) 2022-09-08T02:45:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:46 smithi055 ceph-mon[97033]: pgmap v9921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:45:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:46 smithi059 ceph-mon[97450]: pgmap v9921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:45:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:46.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:46.975314+0000) 2022-09-08T02:45:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:46.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:46.928036+0000) 2022-09-08T02:45:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:47.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:47.975602+0000) 2022-09-08T02:45:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:47.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:47.928260+0000) 2022-09-08T02:45:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:48 smithi055 ceph-mon[97033]: pgmap v9922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:48 smithi059 ceph-mon[97450]: pgmap v9922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:48.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:48.975836+0000) 2022-09-08T02:45:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:48.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:48.928465+0000) 2022-09-08T02:45:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:49.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:49.976011+0000) 2022-09-08T02:45:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:49.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:49.928693+0000) 2022-09-08T02:45:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:50 smithi055 ceph-mon[97033]: pgmap v9923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:50.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:50 smithi059 ceph-mon[97450]: pgmap v9923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:50.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:50.976230+0000) 2022-09-08T02:45:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:50.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:50.928895+0000) 2022-09-08T02:45:52.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:51.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:51.929152+0000) 2022-09-08T02:45:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:51.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:51.976480+0000) 2022-09-08T02:45:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:52 smithi055 ceph-mon[97033]: pgmap v9924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:45:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:52 smithi059 ceph-mon[97450]: pgmap v9924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:45:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:52.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:52.976740+0000) 2022-09-08T02:45:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:52.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:52.929457+0000) 2022-09-08T02:45:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:53.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:53.976926+0000) 2022-09-08T02:45:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:53.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:53.929749+0000) 2022-09-08T02:45:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:54 smithi055 ceph-mon[97033]: pgmap v9925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:54.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:54 smithi059 ceph-mon[97450]: pgmap v9925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:54.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:54.977093+0000) 2022-09-08T02:45:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:54.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:54.929954+0000) 2022-09-08T02:45:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:55.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:55.977319+0000) 2022-09-08T02:45:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:55.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:55.930202+0000) 2022-09-08T02:45:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:56 smithi055 ceph-mon[97033]: pgmap v9926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:45:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:56 smithi059 ceph-mon[97450]: pgmap v9926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:45:56.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:45:56.577+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:45:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:56.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:56.977536+0000) 2022-09-08T02:45:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:56.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:56.930447+0000) 2022-09-08T02:45:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:57.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:57.977713+0000) 2022-09-08T02:45:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:57.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:57.930714+0000) 2022-09-08T02:45:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:45:58 smithi055 ceph-mon[97033]: pgmap v9927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:45:58 smithi059 ceph-mon[97450]: pgmap v9927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:45:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:58.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:58.977897+0000) 2022-09-08T02:45:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:58.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:58.930856+0000) 2022-09-08T02:46:00.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:45:59.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:59.931053+0000) 2022-09-08T02:46:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:45:59.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:45:59.978113+0000) 2022-09-08T02:46:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:00 smithi055 ceph-mon[97033]: pgmap v9928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:00 smithi059 ceph-mon[97450]: pgmap v9928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:00.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:00.978352+0000) 2022-09-08T02:46:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:00.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:00.931268+0000) 2022-09-08T02:46:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:01.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:01.978602+0000) 2022-09-08T02:46:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:01.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:01.931501+0000) 2022-09-08T02:46:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:02 smithi055 ceph-mon[97033]: pgmap v9929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:46:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:02 smithi059 ceph-mon[97450]: pgmap v9929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:46:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:02.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:02.978847+0000) 2022-09-08T02:46:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:02.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:02.931702+0000) 2022-09-08T02:46:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:03.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:03.979031+0000) 2022-09-08T02:46:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:03.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:03.931857+0000) 2022-09-08T02:46:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:04 smithi055 ceph-mon[97033]: pgmap v9930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:04 smithi059 ceph-mon[97450]: pgmap v9930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:04.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:04.979207+0000) 2022-09-08T02:46:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:04.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:04.932027+0000) 2022-09-08T02:46:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:05.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:05.979423+0000) 2022-09-08T02:46:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:05.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:05.932261+0000) 2022-09-08T02:46:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:06 smithi055 ceph-mon[97033]: pgmap v9931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:46:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:46:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:46:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:06 smithi059 ceph-mon[97450]: pgmap v9931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:46:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:46:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:46:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:06.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:06.979694+0000) 2022-09-08T02:46:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:06.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:06.932473+0000) 2022-09-08T02:46:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:07.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:07.979920+0000) 2022-09-08T02:46:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:07.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:07.932682+0000) 2022-09-08T02:46:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:08 smithi055 ceph-mon[97033]: pgmap v9932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:46:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:08 smithi059 ceph-mon[97450]: pgmap v9932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:46:09.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:08.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:08.932927+0000) 2022-09-08T02:46:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:08.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:08.980146+0000) 2022-09-08T02:46:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:09.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:09.980303+0000) 2022-09-08T02:46:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:09.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:09.933164+0000) 2022-09-08T02:46:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:10 smithi055 ceph-mon[97033]: pgmap v9933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:46:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:10 smithi055 ceph-mon[97033]: osdmap e701: 12 total, 11 up, 11 in 2022-09-08T02:46:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:10 smithi059 ceph-mon[97450]: pgmap v9933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:46:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:10 smithi059 ceph-mon[97450]: osdmap e701: 12 total, 11 up, 11 in 2022-09-08T02:46:11.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:10.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:10.933386+0000) 2022-09-08T02:46:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:10.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:10.980544+0000) 2022-09-08T02:46:11.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:46:11.582+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:46:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:11.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:11.980752+0000) 2022-09-08T02:46:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:11.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:11.933583+0000) 2022-09-08T02:46:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:12 smithi055 ceph-mon[97033]: pgmap v9935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:46:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:12 smithi059 ceph-mon[97450]: pgmap v9935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:46:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:12.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:12.980867+0000) 2022-09-08T02:46:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:12.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:12.933723+0000) 2022-09-08T02:46:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:13.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:13.981082+0000) 2022-09-08T02:46:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:13.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:13.933908+0000) 2022-09-08T02:46:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:14 smithi055 ceph-mon[97033]: osdmap e702: 12 total, 11 up, 11 in 2022-09-08T02:46:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:14 smithi055 ceph-mon[97033]: pgmap v9937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:46:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:14 smithi059 ceph-mon[97450]: osdmap e702: 12 total, 11 up, 11 in 2022-09-08T02:46:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:14 smithi059 ceph-mon[97450]: pgmap v9937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:46:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:14.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:14.981308+0000) 2022-09-08T02:46:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:14.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:14.934158+0000) 2022-09-08T02:46:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:15.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:15.981526+0000) 2022-09-08T02:46:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:15.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:15.934407+0000) 2022-09-08T02:46:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:16 smithi055 ceph-mon[97033]: pgmap v9938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:46:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:16 smithi059 ceph-mon[97450]: pgmap v9938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:46:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:16.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:16.981749+0000) 2022-09-08T02:46:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:16.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:16.934677+0000) 2022-09-08T02:46:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:17.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:17.981919+0000) 2022-09-08T02:46:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:17.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:17.934882+0000) 2022-09-08T02:46:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:18 smithi055 ceph-mon[97033]: pgmap v9939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:46:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:18 smithi059 ceph-mon[97450]: pgmap v9939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:46:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:18.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:18.982126+0000) 2022-09-08T02:46:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:18.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:18.935051+0000) 2022-09-08T02:46:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:19.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:19.982322+0000) 2022-09-08T02:46:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:19.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:19.935226+0000) 2022-09-08T02:46:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:20 smithi055 ceph-mon[97033]: pgmap v9940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:46:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:20 smithi059 ceph-mon[97450]: pgmap v9940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:46:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:20.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:20.982537+0000) 2022-09-08T02:46:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:20.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:20.935440+0000) 2022-09-08T02:46:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:21.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:21.982752+0000) 2022-09-08T02:46:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:21.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:21.935648+0000) 2022-09-08T02:46:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:22 smithi055 ceph-mon[97033]: pgmap v9941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:46:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:22 smithi059 ceph-mon[97450]: pgmap v9941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:46:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:22.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:22.982947+0000) 2022-09-08T02:46:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:22.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:22.935781+0000) 2022-09-08T02:46:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:23.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:23.983157+0000) 2022-09-08T02:46:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:23.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:23.936003+0000) 2022-09-08T02:46:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:24 smithi055 ceph-mon[97033]: pgmap v9942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:46:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:24 smithi059 ceph-mon[97450]: pgmap v9942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:46:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:24.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:24.983408+0000) 2022-09-08T02:46:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:24.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:24.936180+0000) 2022-09-08T02:46:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:26 smithi059 ceph-mon[97450]: pgmap v9943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:46:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:46:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:46:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:46:26.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:46:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:25.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:25.983724+0000) 2022-09-08T02:46:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:25.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:25.936393+0000) 2022-09-08T02:46:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:26 smithi055 ceph-mon[97033]: pgmap v9943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:46:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:46:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:46:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:46:26.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:46:26.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:46:26.596+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:46:27.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:26.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:26.936622+0000) 2022-09-08T02:46:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:26.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:26.983875+0000) 2022-09-08T02:46:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:27.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:27.984089+0000) 2022-09-08T02:46:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:27.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:27.936821+0000) 2022-09-08T02:46:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:28 smithi055 ceph-mon[97033]: pgmap v9944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:28 smithi059 ceph-mon[97450]: pgmap v9944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:28.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:28.984330+0000) 2022-09-08T02:46:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:28.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:28.936968+0000) 2022-09-08T02:46:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:29.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:29.984570+0000) 2022-09-08T02:46:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:29.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:29.937117+0000) 2022-09-08T02:46:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:30 smithi055 ceph-mon[97033]: pgmap v9945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:30 smithi059 ceph-mon[97450]: pgmap v9945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:30.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:30.984774+0000) 2022-09-08T02:46:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:30.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:30.937337+0000) 2022-09-08T02:46:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:31.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:31.985000+0000) 2022-09-08T02:46:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:31.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:31.937598+0000) 2022-09-08T02:46:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:32 smithi055 ceph-mon[97033]: pgmap v9946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:46:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:32 smithi059 ceph-mon[97450]: pgmap v9946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:46:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:32.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:32.985253+0000) 2022-09-08T02:46:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:32.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:32.937839+0000) 2022-09-08T02:46:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:33.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:33.985536+0000) 2022-09-08T02:46:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:33.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:33.938028+0000) 2022-09-08T02:46:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:34 smithi055 ceph-mon[97033]: pgmap v9947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:34 smithi059 ceph-mon[97450]: pgmap v9947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:34.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:34.985798+0000) 2022-09-08T02:46:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:34.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:34.938239+0000) 2022-09-08T02:46:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:35.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:35.986041+0000) 2022-09-08T02:46:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:35.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:35.938477+0000) 2022-09-08T02:46:36.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:36 smithi055 ceph-mon[97033]: pgmap v9948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:46:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:36 smithi059 ceph-mon[97450]: pgmap v9948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:46:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:36.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:36.986226+0000) 2022-09-08T02:46:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:36.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:36.938673+0000) 2022-09-08T02:46:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:37.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:37.986427+0000) 2022-09-08T02:46:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:37.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:37.938913+0000) 2022-09-08T02:46:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:38 smithi055 ceph-mon[97033]: pgmap v9949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:38 smithi059 ceph-mon[97450]: pgmap v9949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:38.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:38.986670+0000) 2022-09-08T02:46:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:38.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:38.939123+0000) 2022-09-08T02:46:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:39.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:39.986908+0000) 2022-09-08T02:46:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:39.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:39.939437+0000) 2022-09-08T02:46:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:40 smithi055 ceph-mon[97033]: pgmap v9950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:40.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:40 smithi059 ceph-mon[97450]: pgmap v9950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:40.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:40.987114+0000) 2022-09-08T02:46:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:40.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:40.939584+0000) 2022-09-08T02:46:41.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:46:41.613+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:46:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:41.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:41.987317+0000) 2022-09-08T02:46:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:41.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:41.939817+0000) 2022-09-08T02:46:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:42 smithi055 ceph-mon[97033]: pgmap v9951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:46:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:42 smithi059 ceph-mon[97450]: pgmap v9951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:46:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:42.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:42.987541+0000) 2022-09-08T02:46:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:42.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:42.940052+0000) 2022-09-08T02:46:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:43.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:43.987811+0000) 2022-09-08T02:46:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:43.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:43.940303+0000) 2022-09-08T02:46:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:44 smithi055 ceph-mon[97033]: pgmap v9952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:44 smithi059 ceph-mon[97450]: pgmap v9952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:44.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:44.987950+0000) 2022-09-08T02:46:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:44.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:44.940578+0000) 2022-09-08T02:46:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:45.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:45.988085+0000) 2022-09-08T02:46:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:45.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:45.940785+0000) 2022-09-08T02:46:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:46 smithi055 ceph-mon[97033]: pgmap v9953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:46:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:46 smithi059 ceph-mon[97450]: pgmap v9953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:46:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:46.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:46.988316+0000) 2022-09-08T02:46:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:46.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:46.941017+0000) 2022-09-08T02:46:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:47.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:47.988601+0000) 2022-09-08T02:46:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:47.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:47.941305+0000) 2022-09-08T02:46:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:48 smithi055 ceph-mon[97033]: pgmap v9954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:48 smithi059 ceph-mon[97450]: pgmap v9954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:48.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:48.988831+0000) 2022-09-08T02:46:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:48.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:48.941436+0000) 2022-09-08T02:46:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:49.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:49.989041+0000) 2022-09-08T02:46:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:49.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:49.941708+0000) 2022-09-08T02:46:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:50 smithi055 ceph-mon[97033]: pgmap v9955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:50 smithi059 ceph-mon[97450]: pgmap v9955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:50.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:50.989249+0000) 2022-09-08T02:46:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:50.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:50.941885+0000) 2022-09-08T02:46:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:51.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:51.989503+0000) 2022-09-08T02:46:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:51.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:51.942088+0000) 2022-09-08T02:46:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:52 smithi055 ceph-mon[97033]: pgmap v9956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:46:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:52 smithi059 ceph-mon[97450]: pgmap v9956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:46:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:52.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:52.989717+0000) 2022-09-08T02:46:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:52.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:52.942264+0000) 2022-09-08T02:46:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:53.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:53.989915+0000) 2022-09-08T02:46:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:53.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:53.942490+0000) 2022-09-08T02:46:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:54 smithi055 ceph-mon[97033]: pgmap v9957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:54 smithi059 ceph-mon[97450]: pgmap v9957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:54.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:54.990067+0000) 2022-09-08T02:46:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:54.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:54.942651+0000) 2022-09-08T02:46:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:55.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:55.990258+0000) 2022-09-08T02:46:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:55.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:55.942838+0000) 2022-09-08T02:46:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:56 smithi055 ceph-mon[97033]: pgmap v9958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:46:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:56 smithi059 ceph-mon[97450]: pgmap v9958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:46:56.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:46:56.621+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:46:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:56.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:56.990406+0000) 2022-09-08T02:46:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:56.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:56.943006+0000) 2022-09-08T02:46:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:57.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:57.990585+0000) 2022-09-08T02:46:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:57.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:57.943204+0000) 2022-09-08T02:46:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:46:58 smithi055 ceph-mon[97033]: pgmap v9959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:46:58 smithi059 ceph-mon[97450]: pgmap v9959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:46:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:58.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:58.990790+0000) 2022-09-08T02:46:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:58.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:58.943424+0000) 2022-09-08T02:47:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:46:59.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:59.991034+0000) 2022-09-08T02:47:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:46:59.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:46:59.943687+0000) 2022-09-08T02:47:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:00 smithi055 ceph-mon[97033]: pgmap v9960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:00 smithi059 ceph-mon[97450]: pgmap v9960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:00.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:00.991197+0000) 2022-09-08T02:47:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:00.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:00.943855+0000) 2022-09-08T02:47:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:01.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:01.991418+0000) 2022-09-08T02:47:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:01.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:01.944061+0000) 2022-09-08T02:47:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:02 smithi055 ceph-mon[97033]: pgmap v9961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:47:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:02 smithi059 ceph-mon[97450]: pgmap v9961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:47:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:02.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:02.991660+0000) 2022-09-08T02:47:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:02.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:02.944286+0000) 2022-09-08T02:47:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:03.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:03.991871+0000) 2022-09-08T02:47:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:03.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:03.944528+0000) 2022-09-08T02:47:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:04 smithi055 ceph-mon[97033]: pgmap v9962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:47:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:04 smithi059 ceph-mon[97450]: pgmap v9962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:47:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:04.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:04.992095+0000) 2022-09-08T02:47:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:04.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:04.944782+0000) 2022-09-08T02:47:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:05.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:05.992271+0000) 2022-09-08T02:47:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:05.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:05.944989+0000) 2022-09-08T02:47:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:06 smithi055 ceph-mon[97033]: pgmap v9963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:47:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:47:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:47:06.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:06 smithi059 ceph-mon[97450]: pgmap v9963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:47:06.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:47:06.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:47:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:06.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:06.992506+0000) 2022-09-08T02:47:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:06.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:06.945213+0000) 2022-09-08T02:47:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:07.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:07.992697+0000) 2022-09-08T02:47:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:07.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:07.945460+0000) 2022-09-08T02:47:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:08 smithi055 ceph-mon[97033]: pgmap v9964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:47:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:08 smithi059 ceph-mon[97450]: pgmap v9964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:47:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:08.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:08.992916+0000) 2022-09-08T02:47:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:08.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:08.945715+0000) 2022-09-08T02:47:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:09.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:09.993138+0000) 2022-09-08T02:47:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:09.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:09.945870+0000) 2022-09-08T02:47:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:10 smithi055 ceph-mon[97033]: pgmap v9965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:47:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:10 smithi055 ceph-mon[97033]: osdmap e703: 12 total, 11 up, 11 in 2022-09-08T02:47:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:10 smithi059 ceph-mon[97450]: pgmap v9965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:47:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:10 smithi059 ceph-mon[97450]: osdmap e703: 12 total, 11 up, 11 in 2022-09-08T02:47:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:10.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:10.993367+0000) 2022-09-08T02:47:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:10.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:10.946092+0000) 2022-09-08T02:47:11.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:47:11.624+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:47:12.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:11.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:11.946279+0000) 2022-09-08T02:47:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:11.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:11.993599+0000) 2022-09-08T02:47:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:12 smithi055 ceph-mon[97033]: pgmap v9967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:47:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:12 smithi059 ceph-mon[97450]: pgmap v9967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:47:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:12.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:12.993802+0000) 2022-09-08T02:47:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:12.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:12.946526+0000) 2022-09-08T02:47:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:13.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:13.994010+0000) 2022-09-08T02:47:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:13.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:13.946758+0000) 2022-09-08T02:47:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:14 smithi055 ceph-mon[97033]: pgmap v9968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:47:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:14 smithi059 ceph-mon[97450]: pgmap v9968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:47:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:14.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:14.994212+0000) 2022-09-08T02:47:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:14.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:14.946982+0000) 2022-09-08T02:47:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:15.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:15.994431+0000) 2022-09-08T02:47:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:15.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:15.947183+0000) 2022-09-08T02:47:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:16 smithi055 ceph-mon[97033]: pgmap v9969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:47:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:16 smithi059 ceph-mon[97450]: pgmap v9969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:47:17.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:16.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:16.947365+0000) 2022-09-08T02:47:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:16.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:16.994678+0000) 2022-09-08T02:47:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:17.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:17.994923+0000) 2022-09-08T02:47:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:17.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:17.947597+0000) 2022-09-08T02:47:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:18 smithi055 ceph-mon[97033]: pgmap v9970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:47:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:18 smithi055 ceph-mon[97033]: osdmap e704: 12 total, 11 up, 11 in 2022-09-08T02:47:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:18 smithi059 ceph-mon[97450]: pgmap v9970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:47:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:18 smithi059 ceph-mon[97450]: osdmap e704: 12 total, 11 up, 11 in 2022-09-08T02:47:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:18.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:18.947816+0000) 2022-09-08T02:47:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:18.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:18.995082+0000) 2022-09-08T02:47:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:19.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:19.995247+0000) 2022-09-08T02:47:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:19.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:19.948013+0000) 2022-09-08T02:47:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:20 smithi055 ceph-mon[97033]: pgmap v9972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:47:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:20 smithi059 ceph-mon[97450]: pgmap v9972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:47:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:20.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:20.995504+0000) 2022-09-08T02:47:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:20.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:20.948243+0000) 2022-09-08T02:47:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:21.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:21.995784+0000) 2022-09-08T02:47:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:21.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:21.948502+0000) 2022-09-08T02:47:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:22 smithi055 ceph-mon[97033]: pgmap v9973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:47:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:22 smithi059 ceph-mon[97450]: pgmap v9973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:47:23.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:22.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:22.948714+0000) 2022-09-08T02:47:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:22.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:22.995984+0000) 2022-09-08T02:47:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:23.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:23.996149+0000) 2022-09-08T02:47:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:23.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:23.948929+0000) 2022-09-08T02:47:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:24 smithi055 ceph-mon[97033]: pgmap v9974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:47:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:24 smithi059 ceph-mon[97450]: pgmap v9974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:47:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:24.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:24.996369+0000) 2022-09-08T02:47:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:24.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:24.949090+0000) 2022-09-08T02:47:26.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:26 smithi055 ceph-mon[97033]: pgmap v9975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:47:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:47:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:47:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:47:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:26 smithi059 ceph-mon[97450]: pgmap v9975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:47:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:47:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:47:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:47:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:25.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:25.996598+0000) 2022-09-08T02:47:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:25.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:25.949221+0000) 2022-09-08T02:47:26.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:47:26.627+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:47:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:26.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:26.996840+0000) 2022-09-08T02:47:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:26.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:26.949434+0000) 2022-09-08T02:47:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:28.113 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:28.114 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:28.115 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:28.115 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:27.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:27.997006+0000) 2022-09-08T02:47:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:27.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:27.949719+0000) 2022-09-08T02:47:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:28 smithi055 ceph-mon[97033]: pgmap v9976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:47:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:28 smithi059 ceph-mon[97450]: pgmap v9976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:47:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:28.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:28.997171+0000) 2022-09-08T02:47:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:28.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:28.949951+0000) 2022-09-08T02:47:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:29.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:29.997346+0000) 2022-09-08T02:47:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:29.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:29.950195+0000) 2022-09-08T02:47:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:30 smithi055 ceph-mon[97033]: pgmap v9977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:47:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:30 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T02:47:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:30 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T02:47:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:31.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:31.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:30 smithi059 ceph-mon[97450]: pgmap v9977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:47:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:30 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T02:47:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:30 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T02:47:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:31.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:47:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:30.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:30.997557+0000) 2022-09-08T02:47:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:30.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:30.950374+0000) 2022-09-08T02:47:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:31.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:31.997833+0000) 2022-09-08T02:47:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:31.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:31.950606+0000) 2022-09-08T02:47:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:32 smithi055 ceph-mon[97033]: pgmap v9978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:47:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:32 smithi059 ceph-mon[97450]: pgmap v9978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:47:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:32.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:32.998090+0000) 2022-09-08T02:47:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:32.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:32.950860+0000) 2022-09-08T02:47:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:33.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:33.998357+0000) 2022-09-08T02:47:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:33.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:33.951078+0000) 2022-09-08T02:47:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:34 smithi055 ceph-mon[97033]: pgmap v9979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:34 smithi059 ceph-mon[97450]: pgmap v9979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:34.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:34.998620+0000) 2022-09-08T02:47:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:34.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:34.951341+0000) 2022-09-08T02:47:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:35.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:35.998830+0000) 2022-09-08T02:47:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:35.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:35.951579+0000) 2022-09-08T02:47:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:36 smithi055 ceph-mon[97033]: pgmap v9980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:47:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:36 smithi059 ceph-mon[97450]: pgmap v9980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:47:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:36.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:36.999093+0000) 2022-09-08T02:47:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:36.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:36.951797+0000) 2022-09-08T02:47:38.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:37.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:37.952003+0000) 2022-09-08T02:47:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:37.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:37.999268+0000) 2022-09-08T02:47:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:38 smithi055 ceph-mon[97033]: pgmap v9981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:38 smithi059 ceph-mon[97450]: pgmap v9981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:38.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:38.999447+0000) 2022-09-08T02:47:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:38.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:38.952226+0000) 2022-09-08T02:47:40.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:39.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:39.952453+0000) 2022-09-08T02:47:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:39.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:39.999686+0000) 2022-09-08T02:47:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:40 smithi055 ceph-mon[97033]: pgmap v9982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:40 smithi059 ceph-mon[97450]: pgmap v9982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:41.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:40.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:40.952694+0000) 2022-09-08T02:47:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:40.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:40.999944+0000) 2022-09-08T02:47:41.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:47:41.642+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:47:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:42.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:42.000172+0000) 2022-09-08T02:47:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:41.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:41.952925+0000) 2022-09-08T02:47:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:42 smithi055 ceph-mon[97033]: pgmap v9983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:47:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:42 smithi059 ceph-mon[97450]: pgmap v9983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:47:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:43.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:43.000418+0000) 2022-09-08T02:47:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:42.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:42.953150+0000) 2022-09-08T02:47:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:44.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:44.000628+0000) 2022-09-08T02:47:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:43.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:43.953350+0000) 2022-09-08T02:47:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:44 smithi055 ceph-mon[97033]: pgmap v9984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:44 smithi059 ceph-mon[97450]: pgmap v9984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:45.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:44.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:44.953621+0000) 2022-09-08T02:47:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:45.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:45.000890+0000) 2022-09-08T02:47:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:45.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:46.001032+0000) 2022-09-08T02:47:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:45.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:45.953846+0000) 2022-09-08T02:47:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:46 smithi055 ceph-mon[97033]: pgmap v9985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:47:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:46 smithi059 ceph-mon[97450]: pgmap v9985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:47:47.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:46.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:46.954021+0000) 2022-09-08T02:47:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:47.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:47.001255+0000) 2022-09-08T02:47:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:48.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:48.001491+0000) 2022-09-08T02:47:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:47.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:47.954216+0000) 2022-09-08T02:47:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:48 smithi055 ceph-mon[97033]: pgmap v9986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:48 smithi059 ceph-mon[97450]: pgmap v9986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:49.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:49.001711+0000) 2022-09-08T02:47:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:48.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:48.954451+0000) 2022-09-08T02:47:50.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:49.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:49.954704+0000) 2022-09-08T02:47:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:50.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:50.001890+0000) 2022-09-08T02:47:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:50 smithi055 ceph-mon[97033]: pgmap v9987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:50 smithi059 ceph-mon[97450]: pgmap v9987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:51.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:51.002115+0000) 2022-09-08T02:47:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:50.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:50.954941+0000) 2022-09-08T02:47:52.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:51.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:51.955182+0000) 2022-09-08T02:47:52.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:52.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:52.002338+0000) 2022-09-08T02:47:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:52 smithi055 ceph-mon[97033]: pgmap v9988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:47:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:52 smithi059 ceph-mon[97450]: pgmap v9988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:47:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:53.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:53.002528+0000) 2022-09-08T02:47:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:52.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:52.955384+0000) 2022-09-08T02:47:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:53.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:53.955584+0000) 2022-09-08T02:47:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:54.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:54.002715+0000) 2022-09-08T02:47:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:54 smithi055 ceph-mon[97033]: pgmap v9989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:54 smithi059 ceph-mon[97450]: pgmap v9989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:55.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:54.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:54.955807+0000) 2022-09-08T02:47:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:55.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:55.002952+0000) 2022-09-08T02:47:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:56.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:56.003197+0000) 2022-09-08T02:47:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:55.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:55.955959+0000) 2022-09-08T02:47:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:56 smithi055 ceph-mon[97033]: pgmap v9990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:47:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:56 smithi059 ceph-mon[97450]: pgmap v9990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:47:56.956 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:47:56.656+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:47:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:57.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:57.003392+0000) 2022-09-08T02:47:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:56.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:56.956184+0000) 2022-09-08T02:47:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:58.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:58.003631+0000) 2022-09-08T02:47:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:57.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:57.956420+0000) 2022-09-08T02:47:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:47:58 smithi055 ceph-mon[97033]: pgmap v9991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:47:58 smithi059 ceph-mon[97450]: pgmap v9991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:47:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:47:59.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:59.003884+0000) 2022-09-08T02:47:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:58.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:58.956604+0000) 2022-09-08T02:48:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:00.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:00.004141+0000) 2022-09-08T02:48:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:47:59.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:47:59.956761+0000) 2022-09-08T02:48:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:00 smithi055 ceph-mon[97033]: pgmap v9992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:00 smithi059 ceph-mon[97450]: pgmap v9992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:00.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:00.956901+0000) 2022-09-08T02:48:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:01.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:01.004298+0000) 2022-09-08T02:48:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:02.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:02.004510+0000) 2022-09-08T02:48:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:01.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:01.957112+0000) 2022-09-08T02:48:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:02 smithi055 ceph-mon[97033]: pgmap v9993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:48:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:02 smithi059 ceph-mon[97450]: pgmap v9993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:48:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:03.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:03.004690+0000) 2022-09-08T02:48:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:02.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:02.957348+0000) 2022-09-08T02:48:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:04.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:04.004929+0000) 2022-09-08T02:48:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:03.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:03.957595+0000) 2022-09-08T02:48:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:04 smithi055 ceph-mon[97033]: pgmap v9994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:04 smithi059 ceph-mon[97450]: pgmap v9994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:05.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:05.005132+0000) 2022-09-08T02:48:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:04.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:04.957817+0000) 2022-09-08T02:48:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:06.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:06.005360+0000) 2022-09-08T02:48:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:05.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:05.958053+0000) 2022-09-08T02:48:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:06 smithi055 ceph-mon[97033]: pgmap v9995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:48:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:48:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:48:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:06 smithi059 ceph-mon[97450]: pgmap v9995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:48:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:48:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:48:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:07.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:07.005594+0000) 2022-09-08T02:48:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:06.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:06.958272+0000) 2022-09-08T02:48:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:08.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:08.005818+0000) 2022-09-08T02:48:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:07.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:07.958542+0000) 2022-09-08T02:48:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:08 smithi055 ceph-mon[97033]: pgmap v9996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:48:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:08 smithi059 ceph-mon[97450]: pgmap v9996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:48:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:09.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:09.006032+0000) 2022-09-08T02:48:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:08.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:08.958712+0000) 2022-09-08T02:48:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:10.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:10.006257+0000) 2022-09-08T02:48:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:09.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:09.958955+0000) 2022-09-08T02:48:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:10 smithi055 ceph-mon[97033]: pgmap v9997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:48:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:10 smithi055 ceph-mon[97033]: osdmap e705: 12 total, 11 up, 11 in 2022-09-08T02:48:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:10 smithi059 ceph-mon[97450]: pgmap v9997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:48:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:10 smithi059 ceph-mon[97450]: osdmap e705: 12 total, 11 up, 11 in 2022-09-08T02:48:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:11.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:11.006490+0000) 2022-09-08T02:48:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:10.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:10.959146+0000) 2022-09-08T02:48:11.959 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:48:11.668+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:48:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:12.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:12.006710+0000) 2022-09-08T02:48:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:11.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:11.959369+0000) 2022-09-08T02:48:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:12 smithi055 ceph-mon[97033]: pgmap v9999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:48:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:12 smithi059 ceph-mon[97450]: pgmap v9999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:48:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:13.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:13.006864+0000) 2022-09-08T02:48:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:12.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:12.959586+0000) 2022-09-08T02:48:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:14.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:14.007077+0000) 2022-09-08T02:48:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:13.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:13.959808+0000) 2022-09-08T02:48:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:14 smithi055 ceph-mon[97033]: osdmap e706: 12 total, 11 up, 11 in 2022-09-08T02:48:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:14 smithi055 ceph-mon[97033]: pgmap v10001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:48:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:14 smithi059 ceph-mon[97450]: osdmap e706: 12 total, 11 up, 11 in 2022-09-08T02:48:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:14 smithi059 ceph-mon[97450]: pgmap v10001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:48:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:15.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:15.007315+0000) 2022-09-08T02:48:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:14.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:14.960036+0000) 2022-09-08T02:48:16.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:15.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:15.960211+0000) 2022-09-08T02:48:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:16.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:16.007553+0000) 2022-09-08T02:48:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:16 smithi055 ceph-mon[97033]: pgmap v10002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:48:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:16 smithi059 ceph-mon[97450]: pgmap v10002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:48:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:17.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:17.007797+0000) 2022-09-08T02:48:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:16.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:16.960436+0000) 2022-09-08T02:48:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:18.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:18.008035+0000) 2022-09-08T02:48:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:17.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:17.960643+0000) 2022-09-08T02:48:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:18 smithi055 ceph-mon[97033]: pgmap v10003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:48:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:18 smithi059 ceph-mon[97450]: pgmap v10003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:48:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:19.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:19.008219+0000) 2022-09-08T02:48:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:18.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:18.960853+0000) 2022-09-08T02:48:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:20.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:20.008387+0000) 2022-09-08T02:48:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:19.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:19.961123+0000) 2022-09-08T02:48:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:20 smithi055 ceph-mon[97033]: pgmap v10004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T02:48:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:20 smithi059 ceph-mon[97450]: pgmap v10004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T02:48:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:21.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:21.008565+0000) 2022-09-08T02:48:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:20.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:20.961331+0000) 2022-09-08T02:48:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:22.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:22.008789+0000) 2022-09-08T02:48:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:21.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:21.961561+0000) 2022-09-08T02:48:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:22 smithi055 ceph-mon[97033]: pgmap v10005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:48:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:22 smithi059 ceph-mon[97450]: pgmap v10005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:48:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:23.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:23.008999+0000) 2022-09-08T02:48:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:22.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:22.961808+0000) 2022-09-08T02:48:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:24.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:24.009188+0000) 2022-09-08T02:48:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:23.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:23.962081+0000) 2022-09-08T02:48:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:24 smithi055 ceph-mon[97033]: pgmap v10006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:48:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:24 smithi059 ceph-mon[97450]: pgmap v10006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:48:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:25.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:25.009370+0000) 2022-09-08T02:48:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:24.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:24.962268+0000) 2022-09-08T02:48:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:26.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:26.009558+0000) 2022-09-08T02:48:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:25.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:25.962494+0000) 2022-09-08T02:48:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:26 smithi055 ceph-mon[97033]: pgmap v10007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:48:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:26 smithi059 ceph-mon[97450]: pgmap v10007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:48:26.963 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:48:26.674+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:48:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:27.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:27.009795+0000) 2022-09-08T02:48:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:26.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:26.962737+0000) 2022-09-08T02:48:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:28.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:28.009981+0000) 2022-09-08T02:48:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:27.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:27.962966+0000) 2022-09-08T02:48:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:28 smithi055 ceph-mon[97033]: pgmap v10008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:28 smithi059 ceph-mon[97450]: pgmap v10008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:29.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:29.010214+0000) 2022-09-08T02:48:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:28.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:28.963191+0000) 2022-09-08T02:48:30.216 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:30.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:30.010372+0000) 2022-09-08T02:48:30.217 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:29.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:29.963421+0000) 2022-09-08T02:48:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:30 smithi055 ceph-mon[97033]: pgmap v10009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:48:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:48:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:48:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:30 smithi059 ceph-mon[97450]: pgmap v10009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:48:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:48:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:48:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:31.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:31.010597+0000) 2022-09-08T02:48:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:30.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:30.963625+0000) 2022-09-08T02:48:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:32.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:32.010890+0000) 2022-09-08T02:48:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:31.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:31.963876+0000) 2022-09-08T02:48:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:32 smithi055 ceph-mon[97033]: pgmap v10010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:48:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:32 smithi059 ceph-mon[97450]: pgmap v10010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:48:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:33.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:33.011139+0000) 2022-09-08T02:48:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:32.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:32.964105+0000) 2022-09-08T02:48:34.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:34.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:34.011379+0000) 2022-09-08T02:48:34.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:33.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:33.964366+0000) 2022-09-08T02:48:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:34 smithi055 ceph-mon[97033]: pgmap v10011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:34 smithi059 ceph-mon[97450]: pgmap v10011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:35.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:35.011677+0000) 2022-09-08T02:48:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:34.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:34.964561+0000) 2022-09-08T02:48:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:36.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:36.011862+0000) 2022-09-08T02:48:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:35.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:35.964799+0000) 2022-09-08T02:48:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:36 smithi055 ceph-mon[97033]: pgmap v10012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:48:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:36 smithi059 ceph-mon[97450]: pgmap v10012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:48:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:37.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:37.012080+0000) 2022-09-08T02:48:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:36.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:36.965007+0000) 2022-09-08T02:48:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:38.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:38.012282+0000) 2022-09-08T02:48:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:37.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:37.965210+0000) 2022-09-08T02:48:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:38 smithi055 ceph-mon[97033]: pgmap v10013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:38 smithi059 ceph-mon[97450]: pgmap v10013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:39.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:39.012497+0000) 2022-09-08T02:48:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:38.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:38.965434+0000) 2022-09-08T02:48:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:40.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:40.012660+0000) 2022-09-08T02:48:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:39.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:39.965728+0000) 2022-09-08T02:48:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:40 smithi055 ceph-mon[97033]: pgmap v10014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:40 smithi059 ceph-mon[97450]: pgmap v10014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:41.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:41.012895+0000) 2022-09-08T02:48:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:40.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:40.965905+0000) 2022-09-08T02:48:41.966 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:48:41.678+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:48:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:42.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:42.013093+0000) 2022-09-08T02:48:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:41.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:41.966132+0000) 2022-09-08T02:48:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:42 smithi055 ceph-mon[97033]: pgmap v10015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:48:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:42 smithi059 ceph-mon[97450]: pgmap v10015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:48:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:43.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:43.013384+0000) 2022-09-08T02:48:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:42.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:42.966353+0000) 2022-09-08T02:48:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:44.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:44.013614+0000) 2022-09-08T02:48:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:43.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:43.966573+0000) 2022-09-08T02:48:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:44 smithi055 ceph-mon[97033]: pgmap v10016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:44 smithi059 ceph-mon[97450]: pgmap v10016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:45.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:45.013851+0000) 2022-09-08T02:48:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:44.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:44.966818+0000) 2022-09-08T02:48:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:46.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:46.014012+0000) 2022-09-08T02:48:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:45.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:45.967045+0000) 2022-09-08T02:48:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:46 smithi055 ceph-mon[97033]: pgmap v10017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:48:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:46 smithi059 ceph-mon[97450]: pgmap v10017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:48:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:47.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:47.014200+0000) 2022-09-08T02:48:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:46.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:46.967285+0000) 2022-09-08T02:48:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:48.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:48.014384+0000) 2022-09-08T02:48:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:47.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:47.967555+0000) 2022-09-08T02:48:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:48 smithi055 ceph-mon[97033]: pgmap v10018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:48 smithi059 ceph-mon[97450]: pgmap v10018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:49.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:49.014560+0000) 2022-09-08T02:48:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:48.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:48.967731+0000) 2022-09-08T02:48:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:50.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:50.014773+0000) 2022-09-08T02:48:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:49.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:49.967860+0000) 2022-09-08T02:48:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:50 smithi055 ceph-mon[97033]: pgmap v10019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:50 smithi059 ceph-mon[97450]: pgmap v10019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:51.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:51.015045+0000) 2022-09-08T02:48:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:50.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:50.968084+0000) 2022-09-08T02:48:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:52.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:52.015274+0000) 2022-09-08T02:48:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:51.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:51.968289+0000) 2022-09-08T02:48:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:52 smithi055 ceph-mon[97033]: pgmap v10020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:48:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:52 smithi059 ceph-mon[97450]: pgmap v10020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:48:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:53.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:53.015534+0000) 2022-09-08T02:48:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:52.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:52.968509+0000) 2022-09-08T02:48:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:54.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:54.015799+0000) 2022-09-08T02:48:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:53.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:53.968726+0000) 2022-09-08T02:48:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:54 smithi055 ceph-mon[97033]: pgmap v10021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:54 smithi059 ceph-mon[97450]: pgmap v10021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:55.230 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:55.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:55.016044+0000) 2022-09-08T02:48:55.231 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:54.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:54.968976+0000) 2022-09-08T02:48:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:56.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:56.016232+0000) 2022-09-08T02:48:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:55.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:55.969191+0000) 2022-09-08T02:48:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:56 smithi055 ceph-mon[97033]: pgmap v10022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:48:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:56 smithi059 ceph-mon[97450]: pgmap v10022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:48:56.969 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:48:56.695+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:48:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:57.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:57.016464+0000) 2022-09-08T02:48:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:56.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:56.969405+0000) 2022-09-08T02:48:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:58.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:58.016593+0000) 2022-09-08T02:48:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:57.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:57.969601+0000) 2022-09-08T02:48:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:48:58 smithi055 ceph-mon[97033]: pgmap v10023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:48:58 smithi059 ceph-mon[97450]: pgmap v10023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:48:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:48:59.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:59.016841+0000) 2022-09-08T02:48:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:58.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:58.969850+0000) 2022-09-08T02:49:00.285 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:00.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:00.017065+0000) 2022-09-08T02:49:00.286 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:48:59.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:48:59.970070+0000) 2022-09-08T02:49:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:00 smithi055 ceph-mon[97033]: pgmap v10024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:00 smithi059 ceph-mon[97450]: pgmap v10024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:01.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:01.017229+0000) 2022-09-08T02:49:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:00.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:00.970315+0000) 2022-09-08T02:49:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:01 smithi055 ceph-mon[97033]: pgmap v10025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-08T02:49:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:49:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:01 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T02:49:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:01 smithi059 ceph-mon[97450]: pgmap v10025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-08T02:49:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:49:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:01 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T02:49:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:02.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:02.017427+0000) 2022-09-08T02:49:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:01.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:01.970540+0000) 2022-09-08T02:49:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:03.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:03.017632+0000) 2022-09-08T02:49:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:02.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:02.970784+0000) 2022-09-08T02:49:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:03 smithi055 ceph-mon[97033]: pgmap v10026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:49:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:03 smithi059 ceph-mon[97450]: pgmap v10026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:49:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:04.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:04.017933+0000) 2022-09-08T02:49:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:03.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:03.970990+0000) 2022-09-08T02:49:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:05.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:05.018175+0000) 2022-09-08T02:49:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:04.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:04.971185+0000) 2022-09-08T02:49:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:05 smithi055 ceph-mon[97033]: pgmap v10027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:49:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:05 smithi059 ceph-mon[97450]: pgmap v10027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:49:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:06.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:06.018387+0000) 2022-09-08T02:49:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:05.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:05.971400+0000) 2022-09-08T02:49:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:49:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:49:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:49:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:49:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:07.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:07.018597+0000) 2022-09-08T02:49:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:06.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:06.971627+0000) 2022-09-08T02:49:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:07 smithi055 ceph-mon[97033]: pgmap v10028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 48 KiB/s wr, 21 op/s 2022-09-08T02:49:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:07 smithi059 ceph-mon[97450]: pgmap v10028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 48 KiB/s wr, 21 op/s 2022-09-08T02:49:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:08.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:08.018807+0000) 2022-09-08T02:49:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:07.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:07.971898+0000) 2022-09-08T02:49:09.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:09.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:09.019043+0000) 2022-09-08T02:49:09.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:08.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:08.972117+0000) 2022-09-08T02:49:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:09 smithi055 ceph-mon[97033]: pgmap v10029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 48 KiB/s wr, 21 op/s 2022-09-08T02:49:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:09 smithi059 ceph-mon[97450]: pgmap v10029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 48 KiB/s wr, 21 op/s 2022-09-08T02:49:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:10.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:10.019236+0000) 2022-09-08T02:49:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:09.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:09.972362+0000) 2022-09-08T02:49:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:10 smithi055 ceph-mon[97033]: osdmap e707: 12 total, 11 up, 11 in 2022-09-08T02:49:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:10 smithi059 ceph-mon[97450]: osdmap e707: 12 total, 11 up, 11 in 2022-09-08T02:49:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:11.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:11.019474+0000) 2022-09-08T02:49:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:10.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:10.972511+0000) 2022-09-08T02:49:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:11 smithi055 ceph-mon[97033]: pgmap v10031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:49:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:11 smithi059 ceph-mon[97450]: pgmap v10031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:49:11.972 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:49:11.710+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:49:12.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:11.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:11.972701+0000) 2022-09-08T02:49:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:12.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:12.019672+0000) 2022-09-08T02:49:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:13.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:13.019886+0000) 2022-09-08T02:49:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:12.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:12.972848+0000) 2022-09-08T02:49:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:13 smithi055 ceph-mon[97033]: pgmap v10032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:49:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:13 smithi055 ceph-mon[97033]: osdmap e708: 12 total, 11 up, 11 in 2022-09-08T02:49:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:13 smithi059 ceph-mon[97450]: pgmap v10032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:49:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:13 smithi059 ceph-mon[97450]: osdmap e708: 12 total, 11 up, 11 in 2022-09-08T02:49:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:14.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:14.020137+0000) 2022-09-08T02:49:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:13.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:13.973072+0000) 2022-09-08T02:49:15.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:14.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:14.973252+0000) 2022-09-08T02:49:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:15.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:15.020336+0000) 2022-09-08T02:49:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:15 smithi055 ceph-mon[97033]: pgmap v10034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:49:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:15 smithi059 ceph-mon[97450]: pgmap v10034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:49:16.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:15.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:15.973473+0000) 2022-09-08T02:49:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:16.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:16.020557+0000) 2022-09-08T02:49:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:17.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:17.020798+0000) 2022-09-08T02:49:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:16.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:16.973701+0000) 2022-09-08T02:49:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:17 smithi055 ceph-mon[97033]: pgmap v10035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:49:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:17 smithi059 ceph-mon[97450]: pgmap v10035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:49:18.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:17.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:17.973848+0000) 2022-09-08T02:49:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:18.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:18.021039+0000) 2022-09-08T02:49:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:19.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:19.021276+0000) 2022-09-08T02:49:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:18.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:18.974057+0000) 2022-09-08T02:49:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:19 smithi055 ceph-mon[97033]: pgmap v10036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2022-09-08T02:49:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:19 smithi059 ceph-mon[97450]: pgmap v10036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2022-09-08T02:49:20.356 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:20.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:20.021579+0000) 2022-09-08T02:49:20.357 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:19.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:19.974292+0000) 2022-09-08T02:49:21.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:20.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:20.974513+0000) 2022-09-08T02:49:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:21.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:21.021831+0000) 2022-09-08T02:49:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:21 smithi055 ceph-mon[97033]: pgmap v10037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:49:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:21 smithi059 ceph-mon[97450]: pgmap v10037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:49:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:22.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:22.022096+0000) 2022-09-08T02:49:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:21.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:21.974755+0000) 2022-09-08T02:49:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:23.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:23.022286+0000) 2022-09-08T02:49:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:22.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:22.974906+0000) 2022-09-08T02:49:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:23 smithi055 ceph-mon[97033]: pgmap v10038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:49:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:23 smithi059 ceph-mon[97450]: pgmap v10038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:49:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:24.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:24.022511+0000) 2022-09-08T02:49:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:23.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:23.975078+0000) 2022-09-08T02:49:25.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:24.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:24.975281+0000) 2022-09-08T02:49:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:25.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:25.022698+0000) 2022-09-08T02:49:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:25 smithi055 ceph-mon[97033]: pgmap v10039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T02:49:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:25 smithi059 ceph-mon[97450]: pgmap v10039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T02:49:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:26.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:26.022910+0000) 2022-09-08T02:49:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:25.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:25.975514+0000) 2022-09-08T02:49:26.975 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:49:26.719+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:49:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:27.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:27.023154+0000) 2022-09-08T02:49:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:26.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:26.975673+0000) 2022-09-08T02:49:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:27 smithi055 ceph-mon[97033]: pgmap v10040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:49:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:27 smithi059 ceph-mon[97450]: pgmap v10040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:49:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:28.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:28.023377+0000) 2022-09-08T02:49:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:27.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:27.975882+0000) 2022-09-08T02:49:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:29.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:29.023579+0000) 2022-09-08T02:49:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:28.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:28.976094+0000) 2022-09-08T02:49:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:29 smithi055 ceph-mon[97033]: pgmap v10041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:29 smithi059 ceph-mon[97450]: pgmap v10041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:30.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:30.023772+0000) 2022-09-08T02:49:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:29.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:29.976260+0000) 2022-09-08T02:49:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:30.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:30.976504+0000) 2022-09-08T02:49:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:31.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:31.023984+0000) 2022-09-08T02:49:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:31 smithi055 ceph-mon[97033]: pgmap v10042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:49:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:31 smithi059 ceph-mon[97450]: pgmap v10042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:49:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:32.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:32.024188+0000) 2022-09-08T02:49:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:31.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:31.976780+0000) 2022-09-08T02:49:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:33.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:33.024424+0000) 2022-09-08T02:49:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:32.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:32.976995+0000) 2022-09-08T02:49:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:33 smithi055 ceph-mon[97033]: pgmap v10043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:33 smithi059 ceph-mon[97450]: pgmap v10043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:34.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:34.024676+0000) 2022-09-08T02:49:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:33.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:33.977174+0000) 2022-09-08T02:49:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:35.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:35.024847+0000) 2022-09-08T02:49:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:34.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:34.977400+0000) 2022-09-08T02:49:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:35 smithi055 ceph-mon[97033]: pgmap v10044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:35 smithi059 ceph-mon[97450]: pgmap v10044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:36.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:36.025048+0000) 2022-09-08T02:49:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:35.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:35.977595+0000) 2022-09-08T02:49:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:37.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:37.025295+0000) 2022-09-08T02:49:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:36.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:36.977829+0000) 2022-09-08T02:49:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:37 smithi055 ceph-mon[97033]: pgmap v10045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:49:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:37 smithi059 ceph-mon[97450]: pgmap v10045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:49:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:38.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:38.025504+0000) 2022-09-08T02:49:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:37.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:37.978081+0000) 2022-09-08T02:49:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:39.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:39.025742+0000) 2022-09-08T02:49:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:38.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:38.978318+0000) 2022-09-08T02:49:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:39 smithi055 ceph-mon[97033]: pgmap v10046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:39 smithi059 ceph-mon[97450]: pgmap v10046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:40.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:40.025961+0000) 2022-09-08T02:49:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:39.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:39.978601+0000) 2022-09-08T02:49:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:41.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:41.026143+0000) 2022-09-08T02:49:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:40.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:40.978851+0000) 2022-09-08T02:49:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:41 smithi055 ceph-mon[97033]: pgmap v10047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:49:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:41 smithi059 ceph-mon[97450]: pgmap v10047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:49:41.979 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:49:41.724+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:49:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:42.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:42.026373+0000) 2022-09-08T02:49:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:41.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:41.979079+0000) 2022-09-08T02:49:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:43.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:43.026614+0000) 2022-09-08T02:49:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:42.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:42.979315+0000) 2022-09-08T02:49:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:43 smithi055 ceph-mon[97033]: pgmap v10048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:43 smithi059 ceph-mon[97450]: pgmap v10048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:44.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:44.026877+0000) 2022-09-08T02:49:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:43.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:43.979550+0000) 2022-09-08T02:49:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:45.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:45.027124+0000) 2022-09-08T02:49:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:44.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:44.979774+0000) 2022-09-08T02:49:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:45 smithi055 ceph-mon[97033]: pgmap v10049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:45 smithi059 ceph-mon[97450]: pgmap v10049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:46.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:46.027353+0000) 2022-09-08T02:49:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:45.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:45.979928+0000) 2022-09-08T02:49:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:47.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:47.027544+0000) 2022-09-08T02:49:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:46.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:46.980173+0000) 2022-09-08T02:49:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:47 smithi055 ceph-mon[97033]: pgmap v10050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:49:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:47 smithi059 ceph-mon[97450]: pgmap v10050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:49:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:48.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:48.027764+0000) 2022-09-08T02:49:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:47.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:47.980404+0000) 2022-09-08T02:49:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:49.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:49.028004+0000) 2022-09-08T02:49:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:48.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:48.980662+0000) 2022-09-08T02:49:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:49 smithi055 ceph-mon[97033]: pgmap v10051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:49 smithi059 ceph-mon[97450]: pgmap v10051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:50.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:50.028225+0000) 2022-09-08T02:49:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:49.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:49.980916+0000) 2022-09-08T02:49:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:51.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:51.028392+0000) 2022-09-08T02:49:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:50.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:50.981168+0000) 2022-09-08T02:49:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:51 smithi055 ceph-mon[97033]: pgmap v10052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:49:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:51 smithi059 ceph-mon[97450]: pgmap v10052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:49:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:52.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:52.028617+0000) 2022-09-08T02:49:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:51.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:51.981361+0000) 2022-09-08T02:49:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:53.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:53.028745+0000) 2022-09-08T02:49:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:52.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:52.981561+0000) 2022-09-08T02:49:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:53 smithi055 ceph-mon[97033]: pgmap v10053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:53 smithi059 ceph-mon[97450]: pgmap v10053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:54.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:54.028986+0000) 2022-09-08T02:49:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:53.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:53.981741+0000) 2022-09-08T02:49:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:55.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:55.029261+0000) 2022-09-08T02:49:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:54.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:54.981975+0000) 2022-09-08T02:49:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:55 smithi055 ceph-mon[97033]: pgmap v10054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:55 smithi059 ceph-mon[97450]: pgmap v10054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:56.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:56.029506+0000) 2022-09-08T02:49:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:55.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:55.982159+0000) 2022-09-08T02:49:57.038 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:49:56.731+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:49:57.039 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:56.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:56.982322+0000) 2022-09-08T02:49:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:57.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:57.029713+0000) 2022-09-08T02:49:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:57 smithi055 ceph-mon[97033]: pgmap v10055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:49:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:57 smithi059 ceph-mon[97450]: pgmap v10055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:49:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:58.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:58.029878+0000) 2022-09-08T02:49:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:57.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:57.982526+0000) 2022-09-08T02:49:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:49:59.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:59.030085+0000) 2022-09-08T02:49:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:58.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:58.982728+0000) 2022-09-08T02:49:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:49:59 smithi055 ceph-mon[97033]: pgmap v10056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:49:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:49:59 smithi059 ceph-mon[97450]: pgmap v10056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:00.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:00.030334+0000) 2022-09-08T02:50:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:49:59.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:49:59.982895+0000) 2022-09-08T02:50:00.460 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T02:50:00.461 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T02:50:00.462 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T02:50:00.463 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T02:50:00.463 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T02:50:00.466 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T02:50:00.467 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T02:50:00.467 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T02:50:00.468 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T02:50:00.468 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T02:50:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:01 smithi055 ceph-mon[97033]: pgmap v10057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:50:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:50:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:50:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:50:01.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:50:01.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:50:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:01.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:01.030598+0000) 2022-09-08T02:50:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:00.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:00.983091+0000) 2022-09-08T02:50:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:01 smithi059 ceph-mon[97450]: pgmap v10057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:50:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:50:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:50:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:50:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:50:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:50:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:02.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:02.030882+0000) 2022-09-08T02:50:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:01.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:01.983320+0000) 2022-09-08T02:50:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:03.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:03.031098+0000) 2022-09-08T02:50:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:02.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:02.983552+0000) 2022-09-08T02:50:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:03 smithi055 ceph-mon[97033]: pgmap v10058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:03 smithi059 ceph-mon[97450]: pgmap v10058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:04.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:04.031345+0000) 2022-09-08T02:50:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:03.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:03.983782+0000) 2022-09-08T02:50:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:50:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:50:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:50:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:50:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:04 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T02:50:04.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:04 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T02:50:04.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:50:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:50:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:50:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:50:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:04 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T02:50:04.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:04 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T02:50:04.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:04.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T02:50:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:05.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:05.031579+0000) 2022-09-08T02:50:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:04.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:04.984024+0000) 2022-09-08T02:50:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:05 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T02:50:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:05 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T02:50:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:05 smithi059 ceph-mon[97450]: pgmap v10059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:50:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:50:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:05 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T02:50:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:05 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T02:50:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:05 smithi055 ceph-mon[97033]: pgmap v10059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:50:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:50:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:06.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:06.031830+0000) 2022-09-08T02:50:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:05.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:05.984238+0000) 2022-09-08T02:50:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:07.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:07.032070+0000) 2022-09-08T02:50:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:06.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:06.984480+0000) 2022-09-08T02:50:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:07 smithi059 ceph-mon[97450]: pgmap v10060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:50:08.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:07 smithi055 ceph-mon[97033]: pgmap v10060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:50:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:08.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:08.032279+0000) 2022-09-08T02:50:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:07.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:07.984737+0000) 2022-09-08T02:50:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:09.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:09.032494+0000) 2022-09-08T02:50:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:08.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:08.985001+0000) 2022-09-08T02:50:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:09 smithi055 ceph-mon[97033]: pgmap v10061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:50:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:09 smithi059 ceph-mon[97450]: pgmap v10061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:50:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:10.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:10.032775+0000) 2022-09-08T02:50:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:09.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:09.985224+0000) 2022-09-08T02:50:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:10 smithi055 ceph-mon[97033]: osdmap e709: 12 total, 11 up, 11 in 2022-09-08T02:50:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:10 smithi059 ceph-mon[97450]: osdmap e709: 12 total, 11 up, 11 in 2022-09-08T02:50:11.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:10.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:10.985436+0000) 2022-09-08T02:50:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:11.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:11.032950+0000) 2022-09-08T02:50:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:11 smithi055 ceph-mon[97033]: pgmap v10063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:50:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:11 smithi059 ceph-mon[97450]: pgmap v10063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:50:12.033 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:50:11.744+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:50:12.034 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:11.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:11.985600+0000) 2022-09-08T02:50:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:12.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:12.033115+0000) 2022-09-08T02:50:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:13.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:13.033272+0000) 2022-09-08T02:50:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:12.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:12.985870+0000) 2022-09-08T02:50:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:13 smithi055 ceph-mon[97033]: pgmap v10064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:50:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:13 smithi055 ceph-mon[97033]: osdmap e710: 12 total, 11 up, 11 in 2022-09-08T02:50:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:13 smithi059 ceph-mon[97450]: pgmap v10064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:50:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:13 smithi059 ceph-mon[97450]: osdmap e710: 12 total, 11 up, 11 in 2022-09-08T02:50:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:14.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:14.033511+0000) 2022-09-08T02:50:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:13.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:13.986114+0000) 2022-09-08T02:50:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:15.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:15.033726+0000) 2022-09-08T02:50:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:14.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:14.986363+0000) 2022-09-08T02:50:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:15 smithi055 ceph-mon[97033]: pgmap v10066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:50:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:15 smithi059 ceph-mon[97450]: pgmap v10066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:50:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:16.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:16.033973+0000) 2022-09-08T02:50:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:15.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:15.986538+0000) 2022-09-08T02:50:17.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:16.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:16.986734+0000) 2022-09-08T02:50:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:17.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:17.034201+0000) 2022-09-08T02:50:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:17 smithi055 ceph-mon[97033]: pgmap v10067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:50:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:17 smithi059 ceph-mon[97450]: pgmap v10067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:50:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:18.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:18.034391+0000) 2022-09-08T02:50:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:17.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:17.986947+0000) 2022-09-08T02:50:19.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:18.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:18.987151+0000) 2022-09-08T02:50:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:19.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:19.034663+0000) 2022-09-08T02:50:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:19 smithi055 ceph-mon[97033]: pgmap v10068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2022-09-08T02:50:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:19 smithi059 ceph-mon[97450]: pgmap v10068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2022-09-08T02:50:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:20.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:20.034861+0000) 2022-09-08T02:50:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:19.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:19.987398+0000) 2022-09-08T02:50:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:21.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:21.035136+0000) 2022-09-08T02:50:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:20.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:20.987584+0000) 2022-09-08T02:50:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:21 smithi055 ceph-mon[97033]: pgmap v10069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:50:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:21 smithi059 ceph-mon[97450]: pgmap v10069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:50:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:22.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:22.035313+0000) 2022-09-08T02:50:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:21.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:21.987805+0000) 2022-09-08T02:50:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:23.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:23.035525+0000) 2022-09-08T02:50:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:22.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:22.988049+0000) 2022-09-08T02:50:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:23 smithi055 ceph-mon[97033]: pgmap v10070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:50:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:23 smithi059 ceph-mon[97450]: pgmap v10070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:50:24.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:23.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:23.988403+0000) 2022-09-08T02:50:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:24.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:24.035769+0000) 2022-09-08T02:50:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:25.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:25.035957+0000) 2022-09-08T02:50:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:24.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:24.988697+0000) 2022-09-08T02:50:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:25 smithi055 ceph-mon[97033]: pgmap v10071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T02:50:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:25 smithi059 ceph-mon[97450]: pgmap v10071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T02:50:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:26.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:26.036172+0000) 2022-09-08T02:50:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:25.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:25.988900+0000) 2022-09-08T02:50:27.037 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:50:26.759+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:50:27.038 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:26.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:26.989105+0000) 2022-09-08T02:50:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:27.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:27.036402+0000) 2022-09-08T02:50:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:27 smithi055 ceph-mon[97033]: pgmap v10072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:50:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:27 smithi059 ceph-mon[97450]: pgmap v10072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:50:28.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:28.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:28.036688+0000) 2022-09-08T02:50:28.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:27.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:27.989343+0000) 2022-09-08T02:50:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:29.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:29.036912+0000) 2022-09-08T02:50:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:28.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:28.989590+0000) 2022-09-08T02:50:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:29 smithi055 ceph-mon[97033]: pgmap v10073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:29 smithi059 ceph-mon[97450]: pgmap v10073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:30.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:30.037132+0000) 2022-09-08T02:50:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:29.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:29.989869+0000) 2022-09-08T02:50:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:31.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:31.037340+0000) 2022-09-08T02:50:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:30.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:30.990110+0000) 2022-09-08T02:50:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:31 smithi055 ceph-mon[97033]: pgmap v10074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:50:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:31 smithi055 ceph-mon[97033]: pgmap v10075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:50:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:31 smithi055 ceph-mon[97033]: pgmap v10076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:50:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:50:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:31 smithi059 ceph-mon[97450]: pgmap v10074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:50:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:31 smithi059 ceph-mon[97450]: pgmap v10075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:50:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:31 smithi059 ceph-mon[97450]: pgmap v10076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:50:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:50:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:32.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:32.037597+0000) 2022-09-08T02:50:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:31.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:31.990352+0000) 2022-09-08T02:50:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:32.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:32.990559+0000) 2022-09-08T02:50:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:33.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:33.037860+0000) 2022-09-08T02:50:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:33 smithi055 ceph-mon[97033]: pgmap v10077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:50:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:33 smithi059 ceph-mon[97450]: pgmap v10077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:50:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:33.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:33.990767+0000) 2022-09-08T02:50:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:34.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:34.038067+0000) 2022-09-08T02:50:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:35.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:35.038281+0000) 2022-09-08T02:50:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:34.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:34.990922+0000) 2022-09-08T02:50:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:35 smithi055 ceph-mon[97033]: pgmap v10078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:50:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:35 smithi059 ceph-mon[97450]: pgmap v10078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:50:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:36.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:36.038488+0000) 2022-09-08T02:50:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:35.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:35.991140+0000) 2022-09-08T02:50:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:37.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:37.038693+0000) 2022-09-08T02:50:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:36.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:36.991385+0000) 2022-09-08T02:50:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:37 smithi055 ceph-mon[97033]: pgmap v10079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:50:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:37 smithi059 ceph-mon[97450]: pgmap v10079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:50:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:38.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:38.038863+0000) 2022-09-08T02:50:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:37.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:37.991613+0000) 2022-09-08T02:50:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:39.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:39.039084+0000) 2022-09-08T02:50:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:38.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:38.991859+0000) 2022-09-08T02:50:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:39 smithi055 ceph-mon[97033]: pgmap v10080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:50:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:39 smithi059 ceph-mon[97450]: pgmap v10080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:50:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:40.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:40.039323+0000) 2022-09-08T02:50:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:39.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:39.992146+0000) 2022-09-08T02:50:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:41.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:41.039548+0000) 2022-09-08T02:50:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:40.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:40.992382+0000) 2022-09-08T02:50:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:41 smithi055 ceph-mon[97033]: pgmap v10081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:50:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:41 smithi059 ceph-mon[97450]: pgmap v10081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:50:42.040 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:50:41.766+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:50:42.041 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:41.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:41.992567+0000) 2022-09-08T02:50:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:42.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:42.039802+0000) 2022-09-08T02:50:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:43.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:43.040052+0000) 2022-09-08T02:50:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:42.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:42.992778+0000) 2022-09-08T02:50:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:43 smithi055 ceph-mon[97033]: pgmap v10082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:43 smithi059 ceph-mon[97450]: pgmap v10082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:44.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:43.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:43.993028+0000) 2022-09-08T02:50:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:44.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:44.040297+0000) 2022-09-08T02:50:45.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:44.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:44.993301+0000) 2022-09-08T02:50:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:45.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:45.040471+0000) 2022-09-08T02:50:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:45 smithi055 ceph-mon[97033]: pgmap v10083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:45 smithi059 ceph-mon[97450]: pgmap v10083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:46.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:46.040732+0000) 2022-09-08T02:50:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:45.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:45.993509+0000) 2022-09-08T02:50:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:47.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:47.040871+0000) 2022-09-08T02:50:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:46.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:46.993724+0000) 2022-09-08T02:50:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:47 smithi055 ceph-mon[97033]: pgmap v10084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:50:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:47 smithi059 ceph-mon[97450]: pgmap v10084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:50:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:48.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:48.041095+0000) 2022-09-08T02:50:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:47.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:47.993967+0000) 2022-09-08T02:50:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:49.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:49.041304+0000) 2022-09-08T02:50:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:48.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:48.994187+0000) 2022-09-08T02:50:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:49 smithi055 ceph-mon[97033]: pgmap v10085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:49 smithi059 ceph-mon[97450]: pgmap v10085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:50.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:50.041453+0000) 2022-09-08T02:50:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:49.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:49.994329+0000) 2022-09-08T02:50:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:51.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:51.041623+0000) 2022-09-08T02:50:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:50.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:50.994520+0000) 2022-09-08T02:50:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:51 smithi055 ceph-mon[97033]: pgmap v10086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:50:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:51 smithi059 ceph-mon[97450]: pgmap v10086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:50:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:52.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:52.041845+0000) 2022-09-08T02:50:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:51.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:51.994760+0000) 2022-09-08T02:50:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:53.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:53.042066+0000) 2022-09-08T02:50:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:52.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:52.994976+0000) 2022-09-08T02:50:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:53 smithi055 ceph-mon[97033]: pgmap v10087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:53 smithi059 ceph-mon[97450]: pgmap v10087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:54.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:54.042312+0000) 2022-09-08T02:50:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:53.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:53.995200+0000) 2022-09-08T02:50:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:54.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:54.995441+0000) 2022-09-08T02:50:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:55.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:55.042525+0000) 2022-09-08T02:50:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:55 smithi055 ceph-mon[97033]: pgmap v10088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:55 smithi059 ceph-mon[97450]: pgmap v10088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:56.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:56.042756+0000) 2022-09-08T02:50:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:55.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:55.995708+0000) 2022-09-08T02:50:57.043 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:50:56.766+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:50:57.044 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:56.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:56.995924+0000) 2022-09-08T02:50:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:57.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:57.043002+0000) 2022-09-08T02:50:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:57 smithi055 ceph-mon[97033]: pgmap v10089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:50:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:57 smithi059 ceph-mon[97450]: pgmap v10089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:50:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:58.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:58.043184+0000) 2022-09-08T02:50:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:57.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:57.996173+0000) 2022-09-08T02:50:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:58.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:58.996424+0000) 2022-09-08T02:50:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:50:59.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:59.043412+0000) 2022-09-08T02:50:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:50:59 smithi055 ceph-mon[97033]: pgmap v10090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:50:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:50:59 smithi059 ceph-mon[97450]: pgmap v10090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:00.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:00.043685+0000) 2022-09-08T02:51:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:50:59.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:50:59.996573+0000) 2022-09-08T02:51:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:00.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:00.996784+0000) 2022-09-08T02:51:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:01.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:01.043896+0000) 2022-09-08T02:51:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:01 smithi055 ceph-mon[97033]: pgmap v10091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:51:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:01 smithi059 ceph-mon[97450]: pgmap v10091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:51:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:02.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:02.044134+0000) 2022-09-08T02:51:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:01.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:01.997014+0000) 2022-09-08T02:51:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:03.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:03.044378+0000) 2022-09-08T02:51:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:02.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:02.997217+0000) 2022-09-08T02:51:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:03 smithi055 ceph-mon[97033]: pgmap v10092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:03 smithi059 ceph-mon[97450]: pgmap v10092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:04.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:04.044627+0000) 2022-09-08T02:51:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:03.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:03.997452+0000) 2022-09-08T02:51:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:05.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:05.044825+0000) 2022-09-08T02:51:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:04.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:04.997624+0000) 2022-09-08T02:51:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:05 smithi055 ceph-mon[97033]: pgmap v10093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:51:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:51:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:05 smithi059 ceph-mon[97450]: pgmap v10093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:51:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:51:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:06.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:06.045007+0000) 2022-09-08T02:51:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:05.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:05.997857+0000) 2022-09-08T02:51:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:07.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:07.045245+0000) 2022-09-08T02:51:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:06.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:06.998117+0000) 2022-09-08T02:51:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:07 smithi055 ceph-mon[97033]: pgmap v10094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:51:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:07 smithi059 ceph-mon[97450]: pgmap v10094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:51:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:08.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:08.045508+0000) 2022-09-08T02:51:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:07.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:07.998393+0000) 2022-09-08T02:51:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:09.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:09.045756+0000) 2022-09-08T02:51:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:08.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:08.998614+0000) 2022-09-08T02:51:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:09 smithi055 ceph-mon[97033]: pgmap v10095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:51:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:09 smithi059 ceph-mon[97450]: pgmap v10095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:51:10.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:09.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:09.998816+0000) 2022-09-08T02:51:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:10.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:10.045977+0000) 2022-09-08T02:51:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:10 smithi055 ceph-mon[97033]: osdmap e711: 12 total, 11 up, 11 in 2022-09-08T02:51:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:10 smithi059 ceph-mon[97450]: osdmap e711: 12 total, 11 up, 11 in 2022-09-08T02:51:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:11.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:11.046195+0000) 2022-09-08T02:51:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:10.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:10.998987+0000) 2022-09-08T02:51:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:11 smithi055 ceph-mon[97033]: pgmap v10097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:51:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:11 smithi059 ceph-mon[97450]: pgmap v10097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:51:12.048 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:51:11.772+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:51:12.048 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:11.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:11.999219+0000) 2022-09-08T02:51:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:12.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:12.046403+0000) 2022-09-08T02:51:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:13.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:13.046664+0000) 2022-09-08T02:51:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:12.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:12.999465+0000) 2022-09-08T02:51:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:13 smithi055 ceph-mon[97033]: pgmap v10098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:51:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:13 smithi055 ceph-mon[97033]: osdmap e712: 12 total, 11 up, 11 in 2022-09-08T02:51:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:13 smithi059 ceph-mon[97450]: pgmap v10098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:51:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:13 smithi059 ceph-mon[97450]: osdmap e712: 12 total, 11 up, 11 in 2022-09-08T02:51:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:14.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:14.046919+0000) 2022-09-08T02:51:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:13.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:13.999702+0000) 2022-09-08T02:51:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:15.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:15.047142+0000) 2022-09-08T02:51:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:14.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:14.999847+0000) 2022-09-08T02:51:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:15 smithi055 ceph-mon[97033]: pgmap v10100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:51:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:15 smithi059 ceph-mon[97450]: pgmap v10100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:51:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:16.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:16.047355+0000) 2022-09-08T02:51:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:15.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:15.999985+0000) 2022-09-08T02:51:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:17.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:17.047564+0000) 2022-09-08T02:51:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:16.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:17.000190+0000) 2022-09-08T02:51:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:17 smithi055 ceph-mon[97033]: pgmap v10101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:51:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:17 smithi059 ceph-mon[97450]: pgmap v10101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:51:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:18.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:18.047761+0000) 2022-09-08T02:51:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:17.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:18.000411+0000) 2022-09-08T02:51:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:19.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:19.047969+0000) 2022-09-08T02:51:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:19.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:19.000601+0000) 2022-09-08T02:51:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:19 smithi055 ceph-mon[97033]: pgmap v10102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 217 B/s wr, 6 op/s 2022-09-08T02:51:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:19 smithi059 ceph-mon[97450]: pgmap v10102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 217 B/s wr, 6 op/s 2022-09-08T02:51:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:20.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:20.048145+0000) 2022-09-08T02:51:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:20.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:20.000814+0000) 2022-09-08T02:51:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:21.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:21.048388+0000) 2022-09-08T02:51:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:21.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:21.000993+0000) 2022-09-08T02:51:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:21 smithi055 ceph-mon[97033]: pgmap v10103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:51:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:21 smithi059 ceph-mon[97450]: pgmap v10103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:51:22.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:22.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:22.001176+0000) 2022-09-08T02:51:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:22.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:22.048603+0000) 2022-09-08T02:51:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:23.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:23.048823+0000) 2022-09-08T02:51:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:23.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:23.001320+0000) 2022-09-08T02:51:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:23 smithi055 ceph-mon[97033]: pgmap v10104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:51:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:23 smithi059 ceph-mon[97450]: pgmap v10104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:51:24.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:24.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:24.001533+0000) 2022-09-08T02:51:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:24.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:24.049087+0000) 2022-09-08T02:51:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:25.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:25.049333+0000) 2022-09-08T02:51:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:25.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:25.001775+0000) 2022-09-08T02:51:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:25 smithi055 ceph-mon[97033]: pgmap v10105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T02:51:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:25 smithi059 ceph-mon[97450]: pgmap v10105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T02:51:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:26.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:26.049564+0000) 2022-09-08T02:51:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:26.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:26.001970+0000) 2022-09-08T02:51:27.050 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:51:26.784+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:51:27.051 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:27.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:27.002140+0000) 2022-09-08T02:51:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:27.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:27.049797+0000) 2022-09-08T02:51:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:27 smithi055 ceph-mon[97033]: pgmap v10106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:51:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:27 smithi059 ceph-mon[97450]: pgmap v10106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:51:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:28.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:28.049976+0000) 2022-09-08T02:51:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:28.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:28.002325+0000) 2022-09-08T02:51:29.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:29.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:29.002501+0000) 2022-09-08T02:51:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:29.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:29.050195+0000) 2022-09-08T02:51:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:29 smithi055 ceph-mon[97033]: pgmap v10107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:29 smithi059 ceph-mon[97450]: pgmap v10107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:30.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:30.050511+0000) 2022-09-08T02:51:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:30.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:30.002750+0000) 2022-09-08T02:51:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:31.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:31.050784+0000) 2022-09-08T02:51:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:31.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:31.002894+0000) 2022-09-08T02:51:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:31 smithi055 ceph-mon[97033]: pgmap v10108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:51:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:51:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:51:31.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:51:31.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:51:31.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:51:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:31 smithi059 ceph-mon[97450]: pgmap v10108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:51:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:51:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:51:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:51:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:51:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:51:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:32.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:32.050998+0000) 2022-09-08T02:51:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:32.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:32.003113+0000) 2022-09-08T02:51:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:33.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:33.051223+0000) 2022-09-08T02:51:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:33.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:33.003351+0000) 2022-09-08T02:51:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:33 smithi055 ceph-mon[97033]: pgmap v10109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:33 smithi059 ceph-mon[97450]: pgmap v10109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:34.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:34.051456+0000) 2022-09-08T02:51:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:34.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:34.003566+0000) 2022-09-08T02:51:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:35.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:35.051733+0000) 2022-09-08T02:51:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:35.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:35.003813+0000) 2022-09-08T02:51:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:35 smithi055 ceph-mon[97033]: pgmap v10110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:35 smithi059 ceph-mon[97450]: pgmap v10110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:36.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:36.051973+0000) 2022-09-08T02:51:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:36.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:36.004004+0000) 2022-09-08T02:51:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:37.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:37.052161+0000) 2022-09-08T02:51:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:37.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:37.004245+0000) 2022-09-08T02:51:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:37 smithi055 ceph-mon[97033]: pgmap v10111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:51:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:37 smithi059 ceph-mon[97450]: pgmap v10111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:51:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:38.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:38.052404+0000) 2022-09-08T02:51:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:38.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:38.004466+0000) 2022-09-08T02:51:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:39.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:39.052680+0000) 2022-09-08T02:51:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:39.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:39.004711+0000) 2022-09-08T02:51:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:39 smithi055 ceph-mon[97033]: pgmap v10112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:39 smithi059 ceph-mon[97450]: pgmap v10112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:40.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:40.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:40.004952+0000) 2022-09-08T02:51:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:40.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:40.052820+0000) 2022-09-08T02:51:41.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:41.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:41.005159+0000) 2022-09-08T02:51:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:41.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:41.053041+0000) 2022-09-08T02:51:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:41 smithi055 ceph-mon[97033]: pgmap v10113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:51:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:41 smithi059 ceph-mon[97450]: pgmap v10113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:51:42.053 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:51:41.797+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:51:42.054 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:42.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:42.005373+0000) 2022-09-08T02:51:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:42.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:42.053254+0000) 2022-09-08T02:51:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:43.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:43.053417+0000) 2022-09-08T02:51:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:43.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:43.005535+0000) 2022-09-08T02:51:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:43 smithi055 ceph-mon[97033]: pgmap v10114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:43 smithi059 ceph-mon[97450]: pgmap v10114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:44.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:44.053621+0000) 2022-09-08T02:51:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:44.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:44.005734+0000) 2022-09-08T02:51:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:45.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:45.053826+0000) 2022-09-08T02:51:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:45.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:45.005993+0000) 2022-09-08T02:51:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:45 smithi055 ceph-mon[97033]: pgmap v10115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:45 smithi059 ceph-mon[97450]: pgmap v10115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:46.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:46.054047+0000) 2022-09-08T02:51:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:46.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:46.006194+0000) 2022-09-08T02:51:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:47.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:47.054261+0000) 2022-09-08T02:51:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:47.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:47.006422+0000) 2022-09-08T02:51:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:47 smithi055 ceph-mon[97033]: pgmap v10116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:51:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:47 smithi059 ceph-mon[97450]: pgmap v10116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:51:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:48.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:48.054462+0000) 2022-09-08T02:51:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:48.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:48.006683+0000) 2022-09-08T02:51:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:49.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:49.054661+0000) 2022-09-08T02:51:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:49.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:49.006885+0000) 2022-09-08T02:51:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:49 smithi055 ceph-mon[97033]: pgmap v10117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:49 smithi059 ceph-mon[97450]: pgmap v10117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:50.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:50.054855+0000) 2022-09-08T02:51:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:50.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:50.007056+0000) 2022-09-08T02:51:51.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:51.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:51.007257+0000) 2022-09-08T02:51:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:51.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:51.055120+0000) 2022-09-08T02:51:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:51 smithi055 ceph-mon[97033]: pgmap v10118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:51:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:51 smithi059 ceph-mon[97450]: pgmap v10118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:51:52.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:52.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:52.007518+0000) 2022-09-08T02:51:52.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:52.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:52.055327+0000) 2022-09-08T02:51:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:53.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:53.055545+0000) 2022-09-08T02:51:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:53.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:53.007748+0000) 2022-09-08T02:51:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:53 smithi055 ceph-mon[97033]: pgmap v10119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:53 smithi059 ceph-mon[97450]: pgmap v10119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:54.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:54.055708+0000) 2022-09-08T02:51:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:54.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:54.007982+0000) 2022-09-08T02:51:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:55.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:55.055931+0000) 2022-09-08T02:51:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:55.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:55.008159+0000) 2022-09-08T02:51:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:55 smithi055 ceph-mon[97033]: pgmap v10120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:55 smithi059 ceph-mon[97450]: pgmap v10120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:56.056 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:56.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:56.008390+0000) 2022-09-08T02:51:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:56.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:56.056053+0000) 2022-09-08T02:51:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:57.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:57.056327+0000) 2022-09-08T02:51:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:57.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:57.008587+0000) 2022-09-08T02:51:57.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:51:56.811+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:51:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:57 smithi055 ceph-mon[97033]: pgmap v10121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:51:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:57 smithi059 ceph-mon[97450]: pgmap v10121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:51:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:58.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:58.056568+0000) 2022-09-08T02:51:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:58.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:58.008803+0000) 2022-09-08T02:51:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:51:59.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:59.056704+0000) 2022-09-08T02:51:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:51:59.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:51:59.009058+0000) 2022-09-08T02:51:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:51:59 smithi055 ceph-mon[97033]: pgmap v10122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:51:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:51:59 smithi059 ceph-mon[97450]: pgmap v10122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:00.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:00.056924+0000) 2022-09-08T02:52:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:00.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:00.009266+0000) 2022-09-08T02:52:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:01.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:01.057074+0000) 2022-09-08T02:52:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:01.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:01.009380+0000) 2022-09-08T02:52:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:01 smithi055 ceph-mon[97033]: pgmap v10123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:52:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:01 smithi055 ceph-mon[97033]: pgmap v10124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:52:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:01 smithi055 ceph-mon[97033]: pgmap v10125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:52:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:52:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:01 smithi059 ceph-mon[97450]: pgmap v10123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:52:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:01 smithi059 ceph-mon[97450]: pgmap v10124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:52:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:01 smithi059 ceph-mon[97450]: pgmap v10125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:52:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:52:02.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:02.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:02.009614+0000) 2022-09-08T02:52:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:02.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:02.057267+0000) 2022-09-08T02:52:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:03.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:03.057471+0000) 2022-09-08T02:52:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:03.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:03.009838+0000) 2022-09-08T02:52:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:03 smithi055 ceph-mon[97033]: pgmap v10126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:52:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:03 smithi059 ceph-mon[97450]: pgmap v10126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:52:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:04.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:04.057721+0000) 2022-09-08T02:52:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:04.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:04.010075+0000) 2022-09-08T02:52:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:05.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:05.057916+0000) 2022-09-08T02:52:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:05.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:05.010255+0000) 2022-09-08T02:52:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:05 smithi055 ceph-mon[97033]: pgmap v10127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 26 op/s 2022-09-08T02:52:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:52:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:52:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:05 smithi059 ceph-mon[97450]: pgmap v10127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 26 op/s 2022-09-08T02:52:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:52:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:52:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:06.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:06.058143+0000) 2022-09-08T02:52:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:06.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:06.010486+0000) 2022-09-08T02:52:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:07.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:07.058395+0000) 2022-09-08T02:52:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:07.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:07.010740+0000) 2022-09-08T02:52:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:07 smithi055 ceph-mon[97033]: pgmap v10128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T02:52:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:07 smithi059 ceph-mon[97450]: pgmap v10128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T02:52:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:08.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:08.058613+0000) 2022-09-08T02:52:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:08.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:08.010948+0000) 2022-09-08T02:52:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:09.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:09.058788+0000) 2022-09-08T02:52:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:09.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:09.011179+0000) 2022-09-08T02:52:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:09 smithi055 ceph-mon[97033]: pgmap v10129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 65 KiB/s wr, 25 op/s 2022-09-08T02:52:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:09 smithi059 ceph-mon[97450]: pgmap v10129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 65 KiB/s wr, 25 op/s 2022-09-08T02:52:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:10.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:10.011361+0000) 2022-09-08T02:52:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:10.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:10.058945+0000) 2022-09-08T02:52:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:10 smithi055 ceph-mon[97033]: osdmap e713: 12 total, 11 up, 11 in 2022-09-08T02:52:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:10 smithi059 ceph-mon[97450]: osdmap e713: 12 total, 11 up, 11 in 2022-09-08T02:52:11.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:11.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:11.011596+0000) 2022-09-08T02:52:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:11.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:11.059134+0000) 2022-09-08T02:52:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:11 smithi055 ceph-mon[97033]: pgmap v10131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:52:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:11 smithi059 ceph-mon[97450]: pgmap v10131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:52:12.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:52:11.816+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:52:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:12.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:12.011812+0000) 2022-09-08T02:52:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:12.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:12.059353+0000) 2022-09-08T02:52:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:13.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:13.059540+0000) 2022-09-08T02:52:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:13.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:13.011997+0000) 2022-09-08T02:52:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:13 smithi055 ceph-mon[97033]: pgmap v10132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:52:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:13 smithi055 ceph-mon[97033]: osdmap e714: 12 total, 11 up, 11 in 2022-09-08T02:52:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:13 smithi059 ceph-mon[97450]: pgmap v10132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:52:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:13 smithi059 ceph-mon[97450]: osdmap e714: 12 total, 11 up, 11 in 2022-09-08T02:52:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:14.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:14.059770+0000) 2022-09-08T02:52:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:14.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:14.012248+0000) 2022-09-08T02:52:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:15.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:15.060020+0000) 2022-09-08T02:52:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:15.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:15.012473+0000) 2022-09-08T02:52:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:15 smithi055 ceph-mon[97033]: pgmap v10134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-08T02:52:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:15 smithi059 ceph-mon[97450]: pgmap v10134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-08T02:52:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:16.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:16.060211+0000) 2022-09-08T02:52:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:16.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:16.012670+0000) 2022-09-08T02:52:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:17.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:17.060444+0000) 2022-09-08T02:52:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:17.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:17.012898+0000) 2022-09-08T02:52:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:17 smithi055 ceph-mon[97033]: pgmap v10135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:52:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:17 smithi059 ceph-mon[97450]: pgmap v10135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:52:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:18.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:18.060618+0000) 2022-09-08T02:52:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:18.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:18.013140+0000) 2022-09-08T02:52:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:19.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:19.060849+0000) 2022-09-08T02:52:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:19.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:19.013367+0000) 2022-09-08T02:52:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:19 smithi055 ceph-mon[97033]: pgmap v10136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 211 B/s wr, 6 op/s 2022-09-08T02:52:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:19 smithi059 ceph-mon[97450]: pgmap v10136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 211 B/s wr, 6 op/s 2022-09-08T02:52:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:20.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:20.061110+0000) 2022-09-08T02:52:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:20.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:20.013611+0000) 2022-09-08T02:52:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:21.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:21.061377+0000) 2022-09-08T02:52:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:21.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:21.013860+0000) 2022-09-08T02:52:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:21 smithi055 ceph-mon[97033]: pgmap v10137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:52:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:21 smithi059 ceph-mon[97450]: pgmap v10137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:52:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:22.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:22.061615+0000) 2022-09-08T02:52:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:22.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:22.014078+0000) 2022-09-08T02:52:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:23.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:23.061869+0000) 2022-09-08T02:52:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:23.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:23.014315+0000) 2022-09-08T02:52:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:23 smithi055 ceph-mon[97033]: pgmap v10138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:52:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:23 smithi059 ceph-mon[97450]: pgmap v10138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:52:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:24.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:24.014555+0000) 2022-09-08T02:52:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:24.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:24.062102+0000) 2022-09-08T02:52:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:25.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:25.062309+0000) 2022-09-08T02:52:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:25.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:25.014741+0000) 2022-09-08T02:52:26.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:25 smithi055 ceph-mon[97033]: pgmap v10139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 87 B/s wr, 6 op/s 2022-09-08T02:52:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:25 smithi059 ceph-mon[97450]: pgmap v10139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 87 B/s wr, 6 op/s 2022-09-08T02:52:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:26.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:26.062528+0000) 2022-09-08T02:52:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:26.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:26.014986+0000) 2022-09-08T02:52:27.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:52:26.826+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:52:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:27.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:27.062784+0000) 2022-09-08T02:52:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:27.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:27.015254+0000) 2022-09-08T02:52:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:27 smithi055 ceph-mon[97033]: pgmap v10140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:52:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:27 smithi059 ceph-mon[97450]: pgmap v10140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:52:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:28.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:28.063019+0000) 2022-09-08T02:52:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:28.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:28.015507+0000) 2022-09-08T02:52:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:29.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:29.063227+0000) 2022-09-08T02:52:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:29.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:29.015763+0000) 2022-09-08T02:52:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:29 smithi055 ceph-mon[97033]: pgmap v10141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:29 smithi059 ceph-mon[97450]: pgmap v10141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:30.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:30.063433+0000) 2022-09-08T02:52:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:30.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:30.015941+0000) 2022-09-08T02:52:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:31.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:31.063621+0000) 2022-09-08T02:52:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:31.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:31.016154+0000) 2022-09-08T02:52:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:31 smithi055 ceph-mon[97033]: pgmap v10142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:52:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:31 smithi059 ceph-mon[97450]: pgmap v10142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:52:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:32.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:32.063888+0000) 2022-09-08T02:52:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:32.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:32.016342+0000) 2022-09-08T02:52:33.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:33.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:33.016675+0000) 2022-09-08T02:52:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:33.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:33.064093+0000) 2022-09-08T02:52:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:33 smithi055 ceph-mon[97033]: pgmap v10143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:33 smithi059 ceph-mon[97450]: pgmap v10143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:34.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:34.064312+0000) 2022-09-08T02:52:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:34.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:34.016830+0000) 2022-09-08T02:52:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:35.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:35.064520+0000) 2022-09-08T02:52:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:35.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:35.017085+0000) 2022-09-08T02:52:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:35 smithi055 ceph-mon[97033]: pgmap v10144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:52:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:35 smithi059 ceph-mon[97450]: pgmap v10144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:52:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:36.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:36.064748+0000) 2022-09-08T02:52:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:36.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:36.017271+0000) 2022-09-08T02:52:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:37.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:37.064950+0000) 2022-09-08T02:52:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:37.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:37.017495+0000) 2022-09-08T02:52:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:37 smithi055 ceph-mon[97033]: pgmap v10145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:52:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:37 smithi059 ceph-mon[97450]: pgmap v10145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:52:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:38.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:38.017733+0000) 2022-09-08T02:52:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:38.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:38.065207+0000) 2022-09-08T02:52:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:39.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:39.065439+0000) 2022-09-08T02:52:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:39.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:39.017974+0000) 2022-09-08T02:52:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:39 smithi055 ceph-mon[97033]: pgmap v10146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:39 smithi059 ceph-mon[97450]: pgmap v10146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:40.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:40.018166+0000) 2022-09-08T02:52:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:40.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:40.065701+0000) 2022-09-08T02:52:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:41.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:41.065929+0000) 2022-09-08T02:52:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:41.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:41.018423+0000) 2022-09-08T02:52:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:42.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:42.018734+0000) 2022-09-08T02:52:42.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:52:41.840+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:52:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:42.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:42.066113+0000) 2022-09-08T02:52:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:41 smithi055 ceph-mon[97033]: pgmap v10147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:52:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:41 smithi059 ceph-mon[97450]: pgmap v10147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:52:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:43.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:43.066327+0000) 2022-09-08T02:52:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:43.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:43.018970+0000) 2022-09-08T02:52:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:43 smithi055 ceph-mon[97033]: pgmap v10148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:43 smithi059 ceph-mon[97450]: pgmap v10148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:44.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:44.066553+0000) 2022-09-08T02:52:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:44.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:44.019242+0000) 2022-09-08T02:52:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:45.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:45.066800+0000) 2022-09-08T02:52:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:45.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:45.019491+0000) 2022-09-08T02:52:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:45 smithi055 ceph-mon[97033]: pgmap v10149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:52:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:45 smithi059 ceph-mon[97450]: pgmap v10149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:52:46.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:46.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:46.019696+0000) 2022-09-08T02:52:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:46.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:46.067036+0000) 2022-09-08T02:52:47.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:47.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:47.019845+0000) 2022-09-08T02:52:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:47.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:47.067283+0000) 2022-09-08T02:52:48.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:47 smithi055 ceph-mon[97033]: pgmap v10150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:47 smithi059 ceph-mon[97450]: pgmap v10150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:48.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:48.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:48.020040+0000) 2022-09-08T02:52:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:48.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:48.067497+0000) 2022-09-08T02:52:49.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:49.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:49.020252+0000) 2022-09-08T02:52:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:49.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:49.067756+0000) 2022-09-08T02:52:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:49 smithi055 ceph-mon[97033]: pgmap v10151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:49 smithi059 ceph-mon[97450]: pgmap v10151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:50.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:50.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:50.020478+0000) 2022-09-08T02:52:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:50.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:50.068019+0000) 2022-09-08T02:52:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:51.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:51.068242+0000) 2022-09-08T02:52:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:51.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:51.020761+0000) 2022-09-08T02:52:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:51 smithi055 ceph-mon[97033]: pgmap v10152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:52:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:51 smithi059 ceph-mon[97450]: pgmap v10152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:52:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:52.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:52.068467+0000) 2022-09-08T02:52:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:52.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:52.021027+0000) 2022-09-08T02:52:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:53.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:53.068730+0000) 2022-09-08T02:52:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:53.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:53.021298+0000) 2022-09-08T02:52:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:53 smithi055 ceph-mon[97033]: pgmap v10153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:53 smithi059 ceph-mon[97450]: pgmap v10153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:52:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:54.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:54.021566+0000) 2022-09-08T02:52:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:54.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:54.068975+0000) 2022-09-08T02:52:55.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:55.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:55.021833+0000) 2022-09-08T02:52:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:55.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:55.069142+0000) 2022-09-08T02:52:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:55 smithi055 ceph-mon[97033]: pgmap v10154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:52:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:55 smithi059 ceph-mon[97450]: pgmap v10154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:52:56.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:56.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:56.022046+0000) 2022-09-08T02:52:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:56.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:56.069333+0000) 2022-09-08T02:52:57.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:52:56.844+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:52:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:57.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:57.069604+0000) 2022-09-08T02:52:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:57.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:57.022319+0000) 2022-09-08T02:52:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:57 smithi055 ceph-mon[97033]: pgmap v10155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:52:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:57 smithi059 ceph-mon[97450]: pgmap v10155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:52:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:58.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:58.069851+0000) 2022-09-08T02:52:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:58.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:58.022602+0000) 2022-09-08T02:52:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:52:59.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:59.070090+0000) 2022-09-08T02:52:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:52:59.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:52:59.022874+0000) 2022-09-08T02:52:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:52:59 smithi055 ceph-mon[97033]: pgmap v10156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:52:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:52:59 smithi059 ceph-mon[97450]: pgmap v10156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:53:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:00.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:00.070361+0000) 2022-09-08T02:53:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:00.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:00.023093+0000) 2022-09-08T02:53:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:01.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:01.070617+0000) 2022-09-08T02:53:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:01.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:01.023262+0000) 2022-09-08T02:53:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:01 smithi055 ceph-mon[97033]: pgmap v10157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:53:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:53:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:53:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:53:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:01 smithi059 ceph-mon[97450]: pgmap v10157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:53:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:53:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:53:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:53:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:02.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:02.070954+0000) 2022-09-08T02:53:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:02.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:02.023480+0000) 2022-09-08T02:53:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:03.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:03.071224+0000) 2022-09-08T02:53:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:03.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:03.023734+0000) 2022-09-08T02:53:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:03 smithi055 ceph-mon[97033]: pgmap v10158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:53:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:03 smithi059 ceph-mon[97450]: pgmap v10158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:53:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:04.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:04.071460+0000) 2022-09-08T02:53:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:04.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:04.023920+0000) 2022-09-08T02:53:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:05.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:05.071649+0000) 2022-09-08T02:53:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:05.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:05.024149+0000) 2022-09-08T02:53:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:05 smithi055 ceph-mon[97033]: pgmap v10159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:53:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:53:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:53:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:05 smithi059 ceph-mon[97450]: pgmap v10159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:53:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:53:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:53:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:06.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:06.071833+0000) 2022-09-08T02:53:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:06.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:06.024334+0000) 2022-09-08T02:53:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:07.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:07.072044+0000) 2022-09-08T02:53:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:07.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:07.024548+0000) 2022-09-08T02:53:08.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:07 smithi055 ceph-mon[97033]: pgmap v10160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:53:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:07 smithi059 ceph-mon[97450]: pgmap v10160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:53:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:08.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:08.072258+0000) 2022-09-08T02:53:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:08.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:08.024790+0000) 2022-09-08T02:53:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:09.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:09.072456+0000) 2022-09-08T02:53:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:09.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:09.025064+0000) 2022-09-08T02:53:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:09 smithi055 ceph-mon[97033]: pgmap v10161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:53:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:09 smithi059 ceph-mon[97450]: pgmap v10161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:53:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:10.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:10.072696+0000) 2022-09-08T02:53:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:10.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:10.025259+0000) 2022-09-08T02:53:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:10 smithi055 ceph-mon[97033]: osdmap e715: 12 total, 11 up, 11 in 2022-09-08T02:53:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:10 smithi059 ceph-mon[97450]: osdmap e715: 12 total, 11 up, 11 in 2022-09-08T02:53:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:11.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:11.072817+0000) 2022-09-08T02:53:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:11.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:11.025488+0000) 2022-09-08T02:53:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:11 smithi055 ceph-mon[97033]: pgmap v10163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:53:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:11 smithi059 ceph-mon[97450]: pgmap v10163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:53:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:12.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:12.025686+0000) 2022-09-08T02:53:12.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:53:11.854+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:53:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:12.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:12.073031+0000) 2022-09-08T02:53:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:13.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:13.073258+0000) 2022-09-08T02:53:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:13.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:13.025810+0000) 2022-09-08T02:53:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:13 smithi055 ceph-mon[97033]: pgmap v10164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:53:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:13 smithi055 ceph-mon[97033]: osdmap e716: 12 total, 11 up, 11 in 2022-09-08T02:53:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:13 smithi059 ceph-mon[97450]: pgmap v10164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:53:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:13 smithi059 ceph-mon[97450]: osdmap e716: 12 total, 11 up, 11 in 2022-09-08T02:53:14.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:14.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:14.025997+0000) 2022-09-08T02:53:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:14.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:14.073482+0000) 2022-09-08T02:53:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:15.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:15.026178+0000) 2022-09-08T02:53:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:15.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:15.073698+0000) 2022-09-08T02:53:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:15 smithi055 ceph-mon[97033]: pgmap v10166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:53:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:15 smithi059 ceph-mon[97450]: pgmap v10166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:53:16.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:16.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:16.026391+0000) 2022-09-08T02:53:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:16.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:16.073812+0000) 2022-09-08T02:53:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:17.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:17.073999+0000) 2022-09-08T02:53:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:17.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:17.026609+0000) 2022-09-08T02:53:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:17 smithi055 ceph-mon[97033]: pgmap v10167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:53:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:17 smithi059 ceph-mon[97450]: pgmap v10167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:53:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:18.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:18.074210+0000) 2022-09-08T02:53:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:18.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:18.026792+0000) 2022-09-08T02:53:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:19.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:19.074434+0000) 2022-09-08T02:53:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:19.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:19.027017+0000) 2022-09-08T02:53:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:19 smithi055 ceph-mon[97033]: pgmap v10168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 211 B/s wr, 6 op/s 2022-09-08T02:53:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:19 smithi059 ceph-mon[97450]: pgmap v10168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 211 B/s wr, 6 op/s 2022-09-08T02:53:20.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:20.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:20.027230+0000) 2022-09-08T02:53:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:20.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:20.074672+0000) 2022-09-08T02:53:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:21.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:21.074798+0000) 2022-09-08T02:53:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:21.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:21.027455+0000) 2022-09-08T02:53:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:21 smithi055 ceph-mon[97033]: pgmap v10169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:53:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:21 smithi059 ceph-mon[97450]: pgmap v10169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:53:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:22.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:22.074934+0000) 2022-09-08T02:53:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:22.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:22.027710+0000) 2022-09-08T02:53:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:23.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:23.075148+0000) 2022-09-08T02:53:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:23.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:23.027920+0000) 2022-09-08T02:53:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:23 smithi055 ceph-mon[97033]: pgmap v10170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:53:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:23 smithi059 ceph-mon[97450]: pgmap v10170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:53:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:24.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:24.075317+0000) 2022-09-08T02:53:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:24.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:24.028057+0000) 2022-09-08T02:53:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:25.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:25.075509+0000) 2022-09-08T02:53:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:25.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:25.028280+0000) 2022-09-08T02:53:26.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:25 smithi055 ceph-mon[97033]: pgmap v10171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-08T02:53:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:25 smithi059 ceph-mon[97450]: pgmap v10171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-08T02:53:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:26.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:26.028498+0000) 2022-09-08T02:53:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:26.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:26.075769+0000) 2022-09-08T02:53:27.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:53:26.855+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:53:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:27.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:27.028755+0000) 2022-09-08T02:53:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:27.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:27.076013+0000) 2022-09-08T02:53:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:27 smithi055 ceph-mon[97033]: pgmap v10172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:53:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:27 smithi059 ceph-mon[97450]: pgmap v10172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:53:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:28.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:28.076215+0000) 2022-09-08T02:53:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:28.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:28.029013+0000) 2022-09-08T02:53:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:29.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:29.076358+0000) 2022-09-08T02:53:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:29.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:29.029187+0000) 2022-09-08T02:53:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:29 smithi055 ceph-mon[97033]: pgmap v10173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:53:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:29 smithi059 ceph-mon[97450]: pgmap v10173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:53:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:30.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:30.076607+0000) 2022-09-08T02:53:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:30.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:30.029395+0000) 2022-09-08T02:53:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:31.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:31.076795+0000) 2022-09-08T02:53:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:31.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:31.029628+0000) 2022-09-08T02:53:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:31 smithi055 ceph-mon[97033]: pgmap v10174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:53:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:53:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:31 smithi059 ceph-mon[97450]: pgmap v10174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:53:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:53:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:32.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:32.077084+0000) 2022-09-08T02:53:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:32.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:32.029848+0000) 2022-09-08T02:53:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:32 smithi055 ceph-mon[97033]: pgmap v10175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:53:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:32 smithi055 ceph-mon[97033]: pgmap v10176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:53:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:32 smithi059 ceph-mon[97450]: pgmap v10175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:53:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:32 smithi059 ceph-mon[97450]: pgmap v10176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:53:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:33.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:33.077392+0000) 2022-09-08T02:53:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:33.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:33.030110+0000) 2022-09-08T02:53:34.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:34.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:34.030283+0000) 2022-09-08T02:53:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:34.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:34.077618+0000) 2022-09-08T02:53:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:34 smithi055 ceph-mon[97033]: pgmap v10177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:53:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:34 smithi059 ceph-mon[97450]: pgmap v10177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:53:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:35.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:35.077887+0000) 2022-09-08T02:53:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:35.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:35.030515+0000) 2022-09-08T02:53:36.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:36.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:36.030750+0000) 2022-09-08T02:53:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:36.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:36.078028+0000) 2022-09-08T02:53:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:36 smithi055 ceph-mon[97033]: pgmap v10178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:53:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:36 smithi059 ceph-mon[97450]: pgmap v10178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:53:37.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:37.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:37.030973+0000) 2022-09-08T02:53:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:37.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:37.078244+0000) 2022-09-08T02:53:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:38.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:38.078468+0000) 2022-09-08T02:53:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:38.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:38.031126+0000) 2022-09-08T02:53:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:38 smithi055 ceph-mon[97033]: pgmap v10179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:53:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:38 smithi059 ceph-mon[97450]: pgmap v10179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:53:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:39.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:39.078690+0000) 2022-09-08T02:53:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:39.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:39.031323+0000) 2022-09-08T02:53:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:40.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:40.078912+0000) 2022-09-08T02:53:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:40.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:40.031553+0000) 2022-09-08T02:53:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:40 smithi055 ceph-mon[97033]: pgmap v10180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:53:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:40 smithi059 ceph-mon[97450]: pgmap v10180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:53:41.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:41.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:41.031807+0000) 2022-09-08T02:53:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:41.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:41.079168+0000) 2022-09-08T02:53:42.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:53:41.871+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:53:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:42.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:42.032023+0000) 2022-09-08T02:53:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:42.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:42.079410+0000) 2022-09-08T02:53:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:42 smithi055 ceph-mon[97033]: pgmap v10181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:53:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:42 smithi059 ceph-mon[97450]: pgmap v10181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:53:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:43.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:43.079662+0000) 2022-09-08T02:53:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:43.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:43.032249+0000) 2022-09-08T02:53:44.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:44.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:44.032523+0000) 2022-09-08T02:53:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:44.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:44.079916+0000) 2022-09-08T02:53:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:44 smithi055 ceph-mon[97033]: pgmap v10182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:53:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:44 smithi059 ceph-mon[97450]: pgmap v10182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:53:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:45.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:45.080137+0000) 2022-09-08T02:53:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:45.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:45.032752+0000) 2022-09-08T02:53:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:46.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:46.080353+0000) 2022-09-08T02:53:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:46.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:46.032919+0000) 2022-09-08T02:53:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:46 smithi055 ceph-mon[97033]: pgmap v10183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:53:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:46 smithi059 ceph-mon[97450]: pgmap v10183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:53:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:47.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:47.080597+0000) 2022-09-08T02:53:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:47.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:47.033144+0000) 2022-09-08T02:53:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:48.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:48.080801+0000) 2022-09-08T02:53:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:48.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:48.033392+0000) 2022-09-08T02:53:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:48 smithi055 ceph-mon[97033]: pgmap v10184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:53:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:48 smithi059 ceph-mon[97450]: pgmap v10184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:53:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:49.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:49.081023+0000) 2022-09-08T02:53:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:49.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:49.033661+0000) 2022-09-08T02:53:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:50.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:50.081232+0000) 2022-09-08T02:53:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:50.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:50.033829+0000) 2022-09-08T02:53:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:50 smithi055 ceph-mon[97033]: pgmap v10185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:53:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:50 smithi059 ceph-mon[97450]: pgmap v10185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:53:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:51.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:51.081383+0000) 2022-09-08T02:53:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:51.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:51.034004+0000) 2022-09-08T02:53:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:52.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:52.081603+0000) 2022-09-08T02:53:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:52.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:52.034235+0000) 2022-09-08T02:53:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:52 smithi055 ceph-mon[97033]: pgmap v10186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:53:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:52 smithi059 ceph-mon[97450]: pgmap v10186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:53:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:53.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:53.081814+0000) 2022-09-08T02:53:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:53.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:53.034483+0000) 2022-09-08T02:53:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:54.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:54.082021+0000) 2022-09-08T02:53:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:54.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:54.034734+0000) 2022-09-08T02:53:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:54 smithi055 ceph-mon[97033]: pgmap v10187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:53:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:54 smithi059 ceph-mon[97450]: pgmap v10187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:53:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:55.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:55.082255+0000) 2022-09-08T02:53:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:55.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:55.034948+0000) 2022-09-08T02:53:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:56.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:56.082492+0000) 2022-09-08T02:53:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:56.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:56.035172+0000) 2022-09-08T02:53:57.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:53:56.878+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:53:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:57.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:57.082735+0000) 2022-09-08T02:53:57.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:57.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:57.035423+0000) 2022-09-08T02:53:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:56 smithi055 ceph-mon[97033]: pgmap v10188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:53:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:56 smithi059 ceph-mon[97450]: pgmap v10188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:53:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:58.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:58.082922+0000) 2022-09-08T02:53:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:58.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:58.035715+0000) 2022-09-08T02:53:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:53:58 smithi055 ceph-mon[97033]: pgmap v10189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:53:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:53:58 smithi059 ceph-mon[97450]: pgmap v10189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:53:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:53:59.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:59.083094+0000) 2022-09-08T02:53:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:53:59.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:53:59.035938+0000) 2022-09-08T02:54:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:00.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:00.083240+0000) 2022-09-08T02:54:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:00.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:00.036165+0000) 2022-09-08T02:54:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:00 smithi055 ceph-mon[97033]: pgmap v10190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:00 smithi059 ceph-mon[97450]: pgmap v10190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:01.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:01.083451+0000) 2022-09-08T02:54:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:01.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:01.036347+0000) 2022-09-08T02:54:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:02.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:02.083688+0000) 2022-09-08T02:54:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:02.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:02.036570+0000) 2022-09-08T02:54:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:02 smithi059 ceph-mon[97450]: pgmap v10191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:54:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:03.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:03.083930+0000) 2022-09-08T02:54:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:03.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:03.036731+0000) 2022-09-08T02:54:03.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:02 smithi055 ceph-mon[97033]: pgmap v10191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:54:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:04.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:04.084110+0000) 2022-09-08T02:54:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:04.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:04.036947+0000) 2022-09-08T02:54:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:04 smithi059 ceph-mon[97450]: pgmap v10192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:05.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:05.084333+0000) 2022-09-08T02:54:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:05.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:05.037192+0000) 2022-09-08T02:54:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:04 smithi055 ceph-mon[97033]: pgmap v10192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:54:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:54:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:06.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:06.084565+0000) 2022-09-08T02:54:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:06.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:06.037441+0000) 2022-09-08T02:54:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:54:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:54:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:06 smithi059 ceph-mon[97450]: pgmap v10193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:54:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:07.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:07.084771+0000) 2022-09-08T02:54:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:07.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:07.037715+0000) 2022-09-08T02:54:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:06 smithi055 ceph-mon[97033]: pgmap v10193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:54:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:08.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:08.085047+0000) 2022-09-08T02:54:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:08.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:08.037969+0000) 2022-09-08T02:54:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:08 smithi059 ceph-mon[97450]: pgmap v10194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:54:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:09.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:09.085333+0000) 2022-09-08T02:54:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:09.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:09.038158+0000) 2022-09-08T02:54:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:08 smithi055 ceph-mon[97033]: pgmap v10194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:54:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:10.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:10.085569+0000) 2022-09-08T02:54:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:10.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:10.038346+0000) 2022-09-08T02:54:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:10 smithi055 ceph-mon[97033]: pgmap v10195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:54:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:10 smithi055 ceph-mon[97033]: osdmap e717: 12 total, 11 up, 11 in 2022-09-08T02:54:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:10 smithi059 ceph-mon[97450]: pgmap v10195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:54:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:10 smithi059 ceph-mon[97450]: osdmap e717: 12 total, 11 up, 11 in 2022-09-08T02:54:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:11.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:11.085783+0000) 2022-09-08T02:54:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:11.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:11.038547+0000) 2022-09-08T02:54:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:12.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:12.086006+0000) 2022-09-08T02:54:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:12.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:12.038811+0000) 2022-09-08T02:54:12.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:54:11.881+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:54:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:12 smithi055 ceph-mon[97033]: pgmap v10197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:54:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:12 smithi059 ceph-mon[97450]: pgmap v10197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:54:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:13.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:13.086165+0000) 2022-09-08T02:54:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:13.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:13.039022+0000) 2022-09-08T02:54:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:14.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:14.086340+0000) 2022-09-08T02:54:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:14.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:14.039202+0000) 2022-09-08T02:54:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:14 smithi055 ceph-mon[97033]: pgmap v10198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:54:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:14 smithi055 ceph-mon[97033]: osdmap e718: 12 total, 11 up, 11 in 2022-09-08T02:54:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:14 smithi059 ceph-mon[97450]: pgmap v10198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:54:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:14 smithi059 ceph-mon[97450]: osdmap e718: 12 total, 11 up, 11 in 2022-09-08T02:54:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:15.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:15.086518+0000) 2022-09-08T02:54:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:15.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:15.039327+0000) 2022-09-08T02:54:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:16.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:16.086681+0000) 2022-09-08T02:54:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:16.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:16.039557+0000) 2022-09-08T02:54:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:16 smithi055 ceph-mon[97033]: pgmap v10200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:54:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:16 smithi059 ceph-mon[97450]: pgmap v10200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:54:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:17.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:17.086890+0000) 2022-09-08T02:54:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:17.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:17.039795+0000) 2022-09-08T02:54:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:18.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:18.087011+0000) 2022-09-08T02:54:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:18.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:18.039992+0000) 2022-09-08T02:54:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:18 smithi055 ceph-mon[97033]: pgmap v10201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:54:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:18 smithi059 ceph-mon[97450]: pgmap v10201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:54:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:19.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:19.087235+0000) 2022-09-08T02:54:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:19.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:19.040185+0000) 2022-09-08T02:54:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:20.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:20.087456+0000) 2022-09-08T02:54:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:20.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:20.040401+0000) 2022-09-08T02:54:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:20 smithi055 ceph-mon[97033]: pgmap v10202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:54:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:20 smithi059 ceph-mon[97450]: pgmap v10202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:54:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:21.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:21.087669+0000) 2022-09-08T02:54:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:21.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:21.040605+0000) 2022-09-08T02:54:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:22.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:22.087912+0000) 2022-09-08T02:54:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:22.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:22.040822+0000) 2022-09-08T02:54:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:22 smithi055 ceph-mon[97033]: pgmap v10203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:54:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:22 smithi059 ceph-mon[97450]: pgmap v10203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:54:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:23.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:23.088103+0000) 2022-09-08T02:54:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:23.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:23.041041+0000) 2022-09-08T02:54:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:24.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:24.088337+0000) 2022-09-08T02:54:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:24.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:24.041250+0000) 2022-09-08T02:54:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:24 smithi055 ceph-mon[97033]: pgmap v10204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:54:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:24 smithi059 ceph-mon[97450]: pgmap v10204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:54:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:25.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:25.088610+0000) 2022-09-08T02:54:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:25.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:25.041403+0000) 2022-09-08T02:54:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:26.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:26.088853+0000) 2022-09-08T02:54:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:26.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:26.041652+0000) 2022-09-08T02:54:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:26 smithi055 ceph-mon[97033]: pgmap v10205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:54:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:26 smithi059 ceph-mon[97450]: pgmap v10205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:54:27.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:54:26.887+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:54:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:27.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:27.089067+0000) 2022-09-08T02:54:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:27.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:27.041808+0000) 2022-09-08T02:54:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:28.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:28.089302+0000) 2022-09-08T02:54:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:28.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:28.042005+0000) 2022-09-08T02:54:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:28 smithi055 ceph-mon[97033]: pgmap v10206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:28 smithi059 ceph-mon[97450]: pgmap v10206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:29.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:29.089477+0000) 2022-09-08T02:54:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:29.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:29.042242+0000) 2022-09-08T02:54:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:30.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:30.089726+0000) 2022-09-08T02:54:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:30.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:30.042455+0000) 2022-09-08T02:54:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:30 smithi055 ceph-mon[97033]: pgmap v10207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:30 smithi059 ceph-mon[97450]: pgmap v10207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:31.321 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:31.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:31.042658+0000) 2022-09-08T02:54:31.322 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:31.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:31.089894+0000) 2022-09-08T02:54:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:32.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:32.090131+0000) 2022-09-08T02:54:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:32.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:32.042813+0000) 2022-09-08T02:54:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:32 smithi055 ceph-mon[97033]: pgmap v10208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:54:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:54:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:54:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:54:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:32 smithi059 ceph-mon[97450]: pgmap v10208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:54:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:54:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:54:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:54:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:33.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:33.090387+0000) 2022-09-08T02:54:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:33.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:33.043008+0000) 2022-09-08T02:54:34.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:34.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:34.043254+0000) 2022-09-08T02:54:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:34.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:34.090609+0000) 2022-09-08T02:54:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:34 smithi055 ceph-mon[97033]: pgmap v10209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:34 smithi059 ceph-mon[97450]: pgmap v10209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:35.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:35.090924+0000) 2022-09-08T02:54:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:35.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:35.043517+0000) 2022-09-08T02:54:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:36.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:36.091218+0000) 2022-09-08T02:54:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:36.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:36.043768+0000) 2022-09-08T02:54:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:36 smithi055 ceph-mon[97033]: pgmap v10210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:54:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:36 smithi059 ceph-mon[97450]: pgmap v10210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:54:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:37.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:37.091481+0000) 2022-09-08T02:54:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:37.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:37.043996+0000) 2022-09-08T02:54:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:38.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:38.091740+0000) 2022-09-08T02:54:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:38.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:38.044254+0000) 2022-09-08T02:54:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:38 smithi055 ceph-mon[97033]: pgmap v10211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:38 smithi059 ceph-mon[97450]: pgmap v10211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:39.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:39.091947+0000) 2022-09-08T02:54:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:39.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:39.044471+0000) 2022-09-08T02:54:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:40.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:40.092195+0000) 2022-09-08T02:54:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:40.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:40.044750+0000) 2022-09-08T02:54:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:40 smithi059 ceph-mon[97450]: pgmap v10212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:40 smithi055 ceph-mon[97033]: pgmap v10212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:41.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:41.092449+0000) 2022-09-08T02:54:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:41.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:41.044981+0000) 2022-09-08T02:54:42.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:54:41.900+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:54:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:42.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:42.092725+0000) 2022-09-08T02:54:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:42.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:42.045211+0000) 2022-09-08T02:54:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:42 smithi059 ceph-mon[97450]: pgmap v10213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:54:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:42 smithi055 ceph-mon[97033]: pgmap v10213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:54:43.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:43.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:43.092927+0000) 2022-09-08T02:54:43.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:43.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:43.045454+0000) 2022-09-08T02:54:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:44.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:44.093155+0000) 2022-09-08T02:54:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:44.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:44.045728+0000) 2022-09-08T02:54:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:44 smithi059 ceph-mon[97450]: pgmap v10214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:44 smithi055 ceph-mon[97033]: pgmap v10214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:45.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:45.093382+0000) 2022-09-08T02:54:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:45.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:45.045927+0000) 2022-09-08T02:54:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:46.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:46.093569+0000) 2022-09-08T02:54:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:46.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:46.046106+0000) 2022-09-08T02:54:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:46 smithi059 ceph-mon[97450]: pgmap v10215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:54:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:46 smithi055 ceph-mon[97033]: pgmap v10215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:54:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:47.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:47.093823+0000) 2022-09-08T02:54:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:47.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:47.046360+0000) 2022-09-08T02:54:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:48.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:48.094037+0000) 2022-09-08T02:54:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:48.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:48.046633+0000) 2022-09-08T02:54:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:48 smithi059 ceph-mon[97450]: pgmap v10216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:48 smithi055 ceph-mon[97033]: pgmap v10216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:49.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:49.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:49.046786+0000) 2022-09-08T02:54:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:49.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:49.094158+0000) 2022-09-08T02:54:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:50.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:50.046947+0000) 2022-09-08T02:54:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:50.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:50.094358+0000) 2022-09-08T02:54:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:50 smithi059 ceph-mon[97450]: pgmap v10217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:50 smithi055 ceph-mon[97033]: pgmap v10217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:51.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:51.047184+0000) 2022-09-08T02:54:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:51.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:51.094548+0000) 2022-09-08T02:54:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:52.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:52.094786+0000) 2022-09-08T02:54:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:52.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:52.047338+0000) 2022-09-08T02:54:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:52 smithi059 ceph-mon[97450]: pgmap v10218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:54:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:52 smithi055 ceph-mon[97033]: pgmap v10218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:54:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:53.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:53.095043+0000) 2022-09-08T02:54:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:53.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:53.047553+0000) 2022-09-08T02:54:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:54.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:54.095257+0000) 2022-09-08T02:54:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:54.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:54.047739+0000) 2022-09-08T02:54:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:54 smithi059 ceph-mon[97450]: pgmap v10219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:54 smithi055 ceph-mon[97033]: pgmap v10219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:54:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:55.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:55.095494+0000) 2022-09-08T02:54:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:55.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:55.047880+0000) 2022-09-08T02:54:56.350 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:56.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:56.095693+0000) 2022-09-08T02:54:56.351 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:56.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:56.048066+0000) 2022-09-08T02:54:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:56 smithi059 ceph-mon[97450]: pgmap v10220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:54:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:56 smithi055 ceph-mon[97033]: pgmap v10220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:54:57.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:54:56.914+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:54:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:57.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:57.095918+0000) 2022-09-08T02:54:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:57.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:57.048224+0000) 2022-09-08T02:54:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:58.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:58.096094+0000) 2022-09-08T02:54:58.417 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:58.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:58.048415+0000) 2022-09-08T02:54:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:54:58 smithi059 ceph-mon[97450]: pgmap v10221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:54:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:54:58 smithi055 ceph-mon[97033]: pgmap v10221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:54:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:54:59.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:59.096268+0000) 2022-09-08T02:54:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:54:59.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:54:59.048671+0000) 2022-09-08T02:55:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:00.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:00.096427+0000) 2022-09-08T02:55:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:00.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:00.048908+0000) 2022-09-08T02:55:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:00 smithi059 ceph-mon[97450]: pgmap v10222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:55:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:00 smithi055 ceph-mon[97033]: pgmap v10222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T02:55:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:01.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:01.096551+0000) 2022-09-08T02:55:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:01.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:01.049058+0000) 2022-09-08T02:55:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:02.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:02.096734+0000) 2022-09-08T02:55:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:02.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:02.049287+0000) 2022-09-08T02:55:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:02 smithi055 ceph-mon[97033]: pgmap v10223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:55:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:02 smithi055 ceph-mon[97033]: pgmap v10224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T02:55:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:02 smithi055 ceph-mon[97033]: pgmap v10225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-09-08T02:55:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:55:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:02 smithi059 ceph-mon[97450]: pgmap v10223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:55:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:02 smithi059 ceph-mon[97450]: pgmap v10224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T02:55:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:02 smithi059 ceph-mon[97450]: pgmap v10225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-09-08T02:55:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:55:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:03.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:03.096972+0000) 2022-09-08T02:55:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:03.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:03.049452+0000) 2022-09-08T02:55:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:04.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:04.097179+0000) 2022-09-08T02:55:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:04.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:04.049699+0000) 2022-09-08T02:55:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:04 smithi055 ceph-mon[97033]: pgmap v10226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:55:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:04 smithi059 ceph-mon[97450]: pgmap v10226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:55:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:05.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:05.097298+0000) 2022-09-08T02:55:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:05.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:05.049849+0000) 2022-09-08T02:55:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:55:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:55:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:06.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:06.097428+0000) 2022-09-08T02:55:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:06.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:06.050054+0000) 2022-09-08T02:55:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:06 smithi055 ceph-mon[97033]: pgmap v10227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T02:55:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:55:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:06 smithi059 ceph-mon[97450]: pgmap v10227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T02:55:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:55:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:07.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:07.097668+0000) 2022-09-08T02:55:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:07.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:07.050249+0000) 2022-09-08T02:55:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:08.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:08.097812+0000) 2022-09-08T02:55:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:08.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:08.050453+0000) 2022-09-08T02:55:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:08 smithi055 ceph-mon[97033]: pgmap v10228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T02:55:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:08 smithi059 ceph-mon[97450]: pgmap v10228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T02:55:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:09.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:09.098030+0000) 2022-09-08T02:55:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:09.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:09.050687+0000) 2022-09-08T02:55:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:10.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:10.098249+0000) 2022-09-08T02:55:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:10.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:10.050832+0000) 2022-09-08T02:55:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:10 smithi055 ceph-mon[97033]: pgmap v10229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T02:55:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:10 smithi055 ceph-mon[97033]: osdmap e719: 12 total, 11 up, 11 in 2022-09-08T02:55:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:10 smithi059 ceph-mon[97450]: pgmap v10229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T02:55:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:10 smithi059 ceph-mon[97450]: osdmap e719: 12 total, 11 up, 11 in 2022-09-08T02:55:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:11.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:11.098458+0000) 2022-09-08T02:55:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:11.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:11.051068+0000) 2022-09-08T02:55:12.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:55:11.915+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:55:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:12.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:12.098647+0000) 2022-09-08T02:55:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:12.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:12.051260+0000) 2022-09-08T02:55:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:12 smithi055 ceph-mon[97033]: pgmap v10231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:55:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:12 smithi059 ceph-mon[97450]: pgmap v10231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:55:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:13.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:13.098879+0000) 2022-09-08T02:55:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:13.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:13.051493+0000) 2022-09-08T02:55:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:14.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:14.099134+0000) 2022-09-08T02:55:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:14.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:14.051763+0000) 2022-09-08T02:55:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:14 smithi055 ceph-mon[97033]: osdmap e720: 12 total, 11 up, 11 in 2022-09-08T02:55:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:14 smithi055 ceph-mon[97033]: pgmap v10233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:55:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:14 smithi059 ceph-mon[97450]: osdmap e720: 12 total, 11 up, 11 in 2022-09-08T02:55:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:14 smithi059 ceph-mon[97450]: pgmap v10233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:55:15.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:15.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:15.051970+0000) 2022-09-08T02:55:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:15.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:15.099434+0000) 2022-09-08T02:55:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:16.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:16.099736+0000) 2022-09-08T02:55:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:16.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:16.052192+0000) 2022-09-08T02:55:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:16 smithi055 ceph-mon[97033]: pgmap v10234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:55:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:16 smithi059 ceph-mon[97450]: pgmap v10234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:55:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:17.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:17.099989+0000) 2022-09-08T02:55:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:17.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:17.052410+0000) 2022-09-08T02:55:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:18.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:18.100223+0000) 2022-09-08T02:55:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:18.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:18.052663+0000) 2022-09-08T02:55:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:18 smithi055 ceph-mon[97033]: pgmap v10235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:55:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:18 smithi059 ceph-mon[97450]: pgmap v10235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:55:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:19.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:19.100507+0000) 2022-09-08T02:55:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:19.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:19.052835+0000) 2022-09-08T02:55:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:20.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:20.100766+0000) 2022-09-08T02:55:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:20.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:20.053031+0000) 2022-09-08T02:55:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:20 smithi055 ceph-mon[97033]: pgmap v10236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:55:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:20 smithi059 ceph-mon[97450]: pgmap v10236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:55:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:21.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:21.100932+0000) 2022-09-08T02:55:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:21.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:21.053176+0000) 2022-09-08T02:55:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:22.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:22.053365+0000) 2022-09-08T02:55:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:22.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:22.101167+0000) 2022-09-08T02:55:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:22 smithi055 ceph-mon[97033]: pgmap v10237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:55:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:22 smithi059 ceph-mon[97450]: pgmap v10237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:55:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:23.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:23.053574+0000) 2022-09-08T02:55:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:23.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:23.101435+0000) 2022-09-08T02:55:24.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:24.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:24.053804+0000) 2022-09-08T02:55:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:24.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:24.101712+0000) 2022-09-08T02:55:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:24 smithi055 ceph-mon[97033]: pgmap v10238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T02:55:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:24 smithi059 ceph-mon[97450]: pgmap v10238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T02:55:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:25.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:25.101919+0000) 2022-09-08T02:55:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:25.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:25.054008+0000) 2022-09-08T02:55:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:26.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:26.102139+0000) 2022-09-08T02:55:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:26.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:26.054243+0000) 2022-09-08T02:55:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:26 smithi055 ceph-mon[97033]: pgmap v10239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:55:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:26 smithi059 ceph-mon[97450]: pgmap v10239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:55:27.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:55:26.927+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:55:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:27.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:27.054493+0000) 2022-09-08T02:55:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:27.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:27.102350+0000) 2022-09-08T02:55:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:28.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:28.102539+0000) 2022-09-08T02:55:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:28.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:28.054716+0000) 2022-09-08T02:55:28.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:28 smithi055 ceph-mon[97033]: pgmap v10240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:28 smithi059 ceph-mon[97450]: pgmap v10240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:29.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:29.102738+0000) 2022-09-08T02:55:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:29.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:29.054906+0000) 2022-09-08T02:55:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:30.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:30.102909+0000) 2022-09-08T02:55:30.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:30.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:30.055115+0000) 2022-09-08T02:55:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:30 smithi055 ceph-mon[97033]: pgmap v10241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:30 smithi059 ceph-mon[97450]: pgmap v10241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:31.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:31.103046+0000) 2022-09-08T02:55:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:31.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:31.055344+0000) 2022-09-08T02:55:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:32.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:32.103274+0000) 2022-09-08T02:55:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:32.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:32.055576+0000) 2022-09-08T02:55:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:32 smithi059 ceph-mon[97450]: pgmap v10242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:55:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:32 smithi055 ceph-mon[97033]: pgmap v10242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:55:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:33.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:33.103497+0000) 2022-09-08T02:55:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:33.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:33.055764+0000) 2022-09-08T02:55:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:34.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:34.103719+0000) 2022-09-08T02:55:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:34.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:34.055944+0000) 2022-09-08T02:55:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:34 smithi055 ceph-mon[97033]: pgmap v10243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:34 smithi059 ceph-mon[97450]: pgmap v10243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:35.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:35.103940+0000) 2022-09-08T02:55:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:35.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:35.056198+0000) 2022-09-08T02:55:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:36.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:36.104170+0000) 2022-09-08T02:55:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:36.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:36.056432+0000) 2022-09-08T02:55:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:36 smithi055 ceph-mon[97033]: pgmap v10244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:55:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:36 smithi059 ceph-mon[97450]: pgmap v10244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:55:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:37.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:37.104401+0000) 2022-09-08T02:55:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:37.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:37.056675+0000) 2022-09-08T02:55:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:38.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:38.104677+0000) 2022-09-08T02:55:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:38.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:38.056922+0000) 2022-09-08T02:55:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:38 smithi055 ceph-mon[97033]: pgmap v10245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:38 smithi059 ceph-mon[97450]: pgmap v10245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:39.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:39.104872+0000) 2022-09-08T02:55:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:39.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:39.057103+0000) 2022-09-08T02:55:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:40.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:40.105101+0000) 2022-09-08T02:55:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:40.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:40.057315+0000) 2022-09-08T02:55:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:40 smithi059 ceph-mon[97450]: pgmap v10246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:40 smithi055 ceph-mon[97033]: pgmap v10246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:41.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:41.105348+0000) 2022-09-08T02:55:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:41.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:41.057518+0000) 2022-09-08T02:55:42.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:55:41.941+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:55:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:42.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:42.105688+0000) 2022-09-08T02:55:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:42.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:42.057700+0000) 2022-09-08T02:55:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:42 smithi059 ceph-mon[97450]: pgmap v10247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:55:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:42 smithi055 ceph-mon[97033]: pgmap v10247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:55:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:43.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:43.105890+0000) 2022-09-08T02:55:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:43.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:43.057914+0000) 2022-09-08T02:55:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:44.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:44.058102+0000) 2022-09-08T02:55:44.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:44.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:44.106116+0000) 2022-09-08T02:55:44.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:44 smithi055 ceph-mon[97033]: pgmap v10248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:44 smithi059 ceph-mon[97450]: pgmap v10248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:45.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:45.058256+0000) 2022-09-08T02:55:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:45.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:45.106336+0000) 2022-09-08T02:55:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:46.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:46.106595+0000) 2022-09-08T02:55:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:46.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:46.058482+0000) 2022-09-08T02:55:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:46 smithi055 ceph-mon[97033]: pgmap v10249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:55:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:46 smithi059 ceph-mon[97450]: pgmap v10249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:55:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:47.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:47.106804+0000) 2022-09-08T02:55:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:47.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:47.058702+0000) 2022-09-08T02:55:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:48.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:48.107016+0000) 2022-09-08T02:55:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:48.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:48.058840+0000) 2022-09-08T02:55:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:48 smithi055 ceph-mon[97033]: pgmap v10250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:48 smithi059 ceph-mon[97450]: pgmap v10250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:49.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:49.059053+0000) 2022-09-08T02:55:49.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:49.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:49.107237+0000) 2022-09-08T02:55:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:50.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:50.107477+0000) 2022-09-08T02:55:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:50.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:50.059288+0000) 2022-09-08T02:55:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:50 smithi059 ceph-mon[97450]: pgmap v10251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:50.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:50 smithi055 ceph-mon[97033]: pgmap v10251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:51.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:51.107732+0000) 2022-09-08T02:55:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:51.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:51.059535+0000) 2022-09-08T02:55:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:52.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:52.059803+0000) 2022-09-08T02:55:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:52.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:52.107935+0000) 2022-09-08T02:55:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:52 smithi059 ceph-mon[97450]: pgmap v10252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:55:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:52 smithi055 ceph-mon[97033]: pgmap v10252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:55:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:53.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:53.108180+0000) 2022-09-08T02:55:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:53.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:53.060035+0000) 2022-09-08T02:55:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:54.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:54.108410+0000) 2022-09-08T02:55:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:54.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:54.060293+0000) 2022-09-08T02:55:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:54 smithi055 ceph-mon[97033]: pgmap v10253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:54 smithi059 ceph-mon[97450]: pgmap v10253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:55.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:55.108622+0000) 2022-09-08T02:55:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:55.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:55.060531+0000) 2022-09-08T02:55:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:56.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:56.108810+0000) 2022-09-08T02:55:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:56.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:56.060773+0000) 2022-09-08T02:55:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:56 smithi055 ceph-mon[97033]: pgmap v10254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:55:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:56 smithi059 ceph-mon[97450]: pgmap v10254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:55:57.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:55:56.954+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:55:57.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:57.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:57.109065+0000) 2022-09-08T02:55:57.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:57.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:57.060947+0000) 2022-09-08T02:55:58.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:58.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:58.109274+0000) 2022-09-08T02:55:58.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:58.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:58.061182+0000) 2022-09-08T02:55:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:55:58 smithi059 ceph-mon[97450]: pgmap v10255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:55:58 smithi055 ceph-mon[97033]: pgmap v10255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:55:59.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:55:59.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:59.109495+0000) 2022-09-08T02:55:59.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:55:59.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:55:59.061351+0000) 2022-09-08T02:56:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:00.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:00.109667+0000) 2022-09-08T02:56:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:00.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:00.061556+0000) 2022-09-08T02:56:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:00 smithi059 ceph-mon[97450]: pgmap v10256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:56:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:00 smithi055 ceph-mon[97033]: pgmap v10256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:56:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:01.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:01.109878+0000) 2022-09-08T02:56:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:01.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:01.061787+0000) 2022-09-08T02:56:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:02.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:02.062057+0000) 2022-09-08T02:56:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:02.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:02.110097+0000) 2022-09-08T02:56:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:02 smithi059 ceph-mon[97450]: pgmap v10257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:56:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:56:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:56:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:56:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:02 smithi055 ceph-mon[97033]: pgmap v10257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:56:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:56:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:56:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:56:03.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:03.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:03.062302+0000) 2022-09-08T02:56:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:03.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:03.110297+0000) 2022-09-08T02:56:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:04.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:04.110523+0000) 2022-09-08T02:56:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:04.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:04.062412+0000) 2022-09-08T02:56:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:04 smithi059 ceph-mon[97450]: pgmap v10258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:56:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:04 smithi055 ceph-mon[97033]: pgmap v10258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:56:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:05.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:05.110777+0000) 2022-09-08T02:56:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:05.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:05.062632+0000) 2022-09-08T02:56:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:56:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:56:06.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:06.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:06.062885+0000) 2022-09-08T02:56:06.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:06.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:06.110918+0000) 2022-09-08T02:56:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:06 smithi055 ceph-mon[97033]: pgmap v10259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:56:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:56:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:06 smithi059 ceph-mon[97450]: pgmap v10259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:56:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:56:07.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:07.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:07.063042+0000) 2022-09-08T02:56:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:07.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:07.111100+0000) 2022-09-08T02:56:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:08.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:08.111317+0000) 2022-09-08T02:56:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:08.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:08.063231+0000) 2022-09-08T02:56:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:08 smithi059 ceph-mon[97450]: pgmap v10260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:56:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:08 smithi055 ceph-mon[97033]: pgmap v10260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:56:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:09.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:09.111514+0000) 2022-09-08T02:56:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:09.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:09.063437+0000) 2022-09-08T02:56:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:10.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:10.111726+0000) 2022-09-08T02:56:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:10.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:10.063603+0000) 2022-09-08T02:56:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:10 smithi059 ceph-mon[97450]: pgmap v10261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:56:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:10 smithi059 ceph-mon[97450]: osdmap e721: 12 total, 11 up, 11 in 2022-09-08T02:56:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:10 smithi055 ceph-mon[97033]: pgmap v10261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:56:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:10 smithi055 ceph-mon[97033]: osdmap e721: 12 total, 11 up, 11 in 2022-09-08T02:56:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:11.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:11.111951+0000) 2022-09-08T02:56:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:11.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:11.063754+0000) 2022-09-08T02:56:12.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:56:11.970+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:56:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:12.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:12.112183+0000) 2022-09-08T02:56:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:12.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:12.063988+0000) 2022-09-08T02:56:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:12 smithi059 ceph-mon[97450]: pgmap v10263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:56:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:12 smithi055 ceph-mon[97033]: pgmap v10263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:56:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:13.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:13.112399+0000) 2022-09-08T02:56:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:13.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:13.064208+0000) 2022-09-08T02:56:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:14.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:14.112614+0000) 2022-09-08T02:56:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:14.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:14.064382+0000) 2022-09-08T02:56:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:14 smithi055 ceph-mon[97033]: pgmap v10264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:56:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:14 smithi059 ceph-mon[97450]: pgmap v10264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T02:56:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:15.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:15.112869+0000) 2022-09-08T02:56:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:15.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:15.064583+0000) 2022-09-08T02:56:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:16.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:16.113127+0000) 2022-09-08T02:56:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:16.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:16.064834+0000) 2022-09-08T02:56:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:16 smithi059 ceph-mon[97450]: pgmap v10265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:56:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:16 smithi055 ceph-mon[97033]: pgmap v10265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:56:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:17.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:17.113389+0000) 2022-09-08T02:56:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:17.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:17.065012+0000) 2022-09-08T02:56:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:18.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:18.113686+0000) 2022-09-08T02:56:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:18.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:18.065243+0000) 2022-09-08T02:56:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:18 smithi059 ceph-mon[97450]: pgmap v10266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:56:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:18 smithi059 ceph-mon[97450]: osdmap e722: 12 total, 11 up, 11 in 2022-09-08T02:56:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:18 smithi055 ceph-mon[97033]: pgmap v10266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:56:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:18 smithi055 ceph-mon[97033]: osdmap e722: 12 total, 11 up, 11 in 2022-09-08T02:56:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:19.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:19.113941+0000) 2022-09-08T02:56:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:19.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:19.065408+0000) 2022-09-08T02:56:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:20.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:20.114216+0000) 2022-09-08T02:56:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:20.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:20.065621+0000) 2022-09-08T02:56:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:20 smithi059 ceph-mon[97450]: pgmap v10268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:56:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:20 smithi055 ceph-mon[97033]: pgmap v10268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T02:56:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:21.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:21.114479+0000) 2022-09-08T02:56:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:21.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:21.065875+0000) 2022-09-08T02:56:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:22.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:22.114775+0000) 2022-09-08T02:56:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:22.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:22.066090+0000) 2022-09-08T02:56:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:22 smithi059 ceph-mon[97450]: pgmap v10269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:56:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:22 smithi055 ceph-mon[97033]: pgmap v10269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:56:23.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:23.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:23.066279+0000) 2022-09-08T02:56:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:23.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:23.114990+0000) 2022-09-08T02:56:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:24.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:24.115170+0000) 2022-09-08T02:56:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:24.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:24.066462+0000) 2022-09-08T02:56:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:24 smithi059 ceph-mon[97450]: pgmap v10270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:56:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:24 smithi055 ceph-mon[97033]: pgmap v10270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:56:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:25.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:25.115326+0000) 2022-09-08T02:56:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:25.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:25.066631+0000) 2022-09-08T02:56:26.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:26.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:26.115620+0000) 2022-09-08T02:56:26.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:26.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:26.066822+0000) 2022-09-08T02:56:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:26 smithi059 ceph-mon[97450]: pgmap v10271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:56:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:26 smithi055 ceph-mon[97033]: pgmap v10271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:56:27.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:56:26.985+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:56:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:27.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:27.115846+0000) 2022-09-08T02:56:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:27.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:27.067036+0000) 2022-09-08T02:56:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:28.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:28.116096+0000) 2022-09-08T02:56:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:28.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:28.067268+0000) 2022-09-08T02:56:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:28 smithi059 ceph-mon[97450]: pgmap v10272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:56:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:28 smithi055 ceph-mon[97033]: pgmap v10272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:56:29.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:29.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:29.116246+0000) 2022-09-08T02:56:29.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:29.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:29.067393+0000) 2022-09-08T02:56:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:30.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:30.116436+0000) 2022-09-08T02:56:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:30.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:30.067567+0000) 2022-09-08T02:56:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:30 smithi059 ceph-mon[97450]: pgmap v10273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:56:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:30 smithi055 ceph-mon[97033]: pgmap v10273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T02:56:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:31.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:31.116625+0000) 2022-09-08T02:56:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:31.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:31.067771+0000) 2022-09-08T02:56:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:32.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:32.116790+0000) 2022-09-08T02:56:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:32.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:32.067967+0000) 2022-09-08T02:56:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:32 smithi055 ceph-mon[97033]: pgmap v10274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:56:32.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:32 smithi055 ceph-mon[97033]: pgmap v10275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:56:32.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:32 smithi055 ceph-mon[97033]: pgmap v10276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:56:32.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:56:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:32 smithi059 ceph-mon[97450]: pgmap v10274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:56:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:32 smithi059 ceph-mon[97450]: pgmap v10275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T02:56:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:32 smithi059 ceph-mon[97450]: pgmap v10276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:56:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:56:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:33.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:33.116985+0000) 2022-09-08T02:56:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:33.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:33.068188+0000) 2022-09-08T02:56:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:34.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:34.117166+0000) 2022-09-08T02:56:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:34.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:34.068365+0000) 2022-09-08T02:56:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:34 smithi059 ceph-mon[97450]: pgmap v10277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:56:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:34 smithi055 ceph-mon[97033]: pgmap v10277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:56:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:35.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:35.117369+0000) 2022-09-08T02:56:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:35.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:35.068540+0000) 2022-09-08T02:56:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:36.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:36.117589+0000) 2022-09-08T02:56:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:36.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:36.068757+0000) 2022-09-08T02:56:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:36 smithi059 ceph-mon[97450]: pgmap v10278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:56:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:36 smithi055 ceph-mon[97033]: pgmap v10278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:56:37.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:37.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:37.068952+0000) 2022-09-08T02:56:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:37.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:37.117764+0000) 2022-09-08T02:56:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:38.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:38.117994+0000) 2022-09-08T02:56:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:38.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:38.069201+0000) 2022-09-08T02:56:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:38 smithi059 ceph-mon[97450]: pgmap v10279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:56:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:38 smithi055 ceph-mon[97033]: pgmap v10279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T02:56:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:39.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:39.118227+0000) 2022-09-08T02:56:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:39.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:39.069418+0000) 2022-09-08T02:56:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:40.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:40.118493+0000) 2022-09-08T02:56:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:40.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:40.069631+0000) 2022-09-08T02:56:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:40 smithi059 ceph-mon[97450]: pgmap v10280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:56:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:40 smithi055 ceph-mon[97033]: pgmap v10280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T02:56:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:41.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:41.118712+0000) 2022-09-08T02:56:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:41.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:41.069892+0000) 2022-09-08T02:56:42.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:56:42.001+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:56:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:42.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:42.118915+0000) 2022-09-08T02:56:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:42.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:42.070114+0000) 2022-09-08T02:56:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:42 smithi059 ceph-mon[97450]: pgmap v10281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:56:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:42 smithi055 ceph-mon[97033]: pgmap v10281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T02:56:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:43.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:43.119170+0000) 2022-09-08T02:56:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:43.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:43.070351+0000) 2022-09-08T02:56:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:44.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:44.119418+0000) 2022-09-08T02:56:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:44.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:44.070475+0000) 2022-09-08T02:56:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:44 smithi059 ceph-mon[97450]: pgmap v10282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:56:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:44 smithi055 ceph-mon[97033]: pgmap v10282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:56:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:45.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:45.119690+0000) 2022-09-08T02:56:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:45.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:45.070661+0000) 2022-09-08T02:56:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:46.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:46.119861+0000) 2022-09-08T02:56:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:46.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:46.070885+0000) 2022-09-08T02:56:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:46 smithi059 ceph-mon[97450]: pgmap v10283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:56:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:46 smithi055 ceph-mon[97033]: pgmap v10283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:56:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:47.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:47.120049+0000) 2022-09-08T02:56:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:47.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:47.071041+0000) 2022-09-08T02:56:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:48.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:48.071265+0000) 2022-09-08T02:56:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:48.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:48.120223+0000) 2022-09-08T02:56:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:48 smithi059 ceph-mon[97450]: pgmap v10284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:56:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:48 smithi055 ceph-mon[97033]: pgmap v10284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:56:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:49.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:49.071450+0000) 2022-09-08T02:56:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:49.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:49.120442+0000) 2022-09-08T02:56:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:50.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:50.071659+0000) 2022-09-08T02:56:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:50.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:50.120681+0000) 2022-09-08T02:56:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:50 smithi059 ceph-mon[97450]: pgmap v10285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:56:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:50 smithi055 ceph-mon[97033]: pgmap v10285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:56:51.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:51.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:51.071902+0000) 2022-09-08T02:56:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:51.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:51.120916+0000) 2022-09-08T02:56:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:52.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:52.072106+0000) 2022-09-08T02:56:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:52.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:52.121062+0000) 2022-09-08T02:56:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:52 smithi059 ceph-mon[97450]: pgmap v10286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:56:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:52 smithi055 ceph-mon[97033]: pgmap v10286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:56:53.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:53.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:53.072338+0000) 2022-09-08T02:56:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:53.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:53.121253+0000) 2022-09-08T02:56:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:54.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:54.121417+0000) 2022-09-08T02:56:54.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:54.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:54.072563+0000) 2022-09-08T02:56:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:54 smithi059 ceph-mon[97450]: pgmap v10287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:56:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:54 smithi055 ceph-mon[97033]: pgmap v10287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:56:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:55.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:55.121604+0000) 2022-09-08T02:56:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:55.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:55.072769+0000) 2022-09-08T02:56:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:56.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:56.121824+0000) 2022-09-08T02:56:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:56.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:56.072989+0000) 2022-09-08T02:56:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:56 smithi059 ceph-mon[97450]: pgmap v10288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:56:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:56 smithi055 ceph-mon[97033]: pgmap v10288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:56:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:57.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:57.121992+0000) 2022-09-08T02:56:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:57.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:57.073239+0000) 2022-09-08T02:56:57.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:56:57.008+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:56:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:58.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:58.122204+0000) 2022-09-08T02:56:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:58.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:58.073425+0000) 2022-09-08T02:56:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:56:58 smithi059 ceph-mon[97450]: pgmap v10289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:56:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:56:58 smithi055 ceph-mon[97033]: pgmap v10289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:56:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:56:59.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:59.122453+0000) 2022-09-08T02:56:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:56:59.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:56:59.073583+0000) 2022-09-08T02:57:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:00.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:00.122611+0000) 2022-09-08T02:57:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:00.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:00.073802+0000) 2022-09-08T02:57:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:00 smithi059 ceph-mon[97450]: pgmap v10290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:00 smithi055 ceph-mon[97033]: pgmap v10290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:01.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:01.122724+0000) 2022-09-08T02:57:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:01.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:01.074017+0000) 2022-09-08T02:57:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:02.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:02.122970+0000) 2022-09-08T02:57:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:02.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:02.074264+0000) 2022-09-08T02:57:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:02 smithi059 ceph-mon[97450]: pgmap v10291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:57:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:02 smithi055 ceph-mon[97033]: pgmap v10291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:57:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:03.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:03.123213+0000) 2022-09-08T02:57:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:03.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:03.074510+0000) 2022-09-08T02:57:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:04.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:04.123451+0000) 2022-09-08T02:57:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:04.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:04.074725+0000) 2022-09-08T02:57:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:04 smithi059 ceph-mon[97450]: pgmap v10292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:04 smithi055 ceph-mon[97033]: pgmap v10292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:05.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:05.123623+0000) 2022-09-08T02:57:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:05.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:05.074943+0000) 2022-09-08T02:57:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:57:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:57:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:06.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:06.123874+0000) 2022-09-08T02:57:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:06.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:06.075180+0000) 2022-09-08T02:57:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:57:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:06 smithi055 ceph-mon[97033]: pgmap v10293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:57:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:57:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:06 smithi059 ceph-mon[97450]: pgmap v10293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:57:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:07.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:07.124049+0000) 2022-09-08T02:57:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:07.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:07.075402+0000) 2022-09-08T02:57:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:08.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:08.124276+0000) 2022-09-08T02:57:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:08.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:08.075589+0000) 2022-09-08T02:57:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:08 smithi059 ceph-mon[97450]: pgmap v10294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:57:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:08 smithi055 ceph-mon[97033]: pgmap v10294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T02:57:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:09.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:09.124512+0000) 2022-09-08T02:57:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:09.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:09.075795+0000) 2022-09-08T02:57:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:10.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:10.124761+0000) 2022-09-08T02:57:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:10.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:10.075979+0000) 2022-09-08T02:57:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:10 smithi059 ceph-mon[97450]: osdmap e723: 12 total, 11 up, 11 in 2022-09-08T02:57:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:10 smithi059 ceph-mon[97450]: pgmap v10296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:57:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:10 smithi055 ceph-mon[97033]: osdmap e723: 12 total, 11 up, 11 in 2022-09-08T02:57:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:10 smithi055 ceph-mon[97033]: pgmap v10296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:57:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:11.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:11.124954+0000) 2022-09-08T02:57:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:11.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:11.076214+0000) 2022-09-08T02:57:12.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:57:12.008+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:57:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:12.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:12.125176+0000) 2022-09-08T02:57:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:12.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:12.076467+0000) 2022-09-08T02:57:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:12 smithi059 ceph-mon[97450]: pgmap v10297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:57:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:12 smithi055 ceph-mon[97033]: pgmap v10297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T02:57:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:13.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:13.125393+0000) 2022-09-08T02:57:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:13.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:13.076728+0000) 2022-09-08T02:57:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:14.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:14.125630+0000) 2022-09-08T02:57:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:14.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:14.076951+0000) 2022-09-08T02:57:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:14 smithi059 ceph-mon[97450]: osdmap e724: 12 total, 11 up, 11 in 2022-09-08T02:57:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:14 smithi059 ceph-mon[97450]: pgmap v10299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:57:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:14 smithi055 ceph-mon[97033]: osdmap e724: 12 total, 11 up, 11 in 2022-09-08T02:57:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:14 smithi055 ceph-mon[97033]: pgmap v10299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:57:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:15.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:15.125904+0000) 2022-09-08T02:57:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:15.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:15.077212+0000) 2022-09-08T02:57:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:16.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:16.126131+0000) 2022-09-08T02:57:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:16.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:16.077450+0000) 2022-09-08T02:57:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:16 smithi059 ceph-mon[97450]: pgmap v10300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:57:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:16 smithi055 ceph-mon[97033]: pgmap v10300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:57:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:17.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:17.126311+0000) 2022-09-08T02:57:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:17.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:17.077672+0000) 2022-09-08T02:57:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:18.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:18.126530+0000) 2022-09-08T02:57:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:18.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:18.077940+0000) 2022-09-08T02:57:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:18 smithi059 ceph-mon[97450]: pgmap v10301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-08T02:57:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:18 smithi055 ceph-mon[97033]: pgmap v10301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-08T02:57:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:19.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:19.126774+0000) 2022-09-08T02:57:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:19.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:19.078189+0000) 2022-09-08T02:57:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:20.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:20.078436+0000) 2022-09-08T02:57:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:20.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:20.126916+0000) 2022-09-08T02:57:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:20 smithi059 ceph-mon[97450]: pgmap v10302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:57:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:20 smithi055 ceph-mon[97033]: pgmap v10302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T02:57:21.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:21.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:21.078633+0000) 2022-09-08T02:57:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:21.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:21.127130+0000) 2022-09-08T02:57:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:22.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:22.127354+0000) 2022-09-08T02:57:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:22.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:22.078892+0000) 2022-09-08T02:57:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:22 smithi059 ceph-mon[97450]: pgmap v10303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:57:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:22 smithi055 ceph-mon[97033]: pgmap v10303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:57:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:23.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:23.127545+0000) 2022-09-08T02:57:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:23.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:23.079100+0000) 2022-09-08T02:57:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:24.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:24.127730+0000) 2022-09-08T02:57:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:24.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:24.079330+0000) 2022-09-08T02:57:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:24 smithi059 ceph-mon[97450]: pgmap v10304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2022-09-08T02:57:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:24 smithi055 ceph-mon[97033]: pgmap v10304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2022-09-08T02:57:25.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:25.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:25.079574+0000) 2022-09-08T02:57:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:25.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:25.127854+0000) 2022-09-08T02:57:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:26.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:26.079751+0000) 2022-09-08T02:57:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:26.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:26.128061+0000) 2022-09-08T02:57:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:26 smithi059 ceph-mon[97450]: pgmap v10305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:57:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:26 smithi055 ceph-mon[97033]: pgmap v10305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:57:27.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:57:27.013+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:57:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:27.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:27.128310+0000) 2022-09-08T02:57:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:27.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:27.079966+0000) 2022-09-08T02:57:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:28.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:28.128538+0000) 2022-09-08T02:57:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:28.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:28.080133+0000) 2022-09-08T02:57:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:28 smithi059 ceph-mon[97450]: pgmap v10306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:28 smithi055 ceph-mon[97033]: pgmap v10306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:29.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:29.128779+0000) 2022-09-08T02:57:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:29.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:29.080336+0000) 2022-09-08T02:57:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:30.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:30.128978+0000) 2022-09-08T02:57:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:30.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:30.080566+0000) 2022-09-08T02:57:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:30 smithi059 ceph-mon[97450]: pgmap v10307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:30 smithi055 ceph-mon[97033]: pgmap v10307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:31.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:31.129200+0000) 2022-09-08T02:57:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:31.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:31.080723+0000) 2022-09-08T02:57:32.129 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:32.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:32.081093+0000) 2022-09-08T02:57:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:32.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:32.129404+0000) 2022-09-08T02:57:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:32 smithi059 ceph-mon[97450]: pgmap v10308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:57:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:57:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:57:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:57:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:32 smithi055 ceph-mon[97033]: pgmap v10308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:57:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:57:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:57:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:57:33.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:33.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:33.081313+0000) 2022-09-08T02:57:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:33.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:33.129609+0000) 2022-09-08T02:57:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:34.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:34.081487+0000) 2022-09-08T02:57:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:34.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:34.129847+0000) 2022-09-08T02:57:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:34 smithi059 ceph-mon[97450]: pgmap v10309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:34 smithi055 ceph-mon[97033]: pgmap v10309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:35.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:35.130077+0000) 2022-09-08T02:57:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:35.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:35.081719+0000) 2022-09-08T02:57:36.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:36.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:36.081867+0000) 2022-09-08T02:57:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:36.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:36.130211+0000) 2022-09-08T02:57:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:36 smithi059 ceph-mon[97450]: pgmap v10310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:57:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:36 smithi055 ceph-mon[97033]: pgmap v10310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:57:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:37.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:37.130390+0000) 2022-09-08T02:57:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:37.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:37.082108+0000) 2022-09-08T02:57:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:38.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:38.130583+0000) 2022-09-08T02:57:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:38.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:38.082268+0000) 2022-09-08T02:57:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:38 smithi059 ceph-mon[97450]: pgmap v10311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:38 smithi055 ceph-mon[97033]: pgmap v10311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:39.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:39.130794+0000) 2022-09-08T02:57:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:39.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:39.082482+0000) 2022-09-08T02:57:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:40.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:40.082668+0000) 2022-09-08T02:57:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:40.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:40.131001+0000) 2022-09-08T02:57:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:40 smithi059 ceph-mon[97450]: pgmap v10312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:40 smithi055 ceph-mon[97033]: pgmap v10312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:41.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:41.131204+0000) 2022-09-08T02:57:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:41.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:41.082871+0000) 2022-09-08T02:57:42.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:57:42.025+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:57:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:42.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:42.131408+0000) 2022-09-08T02:57:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:42.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:42.083040+0000) 2022-09-08T02:57:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:42 smithi059 ceph-mon[97450]: pgmap v10313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:57:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:42 smithi055 ceph-mon[97033]: pgmap v10313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:57:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:43.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:43.131633+0000) 2022-09-08T02:57:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:43.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:43.083292+0000) 2022-09-08T02:57:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:44.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:44.131827+0000) 2022-09-08T02:57:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:44.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:44.083466+0000) 2022-09-08T02:57:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:44 smithi059 ceph-mon[97450]: pgmap v10314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:44 smithi055 ceph-mon[97033]: pgmap v10314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:45.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:45.132048+0000) 2022-09-08T02:57:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:45.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:45.083605+0000) 2022-09-08T02:57:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:46.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:46.132267+0000) 2022-09-08T02:57:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:46.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:46.083794+0000) 2022-09-08T02:57:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:46 smithi059 ceph-mon[97450]: pgmap v10315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:57:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:46 smithi055 ceph-mon[97033]: pgmap v10315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:57:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:47.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:47.132507+0000) 2022-09-08T02:57:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:47.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:47.084039+0000) 2022-09-08T02:57:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:48.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:48.132693+0000) 2022-09-08T02:57:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:48.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:48.084238+0000) 2022-09-08T02:57:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:48 smithi059 ceph-mon[97450]: pgmap v10316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:49.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:48 smithi055 ceph-mon[97033]: pgmap v10316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:49.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:49.132857+0000) 2022-09-08T02:57:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:49.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:49.084403+0000) 2022-09-08T02:57:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:50.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:50.133107+0000) 2022-09-08T02:57:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:50.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:50.084652+0000) 2022-09-08T02:57:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:50 smithi059 ceph-mon[97450]: pgmap v10317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:50 smithi055 ceph-mon[97033]: pgmap v10317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:51.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:51.133354+0000) 2022-09-08T02:57:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:51.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:51.084813+0000) 2022-09-08T02:57:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:52.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:52.133596+0000) 2022-09-08T02:57:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:52.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:52.085013+0000) 2022-09-08T02:57:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:52 smithi059 ceph-mon[97450]: pgmap v10318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:57:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:52 smithi055 ceph-mon[97033]: pgmap v10318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:57:53.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:53.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:53.085200+0000) 2022-09-08T02:57:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:53.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:53.133718+0000) 2022-09-08T02:57:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:54.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:54.133874+0000) 2022-09-08T02:57:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:54.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:54.085417+0000) 2022-09-08T02:57:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:54 smithi059 ceph-mon[97450]: pgmap v10319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:54 smithi055 ceph-mon[97033]: pgmap v10319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:55.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:55.134107+0000) 2022-09-08T02:57:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:55.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:55.085670+0000) 2022-09-08T02:57:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:56.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:56.134301+0000) 2022-09-08T02:57:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:56.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:56.085893+0000) 2022-09-08T02:57:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:56 smithi059 ceph-mon[97450]: pgmap v10320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:57:57.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:57:57.040+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:57:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:57.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:57.134515+0000) 2022-09-08T02:57:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:57.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:57.086060+0000) 2022-09-08T02:57:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:56 smithi055 ceph-mon[97033]: pgmap v10320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:57:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:58.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:58.134712+0000) 2022-09-08T02:57:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:58.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:58.086281+0000) 2022-09-08T02:57:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:57:58 smithi059 ceph-mon[97450]: pgmap v10321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:57:58 smithi055 ceph-mon[97033]: pgmap v10321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:57:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:57:59.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:59.134958+0000) 2022-09-08T02:57:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:57:59.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:57:59.086488+0000) 2022-09-08T02:58:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:00.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:00.135183+0000) 2022-09-08T02:58:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:00.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:00.086745+0000) 2022-09-08T02:58:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:00 smithi059 ceph-mon[97450]: pgmap v10322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:00 smithi055 ceph-mon[97033]: pgmap v10322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:01.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:01.135348+0000) 2022-09-08T02:58:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:01.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:01.086962+0000) 2022-09-08T02:58:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:02.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:02.135585+0000) 2022-09-08T02:58:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:02.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:02.087163+0000) 2022-09-08T02:58:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:02 smithi059 ceph-mon[97450]: pgmap v10323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:58:03.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:02 smithi055 ceph-mon[97033]: pgmap v10323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:58:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:03.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:03.135788+0000) 2022-09-08T02:58:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:03.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:03.087382+0000) 2022-09-08T02:58:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:04.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:04.135919+0000) 2022-09-08T02:58:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:04.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:04.087613+0000) 2022-09-08T02:58:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:04 smithi059 ceph-mon[97450]: pgmap v10324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:04 smithi055 ceph-mon[97033]: pgmap v10324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:05.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:05.136111+0000) 2022-09-08T02:58:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:05.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:05.087873+0000) 2022-09-08T02:58:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:58:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:58:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:06.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:06.136337+0000) 2022-09-08T02:58:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:06.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:06.088107+0000) 2022-09-08T02:58:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:58:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:06 smithi055 ceph-mon[97033]: pgmap v10325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:58:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:58:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:06 smithi059 ceph-mon[97450]: pgmap v10325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:58:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:07.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:07.136594+0000) 2022-09-08T02:58:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:07.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:07.088362+0000) 2022-09-08T02:58:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:58:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:58:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:07 smithi059 ceph-mon[97450]: pgmap v10326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 49 KiB/s wr, 22 op/s 2022-09-08T02:58:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:07 smithi059 ceph-mon[97450]: pgmap v10327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 60 KiB/s wr, 27 op/s 2022-09-08T02:58:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:58:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:58:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:58:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:07 smithi055 ceph-mon[97033]: pgmap v10326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 49 KiB/s wr, 22 op/s 2022-09-08T02:58:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:07 smithi055 ceph-mon[97033]: pgmap v10327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 60 KiB/s wr, 27 op/s 2022-09-08T02:58:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:58:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:08.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:08.136766+0000) 2022-09-08T02:58:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:08.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:08.088596+0000) 2022-09-08T02:58:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:09.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:09.136990+0000) 2022-09-08T02:58:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:09.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:09.088726+0000) 2022-09-08T02:58:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:09 smithi059 ceph-mon[97450]: pgmap v10328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 60 KiB/s wr, 27 op/s 2022-09-08T02:58:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:09 smithi055 ceph-mon[97033]: pgmap v10328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 60 KiB/s wr, 27 op/s 2022-09-08T02:58:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:10.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:10.137138+0000) 2022-09-08T02:58:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:10.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:10.088923+0000) 2022-09-08T02:58:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:10 smithi059 ceph-mon[97450]: osdmap e725: 12 total, 11 up, 11 in 2022-09-08T02:58:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:10 smithi055 ceph-mon[97033]: osdmap e725: 12 total, 11 up, 11 in 2022-09-08T02:58:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:11.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:11.137385+0000) 2022-09-08T02:58:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:11.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:11.089168+0000) 2022-09-08T02:58:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:11 smithi059 ceph-mon[97450]: pgmap v10330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 20 KiB/s rd, 78 KiB/s wr, 35 op/s 2022-09-08T02:58:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:11 smithi055 ceph-mon[97033]: pgmap v10330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 20 KiB/s rd, 78 KiB/s wr, 35 op/s 2022-09-08T02:58:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:12.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:12.137536+0000) 2022-09-08T02:58:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:12.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:12.089408+0000) 2022-09-08T02:58:12.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:58:12.041+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:58:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:13.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:13.137724+0000) 2022-09-08T02:58:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:13.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:13.089673+0000) 2022-09-08T02:58:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:13 smithi055 ceph-mon[97033]: pgmap v10331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 150 B/s wr, 4 op/s 2022-09-08T02:58:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:13 smithi055 ceph-mon[97033]: osdmap e726: 12 total, 11 up, 11 in 2022-09-08T02:58:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:13 smithi059 ceph-mon[97450]: pgmap v10331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 150 B/s wr, 4 op/s 2022-09-08T02:58:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:13 smithi059 ceph-mon[97450]: osdmap e726: 12 total, 11 up, 11 in 2022-09-08T02:58:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:14.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:14.137903+0000) 2022-09-08T02:58:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:14.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:14.089945+0000) 2022-09-08T02:58:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:15.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:15.138143+0000) 2022-09-08T02:58:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:15.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:15.090087+0000) 2022-09-08T02:58:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:15 smithi055 ceph-mon[97033]: pgmap v10333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:58:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:15 smithi059 ceph-mon[97450]: pgmap v10333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T02:58:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:16.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:16.090231+0000) 2022-09-08T02:58:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:16.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:16.138327+0000) 2022-09-08T02:58:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:17.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:17.138533+0000) 2022-09-08T02:58:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:17.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:17.090485+0000) 2022-09-08T02:58:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:17 smithi055 ceph-mon[97033]: pgmap v10334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:58:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:17 smithi059 ceph-mon[97450]: pgmap v10334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:58:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:18.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:18.138767+0000) 2022-09-08T02:58:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:18.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:18.090746+0000) 2022-09-08T02:58:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:19.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:19.139011+0000) 2022-09-08T02:58:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:19.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:19.090994+0000) 2022-09-08T02:58:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:19 smithi059 ceph-mon[97450]: pgmap v10335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s 2022-09-08T02:58:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:19 smithi055 ceph-mon[97033]: pgmap v10335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s 2022-09-08T02:58:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:20.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:20.139229+0000) 2022-09-08T02:58:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:20.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:20.091198+0000) 2022-09-08T02:58:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:21.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:21.139478+0000) 2022-09-08T02:58:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:21.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:21.091361+0000) 2022-09-08T02:58:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:21 smithi055 ceph-mon[97033]: pgmap v10336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:58:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:21 smithi059 ceph-mon[97450]: pgmap v10336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:58:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:22.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:22.091603+0000) 2022-09-08T02:58:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:22.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:22.139764+0000) 2022-09-08T02:58:23.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:23.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:23.091868+0000) 2022-09-08T02:58:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:23.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:23.140040+0000) 2022-09-08T02:58:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:23 smithi055 ceph-mon[97033]: pgmap v10337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:58:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:23 smithi059 ceph-mon[97450]: pgmap v10337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:58:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:24.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:24.140280+0000) 2022-09-08T02:58:24.426 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:24.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:24.092065+0000) 2022-09-08T02:58:25.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:25.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:25.092230+0000) 2022-09-08T02:58:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:25.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:25.140540+0000) 2022-09-08T02:58:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:25 smithi055 ceph-mon[97033]: pgmap v10338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-08T02:58:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:25 smithi059 ceph-mon[97450]: pgmap v10338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-08T02:58:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:26.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:26.092461+0000) 2022-09-08T02:58:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:26.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:26.140791+0000) 2022-09-08T02:58:27.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:58:27.049+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:58:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:27.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:27.092689+0000) 2022-09-08T02:58:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:27.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:27.141025+0000) 2022-09-08T02:58:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:27 smithi055 ceph-mon[97033]: pgmap v10339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:58:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:27 smithi059 ceph-mon[97450]: pgmap v10339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T02:58:28.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:28.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:28.092896+0000) 2022-09-08T02:58:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:28.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:28.141215+0000) 2022-09-08T02:58:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:29.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:29.093103+0000) 2022-09-08T02:58:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:29.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:29.141451+0000) 2022-09-08T02:58:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:29 smithi059 ceph-mon[97450]: pgmap v10340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:29 smithi055 ceph-mon[97033]: pgmap v10340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:30.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:30.093351+0000) 2022-09-08T02:58:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:30.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:30.141652+0000) 2022-09-08T02:58:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:31.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:31.141806+0000) 2022-09-08T02:58:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:31.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:31.093568+0000) 2022-09-08T02:58:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:31 smithi055 ceph-mon[97033]: pgmap v10341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:58:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:31 smithi059 ceph-mon[97450]: pgmap v10341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:58:32.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:32.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:32.093772+0000) 2022-09-08T02:58:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:32.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:32.141968+0000) 2022-09-08T02:58:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:33.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:33.142129+0000) 2022-09-08T02:58:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:33.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:33.093987+0000) 2022-09-08T02:58:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:33 smithi055 ceph-mon[97033]: pgmap v10342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:33 smithi059 ceph-mon[97450]: pgmap v10342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:34.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:34.142254+0000) 2022-09-08T02:58:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:34.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:34.094207+0000) 2022-09-08T02:58:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:35.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:35.142466+0000) 2022-09-08T02:58:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:35.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:35.094434+0000) 2022-09-08T02:58:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:35 smithi055 ceph-mon[97033]: pgmap v10343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:35 smithi059 ceph-mon[97450]: pgmap v10343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:36.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:36.142740+0000) 2022-09-08T02:58:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:36.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:36.094580+0000) 2022-09-08T02:58:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:37.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:37.142949+0000) 2022-09-08T02:58:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:37.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:37.094734+0000) 2022-09-08T02:58:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:37 smithi055 ceph-mon[97033]: pgmap v10344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:58:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:37 smithi059 ceph-mon[97450]: pgmap v10344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:58:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:38.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:38.143119+0000) 2022-09-08T02:58:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:38.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:38.094978+0000) 2022-09-08T02:58:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:39.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:39.143316+0000) 2022-09-08T02:58:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:39.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:39.095193+0000) 2022-09-08T02:58:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:39 smithi059 ceph-mon[97450]: pgmap v10345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:39 smithi055 ceph-mon[97033]: pgmap v10345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:40.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:40.143549+0000) 2022-09-08T02:58:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:40.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:40.095370+0000) 2022-09-08T02:58:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:41.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:41.143745+0000) 2022-09-08T02:58:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:41.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:41.095620+0000) 2022-09-08T02:58:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:41 smithi055 ceph-mon[97033]: pgmap v10346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:58:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:41 smithi059 ceph-mon[97450]: pgmap v10346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:58:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:42.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:42.143928+0000) 2022-09-08T02:58:42.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:58:42.057+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:58:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:42.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:42.095864+0000) 2022-09-08T02:58:43.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:43.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:43.096040+0000) 2022-09-08T02:58:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:43.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:43.144161+0000) 2022-09-08T02:58:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:43 smithi055 ceph-mon[97033]: pgmap v10347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:43 smithi059 ceph-mon[97450]: pgmap v10347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:44.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:44.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:44.096237+0000) 2022-09-08T02:58:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:44.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:44.144353+0000) 2022-09-08T02:58:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:45.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:45.144489+0000) 2022-09-08T02:58:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:45.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:45.096458+0000) 2022-09-08T02:58:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:45 smithi055 ceph-mon[97033]: pgmap v10348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:45 smithi059 ceph-mon[97450]: pgmap v10348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:46.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:46.096713+0000) 2022-09-08T02:58:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:46.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:46.144759+0000) 2022-09-08T02:58:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:47.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:47.145024+0000) 2022-09-08T02:58:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:47.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:47.096950+0000) 2022-09-08T02:58:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:47 smithi055 ceph-mon[97033]: pgmap v10349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:58:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:47 smithi059 ceph-mon[97450]: pgmap v10349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:58:48.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:48.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:48.097164+0000) 2022-09-08T02:58:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:48.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:48.145272+0000) 2022-09-08T02:58:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:49.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:49.145433+0000) 2022-09-08T02:58:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:49.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:49.097396+0000) 2022-09-08T02:58:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:49 smithi059 ceph-mon[97450]: pgmap v10350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:49 smithi055 ceph-mon[97033]: pgmap v10350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:50.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:50.145681+0000) 2022-09-08T02:58:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:50.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:50.097687+0000) 2022-09-08T02:58:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:51.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:51.145915+0000) 2022-09-08T02:58:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:51.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:51.097852+0000) 2022-09-08T02:58:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:51 smithi055 ceph-mon[97033]: pgmap v10351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:58:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:51 smithi059 ceph-mon[97450]: pgmap v10351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:58:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:52.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:52.146149+0000) 2022-09-08T02:58:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:52.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:52.098038+0000) 2022-09-08T02:58:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:53.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:53.146324+0000) 2022-09-08T02:58:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:53.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:53.098239+0000) 2022-09-08T02:58:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:53 smithi055 ceph-mon[97033]: pgmap v10352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:53 smithi059 ceph-mon[97450]: pgmap v10352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:54.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:54.098466+0000) 2022-09-08T02:58:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:54.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:54.146532+0000) 2022-09-08T02:58:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:55.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:55.146791+0000) 2022-09-08T02:58:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:55.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:55.098727+0000) 2022-09-08T02:58:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:55 smithi055 ceph-mon[97033]: pgmap v10353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:55 smithi059 ceph-mon[97450]: pgmap v10353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:56.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:56.146988+0000) 2022-09-08T02:58:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:56.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:56.098973+0000) 2022-09-08T02:58:57.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:58:57.067+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:58:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:57.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:57.147191+0000) 2022-09-08T02:58:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:57.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:57.099120+0000) 2022-09-08T02:58:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:57 smithi055 ceph-mon[97033]: pgmap v10354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:58:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:57 smithi059 ceph-mon[97450]: pgmap v10354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:58:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:58.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:58.147414+0000) 2022-09-08T02:58:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:58.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:58.099346+0000) 2022-09-08T02:58:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:58:59.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:59.147649+0000) 2022-09-08T02:58:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:58:59.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:58:59.099569+0000) 2022-09-08T02:58:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:58:59 smithi059 ceph-mon[97450]: pgmap v10355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:58:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:58:59 smithi055 ceph-mon[97033]: pgmap v10355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:00.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:00.147883+0000) 2022-09-08T02:59:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:00.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:00.099786+0000) 2022-09-08T02:59:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:01.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:01.099922+0000) 2022-09-08T02:59:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:01.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:01.148025+0000) 2022-09-08T02:59:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:01 smithi055 ceph-mon[97033]: pgmap v10356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:59:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:01 smithi059 ceph-mon[97450]: pgmap v10356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:59:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:02.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:02.148224+0000) 2022-09-08T02:59:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:02.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:02.100190+0000) 2022-09-08T02:59:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:03.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:03.148422+0000) 2022-09-08T02:59:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:03.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:03.100336+0000) 2022-09-08T02:59:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:03 smithi055 ceph-mon[97033]: pgmap v10357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:03 smithi059 ceph-mon[97450]: pgmap v10357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:04.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:04.148682+0000) 2022-09-08T02:59:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:04.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:04.100564+0000) 2022-09-08T02:59:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:05.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:05.148923+0000) 2022-09-08T02:59:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:05.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:05.100792+0000) 2022-09-08T02:59:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:05 smithi055 ceph-mon[97033]: pgmap v10358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:59:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:05 smithi059 ceph-mon[97450]: pgmap v10358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T02:59:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:06.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:06.149145+0000) 2022-09-08T02:59:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:06.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:06.101036+0000) 2022-09-08T02:59:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:59:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T02:59:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:07.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:07.149341+0000) 2022-09-08T02:59:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:07.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:07.101271+0000) 2022-09-08T02:59:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:07 smithi055 ceph-mon[97033]: pgmap v10359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:59:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:59:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:59:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:59:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:07 smithi055 ceph-mon[97033]: pgmap v10360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T02:59:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:59:08.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:07 smithi059 ceph-mon[97450]: pgmap v10359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T02:59:08.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T02:59:08.041 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T02:59:08.042 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T02:59:08.042 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:07 smithi059 ceph-mon[97450]: pgmap v10360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T02:59:08.042 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T02:59:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:08.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:08.149527+0000) 2022-09-08T02:59:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:08.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:08.101517+0000) 2022-09-08T02:59:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:08 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T02:59:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:08 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T02:59:09.413 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:09.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:09.149741+0000) 2022-09-08T02:59:09.414 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:09.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:09.101714+0000) 2022-09-08T02:59:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:09 smithi055 ceph-mon[97033]: pgmap v10361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T02:59:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:09 smithi055 ceph-mon[97033]: osdmap e727: 12 total, 11 up, 11 in 2022-09-08T02:59:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:09 smithi059 ceph-mon[97450]: pgmap v10361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T02:59:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:09 smithi059 ceph-mon[97450]: osdmap e727: 12 total, 11 up, 11 in 2022-09-08T02:59:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:10.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:10.149988+0000) 2022-09-08T02:59:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:10.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:10.101961+0000) 2022-09-08T02:59:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:11.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:11.150226+0000) 2022-09-08T02:59:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:11.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:11.102146+0000) 2022-09-08T02:59:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:11 smithi055 ceph-mon[97033]: pgmap v10363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2022-09-08T02:59:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:11 smithi059 ceph-mon[97450]: pgmap v10363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2022-09-08T02:59:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:12.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:12.150461+0000) 2022-09-08T02:59:12.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:59:12.084+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:59:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:12.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:12.102377+0000) 2022-09-08T02:59:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:13.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:13.150581+0000) 2022-09-08T02:59:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:13.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:13.102609+0000) 2022-09-08T02:59:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:13 smithi055 ceph-mon[97033]: pgmap v10364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2022-09-08T02:59:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:13 smithi059 ceph-mon[97450]: pgmap v10364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2022-09-08T02:59:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:14.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:14.150803+0000) 2022-09-08T02:59:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:14.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:14.102801+0000) 2022-09-08T02:59:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:14 smithi059 ceph-mon[97450]: osdmap e728: 12 total, 11 up, 11 in 2022-09-08T02:59:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:14 smithi055 ceph-mon[97033]: osdmap e728: 12 total, 11 up, 11 in 2022-09-08T02:59:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:15.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:15.150989+0000) 2022-09-08T02:59:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:15.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:15.103013+0000) 2022-09-08T02:59:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:15 smithi055 ceph-mon[97033]: pgmap v10366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:59:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:15 smithi059 ceph-mon[97450]: pgmap v10366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:59:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:16.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:16.151165+0000) 2022-09-08T02:59:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:16.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:16.103183+0000) 2022-09-08T02:59:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:17.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:17.151374+0000) 2022-09-08T02:59:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:17.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:17.103365+0000) 2022-09-08T02:59:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:17 smithi055 ceph-mon[97033]: pgmap v10367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:59:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:17 smithi059 ceph-mon[97450]: pgmap v10367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T02:59:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:18.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:18.151602+0000) 2022-09-08T02:59:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:18.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:18.103620+0000) 2022-09-08T02:59:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:19.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:19.151800+0000) 2022-09-08T02:59:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:19.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:19.103858+0000) 2022-09-08T02:59:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:19 smithi059 ceph-mon[97450]: pgmap v10368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T02:59:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:19 smithi055 ceph-mon[97033]: pgmap v10368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T02:59:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:20.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:20.104090+0000) 2022-09-08T02:59:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:20.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:20.152035+0000) 2022-09-08T02:59:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:21.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:21.104318+0000) 2022-09-08T02:59:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:21.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:21.152175+0000) 2022-09-08T02:59:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:21 smithi055 ceph-mon[97033]: pgmap v10369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:59:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:21 smithi059 ceph-mon[97450]: pgmap v10369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:59:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:22.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:22.104541+0000) 2022-09-08T02:59:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:22.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:22.152407+0000) 2022-09-08T02:59:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:23.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:23.152593+0000) 2022-09-08T02:59:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:23.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:23.104722+0000) 2022-09-08T02:59:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:23 smithi055 ceph-mon[97033]: pgmap v10370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:59:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:23 smithi059 ceph-mon[97450]: pgmap v10370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T02:59:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:24.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:24.104936+0000) 2022-09-08T02:59:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:24.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:24.152847+0000) 2022-09-08T02:59:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:25.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:25.153004+0000) 2022-09-08T02:59:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:25.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:25.105119+0000) 2022-09-08T02:59:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:25 smithi055 ceph-mon[97033]: pgmap v10371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T02:59:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:25 smithi059 ceph-mon[97450]: pgmap v10371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T02:59:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:26.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:26.153223+0000) 2022-09-08T02:59:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:26.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:26.105351+0000) 2022-09-08T02:59:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:27.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:27.153490+0000) 2022-09-08T02:59:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:27.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:27.105589+0000) 2022-09-08T02:59:27.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:59:27.096+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:59:27.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:27 smithi055 ceph-mon[97033]: pgmap v10372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:27 smithi059 ceph-mon[97450]: pgmap v10372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:28.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:28.153781+0000) 2022-09-08T02:59:28.450 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:28.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:28.105768+0000) 2022-09-08T02:59:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:29.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:29.154039+0000) 2022-09-08T02:59:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:29.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:29.105921+0000) 2022-09-08T02:59:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:29 smithi055 ceph-mon[97033]: pgmap v10373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:29 smithi059 ceph-mon[97450]: pgmap v10373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:30.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:30.154260+0000) 2022-09-08T02:59:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:30.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:30.106101+0000) 2022-09-08T02:59:31.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:31.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:31.106282+0000) 2022-09-08T02:59:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:31.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:31.154502+0000) 2022-09-08T02:59:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:31 smithi055 ceph-mon[97033]: pgmap v10374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:59:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:31 smithi059 ceph-mon[97450]: pgmap v10374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:59:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:32.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:32.154747+0000) 2022-09-08T02:59:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:32.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:32.106501+0000) 2022-09-08T02:59:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:33.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:33.154994+0000) 2022-09-08T02:59:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:33.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:33.106761+0000) 2022-09-08T02:59:33.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:33 smithi055 ceph-mon[97033]: pgmap v10375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:33 smithi059 ceph-mon[97450]: pgmap v10375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:34.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:34.106932+0000) 2022-09-08T02:59:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:34.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:34.155261+0000) 2022-09-08T02:59:35.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:35.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:35.107123+0000) 2022-09-08T02:59:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:35.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:35.155454+0000) 2022-09-08T02:59:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:35 smithi055 ceph-mon[97033]: pgmap v10376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:59:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:35 smithi059 ceph-mon[97450]: pgmap v10376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:59:36.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:36.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:36.107331+0000) 2022-09-08T02:59:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:36.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:36.155718+0000) 2022-09-08T02:59:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:37.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:37.155934+0000) 2022-09-08T02:59:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:37.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:37.107562+0000) 2022-09-08T02:59:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:37 smithi055 ceph-mon[97033]: pgmap v10377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:37 smithi059 ceph-mon[97450]: pgmap v10377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:38.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:38.156203+0000) 2022-09-08T02:59:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:38.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:38.107768+0000) 2022-09-08T02:59:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:39.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:39.156418+0000) 2022-09-08T02:59:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:39.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:39.107918+0000) 2022-09-08T02:59:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:39 smithi055 ceph-mon[97033]: pgmap v10378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:39 smithi059 ceph-mon[97450]: pgmap v10378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:40.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:40.156717+0000) 2022-09-08T02:59:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:40.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:40.108054+0000) 2022-09-08T02:59:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:41.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:41.108238+0000) 2022-09-08T02:59:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:41.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:41.156917+0000) 2022-09-08T02:59:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:41 smithi055 ceph-mon[97033]: pgmap v10379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:59:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:41 smithi059 ceph-mon[97450]: pgmap v10379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:59:42.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:59:42.110+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:59:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:42.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:42.108371+0000) 2022-09-08T02:59:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:42.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:42.157162+0000) 2022-09-08T02:59:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:43.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:43.157328+0000) 2022-09-08T02:59:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:43.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:43.108541+0000) 2022-09-08T02:59:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:43 smithi055 ceph-mon[97033]: pgmap v10380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:43 smithi059 ceph-mon[97450]: pgmap v10380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:44.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:44.157508+0000) 2022-09-08T02:59:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:44.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:44.108716+0000) 2022-09-08T02:59:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:45.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:45.157754+0000) 2022-09-08T02:59:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:45.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:45.108943+0000) 2022-09-08T02:59:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:45 smithi055 ceph-mon[97033]: pgmap v10381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:59:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:45 smithi059 ceph-mon[97450]: pgmap v10381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:59:46.408 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:46.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:46.157980+0000) 2022-09-08T02:59:46.409 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:46.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:46.109177+0000) 2022-09-08T02:59:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:47.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:47.158207+0000) 2022-09-08T02:59:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:47.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:47.109408+0000) 2022-09-08T02:59:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:47 smithi055 ceph-mon[97033]: pgmap v10382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:47 smithi059 ceph-mon[97450]: pgmap v10382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:48.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:48.158437+0000) 2022-09-08T02:59:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:48.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:48.109679+0000) 2022-09-08T02:59:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:49.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:49.109909+0000) 2022-09-08T02:59:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:49.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:49.158685+0000) 2022-09-08T02:59:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:49 smithi055 ceph-mon[97033]: pgmap v10383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:49 smithi059 ceph-mon[97450]: pgmap v10383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:50.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:50.158869+0000) 2022-09-08T02:59:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:50.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:50.110127+0000) 2022-09-08T02:59:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:51.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:51.159018+0000) 2022-09-08T02:59:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:51.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:51.110339+0000) 2022-09-08T02:59:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:51 smithi055 ceph-mon[97033]: pgmap v10384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:59:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:51 smithi059 ceph-mon[97450]: pgmap v10384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:59:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:52.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:52.159191+0000) 2022-09-08T02:59:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:52.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:52.110566+0000) 2022-09-08T02:59:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:53.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:53.159406+0000) 2022-09-08T02:59:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:53.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:53.110747+0000) 2022-09-08T02:59:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:53 smithi055 ceph-mon[97033]: pgmap v10385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:53 smithi059 ceph-mon[97450]: pgmap v10385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:54.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:54.159551+0000) 2022-09-08T02:59:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:54.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:54.110920+0000) 2022-09-08T02:59:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:55.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:55.159737+0000) 2022-09-08T02:59:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:55.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:55.111041+0000) 2022-09-08T02:59:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:55 smithi055 ceph-mon[97033]: pgmap v10386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:59:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:55 smithi059 ceph-mon[97450]: pgmap v10386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T02:59:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:56.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:56.159959+0000) 2022-09-08T02:59:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:56.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:56.111248+0000) 2022-09-08T02:59:57.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 02:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T02:59:57.127+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T02:59:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:57.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:57.160197+0000) 2022-09-08T02:59:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:57.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:57.111464+0000) 2022-09-08T02:59:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:57 smithi059 ceph-mon[97450]: pgmap v10387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:57 smithi055 ceph-mon[97033]: pgmap v10387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:58.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:58.160463+0000) 2022-09-08T02:59:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:58.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:58.111624+0000) 2022-09-08T02:59:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 02:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T02:59:59.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:59.160612+0000) 2022-09-08T02:59:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 02:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T02:59:59.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T01:59:59.111813+0000) 2022-09-08T02:59:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 02:59:59 smithi055 ceph-mon[97033]: pgmap v10388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T02:59:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 02:59:59 smithi059 ceph-mon[97450]: pgmap v10388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:00:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:00.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:00.112062+0000) 2022-09-08T03:00:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:00.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:00.160830+0000) 2022-09-08T03:00:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T03:00:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T03:00:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T03:00:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T03:00:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T03:00:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T03:00:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:01.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:01.161002+0000) 2022-09-08T03:00:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:01.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:01.112268+0000) 2022-09-08T03:00:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:01 smithi055 ceph-mon[97033]: pgmap v10389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:00:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:01 smithi059 ceph-mon[97450]: pgmap v10389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:00:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:02.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:02.161221+0000) 2022-09-08T03:00:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:02.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:02.112545+0000) 2022-09-08T03:00:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:03.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:03.112805+0000) 2022-09-08T03:00:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:03.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:03.161455+0000) 2022-09-08T03:00:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:03 smithi059 ceph-mon[97450]: pgmap v10390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:00:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:03 smithi055 ceph-mon[97033]: pgmap v10390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:00:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:04.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:04.161693+0000) 2022-09-08T03:00:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:04.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:04.113046+0000) 2022-09-08T03:00:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:05.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:05.161903+0000) 2022-09-08T03:00:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:05.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:05.113229+0000) 2022-09-08T03:00:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:05 smithi055 ceph-mon[97033]: pgmap v10391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:00:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:05 smithi059 ceph-mon[97450]: pgmap v10391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:00:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:06.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:06.113391+0000) 2022-09-08T03:00:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:06.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:06.162138+0000) 2022-09-08T03:00:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:00:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:00:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:00:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:00:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:07.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:07.113654+0000) 2022-09-08T03:00:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:07.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:07.162372+0000) 2022-09-08T03:00:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:07 smithi059 ceph-mon[97450]: pgmap v10392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:00:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:07 smithi055 ceph-mon[97033]: pgmap v10392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:00:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:08.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:08.162614+0000) 2022-09-08T03:00:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:08.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:08.113828+0000) 2022-09-08T03:00:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:00:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:00:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:00:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:00:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:00:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:00:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:00:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:00:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:00:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:00:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:09.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:09.162765+0000) 2022-09-08T03:00:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:09.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:09.114043+0000) 2022-09-08T03:00:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:09 smithi055 ceph-mon[97033]: pgmap v10393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:00:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:09 smithi059 ceph-mon[97450]: pgmap v10393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:00:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:10.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:10.162975+0000) 2022-09-08T03:00:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:10.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:10.114239+0000) 2022-09-08T03:00:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:10 smithi059 ceph-mon[97450]: osdmap e729: 12 total, 11 up, 11 in 2022-09-08T03:00:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:00:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:00:10.567 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:10 smithi055 ceph-mon[97033]: osdmap e729: 12 total, 11 up, 11 in 2022-09-08T03:00:10.568 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:00:10.568 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:00:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:11.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:11.163179+0000) 2022-09-08T03:00:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:11.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:11.114442+0000) 2022-09-08T03:00:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:00:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:00:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:11.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T03:00:11.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T03:00:11.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:11.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:11.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:11.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:11.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T03:00:11.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T03:00:11.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:00:11.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:11 smithi055 ceph-mon[97033]: pgmap v10395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:00:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:00:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:00:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T03:00:12.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T03:00:12.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:12.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:12.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:12.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:00:12.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T03:00:12.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T03:00:12.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:00:12.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:11 smithi059 ceph-mon[97450]: pgmap v10395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:00:12.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:00:12.142+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:00:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:12.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:12.163405+0000) 2022-09-08T03:00:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:12.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:12.114618+0000) 2022-09-08T03:00:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:13.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:13.163664+0000) 2022-09-08T03:00:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:13.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:13.114831+0000) 2022-09-08T03:00:13.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:13 smithi059 ceph-mon[97450]: pgmap v10396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:00:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:13 smithi055 ceph-mon[97033]: pgmap v10396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:00:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:14.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:14.163834+0000) 2022-09-08T03:00:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:14.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:14.115054+0000) 2022-09-08T03:00:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:15.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:15.164066+0000) 2022-09-08T03:00:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:15.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:15.115271+0000) 2022-09-08T03:00:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:15 smithi055 ceph-mon[97033]: pgmap v10397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:00:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:15 smithi059 ceph-mon[97450]: pgmap v10397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:00:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:16.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:16.164272+0000) 2022-09-08T03:00:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:16.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:16.115416+0000) 2022-09-08T03:00:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:17.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:17.164458+0000) 2022-09-08T03:00:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:17.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:17.115686+0000) 2022-09-08T03:00:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:17 smithi059 ceph-mon[97450]: pgmap v10398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:00:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:17 smithi055 ceph-mon[97033]: pgmap v10398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:00:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:18.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:18.164739+0000) 2022-09-08T03:00:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:18.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:18.115897+0000) 2022-09-08T03:00:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:19.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:19.164905+0000) 2022-09-08T03:00:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:19.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:19.116143+0000) 2022-09-08T03:00:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:19 smithi055 ceph-mon[97033]: osdmap e730: 12 total, 11 up, 11 in 2022-09-08T03:00:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:19 smithi055 ceph-mon[97033]: pgmap v10400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T03:00:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:19 smithi059 ceph-mon[97450]: osdmap e730: 12 total, 11 up, 11 in 2022-09-08T03:00:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:19 smithi059 ceph-mon[97450]: pgmap v10400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T03:00:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:20.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:20.165029+0000) 2022-09-08T03:00:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:20.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:20.116362+0000) 2022-09-08T03:00:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:21.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:21.165239+0000) 2022-09-08T03:00:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:21.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:21.116587+0000) 2022-09-08T03:00:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:21 smithi059 ceph-mon[97450]: pgmap v10401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:00:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:21 smithi055 ceph-mon[97033]: pgmap v10401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:00:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:22.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:22.165472+0000) 2022-09-08T03:00:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:22.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:22.116780+0000) 2022-09-08T03:00:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:23.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:23.165684+0000) 2022-09-08T03:00:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:23.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:23.117004+0000) 2022-09-08T03:00:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:23 smithi059 ceph-mon[97450]: pgmap v10402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:00:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:23 smithi055 ceph-mon[97033]: pgmap v10402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:00:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:24.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:24.165855+0000) 2022-09-08T03:00:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:24.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:24.117234+0000) 2022-09-08T03:00:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:25.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:25.166067+0000) 2022-09-08T03:00:25.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:25.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:25.117489+0000) 2022-09-08T03:00:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:25 smithi059 ceph-mon[97450]: pgmap v10403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:00:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:25 smithi055 ceph-mon[97033]: pgmap v10403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:00:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:26.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:26.166206+0000) 2022-09-08T03:00:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:26.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:26.117692+0000) 2022-09-08T03:00:27.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:00:27.146+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:00:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:27.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:27.166428+0000) 2022-09-08T03:00:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:27.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:27.117832+0000) 2022-09-08T03:00:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:27 smithi059 ceph-mon[97450]: pgmap v10404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:00:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:27 smithi055 ceph-mon[97033]: pgmap v10404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:00:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:28.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:28.166675+0000) 2022-09-08T03:00:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:28.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:28.118078+0000) 2022-09-08T03:00:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:29.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:29.166800+0000) 2022-09-08T03:00:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:29.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:29.118273+0000) 2022-09-08T03:00:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:29 smithi055 ceph-mon[97033]: pgmap v10405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:00:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:29 smithi059 ceph-mon[97450]: pgmap v10405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:00:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:30.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:30.166997+0000) 2022-09-08T03:00:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:30.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:30.118480+0000) 2022-09-08T03:00:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:31.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:31.167184+0000) 2022-09-08T03:00:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:31.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:31.118729+0000) 2022-09-08T03:00:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:31 smithi059 ceph-mon[97450]: pgmap v10406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:00:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:31 smithi055 ceph-mon[97033]: pgmap v10406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:00:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:32.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:32.167412+0000) 2022-09-08T03:00:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:32.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:32.119001+0000) 2022-09-08T03:00:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:33.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:33.167599+0000) 2022-09-08T03:00:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:33.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:33.119226+0000) 2022-09-08T03:00:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:33 smithi059 ceph-mon[97450]: pgmap v10407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:00:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:33 smithi055 ceph-mon[97033]: pgmap v10407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:00:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:34.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:34.167824+0000) 2022-09-08T03:00:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:34.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:34.119429+0000) 2022-09-08T03:00:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:35.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:35.168062+0000) 2022-09-08T03:00:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:35.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:35.119599+0000) 2022-09-08T03:00:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:35 smithi059 ceph-mon[97450]: pgmap v10408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:00:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:35 smithi055 ceph-mon[97033]: pgmap v10408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:00:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:36.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:36.168292+0000) 2022-09-08T03:00:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:36.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:36.119818+0000) 2022-09-08T03:00:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:37.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:37.168533+0000) 2022-09-08T03:00:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:37.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:37.120074+0000) 2022-09-08T03:00:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:37 smithi059 ceph-mon[97450]: pgmap v10409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:00:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:37 smithi055 ceph-mon[97033]: pgmap v10409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:00:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:38.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:38.168771+0000) 2022-09-08T03:00:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:38.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:38.120317+0000) 2022-09-08T03:00:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:39.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:39.169014+0000) 2022-09-08T03:00:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:39.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:39.120493+0000) 2022-09-08T03:00:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:39 smithi055 ceph-mon[97033]: pgmap v10410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:00:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:39 smithi059 ceph-mon[97450]: pgmap v10410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:00:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:40.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:40.169177+0000) 2022-09-08T03:00:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:40.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:40.120737+0000) 2022-09-08T03:00:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:41.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:41.169392+0000) 2022-09-08T03:00:41.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:41.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:41.120984+0000) 2022-09-08T03:00:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:41 smithi059 ceph-mon[97450]: pgmap v10411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:00:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:41 smithi055 ceph-mon[97033]: pgmap v10411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:00:42.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:00:42.159+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:00:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:42.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:42.169555+0000) 2022-09-08T03:00:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:42.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:42.121187+0000) 2022-09-08T03:00:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:43.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:43.169785+0000) 2022-09-08T03:00:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:43.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:43.121392+0000) 2022-09-08T03:00:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:43 smithi059 ceph-mon[97450]: pgmap v10412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:00:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:43 smithi055 ceph-mon[97033]: pgmap v10412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:00:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:44.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:44.170067+0000) 2022-09-08T03:00:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:44.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:44.121619+0000) 2022-09-08T03:00:45.409 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:45.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:45.170304+0000) 2022-09-08T03:00:45.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:45.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:45.121875+0000) 2022-09-08T03:00:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:45 smithi059 ceph-mon[97450]: pgmap v10413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:00:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:45 smithi055 ceph-mon[97033]: pgmap v10413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:00:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:46.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:46.170493+0000) 2022-09-08T03:00:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:46.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:46.122113+0000) 2022-09-08T03:00:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:47.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:47.170737+0000) 2022-09-08T03:00:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:47.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:47.122376+0000) 2022-09-08T03:00:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:47 smithi059 ceph-mon[97450]: pgmap v10414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:00:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:47 smithi055 ceph-mon[97033]: pgmap v10414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:00:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:48.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:48.170920+0000) 2022-09-08T03:00:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:48.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:48.122622+0000) 2022-09-08T03:00:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:49.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:49.171162+0000) 2022-09-08T03:00:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:49.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:49.122927+0000) 2022-09-08T03:00:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:49 smithi055 ceph-mon[97033]: pgmap v10415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:00:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:49 smithi059 ceph-mon[97450]: pgmap v10415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:00:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:50.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:50.171400+0000) 2022-09-08T03:00:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:50.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:50.123200+0000) 2022-09-08T03:00:51.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:51.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:51.123392+0000) 2022-09-08T03:00:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:51.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:51.171566+0000) 2022-09-08T03:00:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:51 smithi059 ceph-mon[97450]: pgmap v10416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:00:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:51 smithi055 ceph-mon[97033]: pgmap v10416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:00:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:52.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:52.171815+0000) 2022-09-08T03:00:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:52.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:52.123674+0000) 2022-09-08T03:00:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:53.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:53.171978+0000) 2022-09-08T03:00:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:53.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:53.123842+0000) 2022-09-08T03:00:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:53 smithi059 ceph-mon[97450]: pgmap v10417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T03:00:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:53 smithi055 ceph-mon[97033]: pgmap v10417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T03:00:54.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:54.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:54.124062+0000) 2022-09-08T03:00:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:54.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:54.172197+0000) 2022-09-08T03:00:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:55.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:55.172458+0000) 2022-09-08T03:00:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:55.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:55.124235+0000) 2022-09-08T03:00:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:55 smithi059 ceph-mon[97450]: pgmap v10418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:00:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:55 smithi055 ceph-mon[97033]: pgmap v10418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:00:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:56.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:56.172744+0000) 2022-09-08T03:00:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:56.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:56.124442+0000) 2022-09-08T03:00:57.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:00:57.163+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:00:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:57.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:57.172876+0000) 2022-09-08T03:00:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:57.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:57.124734+0000) 2022-09-08T03:00:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:57 smithi059 ceph-mon[97450]: pgmap v10419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:00:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:57 smithi055 ceph-mon[97033]: pgmap v10419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:00:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:58.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:58.173147+0000) 2022-09-08T03:00:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:58.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:58.124884+0000) 2022-09-08T03:00:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:00:59.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:59.173376+0000) 2022-09-08T03:00:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:00:59.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:00:59.125113+0000) 2022-09-08T03:00:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:00:59 smithi055 ceph-mon[97033]: pgmap v10420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:00:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:00:59 smithi059 ceph-mon[97450]: pgmap v10420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:01:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:00.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:00.125362+0000) 2022-09-08T03:01:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:00.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:00.173596+0000) 2022-09-08T03:01:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:01.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:01.173681+0000) 2022-09-08T03:01:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:01.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:01.125572+0000) 2022-09-08T03:01:01.494 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:01 smithi055 ceph-mon[97033]: pgmap v10421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:01:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:01 smithi059 ceph-mon[97450]: pgmap v10421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:01:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:02.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:02.173924+0000) 2022-09-08T03:01:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:02.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:02.125812+0000) 2022-09-08T03:01:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:03.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:03.174136+0000) 2022-09-08T03:01:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:03.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:03.126025+0000) 2022-09-08T03:01:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:03 smithi059 ceph-mon[97450]: pgmap v10422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:01:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:03 smithi055 ceph-mon[97033]: pgmap v10422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:01:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:04.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:04.174339+0000) 2022-09-08T03:01:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:04.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:04.126248+0000) 2022-09-08T03:01:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:05.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:05.174584+0000) 2022-09-08T03:01:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:05.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:05.126432+0000) 2022-09-08T03:01:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:06.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:06.174796+0000) 2022-09-08T03:01:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:06.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:06.126715+0000) 2022-09-08T03:01:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:06 smithi059 ceph-mon[97450]: pgmap v10423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:01:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:01:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:01:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:06 smithi055 ceph-mon[97033]: pgmap v10423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:01:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:01:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:01:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:07.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:07.175060+0000) 2022-09-08T03:01:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:07.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:07.126959+0000) 2022-09-08T03:01:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:08.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:08.127209+0000) 2022-09-08T03:01:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:08.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:08.175336+0000) 2022-09-08T03:01:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:08 smithi059 ceph-mon[97450]: pgmap v10424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:01:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:08 smithi055 ceph-mon[97033]: pgmap v10424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:01:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:09.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:09.175548+0000) 2022-09-08T03:01:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:09.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:09.127462+0000) 2022-09-08T03:01:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:10.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:10.127719+0000) 2022-09-08T03:01:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:10.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:10.175732+0000) 2022-09-08T03:01:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:10 smithi055 ceph-mon[97033]: pgmap v10425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:01:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:10 smithi055 ceph-mon[97033]: osdmap e731: 12 total, 11 up, 11 in 2022-09-08T03:01:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:10 smithi059 ceph-mon[97450]: pgmap v10425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:01:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:10 smithi059 ceph-mon[97450]: osdmap e731: 12 total, 11 up, 11 in 2022-09-08T03:01:11.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:11.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:11.127907+0000) 2022-09-08T03:01:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:11.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:11.175956+0000) 2022-09-08T03:01:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:01:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:01:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:01:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:01:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:01:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:01:12.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:01:12.174+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:01:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:12.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:12.176167+0000) 2022-09-08T03:01:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:12.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:12.128150+0000) 2022-09-08T03:01:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:12 smithi055 ceph-mon[97033]: pgmap v10427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:01:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:01:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:12 smithi059 ceph-mon[97450]: pgmap v10427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:01:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:01:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:13.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:13.176369+0000) 2022-09-08T03:01:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:13.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:13.128394+0000) 2022-09-08T03:01:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:14.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:14.176572+0000) 2022-09-08T03:01:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:14.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:14.128597+0000) 2022-09-08T03:01:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:14 smithi055 ceph-mon[97033]: pgmap v10428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:01:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:14 smithi059 ceph-mon[97450]: pgmap v10428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:01:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:15.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:15.176786+0000) 2022-09-08T03:01:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:15.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:15.128808+0000) 2022-09-08T03:01:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:16.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:16.177004+0000) 2022-09-08T03:01:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:16.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:16.129022+0000) 2022-09-08T03:01:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:16 smithi055 ceph-mon[97033]: pgmap v10429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:01:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:16 smithi059 ceph-mon[97450]: pgmap v10429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:01:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:17.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:17.177199+0000) 2022-09-08T03:01:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:17.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:17.129280+0000) 2022-09-08T03:01:18.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:18.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:18.129551+0000) 2022-09-08T03:01:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:18.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:18.177423+0000) 2022-09-08T03:01:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:18 smithi055 ceph-mon[97033]: pgmap v10430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:01:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:18 smithi055 ceph-mon[97033]: osdmap e732: 12 total, 11 up, 11 in 2022-09-08T03:01:19.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:18 smithi059 ceph-mon[97450]: pgmap v10430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:01:19.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:18 smithi059 ceph-mon[97450]: osdmap e732: 12 total, 11 up, 11 in 2022-09-08T03:01:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:19.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:19.177586+0000) 2022-09-08T03:01:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:19.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:19.129728+0000) 2022-09-08T03:01:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:20.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:20.177786+0000) 2022-09-08T03:01:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:20.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:20.129877+0000) 2022-09-08T03:01:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:20 smithi055 ceph-mon[97033]: pgmap v10432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T03:01:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:20 smithi059 ceph-mon[97450]: pgmap v10432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T03:01:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:21.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:21.178026+0000) 2022-09-08T03:01:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:21.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:21.130080+0000) 2022-09-08T03:01:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:22.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:22.178273+0000) 2022-09-08T03:01:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:22.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:22.130320+0000) 2022-09-08T03:01:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:22 smithi055 ceph-mon[97033]: pgmap v10433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:01:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:22 smithi059 ceph-mon[97450]: pgmap v10433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:01:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:23.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:23.178445+0000) 2022-09-08T03:01:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:23.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:23.130569+0000) 2022-09-08T03:01:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:24.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:24.178649+0000) 2022-09-08T03:01:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:24.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:24.130788+0000) 2022-09-08T03:01:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:24 smithi055 ceph-mon[97033]: pgmap v10434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:01:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:24 smithi059 ceph-mon[97450]: pgmap v10434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:01:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:25.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:25.178819+0000) 2022-09-08T03:01:25.417 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:25.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:25.131010+0000) 2022-09-08T03:01:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:26.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:26.179055+0000) 2022-09-08T03:01:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:26.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:26.131244+0000) 2022-09-08T03:01:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:26 smithi055 ceph-mon[97033]: pgmap v10435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:01:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:26 smithi059 ceph-mon[97450]: pgmap v10435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:01:27.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:01:27.187+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:01:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:27.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:27.179306+0000) 2022-09-08T03:01:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:27.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:27.131433+0000) 2022-09-08T03:01:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:28.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:28.179589+0000) 2022-09-08T03:01:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:28.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:28.131683+0000) 2022-09-08T03:01:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:28 smithi055 ceph-mon[97033]: pgmap v10436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:01:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:28 smithi059 ceph-mon[97450]: pgmap v10436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:01:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:29.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:29.179901+0000) 2022-09-08T03:01:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:29.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:29.131890+0000) 2022-09-08T03:01:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:30.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:30.180127+0000) 2022-09-08T03:01:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:30.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:30.132157+0000) 2022-09-08T03:01:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:30 smithi055 ceph-mon[97033]: pgmap v10437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:01:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:30 smithi059 ceph-mon[97450]: pgmap v10437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:01:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:31.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:31.180388+0000) 2022-09-08T03:01:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:31.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:31.132370+0000) 2022-09-08T03:01:32.181 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:32.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:32.132610+0000) 2022-09-08T03:01:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:32.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:32.180672+0000) 2022-09-08T03:01:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:32 smithi055 ceph-mon[97033]: pgmap v10438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:01:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:32 smithi059 ceph-mon[97450]: pgmap v10438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:01:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:33.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:33.180906+0000) 2022-09-08T03:01:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:33.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:33.132898+0000) 2022-09-08T03:01:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:34.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:34.181171+0000) 2022-09-08T03:01:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:34.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:34.133128+0000) 2022-09-08T03:01:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:34 smithi059 ceph-mon[97450]: pgmap v10439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:01:35.240 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:34 smithi055 ceph-mon[97033]: pgmap v10439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:01:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:35.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:35.181388+0000) 2022-09-08T03:01:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:35.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:35.133361+0000) 2022-09-08T03:01:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:36.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:36.181594+0000) 2022-09-08T03:01:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:36.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:36.133559+0000) 2022-09-08T03:01:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:36 smithi059 ceph-mon[97450]: pgmap v10440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:01:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:36 smithi055 ceph-mon[97033]: pgmap v10440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:01:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:37.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:37.181823+0000) 2022-09-08T03:01:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:37.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:37.133733+0000) 2022-09-08T03:01:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:38.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:38.182055+0000) 2022-09-08T03:01:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:38.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:38.133914+0000) 2022-09-08T03:01:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:38 smithi059 ceph-mon[97450]: pgmap v10441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:01:39.239 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:38 smithi055 ceph-mon[97033]: pgmap v10441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:01:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:39.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:39.182266+0000) 2022-09-08T03:01:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:39.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:39.134154+0000) 2022-09-08T03:01:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:40.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:40.134390+0000) 2022-09-08T03:01:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:40.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:40.182478+0000) 2022-09-08T03:01:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:40 smithi059 ceph-mon[97450]: pgmap v10442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:01:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:40 smithi055 ceph-mon[97033]: pgmap v10442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:01:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:41.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:41.182612+0000) 2022-09-08T03:01:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:41.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:41.134588+0000) 2022-09-08T03:01:42.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:01:42.200+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:01:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:42.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:42.182860+0000) 2022-09-08T03:01:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:42.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:42.134820+0000) 2022-09-08T03:01:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:42 smithi059 ceph-mon[97450]: pgmap v10443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:01:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:42 smithi055 ceph-mon[97033]: pgmap v10443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:01:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:43.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:43.135038+0000) 2022-09-08T03:01:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:43.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:43.183093+0000) 2022-09-08T03:01:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:44.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:44.183360+0000) 2022-09-08T03:01:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:44.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:44.135276+0000) 2022-09-08T03:01:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:44 smithi059 ceph-mon[97450]: pgmap v10444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:01:45.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:44 smithi055 ceph-mon[97033]: pgmap v10444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:01:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:45.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:45.183581+0000) 2022-09-08T03:01:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:45.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:45.135455+0000) 2022-09-08T03:01:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:46.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:46.183762+0000) 2022-09-08T03:01:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:46.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:46.135722+0000) 2022-09-08T03:01:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:46 smithi059 ceph-mon[97450]: pgmap v10445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:01:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:46 smithi055 ceph-mon[97033]: pgmap v10445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:01:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:47.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:47.183991+0000) 2022-09-08T03:01:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:47.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:47.135917+0000) 2022-09-08T03:01:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:48.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:48.184204+0000) 2022-09-08T03:01:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:48.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:48.136193+0000) 2022-09-08T03:01:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:48 smithi059 ceph-mon[97450]: pgmap v10446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:01:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:48 smithi055 ceph-mon[97033]: pgmap v10446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:01:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:49.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:49.184437+0000) 2022-09-08T03:01:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:49.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:49.136411+0000) 2022-09-08T03:01:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:50.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:50.184672+0000) 2022-09-08T03:01:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:50.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:50.136678+0000) 2022-09-08T03:01:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:50 smithi059 ceph-mon[97450]: pgmap v10447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:01:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:50 smithi055 ceph-mon[97033]: pgmap v10447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:01:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:51.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:51.184846+0000) 2022-09-08T03:01:51.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:51.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:51.136917+0000) 2022-09-08T03:01:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:52.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:52.137132+0000) 2022-09-08T03:01:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:52.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:52.185060+0000) 2022-09-08T03:01:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:52 smithi059 ceph-mon[97450]: pgmap v10448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:01:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:52 smithi055 ceph-mon[97033]: pgmap v10448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:01:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:53.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:53.185220+0000) 2022-09-08T03:01:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:53.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:53.137340+0000) 2022-09-08T03:01:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:54.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:54.137580+0000) 2022-09-08T03:01:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:54.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:54.185470+0000) 2022-09-08T03:01:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:54 smithi059 ceph-mon[97450]: pgmap v10449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:01:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:54 smithi055 ceph-mon[97033]: pgmap v10449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:01:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:55.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:55.185619+0000) 2022-09-08T03:01:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:55.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:55.137777+0000) 2022-09-08T03:01:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:56.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:56.185763+0000) 2022-09-08T03:01:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:56.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:56.138011+0000) 2022-09-08T03:01:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:56 smithi059 ceph-mon[97450]: pgmap v10450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:01:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:56 smithi055 ceph-mon[97033]: pgmap v10450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:01:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:57.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:57.138245+0000) 2022-09-08T03:01:57.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:01:57.211+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:01:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:57.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:57.185954+0000) 2022-09-08T03:01:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:58.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:58.186167+0000) 2022-09-08T03:01:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:58.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:58.138497+0000) 2022-09-08T03:01:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:01:58 smithi055 ceph-mon[97033]: pgmap v10451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:01:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:01:58 smithi059 ceph-mon[97450]: pgmap v10451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:01:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:01:59.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:59.186387+0000) 2022-09-08T03:01:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:01:59.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:01:59.138731+0000) 2022-09-08T03:02:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:00.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:00.186622+0000) 2022-09-08T03:02:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:00.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:00.138979+0000) 2022-09-08T03:02:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:00 smithi055 ceph-mon[97033]: pgmap v10452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:00 smithi059 ceph-mon[97450]: pgmap v10452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:01.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:01.186870+0000) 2022-09-08T03:02:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:01.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:01.139133+0000) 2022-09-08T03:02:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:02.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:02.187111+0000) 2022-09-08T03:02:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:02.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:02.139338+0000) 2022-09-08T03:02:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:02 smithi055 ceph-mon[97033]: pgmap v10453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:02:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:02 smithi059 ceph-mon[97450]: pgmap v10453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:02:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:03.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:03.187298+0000) 2022-09-08T03:02:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:03.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:03.139584+0000) 2022-09-08T03:02:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:04.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:04.187450+0000) 2022-09-08T03:02:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:04.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:04.139809+0000) 2022-09-08T03:02:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:04 smithi055 ceph-mon[97033]: pgmap v10454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:04 smithi059 ceph-mon[97450]: pgmap v10454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:05.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:05.187698+0000) 2022-09-08T03:02:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:05.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:05.140006+0000) 2022-09-08T03:02:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:02:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:02:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:02:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:02:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:06.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:06.187857+0000) 2022-09-08T03:02:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:06.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:06.140239+0000) 2022-09-08T03:02:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:06 smithi055 ceph-mon[97033]: pgmap v10455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:02:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:06 smithi059 ceph-mon[97450]: pgmap v10455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:02:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:07.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:07.188001+0000) 2022-09-08T03:02:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:07.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:07.140486+0000) 2022-09-08T03:02:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:08.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:08.140730+0000) 2022-09-08T03:02:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:08.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:08.188186+0000) 2022-09-08T03:02:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:08 smithi055 ceph-mon[97033]: pgmap v10456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:02:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:08 smithi059 ceph-mon[97450]: pgmap v10456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:02:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:09.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:09.188363+0000) 2022-09-08T03:02:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:09.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:09.140884+0000) 2022-09-08T03:02:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:10.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:10.188549+0000) 2022-09-08T03:02:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:10.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:10.141125+0000) 2022-09-08T03:02:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:10 smithi055 ceph-mon[97033]: pgmap v10457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:02:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:10 smithi055 ceph-mon[97033]: osdmap e733: 12 total, 11 up, 11 in 2022-09-08T03:02:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:10 smithi059 ceph-mon[97450]: pgmap v10457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:02:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:10 smithi059 ceph-mon[97450]: osdmap e733: 12 total, 11 up, 11 in 2022-09-08T03:02:11.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:11.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:11.141337+0000) 2022-09-08T03:02:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:11.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:11.188741+0000) 2022-09-08T03:02:12.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:02:12.222+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:02:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:12.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:12.188889+0000) 2022-09-08T03:02:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:12.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:12.141560+0000) 2022-09-08T03:02:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:12 smithi055 ceph-mon[97033]: pgmap v10459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:02:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:02:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:02:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:02:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:02:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:02:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:12 smithi059 ceph-mon[97450]: pgmap v10459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:02:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:02:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:02:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:02:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:02:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:02:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:13.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:13.141789+0000) 2022-09-08T03:02:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:13.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:13.189012+0000) 2022-09-08T03:02:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:02:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:13 smithi055 ceph-mon[97033]: osdmap e734: 12 total, 11 up, 11 in 2022-09-08T03:02:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:02:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:13 smithi059 ceph-mon[97450]: osdmap e734: 12 total, 11 up, 11 in 2022-09-08T03:02:14.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:14.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:14.141994+0000) 2022-09-08T03:02:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:14.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:14.189198+0000) 2022-09-08T03:02:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:14 smithi055 ceph-mon[97033]: pgmap v10460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:02:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:14 smithi059 ceph-mon[97450]: pgmap v10460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:02:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:15.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:15.189359+0000) 2022-09-08T03:02:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:15.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:15.142230+0000) 2022-09-08T03:02:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:16.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:16.189497+0000) 2022-09-08T03:02:16.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:16.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:16.142370+0000) 2022-09-08T03:02:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:16 smithi055 ceph-mon[97033]: pgmap v10462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:02:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:16 smithi059 ceph-mon[97450]: pgmap v10462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:02:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:17.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:17.189694+0000) 2022-09-08T03:02:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:17.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:17.142607+0000) 2022-09-08T03:02:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:18.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:18.189959+0000) 2022-09-08T03:02:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:18.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:18.142795+0000) 2022-09-08T03:02:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:18 smithi055 ceph-mon[97033]: pgmap v10463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:02:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:18 smithi059 ceph-mon[97450]: pgmap v10463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:02:19.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:19.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:19.143028+0000) 2022-09-08T03:02:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:19.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:19.190174+0000) 2022-09-08T03:02:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:20.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:20.143199+0000) 2022-09-08T03:02:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:20.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:20.190338+0000) 2022-09-08T03:02:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:20 smithi055 ceph-mon[97033]: pgmap v10464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T03:02:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:20 smithi059 ceph-mon[97450]: pgmap v10464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T03:02:21.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:21.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:21.143353+0000) 2022-09-08T03:02:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:21.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:21.190544+0000) 2022-09-08T03:02:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:22.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:22.190702+0000) 2022-09-08T03:02:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:22.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:22.143601+0000) 2022-09-08T03:02:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:22 smithi055 ceph-mon[97033]: pgmap v10465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:02:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:22 smithi059 ceph-mon[97450]: pgmap v10465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:02:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:23.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:23.190885+0000) 2022-09-08T03:02:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:23.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:23.143842+0000) 2022-09-08T03:02:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:24.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:24.191116+0000) 2022-09-08T03:02:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:24.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:24.144052+0000) 2022-09-08T03:02:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:24 smithi055 ceph-mon[97033]: pgmap v10466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:02:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:24 smithi059 ceph-mon[97450]: pgmap v10466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:02:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:25.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:25.144262+0000) 2022-09-08T03:02:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:25.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:25.191272+0000) 2022-09-08T03:02:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:26.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:26.191498+0000) 2022-09-08T03:02:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:26.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:26.144469+0000) 2022-09-08T03:02:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:26 smithi055 ceph-mon[97033]: pgmap v10467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T03:02:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:26 smithi059 ceph-mon[97450]: pgmap v10467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T03:02:27.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:02:27.236+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:02:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:27.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:27.191721+0000) 2022-09-08T03:02:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:27.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:27.144740+0000) 2022-09-08T03:02:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:28.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:28.191923+0000) 2022-09-08T03:02:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:28.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:28.144931+0000) 2022-09-08T03:02:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:28 smithi055 ceph-mon[97033]: pgmap v10468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:28 smithi059 ceph-mon[97450]: pgmap v10468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:29.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:29.192102+0000) 2022-09-08T03:02:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:29.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:29.145169+0000) 2022-09-08T03:02:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:30.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:30.192275+0000) 2022-09-08T03:02:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:30.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:30.145323+0000) 2022-09-08T03:02:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:30 smithi055 ceph-mon[97033]: pgmap v10469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:30 smithi059 ceph-mon[97450]: pgmap v10469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:31.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:31.192447+0000) 2022-09-08T03:02:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:31.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:31.145507+0000) 2022-09-08T03:02:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:32.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:32.192705+0000) 2022-09-08T03:02:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:32.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:32.145741+0000) 2022-09-08T03:02:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:32 smithi055 ceph-mon[97033]: pgmap v10470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:02:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:32 smithi059 ceph-mon[97450]: pgmap v10470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:02:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:33.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:33.192946+0000) 2022-09-08T03:02:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:33.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:33.145925+0000) 2022-09-08T03:02:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:34.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:34.193188+0000) 2022-09-08T03:02:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:34.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:34.146158+0000) 2022-09-08T03:02:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:34 smithi055 ceph-mon[97033]: pgmap v10471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:34 smithi059 ceph-mon[97450]: pgmap v10471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:35.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:35.193431+0000) 2022-09-08T03:02:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:35.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:35.146442+0000) 2022-09-08T03:02:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:36.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:36.193709+0000) 2022-09-08T03:02:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:36.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:36.146673+0000) 2022-09-08T03:02:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:36 smithi055 ceph-mon[97033]: pgmap v10472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:02:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:36 smithi059 ceph-mon[97450]: pgmap v10472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:02:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:37.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:37.193948+0000) 2022-09-08T03:02:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:37.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:37.146907+0000) 2022-09-08T03:02:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:38.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:38.194197+0000) 2022-09-08T03:02:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:38.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:38.147156+0000) 2022-09-08T03:02:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:38 smithi055 ceph-mon[97033]: pgmap v10473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:38 smithi059 ceph-mon[97450]: pgmap v10473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:39.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:39.194412+0000) 2022-09-08T03:02:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:39.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:39.147384+0000) 2022-09-08T03:02:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:40.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:40.147607+0000) 2022-09-08T03:02:40.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:40.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:40.194666+0000) 2022-09-08T03:02:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:40 smithi055 ceph-mon[97033]: pgmap v10474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:40 smithi059 ceph-mon[97450]: pgmap v10474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:41.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:41.194909+0000) 2022-09-08T03:02:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:41.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:41.147864+0000) 2022-09-08T03:02:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:42.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:42.195126+0000) 2022-09-08T03:02:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:42.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:42.148113+0000) 2022-09-08T03:02:42.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:02:42.237+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:02:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:42 smithi055 ceph-mon[97033]: pgmap v10475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:02:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:42 smithi059 ceph-mon[97450]: pgmap v10475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:02:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:43.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:43.195265+0000) 2022-09-08T03:02:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:43.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:43.148331+0000) 2022-09-08T03:02:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:44.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:44.195487+0000) 2022-09-08T03:02:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:44.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:44.148563+0000) 2022-09-08T03:02:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:44 smithi055 ceph-mon[97033]: pgmap v10476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:44 smithi059 ceph-mon[97450]: pgmap v10476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:45.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:45.195698+0000) 2022-09-08T03:02:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:45.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:45.148803+0000) 2022-09-08T03:02:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:46.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:46.195913+0000) 2022-09-08T03:02:46.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:46.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:46.148984+0000) 2022-09-08T03:02:47.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:46 smithi055 ceph-mon[97033]: pgmap v10477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:02:47.293 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:46 smithi059 ceph-mon[97450]: pgmap v10477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:02:47.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:47.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:47.196134+0000) 2022-09-08T03:02:47.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:47.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:47.149220+0000) 2022-09-08T03:02:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:48.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:48.196332+0000) 2022-09-08T03:02:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:48.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:48.149423+0000) 2022-09-08T03:02:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:48 smithi055 ceph-mon[97033]: pgmap v10478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:48 smithi059 ceph-mon[97450]: pgmap v10478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:49.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:49.196559+0000) 2022-09-08T03:02:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:49.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:49.149598+0000) 2022-09-08T03:02:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:50.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:50.196818+0000) 2022-09-08T03:02:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:50.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:50.149841+0000) 2022-09-08T03:02:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:50 smithi055 ceph-mon[97033]: pgmap v10479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:50 smithi059 ceph-mon[97450]: pgmap v10479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:51.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:51.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:51.196935+0000) 2022-09-08T03:02:51.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:51.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:51.150046+0000) 2022-09-08T03:02:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:52.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:52.197174+0000) 2022-09-08T03:02:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:52.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:52.150227+0000) 2022-09-08T03:02:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:52 smithi055 ceph-mon[97033]: pgmap v10480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:02:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:52 smithi059 ceph-mon[97450]: pgmap v10480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:02:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:53.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:53.197445+0000) 2022-09-08T03:02:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:53.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:53.150476+0000) 2022-09-08T03:02:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:54.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:54.197680+0000) 2022-09-08T03:02:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:54.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:54.150700+0000) 2022-09-08T03:02:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:54 smithi055 ceph-mon[97033]: pgmap v10481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:54 smithi059 ceph-mon[97450]: pgmap v10481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:55.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:55.197857+0000) 2022-09-08T03:02:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:55.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:55.150867+0000) 2022-09-08T03:02:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:56.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:56.197987+0000) 2022-09-08T03:02:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:56.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:56.151010+0000) 2022-09-08T03:02:57.199 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:57.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:57.151152+0000) 2022-09-08T03:02:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:56 smithi055 ceph-mon[97033]: pgmap v10482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:02:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:56 smithi059 ceph-mon[97450]: pgmap v10482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:02:57.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:02:57.247+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:02:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:57.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:57.198117+0000) 2022-09-08T03:02:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:58.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:58.198357+0000) 2022-09-08T03:02:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:58.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:58.151355+0000) 2022-09-08T03:02:59.199 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:02:59.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:59.151587+0000) 2022-09-08T03:02:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:02:58 smithi055 ceph-mon[97033]: pgmap v10483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:02:58 smithi059 ceph-mon[97450]: pgmap v10483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:02:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:02:59.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:02:59.198573+0000) 2022-09-08T03:03:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:00.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:00.198735+0000) 2022-09-08T03:03:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:00.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:00.151695+0000) 2022-09-08T03:03:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:00 smithi055 ceph-mon[97033]: pgmap v10484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:00 smithi059 ceph-mon[97450]: pgmap v10484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:01.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:01.198957+0000) 2022-09-08T03:03:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:01.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:01.151855+0000) 2022-09-08T03:03:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:02.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:02.199188+0000) 2022-09-08T03:03:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:02.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:02.152103+0000) 2022-09-08T03:03:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:02 smithi055 ceph-mon[97033]: pgmap v10485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:03:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:02 smithi059 ceph-mon[97450]: pgmap v10485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:03:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:03.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:03.199434+0000) 2022-09-08T03:03:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:03.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:03.152289+0000) 2022-09-08T03:03:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:04.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:04.199706+0000) 2022-09-08T03:03:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:04.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:04.152506+0000) 2022-09-08T03:03:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:04 smithi055 ceph-mon[97033]: pgmap v10486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:04 smithi059 ceph-mon[97450]: pgmap v10486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:05.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:05.199970+0000) 2022-09-08T03:03:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:05.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:05.152734+0000) 2022-09-08T03:03:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:03:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:03:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:03:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:03:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:06.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:06.200215+0000) 2022-09-08T03:03:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:06.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:06.152939+0000) 2022-09-08T03:03:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:06 smithi055 ceph-mon[97033]: pgmap v10487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:03:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:06 smithi059 ceph-mon[97450]: pgmap v10487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:03:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:07.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:07.200341+0000) 2022-09-08T03:03:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:07.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:07.153195+0000) 2022-09-08T03:03:08.201 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:08.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:08.153403+0000) 2022-09-08T03:03:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:08.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:08.200581+0000) 2022-09-08T03:03:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:08 smithi055 ceph-mon[97033]: pgmap v10488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:03:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:08 smithi059 ceph-mon[97450]: pgmap v10488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:03:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:09.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:09.200764+0000) 2022-09-08T03:03:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:09.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:09.153650+0000) 2022-09-08T03:03:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:10.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:10.201012+0000) 2022-09-08T03:03:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:10.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:10.153804+0000) 2022-09-08T03:03:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:10 smithi055 ceph-mon[97033]: pgmap v10489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:03:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:10 smithi055 ceph-mon[97033]: osdmap e735: 12 total, 11 up, 11 in 2022-09-08T03:03:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:10 smithi059 ceph-mon[97450]: pgmap v10489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:03:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:10 smithi059 ceph-mon[97450]: osdmap e735: 12 total, 11 up, 11 in 2022-09-08T03:03:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:11.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:11.201275+0000) 2022-09-08T03:03:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:11.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:11.153974+0000) 2022-09-08T03:03:12.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:03:12.255+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:03:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:12.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:12.154206+0000) 2022-09-08T03:03:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:12.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:12.201511+0000) 2022-09-08T03:03:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:12 smithi055 ceph-mon[97033]: pgmap v10491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:03:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:12 smithi059 ceph-mon[97450]: pgmap v10491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:03:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:13.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:13.201774+0000) 2022-09-08T03:03:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:13.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:13.154349+0000) 2022-09-08T03:03:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:03:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:03:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:03:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:13 smithi055 ceph-mon[97033]: osdmap e736: 12 total, 11 up, 11 in 2022-09-08T03:03:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:03:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:03:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:03:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:13 smithi059 ceph-mon[97450]: osdmap e736: 12 total, 11 up, 11 in 2022-09-08T03:03:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:14.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:14.202011+0000) 2022-09-08T03:03:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:14.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:14.154578+0000) 2022-09-08T03:03:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:14 smithi055 ceph-mon[97033]: pgmap v10492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:03:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:03:14.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:14 smithi059 ceph-mon[97450]: pgmap v10492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:03:14.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:03:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:15.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:15.202216+0000) 2022-09-08T03:03:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:15.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:15.154758+0000) 2022-09-08T03:03:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:16.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:16.154971+0000) 2022-09-08T03:03:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:16.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:16.202405+0000) 2022-09-08T03:03:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:16 smithi055 ceph-mon[97033]: pgmap v10494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:03:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:16 smithi059 ceph-mon[97450]: pgmap v10494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:03:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:17.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:17.155222+0000) 2022-09-08T03:03:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:17.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:17.202673+0000) 2022-09-08T03:03:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:18.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:18.202854+0000) 2022-09-08T03:03:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:18.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:18.155445+0000) 2022-09-08T03:03:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:18 smithi055 ceph-mon[97033]: pgmap v10495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:03:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:18 smithi059 ceph-mon[97450]: pgmap v10495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:03:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:19.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:19.155712+0000) 2022-09-08T03:03:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:19.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:19.203056+0000) 2022-09-08T03:03:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:20.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:20.203203+0000) 2022-09-08T03:03:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:20.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:20.155862+0000) 2022-09-08T03:03:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:20 smithi055 ceph-mon[97033]: pgmap v10496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T03:03:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:20 smithi059 ceph-mon[97450]: pgmap v10496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T03:03:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:21.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:21.203344+0000) 2022-09-08T03:03:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:21.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:21.156002+0000) 2022-09-08T03:03:22.204 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:22.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:22.156241+0000) 2022-09-08T03:03:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:22.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:22.203555+0000) 2022-09-08T03:03:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:22 smithi055 ceph-mon[97033]: pgmap v10497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:03:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:22 smithi059 ceph-mon[97450]: pgmap v10497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:03:23.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:23.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:23.203742+0000) 2022-09-08T03:03:23.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:23.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:23.156424+0000) 2022-09-08T03:03:24.204 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:24.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:24.156693+0000) 2022-09-08T03:03:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:24.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:24.203913+0000) 2022-09-08T03:03:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:24 smithi055 ceph-mon[97033]: pgmap v10498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:03:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:24 smithi059 ceph-mon[97450]: pgmap v10498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:03:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:25.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:25.204106+0000) 2022-09-08T03:03:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:25.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:25.156870+0000) 2022-09-08T03:03:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:26.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:26.204236+0000) 2022-09-08T03:03:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:26.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:26.157102+0000) 2022-09-08T03:03:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:26 smithi055 ceph-mon[97033]: pgmap v10499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T03:03:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:26 smithi059 ceph-mon[97450]: pgmap v10499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T03:03:27.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:03:27.261+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:03:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:27.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:27.204490+0000) 2022-09-08T03:03:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:27.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:27.157370+0000) 2022-09-08T03:03:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:28.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:28.204768+0000) 2022-09-08T03:03:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:28.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:28.157595+0000) 2022-09-08T03:03:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:28 smithi055 ceph-mon[97033]: pgmap v10500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:28.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:28 smithi059 ceph-mon[97450]: pgmap v10500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:29.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:29.204927+0000) 2022-09-08T03:03:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:29.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:29.157827+0000) 2022-09-08T03:03:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:30.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:30.205156+0000) 2022-09-08T03:03:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:30.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:30.158058+0000) 2022-09-08T03:03:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:30 smithi055 ceph-mon[97033]: pgmap v10501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:30 smithi059 ceph-mon[97450]: pgmap v10501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:31.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:31.205382+0000) 2022-09-08T03:03:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:31.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:31.158241+0000) 2022-09-08T03:03:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:32.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:32.205616+0000) 2022-09-08T03:03:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:32.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:32.158454+0000) 2022-09-08T03:03:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:32 smithi055 ceph-mon[97033]: pgmap v10502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:03:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:32 smithi059 ceph-mon[97450]: pgmap v10502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:03:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:33.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:33.205886+0000) 2022-09-08T03:03:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:33.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:33.158632+0000) 2022-09-08T03:03:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:34.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:34.206084+0000) 2022-09-08T03:03:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:34.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:34.158783+0000) 2022-09-08T03:03:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:34 smithi055 ceph-mon[97033]: pgmap v10503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:34 smithi059 ceph-mon[97450]: pgmap v10503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:35.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:35.206286+0000) 2022-09-08T03:03:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:35.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:35.158908+0000) 2022-09-08T03:03:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:36.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:36.159128+0000) 2022-09-08T03:03:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:36.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:36.206494+0000) 2022-09-08T03:03:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:36 smithi055 ceph-mon[97033]: pgmap v10504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:03:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:36 smithi059 ceph-mon[97450]: pgmap v10504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:03:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:37.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:37.206753+0000) 2022-09-08T03:03:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:37.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:37.159364+0000) 2022-09-08T03:03:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:38.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:38.206983+0000) 2022-09-08T03:03:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:38.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:38.159627+0000) 2022-09-08T03:03:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:38 smithi055 ceph-mon[97033]: pgmap v10505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:38 smithi059 ceph-mon[97450]: pgmap v10505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:39.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:39.207222+0000) 2022-09-08T03:03:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:39.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:39.159909+0000) 2022-09-08T03:03:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:40.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:40.207430+0000) 2022-09-08T03:03:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:40.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:40.160132+0000) 2022-09-08T03:03:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:40 smithi055 ceph-mon[97033]: pgmap v10506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:40 smithi059 ceph-mon[97450]: pgmap v10506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:41.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:41.207724+0000) 2022-09-08T03:03:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:41.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:41.160385+0000) 2022-09-08T03:03:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:42.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:42.207900+0000) 2022-09-08T03:03:42.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:03:42.273+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:03:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:42.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:42.160596+0000) 2022-09-08T03:03:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:42 smithi055 ceph-mon[97033]: pgmap v10507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:03:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:42 smithi059 ceph-mon[97450]: pgmap v10507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:03:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:43.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:43.208135+0000) 2022-09-08T03:03:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:43.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:43.160829+0000) 2022-09-08T03:03:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:44.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:44.208316+0000) 2022-09-08T03:03:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:44.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:44.161033+0000) 2022-09-08T03:03:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:44 smithi055 ceph-mon[97033]: pgmap v10508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:44 smithi059 ceph-mon[97450]: pgmap v10508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:45.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:45.208466+0000) 2022-09-08T03:03:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:45.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:45.161173+0000) 2022-09-08T03:03:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:46.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:46.208592+0000) 2022-09-08T03:03:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:46.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:46.161324+0000) 2022-09-08T03:03:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:46 smithi055 ceph-mon[97033]: pgmap v10509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:03:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:46 smithi059 ceph-mon[97450]: pgmap v10509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:03:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:47.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:47.208781+0000) 2022-09-08T03:03:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:47.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:47.161545+0000) 2022-09-08T03:03:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:48.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:48.209003+0000) 2022-09-08T03:03:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:48.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:48.161785+0000) 2022-09-08T03:03:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:48 smithi055 ceph-mon[97033]: pgmap v10510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:48 smithi059 ceph-mon[97450]: pgmap v10510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:49.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:49.161961+0000) 2022-09-08T03:03:49.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:49.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:49.209237+0000) 2022-09-08T03:03:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:50.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:50.209381+0000) 2022-09-08T03:03:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:50.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:50.162186+0000) 2022-09-08T03:03:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:50 smithi059 ceph-mon[97450]: pgmap v10511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:50 smithi055 ceph-mon[97033]: pgmap v10511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:51.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:51.209504+0000) 2022-09-08T03:03:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:51.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:51.162402+0000) 2022-09-08T03:03:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:52.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:52.162628+0000) 2022-09-08T03:03:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:52.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:52.209657+0000) 2022-09-08T03:03:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:52 smithi059 ceph-mon[97450]: pgmap v10512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:03:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:52 smithi055 ceph-mon[97033]: pgmap v10512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:03:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:53.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:53.162815+0000) 2022-09-08T03:03:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:53.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:53.209784+0000) 2022-09-08T03:03:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:54.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:54.209933+0000) 2022-09-08T03:03:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:54.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:54.162996+0000) 2022-09-08T03:03:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:54 smithi059 ceph-mon[97450]: pgmap v10513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:54 smithi055 ceph-mon[97033]: pgmap v10513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:55.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:55.210103+0000) 2022-09-08T03:03:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:55.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:55.163199+0000) 2022-09-08T03:03:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:56.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:56.210254+0000) 2022-09-08T03:03:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:56.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:56.163397+0000) 2022-09-08T03:03:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:56 smithi059 ceph-mon[97450]: pgmap v10514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:03:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:56 smithi055 ceph-mon[97033]: pgmap v10514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:03:57.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:03:57.280+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:03:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:57.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:57.210439+0000) 2022-09-08T03:03:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:57.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:57.163625+0000) 2022-09-08T03:03:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:58.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:58.210583+0000) 2022-09-08T03:03:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:58.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:58.163885+0000) 2022-09-08T03:03:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:03:58 smithi059 ceph-mon[97450]: pgmap v10515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:03:58 smithi055 ceph-mon[97033]: pgmap v10515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:03:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:03:59.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:59.210792+0000) 2022-09-08T03:03:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:03:59.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:03:59.164134+0000) 2022-09-08T03:04:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:00.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:00.211033+0000) 2022-09-08T03:04:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:00.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:00.164333+0000) 2022-09-08T03:04:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:00 smithi059 ceph-mon[97450]: pgmap v10516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:00 smithi055 ceph-mon[97033]: pgmap v10516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:01.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:01.211158+0000) 2022-09-08T03:04:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:01.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:01.164499+0000) 2022-09-08T03:04:02.612 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:02.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:02.211308+0000) 2022-09-08T03:04:02.613 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:02.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:02.164657+0000) 2022-09-08T03:04:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:02 smithi059 ceph-mon[97450]: pgmap v10517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:04:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:02 smithi055 ceph-mon[97033]: pgmap v10517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:04:03.440 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:03.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:03.164791+0000) 2022-09-08T03:04:03.441 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:03.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:03.211467+0000) 2022-09-08T03:04:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:04.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:04.211733+0000) 2022-09-08T03:04:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:04.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:04.165052+0000) 2022-09-08T03:04:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:04 smithi059 ceph-mon[97450]: pgmap v10518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:04 smithi055 ceph-mon[97033]: pgmap v10518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:05.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:05.211903+0000) 2022-09-08T03:04:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:05.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:05.165241+0000) 2022-09-08T03:04:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:04:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:04:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:06.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:06.212086+0000) 2022-09-08T03:04:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:06.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:06.165472+0000) 2022-09-08T03:04:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:06 smithi059 ceph-mon[97450]: pgmap v10519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:04:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:04:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:06 smithi055 ceph-mon[97033]: pgmap v10519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:04:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:04:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:07.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:07.212294+0000) 2022-09-08T03:04:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:07.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:07.165714+0000) 2022-09-08T03:04:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:08.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:08.212439+0000) 2022-09-08T03:04:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:08.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:08.165853+0000) 2022-09-08T03:04:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:08 smithi059 ceph-mon[97450]: pgmap v10520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:04:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:08 smithi055 ceph-mon[97033]: pgmap v10520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:04:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:09.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:09.212628+0000) 2022-09-08T03:04:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:09.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:09.165990+0000) 2022-09-08T03:04:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:10.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:10.166214+0000) 2022-09-08T03:04:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:10.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:10.212776+0000) 2022-09-08T03:04:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:10 smithi059 ceph-mon[97450]: pgmap v10521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:04:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:10 smithi059 ceph-mon[97450]: osdmap e737: 12 total, 11 up, 11 in 2022-09-08T03:04:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:10 smithi055 ceph-mon[97033]: pgmap v10521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:04:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:10 smithi055 ceph-mon[97033]: osdmap e737: 12 total, 11 up, 11 in 2022-09-08T03:04:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:11.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:11.166372+0000) 2022-09-08T03:04:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:11.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:11.212914+0000) 2022-09-08T03:04:12.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:04:12.285+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:04:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:12.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:12.213116+0000) 2022-09-08T03:04:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:12.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:12.166580+0000) 2022-09-08T03:04:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:12 smithi059 ceph-mon[97450]: pgmap v10523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:04:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:12 smithi055 ceph-mon[97033]: pgmap v10523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:04:13.427 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:13.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:13.213348+0000) 2022-09-08T03:04:13.428 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:13.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:13.166768+0000) 2022-09-08T03:04:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:14.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:14.213512+0000) 2022-09-08T03:04:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:14.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:14.166955+0000) 2022-09-08T03:04:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:14 smithi055 ceph-mon[97033]: pgmap v10524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:04:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:14 smithi055 ceph-mon[97033]: osdmap e738: 12 total, 11 up, 11 in 2022-09-08T03:04:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:04:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:04:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:04:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:04:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:14 smithi059 ceph-mon[97450]: pgmap v10524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:04:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:14 smithi059 ceph-mon[97450]: osdmap e738: 12 total, 11 up, 11 in 2022-09-08T03:04:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:04:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:04:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:04:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:04:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:15.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:15.213676+0000) 2022-09-08T03:04:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:15.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:15.167160+0000) 2022-09-08T03:04:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:16.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:16.213798+0000) 2022-09-08T03:04:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:16.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:16.167400+0000) 2022-09-08T03:04:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:16 smithi055 ceph-mon[97033]: pgmap v10526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:04:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:16 smithi059 ceph-mon[97450]: pgmap v10526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:04:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:17.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:17.213922+0000) 2022-09-08T03:04:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:17.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:17.167606+0000) 2022-09-08T03:04:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:18.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:18.214054+0000) 2022-09-08T03:04:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:18.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:18.167863+0000) 2022-09-08T03:04:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:18 smithi055 ceph-mon[97033]: pgmap v10527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:04:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:18 smithi059 ceph-mon[97450]: pgmap v10527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:04:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:19.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:19.214257+0000) 2022-09-08T03:04:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:19.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:19.168083+0000) 2022-09-08T03:04:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:20.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:20.214469+0000) 2022-09-08T03:04:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:20.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:20.168261+0000) 2022-09-08T03:04:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:20 smithi055 ceph-mon[97033]: pgmap v10528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 210 B/s wr, 6 op/s 2022-09-08T03:04:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:20 smithi059 ceph-mon[97450]: pgmap v10528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 210 B/s wr, 6 op/s 2022-09-08T03:04:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:21.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:21.214616+0000) 2022-09-08T03:04:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:21.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:21.168520+0000) 2022-09-08T03:04:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:22.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:22.214788+0000) 2022-09-08T03:04:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:22.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:22.168774+0000) 2022-09-08T03:04:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:22 smithi055 ceph-mon[97033]: pgmap v10529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:04:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:22 smithi059 ceph-mon[97450]: pgmap v10529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:04:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:23.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:23.215001+0000) 2022-09-08T03:04:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:23.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:23.169029+0000) 2022-09-08T03:04:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:24.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:24.215175+0000) 2022-09-08T03:04:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:24.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:24.169254+0000) 2022-09-08T03:04:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:24 smithi055 ceph-mon[97033]: pgmap v10530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:04:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:24 smithi059 ceph-mon[97450]: pgmap v10530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:04:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:25.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:25.215326+0000) 2022-09-08T03:04:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:25.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:25.169474+0000) 2022-09-08T03:04:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:26.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:26.215448+0000) 2022-09-08T03:04:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:26.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:26.169653+0000) 2022-09-08T03:04:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:26 smithi055 ceph-mon[97033]: pgmap v10531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T03:04:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:26 smithi059 ceph-mon[97450]: pgmap v10531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T03:04:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:27.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:27.215581+0000) 2022-09-08T03:04:27.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:04:27.294+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:04:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:27.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:27.169896+0000) 2022-09-08T03:04:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:28.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:28.215747+0000) 2022-09-08T03:04:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:28.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:28.170198+0000) 2022-09-08T03:04:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:28 smithi055 ceph-mon[97033]: pgmap v10532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:28 smithi059 ceph-mon[97450]: pgmap v10532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:29.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:29.215912+0000) 2022-09-08T03:04:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:29.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:29.170440+0000) 2022-09-08T03:04:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:30.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:30.216135+0000) 2022-09-08T03:04:30.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:30.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:30.170679+0000) 2022-09-08T03:04:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:30 smithi055 ceph-mon[97033]: pgmap v10533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:30 smithi059 ceph-mon[97450]: pgmap v10533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:31.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:31.216259+0000) 2022-09-08T03:04:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:31.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:31.170899+0000) 2022-09-08T03:04:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:32.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:32.216425+0000) 2022-09-08T03:04:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:32.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:32.171139+0000) 2022-09-08T03:04:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:32 smithi055 ceph-mon[97033]: pgmap v10534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:04:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:32 smithi059 ceph-mon[97450]: pgmap v10534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:04:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:33.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:33.216668+0000) 2022-09-08T03:04:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:33.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:33.171397+0000) 2022-09-08T03:04:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:34.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:34.216836+0000) 2022-09-08T03:04:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:34.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:34.171650+0000) 2022-09-08T03:04:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:34 smithi055 ceph-mon[97033]: pgmap v10535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:34 smithi059 ceph-mon[97450]: pgmap v10535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:35.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:35.217080+0000) 2022-09-08T03:04:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:35.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:35.171847+0000) 2022-09-08T03:04:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:36.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:36.217254+0000) 2022-09-08T03:04:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:36.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:36.171985+0000) 2022-09-08T03:04:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:36 smithi055 ceph-mon[97033]: pgmap v10536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:04:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:36 smithi059 ceph-mon[97450]: pgmap v10536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:04:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:37.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:37.217412+0000) 2022-09-08T03:04:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:37.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:37.172187+0000) 2022-09-08T03:04:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:38.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:38.217575+0000) 2022-09-08T03:04:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:38.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:38.172446+0000) 2022-09-08T03:04:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:38 smithi055 ceph-mon[97033]: pgmap v10537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:38 smithi059 ceph-mon[97450]: pgmap v10537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:39.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:39.217731+0000) 2022-09-08T03:04:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:39.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:39.172607+0000) 2022-09-08T03:04:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:40.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:40.217890+0000) 2022-09-08T03:04:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:40.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:40.172790+0000) 2022-09-08T03:04:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:40 smithi055 ceph-mon[97033]: pgmap v10538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:40 smithi059 ceph-mon[97450]: pgmap v10538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:41.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:41.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:41.218041+0000) 2022-09-08T03:04:41.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:41.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:41.173023+0000) 2022-09-08T03:04:42.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:04:42.310+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:04:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:42.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:42.173190+0000) 2022-09-08T03:04:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:42.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:42.218196+0000) 2022-09-08T03:04:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:42 smithi055 ceph-mon[97033]: pgmap v10539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:04:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:42 smithi059 ceph-mon[97450]: pgmap v10539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:04:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:43.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:43.218344+0000) 2022-09-08T03:04:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:43.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:43.173443+0000) 2022-09-08T03:04:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:44.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:44.173716+0000) 2022-09-08T03:04:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:44.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:44.218569+0000) 2022-09-08T03:04:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:44 smithi055 ceph-mon[97033]: pgmap v10540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:44 smithi059 ceph-mon[97450]: pgmap v10540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:45.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:45.218761+0000) 2022-09-08T03:04:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:45.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:45.173911+0000) 2022-09-08T03:04:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:46.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:46.218973+0000) 2022-09-08T03:04:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:46.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:46.174149+0000) 2022-09-08T03:04:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:46 smithi055 ceph-mon[97033]: pgmap v10541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:04:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:46 smithi059 ceph-mon[97450]: pgmap v10541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:04:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:47.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:47.219165+0000) 2022-09-08T03:04:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:47.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:47.174375+0000) 2022-09-08T03:04:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:48.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:48.219298+0000) 2022-09-08T03:04:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:48.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:48.174671+0000) 2022-09-08T03:04:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:48 smithi055 ceph-mon[97033]: pgmap v10542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:48 smithi059 ceph-mon[97450]: pgmap v10542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:49.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:49.219449+0000) 2022-09-08T03:04:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:49.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:49.174910+0000) 2022-09-08T03:04:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:50.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:50.175035+0000) 2022-09-08T03:04:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:50.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:50.219595+0000) 2022-09-08T03:04:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:50 smithi055 ceph-mon[97033]: pgmap v10543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:50 smithi059 ceph-mon[97450]: pgmap v10543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:51.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:51.219726+0000) 2022-09-08T03:04:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:51.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:51.175244+0000) 2022-09-08T03:04:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:52.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:52.219858+0000) 2022-09-08T03:04:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:52.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:52.175481+0000) 2022-09-08T03:04:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:52 smithi055 ceph-mon[97033]: pgmap v10544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:04:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:52 smithi059 ceph-mon[97450]: pgmap v10544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:04:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:53.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:53.220068+0000) 2022-09-08T03:04:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:53.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:53.175764+0000) 2022-09-08T03:04:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:54.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:54.220225+0000) 2022-09-08T03:04:54.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:54.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:54.175941+0000) 2022-09-08T03:04:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:54 smithi055 ceph-mon[97033]: pgmap v10545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:54 smithi059 ceph-mon[97450]: pgmap v10545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:55.663 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:55.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:55.220350+0000) 2022-09-08T03:04:55.664 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:55.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:55.176086+0000) 2022-09-08T03:04:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:56.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:56.220558+0000) 2022-09-08T03:04:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:56.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:56.176321+0000) 2022-09-08T03:04:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:56 smithi055 ceph-mon[97033]: pgmap v10546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:04:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:56 smithi059 ceph-mon[97450]: pgmap v10546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:04:57.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:04:57.326+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:04:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:57.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:57.220695+0000) 2022-09-08T03:04:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:57.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:57.176558+0000) 2022-09-08T03:04:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:58.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:58.220875+0000) 2022-09-08T03:04:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:58.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:58.176822+0000) 2022-09-08T03:04:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:04:58 smithi055 ceph-mon[97033]: pgmap v10547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:04:58 smithi059 ceph-mon[97450]: pgmap v10547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:04:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:04:59.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:59.176950+0000) 2022-09-08T03:04:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:04:59.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:04:59.221010+0000) 2022-09-08T03:05:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:00.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:00.221231+0000) 2022-09-08T03:05:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:00.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:00.177145+0000) 2022-09-08T03:05:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:00 smithi055 ceph-mon[97033]: pgmap v10548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:00 smithi059 ceph-mon[97450]: pgmap v10548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:01.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:01.221357+0000) 2022-09-08T03:05:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:01.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:01.177378+0000) 2022-09-08T03:05:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:02.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:02.221565+0000) 2022-09-08T03:05:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:02.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:02.177611+0000) 2022-09-08T03:05:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:02 smithi055 ceph-mon[97033]: pgmap v10549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:05:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:02 smithi059 ceph-mon[97450]: pgmap v10549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:05:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:03.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:03.221773+0000) 2022-09-08T03:05:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:03.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:03.177879+0000) 2022-09-08T03:05:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:04.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:04.178134+0000) 2022-09-08T03:05:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:04.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:04.221918+0000) 2022-09-08T03:05:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:04 smithi055 ceph-mon[97033]: pgmap v10550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:04 smithi059 ceph-mon[97450]: pgmap v10550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:05.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:05.222057+0000) 2022-09-08T03:05:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:05.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:05.178344+0000) 2022-09-08T03:05:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:05:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:05:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:06.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:06.222203+0000) 2022-09-08T03:05:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:06.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:06.178592+0000) 2022-09-08T03:05:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:06 smithi055 ceph-mon[97033]: pgmap v10551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:05:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:05:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:06 smithi059 ceph-mon[97450]: pgmap v10551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:05:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:05:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:07.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:07.222326+0000) 2022-09-08T03:05:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:07.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:07.178783+0000) 2022-09-08T03:05:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:08.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:08.222494+0000) 2022-09-08T03:05:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:08.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:08.178961+0000) 2022-09-08T03:05:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:08 smithi055 ceph-mon[97033]: pgmap v10552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:05:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:08 smithi059 ceph-mon[97450]: pgmap v10552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:05:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:09.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:09.222704+0000) 2022-09-08T03:05:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:09.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:09.179191+0000) 2022-09-08T03:05:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:10.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:10.222870+0000) 2022-09-08T03:05:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:10.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:10.179399+0000) 2022-09-08T03:05:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:10 smithi055 ceph-mon[97033]: pgmap v10553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:05:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:10 smithi055 ceph-mon[97033]: osdmap e739: 12 total, 11 up, 11 in 2022-09-08T03:05:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:10 smithi059 ceph-mon[97450]: pgmap v10553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:05:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:10 smithi059 ceph-mon[97450]: osdmap e739: 12 total, 11 up, 11 in 2022-09-08T03:05:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:11.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:11.222997+0000) 2022-09-08T03:05:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:11.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:11.179578+0000) 2022-09-08T03:05:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:12.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:12.179698+0000) 2022-09-08T03:05:12.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:05:12.332+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:05:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:12.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:12.223206+0000) 2022-09-08T03:05:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:12 smithi059 ceph-mon[97450]: pgmap v10555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:05:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:12 smithi055 ceph-mon[97033]: pgmap v10555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:05:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:13.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:13.179937+0000) 2022-09-08T03:05:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:13.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:13.223394+0000) 2022-09-08T03:05:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:14 smithi055 ceph-mon[97033]: pgmap v10556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:05:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:14 smithi055 ceph-mon[97033]: osdmap e740: 12 total, 11 up, 11 in 2022-09-08T03:05:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:05:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:05:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:05:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:14 smithi059 ceph-mon[97450]: pgmap v10556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:05:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:14 smithi059 ceph-mon[97450]: osdmap e740: 12 total, 11 up, 11 in 2022-09-08T03:05:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:05:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:05:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:05:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:14.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:14.180135+0000) 2022-09-08T03:05:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:14.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:14.223522+0000) 2022-09-08T03:05:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:15.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:15.223755+0000) 2022-09-08T03:05:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:15.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:15.180357+0000) 2022-09-08T03:05:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:05:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:05:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:16.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:16.223970+0000) 2022-09-08T03:05:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:16.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:16.180594+0000) 2022-09-08T03:05:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:16 smithi055 ceph-mon[97033]: pgmap v10558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:05:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:16 smithi059 ceph-mon[97450]: pgmap v10558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:05:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:17.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:17.224185+0000) 2022-09-08T03:05:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:17.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:17.180846+0000) 2022-09-08T03:05:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:18.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:18.224320+0000) 2022-09-08T03:05:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:18.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:18.181105+0000) 2022-09-08T03:05:18.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:18 smithi055 ceph-mon[97033]: pgmap v10559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:05:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:18 smithi059 ceph-mon[97450]: pgmap v10559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:05:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:19.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:19.224518+0000) 2022-09-08T03:05:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:19.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:19.181329+0000) 2022-09-08T03:05:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:20.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:20.224750+0000) 2022-09-08T03:05:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:20.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:20.181550+0000) 2022-09-08T03:05:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:20 smithi055 ceph-mon[97033]: pgmap v10560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2022-09-08T03:05:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:20 smithi059 ceph-mon[97450]: pgmap v10560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2022-09-08T03:05:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:21.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:21.224896+0000) 2022-09-08T03:05:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:21.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:21.181755+0000) 2022-09-08T03:05:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:22.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:22.225114+0000) 2022-09-08T03:05:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:22.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:22.181934+0000) 2022-09-08T03:05:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:22 smithi055 ceph-mon[97033]: pgmap v10561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:05:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:22 smithi059 ceph-mon[97450]: pgmap v10561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:05:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:23.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:23.225334+0000) 2022-09-08T03:05:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:23.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:23.182174+0000) 2022-09-08T03:05:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:24.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:24.225509+0000) 2022-09-08T03:05:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:24.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:24.182428+0000) 2022-09-08T03:05:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:24 smithi055 ceph-mon[97033]: pgmap v10562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:05:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:24 smithi059 ceph-mon[97450]: pgmap v10562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:05:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:25.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:25.225692+0000) 2022-09-08T03:05:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:25.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:25.182713+0000) 2022-09-08T03:05:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:26.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:26.225820+0000) 2022-09-08T03:05:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:26.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:26.182924+0000) 2022-09-08T03:05:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:26 smithi055 ceph-mon[97033]: pgmap v10563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T03:05:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:26 smithi059 ceph-mon[97450]: pgmap v10563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T03:05:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:27.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:27.226000+0000) 2022-09-08T03:05:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:27.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:27.183186+0000) 2022-09-08T03:05:27.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:05:27.340+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:05:28.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:28.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:28.183381+0000) 2022-09-08T03:05:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:28.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:28.226167+0000) 2022-09-08T03:05:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:28 smithi055 ceph-mon[97033]: pgmap v10564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:28 smithi059 ceph-mon[97450]: pgmap v10564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:29.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:29.226326+0000) 2022-09-08T03:05:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:29.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:29.183515+0000) 2022-09-08T03:05:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:30.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:30.226467+0000) 2022-09-08T03:05:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:30.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:30.183746+0000) 2022-09-08T03:05:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:30 smithi055 ceph-mon[97033]: pgmap v10565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:30 smithi059 ceph-mon[97450]: pgmap v10565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:31.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:31.226666+0000) 2022-09-08T03:05:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:31.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:31.183941+0000) 2022-09-08T03:05:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:32.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:32.226885+0000) 2022-09-08T03:05:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:32.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:32.184162+0000) 2022-09-08T03:05:32.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:32 smithi055 ceph-mon[97033]: pgmap v10566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:05:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:32 smithi059 ceph-mon[97450]: pgmap v10566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:05:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:33.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:33.227054+0000) 2022-09-08T03:05:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:33.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:33.184395+0000) 2022-09-08T03:05:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:34.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:34.227204+0000) 2022-09-08T03:05:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:34.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:34.184628+0000) 2022-09-08T03:05:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:34 smithi055 ceph-mon[97033]: pgmap v10567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:34 smithi059 ceph-mon[97450]: pgmap v10567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:35.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:35.227371+0000) 2022-09-08T03:05:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:35.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:35.184823+0000) 2022-09-08T03:05:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:36.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:36.227537+0000) 2022-09-08T03:05:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:36.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:36.185057+0000) 2022-09-08T03:05:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:36 smithi055 ceph-mon[97033]: pgmap v10568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:05:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:36 smithi059 ceph-mon[97450]: pgmap v10568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:05:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:37.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:37.227771+0000) 2022-09-08T03:05:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:37.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:37.185268+0000) 2022-09-08T03:05:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:38.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:38.227957+0000) 2022-09-08T03:05:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:38.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:38.185502+0000) 2022-09-08T03:05:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:38 smithi055 ceph-mon[97033]: pgmap v10569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:38 smithi059 ceph-mon[97450]: pgmap v10569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:39.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:39.228213+0000) 2022-09-08T03:05:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:39.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:39.185671+0000) 2022-09-08T03:05:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:40.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:40.185908+0000) 2022-09-08T03:05:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:40.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:40.228363+0000) 2022-09-08T03:05:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:40 smithi055 ceph-mon[97033]: pgmap v10570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:40 smithi059 ceph-mon[97450]: pgmap v10570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:41.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:41.228484+0000) 2022-09-08T03:05:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:41.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:41.186140+0000) 2022-09-08T03:05:42.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:05:42.347+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:05:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:42.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:42.186367+0000) 2022-09-08T03:05:42.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:42.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:42.228678+0000) 2022-09-08T03:05:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:42 smithi055 ceph-mon[97033]: pgmap v10571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:05:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:42 smithi059 ceph-mon[97450]: pgmap v10571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:05:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:43.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:43.186612+0000) 2022-09-08T03:05:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:43.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:43.228868+0000) 2022-09-08T03:05:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:44.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:44.229043+0000) 2022-09-08T03:05:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:44.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:44.186817+0000) 2022-09-08T03:05:44.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:44 smithi055 ceph-mon[97033]: pgmap v10572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:44 smithi059 ceph-mon[97450]: pgmap v10572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:45.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:45.229226+0000) 2022-09-08T03:05:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:45.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:45.187001+0000) 2022-09-08T03:05:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:46.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:46.187197+0000) 2022-09-08T03:05:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:46.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:46.229390+0000) 2022-09-08T03:05:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:46 smithi055 ceph-mon[97033]: pgmap v10573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:05:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:46 smithi059 ceph-mon[97450]: pgmap v10573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:05:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:47.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:47.187381+0000) 2022-09-08T03:05:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:47.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:47.229662+0000) 2022-09-08T03:05:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:48.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:48.229829+0000) 2022-09-08T03:05:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:48.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:48.187586+0000) 2022-09-08T03:05:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:48 smithi055 ceph-mon[97033]: pgmap v10574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:48 smithi059 ceph-mon[97450]: pgmap v10574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:49.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:49.187750+0000) 2022-09-08T03:05:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:49.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:49.230041+0000) 2022-09-08T03:05:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:50.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:50.187961+0000) 2022-09-08T03:05:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:50.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:50.230257+0000) 2022-09-08T03:05:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:50 smithi055 ceph-mon[97033]: pgmap v10575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:50 smithi059 ceph-mon[97450]: pgmap v10575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:51.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:51.230443+0000) 2022-09-08T03:05:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:51.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:51.188139+0000) 2022-09-08T03:05:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:52.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:52.230631+0000) 2022-09-08T03:05:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:52.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:52.188356+0000) 2022-09-08T03:05:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:52 smithi055 ceph-mon[97033]: pgmap v10576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:05:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:52 smithi059 ceph-mon[97450]: pgmap v10576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:05:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:53.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:53.230844+0000) 2022-09-08T03:05:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:53.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:53.188601+0000) 2022-09-08T03:05:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:54.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:54.231030+0000) 2022-09-08T03:05:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:54.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:54.188822+0000) 2022-09-08T03:05:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:54 smithi059 ceph-mon[97450]: pgmap v10577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:54 smithi055 ceph-mon[97033]: pgmap v10577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:55.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:55.231218+0000) 2022-09-08T03:05:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:55.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:55.189062+0000) 2022-09-08T03:05:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:56.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:56.231415+0000) 2022-09-08T03:05:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:56.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:56.189188+0000) 2022-09-08T03:05:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:56 smithi059 ceph-mon[97450]: pgmap v10578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:05:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:56 smithi055 ceph-mon[97033]: pgmap v10578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:05:57.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:05:57.355+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:05:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:57.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:57.231596+0000) 2022-09-08T03:05:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:57.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:57.189352+0000) 2022-09-08T03:05:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:58.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:58.231782+0000) 2022-09-08T03:05:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:58.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:58.189597+0000) 2022-09-08T03:05:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:05:58 smithi059 ceph-mon[97450]: pgmap v10579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:05:58 smithi055 ceph-mon[97033]: pgmap v10579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:05:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:05:59.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:59.231925+0000) 2022-09-08T03:05:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:05:59.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:05:59.189831+0000) 2022-09-08T03:06:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:00.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:00.232090+0000) 2022-09-08T03:06:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:00.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:00.190075+0000) 2022-09-08T03:06:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:00 smithi059 ceph-mon[97450]: pgmap v10580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:00 smithi055 ceph-mon[97033]: pgmap v10580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:01.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:01.232232+0000) 2022-09-08T03:06:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:01.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:01.190225+0000) 2022-09-08T03:06:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-c[97426]: 2022-09-08T03:06:01.786+0000 7fe6d1767700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-c[97426]: 2022-09-08T03:06:01.786+0000 7fe6d1767700 -1 mon.c@1(leader) e3 *** Got Signal Hangup *** 2022-09-08T03:06:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-c[97426]: 2022-09-08T03:06:01.833+0000 7fe6d1767700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-c[97426]: 2022-09-08T03:06:01.833+0000 7fe6d1767700 -1 mon.c@1(leader) e3 *** Got Signal Hangup *** 2022-09-08T03:06:02.034 INFO:journalctl@ceph.osd.5.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-5[109228]: 2022-09-08T03:06:01.786+0000 7f37d3827700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.035 INFO:journalctl@ceph.osd.5.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-5[109228]: 2022-09-08T03:06:01.833+0000 7f37d3827700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.036 INFO:journalctl@ceph.osd.8.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-8[115065]: 2022-09-08T03:06:01.786+0000 7feb038d3700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.036 INFO:journalctl@ceph.osd.8.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-8[115065]: 2022-09-08T03:06:01.833+0000 7feb038d3700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.037 INFO:journalctl@ceph.osd.11.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-11[120911]: 2022-09-08T03:06:01.786+0000 7f2dd630b700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.038 INFO:journalctl@ceph.osd.11.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-11[120911]: 2022-09-08T03:06:01.833+0000 7f2dd630b700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.038 INFO:journalctl@ceph.mds.c.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-c[124719]: 2022-09-08T03:06:01.786+0000 7ff088596700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.039 INFO:journalctl@ceph.mds.c.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-c[124719]: 2022-09-08T03:06:01.833+0000 7ff088596700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.040 INFO:journalctl@ceph.mds.i.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-i[125064]: 2022-09-08T03:06:01.786+0000 7f45ee540700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.040 INFO:journalctl@ceph.mds.i.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-i[125064]: 2022-09-08T03:06:01.834+0000 7f45ee540700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.041 INFO:journalctl@ceph.mds.l.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-l[124021]: 2022-09-08T03:06:01.786+0000 7f95c0335700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.042 INFO:journalctl@ceph.mds.l.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-l[124021]: 2022-09-08T03:06:01.833+0000 7f95c0335700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.042 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-08T03:06:01.786+0000 7f0383075700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.043 INFO:journalctl@ceph.mgr.z.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-z[99507]: 2022-09-08T03:06:01.833+0000 7f0383075700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.044 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-2[103395]: 2022-09-08T03:06:01.786+0000 7fb849418700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.044 INFO:journalctl@ceph.osd.2.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-2[103395]: 2022-09-08T03:06:01.833+0000 7fb849418700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.044 INFO:journalctl@ceph.mds.f.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-f[124373]: 2022-09-08T03:06:01.786+0000 7f21c88c8700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.045 INFO:journalctl@ceph.mds.f.smithi059.stdout:Sep 08 03:06:01 smithi059 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-f[124373]: 2022-09-08T03:06:01.834+0000 7f21c88c8700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:06:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:02.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:02.190464+0000) 2022-09-08T03:06:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:02.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:02.232457+0000) 2022-09-08T03:06:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:02 smithi059 ceph-mon[97450]: pgmap v10581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:06:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:02 smithi055 ceph-mon[97033]: pgmap v10581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:06:03.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:03.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:03.190734+0000) 2022-09-08T03:06:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:03.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:03.232712+0000) 2022-09-08T03:06:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:04.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:04.232943+0000) 2022-09-08T03:06:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:04.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:04.190922+0000) 2022-09-08T03:06:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:04 smithi059 ceph-mon[97450]: pgmap v10582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:04 smithi055 ceph-mon[97033]: pgmap v10582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:05.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:05.233140+0000) 2022-09-08T03:06:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:05.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:05.191091+0000) 2022-09-08T03:06:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:06:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:06:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:06:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:06:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:06.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:06.191289+0000) 2022-09-08T03:06:06.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:06.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:06.233308+0000) 2022-09-08T03:06:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:06 smithi059 ceph-mon[97450]: pgmap v10583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:06:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:06 smithi055 ceph-mon[97033]: pgmap v10583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:06:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:07.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:07.191525+0000) 2022-09-08T03:06:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:07.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:07.233476+0000) 2022-09-08T03:06:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:08.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:08.191894+0000) 2022-09-08T03:06:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:08.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:08.233650+0000) 2022-09-08T03:06:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:08 smithi059 ceph-mon[97450]: pgmap v10584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:06:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:08 smithi055 ceph-mon[97033]: pgmap v10584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:06:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:09.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:09.233836+0000) 2022-09-08T03:06:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:09.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:09.192110+0000) 2022-09-08T03:06:10.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:10.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:10.192326+0000) 2022-09-08T03:06:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:10.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:10.234012+0000) 2022-09-08T03:06:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:10 smithi055 ceph-mon[97033]: pgmap v10585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:06:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:10 smithi055 ceph-mon[97033]: osdmap e741: 12 total, 11 up, 11 in 2022-09-08T03:06:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:10 smithi059 ceph-mon[97450]: pgmap v10585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:06:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:10 smithi059 ceph-mon[97450]: osdmap e741: 12 total, 11 up, 11 in 2022-09-08T03:06:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:11.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:11.234250+0000) 2022-09-08T03:06:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:11.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:11.192572+0000) 2022-09-08T03:06:12.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:06:12.372+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:06:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:12.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:12.234431+0000) 2022-09-08T03:06:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:12.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:12.192848+0000) 2022-09-08T03:06:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:12 smithi055 ceph-mon[97033]: pgmap v10587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:06:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:12 smithi059 ceph-mon[97450]: pgmap v10587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:06:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:13.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:13.234590+0000) 2022-09-08T03:06:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:13.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:13.193022+0000) 2022-09-08T03:06:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:14.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:14.234827+0000) 2022-09-08T03:06:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:14.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:14.193279+0000) 2022-09-08T03:06:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:14 smithi055 ceph-mon[97033]: pgmap v10588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:06:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:14 smithi055 ceph-mon[97033]: osdmap e742: 12 total, 11 up, 11 in 2022-09-08T03:06:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:14 smithi059 ceph-mon[97450]: pgmap v10588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:06:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:14 smithi059 ceph-mon[97450]: osdmap e742: 12 total, 11 up, 11 in 2022-09-08T03:06:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:15.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:15.235067+0000) 2022-09-08T03:06:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:15.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:15.193534+0000) 2022-09-08T03:06:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:06:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:06:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:06:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:06:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:06:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:06:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:06:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:06:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:16.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:16.235276+0000) 2022-09-08T03:06:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:16.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:16.193804+0000) 2022-09-08T03:06:16.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:16 smithi055 ceph-mon[97033]: pgmap v10590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:06:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:16 smithi059 ceph-mon[97450]: pgmap v10590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:06:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:17.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:17.194072+0000) 2022-09-08T03:06:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:17.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:17.235474+0000) 2022-09-08T03:06:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:18.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:18.235716+0000) 2022-09-08T03:06:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:18.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:18.194361+0000) 2022-09-08T03:06:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:18 smithi055 ceph-mon[97033]: pgmap v10591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:06:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:18 smithi059 ceph-mon[97450]: pgmap v10591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:06:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:19.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:19.194543+0000) 2022-09-08T03:06:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:19.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:19.235923+0000) 2022-09-08T03:06:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:20.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:20.236081+0000) 2022-09-08T03:06:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:20.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:20.194781+0000) 2022-09-08T03:06:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:20 smithi055 ceph-mon[97033]: pgmap v10592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T03:06:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:20 smithi059 ceph-mon[97450]: pgmap v10592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T03:06:21.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:21.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:21.195007+0000) 2022-09-08T03:06:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:21.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:21.236308+0000) 2022-09-08T03:06:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:22.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:22.236485+0000) 2022-09-08T03:06:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:22.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:22.195189+0000) 2022-09-08T03:06:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:22 smithi055 ceph-mon[97033]: pgmap v10593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:06:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:22 smithi059 ceph-mon[97450]: pgmap v10593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:06:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:23.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:23.236737+0000) 2022-09-08T03:06:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:23.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:23.195427+0000) 2022-09-08T03:06:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:24.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:24.236916+0000) 2022-09-08T03:06:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:24.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:24.195729+0000) 2022-09-08T03:06:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:24 smithi055 ceph-mon[97033]: pgmap v10594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:06:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:24 smithi059 ceph-mon[97450]: pgmap v10594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:06:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:25.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:25.237037+0000) 2022-09-08T03:06:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:25.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:25.195975+0000) 2022-09-08T03:06:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:26.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:26.237271+0000) 2022-09-08T03:06:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:26.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:26.196175+0000) 2022-09-08T03:06:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:26 smithi055 ceph-mon[97033]: pgmap v10595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T03:06:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:26 smithi059 ceph-mon[97450]: pgmap v10595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T03:06:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:27.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:27.196403+0000) 2022-09-08T03:06:27.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:06:27.387+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:06:27.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:27.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:27.237477+0000) 2022-09-08T03:06:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:28.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:28.237757+0000) 2022-09-08T03:06:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:28.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:28.196691+0000) 2022-09-08T03:06:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:28 smithi055 ceph-mon[97033]: pgmap v10596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:28 smithi059 ceph-mon[97450]: pgmap v10596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:29.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:29.237945+0000) 2022-09-08T03:06:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:29.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:29.196965+0000) 2022-09-08T03:06:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:30.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:30.238067+0000) 2022-09-08T03:06:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:30.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:30.197215+0000) 2022-09-08T03:06:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:30 smithi055 ceph-mon[97033]: pgmap v10597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:30 smithi059 ceph-mon[97450]: pgmap v10597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:31.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:31.238221+0000) 2022-09-08T03:06:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:31.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:31.197433+0000) 2022-09-08T03:06:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:32.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:32.197718+0000) 2022-09-08T03:06:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:32.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:32.238452+0000) 2022-09-08T03:06:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:32 smithi055 ceph-mon[97033]: pgmap v10598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:06:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:32 smithi059 ceph-mon[97450]: pgmap v10598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:06:33.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:33.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:33.238629+0000) 2022-09-08T03:06:33.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:33.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:33.197955+0000) 2022-09-08T03:06:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:34.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:34.238856+0000) 2022-09-08T03:06:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:34.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:34.198173+0000) 2022-09-08T03:06:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:34 smithi055 ceph-mon[97033]: pgmap v10599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:34 smithi059 ceph-mon[97450]: pgmap v10599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:35.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:35.239105+0000) 2022-09-08T03:06:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:35.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:35.198392+0000) 2022-09-08T03:06:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:36.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:36.239313+0000) 2022-09-08T03:06:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:36.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:36.198588+0000) 2022-09-08T03:06:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:36 smithi055 ceph-mon[97033]: pgmap v10600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:06:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:36 smithi059 ceph-mon[97450]: pgmap v10600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:06:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:37.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:37.239510+0000) 2022-09-08T03:06:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:37.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:37.198778+0000) 2022-09-08T03:06:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:38.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:38.199033+0000) 2022-09-08T03:06:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:38.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:38.239756+0000) 2022-09-08T03:06:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:38 smithi055 ceph-mon[97033]: pgmap v10601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:38 smithi059 ceph-mon[97450]: pgmap v10601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:39.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:39.240000+0000) 2022-09-08T03:06:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:39.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:39.199298+0000) 2022-09-08T03:06:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:40.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:40.199566+0000) 2022-09-08T03:06:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:40.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:40.240229+0000) 2022-09-08T03:06:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:40 smithi055 ceph-mon[97033]: pgmap v10602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:40 smithi059 ceph-mon[97450]: pgmap v10602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:41.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:41.240468+0000) 2022-09-08T03:06:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:41.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:41.199821+0000) 2022-09-08T03:06:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:42.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:42.240741+0000) 2022-09-08T03:06:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:42.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:42.200005+0000) 2022-09-08T03:06:42.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:06:42.401+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:06:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:42 smithi055 ceph-mon[97033]: pgmap v10603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:06:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:42 smithi059 ceph-mon[97450]: pgmap v10603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:06:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:43.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:43.240909+0000) 2022-09-08T03:06:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:43.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:43.200273+0000) 2022-09-08T03:06:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:44.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:44.241080+0000) 2022-09-08T03:06:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:44.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:44.200516+0000) 2022-09-08T03:06:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:44 smithi055 ceph-mon[97033]: pgmap v10604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:44 smithi059 ceph-mon[97450]: pgmap v10604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:45.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:45.241317+0000) 2022-09-08T03:06:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:45.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:45.200758+0000) 2022-09-08T03:06:46.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:46.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:46.201014+0000) 2022-09-08T03:06:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:46.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:46.241537+0000) 2022-09-08T03:06:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:46 smithi055 ceph-mon[97033]: pgmap v10605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:06:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:46 smithi059 ceph-mon[97450]: pgmap v10605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:06:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:47.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:47.241758+0000) 2022-09-08T03:06:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:47.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:47.201220+0000) 2022-09-08T03:06:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:48.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:48.241952+0000) 2022-09-08T03:06:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:48.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:48.201437+0000) 2022-09-08T03:06:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:48 smithi055 ceph-mon[97033]: pgmap v10606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:48 smithi059 ceph-mon[97450]: pgmap v10606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:49.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:49.242164+0000) 2022-09-08T03:06:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:49.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:49.201732+0000) 2022-09-08T03:06:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:50.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:50.242440+0000) 2022-09-08T03:06:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:50.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:50.201948+0000) 2022-09-08T03:06:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:50 smithi055 ceph-mon[97033]: pgmap v10607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:50 smithi059 ceph-mon[97450]: pgmap v10607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:51.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:51.242738+0000) 2022-09-08T03:06:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:51.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:51.202153+0000) 2022-09-08T03:06:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:52.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:52.242977+0000) 2022-09-08T03:06:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:52.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:52.202377+0000) 2022-09-08T03:06:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:52 smithi055 ceph-mon[97033]: pgmap v10608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:06:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:52 smithi059 ceph-mon[97450]: pgmap v10608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:06:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:53.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:53.243213+0000) 2022-09-08T03:06:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:53.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:53.202552+0000) 2022-09-08T03:06:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:54.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:54.243458+0000) 2022-09-08T03:06:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:54.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:54.202785+0000) 2022-09-08T03:06:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:54 smithi055 ceph-mon[97033]: pgmap v10609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:54 smithi059 ceph-mon[97450]: pgmap v10609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:55.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:55.243731+0000) 2022-09-08T03:06:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:55.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:55.202995+0000) 2022-09-08T03:06:56.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:56.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:56.203191+0000) 2022-09-08T03:06:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:56.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:56.243916+0000) 2022-09-08T03:06:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:56 smithi055 ceph-mon[97033]: pgmap v10610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:06:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:56 smithi059 ceph-mon[97450]: pgmap v10610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:06:57.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:57.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:57.203410+0000) 2022-09-08T03:06:57.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:06:57.417+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:06:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:57.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:57.244169+0000) 2022-09-08T03:06:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:58.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:58.244387+0000) 2022-09-08T03:06:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:58.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:58.203576+0000) 2022-09-08T03:06:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:06:58 smithi055 ceph-mon[97033]: pgmap v10611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:06:58 smithi059 ceph-mon[97450]: pgmap v10611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:06:59.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:06:59.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:59.244557+0000) 2022-09-08T03:06:59.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:06:59.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:06:59.203772+0000) 2022-09-08T03:07:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:00.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:00.244707+0000) 2022-09-08T03:07:00.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:00.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:00.203936+0000) 2022-09-08T03:07:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:00 smithi055 ceph-mon[97033]: pgmap v10612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:00 smithi059 ceph-mon[97450]: pgmap v10612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:01.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:01.244891+0000) 2022-09-08T03:07:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:01.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:01.204134+0000) 2022-09-08T03:07:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:02.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:02.245051+0000) 2022-09-08T03:07:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:02.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:02.204361+0000) 2022-09-08T03:07:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:02 smithi055 ceph-mon[97033]: pgmap v10613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:07:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:02 smithi059 ceph-mon[97450]: pgmap v10613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:07:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:03.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:03.204561+0000) 2022-09-08T03:07:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:03.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:03.245240+0000) 2022-09-08T03:07:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:04.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:04.245492+0000) 2022-09-08T03:07:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:04.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:04.204792+0000) 2022-09-08T03:07:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:04 smithi055 ceph-mon[97033]: pgmap v10614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:04 smithi059 ceph-mon[97450]: pgmap v10614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:05.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:05.204968+0000) 2022-09-08T03:07:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:05.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:05.245738+0000) 2022-09-08T03:07:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:07:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:07:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:06.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:06.245994+0000) 2022-09-08T03:07:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:06.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:06.205178+0000) 2022-09-08T03:07:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:06 smithi055 ceph-mon[97033]: pgmap v10615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:07:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:07:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:06 smithi059 ceph-mon[97450]: pgmap v10615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:07:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:07:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:07.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:07.205409+0000) 2022-09-08T03:07:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:07.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:07.246200+0000) 2022-09-08T03:07:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:08.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:08.205594+0000) 2022-09-08T03:07:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:08.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:08.246408+0000) 2022-09-08T03:07:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:08 smithi055 ceph-mon[97033]: pgmap v10616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:07:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:08 smithi059 ceph-mon[97450]: pgmap v10616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:07:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:09.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:09.246609+0000) 2022-09-08T03:07:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:09.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:09.205812+0000) 2022-09-08T03:07:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:10.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:10.246872+0000) 2022-09-08T03:07:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:10.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:10.206026+0000) 2022-09-08T03:07:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:10 smithi055 ceph-mon[97033]: pgmap v10617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:07:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:10 smithi055 ceph-mon[97033]: osdmap e743: 12 total, 11 up, 11 in 2022-09-08T03:07:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:10 smithi059 ceph-mon[97450]: pgmap v10617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:07:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:10 smithi059 ceph-mon[97450]: osdmap e743: 12 total, 11 up, 11 in 2022-09-08T03:07:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:11.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:11.247067+0000) 2022-09-08T03:07:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:11.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:11.206245+0000) 2022-09-08T03:07:12.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:07:12.433+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:07:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:12.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:12.247304+0000) 2022-09-08T03:07:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:12.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:12.206443+0000) 2022-09-08T03:07:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:12 smithi055 ceph-mon[97033]: pgmap v10619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:07:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:12 smithi059 ceph-mon[97450]: pgmap v10619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:07:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:13.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:13.247540+0000) 2022-09-08T03:07:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:13.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:13.206714+0000) 2022-09-08T03:07:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:14.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:14.247758+0000) 2022-09-08T03:07:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:14.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:14.206942+0000) 2022-09-08T03:07:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:14 smithi055 ceph-mon[97033]: pgmap v10620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:07:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:14 smithi055 ceph-mon[97033]: osdmap e744: 12 total, 11 up, 11 in 2022-09-08T03:07:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:14 smithi059 ceph-mon[97450]: pgmap v10620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:07:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:14 smithi059 ceph-mon[97450]: osdmap e744: 12 total, 11 up, 11 in 2022-09-08T03:07:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:15.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:15.247930+0000) 2022-09-08T03:07:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:15.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:15.207145+0000) 2022-09-08T03:07:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:16.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:16.248126+0000) 2022-09-08T03:07:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:16.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:16.207365+0000) 2022-09-08T03:07:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:16 smithi055 ceph-mon[97033]: pgmap v10622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:07:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:07:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:07:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:07:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:07:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:16 smithi059 ceph-mon[97450]: pgmap v10622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:07:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:07:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:07:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:07:16.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:07:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:17.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:17.248414+0000) 2022-09-08T03:07:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:17.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:17.207535+0000) 2022-09-08T03:07:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:18.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:18.248687+0000) 2022-09-08T03:07:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:18.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:18.207744+0000) 2022-09-08T03:07:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:18 smithi055 ceph-mon[97033]: pgmap v10623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:07:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:18 smithi059 ceph-mon[97450]: pgmap v10623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:07:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:19.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:19.248943+0000) 2022-09-08T03:07:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:19.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:19.207897+0000) 2022-09-08T03:07:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:20.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:20.249179+0000) 2022-09-08T03:07:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:20.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:20.208150+0000) 2022-09-08T03:07:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:20 smithi055 ceph-mon[97033]: pgmap v10624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2022-09-08T03:07:20.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:20 smithi059 ceph-mon[97450]: pgmap v10624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2022-09-08T03:07:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:21.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:21.249400+0000) 2022-09-08T03:07:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:21.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:21.208332+0000) 2022-09-08T03:07:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:22.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:22.208506+0000) 2022-09-08T03:07:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:22.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:22.249621+0000) 2022-09-08T03:07:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:22 smithi055 ceph-mon[97033]: pgmap v10625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:07:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:22 smithi059 ceph-mon[97450]: pgmap v10625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:07:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:23.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:23.249864+0000) 2022-09-08T03:07:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:23.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:23.208730+0000) 2022-09-08T03:07:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:24.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:24.250040+0000) 2022-09-08T03:07:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:24.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:24.208964+0000) 2022-09-08T03:07:24.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:24 smithi055 ceph-mon[97033]: pgmap v10626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:07:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:24 smithi059 ceph-mon[97450]: pgmap v10626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:07:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:25.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:25.209114+0000) 2022-09-08T03:07:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:25.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:25.250174+0000) 2022-09-08T03:07:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:26.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:26.250366+0000) 2022-09-08T03:07:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:26.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:26.209325+0000) 2022-09-08T03:07:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:26 smithi055 ceph-mon[97033]: pgmap v10627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:07:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:26 smithi059 ceph-mon[97450]: pgmap v10627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:07:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:27.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:27.250536+0000) 2022-09-08T03:07:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:27.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:27.209575+0000) 2022-09-08T03:07:27.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:07:27.440+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:07:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:28.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:28.250853+0000) 2022-09-08T03:07:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:28.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:28.209862+0000) 2022-09-08T03:07:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:28 smithi055 ceph-mon[97033]: pgmap v10628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:28 smithi059 ceph-mon[97450]: pgmap v10628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:29.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:29.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:29.251043+0000) 2022-09-08T03:07:29.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:29.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:29.210053+0000) 2022-09-08T03:07:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:30.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:30.251218+0000) 2022-09-08T03:07:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:30.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:30.210324+0000) 2022-09-08T03:07:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:30 smithi055 ceph-mon[97033]: pgmap v10629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:30 smithi059 ceph-mon[97450]: pgmap v10629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:31.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:31.251473+0000) 2022-09-08T03:07:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:31.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:31.210602+0000) 2022-09-08T03:07:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:32.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:32.251751+0000) 2022-09-08T03:07:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:32.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:32.210831+0000) 2022-09-08T03:07:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:32 smithi055 ceph-mon[97033]: pgmap v10630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:07:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:32 smithi059 ceph-mon[97450]: pgmap v10630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:07:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:33.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:33.251972+0000) 2022-09-08T03:07:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:33.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:33.211088+0000) 2022-09-08T03:07:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:34.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:34.252125+0000) 2022-09-08T03:07:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:34.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:34.211334+0000) 2022-09-08T03:07:34.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:34 smithi055 ceph-mon[97033]: pgmap v10631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:34 smithi059 ceph-mon[97450]: pgmap v10631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:35.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:35.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:35.252381+0000) 2022-09-08T03:07:35.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:35.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:35.211500+0000) 2022-09-08T03:07:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:36.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:36.252602+0000) 2022-09-08T03:07:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:36.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:36.211760+0000) 2022-09-08T03:07:36.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:36 smithi055 ceph-mon[97033]: pgmap v10632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:07:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:36 smithi059 ceph-mon[97450]: pgmap v10632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:07:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:37.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:37.252765+0000) 2022-09-08T03:07:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:37.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:37.211983+0000) 2022-09-08T03:07:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:38.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:38.212267+0000) 2022-09-08T03:07:38.750 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:38.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:38.252991+0000) 2022-09-08T03:07:38.751 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:38 smithi055 ceph-mon[97033]: pgmap v10633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:38 smithi059 ceph-mon[97450]: pgmap v10633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:39.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:39.253191+0000) 2022-09-08T03:07:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:39.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:39.212503+0000) 2022-09-08T03:07:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:40.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:40.212750+0000) 2022-09-08T03:07:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:40.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:40.253396+0000) 2022-09-08T03:07:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:40 smithi055 ceph-mon[97033]: pgmap v10634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:40 smithi059 ceph-mon[97450]: pgmap v10634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:41.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:41.253622+0000) 2022-09-08T03:07:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:41.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:41.212941+0000) 2022-09-08T03:07:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:42.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:42.213162+0000) 2022-09-08T03:07:42.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:07:42.453+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:07:42.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:42.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:42.253902+0000) 2022-09-08T03:07:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:42 smithi055 ceph-mon[97033]: pgmap v10635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:07:42.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:42 smithi059 ceph-mon[97450]: pgmap v10635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:07:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:43.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:43.213442+0000) 2022-09-08T03:07:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:43.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:43.254163+0000) 2022-09-08T03:07:44.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:44.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:44.213730+0000) 2022-09-08T03:07:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:44.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:44.254403+0000) 2022-09-08T03:07:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:44 smithi055 ceph-mon[97033]: pgmap v10636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:44 smithi059 ceph-mon[97450]: pgmap v10636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:45.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:45.254629+0000) 2022-09-08T03:07:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:45.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:45.213941+0000) 2022-09-08T03:07:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:46.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:46.254858+0000) 2022-09-08T03:07:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:46.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:46.214192+0000) 2022-09-08T03:07:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:46 smithi055 ceph-mon[97033]: pgmap v10637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:07:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:46 smithi059 ceph-mon[97450]: pgmap v10637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:07:47.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:47.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:47.214364+0000) 2022-09-08T03:07:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:47.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:47.255081+0000) 2022-09-08T03:07:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:48.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:48.214601+0000) 2022-09-08T03:07:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:48.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:48.255296+0000) 2022-09-08T03:07:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:48 smithi055 ceph-mon[97033]: pgmap v10638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:48 smithi059 ceph-mon[97450]: pgmap v10638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:49.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:49.214881+0000) 2022-09-08T03:07:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:49.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:49.255524+0000) 2022-09-08T03:07:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:50.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:50.255730+0000) 2022-09-08T03:07:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:50.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:50.215117+0000) 2022-09-08T03:07:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:50 smithi055 ceph-mon[97033]: pgmap v10639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:50 smithi059 ceph-mon[97450]: pgmap v10639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:51.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:51.255866+0000) 2022-09-08T03:07:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:51.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:51.215324+0000) 2022-09-08T03:07:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:52.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:52.256055+0000) 2022-09-08T03:07:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:52.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:52.215552+0000) 2022-09-08T03:07:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:52 smithi055 ceph-mon[97033]: pgmap v10640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:07:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:52 smithi059 ceph-mon[97450]: pgmap v10640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:07:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:53.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:53.256278+0000) 2022-09-08T03:07:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:53.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:53.215798+0000) 2022-09-08T03:07:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:54.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:54.256509+0000) 2022-09-08T03:07:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:54.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:54.215950+0000) 2022-09-08T03:07:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:54 smithi055 ceph-mon[97033]: pgmap v10641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:54 smithi059 ceph-mon[97450]: pgmap v10641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:55.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:55.256687+0000) 2022-09-08T03:07:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:55.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:55.216147+0000) 2022-09-08T03:07:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:56.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:56.256900+0000) 2022-09-08T03:07:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:56.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:56.216379+0000) 2022-09-08T03:07:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:56 smithi055 ceph-mon[97033]: pgmap v10642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:07:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:56 smithi059 ceph-mon[97450]: pgmap v10642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:07:57.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:07:57.454+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:07:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:57.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:57.257053+0000) 2022-09-08T03:07:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:57.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:57.216598+0000) 2022-09-08T03:07:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:58.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:58.257278+0000) 2022-09-08T03:07:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:58.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:58.216882+0000) 2022-09-08T03:07:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:07:58 smithi055 ceph-mon[97033]: pgmap v10643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:07:58 smithi059 ceph-mon[97450]: pgmap v10643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:07:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:07:59.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:59.257561+0000) 2022-09-08T03:07:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:07:59.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:07:59.217055+0000) 2022-09-08T03:08:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:00.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:00.257795+0000) 2022-09-08T03:08:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:00.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:00.217261+0000) 2022-09-08T03:08:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:00 smithi055 ceph-mon[97033]: pgmap v10644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:08:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:00 smithi059 ceph-mon[97450]: pgmap v10644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:08:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:01.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:01.217461+0000) 2022-09-08T03:08:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:01.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:01.258018+0000) 2022-09-08T03:08:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:02.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:02.258154+0000) 2022-09-08T03:08:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:02.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:02.217657+0000) 2022-09-08T03:08:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:02 smithi055 ceph-mon[97033]: pgmap v10645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:08:02.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:02 smithi059 ceph-mon[97450]: pgmap v10645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:08:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:03.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:03.217828+0000) 2022-09-08T03:08:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:03.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:03.258353+0000) 2022-09-08T03:08:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:04.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:04.258572+0000) 2022-09-08T03:08:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:04.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:04.218033+0000) 2022-09-08T03:08:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:04 smithi055 ceph-mon[97033]: pgmap v10646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:08:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:04 smithi059 ceph-mon[97450]: pgmap v10646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:08:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:05.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:05.258783+0000) 2022-09-08T03:08:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:05.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:05.218276+0000) 2022-09-08T03:08:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:08:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:08:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:06.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:06.259013+0000) 2022-09-08T03:08:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:06.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:06.218515+0000) 2022-09-08T03:08:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:06 smithi059 ceph-mon[97450]: pgmap v10647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:08:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:08:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:06 smithi055 ceph-mon[97033]: pgmap v10647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:08:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:08:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:07.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:07.259262+0000) 2022-09-08T03:08:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:07.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:07.218681+0000) 2022-09-08T03:08:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:08.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:08.218879+0000) 2022-09-08T03:08:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:08.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:08.259451+0000) 2022-09-08T03:08:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:08 smithi059 ceph-mon[97450]: pgmap v10648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:08:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:08 smithi055 ceph-mon[97033]: pgmap v10648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:08:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:09.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:09.259715+0000) 2022-09-08T03:08:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:09.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:09.219097+0000) 2022-09-08T03:08:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:10.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:10.259914+0000) 2022-09-08T03:08:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:10.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:10.219331+0000) 2022-09-08T03:08:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:10 smithi059 ceph-mon[97450]: pgmap v10649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:08:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:10 smithi059 ceph-mon[97450]: osdmap e745: 12 total, 11 up, 11 in 2022-09-08T03:08:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:10 smithi055 ceph-mon[97033]: pgmap v10649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:08:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:10 smithi055 ceph-mon[97033]: osdmap e745: 12 total, 11 up, 11 in 2022-09-08T03:08:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:11.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:11.260121+0000) 2022-09-08T03:08:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:11.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:11.219555+0000) 2022-09-08T03:08:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:12.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:12.260323+0000) 2022-09-08T03:08:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:12.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:12.219856+0000) 2022-09-08T03:08:12.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:08:12.467+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:08:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:12 smithi059 ceph-mon[97450]: pgmap v10651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:08:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:12 smithi055 ceph-mon[97033]: pgmap v10651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:08:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:13.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:13.260558+0000) 2022-09-08T03:08:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:13.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:13.220074+0000) 2022-09-08T03:08:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:14.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:14.260781+0000) 2022-09-08T03:08:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:14.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:14.220314+0000) 2022-09-08T03:08:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:14 smithi059 ceph-mon[97450]: pgmap v10652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:08:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:14 smithi055 ceph-mon[97033]: pgmap v10652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:08:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:15.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:15.220523+0000) 2022-09-08T03:08:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:15.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:15.261021+0000) 2022-09-08T03:08:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:16.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:16.261168+0000) 2022-09-08T03:08:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:16.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:16.220759+0000) 2022-09-08T03:08:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:16 smithi055 ceph-mon[97033]: pgmap v10653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:08:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:08:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:08:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:08:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:16 smithi059 ceph-mon[97450]: pgmap v10653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:08:16.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:08:16.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:08:16.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:08:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:17.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:17.221009+0000) 2022-09-08T03:08:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:17.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:17.261380+0000) 2022-09-08T03:08:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:18.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:18.261612+0000) 2022-09-08T03:08:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:18.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:18.221281+0000) 2022-09-08T03:08:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:18 smithi059 ceph-mon[97450]: pgmap v10654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:08:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:18 smithi059 ceph-mon[97450]: osdmap e746: 12 total, 11 up, 11 in 2022-09-08T03:08:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:18 smithi055 ceph-mon[97033]: pgmap v10654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:08:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:18 smithi055 ceph-mon[97033]: osdmap e746: 12 total, 11 up, 11 in 2022-09-08T03:08:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:19.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:19.261918+0000) 2022-09-08T03:08:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:19.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:19.221484+0000) 2022-09-08T03:08:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:20.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:20.262143+0000) 2022-09-08T03:08:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:20.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:20.221702+0000) 2022-09-08T03:08:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:20 smithi059 ceph-mon[97450]: pgmap v10656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2022-09-08T03:08:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:20 smithi055 ceph-mon[97033]: pgmap v10656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2022-09-08T03:08:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:21.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:21.262304+0000) 2022-09-08T03:08:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:21.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:21.221883+0000) 2022-09-08T03:08:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:22.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:22.262573+0000) 2022-09-08T03:08:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:22.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:22.222120+0000) 2022-09-08T03:08:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:22 smithi059 ceph-mon[97450]: pgmap v10657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:08:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:22 smithi055 ceph-mon[97033]: pgmap v10657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:08:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:23.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:23.262713+0000) 2022-09-08T03:08:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:23.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:23.222405+0000) 2022-09-08T03:08:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:24.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:24.262838+0000) 2022-09-08T03:08:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:24.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:24.222705+0000) 2022-09-08T03:08:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:24 smithi059 ceph-mon[97450]: pgmap v10658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:08:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:24 smithi055 ceph-mon[97033]: pgmap v10658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:08:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:25.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:25.262991+0000) 2022-09-08T03:08:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:25.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:25.222874+0000) 2022-09-08T03:08:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:26.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:26.263217+0000) 2022-09-08T03:08:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:26.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:26.223148+0000) 2022-09-08T03:08:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:26 smithi059 ceph-mon[97450]: pgmap v10659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:08:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:26 smithi055 ceph-mon[97033]: pgmap v10659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:08:27.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:08:27.471+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:08:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:27.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:27.263370+0000) 2022-09-08T03:08:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:27.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:27.223397+0000) 2022-09-08T03:08:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:28.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:28.263568+0000) 2022-09-08T03:08:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:28.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:28.223711+0000) 2022-09-08T03:08:28.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:28 smithi059 ceph-mon[97450]: pgmap v10660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:08:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:28 smithi055 ceph-mon[97033]: pgmap v10660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:08:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:29.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:29.223950+0000) 2022-09-08T03:08:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:29.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:29.263750+0000) 2022-09-08T03:08:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:30.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:30.224181+0000) 2022-09-08T03:08:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:30.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:30.264002+0000) 2022-09-08T03:08:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:30 smithi059 ceph-mon[97450]: pgmap v10661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:08:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:30 smithi055 ceph-mon[97033]: pgmap v10661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:08:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:31.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:31.264227+0000) 2022-09-08T03:08:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:31.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:31.224378+0000) 2022-09-08T03:08:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:32.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:32.224610+0000) 2022-09-08T03:08:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:32.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:32.264454+0000) 2022-09-08T03:08:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:32 smithi059 ceph-mon[97450]: pgmap v10662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:08:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:32 smithi055 ceph-mon[97033]: pgmap v10662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:08:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:33.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:33.264690+0000) 2022-09-08T03:08:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:33.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:33.224893+0000) 2022-09-08T03:08:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:34.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:34.264830+0000) 2022-09-08T03:08:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:34.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:34.225087+0000) 2022-09-08T03:08:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:34 smithi055 ceph-mon[97033]: pgmap v10663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:08:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:34 smithi059 ceph-mon[97450]: pgmap v10663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:08:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:35.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:35.265054+0000) 2022-09-08T03:08:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:35.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:35.225319+0000) 2022-09-08T03:08:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:36.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:36.265284+0000) 2022-09-08T03:08:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:36.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:36.225536+0000) 2022-09-08T03:08:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:36 smithi055 ceph-mon[97033]: pgmap v10664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:08:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:36 smithi059 ceph-mon[97450]: pgmap v10664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:08:37.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:37.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:37.225738+0000) 2022-09-08T03:08:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:37.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:37.265527+0000) 2022-09-08T03:08:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:38.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:38.226006+0000) 2022-09-08T03:08:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:38.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:38.265740+0000) 2022-09-08T03:08:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:38 smithi055 ceph-mon[97033]: pgmap v10665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:08:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:38 smithi059 ceph-mon[97450]: pgmap v10665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:08:39.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:39.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:39.226238+0000) 2022-09-08T03:08:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:39.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:39.265871+0000) 2022-09-08T03:08:40.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:40.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:40.266092+0000) 2022-09-08T03:08:40.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:40.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:40.226482+0000) 2022-09-08T03:08:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:40 smithi055 ceph-mon[97033]: pgmap v10666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:08:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:40 smithi059 ceph-mon[97450]: pgmap v10666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:08:41.517 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:41.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:41.266241+0000) 2022-09-08T03:08:41.518 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:41.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:41.226757+0000) 2022-09-08T03:08:42.487 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:42.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:42.266480+0000) 2022-09-08T03:08:42.488 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:42.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:42.227019+0000) 2022-09-08T03:08:42.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:08:42.486+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:08:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:42 smithi055 ceph-mon[97033]: pgmap v10667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:08:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:42 smithi059 ceph-mon[97450]: pgmap v10667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:08:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:43.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:43.266732+0000) 2022-09-08T03:08:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:43.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:43.227253+0000) 2022-09-08T03:08:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:44.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:44.266984+0000) 2022-09-08T03:08:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:44.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:44.227414+0000) 2022-09-08T03:08:44.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:44 smithi055 ceph-mon[97033]: pgmap v10668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:08:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:44 smithi059 ceph-mon[97450]: pgmap v10668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:08:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:45.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:45.267159+0000) 2022-09-08T03:08:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:45.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:45.227622+0000) 2022-09-08T03:08:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:46.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:46.267342+0000) 2022-09-08T03:08:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:46.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:46.227855+0000) 2022-09-08T03:08:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:46 smithi055 ceph-mon[97033]: pgmap v10669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:08:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:08:47.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:46 smithi059 ceph-mon[97450]: pgmap v10669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:08:47.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:08:47.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:47.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:47.228048+0000) 2022-09-08T03:08:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:47.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:47.267561+0000) 2022-09-08T03:08:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:47 smithi055 ceph-mon[97033]: pgmap v10670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:08:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:47 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T03:08:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:47 smithi059 ceph-mon[97450]: pgmap v10670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:08:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:47 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T03:08:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:48.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:48.267780+0000) 2022-09-08T03:08:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:48.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:48.228245+0000) 2022-09-08T03:08:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:49.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:49.267996+0000) 2022-09-08T03:08:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:49.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:49.228473+0000) 2022-09-08T03:08:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:49 smithi055 ceph-mon[97033]: pgmap v10671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:08:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:49 smithi059 ceph-mon[97450]: pgmap v10671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:08:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:50.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:50.268148+0000) 2022-09-08T03:08:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:50.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:50.228742+0000) 2022-09-08T03:08:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:51.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:51.268336+0000) 2022-09-08T03:08:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:51.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:51.228938+0000) 2022-09-08T03:08:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:51 smithi055 ceph-mon[97033]: pgmap v10672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-08T03:08:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:51 smithi059 ceph-mon[97450]: pgmap v10672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-08T03:08:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:52.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:52.268528+0000) 2022-09-08T03:08:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:52.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:52.229173+0000) 2022-09-08T03:08:53.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:53.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:53.268715+0000) 2022-09-08T03:08:53.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:53.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:53.229465+0000) 2022-09-08T03:08:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:53 smithi055 ceph-mon[97033]: pgmap v10673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:08:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:53 smithi059 ceph-mon[97450]: pgmap v10673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:08:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:54.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:54.268876+0000) 2022-09-08T03:08:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:54.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:54.229745+0000) 2022-09-08T03:08:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:55.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:55.269082+0000) 2022-09-08T03:08:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:55.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:55.230009+0000) 2022-09-08T03:08:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:55 smithi055 ceph-mon[97033]: pgmap v10674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:08:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:55 smithi059 ceph-mon[97450]: pgmap v10674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:08:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:56.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:56.269251+0000) 2022-09-08T03:08:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:56.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:56.230254+0000) 2022-09-08T03:08:57.489 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:57.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:57.269480+0000) 2022-09-08T03:08:57.490 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:57.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:57.230508+0000) 2022-09-08T03:08:57.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:08:57.488+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:08:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:57 smithi055 ceph-mon[97033]: pgmap v10675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:08:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:57 smithi059 ceph-mon[97450]: pgmap v10675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:08:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:58.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:58.269746+0000) 2022-09-08T03:08:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:58.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:58.230797+0000) 2022-09-08T03:08:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:08:59.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:59.269961+0000) 2022-09-08T03:08:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:08:59.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:08:59.231056+0000) 2022-09-08T03:08:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:08:59 smithi055 ceph-mon[97033]: pgmap v10676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:08:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:08:59 smithi059 ceph-mon[97450]: pgmap v10676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:00.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:00.270138+0000) 2022-09-08T03:09:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:00.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:00.231304+0000) 2022-09-08T03:09:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:01.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:01.270357+0000) 2022-09-08T03:09:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:01.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:01.231547+0000) 2022-09-08T03:09:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:01 smithi055 ceph-mon[97033]: pgmap v10677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:09:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:01 smithi059 ceph-mon[97450]: pgmap v10677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:09:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:02.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:02.270621+0000) 2022-09-08T03:09:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:02.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:02.231775+0000) 2022-09-08T03:09:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:03.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:03.270802+0000) 2022-09-08T03:09:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:03.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:03.232020+0000) 2022-09-08T03:09:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:03 smithi055 ceph-mon[97033]: pgmap v10678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:03 smithi059 ceph-mon[97450]: pgmap v10678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:04.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:04.270988+0000) 2022-09-08T03:09:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:04.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:04.232265+0000) 2022-09-08T03:09:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:05.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:05.271206+0000) 2022-09-08T03:09:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:05.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:05.232517+0000) 2022-09-08T03:09:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:05 smithi055 ceph-mon[97033]: pgmap v10679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:09:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:05 smithi059 ceph-mon[97450]: pgmap v10679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:09:06.540 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:06.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:06.271400+0000) 2022-09-08T03:09:06.541 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:06.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:06.232692+0000) 2022-09-08T03:09:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:09:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:09:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:07.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:07.271522+0000) 2022-09-08T03:09:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:07.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:07.232934+0000) 2022-09-08T03:09:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:07 smithi055 ceph-mon[97033]: pgmap v10680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:09:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:07 smithi059 ceph-mon[97450]: pgmap v10680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:09:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:08.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:08.271744+0000) 2022-09-08T03:09:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:08.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:08.233144+0000) 2022-09-08T03:09:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:09.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:09.271973+0000) 2022-09-08T03:09:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:09.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:09.233324+0000) 2022-09-08T03:09:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:09 smithi055 ceph-mon[97033]: pgmap v10681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:09:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:09 smithi059 ceph-mon[97450]: pgmap v10681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:09:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:10.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:10.272152+0000) 2022-09-08T03:09:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:10.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:10.233571+0000) 2022-09-08T03:09:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:10 smithi055 ceph-mon[97033]: osdmap e747: 12 total, 11 up, 11 in 2022-09-08T03:09:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:10 smithi059 ceph-mon[97450]: osdmap e747: 12 total, 11 up, 11 in 2022-09-08T03:09:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:11.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:11.272358+0000) 2022-09-08T03:09:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:11.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:11.233705+0000) 2022-09-08T03:09:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:11 smithi055 ceph-mon[97033]: pgmap v10683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:09:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:11 smithi059 ceph-mon[97450]: pgmap v10683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:09:12.502 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:12.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:12.272569+0000) 2022-09-08T03:09:12.503 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:12.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:12.233908+0000) 2022-09-08T03:09:12.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:09:12.500+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:09:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:13.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:13.272815+0000) 2022-09-08T03:09:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:13.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:13.234157+0000) 2022-09-08T03:09:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:13 smithi055 ceph-mon[97033]: pgmap v10684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:09:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:13 smithi055 ceph-mon[97033]: osdmap e748: 12 total, 11 up, 11 in 2022-09-08T03:09:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:13 smithi059 ceph-mon[97450]: pgmap v10684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:09:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:13 smithi059 ceph-mon[97450]: osdmap e748: 12 total, 11 up, 11 in 2022-09-08T03:09:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:14.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:14.273060+0000) 2022-09-08T03:09:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:14.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:14.234404+0000) 2022-09-08T03:09:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:15.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:15.273295+0000) 2022-09-08T03:09:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:15.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:15.234680+0000) 2022-09-08T03:09:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:15 smithi055 ceph-mon[97033]: pgmap v10686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:09:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:15 smithi059 ceph-mon[97450]: pgmap v10686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:09:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:16.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:16.273522+0000) 2022-09-08T03:09:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:16.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:16.234916+0000) 2022-09-08T03:09:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:17.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:17.273741+0000) 2022-09-08T03:09:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:17.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:17.235139+0000) 2022-09-08T03:09:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:17 smithi055 ceph-mon[97033]: pgmap v10687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:09:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:17 smithi059 ceph-mon[97450]: pgmap v10687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:09:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:18.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:18.273945+0000) 2022-09-08T03:09:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:18.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:18.235349+0000) 2022-09-08T03:09:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:19.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:19.274147+0000) 2022-09-08T03:09:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:19.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:19.235550+0000) 2022-09-08T03:09:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:19 smithi055 ceph-mon[97033]: pgmap v10688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 226 B/s wr, 7 op/s 2022-09-08T03:09:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:19 smithi059 ceph-mon[97450]: pgmap v10688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 226 B/s wr, 7 op/s 2022-09-08T03:09:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:20.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:20.274409+0000) 2022-09-08T03:09:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:20.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:20.235753+0000) 2022-09-08T03:09:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:21.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:21.274593+0000) 2022-09-08T03:09:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:21.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:21.235960+0000) 2022-09-08T03:09:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:21 smithi055 ceph-mon[97033]: pgmap v10689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:09:22.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:21 smithi059 ceph-mon[97450]: pgmap v10689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:09:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:22.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:22.274795+0000) 2022-09-08T03:09:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:22.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:22.236124+0000) 2022-09-08T03:09:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:23.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:23.275012+0000) 2022-09-08T03:09:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:23.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:23.236342+0000) 2022-09-08T03:09:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:23 smithi055 ceph-mon[97033]: pgmap v10690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:09:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:23 smithi059 ceph-mon[97450]: pgmap v10690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:09:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:24.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:24.275230+0000) 2022-09-08T03:09:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:24.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:24.236490+0000) 2022-09-08T03:09:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:25.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:25.275469+0000) 2022-09-08T03:09:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:25.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:25.236741+0000) 2022-09-08T03:09:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:25 smithi055 ceph-mon[97033]: pgmap v10691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T03:09:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:25 smithi059 ceph-mon[97450]: pgmap v10691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T03:09:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:26.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:26.275649+0000) 2022-09-08T03:09:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:26.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:26.236970+0000) 2022-09-08T03:09:27.506 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:27.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:27.275764+0000) 2022-09-08T03:09:27.506 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:27.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:27.237206+0000) 2022-09-08T03:09:27.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:09:27.504+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:09:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:27 smithi055 ceph-mon[97033]: pgmap v10692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:09:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:27 smithi059 ceph-mon[97450]: pgmap v10692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:09:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:28.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:28.275889+0000) 2022-09-08T03:09:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:28.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:28.237422+0000) 2022-09-08T03:09:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:29.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:29.276048+0000) 2022-09-08T03:09:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:29.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:29.237665+0000) 2022-09-08T03:09:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:29 smithi055 ceph-mon[97033]: pgmap v10693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:29 smithi059 ceph-mon[97450]: pgmap v10693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:30.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:30.276230+0000) 2022-09-08T03:09:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:30.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:30.237916+0000) 2022-09-08T03:09:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:31.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:31.276394+0000) 2022-09-08T03:09:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:31.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:31.238104+0000) 2022-09-08T03:09:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:31 smithi055 ceph-mon[97033]: pgmap v10694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:09:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:31 smithi059 ceph-mon[97450]: pgmap v10694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:09:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:32.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:32.276611+0000) 2022-09-08T03:09:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:32.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:32.238289+0000) 2022-09-08T03:09:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:33.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:33.276844+0000) 2022-09-08T03:09:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:33.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:33.238508+0000) 2022-09-08T03:09:33.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:33 smithi055 ceph-mon[97033]: pgmap v10695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:33 smithi059 ceph-mon[97450]: pgmap v10695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:34.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:34.277043+0000) 2022-09-08T03:09:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:34.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:34.238759+0000) 2022-09-08T03:09:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:35.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:35.238990+0000) 2022-09-08T03:09:35.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:35.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:35.277239+0000) 2022-09-08T03:09:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:35 smithi055 ceph-mon[97033]: pgmap v10696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:35 smithi059 ceph-mon[97450]: pgmap v10696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:36.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:36.239188+0000) 2022-09-08T03:09:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:36.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:36.277427+0000) 2022-09-08T03:09:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:37.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:37.277681+0000) 2022-09-08T03:09:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:37.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:37.239431+0000) 2022-09-08T03:09:37.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:37 smithi055 ceph-mon[97033]: pgmap v10697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:09:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:37 smithi059 ceph-mon[97450]: pgmap v10697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:09:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:38.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:38.277818+0000) 2022-09-08T03:09:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:38.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:38.239714+0000) 2022-09-08T03:09:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:39.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:39.278050+0000) 2022-09-08T03:09:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:39.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:39.239918+0000) 2022-09-08T03:09:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:39 smithi055 ceph-mon[97033]: pgmap v10698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:39 smithi059 ceph-mon[97450]: pgmap v10698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:40.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:40.278258+0000) 2022-09-08T03:09:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:40.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:40.240074+0000) 2022-09-08T03:09:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:41.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:41.278405+0000) 2022-09-08T03:09:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:41.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:41.240262+0000) 2022-09-08T03:09:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:41 smithi055 ceph-mon[97033]: pgmap v10699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:09:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:41 smithi059 ceph-mon[97450]: pgmap v10699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:09:42.509 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:42.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:42.278599+0000) 2022-09-08T03:09:42.509 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:42.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:42.240443+0000) 2022-09-08T03:09:42.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:09:42.508+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:09:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:43.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:43.278869+0000) 2022-09-08T03:09:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:43.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:43.240723+0000) 2022-09-08T03:09:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:43 smithi055 ceph-mon[97033]: pgmap v10700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:43 smithi059 ceph-mon[97450]: pgmap v10700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:44.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:44.279130+0000) 2022-09-08T03:09:44.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:44.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:44.240873+0000) 2022-09-08T03:09:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:45.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:45.279270+0000) 2022-09-08T03:09:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:45.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:45.241003+0000) 2022-09-08T03:09:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:45 smithi055 ceph-mon[97033]: pgmap v10701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:45 smithi059 ceph-mon[97450]: pgmap v10701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:46.615 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:46.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:46.241128+0000) 2022-09-08T03:09:46.616 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:46.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:46.279482+0000) 2022-09-08T03:09:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:47.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:47.279732+0000) 2022-09-08T03:09:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:47.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:47.241358+0000) 2022-09-08T03:09:47.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:47 smithi055 ceph-mon[97033]: pgmap v10702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:09:47.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:09:47.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:09:47.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:09:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:47 smithi059 ceph-mon[97450]: pgmap v10702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:09:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:09:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:09:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:09:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:48.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:48.279954+0000) 2022-09-08T03:09:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:48.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:48.241591+0000) 2022-09-08T03:09:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:49.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:49.280183+0000) 2022-09-08T03:09:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:49.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:49.241775+0000) 2022-09-08T03:09:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:49 smithi055 ceph-mon[97033]: pgmap v10703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:49 smithi059 ceph-mon[97450]: pgmap v10703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:50.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:50.280416+0000) 2022-09-08T03:09:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:50.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:50.241953+0000) 2022-09-08T03:09:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:51.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:51.280665+0000) 2022-09-08T03:09:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:51.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:51.242166+0000) 2022-09-08T03:09:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:51 smithi055 ceph-mon[97033]: pgmap v10704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:09:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:51 smithi059 ceph-mon[97450]: pgmap v10704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:09:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:52.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:52.280925+0000) 2022-09-08T03:09:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:52.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:52.242426+0000) 2022-09-08T03:09:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:53.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:53.281131+0000) 2022-09-08T03:09:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:53.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:53.242672+0000) 2022-09-08T03:09:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:53 smithi055 ceph-mon[97033]: pgmap v10705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:53 smithi059 ceph-mon[97450]: pgmap v10705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:54.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:54.281324+0000) 2022-09-08T03:09:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:54.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:54.242904+0000) 2022-09-08T03:09:55.663 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:55.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:55.281565+0000) 2022-09-08T03:09:55.664 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:55.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:55.243139+0000) 2022-09-08T03:09:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:55 smithi055 ceph-mon[97033]: pgmap v10706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:55 smithi059 ceph-mon[97450]: pgmap v10706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:56.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:56.281813+0000) 2022-09-08T03:09:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:56.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:56.243332+0000) 2022-09-08T03:09:57.515 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:57.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:57.281948+0000) 2022-09-08T03:09:57.516 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:57.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:57.243575+0000) 2022-09-08T03:09:57.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:09:57.514+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:09:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:57 smithi055 ceph-mon[97033]: pgmap v10707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:09:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:57 smithi059 ceph-mon[97450]: pgmap v10707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:09:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:58.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:58.282149+0000) 2022-09-08T03:09:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:58.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:58.243843+0000) 2022-09-08T03:09:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:09:59.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:59.282318+0000) 2022-09-08T03:09:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:09:59.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:09:59.244046+0000) 2022-09-08T03:09:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:09:59 smithi055 ceph-mon[97033]: pgmap v10708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:09:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:09:59 smithi059 ceph-mon[97450]: pgmap v10708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:00.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:00.282441+0000) 2022-09-08T03:10:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:00.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:00.244191+0000) 2022-09-08T03:10:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T03:10:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T03:10:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T03:10:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T03:10:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T03:10:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T03:10:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T03:10:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T03:10:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T03:10:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T03:10:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:01.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:01.244385+0000) 2022-09-08T03:10:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:01.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:01.282674+0000) 2022-09-08T03:10:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:01 smithi055 ceph-mon[97033]: pgmap v10709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:10:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:01 smithi059 ceph-mon[97450]: pgmap v10709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:10:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:02.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:02.282907+0000) 2022-09-08T03:10:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:02.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:02.244666+0000) 2022-09-08T03:10:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:03.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:03.283037+0000) 2022-09-08T03:10:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:03.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:03.244889+0000) 2022-09-08T03:10:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:03 smithi055 ceph-mon[97033]: pgmap v10710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:03 smithi059 ceph-mon[97450]: pgmap v10710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:04.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:04.283198+0000) 2022-09-08T03:10:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:04.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:04.245076+0000) 2022-09-08T03:10:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:05.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:05.283458+0000) 2022-09-08T03:10:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:05.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:05.245313+0000) 2022-09-08T03:10:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:05 smithi055 ceph-mon[97033]: pgmap v10711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:10:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:05 smithi059 ceph-mon[97450]: pgmap v10711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:10:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:06.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:06.283710+0000) 2022-09-08T03:10:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:06.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:06.245482+0000) 2022-09-08T03:10:06.872 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:10:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:10:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:07.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:07.283967+0000) 2022-09-08T03:10:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:07.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:07.245677+0000) 2022-09-08T03:10:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:07 smithi055 ceph-mon[97033]: pgmap v10712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:10:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:07 smithi059 ceph-mon[97450]: pgmap v10712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:10:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:08.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:08.284113+0000) 2022-09-08T03:10:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:08.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:08.245852+0000) 2022-09-08T03:10:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:09.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:09.284271+0000) 2022-09-08T03:10:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:09.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:09.246080+0000) 2022-09-08T03:10:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:09 smithi055 ceph-mon[97033]: pgmap v10713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:10:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:09 smithi059 ceph-mon[97450]: pgmap v10713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:10:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:10.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:10.284491+0000) 2022-09-08T03:10:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:10.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:10.246300+0000) 2022-09-08T03:10:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:10 smithi055 ceph-mon[97033]: osdmap e749: 12 total, 11 up, 11 in 2022-09-08T03:10:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:10 smithi059 ceph-mon[97450]: osdmap e749: 12 total, 11 up, 11 in 2022-09-08T03:10:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:11.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:11.284683+0000) 2022-09-08T03:10:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:11.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:11.246529+0000) 2022-09-08T03:10:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:11 smithi055 ceph-mon[97033]: pgmap v10715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:10:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:11 smithi059 ceph-mon[97450]: pgmap v10715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:10:12.524 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:12.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:12.284918+0000) 2022-09-08T03:10:12.524 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:12.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:12.246775+0000) 2022-09-08T03:10:12.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:10:12.522+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:10:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:13.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:13.285147+0000) 2022-09-08T03:10:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:13.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:13.247019+0000) 2022-09-08T03:10:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:13 smithi055 ceph-mon[97033]: pgmap v10716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:10:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:13 smithi059 ceph-mon[97450]: pgmap v10716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:10:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:14.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:14.285352+0000) 2022-09-08T03:10:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:14.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:14.247264+0000) 2022-09-08T03:10:15.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:15.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:15.285588+0000) 2022-09-08T03:10:15.662 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:15.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:15.247474+0000) 2022-09-08T03:10:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:15 smithi055 ceph-mon[97033]: pgmap v10717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:10:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:15 smithi059 ceph-mon[97450]: pgmap v10717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:10:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:16.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:16.285827+0000) 2022-09-08T03:10:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:16.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:16.247732+0000) 2022-09-08T03:10:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:17.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:17.286025+0000) 2022-09-08T03:10:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:17.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:17.247882+0000) 2022-09-08T03:10:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:17 smithi059 ceph-mon[97450]: pgmap v10718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:10:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:17 smithi059 ceph-mon[97450]: pgmap v10719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T03:10:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:17 smithi059 ceph-mon[97450]: pgmap v10720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 280 B/s wr, 8 op/s 2022-09-08T03:10:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:10:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:17 smithi055 ceph-mon[97033]: pgmap v10718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:10:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:17 smithi055 ceph-mon[97033]: pgmap v10719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T03:10:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:17 smithi055 ceph-mon[97033]: pgmap v10720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 280 B/s wr, 8 op/s 2022-09-08T03:10:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:10:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:18.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:18.248101+0000) 2022-09-08T03:10:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:18.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:18.286254+0000) 2022-09-08T03:10:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:19.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:19.286502+0000) 2022-09-08T03:10:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:19.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:19.248322+0000) 2022-09-08T03:10:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:19 smithi055 ceph-mon[97033]: osdmap e750: 12 total, 11 up, 11 in 2022-09-08T03:10:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:19 smithi055 ceph-mon[97033]: pgmap v10722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T03:10:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:19 smithi059 ceph-mon[97450]: osdmap e750: 12 total, 11 up, 11 in 2022-09-08T03:10:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:19 smithi059 ceph-mon[97450]: pgmap v10722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T03:10:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:20.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:20.286727+0000) 2022-09-08T03:10:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:20.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:20.248556+0000) 2022-09-08T03:10:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:21.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:21.286948+0000) 2022-09-08T03:10:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:21.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:21.248736+0000) 2022-09-08T03:10:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:21 smithi055 ceph-mon[97033]: pgmap v10723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.0 KiB/s rd, 164 B/s wr, 9 op/s 2022-09-08T03:10:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:21 smithi059 ceph-mon[97450]: pgmap v10723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.0 KiB/s rd, 164 B/s wr, 9 op/s 2022-09-08T03:10:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:22.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:22.287121+0000) 2022-09-08T03:10:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:22.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:22.248970+0000) 2022-09-08T03:10:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:23.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:23.287345+0000) 2022-09-08T03:10:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:23.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:23.249140+0000) 2022-09-08T03:10:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:23 smithi055 ceph-mon[97033]: pgmap v10724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:10:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:23 smithi059 ceph-mon[97450]: pgmap v10724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:10:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:24.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:24.287570+0000) 2022-09-08T03:10:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:24.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:24.249294+0000) 2022-09-08T03:10:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:25.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:25.287808+0000) 2022-09-08T03:10:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:25.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:25.249515+0000) 2022-09-08T03:10:26.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:25 smithi055 ceph-mon[97033]: pgmap v10725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s 2022-09-08T03:10:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:25 smithi059 ceph-mon[97450]: pgmap v10725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 3 op/s 2022-09-08T03:10:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:26.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:26.288038+0000) 2022-09-08T03:10:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:26.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:26.249742+0000) 2022-09-08T03:10:27.539 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:27.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:27.288262+0000) 2022-09-08T03:10:27.540 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:27.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:27.249983+0000) 2022-09-08T03:10:27.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:10:27.538+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:10:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:27 smithi055 ceph-mon[97033]: pgmap v10726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:10:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:27 smithi059 ceph-mon[97450]: pgmap v10726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:10:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:28.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:28.288463+0000) 2022-09-08T03:10:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:28.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:28.250225+0000) 2022-09-08T03:10:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:29.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:29.288628+0000) 2022-09-08T03:10:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:29.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:29.250431+0000) 2022-09-08T03:10:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:29 smithi055 ceph-mon[97033]: pgmap v10727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:10:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:29 smithi059 ceph-mon[97450]: pgmap v10727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:10:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:30.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:30.288861+0000) 2022-09-08T03:10:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:30.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:30.250576+0000) 2022-09-08T03:10:31.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:31.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:31.250760+0000) 2022-09-08T03:10:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:31.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:31.289093+0000) 2022-09-08T03:10:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:31 smithi055 ceph-mon[97033]: pgmap v10728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:10:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:31 smithi059 ceph-mon[97450]: pgmap v10728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:10:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:32.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:32.289284+0000) 2022-09-08T03:10:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:32.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:32.250980+0000) 2022-09-08T03:10:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:33.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:33.289407+0000) 2022-09-08T03:10:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:33.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:33.251172+0000) 2022-09-08T03:10:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:33 smithi055 ceph-mon[97033]: pgmap v10729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:33 smithi059 ceph-mon[97450]: pgmap v10729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:34.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:34.289590+0000) 2022-09-08T03:10:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:34.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:34.251417+0000) 2022-09-08T03:10:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:35.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:35.289804+0000) 2022-09-08T03:10:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:35.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:35.251694+0000) 2022-09-08T03:10:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:35 smithi055 ceph-mon[97033]: pgmap v10730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:35 smithi059 ceph-mon[97450]: pgmap v10730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:36.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:36.290039+0000) 2022-09-08T03:10:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:36.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:36.251823+0000) 2022-09-08T03:10:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:37.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:37.252041+0000) 2022-09-08T03:10:37.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:37.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:37.290289+0000) 2022-09-08T03:10:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:37 smithi055 ceph-mon[97033]: pgmap v10731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:10:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:37 smithi059 ceph-mon[97450]: pgmap v10731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:10:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:38.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:38.290465+0000) 2022-09-08T03:10:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:38.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:38.252265+0000) 2022-09-08T03:10:39.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:39.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:39.290698+0000) 2022-09-08T03:10:39.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:39.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:39.252551+0000) 2022-09-08T03:10:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:39 smithi055 ceph-mon[97033]: pgmap v10732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:39 smithi059 ceph-mon[97450]: pgmap v10732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:40.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:40.290907+0000) 2022-09-08T03:10:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:40.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:40.252782+0000) 2022-09-08T03:10:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:41.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:41.291115+0000) 2022-09-08T03:10:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:41.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:41.253034+0000) 2022-09-08T03:10:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:41 smithi055 ceph-mon[97033]: pgmap v10733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:10:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:41 smithi059 ceph-mon[97450]: pgmap v10733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:10:42.558 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:42.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:42.291350+0000) 2022-09-08T03:10:42.559 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:42.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:42.253211+0000) 2022-09-08T03:10:42.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:10:42.554+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:10:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:43.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:43.253395+0000) 2022-09-08T03:10:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:43.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:43.291516+0000) 2022-09-08T03:10:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:43 smithi055 ceph-mon[97033]: pgmap v10734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:43 smithi059 ceph-mon[97450]: pgmap v10734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:44.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:44.291762+0000) 2022-09-08T03:10:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:44.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:44.253667+0000) 2022-09-08T03:10:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:45.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:45.291958+0000) 2022-09-08T03:10:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:45.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:45.253956+0000) 2022-09-08T03:10:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:45 smithi055 ceph-mon[97033]: pgmap v10735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:45 smithi059 ceph-mon[97450]: pgmap v10735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:46.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:46.292109+0000) 2022-09-08T03:10:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:46.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:46.254230+0000) 2022-09-08T03:10:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:47.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:47.292344+0000) 2022-09-08T03:10:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:47.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:47.254406+0000) 2022-09-08T03:10:48.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:47 smithi055 ceph-mon[97033]: pgmap v10736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:10:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:47 smithi059 ceph-mon[97450]: pgmap v10736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:10:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:48.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:48.292569+0000) 2022-09-08T03:10:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:48.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:48.254593+0000) 2022-09-08T03:10:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:49.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:49.254839+0000) 2022-09-08T03:10:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:49.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:49.292789+0000) 2022-09-08T03:10:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:49 smithi055 ceph-mon[97033]: pgmap v10737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:49 smithi059 ceph-mon[97450]: pgmap v10737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:50.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:50.292982+0000) 2022-09-08T03:10:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:50.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:50.255004+0000) 2022-09-08T03:10:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:51.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:51.293154+0000) 2022-09-08T03:10:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:51.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:51.255234+0000) 2022-09-08T03:10:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:51 smithi055 ceph-mon[97033]: pgmap v10738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:10:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:51 smithi059 ceph-mon[97450]: pgmap v10738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:10:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:52.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:52.255475+0000) 2022-09-08T03:10:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:52.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:52.293316+0000) 2022-09-08T03:10:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:53.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:53.293544+0000) 2022-09-08T03:10:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:53.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:53.255735+0000) 2022-09-08T03:10:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:53 smithi055 ceph-mon[97033]: pgmap v10739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:53 smithi059 ceph-mon[97450]: pgmap v10739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:54.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:54.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:54.255981+0000) 2022-09-08T03:10:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:54.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:54.293772+0000) 2022-09-08T03:10:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:55.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:55.294012+0000) 2022-09-08T03:10:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:55.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:55.256231+0000) 2022-09-08T03:10:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:55 smithi055 ceph-mon[97033]: pgmap v10740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:56.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:55 smithi059 ceph-mon[97450]: pgmap v10740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:10:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:56.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:56.294224+0000) 2022-09-08T03:10:56.673 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:56.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:56.256481+0000) 2022-09-08T03:10:57.558 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:57.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:57.294367+0000) 2022-09-08T03:10:57.559 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:57.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:57.256739+0000) 2022-09-08T03:10:57.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:10:57.556+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:10:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:57 smithi055 ceph-mon[97033]: pgmap v10741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:10:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:57 smithi059 ceph-mon[97450]: pgmap v10741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:10:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:58.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:58.257011+0000) 2022-09-08T03:10:58.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:58.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:58.294557+0000) 2022-09-08T03:10:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:10:59.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:59.257284+0000) 2022-09-08T03:10:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:10:59.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:10:59.294774+0000) 2022-09-08T03:10:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:10:59 smithi055 ceph-mon[97033]: pgmap v10742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T03:10:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:10:59 smithi059 ceph-mon[97450]: pgmap v10742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T03:11:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:00.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:00.257506+0000) 2022-09-08T03:11:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:00.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:00.294996+0000) 2022-09-08T03:11:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:01.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:01.257762+0000) 2022-09-08T03:11:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:01.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:01.295143+0000) 2022-09-08T03:11:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:01 smithi055 ceph-mon[97033]: pgmap v10743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:11:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:01 smithi059 ceph-mon[97450]: pgmap v10743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:11:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:02.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:02.258016+0000) 2022-09-08T03:11:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:02.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:02.295324+0000) 2022-09-08T03:11:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:03.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:03.295508+0000) 2022-09-08T03:11:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:03.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:03.258295+0000) 2022-09-08T03:11:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:03 smithi055 ceph-mon[97033]: pgmap v10744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T03:11:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:03 smithi059 ceph-mon[97450]: pgmap v10744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T03:11:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:04.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:04.295739+0000) 2022-09-08T03:11:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:04.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:04.258480+0000) 2022-09-08T03:11:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:05.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:05.295959+0000) 2022-09-08T03:11:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:05.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:05.258747+0000) 2022-09-08T03:11:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:05 smithi055 ceph-mon[97033]: pgmap v10745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T03:11:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:11:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:11:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:05 smithi059 ceph-mon[97450]: pgmap v10745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T03:11:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:11:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:11:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:06.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:06.296186+0000) 2022-09-08T03:11:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:06.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:06.258958+0000) 2022-09-08T03:11:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:07.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:07.296409+0000) 2022-09-08T03:11:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:07.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:07.259182+0000) 2022-09-08T03:11:08.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:07 smithi055 ceph-mon[97033]: pgmap v10746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T03:11:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:07 smithi059 ceph-mon[97450]: pgmap v10746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T03:11:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:08.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:08.259444+0000) 2022-09-08T03:11:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:08.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:08.296586+0000) 2022-09-08T03:11:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:09.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:09.296851+0000) 2022-09-08T03:11:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:09.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:09.259589+0000) 2022-09-08T03:11:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:09 smithi055 ceph-mon[97033]: pgmap v10747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:11:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:09 smithi059 ceph-mon[97450]: pgmap v10747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:11:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:10.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:10.297037+0000) 2022-09-08T03:11:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:10.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:10.259808+0000) 2022-09-08T03:11:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:10 smithi055 ceph-mon[97033]: osdmap e751: 12 total, 11 up, 11 in 2022-09-08T03:11:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:10 smithi059 ceph-mon[97450]: osdmap e751: 12 total, 11 up, 11 in 2022-09-08T03:11:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:11.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:11.297254+0000) 2022-09-08T03:11:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:11.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:11.260035+0000) 2022-09-08T03:11:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:11 smithi055 ceph-mon[97033]: pgmap v10749: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:11:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:11 smithi059 ceph-mon[97450]: pgmap v10749: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:11:12.568 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:12.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:12.297398+0000) 2022-09-08T03:11:12.570 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:12.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:12.260305+0000) 2022-09-08T03:11:12.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:11:12.567+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:11:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:13.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:13.297627+0000) 2022-09-08T03:11:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:13.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:13.260524+0000) 2022-09-08T03:11:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:13 smithi055 ceph-mon[97033]: pgmap v10750: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:11:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:13 smithi059 ceph-mon[97450]: pgmap v10750: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:11:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:14.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:14.297822+0000) 2022-09-08T03:11:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:14.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:14.260781+0000) 2022-09-08T03:11:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:15.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:15.297954+0000) 2022-09-08T03:11:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:15.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:15.261019+0000) 2022-09-08T03:11:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:15 smithi055 ceph-mon[97033]: pgmap v10751: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 204 B/s wr, 7 op/s 2022-09-08T03:11:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:15 smithi059 ceph-mon[97450]: pgmap v10751: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 204 B/s wr, 7 op/s 2022-09-08T03:11:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:16.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:16.298092+0000) 2022-09-08T03:11:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:16.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:16.261203+0000) 2022-09-08T03:11:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:17.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:17.298321+0000) 2022-09-08T03:11:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:17.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:17.261375+0000) 2022-09-08T03:11:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:17 smithi055 ceph-mon[97033]: pgmap v10752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:11:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:11:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:11:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:11:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:11:18.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:11:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:17 smithi059 ceph-mon[97450]: pgmap v10752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:11:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:11:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:11:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:11:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:11:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:11:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:18.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:18.298551+0000) 2022-09-08T03:11:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:18.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:18.261542+0000) 2022-09-08T03:11:19.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:19.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:19.298778+0000) 2022-09-08T03:11:19.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:19.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:19.261724+0000) 2022-09-08T03:11:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:19 smithi055 ceph-mon[97033]: osdmap e752: 12 total, 11 up, 11 in 2022-09-08T03:11:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:19 smithi055 ceph-mon[97033]: pgmap v10754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-08T03:11:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:19 smithi059 ceph-mon[97450]: osdmap e752: 12 total, 11 up, 11 in 2022-09-08T03:11:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:19 smithi059 ceph-mon[97450]: pgmap v10754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-08T03:11:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:20.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:20.298923+0000) 2022-09-08T03:11:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:20.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:20.261968+0000) 2022-09-08T03:11:21.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:11:21.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:11:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:11:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:11:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:11:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:11:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:11:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:11:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:11:21.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:21.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:21.262245+0000) 2022-09-08T03:11:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:21.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:21.299118+0000) 2022-09-08T03:11:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:22 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T03:11:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:22 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T03:11:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:22 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T03:11:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:22 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T03:11:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:22 smithi055 ceph-mon[97033]: pgmap v10755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:11:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:22 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T03:11:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:22 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T03:11:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:22 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T03:11:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:22 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T03:11:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:22 smithi059 ceph-mon[97450]: pgmap v10755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:11:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:22.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:22.299309+0000) 2022-09-08T03:11:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:22.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:22.262474+0000) 2022-09-08T03:11:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:23.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:23.299488+0000) 2022-09-08T03:11:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:23.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:23.262732+0000) 2022-09-08T03:11:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:24 smithi055 ceph-mon[97033]: pgmap v10756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:11:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:24 smithi059 ceph-mon[97450]: pgmap v10756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:11:24.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:24.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:24.262933+0000) 2022-09-08T03:11:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:24.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:24.299616+0000) 2022-09-08T03:11:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:25.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:25.299821+0000) 2022-09-08T03:11:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:25.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:25.263114+0000) 2022-09-08T03:11:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:26 smithi055 ceph-mon[97033]: pgmap v10757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:11:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:26 smithi059 ceph-mon[97450]: pgmap v10757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:11:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:26.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:26.300037+0000) 2022-09-08T03:11:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:26.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:26.263323+0000) 2022-09-08T03:11:27.582 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:27.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:27.300171+0000) 2022-09-08T03:11:27.583 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:27.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:27.263553+0000) 2022-09-08T03:11:27.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:11:27.581+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:11:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:28 smithi055 ceph-mon[97033]: pgmap v10758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:11:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:28 smithi059 ceph-mon[97450]: pgmap v10758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:11:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:28.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:28.300338+0000) 2022-09-08T03:11:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:28.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:28.263812+0000) 2022-09-08T03:11:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:29.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:29.300578+0000) 2022-09-08T03:11:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:29.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:29.264050+0000) 2022-09-08T03:11:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:29 smithi055 ceph-mon[97033]: pgmap v10759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:11:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:29 smithi059 ceph-mon[97450]: pgmap v10759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:11:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:30.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:30.300794+0000) 2022-09-08T03:11:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:30.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:30.264286+0000) 2022-09-08T03:11:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:31.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:31.300971+0000) 2022-09-08T03:11:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:31.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:31.264512+0000) 2022-09-08T03:11:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:31 smithi055 ceph-mon[97033]: pgmap v10760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:11:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:31 smithi059 ceph-mon[97450]: pgmap v10760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:11:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:32.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:32.301219+0000) 2022-09-08T03:11:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:32.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:32.264754+0000) 2022-09-08T03:11:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:33.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:33.301452+0000) 2022-09-08T03:11:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:33.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:33.264952+0000) 2022-09-08T03:11:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:33 smithi055 ceph-mon[97033]: pgmap v10761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:11:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:33 smithi059 ceph-mon[97450]: pgmap v10761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:11:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:34.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:34.301724+0000) 2022-09-08T03:11:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:34.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:34.265154+0000) 2022-09-08T03:11:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:35.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:35.301973+0000) 2022-09-08T03:11:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:35.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:35.265417+0000) 2022-09-08T03:11:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:35 smithi055 ceph-mon[97033]: pgmap v10762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:11:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:35 smithi059 ceph-mon[97450]: pgmap v10762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:11:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:36.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:36.302201+0000) 2022-09-08T03:11:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:36.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:36.265763+0000) 2022-09-08T03:11:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:37.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:37.302414+0000) 2022-09-08T03:11:37.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:37.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:37.265977+0000) 2022-09-08T03:11:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:37 smithi055 ceph-mon[97033]: pgmap v10763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:11:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:37 smithi059 ceph-mon[97450]: pgmap v10763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:11:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:38.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:38.302741+0000) 2022-09-08T03:11:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:38.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:38.266188+0000) 2022-09-08T03:11:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:39.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:39.302943+0000) 2022-09-08T03:11:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:39.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:39.266330+0000) 2022-09-08T03:11:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:39 smithi055 ceph-mon[97033]: pgmap v10764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:11:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:39 smithi059 ceph-mon[97450]: pgmap v10764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:11:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:40.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:40.303178+0000) 2022-09-08T03:11:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:40.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:40.266561+0000) 2022-09-08T03:11:41.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:41.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:41.266825+0000) 2022-09-08T03:11:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:41.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:41.303390+0000) 2022-09-08T03:11:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:41 smithi055 ceph-mon[97033]: pgmap v10765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:11:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:41 smithi059 ceph-mon[97450]: pgmap v10765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:11:42.590 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:42.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:42.303647+0000) 2022-09-08T03:11:42.591 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:42.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:42.267075+0000) 2022-09-08T03:11:42.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:11:42.588+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:11:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:43.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:43.303888+0000) 2022-09-08T03:11:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:43.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:43.267297+0000) 2022-09-08T03:11:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:43 smithi055 ceph-mon[97033]: pgmap v10766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:11:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:43 smithi059 ceph-mon[97450]: pgmap v10766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:11:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:44.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:44.304061+0000) 2022-09-08T03:11:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:44.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:44.267576+0000) 2022-09-08T03:11:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:45.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:45.304250+0000) 2022-09-08T03:11:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:45.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:45.267782+0000) 2022-09-08T03:11:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:45 smithi055 ceph-mon[97033]: pgmap v10767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:11:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:45 smithi059 ceph-mon[97450]: pgmap v10767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:11:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:46.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:46.304423+0000) 2022-09-08T03:11:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:46.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:46.268079+0000) 2022-09-08T03:11:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:47.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:47.304615+0000) 2022-09-08T03:11:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:47.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:47.268359+0000) 2022-09-08T03:11:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:48 smithi059 ceph-mon[97450]: pgmap v10768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:11:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:48 smithi059 ceph-mon[97450]: pgmap v10769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:11:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:48 smithi059 ceph-mon[97450]: pgmap v10770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:11:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:11:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:48 smithi055 ceph-mon[97033]: pgmap v10768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:11:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:48 smithi055 ceph-mon[97033]: pgmap v10769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:11:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:48 smithi055 ceph-mon[97033]: pgmap v10770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:11:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:11:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:48.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:48.268673+0000) 2022-09-08T03:11:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:48.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:48.304853+0000) 2022-09-08T03:11:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:49.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:49.305046+0000) 2022-09-08T03:11:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:49.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:49.268958+0000) 2022-09-08T03:11:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:49 smithi055 ceph-mon[97033]: pgmap v10771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:11:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:49 smithi059 ceph-mon[97450]: pgmap v10771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:11:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:50.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:50.305224+0000) 2022-09-08T03:11:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:50.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:50.269202+0000) 2022-09-08T03:11:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:51.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:51.305437+0000) 2022-09-08T03:11:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:51.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:51.269460+0000) 2022-09-08T03:11:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:52 smithi055 ceph-mon[97033]: pgmap v10772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:11:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:52 smithi059 ceph-mon[97450]: pgmap v10772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:11:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:52.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:52.305654+0000) 2022-09-08T03:11:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:52.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:52.269764+0000) 2022-09-08T03:11:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:53.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:53.305872+0000) 2022-09-08T03:11:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:53.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:53.270011+0000) 2022-09-08T03:11:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:54 smithi055 ceph-mon[97033]: pgmap v10773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:11:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:54 smithi059 ceph-mon[97450]: pgmap v10773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:11:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:54.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:54.306047+0000) 2022-09-08T03:11:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:54.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:54.270211+0000) 2022-09-08T03:11:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:55.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:55.306281+0000) 2022-09-08T03:11:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:55.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:55.270435+0000) 2022-09-08T03:11:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:56 smithi055 ceph-mon[97033]: pgmap v10774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:11:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:56 smithi059 ceph-mon[97450]: pgmap v10774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:11:56.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:56.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:56.270715+0000) 2022-09-08T03:11:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:56.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:56.306540+0000) 2022-09-08T03:11:57.592 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:57.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:57.270953+0000) 2022-09-08T03:11:57.592 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:57.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:57.306808+0000) 2022-09-08T03:11:57.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:11:57.591+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:11:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:58 smithi055 ceph-mon[97033]: pgmap v10775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:11:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:58 smithi059 ceph-mon[97450]: pgmap v10775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:11:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:58.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:58.307013+0000) 2022-09-08T03:11:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:58.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:58.271224+0000) 2022-09-08T03:11:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:11:59.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:59.307238+0000) 2022-09-08T03:11:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:11:59.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:11:59.271505+0000) 2022-09-08T03:11:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:11:59 smithi055 ceph-mon[97033]: pgmap v10776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:11:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:11:59 smithi059 ceph-mon[97450]: pgmap v10776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:00.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:00.307500+0000) 2022-09-08T03:12:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:00.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:00.271778+0000) 2022-09-08T03:12:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:01.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:01.307658+0000) 2022-09-08T03:12:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:01.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:01.272019+0000) 2022-09-08T03:12:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:02 smithi055 ceph-mon[97033]: pgmap v10777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:12:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:02 smithi059 ceph-mon[97450]: pgmap v10777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:12:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:02.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:02.307803+0000) 2022-09-08T03:12:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:02.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:02.272292+0000) 2022-09-08T03:12:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:03.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:03.307966+0000) 2022-09-08T03:12:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:03.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:03.272572+0000) 2022-09-08T03:12:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:04 smithi055 ceph-mon[97033]: pgmap v10778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:04 smithi059 ceph-mon[97450]: pgmap v10778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:04.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:04.308109+0000) 2022-09-08T03:12:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:04.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:04.272851+0000) 2022-09-08T03:12:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:05.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:05.273082+0000) 2022-09-08T03:12:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:05.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:05.308323+0000) 2022-09-08T03:12:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:06 smithi055 ceph-mon[97033]: pgmap v10779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:12:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:12:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:12:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:06 smithi059 ceph-mon[97450]: pgmap v10779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:12:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:12:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:12:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:06.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:06.273327+0000) 2022-09-08T03:12:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:06.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:06.308525+0000) 2022-09-08T03:12:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:07.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:07.308748+0000) 2022-09-08T03:12:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:07.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:07.273570+0000) 2022-09-08T03:12:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:08 smithi055 ceph-mon[97033]: pgmap v10780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:12:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:08 smithi059 ceph-mon[97450]: pgmap v10780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:12:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:08.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:08.273800+0000) 2022-09-08T03:12:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:08.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:08.308977+0000) 2022-09-08T03:12:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:09.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:09.309118+0000) 2022-09-08T03:12:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:09.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:09.274065+0000) 2022-09-08T03:12:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:09 smithi055 ceph-mon[97033]: pgmap v10781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:12:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:09 smithi059 ceph-mon[97450]: pgmap v10781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:12:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:10.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:10.309286+0000) 2022-09-08T03:12:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:10.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:10.274274+0000) 2022-09-08T03:12:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:10 smithi055 ceph-mon[97033]: osdmap e753: 12 total, 11 up, 11 in 2022-09-08T03:12:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:10 smithi059 ceph-mon[97450]: osdmap e753: 12 total, 11 up, 11 in 2022-09-08T03:12:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:11.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:11.309405+0000) 2022-09-08T03:12:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:11.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:11.274514+0000) 2022-09-08T03:12:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:11 smithi055 ceph-mon[97033]: pgmap v10783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:12:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:11 smithi059 ceph-mon[97450]: pgmap v10783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:12:12.607 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:12.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:12.309579+0000) 2022-09-08T03:12:12.609 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:12.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:12.274804+0000) 2022-09-08T03:12:12.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:12:12.607+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:12:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:13.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:13.309764+0000) 2022-09-08T03:12:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:13.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:13.275037+0000) 2022-09-08T03:12:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:14 smithi055 ceph-mon[97033]: pgmap v10784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:12:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:14 smithi059 ceph-mon[97450]: pgmap v10784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:12:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:14.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:14.310025+0000) 2022-09-08T03:12:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:14.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:14.275313+0000) 2022-09-08T03:12:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:15.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:15.310191+0000) 2022-09-08T03:12:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:15.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:15.275537+0000) 2022-09-08T03:12:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:16 smithi055 ceph-mon[97033]: pgmap v10785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:12:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:16 smithi059 ceph-mon[97450]: pgmap v10785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:12:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:16.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:16.310404+0000) 2022-09-08T03:12:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:16.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:16.275778+0000) 2022-09-08T03:12:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:17.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:17.310673+0000) 2022-09-08T03:12:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:17.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:17.275967+0000) 2022-09-08T03:12:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:18 smithi055 ceph-mon[97033]: pgmap v10786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:12:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:18 smithi059 ceph-mon[97450]: pgmap v10786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:12:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:18.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:18.310933+0000) 2022-09-08T03:12:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:18.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:18.276221+0000) 2022-09-08T03:12:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:19.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:19.311147+0000) 2022-09-08T03:12:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:19.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:19.276464+0000) 2022-09-08T03:12:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:19 smithi055 ceph-mon[97033]: osdmap e754: 12 total, 11 up, 11 in 2022-09-08T03:12:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:19 smithi055 ceph-mon[97033]: pgmap v10788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T03:12:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:19 smithi059 ceph-mon[97450]: osdmap e754: 12 total, 11 up, 11 in 2022-09-08T03:12:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:19 smithi059 ceph-mon[97450]: pgmap v10788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T03:12:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:20.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:20.311334+0000) 2022-09-08T03:12:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:20.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:20.276608+0000) 2022-09-08T03:12:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:21.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:21.311578+0000) 2022-09-08T03:12:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:21.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:21.276838+0000) 2022-09-08T03:12:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:22 smithi055 ceph-mon[97033]: pgmap v10789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:12:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:22 smithi059 ceph-mon[97450]: pgmap v10789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:12:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:22.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:22.311796+0000) 2022-09-08T03:12:22.686 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:22.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:22.277089+0000) 2022-09-08T03:12:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:23.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:23.311972+0000) 2022-09-08T03:12:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:23.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:23.277293+0000) 2022-09-08T03:12:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:24 smithi055 ceph-mon[97033]: pgmap v10790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:12:24.538 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:24 smithi059 ceph-mon[97450]: pgmap v10790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:12:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:24.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:24.312116+0000) 2022-09-08T03:12:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:24.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:24.277535+0000) 2022-09-08T03:12:25.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:25.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:25.277740+0000) 2022-09-08T03:12:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:25.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:25.312359+0000) 2022-09-08T03:12:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:26 smithi055 ceph-mon[97033]: pgmap v10791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:12:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:26 smithi059 ceph-mon[97450]: pgmap v10791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:12:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:26.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:26.312526+0000) 2022-09-08T03:12:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:26.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:26.277999+0000) 2022-09-08T03:12:27.613 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:27.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:27.312695+0000) 2022-09-08T03:12:27.615 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:27.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:27.278284+0000) 2022-09-08T03:12:27.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:12:27.611+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:12:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:28 smithi055 ceph-mon[97033]: pgmap v10792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:12:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:28 smithi059 ceph-mon[97450]: pgmap v10792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:12:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:28.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:28.312867+0000) 2022-09-08T03:12:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:28.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:28.278480+0000) 2022-09-08T03:12:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:29.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:29.313092+0000) 2022-09-08T03:12:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:29.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:29.278717+0000) 2022-09-08T03:12:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:29 smithi055 ceph-mon[97033]: pgmap v10793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:12:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:29 smithi059 ceph-mon[97450]: pgmap v10793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:12:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:30.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:30.313322+0000) 2022-09-08T03:12:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:30.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:30.278970+0000) 2022-09-08T03:12:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:31.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:31.313526+0000) 2022-09-08T03:12:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:31.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:31.279172+0000) 2022-09-08T03:12:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:32 smithi055 ceph-mon[97033]: pgmap v10794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:12:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:32 smithi059 ceph-mon[97450]: pgmap v10794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:12:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:32.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:32.313696+0000) 2022-09-08T03:12:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:32.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:32.279382+0000) 2022-09-08T03:12:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:33.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:33.313927+0000) 2022-09-08T03:12:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:33.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:33.279623+0000) 2022-09-08T03:12:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:34 smithi055 ceph-mon[97033]: pgmap v10795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:34 smithi059 ceph-mon[97450]: pgmap v10795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:34.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:34.314138+0000) 2022-09-08T03:12:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:34.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:34.279914+0000) 2022-09-08T03:12:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:35.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:35.314342+0000) 2022-09-08T03:12:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:35.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:35.280144+0000) 2022-09-08T03:12:36.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:36 smithi055 ceph-mon[97033]: pgmap v10796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:12:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:36 smithi059 ceph-mon[97450]: pgmap v10796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:12:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:36.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:36.314490+0000) 2022-09-08T03:12:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:36.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:36.280330+0000) 2022-09-08T03:12:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:37.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:37.314674+0000) 2022-09-08T03:12:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:37.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:37.280579+0000) 2022-09-08T03:12:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:38 smithi055 ceph-mon[97033]: pgmap v10797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:38 smithi059 ceph-mon[97450]: pgmap v10797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:38.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:38.314844+0000) 2022-09-08T03:12:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:38.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:38.280763+0000) 2022-09-08T03:12:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:39.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:39.315093+0000) 2022-09-08T03:12:39.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:39.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:39.281009+0000) 2022-09-08T03:12:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:39 smithi059 ceph-mon[97450]: pgmap v10798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:39 smithi055 ceph-mon[97033]: pgmap v10798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:40.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:40.315322+0000) 2022-09-08T03:12:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:40.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:40.281237+0000) 2022-09-08T03:12:41.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:41.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:41.281478+0000) 2022-09-08T03:12:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:41.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:41.315485+0000) 2022-09-08T03:12:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:42 smithi055 ceph-mon[97033]: pgmap v10799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:12:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:42 smithi059 ceph-mon[97450]: pgmap v10799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:12:42.622 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:42.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:42.315675+0000) 2022-09-08T03:12:42.623 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:42.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:42.281744+0000) 2022-09-08T03:12:42.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:12:42.621+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:12:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:43.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:43.315894+0000) 2022-09-08T03:12:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:43.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:43.281930+0000) 2022-09-08T03:12:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:44 smithi055 ceph-mon[97033]: pgmap v10800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:44 smithi059 ceph-mon[97450]: pgmap v10800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:44.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:44.316107+0000) 2022-09-08T03:12:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:44.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:44.282105+0000) 2022-09-08T03:12:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:45.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:45.282298+0000) 2022-09-08T03:12:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:45.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:45.316233+0000) 2022-09-08T03:12:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:46 smithi055 ceph-mon[97033]: pgmap v10801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:12:46.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:46 smithi059 ceph-mon[97450]: pgmap v10801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:12:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:46.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:46.316354+0000) 2022-09-08T03:12:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:46.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:46.282498+0000) 2022-09-08T03:12:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:47.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:47.316579+0000) 2022-09-08T03:12:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:47.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:47.282752+0000) 2022-09-08T03:12:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:48 smithi055 ceph-mon[97033]: pgmap v10802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:12:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:12:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:12:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:12:48.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:12:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:48 smithi059 ceph-mon[97450]: pgmap v10802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:12:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:12:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:12:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:12:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:12:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:48.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:48.316772+0000) 2022-09-08T03:12:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:48.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:48.282968+0000) 2022-09-08T03:12:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:49.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:49.317008+0000) 2022-09-08T03:12:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:49.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:49.283203+0000) 2022-09-08T03:12:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:49 smithi055 ceph-mon[97033]: pgmap v10803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:49 smithi059 ceph-mon[97450]: pgmap v10803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:50.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:50.317307+0000) 2022-09-08T03:12:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:50.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:50.283440+0000) 2022-09-08T03:12:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:51.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:51.317577+0000) 2022-09-08T03:12:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:51.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:51.283685+0000) 2022-09-08T03:12:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:52 smithi055 ceph-mon[97033]: pgmap v10804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:12:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:52 smithi059 ceph-mon[97450]: pgmap v10804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:12:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:52.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:52.283931+0000) 2022-09-08T03:12:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:52.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:52.317912+0000) 2022-09-08T03:12:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:53.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:53.318154+0000) 2022-09-08T03:12:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:53.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:53.284168+0000) 2022-09-08T03:12:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:54 smithi055 ceph-mon[97033]: pgmap v10805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:54 smithi059 ceph-mon[97450]: pgmap v10805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:54.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:54.318305+0000) 2022-09-08T03:12:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:54.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:54.284479+0000) 2022-09-08T03:12:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:55.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:55.318434+0000) 2022-09-08T03:12:55.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:55.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:55.284792+0000) 2022-09-08T03:12:56.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:56 smithi055 ceph-mon[97033]: pgmap v10806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:12:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:56 smithi059 ceph-mon[97450]: pgmap v10806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:12:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:56.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:56.318632+0000) 2022-09-08T03:12:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:56.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:56.285014+0000) 2022-09-08T03:12:57.624 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:57.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:57.318881+0000) 2022-09-08T03:12:57.625 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:57.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:57.285228+0000) 2022-09-08T03:12:57.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:12:57.623+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:12:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:58 smithi055 ceph-mon[97033]: pgmap v10807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:58 smithi059 ceph-mon[97450]: pgmap v10807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:58.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:58.319068+0000) 2022-09-08T03:12:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:58.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:58.285530+0000) 2022-09-08T03:12:59.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:12:59.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:59.285803+0000) 2022-09-08T03:12:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:12:59.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:12:59.319263+0000) 2022-09-08T03:12:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:12:59 smithi059 ceph-mon[97450]: pgmap v10808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:12:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:12:59 smithi055 ceph-mon[97033]: pgmap v10808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:00.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:00.319531+0000) 2022-09-08T03:13:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:00.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:00.286010+0000) 2022-09-08T03:13:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:01.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:01.319810+0000) 2022-09-08T03:13:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:01.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:01.286149+0000) 2022-09-08T03:13:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:02 smithi055 ceph-mon[97033]: pgmap v10809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:02 smithi059 ceph-mon[97450]: pgmap v10809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:02.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:02.320007+0000) 2022-09-08T03:13:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:02.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:02.286373+0000) 2022-09-08T03:13:03.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:03.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:03.320230+0000) 2022-09-08T03:13:03.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:03.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:03.286556+0000) 2022-09-08T03:13:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:04 smithi055 ceph-mon[97033]: pgmap v10810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:04 smithi059 ceph-mon[97450]: pgmap v10810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:04.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:04.320463+0000) 2022-09-08T03:13:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:04.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:04.286779+0000) 2022-09-08T03:13:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:05.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:05.320719+0000) 2022-09-08T03:13:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:05.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:05.286930+0000) 2022-09-08T03:13:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:06 smithi055 ceph-mon[97033]: pgmap v10811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:13:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:13:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:13:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:06 smithi059 ceph-mon[97450]: pgmap v10811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:13:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:13:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:13:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:06.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:06.320982+0000) 2022-09-08T03:13:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:06.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:06.287158+0000) 2022-09-08T03:13:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:07.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:07.321188+0000) 2022-09-08T03:13:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:07.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:07.287326+0000) 2022-09-08T03:13:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:08 smithi055 ceph-mon[97033]: pgmap v10812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:13:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:08 smithi059 ceph-mon[97450]: pgmap v10812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:13:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:08.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:08.321358+0000) 2022-09-08T03:13:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:08.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:08.287538+0000) 2022-09-08T03:13:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:09.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:09.321480+0000) 2022-09-08T03:13:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:09.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:09.287760+0000) 2022-09-08T03:13:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:09 smithi059 ceph-mon[97450]: pgmap v10813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:13:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:09 smithi055 ceph-mon[97033]: pgmap v10813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:13:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:10.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:10.321663+0000) 2022-09-08T03:13:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:10.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:10.287956+0000) 2022-09-08T03:13:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:10 smithi059 ceph-mon[97450]: osdmap e755: 12 total, 11 up, 11 in 2022-09-08T03:13:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:10 smithi055 ceph-mon[97033]: osdmap e755: 12 total, 11 up, 11 in 2022-09-08T03:13:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:11.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:11.321852+0000) 2022-09-08T03:13:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:11.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:11.288100+0000) 2022-09-08T03:13:11.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:11 smithi059 ceph-mon[97450]: pgmap v10815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:13:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:11 smithi055 ceph-mon[97033]: pgmap v10815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:13:12.640 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:12.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:12.322072+0000) 2022-09-08T03:13:12.641 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:12.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:12.288278+0000) 2022-09-08T03:13:12.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:13:12.638+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:13:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:13.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:13.322324+0000) 2022-09-08T03:13:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:13.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:13.288503+0000) 2022-09-08T03:13:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:14 smithi055 ceph-mon[97033]: pgmap v10816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:13:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:14 smithi059 ceph-mon[97450]: pgmap v10816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:13:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:14.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:14.322573+0000) 2022-09-08T03:13:14.664 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:14.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:14.288724+0000) 2022-09-08T03:13:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:15.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:15.322756+0000) 2022-09-08T03:13:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:15.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:15.288881+0000) 2022-09-08T03:13:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:16 smithi055 ceph-mon[97033]: pgmap v10817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:13:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:16 smithi059 ceph-mon[97450]: pgmap v10817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:13:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:16.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:16.323019+0000) 2022-09-08T03:13:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:16.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:16.289062+0000) 2022-09-08T03:13:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:17.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:17.323218+0000) 2022-09-08T03:13:17.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:17.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:17.289265+0000) 2022-09-08T03:13:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:18.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:18.323467+0000) 2022-09-08T03:13:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:18.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:18.289484+0000) 2022-09-08T03:13:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:18 smithi055 ceph-mon[97033]: pgmap v10818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:13:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:18 smithi055 ceph-mon[97033]: pgmap v10819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T03:13:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:18 smithi055 ceph-mon[97033]: pgmap v10820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 262 B/s wr, 8 op/s 2022-09-08T03:13:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:13:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:18 smithi059 ceph-mon[97450]: pgmap v10818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:13:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:18 smithi059 ceph-mon[97450]: pgmap v10819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T03:13:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:18 smithi059 ceph-mon[97450]: pgmap v10820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 262 B/s wr, 8 op/s 2022-09-08T03:13:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:13:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:19.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:19.323705+0000) 2022-09-08T03:13:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:19.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:19.289746+0000) 2022-09-08T03:13:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:19 smithi059 ceph-mon[97450]: osdmap e756: 12 total, 11 up, 11 in 2022-09-08T03:13:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:19 smithi059 ceph-mon[97450]: pgmap v10822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T03:13:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:19 smithi055 ceph-mon[97033]: osdmap e756: 12 total, 11 up, 11 in 2022-09-08T03:13:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:19 smithi055 ceph-mon[97033]: pgmap v10822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T03:13:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:20.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:20.323945+0000) 2022-09-08T03:13:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:20.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:20.290002+0000) 2022-09-08T03:13:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:21.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:21.324167+0000) 2022-09-08T03:13:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:21.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:21.290246+0000) 2022-09-08T03:13:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:21 smithi055 ceph-mon[97033]: pgmap v10823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:13:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:21 smithi059 ceph-mon[97450]: pgmap v10823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:13:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:22.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:22.324409+0000) 2022-09-08T03:13:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:22.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:22.290473+0000) 2022-09-08T03:13:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:23.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:23.324672+0000) 2022-09-08T03:13:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:23.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:23.290630+0000) 2022-09-08T03:13:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:23 smithi055 ceph-mon[97033]: pgmap v10824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:13:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:23 smithi059 ceph-mon[97450]: pgmap v10824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:13:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:24.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:24.324874+0000) 2022-09-08T03:13:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:24.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:24.290836+0000) 2022-09-08T03:13:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:25.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:25.325035+0000) 2022-09-08T03:13:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:25.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:25.291057+0000) 2022-09-08T03:13:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:25 smithi055 ceph-mon[97033]: pgmap v10825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:25 smithi059 ceph-mon[97450]: pgmap v10825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:26.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:26.325198+0000) 2022-09-08T03:13:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:26.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:26.291255+0000) 2022-09-08T03:13:27.641 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:27.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:27.325425+0000) 2022-09-08T03:13:27.642 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:27.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:27.291470+0000) 2022-09-08T03:13:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:27 smithi055 ceph-mon[97033]: pgmap v10826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:13:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:27 smithi059 ceph-mon[97450]: pgmap v10826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:13:27.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:13:27.640+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:13:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:28.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:28.325693+0000) 2022-09-08T03:13:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:28.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:28.291629+0000) 2022-09-08T03:13:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:29.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:29.325941+0000) 2022-09-08T03:13:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:29.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:29.291869+0000) 2022-09-08T03:13:29.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:29 smithi059 ceph-mon[97450]: pgmap v10827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:13:29.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:29 smithi055 ceph-mon[97033]: pgmap v10827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:13:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:30.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:30.326085+0000) 2022-09-08T03:13:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:30.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:30.292053+0000) 2022-09-08T03:13:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:31.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:31.326227+0000) 2022-09-08T03:13:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:31.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:31.292238+0000) 2022-09-08T03:13:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:31 smithi055 ceph-mon[97033]: pgmap v10828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:31 smithi059 ceph-mon[97450]: pgmap v10828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:32.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:32.292468+0000) 2022-09-08T03:13:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:32.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:32.326474+0000) 2022-09-08T03:13:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:33.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:33.292764+0000) 2022-09-08T03:13:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:33.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:33.326687+0000) 2022-09-08T03:13:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:33 smithi055 ceph-mon[97033]: pgmap v10829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:33 smithi059 ceph-mon[97450]: pgmap v10829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:34.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:34.293005+0000) 2022-09-08T03:13:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:34.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:34.326918+0000) 2022-09-08T03:13:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:35.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:35.327149+0000) 2022-09-08T03:13:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:35.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:35.293240+0000) 2022-09-08T03:13:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:35 smithi055 ceph-mon[97033]: pgmap v10830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:35 smithi059 ceph-mon[97450]: pgmap v10830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:36.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:36.327384+0000) 2022-09-08T03:13:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:36.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:36.293464+0000) 2022-09-08T03:13:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:37.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:37.327721+0000) 2022-09-08T03:13:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:37.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:37.293725+0000) 2022-09-08T03:13:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:37 smithi055 ceph-mon[97033]: pgmap v10831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:37 smithi059 ceph-mon[97450]: pgmap v10831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:38.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:38.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:38.327912+0000) 2022-09-08T03:13:38.662 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:38.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:38.294028+0000) 2022-09-08T03:13:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:39.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:39.328035+0000) 2022-09-08T03:13:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:39.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:39.294254+0000) 2022-09-08T03:13:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:39 smithi059 ceph-mon[97450]: pgmap v10832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:39 smithi055 ceph-mon[97033]: pgmap v10832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:40.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:40.328197+0000) 2022-09-08T03:13:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:40.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:40.294426+0000) 2022-09-08T03:13:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:41.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:41.328449+0000) 2022-09-08T03:13:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:41.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:41.294693+0000) 2022-09-08T03:13:41.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:41 smithi055 ceph-mon[97033]: pgmap v10833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:41 smithi059 ceph-mon[97450]: pgmap v10833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:42.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:42.328690+0000) 2022-09-08T03:13:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:42.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:42.294889+0000) 2022-09-08T03:13:43.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:13:42.655+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:13:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:43.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:43.295135+0000) 2022-09-08T03:13:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:43.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:43.328874+0000) 2022-09-08T03:13:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:43 smithi055 ceph-mon[97033]: pgmap v10834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:43 smithi059 ceph-mon[97450]: pgmap v10834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:44.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:44.329140+0000) 2022-09-08T03:13:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:44.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:44.295404+0000) 2022-09-08T03:13:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:45.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:45.329406+0000) 2022-09-08T03:13:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:45.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:45.295557+0000) 2022-09-08T03:13:45.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:45 smithi055 ceph-mon[97033]: pgmap v10835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:45 smithi059 ceph-mon[97450]: pgmap v10835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:46.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:46.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:46.295801+0000) 2022-09-08T03:13:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:46.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:46.329656+0000) 2022-09-08T03:13:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:47.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:47.329849+0000) 2022-09-08T03:13:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:47.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:47.295980+0000) 2022-09-08T03:13:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:47 smithi055 ceph-mon[97033]: pgmap v10836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:47 smithi059 ceph-mon[97450]: pgmap v10836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:48.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:48.330066+0000) 2022-09-08T03:13:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:48.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:48.296236+0000) 2022-09-08T03:13:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:49.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:49.330329+0000) 2022-09-08T03:13:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:49.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:49.296504+0000) 2022-09-08T03:13:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:49 smithi059 ceph-mon[97450]: pgmap v10837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:49 smithi055 ceph-mon[97033]: pgmap v10837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:50.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:50.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:50.296743+0000) 2022-09-08T03:13:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:50.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:50.330486+0000) 2022-09-08T03:13:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:51.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:51.330715+0000) 2022-09-08T03:13:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:51.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:51.296984+0000) 2022-09-08T03:13:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:51 smithi055 ceph-mon[97033]: pgmap v10838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:51 smithi059 ceph-mon[97450]: pgmap v10838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:52.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:52.297219+0000) 2022-09-08T03:13:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:52.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:52.330916+0000) 2022-09-08T03:13:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:53.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:53.331137+0000) 2022-09-08T03:13:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:53.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:53.297487+0000) 2022-09-08T03:13:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:53 smithi055 ceph-mon[97033]: pgmap v10839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:53 smithi059 ceph-mon[97450]: pgmap v10839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:54.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:54.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:54.297736+0000) 2022-09-08T03:13:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:54.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:54.331344+0000) 2022-09-08T03:13:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:55.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:55.331572+0000) 2022-09-08T03:13:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:55.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:55.297994+0000) 2022-09-08T03:13:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:55 smithi055 ceph-mon[97033]: pgmap v10840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:55 smithi059 ceph-mon[97450]: pgmap v10840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:13:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:56.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:56.331797+0000) 2022-09-08T03:13:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:56.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:56.298265+0000) 2022-09-08T03:13:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:57.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:57.331994+0000) 2022-09-08T03:13:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:57.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:57.298532+0000) 2022-09-08T03:13:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:57 smithi055 ceph-mon[97033]: pgmap v10841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:57 smithi059 ceph-mon[97450]: pgmap v10841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:58.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:13:57.671+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:13:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:58.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:58.332193+0000) 2022-09-08T03:13:58.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:58.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:58.298725+0000) 2022-09-08T03:13:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:13:59.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:59.332335+0000) 2022-09-08T03:13:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:13:59.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:13:59.298961+0000) 2022-09-08T03:13:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:13:59 smithi059 ceph-mon[97450]: pgmap v10842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:13:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:13:59 smithi055 ceph-mon[97033]: pgmap v10842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:14:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:00.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:00.332473+0000) 2022-09-08T03:14:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:00.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:00.299115+0000) 2022-09-08T03:14:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:01.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:01.332671+0000) 2022-09-08T03:14:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:01.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:01.299342+0000) 2022-09-08T03:14:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:01 smithi055 ceph-mon[97033]: pgmap v10843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:14:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:01 smithi059 ceph-mon[97450]: pgmap v10843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:14:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:02.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:02.332827+0000) 2022-09-08T03:14:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:02.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:02.299593+0000) 2022-09-08T03:14:03.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:03.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:03.299808+0000) 2022-09-08T03:14:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:03.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:03.332998+0000) 2022-09-08T03:14:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:03 smithi055 ceph-mon[97033]: pgmap v10844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:14:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:03 smithi059 ceph-mon[97450]: pgmap v10844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:14:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:04.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:04.299960+0000) 2022-09-08T03:14:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:04.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:04.333227+0000) 2022-09-08T03:14:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:05.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:05.333457+0000) 2022-09-08T03:14:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:05.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:05.300158+0000) 2022-09-08T03:14:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:05 smithi055 ceph-mon[97033]: pgmap v10845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:14:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:05 smithi059 ceph-mon[97450]: pgmap v10845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:14:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:06.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:06.333681+0000) 2022-09-08T03:14:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:06.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:06.300396+0000) 2022-09-08T03:14:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:14:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:14:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:14:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:14:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:07.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:07.333888+0000) 2022-09-08T03:14:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:07.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:07.300625+0000) 2022-09-08T03:14:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:07 smithi055 ceph-mon[97033]: pgmap v10846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:14:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:07 smithi059 ceph-mon[97450]: pgmap v10846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:14:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:08.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:08.300930+0000) 2022-09-08T03:14:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:08.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:08.334145+0000) 2022-09-08T03:14:09.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:09.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:09.301120+0000) 2022-09-08T03:14:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:09.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:09.334387+0000) 2022-09-08T03:14:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:09 smithi059 ceph-mon[97450]: pgmap v10847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:14:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:09 smithi055 ceph-mon[97033]: pgmap v10847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:14:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:10.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:10.334565+0000) 2022-09-08T03:14:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:10.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:10.301330+0000) 2022-09-08T03:14:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:10 smithi059 ceph-mon[97450]: osdmap e757: 12 total, 11 up, 11 in 2022-09-08T03:14:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:10 smithi055 ceph-mon[97033]: osdmap e757: 12 total, 11 up, 11 in 2022-09-08T03:14:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:11.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:11.334788+0000) 2022-09-08T03:14:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:11.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:11.301558+0000) 2022-09-08T03:14:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:11 smithi059 ceph-mon[97450]: pgmap v10849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:14:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:11 smithi055 ceph-mon[97033]: pgmap v10849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:14:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:12.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:12.334975+0000) 2022-09-08T03:14:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:12.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:12.301701+0000) 2022-09-08T03:14:13.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:14:12.683+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:14:13.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:13.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:13.301918+0000) 2022-09-08T03:14:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:13.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:13.335160+0000) 2022-09-08T03:14:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:13 smithi055 ceph-mon[97033]: pgmap v10850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:14:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:13 smithi059 ceph-mon[97450]: pgmap v10850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:14:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:14.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:14.302179+0000) 2022-09-08T03:14:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:14.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:14.335373+0000) 2022-09-08T03:14:15.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:15.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:15.302411+0000) 2022-09-08T03:14:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:15.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:15.335622+0000) 2022-09-08T03:14:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:15 smithi055 ceph-mon[97033]: pgmap v10851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:14:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:15 smithi059 ceph-mon[97450]: pgmap v10851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:14:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:16.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:16.335855+0000) 2022-09-08T03:14:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:16.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:16.302687+0000) 2022-09-08T03:14:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:17.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:17.336094+0000) 2022-09-08T03:14:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:17.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:17.302904+0000) 2022-09-08T03:14:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:17 smithi055 ceph-mon[97033]: pgmap v10852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:14:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:17 smithi059 ceph-mon[97450]: pgmap v10852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:14:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:18.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:18.336308+0000) 2022-09-08T03:14:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:18.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:18.303134+0000) 2022-09-08T03:14:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:14:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:14:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:14:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:14:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:14:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:14:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:19.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:19.336515+0000) 2022-09-08T03:14:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:19.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:19.303323+0000) 2022-09-08T03:14:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:19 smithi059 ceph-mon[97450]: osdmap e758: 12 total, 11 up, 11 in 2022-09-08T03:14:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:19 smithi059 ceph-mon[97450]: pgmap v10854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2022-09-08T03:14:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:19 smithi055 ceph-mon[97033]: osdmap e758: 12 total, 11 up, 11 in 2022-09-08T03:14:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:19 smithi055 ceph-mon[97033]: pgmap v10854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2022-09-08T03:14:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:20.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:20.303547+0000) 2022-09-08T03:14:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:20.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:20.336709+0000) 2022-09-08T03:14:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:21.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:21.336879+0000) 2022-09-08T03:14:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:21.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:21.303801+0000) 2022-09-08T03:14:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:21 smithi055 ceph-mon[97033]: pgmap v10855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:14:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:21 smithi059 ceph-mon[97450]: pgmap v10855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:14:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:22.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:22.337086+0000) 2022-09-08T03:14:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:22.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:22.303966+0000) 2022-09-08T03:14:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:23.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:23.337271+0000) 2022-09-08T03:14:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:23.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:23.304193+0000) 2022-09-08T03:14:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:23 smithi055 ceph-mon[97033]: pgmap v10856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:14:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:23 smithi059 ceph-mon[97450]: pgmap v10856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:14:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:24.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:24.337451+0000) 2022-09-08T03:14:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:24.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:24.304424+0000) 2022-09-08T03:14:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:25.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:25.337712+0000) 2022-09-08T03:14:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:25.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:25.304689+0000) 2022-09-08T03:14:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:25 smithi055 ceph-mon[97033]: pgmap v10857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:14:25.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:25 smithi059 ceph-mon[97450]: pgmap v10857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:14:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:26.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:26.337989+0000) 2022-09-08T03:14:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:26.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:26.304934+0000) 2022-09-08T03:14:27.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:27.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:27.305157+0000) 2022-09-08T03:14:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:27.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:27.338204+0000) 2022-09-08T03:14:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:27 smithi055 ceph-mon[97033]: pgmap v10858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:14:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:27 smithi059 ceph-mon[97450]: pgmap v10858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:14:28.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:14:27.687+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:14:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:28.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:28.338384+0000) 2022-09-08T03:14:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:28.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:28.305338+0000) 2022-09-08T03:14:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:29.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:29.338608+0000) 2022-09-08T03:14:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:29.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:29.305536+0000) 2022-09-08T03:14:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:29 smithi059 ceph-mon[97450]: pgmap v10859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:14:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:29 smithi055 ceph-mon[97033]: pgmap v10859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:14:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:30.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:30.338812+0000) 2022-09-08T03:14:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:30.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:30.305723+0000) 2022-09-08T03:14:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:31.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:31.338964+0000) 2022-09-08T03:14:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:31.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:31.305919+0000) 2022-09-08T03:14:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:31 smithi055 ceph-mon[97033]: pgmap v10860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:14:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:31 smithi059 ceph-mon[97450]: pgmap v10860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:14:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:32.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:32.339198+0000) 2022-09-08T03:14:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:32.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:32.306147+0000) 2022-09-08T03:14:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:33.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:33.306388+0000) 2022-09-08T03:14:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:33.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:33.339446+0000) 2022-09-08T03:14:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:33 smithi055 ceph-mon[97033]: pgmap v10861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:14:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:33 smithi059 ceph-mon[97450]: pgmap v10861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:14:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:34.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:34.339679+0000) 2022-09-08T03:14:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:34.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:34.306692+0000) 2022-09-08T03:14:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:35.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:35.339850+0000) 2022-09-08T03:14:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:35.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:35.306879+0000) 2022-09-08T03:14:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:35 smithi055 ceph-mon[97033]: pgmap v10862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:14:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:35 smithi059 ceph-mon[97450]: pgmap v10862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:14:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:36.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:36.340064+0000) 2022-09-08T03:14:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:36.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:36.307085+0000) 2022-09-08T03:14:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:37.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:37.340223+0000) 2022-09-08T03:14:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:37.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:37.307319+0000) 2022-09-08T03:14:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:37 smithi055 ceph-mon[97033]: pgmap v10863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:14:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:37 smithi059 ceph-mon[97450]: pgmap v10863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:14:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:38.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:38.340438+0000) 2022-09-08T03:14:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:38.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:38.307565+0000) 2022-09-08T03:14:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:39.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:39.340709+0000) 2022-09-08T03:14:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:39.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:39.307739+0000) 2022-09-08T03:14:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:39 smithi059 ceph-mon[97450]: pgmap v10864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:14:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:39 smithi055 ceph-mon[97033]: pgmap v10864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:14:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:40.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:40.340890+0000) 2022-09-08T03:14:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:40.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:40.307955+0000) 2022-09-08T03:14:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:41.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:41.341083+0000) 2022-09-08T03:14:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:41.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:41.308184+0000) 2022-09-08T03:14:41.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:41 smithi055 ceph-mon[97033]: pgmap v10865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:14:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:41 smithi059 ceph-mon[97450]: pgmap v10865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:14:42.567 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:42.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:42.341279+0000) 2022-09-08T03:14:42.568 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:42.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:42.308378+0000) 2022-09-08T03:14:42.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:14:42.700+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:14:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:43.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:43.341498+0000) 2022-09-08T03:14:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:43.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:43.308617+0000) 2022-09-08T03:14:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:43 smithi055 ceph-mon[97033]: pgmap v10866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:14:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:43 smithi059 ceph-mon[97450]: pgmap v10866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:14:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:44.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:44.341653+0000) 2022-09-08T03:14:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:44.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:44.308901+0000) 2022-09-08T03:14:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:45.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:45.341795+0000) 2022-09-08T03:14:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:45.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:45.309133+0000) 2022-09-08T03:14:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:45 smithi055 ceph-mon[97033]: pgmap v10867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:14:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:45 smithi059 ceph-mon[97450]: pgmap v10867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:14:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:46.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:46.341943+0000) 2022-09-08T03:14:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:46.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:46.309336+0000) 2022-09-08T03:14:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:47.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:47.342139+0000) 2022-09-08T03:14:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:47.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:47.309494+0000) 2022-09-08T03:14:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:47 smithi055 ceph-mon[97033]: pgmap v10868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:14:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:47 smithi059 ceph-mon[97450]: pgmap v10868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:14:48.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:48.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:48.309680+0000) 2022-09-08T03:14:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:48.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:48.342355+0000) 2022-09-08T03:14:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:48 smithi059 ceph-mon[97450]: pgmap v10869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:14:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:48 smithi059 ceph-mon[97450]: pgmap v10870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:14:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:14:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:48 smithi055 ceph-mon[97033]: pgmap v10869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:14:48.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:48 smithi055 ceph-mon[97033]: pgmap v10870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:14:48.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:14:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:49.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:49.342530+0000) 2022-09-08T03:14:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:49.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:49.309909+0000) 2022-09-08T03:14:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:50.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:50.342737+0000) 2022-09-08T03:14:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:50.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:50.310109+0000) 2022-09-08T03:14:50.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:50 smithi055 ceph-mon[97033]: pgmap v10871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:14:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:50 smithi059 ceph-mon[97450]: pgmap v10871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:14:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:51.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:51.342919+0000) 2022-09-08T03:14:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:51.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:51.310358+0000) 2022-09-08T03:14:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:52.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:52.343141+0000) 2022-09-08T03:14:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:52.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:52.310569+0000) 2022-09-08T03:14:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:52 smithi055 ceph-mon[97033]: pgmap v10872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:14:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:52 smithi059 ceph-mon[97450]: pgmap v10872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:14:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:53.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:53.343346+0000) 2022-09-08T03:14:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:53.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:53.310789+0000) 2022-09-08T03:14:54.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:54.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:54.311044+0000) 2022-09-08T03:14:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:54.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:54.343609+0000) 2022-09-08T03:14:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:54 smithi059 ceph-mon[97450]: pgmap v10873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:14:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:54 smithi055 ceph-mon[97033]: pgmap v10873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:14:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:55.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:55.343868+0000) 2022-09-08T03:14:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:55.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:55.311298+0000) 2022-09-08T03:14:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:56.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:56.311535+0000) 2022-09-08T03:14:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:56.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:56.344011+0000) 2022-09-08T03:14:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:56 smithi055 ceph-mon[97033]: pgmap v10874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:14:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:56 smithi059 ceph-mon[97450]: pgmap v10874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:14:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:57.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:57.344227+0000) 2022-09-08T03:14:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:57.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:57.311714+0000) 2022-09-08T03:14:58.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:14:57.710+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:14:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:58.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:58.344450+0000) 2022-09-08T03:14:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:58.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:58.311888+0000) 2022-09-08T03:14:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:14:58 smithi055 ceph-mon[97033]: pgmap v10875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:14:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:14:58 smithi059 ceph-mon[97450]: pgmap v10875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:14:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:14:59.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:59.344706+0000) 2022-09-08T03:14:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:14:59.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:14:59.312121+0000) 2022-09-08T03:15:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:00.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:00.344957+0000) 2022-09-08T03:15:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:00.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:00.312349+0000) 2022-09-08T03:15:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:00 smithi055 ceph-mon[97033]: pgmap v10876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:00 smithi059 ceph-mon[97450]: pgmap v10876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:01.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:01.345159+0000) 2022-09-08T03:15:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:01.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:01.312471+0000) 2022-09-08T03:15:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:02.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:02.345403+0000) 2022-09-08T03:15:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:02.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:02.312716+0000) 2022-09-08T03:15:02.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:02 smithi055 ceph-mon[97033]: pgmap v10877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:15:03.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:02 smithi059 ceph-mon[97450]: pgmap v10877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:15:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:03.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:03.345674+0000) 2022-09-08T03:15:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:03.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:03.312946+0000) 2022-09-08T03:15:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:04.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:04.313195+0000) 2022-09-08T03:15:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:04.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:04.345918+0000) 2022-09-08T03:15:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:04 smithi059 ceph-mon[97450]: pgmap v10878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:04 smithi055 ceph-mon[97033]: pgmap v10878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:05.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:05.346082+0000) 2022-09-08T03:15:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:05.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:05.313348+0000) 2022-09-08T03:15:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:15:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:15:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:06.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:06.346289+0000) 2022-09-08T03:15:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:06.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:06.313586+0000) 2022-09-08T03:15:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:06 smithi059 ceph-mon[97450]: pgmap v10879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:15:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:15:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:06 smithi055 ceph-mon[97033]: pgmap v10879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:15:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:15:07.578 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:07.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:07.313790+0000) 2022-09-08T03:15:07.579 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:07.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:07.346497+0000) 2022-09-08T03:15:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:08.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:08.346739+0000) 2022-09-08T03:15:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:08.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:08.313924+0000) 2022-09-08T03:15:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:08 smithi055 ceph-mon[97033]: pgmap v10880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:15:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:08 smithi059 ceph-mon[97450]: pgmap v10880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:15:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:09.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:09.346907+0000) 2022-09-08T03:15:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:09.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:09.314071+0000) 2022-09-08T03:15:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:10.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:10.347087+0000) 2022-09-08T03:15:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:10.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:10.314282+0000) 2022-09-08T03:15:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:10 smithi055 ceph-mon[97033]: pgmap v10881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:15:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:10 smithi055 ceph-mon[97033]: osdmap e759: 12 total, 11 up, 11 in 2022-09-08T03:15:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:10 smithi059 ceph-mon[97450]: pgmap v10881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:15:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:10 smithi059 ceph-mon[97450]: osdmap e759: 12 total, 11 up, 11 in 2022-09-08T03:15:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:11.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:11.347231+0000) 2022-09-08T03:15:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:11.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:11.314438+0000) 2022-09-08T03:15:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:12.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:12.347444+0000) 2022-09-08T03:15:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:12.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:12.314619+0000) 2022-09-08T03:15:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:12 smithi055 ceph-mon[97033]: pgmap v10883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:15:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:12 smithi059 ceph-mon[97450]: pgmap v10883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:15:13.161 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:15:12.723+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:15:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:13.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:13.347711+0000) 2022-09-08T03:15:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:13.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:13.314744+0000) 2022-09-08T03:15:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:14.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:14.347876+0000) 2022-09-08T03:15:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:14.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:14.314895+0000) 2022-09-08T03:15:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:14 smithi059 ceph-mon[97450]: pgmap v10884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:15:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:14 smithi055 ceph-mon[97033]: pgmap v10884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:15:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:15.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:15.348140+0000) 2022-09-08T03:15:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:15.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:15.315136+0000) 2022-09-08T03:15:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:16.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:16.348316+0000) 2022-09-08T03:15:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:16.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:16.315364+0000) 2022-09-08T03:15:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:16 smithi055 ceph-mon[97033]: pgmap v10885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:15:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:16 smithi059 ceph-mon[97450]: pgmap v10885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:15:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:17.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:17.348522+0000) 2022-09-08T03:15:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:17.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:17.315595+0000) 2022-09-08T03:15:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:18.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:18.348728+0000) 2022-09-08T03:15:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:18.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:18.315776+0000) 2022-09-08T03:15:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:18 smithi055 ceph-mon[97033]: pgmap v10886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:15:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:18 smithi055 ceph-mon[97033]: osdmap e760: 12 total, 11 up, 11 in 2022-09-08T03:15:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:18 smithi059 ceph-mon[97450]: pgmap v10886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:15:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:18 smithi059 ceph-mon[97450]: osdmap e760: 12 total, 11 up, 11 in 2022-09-08T03:15:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:19.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:19.348900+0000) 2022-09-08T03:15:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:19.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:19.315979+0000) 2022-09-08T03:15:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:20.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:20.349197+0000) 2022-09-08T03:15:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:20.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:20.316147+0000) 2022-09-08T03:15:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:20 smithi055 ceph-mon[97033]: pgmap v10888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T03:15:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:20 smithi059 ceph-mon[97450]: pgmap v10888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T03:15:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:21.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:21.349418+0000) 2022-09-08T03:15:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:21.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:21.316330+0000) 2022-09-08T03:15:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:22.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:22.349699+0000) 2022-09-08T03:15:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:22.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:22.316538+0000) 2022-09-08T03:15:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:22 smithi055 ceph-mon[97033]: pgmap v10889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:15:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:22 smithi059 ceph-mon[97450]: pgmap v10889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:15:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:23.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:23.349956+0000) 2022-09-08T03:15:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:23.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:23.316732+0000) 2022-09-08T03:15:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:24.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:24.350183+0000) 2022-09-08T03:15:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:24.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:24.316971+0000) 2022-09-08T03:15:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:24 smithi059 ceph-mon[97450]: pgmap v10890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:15:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:24 smithi055 ceph-mon[97033]: pgmap v10890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:15:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:25.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:25.350381+0000) 2022-09-08T03:15:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:25.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:25.317115+0000) 2022-09-08T03:15:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:26.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:26.350591+0000) 2022-09-08T03:15:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:26.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:26.317244+0000) 2022-09-08T03:15:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:26 smithi055 ceph-mon[97033]: pgmap v10891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:15:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:26 smithi059 ceph-mon[97450]: pgmap v10891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:15:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:27.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:27.350834+0000) 2022-09-08T03:15:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:27.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:27.317456+0000) 2022-09-08T03:15:28.070 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:15:27.732+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:15:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:28.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:28.351078+0000) 2022-09-08T03:15:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:28.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:28.317715+0000) 2022-09-08T03:15:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:28 smithi055 ceph-mon[97033]: pgmap v10892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:15:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:28 smithi059 ceph-mon[97450]: pgmap v10892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:15:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:29.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:29.351278+0000) 2022-09-08T03:15:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:29.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:29.317921+0000) 2022-09-08T03:15:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:30.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:30.351528+0000) 2022-09-08T03:15:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:30.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:30.318104+0000) 2022-09-08T03:15:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:30 smithi055 ceph-mon[97033]: pgmap v10893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:15:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:30 smithi059 ceph-mon[97450]: pgmap v10893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:15:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:31.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:31.351766+0000) 2022-09-08T03:15:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:31.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:31.318325+0000) 2022-09-08T03:15:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:32.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:32.351951+0000) 2022-09-08T03:15:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:32.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:32.318539+0000) 2022-09-08T03:15:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:32 smithi055 ceph-mon[97033]: pgmap v10894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:15:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:32 smithi059 ceph-mon[97450]: pgmap v10894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:15:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:33.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:33.352170+0000) 2022-09-08T03:15:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:33.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:33.318793+0000) 2022-09-08T03:15:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:34.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:34.352414+0000) 2022-09-08T03:15:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:34.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:34.318990+0000) 2022-09-08T03:15:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:34 smithi059 ceph-mon[97450]: pgmap v10895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:34 smithi055 ceph-mon[97033]: pgmap v10895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:35.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:35.352689+0000) 2022-09-08T03:15:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:35.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:35.319149+0000) 2022-09-08T03:15:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:36.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:36.352925+0000) 2022-09-08T03:15:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:36.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:36.319397+0000) 2022-09-08T03:15:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:36 smithi055 ceph-mon[97033]: pgmap v10896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:15:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:36 smithi059 ceph-mon[97450]: pgmap v10896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:15:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:37.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:37.353178+0000) 2022-09-08T03:15:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:37.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:37.319653+0000) 2022-09-08T03:15:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:38.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:38.353377+0000) 2022-09-08T03:15:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:38.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:38.319884+0000) 2022-09-08T03:15:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:38 smithi055 ceph-mon[97033]: pgmap v10897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:38 smithi059 ceph-mon[97450]: pgmap v10897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:39.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:39.353537+0000) 2022-09-08T03:15:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:39.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:39.320123+0000) 2022-09-08T03:15:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:40.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:40.353751+0000) 2022-09-08T03:15:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:40.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:40.320364+0000) 2022-09-08T03:15:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:40 smithi055 ceph-mon[97033]: pgmap v10898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:40 smithi059 ceph-mon[97450]: pgmap v10898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:41.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:41.353969+0000) 2022-09-08T03:15:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:41.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:41.320568+0000) 2022-09-08T03:15:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:42.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:42.354179+0000) 2022-09-08T03:15:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:42.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:42.320803+0000) 2022-09-08T03:15:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:42 smithi055 ceph-mon[97033]: pgmap v10899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:15:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:42 smithi059 ceph-mon[97450]: pgmap v10899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:15:43.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:15:42.739+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:15:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:43.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:43.354413+0000) 2022-09-08T03:15:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:43.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:43.321025+0000) 2022-09-08T03:15:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:44.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:44.354567+0000) 2022-09-08T03:15:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:44.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:44.321238+0000) 2022-09-08T03:15:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:44 smithi059 ceph-mon[97450]: pgmap v10900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:44.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:44 smithi055 ceph-mon[97033]: pgmap v10900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:45.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:45.354710+0000) 2022-09-08T03:15:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:45.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:45.321467+0000) 2022-09-08T03:15:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:46.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:46.354893+0000) 2022-09-08T03:15:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:46.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:46.321736+0000) 2022-09-08T03:15:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:46 smithi055 ceph-mon[97033]: pgmap v10901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:15:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:46 smithi059 ceph-mon[97450]: pgmap v10901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:15:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:47.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:47.355089+0000) 2022-09-08T03:15:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:47.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:47.321956+0000) 2022-09-08T03:15:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:48.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:48.355290+0000) 2022-09-08T03:15:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:48.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:48.322196+0000) 2022-09-08T03:15:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:48 smithi055 ceph-mon[97033]: pgmap v10902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:48.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:15:48.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:15:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:15:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:48 smithi059 ceph-mon[97450]: pgmap v10902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:15:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:15:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:15:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:49.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:49.355454+0000) 2022-09-08T03:15:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:49.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:49.322474+0000) 2022-09-08T03:15:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:50.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:50.355595+0000) 2022-09-08T03:15:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:50.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:50.322698+0000) 2022-09-08T03:15:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:50 smithi055 ceph-mon[97033]: pgmap v10903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:50 smithi059 ceph-mon[97450]: pgmap v10903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:51.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:51.355812+0000) 2022-09-08T03:15:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:51.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:51.322939+0000) 2022-09-08T03:15:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:52.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:52.356068+0000) 2022-09-08T03:15:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:52.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:52.323212+0000) 2022-09-08T03:15:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:52 smithi055 ceph-mon[97033]: pgmap v10904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:15:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:52 smithi059 ceph-mon[97450]: pgmap v10904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:15:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:53.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:53.356254+0000) 2022-09-08T03:15:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:53.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:53.323469+0000) 2022-09-08T03:15:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:54.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:54.356478+0000) 2022-09-08T03:15:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:54.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:54.323763+0000) 2022-09-08T03:15:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:54 smithi059 ceph-mon[97450]: pgmap v10905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:54.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:54 smithi055 ceph-mon[97033]: pgmap v10905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:55.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:55.356725+0000) 2022-09-08T03:15:55.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:55.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:55.324016+0000) 2022-09-08T03:15:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:56.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:56.356946+0000) 2022-09-08T03:15:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:56.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:56.324277+0000) 2022-09-08T03:15:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:56 smithi055 ceph-mon[97033]: pgmap v10906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:15:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:56 smithi059 ceph-mon[97450]: pgmap v10906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:15:57.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:57.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:57.357128+0000) 2022-09-08T03:15:57.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:57.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:57.324526+0000) 2022-09-08T03:15:58.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:15:57.754+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:15:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:58.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:58.357358+0000) 2022-09-08T03:15:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:58.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:58.324791+0000) 2022-09-08T03:15:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:15:58 smithi055 ceph-mon[97033]: pgmap v10907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:15:58 smithi059 ceph-mon[97450]: pgmap v10907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:15:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:15:59.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:59.357547+0000) 2022-09-08T03:15:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:15:59.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:15:59.325020+0000) 2022-09-08T03:16:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:00.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:00.357730+0000) 2022-09-08T03:16:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:00.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:00.325274+0000) 2022-09-08T03:16:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:00 smithi055 ceph-mon[97033]: pgmap v10908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:00 smithi059 ceph-mon[97450]: pgmap v10908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:01.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:01.357887+0000) 2022-09-08T03:16:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:01.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:01.325464+0000) 2022-09-08T03:16:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:02.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:02.358132+0000) 2022-09-08T03:16:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:02.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:02.325697+0000) 2022-09-08T03:16:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:02 smithi055 ceph-mon[97033]: pgmap v10909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:16:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:02 smithi059 ceph-mon[97450]: pgmap v10909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:16:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:03.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:03.358357+0000) 2022-09-08T03:16:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:03.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:03.325962+0000) 2022-09-08T03:16:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:04.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:04.358560+0000) 2022-09-08T03:16:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:04.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:04.326213+0000) 2022-09-08T03:16:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:04 smithi059 ceph-mon[97450]: pgmap v10910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:04 smithi055 ceph-mon[97033]: pgmap v10910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:05.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:05.358752+0000) 2022-09-08T03:16:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:05.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:05.326383+0000) 2022-09-08T03:16:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:16:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:16:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:06.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:06.358908+0000) 2022-09-08T03:16:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:06.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:06.326539+0000) 2022-09-08T03:16:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:06 smithi059 ceph-mon[97450]: pgmap v10911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:16:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:16:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:06 smithi055 ceph-mon[97033]: pgmap v10911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:16:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:16:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:07.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:07.359121+0000) 2022-09-08T03:16:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:07.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:07.326761+0000) 2022-09-08T03:16:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:08.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:08.359266+0000) 2022-09-08T03:16:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:08.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:08.327036+0000) 2022-09-08T03:16:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:08 smithi055 ceph-mon[97033]: pgmap v10912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:16:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:08 smithi059 ceph-mon[97450]: pgmap v10912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:16:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:09.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:09.359412+0000) 2022-09-08T03:16:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:09.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:09.327268+0000) 2022-09-08T03:16:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:10.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:10.359583+0000) 2022-09-08T03:16:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:10.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:10.327420+0000) 2022-09-08T03:16:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:10 smithi055 ceph-mon[97033]: pgmap v10913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:16:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:10 smithi055 ceph-mon[97033]: osdmap e761: 12 total, 11 up, 11 in 2022-09-08T03:16:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:10 smithi059 ceph-mon[97450]: pgmap v10913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:16:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:10 smithi059 ceph-mon[97450]: osdmap e761: 12 total, 11 up, 11 in 2022-09-08T03:16:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:11.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:11.359830+0000) 2022-09-08T03:16:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:11.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:11.327666+0000) 2022-09-08T03:16:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:12.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:12.360024+0000) 2022-09-08T03:16:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:12.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:12.327898+0000) 2022-09-08T03:16:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:12 smithi055 ceph-mon[97033]: pgmap v10915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:16:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:12 smithi059 ceph-mon[97450]: pgmap v10915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:16:13.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:16:12.764+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:16:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:13.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:13.360263+0000) 2022-09-08T03:16:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:13.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:13.328132+0000) 2022-09-08T03:16:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:14.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:14.360443+0000) 2022-09-08T03:16:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:14.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:14.328363+0000) 2022-09-08T03:16:14.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:14 smithi059 ceph-mon[97450]: pgmap v10916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:16:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:14 smithi055 ceph-mon[97033]: pgmap v10916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:16:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:15.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:15.360708+0000) 2022-09-08T03:16:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:15.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:15.328663+0000) 2022-09-08T03:16:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:16.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:16.360951+0000) 2022-09-08T03:16:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:16.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:16.328925+0000) 2022-09-08T03:16:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:16 smithi055 ceph-mon[97033]: pgmap v10917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:16:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:16 smithi059 ceph-mon[97450]: pgmap v10917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:16:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:17.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:17.361168+0000) 2022-09-08T03:16:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:17.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:17.329156+0000) 2022-09-08T03:16:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:18.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:18.361432+0000) 2022-09-08T03:16:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:18.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:18.329389+0000) 2022-09-08T03:16:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:18 smithi055 ceph-mon[97033]: pgmap v10918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:16:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:18 smithi055 ceph-mon[97033]: pgmap v10919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T03:16:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:16:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:18 smithi055 ceph-mon[97033]: osdmap e762: 12 total, 11 up, 11 in 2022-09-08T03:16:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:18 smithi059 ceph-mon[97450]: pgmap v10918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:16:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:18 smithi059 ceph-mon[97450]: pgmap v10919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T03:16:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:16:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:18 smithi059 ceph-mon[97450]: osdmap e762: 12 total, 11 up, 11 in 2022-09-08T03:16:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:19.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:19.361707+0000) 2022-09-08T03:16:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:19.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:19.329713+0000) 2022-09-08T03:16:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:20.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:20.361887+0000) 2022-09-08T03:16:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:20.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:20.329942+0000) 2022-09-08T03:16:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:20 smithi055 ceph-mon[97033]: pgmap v10921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T03:16:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:20 smithi059 ceph-mon[97450]: pgmap v10921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T03:16:21.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:21.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:21.330123+0000) 2022-09-08T03:16:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:21.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:21.362100+0000) 2022-09-08T03:16:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:22.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:22.362353+0000) 2022-09-08T03:16:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:22.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:22.330345+0000) 2022-09-08T03:16:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:22 smithi055 ceph-mon[97033]: pgmap v10922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T03:16:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:22 smithi059 ceph-mon[97450]: pgmap v10922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T03:16:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:23.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:23.362502+0000) 2022-09-08T03:16:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:23.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:23.330586+0000) 2022-09-08T03:16:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:24.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:24.362696+0000) 2022-09-08T03:16:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:24.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:24.330866+0000) 2022-09-08T03:16:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:24 smithi059 ceph-mon[97450]: pgmap v10923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:16:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:24 smithi055 ceph-mon[97033]: pgmap v10923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:16:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:25.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:25.362818+0000) 2022-09-08T03:16:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:25.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:25.331065+0000) 2022-09-08T03:16:26.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:26.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:26.331283+0000) 2022-09-08T03:16:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:26.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:26.363049+0000) 2022-09-08T03:16:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:26 smithi055 ceph-mon[97033]: pgmap v10924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:16:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:26 smithi059 ceph-mon[97450]: pgmap v10924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:16:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:27.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:27.363288+0000) 2022-09-08T03:16:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:27.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:27.331537+0000) 2022-09-08T03:16:28.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:16:27.769+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:16:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:28.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:28.363551+0000) 2022-09-08T03:16:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:28.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:28.331718+0000) 2022-09-08T03:16:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:28 smithi055 ceph-mon[97033]: pgmap v10925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:16:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:28 smithi059 ceph-mon[97450]: pgmap v10925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:16:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:29.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:29.331897+0000) 2022-09-08T03:16:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:29.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:29.363767+0000) 2022-09-08T03:16:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:30.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:30.363904+0000) 2022-09-08T03:16:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:30.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:30.332134+0000) 2022-09-08T03:16:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:30 smithi055 ceph-mon[97033]: pgmap v10926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T03:16:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:30 smithi059 ceph-mon[97450]: pgmap v10926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T03:16:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:31.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:31.364080+0000) 2022-09-08T03:16:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:31.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:31.332422+0000) 2022-09-08T03:16:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:32.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:32.364332+0000) 2022-09-08T03:16:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:32.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:32.332670+0000) 2022-09-08T03:16:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:32 smithi055 ceph-mon[97033]: pgmap v10927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:32 smithi059 ceph-mon[97450]: pgmap v10927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:33.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:33.332893+0000) 2022-09-08T03:16:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:33.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:33.364591+0000) 2022-09-08T03:16:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:34.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:34.333169+0000) 2022-09-08T03:16:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:34.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:34.364809+0000) 2022-09-08T03:16:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:34 smithi059 ceph-mon[97450]: pgmap v10928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:34 smithi055 ceph-mon[97033]: pgmap v10928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:35.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:35.365045+0000) 2022-09-08T03:16:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:35.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:35.333364+0000) 2022-09-08T03:16:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:36.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:36.365250+0000) 2022-09-08T03:16:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:36.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:36.333503+0000) 2022-09-08T03:16:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:36 smithi055 ceph-mon[97033]: pgmap v10929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:16:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:36 smithi059 ceph-mon[97450]: pgmap v10929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:16:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:37.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:37.365460+0000) 2022-09-08T03:16:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:37.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:37.333665+0000) 2022-09-08T03:16:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:38.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:38.365663+0000) 2022-09-08T03:16:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:38.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:38.333870+0000) 2022-09-08T03:16:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:38 smithi055 ceph-mon[97033]: pgmap v10930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:38 smithi059 ceph-mon[97450]: pgmap v10930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:39.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:39.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:39.334102+0000) 2022-09-08T03:16:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:39.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:39.365871+0000) 2022-09-08T03:16:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:40.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:40.334325+0000) 2022-09-08T03:16:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:40.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:40.366105+0000) 2022-09-08T03:16:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:40 smithi055 ceph-mon[97033]: pgmap v10931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:16:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:40 smithi059 ceph-mon[97450]: pgmap v10931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:16:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:41.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:41.366266+0000) 2022-09-08T03:16:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:41.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:41.334548+0000) 2022-09-08T03:16:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:42.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:42.366476+0000) 2022-09-08T03:16:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:42.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:42.334747+0000) 2022-09-08T03:16:43.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:16:42.776+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:16:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:42 smithi055 ceph-mon[97033]: pgmap v10932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:42 smithi059 ceph-mon[97450]: pgmap v10932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:43.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:43.366738+0000) 2022-09-08T03:16:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:43.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:43.334977+0000) 2022-09-08T03:16:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:44.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:44.366994+0000) 2022-09-08T03:16:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:44.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:44.335206+0000) 2022-09-08T03:16:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:44 smithi059 ceph-mon[97450]: pgmap v10933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:44 smithi055 ceph-mon[97033]: pgmap v10933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:45.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:45.367258+0000) 2022-09-08T03:16:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:45.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:45.335459+0000) 2022-09-08T03:16:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:46.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:46.367512+0000) 2022-09-08T03:16:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:46.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:46.335718+0000) 2022-09-08T03:16:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:46 smithi055 ceph-mon[97033]: pgmap v10934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:16:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:46 smithi059 ceph-mon[97450]: pgmap v10934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:16:47.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:47.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:47.335879+0000) 2022-09-08T03:16:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:47.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:47.367676+0000) 2022-09-08T03:16:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:48.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:48.367801+0000) 2022-09-08T03:16:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:48.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:48.336055+0000) 2022-09-08T03:16:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:48 smithi055 ceph-mon[97033]: pgmap v10935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:48 smithi059 ceph-mon[97450]: pgmap v10935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:49.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:49.368042+0000) 2022-09-08T03:16:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:49.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:49.336297+0000) 2022-09-08T03:16:50.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:50.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:50.336480+0000) 2022-09-08T03:16:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:50.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:50.368226+0000) 2022-09-08T03:16:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:50 smithi055 ceph-mon[97033]: pgmap v10936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:16:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:50 smithi059 ceph-mon[97450]: pgmap v10936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:16:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:51.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:51.368414+0000) 2022-09-08T03:16:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:51.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:51.336747+0000) 2022-09-08T03:16:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:52.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:52.368649+0000) 2022-09-08T03:16:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:52.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:52.336915+0000) 2022-09-08T03:16:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:52 smithi055 ceph-mon[97033]: pgmap v10937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:52 smithi059 ceph-mon[97450]: pgmap v10937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:53.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:53.368771+0000) 2022-09-08T03:16:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:53.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:53.337024+0000) 2022-09-08T03:16:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:54.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:54.368984+0000) 2022-09-08T03:16:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:54.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:54.337230+0000) 2022-09-08T03:16:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:54 smithi055 ceph-mon[97033]: pgmap v10938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:54 smithi059 ceph-mon[97450]: pgmap v10938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:55.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:55.369156+0000) 2022-09-08T03:16:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:55.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:55.337374+0000) 2022-09-08T03:16:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:56.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:56.369386+0000) 2022-09-08T03:16:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:56.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:56.337594+0000) 2022-09-08T03:16:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:56 smithi055 ceph-mon[97033]: pgmap v10939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:16:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:56 smithi059 ceph-mon[97450]: pgmap v10939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:16:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:57.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:57.369622+0000) 2022-09-08T03:16:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:57.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:57.337772+0000) 2022-09-08T03:16:58.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:16:57.783+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:16:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:58.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:58.369895+0000) 2022-09-08T03:16:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:58.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:58.337978+0000) 2022-09-08T03:16:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:16:58 smithi055 ceph-mon[97033]: pgmap v10940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:16:58 smithi059 ceph-mon[97450]: pgmap v10940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:16:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:16:59.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:59.338166+0000) 2022-09-08T03:16:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:16:59.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:16:59.370133+0000) 2022-09-08T03:17:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:00.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:00.370279+0000) 2022-09-08T03:17:00.689 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:00.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:00.338318+0000) 2022-09-08T03:17:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:00 smithi055 ceph-mon[97033]: pgmap v10941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:17:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:00 smithi059 ceph-mon[97450]: pgmap v10941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:17:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:01.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:01.370486+0000) 2022-09-08T03:17:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:01.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:01.338489+0000) 2022-09-08T03:17:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:02.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:02.370749+0000) 2022-09-08T03:17:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:02.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:02.338728+0000) 2022-09-08T03:17:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:02 smithi055 ceph-mon[97033]: pgmap v10942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:02 smithi059 ceph-mon[97450]: pgmap v10942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:03.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:03.370882+0000) 2022-09-08T03:17:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:03.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:03.338949+0000) 2022-09-08T03:17:04.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:04.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:04.371089+0000) 2022-09-08T03:17:04.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:04.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:04.339142+0000) 2022-09-08T03:17:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:04 smithi059 ceph-mon[97450]: pgmap v10943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:04 smithi055 ceph-mon[97033]: pgmap v10943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:05.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:05.371325+0000) 2022-09-08T03:17:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:05.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:05.339356+0000) 2022-09-08T03:17:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:17:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:17:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:06.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:06.371528+0000) 2022-09-08T03:17:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:06.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:06.339564+0000) 2022-09-08T03:17:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:17:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:06 smithi055 ceph-mon[97033]: pgmap v10944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:17:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:17:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:06 smithi059 ceph-mon[97450]: pgmap v10944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:17:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:07.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:07.371709+0000) 2022-09-08T03:17:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:07.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:07.339767+0000) 2022-09-08T03:17:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:08.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:08.371879+0000) 2022-09-08T03:17:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:08.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:08.339973+0000) 2022-09-08T03:17:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:08 smithi055 ceph-mon[97033]: pgmap v10945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:17:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:08 smithi059 ceph-mon[97450]: pgmap v10945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:17:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:09.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:09.372069+0000) 2022-09-08T03:17:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:09.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:09.340165+0000) 2022-09-08T03:17:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:10.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:10.372250+0000) 2022-09-08T03:17:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:10.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:10.340308+0000) 2022-09-08T03:17:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:10 smithi055 ceph-mon[97033]: osdmap e763: 12 total, 11 up, 11 in 2022-09-08T03:17:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:10 smithi055 ceph-mon[97033]: pgmap v10947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:17:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:10 smithi059 ceph-mon[97450]: osdmap e763: 12 total, 11 up, 11 in 2022-09-08T03:17:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:10 smithi059 ceph-mon[97450]: pgmap v10947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:17:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:11.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:11.372419+0000) 2022-09-08T03:17:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:11.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:11.340525+0000) 2022-09-08T03:17:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:12.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:12.340767+0000) 2022-09-08T03:17:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:12.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:12.372599+0000) 2022-09-08T03:17:13.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:17:12.797+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:17:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:12 smithi055 ceph-mon[97033]: pgmap v10948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:17:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:12 smithi059 ceph-mon[97450]: pgmap v10948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:17:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:13.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:13.372835+0000) 2022-09-08T03:17:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:13.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:13.341001+0000) 2022-09-08T03:17:14.663 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:14.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:14.373013+0000) 2022-09-08T03:17:14.664 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:14.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:14.341214+0000) 2022-09-08T03:17:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:14 smithi055 ceph-mon[97033]: pgmap v10949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:17:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:14 smithi059 ceph-mon[97450]: pgmap v10949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:17:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:15.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:15.373302+0000) 2022-09-08T03:17:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:15.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:15.341388+0000) 2022-09-08T03:17:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:16.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:16.373520+0000) 2022-09-08T03:17:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:16.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:16.341577+0000) 2022-09-08T03:17:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:16 smithi055 ceph-mon[97033]: pgmap v10950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:17:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:16 smithi059 ceph-mon[97450]: pgmap v10950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:17:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:17.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:17.373732+0000) 2022-09-08T03:17:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:17.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:17.341789+0000) 2022-09-08T03:17:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:18.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:18.373904+0000) 2022-09-08T03:17:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:18.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:18.342049+0000) 2022-09-08T03:17:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:18 smithi055 ceph-mon[97033]: pgmap v10951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:17:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:17:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:17:19.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:17:19.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:18 smithi055 ceph-mon[97033]: osdmap e764: 12 total, 11 up, 11 in 2022-09-08T03:17:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:18 smithi059 ceph-mon[97450]: pgmap v10951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:17:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:17:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:17:19.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:17:19.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:18 smithi059 ceph-mon[97450]: osdmap e764: 12 total, 11 up, 11 in 2022-09-08T03:17:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:19.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:19.374143+0000) 2022-09-08T03:17:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:19.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:19.342248+0000) 2022-09-08T03:17:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:20.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:20.374393+0000) 2022-09-08T03:17:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:20.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:20.342417+0000) 2022-09-08T03:17:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:20 smithi055 ceph-mon[97033]: pgmap v10953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:17:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:20 smithi059 ceph-mon[97450]: pgmap v10953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:17:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:21.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:21.374615+0000) 2022-09-08T03:17:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:21.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:21.342674+0000) 2022-09-08T03:17:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:22.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:22.374779+0000) 2022-09-08T03:17:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:22.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:22.342918+0000) 2022-09-08T03:17:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:22 smithi055 ceph-mon[97033]: pgmap v10954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:17:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:22 smithi059 ceph-mon[97450]: pgmap v10954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:17:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:23.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:23.374959+0000) 2022-09-08T03:17:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:23.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:23.343136+0000) 2022-09-08T03:17:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:24.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:24.375174+0000) 2022-09-08T03:17:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:24.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:24.343346+0000) 2022-09-08T03:17:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:24 smithi055 ceph-mon[97033]: pgmap v10955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:17:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:24 smithi059 ceph-mon[97450]: pgmap v10955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:17:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:25.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:25.375303+0000) 2022-09-08T03:17:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:25.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:25.343573+0000) 2022-09-08T03:17:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:26.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:26.375555+0000) 2022-09-08T03:17:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:26.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:26.343801+0000) 2022-09-08T03:17:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:26 smithi055 ceph-mon[97033]: pgmap v10956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:17:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:26 smithi059 ceph-mon[97450]: pgmap v10956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:17:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:27.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:27.375796+0000) 2022-09-08T03:17:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:27.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:27.344005+0000) 2022-09-08T03:17:28.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:17:27.809+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:17:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:28.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:28.375985+0000) 2022-09-08T03:17:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:28.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:28.344241+0000) 2022-09-08T03:17:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:28 smithi055 ceph-mon[97033]: pgmap v10957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:17:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:28 smithi059 ceph-mon[97450]: pgmap v10957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:17:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:29.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:29.376106+0000) 2022-09-08T03:17:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:29.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:29.344492+0000) 2022-09-08T03:17:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:30.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:30.376353+0000) 2022-09-08T03:17:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:30.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:30.344738+0000) 2022-09-08T03:17:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:30 smithi055 ceph-mon[97033]: pgmap v10958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T03:17:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:30 smithi059 ceph-mon[97450]: pgmap v10958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T03:17:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:31.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:31.345006+0000) 2022-09-08T03:17:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:31.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:31.376599+0000) 2022-09-08T03:17:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:32.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:32.376866+0000) 2022-09-08T03:17:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:32.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:32.345262+0000) 2022-09-08T03:17:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:32 smithi055 ceph-mon[97033]: pgmap v10959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:32 smithi059 ceph-mon[97450]: pgmap v10959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:33.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:33.377110+0000) 2022-09-08T03:17:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:33.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:33.345420+0000) 2022-09-08T03:17:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:34.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:34.345684+0000) 2022-09-08T03:17:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:34.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:34.377353+0000) 2022-09-08T03:17:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:34 smithi055 ceph-mon[97033]: pgmap v10960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:34 smithi059 ceph-mon[97450]: pgmap v10960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:35.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:35.377590+0000) 2022-09-08T03:17:35.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:35.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:35.345919+0000) 2022-09-08T03:17:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:36.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:36.377799+0000) 2022-09-08T03:17:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:36.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:36.346156+0000) 2022-09-08T03:17:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:36 smithi055 ceph-mon[97033]: pgmap v10961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:17:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:36 smithi059 ceph-mon[97450]: pgmap v10961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:17:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:37.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:37.378024+0000) 2022-09-08T03:17:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:37.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:37.346356+0000) 2022-09-08T03:17:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:38.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:38.378247+0000) 2022-09-08T03:17:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:38.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:38.346586+0000) 2022-09-08T03:17:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:38 smithi055 ceph-mon[97033]: pgmap v10962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:38 smithi059 ceph-mon[97450]: pgmap v10962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:39.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:39.378468+0000) 2022-09-08T03:17:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:39.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:39.346883+0000) 2022-09-08T03:17:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:40.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:40.378674+0000) 2022-09-08T03:17:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:40.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:40.347126+0000) 2022-09-08T03:17:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:40 smithi055 ceph-mon[97033]: pgmap v10963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:17:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:40 smithi059 ceph-mon[97450]: pgmap v10963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:17:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:41.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:41.378918+0000) 2022-09-08T03:17:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:41.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:41.347363+0000) 2022-09-08T03:17:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:42.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:42.379095+0000) 2022-09-08T03:17:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:42.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:42.347553+0000) 2022-09-08T03:17:43.099 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:17:42.817+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:17:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:42 smithi055 ceph-mon[97033]: pgmap v10964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:42 smithi059 ceph-mon[97450]: pgmap v10964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:43.379 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:43.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:43.347830+0000) 2022-09-08T03:17:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:43.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:43.379341+0000) 2022-09-08T03:17:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:44.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:44.379493+0000) 2022-09-08T03:17:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:44.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:44.347989+0000) 2022-09-08T03:17:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:44 smithi055 ceph-mon[97033]: pgmap v10965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:44 smithi059 ceph-mon[97450]: pgmap v10965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:45.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:45.348215+0000) 2022-09-08T03:17:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:45.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:45.379731+0000) 2022-09-08T03:17:46.661 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:46.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:46.379971+0000) 2022-09-08T03:17:46.662 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:46.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:46.348423+0000) 2022-09-08T03:17:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:46 smithi055 ceph-mon[97033]: pgmap v10966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:17:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:46 smithi059 ceph-mon[97450]: pgmap v10966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:17:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:47.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:47.380217+0000) 2022-09-08T03:17:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:47.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:47.348678+0000) 2022-09-08T03:17:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:48.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:48.380460+0000) 2022-09-08T03:17:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:48.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:48.348965+0000) 2022-09-08T03:17:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:48 smithi055 ceph-mon[97033]: pgmap v10967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:48 smithi059 ceph-mon[97450]: pgmap v10967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:49.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:49.380678+0000) 2022-09-08T03:17:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:49.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:49.349184+0000) 2022-09-08T03:17:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:50.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:50.380890+0000) 2022-09-08T03:17:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:50.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:50.349404+0000) 2022-09-08T03:17:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:50 smithi055 ceph-mon[97033]: pgmap v10968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:17:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:50 smithi059 ceph-mon[97450]: pgmap v10968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:17:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:51.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:51.349738+0000) 2022-09-08T03:17:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:51.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:51.381064+0000) 2022-09-08T03:17:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:52.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:52.381256+0000) 2022-09-08T03:17:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:52.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:52.349992+0000) 2022-09-08T03:17:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:52 smithi055 ceph-mon[97033]: pgmap v10969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:17:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:17:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:17:53.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:52 smithi059 ceph-mon[97450]: pgmap v10969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:17:53.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:17:53.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:17:53.289 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:17:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:53.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:53.381447+0000) 2022-09-08T03:17:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:53.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:53.350205+0000) 2022-09-08T03:17:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:53 smithi055 ceph-mon[97033]: pgmap v10970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:17:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:53 smithi055 ceph-mon[97033]: pgmap v10971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:17:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:53 smithi059 ceph-mon[97450]: pgmap v10970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:17:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:53 smithi059 ceph-mon[97450]: pgmap v10971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:17:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:54.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:54.381671+0000) 2022-09-08T03:17:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:54.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:54.350402+0000) 2022-09-08T03:17:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:55.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:55.381805+0000) 2022-09-08T03:17:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:55.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:55.350564+0000) 2022-09-08T03:17:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:55 smithi055 ceph-mon[97033]: pgmap v10972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T03:17:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:55 smithi059 ceph-mon[97450]: pgmap v10972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T03:17:56.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:56.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:56.350794+0000) 2022-09-08T03:17:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:56.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:56.381927+0000) 2022-09-08T03:17:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:57.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:57.382145+0000) 2022-09-08T03:17:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:57.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:57.350975+0000) 2022-09-08T03:17:58.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:17:57.831+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:17:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:57 smithi055 ceph-mon[97033]: pgmap v10973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:17:58.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:57 smithi059 ceph-mon[97450]: pgmap v10973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:17:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:58.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:58.382372+0000) 2022-09-08T03:17:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:58.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:58.351213+0000) 2022-09-08T03:17:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:17:59.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:59.382618+0000) 2022-09-08T03:17:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:17:59.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:17:59.351401+0000) 2022-09-08T03:18:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:17:59 smithi055 ceph-mon[97033]: pgmap v10974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T03:18:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:17:59 smithi059 ceph-mon[97450]: pgmap v10974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T03:18:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:00.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:00.351555+0000) 2022-09-08T03:18:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:00.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:00.382759+0000) 2022-09-08T03:18:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:01.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:01.382901+0000) 2022-09-08T03:18:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:01.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:01.351753+0000) 2022-09-08T03:18:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:01 smithi055 ceph-mon[97033]: pgmap v10975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:18:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:01 smithi059 ceph-mon[97450]: pgmap v10975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:18:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:02.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:02.383056+0000) 2022-09-08T03:18:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:02.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:02.351969+0000) 2022-09-08T03:18:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:03.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:03.383207+0000) 2022-09-08T03:18:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:03.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:03.352201+0000) 2022-09-08T03:18:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:03 smithi055 ceph-mon[97033]: pgmap v10976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:18:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:03 smithi059 ceph-mon[97450]: pgmap v10976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:18:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:04.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:04.383463+0000) 2022-09-08T03:18:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:04.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:04.352432+0000) 2022-09-08T03:18:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:05.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:05.383725+0000) 2022-09-08T03:18:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:05.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:05.352674+0000) 2022-09-08T03:18:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:05 smithi055 ceph-mon[97033]: pgmap v10977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:18:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:18:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:05 smithi059 ceph-mon[97450]: pgmap v10977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:18:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:18:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:06.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:06.383967+0000) 2022-09-08T03:18:06.671 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:06.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:06.352864+0000) 2022-09-08T03:18:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:07.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:07.384192+0000) 2022-09-08T03:18:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:07.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:07.353029+0000) 2022-09-08T03:18:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:07 smithi055 ceph-mon[97033]: pgmap v10978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:18:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:07 smithi059 ceph-mon[97450]: pgmap v10978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:18:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:08.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:08.384432+0000) 2022-09-08T03:18:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:08.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:08.353294+0000) 2022-09-08T03:18:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:09.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:09.384682+0000) 2022-09-08T03:18:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:09.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:09.353519+0000) 2022-09-08T03:18:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:09 smithi055 ceph-mon[97033]: pgmap v10979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:18:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:09 smithi055 ceph-mon[97033]: osdmap e765: 12 total, 11 up, 11 in 2022-09-08T03:18:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:09 smithi059 ceph-mon[97450]: pgmap v10979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:18:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:09 smithi059 ceph-mon[97450]: osdmap e765: 12 total, 11 up, 11 in 2022-09-08T03:18:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:10.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:10.384942+0000) 2022-09-08T03:18:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:10.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:10.353613+0000) 2022-09-08T03:18:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:11.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:11.385158+0000) 2022-09-08T03:18:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:11.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:11.353706+0000) 2022-09-08T03:18:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:11 smithi055 ceph-mon[97033]: pgmap v10981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:18:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:11 smithi059 ceph-mon[97450]: pgmap v10981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:18:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:12.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:12.385359+0000) 2022-09-08T03:18:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:12.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:12.353942+0000) 2022-09-08T03:18:13.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:18:12.836+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:18:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:13.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:13.385504+0000) 2022-09-08T03:18:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:13.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:13.354163+0000) 2022-09-08T03:18:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:13 smithi055 ceph-mon[97033]: pgmap v10982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:18:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:13 smithi055 ceph-mon[97033]: osdmap e766: 12 total, 11 up, 11 in 2022-09-08T03:18:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:13 smithi059 ceph-mon[97450]: pgmap v10982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:18:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:13 smithi059 ceph-mon[97450]: osdmap e766: 12 total, 11 up, 11 in 2022-09-08T03:18:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:14.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:14.385746+0000) 2022-09-08T03:18:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:14.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:14.354340+0000) 2022-09-08T03:18:15.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:15.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:15.354504+0000) 2022-09-08T03:18:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:15.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:15.385954+0000) 2022-09-08T03:18:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:15 smithi055 ceph-mon[97033]: pgmap v10984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:18:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:15 smithi059 ceph-mon[97450]: pgmap v10984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:18:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:16.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:16.386189+0000) 2022-09-08T03:18:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:16.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:16.354761+0000) 2022-09-08T03:18:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:17.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:17.355008+0000) 2022-09-08T03:18:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:17.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:17.386416+0000) 2022-09-08T03:18:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:17 smithi055 ceph-mon[97033]: pgmap v10985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:18:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:17 smithi059 ceph-mon[97450]: pgmap v10985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:18:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:18.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:18.355227+0000) 2022-09-08T03:18:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:18.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:18.386682+0000) 2022-09-08T03:18:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:19.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:19.386925+0000) 2022-09-08T03:18:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:19.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:19.355468+0000) 2022-09-08T03:18:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:19 smithi055 ceph-mon[97033]: pgmap v10986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T03:18:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:19 smithi059 ceph-mon[97450]: pgmap v10986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T03:18:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:20.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:20.387139+0000) 2022-09-08T03:18:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:20.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:20.355735+0000) 2022-09-08T03:18:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:21.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:21.387307+0000) 2022-09-08T03:18:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:21.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:21.355964+0000) 2022-09-08T03:18:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:21 smithi055 ceph-mon[97033]: pgmap v10987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:18:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:21 smithi059 ceph-mon[97450]: pgmap v10987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:18:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:22.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:22.387560+0000) 2022-09-08T03:18:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:22.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:22.356194+0000) 2022-09-08T03:18:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:23.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:23.356441+0000) 2022-09-08T03:18:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:23.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:23.387817+0000) 2022-09-08T03:18:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:23 smithi055 ceph-mon[97033]: pgmap v10988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:18:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:23 smithi059 ceph-mon[97450]: pgmap v10988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:18:24.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:24.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:24.356746+0000) 2022-09-08T03:18:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:24.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:24.388108+0000) 2022-09-08T03:18:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:25.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:25.388309+0000) 2022-09-08T03:18:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:25.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:25.356949+0000) 2022-09-08T03:18:26.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:25 smithi055 ceph-mon[97033]: pgmap v10989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T03:18:26.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:25 smithi059 ceph-mon[97450]: pgmap v10989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T03:18:26.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:26.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:26.357165+0000) 2022-09-08T03:18:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:26.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:26.388514+0000) 2022-09-08T03:18:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:27.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:27.388776+0000) 2022-09-08T03:18:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:27.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:27.357369+0000) 2022-09-08T03:18:28.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:18:27.852+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:18:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:27 smithi055 ceph-mon[97033]: pgmap v10990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:18:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:27 smithi059 ceph-mon[97450]: pgmap v10990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:18:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:28.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:28.388957+0000) 2022-09-08T03:18:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:28.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:28.357609+0000) 2022-09-08T03:18:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:29.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:29.389169+0000) 2022-09-08T03:18:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:29.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:29.357783+0000) 2022-09-08T03:18:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:29 smithi055 ceph-mon[97033]: pgmap v10991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:29 smithi059 ceph-mon[97450]: pgmap v10991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:30.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:30.389359+0000) 2022-09-08T03:18:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:30.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:30.357988+0000) 2022-09-08T03:18:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:31.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:31.389562+0000) 2022-09-08T03:18:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:31.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:31.358130+0000) 2022-09-08T03:18:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:31 smithi055 ceph-mon[97033]: pgmap v10992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:18:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:31 smithi059 ceph-mon[97450]: pgmap v10992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:18:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:32.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:32.389810+0000) 2022-09-08T03:18:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:32.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:32.358333+0000) 2022-09-08T03:18:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:33.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:33.358565+0000) 2022-09-08T03:18:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:33.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:33.390009+0000) 2022-09-08T03:18:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:33 smithi055 ceph-mon[97033]: pgmap v10993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:33 smithi059 ceph-mon[97450]: pgmap v10993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:34.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:34.390241+0000) 2022-09-08T03:18:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:34.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:34.358732+0000) 2022-09-08T03:18:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:35.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:35.390478+0000) 2022-09-08T03:18:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:35.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:35.358953+0000) 2022-09-08T03:18:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:35 smithi055 ceph-mon[97033]: pgmap v10994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:35 smithi059 ceph-mon[97450]: pgmap v10994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:36.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:36.359206+0000) 2022-09-08T03:18:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:36.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:36.390730+0000) 2022-09-08T03:18:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:37.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:37.390918+0000) 2022-09-08T03:18:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:37.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:37.359454+0000) 2022-09-08T03:18:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:37 smithi055 ceph-mon[97033]: pgmap v10995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:18:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:37 smithi059 ceph-mon[97450]: pgmap v10995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:18:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:38.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:38.391125+0000) 2022-09-08T03:18:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:38.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:38.359665+0000) 2022-09-08T03:18:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:39.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:39.359875+0000) 2022-09-08T03:18:39.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:39.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:39.391378+0000) 2022-09-08T03:18:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:39 smithi055 ceph-mon[97033]: pgmap v10996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:39 smithi059 ceph-mon[97450]: pgmap v10996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:40.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:40.391602+0000) 2022-09-08T03:18:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:40.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:40.360093+0000) 2022-09-08T03:18:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:41.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:41.360277+0000) 2022-09-08T03:18:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:41.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:41.391869+0000) 2022-09-08T03:18:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:41 smithi055 ceph-mon[97033]: pgmap v10997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:18:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:41 smithi059 ceph-mon[97450]: pgmap v10997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:18:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:42.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:42.392095+0000) 2022-09-08T03:18:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:42.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:42.360499+0000) 2022-09-08T03:18:43.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:18:42.865+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:18:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:43.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:43.360751+0000) 2022-09-08T03:18:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:43.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:43.392291+0000) 2022-09-08T03:18:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:43 smithi055 ceph-mon[97033]: pgmap v10998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:43 smithi059 ceph-mon[97450]: pgmap v10998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:44.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:44.392430+0000) 2022-09-08T03:18:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:44.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:44.360920+0000) 2022-09-08T03:18:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:45.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:45.392690+0000) 2022-09-08T03:18:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:45.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:45.361112+0000) 2022-09-08T03:18:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:45 smithi059 ceph-mon[97450]: pgmap v10999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:45 smithi055 ceph-mon[97033]: pgmap v10999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:46.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:46.392873+0000) 2022-09-08T03:18:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:46.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:46.361326+0000) 2022-09-08T03:18:47.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:47.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:47.393113+0000) 2022-09-08T03:18:47.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:47.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:47.361602+0000) 2022-09-08T03:18:48.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:47 smithi059 ceph-mon[97450]: pgmap v11000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:18:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:47 smithi055 ceph-mon[97033]: pgmap v11000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:18:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:48.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:48.393341+0000) 2022-09-08T03:18:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:48.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:48.361821+0000) 2022-09-08T03:18:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:49.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:49.393557+0000) 2022-09-08T03:18:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:49.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:49.361970+0000) 2022-09-08T03:18:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:49 smithi059 ceph-mon[97450]: pgmap v11001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:49 smithi055 ceph-mon[97033]: pgmap v11001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:50.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:50.393800+0000) 2022-09-08T03:18:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:50.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:50.362204+0000) 2022-09-08T03:18:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:51.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:51.394038+0000) 2022-09-08T03:18:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:51.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:51.362424+0000) 2022-09-08T03:18:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:51 smithi059 ceph-mon[97450]: pgmap v11002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:18:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:51 smithi055 ceph-mon[97033]: pgmap v11002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:18:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:52.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:52.394338+0000) 2022-09-08T03:18:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:52.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:52.362617+0000) 2022-09-08T03:18:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:18:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:18:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:18:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:18:53.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:18:53.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:18:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:53.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:53.394502+0000) 2022-09-08T03:18:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:53.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:53.362820+0000) 2022-09-08T03:18:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:53 smithi059 ceph-mon[97450]: pgmap v11003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:53 smithi055 ceph-mon[97033]: pgmap v11003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:54.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:54.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:54.394772+0000) 2022-09-08T03:18:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:54.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:54.363017+0000) 2022-09-08T03:18:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:55.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:55.395056+0000) 2022-09-08T03:18:55.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:55.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:55.363234+0000) 2022-09-08T03:18:56.253 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:56 smithi055 ceph-mon[97033]: pgmap v11004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:55 smithi059 ceph-mon[97450]: pgmap v11004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:18:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:56.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:56.395318+0000) 2022-09-08T03:18:56.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:56.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:56.363449+0000) 2022-09-08T03:18:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:57.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:57.363596+0000) 2022-09-08T03:18:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:57.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:57.395567+0000) 2022-09-08T03:18:57.883 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:57 smithi059 ceph-mon[97450]: pgmap v11005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:18:57.883 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:57 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T03:18:57.884 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:57.885 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:57.928 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:57 smithi055 ceph-mon[97033]: pgmap v11005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:18:57.928 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:57 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T03:18:57.929 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:57.929 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:58.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:18:57.880+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:18:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:58.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:58.363799+0000) 2022-09-08T03:18:58.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:58.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:58.395783+0000) 2022-09-08T03:18:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:58 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T03:18:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:58 smithi055 ceph-mon[97033]: pgmap v11006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T03:18:59.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:58 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T03:18:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:58 smithi059 ceph-mon[97450]: pgmap v11006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T03:18:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:18:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:18:59.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:59.396014+0000) 2022-09-08T03:18:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:18:59.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:18:59.363963+0000) 2022-09-08T03:19:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:18:59 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T03:19:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:18:59 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T03:19:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:00.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:00.396252+0000) 2022-09-08T03:19:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:00.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:00.364193+0000) 2022-09-08T03:19:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:00 smithi055 ceph-mon[97033]: pgmap v11007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T03:19:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:00 smithi059 ceph-mon[97450]: pgmap v11007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T03:19:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:01.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:01.396495+0000) 2022-09-08T03:19:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:01.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:01.364415+0000) 2022-09-08T03:19:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:02.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:02.396747+0000) 2022-09-08T03:19:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:02.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:02.364605+0000) 2022-09-08T03:19:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:02 smithi055 ceph-mon[97033]: pgmap v11008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T03:19:03.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:02 smithi059 ceph-mon[97450]: pgmap v11008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T03:19:03.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:03.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:03.364752+0000) 2022-09-08T03:19:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:03.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:03.396947+0000) 2022-09-08T03:19:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:04.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:04.397174+0000) 2022-09-08T03:19:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:04.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:04.364953+0000) 2022-09-08T03:19:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:04 smithi055 ceph-mon[97033]: pgmap v11009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T03:19:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:04 smithi059 ceph-mon[97450]: pgmap v11009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T03:19:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:05.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:05.397363+0000) 2022-09-08T03:19:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:05.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:05.365180+0000) 2022-09-08T03:19:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:19:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:19:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:06.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:06.397552+0000) 2022-09-08T03:19:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:06.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:06.365349+0000) 2022-09-08T03:19:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:19:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:06 smithi055 ceph-mon[97033]: pgmap v11010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 47 KiB/s wr, 23 op/s 2022-09-08T03:19:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:19:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:06 smithi059 ceph-mon[97450]: pgmap v11010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 47 KiB/s wr, 23 op/s 2022-09-08T03:19:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:07.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:07.397751+0000) 2022-09-08T03:19:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:07.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:07.365592+0000) 2022-09-08T03:19:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:08.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:08.397899+0000) 2022-09-08T03:19:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:08.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:08.365871+0000) 2022-09-08T03:19:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:08 smithi055 ceph-mon[97033]: pgmap v11011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 47 KiB/s wr, 21 op/s 2022-09-08T03:19:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:08 smithi059 ceph-mon[97450]: pgmap v11011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 47 KiB/s wr, 21 op/s 2022-09-08T03:19:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:09.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:09.398124+0000) 2022-09-08T03:19:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:09.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:09.366126+0000) 2022-09-08T03:19:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:10.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:10.398257+0000) 2022-09-08T03:19:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:10.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:10.366247+0000) 2022-09-08T03:19:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:10 smithi055 ceph-mon[97033]: osdmap e767: 12 total, 11 up, 11 in 2022-09-08T03:19:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:10 smithi055 ceph-mon[97033]: pgmap v11013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:19:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:10 smithi059 ceph-mon[97450]: osdmap e767: 12 total, 11 up, 11 in 2022-09-08T03:19:11.073 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:10 smithi059 ceph-mon[97450]: pgmap v11013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:19:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:11.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:11.398497+0000) 2022-09-08T03:19:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:11.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:11.366392+0000) 2022-09-08T03:19:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:12.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:12.366597+0000) 2022-09-08T03:19:12.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:12.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:12.398766+0000) 2022-09-08T03:19:13.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:19:12.892+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:19:13.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:12 smithi055 ceph-mon[97033]: pgmap v11014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:19:13.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:12 smithi059 ceph-mon[97450]: pgmap v11014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:19:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:13.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:13.398963+0000) 2022-09-08T03:19:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:13.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:13.366850+0000) 2022-09-08T03:19:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:14.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:14.399100+0000) 2022-09-08T03:19:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:14.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:14.366981+0000) 2022-09-08T03:19:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:14 smithi055 ceph-mon[97033]: pgmap v11015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:19:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:14 smithi059 ceph-mon[97450]: pgmap v11015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:19:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:15.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:15.399294+0000) 2022-09-08T03:19:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:15.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:15.367226+0000) 2022-09-08T03:19:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:16.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:16.399449+0000) 2022-09-08T03:19:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:16.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:16.367420+0000) 2022-09-08T03:19:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:16 smithi055 ceph-mon[97033]: pgmap v11016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:19:17.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:16 smithi059 ceph-mon[97450]: pgmap v11016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:19:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:17.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:17.367696+0000) 2022-09-08T03:19:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:17.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:17.399684+0000) 2022-09-08T03:19:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:18.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:18.399899+0000) 2022-09-08T03:19:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:18.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:18.367926+0000) 2022-09-08T03:19:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:18 smithi055 ceph-mon[97033]: pgmap v11017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:19:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:18 smithi055 ceph-mon[97033]: osdmap e768: 12 total, 11 up, 11 in 2022-09-08T03:19:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:18 smithi059 ceph-mon[97450]: pgmap v11017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:19:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:18 smithi059 ceph-mon[97450]: osdmap e768: 12 total, 11 up, 11 in 2022-09-08T03:19:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:19.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:19.400189+0000) 2022-09-08T03:19:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:19.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:19.368165+0000) 2022-09-08T03:19:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:20.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:20.400409+0000) 2022-09-08T03:19:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:20.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:20.368404+0000) 2022-09-08T03:19:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:20 smithi055 ceph-mon[97033]: pgmap v11019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:19:21.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:20 smithi059 ceph-mon[97450]: pgmap v11019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:19:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:21.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:21.400647+0000) 2022-09-08T03:19:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:21.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:21.368618+0000) 2022-09-08T03:19:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:22.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:22.400888+0000) 2022-09-08T03:19:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:22.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:22.368832+0000) 2022-09-08T03:19:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:22 smithi055 ceph-mon[97033]: pgmap v11020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:19:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:22 smithi059 ceph-mon[97450]: pgmap v11020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:19:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:23.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:23.401100+0000) 2022-09-08T03:19:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:23.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:23.369059+0000) 2022-09-08T03:19:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:24.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:24.401284+0000) 2022-09-08T03:19:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:24.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:24.369335+0000) 2022-09-08T03:19:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:24 smithi055 ceph-mon[97033]: pgmap v11021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:19:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:24 smithi059 ceph-mon[97450]: pgmap v11021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:19:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:25.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:25.401516+0000) 2022-09-08T03:19:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:25.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:25.369532+0000) 2022-09-08T03:19:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:26.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:26.401697+0000) 2022-09-08T03:19:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:26.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:26.369749+0000) 2022-09-08T03:19:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:26 smithi055 ceph-mon[97033]: pgmap v11022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:19:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:26 smithi059 ceph-mon[97450]: pgmap v11022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:19:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:27.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:27.401846+0000) 2022-09-08T03:19:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:27.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:27.369950+0000) 2022-09-08T03:19:28.370 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:19:27.906+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:19:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:28.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:28.402009+0000) 2022-09-08T03:19:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:28.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:28.370108+0000) 2022-09-08T03:19:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:28 smithi055 ceph-mon[97033]: pgmap v11023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:19:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:28 smithi059 ceph-mon[97450]: pgmap v11023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:19:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:29.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:29.402175+0000) 2022-09-08T03:19:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:29.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:29.370324+0000) 2022-09-08T03:19:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:30.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:30.402421+0000) 2022-09-08T03:19:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:30.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:30.370487+0000) 2022-09-08T03:19:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:30 smithi055 ceph-mon[97033]: pgmap v11024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T03:19:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:30 smithi059 ceph-mon[97450]: pgmap v11024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T03:19:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:31.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:31.402588+0000) 2022-09-08T03:19:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:31.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:31.370737+0000) 2022-09-08T03:19:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:32.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:32.370963+0000) 2022-09-08T03:19:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:32.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:32.402804+0000) 2022-09-08T03:19:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:32 smithi055 ceph-mon[97033]: pgmap v11025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:32 smithi059 ceph-mon[97450]: pgmap v11025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:33.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:33.371106+0000) 2022-09-08T03:19:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:33.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:33.403037+0000) 2022-09-08T03:19:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:34.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:34.403219+0000) 2022-09-08T03:19:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:34.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:34.371301+0000) 2022-09-08T03:19:34.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:34 smithi055 ceph-mon[97033]: pgmap v11026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:34 smithi059 ceph-mon[97450]: pgmap v11026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:35.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:35.403439+0000) 2022-09-08T03:19:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:35.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:35.371528+0000) 2022-09-08T03:19:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:36.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:36.403719+0000) 2022-09-08T03:19:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:36.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:36.371715+0000) 2022-09-08T03:19:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:36 smithi055 ceph-mon[97033]: pgmap v11027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:19:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:36 smithi059 ceph-mon[97450]: pgmap v11027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:19:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:37.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:37.403977+0000) 2022-09-08T03:19:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:37.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:37.371880+0000) 2022-09-08T03:19:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:38.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:38.404172+0000) 2022-09-08T03:19:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:38.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:38.372028+0000) 2022-09-08T03:19:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:38 smithi055 ceph-mon[97033]: pgmap v11028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:39.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:38 smithi059 ceph-mon[97450]: pgmap v11028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:39.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:39.404394+0000) 2022-09-08T03:19:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:39.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:39.372170+0000) 2022-09-08T03:19:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:40.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:40.404656+0000) 2022-09-08T03:19:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:40.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:40.372351+0000) 2022-09-08T03:19:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:40 smithi055 ceph-mon[97033]: pgmap v11029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:19:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:40 smithi059 ceph-mon[97450]: pgmap v11029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:19:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:41.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:41.404802+0000) 2022-09-08T03:19:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:41.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:41.372559+0000) 2022-09-08T03:19:42.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:42.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:42.405109+0000) 2022-09-08T03:19:42.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:42.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:42.372702+0000) 2022-09-08T03:19:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:42 smithi055 ceph-mon[97033]: pgmap v11030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:42 smithi059 ceph-mon[97450]: pgmap v11030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:43.372 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:19:42.917+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:19:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:43.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:43.372894+0000) 2022-09-08T03:19:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:43.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:43.405337+0000) 2022-09-08T03:19:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:44.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:44.405556+0000) 2022-09-08T03:19:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:44.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:44.373100+0000) 2022-09-08T03:19:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:44 smithi055 ceph-mon[97033]: pgmap v11031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:44 smithi059 ceph-mon[97450]: pgmap v11031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:45.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:45.405811+0000) 2022-09-08T03:19:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:45.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:45.373350+0000) 2022-09-08T03:19:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:46.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:46.406093+0000) 2022-09-08T03:19:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:46.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:46.373590+0000) 2022-09-08T03:19:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:46 smithi055 ceph-mon[97033]: pgmap v11032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:19:47.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:46 smithi059 ceph-mon[97450]: pgmap v11032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:19:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:47.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:47.406323+0000) 2022-09-08T03:19:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:47.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:47.373863+0000) 2022-09-08T03:19:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:48.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:48.406535+0000) 2022-09-08T03:19:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:48.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:48.374012+0000) 2022-09-08T03:19:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:48 smithi055 ceph-mon[97033]: pgmap v11033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:48 smithi059 ceph-mon[97450]: pgmap v11033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:49.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:49.374250+0000) 2022-09-08T03:19:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:49.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:49.406720+0000) 2022-09-08T03:19:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:50.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:50.406972+0000) 2022-09-08T03:19:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:50.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:50.374396+0000) 2022-09-08T03:19:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:50 smithi055 ceph-mon[97033]: pgmap v11034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:19:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:50 smithi059 ceph-mon[97450]: pgmap v11034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:19:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:51.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:51.407183+0000) 2022-09-08T03:19:51.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:51.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:51.374572+0000) 2022-09-08T03:19:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:52.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:52.407460+0000) 2022-09-08T03:19:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:52.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:52.374731+0000) 2022-09-08T03:19:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:52 smithi055 ceph-mon[97033]: pgmap v11035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:53.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:52 smithi059 ceph-mon[97450]: pgmap v11035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:53.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:53.407724+0000) 2022-09-08T03:19:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:53.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:53.374960+0000) 2022-09-08T03:19:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:54.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:54.407986+0000) 2022-09-08T03:19:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:54.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:54.375149+0000) 2022-09-08T03:19:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:54 smithi055 ceph-mon[97033]: pgmap v11036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:54 smithi059 ceph-mon[97450]: pgmap v11036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:55.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:55.408158+0000) 2022-09-08T03:19:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:55.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:55.375431+0000) 2022-09-08T03:19:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:56.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:56.408382+0000) 2022-09-08T03:19:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:56.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:56.375689+0000) 2022-09-08T03:19:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:56 smithi055 ceph-mon[97033]: pgmap v11037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:19:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:56 smithi059 ceph-mon[97450]: pgmap v11037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:19:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:57.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:57.408578+0000) 2022-09-08T03:19:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:57.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:57.375970+0000) 2022-09-08T03:19:58.376 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:19:57.920+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:19:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:58.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:58.408780+0000) 2022-09-08T03:19:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:58.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:58.376166+0000) 2022-09-08T03:19:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:58 smithi055 ceph-mon[97033]: pgmap v11038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:19:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:19:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:19:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:19:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:19:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:58 smithi059 ceph-mon[97450]: pgmap v11038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:19:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:19:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:19:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:19:59.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:19:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:19:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:19:59.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:59.409008+0000) 2022-09-08T03:19:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:19:59.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:19:59.376451+0000) 2022-09-08T03:20:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:00.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:00.409242+0000) 2022-09-08T03:20:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:00.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:00.376680+0000) 2022-09-08T03:20:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:00 smithi055 ceph-mon[97033]: pgmap v11039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:20:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T03:20:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T03:20:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T03:20:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:00 smithi059 ceph-mon[97450]: pgmap v11039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:20:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T03:20:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T03:20:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T03:20:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:01.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:01.409476+0000) 2022-09-08T03:20:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:01.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:01.376812+0000) 2022-09-08T03:20:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:02.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:02.409692+0000) 2022-09-08T03:20:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:02.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:02.376992+0000) 2022-09-08T03:20:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:02 smithi055 ceph-mon[97033]: pgmap v11040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:02 smithi059 ceph-mon[97450]: pgmap v11040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:03.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:03.409901+0000) 2022-09-08T03:20:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:03.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:03.377197+0000) 2022-09-08T03:20:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:04.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:04.410061+0000) 2022-09-08T03:20:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:04.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:04.377402+0000) 2022-09-08T03:20:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:04 smithi055 ceph-mon[97033]: pgmap v11041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:04 smithi059 ceph-mon[97450]: pgmap v11041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:05.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:05.410264+0000) 2022-09-08T03:20:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:05.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:05.377588+0000) 2022-09-08T03:20:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:20:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:20:06.662 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:06.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:06.410496+0000) 2022-09-08T03:20:06.663 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:06.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:06.377720+0000) 2022-09-08T03:20:06.862 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:20:06.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:06 smithi059 ceph-mon[97450]: pgmap v11042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:20:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:20:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:06 smithi055 ceph-mon[97033]: pgmap v11042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:20:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:07.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:07.410740+0000) 2022-09-08T03:20:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:07.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:07.377941+0000) 2022-09-08T03:20:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:08.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:08.410901+0000) 2022-09-08T03:20:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:08.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:08.378182+0000) 2022-09-08T03:20:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:08 smithi055 ceph-mon[97033]: pgmap v11043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:20:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:08 smithi059 ceph-mon[97450]: pgmap v11043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:20:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:09.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:09.411118+0000) 2022-09-08T03:20:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:09.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:09.378358+0000) 2022-09-08T03:20:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:10.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:10.411378+0000) 2022-09-08T03:20:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:10.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:10.378587+0000) 2022-09-08T03:20:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:10 smithi055 ceph-mon[97033]: osdmap e769: 12 total, 11 up, 11 in 2022-09-08T03:20:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:10 smithi055 ceph-mon[97033]: pgmap v11045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:20:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:10 smithi059 ceph-mon[97450]: osdmap e769: 12 total, 11 up, 11 in 2022-09-08T03:20:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:10 smithi059 ceph-mon[97450]: pgmap v11045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:20:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:11.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:11.411646+0000) 2022-09-08T03:20:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:11.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:11.378828+0000) 2022-09-08T03:20:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:12.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:12.411851+0000) 2022-09-08T03:20:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:12.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:12.378985+0000) 2022-09-08T03:20:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:12 smithi055 ceph-mon[97033]: pgmap v11046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:20:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:12 smithi059 ceph-mon[97450]: pgmap v11046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:20:13.379 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:20:12.934+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:20:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:13.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:13.412097+0000) 2022-09-08T03:20:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:13.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:13.379252+0000) 2022-09-08T03:20:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:14.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:14.379503+0000) 2022-09-08T03:20:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:14.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:14.412367+0000) 2022-09-08T03:20:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:14 smithi055 ceph-mon[97033]: osdmap e770: 12 total, 11 up, 11 in 2022-09-08T03:20:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:14 smithi055 ceph-mon[97033]: pgmap v11048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:20:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:14 smithi059 ceph-mon[97450]: osdmap e770: 12 total, 11 up, 11 in 2022-09-08T03:20:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:14 smithi059 ceph-mon[97450]: pgmap v11048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:20:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:15.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:15.412602+0000) 2022-09-08T03:20:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:15.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:15.379744+0000) 2022-09-08T03:20:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:16.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:16.412831+0000) 2022-09-08T03:20:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:16.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:16.379990+0000) 2022-09-08T03:20:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:16 smithi055 ceph-mon[97033]: pgmap v11049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:20:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:16 smithi059 ceph-mon[97450]: pgmap v11049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:20:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:17.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:17.413056+0000) 2022-09-08T03:20:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:17.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:17.380245+0000) 2022-09-08T03:20:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:18.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:18.413244+0000) 2022-09-08T03:20:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:18.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:18.380450+0000) 2022-09-08T03:20:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:18 smithi055 ceph-mon[97033]: pgmap v11050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T03:20:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:18 smithi059 ceph-mon[97450]: pgmap v11050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T03:20:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:19.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:19.413403+0000) 2022-09-08T03:20:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:19.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:19.380696+0000) 2022-09-08T03:20:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:20.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:20.413671+0000) 2022-09-08T03:20:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:20.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:20.380908+0000) 2022-09-08T03:20:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:20 smithi055 ceph-mon[97033]: pgmap v11051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T03:20:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:20 smithi059 ceph-mon[97450]: pgmap v11051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T03:20:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:21.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:21.413920+0000) 2022-09-08T03:20:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:21.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:21.381158+0000) 2022-09-08T03:20:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:22.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:22.414149+0000) 2022-09-08T03:20:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:22.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:22.381347+0000) 2022-09-08T03:20:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:22 smithi059 ceph-mon[97450]: pgmap v11052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T03:20:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:22 smithi055 ceph-mon[97033]: pgmap v11052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T03:20:23.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:23.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:23.381657+0000) 2022-09-08T03:20:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:23.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:23.414381+0000) 2022-09-08T03:20:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:24.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:24.414591+0000) 2022-09-08T03:20:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:24.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:24.381851+0000) 2022-09-08T03:20:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:24 smithi055 ceph-mon[97033]: pgmap v11053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T03:20:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:24 smithi059 ceph-mon[97450]: pgmap v11053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T03:20:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:25.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:25.414781+0000) 2022-09-08T03:20:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:25.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:25.382004+0000) 2022-09-08T03:20:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:26.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:26.414964+0000) 2022-09-08T03:20:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:26.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:26.382172+0000) 2022-09-08T03:20:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:26 smithi055 ceph-mon[97033]: pgmap v11054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:20:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:26 smithi059 ceph-mon[97450]: pgmap v11054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:20:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:27.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:27.415089+0000) 2022-09-08T03:20:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:27.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:27.382441+0000) 2022-09-08T03:20:28.382 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:20:27.944+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:20:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:28.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:28.415307+0000) 2022-09-08T03:20:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:28.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:28.382756+0000) 2022-09-08T03:20:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:28 smithi059 ceph-mon[97450]: pgmap v11055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:28 smithi055 ceph-mon[97033]: pgmap v11055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:29.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:29.415569+0000) 2022-09-08T03:20:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:29.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:29.382946+0000) 2022-09-08T03:20:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:30.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:30.415799+0000) 2022-09-08T03:20:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:30.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:30.383178+0000) 2022-09-08T03:20:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:30 smithi055 ceph-mon[97033]: pgmap v11056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:20:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:30 smithi059 ceph-mon[97450]: pgmap v11056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:20:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:31.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:31.416026+0000) 2022-09-08T03:20:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:31.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:31.383414+0000) 2022-09-08T03:20:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:32.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:32.383722+0000) 2022-09-08T03:20:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:32.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:32.416305+0000) 2022-09-08T03:20:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:32 smithi059 ceph-mon[97450]: pgmap v11057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:32 smithi055 ceph-mon[97033]: pgmap v11057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:33.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:33.416476+0000) 2022-09-08T03:20:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:33.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:33.383961+0000) 2022-09-08T03:20:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:34.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:34.416733+0000) 2022-09-08T03:20:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:34.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:34.384105+0000) 2022-09-08T03:20:34.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:34 smithi055 ceph-mon[97033]: pgmap v11058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:34 smithi059 ceph-mon[97450]: pgmap v11058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:35.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:35.416980+0000) 2022-09-08T03:20:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:35.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:35.384274+0000) 2022-09-08T03:20:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:36.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:36.384521+0000) 2022-09-08T03:20:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:36.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:36.417205+0000) 2022-09-08T03:20:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:36 smithi055 ceph-mon[97033]: pgmap v11059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:20:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:36 smithi059 ceph-mon[97450]: pgmap v11059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:20:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:37.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:37.417451+0000) 2022-09-08T03:20:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:37.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:37.384791+0000) 2022-09-08T03:20:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:38.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:38.385061+0000) 2022-09-08T03:20:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:38.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:38.417712+0000) 2022-09-08T03:20:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:38 smithi055 ceph-mon[97033]: pgmap v11060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:38 smithi059 ceph-mon[97450]: pgmap v11060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:39.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:39.417940+0000) 2022-09-08T03:20:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:39.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:39.385380+0000) 2022-09-08T03:20:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:40.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:40.385591+0000) 2022-09-08T03:20:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:40.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:40.418116+0000) 2022-09-08T03:20:41.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:40 smithi059 ceph-mon[97450]: pgmap v11061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:20:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:40 smithi055 ceph-mon[97033]: pgmap v11061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:20:41.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:41.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:41.385862+0000) 2022-09-08T03:20:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:41.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:41.418360+0000) 2022-09-08T03:20:42.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:42.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:42.386115+0000) 2022-09-08T03:20:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:42.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:42.418598+0000) 2022-09-08T03:20:43.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:42 smithi059 ceph-mon[97450]: pgmap v11062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:43.386 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:20:42.954+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:20:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:42 smithi055 ceph-mon[97033]: pgmap v11062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:43.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:43.418733+0000) 2022-09-08T03:20:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:43.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:43.386368+0000) 2022-09-08T03:20:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:44.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:44.418924+0000) 2022-09-08T03:20:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:44.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:44.386633+0000) 2022-09-08T03:20:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:44 smithi055 ceph-mon[97033]: pgmap v11063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:45.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:44 smithi059 ceph-mon[97450]: pgmap v11063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:45.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:45.386866+0000) 2022-09-08T03:20:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:45.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:45.419148+0000) 2022-09-08T03:20:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:46.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:46.419373+0000) 2022-09-08T03:20:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:46.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:46.387009+0000) 2022-09-08T03:20:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:46 smithi055 ceph-mon[97033]: pgmap v11064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:20:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:46 smithi059 ceph-mon[97450]: pgmap v11064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:20:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:47.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:47.419594+0000) 2022-09-08T03:20:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:47.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:47.387257+0000) 2022-09-08T03:20:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:48.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:48.419814+0000) 2022-09-08T03:20:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:48.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:48.387564+0000) 2022-09-08T03:20:49.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:48 smithi059 ceph-mon[97450]: pgmap v11065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:48 smithi055 ceph-mon[97033]: pgmap v11065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:49.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:49.419972+0000) 2022-09-08T03:20:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:49.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:49.387794+0000) 2022-09-08T03:20:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:50.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:50.420147+0000) 2022-09-08T03:20:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:50.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:50.388036+0000) 2022-09-08T03:20:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:50 smithi059 ceph-mon[97450]: pgmap v11066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:20:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:50 smithi055 ceph-mon[97033]: pgmap v11066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:20:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:51.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:51.420333+0000) 2022-09-08T03:20:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:51.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:51.388296+0000) 2022-09-08T03:20:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:52.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:52.420549+0000) 2022-09-08T03:20:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:52.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:52.388541+0000) 2022-09-08T03:20:53.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:52 smithi059 ceph-mon[97450]: pgmap v11067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:52 smithi055 ceph-mon[97033]: pgmap v11067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:53.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:53.420767+0000) 2022-09-08T03:20:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:53.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:53.388808+0000) 2022-09-08T03:20:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:54.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:54.421004+0000) 2022-09-08T03:20:54.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:54.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:54.389050+0000) 2022-09-08T03:20:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:54 smithi055 ceph-mon[97033]: pgmap v11068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:54 smithi059 ceph-mon[97450]: pgmap v11068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:55.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:55.421243+0000) 2022-09-08T03:20:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:55.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:55.389216+0000) 2022-09-08T03:20:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:56.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:56.421477+0000) 2022-09-08T03:20:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:56.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:56.389454+0000) 2022-09-08T03:20:57.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:56 smithi059 ceph-mon[97450]: pgmap v11069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:20:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:56 smithi055 ceph-mon[97033]: pgmap v11069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:20:57.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:57.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:57.389724+0000) 2022-09-08T03:20:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:57.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:57.421731+0000) 2022-09-08T03:20:58.389 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:20:57.955+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:20:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:58.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:58.421864+0000) 2022-09-08T03:20:58.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:58.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:58.389934+0000) 2022-09-08T03:20:59.266 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:58 smithi059 ceph-mon[97450]: pgmap v11070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:59.266 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:20:59.267 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:20:59.267 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:20:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:20:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:58 smithi055 ceph-mon[97033]: pgmap v11070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:20:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:20:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:20:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:20:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:20:59.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:20:59.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:59.390154+0000) 2022-09-08T03:20:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:20:59.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:20:59.422092+0000) 2022-09-08T03:21:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:21:00.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:21:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:00.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:00.390291+0000) 2022-09-08T03:21:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:00.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:00.422244+0000) 2022-09-08T03:21:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:01 smithi055 ceph-mon[97033]: pgmap v11071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:21:01.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:01 smithi059 ceph-mon[97450]: pgmap v11071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:21:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:01.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:01.422457+0000) 2022-09-08T03:21:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:01.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:01.390537+0000) 2022-09-08T03:21:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:02.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:02.422701+0000) 2022-09-08T03:21:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:02.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:02.390718+0000) 2022-09-08T03:21:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:03 smithi055 ceph-mon[97033]: pgmap v11072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:03 smithi059 ceph-mon[97450]: pgmap v11072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:03.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:03.422938+0000) 2022-09-08T03:21:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:03.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:03.390939+0000) 2022-09-08T03:21:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:04.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:04.391212+0000) 2022-09-08T03:21:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:04.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:04.423181+0000) 2022-09-08T03:21:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:04 smithi055 ceph-mon[97033]: pgmap v11073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:04 smithi059 ceph-mon[97450]: pgmap v11073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:05.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:05.391421+0000) 2022-09-08T03:21:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:05.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:05.423421+0000) 2022-09-08T03:21:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:21:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:21:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:06.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:06.423631+0000) 2022-09-08T03:21:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:06.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:06.391665+0000) 2022-09-08T03:21:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:21:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:06 smithi055 ceph-mon[97033]: pgmap v11074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:21:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:21:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:06 smithi059 ceph-mon[97450]: pgmap v11074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:21:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:07.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:07.423834+0000) 2022-09-08T03:21:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:07.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:07.391912+0000) 2022-09-08T03:21:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:08.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:08.424069+0000) 2022-09-08T03:21:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:08.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:08.392110+0000) 2022-09-08T03:21:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:08 smithi059 ceph-mon[97450]: pgmap v11075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:21:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:08 smithi055 ceph-mon[97033]: pgmap v11075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:21:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:09.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:09.424272+0000) 2022-09-08T03:21:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:09.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:09.392355+0000) 2022-09-08T03:21:10.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:10.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:10.392586+0000) 2022-09-08T03:21:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:10.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:10.424517+0000) 2022-09-08T03:21:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:10 smithi055 ceph-mon[97033]: osdmap e771: 12 total, 11 up, 11 in 2022-09-08T03:21:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:10 smithi055 ceph-mon[97033]: pgmap v11077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:21:11.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:10 smithi059 ceph-mon[97450]: osdmap e771: 12 total, 11 up, 11 in 2022-09-08T03:21:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:10 smithi059 ceph-mon[97450]: pgmap v11077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:21:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:11.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:11.424744+0000) 2022-09-08T03:21:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:11.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:11.392861+0000) 2022-09-08T03:21:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:12.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:12.393105+0000) 2022-09-08T03:21:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:12.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:12.424984+0000) 2022-09-08T03:21:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:12 smithi059 ceph-mon[97450]: pgmap v11078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:21:13.394 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:21:12.957+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:21:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:12 smithi055 ceph-mon[97033]: pgmap v11078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:21:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:13.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:13.425146+0000) 2022-09-08T03:21:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:13.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:13.393261+0000) 2022-09-08T03:21:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:14.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:14.425362+0000) 2022-09-08T03:21:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:14.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:14.393425+0000) 2022-09-08T03:21:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:14 smithi055 ceph-mon[97033]: osdmap e772: 12 total, 11 up, 11 in 2022-09-08T03:21:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:14 smithi055 ceph-mon[97033]: pgmap v11080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:21:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:14 smithi059 ceph-mon[97450]: osdmap e772: 12 total, 11 up, 11 in 2022-09-08T03:21:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:14 smithi059 ceph-mon[97450]: pgmap v11080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:21:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:15.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:15.425562+0000) 2022-09-08T03:21:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:15.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:15.393686+0000) 2022-09-08T03:21:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:16.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:16.425705+0000) 2022-09-08T03:21:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:16.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:16.393931+0000) 2022-09-08T03:21:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:16 smithi059 ceph-mon[97450]: pgmap v11081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:21:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:16 smithi055 ceph-mon[97033]: pgmap v11081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:21:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:17.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:17.425946+0000) 2022-09-08T03:21:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:17.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:17.394148+0000) 2022-09-08T03:21:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:18.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:18.394299+0000) 2022-09-08T03:21:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:18.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:18.426167+0000) 2022-09-08T03:21:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:18 smithi059 ceph-mon[97450]: pgmap v11082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T03:21:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:18 smithi055 ceph-mon[97033]: pgmap v11082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T03:21:19.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:19.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:19.394502+0000) 2022-09-08T03:21:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:19.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:19.426371+0000) 2022-09-08T03:21:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:20.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:20.394770+0000) 2022-09-08T03:21:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:20.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:20.426684+0000) 2022-09-08T03:21:21.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:20 smithi059 ceph-mon[97450]: pgmap v11083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:21:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:20 smithi055 ceph-mon[97033]: pgmap v11083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:21:21.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:21.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:21.395022+0000) 2022-09-08T03:21:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:21.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:21.426859+0000) 2022-09-08T03:21:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:22.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:22.427080+0000) 2022-09-08T03:21:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:22.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:22.395193+0000) 2022-09-08T03:21:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:22 smithi059 ceph-mon[97450]: pgmap v11084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:21:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:22 smithi055 ceph-mon[97033]: pgmap v11084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:21:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:23.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:23.427288+0000) 2022-09-08T03:21:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:23.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:23.395425+0000) 2022-09-08T03:21:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:24.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:24.427539+0000) 2022-09-08T03:21:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:24.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:24.395662+0000) 2022-09-08T03:21:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:24 smithi055 ceph-mon[97033]: pgmap v11085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T03:21:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:24 smithi059 ceph-mon[97450]: pgmap v11085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T03:21:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:25.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:25.427744+0000) 2022-09-08T03:21:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:25.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:25.395896+0000) 2022-09-08T03:21:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:26.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:26.427920+0000) 2022-09-08T03:21:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:26.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:26.396135+0000) 2022-09-08T03:21:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:26 smithi059 ceph-mon[97450]: pgmap v11086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:21:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:26 smithi055 ceph-mon[97033]: pgmap v11086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:21:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:27.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:27.428150+0000) 2022-09-08T03:21:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:27.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:27.396364+0000) 2022-09-08T03:21:28.396 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:21:27.964+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:21:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:28.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:28.428411+0000) 2022-09-08T03:21:28.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:28.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:28.396500+0000) 2022-09-08T03:21:29.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:28 smithi059 ceph-mon[97450]: pgmap v11087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:28 smithi055 ceph-mon[97033]: pgmap v11087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:29.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:29.428697+0000) 2022-09-08T03:21:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:29.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:29.396743+0000) 2022-09-08T03:21:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:30.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:30.428884+0000) 2022-09-08T03:21:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:30.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:30.396987+0000) 2022-09-08T03:21:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:30 smithi059 ceph-mon[97450]: pgmap v11088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:21:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:30 smithi055 ceph-mon[97033]: pgmap v11088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:21:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:31.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:31.429106+0000) 2022-09-08T03:21:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:31.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:31.397226+0000) 2022-09-08T03:21:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:32.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:32.397461+0000) 2022-09-08T03:21:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:32.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:32.429361+0000) 2022-09-08T03:21:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:32 smithi059 ceph-mon[97450]: pgmap v11089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:32 smithi055 ceph-mon[97033]: pgmap v11089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:33.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:33.429597+0000) 2022-09-08T03:21:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:33.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:33.397728+0000) 2022-09-08T03:21:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:34.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:34.429775+0000) 2022-09-08T03:21:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:34.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:34.397959+0000) 2022-09-08T03:21:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:34 smithi055 ceph-mon[97033]: pgmap v11090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:34 smithi059 ceph-mon[97450]: pgmap v11090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:35.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:35.429955+0000) 2022-09-08T03:21:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:35.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:35.398159+0000) 2022-09-08T03:21:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:36.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:36.430150+0000) 2022-09-08T03:21:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:36.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:36.398340+0000) 2022-09-08T03:21:37.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:36 smithi059 ceph-mon[97450]: pgmap v11091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:21:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:36 smithi055 ceph-mon[97033]: pgmap v11091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:21:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:37.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:37.430350+0000) 2022-09-08T03:21:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:37.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:37.398545+0000) 2022-09-08T03:21:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:38.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:38.398790+0000) 2022-09-08T03:21:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:38.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:38.430546+0000) 2022-09-08T03:21:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:38 smithi059 ceph-mon[97450]: pgmap v11092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:39 smithi055 ceph-mon[97033]: pgmap v11092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:39.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:39.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:39.430758+0000) 2022-09-08T03:21:39.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:39.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:39.398914+0000) 2022-09-08T03:21:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:40.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:40.430890+0000) 2022-09-08T03:21:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:40.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:40.399157+0000) 2022-09-08T03:21:41.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:40 smithi059 ceph-mon[97450]: pgmap v11093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:21:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:41 smithi055 ceph-mon[97033]: pgmap v11093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:21:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:41.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:41.431119+0000) 2022-09-08T03:21:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:41.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:41.399383+0000) 2022-09-08T03:21:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:42.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:42.431397+0000) 2022-09-08T03:21:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:42.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:42.399608+0000) 2022-09-08T03:21:43.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:43 smithi059 ceph-mon[97450]: pgmap v11094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:43.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:21:42.973+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:21:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:43 smithi055 ceph-mon[97033]: pgmap v11094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:43.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:43.399820+0000) 2022-09-08T03:21:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:43.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:43.431677+0000) 2022-09-08T03:21:44.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:44.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:44.400006+0000) 2022-09-08T03:21:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:44.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:44.431930+0000) 2022-09-08T03:21:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:44 smithi055 ceph-mon[97033]: pgmap v11095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:45.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:44 smithi059 ceph-mon[97450]: pgmap v11095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:45.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:45.400223+0000) 2022-09-08T03:21:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:45.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:45.432198+0000) 2022-09-08T03:21:46.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:46.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:46.400460+0000) 2022-09-08T03:21:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:46.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:46.432403+0000) 2022-09-08T03:21:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:46 smithi059 ceph-mon[97450]: pgmap v11096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:21:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:46 smithi055 ceph-mon[97033]: pgmap v11096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:21:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:47.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:47.432625+0000) 2022-09-08T03:21:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:47.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:47.400721+0000) 2022-09-08T03:21:48.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:48.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:48.400889+0000) 2022-09-08T03:21:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:48.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:48.432895+0000) 2022-09-08T03:21:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:48 smithi059 ceph-mon[97450]: pgmap v11097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:49 smithi055 ceph-mon[97033]: pgmap v11097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:49.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:49.433122+0000) 2022-09-08T03:21:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:49.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:49.401023+0000) 2022-09-08T03:21:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:50.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:50.401143+0000) 2022-09-08T03:21:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:50.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:50.433354+0000) 2022-09-08T03:21:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:51 smithi059 ceph-mon[97450]: pgmap v11098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:21:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:51 smithi055 ceph-mon[97033]: pgmap v11098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:21:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:51.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:51.433600+0000) 2022-09-08T03:21:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:51.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:51.401324+0000) 2022-09-08T03:21:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:52.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:52.401507+0000) 2022-09-08T03:21:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:52.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:52.433843+0000) 2022-09-08T03:21:53.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:53 smithi059 ceph-mon[97450]: pgmap v11099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:53 smithi055 ceph-mon[97033]: pgmap v11099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:53.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:53.434028+0000) 2022-09-08T03:21:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:53.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:53.401668+0000) 2022-09-08T03:21:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:54.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:54.434292+0000) 2022-09-08T03:21:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:54.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:54.401846+0000) 2022-09-08T03:21:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:54 smithi055 ceph-mon[97033]: pgmap v11100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:54 smithi059 ceph-mon[97450]: pgmap v11100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:55.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:55.434550+0000) 2022-09-08T03:21:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:55.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:55.402011+0000) 2022-09-08T03:21:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:56.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:56.434798+0000) 2022-09-08T03:21:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:56.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:56.402252+0000) 2022-09-08T03:21:57.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:57 smithi059 ceph-mon[97450]: pgmap v11101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:21:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:57 smithi055 ceph-mon[97033]: pgmap v11101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:21:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:57.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:57.435040+0000) 2022-09-08T03:21:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:57.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:57.402516+0000) 2022-09-08T03:21:58.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:21:57.979+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:21:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:58.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:58.435295+0000) 2022-09-08T03:21:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:58.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:58.402755+0000) 2022-09-08T03:21:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:21:59 smithi059 ceph-mon[97450]: pgmap v11102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:59.330 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:21:59 smithi055 ceph-mon[97033]: pgmap v11102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:21:59.914 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:21:59.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:59.435546+0000) 2022-09-08T03:21:59.915 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:21:59.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:21:59.402992+0000) 2022-09-08T03:22:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:22:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:22:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:22:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:22:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:22:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:22:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:22:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:22:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:22:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:22:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:00.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:00.435831+0000) 2022-09-08T03:22:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:00.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:00.403240+0000) 2022-09-08T03:22:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:01 smithi059 ceph-mon[97450]: pgmap v11103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:22:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:01 smithi055 ceph-mon[97033]: pgmap v11103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:22:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:01.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:01.436050+0000) 2022-09-08T03:22:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:01.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:01.403430+0000) 2022-09-08T03:22:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:02.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:02.436246+0000) 2022-09-08T03:22:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:02.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:02.403656+0000) 2022-09-08T03:22:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:03 smithi059 ceph-mon[97450]: pgmap v11104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:22:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:22:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:22:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:22:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:03.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:03.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:03 smithi055 ceph-mon[97033]: pgmap v11104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:03.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:22:03.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:22:03.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:22:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:22:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:03.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:03.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:03.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:03.436501+0000) 2022-09-08T03:22:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:03.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:03.403824+0000) 2022-09-08T03:22:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:04 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T03:22:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:04 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T03:22:04.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:04.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:04.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:04.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:04.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:04 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T03:22:04.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:04 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T03:22:04.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:22:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:04 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T03:22:04.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:04 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T03:22:04.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:04.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:04.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:04.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:22:04.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:04 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T03:22:04.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:04 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T03:22:04.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:22:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:04.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:04.436762+0000) 2022-09-08T03:22:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:04.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:04.404067+0000) 2022-09-08T03:22:05.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:05 smithi059 ceph-mon[97450]: pgmap v11105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:05 smithi055 ceph-mon[97033]: pgmap v11105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:05.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:05.436999+0000) 2022-09-08T03:22:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:05.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:05.404322+0000) 2022-09-08T03:22:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:22:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:22:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:22:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:22:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:06.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:06.437280+0000) 2022-09-08T03:22:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:06.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:06.404572+0000) 2022-09-08T03:22:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:07 smithi059 ceph-mon[97450]: pgmap v11106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:22:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:07 smithi055 ceph-mon[97033]: pgmap v11106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:22:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:07.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:07.437533+0000) 2022-09-08T03:22:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:07.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:07.404813+0000) 2022-09-08T03:22:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:08.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:08.437819+0000) 2022-09-08T03:22:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:08.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:08.405016+0000) 2022-09-08T03:22:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:09 smithi059 ceph-mon[97450]: pgmap v11107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:22:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:09 smithi055 ceph-mon[97033]: pgmap v11107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:22:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:09.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:09.438126+0000) 2022-09-08T03:22:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:09.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:09.405242+0000) 2022-09-08T03:22:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:10.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:10.438329+0000) 2022-09-08T03:22:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:10.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:10.405431+0000) 2022-09-08T03:22:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:10 smithi055 ceph-mon[97033]: osdmap e773: 12 total, 11 up, 11 in 2022-09-08T03:22:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:10 smithi055 ceph-mon[97033]: pgmap v11109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:22:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:10 smithi059 ceph-mon[97450]: osdmap e773: 12 total, 11 up, 11 in 2022-09-08T03:22:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:10 smithi059 ceph-mon[97450]: pgmap v11109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:22:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:11.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:11.438561+0000) 2022-09-08T03:22:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:11.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:11.405735+0000) 2022-09-08T03:22:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:12.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:12.438867+0000) 2022-09-08T03:22:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:12.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:12.405967+0000) 2022-09-08T03:22:13.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:13 smithi059 ceph-mon[97450]: pgmap v11110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:22:13.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:22:12.987+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:22:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:13 smithi055 ceph-mon[97033]: pgmap v11110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:22:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:13.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:13.438991+0000) 2022-09-08T03:22:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:13.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:13.406152+0000) 2022-09-08T03:22:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:14.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:14.439196+0000) 2022-09-08T03:22:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:14.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:14.406366+0000) 2022-09-08T03:22:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:14 smithi055 ceph-mon[97033]: pgmap v11111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:22:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:14 smithi059 ceph-mon[97450]: pgmap v11111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:22:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:15.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:15.439369+0000) 2022-09-08T03:22:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:15.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:15.406546+0000) 2022-09-08T03:22:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:16.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:16.439569+0000) 2022-09-08T03:22:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:16.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:16.406746+0000) 2022-09-08T03:22:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:17 smithi059 ceph-mon[97450]: pgmap v11112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:22:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:17 smithi055 ceph-mon[97033]: pgmap v11112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:22:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:17.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:17.407009+0000) 2022-09-08T03:22:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:17.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:17.439742+0000) 2022-09-08T03:22:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:18.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:18.439952+0000) 2022-09-08T03:22:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:18.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:18.407274+0000) 2022-09-08T03:22:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:19 smithi059 ceph-mon[97450]: pgmap v11113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:22:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:19 smithi059 ceph-mon[97450]: osdmap e774: 12 total, 11 up, 11 in 2022-09-08T03:22:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:19 smithi055 ceph-mon[97033]: pgmap v11113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:22:19.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:19 smithi055 ceph-mon[97033]: osdmap e774: 12 total, 11 up, 11 in 2022-09-08T03:22:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:19.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:19.440187+0000) 2022-09-08T03:22:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:19.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:19.407491+0000) 2022-09-08T03:22:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:20.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:20.440426+0000) 2022-09-08T03:22:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:20.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:20.407734+0000) 2022-09-08T03:22:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:21 smithi059 ceph-mon[97450]: pgmap v11115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:22:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:21 smithi055 ceph-mon[97033]: pgmap v11115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:22:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:21.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:21.440714+0000) 2022-09-08T03:22:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:21.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:21.407954+0000) 2022-09-08T03:22:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:22.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:22.440984+0000) 2022-09-08T03:22:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:22.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:22.408161+0000) 2022-09-08T03:22:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:23 smithi059 ceph-mon[97450]: pgmap v11116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:22:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:23 smithi055 ceph-mon[97033]: pgmap v11116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:22:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:23.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:23.441170+0000) 2022-09-08T03:22:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:23.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:23.408412+0000) 2022-09-08T03:22:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:24.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:24.408632+0000) 2022-09-08T03:22:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:24.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:24.441379+0000) 2022-09-08T03:22:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:24 smithi055 ceph-mon[97033]: pgmap v11117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:22:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:24 smithi059 ceph-mon[97450]: pgmap v11117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:22:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:25.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:25.441594+0000) 2022-09-08T03:22:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:25.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:25.408834+0000) 2022-09-08T03:22:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:26.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:26.441853+0000) 2022-09-08T03:22:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:26.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:26.409059+0000) 2022-09-08T03:22:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:27 smithi059 ceph-mon[97450]: pgmap v11118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:22:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:27 smithi055 ceph-mon[97033]: pgmap v11118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:22:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:27.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:27.442024+0000) 2022-09-08T03:22:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:27.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:27.409283+0000) 2022-09-08T03:22:28.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:22:27.995+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:22:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:28.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:28.442173+0000) 2022-09-08T03:22:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:28.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:28.409509+0000) 2022-09-08T03:22:29.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:29 smithi059 ceph-mon[97450]: pgmap v11119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:22:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:29 smithi055 ceph-mon[97033]: pgmap v11119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:22:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:29.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:29.442413+0000) 2022-09-08T03:22:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:29.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:29.409730+0000) 2022-09-08T03:22:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:30.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:30.442688+0000) 2022-09-08T03:22:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:30.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:30.409965+0000) 2022-09-08T03:22:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:31 smithi059 ceph-mon[97450]: pgmap v11120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T03:22:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:31 smithi055 ceph-mon[97033]: pgmap v11120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T03:22:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:31.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:31.442891+0000) 2022-09-08T03:22:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:31.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:31.410212+0000) 2022-09-08T03:22:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:32.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:32.443119+0000) 2022-09-08T03:22:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:32.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:32.410405+0000) 2022-09-08T03:22:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:33 smithi059 ceph-mon[97450]: pgmap v11121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:33 smithi055 ceph-mon[97033]: pgmap v11121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:33.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:33.443363+0000) 2022-09-08T03:22:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:33.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:33.410618+0000) 2022-09-08T03:22:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:34.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:34.410860+0000) 2022-09-08T03:22:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:34.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:34.443581+0000) 2022-09-08T03:22:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:34 smithi055 ceph-mon[97033]: pgmap v11122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:34 smithi059 ceph-mon[97450]: pgmap v11122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:35.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:35.443787+0000) 2022-09-08T03:22:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:35.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:35.411104+0000) 2022-09-08T03:22:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:36.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:36.444074+0000) 2022-09-08T03:22:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:36.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:36.411353+0000) 2022-09-08T03:22:37.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:37 smithi059 ceph-mon[97450]: pgmap v11123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:22:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:37 smithi055 ceph-mon[97033]: pgmap v11123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:22:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:37.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:37.444321+0000) 2022-09-08T03:22:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:37.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:37.411579+0000) 2022-09-08T03:22:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:38.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:38.444547+0000) 2022-09-08T03:22:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:38.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:38.411825+0000) 2022-09-08T03:22:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:39 smithi059 ceph-mon[97450]: pgmap v11124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:39 smithi055 ceph-mon[97033]: pgmap v11124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:39.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:39.444777+0000) 2022-09-08T03:22:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:39.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:39.412031+0000) 2022-09-08T03:22:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:40.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:40.444939+0000) 2022-09-08T03:22:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:40.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:40.412259+0000) 2022-09-08T03:22:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:41 smithi059 ceph-mon[97450]: pgmap v11125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:22:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:41 smithi055 ceph-mon[97033]: pgmap v11125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:22:41.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:41.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:41.445179+0000) 2022-09-08T03:22:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:41.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:41.412488+0000) 2022-09-08T03:22:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:42.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:42.445465+0000) 2022-09-08T03:22:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:42.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:42.412668+0000) 2022-09-08T03:22:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:43 smithi059 ceph-mon[97450]: pgmap v11126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:43.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:22:43.005+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:22:43.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:43 smithi055 ceph-mon[97033]: pgmap v11126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:43.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:43.445766+0000) 2022-09-08T03:22:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:43.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:43.412815+0000) 2022-09-08T03:22:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:44.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:44.445948+0000) 2022-09-08T03:22:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:44.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:44.413000+0000) 2022-09-08T03:22:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:44 smithi055 ceph-mon[97033]: pgmap v11127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:45.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:44 smithi059 ceph-mon[97450]: pgmap v11127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:45.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:45.446161+0000) 2022-09-08T03:22:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:45.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:45.413224+0000) 2022-09-08T03:22:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:46.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:46.446340+0000) 2022-09-08T03:22:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:46.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:46.413434+0000) 2022-09-08T03:22:47.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:47 smithi059 ceph-mon[97450]: pgmap v11128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:22:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:47 smithi055 ceph-mon[97033]: pgmap v11128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:22:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:47.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:47.446605+0000) 2022-09-08T03:22:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:47.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:47.413627+0000) 2022-09-08T03:22:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:48.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:48.413792+0000) 2022-09-08T03:22:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:48.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:48.446915+0000) 2022-09-08T03:22:49.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:49 smithi059 ceph-mon[97450]: pgmap v11129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:49 smithi055 ceph-mon[97033]: pgmap v11129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:49.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:49.447079+0000) 2022-09-08T03:22:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:49.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:49.413972+0000) 2022-09-08T03:22:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:50.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:50.447249+0000) 2022-09-08T03:22:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:50.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:50.414216+0000) 2022-09-08T03:22:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:51 smithi059 ceph-mon[97450]: pgmap v11130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:22:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:51 smithi055 ceph-mon[97033]: pgmap v11130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:22:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:51.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:51.414469+0000) 2022-09-08T03:22:51.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:51.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:51.447476+0000) 2022-09-08T03:22:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:52.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:52.447703+0000) 2022-09-08T03:22:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:52.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:52.414743+0000) 2022-09-08T03:22:53.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:53 smithi059 ceph-mon[97450]: pgmap v11131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:53 smithi055 ceph-mon[97033]: pgmap v11131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:53.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:53.447909+0000) 2022-09-08T03:22:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:53.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:53.414890+0000) 2022-09-08T03:22:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:54.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:54.448125+0000) 2022-09-08T03:22:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:54.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:54.415088+0000) 2022-09-08T03:22:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:54 smithi055 ceph-mon[97033]: pgmap v11132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:54 smithi059 ceph-mon[97450]: pgmap v11132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:55.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:55.448350+0000) 2022-09-08T03:22:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:55.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:55.415266+0000) 2022-09-08T03:22:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:56.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:56.415502+0000) 2022-09-08T03:22:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:56.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:56.448596+0000) 2022-09-08T03:22:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:57 smithi059 ceph-mon[97450]: pgmap v11133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:22:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:57 smithi055 ceph-mon[97033]: pgmap v11133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:22:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:57.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:57.448750+0000) 2022-09-08T03:22:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:57.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:57.415746+0000) 2022-09-08T03:22:58.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:22:58.021+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:22:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:58.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:58.449020+0000) 2022-09-08T03:22:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:58.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:58.415941+0000) 2022-09-08T03:22:59.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:22:59 smithi059 ceph-mon[97450]: pgmap v11134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:22:59 smithi055 ceph-mon[97033]: pgmap v11134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:22:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:22:59.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:59.449263+0000) 2022-09-08T03:22:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:22:59.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:22:59.416169+0000) 2022-09-08T03:23:00.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:00.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:00.449531+0000) 2022-09-08T03:23:00.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:00.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:00.416414+0000) 2022-09-08T03:23:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:01 smithi059 ceph-mon[97450]: pgmap v11135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:23:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:01 smithi055 ceph-mon[97033]: pgmap v11135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:23:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:01.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:01.449676+0000) 2022-09-08T03:23:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:01.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:01.416661+0000) 2022-09-08T03:23:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:02.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:02.449870+0000) 2022-09-08T03:23:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:02.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:02.416833+0000) 2022-09-08T03:23:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:03 smithi059 ceph-mon[97450]: pgmap v11136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:03 smithi055 ceph-mon[97033]: pgmap v11136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:03.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:03.450087+0000) 2022-09-08T03:23:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:03.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:03.417098+0000) 2022-09-08T03:23:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:23:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:23:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:23:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:23:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:23:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:23:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:23:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:23:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:23:04.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:23:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:04.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:04.450310+0000) 2022-09-08T03:23:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:04.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:04.417323+0000) 2022-09-08T03:23:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:05 smithi055 ceph-mon[97033]: pgmap v11137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:05.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:23:05.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:05 smithi059 ceph-mon[97450]: pgmap v11137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:23:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:05.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:05.417528+0000) 2022-09-08T03:23:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:05.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:05.450434+0000) 2022-09-08T03:23:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:23:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:23:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:23:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:23:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:06.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:06.450603+0000) 2022-09-08T03:23:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:06.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:06.417706+0000) 2022-09-08T03:23:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:07 smithi055 ceph-mon[97033]: pgmap v11138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:23:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:07 smithi059 ceph-mon[97450]: pgmap v11138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:23:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:07.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:07.450840+0000) 2022-09-08T03:23:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:07.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:07.417881+0000) 2022-09-08T03:23:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:08.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:08.451095+0000) 2022-09-08T03:23:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:08.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:08.418130+0000) 2022-09-08T03:23:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:09 smithi055 ceph-mon[97033]: pgmap v11139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:23:09.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:09 smithi059 ceph-mon[97450]: pgmap v11139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:23:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:09.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:09.451341+0000) 2022-09-08T03:23:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:09.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:09.418324+0000) 2022-09-08T03:23:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:10.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:10.451510+0000) 2022-09-08T03:23:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:10.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:10.418510+0000) 2022-09-08T03:23:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:10 smithi055 ceph-mon[97033]: osdmap e775: 12 total, 11 up, 11 in 2022-09-08T03:23:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:10 smithi055 ceph-mon[97033]: pgmap v11141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:23:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:10 smithi059 ceph-mon[97450]: osdmap e775: 12 total, 11 up, 11 in 2022-09-08T03:23:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:10 smithi059 ceph-mon[97450]: pgmap v11141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:23:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:11.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:11.451777+0000) 2022-09-08T03:23:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:11.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:11.418757+0000) 2022-09-08T03:23:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:12.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:12.452009+0000) 2022-09-08T03:23:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:12.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:12.418968+0000) 2022-09-08T03:23:13.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:13 smithi059 ceph-mon[97450]: pgmap v11142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:23:13.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:23:13.028+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:23:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:13 smithi055 ceph-mon[97033]: pgmap v11142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:23:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:13.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:13.419186+0000) 2022-09-08T03:23:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:13.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:13.452233+0000) 2022-09-08T03:23:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:14.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:14.419392+0000) 2022-09-08T03:23:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:14.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:14.452373+0000) 2022-09-08T03:23:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:14 smithi055 ceph-mon[97033]: osdmap e776: 12 total, 11 up, 11 in 2022-09-08T03:23:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:14 smithi055 ceph-mon[97033]: pgmap v11144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:23:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:14 smithi059 ceph-mon[97450]: osdmap e776: 12 total, 11 up, 11 in 2022-09-08T03:23:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:14 smithi059 ceph-mon[97450]: pgmap v11144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:23:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:15.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:15.419693+0000) 2022-09-08T03:23:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:15.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:15.452596+0000) 2022-09-08T03:23:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:16.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:16.452780+0000) 2022-09-08T03:23:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:16.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:16.419977+0000) 2022-09-08T03:23:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:17 smithi059 ceph-mon[97450]: pgmap v11145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:23:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:17 smithi055 ceph-mon[97033]: pgmap v11145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:23:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:17.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:17.453018+0000) 2022-09-08T03:23:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:17.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:17.420261+0000) 2022-09-08T03:23:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:18.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:18.420506+0000) 2022-09-08T03:23:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:18.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:18.453288+0000) 2022-09-08T03:23:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:19 smithi059 ceph-mon[97450]: pgmap v11146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 244 B/s wr, 7 op/s 2022-09-08T03:23:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:19 smithi055 ceph-mon[97033]: pgmap v11146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 244 B/s wr, 7 op/s 2022-09-08T03:23:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:19.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:19.453534+0000) 2022-09-08T03:23:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:19.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:19.420731+0000) 2022-09-08T03:23:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:20.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:20.420989+0000) 2022-09-08T03:23:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:20.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:20.453795+0000) 2022-09-08T03:23:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:21 smithi055 ceph-mon[97033]: pgmap v11147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:23:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:21 smithi059 ceph-mon[97450]: pgmap v11147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:23:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:21.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:21.421161+0000) 2022-09-08T03:23:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:21.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:21.454063+0000) 2022-09-08T03:23:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:22.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:22.454333+0000) 2022-09-08T03:23:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:22.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:22.421413+0000) 2022-09-08T03:23:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:23 smithi055 ceph-mon[97033]: pgmap v11148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:23:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:23 smithi059 ceph-mon[97450]: pgmap v11148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:23:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:23.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:23.454577+0000) 2022-09-08T03:23:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:23.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:23.421711+0000) 2022-09-08T03:23:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:24.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:24.454871+0000) 2022-09-08T03:23:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:24.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:24.421955+0000) 2022-09-08T03:23:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:24 smithi055 ceph-mon[97033]: pgmap v11149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T03:23:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:24 smithi059 ceph-mon[97450]: pgmap v11149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T03:23:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:25.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:25.455105+0000) 2022-09-08T03:23:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:25.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:25.422143+0000) 2022-09-08T03:23:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:26.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:26.455308+0000) 2022-09-08T03:23:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:26.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:26.422342+0000) 2022-09-08T03:23:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:27 smithi059 ceph-mon[97450]: pgmap v11150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:23:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:27 smithi055 ceph-mon[97033]: pgmap v11150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:23:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:27.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:27.455473+0000) 2022-09-08T03:23:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:27.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:27.422571+0000) 2022-09-08T03:23:28.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:23:28.044+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:23:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:28.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:28.455695+0000) 2022-09-08T03:23:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:28.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:28.422815+0000) 2022-09-08T03:23:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:29 smithi055 ceph-mon[97033]: pgmap v11151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:29.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:29 smithi059 ceph-mon[97450]: pgmap v11151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:29.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:29.455861+0000) 2022-09-08T03:23:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:29.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:29.423003+0000) 2022-09-08T03:23:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:30.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:30.456065+0000) 2022-09-08T03:23:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:30.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:30.423205+0000) 2022-09-08T03:23:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:31 smithi055 ceph-mon[97033]: pgmap v11152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:23:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:31 smithi059 ceph-mon[97450]: pgmap v11152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:23:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:31.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:31.456242+0000) 2022-09-08T03:23:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:31.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:31.423444+0000) 2022-09-08T03:23:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:32.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:32.456387+0000) 2022-09-08T03:23:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:32.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:32.423619+0000) 2022-09-08T03:23:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:33 smithi055 ceph-mon[97033]: pgmap v11153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:33 smithi059 ceph-mon[97450]: pgmap v11153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:33.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:33.456573+0000) 2022-09-08T03:23:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:33.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:33.423806+0000) 2022-09-08T03:23:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:34.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:34.456806+0000) 2022-09-08T03:23:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:34.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:34.423996+0000) 2022-09-08T03:23:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:34 smithi055 ceph-mon[97033]: pgmap v11154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:34 smithi059 ceph-mon[97450]: pgmap v11154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:35.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:35.456999+0000) 2022-09-08T03:23:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:35.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:35.424195+0000) 2022-09-08T03:23:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:36.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:36.457226+0000) 2022-09-08T03:23:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:36.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:36.424326+0000) 2022-09-08T03:23:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:37 smithi055 ceph-mon[97033]: pgmap v11155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:23:37.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:37 smithi059 ceph-mon[97450]: pgmap v11155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:23:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:37.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:37.457360+0000) 2022-09-08T03:23:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:37.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:37.424551+0000) 2022-09-08T03:23:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:38.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:38.457566+0000) 2022-09-08T03:23:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:38.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:38.424770+0000) 2022-09-08T03:23:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:39 smithi055 ceph-mon[97033]: pgmap v11156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:39 smithi059 ceph-mon[97450]: pgmap v11156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:39.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:39.457793+0000) 2022-09-08T03:23:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:39.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:39.425054+0000) 2022-09-08T03:23:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:40.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:40.457996+0000) 2022-09-08T03:23:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:40.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:40.425286+0000) 2022-09-08T03:23:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:41 smithi055 ceph-mon[97033]: pgmap v11157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:23:41.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:41 smithi059 ceph-mon[97450]: pgmap v11157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:23:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:41.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:41.458194+0000) 2022-09-08T03:23:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:41.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:41.425511+0000) 2022-09-08T03:23:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:42.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:42.458396+0000) 2022-09-08T03:23:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:42.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:42.425714+0000) 2022-09-08T03:23:43.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:23:43.048+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:23:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:43 smithi055 ceph-mon[97033]: pgmap v11158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:43.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:43 smithi059 ceph-mon[97450]: pgmap v11158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:43.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:43.458610+0000) 2022-09-08T03:23:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:43.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:43.425959+0000) 2022-09-08T03:23:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:44.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:44.458829+0000) 2022-09-08T03:23:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:44.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:44.426142+0000) 2022-09-08T03:23:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:44 smithi055 ceph-mon[97033]: pgmap v11159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:44 smithi059 ceph-mon[97450]: pgmap v11159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:45.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:45.459020+0000) 2022-09-08T03:23:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:45.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:45.426388+0000) 2022-09-08T03:23:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:46.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:46.459255+0000) 2022-09-08T03:23:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:46.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:46.426687+0000) 2022-09-08T03:23:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:47 smithi055 ceph-mon[97033]: pgmap v11160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:23:47.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:47 smithi059 ceph-mon[97450]: pgmap v11160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:23:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:47.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:47.459454+0000) 2022-09-08T03:23:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:47.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:47.426936+0000) 2022-09-08T03:23:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:48.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:48.459685+0000) 2022-09-08T03:23:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:48.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:48.427215+0000) 2022-09-08T03:23:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:49 smithi055 ceph-mon[97033]: pgmap v11161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:49 smithi059 ceph-mon[97450]: pgmap v11161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:49.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:49.459845+0000) 2022-09-08T03:23:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:49.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:49.427432+0000) 2022-09-08T03:23:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:50.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:50.460012+0000) 2022-09-08T03:23:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:50.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:50.427704+0000) 2022-09-08T03:23:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:51 smithi055 ceph-mon[97033]: pgmap v11162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:23:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:51 smithi059 ceph-mon[97450]: pgmap v11162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:23:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:51.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:51.460191+0000) 2022-09-08T03:23:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:51.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:51.427979+0000) 2022-09-08T03:23:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:52.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:52.460427+0000) 2022-09-08T03:23:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:52.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:52.428211+0000) 2022-09-08T03:23:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:53 smithi055 ceph-mon[97033]: pgmap v11163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:53.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:53 smithi059 ceph-mon[97450]: pgmap v11163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:53.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:53.460583+0000) 2022-09-08T03:23:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:53.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:53.428429+0000) 2022-09-08T03:23:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:54.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:54.460792+0000) 2022-09-08T03:23:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:54.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:54.428607+0000) 2022-09-08T03:23:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:54 smithi055 ceph-mon[97033]: pgmap v11164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:54 smithi059 ceph-mon[97450]: pgmap v11164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:55.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:55.461033+0000) 2022-09-08T03:23:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:55.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:55.428824+0000) 2022-09-08T03:23:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:56.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:56.461270+0000) 2022-09-08T03:23:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:56.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:56.429078+0000) 2022-09-08T03:23:57.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:57 smithi055 ceph-mon[97033]: pgmap v11165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:23:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:57 smithi059 ceph-mon[97450]: pgmap v11165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:23:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:57.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:57.461536+0000) 2022-09-08T03:23:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:57.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:57.429292+0000) 2022-09-08T03:23:58.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:23:58.051+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:23:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:58.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:58.461779+0000) 2022-09-08T03:23:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:58.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:58.429488+0000) 2022-09-08T03:23:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:23:59 smithi055 ceph-mon[97033]: pgmap v11166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:23:59 smithi059 ceph-mon[97450]: pgmap v11166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:23:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:23:59.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:59.429729+0000) 2022-09-08T03:23:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:23:59.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:23:59.462027+0000) 2022-09-08T03:24:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:00.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:00.462301+0000) 2022-09-08T03:24:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:00.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:00.429851+0000) 2022-09-08T03:24:01.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:01 smithi055 ceph-mon[97033]: pgmap v11167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:24:01.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:01 smithi059 ceph-mon[97450]: pgmap v11167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:24:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:01.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:01.430025+0000) 2022-09-08T03:24:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:01.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:01.462511+0000) 2022-09-08T03:24:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:02.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:02.462707+0000) 2022-09-08T03:24:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:02.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:02.430149+0000) 2022-09-08T03:24:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:03 smithi055 ceph-mon[97033]: pgmap v11168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:03.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:03 smithi059 ceph-mon[97450]: pgmap v11168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:03.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:03.462859+0000) 2022-09-08T03:24:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:03.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:03.430411+0000) 2022-09-08T03:24:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:04.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:04.463009+0000) 2022-09-08T03:24:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:04.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:04.430712+0000) 2022-09-08T03:24:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:04 smithi055 ceph-mon[97033]: pgmap v11169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:24:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:24:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:24:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:04 smithi059 ceph-mon[97450]: pgmap v11169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:24:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:24:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:24:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:05.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:05.430958+0000) 2022-09-08T03:24:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:05.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:05.463243+0000) 2022-09-08T03:24:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:24:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:24:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:24:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:24:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:24:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:24:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:06.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:06.431235+0000) 2022-09-08T03:24:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:06.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:06.463472+0000) 2022-09-08T03:24:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:06 smithi055 ceph-mon[97033]: pgmap v11170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:24:07.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:06 smithi059 ceph-mon[97450]: pgmap v11170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:24:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:07.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:07.463743+0000) 2022-09-08T03:24:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:07.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:07.431440+0000) 2022-09-08T03:24:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:08.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:08.463933+0000) 2022-09-08T03:24:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:08.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:08.431696+0000) 2022-09-08T03:24:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:09 smithi055 ceph-mon[97033]: pgmap v11171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:24:09.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:09 smithi059 ceph-mon[97450]: pgmap v11171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:24:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:09.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:09.464050+0000) 2022-09-08T03:24:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:09.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:09.431918+0000) 2022-09-08T03:24:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:10.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:10.464270+0000) 2022-09-08T03:24:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:10.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:10.432150+0000) 2022-09-08T03:24:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:10 smithi055 ceph-mon[97033]: osdmap e777: 12 total, 11 up, 11 in 2022-09-08T03:24:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:10 smithi055 ceph-mon[97033]: pgmap v11173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:24:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:10 smithi059 ceph-mon[97450]: osdmap e777: 12 total, 11 up, 11 in 2022-09-08T03:24:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:10 smithi059 ceph-mon[97450]: pgmap v11173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:24:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:11.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:11.464494+0000) 2022-09-08T03:24:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:11.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:11.432368+0000) 2022-09-08T03:24:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:12.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:12.464693+0000) 2022-09-08T03:24:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:12.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:12.432594+0000) 2022-09-08T03:24:13.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:24:13.063+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:24:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:13 smithi055 ceph-mon[97033]: pgmap v11174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:24:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:13 smithi059 ceph-mon[97450]: pgmap v11174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:24:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:13.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:13.464912+0000) 2022-09-08T03:24:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:13.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:13.432899+0000) 2022-09-08T03:24:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:14.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:14.465054+0000) 2022-09-08T03:24:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:14.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:14.433126+0000) 2022-09-08T03:24:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:14 smithi055 ceph-mon[97033]: pgmap v11175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:24:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:14 smithi059 ceph-mon[97450]: pgmap v11175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:24:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:15.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:15.433346+0000) 2022-09-08T03:24:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:15.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:15.465286+0000) 2022-09-08T03:24:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:16.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:16.433499+0000) 2022-09-08T03:24:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:16.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:16.465492+0000) 2022-09-08T03:24:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:17 smithi055 ceph-mon[97033]: pgmap v11176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:24:17.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:17 smithi059 ceph-mon[97450]: pgmap v11176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:24:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:17.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:17.433734+0000) 2022-09-08T03:24:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:17.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:17.465744+0000) 2022-09-08T03:24:18.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:18.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:18.433862+0000) 2022-09-08T03:24:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:18.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:18.465969+0000) 2022-09-08T03:24:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:19 smithi055 ceph-mon[97033]: pgmap v11177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:24:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:19 smithi055 ceph-mon[97033]: osdmap e778: 12 total, 11 up, 11 in 2022-09-08T03:24:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:19 smithi059 ceph-mon[97450]: pgmap v11177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:24:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:19 smithi059 ceph-mon[97450]: osdmap e778: 12 total, 11 up, 11 in 2022-09-08T03:24:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:19.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:19.466182+0000) 2022-09-08T03:24:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:19.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:19.434103+0000) 2022-09-08T03:24:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:20.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:20.466434+0000) 2022-09-08T03:24:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:20.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:20.434342+0000) 2022-09-08T03:24:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:21 smithi055 ceph-mon[97033]: pgmap v11179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:24:21.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:21 smithi059 ceph-mon[97450]: pgmap v11179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:24:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:21.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:21.466702+0000) 2022-09-08T03:24:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:21.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:21.434608+0000) 2022-09-08T03:24:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:22.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:22.466937+0000) 2022-09-08T03:24:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:22.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:22.434849+0000) 2022-09-08T03:24:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:23 smithi055 ceph-mon[97033]: pgmap v11180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:24:23.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:23 smithi059 ceph-mon[97450]: pgmap v11180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:24:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:23.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:23.467117+0000) 2022-09-08T03:24:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:23.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:23.435007+0000) 2022-09-08T03:24:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:24.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:24.467344+0000) 2022-09-08T03:24:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:24.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:24.435194+0000) 2022-09-08T03:24:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:24 smithi055 ceph-mon[97033]: pgmap v11181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:24:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:24 smithi059 ceph-mon[97450]: pgmap v11181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:24:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:25.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:25.467529+0000) 2022-09-08T03:24:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:25.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:25.435354+0000) 2022-09-08T03:24:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:26.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:26.435563+0000) 2022-09-08T03:24:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:26.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:26.467684+0000) 2022-09-08T03:24:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:27 smithi055 ceph-mon[97033]: pgmap v11182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:24:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:27 smithi059 ceph-mon[97450]: pgmap v11182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:24:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:27.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:27.467941+0000) 2022-09-08T03:24:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:27.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:27.435753+0000) 2022-09-08T03:24:28.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:24:28.065+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:24:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:28.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:28.468202+0000) 2022-09-08T03:24:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:28.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:28.435973+0000) 2022-09-08T03:24:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:29 smithi055 ceph-mon[97033]: pgmap v11183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:24:29.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:29 smithi059 ceph-mon[97450]: pgmap v11183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:24:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:29.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:29.468451+0000) 2022-09-08T03:24:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:29.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:29.436187+0000) 2022-09-08T03:24:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:30.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:30.468740+0000) 2022-09-08T03:24:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:30.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:30.436404+0000) 2022-09-08T03:24:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:31 smithi055 ceph-mon[97033]: pgmap v11184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T03:24:31.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:31 smithi059 ceph-mon[97450]: pgmap v11184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T03:24:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:31.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:31.468940+0000) 2022-09-08T03:24:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:31.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:31.436674+0000) 2022-09-08T03:24:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:32.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:32.469206+0000) 2022-09-08T03:24:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:32.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:32.436909+0000) 2022-09-08T03:24:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:33 smithi055 ceph-mon[97033]: pgmap v11185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:33 smithi059 ceph-mon[97450]: pgmap v11185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:33.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:33.469458+0000) 2022-09-08T03:24:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:33.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:33.437168+0000) 2022-09-08T03:24:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:34.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:34.469734+0000) 2022-09-08T03:24:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:34.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:34.437409+0000) 2022-09-08T03:24:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:34 smithi055 ceph-mon[97033]: pgmap v11186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:34 smithi059 ceph-mon[97450]: pgmap v11186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:35.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:35.469990+0000) 2022-09-08T03:24:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:35.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:35.437624+0000) 2022-09-08T03:24:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:36.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:36.470117+0000) 2022-09-08T03:24:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:36.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:36.437865+0000) 2022-09-08T03:24:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:37 smithi055 ceph-mon[97033]: pgmap v11187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:24:37.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:37 smithi059 ceph-mon[97450]: pgmap v11187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:24:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:37.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:37.470331+0000) 2022-09-08T03:24:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:37.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:37.438095+0000) 2022-09-08T03:24:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:38.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:38.470576+0000) 2022-09-08T03:24:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:38.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:38.438330+0000) 2022-09-08T03:24:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:39 smithi055 ceph-mon[97033]: pgmap v11188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:39 smithi059 ceph-mon[97450]: pgmap v11188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:39.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:39.470708+0000) 2022-09-08T03:24:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:39.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:39.438506+0000) 2022-09-08T03:24:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:40.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:40.470830+0000) 2022-09-08T03:24:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:40.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:40.438669+0000) 2022-09-08T03:24:41.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:41 smithi055 ceph-mon[97033]: pgmap v11189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:24:41.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:41 smithi059 ceph-mon[97450]: pgmap v11189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:24:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:41.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:41.471023+0000) 2022-09-08T03:24:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:41.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:41.438830+0000) 2022-09-08T03:24:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:42.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:42.471250+0000) 2022-09-08T03:24:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:42.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:42.439037+0000) 2022-09-08T03:24:43.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:24:43.070+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:24:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:43 smithi055 ceph-mon[97033]: pgmap v11190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:43 smithi059 ceph-mon[97450]: pgmap v11190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:43.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:43.471427+0000) 2022-09-08T03:24:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:43.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:43.439242+0000) 2022-09-08T03:24:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:44.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:44.471668+0000) 2022-09-08T03:24:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:44.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:44.439421+0000) 2022-09-08T03:24:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:44 smithi055 ceph-mon[97033]: pgmap v11191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:45.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:44 smithi059 ceph-mon[97450]: pgmap v11191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:45.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:45.439593+0000) 2022-09-08T03:24:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:45.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:45.471807+0000) 2022-09-08T03:24:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:46.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:46.472027+0000) 2022-09-08T03:24:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:46.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:46.439881+0000) 2022-09-08T03:24:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:47 smithi055 ceph-mon[97033]: pgmap v11192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:24:47.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:47 smithi059 ceph-mon[97450]: pgmap v11192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:24:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:47.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:47.472195+0000) 2022-09-08T03:24:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:47.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:47.440146+0000) 2022-09-08T03:24:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:48.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:48.472417+0000) 2022-09-08T03:24:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:48.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:48.440394+0000) 2022-09-08T03:24:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:49 smithi055 ceph-mon[97033]: pgmap v11193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:49.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:49 smithi059 ceph-mon[97450]: pgmap v11193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:49.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:49.472625+0000) 2022-09-08T03:24:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:49.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:49.440671+0000) 2022-09-08T03:24:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:50.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:50.440829+0000) 2022-09-08T03:24:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:50.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:50.472779+0000) 2022-09-08T03:24:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:51 smithi055 ceph-mon[97033]: pgmap v11194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:24:51.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:51 smithi059 ceph-mon[97450]: pgmap v11194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:24:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:51.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:51.472966+0000) 2022-09-08T03:24:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:51.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:51.441073+0000) 2022-09-08T03:24:52.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:52.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:52.473173+0000) 2022-09-08T03:24:52.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:52.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:52.441283+0000) 2022-09-08T03:24:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:53 smithi055 ceph-mon[97033]: pgmap v11195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:53 smithi059 ceph-mon[97450]: pgmap v11195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:53.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:53.473336+0000) 2022-09-08T03:24:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:53.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:53.441522+0000) 2022-09-08T03:24:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:54.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:54.473570+0000) 2022-09-08T03:24:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:54.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:54.441717+0000) 2022-09-08T03:24:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:54 smithi055 ceph-mon[97033]: pgmap v11196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:54 smithi059 ceph-mon[97450]: pgmap v11196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:55.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:55.441888+0000) 2022-09-08T03:24:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:55.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:55.473783+0000) 2022-09-08T03:24:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:56.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:56.442104+0000) 2022-09-08T03:24:56.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:56.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:56.474028+0000) 2022-09-08T03:24:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:57 smithi055 ceph-mon[97033]: pgmap v11197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:24:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:57 smithi059 ceph-mon[97450]: pgmap v11197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:24:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:57.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:57.442329+0000) 2022-09-08T03:24:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:57.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:57.474236+0000) 2022-09-08T03:24:58.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:24:58.072+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:24:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:58.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:58.474426+0000) 2022-09-08T03:24:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:58.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:58.442538+0000) 2022-09-08T03:24:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:24:59 smithi055 ceph-mon[97033]: pgmap v11198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:59.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:24:59 smithi059 ceph-mon[97450]: pgmap v11198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:24:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:24:59.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:59.474621+0000) 2022-09-08T03:24:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:24:59.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:24:59.442768+0000) 2022-09-08T03:25:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:00.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:00.474774+0000) 2022-09-08T03:25:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:00.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:00.442936+0000) 2022-09-08T03:25:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:01 smithi055 ceph-mon[97033]: pgmap v11199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:25:01.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:01 smithi059 ceph-mon[97450]: pgmap v11199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:25:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:01.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:01.474937+0000) 2022-09-08T03:25:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:01.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:01.443183+0000) 2022-09-08T03:25:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:02.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:02.475112+0000) 2022-09-08T03:25:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:02.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:02.443390+0000) 2022-09-08T03:25:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:03 smithi055 ceph-mon[97033]: pgmap v11200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:03.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:03 smithi059 ceph-mon[97450]: pgmap v11200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:03.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:03.475364+0000) 2022-09-08T03:25:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:03.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:03.443686+0000) 2022-09-08T03:25:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:04.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:04.475552+0000) 2022-09-08T03:25:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:04.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:04.443918+0000) 2022-09-08T03:25:04.905 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:04 smithi059 ceph-mon[97450]: pgmap v11201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:04.920 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:04 smithi055 ceph-mon[97033]: pgmap v11201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:25:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:25:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:25:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:25:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:25:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:25:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:25:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:25:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:25:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:25:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:05.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:05.475688+0000) 2022-09-08T03:25:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:05.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:05.444175+0000) 2022-09-08T03:25:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:06.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:06.475930+0000) 2022-09-08T03:25:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:06.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:06.444400+0000) 2022-09-08T03:25:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:25:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:06 smithi055 ceph-mon[97033]: pgmap v11202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:25:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:25:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:06 smithi059 ceph-mon[97450]: pgmap v11202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:25:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:07.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:07.476097+0000) 2022-09-08T03:25:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:07.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:07.444585+0000) 2022-09-08T03:25:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:08.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:08.476259+0000) 2022-09-08T03:25:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:08.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:08.444859+0000) 2022-09-08T03:25:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:09 smithi055 ceph-mon[97033]: pgmap v11203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:25:09.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:09 smithi059 ceph-mon[97450]: pgmap v11203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:25:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:09.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:09.476469+0000) 2022-09-08T03:25:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:09.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:09.445056+0000) 2022-09-08T03:25:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:10.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:10.445221+0000) 2022-09-08T03:25:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:10.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:10.476668+0000) 2022-09-08T03:25:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:10 smithi055 ceph-mon[97033]: osdmap e779: 12 total, 11 up, 11 in 2022-09-08T03:25:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:10 smithi055 ceph-mon[97033]: pgmap v11205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:25:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:10 smithi059 ceph-mon[97450]: osdmap e779: 12 total, 11 up, 11 in 2022-09-08T03:25:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:10 smithi059 ceph-mon[97450]: pgmap v11205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:25:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:11.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:11.476931+0000) 2022-09-08T03:25:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:11.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:11.445458+0000) 2022-09-08T03:25:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:12.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:12.477168+0000) 2022-09-08T03:25:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:12.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:12.445722+0000) 2022-09-08T03:25:13.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:25:13.080+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:25:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:13 smithi055 ceph-mon[97033]: pgmap v11206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:25:13.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:13 smithi059 ceph-mon[97450]: pgmap v11206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:25:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:13.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:13.477309+0000) 2022-09-08T03:25:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:13.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:13.445967+0000) 2022-09-08T03:25:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:14.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:14.446204+0000) 2022-09-08T03:25:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:14.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:14.477517+0000) 2022-09-08T03:25:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:14 smithi055 ceph-mon[97033]: osdmap e780: 12 total, 11 up, 11 in 2022-09-08T03:25:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:14 smithi055 ceph-mon[97033]: pgmap v11208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:25:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:14 smithi059 ceph-mon[97450]: osdmap e780: 12 total, 11 up, 11 in 2022-09-08T03:25:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:14 smithi059 ceph-mon[97450]: pgmap v11208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:25:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:15.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:15.446434+0000) 2022-09-08T03:25:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:15.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:15.477751+0000) 2022-09-08T03:25:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:16.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:16.477933+0000) 2022-09-08T03:25:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:16.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:16.446709+0000) 2022-09-08T03:25:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:17 smithi055 ceph-mon[97033]: pgmap v11209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:25:17.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:17 smithi059 ceph-mon[97450]: pgmap v11209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:25:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:17.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:17.446961+0000) 2022-09-08T03:25:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:17.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:17.478168+0000) 2022-09-08T03:25:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:18.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:18.478339+0000) 2022-09-08T03:25:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:18.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:18.447096+0000) 2022-09-08T03:25:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:19 smithi055 ceph-mon[97033]: pgmap v11210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 244 B/s wr, 7 op/s 2022-09-08T03:25:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:19 smithi059 ceph-mon[97450]: pgmap v11210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 244 B/s wr, 7 op/s 2022-09-08T03:25:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:19.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:19.447274+0000) 2022-09-08T03:25:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:19.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:19.478498+0000) 2022-09-08T03:25:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:20.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:20.447495+0000) 2022-09-08T03:25:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:20.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:20.478623+0000) 2022-09-08T03:25:21.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:21 smithi055 ceph-mon[97033]: pgmap v11211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:25:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:21 smithi059 ceph-mon[97450]: pgmap v11211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:25:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:21.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:21.447743+0000) 2022-09-08T03:25:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:21.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:21.478722+0000) 2022-09-08T03:25:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:22.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:22.478928+0000) 2022-09-08T03:25:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:22.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:22.447929+0000) 2022-09-08T03:25:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:23 smithi055 ceph-mon[97033]: pgmap v11212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:25:23.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:23 smithi059 ceph-mon[97450]: pgmap v11212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:25:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:23.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:23.479097+0000) 2022-09-08T03:25:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:23.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:23.448168+0000) 2022-09-08T03:25:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:24.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:24.448445+0000) 2022-09-08T03:25:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:24.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:24.479296+0000) 2022-09-08T03:25:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:24 smithi055 ceph-mon[97033]: pgmap v11213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T03:25:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:24 smithi059 ceph-mon[97450]: pgmap v11213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T03:25:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:25.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:25.479472+0000) 2022-09-08T03:25:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:25.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:25.448758+0000) 2022-09-08T03:25:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:26.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:26.479731+0000) 2022-09-08T03:25:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:26.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:26.448982+0000) 2022-09-08T03:25:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:27 smithi055 ceph-mon[97033]: pgmap v11214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:25:27.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:27 smithi059 ceph-mon[97450]: pgmap v11214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:25:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:27.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:27.479950+0000) 2022-09-08T03:25:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:27.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:27.449213+0000) 2022-09-08T03:25:28.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:25:28.092+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:25:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:28.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:28.449466+0000) 2022-09-08T03:25:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:28.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:28.480126+0000) 2022-09-08T03:25:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:29 smithi055 ceph-mon[97033]: pgmap v11215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:29.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:29 smithi059 ceph-mon[97450]: pgmap v11215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:29.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:29.449702+0000) 2022-09-08T03:25:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:29.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:29.480281+0000) 2022-09-08T03:25:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:30.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:30.480452+0000) 2022-09-08T03:25:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:30.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:30.449860+0000) 2022-09-08T03:25:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:31 smithi055 ceph-mon[97033]: pgmap v11216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:25:31.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:31 smithi059 ceph-mon[97450]: pgmap v11216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:25:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:31.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:31.480676+0000) 2022-09-08T03:25:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:31.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:31.450101+0000) 2022-09-08T03:25:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:32.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:32.480912+0000) 2022-09-08T03:25:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:32.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:32.450370+0000) 2022-09-08T03:25:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:33 smithi055 ceph-mon[97033]: pgmap v11217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:33.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:33 smithi059 ceph-mon[97450]: pgmap v11217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:33.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:33.481173+0000) 2022-09-08T03:25:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:33.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:33.450615+0000) 2022-09-08T03:25:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:34.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:34.481400+0000) 2022-09-08T03:25:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:34.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:34.450870+0000) 2022-09-08T03:25:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:34 smithi055 ceph-mon[97033]: pgmap v11218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:34 smithi059 ceph-mon[97450]: pgmap v11218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:35.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:35.481601+0000) 2022-09-08T03:25:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:35.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:35.451051+0000) 2022-09-08T03:25:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:36.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:36.481729+0000) 2022-09-08T03:25:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:36.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:36.451237+0000) 2022-09-08T03:25:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:37 smithi055 ceph-mon[97033]: pgmap v11219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:25:37.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:37 smithi059 ceph-mon[97450]: pgmap v11219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:25:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:37.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:37.481952+0000) 2022-09-08T03:25:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:37.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:37.451466+0000) 2022-09-08T03:25:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:38.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:38.482131+0000) 2022-09-08T03:25:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:38.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:38.451723+0000) 2022-09-08T03:25:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:39 smithi055 ceph-mon[97033]: pgmap v11220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:39 smithi059 ceph-mon[97450]: pgmap v11220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:39.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:39.451972+0000) 2022-09-08T03:25:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:39.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:39.482363+0000) 2022-09-08T03:25:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:40.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:40.482615+0000) 2022-09-08T03:25:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:40.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:40.452230+0000) 2022-09-08T03:25:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:41 smithi055 ceph-mon[97033]: pgmap v11221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:25:41.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:41 smithi059 ceph-mon[97450]: pgmap v11221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:25:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:41.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:41.482801+0000) 2022-09-08T03:25:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:41.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:41.452447+0000) 2022-09-08T03:25:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:42.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:42.482971+0000) 2022-09-08T03:25:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:42.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:42.452716+0000) 2022-09-08T03:25:43.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:25:43.107+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:25:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:43 smithi055 ceph-mon[97033]: pgmap v11222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:43.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:43 smithi059 ceph-mon[97450]: pgmap v11222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:43.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:43.483173+0000) 2022-09-08T03:25:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:43.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:43.452941+0000) 2022-09-08T03:25:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:44.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:44.483341+0000) 2022-09-08T03:25:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:44.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:44.453147+0000) 2022-09-08T03:25:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:44 smithi055 ceph-mon[97033]: pgmap v11223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:45.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:44 smithi059 ceph-mon[97450]: pgmap v11223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:45.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:45.483507+0000) 2022-09-08T03:25:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:45.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:45.453354+0000) 2022-09-08T03:25:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:46.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:46.483624+0000) 2022-09-08T03:25:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:46.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:46.453573+0000) 2022-09-08T03:25:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:47 smithi055 ceph-mon[97033]: pgmap v11224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:25:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:47 smithi059 ceph-mon[97450]: pgmap v11224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:25:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:47.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:47.483800+0000) 2022-09-08T03:25:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:47.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:47.453875+0000) 2022-09-08T03:25:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:48.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:48.484046+0000) 2022-09-08T03:25:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:48.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:48.454063+0000) 2022-09-08T03:25:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:49 smithi055 ceph-mon[97033]: pgmap v11225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:49.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:49 smithi059 ceph-mon[97450]: pgmap v11225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:49.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:49.484235+0000) 2022-09-08T03:25:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:49.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:49.454249+0000) 2022-09-08T03:25:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:50.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:50.484442+0000) 2022-09-08T03:25:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:50.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:50.454482+0000) 2022-09-08T03:25:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:51 smithi055 ceph-mon[97033]: pgmap v11226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:25:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:51 smithi059 ceph-mon[97450]: pgmap v11226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:25:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:51.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:51.484756+0000) 2022-09-08T03:25:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:51.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:51.454728+0000) 2022-09-08T03:25:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:52.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:52.485020+0000) 2022-09-08T03:25:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:52.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:52.454957+0000) 2022-09-08T03:25:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:53 smithi055 ceph-mon[97033]: pgmap v11227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:53 smithi059 ceph-mon[97450]: pgmap v11227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:53.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:53.485218+0000) 2022-09-08T03:25:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:53.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:53.455132+0000) 2022-09-08T03:25:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:54.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:54.455363+0000) 2022-09-08T03:25:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:54.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:54.485435+0000) 2022-09-08T03:25:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:54 smithi055 ceph-mon[97033]: pgmap v11228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:54 smithi059 ceph-mon[97450]: pgmap v11228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:55.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:55.485719+0000) 2022-09-08T03:25:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:55.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:55.455605+0000) 2022-09-08T03:25:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:56.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:56.485928+0000) 2022-09-08T03:25:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:56.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:56.455888+0000) 2022-09-08T03:25:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:57 smithi055 ceph-mon[97033]: pgmap v11229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:25:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:57 smithi059 ceph-mon[97450]: pgmap v11229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:25:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:57.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:57.486132+0000) 2022-09-08T03:25:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:57.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:57.456159+0000) 2022-09-08T03:25:58.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:25:58.117+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:25:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:58.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:58.456408+0000) 2022-09-08T03:25:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:58.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:58.486372+0000) 2022-09-08T03:25:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:25:59 smithi055 ceph-mon[97033]: pgmap v11230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:59.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:25:59 smithi059 ceph-mon[97450]: pgmap v11230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:25:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:25:59.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:59.486601+0000) 2022-09-08T03:25:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:25:59.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:25:59.456719+0000) 2022-09-08T03:26:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:00.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:00.486828+0000) 2022-09-08T03:26:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:00.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:00.456930+0000) 2022-09-08T03:26:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:01 smithi055 ceph-mon[97033]: pgmap v11231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:26:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:01 smithi059 ceph-mon[97450]: pgmap v11231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:26:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:01.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:01.486997+0000) 2022-09-08T03:26:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:01.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:01.457170+0000) 2022-09-08T03:26:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:02.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:02.487131+0000) 2022-09-08T03:26:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:02.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:02.457352+0000) 2022-09-08T03:26:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:03 smithi055 ceph-mon[97033]: pgmap v11232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:03 smithi059 ceph-mon[97450]: pgmap v11232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:03.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:03.487356+0000) 2022-09-08T03:26:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:03.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:03.457593+0000) 2022-09-08T03:26:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:04.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:04.487540+0000) 2022-09-08T03:26:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:04.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:04.457811+0000) 2022-09-08T03:26:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:04 smithi055 ceph-mon[97033]: pgmap v11233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:05.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:04 smithi059 ceph-mon[97450]: pgmap v11233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:05.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:26:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:26:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:26:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:26:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:05.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:05.487722+0000) 2022-09-08T03:26:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:05.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:05.458007+0000) 2022-09-08T03:26:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:26:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:26:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:26:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:26:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:06.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:06.487888+0000) 2022-09-08T03:26:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:06.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:06.458240+0000) 2022-09-08T03:26:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:26:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:06 smithi055 ceph-mon[97033]: pgmap v11234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:26:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:26:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:26:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:06 smithi059 ceph-mon[97450]: pgmap v11234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:26:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:26:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:07.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:07.488013+0000) 2022-09-08T03:26:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:07.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:07.458511+0000) 2022-09-08T03:26:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:08.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:08.488266+0000) 2022-09-08T03:26:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:08.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:08.458823+0000) 2022-09-08T03:26:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:09 smithi055 ceph-mon[97033]: pgmap v11235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:26:09.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:09 smithi059 ceph-mon[97450]: pgmap v11235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:26:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:09.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:09.488553+0000) 2022-09-08T03:26:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:09.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:09.459086+0000) 2022-09-08T03:26:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:10 smithi055 ceph-mon[97033]: osdmap e781: 12 total, 11 up, 11 in 2022-09-08T03:26:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:10 smithi059 ceph-mon[97450]: osdmap e781: 12 total, 11 up, 11 in 2022-09-08T03:26:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:10.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:10.488790+0000) 2022-09-08T03:26:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:10.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:10.459320+0000) 2022-09-08T03:26:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:11 smithi055 ceph-mon[97033]: pgmap v11237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:26:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:11 smithi059 ceph-mon[97450]: pgmap v11237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:26:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:11.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:11.489005+0000) 2022-09-08T03:26:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:11.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:11.459613+0000) 2022-09-08T03:26:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:12.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:12.489230+0000) 2022-09-08T03:26:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:12.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:12.459861+0000) 2022-09-08T03:26:13.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:26:13.128+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:26:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:13 smithi055 ceph-mon[97033]: pgmap v11238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:26:13.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:13 smithi059 ceph-mon[97450]: pgmap v11238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:26:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:13.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:13.489537+0000) 2022-09-08T03:26:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:13.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:13.460155+0000) 2022-09-08T03:26:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:14.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:14.489783+0000) 2022-09-08T03:26:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:14.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:14.460352+0000) 2022-09-08T03:26:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:14 smithi055 ceph-mon[97033]: pgmap v11239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:26:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:14 smithi059 ceph-mon[97450]: pgmap v11239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:26:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:15.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:15.489990+0000) 2022-09-08T03:26:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:15.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:15.460618+0000) 2022-09-08T03:26:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:16.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:16.460938+0000) 2022-09-08T03:26:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:16.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:16.490211+0000) 2022-09-08T03:26:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:17 smithi055 ceph-mon[97033]: pgmap v11240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:26:17.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:17 smithi059 ceph-mon[97450]: pgmap v11240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:26:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:17.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:17.461083+0000) 2022-09-08T03:26:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:17.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:17.490426+0000) 2022-09-08T03:26:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:18.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:18.490651+0000) 2022-09-08T03:26:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:18.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:18.461341+0000) 2022-09-08T03:26:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:19 smithi055 ceph-mon[97033]: pgmap v11241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:26:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:19 smithi055 ceph-mon[97033]: osdmap e782: 12 total, 11 up, 11 in 2022-09-08T03:26:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:19 smithi059 ceph-mon[97450]: pgmap v11241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:26:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:19 smithi059 ceph-mon[97450]: osdmap e782: 12 total, 11 up, 11 in 2022-09-08T03:26:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:19.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:19.461614+0000) 2022-09-08T03:26:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:19.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:19.490872+0000) 2022-09-08T03:26:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:20.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:20.461834+0000) 2022-09-08T03:26:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:20.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:20.491146+0000) 2022-09-08T03:26:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:21 smithi055 ceph-mon[97033]: pgmap v11243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:26:21.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:21 smithi059 ceph-mon[97450]: pgmap v11243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:26:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:21.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:21.491331+0000) 2022-09-08T03:26:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:21.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:21.462065+0000) 2022-09-08T03:26:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:22.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:22.491586+0000) 2022-09-08T03:26:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:22.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:22.462213+0000) 2022-09-08T03:26:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:23 smithi055 ceph-mon[97033]: pgmap v11244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:26:23.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:23 smithi059 ceph-mon[97450]: pgmap v11244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:26:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:23.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:23.491793+0000) 2022-09-08T03:26:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:23.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:23.462447+0000) 2022-09-08T03:26:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:24.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:24.491999+0000) 2022-09-08T03:26:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:24.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:24.462711+0000) 2022-09-08T03:26:24.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:24 smithi055 ceph-mon[97033]: pgmap v11245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:26:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:24 smithi059 ceph-mon[97450]: pgmap v11245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:26:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:25.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:25.492245+0000) 2022-09-08T03:26:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:25.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:25.462960+0000) 2022-09-08T03:26:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:26.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:26.492498+0000) 2022-09-08T03:26:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:26.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:26.463184+0000) 2022-09-08T03:26:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:27 smithi055 ceph-mon[97033]: pgmap v11246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:26:27.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:27 smithi059 ceph-mon[97450]: pgmap v11246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:26:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:27.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:27.492750+0000) 2022-09-08T03:26:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:27.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:27.463460+0000) 2022-09-08T03:26:28.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:26:28.143+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:26:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:28.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:28.492941+0000) 2022-09-08T03:26:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:28.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:28.463723+0000) 2022-09-08T03:26:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:29 smithi055 ceph-mon[97033]: pgmap v11247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:26:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:29 smithi059 ceph-mon[97450]: pgmap v11247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:26:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:29.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:29.493102+0000) 2022-09-08T03:26:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:29.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:29.463863+0000) 2022-09-08T03:26:30.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:30.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:30.493354+0000) 2022-09-08T03:26:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:30.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:30.464037+0000) 2022-09-08T03:26:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:31 smithi055 ceph-mon[97033]: pgmap v11248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T03:26:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:31 smithi059 ceph-mon[97450]: pgmap v11248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T03:26:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:31.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:31.493586+0000) 2022-09-08T03:26:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:31.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:31.464296+0000) 2022-09-08T03:26:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:32.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:32.464505+0000) 2022-09-08T03:26:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:32.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:32.493856+0000) 2022-09-08T03:26:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:33 smithi055 ceph-mon[97033]: pgmap v11249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:33 smithi059 ceph-mon[97450]: pgmap v11249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:33.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:33.464735+0000) 2022-09-08T03:26:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:33.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:33.494055+0000) 2022-09-08T03:26:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:34.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:34.494179+0000) 2022-09-08T03:26:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:34.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:34.464865+0000) 2022-09-08T03:26:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:34 smithi055 ceph-mon[97033]: pgmap v11250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:34 smithi059 ceph-mon[97450]: pgmap v11250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:35.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:35.494440+0000) 2022-09-08T03:26:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:35.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:35.465091+0000) 2022-09-08T03:26:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:36.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:36.494702+0000) 2022-09-08T03:26:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:36.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:36.465321+0000) 2022-09-08T03:26:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:37 smithi055 ceph-mon[97033]: pgmap v11251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:26:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:37 smithi059 ceph-mon[97450]: pgmap v11251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:26:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:37.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:37.494881+0000) 2022-09-08T03:26:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:37.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:37.465563+0000) 2022-09-08T03:26:38.790 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:38.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:38.495095+0000) 2022-09-08T03:26:38.791 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:38.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:38.465813+0000) 2022-09-08T03:26:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:39 smithi055 ceph-mon[97033]: pgmap v11252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:39 smithi059 ceph-mon[97450]: pgmap v11252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:39.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:39.495270+0000) 2022-09-08T03:26:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:39.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:39.465986+0000) 2022-09-08T03:26:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:40.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:40.466199+0000) 2022-09-08T03:26:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:40.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:40.495514+0000) 2022-09-08T03:26:41.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:41 smithi055 ceph-mon[97033]: pgmap v11253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:26:41.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:41 smithi059 ceph-mon[97450]: pgmap v11253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:26:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:41.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:41.495724+0000) 2022-09-08T03:26:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:41.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:41.466405+0000) 2022-09-08T03:26:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:42.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:42.495855+0000) 2022-09-08T03:26:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:42.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:42.466621+0000) 2022-09-08T03:26:43.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:26:43.147+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:26:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:43 smithi055 ceph-mon[97033]: pgmap v11254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:43.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:43 smithi059 ceph-mon[97450]: pgmap v11254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:43.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:43.496062+0000) 2022-09-08T03:26:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:43.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:43.466851+0000) 2022-09-08T03:26:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:44.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:44.496256+0000) 2022-09-08T03:26:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:44.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:44.467003+0000) 2022-09-08T03:26:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:44 smithi055 ceph-mon[97033]: pgmap v11255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:45.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:44 smithi059 ceph-mon[97450]: pgmap v11255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:45.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:45.496493+0000) 2022-09-08T03:26:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:45.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:45.467249+0000) 2022-09-08T03:26:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:46.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:46.496754+0000) 2022-09-08T03:26:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:46.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:46.467439+0000) 2022-09-08T03:26:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:47 smithi055 ceph-mon[97033]: pgmap v11256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:26:47.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:47 smithi059 ceph-mon[97450]: pgmap v11256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:26:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:47.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:47.496977+0000) 2022-09-08T03:26:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:47.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:47.467716+0000) 2022-09-08T03:26:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:48.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:48.497194+0000) 2022-09-08T03:26:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:48.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:48.467871+0000) 2022-09-08T03:26:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:49 smithi055 ceph-mon[97033]: pgmap v11257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:49.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:49 smithi059 ceph-mon[97450]: pgmap v11257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:49.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:49.497385+0000) 2022-09-08T03:26:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:49.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:49.468039+0000) 2022-09-08T03:26:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:50.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:50.468176+0000) 2022-09-08T03:26:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:50.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:50.497592+0000) 2022-09-08T03:26:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:51 smithi055 ceph-mon[97033]: pgmap v11258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:26:51.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:51 smithi059 ceph-mon[97450]: pgmap v11258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:26:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:51.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:51.497819+0000) 2022-09-08T03:26:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:51.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:51.468346+0000) 2022-09-08T03:26:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:52.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:52.498077+0000) 2022-09-08T03:26:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:52.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:52.468583+0000) 2022-09-08T03:26:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:53 smithi055 ceph-mon[97033]: pgmap v11259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:53.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:53 smithi059 ceph-mon[97450]: pgmap v11259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:53.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:53.498323+0000) 2022-09-08T03:26:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:53.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:53.468816+0000) 2022-09-08T03:26:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:54.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:54.498556+0000) 2022-09-08T03:26:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:54.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:54.468974+0000) 2022-09-08T03:26:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:54 smithi055 ceph-mon[97033]: pgmap v11260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:54 smithi059 ceph-mon[97450]: pgmap v11260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:55.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:55.498794+0000) 2022-09-08T03:26:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:55.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:55.469153+0000) 2022-09-08T03:26:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:56.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:56.499035+0000) 2022-09-08T03:26:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:56.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:56.469323+0000) 2022-09-08T03:26:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:57 smithi055 ceph-mon[97033]: pgmap v11261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:26:57.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:57 smithi059 ceph-mon[97450]: pgmap v11261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:26:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:57.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:57.499308+0000) 2022-09-08T03:26:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:57.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:57.469556+0000) 2022-09-08T03:26:58.469 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:26:58.163+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:26:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:58.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:58.499527+0000) 2022-09-08T03:26:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:58.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:58.469757+0000) 2022-09-08T03:26:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:26:59 smithi055 ceph-mon[97033]: pgmap v11262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:59.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:26:59 smithi059 ceph-mon[97450]: pgmap v11262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:26:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:26:59.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:59.469984+0000) 2022-09-08T03:26:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:26:59.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:26:59.499713+0000) 2022-09-08T03:27:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:00.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:00.499876+0000) 2022-09-08T03:27:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:00.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:00.470239+0000) 2022-09-08T03:27:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:01 smithi055 ceph-mon[97033]: pgmap v11263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:27:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:01 smithi059 ceph-mon[97450]: pgmap v11263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:27:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:01.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:01.500029+0000) 2022-09-08T03:27:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:01.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:01.470447+0000) 2022-09-08T03:27:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:02.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:02.500199+0000) 2022-09-08T03:27:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:02.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:02.470702+0000) 2022-09-08T03:27:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:03 smithi055 ceph-mon[97033]: pgmap v11264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:03 smithi059 ceph-mon[97450]: pgmap v11264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:03.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:03.500440+0000) 2022-09-08T03:27:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:03.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:03.470923+0000) 2022-09-08T03:27:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:04.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:04.500699+0000) 2022-09-08T03:27:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:04.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:04.471160+0000) 2022-09-08T03:27:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:04 smithi055 ceph-mon[97033]: pgmap v11265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:05.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:04 smithi059 ceph-mon[97450]: pgmap v11265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:05.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:05.500937+0000) 2022-09-08T03:27:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:05.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:05.471328+0000) 2022-09-08T03:27:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:27:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:27:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:06.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:06.501172+0000) 2022-09-08T03:27:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:06.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:06.471560+0000) 2022-09-08T03:27:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:27:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:06 smithi055 ceph-mon[97033]: pgmap v11266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:27:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:27:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:27:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:27:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:27:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:06 smithi059 ceph-mon[97450]: pgmap v11266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:27:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:27:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:27:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:27:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:07.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:07.501391+0000) 2022-09-08T03:27:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:07.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:07.471805+0000) 2022-09-08T03:27:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:27:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:27:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:08.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:08.501666+0000) 2022-09-08T03:27:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:08.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:08.472059+0000) 2022-09-08T03:27:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:08 smithi055 ceph-mon[97033]: pgmap v11267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:27:09.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:08 smithi059 ceph-mon[97450]: pgmap v11267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:27:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:09.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:09.501830+0000) 2022-09-08T03:27:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:09.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:09.472267+0000) 2022-09-08T03:27:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:09 smithi055 ceph-mon[97033]: osdmap e783: 12 total, 11 up, 11 in 2022-09-08T03:27:10.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:09 smithi059 ceph-mon[97450]: osdmap e783: 12 total, 11 up, 11 in 2022-09-08T03:27:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:10.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:10.501978+0000) 2022-09-08T03:27:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:10.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:10.472404+0000) 2022-09-08T03:27:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:10 smithi055 ceph-mon[97033]: pgmap v11269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:27:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:10 smithi059 ceph-mon[97450]: pgmap v11269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:27:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:11.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:11.502146+0000) 2022-09-08T03:27:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:11.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:11.472614+0000) 2022-09-08T03:27:12.913 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:12.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:12.502370+0000) 2022-09-08T03:27:12.914 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:12.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:12.472836+0000) 2022-09-08T03:27:13.473 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:27:13.170+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:27:13.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:13 smithi055 ceph-mon[97033]: pgmap v11270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:27:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:13 smithi059 ceph-mon[97450]: pgmap v11270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:27:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:13.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:13.502612+0000) 2022-09-08T03:27:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:13.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:13.473073+0000) 2022-09-08T03:27:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:14.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:14.502874+0000) 2022-09-08T03:27:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:14.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:14.473291+0000) 2022-09-08T03:27:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:14 smithi055 ceph-mon[97033]: osdmap e784: 12 total, 11 up, 11 in 2022-09-08T03:27:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:14 smithi055 ceph-mon[97033]: pgmap v11272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:27:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:14 smithi059 ceph-mon[97450]: osdmap e784: 12 total, 11 up, 11 in 2022-09-08T03:27:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:14 smithi059 ceph-mon[97450]: pgmap v11272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:27:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:15.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:15.503013+0000) 2022-09-08T03:27:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:15.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:15.473565+0000) 2022-09-08T03:27:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:16.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:16.473741+0000) 2022-09-08T03:27:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:16.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:16.503252+0000) 2022-09-08T03:27:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:17 smithi055 ceph-mon[97033]: pgmap v11273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:27:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:17 smithi059 ceph-mon[97450]: pgmap v11273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:27:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:17.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:17.473929+0000) 2022-09-08T03:27:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:17.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:17.503496+0000) 2022-09-08T03:27:18.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:18.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:18.474177+0000) 2022-09-08T03:27:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:18.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:18.503736+0000) 2022-09-08T03:27:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:19 smithi055 ceph-mon[97033]: pgmap v11274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 218 B/s wr, 6 op/s 2022-09-08T03:27:19.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:19 smithi059 ceph-mon[97450]: pgmap v11274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 218 B/s wr, 6 op/s 2022-09-08T03:27:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:19.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:19.474419+0000) 2022-09-08T03:27:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:19.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:19.503973+0000) 2022-09-08T03:27:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:20.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:20.504197+0000) 2022-09-08T03:27:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:20.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:20.474710+0000) 2022-09-08T03:27:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:21 smithi055 ceph-mon[97033]: pgmap v11275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:27:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:21 smithi059 ceph-mon[97450]: pgmap v11275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:27:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:21.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:21.504314+0000) 2022-09-08T03:27:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:21.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:21.474898+0000) 2022-09-08T03:27:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:22.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:22.504540+0000) 2022-09-08T03:27:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:22.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:22.475126+0000) 2022-09-08T03:27:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:23 smithi055 ceph-mon[97033]: pgmap v11276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:27:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:23 smithi059 ceph-mon[97450]: pgmap v11276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:27:23.913 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:23.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:23.504775+0000) 2022-09-08T03:27:23.914 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:23.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:23.475429+0000) 2022-09-08T03:27:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:24.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:24.505017+0000) 2022-09-08T03:27:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:24.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:24.475665+0000) 2022-09-08T03:27:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:24 smithi055 ceph-mon[97033]: pgmap v11277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T03:27:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:24 smithi059 ceph-mon[97450]: pgmap v11277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T03:27:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:25.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:25.505228+0000) 2022-09-08T03:27:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:25.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:25.475942+0000) 2022-09-08T03:27:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:26.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:26.505417+0000) 2022-09-08T03:27:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:26.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:26.476186+0000) 2022-09-08T03:27:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:27 smithi055 ceph-mon[97033]: pgmap v11278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:27:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:27 smithi059 ceph-mon[97450]: pgmap v11278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:27:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:27.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:27.505629+0000) 2022-09-08T03:27:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:27.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:27.476472+0000) 2022-09-08T03:27:28.477 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:27:28.184+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:27:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:28.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:28.505907+0000) 2022-09-08T03:27:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:28.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:28.476749+0000) 2022-09-08T03:27:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:29 smithi055 ceph-mon[97033]: pgmap v11279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:29 smithi059 ceph-mon[97450]: pgmap v11279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:29.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:29.506133+0000) 2022-09-08T03:27:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:29.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:29.476994+0000) 2022-09-08T03:27:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:30.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:30.506381+0000) 2022-09-08T03:27:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:30.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:30.477165+0000) 2022-09-08T03:27:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:31 smithi055 ceph-mon[97033]: pgmap v11280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:27:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:31 smithi059 ceph-mon[97450]: pgmap v11280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:27:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:31.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:31.506554+0000) 2022-09-08T03:27:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:31.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:31.477368+0000) 2022-09-08T03:27:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:32.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:32.506795+0000) 2022-09-08T03:27:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:32.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:32.477600+0000) 2022-09-08T03:27:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:33 smithi055 ceph-mon[97033]: pgmap v11281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:33 smithi059 ceph-mon[97450]: pgmap v11281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:33.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:33.507009+0000) 2022-09-08T03:27:33.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:33.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:33.477805+0000) 2022-09-08T03:27:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:34.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:34.507205+0000) 2022-09-08T03:27:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:34.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:34.477925+0000) 2022-09-08T03:27:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:34 smithi055 ceph-mon[97033]: pgmap v11282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:34 smithi059 ceph-mon[97450]: pgmap v11282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:35.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:35.507409+0000) 2022-09-08T03:27:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:35.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:35.478163+0000) 2022-09-08T03:27:36.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:36.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:36.507736+0000) 2022-09-08T03:27:36.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:36.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:36.478381+0000) 2022-09-08T03:27:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:37 smithi055 ceph-mon[97033]: pgmap v11283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:27:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:37 smithi059 ceph-mon[97450]: pgmap v11283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:27:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:37.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:37.507934+0000) 2022-09-08T03:27:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:37.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:37.478617+0000) 2022-09-08T03:27:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:38.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:38.508137+0000) 2022-09-08T03:27:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:38.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:38.478829+0000) 2022-09-08T03:27:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:39 smithi055 ceph-mon[97033]: pgmap v11284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:39 smithi059 ceph-mon[97450]: pgmap v11284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:39.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:39.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:39.508276+0000) 2022-09-08T03:27:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:39.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:39.479069+0000) 2022-09-08T03:27:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:40.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:40.508453+0000) 2022-09-08T03:27:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:40.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:40.479319+0000) 2022-09-08T03:27:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:41 smithi055 ceph-mon[97033]: pgmap v11285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:27:41.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:41 smithi059 ceph-mon[97450]: pgmap v11285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:27:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:41.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:41.508602+0000) 2022-09-08T03:27:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:41.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:41.479558+0000) 2022-09-08T03:27:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:42.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:42.508818+0000) 2022-09-08T03:27:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:42.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:42.479782+0000) 2022-09-08T03:27:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:43 smithi055 ceph-mon[97033]: pgmap v11286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:43.479 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:27:43.196+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:27:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:43 smithi059 ceph-mon[97450]: pgmap v11286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:43.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:43.508965+0000) 2022-09-08T03:27:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:43.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:43.480016+0000) 2022-09-08T03:27:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:44.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:44.509129+0000) 2022-09-08T03:27:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:44.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:44.480185+0000) 2022-09-08T03:27:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:44 smithi055 ceph-mon[97033]: pgmap v11287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:44 smithi059 ceph-mon[97450]: pgmap v11287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:45.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:45.509325+0000) 2022-09-08T03:27:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:45.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:45.480313+0000) 2022-09-08T03:27:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:46.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:46.509523+0000) 2022-09-08T03:27:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:46.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:46.480498+0000) 2022-09-08T03:27:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:47 smithi055 ceph-mon[97033]: pgmap v11288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:27:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:47 smithi059 ceph-mon[97450]: pgmap v11288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:27:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:47.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:47.509705+0000) 2022-09-08T03:27:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:47.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:47.480758+0000) 2022-09-08T03:27:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:48.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:48.509918+0000) 2022-09-08T03:27:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:48.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:48.481007+0000) 2022-09-08T03:27:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:49 smithi055 ceph-mon[97033]: pgmap v11289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:49.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:49 smithi059 ceph-mon[97450]: pgmap v11289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:49.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:49.510055+0000) 2022-09-08T03:27:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:49.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:49.481218+0000) 2022-09-08T03:27:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:50.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:50.510222+0000) 2022-09-08T03:27:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:50.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:50.481433+0000) 2022-09-08T03:27:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:51 smithi055 ceph-mon[97033]: pgmap v11290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:27:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:51 smithi059 ceph-mon[97450]: pgmap v11290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:27:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:51.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:51.510375+0000) 2022-09-08T03:27:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:51.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:51.481754+0000) 2022-09-08T03:27:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:52.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:52.510576+0000) 2022-09-08T03:27:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:52.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:52.481930+0000) 2022-09-08T03:27:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:53 smithi055 ceph-mon[97033]: pgmap v11291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:53.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:53 smithi059 ceph-mon[97450]: pgmap v11291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:53.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:53.510746+0000) 2022-09-08T03:27:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:53.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:53.482216+0000) 2022-09-08T03:27:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:54.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:54.482463+0000) 2022-09-08T03:27:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:54.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:54.510977+0000) 2022-09-08T03:27:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:54 smithi055 ceph-mon[97033]: pgmap v11292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:54 smithi059 ceph-mon[97450]: pgmap v11292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:55.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:55.511196+0000) 2022-09-08T03:27:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:55.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:55.482755+0000) 2022-09-08T03:27:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:56.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:56.511389+0000) 2022-09-08T03:27:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:56.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:56.482982+0000) 2022-09-08T03:27:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:57 smithi055 ceph-mon[97033]: pgmap v11293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:27:57.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:57 smithi059 ceph-mon[97450]: pgmap v11293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:27:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:57.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:57.483229+0000) 2022-09-08T03:27:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:57.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:57.511626+0000) 2022-09-08T03:27:58.484 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:27:58.212+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:27:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:58.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:58.483488+0000) 2022-09-08T03:27:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:58.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:58.511956+0000) 2022-09-08T03:27:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:27:59 smithi055 ceph-mon[97033]: pgmap v11294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:27:59 smithi059 ceph-mon[97450]: pgmap v11294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:27:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:27:59.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:59.483731+0000) 2022-09-08T03:27:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:27:59.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:27:59.512216+0000) 2022-09-08T03:28:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:00.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:00.483864+0000) 2022-09-08T03:28:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:00.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:00.512383+0000) 2022-09-08T03:28:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:01 smithi055 ceph-mon[97033]: pgmap v11295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:28:01.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:01 smithi059 ceph-mon[97450]: pgmap v11295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:28:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:01.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:01.484125+0000) 2022-09-08T03:28:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:01.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:01.512589+0000) 2022-09-08T03:28:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:02.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:02.512758+0000) 2022-09-08T03:28:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:02.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:02.484412+0000) 2022-09-08T03:28:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:03 smithi055 ceph-mon[97033]: pgmap v11296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:03.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:03 smithi059 ceph-mon[97450]: pgmap v11296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:03.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:03.484622+0000) 2022-09-08T03:28:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:03.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:03.513003+0000) 2022-09-08T03:28:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:04.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:04.484833+0000) 2022-09-08T03:28:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:04.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:04.513284+0000) 2022-09-08T03:28:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:04 smithi055 ceph-mon[97033]: pgmap v11297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:04 smithi059 ceph-mon[97450]: pgmap v11297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:05.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:05.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:05.513512+0000) 2022-09-08T03:28:05.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:05.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:05.485122+0000) 2022-09-08T03:28:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:28:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:28:06.804 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:06.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:06.513686+0000) 2022-09-08T03:28:06.805 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:06.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:06.485301+0000) 2022-09-08T03:28:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:28:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:06 smithi055 ceph-mon[97033]: pgmap v11298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:28:07.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:28:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:06 smithi059 ceph-mon[97450]: pgmap v11298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:28:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:07.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:07.513923+0000) 2022-09-08T03:28:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:07.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:07.485555+0000) 2022-09-08T03:28:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:28:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:28:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:28:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:28:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:28:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:28:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:08.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:08.514116+0000) 2022-09-08T03:28:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:08.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:08.485828+0000) 2022-09-08T03:28:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:08 smithi055 ceph-mon[97033]: pgmap v11299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:28:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:08 smithi059 ceph-mon[97450]: pgmap v11299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:28:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:09.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:09.486127+0000) 2022-09-08T03:28:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:09.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:09.514349+0000) 2022-09-08T03:28:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:09 smithi055 ceph-mon[97033]: osdmap e785: 12 total, 11 up, 11 in 2022-09-08T03:28:10.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:09 smithi059 ceph-mon[97450]: osdmap e785: 12 total, 11 up, 11 in 2022-09-08T03:28:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:10.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:10.514577+0000) 2022-09-08T03:28:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:10.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:10.486401+0000) 2022-09-08T03:28:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:10 smithi055 ceph-mon[97033]: pgmap v11301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:28:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:10 smithi059 ceph-mon[97450]: pgmap v11301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:28:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:11.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:11.514815+0000) 2022-09-08T03:28:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:11.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:11.486619+0000) 2022-09-08T03:28:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:12.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:12.514999+0000) 2022-09-08T03:28:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:12.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:12.486800+0000) 2022-09-08T03:28:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:13 smithi055 ceph-mon[97033]: pgmap v11302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:28:13.486 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:28:13.227+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:28:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:13 smithi059 ceph-mon[97450]: pgmap v11302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:28:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:13.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:13.515195+0000) 2022-09-08T03:28:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:13.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:13.487018+0000) 2022-09-08T03:28:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:14.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:14.515428+0000) 2022-09-08T03:28:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:14.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:14.487179+0000) 2022-09-08T03:28:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:14 smithi055 ceph-mon[97033]: osdmap e786: 12 total, 11 up, 11 in 2022-09-08T03:28:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:14 smithi055 ceph-mon[97033]: pgmap v11304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:28:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:14 smithi059 ceph-mon[97450]: osdmap e786: 12 total, 11 up, 11 in 2022-09-08T03:28:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:14 smithi059 ceph-mon[97450]: pgmap v11304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:28:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:15.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:15.515649+0000) 2022-09-08T03:28:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:15.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:15.487411+0000) 2022-09-08T03:28:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:16.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:16.515888+0000) 2022-09-08T03:28:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:16.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:16.487555+0000) 2022-09-08T03:28:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:17 smithi055 ceph-mon[97033]: pgmap v11305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:28:17.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:17 smithi059 ceph-mon[97450]: pgmap v11305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:28:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:17.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:17.516153+0000) 2022-09-08T03:28:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:17.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:17.487808+0000) 2022-09-08T03:28:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:18.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:18.516319+0000) 2022-09-08T03:28:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:18.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:18.488024+0000) 2022-09-08T03:28:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:19 smithi055 ceph-mon[97033]: pgmap v11306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 218 B/s wr, 6 op/s 2022-09-08T03:28:19.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:19 smithi059 ceph-mon[97450]: pgmap v11306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 218 B/s wr, 6 op/s 2022-09-08T03:28:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:19.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:19.516498+0000) 2022-09-08T03:28:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:19.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:19.488186+0000) 2022-09-08T03:28:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:20.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:20.516752+0000) 2022-09-08T03:28:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:20.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:20.488434+0000) 2022-09-08T03:28:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:21 smithi055 ceph-mon[97033]: pgmap v11307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:28:21.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:21 smithi059 ceph-mon[97450]: pgmap v11307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:28:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:21.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:21.488733+0000) 2022-09-08T03:28:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:21.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:21.516949+0000) 2022-09-08T03:28:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:22.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:22.488974+0000) 2022-09-08T03:28:22.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:22.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:22.517228+0000) 2022-09-08T03:28:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:23 smithi055 ceph-mon[97033]: pgmap v11308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:28:23.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:23 smithi059 ceph-mon[97450]: pgmap v11308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:28:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:23.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:23.517500+0000) 2022-09-08T03:28:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:23.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:23.489265+0000) 2022-09-08T03:28:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:24.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:24.517777+0000) 2022-09-08T03:28:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:24.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:24.489438+0000) 2022-09-08T03:28:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:24 smithi055 ceph-mon[97033]: pgmap v11309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T03:28:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:24 smithi059 ceph-mon[97450]: pgmap v11309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T03:28:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:25.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:25.518032+0000) 2022-09-08T03:28:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:25.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:25.489694+0000) 2022-09-08T03:28:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:26.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:26.518246+0000) 2022-09-08T03:28:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:26.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:26.489852+0000) 2022-09-08T03:28:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:27 smithi055 ceph-mon[97033]: pgmap v11310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:28:27.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:27 smithi059 ceph-mon[97450]: pgmap v11310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:28:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:27.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:27.518467+0000) 2022-09-08T03:28:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:27.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:27.490100+0000) 2022-09-08T03:28:28.518 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:28:28.243+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:28:28.519 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:28.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:28.490378+0000) 2022-09-08T03:28:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:28.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:28.518752+0000) 2022-09-08T03:28:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:29 smithi055 ceph-mon[97033]: pgmap v11311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:29 smithi059 ceph-mon[97450]: pgmap v11311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:29.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:29.518981+0000) 2022-09-08T03:28:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:29.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:29.490706+0000) 2022-09-08T03:28:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:30.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:30.519236+0000) 2022-09-08T03:28:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:30.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:30.490955+0000) 2022-09-08T03:28:31.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:31 smithi055 ceph-mon[97033]: pgmap v11312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:28:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:31 smithi059 ceph-mon[97450]: pgmap v11312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:28:31.860 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:31.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:31.519456+0000) 2022-09-08T03:28:31.861 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:31.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:31.491206+0000) 2022-09-08T03:28:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:32.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:32.519759+0000) 2022-09-08T03:28:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:32.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:32.491453+0000) 2022-09-08T03:28:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:33 smithi055 ceph-mon[97033]: pgmap v11313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:33 smithi059 ceph-mon[97450]: pgmap v11313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:33.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:33.520032+0000) 2022-09-08T03:28:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:33.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:33.491768+0000) 2022-09-08T03:28:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:34.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:34.520231+0000) 2022-09-08T03:28:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:34.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:34.491981+0000) 2022-09-08T03:28:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:34 smithi055 ceph-mon[97033]: pgmap v11314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:34 smithi059 ceph-mon[97450]: pgmap v11314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:35.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:35.520461+0000) 2022-09-08T03:28:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:35.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:35.492233+0000) 2022-09-08T03:28:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:36.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:36.520714+0000) 2022-09-08T03:28:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:36.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:36.492405+0000) 2022-09-08T03:28:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:37 smithi055 ceph-mon[97033]: pgmap v11315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:28:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:28:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:37 smithi059 ceph-mon[97450]: pgmap v11315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:28:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:28:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:37.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:37.520862+0000) 2022-09-08T03:28:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:37.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:37.492672+0000) 2022-09-08T03:28:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:38 smithi055 ceph-mon[97033]: pgmap v11316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:28:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:38 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T03:28:38.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:38 smithi059 ceph-mon[97450]: pgmap v11316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:28:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:38 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T03:28:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:38.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:38.521075+0000) 2022-09-08T03:28:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:38.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:38.492929+0000) 2022-09-08T03:28:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:39.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:39.521266+0000) 2022-09-08T03:28:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:39.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:39.493148+0000) 2022-09-08T03:28:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:39 smithi055 ceph-mon[97033]: pgmap v11317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:28:40.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:39 smithi059 ceph-mon[97450]: pgmap v11317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:28:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:40.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:40.521425+0000) 2022-09-08T03:28:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:40.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:40.493378+0000) 2022-09-08T03:28:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:41.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:41.521672+0000) 2022-09-08T03:28:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:41.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:41.493668+0000) 2022-09-08T03:28:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:41 smithi055 ceph-mon[97033]: pgmap v11318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:28:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:41 smithi059 ceph-mon[97450]: pgmap v11318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:28:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:42.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:42.521924+0000) 2022-09-08T03:28:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:42.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:42.493944+0000) 2022-09-08T03:28:43.522 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:28:43.246+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:28:43.522 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:43.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:43.494203+0000) 2022-09-08T03:28:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:43.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:43.522197+0000) 2022-09-08T03:28:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:43 smithi055 ceph-mon[97033]: pgmap v11319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:28:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:43 smithi059 ceph-mon[97450]: pgmap v11319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:28:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:44.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:44.522395+0000) 2022-09-08T03:28:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:44.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:44.494490+0000) 2022-09-08T03:28:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:45.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:45.522622+0000) 2022-09-08T03:28:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:45.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:45.494770+0000) 2022-09-08T03:28:46.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:45 smithi059 ceph-mon[97450]: pgmap v11320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T03:28:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:45 smithi055 ceph-mon[97033]: pgmap v11320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T03:28:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:46.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:46.522803+0000) 2022-09-08T03:28:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:46.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:46.495044+0000) 2022-09-08T03:28:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:47.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:47.522941+0000) 2022-09-08T03:28:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:47.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:47.495333+0000) 2022-09-08T03:28:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:47 smithi059 ceph-mon[97450]: pgmap v11321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:28:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:47 smithi055 ceph-mon[97033]: pgmap v11321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:28:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:48.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:48.523169+0000) 2022-09-08T03:28:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:48.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:48.495590+0000) 2022-09-08T03:28:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:49.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:49.523408+0000) 2022-09-08T03:28:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:49.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:49.495861+0000) 2022-09-08T03:28:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:49 smithi055 ceph-mon[97033]: pgmap v11322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:49 smithi059 ceph-mon[97450]: pgmap v11322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:50.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:50.523662+0000) 2022-09-08T03:28:50.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:50.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:50.496092+0000) 2022-09-08T03:28:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:51.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:51.523841+0000) 2022-09-08T03:28:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:51.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:51.496351+0000) 2022-09-08T03:28:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:51 smithi055 ceph-mon[97033]: pgmap v11323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:28:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:51 smithi059 ceph-mon[97450]: pgmap v11323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:28:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:52.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:52.524089+0000) 2022-09-08T03:28:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:52.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:52.496681+0000) 2022-09-08T03:28:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:53.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:53.524361+0000) 2022-09-08T03:28:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:53.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:53.496832+0000) 2022-09-08T03:28:54.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:53 smithi059 ceph-mon[97450]: pgmap v11324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:53 smithi055 ceph-mon[97033]: pgmap v11324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:54.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:54.524574+0000) 2022-09-08T03:28:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:54.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:54.497021+0000) 2022-09-08T03:28:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:55.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:55.524751+0000) 2022-09-08T03:28:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:55.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:55.497283+0000) 2022-09-08T03:28:56.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:55 smithi059 ceph-mon[97450]: pgmap v11325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:28:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:55 smithi055 ceph-mon[97033]: pgmap v11325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:28:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:56.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:56.524875+0000) 2022-09-08T03:28:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:56.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:56.497555+0000) 2022-09-08T03:28:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:57.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:57.525048+0000) 2022-09-08T03:28:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:57.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:57.497855+0000) 2022-09-08T03:28:58.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:57 smithi059 ceph-mon[97450]: pgmap v11326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:57 smithi055 ceph-mon[97033]: pgmap v11326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:28:58.525 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:28:58.259+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:28:58.526 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:58.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:58.498073+0000) 2022-09-08T03:28:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:58.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:58.525268+0000) 2022-09-08T03:28:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:28:59.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:59.498294+0000) 2022-09-08T03:28:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:28:59.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:28:59.525556+0000) 2022-09-08T03:28:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:28:59 smithi055 ceph-mon[97033]: pgmap v11327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:00.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:28:59 smithi059 ceph-mon[97450]: pgmap v11327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:00.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:00.525820+0000) 2022-09-08T03:29:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:00.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:00.498477+0000) 2022-09-08T03:29:01.499 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:29:01.377+0000 7f8354969700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.500 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:29:01.377+0000 7f8354969700 -1 mon.a@0(leader) e3 *** Got Signal Hangup *** 2022-09-08T03:29:01.501 INFO:journalctl@ceph.osd.3.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-3[108196]: 2022-09-08T03:29:01.377+0000 7fd275230700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.502 INFO:journalctl@ceph.osd.9.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-9[120046]: 2022-09-08T03:29:01.377+0000 7facfe449700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.503 INFO:journalctl@ceph.mds.d.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-d[124487]: 2022-09-08T03:29:01.377+0000 7fca2972e700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.503 INFO:journalctl@ceph.mds.g.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-g[125105]: 2022-09-08T03:29:01.377+0000 7f188ebed700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.504 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:01.377+0000 7fdbca3d5700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.506 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:01.377+0000 7f7547809700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.507 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-08T03:29:01.377+0000 7f3fe2232700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.507 INFO:journalctl@ceph.osd.6.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-6[114133]: 2022-09-08T03:29:01.377+0000 7f2f79e5f700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.508 INFO:journalctl@ceph.mds.a.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-a[125500]: 2022-09-08T03:29:01.377+0000 7fdbbf180700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.901 INFO:journalctl@ceph.osd.3.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-3[108196]: 2022-09-08T03:29:01.518+0000 7fd275230700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.901 INFO:journalctl@ceph.osd.9.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-9[120046]: 2022-09-08T03:29:01.518+0000 7facfe449700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.902 INFO:journalctl@ceph.mds.d.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-d[124487]: 2022-09-08T03:29:01.518+0000 7fca2972e700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.903 INFO:journalctl@ceph.mds.g.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-g[125105]: 2022-09-08T03:29:01.519+0000 7f188ebed700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:29:01.518+0000 7f8354969700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:29:01.518+0000 7f8354969700 -1 mon.a@0(leader) e3 *** Got Signal Hangup *** 2022-09-08T03:29:01.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:01.518+0000 7f7547809700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:01.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:01.525990+0000) 2022-09-08T03:29:01.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:01.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:01.498720+0000) 2022-09-08T03:29:01.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:01.518+0000 7fdbca3d5700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.908 INFO:journalctl@ceph.mgr.x.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-x[90336]: 2022-09-08T03:29:01.518+0000 7f3fe2232700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.909 INFO:journalctl@ceph.osd.6.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-6[114133]: 2022-09-08T03:29:01.519+0000 7f2f79e5f700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:01.910 INFO:journalctl@ceph.mds.a.smithi047.stdout:Sep 08 03:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-a[125500]: 2022-09-08T03:29:01.518+0000 7fdbbf180700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:29:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:01 smithi055 ceph-mon[97033]: pgmap v11328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:29:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:01 smithi059 ceph-mon[97450]: pgmap v11328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:29:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:02.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:02.526142+0000) 2022-09-08T03:29:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:02.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:02.498813+0000) 2022-09-08T03:29:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:03.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:03.526282+0000) 2022-09-08T03:29:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:03.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:03.498937+0000) 2022-09-08T03:29:04.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:03 smithi059 ceph-mon[97450]: pgmap v11329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:03 smithi055 ceph-mon[97033]: pgmap v11329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:04.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:04.526422+0000) 2022-09-08T03:29:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:04.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:04.499083+0000) 2022-09-08T03:29:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:05.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:05.526539+0000) 2022-09-08T03:29:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:05.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:05.499224+0000) 2022-09-08T03:29:06.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:05 smithi059 ceph-mon[97450]: pgmap v11330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:29:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:29:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:29:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:05 smithi055 ceph-mon[97033]: pgmap v11330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:29:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:29:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:29:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:06.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:06.526680+0000) 2022-09-08T03:29:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:06.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:06.499361+0000) 2022-09-08T03:29:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:07.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:07.499514+0000) 2022-09-08T03:29:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:07.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:07.526812+0000) 2022-09-08T03:29:08.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:07 smithi059 ceph-mon[97450]: pgmap v11331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:29:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:07 smithi055 ceph-mon[97033]: pgmap v11331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:29:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:08.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:08.526938+0000) 2022-09-08T03:29:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:08.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:08.499692+0000) 2022-09-08T03:29:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:09.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:09.527072+0000) 2022-09-08T03:29:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:09.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:09.499823+0000) 2022-09-08T03:29:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:09 smithi059 ceph-mon[97450]: pgmap v11332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:29:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:09 smithi059 ceph-mon[97450]: osdmap e787: 12 total, 11 up, 11 in 2022-09-08T03:29:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:09 smithi055 ceph-mon[97033]: pgmap v11332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:29:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:09 smithi055 ceph-mon[97033]: osdmap e787: 12 total, 11 up, 11 in 2022-09-08T03:29:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:10.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:10.527205+0000) 2022-09-08T03:29:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:10.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:10.499950+0000) 2022-09-08T03:29:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:11.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:11.527331+0000) 2022-09-08T03:29:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:11.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:11.500093+0000) 2022-09-08T03:29:12.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:11 smithi059 ceph-mon[97450]: pgmap v11334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:29:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:11 smithi055 ceph-mon[97033]: pgmap v11334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:29:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:12.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:12.527462+0000) 2022-09-08T03:29:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:12.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:12.500246+0000) 2022-09-08T03:29:13.527 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:29:13.260+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:29:13.528 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:13.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:13.500476+0000) 2022-09-08T03:29:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:13.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:13.527585+0000) 2022-09-08T03:29:14.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:13 smithi059 ceph-mon[97450]: pgmap v11335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:29:14.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:13 smithi059 ceph-mon[97450]: osdmap e788: 12 total, 11 up, 11 in 2022-09-08T03:29:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:13 smithi055 ceph-mon[97033]: pgmap v11335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:29:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:13 smithi055 ceph-mon[97033]: osdmap e788: 12 total, 11 up, 11 in 2022-09-08T03:29:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:14.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:14.527772+0000) 2022-09-08T03:29:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:14.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:14.500758+0000) 2022-09-08T03:29:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:15.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:15.527887+0000) 2022-09-08T03:29:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:15.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:15.500937+0000) 2022-09-08T03:29:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:16 smithi059 ceph-mon[97450]: pgmap v11337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:29:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:16 smithi055 ceph-mon[97033]: pgmap v11337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:29:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:16.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:16.501085+0000) 2022-09-08T03:29:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:16.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:16.528018+0000) 2022-09-08T03:29:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:17.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:17.528147+0000) 2022-09-08T03:29:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:17.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:17.501234+0000) 2022-09-08T03:29:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:18 smithi059 ceph-mon[97450]: pgmap v11338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:29:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:18 smithi055 ceph-mon[97033]: pgmap v11338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:29:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:18.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:18.501387+0000) 2022-09-08T03:29:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:18.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:18.528280+0000) 2022-09-08T03:29:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:19.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:19.528417+0000) 2022-09-08T03:29:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:19.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:19.501535+0000) 2022-09-08T03:29:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:19 smithi055 ceph-mon[97033]: pgmap v11339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T03:29:20.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:19 smithi059 ceph-mon[97450]: pgmap v11339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T03:29:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:20.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:20.528551+0000) 2022-09-08T03:29:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:20.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:20.501690+0000) 2022-09-08T03:29:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:21.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:21.528683+0000) 2022-09-08T03:29:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:21.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:21.501840+0000) 2022-09-08T03:29:22.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:21 smithi059 ceph-mon[97450]: pgmap v11340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:29:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:21 smithi055 ceph-mon[97033]: pgmap v11340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:29:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:22.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:22.528828+0000) 2022-09-08T03:29:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:22.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:22.501968+0000) 2022-09-08T03:29:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:23.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:23.528966+0000) 2022-09-08T03:29:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:23.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:23.502116+0000) 2022-09-08T03:29:24.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:23 smithi059 ceph-mon[97450]: pgmap v11341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:29:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:23 smithi055 ceph-mon[97033]: pgmap v11341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:29:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:24.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:24.529090+0000) 2022-09-08T03:29:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:24.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:24.502269+0000) 2022-09-08T03:29:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:25.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:25.529220+0000) 2022-09-08T03:29:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:25.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:25.502415+0000) 2022-09-08T03:29:26.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:25 smithi059 ceph-mon[97450]: pgmap v11342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 91 B/s wr, 8 op/s 2022-09-08T03:29:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:25 smithi055 ceph-mon[97033]: pgmap v11342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 91 B/s wr, 8 op/s 2022-09-08T03:29:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:26.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:26.529348+0000) 2022-09-08T03:29:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:26.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:26.502562+0000) 2022-09-08T03:29:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:27.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:27.529482+0000) 2022-09-08T03:29:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:27.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:27.502711+0000) 2022-09-08T03:29:28.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:27 smithi059 ceph-mon[97450]: pgmap v11343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:27 smithi055 ceph-mon[97033]: pgmap v11343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:28.529 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:29:28.266+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:29:28.530 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:28.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:28.502855+0000) 2022-09-08T03:29:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:28.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:28.529612+0000) 2022-09-08T03:29:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:29.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:29.529683+0000) 2022-09-08T03:29:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:29.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:29.502985+0000) 2022-09-08T03:29:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:29 smithi055 ceph-mon[97033]: pgmap v11344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:30.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:29 smithi059 ceph-mon[97450]: pgmap v11344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:30.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:30.529794+0000) 2022-09-08T03:29:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:30.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:30.503127+0000) 2022-09-08T03:29:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:31.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:31.529903+0000) 2022-09-08T03:29:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:31.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:31.503275+0000) 2022-09-08T03:29:32.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:31 smithi059 ceph-mon[97450]: pgmap v11345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:29:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:31 smithi055 ceph-mon[97033]: pgmap v11345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:29:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:32.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:32.530042+0000) 2022-09-08T03:29:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:32.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:32.503427+0000) 2022-09-08T03:29:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:33.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:33.530164+0000) 2022-09-08T03:29:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:33.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:33.503577+0000) 2022-09-08T03:29:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:33 smithi059 ceph-mon[97450]: pgmap v11346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:33 smithi055 ceph-mon[97033]: pgmap v11346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:34.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:34.530303+0000) 2022-09-08T03:29:34.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:34.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:34.503793+0000) 2022-09-08T03:29:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:35.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:35.530439+0000) 2022-09-08T03:29:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:35.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:35.503968+0000) 2022-09-08T03:29:36.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:35 smithi059 ceph-mon[97450]: pgmap v11347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:29:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:35 smithi055 ceph-mon[97033]: pgmap v11347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:29:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:36.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:36.530575+0000) 2022-09-08T03:29:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:36.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:36.504118+0000) 2022-09-08T03:29:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:37.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:37.530709+0000) 2022-09-08T03:29:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:37.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:37.504268+0000) 2022-09-08T03:29:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:37 smithi059 ceph-mon[97450]: pgmap v11348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:29:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:29:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:29:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:37 smithi055 ceph-mon[97033]: pgmap v11348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:29:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:29:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:29:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:38.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:38.530822+0000) 2022-09-08T03:29:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:38.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:38.504419+0000) 2022-09-08T03:29:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:39.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:39.530991+0000) 2022-09-08T03:29:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:39.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:39.504701+0000) 2022-09-08T03:29:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:39 smithi055 ceph-mon[97033]: pgmap v11349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:39 smithi059 ceph-mon[97450]: pgmap v11349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:40.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:40.531140+0000) 2022-09-08T03:29:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:40.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:40.504869+0000) 2022-09-08T03:29:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:41.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:41.531269+0000) 2022-09-08T03:29:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:41.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:41.505008+0000) 2022-09-08T03:29:42.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:41 smithi059 ceph-mon[97450]: pgmap v11350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:29:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:41 smithi055 ceph-mon[97033]: pgmap v11350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:29:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:42.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:42.505134+0000) 2022-09-08T03:29:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:42.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:42.531415+0000) 2022-09-08T03:29:43.531 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:29:43.267+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:29:43.532 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:43.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:43.505273+0000) 2022-09-08T03:29:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:43.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:43.531548+0000) 2022-09-08T03:29:44.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:43 smithi059 ceph-mon[97450]: pgmap v11351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:43 smithi055 ceph-mon[97033]: pgmap v11351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:44.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:44.505418+0000) 2022-09-08T03:29:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:44.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:44.531683+0000) 2022-09-08T03:29:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:45.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:45.531793+0000) 2022-09-08T03:29:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:45.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:45.505557+0000) 2022-09-08T03:29:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:45 smithi059 ceph-mon[97450]: pgmap v11352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:29:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:45 smithi055 ceph-mon[97033]: pgmap v11352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:29:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:46.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:46.531906+0000) 2022-09-08T03:29:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:46.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:46.505698+0000) 2022-09-08T03:29:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:47.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:47.532035+0000) 2022-09-08T03:29:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:47.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:47.505818+0000) 2022-09-08T03:29:48.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:48 smithi059 ceph-mon[97450]: pgmap v11353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:48 smithi055 ceph-mon[97033]: pgmap v11353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:48.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:48.532166+0000) 2022-09-08T03:29:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:48.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:48.505955+0000) 2022-09-08T03:29:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:49.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:49.532296+0000) 2022-09-08T03:29:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:49.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:49.506088+0000) 2022-09-08T03:29:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:49 smithi055 ceph-mon[97033]: pgmap v11354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:49 smithi059 ceph-mon[97450]: pgmap v11354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:50.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:50.506226+0000) 2022-09-08T03:29:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:50.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:50.532430+0000) 2022-09-08T03:29:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:51.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:51.532546+0000) 2022-09-08T03:29:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:51.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:51.506359+0000) 2022-09-08T03:29:52.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:51 smithi059 ceph-mon[97450]: pgmap v11355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:29:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:51 smithi055 ceph-mon[97033]: pgmap v11355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:29:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:52.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:52.532683+0000) 2022-09-08T03:29:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:52.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:52.506511+0000) 2022-09-08T03:29:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:53.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:53.506667+0000) 2022-09-08T03:29:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:53.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:53.532794+0000) 2022-09-08T03:29:54.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:53 smithi059 ceph-mon[97450]: pgmap v11356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:53 smithi055 ceph-mon[97033]: pgmap v11356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:29:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:54.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:54.506803+0000) 2022-09-08T03:29:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:54.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:54.532929+0000) 2022-09-08T03:29:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:55.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:55.533062+0000) 2022-09-08T03:29:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:55.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:55.506920+0000) 2022-09-08T03:29:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:55 smithi059 ceph-mon[97450]: pgmap v11357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:29:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:55 smithi055 ceph-mon[97033]: pgmap v11357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:29:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:56.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:56.533198+0000) 2022-09-08T03:29:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:56.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:56.507062+0000) 2022-09-08T03:29:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:57.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:57.533325+0000) 2022-09-08T03:29:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:57.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:57.507201+0000) 2022-09-08T03:29:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:58 smithi059 ceph-mon[97450]: pgmap v11358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:29:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:58 smithi055 ceph-mon[97033]: pgmap v11358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:29:58.533 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:29:58.272+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:29:58.533 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:58.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:58.507349+0000) 2022-09-08T03:29:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:58.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:58.533457+0000) 2022-09-08T03:29:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:29:59.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:59.507487+0000) 2022-09-08T03:29:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:29:59.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:29:59.533586+0000) 2022-09-08T03:29:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:29:59 smithi055 ceph-mon[97033]: pgmap v11359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:30:00.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:29:59 smithi059 ceph-mon[97450]: pgmap v11359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:30:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:00.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:00.533708+0000) 2022-09-08T03:30:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:00.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:00.507787+0000) 2022-09-08T03:30:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T03:30:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T03:30:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T03:30:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T03:30:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T03:30:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T03:30:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T03:30:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T03:30:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T03:30:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T03:30:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:01.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:01.507934+0000) 2022-09-08T03:30:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:01.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:01.533835+0000) 2022-09-08T03:30:01.979 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-08T03:30:01.638+0000 7f4309522700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.979 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mgr-y[98412]: 2022-09-08T03:30:01.697+0000 7f4309522700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.980 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-1[102223]: 2022-09-08T03:30:01.638+0000 7ff5d6c0c700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.980 INFO:journalctl@ceph.osd.1.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-1[102223]: 2022-09-08T03:30:01.697+0000 7ff5d6c0c700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.981 INFO:journalctl@ceph.osd.4.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-4[108024]: 2022-09-08T03:30:01.638+0000 7f4c6c10e700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.981 INFO:journalctl@ceph.osd.4.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-4[108024]: 2022-09-08T03:30:01.697+0000 7f4c6c10e700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.982 INFO:journalctl@ceph.osd.7.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-7[113964]: 2022-09-08T03:30:01.638+0000 7f17e9531700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.983 INFO:journalctl@ceph.osd.7.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-7[113964]: 2022-09-08T03:30:01.697+0000 7f17e9531700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.983 INFO:journalctl@ceph.osd.10.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-10[119927]: 2022-09-08T03:30:01.638+0000 7fdb14a30700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.984 INFO:journalctl@ceph.osd.10.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-10[119927]: 2022-09-08T03:30:01.697+0000 7fdb14a30700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.984 INFO:journalctl@ceph.mds.b.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-b[124035]: 2022-09-08T03:30:01.638+0000 7fc2a5733700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.985 INFO:journalctl@ceph.mds.b.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-b[124035]: 2022-09-08T03:30:01.697+0000 7fc2a5733700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.986 INFO:journalctl@ceph.mds.h.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-h[123688]: 2022-09-08T03:30:01.638+0000 7f1ee37dc700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.986 INFO:journalctl@ceph.mds.h.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-h[123688]: 2022-09-08T03:30:01.697+0000 7f1ee37dc700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.987 INFO:journalctl@ceph.mds.k.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-k[122989]: 2022-09-08T03:30:01.638+0000 7f48697d1700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.987 INFO:journalctl@ceph.mds.k.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-k[122989]: 2022-09-08T03:30:01.697+0000 7f48697d1700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.988 INFO:journalctl@ceph.mds.e.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-e[123341]: 2022-09-08T03:30:01.638+0000 7fc20f8a0700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.989 INFO:journalctl@ceph.mds.e.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-e[123341]: 2022-09-08T03:30:01.697+0000 7fc20f8a0700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-b[97009]: 2022-09-08T03:30:01.638+0000 7f33dacd9700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-b[97009]: 2022-09-08T03:30:01.638+0000 7f33dacd9700 -1 mon.b@2(peon) e3 *** Got Signal Hangup *** 2022-09-08T03:30:01.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-b[97009]: 2022-09-08T03:30:01.697+0000 7f33dacd9700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-09-08T03:30:01.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-b[97009]: 2022-09-08T03:30:01.697+0000 7f33dacd9700 -1 mon.b@2(peon) e3 *** Got Signal Hangup *** 2022-09-08T03:30:01.991 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:01 smithi055 ceph-mon[97033]: pgmap v11360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:30:02.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:01 smithi059 ceph-mon[97450]: pgmap v11360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:30:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:02.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:02.533984+0000) 2022-09-08T03:30:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:02.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:02.508070+0000) 2022-09-08T03:30:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:03.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:03.534112+0000) 2022-09-08T03:30:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:03.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:03.508208+0000) 2022-09-08T03:30:04.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:03 smithi059 ceph-mon[97450]: pgmap v11361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-09-08T03:30:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:03 smithi055 ceph-mon[97033]: pgmap v11361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-09-08T03:30:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:04.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:04.534251+0000) 2022-09-08T03:30:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:04.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:04.508351+0000) 2022-09-08T03:30:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:05.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:05.534385+0000) 2022-09-08T03:30:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:05.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:05.508490+0000) 2022-09-08T03:30:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:05 smithi059 ceph-mon[97450]: pgmap v11362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T03:30:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:30:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:30:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:05 smithi055 ceph-mon[97033]: pgmap v11362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T03:30:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:30:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:30:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:06.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:06.534523+0000) 2022-09-08T03:30:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:06.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:06.508608+0000) 2022-09-08T03:30:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:07.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:07.534668+0000) 2022-09-08T03:30:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:07.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:07.508707+0000) 2022-09-08T03:30:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:08 smithi055 ceph-mon[97033]: pgmap v11363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:30:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:08 smithi055 ceph-mon[97033]: pgmap v11364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 22 op/s 2022-09-08T03:30:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:08 smithi055 ceph-mon[97033]: pgmap v11365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T03:30:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:30:08.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:08 smithi059 ceph-mon[97450]: pgmap v11363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:30:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:08 smithi059 ceph-mon[97450]: pgmap v11364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 22 op/s 2022-09-08T03:30:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:08 smithi059 ceph-mon[97450]: pgmap v11365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T03:30:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:30:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:08.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:08.534799+0000) 2022-09-08T03:30:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:08.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:08.508861+0000) 2022-09-08T03:30:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:09.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:09.534917+0000) 2022-09-08T03:30:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:09.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:09.509002+0000) 2022-09-08T03:30:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:10 smithi055 ceph-mon[97033]: pgmap v11366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T03:30:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:10 smithi055 ceph-mon[97033]: osdmap e789: 12 total, 11 up, 11 in 2022-09-08T03:30:10.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:10 smithi059 ceph-mon[97450]: pgmap v11366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T03:30:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:10 smithi059 ceph-mon[97450]: osdmap e789: 12 total, 11 up, 11 in 2022-09-08T03:30:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:10.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:10.535065+0000) 2022-09-08T03:30:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:10.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:10.509140+0000) 2022-09-08T03:30:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:11.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:11.535287+0000) 2022-09-08T03:30:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:11.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:11.509370+0000) 2022-09-08T03:30:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:12 smithi055 ceph-mon[97033]: pgmap v11368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T03:30:12.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:12 smithi059 ceph-mon[97450]: pgmap v11368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T03:30:12.806 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:12.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:12.535469+0000) 2022-09-08T03:30:12.807 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:12.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:12.509550+0000) 2022-09-08T03:30:13.547 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:30:13.273+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:30:13.548 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:13.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:13.509691+0000) 2022-09-08T03:30:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:13.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:13.535599+0000) 2022-09-08T03:30:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:14 smithi055 ceph-mon[97033]: pgmap v11369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T03:30:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:14 smithi059 ceph-mon[97450]: pgmap v11369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T03:30:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:14.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:14.509829+0000) 2022-09-08T03:30:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:14.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:14.535731+0000) 2022-09-08T03:30:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:15.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:15.509970+0000) 2022-09-08T03:30:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:15.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:15.535866+0000) 2022-09-08T03:30:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:16 smithi055 ceph-mon[97033]: pgmap v11370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:30:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:16 smithi059 ceph-mon[97450]: pgmap v11370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:30:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:16.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:16.510117+0000) 2022-09-08T03:30:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:16.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:16.535978+0000) 2022-09-08T03:30:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:17.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:17.510249+0000) 2022-09-08T03:30:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:17.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:17.536089+0000) 2022-09-08T03:30:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:18 smithi055 ceph-mon[97033]: pgmap v11371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:30:18.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:18 smithi059 ceph-mon[97450]: pgmap v11371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:30:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:18.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:18.536221+0000) 2022-09-08T03:30:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:18.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:18.510399+0000) 2022-09-08T03:30:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:19.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:19.536367+0000) 2022-09-08T03:30:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:19.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:19.510534+0000) 2022-09-08T03:30:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:19 smithi055 ceph-mon[97033]: osdmap e790: 12 total, 11 up, 11 in 2022-09-08T03:30:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:19 smithi055 ceph-mon[97033]: pgmap v11373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:30:20.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:19 smithi059 ceph-mon[97450]: osdmap e790: 12 total, 11 up, 11 in 2022-09-08T03:30:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:19 smithi059 ceph-mon[97450]: pgmap v11373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:30:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:20.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:20.536538+0000) 2022-09-08T03:30:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:20.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:20.510675+0000) 2022-09-08T03:30:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:21.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:21.536669+0000) 2022-09-08T03:30:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:21.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:21.510806+0000) 2022-09-08T03:30:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:22 smithi059 ceph-mon[97450]: pgmap v11374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:30:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:22 smithi055 ceph-mon[97033]: pgmap v11374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:30:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:22.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:22.536809+0000) 2022-09-08T03:30:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:22.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:22.510939+0000) 2022-09-08T03:30:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:23.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:23.536944+0000) 2022-09-08T03:30:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:23.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:23.511071+0000) 2022-09-08T03:30:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:24 smithi059 ceph-mon[97450]: pgmap v11375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:30:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:24 smithi055 ceph-mon[97033]: pgmap v11375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:30:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:24.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:24.537083+0000) 2022-09-08T03:30:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:24.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:24.511213+0000) 2022-09-08T03:30:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:25.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:25.537223+0000) 2022-09-08T03:30:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:25.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:25.511344+0000) 2022-09-08T03:30:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:26 smithi059 ceph-mon[97450]: pgmap v11376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:30:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:26 smithi055 ceph-mon[97033]: pgmap v11376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:30:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:26.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:26.537369+0000) 2022-09-08T03:30:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:26.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:26.511480+0000) 2022-09-08T03:30:27.537 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:27.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:27.511616+0000) 2022-09-08T03:30:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:27.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:27.537518+0000) 2022-09-08T03:30:28.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:28 smithi059 ceph-mon[97450]: pgmap v11377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:30:28.537 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:30:28.283+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:30:28.538 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:28.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:28.511772+0000) 2022-09-08T03:30:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:28 smithi055 ceph-mon[97033]: pgmap v11377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:30:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:28.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:28.537679+0000) 2022-09-08T03:30:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:29.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:29.537794+0000) 2022-09-08T03:30:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:29.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:29.511906+0000) 2022-09-08T03:30:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:29 smithi055 ceph-mon[97033]: pgmap v11378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:30:30.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:29 smithi059 ceph-mon[97450]: pgmap v11378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:30:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:30.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:30.537924+0000) 2022-09-08T03:30:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:30.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:30.512038+0000) 2022-09-08T03:30:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:31.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:31.512169+0000) 2022-09-08T03:30:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:31.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:31.538058+0000) 2022-09-08T03:30:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:32 smithi059 ceph-mon[97450]: pgmap v11379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:30:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:32 smithi055 ceph-mon[97033]: pgmap v11379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:30:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:32.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:32.538199+0000) 2022-09-08T03:30:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:32.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:32.512314+0000) 2022-09-08T03:30:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:33.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:33.538328+0000) 2022-09-08T03:30:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:33.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:33.512450+0000) 2022-09-08T03:30:34.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:34 smithi059 ceph-mon[97450]: pgmap v11380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:34 smithi055 ceph-mon[97033]: pgmap v11380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:34.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:34.538471+0000) 2022-09-08T03:30:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:34.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:34.512581+0000) 2022-09-08T03:30:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:35.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:35.538616+0000) 2022-09-08T03:30:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:35.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:35.512718+0000) 2022-09-08T03:30:36.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:36 smithi059 ceph-mon[97450]: pgmap v11381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:30:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:36 smithi055 ceph-mon[97033]: pgmap v11381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:30:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:36.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:36.538770+0000) 2022-09-08T03:30:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:36.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:36.512858+0000) 2022-09-08T03:30:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:37.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:37.512997+0000) 2022-09-08T03:30:37.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:37.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:37.538895+0000) 2022-09-08T03:30:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:38 smithi059 ceph-mon[97450]: pgmap v11382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:38 smithi055 ceph-mon[97033]: pgmap v11382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:38.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:38.539030+0000) 2022-09-08T03:30:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:38.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:38.513145+0000) 2022-09-08T03:30:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:39.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:39.539166+0000) 2022-09-08T03:30:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:39.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:39.513276+0000) 2022-09-08T03:30:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:39 smithi055 ceph-mon[97033]: pgmap v11383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:39 smithi059 ceph-mon[97450]: pgmap v11383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:40.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:40.539291+0000) 2022-09-08T03:30:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:40.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:40.513434+0000) 2022-09-08T03:30:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:41.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:41.539417+0000) 2022-09-08T03:30:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:41.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:41.513568+0000) 2022-09-08T03:30:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:42 smithi059 ceph-mon[97450]: pgmap v11384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:30:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:42 smithi055 ceph-mon[97033]: pgmap v11384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:30:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:42.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:42.539569+0000) 2022-09-08T03:30:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:42.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:42.513705+0000) 2022-09-08T03:30:43.539 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:30:43.283+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:30:43.541 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:43.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:43.513915+0000) 2022-09-08T03:30:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:43.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:43.539754+0000) 2022-09-08T03:30:44.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:44 smithi059 ceph-mon[97450]: pgmap v11385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:44 smithi055 ceph-mon[97033]: pgmap v11385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:44.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:44.539898+0000) 2022-09-08T03:30:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:44.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:44.514059+0000) 2022-09-08T03:30:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:45.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:45.540023+0000) 2022-09-08T03:30:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:45.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:45.514198+0000) 2022-09-08T03:30:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:46 smithi055 ceph-mon[97033]: pgmap v11386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:30:46.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:46 smithi059 ceph-mon[97450]: pgmap v11386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:30:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:46.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:46.540150+0000) 2022-09-08T03:30:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:46.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:46.514333+0000) 2022-09-08T03:30:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:47.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:47.540282+0000) 2022-09-08T03:30:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:47.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:47.514474+0000) 2022-09-08T03:30:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:48 smithi055 ceph-mon[97033]: pgmap v11387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:48.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:48 smithi059 ceph-mon[97450]: pgmap v11387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:48.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:48.540417+0000) 2022-09-08T03:30:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:48.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:48.514630+0000) 2022-09-08T03:30:49.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:49.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:49.514782+0000) 2022-09-08T03:30:49.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:49.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:49.540552+0000) 2022-09-08T03:30:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:49 smithi055 ceph-mon[97033]: pgmap v11388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:50.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:49 smithi059 ceph-mon[97450]: pgmap v11388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:50.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:50.514921+0000) 2022-09-08T03:30:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:50.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:50.540693+0000) 2022-09-08T03:30:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:51.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:51.540817+0000) 2022-09-08T03:30:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:51.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:51.515061+0000) 2022-09-08T03:30:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:52 smithi059 ceph-mon[97450]: pgmap v11389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:30:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:52 smithi055 ceph-mon[97033]: pgmap v11389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:30:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:52.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:52.540959+0000) 2022-09-08T03:30:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:52.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:52.515197+0000) 2022-09-08T03:30:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:53.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:53.541103+0000) 2022-09-08T03:30:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:53.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:53.515355+0000) 2022-09-08T03:30:54.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:54 smithi059 ceph-mon[97450]: pgmap v11390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:54 smithi055 ceph-mon[97033]: pgmap v11390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:54.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:54.541238+0000) 2022-09-08T03:30:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:54.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:54.515491+0000) 2022-09-08T03:30:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:55.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:55.541374+0000) 2022-09-08T03:30:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:55.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:55.515631+0000) 2022-09-08T03:30:56.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:56 smithi059 ceph-mon[97450]: pgmap v11391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:30:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:56 smithi055 ceph-mon[97033]: pgmap v11391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:30:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:56.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:56.541557+0000) 2022-09-08T03:30:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:56.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:56.515845+0000) 2022-09-08T03:30:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:57.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:57.541702+0000) 2022-09-08T03:30:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:57.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:57.516017+0000) 2022-09-08T03:30:58.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:58 smithi059 ceph-mon[97450]: pgmap v11392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:58.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:30:58.299+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:30:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:58.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:58.541837+0000) 2022-09-08T03:30:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:58.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:58.516166+0000) 2022-09-08T03:30:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:58 smithi055 ceph-mon[97033]: pgmap v11392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:30:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:30:59.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:59.541972+0000) 2022-09-08T03:30:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:30:59.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:30:59.516307+0000) 2022-09-08T03:30:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:30:59 smithi055 ceph-mon[97033]: pgmap v11393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:00.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:30:59 smithi059 ceph-mon[97450]: pgmap v11393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:00.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:00.542114+0000) 2022-09-08T03:31:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:00.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:00.516441+0000) 2022-09-08T03:31:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:01.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:01.542237+0000) 2022-09-08T03:31:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:01.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:01.516583+0000) 2022-09-08T03:31:02.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:02 smithi059 ceph-mon[97450]: pgmap v11394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:31:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:02 smithi055 ceph-mon[97033]: pgmap v11394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:31:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:02.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:02.542384+0000) 2022-09-08T03:31:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:02.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:02.516731+0000) 2022-09-08T03:31:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:03.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:03.542523+0000) 2022-09-08T03:31:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:03.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:03.516876+0000) 2022-09-08T03:31:04.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:04 smithi059 ceph-mon[97450]: pgmap v11395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:04 smithi055 ceph-mon[97033]: pgmap v11395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:04.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:04.542672+0000) 2022-09-08T03:31:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:04.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:04.517012+0000) 2022-09-08T03:31:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:05.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:05.542805+0000) 2022-09-08T03:31:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:05.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:05.517151+0000) 2022-09-08T03:31:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:06 smithi059 ceph-mon[97450]: pgmap v11396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:31:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:31:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:31:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:06 smithi055 ceph-mon[97033]: pgmap v11396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:31:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:31:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:31:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:06.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:06.542939+0000) 2022-09-08T03:31:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:06.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:06.517290+0000) 2022-09-08T03:31:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:07.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:07.543076+0000) 2022-09-08T03:31:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:07.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:07.517431+0000) 2022-09-08T03:31:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:08 smithi055 ceph-mon[97033]: pgmap v11397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:31:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:31:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:31:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:31:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:08 smithi059 ceph-mon[97450]: pgmap v11397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:31:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:31:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:31:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:31:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:08.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:08.543210+0000) 2022-09-08T03:31:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:08.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:08.517587+0000) 2022-09-08T03:31:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:09.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:09.543336+0000) 2022-09-08T03:31:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:09.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:09.517699+0000) 2022-09-08T03:31:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:10 smithi055 ceph-mon[97033]: pgmap v11398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:31:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:10 smithi055 ceph-mon[97033]: osdmap e791: 12 total, 11 up, 11 in 2022-09-08T03:31:10.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:10 smithi059 ceph-mon[97450]: pgmap v11398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:31:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:10 smithi059 ceph-mon[97450]: osdmap e791: 12 total, 11 up, 11 in 2022-09-08T03:31:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:10.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:10.543459+0000) 2022-09-08T03:31:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:10.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:10.517831+0000) 2022-09-08T03:31:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:11.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:11.543590+0000) 2022-09-08T03:31:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:11.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:11.517984+0000) 2022-09-08T03:31:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:12 smithi055 ceph-mon[97033]: pgmap v11400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:31:12.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:12 smithi059 ceph-mon[97450]: pgmap v11400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:31:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:12.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:12.543728+0000) 2022-09-08T03:31:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:12.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:12.518128+0000) 2022-09-08T03:31:13.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:31:13.305+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:31:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:13.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:13.543845+0000) 2022-09-08T03:31:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:13.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:13.518263+0000) 2022-09-08T03:31:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:14 smithi055 ceph-mon[97033]: pgmap v11401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:31:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:14 smithi055 ceph-mon[97033]: osdmap e792: 12 total, 11 up, 11 in 2022-09-08T03:31:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:14 smithi059 ceph-mon[97450]: pgmap v11401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:31:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:14 smithi059 ceph-mon[97450]: osdmap e792: 12 total, 11 up, 11 in 2022-09-08T03:31:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:14.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:14.543960+0000) 2022-09-08T03:31:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:14.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:14.518399+0000) 2022-09-08T03:31:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:15.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:15.544100+0000) 2022-09-08T03:31:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:15.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:15.518541+0000) 2022-09-08T03:31:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:16 smithi055 ceph-mon[97033]: pgmap v11403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:31:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:16 smithi059 ceph-mon[97450]: pgmap v11403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:31:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:16.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:16.544232+0000) 2022-09-08T03:31:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:16.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:16.518687+0000) 2022-09-08T03:31:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:17.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:17.544375+0000) 2022-09-08T03:31:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:17.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:17.518828+0000) 2022-09-08T03:31:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:18 smithi055 ceph-mon[97033]: pgmap v11404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:31:18.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:18 smithi059 ceph-mon[97450]: pgmap v11404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:31:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:18.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:18.544497+0000) 2022-09-08T03:31:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:18.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:18.518979+0000) 2022-09-08T03:31:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:19.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:19.544650+0000) 2022-09-08T03:31:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:19.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:19.519122+0000) 2022-09-08T03:31:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:19 smithi055 ceph-mon[97033]: pgmap v11405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T03:31:20.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:19 smithi059 ceph-mon[97450]: pgmap v11405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T03:31:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:20.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:20.544769+0000) 2022-09-08T03:31:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:20.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:20.519265+0000) 2022-09-08T03:31:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:21.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:21.544900+0000) 2022-09-08T03:31:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:21.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:21.519400+0000) 2022-09-08T03:31:22.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:22 smithi059 ceph-mon[97450]: pgmap v11406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:31:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:22 smithi055 ceph-mon[97033]: pgmap v11406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:31:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:22.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:22.545017+0000) 2022-09-08T03:31:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:22.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:22.519535+0000) 2022-09-08T03:31:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:23.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:23.545160+0000) 2022-09-08T03:31:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:23.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:23.519686+0000) 2022-09-08T03:31:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:24 smithi055 ceph-mon[97033]: pgmap v11407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:31:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:24 smithi059 ceph-mon[97450]: pgmap v11407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:31:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:24.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:24.545291+0000) 2022-09-08T03:31:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:24.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:24.519831+0000) 2022-09-08T03:31:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:25.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:25.545420+0000) 2022-09-08T03:31:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:25.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:25.519953+0000) 2022-09-08T03:31:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:26 smithi055 ceph-mon[97033]: pgmap v11408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 89 B/s wr, 7 op/s 2022-09-08T03:31:26.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:26 smithi059 ceph-mon[97450]: pgmap v11408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 89 B/s wr, 7 op/s 2022-09-08T03:31:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:26.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:26.545558+0000) 2022-09-08T03:31:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:26.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:26.520088+0000) 2022-09-08T03:31:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:27.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:27.545688+0000) 2022-09-08T03:31:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:27.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:27.520209+0000) 2022-09-08T03:31:28.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:31:28.319+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:31:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:28.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:28.545816+0000) 2022-09-08T03:31:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:28.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:28.520360+0000) 2022-09-08T03:31:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:28 smithi055 ceph-mon[97033]: pgmap v11409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:28.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:28 smithi059 ceph-mon[97450]: pgmap v11409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:29.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:29.545934+0000) 2022-09-08T03:31:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:29.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:29.520499+0000) 2022-09-08T03:31:30.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:29 smithi059 ceph-mon[97450]: pgmap v11410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:29 smithi055 ceph-mon[97033]: pgmap v11410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:30.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:30.546065+0000) 2022-09-08T03:31:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:30.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:30.520647+0000) 2022-09-08T03:31:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:31.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:31.546203+0000) 2022-09-08T03:31:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:31.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:31.520778+0000) 2022-09-08T03:31:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:32 smithi055 ceph-mon[97033]: pgmap v11411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:31:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:32 smithi059 ceph-mon[97450]: pgmap v11411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:31:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:32.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:32.546338+0000) 2022-09-08T03:31:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:32.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:32.520916+0000) 2022-09-08T03:31:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:33.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:33.546470+0000) 2022-09-08T03:31:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:33.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:33.521063+0000) 2022-09-08T03:31:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:34 smithi055 ceph-mon[97033]: pgmap v11412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:34.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:34 smithi059 ceph-mon[97450]: pgmap v11412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:34.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:34.546585+0000) 2022-09-08T03:31:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:34.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:34.521213+0000) 2022-09-08T03:31:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:35.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:35.546708+0000) 2022-09-08T03:31:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:35.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:35.521362+0000) 2022-09-08T03:31:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:36 smithi055 ceph-mon[97033]: pgmap v11413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:31:36.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:36 smithi059 ceph-mon[97450]: pgmap v11413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:31:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:36.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:36.546836+0000) 2022-09-08T03:31:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:36.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:36.521514+0000) 2022-09-08T03:31:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:37.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:37.546971+0000) 2022-09-08T03:31:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:37.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:37.521668+0000) 2022-09-08T03:31:38.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:38 smithi059 ceph-mon[97450]: pgmap v11414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:38 smithi059 ceph-mon[97450]: pgmap v11415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:31:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:38 smithi059 ceph-mon[97450]: pgmap v11416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:31:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:31:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:38.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:38.547100+0000) 2022-09-08T03:31:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:38.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:38.521826+0000) 2022-09-08T03:31:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:38 smithi055 ceph-mon[97033]: pgmap v11414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:38 smithi055 ceph-mon[97033]: pgmap v11415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:31:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:38 smithi055 ceph-mon[97033]: pgmap v11416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:31:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:31:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:39.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:39.547248+0000) 2022-09-08T03:31:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:39.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:39.521956+0000) 2022-09-08T03:31:40.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:39 smithi059 ceph-mon[97450]: pgmap v11417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:31:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:39 smithi055 ceph-mon[97033]: pgmap v11417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:31:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:40.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:40.547398+0000) 2022-09-08T03:31:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:40.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:40.522106+0000) 2022-09-08T03:31:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:41.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:41.547543+0000) 2022-09-08T03:31:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:41.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:41.522236+0000) 2022-09-08T03:31:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:41 smithi055 ceph-mon[97033]: pgmap v11418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:31:42.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:41 smithi059 ceph-mon[97450]: pgmap v11418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:31:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:42.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:42.547695+0000) 2022-09-08T03:31:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:42.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:42.522383+0000) 2022-09-08T03:31:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:43.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:43.547812+0000) 2022-09-08T03:31:43.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:31:43.325+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:31:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:43.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:43.522509+0000) 2022-09-08T03:31:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:43 smithi055 ceph-mon[97033]: pgmap v11419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:31:44.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:43 smithi059 ceph-mon[97450]: pgmap v11419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:31:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:44.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:44.522667+0000) 2022-09-08T03:31:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:44.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:44.547925+0000) 2022-09-08T03:31:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:45.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:45.522816+0000) 2022-09-08T03:31:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:45.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:45.548056+0000) 2022-09-08T03:31:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:45 smithi055 ceph-mon[97033]: pgmap v11420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:31:46.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:45 smithi059 ceph-mon[97450]: pgmap v11420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:31:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:46.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:46.548181+0000) 2022-09-08T03:31:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:46.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:46.522962+0000) 2022-09-08T03:31:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:47.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:47.548318+0000) 2022-09-08T03:31:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:47.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:47.523110+0000) 2022-09-08T03:31:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:47 smithi055 ceph-mon[97033]: pgmap v11421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:31:48.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:47 smithi059 ceph-mon[97450]: pgmap v11421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:31:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:48.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:48.548455+0000) 2022-09-08T03:31:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:48.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:48.523266+0000) 2022-09-08T03:31:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:49.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:49.548589+0000) 2022-09-08T03:31:49.935 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:49.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:49.523411+0000) 2022-09-08T03:31:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:49 smithi059 ceph-mon[97450]: pgmap v11422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:49 smithi055 ceph-mon[97033]: pgmap v11422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:50.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:50.548709+0000) 2022-09-08T03:31:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:50.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:50.523556+0000) 2022-09-08T03:31:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:51.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:51.548850+0000) 2022-09-08T03:31:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:51.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:51.523691+0000) 2022-09-08T03:31:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:51 smithi055 ceph-mon[97033]: pgmap v11423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:31:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:51 smithi059 ceph-mon[97450]: pgmap v11423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:31:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:52.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:52.548992+0000) 2022-09-08T03:31:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:52.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:52.523845+0000) 2022-09-08T03:31:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:53.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:53.549117+0000) 2022-09-08T03:31:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:53.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:53.523978+0000) 2022-09-08T03:31:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:53 smithi055 ceph-mon[97033]: pgmap v11424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:54.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:53 smithi059 ceph-mon[97450]: pgmap v11424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:54.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:54.549258+0000) 2022-09-08T03:31:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:54.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:54.524108+0000) 2022-09-08T03:31:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:55.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:55.549395+0000) 2022-09-08T03:31:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:55.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:55.524253+0000) 2022-09-08T03:31:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:55 smithi055 ceph-mon[97033]: pgmap v11425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:31:56.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:55 smithi059 ceph-mon[97450]: pgmap v11425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:31:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:56.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:56.549526+0000) 2022-09-08T03:31:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:56.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:56.524388+0000) 2022-09-08T03:31:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:57.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:57.524526+0000) 2022-09-08T03:31:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:57.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:57.549664+0000) 2022-09-08T03:31:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:57 smithi055 ceph-mon[97033]: pgmap v11426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:57 smithi059 ceph-mon[97450]: pgmap v11426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:31:58.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:31:58.338+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:31:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:58.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:58.549796+0000) 2022-09-08T03:31:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:58.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:58.524658+0000) 2022-09-08T03:31:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:31:59.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:59.549912+0000) 2022-09-08T03:31:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:31:59.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:31:59.524792+0000) 2022-09-08T03:32:00.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:31:59 smithi059 ceph-mon[97450]: pgmap v11427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:31:59 smithi055 ceph-mon[97033]: pgmap v11427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:00.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:00.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:00.550055+0000) 2022-09-08T03:32:00.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:00.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:00.524910+0000) 2022-09-08T03:32:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:01.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:01.550194+0000) 2022-09-08T03:32:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:01.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:01.525048+0000) 2022-09-08T03:32:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:01 smithi055 ceph-mon[97033]: pgmap v11428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:32:02.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:01 smithi059 ceph-mon[97450]: pgmap v11428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:32:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:02.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:02.550337+0000) 2022-09-08T03:32:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:02.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:02.525254+0000) 2022-09-08T03:32:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:03.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:03.550523+0000) 2022-09-08T03:32:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:03.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:03.525418+0000) 2022-09-08T03:32:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:03 smithi055 ceph-mon[97033]: pgmap v11429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:04.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:03 smithi059 ceph-mon[97450]: pgmap v11429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:04.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:04.550669+0000) 2022-09-08T03:32:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:04.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:04.525566+0000) 2022-09-08T03:32:05.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:32:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:05.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:05.550801+0000) 2022-09-08T03:32:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:05.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:05.525704+0000) 2022-09-08T03:32:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:32:06.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:06 smithi059 ceph-mon[97450]: pgmap v11430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:32:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:32:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:06.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:06.550916+0000) 2022-09-08T03:32:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:06.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:06.525822+0000) 2022-09-08T03:32:06.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:06 smithi055 ceph-mon[97033]: pgmap v11430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:32:06.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:32:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:07.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:07.551049+0000) 2022-09-08T03:32:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:07.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:07.525960+0000) 2022-09-08T03:32:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:07 smithi055 ceph-mon[97033]: pgmap v11431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:32:08.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:07 smithi059 ceph-mon[97450]: pgmap v11431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:32:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:08.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:08.551172+0000) 2022-09-08T03:32:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:08.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:08.526112+0000) 2022-09-08T03:32:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:09.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:09.551324+0000) 2022-09-08T03:32:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:09.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:09.526253+0000) 2022-09-08T03:32:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:09 smithi055 ceph-mon[97033]: osdmap e793: 12 total, 11 up, 11 in 2022-09-08T03:32:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:09 smithi055 ceph-mon[97033]: pgmap v11433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:32:10.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:09 smithi059 ceph-mon[97450]: osdmap e793: 12 total, 11 up, 11 in 2022-09-08T03:32:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:09 smithi059 ceph-mon[97450]: pgmap v11433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:32:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:10.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:10.551459+0000) 2022-09-08T03:32:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:10.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:10.526393+0000) 2022-09-08T03:32:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:11.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:11.551596+0000) 2022-09-08T03:32:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:11.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:11.526532+0000) 2022-09-08T03:32:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:11 smithi055 ceph-mon[97033]: pgmap v11434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:32:12.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:11 smithi059 ceph-mon[97450]: pgmap v11434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:32:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:12.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:12.551701+0000) 2022-09-08T03:32:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:12.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:12.526684+0000) 2022-09-08T03:32:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:13.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:13.551820+0000) 2022-09-08T03:32:13.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:13.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:13.526823+0000) 2022-09-08T03:32:13.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:32:13.346+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:32:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:13 smithi055 ceph-mon[97033]: pgmap v11435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:32:14.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:13 smithi059 ceph-mon[97450]: pgmap v11435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:32:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:14.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:14.551955+0000) 2022-09-08T03:32:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:14.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:14.526963+0000) 2022-09-08T03:32:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:14 smithi059 ceph-mon[97450]: osdmap e794: 12 total, 11 up, 11 in 2022-09-08T03:32:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:14 smithi055 ceph-mon[97033]: osdmap e794: 12 total, 11 up, 11 in 2022-09-08T03:32:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:15.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:15.552089+0000) 2022-09-08T03:32:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:15.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:15.527098+0000) 2022-09-08T03:32:16.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:15 smithi059 ceph-mon[97450]: pgmap v11437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:32:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:15 smithi055 ceph-mon[97033]: pgmap v11437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:32:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:16.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:16.552236+0000) 2022-09-08T03:32:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:16.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:16.527239+0000) 2022-09-08T03:32:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:17.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:17.527378+0000) 2022-09-08T03:32:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:17.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:17.552348+0000) 2022-09-08T03:32:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:17 smithi055 ceph-mon[97033]: pgmap v11438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-08T03:32:18.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:17 smithi059 ceph-mon[97450]: pgmap v11438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-08T03:32:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:18.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:18.552465+0000) 2022-09-08T03:32:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:18.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:18.527526+0000) 2022-09-08T03:32:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:19.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:19.552596+0000) 2022-09-08T03:32:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:19.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:19.527674+0000) 2022-09-08T03:32:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:19 smithi059 ceph-mon[97450]: pgmap v11439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:32:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:19 smithi055 ceph-mon[97033]: pgmap v11439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:32:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:20.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:20.527796+0000) 2022-09-08T03:32:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:20.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:20.552728+0000) 2022-09-08T03:32:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:21.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:21.552854+0000) 2022-09-08T03:32:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:21.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:21.527915+0000) 2022-09-08T03:32:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:21 smithi055 ceph-mon[97033]: pgmap v11440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:32:22.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:21 smithi059 ceph-mon[97450]: pgmap v11440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:32:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:22.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:22.552993+0000) 2022-09-08T03:32:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:22.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:22.528067+0000) 2022-09-08T03:32:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:23.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:23.553130+0000) 2022-09-08T03:32:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:23.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:23.528214+0000) 2022-09-08T03:32:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:23 smithi055 ceph-mon[97033]: pgmap v11441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:32:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:23 smithi059 ceph-mon[97450]: pgmap v11441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:32:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:24.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:24.553269+0000) 2022-09-08T03:32:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:24.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:24.528361+0000) 2022-09-08T03:32:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:25.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:25.553402+0000) 2022-09-08T03:32:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:25.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:25.528500+0000) 2022-09-08T03:32:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:26.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:26.553537+0000) 2022-09-08T03:32:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:26.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:26.528643+0000) 2022-09-08T03:32:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:26 smithi055 ceph-mon[97033]: pgmap v11442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-08T03:32:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:26 smithi059 ceph-mon[97450]: pgmap v11442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-08T03:32:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:27.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:27.528776+0000) 2022-09-08T03:32:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:27.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:27.553673+0000) 2022-09-08T03:32:28.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:28.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:28.528914+0000) 2022-09-08T03:32:28.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:32:28.363+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:32:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:28.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:28.553809+0000) 2022-09-08T03:32:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:28 smithi055 ceph-mon[97033]: pgmap v11443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:28 smithi059 ceph-mon[97450]: pgmap v11443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:29.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:29.529041+0000) 2022-09-08T03:32:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:29.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:29.553948+0000) 2022-09-08T03:32:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:30.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:30.554086+0000) 2022-09-08T03:32:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:30.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:30.529168+0000) 2022-09-08T03:32:31.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:30 smithi059 ceph-mon[97450]: pgmap v11444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:30 smithi055 ceph-mon[97033]: pgmap v11444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:31.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:31.554217+0000) 2022-09-08T03:32:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:31.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:31.529299+0000) 2022-09-08T03:32:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:32.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:32.554355+0000) 2022-09-08T03:32:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:32.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:32.529434+0000) 2022-09-08T03:32:33.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:32 smithi059 ceph-mon[97450]: pgmap v11445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:32:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:32 smithi055 ceph-mon[97033]: pgmap v11445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:32:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:33.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:33.554478+0000) 2022-09-08T03:32:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:33.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:33.529567+0000) 2022-09-08T03:32:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:34.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:34.554610+0000) 2022-09-08T03:32:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:34.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:34.529705+0000) 2022-09-08T03:32:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:34 smithi059 ceph-mon[97450]: pgmap v11446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:35.237 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:34 smithi055 ceph-mon[97033]: pgmap v11446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:35.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:35.529858+0000) 2022-09-08T03:32:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:35.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:35.554781+0000) 2022-09-08T03:32:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:36.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:36.529981+0000) 2022-09-08T03:32:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:36.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:36.554910+0000) 2022-09-08T03:32:37.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:36 smithi059 ceph-mon[97450]: pgmap v11447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:32:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:36 smithi055 ceph-mon[97033]: pgmap v11447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:32:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:37.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:37.555042+0000) 2022-09-08T03:32:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:37.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:37.530121+0000) 2022-09-08T03:32:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:32:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:32:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:32:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:32:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:32:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:32:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:38.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:38.555172+0000) 2022-09-08T03:32:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:38.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:38.530268+0000) 2022-09-08T03:32:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:39 smithi055 ceph-mon[97033]: pgmap v11448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:32:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:32:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:39 smithi059 ceph-mon[97450]: pgmap v11448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:32:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:32:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:39.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:39.555305+0000) 2022-09-08T03:32:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:39.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:39.530402+0000) 2022-09-08T03:32:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:40.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:40.555437+0000) 2022-09-08T03:32:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:40.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:40.530537+0000) 2022-09-08T03:32:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:41 smithi055 ceph-mon[97033]: pgmap v11449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:32:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:32:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:41.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:41.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:41.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:41.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:41 smithi059 ceph-mon[97450]: pgmap v11449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:32:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:32:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:41.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:41.555576+0000) 2022-09-08T03:32:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:41.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:41.530668+0000) 2022-09-08T03:32:42.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:42 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T03:32:42.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:42 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T03:32:42.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:32:42.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:32:42.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:42.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:42.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:42.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:42.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:42 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T03:32:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:42 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T03:32:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:32:42.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:32:42.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:42.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:42.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:42.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:32:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:42.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:42.530803+0000) 2022-09-08T03:32:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:42.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:42.555707+0000) 2022-09-08T03:32:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:43.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:43.530938+0000) 2022-09-08T03:32:43.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:32:43.378+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:32:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:43.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:43.555847+0000) 2022-09-08T03:32:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:43 smithi055 ceph-mon[97033]: pgmap v11450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:32:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:43 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T03:32:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:43 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T03:32:43.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:43 smithi059 ceph-mon[97450]: pgmap v11450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:32:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:43 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T03:32:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:43 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T03:32:44.870 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:44.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:44.531075+0000) 2022-09-08T03:32:44.871 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:44.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:44.555980+0000) 2022-09-08T03:32:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:45 smithi055 ceph-mon[97033]: pgmap v11451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:45.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:45 smithi059 ceph-mon[97450]: pgmap v11451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:45.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:45.531217+0000) 2022-09-08T03:32:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:45.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:45.556095+0000) 2022-09-08T03:32:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:46.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:46.556235+0000) 2022-09-08T03:32:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:46.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:46.531346+0000) 2022-09-08T03:32:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:47 smithi055 ceph-mon[97033]: pgmap v11452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:32:47.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:47 smithi059 ceph-mon[97450]: pgmap v11452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:32:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:47.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:47.556378+0000) 2022-09-08T03:32:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:47.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:47.531487+0000) 2022-09-08T03:32:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:48.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:48.556512+0000) 2022-09-08T03:32:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:48.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:48.531633+0000) 2022-09-08T03:32:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:49 smithi055 ceph-mon[97033]: pgmap v11453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:49.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:49 smithi059 ceph-mon[97450]: pgmap v11453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:49.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:49.556654+0000) 2022-09-08T03:32:49.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:49.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:49.531775+0000) 2022-09-08T03:32:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:50.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:50.531910+0000) 2022-09-08T03:32:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:50.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:50.556770+0000) 2022-09-08T03:32:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:51 smithi055 ceph-mon[97033]: pgmap v11454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:51.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:51 smithi059 ceph-mon[97450]: pgmap v11454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:51.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:51.556882+0000) 2022-09-08T03:32:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:51.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:51.532045+0000) 2022-09-08T03:32:52.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:52.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:52.557021+0000) 2022-09-08T03:32:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:52.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:52.532184+0000) 2022-09-08T03:32:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:53 smithi055 ceph-mon[97033]: pgmap v11455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:32:53.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:53 smithi059 ceph-mon[97450]: pgmap v11455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:32:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:53.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:53.557158+0000) 2022-09-08T03:32:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:53.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:53.532324+0000) 2022-09-08T03:32:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:54.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:54.557292+0000) 2022-09-08T03:32:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:54.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:54.532456+0000) 2022-09-08T03:32:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:55 smithi055 ceph-mon[97033]: pgmap v11456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:55 smithi059 ceph-mon[97450]: pgmap v11456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:55.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:55.557423+0000) 2022-09-08T03:32:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:55.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:55.532589+0000) 2022-09-08T03:32:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:56.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:56.557559+0000) 2022-09-08T03:32:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:56.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:56.532732+0000) 2022-09-08T03:32:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:57 smithi055 ceph-mon[97033]: pgmap v11457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:32:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:57 smithi059 ceph-mon[97450]: pgmap v11457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:32:57.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:57.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:57.557694+0000) 2022-09-08T03:32:57.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:57.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:57.532845+0000) 2022-09-08T03:32:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:58.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:58.557809+0000) 2022-09-08T03:32:58.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:32:58.384+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:32:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:58.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:58.532961+0000) 2022-09-08T03:32:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:32:59 smithi055 ceph-mon[97033]: pgmap v11458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:32:59 smithi059 ceph-mon[97450]: pgmap v11458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:32:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:32:59.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:59.557939+0000) 2022-09-08T03:32:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:32:59.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:32:59.533085+0000) 2022-09-08T03:33:00.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:00.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:00.558076+0000) 2022-09-08T03:33:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:00.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:00.533224+0000) 2022-09-08T03:33:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:01 smithi055 ceph-mon[97033]: pgmap v11459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:01.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:01 smithi059 ceph-mon[97450]: pgmap v11459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:01.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:01.558222+0000) 2022-09-08T03:33:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:01.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:01.533356+0000) 2022-09-08T03:33:02.789 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:02.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:02.558377+0000) 2022-09-08T03:33:02.790 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:02.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:02.533485+0000) 2022-09-08T03:33:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:03 smithi055 ceph-mon[97033]: pgmap v11460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:33:03.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:03 smithi059 ceph-mon[97450]: pgmap v11460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:33:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:03.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:03.533627+0000) 2022-09-08T03:33:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:03.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:03.558515+0000) 2022-09-08T03:33:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:04.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:04.558663+0000) 2022-09-08T03:33:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:04.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:04.533778+0000) 2022-09-08T03:33:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:05 smithi055 ceph-mon[97033]: pgmap v11461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:33:05.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:05 smithi059 ceph-mon[97450]: pgmap v11461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:33:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:05.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:05.558798+0000) 2022-09-08T03:33:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:05.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:05.533911+0000) 2022-09-08T03:33:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:33:06.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:33:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:06.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:06.558938+0000) 2022-09-08T03:33:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:06.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:06.534047+0000) 2022-09-08T03:33:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:07 smithi055 ceph-mon[97033]: pgmap v11462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:33:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:07 smithi059 ceph-mon[97450]: pgmap v11462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:33:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:07.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:07.559150+0000) 2022-09-08T03:33:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:07.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:07.534280+0000) 2022-09-08T03:33:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:08.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:08.559303+0000) 2022-09-08T03:33:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:08.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:08.534446+0000) 2022-09-08T03:33:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:08 smithi059 ceph-mon[97450]: pgmap v11463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:33:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:08 smithi059 ceph-mon[97450]: pgmap v11464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T03:33:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:08 smithi059 ceph-mon[97450]: pgmap v11465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T03:33:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:33:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:08 smithi055 ceph-mon[97033]: pgmap v11463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:33:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:08 smithi055 ceph-mon[97033]: pgmap v11464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T03:33:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:08 smithi055 ceph-mon[97033]: pgmap v11465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T03:33:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:33:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:09.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:09.559434+0000) 2022-09-08T03:33:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:09.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:09.534583+0000) 2022-09-08T03:33:10.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:10 smithi059 ceph-mon[97450]: osdmap e795: 12 total, 11 up, 11 in 2022-09-08T03:33:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:10.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:10.559565+0000) 2022-09-08T03:33:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:10.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:10.534716+0000) 2022-09-08T03:33:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:10 smithi055 ceph-mon[97033]: osdmap e795: 12 total, 11 up, 11 in 2022-09-08T03:33:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:11 smithi059 ceph-mon[97450]: pgmap v11467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 19 KiB/s rd, 85 KiB/s wr, 33 op/s 2022-09-08T03:33:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:11.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:11.559706+0000) 2022-09-08T03:33:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:11.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:11.534851+0000) 2022-09-08T03:33:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:11 smithi055 ceph-mon[97033]: pgmap v11467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 19 KiB/s rd, 85 KiB/s wr, 33 op/s 2022-09-08T03:33:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:12.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:12.559875+0000) 2022-09-08T03:33:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:12.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:12.534969+0000) 2022-09-08T03:33:13.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:33:13.397+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:33:13.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:13.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:13.560017+0000) 2022-09-08T03:33:13.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:13.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:13.535108+0000) 2022-09-08T03:33:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:13 smithi059 ceph-mon[97450]: pgmap v11468: 129 pgs: 2 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 114 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T03:33:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:13 smithi055 ceph-mon[97033]: pgmap v11468: 129 pgs: 2 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 114 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T03:33:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:14.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:14.560155+0000) 2022-09-08T03:33:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:14.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:14.535231+0000) 2022-09-08T03:33:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:14 smithi059 ceph-mon[97450]: pgmap v11469: 129 pgs: 2 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 114 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T03:33:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:14 smithi055 ceph-mon[97033]: pgmap v11469: 129 pgs: 2 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 114 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T03:33:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:15.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:15.560300+0000) 2022-09-08T03:33:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:15.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:15.535364+0000) 2022-09-08T03:33:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:16.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:16.560442+0000) 2022-09-08T03:33:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:16.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:16.535505+0000) 2022-09-08T03:33:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:16 smithi055 ceph-mon[97033]: pgmap v11470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:33:17.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:16 smithi059 ceph-mon[97450]: pgmap v11470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:33:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:17.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:17.560578+0000) 2022-09-08T03:33:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:17.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:17.535660+0000) 2022-09-08T03:33:18.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:18.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:18.560698+0000) 2022-09-08T03:33:18.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:18.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:18.535831+0000) 2022-09-08T03:33:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:18 smithi055 ceph-mon[97033]: pgmap v11471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:33:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:18 smithi055 ceph-mon[97033]: osdmap e796: 12 total, 11 up, 11 in 2022-09-08T03:33:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:18 smithi059 ceph-mon[97450]: pgmap v11471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:33:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:18 smithi059 ceph-mon[97450]: osdmap e796: 12 total, 11 up, 11 in 2022-09-08T03:33:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:19.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:19.560844+0000) 2022-09-08T03:33:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:19.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:19.535974+0000) 2022-09-08T03:33:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:20.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:20.560957+0000) 2022-09-08T03:33:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:20.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:20.536115+0000) 2022-09-08T03:33:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:20 smithi055 ceph-mon[97033]: pgmap v11473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:33:21.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:20 smithi059 ceph-mon[97450]: pgmap v11473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:33:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:21.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:21.561104+0000) 2022-09-08T03:33:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:21.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:21.536258+0000) 2022-09-08T03:33:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:22.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:22.561238+0000) 2022-09-08T03:33:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:22.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:22.536398+0000) 2022-09-08T03:33:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:22 smithi055 ceph-mon[97033]: pgmap v11474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:33:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:22 smithi059 ceph-mon[97450]: pgmap v11474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:33:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:23.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:23.561374+0000) 2022-09-08T03:33:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:23.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:23.536541+0000) 2022-09-08T03:33:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:24.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:24.561510+0000) 2022-09-08T03:33:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:24.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:24.536692+0000) 2022-09-08T03:33:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:24 smithi059 ceph-mon[97450]: pgmap v11475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:33:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:24 smithi055 ceph-mon[97033]: pgmap v11475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:33:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:25.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:25.561649+0000) 2022-09-08T03:33:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:25.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:25.536832+0000) 2022-09-08T03:33:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:26.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:26.561800+0000) 2022-09-08T03:33:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:26.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:26.536947+0000) 2022-09-08T03:33:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:26 smithi055 ceph-mon[97033]: pgmap v11476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:33:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:26 smithi059 ceph-mon[97450]: pgmap v11476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:33:27.807 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:27.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:27.561943+0000) 2022-09-08T03:33:27.808 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:27.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:27.537087+0000) 2022-09-08T03:33:28.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:33:28.404+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:33:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:28.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:28.537230+0000) 2022-09-08T03:33:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:28.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:28.562076+0000) 2022-09-08T03:33:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:28 smithi055 ceph-mon[97033]: pgmap v11477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:33:29.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:28 smithi059 ceph-mon[97450]: pgmap v11477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:33:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:29.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:29.562204+0000) 2022-09-08T03:33:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:29.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:29.537369+0000) 2022-09-08T03:33:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:30.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:30.562339+0000) 2022-09-08T03:33:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:30.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:30.537500+0000) 2022-09-08T03:33:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:30 smithi055 ceph-mon[97033]: pgmap v11478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:33:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:30 smithi059 ceph-mon[97450]: pgmap v11478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T03:33:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:31.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:31.562472+0000) 2022-09-08T03:33:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:31.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:31.537651+0000) 2022-09-08T03:33:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:32.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:32.562614+0000) 2022-09-08T03:33:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:32.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:32.537789+0000) 2022-09-08T03:33:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:32 smithi055 ceph-mon[97033]: pgmap v11479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:33:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:32 smithi059 ceph-mon[97450]: pgmap v11479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:33:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:33.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:33.562753+0000) 2022-09-08T03:33:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:33.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:33.537931+0000) 2022-09-08T03:33:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:34.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:34.562891+0000) 2022-09-08T03:33:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:34.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:34.538059+0000) 2022-09-08T03:33:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:34 smithi059 ceph-mon[97450]: pgmap v11480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:34 smithi055 ceph-mon[97033]: pgmap v11480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:35.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:35.563034+0000) 2022-09-08T03:33:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:35.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:35.538200+0000) 2022-09-08T03:33:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:36.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:36.563188+0000) 2022-09-08T03:33:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:36.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:36.538351+0000) 2022-09-08T03:33:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:36 smithi055 ceph-mon[97033]: pgmap v11481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:33:37.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:36 smithi059 ceph-mon[97450]: pgmap v11481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:33:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:37.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:37.563332+0000) 2022-09-08T03:33:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:37.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:37.538490+0000) 2022-09-08T03:33:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:38.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:38.563475+0000) 2022-09-08T03:33:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:38.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:38.538645+0000) 2022-09-08T03:33:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:38 smithi055 ceph-mon[97033]: pgmap v11482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:39.269 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:38 smithi059 ceph-mon[97450]: pgmap v11482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:39.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:39.563613+0000) 2022-09-08T03:33:39.913 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:39.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:39.538800+0000) 2022-09-08T03:33:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:40.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:40.563744+0000) 2022-09-08T03:33:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:40.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:40.538934+0000) 2022-09-08T03:33:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:40 smithi059 ceph-mon[97450]: pgmap v11483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:40 smithi055 ceph-mon[97033]: pgmap v11483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:41.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:41.539076+0000) 2022-09-08T03:33:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:41.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:41.563879+0000) 2022-09-08T03:33:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:42.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:42.539205+0000) 2022-09-08T03:33:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:42.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:42.564017+0000) 2022-09-08T03:33:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:42 smithi055 ceph-mon[97033]: pgmap v11484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:33:43.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:42 smithi059 ceph-mon[97450]: pgmap v11484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:33:43.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:33:43.418+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:33:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:43.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:43.539345+0000) 2022-09-08T03:33:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:43.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:43.564160+0000) 2022-09-08T03:33:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:44.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:44.539483+0000) 2022-09-08T03:33:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:44.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:44.564310+0000) 2022-09-08T03:33:45.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:44 smithi059 ceph-mon[97450]: pgmap v11485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:44 smithi055 ceph-mon[97033]: pgmap v11485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:45.800 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:45.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:45.539621+0000) 2022-09-08T03:33:45.801 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:45.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:45.564431+0000) 2022-09-08T03:33:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:46.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:46.539750+0000) 2022-09-08T03:33:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:46.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:46.564573+0000) 2022-09-08T03:33:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:46 smithi055 ceph-mon[97033]: pgmap v11486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:33:47.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:46 smithi059 ceph-mon[97450]: pgmap v11486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:33:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:47.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:47.539887+0000) 2022-09-08T03:33:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:47.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:47.564722+0000) 2022-09-08T03:33:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:48.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:48.540029+0000) 2022-09-08T03:33:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:48.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:48.564831+0000) 2022-09-08T03:33:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:48 smithi055 ceph-mon[97033]: pgmap v11487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:49.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:48 smithi059 ceph-mon[97450]: pgmap v11487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:49.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:49.564946+0000) 2022-09-08T03:33:49.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:49.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:49.540163+0000) 2022-09-08T03:33:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:50.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:50.565076+0000) 2022-09-08T03:33:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:50.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:50.540296+0000) 2022-09-08T03:33:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:50 smithi055 ceph-mon[97033]: pgmap v11488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:50 smithi059 ceph-mon[97450]: pgmap v11488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:51.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:51.565200+0000) 2022-09-08T03:33:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:51.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:51.540408+0000) 2022-09-08T03:33:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:52.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:52.565337+0000) 2022-09-08T03:33:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:52.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:52.540540+0000) 2022-09-08T03:33:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:52 smithi055 ceph-mon[97033]: pgmap v11489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:33:53.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:52 smithi059 ceph-mon[97450]: pgmap v11489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:33:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:53.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:53.565450+0000) 2022-09-08T03:33:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:53.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:53.540685+0000) 2022-09-08T03:33:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:54.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:54.565581+0000) 2022-09-08T03:33:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:54.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:54.540817+0000) 2022-09-08T03:33:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:54 smithi059 ceph-mon[97450]: pgmap v11490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:54 smithi055 ceph-mon[97033]: pgmap v11490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:55.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:55.565689+0000) 2022-09-08T03:33:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:55.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:55.540956+0000) 2022-09-08T03:33:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:56.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:56.565822+0000) 2022-09-08T03:33:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:56.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:56.541086+0000) 2022-09-08T03:33:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:56 smithi055 ceph-mon[97033]: pgmap v11491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:33:57.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:56 smithi059 ceph-mon[97450]: pgmap v11491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:33:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:57.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:57.565949+0000) 2022-09-08T03:33:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:57.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:57.541223+0000) 2022-09-08T03:33:58.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:33:58.431+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:33:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:58.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:58.566079+0000) 2022-09-08T03:33:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:58.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:58.541355+0000) 2022-09-08T03:33:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:33:58 smithi055 ceph-mon[97033]: pgmap v11492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:59.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:33:58 smithi059 ceph-mon[97450]: pgmap v11492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:33:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:33:59.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:59.566226+0000) 2022-09-08T03:33:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:33:59.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:33:59.541492+0000) 2022-09-08T03:34:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:00.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:00.566361+0000) 2022-09-08T03:34:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:00.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:00.541628+0000) 2022-09-08T03:34:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:00 smithi055 ceph-mon[97033]: pgmap v11493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:01.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:00 smithi059 ceph-mon[97450]: pgmap v11493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:01.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:01.541776+0000) 2022-09-08T03:34:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:01.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:01.566493+0000) 2022-09-08T03:34:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:02.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:02.566663+0000) 2022-09-08T03:34:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:02.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:02.541906+0000) 2022-09-08T03:34:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:02 smithi055 ceph-mon[97033]: pgmap v11494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:34:03.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:02 smithi059 ceph-mon[97450]: pgmap v11494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:34:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:03.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:03.566789+0000) 2022-09-08T03:34:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:03.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:03.542021+0000) 2022-09-08T03:34:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:04.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:04.566937+0000) 2022-09-08T03:34:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:04.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:04.542154+0000) 2022-09-08T03:34:05.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:04 smithi059 ceph-mon[97450]: pgmap v11495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:04 smithi055 ceph-mon[97033]: pgmap v11495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:05.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:05.567079+0000) 2022-09-08T03:34:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:05.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:05.542288+0000) 2022-09-08T03:34:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:34:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:34:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:06.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:06.567214+0000) 2022-09-08T03:34:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:06.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:06.542416+0000) 2022-09-08T03:34:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:06 smithi055 ceph-mon[97033]: pgmap v11496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:34:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:34:07.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:06 smithi059 ceph-mon[97450]: pgmap v11496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:34:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:34:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:07.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:07.542550+0000) 2022-09-08T03:34:07.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:07.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:07.567340+0000) 2022-09-08T03:34:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:08.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:08.567473+0000) 2022-09-08T03:34:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:08.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:08.542700+0000) 2022-09-08T03:34:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:08 smithi055 ceph-mon[97033]: pgmap v11497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:34:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:34:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:34:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:34:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:34:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:34:09.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:08 smithi059 ceph-mon[97450]: pgmap v11497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:34:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:34:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:34:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:34:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:34:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:34:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:09.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:09.567603+0000) 2022-09-08T03:34:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:09.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:09.542818+0000) 2022-09-08T03:34:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:10 smithi055 ceph-mon[97033]: osdmap e797: 12 total, 11 up, 11 in 2022-09-08T03:34:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:10 smithi059 ceph-mon[97450]: osdmap e797: 12 total, 11 up, 11 in 2022-09-08T03:34:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:10.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:10.567725+0000) 2022-09-08T03:34:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:10.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:10.542931+0000) 2022-09-08T03:34:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:11 smithi055 ceph-mon[97033]: pgmap v11499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:34:11.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:11 smithi059 ceph-mon[97450]: pgmap v11499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:34:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:11.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:11.567854+0000) 2022-09-08T03:34:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:11.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:11.543128+0000) 2022-09-08T03:34:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:12.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:12.568011+0000) 2022-09-08T03:34:12.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:12.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:12.543278+0000) 2022-09-08T03:34:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:13 smithi055 ceph-mon[97033]: pgmap v11500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:34:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:13 smithi059 ceph-mon[97450]: pgmap v11500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:34:13.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:34:13.445+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:34:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:13.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:13.568158+0000) 2022-09-08T03:34:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:13.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:13.543413+0000) 2022-09-08T03:34:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:14.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:14.543544+0000) 2022-09-08T03:34:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:14.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:14.568297+0000) 2022-09-08T03:34:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:14 smithi059 ceph-mon[97450]: osdmap e798: 12 total, 11 up, 11 in 2022-09-08T03:34:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:14 smithi059 ceph-mon[97450]: pgmap v11502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:34:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:14 smithi055 ceph-mon[97033]: osdmap e798: 12 total, 11 up, 11 in 2022-09-08T03:34:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:14 smithi055 ceph-mon[97033]: pgmap v11502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:34:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:15.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:15.568434+0000) 2022-09-08T03:34:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:15.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:15.543692+0000) 2022-09-08T03:34:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:16.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:16.568575+0000) 2022-09-08T03:34:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:16.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:16.543824+0000) 2022-09-08T03:34:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:16 smithi055 ceph-mon[97033]: pgmap v11503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:34:17.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:16 smithi059 ceph-mon[97450]: pgmap v11503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:34:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:17.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:17.568697+0000) 2022-09-08T03:34:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:17.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:17.543960+0000) 2022-09-08T03:34:18.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:18.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:18.568839+0000) 2022-09-08T03:34:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:18.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:18.544096+0000) 2022-09-08T03:34:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:18 smithi055 ceph-mon[97033]: pgmap v11504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 243 B/s wr, 7 op/s 2022-09-08T03:34:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:18 smithi059 ceph-mon[97450]: pgmap v11504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 243 B/s wr, 7 op/s 2022-09-08T03:34:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:19.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:19.568980+0000) 2022-09-08T03:34:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:19.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:19.544243+0000) 2022-09-08T03:34:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:20.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:20.569113+0000) 2022-09-08T03:34:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:20.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:20.544375+0000) 2022-09-08T03:34:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:20 smithi055 ceph-mon[97033]: pgmap v11505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:34:21.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:20 smithi059 ceph-mon[97450]: pgmap v11505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:34:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:21.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:21.569255+0000) 2022-09-08T03:34:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:21.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:21.544514+0000) 2022-09-08T03:34:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:22.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:22.569394+0000) 2022-09-08T03:34:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:22.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:22.544658+0000) 2022-09-08T03:34:23.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:22 smithi055 ceph-mon[97033]: pgmap v11506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:34:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:22 smithi059 ceph-mon[97450]: pgmap v11506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:34:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:23.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:23.569508+0000) 2022-09-08T03:34:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:23.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:23.544775+0000) 2022-09-08T03:34:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:24.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:24.569655+0000) 2022-09-08T03:34:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:24.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:24.544917+0000) 2022-09-08T03:34:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:24 smithi059 ceph-mon[97450]: pgmap v11507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:34:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:24 smithi055 ceph-mon[97033]: pgmap v11507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:34:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:25.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:25.569786+0000) 2022-09-08T03:34:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:25.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:25.545054+0000) 2022-09-08T03:34:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:26.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:26.569924+0000) 2022-09-08T03:34:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:26.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:26.545184+0000) 2022-09-08T03:34:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:26 smithi055 ceph-mon[97033]: pgmap v11508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:34:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:26 smithi059 ceph-mon[97450]: pgmap v11508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:34:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:27.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:27.570055+0000) 2022-09-08T03:34:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:27.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:27.545320+0000) 2022-09-08T03:34:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:28.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:28.570194+0000) 2022-09-08T03:34:28.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:34:28.460+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:34:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:28.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:28.545456+0000) 2022-09-08T03:34:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:28 smithi055 ceph-mon[97033]: pgmap v11509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:29.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:28 smithi059 ceph-mon[97450]: pgmap v11509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:29.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:29.570335+0000) 2022-09-08T03:34:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:29.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:29.545591+0000) 2022-09-08T03:34:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:30.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:30.570485+0000) 2022-09-08T03:34:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:30.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:30.545701+0000) 2022-09-08T03:34:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:30 smithi055 ceph-mon[97033]: pgmap v11510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:30 smithi059 ceph-mon[97450]: pgmap v11510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:31.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:31.570633+0000) 2022-09-08T03:34:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:31.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:31.545824+0000) 2022-09-08T03:34:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:32.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:32.570794+0000) 2022-09-08T03:34:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:32.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:32.545958+0000) 2022-09-08T03:34:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:32 smithi055 ceph-mon[97033]: pgmap v11511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:34:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:32 smithi059 ceph-mon[97450]: pgmap v11511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:34:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:33.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:33.570942+0000) 2022-09-08T03:34:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:33.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:33.546094+0000) 2022-09-08T03:34:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:34.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:34.571084+0000) 2022-09-08T03:34:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:34.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:34.546222+0000) 2022-09-08T03:34:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:34 smithi059 ceph-mon[97450]: pgmap v11512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:34 smithi055 ceph-mon[97033]: pgmap v11512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:35.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:35.571223+0000) 2022-09-08T03:34:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:35.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:35.546355+0000) 2022-09-08T03:34:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:36.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:36.571374+0000) 2022-09-08T03:34:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:36.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:36.546490+0000) 2022-09-08T03:34:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:36 smithi055 ceph-mon[97033]: pgmap v11513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:34:37.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:36 smithi059 ceph-mon[97450]: pgmap v11513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:34:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:37.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:37.571522+0000) 2022-09-08T03:34:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:37.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:37.546629+0000) 2022-09-08T03:34:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:38.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:38.571681+0000) 2022-09-08T03:34:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:38.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:38.546775+0000) 2022-09-08T03:34:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:39 smithi055 ceph-mon[97033]: pgmap v11514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:39 smithi055 ceph-mon[97033]: pgmap v11515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:34:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:39 smithi055 ceph-mon[97033]: pgmap v11516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:34:39.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:34:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:39 smithi059 ceph-mon[97450]: pgmap v11514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:39 smithi059 ceph-mon[97450]: pgmap v11515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:34:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:39 smithi059 ceph-mon[97450]: pgmap v11516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:34:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:34:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:39.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:39.571828+0000) 2022-09-08T03:34:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:39.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:39.546921+0000) 2022-09-08T03:34:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:40.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:40.547061+0000) 2022-09-08T03:34:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:40.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:40.571958+0000) 2022-09-08T03:34:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:41 smithi055 ceph-mon[97033]: pgmap v11517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:34:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:41 smithi059 ceph-mon[97450]: pgmap v11517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:34:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:41.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:41.547196+0000) 2022-09-08T03:34:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:41.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:41.572089+0000) 2022-09-08T03:34:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:42.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:42.547347+0000) 2022-09-08T03:34:42.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:42.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:42.572220+0000) 2022-09-08T03:34:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:43 smithi055 ceph-mon[97033]: pgmap v11518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:34:43.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:43 smithi059 ceph-mon[97450]: pgmap v11518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:34:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:43.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:43.547488+0000) 2022-09-08T03:34:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:43.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:43.572331+0000) 2022-09-08T03:34:43.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:34:43.465+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:34:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:44.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:44.572564+0000) 2022-09-08T03:34:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:44.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:44.547746+0000) 2022-09-08T03:34:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:44 smithi055 ceph-mon[97033]: pgmap v11519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:34:45.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:44 smithi059 ceph-mon[97450]: pgmap v11519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:34:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:45.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:45.572740+0000) 2022-09-08T03:34:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:45.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:45.547902+0000) 2022-09-08T03:34:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:46.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:46.548031+0000) 2022-09-08T03:34:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:46.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:46.572955+0000) 2022-09-08T03:34:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:47 smithi055 ceph-mon[97033]: pgmap v11520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:34:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:47 smithi059 ceph-mon[97450]: pgmap v11520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:34:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:47.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:47.548168+0000) 2022-09-08T03:34:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:47.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:47.573116+0000) 2022-09-08T03:34:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:48.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:48.573267+0000) 2022-09-08T03:34:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:48.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:48.548319+0000) 2022-09-08T03:34:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:49 smithi055 ceph-mon[97033]: pgmap v11521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:34:49.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:49 smithi059 ceph-mon[97450]: pgmap v11521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:34:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:49.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:49.573418+0000) 2022-09-08T03:34:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:49.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:49.548461+0000) 2022-09-08T03:34:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:50.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:50.548599+0000) 2022-09-08T03:34:50.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:50.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:50.573571+0000) 2022-09-08T03:34:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:51 smithi055 ceph-mon[97033]: pgmap v11522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:34:51.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:51 smithi059 ceph-mon[97450]: pgmap v11522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:34:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:51.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:51.573724+0000) 2022-09-08T03:34:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:51.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:51.548710+0000) 2022-09-08T03:34:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:52.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:52.573886+0000) 2022-09-08T03:34:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:52.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:52.548851+0000) 2022-09-08T03:34:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:53 smithi055 ceph-mon[97033]: pgmap v11523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:53.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:53 smithi059 ceph-mon[97450]: pgmap v11523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:53.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:53.574023+0000) 2022-09-08T03:34:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:53.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:53.548990+0000) 2022-09-08T03:34:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:54.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:54.549126+0000) 2022-09-08T03:34:54.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:54.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:54.574159+0000) 2022-09-08T03:34:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:54 smithi055 ceph-mon[97033]: pgmap v11524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:55.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:54 smithi059 ceph-mon[97450]: pgmap v11524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:55.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:55.549263+0000) 2022-09-08T03:34:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:55.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:55.574301+0000) 2022-09-08T03:34:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:56.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:56.574438+0000) 2022-09-08T03:34:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:56.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:56.549398+0000) 2022-09-08T03:34:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:57 smithi055 ceph-mon[97033]: pgmap v11525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:34:57.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:57 smithi059 ceph-mon[97450]: pgmap v11525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:34:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:57.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:57.574574+0000) 2022-09-08T03:34:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:57.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:57.549537+0000) 2022-09-08T03:34:58.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:34:58.472+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:34:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:58.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:58.574719+0000) 2022-09-08T03:34:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:58.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:58.549694+0000) 2022-09-08T03:34:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:34:59 smithi055 ceph-mon[97033]: pgmap v11526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:59.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:34:59 smithi059 ceph-mon[97450]: pgmap v11526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:34:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:34:59.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:59.574866+0000) 2022-09-08T03:34:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:34:59.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:34:59.549818+0000) 2022-09-08T03:35:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:00.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:00.575012+0000) 2022-09-08T03:35:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:00.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:00.549941+0000) 2022-09-08T03:35:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:01 smithi055 ceph-mon[97033]: pgmap v11527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:35:01.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:01 smithi059 ceph-mon[97450]: pgmap v11527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:35:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:01.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:01.575134+0000) 2022-09-08T03:35:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:01.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:01.550073+0000) 2022-09-08T03:35:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:02.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:02.575294+0000) 2022-09-08T03:35:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:02.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:02.550213+0000) 2022-09-08T03:35:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:03 smithi055 ceph-mon[97033]: pgmap v11528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:03.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:03 smithi059 ceph-mon[97450]: pgmap v11528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:03.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:03.575441+0000) 2022-09-08T03:35:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:03.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:03.550357+0000) 2022-09-08T03:35:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:04.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:04.550475+0000) 2022-09-08T03:35:04.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:04.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:04.575576+0000) 2022-09-08T03:35:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:04 smithi055 ceph-mon[97033]: pgmap v11529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:05.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:04 smithi059 ceph-mon[97450]: pgmap v11529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:05.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:05.550618+0000) 2022-09-08T03:35:05.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:05.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:05.575710+0000) 2022-09-08T03:35:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:35:06.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:35:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:06.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:06.575821+0000) 2022-09-08T03:35:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:06.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:06.550707+0000) 2022-09-08T03:35:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:35:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:06 smithi055 ceph-mon[97033]: pgmap v11530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:35:07.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:35:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:06 smithi059 ceph-mon[97450]: pgmap v11530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:35:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:07.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:07.550831+0000) 2022-09-08T03:35:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:07.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:07.575981+0000) 2022-09-08T03:35:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:08.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:08.576107+0000) 2022-09-08T03:35:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:08.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:08.550960+0000) 2022-09-08T03:35:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:09 smithi055 ceph-mon[97033]: pgmap v11531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:35:09.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:09 smithi059 ceph-mon[97450]: pgmap v11531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:35:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:09.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:09.576236+0000) 2022-09-08T03:35:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:09.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:09.551083+0000) 2022-09-08T03:35:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:10 smithi055 ceph-mon[97033]: osdmap e799: 12 total, 11 up, 11 in 2022-09-08T03:35:10.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:10 smithi059 ceph-mon[97450]: osdmap e799: 12 total, 11 up, 11 in 2022-09-08T03:35:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:10.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:10.576369+0000) 2022-09-08T03:35:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:10.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:10.551222+0000) 2022-09-08T03:35:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:11 smithi055 ceph-mon[97033]: pgmap v11533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:35:11.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:11 smithi059 ceph-mon[97450]: pgmap v11533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:35:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:11.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:11.551360+0000) 2022-09-08T03:35:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:11.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:11.576501+0000) 2022-09-08T03:35:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:12.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:12.551510+0000) 2022-09-08T03:35:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:12.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:12.576658+0000) 2022-09-08T03:35:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:13 smithi055 ceph-mon[97033]: pgmap v11534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:35:13.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:13 smithi059 ceph-mon[97450]: pgmap v11534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:35:13.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:35:13.487+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:35:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:13.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:13.551666+0000) 2022-09-08T03:35:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:13.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:13.576794+0000) 2022-09-08T03:35:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:14.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:14.551783+0000) 2022-09-08T03:35:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:14.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:14.576908+0000) 2022-09-08T03:35:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:14 smithi055 ceph-mon[97033]: osdmap e800: 12 total, 11 up, 11 in 2022-09-08T03:35:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:14 smithi055 ceph-mon[97033]: pgmap v11536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:35:15.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:14 smithi059 ceph-mon[97450]: osdmap e800: 12 total, 11 up, 11 in 2022-09-08T03:35:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:14 smithi059 ceph-mon[97450]: pgmap v11536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:35:16.293 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:15.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:15.552002+0000) 2022-09-08T03:35:16.294 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:15.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:15.577023+0000) 2022-09-08T03:35:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:16.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:16.552172+0000) 2022-09-08T03:35:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:16.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:16.577148+0000) 2022-09-08T03:35:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:17 smithi055 ceph-mon[97033]: pgmap v11537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:35:17.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:17 smithi059 ceph-mon[97450]: pgmap v11537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:35:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:17.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:17.577277+0000) 2022-09-08T03:35:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:17.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:17.552316+0000) 2022-09-08T03:35:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:18.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:18.577409+0000) 2022-09-08T03:35:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:18.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:18.552463+0000) 2022-09-08T03:35:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:19 smithi055 ceph-mon[97033]: pgmap v11538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T03:35:19.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:19 smithi059 ceph-mon[97450]: pgmap v11538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T03:35:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:19.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:19.577536+0000) 2022-09-08T03:35:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:19.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:19.552606+0000) 2022-09-08T03:35:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:20.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:20.577680+0000) 2022-09-08T03:35:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:20.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:20.552711+0000) 2022-09-08T03:35:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:21 smithi055 ceph-mon[97033]: pgmap v11539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:35:21.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:21 smithi059 ceph-mon[97450]: pgmap v11539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:35:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:21.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:21.577810+0000) 2022-09-08T03:35:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:21.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:21.552831+0000) 2022-09-08T03:35:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:22.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:22.577926+0000) 2022-09-08T03:35:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:22.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:22.552951+0000) 2022-09-08T03:35:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:23 smithi055 ceph-mon[97033]: pgmap v11540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:35:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:23 smithi059 ceph-mon[97450]: pgmap v11540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:35:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:23.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:23.578051+0000) 2022-09-08T03:35:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:23.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:23.553088+0000) 2022-09-08T03:35:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:24.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:24.553229+0000) 2022-09-08T03:35:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:24.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:24.578179+0000) 2022-09-08T03:35:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:24 smithi055 ceph-mon[97033]: pgmap v11541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-08T03:35:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:24 smithi059 ceph-mon[97450]: pgmap v11541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-08T03:35:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:25.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:25.553347+0000) 2022-09-08T03:35:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:25.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:25.578303+0000) 2022-09-08T03:35:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:26.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:26.553479+0000) 2022-09-08T03:35:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:26.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:26.578433+0000) 2022-09-08T03:35:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:27 smithi055 ceph-mon[97033]: pgmap v11542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:35:27.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:27 smithi059 ceph-mon[97450]: pgmap v11542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:35:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:27.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:27.578562+0000) 2022-09-08T03:35:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:27.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:27.553600+0000) 2022-09-08T03:35:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:28.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:28.553716+0000) 2022-09-08T03:35:28.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:35:28.498+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:35:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:28.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:28.578693+0000) 2022-09-08T03:35:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:29 smithi055 ceph-mon[97033]: pgmap v11543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:29.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:29 smithi059 ceph-mon[97450]: pgmap v11543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:29.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:29.578822+0000) 2022-09-08T03:35:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:29.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:29.553835+0000) 2022-09-08T03:35:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:30.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:30.578939+0000) 2022-09-08T03:35:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:30.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:30.553948+0000) 2022-09-08T03:35:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:31 smithi055 ceph-mon[97033]: pgmap v11544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:35:31.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:31 smithi059 ceph-mon[97450]: pgmap v11544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:35:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:31.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:31.579061+0000) 2022-09-08T03:35:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:31.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:31.554081+0000) 2022-09-08T03:35:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:32.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:32.579190+0000) 2022-09-08T03:35:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:32.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:32.554204+0000) 2022-09-08T03:35:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:33 smithi055 ceph-mon[97033]: pgmap v11545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:33.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:33 smithi059 ceph-mon[97450]: pgmap v11545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:33.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:33.579333+0000) 2022-09-08T03:35:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:33.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:33.554347+0000) 2022-09-08T03:35:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:34.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:34.579470+0000) 2022-09-08T03:35:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:34.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:34.554487+0000) 2022-09-08T03:35:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:34 smithi055 ceph-mon[97033]: pgmap v11546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:35.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:34 smithi059 ceph-mon[97450]: pgmap v11546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:35.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:35.579606+0000) 2022-09-08T03:35:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:35.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:35.554630+0000) 2022-09-08T03:35:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:36.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:36.579707+0000) 2022-09-08T03:35:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:36.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:36.554753+0000) 2022-09-08T03:35:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:37 smithi055 ceph-mon[97033]: pgmap v11547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:35:37.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:37 smithi059 ceph-mon[97450]: pgmap v11547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:35:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:37.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:37.579887+0000) 2022-09-08T03:35:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:37.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:37.554895+0000) 2022-09-08T03:35:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:38.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:38.580020+0000) 2022-09-08T03:35:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:38.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:38.555049+0000) 2022-09-08T03:35:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:39 smithi055 ceph-mon[97033]: pgmap v11548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:35:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:35:39.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:35:39.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:39 smithi059 ceph-mon[97450]: pgmap v11548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:35:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:35:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:35:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:39.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:39.580130+0000) 2022-09-08T03:35:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:39.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:39.555194+0000) 2022-09-08T03:35:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:40.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:40.580259+0000) 2022-09-08T03:35:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:40.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:40.555336+0000) 2022-09-08T03:35:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:41 smithi055 ceph-mon[97033]: pgmap v11549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:35:41.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:41 smithi059 ceph-mon[97450]: pgmap v11549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:35:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:41.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:41.580400+0000) 2022-09-08T03:35:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:41.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:41.555475+0000) 2022-09-08T03:35:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:42.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:42.555616+0000) 2022-09-08T03:35:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:42.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:42.580549+0000) 2022-09-08T03:35:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:43 smithi055 ceph-mon[97033]: pgmap v11550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:43.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:43 smithi059 ceph-mon[97450]: pgmap v11550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:43.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:35:43.513+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:35:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:43.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:43.555768+0000) 2022-09-08T03:35:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:43.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:43.580689+0000) 2022-09-08T03:35:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:44.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:44.555909+0000) 2022-09-08T03:35:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:44.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:44.580823+0000) 2022-09-08T03:35:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:44 smithi055 ceph-mon[97033]: pgmap v11551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:45.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:44 smithi059 ceph-mon[97450]: pgmap v11551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:45.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:45.556049+0000) 2022-09-08T03:35:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:45.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:45.580956+0000) 2022-09-08T03:35:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:46.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:46.581089+0000) 2022-09-08T03:35:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:46.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:46.556175+0000) 2022-09-08T03:35:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:47 smithi055 ceph-mon[97033]: pgmap v11552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:35:47.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:47 smithi059 ceph-mon[97450]: pgmap v11552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:35:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:47.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:47.581224+0000) 2022-09-08T03:35:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:47.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:47.556322+0000) 2022-09-08T03:35:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:48.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:48.581377+0000) 2022-09-08T03:35:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:48.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:48.556487+0000) 2022-09-08T03:35:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:49 smithi055 ceph-mon[97033]: pgmap v11553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:49.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:49 smithi059 ceph-mon[97450]: pgmap v11553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:49.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:49.581507+0000) 2022-09-08T03:35:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:49.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:49.556628+0000) 2022-09-08T03:35:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:50.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:50.581650+0000) 2022-09-08T03:35:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:50.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:50.556776+0000) 2022-09-08T03:35:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:51 smithi055 ceph-mon[97033]: pgmap v11554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:35:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:51 smithi059 ceph-mon[97450]: pgmap v11554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:35:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:51.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:51.581783+0000) 2022-09-08T03:35:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:51.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:51.556896+0000) 2022-09-08T03:35:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:52.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:52.581924+0000) 2022-09-08T03:35:52.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:52.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:52.557037+0000) 2022-09-08T03:35:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:53 smithi055 ceph-mon[97033]: pgmap v11555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:53.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:53 smithi059 ceph-mon[97450]: pgmap v11555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:53.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:53.582059+0000) 2022-09-08T03:35:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:53.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:53.557180+0000) 2022-09-08T03:35:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:54.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:54.582198+0000) 2022-09-08T03:35:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:54.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:54.557309+0000) 2022-09-08T03:35:55.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:54 smithi055 ceph-mon[97033]: pgmap v11556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:55.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:54 smithi059 ceph-mon[97450]: pgmap v11556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:55.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:55.582333+0000) 2022-09-08T03:35:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:55.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:55.557457+0000) 2022-09-08T03:35:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:56.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:56.582501+0000) 2022-09-08T03:35:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:56.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:56.557592+0000) 2022-09-08T03:35:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:57 smithi055 ceph-mon[97033]: pgmap v11557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:35:57.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:57 smithi059 ceph-mon[97450]: pgmap v11557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:35:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:57.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:57.582632+0000) 2022-09-08T03:35:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:57.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:57.557740+0000) 2022-09-08T03:35:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:58.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:58.582792+0000) 2022-09-08T03:35:58.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:35:58.528+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:35:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:58.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:58.557887+0000) 2022-09-08T03:35:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:35:59 smithi055 ceph-mon[97033]: pgmap v11558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:59.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:35:59 smithi059 ceph-mon[97450]: pgmap v11558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:35:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:35:59.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:59.582921+0000) 2022-09-08T03:35:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:35:59.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:35:59.558034+0000) 2022-09-08T03:36:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:00.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:00.583057+0000) 2022-09-08T03:36:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:00.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:00.558148+0000) 2022-09-08T03:36:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:01 smithi055 ceph-mon[97033]: pgmap v11559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:36:01.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:01 smithi059 ceph-mon[97450]: pgmap v11559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:36:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:01.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:01.558289+0000) 2022-09-08T03:36:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:01.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:01.583194+0000) 2022-09-08T03:36:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:02.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:02.583345+0000) 2022-09-08T03:36:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:02.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:02.558420+0000) 2022-09-08T03:36:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:03 smithi055 ceph-mon[97033]: pgmap v11560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:03.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:03 smithi059 ceph-mon[97450]: pgmap v11560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:03.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:03.583506+0000) 2022-09-08T03:36:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:03.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:03.558538+0000) 2022-09-08T03:36:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:04.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:04.583645+0000) 2022-09-08T03:36:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:04.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:04.558679+0000) 2022-09-08T03:36:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:04 smithi055 ceph-mon[97033]: pgmap v11561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:05.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:04 smithi059 ceph-mon[97450]: pgmap v11561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:05.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:05.583757+0000) 2022-09-08T03:36:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:05.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:05.558816+0000) 2022-09-08T03:36:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:36:06.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:36:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:06.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:06.583889+0000) 2022-09-08T03:36:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:06.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:06.558949+0000) 2022-09-08T03:36:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:36:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:06 smithi055 ceph-mon[97033]: pgmap v11562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:36:07.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:36:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:06 smithi059 ceph-mon[97450]: pgmap v11562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:36:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:07.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:07.584022+0000) 2022-09-08T03:36:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:07.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:07.559096+0000) 2022-09-08T03:36:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:08.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:08.584156+0000) 2022-09-08T03:36:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:08.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:08.559243+0000) 2022-09-08T03:36:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:09 smithi055 ceph-mon[97033]: pgmap v11563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:36:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:09 smithi055 ceph-mon[97033]: pgmap v11564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T03:36:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:09 smithi055 ceph-mon[97033]: pgmap v11565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 65 KiB/s wr, 25 op/s 2022-09-08T03:36:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:36:09.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:09 smithi059 ceph-mon[97450]: pgmap v11563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:36:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:09 smithi059 ceph-mon[97450]: pgmap v11564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T03:36:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:09 smithi059 ceph-mon[97450]: pgmap v11565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 65 KiB/s wr, 25 op/s 2022-09-08T03:36:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:36:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:09.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:09.559384+0000) 2022-09-08T03:36:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:09.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:09.584293+0000) 2022-09-08T03:36:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:10 smithi055 ceph-mon[97033]: osdmap e801: 12 total, 11 up, 11 in 2022-09-08T03:36:10.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:10 smithi059 ceph-mon[97450]: osdmap e801: 12 total, 11 up, 11 in 2022-09-08T03:36:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:10.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:10.559518+0000) 2022-09-08T03:36:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:10.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:10.584432+0000) 2022-09-08T03:36:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:11 smithi055 ceph-mon[97033]: pgmap v11567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T03:36:11.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:11 smithi059 ceph-mon[97450]: pgmap v11567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T03:36:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:11.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:11.559668+0000) 2022-09-08T03:36:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:11.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:11.584563+0000) 2022-09-08T03:36:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:12.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:12.559803+0000) 2022-09-08T03:36:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:12.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:12.584717+0000) 2022-09-08T03:36:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:13 smithi055 ceph-mon[97033]: pgmap v11568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T03:36:13.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:13 smithi059 ceph-mon[97450]: pgmap v11568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T03:36:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:13.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:13.584857+0000) 2022-09-08T03:36:13.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:36:13.542+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:36:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:13.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:13.559945+0000) 2022-09-08T03:36:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:14.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:14.585011+0000) 2022-09-08T03:36:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:14.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:14.560082+0000) 2022-09-08T03:36:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:14 smithi055 ceph-mon[97033]: osdmap e802: 12 total, 11 up, 11 in 2022-09-08T03:36:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:14 smithi055 ceph-mon[97033]: pgmap v11570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2022-09-08T03:36:15.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:14 smithi059 ceph-mon[97450]: osdmap e802: 12 total, 11 up, 11 in 2022-09-08T03:36:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:14 smithi059 ceph-mon[97450]: pgmap v11570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2022-09-08T03:36:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:15.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:15.585154+0000) 2022-09-08T03:36:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:15.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:15.560218+0000) 2022-09-08T03:36:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:16.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:16.585279+0000) 2022-09-08T03:36:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:16.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:16.560355+0000) 2022-09-08T03:36:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:17 smithi055 ceph-mon[97033]: pgmap v11571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:36:17.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:17 smithi059 ceph-mon[97450]: pgmap v11571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:36:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:17.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:17.585410+0000) 2022-09-08T03:36:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:17.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:17.560494+0000) 2022-09-08T03:36:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:18.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:18.585555+0000) 2022-09-08T03:36:18.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:18.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:18.560644+0000) 2022-09-08T03:36:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:19 smithi055 ceph-mon[97033]: pgmap v11572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T03:36:19.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:19 smithi059 ceph-mon[97450]: pgmap v11572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T03:36:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:19.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:19.585708+0000) 2022-09-08T03:36:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:19.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:19.560788+0000) 2022-09-08T03:36:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:20.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:20.585853+0000) 2022-09-08T03:36:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:20.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:20.560923+0000) 2022-09-08T03:36:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:21 smithi055 ceph-mon[97033]: pgmap v11573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:36:21.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:21 smithi059 ceph-mon[97450]: pgmap v11573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:36:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:21.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:21.585976+0000) 2022-09-08T03:36:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:21.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:21.561061+0000) 2022-09-08T03:36:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:22.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:22.586119+0000) 2022-09-08T03:36:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:22.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:22.561188+0000) 2022-09-08T03:36:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:23 smithi055 ceph-mon[97033]: pgmap v11574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:36:23.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:23 smithi059 ceph-mon[97450]: pgmap v11574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:36:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:23.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:23.586245+0000) 2022-09-08T03:36:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:23.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:23.561330+0000) 2022-09-08T03:36:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:24.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:24.586380+0000) 2022-09-08T03:36:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:24.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:24.561465+0000) 2022-09-08T03:36:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:24 smithi055 ceph-mon[97033]: pgmap v11575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2022-09-08T03:36:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:24 smithi059 ceph-mon[97450]: pgmap v11575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2022-09-08T03:36:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:25.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:25.586521+0000) 2022-09-08T03:36:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:25.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:25.561602+0000) 2022-09-08T03:36:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:26.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:26.586667+0000) 2022-09-08T03:36:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:26.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:26.561739+0000) 2022-09-08T03:36:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:27 smithi055 ceph-mon[97033]: pgmap v11576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:36:27.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:27 smithi059 ceph-mon[97450]: pgmap v11576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:36:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:27.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:27.586811+0000) 2022-09-08T03:36:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:27.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:27.561875+0000) 2022-09-08T03:36:28.586 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:36:28.547+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:36:28.587 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:28.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:28.562017+0000) 2022-09-08T03:36:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:28.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:28.586940+0000) 2022-09-08T03:36:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:29 smithi055 ceph-mon[97033]: pgmap v11577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:29.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:29 smithi059 ceph-mon[97450]: pgmap v11577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:29.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:29.587058+0000) 2022-09-08T03:36:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:29.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:29.562159+0000) 2022-09-08T03:36:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:30.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:30.587199+0000) 2022-09-08T03:36:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:30.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:30.562291+0000) 2022-09-08T03:36:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:31 smithi055 ceph-mon[97033]: pgmap v11578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:36:31.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:31 smithi059 ceph-mon[97450]: pgmap v11578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:36:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:31.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:31.587347+0000) 2022-09-08T03:36:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:31.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:31.562423+0000) 2022-09-08T03:36:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:32.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:32.587497+0000) 2022-09-08T03:36:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:32.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:32.562568+0000) 2022-09-08T03:36:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:33 smithi055 ceph-mon[97033]: pgmap v11579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:33.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:33 smithi059 ceph-mon[97450]: pgmap v11579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:33.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:33.587653+0000) 2022-09-08T03:36:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:33.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:33.562707+0000) 2022-09-08T03:36:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:34.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:34.587806+0000) 2022-09-08T03:36:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:34.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:34.562840+0000) 2022-09-08T03:36:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:34 smithi055 ceph-mon[97033]: pgmap v11580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:35.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:34 smithi059 ceph-mon[97450]: pgmap v11580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:35.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:35.587943+0000) 2022-09-08T03:36:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:35.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:35.562976+0000) 2022-09-08T03:36:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:36.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:36.588086+0000) 2022-09-08T03:36:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:36.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:36.563109+0000) 2022-09-08T03:36:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:37 smithi055 ceph-mon[97033]: pgmap v11581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:36:37.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:37 smithi059 ceph-mon[97450]: pgmap v11581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:36:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:37.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:37.588227+0000) 2022-09-08T03:36:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:37.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:37.563245+0000) 2022-09-08T03:36:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:38.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:38.588362+0000) 2022-09-08T03:36:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:38.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:38.563392+0000) 2022-09-08T03:36:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:39 smithi055 ceph-mon[97033]: pgmap v11582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:39.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:39 smithi059 ceph-mon[97450]: pgmap v11582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:39.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:39.588501+0000) 2022-09-08T03:36:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:39.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:39.563529+0000) 2022-09-08T03:36:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:40.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:40.588647+0000) 2022-09-08T03:36:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:40.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:40.563678+0000) 2022-09-08T03:36:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:41 smithi055 ceph-mon[97033]: pgmap v11583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:36:41.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:41 smithi059 ceph-mon[97450]: pgmap v11583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:36:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:41.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:41.588780+0000) 2022-09-08T03:36:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:41.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:41.563798+0000) 2022-09-08T03:36:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:42.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:42.588920+0000) 2022-09-08T03:36:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:42.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:42.563913+0000) 2022-09-08T03:36:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:43 smithi055 ceph-mon[97033]: pgmap v11584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:43.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:43 smithi059 ceph-mon[97450]: pgmap v11584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:43.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:43.589055+0000) 2022-09-08T03:36:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:43.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:43.564037+0000) 2022-09-08T03:36:43.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:36:43.562+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:36:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:44.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:44.589185+0000) 2022-09-08T03:36:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:44.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:44.564171+0000) 2022-09-08T03:36:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:44 smithi055 ceph-mon[97033]: pgmap v11585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:45.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:44 smithi059 ceph-mon[97450]: pgmap v11585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:45.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:45.589314+0000) 2022-09-08T03:36:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:45.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:45.564307+0000) 2022-09-08T03:36:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:46.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:46.589434+0000) 2022-09-08T03:36:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:46.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:46.564437+0000) 2022-09-08T03:36:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:47 smithi055 ceph-mon[97033]: pgmap v11586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:36:47.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:47 smithi059 ceph-mon[97450]: pgmap v11586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:36:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:47.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:47.589563+0000) 2022-09-08T03:36:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:47.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:47.564573+0000) 2022-09-08T03:36:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:48.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:48.589689+0000) 2022-09-08T03:36:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:48.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:48.564715+0000) 2022-09-08T03:36:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:49 smithi055 ceph-mon[97033]: pgmap v11587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:49.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:49 smithi059 ceph-mon[97450]: pgmap v11587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:49.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:49.589830+0000) 2022-09-08T03:36:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:49.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:49.564832+0000) 2022-09-08T03:36:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:50.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:50.589972+0000) 2022-09-08T03:36:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:50.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:50.564965+0000) 2022-09-08T03:36:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:51 smithi055 ceph-mon[97033]: pgmap v11588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:36:51.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:51 smithi059 ceph-mon[97450]: pgmap v11588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:36:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:51.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:51.590113+0000) 2022-09-08T03:36:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:51.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:51.565094+0000) 2022-09-08T03:36:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:52.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:52.590361+0000) 2022-09-08T03:36:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:52.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:52.565241+0000) 2022-09-08T03:36:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:53 smithi055 ceph-mon[97033]: pgmap v11589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:53.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:53 smithi059 ceph-mon[97450]: pgmap v11589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:53.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:53.590503+0000) 2022-09-08T03:36:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:53.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:53.565372+0000) 2022-09-08T03:36:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:54.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:54.590653+0000) 2022-09-08T03:36:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:54.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:54.565513+0000) 2022-09-08T03:36:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:54 smithi055 ceph-mon[97033]: pgmap v11590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:55.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:54 smithi059 ceph-mon[97450]: pgmap v11590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:55.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:55.590771+0000) 2022-09-08T03:36:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:55.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:55.565661+0000) 2022-09-08T03:36:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:56.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:56.590906+0000) 2022-09-08T03:36:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:56.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:56.565787+0000) 2022-09-08T03:36:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:57 smithi055 ceph-mon[97033]: pgmap v11591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:36:57.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:57 smithi059 ceph-mon[97450]: pgmap v11591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:36:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:57.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:57.591037+0000) 2022-09-08T03:36:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:57.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:57.565923+0000) 2022-09-08T03:36:58.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:36:58.571+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:36:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:58.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:58.591168+0000) 2022-09-08T03:36:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:58.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:58.566060+0000) 2022-09-08T03:36:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:36:59 smithi055 ceph-mon[97033]: pgmap v11592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:36:59 smithi059 ceph-mon[97450]: pgmap v11592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:36:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:36:59.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:59.591300+0000) 2022-09-08T03:36:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:36:59.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:36:59.566190+0000) 2022-09-08T03:37:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:00.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:00.591511+0000) 2022-09-08T03:37:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:00.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:00.566319+0000) 2022-09-08T03:37:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:01 smithi055 ceph-mon[97033]: pgmap v11593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:37:01.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:01 smithi059 ceph-mon[97450]: pgmap v11593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:37:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:01.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:01.591653+0000) 2022-09-08T03:37:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:01.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:01.566459+0000) 2022-09-08T03:37:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:02.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:02.591800+0000) 2022-09-08T03:37:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:02.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:02.566589+0000) 2022-09-08T03:37:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:03 smithi055 ceph-mon[97033]: pgmap v11594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:03.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:03 smithi059 ceph-mon[97450]: pgmap v11594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:03.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:03.591937+0000) 2022-09-08T03:37:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:03.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:03.566699+0000) 2022-09-08T03:37:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:04.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:04.592068+0000) 2022-09-08T03:37:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:04.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:04.566836+0000) 2022-09-08T03:37:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:04 smithi055 ceph-mon[97033]: pgmap v11595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:05.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:04 smithi059 ceph-mon[97450]: pgmap v11595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:05.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:05.592196+0000) 2022-09-08T03:37:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:05.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:05.566972+0000) 2022-09-08T03:37:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:37:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:37:06.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:37:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:37:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:06.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:06.592334+0000) 2022-09-08T03:37:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:06.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:06.567098+0000) 2022-09-08T03:37:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:06 smithi055 ceph-mon[97033]: pgmap v11596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:37:07.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:06 smithi059 ceph-mon[97450]: pgmap v11596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:37:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:07.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:07.592482+0000) 2022-09-08T03:37:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:07.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:07.567235+0000) 2022-09-08T03:37:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:08.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:08.592625+0000) 2022-09-08T03:37:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:08.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:08.567376+0000) 2022-09-08T03:37:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:09 smithi055 ceph-mon[97033]: pgmap v11597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:37:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:37:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:37:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:37:09.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:09 smithi059 ceph-mon[97450]: pgmap v11597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:37:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:37:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:37:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:37:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:09.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:09.567516+0000) 2022-09-08T03:37:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:09.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:09.592773+0000) 2022-09-08T03:37:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:10 smithi055 ceph-mon[97033]: osdmap e803: 12 total, 11 up, 11 in 2022-09-08T03:37:10.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:10 smithi059 ceph-mon[97450]: osdmap e803: 12 total, 11 up, 11 in 2022-09-08T03:37:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:10.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:10.592914+0000) 2022-09-08T03:37:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:10.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:10.567657+0000) 2022-09-08T03:37:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:11 smithi055 ceph-mon[97033]: pgmap v11599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:37:11.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:11 smithi059 ceph-mon[97450]: pgmap v11599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:37:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:11.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:11.593052+0000) 2022-09-08T03:37:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:11.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:11.567787+0000) 2022-09-08T03:37:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:12.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:12.593193+0000) 2022-09-08T03:37:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:12.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:12.567923+0000) 2022-09-08T03:37:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:13 smithi055 ceph-mon[97033]: pgmap v11600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:37:13.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:13 smithi059 ceph-mon[97450]: pgmap v11600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:37:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:13.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:13.593318+0000) 2022-09-08T03:37:13.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:37:13.584+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:37:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:13.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:13.568059+0000) 2022-09-08T03:37:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:14.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:14.593454+0000) 2022-09-08T03:37:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:14.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:14.568192+0000) 2022-09-08T03:37:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:14 smithi055 ceph-mon[97033]: osdmap e804: 12 total, 11 up, 11 in 2022-09-08T03:37:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:14 smithi055 ceph-mon[97033]: pgmap v11602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:37:15.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:14 smithi059 ceph-mon[97450]: osdmap e804: 12 total, 11 up, 11 in 2022-09-08T03:37:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:14 smithi059 ceph-mon[97450]: pgmap v11602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:37:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:15.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:15.568323+0000) 2022-09-08T03:37:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:15.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:15.593584+0000) 2022-09-08T03:37:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:16.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:16.568455+0000) 2022-09-08T03:37:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:16.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:16.593704+0000) 2022-09-08T03:37:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:17 smithi055 ceph-mon[97033]: pgmap v11603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:37:17.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:17 smithi059 ceph-mon[97450]: pgmap v11603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:37:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:17.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:17.593837+0000) 2022-09-08T03:37:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:17.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:17.568590+0000) 2022-09-08T03:37:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:18.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:18.593969+0000) 2022-09-08T03:37:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:18.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:18.568681+0000) 2022-09-08T03:37:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:19 smithi055 ceph-mon[97033]: pgmap v11604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T03:37:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:19 smithi059 ceph-mon[97450]: pgmap v11604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T03:37:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:19.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:19.594110+0000) 2022-09-08T03:37:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:19.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:19.568794+0000) 2022-09-08T03:37:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:20.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:20.594245+0000) 2022-09-08T03:37:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:20.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:20.568926+0000) 2022-09-08T03:37:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:21 smithi055 ceph-mon[97033]: pgmap v11605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:37:21.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:21 smithi059 ceph-mon[97450]: pgmap v11605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:37:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:21.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:21.594374+0000) 2022-09-08T03:37:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:21.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:21.569050+0000) 2022-09-08T03:37:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:22.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:22.594536+0000) 2022-09-08T03:37:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:22.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:22.569174+0000) 2022-09-08T03:37:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:23 smithi055 ceph-mon[97033]: pgmap v11606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:37:23.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:23 smithi059 ceph-mon[97450]: pgmap v11606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:37:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:23.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:23.594700+0000) 2022-09-08T03:37:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:23.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:23.569306+0000) 2022-09-08T03:37:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:24.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:24.594815+0000) 2022-09-08T03:37:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:24.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:24.569436+0000) 2022-09-08T03:37:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:24 smithi055 ceph-mon[97033]: pgmap v11607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2022-09-08T03:37:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:24 smithi059 ceph-mon[97450]: pgmap v11607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2022-09-08T03:37:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:25.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:25.594965+0000) 2022-09-08T03:37:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:25.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:25.569692+0000) 2022-09-08T03:37:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:26.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:26.595136+0000) 2022-09-08T03:37:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:26.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:26.569831+0000) 2022-09-08T03:37:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:27 smithi055 ceph-mon[97033]: pgmap v11608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:37:27.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:27 smithi059 ceph-mon[97450]: pgmap v11608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:37:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:27.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:27.595266+0000) 2022-09-08T03:37:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:27.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:27.569964+0000) 2022-09-08T03:37:28.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:37:28.592+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:37:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:28.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:28.595400+0000) 2022-09-08T03:37:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:28.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:28.570100+0000) 2022-09-08T03:37:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:29 smithi055 ceph-mon[97033]: pgmap v11609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:29.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:29 smithi059 ceph-mon[97450]: pgmap v11609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:29.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:29.595506+0000) 2022-09-08T03:37:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:29.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:29.570234+0000) 2022-09-08T03:37:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:30.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:30.595643+0000) 2022-09-08T03:37:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:30.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:30.570361+0000) 2022-09-08T03:37:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:31 smithi055 ceph-mon[97033]: pgmap v11610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:37:31.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:31 smithi059 ceph-mon[97450]: pgmap v11610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:37:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:31.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:31.595776+0000) 2022-09-08T03:37:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:31.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:31.570491+0000) 2022-09-08T03:37:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:32.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:32.570624+0000) 2022-09-08T03:37:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:32.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:32.595914+0000) 2022-09-08T03:37:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:33 smithi055 ceph-mon[97033]: pgmap v11611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:33.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:33 smithi059 ceph-mon[97450]: pgmap v11611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:33.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:33.570775+0000) 2022-09-08T03:37:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:33.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:33.596039+0000) 2022-09-08T03:37:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:34.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:34.596166+0000) 2022-09-08T03:37:34.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:34.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:34.570908+0000) 2022-09-08T03:37:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:34 smithi055 ceph-mon[97033]: pgmap v11612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:35.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:34 smithi059 ceph-mon[97450]: pgmap v11612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:35.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:35.571044+0000) 2022-09-08T03:37:35.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:35.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:35.596317+0000) 2022-09-08T03:37:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:36.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:36.596438+0000) 2022-09-08T03:37:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:36.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:36.571180+0000) 2022-09-08T03:37:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:37 smithi055 ceph-mon[97033]: pgmap v11613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:37:37.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:37 smithi059 ceph-mon[97450]: pgmap v11613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:37:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:37.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:37.596567+0000) 2022-09-08T03:37:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:37.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:37.571310+0000) 2022-09-08T03:37:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:38.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:38.596698+0000) 2022-09-08T03:37:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:38.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:38.571462+0000) 2022-09-08T03:37:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:39 smithi055 ceph-mon[97033]: pgmap v11614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:39.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:39 smithi059 ceph-mon[97450]: pgmap v11614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:39.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:39.596810+0000) 2022-09-08T03:37:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:39.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:39.571602+0000) 2022-09-08T03:37:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:40.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:40.596917+0000) 2022-09-08T03:37:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:40.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:40.571740+0000) 2022-09-08T03:37:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:41 smithi055 ceph-mon[97033]: pgmap v11615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:37:41.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:41 smithi059 ceph-mon[97450]: pgmap v11615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:37:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:41.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:41.597050+0000) 2022-09-08T03:37:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:41.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:41.571876+0000) 2022-09-08T03:37:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:42.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:42.597188+0000) 2022-09-08T03:37:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:42.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:42.572013+0000) 2022-09-08T03:37:43.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:43 smithi059 ceph-mon[97450]: pgmap v11616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:37:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:37:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:37:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:43 smithi055 ceph-mon[97033]: pgmap v11616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:37:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:37:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:37:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:43.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:43.597301+0000) 2022-09-08T03:37:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:43.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:43.572162+0000) 2022-09-08T03:37:43.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:37:43.608+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:37:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:44 smithi055 ceph-mon[97033]: pgmap v11617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:37:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:44 smithi055 ceph-mon[97033]: pgmap v11618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:37:44.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:44 smithi059 ceph-mon[97450]: pgmap v11617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:37:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:44 smithi059 ceph-mon[97450]: pgmap v11618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:37:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:44.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:44.597440+0000) 2022-09-08T03:37:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:44.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:44.572282+0000) 2022-09-08T03:37:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:45.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:45.597583+0000) 2022-09-08T03:37:45.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:45.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:45.572419+0000) 2022-09-08T03:37:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:46 smithi055 ceph-mon[97033]: pgmap v11619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:37:46.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:46 smithi059 ceph-mon[97450]: pgmap v11619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:37:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:46.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:46.597718+0000) 2022-09-08T03:37:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:46.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:46.572556+0000) 2022-09-08T03:37:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:47.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:47.597830+0000) 2022-09-08T03:37:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:47.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:47.572695+0000) 2022-09-08T03:37:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:48 smithi055 ceph-mon[97033]: pgmap v11620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:37:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:48 smithi059 ceph-mon[97450]: pgmap v11620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:37:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:48.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:48.598052+0000) 2022-09-08T03:37:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:48.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:48.572846+0000) 2022-09-08T03:37:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:49.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:49.598281+0000) 2022-09-08T03:37:49.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:49.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:49.572966+0000) 2022-09-08T03:37:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:49 smithi055 ceph-mon[97033]: pgmap v11621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T03:37:50.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:49 smithi059 ceph-mon[97450]: pgmap v11621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T03:37:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:50.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:50.598466+0000) 2022-09-08T03:37:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:50.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:50.573105+0000) 2022-09-08T03:37:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:51.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:51.598602+0000) 2022-09-08T03:37:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:51.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:51.573239+0000) 2022-09-08T03:37:52.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:52 smithi055 ceph-mon[97033]: pgmap v11622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:37:52.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:52 smithi059 ceph-mon[97450]: pgmap v11622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T03:37:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:52.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:52.598704+0000) 2022-09-08T03:37:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:52.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:52.573377+0000) 2022-09-08T03:37:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:53.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:53.598832+0000) 2022-09-08T03:37:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:53.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:53.573516+0000) 2022-09-08T03:37:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:54 smithi055 ceph-mon[97033]: pgmap v11623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:37:54.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:54 smithi059 ceph-mon[97450]: pgmap v11623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:37:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:54.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:54.598964+0000) 2022-09-08T03:37:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:54.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:54.573639+0000) 2022-09-08T03:37:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:55.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:55.599085+0000) 2022-09-08T03:37:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:55.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:55.573771+0000) 2022-09-08T03:37:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:56 smithi055 ceph-mon[97033]: pgmap v11624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:37:56.601 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:56 smithi059 ceph-mon[97450]: pgmap v11624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:37:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:56.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:56.599215+0000) 2022-09-08T03:37:56.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:56.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:56.573910+0000) 2022-09-08T03:37:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:57.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:57.599345+0000) 2022-09-08T03:37:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:57.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:57.574044+0000) 2022-09-08T03:37:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:58 smithi055 ceph-mon[97033]: pgmap v11625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:58 smithi059 ceph-mon[97450]: pgmap v11625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:37:58.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:37:58.622+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:37:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:58.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:58.599473+0000) 2022-09-08T03:37:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:58.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:58.574181+0000) 2022-09-08T03:37:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:37:59.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:59.599615+0000) 2022-09-08T03:37:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:37:59.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:37:59.574312+0000) 2022-09-08T03:38:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:37:59 smithi055 ceph-mon[97033]: pgmap v11626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:00.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:37:59 smithi059 ceph-mon[97450]: pgmap v11626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:00.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:00.599764+0000) 2022-09-08T03:38:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:00.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:00.574443+0000) 2022-09-08T03:38:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:01.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:01.599908+0000) 2022-09-08T03:38:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:01.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:01.574570+0000) 2022-09-08T03:38:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:02 smithi055 ceph-mon[97033]: pgmap v11627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:38:02.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:02 smithi059 ceph-mon[97450]: pgmap v11627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:38:02.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:02.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:02.600058+0000) 2022-09-08T03:38:02.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:02.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:02.574710+0000) 2022-09-08T03:38:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:03.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:03.600201+0000) 2022-09-08T03:38:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:03.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:03.574843+0000) 2022-09-08T03:38:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:04 smithi055 ceph-mon[97033]: pgmap v11628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:04.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:04 smithi059 ceph-mon[97450]: pgmap v11628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:04.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:04.574967+0000) 2022-09-08T03:38:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:04.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:04.600332+0000) 2022-09-08T03:38:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:05.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:05.600466+0000) 2022-09-08T03:38:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:05.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:05.575100+0000) 2022-09-08T03:38:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:06 smithi055 ceph-mon[97033]: pgmap v11629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:38:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:38:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:38:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:06 smithi059 ceph-mon[97450]: pgmap v11629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:38:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:38:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:38:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:06.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:06.600604+0000) 2022-09-08T03:38:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:06.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:06.575225+0000) 2022-09-08T03:38:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:07.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:07.600732+0000) 2022-09-08T03:38:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:07.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:07.575350+0000) 2022-09-08T03:38:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:08 smithi055 ceph-mon[97033]: pgmap v11630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:38:08.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:08 smithi059 ceph-mon[97450]: pgmap v11630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:38:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:08.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:08.600838+0000) 2022-09-08T03:38:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:08.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:08.575502+0000) 2022-09-08T03:38:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:09.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:09.600967+0000) 2022-09-08T03:38:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:09.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:09.575632+0000) 2022-09-08T03:38:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:10 smithi055 ceph-mon[97033]: pgmap v11631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:38:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:10 smithi055 ceph-mon[97033]: osdmap e805: 12 total, 11 up, 11 in 2022-09-08T03:38:10.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:10 smithi059 ceph-mon[97450]: pgmap v11631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:38:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:10 smithi059 ceph-mon[97450]: osdmap e805: 12 total, 11 up, 11 in 2022-09-08T03:38:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:10.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:10.601108+0000) 2022-09-08T03:38:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:10.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:10.575763+0000) 2022-09-08T03:38:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:11.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:11.601254+0000) 2022-09-08T03:38:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:11.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:11.575884+0000) 2022-09-08T03:38:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:12 smithi055 ceph-mon[97033]: pgmap v11633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:38:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:12 smithi059 ceph-mon[97450]: pgmap v11633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:38:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:12.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:12.601397+0000) 2022-09-08T03:38:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:12.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:12.576015+0000) 2022-09-08T03:38:13.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:38:13.630+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:38:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:13.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:13.601526+0000) 2022-09-08T03:38:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:13.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:13.576135+0000) 2022-09-08T03:38:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:14 smithi055 ceph-mon[97033]: pgmap v11634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:38:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:14 smithi055 ceph-mon[97033]: osdmap e806: 12 total, 11 up, 11 in 2022-09-08T03:38:14.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:14 smithi059 ceph-mon[97450]: pgmap v11634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:38:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:14 smithi059 ceph-mon[97450]: osdmap e806: 12 total, 11 up, 11 in 2022-09-08T03:38:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:14.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:14.601675+0000) 2022-09-08T03:38:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:14.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:14.576276+0000) 2022-09-08T03:38:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:15.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:15.576413+0000) 2022-09-08T03:38:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:15.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:15.601807+0000) 2022-09-08T03:38:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:16 smithi055 ceph-mon[97033]: pgmap v11636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:38:16.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:16 smithi059 ceph-mon[97450]: pgmap v11636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:38:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:16.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:16.576527+0000) 2022-09-08T03:38:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:16.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:16.601934+0000) 2022-09-08T03:38:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:17.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:17.576672+0000) 2022-09-08T03:38:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:17.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:17.602063+0000) 2022-09-08T03:38:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:18 smithi055 ceph-mon[97033]: pgmap v11637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:38:18.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:18 smithi059 ceph-mon[97450]: pgmap v11637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:38:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:18.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:18.576818+0000) 2022-09-08T03:38:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:18.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:18.602174+0000) 2022-09-08T03:38:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:19.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:19.602302+0000) 2022-09-08T03:38:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:19.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:19.576933+0000) 2022-09-08T03:38:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:19 smithi055 ceph-mon[97033]: pgmap v11638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T03:38:20.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:19 smithi059 ceph-mon[97450]: pgmap v11638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T03:38:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:20.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:20.577053+0000) 2022-09-08T03:38:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:20.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:20.602428+0000) 2022-09-08T03:38:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:21.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:21.577170+0000) 2022-09-08T03:38:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:21.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:21.602557+0000) 2022-09-08T03:38:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:22 smithi055 ceph-mon[97033]: pgmap v11639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:38:22.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:22 smithi059 ceph-mon[97450]: pgmap v11639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:38:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:22.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:22.602693+0000) 2022-09-08T03:38:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:22.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:22.577297+0000) 2022-09-08T03:38:23.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:23.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:23.577436+0000) 2022-09-08T03:38:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:23.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:23.602823+0000) 2022-09-08T03:38:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:24 smithi055 ceph-mon[97033]: pgmap v11640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:38:24.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:24 smithi059 ceph-mon[97450]: pgmap v11640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:38:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:24.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:24.577572+0000) 2022-09-08T03:38:24.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:24.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:24.602952+0000) 2022-09-08T03:38:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:25.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:25.577703+0000) 2022-09-08T03:38:25.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:25.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:25.603077+0000) 2022-09-08T03:38:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:26 smithi055 ceph-mon[97033]: pgmap v11641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 91 B/s wr, 8 op/s 2022-09-08T03:38:26.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:26 smithi059 ceph-mon[97450]: pgmap v11641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 91 B/s wr, 8 op/s 2022-09-08T03:38:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:26.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:26.577849+0000) 2022-09-08T03:38:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:26.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:26.603207+0000) 2022-09-08T03:38:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:27.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:27.603339+0000) 2022-09-08T03:38:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:27.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:27.577994+0000) 2022-09-08T03:38:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:28 smithi055 ceph-mon[97033]: pgmap v11642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:28 smithi059 ceph-mon[97450]: pgmap v11642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:28.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:38:28.635+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:38:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:28.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:28.603475+0000) 2022-09-08T03:38:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:28.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:28.578146+0000) 2022-09-08T03:38:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:29.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:29.603610+0000) 2022-09-08T03:38:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:29.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:29.578279+0000) 2022-09-08T03:38:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:29 smithi055 ceph-mon[97033]: pgmap v11643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:30.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:29 smithi059 ceph-mon[97450]: pgmap v11643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:30.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:30.603760+0000) 2022-09-08T03:38:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:30.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:30.578412+0000) 2022-09-08T03:38:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:31.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:31.603877+0000) 2022-09-08T03:38:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:31.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:31.578546+0000) 2022-09-08T03:38:32.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:32 smithi055 ceph-mon[97033]: pgmap v11644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:38:32.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:32 smithi059 ceph-mon[97450]: pgmap v11644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:38:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:32.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:32.604012+0000) 2022-09-08T03:38:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:32.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:32.578696+0000) 2022-09-08T03:38:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:33.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:33.604139+0000) 2022-09-08T03:38:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:33.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:33.578815+0000) 2022-09-08T03:38:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:34 smithi055 ceph-mon[97033]: pgmap v11645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:34.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:34 smithi059 ceph-mon[97450]: pgmap v11645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:34.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:34.604249+0000) 2022-09-08T03:38:34.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:34.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:34.578933+0000) 2022-09-08T03:38:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:35.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:35.604377+0000) 2022-09-08T03:38:35.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:35.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:35.579067+0000) 2022-09-08T03:38:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:36 smithi055 ceph-mon[97033]: pgmap v11646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:38:36.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:36 smithi059 ceph-mon[97450]: pgmap v11646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:38:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:36.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:36.604499+0000) 2022-09-08T03:38:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:36.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:36.579182+0000) 2022-09-08T03:38:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:37.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:37.604643+0000) 2022-09-08T03:38:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:37.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:37.579328+0000) 2022-09-08T03:38:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:38 smithi055 ceph-mon[97033]: pgmap v11647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:38.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:38 smithi059 ceph-mon[97450]: pgmap v11647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:38.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:38.604782+0000) 2022-09-08T03:38:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:38.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:38.579456+0000) 2022-09-08T03:38:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:39.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:39.604924+0000) 2022-09-08T03:38:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:39.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:39.579595+0000) 2022-09-08T03:38:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:39 smithi055 ceph-mon[97033]: pgmap v11648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:40.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:39 smithi059 ceph-mon[97450]: pgmap v11648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:40.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:40.605036+0000) 2022-09-08T03:38:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:40.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:40.579732+0000) 2022-09-08T03:38:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:41.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:41.605167+0000) 2022-09-08T03:38:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:41.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:41.579848+0000) 2022-09-08T03:38:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:42 smithi055 ceph-mon[97033]: pgmap v11649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:38:42.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:42 smithi059 ceph-mon[97450]: pgmap v11649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:38:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:42.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:42.605306+0000) 2022-09-08T03:38:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:42.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:42.579968+0000) 2022-09-08T03:38:43.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:38:43.647+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:38:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:43.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:43.605437+0000) 2022-09-08T03:38:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:43.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:43.580105+0000) 2022-09-08T03:38:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:44 smithi055 ceph-mon[97033]: pgmap v11650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:38:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:38:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:38:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:38:44.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:44 smithi059 ceph-mon[97450]: pgmap v11650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:38:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:38:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:38:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:38:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:44.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:44.605563+0000) 2022-09-08T03:38:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:44.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:44.580242+0000) 2022-09-08T03:38:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:45 smithi055 ceph-mon[97033]: pgmap v11651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:38:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:45 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T03:38:45.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:45 smithi059 ceph-mon[97450]: pgmap v11651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:38:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:45 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T03:38:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:45.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:45.605681+0000) 2022-09-08T03:38:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:45.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:45.580359+0000) 2022-09-08T03:38:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:46.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:46.605781+0000) 2022-09-08T03:38:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:46.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:46.580495+0000) 2022-09-08T03:38:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:47 smithi055 ceph-mon[97033]: pgmap v11652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:38:47.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:47 smithi059 ceph-mon[97450]: pgmap v11652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:38:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:47.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:47.605944+0000) 2022-09-08T03:38:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:47.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:47.580746+0000) 2022-09-08T03:38:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:48.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:48.606110+0000) 2022-09-08T03:38:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:48.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:48.580933+0000) 2022-09-08T03:38:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:49 smithi055 ceph-mon[97033]: pgmap v11653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:38:49.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:49 smithi059 ceph-mon[97450]: pgmap v11653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:38:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:49.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:49.606479+0000) 2022-09-08T03:38:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:49.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:49.581062+0000) 2022-09-08T03:38:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:50.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:50.581179+0000) 2022-09-08T03:38:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:50.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:50.606614+0000) 2022-09-08T03:38:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:51 smithi055 ceph-mon[97033]: pgmap v11654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:38:51.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:51 smithi059 ceph-mon[97450]: pgmap v11654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:38:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:51.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:51.606753+0000) 2022-09-08T03:38:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:51.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:51.581306+0000) 2022-09-08T03:38:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:52.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:52.606901+0000) 2022-09-08T03:38:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:52.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:52.581421+0000) 2022-09-08T03:38:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:53 smithi055 ceph-mon[97033]: pgmap v11655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:38:53.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:53 smithi059 ceph-mon[97450]: pgmap v11655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:38:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:53.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:53.607029+0000) 2022-09-08T03:38:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:53.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:53.581550+0000) 2022-09-08T03:38:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:54.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:54.607164+0000) 2022-09-08T03:38:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:54.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:54.581692+0000) 2022-09-08T03:38:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:55 smithi055 ceph-mon[97033]: pgmap v11656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:38:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:55 smithi059 ceph-mon[97450]: pgmap v11656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:38:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:55.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:55.607300+0000) 2022-09-08T03:38:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:55.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:55.581826+0000) 2022-09-08T03:38:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:56.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:56.607434+0000) 2022-09-08T03:38:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:56.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:56.581963+0000) 2022-09-08T03:38:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:57 smithi055 ceph-mon[97033]: pgmap v11657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:38:57.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:57 smithi059 ceph-mon[97450]: pgmap v11657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:38:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:57.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:57.607569+0000) 2022-09-08T03:38:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:57.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:57.582093+0000) 2022-09-08T03:38:58.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:38:58.663+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:38:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:58.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:58.607699+0000) 2022-09-08T03:38:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:58.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:58.582242+0000) 2022-09-08T03:38:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:38:59 smithi055 ceph-mon[97033]: pgmap v11658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:59.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:38:59 smithi059 ceph-mon[97450]: pgmap v11658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:38:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:38:59.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:59.607830+0000) 2022-09-08T03:38:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:38:59.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:38:59.582380+0000) 2022-09-08T03:39:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:00.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:00.607960+0000) 2022-09-08T03:39:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:00.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:00.582515+0000) 2022-09-08T03:39:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:01 smithi055 ceph-mon[97033]: pgmap v11659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:01.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:01 smithi059 ceph-mon[97450]: pgmap v11659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:01.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:01.608091+0000) 2022-09-08T03:39:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:01.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:01.582659+0000) 2022-09-08T03:39:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:02.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:02.608229+0000) 2022-09-08T03:39:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:02.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:02.582793+0000) 2022-09-08T03:39:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:03 smithi055 ceph-mon[97033]: pgmap v11660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:39:03.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:03 smithi059 ceph-mon[97450]: pgmap v11660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:39:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:03.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:03.608363+0000) 2022-09-08T03:39:03.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:03.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:03.582922+0000) 2022-09-08T03:39:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:04.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:04.608500+0000) 2022-09-08T03:39:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:04.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:04.583065+0000) 2022-09-08T03:39:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:05 smithi055 ceph-mon[97033]: pgmap v11661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:05.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:05 smithi059 ceph-mon[97450]: pgmap v11661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:05.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:05.608630+0000) 2022-09-08T03:39:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:05.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:05.583195+0000) 2022-09-08T03:39:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:39:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:39:06.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:39:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:39:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:06.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:06.608782+0000) 2022-09-08T03:39:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:06.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:06.583331+0000) 2022-09-08T03:39:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:07 smithi055 ceph-mon[97033]: pgmap v11662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:39:07.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:07 smithi059 ceph-mon[97450]: pgmap v11662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:39:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:07.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:07.608922+0000) 2022-09-08T03:39:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:07.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:07.583464+0000) 2022-09-08T03:39:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:08.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:08.609058+0000) 2022-09-08T03:39:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:08.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:08.583609+0000) 2022-09-08T03:39:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:09 smithi055 ceph-mon[97033]: pgmap v11663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:39:09.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:09 smithi059 ceph-mon[97450]: pgmap v11663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:39:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:09.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:09.609195+0000) 2022-09-08T03:39:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:09.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:09.583701+0000) 2022-09-08T03:39:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:10 smithi055 ceph-mon[97033]: osdmap e807: 12 total, 11 up, 11 in 2022-09-08T03:39:10.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:10 smithi059 ceph-mon[97450]: osdmap e807: 12 total, 11 up, 11 in 2022-09-08T03:39:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:10.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:10.609327+0000) 2022-09-08T03:39:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:10.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:10.583835+0000) 2022-09-08T03:39:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:11 smithi055 ceph-mon[97033]: pgmap v11665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:39:11.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:11 smithi059 ceph-mon[97450]: pgmap v11665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:39:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:11.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:11.609461+0000) 2022-09-08T03:39:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:11.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:11.583972+0000) 2022-09-08T03:39:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:12.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:12.609606+0000) 2022-09-08T03:39:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:12.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:12.584106+0000) 2022-09-08T03:39:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:13 smithi055 ceph-mon[97033]: pgmap v11666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:39:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:13 smithi059 ceph-mon[97450]: pgmap v11666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:39:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:13.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:13.609756+0000) 2022-09-08T03:39:13.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:39:13.678+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:39:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:13.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:13.584245+0000) 2022-09-08T03:39:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:14.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:14.609891+0000) 2022-09-08T03:39:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:14.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:14.584379+0000) 2022-09-08T03:39:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:14 smithi055 ceph-mon[97033]: pgmap v11667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:39:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:14 smithi055 ceph-mon[97033]: osdmap e808: 12 total, 11 up, 11 in 2022-09-08T03:39:15.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:14 smithi059 ceph-mon[97450]: pgmap v11667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:39:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:14 smithi059 ceph-mon[97450]: osdmap e808: 12 total, 11 up, 11 in 2022-09-08T03:39:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:15.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:15.610024+0000) 2022-09-08T03:39:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:15.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:15.584512+0000) 2022-09-08T03:39:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:16.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:16.610153+0000) 2022-09-08T03:39:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:16.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:16.584652+0000) 2022-09-08T03:39:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:16 smithi055 ceph-mon[97033]: pgmap v11669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:39:17.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:16 smithi059 ceph-mon[97450]: pgmap v11669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:39:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:17.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:17.610286+0000) 2022-09-08T03:39:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:17.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:17.584791+0000) 2022-09-08T03:39:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:18.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:18.610416+0000) 2022-09-08T03:39:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:18.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:18.584935+0000) 2022-09-08T03:39:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:18 smithi055 ceph-mon[97033]: pgmap v11670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 235 B/s wr, 7 op/s 2022-09-08T03:39:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:18 smithi059 ceph-mon[97450]: pgmap v11670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 235 B/s wr, 7 op/s 2022-09-08T03:39:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:19.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:19.610554+0000) 2022-09-08T03:39:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:19.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:19.585070+0000) 2022-09-08T03:39:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:20.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:20.610696+0000) 2022-09-08T03:39:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:20.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:20.585205+0000) 2022-09-08T03:39:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:20 smithi055 ceph-mon[97033]: pgmap v11671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:39:21.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:20 smithi059 ceph-mon[97450]: pgmap v11671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:39:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:21.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:21.585328+0000) 2022-09-08T03:39:21.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:21.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:21.610834+0000) 2022-09-08T03:39:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:22.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:22.610969+0000) 2022-09-08T03:39:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:22.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:22.585459+0000) 2022-09-08T03:39:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:22 smithi055 ceph-mon[97033]: pgmap v11672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:39:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:22 smithi059 ceph-mon[97450]: pgmap v11672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:39:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:23.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:23.611110+0000) 2022-09-08T03:39:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:23.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:23.585611+0000) 2022-09-08T03:39:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:24.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:24.611258+0000) 2022-09-08T03:39:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:24.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:24.585802+0000) 2022-09-08T03:39:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:24 smithi055 ceph-mon[97033]: pgmap v11673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:39:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:24 smithi059 ceph-mon[97450]: pgmap v11673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:39:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:25.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:25.611388+0000) 2022-09-08T03:39:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:25.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:25.585922+0000) 2022-09-08T03:39:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:26.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:26.611524+0000) 2022-09-08T03:39:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:26.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:26.586046+0000) 2022-09-08T03:39:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:26 smithi055 ceph-mon[97033]: pgmap v11674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:39:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:26 smithi059 ceph-mon[97450]: pgmap v11674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:39:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:27.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:27.611666+0000) 2022-09-08T03:39:27.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:27.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:27.586183+0000) 2022-09-08T03:39:28.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:39:28.692+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:39:28.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:28.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:28.611799+0000) 2022-09-08T03:39:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:28.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:28.586323+0000) 2022-09-08T03:39:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:28 smithi055 ceph-mon[97033]: pgmap v11675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:29.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:28 smithi059 ceph-mon[97450]: pgmap v11675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:29.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:29.611932+0000) 2022-09-08T03:39:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:29.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:29.586465+0000) 2022-09-08T03:39:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:30.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:30.612064+0000) 2022-09-08T03:39:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:30.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:30.586603+0000) 2022-09-08T03:39:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:30 smithi055 ceph-mon[97033]: pgmap v11676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:30 smithi059 ceph-mon[97450]: pgmap v11676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:31.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:31.612191+0000) 2022-09-08T03:39:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:31.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:31.586737+0000) 2022-09-08T03:39:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:32.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:32.612339+0000) 2022-09-08T03:39:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:32.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:32.586869+0000) 2022-09-08T03:39:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:32 smithi055 ceph-mon[97033]: pgmap v11677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:39:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:32 smithi059 ceph-mon[97450]: pgmap v11677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:39:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:33.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:33.612475+0000) 2022-09-08T03:39:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:33.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:33.587013+0000) 2022-09-08T03:39:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:34.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:34.612612+0000) 2022-09-08T03:39:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:34.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:34.587150+0000) 2022-09-08T03:39:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:34 smithi055 ceph-mon[97033]: pgmap v11678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:35.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:34 smithi059 ceph-mon[97450]: pgmap v11678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:35.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:35.612742+0000) 2022-09-08T03:39:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:35.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:35.587290+0000) 2022-09-08T03:39:36.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:36.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:36.612872+0000) 2022-09-08T03:39:36.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:36.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:36.587419+0000) 2022-09-08T03:39:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:36 smithi055 ceph-mon[97033]: pgmap v11679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:39:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:36 smithi059 ceph-mon[97450]: pgmap v11679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:39:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:37.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:37.612980+0000) 2022-09-08T03:39:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:37.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:37.587556+0000) 2022-09-08T03:39:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:38.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:38.613108+0000) 2022-09-08T03:39:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:38.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:38.587706+0000) 2022-09-08T03:39:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:38 smithi055 ceph-mon[97033]: pgmap v11680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:39.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:38 smithi059 ceph-mon[97450]: pgmap v11680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:39.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:39.613234+0000) 2022-09-08T03:39:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:39.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:39.587839+0000) 2022-09-08T03:39:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:40.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:40.613362+0000) 2022-09-08T03:39:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:40.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:40.587972+0000) 2022-09-08T03:39:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:40 smithi055 ceph-mon[97033]: pgmap v11681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:40 smithi059 ceph-mon[97450]: pgmap v11681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:41.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:41.613490+0000) 2022-09-08T03:39:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:41.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:41.588111+0000) 2022-09-08T03:39:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:42.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:42.613627+0000) 2022-09-08T03:39:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:42.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:42.588242+0000) 2022-09-08T03:39:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:42 smithi055 ceph-mon[97033]: pgmap v11682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:39:43.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:42 smithi059 ceph-mon[97450]: pgmap v11682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:39:43.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:39:43.694+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:39:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:43.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:43.613704+0000) 2022-09-08T03:39:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:43.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:43.588381+0000) 2022-09-08T03:39:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:44.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:44.613817+0000) 2022-09-08T03:39:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:44.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:44.588514+0000) 2022-09-08T03:39:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:44 smithi055 ceph-mon[97033]: pgmap v11683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:39:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:39:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:39:45.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:39:45.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:44 smithi059 ceph-mon[97450]: pgmap v11683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:39:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:39:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:39:45.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:39:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:45.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:45.613922+0000) 2022-09-08T03:39:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:45.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:45.588665+0000) 2022-09-08T03:39:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:46.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:46.588801+0000) 2022-09-08T03:39:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:46.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:46.614039+0000) 2022-09-08T03:39:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:46 smithi055 ceph-mon[97033]: pgmap v11684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:39:47.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:46 smithi059 ceph-mon[97450]: pgmap v11684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:39:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:47.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:47.588941+0000) 2022-09-08T03:39:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:47.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:47.614173+0000) 2022-09-08T03:39:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:48.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:48.589089+0000) 2022-09-08T03:39:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:48.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:48.614287+0000) 2022-09-08T03:39:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:48 smithi055 ceph-mon[97033]: pgmap v11685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:49.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:48 smithi059 ceph-mon[97450]: pgmap v11685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:49.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:49.614395+0000) 2022-09-08T03:39:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:49.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:49.589225+0000) 2022-09-08T03:39:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:50.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:50.614526+0000) 2022-09-08T03:39:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:50.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:50.589352+0000) 2022-09-08T03:39:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:50 smithi055 ceph-mon[97033]: pgmap v11686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:50 smithi059 ceph-mon[97450]: pgmap v11686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:51.889 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:51.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:51.614668+0000) 2022-09-08T03:39:51.890 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:51.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:51.589493+0000) 2022-09-08T03:39:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:52.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:52.614800+0000) 2022-09-08T03:39:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:52.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:52.589652+0000) 2022-09-08T03:39:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:52 smithi055 ceph-mon[97033]: pgmap v11687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:39:53.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:52 smithi059 ceph-mon[97450]: pgmap v11687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:39:53.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:53.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:53.614915+0000) 2022-09-08T03:39:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:53.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:53.589804+0000) 2022-09-08T03:39:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:54.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:54.615052+0000) 2022-09-08T03:39:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:54.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:54.589934+0000) 2022-09-08T03:39:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:54 smithi055 ceph-mon[97033]: pgmap v11688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:55.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:54 smithi059 ceph-mon[97450]: pgmap v11688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:55.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:55.590062+0000) 2022-09-08T03:39:55.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:55.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:55.615224+0000) 2022-09-08T03:39:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:56.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:56.615372+0000) 2022-09-08T03:39:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:56.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:56.590205+0000) 2022-09-08T03:39:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:56 smithi055 ceph-mon[97033]: pgmap v11689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:39:57.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:56 smithi059 ceph-mon[97450]: pgmap v11689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:39:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:57.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:57.615515+0000) 2022-09-08T03:39:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:57.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:57.590354+0000) 2022-09-08T03:39:58.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:39:58.707+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:39:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:58.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:58.615667+0000) 2022-09-08T03:39:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:58.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:58.590490+0000) 2022-09-08T03:39:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:39:58 smithi055 ceph-mon[97033]: pgmap v11690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:59.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:39:58 smithi059 ceph-mon[97450]: pgmap v11690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:39:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:39:59.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:59.615785+0000) 2022-09-08T03:39:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:39:59.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:39:59.590609+0000) 2022-09-08T03:40:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:00.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:00.615917+0000) 2022-09-08T03:40:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:00.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:00.590697+0000) 2022-09-08T03:40:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:00 smithi055 ceph-mon[97033]: pgmap v11691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T03:40:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T03:40:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T03:40:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T03:40:01.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:00 smithi059 ceph-mon[97450]: pgmap v11691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T03:40:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T03:40:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T03:40:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T03:40:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:01.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:01.616058+0000) 2022-09-08T03:40:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:01.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:01.590808+0000) 2022-09-08T03:40:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:02.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:02.616197+0000) 2022-09-08T03:40:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:02.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:02.590920+0000) 2022-09-08T03:40:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:02 smithi055 ceph-mon[97033]: pgmap v11692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:40:03.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:02 smithi059 ceph-mon[97450]: pgmap v11692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:40:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:03.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:03.616327+0000) 2022-09-08T03:40:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:03.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:03.591055+0000) 2022-09-08T03:40:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:04.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:04.616461+0000) 2022-09-08T03:40:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:04.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:04.591181+0000) 2022-09-08T03:40:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:04 smithi055 ceph-mon[97033]: pgmap v11693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:05.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:04 smithi059 ceph-mon[97450]: pgmap v11693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:05.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:05.616593+0000) 2022-09-08T03:40:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:05.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:05.591317+0000) 2022-09-08T03:40:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:40:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:40:06.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:40:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:40:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:06.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:06.616725+0000) 2022-09-08T03:40:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:06.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:06.591456+0000) 2022-09-08T03:40:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:06 smithi055 ceph-mon[97033]: pgmap v11694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:40:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:06 smithi059 ceph-mon[97450]: pgmap v11694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:40:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:07.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:07.616839+0000) 2022-09-08T03:40:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:07.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:07.591596+0000) 2022-09-08T03:40:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:08.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:08.616976+0000) 2022-09-08T03:40:08.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:08.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:08.591737+0000) 2022-09-08T03:40:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:08 smithi055 ceph-mon[97033]: pgmap v11695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:40:09.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:08 smithi059 ceph-mon[97450]: pgmap v11695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:40:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:09.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:09.591874+0000) 2022-09-08T03:40:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:09.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:09.617112+0000) 2022-09-08T03:40:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:09 smithi055 ceph-mon[97033]: osdmap e809: 12 total, 11 up, 11 in 2022-09-08T03:40:10.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:09 smithi059 ceph-mon[97450]: osdmap e809: 12 total, 11 up, 11 in 2022-09-08T03:40:10.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:10.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:10.617244+0000) 2022-09-08T03:40:10.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:10.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:10.592008+0000) 2022-09-08T03:40:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:10 smithi055 ceph-mon[97033]: pgmap v11697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:40:11.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:10 smithi059 ceph-mon[97450]: pgmap v11697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:40:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:11.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:11.617373+0000) 2022-09-08T03:40:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:11.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:11.592130+0000) 2022-09-08T03:40:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:12.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:12.617488+0000) 2022-09-08T03:40:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:12.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:12.592261+0000) 2022-09-08T03:40:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:12 smithi055 ceph-mon[97033]: pgmap v11698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:40:13.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:12 smithi059 ceph-mon[97450]: pgmap v11698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:40:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:13.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:13.617616+0000) 2022-09-08T03:40:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:13.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:13.592398+0000) 2022-09-08T03:40:13.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:40:13.721+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:40:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:14.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:14.592528+0000) 2022-09-08T03:40:14.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:14.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:14.617751+0000) 2022-09-08T03:40:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:14 smithi055 ceph-mon[97033]: pgmap v11699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:40:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:14 smithi055 ceph-mon[97033]: osdmap e810: 12 total, 11 up, 11 in 2022-09-08T03:40:15.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:14 smithi059 ceph-mon[97450]: pgmap v11699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:40:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:14 smithi059 ceph-mon[97450]: osdmap e810: 12 total, 11 up, 11 in 2022-09-08T03:40:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:15.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:15.617891+0000) 2022-09-08T03:40:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:15.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:15.592671+0000) 2022-09-08T03:40:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:16.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:16.618024+0000) 2022-09-08T03:40:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:16.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:16.592799+0000) 2022-09-08T03:40:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:16 smithi055 ceph-mon[97033]: pgmap v11701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:40:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:16 smithi059 ceph-mon[97450]: pgmap v11701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:40:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:17.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:17.618153+0000) 2022-09-08T03:40:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:17.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:17.592945+0000) 2022-09-08T03:40:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:18.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:18.618284+0000) 2022-09-08T03:40:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:18.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:18.593093+0000) 2022-09-08T03:40:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:18 smithi055 ceph-mon[97033]: pgmap v11702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T03:40:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:18 smithi059 ceph-mon[97450]: pgmap v11702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T03:40:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:19.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:19.618417+0000) 2022-09-08T03:40:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:19.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:19.593222+0000) 2022-09-08T03:40:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:20.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:20.593353+0000) 2022-09-08T03:40:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:20.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:20.618549+0000) 2022-09-08T03:40:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:20 smithi055 ceph-mon[97033]: pgmap v11703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:40:21.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:20 smithi059 ceph-mon[97450]: pgmap v11703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:40:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:21.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:21.593495+0000) 2022-09-08T03:40:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:21.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:21.618687+0000) 2022-09-08T03:40:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:22.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:22.593629+0000) 2022-09-08T03:40:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:22.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:22.618845+0000) 2022-09-08T03:40:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:22 smithi055 ceph-mon[97033]: pgmap v11704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:40:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:22 smithi059 ceph-mon[97450]: pgmap v11704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:40:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:23.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:23.593775+0000) 2022-09-08T03:40:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:23.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:23.618989+0000) 2022-09-08T03:40:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:24.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:24.593887+0000) 2022-09-08T03:40:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:24.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:24.619131+0000) 2022-09-08T03:40:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:24 smithi055 ceph-mon[97033]: pgmap v11705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:40:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:24 smithi059 ceph-mon[97450]: pgmap v11705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:40:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:25.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:25.619255+0000) 2022-09-08T03:40:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:25.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:25.594039+0000) 2022-09-08T03:40:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:26.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:26.619404+0000) 2022-09-08T03:40:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:26.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:26.594160+0000) 2022-09-08T03:40:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:26 smithi055 ceph-mon[97033]: pgmap v11706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:40:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:26 smithi059 ceph-mon[97450]: pgmap v11706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:40:27.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:27.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:27.619567+0000) 2022-09-08T03:40:27.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:27.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:27.594381+0000) 2022-09-08T03:40:28.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:40:28.723+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:40:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:28.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:28.619730+0000) 2022-09-08T03:40:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:28.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:28.594554+0000) 2022-09-08T03:40:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:28 smithi055 ceph-mon[97033]: pgmap v11707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:28 smithi059 ceph-mon[97450]: pgmap v11707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:29.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:29.619851+0000) 2022-09-08T03:40:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:29.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:29.594691+0000) 2022-09-08T03:40:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:30.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:30.619967+0000) 2022-09-08T03:40:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:30.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:30.594827+0000) 2022-09-08T03:40:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:30 smithi055 ceph-mon[97033]: pgmap v11708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T03:40:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:30 smithi059 ceph-mon[97450]: pgmap v11708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T03:40:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:31.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:31.620099+0000) 2022-09-08T03:40:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:31.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:31.594937+0000) 2022-09-08T03:40:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:32.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:32.620237+0000) 2022-09-08T03:40:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:32.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:32.595072+0000) 2022-09-08T03:40:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:32 smithi055 ceph-mon[97033]: pgmap v11709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T03:40:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:32 smithi059 ceph-mon[97450]: pgmap v11709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T03:40:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:33.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:33.620368+0000) 2022-09-08T03:40:33.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:33.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:33.595196+0000) 2022-09-08T03:40:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:34.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:34.620495+0000) 2022-09-08T03:40:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:34.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:34.595330+0000) 2022-09-08T03:40:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:34 smithi055 ceph-mon[97033]: pgmap v11710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:35.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:34 smithi059 ceph-mon[97450]: pgmap v11710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:35.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:35.620630+0000) 2022-09-08T03:40:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:35.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:35.595467+0000) 2022-09-08T03:40:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:36.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:36.620777+0000) 2022-09-08T03:40:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:36.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:36.595599+0000) 2022-09-08T03:40:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:36 smithi055 ceph-mon[97033]: pgmap v11711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:40:37.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:36 smithi059 ceph-mon[97450]: pgmap v11711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:40:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:37.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:37.620907+0000) 2022-09-08T03:40:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:37.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:37.595736+0000) 2022-09-08T03:40:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:38.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:38.621041+0000) 2022-09-08T03:40:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:38.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:38.595875+0000) 2022-09-08T03:40:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:38 smithi055 ceph-mon[97033]: pgmap v11712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:39.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:38 smithi059 ceph-mon[97450]: pgmap v11712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:39.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:39.621172+0000) 2022-09-08T03:40:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:39.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:39.595989+0000) 2022-09-08T03:40:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:40.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:40.621308+0000) 2022-09-08T03:40:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:40.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:40.596104+0000) 2022-09-08T03:40:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:40 smithi055 ceph-mon[97033]: pgmap v11713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T03:40:41.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:40 smithi059 ceph-mon[97450]: pgmap v11713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T03:40:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:41.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:41.621440+0000) 2022-09-08T03:40:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:41.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:41.596227+0000) 2022-09-08T03:40:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:42.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:42.621588+0000) 2022-09-08T03:40:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:42.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:42.596358+0000) 2022-09-08T03:40:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:42 smithi055 ceph-mon[97033]: pgmap v11714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T03:40:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:42 smithi059 ceph-mon[97450]: pgmap v11714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T03:40:43.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:40:43.739+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:40:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:43.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:43.596492+0000) 2022-09-08T03:40:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:43.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:43.621722+0000) 2022-09-08T03:40:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:44.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:44.621860+0000) 2022-09-08T03:40:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:44.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:44.596629+0000) 2022-09-08T03:40:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:44 smithi055 ceph-mon[97033]: pgmap v11715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:40:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:40:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:40:45.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:44 smithi059 ceph-mon[97450]: pgmap v11715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:45.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:40:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:40:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:40:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:45.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:45.596773+0000) 2022-09-08T03:40:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:45.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:45.621993+0000) 2022-09-08T03:40:46.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:40:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:40:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:46.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:46.596884+0000) 2022-09-08T03:40:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:46.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:46.622134+0000) 2022-09-08T03:40:47.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:47 smithi059 ceph-mon[97450]: pgmap v11716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:40:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:47 smithi055 ceph-mon[97033]: pgmap v11716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:40:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:47.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:47.622267+0000) 2022-09-08T03:40:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:47.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:47.596998+0000) 2022-09-08T03:40:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:48.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:48.622399+0000) 2022-09-08T03:40:48.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:48.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:48.597138+0000) 2022-09-08T03:40:49.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:49 smithi059 ceph-mon[97450]: pgmap v11717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:49 smithi055 ceph-mon[97033]: pgmap v11717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:49.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:49.622533+0000) 2022-09-08T03:40:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:49.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:49.597272+0000) 2022-09-08T03:40:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:50.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:50.622681+0000) 2022-09-08T03:40:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:50.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:50.597408+0000) 2022-09-08T03:40:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:51 smithi055 ceph-mon[97033]: pgmap v11718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T03:40:51.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:51 smithi059 ceph-mon[97450]: pgmap v11718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T03:40:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:51.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:51.622798+0000) 2022-09-08T03:40:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:51.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:51.597540+0000) 2022-09-08T03:40:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:52.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:52.622938+0000) 2022-09-08T03:40:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:52.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:52.597680+0000) 2022-09-08T03:40:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:53 smithi055 ceph-mon[97033]: pgmap v11719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T03:40:53.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:53 smithi059 ceph-mon[97450]: pgmap v11719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T03:40:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:53.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:53.623066+0000) 2022-09-08T03:40:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:53.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:53.597824+0000) 2022-09-08T03:40:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:54.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:54.623187+0000) 2022-09-08T03:40:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:54.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:54.597953+0000) 2022-09-08T03:40:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:55 smithi055 ceph-mon[97033]: pgmap v11720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:55.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:55 smithi059 ceph-mon[97450]: pgmap v11720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:55.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:55.623315+0000) 2022-09-08T03:40:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:55.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:55.598088+0000) 2022-09-08T03:40:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:56.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:56.623444+0000) 2022-09-08T03:40:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:56.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:56.598221+0000) 2022-09-08T03:40:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:57 smithi055 ceph-mon[97033]: pgmap v11721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:40:57.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:57 smithi059 ceph-mon[97450]: pgmap v11721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:40:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:57.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:57.623571+0000) 2022-09-08T03:40:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:57.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:57.598348+0000) 2022-09-08T03:40:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:58.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:58.623700+0000) 2022-09-08T03:40:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:58.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:58.598480+0000) 2022-09-08T03:40:58.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:40:58.755+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:40:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:40:59 smithi055 ceph-mon[97033]: pgmap v11722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:59.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:40:59 smithi059 ceph-mon[97450]: pgmap v11722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:40:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:40:59.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:59.623832+0000) 2022-09-08T03:40:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:40:59.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:40:59.598623+0000) 2022-09-08T03:41:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:00.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:00.623989+0000) 2022-09-08T03:41:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:00.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:00.598755+0000) 2022-09-08T03:41:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:01 smithi055 ceph-mon[97033]: pgmap v11723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T03:41:01.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:01 smithi059 ceph-mon[97450]: pgmap v11723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T03:41:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:01.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:01.624118+0000) 2022-09-08T03:41:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:01.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:01.598887+0000) 2022-09-08T03:41:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:02.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:02.624258+0000) 2022-09-08T03:41:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:02.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:02.599032+0000) 2022-09-08T03:41:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:03 smithi055 ceph-mon[97033]: pgmap v11724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T03:41:03.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:03 smithi059 ceph-mon[97450]: pgmap v11724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T03:41:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:03.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:03.624402+0000) 2022-09-08T03:41:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:03.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:03.599183+0000) 2022-09-08T03:41:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:04.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:04.624541+0000) 2022-09-08T03:41:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:04.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:04.599325+0000) 2022-09-08T03:41:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:05 smithi055 ceph-mon[97033]: pgmap v11725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:05.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:05 smithi059 ceph-mon[97450]: pgmap v11725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:05.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:05.624691+0000) 2022-09-08T03:41:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:05.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:05.599474+0000) 2022-09-08T03:41:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:41:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:41:06.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:41:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:41:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:06.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:06.599610+0000) 2022-09-08T03:41:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:06.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:06.624835+0000) 2022-09-08T03:41:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:07 smithi055 ceph-mon[97033]: pgmap v11726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:41:07.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:07 smithi059 ceph-mon[97450]: pgmap v11726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:41:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:07.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:07.599742+0000) 2022-09-08T03:41:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:07.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:07.624978+0000) 2022-09-08T03:41:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:08.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:08.599881+0000) 2022-09-08T03:41:08.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:08.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:08.625126+0000) 2022-09-08T03:41:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:09 smithi055 ceph-mon[97033]: pgmap v11727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:41:09.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:09 smithi059 ceph-mon[97450]: pgmap v11727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:41:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:09.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:09.625260+0000) 2022-09-08T03:41:09.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:09.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:09.600001+0000) 2022-09-08T03:41:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:10 smithi055 ceph-mon[97033]: osdmap e811: 12 total, 11 up, 11 in 2022-09-08T03:41:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:10 smithi059 ceph-mon[97450]: osdmap e811: 12 total, 11 up, 11 in 2022-09-08T03:41:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:10.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:10.625384+0000) 2022-09-08T03:41:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:10.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:10.600106+0000) 2022-09-08T03:41:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:11 smithi055 ceph-mon[97033]: pgmap v11729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:41:11.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:11 smithi059 ceph-mon[97450]: pgmap v11729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:41:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:11.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:11.625513+0000) 2022-09-08T03:41:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:11.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:11.600235+0000) 2022-09-08T03:41:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:12.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:12.625668+0000) 2022-09-08T03:41:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:12.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:12.600365+0000) 2022-09-08T03:41:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:13 smithi055 ceph-mon[97033]: pgmap v11730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:41:13.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:13 smithi059 ceph-mon[97450]: pgmap v11730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:41:13.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:41:13.761+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:41:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:13.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:13.625794+0000) 2022-09-08T03:41:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:13.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:13.600498+0000) 2022-09-08T03:41:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:14.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:14.625924+0000) 2022-09-08T03:41:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:14.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:14.600640+0000) 2022-09-08T03:41:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:14 smithi055 ceph-mon[97033]: pgmap v11731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:41:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:14 smithi055 ceph-mon[97033]: osdmap e812: 12 total, 11 up, 11 in 2022-09-08T03:41:15.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:14 smithi059 ceph-mon[97450]: pgmap v11731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:41:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:14 smithi059 ceph-mon[97450]: osdmap e812: 12 total, 11 up, 11 in 2022-09-08T03:41:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:15.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:15.626061+0000) 2022-09-08T03:41:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:15.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:15.600777+0000) 2022-09-08T03:41:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:16.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:16.626178+0000) 2022-09-08T03:41:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:16.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:16.600909+0000) 2022-09-08T03:41:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:16 smithi055 ceph-mon[97033]: pgmap v11733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:41:17.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:16 smithi059 ceph-mon[97450]: pgmap v11733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:41:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:17.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:17.601043+0000) 2022-09-08T03:41:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:17.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:17.626309+0000) 2022-09-08T03:41:18.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:18.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:18.601173+0000) 2022-09-08T03:41:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:18.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:18.626446+0000) 2022-09-08T03:41:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:18 smithi055 ceph-mon[97033]: pgmap v11734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 233 B/s wr, 7 op/s 2022-09-08T03:41:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:18 smithi059 ceph-mon[97450]: pgmap v11734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 233 B/s wr, 7 op/s 2022-09-08T03:41:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:19.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:19.601296+0000) 2022-09-08T03:41:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:19.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:19.626582+0000) 2022-09-08T03:41:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:20.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:20.601429+0000) 2022-09-08T03:41:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:20.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:20.626690+0000) 2022-09-08T03:41:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:20 smithi055 ceph-mon[97033]: pgmap v11735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:41:21.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:20 smithi059 ceph-mon[97450]: pgmap v11735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:41:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:21.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:21.626813+0000) 2022-09-08T03:41:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:21.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:21.601572+0000) 2022-09-08T03:41:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:22.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:22.626947+0000) 2022-09-08T03:41:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:22.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:22.601715+0000) 2022-09-08T03:41:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:22 smithi055 ceph-mon[97033]: pgmap v11736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:41:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:22 smithi059 ceph-mon[97450]: pgmap v11736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:41:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:23.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:23.601849+0000) 2022-09-08T03:41:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:23.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:23.627082+0000) 2022-09-08T03:41:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:24.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:24.627214+0000) 2022-09-08T03:41:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:24.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:24.601981+0000) 2022-09-08T03:41:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:24 smithi055 ceph-mon[97033]: pgmap v11737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:41:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:24 smithi059 ceph-mon[97450]: pgmap v11737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:41:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:25.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:25.627326+0000) 2022-09-08T03:41:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:25.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:25.602121+0000) 2022-09-08T03:41:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:26.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:26.627446+0000) 2022-09-08T03:41:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:26.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:26.602253+0000) 2022-09-08T03:41:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:26 smithi055 ceph-mon[97033]: pgmap v11738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:41:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:26 smithi059 ceph-mon[97450]: pgmap v11738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:41:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:27.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:27.627580+0000) 2022-09-08T03:41:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:27.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:27.602370+0000) 2022-09-08T03:41:28.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:28.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:28.627691+0000) 2022-09-08T03:41:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:28.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:28.602521+0000) 2022-09-08T03:41:28.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:41:28.777+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:41:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:28 smithi055 ceph-mon[97033]: pgmap v11739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:29.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:28 smithi059 ceph-mon[97450]: pgmap v11739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:29.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:29.627802+0000) 2022-09-08T03:41:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:29.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:29.602676+0000) 2022-09-08T03:41:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:30.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:30.627913+0000) 2022-09-08T03:41:30.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:30.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:30.602791+0000) 2022-09-08T03:41:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:30 smithi055 ceph-mon[97033]: pgmap v11740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:41:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:30 smithi059 ceph-mon[97450]: pgmap v11740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:41:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:31.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:31.628043+0000) 2022-09-08T03:41:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:31.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:31.602900+0000) 2022-09-08T03:41:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:32.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:32.628184+0000) 2022-09-08T03:41:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:32.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:32.603041+0000) 2022-09-08T03:41:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:32 smithi055 ceph-mon[97033]: pgmap v11741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:32 smithi059 ceph-mon[97450]: pgmap v11741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:33.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:33.628385+0000) 2022-09-08T03:41:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:33.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:33.603189+0000) 2022-09-08T03:41:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:34.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:34.628537+0000) 2022-09-08T03:41:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:34.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:34.603332+0000) 2022-09-08T03:41:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:34 smithi055 ceph-mon[97033]: pgmap v11742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:35.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:34 smithi059 ceph-mon[97450]: pgmap v11742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:35.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:35.628680+0000) 2022-09-08T03:41:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:35.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:35.603482+0000) 2022-09-08T03:41:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:36.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:36.628816+0000) 2022-09-08T03:41:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:36.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:36.603616+0000) 2022-09-08T03:41:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:36 smithi055 ceph-mon[97033]: pgmap v11743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:41:37.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:36 smithi059 ceph-mon[97450]: pgmap v11743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:41:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:37.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:37.628938+0000) 2022-09-08T03:41:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:37.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:37.603754+0000) 2022-09-08T03:41:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:38.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:38.629084+0000) 2022-09-08T03:41:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:38.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:38.603894+0000) 2022-09-08T03:41:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:38 smithi055 ceph-mon[97033]: pgmap v11744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:39.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:38 smithi059 ceph-mon[97450]: pgmap v11744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:39.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:39.629307+0000) 2022-09-08T03:41:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:39.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:39.604075+0000) 2022-09-08T03:41:40.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:40.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:40.629451+0000) 2022-09-08T03:41:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:40.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:40.604189+0000) 2022-09-08T03:41:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:40 smithi055 ceph-mon[97033]: pgmap v11745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:41:41.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:40 smithi059 ceph-mon[97450]: pgmap v11745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:41:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:41.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:41.629588+0000) 2022-09-08T03:41:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:41.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:41.604320+0000) 2022-09-08T03:41:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:42.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:42.629736+0000) 2022-09-08T03:41:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:42.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:42.604451+0000) 2022-09-08T03:41:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:42 smithi055 ceph-mon[97033]: pgmap v11746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:41:43.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:42 smithi059 ceph-mon[97450]: pgmap v11746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:41:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:43.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:43.629879+0000) 2022-09-08T03:41:43.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:41:43.792+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:41:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:43.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:43.604596+0000) 2022-09-08T03:41:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:44.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:44.630024+0000) 2022-09-08T03:41:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:44.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:44.604692+0000) 2022-09-08T03:41:45.186 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:44 smithi055 ceph-mon[97033]: pgmap v11747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:41:45.195 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:44 smithi059 ceph-mon[97450]: pgmap v11747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:41:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:45.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:45.630170+0000) 2022-09-08T03:41:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:45.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:45.604809+0000) 2022-09-08T03:41:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:41:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:41:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:41:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:41:46.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:41:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:41:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:41:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:41:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:46.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:46.630317+0000) 2022-09-08T03:41:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:46.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:46.604939+0000) 2022-09-08T03:41:47.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:46 smithi055 ceph-mon[97033]: pgmap v11748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:41:47.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:46 smithi059 ceph-mon[97450]: pgmap v11748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:41:47.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:47.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:47.630464+0000) 2022-09-08T03:41:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:47.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:47.605068+0000) 2022-09-08T03:41:48.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:48.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:48.630616+0000) 2022-09-08T03:41:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:48.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:48.605199+0000) 2022-09-08T03:41:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:48 smithi055 ceph-mon[97033]: pgmap v11749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:41:49.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:48 smithi059 ceph-mon[97450]: pgmap v11749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:41:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:49.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:49.630760+0000) 2022-09-08T03:41:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:49.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:49.605342+0000) 2022-09-08T03:41:50.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:50.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:50.630897+0000) 2022-09-08T03:41:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:50.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:50.605476+0000) 2022-09-08T03:41:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:50 smithi055 ceph-mon[97033]: pgmap v11750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:41:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:50 smithi059 ceph-mon[97450]: pgmap v11750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:41:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:51.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:51.631116+0000) 2022-09-08T03:41:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:51.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:51.605705+0000) 2022-09-08T03:41:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:52.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:52.631269+0000) 2022-09-08T03:41:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:52.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:52.605916+0000) 2022-09-08T03:41:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:52 smithi055 ceph-mon[97033]: pgmap v11751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:53.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:52 smithi059 ceph-mon[97450]: pgmap v11751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:53.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:53.631401+0000) 2022-09-08T03:41:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:53.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:53.606050+0000) 2022-09-08T03:41:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:54.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:54.631536+0000) 2022-09-08T03:41:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:54.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:54.606181+0000) 2022-09-08T03:41:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:54 smithi055 ceph-mon[97033]: pgmap v11752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:55.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:54 smithi059 ceph-mon[97450]: pgmap v11752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:55.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:55.631671+0000) 2022-09-08T03:41:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:55.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:55.606318+0000) 2022-09-08T03:41:56.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:56.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:56.631801+0000) 2022-09-08T03:41:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:56.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:56.606434+0000) 2022-09-08T03:41:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:56 smithi055 ceph-mon[97033]: pgmap v11753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:41:57.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:56 smithi059 ceph-mon[97450]: pgmap v11753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:41:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:57.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:57.631935+0000) 2022-09-08T03:41:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:57.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:57.606567+0000) 2022-09-08T03:41:58.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:41:58.792+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:41:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:58.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:58.632067+0000) 2022-09-08T03:41:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:58.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:58.606709+0000) 2022-09-08T03:41:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:41:58 smithi055 ceph-mon[97033]: pgmap v11754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:59.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:41:58 smithi059 ceph-mon[97450]: pgmap v11754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:41:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:41:59.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:59.632198+0000) 2022-09-08T03:41:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:41:59.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:41:59.606851+0000) 2022-09-08T03:42:00.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:00.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:00.632337+0000) 2022-09-08T03:42:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:00.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:00.606986+0000) 2022-09-08T03:42:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:00 smithi055 ceph-mon[97033]: pgmap v11755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:42:01.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:00 smithi059 ceph-mon[97450]: pgmap v11755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:42:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:01.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:01.632465+0000) 2022-09-08T03:42:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:01.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:01.607127+0000) 2022-09-08T03:42:02.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:02.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:02.632608+0000) 2022-09-08T03:42:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:02.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:02.607253+0000) 2022-09-08T03:42:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:02 smithi055 ceph-mon[97033]: pgmap v11756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:03.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:02 smithi059 ceph-mon[97450]: pgmap v11756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:03.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:03.632743+0000) 2022-09-08T03:42:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:03.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:03.607382+0000) 2022-09-08T03:42:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:04.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:04.632871+0000) 2022-09-08T03:42:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:04.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:04.607515+0000) 2022-09-08T03:42:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:04 smithi055 ceph-mon[97033]: pgmap v11757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:05.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:04 smithi059 ceph-mon[97450]: pgmap v11757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:05.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:05.633016+0000) 2022-09-08T03:42:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:05.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:05.607763+0000) 2022-09-08T03:42:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:42:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:42:06.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:42:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:42:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:06.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:06.633153+0000) 2022-09-08T03:42:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:06.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:06.607877+0000) 2022-09-08T03:42:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:06 smithi055 ceph-mon[97033]: pgmap v11758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:42:07.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:06 smithi059 ceph-mon[97450]: pgmap v11758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:42:07.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:07.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:07.633284+0000) 2022-09-08T03:42:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:07.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:07.608011+0000) 2022-09-08T03:42:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:08.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:08.633418+0000) 2022-09-08T03:42:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:08.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:08.608134+0000) 2022-09-08T03:42:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:08 smithi055 ceph-mon[97033]: pgmap v11759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:42:09.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:08 smithi059 ceph-mon[97450]: pgmap v11759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:42:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:09.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:09.633563+0000) 2022-09-08T03:42:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:09.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:09.608270+0000) 2022-09-08T03:42:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:09 smithi055 ceph-mon[97033]: osdmap e813: 12 total, 11 up, 11 in 2022-09-08T03:42:10.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:09 smithi059 ceph-mon[97450]: osdmap e813: 12 total, 11 up, 11 in 2022-09-08T03:42:10.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:10.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:10.633695+0000) 2022-09-08T03:42:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:10.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:10.608402+0000) 2022-09-08T03:42:11.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:10 smithi059 ceph-mon[97450]: pgmap v11761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:42:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:10 smithi055 ceph-mon[97033]: pgmap v11761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:42:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:11.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:11.633826+0000) 2022-09-08T03:42:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:11.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:11.608618+0000) 2022-09-08T03:42:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:12.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:12.633963+0000) 2022-09-08T03:42:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:12.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:12.608773+0000) 2022-09-08T03:42:13.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:12 smithi059 ceph-mon[97450]: pgmap v11762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:42:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:12 smithi055 ceph-mon[97033]: pgmap v11762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:42:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:13.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:13.634093+0000) 2022-09-08T03:42:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:13.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:13.608907+0000) 2022-09-08T03:42:13.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:42:13.802+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:42:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:14.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:14.634230+0000) 2022-09-08T03:42:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:14.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:14.609032+0000) 2022-09-08T03:42:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:14 smithi055 ceph-mon[97033]: pgmap v11763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:42:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:14 smithi055 ceph-mon[97033]: osdmap e814: 12 total, 11 up, 11 in 2022-09-08T03:42:15.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:14 smithi059 ceph-mon[97450]: pgmap v11763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:42:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:14 smithi059 ceph-mon[97450]: osdmap e814: 12 total, 11 up, 11 in 2022-09-08T03:42:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:15.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:15.634363+0000) 2022-09-08T03:42:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:15.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:15.609168+0000) 2022-09-08T03:42:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:16.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:16.634490+0000) 2022-09-08T03:42:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:16.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:16.609296+0000) 2022-09-08T03:42:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:16 smithi055 ceph-mon[97033]: pgmap v11765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:42:17.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:16 smithi059 ceph-mon[97450]: pgmap v11765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:42:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:17.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:17.634621+0000) 2022-09-08T03:42:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:17.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:17.609431+0000) 2022-09-08T03:42:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:18.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:18.634709+0000) 2022-09-08T03:42:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:18.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:18.609570+0000) 2022-09-08T03:42:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:18 smithi055 ceph-mon[97033]: pgmap v11766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T03:42:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:18 smithi059 ceph-mon[97450]: pgmap v11766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T03:42:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:19.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:19.634842+0000) 2022-09-08T03:42:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:19.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:19.609682+0000) 2022-09-08T03:42:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:20.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:20.634970+0000) 2022-09-08T03:42:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:20.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:20.609812+0000) 2022-09-08T03:42:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:20 smithi055 ceph-mon[97033]: pgmap v11767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:42:21.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:20 smithi059 ceph-mon[97450]: pgmap v11767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:42:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:21.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:21.635102+0000) 2022-09-08T03:42:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:21.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:21.609926+0000) 2022-09-08T03:42:22.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:22.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:22.635220+0000) 2022-09-08T03:42:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:22.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:22.610055+0000) 2022-09-08T03:42:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:22 smithi055 ceph-mon[97033]: pgmap v11768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:42:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:22 smithi059 ceph-mon[97450]: pgmap v11768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:42:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:23.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:23.635351+0000) 2022-09-08T03:42:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:23.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:23.610201+0000) 2022-09-08T03:42:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:24.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:24.635473+0000) 2022-09-08T03:42:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:24.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:24.610335+0000) 2022-09-08T03:42:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:24 smithi055 ceph-mon[97033]: pgmap v11769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:42:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:24 smithi059 ceph-mon[97450]: pgmap v11769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:42:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:25.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:25.635618+0000) 2022-09-08T03:42:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:25.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:25.610472+0000) 2022-09-08T03:42:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:26.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:26.635774+0000) 2022-09-08T03:42:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:26.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:26.610594+0000) 2022-09-08T03:42:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:26 smithi055 ceph-mon[97033]: pgmap v11770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:42:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:26 smithi059 ceph-mon[97450]: pgmap v11770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:42:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:27.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:27.610707+0000) 2022-09-08T03:42:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:27.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:27.635907+0000) 2022-09-08T03:42:28.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:42:28.817+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:42:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:28.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:28.636016+0000) 2022-09-08T03:42:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:28.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:28.610849+0000) 2022-09-08T03:42:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:28 smithi055 ceph-mon[97033]: pgmap v11771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:29.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:28 smithi059 ceph-mon[97450]: pgmap v11771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:29.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:29.636149+0000) 2022-09-08T03:42:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:29.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:29.610981+0000) 2022-09-08T03:42:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:30.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:30.636278+0000) 2022-09-08T03:42:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:30.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:30.611113+0000) 2022-09-08T03:42:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:30 smithi055 ceph-mon[97033]: pgmap v11772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:42:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:30 smithi059 ceph-mon[97450]: pgmap v11772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:42:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:31.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:31.636407+0000) 2022-09-08T03:42:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:31.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:31.611239+0000) 2022-09-08T03:42:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:32.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:32.636546+0000) 2022-09-08T03:42:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:32.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:32.611374+0000) 2022-09-08T03:42:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:32 smithi055 ceph-mon[97033]: pgmap v11773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:32 smithi059 ceph-mon[97450]: pgmap v11773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:33.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:33.636766+0000) 2022-09-08T03:42:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:33.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:33.611490+0000) 2022-09-08T03:42:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:34.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:34.636915+0000) 2022-09-08T03:42:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:34.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:34.611624+0000) 2022-09-08T03:42:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:34 smithi055 ceph-mon[97033]: pgmap v11774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:35.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:34 smithi059 ceph-mon[97450]: pgmap v11774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:35.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:35.637043+0000) 2022-09-08T03:42:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:35.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:35.611769+0000) 2022-09-08T03:42:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:36.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:36.637180+0000) 2022-09-08T03:42:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:36.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:36.611880+0000) 2022-09-08T03:42:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:36 smithi055 ceph-mon[97033]: pgmap v11775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:42:37.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:36 smithi059 ceph-mon[97450]: pgmap v11775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:42:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:37.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:37.637408+0000) 2022-09-08T03:42:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:37.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:37.612049+0000) 2022-09-08T03:42:38.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:38.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:38.637595+0000) 2022-09-08T03:42:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:38.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:38.612195+0000) 2022-09-08T03:42:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:38 smithi055 ceph-mon[97033]: pgmap v11776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:39.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:38 smithi059 ceph-mon[97450]: pgmap v11776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:39.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:39.637698+0000) 2022-09-08T03:42:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:39.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:39.612329+0000) 2022-09-08T03:42:40.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:40.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:40.637841+0000) 2022-09-08T03:42:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:40.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:40.612463+0000) 2022-09-08T03:42:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:40 smithi055 ceph-mon[97033]: pgmap v11777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:42:41.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:40 smithi059 ceph-mon[97450]: pgmap v11777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:42:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:41.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:41.637979+0000) 2022-09-08T03:42:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:41.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:41.612595+0000) 2022-09-08T03:42:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:42.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:42.612709+0000) 2022-09-08T03:42:42.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:42.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:42.638103+0000) 2022-09-08T03:42:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:42 smithi055 ceph-mon[97033]: pgmap v11778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:43.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:42 smithi059 ceph-mon[97450]: pgmap v11778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:43.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:42:43.826+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:42:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:43.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:43.612824+0000) 2022-09-08T03:42:43.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:43.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:43.638233+0000) 2022-09-08T03:42:44.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:44.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:44.638364+0000) 2022-09-08T03:42:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:44.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:44.612956+0000) 2022-09-08T03:42:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:44 smithi055 ceph-mon[97033]: pgmap v11779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:45.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:44 smithi059 ceph-mon[97450]: pgmap v11779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:45.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:45.613072+0000) 2022-09-08T03:42:45.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:45.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:45.638495+0000) 2022-09-08T03:42:46.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:42:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:42:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:42:46.208 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:42:46.209 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:42:46.209 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:42:46.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:46.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:46.613203+0000) 2022-09-08T03:42:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:46.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:46.638624+0000) 2022-09-08T03:42:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:47 smithi055 ceph-mon[97033]: pgmap v11780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:42:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:42:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:42:47.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:47 smithi059 ceph-mon[97450]: pgmap v11780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:42:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:42:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:42:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:47.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:47.638762+0000) 2022-09-08T03:42:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:47.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:47.613338+0000) 2022-09-08T03:42:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:48.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:48.613477+0000) 2022-09-08T03:42:48.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:48.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:48.638901+0000) 2022-09-08T03:42:49.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:49 smithi059 ceph-mon[97450]: pgmap v11781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:42:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:42:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:49.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:49.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:49.639043+0000) 2022-09-08T03:42:49.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:49.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:49.613608+0000) 2022-09-08T03:42:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:49 smithi055 ceph-mon[97033]: pgmap v11781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:42:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:42:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:49.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:49.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:49.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:50.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:50 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T03:42:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:50 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T03:42:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:50.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:50.639192+0000) 2022-09-08T03:42:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:50.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:50.613738+0000) 2022-09-08T03:42:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:50 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T03:42:50.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:50 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T03:42:51.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:51 smithi059 ceph-mon[97450]: pgmap v11782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:42:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:42:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:42:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:51.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:51.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:51.639342+0000) 2022-09-08T03:42:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:51.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:51.613871+0000) 2022-09-08T03:42:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:51 smithi055 ceph-mon[97033]: pgmap v11782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:42:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:42:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:42:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:52.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:52 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T03:42:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:52 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T03:42:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:42:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:52.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:52.639504+0000) 2022-09-08T03:42:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:52.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:52.614003+0000) 2022-09-08T03:42:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:42:52.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:52 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T03:42:52.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:52 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T03:42:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:42:53.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:53 smithi059 ceph-mon[97450]: pgmap v11783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:53.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:53.614140+0000) 2022-09-08T03:42:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:53.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:53.639659+0000) 2022-09-08T03:42:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:53 smithi055 ceph-mon[97033]: pgmap v11783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:54.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:54.614273+0000) 2022-09-08T03:42:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:54.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:54.639802+0000) 2022-09-08T03:42:55.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:55 smithi059 ceph-mon[97450]: pgmap v11784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:55.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:55.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:55.614420+0000) 2022-09-08T03:42:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:55.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:55.639952+0000) 2022-09-08T03:42:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:55 smithi055 ceph-mon[97033]: pgmap v11784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:56.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:56.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:56.640077+0000) 2022-09-08T03:42:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:56.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:56.614545+0000) 2022-09-08T03:42:57.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:57 smithi059 ceph-mon[97450]: pgmap v11785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:42:57.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:57.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:57.614701+0000) 2022-09-08T03:42:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:57.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:57.640198+0000) 2022-09-08T03:42:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:57 smithi055 ceph-mon[97033]: pgmap v11785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:42:58.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:42:58.830+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:42:58.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:58.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:58.614843+0000) 2022-09-08T03:42:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:58.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:58.640332+0000) 2022-09-08T03:42:59.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:42:59 smithi059 ceph-mon[97450]: pgmap v11786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:42:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:42:59.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:59.640472+0000) 2022-09-08T03:42:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:42:59.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:42:59.614976+0000) 2022-09-08T03:42:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:42:59 smithi055 ceph-mon[97033]: pgmap v11786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:00.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:00.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:00.640596+0000) 2022-09-08T03:43:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:00.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:00.615106+0000) 2022-09-08T03:43:01.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:01 smithi059 ceph-mon[97450]: pgmap v11787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:43:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:01.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:01.615239+0000) 2022-09-08T03:43:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:01.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:01.640726+0000) 2022-09-08T03:43:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:01 smithi055 ceph-mon[97033]: pgmap v11787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:43:02.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:02.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:02.615376+0000) 2022-09-08T03:43:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:02.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:02.640913+0000) 2022-09-08T03:43:03.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:03 smithi059 ceph-mon[97450]: pgmap v11788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:03.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:03.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:03.641048+0000) 2022-09-08T03:43:03.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:03.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:03.615519+0000) 2022-09-08T03:43:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:03 smithi055 ceph-mon[97033]: pgmap v11788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:04.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:04.641184+0000) 2022-09-08T03:43:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:04.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:04.615660+0000) 2022-09-08T03:43:05.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:05 smithi059 ceph-mon[97450]: pgmap v11789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:43:05.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:05.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:05.641317+0000) 2022-09-08T03:43:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:05.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:05.615776+0000) 2022-09-08T03:43:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:05 smithi055 ceph-mon[97033]: pgmap v11789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:43:06.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:43:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:06.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:06.641455+0000) 2022-09-08T03:43:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:06.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:06.615908+0000) 2022-09-08T03:43:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:43:07.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:07 smithi059 ceph-mon[97450]: pgmap v11790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:43:07.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:07.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:07.641585+0000) 2022-09-08T03:43:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:07.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:07.616041+0000) 2022-09-08T03:43:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:07 smithi055 ceph-mon[97033]: pgmap v11790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:43:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:08.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:08.641720+0000) 2022-09-08T03:43:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:08.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:08.616180+0000) 2022-09-08T03:43:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:09.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:09.616319+0000) 2022-09-08T03:43:09.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:09.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:09.641858+0000) 2022-09-08T03:43:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:09 smithi055 ceph-mon[97033]: pgmap v11791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:43:10.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:09 smithi059 ceph-mon[97450]: pgmap v11791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:43:10.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:10.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:10.642050+0000) 2022-09-08T03:43:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:10.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:10.616527+0000) 2022-09-08T03:43:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:10 smithi055 ceph-mon[97033]: osdmap e815: 12 total, 11 up, 11 in 2022-09-08T03:43:11.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:10 smithi059 ceph-mon[97450]: osdmap e815: 12 total, 11 up, 11 in 2022-09-08T03:43:11.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:11.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:11.642208+0000) 2022-09-08T03:43:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:11.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:11.616708+0000) 2022-09-08T03:43:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:11 smithi055 ceph-mon[97033]: pgmap v11793: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:43:12.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:11 smithi059 ceph-mon[97450]: pgmap v11793: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:43:12.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:12.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:12.642341+0000) 2022-09-08T03:43:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:12.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:12.616847+0000) 2022-09-08T03:43:13.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:43:13.846+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:43:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:13.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:13.642471+0000) 2022-09-08T03:43:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:13.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:13.616971+0000) 2022-09-08T03:43:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:13 smithi055 ceph-mon[97033]: pgmap v11794: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:43:14.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:13 smithi059 ceph-mon[97450]: pgmap v11794: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:43:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:14.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:14.642602+0000) 2022-09-08T03:43:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:14.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:14.617112+0000) 2022-09-08T03:43:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:15.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:15.642710+0000) 2022-09-08T03:43:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:15.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:15.617257+0000) 2022-09-08T03:43:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:15 smithi055 ceph-mon[97033]: pgmap v11795: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:43:16.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:15 smithi059 ceph-mon[97450]: pgmap v11795: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:43:16.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:16.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:16.642843+0000) 2022-09-08T03:43:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:16.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:16.617397+0000) 2022-09-08T03:43:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:17.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:17.642979+0000) 2022-09-08T03:43:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:17.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:17.617533+0000) 2022-09-08T03:43:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:17 smithi055 ceph-mon[97033]: pgmap v11796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2022-09-08T03:43:18.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:17 smithi059 ceph-mon[97450]: pgmap v11796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2022-09-08T03:43:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:18.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:18.643112+0000) 2022-09-08T03:43:18.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:18.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:18.617681+0000) 2022-09-08T03:43:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:18 smithi055 ceph-mon[97033]: pgmap v11797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2022-09-08T03:43:19.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:18 smithi059 ceph-mon[97450]: pgmap v11797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2022-09-08T03:43:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:19.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:19.643245+0000) 2022-09-08T03:43:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:19.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:19.617815+0000) 2022-09-08T03:43:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:19 smithi055 ceph-mon[97033]: osdmap e816: 12 total, 11 up, 11 in 2022-09-08T03:43:20.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:19 smithi059 ceph-mon[97450]: osdmap e816: 12 total, 11 up, 11 in 2022-09-08T03:43:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:20.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:20.643380+0000) 2022-09-08T03:43:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:20.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:20.617954+0000) 2022-09-08T03:43:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:20 smithi055 ceph-mon[97033]: pgmap v11799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2022-09-08T03:43:21.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:20 smithi059 ceph-mon[97450]: pgmap v11799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2022-09-08T03:43:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:21.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:21.643512+0000) 2022-09-08T03:43:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:21.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:21.618088+0000) 2022-09-08T03:43:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:22.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:22.643667+0000) 2022-09-08T03:43:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:22.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:22.618228+0000) 2022-09-08T03:43:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:22 smithi055 ceph-mon[97033]: pgmap v11800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2022-09-08T03:43:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:22 smithi059 ceph-mon[97450]: pgmap v11800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2022-09-08T03:43:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:23.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:23.643802+0000) 2022-09-08T03:43:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:23.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:23.618364+0000) 2022-09-08T03:43:24.871 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:24.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:24.643940+0000) 2022-09-08T03:43:24.872 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:24.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:24.618501+0000) 2022-09-08T03:43:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:24 smithi055 ceph-mon[97033]: pgmap v11801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2022-09-08T03:43:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:24 smithi059 ceph-mon[97450]: pgmap v11801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2022-09-08T03:43:25.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:25.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:25.644076+0000) 2022-09-08T03:43:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:25.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:25.618645+0000) 2022-09-08T03:43:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:26.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:26.644213+0000) 2022-09-08T03:43:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:26.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:26.618777+0000) 2022-09-08T03:43:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:26 smithi055 ceph-mon[97033]: pgmap v11802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:43:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:26 smithi059 ceph-mon[97450]: pgmap v11802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:43:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:27.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:27.644393+0000) 2022-09-08T03:43:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:27.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:27.618931+0000) 2022-09-08T03:43:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:28.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:28.644533+0000) 2022-09-08T03:43:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:28.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:28.619095+0000) 2022-09-08T03:43:28.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:43:28.850+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:43:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:28 smithi055 ceph-mon[97033]: pgmap v11803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:43:29.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:28 smithi059 ceph-mon[97450]: pgmap v11803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:43:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:29.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:29.644691+0000) 2022-09-08T03:43:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:29.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:29.619245+0000) 2022-09-08T03:43:30.647 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:30.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:30.619382+0000) 2022-09-08T03:43:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:30.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:30.644828+0000) 2022-09-08T03:43:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:30 smithi055 ceph-mon[97033]: pgmap v11804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T03:43:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:30 smithi059 ceph-mon[97450]: pgmap v11804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T03:43:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:31.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:31.644977+0000) 2022-09-08T03:43:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:31.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:31.619513+0000) 2022-09-08T03:43:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:32.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:32.645133+0000) 2022-09-08T03:43:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:32.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:32.619652+0000) 2022-09-08T03:43:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:32 smithi055 ceph-mon[97033]: pgmap v11805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:32 smithi059 ceph-mon[97450]: pgmap v11805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:33.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:33.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:33.645279+0000) 2022-09-08T03:43:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:33.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:33.619767+0000) 2022-09-08T03:43:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:34.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:34.645427+0000) 2022-09-08T03:43:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:34.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:34.619901+0000) 2022-09-08T03:43:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:34 smithi055 ceph-mon[97033]: pgmap v11806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:35.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:34 smithi059 ceph-mon[97450]: pgmap v11806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:35.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:35.645577+0000) 2022-09-08T03:43:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:35.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:35.620048+0000) 2022-09-08T03:43:36.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:36.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:36.645724+0000) 2022-09-08T03:43:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:36.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:36.620193+0000) 2022-09-08T03:43:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:36 smithi055 ceph-mon[97033]: pgmap v11807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:43:37.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:36 smithi059 ceph-mon[97450]: pgmap v11807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:43:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:37.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:37.645879+0000) 2022-09-08T03:43:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:37.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:37.620357+0000) 2022-09-08T03:43:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:38.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:38.646035+0000) 2022-09-08T03:43:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:38.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:38.620515+0000) 2022-09-08T03:43:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:38 smithi055 ceph-mon[97033]: pgmap v11808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:39.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:38 smithi059 ceph-mon[97450]: pgmap v11808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:39.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:39.646179+0000) 2022-09-08T03:43:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:39.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:39.620661+0000) 2022-09-08T03:43:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:40.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:40.646293+0000) 2022-09-08T03:43:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:40.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:40.620772+0000) 2022-09-08T03:43:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:40 smithi055 ceph-mon[97033]: pgmap v11809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:43:41.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:40 smithi059 ceph-mon[97450]: pgmap v11809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:43:41.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:41.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:41.646423+0000) 2022-09-08T03:43:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:41.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:41.620905+0000) 2022-09-08T03:43:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:42.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:42.646567+0000) 2022-09-08T03:43:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:42.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:42.621035+0000) 2022-09-08T03:43:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:42 smithi055 ceph-mon[97033]: pgmap v11810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:43.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:42 smithi059 ceph-mon[97450]: pgmap v11810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:43.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:43.646713+0000) 2022-09-08T03:43:43.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:43:43.864+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:43:43.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:43.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:43.621189+0000) 2022-09-08T03:43:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:44.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:44.646847+0000) 2022-09-08T03:43:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:44.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:44.621330+0000) 2022-09-08T03:43:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:44 smithi055 ceph-mon[97033]: pgmap v11811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:45.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:44 smithi059 ceph-mon[97450]: pgmap v11811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:45.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:45.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:45.646977+0000) 2022-09-08T03:43:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:45.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:45.621464+0000) 2022-09-08T03:43:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:46.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:46.647110+0000) 2022-09-08T03:43:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:46.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:46.621594+0000) 2022-09-08T03:43:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:46 smithi055 ceph-mon[97033]: pgmap v11812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:43:47.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:46 smithi059 ceph-mon[97450]: pgmap v11812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:43:47.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:47.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:47.647246+0000) 2022-09-08T03:43:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:47.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:47.621711+0000) 2022-09-08T03:43:48.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:48.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:48.647394+0000) 2022-09-08T03:43:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:48.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:48.621850+0000) 2022-09-08T03:43:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:48 smithi055 ceph-mon[97033]: pgmap v11813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:49.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:48 smithi059 ceph-mon[97450]: pgmap v11813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:49.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:49.647540+0000) 2022-09-08T03:43:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:49.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:49.621965+0000) 2022-09-08T03:43:50.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:50.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:50.647694+0000) 2022-09-08T03:43:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:50.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:50.622093+0000) 2022-09-08T03:43:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:50 smithi059 ceph-mon[97450]: pgmap v11814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:43:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:50 smithi055 ceph-mon[97033]: pgmap v11814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:43:51.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:51.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:51.647847+0000) 2022-09-08T03:43:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:51.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:51.622234+0000) 2022-09-08T03:43:52.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:43:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:43:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:43:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:43:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:43:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:43:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:52.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:52.622364+0000) 2022-09-08T03:43:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:52.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:52.647995+0000) 2022-09-08T03:43:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:53 smithi055 ceph-mon[97033]: pgmap v11815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:53.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:43:53.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:53 smithi059 ceph-mon[97450]: pgmap v11815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:43:53.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:53.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:53.648146+0000) 2022-09-08T03:43:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:53.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:53.622508+0000) 2022-09-08T03:43:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:54.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:54.648293+0000) 2022-09-08T03:43:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:54.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:54.622671+0000) 2022-09-08T03:43:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:55 smithi055 ceph-mon[97033]: pgmap v11816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:55.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:55 smithi059 ceph-mon[97450]: pgmap v11816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:55.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:55.648431+0000) 2022-09-08T03:43:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:55.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:55.622786+0000) 2022-09-08T03:43:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:56.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:56.648565+0000) 2022-09-08T03:43:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:56.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:56.622900+0000) 2022-09-08T03:43:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:57 smithi055 ceph-mon[97033]: pgmap v11817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:43:57.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:57 smithi059 ceph-mon[97450]: pgmap v11817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:43:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:57.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:57.648690+0000) 2022-09-08T03:43:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:57.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:57.623032+0000) 2022-09-08T03:43:58.877 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:58.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:58.648839+0000) 2022-09-08T03:43:58.879 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:58.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:58.623170+0000) 2022-09-08T03:43:59.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:43:58.876+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:43:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:43:59 smithi055 ceph-mon[97033]: pgmap v11818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:59.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:43:59 smithi059 ceph-mon[97450]: pgmap v11818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:43:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:43:59.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:59.648990+0000) 2022-09-08T03:43:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:43:59.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:43:59.623305+0000) 2022-09-08T03:44:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:00.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:00.649132+0000) 2022-09-08T03:44:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:00.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:00.623424+0000) 2022-09-08T03:44:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:01 smithi055 ceph-mon[97033]: pgmap v11819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:44:01.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:01 smithi059 ceph-mon[97450]: pgmap v11819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:44:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:01.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:01.649270+0000) 2022-09-08T03:44:01.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:01.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:01.623558+0000) 2022-09-08T03:44:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:02.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:02.649422+0000) 2022-09-08T03:44:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:02.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:02.623691+0000) 2022-09-08T03:44:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:03 smithi055 ceph-mon[97033]: pgmap v11820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:03.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:03 smithi059 ceph-mon[97450]: pgmap v11820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:03.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:03.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:03.649565+0000) 2022-09-08T03:44:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:03.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:03.623827+0000) 2022-09-08T03:44:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:04.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:04.649710+0000) 2022-09-08T03:44:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:04.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:04.623960+0000) 2022-09-08T03:44:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:05 smithi055 ceph-mon[97033]: pgmap v11821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:05.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:05 smithi059 ceph-mon[97450]: pgmap v11821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:05.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:05.649825+0000) 2022-09-08T03:44:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:05.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:05.624095+0000) 2022-09-08T03:44:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:44:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:44:06.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:44:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:44:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:06.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:06.649960+0000) 2022-09-08T03:44:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:06.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:06.624225+0000) 2022-09-08T03:44:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:07 smithi055 ceph-mon[97033]: pgmap v11822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:44:07.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:07 smithi059 ceph-mon[97450]: pgmap v11822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:44:07.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:07.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:07.650092+0000) 2022-09-08T03:44:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:07.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:07.624354+0000) 2022-09-08T03:44:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:08.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:08.650225+0000) 2022-09-08T03:44:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:08.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:08.624502+0000) 2022-09-08T03:44:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:09 smithi055 ceph-mon[97033]: pgmap v11823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:44:09.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:09 smithi059 ceph-mon[97450]: pgmap v11823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:44:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:09.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:09.650364+0000) 2022-09-08T03:44:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:09.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:09.624649+0000) 2022-09-08T03:44:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:10 smithi055 ceph-mon[97033]: osdmap e817: 12 total, 11 up, 11 in 2022-09-08T03:44:10.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:10 smithi059 ceph-mon[97450]: osdmap e817: 12 total, 11 up, 11 in 2022-09-08T03:44:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:10.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:10.650497+0000) 2022-09-08T03:44:10.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:10.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:10.624782+0000) 2022-09-08T03:44:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:11 smithi055 ceph-mon[97033]: pgmap v11825: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:44:11.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:11 smithi059 ceph-mon[97450]: pgmap v11825: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:44:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:11.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:11.650644+0000) 2022-09-08T03:44:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:11.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:11.624916+0000) 2022-09-08T03:44:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:12.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:12.650791+0000) 2022-09-08T03:44:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:12.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:12.625047+0000) 2022-09-08T03:44:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:13 smithi055 ceph-mon[97033]: pgmap v11826: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:44:13.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:13 smithi059 ceph-mon[97450]: pgmap v11826: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:44:13.885 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:13.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:13.650919+0000) 2022-09-08T03:44:13.886 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:13.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:13.625184+0000) 2022-09-08T03:44:14.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:44:13.885+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:44:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:14.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:14.651057+0000) 2022-09-08T03:44:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:14.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:14.625297+0000) 2022-09-08T03:44:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:14 smithi055 ceph-mon[97033]: pgmap v11827: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:44:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:14 smithi055 ceph-mon[97033]: osdmap e818: 12 total, 11 up, 11 in 2022-09-08T03:44:15.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:14 smithi059 ceph-mon[97450]: pgmap v11827: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:44:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:14 smithi059 ceph-mon[97450]: osdmap e818: 12 total, 11 up, 11 in 2022-09-08T03:44:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:15.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:15.651183+0000) 2022-09-08T03:44:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:15.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:15.625430+0000) 2022-09-08T03:44:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:16.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:16.651322+0000) 2022-09-08T03:44:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:16.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:16.625560+0000) 2022-09-08T03:44:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:16 smithi055 ceph-mon[97033]: pgmap v11829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:44:17.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:16 smithi059 ceph-mon[97450]: pgmap v11829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:44:17.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:17.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:17.651441+0000) 2022-09-08T03:44:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:17.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:17.625698+0000) 2022-09-08T03:44:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:18.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:18.651582+0000) 2022-09-08T03:44:18.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:18.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:18.625818+0000) 2022-09-08T03:44:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:18 smithi055 ceph-mon[97033]: pgmap v11830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-08T03:44:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:18 smithi059 ceph-mon[97450]: pgmap v11830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-08T03:44:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:19.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:19.651693+0000) 2022-09-08T03:44:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:19.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:19.625929+0000) 2022-09-08T03:44:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:19 smithi055 ceph-mon[97033]: osdmap e819: 12 total, 11 up, 11 in 2022-09-08T03:44:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:19 smithi059 ceph-mon[97450]: osdmap e819: 12 total, 11 up, 11 in 2022-09-08T03:44:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:20.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:20.651821+0000) 2022-09-08T03:44:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:20.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:20.626063+0000) 2022-09-08T03:44:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:20 smithi055 ceph-mon[97033]: pgmap v11832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T03:44:21.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:20 smithi059 ceph-mon[97450]: pgmap v11832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T03:44:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:21.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:21.651946+0000) 2022-09-08T03:44:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:21.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:21.626195+0000) 2022-09-08T03:44:22.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:22.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:22.652091+0000) 2022-09-08T03:44:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:22.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:22.626323+0000) 2022-09-08T03:44:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:22 smithi055 ceph-mon[97033]: pgmap v11833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T03:44:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:22 smithi059 ceph-mon[97450]: pgmap v11833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T03:44:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:23.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:23.652221+0000) 2022-09-08T03:44:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:23.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:23.626466+0000) 2022-09-08T03:44:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:24.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:24.652355+0000) 2022-09-08T03:44:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:24.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:24.626598+0000) 2022-09-08T03:44:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:24 smithi055 ceph-mon[97033]: pgmap v11834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:44:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:24 smithi059 ceph-mon[97450]: pgmap v11834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:44:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:25.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:25.652588+0000) 2022-09-08T03:44:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:25.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:25.626770+0000) 2022-09-08T03:44:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:26.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:26.652786+0000) 2022-09-08T03:44:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:26.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:26.626918+0000) 2022-09-08T03:44:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:26 smithi055 ceph-mon[97033]: pgmap v11835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:44:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:26 smithi059 ceph-mon[97450]: pgmap v11835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:44:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:27.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:27.652901+0000) 2022-09-08T03:44:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:27.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:27.627051+0000) 2022-09-08T03:44:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:28.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:28.653032+0000) 2022-09-08T03:44:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:28.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:28.627192+0000) 2022-09-08T03:44:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:28 smithi055 ceph-mon[97033]: pgmap v11836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:44:29.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:28 smithi059 ceph-mon[97450]: pgmap v11836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:44:29.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:44:28.899+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:44:29.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:29.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:29.627332+0000) 2022-09-08T03:44:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:29.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:29.653144+0000) 2022-09-08T03:44:30.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:30.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:30.653278+0000) 2022-09-08T03:44:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:30.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:30.627472+0000) 2022-09-08T03:44:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:30 smithi055 ceph-mon[97033]: pgmap v11837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T03:44:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:30 smithi059 ceph-mon[97450]: pgmap v11837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T03:44:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:31.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:31.653405+0000) 2022-09-08T03:44:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:31.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:31.627612+0000) 2022-09-08T03:44:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:32.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:32.627753+0000) 2022-09-08T03:44:32.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:32.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:32.653549+0000) 2022-09-08T03:44:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:32 smithi055 ceph-mon[97033]: pgmap v11838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:32 smithi059 ceph-mon[97450]: pgmap v11838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:33.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:33.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:33.653682+0000) 2022-09-08T03:44:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:33.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:33.627893+0000) 2022-09-08T03:44:34.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:34.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:34.653808+0000) 2022-09-08T03:44:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:34.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:34.628030+0000) 2022-09-08T03:44:35.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:34 smithi059 ceph-mon[97450]: pgmap v11839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:34 smithi055 ceph-mon[97033]: pgmap v11839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:35.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:35.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:35.653937+0000) 2022-09-08T03:44:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:35.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:35.628178+0000) 2022-09-08T03:44:36.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:36.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:36.654080+0000) 2022-09-08T03:44:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:36.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:36.628315+0000) 2022-09-08T03:44:37.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:36 smithi059 ceph-mon[97450]: pgmap v11840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:44:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:36 smithi055 ceph-mon[97033]: pgmap v11840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:44:37.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:37.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:37.654225+0000) 2022-09-08T03:44:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:37.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:37.628457+0000) 2022-09-08T03:44:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:38.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:38.654371+0000) 2022-09-08T03:44:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:38.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:38.628605+0000) 2022-09-08T03:44:39.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:38 smithi059 ceph-mon[97450]: pgmap v11841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:38 smithi055 ceph-mon[97033]: pgmap v11841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:39.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:39.654511+0000) 2022-09-08T03:44:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:39.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:39.628748+0000) 2022-09-08T03:44:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:40.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:40.654650+0000) 2022-09-08T03:44:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:40.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:40.628881+0000) 2022-09-08T03:44:41.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:40 smithi059 ceph-mon[97450]: pgmap v11842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:44:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:40 smithi055 ceph-mon[97033]: pgmap v11842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:44:41.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:41.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:41.654781+0000) 2022-09-08T03:44:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:41.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:41.628998+0000) 2022-09-08T03:44:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:42.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:42.654899+0000) 2022-09-08T03:44:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:42.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:42.629139+0000) 2022-09-08T03:44:43.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:42 smithi059 ceph-mon[97450]: pgmap v11843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:42 smithi055 ceph-mon[97033]: pgmap v11843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:43.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:43.629273+0000) 2022-09-08T03:44:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:43.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:43.655029+0000) 2022-09-08T03:44:44.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:44:43.915+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:44:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:44.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:44.655153+0000) 2022-09-08T03:44:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:44.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:44.629403+0000) 2022-09-08T03:44:45.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:44 smithi059 ceph-mon[97450]: pgmap v11844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:44 smithi055 ceph-mon[97033]: pgmap v11844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:45.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:45.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:45.655284+0000) 2022-09-08T03:44:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:45.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:45.629538+0000) 2022-09-08T03:44:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:46.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:46.655421+0000) 2022-09-08T03:44:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:46.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:46.629676+0000) 2022-09-08T03:44:47.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:47 smithi059 ceph-mon[97450]: pgmap v11845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:44:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:47 smithi055 ceph-mon[97033]: pgmap v11845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:44:47.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:47.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:47.655556+0000) 2022-09-08T03:44:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:47.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:47.629818+0000) 2022-09-08T03:44:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:48.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:48.655690+0000) 2022-09-08T03:44:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:48.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:48.629957+0000) 2022-09-08T03:44:49.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:49 smithi059 ceph-mon[97450]: pgmap v11846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:49 smithi055 ceph-mon[97033]: pgmap v11846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:49.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:49.655815+0000) 2022-09-08T03:44:49.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:49.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:49.630091+0000) 2022-09-08T03:44:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:50.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:50.655967+0000) 2022-09-08T03:44:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:50.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:50.630203+0000) 2022-09-08T03:44:51.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:51 smithi059 ceph-mon[97450]: pgmap v11847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:44:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:51 smithi055 ceph-mon[97033]: pgmap v11847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:44:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:51.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:51.656108+0000) 2022-09-08T03:44:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:51.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:51.630336+0000) 2022-09-08T03:44:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:52.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:52.656252+0000) 2022-09-08T03:44:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:52.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:52.630468+0000) 2022-09-08T03:44:53.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:53 smithi059 ceph-mon[97450]: pgmap v11848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:44:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:44:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:44:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:44:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:44:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:53 smithi055 ceph-mon[97033]: pgmap v11848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:44:53.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:44:53.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:44:53.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:44:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:44:53.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:53.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:53.656390+0000) 2022-09-08T03:44:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:53.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:53.630603+0000) 2022-09-08T03:44:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:44:54.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:44:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:54.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:54.656522+0000) 2022-09-08T03:44:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:54.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:54.630735+0000) 2022-09-08T03:44:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:55 smithi055 ceph-mon[97033]: pgmap v11849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:55.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:55 smithi059 ceph-mon[97450]: pgmap v11849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:55.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:55.630871+0000) 2022-09-08T03:44:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:55.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:55.656660+0000) 2022-09-08T03:44:56.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:56.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:56.656779+0000) 2022-09-08T03:44:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:56.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:56.630981+0000) 2022-09-08T03:44:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:57 smithi055 ceph-mon[97033]: pgmap v11850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:44:57.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:57 smithi059 ceph-mon[97450]: pgmap v11850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:44:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:57.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:57.631133+0000) 2022-09-08T03:44:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:57.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:57.656960+0000) 2022-09-08T03:44:58.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:58.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:58.657108+0000) 2022-09-08T03:44:58.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:58.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:58.631275+0000) 2022-09-08T03:44:59.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:44:58.916+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:44:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:44:59 smithi055 ceph-mon[97033]: pgmap v11851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:59.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:44:59 smithi059 ceph-mon[97450]: pgmap v11851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:44:59.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:44:59.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:59.657254+0000) 2022-09-08T03:44:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:44:59.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:44:59.631408+0000) 2022-09-08T03:45:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:00.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:00.657386+0000) 2022-09-08T03:45:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:00.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:00.631517+0000) 2022-09-08T03:45:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:01 smithi055 ceph-mon[97033]: pgmap v11852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:45:01.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:01 smithi059 ceph-mon[97450]: pgmap v11852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:45:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:01.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:01.657523+0000) 2022-09-08T03:45:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:01.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:01.631672+0000) 2022-09-08T03:45:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:02.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:02.657677+0000) 2022-09-08T03:45:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:02.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:02.631805+0000) 2022-09-08T03:45:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:03 smithi055 ceph-mon[97033]: pgmap v11853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:03.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:03 smithi059 ceph-mon[97450]: pgmap v11853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:03.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:03.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:03.657806+0000) 2022-09-08T03:45:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:03.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:03.631918+0000) 2022-09-08T03:45:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:04.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:04.657935+0000) 2022-09-08T03:45:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:04.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:04.632059+0000) 2022-09-08T03:45:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:05 smithi055 ceph-mon[97033]: pgmap v11854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:05.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:05 smithi059 ceph-mon[97450]: pgmap v11854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:05.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:05.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:05.658088+0000) 2022-09-08T03:45:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:05.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:05.632194+0000) 2022-09-08T03:45:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:45:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:45:06.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:45:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:45:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:06.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:06.658220+0000) 2022-09-08T03:45:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:06.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:06.632326+0000) 2022-09-08T03:45:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:07 smithi055 ceph-mon[97033]: pgmap v11855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:45:07.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:07 smithi059 ceph-mon[97450]: pgmap v11855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:45:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:07.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:07.658363+0000) 2022-09-08T03:45:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:07.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:07.632460+0000) 2022-09-08T03:45:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:08.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:08.658489+0000) 2022-09-08T03:45:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:08.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:08.632600+0000) 2022-09-08T03:45:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:09 smithi055 ceph-mon[97033]: pgmap v11856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:45:09.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:09 smithi059 ceph-mon[97450]: pgmap v11856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:45:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:09.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:09.658618+0000) 2022-09-08T03:45:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:09.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:09.632732+0000) 2022-09-08T03:45:10.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:10 smithi059 ceph-mon[97450]: osdmap e820: 12 total, 11 up, 11 in 2022-09-08T03:45:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:10 smithi055 ceph-mon[97033]: osdmap e820: 12 total, 11 up, 11 in 2022-09-08T03:45:10.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:10.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:10.658744+0000) 2022-09-08T03:45:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:10.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:10.632862+0000) 2022-09-08T03:45:11.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:11 smithi059 ceph-mon[97450]: pgmap v11858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:45:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:11 smithi055 ceph-mon[97033]: pgmap v11858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:45:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:11.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:11.658876+0000) 2022-09-08T03:45:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:11.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:11.632985+0000) 2022-09-08T03:45:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:12.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:12.659014+0000) 2022-09-08T03:45:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:12.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:12.633118+0000) 2022-09-08T03:45:13.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:13 smithi059 ceph-mon[97450]: pgmap v11859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:45:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:13 smithi055 ceph-mon[97033]: pgmap v11859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:45:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:13.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:13.659145+0000) 2022-09-08T03:45:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:13.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:13.633231+0000) 2022-09-08T03:45:14.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:45:13.929+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:45:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:14.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:14.659283+0000) 2022-09-08T03:45:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:14.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:14.633342+0000) 2022-09-08T03:45:15.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:15 smithi059 ceph-mon[97450]: pgmap v11860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:45:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:15 smithi055 ceph-mon[97033]: pgmap v11860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:45:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:15.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:15.659414+0000) 2022-09-08T03:45:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:15.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:15.633480+0000) 2022-09-08T03:45:16.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:16.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:16.659553+0000) 2022-09-08T03:45:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:16.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:16.633612+0000) 2022-09-08T03:45:17.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:17 smithi059 ceph-mon[97450]: pgmap v11861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:45:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:17 smithi055 ceph-mon[97033]: pgmap v11861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:45:17.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:17.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:17.659685+0000) 2022-09-08T03:45:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:17.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:17.633751+0000) 2022-09-08T03:45:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:18.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:18.659821+0000) 2022-09-08T03:45:18.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:18.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:18.633956+0000) 2022-09-08T03:45:19.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:19 smithi059 ceph-mon[97450]: pgmap v11862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:45:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:19 smithi059 ceph-mon[97450]: osdmap e821: 12 total, 11 up, 11 in 2022-09-08T03:45:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:19 smithi055 ceph-mon[97033]: pgmap v11862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:45:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:19 smithi055 ceph-mon[97033]: osdmap e821: 12 total, 11 up, 11 in 2022-09-08T03:45:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:19.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:19.659946+0000) 2022-09-08T03:45:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:19.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:19.634107+0000) 2022-09-08T03:45:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:20.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:20.660053+0000) 2022-09-08T03:45:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:20.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:20.634240+0000) 2022-09-08T03:45:21.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:21 smithi059 ceph-mon[97450]: pgmap v11864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:45:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:21 smithi055 ceph-mon[97033]: pgmap v11864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:45:21.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:21.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:21.660187+0000) 2022-09-08T03:45:21.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:21.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:21.634374+0000) 2022-09-08T03:45:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:22.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:22.660322+0000) 2022-09-08T03:45:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:22.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:22.634506+0000) 2022-09-08T03:45:23.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:23 smithi059 ceph-mon[97450]: pgmap v11865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:45:23.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:23 smithi055 ceph-mon[97033]: pgmap v11865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:45:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:23.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:23.660456+0000) 2022-09-08T03:45:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:23.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:23.634664+0000) 2022-09-08T03:45:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:24.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:24.660593+0000) 2022-09-08T03:45:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:24.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:24.634803+0000) 2022-09-08T03:45:25.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:25 smithi059 ceph-mon[97450]: pgmap v11866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:45:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:25 smithi055 ceph-mon[97033]: pgmap v11866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:45:25.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:25.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:25.660724+0000) 2022-09-08T03:45:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:25.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:25.634942+0000) 2022-09-08T03:45:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:26.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:26.660833+0000) 2022-09-08T03:45:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:26.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:26.635071+0000) 2022-09-08T03:45:27.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:27 smithi059 ceph-mon[97450]: pgmap v11867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:45:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:27 smithi055 ceph-mon[97033]: pgmap v11867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:45:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:27.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:27.635215+0000) 2022-09-08T03:45:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:27.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:27.660956+0000) 2022-09-08T03:45:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:28.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:28.661094+0000) 2022-09-08T03:45:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:28.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:28.635338+0000) 2022-09-08T03:45:29.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:45:28.945+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:45:29.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:29 smithi059 ceph-mon[97450]: pgmap v11868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:45:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:29 smithi055 ceph-mon[97033]: pgmap v11868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:45:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:29.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:29.661284+0000) 2022-09-08T03:45:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:29.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:29.635543+0000) 2022-09-08T03:45:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:30.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:30.661416+0000) 2022-09-08T03:45:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:30.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:30.635700+0000) 2022-09-08T03:45:31.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:31 smithi059 ceph-mon[97450]: pgmap v11869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T03:45:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:31 smithi055 ceph-mon[97033]: pgmap v11869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T03:45:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:31.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:31.661546+0000) 2022-09-08T03:45:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:31.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:31.635811+0000) 2022-09-08T03:45:32.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:32.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:32.635926+0000) 2022-09-08T03:45:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:32.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:32.661694+0000) 2022-09-08T03:45:33.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:33 smithi059 ceph-mon[97450]: pgmap v11870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:33 smithi055 ceph-mon[97033]: pgmap v11870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:33.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:33.661818+0000) 2022-09-08T03:45:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:33.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:33.636053+0000) 2022-09-08T03:45:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:34.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:34.636186+0000) 2022-09-08T03:45:34.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:34.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:34.661959+0000) 2022-09-08T03:45:35.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:35 smithi059 ceph-mon[97450]: pgmap v11871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:35 smithi055 ceph-mon[97033]: pgmap v11871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:35.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:35.662095+0000) 2022-09-08T03:45:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:35.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:35.636328+0000) 2022-09-08T03:45:36.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:36.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:36.662223+0000) 2022-09-08T03:45:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:36.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:36.636467+0000) 2022-09-08T03:45:37.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:37 smithi059 ceph-mon[97450]: pgmap v11872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:45:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:37 smithi055 ceph-mon[97033]: pgmap v11872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:45:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:37.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:37.662356+0000) 2022-09-08T03:45:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:37.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:37.636606+0000) 2022-09-08T03:45:38.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:38.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:38.636749+0000) 2022-09-08T03:45:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:38.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:38.662491+0000) 2022-09-08T03:45:39.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:39 smithi059 ceph-mon[97450]: pgmap v11873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:39 smithi055 ceph-mon[97033]: pgmap v11873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:39.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:39.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:39.662623+0000) 2022-09-08T03:45:39.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:39.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:39.636886+0000) 2022-09-08T03:45:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:40.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:40.662773+0000) 2022-09-08T03:45:40.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:40.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:40.637023+0000) 2022-09-08T03:45:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:41 smithi055 ceph-mon[97033]: pgmap v11874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:45:41.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:41 smithi059 ceph-mon[97450]: pgmap v11874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:45:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:41.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:41.662907+0000) 2022-09-08T03:45:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:41.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:41.637160+0000) 2022-09-08T03:45:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:42.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:42.637293+0000) 2022-09-08T03:45:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:42.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:42.663052+0000) 2022-09-08T03:45:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:43 smithi055 ceph-mon[97033]: pgmap v11875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:43.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:43 smithi059 ceph-mon[97450]: pgmap v11875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:43.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:43.663190+0000) 2022-09-08T03:45:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:43.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:43.637429+0000) 2022-09-08T03:45:44.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:45:43.960+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:45:44.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:44.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:44.663325+0000) 2022-09-08T03:45:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:44.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:44.637566+0000) 2022-09-08T03:45:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:45 smithi055 ceph-mon[97033]: pgmap v11876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:45.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:45 smithi059 ceph-mon[97450]: pgmap v11876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:45.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:45.663457+0000) 2022-09-08T03:45:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:45.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:45.637711+0000) 2022-09-08T03:45:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:46.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:46.663588+0000) 2022-09-08T03:45:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:46.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:46.637844+0000) 2022-09-08T03:45:47.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:47 smithi055 ceph-mon[97033]: pgmap v11877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:45:47.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:47 smithi059 ceph-mon[97450]: pgmap v11877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:45:47.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:47.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:47.663722+0000) 2022-09-08T03:45:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:47.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:47.637981+0000) 2022-09-08T03:45:48.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:48.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:48.663854+0000) 2022-09-08T03:45:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:48.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:48.638120+0000) 2022-09-08T03:45:49.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:49 smithi055 ceph-mon[97033]: pgmap v11878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:49.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:49 smithi059 ceph-mon[97450]: pgmap v11878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:49.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:49.663986+0000) 2022-09-08T03:45:49.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:49.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:49.638257+0000) 2022-09-08T03:45:50.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:50.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:50.664129+0000) 2022-09-08T03:45:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:50.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:50.638391+0000) 2022-09-08T03:45:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:51 smithi055 ceph-mon[97033]: pgmap v11879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:45:51.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:51 smithi059 ceph-mon[97450]: pgmap v11879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:45:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:51.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:51.664265+0000) 2022-09-08T03:45:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:51.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:51.638525+0000) 2022-09-08T03:45:52.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:52.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:52.664404+0000) 2022-09-08T03:45:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:52.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:52.638668+0000) 2022-09-08T03:45:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:53 smithi055 ceph-mon[97033]: pgmap v11880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:45:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:45:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:45:53.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:53 smithi059 ceph-mon[97450]: pgmap v11880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:45:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:45:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:45:53.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:53.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:53.664537+0000) 2022-09-08T03:45:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:53.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:53.638808+0000) 2022-09-08T03:45:54.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:54.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:54.664683+0000) 2022-09-08T03:45:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:54.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:54.638934+0000) 2022-09-08T03:45:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:45:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:54 smithi055 ceph-mon[97033]: pgmap v11881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:55.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:45:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:54 smithi059 ceph-mon[97450]: pgmap v11881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:55.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:55.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:55.664825+0000) 2022-09-08T03:45:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:55.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:55.639069+0000) 2022-09-08T03:45:56.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:56.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:56.664954+0000) 2022-09-08T03:45:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:56.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:56.639178+0000) 2022-09-08T03:45:57.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:56 smithi059 ceph-mon[97450]: pgmap v11882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:45:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:56 smithi055 ceph-mon[97033]: pgmap v11882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:45:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:57.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:57.665099+0000) 2022-09-08T03:45:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:57.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:57.639320+0000) 2022-09-08T03:45:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:58.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:58.665231+0000) 2022-09-08T03:45:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:58.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:58.639459+0000) 2022-09-08T03:45:59.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:45:58 smithi059 ceph-mon[97450]: pgmap v11883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:59.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:45:58.961+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:45:59.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:45:58 smithi055 ceph-mon[97033]: pgmap v11883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:45:59.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:45:59.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:59.665363+0000) 2022-09-08T03:45:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:45:59.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:45:59.639592+0000) 2022-09-08T03:46:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:00.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:00.665586+0000) 2022-09-08T03:46:00.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:00.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:00.639713+0000) 2022-09-08T03:46:01.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:00 smithi059 ceph-mon[97450]: pgmap v11884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:46:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:00 smithi055 ceph-mon[97033]: pgmap v11884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:46:01.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:01.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:01.665762+0000) 2022-09-08T03:46:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:01.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:01.639870+0000) 2022-09-08T03:46:02.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:02.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:02.639986+0000) 2022-09-08T03:46:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:02.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:02.665903+0000) 2022-09-08T03:46:03.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:02 smithi059 ceph-mon[97450]: pgmap v11885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:02 smithi055 ceph-mon[97033]: pgmap v11885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:03.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:03.640119+0000) 2022-09-08T03:46:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:03.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:03.666043+0000) 2022-09-08T03:46:04.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:04.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:04.640251+0000) 2022-09-08T03:46:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:04.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:04.666177+0000) 2022-09-08T03:46:05.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:04 smithi059 ceph-mon[97450]: pgmap v11886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:04 smithi055 ceph-mon[97033]: pgmap v11886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:05.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:05.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:05.640386+0000) 2022-09-08T03:46:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:05.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:05.666316+0000) 2022-09-08T03:46:06.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:46:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:46:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:46:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:46:06.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:06.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:06.640516+0000) 2022-09-08T03:46:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:06.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:06.666447+0000) 2022-09-08T03:46:07.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:06 smithi059 ceph-mon[97450]: pgmap v11887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:46:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:06 smithi055 ceph-mon[97033]: pgmap v11887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:46:07.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:07.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:07.640681+0000) 2022-09-08T03:46:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:07.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:07.666577+0000) 2022-09-08T03:46:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:08.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:08.666700+0000) 2022-09-08T03:46:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:08.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:08.640800+0000) 2022-09-08T03:46:09.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:09 smithi059 ceph-mon[97450]: pgmap v11888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:46:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:09 smithi055 ceph-mon[97033]: pgmap v11888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:46:09.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:09.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:09.666818+0000) 2022-09-08T03:46:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:09.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:09.640933+0000) 2022-09-08T03:46:10.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:10 smithi059 ceph-mon[97450]: osdmap e822: 12 total, 11 up, 11 in 2022-09-08T03:46:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:10 smithi055 ceph-mon[97033]: osdmap e822: 12 total, 11 up, 11 in 2022-09-08T03:46:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:10.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:10.641058+0000) 2022-09-08T03:46:10.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:10.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:10.666957+0000) 2022-09-08T03:46:11.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:11 smithi059 ceph-mon[97450]: pgmap v11890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:46:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:11 smithi055 ceph-mon[97033]: pgmap v11890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:46:11.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:11.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:11.641213+0000) 2022-09-08T03:46:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:11.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:11.667059+0000) 2022-09-08T03:46:12.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:12.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:12.667195+0000) 2022-09-08T03:46:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:12.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:12.641366+0000) 2022-09-08T03:46:13.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:13 smithi059 ceph-mon[97450]: pgmap v11891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:46:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:13 smithi055 ceph-mon[97033]: pgmap v11891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:46:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:13.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:13.667305+0000) 2022-09-08T03:46:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:13.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:13.641503+0000) 2022-09-08T03:46:14.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:46:13.976+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:46:14.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:14.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:14.667414+0000) 2022-09-08T03:46:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:14.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:14.641639+0000) 2022-09-08T03:46:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:14 smithi055 ceph-mon[97033]: pgmap v11892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:46:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:14 smithi055 ceph-mon[97033]: osdmap e823: 12 total, 11 up, 11 in 2022-09-08T03:46:15.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:14 smithi059 ceph-mon[97450]: pgmap v11892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:46:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:14 smithi059 ceph-mon[97450]: osdmap e823: 12 total, 11 up, 11 in 2022-09-08T03:46:15.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:15.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:15.641774+0000) 2022-09-08T03:46:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:15.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:15.667543+0000) 2022-09-08T03:46:16.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:16.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:16.667677+0000) 2022-09-08T03:46:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:16.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:16.641886+0000) 2022-09-08T03:46:17.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:16 smithi055 ceph-mon[97033]: pgmap v11894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:46:17.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:16 smithi059 ceph-mon[97450]: pgmap v11894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:46:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:17.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:17.667785+0000) 2022-09-08T03:46:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:17.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:17.642022+0000) 2022-09-08T03:46:18.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:18.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:18.667893+0000) 2022-09-08T03:46:18.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:18.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:18.642162+0000) 2022-09-08T03:46:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:18 smithi059 ceph-mon[97450]: pgmap v11895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T03:46:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:18 smithi055 ceph-mon[97033]: pgmap v11895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T03:46:19.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:19.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:19.668015+0000) 2022-09-08T03:46:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:19.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:19.642289+0000) 2022-09-08T03:46:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:20.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:20.668140+0000) 2022-09-08T03:46:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:20.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:20.642420+0000) 2022-09-08T03:46:21.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:20 smithi059 ceph-mon[97450]: pgmap v11896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:46:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:20 smithi055 ceph-mon[97033]: pgmap v11896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:46:21.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:21.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:21.668266+0000) 2022-09-08T03:46:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:21.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:21.642549+0000) 2022-09-08T03:46:22.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:22.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:22.668409+0000) 2022-09-08T03:46:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:22.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:22.642663+0000) 2022-09-08T03:46:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:22 smithi059 ceph-mon[97450]: pgmap v11897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:46:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:22 smithi055 ceph-mon[97033]: pgmap v11897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:46:23.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:23.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:23.642797+0000) 2022-09-08T03:46:23.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:23.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:23.668550+0000) 2022-09-08T03:46:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:24.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:24.668681+0000) 2022-09-08T03:46:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:24.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:24.642911+0000) 2022-09-08T03:46:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:24 smithi059 ceph-mon[97450]: pgmap v11898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:46:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:24 smithi055 ceph-mon[97033]: pgmap v11898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:46:25.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:25.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:25.668790+0000) 2022-09-08T03:46:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:25.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:25.643045+0000) 2022-09-08T03:46:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:26.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:26.668903+0000) 2022-09-08T03:46:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:26.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:26.643175+0000) 2022-09-08T03:46:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:26 smithi059 ceph-mon[97450]: pgmap v11899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:46:27.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:26 smithi055 ceph-mon[97033]: pgmap v11899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:46:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:27.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:27.669033+0000) 2022-09-08T03:46:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:27.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:27.643310+0000) 2022-09-08T03:46:28.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:28.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:28.669155+0000) 2022-09-08T03:46:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:28.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:28.643499+0000) 2022-09-08T03:46:29.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:28 smithi059 ceph-mon[97450]: pgmap v11900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:29.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:46:28.991+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:46:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:28 smithi055 ceph-mon[97033]: pgmap v11900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:29.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:29.669289+0000) 2022-09-08T03:46:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:29.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:29.643622+0000) 2022-09-08T03:46:30.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:30.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:30.669418+0000) 2022-09-08T03:46:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:30.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:30.643760+0000) 2022-09-08T03:46:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:31 smithi059 ceph-mon[97450]: pgmap v11901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:46:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:31 smithi055 ceph-mon[97033]: pgmap v11901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:46:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:31.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:31.669545+0000) 2022-09-08T03:46:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:31.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:31.643874+0000) 2022-09-08T03:46:33.061 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:32.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:32.669761+0000) 2022-09-08T03:46:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:33 smithi059 ceph-mon[97450]: pgmap v11902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:33.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:32.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:32.644041+0000) 2022-09-08T03:46:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:33 smithi055 ceph-mon[97033]: pgmap v11902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:33.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:33.669908+0000) 2022-09-08T03:46:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:33.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:33.644218+0000) 2022-09-08T03:46:34.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:34.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:34.670034+0000) 2022-09-08T03:46:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:34.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:34.644352+0000) 2022-09-08T03:46:35.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:35 smithi059 ceph-mon[97450]: pgmap v11903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:35 smithi055 ceph-mon[97033]: pgmap v11903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:35.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:35.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:35.670169+0000) 2022-09-08T03:46:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:35.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:35.644486+0000) 2022-09-08T03:46:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:36.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:36.670302+0000) 2022-09-08T03:46:36.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:36.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:36.644618+0000) 2022-09-08T03:46:37.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:37 smithi059 ceph-mon[97450]: pgmap v11904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:46:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:37 smithi055 ceph-mon[97033]: pgmap v11904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:46:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:37.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:37.670435+0000) 2022-09-08T03:46:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:37.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:37.644707+0000) 2022-09-08T03:46:38.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:38.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:38.644824+0000) 2022-09-08T03:46:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:38.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:38.670583+0000) 2022-09-08T03:46:39.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:39 smithi059 ceph-mon[97450]: pgmap v11905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:39.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:39 smithi055 ceph-mon[97033]: pgmap v11905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:39.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:39.644956+0000) 2022-09-08T03:46:39.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:39.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:39.670716+0000) 2022-09-08T03:46:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:40.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:40.670853+0000) 2022-09-08T03:46:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:40.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:40.645073+0000) 2022-09-08T03:46:41.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:41 smithi059 ceph-mon[97450]: pgmap v11906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:46:41.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:41 smithi055 ceph-mon[97033]: pgmap v11906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:46:41.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:41.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:41.670987+0000) 2022-09-08T03:46:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:41.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:41.645207+0000) 2022-09-08T03:46:42.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:42.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:42.671123+0000) 2022-09-08T03:46:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:42.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:42.645339+0000) 2022-09-08T03:46:43.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:43 smithi059 ceph-mon[97450]: pgmap v11907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:43 smithi055 ceph-mon[97033]: pgmap v11907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:43.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:43.671253+0000) 2022-09-08T03:46:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:43.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:43.645470+0000) 2022-09-08T03:46:44.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:46:44.006+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:46:44.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:44.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:44.671384+0000) 2022-09-08T03:46:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:44.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:44.645604+0000) 2022-09-08T03:46:45.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:45 smithi059 ceph-mon[97450]: pgmap v11908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:45.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:45 smithi055 ceph-mon[97033]: pgmap v11908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:45.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:45.671510+0000) 2022-09-08T03:46:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:45.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:45.645705+0000) 2022-09-08T03:46:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:46.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:46.645815+0000) 2022-09-08T03:46:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:46.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:46.671649+0000) 2022-09-08T03:46:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:47 smithi055 ceph-mon[97033]: pgmap v11909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:46:47.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:47 smithi059 ceph-mon[97450]: pgmap v11909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:46:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:47.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:47.645982+0000) 2022-09-08T03:46:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:47.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:47.671788+0000) 2022-09-08T03:46:48.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:48.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:48.646121+0000) 2022-09-08T03:46:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:48.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:48.671922+0000) 2022-09-08T03:46:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:49 smithi055 ceph-mon[97033]: pgmap v11910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:49.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:49 smithi059 ceph-mon[97450]: pgmap v11910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:49.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:49.672055+0000) 2022-09-08T03:46:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:49.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:49.646237+0000) 2022-09-08T03:46:50.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:50.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:50.672184+0000) 2022-09-08T03:46:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:50.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:50.646369+0000) 2022-09-08T03:46:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:51 smithi055 ceph-mon[97033]: pgmap v11911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:46:51.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:51 smithi059 ceph-mon[97450]: pgmap v11911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:46:51.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:51.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:51.672316+0000) 2022-09-08T03:46:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:51.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:51.646512+0000) 2022-09-08T03:46:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:52.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:52.672454+0000) 2022-09-08T03:46:52.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:52.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:52.646667+0000) 2022-09-08T03:46:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:53 smithi055 ceph-mon[97033]: pgmap v11912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:53.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:53 smithi059 ceph-mon[97450]: pgmap v11912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:53.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:53.672587+0000) 2022-09-08T03:46:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:53.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:53.646818+0000) 2022-09-08T03:46:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:46:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:46:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:46:54.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:46:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:46:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:46:54.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:54.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:54.672724+0000) 2022-09-08T03:46:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:54.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:54.646965+0000) 2022-09-08T03:46:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:55 smithi055 ceph-mon[97033]: pgmap v11913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:46:55.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:55 smithi059 ceph-mon[97450]: pgmap v11913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:46:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:55.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:55.672857+0000) 2022-09-08T03:46:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:55.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:55.647116+0000) 2022-09-08T03:46:56.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:56.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:56.672987+0000) 2022-09-08T03:46:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:56.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:56.647265+0000) 2022-09-08T03:46:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:57 smithi055 ceph-mon[97033]: pgmap v11914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:46:57.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:57 smithi059 ceph-mon[97450]: pgmap v11914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:46:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:57.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:57.673119+0000) 2022-09-08T03:46:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:57.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:57.647397+0000) 2022-09-08T03:46:58.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:58.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:58.673278+0000) 2022-09-08T03:46:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:58.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:58.647554+0000) 2022-09-08T03:46:59.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:46:59.016+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:46:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:46:59 smithi055 ceph-mon[97033]: pgmap v11915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:59.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:46:59 smithi059 ceph-mon[97450]: pgmap v11915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:46:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:46:59.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:59.673406+0000) 2022-09-08T03:46:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:46:59.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:46:59.647654+0000) 2022-09-08T03:47:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:00.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:00.673539+0000) 2022-09-08T03:47:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:00.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:00.647778+0000) 2022-09-08T03:47:01.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:01 smithi055 ceph-mon[97033]: pgmap v11916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:47:01.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:01 smithi059 ceph-mon[97450]: pgmap v11916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:47:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:01.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:01.673675+0000) 2022-09-08T03:47:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:01.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:01.647898+0000) 2022-09-08T03:47:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:02.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:02.648029+0000) 2022-09-08T03:47:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:02.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:02.673811+0000) 2022-09-08T03:47:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:03 smithi055 ceph-mon[97033]: pgmap v11917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:03.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:03 smithi059 ceph-mon[97450]: pgmap v11917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:03.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:03.648173+0000) 2022-09-08T03:47:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:03.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:03.673922+0000) 2022-09-08T03:47:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:04.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:04.674102+0000) 2022-09-08T03:47:04.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:04.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:04.648342+0000) 2022-09-08T03:47:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:05 smithi055 ceph-mon[97033]: pgmap v11918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:05 smithi059 ceph-mon[97450]: pgmap v11918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:05.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:05.674235+0000) 2022-09-08T03:47:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:05.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:05.648493+0000) 2022-09-08T03:47:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:47:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:47:06.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:47:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:47:06.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:06.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:06.674366+0000) 2022-09-08T03:47:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:06.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:06.648625+0000) 2022-09-08T03:47:07.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:07 smithi055 ceph-mon[97033]: pgmap v11919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:47:07.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:07 smithi059 ceph-mon[97450]: pgmap v11919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:47:07.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:07.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:07.648708+0000) 2022-09-08T03:47:07.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:07.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:07.674494+0000) 2022-09-08T03:47:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:08.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:08.674631+0000) 2022-09-08T03:47:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:08.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:08.648829+0000) 2022-09-08T03:47:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:09 smithi055 ceph-mon[97033]: pgmap v11920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:47:09.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:09 smithi059 ceph-mon[97450]: pgmap v11920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:47:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:09.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:09.674775+0000) 2022-09-08T03:47:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:09.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:09.648978+0000) 2022-09-08T03:47:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:10 smithi055 ceph-mon[97033]: osdmap e824: 12 total, 11 up, 11 in 2022-09-08T03:47:10.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:10 smithi059 ceph-mon[97450]: osdmap e824: 12 total, 11 up, 11 in 2022-09-08T03:47:10.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:10.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:10.674899+0000) 2022-09-08T03:47:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:10.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:10.649108+0000) 2022-09-08T03:47:11.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:11 smithi055 ceph-mon[97033]: pgmap v11922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:47:11.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:11 smithi059 ceph-mon[97450]: pgmap v11922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:47:11.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:11.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:11.675042+0000) 2022-09-08T03:47:11.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:11.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:11.649236+0000) 2022-09-08T03:47:12.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:12.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:12.675177+0000) 2022-09-08T03:47:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:12.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:12.649367+0000) 2022-09-08T03:47:13.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:13 smithi055 ceph-mon[97033]: pgmap v11923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:47:13.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:13 smithi059 ceph-mon[97450]: pgmap v11923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:47:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:13.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:13.675305+0000) 2022-09-08T03:47:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:13.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:13.649500+0000) 2022-09-08T03:47:14.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:47:14.032+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:47:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:14.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:14.675409+0000) 2022-09-08T03:47:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:14.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:14.649614+0000) 2022-09-08T03:47:15.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:14 smithi059 ceph-mon[97450]: pgmap v11924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:47:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:14 smithi059 ceph-mon[97450]: osdmap e825: 12 total, 11 up, 11 in 2022-09-08T03:47:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:14 smithi055 ceph-mon[97033]: pgmap v11924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:47:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:14 smithi055 ceph-mon[97033]: osdmap e825: 12 total, 11 up, 11 in 2022-09-08T03:47:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:15.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:15.675546+0000) 2022-09-08T03:47:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:15.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:15.649708+0000) 2022-09-08T03:47:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:16.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:16.675684+0000) 2022-09-08T03:47:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:16.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:16.649819+0000) 2022-09-08T03:47:17.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:16 smithi059 ceph-mon[97450]: pgmap v11926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:47:17.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:16 smithi055 ceph-mon[97033]: pgmap v11926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:47:17.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:17.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:17.675797+0000) 2022-09-08T03:47:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:17.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:17.649943+0000) 2022-09-08T03:47:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:18.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:18.675932+0000) 2022-09-08T03:47:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:18.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:18.650073+0000) 2022-09-08T03:47:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:19 smithi059 ceph-mon[97450]: pgmap v11927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 240 B/s wr, 7 op/s 2022-09-08T03:47:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:19 smithi055 ceph-mon[97033]: pgmap v11927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 240 B/s wr, 7 op/s 2022-09-08T03:47:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:19.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:19.676065+0000) 2022-09-08T03:47:19.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:19.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:19.650204+0000) 2022-09-08T03:47:20.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:20.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:20.676193+0000) 2022-09-08T03:47:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:20.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:20.650338+0000) 2022-09-08T03:47:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:21 smithi059 ceph-mon[97450]: pgmap v11928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:47:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:21 smithi055 ceph-mon[97033]: pgmap v11928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:47:21.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:21.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:21.676332+0000) 2022-09-08T03:47:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:21.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:21.650467+0000) 2022-09-08T03:47:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:22.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:22.676479+0000) 2022-09-08T03:47:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:22.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:22.650597+0000) 2022-09-08T03:47:23.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:23 smithi059 ceph-mon[97450]: pgmap v11929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:47:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:23 smithi055 ceph-mon[97033]: pgmap v11929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:47:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:23.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:23.676619+0000) 2022-09-08T03:47:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:23.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:23.650706+0000) 2022-09-08T03:47:24.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:24.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:24.676765+0000) 2022-09-08T03:47:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:24.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:24.650817+0000) 2022-09-08T03:47:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:25 smithi059 ceph-mon[97450]: pgmap v11930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:47:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:25 smithi055 ceph-mon[97033]: pgmap v11930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:47:25.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:25.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:25.676873+0000) 2022-09-08T03:47:25.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:25.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:25.650987+0000) 2022-09-08T03:47:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:26.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:26.677006+0000) 2022-09-08T03:47:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:26.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:26.651138+0000) 2022-09-08T03:47:27.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:27 smithi059 ceph-mon[97450]: pgmap v11931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:47:27.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:27 smithi055 ceph-mon[97033]: pgmap v11931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:47:27.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:27.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:27.651288+0000) 2022-09-08T03:47:27.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:27.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:27.677132+0000) 2022-09-08T03:47:29.041 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:28.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:28.677263+0000) 2022-09-08T03:47:29.041 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:28.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:28.651441+0000) 2022-09-08T03:47:29.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:29 smithi059 ceph-mon[97450]: pgmap v11932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:29.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:47:29.040+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:47:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:29 smithi055 ceph-mon[97033]: pgmap v11932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:29.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:29.677404+0000) 2022-09-08T03:47:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:29.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:29.651590+0000) 2022-09-08T03:47:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:30.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:30.677545+0000) 2022-09-08T03:47:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:30.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:30.651739+0000) 2022-09-08T03:47:31.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:31 smithi059 ceph-mon[97450]: pgmap v11933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:47:31.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:31 smithi055 ceph-mon[97033]: pgmap v11933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:47:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:31.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:31.677682+0000) 2022-09-08T03:47:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:31.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:31.651868+0000) 2022-09-08T03:47:33.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:32.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:32.677801+0000) 2022-09-08T03:47:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:32.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:32.652010+0000) 2022-09-08T03:47:33.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:33 smithi059 ceph-mon[97450]: pgmap v11934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:33 smithi055 ceph-mon[97033]: pgmap v11934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:33.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:33.677911+0000) 2022-09-08T03:47:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:33.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:33.652160+0000) 2022-09-08T03:47:35.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:34.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:34.678041+0000) 2022-09-08T03:47:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:34.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:34.652298+0000) 2022-09-08T03:47:35.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:35 smithi059 ceph-mon[97450]: pgmap v11935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:35.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:35 smithi055 ceph-mon[97033]: pgmap v11935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:35.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:35.678171+0000) 2022-09-08T03:47:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:35.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:35.652430+0000) 2022-09-08T03:47:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:36.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:36.678307+0000) 2022-09-08T03:47:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:36.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:36.652575+0000) 2022-09-08T03:47:37.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:37 smithi059 ceph-mon[97450]: pgmap v11936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:47:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:37 smithi055 ceph-mon[97033]: pgmap v11936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:47:38.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:37.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:37.652712+0000) 2022-09-08T03:47:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:37.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:37.678419+0000) 2022-09-08T03:47:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:38.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:38.678550+0000) 2022-09-08T03:47:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:38.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:38.652839+0000) 2022-09-08T03:47:39.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:39 smithi059 ceph-mon[97450]: pgmap v11937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:39.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:39 smithi055 ceph-mon[97033]: pgmap v11937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:39.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:39.678675+0000) 2022-09-08T03:47:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:39.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:39.652972+0000) 2022-09-08T03:47:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:40.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:40.678792+0000) 2022-09-08T03:47:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:40.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:40.653093+0000) 2022-09-08T03:47:41.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:41 smithi055 ceph-mon[97033]: pgmap v11938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:47:41.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:41 smithi059 ceph-mon[97450]: pgmap v11938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:47:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:41.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:41.678911+0000) 2022-09-08T03:47:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:41.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:41.653228+0000) 2022-09-08T03:47:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:42.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:42.679057+0000) 2022-09-08T03:47:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:42.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:42.653365+0000) 2022-09-08T03:47:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:43 smithi055 ceph-mon[97033]: pgmap v11939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:43.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:43 smithi059 ceph-mon[97450]: pgmap v11939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:44.047 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:43.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:43.679200+0000) 2022-09-08T03:47:44.048 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:43.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:43.653502+0000) 2022-09-08T03:47:44.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:47:44.046+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:47:45.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:44.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:44.679341+0000) 2022-09-08T03:47:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:44.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:44.653644+0000) 2022-09-08T03:47:45.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:45 smithi055 ceph-mon[97033]: pgmap v11940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:45.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:45 smithi059 ceph-mon[97450]: pgmap v11940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:46.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:45.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:45.679477+0000) 2022-09-08T03:47:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:45.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:45.653794+0000) 2022-09-08T03:47:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:46.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:46.653908+0000) 2022-09-08T03:47:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:46.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:46.679644+0000) 2022-09-08T03:47:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:47 smithi055 ceph-mon[97033]: pgmap v11941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:47:47.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:47 smithi059 ceph-mon[97450]: pgmap v11941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:47:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:47.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:47.679756+0000) 2022-09-08T03:47:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:47.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:47.654042+0000) 2022-09-08T03:47:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:48.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:48.679892+0000) 2022-09-08T03:47:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:48.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:48.654179+0000) 2022-09-08T03:47:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:49 smithi055 ceph-mon[97033]: pgmap v11942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:49.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:49 smithi059 ceph-mon[97450]: pgmap v11942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:49.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:49.680024+0000) 2022-09-08T03:47:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:49.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:49.654315+0000) 2022-09-08T03:47:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:50.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:50.654445+0000) 2022-09-08T03:47:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:50.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:50.680167+0000) 2022-09-08T03:47:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:51 smithi055 ceph-mon[97033]: pgmap v11943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:47:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:51 smithi059 ceph-mon[97450]: pgmap v11943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:47:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:51.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:51.680315+0000) 2022-09-08T03:47:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:51.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:51.654574+0000) 2022-09-08T03:47:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:52.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:52.680459+0000) 2022-09-08T03:47:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:52.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:52.654708+0000) 2022-09-08T03:47:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:53 smithi055 ceph-mon[97033]: pgmap v11944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:53.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:53 smithi059 ceph-mon[97450]: pgmap v11944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:54.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:53.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:53.680588+0000) 2022-09-08T03:47:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:53.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:53.654842+0000) 2022-09-08T03:47:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:54.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:54.680698+0000) 2022-09-08T03:47:54.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:54.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:54.654975+0000) 2022-09-08T03:47:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:55 smithi055 ceph-mon[97033]: pgmap v11945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:47:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:47:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:47:55.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:55 smithi059 ceph-mon[97450]: pgmap v11945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:47:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:47:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:47:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:55.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:55.680815+0000) 2022-09-08T03:47:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:55.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:55.655110+0000) 2022-09-08T03:47:57.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:56.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:56.680924+0000) 2022-09-08T03:47:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:56.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:56.655238+0000) 2022-09-08T03:47:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:57 smithi055 ceph-mon[97033]: pgmap v11946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:47:57.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:57 smithi059 ceph-mon[97450]: pgmap v11946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:47:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:57.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:57.681054+0000) 2022-09-08T03:47:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:57.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:57.655393+0000) 2022-09-08T03:47:59.063 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:58.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:58.681192+0000) 2022-09-08T03:47:59.064 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:58.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:58.655543+0000) 2022-09-08T03:47:59.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:47:59.062+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:47:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:47:59 smithi055 ceph-mon[97033]: pgmap v11947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:47:59.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:47:59 smithi059 ceph-mon[97450]: pgmap v11947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:00.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:47:59.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:59.681329+0000) 2022-09-08T03:48:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:47:59.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:47:59.655684+0000) 2022-09-08T03:48:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:00.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:00.655804+0000) 2022-09-08T03:48:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:00.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:00.681456+0000) 2022-09-08T03:48:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:01 smithi055 ceph-mon[97033]: pgmap v11948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:48:01.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:01 smithi059 ceph-mon[97450]: pgmap v11948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:48:02.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:01.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:01.655948+0000) 2022-09-08T03:48:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:01.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:01.681594+0000) 2022-09-08T03:48:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:02.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:02.681735+0000) 2022-09-08T03:48:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:02.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:02.656088+0000) 2022-09-08T03:48:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:03 smithi055 ceph-mon[97033]: pgmap v11949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:03.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:03 smithi059 ceph-mon[97450]: pgmap v11949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:04.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:03.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:03.656229+0000) 2022-09-08T03:48:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:03.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:03.681864+0000) 2022-09-08T03:48:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:04.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:04.681989+0000) 2022-09-08T03:48:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:04.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:04.656360+0000) 2022-09-08T03:48:05.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:05 smithi055 ceph-mon[97033]: pgmap v11950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:05.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:05 smithi059 ceph-mon[97450]: pgmap v11950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:05.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:05.682119+0000) 2022-09-08T03:48:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:05.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:05.656496+0000) 2022-09-08T03:48:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:48:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:48:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:48:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:48:07.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:06.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:06.682242+0000) 2022-09-08T03:48:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:06.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:06.656630+0000) 2022-09-08T03:48:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:07 smithi055 ceph-mon[97033]: pgmap v11951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:48:07.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:07 smithi059 ceph-mon[97450]: pgmap v11951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:48:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:07.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:07.656761+0000) 2022-09-08T03:48:08.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:07.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:07.682431+0000) 2022-09-08T03:48:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:08.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:08.682552+0000) 2022-09-08T03:48:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:08.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:08.656924+0000) 2022-09-08T03:48:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:09 smithi055 ceph-mon[97033]: pgmap v11952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:48:09.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:09 smithi059 ceph-mon[97450]: pgmap v11952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:48:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:09.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:09.682704+0000) 2022-09-08T03:48:10.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:09.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:09.657060+0000) 2022-09-08T03:48:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:10 smithi055 ceph-mon[97033]: osdmap e826: 12 total, 11 up, 11 in 2022-09-08T03:48:10.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:10 smithi059 ceph-mon[97450]: osdmap e826: 12 total, 11 up, 11 in 2022-09-08T03:48:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:10.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:10.682829+0000) 2022-09-08T03:48:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:10.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:10.657194+0000) 2022-09-08T03:48:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:11 smithi055 ceph-mon[97033]: pgmap v11954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:48:11.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:11 smithi059 ceph-mon[97450]: pgmap v11954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:48:12.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:11.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:11.682981+0000) 2022-09-08T03:48:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:11.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:11.657330+0000) 2022-09-08T03:48:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:12.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:12.683133+0000) 2022-09-08T03:48:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:12.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:12.657462+0000) 2022-09-08T03:48:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:13 smithi055 ceph-mon[97033]: pgmap v11955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:48:13.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:13 smithi059 ceph-mon[97450]: pgmap v11955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:48:14.079 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:13.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:13.657598+0000) 2022-09-08T03:48:14.081 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:13.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:13.683280+0000) 2022-09-08T03:48:14.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:48:14.078+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:48:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:14.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:14.683419+0000) 2022-09-08T03:48:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:14.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:14.657732+0000) 2022-09-08T03:48:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:15 smithi059 ceph-mon[97450]: pgmap v11956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:48:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:15 smithi059 ceph-mon[97450]: osdmap e827: 12 total, 11 up, 11 in 2022-09-08T03:48:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:15 smithi055 ceph-mon[97033]: pgmap v11956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:48:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:15 smithi055 ceph-mon[97033]: osdmap e827: 12 total, 11 up, 11 in 2022-09-08T03:48:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:15.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:15.683548+0000) 2022-09-08T03:48:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:15.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:15.657870+0000) 2022-09-08T03:48:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:16.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:16.683681+0000) 2022-09-08T03:48:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:16.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:16.658004+0000) 2022-09-08T03:48:17.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:17 smithi059 ceph-mon[97450]: pgmap v11958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:48:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:17 smithi055 ceph-mon[97033]: pgmap v11958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:48:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:17.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:17.683790+0000) 2022-09-08T03:48:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:17.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:17.658145+0000) 2022-09-08T03:48:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:18.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:18.683933+0000) 2022-09-08T03:48:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:18.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:18.658292+0000) 2022-09-08T03:48:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:19 smithi059 ceph-mon[97450]: pgmap v11959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T03:48:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:19 smithi055 ceph-mon[97033]: pgmap v11959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T03:48:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:19.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:19.684081+0000) 2022-09-08T03:48:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:19.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:19.658443+0000) 2022-09-08T03:48:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:20.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:20.684226+0000) 2022-09-08T03:48:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:20.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:20.658583+0000) 2022-09-08T03:48:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:21 smithi059 ceph-mon[97450]: pgmap v11960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:48:21.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:21 smithi055 ceph-mon[97033]: pgmap v11960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:48:22.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:21.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:21.684362+0000) 2022-09-08T03:48:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:21.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:21.658732+0000) 2022-09-08T03:48:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:22.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:22.684497+0000) 2022-09-08T03:48:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:22.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:22.658855+0000) 2022-09-08T03:48:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:23 smithi059 ceph-mon[97450]: pgmap v11961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:48:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:23 smithi055 ceph-mon[97033]: pgmap v11961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:48:24.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:23.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:23.684628+0000) 2022-09-08T03:48:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:23.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:23.659005+0000) 2022-09-08T03:48:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:24.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:24.684753+0000) 2022-09-08T03:48:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:24.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:24.659151+0000) 2022-09-08T03:48:25.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:25 smithi059 ceph-mon[97450]: pgmap v11962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:48:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:48:25.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:25 smithi055 ceph-mon[97033]: pgmap v11962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:48:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:48:26.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:25.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:25.659303+0000) 2022-09-08T03:48:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:25.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:25.684887+0000) 2022-09-08T03:48:26.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:26 smithi059 ceph-mon[97450]: pgmap v11963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T03:48:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:26 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T03:48:26.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:26 smithi055 ceph-mon[97033]: pgmap v11963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T03:48:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:26 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T03:48:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:26.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:26.659450+0000) 2022-09-08T03:48:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:26.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:26.685018+0000) 2022-09-08T03:48:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:27.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:27.685150+0000) 2022-09-08T03:48:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:27.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:27.659583+0000) 2022-09-08T03:48:28.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:28 smithi059 ceph-mon[97450]: pgmap v11964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:48:28.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:28 smithi055 ceph-mon[97033]: pgmap v11964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:48:28.998 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:28.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:28.685284+0000) 2022-09-08T03:48:28.999 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:28.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:28.659739+0000) 2022-09-08T03:48:29.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:48:29.079+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:48:30.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:29.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:29.685449+0000) 2022-09-08T03:48:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:29.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:29.659888+0000) 2022-09-08T03:48:30.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:30 smithi055 ceph-mon[97033]: pgmap v11965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:48:30.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:30 smithi059 ceph-mon[97450]: pgmap v11965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:48:31.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:30.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:30.685608+0000) 2022-09-08T03:48:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:30.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:30.660035+0000) 2022-09-08T03:48:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:31.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:31.685689+0000) 2022-09-08T03:48:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:31.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:31.660181+0000) 2022-09-08T03:48:32.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:32 smithi055 ceph-mon[97033]: pgmap v11966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:48:32.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:32 smithi059 ceph-mon[97450]: pgmap v11966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:48:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:32.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:32.685844+0000) 2022-09-08T03:48:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:32.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:32.660327+0000) 2022-09-08T03:48:34.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:33.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:33.685972+0000) 2022-09-08T03:48:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:33.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:33.660474+0000) 2022-09-08T03:48:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:34 smithi055 ceph-mon[97033]: pgmap v11967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:48:34.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:34 smithi059 ceph-mon[97450]: pgmap v11967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:48:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:34.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:34.686115+0000) 2022-09-08T03:48:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:34.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:34.660649+0000) 2022-09-08T03:48:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:35.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:35.686249+0000) 2022-09-08T03:48:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:35.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:35.660784+0000) 2022-09-08T03:48:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:36 smithi055 ceph-mon[97033]: pgmap v11968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:48:36.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:36 smithi059 ceph-mon[97450]: pgmap v11968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:48:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:36.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:36.686389+0000) 2022-09-08T03:48:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:36.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:36.660922+0000) 2022-09-08T03:48:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:37.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:37.686525+0000) 2022-09-08T03:48:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:37.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:37.661048+0000) 2022-09-08T03:48:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:38 smithi055 ceph-mon[97033]: pgmap v11969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:48:38.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:38 smithi059 ceph-mon[97450]: pgmap v11969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:48:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:38.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:38.686684+0000) 2022-09-08T03:48:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:38.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:38.661186+0000) 2022-09-08T03:48:40.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:39.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:39.686817+0000) 2022-09-08T03:48:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:39.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:39.661318+0000) 2022-09-08T03:48:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:40 smithi055 ceph-mon[97033]: pgmap v11970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:40.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:40 smithi059 ceph-mon[97450]: pgmap v11970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:40.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:40.687040+0000) 2022-09-08T03:48:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:40.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:40.661467+0000) 2022-09-08T03:48:42.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:41.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:41.687185+0000) 2022-09-08T03:48:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:41.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:41.661598+0000) 2022-09-08T03:48:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:42 smithi055 ceph-mon[97033]: pgmap v11971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:48:42.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:42 smithi059 ceph-mon[97450]: pgmap v11971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:48:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:42.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:42.687324+0000) 2022-09-08T03:48:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:42.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:42.661731+0000) 2022-09-08T03:48:44.091 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:43.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:43.687452+0000) 2022-09-08T03:48:44.092 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:43.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:43.661864+0000) 2022-09-08T03:48:44.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:48:44.091+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:48:44.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:44 smithi055 ceph-mon[97033]: pgmap v11972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:44.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:44 smithi059 ceph-mon[97450]: pgmap v11972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:44.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:44.687591+0000) 2022-09-08T03:48:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:44.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:44.661997+0000) 2022-09-08T03:48:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:45.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:45.687726+0000) 2022-09-08T03:48:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:45.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:45.662133+0000) 2022-09-08T03:48:46.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:46 smithi055 ceph-mon[97033]: pgmap v11973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:46.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:46 smithi059 ceph-mon[97450]: pgmap v11973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:47.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:46.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:46.687863+0000) 2022-09-08T03:48:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:46.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:46.662262+0000) 2022-09-08T03:48:48.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:47.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:47.687994+0000) 2022-09-08T03:48:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:47.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:47.662397+0000) 2022-09-08T03:48:48.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:48 smithi055 ceph-mon[97033]: pgmap v11974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:48:48.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:48 smithi059 ceph-mon[97450]: pgmap v11974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:48:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:48.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:48.688107+0000) 2022-09-08T03:48:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:48.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:48.662534+0000) 2022-09-08T03:48:50.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:49.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:49.662673+0000) 2022-09-08T03:48:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:49.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:49.688234+0000) 2022-09-08T03:48:50.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:50 smithi055 ceph-mon[97033]: pgmap v11975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:50.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:50 smithi059 ceph-mon[97450]: pgmap v11975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:50.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:50.688377+0000) 2022-09-08T03:48:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:50.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:50.662833+0000) 2022-09-08T03:48:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:51.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:51.688522+0000) 2022-09-08T03:48:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:51.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:51.662956+0000) 2022-09-08T03:48:52.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:52 smithi055 ceph-mon[97033]: pgmap v11976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:48:52.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:52 smithi059 ceph-mon[97450]: pgmap v11976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:48:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:52.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:52.688676+0000) 2022-09-08T03:48:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:52.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:52.663102+0000) 2022-09-08T03:48:54.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:53.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:53.688786+0000) 2022-09-08T03:48:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:53.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:53.663245+0000) 2022-09-08T03:48:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:54 smithi055 ceph-mon[97033]: pgmap v11977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:54.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:54 smithi059 ceph-mon[97450]: pgmap v11977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:54.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:54.688928+0000) 2022-09-08T03:48:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:54.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:54.663391+0000) 2022-09-08T03:48:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:55.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:55.689076+0000) 2022-09-08T03:48:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:55.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:55.663542+0000) 2022-09-08T03:48:56.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:56 smithi055 ceph-mon[97033]: pgmap v11978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:56.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:56 smithi059 ceph-mon[97450]: pgmap v11978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:48:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:56.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:56.663698+0000) 2022-09-08T03:48:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:56.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:56.689216+0000) 2022-09-08T03:48:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:57.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:57.689358+0000) 2022-09-08T03:48:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:57.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:57.663846+0000) 2022-09-08T03:48:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:48:58 smithi055 ceph-mon[97033]: pgmap v11979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:48:58.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:48:58 smithi059 ceph-mon[97450]: pgmap v11979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:48:59.103 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:58.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:58.689497+0000) 2022-09-08T03:48:59.104 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:58.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:58.664006+0000) 2022-09-08T03:48:59.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:48:59.103+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:49:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:48:59.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:59.664132+0000) 2022-09-08T03:49:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:48:59.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:48:59.689617+0000) 2022-09-08T03:49:00.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:00 smithi055 ceph-mon[97033]: pgmap v11980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:00.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:00 smithi059 ceph-mon[97450]: pgmap v11980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:00.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:00.664277+0000) 2022-09-08T03:49:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:00.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:00.689765+0000) 2022-09-08T03:49:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:01.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:01.689898+0000) 2022-09-08T03:49:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:01.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:01.664424+0000) 2022-09-08T03:49:02.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:02 smithi055 ceph-mon[97033]: pgmap v11981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:49:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:02 smithi059 ceph-mon[97450]: pgmap v11981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:49:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:02.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:02.690037+0000) 2022-09-08T03:49:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:02.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:02.664571+0000) 2022-09-08T03:49:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:03.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:03.690171+0000) 2022-09-08T03:49:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:03.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:03.664705+0000) 2022-09-08T03:49:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:04 smithi055 ceph-mon[97033]: pgmap v11982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:04.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:04 smithi059 ceph-mon[97450]: pgmap v11982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:04.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:04.690309+0000) 2022-09-08T03:49:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:04.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:04.664857+0000) 2022-09-08T03:49:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:05.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:05.690437+0000) 2022-09-08T03:49:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:05.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:05.664987+0000) 2022-09-08T03:49:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:06 smithi055 ceph-mon[97033]: pgmap v11983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:49:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:49:06.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:06 smithi059 ceph-mon[97450]: pgmap v11983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:49:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:49:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:06.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:06.690570+0000) 2022-09-08T03:49:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:06.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:06.665122+0000) 2022-09-08T03:49:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:07.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:07.690701+0000) 2022-09-08T03:49:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:07.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:07.665254+0000) 2022-09-08T03:49:08.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:08 smithi055 ceph-mon[97033]: pgmap v11984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:49:08.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:08 smithi059 ceph-mon[97450]: pgmap v11984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:49:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:08.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:08.690821+0000) 2022-09-08T03:49:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:08.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:08.665386+0000) 2022-09-08T03:49:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:09.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:09.690936+0000) 2022-09-08T03:49:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:09.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:09.665523+0000) 2022-09-08T03:49:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:10 smithi055 ceph-mon[97033]: pgmap v11985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:49:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:10 smithi055 ceph-mon[97033]: osdmap e828: 12 total, 11 up, 11 in 2022-09-08T03:49:10.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:10 smithi059 ceph-mon[97450]: pgmap v11985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:49:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:10 smithi059 ceph-mon[97450]: osdmap e828: 12 total, 11 up, 11 in 2022-09-08T03:49:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:10.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:10.691076+0000) 2022-09-08T03:49:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:10.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:10.665666+0000) 2022-09-08T03:49:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:11.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:11.691211+0000) 2022-09-08T03:49:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:11.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:11.665830+0000) 2022-09-08T03:49:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:12 smithi055 ceph-mon[97033]: pgmap v11987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:49:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:12 smithi059 ceph-mon[97450]: pgmap v11987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:49:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:12.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:12.691445+0000) 2022-09-08T03:49:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:12.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:12.666080+0000) 2022-09-08T03:49:14.117 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:13.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:13.691592+0000) 2022-09-08T03:49:14.117 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:13.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:13.666235+0000) 2022-09-08T03:49:14.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:49:14.115+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:49:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:14 smithi055 ceph-mon[97033]: pgmap v11988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:49:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:14 smithi055 ceph-mon[97033]: osdmap e829: 12 total, 11 up, 11 in 2022-09-08T03:49:14.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:14 smithi059 ceph-mon[97450]: pgmap v11988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:49:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:14 smithi059 ceph-mon[97450]: osdmap e829: 12 total, 11 up, 11 in 2022-09-08T03:49:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:14.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:14.691691+0000) 2022-09-08T03:49:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:14.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:14.666383+0000) 2022-09-08T03:49:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:15.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:15.691817+0000) 2022-09-08T03:49:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:15.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:15.666532+0000) 2022-09-08T03:49:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:16 smithi055 ceph-mon[97033]: pgmap v11990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:49:16.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:16 smithi059 ceph-mon[97450]: pgmap v11990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:49:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:16.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:16.691927+0000) 2022-09-08T03:49:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:16.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:16.666687+0000) 2022-09-08T03:49:18.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:17.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:17.692047+0000) 2022-09-08T03:49:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:17.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:17.666817+0000) 2022-09-08T03:49:18.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:18 smithi055 ceph-mon[97033]: pgmap v11991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:49:18.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:18 smithi059 ceph-mon[97450]: pgmap v11991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:49:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:18.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:18.692180+0000) 2022-09-08T03:49:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:18.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:18.666971+0000) 2022-09-08T03:49:20.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:19.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:19.667118+0000) 2022-09-08T03:49:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:19.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:19.692318+0000) 2022-09-08T03:49:20.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:20 smithi055 ceph-mon[97033]: pgmap v11992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T03:49:20.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:20 smithi059 ceph-mon[97450]: pgmap v11992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T03:49:21.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:20.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:20.692464+0000) 2022-09-08T03:49:21.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:20.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:20.667264+0000) 2022-09-08T03:49:22.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:21.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:21.692592+0000) 2022-09-08T03:49:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:21.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:21.667410+0000) 2022-09-08T03:49:22.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:22 smithi055 ceph-mon[97033]: pgmap v11993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:49:22.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:22 smithi059 ceph-mon[97450]: pgmap v11993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:49:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:22.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:22.692707+0000) 2022-09-08T03:49:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:22.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:22.667556+0000) 2022-09-08T03:49:24.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:23.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:23.692818+0000) 2022-09-08T03:49:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:23.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:23.667703+0000) 2022-09-08T03:49:24.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:24 smithi055 ceph-mon[97033]: pgmap v11994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:49:24.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:24 smithi059 ceph-mon[97450]: pgmap v11994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:49:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:24.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:24.667849+0000) 2022-09-08T03:49:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:24.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:24.692927+0000) 2022-09-08T03:49:25.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:49:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:49:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:49:25.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:49:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:49:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:49:26.124 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:26 smithi055 ceph-mon[97033]: pgmap v11995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T03:49:26.125 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:26.125 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:26.126 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:26.126 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:26.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:25.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:25.667977+0000) 2022-09-08T03:49:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:25.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:25.693041+0000) 2022-09-08T03:49:26.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:26 smithi059 ceph-mon[97450]: pgmap v11995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T03:49:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:26.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:26.668118+0000) 2022-09-08T03:49:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:26.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:26.693174+0000) 2022-09-08T03:49:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:27.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:27.693297+0000) 2022-09-08T03:49:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:27.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:27.668264+0000) 2022-09-08T03:49:28.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:28 smithi055 ceph-mon[97033]: pgmap v11996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:49:28.494 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:28 smithi059 ceph-mon[97450]: pgmap v11996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T03:49:29.118 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:28.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:28.693431+0000) 2022-09-08T03:49:29.119 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:28.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:28.668419+0000) 2022-09-08T03:49:29.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:49:29.117+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:49:29.926 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:29.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:29.693560+0000) 2022-09-08T03:49:29.927 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:29.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:29.668569+0000) 2022-09-08T03:49:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:29 smithi055 ceph-mon[97033]: pgmap v11997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:29 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T03:49:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:30.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:29 smithi059 ceph-mon[97450]: pgmap v11997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:29 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T03:49:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:30.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:30.693686+0000) 2022-09-08T03:49:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:30.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:30.668723+0000) 2022-09-08T03:49:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:31 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T03:49:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:32.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:31 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T03:49:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:31.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:31.693795+0000) 2022-09-08T03:49:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:31.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:31.668877+0000) 2022-09-08T03:49:32.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:32 smithi055 ceph-mon[97033]: pgmap v11998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:49:33.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:32 smithi059 ceph-mon[97450]: pgmap v11998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:49:33.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:32.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:32.693941+0000) 2022-09-08T03:49:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:32.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:32.669022+0000) 2022-09-08T03:49:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:33.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:33.694075+0000) 2022-09-08T03:49:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:33.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:33.669170+0000) 2022-09-08T03:49:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:34 smithi055 ceph-mon[97033]: pgmap v11999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:34 smithi059 ceph-mon[97450]: pgmap v11999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:35.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:34.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:34.694202+0000) 2022-09-08T03:49:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:34.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:34.669314+0000) 2022-09-08T03:49:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:35.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:35.694310+0000) 2022-09-08T03:49:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:35.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:35.669464+0000) 2022-09-08T03:49:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:36 smithi055 ceph-mon[97033]: pgmap v12000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:37.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:36 smithi059 ceph-mon[97450]: pgmap v12000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:36.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:36.694436+0000) 2022-09-08T03:49:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:36.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:36.669600+0000) 2022-09-08T03:49:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:37.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:37.694564+0000) 2022-09-08T03:49:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:37.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:37.669733+0000) 2022-09-08T03:49:38.991 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:38 smithi055 ceph-mon[97033]: pgmap v12001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:49:39.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:38 smithi059 ceph-mon[97450]: pgmap v12001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:49:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:38.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:38.669874+0000) 2022-09-08T03:49:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:38.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:38.694708+0000) 2022-09-08T03:49:40.163 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:39.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:39.694836+0000) 2022-09-08T03:49:40.164 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:39.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:39.670007+0000) 2022-09-08T03:49:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:40 smithi055 ceph-mon[97033]: pgmap v12002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:41.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:40 smithi059 ceph-mon[97450]: pgmap v12002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:40.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:40.694947+0000) 2022-09-08T03:49:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:40.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:40.670131+0000) 2022-09-08T03:49:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:41.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:41.695068+0000) 2022-09-08T03:49:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:41.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:41.670243+0000) 2022-09-08T03:49:42.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:42 smithi055 ceph-mon[97033]: pgmap v12003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:49:43.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:42 smithi059 ceph-mon[97450]: pgmap v12003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:49:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:42.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:42.695205+0000) 2022-09-08T03:49:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:42.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:42.670373+0000) 2022-09-08T03:49:44.119 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:43.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:43.695333+0000) 2022-09-08T03:49:44.120 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:43.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:43.670487+0000) 2022-09-08T03:49:44.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:49:44.119+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:49:44.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:44 smithi055 ceph-mon[97033]: pgmap v12004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:45.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:44 smithi059 ceph-mon[97450]: pgmap v12004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:45.049 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:44.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:44.670720+0000) 2022-09-08T03:49:45.400 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:44.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:44.695507+0000) 2022-09-08T03:49:46.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:45.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:45.695617+0000) 2022-09-08T03:49:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:45.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:45.670850+0000) 2022-09-08T03:49:46.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:46 smithi055 ceph-mon[97033]: pgmap v12005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:47.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:46 smithi059 ceph-mon[97450]: pgmap v12005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:46.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:46.695695+0000) 2022-09-08T03:49:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:46.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:46.670959+0000) 2022-09-08T03:49:48.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:47.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:47.695805+0000) 2022-09-08T03:49:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:47.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:47.671078+0000) 2022-09-08T03:49:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:48 smithi055 ceph-mon[97033]: pgmap v12006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:49:49.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:48 smithi059 ceph-mon[97450]: pgmap v12006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:49:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:48.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:48.695920+0000) 2022-09-08T03:49:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:48.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:48.671203+0000) 2022-09-08T03:49:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:49.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:49.696030+0000) 2022-09-08T03:49:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:49.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:49.671343+0000) 2022-09-08T03:49:50.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:50 smithi055 ceph-mon[97033]: pgmap v12007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:51.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:50 smithi059 ceph-mon[97450]: pgmap v12007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:50.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:50.696158+0000) 2022-09-08T03:49:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:50.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:50.671478+0000) 2022-09-08T03:49:52.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:51.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:51.696289+0000) 2022-09-08T03:49:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:51.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:51.671616+0000) 2022-09-08T03:49:52.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:52 smithi055 ceph-mon[97033]: pgmap v12008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:49:53.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:52 smithi059 ceph-mon[97450]: pgmap v12008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:49:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:52.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:52.696421+0000) 2022-09-08T03:49:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:52.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:52.671769+0000) 2022-09-08T03:49:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:53.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:53.696551+0000) 2022-09-08T03:49:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:53.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:53.671907+0000) 2022-09-08T03:49:54.908 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:54 smithi055 ceph-mon[97033]: pgmap v12009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:54.961 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:54 smithi059 ceph-mon[97450]: pgmap v12009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:55.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:54.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:54.696673+0000) 2022-09-08T03:49:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:54.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:54.672044+0000) 2022-09-08T03:49:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:55.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:55.696811+0000) 2022-09-08T03:49:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:55.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:55.672161+0000) 2022-09-08T03:49:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:55 smithi055 ceph-mon[97033]: pgmap v12010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:55 smithi055 ceph-mon[97033]: pgmap v12011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 7 op/s 2022-09-08T03:49:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:55 smithi055 ceph-mon[97033]: pgmap v12012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-08T03:49:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:56.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:55 smithi059 ceph-mon[97450]: pgmap v12010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:49:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:55 smithi059 ceph-mon[97450]: pgmap v12011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 7 op/s 2022-09-08T03:49:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:55 smithi059 ceph-mon[97450]: pgmap v12012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-08T03:49:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:49:57.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:56.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:56.696943+0000) 2022-09-08T03:49:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:56.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:56.672298+0000) 2022-09-08T03:49:58.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:57.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:57.697085+0000) 2022-09-08T03:49:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:57.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:57.672440+0000) 2022-09-08T03:49:58.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:49:57 smithi055 ceph-mon[97033]: pgmap v12013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:49:58.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:49:57 smithi059 ceph-mon[97450]: pgmap v12013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:49:59.134 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:58.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:58.697220+0000) 2022-09-08T03:49:59.135 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:58.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:58.672569+0000) 2022-09-08T03:49:59.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:49:59.134+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:50:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:49:59.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:59.697341+0000) 2022-09-08T03:50:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:49:59.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:49:59.672704+0000) 2022-09-08T03:50:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:00 smithi055 ceph-mon[97033]: pgmap v12014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:50:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T03:50:00.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T03:50:00.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T03:50:00.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T03:50:00.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T03:50:00.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:00 smithi059 ceph-mon[97450]: pgmap v12014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:50:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T03:50:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T03:50:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T03:50:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T03:50:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T03:50:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:00.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:00.697473+0000) 2022-09-08T03:50:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:00.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:00.672844+0000) 2022-09-08T03:50:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:01.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:01.697625+0000) 2022-09-08T03:50:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:01.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:01.672980+0000) 2022-09-08T03:50:02.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:02 smithi055 ceph-mon[97033]: pgmap v12015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:50:02.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:02 smithi059 ceph-mon[97450]: pgmap v12015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:50:03.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:02.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:02.697777+0000) 2022-09-08T03:50:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:02.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:02.673092+0000) 2022-09-08T03:50:04.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:03.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:03.697905+0000) 2022-09-08T03:50:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:03.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:03.673225+0000) 2022-09-08T03:50:04.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:04 smithi055 ceph-mon[97033]: pgmap v12016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:50:04.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:04 smithi059 ceph-mon[97450]: pgmap v12016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:50:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:04.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:04.698051+0000) 2022-09-08T03:50:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:04.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:04.673366+0000) 2022-09-08T03:50:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:05.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:05.698180+0000) 2022-09-08T03:50:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:05.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:05.673501+0000) 2022-09-08T03:50:06.317 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:06 smithi055 ceph-mon[97033]: pgmap v12017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 27 KiB/s wr, 8 op/s 2022-09-08T03:50:06.318 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:50:06.318 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:50:06.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:06 smithi059 ceph-mon[97450]: pgmap v12017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 27 KiB/s wr, 8 op/s 2022-09-08T03:50:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:50:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:50:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:06.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:06.698291+0000) 2022-09-08T03:50:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:06.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:06.673629+0000) 2022-09-08T03:50:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:07.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:07.698419+0000) 2022-09-08T03:50:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:07.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:07.673777+0000) 2022-09-08T03:50:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:08 smithi055 ceph-mon[97033]: pgmap v12018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T03:50:08.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:08 smithi059 ceph-mon[97450]: pgmap v12018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T03:50:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:08.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:08.698533+0000) 2022-09-08T03:50:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:08.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:08.673915+0000) 2022-09-08T03:50:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:09.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:09.698678+0000) 2022-09-08T03:50:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:09.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:09.674047+0000) 2022-09-08T03:50:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:10 smithi055 ceph-mon[97033]: pgmap v12019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:50:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:10 smithi055 ceph-mon[97033]: osdmap e830: 12 total, 11 up, 11 in 2022-09-08T03:50:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:10 smithi059 ceph-mon[97450]: pgmap v12019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:50:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:10 smithi059 ceph-mon[97450]: osdmap e830: 12 total, 11 up, 11 in 2022-09-08T03:50:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:10.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:10.698809+0000) 2022-09-08T03:50:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:10.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:10.674189+0000) 2022-09-08T03:50:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:11.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:11.698945+0000) 2022-09-08T03:50:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:11.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:11.674323+0000) 2022-09-08T03:50:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:12 smithi055 ceph-mon[97033]: pgmap v12021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:50:12.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:12 smithi059 ceph-mon[97450]: pgmap v12021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:50:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:12.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:12.699115+0000) 2022-09-08T03:50:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:12.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:12.674455+0000) 2022-09-08T03:50:14.139 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:13.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:13.674599+0000) 2022-09-08T03:50:14.140 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:13.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:13.699252+0000) 2022-09-08T03:50:14.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:50:14.138+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:50:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:14 smithi055 ceph-mon[97033]: pgmap v12022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:50:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:14 smithi055 ceph-mon[97033]: osdmap e831: 12 total, 11 up, 11 in 2022-09-08T03:50:14.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:14 smithi059 ceph-mon[97450]: pgmap v12022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:50:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:14 smithi059 ceph-mon[97450]: osdmap e831: 12 total, 11 up, 11 in 2022-09-08T03:50:14.924 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:14.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:14.699382+0000) 2022-09-08T03:50:14.925 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:14.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:14.674733+0000) 2022-09-08T03:50:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:15.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:15.699514+0000) 2022-09-08T03:50:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:15.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:15.674866+0000) 2022-09-08T03:50:16.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:16 smithi055 ceph-mon[97033]: pgmap v12024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-08T03:50:16.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:16 smithi059 ceph-mon[97450]: pgmap v12024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-08T03:50:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:16.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:16.699756+0000) 2022-09-08T03:50:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:16.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:16.675028+0000) 2022-09-08T03:50:18.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:17.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:17.699890+0000) 2022-09-08T03:50:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:17.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:17.675168+0000) 2022-09-08T03:50:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:18 smithi055 ceph-mon[97033]: pgmap v12025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:50:18.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:18 smithi059 ceph-mon[97450]: pgmap v12025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:50:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:18.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:18.700019+0000) 2022-09-08T03:50:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:18.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:18.675297+0000) 2022-09-08T03:50:20.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:19.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:19.700149+0000) 2022-09-08T03:50:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:19.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:19.675419+0000) 2022-09-08T03:50:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:20 smithi055 ceph-mon[97033]: pgmap v12026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2022-09-08T03:50:20.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:20 smithi059 ceph-mon[97450]: pgmap v12026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2022-09-08T03:50:21.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:20.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:20.700310+0000) 2022-09-08T03:50:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:20.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:20.675602+0000) 2022-09-08T03:50:22.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:21.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:21.700442+0000) 2022-09-08T03:50:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:21.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:21.675737+0000) 2022-09-08T03:50:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:22 smithi055 ceph-mon[97033]: pgmap v12027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:50:22.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:22 smithi059 ceph-mon[97450]: pgmap v12027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:50:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:22.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:22.700582+0000) 2022-09-08T03:50:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:22.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:22.675849+0000) 2022-09-08T03:50:24.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:23.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:23.700730+0000) 2022-09-08T03:50:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:23.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:23.675992+0000) 2022-09-08T03:50:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:24 smithi055 ceph-mon[97033]: pgmap v12028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:50:24.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:24 smithi059 ceph-mon[97450]: pgmap v12028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:50:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:24.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:24.700861+0000) 2022-09-08T03:50:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:24.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:24.676120+0000) 2022-09-08T03:50:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:25.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:25.700997+0000) 2022-09-08T03:50:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:25.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:25.676238+0000) 2022-09-08T03:50:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:26 smithi055 ceph-mon[97033]: pgmap v12029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 94 B/s wr, 7 op/s 2022-09-08T03:50:26.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:26 smithi059 ceph-mon[97450]: pgmap v12029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 94 B/s wr, 7 op/s 2022-09-08T03:50:27.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:26.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:26.701116+0000) 2022-09-08T03:50:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:26.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:26.676370+0000) 2022-09-08T03:50:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:27.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:27.676498+0000) 2022-09-08T03:50:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:27.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:27.701241+0000) 2022-09-08T03:50:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:28 smithi055 ceph-mon[97033]: pgmap v12030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T03:50:28.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:28 smithi059 ceph-mon[97450]: pgmap v12030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T03:50:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:28.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:28.676633+0000) 2022-09-08T03:50:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:28.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:28.701372+0000) 2022-09-08T03:50:29.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:50:29.154+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:50:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:29.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:29.701503+0000) 2022-09-08T03:50:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:29.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:29.676793+0000) 2022-09-08T03:50:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:30 smithi055 ceph-mon[97033]: pgmap v12031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:50:30.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:30 smithi059 ceph-mon[97450]: pgmap v12031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:50:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:30.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:30.701643+0000) 2022-09-08T03:50:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:30.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:30.676941+0000) 2022-09-08T03:50:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:31.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:31.701752+0000) 2022-09-08T03:50:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:31.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:31.677087+0000) 2022-09-08T03:50:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:32 smithi055 ceph-mon[97033]: pgmap v12032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:50:32.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:32 smithi059 ceph-mon[97450]: pgmap v12032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:50:33.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:32.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:32.701867+0000) 2022-09-08T03:50:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:32.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:32.677234+0000) 2022-09-08T03:50:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:33.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:33.701998+0000) 2022-09-08T03:50:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:33.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:33.677383+0000) 2022-09-08T03:50:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:34 smithi055 ceph-mon[97033]: pgmap v12033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:50:34.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:34 smithi059 ceph-mon[97450]: pgmap v12033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:50:35.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:34.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:34.702128+0000) 2022-09-08T03:50:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:34.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:34.677530+0000) 2022-09-08T03:50:36.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:35.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:35.702254+0000) 2022-09-08T03:50:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:35.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:35.677689+0000) 2022-09-08T03:50:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:36 smithi055 ceph-mon[97033]: pgmap v12034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T03:50:36.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:36 smithi059 ceph-mon[97450]: pgmap v12034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T03:50:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:36.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:36.702385+0000) 2022-09-08T03:50:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:36.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:36.677823+0000) 2022-09-08T03:50:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:37.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:37.702542+0000) 2022-09-08T03:50:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:37.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:37.677937+0000) 2022-09-08T03:50:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:38 smithi055 ceph-mon[97033]: pgmap v12035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T03:50:38.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:38 smithi059 ceph-mon[97450]: pgmap v12035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T03:50:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:38.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:38.702681+0000) 2022-09-08T03:50:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:38.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:38.678053+0000) 2022-09-08T03:50:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:39.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:39.702786+0000) 2022-09-08T03:50:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:39.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:39.678173+0000) 2022-09-08T03:50:40.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:40 smithi055 ceph-mon[97033]: pgmap v12036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:50:40.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:40 smithi059 ceph-mon[97450]: pgmap v12036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:50:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:40.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:40.702902+0000) 2022-09-08T03:50:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:40.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:40.678308+0000) 2022-09-08T03:50:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:41.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:41.703028+0000) 2022-09-08T03:50:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:41.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:41.678439+0000) 2022-09-08T03:50:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:42 smithi055 ceph-mon[97033]: pgmap v12037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:50:42.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:42 smithi059 ceph-mon[97450]: pgmap v12037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:50:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:42.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:42.703159+0000) 2022-09-08T03:50:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:42.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:42.678568+0000) 2022-09-08T03:50:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:43.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:43.703276+0000) 2022-09-08T03:50:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:43.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:43.678708+0000) 2022-09-08T03:50:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:44 smithi055 ceph-mon[97033]: pgmap v12038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:50:44.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:44 smithi059 ceph-mon[97450]: pgmap v12038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:50:44.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:50:44.159+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:50:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:44.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:44.703419+0000) 2022-09-08T03:50:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:44.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:44.678840+0000) 2022-09-08T03:50:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:45.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:45.703554+0000) 2022-09-08T03:50:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:45.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:45.678976+0000) 2022-09-08T03:50:46.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:46 smithi055 ceph-mon[97033]: pgmap v12039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T03:50:46.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:46 smithi059 ceph-mon[97450]: pgmap v12039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T03:50:47.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:46.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:46.703670+0000) 2022-09-08T03:50:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:46.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:46.679101+0000) 2022-09-08T03:50:48.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:47.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:47.703783+0000) 2022-09-08T03:50:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:47.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:47.679243+0000) 2022-09-08T03:50:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:48 smithi055 ceph-mon[97033]: pgmap v12040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T03:50:48.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:48 smithi059 ceph-mon[97450]: pgmap v12040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T03:50:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:48.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:48.703931+0000) 2022-09-08T03:50:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:48.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:48.679387+0000) 2022-09-08T03:50:50.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:49.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:49.704076+0000) 2022-09-08T03:50:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:49.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:49.679520+0000) 2022-09-08T03:50:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:50 smithi055 ceph-mon[97033]: pgmap v12041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:50:50.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:50 smithi059 ceph-mon[97450]: pgmap v12041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:50:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:50.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:50.704206+0000) 2022-09-08T03:50:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:50.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:50.679672+0000) 2022-09-08T03:50:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:51.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:51.704335+0000) 2022-09-08T03:50:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:51.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:51.679794+0000) 2022-09-08T03:50:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:52 smithi055 ceph-mon[97033]: pgmap v12042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:50:52.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:52 smithi059 ceph-mon[97450]: pgmap v12042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:50:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:52.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:52.704474+0000) 2022-09-08T03:50:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:52.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:52.679945+0000) 2022-09-08T03:50:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:53.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:53.704595+0000) 2022-09-08T03:50:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:53.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:53.680093+0000) 2022-09-08T03:50:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:54 smithi055 ceph-mon[97033]: pgmap v12043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:50:54.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:54 smithi059 ceph-mon[97450]: pgmap v12043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:50:55.017 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:54.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:54.680240+0000) 2022-09-08T03:50:55.018 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:54.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:54.704706+0000) 2022-09-08T03:50:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:50:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:50:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:50:55.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:50:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:50:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:50:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:55.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:55.704822+0000) 2022-09-08T03:50:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:55.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:55.680389+0000) 2022-09-08T03:50:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:56 smithi055 ceph-mon[97033]: pgmap v12044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T03:50:56.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:56 smithi059 ceph-mon[97450]: pgmap v12044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T03:50:57.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:56.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:56.704935+0000) 2022-09-08T03:50:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:56.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:56.680514+0000) 2022-09-08T03:50:58.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:57.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:57.705059+0000) 2022-09-08T03:50:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:57.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:57.680673+0000) 2022-09-08T03:50:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:50:58 smithi055 ceph-mon[97033]: pgmap v12045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T03:50:58.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:50:58 smithi059 ceph-mon[97450]: pgmap v12045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T03:50:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:58.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:58.705197+0000) 2022-09-08T03:50:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:58.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:58.680833+0000) 2022-09-08T03:50:59.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:50:59.174+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:51:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:50:59.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:59.705330+0000) 2022-09-08T03:51:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:50:59.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:50:59.680982+0000) 2022-09-08T03:51:00.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:00 smithi055 ceph-mon[97033]: pgmap v12046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:00.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:00 smithi059 ceph-mon[97450]: pgmap v12046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:01.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:00.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:00.681125+0000) 2022-09-08T03:51:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:00.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:00.705469+0000) 2022-09-08T03:51:02.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:01.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:01.705601+0000) 2022-09-08T03:51:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:01.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:01.681263+0000) 2022-09-08T03:51:02.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:02 smithi055 ceph-mon[97033]: pgmap v12047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:51:02.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:02 smithi059 ceph-mon[97450]: pgmap v12047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:51:03.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:02.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:02.681409+0000) 2022-09-08T03:51:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:02.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:02.705739+0000) 2022-09-08T03:51:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:03.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:03.705869+0000) 2022-09-08T03:51:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:03.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:03.681548+0000) 2022-09-08T03:51:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:04 smithi055 ceph-mon[97033]: pgmap v12048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:04.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:04 smithi059 ceph-mon[97450]: pgmap v12048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:04.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:04.706009+0000) 2022-09-08T03:51:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:04.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:04.681699+0000) 2022-09-08T03:51:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:05.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:05.706150+0000) 2022-09-08T03:51:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:05.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:05.681820+0000) 2022-09-08T03:51:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:06 smithi055 ceph-mon[97033]: pgmap v12049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T03:51:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:51:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:51:06.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:06 smithi059 ceph-mon[97450]: pgmap v12049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T03:51:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:51:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:51:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:06.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:06.706281+0000) 2022-09-08T03:51:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:06.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:06.681956+0000) 2022-09-08T03:51:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:07.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:07.706445+0000) 2022-09-08T03:51:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:07.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:07.682096+0000) 2022-09-08T03:51:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:08 smithi055 ceph-mon[97033]: pgmap v12050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T03:51:08.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:08 smithi059 ceph-mon[97450]: pgmap v12050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T03:51:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:08.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:08.706565+0000) 2022-09-08T03:51:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:08.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:08.682243+0000) 2022-09-08T03:51:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:09.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:09.706717+0000) 2022-09-08T03:51:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:09.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:09.682362+0000) 2022-09-08T03:51:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:10 smithi055 ceph-mon[97033]: pgmap v12051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:51:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:10 smithi055 ceph-mon[97033]: osdmap e832: 12 total, 11 up, 11 in 2022-09-08T03:51:10.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:10 smithi059 ceph-mon[97450]: pgmap v12051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:51:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:10 smithi059 ceph-mon[97450]: osdmap e832: 12 total, 11 up, 11 in 2022-09-08T03:51:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:10.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:10.706837+0000) 2022-09-08T03:51:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:10.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:10.682499+0000) 2022-09-08T03:51:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:11.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:11.706980+0000) 2022-09-08T03:51:12.188 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:11.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:11.682621+0000) 2022-09-08T03:51:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:12 smithi055 ceph-mon[97033]: pgmap v12053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:51:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:12 smithi059 ceph-mon[97450]: pgmap v12053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:51:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:12.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:12.707101+0000) 2022-09-08T03:51:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:12.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:12.682772+0000) 2022-09-08T03:51:14.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:13.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:13.707212+0000) 2022-09-08T03:51:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:13.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:13.682911+0000) 2022-09-08T03:51:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:14 smithi055 ceph-mon[97033]: pgmap v12054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:51:14.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:14 smithi059 ceph-mon[97450]: pgmap v12054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:51:14.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:51:14.189+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:51:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:14.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:14.707335+0000) 2022-09-08T03:51:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:14.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:14.683039+0000) 2022-09-08T03:51:16.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:15.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:15.707465+0000) 2022-09-08T03:51:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:15.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:15.683171+0000) 2022-09-08T03:51:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:16 smithi055 ceph-mon[97033]: pgmap v12055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:51:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:16 smithi059 ceph-mon[97450]: pgmap v12055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:51:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:16.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:16.707577+0000) 2022-09-08T03:51:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:16.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:16.683303+0000) 2022-09-08T03:51:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:17.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:17.707710+0000) 2022-09-08T03:51:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:17.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:17.683437+0000) 2022-09-08T03:51:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:18 smithi055 ceph-mon[97033]: pgmap v12056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:51:18.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:18 smithi059 ceph-mon[97450]: pgmap v12056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:51:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:18.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:18.707817+0000) 2022-09-08T03:51:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:18.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:18.683574+0000) 2022-09-08T03:51:20.069 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:19.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:19.707928+0000) 2022-09-08T03:51:20.070 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:19.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:19.683714+0000) 2022-09-08T03:51:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:20 smithi055 ceph-mon[97033]: pgmap v12057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:51:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:20 smithi055 ceph-mon[97033]: osdmap e833: 12 total, 11 up, 11 in 2022-09-08T03:51:20.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:20 smithi059 ceph-mon[97450]: pgmap v12057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:51:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:20 smithi059 ceph-mon[97450]: osdmap e833: 12 total, 11 up, 11 in 2022-09-08T03:51:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:20.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:20.683843+0000) 2022-09-08T03:51:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:20.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:20.708114+0000) 2022-09-08T03:51:22.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:21.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:21.708260+0000) 2022-09-08T03:51:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:21.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:21.683990+0000) 2022-09-08T03:51:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:22 smithi055 ceph-mon[97033]: pgmap v12059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:51:22.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:22 smithi059 ceph-mon[97450]: pgmap v12059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:51:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:22.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:22.684117+0000) 2022-09-08T03:51:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:22.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:22.708403+0000) 2022-09-08T03:51:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:23.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:23.708997+0000) 2022-09-08T03:51:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:23.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:23.684260+0000) 2022-09-08T03:51:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:24 smithi055 ceph-mon[97033]: pgmap v12060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:51:24.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:24 smithi059 ceph-mon[97450]: pgmap v12060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:51:25.122 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:24.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:24.709262+0000) 2022-09-08T03:51:25.123 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:24.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:24.684430+0000) 2022-09-08T03:51:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:25.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:25.709536+0000) 2022-09-08T03:51:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:25.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:25.684645+0000) 2022-09-08T03:51:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:26 smithi055 ceph-mon[97033]: pgmap v12061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:51:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:26 smithi055 ceph-mon[97033]: pgmap v12062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:51:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:26 smithi055 ceph-mon[97033]: pgmap v12063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-08T03:51:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:51:26.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:26 smithi059 ceph-mon[97450]: pgmap v12061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:51:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:26 smithi059 ceph-mon[97450]: pgmap v12062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:51:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:26 smithi059 ceph-mon[97450]: pgmap v12063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-08T03:51:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:51:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:26.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:26.684866+0000) 2022-09-08T03:51:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:26.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:26.709718+0000) 2022-09-08T03:51:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:27.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:27.709916+0000) 2022-09-08T03:51:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:27.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:27.685052+0000) 2022-09-08T03:51:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:28 smithi055 ceph-mon[97033]: pgmap v12064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:51:28.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:28 smithi059 ceph-mon[97450]: pgmap v12064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:51:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:28.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:28.685179+0000) 2022-09-08T03:51:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:28.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:28.710138+0000) 2022-09-08T03:51:29.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:51:29.205+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:51:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:29.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:29.710382+0000) 2022-09-08T03:51:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:29.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:29.685408+0000) 2022-09-08T03:51:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:30 smithi055 ceph-mon[97033]: pgmap v12065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:51:30.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:30 smithi059 ceph-mon[97450]: pgmap v12065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:51:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:30.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:30.710608+0000) 2022-09-08T03:51:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:30.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:30.685628+0000) 2022-09-08T03:51:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:31.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:31.710906+0000) 2022-09-08T03:51:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:31.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:31.685831+0000) 2022-09-08T03:51:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:32 smithi055 ceph-mon[97033]: pgmap v12066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:51:32.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:32 smithi059 ceph-mon[97450]: pgmap v12066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:51:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:32.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:32.711163+0000) 2022-09-08T03:51:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:32.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:32.685997+0000) 2022-09-08T03:51:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:33.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:33.711429+0000) 2022-09-08T03:51:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:33.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:33.686295+0000) 2022-09-08T03:51:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:34 smithi055 ceph-mon[97033]: pgmap v12067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-08T03:51:34.538 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:34 smithi059 ceph-mon[97450]: pgmap v12067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-08T03:51:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:34.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:34.711693+0000) 2022-09-08T03:51:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:34.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:34.686487+0000) 2022-09-08T03:51:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:35.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:35.711892+0000) 2022-09-08T03:51:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:35.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:35.686758+0000) 2022-09-08T03:51:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:36 smithi055 ceph-mon[97033]: pgmap v12068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:51:36.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:36 smithi059 ceph-mon[97450]: pgmap v12068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:51:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:36.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:36.712055+0000) 2022-09-08T03:51:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:36.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:36.686934+0000) 2022-09-08T03:51:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:37.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:37.712247+0000) 2022-09-08T03:51:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:37.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:37.687097+0000) 2022-09-08T03:51:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:38 smithi055 ceph-mon[97033]: pgmap v12069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:38.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:38 smithi059 ceph-mon[97450]: pgmap v12069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:38.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:38.687308+0000) 2022-09-08T03:51:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:38.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:38.712401+0000) 2022-09-08T03:51:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:39.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:39.712594+0000) 2022-09-08T03:51:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:39.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:39.687579+0000) 2022-09-08T03:51:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:40 smithi055 ceph-mon[97033]: pgmap v12070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:40.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:40 smithi059 ceph-mon[97450]: pgmap v12070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:40.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:40.712805+0000) 2022-09-08T03:51:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:40.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:40.687777+0000) 2022-09-08T03:51:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:41.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:41.713038+0000) 2022-09-08T03:51:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:41.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:41.688042+0000) 2022-09-08T03:51:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:42 smithi055 ceph-mon[97033]: pgmap v12071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:51:42.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:42 smithi059 ceph-mon[97450]: pgmap v12071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:51:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:42.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:42.713294+0000) 2022-09-08T03:51:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:42.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:42.688261+0000) 2022-09-08T03:51:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:43.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:43.688497+0000) 2022-09-08T03:51:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:43.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:43.713493+0000) 2022-09-08T03:51:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:44 smithi055 ceph-mon[97033]: pgmap v12072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:44.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:44 smithi059 ceph-mon[97450]: pgmap v12072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:44.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:51:44.216+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:51:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:44.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:44.688742+0000) 2022-09-08T03:51:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:44.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:44.713742+0000) 2022-09-08T03:51:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:45.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:45.688913+0000) 2022-09-08T03:51:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:45.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:45.713910+0000) 2022-09-08T03:51:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:46 smithi055 ceph-mon[97033]: pgmap v12073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:51:46.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:46 smithi059 ceph-mon[97450]: pgmap v12073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:51:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:46.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:46.689111+0000) 2022-09-08T03:51:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:46.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:46.714075+0000) 2022-09-08T03:51:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:47.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:47.714304+0000) 2022-09-08T03:51:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:47.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:47.689351+0000) 2022-09-08T03:51:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:48 smithi055 ceph-mon[97033]: pgmap v12074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:48.538 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:48 smithi059 ceph-mon[97450]: pgmap v12074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:49.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:48.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:48.714511+0000) 2022-09-08T03:51:49.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:48.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:48.689528+0000) 2022-09-08T03:51:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:49.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:49.714758+0000) 2022-09-08T03:51:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:49.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:49.689750+0000) 2022-09-08T03:51:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:50 smithi055 ceph-mon[97033]: pgmap v12075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:50.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:50 smithi059 ceph-mon[97450]: pgmap v12075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:50.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:50.714995+0000) 2022-09-08T03:51:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:50.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:50.689968+0000) 2022-09-08T03:51:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:51.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:51.690239+0000) 2022-09-08T03:51:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:51.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:51.715231+0000) 2022-09-08T03:51:52.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:52 smithi055 ceph-mon[97033]: pgmap v12076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:51:52.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:52 smithi059 ceph-mon[97450]: pgmap v12076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:51:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:52.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:52.715468+0000) 2022-09-08T03:51:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:52.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:52.690520+0000) 2022-09-08T03:51:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:53.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:53.715721+0000) 2022-09-08T03:51:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:53.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:53.690809+0000) 2022-09-08T03:51:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:54 smithi055 ceph-mon[97033]: pgmap v12077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:54 smithi059 ceph-mon[97450]: pgmap v12077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:51:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:54.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:54.715967+0000) 2022-09-08T03:51:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:54.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:54.690975+0000) 2022-09-08T03:51:56.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:55.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:55.716137+0000) 2022-09-08T03:51:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:55.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:55.691239+0000) 2022-09-08T03:51:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:56 smithi055 ceph-mon[97033]: pgmap v12078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:51:56.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:56 smithi059 ceph-mon[97450]: pgmap v12078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:51:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:56.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:56.716334+0000) 2022-09-08T03:51:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:56.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:56.691494+0000) 2022-09-08T03:51:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:57.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:57.716548+0000) 2022-09-08T03:51:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:57.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:57.691773+0000) 2022-09-08T03:51:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:51:58 smithi055 ceph-mon[97033]: pgmap v12079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:51:58.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:51:58 smithi059 ceph-mon[97450]: pgmap v12079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:51:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:58.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:58.716728+0000) 2022-09-08T03:51:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:58.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:58.691931+0000) 2022-09-08T03:51:59.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:51:59.228+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:52:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:51:59.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:59.716894+0000) 2022-09-08T03:52:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:51:59.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:51:59.692074+0000) 2022-09-08T03:52:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:00 smithi055 ceph-mon[97033]: pgmap v12080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:52:00.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:00 smithi059 ceph-mon[97450]: pgmap v12080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:52:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:00.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:00.717109+0000) 2022-09-08T03:52:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:00.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:00.692270+0000) 2022-09-08T03:52:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:01.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:01.692523+0000) 2022-09-08T03:52:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:01.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:01.717298+0000) 2022-09-08T03:52:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:02 smithi055 ceph-mon[97033]: pgmap v12081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:52:02.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:02 smithi059 ceph-mon[97450]: pgmap v12081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:52:03.014 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:02.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:02.692743+0000) 2022-09-08T03:52:03.015 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:02.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:02.717492+0000) 2022-09-08T03:52:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:03.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:03.692985+0000) 2022-09-08T03:52:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:03.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:03.717722+0000) 2022-09-08T03:52:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:04 smithi055 ceph-mon[97033]: pgmap v12082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T03:52:04.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:04 smithi059 ceph-mon[97450]: pgmap v12082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T03:52:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:04.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:04.717897+0000) 2022-09-08T03:52:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:04.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:04.693194+0000) 2022-09-08T03:52:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:05.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:05.718066+0000) 2022-09-08T03:52:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:05.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:05.693432+0000) 2022-09-08T03:52:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:06 smithi055 ceph-mon[97033]: pgmap v12083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:52:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:52:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:52:06.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:06 smithi059 ceph-mon[97450]: pgmap v12083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:52:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:52:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:52:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:06.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:06.718294+0000) 2022-09-08T03:52:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:06.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:06.693586+0000) 2022-09-08T03:52:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:07.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:07.718510+0000) 2022-09-08T03:52:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:07.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:07.693759+0000) 2022-09-08T03:52:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:08 smithi055 ceph-mon[97033]: pgmap v12084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:52:08.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:08 smithi059 ceph-mon[97450]: pgmap v12084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:52:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:08.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:08.718742+0000) 2022-09-08T03:52:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:08.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:08.693951+0000) 2022-09-08T03:52:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:09.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:09.718974+0000) 2022-09-08T03:52:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:09.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:09.694090+0000) 2022-09-08T03:52:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:10 smithi055 ceph-mon[97033]: pgmap v12085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:52:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:10 smithi055 ceph-mon[97033]: osdmap e834: 12 total, 11 up, 11 in 2022-09-08T03:52:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:10 smithi059 ceph-mon[97450]: pgmap v12085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:52:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:10 smithi059 ceph-mon[97450]: osdmap e834: 12 total, 11 up, 11 in 2022-09-08T03:52:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:10.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:10.719127+0000) 2022-09-08T03:52:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:10.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:10.694340+0000) 2022-09-08T03:52:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:11.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:11.719362+0000) 2022-09-08T03:52:12.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:11.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:11.694565+0000) 2022-09-08T03:52:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:12 smithi055 ceph-mon[97033]: pgmap v12087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:52:12.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:12 smithi059 ceph-mon[97450]: pgmap v12087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:52:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:12.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:12.719630+0000) 2022-09-08T03:52:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:12.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:12.694802+0000) 2022-09-08T03:52:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:13.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:13.719897+0000) 2022-09-08T03:52:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:13.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:13.694991+0000) 2022-09-08T03:52:14.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:14 smithi055 ceph-mon[97033]: pgmap v12088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:52:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:14 smithi055 ceph-mon[97033]: osdmap e835: 12 total, 11 up, 11 in 2022-09-08T03:52:14.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:14 smithi059 ceph-mon[97450]: pgmap v12088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:52:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:14 smithi059 ceph-mon[97450]: osdmap e835: 12 total, 11 up, 11 in 2022-09-08T03:52:14.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:52:14.237+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:52:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:14.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:14.720131+0000) 2022-09-08T03:52:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:14.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:14.695127+0000) 2022-09-08T03:52:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:15.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:15.720383+0000) 2022-09-08T03:52:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:15.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:15.695263+0000) 2022-09-08T03:52:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:16 smithi055 ceph-mon[97033]: pgmap v12090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:52:16.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:16 smithi059 ceph-mon[97450]: pgmap v12090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:52:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:16.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:16.720519+0000) 2022-09-08T03:52:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:16.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:16.695482+0000) 2022-09-08T03:52:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:17.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:17.720758+0000) 2022-09-08T03:52:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:17.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:17.695726+0000) 2022-09-08T03:52:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:18 smithi055 ceph-mon[97033]: pgmap v12091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:52:18.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:18 smithi059 ceph-mon[97450]: pgmap v12091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:52:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:18.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:18.720879+0000) 2022-09-08T03:52:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:18.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:18.695904+0000) 2022-09-08T03:52:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:19.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:19.721087+0000) 2022-09-08T03:52:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:19.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:19.696153+0000) 2022-09-08T03:52:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:20 smithi055 ceph-mon[97033]: pgmap v12092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T03:52:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:20 smithi059 ceph-mon[97450]: pgmap v12092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T03:52:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:20.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:20.721247+0000) 2022-09-08T03:52:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:20.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:20.696320+0000) 2022-09-08T03:52:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:21.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:21.721375+0000) 2022-09-08T03:52:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:21.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:21.696511+0000) 2022-09-08T03:52:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:22 smithi055 ceph-mon[97033]: pgmap v12093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:52:22.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:22 smithi059 ceph-mon[97450]: pgmap v12093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:52:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:22.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:22.696661+0000) 2022-09-08T03:52:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:22.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:22.721580+0000) 2022-09-08T03:52:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:23.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:23.721725+0000) 2022-09-08T03:52:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:23.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:23.696790+0000) 2022-09-08T03:52:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:24 smithi055 ceph-mon[97033]: pgmap v12094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:52:24.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:24 smithi059 ceph-mon[97450]: pgmap v12094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:52:25.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:24.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:24.721903+0000) 2022-09-08T03:52:25.164 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:24.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:24.696927+0000) 2022-09-08T03:52:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:25.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:25.722036+0000) 2022-09-08T03:52:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:25.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:25.697167+0000) 2022-09-08T03:52:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:26 smithi055 ceph-mon[97033]: pgmap v12095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 92 B/s wr, 8 op/s 2022-09-08T03:52:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:52:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:52:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:52:26.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:26 smithi059 ceph-mon[97450]: pgmap v12095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 92 B/s wr, 8 op/s 2022-09-08T03:52:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:52:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:52:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:52:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:26.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:26.722170+0000) 2022-09-08T03:52:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:26.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:26.697337+0000) 2022-09-08T03:52:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:27.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:27.722398+0000) 2022-09-08T03:52:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:27.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:27.697568+0000) 2022-09-08T03:52:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:28 smithi055 ceph-mon[97033]: pgmap v12096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:28.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:28 smithi059 ceph-mon[97450]: pgmap v12096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:28.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:28.722579+0000) 2022-09-08T03:52:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:28.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:28.697846+0000) 2022-09-08T03:52:29.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:52:29.238+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:52:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:29.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:29.722784+0000) 2022-09-08T03:52:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:29.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:29.698099+0000) 2022-09-08T03:52:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:30 smithi055 ceph-mon[97033]: pgmap v12097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:30.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:30 smithi059 ceph-mon[97450]: pgmap v12097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:30.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:30.698310+0000) 2022-09-08T03:52:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:30.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:30.722958+0000) 2022-09-08T03:52:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:31.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:31.698542+0000) 2022-09-08T03:52:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:31.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:31.723163+0000) 2022-09-08T03:52:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:32 smithi055 ceph-mon[97033]: pgmap v12098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:52:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:32 smithi059 ceph-mon[97450]: pgmap v12098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:52:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:32.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:32.723385+0000) 2022-09-08T03:52:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:32.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:32.698783+0000) 2022-09-08T03:52:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:33.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:33.723576+0000) 2022-09-08T03:52:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:33.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:33.698966+0000) 2022-09-08T03:52:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:34 smithi055 ceph-mon[97033]: pgmap v12099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:34.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:34 smithi059 ceph-mon[97450]: pgmap v12099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:34.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:34.723804+0000) 2022-09-08T03:52:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:34.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:34.699159+0000) 2022-09-08T03:52:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:35.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:35.724059+0000) 2022-09-08T03:52:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:35.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:35.699384+0000) 2022-09-08T03:52:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:36 smithi055 ceph-mon[97033]: pgmap v12100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:52:36.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:36 smithi059 ceph-mon[97450]: pgmap v12100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:52:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:36.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:36.724204+0000) 2022-09-08T03:52:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:36.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:36.699576+0000) 2022-09-08T03:52:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:37.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:37.699783+0000) 2022-09-08T03:52:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:37.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:37.724372+0000) 2022-09-08T03:52:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:38 smithi055 ceph-mon[97033]: pgmap v12101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:38.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:38 smithi059 ceph-mon[97450]: pgmap v12101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:38.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:38.724586+0000) 2022-09-08T03:52:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:38.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:38.699999+0000) 2022-09-08T03:52:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:39.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:39.724828+0000) 2022-09-08T03:52:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:39.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:39.700185+0000) 2022-09-08T03:52:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:40 smithi055 ceph-mon[97033]: pgmap v12102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:40.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:40 smithi059 ceph-mon[97450]: pgmap v12102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:40.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:40.725011+0000) 2022-09-08T03:52:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:40.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:40.700403+0000) 2022-09-08T03:52:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:41.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:41.725267+0000) 2022-09-08T03:52:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:41.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:41.700685+0000) 2022-09-08T03:52:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:42 smithi055 ceph-mon[97033]: pgmap v12103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:52:42.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:42 smithi059 ceph-mon[97450]: pgmap v12103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:52:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:42.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:42.725477+0000) 2022-09-08T03:52:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:42.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:42.700834+0000) 2022-09-08T03:52:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:43.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:43.725653+0000) 2022-09-08T03:52:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:43.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:43.701035+0000) 2022-09-08T03:52:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:44 smithi055 ceph-mon[97033]: pgmap v12104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:44.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:44 smithi059 ceph-mon[97450]: pgmap v12104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:44.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:52:44.249+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:52:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:44.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:44.725845+0000) 2022-09-08T03:52:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:44.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:44.701270+0000) 2022-09-08T03:52:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:45.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:45.725997+0000) 2022-09-08T03:52:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:45.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:45.701518+0000) 2022-09-08T03:52:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:46 smithi055 ceph-mon[97033]: pgmap v12105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:52:46.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:46 smithi059 ceph-mon[97450]: pgmap v12105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:52:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:46.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:46.726177+0000) 2022-09-08T03:52:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:46.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:46.701767+0000) 2022-09-08T03:52:48.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:47.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:47.726316+0000) 2022-09-08T03:52:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:47.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:47.701984+0000) 2022-09-08T03:52:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:48 smithi055 ceph-mon[97033]: pgmap v12106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:48.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:48 smithi059 ceph-mon[97450]: pgmap v12106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:48.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:48.702230+0000) 2022-09-08T03:52:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:48.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:48.726541+0000) 2022-09-08T03:52:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:49.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:49.726703+0000) 2022-09-08T03:52:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:49.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:49.702417+0000) 2022-09-08T03:52:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:50 smithi055 ceph-mon[97033]: pgmap v12107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:50 smithi059 ceph-mon[97450]: pgmap v12107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:50.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:50.726942+0000) 2022-09-08T03:52:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:50.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:50.702663+0000) 2022-09-08T03:52:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:51.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:51.702907+0000) 2022-09-08T03:52:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:51.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:51.727187+0000) 2022-09-08T03:52:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:52 smithi055 ceph-mon[97033]: pgmap v12108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:52:52.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:52 smithi059 ceph-mon[97450]: pgmap v12108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:52:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:52.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:52.727403+0000) 2022-09-08T03:52:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:52.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:52.703156+0000) 2022-09-08T03:52:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:53.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:53.727691+0000) 2022-09-08T03:52:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:53.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:53.703418+0000) 2022-09-08T03:52:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:54 smithi055 ceph-mon[97033]: pgmap v12109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:54.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:54 smithi059 ceph-mon[97450]: pgmap v12109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:52:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:54.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:54.727941+0000) 2022-09-08T03:52:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:54.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:54.703631+0000) 2022-09-08T03:52:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:55.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:55.728093+0000) 2022-09-08T03:52:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:55.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:55.703886+0000) 2022-09-08T03:52:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:56 smithi055 ceph-mon[97033]: pgmap v12110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:52:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:56 smithi055 ceph-mon[97033]: pgmap v12111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:52:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:56 smithi055 ceph-mon[97033]: pgmap v12112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:52:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:52:56.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:56 smithi059 ceph-mon[97450]: pgmap v12110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:52:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:56 smithi059 ceph-mon[97450]: pgmap v12111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:52:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:56 smithi059 ceph-mon[97450]: pgmap v12112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:52:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:52:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:56.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:56.728297+0000) 2022-09-08T03:52:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:56.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:56.704185+0000) 2022-09-08T03:52:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:57.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:57.728509+0000) 2022-09-08T03:52:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:57.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:57.704419+0000) 2022-09-08T03:52:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:52:58 smithi055 ceph-mon[97033]: pgmap v12113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:52:58.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:52:58 smithi059 ceph-mon[97450]: pgmap v12113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:52:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:58.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:58.704744+0000) 2022-09-08T03:52:59.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:58.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:58.728729+0000) 2022-09-08T03:52:59.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:52:59.263+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:53:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:52:59.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:59.728886+0000) 2022-09-08T03:53:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:52:59.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:52:59.705001+0000) 2022-09-08T03:53:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:00 smithi055 ceph-mon[97033]: pgmap v12114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:53:00.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:00 smithi059 ceph-mon[97450]: pgmap v12114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:53:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:00.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:00.729012+0000) 2022-09-08T03:53:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:00.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:00.705215+0000) 2022-09-08T03:53:02.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:01.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:01.705453+0000) 2022-09-08T03:53:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:01.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:01.729148+0000) 2022-09-08T03:53:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:02 smithi055 ceph-mon[97033]: pgmap v12115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:53:02.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:02 smithi059 ceph-mon[97450]: pgmap v12115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:53:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:02.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:02.705708+0000) 2022-09-08T03:53:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:02.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:02.729421+0000) 2022-09-08T03:53:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:03.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:03.729739+0000) 2022-09-08T03:53:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:03.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:03.705919+0000) 2022-09-08T03:53:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:04 smithi055 ceph-mon[97033]: pgmap v12116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:53:04.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:04 smithi059 ceph-mon[97450]: pgmap v12116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:53:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:04.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:04.730003+0000) 2022-09-08T03:53:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:04.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:04.706195+0000) 2022-09-08T03:53:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:05.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:05.730217+0000) 2022-09-08T03:53:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:05.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:05.706463+0000) 2022-09-08T03:53:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:06 smithi055 ceph-mon[97033]: pgmap v12117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:53:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:53:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:53:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:06 smithi059 ceph-mon[97450]: pgmap v12117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:53:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:53:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:53:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:06.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:06.730443+0000) 2022-09-08T03:53:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:06.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:06.706726+0000) 2022-09-08T03:53:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:07.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:07.730723+0000) 2022-09-08T03:53:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:07.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:07.706958+0000) 2022-09-08T03:53:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:08 smithi055 ceph-mon[97033]: pgmap v12118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:53:08.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:08 smithi059 ceph-mon[97450]: pgmap v12118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:53:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:08.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:08.730990+0000) 2022-09-08T03:53:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:08.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:08.707110+0000) 2022-09-08T03:53:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:09.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:09.731145+0000) 2022-09-08T03:53:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:09.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:09.707287+0000) 2022-09-08T03:53:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:10 smithi055 ceph-mon[97033]: pgmap v12119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:53:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:10 smithi055 ceph-mon[97033]: osdmap e836: 12 total, 11 up, 11 in 2022-09-08T03:53:10.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:10 smithi059 ceph-mon[97450]: pgmap v12119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:53:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:10 smithi059 ceph-mon[97450]: osdmap e836: 12 total, 11 up, 11 in 2022-09-08T03:53:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:10.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:10.731357+0000) 2022-09-08T03:53:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:10.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:10.707444+0000) 2022-09-08T03:53:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:11.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:11.731543+0000) 2022-09-08T03:53:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:11.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:11.707683+0000) 2022-09-08T03:53:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:12 smithi055 ceph-mon[97033]: pgmap v12121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:53:12.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:12 smithi059 ceph-mon[97450]: pgmap v12121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:53:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:12.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:12.731757+0000) 2022-09-08T03:53:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:12.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:12.707916+0000) 2022-09-08T03:53:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:13.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:13.731981+0000) 2022-09-08T03:53:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:13.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:13.708157+0000) 2022-09-08T03:53:14.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:53:14.276+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:53:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:14 smithi055 ceph-mon[97033]: pgmap v12122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:53:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:14 smithi055 ceph-mon[97033]: osdmap e837: 12 total, 11 up, 11 in 2022-09-08T03:53:14.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:14 smithi059 ceph-mon[97450]: pgmap v12122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:53:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:14 smithi059 ceph-mon[97450]: osdmap e837: 12 total, 11 up, 11 in 2022-09-08T03:53:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:14.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:14.732187+0000) 2022-09-08T03:53:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:14.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:14.708379+0000) 2022-09-08T03:53:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:15.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:15.732441+0000) 2022-09-08T03:53:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:15.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:15.708678+0000) 2022-09-08T03:53:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:16 smithi055 ceph-mon[97033]: pgmap v12124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:53:16.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:16 smithi059 ceph-mon[97450]: pgmap v12124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:53:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:16.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:16.732667+0000) 2022-09-08T03:53:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:16.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:16.708907+0000) 2022-09-08T03:53:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:17.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:17.732870+0000) 2022-09-08T03:53:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:17.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:17.709122+0000) 2022-09-08T03:53:18.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:18 smithi059 ceph-mon[97450]: pgmap v12125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:53:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:18 smithi055 ceph-mon[97033]: pgmap v12125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:53:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:18.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:18.733131+0000) 2022-09-08T03:53:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:18.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:18.709399+0000) 2022-09-08T03:53:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:19.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:19.733302+0000) 2022-09-08T03:53:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:19.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:19.709686+0000) 2022-09-08T03:53:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:20 smithi055 ceph-mon[97033]: pgmap v12126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T03:53:20.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:20 smithi059 ceph-mon[97450]: pgmap v12126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T03:53:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:20.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:20.733492+0000) 2022-09-08T03:53:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:20.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:20.709938+0000) 2022-09-08T03:53:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:21.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:21.733684+0000) 2022-09-08T03:53:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:21.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:21.710132+0000) 2022-09-08T03:53:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:22 smithi055 ceph-mon[97033]: pgmap v12127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:53:22.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:22 smithi059 ceph-mon[97450]: pgmap v12127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:53:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:22.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:22.733902+0000) 2022-09-08T03:53:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:22.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:22.710363+0000) 2022-09-08T03:53:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:23.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:23.734171+0000) 2022-09-08T03:53:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:23.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:23.710573+0000) 2022-09-08T03:53:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:24 smithi055 ceph-mon[97033]: pgmap v12128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:53:24.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:24 smithi059 ceph-mon[97450]: pgmap v12128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:53:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:24.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:24.734303+0000) 2022-09-08T03:53:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:24.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:24.710743+0000) 2022-09-08T03:53:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:25.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:25.734447+0000) 2022-09-08T03:53:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:25.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:25.711060+0000) 2022-09-08T03:53:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:26 smithi055 ceph-mon[97033]: pgmap v12129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2022-09-08T03:53:26.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:26 smithi059 ceph-mon[97450]: pgmap v12129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2022-09-08T03:53:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:26.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:26.734657+0000) 2022-09-08T03:53:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:26.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:26.711297+0000) 2022-09-08T03:53:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:27.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:27.734862+0000) 2022-09-08T03:53:28.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:27.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:27.711440+0000) 2022-09-08T03:53:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:28 smithi055 ceph-mon[97033]: pgmap v12130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:53:28.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:28 smithi059 ceph-mon[97450]: pgmap v12130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:53:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:28.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:28.735065+0000) 2022-09-08T03:53:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:28.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:28.711584+0000) 2022-09-08T03:53:29.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:53:29.288+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:53:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:29.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:29.711811+0000) 2022-09-08T03:53:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:29.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:29.735275+0000) 2022-09-08T03:53:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:30 smithi055 ceph-mon[97033]: pgmap v12131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:53:30.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:30 smithi059 ceph-mon[97450]: pgmap v12131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:53:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:30.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:30.735517+0000) 2022-09-08T03:53:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:30.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:30.712015+0000) 2022-09-08T03:53:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:31.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:31.735782+0000) 2022-09-08T03:53:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:31.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:31.712216+0000) 2022-09-08T03:53:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:32 smithi055 ceph-mon[97033]: pgmap v12132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:53:32.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:32 smithi059 ceph-mon[97450]: pgmap v12132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:53:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:32.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:32.735967+0000) 2022-09-08T03:53:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:32.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:32.712428+0000) 2022-09-08T03:53:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:33.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:33.736171+0000) 2022-09-08T03:53:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:33.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:33.712655+0000) 2022-09-08T03:53:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:34 smithi055 ceph-mon[97033]: pgmap v12133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:53:34.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:34 smithi059 ceph-mon[97450]: pgmap v12133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:53:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:34.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:34.736362+0000) 2022-09-08T03:53:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:34.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:34.712856+0000) 2022-09-08T03:53:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:35.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:35.736554+0000) 2022-09-08T03:53:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:35.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:35.713085+0000) 2022-09-08T03:53:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:36 smithi055 ceph-mon[97033]: pgmap v12134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:53:36.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:36 smithi059 ceph-mon[97450]: pgmap v12134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:53:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:36.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:36.736746+0000) 2022-09-08T03:53:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:36.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:36.713256+0000) 2022-09-08T03:53:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:37.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:37.736998+0000) 2022-09-08T03:53:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:37.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:37.713427+0000) 2022-09-08T03:53:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:38 smithi055 ceph-mon[97033]: pgmap v12135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:53:38.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:38 smithi059 ceph-mon[97450]: pgmap v12135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:53:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:38.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:38.737204+0000) 2022-09-08T03:53:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:38.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:38.713669+0000) 2022-09-08T03:53:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:39.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:39.737423+0000) 2022-09-08T03:53:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:39.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:39.713917+0000) 2022-09-08T03:53:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:40 smithi055 ceph-mon[97033]: pgmap v12136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:53:40.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:40 smithi059 ceph-mon[97450]: pgmap v12136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T03:53:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:40.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:40.737650+0000) 2022-09-08T03:53:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:40.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:40.714159+0000) 2022-09-08T03:53:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:41.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:41.737843+0000) 2022-09-08T03:53:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:41.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:41.714379+0000) 2022-09-08T03:53:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:42 smithi055 ceph-mon[97033]: pgmap v12137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:53:42.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:42 smithi059 ceph-mon[97450]: pgmap v12137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:53:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:42.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:42.737997+0000) 2022-09-08T03:53:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:42.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:42.714601+0000) 2022-09-08T03:53:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:43.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:43.738131+0000) 2022-09-08T03:53:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:43.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:43.714807+0000) 2022-09-08T03:53:44.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:53:44.290+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:53:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:44 smithi055 ceph-mon[97033]: pgmap v12138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:53:44.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:44 smithi059 ceph-mon[97450]: pgmap v12138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:53:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:44.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:44.738336+0000) 2022-09-08T03:53:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:44.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:44.715008+0000) 2022-09-08T03:53:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:45.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:45.738517+0000) 2022-09-08T03:53:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:45.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:45.715225+0000) 2022-09-08T03:53:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:46 smithi055 ceph-mon[97033]: pgmap v12139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:53:46.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:46 smithi059 ceph-mon[97450]: pgmap v12139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:53:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:46.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:46.738714+0000) 2022-09-08T03:53:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:46.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:46.715449+0000) 2022-09-08T03:53:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:47.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:47.738902+0000) 2022-09-08T03:53:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:47.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:47.715722+0000) 2022-09-08T03:53:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:48 smithi055 ceph-mon[97033]: pgmap v12140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:53:48.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:48 smithi059 ceph-mon[97450]: pgmap v12140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:53:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:48.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:48.739058+0000) 2022-09-08T03:53:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:48.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:48.715920+0000) 2022-09-08T03:53:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:49.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:49.739232+0000) 2022-09-08T03:53:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:49.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:49.716111+0000) 2022-09-08T03:53:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:50 smithi055 ceph-mon[97033]: pgmap v12141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:53:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:50 smithi059 ceph-mon[97450]: pgmap v12141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:53:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:50.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:50.739468+0000) 2022-09-08T03:53:51.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:50.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:50.716329+0000) 2022-09-08T03:53:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:51.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:51.739772+0000) 2022-09-08T03:53:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:51.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:51.716602+0000) 2022-09-08T03:53:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:52 smithi055 ceph-mon[97033]: pgmap v12142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:53:52.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:52 smithi059 ceph-mon[97450]: pgmap v12142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:53:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:52.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:52.739947+0000) 2022-09-08T03:53:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:52.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:52.716838+0000) 2022-09-08T03:53:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:53.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:53.740063+0000) 2022-09-08T03:53:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:53.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:53.716982+0000) 2022-09-08T03:53:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:54 smithi055 ceph-mon[97033]: pgmap v12143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:53:54.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:54 smithi059 ceph-mon[97450]: pgmap v12143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:53:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:54.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:54.740298+0000) 2022-09-08T03:53:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:54.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:54.717181+0000) 2022-09-08T03:53:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:55.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:55.740471+0000) 2022-09-08T03:53:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:55.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:55.717449+0000) 2022-09-08T03:53:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:56 smithi055 ceph-mon[97033]: pgmap v12144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:53:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:53:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:53:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:53:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:53:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:53:56.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:56 smithi059 ceph-mon[97450]: pgmap v12144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:53:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:53:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:53:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:53:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:53:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:53:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:56.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:56.740591+0000) 2022-09-08T03:53:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:56.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:56.717686+0000) 2022-09-08T03:53:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:57.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:57.740762+0000) 2022-09-08T03:53:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:57.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:57.717914+0000) 2022-09-08T03:53:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:58 smithi055 ceph-mon[97033]: pgmap v12145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:53:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:53:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:53:58.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:58 smithi059 ceph-mon[97450]: pgmap v12145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:53:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:53:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:53:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:58.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:58.741011+0000) 2022-09-08T03:53:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:58.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:58.718162+0000) 2022-09-08T03:53:59.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:53:59.302+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:53:59.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:53:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:53:59.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:53:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:53:59.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:53:59.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:53:59.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:53:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T03:54:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:53:59.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:59.741157+0000) 2022-09-08T03:54:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:53:59.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:53:59.718416+0000) 2022-09-08T03:54:00.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:00 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T03:54:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:00 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T03:54:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:00 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T03:54:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:00 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T03:54:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:00 smithi059 ceph-mon[97450]: pgmap v12146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:00 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T03:54:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:00 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T03:54:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:00 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T03:54:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:00 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T03:54:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:00 smithi055 ceph-mon[97033]: pgmap v12146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:00.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:00.741326+0000) 2022-09-08T03:54:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:00.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:00.718560+0000) 2022-09-08T03:54:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:01.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:01.741529+0000) 2022-09-08T03:54:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:01.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:01.718778+0000) 2022-09-08T03:54:02.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:02 smithi059 ceph-mon[97450]: pgmap v12147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:54:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:02 smithi055 ceph-mon[97033]: pgmap v12147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:54:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:02.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:02.741762+0000) 2022-09-08T03:54:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:02.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:02.719032+0000) 2022-09-08T03:54:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:03.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:03.741938+0000) 2022-09-08T03:54:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:03.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:03.719212+0000) 2022-09-08T03:54:04.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:04 smithi059 ceph-mon[97450]: pgmap v12148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:04 smithi055 ceph-mon[97033]: pgmap v12148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:04.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:04.742173+0000) 2022-09-08T03:54:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:04.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:04.719399+0000) 2022-09-08T03:54:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:54:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:54:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:05.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:05.742338+0000) 2022-09-08T03:54:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:05.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:05.719625+0000) 2022-09-08T03:54:06.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:06 smithi059 ceph-mon[97450]: pgmap v12149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:54:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:54:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:06 smithi055 ceph-mon[97033]: pgmap v12149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:54:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:54:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:06.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:06.742546+0000) 2022-09-08T03:54:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:06.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:06.719790+0000) 2022-09-08T03:54:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:07.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:07.742720+0000) 2022-09-08T03:54:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:07.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:07.720023+0000) 2022-09-08T03:54:08.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:08 smithi059 ceph-mon[97450]: pgmap v12150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:54:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:08 smithi055 ceph-mon[97033]: pgmap v12150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:54:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:08.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:08.742891+0000) 2022-09-08T03:54:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:08.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:08.720258+0000) 2022-09-08T03:54:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:09.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:09.743093+0000) 2022-09-08T03:54:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:09.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:09.720423+0000) 2022-09-08T03:54:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:10 smithi059 ceph-mon[97450]: pgmap v12151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:54:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:10 smithi059 ceph-mon[97450]: osdmap e838: 12 total, 11 up, 11 in 2022-09-08T03:54:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:10 smithi055 ceph-mon[97033]: pgmap v12151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:54:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:10 smithi055 ceph-mon[97033]: osdmap e838: 12 total, 11 up, 11 in 2022-09-08T03:54:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:10.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:10.743333+0000) 2022-09-08T03:54:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:10.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:10.720659+0000) 2022-09-08T03:54:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:11.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:11.743495+0000) 2022-09-08T03:54:12.236 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:11.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:11.720882+0000) 2022-09-08T03:54:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:12 smithi059 ceph-mon[97450]: pgmap v12153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:54:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:12 smithi055 ceph-mon[97033]: pgmap v12153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:54:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:12.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:12.743700+0000) 2022-09-08T03:54:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:12.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:12.721063+0000) 2022-09-08T03:54:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:13.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:13.743849+0000) 2022-09-08T03:54:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:13.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:13.721275+0000) 2022-09-08T03:54:14.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:54:14.317+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:54:14.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:14 smithi059 ceph-mon[97450]: pgmap v12154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:54:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:14 smithi055 ceph-mon[97033]: pgmap v12154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:54:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:14.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:14.744009+0000) 2022-09-08T03:54:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:14.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:14.721517+0000) 2022-09-08T03:54:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:15.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:15.744237+0000) 2022-09-08T03:54:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:15.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:15.721800+0000) 2022-09-08T03:54:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:16 smithi059 ceph-mon[97450]: pgmap v12155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:54:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:16 smithi055 ceph-mon[97033]: pgmap v12155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:54:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:16.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:16.744502+0000) 2022-09-08T03:54:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:16.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:16.722030+0000) 2022-09-08T03:54:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:17.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:17.744699+0000) 2022-09-08T03:54:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:17.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:17.722194+0000) 2022-09-08T03:54:18.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:18 smithi059 ceph-mon[97450]: pgmap v12156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:54:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:18 smithi055 ceph-mon[97033]: pgmap v12156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:54:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:18.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:18.744940+0000) 2022-09-08T03:54:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:18.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:18.722375+0000) 2022-09-08T03:54:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:19.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:19.745090+0000) 2022-09-08T03:54:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:19.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:19.722584+0000) 2022-09-08T03:54:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:20 smithi055 ceph-mon[97033]: osdmap e839: 12 total, 11 up, 11 in 2022-09-08T03:54:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:20 smithi055 ceph-mon[97033]: pgmap v12158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T03:54:20.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:20 smithi059 ceph-mon[97450]: osdmap e839: 12 total, 11 up, 11 in 2022-09-08T03:54:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:20 smithi059 ceph-mon[97450]: pgmap v12158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T03:54:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:20.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:20.745290+0000) 2022-09-08T03:54:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:20.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:20.722784+0000) 2022-09-08T03:54:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:21.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:21.745515+0000) 2022-09-08T03:54:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:21.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:21.722994+0000) 2022-09-08T03:54:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:22 smithi059 ceph-mon[97450]: pgmap v12159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:54:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:22 smithi055 ceph-mon[97033]: pgmap v12159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:54:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:22.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:22.723224+0000) 2022-09-08T03:54:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:22.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:22.745736+0000) 2022-09-08T03:54:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:23.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:23.745922+0000) 2022-09-08T03:54:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:23.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:23.723490+0000) 2022-09-08T03:54:24.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:24 smithi059 ceph-mon[97450]: pgmap v12160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:54:24.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:24 smithi055 ceph-mon[97033]: pgmap v12160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:54:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:24.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:24.746153+0000) 2022-09-08T03:54:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:24.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:24.723738+0000) 2022-09-08T03:54:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:25.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:25.723890+0000) 2022-09-08T03:54:26.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:25.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:25.746420+0000) 2022-09-08T03:54:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:26 smithi059 ceph-mon[97450]: pgmap v12161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:54:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:54:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:26 smithi055 ceph-mon[97033]: pgmap v12161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:54:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:54:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:26.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:26.724111+0000) 2022-09-08T03:54:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:26.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:26.746704+0000) 2022-09-08T03:54:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:27 smithi059 ceph-mon[97450]: pgmap v12162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:54:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:27 smithi059 ceph-mon[97450]: pgmap v12163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-08T03:54:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:27 smithi055 ceph-mon[97033]: pgmap v12162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:54:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:27 smithi055 ceph-mon[97033]: pgmap v12163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-08T03:54:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:27.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:27.746918+0000) 2022-09-08T03:54:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:27.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:27.724300+0000) 2022-09-08T03:54:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:28.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:28.724527+0000) 2022-09-08T03:54:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:28.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:28.747144+0000) 2022-09-08T03:54:29.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:54:29.326+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:54:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:29 smithi059 ceph-mon[97450]: pgmap v12164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:54:29.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:29 smithi055 ceph-mon[97033]: pgmap v12164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:54:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:29.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:29.747370+0000) 2022-09-08T03:54:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:29.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:29.724789+0000) 2022-09-08T03:54:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:30.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:30.725036+0000) 2022-09-08T03:54:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:30.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:30.747588+0000) 2022-09-08T03:54:31.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:31 smithi059 ceph-mon[97450]: pgmap v12165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:54:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:31 smithi055 ceph-mon[97033]: pgmap v12165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:54:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:31.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:31.747823+0000) 2022-09-08T03:54:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:31.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:31.725294+0000) 2022-09-08T03:54:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:32.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:32.748057+0000) 2022-09-08T03:54:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:32.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:32.725466+0000) 2022-09-08T03:54:33.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:33 smithi059 ceph-mon[97450]: pgmap v12166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:54:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:33 smithi055 ceph-mon[97033]: pgmap v12166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:54:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:33.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:33.725732+0000) 2022-09-08T03:54:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:33.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:33.748329+0000) 2022-09-08T03:54:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:34.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:34.748575+0000) 2022-09-08T03:54:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:34.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:34.725922+0000) 2022-09-08T03:54:35.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:35 smithi059 ceph-mon[97450]: pgmap v12167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:54:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:35 smithi055 ceph-mon[97033]: pgmap v12167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:54:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:35.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:35.748806+0000) 2022-09-08T03:54:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:35.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:35.726170+0000) 2022-09-08T03:54:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:36.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:36.748947+0000) 2022-09-08T03:54:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:36.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:36.726394+0000) 2022-09-08T03:54:37.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:37 smithi059 ceph-mon[97450]: pgmap v12168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:54:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:37 smithi055 ceph-mon[97033]: pgmap v12168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:54:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:37.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:37.726616+0000) 2022-09-08T03:54:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:37.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:37.749103+0000) 2022-09-08T03:54:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:38.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:38.726827+0000) 2022-09-08T03:54:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:38.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:38.749316+0000) 2022-09-08T03:54:39.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:39 smithi059 ceph-mon[97450]: pgmap v12169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:39 smithi055 ceph-mon[97033]: pgmap v12169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:39.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:39.749479+0000) 2022-09-08T03:54:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:39.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:39.726954+0000) 2022-09-08T03:54:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:40.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:40.727150+0000) 2022-09-08T03:54:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:40.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:40.749728+0000) 2022-09-08T03:54:41.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:41 smithi059 ceph-mon[97450]: pgmap v12170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:41 smithi055 ceph-mon[97033]: pgmap v12170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:41.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:41.727362+0000) 2022-09-08T03:54:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:41.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:41.749962+0000) 2022-09-08T03:54:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:42.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:42.727603+0000) 2022-09-08T03:54:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:42.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:42.750243+0000) 2022-09-08T03:54:43.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:43 smithi059 ceph-mon[97450]: pgmap v12171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:54:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:43 smithi055 ceph-mon[97033]: pgmap v12171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:54:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:43.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:43.727808+0000) 2022-09-08T03:54:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:43.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:43.750491+0000) 2022-09-08T03:54:44.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:54:44.331+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:54:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:44.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:44.750727+0000) 2022-09-08T03:54:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:44.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:44.727960+0000) 2022-09-08T03:54:45.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:45 smithi059 ceph-mon[97450]: pgmap v12172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:45 smithi055 ceph-mon[97033]: pgmap v12172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:45.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:45.728094+0000) 2022-09-08T03:54:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:45.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:45.750958+0000) 2022-09-08T03:54:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:46.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:46.728252+0000) 2022-09-08T03:54:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:46.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:46.751147+0000) 2022-09-08T03:54:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:47 smithi059 ceph-mon[97450]: pgmap v12173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:54:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:47 smithi055 ceph-mon[97033]: pgmap v12173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:54:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:47.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:47.728476+0000) 2022-09-08T03:54:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:47.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:47.751378+0000) 2022-09-08T03:54:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:48.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:48.728761+0000) 2022-09-08T03:54:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:48.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:48.751547+0000) 2022-09-08T03:54:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:49 smithi055 ceph-mon[97033]: pgmap v12174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:50.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:49 smithi059 ceph-mon[97450]: pgmap v12174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:50.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:49.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:49.728937+0000) 2022-09-08T03:54:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:49.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:49.751790+0000) 2022-09-08T03:54:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:50.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:50.751930+0000) 2022-09-08T03:54:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:50.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:50.729186+0000) 2022-09-08T03:54:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:51 smithi055 ceph-mon[97033]: pgmap v12175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:52.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:51 smithi059 ceph-mon[97450]: pgmap v12175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:51.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:51.752126+0000) 2022-09-08T03:54:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:51.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:51.729352+0000) 2022-09-08T03:54:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:52.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:52.752352+0000) 2022-09-08T03:54:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:52.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:52.729584+0000) 2022-09-08T03:54:53.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:53 smithi055 ceph-mon[97033]: pgmap v12176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:54:54.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:53 smithi059 ceph-mon[97450]: pgmap v12176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:54:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:53.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:53.752576+0000) 2022-09-08T03:54:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:53.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:53.729800+0000) 2022-09-08T03:54:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:54.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:54.752777+0000) 2022-09-08T03:54:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:54.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:54.730029+0000) 2022-09-08T03:54:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:55 smithi055 ceph-mon[97033]: pgmap v12177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:56.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:55 smithi059 ceph-mon[97450]: pgmap v12177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:54:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:55.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:55.753040+0000) 2022-09-08T03:54:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:55.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:55.730243+0000) 2022-09-08T03:54:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:56.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:56.753299+0000) 2022-09-08T03:54:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:56.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:56.730471+0000) 2022-09-08T03:54:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:57 smithi055 ceph-mon[97033]: pgmap v12178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:54:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:57 smithi059 ceph-mon[97450]: pgmap v12178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:54:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:57.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:57.730748+0000) 2022-09-08T03:54:58.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:57.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:57.753561+0000) 2022-09-08T03:54:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:58.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:58.753813+0000) 2022-09-08T03:54:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:58.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:58.730921+0000) 2022-09-08T03:54:59.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:54:59.333+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:54:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:54:59 smithi055 ceph-mon[97033]: pgmap v12179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:00.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:54:59 smithi059 ceph-mon[97450]: pgmap v12179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:54:59.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:59.753982+0000) 2022-09-08T03:55:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:54:59.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:54:59.731053+0000) 2022-09-08T03:55:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:00.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:00.754156+0000) 2022-09-08T03:55:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:00.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:00.731227+0000) 2022-09-08T03:55:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:01 smithi055 ceph-mon[97033]: pgmap v12180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:02.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:01 smithi059 ceph-mon[97450]: pgmap v12180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:01.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:01.754381+0000) 2022-09-08T03:55:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:01.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:01.731476+0000) 2022-09-08T03:55:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:02.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:02.754624+0000) 2022-09-08T03:55:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:02.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:02.731768+0000) 2022-09-08T03:55:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:03 smithi055 ceph-mon[97033]: pgmap v12181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:55:04.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:03 smithi059 ceph-mon[97450]: pgmap v12181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:55:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:03.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:03.731967+0000) 2022-09-08T03:55:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:03.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:03.754861+0000) 2022-09-08T03:55:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:04.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:04.732156+0000) 2022-09-08T03:55:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:04.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:04.754991+0000) 2022-09-08T03:55:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:05 smithi055 ceph-mon[97033]: pgmap v12182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:55:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:05 smithi059 ceph-mon[97450]: pgmap v12182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:55:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:05.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:05.755198+0000) 2022-09-08T03:55:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:05.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:05.732403+0000) 2022-09-08T03:55:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:55:07.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:55:07.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:06.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:06.755373+0000) 2022-09-08T03:55:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:06.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:06.732697+0000) 2022-09-08T03:55:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:07 smithi055 ceph-mon[97033]: pgmap v12183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:55:08.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:07 smithi059 ceph-mon[97450]: pgmap v12183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:55:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:07.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:07.755578+0000) 2022-09-08T03:55:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:07.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:07.732939+0000) 2022-09-08T03:55:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:08.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:08.755745+0000) 2022-09-08T03:55:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:08.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:08.733219+0000) 2022-09-08T03:55:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:09 smithi055 ceph-mon[97033]: pgmap v12184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:55:10.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:09 smithi059 ceph-mon[97450]: pgmap v12184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:55:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:09.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:09.755947+0000) 2022-09-08T03:55:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:09.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:09.733520+0000) 2022-09-08T03:55:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:10 smithi055 ceph-mon[97033]: osdmap e840: 12 total, 11 up, 11 in 2022-09-08T03:55:11.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:10 smithi059 ceph-mon[97450]: osdmap e840: 12 total, 11 up, 11 in 2022-09-08T03:55:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:10.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:10.756089+0000) 2022-09-08T03:55:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:10.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:10.733763+0000) 2022-09-08T03:55:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:11 smithi055 ceph-mon[97033]: pgmap v12186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:55:12.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:11 smithi059 ceph-mon[97450]: pgmap v12186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:55:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:11.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:11.756292+0000) 2022-09-08T03:55:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:11.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:11.733956+0000) 2022-09-08T03:55:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:12.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:12.734155+0000) 2022-09-08T03:55:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:12.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:12.756524+0000) 2022-09-08T03:55:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:13 smithi055 ceph-mon[97033]: pgmap v12187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:55:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:13 smithi059 ceph-mon[97450]: pgmap v12187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:55:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:13.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:13.756743+0000) 2022-09-08T03:55:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:13.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:13.734404+0000) 2022-09-08T03:55:14.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:55:14.339+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:55:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:14.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:14.756878+0000) 2022-09-08T03:55:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:14.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:14.734595+0000) 2022-09-08T03:55:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:15 smithi055 ceph-mon[97033]: pgmap v12188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:55:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:15 smithi059 ceph-mon[97450]: pgmap v12188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:55:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:15.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:15.757095+0000) 2022-09-08T03:55:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:15.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:15.734846+0000) 2022-09-08T03:55:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:16.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:16.757253+0000) 2022-09-08T03:55:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:16.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:16.735073+0000) 2022-09-08T03:55:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:17 smithi055 ceph-mon[97033]: pgmap v12189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:55:18.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:17 smithi059 ceph-mon[97450]: pgmap v12189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:55:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:17.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:17.757460+0000) 2022-09-08T03:55:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:17.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:17.735323+0000) 2022-09-08T03:55:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:18.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:18.757628+0000) 2022-09-08T03:55:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:18.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:18.735552+0000) 2022-09-08T03:55:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:19 smithi055 ceph-mon[97033]: pgmap v12190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:55:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:19 smithi055 ceph-mon[97033]: osdmap e841: 12 total, 11 up, 11 in 2022-09-08T03:55:20.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:19 smithi059 ceph-mon[97450]: pgmap v12190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:55:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:19 smithi059 ceph-mon[97450]: osdmap e841: 12 total, 11 up, 11 in 2022-09-08T03:55:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:19.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:19.757890+0000) 2022-09-08T03:55:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:19.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:19.735771+0000) 2022-09-08T03:55:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:20.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:20.758062+0000) 2022-09-08T03:55:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:20.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:20.736008+0000) 2022-09-08T03:55:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:21 smithi055 ceph-mon[97033]: pgmap v12192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:55:22.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:21 smithi059 ceph-mon[97450]: pgmap v12192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T03:55:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:21.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:21.758224+0000) 2022-09-08T03:55:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:21.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:21.736277+0000) 2022-09-08T03:55:23.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:22.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:22.758466+0000) 2022-09-08T03:55:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:22.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:22.736516+0000) 2022-09-08T03:55:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:23 smithi055 ceph-mon[97033]: pgmap v12193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:55:24.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:23 smithi059 ceph-mon[97450]: pgmap v12193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:55:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:23.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:23.758721+0000) 2022-09-08T03:55:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:23.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:23.736749+0000) 2022-09-08T03:55:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:24.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:24.758982+0000) 2022-09-08T03:55:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:24.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:24.736948+0000) 2022-09-08T03:55:25.901 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:25 smithi055 ceph-mon[97033]: pgmap v12194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:55:25.903 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:25 smithi059 ceph-mon[97450]: pgmap v12194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:55:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:25.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:25.759178+0000) 2022-09-08T03:55:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:25.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:25.737210+0000) 2022-09-08T03:55:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:55:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:55:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:55:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:55:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:55:27.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:55:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:55:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:55:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:55:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:55:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:26.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:26.759330+0000) 2022-09-08T03:55:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:26.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:26.737426+0000) 2022-09-08T03:55:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:27 smithi055 ceph-mon[97033]: pgmap v12195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:55:28.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:27 smithi059 ceph-mon[97450]: pgmap v12195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:55:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:27.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:27.759551+0000) 2022-09-08T03:55:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:27.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:27.737716+0000) 2022-09-08T03:55:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:28.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:28.759802+0000) 2022-09-08T03:55:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:28.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:28.737962+0000) 2022-09-08T03:55:29.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:55:29.354+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:55:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:29 smithi055 ceph-mon[97033]: pgmap v12196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:55:30.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:29 smithi059 ceph-mon[97450]: pgmap v12196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:55:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:29.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:29.759950+0000) 2022-09-08T03:55:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:29.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:29.738192+0000) 2022-09-08T03:55:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:30.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:30.738443+0000) 2022-09-08T03:55:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:30.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:30.760123+0000) 2022-09-08T03:55:31.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:31 smithi055 ceph-mon[97033]: pgmap v12197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:55:32.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:31 smithi059 ceph-mon[97450]: pgmap v12197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:55:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:31.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:31.760431+0000) 2022-09-08T03:55:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:31.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:31.738731+0000) 2022-09-08T03:55:33.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:32.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:32.760661+0000) 2022-09-08T03:55:33.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:32.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:32.738953+0000) 2022-09-08T03:55:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:33 smithi055 ceph-mon[97033]: pgmap v12198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:55:34.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:33 smithi059 ceph-mon[97450]: pgmap v12198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:55:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:33.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:33.760902+0000) 2022-09-08T03:55:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:33.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:33.739154+0000) 2022-09-08T03:55:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:34.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:34.761107+0000) 2022-09-08T03:55:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:34.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:34.739377+0000) 2022-09-08T03:55:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:35 smithi055 ceph-mon[97033]: pgmap v12199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:35 smithi059 ceph-mon[97450]: pgmap v12199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:35.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:35.761349+0000) 2022-09-08T03:55:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:35.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:35.739670+0000) 2022-09-08T03:55:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:36.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:36.739937+0000) 2022-09-08T03:55:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:36.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:36.761591+0000) 2022-09-08T03:55:37.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:37 smithi055 ceph-mon[97033]: pgmap v12200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:55:38.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:37 smithi059 ceph-mon[97450]: pgmap v12200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:55:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:37.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:37.761826+0000) 2022-09-08T03:55:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:37.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:37.740207+0000) 2022-09-08T03:55:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:38.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:38.762007+0000) 2022-09-08T03:55:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:38.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:38.740450+0000) 2022-09-08T03:55:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:39 smithi055 ceph-mon[97033]: pgmap v12201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:40.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:39 smithi059 ceph-mon[97450]: pgmap v12201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:40.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:39.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:39.762224+0000) 2022-09-08T03:55:40.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:39.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:39.740738+0000) 2022-09-08T03:55:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:40.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:40.740960+0000) 2022-09-08T03:55:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:40.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:40.762462+0000) 2022-09-08T03:55:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:41 smithi055 ceph-mon[97033]: pgmap v12202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:42.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:41 smithi059 ceph-mon[97450]: pgmap v12202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:41.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:41.741136+0000) 2022-09-08T03:55:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:41.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:41.762733+0000) 2022-09-08T03:55:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:42.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:42.741348+0000) 2022-09-08T03:55:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:42.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:42.762880+0000) 2022-09-08T03:55:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:43 smithi055 ceph-mon[97033]: pgmap v12203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:55:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:43 smithi059 ceph-mon[97450]: pgmap v12203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:55:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:43.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:43.741583+0000) 2022-09-08T03:55:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:43.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:43.763096+0000) 2022-09-08T03:55:44.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:55:44.360+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:55:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:44.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:44.763342+0000) 2022-09-08T03:55:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:44.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:44.741830+0000) 2022-09-08T03:55:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:45 smithi055 ceph-mon[97033]: pgmap v12204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:46.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:45 smithi059 ceph-mon[97450]: pgmap v12204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:45.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:45.763546+0000) 2022-09-08T03:55:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:45.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:45.742017+0000) 2022-09-08T03:55:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:46.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:46.763752+0000) 2022-09-08T03:55:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:46.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:46.742244+0000) 2022-09-08T03:55:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:47 smithi055 ceph-mon[97033]: pgmap v12205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:55:48.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:47 smithi059 ceph-mon[97450]: pgmap v12205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:55:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:47.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:47.742500+0000) 2022-09-08T03:55:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:47.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:47.763957+0000) 2022-09-08T03:55:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:48.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:48.764178+0000) 2022-09-08T03:55:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:48.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:48.742748+0000) 2022-09-08T03:55:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:49 smithi055 ceph-mon[97033]: pgmap v12206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:50.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:49 smithi059 ceph-mon[97450]: pgmap v12206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:49.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:49.742931+0000) 2022-09-08T03:55:50.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:49.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:49.764388+0000) 2022-09-08T03:55:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:50.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:50.764617+0000) 2022-09-08T03:55:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:50.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:50.743151+0000) 2022-09-08T03:55:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:51 smithi055 ceph-mon[97033]: pgmap v12207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:51 smithi059 ceph-mon[97450]: pgmap v12207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:51.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:51.743402+0000) 2022-09-08T03:55:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:51.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:51.764802+0000) 2022-09-08T03:55:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:52 smithi055 ceph-mon[97033]: pgmap v12208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:55:53.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:52 smithi059 ceph-mon[97450]: pgmap v12208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:55:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:52.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:52.743675+0000) 2022-09-08T03:55:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:52.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:52.765084+0000) 2022-09-08T03:55:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:53.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:53.765308+0000) 2022-09-08T03:55:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:53.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:53.743854+0000) 2022-09-08T03:55:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:54.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:54.765517+0000) 2022-09-08T03:55:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:54.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:54.743991+0000) 2022-09-08T03:55:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:55 smithi055 ceph-mon[97033]: pgmap v12209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:55 smithi059 ceph-mon[97450]: pgmap v12209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:55:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:55.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:55.765692+0000) 2022-09-08T03:55:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:55.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:55.744176+0000) 2022-09-08T03:55:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:56.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:56.765873+0000) 2022-09-08T03:55:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:56.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:56.744401+0000) 2022-09-08T03:55:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:56 smithi055 ceph-mon[97033]: pgmap v12210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:55:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:56 smithi055 ceph-mon[97033]: pgmap v12211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:55:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:56 smithi055 ceph-mon[97033]: pgmap v12212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:55:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:55:57.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:56 smithi059 ceph-mon[97450]: pgmap v12210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:55:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:56 smithi059 ceph-mon[97450]: pgmap v12211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:55:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:56 smithi059 ceph-mon[97450]: pgmap v12212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:55:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:55:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:57.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:57.766085+0000) 2022-09-08T03:55:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:57.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:57.744671+0000) 2022-09-08T03:55:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:58.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:58.766333+0000) 2022-09-08T03:55:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:58.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:58.744896+0000) 2022-09-08T03:55:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:55:58 smithi055 ceph-mon[97033]: pgmap v12213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:55:59.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:55:58 smithi059 ceph-mon[97450]: pgmap v12213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:55:59.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:55:59.369+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:56:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:55:59.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:59.745100+0000) 2022-09-08T03:56:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:55:59.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:55:59.766571+0000) 2022-09-08T03:56:01.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:00.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:00.745269+0000) 2022-09-08T03:56:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:00.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:00.766800+0000) 2022-09-08T03:56:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:01 smithi055 ceph-mon[97033]: pgmap v12214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-08T03:56:01.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:01 smithi059 ceph-mon[97450]: pgmap v12214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-08T03:56:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:01.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:01.767051+0000) 2022-09-08T03:56:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:01.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:01.745504+0000) 2022-09-08T03:56:03.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:02.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:02.745647+0000) 2022-09-08T03:56:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:02.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:02.767309+0000) 2022-09-08T03:56:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:03 smithi055 ceph-mon[97033]: pgmap v12215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:56:03.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:03 smithi059 ceph-mon[97450]: pgmap v12215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T03:56:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:03.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:03.767565+0000) 2022-09-08T03:56:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:03.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:03.745783+0000) 2022-09-08T03:56:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:04.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:04.767815+0000) 2022-09-08T03:56:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:04.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:04.745992+0000) 2022-09-08T03:56:05.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:05 smithi055 ceph-mon[97033]: pgmap v12216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:56:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:05 smithi059 ceph-mon[97450]: pgmap v12216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T03:56:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:05.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:05.746165+0000) 2022-09-08T03:56:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:05.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:05.768040+0000) 2022-09-08T03:56:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:56:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:56:06.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:56:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:56:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:06.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:06.768257+0000) 2022-09-08T03:56:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:06.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:06.746318+0000) 2022-09-08T03:56:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:07 smithi055 ceph-mon[97033]: pgmap v12217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:56:07.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:07 smithi059 ceph-mon[97450]: pgmap v12217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:56:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:07.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:07.768447+0000) 2022-09-08T03:56:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:07.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:07.746495+0000) 2022-09-08T03:56:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:08.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:08.768583+0000) 2022-09-08T03:56:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:08.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:08.746697+0000) 2022-09-08T03:56:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:09 smithi055 ceph-mon[97033]: pgmap v12218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:56:09.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:09 smithi059 ceph-mon[97450]: pgmap v12218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:56:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:09.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:09.768771+0000) 2022-09-08T03:56:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:09.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:09.746868+0000) 2022-09-08T03:56:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:10 smithi055 ceph-mon[97033]: osdmap e842: 12 total, 11 up, 11 in 2022-09-08T03:56:10.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:10 smithi059 ceph-mon[97450]: osdmap e842: 12 total, 11 up, 11 in 2022-09-08T03:56:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:10.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:10.768971+0000) 2022-09-08T03:56:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:10.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:10.747047+0000) 2022-09-08T03:56:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:11 smithi055 ceph-mon[97033]: pgmap v12220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:56:11.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:11 smithi059 ceph-mon[97450]: pgmap v12220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:56:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:11.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:11.769217+0000) 2022-09-08T03:56:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:11.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:11.747202+0000) 2022-09-08T03:56:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:12.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:12.769461+0000) 2022-09-08T03:56:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:12.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:12.747453+0000) 2022-09-08T03:56:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:13 smithi055 ceph-mon[97033]: pgmap v12221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:56:13.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:13 smithi059 ceph-mon[97450]: pgmap v12221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:56:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:13.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:13.769726+0000) 2022-09-08T03:56:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:13.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:13.747733+0000) 2022-09-08T03:56:14.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:56:14.380+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:56:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:14.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:14.769952+0000) 2022-09-08T03:56:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:14.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:14.747864+0000) 2022-09-08T03:56:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:15 smithi055 ceph-mon[97033]: pgmap v12222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:56:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:15 smithi055 ceph-mon[97033]: osdmap e843: 12 total, 11 up, 11 in 2022-09-08T03:56:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:15 smithi059 ceph-mon[97450]: pgmap v12222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T03:56:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:15 smithi059 ceph-mon[97450]: osdmap e843: 12 total, 11 up, 11 in 2022-09-08T03:56:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:15.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:15.748085+0000) 2022-09-08T03:56:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:15.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:15.770182+0000) 2022-09-08T03:56:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:16.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:16.770442+0000) 2022-09-08T03:56:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:16.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:16.748284+0000) 2022-09-08T03:56:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:17 smithi055 ceph-mon[97033]: pgmap v12224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:56:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:17 smithi059 ceph-mon[97450]: pgmap v12224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:56:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:17.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:17.770573+0000) 2022-09-08T03:56:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:17.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:17.748482+0000) 2022-09-08T03:56:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:18.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:18.770814+0000) 2022-09-08T03:56:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:18.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:18.748686+0000) 2022-09-08T03:56:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:19 smithi055 ceph-mon[97033]: pgmap v12225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-08T03:56:19.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:19 smithi059 ceph-mon[97450]: pgmap v12225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-08T03:56:20.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:19.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:19.770988+0000) 2022-09-08T03:56:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:19.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:19.748850+0000) 2022-09-08T03:56:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:20.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:20.771237+0000) 2022-09-08T03:56:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:20.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:20.749094+0000) 2022-09-08T03:56:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:21 smithi055 ceph-mon[97033]: pgmap v12226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:56:21.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:21 smithi059 ceph-mon[97450]: pgmap v12226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:56:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:21.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:21.771479+0000) 2022-09-08T03:56:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:21.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:21.749303+0000) 2022-09-08T03:56:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:22.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:22.771762+0000) 2022-09-08T03:56:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:22.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:22.749509+0000) 2022-09-08T03:56:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:23 smithi055 ceph-mon[97033]: pgmap v12227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:56:23.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:23 smithi059 ceph-mon[97450]: pgmap v12227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:56:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:23.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:23.771996+0000) 2022-09-08T03:56:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:23.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:23.749700+0000) 2022-09-08T03:56:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:24.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:24.772188+0000) 2022-09-08T03:56:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:24.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:24.749917+0000) 2022-09-08T03:56:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:25 smithi055 ceph-mon[97033]: pgmap v12228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:56:25.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:25 smithi059 ceph-mon[97450]: pgmap v12228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:56:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:25.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:25.772418+0000) 2022-09-08T03:56:26.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:25.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:25.750179+0000) 2022-09-08T03:56:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:26.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:26.772648+0000) 2022-09-08T03:56:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:26.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:26.750418+0000) 2022-09-08T03:56:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:27 smithi055 ceph-mon[97033]: pgmap v12229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T03:56:27.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:27 smithi059 ceph-mon[97450]: pgmap v12229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T03:56:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:27.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:27.772875+0000) 2022-09-08T03:56:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:27.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:27.750726+0000) 2022-09-08T03:56:29.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:28.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:28.773108+0000) 2022-09-08T03:56:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:28.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:28.750933+0000) 2022-09-08T03:56:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:29 smithi055 ceph-mon[97033]: pgmap v12230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:29.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:29 smithi059 ceph-mon[97450]: pgmap v12230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:29.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:56:29.387+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:56:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:29.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:29.751059+0000) 2022-09-08T03:56:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:29.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:29.773363+0000) 2022-09-08T03:56:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:30.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:30.773576+0000) 2022-09-08T03:56:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:30.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:30.751285+0000) 2022-09-08T03:56:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:31 smithi055 ceph-mon[97033]: pgmap v12231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:56:31.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:31 smithi059 ceph-mon[97450]: pgmap v12231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:56:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:31.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:31.751541+0000) 2022-09-08T03:56:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:31.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:31.773728+0000) 2022-09-08T03:56:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:32.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:32.773937+0000) 2022-09-08T03:56:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:32.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:32.751747+0000) 2022-09-08T03:56:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:33 smithi055 ceph-mon[97033]: pgmap v12232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:33.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:33 smithi059 ceph-mon[97450]: pgmap v12232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:33.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:33.774196+0000) 2022-09-08T03:56:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:33.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:33.751883+0000) 2022-09-08T03:56:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:34.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:34.752002+0000) 2022-09-08T03:56:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:34.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:34.774398+0000) 2022-09-08T03:56:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:35 smithi055 ceph-mon[97033]: pgmap v12233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:35.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:35 smithi059 ceph-mon[97450]: pgmap v12233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:35.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:35.774703+0000) 2022-09-08T03:56:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:35.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:35.752168+0000) 2022-09-08T03:56:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:36.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:36.774970+0000) 2022-09-08T03:56:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:36.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:36.752366+0000) 2022-09-08T03:56:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:37 smithi055 ceph-mon[97033]: pgmap v12234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:56:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:37 smithi059 ceph-mon[97450]: pgmap v12234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:56:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:37.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:37.775185+0000) 2022-09-08T03:56:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:37.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:37.752534+0000) 2022-09-08T03:56:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:38.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:38.775391+0000) 2022-09-08T03:56:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:38.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:38.752683+0000) 2022-09-08T03:56:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:39 smithi055 ceph-mon[97033]: pgmap v12235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:39.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:39 smithi059 ceph-mon[97450]: pgmap v12235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:39.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:39.775680+0000) 2022-09-08T03:56:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:39.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:39.752859+0000) 2022-09-08T03:56:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:40.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:40.775906+0000) 2022-09-08T03:56:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:40.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:40.753096+0000) 2022-09-08T03:56:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:41 smithi055 ceph-mon[97033]: pgmap v12236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:56:41.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:41 smithi059 ceph-mon[97450]: pgmap v12236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:56:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:41.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:41.776149+0000) 2022-09-08T03:56:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:41.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:41.753314+0000) 2022-09-08T03:56:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:42.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:42.776378+0000) 2022-09-08T03:56:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:42.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:42.753522+0000) 2022-09-08T03:56:43.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:43 smithi055 ceph-mon[97033]: pgmap v12237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:43.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:43 smithi059 ceph-mon[97450]: pgmap v12237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:43.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:43.776600+0000) 2022-09-08T03:56:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:43.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:43.753787+0000) 2022-09-08T03:56:44.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:56:44.394+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:56:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:44.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:44.776900+0000) 2022-09-08T03:56:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:44.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:44.753957+0000) 2022-09-08T03:56:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:45 smithi055 ceph-mon[97033]: pgmap v12238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:45.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:45 smithi059 ceph-mon[97450]: pgmap v12238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:46.160 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:45.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:45.777124+0000) 2022-09-08T03:56:46.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:45.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:45.754132+0000) 2022-09-08T03:56:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:46.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:46.777358+0000) 2022-09-08T03:56:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:46.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:46.754322+0000) 2022-09-08T03:56:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:47 smithi055 ceph-mon[97033]: pgmap v12239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:56:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:47 smithi059 ceph-mon[97450]: pgmap v12239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:56:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:47.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:47.777536+0000) 2022-09-08T03:56:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:47.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:47.754559+0000) 2022-09-08T03:56:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:48.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:48.777715+0000) 2022-09-08T03:56:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:48.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:48.754785+0000) 2022-09-08T03:56:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:49 smithi055 ceph-mon[97033]: pgmap v12240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:49.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:49 smithi059 ceph-mon[97450]: pgmap v12240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:49.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:49.777959+0000) 2022-09-08T03:56:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:49.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:49.754929+0000) 2022-09-08T03:56:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:50.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:50.778221+0000) 2022-09-08T03:56:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:50.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:50.755159+0000) 2022-09-08T03:56:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:51 smithi055 ceph-mon[97033]: pgmap v12241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:56:51.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:51 smithi059 ceph-mon[97450]: pgmap v12241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:56:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:51.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:51.755437+0000) 2022-09-08T03:56:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:51.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:51.778461+0000) 2022-09-08T03:56:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:52.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:52.778754+0000) 2022-09-08T03:56:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:52.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:52.755615+0000) 2022-09-08T03:56:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:53 smithi055 ceph-mon[97033]: pgmap v12242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:53.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:53 smithi059 ceph-mon[97450]: pgmap v12242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:53.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:53.778914+0000) 2022-09-08T03:56:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:53.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:53.755873+0000) 2022-09-08T03:56:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:54.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:54.779139+0000) 2022-09-08T03:56:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:54.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:54.756043+0000) 2022-09-08T03:56:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:55 smithi055 ceph-mon[97033]: pgmap v12243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:55.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:55 smithi059 ceph-mon[97450]: pgmap v12243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:56.104 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:55.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:55.779294+0000) 2022-09-08T03:56:56.105 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:55.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:55.756284+0000) 2022-09-08T03:56:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:56:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:56:56.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:56:56.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:56:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:56:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:56:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:56.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:56.779469+0000) 2022-09-08T03:56:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:56.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:56.756514+0000) 2022-09-08T03:56:57.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:57 smithi055 ceph-mon[97033]: pgmap v12244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:56:57.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:57 smithi059 ceph-mon[97450]: pgmap v12244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:56:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:57.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:57.779720+0000) 2022-09-08T03:56:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:57.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:57.756776+0000) 2022-09-08T03:56:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:58.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:58.779957+0000) 2022-09-08T03:56:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:58.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:58.757021+0000) 2022-09-08T03:56:59.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:56:59 smithi059 ceph-mon[97450]: pgmap v12245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:56:59 smithi055 ceph-mon[97033]: pgmap v12245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:56:59.757 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:56:59.406+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:57:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:56:59.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:59.780179+0000) 2022-09-08T03:57:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:56:59.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:56:59.757201+0000) 2022-09-08T03:57:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:00.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:00.780359+0000) 2022-09-08T03:57:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:00.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:00.757455+0000) 2022-09-08T03:57:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:01 smithi059 ceph-mon[97450]: pgmap v12246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:57:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:01 smithi055 ceph-mon[97033]: pgmap v12246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:57:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:01.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:01.780591+0000) 2022-09-08T03:57:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:01.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:01.757705+0000) 2022-09-08T03:57:03.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:02.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:02.780917+0000) 2022-09-08T03:57:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:02.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:02.757935+0000) 2022-09-08T03:57:03.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:03 smithi059 ceph-mon[97450]: pgmap v12247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:57:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:03 smithi055 ceph-mon[97033]: pgmap v12247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:57:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:03.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:03.781176+0000) 2022-09-08T03:57:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:03.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:03.758118+0000) 2022-09-08T03:57:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:04.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:04.781459+0000) 2022-09-08T03:57:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:04.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:04.758265+0000) 2022-09-08T03:57:05.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:05 smithi059 ceph-mon[97450]: pgmap v12248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:57:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:05 smithi055 ceph-mon[97033]: pgmap v12248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:57:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:05.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:05.781702+0000) 2022-09-08T03:57:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:05.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:05.758511+0000) 2022-09-08T03:57:06.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:57:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:57:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:57:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:57:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:06.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:06.758740+0000) 2022-09-08T03:57:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:06.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:06.781916+0000) 2022-09-08T03:57:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:07 smithi059 ceph-mon[97450]: pgmap v12249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:57:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:07 smithi055 ceph-mon[97033]: pgmap v12249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:57:08.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:07.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:07.758879+0000) 2022-09-08T03:57:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:07.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:07.782195+0000) 2022-09-08T03:57:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:08.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:08.782440+0000) 2022-09-08T03:57:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:08.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:08.759101+0000) 2022-09-08T03:57:09.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:09 smithi059 ceph-mon[97450]: pgmap v12250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:57:09.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:09 smithi055 ceph-mon[97033]: pgmap v12250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:57:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:09.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:09.782690+0000) 2022-09-08T03:57:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:09.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:09.759308+0000) 2022-09-08T03:57:10.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:10 smithi059 ceph-mon[97450]: osdmap e844: 12 total, 11 up, 11 in 2022-09-08T03:57:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:10 smithi055 ceph-mon[97033]: osdmap e844: 12 total, 11 up, 11 in 2022-09-08T03:57:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:10.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:10.782876+0000) 2022-09-08T03:57:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:10.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:10.759441+0000) 2022-09-08T03:57:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:11 smithi059 ceph-mon[97450]: pgmap v12252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:57:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:11 smithi055 ceph-mon[97033]: pgmap v12252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:57:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:11.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:11.759654+0000) 2022-09-08T03:57:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:11.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:11.783024+0000) 2022-09-08T03:57:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:12.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:12.783251+0000) 2022-09-08T03:57:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:12.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:12.759899+0000) 2022-09-08T03:57:13.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:13 smithi059 ceph-mon[97450]: pgmap v12253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:57:13.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:13 smithi055 ceph-mon[97033]: pgmap v12253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:57:14.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:13.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:13.783473+0000) 2022-09-08T03:57:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:13.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:13.760122+0000) 2022-09-08T03:57:14.760 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:57:14.422+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:57:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:14.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:14.783664+0000) 2022-09-08T03:57:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:14.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:14.760309+0000) 2022-09-08T03:57:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:15 smithi055 ceph-mon[97033]: pgmap v12254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:57:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:15 smithi055 ceph-mon[97033]: osdmap e845: 12 total, 11 up, 11 in 2022-09-08T03:57:15.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:15 smithi059 ceph-mon[97450]: pgmap v12254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:57:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:15 smithi059 ceph-mon[97450]: osdmap e845: 12 total, 11 up, 11 in 2022-09-08T03:57:16.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:15.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:15.783806+0000) 2022-09-08T03:57:16.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:15.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:15.760579+0000) 2022-09-08T03:57:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:16.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:16.784013+0000) 2022-09-08T03:57:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:16.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:16.760754+0000) 2022-09-08T03:57:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:17 smithi055 ceph-mon[97033]: pgmap v12256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:57:17.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:17 smithi059 ceph-mon[97450]: pgmap v12256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:57:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:17.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:17.784226+0000) 2022-09-08T03:57:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:17.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:17.760970+0000) 2022-09-08T03:57:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:18.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:18.784433+0000) 2022-09-08T03:57:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:18.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:18.761212+0000) 2022-09-08T03:57:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:19 smithi055 ceph-mon[97033]: pgmap v12257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-08T03:57:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:19 smithi059 ceph-mon[97450]: pgmap v12257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-08T03:57:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:19.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:19.784593+0000) 2022-09-08T03:57:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:19.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:19.761424+0000) 2022-09-08T03:57:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:20.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:20.784804+0000) 2022-09-08T03:57:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:20.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:20.761725+0000) 2022-09-08T03:57:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:21 smithi055 ceph-mon[97033]: pgmap v12258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:57:21.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:21 smithi059 ceph-mon[97450]: pgmap v12258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:57:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:21.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:21.784998+0000) 2022-09-08T03:57:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:21.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:21.762871+0000) 2022-09-08T03:57:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:22.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:22.763022+0000) 2022-09-08T03:57:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:22.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:22.785178+0000) 2022-09-08T03:57:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:23 smithi055 ceph-mon[97033]: pgmap v12259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:57:23.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:23 smithi059 ceph-mon[97450]: pgmap v12259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:57:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:23.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:23.785361+0000) 2022-09-08T03:57:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:23.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:23.763260+0000) 2022-09-08T03:57:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:24.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:24.785609+0000) 2022-09-08T03:57:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:24.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:24.763495+0000) 2022-09-08T03:57:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:25 smithi055 ceph-mon[97033]: pgmap v12260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:57:25.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:25 smithi059 ceph-mon[97450]: pgmap v12260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:57:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:25.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:25.785782+0000) 2022-09-08T03:57:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:25.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:25.763687+0000) 2022-09-08T03:57:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:26.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:26.785952+0000) 2022-09-08T03:57:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:26.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:26.763911+0000) 2022-09-08T03:57:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:27 smithi055 ceph-mon[97033]: pgmap v12261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T03:57:27.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:27 smithi059 ceph-mon[97450]: pgmap v12261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T03:57:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:27.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:27.786196+0000) 2022-09-08T03:57:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:27.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:27.764113+0000) 2022-09-08T03:57:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:28.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:28.786362+0000) 2022-09-08T03:57:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:28.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:28.764308+0000) 2022-09-08T03:57:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:29 smithi055 ceph-mon[97033]: pgmap v12262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:57:29.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:29 smithi059 ceph-mon[97450]: pgmap v12262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:57:29.764 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:57:29.428+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:57:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:29.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:29.786579+0000) 2022-09-08T03:57:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:29.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:29.764514+0000) 2022-09-08T03:57:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:30.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:30.764684+0000) 2022-09-08T03:57:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:30.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:30.786752+0000) 2022-09-08T03:57:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:31 smithi055 ceph-mon[97033]: pgmap v12263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:57:31.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:57:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:57:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:57:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:31 smithi059 ceph-mon[97450]: pgmap v12263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:57:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:57:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:57:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:57:32.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:31.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:31.764815+0000) 2022-09-08T03:57:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:31.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:31.786870+0000) 2022-09-08T03:57:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:32 smithi055 ceph-mon[97033]: pgmap v12264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:57:32.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:32 smithi059 ceph-mon[97450]: pgmap v12264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:57:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:32.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:32.764946+0000) 2022-09-08T03:57:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:32.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:32.787047+0000) 2022-09-08T03:57:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:33.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:33.765172+0000) 2022-09-08T03:57:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:33.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:33.787237+0000) 2022-09-08T03:57:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:34 smithi055 ceph-mon[97033]: pgmap v12265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:57:34.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:34 smithi059 ceph-mon[97450]: pgmap v12265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:57:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:34.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:34.765336+0000) 2022-09-08T03:57:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:34.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:34.787479+0000) 2022-09-08T03:57:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:35.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:35.787719+0000) 2022-09-08T03:57:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:35.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:35.765523+0000) 2022-09-08T03:57:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:36 smithi055 ceph-mon[97033]: pgmap v12266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:57:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:36 smithi059 ceph-mon[97450]: pgmap v12266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:57:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:36.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:36.787889+0000) 2022-09-08T03:57:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:36.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:36.765762+0000) 2022-09-08T03:57:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:37.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:37.788073+0000) 2022-09-08T03:57:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:37.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:37.765932+0000) 2022-09-08T03:57:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:38 smithi055 ceph-mon[97033]: pgmap v12267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:57:38.539 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:38 smithi059 ceph-mon[97450]: pgmap v12267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:57:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:38.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:38.788258+0000) 2022-09-08T03:57:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:38.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:38.766159+0000) 2022-09-08T03:57:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:39.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:39.788456+0000) 2022-09-08T03:57:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:39.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:39.766345+0000) 2022-09-08T03:57:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:40 smithi055 ceph-mon[97033]: pgmap v12268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:57:40.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:40 smithi059 ceph-mon[97450]: pgmap v12268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:57:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:40.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:40.788650+0000) 2022-09-08T03:57:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:40.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:40.766572+0000) 2022-09-08T03:57:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:41.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:41.788781+0000) 2022-09-08T03:57:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:41.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:41.766743+0000) 2022-09-08T03:57:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:42 smithi055 ceph-mon[97033]: pgmap v12269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:57:42.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:42 smithi059 ceph-mon[97450]: pgmap v12269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:57:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:42.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:42.788959+0000) 2022-09-08T03:57:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:42.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:42.766962+0000) 2022-09-08T03:57:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:43.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:43.789169+0000) 2022-09-08T03:57:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:43.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:43.767128+0000) 2022-09-08T03:57:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:44 smithi055 ceph-mon[97033]: pgmap v12270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:57:44.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:44 smithi059 ceph-mon[97450]: pgmap v12270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:57:44.769 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:57:44.435+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:57:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:44.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:44.789305+0000) 2022-09-08T03:57:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:44.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:44.767258+0000) 2022-09-08T03:57:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:45.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:45.789564+0000) 2022-09-08T03:57:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:45.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:45.767464+0000) 2022-09-08T03:57:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:46 smithi055 ceph-mon[97033]: pgmap v12271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T03:57:46.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:46 smithi059 ceph-mon[97450]: pgmap v12271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T03:57:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:46.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:46.789782+0000) 2022-09-08T03:57:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:46.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:46.767746+0000) 2022-09-08T03:57:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:47.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:47.789979+0000) 2022-09-08T03:57:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:47.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:47.767927+0000) 2022-09-08T03:57:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:48 smithi055 ceph-mon[97033]: pgmap v12272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:57:48.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:48 smithi059 ceph-mon[97450]: pgmap v12272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:57:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:48.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:48.768186+0000) 2022-09-08T03:57:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:48.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:48.790179+0000) 2022-09-08T03:57:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:49.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:49.790415+0000) 2022-09-08T03:57:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:49.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:49.768367+0000) 2022-09-08T03:57:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:50 smithi055 ceph-mon[97033]: pgmap v12273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:57:50.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:50 smithi059 ceph-mon[97450]: pgmap v12273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:57:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:50.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:50.790718+0000) 2022-09-08T03:57:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:50.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:50.768576+0000) 2022-09-08T03:57:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:51.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:51.790894+0000) 2022-09-08T03:57:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:51.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:51.768826+0000) 2022-09-08T03:57:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:52 smithi055 ceph-mon[97033]: pgmap v12274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:57:52.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:52 smithi059 ceph-mon[97450]: pgmap v12274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:57:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:52.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:52.791105+0000) 2022-09-08T03:57:53.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:52.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:52.769045+0000) 2022-09-08T03:57:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:53.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:53.791317+0000) 2022-09-08T03:57:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:53.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:53.769311+0000) 2022-09-08T03:57:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:54 smithi059 ceph-mon[97450]: pgmap v12275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:57:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:54 smithi055 ceph-mon[97033]: pgmap v12275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:57:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:54.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:54.791532+0000) 2022-09-08T03:57:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:54.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:54.769517+0000) 2022-09-08T03:57:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:55.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:55.791768+0000) 2022-09-08T03:57:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:55.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:55.769794+0000) 2022-09-08T03:57:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:56 smithi059 ceph-mon[97450]: pgmap v12276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T03:57:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:56 smithi055 ceph-mon[97033]: pgmap v12276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T03:57:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:56.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:56.792014+0000) 2022-09-08T03:57:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:56.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:56.770089+0000) 2022-09-08T03:57:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:57.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:57.792225+0000) 2022-09-08T03:57:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:57.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:57.770369+0000) 2022-09-08T03:57:58.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:57:58 smithi059 ceph-mon[97450]: pgmap v12277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:57:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:57:58 smithi055 ceph-mon[97033]: pgmap v12277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:57:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:58.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:58.792348+0000) 2022-09-08T03:57:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:58.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:58.770631+0000) 2022-09-08T03:57:59.770 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:57:59.441+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:58:00.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:57:59.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:59.770940+0000) 2022-09-08T03:58:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:57:59.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:57:59.792586+0000) 2022-09-08T03:58:00.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:00 smithi059 ceph-mon[97450]: pgmap v12278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:00.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:00 smithi055 ceph-mon[97033]: pgmap v12278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:00.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:00.792691+0000) 2022-09-08T03:58:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:00.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:00.771105+0000) 2022-09-08T03:58:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:01.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:01.792815+0000) 2022-09-08T03:58:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:01.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:01.771252+0000) 2022-09-08T03:58:02.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:02 smithi059 ceph-mon[97450]: pgmap v12279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:58:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:02 smithi055 ceph-mon[97033]: pgmap v12279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:58:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:02.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:02.793001+0000) 2022-09-08T03:58:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:02.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:02.771455+0000) 2022-09-08T03:58:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:03.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:03.793221+0000) 2022-09-08T03:58:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:03.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:03.771720+0000) 2022-09-08T03:58:04.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:04 smithi059 ceph-mon[97450]: pgmap v12280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:04 smithi055 ceph-mon[97033]: pgmap v12280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:04.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:04.793469+0000) 2022-09-08T03:58:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:04.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:04.771866+0000) 2022-09-08T03:58:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:05.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:05.793728+0000) 2022-09-08T03:58:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:05.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:05.772111+0000) 2022-09-08T03:58:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:06 smithi059 ceph-mon[97450]: pgmap v12281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 18 op/s 2022-09-08T03:58:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:58:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:58:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:06 smithi055 ceph-mon[97033]: pgmap v12281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 18 op/s 2022-09-08T03:58:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:58:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:58:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:06.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:06.793951+0000) 2022-09-08T03:58:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:06.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:06.772287+0000) 2022-09-08T03:58:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:07.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:07.794156+0000) 2022-09-08T03:58:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:07.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:07.772512+0000) 2022-09-08T03:58:08.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:08 smithi059 ceph-mon[97450]: pgmap v12282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T03:58:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:08 smithi055 ceph-mon[97033]: pgmap v12282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T03:58:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:08.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:08.794339+0000) 2022-09-08T03:58:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:08.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:08.772706+0000) 2022-09-08T03:58:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:09.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:09.794510+0000) 2022-09-08T03:58:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:09.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:09.772888+0000) 2022-09-08T03:58:10.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:10 smithi059 ceph-mon[97450]: pgmap v12283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:58:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:10 smithi059 ceph-mon[97450]: osdmap e846: 12 total, 11 up, 11 in 2022-09-08T03:58:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:10 smithi055 ceph-mon[97033]: pgmap v12283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:58:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:10 smithi055 ceph-mon[97033]: osdmap e846: 12 total, 11 up, 11 in 2022-09-08T03:58:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:10.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:10.794736+0000) 2022-09-08T03:58:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:10.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:10.773112+0000) 2022-09-08T03:58:12.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:11.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:11.794930+0000) 2022-09-08T03:58:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:11.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:11.773350+0000) 2022-09-08T03:58:12.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:12 smithi059 ceph-mon[97450]: pgmap v12285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:58:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:12 smithi055 ceph-mon[97033]: pgmap v12285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:58:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:12.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:12.795223+0000) 2022-09-08T03:58:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:12.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:12.773577+0000) 2022-09-08T03:58:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:13.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:13.795499+0000) 2022-09-08T03:58:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:13.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:13.773777+0000) 2022-09-08T03:58:14.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:14 smithi059 ceph-mon[97450]: pgmap v12286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:58:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:14 smithi059 ceph-mon[97450]: osdmap e847: 12 total, 11 up, 11 in 2022-09-08T03:58:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:14 smithi055 ceph-mon[97033]: pgmap v12286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:58:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:14 smithi055 ceph-mon[97033]: osdmap e847: 12 total, 11 up, 11 in 2022-09-08T03:58:14.773 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:58:14.456+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:58:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:14.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:14.795784+0000) 2022-09-08T03:58:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:14.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:14.773953+0000) 2022-09-08T03:58:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:15.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:15.795945+0000) 2022-09-08T03:58:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:15.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:15.774178+0000) 2022-09-08T03:58:16.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:16 smithi059 ceph-mon[97450]: pgmap v12288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.1 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-08T03:58:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:16 smithi055 ceph-mon[97033]: pgmap v12288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.1 KiB/s rd, 255 B/s wr, 5 op/s 2022-09-08T03:58:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:16.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:16.796211+0000) 2022-09-08T03:58:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:16.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:16.774367+0000) 2022-09-08T03:58:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:17.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:17.796471+0000) 2022-09-08T03:58:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:17.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:17.774591+0000) 2022-09-08T03:58:18.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:18 smithi059 ceph-mon[97450]: pgmap v12289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:58:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:18 smithi055 ceph-mon[97033]: pgmap v12289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:58:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:18.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:18.796754+0000) 2022-09-08T03:58:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:18.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:18.774799+0000) 2022-09-08T03:58:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:19.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:19.796994+0000) 2022-09-08T03:58:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:19.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:19.775023+0000) 2022-09-08T03:58:20.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:20 smithi059 ceph-mon[97450]: pgmap v12290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2022-09-08T03:58:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:20 smithi055 ceph-mon[97033]: pgmap v12290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2022-09-08T03:58:21.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:20.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:20.797196+0000) 2022-09-08T03:58:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:20.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:20.775235+0000) 2022-09-08T03:58:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:21.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:21.775447+0000) 2022-09-08T03:58:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:21.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:21.797414+0000) 2022-09-08T03:58:22.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:22 smithi059 ceph-mon[97450]: pgmap v12291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:58:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:22 smithi055 ceph-mon[97033]: pgmap v12291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:58:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:22.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:22.797663+0000) 2022-09-08T03:58:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:22.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:22.775679+0000) 2022-09-08T03:58:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:23.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:23.775878+0000) 2022-09-08T03:58:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:23.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:23.797852+0000) 2022-09-08T03:58:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:24 smithi055 ceph-mon[97033]: pgmap v12292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:58:24.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:24 smithi059 ceph-mon[97450]: pgmap v12292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T03:58:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:24.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:24.798002+0000) 2022-09-08T03:58:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:24.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:24.776018+0000) 2022-09-08T03:58:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:25.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:25.798166+0000) 2022-09-08T03:58:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:25.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:25.776230+0000) 2022-09-08T03:58:26.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:26 smithi055 ceph-mon[97033]: pgmap v12293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 6 op/s 2022-09-08T03:58:26.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:26 smithi059 ceph-mon[97450]: pgmap v12293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 6 op/s 2022-09-08T03:58:27.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:26.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:26.798408+0000) 2022-09-08T03:58:27.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:26.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:26.776425+0000) 2022-09-08T03:58:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:27.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:27.798688+0000) 2022-09-08T03:58:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:27.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:27.776689+0000) 2022-09-08T03:58:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:28 smithi055 ceph-mon[97033]: pgmap v12294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T03:58:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:28 smithi059 ceph-mon[97450]: pgmap v12294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T03:58:29.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:28.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:28.798968+0000) 2022-09-08T03:58:29.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:28.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:28.776820+0000) 2022-09-08T03:58:29.776 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:58:29.463+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:58:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:29.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:29.799223+0000) 2022-09-08T03:58:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:29.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:29.776971+0000) 2022-09-08T03:58:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:30 smithi055 ceph-mon[97033]: pgmap v12295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:30.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:30 smithi059 ceph-mon[97450]: pgmap v12295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:31.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:30.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:30.799496+0000) 2022-09-08T03:58:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:30.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:30.777105+0000) 2022-09-08T03:58:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:58:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:58:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:58:31.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:58:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:58:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:58:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:31.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:31.799765+0000) 2022-09-08T03:58:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:31.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:31.777309+0000) 2022-09-08T03:58:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:32 smithi055 ceph-mon[97033]: pgmap v12296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:58:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:32 smithi055 ceph-mon[97033]: pgmap v12297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:58:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:58:32.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:32 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T03:58:32.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:32 smithi059 ceph-mon[97450]: pgmap v12296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:58:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:32 smithi059 ceph-mon[97450]: pgmap v12297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:58:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:58:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:32 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T03:58:33.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:32.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:32.800005+0000) 2022-09-08T03:58:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:32.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:32.777489+0000) 2022-09-08T03:58:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:33.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:33.800188+0000) 2022-09-08T03:58:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:33.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:33.777731+0000) 2022-09-08T03:58:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:34 smithi055 ceph-mon[97033]: pgmap v12298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:58:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:34 smithi059 ceph-mon[97450]: pgmap v12298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:58:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:34.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:34.800365+0000) 2022-09-08T03:58:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:34.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:34.777892+0000) 2022-09-08T03:58:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:35.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:35.800569+0000) 2022-09-08T03:58:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:35.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:35.778073+0000) 2022-09-08T03:58:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:36 smithi055 ceph-mon[97033]: pgmap v12299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 7 op/s 2022-09-08T03:58:36.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:36 smithi059 ceph-mon[97450]: pgmap v12299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 7 op/s 2022-09-08T03:58:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:36.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:36.800702+0000) 2022-09-08T03:58:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:36.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:36.778274+0000) 2022-09-08T03:58:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:37.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:37.800938+0000) 2022-09-08T03:58:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:37.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:37.778506+0000) 2022-09-08T03:58:38.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:38 smithi055 ceph-mon[97033]: pgmap v12300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:58:38.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:38 smithi059 ceph-mon[97450]: pgmap v12300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:58:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:38.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:38.801090+0000) 2022-09-08T03:58:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:38.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:38.778746+0000) 2022-09-08T03:58:40.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:39.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:39.801303+0000) 2022-09-08T03:58:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:39.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:39.779017+0000) 2022-09-08T03:58:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:40 smithi055 ceph-mon[97033]: pgmap v12301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:58:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:40 smithi059 ceph-mon[97450]: pgmap v12301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:58:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:40.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:40.801520+0000) 2022-09-08T03:58:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:40.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:40.779191+0000) 2022-09-08T03:58:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:41.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:41.801738+0000) 2022-09-08T03:58:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:41.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:41.779400+0000) 2022-09-08T03:58:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:42 smithi055 ceph-mon[97033]: pgmap v12302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:58:42.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:42 smithi059 ceph-mon[97450]: pgmap v12302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T03:58:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:42.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:42.801921+0000) 2022-09-08T03:58:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:42.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:42.779674+0000) 2022-09-08T03:58:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:43.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:43.802158+0000) 2022-09-08T03:58:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:43.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:43.779849+0000) 2022-09-08T03:58:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:44 smithi055 ceph-mon[97033]: pgmap v12303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:44.779 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:58:44.469+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:58:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:44 smithi059 ceph-mon[97450]: pgmap v12303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:44.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:44.779987+0000) 2022-09-08T03:58:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:44.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:44.802342+0000) 2022-09-08T03:58:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:45.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:45.780198+0000) 2022-09-08T03:58:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:45.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:45.802580+0000) 2022-09-08T03:58:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:46 smithi055 ceph-mon[97033]: pgmap v12304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:58:46.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:46 smithi059 ceph-mon[97450]: pgmap v12304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:58:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:46.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:46.780415+0000) 2022-09-08T03:58:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:46.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:46.802769+0000) 2022-09-08T03:58:48.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:47.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:47.802966+0000) 2022-09-08T03:58:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:47.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:47.780657+0000) 2022-09-08T03:58:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:48 smithi055 ceph-mon[97033]: pgmap v12305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:48.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:48 smithi059 ceph-mon[97450]: pgmap v12305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:48.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:48.803099+0000) 2022-09-08T03:58:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:48.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:48.780870+0000) 2022-09-08T03:58:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:49.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:49.803289+0000) 2022-09-08T03:58:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:49.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:49.781045+0000) 2022-09-08T03:58:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:50 smithi055 ceph-mon[97033]: pgmap v12306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:50.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:50 smithi059 ceph-mon[97450]: pgmap v12306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:50.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:50.781223+0000) 2022-09-08T03:58:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:50.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:50.803467+0000) 2022-09-08T03:58:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:51.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:51.803710+0000) 2022-09-08T03:58:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:51.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:51.781444+0000) 2022-09-08T03:58:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:52 smithi055 ceph-mon[97033]: pgmap v12307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:58:52.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:52 smithi059 ceph-mon[97450]: pgmap v12307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:58:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:52.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:52.803954+0000) 2022-09-08T03:58:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:52.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:52.781715+0000) 2022-09-08T03:58:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:53.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:53.804140+0000) 2022-09-08T03:58:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:53.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:53.781952+0000) 2022-09-08T03:58:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:54 smithi055 ceph-mon[97033]: pgmap v12308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:54.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:54 smithi059 ceph-mon[97450]: pgmap v12308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:54.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:54.804415+0000) 2022-09-08T03:58:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:54.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:54.782189+0000) 2022-09-08T03:58:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:55.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:55.804607+0000) 2022-09-08T03:58:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:55.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:55.782386+0000) 2022-09-08T03:58:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:56 smithi055 ceph-mon[97033]: pgmap v12309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:58:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:56 smithi059 ceph-mon[97450]: pgmap v12309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:58:57.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:56.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:56.804838+0000) 2022-09-08T03:58:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:56.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:56.782614+0000) 2022-09-08T03:58:58.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:57.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:57.805092+0000) 2022-09-08T03:58:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:57.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:57.782868+0000) 2022-09-08T03:58:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:58:58 smithi055 ceph-mon[97033]: pgmap v12310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:58.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:58:58 smithi059 ceph-mon[97450]: pgmap v12310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:58:59.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:58.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:58.783095+0000) 2022-09-08T03:58:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:58.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:58.805294+0000) 2022-09-08T03:58:59.782 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:58:59.479+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:59:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:58:59.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:59.805521+0000) 2022-09-08T03:59:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:58:59.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:58:59.783281+0000) 2022-09-08T03:59:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:00 smithi055 ceph-mon[97033]: pgmap v12311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:00.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:00 smithi059 ceph-mon[97450]: pgmap v12311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:00.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:00.805702+0000) 2022-09-08T03:59:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:00.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:00.783501+0000) 2022-09-08T03:59:02.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:01.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:01.805947+0000) 2022-09-08T03:59:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:01.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:01.783744+0000) 2022-09-08T03:59:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:02 smithi055 ceph-mon[97033]: pgmap v12312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:59:02.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:02 smithi059 ceph-mon[97450]: pgmap v12312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:59:03.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:02.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:02.806187+0000) 2022-09-08T03:59:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:02.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:02.783944+0000) 2022-09-08T03:59:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:03.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:03.806400+0000) 2022-09-08T03:59:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:03.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:03.784193+0000) 2022-09-08T03:59:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:04 smithi055 ceph-mon[97033]: pgmap v12313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:04 smithi059 ceph-mon[97450]: pgmap v12313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:04.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:04.806594+0000) 2022-09-08T03:59:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:04.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:04.784442+0000) 2022-09-08T03:59:06.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:05.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:05.806801+0000) 2022-09-08T03:59:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:05.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:05.784599+0000) 2022-09-08T03:59:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:06 smithi055 ceph-mon[97033]: pgmap v12314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:59:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:59:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:59:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:06 smithi059 ceph-mon[97450]: pgmap v12314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T03:59:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T03:59:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T03:59:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:06.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:06.807022+0000) 2022-09-08T03:59:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:06.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:06.784823+0000) 2022-09-08T03:59:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:07.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:07.807202+0000) 2022-09-08T03:59:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:07.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:07.785011+0000) 2022-09-08T03:59:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:08 smithi055 ceph-mon[97033]: pgmap v12315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:59:08.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:08 smithi059 ceph-mon[97450]: pgmap v12315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:59:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:08.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:08.785202+0000) 2022-09-08T03:59:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:08.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:08.807377+0000) 2022-09-08T03:59:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:09.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:09.807584+0000) 2022-09-08T03:59:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:09.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:09.785346+0000) 2022-09-08T03:59:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:10 smithi059 ceph-mon[97450]: pgmap v12316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:59:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:10 smithi059 ceph-mon[97450]: osdmap e848: 12 total, 11 up, 11 in 2022-09-08T03:59:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:10 smithi055 ceph-mon[97033]: pgmap v12316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T03:59:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:10 smithi055 ceph-mon[97033]: osdmap e848: 12 total, 11 up, 11 in 2022-09-08T03:59:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:10.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:10.807787+0000) 2022-09-08T03:59:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:10.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:10.785487+0000) 2022-09-08T03:59:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:11.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:11.808034+0000) 2022-09-08T03:59:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:11.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:11.785743+0000) 2022-09-08T03:59:12.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:12 smithi059 ceph-mon[97450]: pgmap v12318: 129 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:59:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:12 smithi055 ceph-mon[97033]: pgmap v12318: 129 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:59:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:12.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:12.808296+0000) 2022-09-08T03:59:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:12.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:12.785936+0000) 2022-09-08T03:59:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:13.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:13.808484+0000) 2022-09-08T03:59:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:13.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:13.786188+0000) 2022-09-08T03:59:14.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:14 smithi059 ceph-mon[97450]: pgmap v12319: 129 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:59:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:14 smithi059 ceph-mon[97450]: osdmap e849: 12 total, 11 up, 11 in 2022-09-08T03:59:14.785 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:59:14.496+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:59:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:14 smithi055 ceph-mon[97033]: pgmap v12319: 129 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T03:59:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:14 smithi055 ceph-mon[97033]: osdmap e849: 12 total, 11 up, 11 in 2022-09-08T03:59:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:14.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:14.808744+0000) 2022-09-08T03:59:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:14.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:14.786411+0000) 2022-09-08T03:59:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:15.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:15.808943+0000) 2022-09-08T03:59:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:15.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:15.786557+0000) 2022-09-08T03:59:16.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:16 smithi059 ceph-mon[97450]: pgmap v12321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:59:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:16 smithi055 ceph-mon[97033]: pgmap v12321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:59:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:16.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:16.809162+0000) 2022-09-08T03:59:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:16.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:16.786768+0000) 2022-09-08T03:59:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:17.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:17.809374+0000) 2022-09-08T03:59:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:17.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:17.786999+0000) 2022-09-08T03:59:18.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:18 smithi059 ceph-mon[97450]: pgmap v12322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:59:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:18 smithi055 ceph-mon[97033]: pgmap v12322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T03:59:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:18.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:18.809541+0000) 2022-09-08T03:59:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:18.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:18.787258+0000) 2022-09-08T03:59:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:19.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:19.809712+0000) 2022-09-08T03:59:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:19.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:19.787523+0000) 2022-09-08T03:59:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:20 smithi055 ceph-mon[97033]: osdmap e850: 12 total, 11 up, 11 in 2022-09-08T03:59:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:20 smithi055 ceph-mon[97033]: pgmap v12324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:59:20.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:20 smithi059 ceph-mon[97450]: osdmap e850: 12 total, 11 up, 11 in 2022-09-08T03:59:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:20 smithi059 ceph-mon[97450]: pgmap v12324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T03:59:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:20.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:20.809902+0000) 2022-09-08T03:59:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:20.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:20.787709+0000) 2022-09-08T03:59:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:21.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:21.787857+0000) 2022-09-08T03:59:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:21.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:21.810116+0000) 2022-09-08T03:59:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:22 smithi055 ceph-mon[97033]: pgmap v12325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T03:59:22.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:22 smithi059 ceph-mon[97450]: pgmap v12325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T03:59:23.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:22.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:22.788071+0000) 2022-09-08T03:59:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:22.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:22.810335+0000) 2022-09-08T03:59:24.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:23.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:23.788328+0000) 2022-09-08T03:59:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:23.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:23.810539+0000) 2022-09-08T03:59:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:24 smithi055 ceph-mon[97033]: pgmap v12326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 111 B/s wr, 6 op/s 2022-09-08T03:59:24.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:24 smithi059 ceph-mon[97450]: pgmap v12326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 111 B/s wr, 6 op/s 2022-09-08T03:59:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:24.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:24.810703+0000) 2022-09-08T03:59:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:24.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:24.788549+0000) 2022-09-08T03:59:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:25.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:25.810879+0000) 2022-09-08T03:59:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:25.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:25.788787+0000) 2022-09-08T03:59:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:26 smithi059 ceph-mon[97450]: pgmap v12327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:59:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:26 smithi055 ceph-mon[97033]: pgmap v12327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:59:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:26.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:26.811169+0000) 2022-09-08T03:59:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:26.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:26.788995+0000) 2022-09-08T03:59:28.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:27.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:27.811398+0000) 2022-09-08T03:59:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:27.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:27.789193+0000) 2022-09-08T03:59:28.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:28 smithi059 ceph-mon[97450]: pgmap v12328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:59:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:28 smithi055 ceph-mon[97033]: pgmap v12328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T03:59:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:28.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:28.811610+0000) 2022-09-08T03:59:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:28.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:28.789418+0000) 2022-09-08T03:59:29.789 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:59:29.509+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:59:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:29.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:29.789624+0000) 2022-09-08T03:59:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:29.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:29.811793+0000) 2022-09-08T03:59:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:30 smithi055 ceph-mon[97033]: pgmap v12329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:59:30.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:30 smithi059 ceph-mon[97450]: pgmap v12329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T03:59:31.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:30.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:30.811943+0000) 2022-09-08T03:59:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:30.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:30.789797+0000) 2022-09-08T03:59:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:31.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:31.812170+0000) 2022-09-08T03:59:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:31.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:31.790039+0000) 2022-09-08T03:59:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:32 smithi055 ceph-mon[97033]: pgmap v12330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:59:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:59:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:59:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:59:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:59:32.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:32 smithi059 ceph-mon[97450]: pgmap v12330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:59:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T03:59:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T03:59:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T03:59:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T03:59:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:32.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:32.812356+0000) 2022-09-08T03:59:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:32.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:32.790296+0000) 2022-09-08T03:59:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:33.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:33.812546+0000) 2022-09-08T03:59:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:33.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:33.790546+0000) 2022-09-08T03:59:34.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:34 smithi059 ceph-mon[97450]: pgmap v12331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:34 smithi055 ceph-mon[97033]: pgmap v12331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:34.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:34.812780+0000) 2022-09-08T03:59:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:34.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:34.790730+0000) 2022-09-08T03:59:36.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:35.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:35.812999+0000) 2022-09-08T03:59:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:35.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:35.790934+0000) 2022-09-08T03:59:36.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:36 smithi059 ceph-mon[97450]: pgmap v12332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:59:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:36 smithi055 ceph-mon[97033]: pgmap v12332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:59:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:36.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:36.813227+0000) 2022-09-08T03:59:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:36.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:36.791145+0000) 2022-09-08T03:59:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:37.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:37.813476+0000) 2022-09-08T03:59:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:37.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:37.791379+0000) 2022-09-08T03:59:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:38 smithi059 ceph-mon[97450]: pgmap v12333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:38 smithi055 ceph-mon[97033]: pgmap v12333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:38.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:38.813680+0000) 2022-09-08T03:59:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:38.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:38.791611+0000) 2022-09-08T03:59:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:39.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:39.813859+0000) 2022-09-08T03:59:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:39.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:39.791719+0000) 2022-09-08T03:59:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:40 smithi055 ceph-mon[97033]: pgmap v12334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:40 smithi059 ceph-mon[97450]: pgmap v12334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:40.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:40.814066+0000) 2022-09-08T03:59:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:40.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:40.791974+0000) 2022-09-08T03:59:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:41.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:41.814305+0000) 2022-09-08T03:59:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:41.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:41.792185+0000) 2022-09-08T03:59:42.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:42 smithi059 ceph-mon[97450]: pgmap v12335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:59:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:42 smithi055 ceph-mon[97033]: pgmap v12335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:59:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:42.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:42.814543+0000) 2022-09-08T03:59:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:42.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:42.792444+0000) 2022-09-08T03:59:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:43.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:43.814776+0000) 2022-09-08T03:59:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:43.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:43.792668+0000) 2022-09-08T03:59:44.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:44 smithi059 ceph-mon[97450]: pgmap v12336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:44.792 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:59:44.522+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T03:59:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:44 smithi055 ceph-mon[97033]: pgmap v12336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:44.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:44.814964+0000) 2022-09-08T03:59:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:44.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:44.792810+0000) 2022-09-08T03:59:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:45.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:45.815116+0000) 2022-09-08T03:59:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:45.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:45.793068+0000) 2022-09-08T03:59:46.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:46 smithi059 ceph-mon[97450]: pgmap v12337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:59:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:46 smithi055 ceph-mon[97033]: pgmap v12337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:59:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:46.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:46.815326+0000) 2022-09-08T03:59:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:46.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:46.793203+0000) 2022-09-08T03:59:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:47.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:47.815515+0000) 2022-09-08T03:59:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:47.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:47.793393+0000) 2022-09-08T03:59:48.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:48 smithi059 ceph-mon[97450]: pgmap v12338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:48 smithi055 ceph-mon[97033]: pgmap v12338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:48.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:48.815750+0000) 2022-09-08T03:59:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:48.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:48.793613+0000) 2022-09-08T03:59:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:49.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:49.815943+0000) 2022-09-08T03:59:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:49.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:49.793867+0000) 2022-09-08T03:59:50.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:50 smithi055 ceph-mon[97033]: pgmap v12339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:50.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:50 smithi059 ceph-mon[97450]: pgmap v12339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:50.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:50.816173+0000) 2022-09-08T03:59:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:50.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:50.794063+0000) 2022-09-08T03:59:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:51.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:51.794272+0000) 2022-09-08T03:59:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:51.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:51.816420+0000) 2022-09-08T03:59:52.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:52 smithi059 ceph-mon[97450]: pgmap v12340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:59:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:52 smithi055 ceph-mon[97033]: pgmap v12340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:59:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:52.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:52.816630+0000) 2022-09-08T03:59:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:52.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:52.794417+0000) 2022-09-08T03:59:54.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:53.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:53.816847+0000) 2022-09-08T03:59:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:53.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:53.794676+0000) 2022-09-08T03:59:54.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:54 smithi059 ceph-mon[97450]: pgmap v12341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:54 smithi055 ceph-mon[97033]: pgmap v12341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:55.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:54.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:54.817059+0000) 2022-09-08T03:59:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:54.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:54.794927+0000) 2022-09-08T03:59:56.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:55.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:55.817285+0000) 2022-09-08T03:59:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:55.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:55.795182+0000) 2022-09-08T03:59:56.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:56 smithi059 ceph-mon[97450]: pgmap v12342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:59:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:56 smithi055 ceph-mon[97033]: pgmap v12342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T03:59:57.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:56.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:56.817490+0000) 2022-09-08T03:59:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:56.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:56.795374+0000) 2022-09-08T03:59:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:57.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:57.817767+0000) 2022-09-08T03:59:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:57.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:57.795600+0000) 2022-09-08T03:59:58.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 03:59:58 smithi059 ceph-mon[97450]: pgmap v12343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 03:59:58 smithi055 ceph-mon[97033]: pgmap v12343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T03:59:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:58.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:58.817962+0000) 2022-09-08T03:59:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:58.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:58.795785+0000) 2022-09-08T03:59:59.795 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 03:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T03:59:59.533+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:00:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 03:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T03:59:59.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:59.818137+0000) 2022-09-08T04:00:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 03:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T03:59:59.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T02:59:59.795911+0000) 2022-09-08T04:00:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:00 smithi055 ceph-mon[97033]: pgmap v12344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T04:00:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T04:00:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T04:00:00.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:00 smithi059 ceph-mon[97450]: pgmap v12344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T04:00:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T04:00:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T04:00:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:00.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:00.818318+0000) 2022-09-08T04:00:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:00.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:00.796123+0000) 2022-09-08T04:00:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:01.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:01.818537+0000) 2022-09-08T04:00:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:01.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:01.796319+0000) 2022-09-08T04:00:02.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:02 smithi059 ceph-mon[97450]: pgmap v12345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:00:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:02 smithi055 ceph-mon[97033]: pgmap v12345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:00:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:02.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:02.796536+0000) 2022-09-08T04:00:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:02.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:02.818794+0000) 2022-09-08T04:00:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:03.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:03.819055+0000) 2022-09-08T04:00:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:03.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:03.796778+0000) 2022-09-08T04:00:04.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:04 smithi059 ceph-mon[97450]: pgmap v12346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:04 smithi055 ceph-mon[97033]: pgmap v12346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:04.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:04.819294+0000) 2022-09-08T04:00:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:04.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:04.796950+0000) 2022-09-08T04:00:05.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:00:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:00:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:05.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:05.797113+0000) 2022-09-08T04:00:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:05.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:05.819552+0000) 2022-09-08T04:00:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:06 smithi059 ceph-mon[97450]: pgmap v12347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:00:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:00:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:06 smithi055 ceph-mon[97033]: pgmap v12347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:00:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:00:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:06.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:06.797312+0000) 2022-09-08T04:00:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:06.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:06.819751+0000) 2022-09-08T04:00:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:07.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:07.819925+0000) 2022-09-08T04:00:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:07.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:07.797496+0000) 2022-09-08T04:00:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:08 smithi059 ceph-mon[97450]: pgmap v12348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:00:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:08 smithi055 ceph-mon[97033]: pgmap v12348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:00:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:08.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:08.797741+0000) 2022-09-08T04:00:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:08.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:08.820169+0000) 2022-09-08T04:00:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:09.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:09.820333+0000) 2022-09-08T04:00:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:09.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:09.797895+0000) 2022-09-08T04:00:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:10 smithi059 ceph-mon[97450]: pgmap v12349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:00:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:10 smithi059 ceph-mon[97450]: osdmap e851: 12 total, 11 up, 11 in 2022-09-08T04:00:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:10 smithi055 ceph-mon[97033]: pgmap v12349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:00:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:10 smithi055 ceph-mon[97033]: osdmap e851: 12 total, 11 up, 11 in 2022-09-08T04:00:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:10.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:10.820507+0000) 2022-09-08T04:00:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:10.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:10.798060+0000) 2022-09-08T04:00:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:11.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:11.820680+0000) 2022-09-08T04:00:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:11.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:11.798261+0000) 2022-09-08T04:00:12.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:12 smithi059 ceph-mon[97450]: pgmap v12351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:00:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:12 smithi055 ceph-mon[97033]: pgmap v12351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:00:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:12.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:12.820879+0000) 2022-09-08T04:00:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:12.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:12.798536+0000) 2022-09-08T04:00:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:13.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:13.821129+0000) 2022-09-08T04:00:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:13.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:13.798796+0000) 2022-09-08T04:00:14.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:14 smithi059 ceph-mon[97450]: pgmap v12352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:00:14.798 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:00:14.547+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:00:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:14 smithi055 ceph-mon[97033]: pgmap v12352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:00:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:14.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:14.799008+0000) 2022-09-08T04:00:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:14.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:14.821374+0000) 2022-09-08T04:00:16.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:15.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:15.821597+0000) 2022-09-08T04:00:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:15.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:15.799188+0000) 2022-09-08T04:00:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:16 smithi059 ceph-mon[97450]: pgmap v12353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:00:16.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:16 smithi055 ceph-mon[97033]: pgmap v12353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:00:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:16.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:16.821856+0000) 2022-09-08T04:00:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:16.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:16.799387+0000) 2022-09-08T04:00:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:17.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:17.822100+0000) 2022-09-08T04:00:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:17.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:17.799630+0000) 2022-09-08T04:00:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:18 smithi059 ceph-mon[97450]: pgmap v12354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:00:18.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:18 smithi055 ceph-mon[97033]: pgmap v12354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:00:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:18.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:18.799941+0000) 2022-09-08T04:00:19.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:18.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:18.822417+0000) 2022-09-08T04:00:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:19.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:19.822673+0000) 2022-09-08T04:00:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:19.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:19.800192+0000) 2022-09-08T04:00:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:20 smithi055 ceph-mon[97033]: osdmap e852: 12 total, 11 up, 11 in 2022-09-08T04:00:20.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:20 smithi055 ceph-mon[97033]: pgmap v12356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T04:00:20.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:20 smithi059 ceph-mon[97450]: osdmap e852: 12 total, 11 up, 11 in 2022-09-08T04:00:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:20 smithi059 ceph-mon[97450]: pgmap v12356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T04:00:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:20.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:20.822895+0000) 2022-09-08T04:00:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:20.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:20.800387+0000) 2022-09-08T04:00:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:21.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:21.800610+0000) 2022-09-08T04:00:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:21.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:21.823123+0000) 2022-09-08T04:00:22.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:22 smithi059 ceph-mon[97450]: pgmap v12357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:00:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:22 smithi055 ceph-mon[97033]: pgmap v12357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:00:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:22.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:22.800874+0000) 2022-09-08T04:00:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:22.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:22.823334+0000) 2022-09-08T04:00:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:23.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:23.823495+0000) 2022-09-08T04:00:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:23.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:23.801092+0000) 2022-09-08T04:00:24.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:24 smithi059 ceph-mon[97450]: pgmap v12358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:00:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:24 smithi055 ceph-mon[97033]: pgmap v12358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:00:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:24.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:24.823690+0000) 2022-09-08T04:00:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:24.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:24.801328+0000) 2022-09-08T04:00:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:25.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:25.823823+0000) 2022-09-08T04:00:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:25.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:25.801541+0000) 2022-09-08T04:00:26.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:26 smithi059 ceph-mon[97450]: pgmap v12359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:00:26.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:26 smithi055 ceph-mon[97033]: pgmap v12359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:00:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:26.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:26.824069+0000) 2022-09-08T04:00:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:26.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:26.801760+0000) 2022-09-08T04:00:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:27.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:27.824276+0000) 2022-09-08T04:00:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:27.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:27.802001+0000) 2022-09-08T04:00:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:28 smithi059 ceph-mon[97450]: pgmap v12360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:00:28.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:28 smithi055 ceph-mon[97033]: pgmap v12360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:00:29.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:28.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:28.824493+0000) 2022-09-08T04:00:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:28.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:28.802200+0000) 2022-09-08T04:00:29.824 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:00:29.562+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:00:29.825 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:29.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:29.802377+0000) 2022-09-08T04:00:30.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:29.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:29.824666+0000) 2022-09-08T04:00:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:30 smithi055 ceph-mon[97033]: pgmap v12361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:00:30.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:30 smithi059 ceph-mon[97450]: pgmap v12361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:00:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:30.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:30.824829+0000) 2022-09-08T04:00:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:30.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:30.802614+0000) 2022-09-08T04:00:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:31.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:31.825056+0000) 2022-09-08T04:00:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:31.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:31.802878+0000) 2022-09-08T04:00:32.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:32 smithi059 ceph-mon[97450]: pgmap v12362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:00:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:00:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:00:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:00:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:32 smithi055 ceph-mon[97033]: pgmap v12362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:00:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:00:32.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:00:32.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:00:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:32.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:32.825250+0000) 2022-09-08T04:00:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:32.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:32.803141+0000) 2022-09-08T04:00:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:00:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:00:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:33.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:33.825495+0000) 2022-09-08T04:00:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:33.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:33.803492+0000) 2022-09-08T04:00:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:34 smithi055 ceph-mon[97033]: pgmap v12363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:34 smithi059 ceph-mon[97450]: pgmap v12363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:34.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:34.825612+0000) 2022-09-08T04:00:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:34.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:34.803631+0000) 2022-09-08T04:00:36.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:35.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:35.825822+0000) 2022-09-08T04:00:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:35.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:35.803880+0000) 2022-09-08T04:00:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:36 smithi055 ceph-mon[97033]: pgmap v12364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:00:37.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:36 smithi059 ceph-mon[97450]: pgmap v12364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:00:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:36.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:36.826044+0000) 2022-09-08T04:00:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:36.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:36.804106+0000) 2022-09-08T04:00:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:37.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:37.826268+0000) 2022-09-08T04:00:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:37.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:37.804345+0000) 2022-09-08T04:00:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:38 smithi055 ceph-mon[97033]: pgmap v12365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:39.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:38 smithi059 ceph-mon[97450]: pgmap v12365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:38.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:38.826499+0000) 2022-09-08T04:00:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:38.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:38.804582+0000) 2022-09-08T04:00:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:39.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:39.804773+0000) 2022-09-08T04:00:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:39.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:39.826718+0000) 2022-09-08T04:00:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:40 smithi059 ceph-mon[97450]: pgmap v12366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:40 smithi055 ceph-mon[97033]: pgmap v12366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:40.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:40.826912+0000) 2022-09-08T04:00:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:40.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:40.804966+0000) 2022-09-08T04:00:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:41.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:41.827105+0000) 2022-09-08T04:00:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:41.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:41.805160+0000) 2022-09-08T04:00:42.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:42 smithi059 ceph-mon[97450]: pgmap v12367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:00:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:42 smithi055 ceph-mon[97033]: pgmap v12367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:00:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:42.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:42.827378+0000) 2022-09-08T04:00:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:42.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:42.805386+0000) 2022-09-08T04:00:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:43.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:43.827615+0000) 2022-09-08T04:00:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:43.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:43.805650+0000) 2022-09-08T04:00:44.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:44 smithi059 ceph-mon[97450]: pgmap v12368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:44.827 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:00:44.565+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:00:44.828 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:44.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:44.805906+0000) 2022-09-08T04:00:44.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:44 smithi055 ceph-mon[97033]: pgmap v12368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:44.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:44.827927+0000) 2022-09-08T04:00:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:45.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:45.828099+0000) 2022-09-08T04:00:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:45.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:45.806109+0000) 2022-09-08T04:00:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:46 smithi059 ceph-mon[97450]: pgmap v12369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:00:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:46 smithi055 ceph-mon[97033]: pgmap v12369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:00:47.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:46.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:46.828294+0000) 2022-09-08T04:00:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:46.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:46.806314+0000) 2022-09-08T04:00:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:47.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:47.828543+0000) 2022-09-08T04:00:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:47.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:47.806522+0000) 2022-09-08T04:00:48.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:48 smithi059 ceph-mon[97450]: pgmap v12370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:48 smithi055 ceph-mon[97033]: pgmap v12370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:48.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:48.806737+0000) 2022-09-08T04:00:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:48.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:48.828754+0000) 2022-09-08T04:00:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:49.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:49.828968+0000) 2022-09-08T04:00:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:49.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:49.806889+0000) 2022-09-08T04:00:50.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:50 smithi059 ceph-mon[97450]: pgmap v12371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:50 smithi055 ceph-mon[97033]: pgmap v12371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:50.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:50.829220+0000) 2022-09-08T04:00:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:50.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:50.807087+0000) 2022-09-08T04:00:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:51.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:51.829406+0000) 2022-09-08T04:00:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:51.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:51.807332+0000) 2022-09-08T04:00:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:52 smithi059 ceph-mon[97450]: pgmap v12372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:00:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:52 smithi055 ceph-mon[97033]: pgmap v12372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:00:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:52.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:52.829666+0000) 2022-09-08T04:00:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:52.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:52.807555+0000) 2022-09-08T04:00:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:53.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:53.807717+0000) 2022-09-08T04:00:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:53.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:53.829852+0000) 2022-09-08T04:00:54.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:54 smithi059 ceph-mon[97450]: pgmap v12373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:54 smithi055 ceph-mon[97033]: pgmap v12373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:00:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:54.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:54.830098+0000) 2022-09-08T04:00:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:54.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:54.807936+0000) 2022-09-08T04:00:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:55.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:55.808150+0000) 2022-09-08T04:00:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:55.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:55.830328+0000) 2022-09-08T04:00:56.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:56 smithi059 ceph-mon[97450]: pgmap v12374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:00:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:56 smithi055 ceph-mon[97033]: pgmap v12374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:00:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:56.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:56.830548+0000) 2022-09-08T04:00:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:56.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:56.808385+0000) 2022-09-08T04:00:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:57.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:57.830788+0000) 2022-09-08T04:00:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:57.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:57.808609+0000) 2022-09-08T04:00:58.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:00:58 smithi059 ceph-mon[97450]: pgmap v12375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:00:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:00:58 smithi055 ceph-mon[97033]: pgmap v12375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:00:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:58.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:58.831033+0000) 2022-09-08T04:00:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:58.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:58.808837+0000) 2022-09-08T04:00:59.830 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:00:59.579+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:00:59.831 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:00:59.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:59.809063+0000) 2022-09-08T04:01:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:00:59.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:00:59.831263+0000) 2022-09-08T04:01:00.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:00 smithi059 ceph-mon[97450]: pgmap v12376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:01:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:00 smithi055 ceph-mon[97033]: pgmap v12376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:01:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:00.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:00.831452+0000) 2022-09-08T04:01:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:00.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:00.809271+0000) 2022-09-08T04:01:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:01.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:01.831659+0000) 2022-09-08T04:01:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:01.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:01.809486+0000) 2022-09-08T04:01:02.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:02 smithi059 ceph-mon[97450]: pgmap v12377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:01:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:02 smithi055 ceph-mon[97033]: pgmap v12377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:01:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:02.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:02.809798+0000) 2022-09-08T04:01:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:02.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:02.831822+0000) 2022-09-08T04:01:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:03.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:03.832033+0000) 2022-09-08T04:01:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:03.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:03.809984+0000) 2022-09-08T04:01:04.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:04 smithi059 ceph-mon[97450]: pgmap v12378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T04:01:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:04 smithi055 ceph-mon[97033]: pgmap v12378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T04:01:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:04.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:04.810236+0000) 2022-09-08T04:01:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:04.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:04.832227+0000) 2022-09-08T04:01:05.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:01:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:01:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:05.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:05.832500+0000) 2022-09-08T04:01:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:05.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:05.810448+0000) 2022-09-08T04:01:06.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:06 smithi059 ceph-mon[97450]: pgmap v12379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:01:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:01:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:06 smithi055 ceph-mon[97033]: pgmap v12379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:01:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:01:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:06.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:06.832778+0000) 2022-09-08T04:01:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:06.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:06.810631+0000) 2022-09-08T04:01:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:07.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:07.833026+0000) 2022-09-08T04:01:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:07.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:07.810898+0000) 2022-09-08T04:01:08.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:08 smithi059 ceph-mon[97450]: pgmap v12380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:01:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:08 smithi055 ceph-mon[97033]: pgmap v12380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:01:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:08.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:08.833209+0000) 2022-09-08T04:01:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:08.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:08.811147+0000) 2022-09-08T04:01:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:09.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:09.833368+0000) 2022-09-08T04:01:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:09.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:09.812075+0000) 2022-09-08T04:01:10.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:10 smithi059 ceph-mon[97450]: pgmap v12381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:01:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:10 smithi059 ceph-mon[97450]: osdmap e853: 12 total, 11 up, 11 in 2022-09-08T04:01:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:10 smithi055 ceph-mon[97033]: pgmap v12381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:01:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:10 smithi055 ceph-mon[97033]: osdmap e853: 12 total, 11 up, 11 in 2022-09-08T04:01:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:10.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:10.833609+0000) 2022-09-08T04:01:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:10.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:10.812265+0000) 2022-09-08T04:01:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:11.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:11.833820+0000) 2022-09-08T04:01:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:11.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:11.812491+0000) 2022-09-08T04:01:12.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:12 smithi059 ceph-mon[97450]: pgmap v12383: 129 pgs: 16 active+clean+snaptrim_wait, 113 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:01:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:12 smithi055 ceph-mon[97033]: pgmap v12383: 129 pgs: 16 active+clean+snaptrim_wait, 113 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:01:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:12.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:12.834044+0000) 2022-09-08T04:01:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:12.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:12.812783+0000) 2022-09-08T04:01:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:13.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:13.834220+0000) 2022-09-08T04:01:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:13.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:13.813093+0000) 2022-09-08T04:01:14.834 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:01:14.582+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:01:14.834 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:14.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:14.813333+0000) 2022-09-08T04:01:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:14 smithi055 ceph-mon[97033]: pgmap v12384: 129 pgs: 16 active+clean+snaptrim_wait, 113 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:01:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:14 smithi055 ceph-mon[97033]: osdmap e854: 12 total, 11 up, 11 in 2022-09-08T04:01:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:14 smithi059 ceph-mon[97450]: pgmap v12384: 129 pgs: 16 active+clean+snaptrim_wait, 113 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:01:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:14 smithi059 ceph-mon[97450]: osdmap e854: 12 total, 11 up, 11 in 2022-09-08T04:01:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:14.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:14.834422+0000) 2022-09-08T04:01:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:15.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:15.834613+0000) 2022-09-08T04:01:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:15.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:15.813591+0000) 2022-09-08T04:01:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:16 smithi055 ceph-mon[97033]: pgmap v12386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:01:17.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:16 smithi059 ceph-mon[97450]: pgmap v12386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:01:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:16.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:16.834821+0000) 2022-09-08T04:01:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:16.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:16.813817+0000) 2022-09-08T04:01:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:17.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:17.834962+0000) 2022-09-08T04:01:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:17.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:17.814054+0000) 2022-09-08T04:01:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:18 smithi055 ceph-mon[97033]: pgmap v12387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:01:19.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:18 smithi059 ceph-mon[97450]: pgmap v12387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:01:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:18.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:18.814229+0000) 2022-09-08T04:01:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:18.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:18.835100+0000) 2022-09-08T04:01:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:19.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:19.814477+0000) 2022-09-08T04:01:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:19.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:19.835284+0000) 2022-09-08T04:01:20.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:20 smithi059 ceph-mon[97450]: osdmap e855: 12 total, 11 up, 11 in 2022-09-08T04:01:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:20 smithi059 ceph-mon[97450]: pgmap v12389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T04:01:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:20 smithi055 ceph-mon[97033]: osdmap e855: 12 total, 11 up, 11 in 2022-09-08T04:01:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:20 smithi055 ceph-mon[97033]: pgmap v12389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T04:01:21.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:20.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:20.835437+0000) 2022-09-08T04:01:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:20.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:20.814733+0000) 2022-09-08T04:01:22.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:21.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:21.814968+0000) 2022-09-08T04:01:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:21.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:21.835678+0000) 2022-09-08T04:01:22.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:22 smithi059 ceph-mon[97450]: pgmap v12390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T04:01:22.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:22 smithi055 ceph-mon[97033]: pgmap v12390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T04:01:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:22.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:22.835917+0000) 2022-09-08T04:01:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:22.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:22.815178+0000) 2022-09-08T04:01:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:23.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:23.815308+0000) 2022-09-08T04:01:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:23.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:23.836105+0000) 2022-09-08T04:01:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:24 smithi059 ceph-mon[97450]: pgmap v12391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 110 B/s wr, 6 op/s 2022-09-08T04:01:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:24 smithi055 ceph-mon[97033]: pgmap v12391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 110 B/s wr, 6 op/s 2022-09-08T04:01:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:24.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:24.836321+0000) 2022-09-08T04:01:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:24.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:24.815536+0000) 2022-09-08T04:01:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:25.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:25.836500+0000) 2022-09-08T04:01:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:25.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:25.815771+0000) 2022-09-08T04:01:26.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:26 smithi059 ceph-mon[97450]: pgmap v12392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:01:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:26 smithi055 ceph-mon[97033]: pgmap v12392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:01:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:26.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:26.836703+0000) 2022-09-08T04:01:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:26.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:26.815916+0000) 2022-09-08T04:01:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:27.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:27.836917+0000) 2022-09-08T04:01:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:27.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:27.816075+0000) 2022-09-08T04:01:28.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:28 smithi059 ceph-mon[97450]: pgmap v12393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:01:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:28 smithi055 ceph-mon[97033]: pgmap v12393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:01:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:28.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:28.837082+0000) 2022-09-08T04:01:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:28.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:28.816272+0000) 2022-09-08T04:01:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:29.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:29.837310+0000) 2022-09-08T04:01:29.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:01:29.598+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:01:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:29.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:29.816453+0000) 2022-09-08T04:01:30.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:30 smithi059 ceph-mon[97450]: pgmap v12394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:01:30.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:30 smithi055 ceph-mon[97033]: pgmap v12394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:01:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:30.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:30.837586+0000) 2022-09-08T04:01:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:30.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:30.816699+0000) 2022-09-08T04:01:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:31.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:31.837788+0000) 2022-09-08T04:01:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:31.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:31.816982+0000) 2022-09-08T04:01:32.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:32 smithi059 ceph-mon[97450]: pgmap v12395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:01:32.781 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:32 smithi055 ceph-mon[97033]: pgmap v12395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:01:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:32.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:32.838021+0000) 2022-09-08T04:01:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:32.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:32.817165+0000) 2022-09-08T04:01:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:01:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:01:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:01:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:01:33.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:01:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:01:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:01:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:01:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:33.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:33.838200+0000) 2022-09-08T04:01:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:33.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:33.817416+0000) 2022-09-08T04:01:34.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:34 smithi059 ceph-mon[97450]: pgmap v12396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:34.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:34 smithi055 ceph-mon[97033]: pgmap v12396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:34.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:34.838387+0000) 2022-09-08T04:01:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:34.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:34.817537+0000) 2022-09-08T04:01:36.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:35.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:35.838584+0000) 2022-09-08T04:01:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:35.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:35.817729+0000) 2022-09-08T04:01:36.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:36 smithi059 ceph-mon[97450]: pgmap v12397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:01:36.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:36 smithi055 ceph-mon[97033]: pgmap v12397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:01:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:36.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:36.838774+0000) 2022-09-08T04:01:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:36.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:36.817958+0000) 2022-09-08T04:01:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:37.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:37.839020+0000) 2022-09-08T04:01:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:37.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:37.818213+0000) 2022-09-08T04:01:38.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:38 smithi059 ceph-mon[97450]: pgmap v12398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:38 smithi055 ceph-mon[97033]: pgmap v12398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:38.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:38.839336+0000) 2022-09-08T04:01:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:38.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:38.818483+0000) 2022-09-08T04:01:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:39.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:39.839627+0000) 2022-09-08T04:01:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:39.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:39.818726+0000) 2022-09-08T04:01:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:40 smithi059 ceph-mon[97450]: pgmap v12399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:40.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:40 smithi055 ceph-mon[97033]: pgmap v12399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:40.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:40.839850+0000) 2022-09-08T04:01:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:40.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:40.818942+0000) 2022-09-08T04:01:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:41.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:41.840087+0000) 2022-09-08T04:01:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:41.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:41.819188+0000) 2022-09-08T04:01:42.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:42 smithi059 ceph-mon[97450]: pgmap v12400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:01:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:42 smithi055 ceph-mon[97033]: pgmap v12400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:01:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:42.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:42.840308+0000) 2022-09-08T04:01:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:42.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:42.819439+0000) 2022-09-08T04:01:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:43.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:43.840538+0000) 2022-09-08T04:01:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:43.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:43.819688+0000) 2022-09-08T04:01:44.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:44 smithi059 ceph-mon[97450]: pgmap v12401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:44.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:01:44.601+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:01:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:44.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:44.819946+0000) 2022-09-08T04:01:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:44.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:44.840744+0000) 2022-09-08T04:01:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:44 smithi055 ceph-mon[97033]: pgmap v12401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:45.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:45.840902+0000) 2022-09-08T04:01:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:45.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:45.820137+0000) 2022-09-08T04:01:46.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:46 smithi059 ceph-mon[97450]: pgmap v12402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:01:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:46 smithi055 ceph-mon[97033]: pgmap v12402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:01:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:46.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:46.841056+0000) 2022-09-08T04:01:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:46.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:46.820347+0000) 2022-09-08T04:01:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:47.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:47.841299+0000) 2022-09-08T04:01:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:47.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:47.820572+0000) 2022-09-08T04:01:48.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:48 smithi059 ceph-mon[97450]: pgmap v12403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:48 smithi055 ceph-mon[97033]: pgmap v12403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:48.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:48.841488+0000) 2022-09-08T04:01:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:48.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:48.820822+0000) 2022-09-08T04:01:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:49.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:49.841689+0000) 2022-09-08T04:01:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:49.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:49.821094+0000) 2022-09-08T04:01:50.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:50 smithi059 ceph-mon[97450]: pgmap v12404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:50 smithi055 ceph-mon[97033]: pgmap v12404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:50.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:50.841918+0000) 2022-09-08T04:01:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:50.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:50.821343+0000) 2022-09-08T04:01:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:51.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:51.842145+0000) 2022-09-08T04:01:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:51.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:51.821619+0000) 2022-09-08T04:01:52.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:52 smithi059 ceph-mon[97450]: pgmap v12405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:01:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:52 smithi055 ceph-mon[97033]: pgmap v12405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:01:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:52.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:52.842403+0000) 2022-09-08T04:01:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:52.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:52.821853+0000) 2022-09-08T04:01:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:53.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:53.842624+0000) 2022-09-08T04:01:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:53.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:53.822129+0000) 2022-09-08T04:01:54.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:54 smithi059 ceph-mon[97450]: pgmap v12406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:54.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:54 smithi055 ceph-mon[97033]: pgmap v12406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:54.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:54.842884+0000) 2022-09-08T04:01:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:54.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:54.822357+0000) 2022-09-08T04:01:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:55.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:55.822511+0000) 2022-09-08T04:01:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:55.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:55.843036+0000) 2022-09-08T04:01:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:56 smithi059 ceph-mon[97450]: pgmap v12407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:01:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:56 smithi055 ceph-mon[97033]: pgmap v12407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:01:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:56.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:56.843267+0000) 2022-09-08T04:01:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:56.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:56.822751+0000) 2022-09-08T04:01:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:57.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:57.843496+0000) 2022-09-08T04:01:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:57.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:57.823027+0000) 2022-09-08T04:01:58.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:01:58 smithi059 ceph-mon[97450]: pgmap v12408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:58.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:01:58 smithi055 ceph-mon[97033]: pgmap v12408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:01:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:58.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:58.843698+0000) 2022-09-08T04:01:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:58.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:58.823253+0000) 2022-09-08T04:01:59.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:01:59.613+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:01:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:01:59.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:59.843819+0000) 2022-09-08T04:01:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:01:59.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:01:59.823426+0000) 2022-09-08T04:02:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:00 smithi059 ceph-mon[97450]: pgmap v12409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:00 smithi055 ceph-mon[97033]: pgmap v12409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:00.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:00.843978+0000) 2022-09-08T04:02:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:00.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:00.823587+0000) 2022-09-08T04:02:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:01.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:01.844160+0000) 2022-09-08T04:02:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:01.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:01.823812+0000) 2022-09-08T04:02:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:02 smithi059 ceph-mon[97450]: pgmap v12410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:02:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:02 smithi055 ceph-mon[97033]: pgmap v12410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:02:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:02.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:02.824073+0000) 2022-09-08T04:02:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:02.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:02.844326+0000) 2022-09-08T04:02:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:03.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:03.844587+0000) 2022-09-08T04:02:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:03.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:03.824342+0000) 2022-09-08T04:02:04.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:04 smithi059 ceph-mon[97450]: pgmap v12411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:04 smithi055 ceph-mon[97033]: pgmap v12411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:04.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:04.844834+0000) 2022-09-08T04:02:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:04.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:04.824587+0000) 2022-09-08T04:02:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:02:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:02:06.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:05.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:05.844986+0000) 2022-09-08T04:02:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:05.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:05.824844+0000) 2022-09-08T04:02:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:06 smithi055 ceph-mon[97033]: pgmap v12412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:02:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:02:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:06 smithi059 ceph-mon[97450]: pgmap v12412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:02:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:02:07.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:06.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:06.825053+0000) 2022-09-08T04:02:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:06.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:06.845155+0000) 2022-09-08T04:02:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:07.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:07.845377+0000) 2022-09-08T04:02:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:07.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:07.825309+0000) 2022-09-08T04:02:08.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:08 smithi055 ceph-mon[97033]: pgmap v12413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:02:09.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:08 smithi059 ceph-mon[97450]: pgmap v12413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:02:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:08.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:08.845588+0000) 2022-09-08T04:02:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:08.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:08.825547+0000) 2022-09-08T04:02:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:09.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:09.845705+0000) 2022-09-08T04:02:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:09.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:09.825790+0000) 2022-09-08T04:02:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:10 smithi055 ceph-mon[97033]: pgmap v12414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:02:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:10 smithi055 ceph-mon[97033]: osdmap e856: 12 total, 11 up, 11 in 2022-09-08T04:02:11.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:10 smithi059 ceph-mon[97450]: pgmap v12414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:02:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:10 smithi059 ceph-mon[97450]: osdmap e856: 12 total, 11 up, 11 in 2022-09-08T04:02:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:10.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:10.845924+0000) 2022-09-08T04:02:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:10.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:10.826013+0000) 2022-09-08T04:02:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:11.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:11.846166+0000) 2022-09-08T04:02:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:11.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:11.826284+0000) 2022-09-08T04:02:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:12 smithi055 ceph-mon[97033]: pgmap v12416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:02:13.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:12 smithi059 ceph-mon[97450]: pgmap v12416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:02:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:12.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:12.846353+0000) 2022-09-08T04:02:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:12.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:12.826513+0000) 2022-09-08T04:02:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:13.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:13.826699+0000) 2022-09-08T04:02:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:13.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:13.846523+0000) 2022-09-08T04:02:14.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:02:14.616+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:02:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:14.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:14.846791+0000) 2022-09-08T04:02:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:14.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:14.826844+0000) 2022-09-08T04:02:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:14 smithi055 ceph-mon[97033]: pgmap v12417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:02:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:14 smithi059 ceph-mon[97450]: pgmap v12417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:02:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:15.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:15.846977+0000) 2022-09-08T04:02:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:15.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:15.826981+0000) 2022-09-08T04:02:16.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:16 smithi055 ceph-mon[97033]: pgmap v12418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:02:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:16 smithi059 ceph-mon[97450]: pgmap v12418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:02:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:16.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:16.847168+0000) 2022-09-08T04:02:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:16.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:16.827229+0000) 2022-09-08T04:02:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:17.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:17.847392+0000) 2022-09-08T04:02:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:17.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:17.827463+0000) 2022-09-08T04:02:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:18 smithi055 ceph-mon[97033]: pgmap v12419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:02:19.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:18 smithi059 ceph-mon[97450]: pgmap v12419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:02:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:18.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:18.827730+0000) 2022-09-08T04:02:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:18.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:18.847512+0000) 2022-09-08T04:02:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:19.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:19.827960+0000) 2022-09-08T04:02:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:19.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:19.847760+0000) 2022-09-08T04:02:20.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:20 smithi059 ceph-mon[97450]: osdmap e857: 12 total, 11 up, 11 in 2022-09-08T04:02:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:20 smithi059 ceph-mon[97450]: pgmap v12421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T04:02:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:20 smithi055 ceph-mon[97033]: osdmap e857: 12 total, 11 up, 11 in 2022-09-08T04:02:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:20 smithi055 ceph-mon[97033]: pgmap v12421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T04:02:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:20.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:20.847978+0000) 2022-09-08T04:02:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:20.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:20.828208+0000) 2022-09-08T04:02:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:21.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:21.848217+0000) 2022-09-08T04:02:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:21.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:21.828427+0000) 2022-09-08T04:02:22.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:22 smithi059 ceph-mon[97450]: pgmap v12422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:02:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:22 smithi055 ceph-mon[97033]: pgmap v12422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:02:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:22.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:22.848446+0000) 2022-09-08T04:02:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:22.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:22.828682+0000) 2022-09-08T04:02:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:23.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:23.848658+0000) 2022-09-08T04:02:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:23.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:23.828903+0000) 2022-09-08T04:02:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:24 smithi055 ceph-mon[97033]: pgmap v12423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:02:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:24 smithi059 ceph-mon[97450]: pgmap v12423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:02:25.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:24.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:24.829113+0000) 2022-09-08T04:02:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:24.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:24.848981+0000) 2022-09-08T04:02:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:25.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:25.849160+0000) 2022-09-08T04:02:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:25.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:25.829330+0000) 2022-09-08T04:02:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:26 smithi055 ceph-mon[97033]: pgmap v12424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:02:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:26 smithi059 ceph-mon[97450]: pgmap v12424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:02:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:26.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:26.849429+0000) 2022-09-08T04:02:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:26.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:26.829471+0000) 2022-09-08T04:02:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:27.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:27.849716+0000) 2022-09-08T04:02:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:27.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:27.829706+0000) 2022-09-08T04:02:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:28 smithi055 ceph-mon[97033]: pgmap v12425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:02:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:28 smithi059 ceph-mon[97450]: pgmap v12425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:02:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:28.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:28.849969+0000) 2022-09-08T04:02:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:28.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:28.829845+0000) 2022-09-08T04:02:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:29.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:29.850129+0000) 2022-09-08T04:02:29.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:29.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:29.830009+0000) 2022-09-08T04:02:29.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:02:29.624+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:02:30.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:30 smithi059 ceph-mon[97450]: pgmap v12426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:02:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:30 smithi055 ceph-mon[97033]: pgmap v12426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:02:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:30.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:30.850342+0000) 2022-09-08T04:02:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:30.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:30.830169+0000) 2022-09-08T04:02:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:31.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:31.830370+0000) 2022-09-08T04:02:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:31.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:31.850602+0000) 2022-09-08T04:02:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:32 smithi055 ceph-mon[97033]: pgmap v12427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:02:33.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:32 smithi059 ceph-mon[97450]: pgmap v12427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:02:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:32.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:32.850948+0000) 2022-09-08T04:02:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:32.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:32.830565+0000) 2022-09-08T04:02:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:02:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:02:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:02:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:02:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:02:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:02:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:33.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:33.851158+0000) 2022-09-08T04:02:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:33.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:33.830734+0000) 2022-09-08T04:02:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:34 smithi055 ceph-mon[97033]: pgmap v12428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:34.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:02:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:34 smithi059 ceph-mon[97450]: pgmap v12428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:02:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:34.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:34.851378+0000) 2022-09-08T04:02:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:34.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:34.830912+0000) 2022-09-08T04:02:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:35.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:35.851542+0000) 2022-09-08T04:02:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:35.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:35.831050+0000) 2022-09-08T04:02:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:36 smithi055 ceph-mon[97033]: pgmap v12429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:02:37.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:36 smithi059 ceph-mon[97450]: pgmap v12429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:02:37.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:36.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:36.851769+0000) 2022-09-08T04:02:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:36.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:36.831274+0000) 2022-09-08T04:02:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:37.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:37.851931+0000) 2022-09-08T04:02:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:37.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:37.831508+0000) 2022-09-08T04:02:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:38 smithi055 ceph-mon[97033]: pgmap v12430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:39.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:38 smithi059 ceph-mon[97450]: pgmap v12430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:38.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:38.852166+0000) 2022-09-08T04:02:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:38.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:38.831736+0000) 2022-09-08T04:02:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:39.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:39.852378+0000) 2022-09-08T04:02:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:39.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:39.831983+0000) 2022-09-08T04:02:40.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:40 smithi059 ceph-mon[97450]: pgmap v12431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:40 smithi055 ceph-mon[97033]: pgmap v12431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:40.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:40.852587+0000) 2022-09-08T04:02:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:40.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:40.832235+0000) 2022-09-08T04:02:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:41.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:41.852780+0000) 2022-09-08T04:02:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:41.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:41.832505+0000) 2022-09-08T04:02:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:42 smithi055 ceph-mon[97033]: pgmap v12432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:02:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:42 smithi059 ceph-mon[97450]: pgmap v12432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:02:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:42.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:42.852959+0000) 2022-09-08T04:02:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:42.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:42.832733+0000) 2022-09-08T04:02:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:43.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:43.853225+0000) 2022-09-08T04:02:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:43.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:43.832954+0000) 2022-09-08T04:02:44.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:02:44.626+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:02:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:44.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:44.833121+0000) 2022-09-08T04:02:44.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:44.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:44.853486+0000) 2022-09-08T04:02:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:44 smithi055 ceph-mon[97033]: pgmap v12433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:45.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:44 smithi059 ceph-mon[97450]: pgmap v12433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:46.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:45.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:45.853733+0000) 2022-09-08T04:02:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:45.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:45.833349+0000) 2022-09-08T04:02:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:46 smithi055 ceph-mon[97033]: pgmap v12434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:02:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:46 smithi059 ceph-mon[97450]: pgmap v12434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:02:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:46.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:46.853921+0000) 2022-09-08T04:02:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:46.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:46.833586+0000) 2022-09-08T04:02:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:47.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:47.854089+0000) 2022-09-08T04:02:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:47.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:47.833859+0000) 2022-09-08T04:02:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:48 smithi055 ceph-mon[97033]: pgmap v12435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:48 smithi059 ceph-mon[97450]: pgmap v12435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:48.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:48.834141+0000) 2022-09-08T04:02:49.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:48.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:48.854335+0000) 2022-09-08T04:02:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:49.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:49.854588+0000) 2022-09-08T04:02:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:49.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:49.834334+0000) 2022-09-08T04:02:50.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:50 smithi059 ceph-mon[97450]: pgmap v12436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:50 smithi055 ceph-mon[97033]: pgmap v12436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:50.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:50.854812+0000) 2022-09-08T04:02:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:50.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:50.834526+0000) 2022-09-08T04:02:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:51.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:51.834749+0000) 2022-09-08T04:02:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:51.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:51.855082+0000) 2022-09-08T04:02:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:52 smithi055 ceph-mon[97033]: pgmap v12437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:02:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:52 smithi059 ceph-mon[97450]: pgmap v12437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:02:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:52.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:52.855344+0000) 2022-09-08T04:02:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:52.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:52.834918+0000) 2022-09-08T04:02:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:53.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:53.855556+0000) 2022-09-08T04:02:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:53.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:53.835084+0000) 2022-09-08T04:02:54.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:54 smithi055 ceph-mon[97033]: pgmap v12438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:54 smithi059 ceph-mon[97450]: pgmap v12438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:54.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:54.855774+0000) 2022-09-08T04:02:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:54.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:54.835255+0000) 2022-09-08T04:02:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:55.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:55.855932+0000) 2022-09-08T04:02:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:55.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:55.835487+0000) 2022-09-08T04:02:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:56 smithi055 ceph-mon[97033]: pgmap v12439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:02:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:56 smithi059 ceph-mon[97450]: pgmap v12439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:02:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:56.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:56.856190+0000) 2022-09-08T04:02:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:56.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:56.835689+0000) 2022-09-08T04:02:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:57.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:57.856398+0000) 2022-09-08T04:02:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:57.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:57.835835+0000) 2022-09-08T04:02:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:02:58 smithi055 ceph-mon[97033]: pgmap v12440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:02:58 smithi059 ceph-mon[97450]: pgmap v12440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:02:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:58.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:58.856656+0000) 2022-09-08T04:02:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:58.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:58.836053+0000) 2022-09-08T04:02:59.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:02:59.639+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:02:59.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:02:59.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:59.856846+0000) 2022-09-08T04:02:59.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:02:59.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:02:59.836283+0000) 2022-09-08T04:03:00.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:00 smithi059 ceph-mon[97450]: pgmap v12441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:00 smithi055 ceph-mon[97033]: pgmap v12441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:00.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:00.856985+0000) 2022-09-08T04:03:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:00.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:00.836488+0000) 2022-09-08T04:03:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:01.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:01.857209+0000) 2022-09-08T04:03:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:01.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:01.836748+0000) 2022-09-08T04:03:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:02 smithi055 ceph-mon[97033]: pgmap v12442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:03:03.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:02 smithi059 ceph-mon[97450]: pgmap v12442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:03:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:02.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:02.857444+0000) 2022-09-08T04:03:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:02.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:02.836935+0000) 2022-09-08T04:03:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:03.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:03.857695+0000) 2022-09-08T04:03:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:03.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:03.837161+0000) 2022-09-08T04:03:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:04 smithi055 ceph-mon[97033]: pgmap v12443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:05.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:04 smithi059 ceph-mon[97450]: pgmap v12443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:04.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:04.857856+0000) 2022-09-08T04:03:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:04.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:04.837393+0000) 2022-09-08T04:03:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:03:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:03:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:05.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:05.858059+0000) 2022-09-08T04:03:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:05.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:05.837694+0000) 2022-09-08T04:03:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:06 smithi055 ceph-mon[97033]: pgmap v12444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:03:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:03:07.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:06 smithi059 ceph-mon[97450]: pgmap v12444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:03:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:03:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:06.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:06.858284+0000) 2022-09-08T04:03:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:06.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:06.837953+0000) 2022-09-08T04:03:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:07.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:07.858497+0000) 2022-09-08T04:03:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:07.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:07.838169+0000) 2022-09-08T04:03:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:08 smithi055 ceph-mon[97033]: pgmap v12445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:03:09.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:08 smithi059 ceph-mon[97450]: pgmap v12445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:03:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:08.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:08.858748+0000) 2022-09-08T04:03:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:08.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:08.838438+0000) 2022-09-08T04:03:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:09.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:09.838667+0000) 2022-09-08T04:03:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:09.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:09.858886+0000) 2022-09-08T04:03:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:10 smithi055 ceph-mon[97033]: pgmap v12446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:03:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:10 smithi055 ceph-mon[97033]: osdmap e858: 12 total, 11 up, 11 in 2022-09-08T04:03:11.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:10 smithi059 ceph-mon[97450]: pgmap v12446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:03:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:10 smithi059 ceph-mon[97450]: osdmap e858: 12 total, 11 up, 11 in 2022-09-08T04:03:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:10.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:10.838893+0000) 2022-09-08T04:03:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:10.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:10.859114+0000) 2022-09-08T04:03:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:11.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:11.859375+0000) 2022-09-08T04:03:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:11.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:11.839080+0000) 2022-09-08T04:03:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:12 smithi055 ceph-mon[97033]: pgmap v12448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:03:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:12 smithi059 ceph-mon[97450]: pgmap v12448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:03:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:12.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:12.859593+0000) 2022-09-08T04:03:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:12.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:12.839333+0000) 2022-09-08T04:03:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:13.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:13.859836+0000) 2022-09-08T04:03:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:13.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:13.839583+0000) 2022-09-08T04:03:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:14.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:14.860066+0000) 2022-09-08T04:03:14.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:03:14.640+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:03:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:14.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:14.839856+0000) 2022-09-08T04:03:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:14 smithi055 ceph-mon[97033]: pgmap v12449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:03:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:14 smithi059 ceph-mon[97450]: pgmap v12449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:03:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:15.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:15.860322+0000) 2022-09-08T04:03:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:15.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:15.840049+0000) 2022-09-08T04:03:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:16 smithi055 ceph-mon[97033]: pgmap v12450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:03:17.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:16 smithi059 ceph-mon[97450]: pgmap v12450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:03:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:16.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:16.860495+0000) 2022-09-08T04:03:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:16.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:16.840279+0000) 2022-09-08T04:03:18.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:17.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:17.860734+0000) 2022-09-08T04:03:18.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:17.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:17.840515+0000) 2022-09-08T04:03:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:18 smithi055 ceph-mon[97033]: pgmap v12451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:03:19.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:18 smithi059 ceph-mon[97450]: pgmap v12451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:03:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:18.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:18.860947+0000) 2022-09-08T04:03:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:18.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:18.840773+0000) 2022-09-08T04:03:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:19.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:19.861091+0000) 2022-09-08T04:03:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:19.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:19.840954+0000) 2022-09-08T04:03:20.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:20 smithi059 ceph-mon[97450]: osdmap e859: 12 total, 11 up, 11 in 2022-09-08T04:03:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:20 smithi059 ceph-mon[97450]: pgmap v12453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T04:03:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:20 smithi055 ceph-mon[97033]: osdmap e859: 12 total, 11 up, 11 in 2022-09-08T04:03:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:20 smithi055 ceph-mon[97033]: pgmap v12453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T04:03:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:20.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:20.861279+0000) 2022-09-08T04:03:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:20.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:20.841171+0000) 2022-09-08T04:03:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:21.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:21.861462+0000) 2022-09-08T04:03:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:21.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:21.841440+0000) 2022-09-08T04:03:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:22 smithi055 ceph-mon[97033]: pgmap v12454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:03:23.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:22 smithi059 ceph-mon[97450]: pgmap v12454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:03:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:22.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:22.861692+0000) 2022-09-08T04:03:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:22.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:22.841699+0000) 2022-09-08T04:03:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:23.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:23.841934+0000) 2022-09-08T04:03:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:23.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:23.861896+0000) 2022-09-08T04:03:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:24 smithi055 ceph-mon[97033]: pgmap v12455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:03:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:24 smithi059 ceph-mon[97450]: pgmap v12455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:03:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:24.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:24.862064+0000) 2022-09-08T04:03:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:24.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:24.842180+0000) 2022-09-08T04:03:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:25.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:25.862317+0000) 2022-09-08T04:03:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:25.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:25.842431+0000) 2022-09-08T04:03:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:26 smithi055 ceph-mon[97033]: pgmap v12456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:03:27.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:26 smithi059 ceph-mon[97450]: pgmap v12456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:03:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:26.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:26.862566+0000) 2022-09-08T04:03:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:26.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:26.842716+0000) 2022-09-08T04:03:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:27.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:27.862723+0000) 2022-09-08T04:03:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:27.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:27.842973+0000) 2022-09-08T04:03:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:28 smithi055 ceph-mon[97033]: pgmap v12457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:03:29.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:28 smithi059 ceph-mon[97450]: pgmap v12457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:03:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:28.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:28.862924+0000) 2022-09-08T04:03:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:28.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:28.843249+0000) 2022-09-08T04:03:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:29.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:29.843512+0000) 2022-09-08T04:03:29.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:03:29.648+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:03:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:29.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:29.863051+0000) 2022-09-08T04:03:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:30 smithi059 ceph-mon[97450]: pgmap v12458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:03:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:30 smithi055 ceph-mon[97033]: pgmap v12458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:03:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:30.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:30.863265+0000) 2022-09-08T04:03:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:30.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:30.843766+0000) 2022-09-08T04:03:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:31.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:31.844036+0000) 2022-09-08T04:03:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:31.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:31.863492+0000) 2022-09-08T04:03:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:32 smithi055 ceph-mon[97033]: pgmap v12459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:03:33.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:32 smithi059 ceph-mon[97450]: pgmap v12459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:03:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:32.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:32.863704+0000) 2022-09-08T04:03:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:32.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:32.844239+0000) 2022-09-08T04:03:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:33.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:33.844459+0000) 2022-09-08T04:03:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:33.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:33.863948+0000) 2022-09-08T04:03:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:34 smithi055 ceph-mon[97033]: pgmap v12460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:03:34.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:03:34.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:03:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:03:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:34 smithi059 ceph-mon[97450]: pgmap v12460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:03:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:03:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:03:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:03:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:34.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:34.864179+0000) 2022-09-08T04:03:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:34.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:34.844622+0000) 2022-09-08T04:03:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:35.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:35.864442+0000) 2022-09-08T04:03:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:35.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:35.844840+0000) 2022-09-08T04:03:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:36 smithi055 ceph-mon[97033]: pgmap v12461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:03:37.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:36 smithi059 ceph-mon[97450]: pgmap v12461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:03:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:36.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:36.864611+0000) 2022-09-08T04:03:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:36.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:36.845066+0000) 2022-09-08T04:03:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:37.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:37.864904+0000) 2022-09-08T04:03:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:37.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:37.845264+0000) 2022-09-08T04:03:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:38 smithi055 ceph-mon[97033]: pgmap v12462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:38 smithi059 ceph-mon[97450]: pgmap v12462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:38.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:38.865126+0000) 2022-09-08T04:03:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:38.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:38.845527+0000) 2022-09-08T04:03:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:39.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:39.865340+0000) 2022-09-08T04:03:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:39.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:39.845784+0000) 2022-09-08T04:03:40.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:40 smithi059 ceph-mon[97450]: pgmap v12463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:40 smithi055 ceph-mon[97033]: pgmap v12463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:40.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:40.865540+0000) 2022-09-08T04:03:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:40.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:40.845981+0000) 2022-09-08T04:03:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:41.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:41.865731+0000) 2022-09-08T04:03:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:41.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:41.846180+0000) 2022-09-08T04:03:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:42 smithi055 ceph-mon[97033]: pgmap v12464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:03:43.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:42 smithi059 ceph-mon[97450]: pgmap v12464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:03:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:42.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:42.865917+0000) 2022-09-08T04:03:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:42.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:42.846427+0000) 2022-09-08T04:03:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:43.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:43.866138+0000) 2022-09-08T04:03:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:43.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:43.846725+0000) 2022-09-08T04:03:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:44.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:44.866301+0000) 2022-09-08T04:03:44.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:44.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:44.846919+0000) 2022-09-08T04:03:44.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:03:44.649+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:03:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:44 smithi055 ceph-mon[97033]: pgmap v12465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:45.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:44 smithi059 ceph-mon[97450]: pgmap v12465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:45.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:45.866486+0000) 2022-09-08T04:03:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:45.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:45.847097+0000) 2022-09-08T04:03:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:46 smithi055 ceph-mon[97033]: pgmap v12466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:03:47.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:46 smithi059 ceph-mon[97450]: pgmap v12466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:03:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:46.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:46.847349+0000) 2022-09-08T04:03:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:46.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:46.866680+0000) 2022-09-08T04:03:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:47.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:47.866879+0000) 2022-09-08T04:03:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:47.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:47.847518+0000) 2022-09-08T04:03:48.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:48 smithi055 ceph-mon[97033]: pgmap v12467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:49.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:48 smithi059 ceph-mon[97450]: pgmap v12467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:48.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:48.867000+0000) 2022-09-08T04:03:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:48.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:48.847747+0000) 2022-09-08T04:03:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:49.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:49.867228+0000) 2022-09-08T04:03:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:49.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:49.847995+0000) 2022-09-08T04:03:50.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:50 smithi059 ceph-mon[97450]: pgmap v12468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:50 smithi055 ceph-mon[97033]: pgmap v12468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:50.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:50.867466+0000) 2022-09-08T04:03:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:50.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:50.848220+0000) 2022-09-08T04:03:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:51.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:51.867768+0000) 2022-09-08T04:03:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:51.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:51.848437+0000) 2022-09-08T04:03:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:52 smithi055 ceph-mon[97033]: pgmap v12469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:03:53.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:52 smithi059 ceph-mon[97450]: pgmap v12469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:03:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:52.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:52.867964+0000) 2022-09-08T04:03:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:52.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:52.848745+0000) 2022-09-08T04:03:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:53.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:53.868200+0000) 2022-09-08T04:03:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:53.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:53.849016+0000) 2022-09-08T04:03:54.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:54 smithi055 ceph-mon[97033]: pgmap v12470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:54 smithi059 ceph-mon[97450]: pgmap v12470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:54.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:54.868394+0000) 2022-09-08T04:03:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:54.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:54.849172+0000) 2022-09-08T04:03:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:55.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:55.868664+0000) 2022-09-08T04:03:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:55.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:55.849414+0000) 2022-09-08T04:03:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:56 smithi055 ceph-mon[97033]: pgmap v12471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:03:57.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:56 smithi059 ceph-mon[97450]: pgmap v12471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:03:57.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:56.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:56.868810+0000) 2022-09-08T04:03:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:56.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:56.849734+0000) 2022-09-08T04:03:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:57.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:57.869007+0000) 2022-09-08T04:03:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:57.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:57.849919+0000) 2022-09-08T04:03:58.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:03:58 smithi055 ceph-mon[97033]: pgmap v12472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:59.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:03:58 smithi059 ceph-mon[97450]: pgmap v12472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:03:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:58.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:58.869249+0000) 2022-09-08T04:03:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:58.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:58.850167+0000) 2022-09-08T04:04:00.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:03:59.661+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:04:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:03:59.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:59.850399+0000) 2022-09-08T04:04:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:03:59.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:03:59.869450+0000) 2022-09-08T04:04:00.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:00 smithi059 ceph-mon[97450]: pgmap v12473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:00 smithi055 ceph-mon[97033]: pgmap v12473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:00.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:00.869712+0000) 2022-09-08T04:04:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:00.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:00.850691+0000) 2022-09-08T04:04:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:01.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:01.869866+0000) 2022-09-08T04:04:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:01.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:01.850907+0000) 2022-09-08T04:04:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:02 smithi055 ceph-mon[97033]: pgmap v12474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:04:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:02 smithi059 ceph-mon[97450]: pgmap v12474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:04:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:02.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:02.870019+0000) 2022-09-08T04:04:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:02.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:02.851170+0000) 2022-09-08T04:04:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:03.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:03.870232+0000) 2022-09-08T04:04:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:03.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:03.851389+0000) 2022-09-08T04:04:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:04 smithi055 ceph-mon[97033]: pgmap v12475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:05.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:04 smithi059 ceph-mon[97450]: pgmap v12475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:04.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:04.870465+0000) 2022-09-08T04:04:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:04.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:04.851563+0000) 2022-09-08T04:04:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:04:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:04:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:05.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:05.870753+0000) 2022-09-08T04:04:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:05.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:05.851789+0000) 2022-09-08T04:04:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:06 smithi055 ceph-mon[97033]: pgmap v12476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:04:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:04:07.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:06 smithi059 ceph-mon[97450]: pgmap v12476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:04:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:04:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:06.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:06.870918+0000) 2022-09-08T04:04:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:06.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:06.852036+0000) 2022-09-08T04:04:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:07.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:07.852295+0000) 2022-09-08T04:04:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:07.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:07.871144+0000) 2022-09-08T04:04:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:08 smithi055 ceph-mon[97033]: pgmap v12477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:04:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:08 smithi059 ceph-mon[97450]: pgmap v12477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:04:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:08.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:08.871335+0000) 2022-09-08T04:04:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:08.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:08.852578+0000) 2022-09-08T04:04:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:09.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:09.852821+0000) 2022-09-08T04:04:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:09.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:09.871519+0000) 2022-09-08T04:04:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:10 smithi055 ceph-mon[97033]: pgmap v12478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:04:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:10 smithi055 ceph-mon[97033]: osdmap e860: 12 total, 11 up, 11 in 2022-09-08T04:04:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:10 smithi059 ceph-mon[97450]: pgmap v12478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:04:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:10 smithi059 ceph-mon[97450]: osdmap e860: 12 total, 11 up, 11 in 2022-09-08T04:04:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:10.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:10.871742+0000) 2022-09-08T04:04:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:10.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:10.853107+0000) 2022-09-08T04:04:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:11.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:11.871867+0000) 2022-09-08T04:04:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:11.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:11.853345+0000) 2022-09-08T04:04:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:12 smithi055 ceph-mon[97033]: pgmap v12480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:04:13.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:12 smithi059 ceph-mon[97450]: pgmap v12480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:04:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:12.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:12.872044+0000) 2022-09-08T04:04:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:12.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:12.853489+0000) 2022-09-08T04:04:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:13.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:13.872225+0000) 2022-09-08T04:04:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:13.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:13.853734+0000) 2022-09-08T04:04:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:14 smithi055 ceph-mon[97033]: pgmap v12481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:04:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:14 smithi059 ceph-mon[97450]: pgmap v12481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:04:15.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:04:14.662+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:04:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:14.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:14.872391+0000) 2022-09-08T04:04:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:14.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:14.853918+0000) 2022-09-08T04:04:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:15.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:15.872600+0000) 2022-09-08T04:04:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:15.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:15.854115+0000) 2022-09-08T04:04:16.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:16 smithi055 ceph-mon[97033]: pgmap v12482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:04:17.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:16 smithi059 ceph-mon[97450]: pgmap v12482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:04:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:16.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:16.872882+0000) 2022-09-08T04:04:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:16.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:16.854347+0000) 2022-09-08T04:04:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:17.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:17.873138+0000) 2022-09-08T04:04:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:17.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:17.854613+0000) 2022-09-08T04:04:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:18 smithi055 ceph-mon[97033]: pgmap v12483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:04:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:18 smithi059 ceph-mon[97450]: pgmap v12483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:04:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:18.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:18.873297+0000) 2022-09-08T04:04:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:18.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:18.854902+0000) 2022-09-08T04:04:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:19.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:19.873533+0000) 2022-09-08T04:04:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:19.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:19.855075+0000) 2022-09-08T04:04:20.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:20 smithi059 ceph-mon[97450]: osdmap e861: 12 total, 11 up, 11 in 2022-09-08T04:04:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:20 smithi059 ceph-mon[97450]: pgmap v12485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T04:04:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:20 smithi055 ceph-mon[97033]: osdmap e861: 12 total, 11 up, 11 in 2022-09-08T04:04:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:20 smithi055 ceph-mon[97033]: pgmap v12485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T04:04:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:20.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:20.855301+0000) 2022-09-08T04:04:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:20.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:20.873755+0000) 2022-09-08T04:04:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:21.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:21.855572+0000) 2022-09-08T04:04:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:21.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:21.873953+0000) 2022-09-08T04:04:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:22 smithi055 ceph-mon[97033]: pgmap v12486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:04:23.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:22 smithi059 ceph-mon[97450]: pgmap v12486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:04:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:22.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:22.855811+0000) 2022-09-08T04:04:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:22.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:22.874164+0000) 2022-09-08T04:04:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:23.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:23.856079+0000) 2022-09-08T04:04:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:23.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:23.874311+0000) 2022-09-08T04:04:24.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:24 smithi055 ceph-mon[97033]: pgmap v12487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:04:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:24 smithi059 ceph-mon[97450]: pgmap v12487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:04:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:24.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:24.856281+0000) 2022-09-08T04:04:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:24.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:24.874551+0000) 2022-09-08T04:04:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:25.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:25.874741+0000) 2022-09-08T04:04:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:25.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:25.856533+0000) 2022-09-08T04:04:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:26 smithi055 ceph-mon[97033]: pgmap v12488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:04:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:26 smithi059 ceph-mon[97450]: pgmap v12488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:04:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:26.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:26.874975+0000) 2022-09-08T04:04:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:26.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:26.856791+0000) 2022-09-08T04:04:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:27.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:27.875177+0000) 2022-09-08T04:04:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:27.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:27.856998+0000) 2022-09-08T04:04:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:28 smithi055 ceph-mon[97033]: pgmap v12489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:04:29.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:28 smithi059 ceph-mon[97450]: pgmap v12489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:04:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:28.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:28.875383+0000) 2022-09-08T04:04:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:28.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:28.857254+0000) 2022-09-08T04:04:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:29.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:29.875632+0000) 2022-09-08T04:04:30.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:04:29.671+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:04:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:29.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:29.857516+0000) 2022-09-08T04:04:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:30 smithi055 ceph-mon[97033]: pgmap v12490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:04:30.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:30 smithi059 ceph-mon[97450]: pgmap v12490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:04:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:30.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:30.875839+0000) 2022-09-08T04:04:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:30.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:30.857783+0000) 2022-09-08T04:04:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:31.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:31.876066+0000) 2022-09-08T04:04:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:31.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:31.857959+0000) 2022-09-08T04:04:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:32 smithi055 ceph-mon[97033]: pgmap v12491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:04:33.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:32 smithi059 ceph-mon[97450]: pgmap v12491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:04:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:32.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:32.876313+0000) 2022-09-08T04:04:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:32.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:32.858195+0000) 2022-09-08T04:04:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:33.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:33.876480+0000) 2022-09-08T04:04:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:33.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:33.858475+0000) 2022-09-08T04:04:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:34 smithi055 ceph-mon[97033]: pgmap v12492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:34 smithi059 ceph-mon[97450]: pgmap v12492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:34.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:34.876664+0000) 2022-09-08T04:04:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:34.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:34.858676+0000) 2022-09-08T04:04:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:04:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:04:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:04:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:04:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:04:36.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:04:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:04:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:04:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:04:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:04:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:35.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:35.876865+0000) 2022-09-08T04:04:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:35.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:35.858915+0000) 2022-09-08T04:04:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:36 smithi055 ceph-mon[97033]: pgmap v12493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:04:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:36 smithi059 ceph-mon[97450]: pgmap v12493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:04:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:36.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:36.877035+0000) 2022-09-08T04:04:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:36.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:36.859172+0000) 2022-09-08T04:04:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:37.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:37.877240+0000) 2022-09-08T04:04:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:37.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:37.859331+0000) 2022-09-08T04:04:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:04:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:04:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:38 smithi055 ceph-mon[97033]: pgmap v12494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:04:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:04:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:04:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:04:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:04:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:38 smithi059 ceph-mon[97450]: pgmap v12494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:04:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:04:38.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:04:38.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:04:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:38.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:38.877472+0000) 2022-09-08T04:04:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:38.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:38.859599+0000) 2022-09-08T04:04:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:39 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T04:04:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:39 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T04:04:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:39 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T04:04:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:39 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T04:04:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:39 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T04:04:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:39 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T04:04:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:39 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T04:04:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:39 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T04:04:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:39.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:39.877701+0000) 2022-09-08T04:04:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:39.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:39.859802+0000) 2022-09-08T04:04:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:40 smithi059 ceph-mon[97450]: pgmap v12495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:40 smithi055 ceph-mon[97033]: pgmap v12495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:40.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:40.860038+0000) 2022-09-08T04:04:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:40.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:40.877951+0000) 2022-09-08T04:04:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:41.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:41.878129+0000) 2022-09-08T04:04:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:41.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:41.860304+0000) 2022-09-08T04:04:42.792 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:42 smithi059 ceph-mon[97450]: pgmap v12496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:04:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:42 smithi055 ceph-mon[97033]: pgmap v12496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:04:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:42.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:42.878370+0000) 2022-09-08T04:04:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:42.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:42.860533+0000) 2022-09-08T04:04:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:43.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:43.878604+0000) 2022-09-08T04:04:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:43.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:43.860731+0000) 2022-09-08T04:04:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:44 smithi055 ceph-mon[97033]: pgmap v12497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:45.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:44 smithi059 ceph-mon[97450]: pgmap v12497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:45.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:04:44.680+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:04:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:44.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:44.878764+0000) 2022-09-08T04:04:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:44.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:44.860927+0000) 2022-09-08T04:04:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:45.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:45.878933+0000) 2022-09-08T04:04:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:45.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:45.861166+0000) 2022-09-08T04:04:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:46 smithi055 ceph-mon[97033]: pgmap v12498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:04:47.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:46 smithi059 ceph-mon[97450]: pgmap v12498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:04:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:46.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:46.879174+0000) 2022-09-08T04:04:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:46.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:46.861398+0000) 2022-09-08T04:04:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:47.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:47.879439+0000) 2022-09-08T04:04:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:47.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:47.861659+0000) 2022-09-08T04:04:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:48 smithi055 ceph-mon[97033]: pgmap v12499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:49.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:48 smithi059 ceph-mon[97450]: pgmap v12499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:48.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:48.879693+0000) 2022-09-08T04:04:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:48.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:48.861813+0000) 2022-09-08T04:04:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:49.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:49.879946+0000) 2022-09-08T04:04:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:49.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:49.862013+0000) 2022-09-08T04:04:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:50 smithi055 ceph-mon[97033]: pgmap v12500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:50 smithi059 ceph-mon[97450]: pgmap v12500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:50.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:50.880179+0000) 2022-09-08T04:04:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:50.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:50.862242+0000) 2022-09-08T04:04:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:51.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:51.880326+0000) 2022-09-08T04:04:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:51.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:51.862438+0000) 2022-09-08T04:04:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:52 smithi055 ceph-mon[97033]: pgmap v12501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:04:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:52 smithi059 ceph-mon[97450]: pgmap v12501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:04:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:52.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:52.880514+0000) 2022-09-08T04:04:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:52.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:52.862695+0000) 2022-09-08T04:04:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:53.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:53.880737+0000) 2022-09-08T04:04:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:53.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:53.862865+0000) 2022-09-08T04:04:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:54 smithi055 ceph-mon[97033]: pgmap v12502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:54 smithi059 ceph-mon[97450]: pgmap v12502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:54.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:54.880910+0000) 2022-09-08T04:04:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:54.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:54.862996+0000) 2022-09-08T04:04:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:55.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:55.881085+0000) 2022-09-08T04:04:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:55.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:55.863237+0000) 2022-09-08T04:04:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:56 smithi055 ceph-mon[97033]: pgmap v12503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:04:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:56 smithi059 ceph-mon[97450]: pgmap v12503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:04:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:56.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:56.881271+0000) 2022-09-08T04:04:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:56.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:56.863467+0000) 2022-09-08T04:04:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:57.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:57.881498+0000) 2022-09-08T04:04:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:57.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:57.863605+0000) 2022-09-08T04:04:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:04:58 smithi055 ceph-mon[97033]: pgmap v12504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:59.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:04:58 smithi059 ceph-mon[97450]: pgmap v12504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:04:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:58.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:58.881753+0000) 2022-09-08T04:04:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:58.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:58.863842+0000) 2022-09-08T04:05:00.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:04:59.683+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:05:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:04:59.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:59.881998+0000) 2022-09-08T04:05:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:04:59.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:04:59.863994+0000) 2022-09-08T04:05:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:00 smithi055 ceph-mon[97033]: pgmap v12505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:00.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:00 smithi059 ceph-mon[97450]: pgmap v12505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:00.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:00.882217+0000) 2022-09-08T04:05:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:00.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:00.864220+0000) 2022-09-08T04:05:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:01.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:01.882452+0000) 2022-09-08T04:05:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:01.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:01.864463+0000) 2022-09-08T04:05:02.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:02 smithi055 ceph-mon[97033]: pgmap v12506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:05:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:02 smithi059 ceph-mon[97450]: pgmap v12506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:05:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:02.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:02.882594+0000) 2022-09-08T04:05:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:02.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:02.864684+0000) 2022-09-08T04:05:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:03.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:03.882849+0000) 2022-09-08T04:05:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:03.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:03.864877+0000) 2022-09-08T04:05:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:04 smithi055 ceph-mon[97033]: pgmap v12507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:05.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:04 smithi059 ceph-mon[97450]: pgmap v12507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:04.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:04.883014+0000) 2022-09-08T04:05:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:04.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:04.865058+0000) 2022-09-08T04:05:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:05:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:05:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:05.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:05.865278+0000) 2022-09-08T04:05:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:05.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:05.883225+0000) 2022-09-08T04:05:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:06 smithi055 ceph-mon[97033]: pgmap v12508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:05:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:05:07.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:06 smithi059 ceph-mon[97450]: pgmap v12508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:05:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:05:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:06.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:06.865520+0000) 2022-09-08T04:05:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:06.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:06.883393+0000) 2022-09-08T04:05:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:07.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:07.883667+0000) 2022-09-08T04:05:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:07.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:07.865745+0000) 2022-09-08T04:05:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:08 smithi055 ceph-mon[97033]: pgmap v12509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:05:09.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:08 smithi059 ceph-mon[97450]: pgmap v12509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:05:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:08.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:08.883856+0000) 2022-09-08T04:05:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:08.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:08.866015+0000) 2022-09-08T04:05:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:09.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:09.884022+0000) 2022-09-08T04:05:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:09.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:09.866284+0000) 2022-09-08T04:05:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:10 smithi055 ceph-mon[97033]: pgmap v12510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:05:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:10 smithi055 ceph-mon[97033]: osdmap e862: 12 total, 11 up, 11 in 2022-09-08T04:05:11.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:10 smithi059 ceph-mon[97450]: pgmap v12510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:05:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:10 smithi059 ceph-mon[97450]: osdmap e862: 12 total, 11 up, 11 in 2022-09-08T04:05:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:10.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:10.884237+0000) 2022-09-08T04:05:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:10.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:10.866538+0000) 2022-09-08T04:05:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:11.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:11.884447+0000) 2022-09-08T04:05:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:11.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:11.866782+0000) 2022-09-08T04:05:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:12 smithi055 ceph-mon[97033]: pgmap v12512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:05:13.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:12 smithi059 ceph-mon[97450]: pgmap v12512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:05:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:12.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:12.884766+0000) 2022-09-08T04:05:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:12.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:12.866972+0000) 2022-09-08T04:05:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:13.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:13.885029+0000) 2022-09-08T04:05:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:13.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:13.867165+0000) 2022-09-08T04:05:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:14 smithi055 ceph-mon[97033]: pgmap v12513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:05:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:14 smithi059 ceph-mon[97450]: pgmap v12513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:05:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:14.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:14.885266+0000) 2022-09-08T04:05:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:14.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:14.867370+0000) 2022-09-08T04:05:15.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:05:14.689+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:05:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:15.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:15.885477+0000) 2022-09-08T04:05:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:15.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:15.867667+0000) 2022-09-08T04:05:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:16 smithi055 ceph-mon[97033]: pgmap v12514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:05:17.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:16 smithi059 ceph-mon[97450]: pgmap v12514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:05:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:16.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:16.885750+0000) 2022-09-08T04:05:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:16.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:16.867918+0000) 2022-09-08T04:05:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:17.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:17.886010+0000) 2022-09-08T04:05:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:17.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:17.868134+0000) 2022-09-08T04:05:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:18 smithi055 ceph-mon[97033]: pgmap v12515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:05:19.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:18 smithi059 ceph-mon[97450]: pgmap v12515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:05:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:18.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:18.886234+0000) 2022-09-08T04:05:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:18.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:18.868387+0000) 2022-09-08T04:05:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:19.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:19.886472+0000) 2022-09-08T04:05:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:19.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:19.868613+0000) 2022-09-08T04:05:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:20 smithi055 ceph-mon[97033]: osdmap e863: 12 total, 11 up, 11 in 2022-09-08T04:05:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:20 smithi055 ceph-mon[97033]: pgmap v12517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T04:05:20.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:20 smithi059 ceph-mon[97450]: osdmap e863: 12 total, 11 up, 11 in 2022-09-08T04:05:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:20 smithi059 ceph-mon[97450]: pgmap v12517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T04:05:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:20.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:20.886630+0000) 2022-09-08T04:05:21.162 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:20.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:20.868870+0000) 2022-09-08T04:05:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:21.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:21.886866+0000) 2022-09-08T04:05:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:21.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:21.869014+0000) 2022-09-08T04:05:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:22 smithi055 ceph-mon[97033]: pgmap v12518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:05:23.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:22 smithi059 ceph-mon[97450]: pgmap v12518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:05:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:22.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:22.887091+0000) 2022-09-08T04:05:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:22.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:22.869239+0000) 2022-09-08T04:05:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:23.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:23.887334+0000) 2022-09-08T04:05:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:23.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:23.869479+0000) 2022-09-08T04:05:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:24 smithi055 ceph-mon[97033]: pgmap v12519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:05:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:24 smithi059 ceph-mon[97450]: pgmap v12519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:05:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:24.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:24.887569+0000) 2022-09-08T04:05:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:24.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:24.869651+0000) 2022-09-08T04:05:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:25.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:25.887771+0000) 2022-09-08T04:05:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:25.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:25.869865+0000) 2022-09-08T04:05:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:26 smithi055 ceph-mon[97033]: pgmap v12520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:05:27.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:26 smithi059 ceph-mon[97450]: pgmap v12520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:05:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:26.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:26.888026+0000) 2022-09-08T04:05:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:26.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:26.870057+0000) 2022-09-08T04:05:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:27.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:27.888270+0000) 2022-09-08T04:05:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:27.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:27.870285+0000) 2022-09-08T04:05:28.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:28 smithi055 ceph-mon[97033]: pgmap v12521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:05:29.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:28 smithi059 ceph-mon[97450]: pgmap v12521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:05:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:28.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:28.888533+0000) 2022-09-08T04:05:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:28.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:28.870529+0000) 2022-09-08T04:05:30.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:05:29.691+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:05:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:29.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:29.888780+0000) 2022-09-08T04:05:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:29.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:29.870734+0000) 2022-09-08T04:05:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:30 smithi055 ceph-mon[97033]: pgmap v12522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:05:30.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:30 smithi059 ceph-mon[97450]: pgmap v12522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:05:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:30.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:30.889030+0000) 2022-09-08T04:05:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:30.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:30.870959+0000) 2022-09-08T04:05:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:31.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:31.889226+0000) 2022-09-08T04:05:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:31.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:31.871109+0000) 2022-09-08T04:05:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:32 smithi055 ceph-mon[97033]: pgmap v12523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:05:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:32 smithi059 ceph-mon[97450]: pgmap v12523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:05:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:32.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:32.889461+0000) 2022-09-08T04:05:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:32.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:32.871307+0000) 2022-09-08T04:05:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:33.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:33.889735+0000) 2022-09-08T04:05:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:33.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:33.871557+0000) 2022-09-08T04:05:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:34 smithi055 ceph-mon[97033]: pgmap v12524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:34 smithi059 ceph-mon[97450]: pgmap v12524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:34.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:34.889886+0000) 2022-09-08T04:05:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:34.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:34.871710+0000) 2022-09-08T04:05:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:35.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:35.871907+0000) 2022-09-08T04:05:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:35.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:35.890059+0000) 2022-09-08T04:05:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:36 smithi055 ceph-mon[97033]: pgmap v12525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:05:37.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:36 smithi059 ceph-mon[97450]: pgmap v12525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:05:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:36.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:36.890173+0000) 2022-09-08T04:05:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:36.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:36.872127+0000) 2022-09-08T04:05:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:37.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:37.890376+0000) 2022-09-08T04:05:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:37.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:37.872370+0000) 2022-09-08T04:05:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:38 smithi059 ceph-mon[97450]: pgmap v12526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:05:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:05:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:05:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:38 smithi055 ceph-mon[97033]: pgmap v12526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:05:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:05:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:05:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:38.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:38.890540+0000) 2022-09-08T04:05:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:38.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:38.872599+0000) 2022-09-08T04:05:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:39.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:39.890746+0000) 2022-09-08T04:05:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:39.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:39.872707+0000) 2022-09-08T04:05:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:05:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:05:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:05:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:39 smithi055 ceph-mon[97033]: pgmap v12527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:40.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:05:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:05:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:05:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:39 smithi059 ceph-mon[97450]: pgmap v12527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:40.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:40.872915+0000) 2022-09-08T04:05:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:40.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:40.890995+0000) 2022-09-08T04:05:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:41.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:41.891175+0000) 2022-09-08T04:05:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:41.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:41.873135+0000) 2022-09-08T04:05:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:42 smithi055 ceph-mon[97033]: pgmap v12528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:05:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:42 smithi059 ceph-mon[97450]: pgmap v12528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:05:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:42.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:42.873365+0000) 2022-09-08T04:05:43.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:42.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:42.891417+0000) 2022-09-08T04:05:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:43.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:43.891700+0000) 2022-09-08T04:05:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:43.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:43.873537+0000) 2022-09-08T04:05:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:44 smithi055 ceph-mon[97033]: pgmap v12529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:45.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:44 smithi059 ceph-mon[97450]: pgmap v12529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:44.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:44.891936+0000) 2022-09-08T04:05:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:44.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:44.873799+0000) 2022-09-08T04:05:45.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:05:44.696+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:05:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:45.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:45.892171+0000) 2022-09-08T04:05:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:45.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:45.874011+0000) 2022-09-08T04:05:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:46 smithi055 ceph-mon[97033]: pgmap v12530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:05:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:46 smithi059 ceph-mon[97450]: pgmap v12530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:05:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:46.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:46.892376+0000) 2022-09-08T04:05:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:46.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:46.874241+0000) 2022-09-08T04:05:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:47.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:47.892602+0000) 2022-09-08T04:05:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:47.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:47.874425+0000) 2022-09-08T04:05:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:48 smithi055 ceph-mon[97033]: pgmap v12531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:48 smithi059 ceph-mon[97450]: pgmap v12531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:48.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:48.892904+0000) 2022-09-08T04:05:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:48.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:48.874689+0000) 2022-09-08T04:05:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:49.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:49.893142+0000) 2022-09-08T04:05:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:49.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:49.874886+0000) 2022-09-08T04:05:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:50 smithi055 ceph-mon[97033]: pgmap v12532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:50.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:50 smithi059 ceph-mon[97450]: pgmap v12532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:50.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:50.893373+0000) 2022-09-08T04:05:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:50.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:50.875094+0000) 2022-09-08T04:05:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:51.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:51.893599+0000) 2022-09-08T04:05:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:51.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:51.875314+0000) 2022-09-08T04:05:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:52 smithi055 ceph-mon[97033]: pgmap v12533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:05:53.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:52 smithi059 ceph-mon[97450]: pgmap v12533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:05:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:52.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:52.893803+0000) 2022-09-08T04:05:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:52.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:52.875544+0000) 2022-09-08T04:05:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:53.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:53.893999+0000) 2022-09-08T04:05:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:53.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:53.875794+0000) 2022-09-08T04:05:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:54 smithi055 ceph-mon[97033]: pgmap v12534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:54 smithi059 ceph-mon[97450]: pgmap v12534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:54.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:54.894174+0000) 2022-09-08T04:05:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:54.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:54.875984+0000) 2022-09-08T04:05:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:55.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:55.894410+0000) 2022-09-08T04:05:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:55.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:55.876218+0000) 2022-09-08T04:05:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:56 smithi055 ceph-mon[97033]: pgmap v12535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:05:57.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:56 smithi059 ceph-mon[97450]: pgmap v12535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:05:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:56.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:56.894673+0000) 2022-09-08T04:05:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:56.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:56.876450+0000) 2022-09-08T04:05:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:57.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:57.894841+0000) 2022-09-08T04:05:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:57.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:57.876600+0000) 2022-09-08T04:05:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:05:58 smithi055 ceph-mon[97033]: pgmap v12536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:59.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:05:58 smithi059 ceph-mon[97450]: pgmap v12536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:05:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:58.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:58.895122+0000) 2022-09-08T04:05:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:58.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:58.876854+0000) 2022-09-08T04:06:00.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:05:59.706+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:06:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:05:59.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:59.895383+0000) 2022-09-08T04:06:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:05:59.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:05:59.877039+0000) 2022-09-08T04:06:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:00 smithi055 ceph-mon[97033]: pgmap v12537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:00.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:00 smithi059 ceph-mon[97450]: pgmap v12537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:01.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:00.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:00.895564+0000) 2022-09-08T04:06:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:00.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:00.877221+0000) 2022-09-08T04:06:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:01.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:01.877429+0000) 2022-09-08T04:06:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:01.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:01.895818+0000) 2022-09-08T04:06:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:02 smithi055 ceph-mon[97033]: pgmap v12538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:06:03.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:02 smithi059 ceph-mon[97450]: pgmap v12538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:06:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:02.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:02.877586+0000) 2022-09-08T04:06:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:02.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:02.895979+0000) 2022-09-08T04:06:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:03.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:03.877779+0000) 2022-09-08T04:06:04.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:03.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:03.896112+0000) 2022-09-08T04:06:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:04 smithi055 ceph-mon[97033]: pgmap v12539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:04 smithi059 ceph-mon[97450]: pgmap v12539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:04.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:04.877959+0000) 2022-09-08T04:06:05.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:04.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:04.896258+0000) 2022-09-08T04:06:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:06:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:06:06.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:05.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:05.896467+0000) 2022-09-08T04:06:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:05.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:05.878233+0000) 2022-09-08T04:06:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:06 smithi055 ceph-mon[97033]: pgmap v12540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:06:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:06:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:06 smithi059 ceph-mon[97450]: pgmap v12540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:06:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:06:07.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:06.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:06.896715+0000) 2022-09-08T04:06:07.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:06.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:06.878455+0000) 2022-09-08T04:06:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:07.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:07.896937+0000) 2022-09-08T04:06:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:07.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:07.878694+0000) 2022-09-08T04:06:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:08 smithi055 ceph-mon[97033]: pgmap v12541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:06:09.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:08 smithi059 ceph-mon[97450]: pgmap v12541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:06:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:08.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:08.878924+0000) 2022-09-08T04:06:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:08.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:08.897066+0000) 2022-09-08T04:06:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:09.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:09.897287+0000) 2022-09-08T04:06:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:09.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:09.879084+0000) 2022-09-08T04:06:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:10 smithi055 ceph-mon[97033]: pgmap v12542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:06:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:10 smithi055 ceph-mon[97033]: osdmap e864: 12 total, 11 up, 11 in 2022-09-08T04:06:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:10 smithi059 ceph-mon[97450]: pgmap v12542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:06:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:10 smithi059 ceph-mon[97450]: osdmap e864: 12 total, 11 up, 11 in 2022-09-08T04:06:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:10.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:10.879276+0000) 2022-09-08T04:06:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:10.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:10.897491+0000) 2022-09-08T04:06:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:11.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:11.897652+0000) 2022-09-08T04:06:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:11.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:11.879513+0000) 2022-09-08T04:06:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:12 smithi055 ceph-mon[97033]: pgmap v12544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:06:13.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:12 smithi059 ceph-mon[97450]: pgmap v12544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:06:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:12.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:12.879764+0000) 2022-09-08T04:06:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:12.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:12.897805+0000) 2022-09-08T04:06:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:13.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:13.898031+0000) 2022-09-08T04:06:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:13.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:13.879981+0000) 2022-09-08T04:06:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:14 smithi055 ceph-mon[97033]: pgmap v12545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:06:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:14 smithi059 ceph-mon[97450]: pgmap v12545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:06:15.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:06:14.716+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:06:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:14.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:14.898264+0000) 2022-09-08T04:06:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:14.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:14.880143+0000) 2022-09-08T04:06:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:15.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:15.880387+0000) 2022-09-08T04:06:16.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:15.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:15.898541+0000) 2022-09-08T04:06:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:16 smithi055 ceph-mon[97033]: pgmap v12546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:06:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:16 smithi059 ceph-mon[97450]: pgmap v12546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:06:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:16.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:16.898705+0000) 2022-09-08T04:06:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:16.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:16.880623+0000) 2022-09-08T04:06:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:17.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:17.898862+0000) 2022-09-08T04:06:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:17.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:17.880829+0000) 2022-09-08T04:06:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:18 smithi055 ceph-mon[97033]: pgmap v12547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:06:19.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:18 smithi059 ceph-mon[97450]: pgmap v12547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:06:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:18.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:18.899115+0000) 2022-09-08T04:06:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:18.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:18.881084+0000) 2022-09-08T04:06:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:19.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:19.899370+0000) 2022-09-08T04:06:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:19.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:19.881274+0000) 2022-09-08T04:06:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:20 smithi055 ceph-mon[97033]: osdmap e865: 12 total, 11 up, 11 in 2022-09-08T04:06:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:20 smithi055 ceph-mon[97033]: pgmap v12549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T04:06:20.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:20 smithi059 ceph-mon[97450]: osdmap e865: 12 total, 11 up, 11 in 2022-09-08T04:06:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:20 smithi059 ceph-mon[97450]: pgmap v12549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T04:06:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:20.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:20.899542+0000) 2022-09-08T04:06:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:20.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:20.881453+0000) 2022-09-08T04:06:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:21.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:21.881659+0000) 2022-09-08T04:06:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:21.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:21.899785+0000) 2022-09-08T04:06:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:22 smithi055 ceph-mon[97033]: pgmap v12550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:06:23.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:22 smithi059 ceph-mon[97450]: pgmap v12550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:06:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:22.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:22.881768+0000) 2022-09-08T04:06:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:22.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:22.900062+0000) 2022-09-08T04:06:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:23.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:23.900328+0000) 2022-09-08T04:06:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:23.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:23.882030+0000) 2022-09-08T04:06:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:24 smithi055 ceph-mon[97033]: pgmap v12551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:06:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:24 smithi059 ceph-mon[97450]: pgmap v12551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:06:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:24.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:24.900584+0000) 2022-09-08T04:06:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:24.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:24.882301+0000) 2022-09-08T04:06:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:25.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:25.900830+0000) 2022-09-08T04:06:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:25.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:25.882513+0000) 2022-09-08T04:06:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:26 smithi055 ceph-mon[97033]: pgmap v12552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:06:27.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:26 smithi059 ceph-mon[97450]: pgmap v12552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:06:27.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:26.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:26.901053+0000) 2022-09-08T04:06:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:26.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:26.882725+0000) 2022-09-08T04:06:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:27.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:27.882976+0000) 2022-09-08T04:06:28.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:27.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:27.901298+0000) 2022-09-08T04:06:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:28 smithi055 ceph-mon[97033]: pgmap v12553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:06:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:28 smithi059 ceph-mon[97450]: pgmap v12553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:06:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:28.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:28.883212+0000) 2022-09-08T04:06:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:28.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:28.901545+0000) 2022-09-08T04:06:30.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:06:29.729+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:06:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:29.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:29.901797+0000) 2022-09-08T04:06:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:29.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:29.883432+0000) 2022-09-08T04:06:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:30 smithi055 ceph-mon[97033]: pgmap v12554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:06:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:30 smithi059 ceph-mon[97450]: pgmap v12554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:06:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:30.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:30.902017+0000) 2022-09-08T04:06:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:30.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:30.883669+0000) 2022-09-08T04:06:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:31.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:31.902244+0000) 2022-09-08T04:06:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:31.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:31.883918+0000) 2022-09-08T04:06:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:32 smithi055 ceph-mon[97033]: pgmap v12555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:06:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:32 smithi059 ceph-mon[97450]: pgmap v12555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:06:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:32.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:32.902508+0000) 2022-09-08T04:06:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:32.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:32.884175+0000) 2022-09-08T04:06:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:33.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:33.902753+0000) 2022-09-08T04:06:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:33.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:33.884397+0000) 2022-09-08T04:06:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:34 smithi055 ceph-mon[97033]: pgmap v12556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:34 smithi059 ceph-mon[97450]: pgmap v12556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:34.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:34.902941+0000) 2022-09-08T04:06:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:34.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:34.884582+0000) 2022-09-08T04:06:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:35.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:35.903183+0000) 2022-09-08T04:06:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:35.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:35.884816+0000) 2022-09-08T04:06:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:36 smithi055 ceph-mon[97033]: pgmap v12557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:06:37.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:36 smithi059 ceph-mon[97450]: pgmap v12557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:06:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:36.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:36.903411+0000) 2022-09-08T04:06:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:36.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:36.885042+0000) 2022-09-08T04:06:38.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:37.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:37.903671+0000) 2022-09-08T04:06:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:37.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:37.885266+0000) 2022-09-08T04:06:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:38 smithi055 ceph-mon[97033]: pgmap v12558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:39.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:38 smithi059 ceph-mon[97450]: pgmap v12558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:38.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:38.903915+0000) 2022-09-08T04:06:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:38.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:38.885524+0000) 2022-09-08T04:06:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:39.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:39.904077+0000) 2022-09-08T04:06:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:39.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:39.885730+0000) 2022-09-08T04:06:40.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:06:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:06:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:06:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:40 smithi059 ceph-mon[97450]: pgmap v12559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:06:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:06:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:06:40.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:06:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:40 smithi055 ceph-mon[97033]: pgmap v12559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:06:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:40.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:40.904305+0000) 2022-09-08T04:06:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:40.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:40.885951+0000) 2022-09-08T04:06:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:41.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:41.904562+0000) 2022-09-08T04:06:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:41.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:41.886151+0000) 2022-09-08T04:06:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:42 smithi055 ceph-mon[97033]: pgmap v12560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:06:43.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:42 smithi059 ceph-mon[97450]: pgmap v12560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:06:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:42.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:42.904773+0000) 2022-09-08T04:06:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:42.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:42.886332+0000) 2022-09-08T04:06:44.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:43.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:43.905017+0000) 2022-09-08T04:06:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:43.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:43.886600+0000) 2022-09-08T04:06:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:44 smithi055 ceph-mon[97033]: pgmap v12561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:44 smithi059 ceph-mon[97450]: pgmap v12561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:44.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:44.886751+0000) 2022-09-08T04:06:45.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:06:44.739+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:06:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:44.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:44.905205+0000) 2022-09-08T04:06:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:45.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:45.905429+0000) 2022-09-08T04:06:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:45.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:45.886978+0000) 2022-09-08T04:06:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:46 smithi055 ceph-mon[97033]: pgmap v12562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:06:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:46 smithi059 ceph-mon[97450]: pgmap v12562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:06:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:46.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:46.905678+0000) 2022-09-08T04:06:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:46.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:46.887238+0000) 2022-09-08T04:06:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:47.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:47.887519+0000) 2022-09-08T04:06:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:47.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:47.905951+0000) 2022-09-08T04:06:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:48 smithi055 ceph-mon[97033]: pgmap v12563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:48 smithi059 ceph-mon[97450]: pgmap v12563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:48.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:48.906218+0000) 2022-09-08T04:06:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:48.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:48.887760+0000) 2022-09-08T04:06:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:49.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:49.906446+0000) 2022-09-08T04:06:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:49.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:49.888012+0000) 2022-09-08T04:06:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:50 smithi055 ceph-mon[97033]: pgmap v12564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:50 smithi059 ceph-mon[97450]: pgmap v12564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:50.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:50.906620+0000) 2022-09-08T04:06:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:50.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:50.888244+0000) 2022-09-08T04:06:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:51.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:51.906822+0000) 2022-09-08T04:06:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:51.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:51.888491+0000) 2022-09-08T04:06:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:52 smithi055 ceph-mon[97033]: pgmap v12565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:06:53.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:52 smithi059 ceph-mon[97450]: pgmap v12565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:06:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:52.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:52.906968+0000) 2022-09-08T04:06:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:52.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:52.888734+0000) 2022-09-08T04:06:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:53.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:53.907193+0000) 2022-09-08T04:06:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:53.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:53.888979+0000) 2022-09-08T04:06:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:54 smithi055 ceph-mon[97033]: pgmap v12566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:54 smithi059 ceph-mon[97450]: pgmap v12566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:54.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:54.889186+0000) 2022-09-08T04:06:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:54.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:54.907379+0000) 2022-09-08T04:06:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:55.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:55.907507+0000) 2022-09-08T04:06:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:55.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:55.889481+0000) 2022-09-08T04:06:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:56 smithi055 ceph-mon[97033]: pgmap v12567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:06:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:56 smithi059 ceph-mon[97450]: pgmap v12567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:06:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:56.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:56.907726+0000) 2022-09-08T04:06:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:56.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:56.889681+0000) 2022-09-08T04:06:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:57.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:57.907970+0000) 2022-09-08T04:06:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:57.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:57.889855+0000) 2022-09-08T04:06:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:06:58 smithi055 ceph-mon[97033]: pgmap v12568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:59.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:06:58 smithi059 ceph-mon[97450]: pgmap v12568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:06:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:58.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:58.908165+0000) 2022-09-08T04:06:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:58.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:58.890051+0000) 2022-09-08T04:07:00.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:06:59.747+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:07:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:06:59.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:59.908386+0000) 2022-09-08T04:07:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:06:59.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:06:59.890248+0000) 2022-09-08T04:07:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:00 smithi055 ceph-mon[97033]: pgmap v12569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:00.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:00 smithi059 ceph-mon[97450]: pgmap v12569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:00.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:00.908599+0000) 2022-09-08T04:07:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:00.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:00.890469+0000) 2022-09-08T04:07:02.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:01.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:01.908824+0000) 2022-09-08T04:07:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:01.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:01.890728+0000) 2022-09-08T04:07:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:02 smithi055 ceph-mon[97033]: pgmap v12570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:07:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:02 smithi059 ceph-mon[97450]: pgmap v12570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:07:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:02.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:02.909088+0000) 2022-09-08T04:07:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:02.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:02.890877+0000) 2022-09-08T04:07:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:03.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:03.909280+0000) 2022-09-08T04:07:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:03.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:03.891120+0000) 2022-09-08T04:07:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:04 smithi055 ceph-mon[97033]: pgmap v12571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:04 smithi059 ceph-mon[97450]: pgmap v12571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:04.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:04.909506+0000) 2022-09-08T04:07:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:04.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:04.891288+0000) 2022-09-08T04:07:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:07:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:07:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:05.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:05.909789+0000) 2022-09-08T04:07:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:05.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:05.891445+0000) 2022-09-08T04:07:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:06 smithi055 ceph-mon[97033]: pgmap v12572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:07:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:07:07.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:06 smithi059 ceph-mon[97450]: pgmap v12572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:07:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:07:07.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:06.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:06.910047+0000) 2022-09-08T04:07:07.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:06.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:06.891682+0000) 2022-09-08T04:07:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:07.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:07.910199+0000) 2022-09-08T04:07:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:07.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:07.891853+0000) 2022-09-08T04:07:08.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:08 smithi055 ceph-mon[97033]: pgmap v12573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:07:09.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:08 smithi059 ceph-mon[97450]: pgmap v12573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:07:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:08.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:08.910432+0000) 2022-09-08T04:07:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:08.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:08.892118+0000) 2022-09-08T04:07:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:09.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:09.910608+0000) 2022-09-08T04:07:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:09.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:09.892369+0000) 2022-09-08T04:07:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:10 smithi055 ceph-mon[97033]: pgmap v12574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:07:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:10 smithi055 ceph-mon[97033]: osdmap e866: 12 total, 11 up, 11 in 2022-09-08T04:07:11.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:10 smithi059 ceph-mon[97450]: pgmap v12574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:07:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:10 smithi059 ceph-mon[97450]: osdmap e866: 12 total, 11 up, 11 in 2022-09-08T04:07:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:10.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:10.910815+0000) 2022-09-08T04:07:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:10.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:10.892592+0000) 2022-09-08T04:07:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:11.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:11.910989+0000) 2022-09-08T04:07:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:11.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:11.892756+0000) 2022-09-08T04:07:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:12 smithi055 ceph-mon[97033]: pgmap v12576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:07:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:12 smithi059 ceph-mon[97450]: pgmap v12576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:07:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:12.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:12.911237+0000) 2022-09-08T04:07:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:12.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:12.892994+0000) 2022-09-08T04:07:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:13.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:13.911454+0000) 2022-09-08T04:07:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:13.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:13.893198+0000) 2022-09-08T04:07:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:14 smithi055 ceph-mon[97033]: pgmap v12577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:07:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:14 smithi059 ceph-mon[97450]: pgmap v12577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:07:15.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:07:14.757+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:07:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:14.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:14.911679+0000) 2022-09-08T04:07:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:14.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:14.893359+0000) 2022-09-08T04:07:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:15.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:15.911923+0000) 2022-09-08T04:07:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:15.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:15.893592+0000) 2022-09-08T04:07:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:16 smithi055 ceph-mon[97033]: pgmap v12578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:07:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:16 smithi059 ceph-mon[97450]: pgmap v12578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:07:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:16.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:16.893805+0000) 2022-09-08T04:07:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:16.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:16.912153+0000) 2022-09-08T04:07:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:17.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:17.912318+0000) 2022-09-08T04:07:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:17.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:17.894080+0000) 2022-09-08T04:07:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:18 smithi055 ceph-mon[97033]: pgmap v12579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:07:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:18 smithi059 ceph-mon[97450]: pgmap v12579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:07:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:18.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:18.912498+0000) 2022-09-08T04:07:19.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:18.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:18.894268+0000) 2022-09-08T04:07:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:19.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:19.912705+0000) 2022-09-08T04:07:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:19.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:19.894479+0000) 2022-09-08T04:07:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:20 smithi055 ceph-mon[97033]: osdmap e867: 12 total, 11 up, 11 in 2022-09-08T04:07:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:20 smithi055 ceph-mon[97033]: pgmap v12581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T04:07:20.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:20 smithi059 ceph-mon[97450]: osdmap e867: 12 total, 11 up, 11 in 2022-09-08T04:07:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:20 smithi059 ceph-mon[97450]: pgmap v12581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T04:07:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:20.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:20.912964+0000) 2022-09-08T04:07:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:20.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:20.894699+0000) 2022-09-08T04:07:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:21.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:21.913203+0000) 2022-09-08T04:07:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:21.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:21.894943+0000) 2022-09-08T04:07:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:22 smithi055 ceph-mon[97033]: pgmap v12582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:07:23.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:22 smithi059 ceph-mon[97450]: pgmap v12582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:07:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:22.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:22.895162+0000) 2022-09-08T04:07:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:22.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:22.913457+0000) 2022-09-08T04:07:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:23.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:23.913685+0000) 2022-09-08T04:07:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:23.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:23.895408+0000) 2022-09-08T04:07:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:24 smithi055 ceph-mon[97033]: pgmap v12583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:07:25.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:24 smithi059 ceph-mon[97450]: pgmap v12583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:07:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:24.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:24.913809+0000) 2022-09-08T04:07:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:24.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:24.895616+0000) 2022-09-08T04:07:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:25.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:25.913980+0000) 2022-09-08T04:07:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:25.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:25.895869+0000) 2022-09-08T04:07:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:26 smithi055 ceph-mon[97033]: pgmap v12584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:07:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:26 smithi059 ceph-mon[97450]: pgmap v12584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:07:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:26.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:26.914188+0000) 2022-09-08T04:07:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:26.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:26.896074+0000) 2022-09-08T04:07:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:27.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:27.914412+0000) 2022-09-08T04:07:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:27.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:27.896285+0000) 2022-09-08T04:07:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:28 smithi055 ceph-mon[97033]: pgmap v12585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:07:29.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:28 smithi059 ceph-mon[97450]: pgmap v12585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:07:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:28.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:28.914596+0000) 2022-09-08T04:07:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:28.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:28.896465+0000) 2022-09-08T04:07:30.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:07:29.765+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:07:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:29.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:29.914834+0000) 2022-09-08T04:07:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:29.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:29.896650+0000) 2022-09-08T04:07:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:30 smithi055 ceph-mon[97033]: pgmap v12586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:07:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:30 smithi059 ceph-mon[97450]: pgmap v12586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:07:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:30.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:30.896871+0000) 2022-09-08T04:07:31.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:30.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:30.915080+0000) 2022-09-08T04:07:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:31.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:31.897140+0000) 2022-09-08T04:07:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:31.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:31.915319+0000) 2022-09-08T04:07:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:32 smithi055 ceph-mon[97033]: pgmap v12587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:07:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:32 smithi059 ceph-mon[97450]: pgmap v12587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:07:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:32.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:32.897346+0000) 2022-09-08T04:07:33.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:32.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:32.915591+0000) 2022-09-08T04:07:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:33.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:33.897532+0000) 2022-09-08T04:07:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:33.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:33.915896+0000) 2022-09-08T04:07:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:34 smithi055 ceph-mon[97033]: pgmap v12588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:34 smithi059 ceph-mon[97450]: pgmap v12588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:34.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:34.897752+0000) 2022-09-08T04:07:35.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:34.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:34.916142+0000) 2022-09-08T04:07:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:35.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:35.916289+0000) 2022-09-08T04:07:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:35.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:35.897982+0000) 2022-09-08T04:07:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:36 smithi055 ceph-mon[97033]: pgmap v12589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:07:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:36 smithi059 ceph-mon[97450]: pgmap v12589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:07:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:36.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:36.916440+0000) 2022-09-08T04:07:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:36.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:36.898178+0000) 2022-09-08T04:07:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:37.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:37.916676+0000) 2022-09-08T04:07:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:37.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:37.898460+0000) 2022-09-08T04:07:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:38 smithi055 ceph-mon[97033]: pgmap v12590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:39.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:38 smithi059 ceph-mon[97450]: pgmap v12590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:38.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:38.916820+0000) 2022-09-08T04:07:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:38.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:38.898743+0000) 2022-09-08T04:07:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:39.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:39.917057+0000) 2022-09-08T04:07:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:39.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:39.898984+0000) 2022-09-08T04:07:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:40 smithi055 ceph-mon[97033]: pgmap v12591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:07:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:07:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:07:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:40 smithi059 ceph-mon[97450]: pgmap v12591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:07:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:07:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:07:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:40.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:40.917471+0000) 2022-09-08T04:07:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:40.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:40.899201+0000) 2022-09-08T04:07:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:41.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:41.917679+0000) 2022-09-08T04:07:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:41.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:41.899402+0000) 2022-09-08T04:07:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:42 smithi055 ceph-mon[97033]: pgmap v12592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:07:43.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:42 smithi059 ceph-mon[97450]: pgmap v12592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:07:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:42.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:42.917895+0000) 2022-09-08T04:07:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:42.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:42.899679+0000) 2022-09-08T04:07:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:43.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:43.918151+0000) 2022-09-08T04:07:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:43.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:43.899881+0000) 2022-09-08T04:07:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:44 smithi055 ceph-mon[97033]: pgmap v12593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:44 smithi059 ceph-mon[97450]: pgmap v12593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:45.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:44.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:44.918395+0000) 2022-09-08T04:07:45.158 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:07:44.780+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:07:45.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:44.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:44.900038+0000) 2022-09-08T04:07:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:45.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:45.918520+0000) 2022-09-08T04:07:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:45.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:45.900229+0000) 2022-09-08T04:07:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:46 smithi055 ceph-mon[97033]: pgmap v12594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:07:47.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:46 smithi059 ceph-mon[97450]: pgmap v12594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:07:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:46.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:46.918695+0000) 2022-09-08T04:07:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:46.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:46.900452+0000) 2022-09-08T04:07:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:47.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:47.918916+0000) 2022-09-08T04:07:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:47.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:47.900629+0000) 2022-09-08T04:07:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:48 smithi055 ceph-mon[97033]: pgmap v12595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:48 smithi059 ceph-mon[97450]: pgmap v12595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:48.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:48.919100+0000) 2022-09-08T04:07:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:48.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:48.900801+0000) 2022-09-08T04:07:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:49.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:49.919300+0000) 2022-09-08T04:07:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:49.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:49.900940+0000) 2022-09-08T04:07:50.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:50 smithi055 ceph-mon[97033]: pgmap v12596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:50.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:50 smithi059 ceph-mon[97450]: pgmap v12596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:50.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:50.919529+0000) 2022-09-08T04:07:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:50.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:50.901142+0000) 2022-09-08T04:07:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:51.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:51.919741+0000) 2022-09-08T04:07:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:51.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:51.901328+0000) 2022-09-08T04:07:52.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:52 smithi055 ceph-mon[97033]: pgmap v12597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:07:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:52 smithi059 ceph-mon[97450]: pgmap v12597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:07:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:52.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:52.919947+0000) 2022-09-08T04:07:53.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:52.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:52.901548+0000) 2022-09-08T04:07:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:53.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:53.920135+0000) 2022-09-08T04:07:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:53.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:53.901704+0000) 2022-09-08T04:07:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:54 smithi055 ceph-mon[97033]: pgmap v12598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:54 smithi059 ceph-mon[97450]: pgmap v12598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:54.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:54.920378+0000) 2022-09-08T04:07:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:54.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:54.901901+0000) 2022-09-08T04:07:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:55.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:55.920592+0000) 2022-09-08T04:07:56.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:55.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:55.902141+0000) 2022-09-08T04:07:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:56 smithi055 ceph-mon[97033]: pgmap v12599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:07:57.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:56 smithi059 ceph-mon[97450]: pgmap v12599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:07:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:56.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:56.920813+0000) 2022-09-08T04:07:57.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:56.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:56.902349+0000) 2022-09-08T04:07:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:57.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:57.921056+0000) 2022-09-08T04:07:58.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:57.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:57.902560+0000) 2022-09-08T04:07:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:07:58 smithi055 ceph-mon[97033]: pgmap v12600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:59.055 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:07:58 smithi059 ceph-mon[97450]: pgmap v12600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:07:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:58.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:58.921231+0000) 2022-09-08T04:07:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:58.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:58.902814+0000) 2022-09-08T04:08:00.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:07:59.789+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:08:00.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:07:59.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:59.921465+0000) 2022-09-08T04:08:00.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:07:59.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:07:59.903013+0000) 2022-09-08T04:08:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:00 smithi055 ceph-mon[97033]: pgmap v12601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:00.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:00 smithi059 ceph-mon[97450]: pgmap v12601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:00.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:00.921629+0000) 2022-09-08T04:08:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:00.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:00.903192+0000) 2022-09-08T04:08:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:01.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:01.921783+0000) 2022-09-08T04:08:02.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:01.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:01.903407+0000) 2022-09-08T04:08:02.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:02 smithi055 ceph-mon[97033]: pgmap v12602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:08:03.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:02 smithi059 ceph-mon[97450]: pgmap v12602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:08:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:02.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:02.922026+0000) 2022-09-08T04:08:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:02.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:02.903668+0000) 2022-09-08T04:08:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:03.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:03.922210+0000) 2022-09-08T04:08:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:03.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:03.903906+0000) 2022-09-08T04:08:04.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:04 smithi055 ceph-mon[97033]: pgmap v12603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:04 smithi059 ceph-mon[97450]: pgmap v12603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:05.240 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:04.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:04.922382+0000) 2022-09-08T04:08:05.241 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:04.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:04.904088+0000) 2022-09-08T04:08:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:08:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:08:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:05.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:05.922584+0000) 2022-09-08T04:08:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:05.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:05.904324+0000) 2022-09-08T04:08:06.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:06 smithi055 ceph-mon[97033]: pgmap v12604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:08:06.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:08:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:06 smithi059 ceph-mon[97450]: pgmap v12604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:08:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:08:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:06.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:06.922835+0000) 2022-09-08T04:08:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:06.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:06.904494+0000) 2022-09-08T04:08:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:07.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:07.923073+0000) 2022-09-08T04:08:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:07.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:07.904756+0000) 2022-09-08T04:08:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:08 smithi055 ceph-mon[97033]: pgmap v12605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:08:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:08 smithi059 ceph-mon[97450]: pgmap v12605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:08:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:08.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:08.923259+0000) 2022-09-08T04:08:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:08.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:08.904988+0000) 2022-09-08T04:08:10.292 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:09.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:09.923505+0000) 2022-09-08T04:08:10.293 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:09.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:09.905210+0000) 2022-09-08T04:08:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:10 smithi055 ceph-mon[97033]: pgmap v12606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:08:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:10 smithi055 ceph-mon[97033]: osdmap e868: 12 total, 11 up, 11 in 2022-09-08T04:08:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:08:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:10 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T04:08:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:10 smithi059 ceph-mon[97450]: pgmap v12606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:08:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:10 smithi059 ceph-mon[97450]: osdmap e868: 12 total, 11 up, 11 in 2022-09-08T04:08:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:08:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:10 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T04:08:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:10.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:10.923755+0000) 2022-09-08T04:08:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:10.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:10.905402+0000) 2022-09-08T04:08:11.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:11 smithi055 ceph-mon[97033]: pgmap v12608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T04:08:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:11 smithi059 ceph-mon[97450]: pgmap v12608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T04:08:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:11.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:11.923950+0000) 2022-09-08T04:08:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:11.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:11.905615+0000) 2022-09-08T04:08:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:12.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:12.924152+0000) 2022-09-08T04:08:13.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:12.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:12.905853+0000) 2022-09-08T04:08:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:13 smithi055 ceph-mon[97033]: pgmap v12609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T04:08:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:13 smithi059 ceph-mon[97450]: pgmap v12609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T04:08:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:13.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:13.924378+0000) 2022-09-08T04:08:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:13.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:13.906018+0000) 2022-09-08T04:08:15.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:08:14.803+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:08:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:14.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:14.924615+0000) 2022-09-08T04:08:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:14.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:14.906186+0000) 2022-09-08T04:08:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:15 smithi055 ceph-mon[97033]: pgmap v12610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 117 B/s wr, 3 op/s 2022-09-08T04:08:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:15 smithi055 ceph-mon[97033]: osdmap e869: 12 total, 11 up, 11 in 2022-09-08T04:08:15.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:15 smithi059 ceph-mon[97450]: pgmap v12610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 117 B/s wr, 3 op/s 2022-09-08T04:08:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:15 smithi059 ceph-mon[97450]: osdmap e869: 12 total, 11 up, 11 in 2022-09-08T04:08:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:15.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:15.924791+0000) 2022-09-08T04:08:16.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:15.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:15.906428+0000) 2022-09-08T04:08:17.409 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:16.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:16.924911+0000) 2022-09-08T04:08:17.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:16.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:16.906699+0000) 2022-09-08T04:08:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:17 smithi055 ceph-mon[97033]: pgmap v12612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.8 KiB/s rd, 305 B/s wr, 9 op/s 2022-09-08T04:08:17.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:17 smithi059 ceph-mon[97450]: pgmap v12612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.8 KiB/s rd, 305 B/s wr, 9 op/s 2022-09-08T04:08:18.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:17.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:17.925085+0000) 2022-09-08T04:08:18.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:17.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:17.906966+0000) 2022-09-08T04:08:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:18.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:18.925253+0000) 2022-09-08T04:08:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:18.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:18.907228+0000) 2022-09-08T04:08:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:19 smithi055 ceph-mon[97033]: pgmap v12613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2022-09-08T04:08:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:19 smithi059 ceph-mon[97450]: pgmap v12613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2022-09-08T04:08:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:19.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:19.925466+0000) 2022-09-08T04:08:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:19.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:19.907472+0000) 2022-09-08T04:08:21.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:20.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:20.925713+0000) 2022-09-08T04:08:21.426 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:20.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:20.907722+0000) 2022-09-08T04:08:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:21 smithi055 ceph-mon[97033]: pgmap v12614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:08:21.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:21 smithi059 ceph-mon[97450]: pgmap v12614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:08:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:21.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:21.925929+0000) 2022-09-08T04:08:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:21.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:21.907891+0000) 2022-09-08T04:08:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:22.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:22.926101+0000) 2022-09-08T04:08:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:22.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:22.908093+0000) 2022-09-08T04:08:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:23 smithi055 ceph-mon[97033]: pgmap v12615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:08:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:23 smithi059 ceph-mon[97450]: pgmap v12615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:08:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:23.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:23.908318+0000) 2022-09-08T04:08:24.477 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:23.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:23.926334+0000) 2022-09-08T04:08:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:24.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:24.926550+0000) 2022-09-08T04:08:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:24.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:24.908520+0000) 2022-09-08T04:08:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:25 smithi055 ceph-mon[97033]: pgmap v12616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:08:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:25 smithi059 ceph-mon[97450]: pgmap v12616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:08:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:25.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:25.926752+0000) 2022-09-08T04:08:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:25.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:25.908758+0000) 2022-09-08T04:08:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:26.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:26.926943+0000) 2022-09-08T04:08:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:26.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:26.908980+0000) 2022-09-08T04:08:27.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:27 smithi055 ceph-mon[97033]: pgmap v12617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T04:08:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:27 smithi059 ceph-mon[97450]: pgmap v12617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T04:08:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:27.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:27.927105+0000) 2022-09-08T04:08:28.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:27.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:27.909152+0000) 2022-09-08T04:08:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:28.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:28.927275+0000) 2022-09-08T04:08:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:28.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:28.909383+0000) 2022-09-08T04:08:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:29 smithi055 ceph-mon[97033]: pgmap v12618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:29 smithi059 ceph-mon[97450]: pgmap v12618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:30.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:08:29.816+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:08:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:29.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:29.927439+0000) 2022-09-08T04:08:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:29.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:29.909582+0000) 2022-09-08T04:08:31.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:30.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:30.909781+0000) 2022-09-08T04:08:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:30.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:30.927611+0000) 2022-09-08T04:08:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:31 smithi055 ceph-mon[97033]: pgmap v12619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:08:31.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:31 smithi059 ceph-mon[97450]: pgmap v12619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:08:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:31.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:31.927836+0000) 2022-09-08T04:08:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:31.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:31.909942+0000) 2022-09-08T04:08:33.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:32.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:32.910131+0000) 2022-09-08T04:08:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:32.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:32.928104+0000) 2022-09-08T04:08:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:33 smithi055 ceph-mon[97033]: pgmap v12620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:33.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:33 smithi059 ceph-mon[97450]: pgmap v12620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:33.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:33.928361+0000) 2022-09-08T04:08:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:33.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:33.910364+0000) 2022-09-08T04:08:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:34.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:34.910574+0000) 2022-09-08T04:08:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:34.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:34.928566+0000) 2022-09-08T04:08:35.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:35 smithi055 ceph-mon[97033]: pgmap v12621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:35.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:35 smithi059 ceph-mon[97450]: pgmap v12621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:36.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:35.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:35.910845+0000) 2022-09-08T04:08:36.410 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:35.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:35.928798+0000) 2022-09-08T04:08:37.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:36.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:36.929005+0000) 2022-09-08T04:08:37.409 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:36.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:36.910994+0000) 2022-09-08T04:08:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:37 smithi055 ceph-mon[97033]: pgmap v12622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:08:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:37 smithi059 ceph-mon[97450]: pgmap v12622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:08:38.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:37.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:37.911165+0000) 2022-09-08T04:08:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:37.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:37.929259+0000) 2022-09-08T04:08:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:38.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:38.911347+0000) 2022-09-08T04:08:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:38.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:38.929476+0000) 2022-09-08T04:08:39.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:39 smithi055 ceph-mon[97033]: pgmap v12623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:39.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:39 smithi059 ceph-mon[97450]: pgmap v12623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:40.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:39.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:39.911590+0000) 2022-09-08T04:08:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:39.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:39.929763+0000) 2022-09-08T04:08:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:40.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:40.930041+0000) 2022-09-08T04:08:41.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:40.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:40.911770+0000) 2022-09-08T04:08:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:41 smithi055 ceph-mon[97033]: pgmap v12624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:08:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:41 smithi059 ceph-mon[97450]: pgmap v12624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:08:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:41.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:41.930254+0000) 2022-09-08T04:08:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:41.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:41.911967+0000) 2022-09-08T04:08:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:42.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:42.930464+0000) 2022-09-08T04:08:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:42.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:42.912134+0000) 2022-09-08T04:08:43.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:43 smithi055 ceph-mon[97033]: pgmap v12625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:43 smithi059 ceph-mon[97450]: pgmap v12625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:43.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:43.930717+0000) 2022-09-08T04:08:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:43.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:43.912354+0000) 2022-09-08T04:08:45.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:08:44.832+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:08:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:44.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:44.930936+0000) 2022-09-08T04:08:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:44.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:44.912583+0000) 2022-09-08T04:08:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:45 smithi055 ceph-mon[97033]: pgmap v12626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:45 smithi059 ceph-mon[97450]: pgmap v12626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:45.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:45.931164+0000) 2022-09-08T04:08:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:45.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:45.912773+0000) 2022-09-08T04:08:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:46.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:46.931423+0000) 2022-09-08T04:08:47.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:46.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:46.912951+0000) 2022-09-08T04:08:47.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:47 smithi055 ceph-mon[97033]: pgmap v12627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:08:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:47 smithi059 ceph-mon[97450]: pgmap v12627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:08:48.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:47.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:47.931687+0000) 2022-09-08T04:08:48.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:47.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:47.913181+0000) 2022-09-08T04:08:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:48.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:48.931948+0000) 2022-09-08T04:08:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:48.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:48.913337+0000) 2022-09-08T04:08:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:49 smithi055 ceph-mon[97033]: pgmap v12628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:49 smithi059 ceph-mon[97450]: pgmap v12628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:49.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:49.932214+0000) 2022-09-08T04:08:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:49.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:49.913573+0000) 2022-09-08T04:08:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:50.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:50.932451+0000) 2022-09-08T04:08:51.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:50.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:50.913794+0000) 2022-09-08T04:08:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:51 smithi055 ceph-mon[97033]: pgmap v12629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:08:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:51 smithi059 ceph-mon[97450]: pgmap v12629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:08:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:51.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:51.932736+0000) 2022-09-08T04:08:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:51.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:51.913958+0000) 2022-09-08T04:08:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:52.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:52.932951+0000) 2022-09-08T04:08:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:52.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:52.914117+0000) 2022-09-08T04:08:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:53 smithi055 ceph-mon[97033]: pgmap v12630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:53 smithi059 ceph-mon[97450]: pgmap v12630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:53.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:53.933125+0000) 2022-09-08T04:08:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:53.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:53.914372+0000) 2022-09-08T04:08:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:54.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:54.933344+0000) 2022-09-08T04:08:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:54.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:54.914622+0000) 2022-09-08T04:08:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:55 smithi055 ceph-mon[97033]: pgmap v12631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:55 smithi059 ceph-mon[97450]: pgmap v12631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:55.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:55.933604+0000) 2022-09-08T04:08:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:55.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:55.914877+0000) 2022-09-08T04:08:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:56.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:56.933845+0000) 2022-09-08T04:08:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:56.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:56.915102+0000) 2022-09-08T04:08:57.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:57 smithi055 ceph-mon[97033]: pgmap v12632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:08:57.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:57 smithi059 ceph-mon[97450]: pgmap v12632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:08:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:57.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:57.933965+0000) 2022-09-08T04:08:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:57.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:57.915337+0000) 2022-09-08T04:08:59.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:58.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:58.915519+0000) 2022-09-08T04:08:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:58.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:58.934191+0000) 2022-09-08T04:08:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:08:59 smithi055 ceph-mon[97033]: pgmap v12633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:08:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:08:59 smithi059 ceph-mon[97450]: pgmap v12633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:00.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:08:59.843+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:09:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:08:59.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:59.934447+0000) 2022-09-08T04:09:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:08:59.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:08:59.915744+0000) 2022-09-08T04:09:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:00.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:00.934610+0000) 2022-09-08T04:09:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:00.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:00.915969+0000) 2022-09-08T04:09:01.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:01 smithi055 ceph-mon[97033]: pgmap v12634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:09:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:01 smithi059 ceph-mon[97450]: pgmap v12634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:09:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:01.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:01.934840+0000) 2022-09-08T04:09:02.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:01.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:01.916156+0000) 2022-09-08T04:09:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:02.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:02.935054+0000) 2022-09-08T04:09:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:02.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:02.916309+0000) 2022-09-08T04:09:03.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:03 smithi055 ceph-mon[97033]: pgmap v12635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:03.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:03 smithi059 ceph-mon[97450]: pgmap v12635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:03.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:03.935221+0000) 2022-09-08T04:09:04.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:03.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:03.916507+0000) 2022-09-08T04:09:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:04.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:04.935439+0000) 2022-09-08T04:09:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:04.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:04.916705+0000) 2022-09-08T04:09:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:05 smithi055 ceph-mon[97033]: pgmap v12636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:05 smithi059 ceph-mon[97450]: pgmap v12636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:05.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:05.935617+0000) 2022-09-08T04:09:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:05.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:05.916887+0000) 2022-09-08T04:09:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:09:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:09:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:09:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:09:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:06.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:06.935831+0000) 2022-09-08T04:09:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:06.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:06.917073+0000) 2022-09-08T04:09:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:07 smithi055 ceph-mon[97033]: pgmap v12637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:09:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:07 smithi059 ceph-mon[97450]: pgmap v12637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:09:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:07.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:07.936045+0000) 2022-09-08T04:09:08.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:07.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:07.917329+0000) 2022-09-08T04:09:09.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:08.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:08.936273+0000) 2022-09-08T04:09:09.413 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:08.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:08.917547+0000) 2022-09-08T04:09:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:09 smithi055 ceph-mon[97033]: pgmap v12638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:09:09.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:09 smithi059 ceph-mon[97450]: pgmap v12638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:09:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:09.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:09.936504+0000) 2022-09-08T04:09:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:09.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:09.917692+0000) 2022-09-08T04:09:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:10 smithi055 ceph-mon[97033]: osdmap e870: 12 total, 11 up, 11 in 2022-09-08T04:09:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:09:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:09:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:09:10.790 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:10 smithi059 ceph-mon[97450]: osdmap e870: 12 total, 11 up, 11 in 2022-09-08T04:09:10.791 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:09:10.792 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:09:10.793 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:09:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:10.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:10.936751+0000) 2022-09-08T04:09:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:10.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:10.917931+0000) 2022-09-08T04:09:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:11 smithi055 ceph-mon[97033]: pgmap v12640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:09:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:11 smithi059 ceph-mon[97450]: pgmap v12640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:09:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:11.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:11.936935+0000) 2022-09-08T04:09:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:11.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:11.918180+0000) 2022-09-08T04:09:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:12.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:12.937186+0000) 2022-09-08T04:09:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:12.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:12.918437+0000) 2022-09-08T04:09:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:13 smithi059 ceph-mon[97450]: pgmap v12641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:09:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:13 smithi055 ceph-mon[97033]: pgmap v12641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:09:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:13.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:13.937439+0000) 2022-09-08T04:09:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:13.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:13.918668+0000) 2022-09-08T04:09:15.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:09:14.849+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:09:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:14.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:14.918873+0000) 2022-09-08T04:09:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:14.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:14.937590+0000) 2022-09-08T04:09:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:15 smithi059 ceph-mon[97450]: pgmap v12642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:09:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:15 smithi055 ceph-mon[97033]: pgmap v12642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:09:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:15.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:15.937777+0000) 2022-09-08T04:09:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:15.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:15.919049+0000) 2022-09-08T04:09:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:16.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:16.937914+0000) 2022-09-08T04:09:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:16.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:16.919256+0000) 2022-09-08T04:09:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:17 smithi059 ceph-mon[97450]: pgmap v12643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:09:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:17 smithi055 ceph-mon[97033]: pgmap v12643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:09:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:17.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:17.919500+0000) 2022-09-08T04:09:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:17.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:17.938158+0000) 2022-09-08T04:09:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:18.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:18.938397+0000) 2022-09-08T04:09:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:18.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:18.919744+0000) 2022-09-08T04:09:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:19 smithi059 ceph-mon[97450]: pgmap v12644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:09:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:19 smithi059 ceph-mon[97450]: osdmap e871: 12 total, 11 up, 11 in 2022-09-08T04:09:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:19 smithi055 ceph-mon[97033]: pgmap v12644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:09:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:19 smithi055 ceph-mon[97033]: osdmap e871: 12 total, 11 up, 11 in 2022-09-08T04:09:20.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:19.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:19.919963+0000) 2022-09-08T04:09:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:19.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:19.938544+0000) 2022-09-08T04:09:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:20.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:20.938721+0000) 2022-09-08T04:09:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:20.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:20.920133+0000) 2022-09-08T04:09:21.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:21 smithi059 ceph-mon[97450]: pgmap v12646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:09:21.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:21 smithi055 ceph-mon[97033]: pgmap v12646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:09:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:21.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:21.938947+0000) 2022-09-08T04:09:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:21.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:21.920372+0000) 2022-09-08T04:09:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:22.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:22.939158+0000) 2022-09-08T04:09:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:22.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:22.920579+0000) 2022-09-08T04:09:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:23 smithi059 ceph-mon[97450]: pgmap v12647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:09:23.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:23 smithi055 ceph-mon[97033]: pgmap v12647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:09:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:23.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:23.939393+0000) 2022-09-08T04:09:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:23.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:23.920761+0000) 2022-09-08T04:09:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:24.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:24.939612+0000) 2022-09-08T04:09:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:24.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:24.920923+0000) 2022-09-08T04:09:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:25 smithi059 ceph-mon[97450]: pgmap v12648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:09:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:25 smithi055 ceph-mon[97033]: pgmap v12648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:09:26.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:25.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:25.939808+0000) 2022-09-08T04:09:26.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:25.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:25.921116+0000) 2022-09-08T04:09:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:26.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:26.940032+0000) 2022-09-08T04:09:27.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:26.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:26.921334+0000) 2022-09-08T04:09:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:27 smithi059 ceph-mon[97450]: pgmap v12649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:09:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:27 smithi055 ceph-mon[97033]: pgmap v12649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:09:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:27.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:27.940223+0000) 2022-09-08T04:09:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:27.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:27.921523+0000) 2022-09-08T04:09:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:28.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:28.940454+0000) 2022-09-08T04:09:29.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:28.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:28.921698+0000) 2022-09-08T04:09:29.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:29 smithi059 ceph-mon[97450]: pgmap v12650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:09:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:29 smithi055 ceph-mon[97033]: pgmap v12650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:09:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:29.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:29.940626+0000) 2022-09-08T04:09:30.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:09:29.858+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:09:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:29.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:29.921865+0000) 2022-09-08T04:09:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:30.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:30.940833+0000) 2022-09-08T04:09:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:30.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:30.922035+0000) 2022-09-08T04:09:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:31 smithi059 ceph-mon[97450]: pgmap v12651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T04:09:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:31 smithi055 ceph-mon[97033]: pgmap v12651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T04:09:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:31.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:31.940995+0000) 2022-09-08T04:09:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:31.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:31.922246+0000) 2022-09-08T04:09:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:32.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:32.941198+0000) 2022-09-08T04:09:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:32.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:32.922417+0000) 2022-09-08T04:09:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:33 smithi059 ceph-mon[97450]: pgmap v12652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:33 smithi055 ceph-mon[97033]: pgmap v12652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:33.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:33.941384+0000) 2022-09-08T04:09:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:33.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:33.922663+0000) 2022-09-08T04:09:35.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:34.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:34.922909+0000) 2022-09-08T04:09:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:34.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:34.941621+0000) 2022-09-08T04:09:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:35 smithi059 ceph-mon[97450]: pgmap v12653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:35 smithi055 ceph-mon[97033]: pgmap v12653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:35.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:35.941887+0000) 2022-09-08T04:09:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:35.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:35.923161+0000) 2022-09-08T04:09:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:36.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:36.942020+0000) 2022-09-08T04:09:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:36.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:36.923351+0000) 2022-09-08T04:09:37.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:37 smithi059 ceph-mon[97450]: pgmap v12654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:09:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:37 smithi055 ceph-mon[97033]: pgmap v12654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:09:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:37.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:37.942162+0000) 2022-09-08T04:09:38.452 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:37.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:37.923525+0000) 2022-09-08T04:09:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:38.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:38.942352+0000) 2022-09-08T04:09:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:38.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:38.923782+0000) 2022-09-08T04:09:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:39 smithi059 ceph-mon[97450]: pgmap v12655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:39 smithi055 ceph-mon[97033]: pgmap v12655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:39.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:39.942524+0000) 2022-09-08T04:09:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:39.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:39.923973+0000) 2022-09-08T04:09:41.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:40.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:40.942750+0000) 2022-09-08T04:09:41.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:40.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:40.924129+0000) 2022-09-08T04:09:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:41 smithi055 ceph-mon[97033]: pgmap v12656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:09:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:41 smithi055 ceph-mon[97033]: pgmap v12657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:09:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:41 smithi055 ceph-mon[97033]: pgmap v12658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:09:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:09:42.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:41 smithi059 ceph-mon[97450]: pgmap v12656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:09:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:41 smithi059 ceph-mon[97450]: pgmap v12657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:09:42.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:41 smithi059 ceph-mon[97450]: pgmap v12658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:09:42.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:09:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:41.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:41.942915+0000) 2022-09-08T04:09:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:41.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:41.924364+0000) 2022-09-08T04:09:43.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:42.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:42.924556+0000) 2022-09-08T04:09:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:42.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:42.943084+0000) 2022-09-08T04:09:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:43 smithi055 ceph-mon[97033]: pgmap v12659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:09:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:43 smithi059 ceph-mon[97450]: pgmap v12659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:09:44.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:43.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:43.924808+0000) 2022-09-08T04:09:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:43.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:43.943295+0000) 2022-09-08T04:09:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:44.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:44.943503+0000) 2022-09-08T04:09:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:44.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:44.925048+0000) 2022-09-08T04:09:45.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:09:44.861+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:09:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:45 smithi055 ceph-mon[97033]: pgmap v12660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:09:45.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:45 smithi059 ceph-mon[97450]: pgmap v12660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:09:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:45.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:45.943732+0000) 2022-09-08T04:09:46.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:45.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:45.925280+0000) 2022-09-08T04:09:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:46.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:46.943967+0000) 2022-09-08T04:09:47.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:46.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:46.925491+0000) 2022-09-08T04:09:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:47 smithi055 ceph-mon[97033]: pgmap v12661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:09:48.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:47 smithi059 ceph-mon[97450]: pgmap v12661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:09:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:47.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:47.944185+0000) 2022-09-08T04:09:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:47.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:47.925735+0000) 2022-09-08T04:09:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:48.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:48.944394+0000) 2022-09-08T04:09:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:48.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:48.925929+0000) 2022-09-08T04:09:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:49 smithi055 ceph-mon[97033]: pgmap v12662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:09:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:49 smithi059 ceph-mon[97450]: pgmap v12662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:09:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:49.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:49.944614+0000) 2022-09-08T04:09:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:49.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:49.926055+0000) 2022-09-08T04:09:51.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:50.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:50.926197+0000) 2022-09-08T04:09:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:50.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:50.944893+0000) 2022-09-08T04:09:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:51 smithi055 ceph-mon[97033]: pgmap v12663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:09:52.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:51 smithi059 ceph-mon[97450]: pgmap v12663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:09:52.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:51.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:51.926426+0000) 2022-09-08T04:09:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:51.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:51.945154+0000) 2022-09-08T04:09:53.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:52.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:52.926615+0000) 2022-09-08T04:09:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:52.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:52.945389+0000) 2022-09-08T04:09:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:53 smithi055 ceph-mon[97033]: pgmap v12664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:53 smithi059 ceph-mon[97450]: pgmap v12664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:53.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:53.945609+0000) 2022-09-08T04:09:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:53.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:53.926865+0000) 2022-09-08T04:09:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:54.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:54.945841+0000) 2022-09-08T04:09:55.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:54.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:54.927091+0000) 2022-09-08T04:09:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:55 smithi055 ceph-mon[97033]: pgmap v12665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:55 smithi059 ceph-mon[97450]: pgmap v12665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:09:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:55.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:55.946021+0000) 2022-09-08T04:09:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:55.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:55.927282+0000) 2022-09-08T04:09:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:56.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:56.927497+0000) 2022-09-08T04:09:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:56.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:56.946248+0000) 2022-09-08T04:09:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:57 smithi055 ceph-mon[97033]: pgmap v12666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:09:58.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:57 smithi059 ceph-mon[97450]: pgmap v12666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:09:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:57.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:57.946401+0000) 2022-09-08T04:09:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:57.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:57.927741+0000) 2022-09-08T04:09:59.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:58.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:58.927899+0000) 2022-09-08T04:09:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:58.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:58.946600+0000) 2022-09-08T04:09:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:09:59 smithi055 ceph-mon[97033]: pgmap v12667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:09:59 smithi059 ceph-mon[97450]: pgmap v12667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:09:59.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:59.946727+0000) 2022-09-08T04:10:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:09:59.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:09:59.928108+0000) 2022-09-08T04:10:00.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:09:59.878+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:10:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T04:10:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T04:10:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T04:10:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T04:10:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T04:10:01.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T04:10:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T04:10:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T04:10:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T04:10:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T04:10:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:00.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:00.946873+0000) 2022-09-08T04:10:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:00.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:00.928263+0000) 2022-09-08T04:10:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:01 smithi055 ceph-mon[97033]: pgmap v12668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:10:02.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:01 smithi059 ceph-mon[97450]: pgmap v12668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:10:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:01.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:01.947081+0000) 2022-09-08T04:10:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:01.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:01.928438+0000) 2022-09-08T04:10:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:02.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:02.947239+0000) 2022-09-08T04:10:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:02.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:02.928683+0000) 2022-09-08T04:10:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:03 smithi055 ceph-mon[97033]: pgmap v12669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:10:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:03 smithi059 ceph-mon[97450]: pgmap v12669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:10:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:03.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:03.947471+0000) 2022-09-08T04:10:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:03.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:03.928862+0000) 2022-09-08T04:10:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:04.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:04.947623+0000) 2022-09-08T04:10:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:04.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:04.929047+0000) 2022-09-08T04:10:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:05 smithi055 ceph-mon[97033]: pgmap v12670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:10:05.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:05 smithi059 ceph-mon[97450]: pgmap v12670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:10:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:05.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:05.947764+0000) 2022-09-08T04:10:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:05.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:05.929268+0000) 2022-09-08T04:10:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:10:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:10:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:10:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:10:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:06.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:06.947994+0000) 2022-09-08T04:10:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:06.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:06.929501+0000) 2022-09-08T04:10:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:07 smithi055 ceph-mon[97033]: pgmap v12671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:10:07.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:07 smithi059 ceph-mon[97450]: pgmap v12671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:10:08.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:07.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:07.929714+0000) 2022-09-08T04:10:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:07.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:07.948200+0000) 2022-09-08T04:10:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:08.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:08.948434+0000) 2022-09-08T04:10:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:08.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:08.929898+0000) 2022-09-08T04:10:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:09 smithi055 ceph-mon[97033]: pgmap v12672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:10:10.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:09 smithi059 ceph-mon[97450]: pgmap v12672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:10:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:09.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:09.948610+0000) 2022-09-08T04:10:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:09.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:09.930047+0000) 2022-09-08T04:10:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:10 smithi055 ceph-mon[97033]: osdmap e872: 12 total, 11 up, 11 in 2022-09-08T04:10:11.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:10 smithi059 ceph-mon[97450]: osdmap e872: 12 total, 11 up, 11 in 2022-09-08T04:10:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:10.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:10.948815+0000) 2022-09-08T04:10:11.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:10.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:10.930211+0000) 2022-09-08T04:10:11.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:11 smithi055 ceph-mon[97033]: pgmap v12674: 129 pgs: 2 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:10:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:11 smithi059 ceph-mon[97450]: pgmap v12674: 129 pgs: 2 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:10:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:11.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:11.949057+0000) 2022-09-08T04:10:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:11.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:11.930407+0000) 2022-09-08T04:10:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:12.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:12.949253+0000) 2022-09-08T04:10:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:12.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:12.930622+0000) 2022-09-08T04:10:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:13 smithi055 ceph-mon[97033]: pgmap v12675: 129 pgs: 2 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:10:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:13 smithi059 ceph-mon[97450]: pgmap v12675: 129 pgs: 2 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:10:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:13.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:13.949470+0000) 2022-09-08T04:10:14.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:13.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:13.930875+0000) 2022-09-08T04:10:15.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:10:14.893+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:10:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:14.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:14.931030+0000) 2022-09-08T04:10:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:14.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:14.949748+0000) 2022-09-08T04:10:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:15 smithi055 ceph-mon[97033]: osdmap e873: 12 total, 11 up, 11 in 2022-09-08T04:10:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:15 smithi055 ceph-mon[97033]: pgmap v12677: 129 pgs: 2 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T04:10:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:15 smithi059 ceph-mon[97450]: osdmap e873: 12 total, 11 up, 11 in 2022-09-08T04:10:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:15 smithi059 ceph-mon[97450]: pgmap v12677: 129 pgs: 2 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T04:10:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:15.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:15.949947+0000) 2022-09-08T04:10:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:15.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:15.931260+0000) 2022-09-08T04:10:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:16.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:16.950123+0000) 2022-09-08T04:10:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:16.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:16.931435+0000) 2022-09-08T04:10:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:17 smithi055 ceph-mon[97033]: pgmap v12678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:10:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:17 smithi059 ceph-mon[97450]: pgmap v12678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:10:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:17.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:17.950308+0000) 2022-09-08T04:10:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:17.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:17.931717+0000) 2022-09-08T04:10:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:18.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:18.950525+0000) 2022-09-08T04:10:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:18.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:18.931991+0000) 2022-09-08T04:10:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:19 smithi055 ceph-mon[97033]: pgmap v12679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T04:10:19.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:19 smithi055 ceph-mon[97033]: osdmap e874: 12 total, 11 up, 11 in 2022-09-08T04:10:20.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:19 smithi059 ceph-mon[97450]: pgmap v12679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T04:10:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:19 smithi059 ceph-mon[97450]: osdmap e874: 12 total, 11 up, 11 in 2022-09-08T04:10:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:19.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:19.950772+0000) 2022-09-08T04:10:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:19.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:19.932244+0000) 2022-09-08T04:10:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:20.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:20.951001+0000) 2022-09-08T04:10:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:20.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:20.932489+0000) 2022-09-08T04:10:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:21 smithi055 ceph-mon[97033]: pgmap v12681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T04:10:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:21 smithi059 ceph-mon[97450]: pgmap v12681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T04:10:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:21.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:21.932739+0000) 2022-09-08T04:10:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:21.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:21.951235+0000) 2022-09-08T04:10:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:22.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:22.951485+0000) 2022-09-08T04:10:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:22.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:22.932876+0000) 2022-09-08T04:10:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:23 smithi055 ceph-mon[97033]: pgmap v12682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T04:10:24.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:23 smithi059 ceph-mon[97450]: pgmap v12682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T04:10:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:23.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:23.951733+0000) 2022-09-08T04:10:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:23.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:23.933114+0000) 2022-09-08T04:10:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:24.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:24.951987+0000) 2022-09-08T04:10:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:24.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:24.933366+0000) 2022-09-08T04:10:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:25 smithi055 ceph-mon[97033]: pgmap v12683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:10:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:25 smithi059 ceph-mon[97450]: pgmap v12683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:10:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:25.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:25.952262+0000) 2022-09-08T04:10:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:25.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:25.933600+0000) 2022-09-08T04:10:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:26.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:26.952458+0000) 2022-09-08T04:10:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:26.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:26.933873+0000) 2022-09-08T04:10:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:27 smithi055 ceph-mon[97033]: pgmap v12684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:10:28.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:27 smithi059 ceph-mon[97450]: pgmap v12684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:10:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:27.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:27.952672+0000) 2022-09-08T04:10:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:27.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:27.934126+0000) 2022-09-08T04:10:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:28.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:28.952908+0000) 2022-09-08T04:10:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:28.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:28.934309+0000) 2022-09-08T04:10:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:29 smithi055 ceph-mon[97033]: pgmap v12685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:10:30.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:29 smithi059 ceph-mon[97450]: pgmap v12685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:10:30.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:10:29.900+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:10:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:29.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:29.953091+0000) 2022-09-08T04:10:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:29.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:29.934522+0000) 2022-09-08T04:10:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:30.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:30.953322+0000) 2022-09-08T04:10:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:30.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:30.934734+0000) 2022-09-08T04:10:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:31 smithi055 ceph-mon[97033]: pgmap v12686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T04:10:32.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:31 smithi059 ceph-mon[97450]: pgmap v12686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T04:10:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:31.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:31.953533+0000) 2022-09-08T04:10:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:31.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:31.934963+0000) 2022-09-08T04:10:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:32.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:32.953720+0000) 2022-09-08T04:10:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:32.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:32.935183+0000) 2022-09-08T04:10:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:33 smithi055 ceph-mon[97033]: pgmap v12687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:34.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:33 smithi059 ceph-mon[97450]: pgmap v12687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:33.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:33.953955+0000) 2022-09-08T04:10:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:33.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:33.935337+0000) 2022-09-08T04:10:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:34.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:34.954176+0000) 2022-09-08T04:10:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:34.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:34.935499+0000) 2022-09-08T04:10:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:35 smithi055 ceph-mon[97033]: pgmap v12688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:35 smithi059 ceph-mon[97450]: pgmap v12688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:35.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:35.954406+0000) 2022-09-08T04:10:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:35.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:35.935748+0000) 2022-09-08T04:10:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:36.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:36.954584+0000) 2022-09-08T04:10:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:36.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:36.935943+0000) 2022-09-08T04:10:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:37 smithi055 ceph-mon[97033]: pgmap v12689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:10:38.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:37 smithi059 ceph-mon[97450]: pgmap v12689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:10:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:37.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:37.954810+0000) 2022-09-08T04:10:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:37.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:37.936184+0000) 2022-09-08T04:10:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:38.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:38.955037+0000) 2022-09-08T04:10:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:38.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:38.936365+0000) 2022-09-08T04:10:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:39 smithi055 ceph-mon[97033]: pgmap v12690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:39 smithi059 ceph-mon[97450]: pgmap v12690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:39.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:39.955280+0000) 2022-09-08T04:10:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:39.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:39.936596+0000) 2022-09-08T04:10:41.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:40.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:40.955420+0000) 2022-09-08T04:10:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:40.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:40.936841+0000) 2022-09-08T04:10:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:41 smithi055 ceph-mon[97033]: pgmap v12691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:10:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:10:41.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:10:41.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:10:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:41 smithi059 ceph-mon[97450]: pgmap v12691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:10:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:10:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:10:42.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:10:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:41.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:41.955698+0000) 2022-09-08T04:10:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:41.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:41.937025+0000) 2022-09-08T04:10:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:42.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:42.955842+0000) 2022-09-08T04:10:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:42.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:42.937158+0000) 2022-09-08T04:10:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:43 smithi055 ceph-mon[97033]: pgmap v12692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:43 smithi059 ceph-mon[97450]: pgmap v12692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:43.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:43.956070+0000) 2022-09-08T04:10:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:43.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:43.937394+0000) 2022-09-08T04:10:45.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:10:44.905+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:10:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:44.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:44.956275+0000) 2022-09-08T04:10:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:44.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:44.937607+0000) 2022-09-08T04:10:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:45 smithi055 ceph-mon[97033]: pgmap v12693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:45 smithi059 ceph-mon[97450]: pgmap v12693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:46.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:45.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:45.937870+0000) 2022-09-08T04:10:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:45.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:45.956507+0000) 2022-09-08T04:10:47.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:46.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:46.938026+0000) 2022-09-08T04:10:47.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:46.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:46.956720+0000) 2022-09-08T04:10:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:47 smithi055 ceph-mon[97033]: pgmap v12694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:10:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:47 smithi059 ceph-mon[97450]: pgmap v12694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:10:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:47.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:47.956893+0000) 2022-09-08T04:10:48.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:47.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:47.938277+0000) 2022-09-08T04:10:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:48.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:48.957121+0000) 2022-09-08T04:10:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:48.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:48.938498+0000) 2022-09-08T04:10:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:49 smithi055 ceph-mon[97033]: pgmap v12695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:49 smithi059 ceph-mon[97450]: pgmap v12695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:49.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:49.957373+0000) 2022-09-08T04:10:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:49.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:49.938723+0000) 2022-09-08T04:10:51.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:50.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:50.938904+0000) 2022-09-08T04:10:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:50.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:50.957608+0000) 2022-09-08T04:10:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:51 smithi055 ceph-mon[97033]: pgmap v12696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:10:52.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:51 smithi059 ceph-mon[97450]: pgmap v12696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:10:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:51.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:51.957860+0000) 2022-09-08T04:10:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:51.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:51.939158+0000) 2022-09-08T04:10:53.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:52.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:52.958002+0000) 2022-09-08T04:10:53.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:52.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:52.939420+0000) 2022-09-08T04:10:53.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:53 smithi055 ceph-mon[97033]: pgmap v12697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:53 smithi059 ceph-mon[97450]: pgmap v12697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:53.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:53.958214+0000) 2022-09-08T04:10:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:53.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:53.939683+0000) 2022-09-08T04:10:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:54.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:54.958460+0000) 2022-09-08T04:10:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:54.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:54.939899+0000) 2022-09-08T04:10:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:55 smithi055 ceph-mon[97033]: pgmap v12698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:55.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:55 smithi059 ceph-mon[97450]: pgmap v12698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:10:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:55.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:55.958735+0000) 2022-09-08T04:10:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:55.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:55.940130+0000) 2022-09-08T04:10:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:56.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:56.958948+0000) 2022-09-08T04:10:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:56.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:56.940377+0000) 2022-09-08T04:10:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:57 smithi055 ceph-mon[97033]: pgmap v12699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:10:58.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:57 smithi059 ceph-mon[97450]: pgmap v12699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:10:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:57.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:57.959158+0000) 2022-09-08T04:10:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:57.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:57.940623+0000) 2022-09-08T04:10:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:58.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:58.959353+0000) 2022-09-08T04:10:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:58.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:58.940927+0000) 2022-09-08T04:10:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:10:59 smithi055 ceph-mon[97033]: pgmap v12700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:10:59 smithi059 ceph-mon[97450]: pgmap v12700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:00.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:10:59.907+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:11:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:10:59.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:59.941172+0000) 2022-09-08T04:11:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:10:59.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:10:59.959586+0000) 2022-09-08T04:11:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:00.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:00.959841+0000) 2022-09-08T04:11:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:00.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:00.941359+0000) 2022-09-08T04:11:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:01 smithi055 ceph-mon[97033]: pgmap v12701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:11:02.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:01 smithi059 ceph-mon[97450]: pgmap v12701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:11:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:01.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:01.941551+0000) 2022-09-08T04:11:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:01.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:01.960042+0000) 2022-09-08T04:11:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:02.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:02.960278+0000) 2022-09-08T04:11:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:02.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:02.941766+0000) 2022-09-08T04:11:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:03 smithi055 ceph-mon[97033]: pgmap v12702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:04.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:03 smithi059 ceph-mon[97450]: pgmap v12702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:03.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:03.960415+0000) 2022-09-08T04:11:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:03.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:03.942025+0000) 2022-09-08T04:11:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:04.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:04.960562+0000) 2022-09-08T04:11:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:04.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:04.942239+0000) 2022-09-08T04:11:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:05 smithi055 ceph-mon[97033]: pgmap v12703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:05.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:05 smithi059 ceph-mon[97450]: pgmap v12703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:05.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:05.960798+0000) 2022-09-08T04:11:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:05.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:05.942468+0000) 2022-09-08T04:11:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:11:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:11:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:11:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:11:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:06.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:06.961051+0000) 2022-09-08T04:11:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:06.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:06.942675+0000) 2022-09-08T04:11:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:07 smithi055 ceph-mon[97033]: pgmap v12704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:11:07.792 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:07 smithi059 ceph-mon[97450]: pgmap v12704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:11:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:07.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:07.961259+0000) 2022-09-08T04:11:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:07.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:07.942869+0000) 2022-09-08T04:11:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:08.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:08.961453+0000) 2022-09-08T04:11:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:08.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:08.943122+0000) 2022-09-08T04:11:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:09 smithi055 ceph-mon[97033]: pgmap v12705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:11:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:09 smithi059 ceph-mon[97450]: pgmap v12705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:11:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:09.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:09.961627+0000) 2022-09-08T04:11:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:09.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:09.943362+0000) 2022-09-08T04:11:10.955 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:10 smithi059 ceph-mon[97450]: osdmap e875: 12 total, 11 up, 11 in 2022-09-08T04:11:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:10 smithi055 ceph-mon[97033]: osdmap e875: 12 total, 11 up, 11 in 2022-09-08T04:11:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:10.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:10.961802+0000) 2022-09-08T04:11:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:10.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:10.943579+0000) 2022-09-08T04:11:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:11 smithi055 ceph-mon[97033]: pgmap v12707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:11:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:11:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:11 smithi059 ceph-mon[97450]: pgmap v12707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:11:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:11:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:11.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:11.962037+0000) 2022-09-08T04:11:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:11.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:11.943817+0000) 2022-09-08T04:11:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:12 smithi055 ceph-mon[97033]: pgmap v12708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T04:11:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:12 smithi055 ceph-mon[97033]: pgmap v12709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T04:11:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:12 smithi059 ceph-mon[97450]: pgmap v12708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T04:11:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:12 smithi059 ceph-mon[97450]: pgmap v12709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T04:11:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:12.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:12.962267+0000) 2022-09-08T04:11:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:12.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:12.944057+0000) 2022-09-08T04:11:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:13.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:13.962492+0000) 2022-09-08T04:11:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:13.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:13.944309+0000) 2022-09-08T04:11:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:14 smithi055 ceph-mon[97033]: pgmap v12710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:11:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:14 smithi059 ceph-mon[97450]: pgmap v12710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:11:15.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:11:14.919+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:11:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:14.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:14.962691+0000) 2022-09-08T04:11:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:14.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:14.944578+0000) 2022-09-08T04:11:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:15.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:15.944798+0000) 2022-09-08T04:11:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:15.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:15.962888+0000) 2022-09-08T04:11:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:16 smithi055 ceph-mon[97033]: pgmap v12711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.1 KiB/s rd, 327 B/s wr, 9 op/s 2022-09-08T04:11:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:16 smithi059 ceph-mon[97450]: pgmap v12711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.1 KiB/s rd, 327 B/s wr, 9 op/s 2022-09-08T04:11:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:16.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:16.963075+0000) 2022-09-08T04:11:17.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:16.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:16.945025+0000) 2022-09-08T04:11:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:17.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:17.963311+0000) 2022-09-08T04:11:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:17.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:17.945264+0000) 2022-09-08T04:11:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:18 smithi055 ceph-mon[97033]: pgmap v12712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 284 B/s wr, 8 op/s 2022-09-08T04:11:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:18 smithi059 ceph-mon[97450]: pgmap v12712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 284 B/s wr, 8 op/s 2022-09-08T04:11:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:18.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:18.963519+0000) 2022-09-08T04:11:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:18.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:18.945528+0000) 2022-09-08T04:11:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:19.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:19.963756+0000) 2022-09-08T04:11:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:19.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:19.945780+0000) 2022-09-08T04:11:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:20 smithi055 ceph-mon[97033]: pgmap v12713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 123 B/s wr, 3 op/s 2022-09-08T04:11:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:20 smithi055 ceph-mon[97033]: osdmap e876: 12 total, 11 up, 11 in 2022-09-08T04:11:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:20 smithi059 ceph-mon[97450]: pgmap v12713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 123 B/s wr, 3 op/s 2022-09-08T04:11:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:20 smithi059 ceph-mon[97450]: osdmap e876: 12 total, 11 up, 11 in 2022-09-08T04:11:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:20.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:20.963986+0000) 2022-09-08T04:11:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:20.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:20.945928+0000) 2022-09-08T04:11:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:21.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:21.964212+0000) 2022-09-08T04:11:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:21.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:21.946097+0000) 2022-09-08T04:11:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:22 smithi055 ceph-mon[97033]: pgmap v12715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:11:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:22 smithi059 ceph-mon[97450]: pgmap v12715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:11:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:22.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:22.964460+0000) 2022-09-08T04:11:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:22.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:22.946291+0000) 2022-09-08T04:11:24.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:23.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:23.964695+0000) 2022-09-08T04:11:24.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:23.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:23.946520+0000) 2022-09-08T04:11:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:24 smithi055 ceph-mon[97033]: pgmap v12716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:11:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:24 smithi059 ceph-mon[97450]: pgmap v12716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:11:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:24.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:24.964823+0000) 2022-09-08T04:11:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:24.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:24.946699+0000) 2022-09-08T04:11:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:25.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:25.965008+0000) 2022-09-08T04:11:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:25.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:25.946924+0000) 2022-09-08T04:11:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:26 smithi055 ceph-mon[97033]: pgmap v12717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:11:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:26 smithi059 ceph-mon[97450]: pgmap v12717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:11:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:26.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:26.965158+0000) 2022-09-08T04:11:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:26.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:26.947104+0000) 2022-09-08T04:11:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:27.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:27.965366+0000) 2022-09-08T04:11:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:27.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:27.947304+0000) 2022-09-08T04:11:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:28 smithi055 ceph-mon[97033]: pgmap v12718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:11:28.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:28 smithi059 ceph-mon[97450]: pgmap v12718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:11:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:28.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:28.965568+0000) 2022-09-08T04:11:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:28.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:28.947518+0000) 2022-09-08T04:11:30.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:11:29.935+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:11:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:29.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:29.965788+0000) 2022-09-08T04:11:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:29.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:29.947742+0000) 2022-09-08T04:11:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:30 smithi055 ceph-mon[97033]: pgmap v12719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:11:30.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:30 smithi059 ceph-mon[97450]: pgmap v12719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:11:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:30.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:30.966005+0000) 2022-09-08T04:11:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:30.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:30.947922+0000) 2022-09-08T04:11:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:31.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:31.966193+0000) 2022-09-08T04:11:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:31.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:31.948129+0000) 2022-09-08T04:11:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:32 smithi055 ceph-mon[97033]: pgmap v12720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:11:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:32 smithi059 ceph-mon[97450]: pgmap v12720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:11:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:32.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:32.966420+0000) 2022-09-08T04:11:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:32.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:32.948319+0000) 2022-09-08T04:11:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:33.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:33.966657+0000) 2022-09-08T04:11:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:33.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:33.948543+0000) 2022-09-08T04:11:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:34 smithi055 ceph-mon[97033]: pgmap v12721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:34.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:34 smithi059 ceph-mon[97450]: pgmap v12721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:34.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:34.948775+0000) 2022-09-08T04:11:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:34.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:34.966850+0000) 2022-09-08T04:11:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:35.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:35.967116+0000) 2022-09-08T04:11:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:35.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:35.949014+0000) 2022-09-08T04:11:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:36 smithi055 ceph-mon[97033]: pgmap v12722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:11:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:36 smithi059 ceph-mon[97450]: pgmap v12722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:11:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:36.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:36.967377+0000) 2022-09-08T04:11:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:36.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:36.949224+0000) 2022-09-08T04:11:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:37.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:37.967557+0000) 2022-09-08T04:11:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:37.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:37.949432+0000) 2022-09-08T04:11:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:38 smithi055 ceph-mon[97033]: pgmap v12723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:11:38.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:38 smithi059 ceph-mon[97450]: pgmap v12723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:11:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:38.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:38.967782+0000) 2022-09-08T04:11:39.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:38.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:38.949726+0000) 2022-09-08T04:11:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:39.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:39.967982+0000) 2022-09-08T04:11:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:39.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:39.949975+0000) 2022-09-08T04:11:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:40 smithi055 ceph-mon[97033]: pgmap v12724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:40.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:40 smithi059 ceph-mon[97450]: pgmap v12724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:41.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:40.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:40.950212+0000) 2022-09-08T04:11:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:40.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:40.968210+0000) 2022-09-08T04:11:42.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:41.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:41.950378+0000) 2022-09-08T04:11:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:41.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:41.968423+0000) 2022-09-08T04:11:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:42 smithi055 ceph-mon[97033]: pgmap v12725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:11:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:42 smithi059 ceph-mon[97450]: pgmap v12725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:11:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:42.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:42.950574+0000) 2022-09-08T04:11:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:42.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:42.968647+0000) 2022-09-08T04:11:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:43.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:43.968776+0000) 2022-09-08T04:11:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:43.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:43.950853+0000) 2022-09-08T04:11:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:44 smithi055 ceph-mon[97033]: pgmap v12726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:44 smithi059 ceph-mon[97450]: pgmap v12726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:45.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:11:44.951+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:11:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:44.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:44.951096+0000) 2022-09-08T04:11:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:44.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:44.969025+0000) 2022-09-08T04:11:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:45.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:45.969231+0000) 2022-09-08T04:11:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:45.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:45.951319+0000) 2022-09-08T04:11:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:46 smithi055 ceph-mon[97033]: pgmap v12727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:11:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:46 smithi059 ceph-mon[97450]: pgmap v12727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:11:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:46.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:46.969432+0000) 2022-09-08T04:11:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:46.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:46.951539+0000) 2022-09-08T04:11:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:47.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:47.969625+0000) 2022-09-08T04:11:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:47.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:47.951763+0000) 2022-09-08T04:11:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:48 smithi055 ceph-mon[97033]: pgmap v12728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:11:48.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:48 smithi059 ceph-mon[97450]: pgmap v12728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:11:49.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:48.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:48.951971+0000) 2022-09-08T04:11:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:48.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:48.969878+0000) 2022-09-08T04:11:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:49.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:49.970115+0000) 2022-09-08T04:11:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:49.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:49.952188+0000) 2022-09-08T04:11:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:50 smithi055 ceph-mon[97033]: pgmap v12729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:50 smithi059 ceph-mon[97450]: pgmap v12729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:50.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:50.970362+0000) 2022-09-08T04:11:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:50.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:50.952410+0000) 2022-09-08T04:11:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:51.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:51.970597+0000) 2022-09-08T04:11:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:51.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:51.952605+0000) 2022-09-08T04:11:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:52 smithi055 ceph-mon[97033]: pgmap v12730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:11:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:52 smithi059 ceph-mon[97450]: pgmap v12730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:11:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:52.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:52.970793+0000) 2022-09-08T04:11:53.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:52.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:52.952848+0000) 2022-09-08T04:11:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:53.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:53.970974+0000) 2022-09-08T04:11:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:53.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:53.953065+0000) 2022-09-08T04:11:54.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:54 smithi055 ceph-mon[97033]: pgmap v12731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:54 smithi059 ceph-mon[97450]: pgmap v12731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:11:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:54.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:54.971167+0000) 2022-09-08T04:11:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:54.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:54.953284+0000) 2022-09-08T04:11:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:55.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:55.971425+0000) 2022-09-08T04:11:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:55.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:55.953469+0000) 2022-09-08T04:11:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:56 smithi055 ceph-mon[97033]: pgmap v12732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:11:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:56 smithi059 ceph-mon[97450]: pgmap v12732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:11:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:56.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:56.971631+0000) 2022-09-08T04:11:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:56.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:56.953702+0000) 2022-09-08T04:11:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:57.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:57.971900+0000) 2022-09-08T04:11:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:57.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:57.953900+0000) 2022-09-08T04:11:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:11:58 smithi055 ceph-mon[97033]: pgmap v12733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:11:58.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:11:58 smithi059 ceph-mon[97450]: pgmap v12733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:11:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:58.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:58.972094+0000) 2022-09-08T04:11:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:58.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:58.954084+0000) 2022-09-08T04:12:00.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:11:59.965+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:12:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:11:59.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:59.972324+0000) 2022-09-08T04:12:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:11:59.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:11:59.954372+0000) 2022-09-08T04:12:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:00 smithi055 ceph-mon[97033]: pgmap v12734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:12:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:00 smithi059 ceph-mon[97450]: pgmap v12734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:12:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:00.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:00.972483+0000) 2022-09-08T04:12:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:00.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:00.954597+0000) 2022-09-08T04:12:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:01.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:01.972658+0000) 2022-09-08T04:12:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:01.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:01.954828+0000) 2022-09-08T04:12:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:02 smithi055 ceph-mon[97033]: pgmap v12735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:12:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:02 smithi059 ceph-mon[97450]: pgmap v12735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:12:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:02.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:02.972830+0000) 2022-09-08T04:12:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:02.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:02.955059+0000) 2022-09-08T04:12:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:03.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:03.973014+0000) 2022-09-08T04:12:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:03.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:03.955305+0000) 2022-09-08T04:12:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:04 smithi059 ceph-mon[97450]: pgmap v12736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:12:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:04 smithi055 ceph-mon[97033]: pgmap v12736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:12:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:04.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:04.973236+0000) 2022-09-08T04:12:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:04.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:04.955539+0000) 2022-09-08T04:12:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:05.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:05.973478+0000) 2022-09-08T04:12:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:05.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:05.955770+0000) 2022-09-08T04:12:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:06 smithi059 ceph-mon[97450]: pgmap v12737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T04:12:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:12:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:12:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:06 smithi055 ceph-mon[97033]: pgmap v12737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T04:12:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:12:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:12:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:06.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:06.973684+0000) 2022-09-08T04:12:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:06.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:06.955988+0000) 2022-09-08T04:12:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:07.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:07.973851+0000) 2022-09-08T04:12:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:07.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:07.956167+0000) 2022-09-08T04:12:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:08 smithi059 ceph-mon[97450]: pgmap v12738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T04:12:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:08 smithi055 ceph-mon[97033]: pgmap v12738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T04:12:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:08.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:08.974006+0000) 2022-09-08T04:12:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:08.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:08.956411+0000) 2022-09-08T04:12:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:09.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:09.974217+0000) 2022-09-08T04:12:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:09.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:09.956665+0000) 2022-09-08T04:12:10.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:10 smithi059 ceph-mon[97450]: pgmap v12739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:12:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:10 smithi059 ceph-mon[97450]: osdmap e877: 12 total, 11 up, 11 in 2022-09-08T04:12:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:10 smithi055 ceph-mon[97033]: pgmap v12739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:12:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:10 smithi055 ceph-mon[97033]: osdmap e877: 12 total, 11 up, 11 in 2022-09-08T04:12:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:10.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:10.974515+0000) 2022-09-08T04:12:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:10.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:10.956875+0000) 2022-09-08T04:12:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:12:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:12:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:12:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:12:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:12:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:12:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:11.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:11.974719+0000) 2022-09-08T04:12:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:11.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:11.957096+0000) 2022-09-08T04:12:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:12 smithi059 ceph-mon[97450]: pgmap v12741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:12:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:12 smithi055 ceph-mon[97033]: pgmap v12741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:12:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:12.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:12.974930+0000) 2022-09-08T04:12:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:12.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:12.957347+0000) 2022-09-08T04:12:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:13.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:13.975133+0000) 2022-09-08T04:12:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:13.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:13.957565+0000) 2022-09-08T04:12:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:14 smithi059 ceph-mon[97450]: pgmap v12742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:12:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:14 smithi055 ceph-mon[97033]: pgmap v12742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:12:15.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:12:14.973+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:12:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:14.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:14.975273+0000) 2022-09-08T04:12:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:14.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:14.957753+0000) 2022-09-08T04:12:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:15.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:15.975492+0000) 2022-09-08T04:12:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:15.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:15.957919+0000) 2022-09-08T04:12:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:16 smithi059 ceph-mon[97450]: pgmap v12743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:12:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:16 smithi055 ceph-mon[97033]: pgmap v12743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:12:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:16.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:16.975681+0000) 2022-09-08T04:12:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:16.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:16.958083+0000) 2022-09-08T04:12:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:17.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:17.975930+0000) 2022-09-08T04:12:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:17.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:17.958279+0000) 2022-09-08T04:12:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:18 smithi059 ceph-mon[97450]: pgmap v12744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:12:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:18 smithi055 ceph-mon[97033]: pgmap v12744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:12:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:18.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:18.976105+0000) 2022-09-08T04:12:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:18.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:18.958443+0000) 2022-09-08T04:12:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:19.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:19.976306+0000) 2022-09-08T04:12:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:19.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:19.958749+0000) 2022-09-08T04:12:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:20 smithi055 ceph-mon[97033]: pgmap v12745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:12:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:20 smithi055 ceph-mon[97033]: osdmap e878: 12 total, 11 up, 11 in 2022-09-08T04:12:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:20 smithi059 ceph-mon[97450]: pgmap v12745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:12:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:20 smithi059 ceph-mon[97450]: osdmap e878: 12 total, 11 up, 11 in 2022-09-08T04:12:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:20.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:20.976532+0000) 2022-09-08T04:12:21.447 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:20.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:20.958880+0000) 2022-09-08T04:12:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:21.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:21.976762+0000) 2022-09-08T04:12:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:21.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:21.959065+0000) 2022-09-08T04:12:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:22 smithi055 ceph-mon[97033]: pgmap v12747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:12:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:22 smithi059 ceph-mon[97450]: pgmap v12747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:12:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:22.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:22.976990+0000) 2022-09-08T04:12:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:22.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:22.959291+0000) 2022-09-08T04:12:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:23.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:23.977228+0000) 2022-09-08T04:12:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:23.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:23.959560+0000) 2022-09-08T04:12:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:24 smithi055 ceph-mon[97033]: pgmap v12748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:12:24.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:24 smithi059 ceph-mon[97450]: pgmap v12748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:12:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:24.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:24.977477+0000) 2022-09-08T04:12:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:24.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:24.959710+0000) 2022-09-08T04:12:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:25.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:25.977664+0000) 2022-09-08T04:12:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:25.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:25.959852+0000) 2022-09-08T04:12:26.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:26 smithi055 ceph-mon[97033]: pgmap v12749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:12:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:26 smithi059 ceph-mon[97450]: pgmap v12749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:12:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:26.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:26.977892+0000) 2022-09-08T04:12:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:26.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:26.960027+0000) 2022-09-08T04:12:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:27.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:27.978079+0000) 2022-09-08T04:12:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:27.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:27.960208+0000) 2022-09-08T04:12:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:28 smithi055 ceph-mon[97033]: pgmap v12750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:12:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:28 smithi059 ceph-mon[97450]: pgmap v12750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:12:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:28.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:28.978212+0000) 2022-09-08T04:12:29.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:28.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:28.960457+0000) 2022-09-08T04:12:30.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:12:29.989+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:12:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:29.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:29.978432+0000) 2022-09-08T04:12:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:29.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:29.960663+0000) 2022-09-08T04:12:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:30 smithi055 ceph-mon[97033]: pgmap v12751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:12:30.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:30 smithi059 ceph-mon[97450]: pgmap v12751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:12:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:30.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:30.978692+0000) 2022-09-08T04:12:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:30.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:30.960855+0000) 2022-09-08T04:12:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:31.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:31.978974+0000) 2022-09-08T04:12:32.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:31.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:31.960997+0000) 2022-09-08T04:12:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:32 smithi055 ceph-mon[97033]: pgmap v12752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:12:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:32 smithi059 ceph-mon[97450]: pgmap v12752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:12:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:32.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:32.979148+0000) 2022-09-08T04:12:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:32.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:32.961175+0000) 2022-09-08T04:12:34.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:33.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:33.961407+0000) 2022-09-08T04:12:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:33.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:33.979322+0000) 2022-09-08T04:12:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:34 smithi055 ceph-mon[97033]: pgmap v12753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:12:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:34 smithi059 ceph-mon[97450]: pgmap v12753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:12:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:34.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:34.979565+0000) 2022-09-08T04:12:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:34.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:34.961694+0000) 2022-09-08T04:12:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:35.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:35.979840+0000) 2022-09-08T04:12:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:35.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:35.961908+0000) 2022-09-08T04:12:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:36 smithi055 ceph-mon[97033]: pgmap v12754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:12:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:36 smithi059 ceph-mon[97450]: pgmap v12754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:12:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:36.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:36.979986+0000) 2022-09-08T04:12:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:36.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:36.962087+0000) 2022-09-08T04:12:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:37.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:37.980136+0000) 2022-09-08T04:12:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:37.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:37.962258+0000) 2022-09-08T04:12:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:38 smithi055 ceph-mon[97033]: pgmap v12755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:12:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:38 smithi059 ceph-mon[97450]: pgmap v12755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:12:39.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:38.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:38.962477+0000) 2022-09-08T04:12:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:38.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:38.980370+0000) 2022-09-08T04:12:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:39.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:39.980687+0000) 2022-09-08T04:12:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:39.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:39.962739+0000) 2022-09-08T04:12:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:40 smithi055 ceph-mon[97033]: pgmap v12756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:12:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:40 smithi059 ceph-mon[97450]: pgmap v12756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:12:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:40.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:40.980850+0000) 2022-09-08T04:12:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:40.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:40.962872+0000) 2022-09-08T04:12:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:41.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:41.981082+0000) 2022-09-08T04:12:42.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:41.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:41.963016+0000) 2022-09-08T04:12:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:42 smithi055 ceph-mon[97033]: pgmap v12757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:12:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:42 smithi055 ceph-mon[97033]: pgmap v12758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:12:42.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:42 smithi055 ceph-mon[97033]: pgmap v12759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:12:42.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:12:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:42 smithi059 ceph-mon[97450]: pgmap v12757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:12:42.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:42 smithi059 ceph-mon[97450]: pgmap v12758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:12:42.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:42 smithi059 ceph-mon[97450]: pgmap v12759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:12:42.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:12:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:42.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:42.981270+0000) 2022-09-08T04:12:43.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:42.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:42.963209+0000) 2022-09-08T04:12:44.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:43.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:43.963429+0000) 2022-09-08T04:12:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:43.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:43.981524+0000) 2022-09-08T04:12:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:44 smithi055 ceph-mon[97033]: pgmap v12760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-08T04:12:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:44 smithi059 ceph-mon[97450]: pgmap v12760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 4 op/s 2022-09-08T04:12:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:44.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:44.963704+0000) 2022-09-08T04:12:45.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:12:44.999+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:12:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:44.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:44.981801+0000) 2022-09-08T04:12:46.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:45.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:45.963941+0000) 2022-09-08T04:12:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:45.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:45.982021+0000) 2022-09-08T04:12:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:46 smithi055 ceph-mon[97033]: pgmap v12761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:12:46.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:46 smithi059 ceph-mon[97450]: pgmap v12761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:12:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:46.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:46.982305+0000) 2022-09-08T04:12:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:46.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:46.964153+0000) 2022-09-08T04:12:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:47.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:47.982577+0000) 2022-09-08T04:12:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:47.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:47.964367+0000) 2022-09-08T04:12:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:48 smithi055 ceph-mon[97033]: pgmap v12762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:12:48.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:48 smithi059 ceph-mon[97450]: pgmap v12762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:12:49.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:48.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:48.964554+0000) 2022-09-08T04:12:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:48.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:48.982816+0000) 2022-09-08T04:12:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:49.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:49.983104+0000) 2022-09-08T04:12:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:49.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:49.964779+0000) 2022-09-08T04:12:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:50 smithi055 ceph-mon[97033]: pgmap v12763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:12:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:50 smithi059 ceph-mon[97450]: pgmap v12763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:12:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:50.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:50.983248+0000) 2022-09-08T04:12:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:50.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:50.964936+0000) 2022-09-08T04:12:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:51.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:51.983455+0000) 2022-09-08T04:12:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:51.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:51.965103+0000) 2022-09-08T04:12:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:52 smithi055 ceph-mon[97033]: pgmap v12764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:12:52.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:52 smithi059 ceph-mon[97450]: pgmap v12764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:12:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:52.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:52.983707+0000) 2022-09-08T04:12:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:52.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:52.965330+0000) 2022-09-08T04:12:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:53.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:53.983985+0000) 2022-09-08T04:12:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:53.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:53.965570+0000) 2022-09-08T04:12:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:54 smithi055 ceph-mon[97033]: pgmap v12765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:12:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:54 smithi059 ceph-mon[97450]: pgmap v12765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:12:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:54.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:54.984182+0000) 2022-09-08T04:12:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:54.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:54.965808+0000) 2022-09-08T04:12:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:55.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:55.984388+0000) 2022-09-08T04:12:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:55.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:55.966150+0000) 2022-09-08T04:12:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:56 smithi055 ceph-mon[97033]: pgmap v12766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:12:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:56 smithi059 ceph-mon[97450]: pgmap v12766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:12:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:56.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:56.966332+0000) 2022-09-08T04:12:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:56.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:56.984573+0000) 2022-09-08T04:12:58.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:57.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:57.966564+0000) 2022-09-08T04:12:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:57.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:57.984718+0000) 2022-09-08T04:12:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:12:58 smithi055 ceph-mon[97033]: pgmap v12767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:12:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:12:58 smithi059 ceph-mon[97450]: pgmap v12767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:12:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:58.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:58.984849+0000) 2022-09-08T04:12:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:58.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:58.966813+0000) 2022-09-08T04:13:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:12:59.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:59.985011+0000) 2022-09-08T04:13:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:12:59.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:12:59.967053+0000) 2022-09-08T04:13:00.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:13:00.003+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:13:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:00 smithi055 ceph-mon[97033]: pgmap v12768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:00.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:00 smithi059 ceph-mon[97450]: pgmap v12768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:00.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:00.967204+0000) 2022-09-08T04:13:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:00.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:00.985260+0000) 2022-09-08T04:13:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:01.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:01.967454+0000) 2022-09-08T04:13:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:01.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:01.985494+0000) 2022-09-08T04:13:02.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:02 smithi055 ceph-mon[97033]: pgmap v12769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:13:02.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:02 smithi059 ceph-mon[97450]: pgmap v12769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:13:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:02.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:02.985758+0000) 2022-09-08T04:13:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:02.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:02.967601+0000) 2022-09-08T04:13:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:03.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:03.985933+0000) 2022-09-08T04:13:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:03.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:03.967806+0000) 2022-09-08T04:13:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:04 smithi055 ceph-mon[97033]: pgmap v12770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:04.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:04 smithi059 ceph-mon[97450]: pgmap v12770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:04.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:04.986090+0000) 2022-09-08T04:13:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:04.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:04.967939+0000) 2022-09-08T04:13:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:05.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:05.986265+0000) 2022-09-08T04:13:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:05.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:05.968152+0000) 2022-09-08T04:13:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:06 smithi055 ceph-mon[97033]: pgmap v12771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:13:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:13:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:13:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:06 smithi059 ceph-mon[97450]: pgmap v12771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:13:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:13:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:13:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:06.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:06.986481+0000) 2022-09-08T04:13:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:06.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:06.968380+0000) 2022-09-08T04:13:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:07.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:07.986701+0000) 2022-09-08T04:13:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:07.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:07.968630+0000) 2022-09-08T04:13:08.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:08 smithi055 ceph-mon[97033]: pgmap v12772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:13:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:08 smithi059 ceph-mon[97450]: pgmap v12772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:13:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:08.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:08.986937+0000) 2022-09-08T04:13:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:08.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:08.968863+0000) 2022-09-08T04:13:10.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:09.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:09.969106+0000) 2022-09-08T04:13:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:09.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:09.987113+0000) 2022-09-08T04:13:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:10 smithi055 ceph-mon[97033]: pgmap v12773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:13:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:10 smithi055 ceph-mon[97033]: osdmap e879: 12 total, 11 up, 11 in 2022-09-08T04:13:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:10 smithi059 ceph-mon[97450]: pgmap v12773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:13:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:10 smithi059 ceph-mon[97450]: osdmap e879: 12 total, 11 up, 11 in 2022-09-08T04:13:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:10.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:10.987348+0000) 2022-09-08T04:13:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:10.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:10.969358+0000) 2022-09-08T04:13:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:11.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:11.987613+0000) 2022-09-08T04:13:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:11.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:11.969566+0000) 2022-09-08T04:13:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:12 smithi055 ceph-mon[97033]: pgmap v12775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:13:12.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:12 smithi059 ceph-mon[97450]: pgmap v12775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:13:13.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:12.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:12.969762+0000) 2022-09-08T04:13:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:12.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:12.987903+0000) 2022-09-08T04:13:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:13.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:13.988138+0000) 2022-09-08T04:13:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:13.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:13.969985+0000) 2022-09-08T04:13:14.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:14 smithi059 ceph-mon[97450]: pgmap v12776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:13:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:14 smithi055 ceph-mon[97033]: pgmap v12776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:13:15.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:13:15.019+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:13:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:14.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:14.988342+0000) 2022-09-08T04:13:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:14.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:14.970202+0000) 2022-09-08T04:13:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:15.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:15.988580+0000) 2022-09-08T04:13:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:15.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:15.970438+0000) 2022-09-08T04:13:16.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:16 smithi059 ceph-mon[97450]: pgmap v12777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:13:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:16 smithi055 ceph-mon[97033]: pgmap v12777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:13:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:16.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:16.988798+0000) 2022-09-08T04:13:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:16.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:16.970609+0000) 2022-09-08T04:13:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:17.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:17.988950+0000) 2022-09-08T04:13:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:17.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:17.970833+0000) 2022-09-08T04:13:18.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:18 smithi059 ceph-mon[97450]: pgmap v12778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:13:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:18 smithi055 ceph-mon[97033]: pgmap v12778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:13:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:18.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:18.989134+0000) 2022-09-08T04:13:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:18.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:18.971031+0000) 2022-09-08T04:13:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:19.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:19.989334+0000) 2022-09-08T04:13:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:19.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:19.971292+0000) 2022-09-08T04:13:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:20 smithi055 ceph-mon[97033]: pgmap v12779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:13:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:20 smithi055 ceph-mon[97033]: osdmap e880: 12 total, 11 up, 11 in 2022-09-08T04:13:20.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:20 smithi059 ceph-mon[97450]: pgmap v12779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:13:20.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:20 smithi059 ceph-mon[97450]: osdmap e880: 12 total, 11 up, 11 in 2022-09-08T04:13:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:20.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:20.989551+0000) 2022-09-08T04:13:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:20.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:20.971592+0000) 2022-09-08T04:13:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:21.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:21.989798+0000) 2022-09-08T04:13:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:21.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:21.971821+0000) 2022-09-08T04:13:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:22 smithi055 ceph-mon[97033]: pgmap v12781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:13:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:22 smithi059 ceph-mon[97450]: pgmap v12781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:13:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:22.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:22.990023+0000) 2022-09-08T04:13:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:22.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:22.972051+0000) 2022-09-08T04:13:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:23.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:23.990251+0000) 2022-09-08T04:13:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:23.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:23.972316+0000) 2022-09-08T04:13:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:24 smithi055 ceph-mon[97033]: pgmap v12782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:13:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:24 smithi059 ceph-mon[97450]: pgmap v12782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:13:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:24.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:24.990470+0000) 2022-09-08T04:13:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:24.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:24.972536+0000) 2022-09-08T04:13:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:25.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:25.990697+0000) 2022-09-08T04:13:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:25.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:25.972749+0000) 2022-09-08T04:13:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:26 smithi055 ceph-mon[97033]: pgmap v12783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:13:26.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:26 smithi059 ceph-mon[97450]: pgmap v12783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:13:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:26.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:26.990885+0000) 2022-09-08T04:13:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:26.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:26.973000+0000) 2022-09-08T04:13:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:27.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:27.991084+0000) 2022-09-08T04:13:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:27.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:27.973238+0000) 2022-09-08T04:13:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:28 smithi055 ceph-mon[97033]: pgmap v12784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:13:28.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:28 smithi059 ceph-mon[97450]: pgmap v12784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:13:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:28.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:28.991324+0000) 2022-09-08T04:13:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:28.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:28.973421+0000) 2022-09-08T04:13:30.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:13:30.023+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:13:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:29.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:29.991570+0000) 2022-09-08T04:13:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:29.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:29.973605+0000) 2022-09-08T04:13:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:30 smithi055 ceph-mon[97033]: pgmap v12785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:13:30.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:30 smithi059 ceph-mon[97450]: pgmap v12785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:13:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:30.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:30.991803+0000) 2022-09-08T04:13:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:30.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:30.973807+0000) 2022-09-08T04:13:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:31.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:31.992044+0000) 2022-09-08T04:13:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:31.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:31.974046+0000) 2022-09-08T04:13:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:32 smithi055 ceph-mon[97033]: pgmap v12786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T04:13:32.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:32 smithi059 ceph-mon[97450]: pgmap v12786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T04:13:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:32.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:32.992280+0000) 2022-09-08T04:13:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:32.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:32.974222+0000) 2022-09-08T04:13:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:33.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:33.992520+0000) 2022-09-08T04:13:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:33.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:33.974408+0000) 2022-09-08T04:13:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:34 smithi055 ceph-mon[97033]: pgmap v12787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:34.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:34 smithi059 ceph-mon[97450]: pgmap v12787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:34.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:34.992742+0000) 2022-09-08T04:13:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:34.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:34.974577+0000) 2022-09-08T04:13:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:35.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:35.992924+0000) 2022-09-08T04:13:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:35.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:35.974778+0000) 2022-09-08T04:13:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:36 smithi055 ceph-mon[97033]: pgmap v12788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:13:36.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:36 smithi059 ceph-mon[97450]: pgmap v12788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:13:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:36.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:36.993139+0000) 2022-09-08T04:13:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:36.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:36.975000+0000) 2022-09-08T04:13:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:37.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:37.993370+0000) 2022-09-08T04:13:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:37.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:37.975166+0000) 2022-09-08T04:13:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:38 smithi055 ceph-mon[97033]: pgmap v12789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:38 smithi059 ceph-mon[97450]: pgmap v12789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:38.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:38.993513+0000) 2022-09-08T04:13:39.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:38.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:38.975405+0000) 2022-09-08T04:13:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:39.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:39.993741+0000) 2022-09-08T04:13:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:39.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:39.975611+0000) 2022-09-08T04:13:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:40 smithi055 ceph-mon[97033]: pgmap v12790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:40 smithi059 ceph-mon[97450]: pgmap v12790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:41.285 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:40.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:40.994008+0000) 2022-09-08T04:13:41.287 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:40.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:40.975874+0000) 2022-09-08T04:13:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:13:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:13:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:13:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:13:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:13:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:13:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:41.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:41.994289+0000) 2022-09-08T04:13:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:41.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:41.976049+0000) 2022-09-08T04:13:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:42 smithi055 ceph-mon[97033]: pgmap v12791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:13:42.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:42 smithi059 ceph-mon[97450]: pgmap v12791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:13:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:42.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:42.994505+0000) 2022-09-08T04:13:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:42.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:42.976225+0000) 2022-09-08T04:13:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:43.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:43.994756+0000) 2022-09-08T04:13:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:43.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:43.976438+0000) 2022-09-08T04:13:44.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:44 smithi055 ceph-mon[97033]: pgmap v12792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:44.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:44 smithi059 ceph-mon[97450]: pgmap v12792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:45.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:13:45.039+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:13:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:44.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:44.995003+0000) 2022-09-08T04:13:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:44.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:44.976678+0000) 2022-09-08T04:13:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:45.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:45.995216+0000) 2022-09-08T04:13:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:45.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:45.976893+0000) 2022-09-08T04:13:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:46 smithi055 ceph-mon[97033]: pgmap v12793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:13:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:46 smithi059 ceph-mon[97450]: pgmap v12793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:13:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:46.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:46.995434+0000) 2022-09-08T04:13:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:46.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:46.977113+0000) 2022-09-08T04:13:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:47.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:47.995672+0000) 2022-09-08T04:13:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:47.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:47.977345+0000) 2022-09-08T04:13:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:48 smithi059 ceph-mon[97450]: pgmap v12794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:48 smithi055 ceph-mon[97033]: pgmap v12794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:48.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:48.995945+0000) 2022-09-08T04:13:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:48.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:48.977662+0000) 2022-09-08T04:13:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:49.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:49.996234+0000) 2022-09-08T04:13:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:49.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:49.977933+0000) 2022-09-08T04:13:50.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:50 smithi059 ceph-mon[97450]: pgmap v12795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:50 smithi055 ceph-mon[97033]: pgmap v12795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:50.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:50.996513+0000) 2022-09-08T04:13:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:50.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:50.978096+0000) 2022-09-08T04:13:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:51.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:51.996734+0000) 2022-09-08T04:13:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:51.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:51.978340+0000) 2022-09-08T04:13:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:52 smithi059 ceph-mon[97450]: pgmap v12796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:13:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:52 smithi055 ceph-mon[97033]: pgmap v12796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:13:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:52.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:52.996977+0000) 2022-09-08T04:13:53.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:52.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:52.978550+0000) 2022-09-08T04:13:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:53.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:53.997256+0000) 2022-09-08T04:13:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:53.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:53.978816+0000) 2022-09-08T04:13:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:54 smithi059 ceph-mon[97450]: pgmap v12797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:54 smithi055 ceph-mon[97033]: pgmap v12797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:54.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:54.997536+0000) 2022-09-08T04:13:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:54.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:54.979048+0000) 2022-09-08T04:13:56.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:55.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:55.997716+0000) 2022-09-08T04:13:56.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:55.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:55.979279+0000) 2022-09-08T04:13:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:56 smithi059 ceph-mon[97450]: pgmap v12798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:13:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:56 smithi055 ceph-mon[97033]: pgmap v12798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:13:57.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:56.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:56.997962+0000) 2022-09-08T04:13:57.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:56.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:56.979514+0000) 2022-09-08T04:13:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:57.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:57.998198+0000) 2022-09-08T04:13:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:57.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:57.979780+0000) 2022-09-08T04:13:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:13:58 smithi059 ceph-mon[97450]: pgmap v12799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:13:58 smithi055 ceph-mon[97033]: pgmap v12799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:13:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:58.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:58.998400+0000) 2022-09-08T04:13:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:58.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:58.980051+0000) 2022-09-08T04:14:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:13:59.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:59.998626+0000) 2022-09-08T04:14:00.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:14:00.054+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:14:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:13:59.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:13:59.980278+0000) 2022-09-08T04:14:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:00 smithi059 ceph-mon[97450]: pgmap v12800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:00 smithi055 ceph-mon[97033]: pgmap v12800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:00.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:00.998793+0000) 2022-09-08T04:14:01.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:00.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:00.980421+0000) 2022-09-08T04:14:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:01.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:01.999019+0000) 2022-09-08T04:14:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:01.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:01.980647+0000) 2022-09-08T04:14:02.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:02 smithi059 ceph-mon[97450]: pgmap v12801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:14:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:02 smithi055 ceph-mon[97033]: pgmap v12801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:14:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:02.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:02.999266+0000) 2022-09-08T04:14:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:02.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:02.980799+0000) 2022-09-08T04:14:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:03.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:03.999525+0000) 2022-09-08T04:14:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:03.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:03.981032+0000) 2022-09-08T04:14:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:04 smithi059 ceph-mon[97450]: pgmap v12802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:04 smithi055 ceph-mon[97033]: pgmap v12802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:04.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:04.999764+0000) 2022-09-08T04:14:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:04.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:04.981198+0000) 2022-09-08T04:14:06.358 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:05.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:05.981440+0000) 2022-09-08T04:14:06.359 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:05.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:05.999903+0000) 2022-09-08T04:14:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:06 smithi059 ceph-mon[97450]: pgmap v12803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:14:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:14:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:14:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:06 smithi055 ceph-mon[97033]: pgmap v12803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:14:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:14:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:14:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:06.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:07.000163+0000) 2022-09-08T04:14:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:06.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:06.981715+0000) 2022-09-08T04:14:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:08.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:08.000451+0000) 2022-09-08T04:14:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:07.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:07.981858+0000) 2022-09-08T04:14:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:08 smithi059 ceph-mon[97450]: pgmap v12804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:14:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:08 smithi055 ceph-mon[97033]: pgmap v12804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:14:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:09.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:09.000690+0000) 2022-09-08T04:14:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:08.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:08.982083+0000) 2022-09-08T04:14:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:10.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:10.000913+0000) 2022-09-08T04:14:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:09.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:09.982255+0000) 2022-09-08T04:14:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:10 smithi059 ceph-mon[97450]: pgmap v12805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:14:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:10 smithi059 ceph-mon[97450]: osdmap e881: 12 total, 11 up, 11 in 2022-09-08T04:14:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:10 smithi055 ceph-mon[97033]: pgmap v12805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:14:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:10 smithi055 ceph-mon[97033]: osdmap e881: 12 total, 11 up, 11 in 2022-09-08T04:14:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:11.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:11.001151+0000) 2022-09-08T04:14:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:10.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:10.982515+0000) 2022-09-08T04:14:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:12.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:12.001336+0000) 2022-09-08T04:14:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:11.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:11.982759+0000) 2022-09-08T04:14:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:12 smithi055 ceph-mon[97033]: pgmap v12807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:14:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:12 smithi055 ceph-mon[97033]: pgmap v12808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T04:14:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:12 smithi055 ceph-mon[97033]: pgmap v12809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:14:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:14:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:12 smithi059 ceph-mon[97450]: pgmap v12807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:14:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:12 smithi059 ceph-mon[97450]: pgmap v12808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T04:14:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:12 smithi059 ceph-mon[97450]: pgmap v12809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:14:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:14:13.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:12.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:12.982895+0000) 2022-09-08T04:14:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:13.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:13.001505+0000) 2022-09-08T04:14:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:14.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:14.001731+0000) 2022-09-08T04:14:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:13.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:13.983087+0000) 2022-09-08T04:14:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:14 smithi055 ceph-mon[97033]: pgmap v12810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:14:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:14 smithi059 ceph-mon[97450]: pgmap v12810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:14:15.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:14:15.061+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:14:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:15.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:15.001878+0000) 2022-09-08T04:14:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:14.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:14.983264+0000) 2022-09-08T04:14:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:16.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:16.002087+0000) 2022-09-08T04:14:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:15.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:15.983502+0000) 2022-09-08T04:14:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:16 smithi055 ceph-mon[97033]: pgmap v12811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.1 KiB/s rd, 327 B/s wr, 10 op/s 2022-09-08T04:14:16.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:16 smithi059 ceph-mon[97450]: pgmap v12811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.1 KiB/s rd, 327 B/s wr, 10 op/s 2022-09-08T04:14:17.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:16.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:16.983740+0000) 2022-09-08T04:14:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:17.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:17.002240+0000) 2022-09-08T04:14:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:18.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:18.002442+0000) 2022-09-08T04:14:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:17.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:17.983933+0000) 2022-09-08T04:14:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:18 smithi055 ceph-mon[97033]: pgmap v12812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 259 B/s wr, 8 op/s 2022-09-08T04:14:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:18 smithi059 ceph-mon[97450]: pgmap v12812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 259 B/s wr, 8 op/s 2022-09-08T04:14:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:19.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:19.002714+0000) 2022-09-08T04:14:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:18.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:18.984111+0000) 2022-09-08T04:14:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:20.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:20.002871+0000) 2022-09-08T04:14:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:19.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:19.984355+0000) 2022-09-08T04:14:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:20 smithi055 ceph-mon[97033]: pgmap v12813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 123 B/s wr, 3 op/s 2022-09-08T04:14:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:20 smithi055 ceph-mon[97033]: osdmap e882: 12 total, 11 up, 11 in 2022-09-08T04:14:20.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:20 smithi059 ceph-mon[97450]: pgmap v12813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 123 B/s wr, 3 op/s 2022-09-08T04:14:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:20 smithi059 ceph-mon[97450]: osdmap e882: 12 total, 11 up, 11 in 2022-09-08T04:14:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:21.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:21.003000+0000) 2022-09-08T04:14:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:20.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:20.984587+0000) 2022-09-08T04:14:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:22.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:22.003157+0000) 2022-09-08T04:14:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:21.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:21.984799+0000) 2022-09-08T04:14:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:22 smithi055 ceph-mon[97033]: pgmap v12815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:14:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:22 smithi059 ceph-mon[97450]: pgmap v12815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:14:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:23.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:23.003394+0000) 2022-09-08T04:14:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:22.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:22.985007+0000) 2022-09-08T04:14:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:24.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:24.003679+0000) 2022-09-08T04:14:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:23.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:23.985257+0000) 2022-09-08T04:14:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:24 smithi055 ceph-mon[97033]: pgmap v12816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:14:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:24 smithi059 ceph-mon[97450]: pgmap v12816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:14:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:25.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:25.003828+0000) 2022-09-08T04:14:25.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:24.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:24.985456+0000) 2022-09-08T04:14:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:26.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:26.004000+0000) 2022-09-08T04:14:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:25.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:25.985719+0000) 2022-09-08T04:14:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:26 smithi055 ceph-mon[97033]: pgmap v12817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:14:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:26 smithi059 ceph-mon[97450]: pgmap v12817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:14:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:27.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:27.004224+0000) 2022-09-08T04:14:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:26.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:26.985939+0000) 2022-09-08T04:14:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:28.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:28.004392+0000) 2022-09-08T04:14:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:27.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:27.986203+0000) 2022-09-08T04:14:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:28 smithi055 ceph-mon[97033]: pgmap v12818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:14:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:28 smithi059 ceph-mon[97450]: pgmap v12818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:14:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:29.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:29.004592+0000) 2022-09-08T04:14:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:28.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:28.986418+0000) 2022-09-08T04:14:30.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:14:30.069+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:14:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:30.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:30.004745+0000) 2022-09-08T04:14:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:29.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:29.986690+0000) 2022-09-08T04:14:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:30 smithi055 ceph-mon[97033]: pgmap v12819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:14:30.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:30 smithi059 ceph-mon[97450]: pgmap v12819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:14:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:31.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:31.004963+0000) 2022-09-08T04:14:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:30.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:30.986922+0000) 2022-09-08T04:14:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:32.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:32.005227+0000) 2022-09-08T04:14:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:31.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:31.987152+0000) 2022-09-08T04:14:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:32 smithi055 ceph-mon[97033]: pgmap v12820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T04:14:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:32 smithi059 ceph-mon[97450]: pgmap v12820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T04:14:33.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:32.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:32.987429+0000) 2022-09-08T04:14:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:33.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:33.005469+0000) 2022-09-08T04:14:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:34.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:34.005732+0000) 2022-09-08T04:14:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:33.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:33.987631+0000) 2022-09-08T04:14:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:34 smithi055 ceph-mon[97033]: pgmap v12821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:34.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:34 smithi059 ceph-mon[97450]: pgmap v12821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:35.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:35.005993+0000) 2022-09-08T04:14:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:34.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:34.987779+0000) 2022-09-08T04:14:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:36.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:36.006216+0000) 2022-09-08T04:14:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:35.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:35.988037+0000) 2022-09-08T04:14:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:36 smithi055 ceph-mon[97033]: pgmap v12822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:14:36.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:36 smithi059 ceph-mon[97450]: pgmap v12822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:14:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:37.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:37.006440+0000) 2022-09-08T04:14:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:36.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:36.988299+0000) 2022-09-08T04:14:38.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:37.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:37.988534+0000) 2022-09-08T04:14:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:38.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:38.006658+0000) 2022-09-08T04:14:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:38 smithi055 ceph-mon[97033]: pgmap v12823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:38 smithi059 ceph-mon[97450]: pgmap v12823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:39.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:39.006860+0000) 2022-09-08T04:14:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:38.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:38.988762+0000) 2022-09-08T04:14:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:40.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:40.007076+0000) 2022-09-08T04:14:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:39.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:39.988951+0000) 2022-09-08T04:14:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:40 smithi059 ceph-mon[97450]: pgmap v12824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:40 smithi055 ceph-mon[97033]: pgmap v12824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:41.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:41.007314+0000) 2022-09-08T04:14:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:40.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:40.989169+0000) 2022-09-08T04:14:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:42.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:42.007525+0000) 2022-09-08T04:14:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:41.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:41.989405+0000) 2022-09-08T04:14:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:42 smithi059 ceph-mon[97450]: pgmap v12825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:14:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:42 smithi055 ceph-mon[97033]: pgmap v12825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:14:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:43.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:43.007769+0000) 2022-09-08T04:14:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:42.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:42.989682+0000) 2022-09-08T04:14:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:44.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:44.007960+0000) 2022-09-08T04:14:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:43.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:43.989948+0000) 2022-09-08T04:14:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:44 smithi059 ceph-mon[97450]: pgmap v12826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:44 smithi055 ceph-mon[97033]: pgmap v12826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:45.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:45.008146+0000) 2022-09-08T04:14:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:44.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:44.990113+0000) 2022-09-08T04:14:45.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:14:45.079+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:14:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:46.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:46.008380+0000) 2022-09-08T04:14:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:45.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:45.990358+0000) 2022-09-08T04:14:46.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:46 smithi059 ceph-mon[97450]: pgmap v12827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:14:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:46 smithi055 ceph-mon[97033]: pgmap v12827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:14:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:47.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:47.008613+0000) 2022-09-08T04:14:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:46.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:46.990614+0000) 2022-09-08T04:14:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:48.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:48.008892+0000) 2022-09-08T04:14:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:47.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:47.990915+0000) 2022-09-08T04:14:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:48 smithi059 ceph-mon[97450]: pgmap v12828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:48 smithi055 ceph-mon[97033]: pgmap v12828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:49.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:49.009076+0000) 2022-09-08T04:14:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:48.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:48.991153+0000) 2022-09-08T04:14:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:50.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:50.009286+0000) 2022-09-08T04:14:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:49.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:49.991333+0000) 2022-09-08T04:14:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:50 smithi059 ceph-mon[97450]: pgmap v12829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:50 smithi055 ceph-mon[97033]: pgmap v12829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:51.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:51.009451+0000) 2022-09-08T04:14:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:50.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:50.991503+0000) 2022-09-08T04:14:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:52.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:52.009597+0000) 2022-09-08T04:14:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:51.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:51.991704+0000) 2022-09-08T04:14:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:52 smithi059 ceph-mon[97450]: pgmap v12830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:14:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:52 smithi055 ceph-mon[97033]: pgmap v12830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:14:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:53.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:53.009791+0000) 2022-09-08T04:14:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:52.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:52.991876+0000) 2022-09-08T04:14:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:54.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:54.009998+0000) 2022-09-08T04:14:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:53.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:53.992127+0000) 2022-09-08T04:14:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:54 smithi059 ceph-mon[97450]: pgmap v12831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:54 smithi055 ceph-mon[97033]: pgmap v12831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:55.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:55.010131+0000) 2022-09-08T04:14:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:54.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:54.992303+0000) 2022-09-08T04:14:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:56.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:56.010317+0000) 2022-09-08T04:14:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:55.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:55.992556+0000) 2022-09-08T04:14:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:56 smithi059 ceph-mon[97450]: pgmap v12832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:14:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:56 smithi055 ceph-mon[97033]: pgmap v12832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:14:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:57.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:57.010483+0000) 2022-09-08T04:14:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:56.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:56.992813+0000) 2022-09-08T04:14:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:58.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:58.010729+0000) 2022-09-08T04:14:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:57.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:57.993009+0000) 2022-09-08T04:14:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:14:58 smithi059 ceph-mon[97450]: pgmap v12833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:14:58 smithi055 ceph-mon[97033]: pgmap v12833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:14:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:14:59.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:59.010904+0000) 2022-09-08T04:14:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:58.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:58.993310+0000) 2022-09-08T04:15:00.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:15:00.084+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:15:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:00.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:00.011178+0000) 2022-09-08T04:15:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:14:59.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:14:59.993532+0000) 2022-09-08T04:15:00.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:00 smithi059 ceph-mon[97450]: pgmap v12834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:15:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:00 smithi055 ceph-mon[97033]: pgmap v12834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:15:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:00.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:00.993774+0000) 2022-09-08T04:15:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:01.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:01.011415+0000) 2022-09-08T04:15:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:02.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:02.011649+0000) 2022-09-08T04:15:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:01.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:01.994010+0000) 2022-09-08T04:15:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:02 smithi059 ceph-mon[97450]: pgmap v12835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:15:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:02 smithi055 ceph-mon[97033]: pgmap v12835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:15:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:02.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:02.994243+0000) 2022-09-08T04:15:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:03.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:03.011816+0000) 2022-09-08T04:15:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:04.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:04.012033+0000) 2022-09-08T04:15:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:03.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:03.994483+0000) 2022-09-08T04:15:04.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:04 smithi059 ceph-mon[97450]: pgmap v12836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:15:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:04 smithi055 ceph-mon[97033]: pgmap v12836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:15:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:05.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:05.012194+0000) 2022-09-08T04:15:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:04.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:04.994763+0000) 2022-09-08T04:15:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:15:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:15:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:06.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:06.012348+0000) 2022-09-08T04:15:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:05.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:05.995040+0000) 2022-09-08T04:15:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:06 smithi055 ceph-mon[97033]: pgmap v12837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:15:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:15:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:06 smithi059 ceph-mon[97450]: pgmap v12837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:15:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:15:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:07.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:07.012578+0000) 2022-09-08T04:15:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:06.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:06.995228+0000) 2022-09-08T04:15:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:07.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:07.995461+0000) 2022-09-08T04:15:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:08.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:08.012797+0000) 2022-09-08T04:15:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:08 smithi055 ceph-mon[97033]: pgmap v12838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:15:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:08 smithi059 ceph-mon[97450]: pgmap v12838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:15:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:09.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:09.012999+0000) 2022-09-08T04:15:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:08.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:08.995738+0000) 2022-09-08T04:15:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:10.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:10.013134+0000) 2022-09-08T04:15:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:09.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:09.995983+0000) 2022-09-08T04:15:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:10 smithi055 ceph-mon[97033]: pgmap v12839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:15:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:10 smithi055 ceph-mon[97033]: osdmap e883: 12 total, 11 up, 11 in 2022-09-08T04:15:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:10 smithi059 ceph-mon[97450]: pgmap v12839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:15:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:10 smithi059 ceph-mon[97450]: osdmap e883: 12 total, 11 up, 11 in 2022-09-08T04:15:11.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:10.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:10.996198+0000) 2022-09-08T04:15:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:11.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:11.013269+0000) 2022-09-08T04:15:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:15:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:15:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:15:12.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:15:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:15:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:15:12.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:11.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:11.996405+0000) 2022-09-08T04:15:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:12.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:12.013507+0000) 2022-09-08T04:15:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:12 smithi055 ceph-mon[97033]: pgmap v12841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:15:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:15:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:15:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:12 smithi059 ceph-mon[97450]: pgmap v12841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:15:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:15:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:15:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:12.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:12.996675+0000) 2022-09-08T04:15:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:13.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:13.013723+0000) 2022-09-08T04:15:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:13.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:13.996883+0000) 2022-09-08T04:15:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:14.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:14.013929+0000) 2022-09-08T04:15:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:14 smithi059 ceph-mon[97450]: pgmap v12842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:15:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:15:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:15:14.817 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:14 smithi055 ceph-mon[97033]: pgmap v12842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:15:14.818 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:15:14.819 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:15:15.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:15:15.086+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:15:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:15.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:15.014162+0000) 2022-09-08T04:15:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:14.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:14.997126+0000) 2022-09-08T04:15:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:15:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:15:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:15 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T04:15:16.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:15 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T04:15:16.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:15:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:15:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:15 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T04:15:16.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:15 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T04:15:16.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:15:16.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:15.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:15.997348+0000) 2022-09-08T04:15:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:16.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:16.014418+0000) 2022-09-08T04:15:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:16 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T04:15:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:16 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T04:15:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:16 smithi055 ceph-mon[97033]: pgmap v12843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:15:17.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:16 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T04:15:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:16 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T04:15:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:16 smithi059 ceph-mon[97450]: pgmap v12843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:15:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:17.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:17.014579+0000) 2022-09-08T04:15:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:16.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:16.997528+0000) 2022-09-08T04:15:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:18.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:18.014789+0000) 2022-09-08T04:15:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:17.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:17.997782+0000) 2022-09-08T04:15:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:18 smithi055 ceph-mon[97033]: pgmap v12844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:15:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:18 smithi059 ceph-mon[97450]: pgmap v12844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:15:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:19.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:19.015022+0000) 2022-09-08T04:15:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:18.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:18.998006+0000) 2022-09-08T04:15:20.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:19.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:19.998196+0000) 2022-09-08T04:15:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:20.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:20.015274+0000) 2022-09-08T04:15:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:20 smithi055 ceph-mon[97033]: pgmap v12845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:15:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:20 smithi055 ceph-mon[97033]: osdmap e884: 12 total, 11 up, 11 in 2022-09-08T04:15:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:20 smithi059 ceph-mon[97450]: pgmap v12845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:15:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:20 smithi059 ceph-mon[97450]: osdmap e884: 12 total, 11 up, 11 in 2022-09-08T04:15:21.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:20.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:20.998423+0000) 2022-09-08T04:15:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:21.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:21.015498+0000) 2022-09-08T04:15:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:22.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:22.015714+0000) 2022-09-08T04:15:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:21.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:21.998671+0000) 2022-09-08T04:15:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:22 smithi055 ceph-mon[97033]: pgmap v12847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:15:22.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:22 smithi059 ceph-mon[97450]: pgmap v12847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:15:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:23.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:23.015957+0000) 2022-09-08T04:15:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:22.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:22.998877+0000) 2022-09-08T04:15:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:24.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:24.016188+0000) 2022-09-08T04:15:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:23.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:23.999155+0000) 2022-09-08T04:15:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:24 smithi055 ceph-mon[97033]: pgmap v12848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:15:24.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:24 smithi059 ceph-mon[97450]: pgmap v12848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:15:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:25.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:25.016409+0000) 2022-09-08T04:15:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:24.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:24.999417+0000) 2022-09-08T04:15:26.413 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:26.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:26.016623+0000) 2022-09-08T04:15:26.414 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:25.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:25.999584+0000) 2022-09-08T04:15:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:26 smithi059 ceph-mon[97450]: pgmap v12849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:15:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:26 smithi055 ceph-mon[97033]: pgmap v12849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:15:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:27.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:27.016844+0000) 2022-09-08T04:15:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:26.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:26.999731+0000) 2022-09-08T04:15:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:28.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:28.017045+0000) 2022-09-08T04:15:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:27.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:27.999903+0000) 2022-09-08T04:15:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:28 smithi059 ceph-mon[97450]: pgmap v12850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:15:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:28 smithi055 ceph-mon[97033]: pgmap v12850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:15:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:29.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:29.017195+0000) 2022-09-08T04:15:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:28.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:29.000153+0000) 2022-09-08T04:15:30.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:15:30.102+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:15:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:30.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:30.017418+0000) 2022-09-08T04:15:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:29.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:30.000403+0000) 2022-09-08T04:15:30.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:30 smithi059 ceph-mon[97450]: pgmap v12851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:15:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:30 smithi055 ceph-mon[97033]: pgmap v12851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:15:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:31.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:31.017671+0000) 2022-09-08T04:15:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:31.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:31.000682+0000) 2022-09-08T04:15:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:32.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:32.017844+0000) 2022-09-08T04:15:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:32.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:32.000915+0000) 2022-09-08T04:15:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:32 smithi059 ceph-mon[97450]: pgmap v12852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T04:15:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:32 smithi055 ceph-mon[97033]: pgmap v12852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T04:15:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:33.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:33.018071+0000) 2022-09-08T04:15:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:33.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:33.001141+0000) 2022-09-08T04:15:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:34.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:34.018255+0000) 2022-09-08T04:15:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:34.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:34.001385+0000) 2022-09-08T04:15:34.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:34 smithi059 ceph-mon[97450]: pgmap v12853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:15:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:34 smithi055 ceph-mon[97033]: pgmap v12853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:15:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:35.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:35.018469+0000) 2022-09-08T04:15:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:35.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:35.001596+0000) 2022-09-08T04:15:36.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:36.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:36.001803+0000) 2022-09-08T04:15:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:36.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:36.018718+0000) 2022-09-08T04:15:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:36 smithi059 ceph-mon[97450]: pgmap v12854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:15:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:36 smithi055 ceph-mon[97033]: pgmap v12854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:15:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:37.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:37.018844+0000) 2022-09-08T04:15:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:37.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:37.002056+0000) 2022-09-08T04:15:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:38.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:38.019050+0000) 2022-09-08T04:15:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:38.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:38.002294+0000) 2022-09-08T04:15:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:38 smithi059 ceph-mon[97450]: pgmap v12855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:15:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:38 smithi055 ceph-mon[97033]: pgmap v12855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:15:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:39.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:39.019280+0000) 2022-09-08T04:15:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:39.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:39.002560+0000) 2022-09-08T04:15:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:40.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:40.019531+0000) 2022-09-08T04:15:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:40.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:40.002756+0000) 2022-09-08T04:15:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:40 smithi059 ceph-mon[97450]: pgmap v12856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:15:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:40 smithi055 ceph-mon[97033]: pgmap v12856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:15:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:41.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:41.019745+0000) 2022-09-08T04:15:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:41.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:41.002983+0000) 2022-09-08T04:15:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:42.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:42.019971+0000) 2022-09-08T04:15:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:42.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:42.003221+0000) 2022-09-08T04:15:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:42 smithi055 ceph-mon[97033]: pgmap v12857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:15:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:42 smithi055 ceph-mon[97033]: pgmap v12858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:15:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:42 smithi055 ceph-mon[97033]: pgmap v12859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:15:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:15:43.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:42 smithi059 ceph-mon[97450]: pgmap v12857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:15:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:42 smithi059 ceph-mon[97450]: pgmap v12858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:15:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:42 smithi059 ceph-mon[97450]: pgmap v12859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:15:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:15:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:43.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:43.020205+0000) 2022-09-08T04:15:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:43.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:43.003458+0000) 2022-09-08T04:15:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:44.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:44.020460+0000) 2022-09-08T04:15:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:44.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:44.003746+0000) 2022-09-08T04:15:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:44 smithi055 ceph-mon[97033]: pgmap v12860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:15:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:44 smithi059 ceph-mon[97450]: pgmap v12860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:15:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:45.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:45.020678+0000) 2022-09-08T04:15:45.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:15:45.105+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:15:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:45.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:45.003969+0000) 2022-09-08T04:15:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:46.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:46.020867+0000) 2022-09-08T04:15:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:46.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:46.004171+0000) 2022-09-08T04:15:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:46 smithi055 ceph-mon[97033]: pgmap v12861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:15:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:46 smithi059 ceph-mon[97450]: pgmap v12861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:15:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:47.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:47.021096+0000) 2022-09-08T04:15:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:47.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:47.004401+0000) 2022-09-08T04:15:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:48.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:48.021338+0000) 2022-09-08T04:15:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:48.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:48.004664+0000) 2022-09-08T04:15:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:48 smithi055 ceph-mon[97033]: pgmap v12862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:15:49.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:48 smithi059 ceph-mon[97450]: pgmap v12862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:15:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:49.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:49.021527+0000) 2022-09-08T04:15:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:49.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:49.004885+0000) 2022-09-08T04:15:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:50.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:50.021778+0000) 2022-09-08T04:15:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:50.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:50.005081+0000) 2022-09-08T04:15:50.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:50 smithi059 ceph-mon[97450]: pgmap v12863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:15:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:50 smithi055 ceph-mon[97033]: pgmap v12863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:15:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:51.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:51.022034+0000) 2022-09-08T04:15:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:51.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:51.005328+0000) 2022-09-08T04:15:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:52.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:52.022250+0000) 2022-09-08T04:15:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:52.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:52.005557+0000) 2022-09-08T04:15:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:52 smithi059 ceph-mon[97450]: pgmap v12864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:15:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:52 smithi055 ceph-mon[97033]: pgmap v12864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:15:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:53.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:53.022530+0000) 2022-09-08T04:15:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:53.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:53.005734+0000) 2022-09-08T04:15:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:54.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:54.022748+0000) 2022-09-08T04:15:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:54.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:54.005904+0000) 2022-09-08T04:15:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:54 smithi059 ceph-mon[97450]: pgmap v12865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:15:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:54 smithi055 ceph-mon[97033]: pgmap v12865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:15:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:55.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:55.022918+0000) 2022-09-08T04:15:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:55.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:55.006086+0000) 2022-09-08T04:15:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:56.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:56.023155+0000) 2022-09-08T04:15:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:56.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:56.006340+0000) 2022-09-08T04:15:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:56 smithi059 ceph-mon[97450]: pgmap v12866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:15:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:56 smithi055 ceph-mon[97033]: pgmap v12866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:15:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:57.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:57.023371+0000) 2022-09-08T04:15:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:57.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:57.006564+0000) 2022-09-08T04:15:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:58.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:58.023605+0000) 2022-09-08T04:15:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:58.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:58.006697+0000) 2022-09-08T04:15:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:15:58 smithi059 ceph-mon[97450]: pgmap v12867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:15:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:15:58 smithi055 ceph-mon[97033]: pgmap v12867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:15:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:15:59.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:59.023893+0000) 2022-09-08T04:15:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:15:59.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:15:59.006852+0000) 2022-09-08T04:16:00.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:16:00.110+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:16:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:00.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:00.024079+0000) 2022-09-08T04:16:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:00.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:00.007084+0000) 2022-09-08T04:16:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:00 smithi059 ceph-mon[97450]: pgmap v12868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:00 smithi055 ceph-mon[97033]: pgmap v12868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:01.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:01.007315+0000) 2022-09-08T04:16:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:01.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:01.024222+0000) 2022-09-08T04:16:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:02.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:02.024402+0000) 2022-09-08T04:16:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:02.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:02.007538+0000) 2022-09-08T04:16:03.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:02 smithi059 ceph-mon[97450]: pgmap v12869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:16:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:02 smithi055 ceph-mon[97033]: pgmap v12869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:16:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:03.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:03.024689+0000) 2022-09-08T04:16:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:03.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:03.007767+0000) 2022-09-08T04:16:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:04.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:04.024948+0000) 2022-09-08T04:16:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:04.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:04.008015+0000) 2022-09-08T04:16:05.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:04 smithi059 ceph-mon[97450]: pgmap v12870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:04 smithi055 ceph-mon[97033]: pgmap v12870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:05.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:05.025230+0000) 2022-09-08T04:16:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:05.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:05.008239+0000) 2022-09-08T04:16:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:16:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:16:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:06.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:06.025456+0000) 2022-09-08T04:16:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:06.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:06.008511+0000) 2022-09-08T04:16:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:06 smithi059 ceph-mon[97450]: pgmap v12871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:16:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:16:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:06 smithi055 ceph-mon[97033]: pgmap v12871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:16:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:16:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:07.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:07.025668+0000) 2022-09-08T04:16:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:07.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:07.008721+0000) 2022-09-08T04:16:08.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:08.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:08.008953+0000) 2022-09-08T04:16:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:08.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:08.025861+0000) 2022-09-08T04:16:09.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:08 smithi059 ceph-mon[97450]: pgmap v12872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:16:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:08 smithi055 ceph-mon[97033]: pgmap v12872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:16:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:09.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:09.026077+0000) 2022-09-08T04:16:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:09.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:09.009224+0000) 2022-09-08T04:16:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:09 smithi055 ceph-mon[97033]: osdmap e885: 12 total, 11 up, 11 in 2022-09-08T04:16:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:09 smithi059 ceph-mon[97450]: osdmap e885: 12 total, 11 up, 11 in 2022-09-08T04:16:10.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:10.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:10.009364+0000) 2022-09-08T04:16:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:10.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:10.026286+0000) 2022-09-08T04:16:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:10 smithi055 ceph-mon[97033]: pgmap v12874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:16:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:10 smithi059 ceph-mon[97450]: pgmap v12874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:16:11.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:11.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:11.009586+0000) 2022-09-08T04:16:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:11.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:11.026544+0000) 2022-09-08T04:16:12.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:12.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:12.009799+0000) 2022-09-08T04:16:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:12.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:12.026796+0000) 2022-09-08T04:16:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:12 smithi055 ceph-mon[97033]: pgmap v12875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:16:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:12 smithi059 ceph-mon[97450]: pgmap v12875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:16:13.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:13.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:13.026950+0000) 2022-09-08T04:16:13.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:13.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:13.010026+0000) 2022-09-08T04:16:14.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:14.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:14.010233+0000) 2022-09-08T04:16:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:14.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:14.027197+0000) 2022-09-08T04:16:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:14 smithi055 ceph-mon[97033]: pgmap v12876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:16:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:14 smithi055 ceph-mon[97033]: osdmap e886: 12 total, 11 up, 11 in 2022-09-08T04:16:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:14 smithi059 ceph-mon[97450]: pgmap v12876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:16:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:14 smithi059 ceph-mon[97450]: osdmap e886: 12 total, 11 up, 11 in 2022-09-08T04:16:15.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:16:15.112+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:16:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:15.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:15.027444+0000) 2022-09-08T04:16:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:15.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:15.010477+0000) 2022-09-08T04:16:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:16.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:16.027734+0000) 2022-09-08T04:16:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:16.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:16.010700+0000) 2022-09-08T04:16:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:16 smithi055 ceph-mon[97033]: pgmap v12878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:16:17.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:16 smithi059 ceph-mon[97450]: pgmap v12878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:16:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:17.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:17.010893+0000) 2022-09-08T04:16:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:17.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:17.027894+0000) 2022-09-08T04:16:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:18.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:18.028058+0000) 2022-09-08T04:16:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:18.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:18.011122+0000) 2022-09-08T04:16:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:18 smithi055 ceph-mon[97033]: pgmap v12879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T04:16:19.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:18 smithi059 ceph-mon[97450]: pgmap v12879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T04:16:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:19.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:19.028287+0000) 2022-09-08T04:16:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:19.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:19.011373+0000) 2022-09-08T04:16:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:20.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:20.028506+0000) 2022-09-08T04:16:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:20.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:20.011605+0000) 2022-09-08T04:16:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:20 smithi059 ceph-mon[97450]: pgmap v12880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:16:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:20 smithi055 ceph-mon[97033]: pgmap v12880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:16:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:21.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:21.028730+0000) 2022-09-08T04:16:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:21.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:21.011850+0000) 2022-09-08T04:16:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:22.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:22.028904+0000) 2022-09-08T04:16:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:22.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:22.012028+0000) 2022-09-08T04:16:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:22 smithi059 ceph-mon[97450]: pgmap v12881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:16:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:22 smithi055 ceph-mon[97033]: pgmap v12881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:16:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:23.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:23.029086+0000) 2022-09-08T04:16:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:23.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:23.012257+0000) 2022-09-08T04:16:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:24.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:24.029280+0000) 2022-09-08T04:16:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:24.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:24.012521+0000) 2022-09-08T04:16:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:24 smithi059 ceph-mon[97450]: pgmap v12882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:16:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:24 smithi055 ceph-mon[97033]: pgmap v12882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:16:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:25.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:25.029468+0000) 2022-09-08T04:16:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:25.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:25.012769+0000) 2022-09-08T04:16:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:26.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:26.029736+0000) 2022-09-08T04:16:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:26.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:26.013009+0000) 2022-09-08T04:16:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:26 smithi059 ceph-mon[97450]: pgmap v12883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 91 B/s wr, 8 op/s 2022-09-08T04:16:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:26 smithi055 ceph-mon[97033]: pgmap v12883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 91 B/s wr, 8 op/s 2022-09-08T04:16:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:27.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:27.029984+0000) 2022-09-08T04:16:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:27.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:27.013240+0000) 2022-09-08T04:16:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:28.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:28.030193+0000) 2022-09-08T04:16:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:28.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:28.013504+0000) 2022-09-08T04:16:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:28 smithi059 ceph-mon[97450]: pgmap v12884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:28 smithi055 ceph-mon[97033]: pgmap v12884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:29.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:29.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:29.013765+0000) 2022-09-08T04:16:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:29.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:29.030440+0000) 2022-09-08T04:16:30.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:16:30.121+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:16:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:30.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:30.030705+0000) 2022-09-08T04:16:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:30.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:30.014006+0000) 2022-09-08T04:16:30.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:30 smithi059 ceph-mon[97450]: pgmap v12885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:30 smithi055 ceph-mon[97033]: pgmap v12885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:31.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:31.030873+0000) 2022-09-08T04:16:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:31.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:31.014165+0000) 2022-09-08T04:16:32.409 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:32.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:32.031090+0000) 2022-09-08T04:16:32.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:32.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:32.014394+0000) 2022-09-08T04:16:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:32 smithi059 ceph-mon[97450]: pgmap v12886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:16:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:32 smithi055 ceph-mon[97033]: pgmap v12886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:16:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:33.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:33.031314+0000) 2022-09-08T04:16:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:33.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:33.014600+0000) 2022-09-08T04:16:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:34.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:34.031484+0000) 2022-09-08T04:16:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:34.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:34.014872+0000) 2022-09-08T04:16:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:34 smithi059 ceph-mon[97450]: pgmap v12887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:34 smithi055 ceph-mon[97033]: pgmap v12887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:35.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:35.031688+0000) 2022-09-08T04:16:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:35.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:35.015096+0000) 2022-09-08T04:16:36.409 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:36.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:36.031863+0000) 2022-09-08T04:16:36.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:36.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:36.015266+0000) 2022-09-08T04:16:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:36 smithi059 ceph-mon[97450]: pgmap v12888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:16:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:36 smithi055 ceph-mon[97033]: pgmap v12888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:16:37.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:37.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:37.015486+0000) 2022-09-08T04:16:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:37.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:37.032057+0000) 2022-09-08T04:16:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:38.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:38.032269+0000) 2022-09-08T04:16:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:38.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:38.015728+0000) 2022-09-08T04:16:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:38 smithi059 ceph-mon[97450]: pgmap v12889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:38 smithi055 ceph-mon[97033]: pgmap v12889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:39.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:39.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:39.015911+0000) 2022-09-08T04:16:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:39.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:39.032427+0000) 2022-09-08T04:16:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:40.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:40.032617+0000) 2022-09-08T04:16:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:40.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:40.016085+0000) 2022-09-08T04:16:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:40 smithi059 ceph-mon[97450]: pgmap v12890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:40 smithi055 ceph-mon[97033]: pgmap v12890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:41.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:41.032877+0000) 2022-09-08T04:16:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:41.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:41.016259+0000) 2022-09-08T04:16:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:42.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:42.033118+0000) 2022-09-08T04:16:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:42.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:42.016488+0000) 2022-09-08T04:16:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:42 smithi059 ceph-mon[97450]: pgmap v12891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:16:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:16:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:16:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:16:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:16:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:16:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:42 smithi055 ceph-mon[97033]: pgmap v12891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:16:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:16:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:16:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:16:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:16:43.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:16:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:43.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:43.033300+0000) 2022-09-08T04:16:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:43.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:43.016707+0000) 2022-09-08T04:16:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:44.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:44.033529+0000) 2022-09-08T04:16:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:44.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:44.016922+0000) 2022-09-08T04:16:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:44 smithi059 ceph-mon[97450]: pgmap v12892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:44 smithi055 ceph-mon[97033]: pgmap v12892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:45.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:16:45.131+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:16:45.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:45.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:45.033712+0000) 2022-09-08T04:16:45.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:45.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:45.017153+0000) 2022-09-08T04:16:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:46.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:46.033927+0000) 2022-09-08T04:16:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:46.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:46.017371+0000) 2022-09-08T04:16:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:46 smithi059 ceph-mon[97450]: pgmap v12893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:16:47.241 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:46 smithi055 ceph-mon[97033]: pgmap v12893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:16:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:47.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:47.034162+0000) 2022-09-08T04:16:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:47.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:47.017595+0000) 2022-09-08T04:16:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:48.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:48.034427+0000) 2022-09-08T04:16:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:48.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:48.017871+0000) 2022-09-08T04:16:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:48 smithi059 ceph-mon[97450]: pgmap v12894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:16:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:48 smithi055 ceph-mon[97033]: pgmap v12894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:16:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:49.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:49.034682+0000) 2022-09-08T04:16:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:49.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:49.018131+0000) 2022-09-08T04:16:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:50.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:50.034894+0000) 2022-09-08T04:16:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:50.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:50.018356+0000) 2022-09-08T04:16:50.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:50 smithi059 ceph-mon[97450]: pgmap v12895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:16:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:50 smithi055 ceph-mon[97033]: pgmap v12895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:16:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:51.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:51.035152+0000) 2022-09-08T04:16:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:51.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:51.018588+0000) 2022-09-08T04:16:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:52.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:52.035370+0000) 2022-09-08T04:16:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:52.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:52.018761+0000) 2022-09-08T04:16:53.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:52 smithi059 ceph-mon[97450]: pgmap v12896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:16:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:52 smithi055 ceph-mon[97033]: pgmap v12896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:16:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:53.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:53.035602+0000) 2022-09-08T04:16:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:53.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:53.018968+0000) 2022-09-08T04:16:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:54.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:54.035850+0000) 2022-09-08T04:16:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:54.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:54.019165+0000) 2022-09-08T04:16:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:54 smithi059 ceph-mon[97450]: pgmap v12897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:16:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:54 smithi055 ceph-mon[97033]: pgmap v12897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:16:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:55.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:55.036086+0000) 2022-09-08T04:16:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:55.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:55.019348+0000) 2022-09-08T04:16:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:56.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:56.036339+0000) 2022-09-08T04:16:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:56.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:56.019539+0000) 2022-09-08T04:16:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:56 smithi059 ceph-mon[97450]: pgmap v12898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:16:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:56 smithi055 ceph-mon[97033]: pgmap v12898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:16:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:57.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:57.036507+0000) 2022-09-08T04:16:57.464 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:57.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:57.019786+0000) 2022-09-08T04:16:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:58.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:58.036762+0000) 2022-09-08T04:16:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:58.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:58.020030+0000) 2022-09-08T04:16:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:16:58 smithi055 ceph-mon[97033]: pgmap v12899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:16:58 smithi059 ceph-mon[97450]: pgmap v12899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:16:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:16:59.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:59.037008+0000) 2022-09-08T04:16:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:16:59.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:16:59.020310+0000) 2022-09-08T04:17:00.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:17:00.136+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:17:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:00.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:00.037250+0000) 2022-09-08T04:17:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:00.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:00.020605+0000) 2022-09-08T04:17:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:00 smithi059 ceph-mon[97450]: pgmap v12900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:00 smithi055 ceph-mon[97033]: pgmap v12900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:01.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:01.037403+0000) 2022-09-08T04:17:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:01.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:01.020778+0000) 2022-09-08T04:17:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:02.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:02.037612+0000) 2022-09-08T04:17:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:02.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:02.020975+0000) 2022-09-08T04:17:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:02 smithi059 ceph-mon[97450]: pgmap v12901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:17:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:02 smithi055 ceph-mon[97033]: pgmap v12901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:17:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:03.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:03.037802+0000) 2022-09-08T04:17:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:03.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:03.021211+0000) 2022-09-08T04:17:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:04.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:04.037952+0000) 2022-09-08T04:17:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:04.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:04.021477+0000) 2022-09-08T04:17:05.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:04 smithi059 ceph-mon[97450]: pgmap v12902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:04 smithi055 ceph-mon[97033]: pgmap v12902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:05.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:05.038119+0000) 2022-09-08T04:17:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:05.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:05.021752+0000) 2022-09-08T04:17:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:17:06.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:17:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:06.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:06.038310+0000) 2022-09-08T04:17:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:06.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:06.021926+0000) 2022-09-08T04:17:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:06 smithi055 ceph-mon[97033]: pgmap v12903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:17:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:17:07.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:06 smithi059 ceph-mon[97450]: pgmap v12903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:17:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:17:07.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:07.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:07.022114+0000) 2022-09-08T04:17:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:07.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:07.038531+0000) 2022-09-08T04:17:08.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:08.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:08.022321+0000) 2022-09-08T04:17:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:08.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:08.038845+0000) 2022-09-08T04:17:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:08 smithi055 ceph-mon[97033]: pgmap v12904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:17:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:08 smithi059 ceph-mon[97450]: pgmap v12904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:17:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:09.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:09.039091+0000) 2022-09-08T04:17:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:09.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:09.022602+0000) 2022-09-08T04:17:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:09 smithi055 ceph-mon[97033]: osdmap e887: 12 total, 11 up, 11 in 2022-09-08T04:17:10.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:09 smithi059 ceph-mon[97450]: osdmap e887: 12 total, 11 up, 11 in 2022-09-08T04:17:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:10.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:10.039333+0000) 2022-09-08T04:17:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:10.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:10.022798+0000) 2022-09-08T04:17:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:10 smithi055 ceph-mon[97033]: pgmap v12906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:17:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:10 smithi059 ceph-mon[97450]: pgmap v12906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:17:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:11.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:11.039553+0000) 2022-09-08T04:17:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:11.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:11.023062+0000) 2022-09-08T04:17:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:12.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:12.039752+0000) 2022-09-08T04:17:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:12.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:12.023275+0000) 2022-09-08T04:17:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:12 smithi055 ceph-mon[97033]: pgmap v12907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:17:13.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:12 smithi059 ceph-mon[97450]: pgmap v12907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:17:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:13.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:13.039937+0000) 2022-09-08T04:17:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:13.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:13.023420+0000) 2022-09-08T04:17:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:14.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:14.040075+0000) 2022-09-08T04:17:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:14.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:14.023588+0000) 2022-09-08T04:17:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:14 smithi055 ceph-mon[97033]: pgmap v12908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:17:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:14 smithi055 ceph-mon[97033]: osdmap e888: 12 total, 11 up, 11 in 2022-09-08T04:17:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:14 smithi059 ceph-mon[97450]: pgmap v12908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:17:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:14 smithi059 ceph-mon[97450]: osdmap e888: 12 total, 11 up, 11 in 2022-09-08T04:17:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:15.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:15.040276+0000) 2022-09-08T04:17:15.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:17:15.148+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:17:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:15.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:15.023829+0000) 2022-09-08T04:17:16.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:16.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:16.024061+0000) 2022-09-08T04:17:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:16.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:16.040496+0000) 2022-09-08T04:17:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:16 smithi059 ceph-mon[97450]: pgmap v12910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:17:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:17:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:17:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:17:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:16 smithi055 ceph-mon[97033]: pgmap v12910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:17:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:17:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:17:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:17:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:17.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:17.040648+0000) 2022-09-08T04:17:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:17.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:17.024294+0000) 2022-09-08T04:17:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:17 smithi055 ceph-mon[97033]: pgmap v12911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 265 B/s wr, 8 op/s 2022-09-08T04:17:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:17 smithi055 ceph-mon[97033]: pgmap v12912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.7 KiB/s rd, 301 B/s wr, 9 op/s 2022-09-08T04:17:18.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:17 smithi059 ceph-mon[97450]: pgmap v12911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 265 B/s wr, 8 op/s 2022-09-08T04:17:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:17 smithi059 ceph-mon[97450]: pgmap v12912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.7 KiB/s rd, 301 B/s wr, 9 op/s 2022-09-08T04:17:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:18.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:18.040863+0000) 2022-09-08T04:17:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:18.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:18.024565+0000) 2022-09-08T04:17:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:19.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:19.041082+0000) 2022-09-08T04:17:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:19.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:19.024778+0000) 2022-09-08T04:17:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:19 smithi055 ceph-mon[97033]: pgmap v12913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 150 B/s wr, 4 op/s 2022-09-08T04:17:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:19 smithi059 ceph-mon[97450]: pgmap v12913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 150 B/s wr, 4 op/s 2022-09-08T04:17:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:20.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:20.041264+0000) 2022-09-08T04:17:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:20.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:20.024960+0000) 2022-09-08T04:17:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:21.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:21.041485+0000) 2022-09-08T04:17:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:21.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:21.025132+0000) 2022-09-08T04:17:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:21 smithi055 ceph-mon[97033]: pgmap v12914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.6 KiB/s rd, 150 B/s wr, 9 op/s 2022-09-08T04:17:22.279 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:21 smithi059 ceph-mon[97450]: pgmap v12914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.6 KiB/s rd, 150 B/s wr, 9 op/s 2022-09-08T04:17:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:22.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:22.041729+0000) 2022-09-08T04:17:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:22.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:22.025345+0000) 2022-09-08T04:17:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:23.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:23.041972+0000) 2022-09-08T04:17:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:23.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:23.025546+0000) 2022-09-08T04:17:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:23 smithi055 ceph-mon[97033]: pgmap v12915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T04:17:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:23 smithi059 ceph-mon[97450]: pgmap v12915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T04:17:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:24.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:24.042239+0000) 2022-09-08T04:17:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:24.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:24.025714+0000) 2022-09-08T04:17:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:25.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:25.042454+0000) 2022-09-08T04:17:25.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:25.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:25.025962+0000) 2022-09-08T04:17:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:25 smithi059 ceph-mon[97450]: pgmap v12916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T04:17:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:25 smithi055 ceph-mon[97033]: pgmap v12916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T04:17:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:26.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:26.042650+0000) 2022-09-08T04:17:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:26.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:26.026180+0000) 2022-09-08T04:17:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:27.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:27.042784+0000) 2022-09-08T04:17:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:27.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:27.026414+0000) 2022-09-08T04:17:27.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:27 smithi055 ceph-mon[97033]: pgmap v12917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:17:28.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:27 smithi059 ceph-mon[97450]: pgmap v12917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:17:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:28.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:28.043030+0000) 2022-09-08T04:17:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:28.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:28.026610+0000) 2022-09-08T04:17:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:29.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:29.043226+0000) 2022-09-08T04:17:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:29.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:29.026843+0000) 2022-09-08T04:17:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:29 smithi055 ceph-mon[97033]: pgmap v12918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:30.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:29 smithi059 ceph-mon[97450]: pgmap v12918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:30.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:17:30.154+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:17:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:30.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:30.043439+0000) 2022-09-08T04:17:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:30.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:30.027000+0000) 2022-09-08T04:17:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:31.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:31.043602+0000) 2022-09-08T04:17:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:31.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:31.027153+0000) 2022-09-08T04:17:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:31 smithi055 ceph-mon[97033]: pgmap v12919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:17:32.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:31 smithi059 ceph-mon[97450]: pgmap v12919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:17:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:32.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:32.043798+0000) 2022-09-08T04:17:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:32.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:32.027331+0000) 2022-09-08T04:17:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:33.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:33.044018+0000) 2022-09-08T04:17:33.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:33.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:33.027508+0000) 2022-09-08T04:17:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:33 smithi055 ceph-mon[97033]: pgmap v12920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:17:34.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:33 smithi059 ceph-mon[97450]: pgmap v12920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:17:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:34.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:34.044181+0000) 2022-09-08T04:17:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:34.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:34.027688+0000) 2022-09-08T04:17:35.410 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:35.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:35.044391+0000) 2022-09-08T04:17:35.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:35.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:35.027917+0000) 2022-09-08T04:17:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:35 smithi059 ceph-mon[97450]: pgmap v12921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:17:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:35 smithi055 ceph-mon[97033]: pgmap v12921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:17:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:36.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:36.044596+0000) 2022-09-08T04:17:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:36.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:36.028073+0000) 2022-09-08T04:17:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:37.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:37.044793+0000) 2022-09-08T04:17:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:37.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:37.028251+0000) 2022-09-08T04:17:37.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:37 smithi055 ceph-mon[97033]: pgmap v12922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:17:38.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:37 smithi059 ceph-mon[97450]: pgmap v12922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:17:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:38.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:38.045039+0000) 2022-09-08T04:17:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:38.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:38.028428+0000) 2022-09-08T04:17:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:39.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:39.045278+0000) 2022-09-08T04:17:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:39.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:39.028708+0000) 2022-09-08T04:17:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:39 smithi055 ceph-mon[97033]: pgmap v12923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:17:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:39 smithi059 ceph-mon[97450]: pgmap v12923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:17:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:40.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:40.045549+0000) 2022-09-08T04:17:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:40.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:40.028854+0000) 2022-09-08T04:17:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:41.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:41.045799+0000) 2022-09-08T04:17:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:41.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:41.029025+0000) 2022-09-08T04:17:41.752 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:41 smithi055 ceph-mon[97033]: pgmap v12924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:17:42.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:41 smithi059 ceph-mon[97450]: pgmap v12924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:17:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:42.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:42.046054+0000) 2022-09-08T04:17:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:42.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:42.029244+0000) 2022-09-08T04:17:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:43.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:43.046319+0000) 2022-09-08T04:17:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:43.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:43.029465+0000) 2022-09-08T04:17:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:43 smithi055 ceph-mon[97033]: pgmap v12925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:43 smithi059 ceph-mon[97450]: pgmap v12925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:44.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:44.046558+0000) 2022-09-08T04:17:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:44.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:44.029727+0000) 2022-09-08T04:17:45.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:17:45.161+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:17:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:45.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:45.046795+0000) 2022-09-08T04:17:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:45.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:45.029893+0000) 2022-09-08T04:17:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:45 smithi059 ceph-mon[97450]: pgmap v12926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:45 smithi055 ceph-mon[97033]: pgmap v12926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:46.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:46.047009+0000) 2022-09-08T04:17:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:46.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:46.030123+0000) 2022-09-08T04:17:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:47.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:47.047228+0000) 2022-09-08T04:17:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:47.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:47.030352+0000) 2022-09-08T04:17:47.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:47 smithi055 ceph-mon[97033]: pgmap v12927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:17:48.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:47 smithi059 ceph-mon[97450]: pgmap v12927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:17:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:48.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:48.047462+0000) 2022-09-08T04:17:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:48.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:48.030599+0000) 2022-09-08T04:17:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:49.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:49.047600+0000) 2022-09-08T04:17:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:49.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:49.030871+0000) 2022-09-08T04:17:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:49 smithi055 ceph-mon[97033]: pgmap v12928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:50.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:49 smithi059 ceph-mon[97450]: pgmap v12928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:50.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:50.047789+0000) 2022-09-08T04:17:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:50.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:50.031101+0000) 2022-09-08T04:17:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:51.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:51.047992+0000) 2022-09-08T04:17:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:51.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:51.031299+0000) 2022-09-08T04:17:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:51 smithi055 ceph-mon[97033]: pgmap v12929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:17:52.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:51 smithi059 ceph-mon[97450]: pgmap v12929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:17:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:52.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:52.048143+0000) 2022-09-08T04:17:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:52.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:52.031520+0000) 2022-09-08T04:17:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:53.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:53.048406+0000) 2022-09-08T04:17:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:53.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:53.031739+0000) 2022-09-08T04:17:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:53 smithi055 ceph-mon[97033]: pgmap v12930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:53 smithi059 ceph-mon[97450]: pgmap v12930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:54.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:54.048595+0000) 2022-09-08T04:17:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:54.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:54.031991+0000) 2022-09-08T04:17:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:55.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:55.048835+0000) 2022-09-08T04:17:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:55.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:55.032227+0000) 2022-09-08T04:17:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:55 smithi059 ceph-mon[97450]: pgmap v12931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:55 smithi055 ceph-mon[97033]: pgmap v12931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:17:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:56.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:56.049071+0000) 2022-09-08T04:17:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:56.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:56.032484+0000) 2022-09-08T04:17:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:57.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:57.049296+0000) 2022-09-08T04:17:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:57.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:57.032691+0000) 2022-09-08T04:17:57.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:57 smithi055 ceph-mon[97033]: pgmap v12932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:17:58.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:57 smithi059 ceph-mon[97450]: pgmap v12932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:17:58.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:58.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:58.032855+0000) 2022-09-08T04:17:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:58.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:58.049499+0000) 2022-09-08T04:17:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:17:59.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:59.049700+0000) 2022-09-08T04:17:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:17:59.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:17:59.033008+0000) 2022-09-08T04:17:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:17:59 smithi055 ceph-mon[97033]: pgmap v12933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:00.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:17:59 smithi059 ceph-mon[97450]: pgmap v12933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:00.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:00.049946+0000) 2022-09-08T04:18:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:00.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:00.033235+0000) 2022-09-08T04:18:00.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:18:00.162+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:18:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:01.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:01.050095+0000) 2022-09-08T04:18:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:01.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:01.033407+0000) 2022-09-08T04:18:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:01 smithi055 ceph-mon[97033]: pgmap v12934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:18:02.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:01 smithi059 ceph-mon[97450]: pgmap v12934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:18:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:02.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:02.050325+0000) 2022-09-08T04:18:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:02.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:02.033725+0000) 2022-09-08T04:18:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:03.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:03.050555+0000) 2022-09-08T04:18:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:03.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:03.033959+0000) 2022-09-08T04:18:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:03 smithi055 ceph-mon[97033]: pgmap v12935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:04.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:03 smithi059 ceph-mon[97450]: pgmap v12935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:04.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:04.050754+0000) 2022-09-08T04:18:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:04.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:04.034255+0000) 2022-09-08T04:18:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:05.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:05.050950+0000) 2022-09-08T04:18:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:05.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:05.034495+0000) 2022-09-08T04:18:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:05 smithi059 ceph-mon[97450]: pgmap v12936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:05 smithi055 ceph-mon[97033]: pgmap v12936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:06.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:06.051119+0000) 2022-09-08T04:18:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:06.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:06.034684+0000) 2022-09-08T04:18:06.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:18:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:18:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:18:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:18:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:07.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:07.051302+0000) 2022-09-08T04:18:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:07.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:07.034903+0000) 2022-09-08T04:18:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:07 smithi059 ceph-mon[97450]: pgmap v12937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:18:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:07 smithi055 ceph-mon[97033]: pgmap v12937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:18:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:08.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:08.051545+0000) 2022-09-08T04:18:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:08.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:08.035105+0000) 2022-09-08T04:18:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:09.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:09.051783+0000) 2022-09-08T04:18:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:09.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:09.035310+0000) 2022-09-08T04:18:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:09 smithi055 ceph-mon[97033]: pgmap v12938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:18:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:09 smithi059 ceph-mon[97450]: pgmap v12938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:18:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:10.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:10.051957+0000) 2022-09-08T04:18:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:10.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:10.035494+0000) 2022-09-08T04:18:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:10 smithi055 ceph-mon[97033]: osdmap e889: 12 total, 11 up, 11 in 2022-09-08T04:18:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:10 smithi059 ceph-mon[97450]: osdmap e889: 12 total, 11 up, 11 in 2022-09-08T04:18:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:11.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:11.052166+0000) 2022-09-08T04:18:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:11.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:11.035785+0000) 2022-09-08T04:18:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:11 smithi055 ceph-mon[97033]: pgmap v12940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:18:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:11 smithi059 ceph-mon[97450]: pgmap v12940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:18:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:12.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:12.052304+0000) 2022-09-08T04:18:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:12.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:12.035987+0000) 2022-09-08T04:18:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:13.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:13.052540+0000) 2022-09-08T04:18:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:13.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:13.036155+0000) 2022-09-08T04:18:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:13 smithi055 ceph-mon[97033]: pgmap v12941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:18:14.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:13 smithi059 ceph-mon[97450]: pgmap v12941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:18:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:14.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:14.052749+0000) 2022-09-08T04:18:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:14.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:14.036401+0000) 2022-09-08T04:18:15.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:18:15.176+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:18:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:15.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:15.052945+0000) 2022-09-08T04:18:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:15.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:15.036595+0000) 2022-09-08T04:18:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:15 smithi059 ceph-mon[97450]: pgmap v12942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:18:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:15 smithi055 ceph-mon[97033]: pgmap v12942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:18:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:16.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:16.053071+0000) 2022-09-08T04:18:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:16.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:16.036838+0000) 2022-09-08T04:18:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:17.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:17.053294+0000) 2022-09-08T04:18:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:17.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:17.037117+0000) 2022-09-08T04:18:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:17 smithi055 ceph-mon[97033]: pgmap v12943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:18:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:18:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:18:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:18:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:18:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:17 smithi059 ceph-mon[97450]: pgmap v12943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:18:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:18:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:18:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:18:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:18:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:18.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:18.053533+0000) 2022-09-08T04:18:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:18.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:18.037372+0000) 2022-09-08T04:18:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:18 smithi055 ceph-mon[97033]: pgmap v12944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2022-09-08T04:18:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:18 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T04:18:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:18 smithi059 ceph-mon[97450]: pgmap v12944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2022-09-08T04:18:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:18 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T04:18:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:19.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:19.053774+0000) 2022-09-08T04:18:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:19.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:19.037624+0000) 2022-09-08T04:18:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:20.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:20.053967+0000) 2022-09-08T04:18:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:20.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:20.037861+0000) 2022-09-08T04:18:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:20 smithi059 ceph-mon[97450]: pgmap v12945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T04:18:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:20 smithi059 ceph-mon[97450]: osdmap e890: 12 total, 11 up, 11 in 2022-09-08T04:18:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:20 smithi055 ceph-mon[97033]: pgmap v12945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T04:18:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:20 smithi055 ceph-mon[97033]: osdmap e890: 12 total, 11 up, 11 in 2022-09-08T04:18:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:21.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:21.054197+0000) 2022-09-08T04:18:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:21.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:21.038035+0000) 2022-09-08T04:18:22.412 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:22.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:22.054400+0000) 2022-09-08T04:18:22.413 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:22.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:22.038250+0000) 2022-09-08T04:18:22.779 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:22 smithi059 ceph-mon[97450]: pgmap v12947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T04:18:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:22 smithi055 ceph-mon[97033]: pgmap v12947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T04:18:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:23.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:23.054533+0000) 2022-09-08T04:18:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:23.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:23.038468+0000) 2022-09-08T04:18:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:24.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:24.054746+0000) 2022-09-08T04:18:24.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:24.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:24.038788+0000) 2022-09-08T04:18:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:24 smithi059 ceph-mon[97450]: pgmap v12948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T04:18:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:24 smithi055 ceph-mon[97033]: pgmap v12948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T04:18:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:25.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:25.054946+0000) 2022-09-08T04:18:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:25.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:25.039037+0000) 2022-09-08T04:18:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:26.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:26.055197+0000) 2022-09-08T04:18:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:26.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:26.039242+0000) 2022-09-08T04:18:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:26 smithi059 ceph-mon[97450]: pgmap v12949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T04:18:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:26 smithi055 ceph-mon[97033]: pgmap v12949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T04:18:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:27.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:27.055428+0000) 2022-09-08T04:18:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:27.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:27.039454+0000) 2022-09-08T04:18:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:28.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:28.055596+0000) 2022-09-08T04:18:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:28.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:28.039730+0000) 2022-09-08T04:18:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:28 smithi059 ceph-mon[97450]: pgmap v12950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:18:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:28 smithi055 ceph-mon[97033]: pgmap v12950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:18:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:29.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:29.055810+0000) 2022-09-08T04:18:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:29.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:29.039978+0000) 2022-09-08T04:18:30.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:18:30.193+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:18:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:30.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:30.056042+0000) 2022-09-08T04:18:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:30.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:30.040215+0000) 2022-09-08T04:18:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:30 smithi055 ceph-mon[97033]: pgmap v12951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:18:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:30 smithi059 ceph-mon[97450]: pgmap v12951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:18:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:31.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:31.056313+0000) 2022-09-08T04:18:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:31.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:31.040482+0000) 2022-09-08T04:18:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:32.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:32.056549+0000) 2022-09-08T04:18:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:32.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:32.040754+0000) 2022-09-08T04:18:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:32 smithi055 ceph-mon[97033]: pgmap v12952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:18:33.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:32 smithi059 ceph-mon[97450]: pgmap v12952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:18:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:33.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:33.056803+0000) 2022-09-08T04:18:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:33.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:33.040997+0000) 2022-09-08T04:18:34.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:34.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:34.041135+0000) 2022-09-08T04:18:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:34.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:34.056996+0000) 2022-09-08T04:18:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:34 smithi055 ceph-mon[97033]: pgmap v12953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:34 smithi059 ceph-mon[97450]: pgmap v12953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:35.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:35.057141+0000) 2022-09-08T04:18:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:35.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:35.041256+0000) 2022-09-08T04:18:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:36.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:36.057347+0000) 2022-09-08T04:18:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:36.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:36.041416+0000) 2022-09-08T04:18:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:36 smithi055 ceph-mon[97033]: pgmap v12954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:18:37.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:36 smithi059 ceph-mon[97450]: pgmap v12954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:18:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:37.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:37.057585+0000) 2022-09-08T04:18:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:37.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:37.041629+0000) 2022-09-08T04:18:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:38.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:38.057778+0000) 2022-09-08T04:18:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:38.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:38.041891+0000) 2022-09-08T04:18:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:38 smithi055 ceph-mon[97033]: pgmap v12955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:38 smithi059 ceph-mon[97450]: pgmap v12955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:39.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:39.057980+0000) 2022-09-08T04:18:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:39.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:39.042099+0000) 2022-09-08T04:18:40.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:40.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:40.042349+0000) 2022-09-08T04:18:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:40.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:40.058196+0000) 2022-09-08T04:18:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:40 smithi055 ceph-mon[97033]: pgmap v12956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:41.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:40 smithi059 ceph-mon[97450]: pgmap v12956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:41.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:41.058390+0000) 2022-09-08T04:18:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:41.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:41.042552+0000) 2022-09-08T04:18:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:42.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:42.058666+0000) 2022-09-08T04:18:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:42.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:42.042767+0000) 2022-09-08T04:18:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:42 smithi055 ceph-mon[97033]: pgmap v12957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:18:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:42 smithi059 ceph-mon[97450]: pgmap v12957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:18:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:43.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:43.058995+0000) 2022-09-08T04:18:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:43.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:43.042953+0000) 2022-09-08T04:18:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:44.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:44.043171+0000) 2022-09-08T04:18:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:44.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:44.059146+0000) 2022-09-08T04:18:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:44 smithi055 ceph-mon[97033]: pgmap v12958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:45.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:44 smithi059 ceph-mon[97450]: pgmap v12958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:45.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:18:45.194+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:18:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:45.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:45.059295+0000) 2022-09-08T04:18:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:45.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:45.043413+0000) 2022-09-08T04:18:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:46.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:46.059507+0000) 2022-09-08T04:18:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:46.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:46.043605+0000) 2022-09-08T04:18:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:46 smithi055 ceph-mon[97033]: pgmap v12959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:18:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:46 smithi059 ceph-mon[97450]: pgmap v12959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:18:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:47.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:47.059694+0000) 2022-09-08T04:18:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:47.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:47.043852+0000) 2022-09-08T04:18:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:48.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:48.059867+0000) 2022-09-08T04:18:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:48.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:48.044091+0000) 2022-09-08T04:18:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:48 smithi055 ceph-mon[97033]: pgmap v12960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:48 smithi059 ceph-mon[97450]: pgmap v12960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:49.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:49.060057+0000) 2022-09-08T04:18:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:49.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:49.044336+0000) 2022-09-08T04:18:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:50.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:50.060195+0000) 2022-09-08T04:18:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:50.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:50.044566+0000) 2022-09-08T04:18:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:50 smithi055 ceph-mon[97033]: pgmap v12961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:50 smithi059 ceph-mon[97450]: pgmap v12961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:51.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:51.060427+0000) 2022-09-08T04:18:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:51.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:51.044791+0000) 2022-09-08T04:18:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:52.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:52.060560+0000) 2022-09-08T04:18:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:52.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:52.045016+0000) 2022-09-08T04:18:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:52 smithi055 ceph-mon[97033]: pgmap v12962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:18:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:52 smithi059 ceph-mon[97450]: pgmap v12962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:18:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:53.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:53.060757+0000) 2022-09-08T04:18:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:53.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:53.045216+0000) 2022-09-08T04:18:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:54.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:54.061018+0000) 2022-09-08T04:18:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:54.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:54.045474+0000) 2022-09-08T04:18:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:54 smithi055 ceph-mon[97033]: pgmap v12963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:54 smithi059 ceph-mon[97450]: pgmap v12963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:55.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:55.061273+0000) 2022-09-08T04:18:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:55.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:55.045711+0000) 2022-09-08T04:18:56.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:56.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:56.045930+0000) 2022-09-08T04:18:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:56.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:56.061500+0000) 2022-09-08T04:18:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:56 smithi055 ceph-mon[97033]: pgmap v12964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:18:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:56 smithi059 ceph-mon[97450]: pgmap v12964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:18:57.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:57.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:57.046182+0000) 2022-09-08T04:18:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:57.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:57.061763+0000) 2022-09-08T04:18:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:58.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:58.062005+0000) 2022-09-08T04:18:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:58.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:58.046436+0000) 2022-09-08T04:18:58.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:18:58 smithi055 ceph-mon[97033]: pgmap v12965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:18:58 smithi059 ceph-mon[97450]: pgmap v12965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:18:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:18:59.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:59.062243+0000) 2022-09-08T04:18:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:18:59.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:18:59.046598+0000) 2022-09-08T04:19:00.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:19:00.205+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:19:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:00.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:00.046777+0000) 2022-09-08T04:19:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:00.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:00.062490+0000) 2022-09-08T04:19:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:00 smithi055 ceph-mon[97033]: pgmap v12966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:01.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:00 smithi059 ceph-mon[97450]: pgmap v12966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:01.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:01.062713+0000) 2022-09-08T04:19:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:01.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:01.046932+0000) 2022-09-08T04:19:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:02.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:02.062895+0000) 2022-09-08T04:19:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:02.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:02.047145+0000) 2022-09-08T04:19:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:02 smithi055 ceph-mon[97033]: pgmap v12967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:19:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:02 smithi059 ceph-mon[97450]: pgmap v12967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:19:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:03.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:03.063141+0000) 2022-09-08T04:19:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:03.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:03.047298+0000) 2022-09-08T04:19:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:04.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:04.063404+0000) 2022-09-08T04:19:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:04.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:04.047551+0000) 2022-09-08T04:19:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:04 smithi055 ceph-mon[97033]: pgmap v12968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:04 smithi059 ceph-mon[97450]: pgmap v12968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:05.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:05.063672+0000) 2022-09-08T04:19:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:05.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:05.047758+0000) 2022-09-08T04:19:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:19:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:19:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:06.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:06.063893+0000) 2022-09-08T04:19:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:06.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:06.047903+0000) 2022-09-08T04:19:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:06 smithi055 ceph-mon[97033]: pgmap v12969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:19:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:19:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:06 smithi059 ceph-mon[97450]: pgmap v12969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:19:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:19:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:07.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:07.064085+0000) 2022-09-08T04:19:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:07.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:07.048138+0000) 2022-09-08T04:19:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:08.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:08.064266+0000) 2022-09-08T04:19:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:08.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:08.048371+0000) 2022-09-08T04:19:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:08 smithi055 ceph-mon[97033]: pgmap v12970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:19:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:08 smithi059 ceph-mon[97450]: pgmap v12970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:19:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:09.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:09.064418+0000) 2022-09-08T04:19:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:09.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:09.048632+0000) 2022-09-08T04:19:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:10.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:10.064630+0000) 2022-09-08T04:19:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:10.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:10.048909+0000) 2022-09-08T04:19:10.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:10 smithi055 ceph-mon[97033]: pgmap v12971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:19:10.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:10 smithi055 ceph-mon[97033]: osdmap e891: 12 total, 11 up, 11 in 2022-09-08T04:19:11.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:10 smithi059 ceph-mon[97450]: pgmap v12971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:19:11.041 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:10 smithi059 ceph-mon[97450]: osdmap e891: 12 total, 11 up, 11 in 2022-09-08T04:19:11.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:11.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:11.064797+0000) 2022-09-08T04:19:11.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:11.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:11.049138+0000) 2022-09-08T04:19:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:12.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:12.065046+0000) 2022-09-08T04:19:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:12.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:12.049347+0000) 2022-09-08T04:19:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:12 smithi055 ceph-mon[97033]: pgmap v12973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:19:13.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:12 smithi059 ceph-mon[97450]: pgmap v12973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:19:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:13.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:13.065342+0000) 2022-09-08T04:19:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:13.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:13.049584+0000) 2022-09-08T04:19:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:14.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:14.065577+0000) 2022-09-08T04:19:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:14.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:14.049826+0000) 2022-09-08T04:19:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:14 smithi055 ceph-mon[97033]: pgmap v12974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:19:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:14 smithi059 ceph-mon[97450]: pgmap v12974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:19:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:15.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:15.065855+0000) 2022-09-08T04:19:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:15.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:15.050016+0000) 2022-09-08T04:19:15.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:19:15.222+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:19:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:16.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:16.065990+0000) 2022-09-08T04:19:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:16.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:16.050234+0000) 2022-09-08T04:19:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:16 smithi055 ceph-mon[97033]: pgmap v12975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:19:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:16 smithi059 ceph-mon[97450]: pgmap v12975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:19:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:17.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:17.066204+0000) 2022-09-08T04:19:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:17.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:17.050428+0000) 2022-09-08T04:19:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:19:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:19:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:19:18.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:19:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:19:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:19:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:18.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:18.066436+0000) 2022-09-08T04:19:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:18.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:18.050673+0000) 2022-09-08T04:19:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:18 smithi055 ceph-mon[97033]: pgmap v12976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:19:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:19:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:18 smithi059 ceph-mon[97450]: pgmap v12976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:19:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:19:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:19.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:19.066616+0000) 2022-09-08T04:19:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:19.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:19.050850+0000) 2022-09-08T04:19:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:20.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:20.066819+0000) 2022-09-08T04:19:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:20.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:20.051091+0000) 2022-09-08T04:19:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:20 smithi055 ceph-mon[97033]: pgmap v12977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:19:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:20 smithi055 ceph-mon[97033]: osdmap e892: 12 total, 11 up, 11 in 2022-09-08T04:19:21.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:20 smithi059 ceph-mon[97450]: pgmap v12977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:19:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:20 smithi059 ceph-mon[97450]: osdmap e892: 12 total, 11 up, 11 in 2022-09-08T04:19:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:21.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:21.067061+0000) 2022-09-08T04:19:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:21.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:21.051324+0000) 2022-09-08T04:19:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:22.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:22.067294+0000) 2022-09-08T04:19:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:22.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:22.051498+0000) 2022-09-08T04:19:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:22 smithi055 ceph-mon[97033]: pgmap v12979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:19:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:22 smithi059 ceph-mon[97450]: pgmap v12979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:19:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:23.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:23.067539+0000) 2022-09-08T04:19:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:23.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:23.051725+0000) 2022-09-08T04:19:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:24.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:24.067760+0000) 2022-09-08T04:19:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:24.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:24.051949+0000) 2022-09-08T04:19:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:24 smithi055 ceph-mon[97033]: pgmap v12980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:19:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:24 smithi059 ceph-mon[97450]: pgmap v12980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:19:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:25.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:25.067982+0000) 2022-09-08T04:19:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:25.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:25.052160+0000) 2022-09-08T04:19:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:26.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:26.068216+0000) 2022-09-08T04:19:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:26.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:26.052442+0000) 2022-09-08T04:19:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:26 smithi055 ceph-mon[97033]: pgmap v12981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:19:27.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:26 smithi059 ceph-mon[97450]: pgmap v12981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:19:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:27.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:27.068424+0000) 2022-09-08T04:19:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:27.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:27.052677+0000) 2022-09-08T04:19:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:28.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:28.068593+0000) 2022-09-08T04:19:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:28.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:28.052811+0000) 2022-09-08T04:19:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:28 smithi055 ceph-mon[97033]: pgmap v12982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:19:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:28 smithi059 ceph-mon[97450]: pgmap v12982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:19:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:29.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:29.068850+0000) 2022-09-08T04:19:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:29.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:29.053037+0000) 2022-09-08T04:19:30.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:19:30.225+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:19:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:30.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:30.069025+0000) 2022-09-08T04:19:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:30.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:30.053339+0000) 2022-09-08T04:19:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:30 smithi055 ceph-mon[97033]: pgmap v12983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:19:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:30 smithi059 ceph-mon[97450]: pgmap v12983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:19:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:31.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:31.069272+0000) 2022-09-08T04:19:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:31.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:31.053551+0000) 2022-09-08T04:19:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:32.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:32.069450+0000) 2022-09-08T04:19:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:32.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:32.053788+0000) 2022-09-08T04:19:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:32 smithi055 ceph-mon[97033]: pgmap v12984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:19:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:32 smithi059 ceph-mon[97450]: pgmap v12984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:19:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:33.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:33.069669+0000) 2022-09-08T04:19:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:33.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:33.054029+0000) 2022-09-08T04:19:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:34.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:34.069877+0000) 2022-09-08T04:19:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:34.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:34.054205+0000) 2022-09-08T04:19:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:34 smithi055 ceph-mon[97033]: pgmap v12985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:34 smithi059 ceph-mon[97450]: pgmap v12985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:35.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:35.070068+0000) 2022-09-08T04:19:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:35.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:35.054360+0000) 2022-09-08T04:19:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:36.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:36.070298+0000) 2022-09-08T04:19:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:36.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:36.054571+0000) 2022-09-08T04:19:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:36 smithi055 ceph-mon[97033]: pgmap v12986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:19:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:36 smithi059 ceph-mon[97450]: pgmap v12986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:19:37.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:37.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:37.054841+0000) 2022-09-08T04:19:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:37.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:37.070497+0000) 2022-09-08T04:19:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:38.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:38.070749+0000) 2022-09-08T04:19:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:38.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:38.055025+0000) 2022-09-08T04:19:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:38 smithi055 ceph-mon[97033]: pgmap v12987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:38 smithi059 ceph-mon[97450]: pgmap v12987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:39.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:39.071006+0000) 2022-09-08T04:19:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:39.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:39.055243+0000) 2022-09-08T04:19:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:40.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:40.071234+0000) 2022-09-08T04:19:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:40.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:40.055444+0000) 2022-09-08T04:19:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:40 smithi055 ceph-mon[97033]: pgmap v12988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:40 smithi059 ceph-mon[97450]: pgmap v12988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:41.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:41.071471+0000) 2022-09-08T04:19:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:41.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:41.055707+0000) 2022-09-08T04:19:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:42.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:42.071678+0000) 2022-09-08T04:19:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:42.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:42.055946+0000) 2022-09-08T04:19:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:42 smithi055 ceph-mon[97033]: pgmap v12989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:19:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:42 smithi059 ceph-mon[97450]: pgmap v12989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:19:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:43.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:43.071890+0000) 2022-09-08T04:19:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:43.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:43.056195+0000) 2022-09-08T04:19:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:44.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:44.072068+0000) 2022-09-08T04:19:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:44.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:44.056483+0000) 2022-09-08T04:19:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:44 smithi055 ceph-mon[97033]: pgmap v12990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:44 smithi059 ceph-mon[97450]: pgmap v12990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:45.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:19:45.236+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:19:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:45.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:45.072223+0000) 2022-09-08T04:19:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:45.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:45.056747+0000) 2022-09-08T04:19:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:46.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:46.072461+0000) 2022-09-08T04:19:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:46.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:46.056919+0000) 2022-09-08T04:19:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:46 smithi055 ceph-mon[97033]: pgmap v12991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:19:47.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:46 smithi059 ceph-mon[97450]: pgmap v12991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:19:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:47.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:47.072721+0000) 2022-09-08T04:19:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:47.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:47.057165+0000) 2022-09-08T04:19:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:48.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:48.072951+0000) 2022-09-08T04:19:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:48.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:48.057406+0000) 2022-09-08T04:19:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:48 smithi055 ceph-mon[97033]: pgmap v12992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:48 smithi059 ceph-mon[97450]: pgmap v12992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:49.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:49.073111+0000) 2022-09-08T04:19:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:49.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:49.057717+0000) 2022-09-08T04:19:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:50.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:50.073287+0000) 2022-09-08T04:19:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:50.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:50.057977+0000) 2022-09-08T04:19:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:50 smithi055 ceph-mon[97033]: pgmap v12993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:51.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:50 smithi059 ceph-mon[97450]: pgmap v12993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:51.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:51.073546+0000) 2022-09-08T04:19:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:51.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:51.058196+0000) 2022-09-08T04:19:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:52.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:52.073731+0000) 2022-09-08T04:19:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:52.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:52.058420+0000) 2022-09-08T04:19:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:52 smithi055 ceph-mon[97033]: pgmap v12994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:19:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:52 smithi059 ceph-mon[97450]: pgmap v12994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:19:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:53.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:53.073981+0000) 2022-09-08T04:19:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:53.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:53.058711+0000) 2022-09-08T04:19:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:54.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:54.074209+0000) 2022-09-08T04:19:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:54.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:54.058983+0000) 2022-09-08T04:19:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:54 smithi055 ceph-mon[97033]: pgmap v12995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:54 smithi059 ceph-mon[97450]: pgmap v12995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:55.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:55.074372+0000) 2022-09-08T04:19:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:55.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:55.059146+0000) 2022-09-08T04:19:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:56.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:56.074611+0000) 2022-09-08T04:19:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:56.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:56.059390+0000) 2022-09-08T04:19:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:56 smithi055 ceph-mon[97033]: pgmap v12996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:19:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:56 smithi059 ceph-mon[97450]: pgmap v12996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:19:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:57.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:57.074869+0000) 2022-09-08T04:19:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:57.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:57.059629+0000) 2022-09-08T04:19:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:58.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:58.075000+0000) 2022-09-08T04:19:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:58.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:58.059908+0000) 2022-09-08T04:19:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:19:58 smithi055 ceph-mon[97033]: pgmap v12997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:19:58 smithi059 ceph-mon[97450]: pgmap v12997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:19:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:19:59.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:59.075228+0000) 2022-09-08T04:19:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:19:59.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:19:59.060159+0000) 2022-09-08T04:20:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:00.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:00.075476+0000) 2022-09-08T04:20:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:00.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:00.060383+0000) 2022-09-08T04:20:00.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:20:00.238+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:20:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:00 smithi055 ceph-mon[97033]: pgmap v12998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T04:20:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T04:20:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T04:20:01.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:00 smithi059 ceph-mon[97450]: pgmap v12998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T04:20:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T04:20:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T04:20:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:01.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:01.060620+0000) 2022-09-08T04:20:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:01.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:01.075695+0000) 2022-09-08T04:20:02.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:02.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:02.060894+0000) 2022-09-08T04:20:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:02.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:02.075810+0000) 2022-09-08T04:20:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:02 smithi055 ceph-mon[97033]: pgmap v12999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:20:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:02 smithi059 ceph-mon[97450]: pgmap v12999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:20:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:03.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:03.076002+0000) 2022-09-08T04:20:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:03.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:03.061034+0000) 2022-09-08T04:20:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:04.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:04.076180+0000) 2022-09-08T04:20:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:04.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:04.061228+0000) 2022-09-08T04:20:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:04 smithi055 ceph-mon[97033]: pgmap v13000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:04 smithi059 ceph-mon[97450]: pgmap v13000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:05.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:05.076386+0000) 2022-09-08T04:20:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:05.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:05.061454+0000) 2022-09-08T04:20:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:20:06.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:20:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:06.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:06.076548+0000) 2022-09-08T04:20:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:06.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:06.061721+0000) 2022-09-08T04:20:06.868 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:06 smithi059 ceph-mon[97450]: pgmap v13001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:20:06.868 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:20:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:06 smithi055 ceph-mon[97033]: pgmap v13001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:20:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:20:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:07.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:07.076739+0000) 2022-09-08T04:20:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:07.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:07.061961+0000) 2022-09-08T04:20:08.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:08.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:08.062180+0000) 2022-09-08T04:20:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:08.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:08.076941+0000) 2022-09-08T04:20:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:08 smithi055 ceph-mon[97033]: pgmap v13002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:20:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:08 smithi059 ceph-mon[97450]: pgmap v13002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:20:09.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:09.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:09.062427+0000) 2022-09-08T04:20:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:09.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:09.077119+0000) 2022-09-08T04:20:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:10.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:10.077339+0000) 2022-09-08T04:20:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:10.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:10.062689+0000) 2022-09-08T04:20:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:10 smithi055 ceph-mon[97033]: pgmap v13003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:20:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:10 smithi055 ceph-mon[97033]: osdmap e893: 12 total, 11 up, 11 in 2022-09-08T04:20:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:10 smithi059 ceph-mon[97450]: pgmap v13003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:20:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:10 smithi059 ceph-mon[97450]: osdmap e893: 12 total, 11 up, 11 in 2022-09-08T04:20:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:11.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:11.077525+0000) 2022-09-08T04:20:11.418 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:11.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:11.062868+0000) 2022-09-08T04:20:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:12.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:12.077720+0000) 2022-09-08T04:20:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:12.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:12.063102+0000) 2022-09-08T04:20:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:12 smithi055 ceph-mon[97033]: pgmap v13005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:20:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:12 smithi059 ceph-mon[97450]: pgmap v13005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:20:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:13.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:13.077950+0000) 2022-09-08T04:20:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:13.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:13.063268+0000) 2022-09-08T04:20:14.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:14.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:14.063495+0000) 2022-09-08T04:20:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:14.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:14.078230+0000) 2022-09-08T04:20:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:14 smithi055 ceph-mon[97033]: pgmap v13006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:20:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:14 smithi059 ceph-mon[97450]: pgmap v13006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:20:15.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:20:15.247+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:20:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:15.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:15.078500+0000) 2022-09-08T04:20:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:15.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:15.063748+0000) 2022-09-08T04:20:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:16.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:16.078743+0000) 2022-09-08T04:20:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:16.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:16.063904+0000) 2022-09-08T04:20:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:16 smithi055 ceph-mon[97033]: pgmap v13007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:20:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:16 smithi059 ceph-mon[97450]: pgmap v13007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:20:17.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:17.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:17.064126+0000) 2022-09-08T04:20:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:17.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:17.078910+0000) 2022-09-08T04:20:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:18.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:18.079127+0000) 2022-09-08T04:20:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:18.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:18.064396+0000) 2022-09-08T04:20:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:18 smithi055 ceph-mon[97033]: pgmap v13008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:20:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:20:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:20:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:20:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:18 smithi059 ceph-mon[97450]: pgmap v13008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:20:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:20:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:20:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:20:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:19.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:19.079363+0000) 2022-09-08T04:20:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:19.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:19.064680+0000) 2022-09-08T04:20:20.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:20.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:19 smithi059 ceph-mon[97450]: osdmap e894: 12 total, 11 up, 11 in 2022-09-08T04:20:20.089 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:20.090 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:20.091 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:20.091 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:20.091 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:19 smithi055 ceph-mon[97033]: osdmap e894: 12 total, 11 up, 11 in 2022-09-08T04:20:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:20.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:20.079500+0000) 2022-09-08T04:20:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:20.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:20.064959+0000) 2022-09-08T04:20:21.101 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:20 smithi059 ceph-mon[97450]: pgmap v13009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:20:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:20 smithi055 ceph-mon[97033]: pgmap v13009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:20:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:21.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:21.079749+0000) 2022-09-08T04:20:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:21.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:21.065156+0000) 2022-09-08T04:20:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:22.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:22.079927+0000) 2022-09-08T04:20:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:22.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:22.065385+0000) 2022-09-08T04:20:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:22 smithi059 ceph-mon[97450]: pgmap v13011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:20:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:22 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T04:20:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:22 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T04:20:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:22.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:22 smithi055 ceph-mon[97033]: pgmap v13011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:20:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:22 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T04:20:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:22 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T04:20:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:22.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:20:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:23.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:23.080137+0000) 2022-09-08T04:20:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:23.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:23.065605+0000) 2022-09-08T04:20:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:24.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:24.080355+0000) 2022-09-08T04:20:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:24.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:24.065749+0000) 2022-09-08T04:20:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:24 smithi055 ceph-mon[97033]: pgmap v13012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:20:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:24 smithi059 ceph-mon[97450]: pgmap v13012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:20:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:25.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:25.080632+0000) 2022-09-08T04:20:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:25.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:25.065998+0000) 2022-09-08T04:20:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:26.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:26.080886+0000) 2022-09-08T04:20:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:26.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:26.066176+0000) 2022-09-08T04:20:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:26 smithi055 ceph-mon[97033]: pgmap v13013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:20:27.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:26 smithi059 ceph-mon[97450]: pgmap v13013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:20:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:27.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:27.081061+0000) 2022-09-08T04:20:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:27.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:27.066368+0000) 2022-09-08T04:20:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:28.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:28.081252+0000) 2022-09-08T04:20:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:28.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:28.066592+0000) 2022-09-08T04:20:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:28 smithi055 ceph-mon[97033]: pgmap v13014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:20:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:28 smithi059 ceph-mon[97450]: pgmap v13014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:20:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:29.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:29.081450+0000) 2022-09-08T04:20:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:29.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:29.066902+0000) 2022-09-08T04:20:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:30.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:30.081655+0000) 2022-09-08T04:20:30.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:20:30.248+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:20:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:30.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:30.067114+0000) 2022-09-08T04:20:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:30 smithi055 ceph-mon[97033]: pgmap v13015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:20:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:30 smithi059 ceph-mon[97450]: pgmap v13015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:20:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:31.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:31.081823+0000) 2022-09-08T04:20:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:31.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:31.067379+0000) 2022-09-08T04:20:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:32.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:32.082046+0000) 2022-09-08T04:20:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:32.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:32.067615+0000) 2022-09-08T04:20:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:32 smithi055 ceph-mon[97033]: pgmap v13016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T04:20:33.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:32 smithi059 ceph-mon[97450]: pgmap v13016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T04:20:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:33.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:33.082284+0000) 2022-09-08T04:20:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:33.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:33.067862+0000) 2022-09-08T04:20:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:34.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:34.082522+0000) 2022-09-08T04:20:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:34.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:34.068017+0000) 2022-09-08T04:20:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:34 smithi055 ceph-mon[97033]: pgmap v13017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:20:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:34 smithi059 ceph-mon[97450]: pgmap v13017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:20:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:35.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:35.082744+0000) 2022-09-08T04:20:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:35.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:35.068196+0000) 2022-09-08T04:20:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:36.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:36.082879+0000) 2022-09-08T04:20:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:36.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:36.068357+0000) 2022-09-08T04:20:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:36 smithi055 ceph-mon[97033]: pgmap v13018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:20:37.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:36 smithi059 ceph-mon[97450]: pgmap v13018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:20:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:37.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:37.083051+0000) 2022-09-08T04:20:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:37.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:37.068574+0000) 2022-09-08T04:20:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:38.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:38.083295+0000) 2022-09-08T04:20:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:38.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:38.068744+0000) 2022-09-08T04:20:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:38 smithi055 ceph-mon[97033]: pgmap v13019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:38 smithi059 ceph-mon[97450]: pgmap v13019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:39.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:39.083509+0000) 2022-09-08T04:20:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:39.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:39.068956+0000) 2022-09-08T04:20:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:40.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:40.083745+0000) 2022-09-08T04:20:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:40.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:40.069188+0000) 2022-09-08T04:20:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:40 smithi055 ceph-mon[97033]: pgmap v13020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:40 smithi059 ceph-mon[97450]: pgmap v13020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:41.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:41.083984+0000) 2022-09-08T04:20:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:41.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:41.069434+0000) 2022-09-08T04:20:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:42.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:42.084138+0000) 2022-09-08T04:20:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:42.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:42.069718+0000) 2022-09-08T04:20:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:42 smithi055 ceph-mon[97033]: pgmap v13021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:20:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:42 smithi059 ceph-mon[97450]: pgmap v13021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:20:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:43.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:43.084323+0000) 2022-09-08T04:20:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:43.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:43.069968+0000) 2022-09-08T04:20:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:44.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:44.084546+0000) 2022-09-08T04:20:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:44.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:44.070197+0000) 2022-09-08T04:20:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:44 smithi055 ceph-mon[97033]: pgmap v13022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:44 smithi059 ceph-mon[97450]: pgmap v13022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:45.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:45.084768+0000) 2022-09-08T04:20:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:45.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:45.070465+0000) 2022-09-08T04:20:45.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:20:45.253+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:20:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:46.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:46.084968+0000) 2022-09-08T04:20:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:46.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:46.070729+0000) 2022-09-08T04:20:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:46 smithi055 ceph-mon[97033]: pgmap v13023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:20:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:46 smithi059 ceph-mon[97450]: pgmap v13023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:20:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:47.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:47.085158+0000) 2022-09-08T04:20:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:47.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:47.070912+0000) 2022-09-08T04:20:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:48.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:48.085315+0000) 2022-09-08T04:20:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:48.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:48.071102+0000) 2022-09-08T04:20:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:48 smithi055 ceph-mon[97033]: pgmap v13024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:49.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:48 smithi059 ceph-mon[97450]: pgmap v13024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:49.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:49.085536+0000) 2022-09-08T04:20:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:49.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:49.071246+0000) 2022-09-08T04:20:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:50.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:50.085705+0000) 2022-09-08T04:20:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:50.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:50.071371+0000) 2022-09-08T04:20:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:50 smithi055 ceph-mon[97033]: pgmap v13025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:50 smithi059 ceph-mon[97450]: pgmap v13025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:51.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:51.085835+0000) 2022-09-08T04:20:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:51.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:51.071575+0000) 2022-09-08T04:20:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:52.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:52.086069+0000) 2022-09-08T04:20:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:52.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:52.071793+0000) 2022-09-08T04:20:52.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:52 smithi055 ceph-mon[97033]: pgmap v13026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:20:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:52 smithi059 ceph-mon[97450]: pgmap v13026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:20:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:53.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:53.086354+0000) 2022-09-08T04:20:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:53.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:53.072006+0000) 2022-09-08T04:20:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:54.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:54.086606+0000) 2022-09-08T04:20:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:54.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:54.072232+0000) 2022-09-08T04:20:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:54 smithi055 ceph-mon[97033]: pgmap v13027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:54 smithi059 ceph-mon[97450]: pgmap v13027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:55.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:55.086796+0000) 2022-09-08T04:20:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:55.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:55.072463+0000) 2022-09-08T04:20:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:56.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:56.086964+0000) 2022-09-08T04:20:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:56.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:56.072686+0000) 2022-09-08T04:20:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:56 smithi055 ceph-mon[97033]: pgmap v13028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:20:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:56 smithi059 ceph-mon[97450]: pgmap v13028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:20:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:57.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:57.087113+0000) 2022-09-08T04:20:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:57.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:57.072928+0000) 2022-09-08T04:20:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:58.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:58.087281+0000) 2022-09-08T04:20:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:58.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:58.073209+0000) 2022-09-08T04:20:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:20:58 smithi055 ceph-mon[97033]: pgmap v13029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:59.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:20:58 smithi059 ceph-mon[97450]: pgmap v13029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:20:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:20:59.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:59.087510+0000) 2022-09-08T04:20:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:20:59.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:20:59.073483+0000) 2022-09-08T04:21:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:00.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:00.087770+0000) 2022-09-08T04:21:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:00.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:00.073784+0000) 2022-09-08T04:21:00.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:21:00.258+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:21:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:00 smithi055 ceph-mon[97033]: pgmap v13030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:01.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:00 smithi059 ceph-mon[97450]: pgmap v13030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:01.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:01.087955+0000) 2022-09-08T04:21:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:01.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:01.073919+0000) 2022-09-08T04:21:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:02.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:02.088164+0000) 2022-09-08T04:21:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:02.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:02.074154+0000) 2022-09-08T04:21:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:02 smithi055 ceph-mon[97033]: pgmap v13031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:21:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:02 smithi059 ceph-mon[97450]: pgmap v13031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:21:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:03.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:03.088408+0000) 2022-09-08T04:21:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:03.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:03.074380+0000) 2022-09-08T04:21:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:04.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:04.088617+0000) 2022-09-08T04:21:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:04.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:04.074623+0000) 2022-09-08T04:21:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:04 smithi055 ceph-mon[97033]: pgmap v13032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:04 smithi059 ceph-mon[97450]: pgmap v13032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:05.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:05.088918+0000) 2022-09-08T04:21:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:05.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:05.074859+0000) 2022-09-08T04:21:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:21:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:21:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:06.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:06.089136+0000) 2022-09-08T04:21:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:06.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:06.075082+0000) 2022-09-08T04:21:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:06 smithi055 ceph-mon[97033]: pgmap v13033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:21:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:21:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:06 smithi059 ceph-mon[97450]: pgmap v13033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:21:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:21:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:07.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:07.089376+0000) 2022-09-08T04:21:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:07.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:07.075287+0000) 2022-09-08T04:21:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:08.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:08.089578+0000) 2022-09-08T04:21:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:08.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:08.075478+0000) 2022-09-08T04:21:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:08 smithi055 ceph-mon[97033]: pgmap v13034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:21:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:08 smithi059 ceph-mon[97450]: pgmap v13034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:21:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:09.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:09.089782+0000) 2022-09-08T04:21:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:09.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:09.075751+0000) 2022-09-08T04:21:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:10.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:10.089998+0000) 2022-09-08T04:21:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:10.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:10.075923+0000) 2022-09-08T04:21:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:10 smithi055 ceph-mon[97033]: pgmap v13035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:21:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:10 smithi055 ceph-mon[97033]: osdmap e895: 12 total, 11 up, 11 in 2022-09-08T04:21:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:10 smithi059 ceph-mon[97450]: pgmap v13035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:21:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:10 smithi059 ceph-mon[97450]: osdmap e895: 12 total, 11 up, 11 in 2022-09-08T04:21:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:11.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:11.090239+0000) 2022-09-08T04:21:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:11.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:11.076120+0000) 2022-09-08T04:21:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:12.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:12.090372+0000) 2022-09-08T04:21:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:12.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:12.076350+0000) 2022-09-08T04:21:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:12 smithi055 ceph-mon[97033]: pgmap v13037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:21:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:12 smithi059 ceph-mon[97450]: pgmap v13037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:21:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:13.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:13.090586+0000) 2022-09-08T04:21:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:13.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:13.076500+0000) 2022-09-08T04:21:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:14.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:14.090731+0000) 2022-09-08T04:21:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:14.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:14.076716+0000) 2022-09-08T04:21:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:14 smithi055 ceph-mon[97033]: pgmap v13038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:21:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:14 smithi055 ceph-mon[97033]: osdmap e896: 12 total, 11 up, 11 in 2022-09-08T04:21:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:14 smithi059 ceph-mon[97450]: pgmap v13038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:21:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:14 smithi059 ceph-mon[97450]: osdmap e896: 12 total, 11 up, 11 in 2022-09-08T04:21:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:15.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:15.090960+0000) 2022-09-08T04:21:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:15.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:15.076914+0000) 2022-09-08T04:21:15.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:21:15.273+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:21:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:16.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:16.091206+0000) 2022-09-08T04:21:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:16.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:16.077114+0000) 2022-09-08T04:21:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:16 smithi055 ceph-mon[97033]: pgmap v13040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:21:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:16 smithi059 ceph-mon[97450]: pgmap v13040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:21:17.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:17.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:17.077328+0000) 2022-09-08T04:21:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:17.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:17.091415+0000) 2022-09-08T04:21:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:18.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:18.091689+0000) 2022-09-08T04:21:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:18.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:18.077557+0000) 2022-09-08T04:21:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:18 smithi055 ceph-mon[97033]: pgmap v13041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:21:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:18 smithi059 ceph-mon[97450]: pgmap v13041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:21:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:19.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:19.091841+0000) 2022-09-08T04:21:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:19.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:19.077801+0000) 2022-09-08T04:21:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:20.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:20.092047+0000) 2022-09-08T04:21:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:20.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:20.078070+0000) 2022-09-08T04:21:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:20 smithi055 ceph-mon[97033]: pgmap v13042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2022-09-08T04:21:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:20 smithi059 ceph-mon[97450]: pgmap v13042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2022-09-08T04:21:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:21.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:21.092271+0000) 2022-09-08T04:21:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:21.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:21.078325+0000) 2022-09-08T04:21:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:22.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:22.092512+0000) 2022-09-08T04:21:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:22.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:22.078567+0000) 2022-09-08T04:21:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:22 smithi059 ceph-mon[97450]: pgmap v13043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:21:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:21:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:21:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:21:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:21:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:22 smithi055 ceph-mon[97033]: pgmap v13043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:21:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:21:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:21:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:21:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:21:23.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:23.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:23.078793+0000) 2022-09-08T04:21:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:23.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:23.092784+0000) 2022-09-08T04:21:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:24.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:24.093006+0000) 2022-09-08T04:21:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:24.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:24.078959+0000) 2022-09-08T04:21:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:24 smithi055 ceph-mon[97033]: pgmap v13044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:21:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:24 smithi059 ceph-mon[97450]: pgmap v13044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:21:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:25.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:25.093254+0000) 2022-09-08T04:21:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:25.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:25.079115+0000) 2022-09-08T04:21:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:26.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:26.093397+0000) 2022-09-08T04:21:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:26.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:26.079280+0000) 2022-09-08T04:21:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:26 smithi055 ceph-mon[97033]: pgmap v13045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-08T04:21:27.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:26 smithi059 ceph-mon[97450]: pgmap v13045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-08T04:21:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:27.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:27.093606+0000) 2022-09-08T04:21:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:27.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:27.079496+0000) 2022-09-08T04:21:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:28.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:28.093874+0000) 2022-09-08T04:21:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:28.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:28.079690+0000) 2022-09-08T04:21:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:28 smithi055 ceph-mon[97033]: pgmap v13046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:28 smithi059 ceph-mon[97450]: pgmap v13046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:29.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:29.094108+0000) 2022-09-08T04:21:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:29.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:29.079889+0000) 2022-09-08T04:21:30.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:30.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:30.080065+0000) 2022-09-08T04:21:30.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:21:30.274+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:21:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:30.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:30.094359+0000) 2022-09-08T04:21:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:30 smithi055 ceph-mon[97033]: pgmap v13047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:30 smithi059 ceph-mon[97450]: pgmap v13047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:31.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:31.094529+0000) 2022-09-08T04:21:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:31.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:31.080261+0000) 2022-09-08T04:21:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:32.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:32.094747+0000) 2022-09-08T04:21:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:32.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:32.080460+0000) 2022-09-08T04:21:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:32 smithi055 ceph-mon[97033]: pgmap v13048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:21:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:32 smithi059 ceph-mon[97450]: pgmap v13048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:21:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:33.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:33.095000+0000) 2022-09-08T04:21:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:33.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:33.080757+0000) 2022-09-08T04:21:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:34.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:34.095223+0000) 2022-09-08T04:21:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:34.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:34.080902+0000) 2022-09-08T04:21:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:34 smithi055 ceph-mon[97033]: pgmap v13049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:34 smithi059 ceph-mon[97450]: pgmap v13049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:35.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:35.095423+0000) 2022-09-08T04:21:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:35.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:35.081108+0000) 2022-09-08T04:21:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:36.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:36.095607+0000) 2022-09-08T04:21:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:36.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:36.081383+0000) 2022-09-08T04:21:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:36 smithi055 ceph-mon[97033]: pgmap v13050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:21:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:36 smithi059 ceph-mon[97450]: pgmap v13050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:21:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:37.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:37.095850+0000) 2022-09-08T04:21:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:37.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:37.081698+0000) 2022-09-08T04:21:38.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:38.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:38.081857+0000) 2022-09-08T04:21:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:38.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:38.096111+0000) 2022-09-08T04:21:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:38 smithi055 ceph-mon[97033]: pgmap v13051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:38 smithi059 ceph-mon[97450]: pgmap v13051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:39.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:39.096351+0000) 2022-09-08T04:21:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:39.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:39.081996+0000) 2022-09-08T04:21:40.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:40.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:40.082163+0000) 2022-09-08T04:21:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:40.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:40.096591+0000) 2022-09-08T04:21:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:40 smithi055 ceph-mon[97033]: pgmap v13052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:40 smithi059 ceph-mon[97450]: pgmap v13052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:41.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:41.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:41.082363+0000) 2022-09-08T04:21:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:41.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:41.096814+0000) 2022-09-08T04:21:42.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:42.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:42.082589+0000) 2022-09-08T04:21:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:42.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:42.097038+0000) 2022-09-08T04:21:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:42 smithi055 ceph-mon[97033]: pgmap v13053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:21:43.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:42 smithi059 ceph-mon[97450]: pgmap v13053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:21:43.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:43.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:43.082805+0000) 2022-09-08T04:21:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:43.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:43.097225+0000) 2022-09-08T04:21:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:44.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:44.097423+0000) 2022-09-08T04:21:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:44.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:44.083014+0000) 2022-09-08T04:21:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:44 smithi055 ceph-mon[97033]: pgmap v13054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:44 smithi059 ceph-mon[97450]: pgmap v13054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:45.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:45.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:45.083261+0000) 2022-09-08T04:21:45.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:21:45.291+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:21:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:45.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:45.097654+0000) 2022-09-08T04:21:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:46.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:46.083484+0000) 2022-09-08T04:21:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:46.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:46.097904+0000) 2022-09-08T04:21:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:46 smithi055 ceph-mon[97033]: pgmap v13055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:21:47.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:46 smithi059 ceph-mon[97450]: pgmap v13055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:21:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:47.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:47.098144+0000) 2022-09-08T04:21:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:47.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:47.083763+0000) 2022-09-08T04:21:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:48.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:48.098378+0000) 2022-09-08T04:21:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:48.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:48.083966+0000) 2022-09-08T04:21:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:48 smithi055 ceph-mon[97033]: pgmap v13056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:49.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:48 smithi059 ceph-mon[97450]: pgmap v13056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:49.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:49.098630+0000) 2022-09-08T04:21:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:49.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:49.084195+0000) 2022-09-08T04:21:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:50.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:50.098817+0000) 2022-09-08T04:21:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:50.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:50.084440+0000) 2022-09-08T04:21:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:50 smithi055 ceph-mon[97033]: pgmap v13057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:51.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:50 smithi059 ceph-mon[97450]: pgmap v13057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:51.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:51.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:51.084624+0000) 2022-09-08T04:21:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:51.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:51.098971+0000) 2022-09-08T04:21:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:52.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:52.099150+0000) 2022-09-08T04:21:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:52.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:52.084792+0000) 2022-09-08T04:21:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:52 smithi055 ceph-mon[97033]: pgmap v13058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:21:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:52 smithi059 ceph-mon[97450]: pgmap v13058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:21:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:53.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:53.099380+0000) 2022-09-08T04:21:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:53.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:53.084977+0000) 2022-09-08T04:21:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:54.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:54.099608+0000) 2022-09-08T04:21:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:54.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:54.085193+0000) 2022-09-08T04:21:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:54 smithi055 ceph-mon[97033]: pgmap v13059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:54 smithi059 ceph-mon[97450]: pgmap v13059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:55.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:55.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:55.085495+0000) 2022-09-08T04:21:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:55.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:55.099870+0000) 2022-09-08T04:21:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:56.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:56.100083+0000) 2022-09-08T04:21:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:56.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:56.085797+0000) 2022-09-08T04:21:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:56 smithi055 ceph-mon[97033]: pgmap v13060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:21:57.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:56 smithi059 ceph-mon[97450]: pgmap v13060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:21:57.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:57.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:57.086046+0000) 2022-09-08T04:21:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:57.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:57.100295+0000) 2022-09-08T04:21:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:58.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:58.100492+0000) 2022-09-08T04:21:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:58.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:58.086292+0000) 2022-09-08T04:21:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:21:58 smithi055 ceph-mon[97033]: pgmap v13061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:59.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:21:58 smithi059 ceph-mon[97450]: pgmap v13061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:21:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:21:59.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:59.100731+0000) 2022-09-08T04:21:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:21:59.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:21:59.086530+0000) 2022-09-08T04:22:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:00.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:00.100873+0000) 2022-09-08T04:22:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:00.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:00.086718+0000) 2022-09-08T04:22:00.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:22:00.306+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:22:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:00 smithi055 ceph-mon[97033]: pgmap v13062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:01.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:00 smithi059 ceph-mon[97450]: pgmap v13062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:01.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:01.086861+0000) 2022-09-08T04:22:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:01.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:01.101023+0000) 2022-09-08T04:22:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:02.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:02.101258+0000) 2022-09-08T04:22:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:02.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:02.087034+0000) 2022-09-08T04:22:03.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:02 smithi059 ceph-mon[97450]: pgmap v13063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:22:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:02 smithi055 ceph-mon[97033]: pgmap v13063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:22:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:03.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:03.101499+0000) 2022-09-08T04:22:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:03.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:03.087236+0000) 2022-09-08T04:22:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:04.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:04.101743+0000) 2022-09-08T04:22:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:04.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:04.087503+0000) 2022-09-08T04:22:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:04 smithi059 ceph-mon[97450]: pgmap v13064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:04 smithi055 ceph-mon[97033]: pgmap v13064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:05.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:05.101981+0000) 2022-09-08T04:22:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:05.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:05.087755+0000) 2022-09-08T04:22:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:22:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:22:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:06.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:06.102144+0000) 2022-09-08T04:22:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:06.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:06.087938+0000) 2022-09-08T04:22:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:06 smithi059 ceph-mon[97450]: pgmap v13065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:22:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:22:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:06 smithi055 ceph-mon[97033]: pgmap v13065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:22:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:22:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:07.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:07.102286+0000) 2022-09-08T04:22:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:07.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:07.088182+0000) 2022-09-08T04:22:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:08.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:08.102527+0000) 2022-09-08T04:22:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:08.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:08.088441+0000) 2022-09-08T04:22:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:08 smithi059 ceph-mon[97450]: pgmap v13066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:22:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:08 smithi055 ceph-mon[97033]: pgmap v13066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:22:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:09.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:09.102759+0000) 2022-09-08T04:22:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:09.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:09.088769+0000) 2022-09-08T04:22:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:09 smithi059 ceph-mon[97450]: osdmap e897: 12 total, 11 up, 11 in 2022-09-08T04:22:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:09 smithi055 ceph-mon[97033]: osdmap e897: 12 total, 11 up, 11 in 2022-09-08T04:22:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:10.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:10.102934+0000) 2022-09-08T04:22:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:10.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:10.089029+0000) 2022-09-08T04:22:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:10 smithi059 ceph-mon[97450]: pgmap v13068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:22:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:10 smithi055 ceph-mon[97033]: pgmap v13068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:22:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:11.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:11.103154+0000) 2022-09-08T04:22:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:11.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:11.089253+0000) 2022-09-08T04:22:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:12.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:12.103399+0000) 2022-09-08T04:22:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:12.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:12.089452+0000) 2022-09-08T04:22:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:12 smithi059 ceph-mon[97450]: pgmap v13069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:22:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:12 smithi055 ceph-mon[97033]: pgmap v13069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:22:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:13.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:13.103664+0000) 2022-09-08T04:22:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:13.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:13.089687+0000) 2022-09-08T04:22:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:14.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:14.103909+0000) 2022-09-08T04:22:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:14.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:14.089855+0000) 2022-09-08T04:22:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:14 smithi059 ceph-mon[97450]: pgmap v13070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:22:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:14 smithi059 ceph-mon[97450]: osdmap e898: 12 total, 11 up, 11 in 2022-09-08T04:22:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:14 smithi055 ceph-mon[97033]: pgmap v13070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:22:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:14 smithi055 ceph-mon[97033]: osdmap e898: 12 total, 11 up, 11 in 2022-09-08T04:22:15.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:22:15.321+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:22:15.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:15.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:15.104184+0000) 2022-09-08T04:22:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:15.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:15.090059+0000) 2022-09-08T04:22:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:16.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:16.104461+0000) 2022-09-08T04:22:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:16.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:16.090313+0000) 2022-09-08T04:22:17.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:16 smithi059 ceph-mon[97450]: pgmap v13072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:22:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:16 smithi055 ceph-mon[97033]: pgmap v13072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:22:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:17.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:17.104738+0000) 2022-09-08T04:22:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:17.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:17.090585+0000) 2022-09-08T04:22:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:18.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:18.104892+0000) 2022-09-08T04:22:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:18.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:18.090799+0000) 2022-09-08T04:22:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:18 smithi059 ceph-mon[97450]: pgmap v13073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 241 B/s wr, 7 op/s 2022-09-08T04:22:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:18 smithi055 ceph-mon[97033]: pgmap v13073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 241 B/s wr, 7 op/s 2022-09-08T04:22:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:19.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:19.105061+0000) 2022-09-08T04:22:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:19.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:19.091009+0000) 2022-09-08T04:22:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:20.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:20.105306+0000) 2022-09-08T04:22:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:20.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:20.091250+0000) 2022-09-08T04:22:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:20 smithi059 ceph-mon[97450]: pgmap v13074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:22:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:20 smithi055 ceph-mon[97033]: pgmap v13074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:22:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:21.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:21.105519+0000) 2022-09-08T04:22:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:21.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:21.091483+0000) 2022-09-08T04:22:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:22.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:22.105761+0000) 2022-09-08T04:22:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:22.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:22.091703+0000) 2022-09-08T04:22:23.048 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:22 smithi059 ceph-mon[97450]: pgmap v13075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:22:23.049 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:22:23.049 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:22:23.050 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:22:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:22 smithi055 ceph-mon[97033]: pgmap v13075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:22:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:22:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:22:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:22:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:23.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:23.105987+0000) 2022-09-08T04:22:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:23.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:23.091897+0000) 2022-09-08T04:22:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:22:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:23 smithi059 ceph-mon[97450]: pgmap v13076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:22:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:24.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:24.106219+0000) 2022-09-08T04:22:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:24.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:24.092085+0000) 2022-09-08T04:22:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:22:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:23 smithi055 ceph-mon[97033]: pgmap v13076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:22:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:25.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:25.106475+0000) 2022-09-08T04:22:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:25.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:25.092291+0000) 2022-09-08T04:22:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:26.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:26.106738+0000) 2022-09-08T04:22:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:26.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:26.092503+0000) 2022-09-08T04:22:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:26 smithi055 ceph-mon[97033]: pgmap v13077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-08T04:22:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:26 smithi059 ceph-mon[97450]: pgmap v13077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-08T04:22:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:27.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:27.106891+0000) 2022-09-08T04:22:27.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:27.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:27.092748+0000) 2022-09-08T04:22:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:28.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:28.107050+0000) 2022-09-08T04:22:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:28.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:28.092929+0000) 2022-09-08T04:22:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:28 smithi055 ceph-mon[97033]: pgmap v13078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:29.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:28 smithi059 ceph-mon[97450]: pgmap v13078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:29.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:29.107275+0000) 2022-09-08T04:22:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:29.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:29.093164+0000) 2022-09-08T04:22:30.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:22:30.334+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:22:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:30.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:30.107502+0000) 2022-09-08T04:22:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:30.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:30.093433+0000) 2022-09-08T04:22:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:30 smithi055 ceph-mon[97033]: pgmap v13079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:31.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:30 smithi059 ceph-mon[97450]: pgmap v13079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:31.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:31.107689+0000) 2022-09-08T04:22:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:31.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:31.093675+0000) 2022-09-08T04:22:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:32.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:32.107907+0000) 2022-09-08T04:22:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:32.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:32.093927+0000) 2022-09-08T04:22:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:32 smithi055 ceph-mon[97033]: pgmap v13080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:22:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:32 smithi059 ceph-mon[97450]: pgmap v13080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:22:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:33.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:33.108115+0000) 2022-09-08T04:22:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:33.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:33.094112+0000) 2022-09-08T04:22:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:34.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:34.108273+0000) 2022-09-08T04:22:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:34.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:34.094270+0000) 2022-09-08T04:22:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:34 smithi055 ceph-mon[97033]: pgmap v13081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:34 smithi059 ceph-mon[97450]: pgmap v13081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:35.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:35.108443+0000) 2022-09-08T04:22:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:35.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:35.094395+0000) 2022-09-08T04:22:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:36.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:36.108651+0000) 2022-09-08T04:22:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:36.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:36.094612+0000) 2022-09-08T04:22:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:36 smithi055 ceph-mon[97033]: pgmap v13082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:22:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:36 smithi059 ceph-mon[97450]: pgmap v13082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:22:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:37.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:37.108862+0000) 2022-09-08T04:22:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:37.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:37.094802+0000) 2022-09-08T04:22:38.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:38.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:38.095028+0000) 2022-09-08T04:22:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:38.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:38.109039+0000) 2022-09-08T04:22:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:38 smithi055 ceph-mon[97033]: pgmap v13083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:39.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:38 smithi059 ceph-mon[97450]: pgmap v13083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:39.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:39.109239+0000) 2022-09-08T04:22:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:39.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:39.095223+0000) 2022-09-08T04:22:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:40.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:40.109454+0000) 2022-09-08T04:22:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:40.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:40.095407+0000) 2022-09-08T04:22:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:40 smithi055 ceph-mon[97033]: pgmap v13084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:41.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:40 smithi059 ceph-mon[97450]: pgmap v13084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:41.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:41.109691+0000) 2022-09-08T04:22:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:41.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:41.095632+0000) 2022-09-08T04:22:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:42.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:42.109833+0000) 2022-09-08T04:22:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:42.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:42.095920+0000) 2022-09-08T04:22:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:42 smithi055 ceph-mon[97033]: pgmap v13085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:22:43.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:42 smithi059 ceph-mon[97450]: pgmap v13085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:22:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:43.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:43.110004+0000) 2022-09-08T04:22:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:43.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:43.096134+0000) 2022-09-08T04:22:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:44.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:44.110250+0000) 2022-09-08T04:22:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:44.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:44.096394+0000) 2022-09-08T04:22:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:44 smithi055 ceph-mon[97033]: pgmap v13086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:44 smithi059 ceph-mon[97450]: pgmap v13086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:45.349 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:45.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:45.110496+0000) 2022-09-08T04:22:45.349 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:45.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:45.096662+0000) 2022-09-08T04:22:45.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:22:45.348+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:22:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:46.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:46.110717+0000) 2022-09-08T04:22:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:46.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:46.096920+0000) 2022-09-08T04:22:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:46 smithi055 ceph-mon[97033]: pgmap v13087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:22:47.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:46 smithi059 ceph-mon[97450]: pgmap v13087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:22:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:47.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:47.110872+0000) 2022-09-08T04:22:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:47.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:47.097159+0000) 2022-09-08T04:22:48.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:48.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:48.097379+0000) 2022-09-08T04:22:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:48.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:48.111094+0000) 2022-09-08T04:22:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:48 smithi055 ceph-mon[97033]: pgmap v13088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:48 smithi059 ceph-mon[97450]: pgmap v13088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:49.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:49.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:49.097611+0000) 2022-09-08T04:22:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:49.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:49.111311+0000) 2022-09-08T04:22:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:50.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:50.111539+0000) 2022-09-08T04:22:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:50.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:50.097795+0000) 2022-09-08T04:22:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:50 smithi055 ceph-mon[97033]: pgmap v13089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:50 smithi059 ceph-mon[97450]: pgmap v13089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:51.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:51.111735+0000) 2022-09-08T04:22:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:51.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:51.097954+0000) 2022-09-08T04:22:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:52.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:52.111943+0000) 2022-09-08T04:22:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:52.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:52.098138+0000) 2022-09-08T04:22:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:52 smithi055 ceph-mon[97033]: pgmap v13090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:22:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:52 smithi059 ceph-mon[97450]: pgmap v13090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:22:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:53.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:53.112170+0000) 2022-09-08T04:22:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:53.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:53.098365+0000) 2022-09-08T04:22:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:54.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:54.112406+0000) 2022-09-08T04:22:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:54.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:54.098655+0000) 2022-09-08T04:22:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:54 smithi055 ceph-mon[97033]: pgmap v13091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:54 smithi059 ceph-mon[97450]: pgmap v13091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:55.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:55.112671+0000) 2022-09-08T04:22:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:55.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:55.098860+0000) 2022-09-08T04:22:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:56.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:56.112863+0000) 2022-09-08T04:22:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:56.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:56.099015+0000) 2022-09-08T04:22:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:56 smithi055 ceph-mon[97033]: pgmap v13092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:22:57.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:56 smithi059 ceph-mon[97450]: pgmap v13092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:22:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:57.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:57.113059+0000) 2022-09-08T04:22:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:57.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:57.099244+0000) 2022-09-08T04:22:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:58.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:58.113270+0000) 2022-09-08T04:22:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:58.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:58.099494+0000) 2022-09-08T04:22:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:22:58 smithi055 ceph-mon[97033]: pgmap v13093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:59.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:22:58 smithi059 ceph-mon[97450]: pgmap v13093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:22:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:22:59.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:59.113491+0000) 2022-09-08T04:22:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:22:59.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:22:59.099706+0000) 2022-09-08T04:23:00.364 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:00.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:00.113732+0000) 2022-09-08T04:23:00.365 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:00.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:00.099942+0000) 2022-09-08T04:23:00.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:23:00.362+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:23:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:00 smithi055 ceph-mon[97033]: pgmap v13094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:00 smithi059 ceph-mon[97450]: pgmap v13094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:01.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:01.113885+0000) 2022-09-08T04:23:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:01.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:01.100182+0000) 2022-09-08T04:23:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:02.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:02.114120+0000) 2022-09-08T04:23:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:02.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:02.100428+0000) 2022-09-08T04:23:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:02 smithi055 ceph-mon[97033]: pgmap v13095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:23:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:02 smithi059 ceph-mon[97450]: pgmap v13095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:23:03.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:03.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:03.100722+0000) 2022-09-08T04:23:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:03.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:03.114345+0000) 2022-09-08T04:23:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:04.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:04.114541+0000) 2022-09-08T04:23:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:04.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:04.100919+0000) 2022-09-08T04:23:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:04 smithi055 ceph-mon[97033]: pgmap v13096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:04 smithi059 ceph-mon[97450]: pgmap v13096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:05.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:05.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:05.101115+0000) 2022-09-08T04:23:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:05.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:05.114794+0000) 2022-09-08T04:23:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:23:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:23:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:06.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:06.114977+0000) 2022-09-08T04:23:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:06.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:06.101303+0000) 2022-09-08T04:23:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:06 smithi055 ceph-mon[97033]: pgmap v13097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:23:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:23:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:06 smithi059 ceph-mon[97450]: pgmap v13097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:23:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:23:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:07.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:07.115180+0000) 2022-09-08T04:23:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:07.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:07.101532+0000) 2022-09-08T04:23:08.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:08.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:08.101768+0000) 2022-09-08T04:23:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:08.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:08.115399+0000) 2022-09-08T04:23:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:08 smithi055 ceph-mon[97033]: pgmap v13098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:23:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:08 smithi059 ceph-mon[97450]: pgmap v13098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:23:09.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:09.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:09.102027+0000) 2022-09-08T04:23:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:09.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:09.115602+0000) 2022-09-08T04:23:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:10.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:10.115881+0000) 2022-09-08T04:23:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:10.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:10.102215+0000) 2022-09-08T04:23:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:10 smithi055 ceph-mon[97033]: pgmap v13099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:23:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:10 smithi055 ceph-mon[97033]: osdmap e899: 12 total, 11 up, 11 in 2022-09-08T04:23:11.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:10 smithi059 ceph-mon[97450]: pgmap v13099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:23:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:10 smithi059 ceph-mon[97450]: osdmap e899: 12 total, 11 up, 11 in 2022-09-08T04:23:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:11.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:11.116019+0000) 2022-09-08T04:23:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:11.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:11.102367+0000) 2022-09-08T04:23:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:12.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:12.116271+0000) 2022-09-08T04:23:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:12.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:12.102598+0000) 2022-09-08T04:23:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:12 smithi055 ceph-mon[97033]: pgmap v13101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:23:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:12 smithi059 ceph-mon[97450]: pgmap v13101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:23:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:13.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:13.116492+0000) 2022-09-08T04:23:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:13.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:13.102793+0000) 2022-09-08T04:23:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:14.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:14.116670+0000) 2022-09-08T04:23:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:14.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:14.102939+0000) 2022-09-08T04:23:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:14 smithi055 ceph-mon[97033]: pgmap v13102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:23:15.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:14 smithi059 ceph-mon[97450]: pgmap v13102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:23:15.376 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:15.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:15.116906+0000) 2022-09-08T04:23:15.377 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:15.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:15.103140+0000) 2022-09-08T04:23:15.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:23:15.375+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:23:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:16.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:16.117092+0000) 2022-09-08T04:23:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:16.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:16.103393+0000) 2022-09-08T04:23:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:16 smithi055 ceph-mon[97033]: pgmap v13103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:23:17.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:16 smithi059 ceph-mon[97450]: pgmap v13103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:23:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:17.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:17.117322+0000) 2022-09-08T04:23:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:17.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:17.103673+0000) 2022-09-08T04:23:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:18.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:18.117509+0000) 2022-09-08T04:23:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:18.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:18.103867+0000) 2022-09-08T04:23:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:18 smithi055 ceph-mon[97033]: pgmap v13104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:23:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:18 smithi059 ceph-mon[97450]: pgmap v13104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:23:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:19.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:19.117626+0000) 2022-09-08T04:23:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:19.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:19.104048+0000) 2022-09-08T04:23:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:20.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:20.117802+0000) 2022-09-08T04:23:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:20.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:20.104309+0000) 2022-09-08T04:23:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:20 smithi055 ceph-mon[97033]: pgmap v13105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:23:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:20 smithi055 ceph-mon[97033]: osdmap e900: 12 total, 11 up, 11 in 2022-09-08T04:23:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:20 smithi059 ceph-mon[97450]: pgmap v13105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:23:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:20 smithi059 ceph-mon[97450]: osdmap e900: 12 total, 11 up, 11 in 2022-09-08T04:23:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:21.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:21.117984+0000) 2022-09-08T04:23:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:21.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:21.104516+0000) 2022-09-08T04:23:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:22.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:22.118228+0000) 2022-09-08T04:23:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:22.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:22.104750+0000) 2022-09-08T04:23:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:22 smithi055 ceph-mon[97033]: pgmap v13107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:23:23.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:22 smithi059 ceph-mon[97450]: pgmap v13107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:23:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:23.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:23.118420+0000) 2022-09-08T04:23:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:23.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:23.104981+0000) 2022-09-08T04:23:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:23:23.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:23:23.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:23:23.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:23:24.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:23:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:23:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:23:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:23:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:24.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:24.118602+0000) 2022-09-08T04:23:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:24.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:24.105125+0000) 2022-09-08T04:23:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:24 smithi055 ceph-mon[97033]: pgmap v13108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:23:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:24 smithi059 ceph-mon[97450]: pgmap v13108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:23:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:25.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:25.118864+0000) 2022-09-08T04:23:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:25.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:25.105338+0000) 2022-09-08T04:23:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:26.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:26.119103+0000) 2022-09-08T04:23:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:26.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:26.105521+0000) 2022-09-08T04:23:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:26 smithi055 ceph-mon[97033]: pgmap v13109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:23:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:26 smithi059 ceph-mon[97450]: pgmap v13109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:23:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:27.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:27.119417+0000) 2022-09-08T04:23:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:27.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:27.105719+0000) 2022-09-08T04:23:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:28.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:28.119674+0000) 2022-09-08T04:23:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:28.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:28.105864+0000) 2022-09-08T04:23:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:28 smithi055 ceph-mon[97033]: pgmap v13110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:23:29.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:28 smithi059 ceph-mon[97450]: pgmap v13110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:23:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:29.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:29.119819+0000) 2022-09-08T04:23:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:29.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:29.106070+0000) 2022-09-08T04:23:30.387 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:30.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:30.120036+0000) 2022-09-08T04:23:30.388 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:30.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:30.106316+0000) 2022-09-08T04:23:30.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:23:30.386+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:23:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:30 smithi055 ceph-mon[97033]: pgmap v13111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:23:31.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:30 smithi059 ceph-mon[97450]: pgmap v13111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:23:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:31.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:31.120204+0000) 2022-09-08T04:23:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:31.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:31.106570+0000) 2022-09-08T04:23:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:32.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:32.120431+0000) 2022-09-08T04:23:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:32.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:32.106790+0000) 2022-09-08T04:23:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:32 smithi055 ceph-mon[97033]: pgmap v13112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:23:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:32 smithi059 ceph-mon[97450]: pgmap v13112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:23:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:33.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:33.120603+0000) 2022-09-08T04:23:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:33.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:33.106960+0000) 2022-09-08T04:23:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:34.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:34.120698+0000) 2022-09-08T04:23:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:34.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:34.107185+0000) 2022-09-08T04:23:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:34 smithi055 ceph-mon[97033]: pgmap v13113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:34 smithi059 ceph-mon[97450]: pgmap v13113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:35.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:35.120900+0000) 2022-09-08T04:23:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:35.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:35.107400+0000) 2022-09-08T04:23:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:36.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:36.121073+0000) 2022-09-08T04:23:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:36.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:36.107671+0000) 2022-09-08T04:23:36.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:36 smithi055 ceph-mon[97033]: pgmap v13114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:23:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:36 smithi059 ceph-mon[97450]: pgmap v13114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:23:37.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:37.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:37.107918+0000) 2022-09-08T04:23:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:37.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:37.121305+0000) 2022-09-08T04:23:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:38.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:38.121525+0000) 2022-09-08T04:23:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:38.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:38.108079+0000) 2022-09-08T04:23:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:38 smithi055 ceph-mon[97033]: pgmap v13115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:38 smithi059 ceph-mon[97450]: pgmap v13115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:39.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:39.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:39.108253+0000) 2022-09-08T04:23:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:39.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:39.121744+0000) 2022-09-08T04:23:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:40.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:40.121953+0000) 2022-09-08T04:23:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:40.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:40.108448+0000) 2022-09-08T04:23:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:40 smithi055 ceph-mon[97033]: pgmap v13116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:40 smithi059 ceph-mon[97450]: pgmap v13116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:41.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:41.122171+0000) 2022-09-08T04:23:41.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:41.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:41.108622+0000) 2022-09-08T04:23:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:42.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:42.122344+0000) 2022-09-08T04:23:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:42.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:42.108810+0000) 2022-09-08T04:23:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:42 smithi055 ceph-mon[97033]: pgmap v13117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:23:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:42 smithi059 ceph-mon[97450]: pgmap v13117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:23:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:43.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:43.122588+0000) 2022-09-08T04:23:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:43.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:43.109038+0000) 2022-09-08T04:23:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:44.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:44.122768+0000) 2022-09-08T04:23:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:44.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:44.109258+0000) 2022-09-08T04:23:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:44 smithi055 ceph-mon[97033]: pgmap v13118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:44 smithi059 ceph-mon[97450]: pgmap v13118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:45.391 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:45.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:45.122977+0000) 2022-09-08T04:23:45.392 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:45.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:45.109502+0000) 2022-09-08T04:23:45.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:23:45.389+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:23:46.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:46.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:46.109781+0000) 2022-09-08T04:23:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:46.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:46.123220+0000) 2022-09-08T04:23:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:46 smithi055 ceph-mon[97033]: pgmap v13119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:23:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:46 smithi059 ceph-mon[97450]: pgmap v13119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:23:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:47.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:47.123448+0000) 2022-09-08T04:23:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:47.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:47.109915+0000) 2022-09-08T04:23:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:48.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:48.123675+0000) 2022-09-08T04:23:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:48.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:48.110058+0000) 2022-09-08T04:23:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:48 smithi055 ceph-mon[97033]: pgmap v13120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:48 smithi059 ceph-mon[97450]: pgmap v13120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:49.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:49.123845+0000) 2022-09-08T04:23:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:49.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:49.110287+0000) 2022-09-08T04:23:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:50.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:50.124029+0000) 2022-09-08T04:23:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:50.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:50.110516+0000) 2022-09-08T04:23:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:50 smithi055 ceph-mon[97033]: pgmap v13121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:50 smithi059 ceph-mon[97450]: pgmap v13121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:51.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:51.124250+0000) 2022-09-08T04:23:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:51.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:51.110743+0000) 2022-09-08T04:23:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:52.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:52.124514+0000) 2022-09-08T04:23:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:52.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:52.111025+0000) 2022-09-08T04:23:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:52 smithi055 ceph-mon[97033]: pgmap v13122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:23:53.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:52 smithi059 ceph-mon[97450]: pgmap v13122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:23:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:53.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:53.124730+0000) 2022-09-08T04:23:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:53.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:53.111231+0000) 2022-09-08T04:23:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:54.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:54.124945+0000) 2022-09-08T04:23:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:54.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:54.111443+0000) 2022-09-08T04:23:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:54 smithi055 ceph-mon[97033]: pgmap v13123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:55.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:54 smithi059 ceph-mon[97450]: pgmap v13123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:55.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:55.125186+0000) 2022-09-08T04:23:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:55.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:55.111688+0000) 2022-09-08T04:23:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:56.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:56.125379+0000) 2022-09-08T04:23:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:56.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:56.111843+0000) 2022-09-08T04:23:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:56 smithi055 ceph-mon[97033]: pgmap v13124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:23:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:56 smithi059 ceph-mon[97450]: pgmap v13124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:23:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:57.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:57.125567+0000) 2022-09-08T04:23:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:57.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:57.112027+0000) 2022-09-08T04:23:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:58.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:58.125806+0000) 2022-09-08T04:23:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:58.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:58.112227+0000) 2022-09-08T04:23:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:23:58 smithi055 ceph-mon[97033]: pgmap v13125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:23:58 smithi059 ceph-mon[97450]: pgmap v13125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:23:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:23:59.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:59.126011+0000) 2022-09-08T04:23:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:23:59.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:23:59.112455+0000) 2022-09-08T04:24:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:00.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:00.126219+0000) 2022-09-08T04:24:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:00.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:00.112697+0000) 2022-09-08T04:24:00.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:24:00.405+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:24:00.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:00 smithi055 ceph-mon[97033]: pgmap v13126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:00 smithi059 ceph-mon[97450]: pgmap v13126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:01.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:01.112930+0000) 2022-09-08T04:24:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:01.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:01.126410+0000) 2022-09-08T04:24:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:02.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:02.126622+0000) 2022-09-08T04:24:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:02.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:02.113155+0000) 2022-09-08T04:24:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:02 smithi055 ceph-mon[97033]: pgmap v13127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:24:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:02 smithi059 ceph-mon[97450]: pgmap v13127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:24:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:03.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:03.126852+0000) 2022-09-08T04:24:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:03.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:03.113330+0000) 2022-09-08T04:24:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:04.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:04.127090+0000) 2022-09-08T04:24:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:04.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:04.113566+0000) 2022-09-08T04:24:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:04 smithi055 ceph-mon[97033]: pgmap v13128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:04 smithi059 ceph-mon[97450]: pgmap v13128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:05.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:05.127333+0000) 2022-09-08T04:24:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:05.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:05.113761+0000) 2022-09-08T04:24:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:24:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:24:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:06.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:06.127609+0000) 2022-09-08T04:24:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:06.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:06.113987+0000) 2022-09-08T04:24:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:06 smithi055 ceph-mon[97033]: pgmap v13129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:24:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:24:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:06 smithi059 ceph-mon[97450]: pgmap v13129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:24:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:24:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:07.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:07.127877+0000) 2022-09-08T04:24:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:07.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:07.114221+0000) 2022-09-08T04:24:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:08.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:08.128094+0000) 2022-09-08T04:24:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:08.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:08.114443+0000) 2022-09-08T04:24:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:08 smithi055 ceph-mon[97033]: pgmap v13130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:24:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:08 smithi059 ceph-mon[97450]: pgmap v13130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:24:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:09.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:09.128315+0000) 2022-09-08T04:24:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:09.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:09.114743+0000) 2022-09-08T04:24:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:10.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:10.128481+0000) 2022-09-08T04:24:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:10.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:10.114900+0000) 2022-09-08T04:24:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:10 smithi055 ceph-mon[97033]: pgmap v13131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:24:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:10 smithi055 ceph-mon[97033]: osdmap e901: 12 total, 11 up, 11 in 2022-09-08T04:24:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:10 smithi059 ceph-mon[97450]: pgmap v13131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:24:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:10 smithi059 ceph-mon[97450]: osdmap e901: 12 total, 11 up, 11 in 2022-09-08T04:24:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:11.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:11.128717+0000) 2022-09-08T04:24:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:11.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:11.115137+0000) 2022-09-08T04:24:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:12.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:12.128913+0000) 2022-09-08T04:24:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:12.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:12.115343+0000) 2022-09-08T04:24:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:12 smithi055 ceph-mon[97033]: pgmap v13133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:24:13.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:12 smithi059 ceph-mon[97450]: pgmap v13133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:24:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:13.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:13.129149+0000) 2022-09-08T04:24:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:13.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:13.115587+0000) 2022-09-08T04:24:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:14.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:14.129392+0000) 2022-09-08T04:24:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:14.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:14.115791+0000) 2022-09-08T04:24:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:14 smithi055 ceph-mon[97033]: pgmap v13134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:24:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:14 smithi059 ceph-mon[97450]: pgmap v13134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:24:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:15.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:15.129537+0000) 2022-09-08T04:24:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:15.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:15.116029+0000) 2022-09-08T04:24:15.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:24:15.421+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:24:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:16.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:16.129745+0000) 2022-09-08T04:24:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:16.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:16.116310+0000) 2022-09-08T04:24:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:16 smithi055 ceph-mon[97033]: pgmap v13135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:24:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:16 smithi059 ceph-mon[97450]: pgmap v13135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:24:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:17.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:17.129929+0000) 2022-09-08T04:24:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:17.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:17.116543+0000) 2022-09-08T04:24:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:18.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:18.130179+0000) 2022-09-08T04:24:18.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:18.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:18.116759+0000) 2022-09-08T04:24:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:18 smithi055 ceph-mon[97033]: pgmap v13136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:24:19.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:18 smithi059 ceph-mon[97450]: pgmap v13136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:24:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:19.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:19.130422+0000) 2022-09-08T04:24:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:19.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:19.116944+0000) 2022-09-08T04:24:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:20.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:20.130590+0000) 2022-09-08T04:24:20.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:20.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:20.117210+0000) 2022-09-08T04:24:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:20 smithi055 ceph-mon[97033]: pgmap v13137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:24:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:20 smithi055 ceph-mon[97033]: osdmap e902: 12 total, 11 up, 11 in 2022-09-08T04:24:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:20 smithi059 ceph-mon[97450]: pgmap v13137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:24:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:20 smithi059 ceph-mon[97450]: osdmap e902: 12 total, 11 up, 11 in 2022-09-08T04:24:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:21.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:21.130761+0000) 2022-09-08T04:24:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:21.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:21.117358+0000) 2022-09-08T04:24:22.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:22.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:22.117597+0000) 2022-09-08T04:24:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:22.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:22.130979+0000) 2022-09-08T04:24:22.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:22 smithi055 ceph-mon[97033]: pgmap v13139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:24:23.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:22 smithi059 ceph-mon[97450]: pgmap v13139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:24:23.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:23.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:23.117731+0000) 2022-09-08T04:24:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:23.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:23.131148+0000) 2022-09-08T04:24:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:24.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:24.131339+0000) 2022-09-08T04:24:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:24.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:24.117869+0000) 2022-09-08T04:24:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:24 smithi055 ceph-mon[97033]: pgmap v13140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:24:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:24:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:24:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:24:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:24:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:24 smithi059 ceph-mon[97450]: pgmap v13140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:24:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:24:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:24:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:24:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:24:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:25.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:25.131609+0000) 2022-09-08T04:24:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:25.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:25.118091+0000) 2022-09-08T04:24:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:26.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:26.131840+0000) 2022-09-08T04:24:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:26.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:26.118317+0000) 2022-09-08T04:24:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:26 smithi055 ceph-mon[97033]: pgmap v13141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:24:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:26 smithi059 ceph-mon[97450]: pgmap v13141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:24:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:27.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:27.132003+0000) 2022-09-08T04:24:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:27.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:27.118537+0000) 2022-09-08T04:24:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:28.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:28.132187+0000) 2022-09-08T04:24:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:28.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:28.118753+0000) 2022-09-08T04:24:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:28 smithi055 ceph-mon[97033]: pgmap v13142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:24:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:28 smithi059 ceph-mon[97450]: pgmap v13142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:24:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:29.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:29.132402+0000) 2022-09-08T04:24:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:29.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:29.119000+0000) 2022-09-08T04:24:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:30.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:30.132611+0000) 2022-09-08T04:24:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:30.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:30.119243+0000) 2022-09-08T04:24:30.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:24:30.428+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:24:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:30 smithi055 ceph-mon[97033]: pgmap v13143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:24:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:30 smithi059 ceph-mon[97450]: pgmap v13143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:24:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:31.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:31.132773+0000) 2022-09-08T04:24:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:31.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:31.119466+0000) 2022-09-08T04:24:32.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:32.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:32.119691+0000) 2022-09-08T04:24:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:32.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:32.132900+0000) 2022-09-08T04:24:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:32 smithi055 ceph-mon[97033]: pgmap v13144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:24:33.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:32 smithi059 ceph-mon[97450]: pgmap v13144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:24:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:33.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:33.133115+0000) 2022-09-08T04:24:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:33.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:33.119922+0000) 2022-09-08T04:24:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:34.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:34.133336+0000) 2022-09-08T04:24:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:34.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:34.120154+0000) 2022-09-08T04:24:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:34 smithi055 ceph-mon[97033]: pgmap v13145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:34 smithi059 ceph-mon[97450]: pgmap v13145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:35.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:35.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:35.120398+0000) 2022-09-08T04:24:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:35.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:35.133513+0000) 2022-09-08T04:24:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:36.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:36.133737+0000) 2022-09-08T04:24:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:36.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:36.120685+0000) 2022-09-08T04:24:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:36 smithi055 ceph-mon[97033]: pgmap v13146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:24:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:36 smithi059 ceph-mon[97450]: pgmap v13146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:24:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:37.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:37.133966+0000) 2022-09-08T04:24:37.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:37.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:37.120873+0000) 2022-09-08T04:24:38.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:38.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:38.121079+0000) 2022-09-08T04:24:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:38.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:38.134183+0000) 2022-09-08T04:24:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:38 smithi055 ceph-mon[97033]: pgmap v13147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:38 smithi059 ceph-mon[97450]: pgmap v13147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:39.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:39.134432+0000) 2022-09-08T04:24:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:39.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:39.121331+0000) 2022-09-08T04:24:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:40.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:40.134726+0000) 2022-09-08T04:24:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:40.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:40.121572+0000) 2022-09-08T04:24:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:40 smithi055 ceph-mon[97033]: pgmap v13148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:40 smithi059 ceph-mon[97450]: pgmap v13148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:41.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:41.134962+0000) 2022-09-08T04:24:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:41.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:41.121800+0000) 2022-09-08T04:24:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:42.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:42.135209+0000) 2022-09-08T04:24:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:42.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:42.122071+0000) 2022-09-08T04:24:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:42 smithi055 ceph-mon[97033]: pgmap v13149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:24:43.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:42 smithi059 ceph-mon[97450]: pgmap v13149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:24:43.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:43.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:43.122242+0000) 2022-09-08T04:24:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:43.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:43.135442+0000) 2022-09-08T04:24:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:44.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:44.135654+0000) 2022-09-08T04:24:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:44.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:44.122484+0000) 2022-09-08T04:24:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:44 smithi055 ceph-mon[97033]: pgmap v13150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:45.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:44 smithi059 ceph-mon[97450]: pgmap v13150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:45.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:45.135848+0000) 2022-09-08T04:24:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:45.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:45.122750+0000) 2022-09-08T04:24:45.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:24:45.438+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:24:46.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:46.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:46.123022+0000) 2022-09-08T04:24:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:46.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:46.135964+0000) 2022-09-08T04:24:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:46 smithi055 ceph-mon[97033]: pgmap v13151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:24:47.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:46 smithi059 ceph-mon[97450]: pgmap v13151: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:24:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:47.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:47.123234+0000) 2022-09-08T04:24:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:47.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:47.136197+0000) 2022-09-08T04:24:48.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:48.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:48.123472+0000) 2022-09-08T04:24:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:48.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:48.136408+0000) 2022-09-08T04:24:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:48 smithi055 ceph-mon[97033]: pgmap v13152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:49.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:48 smithi059 ceph-mon[97450]: pgmap v13152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:49.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:49.123729+0000) 2022-09-08T04:24:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:49.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:49.136680+0000) 2022-09-08T04:24:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:50.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:50.136896+0000) 2022-09-08T04:24:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:50.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:50.123976+0000) 2022-09-08T04:24:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:50 smithi055 ceph-mon[97033]: pgmap v13153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:50 smithi059 ceph-mon[97450]: pgmap v13153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:51.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:51.137104+0000) 2022-09-08T04:24:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:51.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:51.124177+0000) 2022-09-08T04:24:52.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:52.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:52.124345+0000) 2022-09-08T04:24:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:52.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:52.137278+0000) 2022-09-08T04:24:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:52 smithi055 ceph-mon[97033]: pgmap v13154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:24:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:52 smithi059 ceph-mon[97450]: pgmap v13154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:24:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:53.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:53.137468+0000) 2022-09-08T04:24:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:53.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:53.124531+0000) 2022-09-08T04:24:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:54.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:54.137718+0000) 2022-09-08T04:24:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:54.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:54.124776+0000) 2022-09-08T04:24:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:54 smithi055 ceph-mon[97033]: pgmap v13155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:54 smithi059 ceph-mon[97450]: pgmap v13155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:55.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:55.137913+0000) 2022-09-08T04:24:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:55.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:55.125038+0000) 2022-09-08T04:24:56.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:56.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:56.125237+0000) 2022-09-08T04:24:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:56.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:56.138075+0000) 2022-09-08T04:24:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:56 smithi055 ceph-mon[97033]: pgmap v13156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:24:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:56 smithi059 ceph-mon[97450]: pgmap v13156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:24:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:57.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:57.138265+0000) 2022-09-08T04:24:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:57.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:57.125499+0000) 2022-09-08T04:24:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:58.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:58.138461+0000) 2022-09-08T04:24:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:58.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:58.125795+0000) 2022-09-08T04:24:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:24:58 smithi055 ceph-mon[97033]: pgmap v13157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:59.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:24:58 smithi059 ceph-mon[97450]: pgmap v13157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:24:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:24:59.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:59.138668+0000) 2022-09-08T04:24:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:24:59.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:24:59.126070+0000) 2022-09-08T04:25:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:00.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:00.138916+0000) 2022-09-08T04:25:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:00.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:00.126336+0000) 2022-09-08T04:25:00.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:25:00.448+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:25:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:00 smithi055 ceph-mon[97033]: pgmap v13158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:00 smithi059 ceph-mon[97450]: pgmap v13158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:01.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:01.126490+0000) 2022-09-08T04:25:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:01.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:01.139060+0000) 2022-09-08T04:25:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:02.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:02.139228+0000) 2022-09-08T04:25:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:02.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:02.126756+0000) 2022-09-08T04:25:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:02 smithi055 ceph-mon[97033]: pgmap v13159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:25:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:02 smithi059 ceph-mon[97450]: pgmap v13159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:25:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:03.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:03.139483+0000) 2022-09-08T04:25:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:03.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:03.126945+0000) 2022-09-08T04:25:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:04.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:04.139651+0000) 2022-09-08T04:25:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:04.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:04.127157+0000) 2022-09-08T04:25:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:04 smithi055 ceph-mon[97033]: pgmap v13160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:05.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:04 smithi059 ceph-mon[97450]: pgmap v13160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:05.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:05.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:05.127325+0000) 2022-09-08T04:25:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:05.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:05.139837+0000) 2022-09-08T04:25:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:25:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:25:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:06.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:06.140008+0000) 2022-09-08T04:25:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:06.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:06.127534+0000) 2022-09-08T04:25:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:06 smithi055 ceph-mon[97033]: pgmap v13161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:25:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:25:07.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:06 smithi059 ceph-mon[97450]: pgmap v13161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:25:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:25:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:07.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:07.140273+0000) 2022-09-08T04:25:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:07.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:07.127787+0000) 2022-09-08T04:25:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:08.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:08.140521+0000) 2022-09-08T04:25:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:08.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:08.128020+0000) 2022-09-08T04:25:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:08 smithi055 ceph-mon[97033]: pgmap v13162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:25:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:08 smithi059 ceph-mon[97450]: pgmap v13162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:25:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:09.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:09.140783+0000) 2022-09-08T04:25:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:09.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:09.128314+0000) 2022-09-08T04:25:10.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:09 smithi059 ceph-mon[97450]: osdmap e903: 12 total, 11 up, 11 in 2022-09-08T04:25:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:09 smithi055 ceph-mon[97033]: osdmap e903: 12 total, 11 up, 11 in 2022-09-08T04:25:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:10.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:10.141026+0000) 2022-09-08T04:25:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:10.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:10.128533+0000) 2022-09-08T04:25:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:10 smithi059 ceph-mon[97450]: pgmap v13164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:25:11.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:10 smithi055 ceph-mon[97033]: pgmap v13164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:25:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:11.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:11.141304+0000) 2022-09-08T04:25:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:11.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:11.128780+0000) 2022-09-08T04:25:12.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:12.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:12.128997+0000) 2022-09-08T04:25:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:12.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:12.141468+0000) 2022-09-08T04:25:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:12 smithi059 ceph-mon[97450]: pgmap v13165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:25:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:12 smithi055 ceph-mon[97033]: pgmap v13165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:25:13.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:13.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:13.129207+0000) 2022-09-08T04:25:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:13.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:13.141690+0000) 2022-09-08T04:25:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:14.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:14.141937+0000) 2022-09-08T04:25:14.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:14.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:14.129418+0000) 2022-09-08T04:25:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:14 smithi059 ceph-mon[97450]: pgmap v13166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:25:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:14 smithi059 ceph-mon[97450]: osdmap e904: 12 total, 11 up, 11 in 2022-09-08T04:25:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:14 smithi055 ceph-mon[97033]: pgmap v13166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:25:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:14 smithi055 ceph-mon[97033]: osdmap e904: 12 total, 11 up, 11 in 2022-09-08T04:25:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:15.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:15.142140+0000) 2022-09-08T04:25:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:15.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:15.129680+0000) 2022-09-08T04:25:15.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:25:15.460+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:25:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:16.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:16.142369+0000) 2022-09-08T04:25:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:16.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:16.129857+0000) 2022-09-08T04:25:17.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:16 smithi059 ceph-mon[97450]: pgmap v13168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:25:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:16 smithi055 ceph-mon[97033]: pgmap v13168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:25:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:17.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:17.142603+0000) 2022-09-08T04:25:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:17.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:17.130093+0000) 2022-09-08T04:25:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:18.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:18.142821+0000) 2022-09-08T04:25:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:18.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:18.130315+0000) 2022-09-08T04:25:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:18 smithi059 ceph-mon[97450]: pgmap v13169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 238 B/s wr, 7 op/s 2022-09-08T04:25:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:18 smithi055 ceph-mon[97033]: pgmap v13169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 238 B/s wr, 7 op/s 2022-09-08T04:25:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:19.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:19.143023+0000) 2022-09-08T04:25:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:19.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:19.130564+0000) 2022-09-08T04:25:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:20.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:20.143258+0000) 2022-09-08T04:25:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:20.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:20.130778+0000) 2022-09-08T04:25:21.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:20 smithi059 ceph-mon[97450]: pgmap v13170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:25:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:20 smithi055 ceph-mon[97033]: pgmap v13170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:25:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:21.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:21.143431+0000) 2022-09-08T04:25:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:21.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:21.131010+0000) 2022-09-08T04:25:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:22.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:22.143668+0000) 2022-09-08T04:25:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:22.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:22.131185+0000) 2022-09-08T04:25:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:22 smithi059 ceph-mon[97450]: pgmap v13171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:25:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:22 smithi055 ceph-mon[97033]: pgmap v13171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:25:23.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:23.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:23.131408+0000) 2022-09-08T04:25:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:23.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:23.143891+0000) 2022-09-08T04:25:24.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:24.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:24.131673+0000) 2022-09-08T04:25:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:24.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:24.144148+0000) 2022-09-08T04:25:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:24 smithi055 ceph-mon[97033]: pgmap v13172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:25:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:25:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:25:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:25:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:25:24.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:25:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:24 smithi059 ceph-mon[97450]: pgmap v13172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:25:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:25:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:25:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:25:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:25:25.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:25:25.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:25.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:25.131891+0000) 2022-09-08T04:25:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:25.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:25.144369+0000) 2022-09-08T04:25:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:26.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:26.144577+0000) 2022-09-08T04:25:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:26.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:26.132129+0000) 2022-09-08T04:25:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:26 smithi059 ceph-mon[97450]: pgmap v13173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 95 B/s wr, 8 op/s 2022-09-08T04:25:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:26 smithi055 ceph-mon[97033]: pgmap v13173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 95 B/s wr, 8 op/s 2022-09-08T04:25:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:27.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:27.144691+0000) 2022-09-08T04:25:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:27.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:27.132334+0000) 2022-09-08T04:25:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:25:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:25:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: pgmap v13174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:25:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:25:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:25:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T04:25:28.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T04:25:28.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T04:25:28.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T04:25:28.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:25:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:25:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:25:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: pgmap v13174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:25:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:25:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:25:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T04:25:28.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T04:25:28.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:25:28.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T04:25:28.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T04:25:28.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:25:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:28.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:28.144869+0000) 2022-09-08T04:25:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:28.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:28.132549+0000) 2022-09-08T04:25:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:29.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:29.145098+0000) 2022-09-08T04:25:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:29.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:29.132726+0000) 2022-09-08T04:25:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:30.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:30.145276+0000) 2022-09-08T04:25:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:30.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:30.132928+0000) 2022-09-08T04:25:30.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:25:30.472+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:25:30.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:30 smithi055 ceph-mon[97033]: pgmap v13175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:25:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:30 smithi059 ceph-mon[97450]: pgmap v13175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:25:31.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:31.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:31.133082+0000) 2022-09-08T04:25:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:31.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:31.145484+0000) 2022-09-08T04:25:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:32.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:32.145678+0000) 2022-09-08T04:25:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:32.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:32.133279+0000) 2022-09-08T04:25:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:32 smithi055 ceph-mon[97033]: pgmap v13176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:25:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:32 smithi059 ceph-mon[97450]: pgmap v13176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:25:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:33.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:33.145817+0000) 2022-09-08T04:25:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:33.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:33.133521+0000) 2022-09-08T04:25:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:34.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:34.146031+0000) 2022-09-08T04:25:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:34.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:34.133744+0000) 2022-09-08T04:25:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:34 smithi055 ceph-mon[97033]: pgmap v13177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:35.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:34 smithi059 ceph-mon[97450]: pgmap v13177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:35.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:35.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:35.133869+0000) 2022-09-08T04:25:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:35.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:35.146269+0000) 2022-09-08T04:25:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:36.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:36.146504+0000) 2022-09-08T04:25:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:36.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:36.134103+0000) 2022-09-08T04:25:36.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:36 smithi055 ceph-mon[97033]: pgmap v13178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:25:37.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:36 smithi059 ceph-mon[97450]: pgmap v13178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:25:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:37.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:37.146735+0000) 2022-09-08T04:25:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:37.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:37.134353+0000) 2022-09-08T04:25:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:38.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:38.146941+0000) 2022-09-08T04:25:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:38.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:38.134530+0000) 2022-09-08T04:25:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:38 smithi055 ceph-mon[97033]: pgmap v13179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:38 smithi059 ceph-mon[97450]: pgmap v13179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:39.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:39.147164+0000) 2022-09-08T04:25:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:39.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:39.134757+0000) 2022-09-08T04:25:40.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:40.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:40.134967+0000) 2022-09-08T04:25:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:40.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:40.147308+0000) 2022-09-08T04:25:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:40 smithi055 ceph-mon[97033]: pgmap v13180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:41.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:40 smithi059 ceph-mon[97450]: pgmap v13180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:41.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:41.147551+0000) 2022-09-08T04:25:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:41.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:41.135191+0000) 2022-09-08T04:25:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:42.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:42.147774+0000) 2022-09-08T04:25:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:42.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:42.135400+0000) 2022-09-08T04:25:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:42 smithi055 ceph-mon[97033]: pgmap v13181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:25:43.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:42 smithi059 ceph-mon[97450]: pgmap v13181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:25:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:43.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:43.147944+0000) 2022-09-08T04:25:43.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:43.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:43.135655+0000) 2022-09-08T04:25:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:44.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:44.148156+0000) 2022-09-08T04:25:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:44.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:44.135824+0000) 2022-09-08T04:25:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:44 smithi055 ceph-mon[97033]: pgmap v13182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:44 smithi059 ceph-mon[97450]: pgmap v13182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:45.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:45.148350+0000) 2022-09-08T04:25:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:45.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:45.135989+0000) 2022-09-08T04:25:45.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:25:45.488+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:25:46.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:46.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:46.136243+0000) 2022-09-08T04:25:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:46.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:46.148588+0000) 2022-09-08T04:25:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:46 smithi055 ceph-mon[97033]: pgmap v13183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:25:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:46 smithi059 ceph-mon[97450]: pgmap v13183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:25:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:47.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:47.148775+0000) 2022-09-08T04:25:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:47.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:47.136474+0000) 2022-09-08T04:25:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:48.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:48.148980+0000) 2022-09-08T04:25:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:48.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:48.136763+0000) 2022-09-08T04:25:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:48 smithi055 ceph-mon[97033]: pgmap v13184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:48 smithi059 ceph-mon[97450]: pgmap v13184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:49.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:49.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:49.136992+0000) 2022-09-08T04:25:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:49.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:49.149212+0000) 2022-09-08T04:25:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:50.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:50.149418+0000) 2022-09-08T04:25:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:50.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:50.137254+0000) 2022-09-08T04:25:50.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:50 smithi055 ceph-mon[97033]: pgmap v13185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:50 smithi059 ceph-mon[97450]: pgmap v13185: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:51.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:51.149682+0000) 2022-09-08T04:25:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:51.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:51.137540+0000) 2022-09-08T04:25:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:52.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:52.149838+0000) 2022-09-08T04:25:52.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:52.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:52.137836+0000) 2022-09-08T04:25:52.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:52 smithi055 ceph-mon[97033]: pgmap v13186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:25:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:52 smithi059 ceph-mon[97450]: pgmap v13186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:25:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:53.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:53.150020+0000) 2022-09-08T04:25:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:53.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:53.138092+0000) 2022-09-08T04:25:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:54.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:54.150260+0000) 2022-09-08T04:25:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:54.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:54.138358+0000) 2022-09-08T04:25:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:54 smithi055 ceph-mon[97033]: pgmap v13187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:54 smithi059 ceph-mon[97450]: pgmap v13187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:55.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:55.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:55.138526+0000) 2022-09-08T04:25:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:55.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:55.150481+0000) 2022-09-08T04:25:56.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:56.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:56.138756+0000) 2022-09-08T04:25:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:56.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:56.150670+0000) 2022-09-08T04:25:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:56 smithi055 ceph-mon[97033]: pgmap v13188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:25:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:56 smithi059 ceph-mon[97450]: pgmap v13188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:25:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:57.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:57.150875+0000) 2022-09-08T04:25:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:57.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:57.138919+0000) 2022-09-08T04:25:58.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:58.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:58.139099+0000) 2022-09-08T04:25:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:58.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:58.151021+0000) 2022-09-08T04:25:58.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:25:58 smithi055 ceph-mon[97033]: pgmap v13189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:25:58 smithi059 ceph-mon[97450]: pgmap v13189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:25:59.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:25:59.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:59.139306+0000) 2022-09-08T04:25:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:25:59.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:25:59.151266+0000) 2022-09-08T04:26:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:00.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:00.151451+0000) 2022-09-08T04:26:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:00.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:00.139573+0000) 2022-09-08T04:26:00.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:26:00.503+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:26:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:00 smithi055 ceph-mon[97033]: pgmap v13190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:00 smithi059 ceph-mon[97450]: pgmap v13190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:01.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:01.151627+0000) 2022-09-08T04:26:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:01.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:01.139697+0000) 2022-09-08T04:26:02.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:02.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:02.139912+0000) 2022-09-08T04:26:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:02.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:02.151787+0000) 2022-09-08T04:26:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:02 smithi055 ceph-mon[97033]: pgmap v13191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:26:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:02 smithi059 ceph-mon[97450]: pgmap v13191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:26:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:03.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:03.152023+0000) 2022-09-08T04:26:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:03.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:03.140131+0000) 2022-09-08T04:26:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:04.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:04.152265+0000) 2022-09-08T04:26:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:04.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:04.140362+0000) 2022-09-08T04:26:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:04 smithi055 ceph-mon[97033]: pgmap v13192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:04 smithi059 ceph-mon[97450]: pgmap v13192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:05.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:05.152492+0000) 2022-09-08T04:26:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:05.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:05.140551+0000) 2022-09-08T04:26:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:26:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:26:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:06.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:06.152761+0000) 2022-09-08T04:26:06.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:06.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:06.140744+0000) 2022-09-08T04:26:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:06 smithi055 ceph-mon[97033]: pgmap v13193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:26:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:26:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:06 smithi059 ceph-mon[97450]: pgmap v13193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:26:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:26:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:07.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:07.152984+0000) 2022-09-08T04:26:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:07.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:07.140920+0000) 2022-09-08T04:26:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:08.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:08.153183+0000) 2022-09-08T04:26:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:08.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:08.141126+0000) 2022-09-08T04:26:08.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:08 smithi055 ceph-mon[97033]: pgmap v13194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:26:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:08 smithi059 ceph-mon[97450]: pgmap v13194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:26:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:09.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:09.153422+0000) 2022-09-08T04:26:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:09.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:09.141352+0000) 2022-09-08T04:26:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:10.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:10.153661+0000) 2022-09-08T04:26:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:10.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:10.141531+0000) 2022-09-08T04:26:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:10 smithi055 ceph-mon[97033]: pgmap v13195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:26:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:10 smithi055 ceph-mon[97033]: osdmap e905: 12 total, 11 up, 11 in 2022-09-08T04:26:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:10 smithi059 ceph-mon[97450]: pgmap v13195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:26:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:10 smithi059 ceph-mon[97450]: osdmap e905: 12 total, 11 up, 11 in 2022-09-08T04:26:11.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:11.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:11.141684+0000) 2022-09-08T04:26:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:11.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:11.153867+0000) 2022-09-08T04:26:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:12.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:12.154004+0000) 2022-09-08T04:26:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:12.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:12.141866+0000) 2022-09-08T04:26:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:12 smithi055 ceph-mon[97033]: pgmap v13197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:26:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:12 smithi059 ceph-mon[97450]: pgmap v13197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:26:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:13.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:13.154198+0000) 2022-09-08T04:26:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:13.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:13.142077+0000) 2022-09-08T04:26:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:14.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:14.154383+0000) 2022-09-08T04:26:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:14.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:14.142294+0000) 2022-09-08T04:26:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:14 smithi055 ceph-mon[97033]: pgmap v13198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:26:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:14 smithi059 ceph-mon[97450]: pgmap v13198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:26:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:15.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:15.154522+0000) 2022-09-08T04:26:15.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:15.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:15.142527+0000) 2022-09-08T04:26:15.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:26:15.513+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:26:16.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:16.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:16.142769+0000) 2022-09-08T04:26:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:16.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:16.154753+0000) 2022-09-08T04:26:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:16 smithi055 ceph-mon[97033]: pgmap v13199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:26:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:16 smithi059 ceph-mon[97450]: pgmap v13199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:26:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:17.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:17.154951+0000) 2022-09-08T04:26:17.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:17.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:17.143003+0000) 2022-09-08T04:26:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:18.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:18.155127+0000) 2022-09-08T04:26:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:18.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:18.143251+0000) 2022-09-08T04:26:18.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:18 smithi055 ceph-mon[97033]: pgmap v13200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:26:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:18 smithi059 ceph-mon[97450]: pgmap v13200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:26:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:19.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:19.155292+0000) 2022-09-08T04:26:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:19.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:19.143477+0000) 2022-09-08T04:26:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:20.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:20.155500+0000) 2022-09-08T04:26:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:20.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:20.143711+0000) 2022-09-08T04:26:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:20 smithi055 ceph-mon[97033]: pgmap v13201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:26:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:20 smithi055 ceph-mon[97033]: osdmap e906: 12 total, 11 up, 11 in 2022-09-08T04:26:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:20 smithi059 ceph-mon[97450]: pgmap v13201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:26:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:20 smithi059 ceph-mon[97450]: osdmap e906: 12 total, 11 up, 11 in 2022-09-08T04:26:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:21.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:21.155750+0000) 2022-09-08T04:26:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:21.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:21.143923+0000) 2022-09-08T04:26:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:22.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:22.155968+0000) 2022-09-08T04:26:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:22.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:22.144165+0000) 2022-09-08T04:26:22.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:22 smithi055 ceph-mon[97033]: pgmap v13203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:26:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:22 smithi059 ceph-mon[97450]: pgmap v13203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:26:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:23.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:23.156165+0000) 2022-09-08T04:26:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:23.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:23.144413+0000) 2022-09-08T04:26:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:24.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:24.156414+0000) 2022-09-08T04:26:24.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:24.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:24.144656+0000) 2022-09-08T04:26:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:24 smithi055 ceph-mon[97033]: pgmap v13204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:26:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:24 smithi059 ceph-mon[97450]: pgmap v13204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:26:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:25.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:25.156622+0000) 2022-09-08T04:26:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:25.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:25.144799+0000) 2022-09-08T04:26:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:26.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:26.156811+0000) 2022-09-08T04:26:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:26.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:26.145071+0000) 2022-09-08T04:26:26.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:26 smithi055 ceph-mon[97033]: pgmap v13205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:26:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:26 smithi059 ceph-mon[97450]: pgmap v13205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:26:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:27.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:27.156992+0000) 2022-09-08T04:26:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:27.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:27.145323+0000) 2022-09-08T04:26:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:28.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:28.157160+0000) 2022-09-08T04:26:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:28.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:28.145589+0000) 2022-09-08T04:26:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:28 smithi055 ceph-mon[97033]: pgmap v13206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:26:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:26:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:26:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:26:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:26:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:28 smithi059 ceph-mon[97450]: pgmap v13206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:26:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:26:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:26:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:26:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:26:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:29.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:29.157405+0000) 2022-09-08T04:26:29.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:29.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:29.145848+0000) 2022-09-08T04:26:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:30.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:30.157602+0000) 2022-09-08T04:26:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:30.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:30.146092+0000) 2022-09-08T04:26:30.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:26:30.527+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:26:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:30 smithi055 ceph-mon[97033]: pgmap v13207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:26:31.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:30 smithi059 ceph-mon[97450]: pgmap v13207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:26:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:31.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:31.157807+0000) 2022-09-08T04:26:31.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:31.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:31.146273+0000) 2022-09-08T04:26:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:32.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:32.157983+0000) 2022-09-08T04:26:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:32.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:32.146418+0000) 2022-09-08T04:26:32.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:32 smithi055 ceph-mon[97033]: pgmap v13208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:26:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:32 smithi059 ceph-mon[97450]: pgmap v13208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:26:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:33.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:33.158157+0000) 2022-09-08T04:26:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:33.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:33.146706+0000) 2022-09-08T04:26:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:34.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:34.158360+0000) 2022-09-08T04:26:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:34.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:34.146872+0000) 2022-09-08T04:26:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:34 smithi055 ceph-mon[97033]: pgmap v13209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:34 smithi059 ceph-mon[97450]: pgmap v13209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:35.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:35.158591+0000) 2022-09-08T04:26:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:35.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:35.147112+0000) 2022-09-08T04:26:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:36.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:36.158832+0000) 2022-09-08T04:26:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:36.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:36.147287+0000) 2022-09-08T04:26:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:36 smithi055 ceph-mon[97033]: pgmap v13210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:26:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:36 smithi059 ceph-mon[97450]: pgmap v13210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:26:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:37.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:37.159000+0000) 2022-09-08T04:26:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:37.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:37.147517+0000) 2022-09-08T04:26:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:38.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:38.159202+0000) 2022-09-08T04:26:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:38.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:38.147701+0000) 2022-09-08T04:26:38.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:38 smithi055 ceph-mon[97033]: pgmap v13211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:38 smithi059 ceph-mon[97450]: pgmap v13211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:39.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:39.159349+0000) 2022-09-08T04:26:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:39.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:39.147879+0000) 2022-09-08T04:26:40.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:40.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:40.148076+0000) 2022-09-08T04:26:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:40.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:40.159542+0000) 2022-09-08T04:26:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:40 smithi055 ceph-mon[97033]: pgmap v13212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:40 smithi059 ceph-mon[97450]: pgmap v13212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:41.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:41.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:41.148260+0000) 2022-09-08T04:26:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:41.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:41.159741+0000) 2022-09-08T04:26:42.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:42.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:42.148425+0000) 2022-09-08T04:26:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:42.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:42.159896+0000) 2022-09-08T04:26:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:42 smithi055 ceph-mon[97033]: pgmap v13213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:26:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:42 smithi059 ceph-mon[97450]: pgmap v13213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:26:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:43.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:43.160098+0000) 2022-09-08T04:26:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:43.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:43.148690+0000) 2022-09-08T04:26:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:44.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:44.160216+0000) 2022-09-08T04:26:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:44.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:44.148829+0000) 2022-09-08T04:26:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:44 smithi055 ceph-mon[97033]: pgmap v13214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:44 smithi059 ceph-mon[97450]: pgmap v13214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:45.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:45.160374+0000) 2022-09-08T04:26:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:45.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:45.149005+0000) 2022-09-08T04:26:45.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:26:45.541+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:26:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:46.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:46.160579+0000) 2022-09-08T04:26:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:46.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:46.149238+0000) 2022-09-08T04:26:46.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:46 smithi055 ceph-mon[97033]: pgmap v13215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:26:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:46 smithi059 ceph-mon[97450]: pgmap v13215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:26:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:47.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:47.160775+0000) 2022-09-08T04:26:47.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:47.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:47.149505+0000) 2022-09-08T04:26:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:48.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:48.160965+0000) 2022-09-08T04:26:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:48.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:48.149752+0000) 2022-09-08T04:26:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:48 smithi055 ceph-mon[97033]: pgmap v13216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:48 smithi059 ceph-mon[97450]: pgmap v13216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:49.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:49.161119+0000) 2022-09-08T04:26:49.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:49.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:49.149981+0000) 2022-09-08T04:26:50.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:50.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:50.150211+0000) 2022-09-08T04:26:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:50.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:50.161332+0000) 2022-09-08T04:26:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:50 smithi055 ceph-mon[97033]: pgmap v13217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:50 smithi059 ceph-mon[97450]: pgmap v13217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:51.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:51.161549+0000) 2022-09-08T04:26:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:51.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:51.150396+0000) 2022-09-08T04:26:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:52.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:52.161720+0000) 2022-09-08T04:26:52.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:52.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:52.150611+0000) 2022-09-08T04:26:52.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:52 smithi055 ceph-mon[97033]: pgmap v13218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:26:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:52 smithi059 ceph-mon[97450]: pgmap v13218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:26:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:53.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:53.161891+0000) 2022-09-08T04:26:53.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:53.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:53.150815+0000) 2022-09-08T04:26:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:54.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:54.162096+0000) 2022-09-08T04:26:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:54.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:54.150988+0000) 2022-09-08T04:26:54.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:54 smithi055 ceph-mon[97033]: pgmap v13219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:54 smithi059 ceph-mon[97450]: pgmap v13219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:55.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:55.162325+0000) 2022-09-08T04:26:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:55.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:55.151199+0000) 2022-09-08T04:26:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:56.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:56.162504+0000) 2022-09-08T04:26:56.670 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:56.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:56.151435+0000) 2022-09-08T04:26:56.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:56 smithi055 ceph-mon[97033]: pgmap v13220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:26:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:56 smithi059 ceph-mon[97450]: pgmap v13220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:26:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:57.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:57.162735+0000) 2022-09-08T04:26:57.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:57.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:57.151710+0000) 2022-09-08T04:26:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:58.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:58.162944+0000) 2022-09-08T04:26:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:58.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:58.151906+0000) 2022-09-08T04:26:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:26:58 smithi055 ceph-mon[97033]: pgmap v13221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:26:58 smithi059 ceph-mon[97450]: pgmap v13221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:26:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:26:59.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:59.163148+0000) 2022-09-08T04:26:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:26:59.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:26:59.152107+0000) 2022-09-08T04:27:00.556 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:00.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:00.163379+0000) 2022-09-08T04:27:00.557 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:00.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:00.152328+0000) 2022-09-08T04:27:00.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:27:00.555+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:27:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:00 smithi055 ceph-mon[97033]: pgmap v13222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:01.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:00 smithi059 ceph-mon[97450]: pgmap v13222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:01.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:01.152573+0000) 2022-09-08T04:27:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:01.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:01.163567+0000) 2022-09-08T04:27:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:02.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:02.152800+0000) 2022-09-08T04:27:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:02.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:02.163819+0000) 2022-09-08T04:27:03.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:02 smithi059 ceph-mon[97450]: pgmap v13223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:27:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:02 smithi055 ceph-mon[97033]: pgmap v13223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:27:03.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:03.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:03.152991+0000) 2022-09-08T04:27:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:03.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:03.164098+0000) 2022-09-08T04:27:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:04.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:04.164344+0000) 2022-09-08T04:27:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:04.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:04.153201+0000) 2022-09-08T04:27:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:04 smithi059 ceph-mon[97450]: pgmap v13224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:04 smithi055 ceph-mon[97033]: pgmap v13224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:05.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:05.164536+0000) 2022-09-08T04:27:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:05.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:05.153455+0000) 2022-09-08T04:27:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:27:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:27:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:06.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:06.153730+0000) 2022-09-08T04:27:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:06.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:06.164746+0000) 2022-09-08T04:27:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:06 smithi059 ceph-mon[97450]: pgmap v13225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:27:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:27:07.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:06 smithi055 ceph-mon[97033]: pgmap v13225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:27:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:27:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:07.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:07.164987+0000) 2022-09-08T04:27:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:07.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:07.153891+0000) 2022-09-08T04:27:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:08.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:08.154084+0000) 2022-09-08T04:27:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:08.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:08.165231+0000) 2022-09-08T04:27:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:08 smithi059 ceph-mon[97450]: pgmap v13226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:27:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:08 smithi055 ceph-mon[97033]: pgmap v13226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:27:09.662 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:09.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:09.154262+0000) 2022-09-08T04:27:09.662 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:09.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:09.165444+0000) 2022-09-08T04:27:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:09 smithi059 ceph-mon[97450]: osdmap e907: 12 total, 11 up, 11 in 2022-09-08T04:27:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:09 smithi055 ceph-mon[97033]: osdmap e907: 12 total, 11 up, 11 in 2022-09-08T04:27:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:10.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:10.165668+0000) 2022-09-08T04:27:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:10.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:10.154516+0000) 2022-09-08T04:27:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:10 smithi059 ceph-mon[97450]: pgmap v13228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:27:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:10 smithi055 ceph-mon[97033]: pgmap v13228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:27:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:11.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:11.165901+0000) 2022-09-08T04:27:11.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:11.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:11.154771+0000) 2022-09-08T04:27:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:12.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:12.166115+0000) 2022-09-08T04:27:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:12.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:12.154946+0000) 2022-09-08T04:27:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:12 smithi059 ceph-mon[97450]: pgmap v13229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:27:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:12 smithi055 ceph-mon[97033]: pgmap v13229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:27:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:13.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:13.166254+0000) 2022-09-08T04:27:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:13.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:13.155095+0000) 2022-09-08T04:27:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:14.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:14.166454+0000) 2022-09-08T04:27:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:14.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:14.155331+0000) 2022-09-08T04:27:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:14 smithi059 ceph-mon[97450]: pgmap v13230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:27:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:14 smithi059 ceph-mon[97450]: osdmap e908: 12 total, 11 up, 11 in 2022-09-08T04:27:15.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:14 smithi055 ceph-mon[97033]: pgmap v13230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:27:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:14 smithi055 ceph-mon[97033]: osdmap e908: 12 total, 11 up, 11 in 2022-09-08T04:27:15.572 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:15.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:15.166730+0000) 2022-09-08T04:27:15.573 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:15.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:15.155620+0000) 2022-09-08T04:27:15.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:27:15.572+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:27:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:16.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:16.166897+0000) 2022-09-08T04:27:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:16.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:16.155966+0000) 2022-09-08T04:27:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:16 smithi059 ceph-mon[97450]: pgmap v13232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:27:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:16 smithi055 ceph-mon[97033]: pgmap v13232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:27:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:17.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:17.167077+0000) 2022-09-08T04:27:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:17.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:17.156156+0000) 2022-09-08T04:27:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:18.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:18.167244+0000) 2022-09-08T04:27:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:18.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:18.156364+0000) 2022-09-08T04:27:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:18 smithi059 ceph-mon[97450]: pgmap v13233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 238 B/s wr, 7 op/s 2022-09-08T04:27:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:18 smithi055 ceph-mon[97033]: pgmap v13233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 238 B/s wr, 7 op/s 2022-09-08T04:27:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:19.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:19.167474+0000) 2022-09-08T04:27:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:19.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:19.156609+0000) 2022-09-08T04:27:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:20.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:20.167732+0000) 2022-09-08T04:27:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:20.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:20.156799+0000) 2022-09-08T04:27:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:20 smithi059 ceph-mon[97450]: pgmap v13234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:27:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:20 smithi055 ceph-mon[97033]: pgmap v13234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:27:21.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:21.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:21.157038+0000) 2022-09-08T04:27:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:21.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:21.167967+0000) 2022-09-08T04:27:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:22.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:22.168215+0000) 2022-09-08T04:27:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:22.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:22.157267+0000) 2022-09-08T04:27:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:22 smithi059 ceph-mon[97450]: pgmap v13235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:27:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:22 smithi055 ceph-mon[97033]: pgmap v13235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:27:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:23.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:23.168431+0000) 2022-09-08T04:27:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:23.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:23.157517+0000) 2022-09-08T04:27:24.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:24.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:24.157765+0000) 2022-09-08T04:27:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:24.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:24.168664+0000) 2022-09-08T04:27:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:24 smithi055 ceph-mon[97033]: pgmap v13236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:27:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:24 smithi059 ceph-mon[97450]: pgmap v13236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:27:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:25.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:25.168851+0000) 2022-09-08T04:27:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:25.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:25.158028+0000) 2022-09-08T04:27:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:26.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:26.168975+0000) 2022-09-08T04:27:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:26.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:26.158319+0000) 2022-09-08T04:27:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:26 smithi055 ceph-mon[97033]: pgmap v13237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 95 B/s wr, 8 op/s 2022-09-08T04:27:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:26 smithi059 ceph-mon[97450]: pgmap v13237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 95 B/s wr, 8 op/s 2022-09-08T04:27:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:27.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:27.169126+0000) 2022-09-08T04:27:27.681 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:27.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:27.158577+0000) 2022-09-08T04:27:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:28.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:28.169336+0000) 2022-09-08T04:27:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:28.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:28.158784+0000) 2022-09-08T04:27:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:28 smithi055 ceph-mon[97033]: pgmap v13238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:27:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:27:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:27:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:28 smithi059 ceph-mon[97450]: pgmap v13238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:27:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:27:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:27:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:29.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:29.169559+0000) 2022-09-08T04:27:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:29.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:29.158994+0000) 2022-09-08T04:27:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:27:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:27:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:30 smithi055 ceph-mon[97033]: pgmap v13239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:27:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:27:30.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:30 smithi059 ceph-mon[97450]: pgmap v13239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:30.587 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:30.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:30.169832+0000) 2022-09-08T04:27:30.588 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:30.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:30.159271+0000) 2022-09-08T04:27:30.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:27:30.587+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:27:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:31.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:31.170066+0000) 2022-09-08T04:27:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:31.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:31.159573+0000) 2022-09-08T04:27:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:32.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:32.170274+0000) 2022-09-08T04:27:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:32.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:32.159781+0000) 2022-09-08T04:27:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:32 smithi055 ceph-mon[97033]: pgmap v13240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:27:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:32 smithi059 ceph-mon[97450]: pgmap v13240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:27:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:33.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:33.170513+0000) 2022-09-08T04:27:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:33.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:33.160053+0000) 2022-09-08T04:27:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:34.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:34.170739+0000) 2022-09-08T04:27:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:34.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:34.160333+0000) 2022-09-08T04:27:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:34 smithi055 ceph-mon[97033]: pgmap v13241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:34 smithi059 ceph-mon[97450]: pgmap v13241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:35.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:35.170978+0000) 2022-09-08T04:27:35.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:35.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:35.160580+0000) 2022-09-08T04:27:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:36.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:36.171229+0000) 2022-09-08T04:27:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:36.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:36.160802+0000) 2022-09-08T04:27:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:36 smithi055 ceph-mon[97033]: pgmap v13242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:27:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:36 smithi059 ceph-mon[97450]: pgmap v13242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:27:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:37.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:37.171464+0000) 2022-09-08T04:27:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:37.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:37.161002+0000) 2022-09-08T04:27:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:38.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:38.171707+0000) 2022-09-08T04:27:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:38.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:38.161270+0000) 2022-09-08T04:27:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:38 smithi055 ceph-mon[97033]: pgmap v13243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:38 smithi059 ceph-mon[97450]: pgmap v13243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:39.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:39.171936+0000) 2022-09-08T04:27:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:39.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:39.161527+0000) 2022-09-08T04:27:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:40.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:40.172181+0000) 2022-09-08T04:27:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:40.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:40.161816+0000) 2022-09-08T04:27:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:40 smithi055 ceph-mon[97033]: pgmap v13244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:40 smithi059 ceph-mon[97450]: pgmap v13244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:41.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:41.172403+0000) 2022-09-08T04:27:41.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:41.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:41.162074+0000) 2022-09-08T04:27:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:42.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:42.172552+0000) 2022-09-08T04:27:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:42.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:42.162267+0000) 2022-09-08T04:27:42.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:42 smithi055 ceph-mon[97033]: pgmap v13245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:27:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:42 smithi059 ceph-mon[97450]: pgmap v13245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:27:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:43.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:43.172798+0000) 2022-09-08T04:27:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:43.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:43.162517+0000) 2022-09-08T04:27:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:44.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:44.173034+0000) 2022-09-08T04:27:44.680 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:44.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:44.162772+0000) 2022-09-08T04:27:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:44 smithi055 ceph-mon[97033]: pgmap v13246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:44 smithi059 ceph-mon[97450]: pgmap v13246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:45.594 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:45.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:45.173263+0000) 2022-09-08T04:27:45.595 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:45.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:45.162952+0000) 2022-09-08T04:27:45.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:27:45.593+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:27:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:46.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:46.173480+0000) 2022-09-08T04:27:46.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:46.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:46.163182+0000) 2022-09-08T04:27:46.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:46 smithi055 ceph-mon[97033]: pgmap v13247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:27:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:46 smithi059 ceph-mon[97450]: pgmap v13247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:27:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:47.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:47.173738+0000) 2022-09-08T04:27:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:47.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:47.163434+0000) 2022-09-08T04:27:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:48.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:48.173910+0000) 2022-09-08T04:27:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:48.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:48.163633+0000) 2022-09-08T04:27:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:48 smithi055 ceph-mon[97033]: pgmap v13248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:48 smithi059 ceph-mon[97450]: pgmap v13248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:49.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:49.174159+0000) 2022-09-08T04:27:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:49.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:49.163753+0000) 2022-09-08T04:27:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:50.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:50.174406+0000) 2022-09-08T04:27:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:50.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:50.163976+0000) 2022-09-08T04:27:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:50 smithi055 ceph-mon[97033]: pgmap v13249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:50 smithi059 ceph-mon[97450]: pgmap v13249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:51.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:51.174548+0000) 2022-09-08T04:27:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:51.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:51.164150+0000) 2022-09-08T04:27:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:52.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:52.174799+0000) 2022-09-08T04:27:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:52.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:52.164372+0000) 2022-09-08T04:27:52.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:52 smithi055 ceph-mon[97033]: pgmap v13250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:27:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:52 smithi059 ceph-mon[97450]: pgmap v13250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:27:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:53.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:53.175029+0000) 2022-09-08T04:27:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:53.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:53.164614+0000) 2022-09-08T04:27:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:54.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:54.175169+0000) 2022-09-08T04:27:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:54.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:54.164808+0000) 2022-09-08T04:27:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:54 smithi055 ceph-mon[97033]: pgmap v13251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:54 smithi059 ceph-mon[97450]: pgmap v13251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:55.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:55.175391+0000) 2022-09-08T04:27:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:55.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:55.164947+0000) 2022-09-08T04:27:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:56.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:56.175627+0000) 2022-09-08T04:27:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:56.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:56.165160+0000) 2022-09-08T04:27:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:56 smithi055 ceph-mon[97033]: pgmap v13252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:27:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:56 smithi059 ceph-mon[97450]: pgmap v13252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:27:57.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:57.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:57.165411+0000) 2022-09-08T04:27:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:57.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:57.175922+0000) 2022-09-08T04:27:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:58.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:58.176158+0000) 2022-09-08T04:27:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:58.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:58.165629+0000) 2022-09-08T04:27:58.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:58 smithi055 ceph-mon[97033]: pgmap v13253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:58 smithi059 ceph-mon[97450]: pgmap v13253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:27:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:27:59.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:59.176363+0000) 2022-09-08T04:27:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:27:59.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:27:59.165865+0000) 2022-09-08T04:28:00.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:59 smithi055 ceph-mon[97033]: pgmap v13254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T04:28:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:28:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:27:59 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T04:28:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:59 smithi059 ceph-mon[97450]: pgmap v13254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T04:28:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:28:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:27:59 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T04:28:00.595 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:00.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:00.176630+0000) 2022-09-08T04:28:00.596 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:00.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:00.166001+0000) 2022-09-08T04:28:00.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:28:00.594+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:28:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:01.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:01.176892+0000) 2022-09-08T04:28:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:01.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:01.166211+0000) 2022-09-08T04:28:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:01 smithi055 ceph-mon[97033]: pgmap v13255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:28:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:01 smithi059 ceph-mon[97450]: pgmap v13255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T04:28:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:02.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:02.177051+0000) 2022-09-08T04:28:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:02.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:02.166420+0000) 2022-09-08T04:28:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:03.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:03.177268+0000) 2022-09-08T04:28:03.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:03.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:03.166681+0000) 2022-09-08T04:28:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:03 smithi055 ceph-mon[97033]: pgmap v13256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T04:28:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:03 smithi059 ceph-mon[97450]: pgmap v13256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T04:28:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:04.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:04.177462+0000) 2022-09-08T04:28:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:04.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:04.166918+0000) 2022-09-08T04:28:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:05.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:05.177718+0000) 2022-09-08T04:28:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:05.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:05.167145+0000) 2022-09-08T04:28:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:05 smithi055 ceph-mon[97033]: pgmap v13257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 46 KiB/s wr, 23 op/s 2022-09-08T04:28:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:28:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:05 smithi059 ceph-mon[97450]: pgmap v13257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 46 KiB/s wr, 23 op/s 2022-09-08T04:28:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:28:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:06.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:06.177951+0000) 2022-09-08T04:28:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:06.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:06.167319+0000) 2022-09-08T04:28:06.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:28:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:28:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:07.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:07.178151+0000) 2022-09-08T04:28:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:07.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:07.167545+0000) 2022-09-08T04:28:07.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:07 smithi055 ceph-mon[97033]: pgmap v13258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 46 KiB/s wr, 20 op/s 2022-09-08T04:28:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:07 smithi059 ceph-mon[97450]: pgmap v13258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 46 KiB/s wr, 20 op/s 2022-09-08T04:28:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:08.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:08.178402+0000) 2022-09-08T04:28:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:08.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:08.167805+0000) 2022-09-08T04:28:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:09.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:09.178609+0000) 2022-09-08T04:28:09.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:09.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:09.168018+0000) 2022-09-08T04:28:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:09 smithi059 ceph-mon[97450]: osdmap e909: 12 total, 11 up, 11 in 2022-09-08T04:28:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:09 smithi059 ceph-mon[97450]: pgmap v13260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:28:10.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:09 smithi055 ceph-mon[97033]: osdmap e909: 12 total, 11 up, 11 in 2022-09-08T04:28:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:09 smithi055 ceph-mon[97033]: pgmap v13260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:28:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:10.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:10.178889+0000) 2022-09-08T04:28:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:10.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:10.168230+0000) 2022-09-08T04:28:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:11.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:11.179141+0000) 2022-09-08T04:28:11.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:11.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:11.168445+0000) 2022-09-08T04:28:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:11 smithi055 ceph-mon[97033]: pgmap v13261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:28:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:11 smithi059 ceph-mon[97450]: pgmap v13261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:28:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:12.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:12.168719+0000) 2022-09-08T04:28:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:12.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:12.179355+0000) 2022-09-08T04:28:13.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:13.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:13.168922+0000) 2022-09-08T04:28:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:13.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:13.179585+0000) 2022-09-08T04:28:14.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:13 smithi055 ceph-mon[97033]: pgmap v13262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:28:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:13 smithi059 ceph-mon[97450]: pgmap v13262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:28:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:14.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:14.179745+0000) 2022-09-08T04:28:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:14.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:14.169153+0000) 2022-09-08T04:28:15.608 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:15.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:15.179982+0000) 2022-09-08T04:28:15.609 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:15.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:15.169365+0000) 2022-09-08T04:28:15.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:28:15.607+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:28:15.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:15 smithi055 ceph-mon[97033]: osdmap e910: 12 total, 11 up, 11 in 2022-09-08T04:28:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:15 smithi055 ceph-mon[97033]: pgmap v13264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:28:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:15 smithi059 ceph-mon[97450]: osdmap e910: 12 total, 11 up, 11 in 2022-09-08T04:28:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:15 smithi059 ceph-mon[97450]: pgmap v13264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:28:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:16.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:16.180295+0000) 2022-09-08T04:28:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:16.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:16.169600+0000) 2022-09-08T04:28:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:17.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:17.180539+0000) 2022-09-08T04:28:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:17.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:17.169849+0000) 2022-09-08T04:28:18.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:17 smithi055 ceph-mon[97033]: pgmap v13265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-08T04:28:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:17 smithi059 ceph-mon[97450]: pgmap v13265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-08T04:28:18.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:18.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:18.180755+0000) 2022-09-08T04:28:18.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:18.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:18.170072+0000) 2022-09-08T04:28:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:19.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:19.180946+0000) 2022-09-08T04:28:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:19.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:19.170337+0000) 2022-09-08T04:28:20.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:19 smithi055 ceph-mon[97033]: pgmap v13266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:28:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:19 smithi059 ceph-mon[97450]: pgmap v13266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:28:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:20.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:20.181156+0000) 2022-09-08T04:28:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:20.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:20.170554+0000) 2022-09-08T04:28:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:21.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:21.181415+0000) 2022-09-08T04:28:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:21.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:21.170793+0000) 2022-09-08T04:28:22.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:21 smithi055 ceph-mon[97033]: pgmap v13267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:28:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:21 smithi059 ceph-mon[97450]: pgmap v13267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:28:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:22.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:22.181700+0000) 2022-09-08T04:28:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:22.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:22.171045+0000) 2022-09-08T04:28:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:23.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:23.181917+0000) 2022-09-08T04:28:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:23.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:23.171312+0000) 2022-09-08T04:28:24.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:23 smithi055 ceph-mon[97033]: pgmap v13268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:28:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:23 smithi059 ceph-mon[97450]: pgmap v13268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:28:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:24.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:24.182090+0000) 2022-09-08T04:28:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:24.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:24.171543+0000) 2022-09-08T04:28:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:25.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:25.182333+0000) 2022-09-08T04:28:25.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:25.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:25.171780+0000) 2022-09-08T04:28:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:25 smithi055 ceph-mon[97033]: pgmap v13269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.5 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-08T04:28:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:25 smithi059 ceph-mon[97450]: pgmap v13269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.5 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-08T04:28:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:26.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:26.182568+0000) 2022-09-08T04:28:26.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:26.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:26.172008+0000) 2022-09-08T04:28:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:27.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:27.182695+0000) 2022-09-08T04:28:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:27.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:27.172264+0000) 2022-09-08T04:28:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:27 smithi055 ceph-mon[97033]: pgmap v13270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:27 smithi059 ceph-mon[97450]: pgmap v13270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:28.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:28.182934+0000) 2022-09-08T04:28:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:28.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:28.172467+0000) 2022-09-08T04:28:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:29.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:29.183126+0000) 2022-09-08T04:28:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:29.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:29.172681+0000) 2022-09-08T04:28:30.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:29 smithi055 ceph-mon[97033]: pgmap v13271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:29 smithi059 ceph-mon[97450]: pgmap v13271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:30.622 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:30.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:30.183342+0000) 2022-09-08T04:28:30.624 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:30.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:30.172890+0000) 2022-09-08T04:28:30.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:28:30.622+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:28:31.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:31.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:31.173125+0000) 2022-09-08T04:28:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:31.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:31.183610+0000) 2022-09-08T04:28:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:31 smithi055 ceph-mon[97033]: pgmap v13272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:28:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:31 smithi059 ceph-mon[97450]: pgmap v13272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:28:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:32.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:32.183827+0000) 2022-09-08T04:28:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:32.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:32.173296+0000) 2022-09-08T04:28:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:33.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:33.184085+0000) 2022-09-08T04:28:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:33.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:33.173533+0000) 2022-09-08T04:28:34.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:33 smithi055 ceph-mon[97033]: pgmap v13273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:33 smithi059 ceph-mon[97450]: pgmap v13273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:34.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:34.184302+0000) 2022-09-08T04:28:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:34.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:34.173725+0000) 2022-09-08T04:28:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:35.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:35.184555+0000) 2022-09-08T04:28:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:35.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:35.173963+0000) 2022-09-08T04:28:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:35 smithi055 ceph-mon[97033]: pgmap v13274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:28:36.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:35 smithi059 ceph-mon[97450]: pgmap v13274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:28:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:36.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:36.184789+0000) 2022-09-08T04:28:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:36.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:36.174204+0000) 2022-09-08T04:28:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:37.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:37.185032+0000) 2022-09-08T04:28:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:37.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:37.174449+0000) 2022-09-08T04:28:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:37 smithi055 ceph-mon[97033]: pgmap v13275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:37 smithi059 ceph-mon[97450]: pgmap v13275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:38.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:38.185178+0000) 2022-09-08T04:28:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:38.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:38.174721+0000) 2022-09-08T04:28:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:39.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:39.185300+0000) 2022-09-08T04:28:39.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:39.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:39.174954+0000) 2022-09-08T04:28:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:39 smithi055 ceph-mon[97033]: pgmap v13276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:39 smithi059 ceph-mon[97450]: pgmap v13276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:40.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:40.185496+0000) 2022-09-08T04:28:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:40.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:40.175198+0000) 2022-09-08T04:28:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:41.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:41.185734+0000) 2022-09-08T04:28:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:41.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:41.175402+0000) 2022-09-08T04:28:42.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:41 smithi055 ceph-mon[97033]: pgmap v13277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:28:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:41 smithi059 ceph-mon[97450]: pgmap v13277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:28:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:42.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:42.185955+0000) 2022-09-08T04:28:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:42.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:42.175659+0000) 2022-09-08T04:28:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:43.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:43.186209+0000) 2022-09-08T04:28:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:43.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:43.175923+0000) 2022-09-08T04:28:44.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:43 smithi055 ceph-mon[97033]: pgmap v13278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:43 smithi059 ceph-mon[97450]: pgmap v13278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:44.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:44.186361+0000) 2022-09-08T04:28:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:44.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:44.176058+0000) 2022-09-08T04:28:45.637 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:45.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:45.186574+0000) 2022-09-08T04:28:45.638 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:45.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:45.176252+0000) 2022-09-08T04:28:45.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:28:45.635+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:28:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:45 smithi055 ceph-mon[97033]: pgmap v13279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:28:46.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:45 smithi059 ceph-mon[97450]: pgmap v13279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:28:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:46.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:46.186837+0000) 2022-09-08T04:28:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:46.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:46.176516+0000) 2022-09-08T04:28:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:47.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:47.187124+0000) 2022-09-08T04:28:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:47.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:47.176692+0000) 2022-09-08T04:28:48.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:47 smithi055 ceph-mon[97033]: pgmap v13280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:47 smithi059 ceph-mon[97450]: pgmap v13280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:48.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:48.187349+0000) 2022-09-08T04:28:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:48.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:48.176846+0000) 2022-09-08T04:28:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:49.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:49.187544+0000) 2022-09-08T04:28:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:49.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:49.177040+0000) 2022-09-08T04:28:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:49 smithi055 ceph-mon[97033]: pgmap v13281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:49 smithi059 ceph-mon[97450]: pgmap v13281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:50.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:50.187750+0000) 2022-09-08T04:28:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:50.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:50.177275+0000) 2022-09-08T04:28:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:51.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:51.187961+0000) 2022-09-08T04:28:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:51.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:51.177497+0000) 2022-09-08T04:28:52.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:51 smithi055 ceph-mon[97033]: pgmap v13282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:28:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:51 smithi059 ceph-mon[97450]: pgmap v13282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:28:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:52.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:52.188131+0000) 2022-09-08T04:28:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:52.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:52.177755+0000) 2022-09-08T04:28:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:53.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:53.188388+0000) 2022-09-08T04:28:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:53.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:53.177916+0000) 2022-09-08T04:28:54.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:53 smithi055 ceph-mon[97033]: pgmap v13283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:53 smithi059 ceph-mon[97450]: pgmap v13283: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:54.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:54.188612+0000) 2022-09-08T04:28:54.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:54.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:54.178156+0000) 2022-09-08T04:28:55.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:55.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:55.178385+0000) 2022-09-08T04:28:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:55.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:55.188780+0000) 2022-09-08T04:28:55.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:55 smithi055 ceph-mon[97033]: pgmap v13284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:28:56.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:55 smithi059 ceph-mon[97450]: pgmap v13284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:28:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:56.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:56.189023+0000) 2022-09-08T04:28:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:56.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:56.178627+0000) 2022-09-08T04:28:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:57.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:57.189212+0000) 2022-09-08T04:28:57.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:57.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:57.178856+0000) 2022-09-08T04:28:58.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:57 smithi055 ceph-mon[97033]: pgmap v13285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:57 smithi059 ceph-mon[97450]: pgmap v13285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:28:58.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:58.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:58.179093+0000) 2022-09-08T04:28:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:58.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:58.189479+0000) 2022-09-08T04:28:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:28:59.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:59.189715+0000) 2022-09-08T04:28:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:28:59.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:28:59.179265+0000) 2022-09-08T04:29:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:59 smithi055 ceph-mon[97033]: pgmap v13286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:29:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:29:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:29:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:28:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:29:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:59 smithi059 ceph-mon[97450]: pgmap v13286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:29:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:29:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:29:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:28:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:29:00.647 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:00.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:00.189897+0000) 2022-09-08T04:29:00.648 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:00.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:00.179478+0000) 2022-09-08T04:29:00.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:29:00.647+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:29:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:01.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:01.179607+0000) 2022-09-08T04:29:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:01.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:01.190023+0000) 2022-09-08T04:29:02.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:01 smithi055 ceph-mon[97033]: pgmap v13287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:29:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:01 smithi059 ceph-mon[97450]: pgmap v13287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:29:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:02.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:02.190253+0000) 2022-09-08T04:29:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:02.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:02.179783+0000) 2022-09-08T04:29:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:03.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:03.190551+0000) 2022-09-08T04:29:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:03.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:03.180026+0000) 2022-09-08T04:29:04.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:03 smithi055 ceph-mon[97033]: pgmap v13288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:29:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:03 smithi059 ceph-mon[97450]: pgmap v13288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:29:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:04.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:04.190821+0000) 2022-09-08T04:29:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:04.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:04.180273+0000) 2022-09-08T04:29:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:05.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:05.191053+0000) 2022-09-08T04:29:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:05.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:05.180478+0000) 2022-09-08T04:29:05.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:05 smithi055 ceph-mon[97033]: pgmap v13289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:29:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:29:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:05 smithi059 ceph-mon[97450]: pgmap v13289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:29:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:29:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:06.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:06.180703+0000) 2022-09-08T04:29:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:06.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:06.191218+0000) 2022-09-08T04:29:06.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:29:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:29:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:07.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:07.191356+0000) 2022-09-08T04:29:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:07.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:07.180912+0000) 2022-09-08T04:29:07.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:07 smithi055 ceph-mon[97033]: pgmap v13290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:29:08.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:07 smithi059 ceph-mon[97450]: pgmap v13290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:29:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:08.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:08.191654+0000) 2022-09-08T04:29:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:08.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:08.181144+0000) 2022-09-08T04:29:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:09.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:09.191814+0000) 2022-09-08T04:29:09.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:09.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:09.181287+0000) 2022-09-08T04:29:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:09 smithi055 ceph-mon[97033]: osdmap e911: 12 total, 11 up, 11 in 2022-09-08T04:29:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:09 smithi055 ceph-mon[97033]: pgmap v13292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:29:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:09 smithi059 ceph-mon[97450]: osdmap e911: 12 total, 11 up, 11 in 2022-09-08T04:29:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:09 smithi059 ceph-mon[97450]: pgmap v13292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:29:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:10.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:10.192036+0000) 2022-09-08T04:29:10.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:10.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:10.181476+0000) 2022-09-08T04:29:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:11.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:11.192303+0000) 2022-09-08T04:29:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:11.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:11.181727+0000) 2022-09-08T04:29:12.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:11 smithi055 ceph-mon[97033]: pgmap v13293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:29:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:11 smithi059 ceph-mon[97450]: pgmap v13293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:29:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:12.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:12.192577+0000) 2022-09-08T04:29:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:12.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:12.181932+0000) 2022-09-08T04:29:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:13.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:13.192847+0000) 2022-09-08T04:29:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:13.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:13.182140+0000) 2022-09-08T04:29:14.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:13 smithi055 ceph-mon[97033]: pgmap v13294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:29:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:13 smithi059 ceph-mon[97450]: pgmap v13294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:29:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:14.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:14.192971+0000) 2022-09-08T04:29:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:14.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:14.182258+0000) 2022-09-08T04:29:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:15.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:15.193203+0000) 2022-09-08T04:29:15.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:15.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:15.182471+0000) 2022-09-08T04:29:15.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:15 smithi055 ceph-mon[97033]: osdmap e912: 12 total, 11 up, 11 in 2022-09-08T04:29:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:15 smithi055 ceph-mon[97033]: pgmap v13296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:29:16.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:15 smithi059 ceph-mon[97450]: osdmap e912: 12 total, 11 up, 11 in 2022-09-08T04:29:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:15 smithi059 ceph-mon[97450]: pgmap v13296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:29:16.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:29:15.655+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:29:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:16.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:16.193447+0000) 2022-09-08T04:29:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:16.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:16.182730+0000) 2022-09-08T04:29:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:17.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:17.182962+0000) 2022-09-08T04:29:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:17.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:17.193706+0000) 2022-09-08T04:29:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:17 smithi055 ceph-mon[97033]: pgmap v13297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T04:29:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:17 smithi059 ceph-mon[97450]: pgmap v13297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T04:29:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:18.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:18.193895+0000) 2022-09-08T04:29:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:18.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:18.183243+0000) 2022-09-08T04:29:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:19.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:19.194139+0000) 2022-09-08T04:29:19.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:19.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:19.183442+0000) 2022-09-08T04:29:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:19 smithi055 ceph-mon[97033]: pgmap v13298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:29:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:19 smithi059 ceph-mon[97450]: pgmap v13298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:29:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:20.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:20.183723+0000) 2022-09-08T04:29:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:20.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:20.194360+0000) 2022-09-08T04:29:21.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:21.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:21.183940+0000) 2022-09-08T04:29:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:21.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:21.194587+0000) 2022-09-08T04:29:22.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:21 smithi055 ceph-mon[97033]: pgmap v13299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T04:29:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:21 smithi059 ceph-mon[97450]: pgmap v13299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T04:29:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:22.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:22.194820+0000) 2022-09-08T04:29:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:22.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:22.184162+0000) 2022-09-08T04:29:23.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:23.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:23.184388+0000) 2022-09-08T04:29:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:23.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:23.195024+0000) 2022-09-08T04:29:24.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:23 smithi055 ceph-mon[97033]: pgmap v13300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T04:29:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:23 smithi059 ceph-mon[97450]: pgmap v13300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T04:29:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:24.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:24.195239+0000) 2022-09-08T04:29:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:24.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:24.184806+0000) 2022-09-08T04:29:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:25.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:25.195444+0000) 2022-09-08T04:29:25.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:25.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:25.185045+0000) 2022-09-08T04:29:25.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:25 smithi055 ceph-mon[97033]: pgmap v13301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 100 B/s wr, 8 op/s 2022-09-08T04:29:26.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:25 smithi059 ceph-mon[97450]: pgmap v13301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 100 B/s wr, 8 op/s 2022-09-08T04:29:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:26.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:26.195693+0000) 2022-09-08T04:29:26.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:26.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:26.185266+0000) 2022-09-08T04:29:27.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:27.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:27.185416+0000) 2022-09-08T04:29:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:27.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:27.195924+0000) 2022-09-08T04:29:28.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:27 smithi055 ceph-mon[97033]: pgmap v13302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:29:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:27 smithi059 ceph-mon[97450]: pgmap v13302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:29:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:28.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:28.196133+0000) 2022-09-08T04:29:28.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:28.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:28.185630+0000) 2022-09-08T04:29:29.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:29.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:29.185869+0000) 2022-09-08T04:29:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:29.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:29.196350+0000) 2022-09-08T04:29:30.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:29 smithi055 ceph-mon[97033]: pgmap v13303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:29:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:29:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:29 smithi059 ceph-mon[97450]: pgmap v13303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:29:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:29:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:30.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:30.196596+0000) 2022-09-08T04:29:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:30.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:30.186059+0000) 2022-09-08T04:29:31.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:29:30.665+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:29:31.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:30 smithi055 ceph-mon[97033]: pgmap v13304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T04:29:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:30 smithi055 ceph-mon[97033]: pgmap v13305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:29:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:30 smithi059 ceph-mon[97450]: pgmap v13304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T04:29:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:30 smithi059 ceph-mon[97450]: pgmap v13305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:29:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:31.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:31.196871+0000) 2022-09-08T04:29:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:31.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:31.186310+0000) 2022-09-08T04:29:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:32.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:32.186561+0000) 2022-09-08T04:29:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:32.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:32.197020+0000) 2022-09-08T04:29:33.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:32 smithi055 ceph-mon[97033]: pgmap v13306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:29:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:32 smithi059 ceph-mon[97450]: pgmap v13306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:29:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:33.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:33.186806+0000) 2022-09-08T04:29:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:33.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:33.197214+0000) 2022-09-08T04:29:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:34.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:34.186937+0000) 2022-09-08T04:29:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:34.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:34.197393+0000) 2022-09-08T04:29:35.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:34 smithi055 ceph-mon[97033]: pgmap v13307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:29:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:34 smithi059 ceph-mon[97450]: pgmap v13307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:29:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:35.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:35.197613+0000) 2022-09-08T04:29:35.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:35.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:35.187148+0000) 2022-09-08T04:29:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:36.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:36.197699+0000) 2022-09-08T04:29:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:36.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:36.187355+0000) 2022-09-08T04:29:37.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:36 smithi055 ceph-mon[97033]: pgmap v13308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:29:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:36 smithi059 ceph-mon[97450]: pgmap v13308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:29:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:37.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:37.197867+0000) 2022-09-08T04:29:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:37.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:37.187582+0000) 2022-09-08T04:29:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:38.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:38.198107+0000) 2022-09-08T04:29:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:38.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:38.187813+0000) 2022-09-08T04:29:39.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:38 smithi055 ceph-mon[97033]: pgmap v13309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:29:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:38 smithi059 ceph-mon[97450]: pgmap v13309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:29:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:39.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:39.198392+0000) 2022-09-08T04:29:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:39.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:39.188040+0000) 2022-09-08T04:29:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:40.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:40.198603+0000) 2022-09-08T04:29:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:40.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:40.188240+0000) 2022-09-08T04:29:41.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:40 smithi055 ceph-mon[97033]: pgmap v13310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:29:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:40 smithi059 ceph-mon[97450]: pgmap v13310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:29:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:41.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:41.198867+0000) 2022-09-08T04:29:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:41.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:41.188468+0000) 2022-09-08T04:29:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:42.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:42.199029+0000) 2022-09-08T04:29:42.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:42.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:42.188744+0000) 2022-09-08T04:29:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:42 smithi055 ceph-mon[97033]: pgmap v13311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:29:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:42 smithi059 ceph-mon[97450]: pgmap v13311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:29:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:43.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:43.199265+0000) 2022-09-08T04:29:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:43.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:43.188932+0000) 2022-09-08T04:29:44.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:44.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:44.189108+0000) 2022-09-08T04:29:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:44.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:44.199485+0000) 2022-09-08T04:29:45.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:44 smithi055 ceph-mon[97033]: pgmap v13312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:29:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:44 smithi059 ceph-mon[97450]: pgmap v13312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:29:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:45.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:45.199752+0000) 2022-09-08T04:29:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:45.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:45.189351+0000) 2022-09-08T04:29:46.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:29:45.680+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:29:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:46.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:46.199996+0000) 2022-09-08T04:29:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:46.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:46.189591+0000) 2022-09-08T04:29:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:46 smithi055 ceph-mon[97033]: pgmap v13313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:29:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:46 smithi059 ceph-mon[97450]: pgmap v13313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:29:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:47.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:47.200242+0000) 2022-09-08T04:29:47.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:47.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:47.189828+0000) 2022-09-08T04:29:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:48.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:48.200452+0000) 2022-09-08T04:29:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:48.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:48.190085+0000) 2022-09-08T04:29:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:48 smithi055 ceph-mon[97033]: pgmap v13314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:29:49.295 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:48 smithi059 ceph-mon[97450]: pgmap v13314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:29:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:49.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:49.200777+0000) 2022-09-08T04:29:49.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:49.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:49.190329+0000) 2022-09-08T04:29:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:50.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:50.200957+0000) 2022-09-08T04:29:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:50.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:50.190580+0000) 2022-09-08T04:29:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:50 smithi055 ceph-mon[97033]: pgmap v13315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:29:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:50 smithi059 ceph-mon[97450]: pgmap v13315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:29:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:51.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:51.201175+0000) 2022-09-08T04:29:51.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:51.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:51.190795+0000) 2022-09-08T04:29:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:52.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:52.201401+0000) 2022-09-08T04:29:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:52.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:52.190922+0000) 2022-09-08T04:29:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:52 smithi055 ceph-mon[97033]: pgmap v13316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:29:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:52 smithi059 ceph-mon[97450]: pgmap v13316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:29:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:53.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:53.201684+0000) 2022-09-08T04:29:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:53.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:53.191171+0000) 2022-09-08T04:29:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:54.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:54.201878+0000) 2022-09-08T04:29:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:54.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:54.191400+0000) 2022-09-08T04:29:55.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:54 smithi055 ceph-mon[97033]: pgmap v13317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:29:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:54 smithi059 ceph-mon[97450]: pgmap v13317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:29:55.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:55.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:55.191624+0000) 2022-09-08T04:29:55.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:55.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:55.202039+0000) 2022-09-08T04:29:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:56.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:56.202255+0000) 2022-09-08T04:29:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:56.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:56.191841+0000) 2022-09-08T04:29:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:56 smithi055 ceph-mon[97033]: pgmap v13318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:29:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:56 smithi059 ceph-mon[97450]: pgmap v13318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:29:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:57.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:57.202534+0000) 2022-09-08T04:29:57.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:57.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:57.192032+0000) 2022-09-08T04:29:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:58.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:58.202775+0000) 2022-09-08T04:29:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:58.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:58.192265+0000) 2022-09-08T04:29:59.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:29:58 smithi055 ceph-mon[97033]: pgmap v13319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:29:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:29:58 smithi059 ceph-mon[97450]: pgmap v13319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:29:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:29:59.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:59.202937+0000) 2022-09-08T04:29:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:29:59.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:29:59.192524+0000) 2022-09-08T04:30:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:00.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:00.203132+0000) 2022-09-08T04:30:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:00.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:00.192769+0000) 2022-09-08T04:30:01.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:30:00.695+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:30:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:00 smithi055 ceph-mon[97033]: pgmap v13320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T04:30:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T04:30:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T04:30:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T04:30:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T04:30:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:00 smithi059 ceph-mon[97450]: pgmap v13320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T04:30:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T04:30:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T04:30:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T04:30:01.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T04:30:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:01.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:01.203299+0000) 2022-09-08T04:30:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:01.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:01.192906+0000) 2022-09-08T04:30:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:02.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:02.203512+0000) 2022-09-08T04:30:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:02.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:02.193041+0000) 2022-09-08T04:30:03.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:02 smithi055 ceph-mon[97033]: pgmap v13321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:30:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:02 smithi059 ceph-mon[97450]: pgmap v13321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:30:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:03.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:03.203773+0000) 2022-09-08T04:30:03.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:03.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:03.193291+0000) 2022-09-08T04:30:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:04.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:04.203968+0000) 2022-09-08T04:30:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:04.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:04.193530+0000) 2022-09-08T04:30:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:04 smithi055 ceph-mon[97033]: pgmap v13322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:04 smithi059 ceph-mon[97450]: pgmap v13322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:05.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:05.204147+0000) 2022-09-08T04:30:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:05.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:05.193768+0000) 2022-09-08T04:30:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:30:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:30:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:06.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:06.194018+0000) 2022-09-08T04:30:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:06.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:06.204374+0000) 2022-09-08T04:30:07.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:06 smithi055 ceph-mon[97033]: pgmap v13323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:30:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:30:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:06 smithi059 ceph-mon[97450]: pgmap v13323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:30:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:30:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:07.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:07.204576+0000) 2022-09-08T04:30:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:07.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:07.194238+0000) 2022-09-08T04:30:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:08.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:08.204808+0000) 2022-09-08T04:30:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:08.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:08.194470+0000) 2022-09-08T04:30:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:08 smithi059 ceph-mon[97450]: pgmap v13324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:30:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:08 smithi055 ceph-mon[97033]: pgmap v13324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:30:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:09.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:09.205062+0000) 2022-09-08T04:30:09.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:09.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:09.194712+0000) 2022-09-08T04:30:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:09 smithi059 ceph-mon[97450]: osdmap e913: 12 total, 11 up, 11 in 2022-09-08T04:30:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:09 smithi055 ceph-mon[97033]: osdmap e913: 12 total, 11 up, 11 in 2022-09-08T04:30:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:10.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:10.205278+0000) 2022-09-08T04:30:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:10.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:10.194912+0000) 2022-09-08T04:30:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:10 smithi059 ceph-mon[97450]: pgmap v13326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:30:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:11 smithi055 ceph-mon[97033]: pgmap v13326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:30:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:11.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:11.205514+0000) 2022-09-08T04:30:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:11.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:11.195139+0000) 2022-09-08T04:30:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:12.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:12.205752+0000) 2022-09-08T04:30:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:12.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:12.195336+0000) 2022-09-08T04:30:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:12 smithi059 ceph-mon[97450]: pgmap v13327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:30:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:12 smithi055 ceph-mon[97033]: pgmap v13327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:30:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:13.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:13.205966+0000) 2022-09-08T04:30:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:13.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:13.195557+0000) 2022-09-08T04:30:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:14.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:14.206101+0000) 2022-09-08T04:30:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:14.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:14.195710+0000) 2022-09-08T04:30:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:15 smithi059 ceph-mon[97450]: pgmap v13328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:30:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:15 smithi059 ceph-mon[97450]: osdmap e914: 12 total, 11 up, 11 in 2022-09-08T04:30:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:15 smithi055 ceph-mon[97033]: pgmap v13328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:30:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:15 smithi055 ceph-mon[97033]: osdmap e914: 12 total, 11 up, 11 in 2022-09-08T04:30:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:15.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:15.206311+0000) 2022-09-08T04:30:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:15.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:15.195917+0000) 2022-09-08T04:30:16.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:30:15.710+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:30:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:16.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:16.206517+0000) 2022-09-08T04:30:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:16.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:16.196144+0000) 2022-09-08T04:30:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:17 smithi059 ceph-mon[97450]: pgmap v13330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:30:17.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:17 smithi055 ceph-mon[97033]: pgmap v13330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:30:17.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:17.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:17.206666+0000) 2022-09-08T04:30:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:17.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:17.196350+0000) 2022-09-08T04:30:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:18.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:18.206881+0000) 2022-09-08T04:30:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:18.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:18.196573+0000) 2022-09-08T04:30:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:19 smithi059 ceph-mon[97450]: pgmap v13331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T04:30:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:19 smithi055 ceph-mon[97033]: pgmap v13331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T04:30:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:19.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:19.207108+0000) 2022-09-08T04:30:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:19.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:19.196817+0000) 2022-09-08T04:30:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:20.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:20.207295+0000) 2022-09-08T04:30:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:20.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:20.197040+0000) 2022-09-08T04:30:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:21 smithi059 ceph-mon[97450]: pgmap v13332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:30:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:21 smithi055 ceph-mon[97033]: pgmap v13332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:30:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:21.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:21.207535+0000) 2022-09-08T04:30:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:21.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:21.197289+0000) 2022-09-08T04:30:22.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:22.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:22.207682+0000) 2022-09-08T04:30:22.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:22.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:22.197457+0000) 2022-09-08T04:30:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:23 smithi059 ceph-mon[97450]: pgmap v13333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:30:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:23 smithi055 ceph-mon[97033]: pgmap v13333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:30:23.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:23.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:23.207919+0000) 2022-09-08T04:30:23.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:23.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:23.197733+0000) 2022-09-08T04:30:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:24.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:24.208110+0000) 2022-09-08T04:30:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:24.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:24.197949+0000) 2022-09-08T04:30:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:25 smithi059 ceph-mon[97450]: pgmap v13334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:30:25.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:25 smithi055 ceph-mon[97033]: pgmap v13334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:30:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:25.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:25.208289+0000) 2022-09-08T04:30:25.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:25.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:25.198207+0000) 2022-09-08T04:30:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:26.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:26.208494+0000) 2022-09-08T04:30:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:26.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:26.198474+0000) 2022-09-08T04:30:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:27 smithi059 ceph-mon[97450]: pgmap v13335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 98 B/s wr, 8 op/s 2022-09-08T04:30:27.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:27 smithi055 ceph-mon[97033]: pgmap v13335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 98 B/s wr, 8 op/s 2022-09-08T04:30:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:27.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:27.208732+0000) 2022-09-08T04:30:27.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:27.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:27.198756+0000) 2022-09-08T04:30:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:28.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:28.208958+0000) 2022-09-08T04:30:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:28.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:28.198946+0000) 2022-09-08T04:30:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:29 smithi059 ceph-mon[97450]: pgmap v13336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:29.299 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:29 smithi055 ceph-mon[97033]: pgmap v13336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:29.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:29.209093+0000) 2022-09-08T04:30:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:29.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:29.199099+0000) 2022-09-08T04:30:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:30:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:30:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:30:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:30:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:30:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:30:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:30.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:30.209284+0000) 2022-09-08T04:30:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:30.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:30.199351+0000) 2022-09-08T04:30:31.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:30:30.719+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:30:31.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:31 smithi055 ceph-mon[97033]: pgmap v13337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:31 smithi059 ceph-mon[97450]: pgmap v13337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:31.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:31.209477+0000) 2022-09-08T04:30:31.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:31.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:31.199613+0000) 2022-09-08T04:30:32.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:32 smithi055 ceph-mon[97033]: pgmap v13338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:30:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:32 smithi059 ceph-mon[97450]: pgmap v13338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:30:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:32.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:32.209728+0000) 2022-09-08T04:30:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:32.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:32.199891+0000) 2022-09-08T04:30:33.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:33.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:33.209906+0000) 2022-09-08T04:30:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:33.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:33.200152+0000) 2022-09-08T04:30:34.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:34 smithi055 ceph-mon[97033]: pgmap v13339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:34 smithi059 ceph-mon[97450]: pgmap v13339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:34.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:34.210080+0000) 2022-09-08T04:30:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:34.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:34.200384+0000) 2022-09-08T04:30:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:35.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:35.210264+0000) 2022-09-08T04:30:35.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:35.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:35.200592+0000) 2022-09-08T04:30:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:36.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:36.210425+0000) 2022-09-08T04:30:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:36.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:36.200884+0000) 2022-09-08T04:30:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:36 smithi055 ceph-mon[97033]: pgmap v13340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:30:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:36 smithi059 ceph-mon[97450]: pgmap v13340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:30:37.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:37.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:37.201136+0000) 2022-09-08T04:30:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:37.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:37.210679+0000) 2022-09-08T04:30:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:38.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:38.210874+0000) 2022-09-08T04:30:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:38.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:38.201389+0000) 2022-09-08T04:30:38.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:38 smithi055 ceph-mon[97033]: pgmap v13341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:38 smithi059 ceph-mon[97450]: pgmap v13341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:39.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:39.211070+0000) 2022-09-08T04:30:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:39.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:39.201694+0000) 2022-09-08T04:30:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:40.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:40.211303+0000) 2022-09-08T04:30:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:40.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:40.201933+0000) 2022-09-08T04:30:41.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:40 smithi059 ceph-mon[97450]: pgmap v13342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:40 smithi055 ceph-mon[97033]: pgmap v13342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:41.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:41.211502+0000) 2022-09-08T04:30:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:41.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:41.202117+0000) 2022-09-08T04:30:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:42.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:42.211766+0000) 2022-09-08T04:30:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:42.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:42.202359+0000) 2022-09-08T04:30:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:42 smithi059 ceph-mon[97450]: pgmap v13343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:30:43.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:42 smithi055 ceph-mon[97033]: pgmap v13343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:30:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:43.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:43.211927+0000) 2022-09-08T04:30:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:43.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:43.202543+0000) 2022-09-08T04:30:44.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:44.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:44.202809+0000) 2022-09-08T04:30:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:44.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:44.212083+0000) 2022-09-08T04:30:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:44 smithi059 ceph-mon[97450]: pgmap v13344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:45.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:44 smithi055 ceph-mon[97033]: pgmap v13344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:45.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:45.212238+0000) 2022-09-08T04:30:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:45.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:45.203063+0000) 2022-09-08T04:30:46.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:30:45.732+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:30:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:46.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:46.212467+0000) 2022-09-08T04:30:46.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:46.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:46.203225+0000) 2022-09-08T04:30:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:46 smithi059 ceph-mon[97450]: pgmap v13345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:30:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:46 smithi055 ceph-mon[97033]: pgmap v13345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:30:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:47.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:47.212657+0000) 2022-09-08T04:30:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:47.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:47.203407+0000) 2022-09-08T04:30:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:48.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:48.212868+0000) 2022-09-08T04:30:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:48.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:48.203686+0000) 2022-09-08T04:30:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:48 smithi059 ceph-mon[97450]: pgmap v13346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:49.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:48 smithi055 ceph-mon[97033]: pgmap v13346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:49.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:49.203869+0000) 2022-09-08T04:30:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:49.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:49.213058+0000) 2022-09-08T04:30:50.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:50.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:50.204104+0000) 2022-09-08T04:30:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:50.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:50.213256+0000) 2022-09-08T04:30:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:50 smithi059 ceph-mon[97450]: pgmap v13347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:51.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:50 smithi055 ceph-mon[97033]: pgmap v13347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:51.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:51.213486+0000) 2022-09-08T04:30:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:51.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:51.204318+0000) 2022-09-08T04:30:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:52.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:52.204538+0000) 2022-09-08T04:30:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:52.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:52.213721+0000) 2022-09-08T04:30:53.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:52 smithi059 ceph-mon[97450]: pgmap v13348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:30:53.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:52 smithi055 ceph-mon[97033]: pgmap v13348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:30:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:53.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:53.213899+0000) 2022-09-08T04:30:53.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:53.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:53.204801+0000) 2022-09-08T04:30:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:54.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:54.214085+0000) 2022-09-08T04:30:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:54.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:54.205016+0000) 2022-09-08T04:30:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:54 smithi059 ceph-mon[97450]: pgmap v13349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:54 smithi055 ceph-mon[97033]: pgmap v13349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:55.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:55.214285+0000) 2022-09-08T04:30:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:55.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:55.205257+0000) 2022-09-08T04:30:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:56.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:56.214519+0000) 2022-09-08T04:30:56.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:56.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:56.205522+0000) 2022-09-08T04:30:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:56 smithi059 ceph-mon[97450]: pgmap v13350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:30:57.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:56 smithi055 ceph-mon[97033]: pgmap v13350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:30:57.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:57.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:57.205797+0000) 2022-09-08T04:30:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:57.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:57.214737+0000) 2022-09-08T04:30:58.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:58.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:58.205983+0000) 2022-09-08T04:30:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:58.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:58.214931+0000) 2022-09-08T04:30:59.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:30:58 smithi059 ceph-mon[97450]: pgmap v13351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:59.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:30:58 smithi055 ceph-mon[97033]: pgmap v13351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:30:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:30:59.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:59.215053+0000) 2022-09-08T04:30:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:30:59.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:30:59.206223+0000) 2022-09-08T04:31:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:00.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:00.206448+0000) 2022-09-08T04:31:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:00.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:00.215199+0000) 2022-09-08T04:31:00.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:00 smithi055 ceph-mon[97033]: pgmap v13352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:00 smithi055 ceph-mon[97033]: pgmap v13353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:31:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:00 smithi055 ceph-mon[97033]: pgmap v13354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:31:00.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:31:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:00 smithi059 ceph-mon[97450]: pgmap v13352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:00 smithi059 ceph-mon[97450]: pgmap v13353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:31:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:00 smithi059 ceph-mon[97450]: pgmap v13354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:31:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:31:01.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:31:00.746+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:31:01.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:01.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:01.215401+0000) 2022-09-08T04:31:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:01.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:01.206665+0000) 2022-09-08T04:31:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:02.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:02.215605+0000) 2022-09-08T04:31:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:02.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:02.206788+0000) 2022-09-08T04:31:02.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:02 smithi055 ceph-mon[97033]: pgmap v13355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:31:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:02 smithi059 ceph-mon[97450]: pgmap v13355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:31:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:03.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:03.215777+0000) 2022-09-08T04:31:03.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:03.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:03.207029+0000) 2022-09-08T04:31:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:04.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:04.207227+0000) 2022-09-08T04:31:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:04.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:04.215978+0000) 2022-09-08T04:31:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:04 smithi055 ceph-mon[97033]: pgmap v13356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:31:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:04 smithi059 ceph-mon[97450]: pgmap v13356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:31:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:05.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:05.207470+0000) 2022-09-08T04:31:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:05.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:05.216187+0000) 2022-09-08T04:31:06.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:31:06.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:31:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:06.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:06.216366+0000) 2022-09-08T04:31:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:06.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:06.207718+0000) 2022-09-08T04:31:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:06 smithi059 ceph-mon[97450]: pgmap v13357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T04:31:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:31:07.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:06 smithi055 ceph-mon[97033]: pgmap v13357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T04:31:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:31:07.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:07.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:07.216562+0000) 2022-09-08T04:31:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:07.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:07.207960+0000) 2022-09-08T04:31:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:08.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:08.216792+0000) 2022-09-08T04:31:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:08.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:08.208201+0000) 2022-09-08T04:31:09.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:08 smithi059 ceph-mon[97450]: pgmap v13358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T04:31:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:08 smithi055 ceph-mon[97033]: pgmap v13358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T04:31:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:09.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:09.216996+0000) 2022-09-08T04:31:09.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:09.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:09.208446+0000) 2022-09-08T04:31:10.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:09 smithi059 ceph-mon[97450]: osdmap e915: 12 total, 11 up, 11 in 2022-09-08T04:31:10.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:09 smithi055 ceph-mon[97033]: osdmap e915: 12 total, 11 up, 11 in 2022-09-08T04:31:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:10.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:10.217224+0000) 2022-09-08T04:31:10.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:10.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:10.208734+0000) 2022-09-08T04:31:11.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:10 smithi059 ceph-mon[97450]: pgmap v13360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:31:11.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:10 smithi055 ceph-mon[97033]: pgmap v13360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:31:11.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:11.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:11.208965+0000) 2022-09-08T04:31:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:11.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:11.217396+0000) 2022-09-08T04:31:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:12.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:12.217589+0000) 2022-09-08T04:31:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:12.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:12.209269+0000) 2022-09-08T04:31:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:12 smithi059 ceph-mon[97450]: pgmap v13361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:31:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:12 smithi055 ceph-mon[97033]: pgmap v13361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:31:13.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:13.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:13.209556+0000) 2022-09-08T04:31:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:13.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:13.217790+0000) 2022-09-08T04:31:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:14.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:14.217991+0000) 2022-09-08T04:31:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:14.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:14.209767+0000) 2022-09-08T04:31:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:14 smithi059 ceph-mon[97450]: pgmap v13362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:31:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:14 smithi059 ceph-mon[97450]: osdmap e916: 12 total, 11 up, 11 in 2022-09-08T04:31:15.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:14 smithi055 ceph-mon[97033]: pgmap v13362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:31:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:14 smithi055 ceph-mon[97033]: osdmap e916: 12 total, 11 up, 11 in 2022-09-08T04:31:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:15.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:15.218161+0000) 2022-09-08T04:31:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:15.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:15.209994+0000) 2022-09-08T04:31:16.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:31:15.754+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:31:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:16.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:16.218374+0000) 2022-09-08T04:31:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:16.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:16.210210+0000) 2022-09-08T04:31:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:16 smithi059 ceph-mon[97450]: pgmap v13364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:31:17.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:16 smithi055 ceph-mon[97033]: pgmap v13364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:31:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:17.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:17.218596+0000) 2022-09-08T04:31:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:17.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:17.210432+0000) 2022-09-08T04:31:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:18.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:18.210710+0000) 2022-09-08T04:31:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:18.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:18.218795+0000) 2022-09-08T04:31:19.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:18 smithi059 ceph-mon[97450]: pgmap v13365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 238 B/s wr, 7 op/s 2022-09-08T04:31:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:18 smithi055 ceph-mon[97033]: pgmap v13365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 238 B/s wr, 7 op/s 2022-09-08T04:31:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:19.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:19.218949+0000) 2022-09-08T04:31:19.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:19.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:19.211018+0000) 2022-09-08T04:31:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:20.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:20.219151+0000) 2022-09-08T04:31:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:20.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:20.211228+0000) 2022-09-08T04:31:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:20 smithi059 ceph-mon[97450]: pgmap v13366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:31:21.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:20 smithi055 ceph-mon[97033]: pgmap v13366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:31:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:21.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:21.219339+0000) 2022-09-08T04:31:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:21.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:21.211459+0000) 2022-09-08T04:31:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:22.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:22.219572+0000) 2022-09-08T04:31:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:22.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:22.211699+0000) 2022-09-08T04:31:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:22 smithi059 ceph-mon[97450]: pgmap v13367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:31:23.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:22 smithi055 ceph-mon[97033]: pgmap v13367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:31:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:23.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:23.219825+0000) 2022-09-08T04:31:23.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:23.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:23.211912+0000) 2022-09-08T04:31:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:24.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:24.219982+0000) 2022-09-08T04:31:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:24.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:24.212041+0000) 2022-09-08T04:31:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:24 smithi055 ceph-mon[97033]: pgmap v13368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:31:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:24 smithi059 ceph-mon[97450]: pgmap v13368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:31:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:25.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:25.220140+0000) 2022-09-08T04:31:25.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:25.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:25.212253+0000) 2022-09-08T04:31:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:26.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:26.220375+0000) 2022-09-08T04:31:26.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:26.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:26.212478+0000) 2022-09-08T04:31:27.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:26 smithi055 ceph-mon[97033]: pgmap v13369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-08T04:31:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:26 smithi059 ceph-mon[97450]: pgmap v13369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-08T04:31:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:27.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:27.220605+0000) 2022-09-08T04:31:27.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:27.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:27.212735+0000) 2022-09-08T04:31:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:28.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:28.220836+0000) 2022-09-08T04:31:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:28.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:28.212911+0000) 2022-09-08T04:31:29.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:28 smithi055 ceph-mon[97033]: pgmap v13370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:28 smithi059 ceph-mon[97450]: pgmap v13370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:29.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:29.221003+0000) 2022-09-08T04:31:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:29.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:29.213154+0000) 2022-09-08T04:31:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:30.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:30.221209+0000) 2022-09-08T04:31:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:30.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:30.213383+0000) 2022-09-08T04:31:31.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:31:30.763+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:31:31.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:30 smithi055 ceph-mon[97033]: pgmap v13371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:30 smithi059 ceph-mon[97450]: pgmap v13371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:31.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:31.221402+0000) 2022-09-08T04:31:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:31.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:31.213599+0000) 2022-09-08T04:31:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:32.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:32.221599+0000) 2022-09-08T04:31:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:32.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:32.213826+0000) 2022-09-08T04:31:33.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:32 smithi055 ceph-mon[97033]: pgmap v13372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:31:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:32 smithi059 ceph-mon[97450]: pgmap v13372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:31:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:33.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:33.221836+0000) 2022-09-08T04:31:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:33.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:33.214073+0000) 2022-09-08T04:31:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:34.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:34.221963+0000) 2022-09-08T04:31:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:34.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:34.214315+0000) 2022-09-08T04:31:35.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:34 smithi055 ceph-mon[97033]: pgmap v13373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:34 smithi059 ceph-mon[97450]: pgmap v13373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:35.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:35.222122+0000) 2022-09-08T04:31:35.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:35.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:35.214537+0000) 2022-09-08T04:31:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:36.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:36.214785+0000) 2022-09-08T04:31:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:36.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:36.222298+0000) 2022-09-08T04:31:37.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:36 smithi055 ceph-mon[97033]: pgmap v13374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:31:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:36 smithi059 ceph-mon[97450]: pgmap v13374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:31:37.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:37.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:37.214953+0000) 2022-09-08T04:31:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:37.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:37.222507+0000) 2022-09-08T04:31:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:38.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:38.222691+0000) 2022-09-08T04:31:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:38.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:38.215133+0000) 2022-09-08T04:31:39.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:38 smithi055 ceph-mon[97033]: pgmap v13375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:38 smithi059 ceph-mon[97450]: pgmap v13375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:39.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:39.222863+0000) 2022-09-08T04:31:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:39.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:39.215338+0000) 2022-09-08T04:31:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:40.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:40.223010+0000) 2022-09-08T04:31:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:40.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:40.215550+0000) 2022-09-08T04:31:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:40 smithi055 ceph-mon[97033]: pgmap v13376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:40 smithi059 ceph-mon[97450]: pgmap v13376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:41.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:41.223164+0000) 2022-09-08T04:31:41.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:41.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:41.215791+0000) 2022-09-08T04:31:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:42.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:42.223368+0000) 2022-09-08T04:31:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:42.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:42.216026+0000) 2022-09-08T04:31:43.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:42 smithi055 ceph-mon[97033]: pgmap v13377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:31:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:42 smithi059 ceph-mon[97450]: pgmap v13377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:31:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:43.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:43.216294+0000) 2022-09-08T04:31:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:43.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:43.223622+0000) 2022-09-08T04:31:44.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:44.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:44.223904+0000) 2022-09-08T04:31:44.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:44.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:44.216551+0000) 2022-09-08T04:31:45.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:44 smithi055 ceph-mon[97033]: pgmap v13378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:44 smithi059 ceph-mon[97450]: pgmap v13378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:45.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:45.224110+0000) 2022-09-08T04:31:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:45.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:45.216801+0000) 2022-09-08T04:31:46.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:31:45.778+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:31:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:46.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:46.224244+0000) 2022-09-08T04:31:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:46.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:46.217056+0000) 2022-09-08T04:31:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:46 smithi055 ceph-mon[97033]: pgmap v13379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:31:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:46 smithi059 ceph-mon[97450]: pgmap v13379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:31:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:47.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:47.224419+0000) 2022-09-08T04:31:47.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:47.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:47.217270+0000) 2022-09-08T04:31:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:48.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:48.224659+0000) 2022-09-08T04:31:48.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:48.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:48.217442+0000) 2022-09-08T04:31:49.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:48 smithi055 ceph-mon[97033]: pgmap v13380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:48 smithi059 ceph-mon[97450]: pgmap v13380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:49.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:49.224869+0000) 2022-09-08T04:31:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:49.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:49.217718+0000) 2022-09-08T04:31:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:50.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:50.225091+0000) 2022-09-08T04:31:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:50.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:50.217899+0000) 2022-09-08T04:31:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:50 smithi055 ceph-mon[97033]: pgmap v13381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:50 smithi059 ceph-mon[97450]: pgmap v13381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:51.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:51.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:51.225218+0000) 2022-09-08T04:31:51.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:51.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:51.218050+0000) 2022-09-08T04:31:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:52.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:52.225430+0000) 2022-09-08T04:31:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:52.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:52.218258+0000) 2022-09-08T04:31:53.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:52 smithi055 ceph-mon[97033]: pgmap v13382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:31:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:52 smithi059 ceph-mon[97450]: pgmap v13382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:31:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:53.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:53.225698+0000) 2022-09-08T04:31:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:53.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:53.218486+0000) 2022-09-08T04:31:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:54.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:54.225941+0000) 2022-09-08T04:31:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:54.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:54.218651+0000) 2022-09-08T04:31:55.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:54 smithi055 ceph-mon[97033]: pgmap v13383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:54 smithi059 ceph-mon[97450]: pgmap v13383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:55.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:55.226197+0000) 2022-09-08T04:31:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:55.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:55.218867+0000) 2022-09-08T04:31:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:56.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:56.226394+0000) 2022-09-08T04:31:56.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:56.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:56.219118+0000) 2022-09-08T04:31:57.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:56 smithi055 ceph-mon[97033]: pgmap v13384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:31:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:56 smithi059 ceph-mon[97450]: pgmap v13384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:31:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:57.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:57.226625+0000) 2022-09-08T04:31:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:57.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:57.219336+0000) 2022-09-08T04:31:58.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:58.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:58.219558+0000) 2022-09-08T04:31:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:58.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:58.226880+0000) 2022-09-08T04:31:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:58 smithi055 ceph-mon[97033]: pgmap v13385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:58 smithi059 ceph-mon[97450]: pgmap v13385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:31:59.488 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:31:59.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:59.227110+0000) 2022-09-08T04:31:59.489 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:31:59.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:31:59.219785+0000) 2022-09-08T04:32:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:32:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:32:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:31:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:32:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:32:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:32:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:31:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:32:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:00.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:00.220016+0000) 2022-09-08T04:32:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:00.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:00.227333+0000) 2022-09-08T04:32:01.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:32:00.788+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:32:01.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:00 smithi055 ceph-mon[97033]: pgmap v13386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:00 smithi059 ceph-mon[97450]: pgmap v13386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:01.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:01.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:01.227563+0000) 2022-09-08T04:32:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:01.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:01.220233+0000) 2022-09-08T04:32:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:02.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:02.227796+0000) 2022-09-08T04:32:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:02.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:02.220428+0000) 2022-09-08T04:32:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:02 smithi055 ceph-mon[97033]: pgmap v13387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:32:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:02 smithi059 ceph-mon[97450]: pgmap v13387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:32:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:03.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:03.228047+0000) 2022-09-08T04:32:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:03.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:03.220688+0000) 2022-09-08T04:32:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:04.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:04.228240+0000) 2022-09-08T04:32:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:04.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:04.220924+0000) 2022-09-08T04:32:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:04 smithi055 ceph-mon[97033]: pgmap v13388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:04 smithi059 ceph-mon[97450]: pgmap v13388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:05.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:05.228449+0000) 2022-09-08T04:32:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:05.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:05.221073+0000) 2022-09-08T04:32:06.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:32:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:32:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:06.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:06.228652+0000) 2022-09-08T04:32:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:06.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:06.221301+0000) 2022-09-08T04:32:07.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:06 smithi055 ceph-mon[97033]: pgmap v13389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:32:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:32:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:06 smithi059 ceph-mon[97450]: pgmap v13389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:32:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:32:07.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:07.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:07.228872+0000) 2022-09-08T04:32:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:07.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:07.221536+0000) 2022-09-08T04:32:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:08.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:08.229122+0000) 2022-09-08T04:32:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:08.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:08.221748+0000) 2022-09-08T04:32:09.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:08 smithi055 ceph-mon[97033]: pgmap v13390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:32:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:08 smithi059 ceph-mon[97450]: pgmap v13390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:32:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:09.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:09.229263+0000) 2022-09-08T04:32:09.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:09.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:09.221965+0000) 2022-09-08T04:32:10.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:09 smithi055 ceph-mon[97033]: osdmap e917: 12 total, 11 up, 11 in 2022-09-08T04:32:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:09 smithi059 ceph-mon[97450]: osdmap e917: 12 total, 11 up, 11 in 2022-09-08T04:32:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:10.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:10.229430+0000) 2022-09-08T04:32:10.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:10.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:10.222204+0000) 2022-09-08T04:32:11.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:10 smithi055 ceph-mon[97033]: pgmap v13392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:32:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:10 smithi059 ceph-mon[97450]: pgmap v13392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:32:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:11.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:11.229617+0000) 2022-09-08T04:32:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:11.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:11.222449+0000) 2022-09-08T04:32:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:12.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:12.229838+0000) 2022-09-08T04:32:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:12.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:12.222740+0000) 2022-09-08T04:32:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:12 smithi055 ceph-mon[97033]: pgmap v13393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:32:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:12 smithi059 ceph-mon[97450]: pgmap v13393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:32:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:13.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:13.230034+0000) 2022-09-08T04:32:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:13.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:13.222994+0000) 2022-09-08T04:32:14.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:14.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:14.230289+0000) 2022-09-08T04:32:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:14.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:14.223240+0000) 2022-09-08T04:32:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:14 smithi055 ceph-mon[97033]: pgmap v13394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:32:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:14 smithi055 ceph-mon[97033]: osdmap e918: 12 total, 11 up, 11 in 2022-09-08T04:32:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:14 smithi059 ceph-mon[97450]: pgmap v13394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:32:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:14 smithi059 ceph-mon[97450]: osdmap e918: 12 total, 11 up, 11 in 2022-09-08T04:32:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:15.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:15.230563+0000) 2022-09-08T04:32:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:15.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:15.223456+0000) 2022-09-08T04:32:16.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:32:15.802+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:32:16.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:16.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:16.230755+0000) 2022-09-08T04:32:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:16.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:16.223697+0000) 2022-09-08T04:32:17.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:16 smithi055 ceph-mon[97033]: pgmap v13396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:32:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:16 smithi059 ceph-mon[97450]: pgmap v13396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:32:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:17.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:17.230895+0000) 2022-09-08T04:32:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:17.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:17.223943+0000) 2022-09-08T04:32:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:18.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:18.231095+0000) 2022-09-08T04:32:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:18.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:18.224216+0000) 2022-09-08T04:32:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:18 smithi055 ceph-mon[97033]: pgmap v13397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 241 B/s wr, 7 op/s 2022-09-08T04:32:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:18 smithi059 ceph-mon[97450]: pgmap v13397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 241 B/s wr, 7 op/s 2022-09-08T04:32:19.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:19.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:19.231336+0000) 2022-09-08T04:32:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:19.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:19.224540+0000) 2022-09-08T04:32:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:20.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:20.231608+0000) 2022-09-08T04:32:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:20.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:20.224769+0000) 2022-09-08T04:32:21.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:20 smithi055 ceph-mon[97033]: pgmap v13398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:32:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:20 smithi059 ceph-mon[97450]: pgmap v13398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:32:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:21.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:21.231814+0000) 2022-09-08T04:32:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:21.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:21.225007+0000) 2022-09-08T04:32:22.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:22.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:22.231951+0000) 2022-09-08T04:32:22.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:22.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:22.225256+0000) 2022-09-08T04:32:23.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:22 smithi055 ceph-mon[97033]: pgmap v13399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:32:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:22 smithi059 ceph-mon[97450]: pgmap v13399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:32:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:23.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:23.232176+0000) 2022-09-08T04:32:23.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:23.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:23.225513+0000) 2022-09-08T04:32:24.562 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:24.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:24.225730+0000) 2022-09-08T04:32:24.563 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:24.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:24.232379+0000) 2022-09-08T04:32:25.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:24 smithi055 ceph-mon[97033]: pgmap v13400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:32:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:24 smithi059 ceph-mon[97450]: pgmap v13400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:32:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:25.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:25.232625+0000) 2022-09-08T04:32:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:25.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:25.225912+0000) 2022-09-08T04:32:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:26.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:26.232841+0000) 2022-09-08T04:32:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:26.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:26.226157+0000) 2022-09-08T04:32:27.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:26 smithi055 ceph-mon[97033]: pgmap v13401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-08T04:32:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:26 smithi059 ceph-mon[97450]: pgmap v13401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2022-09-08T04:32:27.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:27.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:27.233042+0000) 2022-09-08T04:32:27.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:27.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:27.226350+0000) 2022-09-08T04:32:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:28.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:28.233324+0000) 2022-09-08T04:32:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:28.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:28.226545+0000) 2022-09-08T04:32:29.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:28 smithi055 ceph-mon[97033]: pgmap v13402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:28 smithi059 ceph-mon[97450]: pgmap v13402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:29.616 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:29.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:29.233555+0000) 2022-09-08T04:32:29.617 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:29.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:29.226778+0000) 2022-09-08T04:32:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:30.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:30.233810+0000) 2022-09-08T04:32:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:30.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:30.226969+0000) 2022-09-08T04:32:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:30 smithi055 ceph-mon[97033]: pgmap v13403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:30 smithi055 ceph-mon[97033]: pgmap v13404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:32:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:30 smithi055 ceph-mon[97033]: pgmap v13405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:32:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:32:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:30 smithi059 ceph-mon[97450]: pgmap v13403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:30 smithi059 ceph-mon[97450]: pgmap v13404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:32:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:30 smithi059 ceph-mon[97450]: pgmap v13405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:32:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:32:31.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:32:30.811+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:32:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:31.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:31.234059+0000) 2022-09-08T04:32:31.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:31.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:31.227193+0000) 2022-09-08T04:32:32.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:32.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:32.234334+0000) 2022-09-08T04:32:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:32.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:32.227433+0000) 2022-09-08T04:32:32.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:32 smithi055 ceph-mon[97033]: pgmap v13406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:32:33.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:32 smithi059 ceph-mon[97450]: pgmap v13406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:32:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:33.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:33.227695+0000) 2022-09-08T04:32:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:33.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:33.234591+0000) 2022-09-08T04:32:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:34.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:34.234820+0000) 2022-09-08T04:32:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:34.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:34.227883+0000) 2022-09-08T04:32:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:34 smithi055 ceph-mon[97033]: pgmap v13407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:32:35.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:34 smithi059 ceph-mon[97450]: pgmap v13407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:32:35.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:35.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:35.235023+0000) 2022-09-08T04:32:35.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:35.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:35.228067+0000) 2022-09-08T04:32:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:36.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:36.235195+0000) 2022-09-08T04:32:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:36.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:36.228254+0000) 2022-09-08T04:32:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:36 smithi059 ceph-mon[97450]: pgmap v13408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:32:37.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:36 smithi055 ceph-mon[97033]: pgmap v13408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:32:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:37.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:37.235406+0000) 2022-09-08T04:32:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:37.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:37.228489+0000) 2022-09-08T04:32:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:38.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:38.235574+0000) 2022-09-08T04:32:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:38.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:38.228717+0000) 2022-09-08T04:32:39.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:38 smithi059 ceph-mon[97450]: pgmap v13409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:32:39.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:38 smithi055 ceph-mon[97033]: pgmap v13409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:32:39.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:39.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:39.228875+0000) 2022-09-08T04:32:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:39.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:39.235810+0000) 2022-09-08T04:32:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:40.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:40.236039+0000) 2022-09-08T04:32:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:40.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:40.229018+0000) 2022-09-08T04:32:41.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:40 smithi059 ceph-mon[97450]: pgmap v13410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:32:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:40 smithi055 ceph-mon[97033]: pgmap v13410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:32:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:41.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:41.236259+0000) 2022-09-08T04:32:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:41.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:41.229230+0000) 2022-09-08T04:32:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:42.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:42.236443+0000) 2022-09-08T04:32:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:42.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:42.229437+0000) 2022-09-08T04:32:43.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:42 smithi059 ceph-mon[97450]: pgmap v13411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:32:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:42 smithi055 ceph-mon[97033]: pgmap v13411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:32:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:43.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:43.236573+0000) 2022-09-08T04:32:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:43.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:43.229597+0000) 2022-09-08T04:32:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:44.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:44.236816+0000) 2022-09-08T04:32:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:44.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:44.229784+0000) 2022-09-08T04:32:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:44 smithi059 ceph-mon[97450]: pgmap v13412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:45.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:44 smithi055 ceph-mon[97033]: pgmap v13412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:45.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:45.236973+0000) 2022-09-08T04:32:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:45.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:45.230024+0000) 2022-09-08T04:32:46.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:32:45.821+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:32:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:46.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:46.237091+0000) 2022-09-08T04:32:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:46.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:46.230194+0000) 2022-09-08T04:32:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:46 smithi059 ceph-mon[97450]: pgmap v13413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:32:47.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:46 smithi055 ceph-mon[97033]: pgmap v13413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:32:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:47.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:47.237299+0000) 2022-09-08T04:32:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:47.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:47.230392+0000) 2022-09-08T04:32:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:48.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:48.237534+0000) 2022-09-08T04:32:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:48.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:48.230628+0000) 2022-09-08T04:32:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:48 smithi059 ceph-mon[97450]: pgmap v13414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:49.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:48 smithi055 ceph-mon[97033]: pgmap v13414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:49.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:49.230931+0000) 2022-09-08T04:32:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:49.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:49.237778+0000) 2022-09-08T04:32:50.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:50.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:50.231158+0000) 2022-09-08T04:32:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:50.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:50.237986+0000) 2022-09-08T04:32:51.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:50 smithi059 ceph-mon[97450]: pgmap v13415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:51.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:50 smithi055 ceph-mon[97033]: pgmap v13415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:51.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:51.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:51.231395+0000) 2022-09-08T04:32:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:51.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:51.238209+0000) 2022-09-08T04:32:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:52.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:52.231656+0000) 2022-09-08T04:32:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:52.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:52.238441+0000) 2022-09-08T04:32:53.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:52 smithi055 ceph-mon[97033]: pgmap v13416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:32:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:52 smithi059 ceph-mon[97450]: pgmap v13416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:32:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:53.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:53.238709+0000) 2022-09-08T04:32:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:53.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:53.231894+0000) 2022-09-08T04:32:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:54.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:54.238931+0000) 2022-09-08T04:32:54.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:54.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:54.232161+0000) 2022-09-08T04:32:55.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:54 smithi055 ceph-mon[97033]: pgmap v13417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:54 smithi059 ceph-mon[97450]: pgmap v13417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:55.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:55.239140+0000) 2022-09-08T04:32:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:55.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:55.232397+0000) 2022-09-08T04:32:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:56.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:56.239376+0000) 2022-09-08T04:32:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:56.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:56.232673+0000) 2022-09-08T04:32:57.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:56 smithi055 ceph-mon[97033]: pgmap v13418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:32:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:56 smithi059 ceph-mon[97450]: pgmap v13418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:32:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:57.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:57.239590+0000) 2022-09-08T04:32:57.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:57.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:57.232881+0000) 2022-09-08T04:32:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:58.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:58.239826+0000) 2022-09-08T04:32:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:58.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:58.233108+0000) 2022-09-08T04:32:59.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:32:58 smithi055 ceph-mon[97033]: pgmap v13419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:32:58 smithi059 ceph-mon[97450]: pgmap v13419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:32:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:32:59.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:59.239987+0000) 2022-09-08T04:32:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:32:59.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:32:59.233367+0000) 2022-09-08T04:33:00.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:00.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:00.240125+0000) 2022-09-08T04:33:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:00.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:00.233490+0000) 2022-09-08T04:33:01.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:33:00.833+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:33:01.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:00 smithi055 ceph-mon[97033]: pgmap v13420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:00 smithi059 ceph-mon[97450]: pgmap v13420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:01.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:01.240333+0000) 2022-09-08T04:33:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:01.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:01.233724+0000) 2022-09-08T04:33:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:02.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:02.240553+0000) 2022-09-08T04:33:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:02.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:02.233882+0000) 2022-09-08T04:33:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:02 smithi055 ceph-mon[97033]: pgmap v13421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:33:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:02 smithi059 ceph-mon[97450]: pgmap v13421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:33:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:03.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:03.240765+0000) 2022-09-08T04:33:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:03.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:03.234081+0000) 2022-09-08T04:33:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:04.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:04.240898+0000) 2022-09-08T04:33:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:04.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:04.234273+0000) 2022-09-08T04:33:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:04 smithi055 ceph-mon[97033]: pgmap v13422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:04 smithi059 ceph-mon[97450]: pgmap v13422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:05.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:05.241019+0000) 2022-09-08T04:33:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:05.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:05.234439+0000) 2022-09-08T04:33:06.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:33:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:33:06.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:06.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:06.241239+0000) 2022-09-08T04:33:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:06.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:06.234723+0000) 2022-09-08T04:33:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:06 smithi055 ceph-mon[97033]: pgmap v13423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:33:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:33:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:06 smithi059 ceph-mon[97450]: pgmap v13423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:33:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:33:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:07.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:07.241448+0000) 2022-09-08T04:33:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:07.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:07.234965+0000) 2022-09-08T04:33:08.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:08.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:08.241658+0000) 2022-09-08T04:33:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:08.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:08.235189+0000) 2022-09-08T04:33:09.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:08 smithi055 ceph-mon[97033]: pgmap v13424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:33:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:08 smithi059 ceph-mon[97450]: pgmap v13424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:33:09.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:09.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:09.235427+0000) 2022-09-08T04:33:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:09.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:09.241901+0000) 2022-09-08T04:33:10.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:09 smithi055 ceph-mon[97033]: osdmap e919: 12 total, 11 up, 11 in 2022-09-08T04:33:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:09 smithi059 ceph-mon[97450]: osdmap e919: 12 total, 11 up, 11 in 2022-09-08T04:33:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:10.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:10.242077+0000) 2022-09-08T04:33:10.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:10.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:10.235701+0000) 2022-09-08T04:33:11.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:10 smithi055 ceph-mon[97033]: pgmap v13426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:33:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:10 smithi059 ceph-mon[97450]: pgmap v13426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:33:11.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:11.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:11.242198+0000) 2022-09-08T04:33:11.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:11.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:11.235881+0000) 2022-09-08T04:33:12.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:12.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:12.242406+0000) 2022-09-08T04:33:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:12.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:12.236016+0000) 2022-09-08T04:33:13.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:12 smithi055 ceph-mon[97033]: pgmap v13427: 129 pgs: 1 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 123 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:33:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:12 smithi059 ceph-mon[97450]: pgmap v13427: 129 pgs: 1 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 123 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:33:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:13.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:13.242605+0000) 2022-09-08T04:33:13.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:13.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:13.236233+0000) 2022-09-08T04:33:14.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:14.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:14.242825+0000) 2022-09-08T04:33:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:14.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:14.236474+0000) 2022-09-08T04:33:15.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:14 smithi055 ceph-mon[97033]: pgmap v13428: 129 pgs: 1 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 123 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:33:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:14 smithi055 ceph-mon[97033]: osdmap e920: 12 total, 11 up, 11 in 2022-09-08T04:33:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:14 smithi059 ceph-mon[97450]: pgmap v13428: 129 pgs: 1 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 123 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:33:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:14 smithi059 ceph-mon[97450]: osdmap e920: 12 total, 11 up, 11 in 2022-09-08T04:33:15.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:15.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:15.243046+0000) 2022-09-08T04:33:15.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:15.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:15.236625+0000) 2022-09-08T04:33:16.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:33:15.837+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:33:16.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:16.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:16.243209+0000) 2022-09-08T04:33:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:16.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:16.236775+0000) 2022-09-08T04:33:17.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:16 smithi055 ceph-mon[97033]: pgmap v13430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:33:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:16 smithi059 ceph-mon[97450]: pgmap v13430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:33:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:17.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:17.243430+0000) 2022-09-08T04:33:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:17.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:17.237023+0000) 2022-09-08T04:33:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:18.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:18.237250+0000) 2022-09-08T04:33:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:18.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:18.243688+0000) 2022-09-08T04:33:19.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:18 smithi055 ceph-mon[97033]: pgmap v13431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-08T04:33:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:18 smithi059 ceph-mon[97450]: pgmap v13431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2022-09-08T04:33:19.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:19.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:19.237500+0000) 2022-09-08T04:33:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:19.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:19.243833+0000) 2022-09-08T04:33:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:20.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:20.237677+0000) 2022-09-08T04:33:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:20.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:20.243975+0000) 2022-09-08T04:33:21.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:20 smithi059 ceph-mon[97450]: pgmap v13432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:33:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:20 smithi059 ceph-mon[97450]: osdmap e921: 12 total, 11 up, 11 in 2022-09-08T04:33:21.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:20 smithi055 ceph-mon[97033]: pgmap v13432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:33:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:20 smithi055 ceph-mon[97033]: osdmap e921: 12 total, 11 up, 11 in 2022-09-08T04:33:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:21.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:21.244143+0000) 2022-09-08T04:33:21.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:21.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:21.237844+0000) 2022-09-08T04:33:22.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:22.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:22.238084+0000) 2022-09-08T04:33:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:22.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:22.244318+0000) 2022-09-08T04:33:23.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:22 smithi059 ceph-mon[97450]: pgmap v13434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T04:33:23.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:22 smithi055 ceph-mon[97033]: pgmap v13434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T04:33:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:23.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:23.244548+0000) 2022-09-08T04:33:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:23.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:23.238315+0000) 2022-09-08T04:33:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:24.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:24.244725+0000) 2022-09-08T04:33:24.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:24.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:24.238491+0000) 2022-09-08T04:33:25.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:24 smithi059 ceph-mon[97450]: pgmap v13435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 115 B/s wr, 6 op/s 2022-09-08T04:33:25.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:24 smithi055 ceph-mon[97033]: pgmap v13435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 115 B/s wr, 6 op/s 2022-09-08T04:33:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:25.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:25.244895+0000) 2022-09-08T04:33:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:25.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:25.238708+0000) 2022-09-08T04:33:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:26.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:26.245029+0000) 2022-09-08T04:33:26.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:26.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:26.238831+0000) 2022-09-08T04:33:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:26 smithi059 ceph-mon[97450]: pgmap v13436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:33:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:26 smithi055 ceph-mon[97033]: pgmap v13436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:33:27.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:27.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:27.245224+0000) 2022-09-08T04:33:27.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:27.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:27.239045+0000) 2022-09-08T04:33:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:28.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:28.245419+0000) 2022-09-08T04:33:28.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:28.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:28.239262+0000) 2022-09-08T04:33:29.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:28 smithi059 ceph-mon[97450]: pgmap v13437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:33:29.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:28 smithi055 ceph-mon[97033]: pgmap v13437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:33:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:29.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:29.245586+0000) 2022-09-08T04:33:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:29.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:29.239503+0000) 2022-09-08T04:33:30.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:33:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:33:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:33:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:33:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:33:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:33:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:30.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:30.245818+0000) 2022-09-08T04:33:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:30.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:30.239733+0000) 2022-09-08T04:33:31.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:33:30.851+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:33:31.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:30 smithi055 ceph-mon[97033]: pgmap v13438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:33:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:30 smithi059 ceph-mon[97450]: pgmap v13438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:33:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:31.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:31.245941+0000) 2022-09-08T04:33:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:31.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:31.239901+0000) 2022-09-08T04:33:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:32.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:32.246098+0000) 2022-09-08T04:33:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:32.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:32.240126+0000) 2022-09-08T04:33:33.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:32 smithi055 ceph-mon[97033]: pgmap v13439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:33:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:32 smithi059 ceph-mon[97450]: pgmap v13439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:33:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:33.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:33.246272+0000) 2022-09-08T04:33:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:33.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:33.240365+0000) 2022-09-08T04:33:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:34.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:34.246502+0000) 2022-09-08T04:33:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:34.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:34.240606+0000) 2022-09-08T04:33:35.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:34 smithi055 ceph-mon[97033]: pgmap v13440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:33:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:34 smithi059 ceph-mon[97450]: pgmap v13440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:33:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:35.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:35.246740+0000) 2022-09-08T04:33:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:35.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:35.240878+0000) 2022-09-08T04:33:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:36.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:36.246864+0000) 2022-09-08T04:33:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:36.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:36.241093+0000) 2022-09-08T04:33:37.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:36 smithi055 ceph-mon[97033]: pgmap v13441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:33:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:36 smithi059 ceph-mon[97450]: pgmap v13441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:33:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:37.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:37.246991+0000) 2022-09-08T04:33:37.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:37.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:37.241379+0000) 2022-09-08T04:33:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:38.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:38.247175+0000) 2022-09-08T04:33:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:38.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:38.241624+0000) 2022-09-08T04:33:39.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:38 smithi055 ceph-mon[97033]: pgmap v13442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:38 smithi059 ceph-mon[97450]: pgmap v13442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:39.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:39.247371+0000) 2022-09-08T04:33:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:39.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:39.241863+0000) 2022-09-08T04:33:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:40.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:40.247554+0000) 2022-09-08T04:33:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:40.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:40.242114+0000) 2022-09-08T04:33:41.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:40 smithi055 ceph-mon[97033]: pgmap v13443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:40 smithi059 ceph-mon[97450]: pgmap v13443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:41.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:41.247802+0000) 2022-09-08T04:33:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:41.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:41.242350+0000) 2022-09-08T04:33:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:42.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:42.247948+0000) 2022-09-08T04:33:42.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:42.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:42.242571+0000) 2022-09-08T04:33:43.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:42 smithi055 ceph-mon[97033]: pgmap v13444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:33:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:42 smithi059 ceph-mon[97450]: pgmap v13444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:33:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:43.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:43.248107+0000) 2022-09-08T04:33:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:43.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:43.242774+0000) 2022-09-08T04:33:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:44.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:44.248268+0000) 2022-09-08T04:33:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:44.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:44.243028+0000) 2022-09-08T04:33:45.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:44 smithi055 ceph-mon[97033]: pgmap v13445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:44 smithi059 ceph-mon[97450]: pgmap v13445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:45.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:45.248469+0000) 2022-09-08T04:33:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:45.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:45.243213+0000) 2022-09-08T04:33:46.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:33:45.855+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:33:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:46.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:46.248721+0000) 2022-09-08T04:33:46.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:46.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:46.243443+0000) 2022-09-08T04:33:47.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:46 smithi055 ceph-mon[97033]: pgmap v13446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:33:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:46 smithi059 ceph-mon[97450]: pgmap v13446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:33:47.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:47.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:47.248900+0000) 2022-09-08T04:33:47.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:47.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:47.243620+0000) 2022-09-08T04:33:48.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:48.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:48.243846+0000) 2022-09-08T04:33:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:48.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:48.249061+0000) 2022-09-08T04:33:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:48 smithi055 ceph-mon[97033]: pgmap v13447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:48 smithi059 ceph-mon[97450]: pgmap v13447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:49.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:49.249268+0000) 2022-09-08T04:33:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:49.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:49.244088+0000) 2022-09-08T04:33:50.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:50.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:50.249438+0000) 2022-09-08T04:33:50.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:50.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:50.244305+0000) 2022-09-08T04:33:51.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:50 smithi055 ceph-mon[97033]: pgmap v13448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:50 smithi059 ceph-mon[97450]: pgmap v13448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:51.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:51.249685+0000) 2022-09-08T04:33:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:51.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:51.244516+0000) 2022-09-08T04:33:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:52.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:52.249888+0000) 2022-09-08T04:33:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:52.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:52.244755+0000) 2022-09-08T04:33:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:52 smithi055 ceph-mon[97033]: pgmap v13449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:33:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:52 smithi059 ceph-mon[97450]: pgmap v13449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:33:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:53.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:53.250100+0000) 2022-09-08T04:33:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:53.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:53.244996+0000) 2022-09-08T04:33:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:54.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:54.250359+0000) 2022-09-08T04:33:54.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:54.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:54.245214+0000) 2022-09-08T04:33:55.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:54 smithi055 ceph-mon[97033]: pgmap v13450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:54 smithi059 ceph-mon[97450]: pgmap v13450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:55.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:55.250597+0000) 2022-09-08T04:33:55.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:55.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:55.245399+0000) 2022-09-08T04:33:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:56.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:56.250795+0000) 2022-09-08T04:33:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:56.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:56.245612+0000) 2022-09-08T04:33:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:56 smithi055 ceph-mon[97033]: pgmap v13451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:33:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:56 smithi059 ceph-mon[97450]: pgmap v13451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:33:57.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:57.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:57.250958+0000) 2022-09-08T04:33:57.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:57.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:57.245884+0000) 2022-09-08T04:33:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:58.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:58.251164+0000) 2022-09-08T04:33:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:58.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:58.246052+0000) 2022-09-08T04:33:59.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:33:58 smithi055 ceph-mon[97033]: pgmap v13452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:33:58 smithi059 ceph-mon[97450]: pgmap v13452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:33:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:33:59.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:59.251409+0000) 2022-09-08T04:33:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:33:59.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:33:59.246284+0000) 2022-09-08T04:34:00.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:00.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:00.251678+0000) 2022-09-08T04:34:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:00.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:00.246500+0000) 2022-09-08T04:34:01.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:34:00.870+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:34:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:00 smithi055 ceph-mon[97033]: pgmap v13453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:00 smithi055 ceph-mon[97033]: pgmap v13454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 8 op/s 2022-09-08T04:34:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:00 smithi055 ceph-mon[97033]: pgmap v13455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:34:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:34:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:00 smithi059 ceph-mon[97450]: pgmap v13453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:00 smithi059 ceph-mon[97450]: pgmap v13454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 8 op/s 2022-09-08T04:34:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:00 smithi059 ceph-mon[97450]: pgmap v13455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:34:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:34:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:01.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:01.251846+0000) 2022-09-08T04:34:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:01.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:01.246732+0000) 2022-09-08T04:34:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:02.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:02.252006+0000) 2022-09-08T04:34:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:02.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:02.246976+0000) 2022-09-08T04:34:03.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:02 smithi055 ceph-mon[97033]: pgmap v13456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:34:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:02 smithi059 ceph-mon[97450]: pgmap v13456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:34:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:03.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:03.252185+0000) 2022-09-08T04:34:03.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:03.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:03.247163+0000) 2022-09-08T04:34:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:04.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:04.252431+0000) 2022-09-08T04:34:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:04.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:04.247405+0000) 2022-09-08T04:34:05.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:04 smithi055 ceph-mon[97033]: pgmap v13457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:34:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:04 smithi059 ceph-mon[97450]: pgmap v13457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:34:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:05.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:05.252627+0000) 2022-09-08T04:34:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:05.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:05.247663+0000) 2022-09-08T04:34:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:34:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:34:06.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:06.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:06.252785+0000) 2022-09-08T04:34:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:06.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:06.247813+0000) 2022-09-08T04:34:07.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:06 smithi055 ceph-mon[97033]: pgmap v13458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T04:34:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:34:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:06 smithi059 ceph-mon[97450]: pgmap v13458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T04:34:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:34:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:07.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:07.252930+0000) 2022-09-08T04:34:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:07.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:07.248012+0000) 2022-09-08T04:34:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:08.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:08.253088+0000) 2022-09-08T04:34:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:08.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:08.248264+0000) 2022-09-08T04:34:09.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:08 smithi055 ceph-mon[97033]: pgmap v13459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T04:34:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:08 smithi059 ceph-mon[97450]: pgmap v13459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T04:34:09.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:09.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:09.253335+0000) 2022-09-08T04:34:09.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:09.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:09.248504+0000) 2022-09-08T04:34:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:09 smithi055 ceph-mon[97033]: osdmap e922: 12 total, 11 up, 11 in 2022-09-08T04:34:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:09 smithi059 ceph-mon[97450]: osdmap e922: 12 total, 11 up, 11 in 2022-09-08T04:34:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:10.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:10.253597+0000) 2022-09-08T04:34:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:10.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:10.248774+0000) 2022-09-08T04:34:11.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:10 smithi055 ceph-mon[97033]: pgmap v13461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:34:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:10 smithi059 ceph-mon[97450]: pgmap v13461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:34:11.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:11.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:11.253864+0000) 2022-09-08T04:34:11.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:11.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:11.248981+0000) 2022-09-08T04:34:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:12.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:12.254041+0000) 2022-09-08T04:34:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:12.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:12.249206+0000) 2022-09-08T04:34:13.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:12 smithi055 ceph-mon[97033]: pgmap v13462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:34:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:12 smithi059 ceph-mon[97450]: pgmap v13462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:34:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:13.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:13.254310+0000) 2022-09-08T04:34:13.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:13.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:13.249436+0000) 2022-09-08T04:34:14.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:14.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:14.254559+0000) 2022-09-08T04:34:14.693 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:14.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:14.249721+0000) 2022-09-08T04:34:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:14 smithi055 ceph-mon[97033]: pgmap v13463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:34:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:14 smithi055 ceph-mon[97033]: osdmap e923: 12 total, 11 up, 11 in 2022-09-08T04:34:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:14 smithi059 ceph-mon[97450]: pgmap v13463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:34:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:14 smithi059 ceph-mon[97450]: osdmap e923: 12 total, 11 up, 11 in 2022-09-08T04:34:15.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:15.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:15.254794+0000) 2022-09-08T04:34:15.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:15.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:15.249903+0000) 2022-09-08T04:34:16.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:34:15.873+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:34:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:16.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:16.254971+0000) 2022-09-08T04:34:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:16.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:16.250120+0000) 2022-09-08T04:34:17.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:16 smithi055 ceph-mon[97033]: pgmap v13465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:34:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:16 smithi059 ceph-mon[97450]: pgmap v13465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:34:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:17.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:17.255199+0000) 2022-09-08T04:34:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:17.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:17.250335+0000) 2022-09-08T04:34:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:18.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:18.250574+0000) 2022-09-08T04:34:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:18.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:18.255431+0000) 2022-09-08T04:34:19.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:18 smithi055 ceph-mon[97033]: pgmap v13466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T04:34:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:18 smithi059 ceph-mon[97450]: pgmap v13466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T04:34:19.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:19.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:19.250822+0000) 2022-09-08T04:34:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:19.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:19.255707+0000) 2022-09-08T04:34:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:20.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:20.250999+0000) 2022-09-08T04:34:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:20.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:20.255904+0000) 2022-09-08T04:34:21.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:20 smithi055 ceph-mon[97033]: pgmap v13467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:34:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:20 smithi059 ceph-mon[97450]: pgmap v13467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:34:21.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:21.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:21.251225+0000) 2022-09-08T04:34:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:21.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:21.256115+0000) 2022-09-08T04:34:22.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:22.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:22.251446+0000) 2022-09-08T04:34:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:22.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:22.256322+0000) 2022-09-08T04:34:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:22 smithi055 ceph-mon[97033]: pgmap v13468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:34:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:22 smithi059 ceph-mon[97450]: pgmap v13468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:34:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:23.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:23.256488+0000) 2022-09-08T04:34:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:23.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:23.251705+0000) 2022-09-08T04:34:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:24.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:24.256733+0000) 2022-09-08T04:34:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:24.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:24.251909+0000) 2022-09-08T04:34:25.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:24 smithi055 ceph-mon[97033]: pgmap v13469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:34:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:24 smithi059 ceph-mon[97450]: pgmap v13469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:34:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:25.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:25.256969+0000) 2022-09-08T04:34:25.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:25.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:25.252052+0000) 2022-09-08T04:34:26.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:26.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:26.252232+0000) 2022-09-08T04:34:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:26.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:26.257169+0000) 2022-09-08T04:34:27.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:26 smithi055 ceph-mon[97033]: pgmap v13470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 92 B/s wr, 8 op/s 2022-09-08T04:34:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:26 smithi059 ceph-mon[97450]: pgmap v13470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 92 B/s wr, 8 op/s 2022-09-08T04:34:27.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:27.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:27.252471+0000) 2022-09-08T04:34:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:27.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:27.257366+0000) 2022-09-08T04:34:28.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:28.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:28.252680+0000) 2022-09-08T04:34:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:28.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:28.257520+0000) 2022-09-08T04:34:29.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:28 smithi055 ceph-mon[97033]: pgmap v13471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:28 smithi059 ceph-mon[97450]: pgmap v13471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:29.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:29.252900+0000) 2022-09-08T04:34:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:29.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:29.257756+0000) 2022-09-08T04:34:30.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:30.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:30.253146+0000) 2022-09-08T04:34:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:30.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:30.257895+0000) 2022-09-08T04:34:31.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:34:30.881+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:34:31.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:30 smithi055 ceph-mon[97033]: pgmap v13472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:34:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:30 smithi059 ceph-mon[97450]: pgmap v13472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:34:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:31.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:31.258073+0000) 2022-09-08T04:34:31.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:31.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:31.253374+0000) 2022-09-08T04:34:32.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:32.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:32.258295+0000) 2022-09-08T04:34:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:32.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:32.253627+0000) 2022-09-08T04:34:33.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:32 smithi055 ceph-mon[97033]: pgmap v13473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:32 smithi059 ceph-mon[97450]: pgmap v13473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:33.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:33.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:33.258536+0000) 2022-09-08T04:34:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:33.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:33.253862+0000) 2022-09-08T04:34:34.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:34.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:34.258747+0000) 2022-09-08T04:34:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:34.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:34.254039+0000) 2022-09-08T04:34:35.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:34 smithi055 ceph-mon[97033]: pgmap v13474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:34 smithi059 ceph-mon[97450]: pgmap v13474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:35.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:35.258878+0000) 2022-09-08T04:34:35.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:35.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:35.254242+0000) 2022-09-08T04:34:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:36.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:36.259041+0000) 2022-09-08T04:34:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:36.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:36.254465+0000) 2022-09-08T04:34:37.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:36 smithi055 ceph-mon[97033]: pgmap v13475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:34:37.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:36 smithi059 ceph-mon[97450]: pgmap v13475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:34:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:37.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:37.259265+0000) 2022-09-08T04:34:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:37.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:37.254689+0000) 2022-09-08T04:34:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:38.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:38.259523+0000) 2022-09-08T04:34:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:38.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:38.254958+0000) 2022-09-08T04:34:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:38 smithi055 ceph-mon[97033]: pgmap v13476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:38 smithi059 ceph-mon[97450]: pgmap v13476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:39.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:39.259834+0000) 2022-09-08T04:34:39.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:39.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:39.255178+0000) 2022-09-08T04:34:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:40.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:40.260007+0000) 2022-09-08T04:34:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:40.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:40.255427+0000) 2022-09-08T04:34:41.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:40 smithi055 ceph-mon[97033]: pgmap v13477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:34:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:40 smithi059 ceph-mon[97450]: pgmap v13477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:34:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:41.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:41.260233+0000) 2022-09-08T04:34:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:41.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:41.255700+0000) 2022-09-08T04:34:42.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:42.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:42.260460+0000) 2022-09-08T04:34:42.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:42.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:42.255843+0000) 2022-09-08T04:34:43.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:42 smithi055 ceph-mon[97033]: pgmap v13478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:34:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:42 smithi059 ceph-mon[97450]: pgmap v13478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:34:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:43.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:43.260723+0000) 2022-09-08T04:34:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:43.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:43.256054+0000) 2022-09-08T04:34:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:44.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:44.261002+0000) 2022-09-08T04:34:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:44.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:44.256280+0000) 2022-09-08T04:34:45.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:44 smithi055 ceph-mon[97033]: pgmap v13479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:34:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:44 smithi059 ceph-mon[97450]: pgmap v13479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:34:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:45.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:45.261197+0000) 2022-09-08T04:34:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:45.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:45.256534+0000) 2022-09-08T04:34:46.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:34:45.884+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:34:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:46.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:46.261422+0000) 2022-09-08T04:34:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:46.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:46.256764+0000) 2022-09-08T04:34:47.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:46 smithi055 ceph-mon[97033]: pgmap v13480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:34:47.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:46 smithi059 ceph-mon[97450]: pgmap v13480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:34:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:47.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:47.261692+0000) 2022-09-08T04:34:47.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:47.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:47.256990+0000) 2022-09-08T04:34:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:48.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:48.261861+0000) 2022-09-08T04:34:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:48.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:48.257214+0000) 2022-09-08T04:34:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:48 smithi059 ceph-mon[97450]: pgmap v13481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:34:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:48 smithi055 ceph-mon[97033]: pgmap v13481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:34:49.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:49.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:49.262013+0000) 2022-09-08T04:34:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:49.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:49.257427+0000) 2022-09-08T04:34:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:50.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:50.262165+0000) 2022-09-08T04:34:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:50.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:50.257572+0000) 2022-09-08T04:34:51.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:50 smithi055 ceph-mon[97033]: pgmap v13482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:34:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:50 smithi059 ceph-mon[97450]: pgmap v13482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:34:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:51.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:51.262309+0000) 2022-09-08T04:34:51.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:51.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:51.257787+0000) 2022-09-08T04:34:52.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:52.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:52.262455+0000) 2022-09-08T04:34:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:52.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:52.258048+0000) 2022-09-08T04:34:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:52 smithi055 ceph-mon[97033]: pgmap v13483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:52 smithi059 ceph-mon[97450]: pgmap v13483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:53.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:53.262673+0000) 2022-09-08T04:34:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:53.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:53.258312+0000) 2022-09-08T04:34:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:54.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:54.262883+0000) 2022-09-08T04:34:54.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:54.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:54.258490+0000) 2022-09-08T04:34:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:54 smithi055 ceph-mon[97033]: pgmap v13484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:54 smithi059 ceph-mon[97450]: pgmap v13484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:55.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:55.263100+0000) 2022-09-08T04:34:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:55.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:55.258715+0000) 2022-09-08T04:34:56.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:56.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:56.263329+0000) 2022-09-08T04:34:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:56.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:56.258915+0000) 2022-09-08T04:34:57.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:56 smithi059 ceph-mon[97450]: pgmap v13485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:34:57.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:56 smithi055 ceph-mon[97033]: pgmap v13485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:34:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:57.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:57.263554+0000) 2022-09-08T04:34:57.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:57.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:57.259099+0000) 2022-09-08T04:34:58.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:58.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:58.259316+0000) 2022-09-08T04:34:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:58.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:58.263796+0000) 2022-09-08T04:34:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:34:58 smithi059 ceph-mon[97450]: pgmap v13486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:59.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:34:58 smithi055 ceph-mon[97033]: pgmap v13486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:34:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:34:59.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:59.263998+0000) 2022-09-08T04:34:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:34:59.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:34:59.259538+0000) 2022-09-08T04:35:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:00.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:00.264235+0000) 2022-09-08T04:35:00.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:00.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:00.259768+0000) 2022-09-08T04:35:01.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:35:00.899+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:35:01.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:00 smithi055 ceph-mon[97033]: pgmap v13487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:35:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:35:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:35:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:35:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:00 smithi059 ceph-mon[97450]: pgmap v13487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:35:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:35:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:35:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:35:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:01.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:01.264446+0000) 2022-09-08T04:35:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:01.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:01.259979+0000) 2022-09-08T04:35:02.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:02.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:02.264600+0000) 2022-09-08T04:35:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:02.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:02.260199+0000) 2022-09-08T04:35:03.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:02 smithi055 ceph-mon[97033]: pgmap v13488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:02 smithi059 ceph-mon[97450]: pgmap v13488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:03.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:03.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:03.264809+0000) 2022-09-08T04:35:03.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:03.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:03.260416+0000) 2022-09-08T04:35:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:04.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:04.260712+0000) 2022-09-08T04:35:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:04.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:04.265006+0000) 2022-09-08T04:35:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:04 smithi055 ceph-mon[97033]: pgmap v13489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:04 smithi059 ceph-mon[97450]: pgmap v13489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:05.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:05.265194+0000) 2022-09-08T04:35:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:05.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:05.260879+0000) 2022-09-08T04:35:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:35:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:35:06.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:06.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:06.265473+0000) 2022-09-08T04:35:06.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:06.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:06.261127+0000) 2022-09-08T04:35:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:06 smithi059 ceph-mon[97450]: pgmap v13490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:35:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:35:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:06 smithi055 ceph-mon[97033]: pgmap v13490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:35:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:35:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:07.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:07.261308+0000) 2022-09-08T04:35:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:07.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:07.265697+0000) 2022-09-08T04:35:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:08.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:08.265904+0000) 2022-09-08T04:35:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:08.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:08.261534+0000) 2022-09-08T04:35:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:08 smithi059 ceph-mon[97450]: pgmap v13491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:35:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:08 smithi055 ceph-mon[97033]: pgmap v13491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:35:09.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:09.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:09.266126+0000) 2022-09-08T04:35:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:09.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:09.261751+0000) 2022-09-08T04:35:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:10 smithi059 ceph-mon[97450]: osdmap e924: 12 total, 11 up, 11 in 2022-09-08T04:35:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:10 smithi055 ceph-mon[97033]: osdmap e924: 12 total, 11 up, 11 in 2022-09-08T04:35:10.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:10.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:10.261966+0000) 2022-09-08T04:35:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:10.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:10.266335+0000) 2022-09-08T04:35:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:11 smithi059 ceph-mon[97450]: pgmap v13493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:35:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:11 smithi055 ceph-mon[97033]: pgmap v13493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:35:11.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:11.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:11.266534+0000) 2022-09-08T04:35:11.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:11.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:11.262178+0000) 2022-09-08T04:35:12.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:12.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:12.262393+0000) 2022-09-08T04:35:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:12.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:12.266791+0000) 2022-09-08T04:35:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:13 smithi059 ceph-mon[97450]: pgmap v13494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:35:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:13 smithi055 ceph-mon[97033]: pgmap v13494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:35:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:13.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:13.266983+0000) 2022-09-08T04:35:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:13.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:13.262648+0000) 2022-09-08T04:35:14.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:14.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:14.267208+0000) 2022-09-08T04:35:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:14.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:14.262877+0000) 2022-09-08T04:35:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:15 smithi059 ceph-mon[97450]: pgmap v13495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:35:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:15 smithi059 ceph-mon[97450]: osdmap e925: 12 total, 11 up, 11 in 2022-09-08T04:35:15.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:15 smithi055 ceph-mon[97033]: pgmap v13495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:35:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:15 smithi055 ceph-mon[97033]: osdmap e925: 12 total, 11 up, 11 in 2022-09-08T04:35:15.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:15.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:15.263066+0000) 2022-09-08T04:35:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:15.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:15.267440+0000) 2022-09-08T04:35:16.262 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:35:15.903+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:35:16.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:16.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:16.267602+0000) 2022-09-08T04:35:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:16.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:16.263298+0000) 2022-09-08T04:35:17.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:17 smithi059 ceph-mon[97450]: pgmap v13497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:35:17.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:17 smithi055 ceph-mon[97033]: pgmap v13497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:35:17.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:17.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:17.267801+0000) 2022-09-08T04:35:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:17.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:17.263514+0000) 2022-09-08T04:35:18.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:18.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:18.267942+0000) 2022-09-08T04:35:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:18.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:18.263741+0000) 2022-09-08T04:35:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:19 smithi059 ceph-mon[97450]: pgmap v13498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T04:35:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:19 smithi055 ceph-mon[97033]: pgmap v13498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T04:35:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:19.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:19.268079+0000) 2022-09-08T04:35:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:19.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:19.264010+0000) 2022-09-08T04:35:20.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:20.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:20.268294+0000) 2022-09-08T04:35:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:20.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:20.264230+0000) 2022-09-08T04:35:21.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:20 smithi055 ceph-mon[97033]: pgmap v13499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:35:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:20 smithi059 ceph-mon[97450]: pgmap v13499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:35:21.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:21.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:21.268546+0000) 2022-09-08T04:35:21.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:21.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:21.264447+0000) 2022-09-08T04:35:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:22.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:22.268782+0000) 2022-09-08T04:35:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:22.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:22.264648+0000) 2022-09-08T04:35:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:22 smithi059 ceph-mon[97450]: pgmap v13500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:35:23.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:22 smithi055 ceph-mon[97033]: pgmap v13500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:35:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:23.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:23.268958+0000) 2022-09-08T04:35:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:23.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:23.264878+0000) 2022-09-08T04:35:24.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:24.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:24.269181+0000) 2022-09-08T04:35:24.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:24.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:24.265032+0000) 2022-09-08T04:35:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:24 smithi059 ceph-mon[97450]: pgmap v13501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:35:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:25.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:25.269401+0000) 2022-09-08T04:35:25.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:25.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:25.265190+0000) 2022-09-08T04:35:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:24 smithi055 ceph-mon[97033]: pgmap v13501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:35:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:26.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:26.269621+0000) 2022-09-08T04:35:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:26.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:26.265377+0000) 2022-09-08T04:35:27.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:26 smithi059 ceph-mon[97450]: pgmap v13502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 92 B/s wr, 8 op/s 2022-09-08T04:35:27.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:26 smithi055 ceph-mon[97033]: pgmap v13502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 92 B/s wr, 8 op/s 2022-09-08T04:35:27.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:27.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:27.269763+0000) 2022-09-08T04:35:27.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:27.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:27.265553+0000) 2022-09-08T04:35:28.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:28.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:28.269968+0000) 2022-09-08T04:35:28.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:28.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:28.265688+0000) 2022-09-08T04:35:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:28 smithi059 ceph-mon[97450]: pgmap v13503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:28 smithi055 ceph-mon[97033]: pgmap v13503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:29.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:29.270247+0000) 2022-09-08T04:35:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:29.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:29.265959+0000) 2022-09-08T04:35:30.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:30.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:30.270522+0000) 2022-09-08T04:35:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:30.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:30.266116+0000) 2022-09-08T04:35:31.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:30 smithi055 ceph-mon[97033]: pgmap v13504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:35:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:35:31.266 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:35:30.919+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:35:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:30 smithi059 ceph-mon[97450]: pgmap v13504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:35:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:35:31.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:31.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:31.266329+0000) 2022-09-08T04:35:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:31.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:31.270750+0000) 2022-09-08T04:35:32.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:31 smithi055 ceph-mon[97033]: pgmap v13505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:35:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:31 smithi055 ceph-mon[97033]: pgmap v13506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:35:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:31 smithi059 ceph-mon[97450]: pgmap v13505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:35:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:31 smithi059 ceph-mon[97450]: pgmap v13506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:35:32.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:32.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:32.270920+0000) 2022-09-08T04:35:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:32.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:32.266513+0000) 2022-09-08T04:35:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:33.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:33.271129+0000) 2022-09-08T04:35:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:33.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:33.266740+0000) 2022-09-08T04:35:34.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:33 smithi055 ceph-mon[97033]: pgmap v13507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:35:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:33 smithi059 ceph-mon[97450]: pgmap v13507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:35:34.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:34.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:34.271338+0000) 2022-09-08T04:35:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:34.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:34.266887+0000) 2022-09-08T04:35:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:35.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:35.271524+0000) 2022-09-08T04:35:35.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:35.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:35.267124+0000) 2022-09-08T04:35:36.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:35 smithi055 ceph-mon[97033]: pgmap v13508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:35:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:35 smithi059 ceph-mon[97450]: pgmap v13508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:35:36.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:36.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:36.267344+0000) 2022-09-08T04:35:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:36.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:36.271778+0000) 2022-09-08T04:35:37.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:37.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:37.267610+0000) 2022-09-08T04:35:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:37.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:37.272014+0000) 2022-09-08T04:35:38.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:37 smithi055 ceph-mon[97033]: pgmap v13509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:35:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:37 smithi059 ceph-mon[97450]: pgmap v13509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:35:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:38.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:38.267860+0000) 2022-09-08T04:35:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:38.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:38.272276+0000) 2022-09-08T04:35:39.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:39.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:39.268101+0000) 2022-09-08T04:35:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:39.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:39.272538+0000) 2022-09-08T04:35:40.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:39 smithi055 ceph-mon[97033]: pgmap v13510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:35:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:39 smithi059 ceph-mon[97450]: pgmap v13510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:35:40.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:40.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:40.268322+0000) 2022-09-08T04:35:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:40.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:40.272758+0000) 2022-09-08T04:35:41.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:41.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:41.268521+0000) 2022-09-08T04:35:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:41.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:41.272957+0000) 2022-09-08T04:35:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:41 smithi055 ceph-mon[97033]: pgmap v13511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:35:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:41 smithi059 ceph-mon[97450]: pgmap v13511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:35:42.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:42.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:42.273142+0000) 2022-09-08T04:35:42.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:42.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:42.268731+0000) 2022-09-08T04:35:43.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:43.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:43.273364+0000) 2022-09-08T04:35:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:43.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:43.268961+0000) 2022-09-08T04:35:44.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:43 smithi055 ceph-mon[97033]: pgmap v13512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:44.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:43 smithi059 ceph-mon[97450]: pgmap v13512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:44.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:44.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:44.273620+0000) 2022-09-08T04:35:44.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:44.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:44.269155+0000) 2022-09-08T04:35:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:45.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:45.273878+0000) 2022-09-08T04:35:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:45.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:45.269386+0000) 2022-09-08T04:35:46.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:45 smithi055 ceph-mon[97033]: pgmap v13513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:46.269 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:35:45.922+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:35:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:45 smithi059 ceph-mon[97450]: pgmap v13513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:46.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:46.274034+0000) 2022-09-08T04:35:46.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:46.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:46.269631+0000) 2022-09-08T04:35:47.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:47.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:47.274227+0000) 2022-09-08T04:35:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:47.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:47.269877+0000) 2022-09-08T04:35:48.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:47 smithi055 ceph-mon[97033]: pgmap v13514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:35:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:47 smithi059 ceph-mon[97450]: pgmap v13514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:35:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:48.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:48.274405+0000) 2022-09-08T04:35:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:48.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:48.270051+0000) 2022-09-08T04:35:49.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:49.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:49.274669+0000) 2022-09-08T04:35:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:49.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:49.270259+0000) 2022-09-08T04:35:50.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:49 smithi055 ceph-mon[97033]: pgmap v13515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:49 smithi059 ceph-mon[97450]: pgmap v13515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:50.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:50.274862+0000) 2022-09-08T04:35:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:50.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:50.270495+0000) 2022-09-08T04:35:51.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:51.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:51.275013+0000) 2022-09-08T04:35:51.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:51.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:51.270731+0000) 2022-09-08T04:35:52.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:51 smithi055 ceph-mon[97033]: pgmap v13516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:35:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:51 smithi059 ceph-mon[97450]: pgmap v13516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:35:52.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:52.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:52.275174+0000) 2022-09-08T04:35:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:52.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:52.270982+0000) 2022-09-08T04:35:53.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:53.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:53.275385+0000) 2022-09-08T04:35:53.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:53.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:53.271236+0000) 2022-09-08T04:35:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:53 smithi055 ceph-mon[97033]: pgmap v13517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:53 smithi059 ceph-mon[97450]: pgmap v13517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:54.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:54.275631+0000) 2022-09-08T04:35:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:54.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:54.271434+0000) 2022-09-08T04:35:55.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:55.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:55.275842+0000) 2022-09-08T04:35:55.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:55.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:55.271696+0000) 2022-09-08T04:35:56.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:55 smithi055 ceph-mon[97033]: pgmap v13518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:55 smithi059 ceph-mon[97450]: pgmap v13518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:35:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:56.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:56.275988+0000) 2022-09-08T04:35:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:56.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:56.271960+0000) 2022-09-08T04:35:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:57.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:57.276127+0000) 2022-09-08T04:35:57.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:57.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:57.272200+0000) 2022-09-08T04:35:58.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:57 smithi055 ceph-mon[97033]: pgmap v13519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:35:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:57 smithi059 ceph-mon[97450]: pgmap v13519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:35:58.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:58.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:58.276348+0000) 2022-09-08T04:35:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:58.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:58.272423+0000) 2022-09-08T04:35:59.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:35:59.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:59.276615+0000) 2022-09-08T04:35:59.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:35:59.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:35:59.272697+0000) 2022-09-08T04:36:00.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:35:59 smithi055 ceph-mon[97033]: pgmap v13520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:35:59 smithi059 ceph-mon[97450]: pgmap v13520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:00.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:00.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:00.276867+0000) 2022-09-08T04:36:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:00.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:00.272921+0000) 2022-09-08T04:36:01.272 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:36:00.936+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:36:01.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:01.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:01.277022+0000) 2022-09-08T04:36:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:01.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:01.273084+0000) 2022-09-08T04:36:02.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:01 smithi055 ceph-mon[97033]: pgmap v13521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:36:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:01 smithi059 ceph-mon[97450]: pgmap v13521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:36:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:02.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:02.277220+0000) 2022-09-08T04:36:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:02.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:02.273314+0000) 2022-09-08T04:36:03.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:03.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:03.277460+0000) 2022-09-08T04:36:03.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:03.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:03.273537+0000) 2022-09-08T04:36:04.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:03 smithi055 ceph-mon[97033]: pgmap v13522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:03 smithi059 ceph-mon[97450]: pgmap v13522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:04.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:04.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:04.277682+0000) 2022-09-08T04:36:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:04.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:04.273761+0000) 2022-09-08T04:36:05.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:05.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:05.277862+0000) 2022-09-08T04:36:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:05.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:05.273888+0000) 2022-09-08T04:36:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:05 smithi055 ceph-mon[97033]: pgmap v13523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:36:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:05 smithi059 ceph-mon[97450]: pgmap v13523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:36:06.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:06.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:06.278041+0000) 2022-09-08T04:36:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:06.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:06.274084+0000) 2022-09-08T04:36:07.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:36:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:36:07.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:07.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:07.278196+0000) 2022-09-08T04:36:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:07.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:07.274303+0000) 2022-09-08T04:36:08.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:07 smithi055 ceph-mon[97033]: pgmap v13524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:36:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:07 smithi059 ceph-mon[97450]: pgmap v13524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:36:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:08.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:08.278407+0000) 2022-09-08T04:36:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:08.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:08.274512+0000) 2022-09-08T04:36:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:09.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:09.278558+0000) 2022-09-08T04:36:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:09.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:09.274819+0000) 2022-09-08T04:36:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:09 smithi055 ceph-mon[97033]: pgmap v13525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:36:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:09 smithi055 ceph-mon[97033]: osdmap e926: 12 total, 11 up, 11 in 2022-09-08T04:36:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:09 smithi059 ceph-mon[97450]: pgmap v13525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:36:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:09 smithi059 ceph-mon[97450]: osdmap e926: 12 total, 11 up, 11 in 2022-09-08T04:36:10.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:10.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:10.278795+0000) 2022-09-08T04:36:10.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:10.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:10.275051+0000) 2022-09-08T04:36:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:11.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:11.278982+0000) 2022-09-08T04:36:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:11.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:11.275234+0000) 2022-09-08T04:36:12.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:11 smithi055 ceph-mon[97033]: pgmap v13527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:36:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:11 smithi059 ceph-mon[97450]: pgmap v13527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:36:12.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:12.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:12.279191+0000) 2022-09-08T04:36:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:12.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:12.275369+0000) 2022-09-08T04:36:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:13.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:13.279360+0000) 2022-09-08T04:36:13.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:13.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:13.275665+0000) 2022-09-08T04:36:14.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:13 smithi055 ceph-mon[97033]: pgmap v13528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:36:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:13 smithi059 ceph-mon[97450]: pgmap v13528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:36:14.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:14.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:14.279560+0000) 2022-09-08T04:36:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:14.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:14.275865+0000) 2022-09-08T04:36:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:15.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:15.279753+0000) 2022-09-08T04:36:15.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:15.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:15.276123+0000) 2022-09-08T04:36:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:15 smithi055 ceph-mon[97033]: pgmap v13529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:36:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:15 smithi055 ceph-mon[97033]: osdmap e927: 12 total, 11 up, 11 in 2022-09-08T04:36:16.276 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:36:15.939+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:36:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:15 smithi059 ceph-mon[97450]: pgmap v13529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:36:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:15 smithi059 ceph-mon[97450]: osdmap e927: 12 total, 11 up, 11 in 2022-09-08T04:36:16.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:16.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:16.279941+0000) 2022-09-08T04:36:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:16.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:16.276324+0000) 2022-09-08T04:36:17.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:17.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:17.280179+0000) 2022-09-08T04:36:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:17.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:17.276545+0000) 2022-09-08T04:36:18.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:17 smithi055 ceph-mon[97033]: pgmap v13531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:36:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:17 smithi059 ceph-mon[97450]: pgmap v13531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:36:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:18.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:18.280440+0000) 2022-09-08T04:36:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:18.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:18.276695+0000) 2022-09-08T04:36:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:19.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:19.280620+0000) 2022-09-08T04:36:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:19.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:19.276922+0000) 2022-09-08T04:36:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:19 smithi055 ceph-mon[97033]: pgmap v13532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-08T04:36:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:19 smithi059 ceph-mon[97450]: pgmap v13532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-08T04:36:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:20.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:20.280831+0000) 2022-09-08T04:36:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:20.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:20.277134+0000) 2022-09-08T04:36:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:21.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:21.280997+0000) 2022-09-08T04:36:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:21.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:21.277348+0000) 2022-09-08T04:36:22.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:21 smithi055 ceph-mon[97033]: pgmap v13533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:36:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:21 smithi059 ceph-mon[97450]: pgmap v13533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:36:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:22.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:22.281121+0000) 2022-09-08T04:36:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:22.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:22.277575+0000) 2022-09-08T04:36:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:23.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:23.281303+0000) 2022-09-08T04:36:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:23.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:23.277806+0000) 2022-09-08T04:36:24.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:23 smithi055 ceph-mon[97033]: pgmap v13534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:36:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:23 smithi059 ceph-mon[97450]: pgmap v13534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:36:24.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:24.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:24.278023+0000) 2022-09-08T04:36:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:24.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:24.281502+0000) 2022-09-08T04:36:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:25.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:25.281722+0000) 2022-09-08T04:36:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:25.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:25.278226+0000) 2022-09-08T04:36:26.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:25 smithi055 ceph-mon[97033]: pgmap v13535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:36:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:25 smithi059 ceph-mon[97450]: pgmap v13535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:36:26.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:26.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:26.281890+0000) 2022-09-08T04:36:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:26.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:26.278452+0000) 2022-09-08T04:36:27.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:27.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:27.282021+0000) 2022-09-08T04:36:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:27.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:27.278729+0000) 2022-09-08T04:36:28.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:27 smithi055 ceph-mon[97033]: pgmap v13536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 90 B/s wr, 8 op/s 2022-09-08T04:36:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:27 smithi059 ceph-mon[97450]: pgmap v13536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 90 B/s wr, 8 op/s 2022-09-08T04:36:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:28.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:28.282206+0000) 2022-09-08T04:36:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:28.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:28.278928+0000) 2022-09-08T04:36:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:29.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:29.282400+0000) 2022-09-08T04:36:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:29.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:29.279155+0000) 2022-09-08T04:36:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:29 smithi055 ceph-mon[97033]: pgmap v13537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:29 smithi059 ceph-mon[97450]: pgmap v13537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:30.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:30.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:30.282618+0000) 2022-09-08T04:36:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:30.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:30.279359+0000) 2022-09-08T04:36:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:36:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:36:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:36:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:36:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:36:31.279 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:36:30.953+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:36:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:36:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:36:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:36:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:36:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:36:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:31.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:31.282792+0000) 2022-09-08T04:36:31.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:31.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:31.279578+0000) 2022-09-08T04:36:32.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:31 smithi055 ceph-mon[97033]: pgmap v13538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:36:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:31 smithi059 ceph-mon[97450]: pgmap v13538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:36:32.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:32.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:32.282946+0000) 2022-09-08T04:36:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:32.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:32.279787+0000) 2022-09-08T04:36:33.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:33.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:33.283117+0000) 2022-09-08T04:36:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:33.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:33.279924+0000) 2022-09-08T04:36:34.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:33 smithi055 ceph-mon[97033]: pgmap v13539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:36:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:36:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:36:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:36:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:34.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:33 smithi059 ceph-mon[97450]: pgmap v13539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:36:34.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:36:34.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:36:34.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:36:34.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:34.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:34.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:34.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:34.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:34.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:34.283319+0000) 2022-09-08T04:36:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:34.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:34.280160+0000) 2022-09-08T04:36:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:34 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T04:36:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:34 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T04:36:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:35.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:35.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:34 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T04:36:35.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:34 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T04:36:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:34 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T04:36:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:34 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T04:36:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:35.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:35.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:36:35.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:34 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T04:36:35.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:34 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T04:36:35.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:35.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:35.283532+0000) 2022-09-08T04:36:35.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:35.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:35.280403+0000) 2022-09-08T04:36:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:35 smithi055 ceph-mon[97033]: pgmap v13540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:35 smithi059 ceph-mon[97450]: pgmap v13540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:36.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:36.283783+0000) 2022-09-08T04:36:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:36.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:36.280630+0000) 2022-09-08T04:36:37.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:37.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:37.284013+0000) 2022-09-08T04:36:37.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:37.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:37.280812+0000) 2022-09-08T04:36:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:37 smithi055 ceph-mon[97033]: pgmap v13541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:36:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:37 smithi059 ceph-mon[97450]: pgmap v13541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:36:38.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:38.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:38.284164+0000) 2022-09-08T04:36:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:38.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:38.281095+0000) 2022-09-08T04:36:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:39.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:39.284324+0000) 2022-09-08T04:36:39.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:39.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:39.281344+0000) 2022-09-08T04:36:40.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:39 smithi055 ceph-mon[97033]: pgmap v13542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:39 smithi059 ceph-mon[97450]: pgmap v13542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:40.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:40.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:40.284525+0000) 2022-09-08T04:36:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:40.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:40.281542+0000) 2022-09-08T04:36:41.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:41.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:41.284713+0000) 2022-09-08T04:36:41.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:41.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:41.281681+0000) 2022-09-08T04:36:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:41 smithi055 ceph-mon[97033]: pgmap v13543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:36:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:41 smithi059 ceph-mon[97450]: pgmap v13543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:36:42.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:42.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:42.284855+0000) 2022-09-08T04:36:42.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:42.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:42.281855+0000) 2022-09-08T04:36:43.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:43.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:43.285041+0000) 2022-09-08T04:36:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:43.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:43.282087+0000) 2022-09-08T04:36:44.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:43 smithi055 ceph-mon[97033]: pgmap v13544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:43 smithi059 ceph-mon[97450]: pgmap v13544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:44.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:44.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:44.282349+0000) 2022-09-08T04:36:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:44.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:44.285234+0000) 2022-09-08T04:36:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:45.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:45.285496+0000) 2022-09-08T04:36:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:45.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:45.282571+0000) 2022-09-08T04:36:46.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:45 smithi055 ceph-mon[97033]: pgmap v13545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:45 smithi059 ceph-mon[97450]: pgmap v13545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:46.282 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:36:45.954+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:36:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:46.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:46.285743+0000) 2022-09-08T04:36:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:46.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:46.282823+0000) 2022-09-08T04:36:47.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:47.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:47.285909+0000) 2022-09-08T04:36:47.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:47.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:47.283071+0000) 2022-09-08T04:36:48.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:47 smithi055 ceph-mon[97033]: pgmap v13546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:36:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:47 smithi059 ceph-mon[97450]: pgmap v13546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:36:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:48.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:48.286037+0000) 2022-09-08T04:36:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:48.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:48.283320+0000) 2022-09-08T04:36:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:49.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:49.286248+0000) 2022-09-08T04:36:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:49.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:49.283559+0000) 2022-09-08T04:36:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:49 smithi055 ceph-mon[97033]: pgmap v13547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:49 smithi059 ceph-mon[97450]: pgmap v13547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:50.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:50.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:50.286512+0000) 2022-09-08T04:36:50.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:50.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:50.283762+0000) 2022-09-08T04:36:51.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:51.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:51.286787+0000) 2022-09-08T04:36:51.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:51.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:51.283951+0000) 2022-09-08T04:36:52.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:51 smithi055 ceph-mon[97033]: pgmap v13548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:36:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:51 smithi059 ceph-mon[97450]: pgmap v13548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:36:52.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:52.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:52.286965+0000) 2022-09-08T04:36:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:52.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:52.284145+0000) 2022-09-08T04:36:53.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:53.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:53.287180+0000) 2022-09-08T04:36:53.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:53.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:53.284331+0000) 2022-09-08T04:36:54.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:53 smithi055 ceph-mon[97033]: pgmap v13549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:53 smithi059 ceph-mon[97450]: pgmap v13549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:54.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:54.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:54.287356+0000) 2022-09-08T04:36:54.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:54.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:54.284492+0000) 2022-09-08T04:36:55.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:55.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:55.287511+0000) 2022-09-08T04:36:55.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:55.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:55.284743+0000) 2022-09-08T04:36:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:55 smithi055 ceph-mon[97033]: pgmap v13550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:55 smithi059 ceph-mon[97450]: pgmap v13550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:36:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:56.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:56.287763+0000) 2022-09-08T04:36:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:56.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:56.284942+0000) 2022-09-08T04:36:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:57.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:57.287939+0000) 2022-09-08T04:36:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:57.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:57.285183+0000) 2022-09-08T04:36:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:57 smithi055 ceph-mon[97033]: pgmap v13551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:36:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:57 smithi059 ceph-mon[97450]: pgmap v13551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:36:58.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:58.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:58.285420+0000) 2022-09-08T04:36:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:58.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:58.288061+0000) 2022-09-08T04:36:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:36:59.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:59.288252+0000) 2022-09-08T04:36:59.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:36:59.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:36:59.285698+0000) 2022-09-08T04:37:00.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:36:59 smithi055 ceph-mon[97033]: pgmap v13552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:36:59 smithi059 ceph-mon[97450]: pgmap v13552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:00.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:00.288407+0000) 2022-09-08T04:37:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:00.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:00.285812+0000) 2022-09-08T04:37:01.285 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:37:00.966+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:37:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:01.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:01.288614+0000) 2022-09-08T04:37:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:01.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:01.286025+0000) 2022-09-08T04:37:02.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:01 smithi055 ceph-mon[97033]: pgmap v13553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:37:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:01 smithi059 ceph-mon[97450]: pgmap v13553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:37:02.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:02.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:02.288823+0000) 2022-09-08T04:37:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:02.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:02.286246+0000) 2022-09-08T04:37:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:03.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:03.288994+0000) 2022-09-08T04:37:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:03.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:03.286485+0000) 2022-09-08T04:37:04.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:03 smithi055 ceph-mon[97033]: pgmap v13554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:03 smithi059 ceph-mon[97450]: pgmap v13554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:04.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:04.289168+0000) 2022-09-08T04:37:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:04.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:04.286736+0000) 2022-09-08T04:37:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:05.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:05.289405+0000) 2022-09-08T04:37:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:05.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:05.287011+0000) 2022-09-08T04:37:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:06 smithi059 ceph-mon[97450]: pgmap v13555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:37:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:37:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:06 smithi059 ceph-mon[97450]: pgmap v13556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 49 KiB/s wr, 24 op/s 2022-09-08T04:37:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:06 smithi059 ceph-mon[97450]: pgmap v13557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 60 KiB/s wr, 26 op/s 2022-09-08T04:37:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:37:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:37:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:06 smithi055 ceph-mon[97033]: pgmap v13555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:37:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:37:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:06 smithi055 ceph-mon[97033]: pgmap v13556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 49 KiB/s wr, 24 op/s 2022-09-08T04:37:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:06 smithi055 ceph-mon[97033]: pgmap v13557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 60 KiB/s wr, 26 op/s 2022-09-08T04:37:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:37:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:37:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:06.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:06.289625+0000) 2022-09-08T04:37:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:06.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:06.287254+0000) 2022-09-08T04:37:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:37:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:37:07.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:07.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:07.289822+0000) 2022-09-08T04:37:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:07.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:07.287494+0000) 2022-09-08T04:37:08.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:08 smithi055 ceph-mon[97033]: pgmap v13558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 60 KiB/s wr, 26 op/s 2022-09-08T04:37:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:08 smithi059 ceph-mon[97450]: pgmap v13558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 60 KiB/s wr, 26 op/s 2022-09-08T04:37:08.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:08.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:08.289957+0000) 2022-09-08T04:37:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:08.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:08.287764+0000) 2022-09-08T04:37:09.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:09.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:09.290095+0000) 2022-09-08T04:37:09.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:09.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:09.288019+0000) 2022-09-08T04:37:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:10 smithi055 ceph-mon[97033]: pgmap v13559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 60 KiB/s wr, 23 op/s 2022-09-08T04:37:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:10 smithi055 ceph-mon[97033]: osdmap e928: 12 total, 11 up, 11 in 2022-09-08T04:37:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:10 smithi059 ceph-mon[97450]: pgmap v13559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 60 KiB/s wr, 23 op/s 2022-09-08T04:37:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:10 smithi059 ceph-mon[97450]: osdmap e928: 12 total, 11 up, 11 in 2022-09-08T04:37:10.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:10.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:10.290255+0000) 2022-09-08T04:37:10.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:10.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:10.288241+0000) 2022-09-08T04:37:11.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:11.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:11.290466+0000) 2022-09-08T04:37:11.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:11.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:11.288418+0000) 2022-09-08T04:37:12.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:12 smithi055 ceph-mon[97033]: pgmap v13561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 20 KiB/s rd, 78 KiB/s wr, 35 op/s 2022-09-08T04:37:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:12 smithi059 ceph-mon[97450]: pgmap v13561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 20 KiB/s rd, 78 KiB/s wr, 35 op/s 2022-09-08T04:37:12.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:12.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:12.290677+0000) 2022-09-08T04:37:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:12.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:12.288683+0000) 2022-09-08T04:37:13.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:13.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:13.290838+0000) 2022-09-08T04:37:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:13.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:13.288933+0000) 2022-09-08T04:37:14.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:14 smithi055 ceph-mon[97033]: pgmap v13562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T04:37:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:14 smithi059 ceph-mon[97450]: pgmap v13562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T04:37:14.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:14.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:14.290965+0000) 2022-09-08T04:37:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:14.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:14.289163+0000) 2022-09-08T04:37:15.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:15.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:15.291141+0000) 2022-09-08T04:37:15.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:15.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:15.289423+0000) 2022-09-08T04:37:16.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:15 smithi055 ceph-mon[97033]: pgmap v13563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:37:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:15 smithi059 ceph-mon[97450]: pgmap v13563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:37:16.288 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:37:15.981+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:37:16.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:16.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:16.291342+0000) 2022-09-08T04:37:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:16.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:16.289682+0000) 2022-09-08T04:37:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:17.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:17.291529+0000) 2022-09-08T04:37:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:17.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:17.289909+0000) 2022-09-08T04:37:18.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:18 smithi055 ceph-mon[97033]: pgmap v13564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:37:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:18 smithi059 ceph-mon[97450]: pgmap v13564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:37:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:18.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:18.291771+0000) 2022-09-08T04:37:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:18.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:18.290085+0000) 2022-09-08T04:37:19.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:19.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:19.291903+0000) 2022-09-08T04:37:19.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:19.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:19.290322+0000) 2022-09-08T04:37:20.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:20 smithi055 ceph-mon[97033]: pgmap v13565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:37:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:20 smithi055 ceph-mon[97033]: osdmap e929: 12 total, 11 up, 11 in 2022-09-08T04:37:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:20 smithi059 ceph-mon[97450]: pgmap v13565: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:37:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:20 smithi059 ceph-mon[97450]: osdmap e929: 12 total, 11 up, 11 in 2022-09-08T04:37:20.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:20.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:20.292077+0000) 2022-09-08T04:37:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:20.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:20.290505+0000) 2022-09-08T04:37:21.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:21.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:21.292313+0000) 2022-09-08T04:37:21.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:21.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:21.290744+0000) 2022-09-08T04:37:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:22 smithi055 ceph-mon[97033]: pgmap v13567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:37:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:22 smithi059 ceph-mon[97450]: pgmap v13567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:37:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:22.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:22.292570+0000) 2022-09-08T04:37:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:22.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:22.290978+0000) 2022-09-08T04:37:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:23.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:23.292821+0000) 2022-09-08T04:37:23.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:23.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:23.291200+0000) 2022-09-08T04:37:24.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:24 smithi055 ceph-mon[97033]: pgmap v13568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:37:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:24 smithi059 ceph-mon[97450]: pgmap v13568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:37:24.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:24.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:24.292979+0000) 2022-09-08T04:37:24.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:24.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:24.291435+0000) 2022-09-08T04:37:25.400 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:25.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:25.293146+0000) 2022-09-08T04:37:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:25.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:25.291714+0000) 2022-09-08T04:37:26.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:25 smithi055 ceph-mon[97033]: pgmap v13569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:37:26.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:25 smithi059 ceph-mon[97450]: pgmap v13569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:37:26.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:26.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:26.293374+0000) 2022-09-08T04:37:26.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:26.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:26.291905+0000) 2022-09-08T04:37:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:27.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:27.293607+0000) 2022-09-08T04:37:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:27.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:27.292131+0000) 2022-09-08T04:37:28.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:28 smithi055 ceph-mon[97033]: pgmap v13570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:37:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:28 smithi059 ceph-mon[97450]: pgmap v13570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:37:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:28.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:28.293787+0000) 2022-09-08T04:37:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:28.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:28.292416+0000) 2022-09-08T04:37:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:29.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:29.294013+0000) 2022-09-08T04:37:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:29.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:29.292722+0000) 2022-09-08T04:37:30.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:30 smithi055 ceph-mon[97033]: pgmap v13571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:37:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:30 smithi059 ceph-mon[97450]: pgmap v13571: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:37:30.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:30.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:30.294233+0000) 2022-09-08T04:37:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:30.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:30.292976+0000) 2022-09-08T04:37:31.292 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:37:30.996+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:37:31.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:31.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:31.294371+0000) 2022-09-08T04:37:31.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:31.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:31.293215+0000) 2022-09-08T04:37:32.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:32 smithi055 ceph-mon[97033]: pgmap v13572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-08T04:37:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:32 smithi059 ceph-mon[97450]: pgmap v13572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-08T04:37:32.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:32.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:32.294529+0000) 2022-09-08T04:37:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:32.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:32.293456+0000) 2022-09-08T04:37:33.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:33.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:33.294787+0000) 2022-09-08T04:37:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:33.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:33.293700+0000) 2022-09-08T04:37:34.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:34 smithi055 ceph-mon[97033]: pgmap v13573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:34 smithi059 ceph-mon[97450]: pgmap v13573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:34.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:34.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:34.294963+0000) 2022-09-08T04:37:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:34.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:34.293936+0000) 2022-09-08T04:37:35.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:35.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:35.295184+0000) 2022-09-08T04:37:35.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:35.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:35.294168+0000) 2022-09-08T04:37:36.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:35 smithi055 ceph-mon[97033]: pgmap v13574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:37:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:35 smithi059 ceph-mon[97450]: pgmap v13574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:37:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:36.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:36.295370+0000) 2022-09-08T04:37:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:36.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:36.294408+0000) 2022-09-08T04:37:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:37.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:37.295568+0000) 2022-09-08T04:37:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:37.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:37.294655+0000) 2022-09-08T04:37:38.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:38 smithi055 ceph-mon[97033]: pgmap v13575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:38 smithi059 ceph-mon[97450]: pgmap v13575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:38.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:38.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:38.295718+0000) 2022-09-08T04:37:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:38.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:38.294900+0000) 2022-09-08T04:37:39.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:39.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:39.295865+0000) 2022-09-08T04:37:39.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:39.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:39.295167+0000) 2022-09-08T04:37:40.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:40 smithi055 ceph-mon[97033]: pgmap v13576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:40 smithi059 ceph-mon[97450]: pgmap v13576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:40.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:40.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:40.295992+0000) 2022-09-08T04:37:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:40.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:40.295372+0000) 2022-09-08T04:37:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:41.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:41.296117+0000) 2022-09-08T04:37:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:41.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:41.295606+0000) 2022-09-08T04:37:42.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:42 smithi055 ceph-mon[97033]: pgmap v13577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:37:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:42 smithi059 ceph-mon[97450]: pgmap v13577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:37:42.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:42.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:42.296242+0000) 2022-09-08T04:37:42.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:42.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:42.295909+0000) 2022-09-08T04:37:43.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:43.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:43.296376+0000) 2022-09-08T04:37:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:43.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:43.296167+0000) 2022-09-08T04:37:44.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:44 smithi055 ceph-mon[97033]: pgmap v13578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:44 smithi059 ceph-mon[97450]: pgmap v13578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:44.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:44.296605+0000) 2022-09-08T04:37:44.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:44.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:44.296395+0000) 2022-09-08T04:37:45.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:45.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:45.296677+0000) 2022-09-08T04:37:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:45.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:45.296844+0000) 2022-09-08T04:37:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:45 smithi055 ceph-mon[97033]: pgmap v13579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:37:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:45 smithi059 ceph-mon[97450]: pgmap v13579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:37:46.296 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:37:46.007+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:37:46.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:46.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:46.296997+0000) 2022-09-08T04:37:46.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:46.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:46.296894+0000) 2022-09-08T04:37:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:47.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:47.297198+0000) 2022-09-08T04:37:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:47.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:47.297074+0000) 2022-09-08T04:37:48.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:48 smithi055 ceph-mon[97033]: pgmap v13580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:48 smithi059 ceph-mon[97450]: pgmap v13580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:48.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:48.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:48.297353+0000) 2022-09-08T04:37:48.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:48.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:48.297255+0000) 2022-09-08T04:37:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:49.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:49.297515+0000) 2022-09-08T04:37:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:49.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:49.297480+0000) 2022-09-08T04:37:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:50 smithi055 ceph-mon[97033]: pgmap v13581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:50 smithi059 ceph-mon[97450]: pgmap v13581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:50.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:50.297759+0000) 2022-09-08T04:37:50.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:50.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:50.297763+0000) 2022-09-08T04:37:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:51.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:51.297993+0000) 2022-09-08T04:37:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:51.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:51.297999+0000) 2022-09-08T04:37:52.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:52 smithi055 ceph-mon[97033]: pgmap v13582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:37:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:52 smithi059 ceph-mon[97450]: pgmap v13582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:37:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:52.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:52.298176+0000) 2022-09-08T04:37:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:52.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:52.298177+0000) 2022-09-08T04:37:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:53.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:53.298427+0000) 2022-09-08T04:37:53.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:53.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:53.298411+0000) 2022-09-08T04:37:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:54 smithi055 ceph-mon[97033]: pgmap v13583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:54.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:54 smithi059 ceph-mon[97450]: pgmap v13583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:54.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:54.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:54.298692+0000) 2022-09-08T04:37:54.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:54.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:54.298710+0000) 2022-09-08T04:37:55.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:55.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:55.298858+0000) 2022-09-08T04:37:55.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:55.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:55.298876+0000) 2022-09-08T04:37:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:55 smithi055 ceph-mon[97033]: pgmap v13584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:37:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:55 smithi059 ceph-mon[97450]: pgmap v13584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:37:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:56.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:56.299077+0000) 2022-09-08T04:37:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:56.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:56.299106+0000) 2022-09-08T04:37:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:57.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:57.299340+0000) 2022-09-08T04:37:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:57.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:57.299342+0000) 2022-09-08T04:37:58.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:37:58 smithi055 ceph-mon[97033]: pgmap v13585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:37:58 smithi059 ceph-mon[97450]: pgmap v13585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:37:58.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:58.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:58.299554+0000) 2022-09-08T04:37:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:58.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:58.299559+0000) 2022-09-08T04:37:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:37:59.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:59.299821+0000) 2022-09-08T04:37:59.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:37:59.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:37:59.299815+0000) 2022-09-08T04:38:00.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:00 smithi055 ceph-mon[97033]: pgmap v13586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:00 smithi059 ceph-mon[97450]: pgmap v13586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:00.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:00.300124+0000) 2022-09-08T04:38:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:00.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:00.300102+0000) 2022-09-08T04:38:01.299 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:38:01.012+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:38:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:01.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:01.300306+0000) 2022-09-08T04:38:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:01.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:01.300269+0000) 2022-09-08T04:38:02.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:02 smithi055 ceph-mon[97033]: pgmap v13587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:38:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:02 smithi059 ceph-mon[97450]: pgmap v13587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:38:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:02.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:02.300542+0000) 2022-09-08T04:38:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:02.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:02.300505+0000) 2022-09-08T04:38:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:03.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:03.300810+0000) 2022-09-08T04:38:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:03.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:03.300791+0000) 2022-09-08T04:38:04.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:04 smithi055 ceph-mon[97033]: pgmap v13588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:04 smithi059 ceph-mon[97450]: pgmap v13588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:04.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:04.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:04.301051+0000) 2022-09-08T04:38:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:04.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:04.301011+0000) 2022-09-08T04:38:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:05.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:05.301346+0000) 2022-09-08T04:38:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:05.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:05.301293+0000) 2022-09-08T04:38:06.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:05 smithi059 ceph-mon[97450]: pgmap v13589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:38:06.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:38:06.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:38:06.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:38:06.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:38:06.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:38:06.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:38:06.138 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:05 smithi055 ceph-mon[97033]: pgmap v13589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:38:06.138 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:38:06.139 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:38:06.139 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:38:06.139 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:38:06.140 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:38:06.140 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:38:06.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:06.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:06.301563+0000) 2022-09-08T04:38:06.663 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:06.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:06.301544+0000) 2022-09-08T04:38:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:06 smithi055 ceph-mon[97033]: pgmap v13590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 48 KiB/s wr, 21 op/s 2022-09-08T04:38:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:38:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:38:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:06 smithi059 ceph-mon[97450]: pgmap v13590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 48 KiB/s wr, 21 op/s 2022-09-08T04:38:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:38:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:38:07.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:07.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:07.301761+0000) 2022-09-08T04:38:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:07.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:07.301747+0000) 2022-09-08T04:38:08.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:07 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T04:38:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:07 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T04:38:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:08.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:08.301934+0000) 2022-09-08T04:38:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:08.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:08.301989+0000) 2022-09-08T04:38:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:08 smithi055 ceph-mon[97033]: pgmap v13591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 48 KiB/s wr, 21 op/s 2022-09-08T04:38:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:08 smithi059 ceph-mon[97450]: pgmap v13591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 48 KiB/s wr, 21 op/s 2022-09-08T04:38:09.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:09.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:09.302147+0000) 2022-09-08T04:38:09.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:09.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:09.302163+0000) 2022-09-08T04:38:10.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:09 smithi055 ceph-mon[97033]: osdmap e930: 12 total, 11 up, 11 in 2022-09-08T04:38:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:09 smithi059 ceph-mon[97450]: osdmap e930: 12 total, 11 up, 11 in 2022-09-08T04:38:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:10.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:10.302363+0000) 2022-09-08T04:38:10.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:10.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:10.302387+0000) 2022-09-08T04:38:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:10 smithi055 ceph-mon[97033]: pgmap v13593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 59 KiB/s wr, 26 op/s 2022-09-08T04:38:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:10 smithi059 ceph-mon[97450]: pgmap v13593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 59 KiB/s wr, 26 op/s 2022-09-08T04:38:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:11.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:11.302628+0000) 2022-09-08T04:38:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:11.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:11.302607+0000) 2022-09-08T04:38:12.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:12 smithi055 ceph-mon[97033]: pgmap v13594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 59 KiB/s wr, 26 op/s 2022-09-08T04:38:12.560 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:12 smithi059 ceph-mon[97450]: pgmap v13594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 59 KiB/s wr, 26 op/s 2022-09-08T04:38:12.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:12.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:12.302874+0000) 2022-09-08T04:38:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:12.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:12.302861+0000) 2022-09-08T04:38:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:13.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:13.303095+0000) 2022-09-08T04:38:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:13.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:13.303074+0000) 2022-09-08T04:38:14.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:14 smithi055 ceph-mon[97033]: pgmap v13595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 112 B/s wr, 3 op/s 2022-09-08T04:38:14.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:14 smithi059 ceph-mon[97450]: pgmap v13595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 112 B/s wr, 3 op/s 2022-09-08T04:38:14.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:14.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:14.303296+0000) 2022-09-08T04:38:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:14.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:14.303319+0000) 2022-09-08T04:38:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:15.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:15.303544+0000) 2022-09-08T04:38:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:15.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:15.303592+0000) 2022-09-08T04:38:16.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:15 smithi055 ceph-mon[97033]: osdmap e931: 12 total, 11 up, 11 in 2022-09-08T04:38:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:15 smithi059 ceph-mon[97450]: osdmap e931: 12 total, 11 up, 11 in 2022-09-08T04:38:16.304 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:38:16.028+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:38:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:16.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:16.303804+0000) 2022-09-08T04:38:16.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:16.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:16.303757+0000) 2022-09-08T04:38:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:16 smithi055 ceph-mon[97033]: pgmap v13597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:38:17.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:16 smithi059 ceph-mon[97450]: pgmap v13597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:38:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:17.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:17.304018+0000) 2022-09-08T04:38:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:17.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:17.303976+0000) 2022-09-08T04:38:18.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:18 smithi055 ceph-mon[97033]: pgmap v13598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s 2022-09-08T04:38:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:18 smithi059 ceph-mon[97450]: pgmap v13598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s 2022-09-08T04:38:18.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:18.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:18.304186+0000) 2022-09-08T04:38:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:18.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:18.304214+0000) 2022-09-08T04:38:19.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:19.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:19.304359+0000) 2022-09-08T04:38:19.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:19.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:19.304431+0000) 2022-09-08T04:38:20.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:20.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:20.304560+0000) 2022-09-08T04:38:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:20.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:20.304599+0000) 2022-09-08T04:38:21.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:20 smithi055 ceph-mon[97033]: pgmap v13599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:38:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:20 smithi059 ceph-mon[97450]: pgmap v13599: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:38:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:21.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:21.304724+0000) 2022-09-08T04:38:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:21.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:21.304753+0000) 2022-09-08T04:38:22.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:22 smithi055 ceph-mon[97033]: pgmap v13600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:38:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:22 smithi059 ceph-mon[97450]: pgmap v13600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:38:22.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:22.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:22.304918+0000) 2022-09-08T04:38:22.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:22.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:22.304938+0000) 2022-09-08T04:38:23.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:23.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:23.305068+0000) 2022-09-08T04:38:23.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:23.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:23.305084+0000) 2022-09-08T04:38:24.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:24 smithi055 ceph-mon[97033]: pgmap v13601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:38:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:24 smithi059 ceph-mon[97450]: pgmap v13601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:38:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:24.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:24.305324+0000) 2022-09-08T04:38:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:24.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:24.305328+0000) 2022-09-08T04:38:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:25.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:25.305598+0000) 2022-09-08T04:38:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:25.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:25.305609+0000) 2022-09-08T04:38:26.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:26.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:26.305864+0000) 2022-09-08T04:38:26.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:26.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:26.305857+0000) 2022-09-08T04:38:27.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:27 smithi055 ceph-mon[97033]: pgmap v13602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T04:38:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:27 smithi059 ceph-mon[97450]: pgmap v13602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T04:38:27.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:27.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:27.306115+0000) 2022-09-08T04:38:27.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:27.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:27.306103+0000) 2022-09-08T04:38:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:28.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:28.306338+0000) 2022-09-08T04:38:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:28.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:28.306359+0000) 2022-09-08T04:38:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:29 smithi055 ceph-mon[97033]: pgmap v13603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:29 smithi059 ceph-mon[97450]: pgmap v13603: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:29.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:29.306598+0000) 2022-09-08T04:38:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:29.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:29.306665+0000) 2022-09-08T04:38:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:30.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:30.306878+0000) 2022-09-08T04:38:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:30.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:30.306930+0000) 2022-09-08T04:38:31.306 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:38:31.034+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:38:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:31.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:31.307093+0000) 2022-09-08T04:38:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:31.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:31.307125+0000) 2022-09-08T04:38:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:31 smithi055 ceph-mon[97033]: pgmap v13604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:38:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:31 smithi059 ceph-mon[97450]: pgmap v13604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:38:32.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:32.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:32.307299+0000) 2022-09-08T04:38:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:32.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:32.307320+0000) 2022-09-08T04:38:33.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:33.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:33.307546+0000) 2022-09-08T04:38:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:33.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:33.307551+0000) 2022-09-08T04:38:34.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:33 smithi055 ceph-mon[97033]: pgmap v13605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:33 smithi059 ceph-mon[97450]: pgmap v13605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:34.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:34.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:34.307722+0000) 2022-09-08T04:38:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:34.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:34.307725+0000) 2022-09-08T04:38:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:35.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:35.307984+0000) 2022-09-08T04:38:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:35.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:35.307954+0000) 2022-09-08T04:38:36.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:35 smithi055 ceph-mon[97033]: pgmap v13606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:35 smithi059 ceph-mon[97450]: pgmap v13606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:36.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:36.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:36.308187+0000) 2022-09-08T04:38:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:36.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:36.308171+0000) 2022-09-08T04:38:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:37.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:37.308368+0000) 2022-09-08T04:38:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:37.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:37.308403+0000) 2022-09-08T04:38:38.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:37 smithi055 ceph-mon[97033]: pgmap v13607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:38:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:37 smithi059 ceph-mon[97450]: pgmap v13607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:38:38.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:38.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:38.308592+0000) 2022-09-08T04:38:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:38.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:38.308596+0000) 2022-09-08T04:38:39.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:39.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:39.308832+0000) 2022-09-08T04:38:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:39.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:39.308844+0000) 2022-09-08T04:38:40.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:39 smithi055 ceph-mon[97033]: pgmap v13608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:40.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:39 smithi059 ceph-mon[97450]: pgmap v13608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:40.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:40.309037+0000) 2022-09-08T04:38:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:40.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:40.309023+0000) 2022-09-08T04:38:41.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:41.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:41.309249+0000) 2022-09-08T04:38:41.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:41.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:41.309308+0000) 2022-09-08T04:38:42.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:41 smithi055 ceph-mon[97033]: pgmap v13609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:38:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:41 smithi059 ceph-mon[97450]: pgmap v13609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:38:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:42.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:42.309428+0000) 2022-09-08T04:38:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:42.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:42.309494+0000) 2022-09-08T04:38:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:43.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:43.309675+0000) 2022-09-08T04:38:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:43.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:43.309731+0000) 2022-09-08T04:38:44.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:43 smithi055 ceph-mon[97033]: pgmap v13610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:43 smithi059 ceph-mon[97450]: pgmap v13610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:44.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:44.309876+0000) 2022-09-08T04:38:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:44.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:44.309968+0000) 2022-09-08T04:38:45.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:45.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:45.310159+0000) 2022-09-08T04:38:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:45.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:45.310216+0000) 2022-09-08T04:38:46.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:45 smithi055 ceph-mon[97033]: pgmap v13611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:45 smithi059 ceph-mon[97450]: pgmap v13611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:46.309 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:38:46.044+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:38:46.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:46.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:46.310432+0000) 2022-09-08T04:38:46.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:46.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:46.310445+0000) 2022-09-08T04:38:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:47.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:47.310679+0000) 2022-09-08T04:38:47.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:47.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:47.310678+0000) 2022-09-08T04:38:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:47 smithi055 ceph-mon[97033]: pgmap v13612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:38:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:47 smithi059 ceph-mon[97450]: pgmap v13612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:38:48.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:48.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:48.310893+0000) 2022-09-08T04:38:48.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:48.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:48.310869+0000) 2022-09-08T04:38:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:49.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:49.311047+0000) 2022-09-08T04:38:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:49.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:49.311025+0000) 2022-09-08T04:38:50.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:49 smithi055 ceph-mon[97033]: pgmap v13613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:49 smithi059 ceph-mon[97450]: pgmap v13613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:50.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:50.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:50.311237+0000) 2022-09-08T04:38:50.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:50.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:50.311233+0000) 2022-09-08T04:38:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:51.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:51.311376+0000) 2022-09-08T04:38:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:51.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:51.311398+0000) 2022-09-08T04:38:52.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:51 smithi055 ceph-mon[97033]: pgmap v13614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:38:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:51 smithi059 ceph-mon[97450]: pgmap v13614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:38:52.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:52.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:52.311541+0000) 2022-09-08T04:38:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:52.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:52.311553+0000) 2022-09-08T04:38:53.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:53.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:53.311757+0000) 2022-09-08T04:38:53.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:53.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:53.311760+0000) 2022-09-08T04:38:54.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:53 smithi055 ceph-mon[97033]: pgmap v13615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:53 smithi059 ceph-mon[97450]: pgmap v13615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:54.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:54.311932+0000) 2022-09-08T04:38:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:54.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:54.311922+0000) 2022-09-08T04:38:55.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:55.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:55.312155+0000) 2022-09-08T04:38:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:55.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:55.312177+0000) 2022-09-08T04:38:56.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:55 smithi055 ceph-mon[97033]: pgmap v13616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:55 smithi059 ceph-mon[97450]: pgmap v13616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:38:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:56.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:56.312369+0000) 2022-09-08T04:38:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:56.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:56.312360+0000) 2022-09-08T04:38:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:57.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:57.312582+0000) 2022-09-08T04:38:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:57.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:57.312551+0000) 2022-09-08T04:38:58.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:57 smithi055 ceph-mon[97033]: pgmap v13617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:38:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:57 smithi059 ceph-mon[97450]: pgmap v13617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:38:58.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:58.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:58.312750+0000) 2022-09-08T04:38:58.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:58.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:58.312717+0000) 2022-09-08T04:38:59.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:38:59.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:59.312963+0000) 2022-09-08T04:38:59.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:38:59.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:38:59.312919+0000) 2022-09-08T04:39:00.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:38:59 smithi055 ceph-mon[97033]: pgmap v13618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:38:59 smithi059 ceph-mon[97450]: pgmap v13618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:00.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:00.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:00.313166+0000) 2022-09-08T04:39:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:00.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:00.313156+0000) 2022-09-08T04:39:01.312 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:39:01.059+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:39:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:01.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:01.313330+0000) 2022-09-08T04:39:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:01.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:01.313341+0000) 2022-09-08T04:39:02.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:01 smithi055 ceph-mon[97033]: pgmap v13619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:39:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:01 smithi059 ceph-mon[97450]: pgmap v13619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:39:02.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:02.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:02.313551+0000) 2022-09-08T04:39:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:02.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:02.313584+0000) 2022-09-08T04:39:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:03.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:03.313692+0000) 2022-09-08T04:39:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:03.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:03.313735+0000) 2022-09-08T04:39:04.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:03 smithi055 ceph-mon[97033]: pgmap v13620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:03 smithi059 ceph-mon[97450]: pgmap v13620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:04.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:04.313890+0000) 2022-09-08T04:39:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:04.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:04.313951+0000) 2022-09-08T04:39:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:05.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:05.314119+0000) 2022-09-08T04:39:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:05.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:05.314058+0000) 2022-09-08T04:39:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:05 smithi055 ceph-mon[97033]: pgmap v13621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:39:06.254 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:05 smithi059 ceph-mon[97450]: pgmap v13621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:06.255 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:39:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:06.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:06.314207+0000) 2022-09-08T04:39:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:06.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:06.314244+0000) 2022-09-08T04:39:07.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:39:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:39:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:39:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:39:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:39:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:39:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:39:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:39:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:39:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:39:07.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:07.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:07.314425+0000) 2022-09-08T04:39:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:07.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:07.314435+0000) 2022-09-08T04:39:08.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:07 smithi055 ceph-mon[97033]: pgmap v13622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:39:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:07 smithi059 ceph-mon[97450]: pgmap v13622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:39:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:08.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:08.314757+0000) 2022-09-08T04:39:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:08.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:08.314700+0000) 2022-09-08T04:39:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:09.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:09.315003+0000) 2022-09-08T04:39:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:09.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:09.314895+0000) 2022-09-08T04:39:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:09 smithi055 ceph-mon[97033]: pgmap v13623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:39:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:09 smithi055 ceph-mon[97033]: osdmap e932: 12 total, 11 up, 11 in 2022-09-08T04:39:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:09 smithi059 ceph-mon[97450]: pgmap v13623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:39:10.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:09 smithi059 ceph-mon[97450]: osdmap e932: 12 total, 11 up, 11 in 2022-09-08T04:39:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:10.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:10.315164+0000) 2022-09-08T04:39:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:10.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:10.315137+0000) 2022-09-08T04:39:11.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:11.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:11.315364+0000) 2022-09-08T04:39:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:11.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:11.315408+0000) 2022-09-08T04:39:12.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:11 smithi055 ceph-mon[97033]: pgmap v13625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:39:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:11 smithi059 ceph-mon[97450]: pgmap v13625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:39:12.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:12.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:12.315620+0000) 2022-09-08T04:39:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:12.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:12.315666+0000) 2022-09-08T04:39:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:13.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:13.315966+0000) 2022-09-08T04:39:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:13.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:13.315938+0000) 2022-09-08T04:39:14.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:13 smithi055 ceph-mon[97033]: pgmap v13626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:39:14.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:13 smithi059 ceph-mon[97450]: pgmap v13626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:39:14.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:14.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:14.316171+0000) 2022-09-08T04:39:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:14.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:14.316176+0000) 2022-09-08T04:39:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:15.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:15.316323+0000) 2022-09-08T04:39:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:15.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:15.316312+0000) 2022-09-08T04:39:16.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:15 smithi055 ceph-mon[97033]: pgmap v13627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:39:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:15 smithi055 ceph-mon[97033]: osdmap e933: 12 total, 11 up, 11 in 2022-09-08T04:39:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:15 smithi059 ceph-mon[97450]: pgmap v13627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:39:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:15 smithi059 ceph-mon[97450]: osdmap e933: 12 total, 11 up, 11 in 2022-09-08T04:39:16.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:39:16.065+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:39:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:16.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:16.316567+0000) 2022-09-08T04:39:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:16.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:16.316539+0000) 2022-09-08T04:39:17.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:17.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:17.316838+0000) 2022-09-08T04:39:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:17.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:17.316783+0000) 2022-09-08T04:39:18.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:17 smithi055 ceph-mon[97033]: pgmap v13629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:39:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:17 smithi059 ceph-mon[97450]: pgmap v13629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:39:18.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:18.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:18.317078+0000) 2022-09-08T04:39:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:18.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:18.317115+0000) 2022-09-08T04:39:19.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:19.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:19.317366+0000) 2022-09-08T04:39:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:19.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:19.317391+0000) 2022-09-08T04:39:20.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:19 smithi055 ceph-mon[97033]: pgmap v13630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2022-09-08T04:39:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:19 smithi059 ceph-mon[97450]: pgmap v13630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2022-09-08T04:39:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:20.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:20.317612+0000) 2022-09-08T04:39:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:20.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:20.317613+0000) 2022-09-08T04:39:21.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:21.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:21.317867+0000) 2022-09-08T04:39:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:21.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:21.317888+0000) 2022-09-08T04:39:22.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:21 smithi055 ceph-mon[97033]: pgmap v13631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T04:39:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:21 smithi059 ceph-mon[97450]: pgmap v13631: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T04:39:22.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:22.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:22.318127+0000) 2022-09-08T04:39:22.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:22.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:22.318131+0000) 2022-09-08T04:39:23.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:23.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:23.318374+0000) 2022-09-08T04:39:23.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:23.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:23.318356+0000) 2022-09-08T04:39:24.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:23 smithi055 ceph-mon[97033]: pgmap v13632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T04:39:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:23 smithi059 ceph-mon[97450]: pgmap v13632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T04:39:24.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:24.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:24.318611+0000) 2022-09-08T04:39:24.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:24.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:24.318608+0000) 2022-09-08T04:39:25.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:25.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:25.318826+0000) 2022-09-08T04:39:25.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:25.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:25.318871+0000) 2022-09-08T04:39:26.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:25 smithi055 ceph-mon[97033]: pgmap v13633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T04:39:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:25 smithi059 ceph-mon[97450]: pgmap v13633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T04:39:26.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:26.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:26.319032+0000) 2022-09-08T04:39:26.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:26.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:26.319114+0000) 2022-09-08T04:39:27.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:27.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:27.319349+0000) 2022-09-08T04:39:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:27.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:27.319203+0000) 2022-09-08T04:39:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:27 smithi055 ceph-mon[97033]: pgmap v13634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 6 op/s 2022-09-08T04:39:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:27 smithi059 ceph-mon[97450]: pgmap v13634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 6 op/s 2022-09-08T04:39:28.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:28.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:28.319450+0000) 2022-09-08T04:39:28.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:28.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:28.319592+0000) 2022-09-08T04:39:29.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:29.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:29.319726+0000) 2022-09-08T04:39:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:29.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:29.319829+0000) 2022-09-08T04:39:30.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:29 smithi055 ceph-mon[97033]: pgmap v13635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:39:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:29 smithi059 ceph-mon[97450]: pgmap v13635: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:39:30.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:30.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:30.319942+0000) 2022-09-08T04:39:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:30.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:30.320023+0000) 2022-09-08T04:39:31.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:39:31.080+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:39:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:31.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:31.320152+0000) 2022-09-08T04:39:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:31.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:31.320240+0000) 2022-09-08T04:39:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:31 smithi055 ceph-mon[97033]: pgmap v13636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:39:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:31 smithi059 ceph-mon[97450]: pgmap v13636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:39:32.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:32.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:32.320410+0000) 2022-09-08T04:39:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:32.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:32.320442+0000) 2022-09-08T04:39:33.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:33.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:33.320595+0000) 2022-09-08T04:39:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:33.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:33.320617+0000) 2022-09-08T04:39:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:33 smithi055 ceph-mon[97033]: pgmap v13637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:33 smithi059 ceph-mon[97450]: pgmap v13637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:34.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:34.320794+0000) 2022-09-08T04:39:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:34.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:34.320784+0000) 2022-09-08T04:39:35.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:35.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:35.321020+0000) 2022-09-08T04:39:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:35.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:35.321068+0000) 2022-09-08T04:39:36.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:35 smithi055 ceph-mon[97033]: pgmap v13638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:35 smithi059 ceph-mon[97450]: pgmap v13638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:36.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:36.321274+0000) 2022-09-08T04:39:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:36.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:36.321242+0000) 2022-09-08T04:39:37.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:37.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:37.321518+0000) 2022-09-08T04:39:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:37.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:37.321464+0000) 2022-09-08T04:39:38.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:37 smithi055 ceph-mon[97033]: pgmap v13639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:39:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:37 smithi059 ceph-mon[97450]: pgmap v13639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:39:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:38.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:38.321791+0000) 2022-09-08T04:39:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:38.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:38.321739+0000) 2022-09-08T04:39:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:39.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:39.321970+0000) 2022-09-08T04:39:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:39.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:39.321907+0000) 2022-09-08T04:39:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:39 smithi055 ceph-mon[97033]: pgmap v13640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:39 smithi059 ceph-mon[97450]: pgmap v13640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:40.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:40.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:40.322102+0000) 2022-09-08T04:39:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:40.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:40.322141+0000) 2022-09-08T04:39:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:41.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:41.322337+0000) 2022-09-08T04:39:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:41.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:41.322330+0000) 2022-09-08T04:39:42.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:41 smithi055 ceph-mon[97033]: pgmap v13641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:39:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:41 smithi059 ceph-mon[97450]: pgmap v13641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:39:42.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:42.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:42.322588+0000) 2022-09-08T04:39:42.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:42.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:42.322571+0000) 2022-09-08T04:39:43.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:43.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:43.322875+0000) 2022-09-08T04:39:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:43.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:43.322792+0000) 2022-09-08T04:39:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:43 smithi055 ceph-mon[97033]: pgmap v13642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:43 smithi059 ceph-mon[97450]: pgmap v13642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:44.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:44.323045+0000) 2022-09-08T04:39:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:44.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:44.322957+0000) 2022-09-08T04:39:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:45.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:45.323220+0000) 2022-09-08T04:39:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:45.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:45.323194+0000) 2022-09-08T04:39:46.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:45 smithi055 ceph-mon[97033]: pgmap v13643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:45 smithi059 ceph-mon[97450]: pgmap v13643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:46.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:39:46.095+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:39:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:46.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:46.323456+0000) 2022-09-08T04:39:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:46.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:46.323456+0000) 2022-09-08T04:39:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:47.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:47.323617+0000) 2022-09-08T04:39:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:47.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:47.323608+0000) 2022-09-08T04:39:48.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:47 smithi055 ceph-mon[97033]: pgmap v13644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:39:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:47 smithi059 ceph-mon[97450]: pgmap v13644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:39:48.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:48.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:48.323831+0000) 2022-09-08T04:39:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:48.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:48.323909+0000) 2022-09-08T04:39:49.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:49.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:49.324149+0000) 2022-09-08T04:39:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:49.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:49.324101+0000) 2022-09-08T04:39:50.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:49 smithi055 ceph-mon[97033]: pgmap v13645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:49 smithi059 ceph-mon[97450]: pgmap v13645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:50.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:50.324414+0000) 2022-09-08T04:39:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:50.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:50.324343+0000) 2022-09-08T04:39:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:51.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:51.324698+0000) 2022-09-08T04:39:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:51.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:51.324582+0000) 2022-09-08T04:39:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:51 smithi055 ceph-mon[97033]: pgmap v13646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:39:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:51 smithi059 ceph-mon[97450]: pgmap v13646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:39:52.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:52.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:52.324874+0000) 2022-09-08T04:39:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:52.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:52.324814+0000) 2022-09-08T04:39:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:53.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:53.325140+0000) 2022-09-08T04:39:53.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:53.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:53.325068+0000) 2022-09-08T04:39:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:53 smithi055 ceph-mon[97033]: pgmap v13647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:53 smithi059 ceph-mon[97450]: pgmap v13647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:54.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:54.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:54.325379+0000) 2022-09-08T04:39:54.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:54.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:54.325370+0000) 2022-09-08T04:39:55.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:55.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:55.325599+0000) 2022-09-08T04:39:55.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:55.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:55.325605+0000) 2022-09-08T04:39:56.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:55 smithi055 ceph-mon[97033]: pgmap v13648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:55 smithi059 ceph-mon[97450]: pgmap v13648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:39:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:56.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:56.325842+0000) 2022-09-08T04:39:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:56.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:56.325827+0000) 2022-09-08T04:39:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:57.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:57.326096+0000) 2022-09-08T04:39:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:57.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:57.326089+0000) 2022-09-08T04:39:58.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:57 smithi055 ceph-mon[97033]: pgmap v13649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:39:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:57 smithi059 ceph-mon[97450]: pgmap v13649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:39:58.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:58.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:58.326334+0000) 2022-09-08T04:39:58.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:58.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:58.326307+0000) 2022-09-08T04:39:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:39:59.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:59.326531+0000) 2022-09-08T04:39:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:39:59.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:39:59.326535+0000) 2022-09-08T04:40:00.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:39:59 smithi055 ceph-mon[97033]: pgmap v13650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:39:59 smithi059 ceph-mon[97450]: pgmap v13650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:00.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:00.326787+0000) 2022-09-08T04:40:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:00.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:00.326766+0000) 2022-09-08T04:40:01.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T04:40:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T04:40:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T04:40:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T04:40:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T04:40:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T04:40:01.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:40:01.109+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:40:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:01.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:01.327011+0000) 2022-09-08T04:40:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:01.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:01.326980+0000) 2022-09-08T04:40:02.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:01 smithi055 ceph-mon[97033]: pgmap v13651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:40:02.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:01 smithi059 ceph-mon[97450]: pgmap v13651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:40:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:02.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:02.327220+0000) 2022-09-08T04:40:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:02.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:02.327221+0000) 2022-09-08T04:40:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:03.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:03.327478+0000) 2022-09-08T04:40:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:03.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:03.327457+0000) 2022-09-08T04:40:04.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:03 smithi055 ceph-mon[97033]: pgmap v13652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:03 smithi059 ceph-mon[97450]: pgmap v13652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:04.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:04.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:04.327770+0000) 2022-09-08T04:40:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:04.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:04.327755+0000) 2022-09-08T04:40:05.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:05.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:05.327973+0000) 2022-09-08T04:40:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:05.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:05.327900+0000) 2022-09-08T04:40:06.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:05 smithi055 ceph-mon[97033]: pgmap v13653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:40:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:05 smithi059 ceph-mon[97450]: pgmap v13653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:40:06.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:06.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:06.328115+0000) 2022-09-08T04:40:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:06.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:06.328058+0000) 2022-09-08T04:40:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:40:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:40:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:40:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:40:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:40:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:40:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:40:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:40:07.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:07.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:07.328353+0000) 2022-09-08T04:40:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:07.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:07.328292+0000) 2022-09-08T04:40:08.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:07 smithi055 ceph-mon[97033]: pgmap v13654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:40:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:40:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:07 smithi059 ceph-mon[97450]: pgmap v13654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:40:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:40:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:08.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:08.328574+0000) 2022-09-08T04:40:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:08.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:08.328566+0000) 2022-09-08T04:40:09.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:09.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:09.328797+0000) 2022-09-08T04:40:09.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:09.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:09.328829+0000) 2022-09-08T04:40:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:09 smithi059 ceph-mon[97450]: pgmap v13655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:40:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:09 smithi059 ceph-mon[97450]: osdmap e934: 12 total, 11 up, 11 in 2022-09-08T04:40:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:09 smithi055 ceph-mon[97033]: pgmap v13655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:40:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:09 smithi055 ceph-mon[97033]: osdmap e934: 12 total, 11 up, 11 in 2022-09-08T04:40:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:10.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:10.329031+0000) 2022-09-08T04:40:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:10.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:10.329151+0000) 2022-09-08T04:40:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:11.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:11.329247+0000) 2022-09-08T04:40:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:11.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:11.329425+0000) 2022-09-08T04:40:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:11 smithi059 ceph-mon[97450]: pgmap v13657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:40:12.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:11 smithi055 ceph-mon[97033]: pgmap v13657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:40:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:12.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:12.329574+0000) 2022-09-08T04:40:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:12.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:12.329414+0000) 2022-09-08T04:40:13.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:13.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:13.329669+0000) 2022-09-08T04:40:13.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:13.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:13.329745+0000) 2022-09-08T04:40:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:13 smithi059 ceph-mon[97450]: pgmap v13658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:40:14.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:13 smithi055 ceph-mon[97033]: pgmap v13658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:40:14.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:14.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:14.329827+0000) 2022-09-08T04:40:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:14.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:14.329926+0000) 2022-09-08T04:40:15.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:15.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:15.330022+0000) 2022-09-08T04:40:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:15.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:15.330095+0000) 2022-09-08T04:40:16.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:15 smithi055 ceph-mon[97033]: pgmap v13659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:40:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:15 smithi055 ceph-mon[97033]: osdmap e935: 12 total, 11 up, 11 in 2022-09-08T04:40:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:15 smithi059 ceph-mon[97450]: pgmap v13659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:40:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:15 smithi059 ceph-mon[97450]: osdmap e935: 12 total, 11 up, 11 in 2022-09-08T04:40:16.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:40:16.126+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:40:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:16.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:16.330253+0000) 2022-09-08T04:40:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:16.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:16.330358+0000) 2022-09-08T04:40:17.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:17.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:17.330456+0000) 2022-09-08T04:40:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:17.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:17.330603+0000) 2022-09-08T04:40:18.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:17 smithi055 ceph-mon[97033]: pgmap v13661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:40:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:17 smithi059 ceph-mon[97450]: pgmap v13661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:40:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:18.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:18.330724+0000) 2022-09-08T04:40:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:18.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:18.330866+0000) 2022-09-08T04:40:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:19.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:19.330996+0000) 2022-09-08T04:40:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:19.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:19.331124+0000) 2022-09-08T04:40:20.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:19 smithi055 ceph-mon[97033]: pgmap v13662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2022-09-08T04:40:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:19 smithi059 ceph-mon[97450]: pgmap v13662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2022-09-08T04:40:20.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:20.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:20.331191+0000) 2022-09-08T04:40:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:20.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:20.331263+0000) 2022-09-08T04:40:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:21.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:21.331317+0000) 2022-09-08T04:40:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:21.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:21.331423+0000) 2022-09-08T04:40:22.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:21 smithi055 ceph-mon[97033]: pgmap v13663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:40:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:21 smithi059 ceph-mon[97450]: pgmap v13663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:40:22.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:22.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:22.331507+0000) 2022-09-08T04:40:22.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:22.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:22.331584+0000) 2022-09-08T04:40:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:23.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:23.331724+0000) 2022-09-08T04:40:23.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:23.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:23.331796+0000) 2022-09-08T04:40:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:23 smithi055 ceph-mon[97033]: pgmap v13664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:40:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:23 smithi059 ceph-mon[97450]: pgmap v13664: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:40:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:24.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:24.331963+0000) 2022-09-08T04:40:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:24.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:24.332019+0000) 2022-09-08T04:40:25.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:25.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:25.332260+0000) 2022-09-08T04:40:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:25.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:25.332275+0000) 2022-09-08T04:40:26.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:25 smithi055 ceph-mon[97033]: pgmap v13665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:40:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:25 smithi059 ceph-mon[97450]: pgmap v13665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:40:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:26.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:26.332505+0000) 2022-09-08T04:40:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:26.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:26.332504+0000) 2022-09-08T04:40:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:27.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:27.332714+0000) 2022-09-08T04:40:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:27.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:27.332703+0000) 2022-09-08T04:40:28.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:27 smithi055 ceph-mon[97033]: pgmap v13666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:40:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:27 smithi059 ceph-mon[97450]: pgmap v13666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:40:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:28.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:28.332849+0000) 2022-09-08T04:40:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:28.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:28.332861+0000) 2022-09-08T04:40:29.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:29.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:29.333071+0000) 2022-09-08T04:40:29.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:29.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:29.333073+0000) 2022-09-08T04:40:30.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:29 smithi055 ceph-mon[97033]: pgmap v13667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:29 smithi059 ceph-mon[97450]: pgmap v13667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:30.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:30.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:30.333338+0000) 2022-09-08T04:40:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:30.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:30.333322+0000) 2022-09-08T04:40:31.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:40:31.140+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:40:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:31.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:31.333617+0000) 2022-09-08T04:40:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:31.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:31.333607+0000) 2022-09-08T04:40:32.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:31 smithi055 ceph-mon[97033]: pgmap v13668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:40:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:31 smithi059 ceph-mon[97450]: pgmap v13668: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:40:32.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:32.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:32.333876+0000) 2022-09-08T04:40:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:32.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:32.333834+0000) 2022-09-08T04:40:33.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:33.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:33.334034+0000) 2022-09-08T04:40:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:33.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:33.333985+0000) 2022-09-08T04:40:34.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:33 smithi055 ceph-mon[97033]: pgmap v13669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:33 smithi059 ceph-mon[97450]: pgmap v13669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:34.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:34.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:34.334270+0000) 2022-09-08T04:40:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:34.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:34.334219+0000) 2022-09-08T04:40:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:35.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:35.334533+0000) 2022-09-08T04:40:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:35.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:35.334507+0000) 2022-09-08T04:40:36.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:35 smithi055 ceph-mon[97033]: pgmap v13670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:35 smithi059 ceph-mon[97450]: pgmap v13670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:36.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:36.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:36.334745+0000) 2022-09-08T04:40:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:36.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:36.334727+0000) 2022-09-08T04:40:37.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:37.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:37.334991+0000) 2022-09-08T04:40:37.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:37.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:37.334962+0000) 2022-09-08T04:40:38.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:37 smithi055 ceph-mon[97033]: pgmap v13671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:40:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:37 smithi059 ceph-mon[97450]: pgmap v13671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:40:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:38.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:38.335199+0000) 2022-09-08T04:40:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:38.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:38.335177+0000) 2022-09-08T04:40:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:39.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:39.335436+0000) 2022-09-08T04:40:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:39.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:39.335428+0000) 2022-09-08T04:40:40.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:39 smithi055 ceph-mon[97033]: pgmap v13672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:39 smithi059 ceph-mon[97450]: pgmap v13672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:40.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:40.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:40.335659+0000) 2022-09-08T04:40:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:40.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:40.335689+0000) 2022-09-08T04:40:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:41.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:41.335814+0000) 2022-09-08T04:40:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:41.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:41.335849+0000) 2022-09-08T04:40:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:41 smithi055 ceph-mon[97033]: pgmap v13673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:40:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:41 smithi059 ceph-mon[97450]: pgmap v13673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:40:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:42.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:42.336033+0000) 2022-09-08T04:40:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:42.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:42.336083+0000) 2022-09-08T04:40:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:43.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:43.336264+0000) 2022-09-08T04:40:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:43.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:43.336263+0000) 2022-09-08T04:40:44.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:43 smithi055 ceph-mon[97033]: pgmap v13674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:43 smithi059 ceph-mon[97450]: pgmap v13674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:44.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:44.336488+0000) 2022-09-08T04:40:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:44.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:44.336464+0000) 2022-09-08T04:40:45.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:45.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:45.336743+0000) 2022-09-08T04:40:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:45.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:45.336752+0000) 2022-09-08T04:40:46.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:45 smithi055 ceph-mon[97033]: pgmap v13675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:45 smithi059 ceph-mon[97450]: pgmap v13675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:46.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:40:46.149+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:40:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:46.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:46.336892+0000) 2022-09-08T04:40:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:46.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:46.336926+0000) 2022-09-08T04:40:47.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:47.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:47.337117+0000) 2022-09-08T04:40:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:47.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:47.337134+0000) 2022-09-08T04:40:48.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:47 smithi055 ceph-mon[97033]: pgmap v13676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:40:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:47 smithi059 ceph-mon[97450]: pgmap v13676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:40:48.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:48.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:48.337326+0000) 2022-09-08T04:40:48.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:48.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:48.337371+0000) 2022-09-08T04:40:49.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:49.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:49.337471+0000) 2022-09-08T04:40:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:49.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:49.337561+0000) 2022-09-08T04:40:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:49 smithi055 ceph-mon[97033]: pgmap v13677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:49 smithi059 ceph-mon[97450]: pgmap v13677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:50.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:50.337650+0000) 2022-09-08T04:40:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:50.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:50.337718+0000) 2022-09-08T04:40:51.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:51.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:51.337839+0000) 2022-09-08T04:40:51.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:51.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:51.337885+0000) 2022-09-08T04:40:52.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:51 smithi055 ceph-mon[97033]: pgmap v13678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:40:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:51 smithi059 ceph-mon[97450]: pgmap v13678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:40:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:52.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:52.338048+0000) 2022-09-08T04:40:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:52.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:52.337968+0000) 2022-09-08T04:40:53.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:53.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:53.338288+0000) 2022-09-08T04:40:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:53.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:53.338188+0000) 2022-09-08T04:40:54.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:53 smithi055 ceph-mon[97033]: pgmap v13679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:53 smithi059 ceph-mon[97450]: pgmap v13679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:54.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:54.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:54.338377+0000) 2022-09-08T04:40:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:54.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:54.338486+0000) 2022-09-08T04:40:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:55.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:55.338554+0000) 2022-09-08T04:40:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:55.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:55.338656+0000) 2022-09-08T04:40:56.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:55 smithi055 ceph-mon[97033]: pgmap v13680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:55 smithi059 ceph-mon[97450]: pgmap v13680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:40:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:56.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:56.338753+0000) 2022-09-08T04:40:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:56.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:56.338839+0000) 2022-09-08T04:40:57.661 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:57.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:57.339021+0000) 2022-09-08T04:40:57.662 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:57.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:57.339048+0000) 2022-09-08T04:40:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:57 smithi055 ceph-mon[97033]: pgmap v13681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:40:58.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:57 smithi059 ceph-mon[97450]: pgmap v13681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:40:58.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:58.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:58.339282+0000) 2022-09-08T04:40:58.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:58.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:58.339282+0000) 2022-09-08T04:40:59.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:40:59.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:59.339541+0000) 2022-09-08T04:40:59.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:40:59.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:40:59.339540+0000) 2022-09-08T04:41:00.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:40:59 smithi055 ceph-mon[97033]: pgmap v13682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:40:59 smithi059 ceph-mon[97450]: pgmap v13682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:00.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:00.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:00.339692+0000) 2022-09-08T04:41:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:00.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:00.339708+0000) 2022-09-08T04:41:01.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:41:01.151+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:41:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:01.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:01.339909+0000) 2022-09-08T04:41:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:01.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:01.339929+0000) 2022-09-08T04:41:02.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:01 smithi055 ceph-mon[97033]: pgmap v13683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:41:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:01 smithi059 ceph-mon[97450]: pgmap v13683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:41:02.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:02.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:02.340160+0000) 2022-09-08T04:41:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:02.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:02.340160+0000) 2022-09-08T04:41:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:03.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:03.340408+0000) 2022-09-08T04:41:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:03.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:03.340403+0000) 2022-09-08T04:41:04.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:03 smithi055 ceph-mon[97033]: pgmap v13684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:03 smithi059 ceph-mon[97450]: pgmap v13684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:04.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:04.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:04.340597+0000) 2022-09-08T04:41:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:04.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:04.340603+0000) 2022-09-08T04:41:05.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:05.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:05.340821+0000) 2022-09-08T04:41:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:05.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:05.340845+0000) 2022-09-08T04:41:06.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:05 smithi055 ceph-mon[97033]: pgmap v13685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:41:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:05 smithi059 ceph-mon[97450]: pgmap v13685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:41:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:06.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:06.340979+0000) 2022-09-08T04:41:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:06.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:06.341022+0000) 2022-09-08T04:41:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:41:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:41:07.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:07.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:07.341194+0000) 2022-09-08T04:41:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:07.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:07.341242+0000) 2022-09-08T04:41:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:07 smithi059 ceph-mon[97450]: pgmap v13686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:41:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:41:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:41:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:41:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:41:08.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:07 smithi055 ceph-mon[97033]: pgmap v13686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:41:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:41:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:41:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:41:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:41:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:08.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:08.341424+0000) 2022-09-08T04:41:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:08.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:08.341402+0000) 2022-09-08T04:41:09.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:09.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:09.341557+0000) 2022-09-08T04:41:09.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:09.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:09.341579+0000) 2022-09-08T04:41:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:09 smithi059 ceph-mon[97450]: pgmap v13687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:41:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:09 smithi059 ceph-mon[97450]: osdmap e936: 12 total, 11 up, 11 in 2022-09-08T04:41:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:10 smithi055 ceph-mon[97033]: pgmap v13687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:41:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:10 smithi055 ceph-mon[97033]: osdmap e936: 12 total, 11 up, 11 in 2022-09-08T04:41:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:10.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:10.341750+0000) 2022-09-08T04:41:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:10.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:10.341764+0000) 2022-09-08T04:41:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:11.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:11.341887+0000) 2022-09-08T04:41:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:11.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:11.341921+0000) 2022-09-08T04:41:12.289 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:11 smithi059 ceph-mon[97450]: pgmap v13689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:41:12.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:12 smithi055 ceph-mon[97033]: pgmap v13689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:41:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:12.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:12.342096+0000) 2022-09-08T04:41:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:12.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:12.342131+0000) 2022-09-08T04:41:13.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:13.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:13.342326+0000) 2022-09-08T04:41:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:13.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:13.342311+0000) 2022-09-08T04:41:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:14 smithi059 ceph-mon[97450]: pgmap v13690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:41:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:14 smithi055 ceph-mon[97033]: pgmap v13690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:41:14.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:14.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:14.342557+0000) 2022-09-08T04:41:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:14.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:14.342542+0000) 2022-09-08T04:41:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:15.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:15.342795+0000) 2022-09-08T04:41:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:15.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:15.342823+0000) 2022-09-08T04:41:16.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:15 smithi055 ceph-mon[97033]: pgmap v13691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:41:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:15 smithi055 ceph-mon[97033]: osdmap e937: 12 total, 11 up, 11 in 2022-09-08T04:41:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:15 smithi059 ceph-mon[97450]: pgmap v13691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:41:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:15 smithi059 ceph-mon[97450]: osdmap e937: 12 total, 11 up, 11 in 2022-09-08T04:41:16.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:16.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:16.342981+0000) 2022-09-08T04:41:16.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:41:16.155+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:41:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:16.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:16.343089+0000) 2022-09-08T04:41:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:17.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:17.343199+0000) 2022-09-08T04:41:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:17.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:17.343360+0000) 2022-09-08T04:41:18.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:17 smithi055 ceph-mon[97033]: pgmap v13693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:41:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:17 smithi059 ceph-mon[97450]: pgmap v13693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:41:18.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:18.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:18.343498+0000) 2022-09-08T04:41:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:18.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:18.343564+0000) 2022-09-08T04:41:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:19.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:19.343767+0000) 2022-09-08T04:41:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:19.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:19.343786+0000) 2022-09-08T04:41:20.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:19 smithi055 ceph-mon[97033]: pgmap v13694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2022-09-08T04:41:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:19 smithi059 ceph-mon[97450]: pgmap v13694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2022-09-08T04:41:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:20.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:20.343999+0000) 2022-09-08T04:41:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:20.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:20.344012+0000) 2022-09-08T04:41:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:21.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:21.344210+0000) 2022-09-08T04:41:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:21.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:21.344222+0000) 2022-09-08T04:41:22.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:21 smithi055 ceph-mon[97033]: pgmap v13695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:41:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:21 smithi059 ceph-mon[97450]: pgmap v13695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:41:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:22.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:22.344409+0000) 2022-09-08T04:41:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:22.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:22.344413+0000) 2022-09-08T04:41:23.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:23.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:23.344693+0000) 2022-09-08T04:41:23.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:23.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:23.344703+0000) 2022-09-08T04:41:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:23 smithi055 ceph-mon[97033]: pgmap v13696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:41:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:23 smithi059 ceph-mon[97450]: pgmap v13696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:41:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:24.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:24.344842+0000) 2022-09-08T04:41:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:24.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:24.344870+0000) 2022-09-08T04:41:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:25.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:25.345108+0000) 2022-09-08T04:41:25.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:25.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:25.345102+0000) 2022-09-08T04:41:26.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:25 smithi055 ceph-mon[97033]: pgmap v13697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:41:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:25 smithi059 ceph-mon[97450]: pgmap v13697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:41:26.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:26.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:26.345381+0000) 2022-09-08T04:41:26.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:26.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:26.345332+0000) 2022-09-08T04:41:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:27.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:27.345712+0000) 2022-09-08T04:41:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:27.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:27.345599+0000) 2022-09-08T04:41:28.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:27 smithi055 ceph-mon[97033]: pgmap v13698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:41:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:27 smithi059 ceph-mon[97450]: pgmap v13698: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:41:28.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:28.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:28.345844+0000) 2022-09-08T04:41:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:28.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:28.345873+0000) 2022-09-08T04:41:29.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:29.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:29.346056+0000) 2022-09-08T04:41:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:29.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:29.346048+0000) 2022-09-08T04:41:30.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:29 smithi055 ceph-mon[97033]: pgmap v13699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:29 smithi059 ceph-mon[97450]: pgmap v13699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:30.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:30.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:30.346186+0000) 2022-09-08T04:41:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:30.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:30.346185+0000) 2022-09-08T04:41:31.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:41:31.161+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:41:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:31.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:31.346377+0000) 2022-09-08T04:41:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:31.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:31.346366+0000) 2022-09-08T04:41:32.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:31 smithi055 ceph-mon[97033]: pgmap v13700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:41:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:31 smithi059 ceph-mon[97450]: pgmap v13700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:41:32.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:32.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:32.346617+0000) 2022-09-08T04:41:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:32.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:32.346600+0000) 2022-09-08T04:41:33.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:33.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:33.346928+0000) 2022-09-08T04:41:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:33.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:33.346898+0000) 2022-09-08T04:41:34.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:33 smithi055 ceph-mon[97033]: pgmap v13701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:33 smithi059 ceph-mon[97450]: pgmap v13701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:34.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:34.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:34.347194+0000) 2022-09-08T04:41:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:34.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:34.347173+0000) 2022-09-08T04:41:35.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:35.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:35.347410+0000) 2022-09-08T04:41:35.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:35.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:35.347396+0000) 2022-09-08T04:41:36.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:35 smithi055 ceph-mon[97033]: pgmap v13702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:35 smithi059 ceph-mon[97450]: pgmap v13702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:36.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:36.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:36.347619+0000) 2022-09-08T04:41:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:36.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:36.347627+0000) 2022-09-08T04:41:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:37.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:37.347935+0000) 2022-09-08T04:41:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:37.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:37.347924+0000) 2022-09-08T04:41:38.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:37 smithi055 ceph-mon[97033]: pgmap v13703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:41:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:37 smithi059 ceph-mon[97450]: pgmap v13703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:41:38.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:38.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:38.348213+0000) 2022-09-08T04:41:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:38.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:38.348170+0000) 2022-09-08T04:41:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:39.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:39.348440+0000) 2022-09-08T04:41:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:39.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:39.348440+0000) 2022-09-08T04:41:40.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:39 smithi055 ceph-mon[97033]: pgmap v13704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:39 smithi059 ceph-mon[97450]: pgmap v13704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:40.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:40.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:40.348715+0000) 2022-09-08T04:41:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:40.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:40.348714+0000) 2022-09-08T04:41:41.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:41.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:41.348992+0000) 2022-09-08T04:41:41.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:41.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:41.348959+0000) 2022-09-08T04:41:42.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:41 smithi055 ceph-mon[97033]: pgmap v13705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:41:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:41 smithi059 ceph-mon[97450]: pgmap v13705: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:41:42.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:42.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:42.349231+0000) 2022-09-08T04:41:42.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:42.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:42.349161+0000) 2022-09-08T04:41:43.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:43.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:43.349430+0000) 2022-09-08T04:41:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:43.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:43.349377+0000) 2022-09-08T04:41:44.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:43 smithi055 ceph-mon[97033]: pgmap v13706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:43 smithi059 ceph-mon[97450]: pgmap v13706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:44.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:44.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:44.349699+0000) 2022-09-08T04:41:44.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:44.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:44.349629+0000) 2022-09-08T04:41:45.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:45.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:45.350019+0000) 2022-09-08T04:41:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:45.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:45.349960+0000) 2022-09-08T04:41:46.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:45 smithi055 ceph-mon[97033]: pgmap v13707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:45 smithi059 ceph-mon[97450]: pgmap v13707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:46.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:46.350208+0000) 2022-09-08T04:41:46.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:41:46.171+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:41:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:46.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:46.350192+0000) 2022-09-08T04:41:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:47.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:47.350434+0000) 2022-09-08T04:41:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:47.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:47.350418+0000) 2022-09-08T04:41:48.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:47 smithi055 ceph-mon[97033]: pgmap v13708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:41:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:47 smithi059 ceph-mon[97450]: pgmap v13708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:41:48.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:48.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:48.350664+0000) 2022-09-08T04:41:48.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:48.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:48.350654+0000) 2022-09-08T04:41:49.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:49.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:49.350866+0000) 2022-09-08T04:41:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:49.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:49.350878+0000) 2022-09-08T04:41:50.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:49 smithi055 ceph-mon[97033]: pgmap v13709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:49 smithi059 ceph-mon[97450]: pgmap v13709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:50.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:50.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:50.351138+0000) 2022-09-08T04:41:50.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:50.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:50.351161+0000) 2022-09-08T04:41:51.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:51.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:51.351306+0000) 2022-09-08T04:41:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:51.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:51.351299+0000) 2022-09-08T04:41:52.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:51 smithi055 ceph-mon[97033]: pgmap v13710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:41:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:51 smithi059 ceph-mon[97450]: pgmap v13710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:41:52.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:52.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:52.351485+0000) 2022-09-08T04:41:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:52.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:52.351484+0000) 2022-09-08T04:41:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:53.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:53.351725+0000) 2022-09-08T04:41:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:53.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:53.351730+0000) 2022-09-08T04:41:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:53 smithi055 ceph-mon[97033]: pgmap v13711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:53 smithi059 ceph-mon[97450]: pgmap v13711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:54.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:54.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:54.351950+0000) 2022-09-08T04:41:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:54.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:54.351982+0000) 2022-09-08T04:41:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:55.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:55.352189+0000) 2022-09-08T04:41:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:55.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:55.352199+0000) 2022-09-08T04:41:56.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:55 smithi055 ceph-mon[97033]: pgmap v13712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:55 smithi059 ceph-mon[97450]: pgmap v13712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:41:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:56.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:56.352442+0000) 2022-09-08T04:41:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:56.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:56.352450+0000) 2022-09-08T04:41:57.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:57.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:57.352601+0000) 2022-09-08T04:41:57.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:57.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:57.352680+0000) 2022-09-08T04:41:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:57 smithi055 ceph-mon[97033]: pgmap v13713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:41:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:57 smithi059 ceph-mon[97450]: pgmap v13713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:41:58.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:58.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:58.352857+0000) 2022-09-08T04:41:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:58.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:58.352828+0000) 2022-09-08T04:41:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:41:59.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:59.353045+0000) 2022-09-08T04:41:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:41:59.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:41:59.353036+0000) 2022-09-08T04:42:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:41:59 smithi055 ceph-mon[97033]: pgmap v13714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:00.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:41:59 smithi059 ceph-mon[97450]: pgmap v13714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:00.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:00.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:00.353255+0000) 2022-09-08T04:42:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:00.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:00.353262+0000) 2022-09-08T04:42:01.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:42:01.182+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:42:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:01.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:01.353509+0000) 2022-09-08T04:42:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:01.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:01.353515+0000) 2022-09-08T04:42:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:01 smithi059 ceph-mon[97450]: pgmap v13715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:42:02.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:01 smithi055 ceph-mon[97033]: pgmap v13715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:42:02.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:02.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:02.353757+0000) 2022-09-08T04:42:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:02.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:02.353763+0000) 2022-09-08T04:42:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:03.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:03.353989+0000) 2022-09-08T04:42:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:03.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:03.353992+0000) 2022-09-08T04:42:04.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:03 smithi059 ceph-mon[97450]: pgmap v13716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:03 smithi055 ceph-mon[97033]: pgmap v13716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:04.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:04.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:04.354286+0000) 2022-09-08T04:42:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:04.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:04.354284+0000) 2022-09-08T04:42:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:05.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:05.354554+0000) 2022-09-08T04:42:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:05.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:05.354494+0000) 2022-09-08T04:42:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:05 smithi059 ceph-mon[97450]: pgmap v13717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:42:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:05 smithi055 ceph-mon[97033]: pgmap v13717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:42:06.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:06.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:06.354765+0000) 2022-09-08T04:42:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:06.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:06.354733+0000) 2022-09-08T04:42:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:42:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:42:07.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:07.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:07.354939+0000) 2022-09-08T04:42:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:07.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:07.354933+0000) 2022-09-08T04:42:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:07 smithi059 ceph-mon[97450]: pgmap v13718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:42:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:07 smithi055 ceph-mon[97033]: pgmap v13718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:42:08.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:08.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:08.355126+0000) 2022-09-08T04:42:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:08.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:08.355199+0000) 2022-09-08T04:42:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:42:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:42:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:42:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:42:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:42:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:42:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:42:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:42:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:09.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:09.355314+0000) 2022-09-08T04:42:09.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:09.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:09.355428+0000) 2022-09-08T04:42:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:10 smithi059 ceph-mon[97450]: pgmap v13719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:42:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:10 smithi059 ceph-mon[97450]: osdmap e938: 12 total, 11 up, 11 in 2022-09-08T04:42:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:10 smithi055 ceph-mon[97033]: pgmap v13719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:42:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:10 smithi055 ceph-mon[97033]: osdmap e938: 12 total, 11 up, 11 in 2022-09-08T04:42:10.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:10.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:10.355458+0000) 2022-09-08T04:42:10.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:10.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:10.355593+0000) 2022-09-08T04:42:11.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:11.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:11.355675+0000) 2022-09-08T04:42:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:11.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:11.355734+0000) 2022-09-08T04:42:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:12 smithi059 ceph-mon[97450]: pgmap v13721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:42:12.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:12 smithi055 ceph-mon[97033]: pgmap v13721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:42:12.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:12.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:12.355789+0000) 2022-09-08T04:42:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:12.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:12.355873+0000) 2022-09-08T04:42:13.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:13.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:13.356027+0000) 2022-09-08T04:42:13.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:13.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:13.356099+0000) 2022-09-08T04:42:14.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:14 smithi059 ceph-mon[97450]: pgmap v13722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:42:14.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:14 smithi055 ceph-mon[97033]: pgmap v13722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:42:14.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:14.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:14.356267+0000) 2022-09-08T04:42:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:14.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:14.356349+0000) 2022-09-08T04:42:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:15.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:15.356436+0000) 2022-09-08T04:42:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:15.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:15.356531+0000) 2022-09-08T04:42:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:16 smithi059 ceph-mon[97450]: pgmap v13723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:42:16.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:16 smithi055 ceph-mon[97033]: pgmap v13723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:42:16.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:16.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:16.356673+0000) 2022-09-08T04:42:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:16.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:16.356787+0000) 2022-09-08T04:42:16.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:42:16.191+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:42:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:17.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:17.356866+0000) 2022-09-08T04:42:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:17.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:17.356961+0000) 2022-09-08T04:42:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:18 smithi059 ceph-mon[97450]: pgmap v13724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:42:18.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:18 smithi055 ceph-mon[97033]: pgmap v13724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:42:18.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:18.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:18.357072+0000) 2022-09-08T04:42:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:18.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:18.357162+0000) 2022-09-08T04:42:19.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:19.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:19.357314+0000) 2022-09-08T04:42:19.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:19.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:19.357352+0000) 2022-09-08T04:42:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:20 smithi059 ceph-mon[97450]: pgmap v13725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:42:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:20 smithi059 ceph-mon[97450]: osdmap e939: 12 total, 11 up, 11 in 2022-09-08T04:42:20.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:20 smithi055 ceph-mon[97033]: pgmap v13725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:42:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:20 smithi055 ceph-mon[97033]: osdmap e939: 12 total, 11 up, 11 in 2022-09-08T04:42:20.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:20.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:20.357503+0000) 2022-09-08T04:42:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:20.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:20.357529+0000) 2022-09-08T04:42:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:21.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:21.357753+0000) 2022-09-08T04:42:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:21.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:21.357740+0000) 2022-09-08T04:42:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:22 smithi059 ceph-mon[97450]: pgmap v13727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:42:22.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:22 smithi055 ceph-mon[97033]: pgmap v13727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:42:22.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:22.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:22.357926+0000) 2022-09-08T04:42:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:22.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:22.357917+0000) 2022-09-08T04:42:23.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:23.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:23.358160+0000) 2022-09-08T04:42:23.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:23.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:23.358159+0000) 2022-09-08T04:42:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:24 smithi059 ceph-mon[97450]: pgmap v13728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:42:24.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:24 smithi055 ceph-mon[97033]: pgmap v13728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:42:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:24.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:24.358395+0000) 2022-09-08T04:42:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:24.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:24.358386+0000) 2022-09-08T04:42:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:25.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:25.358687+0000) 2022-09-08T04:42:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:25.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:25.358678+0000) 2022-09-08T04:42:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:26 smithi055 ceph-mon[97033]: pgmap v13729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:42:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:26 smithi059 ceph-mon[97450]: pgmap v13729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:42:26.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:26.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:26.358955+0000) 2022-09-08T04:42:26.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:26.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:26.358934+0000) 2022-09-08T04:42:27.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:27.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:27.359136+0000) 2022-09-08T04:42:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:27.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:27.359094+0000) 2022-09-08T04:42:28.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:28 smithi055 ceph-mon[97033]: pgmap v13730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:42:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:28 smithi059 ceph-mon[97450]: pgmap v13730: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:42:28.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:28.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:28.359350+0000) 2022-09-08T04:42:28.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:28.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:28.359362+0000) 2022-09-08T04:42:29.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:29.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:29.359570+0000) 2022-09-08T04:42:29.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:29.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:29.359593+0000) 2022-09-08T04:42:30.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:30 smithi055 ceph-mon[97033]: pgmap v13731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:42:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:30 smithi059 ceph-mon[97450]: pgmap v13731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:42:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:30.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:30.359802+0000) 2022-09-08T04:42:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:30.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:30.359814+0000) 2022-09-08T04:42:31.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:42:31.202+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:42:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:31.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:31.359982+0000) 2022-09-08T04:42:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:31.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:31.360055+0000) 2022-09-08T04:42:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:32 smithi055 ceph-mon[97033]: pgmap v13732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:42:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:32 smithi059 ceph-mon[97450]: pgmap v13732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:42:32.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:32.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:32.360195+0000) 2022-09-08T04:42:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:32.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:32.360305+0000) 2022-09-08T04:42:33.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:33.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:33.360405+0000) 2022-09-08T04:42:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:33.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:33.360473+0000) 2022-09-08T04:42:34.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:34 smithi055 ceph-mon[97033]: pgmap v13733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:34 smithi059 ceph-mon[97450]: pgmap v13733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:34.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:34.360608+0000) 2022-09-08T04:42:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:34.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:34.360705+0000) 2022-09-08T04:42:35.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:35.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:35.360802+0000) 2022-09-08T04:42:35.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:35.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:35.360868+0000) 2022-09-08T04:42:36.493 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:36 smithi055 ceph-mon[97033]: pgmap v13734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:36 smithi059 ceph-mon[97450]: pgmap v13734: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:36.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:36.360935+0000) 2022-09-08T04:42:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:36.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:36.361035+0000) 2022-09-08T04:42:37.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:37.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:37.361133+0000) 2022-09-08T04:42:37.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:37.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:37.361225+0000) 2022-09-08T04:42:38.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:38 smithi055 ceph-mon[97033]: pgmap v13735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:42:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:38 smithi059 ceph-mon[97450]: pgmap v13735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:42:38.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:38.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:38.361373+0000) 2022-09-08T04:42:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:38.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:38.361433+0000) 2022-09-08T04:42:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:39.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:39.361579+0000) 2022-09-08T04:42:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:39.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:39.361629+0000) 2022-09-08T04:42:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:40 smithi055 ceph-mon[97033]: pgmap v13736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:40 smithi059 ceph-mon[97450]: pgmap v13736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:40.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:40.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:40.361796+0000) 2022-09-08T04:42:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:40.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:40.361830+0000) 2022-09-08T04:42:41.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:41.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:41.362023+0000) 2022-09-08T04:42:41.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:41.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:41.362068+0000) 2022-09-08T04:42:42.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:42 smithi055 ceph-mon[97033]: pgmap v13737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:42:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:42 smithi059 ceph-mon[97450]: pgmap v13737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:42:42.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:42.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:42.362280+0000) 2022-09-08T04:42:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:42.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:42.362283+0000) 2022-09-08T04:42:43.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:43.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:43.362467+0000) 2022-09-08T04:42:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:43.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:43.362414+0000) 2022-09-08T04:42:44.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:44 smithi055 ceph-mon[97033]: pgmap v13738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:44 smithi059 ceph-mon[97450]: pgmap v13738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:44.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:44.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:44.362652+0000) 2022-09-08T04:42:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:44.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:44.362568+0000) 2022-09-08T04:42:45.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:45.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:45.362855+0000) 2022-09-08T04:42:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:45.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:45.362834+0000) 2022-09-08T04:42:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:46 smithi055 ceph-mon[97033]: pgmap v13739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:46 smithi059 ceph-mon[97450]: pgmap v13739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:46.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:46.363027+0000) 2022-09-08T04:42:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:46.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:46.363042+0000) 2022-09-08T04:42:46.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:42:46.218+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:42:47.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:47.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:47.363178+0000) 2022-09-08T04:42:47.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:47.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:47.363201+0000) 2022-09-08T04:42:48.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:48 smithi055 ceph-mon[97033]: pgmap v13740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:42:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:48 smithi059 ceph-mon[97450]: pgmap v13740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:42:48.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:48.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:48.363371+0000) 2022-09-08T04:42:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:48.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:48.363382+0000) 2022-09-08T04:42:49.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:49.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:49.363523+0000) 2022-09-08T04:42:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:49.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:49.363526+0000) 2022-09-08T04:42:50.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:50 smithi055 ceph-mon[97033]: pgmap v13741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:50 smithi059 ceph-mon[97450]: pgmap v13741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:50.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:50.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:50.363753+0000) 2022-09-08T04:42:50.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:50.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:50.363745+0000) 2022-09-08T04:42:51.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:51.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:51.363974+0000) 2022-09-08T04:42:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:51.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:51.363972+0000) 2022-09-08T04:42:52.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:52 smithi055 ceph-mon[97033]: pgmap v13742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:42:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:52 smithi059 ceph-mon[97450]: pgmap v13742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:42:52.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:52.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:52.364215+0000) 2022-09-08T04:42:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:52.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:52.364203+0000) 2022-09-08T04:42:53.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:53.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:53.364470+0000) 2022-09-08T04:42:53.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:53.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:53.364438+0000) 2022-09-08T04:42:54.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:54 smithi055 ceph-mon[97033]: pgmap v13743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:54 smithi059 ceph-mon[97450]: pgmap v13743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:54.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:54.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:54.364680+0000) 2022-09-08T04:42:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:54.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:54.364628+0000) 2022-09-08T04:42:55.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:55.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:55.364915+0000) 2022-09-08T04:42:55.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:55.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:55.364881+0000) 2022-09-08T04:42:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:56 smithi055 ceph-mon[97033]: pgmap v13744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:56.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:56 smithi059 ceph-mon[97450]: pgmap v13744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:42:56.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:56.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:56.365184+0000) 2022-09-08T04:42:56.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:56.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:56.365071+0000) 2022-09-08T04:42:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:57.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:57.365420+0000) 2022-09-08T04:42:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:57.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:57.365312+0000) 2022-09-08T04:42:58.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:42:58 smithi055 ceph-mon[97033]: pgmap v13745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:42:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:42:58 smithi059 ceph-mon[97450]: pgmap v13745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:42:58.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:58.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:58.365588+0000) 2022-09-08T04:42:58.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:58.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:58.365498+0000) 2022-09-08T04:42:59.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:42:59.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:59.365810+0000) 2022-09-08T04:42:59.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:42:59.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:42:59.365752+0000) 2022-09-08T04:43:00.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:00 smithi055 ceph-mon[97033]: pgmap v13746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:00 smithi059 ceph-mon[97450]: pgmap v13746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:00.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:00.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:00.365967+0000) 2022-09-08T04:43:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:00.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:00.365923+0000) 2022-09-08T04:43:01.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:43:01.233+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:43:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:01.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:01.366107+0000) 2022-09-08T04:43:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:01.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:01.366057+0000) 2022-09-08T04:43:02.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:02 smithi055 ceph-mon[97033]: pgmap v13747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:43:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:02 smithi059 ceph-mon[97450]: pgmap v13747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:43:02.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:02.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:02.366284+0000) 2022-09-08T04:43:02.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:02.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:02.366269+0000) 2022-09-08T04:43:03.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:03.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:03.366476+0000) 2022-09-08T04:43:03.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:03.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:03.366477+0000) 2022-09-08T04:43:04.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:04 smithi055 ceph-mon[97033]: pgmap v13748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:04 smithi059 ceph-mon[97450]: pgmap v13748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:04.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:04.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:04.366684+0000) 2022-09-08T04:43:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:04.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:04.366682+0000) 2022-09-08T04:43:05.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:05.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:05.366910+0000) 2022-09-08T04:43:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:05.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:05.366867+0000) 2022-09-08T04:43:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:06 smithi055 ceph-mon[97033]: pgmap v13749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:43:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:06 smithi059 ceph-mon[97450]: pgmap v13749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:43:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:06.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:06.367102+0000) 2022-09-08T04:43:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:06.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:06.367136+0000) 2022-09-08T04:43:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:43:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:43:07.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:07.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:07.367350+0000) 2022-09-08T04:43:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:07.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:07.367402+0000) 2022-09-08T04:43:08.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:08 smithi055 ceph-mon[97033]: pgmap v13750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:43:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:08 smithi059 ceph-mon[97450]: pgmap v13750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:43:08.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:08.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:08.367590+0000) 2022-09-08T04:43:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:08.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:08.367593+0000) 2022-09-08T04:43:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:43:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:43:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:43:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:43:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:43:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:43:09.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:09.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:09.367809+0000) 2022-09-08T04:43:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:09.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:09.367818+0000) 2022-09-08T04:43:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:10 smithi055 ceph-mon[97033]: pgmap v13751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:43:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:10 smithi055 ceph-mon[97033]: osdmap e940: 12 total, 11 up, 11 in 2022-09-08T04:43:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:43:10.539 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:10 smithi059 ceph-mon[97450]: pgmap v13751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:43:10.540 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:10 smithi059 ceph-mon[97450]: osdmap e940: 12 total, 11 up, 11 in 2022-09-08T04:43:10.540 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:43:10.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:10.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:10.368016+0000) 2022-09-08T04:43:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:10.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:10.367988+0000) 2022-09-08T04:43:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:11.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:11.368309+0000) 2022-09-08T04:43:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:11.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:11.368229+0000) 2022-09-08T04:43:12.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:12 smithi055 ceph-mon[97033]: pgmap v13753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:43:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:12 smithi059 ceph-mon[97450]: pgmap v13753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:43:12.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:12.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:12.368465+0000) 2022-09-08T04:43:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:12.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:12.368519+0000) 2022-09-08T04:43:13.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:13.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:13.368773+0000) 2022-09-08T04:43:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:13.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:13.368746+0000) 2022-09-08T04:43:14.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:14 smithi055 ceph-mon[97033]: pgmap v13754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:43:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:14 smithi059 ceph-mon[97450]: pgmap v13754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:43:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:14.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:14.368928+0000) 2022-09-08T04:43:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:14.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:14.369020+0000) 2022-09-08T04:43:15.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:15.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:15.369177+0000) 2022-09-08T04:43:15.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:15.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:15.369299+0000) 2022-09-08T04:43:16.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:16 smithi055 ceph-mon[97033]: pgmap v13755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:43:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:16 smithi059 ceph-mon[97450]: pgmap v13755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:43:16.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:16.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:16.369387+0000) 2022-09-08T04:43:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:16.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:16.369589+0000) 2022-09-08T04:43:16.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:43:16.247+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:43:17.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:17.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:17.369603+0000) 2022-09-08T04:43:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:17.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:17.369747+0000) 2022-09-08T04:43:18.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:18 smithi055 ceph-mon[97033]: pgmap v13756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:43:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:18 smithi059 ceph-mon[97450]: pgmap v13756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:43:18.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:18.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:18.369788+0000) 2022-09-08T04:43:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:18.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:18.369902+0000) 2022-09-08T04:43:19.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:19.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:19.369950+0000) 2022-09-08T04:43:19.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:19.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:19.370067+0000) 2022-09-08T04:43:20.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:20 smithi055 ceph-mon[97033]: pgmap v13757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:43:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:20 smithi055 ceph-mon[97033]: osdmap e941: 12 total, 11 up, 11 in 2022-09-08T04:43:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:20 smithi059 ceph-mon[97450]: pgmap v13757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:43:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:20 smithi059 ceph-mon[97450]: osdmap e941: 12 total, 11 up, 11 in 2022-09-08T04:43:20.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:20.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:20.370248+0000) 2022-09-08T04:43:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:20.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:20.370152+0000) 2022-09-08T04:43:21.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:21.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:21.370431+0000) 2022-09-08T04:43:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:21.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:21.370324+0000) 2022-09-08T04:43:22.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:22 smithi055 ceph-mon[97033]: pgmap v13759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:43:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:22 smithi059 ceph-mon[97450]: pgmap v13759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:43:22.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:22.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:22.370629+0000) 2022-09-08T04:43:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:22.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:22.370574+0000) 2022-09-08T04:43:23.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:23.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:23.370806+0000) 2022-09-08T04:43:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:23.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:23.370748+0000) 2022-09-08T04:43:24.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:24 smithi055 ceph-mon[97033]: pgmap v13760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:43:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:24 smithi059 ceph-mon[97450]: pgmap v13760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:43:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:24.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:24.370895+0000) 2022-09-08T04:43:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:24.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:24.370949+0000) 2022-09-08T04:43:25.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:25.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:25.371155+0000) 2022-09-08T04:43:25.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:25.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:25.371172+0000) 2022-09-08T04:43:26.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:26 smithi055 ceph-mon[97033]: pgmap v13761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:43:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:26 smithi059 ceph-mon[97450]: pgmap v13761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:43:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:26.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:26.371395+0000) 2022-09-08T04:43:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:26.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:26.371409+0000) 2022-09-08T04:43:27.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:27.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:27.371612+0000) 2022-09-08T04:43:27.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:27.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:27.371565+0000) 2022-09-08T04:43:28.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:28 smithi055 ceph-mon[97033]: pgmap v13762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:43:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:28 smithi059 ceph-mon[97450]: pgmap v13762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:43:28.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:28.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:28.371757+0000) 2022-09-08T04:43:28.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:28.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:28.371734+0000) 2022-09-08T04:43:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:29.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:29.371948+0000) 2022-09-08T04:43:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:29.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:29.371953+0000) 2022-09-08T04:43:30.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:30 smithi055 ceph-mon[97033]: pgmap v13763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:43:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:30 smithi059 ceph-mon[97450]: pgmap v13763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:43:30.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:30.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:30.372124+0000) 2022-09-08T04:43:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:30.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:30.372137+0000) 2022-09-08T04:43:31.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:43:31.254+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:43:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:31.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:31.372295+0000) 2022-09-08T04:43:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:31.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:31.372321+0000) 2022-09-08T04:43:32.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:32 smithi055 ceph-mon[97033]: pgmap v13764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:43:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:32 smithi059 ceph-mon[97450]: pgmap v13764: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:43:32.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:32.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:32.372540+0000) 2022-09-08T04:43:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:32.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:32.372558+0000) 2022-09-08T04:43:33.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:33 smithi055 ceph-mon[97033]: pgmap v13765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:33 smithi059 ceph-mon[97450]: pgmap v13765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:33.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:33.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:33.372800+0000) 2022-09-08T04:43:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:33.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:33.372803+0000) 2022-09-08T04:43:34.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:34.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:34.372960+0000) 2022-09-08T04:43:34.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:34.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:34.373021+0000) 2022-09-08T04:43:35.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:35.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:35.373201+0000) 2022-09-08T04:43:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:35.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:35.373323+0000) 2022-09-08T04:43:36.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:35 smithi055 ceph-mon[97033]: pgmap v13766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:35 smithi059 ceph-mon[97450]: pgmap v13766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:36.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:36.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:36.373451+0000) 2022-09-08T04:43:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:36.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:36.373551+0000) 2022-09-08T04:43:37.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:37.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:37.373743+0000) 2022-09-08T04:43:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:37.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:37.373723+0000) 2022-09-08T04:43:38.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:37 smithi055 ceph-mon[97033]: pgmap v13767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:43:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:37 smithi059 ceph-mon[97450]: pgmap v13767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:43:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:38.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:38.373890+0000) 2022-09-08T04:43:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:38.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:38.373932+0000) 2022-09-08T04:43:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:39.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:39.374132+0000) 2022-09-08T04:43:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:39.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:39.374248+0000) 2022-09-08T04:43:40.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:39 smithi055 ceph-mon[97033]: pgmap v13768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:40.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:39 smithi059 ceph-mon[97450]: pgmap v13768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:40.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:40.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:40.374314+0000) 2022-09-08T04:43:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:40.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:40.374464+0000) 2022-09-08T04:43:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:41.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:41.374560+0000) 2022-09-08T04:43:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:41.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:41.374694+0000) 2022-09-08T04:43:42.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:41 smithi055 ceph-mon[97033]: pgmap v13769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:43:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:41 smithi059 ceph-mon[97450]: pgmap v13769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:43:42.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:42.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:42.374704+0000) 2022-09-08T04:43:42.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:42.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:42.374831+0000) 2022-09-08T04:43:43.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:43.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:43.374855+0000) 2022-09-08T04:43:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:43.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:43.374967+0000) 2022-09-08T04:43:44.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:43 smithi055 ceph-mon[97033]: pgmap v13770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:43 smithi059 ceph-mon[97450]: pgmap v13770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:44.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:44.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:44.375017+0000) 2022-09-08T04:43:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:44.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:44.375116+0000) 2022-09-08T04:43:45.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:45.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:45.375201+0000) 2022-09-08T04:43:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:45.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:45.375307+0000) 2022-09-08T04:43:46.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:45 smithi055 ceph-mon[97033]: pgmap v13771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:45 smithi059 ceph-mon[97450]: pgmap v13771: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:46.662 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:46.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:46.375375+0000) 2022-09-08T04:43:46.664 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:43:46.261+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:43:46.665 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:46.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:46.375556+0000) 2022-09-08T04:43:47.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:47.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:47.375578+0000) 2022-09-08T04:43:47.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:47.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:47.375725+0000) 2022-09-08T04:43:48.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:47 smithi055 ceph-mon[97033]: pgmap v13772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:43:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:47 smithi059 ceph-mon[97450]: pgmap v13772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:43:48.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:48.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:48.375812+0000) 2022-09-08T04:43:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:48.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:48.375911+0000) 2022-09-08T04:43:49.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:49.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:49.375969+0000) 2022-09-08T04:43:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:49.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:49.376153+0000) 2022-09-08T04:43:50.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:49 smithi055 ceph-mon[97033]: pgmap v13773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:49 smithi059 ceph-mon[97450]: pgmap v13773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:50.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:50.376142+0000) 2022-09-08T04:43:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:50.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:50.376327+0000) 2022-09-08T04:43:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:51.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:51.376381+0000) 2022-09-08T04:43:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:51.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:51.376559+0000) 2022-09-08T04:43:52.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:51 smithi055 ceph-mon[97033]: pgmap v13774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:43:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:51 smithi059 ceph-mon[97450]: pgmap v13774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:43:52.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:52.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:52.376796+0000) 2022-09-08T04:43:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:52.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:52.376605+0000) 2022-09-08T04:43:53.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:53.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:53.376867+0000) 2022-09-08T04:43:53.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:53.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:53.376996+0000) 2022-09-08T04:43:54.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:53 smithi055 ceph-mon[97033]: pgmap v13775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:54.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:53 smithi059 ceph-mon[97450]: pgmap v13775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:54.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:54.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:54.377126+0000) 2022-09-08T04:43:54.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:54.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:54.377241+0000) 2022-09-08T04:43:55.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:55.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:55.377441+0000) 2022-09-08T04:43:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:55.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:55.377349+0000) 2022-09-08T04:43:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:55 smithi059 ceph-mon[97450]: pgmap v13776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:56.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:55 smithi055 ceph-mon[97033]: pgmap v13776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:43:56.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:56.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:56.377583+0000) 2022-09-08T04:43:56.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:56.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:56.377706+0000) 2022-09-08T04:43:57.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:57.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:57.377906+0000) 2022-09-08T04:43:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:57.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:57.377812+0000) 2022-09-08T04:43:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:57 smithi059 ceph-mon[97450]: pgmap v13777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:43:58.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:57 smithi055 ceph-mon[97033]: pgmap v13777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:43:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:58.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:58.378025+0000) 2022-09-08T04:43:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:58.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:58.378062+0000) 2022-09-08T04:43:59.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:43:59.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:59.378145+0000) 2022-09-08T04:43:59.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:43:59.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:43:59.378228+0000) 2022-09-08T04:44:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:43:59 smithi059 ceph-mon[97450]: pgmap v13778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:00.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:43:59 smithi055 ceph-mon[97033]: pgmap v13778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:00.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:00.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:00.378275+0000) 2022-09-08T04:44:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:00.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:00.378372+0000) 2022-09-08T04:44:01.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:01.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:01.378489+0000) 2022-09-08T04:44:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:01.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:01.378624+0000) 2022-09-08T04:44:01.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:44:01.261+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:44:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:01 smithi059 ceph-mon[97450]: pgmap v13779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:44:02.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:01 smithi055 ceph-mon[97033]: pgmap v13779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:44:02.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:02.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:02.378758+0000) 2022-09-08T04:44:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:02.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:02.378884+0000) 2022-09-08T04:44:03.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:03.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:03.378895+0000) 2022-09-08T04:44:03.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:03.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:03.379029+0000) 2022-09-08T04:44:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:03 smithi059 ceph-mon[97450]: pgmap v13780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:04.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:03 smithi055 ceph-mon[97033]: pgmap v13780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:04.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:04.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:04.379065+0000) 2022-09-08T04:44:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:04.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:04.379194+0000) 2022-09-08T04:44:05.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:05.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:05.379224+0000) 2022-09-08T04:44:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:05.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:05.379368+0000) 2022-09-08T04:44:06.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:05 smithi059 ceph-mon[97450]: pgmap v13781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:44:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:05 smithi055 ceph-mon[97033]: pgmap v13781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:44:06.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:06.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:06.379471+0000) 2022-09-08T04:44:06.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:06.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:06.379603+0000) 2022-09-08T04:44:07.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:44:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:44:07.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:07.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:07.379648+0000) 2022-09-08T04:44:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:07.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:07.379797+0000) 2022-09-08T04:44:08.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:08 smithi059 ceph-mon[97450]: pgmap v13782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:44:08.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:08 smithi055 ceph-mon[97033]: pgmap v13782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:44:08.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:08.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:08.379812+0000) 2022-09-08T04:44:08.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:08.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:08.379931+0000) 2022-09-08T04:44:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:09.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:09.379976+0000) 2022-09-08T04:44:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:09.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:09.380144+0000) 2022-09-08T04:44:10.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:10 smithi055 ceph-mon[97033]: pgmap v13783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:44:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:10 smithi055 ceph-mon[97033]: osdmap e942: 12 total, 11 up, 11 in 2022-09-08T04:44:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:44:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:44:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:44:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:44:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:10 smithi059 ceph-mon[97450]: pgmap v13783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:44:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:10 smithi059 ceph-mon[97450]: osdmap e942: 12 total, 11 up, 11 in 2022-09-08T04:44:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:44:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:44:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:44:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:44:10.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:10.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:10.380380+0000) 2022-09-08T04:44:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:10.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:10.380217+0000) 2022-09-08T04:44:11.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:11.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:11.380467+0000) 2022-09-08T04:44:11.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:11.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:11.380617+0000) 2022-09-08T04:44:12.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:12 smithi059 ceph-mon[97450]: pgmap v13785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:44:12.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:12 smithi055 ceph-mon[97033]: pgmap v13785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:44:12.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:12.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:12.380701+0000) 2022-09-08T04:44:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:12.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:12.380799+0000) 2022-09-08T04:44:13.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:13.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:13.381023+0000) 2022-09-08T04:44:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:13.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:13.380924+0000) 2022-09-08T04:44:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:14 smithi059 ceph-mon[97450]: pgmap v13786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:44:14.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:14 smithi055 ceph-mon[97033]: pgmap v13786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:44:14.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:14.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:14.381245+0000) 2022-09-08T04:44:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:14.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:14.381259+0000) 2022-09-08T04:44:15.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:15.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:15.381490+0000) 2022-09-08T04:44:15.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:15.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:15.381483+0000) 2022-09-08T04:44:16.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:15 smithi055 ceph-mon[97033]: pgmap v13787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:44:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:15 smithi055 ceph-mon[97033]: osdmap e943: 12 total, 11 up, 11 in 2022-09-08T04:44:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:15 smithi059 ceph-mon[97450]: pgmap v13787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:44:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:15 smithi059 ceph-mon[97450]: osdmap e943: 12 total, 11 up, 11 in 2022-09-08T04:44:16.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:44:16.272+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:44:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:16.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:16.381696+0000) 2022-09-08T04:44:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:16.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:16.381646+0000) 2022-09-08T04:44:17.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:17.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:17.381859+0000) 2022-09-08T04:44:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:17.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:17.381796+0000) 2022-09-08T04:44:18.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:17 smithi055 ceph-mon[97033]: pgmap v13789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:44:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:17 smithi059 ceph-mon[97450]: pgmap v13789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:44:18.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:18.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:18.381994+0000) 2022-09-08T04:44:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:18.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:18.381935+0000) 2022-09-08T04:44:19.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:19.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:19.382145+0000) 2022-09-08T04:44:19.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:19.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:19.382130+0000) 2022-09-08T04:44:20.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:19 smithi055 ceph-mon[97033]: pgmap v13790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2022-09-08T04:44:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:19 smithi059 ceph-mon[97450]: pgmap v13790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2022-09-08T04:44:20.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:20.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:20.382306+0000) 2022-09-08T04:44:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:20.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:20.382306+0000) 2022-09-08T04:44:21.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:21.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:21.382540+0000) 2022-09-08T04:44:21.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:21.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:21.382469+0000) 2022-09-08T04:44:22.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:21 smithi055 ceph-mon[97033]: pgmap v13791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:44:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:21 smithi059 ceph-mon[97450]: pgmap v13791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:44:22.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:22.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:22.382821+0000) 2022-09-08T04:44:22.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:22.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:22.382723+0000) 2022-09-08T04:44:23.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:23.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:23.383061+0000) 2022-09-08T04:44:23.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:23.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:23.382911+0000) 2022-09-08T04:44:24.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:23 smithi055 ceph-mon[97033]: pgmap v13792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:44:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:23 smithi059 ceph-mon[97450]: pgmap v13792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:44:24.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:24.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:24.383305+0000) 2022-09-08T04:44:24.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:24.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:24.383164+0000) 2022-09-08T04:44:25.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:25.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:25.383488+0000) 2022-09-08T04:44:25.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:25.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:25.383380+0000) 2022-09-08T04:44:26.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:25 smithi055 ceph-mon[97033]: pgmap v13793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:44:26.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:25 smithi059 ceph-mon[97450]: pgmap v13793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:44:26.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:26.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:26.383600+0000) 2022-09-08T04:44:26.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:26.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:26.383761+0000) 2022-09-08T04:44:27.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:27.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:27.383916+0000) 2022-09-08T04:44:27.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:27.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:27.383792+0000) 2022-09-08T04:44:28.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:27 smithi055 ceph-mon[97033]: pgmap v13794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:44:28.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:27 smithi059 ceph-mon[97450]: pgmap v13794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:44:28.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:28.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:28.384129+0000) 2022-09-08T04:44:28.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:28.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:28.384037+0000) 2022-09-08T04:44:29.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:29.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:29.384379+0000) 2022-09-08T04:44:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:29.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:29.384331+0000) 2022-09-08T04:44:30.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:29 smithi055 ceph-mon[97033]: pgmap v13795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:29 smithi059 ceph-mon[97450]: pgmap v13795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:30.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:30.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:30.384574+0000) 2022-09-08T04:44:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:30.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:30.384597+0000) 2022-09-08T04:44:31.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:31.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:31.384811+0000) 2022-09-08T04:44:31.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:31.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:31.384853+0000) 2022-09-08T04:44:31.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:44:31.284+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:44:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:31 smithi059 ceph-mon[97450]: pgmap v13796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:44:32.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:31 smithi055 ceph-mon[97033]: pgmap v13796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:44:32.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:32.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:32.385051+0000) 2022-09-08T04:44:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:32.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:32.385025+0000) 2022-09-08T04:44:33.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:33.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:33.385307+0000) 2022-09-08T04:44:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:33.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:33.385292+0000) 2022-09-08T04:44:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:33 smithi059 ceph-mon[97450]: pgmap v13797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:34.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:33 smithi055 ceph-mon[97033]: pgmap v13797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:34.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:34.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:34.385506+0000) 2022-09-08T04:44:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:34.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:34.385513+0000) 2022-09-08T04:44:35.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:35.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:35.385684+0000) 2022-09-08T04:44:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:35.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:35.385727+0000) 2022-09-08T04:44:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:35 smithi059 ceph-mon[97450]: pgmap v13798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:36.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:35 smithi055 ceph-mon[97033]: pgmap v13798: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:36.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:36.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:36.385803+0000) 2022-09-08T04:44:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:36.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:36.385870+0000) 2022-09-08T04:44:37.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:37.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:37.386008+0000) 2022-09-08T04:44:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:37.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:37.386044+0000) 2022-09-08T04:44:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:37 smithi059 ceph-mon[97450]: pgmap v13799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:44:38.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:37 smithi055 ceph-mon[97033]: pgmap v13799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:44:38.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:38.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:38.386219+0000) 2022-09-08T04:44:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:38.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:38.386247+0000) 2022-09-08T04:44:39.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:39.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:39.386368+0000) 2022-09-08T04:44:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:39.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:39.386458+0000) 2022-09-08T04:44:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:39 smithi059 ceph-mon[97450]: pgmap v13800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:39 smithi055 ceph-mon[97033]: pgmap v13800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:40.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:40.386590+0000) 2022-09-08T04:44:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:40.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:40.386680+0000) 2022-09-08T04:44:41.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:41.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:41.386810+0000) 2022-09-08T04:44:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:41.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:41.386878+0000) 2022-09-08T04:44:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:41 smithi059 ceph-mon[97450]: pgmap v13801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:44:42.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:41 smithi055 ceph-mon[97033]: pgmap v13801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:44:42.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:42.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:42.386974+0000) 2022-09-08T04:44:42.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:42.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:42.387039+0000) 2022-09-08T04:44:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:43.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:43.387189+0000) 2022-09-08T04:44:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:43.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:43.387241+0000) 2022-09-08T04:44:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:43 smithi059 ceph-mon[97450]: pgmap v13802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:44.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:44 smithi055 ceph-mon[97033]: pgmap v13802: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:44.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:44.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:44.387398+0000) 2022-09-08T04:44:44.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:44.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:44.387496+0000) 2022-09-08T04:44:45.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:45.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:45.387684+0000) 2022-09-08T04:44:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:45.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:45.387776+0000) 2022-09-08T04:44:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:46 smithi059 ceph-mon[97450]: pgmap v13803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:46.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:46 smithi055 ceph-mon[97033]: pgmap v13803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:46.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:44:46.285+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:44:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:46.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:46.387839+0000) 2022-09-08T04:44:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:46.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:46.387959+0000) 2022-09-08T04:44:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:47.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:47.388002+0000) 2022-09-08T04:44:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:47.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:47.388161+0000) 2022-09-08T04:44:48.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:48 smithi059 ceph-mon[97450]: pgmap v13804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:44:48.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:48 smithi055 ceph-mon[97033]: pgmap v13804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:44:48.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:48.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:48.388204+0000) 2022-09-08T04:44:48.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:48.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:48.388320+0000) 2022-09-08T04:44:49.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:49.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:49.388436+0000) 2022-09-08T04:44:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:49.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:49.388571+0000) 2022-09-08T04:44:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:50 smithi059 ceph-mon[97450]: pgmap v13805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:50.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:50 smithi055 ceph-mon[97033]: pgmap v13805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:50.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:50.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:50.388711+0000) 2022-09-08T04:44:50.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:50.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:50.388827+0000) 2022-09-08T04:44:51.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:51.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:51.388834+0000) 2022-09-08T04:44:51.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:51.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:51.388981+0000) 2022-09-08T04:44:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:52 smithi059 ceph-mon[97450]: pgmap v13806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:44:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:52 smithi055 ceph-mon[97033]: pgmap v13806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:44:52.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:52.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:52.389028+0000) 2022-09-08T04:44:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:52.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:52.389234+0000) 2022-09-08T04:44:53.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:53.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:53.389321+0000) 2022-09-08T04:44:53.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:53.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:53.389500+0000) 2022-09-08T04:44:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:54 smithi059 ceph-mon[97450]: pgmap v13807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:54.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:54 smithi055 ceph-mon[97033]: pgmap v13807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:54.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:54.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:54.389584+0000) 2022-09-08T04:44:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:54.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:54.389727+0000) 2022-09-08T04:44:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:55.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:55.389771+0000) 2022-09-08T04:44:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:55.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:55.389907+0000) 2022-09-08T04:44:56.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:56 smithi059 ceph-mon[97450]: pgmap v13808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:56.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:56 smithi055 ceph-mon[97033]: pgmap v13808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:44:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:56.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:56.389913+0000) 2022-09-08T04:44:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:56.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:56.390066+0000) 2022-09-08T04:44:57.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:57.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:57.390112+0000) 2022-09-08T04:44:57.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:57.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:57.390253+0000) 2022-09-08T04:44:58.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:44:58 smithi059 ceph-mon[97450]: pgmap v13809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:44:58.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:44:58 smithi055 ceph-mon[97033]: pgmap v13809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:44:58.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:58.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:58.390268+0000) 2022-09-08T04:44:58.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:58.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:58.390413+0000) 2022-09-08T04:44:59.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:44:59.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:59.390595+0000) 2022-09-08T04:44:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:44:59.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:44:59.390464+0000) 2022-09-08T04:45:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:00 smithi059 ceph-mon[97450]: pgmap v13810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:00.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:00 smithi055 ceph-mon[97033]: pgmap v13810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:00.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:00.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:00.390609+0000) 2022-09-08T04:45:00.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:00.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:00.390719+0000) 2022-09-08T04:45:01.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:45:01.300+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:45:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:01.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:01.390843+0000) 2022-09-08T04:45:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:01.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:01.390951+0000) 2022-09-08T04:45:02.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:02 smithi055 ceph-mon[97033]: pgmap v13811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:45:02.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:02 smithi059 ceph-mon[97450]: pgmap v13811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:45:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:02.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:02.391105+0000) 2022-09-08T04:45:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:02.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:02.391169+0000) 2022-09-08T04:45:03.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:03.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:03.391228+0000) 2022-09-08T04:45:03.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:03.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:03.391315+0000) 2022-09-08T04:45:04.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:04 smithi055 ceph-mon[97033]: pgmap v13812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:04 smithi059 ceph-mon[97450]: pgmap v13812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:04.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:04.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:04.391453+0000) 2022-09-08T04:45:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:04.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:04.391516+0000) 2022-09-08T04:45:05.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:05.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:05.391706+0000) 2022-09-08T04:45:05.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:05.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:05.391780+0000) 2022-09-08T04:45:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:06 smithi055 ceph-mon[97033]: pgmap v13813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:45:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:06 smithi059 ceph-mon[97450]: pgmap v13813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:45:06.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:06.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:06.391939+0000) 2022-09-08T04:45:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:06.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:06.391979+0000) 2022-09-08T04:45:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:45:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:45:07.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:07.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:07.392208+0000) 2022-09-08T04:45:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:07.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:07.392192+0000) 2022-09-08T04:45:08.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:08 smithi055 ceph-mon[97033]: pgmap v13814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:45:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:08 smithi059 ceph-mon[97450]: pgmap v13814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:45:08.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:08.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:08.392429+0000) 2022-09-08T04:45:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:08.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:08.392484+0000) 2022-09-08T04:45:09.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:09.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:09.392601+0000) 2022-09-08T04:45:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:09.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:09.392670+0000) 2022-09-08T04:45:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:10 smithi059 ceph-mon[97450]: pgmap v13815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:45:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:10 smithi059 ceph-mon[97450]: osdmap e944: 12 total, 11 up, 11 in 2022-09-08T04:45:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:45:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:45:10.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:45:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:10 smithi055 ceph-mon[97033]: pgmap v13815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:45:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:10 smithi055 ceph-mon[97033]: osdmap e944: 12 total, 11 up, 11 in 2022-09-08T04:45:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:45:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:45:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:45:10.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:10.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:10.392786+0000) 2022-09-08T04:45:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:10.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:10.392838+0000) 2022-09-08T04:45:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:11.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:11.392990+0000) 2022-09-08T04:45:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:11.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:11.392968+0000) 2022-09-08T04:45:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:11 smithi055 ceph-mon[97033]: pgmap v13817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:45:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:45:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:11 smithi059 ceph-mon[97450]: pgmap v13817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:45:11.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:45:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:12.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:12.393195+0000) 2022-09-08T04:45:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:12.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:12.393161+0000) 2022-09-08T04:45:13.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:13.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:13.393422+0000) 2022-09-08T04:45:13.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:13.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:13.393405+0000) 2022-09-08T04:45:13.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:13 smithi055 ceph-mon[97033]: pgmap v13818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:45:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:13 smithi059 ceph-mon[97450]: pgmap v13818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:45:14.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:14.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:14.393616+0000) 2022-09-08T04:45:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:14.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:14.393600+0000) 2022-09-08T04:45:15.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:15.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:15.393839+0000) 2022-09-08T04:45:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:15.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:15.393842+0000) 2022-09-08T04:45:16.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:15 smithi055 ceph-mon[97033]: pgmap v13819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:45:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:15 smithi055 ceph-mon[97033]: osdmap e945: 12 total, 11 up, 11 in 2022-09-08T04:45:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:15 smithi059 ceph-mon[97450]: pgmap v13819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:45:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:15 smithi059 ceph-mon[97450]: osdmap e945: 12 total, 11 up, 11 in 2022-09-08T04:45:16.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:45:16.317+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:45:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:16.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:16.393991+0000) 2022-09-08T04:45:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:16.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:16.394014+0000) 2022-09-08T04:45:17.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:17.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:17.394214+0000) 2022-09-08T04:45:17.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:17.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:17.394257+0000) 2022-09-08T04:45:18.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:17 smithi055 ceph-mon[97033]: pgmap v13821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:45:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:17 smithi059 ceph-mon[97450]: pgmap v13821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:45:18.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:18.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:18.394465+0000) 2022-09-08T04:45:18.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:18.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:18.394522+0000) 2022-09-08T04:45:19.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:19.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:19.394699+0000) 2022-09-08T04:45:19.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:19.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:19.394721+0000) 2022-09-08T04:45:20.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:19 smithi055 ceph-mon[97033]: pgmap v13822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2022-09-08T04:45:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:19 smithi059 ceph-mon[97450]: pgmap v13822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2022-09-08T04:45:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:20.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:20.394869+0000) 2022-09-08T04:45:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:20.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:20.394963+0000) 2022-09-08T04:45:21.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:21.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:21.395098+0000) 2022-09-08T04:45:21.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:21.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:21.395205+0000) 2022-09-08T04:45:22.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:21 smithi059 ceph-mon[97450]: pgmap v13823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:45:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:21 smithi055 ceph-mon[97033]: pgmap v13823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:45:22.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:22.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:22.395344+0000) 2022-09-08T04:45:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:22.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:22.395401+0000) 2022-09-08T04:45:23.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:23.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:23.395622+0000) 2022-09-08T04:45:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:23.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:23.395624+0000) 2022-09-08T04:45:24.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:23 smithi059 ceph-mon[97450]: pgmap v13824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:45:24.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:23 smithi055 ceph-mon[97033]: pgmap v13824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:45:24.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:24.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:24.395855+0000) 2022-09-08T04:45:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:24.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:24.395876+0000) 2022-09-08T04:45:25.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:25.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:25.396117+0000) 2022-09-08T04:45:25.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:25.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:25.396123+0000) 2022-09-08T04:45:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:25 smithi059 ceph-mon[97450]: pgmap v13825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:45:26.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:25 smithi055 ceph-mon[97033]: pgmap v13825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:45:26.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:26.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:26.396400+0000) 2022-09-08T04:45:26.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:26.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:26.396403+0000) 2022-09-08T04:45:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:27.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:27.396679+0000) 2022-09-08T04:45:27.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:27.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:27.396677+0000) 2022-09-08T04:45:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:27 smithi059 ceph-mon[97450]: pgmap v13826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:45:28.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:27 smithi055 ceph-mon[97033]: pgmap v13826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:45:28.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:28.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:28.396865+0000) 2022-09-08T04:45:28.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:28.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:28.396857+0000) 2022-09-08T04:45:29.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:29.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:29.397037+0000) 2022-09-08T04:45:29.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:29.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:29.397057+0000) 2022-09-08T04:45:30.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:29 smithi059 ceph-mon[97450]: pgmap v13827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:30.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:29 smithi055 ceph-mon[97033]: pgmap v13827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:30.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:30.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:30.397235+0000) 2022-09-08T04:45:30.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:30.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:30.397256+0000) 2022-09-08T04:45:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:31.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:31.397461+0000) 2022-09-08T04:45:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:31.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:31.397464+0000) 2022-09-08T04:45:31.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:45:31.326+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:45:32.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:31 smithi059 ceph-mon[97450]: pgmap v13828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:45:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:31 smithi055 ceph-mon[97033]: pgmap v13828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:45:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:32.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:32.397674+0000) 2022-09-08T04:45:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:32.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:32.397675+0000) 2022-09-08T04:45:33.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:33.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:33.397869+0000) 2022-09-08T04:45:33.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:33.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:33.397860+0000) 2022-09-08T04:45:34.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:33 smithi059 ceph-mon[97450]: pgmap v13829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:33 smithi055 ceph-mon[97033]: pgmap v13829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:34.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:34.398071+0000) 2022-09-08T04:45:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:34.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:34.398084+0000) 2022-09-08T04:45:35.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:35.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:35.398243+0000) 2022-09-08T04:45:35.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:35.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:35.398249+0000) 2022-09-08T04:45:36.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:35 smithi059 ceph-mon[97450]: pgmap v13830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:36.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:36 smithi055 ceph-mon[97033]: pgmap v13830: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:36.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:36.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:36.398503+0000) 2022-09-08T04:45:36.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:36.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:36.398484+0000) 2022-09-08T04:45:37.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:37.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:37.398698+0000) 2022-09-08T04:45:37.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:37.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:37.398691+0000) 2022-09-08T04:45:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:38 smithi059 ceph-mon[97450]: pgmap v13831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:45:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:38 smithi055 ceph-mon[97033]: pgmap v13831: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:45:38.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:38.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:38.398861+0000) 2022-09-08T04:45:38.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:38.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:38.398816+0000) 2022-09-08T04:45:39.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:39.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:39.399015+0000) 2022-09-08T04:45:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:39.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:39.399040+0000) 2022-09-08T04:45:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:40 smithi059 ceph-mon[97450]: pgmap v13832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:40.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:40 smithi055 ceph-mon[97033]: pgmap v13832: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:40.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:40.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:40.399236+0000) 2022-09-08T04:45:40.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:40.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:40.399220+0000) 2022-09-08T04:45:41.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:41.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:41.399392+0000) 2022-09-08T04:45:41.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:41.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:41.399403+0000) 2022-09-08T04:45:42.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:42 smithi059 ceph-mon[97450]: pgmap v13833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:45:42.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:42 smithi055 ceph-mon[97033]: pgmap v13833: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:45:42.650 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:42.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:42.399589+0000) 2022-09-08T04:45:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:42.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:42.399576+0000) 2022-09-08T04:45:43.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:43.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:43.399817+0000) 2022-09-08T04:45:43.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:43.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:43.399808+0000) 2022-09-08T04:45:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:44 smithi059 ceph-mon[97450]: pgmap v13834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:44.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:44 smithi055 ceph-mon[97033]: pgmap v13834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:44.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:44.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:44.399960+0000) 2022-09-08T04:45:44.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:44.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:44.400048+0000) 2022-09-08T04:45:45.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:45.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:45.400115+0000) 2022-09-08T04:45:45.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:45.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:45.400216+0000) 2022-09-08T04:45:46.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:46 smithi059 ceph-mon[97450]: pgmap v13835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:46.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:46 smithi055 ceph-mon[97033]: pgmap v13835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:46.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:46.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:46.400307+0000) 2022-09-08T04:45:46.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:46.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:46.400396+0000) 2022-09-08T04:45:46.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:45:46.326+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:45:47.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:47.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:47.400530+0000) 2022-09-08T04:45:47.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:47.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:47.400604+0000) 2022-09-08T04:45:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:48 smithi059 ceph-mon[97450]: pgmap v13836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:45:48.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:48 smithi055 ceph-mon[97033]: pgmap v13836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:45:48.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:48.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:48.400704+0000) 2022-09-08T04:45:48.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:48.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:48.400723+0000) 2022-09-08T04:45:49.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:49.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:49.400874+0000) 2022-09-08T04:45:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:49.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:49.400950+0000) 2022-09-08T04:45:50.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:50 smithi059 ceph-mon[97450]: pgmap v13837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:50.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:50 smithi055 ceph-mon[97033]: pgmap v13837: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:50.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:50.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:50.401148+0000) 2022-09-08T04:45:50.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:50.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:50.401255+0000) 2022-09-08T04:45:51.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:51.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:51.401382+0000) 2022-09-08T04:45:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:51.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:51.401466+0000) 2022-09-08T04:45:52.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:52 smithi059 ceph-mon[97450]: pgmap v13838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:45:52.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:52 smithi055 ceph-mon[97033]: pgmap v13838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:45:52.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:52.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:52.401685+0000) 2022-09-08T04:45:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:52.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:52.401700+0000) 2022-09-08T04:45:53.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:53.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:53.401867+0000) 2022-09-08T04:45:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:53.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:53.401868+0000) 2022-09-08T04:45:54.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:54 smithi055 ceph-mon[97033]: pgmap v13839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:54 smithi059 ceph-mon[97450]: pgmap v13839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:54.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:54.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:54.402036+0000) 2022-09-08T04:45:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:54.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:54.402075+0000) 2022-09-08T04:45:55.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:55.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:55.402336+0000) 2022-09-08T04:45:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:55.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:55.402313+0000) 2022-09-08T04:45:56.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:56 smithi055 ceph-mon[97033]: pgmap v13840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:56 smithi059 ceph-mon[97450]: pgmap v13840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:45:56.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:56.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:56.402554+0000) 2022-09-08T04:45:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:56.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:56.402602+0000) 2022-09-08T04:45:57.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:57.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:57.402720+0000) 2022-09-08T04:45:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:57.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:57.402814+0000) 2022-09-08T04:45:58.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:45:58 smithi055 ceph-mon[97033]: pgmap v13841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:45:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:45:58 smithi059 ceph-mon[97450]: pgmap v13841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:45:58.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:58.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:58.402838+0000) 2022-09-08T04:45:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:58.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:58.402956+0000) 2022-09-08T04:45:59.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:45:59.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:59.402963+0000) 2022-09-08T04:45:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:45:59.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:45:59.403116+0000) 2022-09-08T04:46:00.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:00 smithi055 ceph-mon[97033]: pgmap v13842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:00 smithi059 ceph-mon[97450]: pgmap v13842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:00.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:00.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:00.403160+0000) 2022-09-08T04:46:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:00.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:00.403260+0000) 2022-09-08T04:46:01.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:46:01.336+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:46:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:01.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:01.403272+0000) 2022-09-08T04:46:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:01.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:01.403404+0000) 2022-09-08T04:46:02.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:02 smithi055 ceph-mon[97033]: pgmap v13843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:46:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:02 smithi059 ceph-mon[97450]: pgmap v13843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:46:02.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:02.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:02.403458+0000) 2022-09-08T04:46:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:02.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:02.403564+0000) 2022-09-08T04:46:03.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:03.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:03.403729+0000) 2022-09-08T04:46:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:03.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:03.403761+0000) 2022-09-08T04:46:04.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:04 smithi055 ceph-mon[97033]: pgmap v13844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:04 smithi059 ceph-mon[97450]: pgmap v13844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:04.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:04.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:04.403977+0000) 2022-09-08T04:46:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:04.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:04.403978+0000) 2022-09-08T04:46:05.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:05.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:05.404153+0000) 2022-09-08T04:46:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:05.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:05.404152+0000) 2022-09-08T04:46:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:06 smithi055 ceph-mon[97033]: pgmap v13845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:46:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:06 smithi059 ceph-mon[97450]: pgmap v13845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:46:06.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:06.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:06.404343+0000) 2022-09-08T04:46:06.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:06.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:06.404313+0000) 2022-09-08T04:46:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:46:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:46:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:07.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:07.404573+0000) 2022-09-08T04:46:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:07.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:07.404559+0000) 2022-09-08T04:46:08.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:08 smithi055 ceph-mon[97033]: pgmap v13846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:46:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:08 smithi059 ceph-mon[97450]: pgmap v13846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:46:08.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:08.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:08.404783+0000) 2022-09-08T04:46:08.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:08.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:08.404779+0000) 2022-09-08T04:46:09.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:09.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:09.404970+0000) 2022-09-08T04:46:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:09.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:09.404962+0000) 2022-09-08T04:46:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:10 smithi055 ceph-mon[97033]: pgmap v13847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:46:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:10 smithi055 ceph-mon[97033]: osdmap e946: 12 total, 11 up, 11 in 2022-09-08T04:46:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:10 smithi059 ceph-mon[97450]: pgmap v13847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:46:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:10 smithi059 ceph-mon[97450]: osdmap e946: 12 total, 11 up, 11 in 2022-09-08T04:46:10.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:10.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:10.405206+0000) 2022-09-08T04:46:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:10.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:10.405196+0000) 2022-09-08T04:46:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:46:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:46:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:46:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:46:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:46:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:46:11.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:11.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:11.405486+0000) 2022-09-08T04:46:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:11.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:11.405462+0000) 2022-09-08T04:46:12.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:12 smithi055 ceph-mon[97033]: pgmap v13849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:46:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:46:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:12 smithi059 ceph-mon[97450]: pgmap v13849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:46:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:46:12.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:12.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:12.405788+0000) 2022-09-08T04:46:12.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:12.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:12.405723+0000) 2022-09-08T04:46:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:13.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:13.405973+0000) 2022-09-08T04:46:13.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:13.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:13.406001+0000) 2022-09-08T04:46:14.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:14 smithi055 ceph-mon[97033]: pgmap v13850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:46:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:14 smithi059 ceph-mon[97450]: pgmap v13850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:46:14.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:14.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:14.406253+0000) 2022-09-08T04:46:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:14.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:14.406223+0000) 2022-09-08T04:46:15.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:15.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:15.406505+0000) 2022-09-08T04:46:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:15.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:15.406444+0000) 2022-09-08T04:46:16.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:16 smithi055 ceph-mon[97033]: pgmap v13851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:46:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:16 smithi059 ceph-mon[97450]: pgmap v13851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:46:16.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:46:16.344+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:46:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:16.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:16.406644+0000) 2022-09-08T04:46:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:16.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:16.406721+0000) 2022-09-08T04:46:17.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:17.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:17.406892+0000) 2022-09-08T04:46:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:17.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:17.406973+0000) 2022-09-08T04:46:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:18 smithi055 ceph-mon[97033]: pgmap v13852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:46:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:18 smithi059 ceph-mon[97450]: pgmap v13852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:46:18.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:18.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:18.407184+0000) 2022-09-08T04:46:18.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:18.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:18.407186+0000) 2022-09-08T04:46:19.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:19.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:19.407439+0000) 2022-09-08T04:46:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:19.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:19.407435+0000) 2022-09-08T04:46:20.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:20 smithi055 ceph-mon[97033]: pgmap v13853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:46:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:20 smithi055 ceph-mon[97033]: osdmap e947: 12 total, 11 up, 11 in 2022-09-08T04:46:20.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:20 smithi059 ceph-mon[97450]: pgmap v13853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:46:20.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:20 smithi059 ceph-mon[97450]: osdmap e947: 12 total, 11 up, 11 in 2022-09-08T04:46:20.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:20.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:20.407620+0000) 2022-09-08T04:46:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:20.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:20.407616+0000) 2022-09-08T04:46:21.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:21.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:21.407782+0000) 2022-09-08T04:46:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:21.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:21.407726+0000) 2022-09-08T04:46:22.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:22 smithi055 ceph-mon[97033]: pgmap v13855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:46:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:22 smithi059 ceph-mon[97450]: pgmap v13855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:46:22.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:22.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:22.407958+0000) 2022-09-08T04:46:22.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:22.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:22.407936+0000) 2022-09-08T04:46:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:23.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:23.408127+0000) 2022-09-08T04:46:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:23.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:23.408090+0000) 2022-09-08T04:46:24.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:24 smithi055 ceph-mon[97033]: pgmap v13856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:46:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:24 smithi059 ceph-mon[97450]: pgmap v13856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:46:24.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:24.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:24.408369+0000) 2022-09-08T04:46:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:24.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:24.408307+0000) 2022-09-08T04:46:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:25.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:25.408667+0000) 2022-09-08T04:46:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:25.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:25.408543+0000) 2022-09-08T04:46:26.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:26 smithi055 ceph-mon[97033]: pgmap v13857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:46:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:26 smithi059 ceph-mon[97450]: pgmap v13857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:46:26.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:26.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:26.408859+0000) 2022-09-08T04:46:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:26.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:26.408744+0000) 2022-09-08T04:46:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:27.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:27.409043+0000) 2022-09-08T04:46:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:27.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:27.408921+0000) 2022-09-08T04:46:28.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:28 smithi055 ceph-mon[97033]: pgmap v13858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:46:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:28 smithi059 ceph-mon[97450]: pgmap v13858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:46:28.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:28.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:28.409272+0000) 2022-09-08T04:46:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:28.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:28.409151+0000) 2022-09-08T04:46:29.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:29.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:29.409510+0000) 2022-09-08T04:46:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:29.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:29.409414+0000) 2022-09-08T04:46:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:30 smithi055 ceph-mon[97033]: pgmap v13859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:46:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:30 smithi059 ceph-mon[97450]: pgmap v13859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:46:30.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:30.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:30.409743+0000) 2022-09-08T04:46:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:30.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:30.409673+0000) 2022-09-08T04:46:31.408 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:46:31.349+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:46:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:31.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:31.409906+0000) 2022-09-08T04:46:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:31.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:31.409884+0000) 2022-09-08T04:46:32.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:32 smithi055 ceph-mon[97033]: pgmap v13860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:46:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:32 smithi059 ceph-mon[97450]: pgmap v13860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:46:32.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:32.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:32.410153+0000) 2022-09-08T04:46:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:32.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:32.410135+0000) 2022-09-08T04:46:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:33.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:33.410415+0000) 2022-09-08T04:46:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:33.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:33.410395+0000) 2022-09-08T04:46:34.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:34 smithi055 ceph-mon[97033]: pgmap v13861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:34.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:34 smithi059 ceph-mon[97450]: pgmap v13861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:34.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:34.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:34.410652+0000) 2022-09-08T04:46:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:34.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:34.410630+0000) 2022-09-08T04:46:35.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:35.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:35.410919+0000) 2022-09-08T04:46:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:35.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:35.410886+0000) 2022-09-08T04:46:36.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:36 smithi055 ceph-mon[97033]: pgmap v13862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:36 smithi059 ceph-mon[97450]: pgmap v13862: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:36.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:36.411110+0000) 2022-09-08T04:46:36.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:36.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:36.411184+0000) 2022-09-08T04:46:37.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:37.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:37.411347+0000) 2022-09-08T04:46:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:37.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:37.411426+0000) 2022-09-08T04:46:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:38 smithi055 ceph-mon[97033]: pgmap v13863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:46:38.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:38 smithi059 ceph-mon[97450]: pgmap v13863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:46:38.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:38.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:38.411628+0000) 2022-09-08T04:46:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:38.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:38.411609+0000) 2022-09-08T04:46:39.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:39.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:39.411889+0000) 2022-09-08T04:46:39.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:39.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:39.411886+0000) 2022-09-08T04:46:40.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:40 smithi055 ceph-mon[97033]: pgmap v13864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:40 smithi059 ceph-mon[97450]: pgmap v13864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:40.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:40.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:40.412143+0000) 2022-09-08T04:46:40.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:40.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:40.412115+0000) 2022-09-08T04:46:41.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:41.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:41.412264+0000) 2022-09-08T04:46:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:41.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:41.412376+0000) 2022-09-08T04:46:42.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:42 smithi055 ceph-mon[97033]: pgmap v13865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:46:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:42 smithi059 ceph-mon[97450]: pgmap v13865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:46:42.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:42.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:42.412470+0000) 2022-09-08T04:46:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:42.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:42.412597+0000) 2022-09-08T04:46:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:43.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:43.412667+0000) 2022-09-08T04:46:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:43.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:43.412779+0000) 2022-09-08T04:46:44.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:44 smithi055 ceph-mon[97033]: pgmap v13866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:44 smithi059 ceph-mon[97450]: pgmap v13866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:44.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:44.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:44.412890+0000) 2022-09-08T04:46:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:44.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:44.412993+0000) 2022-09-08T04:46:45.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:45.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:45.413135+0000) 2022-09-08T04:46:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:45.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:45.413230+0000) 2022-09-08T04:46:46.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:46 smithi055 ceph-mon[97033]: pgmap v13867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:46 smithi059 ceph-mon[97450]: pgmap v13867: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:46.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:46.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:46.413319+0000) 2022-09-08T04:46:46.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:46:46.362+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:46:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:46.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:46.413432+0000) 2022-09-08T04:46:47.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:47.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:47.413576+0000) 2022-09-08T04:46:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:47.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:47.413704+0000) 2022-09-08T04:46:48.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:48 smithi055 ceph-mon[97033]: pgmap v13868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:46:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:48 smithi059 ceph-mon[97450]: pgmap v13868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:46:48.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:48.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:48.413882+0000) 2022-09-08T04:46:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:48.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:48.413824+0000) 2022-09-08T04:46:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:49.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:49.414085+0000) 2022-09-08T04:46:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:49.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:49.414134+0000) 2022-09-08T04:46:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:50 smithi055 ceph-mon[97033]: pgmap v13869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:50 smithi059 ceph-mon[97450]: pgmap v13869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:50.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:50.414288+0000) 2022-09-08T04:46:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:50.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:50.414387+0000) 2022-09-08T04:46:51.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:51.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:51.414478+0000) 2022-09-08T04:46:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:51.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:51.414544+0000) 2022-09-08T04:46:52.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:52 smithi055 ceph-mon[97033]: pgmap v13870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:46:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:52 smithi059 ceph-mon[97450]: pgmap v13870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:46:52.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:52.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:52.414680+0000) 2022-09-08T04:46:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:52.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:52.414706+0000) 2022-09-08T04:46:53.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:53.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:53.414907+0000) 2022-09-08T04:46:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:53.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:53.414914+0000) 2022-09-08T04:46:54.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:54 smithi055 ceph-mon[97033]: pgmap v13871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:54 smithi059 ceph-mon[97450]: pgmap v13871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:54.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:54.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:54.415156+0000) 2022-09-08T04:46:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:54.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:54.415148+0000) 2022-09-08T04:46:55.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:55.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:55.415408+0000) 2022-09-08T04:46:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:55.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:55.415372+0000) 2022-09-08T04:46:56.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:56 smithi055 ceph-mon[97033]: pgmap v13872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:56 smithi059 ceph-mon[97450]: pgmap v13872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:46:56.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:56.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:56.415619+0000) 2022-09-08T04:46:56.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:56.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:56.415724+0000) 2022-09-08T04:46:57.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:57.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:57.415852+0000) 2022-09-08T04:46:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:57.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:57.415912+0000) 2022-09-08T04:46:58.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:46:58 smithi055 ceph-mon[97033]: pgmap v13873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:46:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:46:58 smithi059 ceph-mon[97450]: pgmap v13873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:46:58.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:58.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:58.416164+0000) 2022-09-08T04:46:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:58.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:58.416119+0000) 2022-09-08T04:46:59.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:46:59.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:59.416343+0000) 2022-09-08T04:46:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:46:59.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:46:59.416270+0000) 2022-09-08T04:47:00.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:00 smithi055 ceph-mon[97033]: pgmap v13874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:00 smithi059 ceph-mon[97450]: pgmap v13874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:00.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:00.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:00.416553+0000) 2022-09-08T04:47:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:00.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:00.416499+0000) 2022-09-08T04:47:01.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:47:01.367+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:47:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:01.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:01.416728+0000) 2022-09-08T04:47:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:01.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:01.416717+0000) 2022-09-08T04:47:02.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:02 smithi055 ceph-mon[97033]: pgmap v13875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:47:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:02 smithi059 ceph-mon[97450]: pgmap v13875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:47:02.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:02.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:02.416965+0000) 2022-09-08T04:47:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:02.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:02.416953+0000) 2022-09-08T04:47:03.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:03.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:03.417234+0000) 2022-09-08T04:47:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:03.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:03.417184+0000) 2022-09-08T04:47:04.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:04 smithi055 ceph-mon[97033]: pgmap v13876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:04.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:04 smithi059 ceph-mon[97450]: pgmap v13876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:04.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:04.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:04.417497+0000) 2022-09-08T04:47:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:04.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:04.417448+0000) 2022-09-08T04:47:05.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:05.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:05.417682+0000) 2022-09-08T04:47:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:05.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:05.417619+0000) 2022-09-08T04:47:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:06 smithi055 ceph-mon[97033]: pgmap v13877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:47:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:06 smithi059 ceph-mon[97450]: pgmap v13877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:47:06.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:06.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:06.417793+0000) 2022-09-08T04:47:06.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:06.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:06.417857+0000) 2022-09-08T04:47:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:47:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:47:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:07.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:07.418099+0000) 2022-09-08T04:47:07.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:07.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:07.418041+0000) 2022-09-08T04:47:08.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:08 smithi055 ceph-mon[97033]: pgmap v13878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:47:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:08 smithi059 ceph-mon[97450]: pgmap v13878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:47:08.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:08.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:08.418292+0000) 2022-09-08T04:47:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:08.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:08.418265+0000) 2022-09-08T04:47:09.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:09.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:09.418415+0000) 2022-09-08T04:47:09.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:09.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:09.418452+0000) 2022-09-08T04:47:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:10 smithi055 ceph-mon[97033]: pgmap v13879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:47:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:10 smithi055 ceph-mon[97033]: osdmap e948: 12 total, 11 up, 11 in 2022-09-08T04:47:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:10 smithi059 ceph-mon[97450]: pgmap v13879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:47:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:10 smithi059 ceph-mon[97450]: osdmap e948: 12 total, 11 up, 11 in 2022-09-08T04:47:10.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:10.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:10.418721+0000) 2022-09-08T04:47:10.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:10.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:10.418694+0000) 2022-09-08T04:47:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:47:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:47:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:11 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:47:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:47:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:47:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:11 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:47:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:11.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:11.418902+0000) 2022-09-08T04:47:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:11.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:11.418940+0000) 2022-09-08T04:47:12.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:12 smithi055 ceph-mon[97033]: pgmap v13881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:47:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:47:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:47:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:12 smithi059 ceph-mon[97450]: pgmap v13881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:47:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:47:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:47:12.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:12.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:12.419021+0000) 2022-09-08T04:47:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:12.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:12.419084+0000) 2022-09-08T04:47:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:13 smithi055 ceph-mon[97033]: pgmap v13882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:47:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:13 smithi059 ceph-mon[97450]: pgmap v13882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:47:13.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:13.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:13.419250+0000) 2022-09-08T04:47:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:13.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:13.419308+0000) 2022-09-08T04:47:14.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:14.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:14.419433+0000) 2022-09-08T04:47:14.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:14.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:14.419478+0000) 2022-09-08T04:47:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:47:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:47:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:47:15.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:47:15.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:47:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:47:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:47:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:47:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:47:15.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:15.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:15.419578+0000) 2022-09-08T04:47:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:15.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:15.419630+0000) 2022-09-08T04:47:16.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:16 smithi059 ceph-mon[97450]: pgmap v13883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:47:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:16 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T04:47:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:16 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T04:47:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:16 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T04:47:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:16 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T04:47:16.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:16 smithi059 ceph-mon[97450]: osdmap e949: 12 total, 11 up, 11 in 2022-09-08T04:47:16.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:16 smithi055 ceph-mon[97033]: pgmap v13883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:47:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:16 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T04:47:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:16 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T04:47:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:16 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T04:47:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:16 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T04:47:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:16 smithi055 ceph-mon[97033]: osdmap e949: 12 total, 11 up, 11 in 2022-09-08T04:47:16.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:16.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:16.419753+0000) 2022-09-08T04:47:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:16.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:16.419756+0000) 2022-09-08T04:47:16.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:47:16.371+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:47:17.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:17.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:17.419975+0000) 2022-09-08T04:47:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:17.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:17.419973+0000) 2022-09-08T04:47:18.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:18 smithi059 ceph-mon[97450]: pgmap v13885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:47:18.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:18 smithi055 ceph-mon[97033]: pgmap v13885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:47:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:18.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:18.420146+0000) 2022-09-08T04:47:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:18.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:18.420138+0000) 2022-09-08T04:47:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:19.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:19.420356+0000) 2022-09-08T04:47:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:19.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:19.420343+0000) 2022-09-08T04:47:20.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:20 smithi059 ceph-mon[97450]: pgmap v13886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 225 B/s wr, 7 op/s 2022-09-08T04:47:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:20 smithi055 ceph-mon[97033]: pgmap v13886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 225 B/s wr, 7 op/s 2022-09-08T04:47:20.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:20.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:20.420545+0000) 2022-09-08T04:47:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:20.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:20.420534+0000) 2022-09-08T04:47:21.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:21.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:21.420700+0000) 2022-09-08T04:47:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:21.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:21.420704+0000) 2022-09-08T04:47:22.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:22 smithi059 ceph-mon[97450]: pgmap v13887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:47:22.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:22 smithi055 ceph-mon[97033]: pgmap v13887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:47:22.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:22.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:22.420844+0000) 2022-09-08T04:47:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:22.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:22.420861+0000) 2022-09-08T04:47:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:23.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:23.421037+0000) 2022-09-08T04:47:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:23.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:23.421017+0000) 2022-09-08T04:47:24.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:24 smithi059 ceph-mon[97450]: pgmap v13888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:47:24.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:24 smithi055 ceph-mon[97033]: pgmap v13888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:47:24.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:24.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:24.421184+0000) 2022-09-08T04:47:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:24.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:24.421140+0000) 2022-09-08T04:47:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:25.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:25.421426+0000) 2022-09-08T04:47:25.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:25.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:25.421375+0000) 2022-09-08T04:47:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:26 smithi059 ceph-mon[97450]: pgmap v13889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:47:26.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:26 smithi055 ceph-mon[97033]: pgmap v13889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:47:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:26.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:26.421584+0000) 2022-09-08T04:47:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:26.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:26.421578+0000) 2022-09-08T04:47:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:27.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:27.421792+0000) 2022-09-08T04:47:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:27.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:27.421747+0000) 2022-09-08T04:47:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:28 smithi059 ceph-mon[97450]: pgmap v13890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:47:28.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:28 smithi055 ceph-mon[97033]: pgmap v13890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:47:28.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:28.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:28.422055+0000) 2022-09-08T04:47:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:28.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:28.421990+0000) 2022-09-08T04:47:29.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:29.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:29.422252+0000) 2022-09-08T04:47:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:29.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:29.422188+0000) 2022-09-08T04:47:30.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:30 smithi055 ceph-mon[97033]: pgmap v13891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:30 smithi059 ceph-mon[97450]: pgmap v13891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:30.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:30.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:30.422489+0000) 2022-09-08T04:47:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:30.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:30.422420+0000) 2022-09-08T04:47:31.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:47:31.379+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:47:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:31.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:31.422756+0000) 2022-09-08T04:47:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:31.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:31.422683+0000) 2022-09-08T04:47:32.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:32 smithi055 ceph-mon[97033]: pgmap v13892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:47:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:32 smithi059 ceph-mon[97450]: pgmap v13892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:47:32.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:32.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:32.422979+0000) 2022-09-08T04:47:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:32.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:32.422976+0000) 2022-09-08T04:47:33.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:33.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:33.423197+0000) 2022-09-08T04:47:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:33.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:33.423212+0000) 2022-09-08T04:47:34.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:34 smithi055 ceph-mon[97033]: pgmap v13893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:34 smithi059 ceph-mon[97450]: pgmap v13893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:34.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:34.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:34.423384+0000) 2022-09-08T04:47:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:34.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:34.423393+0000) 2022-09-08T04:47:35.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:35.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:35.423672+0000) 2022-09-08T04:47:35.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:35.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:35.423618+0000) 2022-09-08T04:47:36.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:36 smithi055 ceph-mon[97033]: pgmap v13894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:36 smithi059 ceph-mon[97450]: pgmap v13894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:36.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:36.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:36.423856+0000) 2022-09-08T04:47:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:36.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:36.423948+0000) 2022-09-08T04:47:37.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:37.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:37.423999+0000) 2022-09-08T04:47:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:37.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:37.424095+0000) 2022-09-08T04:47:38.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:38 smithi055 ceph-mon[97033]: pgmap v13895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:47:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:38 smithi059 ceph-mon[97450]: pgmap v13895: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:47:38.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:38.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:38.424224+0000) 2022-09-08T04:47:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:38.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:38.424329+0000) 2022-09-08T04:47:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:39.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:39.424411+0000) 2022-09-08T04:47:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:39.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:39.424504+0000) 2022-09-08T04:47:40.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:40 smithi055 ceph-mon[97033]: pgmap v13896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:40 smithi059 ceph-mon[97450]: pgmap v13896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:40.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:40.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:40.424752+0000) 2022-09-08T04:47:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:40.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:40.424679+0000) 2022-09-08T04:47:41.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:41.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:41.424875+0000) 2022-09-08T04:47:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:41.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:41.424925+0000) 2022-09-08T04:47:42.487 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:42 smithi055 ceph-mon[97033]: pgmap v13897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:47:42.488 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:47:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:42 smithi059 ceph-mon[97450]: pgmap v13897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:47:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:47:42.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:42.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:42.425014+0000) 2022-09-08T04:47:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:42.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:42.425092+0000) 2022-09-08T04:47:43.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:43 smithi055 ceph-mon[97033]: pgmap v13898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:47:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:43 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T04:47:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:43 smithi059 ceph-mon[97450]: pgmap v13898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:47:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:43 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T04:47:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:43.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:43.425258+0000) 2022-09-08T04:47:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:43.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:43.425315+0000) 2022-09-08T04:47:44.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:44.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:44.425496+0000) 2022-09-08T04:47:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:44.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:44.425571+0000) 2022-09-08T04:47:45.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:45 smithi055 ceph-mon[97033]: pgmap v13899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:47:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:45 smithi059 ceph-mon[97450]: pgmap v13899: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:47:45.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:45.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:45.425771+0000) 2022-09-08T04:47:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:45.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:45.425828+0000) 2022-09-08T04:47:46.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:47:46.395+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:47:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:46.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:46.425934+0000) 2022-09-08T04:47:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:46.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:46.425997+0000) 2022-09-08T04:47:47.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:47 smithi055 ceph-mon[97033]: pgmap v13900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T04:47:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:47 smithi059 ceph-mon[97450]: pgmap v13900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T04:47:47.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:47.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:47.426170+0000) 2022-09-08T04:47:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:47.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:47.426245+0000) 2022-09-08T04:47:48.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:48.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:48.426380+0000) 2022-09-08T04:47:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:48.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:48.426440+0000) 2022-09-08T04:47:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:49 smithi055 ceph-mon[97033]: pgmap v13901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:47:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:49 smithi059 ceph-mon[97450]: pgmap v13901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:47:49.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:49.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:49.426668+0000) 2022-09-08T04:47:49.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:49.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:49.426716+0000) 2022-09-08T04:47:50.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:50.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:50.426855+0000) 2022-09-08T04:47:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:50.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:50.426912+0000) 2022-09-08T04:47:51.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:51 smithi055 ceph-mon[97033]: pgmap v13902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:47:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:51 smithi059 ceph-mon[97450]: pgmap v13902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:47:51.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:51.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:51.427107+0000) 2022-09-08T04:47:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:51.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:51.427141+0000) 2022-09-08T04:47:52.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:52.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:52.427283+0000) 2022-09-08T04:47:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:52.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:52.427364+0000) 2022-09-08T04:47:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:53 smithi055 ceph-mon[97033]: pgmap v13903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:47:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:53 smithi059 ceph-mon[97450]: pgmap v13903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:47:53.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:53.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:53.427479+0000) 2022-09-08T04:47:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:53.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:53.427533+0000) 2022-09-08T04:47:54.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:54.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:54.427623+0000) 2022-09-08T04:47:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:54.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:54.427682+0000) 2022-09-08T04:47:55.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:55 smithi055 ceph-mon[97033]: pgmap v13904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:55 smithi059 ceph-mon[97450]: pgmap v13904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:55.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:55.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:55.427759+0000) 2022-09-08T04:47:55.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:55.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:55.427830+0000) 2022-09-08T04:47:56.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:56.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:56.428004+0000) 2022-09-08T04:47:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:56.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:56.428041+0000) 2022-09-08T04:47:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:57 smithi055 ceph-mon[97033]: pgmap v13905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:47:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:57 smithi059 ceph-mon[97450]: pgmap v13905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:47:57.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:57.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:57.428169+0000) 2022-09-08T04:47:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:57.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:57.428250+0000) 2022-09-08T04:47:58.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:58.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:58.428288+0000) 2022-09-08T04:47:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:58.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:58.428408+0000) 2022-09-08T04:47:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:47:59 smithi055 ceph-mon[97033]: pgmap v13906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:47:59 smithi059 ceph-mon[97450]: pgmap v13906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:47:59.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:47:59.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:59.428563+0000) 2022-09-08T04:47:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:47:59.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:47:59.428491+0000) 2022-09-08T04:48:00.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:00.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:00.428644+0000) 2022-09-08T04:48:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:00.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:00.428697+0000) 2022-09-08T04:48:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:01 smithi055 ceph-mon[97033]: pgmap v13907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:01 smithi059 ceph-mon[97450]: pgmap v13907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:01.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:48:01.398+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:48:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:01.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:01.428853+0000) 2022-09-08T04:48:01.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:01.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:01.428927+0000) 2022-09-08T04:48:02.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:02.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:02.429100+0000) 2022-09-08T04:48:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:02.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:02.429145+0000) 2022-09-08T04:48:03.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:03 smithi055 ceph-mon[97033]: pgmap v13908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:48:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:03 smithi059 ceph-mon[97450]: pgmap v13908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:48:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:03.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:03.429209+0000) 2022-09-08T04:48:03.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:03.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:03.429321+0000) 2022-09-08T04:48:04.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:04.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:04.429417+0000) 2022-09-08T04:48:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:04.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:04.429480+0000) 2022-09-08T04:48:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:05 smithi055 ceph-mon[97033]: pgmap v13909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:05.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:05 smithi059 ceph-mon[97450]: pgmap v13909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:05.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:05.429668+0000) 2022-09-08T04:48:05.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:05.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:05.429687+0000) 2022-09-08T04:48:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:48:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:48:06.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:06.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:06.429821+0000) 2022-09-08T04:48:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:06.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:06.429828+0000) 2022-09-08T04:48:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:07 smithi055 ceph-mon[97033]: pgmap v13910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:48:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:48:07.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:07 smithi059 ceph-mon[97450]: pgmap v13910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:48:07.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:48:07.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:07.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:07.430074+0000) 2022-09-08T04:48:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:07.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:07.430066+0000) 2022-09-08T04:48:08.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:08.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:08.430247+0000) 2022-09-08T04:48:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:08.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:08.430243+0000) 2022-09-08T04:48:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:09 smithi055 ceph-mon[97033]: pgmap v13911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:48:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:09 smithi059 ceph-mon[97450]: pgmap v13911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:48:09.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:09.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:09.430454+0000) 2022-09-08T04:48:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:09.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:09.430461+0000) 2022-09-08T04:48:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:10 smithi055 ceph-mon[97033]: osdmap e950: 12 total, 11 up, 11 in 2022-09-08T04:48:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:10 smithi059 ceph-mon[97450]: osdmap e950: 12 total, 11 up, 11 in 2022-09-08T04:48:10.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:10.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:10.430699+0000) 2022-09-08T04:48:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:10.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:10.430740+0000) 2022-09-08T04:48:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:11 smithi055 ceph-mon[97033]: pgmap v13913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:48:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:11 smithi059 ceph-mon[97450]: pgmap v13913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:48:11.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:11.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:11.430885+0000) 2022-09-08T04:48:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:11.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:11.430889+0000) 2022-09-08T04:48:12.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:12.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:12.431082+0000) 2022-09-08T04:48:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:12.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:12.431092+0000) 2022-09-08T04:48:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:13 smithi055 ceph-mon[97033]: pgmap v13914: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:48:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:13 smithi059 ceph-mon[97450]: pgmap v13914: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:48:13.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:13.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:13.431296+0000) 2022-09-08T04:48:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:13.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:13.431282+0000) 2022-09-08T04:48:14.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:14.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:14.431459+0000) 2022-09-08T04:48:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:14.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:14.431427+0000) 2022-09-08T04:48:15.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:15 smithi055 ceph-mon[97033]: pgmap v13915: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:48:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:15 smithi055 ceph-mon[97033]: osdmap e951: 12 total, 11 up, 11 in 2022-09-08T04:48:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:15 smithi059 ceph-mon[97450]: pgmap v13915: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:48:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:15 smithi059 ceph-mon[97450]: osdmap e951: 12 total, 11 up, 11 in 2022-09-08T04:48:15.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:15.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:15.431660+0000) 2022-09-08T04:48:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:15.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:15.431625+0000) 2022-09-08T04:48:16.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:16.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:16.431817+0000) 2022-09-08T04:48:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:16.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:16.431809+0000) 2022-09-08T04:48:16.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:48:16.405+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:48:17.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:17 smithi055 ceph-mon[97033]: pgmap v13917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:48:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:17 smithi059 ceph-mon[97450]: pgmap v13917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:48:17.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:17.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:17.432023+0000) 2022-09-08T04:48:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:17.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:17.432022+0000) 2022-09-08T04:48:18.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:18.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:18.432181+0000) 2022-09-08T04:48:18.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:18.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:18.432194+0000) 2022-09-08T04:48:19.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:19 smithi055 ceph-mon[97033]: pgmap v13918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T04:48:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:19 smithi059 ceph-mon[97450]: pgmap v13918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T04:48:19.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:19.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:19.432387+0000) 2022-09-08T04:48:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:19.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:19.432402+0000) 2022-09-08T04:48:20.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:20.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:20.432625+0000) 2022-09-08T04:48:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:20.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:20.432628+0000) 2022-09-08T04:48:21.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:21 smithi055 ceph-mon[97033]: pgmap v13919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:48:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:21 smithi059 ceph-mon[97450]: pgmap v13919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:48:21.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:21.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:21.432802+0000) 2022-09-08T04:48:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:21.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:21.432792+0000) 2022-09-08T04:48:22.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:22.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:22.432940+0000) 2022-09-08T04:48:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:22.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:22.432927+0000) 2022-09-08T04:48:23.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:23 smithi055 ceph-mon[97033]: pgmap v13920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:48:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:23 smithi059 ceph-mon[97450]: pgmap v13920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:48:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:23.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:23.433099+0000) 2022-09-08T04:48:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:23.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:23.433069+0000) 2022-09-08T04:48:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:24.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:24.433266+0000) 2022-09-08T04:48:24.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:24.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:24.433269+0000) 2022-09-08T04:48:25.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:25 smithi055 ceph-mon[97033]: pgmap v13921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:48:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:25 smithi055 ceph-mon[97033]: osdmap e952: 12 total, 11 up, 11 in 2022-09-08T04:48:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:25 smithi059 ceph-mon[97450]: pgmap v13921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:48:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:25 smithi059 ceph-mon[97450]: osdmap e952: 12 total, 11 up, 11 in 2022-09-08T04:48:25.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:25.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:25.433476+0000) 2022-09-08T04:48:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:25.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:25.433516+0000) 2022-09-08T04:48:26.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:26.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:26.433761+0000) 2022-09-08T04:48:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:26.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:26.433779+0000) 2022-09-08T04:48:27.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:27 smithi055 ceph-mon[97033]: pgmap v13923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:48:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:27 smithi059 ceph-mon[97450]: pgmap v13923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:48:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:27.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:27.433919+0000) 2022-09-08T04:48:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:27.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:27.433929+0000) 2022-09-08T04:48:28.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:28.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:28.434124+0000) 2022-09-08T04:48:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:28.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:28.434108+0000) 2022-09-08T04:48:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:29 smithi055 ceph-mon[97033]: pgmap v13924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:48:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:29 smithi059 ceph-mon[97450]: pgmap v13924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:48:29.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:29.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:29.434346+0000) 2022-09-08T04:48:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:29.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:29.434379+0000) 2022-09-08T04:48:30.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:30.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:30.434511+0000) 2022-09-08T04:48:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:30.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:30.434594+0000) 2022-09-08T04:48:31.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:31 smithi055 ceph-mon[97033]: pgmap v13925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:48:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:31 smithi059 ceph-mon[97450]: pgmap v13925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:48:31.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:48:31.411+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:48:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:31.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:31.434784+0000) 2022-09-08T04:48:31.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:31.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:31.434713+0000) 2022-09-08T04:48:32.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:32.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:32.434874+0000) 2022-09-08T04:48:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:32.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:32.434941+0000) 2022-09-08T04:48:33.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:33 smithi055 ceph-mon[97033]: pgmap v13926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:48:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:33 smithi059 ceph-mon[97450]: pgmap v13926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:48:33.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:33.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:33.435032+0000) 2022-09-08T04:48:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:33.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:33.435102+0000) 2022-09-08T04:48:34.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:34.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:34.435184+0000) 2022-09-08T04:48:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:34.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:34.435261+0000) 2022-09-08T04:48:35.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:35 smithi055 ceph-mon[97033]: pgmap v13927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:48:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:35 smithi059 ceph-mon[97450]: pgmap v13927: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:48:35.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:35.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:35.435367+0000) 2022-09-08T04:48:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:35.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:35.435416+0000) 2022-09-08T04:48:36.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:36.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:36.435563+0000) 2022-09-08T04:48:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:36.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:36.435652+0000) 2022-09-08T04:48:37.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:37 smithi055 ceph-mon[97033]: pgmap v13928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:48:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:37 smithi059 ceph-mon[97450]: pgmap v13928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:48:37.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:37.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:37.435726+0000) 2022-09-08T04:48:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:37.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:37.435835+0000) 2022-09-08T04:48:38.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:38.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:38.435889+0000) 2022-09-08T04:48:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:38.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:38.435990+0000) 2022-09-08T04:48:39.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:39 smithi055 ceph-mon[97033]: pgmap v13929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:39 smithi059 ceph-mon[97450]: pgmap v13929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:39.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:39.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:39.436011+0000) 2022-09-08T04:48:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:39.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:39.436132+0000) 2022-09-08T04:48:40.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:40.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:40.436204+0000) 2022-09-08T04:48:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:40.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:40.436299+0000) 2022-09-08T04:48:41.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:41 smithi055 ceph-mon[97033]: pgmap v13930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:41.524 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:41 smithi059 ceph-mon[97450]: pgmap v13930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:41.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:41.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:41.436359+0000) 2022-09-08T04:48:41.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:41.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:41.436470+0000) 2022-09-08T04:48:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:48:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:48:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:48:42.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:48:42.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:48:42.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:48:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:48:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:48:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:48:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:48:42.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:42.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:42.436551+0000) 2022-09-08T04:48:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:42.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:42.436722+0000) 2022-09-08T04:48:43.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:43 smithi055 ceph-mon[97033]: pgmap v13931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:48:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:43 smithi059 ceph-mon[97450]: pgmap v13931: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:48:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:43.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:43.436751+0000) 2022-09-08T04:48:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:43.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:43.436887+0000) 2022-09-08T04:48:44.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:44.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:44.436954+0000) 2022-09-08T04:48:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:44.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:44.437094+0000) 2022-09-08T04:48:45.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:45 smithi055 ceph-mon[97033]: pgmap v13932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:45 smithi059 ceph-mon[97450]: pgmap v13932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:45.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:45.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:45.437177+0000) 2022-09-08T04:48:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:45.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:45.437344+0000) 2022-09-08T04:48:46.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:46.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:46.437401+0000) 2022-09-08T04:48:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:46.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:46.437559+0000) 2022-09-08T04:48:46.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:48:46.426+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:48:47.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:47 smithi055 ceph-mon[97033]: pgmap v13933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:48:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:47 smithi059 ceph-mon[97450]: pgmap v13933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:48:47.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:47.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:47.437738+0000) 2022-09-08T04:48:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:47.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:47.437632+0000) 2022-09-08T04:48:48.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:48.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:48.437891+0000) 2022-09-08T04:48:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:48.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:48.437796+0000) 2022-09-08T04:48:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:49 smithi055 ceph-mon[97033]: pgmap v13934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:49 smithi059 ceph-mon[97450]: pgmap v13934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:49.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:49.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:49.438094+0000) 2022-09-08T04:48:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:49.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:49.438018+0000) 2022-09-08T04:48:50.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:50.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:50.438237+0000) 2022-09-08T04:48:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:50.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:50.438194+0000) 2022-09-08T04:48:51.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:51 smithi055 ceph-mon[97033]: pgmap v13935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:51 smithi059 ceph-mon[97450]: pgmap v13935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:51.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:51.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:51.438369+0000) 2022-09-08T04:48:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:51.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:51.438459+0000) 2022-09-08T04:48:52.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:52.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:52.438742+0000) 2022-09-08T04:48:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:52.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:52.438562+0000) 2022-09-08T04:48:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:53 smithi055 ceph-mon[97033]: pgmap v13936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:48:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:53 smithi059 ceph-mon[97450]: pgmap v13936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:48:53.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:53.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:53.438761+0000) 2022-09-08T04:48:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:53.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:53.438910+0000) 2022-09-08T04:48:54.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:54.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:54.438899+0000) 2022-09-08T04:48:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:54.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:54.439043+0000) 2022-09-08T04:48:55.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:55 smithi055 ceph-mon[97033]: pgmap v13937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:55 smithi059 ceph-mon[97450]: pgmap v13937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:55.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:55.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:55.439130+0000) 2022-09-08T04:48:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:55.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:55.439243+0000) 2022-09-08T04:48:56.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:56.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:56.439293+0000) 2022-09-08T04:48:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:56.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:56.439413+0000) 2022-09-08T04:48:57.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:57 smithi055 ceph-mon[97033]: pgmap v13938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:48:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:57 smithi059 ceph-mon[97450]: pgmap v13938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:48:57.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:57.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:57.439424+0000) 2022-09-08T04:48:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:57.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:57.439562+0000) 2022-09-08T04:48:58.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:58.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:58.439584+0000) 2022-09-08T04:48:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:58.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:58.439708+0000) 2022-09-08T04:48:59.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:48:59 smithi055 ceph-mon[97033]: pgmap v13939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:48:59 smithi059 ceph-mon[97450]: pgmap v13939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:48:59.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:48:59.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:59.439764+0000) 2022-09-08T04:48:59.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:48:59.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:48:59.439914+0000) 2022-09-08T04:49:00.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:00.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:00.440083+0000) 2022-09-08T04:49:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:00.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:00.439931+0000) 2022-09-08T04:49:01.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:01 smithi055 ceph-mon[97033]: pgmap v13940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:49:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:01 smithi059 ceph-mon[97450]: pgmap v13940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:49:01.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:49:01.431+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:49:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:01.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:01.440122+0000) 2022-09-08T04:49:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:01.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:01.440333+0000) 2022-09-08T04:49:02.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:02 smithi059 ceph-mon[97450]: pgmap v13941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:49:02.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:02 smithi055 ceph-mon[97033]: pgmap v13941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:49:02.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:02.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:02.440335+0000) 2022-09-08T04:49:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:02.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:02.440480+0000) 2022-09-08T04:49:03.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:03.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:03.440535+0000) 2022-09-08T04:49:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:03.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:03.440660+0000) 2022-09-08T04:49:04.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:04 smithi055 ceph-mon[97033]: pgmap v13942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:49:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:04 smithi059 ceph-mon[97450]: pgmap v13942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:49:04.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:04.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:04.440722+0000) 2022-09-08T04:49:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:04.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:04.440894+0000) 2022-09-08T04:49:05.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:05.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:05.441143+0000) 2022-09-08T04:49:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:05.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:05.440943+0000) 2022-09-08T04:49:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:49:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:49:06.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:06.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:06.441173+0000) 2022-09-08T04:49:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:06.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:06.441323+0000) 2022-09-08T04:49:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:07 smithi055 ceph-mon[97033]: pgmap v13943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:49:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:49:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:07 smithi059 ceph-mon[97450]: pgmap v13943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:49:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:49:07.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:07.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:07.441321+0000) 2022-09-08T04:49:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:07.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:07.441466+0000) 2022-09-08T04:49:08.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:08.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:08.441545+0000) 2022-09-08T04:49:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:08.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:08.441678+0000) 2022-09-08T04:49:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:09 smithi055 ceph-mon[97033]: pgmap v13944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:49:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:09 smithi059 ceph-mon[97450]: pgmap v13944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:49:09.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:09.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:09.441751+0000) 2022-09-08T04:49:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:09.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:09.441824+0000) 2022-09-08T04:49:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:10 smithi055 ceph-mon[97033]: osdmap e953: 12 total, 11 up, 11 in 2022-09-08T04:49:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:10 smithi059 ceph-mon[97450]: osdmap e953: 12 total, 11 up, 11 in 2022-09-08T04:49:10.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:10.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:10.441945+0000) 2022-09-08T04:49:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:10.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:10.442024+0000) 2022-09-08T04:49:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:11 smithi055 ceph-mon[97033]: pgmap v13946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:49:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:11 smithi059 ceph-mon[97450]: pgmap v13946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:49:11.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:11.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:11.442195+0000) 2022-09-08T04:49:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:11.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:11.442195+0000) 2022-09-08T04:49:12.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:12.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:12.442368+0000) 2022-09-08T04:49:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:12.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:12.442340+0000) 2022-09-08T04:49:12.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:12 smithi055 ceph-mon[97033]: pgmap v13947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:49:12.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:12 smithi055 ceph-mon[97033]: pgmap v13948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T04:49:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:12 smithi055 ceph-mon[97033]: pgmap v13949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:49:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:49:13.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:12 smithi059 ceph-mon[97450]: pgmap v13947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:49:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:12 smithi059 ceph-mon[97450]: pgmap v13948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T04:49:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:12 smithi059 ceph-mon[97450]: pgmap v13949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:49:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:49:13.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:13.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:13.442584+0000) 2022-09-08T04:49:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:13.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:13.442597+0000) 2022-09-08T04:49:14.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:14.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:14.442886+0000) 2022-09-08T04:49:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:14.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:14.442795+0000) 2022-09-08T04:49:14.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:14 smithi055 ceph-mon[97033]: pgmap v13950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:49:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:14 smithi059 ceph-mon[97450]: pgmap v13950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:49:15.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:15.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:15.442942+0000) 2022-09-08T04:49:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:15.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:15.443057+0000) 2022-09-08T04:49:16.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:16 smithi055 ceph-mon[97033]: osdmap e954: 12 total, 11 up, 11 in 2022-09-08T04:49:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:16 smithi059 ceph-mon[97450]: osdmap e954: 12 total, 11 up, 11 in 2022-09-08T04:49:16.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:49:16.447+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:49:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:16.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:16.443131+0000) 2022-09-08T04:49:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:16.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:16.443231+0000) 2022-09-08T04:49:17.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:17 smithi055 ceph-mon[97033]: pgmap v13952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.1 KiB/s rd, 327 B/s wr, 10 op/s 2022-09-08T04:49:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:17 smithi059 ceph-mon[97450]: pgmap v13952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.1 KiB/s rd, 327 B/s wr, 10 op/s 2022-09-08T04:49:17.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:17.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:17.443404+0000) 2022-09-08T04:49:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:17.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:17.443482+0000) 2022-09-08T04:49:18.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:18.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:18.443631+0000) 2022-09-08T04:49:18.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:18.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:18.443786+0000) 2022-09-08T04:49:19.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:19 smithi055 ceph-mon[97033]: pgmap v13953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:49:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:19 smithi059 ceph-mon[97450]: pgmap v13953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:49:19.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:19.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:19.443807+0000) 2022-09-08T04:49:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:19.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:19.444004+0000) 2022-09-08T04:49:20.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:20.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:20.444041+0000) 2022-09-08T04:49:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:20.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:20.444225+0000) 2022-09-08T04:49:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:21 smithi055 ceph-mon[97033]: pgmap v13954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T04:49:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:21 smithi059 ceph-mon[97450]: pgmap v13954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T04:49:21.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:21.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:21.444262+0000) 2022-09-08T04:49:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:21.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:21.444425+0000) 2022-09-08T04:49:22.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:22.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:22.444495+0000) 2022-09-08T04:49:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:22.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:22.444572+0000) 2022-09-08T04:49:23.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:23 smithi055 ceph-mon[97033]: pgmap v13955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:49:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:23 smithi059 ceph-mon[97450]: pgmap v13955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:49:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:23.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:23.444662+0000) 2022-09-08T04:49:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:23.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:23.444720+0000) 2022-09-08T04:49:24.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:24.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:24.444830+0000) 2022-09-08T04:49:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:24.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:24.444876+0000) 2022-09-08T04:49:25.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:25 smithi055 ceph-mon[97033]: pgmap v13956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:49:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:25 smithi059 ceph-mon[97450]: pgmap v13956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:49:25.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:25.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:25.445044+0000) 2022-09-08T04:49:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:25.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:25.444974+0000) 2022-09-08T04:49:26.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:26.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:26.445108+0000) 2022-09-08T04:49:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:26.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:26.445204+0000) 2022-09-08T04:49:27.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:27 smithi055 ceph-mon[97033]: pgmap v13957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:49:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:27 smithi059 ceph-mon[97450]: pgmap v13957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:49:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:27.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:27.445251+0000) 2022-09-08T04:49:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:27.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:27.445357+0000) 2022-09-08T04:49:28.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:28.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:28.445429+0000) 2022-09-08T04:49:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:28.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:28.445538+0000) 2022-09-08T04:49:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:29 smithi055 ceph-mon[97033]: pgmap v13958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:49:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:29 smithi059 ceph-mon[97450]: pgmap v13958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:49:29.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:29.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:29.445748+0000) 2022-09-08T04:49:29.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:29.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:29.445681+0000) 2022-09-08T04:49:30.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:30.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:30.445908+0000) 2022-09-08T04:49:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:30.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:30.446000+0000) 2022-09-08T04:49:31.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:31 smithi055 ceph-mon[97033]: pgmap v13959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:49:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:31 smithi059 ceph-mon[97450]: pgmap v13959: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:49:31.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:49:31.452+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:49:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:31.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:31.446088+0000) 2022-09-08T04:49:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:31.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:31.446198+0000) 2022-09-08T04:49:32.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:32.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:32.446312+0000) 2022-09-08T04:49:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:32.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:32.446417+0000) 2022-09-08T04:49:33.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:33 smithi055 ceph-mon[97033]: pgmap v13960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:49:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:33 smithi059 ceph-mon[97450]: pgmap v13960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:49:33.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:33.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:33.446554+0000) 2022-09-08T04:49:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:33.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:33.446610+0000) 2022-09-08T04:49:34.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:34.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:34.446786+0000) 2022-09-08T04:49:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:34.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:34.446793+0000) 2022-09-08T04:49:35.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:35 smithi055 ceph-mon[97033]: pgmap v13961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T04:49:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:35 smithi059 ceph-mon[97450]: pgmap v13961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T04:49:35.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:35.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:35.447014+0000) 2022-09-08T04:49:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:35.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:35.446998+0000) 2022-09-08T04:49:36.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:36.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:36.447277+0000) 2022-09-08T04:49:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:36.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:36.447247+0000) 2022-09-08T04:49:37.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:37 smithi055 ceph-mon[97033]: pgmap v13962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:49:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:37 smithi059 ceph-mon[97450]: pgmap v13962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:49:37.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:37.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:37.447483+0000) 2022-09-08T04:49:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:37.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:37.447481+0000) 2022-09-08T04:49:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:38.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:38.447697+0000) 2022-09-08T04:49:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:38.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:38.447706+0000) 2022-09-08T04:49:39.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:39 smithi055 ceph-mon[97033]: pgmap v13963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:49:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:39 smithi059 ceph-mon[97450]: pgmap v13963: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:49:39.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:39.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:39.447814+0000) 2022-09-08T04:49:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:39.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:39.447836+0000) 2022-09-08T04:49:40.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:40.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:40.447997+0000) 2022-09-08T04:49:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:40.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:40.448000+0000) 2022-09-08T04:49:41.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:41 smithi055 ceph-mon[97033]: pgmap v13964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:49:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:41 smithi059 ceph-mon[97450]: pgmap v13964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:49:41.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:41.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:41.448197+0000) 2022-09-08T04:49:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:41.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:41.448191+0000) 2022-09-08T04:49:42.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:42.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:42.448386+0000) 2022-09-08T04:49:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:42.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:42.448406+0000) 2022-09-08T04:49:43.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:43 smithi055 ceph-mon[97033]: pgmap v13965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:49:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:43 smithi059 ceph-mon[97450]: pgmap v13965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:49:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:43.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:43.448687+0000) 2022-09-08T04:49:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:43.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:43.448619+0000) 2022-09-08T04:49:44.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:44.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:44.448862+0000) 2022-09-08T04:49:44.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:44.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:44.448809+0000) 2022-09-08T04:49:45.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:45 smithi055 ceph-mon[97033]: pgmap v13966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:49:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:45 smithi059 ceph-mon[97450]: pgmap v13966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:49:45.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:45.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:45.449061+0000) 2022-09-08T04:49:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:45.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:45.449076+0000) 2022-09-08T04:49:46.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:49:46.464+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:49:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:46.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:46.449243+0000) 2022-09-08T04:49:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:46.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:46.449239+0000) 2022-09-08T04:49:47.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:47 smithi055 ceph-mon[97033]: pgmap v13967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:49:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:47 smithi059 ceph-mon[97450]: pgmap v13967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:49:47.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:47.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:47.449451+0000) 2022-09-08T04:49:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:47.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:47.449456+0000) 2022-09-08T04:49:48.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:48.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:48.449731+0000) 2022-09-08T04:49:48.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:48.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:48.449721+0000) 2022-09-08T04:49:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:49 smithi055 ceph-mon[97033]: pgmap v13968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:49:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:49 smithi059 ceph-mon[97450]: pgmap v13968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:49:49.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:49.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:49.449947+0000) 2022-09-08T04:49:49.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:49.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:49.449967+0000) 2022-09-08T04:49:50.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:50.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:50.450141+0000) 2022-09-08T04:49:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:50.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:50.450165+0000) 2022-09-08T04:49:51.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:51 smithi055 ceph-mon[97033]: pgmap v13969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:49:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:51 smithi059 ceph-mon[97450]: pgmap v13969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:49:51.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:51.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:51.450365+0000) 2022-09-08T04:49:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:51.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:51.450370+0000) 2022-09-08T04:49:52.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:52.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:52.450595+0000) 2022-09-08T04:49:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:52.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:52.450669+0000) 2022-09-08T04:49:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:53 smithi055 ceph-mon[97033]: pgmap v13970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:49:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:53 smithi059 ceph-mon[97450]: pgmap v13970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:49:53.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:53.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:53.450762+0000) 2022-09-08T04:49:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:53.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:53.450860+0000) 2022-09-08T04:49:54.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:54.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:54.450919+0000) 2022-09-08T04:49:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:54.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:54.450996+0000) 2022-09-08T04:49:55.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:55 smithi055 ceph-mon[97033]: pgmap v13971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:49:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:55 smithi059 ceph-mon[97450]: pgmap v13971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:49:55.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:55.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:55.451141+0000) 2022-09-08T04:49:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:55.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:55.451245+0000) 2022-09-08T04:49:56.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:56.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:56.451304+0000) 2022-09-08T04:49:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:56.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:56.451423+0000) 2022-09-08T04:49:57.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:57 smithi055 ceph-mon[97033]: pgmap v13972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:49:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:57 smithi059 ceph-mon[97450]: pgmap v13972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:49:57.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:57.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:57.451513+0000) 2022-09-08T04:49:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:57.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:57.451592+0000) 2022-09-08T04:49:58.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:58.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:58.451709+0000) 2022-09-08T04:49:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:58.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:58.451764+0000) 2022-09-08T04:49:59.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:49:59 smithi055 ceph-mon[97033]: pgmap v13973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:49:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:49:59 smithi059 ceph-mon[97450]: pgmap v13973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:49:59.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:49:59.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:59.451933+0000) 2022-09-08T04:49:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:49:59.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:49:59.452002+0000) 2022-09-08T04:50:00.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T04:50:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T04:50:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T04:50:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T04:50:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T04:50:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T04:50:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T04:50:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T04:50:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T04:50:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T04:50:00.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:00.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:00.452150+0000) 2022-09-08T04:50:00.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:00.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:00.452184+0000) 2022-09-08T04:50:01.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:01 smithi055 ceph-mon[97033]: pgmap v13974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:50:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:01 smithi059 ceph-mon[97450]: pgmap v13974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:50:01.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:50:01.480+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:50:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:01.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:01.452364+0000) 2022-09-08T04:50:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:01.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:01.452408+0000) 2022-09-08T04:50:02.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:02.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:02.452612+0000) 2022-09-08T04:50:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:02.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:02.452633+0000) 2022-09-08T04:50:03.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:03 smithi055 ceph-mon[97033]: pgmap v13975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:50:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:03 smithi059 ceph-mon[97450]: pgmap v13975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:50:03.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:03.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:03.452829+0000) 2022-09-08T04:50:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:03.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:03.452881+0000) 2022-09-08T04:50:04.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:04.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:04.453054+0000) 2022-09-08T04:50:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:04.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:04.453062+0000) 2022-09-08T04:50:05.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:05 smithi055 ceph-mon[97033]: pgmap v13976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:50:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:05 smithi059 ceph-mon[97450]: pgmap v13976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:50:05.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:05.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:05.453327+0000) 2022-09-08T04:50:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:05.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:05.453311+0000) 2022-09-08T04:50:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:50:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:50:06.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:06.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:06.453600+0000) 2022-09-08T04:50:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:06.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:06.453574+0000) 2022-09-08T04:50:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:07 smithi055 ceph-mon[97033]: pgmap v13977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:50:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:50:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:07 smithi059 ceph-mon[97450]: pgmap v13977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:50:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:50:07.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:07.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:07.453870+0000) 2022-09-08T04:50:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:07.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:07.453828+0000) 2022-09-08T04:50:08.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:08.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:08.454076+0000) 2022-09-08T04:50:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:08.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:08.454012+0000) 2022-09-08T04:50:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:09 smithi055 ceph-mon[97033]: pgmap v13978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:50:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:09 smithi059 ceph-mon[97450]: pgmap v13978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:50:09.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:09.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:09.454256+0000) 2022-09-08T04:50:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:09.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:09.454237+0000) 2022-09-08T04:50:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:10 smithi055 ceph-mon[97033]: osdmap e955: 12 total, 11 up, 11 in 2022-09-08T04:50:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:10 smithi059 ceph-mon[97450]: osdmap e955: 12 total, 11 up, 11 in 2022-09-08T04:50:10.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:10.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:10.454508+0000) 2022-09-08T04:50:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:10.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:10.454521+0000) 2022-09-08T04:50:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:11 smithi055 ceph-mon[97033]: pgmap v13980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:50:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:11 smithi059 ceph-mon[97450]: pgmap v13980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:50:11.798 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:11.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:11.454774+0000) 2022-09-08T04:50:11.799 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:11.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:11.454756+0000) 2022-09-08T04:50:12.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:50:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:50:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:50:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:50:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:50:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:50:12.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:12.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:12.455010+0000) 2022-09-08T04:50:12.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:12.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:12.454985+0000) 2022-09-08T04:50:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:13 smithi055 ceph-mon[97033]: pgmap v13981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:50:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:13 smithi059 ceph-mon[97450]: pgmap v13981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:50:13.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:13.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:13.455216+0000) 2022-09-08T04:50:13.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:13.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:13.455184+0000) 2022-09-08T04:50:14.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:14.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:14.455437+0000) 2022-09-08T04:50:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:14.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:14.455403+0000) 2022-09-08T04:50:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:15 smithi055 ceph-mon[97033]: pgmap v13982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:50:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:15 smithi059 ceph-mon[97450]: pgmap v13982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:50:15.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:15.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:15.455742+0000) 2022-09-08T04:50:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:15.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:15.455683+0000) 2022-09-08T04:50:16.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:50:16.494+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:50:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:16.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:16.455913+0000) 2022-09-08T04:50:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:16.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:16.455859+0000) 2022-09-08T04:50:17.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:17 smithi055 ceph-mon[97033]: pgmap v13983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:50:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:17 smithi059 ceph-mon[97450]: pgmap v13983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:50:17.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:17.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:17.456149+0000) 2022-09-08T04:50:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:17.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:17.456047+0000) 2022-09-08T04:50:18.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:18.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:18.456335+0000) 2022-09-08T04:50:18.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:18.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:18.456285+0000) 2022-09-08T04:50:19.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:19 smithi055 ceph-mon[97033]: pgmap v13984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:50:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:19 smithi059 ceph-mon[97450]: pgmap v13984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:50:19.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:19.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:19.456520+0000) 2022-09-08T04:50:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:19.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:19.456512+0000) 2022-09-08T04:50:20.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:20.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:20.456714+0000) 2022-09-08T04:50:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:20.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:20.456715+0000) 2022-09-08T04:50:21.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:21 smithi055 ceph-mon[97033]: pgmap v13985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 194 B/s wr, 6 op/s 2022-09-08T04:50:21.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:21 smithi055 ceph-mon[97033]: osdmap e956: 12 total, 11 up, 11 in 2022-09-08T04:50:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:21 smithi059 ceph-mon[97450]: pgmap v13985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 194 B/s wr, 6 op/s 2022-09-08T04:50:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:21 smithi059 ceph-mon[97450]: osdmap e956: 12 total, 11 up, 11 in 2022-09-08T04:50:21.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:21.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:21.456854+0000) 2022-09-08T04:50:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:21.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:21.456882+0000) 2022-09-08T04:50:22.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:22.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:22.457098+0000) 2022-09-08T04:50:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:22.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:22.457115+0000) 2022-09-08T04:50:23.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:23 smithi055 ceph-mon[97033]: pgmap v13987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:50:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:23 smithi059 ceph-mon[97450]: pgmap v13987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:50:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:23.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:23.457333+0000) 2022-09-08T04:50:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:23.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:23.457329+0000) 2022-09-08T04:50:24.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:24.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:24.457541+0000) 2022-09-08T04:50:24.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:24.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:24.457583+0000) 2022-09-08T04:50:25.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:25 smithi055 ceph-mon[97033]: pgmap v13988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:50:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:25 smithi059 ceph-mon[97450]: pgmap v13988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:50:25.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:25.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:25.457788+0000) 2022-09-08T04:50:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:25.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:25.457824+0000) 2022-09-08T04:50:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:26.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:26.458061+0000) 2022-09-08T04:50:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:26.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:26.458080+0000) 2022-09-08T04:50:27.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:27 smithi055 ceph-mon[97033]: pgmap v13989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:50:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:27 smithi059 ceph-mon[97450]: pgmap v13989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:50:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:27.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:27.458359+0000) 2022-09-08T04:50:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:27.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:27.458340+0000) 2022-09-08T04:50:28.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:28.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:28.458666+0000) 2022-09-08T04:50:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:28.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:28.458622+0000) 2022-09-08T04:50:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:29 smithi055 ceph-mon[97033]: pgmap v13990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:50:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:29 smithi059 ceph-mon[97450]: pgmap v13990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:50:29.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:29.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:29.458940+0000) 2022-09-08T04:50:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:29.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:29.458924+0000) 2022-09-08T04:50:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:30.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:30.459202+0000) 2022-09-08T04:50:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:30.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:30.459201+0000) 2022-09-08T04:50:31.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:31 smithi055 ceph-mon[97033]: pgmap v13991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:50:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:31 smithi059 ceph-mon[97450]: pgmap v13991: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:50:31.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:31.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:31.459449+0000) 2022-09-08T04:50:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:31.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:31.459421+0000) 2022-09-08T04:50:31.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:50:31.509+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:50:32.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:32.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:32.459709+0000) 2022-09-08T04:50:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:32.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:32.459702+0000) 2022-09-08T04:50:33.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:33 smithi055 ceph-mon[97033]: pgmap v13992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T04:50:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:33 smithi059 ceph-mon[97450]: pgmap v13992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T04:50:33.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:33.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:33.459928+0000) 2022-09-08T04:50:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:33.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:33.459898+0000) 2022-09-08T04:50:34.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:34.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:34.460174+0000) 2022-09-08T04:50:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:34.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:34.460146+0000) 2022-09-08T04:50:35.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:35 smithi055 ceph-mon[97033]: pgmap v13993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:50:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:35 smithi059 ceph-mon[97450]: pgmap v13993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:50:35.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:35.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:35.460414+0000) 2022-09-08T04:50:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:35.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:35.460373+0000) 2022-09-08T04:50:36.872 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:36.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:36.460614+0000) 2022-09-08T04:50:36.873 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:36.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:36.460556+0000) 2022-09-08T04:50:37.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:37 smithi055 ceph-mon[97033]: pgmap v13994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:50:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:37 smithi059 ceph-mon[97450]: pgmap v13994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:50:37.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:37.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:37.460879+0000) 2022-09-08T04:50:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:37.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:37.460816+0000) 2022-09-08T04:50:38.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:38.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:38.461159+0000) 2022-09-08T04:50:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:38.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:38.461104+0000) 2022-09-08T04:50:39.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:39 smithi055 ceph-mon[97033]: pgmap v13995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:50:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:39 smithi059 ceph-mon[97450]: pgmap v13995: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:50:39.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:39.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:39.461422+0000) 2022-09-08T04:50:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:39.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:39.461379+0000) 2022-09-08T04:50:40.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:40.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:40.461613+0000) 2022-09-08T04:50:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:40.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:40.461604+0000) 2022-09-08T04:50:41.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:41 smithi055 ceph-mon[97033]: pgmap v13996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:50:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:41 smithi059 ceph-mon[97450]: pgmap v13996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:50:41.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:41.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:41.461825+0000) 2022-09-08T04:50:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:41.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:41.461830+0000) 2022-09-08T04:50:42.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:42.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:42.462071+0000) 2022-09-08T04:50:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:42.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:42.462028+0000) 2022-09-08T04:50:43.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:42 smithi055 ceph-mon[97033]: pgmap v13997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:50:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:42 smithi055 ceph-mon[97033]: pgmap v13998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:50:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:42 smithi055 ceph-mon[97033]: pgmap v13999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:50:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:50:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:42 smithi059 ceph-mon[97450]: pgmap v13997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:50:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:42 smithi059 ceph-mon[97450]: pgmap v13998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:50:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:42 smithi059 ceph-mon[97450]: pgmap v13999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:50:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:50:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:43.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:43.462270+0000) 2022-09-08T04:50:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:43.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:43.462225+0000) 2022-09-08T04:50:44.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:44.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:44.462347+0000) 2022-09-08T04:50:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:44.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:44.462409+0000) 2022-09-08T04:50:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:44 smithi055 ceph-mon[97033]: pgmap v14000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:50:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:44 smithi059 ceph-mon[97450]: pgmap v14000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:50:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:45.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:45.462596+0000) 2022-09-08T04:50:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:45.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:45.462568+0000) 2022-09-08T04:50:46.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:50:46.515+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:50:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:46.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:46.462805+0000) 2022-09-08T04:50:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:46.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:46.462793+0000) 2022-09-08T04:50:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:47 smithi055 ceph-mon[97033]: pgmap v14001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:50:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:47 smithi059 ceph-mon[97450]: pgmap v14001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:50:47.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:47.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:47.463065+0000) 2022-09-08T04:50:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:47.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:47.463030+0000) 2022-09-08T04:50:48.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:48.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:48.463308+0000) 2022-09-08T04:50:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:48.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:48.463308+0000) 2022-09-08T04:50:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:49 smithi055 ceph-mon[97033]: pgmap v14002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:50:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:49 smithi059 ceph-mon[97450]: pgmap v14002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:50:49.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:49.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:49.463533+0000) 2022-09-08T04:50:49.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:49.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:49.463510+0000) 2022-09-08T04:50:50.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:50.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:50.463785+0000) 2022-09-08T04:50:50.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:50.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:50.463786+0000) 2022-09-08T04:50:51.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:51 smithi055 ceph-mon[97033]: pgmap v14003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 6 op/s 2022-09-08T04:50:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:51 smithi059 ceph-mon[97450]: pgmap v14003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 6 op/s 2022-09-08T04:50:51.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:51.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:51.464006+0000) 2022-09-08T04:50:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:51.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:51.464006+0000) 2022-09-08T04:50:52.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:52.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:52.464275+0000) 2022-09-08T04:50:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:52.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:52.464275+0000) 2022-09-08T04:50:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:53 smithi055 ceph-mon[97033]: pgmap v14004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:50:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:53 smithi059 ceph-mon[97450]: pgmap v14004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:50:53.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:53.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:53.464557+0000) 2022-09-08T04:50:53.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:53.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:53.464548+0000) 2022-09-08T04:50:54.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:54.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:54.464811+0000) 2022-09-08T04:50:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:54.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:54.464813+0000) 2022-09-08T04:50:55.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:55 smithi055 ceph-mon[97033]: pgmap v14005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:50:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:55 smithi059 ceph-mon[97450]: pgmap v14005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:50:55.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:55.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:55.464964+0000) 2022-09-08T04:50:55.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:55.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:55.464971+0000) 2022-09-08T04:50:56.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:56.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:56.465165+0000) 2022-09-08T04:50:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:56.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:56.465165+0000) 2022-09-08T04:50:57.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:57 smithi055 ceph-mon[97033]: pgmap v14006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:50:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:57 smithi059 ceph-mon[97450]: pgmap v14006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:50:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:57.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:57.465409+0000) 2022-09-08T04:50:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:57.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:57.465407+0000) 2022-09-08T04:50:58.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:58.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:58.465736+0000) 2022-09-08T04:50:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:58.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:58.465736+0000) 2022-09-08T04:50:59.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:50:59 smithi055 ceph-mon[97033]: pgmap v14007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:50:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:50:59 smithi059 ceph-mon[97450]: pgmap v14007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:50:59.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:50:59.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:59.465988+0000) 2022-09-08T04:50:59.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:50:59.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:50:59.465989+0000) 2022-09-08T04:51:00.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:00.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:00.466172+0000) 2022-09-08T04:51:00.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:00.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:00.466173+0000) 2022-09-08T04:51:01.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:01 smithi055 ceph-mon[97033]: pgmap v14008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T04:51:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:01 smithi059 ceph-mon[97450]: pgmap v14008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T04:51:01.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:51:01.518+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:51:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:01.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:01.466365+0000) 2022-09-08T04:51:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:01.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:01.466353+0000) 2022-09-08T04:51:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:02.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:02.466549+0000) 2022-09-08T04:51:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:02.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:02.466549+0000) 2022-09-08T04:51:03.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:03 smithi055 ceph-mon[97033]: pgmap v14009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T04:51:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:03 smithi059 ceph-mon[97450]: pgmap v14009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T04:51:03.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:03.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:03.466821+0000) 2022-09-08T04:51:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:03.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:03.466816+0000) 2022-09-08T04:51:04.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:04.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:04.467023+0000) 2022-09-08T04:51:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:04.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:04.467025+0000) 2022-09-08T04:51:05.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:05 smithi055 ceph-mon[97033]: pgmap v14010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:51:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:05 smithi059 ceph-mon[97450]: pgmap v14010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:51:05.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:05.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:05.467224+0000) 2022-09-08T04:51:05.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:05.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:05.467226+0000) 2022-09-08T04:51:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:51:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:51:06.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:06.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:06.467445+0000) 2022-09-08T04:51:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:06.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:06.467442+0000) 2022-09-08T04:51:07.470 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:07.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:07.467745+0000) 2022-09-08T04:51:07.471 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:07.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:07.467745+0000) 2022-09-08T04:51:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:07 smithi055 ceph-mon[97033]: pgmap v14011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:51:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:51:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:07 smithi059 ceph-mon[97450]: pgmap v14011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:51:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:51:08.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:08.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:08.467902+0000) 2022-09-08T04:51:08.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:08.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:08.467888+0000) 2022-09-08T04:51:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:09 smithi055 ceph-mon[97033]: pgmap v14012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:51:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:09 smithi059 ceph-mon[97450]: pgmap v14012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:51:09.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:09.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:09.468089+0000) 2022-09-08T04:51:09.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:09.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:09.468108+0000) 2022-09-08T04:51:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:10 smithi055 ceph-mon[97033]: osdmap e957: 12 total, 11 up, 11 in 2022-09-08T04:51:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:10 smithi059 ceph-mon[97450]: osdmap e957: 12 total, 11 up, 11 in 2022-09-08T04:51:10.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:10.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:10.468334+0000) 2022-09-08T04:51:10.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:10.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:10.468324+0000) 2022-09-08T04:51:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:11 smithi055 ceph-mon[97033]: pgmap v14014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:51:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:11 smithi059 ceph-mon[97450]: pgmap v14014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:51:11.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:11.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:11.468481+0000) 2022-09-08T04:51:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:11.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:11.468531+0000) 2022-09-08T04:51:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:12.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:12.468727+0000) 2022-09-08T04:51:12.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:12.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:12.468724+0000) 2022-09-08T04:51:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:13 smithi055 ceph-mon[97033]: pgmap v14015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:51:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:13 smithi059 ceph-mon[97450]: pgmap v14015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:51:13.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:13.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:13.468868+0000) 2022-09-08T04:51:13.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:13.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:13.468899+0000) 2022-09-08T04:51:14.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:14.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:14.469052+0000) 2022-09-08T04:51:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:14.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:14.469041+0000) 2022-09-08T04:51:15.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:15 smithi055 ceph-mon[97033]: pgmap v14016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:51:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:15 smithi059 ceph-mon[97450]: pgmap v14016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T04:51:15.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:15.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:15.469299+0000) 2022-09-08T04:51:15.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:15.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:15.469299+0000) 2022-09-08T04:51:16.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:16.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:16.469568+0000) 2022-09-08T04:51:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:16.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:16.469572+0000) 2022-09-08T04:51:16.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:51:16.529+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:51:17.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:17 smithi055 ceph-mon[97033]: pgmap v14017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:51:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:17 smithi059 ceph-mon[97450]: pgmap v14017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:51:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:17.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:17.469718+0000) 2022-09-08T04:51:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:17.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:17.469718+0000) 2022-09-08T04:51:18.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:18.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:18.469872+0000) 2022-09-08T04:51:18.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:18.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:18.469874+0000) 2022-09-08T04:51:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:19 smithi055 ceph-mon[97033]: pgmap v14018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:51:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:19 smithi059 ceph-mon[97450]: pgmap v14018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:51:19.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:19.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:19.470130+0000) 2022-09-08T04:51:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:19.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:19.470131+0000) 2022-09-08T04:51:20.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:20.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:20.470341+0000) 2022-09-08T04:51:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:20.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:20.470340+0000) 2022-09-08T04:51:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:21 smithi055 ceph-mon[97033]: pgmap v14019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 189 B/s wr, 7 op/s 2022-09-08T04:51:21.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:21 smithi055 ceph-mon[97033]: osdmap e958: 12 total, 11 up, 11 in 2022-09-08T04:51:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:21 smithi059 ceph-mon[97450]: pgmap v14019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 189 B/s wr, 7 op/s 2022-09-08T04:51:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:21 smithi059 ceph-mon[97450]: osdmap e958: 12 total, 11 up, 11 in 2022-09-08T04:51:21.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:21.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:21.470562+0000) 2022-09-08T04:51:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:21.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:21.470563+0000) 2022-09-08T04:51:22.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:22.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:22.470838+0000) 2022-09-08T04:51:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:22.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:22.470838+0000) 2022-09-08T04:51:23.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:23 smithi055 ceph-mon[97033]: pgmap v14021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:51:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:23 smithi059 ceph-mon[97450]: pgmap v14021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:51:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:23.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:23.471063+0000) 2022-09-08T04:51:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:23.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:23.471034+0000) 2022-09-08T04:51:24.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:24.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:24.471259+0000) 2022-09-08T04:51:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:24.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:24.471251+0000) 2022-09-08T04:51:25.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:25 smithi055 ceph-mon[97033]: pgmap v14022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:51:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:25 smithi059 ceph-mon[97450]: pgmap v14022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:51:25.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:25.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:25.471477+0000) 2022-09-08T04:51:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:25.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:25.471475+0000) 2022-09-08T04:51:26.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:26.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:26.471713+0000) 2022-09-08T04:51:26.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:26.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:26.471716+0000) 2022-09-08T04:51:27.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:27 smithi055 ceph-mon[97033]: pgmap v14023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:51:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:27 smithi059 ceph-mon[97450]: pgmap v14023: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:51:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:27.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:27.471852+0000) 2022-09-08T04:51:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:27.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:27.471875+0000) 2022-09-08T04:51:28.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:28.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:28.471979+0000) 2022-09-08T04:51:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:28.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:28.472035+0000) 2022-09-08T04:51:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:29 smithi055 ceph-mon[97033]: pgmap v14024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:51:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:29 smithi059 ceph-mon[97450]: pgmap v14024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:51:29.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:29.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:29.472229+0000) 2022-09-08T04:51:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:29.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:29.472246+0000) 2022-09-08T04:51:30.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:30.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:30.472479+0000) 2022-09-08T04:51:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:30.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:30.472479+0000) 2022-09-08T04:51:31.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:31 smithi055 ceph-mon[97033]: pgmap v14025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:51:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:31 smithi059 ceph-mon[97450]: pgmap v14025: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:51:31.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:51:31.542+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:51:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:31.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:31.472752+0000) 2022-09-08T04:51:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:31.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:31.472772+0000) 2022-09-08T04:51:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:32.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:32.473017+0000) 2022-09-08T04:51:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:32.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:32.473017+0000) 2022-09-08T04:51:33.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:33 smithi055 ceph-mon[97033]: pgmap v14026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:51:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:33 smithi059 ceph-mon[97450]: pgmap v14026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:51:33.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:33.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:33.473238+0000) 2022-09-08T04:51:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:33.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:33.473211+0000) 2022-09-08T04:51:34.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:34.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:34.473385+0000) 2022-09-08T04:51:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:34.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:34.473385+0000) 2022-09-08T04:51:35.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:35 smithi055 ceph-mon[97033]: pgmap v14027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:51:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:35 smithi059 ceph-mon[97450]: pgmap v14027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:51:35.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:35.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:35.473548+0000) 2022-09-08T04:51:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:35.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:35.473549+0000) 2022-09-08T04:51:36.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:36.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:36.473831+0000) 2022-09-08T04:51:36.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:36.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:36.473793+0000) 2022-09-08T04:51:37.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:37 smithi055 ceph-mon[97033]: pgmap v14028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:51:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:37 smithi059 ceph-mon[97450]: pgmap v14028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:51:37.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:37.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:37.474072+0000) 2022-09-08T04:51:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:37.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:37.474029+0000) 2022-09-08T04:51:38.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:38.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:38.474229+0000) 2022-09-08T04:51:38.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:38.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:38.474175+0000) 2022-09-08T04:51:39.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:39 smithi055 ceph-mon[97033]: pgmap v14029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:51:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:39 smithi059 ceph-mon[97450]: pgmap v14029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:51:39.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:39.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:39.474456+0000) 2022-09-08T04:51:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:39.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:39.474401+0000) 2022-09-08T04:51:40.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:40.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:40.474688+0000) 2022-09-08T04:51:40.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:40.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:40.474632+0000) 2022-09-08T04:51:41.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:41 smithi055 ceph-mon[97033]: pgmap v14030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T04:51:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:41 smithi059 ceph-mon[97450]: pgmap v14030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T04:51:41.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:41.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:41.474894+0000) 2022-09-08T04:51:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:41.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:41.474825+0000) 2022-09-08T04:51:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:51:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:51:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:51:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:51:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:51:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:51:42.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:42.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:42.475131+0000) 2022-09-08T04:51:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:42.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:42.475074+0000) 2022-09-08T04:51:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:43 smithi059 ceph-mon[97450]: pgmap v14031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T04:51:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:43.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:43.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:43.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:43 smithi055 ceph-mon[97033]: pgmap v14031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T04:51:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:43.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:43.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:43.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:43.475266+0000) 2022-09-08T04:51:43.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:43.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:43.475364+0000) 2022-09-08T04:51:44.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:44.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:44.475541+0000) 2022-09-08T04:51:44.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:44.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:44.475579+0000) 2022-09-08T04:51:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:45 smithi055 ceph-mon[97033]: pgmap v14032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:51:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:45 smithi059 ceph-mon[97450]: pgmap v14032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:51:45.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:45.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:45.475789+0000) 2022-09-08T04:51:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:45.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:45.475995+0000) 2022-09-08T04:51:46.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:51:46.556+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:51:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:46.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:46.476021+0000) 2022-09-08T04:51:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:46.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:46.476241+0000) 2022-09-08T04:51:46.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:46 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T04:51:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:46 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T04:51:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:46.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:46 smithi055 ceph-mon[97033]: pgmap v14033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:51:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:46 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T04:51:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:46 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T04:51:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:51:47.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:46 smithi059 ceph-mon[97450]: pgmap v14033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:51:47.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:47.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:47.476247+0000) 2022-09-08T04:51:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:47.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:47.476457+0000) 2022-09-08T04:51:48.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:48.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:48.476447+0000) 2022-09-08T04:51:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:48.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:48.476616+0000) 2022-09-08T04:51:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:49 smithi059 ceph-mon[97450]: pgmap v14034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:51:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:49 smithi055 ceph-mon[97033]: pgmap v14034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:51:49.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:49.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:49.476655+0000) 2022-09-08T04:51:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:49.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:49.476791+0000) 2022-09-08T04:51:50.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:50.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:50.476876+0000) 2022-09-08T04:51:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:50.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:50.476953+0000) 2022-09-08T04:51:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:51 smithi055 ceph-mon[97033]: pgmap v14035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T04:51:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:51 smithi059 ceph-mon[97450]: pgmap v14035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T04:51:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:51.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:51.477106+0000) 2022-09-08T04:51:51.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:51.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:51.477136+0000) 2022-09-08T04:51:52.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:52.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:52.477272+0000) 2022-09-08T04:51:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:52.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:52.477323+0000) 2022-09-08T04:51:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:53 smithi055 ceph-mon[97033]: pgmap v14036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T04:51:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:53 smithi059 ceph-mon[97450]: pgmap v14036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T04:51:53.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:53.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:53.477514+0000) 2022-09-08T04:51:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:53.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:53.477559+0000) 2022-09-08T04:51:54.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:54.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:54.477686+0000) 2022-09-08T04:51:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:54.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:54.477706+0000) 2022-09-08T04:51:55.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:55 smithi055 ceph-mon[97033]: pgmap v14037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:51:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:55 smithi059 ceph-mon[97450]: pgmap v14037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:51:55.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:55.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:55.477869+0000) 2022-09-08T04:51:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:55.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:55.477874+0000) 2022-09-08T04:51:56.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:56.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:56.478097+0000) 2022-09-08T04:51:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:56.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:56.478098+0000) 2022-09-08T04:51:57.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:57 smithi055 ceph-mon[97033]: pgmap v14038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:51:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:57 smithi059 ceph-mon[97450]: pgmap v14038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:51:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:57.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:57.478359+0000) 2022-09-08T04:51:57.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:57.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:57.478370+0000) 2022-09-08T04:51:58.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:58.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:58.478616+0000) 2022-09-08T04:51:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:58.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:58.478603+0000) 2022-09-08T04:51:59.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:51:59 smithi055 ceph-mon[97033]: pgmap v14039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:51:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:51:59 smithi059 ceph-mon[97450]: pgmap v14039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:51:59.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:51:59.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:59.478839+0000) 2022-09-08T04:51:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:51:59.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:51:59.478872+0000) 2022-09-08T04:52:00.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:00.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:00.478993+0000) 2022-09-08T04:52:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:00.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:00.479013+0000) 2022-09-08T04:52:01.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:01 smithi055 ceph-mon[97033]: pgmap v14040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T04:52:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:01 smithi059 ceph-mon[97450]: pgmap v14040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T04:52:01.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:01.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:01.479220+0000) 2022-09-08T04:52:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:01.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:01.479239+0000) 2022-09-08T04:52:01.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:52:01.557+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:52:02.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:02.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:02.479535+0000) 2022-09-08T04:52:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:02.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:02.479538+0000) 2022-09-08T04:52:03.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:03 smithi055 ceph-mon[97033]: pgmap v14041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 5 op/s 2022-09-08T04:52:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:03 smithi059 ceph-mon[97450]: pgmap v14041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 5 op/s 2022-09-08T04:52:03.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:03.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:03.479806+0000) 2022-09-08T04:52:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:03.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:03.479806+0000) 2022-09-08T04:52:04.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:04.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:04.479986+0000) 2022-09-08T04:52:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:04.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:04.479985+0000) 2022-09-08T04:52:05.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:05 smithi055 ceph-mon[97033]: pgmap v14042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:52:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:05 smithi059 ceph-mon[97450]: pgmap v14042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:52:05.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:05.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:05.480246+0000) 2022-09-08T04:52:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:05.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:05.480244+0000) 2022-09-08T04:52:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:52:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:52:06.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:06.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:06.480525+0000) 2022-09-08T04:52:06.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:06.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:06.480538+0000) 2022-09-08T04:52:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:07 smithi055 ceph-mon[97033]: pgmap v14043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:52:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:52:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:07 smithi059 ceph-mon[97450]: pgmap v14043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:52:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:52:07.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:07.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:07.480770+0000) 2022-09-08T04:52:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:07.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:07.480780+0000) 2022-09-08T04:52:08.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:08.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:08.480995+0000) 2022-09-08T04:52:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:08.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:08.480995+0000) 2022-09-08T04:52:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:09 smithi055 ceph-mon[97033]: pgmap v14044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:52:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:09 smithi059 ceph-mon[97450]: pgmap v14044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:52:09.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:09.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:09.481228+0000) 2022-09-08T04:52:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:09.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:09.481228+0000) 2022-09-08T04:52:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:10 smithi055 ceph-mon[97033]: osdmap e959: 12 total, 11 up, 11 in 2022-09-08T04:52:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:10 smithi059 ceph-mon[97450]: osdmap e959: 12 total, 11 up, 11 in 2022-09-08T04:52:10.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:10.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:10.481392+0000) 2022-09-08T04:52:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:10.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:10.481397+0000) 2022-09-08T04:52:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:11 smithi055 ceph-mon[97033]: pgmap v14046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:52:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:11 smithi059 ceph-mon[97450]: pgmap v14046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:52:11.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:11.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:11.481664+0000) 2022-09-08T04:52:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:11.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:11.481663+0000) 2022-09-08T04:52:12.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:12.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:12.481887+0000) 2022-09-08T04:52:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:12.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:12.481896+0000) 2022-09-08T04:52:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:13 smithi055 ceph-mon[97033]: pgmap v14047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:52:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:13 smithi055 ceph-mon[97033]: pgmap v14048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T04:52:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:13 smithi055 ceph-mon[97033]: pgmap v14049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:52:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:52:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:13 smithi059 ceph-mon[97450]: pgmap v14047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:52:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:13 smithi059 ceph-mon[97450]: pgmap v14048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T04:52:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:13 smithi059 ceph-mon[97450]: pgmap v14049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:52:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:52:13.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:13.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:13.482120+0000) 2022-09-08T04:52:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:13.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:13.482117+0000) 2022-09-08T04:52:14.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:14.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:14.482328+0000) 2022-09-08T04:52:14.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:14.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:14.482328+0000) 2022-09-08T04:52:15.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:15 smithi055 ceph-mon[97033]: pgmap v14050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:52:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:15 smithi055 ceph-mon[97033]: osdmap e960: 12 total, 11 up, 11 in 2022-09-08T04:52:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:15 smithi059 ceph-mon[97450]: pgmap v14050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:52:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:15 smithi059 ceph-mon[97450]: osdmap e960: 12 total, 11 up, 11 in 2022-09-08T04:52:15.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:15.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:15.482608+0000) 2022-09-08T04:52:15.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:15.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:15.482606+0000) 2022-09-08T04:52:16.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:52:16.567+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:52:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:16.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:16.482941+0000) 2022-09-08T04:52:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:16.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:16.482941+0000) 2022-09-08T04:52:17.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:17 smithi055 ceph-mon[97033]: pgmap v14052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 163 B/s wr, 6 op/s 2022-09-08T04:52:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:17 smithi059 ceph-mon[97450]: pgmap v14052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 163 B/s wr, 6 op/s 2022-09-08T04:52:17.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:17.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:17.483167+0000) 2022-09-08T04:52:17.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:17.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:17.483170+0000) 2022-09-08T04:52:18.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:18.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:18.483418+0000) 2022-09-08T04:52:18.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:18.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:18.483418+0000) 2022-09-08T04:52:19.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:19 smithi055 ceph-mon[97033]: pgmap v14053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:52:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:19 smithi059 ceph-mon[97450]: pgmap v14053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T04:52:19.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:19.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:19.483678+0000) 2022-09-08T04:52:19.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:19.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:19.483674+0000) 2022-09-08T04:52:20.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:20.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:20.483902+0000) 2022-09-08T04:52:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:20.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:20.483897+0000) 2022-09-08T04:52:21.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:21 smithi055 ceph-mon[97033]: pgmap v14054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T04:52:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:21 smithi059 ceph-mon[97450]: pgmap v14054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T04:52:21.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:21.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:21.484132+0000) 2022-09-08T04:52:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:21.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:21.484139+0000) 2022-09-08T04:52:22.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:22.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:22.484291+0000) 2022-09-08T04:52:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:22.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:22.484316+0000) 2022-09-08T04:52:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:23 smithi059 ceph-mon[97450]: pgmap v14055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:52:23.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:23 smithi055 ceph-mon[97033]: pgmap v14055: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:52:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:23.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:23.484516+0000) 2022-09-08T04:52:23.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:23.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:23.484515+0000) 2022-09-08T04:52:24.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:24.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:24.484759+0000) 2022-09-08T04:52:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:24.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:24.484725+0000) 2022-09-08T04:52:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:25 smithi059 ceph-mon[97450]: pgmap v14056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:52:25.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:25 smithi055 ceph-mon[97033]: pgmap v14056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:52:25.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:25.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:25.484919+0000) 2022-09-08T04:52:25.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:25.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:25.484863+0000) 2022-09-08T04:52:26.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:26 smithi055 ceph-mon[97033]: pgmap v14057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:52:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:26 smithi059 ceph-mon[97450]: pgmap v14057: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:52:26.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:26.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:26.485084+0000) 2022-09-08T04:52:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:26.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:26.485059+0000) 2022-09-08T04:52:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:27.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:27.485251+0000) 2022-09-08T04:52:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:27.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:27.485265+0000) 2022-09-08T04:52:28.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:28.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:28.485451+0000) 2022-09-08T04:52:28.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:28.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:28.485464+0000) 2022-09-08T04:52:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:29 smithi059 ceph-mon[97450]: pgmap v14058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:29.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:29 smithi055 ceph-mon[97033]: pgmap v14058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:29.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:29.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:29.485592+0000) 2022-09-08T04:52:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:29.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:29.485656+0000) 2022-09-08T04:52:30.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:30.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:30.485721+0000) 2022-09-08T04:52:30.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:30.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:30.485807+0000) 2022-09-08T04:52:31.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:31 smithi055 ceph-mon[97033]: pgmap v14059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:52:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:31 smithi059 ceph-mon[97450]: pgmap v14059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:52:31.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:52:31.581+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:52:31.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:31.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:31.486032+0000) 2022-09-08T04:52:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:31.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:31.485967+0000) 2022-09-08T04:52:32.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:32.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:32.486280+0000) 2022-09-08T04:52:32.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:32.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:32.486279+0000) 2022-09-08T04:52:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:33 smithi059 ceph-mon[97450]: pgmap v14060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:33.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:33 smithi055 ceph-mon[97033]: pgmap v14060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:33.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:33.486578+0000) 2022-09-08T04:52:33.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:33.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:33.486572+0000) 2022-09-08T04:52:34.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:34.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:34.486827+0000) 2022-09-08T04:52:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:34.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:34.486832+0000) 2022-09-08T04:52:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:35 smithi059 ceph-mon[97450]: pgmap v14061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:35.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:35 smithi055 ceph-mon[97033]: pgmap v14061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:35.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:35.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:35.487057+0000) 2022-09-08T04:52:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:35.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:35.487057+0000) 2022-09-08T04:52:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:36 smithi059 ceph-mon[97450]: pgmap v14062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:52:36.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:36 smithi055 ceph-mon[97033]: pgmap v14062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:52:36.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:36.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:36.487339+0000) 2022-09-08T04:52:36.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:36.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:36.487340+0000) 2022-09-08T04:52:37.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:37.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:37.487599+0000) 2022-09-08T04:52:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:37.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:37.487599+0000) 2022-09-08T04:52:38.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:38.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:38.487949+0000) 2022-09-08T04:52:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:38.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:38.487949+0000) 2022-09-08T04:52:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:39 smithi059 ceph-mon[97450]: pgmap v14063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:39.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:39 smithi055 ceph-mon[97033]: pgmap v14063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:39.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:39.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:39.488198+0000) 2022-09-08T04:52:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:39.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:39.488197+0000) 2022-09-08T04:52:40.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:40.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:40.488486+0000) 2022-09-08T04:52:40.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:40.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:40.488492+0000) 2022-09-08T04:52:41.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:41 smithi055 ceph-mon[97033]: pgmap v14064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:52:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:41 smithi059 ceph-mon[97450]: pgmap v14064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:52:41.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:41.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:41.488716+0000) 2022-09-08T04:52:41.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:41.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:41.488718+0000) 2022-09-08T04:52:42.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:42.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:42.489024+0000) 2022-09-08T04:52:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:42.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:42.489007+0000) 2022-09-08T04:52:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:43 smithi059 ceph-mon[97450]: pgmap v14065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:43 smithi055 ceph-mon[97033]: pgmap v14065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:43.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:43.489316+0000) 2022-09-08T04:52:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:43.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:43.489330+0000) 2022-09-08T04:52:44.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:44 smithi055 ceph-mon[97033]: pgmap v14066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:44 smithi059 ceph-mon[97450]: pgmap v14066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:44.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:44.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:44.489483+0000) 2022-09-08T04:52:44.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:44.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:44.489504+0000) 2022-09-08T04:52:45.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:45.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:45.489756+0000) 2022-09-08T04:52:45.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:45.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:45.489773+0000) 2022-09-08T04:52:46.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:52:46.589+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:52:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:46.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:46.489928+0000) 2022-09-08T04:52:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:46.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:46.489955+0000) 2022-09-08T04:52:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:47 smithi059 ceph-mon[97450]: pgmap v14067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:52:47.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:47 smithi055 ceph-mon[97033]: pgmap v14067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:52:47.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:47.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:47.490119+0000) 2022-09-08T04:52:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:47.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:47.490119+0000) 2022-09-08T04:52:48.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:48 smithi055 ceph-mon[97033]: pgmap v14068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:48 smithi059 ceph-mon[97450]: pgmap v14068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:48.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:48.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:48.490378+0000) 2022-09-08T04:52:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:48.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:48.490377+0000) 2022-09-08T04:52:49.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:49.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:49.490618+0000) 2022-09-08T04:52:49.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:49.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:49.490618+0000) 2022-09-08T04:52:50.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:50.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:50.490917+0000) 2022-09-08T04:52:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:50.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:50.490927+0000) 2022-09-08T04:52:51.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:51 smithi055 ceph-mon[97033]: pgmap v14069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:52:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:51 smithi059 ceph-mon[97450]: pgmap v14069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:52:51.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:51.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:51.491170+0000) 2022-09-08T04:52:51.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:51.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:51.491152+0000) 2022-09-08T04:52:52.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:52.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:52.491505+0000) 2022-09-08T04:52:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:52.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:52.491448+0000) 2022-09-08T04:52:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:53 smithi059 ceph-mon[97450]: pgmap v14070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:53.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:53 smithi055 ceph-mon[97033]: pgmap v14070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:53.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:53.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:53.491735+0000) 2022-09-08T04:52:53.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:53.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:53.491769+0000) 2022-09-08T04:52:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:54 smithi059 ceph-mon[97450]: pgmap v14071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:54.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:54 smithi055 ceph-mon[97033]: pgmap v14071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:54.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:54.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:54.491988+0000) 2022-09-08T04:52:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:54.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:54.492033+0000) 2022-09-08T04:52:55.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:55.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:55.492116+0000) 2022-09-08T04:52:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:55.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:55.492203+0000) 2022-09-08T04:52:56.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:56.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:56.492294+0000) 2022-09-08T04:52:56.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:56.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:56.492358+0000) 2022-09-08T04:52:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:57 smithi059 ceph-mon[97450]: pgmap v14072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:52:57.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:57 smithi055 ceph-mon[97033]: pgmap v14072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:52:57.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:57.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:57.492485+0000) 2022-09-08T04:52:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:57.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:57.492504+0000) 2022-09-08T04:52:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:52:58 smithi059 ceph-mon[97450]: pgmap v14073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:58.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:52:58 smithi055 ceph-mon[97033]: pgmap v14073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:52:58.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:58.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:58.492776+0000) 2022-09-08T04:52:58.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:58.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:58.492763+0000) 2022-09-08T04:52:59.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:52:59.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:59.493067+0000) 2022-09-08T04:52:59.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:52:59.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:52:59.493066+0000) 2022-09-08T04:53:00.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:00.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:00.493328+0000) 2022-09-08T04:53:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:00.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:00.493327+0000) 2022-09-08T04:53:01.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:01 smithi055 ceph-mon[97033]: pgmap v14074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:53:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:01 smithi059 ceph-mon[97450]: pgmap v14074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:53:01.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:53:01.601+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:53:01.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:01.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:01.493528+0000) 2022-09-08T04:53:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:01.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:01.493534+0000) 2022-09-08T04:53:02.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:02 smithi055 ceph-mon[97033]: pgmap v14075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:53:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:02 smithi059 ceph-mon[97450]: pgmap v14075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:53:02.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:02.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:02.493774+0000) 2022-09-08T04:53:02.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:02.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:02.493780+0000) 2022-09-08T04:53:03.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:03.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:03.494067+0000) 2022-09-08T04:53:03.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:03.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:03.494065+0000) 2022-09-08T04:53:04.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:04 smithi055 ceph-mon[97033]: pgmap v14076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:53:04.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:04 smithi059 ceph-mon[97450]: pgmap v14076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:53:04.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:04.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:04.494249+0000) 2022-09-08T04:53:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:04.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:04.494249+0000) 2022-09-08T04:53:05.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:05.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:05.494502+0000) 2022-09-08T04:53:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:05.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:05.494502+0000) 2022-09-08T04:53:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:53:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:53:06.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:06.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:06.494812+0000) 2022-09-08T04:53:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:06.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:06.494813+0000) 2022-09-08T04:53:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:07 smithi055 ceph-mon[97033]: pgmap v14077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:53:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:53:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:07 smithi059 ceph-mon[97450]: pgmap v14077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:53:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:53:07.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:07.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:07.495042+0000) 2022-09-08T04:53:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:07.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:07.495042+0000) 2022-09-08T04:53:08.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:08 smithi055 ceph-mon[97033]: pgmap v14078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:53:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:08 smithi059 ceph-mon[97450]: pgmap v14078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:53:08.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:08.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:08.495264+0000) 2022-09-08T04:53:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:08.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:08.495262+0000) 2022-09-08T04:53:09.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:09.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:09.495535+0000) 2022-09-08T04:53:09.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:09.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:09.495541+0000) 2022-09-08T04:53:10.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:10 smithi055 ceph-mon[97033]: osdmap e961: 12 total, 11 up, 11 in 2022-09-08T04:53:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:10 smithi055 ceph-mon[97033]: pgmap v14080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:53:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:10 smithi059 ceph-mon[97450]: osdmap e961: 12 total, 11 up, 11 in 2022-09-08T04:53:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:10 smithi059 ceph-mon[97450]: pgmap v14080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:53:10.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:10.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:10.495786+0000) 2022-09-08T04:53:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:10.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:10.495786+0000) 2022-09-08T04:53:11.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:11.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:11.496062+0000) 2022-09-08T04:53:11.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:11.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:11.496062+0000) 2022-09-08T04:53:12.650 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:12.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:12.496351+0000) 2022-09-08T04:53:12.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:12.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:12.496364+0000) 2022-09-08T04:53:12.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:12 smithi055 ceph-mon[97033]: pgmap v14081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:53:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:12 smithi059 ceph-mon[97450]: pgmap v14081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:53:13.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:53:13.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:53:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:53:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:53:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:53:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:53:13.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:13.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:13.496702+0000) 2022-09-08T04:53:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:13.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:13.496683+0000) 2022-09-08T04:53:14.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:14 smithi055 ceph-mon[97033]: pgmap v14082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:53:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:14 smithi059 ceph-mon[97450]: pgmap v14082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:53:14.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:14.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:14.496884+0000) 2022-09-08T04:53:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:14.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:14.496883+0000) 2022-09-08T04:53:15.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:15.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:15.497106+0000) 2022-09-08T04:53:15.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:15.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:15.497106+0000) 2022-09-08T04:53:16.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:16 smithi055 ceph-mon[97033]: pgmap v14083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:53:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:16 smithi059 ceph-mon[97450]: pgmap v14083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:53:16.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:53:16.616+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:53:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:16.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:16.497381+0000) 2022-09-08T04:53:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:16.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:16.497381+0000) 2022-09-08T04:53:17.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:17.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:17.497721+0000) 2022-09-08T04:53:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:17.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:17.497723+0000) 2022-09-08T04:53:18.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:18 smithi055 ceph-mon[97033]: pgmap v14084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:53:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:18 smithi059 ceph-mon[97450]: pgmap v14084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:53:18.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:18.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:18.497990+0000) 2022-09-08T04:53:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:18.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:18.497991+0000) 2022-09-08T04:53:19.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:19.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:19.498192+0000) 2022-09-08T04:53:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:19.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:19.498161+0000) 2022-09-08T04:53:20.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:20.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:20.498398+0000) 2022-09-08T04:53:20.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:20.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:20.498421+0000) 2022-09-08T04:53:21.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:21 smithi055 ceph-mon[97033]: osdmap e962: 12 total, 11 up, 11 in 2022-09-08T04:53:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:21 smithi055 ceph-mon[97033]: pgmap v14086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:53:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:21 smithi059 ceph-mon[97450]: osdmap e962: 12 total, 11 up, 11 in 2022-09-08T04:53:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:21 smithi059 ceph-mon[97450]: pgmap v14086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:53:21.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:21.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:21.498612+0000) 2022-09-08T04:53:21.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:21.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:21.498624+0000) 2022-09-08T04:53:22.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:22 smithi055 ceph-mon[97033]: pgmap v14087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:53:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:22 smithi059 ceph-mon[97450]: pgmap v14087: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:53:22.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:22.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:22.498827+0000) 2022-09-08T04:53:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:22.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:22.498882+0000) 2022-09-08T04:53:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:23.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:23.499027+0000) 2022-09-08T04:53:23.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:23.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:23.499150+0000) 2022-09-08T04:53:24.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:24 smithi055 ceph-mon[97033]: pgmap v14088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:53:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:24 smithi059 ceph-mon[97450]: pgmap v14088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:53:24.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:24.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:24.499268+0000) 2022-09-08T04:53:24.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:24.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:24.499405+0000) 2022-09-08T04:53:25.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:25.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:25.499534+0000) 2022-09-08T04:53:25.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:25.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:25.499627+0000) 2022-09-08T04:53:26.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:26 smithi055 ceph-mon[97033]: pgmap v14089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:53:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:26 smithi059 ceph-mon[97450]: pgmap v14089: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:53:26.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:26.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:26.499816+0000) 2022-09-08T04:53:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:26.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:26.499887+0000) 2022-09-08T04:53:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:27.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:27.500093+0000) 2022-09-08T04:53:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:27.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:27.500115+0000) 2022-09-08T04:53:28.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:28 smithi055 ceph-mon[97033]: pgmap v14090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:53:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:28 smithi059 ceph-mon[97450]: pgmap v14090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:53:28.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:28.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:28.500396+0000) 2022-09-08T04:53:28.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:28.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:28.500393+0000) 2022-09-08T04:53:29.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:29.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:29.500650+0000) 2022-09-08T04:53:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:29.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:29.500651+0000) 2022-09-08T04:53:30.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:30.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:30.500842+0000) 2022-09-08T04:53:30.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:30.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:30.500842+0000) 2022-09-08T04:53:31.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:31 smithi055 ceph-mon[97033]: pgmap v14091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:53:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:31 smithi059 ceph-mon[97450]: pgmap v14091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:53:31.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:53:31.618+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:53:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:31.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:31.501047+0000) 2022-09-08T04:53:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:31.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:31.501048+0000) 2022-09-08T04:53:32.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:32 smithi055 ceph-mon[97033]: pgmap v14092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:53:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:32 smithi059 ceph-mon[97450]: pgmap v14092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:53:32.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:32.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:32.501314+0000) 2022-09-08T04:53:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:32.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:32.501314+0000) 2022-09-08T04:53:33.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:33.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:33.501595+0000) 2022-09-08T04:53:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:33.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:33.501578+0000) 2022-09-08T04:53:34.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:34 smithi055 ceph-mon[97033]: pgmap v14093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:53:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:34 smithi059 ceph-mon[97450]: pgmap v14093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:53:34.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:34.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:34.501837+0000) 2022-09-08T04:53:34.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:34.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:34.501843+0000) 2022-09-08T04:53:35.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:35.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:35.502101+0000) 2022-09-08T04:53:35.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:35.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:35.502098+0000) 2022-09-08T04:53:36.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:36 smithi055 ceph-mon[97033]: pgmap v14094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:53:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:36 smithi059 ceph-mon[97450]: pgmap v14094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:53:36.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:36.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:36.502336+0000) 2022-09-08T04:53:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:36.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:36.502336+0000) 2022-09-08T04:53:37.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:37.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:37.502514+0000) 2022-09-08T04:53:37.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:37.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:37.502512+0000) 2022-09-08T04:53:38.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:38 smithi055 ceph-mon[97033]: pgmap v14095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:53:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:38 smithi059 ceph-mon[97450]: pgmap v14095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:53:38.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:38.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:38.502788+0000) 2022-09-08T04:53:38.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:38.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:38.502788+0000) 2022-09-08T04:53:39.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:39.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:39.503029+0000) 2022-09-08T04:53:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:39.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:39.503029+0000) 2022-09-08T04:53:40.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:40.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:40.503233+0000) 2022-09-08T04:53:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:40.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:40.503236+0000) 2022-09-08T04:53:41.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:41 smithi055 ceph-mon[97033]: pgmap v14096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:53:41.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:41 smithi059 ceph-mon[97450]: pgmap v14096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:53:41.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:41.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:41.503494+0000) 2022-09-08T04:53:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:41.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:41.503494+0000) 2022-09-08T04:53:42.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:42 smithi055 ceph-mon[97033]: pgmap v14097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:53:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:42 smithi059 ceph-mon[97450]: pgmap v14097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:53:42.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:42.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:42.503782+0000) 2022-09-08T04:53:42.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:42.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:42.503781+0000) 2022-09-08T04:53:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:43 smithi059 ceph-mon[97450]: pgmap v14098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:53:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:43 smithi059 ceph-mon[97450]: pgmap v14099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:53:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:53:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:43.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:43.504007+0000) 2022-09-08T04:53:43.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:43.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:43.504007+0000) 2022-09-08T04:53:43.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:43 smithi055 ceph-mon[97033]: pgmap v14098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:53:43.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:43 smithi055 ceph-mon[97033]: pgmap v14099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:53:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:53:44.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:44.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:44.504201+0000) 2022-09-08T04:53:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:44.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:44.504201+0000) 2022-09-08T04:53:45.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:45.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:45.504449+0000) 2022-09-08T04:53:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:45.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:45.504451+0000) 2022-09-08T04:53:46.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:46 smithi055 ceph-mon[97033]: pgmap v14100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:53:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:46 smithi059 ceph-mon[97450]: pgmap v14100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:53:46.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:53:46.622+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:53:46.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:46.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:46.504777+0000) 2022-09-08T04:53:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:46.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:46.504789+0000) 2022-09-08T04:53:47.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:47.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:47.505083+0000) 2022-09-08T04:53:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:47.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:47.505081+0000) 2022-09-08T04:53:48.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:48 smithi055 ceph-mon[97033]: pgmap v14101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:53:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:48 smithi059 ceph-mon[97450]: pgmap v14101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:53:48.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:48.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:48.505308+0000) 2022-09-08T04:53:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:48.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:48.505307+0000) 2022-09-08T04:53:49.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:49.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:49.505562+0000) 2022-09-08T04:53:49.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:49.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:49.505562+0000) 2022-09-08T04:53:50.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:50 smithi055 ceph-mon[97033]: pgmap v14102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:53:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:50 smithi059 ceph-mon[97450]: pgmap v14102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T04:53:50.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:50.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:50.505908+0000) 2022-09-08T04:53:50.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:50.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:50.505867+0000) 2022-09-08T04:53:51.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:51.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:51.506109+0000) 2022-09-08T04:53:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:51.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:51.506086+0000) 2022-09-08T04:53:52.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:52 smithi055 ceph-mon[97033]: pgmap v14103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:53:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:52 smithi059 ceph-mon[97450]: pgmap v14103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T04:53:52.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:52.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:52.506327+0000) 2022-09-08T04:53:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:52.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:52.506327+0000) 2022-09-08T04:53:53.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:53.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:53.506526+0000) 2022-09-08T04:53:53.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:53.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:53.506531+0000) 2022-09-08T04:53:54.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:54 smithi055 ceph-mon[97033]: pgmap v14104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:53:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:54 smithi059 ceph-mon[97450]: pgmap v14104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:53:54.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:54.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:54.506733+0000) 2022-09-08T04:53:54.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:54.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:54.506736+0000) 2022-09-08T04:53:55.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:55.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:55.506983+0000) 2022-09-08T04:53:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:55.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:55.506983+0000) 2022-09-08T04:53:56.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:56 smithi055 ceph-mon[97033]: pgmap v14105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:53:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:56 smithi059 ceph-mon[97450]: pgmap v14105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:53:56.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:56.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:56.507265+0000) 2022-09-08T04:53:56.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:56.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:56.507273+0000) 2022-09-08T04:53:57.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:57.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:57.507532+0000) 2022-09-08T04:53:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:57.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:57.507531+0000) 2022-09-08T04:53:58.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:53:58 smithi055 ceph-mon[97033]: pgmap v14106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:53:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:53:58 smithi059 ceph-mon[97450]: pgmap v14106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:53:58.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:58.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:58.507747+0000) 2022-09-08T04:53:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:58.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:58.507750+0000) 2022-09-08T04:53:59.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:53:59.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:59.507946+0000) 2022-09-08T04:53:59.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:53:59.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:53:59.507945+0000) 2022-09-08T04:54:00.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:00 smithi055 ceph-mon[97033]: pgmap v14107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:00 smithi059 ceph-mon[97450]: pgmap v14107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:00.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:00.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:00.508178+0000) 2022-09-08T04:54:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:00.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:00.508177+0000) 2022-09-08T04:54:01.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:01.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:01.508407+0000) 2022-09-08T04:54:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:01.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:01.508411+0000) 2022-09-08T04:54:01.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:54:01.630+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:54:02.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:02 smithi055 ceph-mon[97033]: pgmap v14108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:54:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:02 smithi059 ceph-mon[97450]: pgmap v14108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:54:02.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:02.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:02.508649+0000) 2022-09-08T04:54:02.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:02.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:02.508642+0000) 2022-09-08T04:54:03.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:03.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:03.508919+0000) 2022-09-08T04:54:03.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:03.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:03.508916+0000) 2022-09-08T04:54:04.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:04 smithi055 ceph-mon[97033]: pgmap v14109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:04 smithi059 ceph-mon[97450]: pgmap v14109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:04.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:04.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:04.509138+0000) 2022-09-08T04:54:04.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:04.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:04.509138+0000) 2022-09-08T04:54:05.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:05.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:05.509376+0000) 2022-09-08T04:54:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:05.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:05.509376+0000) 2022-09-08T04:54:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:06 smithi055 ceph-mon[97033]: pgmap v14110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:54:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:06 smithi059 ceph-mon[97450]: pgmap v14110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:54:06.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:06.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:06.509609+0000) 2022-09-08T04:54:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:06.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:06.509609+0000) 2022-09-08T04:54:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:54:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:54:07.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:07.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:07.509801+0000) 2022-09-08T04:54:07.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:07.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:07.509796+0000) 2022-09-08T04:54:08.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:08 smithi055 ceph-mon[97033]: pgmap v14111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:54:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:08 smithi059 ceph-mon[97450]: pgmap v14111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:54:08.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:08.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:08.509968+0000) 2022-09-08T04:54:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:08.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:08.509974+0000) 2022-09-08T04:54:09.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:09.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:09.510146+0000) 2022-09-08T04:54:09.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:09.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:09.510146+0000) 2022-09-08T04:54:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:10 smithi055 ceph-mon[97033]: pgmap v14112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:54:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:10 smithi055 ceph-mon[97033]: osdmap e963: 12 total, 11 up, 11 in 2022-09-08T04:54:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:10 smithi059 ceph-mon[97450]: pgmap v14112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:54:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:10 smithi059 ceph-mon[97450]: osdmap e963: 12 total, 11 up, 11 in 2022-09-08T04:54:10.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:10.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:10.510385+0000) 2022-09-08T04:54:10.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:10.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:10.510387+0000) 2022-09-08T04:54:11.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:11.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:11.510691+0000) 2022-09-08T04:54:11.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:11.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:11.510713+0000) 2022-09-08T04:54:12.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:12 smithi055 ceph-mon[97033]: pgmap v14114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:54:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:12 smithi059 ceph-mon[97450]: pgmap v14114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:54:12.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:12.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:12.510921+0000) 2022-09-08T04:54:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:12.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:12.510916+0000) 2022-09-08T04:54:13.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:13.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:13.511095+0000) 2022-09-08T04:54:13.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:13.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:13.511095+0000) 2022-09-08T04:54:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:14 smithi055 ceph-mon[97033]: pgmap v14115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:54:14.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:14 smithi059 ceph-mon[97450]: pgmap v14115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:54:14.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:14.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:14.511364+0000) 2022-09-08T04:54:14.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:14.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:14.511364+0000) 2022-09-08T04:54:15.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:15.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:15.511748+0000) 2022-09-08T04:54:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:15.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:15.511761+0000) 2022-09-08T04:54:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:16 smithi055 ceph-mon[97033]: pgmap v14116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:54:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:16 smithi055 ceph-mon[97033]: osdmap e964: 12 total, 11 up, 11 in 2022-09-08T04:54:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:16 smithi059 ceph-mon[97450]: pgmap v14116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:54:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:16 smithi059 ceph-mon[97450]: osdmap e964: 12 total, 11 up, 11 in 2022-09-08T04:54:16.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:54:16.638+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:54:16.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:16.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:16.512041+0000) 2022-09-08T04:54:16.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:16.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:16.512000+0000) 2022-09-08T04:54:17.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:17.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:17.512269+0000) 2022-09-08T04:54:17.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:17.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:17.512275+0000) 2022-09-08T04:54:18.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:18 smithi055 ceph-mon[97033]: pgmap v14118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:54:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:18 smithi059 ceph-mon[97450]: pgmap v14118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:54:18.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:18.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:18.512581+0000) 2022-09-08T04:54:18.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:18.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:18.512568+0000) 2022-09-08T04:54:19.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:19.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:19.512864+0000) 2022-09-08T04:54:19.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:19.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:19.512836+0000) 2022-09-08T04:54:20.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:20 smithi055 ceph-mon[97033]: pgmap v14119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-08T04:54:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:20 smithi059 ceph-mon[97450]: pgmap v14119: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-08T04:54:20.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:20.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:20.513060+0000) 2022-09-08T04:54:20.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:20.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:20.513094+0000) 2022-09-08T04:54:21.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:21.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:21.513262+0000) 2022-09-08T04:54:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:21.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:21.513266+0000) 2022-09-08T04:54:22.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:22 smithi055 ceph-mon[97033]: pgmap v14120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:54:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:22 smithi059 ceph-mon[97450]: pgmap v14120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:54:22.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:22.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:22.513494+0000) 2022-09-08T04:54:22.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:22.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:22.513494+0000) 2022-09-08T04:54:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:23.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:23.513706+0000) 2022-09-08T04:54:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:23.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:23.513688+0000) 2022-09-08T04:54:24.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:24 smithi055 ceph-mon[97033]: pgmap v14121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:54:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:24 smithi059 ceph-mon[97450]: pgmap v14121: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:54:24.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:24.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:24.513897+0000) 2022-09-08T04:54:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:24.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:24.513896+0000) 2022-09-08T04:54:25.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:25.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:25.514089+0000) 2022-09-08T04:54:25.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:25.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:25.514097+0000) 2022-09-08T04:54:26.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:26 smithi055 ceph-mon[97033]: pgmap v14122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:54:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:26 smithi059 ceph-mon[97450]: pgmap v14122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:54:26.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:26.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:26.514362+0000) 2022-09-08T04:54:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:26.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:26.514361+0000) 2022-09-08T04:54:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:27.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:27.514672+0000) 2022-09-08T04:54:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:27.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:27.514668+0000) 2022-09-08T04:54:28.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:28 smithi055 ceph-mon[97033]: pgmap v14123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T04:54:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:28 smithi059 ceph-mon[97450]: pgmap v14123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T04:54:28.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:28.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:28.514888+0000) 2022-09-08T04:54:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:28.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:28.514886+0000) 2022-09-08T04:54:29.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:29.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:29.515153+0000) 2022-09-08T04:54:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:29.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:29.515153+0000) 2022-09-08T04:54:30.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:30 smithi055 ceph-mon[97033]: pgmap v14124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:30 smithi059 ceph-mon[97450]: pgmap v14124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:30.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:30.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:30.515396+0000) 2022-09-08T04:54:30.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:30.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:30.515396+0000) 2022-09-08T04:54:31.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:54:31.649+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:54:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:31.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:31.515616+0000) 2022-09-08T04:54:31.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:31.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:31.515608+0000) 2022-09-08T04:54:32.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:32 smithi055 ceph-mon[97033]: pgmap v14125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:54:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:32 smithi059 ceph-mon[97450]: pgmap v14125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:54:32.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:32.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:32.515859+0000) 2022-09-08T04:54:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:32.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:32.515862+0000) 2022-09-08T04:54:33.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:33.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:33.516154+0000) 2022-09-08T04:54:33.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:33.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:33.516154+0000) 2022-09-08T04:54:34.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:34 smithi055 ceph-mon[97033]: pgmap v14126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:34 smithi059 ceph-mon[97450]: pgmap v14126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:34.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:34.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:34.516425+0000) 2022-09-08T04:54:34.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:34.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:34.516369+0000) 2022-09-08T04:54:35.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:35.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:35.516611+0000) 2022-09-08T04:54:35.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:35.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:35.516611+0000) 2022-09-08T04:54:36.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:36 smithi055 ceph-mon[97033]: pgmap v14127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:36 smithi059 ceph-mon[97450]: pgmap v14127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:36.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:36.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:36.516853+0000) 2022-09-08T04:54:36.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:36.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:36.516808+0000) 2022-09-08T04:54:37.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:37.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:37.517060+0000) 2022-09-08T04:54:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:37.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:37.517060+0000) 2022-09-08T04:54:38.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:38 smithi055 ceph-mon[97033]: pgmap v14128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:54:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:38 smithi059 ceph-mon[97450]: pgmap v14128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:54:38.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:38.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:38.517314+0000) 2022-09-08T04:54:38.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:38.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:38.517304+0000) 2022-09-08T04:54:39.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:39.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:39.517472+0000) 2022-09-08T04:54:39.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:39.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:39.517472+0000) 2022-09-08T04:54:40.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:40 smithi055 ceph-mon[97033]: pgmap v14129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:40 smithi059 ceph-mon[97450]: pgmap v14129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:40.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:40.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:40.517679+0000) 2022-09-08T04:54:40.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:40.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:40.517684+0000) 2022-09-08T04:54:41.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:41.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:41.517904+0000) 2022-09-08T04:54:41.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:41.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:41.517906+0000) 2022-09-08T04:54:42.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:42 smithi055 ceph-mon[97033]: pgmap v14130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:54:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:42 smithi059 ceph-mon[97450]: pgmap v14130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:54:42.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:42.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:42.518142+0000) 2022-09-08T04:54:42.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:42.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:42.518142+0000) 2022-09-08T04:54:43.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:54:43.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:54:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:54:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:54:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:54:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:54:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:43.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:43.518361+0000) 2022-09-08T04:54:43.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:43.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:43.518366+0000) 2022-09-08T04:54:44.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:44 smithi055 ceph-mon[97033]: pgmap v14131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:44 smithi059 ceph-mon[97450]: pgmap v14131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:44.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:44.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:44.518616+0000) 2022-09-08T04:54:44.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:44.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:44.518613+0000) 2022-09-08T04:54:45.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:45.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:45.518948+0000) 2022-09-08T04:54:45.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:45.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:45.518951+0000) 2022-09-08T04:54:46.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:46 smithi055 ceph-mon[97033]: pgmap v14132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:46 smithi059 ceph-mon[97450]: pgmap v14132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:46.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:54:46.660+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:54:46.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:46.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:46.519238+0000) 2022-09-08T04:54:46.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:46.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:46.519238+0000) 2022-09-08T04:54:47.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:47.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:47.519543+0000) 2022-09-08T04:54:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:47.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:47.519547+0000) 2022-09-08T04:54:48.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:48 smithi055 ceph-mon[97033]: pgmap v14133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:54:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:48 smithi059 ceph-mon[97450]: pgmap v14133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:54:48.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:48.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:48.519745+0000) 2022-09-08T04:54:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:48.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:48.519744+0000) 2022-09-08T04:54:49.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:49.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:49.519999+0000) 2022-09-08T04:54:49.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:49.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:49.519999+0000) 2022-09-08T04:54:50.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:50 smithi055 ceph-mon[97033]: pgmap v14134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:50 smithi059 ceph-mon[97450]: pgmap v14134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:50.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:50.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:50.520240+0000) 2022-09-08T04:54:50.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:50.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:50.520237+0000) 2022-09-08T04:54:51.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:51.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:51.520439+0000) 2022-09-08T04:54:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:51.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:51.520439+0000) 2022-09-08T04:54:52.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:52 smithi055 ceph-mon[97033]: pgmap v14135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:54:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:52 smithi059 ceph-mon[97450]: pgmap v14135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:54:52.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:52.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:52.520610+0000) 2022-09-08T04:54:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:52.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:52.520610+0000) 2022-09-08T04:54:53.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:53.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:53.520901+0000) 2022-09-08T04:54:53.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:53.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:53.520897+0000) 2022-09-08T04:54:54.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:54 smithi055 ceph-mon[97033]: pgmap v14136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:54 smithi059 ceph-mon[97450]: pgmap v14136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:54.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:54.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:54.521147+0000) 2022-09-08T04:54:54.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:54.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:54.521144+0000) 2022-09-08T04:54:55.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:55.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:55.521357+0000) 2022-09-08T04:54:55.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:55.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:55.521360+0000) 2022-09-08T04:54:56.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:56 smithi055 ceph-mon[97033]: pgmap v14137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:56 smithi059 ceph-mon[97450]: pgmap v14137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:54:56.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:56.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:56.521590+0000) 2022-09-08T04:54:56.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:56.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:56.521590+0000) 2022-09-08T04:54:57.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:57.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:57.521851+0000) 2022-09-08T04:54:57.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:57.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:57.521873+0000) 2022-09-08T04:54:58.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:54:58 smithi055 ceph-mon[97033]: pgmap v14138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:54:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:54:58 smithi059 ceph-mon[97450]: pgmap v14138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:54:58.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:58.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:58.522048+0000) 2022-09-08T04:54:58.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:58.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:58.522048+0000) 2022-09-08T04:54:59.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:54:59.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:59.522307+0000) 2022-09-08T04:54:59.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:54:59.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:54:59.522302+0000) 2022-09-08T04:55:00.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:00 smithi055 ceph-mon[97033]: pgmap v14139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:00 smithi059 ceph-mon[97450]: pgmap v14139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:00.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:00.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:00.522542+0000) 2022-09-08T04:55:00.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:00.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:00.522542+0000) 2022-09-08T04:55:01.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:01.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:01.522773+0000) 2022-09-08T04:55:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:01.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:01.522781+0000) 2022-09-08T04:55:01.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:55:01.672+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:55:02.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:02 smithi055 ceph-mon[97033]: pgmap v14140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:55:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:02 smithi059 ceph-mon[97450]: pgmap v14140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:55:02.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:02.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:02.522904+0000) 2022-09-08T04:55:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:02.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:02.522937+0000) 2022-09-08T04:55:03.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:03.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:03.523072+0000) 2022-09-08T04:55:03.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:03.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:03.523096+0000) 2022-09-08T04:55:04.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:04 smithi055 ceph-mon[97033]: pgmap v14141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:04.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:04 smithi059 ceph-mon[97450]: pgmap v14141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:04.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:04.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:04.523293+0000) 2022-09-08T04:55:04.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:04.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:04.523317+0000) 2022-09-08T04:55:05.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:05.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:05.523536+0000) 2022-09-08T04:55:05.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:05.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:05.523570+0000) 2022-09-08T04:55:06.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:06 smithi055 ceph-mon[97033]: pgmap v14142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:55:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:06 smithi059 ceph-mon[97450]: pgmap v14142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:55:06.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:06.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:06.523748+0000) 2022-09-08T04:55:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:06.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:06.523773+0000) 2022-09-08T04:55:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:55:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:55:07.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:07.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:07.523954+0000) 2022-09-08T04:55:07.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:07.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:07.523962+0000) 2022-09-08T04:55:08.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:08 smithi055 ceph-mon[97033]: pgmap v14143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:55:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:08 smithi059 ceph-mon[97450]: pgmap v14143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:55:08.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:08.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:08.524163+0000) 2022-09-08T04:55:08.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:08.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:08.524167+0000) 2022-09-08T04:55:09.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:09.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:09.524377+0000) 2022-09-08T04:55:09.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:09.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:09.524402+0000) 2022-09-08T04:55:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:10 smithi059 ceph-mon[97450]: pgmap v14144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:55:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:10 smithi059 ceph-mon[97450]: osdmap e965: 12 total, 11 up, 11 in 2022-09-08T04:55:10.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:10 smithi055 ceph-mon[97033]: pgmap v14144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:55:10.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:10 smithi055 ceph-mon[97033]: osdmap e965: 12 total, 11 up, 11 in 2022-09-08T04:55:10.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:10.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:10.524557+0000) 2022-09-08T04:55:10.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:10.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:10.524593+0000) 2022-09-08T04:55:11.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:11.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:11.524803+0000) 2022-09-08T04:55:11.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:11.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:11.524845+0000) 2022-09-08T04:55:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:12 smithi059 ceph-mon[97450]: pgmap v14146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:55:12.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:12 smithi055 ceph-mon[97033]: pgmap v14146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:55:12.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:12.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:12.525070+0000) 2022-09-08T04:55:12.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:12.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:12.525032+0000) 2022-09-08T04:55:13.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:13.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:13.525203+0000) 2022-09-08T04:55:13.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:13.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:13.525247+0000) 2022-09-08T04:55:14.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:13 smithi059 ceph-mon[97450]: pgmap v14147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:55:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:13 smithi059 ceph-mon[97450]: pgmap v14148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T04:55:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:13 smithi059 ceph-mon[97450]: pgmap v14149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T04:55:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:55:14.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:13 smithi055 ceph-mon[97033]: pgmap v14147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:55:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:13 smithi055 ceph-mon[97033]: pgmap v14148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T04:55:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:13 smithi055 ceph-mon[97033]: pgmap v14149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T04:55:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:55:14.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:14.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:14.525469+0000) 2022-09-08T04:55:14.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:14.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:14.525469+0000) 2022-09-08T04:55:15.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:15.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:15.525781+0000) 2022-09-08T04:55:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:15.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:15.525781+0000) 2022-09-08T04:55:16.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:16 smithi055 ceph-mon[97033]: pgmap v14150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T04:55:16.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:16 smithi055 ceph-mon[97033]: osdmap e966: 12 total, 11 up, 11 in 2022-09-08T04:55:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:16 smithi059 ceph-mon[97450]: pgmap v14150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T04:55:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:16 smithi059 ceph-mon[97450]: osdmap e966: 12 total, 11 up, 11 in 2022-09-08T04:55:16.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:55:16.677+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:55:16.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:16.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:16.526021+0000) 2022-09-08T04:55:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:16.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:16.526032+0000) 2022-09-08T04:55:17.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:17.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:17.526302+0000) 2022-09-08T04:55:17.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:17.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:17.526305+0000) 2022-09-08T04:55:18.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:18 smithi055 ceph-mon[97033]: pgmap v14152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T04:55:18.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:18 smithi059 ceph-mon[97450]: pgmap v14152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T04:55:18.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:18.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:18.526614+0000) 2022-09-08T04:55:18.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:18.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:18.526620+0000) 2022-09-08T04:55:19.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:19.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:19.526851+0000) 2022-09-08T04:55:19.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:19.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:19.526867+0000) 2022-09-08T04:55:20.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:20 smithi055 ceph-mon[97033]: pgmap v14153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T04:55:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:20 smithi059 ceph-mon[97450]: pgmap v14153: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T04:55:20.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:20.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:20.527115+0000) 2022-09-08T04:55:20.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:20.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:20.527115+0000) 2022-09-08T04:55:21.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:21.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:21.527301+0000) 2022-09-08T04:55:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:21.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:21.527335+0000) 2022-09-08T04:55:22.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:22 smithi055 ceph-mon[97033]: pgmap v14154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T04:55:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:22 smithi059 ceph-mon[97450]: pgmap v14154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T04:55:22.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:22.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:22.527549+0000) 2022-09-08T04:55:22.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:22.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:22.527576+0000) 2022-09-08T04:55:23.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:23.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:23.527814+0000) 2022-09-08T04:55:23.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:23.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:23.527816+0000) 2022-09-08T04:55:24.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:24 smithi055 ceph-mon[97033]: pgmap v14155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:55:24.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:24 smithi059 ceph-mon[97450]: pgmap v14155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:55:24.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:24.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:24.528080+0000) 2022-09-08T04:55:24.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:24.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:24.528020+0000) 2022-09-08T04:55:25.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:25.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:25.528267+0000) 2022-09-08T04:55:25.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:25.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:25.528250+0000) 2022-09-08T04:55:26.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:26 smithi055 ceph-mon[97033]: pgmap v14156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:55:26.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:26 smithi059 ceph-mon[97450]: pgmap v14156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:55:26.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:26.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:26.528469+0000) 2022-09-08T04:55:26.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:26.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:26.528471+0000) 2022-09-08T04:55:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:27.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:27.528760+0000) 2022-09-08T04:55:27.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:27.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:27.528744+0000) 2022-09-08T04:55:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:28 smithi055 ceph-mon[97033]: pgmap v14157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T04:55:28.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:28 smithi059 ceph-mon[97450]: pgmap v14157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T04:55:28.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:28.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:28.528995+0000) 2022-09-08T04:55:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:28.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:28.528987+0000) 2022-09-08T04:55:29.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:29.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:29.529194+0000) 2022-09-08T04:55:29.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:29.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:29.529234+0000) 2022-09-08T04:55:30.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:30 smithi055 ceph-mon[97033]: pgmap v14158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:30.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:30 smithi059 ceph-mon[97450]: pgmap v14158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:30.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:30.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:30.529332+0000) 2022-09-08T04:55:30.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:30.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:30.529429+0000) 2022-09-08T04:55:31.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:31.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:31.529556+0000) 2022-09-08T04:55:31.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:31.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:31.529693+0000) 2022-09-08T04:55:31.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:55:31.690+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:55:32.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:32 smithi055 ceph-mon[97033]: pgmap v14159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:55:32.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:32 smithi059 ceph-mon[97450]: pgmap v14159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:55:32.794 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:32.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:32.529819+0000) 2022-09-08T04:55:32.795 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:32.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:32.529895+0000) 2022-09-08T04:55:33.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:33.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:33.529994+0000) 2022-09-08T04:55:33.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:33.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:33.530118+0000) 2022-09-08T04:55:34.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:34 smithi055 ceph-mon[97033]: pgmap v14160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:34.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:34 smithi059 ceph-mon[97450]: pgmap v14160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:34.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:34.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:34.530229+0000) 2022-09-08T04:55:34.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:34.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:34.530395+0000) 2022-09-08T04:55:35.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:35.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:35.530457+0000) 2022-09-08T04:55:35.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:35.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:35.530690+0000) 2022-09-08T04:55:36.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:36 smithi055 ceph-mon[97033]: pgmap v14161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:36 smithi059 ceph-mon[97450]: pgmap v14161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:36.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:36.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:36.530679+0000) 2022-09-08T04:55:36.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:36.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:36.530835+0000) 2022-09-08T04:55:37.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:37.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:37.530899+0000) 2022-09-08T04:55:37.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:37.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:37.531005+0000) 2022-09-08T04:55:38.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:38 smithi055 ceph-mon[97033]: pgmap v14162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:55:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:38 smithi059 ceph-mon[97450]: pgmap v14162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:55:38.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:38.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:38.531166+0000) 2022-09-08T04:55:38.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:38.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:38.531053+0000) 2022-09-08T04:55:39.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:39.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:39.531281+0000) 2022-09-08T04:55:39.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:39.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:39.531386+0000) 2022-09-08T04:55:40.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:40 smithi055 ceph-mon[97033]: pgmap v14163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:40 smithi059 ceph-mon[97450]: pgmap v14163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:40.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:40.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:40.531508+0000) 2022-09-08T04:55:40.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:40.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:40.531626+0000) 2022-09-08T04:55:41.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:41.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:41.531731+0000) 2022-09-08T04:55:41.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:41.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:41.531870+0000) 2022-09-08T04:55:42.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:42 smithi055 ceph-mon[97033]: pgmap v14164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:55:42.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:42 smithi059 ceph-mon[97450]: pgmap v14164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:55:42.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:42.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:42.532039+0000) 2022-09-08T04:55:42.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:42.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:42.531898+0000) 2022-09-08T04:55:43.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:43.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:43.532253+0000) 2022-09-08T04:55:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:43.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:43.532165+0000) 2022-09-08T04:55:44.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:44 smithi055 ceph-mon[97033]: pgmap v14165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:44 smithi059 ceph-mon[97450]: pgmap v14165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:44.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:44.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:44.532419+0000) 2022-09-08T04:55:44.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:44.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:44.532469+0000) 2022-09-08T04:55:45.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:45.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:45.532696+0000) 2022-09-08T04:55:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:45.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:45.532739+0000) 2022-09-08T04:55:46.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:46 smithi055 ceph-mon[97033]: pgmap v14166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:46 smithi059 ceph-mon[97450]: pgmap v14166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:46.899 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:55:46.701+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:55:46.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:46.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:46.532962+0000) 2022-09-08T04:55:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:46.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:46.533017+0000) 2022-09-08T04:55:47.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:47.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:47.533202+0000) 2022-09-08T04:55:47.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:47.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:47.533277+0000) 2022-09-08T04:55:48.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:48 smithi055 ceph-mon[97033]: pgmap v14167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:55:48.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:48 smithi059 ceph-mon[97450]: pgmap v14167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:55:48.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:48.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:48.533446+0000) 2022-09-08T04:55:48.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:48.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:48.533579+0000) 2022-09-08T04:55:49.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:49.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:49.533647+0000) 2022-09-08T04:55:49.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:49.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:49.533713+0000) 2022-09-08T04:55:50.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:50 smithi055 ceph-mon[97033]: pgmap v14168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:50.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:50 smithi059 ceph-mon[97450]: pgmap v14168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:50.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:50.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:50.533855+0000) 2022-09-08T04:55:50.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:50.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:50.533886+0000) 2022-09-08T04:55:51.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:51.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:51.533978+0000) 2022-09-08T04:55:51.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:51.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:51.534028+0000) 2022-09-08T04:55:52.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:52 smithi055 ceph-mon[97033]: pgmap v14169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:55:52.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:52 smithi059 ceph-mon[97450]: pgmap v14169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:55:52.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:52.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:52.534234+0000) 2022-09-08T04:55:52.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:52.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:52.534190+0000) 2022-09-08T04:55:53.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:53.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:53.534424+0000) 2022-09-08T04:55:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:53.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:53.534466+0000) 2022-09-08T04:55:54.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:54 smithi055 ceph-mon[97033]: pgmap v14170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:54.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:54 smithi059 ceph-mon[97450]: pgmap v14170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:54.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:54.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:54.534700+0000) 2022-09-08T04:55:54.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:54.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:54.534709+0000) 2022-09-08T04:55:55.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:55.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:55.534875+0000) 2022-09-08T04:55:55.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:55.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:55.534946+0000) 2022-09-08T04:55:56.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:56 smithi055 ceph-mon[97033]: pgmap v14171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:56 smithi059 ceph-mon[97450]: pgmap v14171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:55:56.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:56.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:56.535064+0000) 2022-09-08T04:55:56.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:56.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:56.535128+0000) 2022-09-08T04:55:57.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:57.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:57.535301+0000) 2022-09-08T04:55:57.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:57.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:57.535340+0000) 2022-09-08T04:55:58.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:55:58 smithi055 ceph-mon[97033]: pgmap v14172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:55:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:55:58 smithi059 ceph-mon[97450]: pgmap v14172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:55:58.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:58.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:58.535542+0000) 2022-09-08T04:55:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:58.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:58.535603+0000) 2022-09-08T04:55:59.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:55:59.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:59.535772+0000) 2022-09-08T04:55:59.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:55:59.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:55:59.535832+0000) 2022-09-08T04:56:00.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:00 smithi055 ceph-mon[97033]: pgmap v14173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:56:00.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:00 smithi059 ceph-mon[97450]: pgmap v14173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:56:00.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:00.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:00.535909+0000) 2022-09-08T04:56:00.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:00.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:00.535965+0000) 2022-09-08T04:56:01.899 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:56:01.707+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:56:01.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:01.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:01.536031+0000) 2022-09-08T04:56:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:01.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:01.536136+0000) 2022-09-08T04:56:02.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:02 smithi055 ceph-mon[97033]: pgmap v14174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:56:02.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:02 smithi059 ceph-mon[97450]: pgmap v14174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:56:02.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:02.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:02.536308+0000) 2022-09-08T04:56:02.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:02.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:02.536229+0000) 2022-09-08T04:56:03.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:03.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:03.536468+0000) 2022-09-08T04:56:03.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:03.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:03.536527+0000) 2022-09-08T04:56:04.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:04 smithi059 ceph-mon[97450]: pgmap v14175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:56:04.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:04 smithi055 ceph-mon[97033]: pgmap v14175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:56:04.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:04.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:04.536796+0000) 2022-09-08T04:56:04.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:04.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:04.536726+0000) 2022-09-08T04:56:05.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:05.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:05.536919+0000) 2022-09-08T04:56:05.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:05.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:05.536990+0000) 2022-09-08T04:56:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:06 smithi059 ceph-mon[97450]: pgmap v14176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:56:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:56:06.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:06 smithi055 ceph-mon[97033]: pgmap v14176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:56:06.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:56:06.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:06.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:06.537131+0000) 2022-09-08T04:56:06.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:06.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:06.537202+0000) 2022-09-08T04:56:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:56:07.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:56:07.911 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:07.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:07.537350+0000) 2022-09-08T04:56:07.912 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:07.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:07.537379+0000) 2022-09-08T04:56:08.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:08 smithi059 ceph-mon[97450]: pgmap v14177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:56:08.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:08 smithi055 ceph-mon[97033]: pgmap v14177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:56:08.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:08.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:08.537577+0000) 2022-09-08T04:56:08.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:08.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:08.537682+0000) 2022-09-08T04:56:09.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:09.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:09.537725+0000) 2022-09-08T04:56:09.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:09.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:09.537932+0000) 2022-09-08T04:56:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:10 smithi059 ceph-mon[97450]: pgmap v14178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:56:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:10 smithi059 ceph-mon[97450]: osdmap e967: 12 total, 11 up, 11 in 2022-09-08T04:56:10.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:10 smithi055 ceph-mon[97033]: pgmap v14178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:56:10.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:10 smithi055 ceph-mon[97033]: osdmap e967: 12 total, 11 up, 11 in 2022-09-08T04:56:10.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:10.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:10.537850+0000) 2022-09-08T04:56:10.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:10.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:10.538082+0000) 2022-09-08T04:56:11.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:11.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:11.538097+0000) 2022-09-08T04:56:11.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:11.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:11.538306+0000) 2022-09-08T04:56:12.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:12 smithi059 ceph-mon[97450]: pgmap v14180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:56:12.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:12 smithi055 ceph-mon[97033]: pgmap v14180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:56:12.888 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:12.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:12.538489+0000) 2022-09-08T04:56:12.889 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:12.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:12.538303+0000) 2022-09-08T04:56:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:56:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:56:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:56:13.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:56:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:56:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:56:13.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:13.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:13.538430+0000) 2022-09-08T04:56:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:13.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:13.538630+0000) 2022-09-08T04:56:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:14 smithi059 ceph-mon[97450]: pgmap v14181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:56:14.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:14 smithi055 ceph-mon[97033]: pgmap v14181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:56:14.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:14.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:14.538575+0000) 2022-09-08T04:56:14.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:14.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:14.538788+0000) 2022-09-08T04:56:15.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:15.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:15.538809+0000) 2022-09-08T04:56:15.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:15.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:15.538954+0000) 2022-09-08T04:56:16.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:16 smithi059 ceph-mon[97450]: pgmap v14182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:56:16.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:16 smithi055 ceph-mon[97033]: pgmap v14182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:56:16.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:16.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:16.538996+0000) 2022-09-08T04:56:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:16.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:16.539107+0000) 2022-09-08T04:56:16.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:56:16.716+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:56:17.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:17.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:17.539323+0000) 2022-09-08T04:56:17.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:17.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:17.539227+0000) 2022-09-08T04:56:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:18 smithi059 ceph-mon[97450]: pgmap v14183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:56:18.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:18 smithi055 ceph-mon[97033]: pgmap v14183: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:56:18.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:18.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:18.539488+0000) 2022-09-08T04:56:18.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:18.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:18.539589+0000) 2022-09-08T04:56:19.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:19.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:19.539798+0000) 2022-09-08T04:56:19.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:19.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:19.539798+0000) 2022-09-08T04:56:20.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:20 smithi059 ceph-mon[97450]: pgmap v14184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:56:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:20 smithi059 ceph-mon[97450]: osdmap e968: 12 total, 11 up, 11 in 2022-09-08T04:56:20.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:20 smithi055 ceph-mon[97033]: pgmap v14184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:56:20.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:20 smithi055 ceph-mon[97033]: osdmap e968: 12 total, 11 up, 11 in 2022-09-08T04:56:20.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:20.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:20.540008+0000) 2022-09-08T04:56:20.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:20.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:20.540019+0000) 2022-09-08T04:56:21.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:21.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:21.540228+0000) 2022-09-08T04:56:21.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:21.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:21.540235+0000) 2022-09-08T04:56:22.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:22 smithi059 ceph-mon[97450]: pgmap v14186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:56:22.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:22 smithi055 ceph-mon[97033]: pgmap v14186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:56:22.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:22.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:22.540498+0000) 2022-09-08T04:56:22.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:22.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:22.540498+0000) 2022-09-08T04:56:23.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:23.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:23.540768+0000) 2022-09-08T04:56:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:23.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:23.540768+0000) 2022-09-08T04:56:24.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:24 smithi055 ceph-mon[97033]: pgmap v14187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:56:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:24 smithi059 ceph-mon[97450]: pgmap v14187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:56:24.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:24.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:24.541044+0000) 2022-09-08T04:56:24.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:24.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:24.541035+0000) 2022-09-08T04:56:25.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:25.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:25.541338+0000) 2022-09-08T04:56:25.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:25.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:25.541368+0000) 2022-09-08T04:56:26.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:26 smithi055 ceph-mon[97033]: pgmap v14188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 102 B/s wr, 7 op/s 2022-09-08T04:56:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:26 smithi059 ceph-mon[97450]: pgmap v14188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 102 B/s wr, 7 op/s 2022-09-08T04:56:26.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:26.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:26.541665+0000) 2022-09-08T04:56:26.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:26.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:26.541668+0000) 2022-09-08T04:56:27.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:27.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:27.541839+0000) 2022-09-08T04:56:27.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:27.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:27.541839+0000) 2022-09-08T04:56:28.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:28 smithi055 ceph-mon[97033]: pgmap v14189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:56:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:28 smithi059 ceph-mon[97450]: pgmap v14189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:56:28.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:28.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:28.542087+0000) 2022-09-08T04:56:28.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:28.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:28.542085+0000) 2022-09-08T04:56:29.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:29.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:29.542306+0000) 2022-09-08T04:56:29.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:29.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:29.542333+0000) 2022-09-08T04:56:30.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:30 smithi055 ceph-mon[97033]: pgmap v14190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:56:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:30 smithi059 ceph-mon[97450]: pgmap v14190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:56:30.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:30.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:30.542536+0000) 2022-09-08T04:56:30.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:30.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:30.542548+0000) 2022-09-08T04:56:31.899 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:56:31.727+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:56:31.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:31.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:31.542805+0000) 2022-09-08T04:56:31.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:31.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:31.542806+0000) 2022-09-08T04:56:32.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:32 smithi055 ceph-mon[97033]: pgmap v14191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T04:56:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:32 smithi059 ceph-mon[97450]: pgmap v14191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T04:56:32.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:32.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:32.543088+0000) 2022-09-08T04:56:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:32.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:32.543074+0000) 2022-09-08T04:56:33.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:33 smithi055 ceph-mon[97033]: pgmap v14192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:56:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:33 smithi059 ceph-mon[97450]: pgmap v14192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:56:33.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:33.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:33.543391+0000) 2022-09-08T04:56:33.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:33.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:33.543370+0000) 2022-09-08T04:56:34.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:34.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:34.543614+0000) 2022-09-08T04:56:34.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:34.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:34.543666+0000) 2022-09-08T04:56:35.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:35.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:35.543768+0000) 2022-09-08T04:56:35.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:35.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:35.543863+0000) 2022-09-08T04:56:36.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:36 smithi055 ceph-mon[97033]: pgmap v14193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T04:56:36.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:36 smithi059 ceph-mon[97450]: pgmap v14193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T04:56:36.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:36.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:36.544114+0000) 2022-09-08T04:56:36.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:36.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:36.544000+0000) 2022-09-08T04:56:37.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:37.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:37.544299+0000) 2022-09-08T04:56:37.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:37.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:37.544238+0000) 2022-09-08T04:56:38.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:38 smithi055 ceph-mon[97033]: pgmap v14194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T04:56:38.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:38 smithi059 ceph-mon[97450]: pgmap v14194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T04:56:38.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:38.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:38.544529+0000) 2022-09-08T04:56:38.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:38.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:38.544484+0000) 2022-09-08T04:56:39.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:39.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:39.544724+0000) 2022-09-08T04:56:39.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:39.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:39.544688+0000) 2022-09-08T04:56:40.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:40 smithi055 ceph-mon[97033]: pgmap v14195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:56:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:40 smithi059 ceph-mon[97450]: pgmap v14195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:56:40.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:40.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:40.544867+0000) 2022-09-08T04:56:40.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:40.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:40.544814+0000) 2022-09-08T04:56:41.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:41.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:41.545074+0000) 2022-09-08T04:56:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:41.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:41.545048+0000) 2022-09-08T04:56:42.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:42 smithi055 ceph-mon[97033]: pgmap v14196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:56:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:42 smithi059 ceph-mon[97450]: pgmap v14196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:56:42.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:42.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:42.545329+0000) 2022-09-08T04:56:42.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:42.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:42.545263+0000) 2022-09-08T04:56:43.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:43.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:43.545548+0000) 2022-09-08T04:56:43.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:43.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:43.545513+0000) 2022-09-08T04:56:44.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:44 smithi055 ceph-mon[97033]: pgmap v14197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:56:44.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:44 smithi059 ceph-mon[97450]: pgmap v14197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:56:44.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:44.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:44.545717+0000) 2022-09-08T04:56:44.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:44.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:44.545684+0000) 2022-09-08T04:56:45.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:45.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:45.545917+0000) 2022-09-08T04:56:45.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:45.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:45.545891+0000) 2022-09-08T04:56:46.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:46 smithi055 ceph-mon[97033]: pgmap v14198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T04:56:46.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:46 smithi059 ceph-mon[97450]: pgmap v14198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T04:56:46.899 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:56:46.740+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:56:46.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:46.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:46.546183+0000) 2022-09-08T04:56:46.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:46.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:46.546185+0000) 2022-09-08T04:56:47.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:47.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:47.546478+0000) 2022-09-08T04:56:47.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:47.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:47.546482+0000) 2022-09-08T04:56:48.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:48 smithi055 ceph-mon[97033]: pgmap v14199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T04:56:48.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:56:48.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:56:48.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:56:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:48 smithi059 ceph-mon[97450]: pgmap v14199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T04:56:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:56:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:56:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:56:48.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:48.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:48.546776+0000) 2022-09-08T04:56:48.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:48.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:48.546760+0000) 2022-09-08T04:56:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:49 smithi055 ceph-mon[97033]: pgmap v14200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T04:56:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:49 smithi055 ceph-mon[97033]: pgmap v14201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T04:56:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:49 smithi059 ceph-mon[97450]: pgmap v14200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T04:56:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:49 smithi059 ceph-mon[97450]: pgmap v14201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T04:56:49.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:49.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:49.547073+0000) 2022-09-08T04:56:49.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:49.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:49.547122+0000) 2022-09-08T04:56:50.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:50.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:50.547370+0000) 2022-09-08T04:56:50.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:50.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:50.547358+0000) 2022-09-08T04:56:51.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:51 smithi055 ceph-mon[97033]: pgmap v14202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T04:56:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:51 smithi059 ceph-mon[97450]: pgmap v14202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T04:56:51.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:51.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:51.547666+0000) 2022-09-08T04:56:51.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:51.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:51.547660+0000) 2022-09-08T04:56:52.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:52.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:52.547916+0000) 2022-09-08T04:56:52.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:52.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:52.547923+0000) 2022-09-08T04:56:53.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:53 smithi055 ceph-mon[97033]: pgmap v14203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T04:56:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:53 smithi059 ceph-mon[97450]: pgmap v14203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T04:56:53.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:53.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:53.548169+0000) 2022-09-08T04:56:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:53.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:53.548172+0000) 2022-09-08T04:56:54.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:54.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:54.548398+0000) 2022-09-08T04:56:54.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:54.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:54.548401+0000) 2022-09-08T04:56:55.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:55 smithi055 ceph-mon[97033]: pgmap v14204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T04:56:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:55 smithi059 ceph-mon[97450]: pgmap v14204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T04:56:55.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:55.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:55.548653+0000) 2022-09-08T04:56:55.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:55.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:55.548658+0000) 2022-09-08T04:56:56.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:56.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:56.548913+0000) 2022-09-08T04:56:56.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:56.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:56.548918+0000) 2022-09-08T04:56:57.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:57 smithi055 ceph-mon[97033]: pgmap v14205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T04:56:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:57 smithi059 ceph-mon[97450]: pgmap v14205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T04:56:57.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:57.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:57.549060+0000) 2022-09-08T04:56:57.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:57.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:57.549042+0000) 2022-09-08T04:56:58.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:58.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:58.549287+0000) 2022-09-08T04:56:58.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:58.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:58.549279+0000) 2022-09-08T04:56:59.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:56:59 smithi055 ceph-mon[97033]: pgmap v14206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:56:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:56:59 smithi059 ceph-mon[97450]: pgmap v14206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:56:59.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:56:59.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:59.549460+0000) 2022-09-08T04:56:59.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:56:59.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:56:59.549472+0000) 2022-09-08T04:57:00.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:00.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:00.549699+0000) 2022-09-08T04:57:00.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:00.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:00.549693+0000) 2022-09-08T04:57:01.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:01 smithi055 ceph-mon[97033]: pgmap v14207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:01 smithi059 ceph-mon[97450]: pgmap v14207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:01.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:57:01.750+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:57:01.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:01.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:01.549878+0000) 2022-09-08T04:57:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:01.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:01.549864+0000) 2022-09-08T04:57:02.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:02.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:02.550108+0000) 2022-09-08T04:57:02.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:02.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:02.550103+0000) 2022-09-08T04:57:03.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:03 smithi055 ceph-mon[97033]: pgmap v14208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:57:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:03 smithi059 ceph-mon[97450]: pgmap v14208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:57:03.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:03.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:03.550338+0000) 2022-09-08T04:57:03.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:03.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:03.550336+0000) 2022-09-08T04:57:04.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:04.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:04.550592+0000) 2022-09-08T04:57:04.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:04.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:04.550603+0000) 2022-09-08T04:57:05.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:05 smithi055 ceph-mon[97033]: pgmap v14209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:57:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:05 smithi059 ceph-mon[97450]: pgmap v14209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T04:57:05.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:05.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:05.550892+0000) 2022-09-08T04:57:05.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:05.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:05.550897+0000) 2022-09-08T04:57:06.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:57:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:57:06.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:06.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:06.551111+0000) 2022-09-08T04:57:06.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:06.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:06.551121+0000) 2022-09-08T04:57:07.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:07 smithi055 ceph-mon[97033]: pgmap v14210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:57:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:57:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:07 smithi059 ceph-mon[97450]: pgmap v14210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:57:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:57:07.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:07.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:07.551357+0000) 2022-09-08T04:57:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:07.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:07.551341+0000) 2022-09-08T04:57:08.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:08.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:08.551542+0000) 2022-09-08T04:57:08.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:08.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:08.551544+0000) 2022-09-08T04:57:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:09 smithi059 ceph-mon[97450]: pgmap v14211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:57:09.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:09 smithi055 ceph-mon[97033]: pgmap v14211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:57:09.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:09.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:09.551693+0000) 2022-09-08T04:57:09.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:09.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:09.551684+0000) 2022-09-08T04:57:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:10 smithi059 ceph-mon[97450]: osdmap e969: 12 total, 11 up, 11 in 2022-09-08T04:57:10.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:10 smithi055 ceph-mon[97033]: osdmap e969: 12 total, 11 up, 11 in 2022-09-08T04:57:10.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:10.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:10.551924+0000) 2022-09-08T04:57:10.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:10.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:10.551902+0000) 2022-09-08T04:57:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:11 smithi059 ceph-mon[97450]: pgmap v14213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:57:11.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:11 smithi055 ceph-mon[97033]: pgmap v14213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:57:11.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:11.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:11.552136+0000) 2022-09-08T04:57:11.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:11.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:11.552175+0000) 2022-09-08T04:57:12.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:12.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:12.552432+0000) 2022-09-08T04:57:12.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:12.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:12.552384+0000) 2022-09-08T04:57:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:13 smithi059 ceph-mon[97450]: pgmap v14214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:57:13.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:13 smithi055 ceph-mon[97033]: pgmap v14214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:57:13.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:13.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:13.552631+0000) 2022-09-08T04:57:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:13.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:13.552592+0000) 2022-09-08T04:57:14.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:14.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:14.552771+0000) 2022-09-08T04:57:14.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:14.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:14.552775+0000) 2022-09-08T04:57:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:15 smithi059 ceph-mon[97450]: pgmap v14215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:57:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:15 smithi059 ceph-mon[97450]: osdmap e970: 12 total, 11 up, 11 in 2022-09-08T04:57:15.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:15 smithi055 ceph-mon[97033]: pgmap v14215: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:57:15.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:15 smithi055 ceph-mon[97033]: osdmap e970: 12 total, 11 up, 11 in 2022-09-08T04:57:15.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:15.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:15.553010+0000) 2022-09-08T04:57:15.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:15.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:15.553022+0000) 2022-09-08T04:57:16.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:57:16.754+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:57:16.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:16.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:16.553154+0000) 2022-09-08T04:57:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:16.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:16.553162+0000) 2022-09-08T04:57:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:17 smithi059 ceph-mon[97450]: pgmap v14217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:57:17.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:17 smithi055 ceph-mon[97033]: pgmap v14217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:57:17.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:17.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:17.553309+0000) 2022-09-08T04:57:17.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:17.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:17.553370+0000) 2022-09-08T04:57:18.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:18.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:18.553568+0000) 2022-09-08T04:57:18.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:18.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:18.553598+0000) 2022-09-08T04:57:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:19 smithi059 ceph-mon[97450]: pgmap v14218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 246 B/s wr, 7 op/s 2022-09-08T04:57:19.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:19 smithi055 ceph-mon[97033]: pgmap v14218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 246 B/s wr, 7 op/s 2022-09-08T04:57:19.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:19.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:19.553817+0000) 2022-09-08T04:57:19.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:19.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:19.553786+0000) 2022-09-08T04:57:20.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:20.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:20.553927+0000) 2022-09-08T04:57:20.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:20.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:20.553988+0000) 2022-09-08T04:57:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:21 smithi059 ceph-mon[97450]: pgmap v14219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:57:21.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:21 smithi055 ceph-mon[97033]: pgmap v14219: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:57:21.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:21.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:21.554053+0000) 2022-09-08T04:57:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:21.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:21.554122+0000) 2022-09-08T04:57:22.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:22.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:22.554231+0000) 2022-09-08T04:57:22.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:22.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:22.554270+0000) 2022-09-08T04:57:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:23 smithi059 ceph-mon[97450]: pgmap v14220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:57:23.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:23 smithi055 ceph-mon[97033]: pgmap v14220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:57:23.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:23.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:23.554501+0000) 2022-09-08T04:57:23.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:23.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:23.554523+0000) 2022-09-08T04:57:24.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:24.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:24.554757+0000) 2022-09-08T04:57:24.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:24.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:24.554728+0000) 2022-09-08T04:57:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:25 smithi059 ceph-mon[97450]: pgmap v14221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:57:25.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:25 smithi055 ceph-mon[97033]: pgmap v14221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:57:25.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:25.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:25.554959+0000) 2022-09-08T04:57:25.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:25.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:25.554995+0000) 2022-09-08T04:57:26.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:26.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:26.555175+0000) 2022-09-08T04:57:26.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:26.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:26.555211+0000) 2022-09-08T04:57:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:27 smithi059 ceph-mon[97450]: pgmap v14222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:57:27.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:27 smithi055 ceph-mon[97033]: pgmap v14222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:57:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:27.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:27.555414+0000) 2022-09-08T04:57:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:27.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:27.555422+0000) 2022-09-08T04:57:28.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:28.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:28.555574+0000) 2022-09-08T04:57:28.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:28.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:28.555587+0000) 2022-09-08T04:57:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:29 smithi059 ceph-mon[97450]: pgmap v14223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:29.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:29 smithi055 ceph-mon[97033]: pgmap v14223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:29.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:29.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:29.555806+0000) 2022-09-08T04:57:29.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:29.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:29.555751+0000) 2022-09-08T04:57:30.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:30.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:30.555974+0000) 2022-09-08T04:57:30.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:30.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:30.555942+0000) 2022-09-08T04:57:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:31 smithi059 ceph-mon[97450]: pgmap v14224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:31.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:31 smithi055 ceph-mon[97033]: pgmap v14224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:31.899 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:57:31.770+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:57:31.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:31.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:31.556174+0000) 2022-09-08T04:57:31.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:31.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:31.556171+0000) 2022-09-08T04:57:32.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:32.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:32.556438+0000) 2022-09-08T04:57:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:32.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:32.556458+0000) 2022-09-08T04:57:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:33 smithi059 ceph-mon[97450]: pgmap v14225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:57:33.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:33 smithi055 ceph-mon[97033]: pgmap v14225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:57:33.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:33.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:33.556712+0000) 2022-09-08T04:57:33.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:33.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:33.556722+0000) 2022-09-08T04:57:34.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:34.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:34.556941+0000) 2022-09-08T04:57:34.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:34.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:34.556931+0000) 2022-09-08T04:57:35.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:35 smithi055 ceph-mon[97033]: pgmap v14226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:35 smithi059 ceph-mon[97450]: pgmap v14226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:35.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:35.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:35.557203+0000) 2022-09-08T04:57:35.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:35.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:35.557199+0000) 2022-09-08T04:57:36.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:36.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:36.557413+0000) 2022-09-08T04:57:36.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:36.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:36.557406+0000) 2022-09-08T04:57:37.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:37 smithi055 ceph-mon[97033]: pgmap v14227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:57:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:37 smithi059 ceph-mon[97450]: pgmap v14227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:57:37.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:37.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:37.557613+0000) 2022-09-08T04:57:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:37.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:37.557620+0000) 2022-09-08T04:57:38.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:38.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:38.557860+0000) 2022-09-08T04:57:38.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:38.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:38.557865+0000) 2022-09-08T04:57:39.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:39 smithi055 ceph-mon[97033]: pgmap v14228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:39 smithi059 ceph-mon[97450]: pgmap v14228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:39.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:39.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:39.558033+0000) 2022-09-08T04:57:39.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:39.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:39.558060+0000) 2022-09-08T04:57:40.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:40.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:40.558256+0000) 2022-09-08T04:57:40.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:40.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:40.558233+0000) 2022-09-08T04:57:41.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:41 smithi055 ceph-mon[97033]: pgmap v14229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:41 smithi059 ceph-mon[97450]: pgmap v14229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:41.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:41.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:41.558444+0000) 2022-09-08T04:57:41.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:41.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:41.558461+0000) 2022-09-08T04:57:42.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:42.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:42.558731+0000) 2022-09-08T04:57:42.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:42.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:42.558711+0000) 2022-09-08T04:57:43.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:43 smithi055 ceph-mon[97033]: pgmap v14230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:57:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:43 smithi059 ceph-mon[97450]: pgmap v14230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:57:43.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:43.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:43.558940+0000) 2022-09-08T04:57:43.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:43.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:43.558932+0000) 2022-09-08T04:57:44.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:44.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:44.559096+0000) 2022-09-08T04:57:44.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:44.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:44.559093+0000) 2022-09-08T04:57:45.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:45 smithi055 ceph-mon[97033]: pgmap v14231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:45 smithi059 ceph-mon[97450]: pgmap v14231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:45.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:45.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:45.559321+0000) 2022-09-08T04:57:45.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:45.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:45.559298+0000) 2022-09-08T04:57:46.899 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:57:46.785+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:57:46.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:46.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:46.559534+0000) 2022-09-08T04:57:46.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:46.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:46.559528+0000) 2022-09-08T04:57:47.673 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:47 smithi055 ceph-mon[97033]: pgmap v14232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:57:47.674 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:47 smithi059 ceph-mon[97450]: pgmap v14232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:57:47.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:47.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:47.559758+0000) 2022-09-08T04:57:47.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:47.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:47.559764+0000) 2022-09-08T04:57:48.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:57:48.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:57:48.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:57:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:57:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:57:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:57:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:57:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:57:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:57:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:57:48.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:48.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:48.560030+0000) 2022-09-08T04:57:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:48.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:48.560074+0000) 2022-09-08T04:57:49.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:49 smithi055 ceph-mon[97033]: pgmap v14233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:49 smithi059 ceph-mon[97450]: pgmap v14233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:49.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:49.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:49.560398+0000) 2022-09-08T04:57:49.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:49.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:49.560314+0000) 2022-09-08T04:57:50.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:50.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:50.560565+0000) 2022-09-08T04:57:50.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:50.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:50.560678+0000) 2022-09-08T04:57:51.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:51 smithi055 ceph-mon[97033]: pgmap v14234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:57:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:57:51.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:57:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:57:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:51.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:51 smithi059 ceph-mon[97450]: pgmap v14234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:57:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:57:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:57:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:57:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:57:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:51.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:51.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:51.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:51.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:51.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:51.560898+0000) 2022-09-08T04:57:51.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:51.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:51.560787+0000) 2022-09-08T04:57:52.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:52 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T04:57:52.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:52 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T04:57:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:57:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:52 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T04:57:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:52 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T04:57:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:52.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T04:57:52.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:57:52.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:52.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:52.560957+0000) 2022-09-08T04:57:52.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:52.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:52.561109+0000) 2022-09-08T04:57:53.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:53 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T04:57:53.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:53 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T04:57:53.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:53 smithi055 ceph-mon[97033]: pgmap v14235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T04:57:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:53 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T04:57:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:53 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T04:57:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:53 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T04:57:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:53 smithi059 ceph-mon[97450]: pgmap v14235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T04:57:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:53 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T04:57:53.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:53.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:53.561198+0000) 2022-09-08T04:57:53.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:53.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:53.561364+0000) 2022-09-08T04:57:54.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:54 smithi055 ceph-mon[97033]: pgmap v14236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:57:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:54 smithi059 ceph-mon[97450]: pgmap v14236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:57:54.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:54.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:54.561623+0000) 2022-09-08T04:57:54.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:54.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:54.561442+0000) 2022-09-08T04:57:55.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:55.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:55.561668+0000) 2022-09-08T04:57:55.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:55.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:55.561838+0000) 2022-09-08T04:57:56.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:56.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:56.562061+0000) 2022-09-08T04:57:56.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:56.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:56.561880+0000) 2022-09-08T04:57:57.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:57 smithi055 ceph-mon[97033]: pgmap v14237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T04:57:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:57 smithi059 ceph-mon[97450]: pgmap v14237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T04:57:57.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:57.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:57.562058+0000) 2022-09-08T04:57:57.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:57.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:57.562229+0000) 2022-09-08T04:57:58.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:58.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:58.562288+0000) 2022-09-08T04:57:58.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:58.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:58.562472+0000) 2022-09-08T04:57:59.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:57:59 smithi055 ceph-mon[97033]: pgmap v14238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:57:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:57:59 smithi059 ceph-mon[97450]: pgmap v14238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:57:59.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:57:59.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:59.562538+0000) 2022-09-08T04:57:59.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:57:59.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:57:59.562690+0000) 2022-09-08T04:58:00.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:00.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:00.562784+0000) 2022-09-08T04:58:00.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:00.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:00.562837+0000) 2022-09-08T04:58:01.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:01 smithi055 ceph-mon[97033]: pgmap v14239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:58:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:01 smithi059 ceph-mon[97450]: pgmap v14239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T04:58:01.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:01.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:01.562936+0000) 2022-09-08T04:58:01.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:01.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:01.563005+0000) 2022-09-08T04:58:01.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:58:01.792+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:58:02.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:02.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:02.563217+0000) 2022-09-08T04:58:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:02.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:02.563140+0000) 2022-09-08T04:58:03.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:03 smithi055 ceph-mon[97033]: pgmap v14240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T04:58:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:03 smithi059 ceph-mon[97450]: pgmap v14240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T04:58:03.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:03.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:03.563410+0000) 2022-09-08T04:58:03.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:03.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:03.563464+0000) 2022-09-08T04:58:04.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:04.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:04.563597+0000) 2022-09-08T04:58:04.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:04.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:04.563664+0000) 2022-09-08T04:58:05.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:05 smithi055 ceph-mon[97033]: pgmap v14241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:05 smithi059 ceph-mon[97450]: pgmap v14241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:05.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:05.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:05.563729+0000) 2022-09-08T04:58:05.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:05.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:05.563826+0000) 2022-09-08T04:58:06.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:58:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:58:06.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:06.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:06.563896+0000) 2022-09-08T04:58:06.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:06.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:06.563975+0000) 2022-09-08T04:58:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:07 smithi055 ceph-mon[97033]: pgmap v14242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:58:07.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:58:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:07 smithi059 ceph-mon[97450]: pgmap v14242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:58:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:58:07.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:07.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:07.564090+0000) 2022-09-08T04:58:07.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:07.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:07.564137+0000) 2022-09-08T04:58:08.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:08.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:08.564252+0000) 2022-09-08T04:58:08.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:08.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:08.564290+0000) 2022-09-08T04:58:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:09 smithi055 ceph-mon[97033]: pgmap v14243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:58:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:09 smithi059 ceph-mon[97450]: pgmap v14243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:58:09.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:09.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:09.564471+0000) 2022-09-08T04:58:09.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:09.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:09.564476+0000) 2022-09-08T04:58:10.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:10 smithi055 ceph-mon[97033]: osdmap e971: 12 total, 11 up, 11 in 2022-09-08T04:58:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:10 smithi059 ceph-mon[97450]: osdmap e971: 12 total, 11 up, 11 in 2022-09-08T04:58:10.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:10.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:10.564739+0000) 2022-09-08T04:58:10.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:10.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:10.564734+0000) 2022-09-08T04:58:11.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:11 smithi055 ceph-mon[97033]: pgmap v14245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:58:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:11 smithi059 ceph-mon[97450]: pgmap v14245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:58:11.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:11.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:11.564875+0000) 2022-09-08T04:58:11.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:11.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:11.564950+0000) 2022-09-08T04:58:12.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:12.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:12.565008+0000) 2022-09-08T04:58:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:12.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:12.565120+0000) 2022-09-08T04:58:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:13 smithi059 ceph-mon[97450]: pgmap v14246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:58:13.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:13 smithi055 ceph-mon[97033]: pgmap v14246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:58:13.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:13.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:13.565216+0000) 2022-09-08T04:58:13.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:13.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:13.565303+0000) 2022-09-08T04:58:14.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:14.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:14.565420+0000) 2022-09-08T04:58:14.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:14.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:14.565514+0000) 2022-09-08T04:58:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:15 smithi059 ceph-mon[97450]: pgmap v14247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:58:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:15 smithi059 ceph-mon[97450]: osdmap e972: 12 total, 11 up, 11 in 2022-09-08T04:58:15.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:15 smithi055 ceph-mon[97033]: pgmap v14247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:58:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:15 smithi055 ceph-mon[97033]: osdmap e972: 12 total, 11 up, 11 in 2022-09-08T04:58:15.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:15.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:15.565685+0000) 2022-09-08T04:58:15.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:15.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:15.565761+0000) 2022-09-08T04:58:16.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:16.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:16.565899+0000) 2022-09-08T04:58:16.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:16.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:16.565982+0000) 2022-09-08T04:58:16.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:58:16.804+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:58:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:17 smithi059 ceph-mon[97450]: pgmap v14249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:58:17.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:17 smithi055 ceph-mon[97033]: pgmap v14249: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T04:58:17.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:17.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:17.566137+0000) 2022-09-08T04:58:17.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:17.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:17.566204+0000) 2022-09-08T04:58:18.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:18.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:18.566333+0000) 2022-09-08T04:58:18.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:18.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:18.566441+0000) 2022-09-08T04:58:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:19 smithi059 ceph-mon[97450]: pgmap v14250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 251 B/s wr, 7 op/s 2022-09-08T04:58:19.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:19 smithi055 ceph-mon[97033]: pgmap v14250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 251 B/s wr, 7 op/s 2022-09-08T04:58:19.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:19.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:19.566565+0000) 2022-09-08T04:58:19.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:19.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:19.566684+0000) 2022-09-08T04:58:20.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:20.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:20.566812+0000) 2022-09-08T04:58:20.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:20.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:20.566903+0000) 2022-09-08T04:58:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:21 smithi059 ceph-mon[97450]: pgmap v14251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:58:21.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:21 smithi055 ceph-mon[97033]: pgmap v14251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:58:21.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:21.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:21.566958+0000) 2022-09-08T04:58:21.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:21.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:21.567080+0000) 2022-09-08T04:58:22.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:22.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:22.567278+0000) 2022-09-08T04:58:22.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:22.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:22.567208+0000) 2022-09-08T04:58:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:23 smithi059 ceph-mon[97450]: pgmap v14252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:58:23.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:23 smithi055 ceph-mon[97033]: pgmap v14252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:58:23.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:23.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:23.567386+0000) 2022-09-08T04:58:23.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:23.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:23.567459+0000) 2022-09-08T04:58:24.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:24.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:24.567600+0000) 2022-09-08T04:58:24.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:24.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:24.567681+0000) 2022-09-08T04:58:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:25 smithi059 ceph-mon[97450]: pgmap v14253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:58:25.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:25 smithi055 ceph-mon[97033]: pgmap v14253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:58:25.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:25.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:25.567748+0000) 2022-09-08T04:58:25.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:25.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:25.567839+0000) 2022-09-08T04:58:26.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:26.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:26.567982+0000) 2022-09-08T04:58:26.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:26.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:26.568092+0000) 2022-09-08T04:58:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:27 smithi059 ceph-mon[97450]: pgmap v14254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:58:27.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:27 smithi055 ceph-mon[97033]: pgmap v14254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T04:58:27.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:27.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:27.568219+0000) 2022-09-08T04:58:27.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:27.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:27.568326+0000) 2022-09-08T04:58:28.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:28.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:28.568425+0000) 2022-09-08T04:58:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:28.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:28.568493+0000) 2022-09-08T04:58:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:29 smithi059 ceph-mon[97450]: pgmap v14255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:29.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:29 smithi055 ceph-mon[97033]: pgmap v14255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:29.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:29.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:29.568651+0000) 2022-09-08T04:58:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:29.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:29.568688+0000) 2022-09-08T04:58:30.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:30.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:30.568887+0000) 2022-09-08T04:58:30.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:30.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:30.568933+0000) 2022-09-08T04:58:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:31 smithi059 ceph-mon[97450]: pgmap v14256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:31.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:31 smithi055 ceph-mon[97033]: pgmap v14256: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:31.819 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:31.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:31.569148+0000) 2022-09-08T04:58:31.820 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:31.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:31.569178+0000) 2022-09-08T04:58:32.149 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:58:31.820+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:58:32.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:32.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:32.569435+0000) 2022-09-08T04:58:32.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:32.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:32.569418+0000) 2022-09-08T04:58:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:33 smithi059 ceph-mon[97450]: pgmap v14257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:58:33.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:33 smithi055 ceph-mon[97033]: pgmap v14257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:58:33.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:33.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:33.569733+0000) 2022-09-08T04:58:33.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:33.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:33.569722+0000) 2022-09-08T04:58:34.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:34.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:34.569942+0000) 2022-09-08T04:58:34.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:34.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:34.569923+0000) 2022-09-08T04:58:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:35 smithi059 ceph-mon[97450]: pgmap v14258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:35.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:35 smithi055 ceph-mon[97033]: pgmap v14258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:35.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:35.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:35.570192+0000) 2022-09-08T04:58:35.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:35.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:35.570168+0000) 2022-09-08T04:58:36.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:36.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:36.570354+0000) 2022-09-08T04:58:36.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:36.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:36.570330+0000) 2022-09-08T04:58:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:37 smithi059 ceph-mon[97450]: pgmap v14259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:58:37.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:37 smithi055 ceph-mon[97033]: pgmap v14259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:58:37.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:37.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:37.570587+0000) 2022-09-08T04:58:37.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:37.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:37.570569+0000) 2022-09-08T04:58:38.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:38.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:38.570750+0000) 2022-09-08T04:58:38.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:38.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:38.570754+0000) 2022-09-08T04:58:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:39 smithi059 ceph-mon[97450]: pgmap v14260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:39.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:39 smithi055 ceph-mon[97033]: pgmap v14260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:39.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:39.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:39.570990+0000) 2022-09-08T04:58:39.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:39.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:39.570984+0000) 2022-09-08T04:58:40.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:40.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:40.571280+0000) 2022-09-08T04:58:40.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:40.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:40.571239+0000) 2022-09-08T04:58:41.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:41 smithi055 ceph-mon[97033]: pgmap v14261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:41 smithi059 ceph-mon[97450]: pgmap v14261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:41.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:41.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:41.571546+0000) 2022-09-08T04:58:41.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:41.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:41.571525+0000) 2022-09-08T04:58:42.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:42.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:42.571775+0000) 2022-09-08T04:58:42.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:42.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:42.571782+0000) 2022-09-08T04:58:43.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:43 smithi055 ceph-mon[97033]: pgmap v14262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:58:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:43 smithi059 ceph-mon[97450]: pgmap v14262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:58:43.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:43.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:43.572037+0000) 2022-09-08T04:58:43.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:43.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:43.572053+0000) 2022-09-08T04:58:44.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:44.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:44.572280+0000) 2022-09-08T04:58:44.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:44.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:44.572323+0000) 2022-09-08T04:58:45.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:45 smithi055 ceph-mon[97033]: pgmap v14263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:45 smithi059 ceph-mon[97450]: pgmap v14263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:45.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:45.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:45.572514+0000) 2022-09-08T04:58:45.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:45.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:45.572519+0000) 2022-09-08T04:58:46.826 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:46.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:46.572733+0000) 2022-09-08T04:58:46.827 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:46.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:46.572746+0000) 2022-09-08T04:58:47.149 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:58:46.828+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:58:47.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:47 smithi055 ceph-mon[97033]: pgmap v14264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:58:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:47 smithi059 ceph-mon[97450]: pgmap v14264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:58:47.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:47.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:47.572881+0000) 2022-09-08T04:58:47.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:47.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:47.572918+0000) 2022-09-08T04:58:48.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:48.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:48.573096+0000) 2022-09-08T04:58:48.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:48.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:48.573101+0000) 2022-09-08T04:58:49.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:49 smithi055 ceph-mon[97033]: pgmap v14265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:49 smithi059 ceph-mon[97450]: pgmap v14265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:49.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:49.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:49.573344+0000) 2022-09-08T04:58:49.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:49.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:49.573344+0000) 2022-09-08T04:58:50.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:50.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:50.573595+0000) 2022-09-08T04:58:50.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:50.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:50.573631+0000) 2022-09-08T04:58:51.561 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:51 smithi059 ceph-mon[97450]: pgmap v14266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:51.562 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:51 smithi055 ceph-mon[97033]: pgmap v14266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:51.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:51.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:51.573755+0000) 2022-09-08T04:58:51.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:51.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:51.573843+0000) 2022-09-08T04:58:52.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:58:52.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:58:52.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:58:52.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:58:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:58:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:58:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:58:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:58:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:58:52.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:58:52.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:52.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:52.573968+0000) 2022-09-08T04:58:52.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:52.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:52.574089+0000) 2022-09-08T04:58:53.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:53 smithi055 ceph-mon[97033]: pgmap v14267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:58:53.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:58:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:53 smithi059 ceph-mon[97450]: pgmap v14267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:58:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:58:53.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:53.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:53.574212+0000) 2022-09-08T04:58:53.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:53.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:53.574296+0000) 2022-09-08T04:58:54.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:54.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:54.574478+0000) 2022-09-08T04:58:54.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:54.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:54.574561+0000) 2022-09-08T04:58:55.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:55 smithi055 ceph-mon[97033]: pgmap v14268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:55 smithi059 ceph-mon[97450]: pgmap v14268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:55.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:55.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:55.574735+0000) 2022-09-08T04:58:55.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:55.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:55.574780+0000) 2022-09-08T04:58:56.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:56.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:56.574915+0000) 2022-09-08T04:58:56.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:56.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:56.574938+0000) 2022-09-08T04:58:57.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:57 smithi055 ceph-mon[97033]: pgmap v14269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:58:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:57 smithi059 ceph-mon[97450]: pgmap v14269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:58:57.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:57.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:57.575064+0000) 2022-09-08T04:58:57.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:57.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:57.575099+0000) 2022-09-08T04:58:58.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:58.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:58.575332+0000) 2022-09-08T04:58:58.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:58.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:58.575284+0000) 2022-09-08T04:58:59.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:58:59 smithi055 ceph-mon[97033]: pgmap v14270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:58:59 smithi059 ceph-mon[97450]: pgmap v14270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:58:59.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:58:59.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:59.575545+0000) 2022-09-08T04:58:59.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:58:59.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:58:59.575571+0000) 2022-09-08T04:59:00.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:00.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:00.575795+0000) 2022-09-08T04:59:00.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:00.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:00.575807+0000) 2022-09-08T04:59:01.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:01 smithi055 ceph-mon[97033]: pgmap v14271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:01 smithi059 ceph-mon[97450]: pgmap v14271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:01.828 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:01.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:01.576080+0000) 2022-09-08T04:59:01.829 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:01.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:01.576092+0000) 2022-09-08T04:59:02.149 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:59:01.829+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:59:02.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:02.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:02.576355+0000) 2022-09-08T04:59:02.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:02.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:02.576382+0000) 2022-09-08T04:59:03.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:03 smithi055 ceph-mon[97033]: pgmap v14272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:59:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:03 smithi059 ceph-mon[97450]: pgmap v14272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:59:03.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:03.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:03.576630+0000) 2022-09-08T04:59:03.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:03.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:03.576690+0000) 2022-09-08T04:59:04.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:04.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:04.576940+0000) 2022-09-08T04:59:04.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:04.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:04.576976+0000) 2022-09-08T04:59:05.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:05 smithi055 ceph-mon[97033]: pgmap v14273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:05 smithi059 ceph-mon[97450]: pgmap v14273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:05.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:05.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:05.577232+0000) 2022-09-08T04:59:05.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:05.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:05.577254+0000) 2022-09-08T04:59:06.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:59:06.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:59:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T04:59:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T04:59:06.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:06.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:06.577514+0000) 2022-09-08T04:59:06.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:06.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:06.577527+0000) 2022-09-08T04:59:07.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:07 smithi055 ceph-mon[97033]: pgmap v14274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:59:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:07 smithi059 ceph-mon[97450]: pgmap v14274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T04:59:07.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:07.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:07.577819+0000) 2022-09-08T04:59:07.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:07.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:07.577811+0000) 2022-09-08T04:59:08.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:08 smithi055 ceph-mon[97033]: pgmap v14275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:59:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:08 smithi059 ceph-mon[97450]: pgmap v14275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T04:59:08.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:08.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:08.577998+0000) 2022-09-08T04:59:08.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:08.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:08.577984+0000) 2022-09-08T04:59:09.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:09.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:09.578118+0000) 2022-09-08T04:59:09.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:09.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:09.578161+0000) 2022-09-08T04:59:10.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:10 smithi055 ceph-mon[97033]: osdmap e973: 12 total, 11 up, 11 in 2022-09-08T04:59:10.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:10 smithi055 ceph-mon[97033]: pgmap v14277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:59:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:10 smithi059 ceph-mon[97450]: osdmap e973: 12 total, 11 up, 11 in 2022-09-08T04:59:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:10 smithi059 ceph-mon[97450]: pgmap v14277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:59:10.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:10.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:10.578247+0000) 2022-09-08T04:59:10.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:10.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:10.578339+0000) 2022-09-08T04:59:11.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:11.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:11.578485+0000) 2022-09-08T04:59:11.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:11.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:11.578581+0000) 2022-09-08T04:59:12.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:12 smithi055 ceph-mon[97033]: pgmap v14278: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:59:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:12 smithi059 ceph-mon[97450]: pgmap v14278: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:59:12.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:12.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:12.578744+0000) 2022-09-08T04:59:12.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:12.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:12.578857+0000) 2022-09-08T04:59:13.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:13.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:13.579030+0000) 2022-09-08T04:59:13.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:13.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:13.579083+0000) 2022-09-08T04:59:14.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:14 smithi055 ceph-mon[97033]: pgmap v14279: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:59:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:14 smithi059 ceph-mon[97450]: pgmap v14279: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T04:59:14.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:14.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:14.579219+0000) 2022-09-08T04:59:14.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:14.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:14.579286+0000) 2022-09-08T04:59:15.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:15.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:15.579430+0000) 2022-09-08T04:59:15.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:15.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:15.579536+0000) 2022-09-08T04:59:16.844 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:16.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:16.579689+0000) 2022-09-08T04:59:16.845 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:16.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:16.579757+0000) 2022-09-08T04:59:17.149 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:59:16.845+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:59:17.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:17 smithi055 ceph-mon[97033]: pgmap v14280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:59:17.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:17 smithi059 ceph-mon[97450]: pgmap v14280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:59:17.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:17.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:17.579854+0000) 2022-09-08T04:59:17.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:17.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:17.579994+0000) 2022-09-08T04:59:18.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:18.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:18.580078+0000) 2022-09-08T04:59:18.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:18.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:18.580207+0000) 2022-09-08T04:59:19.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:19 smithi055 ceph-mon[97033]: pgmap v14281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:59:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:19 smithi059 ceph-mon[97450]: pgmap v14281: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T04:59:19.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:19.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:19.580379+0000) 2022-09-08T04:59:19.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:19.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:19.580249+0000) 2022-09-08T04:59:20.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:20.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:20.580462+0000) 2022-09-08T04:59:20.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:20.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:20.580574+0000) 2022-09-08T04:59:21.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:21 smithi055 ceph-mon[97033]: pgmap v14282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 203 B/s wr, 6 op/s 2022-09-08T04:59:21.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:21 smithi055 ceph-mon[97033]: osdmap e974: 12 total, 11 up, 11 in 2022-09-08T04:59:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:21 smithi059 ceph-mon[97450]: pgmap v14282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 203 B/s wr, 6 op/s 2022-09-08T04:59:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:21 smithi059 ceph-mon[97450]: osdmap e974: 12 total, 11 up, 11 in 2022-09-08T04:59:21.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:21.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:21.580678+0000) 2022-09-08T04:59:21.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:21.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:21.580740+0000) 2022-09-08T04:59:22.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:22.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:22.580877+0000) 2022-09-08T04:59:22.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:22.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:22.580921+0000) 2022-09-08T04:59:23.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:23 smithi055 ceph-mon[97033]: pgmap v14284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:59:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:23 smithi059 ceph-mon[97450]: pgmap v14284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:59:23.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:23.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:23.581093+0000) 2022-09-08T04:59:23.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:23.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:23.581148+0000) 2022-09-08T04:59:24.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:24.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:24.581334+0000) 2022-09-08T04:59:24.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:24.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:24.581437+0000) 2022-09-08T04:59:25.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:25 smithi055 ceph-mon[97033]: pgmap v14285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:59:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:25 smithi059 ceph-mon[97450]: pgmap v14285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T04:59:25.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:25.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:25.581581+0000) 2022-09-08T04:59:25.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:25.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:25.581702+0000) 2022-09-08T04:59:26.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:26.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:26.581815+0000) 2022-09-08T04:59:26.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:26.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:26.581912+0000) 2022-09-08T04:59:27.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:27 smithi055 ceph-mon[97033]: pgmap v14286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:59:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:27 smithi059 ceph-mon[97450]: pgmap v14286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:59:27.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:27.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:27.582063+0000) 2022-09-08T04:59:27.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:27.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:27.582092+0000) 2022-09-08T04:59:28.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:28.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:28.582330+0000) 2022-09-08T04:59:28.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:28.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:28.582310+0000) 2022-09-08T04:59:29.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:29 smithi055 ceph-mon[97033]: pgmap v14287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:59:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:29 smithi059 ceph-mon[97450]: pgmap v14287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:59:29.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:29.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:29.582577+0000) 2022-09-08T04:59:29.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:29.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:29.582588+0000) 2022-09-08T04:59:30.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:30.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:30.582821+0000) 2022-09-08T04:59:30.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:30.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:30.582893+0000) 2022-09-08T04:59:31.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:31 smithi055 ceph-mon[97033]: pgmap v14288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:59:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:31 smithi059 ceph-mon[97450]: pgmap v14288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T04:59:31.850 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:31.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:31.583055+0000) 2022-09-08T04:59:31.851 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:31.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:31.583149+0000) 2022-09-08T04:59:32.149 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:59:31.852+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:59:32.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:32.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:32.583213+0000) 2022-09-08T04:59:32.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:32.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:32.583312+0000) 2022-09-08T04:59:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:33 smithi059 ceph-mon[97450]: pgmap v14289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:59:33.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:33 smithi055 ceph-mon[97033]: pgmap v14289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T04:59:33.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:33.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:33.583440+0000) 2022-09-08T04:59:33.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:33.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:33.583496+0000) 2022-09-08T04:59:34.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:34.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:34.583613+0000) 2022-09-08T04:59:34.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:34.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:34.583692+0000) 2022-09-08T04:59:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:35 smithi059 ceph-mon[97450]: pgmap v14290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:35.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:35 smithi055 ceph-mon[97033]: pgmap v14290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:35.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:35.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:35.583823+0000) 2022-09-08T04:59:35.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:35.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:35.583897+0000) 2022-09-08T04:59:36.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:36.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:36.584129+0000) 2022-09-08T04:59:36.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:36.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:36.584074+0000) 2022-09-08T04:59:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:37 smithi059 ceph-mon[97450]: pgmap v14291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:59:37.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:37 smithi055 ceph-mon[97033]: pgmap v14291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:59:37.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:37.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:37.584329+0000) 2022-09-08T04:59:37.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:37.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:37.584404+0000) 2022-09-08T04:59:38.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:38.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:38.584565+0000) 2022-09-08T04:59:38.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:38.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:38.584694+0000) 2022-09-08T04:59:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:39 smithi059 ceph-mon[97450]: pgmap v14292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:39.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:39 smithi055 ceph-mon[97033]: pgmap v14292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:39.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:39.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:39.584760+0000) 2022-09-08T04:59:39.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:39.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:39.584909+0000) 2022-09-08T04:59:40.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:40.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:40.584959+0000) 2022-09-08T04:59:40.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:40.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:40.585131+0000) 2022-09-08T04:59:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:41 smithi059 ceph-mon[97450]: pgmap v14293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:41.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:41 smithi055 ceph-mon[97033]: pgmap v14293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:41.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:41.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:41.585131+0000) 2022-09-08T04:59:41.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:41.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:41.585291+0000) 2022-09-08T04:59:42.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:42.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:42.585337+0000) 2022-09-08T04:59:42.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:42.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:42.585513+0000) 2022-09-08T04:59:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:43 smithi059 ceph-mon[97450]: pgmap v14294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:59:43.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:43 smithi055 ceph-mon[97033]: pgmap v14294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:59:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:43.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:43.585605+0000) 2022-09-08T04:59:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:43.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:43.585719+0000) 2022-09-08T04:59:44.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:44.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:44.585847+0000) 2022-09-08T04:59:44.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:44.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:44.585879+0000) 2022-09-08T04:59:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:45 smithi059 ceph-mon[97450]: pgmap v14295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:45.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:45 smithi055 ceph-mon[97033]: pgmap v14295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:45.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:45.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:45.586111+0000) 2022-09-08T04:59:45.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:45.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:45.586091+0000) 2022-09-08T04:59:46.866 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:46.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:46.586286+0000) 2022-09-08T04:59:46.867 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:46.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:46.586269+0000) 2022-09-08T04:59:47.149 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 04:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T04:59:46.866+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T04:59:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:47 smithi059 ceph-mon[97450]: pgmap v14296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:59:47.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:47 smithi055 ceph-mon[97033]: pgmap v14296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:59:47.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:47.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:47.586408+0000) 2022-09-08T04:59:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:47.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:47.586421+0000) 2022-09-08T04:59:48.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:48.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:48.586628+0000) 2022-09-08T04:59:48.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:48.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:48.586618+0000) 2022-09-08T04:59:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:49 smithi059 ceph-mon[97450]: pgmap v14297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:49.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:49 smithi055 ceph-mon[97033]: pgmap v14297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:49.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:49.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:49.586902+0000) 2022-09-08T04:59:49.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:49.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:49.586882+0000) 2022-09-08T04:59:50.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:50.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:50.587207+0000) 2022-09-08T04:59:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:50.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:50.587100+0000) 2022-09-08T04:59:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:51 smithi059 ceph-mon[97450]: pgmap v14298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:51.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:51 smithi055 ceph-mon[97033]: pgmap v14298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:51.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:51.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:51.587381+0000) 2022-09-08T04:59:51.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:51.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:51.587314+0000) 2022-09-08T04:59:52.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:52.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:52.587508+0000) 2022-09-08T04:59:52.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:52.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:52.587574+0000) 2022-09-08T04:59:53.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:53 smithi055 ceph-mon[97033]: pgmap v14299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:59:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:59:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:59:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:59:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:59:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:53 smithi059 ceph-mon[97450]: pgmap v14299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:59:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T04:59:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T04:59:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T04:59:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T04:59:53.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:53.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:53.587744+0000) 2022-09-08T04:59:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:53.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:53.587818+0000) 2022-09-08T04:59:54.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:54.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:54.587987+0000) 2022-09-08T04:59:54.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:54.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:54.588089+0000) 2022-09-08T04:59:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:55 smithi059 ceph-mon[97450]: pgmap v14300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:55.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:55 smithi055 ceph-mon[97033]: pgmap v14300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:55.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:55.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:55.588222+0000) 2022-09-08T04:59:55.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:55.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:55.588279+0000) 2022-09-08T04:59:56.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:56.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:56.588435+0000) 2022-09-08T04:59:56.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:56.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:56.588494+0000) 2022-09-08T04:59:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:57 smithi059 ceph-mon[97450]: pgmap v14301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:59:57.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:57 smithi055 ceph-mon[97033]: pgmap v14301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T04:59:57.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:57.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:57.588702+0000) 2022-09-08T04:59:57.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:57.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:57.588737+0000) 2022-09-08T04:59:58.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:58.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:58.588949+0000) 2022-09-08T04:59:58.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:58.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:58.589013+0000) 2022-09-08T04:59:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 04:59:59 smithi059 ceph-mon[97450]: pgmap v14302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:59.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 04:59:59 smithi055 ceph-mon[97033]: pgmap v14302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T04:59:59.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 04:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T04:59:59.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:59.589196+0000) 2022-09-08T04:59:59.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 04:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T04:59:59.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T03:59:59.589226+0000) 2022-09-08T05:00:00.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T05:00:00.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T05:00:00.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T05:00:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T05:00:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T05:00:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T05:00:00.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:00.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:00.589334+0000) 2022-09-08T05:00:00.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:00.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:00.589364+0000) 2022-09-08T05:00:01.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:01 smithi055 ceph-mon[97033]: pgmap v14303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:01 smithi059 ceph-mon[97450]: pgmap v14303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:01.880 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:01.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:01.589538+0000) 2022-09-08T05:00:01.881 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:01.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:01.589588+0000) 2022-09-08T05:00:02.149 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:00:01.881+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:00:02.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:02.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:02.589775+0000) 2022-09-08T05:00:02.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:02.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:02.589798+0000) 2022-09-08T05:00:03.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:03 smithi055 ceph-mon[97033]: pgmap v14304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:00:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:03 smithi059 ceph-mon[97450]: pgmap v14304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:00:03.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:03.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:03.589988+0000) 2022-09-08T05:00:03.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:03.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:03.589992+0000) 2022-09-08T05:00:04.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:04.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:04.590222+0000) 2022-09-08T05:00:04.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:04.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:04.590199+0000) 2022-09-08T05:00:05.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:05 smithi055 ceph-mon[97033]: pgmap v14305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:05 smithi059 ceph-mon[97450]: pgmap v14305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:05.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:05.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:05.590445+0000) 2022-09-08T05:00:05.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:05.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:05.590426+0000) 2022-09-08T05:00:06.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:00:06.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:00:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:00:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:00:06.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:06.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:06.590665+0000) 2022-09-08T05:00:06.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:06.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:06.590684+0000) 2022-09-08T05:00:07.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:07 smithi055 ceph-mon[97033]: pgmap v14306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:00:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:07 smithi059 ceph-mon[97450]: pgmap v14306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:00:07.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:07.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:07.590877+0000) 2022-09-08T05:00:07.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:07.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:07.590882+0000) 2022-09-08T05:00:08.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:08.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:08.591120+0000) 2022-09-08T05:00:08.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:08.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:08.591121+0000) 2022-09-08T05:00:09.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:09 smithi055 ceph-mon[97033]: pgmap v14307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:00:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:09 smithi059 ceph-mon[97450]: pgmap v14307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:00:09.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:09.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:09.592112+0000) 2022-09-08T05:00:09.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:09.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:09.591822+0000) 2022-09-08T05:00:10.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:10 smithi055 ceph-mon[97033]: osdmap e975: 12 total, 11 up, 11 in 2022-09-08T05:00:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:10 smithi059 ceph-mon[97450]: osdmap e975: 12 total, 11 up, 11 in 2022-09-08T05:00:10.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:10.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:10.592263+0000) 2022-09-08T05:00:10.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:10.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:10.591959+0000) 2022-09-08T05:00:11.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:11 smithi055 ceph-mon[97033]: pgmap v14309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:00:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:11 smithi059 ceph-mon[97450]: pgmap v14309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:00:11.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:11.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:11.592503+0000) 2022-09-08T05:00:11.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:11.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:11.592175+0000) 2022-09-08T05:00:12.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:12.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:12.592723+0000) 2022-09-08T05:00:12.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:12.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:12.592442+0000) 2022-09-08T05:00:13.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:13 smithi055 ceph-mon[97033]: pgmap v14310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:00:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:13 smithi059 ceph-mon[97450]: pgmap v14310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:00:13.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:13.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:13.592972+0000) 2022-09-08T05:00:13.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:13.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:13.592719+0000) 2022-09-08T05:00:14.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:14.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:14.593126+0000) 2022-09-08T05:00:14.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:14.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:14.592947+0000) 2022-09-08T05:00:15.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:15 smithi055 ceph-mon[97033]: pgmap v14311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:00:15.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:15 smithi055 ceph-mon[97033]: osdmap e976: 12 total, 11 up, 11 in 2022-09-08T05:00:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:15 smithi059 ceph-mon[97450]: pgmap v14311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:00:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:15 smithi059 ceph-mon[97450]: osdmap e976: 12 total, 11 up, 11 in 2022-09-08T05:00:15.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:15.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:15.593282+0000) 2022-09-08T05:00:15.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:15.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:15.593124+0000) 2022-09-08T05:00:16.896 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:16.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:16.593475+0000) 2022-09-08T05:00:16.897 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:16.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:16.593342+0000) 2022-09-08T05:00:17.149 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:00:16.897+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:00:17.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:17 smithi055 ceph-mon[97033]: pgmap v14313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:00:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:17 smithi059 ceph-mon[97450]: pgmap v14313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:00:17.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:17.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:17.593605+0000) 2022-09-08T05:00:17.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:17.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:17.593705+0000) 2022-09-08T05:00:18.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:18.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:18.593923+0000) 2022-09-08T05:00:18.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:18.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:18.593890+0000) 2022-09-08T05:00:19.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:19 smithi055 ceph-mon[97033]: pgmap v14314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T05:00:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:19 smithi059 ceph-mon[97450]: pgmap v14314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T05:00:19.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:19.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:19.594152+0000) 2022-09-08T05:00:19.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:19.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:19.594181+0000) 2022-09-08T05:00:20.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:20.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:20.594386+0000) 2022-09-08T05:00:20.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:20.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:20.594433+0000) 2022-09-08T05:00:21.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:21 smithi055 ceph-mon[97033]: pgmap v14315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:00:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:21 smithi059 ceph-mon[97450]: pgmap v14315: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:00:21.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:21.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:21.594628+0000) 2022-09-08T05:00:21.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:21.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:21.594598+0000) 2022-09-08T05:00:22.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:22 smithi055 ceph-mon[97033]: pgmap v14316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:00:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:22 smithi059 ceph-mon[97450]: pgmap v14316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:00:22.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:22.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:22.594790+0000) 2022-09-08T05:00:22.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:22.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:22.594814+0000) 2022-09-08T05:00:23.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:23.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:23.595007+0000) 2022-09-08T05:00:23.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:23.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:23.595008+0000) 2022-09-08T05:00:24.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:24 smithi055 ceph-mon[97033]: pgmap v14317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:00:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:24 smithi059 ceph-mon[97450]: pgmap v14317: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:00:24.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:24.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:24.595222+0000) 2022-09-08T05:00:24.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:24.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:24.595233+0000) 2022-09-08T05:00:25.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:25.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:25.595467+0000) 2022-09-08T05:00:25.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:25.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:25.595483+0000) 2022-09-08T05:00:26.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:26.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:26.595727+0000) 2022-09-08T05:00:26.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:26.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:26.595783+0000) 2022-09-08T05:00:27.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:27 smithi059 ceph-mon[97450]: pgmap v14318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:00:27.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:27 smithi055 ceph-mon[97033]: pgmap v14318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:00:27.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:27.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:27.595998+0000) 2022-09-08T05:00:27.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:27.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:27.596094+0000) 2022-09-08T05:00:28.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:28.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:28.596230+0000) 2022-09-08T05:00:28.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:28.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:28.596364+0000) 2022-09-08T05:00:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:29 smithi059 ceph-mon[97450]: pgmap v14319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:29.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:29 smithi055 ceph-mon[97033]: pgmap v14319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:29.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:29.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:29.596451+0000) 2022-09-08T05:00:29.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:29.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:29.596677+0000) 2022-09-08T05:00:30.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:30.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:30.596691+0000) 2022-09-08T05:00:30.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:30.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:30.596907+0000) 2022-09-08T05:00:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:31 smithi059 ceph-mon[97450]: pgmap v14320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:31.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:31 smithi055 ceph-mon[97033]: pgmap v14320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:31.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:31.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:31.596925+0000) 2022-09-08T05:00:31.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:31.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:31.597143+0000) 2022-09-08T05:00:32.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:00:31.911+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:00:32.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:32.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:32.597113+0000) 2022-09-08T05:00:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:32.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:32.597333+0000) 2022-09-08T05:00:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:33 smithi059 ceph-mon[97450]: pgmap v14321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:00:33.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:33 smithi055 ceph-mon[97033]: pgmap v14321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:00:33.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:33.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:33.597319+0000) 2022-09-08T05:00:33.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:33.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:33.597530+0000) 2022-09-08T05:00:34.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:34.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:34.597596+0000) 2022-09-08T05:00:34.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:34.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:34.597783+0000) 2022-09-08T05:00:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:35 smithi059 ceph-mon[97450]: pgmap v14322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:35.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:35 smithi055 ceph-mon[97033]: pgmap v14322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:35.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:35.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:35.597811+0000) 2022-09-08T05:00:35.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:35.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:35.597953+0000) 2022-09-08T05:00:36.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:36.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:36.598025+0000) 2022-09-08T05:00:36.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:36.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:36.598203+0000) 2022-09-08T05:00:37.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:37 smithi059 ceph-mon[97450]: pgmap v14323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:00:37.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:37 smithi055 ceph-mon[97033]: pgmap v14323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:00:37.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:37.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:37.598163+0000) 2022-09-08T05:00:37.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:37.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:37.598342+0000) 2022-09-08T05:00:38.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:38.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:38.598400+0000) 2022-09-08T05:00:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:38.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:38.598555+0000) 2022-09-08T05:00:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:39 smithi059 ceph-mon[97450]: pgmap v14324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:39.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:39 smithi055 ceph-mon[97033]: pgmap v14324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:39.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:39.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:39.598673+0000) 2022-09-08T05:00:39.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:39.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:39.598868+0000) 2022-09-08T05:00:40.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:40.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:40.598864+0000) 2022-09-08T05:00:40.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:40.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:40.599055+0000) 2022-09-08T05:00:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:41 smithi059 ceph-mon[97450]: pgmap v14325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:41.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:41 smithi055 ceph-mon[97033]: pgmap v14325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:41.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:41.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:41.599007+0000) 2022-09-08T05:00:41.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:41.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:41.599190+0000) 2022-09-08T05:00:42.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:42.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:42.599154+0000) 2022-09-08T05:00:42.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:42.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:42.599321+0000) 2022-09-08T05:00:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:43 smithi059 ceph-mon[97450]: pgmap v14326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:00:43.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:43 smithi055 ceph-mon[97033]: pgmap v14326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:00:43.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:43.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:43.599389+0000) 2022-09-08T05:00:43.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:43.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:43.599497+0000) 2022-09-08T05:00:44.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:44.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:44.599656+0000) 2022-09-08T05:00:44.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:44.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:44.599706+0000) 2022-09-08T05:00:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:45 smithi059 ceph-mon[97450]: pgmap v14327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:45.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:45 smithi055 ceph-mon[97033]: pgmap v14327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:45.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:45.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:45.599811+0000) 2022-09-08T05:00:45.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:45.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:45.599838+0000) 2022-09-08T05:00:46.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:46.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:46.600048+0000) 2022-09-08T05:00:46.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:46.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:46.600069+0000) 2022-09-08T05:00:47.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:00:46.928+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:00:47.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:47 smithi059 ceph-mon[97450]: pgmap v14328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:00:47.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:47 smithi055 ceph-mon[97033]: pgmap v14328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:00:47.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:47.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:47.600260+0000) 2022-09-08T05:00:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:47.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:47.600249+0000) 2022-09-08T05:00:48.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:48.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:48.600488+0000) 2022-09-08T05:00:48.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:48.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:48.600492+0000) 2022-09-08T05:00:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:49 smithi059 ceph-mon[97450]: pgmap v14329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:49.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:49 smithi055 ceph-mon[97033]: pgmap v14329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:49.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:49.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:49.600723+0000) 2022-09-08T05:00:49.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:49.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:49.600727+0000) 2022-09-08T05:00:50.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:50.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:50.600941+0000) 2022-09-08T05:00:50.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:50.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:50.600943+0000) 2022-09-08T05:00:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:51 smithi059 ceph-mon[97450]: pgmap v14330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:51.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:51 smithi055 ceph-mon[97033]: pgmap v14330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:51.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:51.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:51.601150+0000) 2022-09-08T05:00:51.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:51.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:51.601181+0000) 2022-09-08T05:00:52.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:52.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:52.601333+0000) 2022-09-08T05:00:52.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:52.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:52.601357+0000) 2022-09-08T05:00:53.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:53 smithi055 ceph-mon[97033]: pgmap v14331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:00:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:00:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:00:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:00:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:53 smithi059 ceph-mon[97450]: pgmap v14331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:00:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:00:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:00:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:00:53.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:53.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:53.601581+0000) 2022-09-08T05:00:53.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:53.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:53.601576+0000) 2022-09-08T05:00:54.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:54.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:54.601771+0000) 2022-09-08T05:00:54.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:54.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:54.601775+0000) 2022-09-08T05:00:54.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:54 smithi055 ceph-mon[97033]: pgmap v14332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:54.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:00:55.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:54 smithi059 ceph-mon[97450]: pgmap v14332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:00:55.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:55.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:55.601895+0000) 2022-09-08T05:00:55.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:55.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:55.601916+0000) 2022-09-08T05:00:56.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:56.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:56.602120+0000) 2022-09-08T05:00:56.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:56.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:56.602153+0000) 2022-09-08T05:00:57.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:57 smithi059 ceph-mon[97450]: pgmap v14333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:00:57.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:57 smithi055 ceph-mon[97033]: pgmap v14333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:00:57.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:57.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:57.602294+0000) 2022-09-08T05:00:57.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:57.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:57.602429+0000) 2022-09-08T05:00:58.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:58.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:58.602512+0000) 2022-09-08T05:00:58.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:58.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:58.602623+0000) 2022-09-08T05:00:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:00:59 smithi059 ceph-mon[97450]: pgmap v14334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:59.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:00:59 smithi055 ceph-mon[97033]: pgmap v14334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:00:59.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:00:59.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:59.602747+0000) 2022-09-08T05:00:59.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:00:59.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:00:59.602923+0000) 2022-09-08T05:01:00.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:00.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:00.602998+0000) 2022-09-08T05:01:00.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:00.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:00.603161+0000) 2022-09-08T05:01:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:01 smithi059 ceph-mon[97450]: pgmap v14335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:01.568 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:01 smithi055 ceph-mon[97033]: pgmap v14335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:01.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:01.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:01.603196+0000) 2022-09-08T05:01:01.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:01.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:01.603382+0000) 2022-09-08T05:01:02.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:01:01.934+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:01:02.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:02.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:02.603445+0000) 2022-09-08T05:01:02.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:02.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:02.603660+0000) 2022-09-08T05:01:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:03 smithi059 ceph-mon[97450]: pgmap v14336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:01:03.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:03 smithi055 ceph-mon[97033]: pgmap v14336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:01:03.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:03.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:03.603748+0000) 2022-09-08T05:01:03.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:03.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:03.603817+0000) 2022-09-08T05:01:04.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:04.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:04.603964+0000) 2022-09-08T05:01:04.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:04.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:04.604035+0000) 2022-09-08T05:01:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:05 smithi059 ceph-mon[97450]: pgmap v14337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:05.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:05 smithi055 ceph-mon[97033]: pgmap v14337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:05.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:05.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:05.604231+0000) 2022-09-08T05:01:05.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:05.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:05.604190+0000) 2022-09-08T05:01:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:01:06.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:01:06.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:06.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:06.604350+0000) 2022-09-08T05:01:06.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:06.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:06.604385+0000) 2022-09-08T05:01:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:07 smithi059 ceph-mon[97450]: pgmap v14338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:01:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:01:07.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:07 smithi055 ceph-mon[97033]: pgmap v14338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:01:07.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:01:07.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:07.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:07.604593+0000) 2022-09-08T05:01:07.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:07.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:07.604603+0000) 2022-09-08T05:01:08.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:08.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:08.604861+0000) 2022-09-08T05:01:08.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:08.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:08.604878+0000) 2022-09-08T05:01:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:09 smithi059 ceph-mon[97450]: pgmap v14339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:01:09.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:09 smithi055 ceph-mon[97033]: pgmap v14339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:01:09.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:09.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:09.605127+0000) 2022-09-08T05:01:09.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:09.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:09.605155+0000) 2022-09-08T05:01:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:10 smithi059 ceph-mon[97450]: osdmap e977: 12 total, 11 up, 11 in 2022-09-08T05:01:10.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:10 smithi055 ceph-mon[97033]: osdmap e977: 12 total, 11 up, 11 in 2022-09-08T05:01:10.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:10.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:10.605376+0000) 2022-09-08T05:01:10.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:10.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:10.605402+0000) 2022-09-08T05:01:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:11 smithi059 ceph-mon[97450]: pgmap v14341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:01:11.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:11 smithi055 ceph-mon[97033]: pgmap v14341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:01:11.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:11.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:11.605617+0000) 2022-09-08T05:01:11.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:11.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:11.605611+0000) 2022-09-08T05:01:12.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:12.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:12.605819+0000) 2022-09-08T05:01:12.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:12.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:12.605814+0000) 2022-09-08T05:01:13.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:13 smithi055 ceph-mon[97033]: pgmap v14342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:01:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:13 smithi059 ceph-mon[97450]: pgmap v14342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:01:13.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:13.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:13.606043+0000) 2022-09-08T05:01:13.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:13.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:13.605971+0000) 2022-09-08T05:01:14.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:14.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:14.606206+0000) 2022-09-08T05:01:14.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:14.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:14.606286+0000) 2022-09-08T05:01:15.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:15 smithi055 ceph-mon[97033]: pgmap v14343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:01:15.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:15 smithi055 ceph-mon[97033]: osdmap e978: 12 total, 11 up, 11 in 2022-09-08T05:01:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:15 smithi059 ceph-mon[97450]: pgmap v14343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:01:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:15 smithi059 ceph-mon[97450]: osdmap e978: 12 total, 11 up, 11 in 2022-09-08T05:01:15.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:15.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:15.606520+0000) 2022-09-08T05:01:15.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:15.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:15.606458+0000) 2022-09-08T05:01:16.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:16.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:16.606767+0000) 2022-09-08T05:01:16.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:16.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:16.606757+0000) 2022-09-08T05:01:17.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:01:16.941+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:01:17.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:17 smithi055 ceph-mon[97033]: pgmap v14345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:01:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:17 smithi059 ceph-mon[97450]: pgmap v14345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:01:17.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:17.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:17.606982+0000) 2022-09-08T05:01:17.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:17.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:17.606956+0000) 2022-09-08T05:01:18.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:18.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:18.607209+0000) 2022-09-08T05:01:18.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:18.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:18.607198+0000) 2022-09-08T05:01:19.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:19 smithi055 ceph-mon[97033]: pgmap v14346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T05:01:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:19 smithi059 ceph-mon[97450]: pgmap v14346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T05:01:19.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:19.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:19.607410+0000) 2022-09-08T05:01:19.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:19.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:19.607404+0000) 2022-09-08T05:01:20.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:20.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:20.607671+0000) 2022-09-08T05:01:20.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:20.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:20.607680+0000) 2022-09-08T05:01:21.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:21 smithi055 ceph-mon[97033]: pgmap v14347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:01:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:21 smithi059 ceph-mon[97450]: pgmap v14347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:01:21.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:21.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:21.607840+0000) 2022-09-08T05:01:21.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:21.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:21.607828+0000) 2022-09-08T05:01:22.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:22.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:22.608064+0000) 2022-09-08T05:01:22.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:22.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:22.608070+0000) 2022-09-08T05:01:23.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:23 smithi055 ceph-mon[97033]: pgmap v14348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:01:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:23 smithi059 ceph-mon[97450]: pgmap v14348: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:01:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:23.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:23.608335+0000) 2022-09-08T05:01:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:23.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:23.608338+0000) 2022-09-08T05:01:24.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:24.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:24.608536+0000) 2022-09-08T05:01:24.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:24.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:24.608513+0000) 2022-09-08T05:01:25.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:25 smithi055 ceph-mon[97033]: pgmap v14349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:01:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:25 smithi059 ceph-mon[97450]: pgmap v14349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:01:25.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:25.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:25.608757+0000) 2022-09-08T05:01:25.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:25.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:25.608744+0000) 2022-09-08T05:01:26.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:26.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:26.608966+0000) 2022-09-08T05:01:26.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:26.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:26.608961+0000) 2022-09-08T05:01:27.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:27 smithi055 ceph-mon[97033]: pgmap v14350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:01:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:27 smithi059 ceph-mon[97450]: pgmap v14350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:01:27.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:27.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:27.609176+0000) 2022-09-08T05:01:27.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:27.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:27.609124+0000) 2022-09-08T05:01:28.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:28.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:28.609327+0000) 2022-09-08T05:01:28.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:28.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:28.609524+0000) 2022-09-08T05:01:29.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:29 smithi055 ceph-mon[97033]: pgmap v14351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:29 smithi059 ceph-mon[97450]: pgmap v14351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:29.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:29.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:29.609544+0000) 2022-09-08T05:01:29.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:29.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:29.609714+0000) 2022-09-08T05:01:30.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:30.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:30.609831+0000) 2022-09-08T05:01:30.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:30.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:30.609963+0000) 2022-09-08T05:01:31.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:31 smithi055 ceph-mon[97033]: pgmap v14352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:31 smithi059 ceph-mon[97450]: pgmap v14352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:31.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:31.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:31.610017+0000) 2022-09-08T05:01:31.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:31.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:31.610143+0000) 2022-09-08T05:01:32.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:01:31.955+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:01:32.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:32.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:32.610259+0000) 2022-09-08T05:01:32.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:32.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:32.610319+0000) 2022-09-08T05:01:33.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:33 smithi055 ceph-mon[97033]: pgmap v14353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:01:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:33 smithi059 ceph-mon[97450]: pgmap v14353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:01:33.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:33.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:33.610513+0000) 2022-09-08T05:01:33.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:33.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:33.610421+0000) 2022-09-08T05:01:34.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:34.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:34.610709+0000) 2022-09-08T05:01:34.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:34.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:34.610648+0000) 2022-09-08T05:01:35.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:35 smithi055 ceph-mon[97033]: pgmap v14354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:35 smithi059 ceph-mon[97450]: pgmap v14354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:35.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:35.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:35.610859+0000) 2022-09-08T05:01:35.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:35.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:35.610889+0000) 2022-09-08T05:01:36.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:36.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:36.610985+0000) 2022-09-08T05:01:36.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:36.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:36.611052+0000) 2022-09-08T05:01:37.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:37 smithi055 ceph-mon[97033]: pgmap v14355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:01:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:37 smithi059 ceph-mon[97450]: pgmap v14355: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:01:37.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:37.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:37.611154+0000) 2022-09-08T05:01:37.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:37.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:37.611238+0000) 2022-09-08T05:01:38.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:38.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:38.611394+0000) 2022-09-08T05:01:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:38.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:38.611411+0000) 2022-09-08T05:01:39.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:39 smithi055 ceph-mon[97033]: pgmap v14356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:39 smithi059 ceph-mon[97450]: pgmap v14356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:39.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:39.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:39.611612+0000) 2022-09-08T05:01:39.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:39.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:39.611680+0000) 2022-09-08T05:01:40.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:40.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:40.611894+0000) 2022-09-08T05:01:40.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:40.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:40.611802+0000) 2022-09-08T05:01:41.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:41 smithi055 ceph-mon[97033]: pgmap v14357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:41 smithi059 ceph-mon[97450]: pgmap v14357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:41.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:41.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:41.612064+0000) 2022-09-08T05:01:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:41.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:41.612008+0000) 2022-09-08T05:01:42.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:42.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:42.612277+0000) 2022-09-08T05:01:42.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:42.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:42.612245+0000) 2022-09-08T05:01:43.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:43 smithi055 ceph-mon[97033]: pgmap v14358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:01:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:43 smithi059 ceph-mon[97450]: pgmap v14358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:01:43.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:43.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:43.612516+0000) 2022-09-08T05:01:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:43.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:43.612498+0000) 2022-09-08T05:01:44.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:44.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:44.612751+0000) 2022-09-08T05:01:44.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:44.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:44.612785+0000) 2022-09-08T05:01:45.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:45 smithi055 ceph-mon[97033]: pgmap v14359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:45 smithi059 ceph-mon[97450]: pgmap v14359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:45.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:45.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:45.612995+0000) 2022-09-08T05:01:45.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:45.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:45.612974+0000) 2022-09-08T05:01:46.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:46.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:46.613232+0000) 2022-09-08T05:01:46.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:46.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:46.613232+0000) 2022-09-08T05:01:47.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:01:46.960+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:01:47.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:47 smithi055 ceph-mon[97033]: pgmap v14360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:01:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:47 smithi059 ceph-mon[97450]: pgmap v14360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:01:47.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:47.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:47.613437+0000) 2022-09-08T05:01:47.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:47.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:47.613447+0000) 2022-09-08T05:01:48.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:48.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:48.613660+0000) 2022-09-08T05:01:48.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:48.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:48.613631+0000) 2022-09-08T05:01:49.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:49 smithi055 ceph-mon[97033]: pgmap v14361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:49 smithi059 ceph-mon[97450]: pgmap v14361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:49.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:49.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:49.613879+0000) 2022-09-08T05:01:49.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:49.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:49.613848+0000) 2022-09-08T05:01:50.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:50.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:50.614073+0000) 2022-09-08T05:01:50.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:50.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:50.614062+0000) 2022-09-08T05:01:51.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:51 smithi055 ceph-mon[97033]: pgmap v14362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:51 smithi059 ceph-mon[97450]: pgmap v14362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:51.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:51.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:51.614261+0000) 2022-09-08T05:01:51.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:51.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:51.614259+0000) 2022-09-08T05:01:52.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:52.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:52.614474+0000) 2022-09-08T05:01:52.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:52.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:52.614494+0000) 2022-09-08T05:01:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:53 smithi055 ceph-mon[97033]: pgmap v14363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:01:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:53 smithi059 ceph-mon[97450]: pgmap v14363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:01:53.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:53.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:53.614720+0000) 2022-09-08T05:01:53.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:53.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:53.614729+0000) 2022-09-08T05:01:54.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:01:54.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:01:54.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:01:54.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:01:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:01:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:01:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:01:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:01:54.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:54.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:54.614957+0000) 2022-09-08T05:01:54.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:54.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:54.614967+0000) 2022-09-08T05:01:55.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:55 smithi055 ceph-mon[97033]: pgmap v14364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:55 smithi059 ceph-mon[97450]: pgmap v14364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:55.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:55.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:55.615172+0000) 2022-09-08T05:01:55.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:55.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:55.615177+0000) 2022-09-08T05:01:56.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:56.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:56.615350+0000) 2022-09-08T05:01:56.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:56.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:56.615348+0000) 2022-09-08T05:01:57.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:57 smithi055 ceph-mon[97033]: pgmap v14365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:01:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:57 smithi059 ceph-mon[97450]: pgmap v14365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:01:57.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:57.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:57.615549+0000) 2022-09-08T05:01:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:57.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:57.615598+0000) 2022-09-08T05:01:58.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:58.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:58.615740+0000) 2022-09-08T05:01:58.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:58.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:58.615818+0000) 2022-09-08T05:01:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:01:59 smithi055 ceph-mon[97033]: pgmap v14366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:01:59 smithi059 ceph-mon[97450]: pgmap v14366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:01:59.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:01:59.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:59.615966+0000) 2022-09-08T05:01:59.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:01:59.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:01:59.616077+0000) 2022-09-08T05:02:00.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:00.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:00.616238+0000) 2022-09-08T05:02:00.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:00.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:00.616296+0000) 2022-09-08T05:02:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:01 smithi059 ceph-mon[97450]: pgmap v14367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:01.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:01 smithi055 ceph-mon[97033]: pgmap v14367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:01.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:01.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:01.616477+0000) 2022-09-08T05:02:01.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:01.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:01.616365+0000) 2022-09-08T05:02:02.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:02:01.974+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:02:02.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:02.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:02.616554+0000) 2022-09-08T05:02:02.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:02.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:02.616651+0000) 2022-09-08T05:02:03.539 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:03 smithi059 ceph-mon[97450]: pgmap v14368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:02:03.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:03 smithi055 ceph-mon[97033]: pgmap v14368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:02:03.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:03.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:03.616728+0000) 2022-09-08T05:02:03.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:03.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:03.616825+0000) 2022-09-08T05:02:04.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:04.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:04.616885+0000) 2022-09-08T05:02:04.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:04.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:04.617050+0000) 2022-09-08T05:02:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:05 smithi059 ceph-mon[97450]: pgmap v14369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:05.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:05 smithi055 ceph-mon[97033]: pgmap v14369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:05.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:05.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:05.617055+0000) 2022-09-08T05:02:05.982 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:05.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:05.617222+0000) 2022-09-08T05:02:06.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:02:06.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:02:06.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:06.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:06.617266+0000) 2022-09-08T05:02:06.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:06.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:06.617356+0000) 2022-09-08T05:02:07.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:07 smithi059 ceph-mon[97450]: pgmap v14370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:02:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:02:07.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:07 smithi055 ceph-mon[97033]: pgmap v14370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:02:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:02:07.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:07.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:07.617496+0000) 2022-09-08T05:02:07.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:07.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:07.617544+0000) 2022-09-08T05:02:08.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:08.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:08.617662+0000) 2022-09-08T05:02:08.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:08.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:08.617773+0000) 2022-09-08T05:02:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:09 smithi059 ceph-mon[97450]: pgmap v14371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:02:09.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:09 smithi055 ceph-mon[97033]: pgmap v14371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:02:09.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:09.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:09.617904+0000) 2022-09-08T05:02:09.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:09.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:09.617993+0000) 2022-09-08T05:02:10.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:10 smithi055 ceph-mon[97033]: osdmap e979: 12 total, 11 up, 11 in 2022-09-08T05:02:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:10 smithi059 ceph-mon[97450]: osdmap e979: 12 total, 11 up, 11 in 2022-09-08T05:02:10.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:10.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:10.618087+0000) 2022-09-08T05:02:10.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:10.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:10.618172+0000) 2022-09-08T05:02:11.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:11 smithi055 ceph-mon[97033]: pgmap v14373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:02:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:11 smithi059 ceph-mon[97450]: pgmap v14373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:02:11.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:11.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:11.618297+0000) 2022-09-08T05:02:11.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:11.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:11.618398+0000) 2022-09-08T05:02:12.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:12.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:12.618496+0000) 2022-09-08T05:02:12.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:12.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:12.618609+0000) 2022-09-08T05:02:13.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:13 smithi055 ceph-mon[97033]: pgmap v14374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:02:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:13 smithi059 ceph-mon[97450]: pgmap v14374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:02:13.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:13.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:13.618787+0000) 2022-09-08T05:02:13.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:13.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:13.618761+0000) 2022-09-08T05:02:14.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:14.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:14.618998+0000) 2022-09-08T05:02:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:14.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:14.618968+0000) 2022-09-08T05:02:15.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:15 smithi055 ceph-mon[97033]: pgmap v14375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:02:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:15 smithi059 ceph-mon[97450]: pgmap v14375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:02:15.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:15.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:15.619188+0000) 2022-09-08T05:02:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:15.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:15.619232+0000) 2022-09-08T05:02:16.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:16.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:16.619437+0000) 2022-09-08T05:02:16.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:16.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:16.619464+0000) 2022-09-08T05:02:17.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:02:16.983+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:02:17.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:17 smithi055 ceph-mon[97033]: pgmap v14376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:02:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:17 smithi059 ceph-mon[97450]: pgmap v14376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:02:17.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:17.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:17.619689+0000) 2022-09-08T05:02:17.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:17.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:17.619703+0000) 2022-09-08T05:02:18.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:18.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:18.619945+0000) 2022-09-08T05:02:18.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:18.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:18.619965+0000) 2022-09-08T05:02:19.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:19 smithi055 ceph-mon[97033]: pgmap v14377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:02:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:19 smithi059 ceph-mon[97450]: pgmap v14377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:02:19.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:19.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:19.620247+0000) 2022-09-08T05:02:19.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:19.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:19.620240+0000) 2022-09-08T05:02:20.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:20.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:20.620473+0000) 2022-09-08T05:02:20.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:20.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:20.620479+0000) 2022-09-08T05:02:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:21 smithi059 ceph-mon[97450]: pgmap v14378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 201 B/s wr, 6 op/s 2022-09-08T05:02:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:21 smithi059 ceph-mon[97450]: osdmap e980: 12 total, 11 up, 11 in 2022-09-08T05:02:21.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:21 smithi055 ceph-mon[97033]: pgmap v14378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 201 B/s wr, 6 op/s 2022-09-08T05:02:21.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:21 smithi055 ceph-mon[97033]: osdmap e980: 12 total, 11 up, 11 in 2022-09-08T05:02:21.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:21.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:21.620707+0000) 2022-09-08T05:02:21.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:21.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:21.620715+0000) 2022-09-08T05:02:22.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:22.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:22.620839+0000) 2022-09-08T05:02:22.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:22.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:22.620827+0000) 2022-09-08T05:02:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:23 smithi059 ceph-mon[97450]: pgmap v14380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:02:23.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:23 smithi055 ceph-mon[97033]: pgmap v14380: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:02:23.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:23.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:23.621039+0000) 2022-09-08T05:02:23.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:23.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:23.621020+0000) 2022-09-08T05:02:24.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:24.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:24.621252+0000) 2022-09-08T05:02:24.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:24.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:24.621255+0000) 2022-09-08T05:02:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:25 smithi059 ceph-mon[97450]: pgmap v14381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:02:25.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:25 smithi055 ceph-mon[97033]: pgmap v14381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:02:25.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:25.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:25.621497+0000) 2022-09-08T05:02:25.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:25.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:25.621512+0000) 2022-09-08T05:02:26.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:26.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:26.621746+0000) 2022-09-08T05:02:26.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:26.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:26.621735+0000) 2022-09-08T05:02:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:27 smithi059 ceph-mon[97450]: pgmap v14382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:02:27.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:27 smithi055 ceph-mon[97033]: pgmap v14382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:02:27.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:27.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:27.621966+0000) 2022-09-08T05:02:27.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:27.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:27.621981+0000) 2022-09-08T05:02:28.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:28.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:28.622194+0000) 2022-09-08T05:02:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:28.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:28.622199+0000) 2022-09-08T05:02:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:29 smithi059 ceph-mon[97450]: pgmap v14383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:02:29.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:29 smithi055 ceph-mon[97033]: pgmap v14383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:02:29.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:29.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:29.622429+0000) 2022-09-08T05:02:29.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:29.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:29.622426+0000) 2022-09-08T05:02:30.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:30.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:30.622697+0000) 2022-09-08T05:02:30.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:30.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:30.622686+0000) 2022-09-08T05:02:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:31 smithi059 ceph-mon[97450]: pgmap v14384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:02:31.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:31 smithi055 ceph-mon[97033]: pgmap v14384: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:02:31.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:31.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:31.622898+0000) 2022-09-08T05:02:31.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:31.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:31.622897+0000) 2022-09-08T05:02:32.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:02:31.992+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:02:32.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:32.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:32.623130+0000) 2022-09-08T05:02:32.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:32.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:32.623104+0000) 2022-09-08T05:02:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:33 smithi059 ceph-mon[97450]: pgmap v14385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:02:33.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:33 smithi055 ceph-mon[97033]: pgmap v14385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:02:33.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:33.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:33.623395+0000) 2022-09-08T05:02:33.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:33.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:33.623356+0000) 2022-09-08T05:02:34.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:34.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:34.623594+0000) 2022-09-08T05:02:34.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:34.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:34.623556+0000) 2022-09-08T05:02:35.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:35 smithi055 ceph-mon[97033]: pgmap v14386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:35 smithi059 ceph-mon[97450]: pgmap v14386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:35.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:35.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:35.623812+0000) 2022-09-08T05:02:35.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:35.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:35.623795+0000) 2022-09-08T05:02:36.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:36.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:36.624072+0000) 2022-09-08T05:02:36.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:36.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:36.624028+0000) 2022-09-08T05:02:37.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:37 smithi055 ceph-mon[97033]: pgmap v14387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:02:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:37 smithi059 ceph-mon[97450]: pgmap v14387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:02:37.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:37.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:37.624305+0000) 2022-09-08T05:02:37.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:37.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:37.624271+0000) 2022-09-08T05:02:38.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:38.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:38.624486+0000) 2022-09-08T05:02:38.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:38.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:38.624458+0000) 2022-09-08T05:02:39.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:39 smithi055 ceph-mon[97033]: pgmap v14388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:39 smithi059 ceph-mon[97450]: pgmap v14388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:39.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:39.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:39.624684+0000) 2022-09-08T05:02:39.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:39.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:39.624665+0000) 2022-09-08T05:02:40.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:40.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:40.624899+0000) 2022-09-08T05:02:40.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:40.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:40.624899+0000) 2022-09-08T05:02:41.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:41 smithi055 ceph-mon[97033]: pgmap v14389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:41 smithi059 ceph-mon[97450]: pgmap v14389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:41.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:41.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:41.625111+0000) 2022-09-08T05:02:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:41.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:41.625065+0000) 2022-09-08T05:02:42.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:42.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:42.625317+0000) 2022-09-08T05:02:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:42.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:42.625277+0000) 2022-09-08T05:02:43.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:43 smithi055 ceph-mon[97033]: pgmap v14390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:02:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:43 smithi059 ceph-mon[97450]: pgmap v14390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:02:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:43.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:43.625523+0000) 2022-09-08T05:02:43.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:43.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:43.625460+0000) 2022-09-08T05:02:44.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:44.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:44.625699+0000) 2022-09-08T05:02:44.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:44.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:44.625685+0000) 2022-09-08T05:02:45.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:45 smithi055 ceph-mon[97033]: pgmap v14391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:45 smithi059 ceph-mon[97450]: pgmap v14391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:45.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:45.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:45.625833+0000) 2022-09-08T05:02:45.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:45.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:45.625824+0000) 2022-09-08T05:02:46.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:46.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:46.625957+0000) 2022-09-08T05:02:46.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:46.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:46.625951+0000) 2022-09-08T05:02:47.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:02:47.004+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:02:47.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:47 smithi055 ceph-mon[97033]: pgmap v14392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:02:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:47 smithi059 ceph-mon[97450]: pgmap v14392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:02:47.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:47.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:47.626115+0000) 2022-09-08T05:02:47.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:47.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:47.626128+0000) 2022-09-08T05:02:48.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:48.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:48.626327+0000) 2022-09-08T05:02:48.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:48.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:48.626349+0000) 2022-09-08T05:02:49.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:49 smithi055 ceph-mon[97033]: pgmap v14393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:49 smithi059 ceph-mon[97450]: pgmap v14393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:49.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:49.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:49.626590+0000) 2022-09-08T05:02:49.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:49.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:49.626585+0000) 2022-09-08T05:02:50.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:50.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:50.626778+0000) 2022-09-08T05:02:50.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:50.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:50.626742+0000) 2022-09-08T05:02:51.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:51 smithi055 ceph-mon[97033]: pgmap v14394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:51 smithi059 ceph-mon[97450]: pgmap v14394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:51.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:51.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:51.626939+0000) 2022-09-08T05:02:51.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:51.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:51.626880+0000) 2022-09-08T05:02:52.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:52.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:52.627081+0000) 2022-09-08T05:02:52.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:52.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:52.627024+0000) 2022-09-08T05:02:53.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:53 smithi055 ceph-mon[97033]: pgmap v14395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:02:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:53 smithi059 ceph-mon[97450]: pgmap v14395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:02:53.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:53.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:53.627316+0000) 2022-09-08T05:02:53.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:53.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:53.627229+0000) 2022-09-08T05:02:54.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:54.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:54.627570+0000) 2022-09-08T05:02:54.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:54.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:54.627473+0000) 2022-09-08T05:02:55.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:55 smithi055 ceph-mon[97033]: pgmap v14396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:55.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:02:55.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:02:55.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:02:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:02:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:55 smithi059 ceph-mon[97450]: pgmap v14396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:02:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:02:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:02:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:02:55.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:55.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:55.627801+0000) 2022-09-08T05:02:55.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:55.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:55.627693+0000) 2022-09-08T05:02:56.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:56.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:56.627944+0000) 2022-09-08T05:02:56.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:56.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:56.627851+0000) 2022-09-08T05:02:57.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:57 smithi055 ceph-mon[97033]: pgmap v14397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:02:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:57 smithi059 ceph-mon[97450]: pgmap v14397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:02:57.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:57.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:57.628077+0000) 2022-09-08T05:02:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:57.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:57.628025+0000) 2022-09-08T05:02:58.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:58.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:58.628232+0000) 2022-09-08T05:02:58.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:58.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:58.628198+0000) 2022-09-08T05:02:59.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:02:59 smithi055 ceph-mon[97033]: pgmap v14398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:02:59 smithi059 ceph-mon[97450]: pgmap v14398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:02:59.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:02:59.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:59.628436+0000) 2022-09-08T05:02:59.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:02:59.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:02:59.628435+0000) 2022-09-08T05:03:00.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:00.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:00.628679+0000) 2022-09-08T05:03:00.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:00.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:00.628683+0000) 2022-09-08T05:03:01.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:01 smithi055 ceph-mon[97033]: pgmap v14399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:01 smithi059 ceph-mon[97450]: pgmap v14399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:01.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:01.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:01.628820+0000) 2022-09-08T05:03:01.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:01.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:01.628874+0000) 2022-09-08T05:03:02.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:03:02.013+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:03:02.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:02.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:02.628996+0000) 2022-09-08T05:03:02.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:02.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:02.629074+0000) 2022-09-08T05:03:03.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:03 smithi055 ceph-mon[97033]: pgmap v14400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:03:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:03 smithi059 ceph-mon[97450]: pgmap v14400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:03:03.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:03.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:03.629237+0000) 2022-09-08T05:03:03.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:03.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:03.629330+0000) 2022-09-08T05:03:04.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:04.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:04.629537+0000) 2022-09-08T05:03:04.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:04.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:04.629442+0000) 2022-09-08T05:03:05.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:05 smithi055 ceph-mon[97033]: pgmap v14401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:05 smithi059 ceph-mon[97450]: pgmap v14401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:05.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:05.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:05.629601+0000) 2022-09-08T05:03:05.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:05.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:05.629726+0000) 2022-09-08T05:03:06.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:03:06.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:03:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:03:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:03:06.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:06.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:06.629939+0000) 2022-09-08T05:03:06.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:06.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:06.629764+0000) 2022-09-08T05:03:07.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:07 smithi055 ceph-mon[97033]: pgmap v14402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:03:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:07 smithi059 ceph-mon[97450]: pgmap v14402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:03:07.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:07.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:07.629927+0000) 2022-09-08T05:03:07.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:07.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:07.630118+0000) 2022-09-08T05:03:08.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:08.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:08.630155+0000) 2022-09-08T05:03:08.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:08.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:08.630373+0000) 2022-09-08T05:03:09.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:09 smithi055 ceph-mon[97033]: pgmap v14403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:03:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:09 smithi059 ceph-mon[97450]: pgmap v14403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:03:09.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:09.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:09.630320+0000) 2022-09-08T05:03:09.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:09.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:09.630540+0000) 2022-09-08T05:03:10.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:10 smithi055 ceph-mon[97033]: osdmap e981: 12 total, 11 up, 11 in 2022-09-08T05:03:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:10 smithi059 ceph-mon[97450]: osdmap e981: 12 total, 11 up, 11 in 2022-09-08T05:03:10.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:10.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:10.630699+0000) 2022-09-08T05:03:10.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:10.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:10.630518+0000) 2022-09-08T05:03:11.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:11 smithi055 ceph-mon[97033]: pgmap v14405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:03:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:11 smithi059 ceph-mon[97450]: pgmap v14405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:03:11.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:11.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:11.630721+0000) 2022-09-08T05:03:11.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:11.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:11.630848+0000) 2022-09-08T05:03:12.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:12.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:12.630866+0000) 2022-09-08T05:03:12.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:12.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:12.631022+0000) 2022-09-08T05:03:13.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:13 smithi055 ceph-mon[97033]: pgmap v14406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:03:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:13 smithi059 ceph-mon[97450]: pgmap v14406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:03:13.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:13.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:13.631005+0000) 2022-09-08T05:03:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:13.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:13.631164+0000) 2022-09-08T05:03:14.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:14.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:14.631244+0000) 2022-09-08T05:03:14.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:14.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:14.631343+0000) 2022-09-08T05:03:15.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:15 smithi055 ceph-mon[97033]: pgmap v14407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:03:15.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:15 smithi055 ceph-mon[97033]: osdmap e982: 12 total, 11 up, 11 in 2022-09-08T05:03:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:15 smithi059 ceph-mon[97450]: pgmap v14407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:03:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:15 smithi059 ceph-mon[97450]: osdmap e982: 12 total, 11 up, 11 in 2022-09-08T05:03:15.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:15.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:15.631441+0000) 2022-09-08T05:03:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:15.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:15.631565+0000) 2022-09-08T05:03:16.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:16 smithi055 ceph-mon[97033]: pgmap v14409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:03:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:16 smithi059 ceph-mon[97450]: pgmap v14409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:03:16.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:16.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:16.631657+0000) 2022-09-08T05:03:16.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:16.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:16.631733+0000) 2022-09-08T05:03:17.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:03:17.021+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:03:17.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:17.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:17.631894+0000) 2022-09-08T05:03:17.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:17.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:17.631952+0000) 2022-09-08T05:03:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:18 smithi059 ceph-mon[97450]: pgmap v14410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T05:03:18.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:18.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:18.632117+0000) 2022-09-08T05:03:18.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:18.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:18.632138+0000) 2022-09-08T05:03:18.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:18 smithi055 ceph-mon[97033]: pgmap v14410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T05:03:19.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:19.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:19.632317+0000) 2022-09-08T05:03:19.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:19.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:19.632358+0000) 2022-09-08T05:03:20.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:20.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:20.632549+0000) 2022-09-08T05:03:20.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:20.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:20.632602+0000) 2022-09-08T05:03:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:21 smithi059 ceph-mon[97450]: pgmap v14411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:03:21.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:21 smithi055 ceph-mon[97033]: pgmap v14411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:03:21.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:21.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:21.632804+0000) 2022-09-08T05:03:21.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:21.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:21.632923+0000) 2022-09-08T05:03:22.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:22.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:22.632958+0000) 2022-09-08T05:03:22.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:22.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:22.633089+0000) 2022-09-08T05:03:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:23 smithi059 ceph-mon[97450]: pgmap v14412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:03:23.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:23 smithi055 ceph-mon[97033]: pgmap v14412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:03:23.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:23.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:23.633121+0000) 2022-09-08T05:03:23.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:23.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:23.633307+0000) 2022-09-08T05:03:24.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:24.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:24.633358+0000) 2022-09-08T05:03:24.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:24.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:24.633569+0000) 2022-09-08T05:03:25.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:25 smithi055 ceph-mon[97033]: pgmap v14413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:03:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:25 smithi059 ceph-mon[97450]: pgmap v14413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:03:25.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:25.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:25.633487+0000) 2022-09-08T05:03:25.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:25.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:25.633773+0000) 2022-09-08T05:03:26.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:26.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:26.633692+0000) 2022-09-08T05:03:26.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:26.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:26.633893+0000) 2022-09-08T05:03:27.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:27 smithi055 ceph-mon[97033]: pgmap v14414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:03:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:27 smithi059 ceph-mon[97450]: pgmap v14414: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:03:27.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:27.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:27.633821+0000) 2022-09-08T05:03:27.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:27.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:27.634017+0000) 2022-09-08T05:03:28.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:28.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:28.634060+0000) 2022-09-08T05:03:28.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:28.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:28.634187+0000) 2022-09-08T05:03:29.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:29 smithi055 ceph-mon[97033]: pgmap v14415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:29 smithi059 ceph-mon[97450]: pgmap v14415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:29.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:29.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:29.634326+0000) 2022-09-08T05:03:29.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:29.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:29.634381+0000) 2022-09-08T05:03:30.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:30.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:30.634553+0000) 2022-09-08T05:03:30.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:30.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:30.634557+0000) 2022-09-08T05:03:31.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:31 smithi055 ceph-mon[97033]: pgmap v14416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:31 smithi059 ceph-mon[97450]: pgmap v14416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:31.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:31.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:31.634761+0000) 2022-09-08T05:03:31.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:31.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:31.634780+0000) 2022-09-08T05:03:32.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:03:32.036+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:03:32.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:32.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:32.634982+0000) 2022-09-08T05:03:32.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:32.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:32.634977+0000) 2022-09-08T05:03:33.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:33 smithi055 ceph-mon[97033]: pgmap v14417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:03:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:33 smithi059 ceph-mon[97450]: pgmap v14417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:03:33.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:33.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:33.635244+0000) 2022-09-08T05:03:33.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:33.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:33.635223+0000) 2022-09-08T05:03:34.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:34.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:34.635462+0000) 2022-09-08T05:03:34.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:34.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:34.635444+0000) 2022-09-08T05:03:35.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:35 smithi055 ceph-mon[97033]: pgmap v14418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:35 smithi059 ceph-mon[97450]: pgmap v14418: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:35.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:35.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:35.635712+0000) 2022-09-08T05:03:35.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:35.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:35.635740+0000) 2022-09-08T05:03:36.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:36.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:36.635921+0000) 2022-09-08T05:03:36.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:36.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:36.635976+0000) 2022-09-08T05:03:37.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:37 smithi055 ceph-mon[97033]: pgmap v14419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:03:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:37 smithi059 ceph-mon[97450]: pgmap v14419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:03:37.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:37.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:37.636173+0000) 2022-09-08T05:03:37.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:37.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:37.636192+0000) 2022-09-08T05:03:38.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:38.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:38.636410+0000) 2022-09-08T05:03:38.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:38.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:38.636402+0000) 2022-09-08T05:03:39.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:39 smithi055 ceph-mon[97033]: pgmap v14420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:39 smithi059 ceph-mon[97450]: pgmap v14420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:39.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:39.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:39.636625+0000) 2022-09-08T05:03:39.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:39.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:39.636669+0000) 2022-09-08T05:03:40.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:40.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:40.636790+0000) 2022-09-08T05:03:40.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:40.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:40.636775+0000) 2022-09-08T05:03:41.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:41 smithi055 ceph-mon[97033]: pgmap v14421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:41 smithi059 ceph-mon[97450]: pgmap v14421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:41.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:41.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:41.636940+0000) 2022-09-08T05:03:41.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:41.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:41.636925+0000) 2022-09-08T05:03:42.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:42.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:42.637095+0000) 2022-09-08T05:03:42.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:42.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:42.637061+0000) 2022-09-08T05:03:43.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:43 smithi055 ceph-mon[97033]: pgmap v14422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:03:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:43 smithi059 ceph-mon[97450]: pgmap v14422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:03:43.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:43.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:43.637342+0000) 2022-09-08T05:03:43.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:43.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:43.637279+0000) 2022-09-08T05:03:44.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:44.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:44.637607+0000) 2022-09-08T05:03:44.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:44.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:44.637547+0000) 2022-09-08T05:03:45.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:45 smithi055 ceph-mon[97033]: pgmap v14423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:45 smithi059 ceph-mon[97450]: pgmap v14423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:45.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:45.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:45.637820+0000) 2022-09-08T05:03:45.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:45.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:45.637707+0000) 2022-09-08T05:03:46.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:46.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:46.637946+0000) 2022-09-08T05:03:46.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:46.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:46.637810+0000) 2022-09-08T05:03:47.331 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:03:47.044+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:03:47.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:47 smithi055 ceph-mon[97033]: pgmap v14424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:03:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:47 smithi059 ceph-mon[97450]: pgmap v14424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:03:47.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:47.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:47.638008+0000) 2022-09-08T05:03:47.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:47.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:47.638071+0000) 2022-09-08T05:03:48.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:48.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:48.638301+0000) 2022-09-08T05:03:48.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:48.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:48.638270+0000) 2022-09-08T05:03:49.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:49 smithi055 ceph-mon[97033]: pgmap v14425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:49 smithi059 ceph-mon[97450]: pgmap v14425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:49.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:49.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:49.638576+0000) 2022-09-08T05:03:49.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:49.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:49.638573+0000) 2022-09-08T05:03:50.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:50.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:50.638798+0000) 2022-09-08T05:03:50.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:50.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:50.638808+0000) 2022-09-08T05:03:51.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:51 smithi055 ceph-mon[97033]: pgmap v14426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:51 smithi059 ceph-mon[97450]: pgmap v14426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:51.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:51.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:51.639045+0000) 2022-09-08T05:03:51.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:51.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:51.639038+0000) 2022-09-08T05:03:52.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:52.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:52.639276+0000) 2022-09-08T05:03:52.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:52.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:52.639337+0000) 2022-09-08T05:03:53.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:53 smithi055 ceph-mon[97033]: pgmap v14427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:03:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:53 smithi059 ceph-mon[97450]: pgmap v14427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:03:53.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:53.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:53.639546+0000) 2022-09-08T05:03:53.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:53.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:53.639519+0000) 2022-09-08T05:03:54.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:54.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:54.639775+0000) 2022-09-08T05:03:54.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:54.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:54.639732+0000) 2022-09-08T05:03:55.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:55 smithi055 ceph-mon[97033]: pgmap v14428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:55.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:03:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:03:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:03:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:55 smithi059 ceph-mon[97450]: pgmap v14428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:03:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:03:55.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:03:55.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:55.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:55.639917+0000) 2022-09-08T05:03:55.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:55.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:55.640003+0000) 2022-09-08T05:03:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:56 smithi059 ceph-mon[97450]: pgmap v14429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:03:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:03:56.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:56.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:56.640248+0000) 2022-09-08T05:03:56.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:56.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:56.640179+0000) 2022-09-08T05:03:56.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:56 smithi055 ceph-mon[97033]: pgmap v14429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:03:56.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:03:57.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:57.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:57.640412+0000) 2022-09-08T05:03:57.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:57.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:57.640385+0000) 2022-09-08T05:03:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:03:58 smithi059 ceph-mon[97450]: pgmap v14430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:58.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:58.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:58.640615+0000) 2022-09-08T05:03:58.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:58.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:58.640585+0000) 2022-09-08T05:03:58.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:03:58 smithi055 ceph-mon[97033]: pgmap v14430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:03:59.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:03:59.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:59.640791+0000) 2022-09-08T05:03:59.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:03:59.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:03:59.640775+0000) 2022-09-08T05:04:00.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:00.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:00.640898+0000) 2022-09-08T05:04:00.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:00.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:00.640938+0000) 2022-09-08T05:04:01.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:01 smithi055 ceph-mon[97033]: pgmap v14431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:01 smithi059 ceph-mon[97450]: pgmap v14431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:01.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:01.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:01.641155+0000) 2022-09-08T05:04:01.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:01.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:01.641147+0000) 2022-09-08T05:04:02.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:04:02.053+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:04:02.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:02.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:02.641384+0000) 2022-09-08T05:04:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:02.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:02.641385+0000) 2022-09-08T05:04:03.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:03 smithi055 ceph-mon[97033]: pgmap v14432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:04:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:03 smithi059 ceph-mon[97450]: pgmap v14432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:04:03.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:03.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:03.641614+0000) 2022-09-08T05:04:03.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:03.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:03.641616+0000) 2022-09-08T05:04:04.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:04.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:04.641916+0000) 2022-09-08T05:04:04.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:04.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:04.641905+0000) 2022-09-08T05:04:05.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:05 smithi055 ceph-mon[97033]: pgmap v14433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:05 smithi059 ceph-mon[97450]: pgmap v14433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:05.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:05.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:05.642096+0000) 2022-09-08T05:04:05.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:05.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:05.642098+0000) 2022-09-08T05:04:06.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:04:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:04:06.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:06.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:06.642272+0000) 2022-09-08T05:04:06.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:06.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:06.642276+0000) 2022-09-08T05:04:07.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:07 smithi055 ceph-mon[97033]: pgmap v14434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:04:07.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:04:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:07 smithi059 ceph-mon[97450]: pgmap v14434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:04:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:04:07.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:07.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:07.642447+0000) 2022-09-08T05:04:07.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:07.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:07.642444+0000) 2022-09-08T05:04:08.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:08.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:08.642708+0000) 2022-09-08T05:04:08.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:08.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:08.642700+0000) 2022-09-08T05:04:09.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:09 smithi055 ceph-mon[97033]: pgmap v14435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:04:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:09 smithi059 ceph-mon[97450]: pgmap v14435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:04:09.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:09.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:09.642879+0000) 2022-09-08T05:04:09.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:09.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:09.642916+0000) 2022-09-08T05:04:10.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:10 smithi055 ceph-mon[97033]: osdmap e983: 12 total, 11 up, 11 in 2022-09-08T05:04:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:10 smithi059 ceph-mon[97450]: osdmap e983: 12 total, 11 up, 11 in 2022-09-08T05:04:10.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:10.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:10.643019+0000) 2022-09-08T05:04:10.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:10.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:10.643056+0000) 2022-09-08T05:04:11.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:11 smithi055 ceph-mon[97033]: pgmap v14437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:04:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:11 smithi059 ceph-mon[97450]: pgmap v14437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:04:11.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:11.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:11.643186+0000) 2022-09-08T05:04:11.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:11.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:11.643193+0000) 2022-09-08T05:04:12.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:12.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:12.643431+0000) 2022-09-08T05:04:12.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:12.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:12.643409+0000) 2022-09-08T05:04:13.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:13 smithi055 ceph-mon[97033]: pgmap v14438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:04:13.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:13 smithi059 ceph-mon[97450]: pgmap v14438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:04:13.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:13.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:13.643659+0000) 2022-09-08T05:04:13.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:13.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:13.643648+0000) 2022-09-08T05:04:14.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:14.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:14.643908+0000) 2022-09-08T05:04:14.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:14.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:14.643864+0000) 2022-09-08T05:04:15.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:15 smithi055 ceph-mon[97033]: pgmap v14439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:04:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:15 smithi059 ceph-mon[97450]: pgmap v14439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:04:15.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:15.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:15.644161+0000) 2022-09-08T05:04:15.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:15.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:15.644057+0000) 2022-09-08T05:04:16.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:16.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:16.644335+0000) 2022-09-08T05:04:16.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:16.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:16.644271+0000) 2022-09-08T05:04:17.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:04:17.069+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:04:17.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:17 smithi055 ceph-mon[97033]: pgmap v14440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:04:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:17 smithi059 ceph-mon[97450]: pgmap v14440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:04:17.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:17.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:17.644480+0000) 2022-09-08T05:04:17.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:17.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:17.644425+0000) 2022-09-08T05:04:18.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:18.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:18.644753+0000) 2022-09-08T05:04:18.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:18.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:18.644688+0000) 2022-09-08T05:04:19.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:19 smithi055 ceph-mon[97033]: pgmap v14441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:04:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:19 smithi059 ceph-mon[97450]: pgmap v14441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:04:19.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:19.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:19.644970+0000) 2022-09-08T05:04:19.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:19.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:19.644950+0000) 2022-09-08T05:04:20.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:20.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:20.645139+0000) 2022-09-08T05:04:20.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:20.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:20.645106+0000) 2022-09-08T05:04:21.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:21 smithi055 ceph-mon[97033]: pgmap v14442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 201 B/s wr, 6 op/s 2022-09-08T05:04:21.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:21 smithi055 ceph-mon[97033]: osdmap e984: 12 total, 11 up, 11 in 2022-09-08T05:04:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:21 smithi059 ceph-mon[97450]: pgmap v14442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 201 B/s wr, 6 op/s 2022-09-08T05:04:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:21 smithi059 ceph-mon[97450]: osdmap e984: 12 total, 11 up, 11 in 2022-09-08T05:04:21.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:21.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:21.645296+0000) 2022-09-08T05:04:21.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:21.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:21.645257+0000) 2022-09-08T05:04:22.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:22.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:22.645501+0000) 2022-09-08T05:04:22.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:22.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:22.645506+0000) 2022-09-08T05:04:23.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:23 smithi055 ceph-mon[97033]: pgmap v14444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:04:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:23 smithi059 ceph-mon[97450]: pgmap v14444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:04:23.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:23.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:23.645742+0000) 2022-09-08T05:04:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:23.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:23.645728+0000) 2022-09-08T05:04:24.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:24.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:24.645882+0000) 2022-09-08T05:04:24.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:24.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:24.645889+0000) 2022-09-08T05:04:25.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:25 smithi055 ceph-mon[97033]: pgmap v14445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:04:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:25 smithi059 ceph-mon[97450]: pgmap v14445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:04:25.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:25.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:25.646108+0000) 2022-09-08T05:04:25.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:25.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:25.646116+0000) 2022-09-08T05:04:26.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:26.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:26.646336+0000) 2022-09-08T05:04:26.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:26.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:26.646356+0000) 2022-09-08T05:04:27.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:27 smithi055 ceph-mon[97033]: pgmap v14446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:04:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:27 smithi059 ceph-mon[97450]: pgmap v14446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:04:27.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:27.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:27.646590+0000) 2022-09-08T05:04:27.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:27.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:27.646662+0000) 2022-09-08T05:04:28.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:28.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:28.646705+0000) 2022-09-08T05:04:28.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:28.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:28.646887+0000) 2022-09-08T05:04:29.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:29 smithi055 ceph-mon[97033]: pgmap v14447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:04:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:29 smithi059 ceph-mon[97450]: pgmap v14447: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:04:29.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:29.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:29.646942+0000) 2022-09-08T05:04:29.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:29.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:29.647049+0000) 2022-09-08T05:04:30.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:30.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:30.647205+0000) 2022-09-08T05:04:30.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:30.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:30.647221+0000) 2022-09-08T05:04:31.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:31 smithi055 ceph-mon[97033]: pgmap v14448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:04:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:31 smithi059 ceph-mon[97450]: pgmap v14448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:04:31.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:31.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:31.647453+0000) 2022-09-08T05:04:31.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:31.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:31.647440+0000) 2022-09-08T05:04:32.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:04:32.080+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:04:32.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:32.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:32.647722+0000) 2022-09-08T05:04:32.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:32.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:32.647799+0000) 2022-09-08T05:04:33.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:33 smithi055 ceph-mon[97033]: pgmap v14449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:04:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:33 smithi059 ceph-mon[97450]: pgmap v14449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:04:33.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:33.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:33.647989+0000) 2022-09-08T05:04:33.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:33.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:33.648065+0000) 2022-09-08T05:04:34.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:34.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:34.648224+0000) 2022-09-08T05:04:34.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:34.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:34.648319+0000) 2022-09-08T05:04:35.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:35 smithi055 ceph-mon[97033]: pgmap v14450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:35 smithi059 ceph-mon[97450]: pgmap v14450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:35.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:35.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:35.648428+0000) 2022-09-08T05:04:35.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:35.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:35.648543+0000) 2022-09-08T05:04:36.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:36.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:36.648588+0000) 2022-09-08T05:04:36.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:36.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:36.648712+0000) 2022-09-08T05:04:37.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:37 smithi055 ceph-mon[97033]: pgmap v14451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:04:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:37 smithi059 ceph-mon[97450]: pgmap v14451: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:04:37.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:37.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:37.648820+0000) 2022-09-08T05:04:37.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:37.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:37.648873+0000) 2022-09-08T05:04:38.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:38.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:38.649081+0000) 2022-09-08T05:04:38.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:38.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:38.649075+0000) 2022-09-08T05:04:39.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:39 smithi055 ceph-mon[97033]: pgmap v14452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:39 smithi059 ceph-mon[97450]: pgmap v14452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:39.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:39.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:39.649368+0000) 2022-09-08T05:04:39.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:39.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:39.649336+0000) 2022-09-08T05:04:40.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:40.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:40.649563+0000) 2022-09-08T05:04:40.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:40.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:40.649557+0000) 2022-09-08T05:04:41.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:41 smithi055 ceph-mon[97033]: pgmap v14453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:41 smithi059 ceph-mon[97450]: pgmap v14453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:41.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:41.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:41.649804+0000) 2022-09-08T05:04:41.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:41.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:41.649791+0000) 2022-09-08T05:04:42.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:42.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:42.650020+0000) 2022-09-08T05:04:42.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:42.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:42.650016+0000) 2022-09-08T05:04:43.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:43 smithi055 ceph-mon[97033]: pgmap v14454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:04:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:43 smithi059 ceph-mon[97450]: pgmap v14454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:04:43.899 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:43.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:43.650191+0000) 2022-09-08T05:04:43.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:43.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:43.650184+0000) 2022-09-08T05:04:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:44.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:44.650405+0000) 2022-09-08T05:04:44.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:44.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:44.650441+0000) 2022-09-08T05:04:45.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:45 smithi055 ceph-mon[97033]: pgmap v14455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:45 smithi059 ceph-mon[97450]: pgmap v14455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:45.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:45.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:45.650671+0000) 2022-09-08T05:04:45.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:45.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:45.650750+0000) 2022-09-08T05:04:46.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:46.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:46.650931+0000) 2022-09-08T05:04:46.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:46.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:46.650985+0000) 2022-09-08T05:04:47.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:04:47.086+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:04:47.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:47 smithi055 ceph-mon[97033]: pgmap v14456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:04:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:47 smithi059 ceph-mon[97450]: pgmap v14456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:04:47.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:47.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:47.651135+0000) 2022-09-08T05:04:47.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:47.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:47.651266+0000) 2022-09-08T05:04:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:48.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:48.651380+0000) 2022-09-08T05:04:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:48.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:48.651515+0000) 2022-09-08T05:04:49.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:49 smithi055 ceph-mon[97033]: pgmap v14457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:49 smithi059 ceph-mon[97450]: pgmap v14457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:50.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:49.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:49.651596+0000) 2022-09-08T05:04:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:49.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:49.651711+0000) 2022-09-08T05:04:51.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:50.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:50.651834+0000) 2022-09-08T05:04:51.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:50.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:50.651890+0000) 2022-09-08T05:04:51.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:51 smithi055 ceph-mon[97033]: pgmap v14458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:51 smithi059 ceph-mon[97450]: pgmap v14458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:52.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:51.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:51.652077+0000) 2022-09-08T05:04:52.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:51.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:51.652124+0000) 2022-09-08T05:04:53.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:52.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:52.652346+0000) 2022-09-08T05:04:53.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:52.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:52.652368+0000) 2022-09-08T05:04:53.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:53 smithi055 ceph-mon[97033]: pgmap v14459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:04:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:53 smithi059 ceph-mon[97450]: pgmap v14459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:04:54.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:53.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:53.652491+0000) 2022-09-08T05:04:54.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:53.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:53.652511+0000) 2022-09-08T05:04:55.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:54.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:54.652737+0000) 2022-09-08T05:04:55.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:54.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:54.652760+0000) 2022-09-08T05:04:55.644 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:55 smithi059 ceph-mon[97450]: pgmap v14460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:55.652 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:55 smithi055 ceph-mon[97033]: pgmap v14460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:56.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:55.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:55.652919+0000) 2022-09-08T05:04:56.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:55.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:55.652939+0000) 2022-09-08T05:04:56.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:04:56.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:04:56.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:04:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:04:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:04:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:04:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:04:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:04:57.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:56.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:56.653064+0000) 2022-09-08T05:04:57.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:56.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:56.653109+0000) 2022-09-08T05:04:57.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:57 smithi055 ceph-mon[97033]: pgmap v14461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:04:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:57 smithi059 ceph-mon[97450]: pgmap v14461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:04:58.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:57.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:57.653247+0000) 2022-09-08T05:04:58.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:57.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:57.653285+0000) 2022-09-08T05:04:59.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:58.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:58.653401+0000) 2022-09-08T05:04:59.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:58.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:58.653442+0000) 2022-09-08T05:04:59.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:04:59 smithi055 ceph-mon[97033]: pgmap v14462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:04:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:04:59 smithi059 ceph-mon[97450]: pgmap v14462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:04:59.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:59.653665+0000) 2022-09-08T05:05:00.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:04:59.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:04:59.653730+0000) 2022-09-08T05:05:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:00.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:00.653937+0000) 2022-09-08T05:05:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:00.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:00.653997+0000) 2022-09-08T05:05:01.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:01 smithi055 ceph-mon[97033]: pgmap v14463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:01 smithi059 ceph-mon[97450]: pgmap v14463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:02.093 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:01.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:01.654193+0000) 2022-09-08T05:05:02.094 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:01.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:01.654234+0000) 2022-09-08T05:05:02.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:05:02.094+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:05:03.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:02.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:02.654378+0000) 2022-09-08T05:05:03.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:02.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:02.654393+0000) 2022-09-08T05:05:03.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:03 smithi055 ceph-mon[97033]: pgmap v14464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:05:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:03 smithi059 ceph-mon[97450]: pgmap v14464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:05:04.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:03.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:03.654665+0000) 2022-09-08T05:05:04.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:03.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:03.654675+0000) 2022-09-08T05:05:05.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:04.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:04.654873+0000) 2022-09-08T05:05:05.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:04.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:04.654879+0000) 2022-09-08T05:05:05.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:05 smithi055 ceph-mon[97033]: pgmap v14465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:05 smithi059 ceph-mon[97450]: pgmap v14465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:06.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:05.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:05.655091+0000) 2022-09-08T05:05:06.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:05.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:05.655105+0000) 2022-09-08T05:05:06.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:05:06.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:05:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:05:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:05:07.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:06.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:06.655290+0000) 2022-09-08T05:05:07.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:06.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:06.655285+0000) 2022-09-08T05:05:07.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:07 smithi055 ceph-mon[97033]: pgmap v14466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:05:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:07 smithi059 ceph-mon[97450]: pgmap v14466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:05:08.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:07.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:07.655526+0000) 2022-09-08T05:05:08.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:07.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:07.655530+0000) 2022-09-08T05:05:09.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:08.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:08.655807+0000) 2022-09-08T05:05:09.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:08.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:08.655811+0000) 2022-09-08T05:05:09.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:09 smithi055 ceph-mon[97033]: pgmap v14467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:05:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:09 smithi059 ceph-mon[97450]: pgmap v14467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:05:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:09.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:09.656024+0000) 2022-09-08T05:05:10.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:09.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:09.656037+0000) 2022-09-08T05:05:10.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:10 smithi055 ceph-mon[97033]: osdmap e985: 12 total, 11 up, 11 in 2022-09-08T05:05:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:10 smithi059 ceph-mon[97450]: osdmap e985: 12 total, 11 up, 11 in 2022-09-08T05:05:11.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:10.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:10.656243+0000) 2022-09-08T05:05:11.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:10.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:10.656232+0000) 2022-09-08T05:05:11.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:11 smithi055 ceph-mon[97033]: pgmap v14469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:05:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:11 smithi059 ceph-mon[97450]: pgmap v14469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:05:12.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:11.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:11.656483+0000) 2022-09-08T05:05:12.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:11.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:11.656446+0000) 2022-09-08T05:05:12.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:12 smithi055 ceph-mon[97033]: pgmap v14470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:05:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:12 smithi059 ceph-mon[97450]: pgmap v14470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:05:13.149 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:12.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:12.656657+0000) 2022-09-08T05:05:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:12.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:12.656642+0000) 2022-09-08T05:05:14.149 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:13.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:13.656887+0000) 2022-09-08T05:05:14.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:13.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:13.656897+0000) 2022-09-08T05:05:14.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:14 smithi055 ceph-mon[97033]: pgmap v14471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:05:15.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:14 smithi059 ceph-mon[97450]: pgmap v14471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:05:15.149 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:14.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:14.657129+0000) 2022-09-08T05:05:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:14.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:14.657124+0000) 2022-09-08T05:05:16.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:15.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:15.657322+0000) 2022-09-08T05:05:16.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:15.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:15.657321+0000) 2022-09-08T05:05:16.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:16 smithi055 ceph-mon[97033]: osdmap e986: 12 total, 11 up, 11 in 2022-09-08T05:05:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:16 smithi059 ceph-mon[97450]: osdmap e986: 12 total, 11 up, 11 in 2022-09-08T05:05:17.098 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:16.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:16.657553+0000) 2022-09-08T05:05:17.099 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:16.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:16.657566+0000) 2022-09-08T05:05:17.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:05:17.099+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:05:17.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:17 smithi055 ceph-mon[97033]: pgmap v14473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:05:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:17 smithi059 ceph-mon[97450]: pgmap v14473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:05:18.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:17.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:17.657753+0000) 2022-09-08T05:05:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:17.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:17.657720+0000) 2022-09-08T05:05:19.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:18.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:18.657955+0000) 2022-09-08T05:05:19.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:18.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:18.657956+0000) 2022-09-08T05:05:19.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:19 smithi055 ceph-mon[97033]: pgmap v14474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T05:05:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:19 smithi059 ceph-mon[97450]: pgmap v14474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T05:05:20.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:19.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:19.658196+0000) 2022-09-08T05:05:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:19.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:19.658195+0000) 2022-09-08T05:05:21.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:20.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:20.658442+0000) 2022-09-08T05:05:21.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:20.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:20.658449+0000) 2022-09-08T05:05:21.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:21 smithi055 ceph-mon[97033]: pgmap v14475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:05:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:21 smithi059 ceph-mon[97450]: pgmap v14475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:05:22.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:21.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:21.658723+0000) 2022-09-08T05:05:22.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:21.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:21.658725+0000) 2022-09-08T05:05:23.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:22.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:22.658947+0000) 2022-09-08T05:05:23.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:22.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:22.658950+0000) 2022-09-08T05:05:23.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:23 smithi055 ceph-mon[97033]: pgmap v14476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:05:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:23 smithi059 ceph-mon[97450]: pgmap v14476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:05:24.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:23.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:23.659098+0000) 2022-09-08T05:05:24.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:23.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:23.659189+0000) 2022-09-08T05:05:25.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:24.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:24.659286+0000) 2022-09-08T05:05:25.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:24.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:24.659372+0000) 2022-09-08T05:05:25.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:25 smithi055 ceph-mon[97033]: pgmap v14477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:05:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:25 smithi059 ceph-mon[97450]: pgmap v14477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:05:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:25.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:25.659433+0000) 2022-09-08T05:05:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:25.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:25.659561+0000) 2022-09-08T05:05:27.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:26.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:26.659664+0000) 2022-09-08T05:05:27.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:26.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:26.659759+0000) 2022-09-08T05:05:27.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:27 smithi055 ceph-mon[97033]: pgmap v14478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:05:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:27 smithi059 ceph-mon[97450]: pgmap v14478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:05:28.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:27.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:27.659862+0000) 2022-09-08T05:05:28.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:27.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:27.660026+0000) 2022-09-08T05:05:29.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:28.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:28.660059+0000) 2022-09-08T05:05:29.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:28.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:28.660295+0000) 2022-09-08T05:05:29.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:29 smithi055 ceph-mon[97033]: pgmap v14479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:29 smithi059 ceph-mon[97450]: pgmap v14479: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:30.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:29.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:29.660252+0000) 2022-09-08T05:05:30.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:29.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:29.660460+0000) 2022-09-08T05:05:31.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:30.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:30.660483+0000) 2022-09-08T05:05:31.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:30.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:30.660706+0000) 2022-09-08T05:05:31.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:31 smithi055 ceph-mon[97033]: pgmap v14480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:31 smithi059 ceph-mon[97450]: pgmap v14480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:32.103 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:31.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:31.660738+0000) 2022-09-08T05:05:32.104 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:31.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:31.660858+0000) 2022-09-08T05:05:32.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:05:32.104+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:05:33.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:32.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:32.660967+0000) 2022-09-08T05:05:33.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:32.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:32.661067+0000) 2022-09-08T05:05:33.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:33 smithi055 ceph-mon[97033]: pgmap v14481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:05:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:33 smithi059 ceph-mon[97450]: pgmap v14481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:05:34.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:33.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:33.661295+0000) 2022-09-08T05:05:34.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:33.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:33.661270+0000) 2022-09-08T05:05:35.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:34.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:34.661500+0000) 2022-09-08T05:05:35.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:34.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:34.661439+0000) 2022-09-08T05:05:35.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:35 smithi055 ceph-mon[97033]: pgmap v14482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:35 smithi059 ceph-mon[97450]: pgmap v14482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:36.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:35.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:35.661670+0000) 2022-09-08T05:05:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:35.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:35.661651+0000) 2022-09-08T05:05:37.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:36.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:36.661818+0000) 2022-09-08T05:05:37.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:36.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:36.661796+0000) 2022-09-08T05:05:37.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:37 smithi055 ceph-mon[97033]: pgmap v14483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:05:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:37 smithi059 ceph-mon[97450]: pgmap v14483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:05:38.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:37.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:37.662070+0000) 2022-09-08T05:05:38.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:37.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:37.662070+0000) 2022-09-08T05:05:39.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:38.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:38.662349+0000) 2022-09-08T05:05:39.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:38.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:38.662330+0000) 2022-09-08T05:05:39.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:39 smithi055 ceph-mon[97033]: pgmap v14484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:39 smithi059 ceph-mon[97450]: pgmap v14484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:40.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:39.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:39.662503+0000) 2022-09-08T05:05:40.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:39.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:39.662509+0000) 2022-09-08T05:05:41.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:40.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:40.662743+0000) 2022-09-08T05:05:41.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:40.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:40.662766+0000) 2022-09-08T05:05:41.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:41 smithi055 ceph-mon[97033]: pgmap v14485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:41 smithi059 ceph-mon[97450]: pgmap v14485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:41.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:41.662882+0000) 2022-09-08T05:05:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:41.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:41.662999+0000) 2022-09-08T05:05:43.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:42.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:42.663009+0000) 2022-09-08T05:05:43.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:42.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:42.663142+0000) 2022-09-08T05:05:43.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:43 smithi055 ceph-mon[97033]: pgmap v14486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:05:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:43 smithi059 ceph-mon[97450]: pgmap v14486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:05:44.149 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:43.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:43.663326+0000) 2022-09-08T05:05:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:43.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:43.663172+0000) 2022-09-08T05:05:45.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:44.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:44.663374+0000) 2022-09-08T05:05:45.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:44.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:44.663532+0000) 2022-09-08T05:05:45.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:45 smithi055 ceph-mon[97033]: pgmap v14487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:45 smithi059 ceph-mon[97450]: pgmap v14487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:46.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:45.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:45.663628+0000) 2022-09-08T05:05:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:45.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:45.663753+0000) 2022-09-08T05:05:47.109 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:46.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:46.663895+0000) 2022-09-08T05:05:47.109 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:46.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:46.663914+0000) 2022-09-08T05:05:47.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:05:47.109+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:05:47.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:47 smithi055 ceph-mon[97033]: pgmap v14488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:05:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:47 smithi059 ceph-mon[97450]: pgmap v14488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:05:48.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:47.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:47.664129+0000) 2022-09-08T05:05:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:47.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:47.664117+0000) 2022-09-08T05:05:49.149 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:48.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:48.664405+0000) 2022-09-08T05:05:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:48.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:48.664385+0000) 2022-09-08T05:05:49.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:49 smithi055 ceph-mon[97033]: pgmap v14489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:49 smithi059 ceph-mon[97450]: pgmap v14489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:50.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:49.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:49.664686+0000) 2022-09-08T05:05:50.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:49.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:49.664736+0000) 2022-09-08T05:05:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:50.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:50.664886+0000) 2022-09-08T05:05:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:50.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:50.664901+0000) 2022-09-08T05:05:51.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:51 smithi055 ceph-mon[97033]: pgmap v14490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:51 smithi059 ceph-mon[97450]: pgmap v14490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:52.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:51.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:51.665114+0000) 2022-09-08T05:05:52.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:51.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:51.665093+0000) 2022-09-08T05:05:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:52.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:52.665362+0000) 2022-09-08T05:05:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:52.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:52.665318+0000) 2022-09-08T05:05:53.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:53 smithi055 ceph-mon[97033]: pgmap v14491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:05:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:53 smithi059 ceph-mon[97450]: pgmap v14491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:05:54.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:53.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:53.665524+0000) 2022-09-08T05:05:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:53.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:53.665477+0000) 2022-09-08T05:05:55.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:54.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:54.665764+0000) 2022-09-08T05:05:55.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:54.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:54.665715+0000) 2022-09-08T05:05:55.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:55 smithi055 ceph-mon[97033]: pgmap v14492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:55 smithi059 ceph-mon[97450]: pgmap v14492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:56.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:55.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:55.666023+0000) 2022-09-08T05:05:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:55.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:55.665923+0000) 2022-09-08T05:05:56.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:05:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:05:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:05:56.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:05:56.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:05:56.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:05:57.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:56.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:56.666217+0000) 2022-09-08T05:05:57.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:56.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:56.666194+0000) 2022-09-08T05:05:57.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:57 smithi055 ceph-mon[97033]: pgmap v14493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:05:57.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:05:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:57 smithi059 ceph-mon[97450]: pgmap v14493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:05:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:05:58.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:57.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:57.666445+0000) 2022-09-08T05:05:58.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:57.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:57.666450+0000) 2022-09-08T05:05:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:58.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:58.666667+0000) 2022-09-08T05:05:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:58.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:58.666656+0000) 2022-09-08T05:05:59.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:05:59 smithi055 ceph-mon[97033]: pgmap v14494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:05:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:05:59 smithi059 ceph-mon[97450]: pgmap v14494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:00.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:05:59.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:59.666920+0000) 2022-09-08T05:06:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:05:59.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:05:59.666889+0000) 2022-09-08T05:06:01.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:00.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:00.667118+0000) 2022-09-08T05:06:01.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:00.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:00.667112+0000) 2022-09-08T05:06:01.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:01 smithi055 ceph-mon[97033]: pgmap v14495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:01 smithi059 ceph-mon[97450]: pgmap v14495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:02.122 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:01.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:01.667356+0000) 2022-09-08T05:06:02.123 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:01.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:01.667335+0000) 2022-09-08T05:06:02.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:06:02.122+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:06:02.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:02 smithi055 ceph-mon[97033]: pgmap v14496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:06:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:02 smithi059 ceph-mon[97450]: pgmap v14496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:06:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:02.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:02.667561+0000) 2022-09-08T05:06:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:02.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:02.667566+0000) 2022-09-08T05:06:04.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:03.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:03.667804+0000) 2022-09-08T05:06:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:03.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:03.667807+0000) 2022-09-08T05:06:04.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:04 smithi055 ceph-mon[97033]: pgmap v14497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:05.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:04 smithi059 ceph-mon[97450]: pgmap v14497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:05.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:04.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:04.668072+0000) 2022-09-08T05:06:05.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:04.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:04.668060+0000) 2022-09-08T05:06:06.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:05.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:05.668337+0000) 2022-09-08T05:06:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:05.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:05.668363+0000) 2022-09-08T05:06:06.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:06:06.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:06:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:06:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:06:07.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:06.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:06.668625+0000) 2022-09-08T05:06:07.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:06.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:06.668704+0000) 2022-09-08T05:06:07.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:07 smithi055 ceph-mon[97033]: pgmap v14498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:06:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:07 smithi059 ceph-mon[97450]: pgmap v14498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:06:08.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:07.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:07.668879+0000) 2022-09-08T05:06:08.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:07.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:07.668943+0000) 2022-09-08T05:06:09.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:08.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:08.669123+0000) 2022-09-08T05:06:09.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:08.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:08.669147+0000) 2022-09-08T05:06:09.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:09 smithi055 ceph-mon[97033]: pgmap v14499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:06:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:09 smithi059 ceph-mon[97450]: pgmap v14499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:06:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:09.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:09.669367+0000) 2022-09-08T05:06:10.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:09.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:09.669399+0000) 2022-09-08T05:06:10.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:10 smithi055 ceph-mon[97033]: osdmap e987: 12 total, 11 up, 11 in 2022-09-08T05:06:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:10 smithi059 ceph-mon[97450]: osdmap e987: 12 total, 11 up, 11 in 2022-09-08T05:06:11.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:10.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:10.669599+0000) 2022-09-08T05:06:11.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:10.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:10.669598+0000) 2022-09-08T05:06:11.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:11 smithi055 ceph-mon[97033]: pgmap v14501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:06:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:11 smithi059 ceph-mon[97450]: pgmap v14501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:06:12.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:11.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:11.669801+0000) 2022-09-08T05:06:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:11.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:11.669859+0000) 2022-09-08T05:06:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:12.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:12.670043+0000) 2022-09-08T05:06:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:12.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:12.670160+0000) 2022-09-08T05:06:13.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:13 smithi055 ceph-mon[97033]: pgmap v14502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:06:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:13 smithi059 ceph-mon[97450]: pgmap v14502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:06:14.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:13.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:13.670220+0000) 2022-09-08T05:06:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:13.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:13.670358+0000) 2022-09-08T05:06:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:14.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:14.670420+0000) 2022-09-08T05:06:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:14.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:14.670506+0000) 2022-09-08T05:06:15.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:15 smithi055 ceph-mon[97033]: pgmap v14503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:06:15.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:15 smithi055 ceph-mon[97033]: osdmap e988: 12 total, 11 up, 11 in 2022-09-08T05:06:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:15 smithi059 ceph-mon[97450]: pgmap v14503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:06:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:15 smithi059 ceph-mon[97450]: osdmap e988: 12 total, 11 up, 11 in 2022-09-08T05:06:16.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:15.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:15.670607+0000) 2022-09-08T05:06:16.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:15.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:15.670716+0000) 2022-09-08T05:06:17.138 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:16.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:16.670813+0000) 2022-09-08T05:06:17.139 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:16.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:16.670903+0000) 2022-09-08T05:06:17.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:06:17.139+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:06:17.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:17 smithi055 ceph-mon[97033]: pgmap v14505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:06:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:17 smithi059 ceph-mon[97450]: pgmap v14505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:06:18.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:17.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:17.671037+0000) 2022-09-08T05:06:18.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:17.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:17.671120+0000) 2022-09-08T05:06:19.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:18.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:18.671279+0000) 2022-09-08T05:06:19.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:18.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:18.671318+0000) 2022-09-08T05:06:19.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:19 smithi055 ceph-mon[97033]: pgmap v14506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T05:06:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:19 smithi059 ceph-mon[97450]: pgmap v14506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T05:06:20.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:19.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:19.671503+0000) 2022-09-08T05:06:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:19.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:19.671507+0000) 2022-09-08T05:06:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:20.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:20.671742+0000) 2022-09-08T05:06:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:20.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:20.671779+0000) 2022-09-08T05:06:21.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:21 smithi055 ceph-mon[97033]: pgmap v14507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:06:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:21 smithi059 ceph-mon[97450]: pgmap v14507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:06:22.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:21.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:21.671991+0000) 2022-09-08T05:06:22.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:21.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:21.672014+0000) 2022-09-08T05:06:23.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:22.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:22.672248+0000) 2022-09-08T05:06:23.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:22.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:22.672280+0000) 2022-09-08T05:06:23.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:23 smithi055 ceph-mon[97033]: pgmap v14508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:06:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:23 smithi059 ceph-mon[97450]: pgmap v14508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:06:24.149 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:23.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:23.672581+0000) 2022-09-08T05:06:24.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:23.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:23.672584+0000) 2022-09-08T05:06:25.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:24.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:24.672759+0000) 2022-09-08T05:06:25.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:24.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:24.672775+0000) 2022-09-08T05:06:25.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:25 smithi055 ceph-mon[97033]: pgmap v14509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:06:25.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:25 smithi059 ceph-mon[97450]: pgmap v14509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:06:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:25.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:25.673034+0000) 2022-09-08T05:06:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:25.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:25.673033+0000) 2022-09-08T05:06:27.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:26.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:26.673303+0000) 2022-09-08T05:06:27.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:26.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:26.673299+0000) 2022-09-08T05:06:27.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:27 smithi055 ceph-mon[97033]: pgmap v14510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:06:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:27 smithi059 ceph-mon[97450]: pgmap v14510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:06:28.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:27.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:27.673569+0000) 2022-09-08T05:06:28.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:27.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:27.673570+0000) 2022-09-08T05:06:29.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:28.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:28.673767+0000) 2022-09-08T05:06:29.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:28.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:28.673767+0000) 2022-09-08T05:06:29.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:29 smithi055 ceph-mon[97033]: pgmap v14511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:29 smithi059 ceph-mon[97450]: pgmap v14511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:30.149 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:29.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:29.673931+0000) 2022-09-08T05:06:30.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:29.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:29.673918+0000) 2022-09-08T05:06:31.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:30.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:30.674186+0000) 2022-09-08T05:06:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:30.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:30.674182+0000) 2022-09-08T05:06:31.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:31 smithi055 ceph-mon[97033]: pgmap v14512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:31 smithi059 ceph-mon[97450]: pgmap v14512: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:32.146 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:31.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:31.674442+0000) 2022-09-08T05:06:32.147 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:31.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:31.674441+0000) 2022-09-08T05:06:32.399 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:06:32.147+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:06:33.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:32.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:32.674689+0000) 2022-09-08T05:06:33.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:32.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:32.674687+0000) 2022-09-08T05:06:33.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:33 smithi055 ceph-mon[97033]: pgmap v14513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:06:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:33 smithi059 ceph-mon[97450]: pgmap v14513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:06:34.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:33.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:33.674940+0000) 2022-09-08T05:06:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:33.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:33.674939+0000) 2022-09-08T05:06:35.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:34.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:34.675196+0000) 2022-09-08T05:06:35.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:34.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:34.675176+0000) 2022-09-08T05:06:35.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:35 smithi055 ceph-mon[97033]: pgmap v14514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:35 smithi059 ceph-mon[97450]: pgmap v14514: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:36.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:35.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:35.675479+0000) 2022-09-08T05:06:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:35.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:35.675492+0000) 2022-09-08T05:06:37.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:36.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:36.675758+0000) 2022-09-08T05:06:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:36.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:36.675758+0000) 2022-09-08T05:06:37.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:37 smithi055 ceph-mon[97033]: pgmap v14515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:06:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:37 smithi059 ceph-mon[97450]: pgmap v14515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:06:38.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:37.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:37.675973+0000) 2022-09-08T05:06:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:37.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:37.675973+0000) 2022-09-08T05:06:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:38.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:38.676161+0000) 2022-09-08T05:06:39.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:38.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:38.676151+0000) 2022-09-08T05:06:39.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:39 smithi055 ceph-mon[97033]: pgmap v14516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:39 smithi059 ceph-mon[97450]: pgmap v14516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:40.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:39.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:39.676407+0000) 2022-09-08T05:06:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:39.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:39.676404+0000) 2022-09-08T05:06:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:40.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:40.676688+0000) 2022-09-08T05:06:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:40.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:40.676685+0000) 2022-09-08T05:06:41.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:41 smithi055 ceph-mon[97033]: pgmap v14517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:41 smithi059 ceph-mon[97450]: pgmap v14517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:42.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:41.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:41.676971+0000) 2022-09-08T05:06:42.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:41.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:41.676968+0000) 2022-09-08T05:06:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:42.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:42.677159+0000) 2022-09-08T05:06:43.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:42.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:42.677159+0000) 2022-09-08T05:06:43.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:43 smithi055 ceph-mon[97033]: pgmap v14518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:06:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:43 smithi059 ceph-mon[97450]: pgmap v14518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:06:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:43.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:43.677477+0000) 2022-09-08T05:06:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:43.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:43.677461+0000) 2022-09-08T05:06:45.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:44.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:44.677714+0000) 2022-09-08T05:06:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:44.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:44.677630+0000) 2022-09-08T05:06:45.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:45 smithi055 ceph-mon[97033]: pgmap v14519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:45 smithi059 ceph-mon[97450]: pgmap v14519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:46.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:45.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:45.677893+0000) 2022-09-08T05:06:46.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:45.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:45.677897+0000) 2022-09-08T05:06:47.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:46.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:46.678015+0000) 2022-09-08T05:06:47.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:46.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:46.678049+0000) 2022-09-08T05:06:47.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:06:47.150+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:06:47.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:47 smithi055 ceph-mon[97033]: pgmap v14520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:06:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:47 smithi059 ceph-mon[97450]: pgmap v14520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:06:48.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:47.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:47.678169+0000) 2022-09-08T05:06:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:47.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:47.678188+0000) 2022-09-08T05:06:49.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:48.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:48.678378+0000) 2022-09-08T05:06:49.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:48.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:48.678380+0000) 2022-09-08T05:06:49.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:49 smithi055 ceph-mon[97033]: pgmap v14521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:49 smithi059 ceph-mon[97450]: pgmap v14521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:50.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:49.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:49.678596+0000) 2022-09-08T05:06:50.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:49.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:49.678603+0000) 2022-09-08T05:06:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:50.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:50.678820+0000) 2022-09-08T05:06:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:50.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:50.678813+0000) 2022-09-08T05:06:51.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:51 smithi055 ceph-mon[97033]: pgmap v14522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:51 smithi059 ceph-mon[97450]: pgmap v14522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:52.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:51.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:51.679033+0000) 2022-09-08T05:06:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:51.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:51.679057+0000) 2022-09-08T05:06:53.149 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:52.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:52.679284+0000) 2022-09-08T05:06:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:52.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:52.679262+0000) 2022-09-08T05:06:53.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:53 smithi055 ceph-mon[97033]: pgmap v14523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:06:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:53 smithi059 ceph-mon[97450]: pgmap v14523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:06:54.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:53.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:53.679457+0000) 2022-09-08T05:06:54.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:53.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:53.679470+0000) 2022-09-08T05:06:55.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:54.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:54.679672+0000) 2022-09-08T05:06:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:54.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:54.679702+0000) 2022-09-08T05:06:55.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:55 smithi055 ceph-mon[97033]: pgmap v14524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:55.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:55 smithi059 ceph-mon[97450]: pgmap v14524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:56.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:55.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:55.679872+0000) 2022-09-08T05:06:56.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:55.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:55.679864+0000) 2022-09-08T05:06:56.935 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:56.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:56.680037+0000) 2022-09-08T05:06:56.937 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:56.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:56.680027+0000) 2022-09-08T05:06:57.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:57 smithi055 ceph-mon[97033]: pgmap v14525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:06:57.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:06:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:06:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:06:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:57 smithi059 ceph-mon[97450]: pgmap v14525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:06:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:06:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:06:57.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:06:58.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:57.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:57.680291+0000) 2022-09-08T05:06:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:57.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:57.680304+0000) 2022-09-08T05:06:59.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:58.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:58.680526+0000) 2022-09-08T05:06:59.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:58.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:58.680533+0000) 2022-09-08T05:06:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:06:59 smithi055 ceph-mon[97033]: pgmap v14526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:06:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:06:59 smithi059 ceph-mon[97450]: pgmap v14526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:00.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:06:59.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:59.680772+0000) 2022-09-08T05:07:00.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:06:59.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:06:59.680772+0000) 2022-09-08T05:07:01.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:00.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:00.680911+0000) 2022-09-08T05:07:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:00.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:00.680911+0000) 2022-09-08T05:07:01.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:01 smithi055 ceph-mon[97033]: pgmap v14527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:01.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:01 smithi059 ceph-mon[97450]: pgmap v14527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:02.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:01.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:01.681092+0000) 2022-09-08T05:07:02.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:01.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:01.681113+0000) 2022-09-08T05:07:02.649 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:07:02.166+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:07:03.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:02.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:02.681302+0000) 2022-09-08T05:07:03.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:02.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:02.681331+0000) 2022-09-08T05:07:03.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:03 smithi055 ceph-mon[97033]: pgmap v14528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:07:03.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:03 smithi059 ceph-mon[97450]: pgmap v14528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:07:04.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:03.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:03.681549+0000) 2022-09-08T05:07:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:03.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:03.681561+0000) 2022-09-08T05:07:05.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:04.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:04.681794+0000) 2022-09-08T05:07:05.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:04.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:04.681796+0000) 2022-09-08T05:07:05.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:05 smithi055 ceph-mon[97033]: pgmap v14529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:05 smithi059 ceph-mon[97450]: pgmap v14529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:06.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:05.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:05.681948+0000) 2022-09-08T05:07:06.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:05.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:05.681958+0000) 2022-09-08T05:07:06.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:07:06.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:07:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:07:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:07:07.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:06.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:06.682157+0000) 2022-09-08T05:07:07.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:06.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:06.682131+0000) 2022-09-08T05:07:07.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:07 smithi055 ceph-mon[97033]: pgmap v14530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:07:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:07 smithi059 ceph-mon[97450]: pgmap v14530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:07:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:07.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:07.682332+0000) 2022-09-08T05:07:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:07.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:07.682312+0000) 2022-09-08T05:07:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:08.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:08.682512+0000) 2022-09-08T05:07:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:08.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:08.682520+0000) 2022-09-08T05:07:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:09 smithi055 ceph-mon[97033]: pgmap v14531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:07:09.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:09 smithi059 ceph-mon[97450]: pgmap v14531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:07:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:09.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:09.682631+0000) 2022-09-08T05:07:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:09.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:09.682693+0000) 2022-09-08T05:07:10.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:10 smithi055 ceph-mon[97033]: osdmap e989: 12 total, 11 up, 11 in 2022-09-08T05:07:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:10 smithi059 ceph-mon[97450]: osdmap e989: 12 total, 11 up, 11 in 2022-09-08T05:07:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:10.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:10.682841+0000) 2022-09-08T05:07:11.180 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:10.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:10.682862+0000) 2022-09-08T05:07:11.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:11 smithi055 ceph-mon[97033]: pgmap v14533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:07:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:11 smithi059 ceph-mon[97450]: pgmap v14533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:07:12.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:11.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:11.683079+0000) 2022-09-08T05:07:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:11.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:11.683087+0000) 2022-09-08T05:07:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:12.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:12.683281+0000) 2022-09-08T05:07:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:12.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:12.683314+0000) 2022-09-08T05:07:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:13 smithi055 ceph-mon[97033]: pgmap v14534: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:07:13.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:13 smithi059 ceph-mon[97450]: pgmap v14534: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:07:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:13.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:13.683557+0000) 2022-09-08T05:07:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:13.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:13.683523+0000) 2022-09-08T05:07:15.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:14.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:14.683673+0000) 2022-09-08T05:07:15.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:14.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:14.683706+0000) 2022-09-08T05:07:15.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:15 smithi055 ceph-mon[97033]: pgmap v14535: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:07:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:15 smithi059 ceph-mon[97450]: pgmap v14535: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:07:16.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:15.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:15.683895+0000) 2022-09-08T05:07:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:15.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:15.683948+0000) 2022-09-08T05:07:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:16.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:16.684156+0000) 2022-09-08T05:07:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:16.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:16.684160+0000) 2022-09-08T05:07:17.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:07:17.178+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:07:17.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:17 smithi055 ceph-mon[97033]: pgmap v14536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:07:17.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:17 smithi059 ceph-mon[97450]: pgmap v14536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:07:18.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:17.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:17.684405+0000) 2022-09-08T05:07:18.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:17.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:17.684450+0000) 2022-09-08T05:07:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:18.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:18.684712+0000) 2022-09-08T05:07:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:18.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:18.684717+0000) 2022-09-08T05:07:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:19 smithi055 ceph-mon[97033]: pgmap v14537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:07:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:19 smithi059 ceph-mon[97450]: pgmap v14537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:07:20.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:19.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:19.684923+0000) 2022-09-08T05:07:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:19.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:19.684973+0000) 2022-09-08T05:07:21.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:20.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:20.685185+0000) 2022-09-08T05:07:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:20.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:20.685245+0000) 2022-09-08T05:07:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:21 smithi055 ceph-mon[97033]: pgmap v14538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 201 B/s wr, 6 op/s 2022-09-08T05:07:21.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:21 smithi055 ceph-mon[97033]: osdmap e990: 12 total, 11 up, 11 in 2022-09-08T05:07:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:21 smithi059 ceph-mon[97450]: pgmap v14538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 201 B/s wr, 6 op/s 2022-09-08T05:07:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:21 smithi059 ceph-mon[97450]: osdmap e990: 12 total, 11 up, 11 in 2022-09-08T05:07:22.021 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:21.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:21.685378+0000) 2022-09-08T05:07:22.021 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:21.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:21.685433+0000) 2022-09-08T05:07:23.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:22.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:22.685595+0000) 2022-09-08T05:07:23.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:22.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:22.685611+0000) 2022-09-08T05:07:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:23 smithi055 ceph-mon[97033]: pgmap v14540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T05:07:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:23 smithi059 ceph-mon[97450]: pgmap v14540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T05:07:24.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:23.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:23.685843+0000) 2022-09-08T05:07:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:23.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:23.685843+0000) 2022-09-08T05:07:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:24.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:24.686088+0000) 2022-09-08T05:07:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:24.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:24.686093+0000) 2022-09-08T05:07:25.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:25 smithi055 ceph-mon[97033]: pgmap v14541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T05:07:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:25 smithi059 ceph-mon[97450]: pgmap v14541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T05:07:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:25.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:25.686275+0000) 2022-09-08T05:07:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:25.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:25.686278+0000) 2022-09-08T05:07:26.983 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:26.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:26.686525+0000) 2022-09-08T05:07:26.984 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:26.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:26.686526+0000) 2022-09-08T05:07:27.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:27 smithi055 ceph-mon[97033]: pgmap v14542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:07:27.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:07:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:27 smithi059 ceph-mon[97450]: pgmap v14542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:07:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:07:28.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:27.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:27.686770+0000) 2022-09-08T05:07:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:27.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:27.686794+0000) 2022-09-08T05:07:28.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:28 smithi055 ceph-mon[97033]: pgmap v14543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-08T05:07:28.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:28 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T05:07:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:28 smithi059 ceph-mon[97450]: pgmap v14543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-08T05:07:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:28 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T05:07:29.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:28.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:28.687019+0000) 2022-09-08T05:07:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:28.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:28.687023+0000) 2022-09-08T05:07:30.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:29.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:29.687270+0000) 2022-09-08T05:07:30.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:29.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:29.687254+0000) 2022-09-08T05:07:30.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:30 smithi055 ceph-mon[97033]: pgmap v14544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-08T05:07:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:30 smithi059 ceph-mon[97450]: pgmap v14544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-08T05:07:31.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:30.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:30.687456+0000) 2022-09-08T05:07:31.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:30.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:30.687388+0000) 2022-09-08T05:07:32.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:31.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:31.687682+0000) 2022-09-08T05:07:32.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:31.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:31.687613+0000) 2022-09-08T05:07:32.649 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:07:32.186+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:07:32.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:32 smithi055 ceph-mon[97033]: pgmap v14545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:07:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:32 smithi059 ceph-mon[97450]: pgmap v14545: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:07:33.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:32.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:32.687868+0000) 2022-09-08T05:07:33.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:32.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:32.687880+0000) 2022-09-08T05:07:34.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:33.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:33.688099+0000) 2022-09-08T05:07:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:33.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:33.688061+0000) 2022-09-08T05:07:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:34 smithi055 ceph-mon[97033]: pgmap v14546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:07:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:34 smithi059 ceph-mon[97450]: pgmap v14546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:07:35.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:34.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:34.688237+0000) 2022-09-08T05:07:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:34.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:34.688261+0000) 2022-09-08T05:07:36.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:35.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:35.688501+0000) 2022-09-08T05:07:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:35.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:35.688446+0000) 2022-09-08T05:07:36.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:36 smithi055 ceph-mon[97033]: pgmap v14547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T05:07:36.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:36 smithi059 ceph-mon[97450]: pgmap v14547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T05:07:37.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:36.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:36.688689+0000) 2022-09-08T05:07:37.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:36.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:36.688655+0000) 2022-09-08T05:07:38.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:37.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:37.688923+0000) 2022-09-08T05:07:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:37.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:37.688884+0000) 2022-09-08T05:07:38.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:38 smithi055 ceph-mon[97033]: pgmap v14548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:07:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:38 smithi059 ceph-mon[97450]: pgmap v14548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:07:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:38.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:38.689130+0000) 2022-09-08T05:07:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:38.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:38.689069+0000) 2022-09-08T05:07:40.167 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:39.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:39.689254+0000) 2022-09-08T05:07:40.168 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:39.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:39.689222+0000) 2022-09-08T05:07:40.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:40 smithi055 ceph-mon[97033]: pgmap v14549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:40 smithi059 ceph-mon[97450]: pgmap v14549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:40.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:40.689437+0000) 2022-09-08T05:07:41.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:40.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:40.689431+0000) 2022-09-08T05:07:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:41.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:41.689646+0000) 2022-09-08T05:07:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:41.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:41.689676+0000) 2022-09-08T05:07:42.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:42 smithi055 ceph-mon[97033]: pgmap v14550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:07:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:42 smithi059 ceph-mon[97450]: pgmap v14550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:07:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:42.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:42.689880+0000) 2022-09-08T05:07:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:42.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:42.689889+0000) 2022-09-08T05:07:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:43.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:43.690069+0000) 2022-09-08T05:07:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:43.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:43.690073+0000) 2022-09-08T05:07:44.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:44 smithi055 ceph-mon[97033]: pgmap v14551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:44 smithi059 ceph-mon[97450]: pgmap v14551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:45.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:44.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:44.690255+0000) 2022-09-08T05:07:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:44.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:44.690255+0000) 2022-09-08T05:07:46.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:45.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:45.690438+0000) 2022-09-08T05:07:46.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:45.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:45.690440+0000) 2022-09-08T05:07:46.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:46 smithi055 ceph-mon[97033]: pgmap v14552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:07:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:46 smithi059 ceph-mon[97450]: pgmap v14552: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:07:47.040 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:46.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:46.690659+0000) 2022-09-08T05:07:47.041 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:46.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:46.690623+0000) 2022-09-08T05:07:47.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:07:47.197+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:07:48.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:47.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:47.690805+0000) 2022-09-08T05:07:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:47.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:47.690807+0000) 2022-09-08T05:07:48.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:48 smithi055 ceph-mon[97033]: pgmap v14553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:48 smithi059 ceph-mon[97450]: pgmap v14553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:48.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:48.690982+0000) 2022-09-08T05:07:49.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:48.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:48.690968+0000) 2022-09-08T05:07:50.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:49.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:49.691214+0000) 2022-09-08T05:07:50.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:49.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:49.691202+0000) 2022-09-08T05:07:50.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:50 smithi055 ceph-mon[97033]: pgmap v14554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:50 smithi059 ceph-mon[97450]: pgmap v14554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:50.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:50.691443+0000) 2022-09-08T05:07:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:50.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:50.691422+0000) 2022-09-08T05:07:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:51.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:51.691609+0000) 2022-09-08T05:07:52.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:51.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:51.691613+0000) 2022-09-08T05:07:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:52 smithi055 ceph-mon[97033]: pgmap v14555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:07:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:52 smithi059 ceph-mon[97450]: pgmap v14555: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:07:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:52.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:52.691804+0000) 2022-09-08T05:07:53.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:52.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:52.691793+0000) 2022-09-08T05:07:54.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:53.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:53.692022+0000) 2022-09-08T05:07:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:53.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:53.691976+0000) 2022-09-08T05:07:54.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:54 smithi055 ceph-mon[97033]: pgmap v14556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:54 smithi059 ceph-mon[97450]: pgmap v14556: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:54.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:54.692283+0000) 2022-09-08T05:07:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:54.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:54.692192+0000) 2022-09-08T05:07:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:55.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:55.692485+0000) 2022-09-08T05:07:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:55.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:55.692435+0000) 2022-09-08T05:07:56.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:56 smithi055 ceph-mon[97033]: pgmap v14557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:07:56.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:56 smithi059 ceph-mon[97450]: pgmap v14557: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:07:57.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:56.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:56.692631+0000) 2022-09-08T05:07:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:56.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:56.692610+0000) 2022-09-08T05:07:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:57.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:57.692865+0000) 2022-09-08T05:07:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:57.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:57.692822+0000) 2022-09-08T05:07:58.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:07:58 smithi055 ceph-mon[97033]: pgmap v14558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:07:58 smithi059 ceph-mon[97450]: pgmap v14558: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:07:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:58.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:58.693092+0000) 2022-09-08T05:07:59.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:58.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:58.693037+0000) 2022-09-08T05:08:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:07:59.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:59.693333+0000) 2022-09-08T05:08:00.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:07:59.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:07:59.693281+0000) 2022-09-08T05:08:00.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:00 smithi055 ceph-mon[97033]: pgmap v14559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:00.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:00 smithi059 ceph-mon[97450]: pgmap v14559: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:00.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:00.693525+0000) 2022-09-08T05:08:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:00.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:00.693465+0000) 2022-09-08T05:08:02.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:01.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:01.693741+0000) 2022-09-08T05:08:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:01.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:01.693723+0000) 2022-09-08T05:08:02.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:08:02.204+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:08:02.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:02 smithi055 ceph-mon[97033]: pgmap v14560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:08:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:02 smithi059 ceph-mon[97450]: pgmap v14560: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:08:03.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:02.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:02.693966+0000) 2022-09-08T05:08:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:02.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:02.693991+0000) 2022-09-08T05:08:03.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:03 smithi055 ceph-mon[97033]: pgmap v14561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:03 smithi059 ceph-mon[97450]: pgmap v14561: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:04.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:03.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:03.694153+0000) 2022-09-08T05:08:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:03.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:03.694192+0000) 2022-09-08T05:08:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:04.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:04.694360+0000) 2022-09-08T05:08:05.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:04.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:04.694390+0000) 2022-09-08T05:08:06.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:05.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:05.694562+0000) 2022-09-08T05:08:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:05.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:05.694573+0000) 2022-09-08T05:08:06.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:06 smithi055 ceph-mon[97033]: pgmap v14562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:08:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:08:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:08:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:06 smithi059 ceph-mon[97450]: pgmap v14562: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:08:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:08:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:08:07.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:06.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:06.694816+0000) 2022-09-08T05:08:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:06.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:06.694782+0000) 2022-09-08T05:08:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:07.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:07.695029+0000) 2022-09-08T05:08:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:07.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:07.694996+0000) 2022-09-08T05:08:08.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:08 smithi055 ceph-mon[97033]: pgmap v14563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:08:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:08 smithi059 ceph-mon[97450]: pgmap v14563: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:08:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:08.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:08.695238+0000) 2022-09-08T05:08:09.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:08.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:08.695256+0000) 2022-09-08T05:08:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:09.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:09.695510+0000) 2022-09-08T05:08:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:09.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:09.695518+0000) 2022-09-08T05:08:10.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:10 smithi055 ceph-mon[97033]: pgmap v14564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:08:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:10 smithi055 ceph-mon[97033]: osdmap e991: 12 total, 11 up, 11 in 2022-09-08T05:08:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:10 smithi059 ceph-mon[97450]: pgmap v14564: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:08:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:10 smithi059 ceph-mon[97450]: osdmap e991: 12 total, 11 up, 11 in 2022-09-08T05:08:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:10.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:10.695752+0000) 2022-09-08T05:08:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:10.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:10.695741+0000) 2022-09-08T05:08:12.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:11.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:11.696000+0000) 2022-09-08T05:08:12.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:11.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:11.695973+0000) 2022-09-08T05:08:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:12 smithi055 ceph-mon[97033]: pgmap v14566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:08:12.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:12 smithi059 ceph-mon[97450]: pgmap v14566: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:08:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:12.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:12.696160+0000) 2022-09-08T05:08:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:12.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:12.696203+0000) 2022-09-08T05:08:14.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:13.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:13.696343+0000) 2022-09-08T05:08:14.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:13.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:13.696359+0000) 2022-09-08T05:08:14.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:14 smithi055 ceph-mon[97033]: pgmap v14567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:08:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:14 smithi059 ceph-mon[97450]: pgmap v14567: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:08:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:14.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:14.696605+0000) 2022-09-08T05:08:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:14.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:14.696588+0000) 2022-09-08T05:08:16.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:15.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:15.696843+0000) 2022-09-08T05:08:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:15.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:15.696811+0000) 2022-09-08T05:08:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:16 smithi055 ceph-mon[97033]: pgmap v14568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:08:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:16 smithi059 ceph-mon[97450]: pgmap v14568: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:08:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:16.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:16.697092+0000) 2022-09-08T05:08:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:16.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:16.697001+0000) 2022-09-08T05:08:17.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:08:17.214+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:08:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:17.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:17.697220+0000) 2022-09-08T05:08:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:17.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:17.697294+0000) 2022-09-08T05:08:18.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:18 smithi055 ceph-mon[97033]: pgmap v14569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:08:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:18 smithi059 ceph-mon[97450]: pgmap v14569: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:08:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:18.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:18.697518+0000) 2022-09-08T05:08:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:18.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:18.697453+0000) 2022-09-08T05:08:20.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:19.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:19.697694+0000) 2022-09-08T05:08:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:19.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:19.697668+0000) 2022-09-08T05:08:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:20 smithi055 ceph-mon[97033]: pgmap v14570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:08:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:20 smithi055 ceph-mon[97033]: osdmap e992: 12 total, 11 up, 11 in 2022-09-08T05:08:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:20 smithi059 ceph-mon[97450]: pgmap v14570: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:08:20.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:20 smithi059 ceph-mon[97450]: osdmap e992: 12 total, 11 up, 11 in 2022-09-08T05:08:21.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:20.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:20.697808+0000) 2022-09-08T05:08:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:20.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:20.697860+0000) 2022-09-08T05:08:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:21.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:21.698034+0000) 2022-09-08T05:08:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:21.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:21.698095+0000) 2022-09-08T05:08:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:22 smithi055 ceph-mon[97033]: pgmap v14572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:08:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:22 smithi059 ceph-mon[97450]: pgmap v14572: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:08:23.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:22.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:22.698221+0000) 2022-09-08T05:08:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:22.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:22.698287+0000) 2022-09-08T05:08:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:23.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:23.698537+0000) 2022-09-08T05:08:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:23.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:23.698444+0000) 2022-09-08T05:08:24.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:24 smithi055 ceph-mon[97033]: pgmap v14573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:08:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:24 smithi059 ceph-mon[97450]: pgmap v14573: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:08:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:24.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:24.698728+0000) 2022-09-08T05:08:25.163 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:24.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:24.698682+0000) 2022-09-08T05:08:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:25.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:25.698895+0000) 2022-09-08T05:08:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:25.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:25.698881+0000) 2022-09-08T05:08:26.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:26 smithi055 ceph-mon[97033]: pgmap v14574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:08:26.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:26 smithi059 ceph-mon[97450]: pgmap v14574: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:08:27.133 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:26.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:26.699089+0000) 2022-09-08T05:08:27.134 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:26.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:26.699069+0000) 2022-09-08T05:08:27.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:08:27.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:08:27.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:08:27.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:08:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:08:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:08:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:27.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:27.699288+0000) 2022-09-08T05:08:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:27.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:27.699284+0000) 2022-09-08T05:08:28.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:28 smithi055 ceph-mon[97033]: pgmap v14575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:08:28.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:08:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:08:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:28 smithi059 ceph-mon[97450]: pgmap v14575: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:08:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:08:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:08:29.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:28.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:28.699530+0000) 2022-09-08T05:08:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:28.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:28.699517+0000) 2022-09-08T05:08:30.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:29.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:29.699773+0000) 2022-09-08T05:08:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:29.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:29.699794+0000) 2022-09-08T05:08:30.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:30 smithi055 ceph-mon[97033]: pgmap v14576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:08:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:08:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:08:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:08:30.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:08:30.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:30.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:30.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:30.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:30 smithi059 ceph-mon[97450]: pgmap v14576: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:08:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:08:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:08:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:08:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:08:30.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:30.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:30.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:30.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:31.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:30.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:30.700103+0000) 2022-09-08T05:08:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:30.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:30.700082+0000) 2022-09-08T05:08:31.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:31 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T05:08:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:31 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T05:08:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:31 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T05:08:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:31 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T05:08:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:31.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:08:32.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:31.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:31.700336+0000) 2022-09-08T05:08:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:31.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:31.700365+0000) 2022-09-08T05:08:32.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:08:32.230+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:08:32.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:32 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T05:08:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:32 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T05:08:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:32 smithi055 ceph-mon[97033]: pgmap v14577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:08:32.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:32 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T05:08:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:32 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T05:08:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:32 smithi059 ceph-mon[97450]: pgmap v14577: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:08:33.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:32.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:32.700561+0000) 2022-09-08T05:08:33.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:32.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:32.700613+0000) 2022-09-08T05:08:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:33.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:33.700914+0000) 2022-09-08T05:08:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:33.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:33.700829+0000) 2022-09-08T05:08:34.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:34 smithi055 ceph-mon[97033]: pgmap v14578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:34 smithi059 ceph-mon[97450]: pgmap v14578: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:35.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:34.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:34.701098+0000) 2022-09-08T05:08:35.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:34.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:34.701034+0000) 2022-09-08T05:08:36.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:35.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:35.701278+0000) 2022-09-08T05:08:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:35.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:35.701238+0000) 2022-09-08T05:08:36.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:36 smithi055 ceph-mon[97033]: pgmap v14579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:08:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:36 smithi059 ceph-mon[97450]: pgmap v14579: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:08:37.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:36.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:36.701470+0000) 2022-09-08T05:08:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:36.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:36.701528+0000) 2022-09-08T05:08:37.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:37 smithi055 ceph-mon[97033]: pgmap v14580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:37 smithi059 ceph-mon[97450]: pgmap v14580: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:38.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:37.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:37.701725+0000) 2022-09-08T05:08:38.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:37.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:37.701740+0000) 2022-09-08T05:08:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:38.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:38.701931+0000) 2022-09-08T05:08:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:38.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:38.701922+0000) 2022-09-08T05:08:40.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:39.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:39.702158+0000) 2022-09-08T05:08:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:39.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:39.702150+0000) 2022-09-08T05:08:40.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:40 smithi055 ceph-mon[97033]: pgmap v14581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:40.595 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:40 smithi059 ceph-mon[97450]: pgmap v14581: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:40.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:40.702342+0000) 2022-09-08T05:08:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:40.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:40.702345+0000) 2022-09-08T05:08:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:41.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:41.702606+0000) 2022-09-08T05:08:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:41.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:41.702598+0000) 2022-09-08T05:08:42.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:42 smithi055 ceph-mon[97033]: pgmap v14582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:08:42.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:42 smithi059 ceph-mon[97450]: pgmap v14582: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:08:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:42.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:42.702916+0000) 2022-09-08T05:08:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:42.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:42.702895+0000) 2022-09-08T05:08:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:43.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:43.703185+0000) 2022-09-08T05:08:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:43.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:43.703136+0000) 2022-09-08T05:08:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:44 smithi055 ceph-mon[97033]: pgmap v14583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:44 smithi059 ceph-mon[97450]: pgmap v14583: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:45.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:44.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:44.703419+0000) 2022-09-08T05:08:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:44.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:44.703338+0000) 2022-09-08T05:08:46.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:45.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:45.703712+0000) 2022-09-08T05:08:46.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:45.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:45.703586+0000) 2022-09-08T05:08:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:46 smithi055 ceph-mon[97033]: pgmap v14584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:08:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:46 smithi059 ceph-mon[97450]: pgmap v14584: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:08:47.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:46.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:46.703983+0000) 2022-09-08T05:08:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:46.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:46.703881+0000) 2022-09-08T05:08:47.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:08:47.232+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:08:48.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:47.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:47.704221+0000) 2022-09-08T05:08:48.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:47.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:47.704140+0000) 2022-09-08T05:08:48.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:48 smithi055 ceph-mon[97033]: pgmap v14585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:48 smithi059 ceph-mon[97450]: pgmap v14585: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:48.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:48.704450+0000) 2022-09-08T05:08:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:48.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:48.704389+0000) 2022-09-08T05:08:50.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:49.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:49.704615+0000) 2022-09-08T05:08:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:49.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:49.704573+0000) 2022-09-08T05:08:50.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:50 smithi055 ceph-mon[97033]: pgmap v14586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:50.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:50 smithi059 ceph-mon[97450]: pgmap v14586: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:50.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:50.704794+0000) 2022-09-08T05:08:51.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:50.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:50.704691+0000) 2022-09-08T05:08:52.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:51.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:51.704964+0000) 2022-09-08T05:08:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:51.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:51.704861+0000) 2022-09-08T05:08:52.739 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:52 smithi055 ceph-mon[97033]: pgmap v14587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:08:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:52 smithi059 ceph-mon[97450]: pgmap v14587: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:08:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:52.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:52.705151+0000) 2022-09-08T05:08:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:52.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:52.705077+0000) 2022-09-08T05:08:54.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:53.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:53.705361+0000) 2022-09-08T05:08:54.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:53.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:53.705291+0000) 2022-09-08T05:08:54.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:54 smithi055 ceph-mon[97033]: pgmap v14588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:54.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:54 smithi059 ceph-mon[97450]: pgmap v14588: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:55.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:54.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:54.705535+0000) 2022-09-08T05:08:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:54.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:54.705457+0000) 2022-09-08T05:08:56.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:55.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:55.705767+0000) 2022-09-08T05:08:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:55.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:55.705706+0000) 2022-09-08T05:08:56.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:56 smithi055 ceph-mon[97033]: pgmap v14589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:08:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:56 smithi059 ceph-mon[97450]: pgmap v14589: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:08:57.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:56.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:56.705950+0000) 2022-09-08T05:08:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:56.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:56.705883+0000) 2022-09-08T05:08:58.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:57.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:57.706097+0000) 2022-09-08T05:08:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:57.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:57.706023+0000) 2022-09-08T05:08:58.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:58 smithi059 ceph-mon[97450]: pgmap v14590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:58 smithi059 ceph-mon[97450]: pgmap v14591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:08:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:58 smithi059 ceph-mon[97450]: pgmap v14592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:08:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:08:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:08:58.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:58 smithi055 ceph-mon[97033]: pgmap v14590: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:08:58.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:58 smithi055 ceph-mon[97033]: pgmap v14591: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:08:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:58 smithi055 ceph-mon[97033]: pgmap v14592: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:08:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:08:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:08:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:58.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:58.706276+0000) 2022-09-08T05:08:59.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:58.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:58.706215+0000) 2022-09-08T05:09:00.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:08:59.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:59.706454+0000) 2022-09-08T05:09:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:08:59.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:08:59.706426+0000) 2022-09-08T05:09:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:00 smithi059 ceph-mon[97450]: pgmap v14593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:09:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:00 smithi055 ceph-mon[97033]: pgmap v14593: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:09:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:00.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:00.706630+0000) 2022-09-08T05:09:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:00.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:00.706630+0000) 2022-09-08T05:09:02.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:01.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:01.706813+0000) 2022-09-08T05:09:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:01.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:01.706767+0000) 2022-09-08T05:09:02.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:09:02.248+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:09:02.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:02 smithi055 ceph-mon[97033]: pgmap v14594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:09:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:02 smithi059 ceph-mon[97450]: pgmap v14594: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:09:03.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:02.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:02.707036+0000) 2022-09-08T05:09:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:02.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:02.706990+0000) 2022-09-08T05:09:04.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:03.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:03.707347+0000) 2022-09-08T05:09:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:03.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:03.707240+0000) 2022-09-08T05:09:04.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:04 smithi055 ceph-mon[97033]: pgmap v14595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:09:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:04 smithi059 ceph-mon[97450]: pgmap v14595: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:09:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:04.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:04.707661+0000) 2022-09-08T05:09:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:04.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:04.707516+0000) 2022-09-08T05:09:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:05.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:05.707890+0000) 2022-09-08T05:09:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:05.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:05.707783+0000) 2022-09-08T05:09:06.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:06 smithi055 ceph-mon[97033]: pgmap v14596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T05:09:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:09:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:09:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:06 smithi059 ceph-mon[97450]: pgmap v14596: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T05:09:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:09:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:09:07.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:06.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:06.708122+0000) 2022-09-08T05:09:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:06.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:06.708027+0000) 2022-09-08T05:09:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:07.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:07.708325+0000) 2022-09-08T05:09:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:07.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:07.708282+0000) 2022-09-08T05:09:08.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:08 smithi055 ceph-mon[97033]: pgmap v14597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:09:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:08 smithi059 ceph-mon[97450]: pgmap v14597: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:09:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:08.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:08.708506+0000) 2022-09-08T05:09:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:08.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:08.708478+0000) 2022-09-08T05:09:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:09.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:09.708678+0000) 2022-09-08T05:09:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:09.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:09.708628+0000) 2022-09-08T05:09:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:10 smithi055 ceph-mon[97033]: pgmap v14598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:09:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:10 smithi055 ceph-mon[97033]: osdmap e993: 12 total, 11 up, 11 in 2022-09-08T05:09:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:10 smithi059 ceph-mon[97450]: pgmap v14598: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:09:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:10 smithi059 ceph-mon[97450]: osdmap e993: 12 total, 11 up, 11 in 2022-09-08T05:09:11.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:10.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:10.708812+0000) 2022-09-08T05:09:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:10.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:10.708744+0000) 2022-09-08T05:09:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:11.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:11.709021+0000) 2022-09-08T05:09:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:11.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:11.709014+0000) 2022-09-08T05:09:12.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:12 smithi055 ceph-mon[97033]: pgmap v14600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:09:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:12 smithi059 ceph-mon[97450]: pgmap v14600: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:09:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:12.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:12.709217+0000) 2022-09-08T05:09:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:12.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:12.709217+0000) 2022-09-08T05:09:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:13.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:13.709509+0000) 2022-09-08T05:09:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:13.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:13.709475+0000) 2022-09-08T05:09:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:14 smithi055 ceph-mon[97033]: pgmap v14601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:09:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:14 smithi059 ceph-mon[97450]: pgmap v14601: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:09:15.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:14.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:14.709750+0000) 2022-09-08T05:09:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:14.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:14.709803+0000) 2022-09-08T05:09:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:15.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:15.709989+0000) 2022-09-08T05:09:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:15.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:15.709924+0000) 2022-09-08T05:09:16.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:16 smithi055 ceph-mon[97033]: pgmap v14602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:09:16.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:16 smithi055 ceph-mon[97033]: osdmap e994: 12 total, 11 up, 11 in 2022-09-08T05:09:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:16 smithi059 ceph-mon[97450]: pgmap v14602: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:09:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:16 smithi059 ceph-mon[97450]: osdmap e994: 12 total, 11 up, 11 in 2022-09-08T05:09:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:16.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:16.710256+0000) 2022-09-08T05:09:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:16.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:16.710180+0000) 2022-09-08T05:09:17.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:09:17.253+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:09:18.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:17.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:17.710417+0000) 2022-09-08T05:09:18.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:17.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:17.710485+0000) 2022-09-08T05:09:18.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:18 smithi055 ceph-mon[97033]: pgmap v14604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:09:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:18 smithi059 ceph-mon[97450]: pgmap v14604: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:09:19.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:18.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:18.710687+0000) 2022-09-08T05:09:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:18.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:18.710794+0000) 2022-09-08T05:09:20.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:19.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:19.710944+0000) 2022-09-08T05:09:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:19.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:19.710861+0000) 2022-09-08T05:09:20.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:20 smithi055 ceph-mon[97033]: pgmap v14605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T05:09:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:20 smithi059 ceph-mon[97450]: pgmap v14605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T05:09:21.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:20.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:20.711181+0000) 2022-09-08T05:09:21.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:20.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:20.711109+0000) 2022-09-08T05:09:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:21.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:21.711356+0000) 2022-09-08T05:09:22.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:21.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:21.711305+0000) 2022-09-08T05:09:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:22 smithi055 ceph-mon[97033]: pgmap v14606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:09:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:22 smithi059 ceph-mon[97450]: pgmap v14606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:09:23.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:22.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:22.711532+0000) 2022-09-08T05:09:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:22.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:22.711604+0000) 2022-09-08T05:09:24.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:23.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:23.711873+0000) 2022-09-08T05:09:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:23.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:23.711745+0000) 2022-09-08T05:09:24.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:24 smithi055 ceph-mon[97033]: pgmap v14607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:09:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:24 smithi059 ceph-mon[97450]: pgmap v14607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:09:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:24.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:24.712019+0000) 2022-09-08T05:09:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:24.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:24.711881+0000) 2022-09-08T05:09:26.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:25.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:25.712173+0000) 2022-09-08T05:09:26.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:25.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:25.712058+0000) 2022-09-08T05:09:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:26 smithi055 ceph-mon[97033]: pgmap v14608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:09:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:26 smithi059 ceph-mon[97450]: pgmap v14608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:09:27.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:26.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:26.712280+0000) 2022-09-08T05:09:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:26.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:26.712316+0000) 2022-09-08T05:09:28.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:27.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:27.712471+0000) 2022-09-08T05:09:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:27.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:27.712475+0000) 2022-09-08T05:09:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:28 smithi055 ceph-mon[97033]: pgmap v14609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:09:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:28 smithi059 ceph-mon[97450]: pgmap v14609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:09:29.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:28.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:28.712772+0000) 2022-09-08T05:09:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:28.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:28.712772+0000) 2022-09-08T05:09:30.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:29.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:29.713056+0000) 2022-09-08T05:09:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:29.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:29.713056+0000) 2022-09-08T05:09:30.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:30 smithi055 ceph-mon[97033]: pgmap v14610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:30 smithi059 ceph-mon[97450]: pgmap v14610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:30.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:30.713300+0000) 2022-09-08T05:09:31.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:30.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:30.713295+0000) 2022-09-08T05:09:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:31.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:31.713497+0000) 2022-09-08T05:09:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:31.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:31.713497+0000) 2022-09-08T05:09:32.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:09:32.259+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:09:32.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:32 smithi055 ceph-mon[97033]: pgmap v14611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:09:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:32 smithi059 ceph-mon[97450]: pgmap v14611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:09:33.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:32.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:32.713707+0000) 2022-09-08T05:09:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:32.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:32.713707+0000) 2022-09-08T05:09:34.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:33.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:33.713903+0000) 2022-09-08T05:09:34.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:33.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:33.713904+0000) 2022-09-08T05:09:34.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:34 smithi055 ceph-mon[97033]: pgmap v14612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:34 smithi059 ceph-mon[97450]: pgmap v14612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:35.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:34.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:34.714171+0000) 2022-09-08T05:09:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:34.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:34.714170+0000) 2022-09-08T05:09:36.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:35.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:35.714426+0000) 2022-09-08T05:09:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:35.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:35.714425+0000) 2022-09-08T05:09:36.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:36 smithi055 ceph-mon[97033]: pgmap v14613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:09:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:36 smithi059 ceph-mon[97450]: pgmap v14613: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:09:37.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:36.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:36.714712+0000) 2022-09-08T05:09:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:36.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:36.714715+0000) 2022-09-08T05:09:38.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:37.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:37.714986+0000) 2022-09-08T05:09:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:37.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:37.714984+0000) 2022-09-08T05:09:38.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:38 smithi055 ceph-mon[97033]: pgmap v14614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:38 smithi059 ceph-mon[97450]: pgmap v14614: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:38.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:38.715293+0000) 2022-09-08T05:09:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:38.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:38.715293+0000) 2022-09-08T05:09:40.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:39.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:39.715566+0000) 2022-09-08T05:09:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:39.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:39.715566+0000) 2022-09-08T05:09:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:40 smithi055 ceph-mon[97033]: pgmap v14615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:40 smithi059 ceph-mon[97450]: pgmap v14615: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:40.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:40.715867+0000) 2022-09-08T05:09:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:40.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:40.715867+0000) 2022-09-08T05:09:42.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:41.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:41.716159+0000) 2022-09-08T05:09:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:41.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:41.716159+0000) 2022-09-08T05:09:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:42 smithi055 ceph-mon[97033]: pgmap v14616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:09:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:42 smithi059 ceph-mon[97450]: pgmap v14616: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:09:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:42.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:42.716366+0000) 2022-09-08T05:09:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:42.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:42.716403+0000) 2022-09-08T05:09:44.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:43.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:43.716607+0000) 2022-09-08T05:09:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:43.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:43.716617+0000) 2022-09-08T05:09:44.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:44 smithi055 ceph-mon[97033]: pgmap v14617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:44.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:44 smithi059 ceph-mon[97450]: pgmap v14617: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:45.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:44.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:44.716811+0000) 2022-09-08T05:09:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:44.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:44.716775+0000) 2022-09-08T05:09:46.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:45.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:45.717056+0000) 2022-09-08T05:09:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:45.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:45.717055+0000) 2022-09-08T05:09:46.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:46 smithi055 ceph-mon[97033]: pgmap v14618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:09:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:46 smithi059 ceph-mon[97450]: pgmap v14618: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:09:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:46.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:46.717281+0000) 2022-09-08T05:09:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:46.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:46.717280+0000) 2022-09-08T05:09:47.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:09:47.276+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:09:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:47.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:47.717484+0000) 2022-09-08T05:09:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:47.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:47.717477+0000) 2022-09-08T05:09:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:48 smithi055 ceph-mon[97033]: pgmap v14619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:48 smithi059 ceph-mon[97450]: pgmap v14619: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:48.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:48.717708+0000) 2022-09-08T05:09:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:48.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:48.717712+0000) 2022-09-08T05:09:50.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:49.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:49.717910+0000) 2022-09-08T05:09:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:49.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:49.717910+0000) 2022-09-08T05:09:50.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:50 smithi055 ceph-mon[97033]: pgmap v14620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:50.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:50 smithi059 ceph-mon[97450]: pgmap v14620: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:50.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:50.718166+0000) 2022-09-08T05:09:51.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:50.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:50.718179+0000) 2022-09-08T05:09:52.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:51.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:51.718411+0000) 2022-09-08T05:09:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:51.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:51.718411+0000) 2022-09-08T05:09:52.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:52 smithi055 ceph-mon[97033]: pgmap v14621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:09:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:52 smithi059 ceph-mon[97450]: pgmap v14621: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:09:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:52.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:52.718584+0000) 2022-09-08T05:09:53.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:52.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:52.718584+0000) 2022-09-08T05:09:54.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:53.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:53.718862+0000) 2022-09-08T05:09:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:53.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:53.718858+0000) 2022-09-08T05:09:54.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:54 smithi055 ceph-mon[97033]: pgmap v14622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:54 smithi059 ceph-mon[97450]: pgmap v14622: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:54.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:54.719146+0000) 2022-09-08T05:09:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:54.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:54.719146+0000) 2022-09-08T05:09:56.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:55.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:55.719372+0000) 2022-09-08T05:09:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:55.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:55.719377+0000) 2022-09-08T05:09:56.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:56 smithi055 ceph-mon[97033]: pgmap v14623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:09:56.790 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:56 smithi059 ceph-mon[97450]: pgmap v14623: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:09:57.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:56.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:56.719579+0000) 2022-09-08T05:09:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:56.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:56.719578+0000) 2022-09-08T05:09:57.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:57 smithi055 ceph-mon[97033]: pgmap v14624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:09:57.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:09:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:09:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:57 smithi059 ceph-mon[97450]: pgmap v14624: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:09:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:09:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:09:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:09:58.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:57.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:57.719795+0000) 2022-09-08T05:09:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:57.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:57.719803+0000) 2022-09-08T05:09:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:58.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:58.719955+0000) 2022-09-08T05:09:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:58.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:58.719969+0000) 2022-09-08T05:09:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:09:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:09:59.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:09:59.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:10:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:09:59.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:59.720180+0000) 2022-09-08T05:10:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:09:59.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:09:59.720249+0000) 2022-09-08T05:10:00.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:09:59 smithi055 ceph-mon[97033]: pgmap v14625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:09:59 smithi059 ceph-mon[97450]: pgmap v14625: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:00.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:00.720399+0000) 2022-09-08T05:10:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:00.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:00.720490+0000) 2022-09-08T05:10:01.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T05:10:01.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T05:10:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T05:10:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T05:10:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T05:10:01.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T05:10:01.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T05:10:01.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T05:10:01.289 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T05:10:01.290 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T05:10:02.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:01.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:01.720598+0000) 2022-09-08T05:10:02.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:01.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:01.720686+0000) 2022-09-08T05:10:02.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:01 smithi055 ceph-mon[97033]: pgmap v14626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:10:02.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:01 smithi059 ceph-mon[97450]: pgmap v14626: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:10:02.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:10:02.291+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:10:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:02.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:02.720821+0000) 2022-09-08T05:10:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:02.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:02.720884+0000) 2022-09-08T05:10:04.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:03.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:03.721024+0000) 2022-09-08T05:10:04.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:03.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:03.721073+0000) 2022-09-08T05:10:04.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:04 smithi055 ceph-mon[97033]: pgmap v14627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:04 smithi059 ceph-mon[97450]: pgmap v14627: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:04.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:04.721262+0000) 2022-09-08T05:10:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:04.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:04.721311+0000) 2022-09-08T05:10:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:05.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:05.721522+0000) 2022-09-08T05:10:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:05.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:05.721578+0000) 2022-09-08T05:10:06.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:06 smithi055 ceph-mon[97033]: pgmap v14628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:10:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:10:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:10:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:06 smithi059 ceph-mon[97450]: pgmap v14628: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:10:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:10:06.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:10:07.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:06.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:06.721773+0000) 2022-09-08T05:10:07.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:06.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:06.721820+0000) 2022-09-08T05:10:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:07.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:07.721907+0000) 2022-09-08T05:10:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:07.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:07.721967+0000) 2022-09-08T05:10:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:08 smithi055 ceph-mon[97033]: pgmap v14629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:10:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:08 smithi059 ceph-mon[97450]: pgmap v14629: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:10:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:08.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:08.722096+0000) 2022-09-08T05:10:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:08.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:08.722153+0000) 2022-09-08T05:10:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:09.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:09.722256+0000) 2022-09-08T05:10:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:09.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:09.722304+0000) 2022-09-08T05:10:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:10 smithi055 ceph-mon[97033]: pgmap v14630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:10:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:10 smithi055 ceph-mon[97033]: osdmap e995: 12 total, 11 up, 11 in 2022-09-08T05:10:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:10 smithi059 ceph-mon[97450]: pgmap v14630: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:10:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:10 smithi059 ceph-mon[97450]: osdmap e995: 12 total, 11 up, 11 in 2022-09-08T05:10:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:10.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:10.722503+0000) 2022-09-08T05:10:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:10.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:10.722514+0000) 2022-09-08T05:10:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:11.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:11.722738+0000) 2022-09-08T05:10:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:11.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:11.722782+0000) 2022-09-08T05:10:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:12 smithi055 ceph-mon[97033]: pgmap v14632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:10:12.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:12 smithi059 ceph-mon[97450]: pgmap v14632: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:10:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:12.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:12.722898+0000) 2022-09-08T05:10:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:12.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:12.722934+0000) 2022-09-08T05:10:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:13.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:13.723120+0000) 2022-09-08T05:10:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:13.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:13.723148+0000) 2022-09-08T05:10:14.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:14 smithi055 ceph-mon[97033]: pgmap v14633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:10:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:14 smithi059 ceph-mon[97450]: pgmap v14633: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:10:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:14.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:14.723381+0000) 2022-09-08T05:10:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:14.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:14.723365+0000) 2022-09-08T05:10:16.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:15.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:15.723550+0000) 2022-09-08T05:10:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:15.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:15.723545+0000) 2022-09-08T05:10:16.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:16 smithi055 ceph-mon[97033]: pgmap v14634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:10:16.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:16 smithi055 ceph-mon[97033]: osdmap e996: 12 total, 11 up, 11 in 2022-09-08T05:10:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:16 smithi059 ceph-mon[97450]: pgmap v14634: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:10:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:16 smithi059 ceph-mon[97450]: osdmap e996: 12 total, 11 up, 11 in 2022-09-08T05:10:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:16.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:16.723716+0000) 2022-09-08T05:10:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:16.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:16.723721+0000) 2022-09-08T05:10:17.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:10:17.292+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:10:18.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:17.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:17.723899+0000) 2022-09-08T05:10:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:17.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:17.723906+0000) 2022-09-08T05:10:18.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:18 smithi055 ceph-mon[97033]: pgmap v14636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:10:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:18 smithi059 ceph-mon[97450]: pgmap v14636: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:10:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:18.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:18.724047+0000) 2022-09-08T05:10:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:18.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:18.724048+0000) 2022-09-08T05:10:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:19.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:19.724320+0000) 2022-09-08T05:10:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:19.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:19.724320+0000) 2022-09-08T05:10:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:20 smithi055 ceph-mon[97033]: pgmap v14637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T05:10:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:20 smithi059 ceph-mon[97450]: pgmap v14637: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T05:10:21.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:20.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:20.724525+0000) 2022-09-08T05:10:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:20.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:20.724527+0000) 2022-09-08T05:10:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:21.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:21.724785+0000) 2022-09-08T05:10:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:21.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:21.724785+0000) 2022-09-08T05:10:22.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:22 smithi055 ceph-mon[97033]: pgmap v14638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:10:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:22 smithi059 ceph-mon[97450]: pgmap v14638: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:10:22.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:22.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:22.725034+0000) 2022-09-08T05:10:22.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:22.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:22.725025+0000) 2022-09-08T05:10:24.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:23.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:23.725303+0000) 2022-09-08T05:10:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:23.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:23.725303+0000) 2022-09-08T05:10:24.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:24 smithi055 ceph-mon[97033]: pgmap v14639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:10:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:24 smithi059 ceph-mon[97450]: pgmap v14639: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:10:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:24.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:24.725605+0000) 2022-09-08T05:10:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:24.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:24.725605+0000) 2022-09-08T05:10:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:25.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:25.725915+0000) 2022-09-08T05:10:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:25.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:25.725918+0000) 2022-09-08T05:10:26.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:26 smithi055 ceph-mon[97033]: pgmap v14640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:10:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:26 smithi059 ceph-mon[97450]: pgmap v14640: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:10:27.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:26.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:26.726097+0000) 2022-09-08T05:10:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:26.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:26.726090+0000) 2022-09-08T05:10:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:27.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:27.726366+0000) 2022-09-08T05:10:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:27.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:27.726366+0000) 2022-09-08T05:10:28.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:28 smithi055 ceph-mon[97033]: pgmap v14641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:10:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:10:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:28 smithi059 ceph-mon[97450]: pgmap v14641: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:10:28.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:10:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:28.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:28.726596+0000) 2022-09-08T05:10:29.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:28.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:28.726596+0000) 2022-09-08T05:10:29.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:29 smithi055 ceph-mon[97033]: pgmap v14642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:10:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:29 smithi055 ceph-mon[97033]: pgmap v14643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:10:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:29 smithi059 ceph-mon[97450]: pgmap v14642: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:10:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:29 smithi059 ceph-mon[97450]: pgmap v14643: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:10:30.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:29.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:29.726787+0000) 2022-09-08T05:10:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:29.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:29.726789+0000) 2022-09-08T05:10:31.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:30.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:30.726996+0000) 2022-09-08T05:10:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:30.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:30.726987+0000) 2022-09-08T05:10:31.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:31 smithi055 ceph-mon[97033]: pgmap v14644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:10:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:31 smithi059 ceph-mon[97450]: pgmap v14644: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:10:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:31.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:31.727247+0000) 2022-09-08T05:10:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:31.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:31.727247+0000) 2022-09-08T05:10:32.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:10:32.306+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:10:33.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:32.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:32.727537+0000) 2022-09-08T05:10:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:32.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:32.727537+0000) 2022-09-08T05:10:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:33 smithi055 ceph-mon[97033]: pgmap v14645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:10:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:33 smithi059 ceph-mon[97450]: pgmap v14645: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:10:34.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:33.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:33.727734+0000) 2022-09-08T05:10:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:33.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:33.727736+0000) 2022-09-08T05:10:35.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:34.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:34.727884+0000) 2022-09-08T05:10:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:34.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:34.727872+0000) 2022-09-08T05:10:35.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:35 smithi055 ceph-mon[97033]: pgmap v14646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:10:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:35 smithi059 ceph-mon[97450]: pgmap v14646: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:10:36.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:35.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:35.728100+0000) 2022-09-08T05:10:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:35.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:35.728064+0000) 2022-09-08T05:10:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:36.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:36.728332+0000) 2022-09-08T05:10:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:36.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:36.728333+0000) 2022-09-08T05:10:37.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:37 smithi055 ceph-mon[97033]: pgmap v14647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:10:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:37 smithi059 ceph-mon[97450]: pgmap v14647: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:10:38.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:37.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:37.728574+0000) 2022-09-08T05:10:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:37.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:37.728598+0000) 2022-09-08T05:10:39.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:38.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:38.728721+0000) 2022-09-08T05:10:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:38.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:38.728792+0000) 2022-09-08T05:10:39.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:39 smithi055 ceph-mon[97033]: pgmap v14648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:10:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:39 smithi059 ceph-mon[97450]: pgmap v14648: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:10:40.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:39.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:39.729004+0000) 2022-09-08T05:10:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:39.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:39.728952+0000) 2022-09-08T05:10:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:40.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:40.729190+0000) 2022-09-08T05:10:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:40.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:40.729188+0000) 2022-09-08T05:10:41.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:41 smithi055 ceph-mon[97033]: pgmap v14649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:41 smithi059 ceph-mon[97450]: pgmap v14649: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:41.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:41.729438+0000) 2022-09-08T05:10:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:41.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:41.729443+0000) 2022-09-08T05:10:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:42.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:42.729743+0000) 2022-09-08T05:10:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:42.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:42.729742+0000) 2022-09-08T05:10:43.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:43 smithi055 ceph-mon[97033]: pgmap v14650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:10:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:43 smithi059 ceph-mon[97450]: pgmap v14650: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:10:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:43.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:43.729938+0000) 2022-09-08T05:10:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:43.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:43.729887+0000) 2022-09-08T05:10:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:44.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:44.730090+0000) 2022-09-08T05:10:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:44.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:44.730090+0000) 2022-09-08T05:10:45.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:45 smithi055 ceph-mon[97033]: pgmap v14651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:45 smithi059 ceph-mon[97450]: pgmap v14651: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:46.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:45.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:45.730311+0000) 2022-09-08T05:10:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:45.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:45.730286+0000) 2022-09-08T05:10:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:46 smithi059 ceph-mon[97450]: pgmap v14652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:10:46.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:46 smithi055 ceph-mon[97033]: pgmap v14652: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:10:47.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:46.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:46.730562+0000) 2022-09-08T05:10:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:46.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:46.730560+0000) 2022-09-08T05:10:47.600 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:10:47.316+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:10:47.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:47.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:47.730899+0000) 2022-09-08T05:10:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:47.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:47.730820+0000) 2022-09-08T05:10:48.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:48 smithi055 ceph-mon[97033]: pgmap v14653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:49.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:48 smithi059 ceph-mon[97450]: pgmap v14653: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:48.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:48.731156+0000) 2022-09-08T05:10:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:48.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:48.731069+0000) 2022-09-08T05:10:50.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:49.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:49.731360+0000) 2022-09-08T05:10:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:49.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:49.731303+0000) 2022-09-08T05:10:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:50.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:50.731574+0000) 2022-09-08T05:10:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:50.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:50.731511+0000) 2022-09-08T05:10:51.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:51 smithi055 ceph-mon[97033]: pgmap v14654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:51 smithi059 ceph-mon[97450]: pgmap v14654: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:52.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:51.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:51.731814+0000) 2022-09-08T05:10:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:51.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:51.731798+0000) 2022-09-08T05:10:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:52.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:52.732034+0000) 2022-09-08T05:10:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:52.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:52.731967+0000) 2022-09-08T05:10:53.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:53 smithi055 ceph-mon[97033]: pgmap v14655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:10:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:53 smithi059 ceph-mon[97450]: pgmap v14655: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:10:54.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:53.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:53.732314+0000) 2022-09-08T05:10:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:53.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:53.732234+0000) 2022-09-08T05:10:55.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:54.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:54.732393+0000) 2022-09-08T05:10:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:54.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:54.732495+0000) 2022-09-08T05:10:55.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:55 smithi055 ceph-mon[97033]: pgmap v14656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:55 smithi059 ceph-mon[97450]: pgmap v14656: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:56.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:55.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:55.732666+0000) 2022-09-08T05:10:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:55.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:55.732537+0000) 2022-09-08T05:10:57.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:56.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:56.732889+0000) 2022-09-08T05:10:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:56.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:56.732762+0000) 2022-09-08T05:10:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:57 smithi055 ceph-mon[97033]: pgmap v14657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:10:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:57 smithi059 ceph-mon[97450]: pgmap v14657: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:10:58.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:57.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:57.733029+0000) 2022-09-08T05:10:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:57.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:57.732875+0000) 2022-09-08T05:10:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:58.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:58.733247+0000) 2022-09-08T05:10:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:58.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:58.733039+0000) 2022-09-08T05:10:59.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:10:59 smithi055 ceph-mon[97033]: pgmap v14658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:10:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:10:59 smithi059 ceph-mon[97450]: pgmap v14658: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:00.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:10:59.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:59.733433+0000) 2022-09-08T05:11:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:10:59.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:10:59.733326+0000) 2022-09-08T05:11:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:00.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:00.733577+0000) 2022-09-08T05:11:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:00.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:00.733456+0000) 2022-09-08T05:11:01.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:01 smithi055 ceph-mon[97033]: pgmap v14659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:01 smithi059 ceph-mon[97450]: pgmap v14659: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:02.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:01.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:01.733807+0000) 2022-09-08T05:11:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:01.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:01.733747+0000) 2022-09-08T05:11:02.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:11:02.329+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:11:03.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:02.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:02.734040+0000) 2022-09-08T05:11:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:02.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:02.734030+0000) 2022-09-08T05:11:03.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:03 smithi055 ceph-mon[97033]: pgmap v14660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:11:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:03 smithi059 ceph-mon[97450]: pgmap v14660: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:11:04.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:03.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:03.734337+0000) 2022-09-08T05:11:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:03.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:03.734325+0000) 2022-09-08T05:11:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:04.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:04.734614+0000) 2022-09-08T05:11:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:04.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:04.734608+0000) 2022-09-08T05:11:05.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:05 smithi055 ceph-mon[97033]: pgmap v14661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:05 smithi059 ceph-mon[97450]: pgmap v14661: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:06.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:05.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:05.734836+0000) 2022-09-08T05:11:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:05.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:05.734836+0000) 2022-09-08T05:11:06.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:11:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:11:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:11:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:11:07.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:06.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:06.735040+0000) 2022-09-08T05:11:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:06.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:06.735036+0000) 2022-09-08T05:11:07.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:07 smithi055 ceph-mon[97033]: pgmap v14662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:11:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:07 smithi059 ceph-mon[97450]: pgmap v14662: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:11:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:07.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:07.735271+0000) 2022-09-08T05:11:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:07.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:07.735263+0000) 2022-09-08T05:11:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:08.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:08.735475+0000) 2022-09-08T05:11:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:08.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:08.735475+0000) 2022-09-08T05:11:09.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:09 smithi055 ceph-mon[97033]: pgmap v14663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:11:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:09 smithi059 ceph-mon[97450]: pgmap v14663: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:11:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:09.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:09.735706+0000) 2022-09-08T05:11:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:09.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:09.735702+0000) 2022-09-08T05:11:10.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:10 smithi055 ceph-mon[97033]: osdmap e997: 12 total, 11 up, 11 in 2022-09-08T05:11:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:10 smithi059 ceph-mon[97450]: osdmap e997: 12 total, 11 up, 11 in 2022-09-08T05:11:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:10.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:10.735968+0000) 2022-09-08T05:11:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:10.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:10.735969+0000) 2022-09-08T05:11:11.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:11 smithi055 ceph-mon[97033]: pgmap v14665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:11:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:11 smithi059 ceph-mon[97450]: pgmap v14665: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:11:12.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:11.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:11.736117+0000) 2022-09-08T05:11:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:11.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:11.736140+0000) 2022-09-08T05:11:13.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:12.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:12.736310+0000) 2022-09-08T05:11:13.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:12.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:12.736302+0000) 2022-09-08T05:11:13.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:13 smithi055 ceph-mon[97033]: pgmap v14666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:11:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:13 smithi059 ceph-mon[97450]: pgmap v14666: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:11:14.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:13.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:13.736539+0000) 2022-09-08T05:11:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:13.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:13.736541+0000) 2022-09-08T05:11:15.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:14.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:14.736710+0000) 2022-09-08T05:11:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:14.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:14.736709+0000) 2022-09-08T05:11:15.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:15 smithi055 ceph-mon[97033]: pgmap v14667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:11:15.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:15 smithi055 ceph-mon[97033]: osdmap e998: 12 total, 11 up, 11 in 2022-09-08T05:11:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:15 smithi059 ceph-mon[97450]: pgmap v14667: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:11:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:15 smithi059 ceph-mon[97450]: osdmap e998: 12 total, 11 up, 11 in 2022-09-08T05:11:16.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:15.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:15.736893+0000) 2022-09-08T05:11:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:15.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:15.736886+0000) 2022-09-08T05:11:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:16 smithi059 ceph-mon[97450]: pgmap v14669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:11:16.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:16 smithi055 ceph-mon[97033]: pgmap v14669: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:11:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:16.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:16.737123+0000) 2022-09-08T05:11:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:16.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:16.737137+0000) 2022-09-08T05:11:17.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:11:17.345+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:11:18.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:17.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:17.737398+0000) 2022-09-08T05:11:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:17.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:17.737371+0000) 2022-09-08T05:11:18.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:18 smithi055 ceph-mon[97033]: pgmap v14670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T05:11:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:18 smithi059 ceph-mon[97450]: pgmap v14670: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T05:11:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:18.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:18.737596+0000) 2022-09-08T05:11:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:18.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:18.737530+0000) 2022-09-08T05:11:20.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:19.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:19.737782+0000) 2022-09-08T05:11:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:19.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:19.737797+0000) 2022-09-08T05:11:21.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:20.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:20.738063+0000) 2022-09-08T05:11:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:20.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:20.738072+0000) 2022-09-08T05:11:21.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:21 smithi055 ceph-mon[97033]: pgmap v14671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:11:21.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:21 smithi059 ceph-mon[97450]: pgmap v14671: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:11:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:21.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:21.738305+0000) 2022-09-08T05:11:22.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:21.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:21.738299+0000) 2022-09-08T05:11:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:22.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:22.738548+0000) 2022-09-08T05:11:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:22.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:22.738561+0000) 2022-09-08T05:11:23.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:23 smithi055 ceph-mon[97033]: pgmap v14672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:11:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:23 smithi059 ceph-mon[97450]: pgmap v14672: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:11:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:23.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:23.738802+0000) 2022-09-08T05:11:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:23.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:23.738792+0000) 2022-09-08T05:11:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:24.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:24.739061+0000) 2022-09-08T05:11:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:24.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:24.739069+0000) 2022-09-08T05:11:25.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:25 smithi055 ceph-mon[97033]: pgmap v14673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:11:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:25 smithi059 ceph-mon[97450]: pgmap v14673: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:11:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:25.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:25.739187+0000) 2022-09-08T05:11:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:25.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:25.739225+0000) 2022-09-08T05:11:27.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:26.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:26.739415+0000) 2022-09-08T05:11:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:26.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:26.739445+0000) 2022-09-08T05:11:27.722 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:27 smithi055 ceph-mon[97033]: pgmap v14674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:11:27.723 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:27 smithi059 ceph-mon[97450]: pgmap v14674: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:11:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:27.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:27.739580+0000) 2022-09-08T05:11:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:27.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:27.739680+0000) 2022-09-08T05:11:28.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:11:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:11:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:11:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:11:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:11:28.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:11:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:28.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:28.739758+0000) 2022-09-08T05:11:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:28.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:28.739827+0000) 2022-09-08T05:11:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:29 smithi055 ceph-mon[97033]: pgmap v14675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:29.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:29 smithi059 ceph-mon[97450]: pgmap v14675: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:30.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:29.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:29.739886+0000) 2022-09-08T05:11:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:29.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:29.739967+0000) 2022-09-08T05:11:31.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:30.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:30.740081+0000) 2022-09-08T05:11:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:30.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:30.740116+0000) 2022-09-08T05:11:31.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:31 smithi055 ceph-mon[97033]: pgmap v14676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:31.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:31 smithi059 ceph-mon[97450]: pgmap v14676: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:31.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:31.740254+0000) 2022-09-08T05:11:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:31.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:31.740295+0000) 2022-09-08T05:11:32.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:11:32.347+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:11:33.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:32.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:32.740499+0000) 2022-09-08T05:11:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:32.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:32.740519+0000) 2022-09-08T05:11:33.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:33 smithi055 ceph-mon[97033]: pgmap v14677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:11:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:33 smithi059 ceph-mon[97450]: pgmap v14677: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:11:34.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:33.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:33.740756+0000) 2022-09-08T05:11:34.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:33.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:33.740768+0000) 2022-09-08T05:11:35.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:34.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:34.741060+0000) 2022-09-08T05:11:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:34.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:34.741013+0000) 2022-09-08T05:11:35.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:35 smithi055 ceph-mon[97033]: pgmap v14678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:35 smithi059 ceph-mon[97450]: pgmap v14678: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:35.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:35.741312+0000) 2022-09-08T05:11:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:35.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:35.741309+0000) 2022-09-08T05:11:37.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:36.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:36.741483+0000) 2022-09-08T05:11:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:36.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:36.741481+0000) 2022-09-08T05:11:37.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:37 smithi055 ceph-mon[97033]: pgmap v14679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:11:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:37 smithi059 ceph-mon[97450]: pgmap v14679: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:11:38.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:37.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:37.741707+0000) 2022-09-08T05:11:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:37.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:37.741698+0000) 2022-09-08T05:11:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:38.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:38.741912+0000) 2022-09-08T05:11:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:38.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:38.741896+0000) 2022-09-08T05:11:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:39 smithi055 ceph-mon[97033]: pgmap v14680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:39 smithi059 ceph-mon[97450]: pgmap v14680: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:40.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:39.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:39.742161+0000) 2022-09-08T05:11:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:39.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:39.742158+0000) 2022-09-08T05:11:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:40.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:40.742445+0000) 2022-09-08T05:11:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:40.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:40.742445+0000) 2022-09-08T05:11:41.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:41 smithi055 ceph-mon[97033]: pgmap v14681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:41 smithi059 ceph-mon[97450]: pgmap v14681: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:41.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:41.742718+0000) 2022-09-08T05:11:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:41.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:41.742718+0000) 2022-09-08T05:11:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:42.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:42.742959+0000) 2022-09-08T05:11:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:42.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:42.742958+0000) 2022-09-08T05:11:43.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:43 smithi055 ceph-mon[97033]: pgmap v14682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:11:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:43 smithi059 ceph-mon[97450]: pgmap v14682: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:11:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:43.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:43.743195+0000) 2022-09-08T05:11:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:43.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:43.743195+0000) 2022-09-08T05:11:45.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:44.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:44.743370+0000) 2022-09-08T05:11:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:44.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:44.743366+0000) 2022-09-08T05:11:45.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:45 smithi055 ceph-mon[97033]: pgmap v14683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:45 smithi059 ceph-mon[97450]: pgmap v14683: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:46.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:45.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:45.743589+0000) 2022-09-08T05:11:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:45.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:45.743589+0000) 2022-09-08T05:11:47.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:46.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:46.743857+0000) 2022-09-08T05:11:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:46.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:46.743857+0000) 2022-09-08T05:11:47.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:11:47.362+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:11:47.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:47 smithi055 ceph-mon[97033]: pgmap v14684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:11:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:47 smithi059 ceph-mon[97450]: pgmap v14684: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:11:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:47.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:47.744078+0000) 2022-09-08T05:11:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:47.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:47.744029+0000) 2022-09-08T05:11:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:48.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:48.744218+0000) 2022-09-08T05:11:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:48.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:48.744179+0000) 2022-09-08T05:11:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:49 smithi055 ceph-mon[97033]: pgmap v14685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:49 smithi059 ceph-mon[97450]: pgmap v14685: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:50.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:49.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:49.744411+0000) 2022-09-08T05:11:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:49.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:49.744411+0000) 2022-09-08T05:11:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:50.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:50.744705+0000) 2022-09-08T05:11:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:50.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:50.744702+0000) 2022-09-08T05:11:51.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:51 smithi055 ceph-mon[97033]: pgmap v14686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:51 smithi059 ceph-mon[97450]: pgmap v14686: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:52.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:51.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:51.744890+0000) 2022-09-08T05:11:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:51.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:51.744960+0000) 2022-09-08T05:11:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:52.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:52.745167+0000) 2022-09-08T05:11:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:52.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:52.745179+0000) 2022-09-08T05:11:53.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:53 smithi055 ceph-mon[97033]: pgmap v14687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:11:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:53 smithi059 ceph-mon[97450]: pgmap v14687: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:11:54.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:53.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:53.745381+0000) 2022-09-08T05:11:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:53.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:53.745386+0000) 2022-09-08T05:11:55.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:54.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:54.745647+0000) 2022-09-08T05:11:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:54.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:54.745653+0000) 2022-09-08T05:11:55.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:55 smithi055 ceph-mon[97033]: pgmap v14688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:55 smithi059 ceph-mon[97450]: pgmap v14688: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:56.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:55.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:55.745905+0000) 2022-09-08T05:11:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:55.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:55.745906+0000) 2022-09-08T05:11:57.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:56.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:56.746146+0000) 2022-09-08T05:11:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:56.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:56.746148+0000) 2022-09-08T05:11:57.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:57 smithi055 ceph-mon[97033]: pgmap v14689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:11:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:57 smithi059 ceph-mon[97450]: pgmap v14689: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:11:58.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:57.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:57.746386+0000) 2022-09-08T05:11:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:57.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:57.746392+0000) 2022-09-08T05:11:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:58.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:58.746527+0000) 2022-09-08T05:11:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:58.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:58.746523+0000) 2022-09-08T05:11:59.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:58 smithi055 ceph-mon[97033]: pgmap v14690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:59.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:58 smithi055 ceph-mon[97033]: pgmap v14691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:11:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:58 smithi055 ceph-mon[97033]: pgmap v14692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:11:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:11:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:11:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:58 smithi059 ceph-mon[97450]: pgmap v14690: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:11:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:58 smithi059 ceph-mon[97450]: pgmap v14691: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:11:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:58 smithi059 ceph-mon[97450]: pgmap v14692: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:11:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:11:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:12:00.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:11:59.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:59.746796+0000) 2022-09-08T05:12:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:11:59.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:11:59.746795+0000) 2022-09-08T05:12:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:00.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:00.746995+0000) 2022-09-08T05:12:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:00.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:00.746991+0000) 2022-09-08T05:12:01.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:01 smithi055 ceph-mon[97033]: pgmap v14693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:12:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:01 smithi059 ceph-mon[97450]: pgmap v14693: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:12:02.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:01.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:01.747222+0000) 2022-09-08T05:12:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:01.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:01.747240+0000) 2022-09-08T05:12:02.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:12:02.378+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:12:03.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:02.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:02.747457+0000) 2022-09-08T05:12:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:02.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:02.747459+0000) 2022-09-08T05:12:03.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:03 smithi055 ceph-mon[97033]: pgmap v14694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:12:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:03 smithi059 ceph-mon[97450]: pgmap v14694: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:12:04.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:03.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:03.747763+0000) 2022-09-08T05:12:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:03.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:03.747752+0000) 2022-09-08T05:12:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:04.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:04.747997+0000) 2022-09-08T05:12:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:04.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:04.748004+0000) 2022-09-08T05:12:05.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:05 smithi055 ceph-mon[97033]: pgmap v14695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:12:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:05 smithi059 ceph-mon[97450]: pgmap v14695: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:12:06.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:05.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:05.748270+0000) 2022-09-08T05:12:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:05.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:05.748290+0000) 2022-09-08T05:12:06.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:12:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:12:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:12:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:12:07.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:06.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:06.748542+0000) 2022-09-08T05:12:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:06.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:06.748549+0000) 2022-09-08T05:12:07.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:07 smithi055 ceph-mon[97033]: pgmap v14696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T05:12:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:07 smithi059 ceph-mon[97450]: pgmap v14696: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T05:12:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:07.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:07.748801+0000) 2022-09-08T05:12:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:07.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:07.748798+0000) 2022-09-08T05:12:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:08.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:08.748992+0000) 2022-09-08T05:12:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:08.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:08.748996+0000) 2022-09-08T05:12:09.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:09 smithi055 ceph-mon[97033]: pgmap v14697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:12:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:09 smithi059 ceph-mon[97450]: pgmap v14697: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:12:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:09.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:09.749217+0000) 2022-09-08T05:12:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:09.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:09.749224+0000) 2022-09-08T05:12:10.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:10 smithi055 ceph-mon[97033]: osdmap e999: 12 total, 11 up, 11 in 2022-09-08T05:12:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:10 smithi059 ceph-mon[97450]: osdmap e999: 12 total, 11 up, 11 in 2022-09-08T05:12:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:10.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:10.749441+0000) 2022-09-08T05:12:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:10.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:10.749422+0000) 2022-09-08T05:12:11.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:11 smithi055 ceph-mon[97033]: pgmap v14699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:12:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:11 smithi059 ceph-mon[97450]: pgmap v14699: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:12:12.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:11.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:11.749662+0000) 2022-09-08T05:12:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:11.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:11.749677+0000) 2022-09-08T05:12:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:12.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:12.749884+0000) 2022-09-08T05:12:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:12.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:12.749924+0000) 2022-09-08T05:12:13.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:13 smithi055 ceph-mon[97033]: pgmap v14700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:12:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:13 smithi059 ceph-mon[97450]: pgmap v14700: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:12:14.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:13.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:13.750050+0000) 2022-09-08T05:12:14.169 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:13.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:13.750135+0000) 2022-09-08T05:12:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:14.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:14.750227+0000) 2022-09-08T05:12:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:14.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:14.750258+0000) 2022-09-08T05:12:15.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:15 smithi055 ceph-mon[97033]: pgmap v14701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:12:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:15 smithi059 ceph-mon[97450]: pgmap v14701: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:12:16.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:15.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:15.750387+0000) 2022-09-08T05:12:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:15.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:15.750419+0000) 2022-09-08T05:12:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:16.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:16.750632+0000) 2022-09-08T05:12:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:16.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:16.750678+0000) 2022-09-08T05:12:17.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:12:17.393+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:12:17.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:17 smithi055 ceph-mon[97033]: pgmap v14702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:12:17.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:17 smithi059 ceph-mon[97450]: pgmap v14702: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:12:18.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:17.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:17.750909+0000) 2022-09-08T05:12:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:17.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:17.750880+0000) 2022-09-08T05:12:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:18.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:18.751114+0000) 2022-09-08T05:12:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:18.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:18.751134+0000) 2022-09-08T05:12:19.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:19 smithi055 ceph-mon[97033]: pgmap v14703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:12:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:19 smithi059 ceph-mon[97450]: pgmap v14703: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:12:20.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:19.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:19.751350+0000) 2022-09-08T05:12:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:19.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:19.751293+0000) 2022-09-08T05:12:21.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:20.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:20.751536+0000) 2022-09-08T05:12:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:20.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:20.751577+0000) 2022-09-08T05:12:21.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:21 smithi055 ceph-mon[97033]: pgmap v14704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 197 B/s wr, 6 op/s 2022-09-08T05:12:21.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:21 smithi055 ceph-mon[97033]: osdmap e1000: 12 total, 11 up, 11 in 2022-09-08T05:12:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:21 smithi059 ceph-mon[97450]: pgmap v14704: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 197 B/s wr, 6 op/s 2022-09-08T05:12:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:21 smithi059 ceph-mon[97450]: osdmap e1000: 12 total, 11 up, 11 in 2022-09-08T05:12:22.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:21.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:21.751886+0000) 2022-09-08T05:12:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:21.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:21.751893+0000) 2022-09-08T05:12:23.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:22.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:22.752095+0000) 2022-09-08T05:12:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:22.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:22.752053+0000) 2022-09-08T05:12:23.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:23 smithi055 ceph-mon[97033]: pgmap v14706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:12:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:23 smithi059 ceph-mon[97450]: pgmap v14706: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:12:24.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:23.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:23.752308+0000) 2022-09-08T05:12:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:23.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:23.752285+0000) 2022-09-08T05:12:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:24.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:24.752517+0000) 2022-09-08T05:12:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:24.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:24.752512+0000) 2022-09-08T05:12:25.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:25 smithi055 ceph-mon[97033]: pgmap v14707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:12:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:25 smithi059 ceph-mon[97450]: pgmap v14707: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:12:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:25.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:25.752766+0000) 2022-09-08T05:12:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:25.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:25.752746+0000) 2022-09-08T05:12:27.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:26.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:26.753007+0000) 2022-09-08T05:12:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:26.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:26.752984+0000) 2022-09-08T05:12:27.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:27 smithi055 ceph-mon[97033]: pgmap v14708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:12:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:27 smithi059 ceph-mon[97450]: pgmap v14708: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:12:28.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:27.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:27.753110+0000) 2022-09-08T05:12:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:27.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:27.753168+0000) 2022-09-08T05:12:29.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:28.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:28.753379+0000) 2022-09-08T05:12:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:28.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:28.753350+0000) 2022-09-08T05:12:29.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:29 smithi055 ceph-mon[97033]: pgmap v14709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:12:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:29 smithi059 ceph-mon[97450]: pgmap v14709: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:12:30.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:29.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:29.753614+0000) 2022-09-08T05:12:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:29.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:29.753591+0000) 2022-09-08T05:12:31.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:30.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:30.753826+0000) 2022-09-08T05:12:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:30.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:30.753800+0000) 2022-09-08T05:12:31.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:31 smithi055 ceph-mon[97033]: pgmap v14710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:12:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:31 smithi059 ceph-mon[97450]: pgmap v14710: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:12:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:31.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:31.753999+0000) 2022-09-08T05:12:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:31.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:31.753966+0000) 2022-09-08T05:12:32.753 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:12:32.401+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:12:33.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:32.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:32.754183+0000) 2022-09-08T05:12:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:32.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:32.754214+0000) 2022-09-08T05:12:33.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:33 smithi055 ceph-mon[97033]: pgmap v14711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:12:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:33 smithi059 ceph-mon[97450]: pgmap v14711: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:12:34.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:33.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:33.754410+0000) 2022-09-08T05:12:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:33.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:33.754378+0000) 2022-09-08T05:12:35.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:34.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:34.754599+0000) 2022-09-08T05:12:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:34.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:34.754572+0000) 2022-09-08T05:12:35.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:35 smithi055 ceph-mon[97033]: pgmap v14712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:35 smithi059 ceph-mon[97450]: pgmap v14712: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:35.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:35.754820+0000) 2022-09-08T05:12:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:35.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:35.754808+0000) 2022-09-08T05:12:37.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:36.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:36.754999+0000) 2022-09-08T05:12:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:36.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:36.754940+0000) 2022-09-08T05:12:37.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:37 smithi055 ceph-mon[97033]: pgmap v14713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:12:37.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:37 smithi059 ceph-mon[97450]: pgmap v14713: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:12:38.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:37.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:37.755186+0000) 2022-09-08T05:12:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:37.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:37.755250+0000) 2022-09-08T05:12:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:38.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:38.755518+0000) 2022-09-08T05:12:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:38.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:38.755454+0000) 2022-09-08T05:12:39.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:39 smithi055 ceph-mon[97033]: pgmap v14714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:39 smithi059 ceph-mon[97450]: pgmap v14714: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:40.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:39.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:39.755764+0000) 2022-09-08T05:12:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:39.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:39.755732+0000) 2022-09-08T05:12:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:40.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:40.756004+0000) 2022-09-08T05:12:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:40.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:40.755979+0000) 2022-09-08T05:12:41.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:41 smithi055 ceph-mon[97033]: pgmap v14715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:41 smithi059 ceph-mon[97450]: pgmap v14715: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:41.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:41.756285+0000) 2022-09-08T05:12:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:41.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:41.756243+0000) 2022-09-08T05:12:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:42.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:42.756486+0000) 2022-09-08T05:12:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:42.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:42.756417+0000) 2022-09-08T05:12:43.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:43 smithi055 ceph-mon[97033]: pgmap v14716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:12:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:43 smithi059 ceph-mon[97450]: pgmap v14716: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:12:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:43.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:43.756746+0000) 2022-09-08T05:12:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:43.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:43.756746+0000) 2022-09-08T05:12:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:44 smithi059 ceph-mon[97450]: pgmap v14717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:44.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:44 smithi055 ceph-mon[97033]: pgmap v14717: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:45.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:44.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:44.757043+0000) 2022-09-08T05:12:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:44.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:44.757043+0000) 2022-09-08T05:12:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:45.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:45.757288+0000) 2022-09-08T05:12:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:45.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:45.757302+0000) 2022-09-08T05:12:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:46.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:46.757584+0000) 2022-09-08T05:12:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:46.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:46.757583+0000) 2022-09-08T05:12:47.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:47 smithi055 ceph-mon[97033]: pgmap v14718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:12:47.757 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:12:47.418+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:12:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:47 smithi059 ceph-mon[97450]: pgmap v14718: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:12:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:47.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:47.757801+0000) 2022-09-08T05:12:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:47.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:47.757801+0000) 2022-09-08T05:12:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:48.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:48.758014+0000) 2022-09-08T05:12:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:48.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:48.758016+0000) 2022-09-08T05:12:49.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:49 smithi055 ceph-mon[97033]: pgmap v14719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:49 smithi059 ceph-mon[97450]: pgmap v14719: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:49.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:49.758292+0000) 2022-09-08T05:12:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:49.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:49.758292+0000) 2022-09-08T05:12:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:50.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:50.758516+0000) 2022-09-08T05:12:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:50.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:50.758515+0000) 2022-09-08T05:12:51.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:51 smithi055 ceph-mon[97033]: pgmap v14720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:51.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:51 smithi059 ceph-mon[97450]: pgmap v14720: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:52.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:51.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:51.758767+0000) 2022-09-08T05:12:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:51.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:51.758771+0000) 2022-09-08T05:12:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:52.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:52.758983+0000) 2022-09-08T05:12:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:52.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:52.759018+0000) 2022-09-08T05:12:53.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:53 smithi055 ceph-mon[97033]: pgmap v14721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:12:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:53 smithi059 ceph-mon[97450]: pgmap v14721: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:12:54.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:53.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:53.759222+0000) 2022-09-08T05:12:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:53.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:53.759236+0000) 2022-09-08T05:12:55.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:54.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:54.759465+0000) 2022-09-08T05:12:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:54.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:54.759473+0000) 2022-09-08T05:12:55.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:55 smithi055 ceph-mon[97033]: pgmap v14722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:55 smithi059 ceph-mon[97450]: pgmap v14722: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:56.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:55.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:55.759740+0000) 2022-09-08T05:12:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:55.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:55.759739+0000) 2022-09-08T05:12:57.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:56.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:56.759913+0000) 2022-09-08T05:12:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:56.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:56.759920+0000) 2022-09-08T05:12:57.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:57 smithi055 ceph-mon[97033]: pgmap v14723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:12:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:57 smithi059 ceph-mon[97450]: pgmap v14723: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:12:58.039 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:57.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:57.760089+0000) 2022-09-08T05:12:58.040 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:57.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:57.760088+0000) 2022-09-08T05:12:58.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:12:58.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:12:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:12:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:12:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:12:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:12:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:58.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:58.760277+0000) 2022-09-08T05:12:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:58.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:58.760259+0000) 2022-09-08T05:12:59.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:12:59 smithi055 ceph-mon[97033]: pgmap v14724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:12:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:12:59 smithi059 ceph-mon[97450]: pgmap v14724: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:12:59.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:59.760408+0000) 2022-09-08T05:13:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:12:59.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:12:59.760431+0000) 2022-09-08T05:13:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:00.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:00.760621+0000) 2022-09-08T05:13:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:00.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:00.760709+0000) 2022-09-08T05:13:01.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:01 smithi055 ceph-mon[97033]: pgmap v14725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:01 smithi059 ceph-mon[97450]: pgmap v14725: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:02.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:01.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:01.760819+0000) 2022-09-08T05:13:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:01.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:01.760868+0000) 2022-09-08T05:13:02.760 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:13:02.428+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:13:03.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:02.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:02.761024+0000) 2022-09-08T05:13:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:02.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:02.761086+0000) 2022-09-08T05:13:03.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:03 smithi055 ceph-mon[97033]: pgmap v14726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:13:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:03 smithi059 ceph-mon[97450]: pgmap v14726: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:13:04.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:03.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:03.761261+0000) 2022-09-08T05:13:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:03.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:03.761310+0000) 2022-09-08T05:13:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:04.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:04.761459+0000) 2022-09-08T05:13:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:04.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:04.761486+0000) 2022-09-08T05:13:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:05 smithi059 ceph-mon[97450]: pgmap v14727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:05.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:05 smithi055 ceph-mon[97033]: pgmap v14727: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:05.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:05.761745+0000) 2022-09-08T05:13:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:05.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:05.761696+0000) 2022-09-08T05:13:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:13:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:06 smithi059 ceph-mon[97450]: pgmap v14728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:13:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:13:06.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:13:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:06 smithi055 ceph-mon[97033]: pgmap v14728: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:13:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:13:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:06.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:06.761894+0000) 2022-09-08T05:13:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:06.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:06.761929+0000) 2022-09-08T05:13:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:07.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:07.762117+0000) 2022-09-08T05:13:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:07.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:07.762162+0000) 2022-09-08T05:13:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:08.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:08.762400+0000) 2022-09-08T05:13:09.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:08.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:08.762400+0000) 2022-09-08T05:13:09.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:08 smithi055 ceph-mon[97033]: pgmap v14729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:13:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:08 smithi059 ceph-mon[97450]: pgmap v14729: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:13:10.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:09.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:09.762678+0000) 2022-09-08T05:13:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:09.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:09.762616+0000) 2022-09-08T05:13:10.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:09 smithi055 ceph-mon[97033]: osdmap e1001: 12 total, 11 up, 11 in 2022-09-08T05:13:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:09 smithi059 ceph-mon[97450]: osdmap e1001: 12 total, 11 up, 11 in 2022-09-08T05:13:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:10.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:10.762834+0000) 2022-09-08T05:13:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:10.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:10.762832+0000) 2022-09-08T05:13:11.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:11 smithi055 ceph-mon[97033]: pgmap v14731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:13:11.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:11 smithi059 ceph-mon[97450]: pgmap v14731: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:13:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:11.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:11.763045+0000) 2022-09-08T05:13:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:11.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:11.763040+0000) 2022-09-08T05:13:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:12.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:12.763294+0000) 2022-09-08T05:13:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:12.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:12.763297+0000) 2022-09-08T05:13:13.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:13 smithi055 ceph-mon[97033]: pgmap v14732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:13:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:13 smithi059 ceph-mon[97450]: pgmap v14732: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:13:14.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:13.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:13.763495+0000) 2022-09-08T05:13:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:13.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:13.763495+0000) 2022-09-08T05:13:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:14.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:14.763681+0000) 2022-09-08T05:13:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:14.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:14.763627+0000) 2022-09-08T05:13:15.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:15 smithi055 ceph-mon[97033]: pgmap v14733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:13:15.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:15 smithi055 ceph-mon[97033]: osdmap e1002: 12 total, 11 up, 11 in 2022-09-08T05:13:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:15 smithi059 ceph-mon[97450]: pgmap v14733: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:13:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:15 smithi059 ceph-mon[97450]: osdmap e1002: 12 total, 11 up, 11 in 2022-09-08T05:13:16.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:15.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:15.763842+0000) 2022-09-08T05:13:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:15.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:15.763806+0000) 2022-09-08T05:13:17.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:16.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:16.764084+0000) 2022-09-08T05:13:17.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:16.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:16.764037+0000) 2022-09-08T05:13:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:17 smithi055 ceph-mon[97033]: pgmap v14735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:13:17.764 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:13:17.443+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:13:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:17 smithi059 ceph-mon[97450]: pgmap v14735: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:13:18.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:17.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:17.764318+0000) 2022-09-08T05:13:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:17.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:17.764266+0000) 2022-09-08T05:13:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:18.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:18.764571+0000) 2022-09-08T05:13:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:18.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:18.764496+0000) 2022-09-08T05:13:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:19 smithi059 ceph-mon[97450]: pgmap v14736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T05:13:19.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:19 smithi055 ceph-mon[97033]: pgmap v14736: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T05:13:20.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:19.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:19.764774+0000) 2022-09-08T05:13:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:19.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:19.764768+0000) 2022-09-08T05:13:21.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:20.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:20.764963+0000) 2022-09-08T05:13:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:20.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:20.764961+0000) 2022-09-08T05:13:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:21 smithi059 ceph-mon[97450]: pgmap v14737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:13:21.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:21 smithi055 ceph-mon[97033]: pgmap v14737: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:13:22.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:21.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:21.765187+0000) 2022-09-08T05:13:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:21.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:21.765216+0000) 2022-09-08T05:13:22.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:22 smithi059 ceph-mon[97450]: pgmap v14738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:13:22.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:22 smithi055 ceph-mon[97033]: pgmap v14738: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:13:23.070 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:22.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:22.765445+0000) 2022-09-08T05:13:23.071 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:22.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:22.765470+0000) 2022-09-08T05:13:24.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:23.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:23.765712+0000) 2022-09-08T05:13:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:23.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:23.765716+0000) 2022-09-08T05:13:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:24.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:24.765892+0000) 2022-09-08T05:13:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:24.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:24.765891+0000) 2022-09-08T05:13:25.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:24 smithi055 ceph-mon[97033]: pgmap v14739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:13:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:24 smithi059 ceph-mon[97450]: pgmap v14739: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:13:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:25.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:25.766017+0000) 2022-09-08T05:13:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:25.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:25.766052+0000) 2022-09-08T05:13:27.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:26.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:26.766236+0000) 2022-09-08T05:13:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:26.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:26.766257+0000) 2022-09-08T05:13:27.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:27 smithi055 ceph-mon[97033]: pgmap v14740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:13:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:27 smithi059 ceph-mon[97450]: pgmap v14740: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:13:28.122 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:27.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:27.766487+0000) 2022-09-08T05:13:28.123 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:27.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:27.766489+0000) 2022-09-08T05:13:29.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:28.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:28.766759+0000) 2022-09-08T05:13:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:28.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:28.766758+0000) 2022-09-08T05:13:29.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:29 smithi055 ceph-mon[97033]: pgmap v14741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:29 smithi055 ceph-mon[97033]: pgmap v14742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:13:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:29 smithi055 ceph-mon[97033]: pgmap v14743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:13:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:13:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:29 smithi059 ceph-mon[97450]: pgmap v14741: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:29 smithi059 ceph-mon[97450]: pgmap v14742: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:13:29.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:29 smithi059 ceph-mon[97450]: pgmap v14743: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:13:29.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:13:30.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:29.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:29.766921+0000) 2022-09-08T05:13:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:29.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:29.766980+0000) 2022-09-08T05:13:31.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:30.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:30.767156+0000) 2022-09-08T05:13:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:30.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:30.767126+0000) 2022-09-08T05:13:31.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:31 smithi055 ceph-mon[97033]: pgmap v14744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:13:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:31 smithi059 ceph-mon[97450]: pgmap v14744: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:13:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:31.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:31.767353+0000) 2022-09-08T05:13:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:31.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:31.767376+0000) 2022-09-08T05:13:32.766 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:13:32.457+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:13:33.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:32.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:32.767584+0000) 2022-09-08T05:13:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:32.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:32.767606+0000) 2022-09-08T05:13:33.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:33 smithi055 ceph-mon[97033]: pgmap v14745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:13:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:33 smithi059 ceph-mon[97450]: pgmap v14745: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:13:34.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:33.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:33.767822+0000) 2022-09-08T05:13:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:33.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:33.767835+0000) 2022-09-08T05:13:35.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:34.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:34.767945+0000) 2022-09-08T05:13:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:34.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:34.768010+0000) 2022-09-08T05:13:35.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:35 smithi055 ceph-mon[97033]: pgmap v14746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:13:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:35 smithi059 ceph-mon[97450]: pgmap v14746: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:13:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:35.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:35.768142+0000) 2022-09-08T05:13:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:35.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:35.768206+0000) 2022-09-08T05:13:37.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:36.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:36.768387+0000) 2022-09-08T05:13:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:36.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:36.768343+0000) 2022-09-08T05:13:37.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:37 smithi055 ceph-mon[97033]: pgmap v14747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:13:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:37 smithi059 ceph-mon[97450]: pgmap v14747: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:13:38.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:37.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:37.768556+0000) 2022-09-08T05:13:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:37.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:37.768617+0000) 2022-09-08T05:13:39.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:38.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:38.768888+0000) 2022-09-08T05:13:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:38.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:38.768778+0000) 2022-09-08T05:13:39.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:39 smithi055 ceph-mon[97033]: pgmap v14748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:13:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:39 smithi059 ceph-mon[97450]: pgmap v14748: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:13:40.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:39.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:39.768928+0000) 2022-09-08T05:13:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:39.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:39.769081+0000) 2022-09-08T05:13:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:40.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:40.769159+0000) 2022-09-08T05:13:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:40.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:40.769269+0000) 2022-09-08T05:13:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:41 smithi059 ceph-mon[97450]: pgmap v14749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:13:41.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:41 smithi055 ceph-mon[97033]: pgmap v14749: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:13:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:41.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:41.769434+0000) 2022-09-08T05:13:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:41.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:41.769458+0000) 2022-09-08T05:13:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:42.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:42.769693+0000) 2022-09-08T05:13:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:42.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:42.769693+0000) 2022-09-08T05:13:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:43 smithi059 ceph-mon[97450]: pgmap v14750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:43.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:43 smithi055 ceph-mon[97033]: pgmap v14750: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:43.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:43.769882+0000) 2022-09-08T05:13:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:43.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:43.769882+0000) 2022-09-08T05:13:45.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:44.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:44.770106+0000) 2022-09-08T05:13:45.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:44.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:44.770098+0000) 2022-09-08T05:13:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:45 smithi059 ceph-mon[97450]: pgmap v14751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:45.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:45 smithi055 ceph-mon[97033]: pgmap v14751: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:45.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:45.770344+0000) 2022-09-08T05:13:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:45.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:45.770344+0000) 2022-09-08T05:13:46.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:46 smithi055 ceph-mon[97033]: pgmap v14752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:13:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:46 smithi059 ceph-mon[97450]: pgmap v14752: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:13:47.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:46.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:46.770596+0000) 2022-09-08T05:13:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:46.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:46.770595+0000) 2022-09-08T05:13:47.769 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:13:47.469+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:13:48.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:47.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:47.770837+0000) 2022-09-08T05:13:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:47.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:47.770836+0000) 2022-09-08T05:13:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:48.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:48.771039+0000) 2022-09-08T05:13:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:48.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:48.771041+0000) 2022-09-08T05:13:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:49 smithi055 ceph-mon[97033]: pgmap v14753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:49 smithi059 ceph-mon[97450]: pgmap v14753: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:50.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:49.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:49.771216+0000) 2022-09-08T05:13:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:49.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:49.771217+0000) 2022-09-08T05:13:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:50.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:50.771465+0000) 2022-09-08T05:13:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:50.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:50.771465+0000) 2022-09-08T05:13:51.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:51 smithi055 ceph-mon[97033]: pgmap v14754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:13:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:51 smithi059 ceph-mon[97450]: pgmap v14754: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:13:52.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:51.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:51.771778+0000) 2022-09-08T05:13:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:51.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:51.771776+0000) 2022-09-08T05:13:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:52.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:52.772020+0000) 2022-09-08T05:13:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:52.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:52.772020+0000) 2022-09-08T05:13:53.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:53 smithi055 ceph-mon[97033]: pgmap v14755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:53 smithi059 ceph-mon[97450]: pgmap v14755: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:54.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:53.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:53.772201+0000) 2022-09-08T05:13:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:53.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:53.772201+0000) 2022-09-08T05:13:55.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:54.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:54.772463+0000) 2022-09-08T05:13:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:54.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:54.772461+0000) 2022-09-08T05:13:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:55 smithi059 ceph-mon[97450]: pgmap v14756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:55.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:55 smithi055 ceph-mon[97033]: pgmap v14756: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:56.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:55.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:55.772691+0000) 2022-09-08T05:13:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:55.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:55.772691+0000) 2022-09-08T05:13:57.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:56.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:56.772900+0000) 2022-09-08T05:13:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:56.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:56.772902+0000) 2022-09-08T05:13:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:57 smithi059 ceph-mon[97450]: pgmap v14757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:13:57.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:57 smithi055 ceph-mon[97033]: pgmap v14757: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:13:58.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:57.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:57.773215+0000) 2022-09-08T05:13:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:57.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:57.773212+0000) 2022-09-08T05:13:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:58.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:58.773489+0000) 2022-09-08T05:13:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:58.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:58.773489+0000) 2022-09-08T05:13:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:13:59 smithi059 ceph-mon[97450]: pgmap v14758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:13:59.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:13:59 smithi055 ceph-mon[97033]: pgmap v14758: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:00.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:13:59.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:59.773801+0000) 2022-09-08T05:14:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:13:59.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:13:59.773798+0000) 2022-09-08T05:14:01.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:00.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:00.774084+0000) 2022-09-08T05:14:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:00.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:00.774084+0000) 2022-09-08T05:14:01.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:01 smithi055 ceph-mon[97033]: pgmap v14759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:14:01.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:01 smithi059 ceph-mon[97450]: pgmap v14759: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:14:02.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:01.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:01.774351+0000) 2022-09-08T05:14:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:01.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:01.774351+0000) 2022-09-08T05:14:02.773 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:14:02.479+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:14:03.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:02.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:02.774614+0000) 2022-09-08T05:14:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:02.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:02.774611+0000) 2022-09-08T05:14:03.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:03 smithi055 ceph-mon[97033]: pgmap v14760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:03 smithi059 ceph-mon[97450]: pgmap v14760: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:04.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:03.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:03.774848+0000) 2022-09-08T05:14:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:03.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:03.774848+0000) 2022-09-08T05:14:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:04.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:04.775050+0000) 2022-09-08T05:14:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:04.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:04.775093+0000) 2022-09-08T05:14:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:05 smithi059 ceph-mon[97450]: pgmap v14761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:05.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:05 smithi055 ceph-mon[97033]: pgmap v14761: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:06.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:05.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:05.775247+0000) 2022-09-08T05:14:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:05.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:05.775302+0000) 2022-09-08T05:14:06.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:14:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:14:06.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:14:06.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:14:07.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:06.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:06.775484+0000) 2022-09-08T05:14:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:06.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:06.775470+0000) 2022-09-08T05:14:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:07 smithi059 ceph-mon[97450]: pgmap v14762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:14:07.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:07 smithi055 ceph-mon[97033]: pgmap v14762: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:14:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:07.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:07.775772+0000) 2022-09-08T05:14:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:07.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:07.775738+0000) 2022-09-08T05:14:08.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:08 smithi059 ceph-mon[97450]: pgmap v14763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:14:08.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:08 smithi055 ceph-mon[97033]: pgmap v14763: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:14:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:08.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:08.776034+0000) 2022-09-08T05:14:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:08.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:08.775990+0000) 2022-09-08T05:14:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:09.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:09.776307+0000) 2022-09-08T05:14:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:09.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:09.776271+0000) 2022-09-08T05:14:10.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:10 smithi059 ceph-mon[97450]: osdmap e1003: 12 total, 11 up, 11 in 2022-09-08T05:14:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:10 smithi059 ceph-mon[97450]: pgmap v14765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:14:10.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:10 smithi055 ceph-mon[97033]: osdmap e1003: 12 total, 11 up, 11 in 2022-09-08T05:14:10.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:10 smithi055 ceph-mon[97033]: pgmap v14765: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:14:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:10.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:10.776544+0000) 2022-09-08T05:14:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:10.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:10.776516+0000) 2022-09-08T05:14:12.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:11.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:11.776780+0000) 2022-09-08T05:14:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:11.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:11.776776+0000) 2022-09-08T05:14:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:12.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:12.776940+0000) 2022-09-08T05:14:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:12.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:12.776909+0000) 2022-09-08T05:14:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:13 smithi055 ceph-mon[97033]: pgmap v14766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:14:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:13 smithi059 ceph-mon[97450]: pgmap v14766: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:14:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:13.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:13.777102+0000) 2022-09-08T05:14:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:13.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:13.777068+0000) 2022-09-08T05:14:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:14.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:14.777278+0000) 2022-09-08T05:14:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:14.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:14.777302+0000) 2022-09-08T05:14:15.476 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:15 smithi055 ceph-mon[97033]: pgmap v14767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:14:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:15 smithi059 ceph-mon[97450]: pgmap v14767: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:14:16.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:15.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:15.777526+0000) 2022-09-08T05:14:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:15.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:15.777437+0000) 2022-09-08T05:14:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:16.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:16.777650+0000) 2022-09-08T05:14:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:16.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:16.777723+0000) 2022-09-08T05:14:17.777 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:14:17.493+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:14:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:17 smithi059 ceph-mon[97450]: pgmap v14768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:14:17.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:17 smithi055 ceph-mon[97033]: pgmap v14768: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:14:18.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:17.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:17.777884+0000) 2022-09-08T05:14:18.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:17.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:17.777971+0000) 2022-09-08T05:14:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:18.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:18.778048+0000) 2022-09-08T05:14:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:18.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:18.778231+0000) 2022-09-08T05:14:19.790 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:19 smithi059 ceph-mon[97450]: pgmap v14769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:14:19.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:19 smithi055 ceph-mon[97033]: pgmap v14769: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:14:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:19.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:19.778216+0000) 2022-09-08T05:14:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:19.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:19.778425+0000) 2022-09-08T05:14:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:20.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:20.778588+0000) 2022-09-08T05:14:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:20.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:20.778362+0000) 2022-09-08T05:14:21.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:21 smithi055 ceph-mon[97033]: pgmap v14770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 192 B/s wr, 8 op/s 2022-09-08T05:14:21.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:21 smithi055 ceph-mon[97033]: osdmap e1004: 12 total, 11 up, 11 in 2022-09-08T05:14:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:21 smithi059 ceph-mon[97450]: pgmap v14770: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 192 B/s wr, 8 op/s 2022-09-08T05:14:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:21 smithi059 ceph-mon[97450]: osdmap e1004: 12 total, 11 up, 11 in 2022-09-08T05:14:22.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:21.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:21.778795+0000) 2022-09-08T05:14:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:21.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:21.778578+0000) 2022-09-08T05:14:23.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:22.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:22.778927+0000) 2022-09-08T05:14:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:22.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:22.778805+0000) 2022-09-08T05:14:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:23 smithi059 ceph-mon[97450]: pgmap v14772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:14:23.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:23 smithi055 ceph-mon[97033]: pgmap v14772: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:14:24.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:23.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:23.778954+0000) 2022-09-08T05:14:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:23.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:23.779071+0000) 2022-09-08T05:14:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:24.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:24.779104+0000) 2022-09-08T05:14:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:24.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:24.779227+0000) 2022-09-08T05:14:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:25 smithi059 ceph-mon[97450]: pgmap v14773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:14:25.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:25 smithi055 ceph-mon[97033]: pgmap v14773: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:14:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:25.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:25.779339+0000) 2022-09-08T05:14:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:25.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:25.779434+0000) 2022-09-08T05:14:27.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:26.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:26.779548+0000) 2022-09-08T05:14:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:26.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:26.779603+0000) 2022-09-08T05:14:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:27 smithi059 ceph-mon[97450]: pgmap v14774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:14:27.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:27 smithi055 ceph-mon[97033]: pgmap v14774: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:14:28.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:27.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:27.779776+0000) 2022-09-08T05:14:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:27.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:27.779822+0000) 2022-09-08T05:14:28.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:14:28.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:14:28.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:14:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:14:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:14:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:14:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:28.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:28.780026+0000) 2022-09-08T05:14:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:28.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:28.780065+0000) 2022-09-08T05:14:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:29 smithi059 ceph-mon[97450]: pgmap v14775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:14:29.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:29 smithi055 ceph-mon[97033]: pgmap v14775: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:14:30.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:29.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:29.780258+0000) 2022-09-08T05:14:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:29.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:29.780297+0000) 2022-09-08T05:14:31.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:30.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:30.780469+0000) 2022-09-08T05:14:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:30.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:30.780483+0000) 2022-09-08T05:14:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:31 smithi059 ceph-mon[97450]: pgmap v14776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:14:31.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:31 smithi055 ceph-mon[97033]: pgmap v14776: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:14:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:31.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:31.780762+0000) 2022-09-08T05:14:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:31.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:31.780762+0000) 2022-09-08T05:14:32.780 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:14:32.506+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:14:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:32 smithi059 ceph-mon[97450]: pgmap v14777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:14:32.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:32 smithi055 ceph-mon[97033]: pgmap v14777: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:14:33.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:32.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:32.781052+0000) 2022-09-08T05:14:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:32.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:32.781005+0000) 2022-09-08T05:14:34.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:33.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:33.781234+0000) 2022-09-08T05:14:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:33.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:33.781246+0000) 2022-09-08T05:14:35.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:34.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:34.781487+0000) 2022-09-08T05:14:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:34.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:34.781510+0000) 2022-09-08T05:14:35.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:35 smithi055 ceph-mon[97033]: pgmap v14778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:35 smithi059 ceph-mon[97450]: pgmap v14778: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:36.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:35.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:35.781750+0000) 2022-09-08T05:14:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:35.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:35.781783+0000) 2022-09-08T05:14:37.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:36.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:36.782007+0000) 2022-09-08T05:14:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:36.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:36.782041+0000) 2022-09-08T05:14:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:37 smithi059 ceph-mon[97450]: pgmap v14779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:14:37.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:37 smithi055 ceph-mon[97033]: pgmap v14779: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:14:38.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:37.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:37.782299+0000) 2022-09-08T05:14:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:37.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:37.782305+0000) 2022-09-08T05:14:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:38.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:38.782434+0000) 2022-09-08T05:14:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:38.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:38.782497+0000) 2022-09-08T05:14:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:39 smithi059 ceph-mon[97450]: pgmap v14780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:39.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:39 smithi055 ceph-mon[97033]: pgmap v14780: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:40.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:39.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:39.782706+0000) 2022-09-08T05:14:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:39.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:39.782754+0000) 2022-09-08T05:14:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:40.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:40.782861+0000) 2022-09-08T05:14:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:40.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:40.782881+0000) 2022-09-08T05:14:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:41 smithi059 ceph-mon[97450]: pgmap v14781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:14:41.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:41 smithi055 ceph-mon[97033]: pgmap v14781: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:14:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:41.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:41.783113+0000) 2022-09-08T05:14:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:41.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:41.783098+0000) 2022-09-08T05:14:43.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:42.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:42.783333+0000) 2022-09-08T05:14:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:42.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:42.783361+0000) 2022-09-08T05:14:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:43 smithi059 ceph-mon[97450]: pgmap v14782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:43 smithi055 ceph-mon[97033]: pgmap v14782: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:43.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:43.783565+0000) 2022-09-08T05:14:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:43.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:43.783556+0000) 2022-09-08T05:14:45.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:44.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:44.783823+0000) 2022-09-08T05:14:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:44.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:44.784018+0000) 2022-09-08T05:14:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:45 smithi059 ceph-mon[97450]: pgmap v14783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:45.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:45 smithi055 ceph-mon[97033]: pgmap v14783: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:46.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:45.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:45.784081+0000) 2022-09-08T05:14:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:45.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:45.784188+0000) 2022-09-08T05:14:46.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:46 smithi055 ceph-mon[97033]: pgmap v14784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:14:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:46 smithi059 ceph-mon[97450]: pgmap v14784: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:14:47.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:46.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:46.784252+0000) 2022-09-08T05:14:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:46.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:46.784390+0000) 2022-09-08T05:14:47.786 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:14:47.521+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:14:48.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:47.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:47.784453+0000) 2022-09-08T05:14:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:47.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:47.784558+0000) 2022-09-08T05:14:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:48.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:48.784745+0000) 2022-09-08T05:14:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:48.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:48.784782+0000) 2022-09-08T05:14:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:49 smithi055 ceph-mon[97033]: pgmap v14785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:49 smithi059 ceph-mon[97450]: pgmap v14785: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:14:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:49.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:49.784882+0000) 2022-09-08T05:14:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:49.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:49.784933+0000) 2022-09-08T05:14:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:50.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:50.785116+0000) 2022-09-08T05:14:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:50.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:50.785123+0000) 2022-09-08T05:14:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:51 smithi059 ceph-mon[97450]: pgmap v14786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:14:51.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:51 smithi055 ceph-mon[97033]: pgmap v14786: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:14:52.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:51.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:51.785347+0000) 2022-09-08T05:14:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:51.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:51.785353+0000) 2022-09-08T05:14:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:52.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:52.785598+0000) 2022-09-08T05:14:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:52.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:52.785600+0000) 2022-09-08T05:14:53.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:53 smithi059 ceph-mon[97450]: pgmap v14787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:14:53.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:53 smithi055 ceph-mon[97033]: pgmap v14787: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:14:54.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:53.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:53.785832+0000) 2022-09-08T05:14:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:53.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:53.785820+0000) 2022-09-08T05:14:55.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:54.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:54.786017+0000) 2022-09-08T05:14:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:54.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:54.786005+0000) 2022-09-08T05:14:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:55 smithi059 ceph-mon[97450]: pgmap v14788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:14:55.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:55 smithi055 ceph-mon[97033]: pgmap v14788: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:14:56.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:55.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:55.786303+0000) 2022-09-08T05:14:56.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:55.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:55.786240+0000) 2022-09-08T05:14:57.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:56.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:56.786580+0000) 2022-09-08T05:14:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:56.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:56.786522+0000) 2022-09-08T05:14:57.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:57 smithi059 ceph-mon[97450]: pgmap v14789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:14:57.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:57 smithi055 ceph-mon[97033]: pgmap v14789: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:14:58.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:57.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:57.786806+0000) 2022-09-08T05:14:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:57.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:57.786766+0000) 2022-09-08T05:14:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:58.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:58.786992+0000) 2022-09-08T05:14:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:58.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:58.786967+0000) 2022-09-08T05:14:59.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:59 smithi055 ceph-mon[97033]: pgmap v14790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:14:59.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:59 smithi055 ceph-mon[97033]: pgmap v14791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:14:59.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:59 smithi055 ceph-mon[97033]: pgmap v14792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:14:59.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:14:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:14:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:59 smithi059 ceph-mon[97450]: pgmap v14790: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:14:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:59 smithi059 ceph-mon[97450]: pgmap v14791: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:14:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:59 smithi059 ceph-mon[97450]: pgmap v14792: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:14:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:14:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:15:00.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:14:59.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:59.787199+0000) 2022-09-08T05:15:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:14:59.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:14:59.787219+0000) 2022-09-08T05:15:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:00.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:00.787337+0000) 2022-09-08T05:15:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:00.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:00.787389+0000) 2022-09-08T05:15:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:01 smithi059 ceph-mon[97450]: pgmap v14793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:15:01.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:01 smithi055 ceph-mon[97033]: pgmap v14793: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:15:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:01.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:01.787521+0000) 2022-09-08T05:15:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:01.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:01.787555+0000) 2022-09-08T05:15:02.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:15:02.538+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:15:02.909 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:02.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:02.787725+0000) 2022-09-08T05:15:02.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:02.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:02.787738+0000) 2022-09-08T05:15:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:03 smithi059 ceph-mon[97450]: pgmap v14794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:15:03.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:03 smithi055 ceph-mon[97033]: pgmap v14794: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:15:04.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:03.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:03.787899+0000) 2022-09-08T05:15:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:03.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:03.787941+0000) 2022-09-08T05:15:05.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:04.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:04.788119+0000) 2022-09-08T05:15:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:04.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:04.788077+0000) 2022-09-08T05:15:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:05 smithi059 ceph-mon[97450]: pgmap v14795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:15:05.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:05 smithi055 ceph-mon[97033]: pgmap v14795: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:15:06.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:05.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:05.788290+0000) 2022-09-08T05:15:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:05.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:05.788367+0000) 2022-09-08T05:15:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:15:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:15:06.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:15:06.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:15:07.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:06.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:06.788495+0000) 2022-09-08T05:15:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:06.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:06.788547+0000) 2022-09-08T05:15:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:07 smithi059 ceph-mon[97450]: pgmap v14796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T05:15:07.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:07 smithi055 ceph-mon[97033]: pgmap v14796: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T05:15:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:07.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:07.788621+0000) 2022-09-08T05:15:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:07.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:07.788709+0000) 2022-09-08T05:15:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:08 smithi059 ceph-mon[97450]: pgmap v14797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:15:08.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:08 smithi055 ceph-mon[97033]: pgmap v14797: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:15:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:08.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:08.788763+0000) 2022-09-08T05:15:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:08.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:08.788846+0000) 2022-09-08T05:15:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:09.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:09.788880+0000) 2022-09-08T05:15:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:09.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:09.788991+0000) 2022-09-08T05:15:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:10 smithi059 ceph-mon[97450]: osdmap e1005: 12 total, 11 up, 11 in 2022-09-08T05:15:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:10 smithi059 ceph-mon[97450]: pgmap v14799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:15:10.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:10 smithi055 ceph-mon[97033]: osdmap e1005: 12 total, 11 up, 11 in 2022-09-08T05:15:10.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:10 smithi055 ceph-mon[97033]: pgmap v14799: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:15:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:10.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:10.789051+0000) 2022-09-08T05:15:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:10.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:10.789216+0000) 2022-09-08T05:15:12.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:11.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:11.789269+0000) 2022-09-08T05:15:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:11.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:11.789414+0000) 2022-09-08T05:15:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:12.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:12.789483+0000) 2022-09-08T05:15:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:12.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:12.789683+0000) 2022-09-08T05:15:13.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:13 smithi055 ceph-mon[97033]: pgmap v14800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:15:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:13 smithi059 ceph-mon[97450]: pgmap v14800: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:15:14.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:13.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:13.789701+0000) 2022-09-08T05:15:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:13.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:13.789877+0000) 2022-09-08T05:15:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:14.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:14.789870+0000) 2022-09-08T05:15:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:14.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:14.790013+0000) 2022-09-08T05:15:15.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:15 smithi055 ceph-mon[97033]: pgmap v14801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:15:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:15 smithi059 ceph-mon[97450]: pgmap v14801: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:15:16.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:15.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:15.790038+0000) 2022-09-08T05:15:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:15.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:15.790181+0000) 2022-09-08T05:15:16.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:16 smithi059 ceph-mon[97450]: osdmap e1006: 12 total, 11 up, 11 in 2022-09-08T05:15:16.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:16 smithi055 ceph-mon[97033]: osdmap e1006: 12 total, 11 up, 11 in 2022-09-08T05:15:17.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:16.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:16.790259+0000) 2022-09-08T05:15:17.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:16.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:16.790407+0000) 2022-09-08T05:15:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:17 smithi059 ceph-mon[97450]: pgmap v14803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:15:17.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:15:17.545+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:15:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:17.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:17.790618+0000) 2022-09-08T05:15:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:17.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:17.790527+0000) 2022-09-08T05:15:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:17 smithi055 ceph-mon[97033]: pgmap v14803: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:15:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:18.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:18.790786+0000) 2022-09-08T05:15:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:18.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:18.790862+0000) 2022-09-08T05:15:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:19 smithi059 ceph-mon[97450]: pgmap v14804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T05:15:19.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:19 smithi055 ceph-mon[97033]: pgmap v14804: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T05:15:20.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:19.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:19.790925+0000) 2022-09-08T05:15:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:19.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:19.791068+0000) 2022-09-08T05:15:21.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:20.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:20.791120+0000) 2022-09-08T05:15:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:20.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:20.791241+0000) 2022-09-08T05:15:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:21 smithi059 ceph-mon[97450]: pgmap v14805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:15:21.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:21 smithi055 ceph-mon[97033]: pgmap v14805: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:15:22.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:21.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:21.791355+0000) 2022-09-08T05:15:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:21.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:21.791413+0000) 2022-09-08T05:15:23.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:22.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:22.791585+0000) 2022-09-08T05:15:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:22.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:22.791619+0000) 2022-09-08T05:15:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:23 smithi059 ceph-mon[97450]: pgmap v14806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:15:23.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:23 smithi055 ceph-mon[97033]: pgmap v14806: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:15:24.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:23.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:23.791700+0000) 2022-09-08T05:15:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:23.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:23.791801+0000) 2022-09-08T05:15:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:24.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:24.791849+0000) 2022-09-08T05:15:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:24.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:24.791943+0000) 2022-09-08T05:15:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:25 smithi059 ceph-mon[97450]: pgmap v14807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:15:25.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:25 smithi055 ceph-mon[97033]: pgmap v14807: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:15:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:25.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:25.792023+0000) 2022-09-08T05:15:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:25.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:25.792092+0000) 2022-09-08T05:15:27.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:26.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:26.792301+0000) 2022-09-08T05:15:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:26.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:26.792224+0000) 2022-09-08T05:15:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:27 smithi059 ceph-mon[97450]: pgmap v14808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:15:27.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:27 smithi055 ceph-mon[97033]: pgmap v14808: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:15:28.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:27.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:27.792556+0000) 2022-09-08T05:15:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:27.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:27.792457+0000) 2022-09-08T05:15:28.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:28 smithi055 ceph-mon[97033]: pgmap v14809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:28 smithi059 ceph-mon[97450]: pgmap v14809: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:29.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:28.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:28.792720+0000) 2022-09-08T05:15:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:28.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:28.792769+0000) 2022-09-08T05:15:30.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:29.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:29.792880+0000) 2022-09-08T05:15:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:29.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:29.792965+0000) 2022-09-08T05:15:31.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:30.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:30.793125+0000) 2022-09-08T05:15:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:30.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:30.793067+0000) 2022-09-08T05:15:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:31 smithi059 ceph-mon[97450]: pgmap v14810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:15:31.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:31 smithi055 ceph-mon[97033]: pgmap v14810: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:15:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:31.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:31.793290+0000) 2022-09-08T05:15:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:31.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:31.793344+0000) 2022-09-08T05:15:32.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:15:32.555+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:15:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:32.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:32.793519+0000) 2022-09-08T05:15:32.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:32.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:32.793579+0000) 2022-09-08T05:15:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:33 smithi059 ceph-mon[97450]: pgmap v14811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:33.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:33 smithi055 ceph-mon[97033]: pgmap v14811: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:34.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:33.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:33.793792+0000) 2022-09-08T05:15:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:33.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:33.793813+0000) 2022-09-08T05:15:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:34 smithi059 ceph-mon[97450]: pgmap v14812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:34.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:34 smithi055 ceph-mon[97033]: pgmap v14812: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:34.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:34.793958+0000) 2022-09-08T05:15:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:34.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:34.793968+0000) 2022-09-08T05:15:36.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:35.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:35.794170+0000) 2022-09-08T05:15:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:35.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:35.794186+0000) 2022-09-08T05:15:37.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:36.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:36.794415+0000) 2022-09-08T05:15:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:36.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:36.794430+0000) 2022-09-08T05:15:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:37 smithi059 ceph-mon[97450]: pgmap v14813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:15:37.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:37 smithi055 ceph-mon[97033]: pgmap v14813: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:15:38.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:37.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:37.794703+0000) 2022-09-08T05:15:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:37.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:37.794734+0000) 2022-09-08T05:15:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:38.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:38.794964+0000) 2022-09-08T05:15:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:38.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:38.794975+0000) 2022-09-08T05:15:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:39 smithi059 ceph-mon[97450]: pgmap v14814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:39.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:39 smithi055 ceph-mon[97033]: pgmap v14814: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:39.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:39.795196+0000) 2022-09-08T05:15:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:39.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:39.795277+0000) 2022-09-08T05:15:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:40.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:40.795463+0000) 2022-09-08T05:15:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:40.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:40.795543+0000) 2022-09-08T05:15:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:41 smithi059 ceph-mon[97450]: pgmap v14815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:15:41.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:41 smithi055 ceph-mon[97033]: pgmap v14815: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:15:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:41.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:41.795764+0000) 2022-09-08T05:15:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:41.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:41.795789+0000) 2022-09-08T05:15:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:42.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:42.796061+0000) 2022-09-08T05:15:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:42.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:42.796035+0000) 2022-09-08T05:15:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:43 smithi059 ceph-mon[97450]: pgmap v14816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:43 smithi055 ceph-mon[97033]: pgmap v14816: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:43.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:43.796278+0000) 2022-09-08T05:15:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:43.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:43.796240+0000) 2022-09-08T05:15:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:44 smithi059 ceph-mon[97450]: pgmap v14817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:44.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:44 smithi055 ceph-mon[97033]: pgmap v14817: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:45.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:44.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:44.796499+0000) 2022-09-08T05:15:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:44.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:44.796449+0000) 2022-09-08T05:15:46.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:45.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:45.796713+0000) 2022-09-08T05:15:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:45.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:45.796691+0000) 2022-09-08T05:15:47.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:46.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:46.796909+0000) 2022-09-08T05:15:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:46.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:46.796898+0000) 2022-09-08T05:15:47.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:47 smithi059 ceph-mon[97450]: pgmap v14818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:15:47.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:15:47.559+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:15:47.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:47.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:47.797067+0000) 2022-09-08T05:15:47.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:47.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:47.797078+0000) 2022-09-08T05:15:47.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:47 smithi055 ceph-mon[97033]: pgmap v14818: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:15:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:48.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:48.797299+0000) 2022-09-08T05:15:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:48.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:48.797320+0000) 2022-09-08T05:15:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:49 smithi059 ceph-mon[97450]: pgmap v14819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:49.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:49 smithi055 ceph-mon[97033]: pgmap v14819: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:50.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:49.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:49.797542+0000) 2022-09-08T05:15:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:49.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:49.797543+0000) 2022-09-08T05:15:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:50.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:50.797768+0000) 2022-09-08T05:15:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:50.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:50.797756+0000) 2022-09-08T05:15:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:51 smithi059 ceph-mon[97450]: pgmap v14820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:15:51.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:51 smithi055 ceph-mon[97033]: pgmap v14820: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:15:52.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:51.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:51.797928+0000) 2022-09-08T05:15:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:51.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:51.797942+0000) 2022-09-08T05:15:52.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:52 smithi055 ceph-mon[97033]: pgmap v14821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:52 smithi059 ceph-mon[97450]: pgmap v14821: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:52.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:52.798065+0000) 2022-09-08T05:15:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:52.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:52.798105+0000) 2022-09-08T05:15:54.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:53.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:53.798289+0000) 2022-09-08T05:15:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:53.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:53.798239+0000) 2022-09-08T05:15:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:54.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:54.798444+0000) 2022-09-08T05:15:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:54.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:54.798416+0000) 2022-09-08T05:15:55.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:55 smithi055 ceph-mon[97033]: pgmap v14822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:55 smithi059 ceph-mon[97450]: pgmap v14822: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:56.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:55.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:55.798671+0000) 2022-09-08T05:15:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:55.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:55.798699+0000) 2022-09-08T05:15:57.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:56.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:56.798853+0000) 2022-09-08T05:15:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:56.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:56.798917+0000) 2022-09-08T05:15:57.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:57 smithi059 ceph-mon[97450]: pgmap v14823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:15:57.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:57 smithi055 ceph-mon[97033]: pgmap v14823: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:15:58.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:57.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:57.799082+0000) 2022-09-08T05:15:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:57.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:57.799141+0000) 2022-09-08T05:15:58.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:58.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:58.799227+0000) 2022-09-08T05:15:58.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:58.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:58.799341+0000) 2022-09-08T05:15:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:59 smithi059 ceph-mon[97450]: pgmap v14824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:15:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:15:59.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:15:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:15:59.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:59 smithi055 ceph-mon[97033]: pgmap v14824: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:15:59.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:15:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:15:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:15:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:16:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:15:59.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:59.799400+0000) 2022-09-08T05:16:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:15:59.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:15:59.799518+0000) 2022-09-08T05:16:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:00.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:00.799627+0000) 2022-09-08T05:16:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:00.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:00.799702+0000) 2022-09-08T05:16:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:01 smithi059 ceph-mon[97450]: pgmap v14825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:16:01.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:01 smithi055 ceph-mon[97033]: pgmap v14825: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:16:02.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:01.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:01.799896+0000) 2022-09-08T05:16:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:01.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:01.799801+0000) 2022-09-08T05:16:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:02 smithi059 ceph-mon[97450]: pgmap v14826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:16:02.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:16:02.574+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:16:02.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:02.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:02.799937+0000) 2022-09-08T05:16:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:02.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:02.800070+0000) 2022-09-08T05:16:02.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:02 smithi055 ceph-mon[97033]: pgmap v14826: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:16:04.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:03.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:03.800107+0000) 2022-09-08T05:16:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:03.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:03.800223+0000) 2022-09-08T05:16:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:04.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:04.800291+0000) 2022-09-08T05:16:05.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:04.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:04.800373+0000) 2022-09-08T05:16:05.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:05 smithi059 ceph-mon[97450]: pgmap v14827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:16:05.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:05 smithi055 ceph-mon[97033]: pgmap v14827: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:16:06.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:05.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:05.800497+0000) 2022-09-08T05:16:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:05.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:05.800593+0000) 2022-09-08T05:16:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:16:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:16:06.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:16:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:16:07.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:06.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:06.800735+0000) 2022-09-08T05:16:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:06.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:06.800832+0000) 2022-09-08T05:16:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:07 smithi059 ceph-mon[97450]: pgmap v14828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:16:07.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:07 smithi055 ceph-mon[97033]: pgmap v14828: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:16:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:07.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:07.800958+0000) 2022-09-08T05:16:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:07.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:07.801030+0000) 2022-09-08T05:16:08.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:08 smithi059 ceph-mon[97450]: pgmap v14829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:16:08.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:08 smithi055 ceph-mon[97033]: pgmap v14829: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:16:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:08.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:08.801202+0000) 2022-09-08T05:16:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:08.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:08.801251+0000) 2022-09-08T05:16:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:09.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:09.801384+0000) 2022-09-08T05:16:10.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:09.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:09.801439+0000) 2022-09-08T05:16:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:10 smithi059 ceph-mon[97450]: osdmap e1007: 12 total, 11 up, 11 in 2022-09-08T05:16:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:10 smithi059 ceph-mon[97450]: pgmap v14831: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:16:10.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:10 smithi055 ceph-mon[97033]: osdmap e1007: 12 total, 11 up, 11 in 2022-09-08T05:16:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:10 smithi055 ceph-mon[97033]: pgmap v14831: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:16:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:10.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:10.801530+0000) 2022-09-08T05:16:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:10.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:10.801597+0000) 2022-09-08T05:16:12.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:11.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:11.801742+0000) 2022-09-08T05:16:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:11.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:11.801849+0000) 2022-09-08T05:16:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:12.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:12.801975+0000) 2022-09-08T05:16:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:12.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:12.802108+0000) 2022-09-08T05:16:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:13 smithi059 ceph-mon[97450]: pgmap v14832: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:16:13.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:13 smithi055 ceph-mon[97033]: pgmap v14832: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:16:14.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:13.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:13.802267+0000) 2022-09-08T05:16:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:13.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:13.802170+0000) 2022-09-08T05:16:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:14.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:14.802460+0000) 2022-09-08T05:16:15.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:14.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:14.802414+0000) 2022-09-08T05:16:15.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:15 smithi059 ceph-mon[97450]: pgmap v14833: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:16:15.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:15 smithi055 ceph-mon[97033]: pgmap v14833: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:16:16.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:15.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:15.802699+0000) 2022-09-08T05:16:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:15.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:15.802629+0000) 2022-09-08T05:16:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:16.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:16.802820+0000) 2022-09-08T05:16:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:16.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:16.802894+0000) 2022-09-08T05:16:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:17 smithi059 ceph-mon[97450]: pgmap v14834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:16:17.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:16:17.583+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:16:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:17.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:17.803057+0000) 2022-09-08T05:16:17.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:17.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:17.803143+0000) 2022-09-08T05:16:17.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:17 smithi055 ceph-mon[97033]: pgmap v14834: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:16:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:18.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:18.803236+0000) 2022-09-08T05:16:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:18.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:18.803327+0000) 2022-09-08T05:16:19.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:19 smithi059 ceph-mon[97450]: pgmap v14835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:16:19.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:19 smithi055 ceph-mon[97033]: pgmap v14835: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:16:20.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:19.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:19.803473+0000) 2022-09-08T05:16:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:19.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:19.803557+0000) 2022-09-08T05:16:21.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:20.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:20.803655+0000) 2022-09-08T05:16:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:20.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:20.803735+0000) 2022-09-08T05:16:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:21 smithi059 ceph-mon[97450]: pgmap v14836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T05:16:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:21 smithi059 ceph-mon[97450]: osdmap e1008: 12 total, 11 up, 11 in 2022-09-08T05:16:21.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:21 smithi055 ceph-mon[97033]: pgmap v14836: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T05:16:21.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:21 smithi055 ceph-mon[97033]: osdmap e1008: 12 total, 11 up, 11 in 2022-09-08T05:16:22.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:21.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:21.803902+0000) 2022-09-08T05:16:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:21.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:21.803933+0000) 2022-09-08T05:16:23.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:22.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:22.804161+0000) 2022-09-08T05:16:23.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:22.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:22.804167+0000) 2022-09-08T05:16:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:23 smithi059 ceph-mon[97450]: pgmap v14838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:16:23.900 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:23.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:23.804377+0000) 2022-09-08T05:16:23.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:23.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:23.804336+0000) 2022-09-08T05:16:23.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:23 smithi055 ceph-mon[97033]: pgmap v14838: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:16:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:24.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:24.804560+0000) 2022-09-08T05:16:25.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:24.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:24.804554+0000) 2022-09-08T05:16:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:25 smithi059 ceph-mon[97450]: pgmap v14839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:16:25.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:25 smithi055 ceph-mon[97033]: pgmap v14839: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:16:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:25.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:25.804824+0000) 2022-09-08T05:16:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:25.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:25.804810+0000) 2022-09-08T05:16:26.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:26 smithi055 ceph-mon[97033]: pgmap v14840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:16:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:26 smithi059 ceph-mon[97450]: pgmap v14840: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:16:27.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:26.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:26.805040+0000) 2022-09-08T05:16:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:26.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:26.805031+0000) 2022-09-08T05:16:28.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:27.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:27.805283+0000) 2022-09-08T05:16:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:27.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:27.805325+0000) 2022-09-08T05:16:29.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:28.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:28.805564+0000) 2022-09-08T05:16:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:28.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:28.805564+0000) 2022-09-08T05:16:29.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:29 smithi059 ceph-mon[97450]: pgmap v14841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:16:29.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:29 smithi055 ceph-mon[97033]: pgmap v14841: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:16:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:29.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:29.805818+0000) 2022-09-08T05:16:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:29.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:29.805817+0000) 2022-09-08T05:16:31.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:30.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:30.806055+0000) 2022-09-08T05:16:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:30.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:30.806052+0000) 2022-09-08T05:16:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:31 smithi059 ceph-mon[97450]: pgmap v14842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:16:31.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:31 smithi055 ceph-mon[97033]: pgmap v14842: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:16:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:31.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:31.806266+0000) 2022-09-08T05:16:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:31.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:31.806266+0000) 2022-09-08T05:16:32.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:16:32.599+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:16:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:32.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:32.806552+0000) 2022-09-08T05:16:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:32.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:32.806550+0000) 2022-09-08T05:16:33.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:33 smithi055 ceph-mon[97033]: pgmap v14843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:16:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:16:33.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:16:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:16:33.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:33 smithi059 ceph-mon[97450]: pgmap v14843: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:16:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:16:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:16:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:16:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:33.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:33.806735+0000) 2022-09-08T05:16:34.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:33.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:33.806735+0000) 2022-09-08T05:16:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:34 smithi059 ceph-mon[97450]: pgmap v14844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:16:34.976 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:34 smithi055 ceph-mon[97033]: pgmap v14844: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:16:35.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:34.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:34.806952+0000) 2022-09-08T05:16:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:34.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:34.807063+0000) 2022-09-08T05:16:36.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:35.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:35.807202+0000) 2022-09-08T05:16:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:35.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:35.807226+0000) 2022-09-08T05:16:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:36 smithi059 ceph-mon[97450]: pgmap v14845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T05:16:36.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:36 smithi055 ceph-mon[97033]: pgmap v14845: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T05:16:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:36.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:36.807454+0000) 2022-09-08T05:16:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:36.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:36.807454+0000) 2022-09-08T05:16:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:37.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:37.807734+0000) 2022-09-08T05:16:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:37.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:37.807732+0000) 2022-09-08T05:16:38.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:38 smithi059 ceph-mon[97450]: pgmap v14846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:16:38.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:38 smithi055 ceph-mon[97033]: pgmap v14846: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:16:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:38.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:38.807900+0000) 2022-09-08T05:16:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:38.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:38.807897+0000) 2022-09-08T05:16:39.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:39 smithi055 ceph-mon[97033]: pgmap v14847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:16:40.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:39 smithi059 ceph-mon[97450]: pgmap v14847: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:16:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:39.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:39.808147+0000) 2022-09-08T05:16:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:39.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:39.808148+0000) 2022-09-08T05:16:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:40.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:40.808331+0000) 2022-09-08T05:16:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:40.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:40.808331+0000) 2022-09-08T05:16:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:41.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:41.808598+0000) 2022-09-08T05:16:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:41.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:41.808598+0000) 2022-09-08T05:16:42.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:42 smithi059 ceph-mon[97450]: pgmap v14848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:16:42.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:42 smithi055 ceph-mon[97033]: pgmap v14848: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:16:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:42.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:42.808870+0000) 2022-09-08T05:16:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:42.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:42.808870+0000) 2022-09-08T05:16:43.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:43 smithi055 ceph-mon[97033]: pgmap v14849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:16:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:43 smithi059 ceph-mon[97450]: pgmap v14849: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:16:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:43.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:43.809136+0000) 2022-09-08T05:16:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:43.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:43.809136+0000) 2022-09-08T05:16:45.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:44.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:44.809391+0000) 2022-09-08T05:16:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:44.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:44.809391+0000) 2022-09-08T05:16:46.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:45.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:45.809693+0000) 2022-09-08T05:16:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:45.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:45.809693+0000) 2022-09-08T05:16:46.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:46 smithi059 ceph-mon[97450]: pgmap v14850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:16:46.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:46 smithi055 ceph-mon[97033]: pgmap v14850: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:16:47.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:46.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:46.809953+0000) 2022-09-08T05:16:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:46.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:46.809953+0000) 2022-09-08T05:16:47.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:16:47.604+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:16:47.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:47.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:47.810148+0000) 2022-09-08T05:16:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:47.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:47.810192+0000) 2022-09-08T05:16:48.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:48 smithi059 ceph-mon[97450]: pgmap v14851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:16:48.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:48 smithi055 ceph-mon[97033]: pgmap v14851: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:16:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:48.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:48.810345+0000) 2022-09-08T05:16:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:48.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:48.810329+0000) 2022-09-08T05:16:49.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:49 smithi059 ceph-mon[97450]: pgmap v14852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:16:49.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:49 smithi055 ceph-mon[97033]: pgmap v14852: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:16:50.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:49.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:49.810578+0000) 2022-09-08T05:16:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:49.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:49.810570+0000) 2022-09-08T05:16:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:50.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:50.810757+0000) 2022-09-08T05:16:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:50.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:50.810751+0000) 2022-09-08T05:16:52.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:51.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:51.810884+0000) 2022-09-08T05:16:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:51.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:51.810898+0000) 2022-09-08T05:16:52.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:52 smithi059 ceph-mon[97450]: pgmap v14853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:16:52.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:52 smithi055 ceph-mon[97033]: pgmap v14853: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:16:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:52.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:52.811056+0000) 2022-09-08T05:16:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:52.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:52.811078+0000) 2022-09-08T05:16:53.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:53 smithi055 ceph-mon[97033]: pgmap v14854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:16:54.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:53 smithi059 ceph-mon[97450]: pgmap v14854: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:16:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:53.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:53.811280+0000) 2022-09-08T05:16:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:53.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:53.811320+0000) 2022-09-08T05:16:55.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:54.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:54.811573+0000) 2022-09-08T05:16:55.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:54.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:54.811532+0000) 2022-09-08T05:16:56.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:55.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:55.811811+0000) 2022-09-08T05:16:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:55.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:55.811781+0000) 2022-09-08T05:16:56.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:56 smithi059 ceph-mon[97450]: pgmap v14855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:16:56.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:56 smithi055 ceph-mon[97033]: pgmap v14855: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:16:57.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:56.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:56.812059+0000) 2022-09-08T05:16:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:56.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:56.812022+0000) 2022-09-08T05:16:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:57.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:57.812338+0000) 2022-09-08T05:16:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:57.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:57.812260+0000) 2022-09-08T05:16:58.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:58 smithi059 ceph-mon[97450]: pgmap v14856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:16:58.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:58 smithi055 ceph-mon[97033]: pgmap v14856: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:16:59.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:58.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:58.812589+0000) 2022-09-08T05:16:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:58.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:58.812489+0000) 2022-09-08T05:16:59.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:16:59 smithi059 ceph-mon[97450]: pgmap v14857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:16:59.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:16:59 smithi055 ceph-mon[97033]: pgmap v14857: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:17:00.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:16:59.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:59.812842+0000) 2022-09-08T05:17:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:16:59.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:16:59.812729+0000) 2022-09-08T05:17:01.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:00.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:00.813095+0000) 2022-09-08T05:17:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:00.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:00.812986+0000) 2022-09-08T05:17:02.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:01.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:01.813357+0000) 2022-09-08T05:17:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:01.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:01.813223+0000) 2022-09-08T05:17:02.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:02 smithi059 ceph-mon[97450]: pgmap v14858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:17:02.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:17:02.608+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:17:02.901 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:02.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:02.813579+0000) 2022-09-08T05:17:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:02.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:02.813455+0000) 2022-09-08T05:17:02.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:02 smithi055 ceph-mon[97033]: pgmap v14858: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:17:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:03 smithi055 ceph-mon[97033]: pgmap v14859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:17:04.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:03 smithi059 ceph-mon[97450]: pgmap v14859: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:17:04.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:03.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:03.813679+0000) 2022-09-08T05:17:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:03.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:03.813744+0000) 2022-09-08T05:17:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:04.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:04.813873+0000) 2022-09-08T05:17:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:04.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:04.813946+0000) 2022-09-08T05:17:06.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:05.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:05.814105+0000) 2022-09-08T05:17:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:05.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:05.814177+0000) 2022-09-08T05:17:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:06 smithi059 ceph-mon[97450]: pgmap v14860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:17:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:17:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:17:06.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:06 smithi055 ceph-mon[97033]: pgmap v14860: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:17:06.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:17:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:17:07.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:06.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:06.814306+0000) 2022-09-08T05:17:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:06.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:06.814365+0000) 2022-09-08T05:17:07.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:07 smithi059 ceph-mon[97450]: pgmap v14861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:17:07.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:07 smithi055 ceph-mon[97033]: pgmap v14861: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:17:08.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:07.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:07.814670+0000) 2022-09-08T05:17:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:07.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:07.814545+0000) 2022-09-08T05:17:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:08.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:08.814788+0000) 2022-09-08T05:17:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:08.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:08.814902+0000) 2022-09-08T05:17:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:09.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:09.815107+0000) 2022-09-08T05:17:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:09.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:09.814931+0000) 2022-09-08T05:17:10.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:09 smithi055 ceph-mon[97033]: osdmap e1009: 12 total, 11 up, 11 in 2022-09-08T05:17:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:09 smithi055 ceph-mon[97033]: pgmap v14863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:17:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:09 smithi059 ceph-mon[97450]: osdmap e1009: 12 total, 11 up, 11 in 2022-09-08T05:17:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:09 smithi059 ceph-mon[97450]: pgmap v14863: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:17:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:10.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:10.815113+0000) 2022-09-08T05:17:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:10.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:10.815337+0000) 2022-09-08T05:17:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:11 smithi055 ceph-mon[97033]: pgmap v14864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:17:12.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:11 smithi059 ceph-mon[97450]: pgmap v14864: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:17:12.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:11.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:11.815514+0000) 2022-09-08T05:17:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:11.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:11.815302+0000) 2022-09-08T05:17:13.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:12.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:12.815697+0000) 2022-09-08T05:17:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:12.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:12.815494+0000) 2022-09-08T05:17:14.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:13.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:13.815736+0000) 2022-09-08T05:17:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:13.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:13.815878+0000) 2022-09-08T05:17:14.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:14 smithi055 ceph-mon[97033]: pgmap v14865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:17:14.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:14 smithi059 ceph-mon[97450]: pgmap v14865: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:17:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:14.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:14.815889+0000) 2022-09-08T05:17:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:14.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:14.816029+0000) 2022-09-08T05:17:16.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:15.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:15.816051+0000) 2022-09-08T05:17:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:15.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:15.816252+0000) 2022-09-08T05:17:16.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:16 smithi055 ceph-mon[97033]: pgmap v14866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:17:16.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:16 smithi055 ceph-mon[97033]: osdmap e1010: 12 total, 11 up, 11 in 2022-09-08T05:17:16.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:16 smithi059 ceph-mon[97450]: pgmap v14866: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:17:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:16 smithi059 ceph-mon[97450]: osdmap e1010: 12 total, 11 up, 11 in 2022-09-08T05:17:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:16.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:16.816263+0000) 2022-09-08T05:17:17.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:16.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:16.816418+0000) 2022-09-08T05:17:17.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:17.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:17.816546+0000) 2022-09-08T05:17:17.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:17:17.612+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:17:17.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:17.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:17.816440+0000) 2022-09-08T05:17:18.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:18 smithi055 ceph-mon[97033]: pgmap v14868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-08T05:17:18.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:18 smithi059 ceph-mon[97450]: pgmap v14868: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-08T05:17:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:18.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:18.816687+0000) 2022-09-08T05:17:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:18.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:18.816768+0000) 2022-09-08T05:17:20.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:19.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:19.816864+0000) 2022-09-08T05:17:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:19.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:19.816980+0000) 2022-09-08T05:17:20.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:20 smithi055 ceph-mon[97033]: pgmap v14869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:17:20.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:20 smithi059 ceph-mon[97450]: pgmap v14869: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:17:21.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:20.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:20.817203+0000) 2022-09-08T05:17:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:20.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:20.817082+0000) 2022-09-08T05:17:21.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:21 smithi059 ceph-mon[97450]: pgmap v14870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:17:21.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:21 smithi055 ceph-mon[97033]: pgmap v14870: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:17:22.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:21.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:21.817350+0000) 2022-09-08T05:17:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:21.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:21.817361+0000) 2022-09-08T05:17:23.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:22.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:22.817563+0000) 2022-09-08T05:17:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:22.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:22.817568+0000) 2022-09-08T05:17:24.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:23.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:23.817710+0000) 2022-09-08T05:17:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:23.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:23.817711+0000) 2022-09-08T05:17:24.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:24 smithi055 ceph-mon[97033]: pgmap v14871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:17:24.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:24 smithi059 ceph-mon[97450]: pgmap v14871: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:17:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:24.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:24.817872+0000) 2022-09-08T05:17:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:24.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:24.817878+0000) 2022-09-08T05:17:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:25.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:25.818145+0000) 2022-09-08T05:17:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:25.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:25.818145+0000) 2022-09-08T05:17:26.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:26 smithi055 ceph-mon[97033]: pgmap v14872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:17:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:26 smithi059 ceph-mon[97450]: pgmap v14872: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:17:27.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:26.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:26.818400+0000) 2022-09-08T05:17:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:26.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:26.818400+0000) 2022-09-08T05:17:28.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:27.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:27.818657+0000) 2022-09-08T05:17:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:27.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:27.818663+0000) 2022-09-08T05:17:28.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:28 smithi055 ceph-mon[97033]: pgmap v14873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:17:28.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:28 smithi059 ceph-mon[97450]: pgmap v14873: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:17:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:28.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:28.818810+0000) 2022-09-08T05:17:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:28.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:28.818812+0000) 2022-09-08T05:17:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:29.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:29.818964+0000) 2022-09-08T05:17:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:29.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:29.818964+0000) 2022-09-08T05:17:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:30 smithi055 ceph-mon[97033]: pgmap v14874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:17:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:30 smithi059 ceph-mon[97450]: pgmap v14874: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:17:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:30.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:30.819141+0000) 2022-09-08T05:17:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:30.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:30.819141+0000) 2022-09-08T05:17:31.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:31 smithi059 ceph-mon[97450]: pgmap v14875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:17:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:31 smithi055 ceph-mon[97033]: pgmap v14875: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:17:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:31.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:31.819337+0000) 2022-09-08T05:17:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:31.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:31.819337+0000) 2022-09-08T05:17:32.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:17:32.628+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:17:32.901 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:32.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:32.819593+0000) 2022-09-08T05:17:32.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:32.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:32.819589+0000) 2022-09-08T05:17:34.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:33.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:33.819908+0000) 2022-09-08T05:17:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:33.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:33.819871+0000) 2022-09-08T05:17:34.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:34 smithi055 ceph-mon[97033]: pgmap v14876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:17:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:17:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:17:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:17:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:17:34.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:34 smithi059 ceph-mon[97450]: pgmap v14876: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:17:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:17:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:17:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:17:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:17:35.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:34.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:34.820098+0000) 2022-09-08T05:17:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:34.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:34.820162+0000) 2022-09-08T05:17:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:35 smithi055 ceph-mon[97033]: pgmap v14877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:17:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:35 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T05:17:35.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:35 smithi059 ceph-mon[97450]: pgmap v14877: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:17:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:35 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T05:17:36.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:35.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:35.820368+0000) 2022-09-08T05:17:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:35.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:35.820357+0000) 2022-09-08T05:17:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:36 smithi059 ceph-mon[97450]: pgmap v14878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:17:36.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:36 smithi055 ceph-mon[97033]: pgmap v14878: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:17:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:36.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:36.820621+0000) 2022-09-08T05:17:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:36.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:36.820619+0000) 2022-09-08T05:17:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:37.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:37.820832+0000) 2022-09-08T05:17:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:37.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:37.820828+0000) 2022-09-08T05:17:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:38.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:38.821011+0000) 2022-09-08T05:17:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:38.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:38.821011+0000) 2022-09-08T05:17:39.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:38 smithi055 ceph-mon[97033]: pgmap v14879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:17:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:38 smithi059 ceph-mon[97450]: pgmap v14879: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:17:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:39.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:39.821194+0000) 2022-09-08T05:17:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:39.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:39.821194+0000) 2022-09-08T05:17:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:40.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:40.821444+0000) 2022-09-08T05:17:41.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:40.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:40.821444+0000) 2022-09-08T05:17:41.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:40 smithi055 ceph-mon[97033]: pgmap v14880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:17:41.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:40 smithi059 ceph-mon[97450]: pgmap v14880: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:17:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:41.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:41.821759+0000) 2022-09-08T05:17:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:41.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:41.821759+0000) 2022-09-08T05:17:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:42.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:42.822019+0000) 2022-09-08T05:17:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:42.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:42.822019+0000) 2022-09-08T05:17:43.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:42 smithi055 ceph-mon[97033]: pgmap v14881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:17:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:42 smithi059 ceph-mon[97450]: pgmap v14881: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:17:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:43.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:43.822246+0000) 2022-09-08T05:17:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:43.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:43.822244+0000) 2022-09-08T05:17:45.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:44.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:44.822504+0000) 2022-09-08T05:17:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:44.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:44.822504+0000) 2022-09-08T05:17:45.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:44 smithi055 ceph-mon[97033]: pgmap v14882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:17:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:44 smithi059 ceph-mon[97450]: pgmap v14882: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:17:46.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:45.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:45.822749+0000) 2022-09-08T05:17:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:45.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:45.822747+0000) 2022-09-08T05:17:46.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:46 smithi055 ceph-mon[97033]: pgmap v14883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:17:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:46 smithi059 ceph-mon[97450]: pgmap v14883: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:17:47.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:46.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:46.822977+0000) 2022-09-08T05:17:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:46.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:46.822970+0000) 2022-09-08T05:17:47.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:17:47.632+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:17:47.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:47.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:47.823177+0000) 2022-09-08T05:17:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:47.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:47.823185+0000) 2022-09-08T05:17:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:48.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:48.823406+0000) 2022-09-08T05:17:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:48.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:48.823405+0000) 2022-09-08T05:17:49.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:48 smithi055 ceph-mon[97033]: pgmap v14884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:17:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:48 smithi059 ceph-mon[97450]: pgmap v14884: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:17:50.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:49.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:49.823690+0000) 2022-09-08T05:17:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:49.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:49.823697+0000) 2022-09-08T05:17:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:50.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:50.823971+0000) 2022-09-08T05:17:51.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:50.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:50.823971+0000) 2022-09-08T05:17:51.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:50 smithi055 ceph-mon[97033]: pgmap v14885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:17:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:50 smithi059 ceph-mon[97450]: pgmap v14885: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:17:52.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:51.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:51.824260+0000) 2022-09-08T05:17:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:51.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:51.824256+0000) 2022-09-08T05:17:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:52.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:52.824538+0000) 2022-09-08T05:17:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:52.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:52.824538+0000) 2022-09-08T05:17:53.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:52 smithi055 ceph-mon[97033]: pgmap v14886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:17:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:52 smithi059 ceph-mon[97450]: pgmap v14886: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:17:54.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:53.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:53.824786+0000) 2022-09-08T05:17:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:53.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:53.824786+0000) 2022-09-08T05:17:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:54.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:54.825039+0000) 2022-09-08T05:17:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:54.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:54.825030+0000) 2022-09-08T05:17:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:54 smithi055 ceph-mon[97033]: pgmap v14887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:17:55.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:54 smithi059 ceph-mon[97450]: pgmap v14887: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:17:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:55.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:55.825265+0000) 2022-09-08T05:17:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:55.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:55.825266+0000) 2022-09-08T05:17:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:56 smithi055 ceph-mon[97033]: pgmap v14888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:17:57.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:56 smithi059 ceph-mon[97450]: pgmap v14888: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:17:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:56.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:56.825446+0000) 2022-09-08T05:17:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:56.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:56.825448+0000) 2022-09-08T05:17:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:57.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:57.825704+0000) 2022-09-08T05:17:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:57.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:57.825704+0000) 2022-09-08T05:17:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:58.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:58.825850+0000) 2022-09-08T05:17:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:58.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:58.825841+0000) 2022-09-08T05:17:59.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:17:58 smithi055 ceph-mon[97033]: pgmap v14889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:17:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:17:58 smithi059 ceph-mon[97450]: pgmap v14889: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:17:59.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:59.826063+0000) 2022-09-08T05:18:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:17:59.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:17:59.826058+0000) 2022-09-08T05:18:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:00.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:00.826224+0000) 2022-09-08T05:18:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:00.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:00.826224+0000) 2022-09-08T05:18:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:00 smithi055 ceph-mon[97033]: pgmap v14890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:00 smithi059 ceph-mon[97450]: pgmap v14890: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:02.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:01.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:01.826498+0000) 2022-09-08T05:18:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:01.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:01.826498+0000) 2022-09-08T05:18:02.900 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:18:02.648+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:18:02.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:02.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:02.826779+0000) 2022-09-08T05:18:02.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:02.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:02.826779+0000) 2022-09-08T05:18:03.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:02 smithi055 ceph-mon[97033]: pgmap v14891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:18:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:02 smithi059 ceph-mon[97450]: pgmap v14891: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:18:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:03.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:03.827052+0000) 2022-09-08T05:18:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:03.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:03.827041+0000) 2022-09-08T05:18:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:04.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:04.827265+0000) 2022-09-08T05:18:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:04.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:04.827301+0000) 2022-09-08T05:18:05.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:04 smithi055 ceph-mon[97033]: pgmap v14892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:04 smithi059 ceph-mon[97450]: pgmap v14892: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:06.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:05.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:05.827558+0000) 2022-09-08T05:18:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:05.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:05.827579+0000) 2022-09-08T05:18:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:18:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:06 smithi055 ceph-mon[97033]: pgmap v14893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:18:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:18:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:18:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:06 smithi059 ceph-mon[97450]: pgmap v14893: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:18:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:18:07.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:06.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:06.827842+0000) 2022-09-08T05:18:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:06.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:06.827842+0000) 2022-09-08T05:18:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:07.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:07.828060+0000) 2022-09-08T05:18:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:07.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:07.828060+0000) 2022-09-08T05:18:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:08.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:08.828280+0000) 2022-09-08T05:18:09.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:08.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:08.828280+0000) 2022-09-08T05:18:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:08 smithi055 ceph-mon[97033]: pgmap v14894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:18:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:08 smithi059 ceph-mon[97450]: pgmap v14894: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:18:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:09.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:09.828546+0000) 2022-09-08T05:18:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:09.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:09.828546+0000) 2022-09-08T05:18:10.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:09 smithi055 ceph-mon[97033]: osdmap e1011: 12 total, 11 up, 11 in 2022-09-08T05:18:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:09 smithi059 ceph-mon[97450]: osdmap e1011: 12 total, 11 up, 11 in 2022-09-08T05:18:11.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:10.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:10.828843+0000) 2022-09-08T05:18:11.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:10.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:10.828840+0000) 2022-09-08T05:18:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:10 smithi059 ceph-mon[97450]: pgmap v14896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:18:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:10 smithi055 ceph-mon[97033]: pgmap v14896: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:18:12.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:11.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:11.829151+0000) 2022-09-08T05:18:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:11.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:11.829151+0000) 2022-09-08T05:18:13.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:12.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:12.829390+0000) 2022-09-08T05:18:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:12.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:12.829399+0000) 2022-09-08T05:18:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:12 smithi059 ceph-mon[97450]: pgmap v14897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:18:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:12 smithi055 ceph-mon[97033]: pgmap v14897: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:18:14.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:13.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:13.829710+0000) 2022-09-08T05:18:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:13.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:13.829710+0000) 2022-09-08T05:18:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:14.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:14.829879+0000) 2022-09-08T05:18:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:14.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:14.829896+0000) 2022-09-08T05:18:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:14 smithi059 ceph-mon[97450]: pgmap v14898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:18:15.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:14 smithi055 ceph-mon[97033]: pgmap v14898: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:18:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:15.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:15.830082+0000) 2022-09-08T05:18:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:15.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:15.830083+0000) 2022-09-08T05:18:16.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:16 smithi055 ceph-mon[97033]: osdmap e1012: 12 total, 11 up, 11 in 2022-09-08T05:18:16.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:16 smithi055 ceph-mon[97033]: pgmap v14900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:18:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:16 smithi059 ceph-mon[97450]: osdmap e1012: 12 total, 11 up, 11 in 2022-09-08T05:18:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:16 smithi059 ceph-mon[97450]: pgmap v14900: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:18:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:16.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:16.830304+0000) 2022-09-08T05:18:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:16.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:16.830304+0000) 2022-09-08T05:18:18.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:18:17.664+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:18:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:17.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:17.830509+0000) 2022-09-08T05:18:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:17.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:17.830514+0000) 2022-09-08T05:18:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:18.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:18.830701+0000) 2022-09-08T05:18:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:18.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:18.830696+0000) 2022-09-08T05:18:19.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:18 smithi055 ceph-mon[97033]: pgmap v14901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T05:18:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:18 smithi059 ceph-mon[97450]: pgmap v14901: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T05:18:20.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:19.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:19.830909+0000) 2022-09-08T05:18:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:19.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:19.830909+0000) 2022-09-08T05:18:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:20.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:20.831124+0000) 2022-09-08T05:18:21.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:20.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:20.831126+0000) 2022-09-08T05:18:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:20 smithi059 ceph-mon[97450]: pgmap v14902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:18:21.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:20 smithi055 ceph-mon[97033]: pgmap v14902: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:18:22.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:21.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:21.831321+0000) 2022-09-08T05:18:22.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:21.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:21.831322+0000) 2022-09-08T05:18:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:22.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:22.831559+0000) 2022-09-08T05:18:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:22.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:22.831538+0000) 2022-09-08T05:18:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:22 smithi059 ceph-mon[97450]: pgmap v14903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:18:23.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:22 smithi055 ceph-mon[97033]: pgmap v14903: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:18:24.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:23.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:23.831776+0000) 2022-09-08T05:18:24.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:23.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:23.831764+0000) 2022-09-08T05:18:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:24.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:24.831994+0000) 2022-09-08T05:18:25.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:24.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:24.831990+0000) 2022-09-08T05:18:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:24 smithi059 ceph-mon[97450]: pgmap v14904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:18:25.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:24 smithi055 ceph-mon[97033]: pgmap v14904: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:18:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:25.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:25.832197+0000) 2022-09-08T05:18:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:25.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:25.832192+0000) 2022-09-08T05:18:26.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:26 smithi055 ceph-mon[97033]: pgmap v14905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:18:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:26 smithi059 ceph-mon[97450]: pgmap v14905: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:18:27.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:26.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:26.832344+0000) 2022-09-08T05:18:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:26.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:26.832478+0000) 2022-09-08T05:18:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:27.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:27.832489+0000) 2022-09-08T05:18:28.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:27.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:27.832633+0000) 2022-09-08T05:18:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:28.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:28.832805+0000) 2022-09-08T05:18:29.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:28.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:28.832684+0000) 2022-09-08T05:18:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:28 smithi059 ceph-mon[97450]: pgmap v14906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:28 smithi055 ceph-mon[97033]: pgmap v14906: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:30.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:29.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:29.832968+0000) 2022-09-08T05:18:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:29.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:29.832826+0000) 2022-09-08T05:18:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:30.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:30.833038+0000) 2022-09-08T05:18:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:30.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:30.833096+0000) 2022-09-08T05:18:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:30 smithi059 ceph-mon[97450]: pgmap v14907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:18:31.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:30 smithi055 ceph-mon[97033]: pgmap v14907: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:18:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:31.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:31.833311+0000) 2022-09-08T05:18:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:31.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:31.833310+0000) 2022-09-08T05:18:33.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:32.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:32.833550+0000) 2022-09-08T05:18:33.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:32.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:32.833556+0000) 2022-09-08T05:18:33.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:18:32.672+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:18:33.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:32 smithi059 ceph-mon[97450]: pgmap v14908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:18:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:32 smithi055 ceph-mon[97033]: pgmap v14908: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:18:34.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:33.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:33.833729+0000) 2022-09-08T05:18:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:33.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:33.833729+0000) 2022-09-08T05:18:35.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:34.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:34.833937+0000) 2022-09-08T05:18:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:34.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:34.833937+0000) 2022-09-08T05:18:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:34 smithi059 ceph-mon[97450]: pgmap v14909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:18:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:18:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:18:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:18:35.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:18:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:34 smithi055 ceph-mon[97033]: pgmap v14909: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:18:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:18:35.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:18:35.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:18:35.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:18:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:35.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:35.834162+0000) 2022-09-08T05:18:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:35.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:35.834161+0000) 2022-09-08T05:18:36.934 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:36 smithi059 ceph-mon[97450]: pgmap v14910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:18:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:36 smithi055 ceph-mon[97033]: pgmap v14910: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:18:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:36.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:36.834338+0000) 2022-09-08T05:18:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:36.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:36.834338+0000) 2022-09-08T05:18:38.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:37.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:37.834616+0000) 2022-09-08T05:18:38.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:37.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:37.834615+0000) 2022-09-08T05:18:38.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:18:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:18:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:18:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:18:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T05:18:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T05:18:38.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:18:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:18:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:18:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:18:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:18:38.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T05:18:38.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T05:18:38.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:18:38.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:18:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:38.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:38.834846+0000) 2022-09-08T05:18:39.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:38.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:38.834843+0000) 2022-09-08T05:18:39.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:38 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T05:18:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:38 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T05:18:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:38 smithi055 ceph-mon[97033]: pgmap v14911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:38 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T05:18:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:38 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T05:18:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:38 smithi059 ceph-mon[97450]: pgmap v14911: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:40.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:39.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:39.835099+0000) 2022-09-08T05:18:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:39.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:39.835097+0000) 2022-09-08T05:18:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:40.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:40.835375+0000) 2022-09-08T05:18:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:40.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:40.835375+0000) 2022-09-08T05:18:41.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:40 smithi059 ceph-mon[97450]: pgmap v14912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:18:41.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:40 smithi055 ceph-mon[97033]: pgmap v14912: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:18:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:41.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:41.835703+0000) 2022-09-08T05:18:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:41.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:41.835703+0000) 2022-09-08T05:18:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:42.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:42.836010+0000) 2022-09-08T05:18:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:42.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:42.836007+0000) 2022-09-08T05:18:43.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:42 smithi059 ceph-mon[97450]: pgmap v14913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:18:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:42 smithi055 ceph-mon[97033]: pgmap v14913: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:18:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:43.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:43.836280+0000) 2022-09-08T05:18:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:43.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:43.836280+0000) 2022-09-08T05:18:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:44.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:44.836501+0000) 2022-09-08T05:18:45.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:44.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:44.836502+0000) 2022-09-08T05:18:45.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:44 smithi059 ceph-mon[97450]: pgmap v14914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:44 smithi055 ceph-mon[97033]: pgmap v14914: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:45.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:45.836759+0000) 2022-09-08T05:18:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:45.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:45.836775+0000) 2022-09-08T05:18:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:46 smithi055 ceph-mon[97033]: pgmap v14915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:18:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:46 smithi059 ceph-mon[97450]: pgmap v14915: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:18:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:46.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:46.837037+0000) 2022-09-08T05:18:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:46.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:46.837037+0000) 2022-09-08T05:18:48.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:47.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:47.837257+0000) 2022-09-08T05:18:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:47.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:47.837258+0000) 2022-09-08T05:18:48.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:18:47.683+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:18:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:48.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:48.837404+0000) 2022-09-08T05:18:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:48.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:48.837443+0000) 2022-09-08T05:18:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:48 smithi059 ceph-mon[97450]: pgmap v14916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:48 smithi055 ceph-mon[97033]: pgmap v14916: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:50.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:49.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:49.837620+0000) 2022-09-08T05:18:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:49.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:49.837620+0000) 2022-09-08T05:18:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:50.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:50.837897+0000) 2022-09-08T05:18:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:50.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:50.837892+0000) 2022-09-08T05:18:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:50 smithi059 ceph-mon[97450]: pgmap v14917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:18:51.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:50 smithi055 ceph-mon[97033]: pgmap v14917: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:18:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:51.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:51.838187+0000) 2022-09-08T05:18:52.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:51.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:51.838187+0000) 2022-09-08T05:18:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:52.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:52.838489+0000) 2022-09-08T05:18:53.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:52.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:52.838501+0000) 2022-09-08T05:18:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:52 smithi059 ceph-mon[97450]: pgmap v14918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:18:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:52 smithi055 ceph-mon[97033]: pgmap v14918: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:18:54.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:53.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:53.838765+0000) 2022-09-08T05:18:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:53.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:53.838765+0000) 2022-09-08T05:18:55.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:54.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:54.838915+0000) 2022-09-08T05:18:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:54.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:54.838913+0000) 2022-09-08T05:18:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:54 smithi059 ceph-mon[97450]: pgmap v14919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:55.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:54 smithi055 ceph-mon[97033]: pgmap v14919: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:56.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:55.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:55.839135+0000) 2022-09-08T05:18:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:55.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:55.839146+0000) 2022-09-08T05:18:56.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:56 smithi055 ceph-mon[97033]: pgmap v14920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:18:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:56 smithi059 ceph-mon[97450]: pgmap v14920: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:18:57.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:56.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:56.839407+0000) 2022-09-08T05:18:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:56.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:56.839407+0000) 2022-09-08T05:18:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:57.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:57.839573+0000) 2022-09-08T05:18:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:57.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:57.839584+0000) 2022-09-08T05:18:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:58.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:58.839764+0000) 2022-09-08T05:18:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:58.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:58.839764+0000) 2022-09-08T05:18:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:18:58 smithi059 ceph-mon[97450]: pgmap v14921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:18:59.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:18:58 smithi055 ceph-mon[97033]: pgmap v14921: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:00.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:18:59.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:59.839936+0000) 2022-09-08T05:19:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:18:59.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:18:59.839936+0000) 2022-09-08T05:19:01.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:00.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:00.840077+0000) 2022-09-08T05:19:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:00.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:00.840098+0000) 2022-09-08T05:19:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:00 smithi059 ceph-mon[97450]: pgmap v14922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:19:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:00 smithi055 ceph-mon[97033]: pgmap v14922: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:19:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:01.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:01.840334+0000) 2022-09-08T05:19:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:01.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:01.840315+0000) 2022-09-08T05:19:03.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:19:02.692+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:19:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:02.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:02.840532+0000) 2022-09-08T05:19:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:02.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:02.840504+0000) 2022-09-08T05:19:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:02 smithi059 ceph-mon[97450]: pgmap v14923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:19:03.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:02 smithi055 ceph-mon[97033]: pgmap v14923: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:19:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:03.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:03.840779+0000) 2022-09-08T05:19:04.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:03.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:03.840744+0000) 2022-09-08T05:19:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:04.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:04.840973+0000) 2022-09-08T05:19:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:04.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:04.840912+0000) 2022-09-08T05:19:05.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:04 smithi059 ceph-mon[97450]: pgmap v14924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:05.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:04 smithi055 ceph-mon[97033]: pgmap v14924: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:05.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:05.841121+0000) 2022-09-08T05:19:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:05.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:05.841078+0000) 2022-09-08T05:19:06.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:19:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:06 smithi055 ceph-mon[97033]: pgmap v14925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:19:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:19:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:19:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:06 smithi059 ceph-mon[97450]: pgmap v14925: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:19:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:19:07.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:06.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:06.841337+0000) 2022-09-08T05:19:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:06.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:06.841305+0000) 2022-09-08T05:19:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:07.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:07.841551+0000) 2022-09-08T05:19:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:07.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:07.841551+0000) 2022-09-08T05:19:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:08.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:08.841789+0000) 2022-09-08T05:19:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:08.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:08.841779+0000) 2022-09-08T05:19:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:08 smithi059 ceph-mon[97450]: pgmap v14926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:19:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:08 smithi055 ceph-mon[97033]: pgmap v14926: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:19:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:09.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:09.842007+0000) 2022-09-08T05:19:10.193 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:09.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:09.842007+0000) 2022-09-08T05:19:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:09 smithi059 ceph-mon[97450]: osdmap e1013: 12 total, 11 up, 11 in 2022-09-08T05:19:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:09 smithi055 ceph-mon[97033]: osdmap e1013: 12 total, 11 up, 11 in 2022-09-08T05:19:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:10.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:10.842289+0000) 2022-09-08T05:19:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:10.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:10.842288+0000) 2022-09-08T05:19:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:11 smithi059 ceph-mon[97450]: pgmap v14928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:19:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:11 smithi055 ceph-mon[97033]: pgmap v14928: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:19:12.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:11.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:11.842528+0000) 2022-09-08T05:19:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:11.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:11.842536+0000) 2022-09-08T05:19:13.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:12.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:12.842740+0000) 2022-09-08T05:19:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:12.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:12.842742+0000) 2022-09-08T05:19:13.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:13 smithi059 ceph-mon[97450]: pgmap v14929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:19:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:13 smithi055 ceph-mon[97033]: pgmap v14929: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:19:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:13.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:13.843012+0000) 2022-09-08T05:19:14.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:13.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:13.842977+0000) 2022-09-08T05:19:15.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:14.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:14.843256+0000) 2022-09-08T05:19:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:14.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:14.843245+0000) 2022-09-08T05:19:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:15 smithi059 ceph-mon[97450]: pgmap v14930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:19:15.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:15 smithi055 ceph-mon[97033]: pgmap v14930: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:19:16.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:15.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:15.843516+0000) 2022-09-08T05:19:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:15.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:15.843531+0000) 2022-09-08T05:19:16.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:16 smithi055 ceph-mon[97033]: osdmap e1014: 12 total, 11 up, 11 in 2022-09-08T05:19:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:16 smithi055 ceph-mon[97033]: pgmap v14932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:19:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:16 smithi059 ceph-mon[97450]: osdmap e1014: 12 total, 11 up, 11 in 2022-09-08T05:19:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:16 smithi059 ceph-mon[97450]: pgmap v14932: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:19:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:16.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:16.843773+0000) 2022-09-08T05:19:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:16.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:16.843776+0000) 2022-09-08T05:19:18.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:19:17.692+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:19:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:17.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:17.844011+0000) 2022-09-08T05:19:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:17.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:17.843925+0000) 2022-09-08T05:19:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:18.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:18.844273+0000) 2022-09-08T05:19:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:18.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:18.844145+0000) 2022-09-08T05:19:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:18 smithi059 ceph-mon[97450]: pgmap v14933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T05:19:19.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:18 smithi055 ceph-mon[97033]: pgmap v14933: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T05:19:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:19.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:19.844556+0000) 2022-09-08T05:19:20.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:19.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:19.844426+0000) 2022-09-08T05:19:21.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:20.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:20.844801+0000) 2022-09-08T05:19:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:20.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:20.844713+0000) 2022-09-08T05:19:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:20 smithi059 ceph-mon[97450]: pgmap v14934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:19:21.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:20 smithi055 ceph-mon[97033]: pgmap v14934: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:19:22.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:21.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:21.844999+0000) 2022-09-08T05:19:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:21.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:21.844964+0000) 2022-09-08T05:19:23.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:22.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:22.845179+0000) 2022-09-08T05:19:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:22.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:22.845165+0000) 2022-09-08T05:19:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:22 smithi059 ceph-mon[97450]: pgmap v14935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:19:23.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:22 smithi055 ceph-mon[97033]: pgmap v14935: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:19:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:23.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:23.845392+0000) 2022-09-08T05:19:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:23.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:23.845390+0000) 2022-09-08T05:19:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:24.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:24.845556+0000) 2022-09-08T05:19:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:24.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:24.845535+0000) 2022-09-08T05:19:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:25 smithi059 ceph-mon[97450]: pgmap v14936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:19:25.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:25 smithi055 ceph-mon[97033]: pgmap v14936: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:19:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:25.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:25.845765+0000) 2022-09-08T05:19:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:25.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:25.845759+0000) 2022-09-08T05:19:26.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:26 smithi055 ceph-mon[97033]: pgmap v14937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:19:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:26 smithi059 ceph-mon[97450]: pgmap v14937: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:19:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:26.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:26.845946+0000) 2022-09-08T05:19:27.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:26.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:26.845880+0000) 2022-09-08T05:19:28.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:27.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:27.846156+0000) 2022-09-08T05:19:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:27.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:27.846144+0000) 2022-09-08T05:19:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:28.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:28.846399+0000) 2022-09-08T05:19:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:28.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:28.846392+0000) 2022-09-08T05:19:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:28 smithi059 ceph-mon[97450]: pgmap v14938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:28 smithi055 ceph-mon[97033]: pgmap v14938: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:30.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:29.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:29.846674+0000) 2022-09-08T05:19:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:29.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:29.846628+0000) 2022-09-08T05:19:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:30.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:30.846909+0000) 2022-09-08T05:19:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:30.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:30.846893+0000) 2022-09-08T05:19:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:30 smithi059 ceph-mon[97450]: pgmap v14939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:19:31.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:31 smithi055 ceph-mon[97033]: pgmap v14939: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:19:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:31.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:31.847093+0000) 2022-09-08T05:19:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:31.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:31.847053+0000) 2022-09-08T05:19:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:32.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:32.847267+0000) 2022-09-08T05:19:33.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:19:32.703+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:19:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:32.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:32.847263+0000) 2022-09-08T05:19:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:32 smithi059 ceph-mon[97450]: pgmap v14940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:19:33.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:33 smithi055 ceph-mon[97033]: pgmap v14940: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:19:34.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:33.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:33.847438+0000) 2022-09-08T05:19:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:33.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:33.847497+0000) 2022-09-08T05:19:35.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:34.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:34.847749+0000) 2022-09-08T05:19:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:34.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:34.847752+0000) 2022-09-08T05:19:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:35 smithi059 ceph-mon[97450]: pgmap v14941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:35.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:35 smithi055 ceph-mon[97033]: pgmap v14941: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:35.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:35.848071+0000) 2022-09-08T05:19:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:35.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:35.848000+0000) 2022-09-08T05:19:36.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:36 smithi055 ceph-mon[97033]: pgmap v14942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:19:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:36 smithi059 ceph-mon[97450]: pgmap v14942: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:19:37.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:36.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:36.848224+0000) 2022-09-08T05:19:37.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:36.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:36.848318+0000) 2022-09-08T05:19:38.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:37.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:37.848482+0000) 2022-09-08T05:19:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:37.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:37.848399+0000) 2022-09-08T05:19:38.227 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:19:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:19:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:19:38.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:19:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:19:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:19:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:38.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:38.848687+0000) 2022-09-08T05:19:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:38.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:38.848605+0000) 2022-09-08T05:19:39.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:39 smithi055 ceph-mon[97033]: pgmap v14943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:39.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:19:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:39 smithi059 ceph-mon[97450]: pgmap v14943: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:39.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:19:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:39.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:39.848850+0000) 2022-09-08T05:19:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:39.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:39.848855+0000) 2022-09-08T05:19:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:40.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:40.849035+0000) 2022-09-08T05:19:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:40.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:40.848986+0000) 2022-09-08T05:19:41.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:41 smithi055 ceph-mon[97033]: pgmap v14944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:19:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:41 smithi059 ceph-mon[97450]: pgmap v14944: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:19:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:41.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:41.849236+0000) 2022-09-08T05:19:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:41.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:41.849206+0000) 2022-09-08T05:19:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:42.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:42.849447+0000) 2022-09-08T05:19:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:42.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:42.849419+0000) 2022-09-08T05:19:43.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:43 smithi055 ceph-mon[97033]: pgmap v14945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:19:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:43 smithi059 ceph-mon[97450]: pgmap v14945: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:19:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:43.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:43.849737+0000) 2022-09-08T05:19:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:43.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:43.849690+0000) 2022-09-08T05:19:45.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:44.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:44.849830+0000) 2022-09-08T05:19:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:44.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:44.849910+0000) 2022-09-08T05:19:45.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:45 smithi055 ceph-mon[97033]: pgmap v14946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:45 smithi059 ceph-mon[97450]: pgmap v14946: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:46.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:45.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:45.850029+0000) 2022-09-08T05:19:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:45.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:45.850052+0000) 2022-09-08T05:19:46.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:46 smithi055 ceph-mon[97033]: pgmap v14947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:19:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:46 smithi059 ceph-mon[97450]: pgmap v14947: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:19:47.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:46.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:46.850263+0000) 2022-09-08T05:19:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:46.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:46.850233+0000) 2022-09-08T05:19:48.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:19:47.719+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:19:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:47.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:47.850438+0000) 2022-09-08T05:19:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:47.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:47.850421+0000) 2022-09-08T05:19:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:48.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:48.850689+0000) 2022-09-08T05:19:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:48.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:48.850678+0000) 2022-09-08T05:19:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:48 smithi059 ceph-mon[97450]: pgmap v14948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:49 smithi055 ceph-mon[97033]: pgmap v14948: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:50.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:49.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:49.850916+0000) 2022-09-08T05:19:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:49.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:49.850912+0000) 2022-09-08T05:19:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:50.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:50.851142+0000) 2022-09-08T05:19:51.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:50.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:50.851118+0000) 2022-09-08T05:19:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:51 smithi059 ceph-mon[97450]: pgmap v14949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T05:19:51.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:51 smithi055 ceph-mon[97033]: pgmap v14949: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T05:19:52.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:51.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:51.851343+0000) 2022-09-08T05:19:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:51.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:51.851348+0000) 2022-09-08T05:19:53.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:52.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:52.851603+0000) 2022-09-08T05:19:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:52.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:52.851568+0000) 2022-09-08T05:19:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:53 smithi059 ceph-mon[97450]: pgmap v14950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:19:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:53 smithi055 ceph-mon[97033]: pgmap v14950: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:19:54.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:53.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:53.851902+0000) 2022-09-08T05:19:54.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:53.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:53.851893+0000) 2022-09-08T05:19:55.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:54.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:54.852096+0000) 2022-09-08T05:19:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:54.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:54.852074+0000) 2022-09-08T05:19:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:55 smithi059 ceph-mon[97450]: pgmap v14951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:55.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:55 smithi055 ceph-mon[97033]: pgmap v14951: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:56.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:55.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:55.852302+0000) 2022-09-08T05:19:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:55.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:55.852293+0000) 2022-09-08T05:19:56.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:56 smithi055 ceph-mon[97033]: pgmap v14952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:19:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:56 smithi059 ceph-mon[97450]: pgmap v14952: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:19:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:56.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:56.852505+0000) 2022-09-08T05:19:57.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:56.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:56.852514+0000) 2022-09-08T05:19:58.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:57.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:57.852692+0000) 2022-09-08T05:19:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:57.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:57.852724+0000) 2022-09-08T05:19:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:58.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:58.852814+0000) 2022-09-08T05:19:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:58.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:58.852874+0000) 2022-09-08T05:19:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:19:59 smithi059 ceph-mon[97450]: pgmap v14953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:19:59.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:19:59 smithi055 ceph-mon[97033]: pgmap v14953: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:00.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:19:59.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:59.853000+0000) 2022-09-08T05:20:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:19:59.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:19:59.853046+0000) 2022-09-08T05:20:00.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T05:20:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T05:20:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T05:20:00.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T05:20:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T05:20:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T05:20:01.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:00.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:00.853170+0000) 2022-09-08T05:20:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:00.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:00.853202+0000) 2022-09-08T05:20:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:01 smithi059 ceph-mon[97450]: pgmap v14954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T05:20:01.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:01 smithi055 ceph-mon[97033]: pgmap v14954: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T05:20:02.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:01.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:01.853300+0000) 2022-09-08T05:20:02.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:01.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:01.853346+0000) 2022-09-08T05:20:03.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:20:02.732+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:20:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:02.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:02.853540+0000) 2022-09-08T05:20:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:02.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:02.853547+0000) 2022-09-08T05:20:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:03 smithi059 ceph-mon[97450]: pgmap v14955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:20:03.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:03 smithi055 ceph-mon[97033]: pgmap v14955: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:20:04.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:03.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:03.853805+0000) 2022-09-08T05:20:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:03.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:03.853791+0000) 2022-09-08T05:20:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:04.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:04.854068+0000) 2022-09-08T05:20:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:04.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:04.854072+0000) 2022-09-08T05:20:05.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:05 smithi059 ceph-mon[97450]: pgmap v14956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:05.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:05 smithi055 ceph-mon[97033]: pgmap v14956: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:05.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:05.854371+0000) 2022-09-08T05:20:06.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:05.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:05.854371+0000) 2022-09-08T05:20:06.863 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:20:06.863 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:06 smithi059 ceph-mon[97450]: pgmap v14957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:20:06.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:20:06.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:20:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:06 smithi055 ceph-mon[97033]: pgmap v14957: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:20:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:20:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:06.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:06.854668+0000) 2022-09-08T05:20:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:06.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:06.854668+0000) 2022-09-08T05:20:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:07.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:07.854906+0000) 2022-09-08T05:20:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:07.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:07.854904+0000) 2022-09-08T05:20:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:08.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:08.855088+0000) 2022-09-08T05:20:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:08.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:08.855090+0000) 2022-09-08T05:20:09.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:09 smithi059 ceph-mon[97450]: pgmap v14958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:20:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:09 smithi055 ceph-mon[97033]: pgmap v14958: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:20:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:09.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:09.855256+0000) 2022-09-08T05:20:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:09.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:09.855268+0000) 2022-09-08T05:20:10.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:10 smithi059 ceph-mon[97450]: osdmap e1015: 12 total, 11 up, 11 in 2022-09-08T05:20:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:10 smithi055 ceph-mon[97033]: osdmap e1015: 12 total, 11 up, 11 in 2022-09-08T05:20:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:10.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:10.855424+0000) 2022-09-08T05:20:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:10.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:10.855424+0000) 2022-09-08T05:20:11.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:11 smithi059 ceph-mon[97450]: pgmap v14960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:20:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:11 smithi055 ceph-mon[97033]: pgmap v14960: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:20:12.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:11.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:11.855727+0000) 2022-09-08T05:20:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:11.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:11.855730+0000) 2022-09-08T05:20:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:12.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:12.855967+0000) 2022-09-08T05:20:13.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:12.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:12.855968+0000) 2022-09-08T05:20:13.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:13 smithi059 ceph-mon[97450]: pgmap v14961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:20:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:13 smithi055 ceph-mon[97033]: pgmap v14961: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:20:14.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:13.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:13.856219+0000) 2022-09-08T05:20:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:13.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:13.856218+0000) 2022-09-08T05:20:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:14.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:14.856501+0000) 2022-09-08T05:20:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:14.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:14.856501+0000) 2022-09-08T05:20:15.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:15 smithi059 ceph-mon[97450]: pgmap v14962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:20:15.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:15 smithi055 ceph-mon[97033]: pgmap v14962: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:20:16.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:15.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:15.856803+0000) 2022-09-08T05:20:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:15.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:15.856803+0000) 2022-09-08T05:20:16.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:16 smithi055 ceph-mon[97033]: osdmap e1016: 12 total, 11 up, 11 in 2022-09-08T05:20:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:16 smithi055 ceph-mon[97033]: pgmap v14964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:20:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:16 smithi059 ceph-mon[97450]: osdmap e1016: 12 total, 11 up, 11 in 2022-09-08T05:20:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:16 smithi059 ceph-mon[97450]: pgmap v14964: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:20:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:16.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:16.857098+0000) 2022-09-08T05:20:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:16.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:16.857098+0000) 2022-09-08T05:20:18.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:20:17.749+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:20:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:17.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:17.857348+0000) 2022-09-08T05:20:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:17.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:17.857348+0000) 2022-09-08T05:20:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:18.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:18.857525+0000) 2022-09-08T05:20:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:18.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:18.857518+0000) 2022-09-08T05:20:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:19 smithi059 ceph-mon[97450]: pgmap v14965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T05:20:19.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:19 smithi055 ceph-mon[97033]: pgmap v14965: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T05:20:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:19.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:19.857768+0000) 2022-09-08T05:20:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:19.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:19.857768+0000) 2022-09-08T05:20:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:20.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:20.857976+0000) 2022-09-08T05:20:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:20.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:20.857976+0000) 2022-09-08T05:20:21.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:21 smithi059 ceph-mon[97450]: pgmap v14966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:20:21.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:21 smithi055 ceph-mon[97033]: pgmap v14966: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:20:22.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:21.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:21.858139+0000) 2022-09-08T05:20:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:21.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:21.858139+0000) 2022-09-08T05:20:23.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:22.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:22.858382+0000) 2022-09-08T05:20:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:22.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:22.858374+0000) 2022-09-08T05:20:23.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:23 smithi059 ceph-mon[97450]: pgmap v14967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:20:23.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:23 smithi055 ceph-mon[97033]: pgmap v14967: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:20:24.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:23.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:23.858536+0000) 2022-09-08T05:20:24.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:23.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:23.858528+0000) 2022-09-08T05:20:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:24.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:24.858711+0000) 2022-09-08T05:20:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:24.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:24.858711+0000) 2022-09-08T05:20:25.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:25 smithi059 ceph-mon[97450]: pgmap v14968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:20:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:25 smithi055 ceph-mon[97033]: pgmap v14968: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:20:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:25.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:25.858853+0000) 2022-09-08T05:20:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:25.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:25.858859+0000) 2022-09-08T05:20:26.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:26 smithi055 ceph-mon[97033]: pgmap v14969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:20:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:26 smithi059 ceph-mon[97450]: pgmap v14969: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:20:27.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:26.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:26.859089+0000) 2022-09-08T05:20:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:26.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:26.859050+0000) 2022-09-08T05:20:28.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:27.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:27.859273+0000) 2022-09-08T05:20:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:27.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:27.859255+0000) 2022-09-08T05:20:29.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:28.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:28.859422+0000) 2022-09-08T05:20:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:28.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:28.859422+0000) 2022-09-08T05:20:29.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:29 smithi059 ceph-mon[97450]: pgmap v14970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:29 smithi055 ceph-mon[97033]: pgmap v14970: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:29.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:29.859571+0000) 2022-09-08T05:20:30.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:29.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:29.859571+0000) 2022-09-08T05:20:31.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:30.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:30.859739+0000) 2022-09-08T05:20:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:30.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:30.859738+0000) 2022-09-08T05:20:31.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:31 smithi059 ceph-mon[97450]: pgmap v14971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T05:20:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:31 smithi055 ceph-mon[97033]: pgmap v14971: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T05:20:32.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:31.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:31.859904+0000) 2022-09-08T05:20:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:31.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:31.859904+0000) 2022-09-08T05:20:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:32.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:32.860081+0000) 2022-09-08T05:20:33.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:20:32.757+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:20:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:32.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:32.860081+0000) 2022-09-08T05:20:33.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:33 smithi059 ceph-mon[97450]: pgmap v14972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:20:33.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:33 smithi055 ceph-mon[97033]: pgmap v14972: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:20:34.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:33.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:33.860342+0000) 2022-09-08T05:20:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:33.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:33.860342+0000) 2022-09-08T05:20:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:34.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:34.860510+0000) 2022-09-08T05:20:35.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:34.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:34.860504+0000) 2022-09-08T05:20:35.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:35 smithi059 ceph-mon[97450]: pgmap v14973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:35.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:35 smithi055 ceph-mon[97033]: pgmap v14973: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:36.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:35.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:35.860697+0000) 2022-09-08T05:20:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:35.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:35.860697+0000) 2022-09-08T05:20:36.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:36 smithi055 ceph-mon[97033]: pgmap v14974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:20:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:36 smithi059 ceph-mon[97450]: pgmap v14974: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:20:37.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:36.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:36.860845+0000) 2022-09-08T05:20:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:36.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:36.860843+0000) 2022-09-08T05:20:38.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:37.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:37.861096+0000) 2022-09-08T05:20:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:37.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:37.861093+0000) 2022-09-08T05:20:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:38.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:38.861372+0000) 2022-09-08T05:20:39.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:38.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:38.861373+0000) 2022-09-08T05:20:39.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:39 smithi059 ceph-mon[97450]: pgmap v14975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:20:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:20:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:20:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:20:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:20:39.433 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:39 smithi055 ceph-mon[97033]: pgmap v14975: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:39.434 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:20:39.434 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:20:39.435 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:20:39.436 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:20:39.436 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:20:40.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:39.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:39.861618+0000) 2022-09-08T05:20:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:39.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:39.861616+0000) 2022-09-08T05:20:40.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:20:40.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:20:41.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:40.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:40.861794+0000) 2022-09-08T05:20:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:40.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:40.861794+0000) 2022-09-08T05:20:41.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:41 smithi055 ceph-mon[97033]: pgmap v14976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T05:20:41.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:41 smithi059 ceph-mon[97450]: pgmap v14976: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T05:20:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:41.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:41.862008+0000) 2022-09-08T05:20:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:41.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:41.862006+0000) 2022-09-08T05:20:43.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:42.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:42.862299+0000) 2022-09-08T05:20:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:42.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:42.862297+0000) 2022-09-08T05:20:43.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:43 smithi055 ceph-mon[97033]: pgmap v14977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:20:43.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:43 smithi059 ceph-mon[97450]: pgmap v14977: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:20:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:43.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:43.862562+0000) 2022-09-08T05:20:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:43.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:43.862557+0000) 2022-09-08T05:20:45.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:44.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:44.862844+0000) 2022-09-08T05:20:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:44.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:44.862844+0000) 2022-09-08T05:20:45.727 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:45 smithi055 ceph-mon[97033]: pgmap v14978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:45 smithi059 ceph-mon[97450]: pgmap v14978: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:45.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:45.863051+0000) 2022-09-08T05:20:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:45.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:45.863051+0000) 2022-09-08T05:20:46.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:46 smithi055 ceph-mon[97033]: pgmap v14979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:20:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:46 smithi059 ceph-mon[97450]: pgmap v14979: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:20:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:46.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:46.863339+0000) 2022-09-08T05:20:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:46.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:46.863340+0000) 2022-09-08T05:20:48.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:20:47.771+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:20:48.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:47.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:47.863557+0000) 2022-09-08T05:20:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:47.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:47.863557+0000) 2022-09-08T05:20:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:48.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:48.863789+0000) 2022-09-08T05:20:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:48.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:48.863789+0000) 2022-09-08T05:20:49.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:49 smithi059 ceph-mon[97450]: pgmap v14980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:49 smithi055 ceph-mon[97033]: pgmap v14980: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:49.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:49.864028+0000) 2022-09-08T05:20:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:49.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:49.864028+0000) 2022-09-08T05:20:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:50.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:50.864195+0000) 2022-09-08T05:20:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:50.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:50.864190+0000) 2022-09-08T05:20:51.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:51 smithi059 ceph-mon[97450]: pgmap v14981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T05:20:51.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:51 smithi055 ceph-mon[97033]: pgmap v14981: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T05:20:52.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:51.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:51.864395+0000) 2022-09-08T05:20:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:51.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:51.864395+0000) 2022-09-08T05:20:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:52.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:52.864576+0000) 2022-09-08T05:20:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:52.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:52.864576+0000) 2022-09-08T05:20:53.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:53 smithi059 ceph-mon[97450]: pgmap v14982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:20:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:53 smithi055 ceph-mon[97033]: pgmap v14982: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:20:54.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:53.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:53.864750+0000) 2022-09-08T05:20:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:53.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:53.864776+0000) 2022-09-08T05:20:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:54.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:54.864973+0000) 2022-09-08T05:20:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:54.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:54.864953+0000) 2022-09-08T05:20:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:55 smithi059 ceph-mon[97450]: pgmap v14983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:55.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:55 smithi055 ceph-mon[97033]: pgmap v14983: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:56.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:55.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:55.865140+0000) 2022-09-08T05:20:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:55.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:55.865138+0000) 2022-09-08T05:20:56.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:56 smithi055 ceph-mon[97033]: pgmap v14984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:20:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:56 smithi059 ceph-mon[97450]: pgmap v14984: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:20:57.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:56.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:56.865295+0000) 2022-09-08T05:20:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:56.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:56.865271+0000) 2022-09-08T05:20:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:57.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:57.865462+0000) 2022-09-08T05:20:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:57.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:57.865417+0000) 2022-09-08T05:20:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:58.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:58.865631+0000) 2022-09-08T05:20:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:58.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:58.865586+0000) 2022-09-08T05:20:59.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:20:59 smithi059 ceph-mon[97450]: pgmap v14985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:20:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:20:59 smithi055 ceph-mon[97033]: pgmap v14985: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:00.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:20:59.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:59.865804+0000) 2022-09-08T05:21:00.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:20:59.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:20:59.865780+0000) 2022-09-08T05:21:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:00.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:00.865979+0000) 2022-09-08T05:21:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:00.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:00.865962+0000) 2022-09-08T05:21:01.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:01 smithi059 ceph-mon[97450]: pgmap v14986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:21:01.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:01 smithi055 ceph-mon[97033]: pgmap v14986: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:21:02.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:01.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:01.866120+0000) 2022-09-08T05:21:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:01.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:01.866129+0000) 2022-09-08T05:21:03.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:21:02.775+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:21:03.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:02.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:02.866331+0000) 2022-09-08T05:21:03.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:02.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:02.866344+0000) 2022-09-08T05:21:03.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:03 smithi059 ceph-mon[97450]: pgmap v14987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:21:03.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:03 smithi055 ceph-mon[97033]: pgmap v14987: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:21:04.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:03.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:03.866555+0000) 2022-09-08T05:21:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:03.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:03.866574+0000) 2022-09-08T05:21:05.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:04.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:04.866784+0000) 2022-09-08T05:21:05.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:04.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:04.866759+0000) 2022-09-08T05:21:05.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:05 smithi055 ceph-mon[97033]: pgmap v14988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:05 smithi059 ceph-mon[97450]: pgmap v14988: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:05.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:05.866944+0000) 2022-09-08T05:21:06.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:05.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:05.866937+0000) 2022-09-08T05:21:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:21:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:06 smithi055 ceph-mon[97033]: pgmap v14989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:21:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:21:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:21:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:06 smithi059 ceph-mon[97450]: pgmap v14989: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:21:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:21:07.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:06.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:06.867109+0000) 2022-09-08T05:21:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:06.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:06.867088+0000) 2022-09-08T05:21:08.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:07.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:07.867262+0000) 2022-09-08T05:21:08.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:07.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:07.867245+0000) 2022-09-08T05:21:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:08.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:08.867462+0000) 2022-09-08T05:21:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:08.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:08.867466+0000) 2022-09-08T05:21:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:09 smithi055 ceph-mon[97033]: pgmap v14990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:21:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:09 smithi059 ceph-mon[97450]: pgmap v14990: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:21:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:09.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:09.867706+0000) 2022-09-08T05:21:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:09.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:09.867746+0000) 2022-09-08T05:21:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:10 smithi055 ceph-mon[97033]: osdmap e1017: 12 total, 11 up, 11 in 2022-09-08T05:21:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:10 smithi059 ceph-mon[97450]: osdmap e1017: 12 total, 11 up, 11 in 2022-09-08T05:21:11.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:10.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:10.867926+0000) 2022-09-08T05:21:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:10.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:10.867926+0000) 2022-09-08T05:21:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:11 smithi055 ceph-mon[97033]: pgmap v14992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:21:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:11 smithi059 ceph-mon[97450]: pgmap v14992: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:21:12.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:11.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:11.868129+0000) 2022-09-08T05:21:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:11.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:11.868134+0000) 2022-09-08T05:21:13.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:12.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:12.868390+0000) 2022-09-08T05:21:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:12.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:12.868350+0000) 2022-09-08T05:21:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:13 smithi055 ceph-mon[97033]: pgmap v14993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:21:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:13 smithi059 ceph-mon[97450]: pgmap v14993: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:21:14.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:13.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:13.868593+0000) 2022-09-08T05:21:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:13.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:13.868665+0000) 2022-09-08T05:21:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:14.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:14.868825+0000) 2022-09-08T05:21:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:14.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:14.868935+0000) 2022-09-08T05:21:15.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:15 smithi055 ceph-mon[97033]: pgmap v14994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:21:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:15 smithi059 ceph-mon[97450]: pgmap v14994: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:21:16.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:15.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:15.868988+0000) 2022-09-08T05:21:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:15.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:15.869108+0000) 2022-09-08T05:21:16.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:16 smithi055 ceph-mon[97033]: osdmap e1018: 12 total, 11 up, 11 in 2022-09-08T05:21:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:16 smithi055 ceph-mon[97033]: pgmap v14996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:21:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:16 smithi059 ceph-mon[97450]: osdmap e1018: 12 total, 11 up, 11 in 2022-09-08T05:21:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:16 smithi059 ceph-mon[97450]: pgmap v14996: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:21:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:16.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:16.869172+0000) 2022-09-08T05:21:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:16.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:16.869312+0000) 2022-09-08T05:21:18.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:21:17.788+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:21:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:17.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:17.869413+0000) 2022-09-08T05:21:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:17.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:17.869472+0000) 2022-09-08T05:21:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:18.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:18.869618+0000) 2022-09-08T05:21:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:18.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:18.869604+0000) 2022-09-08T05:21:19.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:19 smithi059 ceph-mon[97450]: pgmap v14997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T05:21:19.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:19 smithi055 ceph-mon[97033]: pgmap v14997: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T05:21:20.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:19.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:19.869745+0000) 2022-09-08T05:21:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:19.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:19.869812+0000) 2022-09-08T05:21:21.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:20.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:20.869937+0000) 2022-09-08T05:21:21.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:20.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:20.870011+0000) 2022-09-08T05:21:21.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:21 smithi055 ceph-mon[97033]: pgmap v14998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:21:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:21 smithi059 ceph-mon[97450]: pgmap v14998: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:21:22.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:21.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:21.870257+0000) 2022-09-08T05:21:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:21.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:21.870193+0000) 2022-09-08T05:21:23.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:22.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:22.870486+0000) 2022-09-08T05:21:23.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:22.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:22.870485+0000) 2022-09-08T05:21:23.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:23 smithi055 ceph-mon[97033]: pgmap v14999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:21:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:23 smithi059 ceph-mon[97450]: pgmap v14999: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:21:24.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:23.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:23.870770+0000) 2022-09-08T05:21:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:23.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:23.870770+0000) 2022-09-08T05:21:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:24.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:24.870967+0000) 2022-09-08T05:21:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:24.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:24.870967+0000) 2022-09-08T05:21:25.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:25 smithi055 ceph-mon[97033]: pgmap v15000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:21:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:25 smithi059 ceph-mon[97450]: pgmap v15000: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:21:26.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:25.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:25.871187+0000) 2022-09-08T05:21:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:25.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:25.871187+0000) 2022-09-08T05:21:26.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:26 smithi055 ceph-mon[97033]: pgmap v15001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:21:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:26 smithi059 ceph-mon[97450]: pgmap v15001: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:21:27.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:26.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:26.871407+0000) 2022-09-08T05:21:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:26.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:26.871404+0000) 2022-09-08T05:21:28.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:27.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:27.871670+0000) 2022-09-08T05:21:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:27.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:27.871680+0000) 2022-09-08T05:21:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:28.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:28.871937+0000) 2022-09-08T05:21:29.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:28.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:28.871935+0000) 2022-09-08T05:21:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:29 smithi055 ceph-mon[97033]: pgmap v15002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:29 smithi059 ceph-mon[97450]: pgmap v15002: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:29.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:29.872137+0000) 2022-09-08T05:21:30.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:29.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:29.872150+0000) 2022-09-08T05:21:31.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:30.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:30.872365+0000) 2022-09-08T05:21:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:30.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:30.872367+0000) 2022-09-08T05:21:31.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:31 smithi055 ceph-mon[97033]: pgmap v15003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:21:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:31 smithi059 ceph-mon[97450]: pgmap v15003: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:21:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:31.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:31.872539+0000) 2022-09-08T05:21:32.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:31.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:31.872555+0000) 2022-09-08T05:21:33.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:21:32.803+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:21:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:32.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:32.872753+0000) 2022-09-08T05:21:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:32.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:32.872766+0000) 2022-09-08T05:21:33.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:33 smithi055 ceph-mon[97033]: pgmap v15004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:33 smithi059 ceph-mon[97450]: pgmap v15004: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:34.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:33.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:33.872989+0000) 2022-09-08T05:21:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:33.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:33.873000+0000) 2022-09-08T05:21:35.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:34.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:34.873208+0000) 2022-09-08T05:21:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:34.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:34.873225+0000) 2022-09-08T05:21:35.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:35 smithi055 ceph-mon[97033]: pgmap v15005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:35 smithi059 ceph-mon[97450]: pgmap v15005: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:35.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:35.873418+0000) 2022-09-08T05:21:36.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:35.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:35.873461+0000) 2022-09-08T05:21:36.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:36 smithi055 ceph-mon[97033]: pgmap v15006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:21:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:36 smithi059 ceph-mon[97450]: pgmap v15006: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:21:37.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:36.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:36.873580+0000) 2022-09-08T05:21:37.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:36.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:36.873698+0000) 2022-09-08T05:21:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:37.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:37.873755+0000) 2022-09-08T05:21:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:37.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:37.873941+0000) 2022-09-08T05:21:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:38.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:38.873930+0000) 2022-09-08T05:21:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:38.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:38.874088+0000) 2022-09-08T05:21:39.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:39 smithi055 ceph-mon[97033]: pgmap v15007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:39 smithi059 ceph-mon[97450]: pgmap v15007: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:39.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:39.874166+0000) 2022-09-08T05:21:40.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:39.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:39.874347+0000) 2022-09-08T05:21:40.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:21:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:21:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:21:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:21:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:21:40.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:21:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:21:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:21:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:40.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:40.874382+0000) 2022-09-08T05:21:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:40.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:40.874563+0000) 2022-09-08T05:21:41.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:41 smithi055 ceph-mon[97033]: pgmap v15008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:21:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:41 smithi059 ceph-mon[97450]: pgmap v15008: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:21:42.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:41.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:41.874598+0000) 2022-09-08T05:21:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:41.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:41.874710+0000) 2022-09-08T05:21:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:42.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:42.874797+0000) 2022-09-08T05:21:43.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:42.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:42.874932+0000) 2022-09-08T05:21:43.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:43 smithi055 ceph-mon[97033]: pgmap v15009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:43 smithi059 ceph-mon[97450]: pgmap v15009: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:44.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:43.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:43.875048+0000) 2022-09-08T05:21:44.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:43.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:43.875176+0000) 2022-09-08T05:21:45.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:44.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:44.875350+0000) 2022-09-08T05:21:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:44.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:44.875297+0000) 2022-09-08T05:21:45.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:45 smithi055 ceph-mon[97033]: pgmap v15010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:45.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:45 smithi059 ceph-mon[97450]: pgmap v15010: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:45.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:45.875485+0000) 2022-09-08T05:21:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:45.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:45.875507+0000) 2022-09-08T05:21:46.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:46 smithi055 ceph-mon[97033]: pgmap v15011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:21:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:46 smithi059 ceph-mon[97450]: pgmap v15011: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:21:47.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:46.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:46.875668+0000) 2022-09-08T05:21:47.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:46.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:46.875684+0000) 2022-09-08T05:21:48.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:47.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:47.875876+0000) 2022-09-08T05:21:48.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:47.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:47.875897+0000) 2022-09-08T05:21:48.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:21:47.813+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:21:49.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:48.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:48.876132+0000) 2022-09-08T05:21:49.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:48.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:48.876144+0000) 2022-09-08T05:21:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:49 smithi055 ceph-mon[97033]: pgmap v15012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:49 smithi059 ceph-mon[97450]: pgmap v15012: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:49.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:49.876368+0000) 2022-09-08T05:21:50.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:49.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:49.876378+0000) 2022-09-08T05:21:51.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:50.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:50.876605+0000) 2022-09-08T05:21:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:50.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:50.876582+0000) 2022-09-08T05:21:51.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:51 smithi055 ceph-mon[97033]: pgmap v15013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:21:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:51 smithi059 ceph-mon[97450]: pgmap v15013: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:21:52.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:51.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:51.876727+0000) 2022-09-08T05:21:52.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:51.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:51.876865+0000) 2022-09-08T05:21:53.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:52.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:52.877036+0000) 2022-09-08T05:21:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:52.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:52.876879+0000) 2022-09-08T05:21:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:53 smithi055 ceph-mon[97033]: pgmap v15014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:53 smithi059 ceph-mon[97450]: pgmap v15014: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:53.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:53.877027+0000) 2022-09-08T05:21:54.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:53.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:53.877238+0000) 2022-09-08T05:21:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:54.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:54.877396+0000) 2022-09-08T05:21:55.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:54.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:54.877208+0000) 2022-09-08T05:21:55.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:55 smithi055 ceph-mon[97033]: pgmap v15015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:55 smithi059 ceph-mon[97450]: pgmap v15015: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:56.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:55.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:55.877426+0000) 2022-09-08T05:21:56.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:55.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:55.877575+0000) 2022-09-08T05:21:56.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:56 smithi055 ceph-mon[97033]: pgmap v15016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:21:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:56 smithi059 ceph-mon[97450]: pgmap v15016: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:21:57.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:56.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:56.877777+0000) 2022-09-08T05:21:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:56.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:56.877711+0000) 2022-09-08T05:21:58.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:57.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:57.877995+0000) 2022-09-08T05:21:58.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:57.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:57.877886+0000) 2022-09-08T05:21:59.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:58.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:58.878238+0000) 2022-09-08T05:21:59.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:58.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:58.878056+0000) 2022-09-08T05:21:59.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:21:59 smithi055 ceph-mon[97033]: pgmap v15017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:21:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:21:59 smithi059 ceph-mon[97450]: pgmap v15017: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:21:59.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:59.878418+0000) 2022-09-08T05:22:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:21:59.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:21:59.878235+0000) 2022-09-08T05:22:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:00.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:00.878681+0000) 2022-09-08T05:22:01.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:00.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:00.878439+0000) 2022-09-08T05:22:01.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:01 smithi055 ceph-mon[97033]: pgmap v15018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:22:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:01 smithi059 ceph-mon[97450]: pgmap v15018: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:22:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:01.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:01.878808+0000) 2022-09-08T05:22:02.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:01.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:01.878664+0000) 2022-09-08T05:22:03.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:22:02.819+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:22:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:02.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:02.879006+0000) 2022-09-08T05:22:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:02.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:02.878831+0000) 2022-09-08T05:22:03.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:03 smithi055 ceph-mon[97033]: pgmap v15019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:03 smithi059 ceph-mon[97450]: pgmap v15019: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:04.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:03.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:03.879177+0000) 2022-09-08T05:22:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:03.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:03.879090+0000) 2022-09-08T05:22:05.160 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:04.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:04.879357+0000) 2022-09-08T05:22:05.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:04.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:04.879334+0000) 2022-09-08T05:22:05.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:05 smithi055 ceph-mon[97033]: pgmap v15020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:05.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:05 smithi059 ceph-mon[97450]: pgmap v15020: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:06.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:05.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:05.879542+0000) 2022-09-08T05:22:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:05.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:05.879561+0000) 2022-09-08T05:22:06.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:22:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:06 smithi055 ceph-mon[97033]: pgmap v15021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:22:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:22:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:22:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:06 smithi059 ceph-mon[97450]: pgmap v15021: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:22:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:22:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:06.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:06.879775+0000) 2022-09-08T05:22:07.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:06.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:06.879796+0000) 2022-09-08T05:22:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:07.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:07.880003+0000) 2022-09-08T05:22:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:07.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:07.880040+0000) 2022-09-08T05:22:09.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:08.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:08.880175+0000) 2022-09-08T05:22:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:08.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:08.880224+0000) 2022-09-08T05:22:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:09 smithi055 ceph-mon[97033]: pgmap v15022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:22:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:09 smithi059 ceph-mon[97450]: pgmap v15022: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:22:10.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:09.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:09.880350+0000) 2022-09-08T05:22:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:09.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:09.880396+0000) 2022-09-08T05:22:10.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:10 smithi055 ceph-mon[97033]: osdmap e1019: 12 total, 11 up, 11 in 2022-09-08T05:22:10.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:10 smithi059 ceph-mon[97450]: osdmap e1019: 12 total, 11 up, 11 in 2022-09-08T05:22:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:10.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:10.880579+0000) 2022-09-08T05:22:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:10.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:10.880667+0000) 2022-09-08T05:22:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:11 smithi055 ceph-mon[97033]: pgmap v15024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:22:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:11 smithi055 ceph-mon[97033]: osdmap e1020: 12 total, 11 up, 11 in 2022-09-08T05:22:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:11 smithi059 ceph-mon[97450]: pgmap v15024: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:22:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:11 smithi059 ceph-mon[97450]: osdmap e1020: 12 total, 11 up, 11 in 2022-09-08T05:22:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:11.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:11.880886+0000) 2022-09-08T05:22:12.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:11.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:11.880938+0000) 2022-09-08T05:22:13.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:12.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:12.881075+0000) 2022-09-08T05:22:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:12.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:12.881118+0000) 2022-09-08T05:22:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:13 smithi055 ceph-mon[97033]: pgmap v15026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:22:13.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:13 smithi059 ceph-mon[97450]: pgmap v15026: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:22:14.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:13.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:13.881266+0000) 2022-09-08T05:22:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:13.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:13.881291+0000) 2022-09-08T05:22:15.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:14.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:14.881443+0000) 2022-09-08T05:22:15.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:14.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:14.881468+0000) 2022-09-08T05:22:15.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:15 smithi055 ceph-mon[97033]: pgmap v15027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:22:15.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:15 smithi059 ceph-mon[97450]: pgmap v15027: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:22:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:15.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:15.881620+0000) 2022-09-08T05:22:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:15.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:15.881668+0000) 2022-09-08T05:22:16.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:16 smithi055 ceph-mon[97033]: pgmap v15028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:22:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:16 smithi059 ceph-mon[97450]: pgmap v15028: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:22:17.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:16.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:16.881785+0000) 2022-09-08T05:22:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:16.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:16.881808+0000) 2022-09-08T05:22:18.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:22:17.830+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:22:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:17.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:17.882081+0000) 2022-09-08T05:22:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:17.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:17.881988+0000) 2022-09-08T05:22:19.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:18.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:18.882179+0000) 2022-09-08T05:22:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:18.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:18.882230+0000) 2022-09-08T05:22:19.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:19 smithi055 ceph-mon[97033]: pgmap v15029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T05:22:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:19 smithi059 ceph-mon[97450]: pgmap v15029: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T05:22:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:19.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:19.882397+0000) 2022-09-08T05:22:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:19.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:19.882425+0000) 2022-09-08T05:22:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:20.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:20.882621+0000) 2022-09-08T05:22:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:20.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:20.882692+0000) 2022-09-08T05:22:21.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:21 smithi055 ceph-mon[97033]: pgmap v15030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:22:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:21 smithi059 ceph-mon[97450]: pgmap v15030: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:22:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:21.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:21.882844+0000) 2022-09-08T05:22:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:21.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:21.882942+0000) 2022-09-08T05:22:23.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:22.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:22.883078+0000) 2022-09-08T05:22:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:22.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:22.883115+0000) 2022-09-08T05:22:23.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:23 smithi055 ceph-mon[97033]: pgmap v15031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T05:22:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:23 smithi059 ceph-mon[97450]: pgmap v15031: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T05:22:24.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:23.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:23.883350+0000) 2022-09-08T05:22:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:23.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:23.883361+0000) 2022-09-08T05:22:25.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:24.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:24.883546+0000) 2022-09-08T05:22:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:24.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:24.883508+0000) 2022-09-08T05:22:25.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:25 smithi055 ceph-mon[97033]: pgmap v15032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:22:25.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:25 smithi059 ceph-mon[97450]: pgmap v15032: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:22:26.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:25.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:25.883732+0000) 2022-09-08T05:22:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:25.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:25.883794+0000) 2022-09-08T05:22:26.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:26 smithi055 ceph-mon[97033]: pgmap v15033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:22:27.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:26 smithi059 ceph-mon[97450]: pgmap v15033: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:22:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:26.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:26.883958+0000) 2022-09-08T05:22:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:26.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:26.883914+0000) 2022-09-08T05:22:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:27.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:27.884099+0000) 2022-09-08T05:22:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:27.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:27.884044+0000) 2022-09-08T05:22:29.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:28.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:28.884318+0000) 2022-09-08T05:22:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:28.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:28.884338+0000) 2022-09-08T05:22:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:29 smithi055 ceph-mon[97033]: pgmap v15034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:29.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:29 smithi059 ceph-mon[97450]: pgmap v15034: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:30.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:29.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:29.884556+0000) 2022-09-08T05:22:30.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:29.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:29.884586+0000) 2022-09-08T05:22:31.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:30.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:30.884877+0000) 2022-09-08T05:22:31.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:30.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:30.884831+0000) 2022-09-08T05:22:31.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:31 smithi055 ceph-mon[97033]: pgmap v15035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:22:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:31 smithi059 ceph-mon[97450]: pgmap v15035: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:22:32.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:31.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:31.885097+0000) 2022-09-08T05:22:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:31.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:31.885111+0000) 2022-09-08T05:22:33.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:22:32.844+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:22:33.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:32.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:32.885273+0000) 2022-09-08T05:22:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:32.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:32.885325+0000) 2022-09-08T05:22:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:33 smithi055 ceph-mon[97033]: pgmap v15036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:33 smithi059 ceph-mon[97450]: pgmap v15036: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:34.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:33.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:33.885470+0000) 2022-09-08T05:22:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:33.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:33.885508+0000) 2022-09-08T05:22:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:34.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:34.885674+0000) 2022-09-08T05:22:35.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:34.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:34.885692+0000) 2022-09-08T05:22:35.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:35 smithi055 ceph-mon[97033]: pgmap v15037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:35 smithi059 ceph-mon[97450]: pgmap v15037: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:36.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:35.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:35.885852+0000) 2022-09-08T05:22:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:35.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:35.885869+0000) 2022-09-08T05:22:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:36 smithi055 ceph-mon[97033]: pgmap v15038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:22:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:36 smithi059 ceph-mon[97450]: pgmap v15038: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:22:37.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:36.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:36.886027+0000) 2022-09-08T05:22:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:36.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:36.886096+0000) 2022-09-08T05:22:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:37.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:37.886272+0000) 2022-09-08T05:22:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:37.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:37.886328+0000) 2022-09-08T05:22:39.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:38.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:38.886568+0000) 2022-09-08T05:22:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:38.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:38.886566+0000) 2022-09-08T05:22:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:39 smithi055 ceph-mon[97033]: pgmap v15039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:39 smithi059 ceph-mon[97450]: pgmap v15039: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:40.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:39.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:39.886843+0000) 2022-09-08T05:22:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:39.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:39.886868+0000) 2022-09-08T05:22:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:40.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:40.887131+0000) 2022-09-08T05:22:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:40.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:40.887128+0000) 2022-09-08T05:22:41.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:41 smithi055 ceph-mon[97033]: pgmap v15040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:22:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:22:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:22:41.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:22:41.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:41.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:41 smithi059 ceph-mon[97450]: pgmap v15040: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:22:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:22:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:22:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:22:41.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:41.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:41.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:41.887380+0000) 2022-09-08T05:22:42.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:41.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:41.887383+0000) 2022-09-08T05:22:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:42.257 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:42.258 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:42.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:42.887672+0000) 2022-09-08T05:22:43.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:42.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:42.887621+0000) 2022-09-08T05:22:43.398 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:43 smithi059 ceph-mon[97450]: pgmap v15041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:43.451 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:43 smithi055 ceph-mon[97033]: pgmap v15041: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:44.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:43.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:43.887984+0000) 2022-09-08T05:22:44.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:43.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:43.887937+0000) 2022-09-08T05:22:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:44 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T05:22:44.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:44.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:44 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T05:22:44.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:44.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:44.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:44.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:44 smithi055 ceph-mon[97033]: pgmap v15042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:44 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T05:22:45.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:44 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T05:22:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:22:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:44 smithi059 ceph-mon[97450]: pgmap v15042: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:45.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:44.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:44.888212+0000) 2022-09-08T05:22:45.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:44.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:44.888174+0000) 2022-09-08T05:22:46.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:45.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:45.888449+0000) 2022-09-08T05:22:46.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:45.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:45.888400+0000) 2022-09-08T05:22:46.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:46 smithi055 ceph-mon[97033]: pgmap v15043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:22:47.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:46 smithi059 ceph-mon[97450]: pgmap v15043: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:22:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:46.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:46.888627+0000) 2022-09-08T05:22:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:46.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:46.888557+0000) 2022-09-08T05:22:48.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:22:47.858+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:22:48.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:47.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:47.888895+0000) 2022-09-08T05:22:48.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:47.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:47.888800+0000) 2022-09-08T05:22:49.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:48.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:48.889132+0000) 2022-09-08T05:22:49.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:48.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:48.889075+0000) 2022-09-08T05:22:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:49 smithi055 ceph-mon[97033]: pgmap v15044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:49.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:49 smithi059 ceph-mon[97450]: pgmap v15044: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:50.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:49.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:49.889263+0000) 2022-09-08T05:22:50.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:49.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:49.889250+0000) 2022-09-08T05:22:51.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:50.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:50.889435+0000) 2022-09-08T05:22:51.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:50.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:50.889467+0000) 2022-09-08T05:22:51.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:51 smithi055 ceph-mon[97033]: pgmap v15045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:22:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:51 smithi059 ceph-mon[97450]: pgmap v15045: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:22:52.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:51.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:51.889723+0000) 2022-09-08T05:22:52.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:51.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:51.889740+0000) 2022-09-08T05:22:53.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:52.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:52.889979+0000) 2022-09-08T05:22:53.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:52.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:52.889997+0000) 2022-09-08T05:22:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:53 smithi055 ceph-mon[97033]: pgmap v15046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:53.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:53 smithi059 ceph-mon[97450]: pgmap v15046: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:54.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:53.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:53.890174+0000) 2022-09-08T05:22:54.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:53.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:53.890276+0000) 2022-09-08T05:22:55.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:54.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:54.890412+0000) 2022-09-08T05:22:55.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:54.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:54.890537+0000) 2022-09-08T05:22:55.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:55 smithi055 ceph-mon[97033]: pgmap v15047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:55.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:55 smithi059 ceph-mon[97450]: pgmap v15047: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:56.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:55.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:55.890684+0000) 2022-09-08T05:22:56.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:55.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:55.890772+0000) 2022-09-08T05:22:56.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:56 smithi055 ceph-mon[97033]: pgmap v15048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:22:57.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:56 smithi059 ceph-mon[97450]: pgmap v15048: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:22:57.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:56.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:56.890929+0000) 2022-09-08T05:22:57.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:56.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:56.891017+0000) 2022-09-08T05:22:58.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:57.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:57.891148+0000) 2022-09-08T05:22:58.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:57.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:57.891206+0000) 2022-09-08T05:22:59.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:58.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:58.891375+0000) 2022-09-08T05:22:59.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:58.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:58.891453+0000) 2022-09-08T05:22:59.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:22:59 smithi055 ceph-mon[97033]: pgmap v15049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:22:59.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:22:59 smithi059 ceph-mon[97450]: pgmap v15049: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:00.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:22:59.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:59.891546+0000) 2022-09-08T05:23:00.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:22:59.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:22:59.891677+0000) 2022-09-08T05:23:01.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:00.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:00.891693+0000) 2022-09-08T05:23:01.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:00.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:00.891818+0000) 2022-09-08T05:23:01.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:01 smithi055 ceph-mon[97033]: pgmap v15050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:23:01.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:01 smithi059 ceph-mon[97450]: pgmap v15050: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:23:02.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:01.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:01.891968+0000) 2022-09-08T05:23:02.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:01.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:01.891889+0000) 2022-09-08T05:23:03.150 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:23:02.873+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:23:03.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:02.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:02.892065+0000) 2022-09-08T05:23:03.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:02.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:02.892136+0000) 2022-09-08T05:23:03.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:03 smithi055 ceph-mon[97033]: pgmap v15051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:03.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:03 smithi059 ceph-mon[97450]: pgmap v15051: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:04.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:03.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:03.892308+0000) 2022-09-08T05:23:04.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:03.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:03.892347+0000) 2022-09-08T05:23:05.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:04.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:04.892570+0000) 2022-09-08T05:23:05.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:04.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:04.892595+0000) 2022-09-08T05:23:05.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:05 smithi055 ceph-mon[97033]: pgmap v15052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:05.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:05 smithi059 ceph-mon[97450]: pgmap v15052: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:06.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:05.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:05.892847+0000) 2022-09-08T05:23:06.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:05.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:05.892866+0000) 2022-09-08T05:23:06.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:23:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:06 smithi055 ceph-mon[97033]: pgmap v15053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:23:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:23:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:23:07.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:06 smithi059 ceph-mon[97450]: pgmap v15053: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:23:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:23:07.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:06.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:06.892969+0000) 2022-09-08T05:23:07.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:06.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:06.893030+0000) 2022-09-08T05:23:08.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:07.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:07.893136+0000) 2022-09-08T05:23:08.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:07.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:07.893197+0000) 2022-09-08T05:23:09.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:08.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:08.893360+0000) 2022-09-08T05:23:09.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:08.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:08.893343+0000) 2022-09-08T05:23:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:09 smithi055 ceph-mon[97033]: pgmap v15054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:23:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:09 smithi059 ceph-mon[97450]: pgmap v15054: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:23:10.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:09.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:09.893549+0000) 2022-09-08T05:23:10.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:09.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:09.893599+0000) 2022-09-08T05:23:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:10 smithi055 ceph-mon[97033]: osdmap e1021: 12 total, 11 up, 11 in 2022-09-08T05:23:10.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:10 smithi059 ceph-mon[97450]: osdmap e1021: 12 total, 11 up, 11 in 2022-09-08T05:23:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:10.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:10.893809+0000) 2022-09-08T05:23:11.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:10.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:10.893815+0000) 2022-09-08T05:23:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:11 smithi055 ceph-mon[97033]: pgmap v15056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:23:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:11 smithi055 ceph-mon[97033]: osdmap e1022: 12 total, 11 up, 11 in 2022-09-08T05:23:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:11 smithi059 ceph-mon[97450]: pgmap v15056: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:23:11.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:11 smithi059 ceph-mon[97450]: osdmap e1022: 12 total, 11 up, 11 in 2022-09-08T05:23:12.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:11.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:11.893983+0000) 2022-09-08T05:23:12.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:11.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:11.894008+0000) 2022-09-08T05:23:13.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:12.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:12.894178+0000) 2022-09-08T05:23:13.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:12.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:12.894163+0000) 2022-09-08T05:23:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:13 smithi055 ceph-mon[97033]: pgmap v15058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:23:13.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:13 smithi059 ceph-mon[97450]: pgmap v15058: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:23:14.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:13.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:13.894488+0000) 2022-09-08T05:23:14.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:13.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:13.894376+0000) 2022-09-08T05:23:15.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:14.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:14.894612+0000) 2022-09-08T05:23:15.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:14.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:14.894774+0000) 2022-09-08T05:23:15.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:15 smithi055 ceph-mon[97033]: pgmap v15059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:23:15.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:15 smithi059 ceph-mon[97450]: pgmap v15059: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:23:16.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:15.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:15.894871+0000) 2022-09-08T05:23:16.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:15.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:15.894962+0000) 2022-09-08T05:23:16.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:16 smithi055 ceph-mon[97033]: pgmap v15060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:23:17.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:16 smithi059 ceph-mon[97450]: pgmap v15060: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:23:17.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:16.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:16.895138+0000) 2022-09-08T05:23:17.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:16.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:16.895176+0000) 2022-09-08T05:23:18.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:23:17.889+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:23:18.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:17.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:17.895383+0000) 2022-09-08T05:23:18.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:17.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:17.895416+0000) 2022-09-08T05:23:19.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:18.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:18.895676+0000) 2022-09-08T05:23:19.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:18.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:18.895716+0000) 2022-09-08T05:23:19.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:19 smithi055 ceph-mon[97033]: pgmap v15061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T05:23:19.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:19 smithi059 ceph-mon[97450]: pgmap v15061: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T05:23:20.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:19.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:19.895898+0000) 2022-09-08T05:23:20.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:19.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:19.895943+0000) 2022-09-08T05:23:21.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:20.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:20.896058+0000) 2022-09-08T05:23:21.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:20.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:20.896107+0000) 2022-09-08T05:23:21.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:21 smithi055 ceph-mon[97033]: pgmap v15062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T05:23:21.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:21 smithi059 ceph-mon[97450]: pgmap v15062: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T05:23:22.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:21.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:21.896322+0000) 2022-09-08T05:23:22.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:21.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:21.896310+0000) 2022-09-08T05:23:23.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:22.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:22.896545+0000) 2022-09-08T05:23:23.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:22.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:22.896567+0000) 2022-09-08T05:23:23.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:23 smithi055 ceph-mon[97033]: pgmap v15063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T05:23:23.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:23 smithi059 ceph-mon[97450]: pgmap v15063: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T05:23:24.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:23.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:23.896739+0000) 2022-09-08T05:23:24.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:23.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:23.896773+0000) 2022-09-08T05:23:25.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:24.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:24.896901+0000) 2022-09-08T05:23:25.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:24.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:24.896936+0000) 2022-09-08T05:23:25.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:25 smithi055 ceph-mon[97033]: pgmap v15064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2022-09-08T05:23:25.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:25 smithi059 ceph-mon[97450]: pgmap v15064: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2022-09-08T05:23:26.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:25.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:25.897046+0000) 2022-09-08T05:23:26.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:25.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:25.897099+0000) 2022-09-08T05:23:26.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:26 smithi055 ceph-mon[97033]: pgmap v15065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:23:27.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:26 smithi059 ceph-mon[97450]: pgmap v15065: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:23:27.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:26.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:26.897315+0000) 2022-09-08T05:23:27.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:26.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:26.897306+0000) 2022-09-08T05:23:28.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:27.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:27.897524+0000) 2022-09-08T05:23:28.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:27.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:27.897481+0000) 2022-09-08T05:23:29.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:28.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:28.897716+0000) 2022-09-08T05:23:29.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:28.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:28.897709+0000) 2022-09-08T05:23:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:29 smithi055 ceph-mon[97033]: pgmap v15066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:23:29.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:29 smithi059 ceph-mon[97450]: pgmap v15066: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:23:30.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:29.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:29.897884+0000) 2022-09-08T05:23:30.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:29.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:29.897882+0000) 2022-09-08T05:23:31.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:30.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:30.898022+0000) 2022-09-08T05:23:31.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:30.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:30.898018+0000) 2022-09-08T05:23:31.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:31 smithi055 ceph-mon[97033]: pgmap v15067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:23:31.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:31 smithi059 ceph-mon[97450]: pgmap v15067: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:23:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:31.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:31.898273+0000) 2022-09-08T05:23:32.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:31.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:31.898262+0000) 2022-09-08T05:23:33.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:32.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:32.898552+0000) 2022-09-08T05:23:33.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:23:32.905+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:23:33.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:32.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:32.898536+0000) 2022-09-08T05:23:33.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:33 smithi055 ceph-mon[97033]: pgmap v15068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:33.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:33 smithi059 ceph-mon[97450]: pgmap v15068: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:34.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:33.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:33.898779+0000) 2022-09-08T05:23:34.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:33.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:33.898783+0000) 2022-09-08T05:23:35.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:34.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:34.898903+0000) 2022-09-08T05:23:35.176 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:34.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:34.898939+0000) 2022-09-08T05:23:35.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:35 smithi055 ceph-mon[97033]: pgmap v15069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:35.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:35 smithi059 ceph-mon[97450]: pgmap v15069: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:36.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:35.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:35.899110+0000) 2022-09-08T05:23:36.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:35.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:35.899186+0000) 2022-09-08T05:23:36.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:36 smithi055 ceph-mon[97033]: pgmap v15070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:23:37.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:36 smithi059 ceph-mon[97450]: pgmap v15070: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:23:37.150 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:36.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:36.899316+0000) 2022-09-08T05:23:37.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:36.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:36.899357+0000) 2022-09-08T05:23:38.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:37.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:37.899548+0000) 2022-09-08T05:23:38.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:37.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:37.899574+0000) 2022-09-08T05:23:39.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:38.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:38.899869+0000) 2022-09-08T05:23:39.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:38.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:38.899831+0000) 2022-09-08T05:23:39.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:39 smithi055 ceph-mon[97033]: pgmap v15071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:39.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:39 smithi059 ceph-mon[97450]: pgmap v15071: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:40.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:39.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:39.900124+0000) 2022-09-08T05:23:40.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:39.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:39.900104+0000) 2022-09-08T05:23:41.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:40.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:40.900377+0000) 2022-09-08T05:23:41.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:40.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:40.900379+0000) 2022-09-08T05:23:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:41 smithi055 ceph-mon[97033]: pgmap v15072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:23:41.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:41 smithi059 ceph-mon[97450]: pgmap v15072: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:23:42.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:41.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:41.900716+0000) 2022-09-08T05:23:42.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:41.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:41.900716+0000) 2022-09-08T05:23:43.150 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:42.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:42.901011+0000) 2022-09-08T05:23:43.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:42.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:42.901012+0000) 2022-09-08T05:23:43.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:43 smithi055 ceph-mon[97033]: pgmap v15073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:43.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:43 smithi059 ceph-mon[97450]: pgmap v15073: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:23:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:23:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:23:44.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:23:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:23:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:23:44.400 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:43.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:43.901327+0000) 2022-09-08T05:23:44.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:43.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:43.901335+0000) 2022-09-08T05:23:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:44.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:44.901564+0000) 2022-09-08T05:23:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:44.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:44.901552+0000) 2022-09-08T05:23:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:45 smithi055 ceph-mon[97033]: pgmap v15074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:45.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:23:45.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:45 smithi059 ceph-mon[97450]: pgmap v15074: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:23:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:45.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:45.901778+0000) 2022-09-08T05:23:46.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:45.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:45.901760+0000) 2022-09-08T05:23:46.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:46 smithi055 ceph-mon[97033]: pgmap v15075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:23:47.029 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:46 smithi059 ceph-mon[97450]: pgmap v15075: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:23:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:46.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:46.901941+0000) 2022-09-08T05:23:47.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:46.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:46.901930+0000) 2022-09-08T05:23:48.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:23:47.919+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:23:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:47.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:47.902075+0000) 2022-09-08T05:23:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:47.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:47.902105+0000) 2022-09-08T05:23:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:48.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:48.902303+0000) 2022-09-08T05:23:49.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:48.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:48.902331+0000) 2022-09-08T05:23:49.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:49 smithi055 ceph-mon[97033]: pgmap v15076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:49.529 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:49 smithi059 ceph-mon[97450]: pgmap v15076: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:49.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:49.902548+0000) 2022-09-08T05:23:50.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:49.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:49.902531+0000) 2022-09-08T05:23:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:50.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:50.902788+0000) 2022-09-08T05:23:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:50.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:50.902798+0000) 2022-09-08T05:23:51.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:51 smithi055 ceph-mon[97033]: pgmap v15077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:23:51.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:51 smithi059 ceph-mon[97450]: pgmap v15077: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:23:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:51.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:51.903009+0000) 2022-09-08T05:23:52.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:51.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:51.903008+0000) 2022-09-08T05:23:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:52.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:52.903262+0000) 2022-09-08T05:23:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:52.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:52.903261+0000) 2022-09-08T05:23:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:53 smithi055 ceph-mon[97033]: pgmap v15078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:53 smithi059 ceph-mon[97450]: pgmap v15078: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:53.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:53.903502+0000) 2022-09-08T05:23:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:53.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:53.903502+0000) 2022-09-08T05:23:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:54.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:54.903658+0000) 2022-09-08T05:23:55.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:54.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:54.903658+0000) 2022-09-08T05:23:55.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:55 smithi055 ceph-mon[97033]: pgmap v15079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:55 smithi059 ceph-mon[97450]: pgmap v15079: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:55.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:55.903847+0000) 2022-09-08T05:23:56.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:55.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:55.903845+0000) 2022-09-08T05:23:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:56 smithi055 ceph-mon[97033]: pgmap v15080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:23:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:56 smithi059 ceph-mon[97450]: pgmap v15080: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:23:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:56.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:56.904053+0000) 2022-09-08T05:23:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:56.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:56.904049+0000) 2022-09-08T05:23:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:57.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:57.904289+0000) 2022-09-08T05:23:58.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:57.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:57.904289+0000) 2022-09-08T05:23:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:58.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:58.904570+0000) 2022-09-08T05:23:59.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:58.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:58.904570+0000) 2022-09-08T05:23:59.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:23:59 smithi055 ceph-mon[97033]: pgmap v15081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:23:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:23:59 smithi059 ceph-mon[97450]: pgmap v15081: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:23:59.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:59.904806+0000) 2022-09-08T05:24:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:23:59.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:23:59.904810+0000) 2022-09-08T05:24:01.400 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:00.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:00.905014+0000) 2022-09-08T05:24:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:00.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:00.905021+0000) 2022-09-08T05:24:01.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:01 smithi055 ceph-mon[97033]: pgmap v15082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:24:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:01 smithi059 ceph-mon[97450]: pgmap v15082: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:24:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:01.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:01.905305+0000) 2022-09-08T05:24:02.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:01.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:01.905305+0000) 2022-09-08T05:24:03.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:24:02.929+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:24:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:02.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:02.905546+0000) 2022-09-08T05:24:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:02.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:02.905546+0000) 2022-09-08T05:24:03.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:03 smithi055 ceph-mon[97033]: pgmap v15083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:03 smithi059 ceph-mon[97450]: pgmap v15083: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:03.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:03.905828+0000) 2022-09-08T05:24:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:03.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:03.905826+0000) 2022-09-08T05:24:05.400 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:04.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:04.906129+0000) 2022-09-08T05:24:05.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:04.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:04.906128+0000) 2022-09-08T05:24:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:05 smithi055 ceph-mon[97033]: pgmap v15084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:05 smithi059 ceph-mon[97450]: pgmap v15084: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:05.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:05.906421+0000) 2022-09-08T05:24:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:05.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:05.906426+0000) 2022-09-08T05:24:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:24:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:06 smithi055 ceph-mon[97033]: pgmap v15085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:24:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:24:07.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:24:07.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:06 smithi059 ceph-mon[97450]: pgmap v15085: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:24:07.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:24:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:06.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:06.906735+0000) 2022-09-08T05:24:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:06.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:06.906732+0000) 2022-09-08T05:24:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:07.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:07.907014+0000) 2022-09-08T05:24:08.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:07.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:07.907014+0000) 2022-09-08T05:24:09.400 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:08.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:08.907248+0000) 2022-09-08T05:24:09.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:08.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:08.907247+0000) 2022-09-08T05:24:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:09 smithi055 ceph-mon[97033]: pgmap v15086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:24:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:09 smithi059 ceph-mon[97450]: pgmap v15086: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:24:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:09.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:09.907528+0000) 2022-09-08T05:24:10.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:09.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:09.907535+0000) 2022-09-08T05:24:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:10 smithi055 ceph-mon[97033]: osdmap e1023: 12 total, 11 up, 11 in 2022-09-08T05:24:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:10 smithi059 ceph-mon[97450]: osdmap e1023: 12 total, 11 up, 11 in 2022-09-08T05:24:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:10.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:10.907834+0000) 2022-09-08T05:24:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:10.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:10.907844+0000) 2022-09-08T05:24:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:11 smithi055 ceph-mon[97033]: pgmap v15088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:24:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:11 smithi055 ceph-mon[97033]: osdmap e1024: 12 total, 11 up, 11 in 2022-09-08T05:24:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:11 smithi059 ceph-mon[97450]: pgmap v15088: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:24:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:11 smithi059 ceph-mon[97450]: osdmap e1024: 12 total, 11 up, 11 in 2022-09-08T05:24:12.400 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:11.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:11.908012+0000) 2022-09-08T05:24:12.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:11.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:11.908027+0000) 2022-09-08T05:24:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:12.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:12.908215+0000) 2022-09-08T05:24:13.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:12.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:12.908218+0000) 2022-09-08T05:24:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:13 smithi055 ceph-mon[97033]: pgmap v15090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:24:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:13 smithi059 ceph-mon[97450]: pgmap v15090: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:24:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:13.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:13.908477+0000) 2022-09-08T05:24:14.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:13.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:13.908480+0000) 2022-09-08T05:24:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:14.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:14.908794+0000) 2022-09-08T05:24:15.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:14.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:14.908793+0000) 2022-09-08T05:24:15.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:15 smithi055 ceph-mon[97033]: pgmap v15091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:24:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:15 smithi059 ceph-mon[97450]: pgmap v15091: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:24:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:15.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:15.909038+0000) 2022-09-08T05:24:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:15.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:15.909038+0000) 2022-09-08T05:24:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:16 smithi055 ceph-mon[97033]: pgmap v15092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:24:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:16 smithi059 ceph-mon[97450]: pgmap v15092: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:24:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:16.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:16.909274+0000) 2022-09-08T05:24:17.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:16.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:16.909274+0000) 2022-09-08T05:24:18.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:24:17.942+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:24:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:17.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:17.909575+0000) 2022-09-08T05:24:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:17.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:17.909575+0000) 2022-09-08T05:24:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:18.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:18.909854+0000) 2022-09-08T05:24:19.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:18.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:18.909852+0000) 2022-09-08T05:24:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:19 smithi055 ceph-mon[97033]: pgmap v15093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T05:24:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:19 smithi059 ceph-mon[97450]: pgmap v15093: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T05:24:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:19.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:19.910100+0000) 2022-09-08T05:24:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:19.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:19.910103+0000) 2022-09-08T05:24:21.400 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:20.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:20.910348+0000) 2022-09-08T05:24:21.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:20.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:20.910347+0000) 2022-09-08T05:24:21.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:21 smithi055 ceph-mon[97033]: pgmap v15094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:24:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:21 smithi059 ceph-mon[97450]: pgmap v15094: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:24:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:21.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:21.910498+0000) 2022-09-08T05:24:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:21.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:21.910506+0000) 2022-09-08T05:24:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:22.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:22.910726+0000) 2022-09-08T05:24:23.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:22.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:22.910726+0000) 2022-09-08T05:24:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:23 smithi055 ceph-mon[97033]: pgmap v15095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T05:24:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:23 smithi059 ceph-mon[97450]: pgmap v15095: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T05:24:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:23.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:23.910950+0000) 2022-09-08T05:24:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:23.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:23.910940+0000) 2022-09-08T05:24:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:24.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:24.911141+0000) 2022-09-08T05:24:25.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:24.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:24.911126+0000) 2022-09-08T05:24:25.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:25 smithi055 ceph-mon[97033]: pgmap v15096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:24:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:25 smithi059 ceph-mon[97450]: pgmap v15096: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:24:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:25.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:25.911381+0000) 2022-09-08T05:24:26.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:25.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:25.911381+0000) 2022-09-08T05:24:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:26 smithi055 ceph-mon[97033]: pgmap v15097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:24:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:26 smithi059 ceph-mon[97450]: pgmap v15097: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:24:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:26.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:26.911698+0000) 2022-09-08T05:24:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:26.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:26.911698+0000) 2022-09-08T05:24:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:27.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:27.911991+0000) 2022-09-08T05:24:28.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:27.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:27.911991+0000) 2022-09-08T05:24:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:28.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:28.912273+0000) 2022-09-08T05:24:29.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:28.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:28.912274+0000) 2022-09-08T05:24:29.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:29 smithi055 ceph-mon[97033]: pgmap v15098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:29 smithi059 ceph-mon[97450]: pgmap v15098: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:29.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:29.912494+0000) 2022-09-08T05:24:30.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:29.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:29.912492+0000) 2022-09-08T05:24:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:30.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:30.912732+0000) 2022-09-08T05:24:31.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:30.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:30.912732+0000) 2022-09-08T05:24:31.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:31 smithi055 ceph-mon[97033]: pgmap v15099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:24:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:31 smithi059 ceph-mon[97450]: pgmap v15099: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:24:32.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:31.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:31.912990+0000) 2022-09-08T05:24:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:31.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:31.912996+0000) 2022-09-08T05:24:33.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:24:32.945+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:24:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:32.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:32.913287+0000) 2022-09-08T05:24:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:32.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:32.913286+0000) 2022-09-08T05:24:33.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:33 smithi055 ceph-mon[97033]: pgmap v15100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:33 smithi059 ceph-mon[97450]: pgmap v15100: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:33.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:33.913587+0000) 2022-09-08T05:24:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:33.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:33.913587+0000) 2022-09-08T05:24:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:34.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:34.913851+0000) 2022-09-08T05:24:35.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:34.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:34.913856+0000) 2022-09-08T05:24:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:35 smithi055 ceph-mon[97033]: pgmap v15101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:35 smithi059 ceph-mon[97450]: pgmap v15101: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:36.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:35.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:35.914079+0000) 2022-09-08T05:24:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:35.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:35.914079+0000) 2022-09-08T05:24:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:36 smithi055 ceph-mon[97033]: pgmap v15102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:24:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:36 smithi059 ceph-mon[97450]: pgmap v15102: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:24:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:36.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:36.914311+0000) 2022-09-08T05:24:37.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:36.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:36.914310+0000) 2022-09-08T05:24:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:37.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:37.914506+0000) 2022-09-08T05:24:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:37.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:37.914506+0000) 2022-09-08T05:24:39.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:38.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:38.914768+0000) 2022-09-08T05:24:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:38.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:38.914774+0000) 2022-09-08T05:24:39.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:39 smithi055 ceph-mon[97033]: pgmap v15103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:39 smithi059 ceph-mon[97450]: pgmap v15103: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:39.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:39.915052+0000) 2022-09-08T05:24:40.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:39.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:39.915052+0000) 2022-09-08T05:24:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:40.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:40.915319+0000) 2022-09-08T05:24:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:40.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:40.915319+0000) 2022-09-08T05:24:41.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:41 smithi055 ceph-mon[97033]: pgmap v15104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:24:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:41 smithi059 ceph-mon[97450]: pgmap v15104: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:24:42.400 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:41.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:41.915624+0000) 2022-09-08T05:24:42.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:41.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:41.915624+0000) 2022-09-08T05:24:43.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:42.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:42.915930+0000) 2022-09-08T05:24:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:42.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:42.915930+0000) 2022-09-08T05:24:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:43 smithi055 ceph-mon[97033]: pgmap v15105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:43 smithi059 ceph-mon[97450]: pgmap v15105: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:44.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:43.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:43.916180+0000) 2022-09-08T05:24:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:43.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:43.916183+0000) 2022-09-08T05:24:45.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:44.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:44.916456+0000) 2022-09-08T05:24:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:44.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:44.916456+0000) 2022-09-08T05:24:45.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:45 smithi055 ceph-mon[97033]: pgmap v15106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:24:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:24:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:24:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:24:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:45 smithi059 ceph-mon[97450]: pgmap v15106: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:24:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:24:45.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:24:45.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:24:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:45.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:45.916761+0000) 2022-09-08T05:24:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:45.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:45.916759+0000) 2022-09-08T05:24:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:46 smithi055 ceph-mon[97033]: pgmap v15107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:24:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:46 smithi059 ceph-mon[97450]: pgmap v15107: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:24:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:46.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:46.917070+0000) 2022-09-08T05:24:47.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:46.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:46.917066+0000) 2022-09-08T05:24:48.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:24:47.956+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:24:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:47.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:47.917231+0000) 2022-09-08T05:24:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:47.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:47.917233+0000) 2022-09-08T05:24:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:48.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:48.917482+0000) 2022-09-08T05:24:49.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:48.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:48.917482+0000) 2022-09-08T05:24:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:49 smithi055 ceph-mon[97033]: pgmap v15108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:49 smithi059 ceph-mon[97450]: pgmap v15108: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:50.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:49.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:49.917793+0000) 2022-09-08T05:24:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:49.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:49.917794+0000) 2022-09-08T05:24:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:50.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:50.917972+0000) 2022-09-08T05:24:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:50.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:50.917970+0000) 2022-09-08T05:24:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:51 smithi055 ceph-mon[97033]: pgmap v15109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:24:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:51 smithi059 ceph-mon[97450]: pgmap v15109: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:24:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:51.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:51.918235+0000) 2022-09-08T05:24:52.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:51.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:51.918233+0000) 2022-09-08T05:24:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:52.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:52.918422+0000) 2022-09-08T05:24:53.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:52.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:52.918419+0000) 2022-09-08T05:24:53.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:53 smithi055 ceph-mon[97033]: pgmap v15110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:53 smithi059 ceph-mon[97450]: pgmap v15110: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:54.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:53.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:53.918662+0000) 2022-09-08T05:24:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:53.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:53.918662+0000) 2022-09-08T05:24:55.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:54.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:54.918800+0000) 2022-09-08T05:24:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:54.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:54.918812+0000) 2022-09-08T05:24:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:55 smithi055 ceph-mon[97033]: pgmap v15111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:55 smithi059 ceph-mon[97450]: pgmap v15111: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:55.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:55.918959+0000) 2022-09-08T05:24:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:55.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:55.918956+0000) 2022-09-08T05:24:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:56 smithi055 ceph-mon[97033]: pgmap v15112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:24:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:56 smithi059 ceph-mon[97450]: pgmap v15112: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:24:57.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:56.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:56.919193+0000) 2022-09-08T05:24:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:56.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:56.919212+0000) 2022-09-08T05:24:58.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:57.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:57.919389+0000) 2022-09-08T05:24:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:57.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:57.919400+0000) 2022-09-08T05:24:59.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:58.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:58.919625+0000) 2022-09-08T05:24:59.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:58.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:58.919624+0000) 2022-09-08T05:24:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:24:59 smithi055 ceph-mon[97033]: pgmap v15113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:24:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:24:59 smithi059 ceph-mon[97450]: pgmap v15113: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:24:59.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:59.919891+0000) 2022-09-08T05:25:00.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:24:59.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:24:59.919894+0000) 2022-09-08T05:25:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:00.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:00.920090+0000) 2022-09-08T05:25:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:00.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:00.920100+0000) 2022-09-08T05:25:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:01 smithi055 ceph-mon[97033]: pgmap v15114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:25:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:01 smithi059 ceph-mon[97450]: pgmap v15114: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:25:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:01.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:01.920279+0000) 2022-09-08T05:25:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:01.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:01.920341+0000) 2022-09-08T05:25:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:02.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:02.920522+0000) 2022-09-08T05:25:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:02.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:02.920587+0000) 2022-09-08T05:25:03.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:25:02.964+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:25:03.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:03 smithi055 ceph-mon[97033]: pgmap v15115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:03 smithi059 ceph-mon[97450]: pgmap v15115: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:03.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:03.920742+0000) 2022-09-08T05:25:04.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:03.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:03.920752+0000) 2022-09-08T05:25:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:04.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:04.921007+0000) 2022-09-08T05:25:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:04.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:04.920995+0000) 2022-09-08T05:25:05.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:05 smithi055 ceph-mon[97033]: pgmap v15116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:05 smithi059 ceph-mon[97450]: pgmap v15116: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:05.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:05.921222+0000) 2022-09-08T05:25:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:05.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:05.921235+0000) 2022-09-08T05:25:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:25:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:06 smithi055 ceph-mon[97033]: pgmap v15117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:25:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:25:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:25:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:06 smithi059 ceph-mon[97450]: pgmap v15117: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:25:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:25:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:06.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:06.921478+0000) 2022-09-08T05:25:07.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:06.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:06.921479+0000) 2022-09-08T05:25:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:07.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:07.921726+0000) 2022-09-08T05:25:08.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:07.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:07.921737+0000) 2022-09-08T05:25:09.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:08.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:08.921971+0000) 2022-09-08T05:25:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:08.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:08.921978+0000) 2022-09-08T05:25:09.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:09 smithi055 ceph-mon[97033]: pgmap v15118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:25:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:09 smithi059 ceph-mon[97450]: pgmap v15118: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:25:10.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:09.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:09.922214+0000) 2022-09-08T05:25:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:09.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:09.922209+0000) 2022-09-08T05:25:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:10 smithi055 ceph-mon[97033]: osdmap e1025: 12 total, 11 up, 11 in 2022-09-08T05:25:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:10 smithi059 ceph-mon[97450]: osdmap e1025: 12 total, 11 up, 11 in 2022-09-08T05:25:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:10.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:10.922415+0000) 2022-09-08T05:25:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:10.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:10.922426+0000) 2022-09-08T05:25:11.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:11 smithi055 ceph-mon[97033]: pgmap v15120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:25:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:11 smithi055 ceph-mon[97033]: osdmap e1026: 12 total, 11 up, 11 in 2022-09-08T05:25:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:11 smithi059 ceph-mon[97450]: pgmap v15120: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:25:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:11 smithi059 ceph-mon[97450]: osdmap e1026: 12 total, 11 up, 11 in 2022-09-08T05:25:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:11.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:11.922682+0000) 2022-09-08T05:25:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:11.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:11.922671+0000) 2022-09-08T05:25:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:12.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:12.922924+0000) 2022-09-08T05:25:13.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:12.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:12.922924+0000) 2022-09-08T05:25:13.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:13 smithi055 ceph-mon[97033]: pgmap v15122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:25:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:13 smithi059 ceph-mon[97450]: pgmap v15122: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:25:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:13.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:13.923208+0000) 2022-09-08T05:25:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:13.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:13.923216+0000) 2022-09-08T05:25:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:14.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:14.923398+0000) 2022-09-08T05:25:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:14.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:14.923390+0000) 2022-09-08T05:25:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:15 smithi055 ceph-mon[97033]: pgmap v15123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:25:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:15 smithi059 ceph-mon[97450]: pgmap v15123: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:25:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:15.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:15.923616+0000) 2022-09-08T05:25:16.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:15.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:15.923608+0000) 2022-09-08T05:25:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:16 smithi055 ceph-mon[97033]: pgmap v15124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 7.6 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2022-09-08T05:25:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:16 smithi059 ceph-mon[97450]: pgmap v15124: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 7.6 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2022-09-08T05:25:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:16.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:16.923774+0000) 2022-09-08T05:25:17.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:16.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:16.923802+0000) 2022-09-08T05:25:18.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:25:17.973+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:25:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:17.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:17.923906+0000) 2022-09-08T05:25:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:17.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:17.923945+0000) 2022-09-08T05:25:19.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:18.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:18.924169+0000) 2022-09-08T05:25:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:18.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:18.924168+0000) 2022-09-08T05:25:19.477 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:19 smithi055 ceph-mon[97033]: pgmap v15125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.8 KiB/s rd, 3.5 KiB/s wr, 7 op/s 2022-09-08T05:25:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:19 smithi059 ceph-mon[97450]: pgmap v15125: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.8 KiB/s rd, 3.5 KiB/s wr, 7 op/s 2022-09-08T05:25:20.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:19.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:19.924442+0000) 2022-09-08T05:25:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:19.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:19.924409+0000) 2022-09-08T05:25:21.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:20.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:20.924721+0000) 2022-09-08T05:25:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:20.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:20.924672+0000) 2022-09-08T05:25:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:21 smithi055 ceph-mon[97033]: pgmap v15126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2022-09-08T05:25:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:21 smithi059 ceph-mon[97450]: pgmap v15126: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2022-09-08T05:25:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:21.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:21.924896+0000) 2022-09-08T05:25:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:21.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:21.924921+0000) 2022-09-08T05:25:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:22.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:22.925092+0000) 2022-09-08T05:25:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:22.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:22.925094+0000) 2022-09-08T05:25:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:23 smithi055 ceph-mon[97033]: pgmap v15127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.3 KiB/s rd, 2.6 KiB/s wr, 5 op/s 2022-09-08T05:25:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:23 smithi059 ceph-mon[97450]: pgmap v15127: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.3 KiB/s rd, 2.6 KiB/s wr, 5 op/s 2022-09-08T05:25:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:23.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:23.925322+0000) 2022-09-08T05:25:24.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:23.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:23.925323+0000) 2022-09-08T05:25:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:24.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:24.925470+0000) 2022-09-08T05:25:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:24.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:24.925505+0000) 2022-09-08T05:25:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:25 smithi055 ceph-mon[97033]: pgmap v15128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.0 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2022-09-08T05:25:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:25 smithi059 ceph-mon[97450]: pgmap v15128: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.0 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2022-09-08T05:25:26.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:25.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:25.925732+0000) 2022-09-08T05:25:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:25.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:25.925744+0000) 2022-09-08T05:25:26.977 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:26 smithi055 ceph-mon[97033]: pgmap v15129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.2 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2022-09-08T05:25:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:26 smithi059 ceph-mon[97450]: pgmap v15129: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 6.2 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2022-09-08T05:25:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:26.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:26.925996+0000) 2022-09-08T05:25:27.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:26.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:26.925959+0000) 2022-09-08T05:25:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:27.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:27.926233+0000) 2022-09-08T05:25:28.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:27.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:27.926225+0000) 2022-09-08T05:25:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:28.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:28.926483+0000) 2022-09-08T05:25:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:28.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:28.926498+0000) 2022-09-08T05:25:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:29 smithi055 ceph-mon[97033]: pgmap v15130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:29 smithi059 ceph-mon[97450]: pgmap v15130: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:29.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:29.926712+0000) 2022-09-08T05:25:30.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:29.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:29.926718+0000) 2022-09-08T05:25:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:30.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:30.926876+0000) 2022-09-08T05:25:31.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:30.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:30.926838+0000) 2022-09-08T05:25:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:31 smithi055 ceph-mon[97033]: pgmap v15131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:25:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:31 smithi059 ceph-mon[97450]: pgmap v15131: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:25:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:31.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:31.927091+0000) 2022-09-08T05:25:32.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:31.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:31.927093+0000) 2022-09-08T05:25:33.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:25:32.984+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:25:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:32.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:32.927325+0000) 2022-09-08T05:25:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:32.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:32.927327+0000) 2022-09-08T05:25:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:33 smithi055 ceph-mon[97033]: pgmap v15132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:25:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:33 smithi059 ceph-mon[97450]: pgmap v15132: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:25:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:33.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:33.927586+0000) 2022-09-08T05:25:34.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:33.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:33.927594+0000) 2022-09-08T05:25:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:34.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:34.927743+0000) 2022-09-08T05:25:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:34.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:34.927753+0000) 2022-09-08T05:25:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:35 smithi055 ceph-mon[97033]: pgmap v15133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:25:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:35 smithi059 ceph-mon[97450]: pgmap v15133: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:25:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:35.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:35.927954+0000) 2022-09-08T05:25:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:35.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:35.927986+0000) 2022-09-08T05:25:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:36 smithi055 ceph-mon[97033]: pgmap v15134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:25:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:36 smithi059 ceph-mon[97450]: pgmap v15134: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:25:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:36.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:36.928250+0000) 2022-09-08T05:25:37.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:36.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:36.928230+0000) 2022-09-08T05:25:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:37.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:37.928445+0000) 2022-09-08T05:25:38.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:37.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:37.928420+0000) 2022-09-08T05:25:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:38.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:38.928720+0000) 2022-09-08T05:25:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:38.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:38.928685+0000) 2022-09-08T05:25:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:39 smithi055 ceph-mon[97033]: pgmap v15135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:25:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:39 smithi059 ceph-mon[97450]: pgmap v15135: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:25:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:39.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:39.928983+0000) 2022-09-08T05:25:40.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:39.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:39.928946+0000) 2022-09-08T05:25:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:40.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:40.929218+0000) 2022-09-08T05:25:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:40.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:40.929160+0000) 2022-09-08T05:25:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:41 smithi055 ceph-mon[97033]: pgmap v15136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:25:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:41 smithi059 ceph-mon[97450]: pgmap v15136: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:25:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:41.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:41.929420+0000) 2022-09-08T05:25:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:41.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:41.929367+0000) 2022-09-08T05:25:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:42.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:42.929659+0000) 2022-09-08T05:25:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:42.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:42.929606+0000) 2022-09-08T05:25:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:43 smithi055 ceph-mon[97033]: pgmap v15137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:43 smithi059 ceph-mon[97450]: pgmap v15137: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:43.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:43.929863+0000) 2022-09-08T05:25:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:43.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:43.929853+0000) 2022-09-08T05:25:45.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:44.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:44.930074+0000) 2022-09-08T05:25:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:44.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:44.930075+0000) 2022-09-08T05:25:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:45 smithi055 ceph-mon[97033]: pgmap v15138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:25:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:25:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:25:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:45 smithi059 ceph-mon[97450]: pgmap v15138: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:25:45.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:25:45.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:25:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:45.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:45.930314+0000) 2022-09-08T05:25:46.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:45.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:45.930314+0000) 2022-09-08T05:25:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:25:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:46 smithi055 ceph-mon[97033]: pgmap v15139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:25:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:25:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:46 smithi059 ceph-mon[97450]: pgmap v15139: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:25:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:46.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:46.930530+0000) 2022-09-08T05:25:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:46.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:46.930530+0000) 2022-09-08T05:25:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:47.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:47.930790+0000) 2022-09-08T05:25:48.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:25:47.996+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:25:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:47.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:47.930769+0000) 2022-09-08T05:25:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:48.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:48.931025+0000) 2022-09-08T05:25:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:48.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:48.931023+0000) 2022-09-08T05:25:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:49 smithi055 ceph-mon[97033]: pgmap v15140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:49 smithi059 ceph-mon[97450]: pgmap v15140: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:49.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:49.931310+0000) 2022-09-08T05:25:50.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:49.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:49.931319+0000) 2022-09-08T05:25:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:50.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:50.931508+0000) 2022-09-08T05:25:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:50.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:50.931503+0000) 2022-09-08T05:25:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:51 smithi055 ceph-mon[97033]: pgmap v15141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:25:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:51 smithi059 ceph-mon[97450]: pgmap v15141: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:25:52.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:51.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:51.931738+0000) 2022-09-08T05:25:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:51.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:51.931740+0000) 2022-09-08T05:25:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:52.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:52.931947+0000) 2022-09-08T05:25:53.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:52.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:52.931969+0000) 2022-09-08T05:25:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:53 smithi055 ceph-mon[97033]: pgmap v15142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:53 smithi059 ceph-mon[97450]: pgmap v15142: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:53.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:53.932151+0000) 2022-09-08T05:25:54.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:53.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:53.932155+0000) 2022-09-08T05:25:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:54.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:54.932345+0000) 2022-09-08T05:25:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:54.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:54.932342+0000) 2022-09-08T05:25:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:55 smithi055 ceph-mon[97033]: pgmap v15143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:55 smithi059 ceph-mon[97450]: pgmap v15143: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:56.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:55.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:55.932580+0000) 2022-09-08T05:25:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:55.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:55.932584+0000) 2022-09-08T05:25:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:56 smithi055 ceph-mon[97033]: pgmap v15144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:25:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:56 smithi059 ceph-mon[97450]: pgmap v15144: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:25:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:56.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:56.932788+0000) 2022-09-08T05:25:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:56.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:56.932904+0000) 2022-09-08T05:25:58.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:57.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:57.933152+0000) 2022-09-08T05:25:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:57.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:57.933011+0000) 2022-09-08T05:25:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:58.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:58.933214+0000) 2022-09-08T05:25:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:58.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:58.933294+0000) 2022-09-08T05:25:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:25:59 smithi055 ceph-mon[97033]: pgmap v15145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:25:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:25:59 smithi059 ceph-mon[97450]: pgmap v15145: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:25:59.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:59.933419+0000) 2022-09-08T05:26:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:25:59.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:25:59.933526+0000) 2022-09-08T05:26:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:00.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:00.933581+0000) 2022-09-08T05:26:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:00.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:00.933705+0000) 2022-09-08T05:26:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:01 smithi055 ceph-mon[97033]: pgmap v15146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:26:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:01 smithi059 ceph-mon[97450]: pgmap v15146: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:26:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:01.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:01.933780+0000) 2022-09-08T05:26:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:01.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:01.933913+0000) 2022-09-08T05:26:03.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:26:03.000+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:26:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:02.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:02.933990+0000) 2022-09-08T05:26:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:02.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:02.934140+0000) 2022-09-08T05:26:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:03 smithi055 ceph-mon[97033]: pgmap v15147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:03 smithi059 ceph-mon[97450]: pgmap v15147: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:03.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:03.934209+0000) 2022-09-08T05:26:04.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:03.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:03.934327+0000) 2022-09-08T05:26:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:04.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:04.934439+0000) 2022-09-08T05:26:05.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:04.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:04.934556+0000) 2022-09-08T05:26:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:05 smithi055 ceph-mon[97033]: pgmap v15148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:05 smithi059 ceph-mon[97450]: pgmap v15148: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:06.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:05.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:05.934767+0000) 2022-09-08T05:26:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:05.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:05.934720+0000) 2022-09-08T05:26:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:26:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:06 smithi055 ceph-mon[97033]: pgmap v15149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:26:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:26:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:26:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:06 smithi059 ceph-mon[97450]: pgmap v15149: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:26:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:26:07.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:06.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:06.934935+0000) 2022-09-08T05:26:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:06.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:06.934859+0000) 2022-09-08T05:26:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:07.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:07.935033+0000) 2022-09-08T05:26:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:07.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:07.935078+0000) 2022-09-08T05:26:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:08.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:08.935195+0000) 2022-09-08T05:26:09.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:08.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:08.935290+0000) 2022-09-08T05:26:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:09 smithi055 ceph-mon[97033]: pgmap v15150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:26:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:09 smithi059 ceph-mon[97450]: pgmap v15150: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:26:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:09.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:09.935422+0000) 2022-09-08T05:26:10.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:09.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:09.935556+0000) 2022-09-08T05:26:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:10 smithi055 ceph-mon[97033]: osdmap e1027: 12 total, 11 up, 11 in 2022-09-08T05:26:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:10 smithi059 ceph-mon[97450]: osdmap e1027: 12 total, 11 up, 11 in 2022-09-08T05:26:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:10.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:10.935600+0000) 2022-09-08T05:26:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:10.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:10.935717+0000) 2022-09-08T05:26:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:11 smithi055 ceph-mon[97033]: pgmap v15152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:26:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:11 smithi055 ceph-mon[97033]: osdmap e1028: 12 total, 11 up, 11 in 2022-09-08T05:26:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:11 smithi059 ceph-mon[97450]: pgmap v15152: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:26:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:11 smithi059 ceph-mon[97450]: osdmap e1028: 12 total, 11 up, 11 in 2022-09-08T05:26:12.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:11.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:11.935902+0000) 2022-09-08T05:26:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:11.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:11.935817+0000) 2022-09-08T05:26:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:12.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:12.936045+0000) 2022-09-08T05:26:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:12.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:12.936154+0000) 2022-09-08T05:26:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:13 smithi055 ceph-mon[97033]: pgmap v15154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:26:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:13 smithi059 ceph-mon[97450]: pgmap v15154: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:26:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:13.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:13.936231+0000) 2022-09-08T05:26:14.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:13.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:13.936333+0000) 2022-09-08T05:26:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:14.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:14.936395+0000) 2022-09-08T05:26:15.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:14.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:14.936492+0000) 2022-09-08T05:26:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:15 smithi055 ceph-mon[97033]: pgmap v15155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:26:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:15 smithi059 ceph-mon[97450]: pgmap v15155: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:26:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:15.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:15.936561+0000) 2022-09-08T05:26:16.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:15.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:15.936669+0000) 2022-09-08T05:26:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:16 smithi055 ceph-mon[97033]: pgmap v15156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:26:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:16 smithi059 ceph-mon[97450]: pgmap v15156: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:26:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:16.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:16.936783+0000) 2022-09-08T05:26:17.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:16.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:16.936845+0000) 2022-09-08T05:26:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:17.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:17.936924+0000) 2022-09-08T05:26:18.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:26:18.003+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:26:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:17.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:17.936982+0000) 2022-09-08T05:26:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:18.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:18.937119+0000) 2022-09-08T05:26:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:18.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:18.937128+0000) 2022-09-08T05:26:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:19 smithi055 ceph-mon[97033]: pgmap v15157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T05:26:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:19 smithi059 ceph-mon[97450]: pgmap v15157: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 228 B/s wr, 7 op/s 2022-09-08T05:26:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:19.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:19.937346+0000) 2022-09-08T05:26:20.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:19.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:19.937350+0000) 2022-09-08T05:26:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:20.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:20.937575+0000) 2022-09-08T05:26:21.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:20.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:20.937491+0000) 2022-09-08T05:26:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:21 smithi055 ceph-mon[97033]: pgmap v15158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:26:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:21 smithi059 ceph-mon[97450]: pgmap v15158: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:26:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:21.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:21.937786+0000) 2022-09-08T05:26:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:21.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:21.937739+0000) 2022-09-08T05:26:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:22.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:22.938032+0000) 2022-09-08T05:26:23.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:22.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:22.938025+0000) 2022-09-08T05:26:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:23 smithi055 ceph-mon[97033]: pgmap v15159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T05:26:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:23 smithi059 ceph-mon[97450]: pgmap v15159: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T05:26:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:23.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:23.938244+0000) 2022-09-08T05:26:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:23.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:23.938231+0000) 2022-09-08T05:26:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:24.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:24.938471+0000) 2022-09-08T05:26:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:24.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:24.938458+0000) 2022-09-08T05:26:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:25 smithi055 ceph-mon[97033]: pgmap v15160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:26:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:25 smithi059 ceph-mon[97450]: pgmap v15160: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:26:26.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:25.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:25.938617+0000) 2022-09-08T05:26:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:25.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:25.938617+0000) 2022-09-08T05:26:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:26 smithi055 ceph-mon[97033]: pgmap v15161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:26:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:26 smithi059 ceph-mon[97450]: pgmap v15161: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:26:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:26.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:26.938782+0000) 2022-09-08T05:26:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:26.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:26.938774+0000) 2022-09-08T05:26:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:27.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:27.938988+0000) 2022-09-08T05:26:28.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:27.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:27.938988+0000) 2022-09-08T05:26:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:28.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:28.939270+0000) 2022-09-08T05:26:29.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:28.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:28.939251+0000) 2022-09-08T05:26:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:29 smithi055 ceph-mon[97033]: pgmap v15162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:29 smithi059 ceph-mon[97450]: pgmap v15162: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:30.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:29.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:29.939518+0000) 2022-09-08T05:26:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:29.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:29.939498+0000) 2022-09-08T05:26:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:30.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:30.939745+0000) 2022-09-08T05:26:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:30.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:30.939749+0000) 2022-09-08T05:26:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:31 smithi055 ceph-mon[97033]: pgmap v15163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:26:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:31 smithi059 ceph-mon[97450]: pgmap v15163: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:26:32.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:31.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:31.939869+0000) 2022-09-08T05:26:32.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:31.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:31.939916+0000) 2022-09-08T05:26:33.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:26:33.004+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:26:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:32.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:32.939992+0000) 2022-09-08T05:26:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:32.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:32.940048+0000) 2022-09-08T05:26:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:33 smithi055 ceph-mon[97033]: pgmap v15164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:33 smithi059 ceph-mon[97450]: pgmap v15164: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:33.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:33.940210+0000) 2022-09-08T05:26:34.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:33.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:33.940266+0000) 2022-09-08T05:26:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:34.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:34.940437+0000) 2022-09-08T05:26:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:34.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:34.940453+0000) 2022-09-08T05:26:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:35 smithi055 ceph-mon[97033]: pgmap v15165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:35 smithi059 ceph-mon[97450]: pgmap v15165: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:35.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:35.940663+0000) 2022-09-08T05:26:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:35.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:35.940677+0000) 2022-09-08T05:26:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:36 smithi055 ceph-mon[97033]: pgmap v15166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:26:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:36 smithi059 ceph-mon[97450]: pgmap v15166: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:26:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:36.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:36.940881+0000) 2022-09-08T05:26:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:36.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:36.940888+0000) 2022-09-08T05:26:38.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:37.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:37.941124+0000) 2022-09-08T05:26:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:37.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:37.941043+0000) 2022-09-08T05:26:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:38.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:38.941236+0000) 2022-09-08T05:26:39.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:38.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:38.941322+0000) 2022-09-08T05:26:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:39 smithi055 ceph-mon[97033]: pgmap v15167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:39 smithi059 ceph-mon[97450]: pgmap v15167: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:39.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:39.941402+0000) 2022-09-08T05:26:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:39.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:39.941580+0000) 2022-09-08T05:26:41.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:40.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:40.941776+0000) 2022-09-08T05:26:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:40.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:40.941530+0000) 2022-09-08T05:26:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:41 smithi055 ceph-mon[97033]: pgmap v15168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:26:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:41 smithi059 ceph-mon[97450]: pgmap v15168: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:26:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:41.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:41.941661+0000) 2022-09-08T05:26:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:41.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:41.941912+0000) 2022-09-08T05:26:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:42.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:42.941782+0000) 2022-09-08T05:26:43.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:42.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:42.942031+0000) 2022-09-08T05:26:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:43 smithi055 ceph-mon[97033]: pgmap v15169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:43 smithi059 ceph-mon[97450]: pgmap v15169: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:43.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:43.941962+0000) 2022-09-08T05:26:44.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:43.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:43.942170+0000) 2022-09-08T05:26:45.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:44.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:44.942321+0000) 2022-09-08T05:26:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:44.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:44.942168+0000) 2022-09-08T05:26:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:45 smithi055 ceph-mon[97033]: pgmap v15170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:45 smithi059 ceph-mon[97450]: pgmap v15170: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:46.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:45.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:45.942326+0000) 2022-09-08T05:26:46.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:45.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:45.942452+0000) 2022-09-08T05:26:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:26:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:26:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:26:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:46 smithi055 ceph-mon[97033]: pgmap v15171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:26:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:26:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:26:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:26:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:46 smithi059 ceph-mon[97450]: pgmap v15171: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:26:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:46.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:46.942446+0000) 2022-09-08T05:26:47.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:46.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:46.942533+0000) 2022-09-08T05:26:48.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:26:48.005+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:26:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:47.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:47.942716+0000) 2022-09-08T05:26:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:47.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:47.942718+0000) 2022-09-08T05:26:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:48.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:48.942912+0000) 2022-09-08T05:26:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:48.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:48.942897+0000) 2022-09-08T05:26:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:49 smithi055 ceph-mon[97033]: pgmap v15172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:49 smithi059 ceph-mon[97450]: pgmap v15172: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:49.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:49.943067+0000) 2022-09-08T05:26:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:49.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:49.943034+0000) 2022-09-08T05:26:51.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:50.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:50.943156+0000) 2022-09-08T05:26:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:50.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:50.943213+0000) 2022-09-08T05:26:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:51 smithi055 ceph-mon[97033]: pgmap v15173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:26:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:51 smithi059 ceph-mon[97450]: pgmap v15173: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:26:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:51.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:51.943343+0000) 2022-09-08T05:26:52.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:51.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:51.943281+0000) 2022-09-08T05:26:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:52.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:52.943566+0000) 2022-09-08T05:26:53.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:52.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:52.943529+0000) 2022-09-08T05:26:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:53 smithi055 ceph-mon[97033]: pgmap v15174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:53 smithi059 ceph-mon[97450]: pgmap v15174: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:53.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:53.943755+0000) 2022-09-08T05:26:54.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:53.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:53.943766+0000) 2022-09-08T05:26:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:54.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:54.943912+0000) 2022-09-08T05:26:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:54.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:54.943903+0000) 2022-09-08T05:26:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:55 smithi055 ceph-mon[97033]: pgmap v15175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:55 smithi059 ceph-mon[97450]: pgmap v15175: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:56.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:55.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:55.944083+0000) 2022-09-08T05:26:56.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:55.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:55.944067+0000) 2022-09-08T05:26:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:56 smithi055 ceph-mon[97033]: pgmap v15176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:26:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:56 smithi059 ceph-mon[97450]: pgmap v15176: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:26:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:56.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:56.944238+0000) 2022-09-08T05:26:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:56.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:56.944237+0000) 2022-09-08T05:26:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:57.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:57.944490+0000) 2022-09-08T05:26:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:57.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:57.944458+0000) 2022-09-08T05:26:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:58.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:58.944775+0000) 2022-09-08T05:26:59.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:58.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:58.944763+0000) 2022-09-08T05:26:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:26:59 smithi055 ceph-mon[97033]: pgmap v15177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:26:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:26:59 smithi059 ceph-mon[97450]: pgmap v15177: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:26:59.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:59.944967+0000) 2022-09-08T05:27:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:26:59.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:26:59.944969+0000) 2022-09-08T05:27:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:00.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:00.945128+0000) 2022-09-08T05:27:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:00.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:00.945127+0000) 2022-09-08T05:27:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:01 smithi055 ceph-mon[97033]: pgmap v15178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:27:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:01 smithi059 ceph-mon[97450]: pgmap v15178: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:27:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:01.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:01.945332+0000) 2022-09-08T05:27:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:01.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:01.945343+0000) 2022-09-08T05:27:03.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:27:03.015+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:27:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:02.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:02.945513+0000) 2022-09-08T05:27:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:02.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:02.945531+0000) 2022-09-08T05:27:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:03 smithi055 ceph-mon[97033]: pgmap v15179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:03 smithi059 ceph-mon[97450]: pgmap v15179: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:03.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:03.945697+0000) 2022-09-08T05:27:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:03.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:03.945731+0000) 2022-09-08T05:27:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:04.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:04.945842+0000) 2022-09-08T05:27:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:04.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:04.945907+0000) 2022-09-08T05:27:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:05 smithi055 ceph-mon[97033]: pgmap v15180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:05 smithi059 ceph-mon[97450]: pgmap v15180: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:06.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:05.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:05.946146+0000) 2022-09-08T05:27:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:05.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:05.946049+0000) 2022-09-08T05:27:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:27:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:06 smithi055 ceph-mon[97033]: pgmap v15181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:27:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:27:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:27:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:06 smithi059 ceph-mon[97450]: pgmap v15181: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:27:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:27:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:06.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:06.946224+0000) 2022-09-08T05:27:07.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:06.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:06.946329+0000) 2022-09-08T05:27:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:07.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:07.946385+0000) 2022-09-08T05:27:08.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:07.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:07.946561+0000) 2022-09-08T05:27:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:08.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:08.946581+0000) 2022-09-08T05:27:09.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:08.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:08.946743+0000) 2022-09-08T05:27:09.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:09 smithi055 ceph-mon[97033]: pgmap v15182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:27:09.530 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:09 smithi059 ceph-mon[97450]: pgmap v15182: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:27:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:09.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:09.946761+0000) 2022-09-08T05:27:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:09.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:09.946905+0000) 2022-09-08T05:27:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:10 smithi055 ceph-mon[97033]: osdmap e1029: 12 total, 11 up, 11 in 2022-09-08T05:27:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:10 smithi059 ceph-mon[97450]: osdmap e1029: 12 total, 11 up, 11 in 2022-09-08T05:27:11.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:10.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:10.946885+0000) 2022-09-08T05:27:11.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:10.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:10.947038+0000) 2022-09-08T05:27:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:11 smithi055 ceph-mon[97033]: pgmap v15184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:27:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:11 smithi055 ceph-mon[97033]: osdmap e1030: 12 total, 11 up, 11 in 2022-09-08T05:27:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:11 smithi059 ceph-mon[97450]: pgmap v15184: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:27:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:11 smithi059 ceph-mon[97450]: osdmap e1030: 12 total, 11 up, 11 in 2022-09-08T05:27:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:11.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:11.947039+0000) 2022-09-08T05:27:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:11.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:11.947172+0000) 2022-09-08T05:27:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:12.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:12.947239+0000) 2022-09-08T05:27:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:12.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:12.947358+0000) 2022-09-08T05:27:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:13 smithi055 ceph-mon[97033]: pgmap v15186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T05:27:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:13 smithi059 ceph-mon[97450]: pgmap v15186: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T05:27:14.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:13.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:13.947537+0000) 2022-09-08T05:27:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:13.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:13.947506+0000) 2022-09-08T05:27:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:14.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:14.947729+0000) 2022-09-08T05:27:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:14.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:14.947741+0000) 2022-09-08T05:27:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:15 smithi055 ceph-mon[97033]: pgmap v15187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:27:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:15 smithi059 ceph-mon[97450]: pgmap v15187: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:27:16.151 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:15.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:15.947957+0000) 2022-09-08T05:27:16.152 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:15.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:15.947932+0000) 2022-09-08T05:27:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:16 smithi055 ceph-mon[97033]: pgmap v15188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 263 B/s wr, 8 op/s 2022-09-08T05:27:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:27:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:16 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T05:27:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:16 smithi059 ceph-mon[97450]: pgmap v15188: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 263 B/s wr, 8 op/s 2022-09-08T05:27:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:27:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:16 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T05:27:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:16.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:16.948073+0000) 2022-09-08T05:27:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:16.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:16.948112+0000) 2022-09-08T05:27:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:17.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:17.948198+0000) 2022-09-08T05:27:18.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:17.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:17.948287+0000) 2022-09-08T05:27:18.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:27:18.016+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:27:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:18 smithi055 ceph-mon[97033]: pgmap v15189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-08T05:27:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:18 smithi059 ceph-mon[97450]: pgmap v15189: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-08T05:27:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:18.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:18.948352+0000) 2022-09-08T05:27:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:18.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:18.948433+0000) 2022-09-08T05:27:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:19.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:19.948525+0000) 2022-09-08T05:27:20.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:19.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:19.948700+0000) 2022-09-08T05:27:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:20 smithi055 ceph-mon[97033]: pgmap v15190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 104 B/s wr, 6 op/s 2022-09-08T05:27:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:20 smithi059 ceph-mon[97450]: pgmap v15190: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 104 B/s wr, 6 op/s 2022-09-08T05:27:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:20.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:20.948734+0000) 2022-09-08T05:27:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:20.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:20.948943+0000) 2022-09-08T05:27:22.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:21.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:21.949082+0000) 2022-09-08T05:27:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:21.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:21.948870+0000) 2022-09-08T05:27:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:22 smithi055 ceph-mon[97033]: pgmap v15191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T05:27:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:22 smithi059 ceph-mon[97450]: pgmap v15191: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T05:27:23.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:22.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:22.949217+0000) 2022-09-08T05:27:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:22.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:22.949012+0000) 2022-09-08T05:27:24.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:23.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:23.949366+0000) 2022-09-08T05:27:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:23.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:23.949166+0000) 2022-09-08T05:27:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:24 smithi055 ceph-mon[97033]: pgmap v15192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T05:27:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:24 smithi059 ceph-mon[97450]: pgmap v15192: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T05:27:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:24.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:24.949388+0000) 2022-09-08T05:27:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:24.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:24.949566+0000) 2022-09-08T05:27:26.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:25.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:25.949705+0000) 2022-09-08T05:27:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:25.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:25.949607+0000) 2022-09-08T05:27:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:26 smithi055 ceph-mon[97033]: pgmap v15193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T05:27:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:26 smithi059 ceph-mon[97450]: pgmap v15193: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 86 B/s wr, 7 op/s 2022-09-08T05:27:27.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:26.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:26.949911+0000) 2022-09-08T05:27:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:26.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:26.949803+0000) 2022-09-08T05:27:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:27.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:27.950073+0000) 2022-09-08T05:27:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:27.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:27.950156+0000) 2022-09-08T05:27:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:28 smithi055 ceph-mon[97033]: pgmap v15194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:28 smithi059 ceph-mon[97450]: pgmap v15194: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:28.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:28.950274+0000) 2022-09-08T05:27:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:28.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:28.950333+0000) 2022-09-08T05:27:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:29.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:29.950446+0000) 2022-09-08T05:27:30.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:29.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:29.950488+0000) 2022-09-08T05:27:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:30 smithi055 ceph-mon[97033]: pgmap v15195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:27:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:30 smithi059 ceph-mon[97450]: pgmap v15195: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:27:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:30.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:30.950620+0000) 2022-09-08T05:27:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:30.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:30.950620+0000) 2022-09-08T05:27:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:31.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:31.950838+0000) 2022-09-08T05:27:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:31.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:31.950820+0000) 2022-09-08T05:27:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:32 smithi055 ceph-mon[97033]: pgmap v15196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:32 smithi059 ceph-mon[97450]: pgmap v15196: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:33.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:32.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:32.951041+0000) 2022-09-08T05:27:33.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:27:33.020+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:27:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:32.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:32.951051+0000) 2022-09-08T05:27:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:33.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:33.951287+0000) 2022-09-08T05:27:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:33.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:33.951287+0000) 2022-09-08T05:27:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:34 smithi055 ceph-mon[97033]: pgmap v15197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:34 smithi059 ceph-mon[97450]: pgmap v15197: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:34.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:34.951540+0000) 2022-09-08T05:27:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:34.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:34.951540+0000) 2022-09-08T05:27:36.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:35.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:35.951820+0000) 2022-09-08T05:27:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:35.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:35.951817+0000) 2022-09-08T05:27:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:36 smithi055 ceph-mon[97033]: pgmap v15198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:27:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:36 smithi059 ceph-mon[97450]: pgmap v15198: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:27:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:36.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:36.952106+0000) 2022-09-08T05:27:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:36.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:36.952106+0000) 2022-09-08T05:27:38.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:37.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:37.952397+0000) 2022-09-08T05:27:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:37.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:37.952400+0000) 2022-09-08T05:27:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:38 smithi055 ceph-mon[97033]: pgmap v15199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:38 smithi059 ceph-mon[97450]: pgmap v15199: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:39.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:38.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:38.952694+0000) 2022-09-08T05:27:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:38.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:38.952696+0000) 2022-09-08T05:27:40.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:39.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:39.952922+0000) 2022-09-08T05:27:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:39.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:39.952899+0000) 2022-09-08T05:27:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:40 smithi055 ceph-mon[97033]: pgmap v15200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:27:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:40 smithi059 ceph-mon[97450]: pgmap v15200: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:27:41.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:40.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:40.953134+0000) 2022-09-08T05:27:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:40.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:40.953134+0000) 2022-09-08T05:27:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:41.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:41.953293+0000) 2022-09-08T05:27:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:41.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:41.953293+0000) 2022-09-08T05:27:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:42 smithi055 ceph-mon[97033]: pgmap v15201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:42 smithi059 ceph-mon[97450]: pgmap v15201: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:43.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:42.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:42.953532+0000) 2022-09-08T05:27:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:42.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:42.953532+0000) 2022-09-08T05:27:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:43.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:43.953777+0000) 2022-09-08T05:27:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:43.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:43.953782+0000) 2022-09-08T05:27:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:44 smithi055 ceph-mon[97033]: pgmap v15202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:44 smithi059 ceph-mon[97450]: pgmap v15202: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:44.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:44.954050+0000) 2022-09-08T05:27:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:44.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:44.954050+0000) 2022-09-08T05:27:46.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:45.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:45.954278+0000) 2022-09-08T05:27:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:45.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:45.954278+0000) 2022-09-08T05:27:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:46 smithi055 ceph-mon[97033]: pgmap v15203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:27:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:46 smithi059 ceph-mon[97450]: pgmap v15203: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:27:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:46.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:46.954521+0000) 2022-09-08T05:27:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:46.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:46.954520+0000) 2022-09-08T05:27:48.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:27:48.031+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:27:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:47.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:47.954796+0000) 2022-09-08T05:27:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:47.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:47.954798+0000) 2022-09-08T05:27:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:48 smithi055 ceph-mon[97033]: pgmap v15204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:48 smithi059 ceph-mon[97450]: pgmap v15204: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:49.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:48.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:48.955033+0000) 2022-09-08T05:27:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:48.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:48.955035+0000) 2022-09-08T05:27:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:49.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:49.955282+0000) 2022-09-08T05:27:50.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:49.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:49.955285+0000) 2022-09-08T05:27:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:50 smithi055 ceph-mon[97033]: pgmap v15205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:27:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:50 smithi059 ceph-mon[97450]: pgmap v15205: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:27:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:50.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:50.955477+0000) 2022-09-08T05:27:51.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:50.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:50.955477+0000) 2022-09-08T05:27:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:51.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:51.955772+0000) 2022-09-08T05:27:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:51.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:51.955772+0000) 2022-09-08T05:27:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:52 smithi055 ceph-mon[97033]: pgmap v15206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:52 smithi059 ceph-mon[97450]: pgmap v15206: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:52.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:52.956034+0000) 2022-09-08T05:27:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:52.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:52.956036+0000) 2022-09-08T05:27:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:53.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:53.956244+0000) 2022-09-08T05:27:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:53.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:53.956263+0000) 2022-09-08T05:27:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:54 smithi055 ceph-mon[97033]: pgmap v15207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:54 smithi059 ceph-mon[97450]: pgmap v15207: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:54.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:54.956448+0000) 2022-09-08T05:27:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:54.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:54.956463+0000) 2022-09-08T05:27:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:55.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:55.956615+0000) 2022-09-08T05:27:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:55.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:55.956730+0000) 2022-09-08T05:27:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:56 smithi055 ceph-mon[97033]: pgmap v15208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:27:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:56 smithi059 ceph-mon[97450]: pgmap v15208: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:27:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:56.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:56.956897+0000) 2022-09-08T05:27:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:56.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:56.956945+0000) 2022-09-08T05:27:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:57.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:57.957112+0000) 2022-09-08T05:27:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:57.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:57.957223+0000) 2022-09-08T05:27:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:27:58 smithi055 ceph-mon[97033]: pgmap v15209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:27:58 smithi059 ceph-mon[97450]: pgmap v15209: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:27:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:58.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:58.957336+0000) 2022-09-08T05:27:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:58.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:58.957424+0000) 2022-09-08T05:28:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:27:59.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:59.957605+0000) 2022-09-08T05:28:00.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:27:59.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:27:59.957698+0000) 2022-09-08T05:28:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:00 smithi055 ceph-mon[97033]: pgmap v15210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:28:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:00 smithi059 ceph-mon[97450]: pgmap v15210: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:28:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:00.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:00.957881+0000) 2022-09-08T05:28:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:00.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:00.957927+0000) 2022-09-08T05:28:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:01.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:01.958158+0000) 2022-09-08T05:28:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:01.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:01.958148+0000) 2022-09-08T05:28:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:02 smithi055 ceph-mon[97033]: pgmap v15211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:02 smithi059 ceph-mon[97450]: pgmap v15211: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:03.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:28:03.032+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:28:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:02.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:02.958433+0000) 2022-09-08T05:28:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:02.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:02.958365+0000) 2022-09-08T05:28:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:03.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:03.958780+0000) 2022-09-08T05:28:04.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:03.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:03.958630+0000) 2022-09-08T05:28:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:04 smithi055 ceph-mon[97033]: pgmap v15212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:04 smithi059 ceph-mon[97450]: pgmap v15212: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:05.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:04.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:04.958842+0000) 2022-09-08T05:28:05.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:04.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:04.958970+0000) 2022-09-08T05:28:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:28:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:28:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:05.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:05.959139+0000) 2022-09-08T05:28:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:05.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:05.959041+0000) 2022-09-08T05:28:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:06 smithi055 ceph-mon[97033]: pgmap v15213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:28:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:28:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:06 smithi059 ceph-mon[97450]: pgmap v15213: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:28:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:28:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:06.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:06.959288+0000) 2022-09-08T05:28:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:06.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:06.959190+0000) 2022-09-08T05:28:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:07.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:07.959466+0000) 2022-09-08T05:28:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:07.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:07.959421+0000) 2022-09-08T05:28:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:08 smithi055 ceph-mon[97033]: pgmap v15214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:28:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:08 smithi059 ceph-mon[97450]: pgmap v15214: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:28:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:08.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:08.959699+0000) 2022-09-08T05:28:09.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:08.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:08.959686+0000) 2022-09-08T05:28:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:09 smithi055 ceph-mon[97033]: osdmap e1031: 12 total, 11 up, 11 in 2022-09-08T05:28:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:09 smithi059 ceph-mon[97450]: osdmap e1031: 12 total, 11 up, 11 in 2022-09-08T05:28:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:09.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:09.959969+0000) 2022-09-08T05:28:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:09.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:09.959980+0000) 2022-09-08T05:28:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:10 smithi055 ceph-mon[97033]: pgmap v15216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:28:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:10 smithi059 ceph-mon[97450]: pgmap v15216: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:28:11.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:10.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:10.960151+0000) 2022-09-08T05:28:11.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:10.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:10.960139+0000) 2022-09-08T05:28:12.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:11.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:11.960335+0000) 2022-09-08T05:28:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:11.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:11.960327+0000) 2022-09-08T05:28:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:12 smithi059 ceph-mon[97450]: pgmap v15217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:28:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:12.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:12.960504+0000) 2022-09-08T05:28:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:12.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:12.960526+0000) 2022-09-08T05:28:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:12 smithi055 ceph-mon[97033]: pgmap v15217: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:28:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:13.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:13.960748+0000) 2022-09-08T05:28:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:13.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:13.960774+0000) 2022-09-08T05:28:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:14 smithi059 ceph-mon[97450]: pgmap v15218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:28:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:14.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:14.960997+0000) 2022-09-08T05:28:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:14.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:14.961024+0000) 2022-09-08T05:28:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:14 smithi055 ceph-mon[97033]: pgmap v15218: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:28:16.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:15.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:15.961245+0000) 2022-09-08T05:28:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:15.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:15.961235+0000) 2022-09-08T05:28:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:16 smithi055 ceph-mon[97033]: osdmap e1032: 12 total, 11 up, 11 in 2022-09-08T05:28:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:16 smithi055 ceph-mon[97033]: pgmap v15220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:28:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:28:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:28:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:28:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:16 smithi059 ceph-mon[97450]: osdmap e1032: 12 total, 11 up, 11 in 2022-09-08T05:28:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:16 smithi059 ceph-mon[97450]: pgmap v15220: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:28:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:28:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:28:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:28:17.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:16.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:16.961413+0000) 2022-09-08T05:28:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:16.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:16.961385+0000) 2022-09-08T05:28:18.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:17.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:17.961550+0000) 2022-09-08T05:28:18.406 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:28:18.041+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:28:18.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:17.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:17.961625+0000) 2022-09-08T05:28:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:18 smithi055 ceph-mon[97033]: pgmap v15221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T05:28:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:18 smithi059 ceph-mon[97450]: pgmap v15221: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T05:28:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:18.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:18.961811+0000) 2022-09-08T05:28:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:18.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:18.961842+0000) 2022-09-08T05:28:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:19.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:19.962057+0000) 2022-09-08T05:28:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:19.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:19.962054+0000) 2022-09-08T05:28:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:20 smithi055 ceph-mon[97033]: pgmap v15222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:28:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:20 smithi059 ceph-mon[97450]: pgmap v15222: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:28:21.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:20.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:20.962265+0000) 2022-09-08T05:28:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:20.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:20.962265+0000) 2022-09-08T05:28:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:21.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:21.962473+0000) 2022-09-08T05:28:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:21.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:21.962490+0000) 2022-09-08T05:28:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:22 smithi055 ceph-mon[97033]: pgmap v15223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:28:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:22 smithi059 ceph-mon[97450]: pgmap v15223: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:28:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:22.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:22.962693+0000) 2022-09-08T05:28:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:22.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:22.962694+0000) 2022-09-08T05:28:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:23.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:23.962879+0000) 2022-09-08T05:28:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:23.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:23.962879+0000) 2022-09-08T05:28:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:24 smithi055 ceph-mon[97033]: pgmap v15224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:28:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:24 smithi059 ceph-mon[97450]: pgmap v15224: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:28:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:24.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:24.963051+0000) 2022-09-08T05:28:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:24.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:24.963065+0000) 2022-09-08T05:28:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:25.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:25.963272+0000) 2022-09-08T05:28:26.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:25.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:25.963288+0000) 2022-09-08T05:28:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:26 smithi055 ceph-mon[97033]: pgmap v15225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:28:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:26 smithi059 ceph-mon[97450]: pgmap v15225: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:28:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:26.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:26.963500+0000) 2022-09-08T05:28:27.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:26.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:26.963496+0000) 2022-09-08T05:28:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:27.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:27.963716+0000) 2022-09-08T05:28:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:27.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:27.963729+0000) 2022-09-08T05:28:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:28 smithi055 ceph-mon[97033]: pgmap v15226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:28 smithi059 ceph-mon[97450]: pgmap v15226: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:28.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:28.963941+0000) 2022-09-08T05:28:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:28.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:28.963905+0000) 2022-09-08T05:28:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:29.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:29.964185+0000) 2022-09-08T05:28:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:29.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:29.964209+0000) 2022-09-08T05:28:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:30 smithi055 ceph-mon[97033]: pgmap v15227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:28:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:30 smithi059 ceph-mon[97450]: pgmap v15227: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:28:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:30.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:30.964457+0000) 2022-09-08T05:28:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:30.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:30.964458+0000) 2022-09-08T05:28:32.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:31.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:31.964684+0000) 2022-09-08T05:28:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:31.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:31.964684+0000) 2022-09-08T05:28:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:32 smithi055 ceph-mon[97033]: pgmap v15228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:32 smithi059 ceph-mon[97450]: pgmap v15228: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:33.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:28:33.050+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:28:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:32.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:32.964899+0000) 2022-09-08T05:28:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:32.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:32.964897+0000) 2022-09-08T05:28:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:33.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:33.965123+0000) 2022-09-08T05:28:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:33.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:33.965123+0000) 2022-09-08T05:28:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:34 smithi055 ceph-mon[97033]: pgmap v15229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:34 smithi059 ceph-mon[97450]: pgmap v15229: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:34.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:34.965308+0000) 2022-09-08T05:28:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:34.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:34.965308+0000) 2022-09-08T05:28:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:35.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:35.965559+0000) 2022-09-08T05:28:36.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:35.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:35.965562+0000) 2022-09-08T05:28:36.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:36 smithi055 ceph-mon[97033]: pgmap v15230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:28:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:36 smithi059 ceph-mon[97450]: pgmap v15230: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:28:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:36.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:36.965809+0000) 2022-09-08T05:28:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:36.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:36.965808+0000) 2022-09-08T05:28:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:37.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:37.966039+0000) 2022-09-08T05:28:38.415 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:37.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:37.966039+0000) 2022-09-08T05:28:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:38 smithi055 ceph-mon[97033]: pgmap v15231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:38 smithi059 ceph-mon[97450]: pgmap v15231: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:38.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:38.966182+0000) 2022-09-08T05:28:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:38.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:38.966175+0000) 2022-09-08T05:28:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:39.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:39.966444+0000) 2022-09-08T05:28:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:39.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:39.966417+0000) 2022-09-08T05:28:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:40 smithi055 ceph-mon[97033]: pgmap v15232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:28:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:40 smithi059 ceph-mon[97450]: pgmap v15232: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:28:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:40.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:40.966617+0000) 2022-09-08T05:28:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:40.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:40.966554+0000) 2022-09-08T05:28:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:41.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:41.966828+0000) 2022-09-08T05:28:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:41.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:41.966799+0000) 2022-09-08T05:28:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:42 smithi055 ceph-mon[97033]: pgmap v15233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:42 smithi059 ceph-mon[97450]: pgmap v15233: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:43.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:42.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:42.967051+0000) 2022-09-08T05:28:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:42.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:42.967091+0000) 2022-09-08T05:28:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:43.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:43.967305+0000) 2022-09-08T05:28:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:43.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:43.967283+0000) 2022-09-08T05:28:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:44 smithi055 ceph-mon[97033]: pgmap v15234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:44 smithi059 ceph-mon[97450]: pgmap v15234: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:44.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:44.967555+0000) 2022-09-08T05:28:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:44.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:44.967553+0000) 2022-09-08T05:28:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:45.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:45.967747+0000) 2022-09-08T05:28:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:45.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:45.967763+0000) 2022-09-08T05:28:47.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:46.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:46.967970+0000) 2022-09-08T05:28:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:46.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:46.967916+0000) 2022-09-08T05:28:47.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:47 smithi055 ceph-mon[97033]: pgmap v15235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:28:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:47 smithi055 ceph-mon[97033]: pgmap v15236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:28:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:47 smithi055 ceph-mon[97033]: pgmap v15237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:28:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:28:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:47 smithi059 ceph-mon[97450]: pgmap v15235: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:28:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:47 smithi059 ceph-mon[97450]: pgmap v15236: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:28:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:47 smithi059 ceph-mon[97450]: pgmap v15237: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:28:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:28:48.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:28:48.063+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:28:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:47.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:47.968241+0000) 2022-09-08T05:28:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:47.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:47.968136+0000) 2022-09-08T05:28:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:48.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:48.968346+0000) 2022-09-08T05:28:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:48.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:48.968543+0000) 2022-09-08T05:28:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:49 smithi055 ceph-mon[97033]: pgmap v15238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:28:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:49 smithi059 ceph-mon[97450]: pgmap v15238: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:28:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:49.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:49.968532+0000) 2022-09-08T05:28:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:49.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:49.968753+0000) 2022-09-08T05:28:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:50.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:50.968713+0000) 2022-09-08T05:28:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:50.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:50.968885+0000) 2022-09-08T05:28:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:51 smithi055 ceph-mon[97033]: pgmap v15239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:28:51.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:51 smithi059 ceph-mon[97450]: pgmap v15239: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:28:52.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:51.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:51.968919+0000) 2022-09-08T05:28:52.412 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:51.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:51.969018+0000) 2022-09-08T05:28:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:52.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:52.969134+0000) 2022-09-08T05:28:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:52.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:52.969246+0000) 2022-09-08T05:28:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:53 smithi055 ceph-mon[97033]: pgmap v15240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:28:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:53 smithi059 ceph-mon[97450]: pgmap v15240: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:28:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:53.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:53.969362+0000) 2022-09-08T05:28:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:53.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:53.969425+0000) 2022-09-08T05:28:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:54.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:54.969675+0000) 2022-09-08T05:28:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:54.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:54.969683+0000) 2022-09-08T05:28:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:55 smithi055 ceph-mon[97033]: pgmap v15241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:28:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:55 smithi059 ceph-mon[97450]: pgmap v15241: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:28:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:55.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:55.969951+0000) 2022-09-08T05:28:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:55.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:55.969949+0000) 2022-09-08T05:28:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:56 smithi055 ceph-mon[97033]: pgmap v15242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:28:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:56 smithi059 ceph-mon[97450]: pgmap v15242: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:28:57.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:56.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:56.970169+0000) 2022-09-08T05:28:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:56.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:56.970165+0000) 2022-09-08T05:28:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:57.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:57.970408+0000) 2022-09-08T05:28:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:57.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:57.970426+0000) 2022-09-08T05:28:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:58.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:58.970753+0000) 2022-09-08T05:28:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:58.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:58.970752+0000) 2022-09-08T05:28:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:28:59 smithi055 ceph-mon[97033]: pgmap v15243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:28:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:28:59 smithi059 ceph-mon[97450]: pgmap v15243: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:29:00.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:28:59.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:59.971058+0000) 2022-09-08T05:29:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:28:59.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:28:59.971060+0000) 2022-09-08T05:29:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:00.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:00.971229+0000) 2022-09-08T05:29:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:00.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:00.971225+0000) 2022-09-08T05:29:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:01 smithi055 ceph-mon[97033]: pgmap v15244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:29:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:01 smithi059 ceph-mon[97450]: pgmap v15244: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:29:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:01.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:01.971504+0000) 2022-09-08T05:29:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:01.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:01.971504+0000) 2022-09-08T05:29:03.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:29:03.073+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:29:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:02.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:02.971799+0000) 2022-09-08T05:29:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:02.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:02.971799+0000) 2022-09-08T05:29:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:03 smithi055 ceph-mon[97033]: pgmap v15245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:29:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:03 smithi059 ceph-mon[97450]: pgmap v15245: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:29:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:03.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:03.972006+0000) 2022-09-08T05:29:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:03.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:03.971999+0000) 2022-09-08T05:29:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:04.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:04.972273+0000) 2022-09-08T05:29:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:04.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:04.972268+0000) 2022-09-08T05:29:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:05 smithi055 ceph-mon[97033]: pgmap v15246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:29:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:05 smithi059 ceph-mon[97450]: pgmap v15246: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:29:06.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:05.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:05.972485+0000) 2022-09-08T05:29:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:05.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:05.972504+0000) 2022-09-08T05:29:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:29:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:29:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:06.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:06.972744+0000) 2022-09-08T05:29:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:06.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:06.972736+0000) 2022-09-08T05:29:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:07 smithi055 ceph-mon[97033]: pgmap v15247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:29:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:29:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:07 smithi059 ceph-mon[97450]: pgmap v15247: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:29:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:29:08.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:07.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:07.973026+0000) 2022-09-08T05:29:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:07.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:07.972969+0000) 2022-09-08T05:29:09.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:08.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:08.973293+0000) 2022-09-08T05:29:09.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:08.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:08.973210+0000) 2022-09-08T05:29:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:09 smithi059 ceph-mon[97450]: pgmap v15248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:29:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:09 smithi055 ceph-mon[97033]: pgmap v15248: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:29:10.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:09.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:09.973584+0000) 2022-09-08T05:29:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:09.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:09.973491+0000) 2022-09-08T05:29:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:10 smithi059 ceph-mon[97450]: osdmap e1033: 12 total, 11 up, 11 in 2022-09-08T05:29:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:10 smithi055 ceph-mon[97033]: osdmap e1033: 12 total, 11 up, 11 in 2022-09-08T05:29:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:10.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:10.973754+0000) 2022-09-08T05:29:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:10.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:10.973871+0000) 2022-09-08T05:29:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:11 smithi059 ceph-mon[97450]: pgmap v15250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:29:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:11 smithi055 ceph-mon[97033]: pgmap v15250: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:29:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:11.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:11.973927+0000) 2022-09-08T05:29:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:11.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:11.974002+0000) 2022-09-08T05:29:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:12.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:12.974076+0000) 2022-09-08T05:29:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:12.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:12.974152+0000) 2022-09-08T05:29:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:13 smithi059 ceph-mon[97450]: pgmap v15251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:29:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:13 smithi055 ceph-mon[97033]: pgmap v15251: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:29:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:13.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:13.974323+0000) 2022-09-08T05:29:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:13.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:13.974375+0000) 2022-09-08T05:29:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:14.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:14.974489+0000) 2022-09-08T05:29:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:14.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:14.974516+0000) 2022-09-08T05:29:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:15 smithi059 ceph-mon[97450]: pgmap v15252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:29:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:15 smithi055 ceph-mon[97033]: pgmap v15252: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:29:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:15.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:15.974747+0000) 2022-09-08T05:29:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:15.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:15.974735+0000) 2022-09-08T05:29:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:16 smithi055 ceph-mon[97033]: pgmap v15253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:29:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:16 smithi059 ceph-mon[97450]: pgmap v15253: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:29:17.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:16.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:16.974945+0000) 2022-09-08T05:29:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:16.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:16.974943+0000) 2022-09-08T05:29:18.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:29:18.088+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:29:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:17.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:17.975181+0000) 2022-09-08T05:29:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:17.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:17.975167+0000) 2022-09-08T05:29:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:18.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:18.975383+0000) 2022-09-08T05:29:19.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:18.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:18.975390+0000) 2022-09-08T05:29:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:19 smithi055 ceph-mon[97033]: pgmap v15254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:29:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:19 smithi059 ceph-mon[97450]: pgmap v15254: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:29:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:19.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:19.975589+0000) 2022-09-08T05:29:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:19.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:19.975661+0000) 2022-09-08T05:29:21.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:20.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:20.975945+0000) 2022-09-08T05:29:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:20.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:20.975890+0000) 2022-09-08T05:29:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:21 smithi055 ceph-mon[97033]: pgmap v15255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T05:29:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:21 smithi055 ceph-mon[97033]: osdmap e1034: 12 total, 11 up, 11 in 2022-09-08T05:29:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:21 smithi059 ceph-mon[97450]: pgmap v15255: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T05:29:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:21 smithi059 ceph-mon[97450]: osdmap e1034: 12 total, 11 up, 11 in 2022-09-08T05:29:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:21.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:21.976153+0000) 2022-09-08T05:29:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:21.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:21.976162+0000) 2022-09-08T05:29:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:22.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:22.976400+0000) 2022-09-08T05:29:23.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:22.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:22.976381+0000) 2022-09-08T05:29:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:23 smithi055 ceph-mon[97033]: pgmap v15257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:29:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:23 smithi059 ceph-mon[97450]: pgmap v15257: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:29:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:23.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:23.976622+0000) 2022-09-08T05:29:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:23.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:23.976566+0000) 2022-09-08T05:29:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:24.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:24.976855+0000) 2022-09-08T05:29:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:24.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:24.976802+0000) 2022-09-08T05:29:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:25 smithi055 ceph-mon[97033]: pgmap v15258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:29:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:25 smithi059 ceph-mon[97450]: pgmap v15258: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:29:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:25.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:25.977139+0000) 2022-09-08T05:29:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:25.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:25.977023+0000) 2022-09-08T05:29:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:26 smithi059 ceph-mon[97450]: pgmap v15259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:29:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:26 smithi055 ceph-mon[97033]: pgmap v15259: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:29:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:26.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:26.977321+0000) 2022-09-08T05:29:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:26.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:26.977260+0000) 2022-09-08T05:29:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:27.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:27.977466+0000) 2022-09-08T05:29:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:27.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:27.977443+0000) 2022-09-08T05:29:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:28.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:28.977726+0000) 2022-09-08T05:29:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:28.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:28.977699+0000) 2022-09-08T05:29:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:29 smithi055 ceph-mon[97033]: pgmap v15260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:29:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:29 smithi059 ceph-mon[97450]: pgmap v15260: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:29:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:29.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:29.977908+0000) 2022-09-08T05:29:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:29.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:29.977855+0000) 2022-09-08T05:29:31.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:30.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:30.978038+0000) 2022-09-08T05:29:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:30.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:30.978066+0000) 2022-09-08T05:29:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:31 smithi055 ceph-mon[97033]: pgmap v15261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:29:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:31 smithi059 ceph-mon[97450]: pgmap v15261: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:29:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:31.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:31.978254+0000) 2022-09-08T05:29:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:31.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:31.978242+0000) 2022-09-08T05:29:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:32.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:32.978435+0000) 2022-09-08T05:29:33.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:29:33.100+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:29:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:32.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:32.978444+0000) 2022-09-08T05:29:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:33 smithi055 ceph-mon[97033]: pgmap v15262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:29:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:33 smithi059 ceph-mon[97450]: pgmap v15262: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:29:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:33.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:33.978661+0000) 2022-09-08T05:29:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:33.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:33.978650+0000) 2022-09-08T05:29:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:34.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:34.978918+0000) 2022-09-08T05:29:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:34.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:34.978897+0000) 2022-09-08T05:29:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:35 smithi055 ceph-mon[97033]: pgmap v15263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:29:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:35 smithi059 ceph-mon[97450]: pgmap v15263: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:29:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:35.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:35.979160+0000) 2022-09-08T05:29:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:35.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:35.979128+0000) 2022-09-08T05:29:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:36 smithi059 ceph-mon[97450]: pgmap v15264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:29:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:36 smithi055 ceph-mon[97033]: pgmap v15264: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:29:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:36.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:36.979331+0000) 2022-09-08T05:29:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:36.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:36.979334+0000) 2022-09-08T05:29:38.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:37.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:37.979565+0000) 2022-09-08T05:29:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:37.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:37.979584+0000) 2022-09-08T05:29:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:38.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:38.979834+0000) 2022-09-08T05:29:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:38.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:38.979824+0000) 2022-09-08T05:29:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:39 smithi055 ceph-mon[97033]: pgmap v15265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:29:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:39 smithi059 ceph-mon[97450]: pgmap v15265: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:29:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:39.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:39.980035+0000) 2022-09-08T05:29:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:39.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:39.980044+0000) 2022-09-08T05:29:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:40.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:40.980221+0000) 2022-09-08T05:29:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:40.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:40.980220+0000) 2022-09-08T05:29:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:41 smithi055 ceph-mon[97033]: pgmap v15266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:29:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:41 smithi059 ceph-mon[97450]: pgmap v15266: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:29:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:41.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:41.980451+0000) 2022-09-08T05:29:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:41.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:41.980449+0000) 2022-09-08T05:29:43.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:42.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:42.980696+0000) 2022-09-08T05:29:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:42.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:42.980705+0000) 2022-09-08T05:29:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:43 smithi055 ceph-mon[97033]: pgmap v15267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:29:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:43 smithi059 ceph-mon[97450]: pgmap v15267: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:29:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:43.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:43.980914+0000) 2022-09-08T05:29:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:43.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:43.980918+0000) 2022-09-08T05:29:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:44.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:44.981135+0000) 2022-09-08T05:29:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:44.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:44.981138+0000) 2022-09-08T05:29:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:45 smithi055 ceph-mon[97033]: pgmap v15268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:29:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:45 smithi059 ceph-mon[97450]: pgmap v15268: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:29:46.287 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:45.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:45.981368+0000) 2022-09-08T05:29:46.288 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:45.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:45.981346+0000) 2022-09-08T05:29:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:46 smithi059 ceph-mon[97450]: pgmap v15269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:29:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:29:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:29:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:29:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:46 smithi055 ceph-mon[97033]: pgmap v15269: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:29:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:29:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:29:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:29:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:46.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:46.981607+0000) 2022-09-08T05:29:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:46.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:46.981573+0000) 2022-09-08T05:29:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:29:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:29:48.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:29:48.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:29:48.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:29:48.112+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:29:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:47.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:47.981826+0000) 2022-09-08T05:29:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:47.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:47.981757+0000) 2022-09-08T05:29:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:48 smithi059 ceph-mon[97450]: pgmap v15270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:29:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:48 smithi055 ceph-mon[97033]: pgmap v15270: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:29:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:48.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:48.981987+0000) 2022-09-08T05:29:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:48.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:48.981967+0000) 2022-09-08T05:29:50.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:49.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:49.982270+0000) 2022-09-08T05:29:50.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:49.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:49.982226+0000) 2022-09-08T05:29:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:29:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:29:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:29:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:29:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:50 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T05:29:50.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:50 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T05:29:50.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:29:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:29:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:29:50.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:29:50.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:50 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T05:29:50.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:50 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T05:29:50.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.538 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.538 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:50.539 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:29:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:50.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:50.982521+0000) 2022-09-08T05:29:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:50.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:50.982521+0000) 2022-09-08T05:29:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:51 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T05:29:51.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:51 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T05:29:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:51 smithi055 ceph-mon[97033]: pgmap v15271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:29:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:51 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T05:29:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:51 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T05:29:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:51 smithi059 ceph-mon[97450]: pgmap v15271: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:29:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:51.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:51.982791+0000) 2022-09-08T05:29:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:51.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:51.982789+0000) 2022-09-08T05:29:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:52.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:52.982939+0000) 2022-09-08T05:29:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:52.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:52.982938+0000) 2022-09-08T05:29:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:53 smithi055 ceph-mon[97033]: pgmap v15272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:29:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:53 smithi059 ceph-mon[97450]: pgmap v15272: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:29:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:53.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:53.983128+0000) 2022-09-08T05:29:54.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:53.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:53.983128+0000) 2022-09-08T05:29:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:54.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:54.983353+0000) 2022-09-08T05:29:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:54.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:54.983353+0000) 2022-09-08T05:29:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:55 smithi055 ceph-mon[97033]: pgmap v15273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:29:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:55 smithi059 ceph-mon[97450]: pgmap v15273: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:29:56.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:55.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:55.983597+0000) 2022-09-08T05:29:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:55.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:55.983597+0000) 2022-09-08T05:29:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:56 smithi059 ceph-mon[97450]: pgmap v15274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:29:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:56 smithi055 ceph-mon[97033]: pgmap v15274: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:29:57.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:56.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:56.983793+0000) 2022-09-08T05:29:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:56.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:56.983793+0000) 2022-09-08T05:29:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:57.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:57.984042+0000) 2022-09-08T05:29:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:57.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:57.984043+0000) 2022-09-08T05:29:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:58.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:58.984312+0000) 2022-09-08T05:29:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:58.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:58.984305+0000) 2022-09-08T05:29:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:29:59 smithi055 ceph-mon[97033]: pgmap v15275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:29:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:29:59 smithi059 ceph-mon[97450]: pgmap v15275: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:29:59.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:59.984676+0000) 2022-09-08T05:30:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:29:59.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:29:59.984613+0000) 2022-09-08T05:30:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T05:30:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T05:30:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T05:30:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T05:30:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T05:30:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T05:30:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T05:30:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T05:30:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T05:30:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T05:30:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:00.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:00.984856+0000) 2022-09-08T05:30:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:00.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:00.984815+0000) 2022-09-08T05:30:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:01 smithi055 ceph-mon[97033]: pgmap v15276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:30:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:01 smithi059 ceph-mon[97450]: pgmap v15276: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:30:02.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:01.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:01.985031+0000) 2022-09-08T05:30:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:01.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:01.985067+0000) 2022-09-08T05:30:03.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:30:03.124+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:30:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:02.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:02.985290+0000) 2022-09-08T05:30:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:02.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:02.985265+0000) 2022-09-08T05:30:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:03 smithi059 ceph-mon[97450]: pgmap v15277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:03.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:03 smithi055 ceph-mon[97033]: pgmap v15277: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:04.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:03.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:03.985517+0000) 2022-09-08T05:30:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:03.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:03.985543+0000) 2022-09-08T05:30:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:04.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:04.985766+0000) 2022-09-08T05:30:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:04.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:04.985754+0000) 2022-09-08T05:30:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:05 smithi059 ceph-mon[97450]: pgmap v15278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:05.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:05 smithi055 ceph-mon[97033]: pgmap v15278: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:05.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:05.986009+0000) 2022-09-08T05:30:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:05.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:05.985998+0000) 2022-09-08T05:30:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:30:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:30:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:06.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:06.986243+0000) 2022-09-08T05:30:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:06.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:06.986208+0000) 2022-09-08T05:30:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:07 smithi059 ceph-mon[97450]: pgmap v15279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:30:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:30:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:07 smithi055 ceph-mon[97033]: pgmap v15279: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:30:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:30:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:07.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:07.986385+0000) 2022-09-08T05:30:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:07.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:07.986385+0000) 2022-09-08T05:30:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:08.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:08.986628+0000) 2022-09-08T05:30:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:08.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:08.986630+0000) 2022-09-08T05:30:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:09 smithi059 ceph-mon[97450]: pgmap v15280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:30:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:09 smithi055 ceph-mon[97033]: pgmap v15280: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:30:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:09.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:09.986872+0000) 2022-09-08T05:30:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:09.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:09.986855+0000) 2022-09-08T05:30:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:10 smithi059 ceph-mon[97450]: osdmap e1035: 12 total, 11 up, 11 in 2022-09-08T05:30:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:10 smithi055 ceph-mon[97033]: osdmap e1035: 12 total, 11 up, 11 in 2022-09-08T05:30:11.358 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:10.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:10.987122+0000) 2022-09-08T05:30:11.359 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:10.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:10.987032+0000) 2022-09-08T05:30:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:11 smithi059 ceph-mon[97450]: pgmap v15282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:30:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:11 smithi059 ceph-mon[97450]: osdmap e1036: 12 total, 11 up, 11 in 2022-09-08T05:30:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:11 smithi055 ceph-mon[97033]: pgmap v15282: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:30:11.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:11 smithi055 ceph-mon[97033]: osdmap e1036: 12 total, 11 up, 11 in 2022-09-08T05:30:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:11.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:11.987314+0000) 2022-09-08T05:30:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:11.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:11.987251+0000) 2022-09-08T05:30:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:12.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:12.987479+0000) 2022-09-08T05:30:13.443 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:12.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:12.987435+0000) 2022-09-08T05:30:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:13 smithi059 ceph-mon[97450]: pgmap v15284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:30:13.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:13 smithi055 ceph-mon[97033]: pgmap v15284: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:30:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:13.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:13.987730+0000) 2022-09-08T05:30:14.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:13.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:13.987685+0000) 2022-09-08T05:30:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:14.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:14.987900+0000) 2022-09-08T05:30:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:14.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:14.987825+0000) 2022-09-08T05:30:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:15 smithi059 ceph-mon[97450]: pgmap v15285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:30:15.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:15 smithi055 ceph-mon[97033]: pgmap v15285: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:30:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:15.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:15.988091+0000) 2022-09-08T05:30:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:15.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:15.988051+0000) 2022-09-08T05:30:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:16.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:16.988273+0000) 2022-09-08T05:30:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:16.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:16.988229+0000) 2022-09-08T05:30:17.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:17 smithi055 ceph-mon[97033]: pgmap v15286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:30:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:17 smithi055 ceph-mon[97033]: pgmap v15287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.5 KiB/s rd, 285 B/s wr, 8 op/s 2022-09-08T05:30:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:17 smithi055 ceph-mon[97033]: pgmap v15288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T05:30:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:30:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:17 smithi059 ceph-mon[97450]: pgmap v15286: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:30:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:17 smithi059 ceph-mon[97450]: pgmap v15287: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.5 KiB/s rd, 285 B/s wr, 8 op/s 2022-09-08T05:30:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:17 smithi059 ceph-mon[97450]: pgmap v15288: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T05:30:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:30:18.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:30:18.131+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:30:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:17.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:17.988450+0000) 2022-09-08T05:30:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:17.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:17.988457+0000) 2022-09-08T05:30:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:18.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:18.988686+0000) 2022-09-08T05:30:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:18.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:18.988686+0000) 2022-09-08T05:30:19.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:19 smithi055 ceph-mon[97033]: pgmap v15289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.1 KiB/s rd, 133 B/s wr, 4 op/s 2022-09-08T05:30:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:19 smithi059 ceph-mon[97450]: pgmap v15289: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.1 KiB/s rd, 133 B/s wr, 4 op/s 2022-09-08T05:30:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:19.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:19.988842+0000) 2022-09-08T05:30:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:19.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:19.988839+0000) 2022-09-08T05:30:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:20.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:20.989114+0000) 2022-09-08T05:30:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:20.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:20.989114+0000) 2022-09-08T05:30:21.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:21 smithi055 ceph-mon[97033]: pgmap v15290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T05:30:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:21 smithi059 ceph-mon[97450]: pgmap v15290: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T05:30:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:21.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:21.989351+0000) 2022-09-08T05:30:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:21.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:21.989351+0000) 2022-09-08T05:30:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:22.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:22.989618+0000) 2022-09-08T05:30:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:22.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:22.989618+0000) 2022-09-08T05:30:23.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:23 smithi055 ceph-mon[97033]: pgmap v15291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T05:30:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:23 smithi059 ceph-mon[97450]: pgmap v15291: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T05:30:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:23.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:23.989942+0000) 2022-09-08T05:30:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:23.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:23.989941+0000) 2022-09-08T05:30:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:24.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:24.990194+0000) 2022-09-08T05:30:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:24.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:24.990195+0000) 2022-09-08T05:30:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:25 smithi055 ceph-mon[97033]: pgmap v15292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:30:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:25 smithi059 ceph-mon[97450]: pgmap v15292: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:30:26.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:25.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:25.990433+0000) 2022-09-08T05:30:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:25.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:25.990433+0000) 2022-09-08T05:30:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:26 smithi059 ceph-mon[97450]: pgmap v15293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:30:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:26 smithi055 ceph-mon[97033]: pgmap v15293: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:30:27.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:26.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:26.990688+0000) 2022-09-08T05:30:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:26.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:26.990714+0000) 2022-09-08T05:30:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:27.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:27.990978+0000) 2022-09-08T05:30:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:27.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:27.990972+0000) 2022-09-08T05:30:29.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:28.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:28.991174+0000) 2022-09-08T05:30:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:28.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:28.991158+0000) 2022-09-08T05:30:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:29 smithi059 ceph-mon[97450]: pgmap v15294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:29 smithi055 ceph-mon[97033]: pgmap v15294: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:29.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:29.991394+0000) 2022-09-08T05:30:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:29.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:29.991403+0000) 2022-09-08T05:30:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:30.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:30.991673+0000) 2022-09-08T05:30:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:30.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:30.991660+0000) 2022-09-08T05:30:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:31 smithi059 ceph-mon[97450]: pgmap v15295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:30:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:31 smithi055 ceph-mon[97033]: pgmap v15295: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:30:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:31.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:31.991820+0000) 2022-09-08T05:30:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:31.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:31.991827+0000) 2022-09-08T05:30:33.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:30:33.141+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:30:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:32.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:32.991988+0000) 2022-09-08T05:30:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:32.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:32.991941+0000) 2022-09-08T05:30:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:33 smithi059 ceph-mon[97450]: pgmap v15296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:33 smithi055 ceph-mon[97033]: pgmap v15296: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:33.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:33.992151+0000) 2022-09-08T05:30:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:33.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:33.992197+0000) 2022-09-08T05:30:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:34.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:34.992399+0000) 2022-09-08T05:30:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:34.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:34.992435+0000) 2022-09-08T05:30:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:35 smithi059 ceph-mon[97450]: pgmap v15297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:35 smithi055 ceph-mon[97033]: pgmap v15297: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:35.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:35.992718+0000) 2022-09-08T05:30:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:35.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:35.992713+0000) 2022-09-08T05:30:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:36 smithi059 ceph-mon[97450]: pgmap v15298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:30:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:36 smithi055 ceph-mon[97033]: pgmap v15298: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:30:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:36.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:36.993016+0000) 2022-09-08T05:30:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:36.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:36.992974+0000) 2022-09-08T05:30:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:37.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:37.993290+0000) 2022-09-08T05:30:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:37.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:37.993212+0000) 2022-09-08T05:30:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:38.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:38.993425+0000) 2022-09-08T05:30:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:38.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:38.993409+0000) 2022-09-08T05:30:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:39 smithi059 ceph-mon[97450]: pgmap v15299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:39 smithi055 ceph-mon[97033]: pgmap v15299: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:40.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:39.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:39.993706+0000) 2022-09-08T05:30:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:39.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:39.993712+0000) 2022-09-08T05:30:41.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:40.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:40.993903+0000) 2022-09-08T05:30:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:40.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:40.993870+0000) 2022-09-08T05:30:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:41 smithi059 ceph-mon[97450]: pgmap v15300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:30:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:41 smithi055 ceph-mon[97033]: pgmap v15300: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:30:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:41.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:41.994102+0000) 2022-09-08T05:30:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:41.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:41.994142+0000) 2022-09-08T05:30:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:42.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:42.994313+0000) 2022-09-08T05:30:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:42.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:42.994322+0000) 2022-09-08T05:30:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:43 smithi059 ceph-mon[97450]: pgmap v15301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:43 smithi055 ceph-mon[97033]: pgmap v15301: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:43.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:43.994581+0000) 2022-09-08T05:30:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:43.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:43.994591+0000) 2022-09-08T05:30:45.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:44.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:44.994918+0000) 2022-09-08T05:30:45.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:44.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:44.994849+0000) 2022-09-08T05:30:45.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:45 smithi059 ceph-mon[97450]: pgmap v15302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:45 smithi055 ceph-mon[97033]: pgmap v15302: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:45.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:45.995098+0000) 2022-09-08T05:30:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:45.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:45.995033+0000) 2022-09-08T05:30:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:46 smithi059 ceph-mon[97450]: pgmap v15303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:30:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:46 smithi055 ceph-mon[97033]: pgmap v15303: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:30:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:46.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:46.995276+0000) 2022-09-08T05:30:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:46.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:46.995301+0000) 2022-09-08T05:30:48.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:30:48.152+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:30:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:47.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:47.995543+0000) 2022-09-08T05:30:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:47.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:47.995567+0000) 2022-09-08T05:30:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:48.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:48.995777+0000) 2022-09-08T05:30:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:48.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:48.995819+0000) 2022-09-08T05:30:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:49 smithi059 ceph-mon[97450]: pgmap v15304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:49 smithi055 ceph-mon[97033]: pgmap v15304: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:49.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:49.995951+0000) 2022-09-08T05:30:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:49.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:49.996021+0000) 2022-09-08T05:30:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:50.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:50.996186+0000) 2022-09-08T05:30:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:50.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:50.996246+0000) 2022-09-08T05:30:51.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:51 smithi059 ceph-mon[97450]: pgmap v15305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:30:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:51 smithi055 ceph-mon[97033]: pgmap v15305: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:30:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:51.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:51.996375+0000) 2022-09-08T05:30:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:51.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:51.996430+0000) 2022-09-08T05:30:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:52.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:52.996593+0000) 2022-09-08T05:30:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:52.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:52.996693+0000) 2022-09-08T05:30:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:53 smithi059 ceph-mon[97450]: pgmap v15306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:53.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:53 smithi055 ceph-mon[97033]: pgmap v15306: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:53.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:53.996779+0000) 2022-09-08T05:30:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:53.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:53.996841+0000) 2022-09-08T05:30:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:54.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:54.996962+0000) 2022-09-08T05:30:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:54.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:54.997066+0000) 2022-09-08T05:30:55.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:55 smithi059 ceph-mon[97450]: pgmap v15307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:55 smithi055 ceph-mon[97033]: pgmap v15307: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:55.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:55.997155+0000) 2022-09-08T05:30:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:55.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:55.997215+0000) 2022-09-08T05:30:57.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:56 smithi059 ceph-mon[97450]: pgmap v15308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:30:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:56 smithi055 ceph-mon[97033]: pgmap v15308: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:30:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:56.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:56.997375+0000) 2022-09-08T05:30:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:56.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:56.997422+0000) 2022-09-08T05:30:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:57.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:57.997651+0000) 2022-09-08T05:30:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:57.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:57.997665+0000) 2022-09-08T05:30:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:58.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:58.997897+0000) 2022-09-08T05:30:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:58.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:58.997897+0000) 2022-09-08T05:30:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:30:59 smithi059 ceph-mon[97450]: pgmap v15309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:30:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:30:59 smithi055 ceph-mon[97033]: pgmap v15309: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:30:59.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:59.998124+0000) 2022-09-08T05:31:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:30:59.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:30:59.998122+0000) 2022-09-08T05:31:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:00.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:00.998272+0000) 2022-09-08T05:31:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:00.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:00.998294+0000) 2022-09-08T05:31:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:01 smithi059 ceph-mon[97450]: pgmap v15310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:31:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:01 smithi055 ceph-mon[97033]: pgmap v15310: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:31:02.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:01.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:01.998549+0000) 2022-09-08T05:31:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:01.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:01.998515+0000) 2022-09-08T05:31:03.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:31:03.167+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:31:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:02.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:02.998765+0000) 2022-09-08T05:31:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:02.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:02.998805+0000) 2022-09-08T05:31:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:03 smithi059 ceph-mon[97450]: pgmap v15311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:03.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:03 smithi055 ceph-mon[97033]: pgmap v15311: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:03.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:03.999027+0000) 2022-09-08T05:31:04.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:03.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:03.999067+0000) 2022-09-08T05:31:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:04.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:04.999218+0000) 2022-09-08T05:31:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:04.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:04.999265+0000) 2022-09-08T05:31:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:05 smithi059 ceph-mon[97450]: pgmap v15312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:05.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:05 smithi055 ceph-mon[97033]: pgmap v15312: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:05.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:05.999425+0000) 2022-09-08T05:31:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:05.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:05.999509+0000) 2022-09-08T05:31:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:31:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:31:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:31:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:31:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:06.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:06.999729+0000) 2022-09-08T05:31:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:06.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:06.999769+0000) 2022-09-08T05:31:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:07 smithi059 ceph-mon[97450]: pgmap v15313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:31:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:07 smithi055 ceph-mon[97033]: pgmap v15313: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:31:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:07.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:07.999904+0000) 2022-09-08T05:31:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:07.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:07.999937+0000) 2022-09-08T05:31:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:08.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:09.000062+0000) 2022-09-08T05:31:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:08.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:09.000107+0000) 2022-09-08T05:31:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:09 smithi059 ceph-mon[97450]: pgmap v15314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:31:09.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:09 smithi055 ceph-mon[97033]: pgmap v15314: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:31:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:09.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:10.000226+0000) 2022-09-08T05:31:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:09.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:10.000242+0000) 2022-09-08T05:31:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:10 smithi059 ceph-mon[97450]: osdmap e1037: 12 total, 11 up, 11 in 2022-09-08T05:31:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:10 smithi055 ceph-mon[97033]: osdmap e1037: 12 total, 11 up, 11 in 2022-09-08T05:31:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:10.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:11.000370+0000) 2022-09-08T05:31:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:10.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:11.000422+0000) 2022-09-08T05:31:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:11 smithi059 ceph-mon[97450]: pgmap v15316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:31:11.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:11 smithi059 ceph-mon[97450]: osdmap e1038: 12 total, 11 up, 11 in 2022-09-08T05:31:11.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:11 smithi055 ceph-mon[97033]: pgmap v15316: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:31:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:11 smithi055 ceph-mon[97033]: osdmap e1038: 12 total, 11 up, 11 in 2022-09-08T05:31:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:11.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:12.000543+0000) 2022-09-08T05:31:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:11.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:12.000595+0000) 2022-09-08T05:31:13.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:13.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:13.000700+0000) 2022-09-08T05:31:13.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:13.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:13.000728+0000) 2022-09-08T05:31:13.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:13 smithi055 ceph-mon[97033]: pgmap v15318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:31:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:13 smithi059 ceph-mon[97450]: pgmap v15318: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:31:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:14.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:14.000931+0000) 2022-09-08T05:31:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:14.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:14.000957+0000) 2022-09-08T05:31:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:15.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:15.001134+0000) 2022-09-08T05:31:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:15.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:15.001189+0000) 2022-09-08T05:31:15.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:15 smithi055 ceph-mon[97033]: pgmap v15319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:31:16.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:15 smithi059 ceph-mon[97450]: pgmap v15319: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:31:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:16.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:16.001381+0000) 2022-09-08T05:31:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:16.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:16.001427+0000) 2022-09-08T05:31:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:16 smithi055 ceph-mon[97033]: pgmap v15320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:31:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:31:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:31:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:31:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:16 smithi059 ceph-mon[97450]: pgmap v15320: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:31:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:31:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:31:17.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:31:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:17.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:17.001559+0000) 2022-09-08T05:31:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:17.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:17.001593+0000) 2022-09-08T05:31:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:31:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:31:18.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:31:18.182+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:31:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:18.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:18.001781+0000) 2022-09-08T05:31:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:18.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:18.001808+0000) 2022-09-08T05:31:18.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:31:18.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:31:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:19.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:19.001949+0000) 2022-09-08T05:31:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:19.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:19.001998+0000) 2022-09-08T05:31:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:19 smithi055 ceph-mon[97033]: pgmap v15321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T05:31:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:19 smithi059 ceph-mon[97450]: pgmap v15321: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T05:31:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:20.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:20.002083+0000) 2022-09-08T05:31:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:20.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:20.002162+0000) 2022-09-08T05:31:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:21.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:21.002244+0000) 2022-09-08T05:31:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:21.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:21.002308+0000) 2022-09-08T05:31:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:21 smithi059 ceph-mon[97450]: pgmap v15322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:31:21.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:21 smithi055 ceph-mon[97033]: pgmap v15322: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:31:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:22.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:22.002477+0000) 2022-09-08T05:31:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:22.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:22.002529+0000) 2022-09-08T05:31:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:23.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:23.002699+0000) 2022-09-08T05:31:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:23.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:23.002778+0000) 2022-09-08T05:31:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:23 smithi059 ceph-mon[97450]: pgmap v15323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 87 B/s wr, 5 op/s 2022-09-08T05:31:23.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:23 smithi055 ceph-mon[97033]: pgmap v15323: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 87 B/s wr, 5 op/s 2022-09-08T05:31:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:24.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:24.002957+0000) 2022-09-08T05:31:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:24.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:24.003046+0000) 2022-09-08T05:31:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:25.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:25.003347+0000) 2022-09-08T05:31:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:25.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:25.003168+0000) 2022-09-08T05:31:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:25 smithi059 ceph-mon[97450]: pgmap v15324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:31:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:25 smithi055 ceph-mon[97033]: pgmap v15324: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:31:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:26.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:26.003357+0000) 2022-09-08T05:31:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:26.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:26.003533+0000) 2022-09-08T05:31:27.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:26 smithi059 ceph-mon[97450]: pgmap v15325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:31:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:26 smithi055 ceph-mon[97033]: pgmap v15325: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:31:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:27.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:27.003588+0000) 2022-09-08T05:31:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:27.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:27.003779+0000) 2022-09-08T05:31:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:28.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:28.003793+0000) 2022-09-08T05:31:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:28.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:28.004035+0000) 2022-09-08T05:31:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:29.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:29.003988+0000) 2022-09-08T05:31:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:29.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:29.004189+0000) 2022-09-08T05:31:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:29 smithi059 ceph-mon[97450]: pgmap v15326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:29 smithi055 ceph-mon[97033]: pgmap v15326: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:30.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:30.004202+0000) 2022-09-08T05:31:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:30.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:30.004321+0000) 2022-09-08T05:31:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:31.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:31.004407+0000) 2022-09-08T05:31:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:31.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:31.004439+0000) 2022-09-08T05:31:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:31 smithi059 ceph-mon[97450]: pgmap v15327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:31:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:31 smithi055 ceph-mon[97033]: pgmap v15327: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:31:32.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:32.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:32.004649+0000) 2022-09-08T05:31:32.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:32.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:32.004671+0000) 2022-09-08T05:31:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:33.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:33.004831+0000) 2022-09-08T05:31:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:33.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:33.004919+0000) 2022-09-08T05:31:33.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:31:33.197+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:31:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:33 smithi059 ceph-mon[97450]: pgmap v15328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:33 smithi055 ceph-mon[97033]: pgmap v15328: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:34.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:34.004996+0000) 2022-09-08T05:31:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:34.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:34.005123+0000) 2022-09-08T05:31:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:35.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:35.005206+0000) 2022-09-08T05:31:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:35.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:35.005298+0000) 2022-09-08T05:31:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:35 smithi059 ceph-mon[97450]: pgmap v15329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:35 smithi055 ceph-mon[97033]: pgmap v15329: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:36.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:36.005404+0000) 2022-09-08T05:31:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:36.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:36.005483+0000) 2022-09-08T05:31:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:36 smithi059 ceph-mon[97450]: pgmap v15330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:31:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:36 smithi055 ceph-mon[97033]: pgmap v15330: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:31:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:37.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:37.005676+0000) 2022-09-08T05:31:37.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:37.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:37.005739+0000) 2022-09-08T05:31:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:38.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:38.005916+0000) 2022-09-08T05:31:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:38.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:38.005939+0000) 2022-09-08T05:31:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:39.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:39.006116+0000) 2022-09-08T05:31:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:39.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:39.006124+0000) 2022-09-08T05:31:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:39 smithi059 ceph-mon[97450]: pgmap v15331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:39 smithi055 ceph-mon[97033]: pgmap v15331: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:40.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:40.006325+0000) 2022-09-08T05:31:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:40.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:40.006357+0000) 2022-09-08T05:31:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:41.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:41.006580+0000) 2022-09-08T05:31:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:41.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:41.006579+0000) 2022-09-08T05:31:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:41 smithi059 ceph-mon[97450]: pgmap v15332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:31:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:41 smithi055 ceph-mon[97033]: pgmap v15332: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:31:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:42.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:42.006803+0000) 2022-09-08T05:31:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:42.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:42.006843+0000) 2022-09-08T05:31:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:43.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:43.007020+0000) 2022-09-08T05:31:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:43.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:43.007032+0000) 2022-09-08T05:31:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:43 smithi059 ceph-mon[97450]: pgmap v15333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:43 smithi055 ceph-mon[97033]: pgmap v15333: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:44.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:44.007196+0000) 2022-09-08T05:31:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:44.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:44.007199+0000) 2022-09-08T05:31:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:45.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:45.007464+0000) 2022-09-08T05:31:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:45.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:45.007438+0000) 2022-09-08T05:31:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:45 smithi059 ceph-mon[97450]: pgmap v15334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:45.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:45 smithi055 ceph-mon[97033]: pgmap v15334: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:31:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:46.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:46.007768+0000) 2022-09-08T05:31:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:46.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:46.007739+0000) 2022-09-08T05:31:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:46 smithi055 ceph-mon[97033]: pgmap v15335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:31:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:31:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:46 smithi059 ceph-mon[97450]: pgmap v15335: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:31:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:31:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:47.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:47.008034+0000) 2022-09-08T05:31:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:47.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:47.007995+0000) 2022-09-08T05:31:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:47 smithi059 ceph-mon[97450]: pgmap v15336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:31:48.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:47 smithi055 ceph-mon[97033]: pgmap v15336: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:31:48.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:31:48.209+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:31:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:48.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:48.008232+0000) 2022-09-08T05:31:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:48.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:48.008216+0000) 2022-09-08T05:31:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:49.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:49.008428+0000) 2022-09-08T05:31:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:49.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:49.008428+0000) 2022-09-08T05:31:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:49 smithi059 ceph-mon[97450]: pgmap v15337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:31:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:49 smithi055 ceph-mon[97033]: pgmap v15337: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:31:50.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:50.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:50.008701+0000) 2022-09-08T05:31:50.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:50.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:50.008694+0000) 2022-09-08T05:31:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:51.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:51.008929+0000) 2022-09-08T05:31:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:51.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:51.008888+0000) 2022-09-08T05:31:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:51 smithi059 ceph-mon[97450]: pgmap v15338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:31:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:51 smithi055 ceph-mon[97033]: pgmap v15338: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:31:52.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:52.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:52.009131+0000) 2022-09-08T05:31:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:52.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:52.009134+0000) 2022-09-08T05:31:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:53.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:53.009361+0000) 2022-09-08T05:31:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:53.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:53.009391+0000) 2022-09-08T05:31:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:53 smithi059 ceph-mon[97450]: pgmap v15339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:31:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:53 smithi055 ceph-mon[97033]: pgmap v15339: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:31:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:54.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:54.009522+0000) 2022-09-08T05:31:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:54.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:54.009592+0000) 2022-09-08T05:31:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:55.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:55.009756+0000) 2022-09-08T05:31:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:55.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:55.009785+0000) 2022-09-08T05:31:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:55 smithi059 ceph-mon[97450]: pgmap v15340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:31:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:55 smithi055 ceph-mon[97033]: pgmap v15340: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:31:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:56.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:56.009965+0000) 2022-09-08T05:31:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:56.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:56.009975+0000) 2022-09-08T05:31:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:57.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:57.010189+0000) 2022-09-08T05:31:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:57.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:57.010220+0000) 2022-09-08T05:31:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:57 smithi059 ceph-mon[97450]: pgmap v15341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:31:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:57 smithi055 ceph-mon[97033]: pgmap v15341: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:31:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:58.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:58.010413+0000) 2022-09-08T05:31:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:58.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:58.010555+0000) 2022-09-08T05:31:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:31:59.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:59.010629+0000) 2022-09-08T05:31:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:31:59.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:31:59.010734+0000) 2022-09-08T05:32:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:31:59 smithi055 ceph-mon[97033]: pgmap v15342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:31:59 smithi059 ceph-mon[97450]: pgmap v15342: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:00.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:00.010910+0000) 2022-09-08T05:32:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:00.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:00.010954+0000) 2022-09-08T05:32:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:01.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:01.011080+0000) 2022-09-08T05:32:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:01.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:01.011136+0000) 2022-09-08T05:32:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:01 smithi055 ceph-mon[97033]: pgmap v15343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:32:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:01 smithi059 ceph-mon[97450]: pgmap v15343: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:32:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:02.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:02.011367+0000) 2022-09-08T05:32:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:02.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:02.011378+0000) 2022-09-08T05:32:03.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:32:03.225+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:32:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:03.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:03.011571+0000) 2022-09-08T05:32:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:03.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:03.011571+0000) 2022-09-08T05:32:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:03 smithi055 ceph-mon[97033]: pgmap v15344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:03 smithi059 ceph-mon[97450]: pgmap v15344: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:04.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:04.011814+0000) 2022-09-08T05:32:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:04.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:04.011809+0000) 2022-09-08T05:32:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:05.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:05.012027+0000) 2022-09-08T05:32:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:05.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:05.012019+0000) 2022-09-08T05:32:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:05 smithi055 ceph-mon[97033]: pgmap v15345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:32:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:05 smithi059 ceph-mon[97450]: pgmap v15345: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:32:06.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:06.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:06.012246+0000) 2022-09-08T05:32:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:06.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:06.012238+0000) 2022-09-08T05:32:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:32:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:32:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:07.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:07.012470+0000) 2022-09-08T05:32:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:07.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:07.012464+0000) 2022-09-08T05:32:08.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:07 smithi055 ceph-mon[97033]: pgmap v15346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:32:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:07 smithi059 ceph-mon[97450]: pgmap v15346: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:32:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:08.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:08.012734+0000) 2022-09-08T05:32:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:08.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:08.012738+0000) 2022-09-08T05:32:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:09.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:09.012983+0000) 2022-09-08T05:32:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:09.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:09.012979+0000) 2022-09-08T05:32:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:09 smithi055 ceph-mon[97033]: pgmap v15347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:32:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:09 smithi059 ceph-mon[97450]: pgmap v15347: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:32:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:10.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:10.013118+0000) 2022-09-08T05:32:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:10.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:10.013195+0000) 2022-09-08T05:32:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:10 smithi055 ceph-mon[97033]: osdmap e1039: 12 total, 11 up, 11 in 2022-09-08T05:32:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:10 smithi059 ceph-mon[97450]: osdmap e1039: 12 total, 11 up, 11 in 2022-09-08T05:32:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:11.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:11.013290+0000) 2022-09-08T05:32:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:11.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:11.013391+0000) 2022-09-08T05:32:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:11 smithi055 ceph-mon[97033]: pgmap v15349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:32:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:11 smithi059 ceph-mon[97450]: pgmap v15349: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:32:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:12.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:12.013503+0000) 2022-09-08T05:32:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:12.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:12.013568+0000) 2022-09-08T05:32:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:13.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:13.013701+0000) 2022-09-08T05:32:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:13.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:13.013739+0000) 2022-09-08T05:32:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:13 smithi055 ceph-mon[97033]: pgmap v15350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:32:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:13 smithi059 ceph-mon[97450]: pgmap v15350: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:32:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:14.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:14.013978+0000) 2022-09-08T05:32:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:14.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:14.013960+0000) 2022-09-08T05:32:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:15.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:15.014180+0000) 2022-09-08T05:32:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:15.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:15.014185+0000) 2022-09-08T05:32:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:15 smithi055 ceph-mon[97033]: pgmap v15351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:32:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:15 smithi059 ceph-mon[97450]: pgmap v15351: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:32:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:16.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:16.014419+0000) 2022-09-08T05:32:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:16.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:16.014428+0000) 2022-09-08T05:32:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:17.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:17.014601+0000) 2022-09-08T05:32:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:17.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:17.014570+0000) 2022-09-08T05:32:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:17 smithi055 ceph-mon[97033]: pgmap v15352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:32:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:17 smithi059 ceph-mon[97450]: pgmap v15352: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:32:18.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:32:18.230+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:32:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:18.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:18.014837+0000) 2022-09-08T05:32:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:18.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:18.014849+0000) 2022-09-08T05:32:19.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:19.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:19.015008+0000) 2022-09-08T05:32:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:19.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:19.015115+0000) 2022-09-08T05:32:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:19 smithi055 ceph-mon[97033]: pgmap v15353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:32:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:19 smithi059 ceph-mon[97450]: pgmap v15353: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:32:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:20.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:20.015314+0000) 2022-09-08T05:32:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:20.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:20.015225+0000) 2022-09-08T05:32:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:21.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:21.015515+0000) 2022-09-08T05:32:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:21.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:21.015420+0000) 2022-09-08T05:32:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:21 smithi059 ceph-mon[97450]: pgmap v15354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 94 B/s wr, 6 op/s 2022-09-08T05:32:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:21 smithi059 ceph-mon[97450]: osdmap e1040: 12 total, 11 up, 11 in 2022-09-08T05:32:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:21 smithi055 ceph-mon[97033]: pgmap v15354: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 94 B/s wr, 6 op/s 2022-09-08T05:32:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:21 smithi055 ceph-mon[97033]: osdmap e1040: 12 total, 11 up, 11 in 2022-09-08T05:32:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:22.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:22.015765+0000) 2022-09-08T05:32:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:22.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:22.015631+0000) 2022-09-08T05:32:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:23.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:23.016088+0000) 2022-09-08T05:32:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:23.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:23.015923+0000) 2022-09-08T05:32:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:23 smithi059 ceph-mon[97450]: pgmap v15356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:32:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:23 smithi055 ceph-mon[97033]: pgmap v15356: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:32:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:24.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:24.016330+0000) 2022-09-08T05:32:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:24.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:24.016179+0000) 2022-09-08T05:32:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:25.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:25.016482+0000) 2022-09-08T05:32:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:25.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:25.016395+0000) 2022-09-08T05:32:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:25 smithi059 ceph-mon[97450]: pgmap v15357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:32:26.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:25 smithi055 ceph-mon[97033]: pgmap v15357: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:32:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:26.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:26.016663+0000) 2022-09-08T05:32:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:26.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:26.016588+0000) 2022-09-08T05:32:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:27.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:27.016833+0000) 2022-09-08T05:32:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:27.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:27.016745+0000) 2022-09-08T05:32:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:27 smithi059 ceph-mon[97450]: pgmap v15358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:32:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:27 smithi055 ceph-mon[97033]: pgmap v15358: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:32:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:28.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:28.017050+0000) 2022-09-08T05:32:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:28.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:28.017001+0000) 2022-09-08T05:32:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:29.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:29.017276+0000) 2022-09-08T05:32:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:29.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:29.017235+0000) 2022-09-08T05:32:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:29 smithi059 ceph-mon[97450]: pgmap v15359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:32:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:29 smithi055 ceph-mon[97033]: pgmap v15359: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:32:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:30.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:30.017508+0000) 2022-09-08T05:32:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:30.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:30.017506+0000) 2022-09-08T05:32:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:31.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:31.017724+0000) 2022-09-08T05:32:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:31.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:31.017747+0000) 2022-09-08T05:32:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:31 smithi055 ceph-mon[97033]: pgmap v15360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:32:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:31 smithi059 ceph-mon[97450]: pgmap v15360: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:32:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:32.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:32.017940+0000) 2022-09-08T05:32:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:32.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:32.017949+0000) 2022-09-08T05:32:33.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:32:33.236+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:32:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:33.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:33.018150+0000) 2022-09-08T05:32:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:33.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:33.018163+0000) 2022-09-08T05:32:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:33 smithi055 ceph-mon[97033]: pgmap v15361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:32:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:33 smithi059 ceph-mon[97450]: pgmap v15361: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:32:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:34.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:34.018327+0000) 2022-09-08T05:32:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:34.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:34.018337+0000) 2022-09-08T05:32:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:35.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:35.018543+0000) 2022-09-08T05:32:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:35.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:35.018547+0000) 2022-09-08T05:32:36.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:35 smithi055 ceph-mon[97033]: pgmap v15362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:35 smithi059 ceph-mon[97450]: pgmap v15362: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:36.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:36.018797+0000) 2022-09-08T05:32:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:36.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:36.018813+0000) 2022-09-08T05:32:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:37.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:37.018976+0000) 2022-09-08T05:32:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:37.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:37.018997+0000) 2022-09-08T05:32:38.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:37 smithi055 ceph-mon[97033]: pgmap v15363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:32:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:37 smithi059 ceph-mon[97450]: pgmap v15363: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:32:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:38.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:38.019205+0000) 2022-09-08T05:32:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:38.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:38.019224+0000) 2022-09-08T05:32:39.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:39.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:39.019439+0000) 2022-09-08T05:32:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:39.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:39.019434+0000) 2022-09-08T05:32:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:39 smithi055 ceph-mon[97033]: pgmap v15364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:39 smithi059 ceph-mon[97450]: pgmap v15364: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:40.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:40.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:40.019599+0000) 2022-09-08T05:32:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:40.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:40.019614+0000) 2022-09-08T05:32:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:41.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:41.019793+0000) 2022-09-08T05:32:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:41.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:41.019791+0000) 2022-09-08T05:32:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:41 smithi055 ceph-mon[97033]: pgmap v15365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:32:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:41 smithi059 ceph-mon[97450]: pgmap v15365: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:32:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:42.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:42.020017+0000) 2022-09-08T05:32:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:42.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:42.019961+0000) 2022-09-08T05:32:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:43.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:43.020185+0000) 2022-09-08T05:32:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:43.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:43.020152+0000) 2022-09-08T05:32:44.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:43 smithi055 ceph-mon[97033]: pgmap v15366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:43 smithi059 ceph-mon[97450]: pgmap v15366: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:44.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:44.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:44.020420+0000) 2022-09-08T05:32:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:44.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:44.020406+0000) 2022-09-08T05:32:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:45.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:45.020617+0000) 2022-09-08T05:32:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:45.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:45.020619+0000) 2022-09-08T05:32:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:45 smithi055 ceph-mon[97033]: pgmap v15367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:45 smithi059 ceph-mon[97450]: pgmap v15367: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:46.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:46.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:46.020784+0000) 2022-09-08T05:32:46.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:46.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:46.020841+0000) 2022-09-08T05:32:47.151 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:47.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:47.021003+0000) 2022-09-08T05:32:47.152 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:47.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:47.020971+0000) 2022-09-08T05:32:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:32:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:32:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:32:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:32:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:32:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:32:48.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:47 smithi055 ceph-mon[97033]: pgmap v15368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:32:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:47 smithi059 ceph-mon[97450]: pgmap v15368: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:32:48.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:32:48.245+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:32:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:48.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:48.021216+0000) 2022-09-08T05:32:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:48.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:48.021222+0000) 2022-09-08T05:32:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:49.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:49.021371+0000) 2022-09-08T05:32:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:49.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:49.021422+0000) 2022-09-08T05:32:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:49 smithi055 ceph-mon[97033]: pgmap v15369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:49 smithi059 ceph-mon[97450]: pgmap v15369: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:50.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:50.021551+0000) 2022-09-08T05:32:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:50.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:50.021623+0000) 2022-09-08T05:32:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:51.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:51.021723+0000) 2022-09-08T05:32:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:51.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:51.021808+0000) 2022-09-08T05:32:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:51 smithi055 ceph-mon[97033]: pgmap v15370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:32:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:51 smithi059 ceph-mon[97450]: pgmap v15370: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:32:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:52.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:52.021945+0000) 2022-09-08T05:32:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:52.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:52.022024+0000) 2022-09-08T05:32:53.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:53.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:53.022259+0000) 2022-09-08T05:32:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:53.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:53.022156+0000) 2022-09-08T05:32:54.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:53 smithi055 ceph-mon[97033]: pgmap v15371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:53 smithi059 ceph-mon[97450]: pgmap v15371: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:54.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:54.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:54.022394+0000) 2022-09-08T05:32:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:54.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:54.022499+0000) 2022-09-08T05:32:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:55.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:55.022549+0000) 2022-09-08T05:32:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:55.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:55.022677+0000) 2022-09-08T05:32:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:55 smithi055 ceph-mon[97033]: pgmap v15372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:55 smithi059 ceph-mon[97450]: pgmap v15372: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:32:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:56.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:56.022719+0000) 2022-09-08T05:32:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:56.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:56.022846+0000) 2022-09-08T05:32:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:57.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:57.022924+0000) 2022-09-08T05:32:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:57.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:57.023017+0000) 2022-09-08T05:32:58.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:57 smithi055 ceph-mon[97033]: pgmap v15373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:32:58.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:57 smithi059 ceph-mon[97450]: pgmap v15373: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:32:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:58.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:58.023131+0000) 2022-09-08T05:32:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:58.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:58.023271+0000) 2022-09-08T05:32:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:32:59.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:59.023300+0000) 2022-09-08T05:32:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:32:59.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:32:59.023415+0000) 2022-09-08T05:33:00.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:32:59 smithi055 ceph-mon[97033]: pgmap v15374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:32:59 smithi059 ceph-mon[97450]: pgmap v15374: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:00.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:00.023483+0000) 2022-09-08T05:33:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:00.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:00.023595+0000) 2022-09-08T05:33:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:01.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:01.023732+0000) 2022-09-08T05:33:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:01.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:01.023859+0000) 2022-09-08T05:33:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:01 smithi055 ceph-mon[97033]: pgmap v15375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:33:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:01 smithi059 ceph-mon[97450]: pgmap v15375: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:33:02.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:02.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:02.024113+0000) 2022-09-08T05:33:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:02.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:02.023951+0000) 2022-09-08T05:33:03.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:33:03.253+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:33:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:03.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:03.024114+0000) 2022-09-08T05:33:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:03.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:03.024284+0000) 2022-09-08T05:33:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:03 smithi055 ceph-mon[97033]: pgmap v15376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:03 smithi059 ceph-mon[97450]: pgmap v15376: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:04.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:04.024371+0000) 2022-09-08T05:33:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:04.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:04.024527+0000) 2022-09-08T05:33:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:05.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:05.024631+0000) 2022-09-08T05:33:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:05.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:05.024739+0000) 2022-09-08T05:33:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:05 smithi055 ceph-mon[97033]: pgmap v15377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:33:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:05 smithi059 ceph-mon[97450]: pgmap v15377: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:33:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:06.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:06.024814+0000) 2022-09-08T05:33:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:06.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:06.024932+0000) 2022-09-08T05:33:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:33:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:33:07.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:07.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:07.025160+0000) 2022-09-08T05:33:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:07.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:07.025010+0000) 2022-09-08T05:33:08.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:07 smithi055 ceph-mon[97033]: pgmap v15378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:33:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:07 smithi059 ceph-mon[97450]: pgmap v15378: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:33:08.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:08.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:08.025426+0000) 2022-09-08T05:33:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:08.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:08.025257+0000) 2022-09-08T05:33:09.410 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:09.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:09.025441+0000) 2022-09-08T05:33:09.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:09.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:09.025626+0000) 2022-09-08T05:33:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:09 smithi055 ceph-mon[97033]: pgmap v15379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:33:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:09 smithi055 ceph-mon[97033]: osdmap e1041: 12 total, 11 up, 11 in 2022-09-08T05:33:10.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:09 smithi059 ceph-mon[97450]: pgmap v15379: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:33:10.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:09 smithi059 ceph-mon[97450]: osdmap e1041: 12 total, 11 up, 11 in 2022-09-08T05:33:10.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:10.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:10.025792+0000) 2022-09-08T05:33:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:10.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:10.025650+0000) 2022-09-08T05:33:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:11.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:11.025820+0000) 2022-09-08T05:33:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:11.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:11.025941+0000) 2022-09-08T05:33:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:11 smithi055 ceph-mon[97033]: pgmap v15381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:33:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:11 smithi059 ceph-mon[97450]: pgmap v15381: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:33:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:12.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:12.026038+0000) 2022-09-08T05:33:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:12.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:12.026182+0000) 2022-09-08T05:33:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:13.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:13.026263+0000) 2022-09-08T05:33:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:13.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:13.026357+0000) 2022-09-08T05:33:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:13 smithi055 ceph-mon[97033]: pgmap v15382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:33:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:13 smithi059 ceph-mon[97450]: pgmap v15382: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:33:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:14.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:14.026558+0000) 2022-09-08T05:33:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:14.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:14.026569+0000) 2022-09-08T05:33:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:15.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:15.026774+0000) 2022-09-08T05:33:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:15.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:15.026761+0000) 2022-09-08T05:33:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:15 smithi055 ceph-mon[97033]: pgmap v15383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:33:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:15 smithi055 ceph-mon[97033]: osdmap e1042: 12 total, 11 up, 11 in 2022-09-08T05:33:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:15 smithi059 ceph-mon[97450]: pgmap v15383: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:33:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:15 smithi059 ceph-mon[97450]: osdmap e1042: 12 total, 11 up, 11 in 2022-09-08T05:33:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:16.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:16.027023+0000) 2022-09-08T05:33:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:16.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:16.026999+0000) 2022-09-08T05:33:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:17.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:17.027220+0000) 2022-09-08T05:33:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:17.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:17.027173+0000) 2022-09-08T05:33:18.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:17 smithi055 ceph-mon[97033]: pgmap v15385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:33:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:17 smithi055 ceph-mon[97033]: pgmap v15386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-08T05:33:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:17 smithi055 ceph-mon[97033]: pgmap v15387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T05:33:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:33:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:17 smithi059 ceph-mon[97450]: pgmap v15385: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:33:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:17 smithi059 ceph-mon[97450]: pgmap v15386: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-08T05:33:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:17 smithi059 ceph-mon[97450]: pgmap v15387: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T05:33:18.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:33:18.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:33:18.265+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:33:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:18.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:18.027428+0000) 2022-09-08T05:33:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:18.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:18.027395+0000) 2022-09-08T05:33:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:19.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:19.027685+0000) 2022-09-08T05:33:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:19.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:19.027683+0000) 2022-09-08T05:33:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:19 smithi059 ceph-mon[97450]: pgmap v15388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T05:33:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:20.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:20.027912+0000) 2022-09-08T05:33:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:20.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:20.027913+0000) 2022-09-08T05:33:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:19 smithi055 ceph-mon[97033]: pgmap v15388: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T05:33:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:21.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:21.028155+0000) 2022-09-08T05:33:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:21.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:21.028148+0000) 2022-09-08T05:33:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:21 smithi055 ceph-mon[97033]: pgmap v15389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.0 KiB/s rd, 163 B/s wr, 9 op/s 2022-09-08T05:33:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:21 smithi059 ceph-mon[97450]: pgmap v15389: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5.0 KiB/s rd, 163 B/s wr, 9 op/s 2022-09-08T05:33:22.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:22.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:22.028406+0000) 2022-09-08T05:33:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:22.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:22.028369+0000) 2022-09-08T05:33:23.408 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:23.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:23.028618+0000) 2022-09-08T05:33:23.409 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:23.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:23.028612+0000) 2022-09-08T05:33:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:24 smithi059 ceph-mon[97450]: pgmap v15390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.1 KiB/s rd, 4 op/s 2022-09-08T05:33:24.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:24.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:24.028798+0000) 2022-09-08T05:33:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:24.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:24.028778+0000) 2022-09-08T05:33:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:24 smithi055 ceph-mon[97033]: pgmap v15390: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.1 KiB/s rd, 4 op/s 2022-09-08T05:33:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:25.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:25.029037+0000) 2022-09-08T05:33:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:25.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:25.029028+0000) 2022-09-08T05:33:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:26 smithi059 ceph-mon[97450]: pgmap v15391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:33:26.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:26.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:26.029250+0000) 2022-09-08T05:33:26.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:26.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:26.029251+0000) 2022-09-08T05:33:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:26 smithi055 ceph-mon[97033]: pgmap v15391: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:33:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:27.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:27.029524+0000) 2022-09-08T05:33:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:27.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:27.029508+0000) 2022-09-08T05:33:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:28 smithi059 ceph-mon[97450]: pgmap v15392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:33:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:28.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:28.029795+0000) 2022-09-08T05:33:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:28.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:28.029764+0000) 2022-09-08T05:33:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:28 smithi055 ceph-mon[97033]: pgmap v15392: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:33:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:29.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:29.029990+0000) 2022-09-08T05:33:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:29.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:29.030005+0000) 2022-09-08T05:33:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:30.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:30.030246+0000) 2022-09-08T05:33:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:30.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:30.030272+0000) 2022-09-08T05:33:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:30 smithi055 ceph-mon[97033]: pgmap v15393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:30 smithi059 ceph-mon[97450]: pgmap v15393: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:31.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:31.030445+0000) 2022-09-08T05:33:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:31.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:31.030471+0000) 2022-09-08T05:33:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:31 smithi055 ceph-mon[97033]: pgmap v15394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:33:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:31 smithi059 ceph-mon[97450]: pgmap v15394: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:33:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:32.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:32.030687+0000) 2022-09-08T05:33:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:32.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:32.030689+0000) 2022-09-08T05:33:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:33.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:33.030874+0000) 2022-09-08T05:33:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:33.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:33.030897+0000) 2022-09-08T05:33:33.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:33:33.279+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:33:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:34 smithi059 ceph-mon[97450]: pgmap v15395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:34.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:34.031110+0000) 2022-09-08T05:33:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:34.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:34.031119+0000) 2022-09-08T05:33:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:34 smithi055 ceph-mon[97033]: pgmap v15395: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:35.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:35.031375+0000) 2022-09-08T05:33:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:35.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:35.031373+0000) 2022-09-08T05:33:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:36 smithi059 ceph-mon[97450]: pgmap v15396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:33:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:36.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:36.031545+0000) 2022-09-08T05:33:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:36.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:36.031605+0000) 2022-09-08T05:33:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:36 smithi055 ceph-mon[97033]: pgmap v15396: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:33:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:37.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:37.031709+0000) 2022-09-08T05:33:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:37.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:37.031802+0000) 2022-09-08T05:33:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:38 smithi059 ceph-mon[97450]: pgmap v15397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:38.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:38.031834+0000) 2022-09-08T05:33:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:38.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:38.031955+0000) 2022-09-08T05:33:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:38 smithi055 ceph-mon[97033]: pgmap v15397: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:39.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:39.032057+0000) 2022-09-08T05:33:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:39.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:39.032104+0000) 2022-09-08T05:33:40.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:40.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:40.032351+0000) 2022-09-08T05:33:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:40.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:40.032330+0000) 2022-09-08T05:33:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:40 smithi055 ceph-mon[97033]: pgmap v15398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:40 smithi059 ceph-mon[97450]: pgmap v15398: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:41.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:41.032602+0000) 2022-09-08T05:33:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:41.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:41.032624+0000) 2022-09-08T05:33:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:41 smithi055 ceph-mon[97033]: pgmap v15399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:33:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:41 smithi059 ceph-mon[97450]: pgmap v15399: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:33:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:42.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:42.032882+0000) 2022-09-08T05:33:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:42.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:42.032850+0000) 2022-09-08T05:33:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:43.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:43.033121+0000) 2022-09-08T05:33:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:43.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:43.033156+0000) 2022-09-08T05:33:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:44 smithi059 ceph-mon[97450]: pgmap v15400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:44.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:44.033296+0000) 2022-09-08T05:33:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:44.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:44.033349+0000) 2022-09-08T05:33:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:44 smithi055 ceph-mon[97033]: pgmap v15400: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:45.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:45.033449+0000) 2022-09-08T05:33:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:45.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:45.033496+0000) 2022-09-08T05:33:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:46.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:46.033751+0000) 2022-09-08T05:33:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:46.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:46.033741+0000) 2022-09-08T05:33:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:46 smithi055 ceph-mon[97033]: pgmap v15401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:33:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:46 smithi059 ceph-mon[97450]: pgmap v15401: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:33:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:47.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:47.033956+0000) 2022-09-08T05:33:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:47.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:47.033945+0000) 2022-09-08T05:33:48.401 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:33:48.280+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:33:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:48.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:48.034186+0000) 2022-09-08T05:33:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:48.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:48.034182+0000) 2022-09-08T05:33:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:48 smithi055 ceph-mon[97033]: pgmap v15402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:48 smithi059 ceph-mon[97450]: pgmap v15402: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:49.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:49.034464+0000) 2022-09-08T05:33:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:49.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:49.034456+0000) 2022-09-08T05:33:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:50.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:50.034706+0000) 2022-09-08T05:33:50.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:50.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:50.034700+0000) 2022-09-08T05:33:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:50 smithi055 ceph-mon[97033]: pgmap v15403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:50.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:50 smithi059 ceph-mon[97450]: pgmap v15403: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:51.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:51.034988+0000) 2022-09-08T05:33:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:51.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:51.034961+0000) 2022-09-08T05:33:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:51 smithi055 ceph-mon[97033]: pgmap v15404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:33:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:51 smithi059 ceph-mon[97450]: pgmap v15404: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:33:52.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:52.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:52.035193+0000) 2022-09-08T05:33:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:52.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:52.035247+0000) 2022-09-08T05:33:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:53.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:53.035459+0000) 2022-09-08T05:33:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:53.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:53.035465+0000) 2022-09-08T05:33:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:54.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:54.035687+0000) 2022-09-08T05:33:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:54.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:54.035710+0000) 2022-09-08T05:33:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:54 smithi055 ceph-mon[97033]: pgmap v15405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:54 smithi059 ceph-mon[97450]: pgmap v15405: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:55.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:55.035940+0000) 2022-09-08T05:33:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:55.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:55.035938+0000) 2022-09-08T05:33:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:56.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:56.036198+0000) 2022-09-08T05:33:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:56.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:56.036209+0000) 2022-09-08T05:33:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:56 smithi055 ceph-mon[97033]: pgmap v15406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:33:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:56 smithi059 ceph-mon[97450]: pgmap v15406: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:33:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:57.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:57.036469+0000) 2022-09-08T05:33:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:57.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:57.036440+0000) 2022-09-08T05:33:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:58.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:58.036655+0000) 2022-09-08T05:33:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:58.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:58.036594+0000) 2022-09-08T05:33:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:33:58 smithi055 ceph-mon[97033]: pgmap v15407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:33:58 smithi059 ceph-mon[97450]: pgmap v15407: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:33:59.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:33:59.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:59.036820+0000) 2022-09-08T05:33:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:33:59.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:33:59.036863+0000) 2022-09-08T05:34:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:00.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:00.037109+0000) 2022-09-08T05:34:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:00.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:00.037055+0000) 2022-09-08T05:34:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:00 smithi055 ceph-mon[97033]: pgmap v15408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:34:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:00 smithi059 ceph-mon[97450]: pgmap v15408: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:34:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:01.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:01.037186+0000) 2022-09-08T05:34:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:01.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:01.037266+0000) 2022-09-08T05:34:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:01 smithi055 ceph-mon[97033]: pgmap v15409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:34:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:01 smithi059 ceph-mon[97450]: pgmap v15409: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:34:02.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:02.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:02.037419+0000) 2022-09-08T05:34:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:02.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:02.037483+0000) 2022-09-08T05:34:03.289 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:03.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:03.037734+0000) 2022-09-08T05:34:03.290 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:03.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:03.037684+0000) 2022-09-08T05:34:03.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:34:03.288+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:34:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:04.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:04.037951+0000) 2022-09-08T05:34:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:04.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:04.037939+0000) 2022-09-08T05:34:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:04 smithi055 ceph-mon[97033]: pgmap v15410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:34:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:04 smithi059 ceph-mon[97450]: pgmap v15410: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:34:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:05.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:05.038155+0000) 2022-09-08T05:34:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:05.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:05.038146+0000) 2022-09-08T05:34:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:06.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:06.038365+0000) 2022-09-08T05:34:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:06.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:06.038373+0000) 2022-09-08T05:34:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:06 smithi055 ceph-mon[97033]: pgmap v15411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:34:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:34:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:06 smithi059 ceph-mon[97450]: pgmap v15411: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:34:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:34:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:07.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:07.038695+0000) 2022-09-08T05:34:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:07.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:07.038602+0000) 2022-09-08T05:34:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:34:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:34:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:08.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:08.038899+0000) 2022-09-08T05:34:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:08.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:08.038850+0000) 2022-09-08T05:34:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:08 smithi055 ceph-mon[97033]: pgmap v15412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:34:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:08 smithi059 ceph-mon[97450]: pgmap v15412: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:34:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:09.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:09.039061+0000) 2022-09-08T05:34:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:09.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:09.039026+0000) 2022-09-08T05:34:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:10.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:10.039260+0000) 2022-09-08T05:34:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:10.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:10.039264+0000) 2022-09-08T05:34:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:10 smithi055 ceph-mon[97033]: pgmap v15413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:34:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:10 smithi055 ceph-mon[97033]: osdmap e1043: 12 total, 11 up, 11 in 2022-09-08T05:34:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:10 smithi059 ceph-mon[97450]: pgmap v15413: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:34:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:10 smithi059 ceph-mon[97450]: osdmap e1043: 12 total, 11 up, 11 in 2022-09-08T05:34:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:11.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:11.039510+0000) 2022-09-08T05:34:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:11.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:11.039512+0000) 2022-09-08T05:34:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:11 smithi055 ceph-mon[97033]: pgmap v15415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:34:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:11 smithi059 ceph-mon[97450]: pgmap v15415: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:34:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:12.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:12.039801+0000) 2022-09-08T05:34:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:12.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:12.039802+0000) 2022-09-08T05:34:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:13.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:13.040057+0000) 2022-09-08T05:34:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:13.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:13.040107+0000) 2022-09-08T05:34:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:14.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:14.040315+0000) 2022-09-08T05:34:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:14.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:14.040351+0000) 2022-09-08T05:34:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:14 smithi055 ceph-mon[97033]: pgmap v15416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:34:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:14 smithi059 ceph-mon[97450]: pgmap v15416: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:34:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:15.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:15.040552+0000) 2022-09-08T05:34:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:15.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:15.040569+0000) 2022-09-08T05:34:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:16.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:16.040780+0000) 2022-09-08T05:34:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:16.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:16.040815+0000) 2022-09-08T05:34:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:16 smithi055 ceph-mon[97033]: pgmap v15417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:34:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:16 smithi055 ceph-mon[97033]: osdmap e1044: 12 total, 11 up, 11 in 2022-09-08T05:34:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:16 smithi059 ceph-mon[97450]: pgmap v15417: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:34:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:16 smithi059 ceph-mon[97450]: osdmap e1044: 12 total, 11 up, 11 in 2022-09-08T05:34:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:17.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:17.040974+0000) 2022-09-08T05:34:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:17.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:17.041040+0000) 2022-09-08T05:34:18.299 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:18.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:18.041230+0000) 2022-09-08T05:34:18.300 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:18.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:18.041257+0000) 2022-09-08T05:34:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:18 smithi055 ceph-mon[97033]: pgmap v15419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:34:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:34:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:34:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:34:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:18 smithi059 ceph-mon[97450]: pgmap v15419: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:34:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:34:18.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:34:18.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:34:18.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:34:18.298+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:34:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:19.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:19.041500+0000) 2022-09-08T05:34:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:19.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:19.041500+0000) 2022-09-08T05:34:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:20.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:20.041810+0000) 2022-09-08T05:34:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:20.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:20.041808+0000) 2022-09-08T05:34:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:20 smithi055 ceph-mon[97033]: pgmap v15420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T05:34:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:20 smithi059 ceph-mon[97450]: pgmap v15420: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T05:34:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:21.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:21.042102+0000) 2022-09-08T05:34:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:21.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:21.042086+0000) 2022-09-08T05:34:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:21 smithi055 ceph-mon[97033]: pgmap v15421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:34:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:21 smithi059 ceph-mon[97450]: pgmap v15421: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:34:22.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:22.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:22.042275+0000) 2022-09-08T05:34:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:22.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:22.042275+0000) 2022-09-08T05:34:23.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:23.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:23.042550+0000) 2022-09-08T05:34:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:23.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:23.042544+0000) 2022-09-08T05:34:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:24.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:24.042791+0000) 2022-09-08T05:34:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:24.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:24.042791+0000) 2022-09-08T05:34:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:24 smithi055 ceph-mon[97033]: pgmap v15422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:34:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:24 smithi059 ceph-mon[97450]: pgmap v15422: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:34:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:25.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:25.043040+0000) 2022-09-08T05:34:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:25.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:25.043037+0000) 2022-09-08T05:34:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:26.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:26.043278+0000) 2022-09-08T05:34:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:26.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:26.043276+0000) 2022-09-08T05:34:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:26 smithi055 ceph-mon[97033]: pgmap v15423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:34:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:26 smithi059 ceph-mon[97450]: pgmap v15423: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:34:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:27.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:27.043544+0000) 2022-09-08T05:34:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:27.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:27.043552+0000) 2022-09-08T05:34:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:28.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:28.043747+0000) 2022-09-08T05:34:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:28.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:28.043752+0000) 2022-09-08T05:34:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:28 smithi055 ceph-mon[97033]: pgmap v15424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:34:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:28 smithi059 ceph-mon[97450]: pgmap v15424: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:34:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:29.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:29.044018+0000) 2022-09-08T05:34:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:29.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:29.044018+0000) 2022-09-08T05:34:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:30.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:30.044157+0000) 2022-09-08T05:34:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:30.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:30.044194+0000) 2022-09-08T05:34:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:30 smithi055 ceph-mon[97033]: pgmap v15425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:34:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:30 smithi059 ceph-mon[97450]: pgmap v15425: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:34:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:31.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:31.044374+0000) 2022-09-08T05:34:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:31.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:31.044399+0000) 2022-09-08T05:34:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:31 smithi055 ceph-mon[97033]: pgmap v15426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:34:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:31 smithi059 ceph-mon[97450]: pgmap v15426: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:34:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:32.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:32.044631+0000) 2022-09-08T05:34:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:32.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:32.044631+0000) 2022-09-08T05:34:33.314 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:33.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:33.044864+0000) 2022-09-08T05:34:33.315 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:33.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:33.044863+0000) 2022-09-08T05:34:33.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:34:33.313+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:34:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:34.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:34.045030+0000) 2022-09-08T05:34:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:34.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:34.045034+0000) 2022-09-08T05:34:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:34 smithi055 ceph-mon[97033]: pgmap v15427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:34:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:34 smithi059 ceph-mon[97450]: pgmap v15427: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:34:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:35.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:35.045204+0000) 2022-09-08T05:34:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:35.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:35.045205+0000) 2022-09-08T05:34:36.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:36.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:36.045377+0000) 2022-09-08T05:34:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:36.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:36.045379+0000) 2022-09-08T05:34:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:36 smithi055 ceph-mon[97033]: pgmap v15428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:34:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:36 smithi059 ceph-mon[97450]: pgmap v15428: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:34:37.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:37.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:37.045549+0000) 2022-09-08T05:34:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:37.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:37.045585+0000) 2022-09-08T05:34:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:38.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:38.045754+0000) 2022-09-08T05:34:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:38.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:38.045693+0000) 2022-09-08T05:34:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:38 smithi055 ceph-mon[97033]: pgmap v15429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:34:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:38 smithi059 ceph-mon[97450]: pgmap v15429: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:34:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:39.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:39.045906+0000) 2022-09-08T05:34:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:39.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:39.045878+0000) 2022-09-08T05:34:40.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:40.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:40.046029+0000) 2022-09-08T05:34:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:40.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:40.046069+0000) 2022-09-08T05:34:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:40 smithi055 ceph-mon[97033]: pgmap v15430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:34:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:40 smithi059 ceph-mon[97450]: pgmap v15430: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:34:41.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:41.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:41.046181+0000) 2022-09-08T05:34:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:41.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:41.046202+0000) 2022-09-08T05:34:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:41 smithi055 ceph-mon[97033]: pgmap v15431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:34:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:41 smithi059 ceph-mon[97450]: pgmap v15431: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:34:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:42.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:42.046402+0000) 2022-09-08T05:34:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:42.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:42.046416+0000) 2022-09-08T05:34:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:43.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:43.046545+0000) 2022-09-08T05:34:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:43.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:43.046571+0000) 2022-09-08T05:34:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:44.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:44.046790+0000) 2022-09-08T05:34:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:44.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:44.046792+0000) 2022-09-08T05:34:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:44 smithi055 ceph-mon[97033]: pgmap v15432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:34:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:44 smithi059 ceph-mon[97450]: pgmap v15432: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:34:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:45.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:45.047054+0000) 2022-09-08T05:34:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:45.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:45.047043+0000) 2022-09-08T05:34:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:46.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:46.047237+0000) 2022-09-08T05:34:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:46.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:46.047243+0000) 2022-09-08T05:34:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:46 smithi055 ceph-mon[97033]: pgmap v15433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:34:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:46 smithi059 ceph-mon[97450]: pgmap v15433: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:34:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:47.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:47.047377+0000) 2022-09-08T05:34:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:47.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:47.047433+0000) 2022-09-08T05:34:48.326 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:48.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:48.047567+0000) 2022-09-08T05:34:48.327 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:48.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:48.047606+0000) 2022-09-08T05:34:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:48 smithi059 ceph-mon[97450]: pgmap v15434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:34:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:48 smithi059 ceph-mon[97450]: pgmap v15435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:34:48.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:34:48.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:34:48.324+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:34:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:48 smithi055 ceph-mon[97033]: pgmap v15434: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:34:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:48 smithi055 ceph-mon[97033]: pgmap v15435: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:34:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:34:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:49.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:49.047777+0000) 2022-09-08T05:34:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:49.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:49.047868+0000) 2022-09-08T05:34:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:50.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:50.047923+0000) 2022-09-08T05:34:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:50.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:50.048083+0000) 2022-09-08T05:34:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:50 smithi059 ceph-mon[97450]: pgmap v15436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:34:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:50 smithi055 ceph-mon[97033]: pgmap v15436: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:34:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:51.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:51.048094+0000) 2022-09-08T05:34:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:51.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:51.048281+0000) 2022-09-08T05:34:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:51 smithi055 ceph-mon[97033]: pgmap v15437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:34:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:51 smithi059 ceph-mon[97450]: pgmap v15437: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:34:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:52.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:52.048300+0000) 2022-09-08T05:34:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:52.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:52.048531+0000) 2022-09-08T05:34:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:53.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:53.048536+0000) 2022-09-08T05:34:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:53.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:53.048700+0000) 2022-09-08T05:34:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:54.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:54.048758+0000) 2022-09-08T05:34:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:54.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:54.048910+0000) 2022-09-08T05:34:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:54 smithi055 ceph-mon[97033]: pgmap v15438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:34:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:54 smithi059 ceph-mon[97450]: pgmap v15438: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:34:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:55.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:55.048979+0000) 2022-09-08T05:34:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:55.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:55.049104+0000) 2022-09-08T05:34:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:56.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:56.049185+0000) 2022-09-08T05:34:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:56.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:56.049314+0000) 2022-09-08T05:34:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:56 smithi055 ceph-mon[97033]: pgmap v15439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:34:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:56 smithi059 ceph-mon[97450]: pgmap v15439: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:34:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:57.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:57.049401+0000) 2022-09-08T05:34:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:57.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:57.049476+0000) 2022-09-08T05:34:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:58.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:58.049610+0000) 2022-09-08T05:34:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:58.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:58.049725+0000) 2022-09-08T05:34:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:34:58 smithi055 ceph-mon[97033]: pgmap v15440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:34:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:34:58 smithi059 ceph-mon[97450]: pgmap v15440: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:34:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:34:59.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:59.049773+0000) 2022-09-08T05:34:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:34:59.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:34:59.049914+0000) 2022-09-08T05:35:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:00.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:00.049924+0000) 2022-09-08T05:35:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:00.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:00.050079+0000) 2022-09-08T05:35:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:00 smithi055 ceph-mon[97033]: pgmap v15441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:00 smithi059 ceph-mon[97450]: pgmap v15441: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:01.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:01.050077+0000) 2022-09-08T05:35:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:01.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:01.050230+0000) 2022-09-08T05:35:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:01 smithi055 ceph-mon[97033]: pgmap v15442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:35:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:01 smithi059 ceph-mon[97450]: pgmap v15442: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:35:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:02.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:02.050226+0000) 2022-09-08T05:35:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:02.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:02.050387+0000) 2022-09-08T05:35:03.339 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:03.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:03.050368+0000) 2022-09-08T05:35:03.340 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:03.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:03.050536+0000) 2022-09-08T05:35:03.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:35:03.339+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:35:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:04.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:04.050525+0000) 2022-09-08T05:35:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:04.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:04.050694+0000) 2022-09-08T05:35:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:04 smithi055 ceph-mon[97033]: pgmap v15443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:04 smithi059 ceph-mon[97450]: pgmap v15443: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:05.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:05.050713+0000) 2022-09-08T05:35:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:05.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:05.050828+0000) 2022-09-08T05:35:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:06.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:06.050918+0000) 2022-09-08T05:35:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:06.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:06.051044+0000) 2022-09-08T05:35:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:06 smithi055 ceph-mon[97033]: pgmap v15444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:35:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:35:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:06 smithi059 ceph-mon[97450]: pgmap v15444: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:35:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:35:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:07.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:07.051057+0000) 2022-09-08T05:35:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:07.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:07.051208+0000) 2022-09-08T05:35:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:35:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:35:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:08.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:08.051224+0000) 2022-09-08T05:35:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:08.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:08.051436+0000) 2022-09-08T05:35:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:08 smithi055 ceph-mon[97033]: pgmap v15445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:35:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:08 smithi059 ceph-mon[97450]: pgmap v15445: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:35:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:09.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:09.051365+0000) 2022-09-08T05:35:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:09.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:09.051614+0000) 2022-09-08T05:35:10.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:10.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:10.051757+0000) 2022-09-08T05:35:10.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:10.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:10.051532+0000) 2022-09-08T05:35:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:10 smithi055 ceph-mon[97033]: pgmap v15446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:35:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:10 smithi055 ceph-mon[97033]: osdmap e1045: 12 total, 11 up, 11 in 2022-09-08T05:35:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:10 smithi059 ceph-mon[97450]: pgmap v15446: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:35:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:10 smithi059 ceph-mon[97450]: osdmap e1045: 12 total, 11 up, 11 in 2022-09-08T05:35:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:11.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:11.051743+0000) 2022-09-08T05:35:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:11.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:11.051887+0000) 2022-09-08T05:35:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:11 smithi055 ceph-mon[97033]: pgmap v15448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:35:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:11 smithi059 ceph-mon[97450]: pgmap v15448: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:35:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:12.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:12.051957+0000) 2022-09-08T05:35:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:12.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:12.052041+0000) 2022-09-08T05:35:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:13.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:13.052082+0000) 2022-09-08T05:35:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:13.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:13.052187+0000) 2022-09-08T05:35:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:14.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:14.052285+0000) 2022-09-08T05:35:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:14.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:14.052348+0000) 2022-09-08T05:35:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:14 smithi055 ceph-mon[97033]: pgmap v15449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:35:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:14 smithi059 ceph-mon[97450]: pgmap v15449: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:35:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:15.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:15.052501+0000) 2022-09-08T05:35:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:15.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:15.052542+0000) 2022-09-08T05:35:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:16.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:16.052653+0000) 2022-09-08T05:35:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:16.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:16.052697+0000) 2022-09-08T05:35:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:16 smithi055 ceph-mon[97033]: pgmap v15450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:35:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:16 smithi055 ceph-mon[97033]: osdmap e1046: 12 total, 11 up, 11 in 2022-09-08T05:35:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:16 smithi059 ceph-mon[97450]: pgmap v15450: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:35:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:16 smithi059 ceph-mon[97450]: osdmap e1046: 12 total, 11 up, 11 in 2022-09-08T05:35:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:17.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:17.052868+0000) 2022-09-08T05:35:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:17.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:17.052883+0000) 2022-09-08T05:35:18.344 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:18.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:18.053130+0000) 2022-09-08T05:35:18.345 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:18.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:18.053142+0000) 2022-09-08T05:35:18.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:35:18.342+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:35:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:18 smithi055 ceph-mon[97033]: pgmap v15452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:35:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:18 smithi059 ceph-mon[97450]: pgmap v15452: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:35:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:19.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:19.053388+0000) 2022-09-08T05:35:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:19.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:19.053389+0000) 2022-09-08T05:35:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:20.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:20.053675+0000) 2022-09-08T05:35:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:20.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:20.053665+0000) 2022-09-08T05:35:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:20 smithi055 ceph-mon[97033]: pgmap v15453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T05:35:20.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:20 smithi059 ceph-mon[97450]: pgmap v15453: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T05:35:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:21.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:21.053922+0000) 2022-09-08T05:35:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:21.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:21.053930+0000) 2022-09-08T05:35:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:21 smithi055 ceph-mon[97033]: pgmap v15454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:35:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:21 smithi059 ceph-mon[97450]: pgmap v15454: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:35:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:22.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:22.054118+0000) 2022-09-08T05:35:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:22.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:22.054132+0000) 2022-09-08T05:35:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:23.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:23.054289+0000) 2022-09-08T05:35:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:23.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:23.054287+0000) 2022-09-08T05:35:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:24.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:24.054538+0000) 2022-09-08T05:35:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:24.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:24.054538+0000) 2022-09-08T05:35:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:24 smithi055 ceph-mon[97033]: pgmap v15455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:35:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:24 smithi059 ceph-mon[97450]: pgmap v15455: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:35:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:25.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:25.054778+0000) 2022-09-08T05:35:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:25.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:25.054777+0000) 2022-09-08T05:35:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:26.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:26.054924+0000) 2022-09-08T05:35:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:26.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:26.054924+0000) 2022-09-08T05:35:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:26 smithi055 ceph-mon[97033]: pgmap v15456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:35:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:26 smithi059 ceph-mon[97450]: pgmap v15456: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:35:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:27.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:27.055064+0000) 2022-09-08T05:35:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:27.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:27.055101+0000) 2022-09-08T05:35:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:28.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:28.055292+0000) 2022-09-08T05:35:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:28.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:28.055306+0000) 2022-09-08T05:35:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:28 smithi055 ceph-mon[97033]: pgmap v15457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:35:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:28 smithi059 ceph-mon[97450]: pgmap v15457: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:35:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:29.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:29.055531+0000) 2022-09-08T05:35:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:29.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:29.055540+0000) 2022-09-08T05:35:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:30.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:30.055771+0000) 2022-09-08T05:35:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:30.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:30.055821+0000) 2022-09-08T05:35:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:30 smithi055 ceph-mon[97033]: pgmap v15458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:30 smithi059 ceph-mon[97450]: pgmap v15458: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:31.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:31.056004+0000) 2022-09-08T05:35:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:31.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:31.056019+0000) 2022-09-08T05:35:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:31 smithi055 ceph-mon[97033]: pgmap v15459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:35:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:31 smithi059 ceph-mon[97450]: pgmap v15459: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:35:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:32.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:32.056210+0000) 2022-09-08T05:35:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:32.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:32.056270+0000) 2022-09-08T05:35:33.359 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:33.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:33.056348+0000) 2022-09-08T05:35:33.360 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:33.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:33.056432+0000) 2022-09-08T05:35:33.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:35:33.359+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:35:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:34.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:34.056568+0000) 2022-09-08T05:35:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:34.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:34.056570+0000) 2022-09-08T05:35:34.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:34 smithi055 ceph-mon[97033]: pgmap v15460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:34 smithi059 ceph-mon[97450]: pgmap v15460: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:35.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:35.056836+0000) 2022-09-08T05:35:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:35.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:35.056835+0000) 2022-09-08T05:35:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:36.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:36.057033+0000) 2022-09-08T05:35:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:36.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:36.057034+0000) 2022-09-08T05:35:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:36 smithi055 ceph-mon[97033]: pgmap v15461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:35:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:36 smithi059 ceph-mon[97450]: pgmap v15461: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:35:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:37.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:37.057269+0000) 2022-09-08T05:35:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:37.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:37.057269+0000) 2022-09-08T05:35:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:38.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:38.057478+0000) 2022-09-08T05:35:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:38.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:38.057478+0000) 2022-09-08T05:35:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:38 smithi055 ceph-mon[97033]: pgmap v15462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:38 smithi059 ceph-mon[97450]: pgmap v15462: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:39.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:39.057736+0000) 2022-09-08T05:35:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:39.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:39.057735+0000) 2022-09-08T05:35:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:40.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:40.057879+0000) 2022-09-08T05:35:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:40.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:40.057897+0000) 2022-09-08T05:35:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:40 smithi055 ceph-mon[97033]: pgmap v15463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:40 smithi059 ceph-mon[97450]: pgmap v15463: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:41.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:41.058056+0000) 2022-09-08T05:35:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:41.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:41.058044+0000) 2022-09-08T05:35:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:41 smithi055 ceph-mon[97033]: pgmap v15464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:35:42.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:41 smithi059 ceph-mon[97450]: pgmap v15464: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:35:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:42.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:42.058258+0000) 2022-09-08T05:35:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:42.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:42.058262+0000) 2022-09-08T05:35:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:43.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:43.058387+0000) 2022-09-08T05:35:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:43.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:43.058379+0000) 2022-09-08T05:35:44.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:44.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:44.058593+0000) 2022-09-08T05:35:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:44.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:44.058587+0000) 2022-09-08T05:35:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:44 smithi055 ceph-mon[97033]: pgmap v15465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:44 smithi059 ceph-mon[97450]: pgmap v15465: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:45.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:45.058804+0000) 2022-09-08T05:35:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:45.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:45.058787+0000) 2022-09-08T05:35:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:46.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:46.059013+0000) 2022-09-08T05:35:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:46.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:46.059021+0000) 2022-09-08T05:35:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:46 smithi055 ceph-mon[97033]: pgmap v15466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:35:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:46 smithi059 ceph-mon[97450]: pgmap v15466: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:35:47.389 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:47.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:47.059225+0000) 2022-09-08T05:35:47.390 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:47.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:47.059249+0000) 2022-09-08T05:35:48.369 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:48.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:48.059496+0000) 2022-09-08T05:35:48.370 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:48.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:48.059498+0000) 2022-09-08T05:35:48.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:35:48.368+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:35:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:48 smithi055 ceph-mon[97033]: pgmap v15467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:48.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:35:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:35:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:35:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:48 smithi059 ceph-mon[97450]: pgmap v15467: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:35:48.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:35:48.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:35:49.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:49.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:49.059699+0000) 2022-09-08T05:35:49.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:49.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:49.059695+0000) 2022-09-08T05:35:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:50.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:50.059860+0000) 2022-09-08T05:35:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:50.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:50.059837+0000) 2022-09-08T05:35:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:50 smithi055 ceph-mon[97033]: pgmap v15468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:50 smithi059 ceph-mon[97450]: pgmap v15468: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:51.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:51.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:51.060071+0000) 2022-09-08T05:35:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:51.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:51.060033+0000) 2022-09-08T05:35:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:51 smithi055 ceph-mon[97033]: pgmap v15469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:35:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:51 smithi059 ceph-mon[97450]: pgmap v15469: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:35:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:52.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:52.060341+0000) 2022-09-08T05:35:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:52.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:52.060281+0000) 2022-09-08T05:35:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:53.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:53.060506+0000) 2022-09-08T05:35:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:53.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:53.060584+0000) 2022-09-08T05:35:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:54.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:54.060786+0000) 2022-09-08T05:35:54.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:54.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:54.060792+0000) 2022-09-08T05:35:54.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:54 smithi055 ceph-mon[97033]: pgmap v15470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:54 smithi059 ceph-mon[97450]: pgmap v15470: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:55.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:55.061038+0000) 2022-09-08T05:35:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:55.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:55.061038+0000) 2022-09-08T05:35:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:56.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:56.061250+0000) 2022-09-08T05:35:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:56.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:56.061254+0000) 2022-09-08T05:35:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:56 smithi055 ceph-mon[97033]: pgmap v15471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:35:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:56 smithi059 ceph-mon[97450]: pgmap v15471: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:35:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:57.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:57.061519+0000) 2022-09-08T05:35:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:57.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:57.061489+0000) 2022-09-08T05:35:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:58.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:58.061716+0000) 2022-09-08T05:35:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:58.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:58.061722+0000) 2022-09-08T05:35:58.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:35:58 smithi055 ceph-mon[97033]: pgmap v15472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:35:58 smithi059 ceph-mon[97450]: pgmap v15472: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:35:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:35:59.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:59.061842+0000) 2022-09-08T05:35:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:35:59.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:35:59.061905+0000) 2022-09-08T05:36:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:00.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:00.062075+0000) 2022-09-08T05:36:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:00.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:00.062175+0000) 2022-09-08T05:36:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:00 smithi055 ceph-mon[97033]: pgmap v15473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:00 smithi059 ceph-mon[97450]: pgmap v15473: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:01.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:01.062285+0000) 2022-09-08T05:36:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:01.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:01.062401+0000) 2022-09-08T05:36:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:01 smithi055 ceph-mon[97033]: pgmap v15474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:36:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:01 smithi059 ceph-mon[97450]: pgmap v15474: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:36:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:02.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:02.062424+0000) 2022-09-08T05:36:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:02.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:02.062582+0000) 2022-09-08T05:36:03.373 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:03.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:03.062608+0000) 2022-09-08T05:36:03.374 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:03.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:03.062713+0000) 2022-09-08T05:36:03.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:36:03.371+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:36:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:04.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:04.062692+0000) 2022-09-08T05:36:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:04.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:04.062846+0000) 2022-09-08T05:36:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:04 smithi055 ceph-mon[97033]: pgmap v15475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:04 smithi059 ceph-mon[97450]: pgmap v15475: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:05.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:05.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:05.063052+0000) 2022-09-08T05:36:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:05.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:05.062862+0000) 2022-09-08T05:36:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:06.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:06.063113+0000) 2022-09-08T05:36:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:06.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:06.063264+0000) 2022-09-08T05:36:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:06 smithi055 ceph-mon[97033]: pgmap v15476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:36:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:36:06.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:06 smithi059 ceph-mon[97450]: pgmap v15476: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:36:06.790 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:36:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:07.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:07.063295+0000) 2022-09-08T05:36:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:07.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:07.063412+0000) 2022-09-08T05:36:07.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:36:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:36:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:08.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:08.063524+0000) 2022-09-08T05:36:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:08.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:08.063625+0000) 2022-09-08T05:36:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:08 smithi055 ceph-mon[97033]: pgmap v15477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:36:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:08 smithi059 ceph-mon[97450]: pgmap v15477: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:36:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:09.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:09.063748+0000) 2022-09-08T05:36:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:09.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:09.063846+0000) 2022-09-08T05:36:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:10.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:10.063916+0000) 2022-09-08T05:36:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:10.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:10.064028+0000) 2022-09-08T05:36:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:10 smithi055 ceph-mon[97033]: pgmap v15478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:36:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:10 smithi055 ceph-mon[97033]: osdmap e1047: 12 total, 11 up, 11 in 2022-09-08T05:36:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:10 smithi059 ceph-mon[97450]: pgmap v15478: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:36:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:10 smithi059 ceph-mon[97450]: osdmap e1047: 12 total, 11 up, 11 in 2022-09-08T05:36:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:11.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:11.064141+0000) 2022-09-08T05:36:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:11.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:11.064265+0000) 2022-09-08T05:36:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:11 smithi055 ceph-mon[97033]: pgmap v15480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:36:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:11 smithi059 ceph-mon[97450]: pgmap v15480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:36:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:12.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:12.064391+0000) 2022-09-08T05:36:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:12.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:12.064492+0000) 2022-09-08T05:36:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:13.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:13.064608+0000) 2022-09-08T05:36:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:13.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:13.064705+0000) 2022-09-08T05:36:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:14.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:14.064806+0000) 2022-09-08T05:36:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:14.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:14.064943+0000) 2022-09-08T05:36:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:14 smithi055 ceph-mon[97033]: pgmap v15481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:36:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:14 smithi059 ceph-mon[97450]: pgmap v15481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:36:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:15.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:15.065087+0000) 2022-09-08T05:36:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:15.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:15.065194+0000) 2022-09-08T05:36:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:16.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:16.065373+0000) 2022-09-08T05:36:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:16.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:16.065452+0000) 2022-09-08T05:36:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:16 smithi055 ceph-mon[97033]: pgmap v15482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:36:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:16 smithi055 ceph-mon[97033]: osdmap e1048: 12 total, 11 up, 11 in 2022-09-08T05:36:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:16 smithi059 ceph-mon[97450]: pgmap v15482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:36:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:16 smithi059 ceph-mon[97450]: osdmap e1048: 12 total, 11 up, 11 in 2022-09-08T05:36:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:17.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:17.065534+0000) 2022-09-08T05:36:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:17.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:17.065650+0000) 2022-09-08T05:36:18.378 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:18.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:18.065703+0000) 2022-09-08T05:36:18.379 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:18.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:18.065796+0000) 2022-09-08T05:36:18.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:36:18.377+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:36:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:18 smithi055 ceph-mon[97033]: pgmap v15484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:36:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:18 smithi059 ceph-mon[97450]: pgmap v15484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:36:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:19.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:19.065940+0000) 2022-09-08T05:36:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:19.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:19.066009+0000) 2022-09-08T05:36:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:20.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:20.066230+0000) 2022-09-08T05:36:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:20.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:20.066229+0000) 2022-09-08T05:36:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:20 smithi055 ceph-mon[97033]: pgmap v15485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T05:36:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:20 smithi059 ceph-mon[97450]: pgmap v15485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T05:36:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:21.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:21.066418+0000) 2022-09-08T05:36:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:21.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:21.066418+0000) 2022-09-08T05:36:22.115 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:21 smithi059 ceph-mon[97450]: pgmap v15486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:36:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:21 smithi055 ceph-mon[97033]: pgmap v15486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:36:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:22.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:22.066709+0000) 2022-09-08T05:36:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:22.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:22.066703+0000) 2022-09-08T05:36:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:36:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:36:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:23 smithi059 ceph-mon[97450]: pgmap v15487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 117 B/s wr, 7 op/s 2022-09-08T05:36:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:23 smithi059 ceph-mon[97450]: pgmap v15488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-09-08T05:36:23.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:36:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:23.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:23.066958+0000) 2022-09-08T05:36:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:23.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:23.066953+0000) 2022-09-08T05:36:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:36:23.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:36:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:23 smithi055 ceph-mon[97033]: pgmap v15487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 117 B/s wr, 7 op/s 2022-09-08T05:36:23.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:23 smithi055 ceph-mon[97033]: pgmap v15488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-09-08T05:36:23.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:36:24.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:24.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:24.067245+0000) 2022-09-08T05:36:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:24.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:24.067243+0000) 2022-09-08T05:36:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:25 smithi059 ceph-mon[97450]: pgmap v15489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:36:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:25.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:25.067427+0000) 2022-09-08T05:36:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:25.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:25.067427+0000) 2022-09-08T05:36:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:25 smithi055 ceph-mon[97033]: pgmap v15489: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:36:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:26.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:26.067742+0000) 2022-09-08T05:36:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:26.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:26.067746+0000) 2022-09-08T05:36:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:26 smithi055 ceph-mon[97033]: pgmap v15490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T05:36:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:26 smithi059 ceph-mon[97450]: pgmap v15490: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T05:36:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:27.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:27.067969+0000) 2022-09-08T05:36:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:27.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:27.067969+0000) 2022-09-08T05:36:28.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:28.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:28.068122+0000) 2022-09-08T05:36:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:28.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:28.068102+0000) 2022-09-08T05:36:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:29.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:29.068290+0000) 2022-09-08T05:36:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:29.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:29.068293+0000) 2022-09-08T05:36:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:29 smithi055 ceph-mon[97033]: pgmap v15491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T05:36:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:29 smithi059 ceph-mon[97450]: pgmap v15491: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T05:36:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:30.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:30.068510+0000) 2022-09-08T05:36:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:30.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:30.068523+0000) 2022-09-08T05:36:31.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:31.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:31.068759+0000) 2022-09-08T05:36:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:31.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:31.068755+0000) 2022-09-08T05:36:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:31 smithi055 ceph-mon[97033]: pgmap v15492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T05:36:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:31 smithi059 ceph-mon[97450]: pgmap v15492: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T05:36:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:32.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:32.069018+0000) 2022-09-08T05:36:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:32.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:32.069005+0000) 2022-09-08T05:36:33.386 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:33.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:33.069281+0000) 2022-09-08T05:36:33.387 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:33.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:33.069274+0000) 2022-09-08T05:36:33.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:33 smithi055 ceph-mon[97033]: pgmap v15493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:36:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:33 smithi059 ceph-mon[97450]: pgmap v15493: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:36:33.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:36:33.383+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:36:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:34.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:34.069523+0000) 2022-09-08T05:36:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:34.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:34.069522+0000) 2022-09-08T05:36:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:35.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:35.069812+0000) 2022-09-08T05:36:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:35.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:35.069793+0000) 2022-09-08T05:36:35.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:35 smithi055 ceph-mon[97033]: pgmap v15494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:35 smithi059 ceph-mon[97450]: pgmap v15494: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:36.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:36.070042+0000) 2022-09-08T05:36:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:36.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:36.070064+0000) 2022-09-08T05:36:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:36 smithi055 ceph-mon[97033]: pgmap v15495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:36:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:36 smithi059 ceph-mon[97450]: pgmap v15495: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:36:37.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:37.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:37.070322+0000) 2022-09-08T05:36:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:37.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:37.070332+0000) 2022-09-08T05:36:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:38.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:38.070564+0000) 2022-09-08T05:36:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:38.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:38.070598+0000) 2022-09-08T05:36:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:39.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:39.070813+0000) 2022-09-08T05:36:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:39.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:39.070764+0000) 2022-09-08T05:36:39.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:39 smithi055 ceph-mon[97033]: pgmap v15496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:39 smithi059 ceph-mon[97450]: pgmap v15496: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:40.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:40.070964+0000) 2022-09-08T05:36:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:40.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:40.070896+0000) 2022-09-08T05:36:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:41.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:41.071112+0000) 2022-09-08T05:36:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:41.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:41.071057+0000) 2022-09-08T05:36:41.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:41 smithi055 ceph-mon[97033]: pgmap v15497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:36:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:41 smithi059 ceph-mon[97450]: pgmap v15497: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:36:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:42.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:42.071286+0000) 2022-09-08T05:36:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:42.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:42.071270+0000) 2022-09-08T05:36:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:43.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:43.071430+0000) 2022-09-08T05:36:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:43.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:43.071445+0000) 2022-09-08T05:36:43.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:43 smithi055 ceph-mon[97033]: pgmap v15498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:43 smithi059 ceph-mon[97450]: pgmap v15498: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:44.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:44.071670+0000) 2022-09-08T05:36:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:44.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:44.071676+0000) 2022-09-08T05:36:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:45.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:45.071842+0000) 2022-09-08T05:36:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:45.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:45.071811+0000) 2022-09-08T05:36:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:45 smithi055 ceph-mon[97033]: pgmap v15499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:45 smithi059 ceph-mon[97450]: pgmap v15499: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:46.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:46.072051+0000) 2022-09-08T05:36:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:46.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:46.072023+0000) 2022-09-08T05:36:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:46 smithi055 ceph-mon[97033]: pgmap v15500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:36:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:46 smithi059 ceph-mon[97450]: pgmap v15500: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:36:47.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:47.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:47.072318+0000) 2022-09-08T05:36:47.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:47.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:47.072262+0000) 2022-09-08T05:36:48.387 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:48.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:48.072526+0000) 2022-09-08T05:36:48.388 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:48.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:48.072534+0000) 2022-09-08T05:36:48.651 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:36:48.385+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:36:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:49.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:49.072756+0000) 2022-09-08T05:36:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:49.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:49.072765+0000) 2022-09-08T05:36:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:49 smithi055 ceph-mon[97033]: pgmap v15501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:49 smithi059 ceph-mon[97450]: pgmap v15501: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:50.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:50.072895+0000) 2022-09-08T05:36:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:50.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:50.072946+0000) 2022-09-08T05:36:51.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:51.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:51.073127+0000) 2022-09-08T05:36:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:51.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:51.073111+0000) 2022-09-08T05:36:51.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:51 smithi055 ceph-mon[97033]: pgmap v15502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:36:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:51 smithi059 ceph-mon[97450]: pgmap v15502: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:36:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:52.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:52.073244+0000) 2022-09-08T05:36:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:52.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:52.073261+0000) 2022-09-08T05:36:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:53.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:53.073391+0000) 2022-09-08T05:36:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:53.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:53.073415+0000) 2022-09-08T05:36:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:53 smithi055 ceph-mon[97033]: pgmap v15503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:53 smithi059 ceph-mon[97450]: pgmap v15503: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:54.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:54.073620+0000) 2022-09-08T05:36:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:54.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:54.073626+0000) 2022-09-08T05:36:55.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:55.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:55.073871+0000) 2022-09-08T05:36:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:55.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:55.073875+0000) 2022-09-08T05:36:55.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:55 smithi055 ceph-mon[97033]: pgmap v15504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:55 smithi059 ceph-mon[97450]: pgmap v15504: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:56.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:56.074069+0000) 2022-09-08T05:36:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:56.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:56.074075+0000) 2022-09-08T05:36:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:56 smithi055 ceph-mon[97033]: pgmap v15505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:36:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:56 smithi059 ceph-mon[97450]: pgmap v15505: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:36:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:57.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:57.074311+0000) 2022-09-08T05:36:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:57.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:57.074325+0000) 2022-09-08T05:36:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:58.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:58.074570+0000) 2022-09-08T05:36:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:58.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:58.074583+0000) 2022-09-08T05:36:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:36:59.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:59.074916+0000) 2022-09-08T05:36:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:36:59.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:36:59.074782+0000) 2022-09-08T05:36:59.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:36:59 smithi055 ceph-mon[97033]: pgmap v15506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:36:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:36:59 smithi059 ceph-mon[97450]: pgmap v15506: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:00.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:00.075135+0000) 2022-09-08T05:37:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:00.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:00.075038+0000) 2022-09-08T05:37:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:01.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:01.075291+0000) 2022-09-08T05:37:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:01.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:01.075210+0000) 2022-09-08T05:37:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:01 smithi055 ceph-mon[97033]: pgmap v15507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:37:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:01 smithi059 ceph-mon[97450]: pgmap v15507: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:37:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:02.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:02.075467+0000) 2022-09-08T05:37:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:02.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:02.075413+0000) 2022-09-08T05:37:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:03.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:03.075653+0000) 2022-09-08T05:37:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:03.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:03.075567+0000) 2022-09-08T05:37:03.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:03 smithi055 ceph-mon[97033]: pgmap v15508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:03 smithi059 ceph-mon[97450]: pgmap v15508: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:03.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:37:03.401+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:37:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:04.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:04.075856+0000) 2022-09-08T05:37:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:04.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:04.075813+0000) 2022-09-08T05:37:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:05.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:05.076049+0000) 2022-09-08T05:37:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:05.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:05.076029+0000) 2022-09-08T05:37:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:05 smithi055 ceph-mon[97033]: pgmap v15509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:05 smithi059 ceph-mon[97450]: pgmap v15509: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:06.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:06.076269+0000) 2022-09-08T05:37:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:06.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:06.076262+0000) 2022-09-08T05:37:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:37:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:37:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:07.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:07.076506+0000) 2022-09-08T05:37:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:07.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:07.076493+0000) 2022-09-08T05:37:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:07 smithi055 ceph-mon[97033]: pgmap v15510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:37:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:37:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:07 smithi059 ceph-mon[97450]: pgmap v15510: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:37:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:37:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:08.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:08.076696+0000) 2022-09-08T05:37:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:08.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:08.076695+0000) 2022-09-08T05:37:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:09.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:09.076841+0000) 2022-09-08T05:37:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:09.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:09.076845+0000) 2022-09-08T05:37:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:09 smithi055 ceph-mon[97033]: pgmap v15511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:37:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:09 smithi059 ceph-mon[97450]: pgmap v15511: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:37:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:10.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:10.077080+0000) 2022-09-08T05:37:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:10.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:10.077072+0000) 2022-09-08T05:37:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:10 smithi055 ceph-mon[97033]: osdmap e1049: 12 total, 11 up, 11 in 2022-09-08T05:37:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:10 smithi059 ceph-mon[97450]: osdmap e1049: 12 total, 11 up, 11 in 2022-09-08T05:37:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:11.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:11.077354+0000) 2022-09-08T05:37:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:11.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:11.077309+0000) 2022-09-08T05:37:11.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:11 smithi055 ceph-mon[97033]: pgmap v15513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:37:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:11 smithi055 ceph-mon[97033]: osdmap e1050: 12 total, 11 up, 11 in 2022-09-08T05:37:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:11 smithi059 ceph-mon[97450]: pgmap v15513: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:37:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:11 smithi059 ceph-mon[97450]: osdmap e1050: 12 total, 11 up, 11 in 2022-09-08T05:37:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:12.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:12.077571+0000) 2022-09-08T05:37:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:12.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:12.077534+0000) 2022-09-08T05:37:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:13.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:13.077816+0000) 2022-09-08T05:37:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:13.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:13.077800+0000) 2022-09-08T05:37:13.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:13 smithi055 ceph-mon[97033]: pgmap v15515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:37:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:13 smithi059 ceph-mon[97450]: pgmap v15515: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:37:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:14.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:14.078046+0000) 2022-09-08T05:37:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:14.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:14.078058+0000) 2022-09-08T05:37:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:15.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:15.078273+0000) 2022-09-08T05:37:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:15.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:15.078293+0000) 2022-09-08T05:37:15.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:15 smithi055 ceph-mon[97033]: pgmap v15516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:37:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:15 smithi059 ceph-mon[97450]: pgmap v15516: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:37:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:16.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:16.078539+0000) 2022-09-08T05:37:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:16.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:16.078583+0000) 2022-09-08T05:37:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:16 smithi055 ceph-mon[97033]: pgmap v15517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:37:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:16 smithi059 ceph-mon[97450]: pgmap v15517: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:37:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:17.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:17.078691+0000) 2022-09-08T05:37:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:17.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:17.078714+0000) 2022-09-08T05:37:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:18.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:18.078933+0000) 2022-09-08T05:37:18.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:18.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:18.078961+0000) 2022-09-08T05:37:18.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:37:18.416+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:37:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:19.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:19.079196+0000) 2022-09-08T05:37:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:19.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:19.079224+0000) 2022-09-08T05:37:19.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:19 smithi055 ceph-mon[97033]: pgmap v15518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T05:37:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:19 smithi059 ceph-mon[97450]: pgmap v15518: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T05:37:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:20.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:20.079448+0000) 2022-09-08T05:37:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:20.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:20.079508+0000) 2022-09-08T05:37:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:21.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:21.079626+0000) 2022-09-08T05:37:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:21.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:21.079703+0000) 2022-09-08T05:37:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:21 smithi055 ceph-mon[97033]: pgmap v15519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:37:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:21 smithi059 ceph-mon[97450]: pgmap v15519: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:37:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:22.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:22.079850+0000) 2022-09-08T05:37:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:22.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:22.079879+0000) 2022-09-08T05:37:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:23.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:23.080039+0000) 2022-09-08T05:37:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:23.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:23.080061+0000) 2022-09-08T05:37:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:23 smithi055 ceph-mon[97033]: pgmap v15520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T05:37:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:37:23.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:37:23.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:37:23.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:23 smithi055 ceph-mon[97033]: pgmap v15521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T05:37:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:37:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:23 smithi059 ceph-mon[97450]: pgmap v15520: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T05:37:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:37:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:37:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:37:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:23 smithi059 ceph-mon[97450]: pgmap v15521: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T05:37:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:37:24.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:24.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:24.080291+0000) 2022-09-08T05:37:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:24.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:24.080279+0000) 2022-09-08T05:37:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:24 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T05:37:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:24 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T05:37:25.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:25.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:25.080528+0000) 2022-09-08T05:37:25.413 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:25.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:25.080517+0000) 2022-09-08T05:37:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:25 smithi055 ceph-mon[97033]: pgmap v15522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T05:37:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:25 smithi059 ceph-mon[97450]: pgmap v15522: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T05:37:26.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:26.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:26.080759+0000) 2022-09-08T05:37:26.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:26.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:26.080736+0000) 2022-09-08T05:37:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:26 smithi055 ceph-mon[97033]: pgmap v15523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:37:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:26 smithi059 ceph-mon[97450]: pgmap v15523: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:37:27.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:27.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:27.080980+0000) 2022-09-08T05:37:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:27.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:27.080979+0000) 2022-09-08T05:37:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:28.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:28.081117+0000) 2022-09-08T05:37:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:28.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:28.081151+0000) 2022-09-08T05:37:29.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:29.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:29.081333+0000) 2022-09-08T05:37:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:29.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:29.081295+0000) 2022-09-08T05:37:29.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:29 smithi055 ceph-mon[97033]: pgmap v15524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:37:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:29 smithi059 ceph-mon[97450]: pgmap v15524: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:37:30.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:30.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:30.081542+0000) 2022-09-08T05:37:30.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:30.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:30.081511+0000) 2022-09-08T05:37:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:31.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:31.081751+0000) 2022-09-08T05:37:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:31.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:31.081781+0000) 2022-09-08T05:37:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:31 smithi055 ceph-mon[97033]: pgmap v15525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:37:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:31 smithi059 ceph-mon[97450]: pgmap v15525: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:37:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:32.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:32.081983+0000) 2022-09-08T05:37:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:32.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:32.082044+0000) 2022-09-08T05:37:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:33.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:33.082168+0000) 2022-09-08T05:37:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:33.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:33.082228+0000) 2022-09-08T05:37:33.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:37:33.430+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:37:33.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:33 smithi055 ceph-mon[97033]: pgmap v15526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:37:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:33 smithi059 ceph-mon[97450]: pgmap v15526: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:37:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:34.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:34.082424+0000) 2022-09-08T05:37:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:34.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:34.082471+0000) 2022-09-08T05:37:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:35.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:35.082566+0000) 2022-09-08T05:37:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:35.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:35.082691+0000) 2022-09-08T05:37:35.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:35 smithi055 ceph-mon[97033]: pgmap v15527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:35 smithi059 ceph-mon[97450]: pgmap v15527: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:36.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:36.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:36.082907+0000) 2022-09-08T05:37:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:36.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:36.082805+0000) 2022-09-08T05:37:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:36 smithi055 ceph-mon[97033]: pgmap v15528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:37:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:36 smithi059 ceph-mon[97450]: pgmap v15528: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:37:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:37.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:37.083077+0000) 2022-09-08T05:37:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:37.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:37.083098+0000) 2022-09-08T05:37:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:38.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:38.083288+0000) 2022-09-08T05:37:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:38.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:38.083307+0000) 2022-09-08T05:37:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:39.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:39.083439+0000) 2022-09-08T05:37:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:39.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:39.083452+0000) 2022-09-08T05:37:39.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:39 smithi055 ceph-mon[97033]: pgmap v15529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:39 smithi059 ceph-mon[97450]: pgmap v15529: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:40.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:40.083707+0000) 2022-09-08T05:37:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:40.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:40.083739+0000) 2022-09-08T05:37:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:41.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:41.083880+0000) 2022-09-08T05:37:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:41.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:41.083944+0000) 2022-09-08T05:37:41.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:41 smithi055 ceph-mon[97033]: pgmap v15530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:37:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:41 smithi059 ceph-mon[97450]: pgmap v15530: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:37:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:42.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:42.084012+0000) 2022-09-08T05:37:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:42.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:42.084118+0000) 2022-09-08T05:37:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:43.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:43.084135+0000) 2022-09-08T05:37:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:43.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:43.084275+0000) 2022-09-08T05:37:43.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:43 smithi055 ceph-mon[97033]: pgmap v15531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:43 smithi059 ceph-mon[97450]: pgmap v15531: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:44.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:44.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:44.084521+0000) 2022-09-08T05:37:44.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:44.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:44.084324+0000) 2022-09-08T05:37:45.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:45.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:45.084718+0000) 2022-09-08T05:37:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:45.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:45.084542+0000) 2022-09-08T05:37:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:45 smithi059 ceph-mon[97450]: pgmap v15532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:45 smithi055 ceph-mon[97033]: pgmap v15532: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:46.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:46.084764+0000) 2022-09-08T05:37:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:46.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:46.084968+0000) 2022-09-08T05:37:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:46 smithi055 ceph-mon[97033]: pgmap v15533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:37:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:46 smithi059 ceph-mon[97450]: pgmap v15533: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:37:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:47.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:47.084990+0000) 2022-09-08T05:37:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:47.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:47.085200+0000) 2022-09-08T05:37:48.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:48.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:48.085369+0000) 2022-09-08T05:37:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:48.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:48.085123+0000) 2022-09-08T05:37:48.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:37:48.443+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:37:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:49.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:49.085312+0000) 2022-09-08T05:37:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:49.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:49.085515+0000) 2022-09-08T05:37:49.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:49 smithi055 ceph-mon[97033]: pgmap v15534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:49 smithi059 ceph-mon[97450]: pgmap v15534: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:50.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:50.085567+0000) 2022-09-08T05:37:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:50.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:50.085720+0000) 2022-09-08T05:37:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:51.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:51.085788+0000) 2022-09-08T05:37:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:51.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:51.085888+0000) 2022-09-08T05:37:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:51 smithi059 ceph-mon[97450]: pgmap v15535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:37:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:51 smithi055 ceph-mon[97033]: pgmap v15535: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:37:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:52.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:52.086005+0000) 2022-09-08T05:37:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:52.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:52.086127+0000) 2022-09-08T05:37:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:52 smithi055 ceph-mon[97033]: pgmap v15536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:52 smithi059 ceph-mon[97450]: pgmap v15536: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:53.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:53.086261+0000) 2022-09-08T05:37:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:53.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:53.086325+0000) 2022-09-08T05:37:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:54.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:54.086472+0000) 2022-09-08T05:37:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:54.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:54.086544+0000) 2022-09-08T05:37:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:55.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:55.086747+0000) 2022-09-08T05:37:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:55.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:55.086777+0000) 2022-09-08T05:37:55.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:55 smithi055 ceph-mon[97033]: pgmap v15537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:55 smithi059 ceph-mon[97450]: pgmap v15537: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:37:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:56.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:56.086981+0000) 2022-09-08T05:37:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:56.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:56.087002+0000) 2022-09-08T05:37:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:56 smithi055 ceph-mon[97033]: pgmap v15538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:37:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:56 smithi059 ceph-mon[97450]: pgmap v15538: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:37:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:57.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:57.087194+0000) 2022-09-08T05:37:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:57.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:57.087205+0000) 2022-09-08T05:37:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:58.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:58.087375+0000) 2022-09-08T05:37:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:58.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:58.087394+0000) 2022-09-08T05:37:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:37:59.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:59.087575+0000) 2022-09-08T05:37:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:37:59.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:37:59.087580+0000) 2022-09-08T05:37:59.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:37:59 smithi055 ceph-mon[97033]: pgmap v15539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:37:59 smithi059 ceph-mon[97450]: pgmap v15539: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:00.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:00.087786+0000) 2022-09-08T05:38:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:00.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:00.087772+0000) 2022-09-08T05:38:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:01.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:01.087907+0000) 2022-09-08T05:38:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:01.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:01.087917+0000) 2022-09-08T05:38:01.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:01 smithi055 ceph-mon[97033]: pgmap v15540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:38:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:01 smithi059 ceph-mon[97450]: pgmap v15540: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:38:02.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:02.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:02.088114+0000) 2022-09-08T05:38:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:02.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:02.088128+0000) 2022-09-08T05:38:03.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:03.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:03.088343+0000) 2022-09-08T05:38:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:03.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:03.088373+0000) 2022-09-08T05:38:03.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:38:03.459+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:38:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:03 smithi059 ceph-mon[97450]: pgmap v15541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:04.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:03 smithi055 ceph-mon[97033]: pgmap v15541: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:04.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:04.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:04.088656+0000) 2022-09-08T05:38:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:04.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:04.088678+0000) 2022-09-08T05:38:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:05.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:05.088900+0000) 2022-09-08T05:38:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:05.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:05.088913+0000) 2022-09-08T05:38:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:05 smithi059 ceph-mon[97450]: pgmap v15542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:38:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:05 smithi055 ceph-mon[97033]: pgmap v15542: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:38:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:06.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:06.089130+0000) 2022-09-08T05:38:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:06.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:06.089125+0000) 2022-09-08T05:38:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:38:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:06 smithi055 ceph-mon[97033]: pgmap v15543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:38:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:38:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:06 smithi059 ceph-mon[97450]: pgmap v15543: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:38:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:07.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:07.089332+0000) 2022-09-08T05:38:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:07.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:07.089339+0000) 2022-09-08T05:38:08.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:08.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:08.089526+0000) 2022-09-08T05:38:08.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:08.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:08.089525+0000) 2022-09-08T05:38:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:09.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:09.089773+0000) 2022-09-08T05:38:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:09.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:09.089776+0000) 2022-09-08T05:38:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:09 smithi059 ceph-mon[97450]: pgmap v15544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:38:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:09 smithi055 ceph-mon[97033]: pgmap v15544: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:38:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:10.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:10.089938+0000) 2022-09-08T05:38:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:10.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:10.089918+0000) 2022-09-08T05:38:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:10 smithi059 ceph-mon[97450]: osdmap e1051: 12 total, 11 up, 11 in 2022-09-08T05:38:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:10 smithi055 ceph-mon[97033]: osdmap e1051: 12 total, 11 up, 11 in 2022-09-08T05:38:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:11.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:11.090155+0000) 2022-09-08T05:38:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:11.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:11.090156+0000) 2022-09-08T05:38:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:11 smithi059 ceph-mon[97450]: pgmap v15546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:38:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:11 smithi055 ceph-mon[97033]: pgmap v15546: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:38:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:12.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:12.090385+0000) 2022-09-08T05:38:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:12.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:12.090385+0000) 2022-09-08T05:38:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:12 smithi055 ceph-mon[97033]: pgmap v15547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:38:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:12 smithi059 ceph-mon[97450]: pgmap v15547: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:38:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:13.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:13.090610+0000) 2022-09-08T05:38:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:13.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:13.090633+0000) 2022-09-08T05:38:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:14.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:14.090828+0000) 2022-09-08T05:38:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:14.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:14.090882+0000) 2022-09-08T05:38:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:15.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:15.091061+0000) 2022-09-08T05:38:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:15.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:15.091104+0000) 2022-09-08T05:38:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:15 smithi059 ceph-mon[97450]: pgmap v15548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:38:16.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:15 smithi055 ceph-mon[97033]: pgmap v15548: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:38:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:16.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:16.091323+0000) 2022-09-08T05:38:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:16.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:16.091393+0000) 2022-09-08T05:38:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:16 smithi055 ceph-mon[97033]: pgmap v15549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:38:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:16 smithi059 ceph-mon[97450]: pgmap v15549: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:38:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:17.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:17.091544+0000) 2022-09-08T05:38:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:17.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:17.091677+0000) 2022-09-08T05:38:18.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:18.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:18.091741+0000) 2022-09-08T05:38:18.413 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:18.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:18.091885+0000) 2022-09-08T05:38:18.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:38:18.464+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:38:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:19.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:19.091966+0000) 2022-09-08T05:38:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:19.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:19.092120+0000) 2022-09-08T05:38:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:19 smithi059 ceph-mon[97450]: pgmap v15550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:38:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:19 smithi055 ceph-mon[97033]: pgmap v15550: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:38:20.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:20.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:20.092282+0000) 2022-09-08T05:38:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:20.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:20.092115+0000) 2022-09-08T05:38:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:21.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:21.092460+0000) 2022-09-08T05:38:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:21.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:21.092274+0000) 2022-09-08T05:38:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:21 smithi059 ceph-mon[97450]: pgmap v15551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T05:38:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:21 smithi059 ceph-mon[97450]: osdmap e1052: 12 total, 11 up, 11 in 2022-09-08T05:38:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:21 smithi055 ceph-mon[97033]: pgmap v15551: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T05:38:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:21 smithi055 ceph-mon[97033]: osdmap e1052: 12 total, 11 up, 11 in 2022-09-08T05:38:22.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:22.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:22.092620+0000) 2022-09-08T05:38:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:22.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:22.092459+0000) 2022-09-08T05:38:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:23.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:23.092781+0000) 2022-09-08T05:38:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:23.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:23.092598+0000) 2022-09-08T05:38:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:23 smithi059 ceph-mon[97450]: pgmap v15553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:38:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:38:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:38:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:38:24.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:38:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:23 smithi055 ceph-mon[97033]: pgmap v15553: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:38:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:38:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:38:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:38:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:38:24.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:24.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:24.092984+0000) 2022-09-08T05:38:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:24.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:24.092759+0000) 2022-09-08T05:38:25.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:25.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:25.093165+0000) 2022-09-08T05:38:25.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:25.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:25.092958+0000) 2022-09-08T05:38:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:25 smithi059 ceph-mon[97450]: pgmap v15554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:38:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:25 smithi055 ceph-mon[97033]: pgmap v15554: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:38:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:26.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:26.093148+0000) 2022-09-08T05:38:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:26.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:26.093345+0000) 2022-09-08T05:38:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:26 smithi055 ceph-mon[97033]: pgmap v15555: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:38:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:26 smithi059 ceph-mon[97450]: pgmap v15555: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:38:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:27.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:27.093370+0000) 2022-09-08T05:38:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:27.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:27.093566+0000) 2022-09-08T05:38:28.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:28.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:28.093757+0000) 2022-09-08T05:38:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:28.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:28.093623+0000) 2022-09-08T05:38:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:29.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:29.093929+0000) 2022-09-08T05:38:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:29.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:29.093972+0000) 2022-09-08T05:38:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:29 smithi059 ceph-mon[97450]: pgmap v15556: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:38:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:29 smithi055 ceph-mon[97033]: pgmap v15556: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:38:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:30.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:30.094139+0000) 2022-09-08T05:38:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:30.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:30.094131+0000) 2022-09-08T05:38:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:31.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:31.094375+0000) 2022-09-08T05:38:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:31.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:31.094336+0000) 2022-09-08T05:38:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:31 smithi059 ceph-mon[97450]: pgmap v15557: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:38:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:31 smithi055 ceph-mon[97033]: pgmap v15557: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:38:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:32.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:32.094574+0000) 2022-09-08T05:38:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:32.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:32.094553+0000) 2022-09-08T05:38:33.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:33.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:33.094740+0000) 2022-09-08T05:38:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:33.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:33.094744+0000) 2022-09-08T05:38:33.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:38:33.478+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:38:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:33 smithi059 ceph-mon[97450]: pgmap v15558: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:38:34.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:33 smithi055 ceph-mon[97033]: pgmap v15558: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:38:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:34.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:34.094897+0000) 2022-09-08T05:38:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:34.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:34.094877+0000) 2022-09-08T05:38:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:34 smithi059 ceph-mon[97450]: pgmap v15559: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:34 smithi055 ceph-mon[97033]: pgmap v15559: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:35.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:35.095075+0000) 2022-09-08T05:38:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:35.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:35.095073+0000) 2022-09-08T05:38:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:36.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:36.095202+0000) 2022-09-08T05:38:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:36.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:36.095212+0000) 2022-09-08T05:38:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:36 smithi055 ceph-mon[97033]: pgmap v15560: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:38:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:36 smithi059 ceph-mon[97450]: pgmap v15560: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:38:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:37.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:37.095328+0000) 2022-09-08T05:38:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:37.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:37.095379+0000) 2022-09-08T05:38:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:38.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:38.095538+0000) 2022-09-08T05:38:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:38.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:38.095679+0000) 2022-09-08T05:38:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:39.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:39.095686+0000) 2022-09-08T05:38:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:39.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:39.095837+0000) 2022-09-08T05:38:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:39 smithi059 ceph-mon[97450]: pgmap v15561: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:40.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:39 smithi055 ceph-mon[97033]: pgmap v15561: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:40.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:40.095916+0000) 2022-09-08T05:38:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:40.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:40.096047+0000) 2022-09-08T05:38:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:41.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:41.096139+0000) 2022-09-08T05:38:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:41.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:41.096251+0000) 2022-09-08T05:38:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:41 smithi059 ceph-mon[97450]: pgmap v15562: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:38:42.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:41 smithi055 ceph-mon[97033]: pgmap v15562: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:38:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:42.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:42.096376+0000) 2022-09-08T05:38:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:42.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:42.096479+0000) 2022-09-08T05:38:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:42 smithi055 ceph-mon[97033]: pgmap v15563: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:42 smithi059 ceph-mon[97450]: pgmap v15563: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:43.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:43.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:43.096622+0000) 2022-09-08T05:38:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:43.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:43.096704+0000) 2022-09-08T05:38:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:44.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:44.096789+0000) 2022-09-08T05:38:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:44.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:44.096837+0000) 2022-09-08T05:38:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:45.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:45.096936+0000) 2022-09-08T05:38:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:45.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:45.096984+0000) 2022-09-08T05:38:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:45 smithi059 ceph-mon[97450]: pgmap v15564: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:46.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:45 smithi055 ceph-mon[97033]: pgmap v15564: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:46.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:46.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:46.097162+0000) 2022-09-08T05:38:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:46.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:46.097107+0000) 2022-09-08T05:38:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:46 smithi055 ceph-mon[97033]: pgmap v15565: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:38:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:46 smithi059 ceph-mon[97450]: pgmap v15565: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:38:47.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:47.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:47.097326+0000) 2022-09-08T05:38:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:47.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:47.097297+0000) 2022-09-08T05:38:48.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:48.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:48.097539+0000) 2022-09-08T05:38:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:48.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:48.097541+0000) 2022-09-08T05:38:48.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:38:48.483+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:38:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:49.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:49.097779+0000) 2022-09-08T05:38:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:49.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:49.097790+0000) 2022-09-08T05:38:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:49 smithi059 ceph-mon[97450]: pgmap v15566: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:50.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:49 smithi055 ceph-mon[97033]: pgmap v15566: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:50.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:50.098031+0000) 2022-09-08T05:38:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:50.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:50.097967+0000) 2022-09-08T05:38:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:51.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:51.098188+0000) 2022-09-08T05:38:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:51.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:51.098134+0000) 2022-09-08T05:38:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:51 smithi059 ceph-mon[97450]: pgmap v15567: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:38:52.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:51 smithi055 ceph-mon[97033]: pgmap v15567: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:38:52.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:52.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:52.098374+0000) 2022-09-08T05:38:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:52.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:52.098393+0000) 2022-09-08T05:38:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:52 smithi055 ceph-mon[97033]: pgmap v15568: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:52 smithi059 ceph-mon[97450]: pgmap v15568: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:53.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:53.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:53.098526+0000) 2022-09-08T05:38:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:53.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:53.098533+0000) 2022-09-08T05:38:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:54.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:54.098743+0000) 2022-09-08T05:38:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:54.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:54.098746+0000) 2022-09-08T05:38:55.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:55.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:55.098933+0000) 2022-09-08T05:38:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:55.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:55.098949+0000) 2022-09-08T05:38:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:55 smithi059 ceph-mon[97450]: pgmap v15569: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:56.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:55 smithi055 ceph-mon[97033]: pgmap v15569: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:38:56.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:56.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:56.099078+0000) 2022-09-08T05:38:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:56.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:56.099088+0000) 2022-09-08T05:38:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:56 smithi055 ceph-mon[97033]: pgmap v15570: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:38:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:56 smithi059 ceph-mon[97450]: pgmap v15570: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:38:57.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:57.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:57.099316+0000) 2022-09-08T05:38:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:57.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:57.099324+0000) 2022-09-08T05:38:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:58.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:58.099512+0000) 2022-09-08T05:38:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:58.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:58.099516+0000) 2022-09-08T05:38:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:38:59.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:59.099755+0000) 2022-09-08T05:38:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:38:59.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:38:59.099723+0000) 2022-09-08T05:39:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:38:59 smithi059 ceph-mon[97450]: pgmap v15571: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:00.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:38:59 smithi055 ceph-mon[97033]: pgmap v15571: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:00.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:00.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:00.100025+0000) 2022-09-08T05:39:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:00.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:00.100013+0000) 2022-09-08T05:39:01.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:01.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:01.100188+0000) 2022-09-08T05:39:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:01.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:01.100158+0000) 2022-09-08T05:39:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:01 smithi059 ceph-mon[97450]: pgmap v15572: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:39:02.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:01 smithi055 ceph-mon[97033]: pgmap v15572: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:39:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:02.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:02.100384+0000) 2022-09-08T05:39:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:02.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:02.100387+0000) 2022-09-08T05:39:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:02 smithi059 ceph-mon[97450]: pgmap v15573: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:02 smithi055 ceph-mon[97033]: pgmap v15573: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:03.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:03.100675+0000) 2022-09-08T05:39:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:03.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:03.100591+0000) 2022-09-08T05:39:03.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:39:03.497+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:39:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:04.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:04.100941+0000) 2022-09-08T05:39:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:04.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:04.100819+0000) 2022-09-08T05:39:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:05.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:05.101185+0000) 2022-09-08T05:39:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:05.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:05.101067+0000) 2022-09-08T05:39:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:05 smithi059 ceph-mon[97450]: pgmap v15574: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:39:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:05 smithi055 ceph-mon[97033]: pgmap v15574: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:39:06.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:06.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:06.101363+0000) 2022-09-08T05:39:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:06.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:06.101309+0000) 2022-09-08T05:39:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:39:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:06 smithi055 ceph-mon[97033]: pgmap v15575: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:39:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:39:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:06 smithi059 ceph-mon[97450]: pgmap v15575: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:39:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:07.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:07.101528+0000) 2022-09-08T05:39:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:07.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:07.101496+0000) 2022-09-08T05:39:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:08.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:08.101715+0000) 2022-09-08T05:39:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:08.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:08.101718+0000) 2022-09-08T05:39:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:09.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:09.101930+0000) 2022-09-08T05:39:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:09.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:09.101909+0000) 2022-09-08T05:39:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:09 smithi059 ceph-mon[97450]: pgmap v15576: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:39:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:09 smithi055 ceph-mon[97033]: pgmap v15576: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:39:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:10.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:10.102068+0000) 2022-09-08T05:39:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:10.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:10.102061+0000) 2022-09-08T05:39:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:10 smithi059 ceph-mon[97450]: osdmap e1053: 12 total, 11 up, 11 in 2022-09-08T05:39:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:10 smithi059 ceph-mon[97450]: pgmap v15578: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:39:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:10 smithi055 ceph-mon[97033]: osdmap e1053: 12 total, 11 up, 11 in 2022-09-08T05:39:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:10 smithi055 ceph-mon[97033]: pgmap v15578: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:39:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:11.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:11.102247+0000) 2022-09-08T05:39:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:11.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:11.102232+0000) 2022-09-08T05:39:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:12.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:12.102447+0000) 2022-09-08T05:39:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:12.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:12.102479+0000) 2022-09-08T05:39:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:13.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:13.102753+0000) 2022-09-08T05:39:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:13.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:13.102746+0000) 2022-09-08T05:39:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:13 smithi059 ceph-mon[97450]: pgmap v15579: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:39:14.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:13 smithi055 ceph-mon[97033]: pgmap v15579: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:39:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:14.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:14.102971+0000) 2022-09-08T05:39:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:14.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:14.102939+0000) 2022-09-08T05:39:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:14 smithi055 ceph-mon[97033]: pgmap v15580: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:39:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:14 smithi059 ceph-mon[97450]: pgmap v15580: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:39:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:15.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:15.103164+0000) 2022-09-08T05:39:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:15.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:15.103096+0000) 2022-09-08T05:39:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:16.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:16.103399+0000) 2022-09-08T05:39:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:16.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:16.103379+0000) 2022-09-08T05:39:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:16 smithi055 ceph-mon[97033]: pgmap v15581: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:39:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:16 smithi059 ceph-mon[97450]: pgmap v15581: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:39:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:17.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:17.103610+0000) 2022-09-08T05:39:17.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:17.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:17.103615+0000) 2022-09-08T05:39:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:18.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:18.103872+0000) 2022-09-08T05:39:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:18.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:18.103843+0000) 2022-09-08T05:39:18.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:39:18.504+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:39:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:19.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:19.104091+0000) 2022-09-08T05:39:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:19.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:19.104084+0000) 2022-09-08T05:39:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:19 smithi059 ceph-mon[97450]: pgmap v15582: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:39:20.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:19 smithi055 ceph-mon[97033]: pgmap v15582: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:39:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:20.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:20.104236+0000) 2022-09-08T05:39:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:20.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:20.104298+0000) 2022-09-08T05:39:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:20 smithi059 ceph-mon[97450]: pgmap v15583: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T05:39:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:20 smithi055 ceph-mon[97033]: pgmap v15583: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T05:39:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:21.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:21.104385+0000) 2022-09-08T05:39:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:21.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:21.104456+0000) 2022-09-08T05:39:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:21 smithi055 ceph-mon[97033]: osdmap e1054: 12 total, 11 up, 11 in 2022-09-08T05:39:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:21 smithi059 ceph-mon[97450]: osdmap e1054: 12 total, 11 up, 11 in 2022-09-08T05:39:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:22.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:22.104574+0000) 2022-09-08T05:39:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:22.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:22.104631+0000) 2022-09-08T05:39:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:22 smithi055 ceph-mon[97033]: pgmap v15585: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:39:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:22 smithi059 ceph-mon[97450]: pgmap v15585: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:39:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:23.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:23.104891+0000) 2022-09-08T05:39:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:23.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:23.104802+0000) 2022-09-08T05:39:24.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:39:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:39:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:39:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:39:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:39:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:39:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:39:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:39:24.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:24.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:24.105148+0000) 2022-09-08T05:39:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:24.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:24.105107+0000) 2022-09-08T05:39:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:24 smithi055 ceph-mon[97033]: pgmap v15586: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:39:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:24 smithi059 ceph-mon[97450]: pgmap v15586: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:39:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:25.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:25.105315+0000) 2022-09-08T05:39:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:25.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:25.105288+0000) 2022-09-08T05:39:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:26.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:26.105558+0000) 2022-09-08T05:39:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:26.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:26.105543+0000) 2022-09-08T05:39:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:26 smithi055 ceph-mon[97033]: pgmap v15587: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:39:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:26 smithi059 ceph-mon[97450]: pgmap v15587: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:39:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:27.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:27.105834+0000) 2022-09-08T05:39:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:27.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:27.105784+0000) 2022-09-08T05:39:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:28.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:28.105987+0000) 2022-09-08T05:39:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:28.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:28.105970+0000) 2022-09-08T05:39:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:28 smithi055 ceph-mon[97033]: pgmap v15588: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:39:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:28 smithi059 ceph-mon[97450]: pgmap v15588: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:39:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:29.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:29.106169+0000) 2022-09-08T05:39:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:29.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:29.106172+0000) 2022-09-08T05:39:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:30.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:30.106379+0000) 2022-09-08T05:39:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:30.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:30.106470+0000) 2022-09-08T05:39:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:30 smithi055 ceph-mon[97033]: pgmap v15589: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:39:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:30 smithi059 ceph-mon[97450]: pgmap v15589: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:39:31.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:31.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:31.106586+0000) 2022-09-08T05:39:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:31.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:31.106665+0000) 2022-09-08T05:39:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:32.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:32.106796+0000) 2022-09-08T05:39:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:32.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:32.106890+0000) 2022-09-08T05:39:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:32 smithi055 ceph-mon[97033]: pgmap v15590: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:39:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:32 smithi059 ceph-mon[97450]: pgmap v15590: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:39:33.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:33.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:33.107117+0000) 2022-09-08T05:39:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:33.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:33.107016+0000) 2022-09-08T05:39:33.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:39:33.513+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:39:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:34.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:34.107258+0000) 2022-09-08T05:39:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:34.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:34.107330+0000) 2022-09-08T05:39:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:34 smithi055 ceph-mon[97033]: pgmap v15591: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:34 smithi059 ceph-mon[97450]: pgmap v15591: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:35.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:35.107468+0000) 2022-09-08T05:39:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:35.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:35.107500+0000) 2022-09-08T05:39:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:36.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:36.107776+0000) 2022-09-08T05:39:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:36.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:36.107747+0000) 2022-09-08T05:39:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:36 smithi055 ceph-mon[97033]: pgmap v15592: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:39:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:36 smithi059 ceph-mon[97450]: pgmap v15592: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:39:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:37.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:37.107957+0000) 2022-09-08T05:39:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:37.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:37.107983+0000) 2022-09-08T05:39:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:38.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:38.108191+0000) 2022-09-08T05:39:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:38.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:38.108189+0000) 2022-09-08T05:39:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:38 smithi055 ceph-mon[97033]: pgmap v15593: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:38 smithi059 ceph-mon[97450]: pgmap v15593: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:39.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:39.108422+0000) 2022-09-08T05:39:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:39.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:39.108477+0000) 2022-09-08T05:39:40.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:40.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:40.108769+0000) 2022-09-08T05:39:40.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:40.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:40.108738+0000) 2022-09-08T05:39:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:40 smithi055 ceph-mon[97033]: pgmap v15594: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:39:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:40 smithi059 ceph-mon[97450]: pgmap v15594: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:39:41.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:41.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:41.108945+0000) 2022-09-08T05:39:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:41.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:41.108881+0000) 2022-09-08T05:39:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:42.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:42.109199+0000) 2022-09-08T05:39:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:42.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:42.109139+0000) 2022-09-08T05:39:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:42 smithi055 ceph-mon[97033]: pgmap v15595: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:42 smithi059 ceph-mon[97450]: pgmap v15595: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:43.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:43.109417+0000) 2022-09-08T05:39:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:43.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:43.109377+0000) 2022-09-08T05:39:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:44.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:44.109729+0000) 2022-09-08T05:39:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:44.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:44.109628+0000) 2022-09-08T05:39:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:44 smithi055 ceph-mon[97033]: pgmap v15596: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:44 smithi059 ceph-mon[97450]: pgmap v15596: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:45.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:45.109922+0000) 2022-09-08T05:39:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:45.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:45.109879+0000) 2022-09-08T05:39:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:46.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:46.110088+0000) 2022-09-08T05:39:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:46.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:46.110191+0000) 2022-09-08T05:39:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:46 smithi055 ceph-mon[97033]: pgmap v15597: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:39:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:46 smithi059 ceph-mon[97450]: pgmap v15597: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:39:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:47.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:47.110471+0000) 2022-09-08T05:39:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:47.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:47.110345+0000) 2022-09-08T05:39:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:48.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:48.110752+0000) 2022-09-08T05:39:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:48.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:48.110587+0000) 2022-09-08T05:39:48.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:39:48.515+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:39:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:48 smithi055 ceph-mon[97033]: pgmap v15598: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:48 smithi059 ceph-mon[97450]: pgmap v15598: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:49.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:49.110911+0000) 2022-09-08T05:39:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:49.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:49.110808+0000) 2022-09-08T05:39:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:50.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:50.111085+0000) 2022-09-08T05:39:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:50.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:50.111010+0000) 2022-09-08T05:39:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:50 smithi055 ceph-mon[97033]: pgmap v15599: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:39:51.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:50 smithi059 ceph-mon[97450]: pgmap v15599: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:39:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:51.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:51.111297+0000) 2022-09-08T05:39:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:51.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:51.111263+0000) 2022-09-08T05:39:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:52.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:52.111538+0000) 2022-09-08T05:39:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:52.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:52.111539+0000) 2022-09-08T05:39:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:52 smithi055 ceph-mon[97033]: pgmap v15600: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:52 smithi059 ceph-mon[97450]: pgmap v15600: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:53.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:53.111733+0000) 2022-09-08T05:39:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:53.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:53.111710+0000) 2022-09-08T05:39:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:54.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:54.111879+0000) 2022-09-08T05:39:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:54.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:54.111885+0000) 2022-09-08T05:39:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:54 smithi055 ceph-mon[97033]: pgmap v15601: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:54 smithi059 ceph-mon[97450]: pgmap v15601: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:55.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:55.112035+0000) 2022-09-08T05:39:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:55.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:55.111991+0000) 2022-09-08T05:39:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:56.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:56.112187+0000) 2022-09-08T05:39:56.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:56.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:56.112153+0000) 2022-09-08T05:39:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:56 smithi055 ceph-mon[97033]: pgmap v15602: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:39:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:56 smithi059 ceph-mon[97450]: pgmap v15602: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:39:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:57.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:57.112401+0000) 2022-09-08T05:39:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:57.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:57.112401+0000) 2022-09-08T05:39:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:58.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:58.112651+0000) 2022-09-08T05:39:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:58.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:58.112632+0000) 2022-09-08T05:39:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:39:58 smithi055 ceph-mon[97033]: pgmap v15603: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:39:58 smithi059 ceph-mon[97450]: pgmap v15603: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:39:59.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:39:59.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:59.112780+0000) 2022-09-08T05:39:59.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:39:59.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:39:59.112840+0000) 2022-09-08T05:40:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:00.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:00.113014+0000) 2022-09-08T05:40:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:00.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:00.112965+0000) 2022-09-08T05:40:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T05:40:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T05:40:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T05:40:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T05:40:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T05:40:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T05:40:01.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:01.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:01.113271+0000) 2022-09-08T05:40:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:01.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:01.113325+0000) 2022-09-08T05:40:01.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:01 smithi055 ceph-mon[97033]: pgmap v15604: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:40:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:01 smithi059 ceph-mon[97450]: pgmap v15604: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:40:02.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:02.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:02.113532+0000) 2022-09-08T05:40:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:02.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:02.113527+0000) 2022-09-08T05:40:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:02 smithi055 ceph-mon[97033]: pgmap v15605: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:40:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:02 smithi059 ceph-mon[97450]: pgmap v15605: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:40:03.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:03.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:03.113680+0000) 2022-09-08T05:40:03.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:03.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:03.113680+0000) 2022-09-08T05:40:03.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:40:03.519+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:40:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:04.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:04.113836+0000) 2022-09-08T05:40:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:04.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:04.113809+0000) 2022-09-08T05:40:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:04 smithi055 ceph-mon[97033]: pgmap v15606: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:40:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:04 smithi059 ceph-mon[97450]: pgmap v15606: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:40:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:05.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:05.114088+0000) 2022-09-08T05:40:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:05.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:05.113978+0000) 2022-09-08T05:40:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:40:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:40:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:06.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:06.114327+0000) 2022-09-08T05:40:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:06.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:06.114240+0000) 2022-09-08T05:40:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:40:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:06 smithi055 ceph-mon[97033]: pgmap v15607: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:40:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:40:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:06 smithi059 ceph-mon[97450]: pgmap v15607: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:40:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:07.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:07.114468+0000) 2022-09-08T05:40:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:07.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:07.114576+0000) 2022-09-08T05:40:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:08.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:08.114737+0000) 2022-09-08T05:40:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:08.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:08.114716+0000) 2022-09-08T05:40:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:08 smithi055 ceph-mon[97033]: pgmap v15608: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:40:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:08 smithi059 ceph-mon[97450]: pgmap v15608: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:40:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:09.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:09.114988+0000) 2022-09-08T05:40:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:09.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:09.114956+0000) 2022-09-08T05:40:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:10.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:10.115169+0000) 2022-09-08T05:40:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:10.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:10.115169+0000) 2022-09-08T05:40:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:10 smithi055 ceph-mon[97033]: osdmap e1055: 12 total, 11 up, 11 in 2022-09-08T05:40:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:10 smithi055 ceph-mon[97033]: pgmap v15610: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:40:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:10 smithi059 ceph-mon[97450]: osdmap e1055: 12 total, 11 up, 11 in 2022-09-08T05:40:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:10 smithi059 ceph-mon[97450]: pgmap v15610: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:40:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:11.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:11.115347+0000) 2022-09-08T05:40:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:11.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:11.115367+0000) 2022-09-08T05:40:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:12.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:12.115599+0000) 2022-09-08T05:40:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:12.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:12.115604+0000) 2022-09-08T05:40:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:12 smithi055 ceph-mon[97033]: pgmap v15611: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:40:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:12 smithi059 ceph-mon[97450]: pgmap v15611: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:40:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:13.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:13.115783+0000) 2022-09-08T05:40:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:13.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:13.115849+0000) 2022-09-08T05:40:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:14.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:14.115994+0000) 2022-09-08T05:40:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:14.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:14.116044+0000) 2022-09-08T05:40:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:14 smithi055 ceph-mon[97033]: pgmap v15612: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:40:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:14 smithi059 ceph-mon[97450]: pgmap v15612: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:40:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:15.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:15.116221+0000) 2022-09-08T05:40:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:15.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:15.116230+0000) 2022-09-08T05:40:16.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:16.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:16.116428+0000) 2022-09-08T05:40:16.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:16.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:16.116412+0000) 2022-09-08T05:40:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:16 smithi055 ceph-mon[97033]: pgmap v15613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:40:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:16 smithi059 ceph-mon[97450]: pgmap v15613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:40:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:17.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:17.116619+0000) 2022-09-08T05:40:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:17.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:17.116618+0000) 2022-09-08T05:40:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:18.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:18.116775+0000) 2022-09-08T05:40:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:18.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:18.116887+0000) 2022-09-08T05:40:18.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:40:18.528+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:40:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:18 smithi055 ceph-mon[97033]: pgmap v15614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:40:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:18 smithi059 ceph-mon[97450]: pgmap v15614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:40:19.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:19.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:19.116928+0000) 2022-09-08T05:40:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:19.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:19.117041+0000) 2022-09-08T05:40:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:20.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:20.117134+0000) 2022-09-08T05:40:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:20.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:20.117273+0000) 2022-09-08T05:40:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:20 smithi055 ceph-mon[97033]: pgmap v15615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T05:40:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:20 smithi059 ceph-mon[97450]: pgmap v15615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T05:40:21.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:21.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:21.117319+0000) 2022-09-08T05:40:21.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:21.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:21.117422+0000) 2022-09-08T05:40:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:21 smithi055 ceph-mon[97033]: osdmap e1056: 12 total, 11 up, 11 in 2022-09-08T05:40:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:21 smithi059 ceph-mon[97450]: osdmap e1056: 12 total, 11 up, 11 in 2022-09-08T05:40:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:22.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:22.117573+0000) 2022-09-08T05:40:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:22.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:22.117687+0000) 2022-09-08T05:40:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:22 smithi055 ceph-mon[97033]: pgmap v15617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:40:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:22 smithi059 ceph-mon[97450]: pgmap v15617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:40:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:23.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:23.117865+0000) 2022-09-08T05:40:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:23.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:23.117740+0000) 2022-09-08T05:40:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:40:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:40:24.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:40:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:24.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:24.117914+0000) 2022-09-08T05:40:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:24.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:24.118075+0000) 2022-09-08T05:40:24.473 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:40:24.474 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:40:24.474 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:40:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:25.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:25.118094+0000) 2022-09-08T05:40:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:25.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:25.118249+0000) 2022-09-08T05:40:25.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:40:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:40:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:25 smithi055 ceph-mon[97033]: pgmap v15618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:40:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:40:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:40:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:25 smithi059 ceph-mon[97450]: pgmap v15618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:40:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:26.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:26.118267+0000) 2022-09-08T05:40:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:26.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:26.118415+0000) 2022-09-08T05:40:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:26 smithi055 ceph-mon[97033]: pgmap v15619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:40:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:40:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:26 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:40:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:26 smithi059 ceph-mon[97450]: pgmap v15619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:40:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:40:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:26 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:40:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:27.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:27.118487+0000) 2022-09-08T05:40:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:27.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:27.118624+0000) 2022-09-08T05:40:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:28.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:28.118735+0000) 2022-09-08T05:40:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:28.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:28.118863+0000) 2022-09-08T05:40:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:40:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:40:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T05:40:28.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T05:40:28.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T05:40:28.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T05:40:28.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:40:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:40:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:40:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T05:40:28.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T05:40:28.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:40:28.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T05:40:28.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T05:40:28.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:40:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:29.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:29.118917+0000) 2022-09-08T05:40:29.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:29.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:29.119077+0000) 2022-09-08T05:40:29.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:29 smithi055 ceph-mon[97033]: pgmap v15620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:40:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:29 smithi059 ceph-mon[97450]: pgmap v15620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:40:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:30.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:30.119144+0000) 2022-09-08T05:40:30.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:30.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:30.119296+0000) 2022-09-08T05:40:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:30 smithi055 ceph-mon[97033]: pgmap v15621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:40:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:30 smithi059 ceph-mon[97450]: pgmap v15621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:40:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:31.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:31.119366+0000) 2022-09-08T05:40:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:31.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:31.119541+0000) 2022-09-08T05:40:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:32.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:32.119590+0000) 2022-09-08T05:40:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:32.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:32.119781+0000) 2022-09-08T05:40:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:32 smithi055 ceph-mon[97033]: pgmap v15622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:40:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:32 smithi059 ceph-mon[97450]: pgmap v15622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:40:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:33.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:33.119801+0000) 2022-09-08T05:40:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:33.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:33.119975+0000) 2022-09-08T05:40:33.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:40:33.536+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:40:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:34.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:34.119977+0000) 2022-09-08T05:40:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:34.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:34.120125+0000) 2022-09-08T05:40:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:34 smithi055 ceph-mon[97033]: pgmap v15623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:40:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:34 smithi059 ceph-mon[97450]: pgmap v15623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:40:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:35.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:35.120134+0000) 2022-09-08T05:40:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:35.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:35.120264+0000) 2022-09-08T05:40:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:36.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:36.120346+0000) 2022-09-08T05:40:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:36.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:36.120523+0000) 2022-09-08T05:40:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:36 smithi055 ceph-mon[97033]: pgmap v15624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:40:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:36 smithi059 ceph-mon[97450]: pgmap v15624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:40:37.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:37.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:37.120549+0000) 2022-09-08T05:40:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:37.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:37.120803+0000) 2022-09-08T05:40:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:38.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:38.120707+0000) 2022-09-08T05:40:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:38.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:38.120936+0000) 2022-09-08T05:40:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:38 smithi055 ceph-mon[97033]: pgmap v15625: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:40:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:38 smithi059 ceph-mon[97450]: pgmap v15625: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:40:39.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:39.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:39.120913+0000) 2022-09-08T05:40:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:39.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:39.121093+0000) 2022-09-08T05:40:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:40.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:40.121183+0000) 2022-09-08T05:40:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:40.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:40.121303+0000) 2022-09-08T05:40:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:40 smithi055 ceph-mon[97033]: pgmap v15626: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:40:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:40 smithi059 ceph-mon[97450]: pgmap v15626: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:40:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:41.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:41.121419+0000) 2022-09-08T05:40:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:41.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:41.121561+0000) 2022-09-08T05:40:42.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:42.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:42.121796+0000) 2022-09-08T05:40:42.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:42.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:42.121627+0000) 2022-09-08T05:40:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:42 smithi055 ceph-mon[97033]: pgmap v15627: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:40:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:42 smithi059 ceph-mon[97450]: pgmap v15627: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:40:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:43.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:43.121850+0000) 2022-09-08T05:40:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:43.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:43.121997+0000) 2022-09-08T05:40:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:44.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:44.122069+0000) 2022-09-08T05:40:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:44.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:44.122162+0000) 2022-09-08T05:40:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:44 smithi055 ceph-mon[97033]: pgmap v15628: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:40:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:44 smithi059 ceph-mon[97450]: pgmap v15628: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:40:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:45.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:45.122285+0000) 2022-09-08T05:40:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:45.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:45.122372+0000) 2022-09-08T05:40:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:46.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:46.122537+0000) 2022-09-08T05:40:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:46.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:46.122569+0000) 2022-09-08T05:40:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:46 smithi055 ceph-mon[97033]: pgmap v15629: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:40:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:46 smithi059 ceph-mon[97450]: pgmap v15629: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:40:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:47.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:47.122674+0000) 2022-09-08T05:40:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:47.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:47.122725+0000) 2022-09-08T05:40:48.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:48.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:48.122902+0000) 2022-09-08T05:40:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:48.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:48.122938+0000) 2022-09-08T05:40:48.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:40:48.552+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:40:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:48 smithi055 ceph-mon[97033]: pgmap v15630: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:40:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:48 smithi059 ceph-mon[97450]: pgmap v15630: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:40:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:49.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:49.123182+0000) 2022-09-08T05:40:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:49.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:49.123182+0000) 2022-09-08T05:40:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:50.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:50.123416+0000) 2022-09-08T05:40:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:50.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:50.123423+0000) 2022-09-08T05:40:51.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:50 smithi055 ceph-mon[97033]: pgmap v15631: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:40:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:50 smithi059 ceph-mon[97450]: pgmap v15631: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:40:51.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:51.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:51.123606+0000) 2022-09-08T05:40:51.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:51.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:51.123599+0000) 2022-09-08T05:40:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:52.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:52.123819+0000) 2022-09-08T05:40:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:52.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:52.123844+0000) 2022-09-08T05:40:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:52 smithi055 ceph-mon[97033]: pgmap v15632: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:40:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:52 smithi059 ceph-mon[97450]: pgmap v15632: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:40:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:53.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:53.124046+0000) 2022-09-08T05:40:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:53.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:53.124054+0000) 2022-09-08T05:40:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:54.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:54.124239+0000) 2022-09-08T05:40:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:54.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:54.124224+0000) 2022-09-08T05:40:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:54 smithi055 ceph-mon[97033]: pgmap v15633: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:40:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:54 smithi059 ceph-mon[97450]: pgmap v15633: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:40:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:55.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:55.124429+0000) 2022-09-08T05:40:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:55.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:55.124435+0000) 2022-09-08T05:40:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:56.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:56.124698+0000) 2022-09-08T05:40:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:56.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:56.124732+0000) 2022-09-08T05:40:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:56 smithi055 ceph-mon[97033]: pgmap v15634: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:40:57.289 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:56 smithi059 ceph-mon[97450]: pgmap v15634: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:40:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:57.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:57.124944+0000) 2022-09-08T05:40:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:57.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:57.124951+0000) 2022-09-08T05:40:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:58.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:58.125197+0000) 2022-09-08T05:40:58.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:58.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:58.125165+0000) 2022-09-08T05:40:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:40:58 smithi055 ceph-mon[97033]: pgmap v15635: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:40:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:40:58 smithi059 ceph-mon[97450]: pgmap v15635: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:40:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:40:59.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:59.125473+0000) 2022-09-08T05:40:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:40:59.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:40:59.125422+0000) 2022-09-08T05:41:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:00.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:00.125766+0000) 2022-09-08T05:41:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:00.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:00.125735+0000) 2022-09-08T05:41:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:00 smithi055 ceph-mon[97033]: pgmap v15636: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:41:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:00 smithi059 ceph-mon[97450]: pgmap v15636: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:41:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:01.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:01.125955+0000) 2022-09-08T05:41:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:01.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:01.125975+0000) 2022-09-08T05:41:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:02.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:02.126196+0000) 2022-09-08T05:41:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:02.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:02.126218+0000) 2022-09-08T05:41:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:02 smithi055 ceph-mon[97033]: pgmap v15637: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:02 smithi059 ceph-mon[97450]: pgmap v15637: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:03.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:03.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:03.126411+0000) 2022-09-08T05:41:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:03.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:03.126387+0000) 2022-09-08T05:41:03.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:41:03.567+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:41:04.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:04.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:04.126692+0000) 2022-09-08T05:41:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:04.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:04.126664+0000) 2022-09-08T05:41:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:04 smithi055 ceph-mon[97033]: pgmap v15638: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:04 smithi059 ceph-mon[97450]: pgmap v15638: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:05.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:05.126815+0000) 2022-09-08T05:41:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:05.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:05.126869+0000) 2022-09-08T05:41:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:41:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:41:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:06.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:06.127024+0000) 2022-09-08T05:41:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:06.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:06.127080+0000) 2022-09-08T05:41:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:41:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:06 smithi055 ceph-mon[97033]: pgmap v15639: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:41:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:41:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:06 smithi059 ceph-mon[97450]: pgmap v15639: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:41:07.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:07.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:07.127267+0000) 2022-09-08T05:41:07.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:07.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:07.127313+0000) 2022-09-08T05:41:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:08.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:08.127477+0000) 2022-09-08T05:41:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:08.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:08.127504+0000) 2022-09-08T05:41:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:08 smithi055 ceph-mon[97033]: pgmap v15640: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:41:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:08 smithi059 ceph-mon[97450]: pgmap v15640: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:41:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:09.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:09.127652+0000) 2022-09-08T05:41:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:09.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:09.127674+0000) 2022-09-08T05:41:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:10.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:10.127801+0000) 2022-09-08T05:41:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:10.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:10.127851+0000) 2022-09-08T05:41:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:10 smithi055 ceph-mon[97033]: osdmap e1057: 12 total, 11 up, 11 in 2022-09-08T05:41:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:10 smithi055 ceph-mon[97033]: pgmap v15642: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:41:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:10 smithi059 ceph-mon[97450]: osdmap e1057: 12 total, 11 up, 11 in 2022-09-08T05:41:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:10 smithi059 ceph-mon[97450]: pgmap v15642: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:41:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:11.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:11.127979+0000) 2022-09-08T05:41:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:11.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:11.128012+0000) 2022-09-08T05:41:12.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:12.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:12.128198+0000) 2022-09-08T05:41:12.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:12.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:12.128200+0000) 2022-09-08T05:41:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:12 smithi055 ceph-mon[97033]: pgmap v15643: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:41:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:12 smithi059 ceph-mon[97450]: pgmap v15643: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:41:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:13.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:13.128391+0000) 2022-09-08T05:41:13.409 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:13.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:13.128417+0000) 2022-09-08T05:41:14.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:14.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:14.128589+0000) 2022-09-08T05:41:14.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:14.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:14.128540+0000) 2022-09-08T05:41:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:14 smithi055 ceph-mon[97033]: pgmap v15644: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:41:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:14 smithi059 ceph-mon[97450]: pgmap v15644: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:41:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:15.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:15.128765+0000) 2022-09-08T05:41:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:15.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:15.128829+0000) 2022-09-08T05:41:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:16.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:16.129040+0000) 2022-09-08T05:41:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:16.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:16.129120+0000) 2022-09-08T05:41:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:16 smithi055 ceph-mon[97033]: pgmap v15645: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:41:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:16 smithi059 ceph-mon[97450]: pgmap v15645: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:41:17.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:17.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:17.129286+0000) 2022-09-08T05:41:17.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:17.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:17.129365+0000) 2022-09-08T05:41:18.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:18.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:18.129519+0000) 2022-09-08T05:41:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:18.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:18.129629+0000) 2022-09-08T05:41:18.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:41:18.579+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:41:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:18 smithi055 ceph-mon[97033]: pgmap v15646: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:41:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:18 smithi059 ceph-mon[97450]: pgmap v15646: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:41:19.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:19.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:19.129783+0000) 2022-09-08T05:41:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:19.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:19.129896+0000) 2022-09-08T05:41:20.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:20.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:20.130059+0000) 2022-09-08T05:41:20.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:20.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:20.130155+0000) 2022-09-08T05:41:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:20 smithi055 ceph-mon[97033]: pgmap v15647: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 187 B/s wr, 6 op/s 2022-09-08T05:41:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:20 smithi059 ceph-mon[97450]: pgmap v15647: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 187 B/s wr, 6 op/s 2022-09-08T05:41:21.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:21.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:21.130332+0000) 2022-09-08T05:41:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:21.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:21.130238+0000) 2022-09-08T05:41:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:21 smithi055 ceph-mon[97033]: osdmap e1058: 12 total, 11 up, 11 in 2022-09-08T05:41:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:21 smithi059 ceph-mon[97450]: osdmap e1058: 12 total, 11 up, 11 in 2022-09-08T05:41:22.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:22.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:22.130494+0000) 2022-09-08T05:41:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:22.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:22.130378+0000) 2022-09-08T05:41:23.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:22 smithi055 ceph-mon[97033]: pgmap v15649: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:41:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:22 smithi059 ceph-mon[97450]: pgmap v15649: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:41:23.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:23.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:23.130609+0000) 2022-09-08T05:41:23.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:23.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:23.130680+0000) 2022-09-08T05:41:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:24.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:24.130803+0000) 2022-09-08T05:41:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:24.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:24.130900+0000) 2022-09-08T05:41:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:24 smithi055 ceph-mon[97033]: pgmap v15650: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:41:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:24 smithi059 ceph-mon[97450]: pgmap v15650: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:41:25.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:25.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:25.131037+0000) 2022-09-08T05:41:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:25.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:25.131096+0000) 2022-09-08T05:41:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:26.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:26.131262+0000) 2022-09-08T05:41:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:26.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:26.131285+0000) 2022-09-08T05:41:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:26 smithi055 ceph-mon[97033]: pgmap v15651: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:41:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:26 smithi059 ceph-mon[97450]: pgmap v15651: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:41:27.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:27.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:27.131510+0000) 2022-09-08T05:41:27.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:27.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:27.131461+0000) 2022-09-08T05:41:28.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:41:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:41:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:41:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:41:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:41:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:41:28.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:28.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:28.131684+0000) 2022-09-08T05:41:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:28.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:28.131747+0000) 2022-09-08T05:41:29.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:29.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:29.131903+0000) 2022-09-08T05:41:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:29.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:29.131917+0000) 2022-09-08T05:41:29.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:41:29.538 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:41:29.538 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:29 smithi059 ceph-mon[97450]: pgmap v15652: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:41:29.539 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:41:29.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:41:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:41:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:29 smithi055 ceph-mon[97033]: pgmap v15652: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:41:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:41:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:30.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:30.132042+0000) 2022-09-08T05:41:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:30.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:30.132117+0000) 2022-09-08T05:41:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:30 smithi055 ceph-mon[97033]: pgmap v15653: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:41:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:30 smithi059 ceph-mon[97450]: pgmap v15653: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:41:31.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:31.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:31.132257+0000) 2022-09-08T05:41:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:31.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:31.132183+0000) 2022-09-08T05:41:32.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:32.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:32.132363+0000) 2022-09-08T05:41:32.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:32.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:32.132437+0000) 2022-09-08T05:41:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:32 smithi055 ceph-mon[97033]: pgmap v15654: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:41:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:32 smithi059 ceph-mon[97450]: pgmap v15654: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:41:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:33.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:33.132609+0000) 2022-09-08T05:41:33.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:33.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:33.132655+0000) 2022-09-08T05:41:33.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:41:33.585+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:41:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:34.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:34.132773+0000) 2022-09-08T05:41:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:34.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:34.132825+0000) 2022-09-08T05:41:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:34 smithi055 ceph-mon[97033]: pgmap v15655: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:34 smithi059 ceph-mon[97450]: pgmap v15655: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:35.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:35.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:35.133000+0000) 2022-09-08T05:41:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:35.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:35.133000+0000) 2022-09-08T05:41:36.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:36.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:36.133201+0000) 2022-09-08T05:41:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:36.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:36.133219+0000) 2022-09-08T05:41:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:36 smithi055 ceph-mon[97033]: pgmap v15656: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:41:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:36 smithi059 ceph-mon[97450]: pgmap v15656: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:41:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:37.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:37.133536+0000) 2022-09-08T05:41:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:37.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:37.133483+0000) 2022-09-08T05:41:38.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:38.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:38.133751+0000) 2022-09-08T05:41:38.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:38.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:38.133812+0000) 2022-09-08T05:41:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:38 smithi055 ceph-mon[97033]: pgmap v15657: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:38 smithi059 ceph-mon[97450]: pgmap v15657: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:39.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:39.134072+0000) 2022-09-08T05:41:39.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:39.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:39.134016+0000) 2022-09-08T05:41:40.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:40.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:40.134259+0000) 2022-09-08T05:41:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:40.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:40.134224+0000) 2022-09-08T05:41:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:40 smithi055 ceph-mon[97033]: pgmap v15658: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:41:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:40 smithi059 ceph-mon[97450]: pgmap v15658: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:41:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:41.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:41.134392+0000) 2022-09-08T05:41:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:41.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:41.134351+0000) 2022-09-08T05:41:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:42.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:42.134578+0000) 2022-09-08T05:41:42.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:42.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:42.134533+0000) 2022-09-08T05:41:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:42 smithi055 ceph-mon[97033]: pgmap v15659: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:42 smithi059 ceph-mon[97450]: pgmap v15659: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:43.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:43.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:43.134805+0000) 2022-09-08T05:41:43.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:43.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:43.134777+0000) 2022-09-08T05:41:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:44.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:44.135021+0000) 2022-09-08T05:41:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:44.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:44.135010+0000) 2022-09-08T05:41:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:44 smithi055 ceph-mon[97033]: pgmap v15660: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:44 smithi059 ceph-mon[97450]: pgmap v15660: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:45.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:45.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:45.135263+0000) 2022-09-08T05:41:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:45.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:45.135251+0000) 2022-09-08T05:41:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:46.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:46.135484+0000) 2022-09-08T05:41:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:46.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:46.135476+0000) 2022-09-08T05:41:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:46 smithi055 ceph-mon[97033]: pgmap v15661: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:41:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:46 smithi059 ceph-mon[97450]: pgmap v15661: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:41:47.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:47.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:47.135744+0000) 2022-09-08T05:41:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:47.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:47.135724+0000) 2022-09-08T05:41:48.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:48.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:48.135946+0000) 2022-09-08T05:41:48.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:48.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:48.135922+0000) 2022-09-08T05:41:48.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:41:48.588+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:41:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:48 smithi055 ceph-mon[97033]: pgmap v15662: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:48 smithi059 ceph-mon[97450]: pgmap v15662: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:49.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:49.136188+0000) 2022-09-08T05:41:49.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:49.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:49.136088+0000) 2022-09-08T05:41:50.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:50.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:50.136461+0000) 2022-09-08T05:41:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:50.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:50.136330+0000) 2022-09-08T05:41:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:50 smithi055 ceph-mon[97033]: pgmap v15663: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:41:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:50 smithi059 ceph-mon[97450]: pgmap v15663: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:41:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:51.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:51.136658+0000) 2022-09-08T05:41:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:51.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:51.136520+0000) 2022-09-08T05:41:52.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:52.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:52.136805+0000) 2022-09-08T05:41:52.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:52.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:52.136699+0000) 2022-09-08T05:41:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:52 smithi055 ceph-mon[97033]: pgmap v15664: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:52 smithi059 ceph-mon[97450]: pgmap v15664: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:53.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:53.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:53.137012+0000) 2022-09-08T05:41:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:53.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:53.136865+0000) 2022-09-08T05:41:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:54.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:54.137243+0000) 2022-09-08T05:41:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:54.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:54.137112+0000) 2022-09-08T05:41:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:54 smithi055 ceph-mon[97033]: pgmap v15665: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:54 smithi059 ceph-mon[97450]: pgmap v15665: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:55.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:55.137475+0000) 2022-09-08T05:41:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:55.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:55.137338+0000) 2022-09-08T05:41:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:56.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:56.137754+0000) 2022-09-08T05:41:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:56.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:56.137558+0000) 2022-09-08T05:41:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:56 smithi055 ceph-mon[97033]: pgmap v15666: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:41:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:56 smithi059 ceph-mon[97450]: pgmap v15666: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:41:57.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:57.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:57.137964+0000) 2022-09-08T05:41:57.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:57.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:57.137805+0000) 2022-09-08T05:41:58.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:58.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:58.138119+0000) 2022-09-08T05:41:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:58.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:58.137968+0000) 2022-09-08T05:41:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:41:58 smithi055 ceph-mon[97033]: pgmap v15667: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:41:58 smithi059 ceph-mon[97450]: pgmap v15667: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:41:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:41:59.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:59.138281+0000) 2022-09-08T05:41:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:41:59.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:41:59.138172+0000) 2022-09-08T05:42:00.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:00.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:00.138459+0000) 2022-09-08T05:42:00.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:00.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:00.138364+0000) 2022-09-08T05:42:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:00 smithi055 ceph-mon[97033]: pgmap v15668: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:42:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:00 smithi059 ceph-mon[97450]: pgmap v15668: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:42:01.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:01.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:01.138611+0000) 2022-09-08T05:42:01.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:01.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:01.138525+0000) 2022-09-08T05:42:02.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:02.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:02.138817+0000) 2022-09-08T05:42:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:02.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:02.138753+0000) 2022-09-08T05:42:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:02 smithi055 ceph-mon[97033]: pgmap v15669: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:02 smithi059 ceph-mon[97450]: pgmap v15669: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:03.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:03.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:03.139062+0000) 2022-09-08T05:42:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:03.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:03.139030+0000) 2022-09-08T05:42:03.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:42:03.597+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:42:04.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:04.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:04.139295+0000) 2022-09-08T05:42:04.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:04.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:04.139303+0000) 2022-09-08T05:42:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:04 smithi055 ceph-mon[97033]: pgmap v15670: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:04 smithi059 ceph-mon[97450]: pgmap v15670: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:05.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:05.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:05.139490+0000) 2022-09-08T05:42:05.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:05.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:05.139497+0000) 2022-09-08T05:42:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:42:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:42:06.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:06.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:06.139766+0000) 2022-09-08T05:42:06.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:06.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:06.139758+0000) 2022-09-08T05:42:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:42:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:06 smithi055 ceph-mon[97033]: pgmap v15671: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:42:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:42:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:06 smithi059 ceph-mon[97450]: pgmap v15671: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:42:07.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:07.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:07.139997+0000) 2022-09-08T05:42:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:07.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:07.139957+0000) 2022-09-08T05:42:08.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:08.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:08.140156+0000) 2022-09-08T05:42:08.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:08.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:08.140194+0000) 2022-09-08T05:42:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:08 smithi055 ceph-mon[97033]: pgmap v15672: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:42:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:08 smithi059 ceph-mon[97450]: pgmap v15672: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:42:09.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:09.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:09.140376+0000) 2022-09-08T05:42:09.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:09.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:09.140385+0000) 2022-09-08T05:42:10.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:10.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:10.140596+0000) 2022-09-08T05:42:10.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:10.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:10.140624+0000) 2022-09-08T05:42:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:10 smithi055 ceph-mon[97033]: osdmap e1059: 12 total, 11 up, 11 in 2022-09-08T05:42:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:10 smithi055 ceph-mon[97033]: pgmap v15674: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:42:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:10 smithi059 ceph-mon[97450]: osdmap e1059: 12 total, 11 up, 11 in 2022-09-08T05:42:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:10 smithi059 ceph-mon[97450]: pgmap v15674: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:42:11.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:11.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:11.140849+0000) 2022-09-08T05:42:11.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:11.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:11.140950+0000) 2022-09-08T05:42:12.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:12.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:12.141026+0000) 2022-09-08T05:42:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:12.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:12.141132+0000) 2022-09-08T05:42:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:12 smithi055 ceph-mon[97033]: pgmap v15675: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:42:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:12 smithi059 ceph-mon[97450]: pgmap v15675: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:42:13.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:13.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:13.141243+0000) 2022-09-08T05:42:13.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:13.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:13.141366+0000) 2022-09-08T05:42:14.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:14.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:14.141461+0000) 2022-09-08T05:42:14.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:14.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:14.141527+0000) 2022-09-08T05:42:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:14 smithi055 ceph-mon[97033]: pgmap v15676: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:42:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:14 smithi059 ceph-mon[97450]: pgmap v15676: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:42:15.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:15.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:15.141681+0000) 2022-09-08T05:42:15.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:15.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:15.141681+0000) 2022-09-08T05:42:16.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:16.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:16.141945+0000) 2022-09-08T05:42:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:16.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:16.141917+0000) 2022-09-08T05:42:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:16 smithi055 ceph-mon[97033]: pgmap v15677: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:42:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:16 smithi059 ceph-mon[97450]: pgmap v15677: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:42:17.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:17.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:17.142182+0000) 2022-09-08T05:42:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:17.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:17.142180+0000) 2022-09-08T05:42:18.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:18.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:18.142398+0000) 2022-09-08T05:42:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:18.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:18.142405+0000) 2022-09-08T05:42:18.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:42:18.613+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:42:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:18 smithi055 ceph-mon[97033]: pgmap v15678: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:42:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:18 smithi059 ceph-mon[97450]: pgmap v15678: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:42:19.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:19.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:19.142582+0000) 2022-09-08T05:42:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:19.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:19.142587+0000) 2022-09-08T05:42:20.401 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:20.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:20.142777+0000) 2022-09-08T05:42:20.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:20.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:20.142789+0000) 2022-09-08T05:42:21.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:20 smithi055 ceph-mon[97033]: pgmap v15679: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 187 B/s wr, 6 op/s 2022-09-08T05:42:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:20 smithi059 ceph-mon[97450]: pgmap v15679: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 187 B/s wr, 6 op/s 2022-09-08T05:42:21.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:21.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:21.142961+0000) 2022-09-08T05:42:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:21.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:21.142941+0000) 2022-09-08T05:42:22.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:21 smithi055 ceph-mon[97033]: osdmap e1060: 12 total, 11 up, 11 in 2022-09-08T05:42:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:21 smithi059 ceph-mon[97450]: osdmap e1060: 12 total, 11 up, 11 in 2022-09-08T05:42:22.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:22.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:22.143173+0000) 2022-09-08T05:42:22.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:22.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:22.143157+0000) 2022-09-08T05:42:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:22 smithi055 ceph-mon[97033]: pgmap v15681: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:42:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:22 smithi059 ceph-mon[97450]: pgmap v15681: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:42:23.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:23.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:23.143454+0000) 2022-09-08T05:42:23.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:23.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:23.143424+0000) 2022-09-08T05:42:24.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:24.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:24.143745+0000) 2022-09-08T05:42:24.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:24.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:24.143720+0000) 2022-09-08T05:42:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:24 smithi055 ceph-mon[97033]: pgmap v15682: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:42:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:24 smithi059 ceph-mon[97450]: pgmap v15682: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:42:25.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:25.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:25.143851+0000) 2022-09-08T05:42:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:25.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:25.143914+0000) 2022-09-08T05:42:26.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:26.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:26.144055+0000) 2022-09-08T05:42:26.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:26.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:26.144004+0000) 2022-09-08T05:42:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:26 smithi055 ceph-mon[97033]: pgmap v15683: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:42:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:26 smithi059 ceph-mon[97450]: pgmap v15683: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:42:27.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:27.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:27.144208+0000) 2022-09-08T05:42:27.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:27.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:27.144142+0000) 2022-09-08T05:42:28.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:28.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:28.144413+0000) 2022-09-08T05:42:28.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:28.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:28.144357+0000) 2022-09-08T05:42:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:28 smithi055 ceph-mon[97033]: pgmap v15684: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:42:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:28 smithi059 ceph-mon[97450]: pgmap v15684: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:42:29.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:29.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:29.144666+0000) 2022-09-08T05:42:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:29.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:29.144674+0000) 2022-09-08T05:42:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:42:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:42:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:42:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:42:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:42:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:42:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:42:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:42:30.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:30.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:30.144869+0000) 2022-09-08T05:42:30.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:30.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:30.144880+0000) 2022-09-08T05:42:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:30 smithi055 ceph-mon[97033]: pgmap v15685: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:42:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:30 smithi059 ceph-mon[97450]: pgmap v15685: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:42:31.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:31.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:31.145107+0000) 2022-09-08T05:42:31.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:31.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:31.145119+0000) 2022-09-08T05:42:32.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:32.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:32.145246+0000) 2022-09-08T05:42:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:32.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:32.145288+0000) 2022-09-08T05:42:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:32 smithi055 ceph-mon[97033]: pgmap v15686: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:42:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:32 smithi059 ceph-mon[97450]: pgmap v15686: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:42:33.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:33.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:33.145416+0000) 2022-09-08T05:42:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:33.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:33.145507+0000) 2022-09-08T05:42:33.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:42:33.629+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:42:34.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:34.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:34.145623+0000) 2022-09-08T05:42:34.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:34.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:34.145712+0000) 2022-09-08T05:42:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:34 smithi055 ceph-mon[97033]: pgmap v15687: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:34 smithi059 ceph-mon[97450]: pgmap v15687: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:35.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:35.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:35.145842+0000) 2022-09-08T05:42:35.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:35.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:35.145868+0000) 2022-09-08T05:42:36.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:36.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:36.145962+0000) 2022-09-08T05:42:36.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:36.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:36.146049+0000) 2022-09-08T05:42:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:36 smithi055 ceph-mon[97033]: pgmap v15688: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:42:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:36 smithi059 ceph-mon[97450]: pgmap v15688: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:42:37.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:37.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:37.146164+0000) 2022-09-08T05:42:37.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:37.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:37.146241+0000) 2022-09-08T05:42:38.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:38.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:38.146381+0000) 2022-09-08T05:42:38.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:38.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:38.146457+0000) 2022-09-08T05:42:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:38 smithi055 ceph-mon[97033]: pgmap v15689: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:38 smithi059 ceph-mon[97450]: pgmap v15689: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:39.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:39.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:39.146594+0000) 2022-09-08T05:42:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:39.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:39.146672+0000) 2022-09-08T05:42:40.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:40.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:40.146734+0000) 2022-09-08T05:42:40.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:40.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:40.146833+0000) 2022-09-08T05:42:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:40 smithi055 ceph-mon[97033]: pgmap v15690: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:42:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:40 smithi059 ceph-mon[97450]: pgmap v15690: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:42:41.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:41.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:41.146933+0000) 2022-09-08T05:42:41.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:41.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:41.147005+0000) 2022-09-08T05:42:42.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:42.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:42.147149+0000) 2022-09-08T05:42:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:42.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:42.147211+0000) 2022-09-08T05:42:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:42 smithi055 ceph-mon[97033]: pgmap v15691: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:42 smithi059 ceph-mon[97450]: pgmap v15691: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:43.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:43.147333+0000) 2022-09-08T05:42:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:43.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:43.147373+0000) 2022-09-08T05:42:44.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:44.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:44.147578+0000) 2022-09-08T05:42:44.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:44.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:44.147622+0000) 2022-09-08T05:42:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:44 smithi055 ceph-mon[97033]: pgmap v15692: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:44 smithi059 ceph-mon[97450]: pgmap v15692: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:45.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:45.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:45.147841+0000) 2022-09-08T05:42:45.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:45.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:45.147903+0000) 2022-09-08T05:42:46.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:46.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:46.148115+0000) 2022-09-08T05:42:46.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:46.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:46.148096+0000) 2022-09-08T05:42:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:46 smithi055 ceph-mon[97033]: pgmap v15693: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:42:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:46 smithi059 ceph-mon[97450]: pgmap v15693: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:42:47.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:47.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:47.148314+0000) 2022-09-08T05:42:47.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:47.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:47.148306+0000) 2022-09-08T05:42:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:48.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:48.148619+0000) 2022-09-08T05:42:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:48.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:48.148623+0000) 2022-09-08T05:42:48.901 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:42:48.630+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:42:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:48 smithi055 ceph-mon[97033]: pgmap v15694: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:48 smithi059 ceph-mon[97450]: pgmap v15694: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:49.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:49.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:49.148780+0000) 2022-09-08T05:42:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:49.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:49.148795+0000) 2022-09-08T05:42:50.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:50.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:50.149052+0000) 2022-09-08T05:42:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:50.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:50.149059+0000) 2022-09-08T05:42:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:50 smithi055 ceph-mon[97033]: pgmap v15695: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:42:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:50 smithi059 ceph-mon[97450]: pgmap v15695: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:42:51.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:51.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:51.149263+0000) 2022-09-08T05:42:51.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:51.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:51.149280+0000) 2022-09-08T05:42:52.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:52.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:52.149473+0000) 2022-09-08T05:42:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:52.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:52.149490+0000) 2022-09-08T05:42:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:52 smithi055 ceph-mon[97033]: pgmap v15696: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:52 smithi059 ceph-mon[97450]: pgmap v15696: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:53.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:53.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:53.149672+0000) 2022-09-08T05:42:53.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:53.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:53.149704+0000) 2022-09-08T05:42:54.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:54.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:54.149792+0000) 2022-09-08T05:42:54.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:54.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:54.149850+0000) 2022-09-08T05:42:55.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:54 smithi055 ceph-mon[97033]: pgmap v15697: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:54 smithi059 ceph-mon[97450]: pgmap v15697: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:55.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:55.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:55.150007+0000) 2022-09-08T05:42:55.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:55.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:55.150042+0000) 2022-09-08T05:42:56.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:56.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:56.150238+0000) 2022-09-08T05:42:56.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:56.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:56.150266+0000) 2022-09-08T05:42:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:56 smithi055 ceph-mon[97033]: pgmap v15698: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:42:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:56 smithi059 ceph-mon[97450]: pgmap v15698: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:42:57.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:57.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:57.150408+0000) 2022-09-08T05:42:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:57.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:57.150433+0000) 2022-09-08T05:42:58.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:58.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:58.150623+0000) 2022-09-08T05:42:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:58.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:58.150697+0000) 2022-09-08T05:42:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:42:58 smithi055 ceph-mon[97033]: pgmap v15699: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:42:58 smithi059 ceph-mon[97450]: pgmap v15699: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:42:59.401 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:42:59.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:59.150891+0000) 2022-09-08T05:42:59.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:42:59.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:42:59.150918+0000) 2022-09-08T05:43:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:00.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:00.151169+0000) 2022-09-08T05:43:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:00.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:00.151158+0000) 2022-09-08T05:43:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:00 smithi055 ceph-mon[97033]: pgmap v15700: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:43:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:00 smithi059 ceph-mon[97450]: pgmap v15700: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:43:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:01.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:01.151473+0000) 2022-09-08T05:43:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:01.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:01.151452+0000) 2022-09-08T05:43:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:02.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:02.151776+0000) 2022-09-08T05:43:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:02.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:02.151700+0000) 2022-09-08T05:43:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:02 smithi055 ceph-mon[97033]: pgmap v15701: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:43:03.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:02 smithi059 ceph-mon[97450]: pgmap v15701: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:43:03.648 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:03.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:03.151942+0000) 2022-09-08T05:43:03.649 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:03.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:03.151882+0000) 2022-09-08T05:43:03.902 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:43:03.647+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:43:04.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:04.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:04.152044+0000) 2022-09-08T05:43:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:04.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:04.152114+0000) 2022-09-08T05:43:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:04 smithi055 ceph-mon[97033]: pgmap v15702: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-08T05:43:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:04 smithi059 ceph-mon[97450]: pgmap v15702: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-08T05:43:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:05.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:05.152343+0000) 2022-09-08T05:43:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:05.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:05.152266+0000) 2022-09-08T05:43:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:43:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:43:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:06.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:06.152578+0000) 2022-09-08T05:43:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:06.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:06.152503+0000) 2022-09-08T05:43:07.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:43:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:06 smithi055 ceph-mon[97033]: pgmap v15703: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:43:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:43:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:06 smithi059 ceph-mon[97450]: pgmap v15703: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:43:07.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:07.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:07.152756+0000) 2022-09-08T05:43:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:07.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:07.152758+0000) 2022-09-08T05:43:08.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:08.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:08.152945+0000) 2022-09-08T05:43:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:08.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:08.152924+0000) 2022-09-08T05:43:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:08 smithi055 ceph-mon[97033]: pgmap v15704: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:43:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:08 smithi059 ceph-mon[97450]: pgmap v15704: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:43:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:09.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:09.153173+0000) 2022-09-08T05:43:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:09.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:09.153131+0000) 2022-09-08T05:43:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:09 smithi055 ceph-mon[97033]: osdmap e1061: 12 total, 11 up, 11 in 2022-09-08T05:43:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:09 smithi059 ceph-mon[97450]: osdmap e1061: 12 total, 11 up, 11 in 2022-09-08T05:43:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:10.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:10.153402+0000) 2022-09-08T05:43:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:10.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:10.153386+0000) 2022-09-08T05:43:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:10 smithi055 ceph-mon[97033]: pgmap v15706: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:43:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:10 smithi059 ceph-mon[97450]: pgmap v15706: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:43:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:11.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:11.153678+0000) 2022-09-08T05:43:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:11.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:11.153664+0000) 2022-09-08T05:43:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:11 smithi055 ceph-mon[97033]: osdmap e1062: 12 total, 11 up, 11 in 2022-09-08T05:43:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:11 smithi059 ceph-mon[97450]: osdmap e1062: 12 total, 11 up, 11 in 2022-09-08T05:43:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:12.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:12.153966+0000) 2022-09-08T05:43:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:12.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:12.153992+0000) 2022-09-08T05:43:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:12 smithi055 ceph-mon[97033]: pgmap v15708: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 33 KiB/s wr, 26 op/s 2022-09-08T05:43:13.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:12 smithi059 ceph-mon[97450]: pgmap v15708: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 33 KiB/s wr, 26 op/s 2022-09-08T05:43:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:13.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:13.154217+0000) 2022-09-08T05:43:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:13.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:13.154274+0000) 2022-09-08T05:43:14.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:14.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:14.154499+0000) 2022-09-08T05:43:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:14.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:14.154506+0000) 2022-09-08T05:43:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:14 smithi055 ceph-mon[97033]: pgmap v15709: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:43:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:14 smithi059 ceph-mon[97450]: pgmap v15709: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:43:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:15.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:15.154791+0000) 2022-09-08T05:43:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:15.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:15.154767+0000) 2022-09-08T05:43:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:16.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:16.155030+0000) 2022-09-08T05:43:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:16.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:16.155016+0000) 2022-09-08T05:43:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:16 smithi055 ceph-mon[97033]: pgmap v15710: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:43:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:16 smithi059 ceph-mon[97450]: pgmap v15710: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:43:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:17.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:17.155208+0000) 2022-09-08T05:43:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:17.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:17.155215+0000) 2022-09-08T05:43:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:18.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:18.155415+0000) 2022-09-08T05:43:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:18.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:18.155420+0000) 2022-09-08T05:43:19.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:43:18.663+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:43:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:18 smithi055 ceph-mon[97033]: pgmap v15711: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T05:43:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:18 smithi059 ceph-mon[97450]: pgmap v15711: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T05:43:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:19.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:19.155676+0000) 2022-09-08T05:43:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:19.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:19.155683+0000) 2022-09-08T05:43:20.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:20.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:20.155884+0000) 2022-09-08T05:43:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:20.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:20.155872+0000) 2022-09-08T05:43:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:20 smithi055 ceph-mon[97033]: pgmap v15712: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:43:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:20 smithi059 ceph-mon[97450]: pgmap v15712: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:43:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:21.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:21.156103+0000) 2022-09-08T05:43:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:21.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:21.156107+0000) 2022-09-08T05:43:22.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:22.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:22.156388+0000) 2022-09-08T05:43:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:22.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:22.156304+0000) 2022-09-08T05:43:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:22 smithi055 ceph-mon[97033]: pgmap v15713: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T05:43:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:22 smithi059 ceph-mon[97450]: pgmap v15713: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T05:43:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:23.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:23.156422+0000) 2022-09-08T05:43:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:23.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:23.156556+0000) 2022-09-08T05:43:24.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:24.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:24.156714+0000) 2022-09-08T05:43:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:24.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:24.156598+0000) 2022-09-08T05:43:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:24 smithi055 ceph-mon[97033]: pgmap v15714: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:43:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:24 smithi059 ceph-mon[97450]: pgmap v15714: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:43:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:25.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:25.156817+0000) 2022-09-08T05:43:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:25.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:25.156912+0000) 2022-09-08T05:43:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:26.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:26.157082+0000) 2022-09-08T05:43:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:26.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:26.157095+0000) 2022-09-08T05:43:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:26 smithi055 ceph-mon[97033]: pgmap v15715: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:43:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:26 smithi059 ceph-mon[97450]: pgmap v15715: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:43:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:27.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:27.157272+0000) 2022-09-08T05:43:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:27.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:27.157267+0000) 2022-09-08T05:43:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:28.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:28.157460+0000) 2022-09-08T05:43:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:28.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:28.157477+0000) 2022-09-08T05:43:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:28 smithi055 ceph-mon[97033]: pgmap v15716: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:43:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:28 smithi059 ceph-mon[97450]: pgmap v15716: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:43:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:29.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:29.157677+0000) 2022-09-08T05:43:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:29.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:29.157700+0000) 2022-09-08T05:43:30.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:43:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:43:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:43:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:43:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:43:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:43:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:30.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:30.157827+0000) 2022-09-08T05:43:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:30.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:30.157864+0000) 2022-09-08T05:43:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:43:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:31 smithi059 ceph-mon[97450]: pgmap v15717: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:43:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:43:31.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:31 smithi055 ceph-mon[97033]: pgmap v15717: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:43:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:31.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:31.158050+0000) 2022-09-08T05:43:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:31.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:31.158023+0000) 2022-09-08T05:43:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:32.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:32.158323+0000) 2022-09-08T05:43:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:32.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:32.158261+0000) 2022-09-08T05:43:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:32 smithi055 ceph-mon[97033]: pgmap v15718: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:43:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:32 smithi059 ceph-mon[97450]: pgmap v15718: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:43:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:33.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:33.158484+0000) 2022-09-08T05:43:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:33.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:33.158479+0000) 2022-09-08T05:43:34.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:43:33.664+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:43:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:34.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:34.158739+0000) 2022-09-08T05:43:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:34.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:34.158712+0000) 2022-09-08T05:43:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:34 smithi055 ceph-mon[97033]: pgmap v15719: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 5 op/s 2022-09-08T05:43:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:34 smithi059 ceph-mon[97450]: pgmap v15719: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 5 op/s 2022-09-08T05:43:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:35.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:35.158994+0000) 2022-09-08T05:43:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:35.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:35.158928+0000) 2022-09-08T05:43:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:36.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:36.159209+0000) 2022-09-08T05:43:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:36.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:36.159189+0000) 2022-09-08T05:43:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:36 smithi055 ceph-mon[97033]: pgmap v15720: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:43:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:36 smithi059 ceph-mon[97450]: pgmap v15720: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:43:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:37.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:37.159431+0000) 2022-09-08T05:43:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:37.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:37.159427+0000) 2022-09-08T05:43:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:38.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:38.159663+0000) 2022-09-08T05:43:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:38.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:38.159675+0000) 2022-09-08T05:43:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:38 smithi055 ceph-mon[97033]: pgmap v15721: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:43:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:38 smithi059 ceph-mon[97450]: pgmap v15721: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:43:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:39.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:39.159830+0000) 2022-09-08T05:43:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:39.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:39.159790+0000) 2022-09-08T05:43:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:40.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:40.160069+0000) 2022-09-08T05:43:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:40.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:40.160030+0000) 2022-09-08T05:43:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:40 smithi055 ceph-mon[97033]: pgmap v15722: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:43:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:40 smithi059 ceph-mon[97450]: pgmap v15722: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:43:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:41.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:41.160255+0000) 2022-09-08T05:43:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:41.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:41.160167+0000) 2022-09-08T05:43:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:42.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:42.160479+0000) 2022-09-08T05:43:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:42.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:42.160399+0000) 2022-09-08T05:43:43.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:42 smithi055 ceph-mon[97033]: pgmap v15723: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:43:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:42 smithi059 ceph-mon[97450]: pgmap v15723: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:43:43.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:43.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:43.160756+0000) 2022-09-08T05:43:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:43.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:43.160668+0000) 2022-09-08T05:43:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:44.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:44.161010+0000) 2022-09-08T05:43:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:44.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:44.160918+0000) 2022-09-08T05:43:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:44 smithi055 ceph-mon[97033]: pgmap v15724: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T05:43:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:44 smithi059 ceph-mon[97450]: pgmap v15724: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T05:43:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:45.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:45.161207+0000) 2022-09-08T05:43:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:45.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:45.161163+0000) 2022-09-08T05:43:46.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:46.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:46.161318+0000) 2022-09-08T05:43:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:46.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:46.161356+0000) 2022-09-08T05:43:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:46 smithi055 ceph-mon[97033]: pgmap v15725: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T05:43:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:46 smithi059 ceph-mon[97450]: pgmap v15725: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T05:43:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:47.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:47.161536+0000) 2022-09-08T05:43:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:47.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:47.161527+0000) 2022-09-08T05:43:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:48.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:48.161734+0000) 2022-09-08T05:43:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:48.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:48.161731+0000) 2022-09-08T05:43:49.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:43:48.666+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:43:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:48 smithi055 ceph-mon[97033]: pgmap v15726: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:43:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:48 smithi059 ceph-mon[97450]: pgmap v15726: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:43:49.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:49.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:49.161884+0000) 2022-09-08T05:43:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:49.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:49.161871+0000) 2022-09-08T05:43:50.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:50.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:50.162083+0000) 2022-09-08T05:43:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:50.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:50.162111+0000) 2022-09-08T05:43:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:50 smithi055 ceph-mon[97033]: pgmap v15727: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:43:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:50 smithi059 ceph-mon[97450]: pgmap v15727: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:43:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:51.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:51.162354+0000) 2022-09-08T05:43:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:51.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:51.162296+0000) 2022-09-08T05:43:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:52.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:52.162550+0000) 2022-09-08T05:43:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:52.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:52.162535+0000) 2022-09-08T05:43:53.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:52 smithi055 ceph-mon[97033]: pgmap v15728: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:43:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:52 smithi059 ceph-mon[97450]: pgmap v15728: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:43:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:53.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:53.162771+0000) 2022-09-08T05:43:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:53.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:53.162752+0000) 2022-09-08T05:43:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:54.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:54.163051+0000) 2022-09-08T05:43:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:54.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:54.162999+0000) 2022-09-08T05:43:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:54 smithi055 ceph-mon[97033]: pgmap v15729: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T05:43:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:54 smithi059 ceph-mon[97450]: pgmap v15729: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T05:43:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:55.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:55.163325+0000) 2022-09-08T05:43:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:55.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:55.163277+0000) 2022-09-08T05:43:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:56.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:56.163594+0000) 2022-09-08T05:43:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:56.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:56.163460+0000) 2022-09-08T05:43:57.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:56 smithi055 ceph-mon[97033]: pgmap v15730: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T05:43:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:56 smithi059 ceph-mon[97450]: pgmap v15730: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T05:43:57.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:57.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:57.163809+0000) 2022-09-08T05:43:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:57.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:57.163668+0000) 2022-09-08T05:43:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:58.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:58.164058+0000) 2022-09-08T05:43:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:58.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:58.163935+0000) 2022-09-08T05:43:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:43:58 smithi055 ceph-mon[97033]: pgmap v15731: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:43:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:43:58 smithi059 ceph-mon[97450]: pgmap v15731: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:43:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:43:59.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:59.164196+0000) 2022-09-08T05:43:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:43:59.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:43:59.164315+0000) 2022-09-08T05:44:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:00.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:00.164592+0000) 2022-09-08T05:44:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:00.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:00.164430+0000) 2022-09-08T05:44:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:00 smithi055 ceph-mon[97033]: pgmap v15732: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:44:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:00 smithi059 ceph-mon[97450]: pgmap v15732: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:44:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:01.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:01.164729+0000) 2022-09-08T05:44:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:01.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:01.164604+0000) 2022-09-08T05:44:02.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:02.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:02.164870+0000) 2022-09-08T05:44:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:02.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:02.164761+0000) 2022-09-08T05:44:03.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:02 smithi055 ceph-mon[97033]: pgmap v15733: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:44:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:02 smithi059 ceph-mon[97450]: pgmap v15733: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:44:03.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:03.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:03.165038+0000) 2022-09-08T05:44:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:03.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:03.164958+0000) 2022-09-08T05:44:04.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:44:03.680+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:44:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:04.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:04.165206+0000) 2022-09-08T05:44:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:04.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:04.165100+0000) 2022-09-08T05:44:05.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:04 smithi055 ceph-mon[97033]: pgmap v15734: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 8 op/s 2022-09-08T05:44:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:04 smithi059 ceph-mon[97450]: pgmap v15734: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 8 op/s 2022-09-08T05:44:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:05.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:05.165380+0000) 2022-09-08T05:44:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:05.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:05.165287+0000) 2022-09-08T05:44:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:44:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:44:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:06.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:06.165498+0000) 2022-09-08T05:44:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:06.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:06.165618+0000) 2022-09-08T05:44:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:44:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:06 smithi059 ceph-mon[97450]: pgmap v15735: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T05:44:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:44:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:06 smithi055 ceph-mon[97033]: pgmap v15735: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T05:44:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:07.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:07.165789+0000) 2022-09-08T05:44:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:07.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:07.165680+0000) 2022-09-08T05:44:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:08.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:08.165963+0000) 2022-09-08T05:44:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:08.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:08.165885+0000) 2022-09-08T05:44:09.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:08 smithi055 ceph-mon[97033]: pgmap v15736: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:44:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:08 smithi059 ceph-mon[97450]: pgmap v15736: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:44:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:09.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:09.166111+0000) 2022-09-08T05:44:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:09.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:09.166156+0000) 2022-09-08T05:44:10.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:09 smithi055 ceph-mon[97033]: osdmap e1063: 12 total, 11 up, 11 in 2022-09-08T05:44:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:09 smithi059 ceph-mon[97450]: osdmap e1063: 12 total, 11 up, 11 in 2022-09-08T05:44:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:10.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:10.166413+0000) 2022-09-08T05:44:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:10.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:10.166406+0000) 2022-09-08T05:44:11.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:10 smithi055 ceph-mon[97033]: pgmap v15738: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:44:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:10 smithi059 ceph-mon[97450]: pgmap v15738: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:44:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:11.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:11.166598+0000) 2022-09-08T05:44:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:11.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:11.166621+0000) 2022-09-08T05:44:12.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:11 smithi055 ceph-mon[97033]: osdmap e1064: 12 total, 11 up, 11 in 2022-09-08T05:44:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:11 smithi059 ceph-mon[97450]: osdmap e1064: 12 total, 11 up, 11 in 2022-09-08T05:44:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:12.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:12.166798+0000) 2022-09-08T05:44:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:12.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:12.166796+0000) 2022-09-08T05:44:13.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:12 smithi055 ceph-mon[97033]: pgmap v15740: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 33 KiB/s wr, 25 op/s 2022-09-08T05:44:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:12 smithi059 ceph-mon[97450]: pgmap v15740: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 33 KiB/s wr, 25 op/s 2022-09-08T05:44:13.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:13.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:13.166978+0000) 2022-09-08T05:44:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:13.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:13.166974+0000) 2022-09-08T05:44:14.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:14.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:14.167215+0000) 2022-09-08T05:44:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:14.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:14.167212+0000) 2022-09-08T05:44:15.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:14 smithi055 ceph-mon[97033]: pgmap v15741: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-08T05:44:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:14 smithi059 ceph-mon[97450]: pgmap v15741: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-08T05:44:15.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:15.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:15.167446+0000) 2022-09-08T05:44:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:15.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:15.167437+0000) 2022-09-08T05:44:16.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:16.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:16.167752+0000) 2022-09-08T05:44:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:16.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:16.167754+0000) 2022-09-08T05:44:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:16 smithi059 ceph-mon[97450]: pgmap v15742: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:44:17.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:16 smithi055 ceph-mon[97033]: pgmap v15742: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:44:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:17.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:17.167971+0000) 2022-09-08T05:44:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:17.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:17.167972+0000) 2022-09-08T05:44:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:18.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:18.168249+0000) 2022-09-08T05:44:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:18.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:18.168187+0000) 2022-09-08T05:44:19.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:44:18.680+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:44:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:18 smithi055 ceph-mon[97033]: pgmap v15743: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T05:44:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:18 smithi059 ceph-mon[97450]: pgmap v15743: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T05:44:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:19.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:19.168525+0000) 2022-09-08T05:44:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:19.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:19.168393+0000) 2022-09-08T05:44:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:20.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:20.168778+0000) 2022-09-08T05:44:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:20.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:20.168667+0000) 2022-09-08T05:44:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:20 smithi055 ceph-mon[97033]: pgmap v15744: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:44:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:20 smithi059 ceph-mon[97450]: pgmap v15744: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:44:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:21.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:21.168933+0000) 2022-09-08T05:44:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:21.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:21.168844+0000) 2022-09-08T05:44:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:22.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:22.169154+0000) 2022-09-08T05:44:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:22.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:22.169071+0000) 2022-09-08T05:44:23.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:22 smithi055 ceph-mon[97033]: pgmap v15745: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T05:44:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:22 smithi059 ceph-mon[97450]: pgmap v15745: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 86 B/s wr, 5 op/s 2022-09-08T05:44:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:23.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:23.169372+0000) 2022-09-08T05:44:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:23.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:23.169386+0000) 2022-09-08T05:44:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:24.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:24.169599+0000) 2022-09-08T05:44:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:24.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:24.169600+0000) 2022-09-08T05:44:25.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:24 smithi055 ceph-mon[97033]: pgmap v15746: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 6 op/s 2022-09-08T05:44:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:24 smithi059 ceph-mon[97450]: pgmap v15746: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 6 op/s 2022-09-08T05:44:25.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:25.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:25.169789+0000) 2022-09-08T05:44:25.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:25.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:25.169789+0000) 2022-09-08T05:44:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:26.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:26.170029+0000) 2022-09-08T05:44:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:26.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:26.170012+0000) 2022-09-08T05:44:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:26 smithi059 ceph-mon[97450]: pgmap v15747: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:44:27.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:26 smithi055 ceph-mon[97033]: pgmap v15747: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:44:27.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:27.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:27.170265+0000) 2022-09-08T05:44:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:27.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:27.170293+0000) 2022-09-08T05:44:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:28.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:28.170499+0000) 2022-09-08T05:44:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:28.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:28.170514+0000) 2022-09-08T05:44:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:28 smithi055 ceph-mon[97033]: pgmap v15748: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:44:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:28 smithi059 ceph-mon[97450]: pgmap v15748: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:44:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:29.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:29.170716+0000) 2022-09-08T05:44:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:29.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:29.170735+0000) 2022-09-08T05:44:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:44:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:44:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:44:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:44:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:44:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:44:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:30.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:30.170887+0000) 2022-09-08T05:44:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:30.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:30.171008+0000) 2022-09-08T05:44:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:31.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:31.171186+0000) 2022-09-08T05:44:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:31.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:31.171131+0000) 2022-09-08T05:44:31.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:44:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:31 smithi055 ceph-mon[97033]: pgmap v15749: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:44:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:44:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:31 smithi059 ceph-mon[97450]: pgmap v15749: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:44:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:32.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:32.171355+0000) 2022-09-08T05:44:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:32.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:32.171368+0000) 2022-09-08T05:44:33.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:32 smithi055 ceph-mon[97033]: pgmap v15750: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:44:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:32 smithi059 ceph-mon[97450]: pgmap v15750: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:44:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:33.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:33.171563+0000) 2022-09-08T05:44:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:33.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:33.171587+0000) 2022-09-08T05:44:34.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:44:33.689+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:44:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:34.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:34.171805+0000) 2022-09-08T05:44:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:34.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:34.171819+0000) 2022-09-08T05:44:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:34 smithi055 ceph-mon[97033]: pgmap v15751: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:44:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:34 smithi059 ceph-mon[97450]: pgmap v15751: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:44:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:35.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:35.172045+0000) 2022-09-08T05:44:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:35.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:35.172093+0000) 2022-09-08T05:44:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:36.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:36.172234+0000) 2022-09-08T05:44:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:36.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:36.172323+0000) 2022-09-08T05:44:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:36 smithi055 ceph-mon[97033]: pgmap v15752: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:44:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:36 smithi059 ceph-mon[97450]: pgmap v15752: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:44:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:37.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:37.172410+0000) 2022-09-08T05:44:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:37.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:37.172573+0000) 2022-09-08T05:44:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:38.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:38.172669+0000) 2022-09-08T05:44:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:38.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:38.172836+0000) 2022-09-08T05:44:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:38 smithi055 ceph-mon[97033]: pgmap v15753: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:44:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:38 smithi059 ceph-mon[97450]: pgmap v15753: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:44:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:39.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:39.172923+0000) 2022-09-08T05:44:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:39.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:39.173063+0000) 2022-09-08T05:44:40.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:40.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:40.173155+0000) 2022-09-08T05:44:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:40.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:40.173310+0000) 2022-09-08T05:44:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:41.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:41.173387+0000) 2022-09-08T05:44:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:41.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:41.173492+0000) 2022-09-08T05:44:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:41 smithi055 ceph-mon[97033]: pgmap v15754: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:44:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:41 smithi059 ceph-mon[97450]: pgmap v15754: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:44:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:42.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:42.173651+0000) 2022-09-08T05:44:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:42.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:42.173728+0000) 2022-09-08T05:44:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:43.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:43.173898+0000) 2022-09-08T05:44:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:43.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:43.173876+0000) 2022-09-08T05:44:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:43 smithi059 ceph-mon[97450]: pgmap v15755: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:44:44.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:43 smithi055 ceph-mon[97033]: pgmap v15755: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:44:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:44.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:44.174150+0000) 2022-09-08T05:44:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:44.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:44.174105+0000) 2022-09-08T05:44:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:45.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:45.174347+0000) 2022-09-08T05:44:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:45.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:45.174416+0000) 2022-09-08T05:44:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:45 smithi059 ceph-mon[97450]: pgmap v15756: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:44:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:45 smithi055 ceph-mon[97033]: pgmap v15756: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:44:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:46.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:46.174710+0000) 2022-09-08T05:44:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:46.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:46.174580+0000) 2022-09-08T05:44:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:47.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:47.174914+0000) 2022-09-08T05:44:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:47.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:47.174815+0000) 2022-09-08T05:44:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:47 smithi059 ceph-mon[97450]: pgmap v15757: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:44:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:47 smithi055 ceph-mon[97033]: pgmap v15757: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:44:48.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:48.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:48.175154+0000) 2022-09-08T05:44:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:48.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:48.175082+0000) 2022-09-08T05:44:49.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:44:48.704+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:44:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:49.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:49.175372+0000) 2022-09-08T05:44:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:49.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:49.175321+0000) 2022-09-08T05:44:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:49 smithi059 ceph-mon[97450]: pgmap v15758: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:44:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:49 smithi055 ceph-mon[97033]: pgmap v15758: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:44:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:50.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:50.175559+0000) 2022-09-08T05:44:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:50.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:50.175556+0000) 2022-09-08T05:44:51.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:51.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:51.175809+0000) 2022-09-08T05:44:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:51.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:51.175800+0000) 2022-09-08T05:44:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:51 smithi059 ceph-mon[97450]: pgmap v15759: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:44:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:51 smithi055 ceph-mon[97033]: pgmap v15759: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:44:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:52.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:52.176047+0000) 2022-09-08T05:44:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:52.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:52.176046+0000) 2022-09-08T05:44:53.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:53.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:53.176258+0000) 2022-09-08T05:44:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:53.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:53.176280+0000) 2022-09-08T05:44:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:53 smithi059 ceph-mon[97450]: pgmap v15760: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:44:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:53 smithi055 ceph-mon[97033]: pgmap v15760: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:44:54.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:54.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:54.176497+0000) 2022-09-08T05:44:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:54.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:54.176703+0000) 2022-09-08T05:44:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:55.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:55.176746+0000) 2022-09-08T05:44:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:55.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:55.176914+0000) 2022-09-08T05:44:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:56 smithi059 ceph-mon[97450]: pgmap v15761: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:44:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:56 smithi055 ceph-mon[97033]: pgmap v15761: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:44:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:56.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:56.176926+0000) 2022-09-08T05:44:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:56.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:56.177061+0000) 2022-09-08T05:44:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:57.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:57.177152+0000) 2022-09-08T05:44:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:57.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:57.177304+0000) 2022-09-08T05:44:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:44:58 smithi059 ceph-mon[97450]: pgmap v15762: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:44:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:44:58 smithi055 ceph-mon[97033]: pgmap v15762: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:44:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:58.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:58.177398+0000) 2022-09-08T05:44:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:58.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:58.177563+0000) 2022-09-08T05:44:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:44:59.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:59.177618+0000) 2022-09-08T05:44:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:44:59.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:44:59.177777+0000) 2022-09-08T05:45:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:00 smithi059 ceph-mon[97450]: pgmap v15763: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:45:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:00 smithi055 ceph-mon[97033]: pgmap v15763: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:45:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:00.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:00.177877+0000) 2022-09-08T05:45:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:00.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:00.177981+0000) 2022-09-08T05:45:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:01.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:01.178085+0000) 2022-09-08T05:45:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:01.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:01.178185+0000) 2022-09-08T05:45:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:02 smithi059 ceph-mon[97450]: pgmap v15764: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:45:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:02 smithi055 ceph-mon[97033]: pgmap v15764: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:45:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:02.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:02.178274+0000) 2022-09-08T05:45:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:02.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:02.178341+0000) 2022-09-08T05:45:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:03.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:03.178480+0000) 2022-09-08T05:45:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:03.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:03.178515+0000) 2022-09-08T05:45:04.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:45:03.707+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:45:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:04 smithi059 ceph-mon[97450]: pgmap v15765: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:45:04.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:04 smithi055 ceph-mon[97033]: pgmap v15765: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:45:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:04.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:04.178721+0000) 2022-09-08T05:45:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:04.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:04.178701+0000) 2022-09-08T05:45:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:05.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:05.178920+0000) 2022-09-08T05:45:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:05.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:05.178922+0000) 2022-09-08T05:45:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:06 smithi059 ceph-mon[97450]: pgmap v15766: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 8 op/s 2022-09-08T05:45:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:45:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:06 smithi055 ceph-mon[97033]: pgmap v15766: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 8 op/s 2022-09-08T05:45:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:45:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:06.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:06.179125+0000) 2022-09-08T05:45:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:06.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:06.179107+0000) 2022-09-08T05:45:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:45:07.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:45:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:07.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:07.179368+0000) 2022-09-08T05:45:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:07.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:07.179342+0000) 2022-09-08T05:45:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:08 smithi059 ceph-mon[97450]: pgmap v15767: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T05:45:08.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:08 smithi055 ceph-mon[97033]: pgmap v15767: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T05:45:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:08.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:08.179558+0000) 2022-09-08T05:45:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:08.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:08.179502+0000) 2022-09-08T05:45:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:09.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:09.179782+0000) 2022-09-08T05:45:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:09.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:09.179744+0000) 2022-09-08T05:45:10.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:10 smithi055 ceph-mon[97033]: pgmap v15768: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:45:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:10 smithi055 ceph-mon[97033]: osdmap e1065: 12 total, 11 up, 11 in 2022-09-08T05:45:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:10 smithi059 ceph-mon[97450]: pgmap v15768: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:45:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:10 smithi059 ceph-mon[97450]: osdmap e1065: 12 total, 11 up, 11 in 2022-09-08T05:45:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:10.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:10.180023+0000) 2022-09-08T05:45:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:10.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:10.179936+0000) 2022-09-08T05:45:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:11.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:11.180196+0000) 2022-09-08T05:45:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:11.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:11.180132+0000) 2022-09-08T05:45:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:12 smithi055 ceph-mon[97033]: pgmap v15770: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:45:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:12 smithi059 ceph-mon[97450]: pgmap v15770: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:45:12.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:12.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:12.180444+0000) 2022-09-08T05:45:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:12.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:12.180348+0000) 2022-09-08T05:45:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:13.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:13.180735+0000) 2022-09-08T05:45:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:13.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:13.180574+0000) 2022-09-08T05:45:14.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:14 smithi055 ceph-mon[97033]: pgmap v15771: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:45:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:14 smithi059 ceph-mon[97450]: pgmap v15771: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:45:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:14.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:14.180947+0000) 2022-09-08T05:45:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:14.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:14.180803+0000) 2022-09-08T05:45:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:15.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:15.181141+0000) 2022-09-08T05:45:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:15.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:15.180977+0000) 2022-09-08T05:45:16.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:16 smithi055 ceph-mon[97033]: pgmap v15772: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-08T05:45:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:16 smithi055 ceph-mon[97033]: osdmap e1066: 12 total, 11 up, 11 in 2022-09-08T05:45:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:16 smithi059 ceph-mon[97450]: pgmap v15772: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-08T05:45:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:16 smithi059 ceph-mon[97450]: osdmap e1066: 12 total, 11 up, 11 in 2022-09-08T05:45:16.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:16.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:16.181345+0000) 2022-09-08T05:45:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:16.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:16.181250+0000) 2022-09-08T05:45:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:17.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:17.181520+0000) 2022-09-08T05:45:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:17.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:17.181457+0000) 2022-09-08T05:45:18.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:18 smithi055 ceph-mon[97033]: pgmap v15774: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:45:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:18 smithi059 ceph-mon[97450]: pgmap v15774: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:45:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:18.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:18.181693+0000) 2022-09-08T05:45:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:18.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:18.181669+0000) 2022-09-08T05:45:19.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:45:18.714+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:45:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:19.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:19.181886+0000) 2022-09-08T05:45:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:19.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:19.181780+0000) 2022-09-08T05:45:20.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:20 smithi055 ceph-mon[97033]: pgmap v15775: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2022-09-08T05:45:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:20 smithi059 ceph-mon[97450]: pgmap v15775: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2022-09-08T05:45:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:20.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:20.182039+0000) 2022-09-08T05:45:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:20.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:20.181952+0000) 2022-09-08T05:45:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:21.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:21.182213+0000) 2022-09-08T05:45:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:21.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:21.182080+0000) 2022-09-08T05:45:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:22 smithi055 ceph-mon[97033]: pgmap v15776: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:45:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:22 smithi059 ceph-mon[97450]: pgmap v15776: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:45:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:22.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:22.182456+0000) 2022-09-08T05:45:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:22.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:22.182276+0000) 2022-09-08T05:45:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:23.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:23.182651+0000) 2022-09-08T05:45:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:23.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:23.182435+0000) 2022-09-08T05:45:24.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:24 smithi055 ceph-mon[97033]: pgmap v15777: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:45:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:24 smithi059 ceph-mon[97450]: pgmap v15777: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:45:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:24.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:24.182808+0000) 2022-09-08T05:45:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:24.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:24.182614+0000) 2022-09-08T05:45:25.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:25.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:25.183036+0000) 2022-09-08T05:45:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:25.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:25.182779+0000) 2022-09-08T05:45:26.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:26 smithi055 ceph-mon[97033]: pgmap v15778: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:45:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:26 smithi059 ceph-mon[97450]: pgmap v15778: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:45:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:26.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:26.183270+0000) 2022-09-08T05:45:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:26.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:26.182960+0000) 2022-09-08T05:45:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:27.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:27.183456+0000) 2022-09-08T05:45:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:27.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:27.183147+0000) 2022-09-08T05:45:28.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:28 smithi055 ceph-mon[97033]: pgmap v15779: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:45:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:28 smithi059 ceph-mon[97450]: pgmap v15779: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:45:28.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:28.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:28.183667+0000) 2022-09-08T05:45:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:28.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:28.183384+0000) 2022-09-08T05:45:29.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:29.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:29.183891+0000) 2022-09-08T05:45:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:29.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:29.183661+0000) 2022-09-08T05:45:30.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:30 smithi055 ceph-mon[97033]: pgmap v15780: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:45:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:30 smithi059 ceph-mon[97450]: pgmap v15780: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:45:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:30.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:30.184201+0000) 2022-09-08T05:45:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:30.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:30.183951+0000) 2022-09-08T05:45:31.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:45:31.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:45:31.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:45:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:45:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:45:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:45:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:31.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:31.184397+0000) 2022-09-08T05:45:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:31.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:31.184188+0000) 2022-09-08T05:45:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:32 smithi059 ceph-mon[97450]: pgmap v15781: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:45:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:45:32.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:32.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:32.184605+0000) 2022-09-08T05:45:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:32.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:32.184403+0000) 2022-09-08T05:45:32.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:32 smithi055 ceph-mon[97033]: pgmap v15781: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:45:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:45:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:33.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:33.184860+0000) 2022-09-08T05:45:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:33.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:33.184625+0000) 2022-09-08T05:45:34.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:45:33.730+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:45:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:34 smithi059 ceph-mon[97450]: pgmap v15782: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:45:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:34.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:34.185122+0000) 2022-09-08T05:45:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:34.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:34.184869+0000) 2022-09-08T05:45:34.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:34 smithi055 ceph-mon[97033]: pgmap v15782: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:45:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:35.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:35.185338+0000) 2022-09-08T05:45:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:35.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:35.185092+0000) 2022-09-08T05:45:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:36 smithi059 ceph-mon[97450]: pgmap v15783: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:45:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:36.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:36.185490+0000) 2022-09-08T05:45:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:36.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:36.185318+0000) 2022-09-08T05:45:36.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:36 smithi055 ceph-mon[97033]: pgmap v15783: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:45:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:37.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:37.185665+0000) 2022-09-08T05:45:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:37.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:37.185512+0000) 2022-09-08T05:45:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:38 smithi059 ceph-mon[97450]: pgmap v15784: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:45:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:38.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:38.185839+0000) 2022-09-08T05:45:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:38.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:38.185759+0000) 2022-09-08T05:45:38.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:38 smithi055 ceph-mon[97033]: pgmap v15784: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:45:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:39.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:39.186018+0000) 2022-09-08T05:45:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:39.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:39.185901+0000) 2022-09-08T05:45:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:40 smithi059 ceph-mon[97450]: pgmap v15785: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:45:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:40.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:40.186200+0000) 2022-09-08T05:45:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:40.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:40.186142+0000) 2022-09-08T05:45:40.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:40 smithi055 ceph-mon[97033]: pgmap v15785: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:45:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:41.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:41.186336+0000) 2022-09-08T05:45:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:41.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:41.186297+0000) 2022-09-08T05:45:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:42 smithi059 ceph-mon[97450]: pgmap v15786: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:45:42.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:42.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:42.186559+0000) 2022-09-08T05:45:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:42.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:42.186524+0000) 2022-09-08T05:45:42.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:42 smithi055 ceph-mon[97033]: pgmap v15786: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:45:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:43.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:43.186806+0000) 2022-09-08T05:45:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:43.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:43.186808+0000) 2022-09-08T05:45:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:44 smithi059 ceph-mon[97450]: pgmap v15787: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:45:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:44.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:44.187027+0000) 2022-09-08T05:45:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:44.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:44.187026+0000) 2022-09-08T05:45:44.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:44 smithi055 ceph-mon[97033]: pgmap v15787: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:45:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:45.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:45.187188+0000) 2022-09-08T05:45:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:45.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:45.187193+0000) 2022-09-08T05:45:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:46 smithi059 ceph-mon[97450]: pgmap v15788: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:45:46.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:46.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:46.187419+0000) 2022-09-08T05:45:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:46.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:46.187415+0000) 2022-09-08T05:45:46.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:46 smithi055 ceph-mon[97033]: pgmap v15788: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:45:47.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:47.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:47.187699+0000) 2022-09-08T05:45:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:47.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:47.187710+0000) 2022-09-08T05:45:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:48 smithi059 ceph-mon[97450]: pgmap v15789: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:45:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:48.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:48.187976+0000) 2022-09-08T05:45:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:48.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:48.187978+0000) 2022-09-08T05:45:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:48 smithi055 ceph-mon[97033]: pgmap v15789: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:45:49.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:45:48.745+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:45:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:49.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:49.188200+0000) 2022-09-08T05:45:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:49.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:49.188204+0000) 2022-09-08T05:45:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:50 smithi059 ceph-mon[97450]: pgmap v15790: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:45:50.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:50.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:50.188426+0000) 2022-09-08T05:45:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:50.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:50.188433+0000) 2022-09-08T05:45:50.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:50 smithi055 ceph-mon[97033]: pgmap v15790: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:45:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:51.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:51.188607+0000) 2022-09-08T05:45:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:51.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:51.188658+0000) 2022-09-08T05:45:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:52 smithi059 ceph-mon[97450]: pgmap v15791: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:45:52.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:52.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:52.188842+0000) 2022-09-08T05:45:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:52.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:52.188811+0000) 2022-09-08T05:45:52.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:52 smithi055 ceph-mon[97033]: pgmap v15791: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:45:53.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:53.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:53.188973+0000) 2022-09-08T05:45:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:53.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:53.189015+0000) 2022-09-08T05:45:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:54 smithi059 ceph-mon[97450]: pgmap v15792: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:45:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:54.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:54.189121+0000) 2022-09-08T05:45:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:54.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:54.189186+0000) 2022-09-08T05:45:54.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:54 smithi055 ceph-mon[97033]: pgmap v15792: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:45:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:55.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:55.189354+0000) 2022-09-08T05:45:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:55.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:55.189420+0000) 2022-09-08T05:45:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:56 smithi059 ceph-mon[97450]: pgmap v15793: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:45:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:56.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:56.189586+0000) 2022-09-08T05:45:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:56.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:56.189736+0000) 2022-09-08T05:45:56.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:56 smithi055 ceph-mon[97033]: pgmap v15793: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:45:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:57.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:57.189782+0000) 2022-09-08T05:45:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:57.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:57.189935+0000) 2022-09-08T05:45:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:45:58 smithi059 ceph-mon[97450]: pgmap v15794: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:45:58.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:58.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:58.190031+0000) 2022-09-08T05:45:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:58.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:58.190217+0000) 2022-09-08T05:45:58.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:45:58 smithi055 ceph-mon[97033]: pgmap v15794: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:45:59.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:45:59.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:59.190263+0000) 2022-09-08T05:45:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:45:59.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:45:59.190418+0000) 2022-09-08T05:46:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:00 smithi059 ceph-mon[97450]: pgmap v15795: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:00.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:00.190405+0000) 2022-09-08T05:46:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:00.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:00.190570+0000) 2022-09-08T05:46:00.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:00 smithi055 ceph-mon[97033]: pgmap v15795: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:01.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:01.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:01.190658+0000) 2022-09-08T05:46:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:01.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:01.190716+0000) 2022-09-08T05:46:02.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:02.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:02.190871+0000) 2022-09-08T05:46:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:02.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:02.190873+0000) 2022-09-08T05:46:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:02 smithi055 ceph-mon[97033]: pgmap v15796: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:46:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:02 smithi059 ceph-mon[97450]: pgmap v15796: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:46:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:03.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:03.191031+0000) 2022-09-08T05:46:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:03.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:03.191030+0000) 2022-09-08T05:46:04.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:46:03.756+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:46:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:04.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:04.191282+0000) 2022-09-08T05:46:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:04.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:04.191282+0000) 2022-09-08T05:46:04.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:04 smithi055 ceph-mon[97033]: pgmap v15797: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:04 smithi059 ceph-mon[97450]: pgmap v15797: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:05.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:05.191509+0000) 2022-09-08T05:46:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:05.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:05.191509+0000) 2022-09-08T05:46:06.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:06.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:06.191776+0000) 2022-09-08T05:46:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:06.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:06.191776+0000) 2022-09-08T05:46:06.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:06 smithi055 ceph-mon[97033]: pgmap v15798: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 8 op/s 2022-09-08T05:46:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:46:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:06 smithi059 ceph-mon[97450]: pgmap v15798: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 8 op/s 2022-09-08T05:46:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:46:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:07.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:07.192047+0000) 2022-09-08T05:46:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:07.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:07.192045+0000) 2022-09-08T05:46:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:46:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:46:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:08.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:08.192217+0000) 2022-09-08T05:46:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:08.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:08.192231+0000) 2022-09-08T05:46:08.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:08 smithi055 ceph-mon[97033]: pgmap v15799: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T05:46:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:08 smithi059 ceph-mon[97450]: pgmap v15799: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T05:46:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:09.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:09.192482+0000) 2022-09-08T05:46:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:09.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:09.192481+0000) 2022-09-08T05:46:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:10.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:10.192716+0000) 2022-09-08T05:46:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:10.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:10.192716+0000) 2022-09-08T05:46:10.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:10 smithi055 ceph-mon[97033]: pgmap v15800: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:46:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:10 smithi055 ceph-mon[97033]: osdmap e1067: 12 total, 11 up, 11 in 2022-09-08T05:46:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:10 smithi059 ceph-mon[97450]: pgmap v15800: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:46:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:10 smithi059 ceph-mon[97450]: osdmap e1067: 12 total, 11 up, 11 in 2022-09-08T05:46:11.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:11.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:11.192986+0000) 2022-09-08T05:46:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:11.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:11.192976+0000) 2022-09-08T05:46:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:12.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:12.193238+0000) 2022-09-08T05:46:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:12.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:12.193238+0000) 2022-09-08T05:46:12.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:12 smithi055 ceph-mon[97033]: pgmap v15802: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:46:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:12 smithi059 ceph-mon[97450]: pgmap v15802: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:46:13.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:13.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:13.193505+0000) 2022-09-08T05:46:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:13.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:13.193506+0000) 2022-09-08T05:46:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:14.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:14.193794+0000) 2022-09-08T05:46:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:14.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:14.193797+0000) 2022-09-08T05:46:14.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:14 smithi055 ceph-mon[97033]: pgmap v15803: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:46:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:14 smithi059 ceph-mon[97450]: pgmap v15803: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:46:15.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:15.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:15.193992+0000) 2022-09-08T05:46:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:15.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:15.193984+0000) 2022-09-08T05:46:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:16.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:16.194168+0000) 2022-09-08T05:46:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:16.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:16.194184+0000) 2022-09-08T05:46:16.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:16 smithi055 ceph-mon[97033]: pgmap v15804: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-08T05:46:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:16 smithi055 ceph-mon[97033]: osdmap e1068: 12 total, 11 up, 11 in 2022-09-08T05:46:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:16 smithi059 ceph-mon[97450]: pgmap v15804: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-08T05:46:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:16 smithi059 ceph-mon[97450]: osdmap e1068: 12 total, 11 up, 11 in 2022-09-08T05:46:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:17.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:17.194393+0000) 2022-09-08T05:46:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:17.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:17.194404+0000) 2022-09-08T05:46:18.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:18.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:18.194543+0000) 2022-09-08T05:46:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:18.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:18.194570+0000) 2022-09-08T05:46:18.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:18 smithi055 ceph-mon[97033]: pgmap v15806: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:46:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:18 smithi059 ceph-mon[97450]: pgmap v15806: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:46:19.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:46:18.771+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:46:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:19.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:19.194793+0000) 2022-09-08T05:46:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:19.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:19.194810+0000) 2022-09-08T05:46:20.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:20.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:20.195107+0000) 2022-09-08T05:46:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:20.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:20.195035+0000) 2022-09-08T05:46:20.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:20 smithi055 ceph-mon[97033]: pgmap v15807: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T05:46:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:20 smithi059 ceph-mon[97450]: pgmap v15807: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T05:46:21.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:21.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:21.195389+0000) 2022-09-08T05:46:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:21.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:21.195284+0000) 2022-09-08T05:46:22.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:22.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:22.195627+0000) 2022-09-08T05:46:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:22.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:22.195535+0000) 2022-09-08T05:46:22.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:22 smithi055 ceph-mon[97033]: pgmap v15808: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:46:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:22 smithi059 ceph-mon[97450]: pgmap v15808: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:46:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:23.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:23.195851+0000) 2022-09-08T05:46:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:23.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:23.195790+0000) 2022-09-08T05:46:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:24.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:24.196045+0000) 2022-09-08T05:46:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:24.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:24.196010+0000) 2022-09-08T05:46:24.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:24 smithi055 ceph-mon[97033]: pgmap v15809: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:46:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:24 smithi059 ceph-mon[97450]: pgmap v15809: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:46:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:25.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:25.196216+0000) 2022-09-08T05:46:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:25.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:25.196160+0000) 2022-09-08T05:46:26.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:26.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:26.196423+0000) 2022-09-08T05:46:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:26.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:26.196406+0000) 2022-09-08T05:46:26.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:26 smithi055 ceph-mon[97033]: pgmap v15810: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:46:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:26 smithi059 ceph-mon[97450]: pgmap v15810: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:46:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:27.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:27.196693+0000) 2022-09-08T05:46:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:27.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:27.196673+0000) 2022-09-08T05:46:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:28.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:28.196841+0000) 2022-09-08T05:46:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:28.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:28.196836+0000) 2022-09-08T05:46:28.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:28 smithi055 ceph-mon[97033]: pgmap v15811: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:46:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:28 smithi059 ceph-mon[97450]: pgmap v15811: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:46:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:29.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:29.196968+0000) 2022-09-08T05:46:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:29.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:29.196994+0000) 2022-09-08T05:46:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:30.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:30.197197+0000) 2022-09-08T05:46:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:30.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:30.197186+0000) 2022-09-08T05:46:30.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:30 smithi055 ceph-mon[97033]: pgmap v15812: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:30 smithi059 ceph-mon[97450]: pgmap v15812: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:31.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:31.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:31.197387+0000) 2022-09-08T05:46:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:31.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:31.197383+0000) 2022-09-08T05:46:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:32 smithi059 ceph-mon[97450]: pgmap v15813: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:46:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:46:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:46:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:46:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:32 smithi055 ceph-mon[97033]: pgmap v15813: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:46:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:46:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:46:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:46:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:32.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:32.197585+0000) 2022-09-08T05:46:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:32.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:32.197596+0000) 2022-09-08T05:46:33.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:33.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:33.197803+0000) 2022-09-08T05:46:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:33.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:33.197809+0000) 2022-09-08T05:46:34.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:46:33.779+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:46:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:34 smithi059 ceph-mon[97450]: pgmap v15814: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:34.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:34 smithi055 ceph-mon[97033]: pgmap v15814: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:34.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:34.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:34.198025+0000) 2022-09-08T05:46:34.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:34.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:34.198015+0000) 2022-09-08T05:46:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:35.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:35.198246+0000) 2022-09-08T05:46:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:35.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:35.198214+0000) 2022-09-08T05:46:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:36 smithi059 ceph-mon[97450]: pgmap v15815: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:46:36.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:36 smithi055 ceph-mon[97033]: pgmap v15815: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:46:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:36.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:36.198476+0000) 2022-09-08T05:46:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:36.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:36.198469+0000) 2022-09-08T05:46:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:37.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:37.198706+0000) 2022-09-08T05:46:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:37.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:37.198709+0000) 2022-09-08T05:46:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:38 smithi059 ceph-mon[97450]: pgmap v15816: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:46:38.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:38 smithi055 ceph-mon[97033]: pgmap v15816: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:46:38.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:38.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:38.198933+0000) 2022-09-08T05:46:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:38.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:38.198929+0000) 2022-09-08T05:46:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:39.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:39.199104+0000) 2022-09-08T05:46:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:39.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:39.199104+0000) 2022-09-08T05:46:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:40 smithi059 ceph-mon[97450]: pgmap v15817: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:40.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:40 smithi055 ceph-mon[97033]: pgmap v15817: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:40.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:40.199326+0000) 2022-09-08T05:46:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:40.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:40.199354+0000) 2022-09-08T05:46:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:41.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:41.199564+0000) 2022-09-08T05:46:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:41.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:41.199556+0000) 2022-09-08T05:46:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:42 smithi059 ceph-mon[97450]: pgmap v15818: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:46:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:42 smithi055 ceph-mon[97033]: pgmap v15818: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:46:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:42.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:42.199841+0000) 2022-09-08T05:46:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:42.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:42.199834+0000) 2022-09-08T05:46:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:43.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:43.200029+0000) 2022-09-08T05:46:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:43.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:43.200032+0000) 2022-09-08T05:46:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:44 smithi059 ceph-mon[97450]: pgmap v15819: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:44 smithi055 ceph-mon[97033]: pgmap v15819: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:44.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:44.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:44.200239+0000) 2022-09-08T05:46:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:44.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:44.200201+0000) 2022-09-08T05:46:45.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:45.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:45.200386+0000) 2022-09-08T05:46:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:45.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:45.200326+0000) 2022-09-08T05:46:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:46 smithi059 ceph-mon[97450]: pgmap v15820: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:46:46.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:46 smithi055 ceph-mon[97033]: pgmap v15820: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:46:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:46.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:46.200606+0000) 2022-09-08T05:46:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:46.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:46.200570+0000) 2022-09-08T05:46:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:47.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:47.200884+0000) 2022-09-08T05:46:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:47.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:47.200829+0000) 2022-09-08T05:46:48.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:48 smithi055 ceph-mon[97033]: pgmap v15821: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:46:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:48 smithi059 ceph-mon[97450]: pgmap v15821: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:46:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:48.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:48.201096+0000) 2022-09-08T05:46:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:48.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:48.200994+0000) 2022-09-08T05:46:49.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:46:48.782+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:46:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:49.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:49.201322+0000) 2022-09-08T05:46:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:49.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:49.201233+0000) 2022-09-08T05:46:50.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:50 smithi055 ceph-mon[97033]: pgmap v15822: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:50 smithi059 ceph-mon[97450]: pgmap v15822: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:50.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:50.201539+0000) 2022-09-08T05:46:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:50.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:50.201512+0000) 2022-09-08T05:46:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:51.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:51.201767+0000) 2022-09-08T05:46:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:51.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:51.201726+0000) 2022-09-08T05:46:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:52 smithi055 ceph-mon[97033]: pgmap v15823: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:46:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:52 smithi059 ceph-mon[97450]: pgmap v15823: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:46:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:52.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:52.202008+0000) 2022-09-08T05:46:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:52.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:52.202032+0000) 2022-09-08T05:46:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:53.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:53.202261+0000) 2022-09-08T05:46:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:53.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:53.202311+0000) 2022-09-08T05:46:54.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:54 smithi055 ceph-mon[97033]: pgmap v15824: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:54 smithi059 ceph-mon[97450]: pgmap v15824: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:46:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:54.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:54.202475+0000) 2022-09-08T05:46:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:54.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:54.202508+0000) 2022-09-08T05:46:55.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:55.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:55.202700+0000) 2022-09-08T05:46:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:55.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:55.202693+0000) 2022-09-08T05:46:56.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:56 smithi055 ceph-mon[97033]: pgmap v15825: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:46:56.521 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:56.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:56.202878+0000) 2022-09-08T05:46:56.521 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:56.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:56.202884+0000) 2022-09-08T05:46:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:56 smithi059 ceph-mon[97450]: pgmap v15825: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T05:46:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:57.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:57.203118+0000) 2022-09-08T05:46:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:57.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:57.203103+0000) 2022-09-08T05:46:58.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:46:58 smithi055 ceph-mon[97033]: pgmap v15826: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:46:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:46:58 smithi059 ceph-mon[97450]: pgmap v15826: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:46:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:58.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:58.203365+0000) 2022-09-08T05:46:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:58.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:58.203375+0000) 2022-09-08T05:46:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:46:59.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:59.203619+0000) 2022-09-08T05:46:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:46:59.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:46:59.203610+0000) 2022-09-08T05:47:00.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:00 smithi055 ceph-mon[97033]: pgmap v15827: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:00 smithi059 ceph-mon[97450]: pgmap v15827: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:00.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:00.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:00.203873+0000) 2022-09-08T05:47:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:00.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:00.203784+0000) 2022-09-08T05:47:01.478 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:01.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:01.204044+0000) 2022-09-08T05:47:01.479 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:01.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:01.203957+0000) 2022-09-08T05:47:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:02.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:02.204156+0000) 2022-09-08T05:47:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:02.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:02.204191+0000) 2022-09-08T05:47:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:02 smithi059 ceph-mon[97450]: pgmap v15828: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:47:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:02 smithi059 ceph-mon[97450]: pgmap v15829: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:47:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:47:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:02 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T05:47:02.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:02 smithi055 ceph-mon[97033]: pgmap v15828: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:47:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:02 smithi055 ceph-mon[97033]: pgmap v15829: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:47:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:47:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:02 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T05:47:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:03.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:03.204401+0000) 2022-09-08T05:47:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:03.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:03.204408+0000) 2022-09-08T05:47:04.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:47:03.797+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:47:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:04.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:04.204651+0000) 2022-09-08T05:47:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:04.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:04.204672+0000) 2022-09-08T05:47:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:04 smithi059 ceph-mon[97450]: pgmap v15830: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:47:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:04 smithi055 ceph-mon[97033]: pgmap v15830: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T05:47:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:05.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:05.204853+0000) 2022-09-08T05:47:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:05.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:05.204869+0000) 2022-09-08T05:47:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:06.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:06.205027+0000) 2022-09-08T05:47:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:06.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:06.205072+0000) 2022-09-08T05:47:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:06 smithi059 ceph-mon[97450]: pgmap v15831: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 23 op/s 2022-09-08T05:47:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:47:06.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:06 smithi055 ceph-mon[97033]: pgmap v15831: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 23 op/s 2022-09-08T05:47:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:47:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:07.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:07.205211+0000) 2022-09-08T05:47:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:07.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:07.205276+0000) 2022-09-08T05:47:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:47:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:47:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:08.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:08.205446+0000) 2022-09-08T05:47:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:08.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:08.205493+0000) 2022-09-08T05:47:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:08 smithi059 ceph-mon[97450]: pgmap v15832: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T05:47:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:08 smithi055 ceph-mon[97033]: pgmap v15832: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T05:47:09.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:09.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:09.205760+0000) 2022-09-08T05:47:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:09.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:09.205758+0000) 2022-09-08T05:47:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:10.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:10.206042+0000) 2022-09-08T05:47:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:10.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:10.206046+0000) 2022-09-08T05:47:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:10 smithi059 ceph-mon[97450]: pgmap v15833: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T05:47:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:10 smithi059 ceph-mon[97450]: osdmap e1069: 12 total, 11 up, 11 in 2022-09-08T05:47:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:10 smithi055 ceph-mon[97033]: pgmap v15833: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T05:47:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:10 smithi055 ceph-mon[97033]: osdmap e1069: 12 total, 11 up, 11 in 2022-09-08T05:47:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:11.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:11.206279+0000) 2022-09-08T05:47:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:11.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:11.206285+0000) 2022-09-08T05:47:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:12 smithi059 ceph-mon[97450]: pgmap v15835: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:47:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:12 smithi055 ceph-mon[97033]: pgmap v15835: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:47:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:12.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:12.206541+0000) 2022-09-08T05:47:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:12.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:12.206539+0000) 2022-09-08T05:47:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:13.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:13.206774+0000) 2022-09-08T05:47:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:13.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:13.206796+0000) 2022-09-08T05:47:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:14.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:14.207002+0000) 2022-09-08T05:47:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:14.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:14.207071+0000) 2022-09-08T05:47:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:14 smithi055 ceph-mon[97033]: pgmap v15836: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:47:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:14 smithi059 ceph-mon[97450]: pgmap v15836: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:47:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:15.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:15.207295+0000) 2022-09-08T05:47:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:15.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:15.207315+0000) 2022-09-08T05:47:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:16.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:16.207451+0000) 2022-09-08T05:47:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:16.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:16.207478+0000) 2022-09-08T05:47:16.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:16 smithi055 ceph-mon[97033]: pgmap v15837: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:47:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:16 smithi055 ceph-mon[97033]: osdmap e1070: 12 total, 11 up, 11 in 2022-09-08T05:47:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:16 smithi059 ceph-mon[97450]: pgmap v15837: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:47:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:16 smithi059 ceph-mon[97450]: osdmap e1070: 12 total, 11 up, 11 in 2022-09-08T05:47:17.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:17.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:17.207631+0000) 2022-09-08T05:47:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:17.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:17.207684+0000) 2022-09-08T05:47:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:18.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:18.207824+0000) 2022-09-08T05:47:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:18.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:18.207818+0000) 2022-09-08T05:47:18.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:18 smithi055 ceph-mon[97033]: pgmap v15839: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:47:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:18 smithi059 ceph-mon[97450]: pgmap v15839: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:47:19.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:47:18.804+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:47:19.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:19.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:19.208063+0000) 2022-09-08T05:47:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:19.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:19.208055+0000) 2022-09-08T05:47:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:20.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:20.208317+0000) 2022-09-08T05:47:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:20.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:20.208316+0000) 2022-09-08T05:47:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:20 smithi055 ceph-mon[97033]: pgmap v15840: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T05:47:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:20 smithi059 ceph-mon[97450]: pgmap v15840: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T05:47:21.543 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:21.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:21.208506+0000) 2022-09-08T05:47:21.545 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:21.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:21.208517+0000) 2022-09-08T05:47:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:22 smithi059 ceph-mon[97450]: pgmap v15841: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:47:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:22 smithi055 ceph-mon[97033]: pgmap v15841: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:47:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:22.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:22.208769+0000) 2022-09-08T05:47:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:22.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:22.208776+0000) 2022-09-08T05:47:23.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:23.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:23.209022+0000) 2022-09-08T05:47:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:23.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:23.208983+0000) 2022-09-08T05:47:24.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:24.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:24.209234+0000) 2022-09-08T05:47:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:24.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:24.209233+0000) 2022-09-08T05:47:24.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:24 smithi055 ceph-mon[97033]: pgmap v15842: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:47:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:24 smithi059 ceph-mon[97450]: pgmap v15842: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:47:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:25.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:25.209494+0000) 2022-09-08T05:47:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:25.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:25.209502+0000) 2022-09-08T05:47:26.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:26.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:26.209751+0000) 2022-09-08T05:47:26.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:26.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:26.209774+0000) 2022-09-08T05:47:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:26 smithi055 ceph-mon[97033]: pgmap v15843: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:47:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:26 smithi059 ceph-mon[97450]: pgmap v15843: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:47:27.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:27.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:27.209989+0000) 2022-09-08T05:47:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:27.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:27.210002+0000) 2022-09-08T05:47:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:28.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:28.210198+0000) 2022-09-08T05:47:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:28.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:28.210211+0000) 2022-09-08T05:47:28.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:28 smithi055 ceph-mon[97033]: pgmap v15844: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:47:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:28 smithi059 ceph-mon[97450]: pgmap v15844: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:47:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:29.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:29.210426+0000) 2022-09-08T05:47:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:29.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:29.210443+0000) 2022-09-08T05:47:30.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:30.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:30.210679+0000) 2022-09-08T05:47:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:30.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:30.210716+0000) 2022-09-08T05:47:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:30 smithi055 ceph-mon[97033]: pgmap v15845: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:30 smithi059 ceph-mon[97450]: pgmap v15845: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:31.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:31.210816+0000) 2022-09-08T05:47:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:31.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:31.210851+0000) 2022-09-08T05:47:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:32 smithi059 ceph-mon[97450]: pgmap v15846: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:47:32.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:32 smithi055 ceph-mon[97033]: pgmap v15846: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:47:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:32.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:32.210999+0000) 2022-09-08T05:47:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:32.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:32.211018+0000) 2022-09-08T05:47:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:33.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:33.211209+0000) 2022-09-08T05:47:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:33.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:33.211215+0000) 2022-09-08T05:47:34.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:47:33.820+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:47:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:34.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:34.211448+0000) 2022-09-08T05:47:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:34.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:34.211443+0000) 2022-09-08T05:47:34.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:34 smithi055 ceph-mon[97033]: pgmap v15847: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:34 smithi059 ceph-mon[97450]: pgmap v15847: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:35.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:35.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:35.211714+0000) 2022-09-08T05:47:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:35.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:35.211726+0000) 2022-09-08T05:47:36.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:36.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:36.211992+0000) 2022-09-08T05:47:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:36.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:36.211972+0000) 2022-09-08T05:47:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:36 smithi055 ceph-mon[97033]: pgmap v15848: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:47:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:36 smithi059 ceph-mon[97450]: pgmap v15848: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:47:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:37.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:37.212272+0000) 2022-09-08T05:47:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:37.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:37.212224+0000) 2022-09-08T05:47:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:38.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:38.212466+0000) 2022-09-08T05:47:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:38.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:38.212464+0000) 2022-09-08T05:47:38.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:38 smithi055 ceph-mon[97033]: pgmap v15849: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:38 smithi059 ceph-mon[97450]: pgmap v15849: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:39.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:39.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:39.212727+0000) 2022-09-08T05:47:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:39.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:39.212702+0000) 2022-09-08T05:47:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:40.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:40.212911+0000) 2022-09-08T05:47:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:40.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:40.212894+0000) 2022-09-08T05:47:40.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:40 smithi055 ceph-mon[97033]: pgmap v15850: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:40 smithi059 ceph-mon[97450]: pgmap v15850: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:41.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:41.213134+0000) 2022-09-08T05:47:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:41.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:41.213133+0000) 2022-09-08T05:47:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:42 smithi059 ceph-mon[97450]: pgmap v15851: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:47:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:42 smithi055 ceph-mon[97033]: pgmap v15851: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:47:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:42.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:42.213435+0000) 2022-09-08T05:47:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:42.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:42.213367+0000) 2022-09-08T05:47:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:43.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:43.213650+0000) 2022-09-08T05:47:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:43.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:43.213608+0000) 2022-09-08T05:47:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:44.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:44.213867+0000) 2022-09-08T05:47:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:44.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:44.213842+0000) 2022-09-08T05:47:44.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:44 smithi055 ceph-mon[97033]: pgmap v15852: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:44 smithi059 ceph-mon[97450]: pgmap v15852: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:45.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:45.214089+0000) 2022-09-08T05:47:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:45.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:45.214108+0000) 2022-09-08T05:47:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:46.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:46.214312+0000) 2022-09-08T05:47:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:46.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:46.214324+0000) 2022-09-08T05:47:46.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:46 smithi055 ceph-mon[97033]: pgmap v15853: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:47:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:46 smithi059 ceph-mon[97450]: pgmap v15853: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:47:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:47.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:47.214520+0000) 2022-09-08T05:47:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:47.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:47.214557+0000) 2022-09-08T05:47:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:48.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:48.214756+0000) 2022-09-08T05:47:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:48.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:48.214824+0000) 2022-09-08T05:47:48.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:48 smithi055 ceph-mon[97033]: pgmap v15854: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:48 smithi059 ceph-mon[97450]: pgmap v15854: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:49.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:47:48.833+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:47:49.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:49.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:49.215047+0000) 2022-09-08T05:47:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:49.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:49.215094+0000) 2022-09-08T05:47:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:50.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:50.215339+0000) 2022-09-08T05:47:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:50.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:50.215377+0000) 2022-09-08T05:47:50.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:50 smithi055 ceph-mon[97033]: pgmap v15855: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:50 smithi059 ceph-mon[97450]: pgmap v15855: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:51.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:51.215569+0000) 2022-09-08T05:47:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:51.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:51.215589+0000) 2022-09-08T05:47:52.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:52 smithi055 ceph-mon[97033]: pgmap v15856: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:47:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:52 smithi059 ceph-mon[97450]: pgmap v15856: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:47:52.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:52.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:52.215733+0000) 2022-09-08T05:47:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:52.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:52.215752+0000) 2022-09-08T05:47:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:53.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:53.215962+0000) 2022-09-08T05:47:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:53.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:53.215949+0000) 2022-09-08T05:47:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:54.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:54.216144+0000) 2022-09-08T05:47:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:54.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:54.216135+0000) 2022-09-08T05:47:54.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:54 smithi055 ceph-mon[97033]: pgmap v15857: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:54 smithi059 ceph-mon[97450]: pgmap v15857: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:55.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:55.216344+0000) 2022-09-08T05:47:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:55.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:55.216393+0000) 2022-09-08T05:47:56.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:56.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:56.216569+0000) 2022-09-08T05:47:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:56.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:56.216690+0000) 2022-09-08T05:47:56.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:56 smithi055 ceph-mon[97033]: pgmap v15858: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:47:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:56 smithi059 ceph-mon[97450]: pgmap v15858: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:47:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:57.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:57.216806+0000) 2022-09-08T05:47:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:57.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:57.216923+0000) 2022-09-08T05:47:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:58.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:58.217045+0000) 2022-09-08T05:47:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:58.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:58.217132+0000) 2022-09-08T05:47:58.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:47:58 smithi055 ceph-mon[97033]: pgmap v15859: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:59.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:47:58 smithi059 ceph-mon[97450]: pgmap v15859: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:47:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:47:59.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:59.217265+0000) 2022-09-08T05:47:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:47:59.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:47:59.217333+0000) 2022-09-08T05:48:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:00.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:00.217510+0000) 2022-09-08T05:48:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:00.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:00.217570+0000) 2022-09-08T05:48:00.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:00 smithi055 ceph-mon[97033]: pgmap v15860: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:00 smithi059 ceph-mon[97450]: pgmap v15860: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:01.625 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:01.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:01.217748+0000) 2022-09-08T05:48:01.626 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:01.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:01.217807+0000) 2022-09-08T05:48:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:02 smithi055 ceph-mon[97033]: pgmap v15861: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:48:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:48:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:48:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:48:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:02 smithi059 ceph-mon[97450]: pgmap v15861: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:48:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:48:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:48:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:48:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:02.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:02.218065+0000) 2022-09-08T05:48:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:02.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:02.217973+0000) 2022-09-08T05:48:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:03.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:03.218179+0000) 2022-09-08T05:48:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:03.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:03.218250+0000) 2022-09-08T05:48:04.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:48:03.845+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:48:04.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:04.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:04.218412+0000) 2022-09-08T05:48:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:04.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:04.218452+0000) 2022-09-08T05:48:04.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:04 smithi055 ceph-mon[97033]: pgmap v15862: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:04 smithi059 ceph-mon[97450]: pgmap v15862: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:05.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:05.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:05.218603+0000) 2022-09-08T05:48:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:05.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:05.218620+0000) 2022-09-08T05:48:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:06.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:06.218847+0000) 2022-09-08T05:48:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:06.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:06.218834+0000) 2022-09-08T05:48:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:06 smithi055 ceph-mon[97033]: pgmap v15863: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:48:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:48:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:06 smithi059 ceph-mon[97450]: pgmap v15863: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:48:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:48:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:07.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:07.218998+0000) 2022-09-08T05:48:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:07.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:07.219020+0000) 2022-09-08T05:48:07.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:48:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:48:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:08.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:08.219230+0000) 2022-09-08T05:48:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:08.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:08.219238+0000) 2022-09-08T05:48:08.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:08 smithi055 ceph-mon[97033]: pgmap v15864: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:48:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:08 smithi059 ceph-mon[97450]: pgmap v15864: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:48:09.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:09.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:09.219452+0000) 2022-09-08T05:48:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:09.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:09.219457+0000) 2022-09-08T05:48:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:10.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:10.219711+0000) 2022-09-08T05:48:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:10.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:10.219694+0000) 2022-09-08T05:48:10.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:10 smithi055 ceph-mon[97033]: pgmap v15865: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:48:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:10 smithi055 ceph-mon[97033]: osdmap e1071: 12 total, 11 up, 11 in 2022-09-08T05:48:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:10 smithi059 ceph-mon[97450]: pgmap v15865: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:48:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:10 smithi059 ceph-mon[97450]: osdmap e1071: 12 total, 11 up, 11 in 2022-09-08T05:48:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:11.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:11.219895+0000) 2022-09-08T05:48:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:11.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:11.219881+0000) 2022-09-08T05:48:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:12 smithi055 ceph-mon[97033]: pgmap v15867: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:48:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:12 smithi059 ceph-mon[97450]: pgmap v15867: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:48:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:12.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:12.220059+0000) 2022-09-08T05:48:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:12.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:12.220010+0000) 2022-09-08T05:48:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:13.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:13.220262+0000) 2022-09-08T05:48:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:13.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:13.220242+0000) 2022-09-08T05:48:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:14.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:14.220502+0000) 2022-09-08T05:48:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:14.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:14.220474+0000) 2022-09-08T05:48:14.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:14 smithi055 ceph-mon[97033]: pgmap v15868: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:48:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:14 smithi059 ceph-mon[97450]: pgmap v15868: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:48:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:15.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:15.220694+0000) 2022-09-08T05:48:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:15.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:15.220698+0000) 2022-09-08T05:48:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:16.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:16.220863+0000) 2022-09-08T05:48:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:16.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:16.220928+0000) 2022-09-08T05:48:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:16 smithi055 ceph-mon[97033]: pgmap v15869: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:48:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:16 smithi055 ceph-mon[97033]: osdmap e1072: 12 total, 11 up, 11 in 2022-09-08T05:48:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:16 smithi059 ceph-mon[97450]: pgmap v15869: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:48:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:16 smithi059 ceph-mon[97450]: osdmap e1072: 12 total, 11 up, 11 in 2022-09-08T05:48:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:17.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:17.221074+0000) 2022-09-08T05:48:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:17.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:17.221112+0000) 2022-09-08T05:48:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:18.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:18.221250+0000) 2022-09-08T05:48:18.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:18.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:18.221286+0000) 2022-09-08T05:48:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:18 smithi055 ceph-mon[97033]: pgmap v15871: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:48:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:18 smithi059 ceph-mon[97450]: pgmap v15871: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:48:19.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:48:18.861+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:48:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:19.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:19.221490+0000) 2022-09-08T05:48:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:19.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:19.221482+0000) 2022-09-08T05:48:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:20.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:20.221782+0000) 2022-09-08T05:48:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:20.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:20.221777+0000) 2022-09-08T05:48:20.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:20 smithi055 ceph-mon[97033]: pgmap v15872: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T05:48:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:20 smithi059 ceph-mon[97450]: pgmap v15872: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T05:48:21.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:21.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:21.222089+0000) 2022-09-08T05:48:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:21.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:21.222056+0000) 2022-09-08T05:48:22.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:22 smithi055 ceph-mon[97033]: pgmap v15873: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:48:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:22 smithi059 ceph-mon[97450]: pgmap v15873: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:48:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:22.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:22.222309+0000) 2022-09-08T05:48:22.707 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:22.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:22.222312+0000) 2022-09-08T05:48:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:23.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:23.222540+0000) 2022-09-08T05:48:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:23.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:23.222544+0000) 2022-09-08T05:48:24.651 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:24.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:24.222799+0000) 2022-09-08T05:48:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:24.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:24.222792+0000) 2022-09-08T05:48:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:24 smithi055 ceph-mon[97033]: pgmap v15874: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:48:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:24 smithi059 ceph-mon[97450]: pgmap v15874: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:48:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:25.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:25.222992+0000) 2022-09-08T05:48:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:25.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:25.223006+0000) 2022-09-08T05:48:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:26.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:26.223216+0000) 2022-09-08T05:48:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:26.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:26.223213+0000) 2022-09-08T05:48:26.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:26 smithi055 ceph-mon[97033]: pgmap v15875: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:48:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:26 smithi059 ceph-mon[97450]: pgmap v15875: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:48:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:27.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:27.223455+0000) 2022-09-08T05:48:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:27.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:27.223433+0000) 2022-09-08T05:48:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:28.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:28.223682+0000) 2022-09-08T05:48:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:28.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:28.223676+0000) 2022-09-08T05:48:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:28 smithi055 ceph-mon[97033]: pgmap v15876: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:48:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:28 smithi059 ceph-mon[97450]: pgmap v15876: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:48:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:29.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:29.223912+0000) 2022-09-08T05:48:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:29.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:29.223909+0000) 2022-09-08T05:48:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:30.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:30.224109+0000) 2022-09-08T05:48:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:30.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:30.224112+0000) 2022-09-08T05:48:30.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:30 smithi055 ceph-mon[97033]: pgmap v15877: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:30 smithi059 ceph-mon[97450]: pgmap v15877: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:31.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:31.224316+0000) 2022-09-08T05:48:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:31.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:31.224320+0000) 2022-09-08T05:48:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:32.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:32.224569+0000) 2022-09-08T05:48:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:32.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:32.224560+0000) 2022-09-08T05:48:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:32 smithi059 ceph-mon[97450]: pgmap v15878: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:48:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:32 smithi059 ceph-mon[97450]: pgmap v15879: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:48:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:32 smithi059 ceph-mon[97450]: pgmap v15880: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:48:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:48:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:32 smithi055 ceph-mon[97033]: pgmap v15878: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:48:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:32 smithi055 ceph-mon[97033]: pgmap v15879: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:48:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:32 smithi055 ceph-mon[97033]: pgmap v15880: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:48:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:48:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:33.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:33.224814+0000) 2022-09-08T05:48:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:33.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:33.224791+0000) 2022-09-08T05:48:34.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:48:33.877+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:48:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:34.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:34.225070+0000) 2022-09-08T05:48:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:34.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:34.225077+0000) 2022-09-08T05:48:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:34 smithi059 ceph-mon[97450]: pgmap v15881: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:48:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:34 smithi055 ceph-mon[97033]: pgmap v15881: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:48:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:35.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:35.225351+0000) 2022-09-08T05:48:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:35.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:35.225313+0000) 2022-09-08T05:48:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:36.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:36.225567+0000) 2022-09-08T05:48:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:36.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:36.225498+0000) 2022-09-08T05:48:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:36 smithi059 ceph-mon[97450]: pgmap v15882: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:48:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:36 smithi055 ceph-mon[97033]: pgmap v15882: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:48:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:37.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:37.225723+0000) 2022-09-08T05:48:37.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:37.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:37.225706+0000) 2022-09-08T05:48:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:38.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:38.225864+0000) 2022-09-08T05:48:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:38.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:38.225838+0000) 2022-09-08T05:48:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:38 smithi059 ceph-mon[97450]: pgmap v15883: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:48:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:38 smithi055 ceph-mon[97033]: pgmap v15883: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:48:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:39.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:39.226065+0000) 2022-09-08T05:48:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:39.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:39.226047+0000) 2022-09-08T05:48:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:40.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:40.226301+0000) 2022-09-08T05:48:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:40.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:40.226315+0000) 2022-09-08T05:48:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:40 smithi059 ceph-mon[97450]: pgmap v15884: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:48:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:40 smithi055 ceph-mon[97033]: pgmap v15884: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:48:41.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:41.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:41.226580+0000) 2022-09-08T05:48:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:41.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:41.226584+0000) 2022-09-08T05:48:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:42 smithi055 ceph-mon[97033]: pgmap v15885: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:48:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:42 smithi059 ceph-mon[97450]: pgmap v15885: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:48:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:42.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:42.226791+0000) 2022-09-08T05:48:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:42.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:42.226778+0000) 2022-09-08T05:48:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:43.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:43.226998+0000) 2022-09-08T05:48:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:43.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:43.226987+0000) 2022-09-08T05:48:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:44.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:44.227284+0000) 2022-09-08T05:48:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:44.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:44.227220+0000) 2022-09-08T05:48:45.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:44 smithi055 ceph-mon[97033]: pgmap v15886: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:44 smithi059 ceph-mon[97450]: pgmap v15886: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:45.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:45.227541+0000) 2022-09-08T05:48:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:45.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:45.227468+0000) 2022-09-08T05:48:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:46.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:46.227726+0000) 2022-09-08T05:48:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:46.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:46.227617+0000) 2022-09-08T05:48:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:46 smithi055 ceph-mon[97033]: pgmap v15887: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:48:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:46 smithi059 ceph-mon[97450]: pgmap v15887: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:48:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:47.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:47.227955+0000) 2022-09-08T05:48:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:47.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:47.227774+0000) 2022-09-08T05:48:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:48.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:48.228116+0000) 2022-09-08T05:48:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:48.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:48.228006+0000) 2022-09-08T05:48:49.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:48:48.886+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:48:49.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:48 smithi055 ceph-mon[97033]: pgmap v15888: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:48 smithi059 ceph-mon[97450]: pgmap v15888: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:49.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:49.228329+0000) 2022-09-08T05:48:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:49.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:49.228229+0000) 2022-09-08T05:48:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:50.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:50.228545+0000) 2022-09-08T05:48:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:50.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:50.228489+0000) 2022-09-08T05:48:51.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:50 smithi055 ceph-mon[97033]: pgmap v15889: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:50 smithi059 ceph-mon[97450]: pgmap v15889: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:51.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:51.228734+0000) 2022-09-08T05:48:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:51.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:51.228665+0000) 2022-09-08T05:48:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:52 smithi055 ceph-mon[97033]: pgmap v15890: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:48:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:52 smithi059 ceph-mon[97450]: pgmap v15890: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:48:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:52.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:52.228880+0000) 2022-09-08T05:48:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:52.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:52.228835+0000) 2022-09-08T05:48:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:53.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:53.229093+0000) 2022-09-08T05:48:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:53.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:53.229087+0000) 2022-09-08T05:48:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:54.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:54.229317+0000) 2022-09-08T05:48:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:54.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:54.229293+0000) 2022-09-08T05:48:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:54 smithi055 ceph-mon[97033]: pgmap v15891: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:54 smithi059 ceph-mon[97450]: pgmap v15891: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:55.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:55.229546+0000) 2022-09-08T05:48:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:55.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:55.229487+0000) 2022-09-08T05:48:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:56.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:56.229784+0000) 2022-09-08T05:48:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:56.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:56.229772+0000) 2022-09-08T05:48:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:56 smithi055 ceph-mon[97033]: pgmap v15892: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:48:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:56 smithi059 ceph-mon[97450]: pgmap v15892: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:48:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:57.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:57.230043+0000) 2022-09-08T05:48:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:57.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:57.230032+0000) 2022-09-08T05:48:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:58.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:58.230274+0000) 2022-09-08T05:48:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:58.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:58.230280+0000) 2022-09-08T05:48:59.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:48:58 smithi055 ceph-mon[97033]: pgmap v15893: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:48:58 smithi059 ceph-mon[97450]: pgmap v15893: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:48:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:48:59.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:59.230431+0000) 2022-09-08T05:48:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:48:59.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:48:59.230446+0000) 2022-09-08T05:49:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:00.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:00.230660+0000) 2022-09-08T05:49:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:00.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:00.230647+0000) 2022-09-08T05:49:01.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:00 smithi055 ceph-mon[97033]: pgmap v15894: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:00 smithi059 ceph-mon[97450]: pgmap v15894: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:01.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:01.230800+0000) 2022-09-08T05:49:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:01.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:01.230789+0000) 2022-09-08T05:49:02.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:02 smithi055 ceph-mon[97033]: pgmap v15895: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:49:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:02 smithi059 ceph-mon[97450]: pgmap v15895: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:49:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:02.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:02.231049+0000) 2022-09-08T05:49:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:02.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:02.231011+0000) 2022-09-08T05:49:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:03.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:03.231310+0000) 2022-09-08T05:49:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:03.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:03.231243+0000) 2022-09-08T05:49:04.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:49:03.899+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:49:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:04.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:04.231560+0000) 2022-09-08T05:49:04.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:04.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:04.231498+0000) 2022-09-08T05:49:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:04 smithi055 ceph-mon[97033]: pgmap v15896: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:04 smithi059 ceph-mon[97450]: pgmap v15896: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:05.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:05.231804+0000) 2022-09-08T05:49:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:05.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:05.231753+0000) 2022-09-08T05:49:06.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:49:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:49:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:06.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:06.231980+0000) 2022-09-08T05:49:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:06.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:06.231900+0000) 2022-09-08T05:49:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:06 smithi055 ceph-mon[97033]: pgmap v15897: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:49:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:49:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:06 smithi059 ceph-mon[97450]: pgmap v15897: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:49:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:49:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:07.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:07.232134+0000) 2022-09-08T05:49:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:07.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:07.232049+0000) 2022-09-08T05:49:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:08.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:08.232338+0000) 2022-09-08T05:49:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:08.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:08.232257+0000) 2022-09-08T05:49:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:08 smithi055 ceph-mon[97033]: pgmap v15898: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:49:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:08 smithi059 ceph-mon[97450]: pgmap v15898: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:49:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:09.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:09.232496+0000) 2022-09-08T05:49:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:09.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:09.232408+0000) 2022-09-08T05:49:10.651 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:10.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:10.232687+0000) 2022-09-08T05:49:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:10.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:10.232592+0000) 2022-09-08T05:49:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:10 smithi055 ceph-mon[97033]: pgmap v15899: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:49:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:10 smithi055 ceph-mon[97033]: osdmap e1073: 12 total, 11 up, 11 in 2022-09-08T05:49:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:10 smithi059 ceph-mon[97450]: pgmap v15899: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:49:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:10 smithi059 ceph-mon[97450]: osdmap e1073: 12 total, 11 up, 11 in 2022-09-08T05:49:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:11.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:11.232913+0000) 2022-09-08T05:49:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:11.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:11.232777+0000) 2022-09-08T05:49:12.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:12 smithi055 ceph-mon[97033]: pgmap v15901: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:49:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:12 smithi059 ceph-mon[97450]: pgmap v15901: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:49:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:12.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:12.233084+0000) 2022-09-08T05:49:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:12.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:12.233030+0000) 2022-09-08T05:49:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:13.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:13.233248+0000) 2022-09-08T05:49:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:13.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:13.233220+0000) 2022-09-08T05:49:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:14.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:14.233447+0000) 2022-09-08T05:49:14.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:14.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:14.233452+0000) 2022-09-08T05:49:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:14 smithi055 ceph-mon[97033]: pgmap v15902: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:49:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:14 smithi059 ceph-mon[97450]: pgmap v15902: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:49:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:15.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:15.233616+0000) 2022-09-08T05:49:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:15.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:15.233608+0000) 2022-09-08T05:49:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:16.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:16.233856+0000) 2022-09-08T05:49:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:16.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:16.233863+0000) 2022-09-08T05:49:17.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:16 smithi055 ceph-mon[97033]: pgmap v15903: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:49:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:16 smithi055 ceph-mon[97033]: osdmap e1074: 12 total, 11 up, 11 in 2022-09-08T05:49:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:16 smithi059 ceph-mon[97450]: pgmap v15903: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:49:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:16 smithi059 ceph-mon[97450]: osdmap e1074: 12 total, 11 up, 11 in 2022-09-08T05:49:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:17.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:17.234057+0000) 2022-09-08T05:49:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:17.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:17.234063+0000) 2022-09-08T05:49:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:18.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:18.234289+0000) 2022-09-08T05:49:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:18.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:18.234271+0000) 2022-09-08T05:49:19.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:18 smithi055 ceph-mon[97033]: pgmap v15905: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:49:19.235 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:49:18.914+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:49:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:18 smithi059 ceph-mon[97450]: pgmap v15905: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:49:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:19.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:19.234568+0000) 2022-09-08T05:49:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:19.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:19.234450+0000) 2022-09-08T05:49:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:20.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:20.234805+0000) 2022-09-08T05:49:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:20.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:20.234739+0000) 2022-09-08T05:49:21.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:20 smithi055 ceph-mon[97033]: pgmap v15906: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 206 B/s wr, 5 op/s 2022-09-08T05:49:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:20 smithi059 ceph-mon[97450]: pgmap v15906: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 206 B/s wr, 5 op/s 2022-09-08T05:49:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:21.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:21.235085+0000) 2022-09-08T05:49:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:21.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:21.234968+0000) 2022-09-08T05:49:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:22 smithi055 ceph-mon[97033]: pgmap v15907: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:49:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:22 smithi059 ceph-mon[97450]: pgmap v15907: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:49:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:22.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:22.235315+0000) 2022-09-08T05:49:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:22.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:22.235187+0000) 2022-09-08T05:49:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:23.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:23.235482+0000) 2022-09-08T05:49:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:23.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:23.235356+0000) 2022-09-08T05:49:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:24.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:24.235628+0000) 2022-09-08T05:49:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:24.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:24.235553+0000) 2022-09-08T05:49:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:24 smithi055 ceph-mon[97033]: pgmap v15908: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:49:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:24 smithi059 ceph-mon[97450]: pgmap v15908: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:49:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:25.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:25.235797+0000) 2022-09-08T05:49:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:25.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:25.235767+0000) 2022-09-08T05:49:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:26.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:26.235965+0000) 2022-09-08T05:49:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:26.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:26.235977+0000) 2022-09-08T05:49:27.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:26 smithi055 ceph-mon[97033]: pgmap v15909: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:49:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:26 smithi059 ceph-mon[97450]: pgmap v15909: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:49:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:27.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:27.236199+0000) 2022-09-08T05:49:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:27.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:27.236193+0000) 2022-09-08T05:49:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:28.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:28.236446+0000) 2022-09-08T05:49:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:28.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:28.236453+0000) 2022-09-08T05:49:29.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:28 smithi055 ceph-mon[97033]: pgmap v15910: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:49:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:28 smithi059 ceph-mon[97450]: pgmap v15910: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:49:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:29.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:29.236664+0000) 2022-09-08T05:49:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:29.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:29.236659+0000) 2022-09-08T05:49:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:30.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:30.236884+0000) 2022-09-08T05:49:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:30.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:30.236934+0000) 2022-09-08T05:49:31.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:30 smithi055 ceph-mon[97033]: pgmap v15911: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:30 smithi059 ceph-mon[97450]: pgmap v15911: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:31.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:31.237091+0000) 2022-09-08T05:49:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:31.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:31.237125+0000) 2022-09-08T05:49:32.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:32 smithi055 ceph-mon[97033]: pgmap v15912: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:49:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:49:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:49:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:49:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:32 smithi059 ceph-mon[97450]: pgmap v15912: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:49:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:49:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:49:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:49:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:32.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:32.237348+0000) 2022-09-08T05:49:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:32.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:32.237342+0000) 2022-09-08T05:49:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:33.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:33.237508+0000) 2022-09-08T05:49:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:33.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:33.237539+0000) 2022-09-08T05:49:34.238 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:49:33.929+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:49:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:34.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:34.237788+0000) 2022-09-08T05:49:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:34.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:34.237778+0000) 2022-09-08T05:49:35.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:34 smithi055 ceph-mon[97033]: pgmap v15913: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:34 smithi059 ceph-mon[97450]: pgmap v15913: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:35.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:35.237968+0000) 2022-09-08T05:49:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:35.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:35.237952+0000) 2022-09-08T05:49:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:36.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:36.238211+0000) 2022-09-08T05:49:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:36.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:36.238116+0000) 2022-09-08T05:49:37.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:36 smithi055 ceph-mon[97033]: pgmap v15914: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:49:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:36 smithi059 ceph-mon[97450]: pgmap v15914: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:49:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:37.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:37.238402+0000) 2022-09-08T05:49:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:37.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:37.238349+0000) 2022-09-08T05:49:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:38.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:38.238675+0000) 2022-09-08T05:49:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:38.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:38.238597+0000) 2022-09-08T05:49:39.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:38 smithi055 ceph-mon[97033]: pgmap v15915: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:38 smithi059 ceph-mon[97450]: pgmap v15915: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:39.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:39.238823+0000) 2022-09-08T05:49:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:39.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:39.238823+0000) 2022-09-08T05:49:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:40.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:40.238955+0000) 2022-09-08T05:49:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:40.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:40.238992+0000) 2022-09-08T05:49:41.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:40 smithi055 ceph-mon[97033]: pgmap v15916: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:40 smithi059 ceph-mon[97450]: pgmap v15916: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:41.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:41.239167+0000) 2022-09-08T05:49:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:41.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:41.239218+0000) 2022-09-08T05:49:42.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:42 smithi055 ceph-mon[97033]: pgmap v15917: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:49:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:42 smithi059 ceph-mon[97450]: pgmap v15917: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:49:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:42.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:42.239356+0000) 2022-09-08T05:49:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:42.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:42.239450+0000) 2022-09-08T05:49:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:43.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:43.239576+0000) 2022-09-08T05:49:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:43.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:43.239659+0000) 2022-09-08T05:49:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:44.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:44.239796+0000) 2022-09-08T05:49:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:44.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:44.239889+0000) 2022-09-08T05:49:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:44 smithi055 ceph-mon[97033]: pgmap v15918: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:44 smithi059 ceph-mon[97450]: pgmap v15918: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:45.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:45.240026+0000) 2022-09-08T05:49:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:45.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:45.240147+0000) 2022-09-08T05:49:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:46.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:46.240273+0000) 2022-09-08T05:49:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:46.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:46.240357+0000) 2022-09-08T05:49:47.228 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:46 smithi055 ceph-mon[97033]: pgmap v15919: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:49:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:46 smithi059 ceph-mon[97450]: pgmap v15919: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:49:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:47.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:47.240476+0000) 2022-09-08T05:49:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:47.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:47.240568+0000) 2022-09-08T05:49:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:48.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:48.240789+0000) 2022-09-08T05:49:48.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:48.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:48.240797+0000) 2022-09-08T05:49:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:48 smithi055 ceph-mon[97033]: pgmap v15920: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:49.241 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:49:48.946+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:49:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:48 smithi059 ceph-mon[97450]: pgmap v15920: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:49.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:49.241018+0000) 2022-09-08T05:49:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:49.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:49.241022+0000) 2022-09-08T05:49:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:50.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:50.241247+0000) 2022-09-08T05:49:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:50.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:50.241246+0000) 2022-09-08T05:49:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:50 smithi055 ceph-mon[97033]: pgmap v15921: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:50 smithi059 ceph-mon[97450]: pgmap v15921: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:51.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:51.241537+0000) 2022-09-08T05:49:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:51.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:51.241543+0000) 2022-09-08T05:49:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:52 smithi055 ceph-mon[97033]: pgmap v15922: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:49:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:52 smithi059 ceph-mon[97450]: pgmap v15922: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:49:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:52.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:52.241829+0000) 2022-09-08T05:49:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:52.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:52.241828+0000) 2022-09-08T05:49:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:53.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:53.242081+0000) 2022-09-08T05:49:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:53.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:53.242081+0000) 2022-09-08T05:49:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:54.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:54.242342+0000) 2022-09-08T05:49:54.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:54.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:54.242334+0000) 2022-09-08T05:49:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:54 smithi055 ceph-mon[97033]: pgmap v15923: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:55.280 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:54 smithi059 ceph-mon[97450]: pgmap v15923: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:55.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:55.242632+0000) 2022-09-08T05:49:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:55.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:55.242582+0000) 2022-09-08T05:49:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:56.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:56.242891+0000) 2022-09-08T05:49:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:56.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:56.242810+0000) 2022-09-08T05:49:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:56 smithi055 ceph-mon[97033]: pgmap v15924: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:49:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:56 smithi059 ceph-mon[97450]: pgmap v15924: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:49:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:57.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:57.243088+0000) 2022-09-08T05:49:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:57.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:57.243141+0000) 2022-09-08T05:49:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:58.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:58.243260+0000) 2022-09-08T05:49:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:58.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:58.243323+0000) 2022-09-08T05:49:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:49:58 smithi055 ceph-mon[97033]: pgmap v15925: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:49:58 smithi059 ceph-mon[97450]: pgmap v15925: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:49:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:49:59.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:59.243392+0000) 2022-09-08T05:49:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:49:59.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:49:59.243474+0000) 2022-09-08T05:50:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:00.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:00.243623+0000) 2022-09-08T05:50:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:00.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:00.243542+0000) 2022-09-08T05:50:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:00 smithi055 ceph-mon[97033]: pgmap v15926: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T05:50:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T05:50:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T05:50:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T05:50:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T05:50:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:00 smithi059 ceph-mon[97450]: pgmap v15926: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T05:50:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T05:50:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T05:50:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T05:50:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T05:50:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:01.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:01.243861+0000) 2022-09-08T05:50:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:01.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:01.243818+0000) 2022-09-08T05:50:02.402 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:02.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:02.244030+0000) 2022-09-08T05:50:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:02.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:02.244031+0000) 2022-09-08T05:50:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:02 smithi055 ceph-mon[97033]: pgmap v15927: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:50:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:50:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:02 smithi059 ceph-mon[97450]: pgmap v15927: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:50:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:50:03.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:03 smithi055 ceph-mon[97033]: pgmap v15928: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:50:03.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:03 smithi055 ceph-mon[97033]: pgmap v15929: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:50:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:03 smithi059 ceph-mon[97450]: pgmap v15928: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:50:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:03 smithi059 ceph-mon[97450]: pgmap v15929: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:50:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:03.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:03.244249+0000) 2022-09-08T05:50:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:03.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:03.244260+0000) 2022-09-08T05:50:04.245 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:50:03.947+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:50:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:04.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:04.244500+0000) 2022-09-08T05:50:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:04.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:04.244515+0000) 2022-09-08T05:50:05.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:05 smithi055 ceph-mon[97033]: pgmap v15930: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:50:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:05 smithi059 ceph-mon[97450]: pgmap v15930: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:50:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:05.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:05.244682+0000) 2022-09-08T05:50:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:05.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:05.244708+0000) 2022-09-08T05:50:06.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:50:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:50:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:06.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:06.244926+0000) 2022-09-08T05:50:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:06.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:06.244915+0000) 2022-09-08T05:50:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:07 smithi055 ceph-mon[97033]: pgmap v15931: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T05:50:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:50:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:07 smithi059 ceph-mon[97450]: pgmap v15931: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T05:50:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:50:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:07.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:07.245148+0000) 2022-09-08T05:50:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:07.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:07.245124+0000) 2022-09-08T05:50:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:08.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:08.245374+0000) 2022-09-08T05:50:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:08.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:08.245361+0000) 2022-09-08T05:50:09.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:09 smithi055 ceph-mon[97033]: pgmap v15932: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T05:50:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:09 smithi059 ceph-mon[97450]: pgmap v15932: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T05:50:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:09.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:09.245534+0000) 2022-09-08T05:50:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:09.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:09.245570+0000) 2022-09-08T05:50:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:10 smithi055 ceph-mon[97033]: osdmap e1075: 12 total, 11 up, 11 in 2022-09-08T05:50:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:10 smithi059 ceph-mon[97450]: osdmap e1075: 12 total, 11 up, 11 in 2022-09-08T05:50:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:10.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:10.245788+0000) 2022-09-08T05:50:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:10.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:10.245743+0000) 2022-09-08T05:50:11.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:11 smithi055 ceph-mon[97033]: pgmap v15934: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T05:50:11.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:11 smithi059 ceph-mon[97450]: pgmap v15934: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T05:50:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:11.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:11.245953+0000) 2022-09-08T05:50:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:11.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:11.245990+0000) 2022-09-08T05:50:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:12.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:12.246124+0000) 2022-09-08T05:50:12.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:12.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:12.246154+0000) 2022-09-08T05:50:13.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:13 smithi055 ceph-mon[97033]: pgmap v15935: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:50:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:13 smithi059 ceph-mon[97450]: pgmap v15935: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:50:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:13.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:13.246368+0000) 2022-09-08T05:50:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:13.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:13.246389+0000) 2022-09-08T05:50:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:14.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:14.246599+0000) 2022-09-08T05:50:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:14.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:14.246663+0000) 2022-09-08T05:50:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:15 smithi055 ceph-mon[97033]: pgmap v15936: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:50:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:15 smithi059 ceph-mon[97450]: pgmap v15936: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:50:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:15.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:15.246830+0000) 2022-09-08T05:50:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:15.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:15.246938+0000) 2022-09-08T05:50:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:16.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:16.246977+0000) 2022-09-08T05:50:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:16.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:16.247112+0000) 2022-09-08T05:50:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:17 smithi055 ceph-mon[97033]: pgmap v15937: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:50:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:17 smithi059 ceph-mon[97450]: pgmap v15937: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:50:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:17.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:17.247142+0000) 2022-09-08T05:50:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:17.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:17.247264+0000) 2022-09-08T05:50:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:18.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:18.247386+0000) 2022-09-08T05:50:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:18.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:18.247494+0000) 2022-09-08T05:50:19.248 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:50:18.960+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:50:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:19 smithi055 ceph-mon[97033]: pgmap v15938: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:50:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:19 smithi059 ceph-mon[97450]: pgmap v15938: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:50:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:19.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:19.247618+0000) 2022-09-08T05:50:19.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:19.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:19.247762+0000) 2022-09-08T05:50:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:20.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:20.247929+0000) 2022-09-08T05:50:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:20.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:20.248025+0000) 2022-09-08T05:50:21.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:21 smithi055 ceph-mon[97033]: pgmap v15939: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T05:50:21.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:21 smithi055 ceph-mon[97033]: osdmap e1076: 12 total, 11 up, 11 in 2022-09-08T05:50:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:21 smithi059 ceph-mon[97450]: pgmap v15939: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T05:50:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:21 smithi059 ceph-mon[97450]: osdmap e1076: 12 total, 11 up, 11 in 2022-09-08T05:50:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:21.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:21.248082+0000) 2022-09-08T05:50:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:21.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:21.248182+0000) 2022-09-08T05:50:22.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:22.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:22.248283+0000) 2022-09-08T05:50:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:22.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:22.248363+0000) 2022-09-08T05:50:23.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:23 smithi055 ceph-mon[97033]: pgmap v15941: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:50:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:23 smithi059 ceph-mon[97450]: pgmap v15941: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:50:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:23.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:23.248411+0000) 2022-09-08T05:50:23.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:23.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:23.248531+0000) 2022-09-08T05:50:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:24.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:24.248596+0000) 2022-09-08T05:50:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:24.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:24.248722+0000) 2022-09-08T05:50:25.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:25 smithi055 ceph-mon[97033]: pgmap v15942: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:50:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:25 smithi059 ceph-mon[97450]: pgmap v15942: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:50:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:25.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:25.248740+0000) 2022-09-08T05:50:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:25.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:25.248890+0000) 2022-09-08T05:50:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:26.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:26.248944+0000) 2022-09-08T05:50:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:26.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:26.249097+0000) 2022-09-08T05:50:27.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:27 smithi055 ceph-mon[97033]: pgmap v15943: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:50:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:27 smithi059 ceph-mon[97450]: pgmap v15943: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:50:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:27.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:27.249101+0000) 2022-09-08T05:50:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:27.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:27.249238+0000) 2022-09-08T05:50:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:28.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:28.249276+0000) 2022-09-08T05:50:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:28.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:28.249396+0000) 2022-09-08T05:50:29.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:29 smithi055 ceph-mon[97033]: pgmap v15944: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:50:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:29 smithi059 ceph-mon[97450]: pgmap v15944: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:50:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:29.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:29.249456+0000) 2022-09-08T05:50:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:29.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:29.249551+0000) 2022-09-08T05:50:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:30.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:30.249615+0000) 2022-09-08T05:50:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:30.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:30.249724+0000) 2022-09-08T05:50:31.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:31 smithi055 ceph-mon[97033]: pgmap v15945: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:50:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:31 smithi059 ceph-mon[97450]: pgmap v15945: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:50:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:31.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:31.249878+0000) 2022-09-08T05:50:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:31.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:31.249762+0000) 2022-09-08T05:50:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:32.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:32.250009+0000) 2022-09-08T05:50:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:32.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:32.249904+0000) 2022-09-08T05:50:33.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:33 smithi055 ceph-mon[97033]: pgmap v15946: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:50:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:33 smithi059 ceph-mon[97450]: pgmap v15946: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:50:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:33.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:33.250073+0000) 2022-09-08T05:50:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:33.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:33.250233+0000) 2022-09-08T05:50:34.251 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:50:33.972+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:50:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:34.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:34.250331+0000) 2022-09-08T05:50:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:34.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:34.250439+0000) 2022-09-08T05:50:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:35 smithi055 ceph-mon[97033]: pgmap v15947: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:35 smithi059 ceph-mon[97450]: pgmap v15947: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:35.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:35.250497+0000) 2022-09-08T05:50:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:35.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:35.250580+0000) 2022-09-08T05:50:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:36.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:36.250733+0000) 2022-09-08T05:50:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:36.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:36.250806+0000) 2022-09-08T05:50:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:37 smithi055 ceph-mon[97033]: pgmap v15948: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:50:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:37 smithi059 ceph-mon[97450]: pgmap v15948: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:50:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:37.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:37.250878+0000) 2022-09-08T05:50:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:37.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:37.250986+0000) 2022-09-08T05:50:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:38.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:38.251112+0000) 2022-09-08T05:50:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:38.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:38.251225+0000) 2022-09-08T05:50:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:39 smithi055 ceph-mon[97033]: pgmap v15949: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:39 smithi059 ceph-mon[97450]: pgmap v15949: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:39.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:39.251337+0000) 2022-09-08T05:50:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:39.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:39.251464+0000) 2022-09-08T05:50:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:40.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:40.251507+0000) 2022-09-08T05:50:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:40.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:40.251633+0000) 2022-09-08T05:50:41.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:41 smithi055 ceph-mon[97033]: pgmap v15950: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:50:41.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:41 smithi059 ceph-mon[97450]: pgmap v15950: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:50:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:41.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:41.251651+0000) 2022-09-08T05:50:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:41.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:41.251825+0000) 2022-09-08T05:50:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:42.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:42.251884+0000) 2022-09-08T05:50:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:42.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:42.251992+0000) 2022-09-08T05:50:43.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:43 smithi055 ceph-mon[97033]: pgmap v15951: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:43 smithi059 ceph-mon[97450]: pgmap v15951: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:43.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:43.252110+0000) 2022-09-08T05:50:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:43.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:43.252213+0000) 2022-09-08T05:50:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:44.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:44.252384+0000) 2022-09-08T05:50:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:44.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:44.252483+0000) 2022-09-08T05:50:45.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:45 smithi055 ceph-mon[97033]: pgmap v15952: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:45 smithi059 ceph-mon[97450]: pgmap v15952: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:45.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:45.252631+0000) 2022-09-08T05:50:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:45.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:45.252701+0000) 2022-09-08T05:50:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:46.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:46.252819+0000) 2022-09-08T05:50:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:46.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:46.252852+0000) 2022-09-08T05:50:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:47 smithi055 ceph-mon[97033]: pgmap v15953: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:50:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:47 smithi059 ceph-mon[97450]: pgmap v15953: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:50:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:47.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:47.253031+0000) 2022-09-08T05:50:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:47.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:47.253015+0000) 2022-09-08T05:50:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:48.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:48.253206+0000) 2022-09-08T05:50:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:48.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:48.253196+0000) 2022-09-08T05:50:49.254 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:50:48.981+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:50:49.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:49 smithi055 ceph-mon[97033]: pgmap v15954: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:49 smithi059 ceph-mon[97450]: pgmap v15954: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:49.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:49.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:49.253427+0000) 2022-09-08T05:50:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:49.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:49.253421+0000) 2022-09-08T05:50:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:50.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:50.253630+0000) 2022-09-08T05:50:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:50.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:50.253662+0000) 2022-09-08T05:50:51.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:51 smithi055 ceph-mon[97033]: pgmap v15955: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:50:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:51 smithi059 ceph-mon[97450]: pgmap v15955: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:50:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:51.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:51.253800+0000) 2022-09-08T05:50:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:51.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:51.253817+0000) 2022-09-08T05:50:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:52.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:52.253975+0000) 2022-09-08T05:50:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:52.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:52.253996+0000) 2022-09-08T05:50:53.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:53 smithi055 ceph-mon[97033]: pgmap v15956: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:53 smithi059 ceph-mon[97450]: pgmap v15956: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:53.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:53.254155+0000) 2022-09-08T05:50:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:53.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:53.254161+0000) 2022-09-08T05:50:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:54.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:54.254334+0000) 2022-09-08T05:50:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:54.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:54.254339+0000) 2022-09-08T05:50:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:55 smithi055 ceph-mon[97033]: pgmap v15957: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:55 smithi059 ceph-mon[97450]: pgmap v15957: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:55.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:55.254478+0000) 2022-09-08T05:50:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:55.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:55.254466+0000) 2022-09-08T05:50:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:56.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:56.254717+0000) 2022-09-08T05:50:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:56.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:56.254681+0000) 2022-09-08T05:50:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:57 smithi055 ceph-mon[97033]: pgmap v15958: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:50:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:57 smithi059 ceph-mon[97450]: pgmap v15958: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:50:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:57.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:57.254950+0000) 2022-09-08T05:50:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:57.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:57.254835+0000) 2022-09-08T05:50:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:58.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:58.255216+0000) 2022-09-08T05:50:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:58.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:58.255038+0000) 2022-09-08T05:50:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:50:59 smithi055 ceph-mon[97033]: pgmap v15959: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:50:59 smithi059 ceph-mon[97450]: pgmap v15959: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:50:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:50:59.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:59.255460+0000) 2022-09-08T05:50:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:50:59.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:50:59.255215+0000) 2022-09-08T05:51:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:00.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:00.255615+0000) 2022-09-08T05:51:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:00.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:00.255457+0000) 2022-09-08T05:51:01.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:01 smithi055 ceph-mon[97033]: pgmap v15960: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:51:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:01 smithi059 ceph-mon[97450]: pgmap v15960: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:51:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:01.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:01.255812+0000) 2022-09-08T05:51:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:01.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:01.255726+0000) 2022-09-08T05:51:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:51:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:51:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:51:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:51:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:51:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:51:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:02.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:02.255999+0000) 2022-09-08T05:51:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:02.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:02.255929+0000) 2022-09-08T05:51:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:03.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:03.256174+0000) 2022-09-08T05:51:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:03.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:03.256141+0000) 2022-09-08T05:51:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:03 smithi055 ceph-mon[97033]: pgmap v15961: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:51:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:51:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:51:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:03 smithi059 ceph-mon[97450]: pgmap v15961: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:51:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:51:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:51:04.257 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:51:03.994+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:51:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:04.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:04.256392+0000) 2022-09-08T05:51:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:04.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:04.256375+0000) 2022-09-08T05:51:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:05.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:05.256562+0000) 2022-09-08T05:51:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:05.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:05.256580+0000) 2022-09-08T05:51:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:05 smithi055 ceph-mon[97033]: pgmap v15962: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:51:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:51:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:51:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:51:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:51:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:05.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:05.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:05 smithi059 ceph-mon[97450]: pgmap v15962: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:51:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:51:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:51:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:51:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:51:06.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:06.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:06.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:06.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:06.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:06.256724+0000) 2022-09-08T05:51:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:06.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:06.256741+0000) 2022-09-08T05:51:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:06 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T05:51:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:06 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T05:51:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:51:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:51:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:06 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T05:51:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:06 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T05:51:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:51:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T05:51:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:51:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:07.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:07.256936+0000) 2022-09-08T05:51:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:07.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:07.256973+0000) 2022-09-08T05:51:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:07 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T05:51:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:07 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T05:51:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:07 smithi055 ceph-mon[97033]: pgmap v15963: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:51:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:07 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T05:51:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:07 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T05:51:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:07 smithi059 ceph-mon[97450]: pgmap v15963: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:51:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:08.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:08.257189+0000) 2022-09-08T05:51:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:08.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:08.257213+0000) 2022-09-08T05:51:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:09.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:09.257468+0000) 2022-09-08T05:51:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:09.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:09.257448+0000) 2022-09-08T05:51:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:09 smithi055 ceph-mon[97033]: pgmap v15964: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:51:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:09 smithi059 ceph-mon[97450]: pgmap v15964: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:51:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:10.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:10.257741+0000) 2022-09-08T05:51:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:10.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:10.257754+0000) 2022-09-08T05:51:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:10 smithi055 ceph-mon[97033]: osdmap e1077: 12 total, 11 up, 11 in 2022-09-08T05:51:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:10 smithi059 ceph-mon[97450]: osdmap e1077: 12 total, 11 up, 11 in 2022-09-08T05:51:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:11.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:11.257952+0000) 2022-09-08T05:51:11.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:11.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:11.257994+0000) 2022-09-08T05:51:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:11 smithi055 ceph-mon[97033]: pgmap v15966: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:51:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:11 smithi059 ceph-mon[97450]: pgmap v15966: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:51:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:12.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:12.258153+0000) 2022-09-08T05:51:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:12.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:12.258164+0000) 2022-09-08T05:51:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:13.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:13.258376+0000) 2022-09-08T05:51:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:13.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:13.258425+0000) 2022-09-08T05:51:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:13 smithi055 ceph-mon[97033]: pgmap v15967: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:51:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:13 smithi059 ceph-mon[97450]: pgmap v15967: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:51:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:14.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:14.258680+0000) 2022-09-08T05:51:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:14.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:14.258705+0000) 2022-09-08T05:51:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:15.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:15.258904+0000) 2022-09-08T05:51:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:15.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:15.258906+0000) 2022-09-08T05:51:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:15 smithi055 ceph-mon[97033]: pgmap v15968: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:51:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:15 smithi059 ceph-mon[97450]: pgmap v15968: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T05:51:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:16.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:16.259204+0000) 2022-09-08T05:51:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:16.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:16.259206+0000) 2022-09-08T05:51:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:17.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:17.259442+0000) 2022-09-08T05:51:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:17.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:17.259468+0000) 2022-09-08T05:51:17.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:17 smithi055 ceph-mon[97033]: pgmap v15969: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:51:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:17 smithi059 ceph-mon[97450]: pgmap v15969: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:51:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:18.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:18.259648+0000) 2022-09-08T05:51:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:18.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:18.259628+0000) 2022-09-08T05:51:19.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:51:19.011+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:51:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:19.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:19.259887+0000) 2022-09-08T05:51:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:19.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:19.259870+0000) 2022-09-08T05:51:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:19 smithi055 ceph-mon[97033]: pgmap v15970: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:51:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:19 smithi059 ceph-mon[97450]: pgmap v15970: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:51:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:20.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:20.260144+0000) 2022-09-08T05:51:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:20.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:20.260142+0000) 2022-09-08T05:51:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:21.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:21.260434+0000) 2022-09-08T05:51:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:21.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:21.260434+0000) 2022-09-08T05:51:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:21 smithi055 ceph-mon[97033]: pgmap v15971: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.5 KiB/s rd, 196 B/s wr, 9 op/s 2022-09-08T05:51:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:21 smithi055 ceph-mon[97033]: osdmap e1078: 12 total, 11 up, 11 in 2022-09-08T05:51:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:21 smithi059 ceph-mon[97450]: pgmap v15971: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.5 KiB/s rd, 196 B/s wr, 9 op/s 2022-09-08T05:51:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:21 smithi059 ceph-mon[97450]: osdmap e1078: 12 total, 11 up, 11 in 2022-09-08T05:51:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:22.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:22.260740+0000) 2022-09-08T05:51:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:22.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:22.260740+0000) 2022-09-08T05:51:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:23.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:23.260995+0000) 2022-09-08T05:51:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:23.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:23.260995+0000) 2022-09-08T05:51:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:23 smithi055 ceph-mon[97033]: pgmap v15973: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:51:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:23 smithi059 ceph-mon[97450]: pgmap v15973: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:51:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:24.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:24.261302+0000) 2022-09-08T05:51:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:24.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:24.261301+0000) 2022-09-08T05:51:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:25.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:25.261476+0000) 2022-09-08T05:51:25.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:25.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:25.261481+0000) 2022-09-08T05:51:25.978 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:25 smithi055 ceph-mon[97033]: pgmap v15974: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:51:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:25 smithi059 ceph-mon[97450]: pgmap v15974: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:51:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:26.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:26.261783+0000) 2022-09-08T05:51:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:26.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:26.261783+0000) 2022-09-08T05:51:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:27.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:27.262019+0000) 2022-09-08T05:51:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:27.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:27.262017+0000) 2022-09-08T05:51:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:27 smithi055 ceph-mon[97033]: pgmap v15975: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:51:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:27 smithi059 ceph-mon[97450]: pgmap v15975: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:51:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:28.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:28.262196+0000) 2022-09-08T05:51:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:28.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:28.262174+0000) 2022-09-08T05:51:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:29.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:29.262437+0000) 2022-09-08T05:51:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:29.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:29.262422+0000) 2022-09-08T05:51:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:29 smithi055 ceph-mon[97033]: pgmap v15976: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:51:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:29 smithi059 ceph-mon[97450]: pgmap v15976: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:51:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:30.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:30.262608+0000) 2022-09-08T05:51:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:30.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:30.262614+0000) 2022-09-08T05:51:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:31.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:31.262881+0000) 2022-09-08T05:51:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:31.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:31.262870+0000) 2022-09-08T05:51:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:31 smithi055 ceph-mon[97033]: pgmap v15977: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:51:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:31 smithi059 ceph-mon[97450]: pgmap v15977: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:51:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:32.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:32.263131+0000) 2022-09-08T05:51:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:32.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:32.263129+0000) 2022-09-08T05:51:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:33.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:33.263410+0000) 2022-09-08T05:51:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:33.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:33.263398+0000) 2022-09-08T05:51:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:33 smithi055 ceph-mon[97033]: pgmap v15978: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:51:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:33 smithi055 ceph-mon[97033]: pgmap v15979: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:51:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:33 smithi055 ceph-mon[97033]: pgmap v15980: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:51:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:51:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:33 smithi059 ceph-mon[97450]: pgmap v15978: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:51:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:33 smithi059 ceph-mon[97450]: pgmap v15979: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:51:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:33 smithi059 ceph-mon[97450]: pgmap v15980: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:51:33.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:51:34.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:51:34.021+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:51:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:34.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:34.263727+0000) 2022-09-08T05:51:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:34.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:34.263699+0000) 2022-09-08T05:51:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:35.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:35.263898+0000) 2022-09-08T05:51:35.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:35.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:35.263859+0000) 2022-09-08T05:51:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:35 smithi055 ceph-mon[97033]: pgmap v15981: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:51:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:35 smithi059 ceph-mon[97450]: pgmap v15981: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:51:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:36.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:36.264061+0000) 2022-09-08T05:51:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:36.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:36.264088+0000) 2022-09-08T05:51:37.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:37 smithi055 ceph-mon[97033]: pgmap v15982: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:51:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:37 smithi059 ceph-mon[97450]: pgmap v15982: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:51:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:37.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:37.264318+0000) 2022-09-08T05:51:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:37.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:37.264296+0000) 2022-09-08T05:51:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:38.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:38.264561+0000) 2022-09-08T05:51:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:38.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:38.264554+0000) 2022-09-08T05:51:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:39.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:39.264696+0000) 2022-09-08T05:51:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:39.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:39.264748+0000) 2022-09-08T05:51:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:39 smithi055 ceph-mon[97033]: pgmap v15983: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:51:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:39 smithi059 ceph-mon[97450]: pgmap v15983: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:51:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:40.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:40.264839+0000) 2022-09-08T05:51:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:40.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:40.264942+0000) 2022-09-08T05:51:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:41.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:41.265070+0000) 2022-09-08T05:51:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:41.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:41.265145+0000) 2022-09-08T05:51:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:41 smithi055 ceph-mon[97033]: pgmap v15984: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:51:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:41 smithi059 ceph-mon[97450]: pgmap v15984: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:51:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:42.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:42.265238+0000) 2022-09-08T05:51:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:42.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:42.265362+0000) 2022-09-08T05:51:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:43.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:43.265456+0000) 2022-09-08T05:51:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:43.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:43.265580+0000) 2022-09-08T05:51:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:43 smithi055 ceph-mon[97033]: pgmap v15985: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:51:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:43 smithi059 ceph-mon[97450]: pgmap v15985: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:51:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:44.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:44.265741+0000) 2022-09-08T05:51:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:44.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:44.265822+0000) 2022-09-08T05:51:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:45.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:45.265974+0000) 2022-09-08T05:51:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:45.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:45.266007+0000) 2022-09-08T05:51:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:45 smithi055 ceph-mon[97033]: pgmap v15986: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:51:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:45 smithi059 ceph-mon[97450]: pgmap v15986: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:51:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:46.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:46.266232+0000) 2022-09-08T05:51:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:46.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:46.266230+0000) 2022-09-08T05:51:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:47 smithi055 ceph-mon[97033]: pgmap v15987: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:51:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:47 smithi059 ceph-mon[97450]: pgmap v15987: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:51:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:47.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:47.266475+0000) 2022-09-08T05:51:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:47.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:47.266494+0000) 2022-09-08T05:51:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:48.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:48.266788+0000) 2022-09-08T05:51:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:48.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:48.266756+0000) 2022-09-08T05:51:49.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:51:49.036+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:51:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:49.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:49.267000+0000) 2022-09-08T05:51:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:49.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:49.266997+0000) 2022-09-08T05:51:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:49 smithi055 ceph-mon[97033]: pgmap v15988: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:51:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:49 smithi059 ceph-mon[97450]: pgmap v15988: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:51:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:50.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:50.267248+0000) 2022-09-08T05:51:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:50.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:50.267256+0000) 2022-09-08T05:51:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:51.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:51.267507+0000) 2022-09-08T05:51:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:51.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:51.267466+0000) 2022-09-08T05:51:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:51 smithi055 ceph-mon[97033]: pgmap v15989: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:51:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:51 smithi059 ceph-mon[97450]: pgmap v15989: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:51:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:52.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:52.267743+0000) 2022-09-08T05:51:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:52.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:52.267750+0000) 2022-09-08T05:51:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:53.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:53.268008+0000) 2022-09-08T05:51:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:53.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:53.268012+0000) 2022-09-08T05:51:53.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:53 smithi055 ceph-mon[97033]: pgmap v15990: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:51:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:53 smithi059 ceph-mon[97450]: pgmap v15990: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:51:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:54.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:54.268273+0000) 2022-09-08T05:51:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:54.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:54.268262+0000) 2022-09-08T05:51:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:55.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:55.268562+0000) 2022-09-08T05:51:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:55.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:55.268541+0000) 2022-09-08T05:51:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:55 smithi055 ceph-mon[97033]: pgmap v15991: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:51:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:55 smithi059 ceph-mon[97450]: pgmap v15991: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:51:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:56.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:56.268783+0000) 2022-09-08T05:51:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:56.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:56.268770+0000) 2022-09-08T05:51:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:57 smithi055 ceph-mon[97033]: pgmap v15992: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:51:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:57 smithi059 ceph-mon[97450]: pgmap v15992: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:51:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:57.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:57.269020+0000) 2022-09-08T05:51:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:57.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:57.269011+0000) 2022-09-08T05:51:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:58.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:58.269330+0000) 2022-09-08T05:51:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:58.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:58.269259+0000) 2022-09-08T05:51:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:51:59.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:59.269483+0000) 2022-09-08T05:51:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:51:59.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:51:59.269586+0000) 2022-09-08T05:51:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:51:59 smithi055 ceph-mon[97033]: pgmap v15993: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:51:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:51:59 smithi059 ceph-mon[97450]: pgmap v15993: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:00.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:00.269697+0000) 2022-09-08T05:52:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:00.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:00.269799+0000) 2022-09-08T05:52:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:01.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:01.269862+0000) 2022-09-08T05:52:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:01.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:01.269988+0000) 2022-09-08T05:52:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:01 smithi055 ceph-mon[97033]: pgmap v15994: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:52:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:01 smithi059 ceph-mon[97450]: pgmap v15994: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:52:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:02.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:02.270096+0000) 2022-09-08T05:52:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:02.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:02.270164+0000) 2022-09-08T05:52:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:03.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:03.270275+0000) 2022-09-08T05:52:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:03.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:03.270409+0000) 2022-09-08T05:52:03.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:03 smithi055 ceph-mon[97033]: pgmap v15995: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:03.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:03 smithi059 ceph-mon[97450]: pgmap v15995: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:04.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:52:04.052+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:52:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:04.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:04.270537+0000) 2022-09-08T05:52:04.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:04.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:04.270677+0000) 2022-09-08T05:52:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:05.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:05.270787+0000) 2022-09-08T05:52:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:05.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:05.270926+0000) 2022-09-08T05:52:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:05 smithi055 ceph-mon[97033]: pgmap v15996: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:05 smithi059 ceph-mon[97450]: pgmap v15996: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:06.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:06.271035+0000) 2022-09-08T05:52:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:06.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:06.271167+0000) 2022-09-08T05:52:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:52:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:52:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:07.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:07.271395+0000) 2022-09-08T05:52:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:07.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:07.271322+0000) 2022-09-08T05:52:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:07 smithi055 ceph-mon[97033]: pgmap v15997: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:52:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:52:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:07 smithi059 ceph-mon[97450]: pgmap v15997: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:52:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:52:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:08.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:08.271571+0000) 2022-09-08T05:52:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:08.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:08.271534+0000) 2022-09-08T05:52:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:09.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:09.271744+0000) 2022-09-08T05:52:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:09.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:09.271796+0000) 2022-09-08T05:52:09.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:09 smithi055 ceph-mon[97033]: pgmap v15998: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:52:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:09 smithi059 ceph-mon[97450]: pgmap v15998: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:52:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:10.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:10.271953+0000) 2022-09-08T05:52:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:10.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:10.271983+0000) 2022-09-08T05:52:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:10 smithi055 ceph-mon[97033]: osdmap e1079: 12 total, 11 up, 11 in 2022-09-08T05:52:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:10 smithi059 ceph-mon[97450]: osdmap e1079: 12 total, 11 up, 11 in 2022-09-08T05:52:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:11.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:11.272198+0000) 2022-09-08T05:52:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:11.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:11.272249+0000) 2022-09-08T05:52:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:11 smithi055 ceph-mon[97033]: pgmap v16000: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:52:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:11 smithi055 ceph-mon[97033]: osdmap e1080: 12 total, 11 up, 11 in 2022-09-08T05:52:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:11 smithi059 ceph-mon[97450]: pgmap v16000: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:52:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:11 smithi059 ceph-mon[97450]: osdmap e1080: 12 total, 11 up, 11 in 2022-09-08T05:52:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:12.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:12.272399+0000) 2022-09-08T05:52:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:12.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:12.272449+0000) 2022-09-08T05:52:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:13.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:13.272611+0000) 2022-09-08T05:52:13.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:13.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:13.272626+0000) 2022-09-08T05:52:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:13 smithi055 ceph-mon[97033]: pgmap v16002: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:52:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:13 smithi059 ceph-mon[97450]: pgmap v16002: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:52:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:14.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:14.272870+0000) 2022-09-08T05:52:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:14.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:14.272899+0000) 2022-09-08T05:52:15.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:15.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:15.273149+0000) 2022-09-08T05:52:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:15.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:15.273148+0000) 2022-09-08T05:52:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:15 smithi055 ceph-mon[97033]: pgmap v16003: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:52:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:15 smithi059 ceph-mon[97450]: pgmap v16003: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:52:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:16.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:16.273351+0000) 2022-09-08T05:52:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:16.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:16.273380+0000) 2022-09-08T05:52:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:17 smithi055 ceph-mon[97033]: pgmap v16004: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:52:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:17 smithi059 ceph-mon[97450]: pgmap v16004: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:52:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:17.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:17.273622+0000) 2022-09-08T05:52:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:17.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:17.273699+0000) 2022-09-08T05:52:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:18.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:18.273881+0000) 2022-09-08T05:52:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:18.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:18.273899+0000) 2022-09-08T05:52:19.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:52:19.064+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:52:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:19.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:19.274127+0000) 2022-09-08T05:52:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:19.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:19.274139+0000) 2022-09-08T05:52:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:19 smithi055 ceph-mon[97033]: pgmap v16005: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T05:52:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:19 smithi059 ceph-mon[97450]: pgmap v16005: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T05:52:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:20.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:20.274430+0000) 2022-09-08T05:52:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:20.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:20.274362+0000) 2022-09-08T05:52:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:21.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:21.274606+0000) 2022-09-08T05:52:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:21.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:21.274743+0000) 2022-09-08T05:52:21.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:21 smithi055 ceph-mon[97033]: pgmap v16006: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:52:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:21 smithi059 ceph-mon[97450]: pgmap v16006: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:52:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:22.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:22.274911+0000) 2022-09-08T05:52:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:22.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:22.274950+0000) 2022-09-08T05:52:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:23.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:23.275192+0000) 2022-09-08T05:52:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:23.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:23.275185+0000) 2022-09-08T05:52:23.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:23 smithi055 ceph-mon[97033]: pgmap v16007: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T05:52:23.780 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:23 smithi059 ceph-mon[97450]: pgmap v16007: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T05:52:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:24.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:24.275476+0000) 2022-09-08T05:52:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:24.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:24.275476+0000) 2022-09-08T05:52:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:25.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:25.275754+0000) 2022-09-08T05:52:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:25.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:25.275748+0000) 2022-09-08T05:52:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:25 smithi055 ceph-mon[97033]: pgmap v16008: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:52:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:25 smithi059 ceph-mon[97450]: pgmap v16008: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:52:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:26.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:26.276021+0000) 2022-09-08T05:52:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:26.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:26.275964+0000) 2022-09-08T05:52:27.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:27 smithi055 ceph-mon[97033]: pgmap v16009: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:52:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:27 smithi059 ceph-mon[97450]: pgmap v16009: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:52:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:27.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:27.276182+0000) 2022-09-08T05:52:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:27.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:27.276171+0000) 2022-09-08T05:52:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:28.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:28.276384+0000) 2022-09-08T05:52:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:28.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:28.276385+0000) 2022-09-08T05:52:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:29.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:29.276608+0000) 2022-09-08T05:52:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:29.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:29.276609+0000) 2022-09-08T05:52:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:29 smithi055 ceph-mon[97033]: pgmap v16010: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:29 smithi059 ceph-mon[97450]: pgmap v16010: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:30.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:30.276870+0000) 2022-09-08T05:52:30.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:30.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:30.276820+0000) 2022-09-08T05:52:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:31.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:31.277126+0000) 2022-09-08T05:52:31.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:31.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:31.277075+0000) 2022-09-08T05:52:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:31 smithi055 ceph-mon[97033]: pgmap v16011: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:52:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:31 smithi059 ceph-mon[97450]: pgmap v16011: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:52:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:32.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:32.277348+0000) 2022-09-08T05:52:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:32.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:32.277320+0000) 2022-09-08T05:52:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:33.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:33.277571+0000) 2022-09-08T05:52:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:33.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:33.277582+0000) 2022-09-08T05:52:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:33 smithi055 ceph-mon[97033]: pgmap v16012: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:52:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:52:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:52:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:52:33.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:52:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:33 smithi059 ceph-mon[97450]: pgmap v16012: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:52:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:52:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:52:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:52:33.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:52:34.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:34.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:34.277914+0000) 2022-09-08T05:52:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:34.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:34.277903+0000) 2022-09-08T05:52:34.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:52:34.069+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:52:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:35.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:35.278197+0000) 2022-09-08T05:52:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:35.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:35.278146+0000) 2022-09-08T05:52:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:35 smithi055 ceph-mon[97033]: pgmap v16013: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:35 smithi059 ceph-mon[97450]: pgmap v16013: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:36.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:36.278470+0000) 2022-09-08T05:52:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:36.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:36.278396+0000) 2022-09-08T05:52:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:37 smithi055 ceph-mon[97033]: pgmap v16014: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:52:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:37 smithi059 ceph-mon[97450]: pgmap v16014: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:52:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:37.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:37.278688+0000) 2022-09-08T05:52:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:37.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:37.278677+0000) 2022-09-08T05:52:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:38.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:38.278883+0000) 2022-09-08T05:52:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:38.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:38.278885+0000) 2022-09-08T05:52:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:39.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:39.279107+0000) 2022-09-08T05:52:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:39.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:39.279122+0000) 2022-09-08T05:52:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:39 smithi055 ceph-mon[97033]: pgmap v16015: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:39 smithi059 ceph-mon[97450]: pgmap v16015: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:40.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:40.279360+0000) 2022-09-08T05:52:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:40.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:40.279348+0000) 2022-09-08T05:52:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:41.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:41.279568+0000) 2022-09-08T05:52:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:41.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:41.279576+0000) 2022-09-08T05:52:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:41 smithi055 ceph-mon[97033]: pgmap v16016: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:52:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:41 smithi059 ceph-mon[97450]: pgmap v16016: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:52:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:42.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:42.279804+0000) 2022-09-08T05:52:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:42.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:42.279814+0000) 2022-09-08T05:52:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:43.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:43.280068+0000) 2022-09-08T05:52:43.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:43.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:43.280081+0000) 2022-09-08T05:52:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:43 smithi055 ceph-mon[97033]: pgmap v16017: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:43 smithi059 ceph-mon[97450]: pgmap v16017: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:44.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:44.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:44.280275+0000) 2022-09-08T05:52:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:44.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:44.280248+0000) 2022-09-08T05:52:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:45.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:45.280441+0000) 2022-09-08T05:52:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:45.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:45.280455+0000) 2022-09-08T05:52:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:45 smithi055 ceph-mon[97033]: pgmap v16018: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:45 smithi059 ceph-mon[97450]: pgmap v16018: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:46.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:46.280716+0000) 2022-09-08T05:52:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:46.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:46.280698+0000) 2022-09-08T05:52:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:47 smithi055 ceph-mon[97033]: pgmap v16019: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:52:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:47 smithi059 ceph-mon[97450]: pgmap v16019: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:52:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:47.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:47.280916+0000) 2022-09-08T05:52:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:47.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:47.280860+0000) 2022-09-08T05:52:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:48.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:48.281088+0000) 2022-09-08T05:52:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:48.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:48.281054+0000) 2022-09-08T05:52:49.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:52:49.075+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:52:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:49.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:49.281341+0000) 2022-09-08T05:52:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:49.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:49.281306+0000) 2022-09-08T05:52:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:49 smithi055 ceph-mon[97033]: pgmap v16020: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:49 smithi059 ceph-mon[97450]: pgmap v16020: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:50.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:50.281580+0000) 2022-09-08T05:52:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:50.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:50.281588+0000) 2022-09-08T05:52:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:51.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:51.281766+0000) 2022-09-08T05:52:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:51.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:51.281759+0000) 2022-09-08T05:52:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:51 smithi055 ceph-mon[97033]: pgmap v16021: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:52:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:51 smithi059 ceph-mon[97450]: pgmap v16021: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:52:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:52.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:52.281938+0000) 2022-09-08T05:52:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:52.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:52.281931+0000) 2022-09-08T05:52:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:53.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:53.282192+0000) 2022-09-08T05:52:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:53.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:53.282173+0000) 2022-09-08T05:52:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:53 smithi055 ceph-mon[97033]: pgmap v16022: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:53 smithi059 ceph-mon[97450]: pgmap v16022: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:54.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:54.282475+0000) 2022-09-08T05:52:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:54.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:54.282449+0000) 2022-09-08T05:52:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:55.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:55.282767+0000) 2022-09-08T05:52:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:55.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:55.282736+0000) 2022-09-08T05:52:55.728 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:55 smithi055 ceph-mon[97033]: pgmap v16023: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:55 smithi059 ceph-mon[97450]: pgmap v16023: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:56.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:56.283037+0000) 2022-09-08T05:52:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:56.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:56.282986+0000) 2022-09-08T05:52:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:57 smithi055 ceph-mon[97033]: pgmap v16024: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:52:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:57 smithi059 ceph-mon[97450]: pgmap v16024: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:52:57.606 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:57.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:57.283149+0000) 2022-09-08T05:52:57.607 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:57.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:57.283180+0000) 2022-09-08T05:52:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:58.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:58.283312+0000) 2022-09-08T05:52:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:58.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:58.283284+0000) 2022-09-08T05:52:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:52:59.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:59.283520+0000) 2022-09-08T05:52:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:52:59.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:52:59.283501+0000) 2022-09-08T05:52:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:52:59 smithi055 ceph-mon[97033]: pgmap v16025: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:52:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:52:59 smithi059 ceph-mon[97450]: pgmap v16025: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:53:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:00.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:00.283758+0000) 2022-09-08T05:53:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:00.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:00.283768+0000) 2022-09-08T05:53:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:01.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:01.283895+0000) 2022-09-08T05:53:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:01.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:01.283934+0000) 2022-09-08T05:53:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:01 smithi055 ceph-mon[97033]: pgmap v16026: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:53:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:01 smithi059 ceph-mon[97450]: pgmap v16026: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:53:02.565 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:02.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:02.284108+0000) 2022-09-08T05:53:02.566 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:02.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:02.284161+0000) 2022-09-08T05:53:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:03.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:03.284407+0000) 2022-09-08T05:53:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:03.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:03.284332+0000) 2022-09-08T05:53:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:03 smithi055 ceph-mon[97033]: pgmap v16027: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:53:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:03 smithi055 ceph-mon[97033]: pgmap v16028: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:53:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:03 smithi055 ceph-mon[97033]: pgmap v16029: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:53:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:53:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:03 smithi059 ceph-mon[97450]: pgmap v16027: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:53:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:03 smithi059 ceph-mon[97450]: pgmap v16028: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:53:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:03 smithi059 ceph-mon[97450]: pgmap v16029: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:53:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:53:04.402 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:04.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:04.284696+0000) 2022-09-08T05:53:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:04.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:04.284624+0000) 2022-09-08T05:53:04.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:53:04.087+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:53:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:05.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:05.284884+0000) 2022-09-08T05:53:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:05.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:05.284811+0000) 2022-09-08T05:53:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:05 smithi055 ceph-mon[97033]: pgmap v16030: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:53:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:05 smithi059 ceph-mon[97450]: pgmap v16030: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:53:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:06.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:06.285124+0000) 2022-09-08T05:53:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:06.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:06.285042+0000) 2022-09-08T05:53:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:53:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:53:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:07.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:07.285356+0000) 2022-09-08T05:53:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:07.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:07.285271+0000) 2022-09-08T05:53:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:07 smithi055 ceph-mon[97033]: pgmap v16031: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T05:53:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:53:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:07 smithi059 ceph-mon[97450]: pgmap v16031: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T05:53:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:53:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:08.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:08.285597+0000) 2022-09-08T05:53:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:08.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:08.285508+0000) 2022-09-08T05:53:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:09.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:09.285834+0000) 2022-09-08T05:53:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:09.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:09.285727+0000) 2022-09-08T05:53:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:09 smithi055 ceph-mon[97033]: pgmap v16032: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T05:53:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:09 smithi059 ceph-mon[97450]: pgmap v16032: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T05:53:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:10.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:10.286079+0000) 2022-09-08T05:53:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:10.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:10.285966+0000) 2022-09-08T05:53:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:10 smithi055 ceph-mon[97033]: osdmap e1081: 12 total, 11 up, 11 in 2022-09-08T05:53:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:10 smithi059 ceph-mon[97450]: osdmap e1081: 12 total, 11 up, 11 in 2022-09-08T05:53:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:11.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:11.286263+0000) 2022-09-08T05:53:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:11.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:11.286172+0000) 2022-09-08T05:53:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:11 smithi055 ceph-mon[97033]: pgmap v16034: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:53:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:11 smithi059 ceph-mon[97450]: pgmap v16034: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:53:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:12.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:12.286496+0000) 2022-09-08T05:53:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:12.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:12.286419+0000) 2022-09-08T05:53:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:13.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:13.286760+0000) 2022-09-08T05:53:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:13.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:13.286609+0000) 2022-09-08T05:53:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:13 smithi055 ceph-mon[97033]: pgmap v16035: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:53:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:13 smithi059 ceph-mon[97450]: pgmap v16035: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:53:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:14.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:14.286981+0000) 2022-09-08T05:53:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:14.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:14.286765+0000) 2022-09-08T05:53:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:15.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:15.287143+0000) 2022-09-08T05:53:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:15.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:15.286989+0000) 2022-09-08T05:53:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:15 smithi055 ceph-mon[97033]: pgmap v16036: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:53:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:15 smithi059 ceph-mon[97450]: pgmap v16036: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:53:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:16.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:16.287294+0000) 2022-09-08T05:53:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:16.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:16.287193+0000) 2022-09-08T05:53:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:17 smithi055 ceph-mon[97033]: pgmap v16037: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:53:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:17 smithi059 ceph-mon[97450]: pgmap v16037: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:53:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:17.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:17.287464+0000) 2022-09-08T05:53:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:17.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:17.287409+0000) 2022-09-08T05:53:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:18.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:18.287725+0000) 2022-09-08T05:53:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:18.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:18.287659+0000) 2022-09-08T05:53:19.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:53:19.090+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:53:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:19.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:19.287829+0000) 2022-09-08T05:53:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:19.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:19.287908+0000) 2022-09-08T05:53:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:19 smithi055 ceph-mon[97033]: pgmap v16038: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:53:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:19 smithi059 ceph-mon[97450]: pgmap v16038: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:53:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:20.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:20.288082+0000) 2022-09-08T05:53:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:20.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:20.288030+0000) 2022-09-08T05:53:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:21.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:21.288330+0000) 2022-09-08T05:53:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:21.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:21.288279+0000) 2022-09-08T05:53:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:21 smithi055 ceph-mon[97033]: pgmap v16039: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 186 B/s wr, 8 op/s 2022-09-08T05:53:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:21 smithi055 ceph-mon[97033]: osdmap e1082: 12 total, 11 up, 11 in 2022-09-08T05:53:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:21 smithi059 ceph-mon[97450]: pgmap v16039: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 186 B/s wr, 8 op/s 2022-09-08T05:53:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:21 smithi059 ceph-mon[97450]: osdmap e1082: 12 total, 11 up, 11 in 2022-09-08T05:53:22.616 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:22.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:22.288549+0000) 2022-09-08T05:53:22.617 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:22.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:22.288550+0000) 2022-09-08T05:53:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:23.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:23.288801+0000) 2022-09-08T05:53:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:23.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:23.288792+0000) 2022-09-08T05:53:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:23 smithi055 ceph-mon[97033]: pgmap v16041: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:53:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:23 smithi059 ceph-mon[97450]: pgmap v16041: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:53:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:24.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:24.289019+0000) 2022-09-08T05:53:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:24.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:24.289025+0000) 2022-09-08T05:53:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:25.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:25.289298+0000) 2022-09-08T05:53:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:25.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:25.289294+0000) 2022-09-08T05:53:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:25 smithi055 ceph-mon[97033]: pgmap v16042: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:53:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:25 smithi059 ceph-mon[97450]: pgmap v16042: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:53:26.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:26.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:26.289525+0000) 2022-09-08T05:53:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:26.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:26.289531+0000) 2022-09-08T05:53:27.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:27 smithi055 ceph-mon[97033]: pgmap v16043: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:53:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:27 smithi059 ceph-mon[97450]: pgmap v16043: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:53:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:27.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:27.289718+0000) 2022-09-08T05:53:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:27.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:27.289689+0000) 2022-09-08T05:53:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:28.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:28.289851+0000) 2022-09-08T05:53:28.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:28.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:28.289807+0000) 2022-09-08T05:53:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:29.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:29.290090+0000) 2022-09-08T05:53:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:29.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:29.290057+0000) 2022-09-08T05:53:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:29 smithi055 ceph-mon[97033]: pgmap v16044: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:53:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:29 smithi059 ceph-mon[97450]: pgmap v16044: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:53:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:30.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:30.290335+0000) 2022-09-08T05:53:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:30.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:30.290295+0000) 2022-09-08T05:53:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:31.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:31.290539+0000) 2022-09-08T05:53:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:31.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:31.290528+0000) 2022-09-08T05:53:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:31 smithi055 ceph-mon[97033]: pgmap v16045: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:53:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:31 smithi059 ceph-mon[97450]: pgmap v16045: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:53:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:32.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:32.290768+0000) 2022-09-08T05:53:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:32.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:32.290769+0000) 2022-09-08T05:53:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:33.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:33.291030+0000) 2022-09-08T05:53:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:33.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:33.291021+0000) 2022-09-08T05:53:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:33 smithi055 ceph-mon[97033]: pgmap v16046: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:53:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:33 smithi059 ceph-mon[97450]: pgmap v16046: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T05:53:34.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:53:34.098+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:53:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:34.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:34.291242+0000) 2022-09-08T05:53:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:34.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:34.291251+0000) 2022-09-08T05:53:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:35.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:35.291466+0000) 2022-09-08T05:53:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:35.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:35.291467+0000) 2022-09-08T05:53:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:35 smithi055 ceph-mon[97033]: pgmap v16047: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:53:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:35 smithi059 ceph-mon[97450]: pgmap v16047: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:53:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:36.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:36.291700+0000) 2022-09-08T05:53:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:36.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:36.291697+0000) 2022-09-08T05:53:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:37 smithi055 ceph-mon[97033]: pgmap v16048: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:53:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:37 smithi059 ceph-mon[97450]: pgmap v16048: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:53:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:37.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:37.291910+0000) 2022-09-08T05:53:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:37.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:37.291912+0000) 2022-09-08T05:53:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:38.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:38.292085+0000) 2022-09-08T05:53:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:38.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:38.292080+0000) 2022-09-08T05:53:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:39.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:39.292307+0000) 2022-09-08T05:53:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:39.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:39.292333+0000) 2022-09-08T05:53:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:39 smithi055 ceph-mon[97033]: pgmap v16049: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:53:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:39 smithi059 ceph-mon[97450]: pgmap v16049: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:53:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:40.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:40.292537+0000) 2022-09-08T05:53:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:40.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:40.292548+0000) 2022-09-08T05:53:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:41.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:41.292772+0000) 2022-09-08T05:53:41.666 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:41.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:41.292774+0000) 2022-09-08T05:53:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:41 smithi055 ceph-mon[97033]: pgmap v16050: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:53:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:41 smithi059 ceph-mon[97450]: pgmap v16050: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:53:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:42.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:42.292997+0000) 2022-09-08T05:53:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:42.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:42.292989+0000) 2022-09-08T05:53:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:43.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:43.293253+0000) 2022-09-08T05:53:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:43.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:43.293254+0000) 2022-09-08T05:53:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:43 smithi055 ceph-mon[97033]: pgmap v16051: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:53:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:43 smithi059 ceph-mon[97450]: pgmap v16051: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:53:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:44.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:44.293526+0000) 2022-09-08T05:53:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:44.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:44.293522+0000) 2022-09-08T05:53:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:45.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:45.293795+0000) 2022-09-08T05:53:45.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:45.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:45.293784+0000) 2022-09-08T05:53:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:45 smithi055 ceph-mon[97033]: pgmap v16052: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:53:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:45 smithi059 ceph-mon[97450]: pgmap v16052: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:53:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:46.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:46.294033+0000) 2022-09-08T05:53:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:46.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:46.294034+0000) 2022-09-08T05:53:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:47 smithi055 ceph-mon[97033]: pgmap v16053: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:53:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:47 smithi059 ceph-mon[97450]: pgmap v16053: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:53:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:47.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:47.294284+0000) 2022-09-08T05:53:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:47.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:47.294288+0000) 2022-09-08T05:53:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:48.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:48.294456+0000) 2022-09-08T05:53:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:48.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:48.294468+0000) 2022-09-08T05:53:49.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:53:49.107+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:53:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:49.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:49.294663+0000) 2022-09-08T05:53:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:49.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:49.294667+0000) 2022-09-08T05:53:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:49 smithi055 ceph-mon[97033]: pgmap v16054: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:53:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:49 smithi059 ceph-mon[97450]: pgmap v16054: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:53:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:50.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:50.294865+0000) 2022-09-08T05:53:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:50.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:50.294861+0000) 2022-09-08T05:53:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:51.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:51.295016+0000) 2022-09-08T05:53:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:51.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:51.295021+0000) 2022-09-08T05:53:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:51 smithi055 ceph-mon[97033]: pgmap v16055: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:53:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:51 smithi059 ceph-mon[97450]: pgmap v16055: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:53:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:52.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:52.295165+0000) 2022-09-08T05:53:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:52.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:52.295199+0000) 2022-09-08T05:53:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:53.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:53.295381+0000) 2022-09-08T05:53:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:53.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:53.295412+0000) 2022-09-08T05:53:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:53 smithi055 ceph-mon[97033]: pgmap v16056: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:53:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:53 smithi059 ceph-mon[97450]: pgmap v16056: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:53:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:54.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:54.295624+0000) 2022-09-08T05:53:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:54.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:54.295688+0000) 2022-09-08T05:53:55.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:55.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:55.295901+0000) 2022-09-08T05:53:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:55.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:55.295888+0000) 2022-09-08T05:53:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:55 smithi055 ceph-mon[97033]: pgmap v16057: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:53:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:55 smithi059 ceph-mon[97450]: pgmap v16057: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:53:56.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:56.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:56.296121+0000) 2022-09-08T05:53:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:56.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:56.296096+0000) 2022-09-08T05:53:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:57 smithi055 ceph-mon[97033]: pgmap v16058: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:53:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:57 smithi059 ceph-mon[97450]: pgmap v16058: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:53:57.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:57.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:57.296242+0000) 2022-09-08T05:53:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:57.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:57.296217+0000) 2022-09-08T05:53:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:58.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:58.296436+0000) 2022-09-08T05:53:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:58.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:58.296442+0000) 2022-09-08T05:53:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:53:59.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:59.296656+0000) 2022-09-08T05:53:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:53:59.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:53:59.296654+0000) 2022-09-08T05:53:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:53:59 smithi055 ceph-mon[97033]: pgmap v16059: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:54:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:53:59 smithi059 ceph-mon[97450]: pgmap v16059: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T05:54:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:00.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:00.296808+0000) 2022-09-08T05:54:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:00.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:00.296814+0000) 2022-09-08T05:54:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:01.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:01.297069+0000) 2022-09-08T05:54:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:01.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:01.297061+0000) 2022-09-08T05:54:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:01 smithi055 ceph-mon[97033]: pgmap v16060: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:54:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:01 smithi059 ceph-mon[97450]: pgmap v16060: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:54:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:02.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:02.297257+0000) 2022-09-08T05:54:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:02.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:02.297274+0000) 2022-09-08T05:54:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:03.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:03.297468+0000) 2022-09-08T05:54:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:03.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:03.297471+0000) 2022-09-08T05:54:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:03 smithi055 ceph-mon[97033]: pgmap v16061: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:54:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:54:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:54:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:54:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:03.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:03 smithi059 ceph-mon[97450]: pgmap v16061: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:54:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:54:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:54:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:54:04.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:04.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:04.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:54:04.122+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:54:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:04.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:04.297768+0000) 2022-09-08T05:54:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:04.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:04.297777+0000) 2022-09-08T05:54:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:04.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:04.892 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:04.893 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:05.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:05.297987+0000) 2022-09-08T05:54:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:05.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:05.298057+0000) 2022-09-08T05:54:05.945 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:05 smithi059 ceph-mon[97450]: pgmap v16062: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:54:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:05 smithi055 ceph-mon[97033]: pgmap v16062: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:54:06.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:06.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:06.298269+0000) 2022-09-08T05:54:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:06.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:06.298336+0000) 2022-09-08T05:54:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:54:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:54:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:54:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:54:07.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:07.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:07.298468+0000) 2022-09-08T05:54:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:07.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:07.298530+0000) 2022-09-08T05:54:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:07 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T05:54:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:07 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T05:54:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:07 smithi055 ceph-mon[97033]: pgmap v16063: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:54:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:07 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T05:54:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:07 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T05:54:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:07 smithi059 ceph-mon[97450]: pgmap v16063: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:54:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:08.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:08.298763+0000) 2022-09-08T05:54:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:08.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:08.298696+0000) 2022-09-08T05:54:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:09.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:09.298950+0000) 2022-09-08T05:54:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:09.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:09.298956+0000) 2022-09-08T05:54:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:09 smithi055 ceph-mon[97033]: pgmap v16064: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:54:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:09 smithi059 ceph-mon[97450]: pgmap v16064: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:54:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:10.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:10.299163+0000) 2022-09-08T05:54:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:10.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:10.299163+0000) 2022-09-08T05:54:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:10 smithi055 ceph-mon[97033]: osdmap e1083: 12 total, 11 up, 11 in 2022-09-08T05:54:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:10 smithi059 ceph-mon[97450]: osdmap e1083: 12 total, 11 up, 11 in 2022-09-08T05:54:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:11.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:11.299428+0000) 2022-09-08T05:54:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:11.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:11.299398+0000) 2022-09-08T05:54:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:11 smithi055 ceph-mon[97033]: pgmap v16066: 129 pgs: 2 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:54:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:11 smithi059 ceph-mon[97450]: pgmap v16066: 129 pgs: 2 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:54:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:12.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:12.299652+0000) 2022-09-08T05:54:12.685 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:12.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:12.299689+0000) 2022-09-08T05:54:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:13.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:13.299880+0000) 2022-09-08T05:54:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:13.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:13.299907+0000) 2022-09-08T05:54:13.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:13 smithi055 ceph-mon[97033]: pgmap v16067: 129 pgs: 2 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:54:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:13 smithi059 ceph-mon[97450]: pgmap v16067: 129 pgs: 2 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:54:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:14.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:14.300104+0000) 2022-09-08T05:54:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:14.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:14.300121+0000) 2022-09-08T05:54:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:15.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:15.300293+0000) 2022-09-08T05:54:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:15.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:15.300327+0000) 2022-09-08T05:54:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:15 smithi055 ceph-mon[97033]: pgmap v16068: 129 pgs: 2 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:54:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:15 smithi059 ceph-mon[97450]: pgmap v16068: 129 pgs: 2 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:54:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:16.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:16.300499+0000) 2022-09-08T05:54:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:16.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:16.300575+0000) 2022-09-08T05:54:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:17 smithi055 ceph-mon[97033]: pgmap v16069: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:54:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:17 smithi059 ceph-mon[97450]: pgmap v16069: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:54:17.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:17.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:17.300770+0000) 2022-09-08T05:54:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:17.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:17.300760+0000) 2022-09-08T05:54:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:18.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:18.300974+0000) 2022-09-08T05:54:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:18.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:18.300972+0000) 2022-09-08T05:54:19.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:54:19.136+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:54:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:19.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:19.301212+0000) 2022-09-08T05:54:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:19.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:19.301207+0000) 2022-09-08T05:54:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:19 smithi055 ceph-mon[97033]: pgmap v16070: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:54:20.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:19 smithi059 ceph-mon[97450]: pgmap v16070: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:54:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:20.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:20.301500+0000) 2022-09-08T05:54:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:20.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:20.301482+0000) 2022-09-08T05:54:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:21.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:21.301729+0000) 2022-09-08T05:54:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:21.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:21.301735+0000) 2022-09-08T05:54:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:21 smithi055 ceph-mon[97033]: pgmap v16071: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T05:54:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:21 smithi055 ceph-mon[97033]: osdmap e1084: 12 total, 11 up, 11 in 2022-09-08T05:54:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:21 smithi059 ceph-mon[97450]: pgmap v16071: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T05:54:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:21 smithi059 ceph-mon[97450]: osdmap e1084: 12 total, 11 up, 11 in 2022-09-08T05:54:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:22.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:22.301916+0000) 2022-09-08T05:54:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:22.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:22.301946+0000) 2022-09-08T05:54:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:23.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:23.302173+0000) 2022-09-08T05:54:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:23.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:23.302157+0000) 2022-09-08T05:54:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:23 smithi055 ceph-mon[97033]: pgmap v16073: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:54:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:23 smithi059 ceph-mon[97450]: pgmap v16073: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:54:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:24.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:24.302387+0000) 2022-09-08T05:54:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:24.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:24.302361+0000) 2022-09-08T05:54:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:25.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:25.302765+0000) 2022-09-08T05:54:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:25.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:25.302590+0000) 2022-09-08T05:54:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:25 smithi055 ceph-mon[97033]: pgmap v16074: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:54:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:25 smithi059 ceph-mon[97450]: pgmap v16074: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:54:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:26.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:26.302904+0000) 2022-09-08T05:54:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:26.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:26.302814+0000) 2022-09-08T05:54:27.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:27 smithi055 ceph-mon[97033]: pgmap v16075: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:54:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:27 smithi059 ceph-mon[97450]: pgmap v16075: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:54:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:27.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:27.303006+0000) 2022-09-08T05:54:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:27.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:27.302974+0000) 2022-09-08T05:54:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:28.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:28.303168+0000) 2022-09-08T05:54:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:28.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:28.303174+0000) 2022-09-08T05:54:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:29.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:29.303362+0000) 2022-09-08T05:54:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:29.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:29.303397+0000) 2022-09-08T05:54:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:29 smithi055 ceph-mon[97033]: pgmap v16076: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:54:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:29 smithi059 ceph-mon[97450]: pgmap v16076: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:54:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:30.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:30.303601+0000) 2022-09-08T05:54:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:30.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:30.303652+0000) 2022-09-08T05:54:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:31.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:31.303821+0000) 2022-09-08T05:54:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:31.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:31.303873+0000) 2022-09-08T05:54:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:31 smithi055 ceph-mon[97033]: pgmap v16077: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:54:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:31 smithi059 ceph-mon[97450]: pgmap v16077: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:54:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:32.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:32.304003+0000) 2022-09-08T05:54:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:32.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:32.304096+0000) 2022-09-08T05:54:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:33.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:33.304232+0000) 2022-09-08T05:54:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:33.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:33.304337+0000) 2022-09-08T05:54:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:33 smithi055 ceph-mon[97033]: pgmap v16078: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:54:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:33 smithi055 ceph-mon[97033]: pgmap v16079: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:54:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:33 smithi055 ceph-mon[97033]: pgmap v16080: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:54:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:33 smithi059 ceph-mon[97450]: pgmap v16078: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:54:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:33 smithi059 ceph-mon[97450]: pgmap v16079: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T05:54:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:33 smithi059 ceph-mon[97450]: pgmap v16080: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:54:34.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:54:34.402 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:54:34.139+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:54:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:34.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:34.304471+0000) 2022-09-08T05:54:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:34.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:34.304600+0000) 2022-09-08T05:54:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:35.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:35.304644+0000) 2022-09-08T05:54:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:35.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:35.304787+0000) 2022-09-08T05:54:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:35 smithi055 ceph-mon[97033]: pgmap v16081: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:54:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:35 smithi059 ceph-mon[97450]: pgmap v16081: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:54:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:36.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:36.304800+0000) 2022-09-08T05:54:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:36.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:36.304924+0000) 2022-09-08T05:54:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:37 smithi055 ceph-mon[97033]: pgmap v16082: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:54:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:37 smithi059 ceph-mon[97450]: pgmap v16082: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:54:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:37.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:37.305007+0000) 2022-09-08T05:54:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:37.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:37.305113+0000) 2022-09-08T05:54:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:38.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:38.305363+0000) 2022-09-08T05:54:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:38.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:38.305285+0000) 2022-09-08T05:54:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:39.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:39.305566+0000) 2022-09-08T05:54:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:39.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:39.305623+0000) 2022-09-08T05:54:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:39 smithi055 ceph-mon[97033]: pgmap v16083: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:54:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:39 smithi059 ceph-mon[97450]: pgmap v16083: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T05:54:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:40.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:40.305942+0000) 2022-09-08T05:54:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:40.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:40.305807+0000) 2022-09-08T05:54:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:41.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:41.306025+0000) 2022-09-08T05:54:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:41.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:41.306123+0000) 2022-09-08T05:54:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:41 smithi055 ceph-mon[97033]: pgmap v16084: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:54:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:41 smithi059 ceph-mon[97450]: pgmap v16084: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T05:54:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:42.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:42.306253+0000) 2022-09-08T05:54:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:42.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:42.306349+0000) 2022-09-08T05:54:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:43.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:43.306451+0000) 2022-09-08T05:54:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:43.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:43.306520+0000) 2022-09-08T05:54:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:43 smithi055 ceph-mon[97033]: pgmap v16085: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:54:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:43 smithi059 ceph-mon[97450]: pgmap v16085: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:54:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:44.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:44.306606+0000) 2022-09-08T05:54:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:44.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:44.306715+0000) 2022-09-08T05:54:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:45.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:45.306769+0000) 2022-09-08T05:54:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:45.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:45.306877+0000) 2022-09-08T05:54:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:45 smithi055 ceph-mon[97033]: pgmap v16086: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:54:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:45 smithi059 ceph-mon[97450]: pgmap v16086: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:54:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:46.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:46.306889+0000) 2022-09-08T05:54:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:46.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:46.307017+0000) 2022-09-08T05:54:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:47 smithi055 ceph-mon[97033]: pgmap v16087: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:54:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:47 smithi059 ceph-mon[97450]: pgmap v16087: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:54:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:47.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:47.307042+0000) 2022-09-08T05:54:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:47.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:47.307154+0000) 2022-09-08T05:54:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:48.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:48.307201+0000) 2022-09-08T05:54:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:48.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:48.307327+0000) 2022-09-08T05:54:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:49.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:49.307397+0000) 2022-09-08T05:54:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:49.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:49.307485+0000) 2022-09-08T05:54:49.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:54:49.154+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:54:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:49 smithi055 ceph-mon[97033]: pgmap v16088: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:54:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:49 smithi059 ceph-mon[97450]: pgmap v16088: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:54:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:50.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:50.307516+0000) 2022-09-08T05:54:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:50.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:50.307660+0000) 2022-09-08T05:54:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:51.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:51.307757+0000) 2022-09-08T05:54:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:51.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:51.307893+0000) 2022-09-08T05:54:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:51 smithi055 ceph-mon[97033]: pgmap v16089: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:54:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:51 smithi059 ceph-mon[97450]: pgmap v16089: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:54:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:52.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:52.307998+0000) 2022-09-08T05:54:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:52.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:52.308160+0000) 2022-09-08T05:54:53.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:53.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:53.308357+0000) 2022-09-08T05:54:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:53.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:53.308191+0000) 2022-09-08T05:54:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:53 smithi055 ceph-mon[97033]: pgmap v16090: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:54:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:53 smithi059 ceph-mon[97450]: pgmap v16090: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:54:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:54.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:54.308413+0000) 2022-09-08T05:54:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:54.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:54.308572+0000) 2022-09-08T05:54:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:54 smithi055 ceph-mon[97033]: pgmap v16091: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:54:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:54 smithi059 ceph-mon[97450]: pgmap v16091: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:54:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:55.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:55.308557+0000) 2022-09-08T05:54:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:55.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:55.308719+0000) 2022-09-08T05:54:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:56.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:56.308792+0000) 2022-09-08T05:54:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:56.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:56.308932+0000) 2022-09-08T05:54:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:57 smithi055 ceph-mon[97033]: pgmap v16092: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:54:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:57 smithi059 ceph-mon[97450]: pgmap v16092: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:54:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:57.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:57.308976+0000) 2022-09-08T05:54:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:57.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:57.309188+0000) 2022-09-08T05:54:58.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:58.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:58.309383+0000) 2022-09-08T05:54:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:58.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:58.309200+0000) 2022-09-08T05:54:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:54:59.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:59.309446+0000) 2022-09-08T05:54:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:54:59.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:54:59.309612+0000) 2022-09-08T05:55:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:54:59 smithi055 ceph-mon[97033]: pgmap v16093: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:54:59 smithi059 ceph-mon[97450]: pgmap v16093: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:00.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:00.309716+0000) 2022-09-08T05:55:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:00.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:00.309916+0000) 2022-09-08T05:55:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:01.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:01.309875+0000) 2022-09-08T05:55:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:01.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:01.310051+0000) 2022-09-08T05:55:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:01 smithi055 ceph-mon[97033]: pgmap v16094: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:55:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:01 smithi059 ceph-mon[97450]: pgmap v16094: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:55:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:02.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:02.310104+0000) 2022-09-08T05:55:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:02.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:02.310242+0000) 2022-09-08T05:55:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:02 smithi055 ceph-mon[97033]: pgmap v16095: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:02 smithi059 ceph-mon[97450]: pgmap v16095: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:03.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:03.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:03.310389+0000) 2022-09-08T05:55:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:03.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:03.310310+0000) 2022-09-08T05:55:04.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:55:04.158+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:55:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:04.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:04.310448+0000) 2022-09-08T05:55:04.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:04.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:04.310547+0000) 2022-09-08T05:55:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:05.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:05.310632+0000) 2022-09-08T05:55:05.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:05.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:05.310713+0000) 2022-09-08T05:55:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:05 smithi055 ceph-mon[97033]: pgmap v16096: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:55:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:05 smithi059 ceph-mon[97450]: pgmap v16096: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:55:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:06.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:06.310859+0000) 2022-09-08T05:55:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:06.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:06.310900+0000) 2022-09-08T05:55:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:55:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:07 smithi055 ceph-mon[97033]: pgmap v16097: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:55:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:55:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:07 smithi059 ceph-mon[97450]: pgmap v16097: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:55:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:07.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:07.311112+0000) 2022-09-08T05:55:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:07.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:07.311158+0000) 2022-09-08T05:55:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:08.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:08.311338+0000) 2022-09-08T05:55:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:08.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:08.311405+0000) 2022-09-08T05:55:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:09.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:09.311485+0000) 2022-09-08T05:55:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:09.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:09.311591+0000) 2022-09-08T05:55:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:09 smithi055 ceph-mon[97033]: pgmap v16098: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:55:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:09 smithi055 ceph-mon[97033]: osdmap e1085: 12 total, 11 up, 11 in 2022-09-08T05:55:10.291 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:09 smithi059 ceph-mon[97450]: pgmap v16098: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:55:10.293 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:09 smithi059 ceph-mon[97450]: osdmap e1085: 12 total, 11 up, 11 in 2022-09-08T05:55:10.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:10.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:10.311803+0000) 2022-09-08T05:55:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:10.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:10.311756+0000) 2022-09-08T05:55:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:11.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:11.311940+0000) 2022-09-08T05:55:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:11.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:11.312027+0000) 2022-09-08T05:55:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:11 smithi055 ceph-mon[97033]: pgmap v16100: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:55:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:11 smithi055 ceph-mon[97033]: osdmap e1086: 12 total, 11 up, 11 in 2022-09-08T05:55:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:11 smithi059 ceph-mon[97450]: pgmap v16100: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:55:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:11 smithi059 ceph-mon[97450]: osdmap e1086: 12 total, 11 up, 11 in 2022-09-08T05:55:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:12.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:12.312180+0000) 2022-09-08T05:55:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:12.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:12.312211+0000) 2022-09-08T05:55:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:13.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:13.312330+0000) 2022-09-08T05:55:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:13.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:13.312345+0000) 2022-09-08T05:55:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:13 smithi055 ceph-mon[97033]: pgmap v16102: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:55:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:13 smithi059 ceph-mon[97450]: pgmap v16102: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T05:55:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:14.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:14.312577+0000) 2022-09-08T05:55:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:14.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:14.312597+0000) 2022-09-08T05:55:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:15.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:15.312791+0000) 2022-09-08T05:55:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:15.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:15.312814+0000) 2022-09-08T05:55:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:15 smithi055 ceph-mon[97033]: pgmap v16103: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:55:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:15 smithi059 ceph-mon[97450]: pgmap v16103: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T05:55:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:16.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:16.312986+0000) 2022-09-08T05:55:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:16.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:16.312994+0000) 2022-09-08T05:55:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:17 smithi055 ceph-mon[97033]: pgmap v16104: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:55:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:17 smithi059 ceph-mon[97450]: pgmap v16104: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:55:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:17.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:17.313221+0000) 2022-09-08T05:55:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:17.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:17.313220+0000) 2022-09-08T05:55:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:18.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:18.313383+0000) 2022-09-08T05:55:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:18.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:18.313403+0000) 2022-09-08T05:55:19.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:55:19.164+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:55:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:19.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:19.313589+0000) 2022-09-08T05:55:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:19.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:19.313676+0000) 2022-09-08T05:55:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:19 smithi055 ceph-mon[97033]: pgmap v16105: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T05:55:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:19 smithi059 ceph-mon[97450]: pgmap v16105: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T05:55:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:20.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:20.313786+0000) 2022-09-08T05:55:20.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:20.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:20.313887+0000) 2022-09-08T05:55:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:20 smithi055 ceph-mon[97033]: pgmap v16106: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:55:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:20 smithi059 ceph-mon[97450]: pgmap v16106: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:55:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:21.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:21.313993+0000) 2022-09-08T05:55:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:21.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:21.314050+0000) 2022-09-08T05:55:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:22.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:22.314158+0000) 2022-09-08T05:55:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:22.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:22.314240+0000) 2022-09-08T05:55:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:23.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:23.314336+0000) 2022-09-08T05:55:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:23.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:23.314416+0000) 2022-09-08T05:55:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:23 smithi055 ceph-mon[97033]: pgmap v16107: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 87 B/s wr, 5 op/s 2022-09-08T05:55:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:23 smithi059 ceph-mon[97450]: pgmap v16107: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 87 B/s wr, 5 op/s 2022-09-08T05:55:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:24.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:24.314577+0000) 2022-09-08T05:55:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:24.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:24.314704+0000) 2022-09-08T05:55:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:24 smithi055 ceph-mon[97033]: pgmap v16108: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:55:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:24 smithi059 ceph-mon[97450]: pgmap v16108: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T05:55:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:25.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:25.314758+0000) 2022-09-08T05:55:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:25.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:25.314894+0000) 2022-09-08T05:55:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:26.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:26.314910+0000) 2022-09-08T05:55:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:26.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:26.315042+0000) 2022-09-08T05:55:27.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:27 smithi055 ceph-mon[97033]: pgmap v16109: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:55:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:27 smithi059 ceph-mon[97450]: pgmap v16109: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T05:55:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:27.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:27.315080+0000) 2022-09-08T05:55:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:27.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:27.315243+0000) 2022-09-08T05:55:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:28.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:28.315288+0000) 2022-09-08T05:55:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:28.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:28.315439+0000) 2022-09-08T05:55:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:28 smithi055 ceph-mon[97033]: pgmap v16110: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:28 smithi059 ceph-mon[97450]: pgmap v16110: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:29.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:29.315546+0000) 2022-09-08T05:55:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:29.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:29.315683+0000) 2022-09-08T05:55:30.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:30.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:30.315817+0000) 2022-09-08T05:55:30.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:30.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:30.315871+0000) 2022-09-08T05:55:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:30 smithi055 ceph-mon[97033]: pgmap v16111: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:55:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:30 smithi059 ceph-mon[97450]: pgmap v16111: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:55:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:31.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:31.316048+0000) 2022-09-08T05:55:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:31.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:31.316062+0000) 2022-09-08T05:55:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:32.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:32.316332+0000) 2022-09-08T05:55:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:32.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:32.316291+0000) 2022-09-08T05:55:33.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:32 smithi055 ceph-mon[97033]: pgmap v16112: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:33.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:32 smithi059 ceph-mon[97450]: pgmap v16112: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:33.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:33.316574+0000) 2022-09-08T05:55:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:33.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:33.316559+0000) 2022-09-08T05:55:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:55:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:55:34.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:55:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:55:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:55:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:55:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:34.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:34.316849+0000) 2022-09-08T05:55:34.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:55:34.171+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:55:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:34.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:34.316793+0000) 2022-09-08T05:55:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:34 smithi059 ceph-mon[97450]: pgmap v16113: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T05:55:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:34 smithi055 ceph-mon[97033]: pgmap v16113: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T05:55:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:35.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:35.317053+0000) 2022-09-08T05:55:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:35.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:35.316981+0000) 2022-09-08T05:55:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:36.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:36.317208+0000) 2022-09-08T05:55:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:36.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:36.317163+0000) 2022-09-08T05:55:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:37 smithi055 ceph-mon[97033]: pgmap v16114: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:55:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:37 smithi059 ceph-mon[97450]: pgmap v16114: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:55:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:37.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:37.317368+0000) 2022-09-08T05:55:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:37.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:37.317345+0000) 2022-09-08T05:55:38.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:38.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:38.317505+0000) 2022-09-08T05:55:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:38.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:38.317508+0000) 2022-09-08T05:55:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:38 smithi055 ceph-mon[97033]: pgmap v16115: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:38 smithi059 ceph-mon[97450]: pgmap v16115: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:39.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:39.317725+0000) 2022-09-08T05:55:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:39.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:39.317779+0000) 2022-09-08T05:55:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:40.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:40.317936+0000) 2022-09-08T05:55:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:40.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:40.317988+0000) 2022-09-08T05:55:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:40 smithi055 ceph-mon[97033]: pgmap v16116: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:55:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:40 smithi059 ceph-mon[97450]: pgmap v16116: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:55:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:41.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:41.318159+0000) 2022-09-08T05:55:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:41.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:41.318100+0000) 2022-09-08T05:55:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:42.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:42.318258+0000) 2022-09-08T05:55:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:42.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:42.318289+0000) 2022-09-08T05:55:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:42 smithi055 ceph-mon[97033]: pgmap v16117: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:42 smithi059 ceph-mon[97450]: pgmap v16117: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:43.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:43.318410+0000) 2022-09-08T05:55:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:43.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:43.318420+0000) 2022-09-08T05:55:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:44.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:44.318633+0000) 2022-09-08T05:55:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:44.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:44.318670+0000) 2022-09-08T05:55:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:44 smithi055 ceph-mon[97033]: pgmap v16118: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T05:55:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:44 smithi059 ceph-mon[97450]: pgmap v16118: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T05:55:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:45.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:45.318878+0000) 2022-09-08T05:55:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:45.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:45.318863+0000) 2022-09-08T05:55:46.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:46.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:46.319037+0000) 2022-09-08T05:55:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:46.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:46.319031+0000) 2022-09-08T05:55:47.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:47 smithi055 ceph-mon[97033]: pgmap v16119: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T05:55:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:47 smithi059 ceph-mon[97450]: pgmap v16119: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T05:55:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:47.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:47.319278+0000) 2022-09-08T05:55:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:47.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:47.319272+0000) 2022-09-08T05:55:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:48.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:48.319466+0000) 2022-09-08T05:55:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:48.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:48.319470+0000) 2022-09-08T05:55:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:48 smithi055 ceph-mon[97033]: pgmap v16120: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:48 smithi059 ceph-mon[97450]: pgmap v16120: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:49.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:55:49.188+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:55:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:49.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:49.319724+0000) 2022-09-08T05:55:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:49.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:49.319730+0000) 2022-09-08T05:55:50.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:50.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:50.319953+0000) 2022-09-08T05:55:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:50.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:50.319930+0000) 2022-09-08T05:55:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:50 smithi055 ceph-mon[97033]: pgmap v16121: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:55:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:50 smithi059 ceph-mon[97450]: pgmap v16121: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:55:51.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:51.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:51.320172+0000) 2022-09-08T05:55:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:51.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:51.320169+0000) 2022-09-08T05:55:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:52.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:52.320362+0000) 2022-09-08T05:55:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:52.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:52.320395+0000) 2022-09-08T05:55:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:52 smithi055 ceph-mon[97033]: pgmap v16122: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:53.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:52 smithi059 ceph-mon[97450]: pgmap v16122: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:53.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:53.320527+0000) 2022-09-08T05:55:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:53.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:53.320564+0000) 2022-09-08T05:55:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:54.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:54.320718+0000) 2022-09-08T05:55:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:54.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:54.320743+0000) 2022-09-08T05:55:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:54 smithi055 ceph-mon[97033]: pgmap v16123: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T05:55:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:54 smithi059 ceph-mon[97450]: pgmap v16123: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 5 op/s 2022-09-08T05:55:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:55.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:55.320940+0000) 2022-09-08T05:55:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:55.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:55.320952+0000) 2022-09-08T05:55:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:56.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:56.321116+0000) 2022-09-08T05:55:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:56.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:56.321119+0000) 2022-09-08T05:55:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:57 smithi055 ceph-mon[97033]: pgmap v16124: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T05:55:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:57 smithi059 ceph-mon[97450]: pgmap v16124: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T05:55:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:57.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:57.321343+0000) 2022-09-08T05:55:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:57.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:57.321330+0000) 2022-09-08T05:55:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:58.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:58.321589+0000) 2022-09-08T05:55:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:58.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:58.321605+0000) 2022-09-08T05:55:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:55:58 smithi055 ceph-mon[97033]: pgmap v16125: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:55:58 smithi059 ceph-mon[97450]: pgmap v16125: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:55:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:55:59.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:59.321774+0000) 2022-09-08T05:55:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:55:59.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:55:59.321819+0000) 2022-09-08T05:56:00.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:00.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:00.322038+0000) 2022-09-08T05:56:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:00.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:00.321980+0000) 2022-09-08T05:56:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:00 smithi055 ceph-mon[97033]: pgmap v16126: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:56:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:00 smithi059 ceph-mon[97450]: pgmap v16126: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:56:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:01.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:01.322132+0000) 2022-09-08T05:56:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:01.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:01.322197+0000) 2022-09-08T05:56:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:02.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:02.322334+0000) 2022-09-08T05:56:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:02.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:02.322429+0000) 2022-09-08T05:56:03.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:02 smithi055 ceph-mon[97033]: pgmap v16127: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:03.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:02 smithi059 ceph-mon[97450]: pgmap v16127: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:03.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:03.322557+0000) 2022-09-08T05:56:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:03.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:03.322686+0000) 2022-09-08T05:56:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:04.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:04.322825+0000) 2022-09-08T05:56:04.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:56:04.194+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:56:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:04.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:04.322883+0000) 2022-09-08T05:56:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:04 smithi055 ceph-mon[97033]: pgmap v16128: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:56:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:04 smithi059 ceph-mon[97450]: pgmap v16128: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:56:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:05.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:05.323082+0000) 2022-09-08T05:56:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:05.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:05.323111+0000) 2022-09-08T05:56:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:56:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:56:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:06.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:06.323298+0000) 2022-09-08T05:56:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:06.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:06.323328+0000) 2022-09-08T05:56:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:56:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:07 smithi055 ceph-mon[97033]: pgmap v16129: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T05:56:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:56:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:07 smithi059 ceph-mon[97450]: pgmap v16129: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T05:56:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:07.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:07.323488+0000) 2022-09-08T05:56:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:07.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:07.323591+0000) 2022-09-08T05:56:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:08.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:08.323735+0000) 2022-09-08T05:56:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:08.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:08.323858+0000) 2022-09-08T05:56:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:56:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:56:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:08 smithi059 ceph-mon[97450]: pgmap v16130: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T05:56:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:08 smithi059 ceph-mon[97450]: pgmap v16131: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2022-09-08T05:56:08.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:56:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:56:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:56:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:08 smithi055 ceph-mon[97033]: pgmap v16130: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T05:56:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:08 smithi055 ceph-mon[97033]: pgmap v16131: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2022-09-08T05:56:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:56:09.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:09.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:09.324091+0000) 2022-09-08T05:56:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:09.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:09.324017+0000) 2022-09-08T05:56:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:10.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:10.324329+0000) 2022-09-08T05:56:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:10.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:10.324352+0000) 2022-09-08T05:56:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:10 smithi059 ceph-mon[97450]: pgmap v16133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 18 KiB/s rd, 80 KiB/s wr, 31 op/s 2022-09-08T05:56:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:10 smithi059 ceph-mon[97450]: osdmap e1087: 12 total, 11 up, 11 in 2022-09-08T05:56:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:10 smithi055 ceph-mon[97033]: pgmap v16133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 18 KiB/s rd, 80 KiB/s wr, 31 op/s 2022-09-08T05:56:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:10 smithi055 ceph-mon[97033]: osdmap e1087: 12 total, 11 up, 11 in 2022-09-08T05:56:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:11.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:11.324554+0000) 2022-09-08T05:56:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:11.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:11.324554+0000) 2022-09-08T05:56:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:12 smithi055 ceph-mon[97033]: pgmap v16134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 154 B/s wr, 4 op/s 2022-09-08T05:56:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:12 smithi059 ceph-mon[97450]: pgmap v16134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 154 B/s wr, 4 op/s 2022-09-08T05:56:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:12.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:12.324764+0000) 2022-09-08T05:56:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:12.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:12.324763+0000) 2022-09-08T05:56:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:13.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:13.324904+0000) 2022-09-08T05:56:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:13.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:13.324905+0000) 2022-09-08T05:56:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:14.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:14.325085+0000) 2022-09-08T05:56:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:14.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:14.325083+0000) 2022-09-08T05:56:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:14 smithi055 ceph-mon[97033]: pgmap v16135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 154 B/s wr, 4 op/s 2022-09-08T05:56:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:14 smithi059 ceph-mon[97450]: pgmap v16135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 154 B/s wr, 4 op/s 2022-09-08T05:56:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:15.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:15.325349+0000) 2022-09-08T05:56:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:15.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:15.325348+0000) 2022-09-08T05:56:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:16.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:16.325671+0000) 2022-09-08T05:56:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:16.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:16.325667+0000) 2022-09-08T05:56:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:16 smithi055 ceph-mon[97033]: pgmap v16136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:56:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:16 smithi055 ceph-mon[97033]: osdmap e1088: 12 total, 11 up, 11 in 2022-09-08T05:56:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:16 smithi059 ceph-mon[97450]: pgmap v16136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:56:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:16 smithi059 ceph-mon[97450]: osdmap e1088: 12 total, 11 up, 11 in 2022-09-08T05:56:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:17.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:17.325922+0000) 2022-09-08T05:56:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:17.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:17.325907+0000) 2022-09-08T05:56:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:18.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:18.326211+0000) 2022-09-08T05:56:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:18.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:18.326233+0000) 2022-09-08T05:56:18.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:18 smithi055 ceph-mon[97033]: pgmap v16138: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:56:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:18 smithi059 ceph-mon[97450]: pgmap v16138: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T05:56:19.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:56:19.201+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:56:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:19.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:19.326453+0000) 2022-09-08T05:56:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:19.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:19.326453+0000) 2022-09-08T05:56:20.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:20.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:20.326776+0000) 2022-09-08T05:56:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:20.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:20.326776+0000) 2022-09-08T05:56:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:20 smithi055 ceph-mon[97033]: pgmap v16139: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:56:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:20 smithi059 ceph-mon[97450]: pgmap v16139: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:56:21.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:21.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:21.327079+0000) 2022-09-08T05:56:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:21.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:21.327076+0000) 2022-09-08T05:56:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:22 smithi055 ceph-mon[97033]: pgmap v16140: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:56:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:22 smithi059 ceph-mon[97450]: pgmap v16140: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:56:22.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:22.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:22.327371+0000) 2022-09-08T05:56:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:22.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:22.327371+0000) 2022-09-08T05:56:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:23.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:23.327656+0000) 2022-09-08T05:56:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:23.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:23.327656+0000) 2022-09-08T05:56:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:24.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:24.327831+0000) 2022-09-08T05:56:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:24.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:24.327831+0000) 2022-09-08T05:56:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:24 smithi055 ceph-mon[97033]: pgmap v16141: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:56:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:24 smithi059 ceph-mon[97450]: pgmap v16141: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:56:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:25.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:25.328110+0000) 2022-09-08T05:56:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:25.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:25.328110+0000) 2022-09-08T05:56:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:26.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:26.328271+0000) 2022-09-08T05:56:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:26.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:26.328271+0000) 2022-09-08T05:56:26.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:26 smithi055 ceph-mon[97033]: pgmap v16142: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:56:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:26 smithi059 ceph-mon[97450]: pgmap v16142: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:56:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:27.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:27.328501+0000) 2022-09-08T05:56:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:27.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:27.328498+0000) 2022-09-08T05:56:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:28.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:28.328687+0000) 2022-09-08T05:56:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:28.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:28.328692+0000) 2022-09-08T05:56:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:28 smithi055 ceph-mon[97033]: pgmap v16143: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:56:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:28 smithi059 ceph-mon[97450]: pgmap v16143: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T05:56:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:29.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:29.328841+0000) 2022-09-08T05:56:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:29.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:29.328842+0000) 2022-09-08T05:56:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:30.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:30.328962+0000) 2022-09-08T05:56:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:30.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:30.328997+0000) 2022-09-08T05:56:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:30 smithi055 ceph-mon[97033]: pgmap v16144: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:30 smithi059 ceph-mon[97450]: pgmap v16144: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:31.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:31.329127+0000) 2022-09-08T05:56:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:31.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:31.329142+0000) 2022-09-08T05:56:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:32 smithi055 ceph-mon[97033]: pgmap v16145: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:56:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:32 smithi059 ceph-mon[97450]: pgmap v16145: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:56:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:32.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:32.329288+0000) 2022-09-08T05:56:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:32.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:32.329268+0000) 2022-09-08T05:56:33.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:33.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:33.329423+0000) 2022-09-08T05:56:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:33.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:33.329408+0000) 2022-09-08T05:56:34.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:56:34.218+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:56:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:34.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:34.329576+0000) 2022-09-08T05:56:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:34.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:34.329559+0000) 2022-09-08T05:56:34.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:34 smithi055 ceph-mon[97033]: pgmap v16146: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:34 smithi059 ceph-mon[97450]: pgmap v16146: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:35.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:35.329713+0000) 2022-09-08T05:56:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:35.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:35.329727+0000) 2022-09-08T05:56:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:36.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:36.329844+0000) 2022-09-08T05:56:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:36.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:36.329867+0000) 2022-09-08T05:56:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:36 smithi055 ceph-mon[97033]: pgmap v16147: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:56:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:36 smithi059 ceph-mon[97450]: pgmap v16147: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:56:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:37.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:37.329972+0000) 2022-09-08T05:56:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:37.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:37.330018+0000) 2022-09-08T05:56:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:38.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:38.330148+0000) 2022-09-08T05:56:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:38.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:38.330102+0000) 2022-09-08T05:56:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:38 smithi055 ceph-mon[97033]: pgmap v16148: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:38 smithi059 ceph-mon[97450]: pgmap v16148: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:39.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:39.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:39.330282+0000) 2022-09-08T05:56:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:39.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:39.330247+0000) 2022-09-08T05:56:40.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:40.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:40.330418+0000) 2022-09-08T05:56:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:40.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:40.330403+0000) 2022-09-08T05:56:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:40 smithi055 ceph-mon[97033]: pgmap v16149: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:40 smithi059 ceph-mon[97450]: pgmap v16149: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:41.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:41.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:41.330550+0000) 2022-09-08T05:56:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:41.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:41.330549+0000) 2022-09-08T05:56:42.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:42 smithi055 ceph-mon[97033]: pgmap v16150: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:56:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:42 smithi059 ceph-mon[97450]: pgmap v16150: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:56:42.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:42.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:42.330693+0000) 2022-09-08T05:56:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:42.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:42.330694+0000) 2022-09-08T05:56:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:43.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:43.330822+0000) 2022-09-08T05:56:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:43.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:43.330826+0000) 2022-09-08T05:56:44.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:44.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:44.330976+0000) 2022-09-08T05:56:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:44.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:44.330980+0000) 2022-09-08T05:56:44.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:44 smithi055 ceph-mon[97033]: pgmap v16151: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:44 smithi059 ceph-mon[97450]: pgmap v16151: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:45.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:45.331129+0000) 2022-09-08T05:56:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:45.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:45.331126+0000) 2022-09-08T05:56:46.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:46.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:46.331270+0000) 2022-09-08T05:56:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:46.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:46.331249+0000) 2022-09-08T05:56:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:46 smithi055 ceph-mon[97033]: pgmap v16152: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:56:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:46 smithi059 ceph-mon[97450]: pgmap v16152: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:56:47.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:47.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:47.331418+0000) 2022-09-08T05:56:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:47.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:47.331381+0000) 2022-09-08T05:56:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:48.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:48.331507+0000) 2022-09-08T05:56:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:48.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:48.331561+0000) 2022-09-08T05:56:48.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:48 smithi055 ceph-mon[97033]: pgmap v16153: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:48 smithi059 ceph-mon[97450]: pgmap v16153: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:49.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:49.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:49.331646+0000) 2022-09-08T05:56:49.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:56:49.225+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:56:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:49.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:49.331694+0000) 2022-09-08T05:56:50.596 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:50.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:50.331790+0000) 2022-09-08T05:56:50.597 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:50.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:50.331829+0000) 2022-09-08T05:56:50.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:50 smithi055 ceph-mon[97033]: pgmap v16154: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:50 smithi059 ceph-mon[97450]: pgmap v16154: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:51.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:51.331934+0000) 2022-09-08T05:56:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:51.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:51.331953+0000) 2022-09-08T05:56:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:52 smithi055 ceph-mon[97033]: pgmap v16155: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:56:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:52 smithi059 ceph-mon[97450]: pgmap v16155: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:56:52.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:52.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:52.332081+0000) 2022-09-08T05:56:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:52.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:52.332082+0000) 2022-09-08T05:56:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:53.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:53.332213+0000) 2022-09-08T05:56:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:53.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:53.332213+0000) 2022-09-08T05:56:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:54.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:54.332365+0000) 2022-09-08T05:56:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:54.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:54.332364+0000) 2022-09-08T05:56:54.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:54 smithi055 ceph-mon[97033]: pgmap v16156: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:54 smithi059 ceph-mon[97450]: pgmap v16156: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:55.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:55.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:55.332511+0000) 2022-09-08T05:56:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:55.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:55.332511+0000) 2022-09-08T05:56:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:56.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:56.332689+0000) 2022-09-08T05:56:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:56.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:56.332689+0000) 2022-09-08T05:56:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:56 smithi055 ceph-mon[97033]: pgmap v16157: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:56:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:56 smithi059 ceph-mon[97450]: pgmap v16157: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:56:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:57.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:57.332846+0000) 2022-09-08T05:56:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:57.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:57.332977+0000) 2022-09-08T05:56:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:58.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:58.333082+0000) 2022-09-08T05:56:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:58.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:58.333176+0000) 2022-09-08T05:56:58.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:56:58 smithi055 ceph-mon[97033]: pgmap v16158: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:59.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:56:58 smithi059 ceph-mon[97450]: pgmap v16158: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:56:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:56:59.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:59.333286+0000) 2022-09-08T05:56:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:56:59.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:56:59.333405+0000) 2022-09-08T05:57:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:00.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:00.333446+0000) 2022-09-08T05:57:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:00.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:00.333589+0000) 2022-09-08T05:57:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:00 smithi055 ceph-mon[97033]: pgmap v16159: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:00 smithi059 ceph-mon[97450]: pgmap v16159: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:01.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:01.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:01.333680+0000) 2022-09-08T05:57:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:01.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:01.333866+0000) 2022-09-08T05:57:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:02 smithi055 ceph-mon[97033]: pgmap v16160: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:57:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:02 smithi059 ceph-mon[97450]: pgmap v16160: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:57:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:02.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:02.333895+0000) 2022-09-08T05:57:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:02.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:02.334036+0000) 2022-09-08T05:57:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:03.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:03.334112+0000) 2022-09-08T05:57:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:03.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:03.334191+0000) 2022-09-08T05:57:04.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:04.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:04.334345+0000) 2022-09-08T05:57:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:04.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:04.334406+0000) 2022-09-08T05:57:04.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:57:04.239+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:57:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:04 smithi055 ceph-mon[97033]: pgmap v16161: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:04 smithi059 ceph-mon[97450]: pgmap v16161: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:05.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:05.334673+0000) 2022-09-08T05:57:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:05.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:05.334603+0000) 2022-09-08T05:57:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:06.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:06.334848+0000) 2022-09-08T05:57:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:06.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:06.334831+0000) 2022-09-08T05:57:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:06 smithi055 ceph-mon[97033]: pgmap v16162: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:57:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:57:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:06 smithi059 ceph-mon[97450]: pgmap v16162: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:57:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:57:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:07.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:07.335055+0000) 2022-09-08T05:57:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:07.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:07.335026+0000) 2022-09-08T05:57:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:57:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:57:08.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:08.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:08.335340+0000) 2022-09-08T05:57:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:08.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:08.335312+0000) 2022-09-08T05:57:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:08 smithi055 ceph-mon[97033]: pgmap v16163: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:57:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:57:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:57:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:57:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:57:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:08 smithi059 ceph-mon[97450]: pgmap v16163: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:57:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:57:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:57:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:57:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:57:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:09.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:09.335554+0000) 2022-09-08T05:57:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:09.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:09.335540+0000) 2022-09-08T05:57:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:09 smithi055 ceph-mon[97033]: pgmap v16164: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T05:57:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:09 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T05:57:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:09 smithi059 ceph-mon[97450]: pgmap v16164: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T05:57:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:09 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T05:57:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:10.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:10.335770+0000) 2022-09-08T05:57:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:10.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:10.335781+0000) 2022-09-08T05:57:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:10 smithi055 ceph-mon[97033]: osdmap e1089: 12 total, 11 up, 11 in 2022-09-08T05:57:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:10 smithi059 ceph-mon[97450]: osdmap e1089: 12 total, 11 up, 11 in 2022-09-08T05:57:11.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:11.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:11.335949+0000) 2022-09-08T05:57:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:11.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:11.335962+0000) 2022-09-08T05:57:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:11 smithi055 ceph-mon[97033]: pgmap v16166: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2022-09-08T05:57:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:11 smithi059 ceph-mon[97450]: pgmap v16166: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2022-09-08T05:57:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:12.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:12.336191+0000) 2022-09-08T05:57:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:12.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:12.336174+0000) 2022-09-08T05:57:13.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:13.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:13.336373+0000) 2022-09-08T05:57:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:13.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:13.336367+0000) 2022-09-08T05:57:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:13 smithi055 ceph-mon[97033]: pgmap v16167: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2022-09-08T05:57:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:13 smithi059 ceph-mon[97450]: pgmap v16167: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2022-09-08T05:57:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:14.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:14.336606+0000) 2022-09-08T05:57:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:14.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:14.336613+0000) 2022-09-08T05:57:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:15.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:15.336840+0000) 2022-09-08T05:57:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:15.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:15.336829+0000) 2022-09-08T05:57:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:15 smithi055 ceph-mon[97033]: pgmap v16168: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 118 B/s wr, 3 op/s 2022-09-08T05:57:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:15 smithi059 ceph-mon[97450]: pgmap v16168: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 118 B/s wr, 3 op/s 2022-09-08T05:57:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:16.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:16.337108+0000) 2022-09-08T05:57:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:16.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:16.337110+0000) 2022-09-08T05:57:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:17.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:17.337319+0000) 2022-09-08T05:57:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:17.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:17.337307+0000) 2022-09-08T05:57:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:17 smithi055 ceph-mon[97033]: pgmap v16169: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2022-09-08T05:57:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:17 smithi059 ceph-mon[97450]: pgmap v16169: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2022-09-08T05:57:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:18.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:18.337521+0000) 2022-09-08T05:57:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:18.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:18.337534+0000) 2022-09-08T05:57:19.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:57:19.252+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:57:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:19.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:19.337744+0000) 2022-09-08T05:57:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:19.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:19.337754+0000) 2022-09-08T05:57:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:19 smithi055 ceph-mon[97033]: pgmap v16170: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:57:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:19 smithi059 ceph-mon[97450]: pgmap v16170: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:57:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:20.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:20.337982+0000) 2022-09-08T05:57:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:20.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:20.337980+0000) 2022-09-08T05:57:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:21.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:21.338176+0000) 2022-09-08T05:57:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:21.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:21.338177+0000) 2022-09-08T05:57:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:21 smithi055 ceph-mon[97033]: pgmap v16171: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.5 KiB/s rd, 194 B/s wr, 8 op/s 2022-09-08T05:57:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:21 smithi055 ceph-mon[97033]: osdmap e1090: 12 total, 11 up, 11 in 2022-09-08T05:57:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:21 smithi059 ceph-mon[97450]: pgmap v16171: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.5 KiB/s rd, 194 B/s wr, 8 op/s 2022-09-08T05:57:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:21 smithi059 ceph-mon[97450]: osdmap e1090: 12 total, 11 up, 11 in 2022-09-08T05:57:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:22.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:22.338397+0000) 2022-09-08T05:57:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:22.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:22.338413+0000) 2022-09-08T05:57:23.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:23.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:23.338626+0000) 2022-09-08T05:57:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:23.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:23.338628+0000) 2022-09-08T05:57:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:23 smithi055 ceph-mon[97033]: pgmap v16173: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:57:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:23 smithi059 ceph-mon[97450]: pgmap v16173: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:57:24.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:24.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:24.338896+0000) 2022-09-08T05:57:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:24.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:24.338900+0000) 2022-09-08T05:57:25.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:25.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:25.339188+0000) 2022-09-08T05:57:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:25.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:25.339137+0000) 2022-09-08T05:57:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:25 smithi055 ceph-mon[97033]: pgmap v16174: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:57:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:25 smithi059 ceph-mon[97450]: pgmap v16174: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:57:26.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:26.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:26.339309+0000) 2022-09-08T05:57:26.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:26.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:26.339304+0000) 2022-09-08T05:57:27.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:27.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:27.339488+0000) 2022-09-08T05:57:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:27.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:27.339458+0000) 2022-09-08T05:57:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:27 smithi055 ceph-mon[97033]: pgmap v16175: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:57:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:27 smithi059 ceph-mon[97450]: pgmap v16175: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:57:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:28.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:28.339716+0000) 2022-09-08T05:57:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:28.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:28.339728+0000) 2022-09-08T05:57:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:29.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:29.339915+0000) 2022-09-08T05:57:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:29.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:29.339974+0000) 2022-09-08T05:57:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:29 smithi055 ceph-mon[97033]: pgmap v16176: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:57:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:29 smithi059 ceph-mon[97450]: pgmap v16176: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:57:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:30.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:30.340105+0000) 2022-09-08T05:57:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:30.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:30.340163+0000) 2022-09-08T05:57:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:31.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:31.340306+0000) 2022-09-08T05:57:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:31.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:31.340353+0000) 2022-09-08T05:57:31.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:31 smithi055 ceph-mon[97033]: pgmap v16177: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:57:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:31 smithi059 ceph-mon[97450]: pgmap v16177: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:57:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:32.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:32.340539+0000) 2022-09-08T05:57:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:32.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:32.340552+0000) 2022-09-08T05:57:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:33.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:33.340777+0000) 2022-09-08T05:57:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:33.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:33.340778+0000) 2022-09-08T05:57:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:33 smithi055 ceph-mon[97033]: pgmap v16178: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:57:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:33 smithi059 ceph-mon[97450]: pgmap v16178: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:57:34.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:34.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:34.340963+0000) 2022-09-08T05:57:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:34.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:34.340957+0000) 2022-09-08T05:57:34.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:57:34.255+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:57:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:35.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:35.341165+0000) 2022-09-08T05:57:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:35.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:35.341162+0000) 2022-09-08T05:57:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:35 smithi055 ceph-mon[97033]: pgmap v16179: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:35 smithi059 ceph-mon[97450]: pgmap v16179: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:36.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:36.341442+0000) 2022-09-08T05:57:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:36.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:36.341442+0000) 2022-09-08T05:57:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:37.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:37.341703+0000) 2022-09-08T05:57:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:37.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:37.341703+0000) 2022-09-08T05:57:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:37 smithi055 ceph-mon[97033]: pgmap v16180: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:57:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:37 smithi059 ceph-mon[97450]: pgmap v16180: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:57:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:38.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:38.341952+0000) 2022-09-08T05:57:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:38.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:38.341952+0000) 2022-09-08T05:57:39.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:39.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:39.342215+0000) 2022-09-08T05:57:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:39.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:39.342215+0000) 2022-09-08T05:57:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:39 smithi055 ceph-mon[97033]: pgmap v16181: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:39 smithi059 ceph-mon[97450]: pgmap v16181: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:40.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:40.342388+0000) 2022-09-08T05:57:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:40.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:40.342414+0000) 2022-09-08T05:57:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:41.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:41.342585+0000) 2022-09-08T05:57:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:41.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:41.342627+0000) 2022-09-08T05:57:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:41 smithi055 ceph-mon[97033]: pgmap v16182: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:57:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:41 smithi059 ceph-mon[97450]: pgmap v16182: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:57:42.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:42.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:42.342786+0000) 2022-09-08T05:57:42.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:42.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:42.342863+0000) 2022-09-08T05:57:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:43.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:43.342934+0000) 2022-09-08T05:57:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:43.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:43.343039+0000) 2022-09-08T05:57:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:43 smithi055 ceph-mon[97033]: pgmap v16183: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:43 smithi059 ceph-mon[97450]: pgmap v16183: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:44.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:44.343172+0000) 2022-09-08T05:57:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:44.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:44.343289+0000) 2022-09-08T05:57:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:45.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:45.343338+0000) 2022-09-08T05:57:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:45.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:45.343467+0000) 2022-09-08T05:57:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:45 smithi055 ceph-mon[97033]: pgmap v16184: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:45 smithi059 ceph-mon[97450]: pgmap v16184: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:46.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:46.343553+0000) 2022-09-08T05:57:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:46.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:46.343715+0000) 2022-09-08T05:57:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:47.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:47.343747+0000) 2022-09-08T05:57:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:47.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:47.343879+0000) 2022-09-08T05:57:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:47 smithi055 ceph-mon[97033]: pgmap v16185: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:57:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:47 smithi059 ceph-mon[97450]: pgmap v16185: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:57:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:48.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:48.343912+0000) 2022-09-08T05:57:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:48.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:48.344090+0000) 2022-09-08T05:57:49.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:57:49.258+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:57:49.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:49.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:49.344075+0000) 2022-09-08T05:57:49.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:49.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:49.344251+0000) 2022-09-08T05:57:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:49 smithi055 ceph-mon[97033]: pgmap v16186: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:49 smithi059 ceph-mon[97450]: pgmap v16186: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:50.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:50.344196+0000) 2022-09-08T05:57:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:50.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:50.344415+0000) 2022-09-08T05:57:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:51.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:51.344312+0000) 2022-09-08T05:57:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:51.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:51.344563+0000) 2022-09-08T05:57:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:51 smithi055 ceph-mon[97033]: pgmap v16187: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:57:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:51 smithi059 ceph-mon[97450]: pgmap v16187: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:57:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:52.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:52.344512+0000) 2022-09-08T05:57:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:52.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:52.344713+0000) 2022-09-08T05:57:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:53.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:53.344911+0000) 2022-09-08T05:57:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:53.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:53.344729+0000) 2022-09-08T05:57:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:53 smithi055 ceph-mon[97033]: pgmap v16188: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:53 smithi059 ceph-mon[97450]: pgmap v16188: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:54.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:54.344983+0000) 2022-09-08T05:57:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:54.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:54.345161+0000) 2022-09-08T05:57:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:55.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:55.345168+0000) 2022-09-08T05:57:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:55.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:55.345348+0000) 2022-09-08T05:57:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:55 smithi055 ceph-mon[97033]: pgmap v16189: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:55 smithi059 ceph-mon[97450]: pgmap v16189: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:57:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:56.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:56.345398+0000) 2022-09-08T05:57:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:56.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:56.345586+0000) 2022-09-08T05:57:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:57.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:57.345618+0000) 2022-09-08T05:57:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:57.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:57.345728+0000) 2022-09-08T05:57:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:57 smithi055 ceph-mon[97033]: pgmap v16190: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:57:58.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:57 smithi059 ceph-mon[97450]: pgmap v16190: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:57:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:58.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:58.345850+0000) 2022-09-08T05:57:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:58.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:58.345932+0000) 2022-09-08T05:57:59.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:57:59.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:59.346083+0000) 2022-09-08T05:57:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:57:59.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:57:59.346019+0000) 2022-09-08T05:57:59.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:57:59 smithi055 ceph-mon[97033]: pgmap v16191: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:00.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:57:59 smithi059 ceph-mon[97450]: pgmap v16191: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:00.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:00.346181+0000) 2022-09-08T05:58:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:00.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:00.346243+0000) 2022-09-08T05:58:01.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:01.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:01.346383+0000) 2022-09-08T05:58:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:01.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:01.346338+0000) 2022-09-08T05:58:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:01 smithi055 ceph-mon[97033]: pgmap v16192: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:58:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:01 smithi059 ceph-mon[97450]: pgmap v16192: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:58:02.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:02.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:02.346578+0000) 2022-09-08T05:58:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:02.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:02.346683+0000) 2022-09-08T05:58:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:03.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:03.346726+0000) 2022-09-08T05:58:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:03.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:03.346858+0000) 2022-09-08T05:58:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:03 smithi055 ceph-mon[97033]: pgmap v16193: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:03 smithi059 ceph-mon[97450]: pgmap v16193: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:04.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:58:04.262+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:58:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:04.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:04.346875+0000) 2022-09-08T05:58:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:04.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:04.347022+0000) 2022-09-08T05:58:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:05.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:05.347030+0000) 2022-09-08T05:58:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:05.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:05.347163+0000) 2022-09-08T05:58:05.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:05 smithi055 ceph-mon[97033]: pgmap v16194: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:05 smithi059 ceph-mon[97450]: pgmap v16194: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:06.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:06.347310+0000) 2022-09-08T05:58:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:06.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:06.347166+0000) 2022-09-08T05:58:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:58:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:58:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:58:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:58:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:07.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:07.347439+0000) 2022-09-08T05:58:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:07.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:07.347300+0000) 2022-09-08T05:58:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:07 smithi055 ceph-mon[97033]: pgmap v16195: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:58:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:07 smithi059 ceph-mon[97450]: pgmap v16195: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:58:08.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:08.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:08.347588+0000) 2022-09-08T05:58:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:08.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:08.347471+0000) 2022-09-08T05:58:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:58:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:58:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:58:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:58:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:58:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:58:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:09.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:09.347618+0000) 2022-09-08T05:58:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:09.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:09.347734+0000) 2022-09-08T05:58:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:09 smithi059 ceph-mon[97450]: pgmap v16196: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:58:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:58:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:09 smithi055 ceph-mon[97033]: pgmap v16196: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:58:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:58:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:10.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:10.347849+0000) 2022-09-08T05:58:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:10.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:10.348003+0000) 2022-09-08T05:58:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:10 smithi055 ceph-mon[97033]: osdmap e1091: 12 total, 11 up, 11 in 2022-09-08T05:58:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:10 smithi059 ceph-mon[97450]: osdmap e1091: 12 total, 11 up, 11 in 2022-09-08T05:58:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:11.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:11.347996+0000) 2022-09-08T05:58:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:11.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:11.348184+0000) 2022-09-08T05:58:12.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:11 smithi055 ceph-mon[97033]: pgmap v16198: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:58:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:11 smithi059 ceph-mon[97450]: pgmap v16198: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:58:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:12.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:12.348201+0000) 2022-09-08T05:58:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:12.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:12.348392+0000) 2022-09-08T05:58:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:13.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:13.348395+0000) 2022-09-08T05:58:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:13.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:13.348542+0000) 2022-09-08T05:58:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:13 smithi055 ceph-mon[97033]: pgmap v16199: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:58:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:13 smithi059 ceph-mon[97450]: pgmap v16199: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:58:14.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:14.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:14.348567+0000) 2022-09-08T05:58:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:14.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:14.348717+0000) 2022-09-08T05:58:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:15.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:15.348799+0000) 2022-09-08T05:58:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:15.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:15.348965+0000) 2022-09-08T05:58:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:15 smithi055 ceph-mon[97033]: pgmap v16200: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:58:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:15 smithi059 ceph-mon[97450]: pgmap v16200: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:58:16.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:16.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:16.349187+0000) 2022-09-08T05:58:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:16.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:16.349068+0000) 2022-09-08T05:58:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:17.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:17.349343+0000) 2022-09-08T05:58:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:17.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:17.349419+0000) 2022-09-08T05:58:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:17 smithi055 ceph-mon[97033]: pgmap v16201: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:58:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:17 smithi059 ceph-mon[97450]: pgmap v16201: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:58:18.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:18.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:18.349552+0000) 2022-09-08T05:58:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:18.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:18.349591+0000) 2022-09-08T05:58:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:19.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:19.349812+0000) 2022-09-08T05:58:19.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:58:19.272+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:58:19.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:19.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:19.349816+0000) 2022-09-08T05:58:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:19 smithi055 ceph-mon[97033]: pgmap v16202: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:58:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:19 smithi059 ceph-mon[97450]: pgmap v16202: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:58:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:20.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:20.350074+0000) 2022-09-08T05:58:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:20.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:20.350092+0000) 2022-09-08T05:58:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:21.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:21.350300+0000) 2022-09-08T05:58:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:21.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:21.350294+0000) 2022-09-08T05:58:22.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:21 smithi055 ceph-mon[97033]: pgmap v16203: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 98 B/s wr, 6 op/s 2022-09-08T05:58:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:21 smithi055 ceph-mon[97033]: osdmap e1092: 12 total, 11 up, 11 in 2022-09-08T05:58:22.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:21 smithi059 ceph-mon[97450]: pgmap v16203: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 98 B/s wr, 6 op/s 2022-09-08T05:58:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:21 smithi059 ceph-mon[97450]: osdmap e1092: 12 total, 11 up, 11 in 2022-09-08T05:58:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:22.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:22.350518+0000) 2022-09-08T05:58:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:22.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:22.350502+0000) 2022-09-08T05:58:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:23.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:23.350671+0000) 2022-09-08T05:58:23.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:23.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:23.350651+0000) 2022-09-08T05:58:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:23 smithi055 ceph-mon[97033]: pgmap v16205: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:58:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:23 smithi059 ceph-mon[97450]: pgmap v16205: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:58:24.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:24.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:24.350860+0000) 2022-09-08T05:58:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:24.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:24.350842+0000) 2022-09-08T05:58:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:25.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:25.351041+0000) 2022-09-08T05:58:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:25.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:25.351039+0000) 2022-09-08T05:58:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:25 smithi055 ceph-mon[97033]: pgmap v16206: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:58:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:25 smithi059 ceph-mon[97450]: pgmap v16206: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:58:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:26.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:26.351285+0000) 2022-09-08T05:58:26.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:26.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:26.351294+0000) 2022-09-08T05:58:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:27.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:27.351532+0000) 2022-09-08T05:58:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:27.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:27.351536+0000) 2022-09-08T05:58:28.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:27 smithi055 ceph-mon[97033]: pgmap v16207: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:58:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:27 smithi059 ceph-mon[97450]: pgmap v16207: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:58:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:28.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:28.351732+0000) 2022-09-08T05:58:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:28.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:28.351739+0000) 2022-09-08T05:58:29.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:29.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:29.351906+0000) 2022-09-08T05:58:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:29.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:29.351980+0000) 2022-09-08T05:58:30.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:29 smithi055 ceph-mon[97033]: pgmap v16208: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:58:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:29 smithi059 ceph-mon[97450]: pgmap v16208: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:58:30.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:30.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:30.352129+0000) 2022-09-08T05:58:30.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:30.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:30.352228+0000) 2022-09-08T05:58:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:31.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:31.352363+0000) 2022-09-08T05:58:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:31.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:31.352387+0000) 2022-09-08T05:58:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:31 smithi055 ceph-mon[97033]: pgmap v16209: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:58:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:31 smithi059 ceph-mon[97450]: pgmap v16209: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:58:32.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:32.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:32.352592+0000) 2022-09-08T05:58:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:32.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:32.352608+0000) 2022-09-08T05:58:33.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:33.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:33.352807+0000) 2022-09-08T05:58:33.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:33.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:33.352848+0000) 2022-09-08T05:58:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:33 smithi055 ceph-mon[97033]: pgmap v16210: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:58:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:33 smithi059 ceph-mon[97450]: pgmap v16210: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:58:34.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:58:34.274+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:58:34.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:34.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:34.353099+0000) 2022-09-08T05:58:34.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:34.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:34.353112+0000) 2022-09-08T05:58:35.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:35.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:35.353336+0000) 2022-09-08T05:58:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:35.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:35.353329+0000) 2022-09-08T05:58:36.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:35 smithi055 ceph-mon[97033]: pgmap v16211: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:35 smithi059 ceph-mon[97450]: pgmap v16211: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:36.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:36.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:36.353593+0000) 2022-09-08T05:58:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:36.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:36.353588+0000) 2022-09-08T05:58:37.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:37.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:37.353835+0000) 2022-09-08T05:58:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:37.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:37.353875+0000) 2022-09-08T05:58:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:37 smithi055 ceph-mon[97033]: pgmap v16212: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:58:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:37 smithi059 ceph-mon[97450]: pgmap v16212: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:58:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:38.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:38.353999+0000) 2022-09-08T05:58:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:38.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:38.354085+0000) 2022-09-08T05:58:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:39.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:39.354163+0000) 2022-09-08T05:58:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:39.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:39.354309+0000) 2022-09-08T05:58:40.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:39 smithi055 ceph-mon[97033]: pgmap v16213: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:39 smithi059 ceph-mon[97450]: pgmap v16213: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:40.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:40.354353+0000) 2022-09-08T05:58:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:40.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:40.354521+0000) 2022-09-08T05:58:41.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:41.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:41.354561+0000) 2022-09-08T05:58:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:41.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:41.354762+0000) 2022-09-08T05:58:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:41 smithi055 ceph-mon[97033]: pgmap v16214: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:58:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:41 smithi059 ceph-mon[97450]: pgmap v16214: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:58:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:42.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:42.354785+0000) 2022-09-08T05:58:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:42.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:42.355016+0000) 2022-09-08T05:58:43.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:43.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:43.355037+0000) 2022-09-08T05:58:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:43.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:43.355245+0000) 2022-09-08T05:58:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:43 smithi055 ceph-mon[97033]: pgmap v16215: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:43 smithi059 ceph-mon[97450]: pgmap v16215: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:44.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:44.355275+0000) 2022-09-08T05:58:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:44.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:44.355415+0000) 2022-09-08T05:58:45.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:45.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:45.355543+0000) 2022-09-08T05:58:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:45.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:45.355632+0000) 2022-09-08T05:58:46.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:45 smithi055 ceph-mon[97033]: pgmap v16216: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:45 smithi059 ceph-mon[97450]: pgmap v16216: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:46.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:46.355750+0000) 2022-09-08T05:58:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:46.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:46.355854+0000) 2022-09-08T05:58:47.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:47.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:47.355890+0000) 2022-09-08T05:58:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:47.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:47.356018+0000) 2022-09-08T05:58:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:47 smithi055 ceph-mon[97033]: pgmap v16217: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:58:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:47 smithi059 ceph-mon[97450]: pgmap v16217: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:58:48.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:48.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:48.356141+0000) 2022-09-08T05:58:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:48.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:48.356250+0000) 2022-09-08T05:58:49.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:58:49.281+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:58:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:49.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:49.356320+0000) 2022-09-08T05:58:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:49.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:49.356440+0000) 2022-09-08T05:58:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:49 smithi055 ceph-mon[97033]: pgmap v16218: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:49 smithi059 ceph-mon[97450]: pgmap v16218: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:50.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:50.356529+0000) 2022-09-08T05:58:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:50.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:50.356670+0000) 2022-09-08T05:58:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:51.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:51.356756+0000) 2022-09-08T05:58:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:51.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:51.356911+0000) 2022-09-08T05:58:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:51 smithi055 ceph-mon[97033]: pgmap v16219: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:58:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:51 smithi059 ceph-mon[97450]: pgmap v16219: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:58:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:52.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:52.356992+0000) 2022-09-08T05:58:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:52.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:52.357065+0000) 2022-09-08T05:58:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:53.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:53.357175+0000) 2022-09-08T05:58:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:53.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:53.357219+0000) 2022-09-08T05:58:54.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:53 smithi055 ceph-mon[97033]: pgmap v16220: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:53 smithi059 ceph-mon[97450]: pgmap v16220: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:54.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:54.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:54.357379+0000) 2022-09-08T05:58:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:54.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:54.357376+0000) 2022-09-08T05:58:55.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:55.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:55.357616+0000) 2022-09-08T05:58:55.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:55.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:55.357588+0000) 2022-09-08T05:58:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:55 smithi055 ceph-mon[97033]: pgmap v16221: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:55 smithi059 ceph-mon[97450]: pgmap v16221: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:58:56.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:56.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:56.357783+0000) 2022-09-08T05:58:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:56.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:56.357699+0000) 2022-09-08T05:58:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:57.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:57.358001+0000) 2022-09-08T05:58:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:57.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:57.357922+0000) 2022-09-08T05:58:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:57 smithi055 ceph-mon[97033]: pgmap v16222: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:58:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:57 smithi059 ceph-mon[97450]: pgmap v16222: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:58:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:58.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:58.358235+0000) 2022-09-08T05:58:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:58.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:58.358210+0000) 2022-09-08T05:58:59.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:58:59.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:59.358447+0000) 2022-09-08T05:58:59.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:58:59.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:58:59.358386+0000) 2022-09-08T05:59:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:58:59 smithi055 ceph-mon[97033]: pgmap v16223: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:58:59 smithi059 ceph-mon[97450]: pgmap v16223: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:00.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:00.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:00.358661+0000) 2022-09-08T05:59:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:00.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:00.358593+0000) 2022-09-08T05:59:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:01.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:01.358877+0000) 2022-09-08T05:59:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:01.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:01.358921+0000) 2022-09-08T05:59:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:01 smithi055 ceph-mon[97033]: pgmap v16224: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:59:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:01 smithi059 ceph-mon[97450]: pgmap v16224: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:59:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:02.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:02.359150+0000) 2022-09-08T05:59:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:02.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:02.359154+0000) 2022-09-08T05:59:03.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:03.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:03.359401+0000) 2022-09-08T05:59:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:03.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:03.359402+0000) 2022-09-08T05:59:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:03 smithi055 ceph-mon[97033]: pgmap v16225: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:03 smithi059 ceph-mon[97450]: pgmap v16225: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:04.652 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:59:04.296+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:59:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:04.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:04.359579+0000) 2022-09-08T05:59:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:04.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:04.359605+0000) 2022-09-08T05:59:05.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:05.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:05.359786+0000) 2022-09-08T05:59:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:05.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:05.359863+0000) 2022-09-08T05:59:06.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:05 smithi055 ceph-mon[97033]: pgmap v16226: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:06.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:59:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:05 smithi059 ceph-mon[97450]: pgmap v16226: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T05:59:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:06.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:06.359917+0000) 2022-09-08T05:59:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:06.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:06.360035+0000) 2022-09-08T05:59:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:07 smithi055 ceph-mon[97033]: pgmap v16227: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:59:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:59:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:07 smithi059 ceph-mon[97450]: pgmap v16227: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T05:59:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T05:59:07.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:07.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:07.360102+0000) 2022-09-08T05:59:07.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:07.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:07.360185+0000) 2022-09-08T05:59:08.662 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:08.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:08.360322+0000) 2022-09-08T05:59:08.663 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:08.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:08.360432+0000) 2022-09-08T05:59:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:09 smithi059 ceph-mon[97450]: pgmap v16228: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:59:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:59:09.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:59:09.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:59:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:09.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:09.360563+0000) 2022-09-08T05:59:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:09.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:09.360725+0000) 2022-09-08T05:59:09.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:09 smithi055 ceph-mon[97033]: pgmap v16228: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T05:59:09.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T05:59:09.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T05:59:09.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T05:59:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:10 smithi059 ceph-mon[97450]: osdmap e1093: 12 total, 11 up, 11 in 2022-09-08T05:59:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:59:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:10.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:10.360821+0000) 2022-09-08T05:59:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:10.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:10.360988+0000) 2022-09-08T05:59:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:10 smithi055 ceph-mon[97033]: osdmap e1093: 12 total, 11 up, 11 in 2022-09-08T05:59:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T05:59:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:11.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:11.361034+0000) 2022-09-08T05:59:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:11.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:11.361149+0000) 2022-09-08T05:59:11.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:11 smithi055 ceph-mon[97033]: pgmap v16230: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:59:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:11 smithi059 ceph-mon[97450]: pgmap v16230: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:59:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:12.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:12.361185+0000) 2022-09-08T05:59:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:12.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:12.361307+0000) 2022-09-08T05:59:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:13.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:13.361373+0000) 2022-09-08T05:59:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:13.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:13.361473+0000) 2022-09-08T05:59:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:13 smithi055 ceph-mon[97033]: pgmap v16231: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:59:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:13 smithi059 ceph-mon[97450]: pgmap v16231: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:59:14.667 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:14.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:14.361592+0000) 2022-09-08T05:59:14.667 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:14.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:14.361708+0000) 2022-09-08T05:59:15.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:15.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:15.361821+0000) 2022-09-08T05:59:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:15.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:15.361876+0000) 2022-09-08T05:59:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:15 smithi055 ceph-mon[97033]: pgmap v16232: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:59:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:15 smithi059 ceph-mon[97450]: pgmap v16232: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T05:59:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:16.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:16.361993+0000) 2022-09-08T05:59:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:16.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:16.362077+0000) 2022-09-08T05:59:17.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:17.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:17.362168+0000) 2022-09-08T05:59:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:17.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:17.362228+0000) 2022-09-08T05:59:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:17 smithi055 ceph-mon[97033]: pgmap v16233: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:59:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:17 smithi059 ceph-mon[97450]: pgmap v16233: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:59:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:18.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:18.362388+0000) 2022-09-08T05:59:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:18.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:18.362435+0000) 2022-09-08T05:59:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:19.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:19.362620+0000) 2022-09-08T05:59:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:19.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:19.362738+0000) 2022-09-08T05:59:19.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:59:19.300+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:59:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:19 smithi055 ceph-mon[97033]: pgmap v16234: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:59:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:19 smithi059 ceph-mon[97450]: pgmap v16234: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T05:59:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:20.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:20.362890+0000) 2022-09-08T05:59:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:20.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:20.362945+0000) 2022-09-08T05:59:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:21.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:21.363131+0000) 2022-09-08T05:59:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:21.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:21.363186+0000) 2022-09-08T05:59:21.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:21 smithi055 ceph-mon[97033]: pgmap v16235: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 186 B/s wr, 8 op/s 2022-09-08T05:59:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:21 smithi055 ceph-mon[97033]: osdmap e1094: 12 total, 11 up, 11 in 2022-09-08T05:59:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:21 smithi059 ceph-mon[97450]: pgmap v16235: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 186 B/s wr, 8 op/s 2022-09-08T05:59:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:21 smithi059 ceph-mon[97450]: osdmap e1094: 12 total, 11 up, 11 in 2022-09-08T05:59:22.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:22.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:22.363370+0000) 2022-09-08T05:59:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:22.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:22.363391+0000) 2022-09-08T05:59:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:23.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:23.363685+0000) 2022-09-08T05:59:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:23.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:23.363620+0000) 2022-09-08T05:59:23.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:23 smithi055 ceph-mon[97033]: pgmap v16237: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:59:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:23 smithi059 ceph-mon[97450]: pgmap v16237: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:59:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:24.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:24.363893+0000) 2022-09-08T05:59:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:24.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:24.363866+0000) 2022-09-08T05:59:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:25.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:25.364122+0000) 2022-09-08T05:59:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:25.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:25.364114+0000) 2022-09-08T05:59:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:25 smithi055 ceph-mon[97033]: pgmap v16238: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:59:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:25 smithi059 ceph-mon[97450]: pgmap v16238: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T05:59:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:26.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:26.364371+0000) 2022-09-08T05:59:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:26.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:26.364369+0000) 2022-09-08T05:59:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:27.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:27.364651+0000) 2022-09-08T05:59:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:27.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:27.364664+0000) 2022-09-08T05:59:27.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:27 smithi055 ceph-mon[97033]: pgmap v16239: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:59:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:27 smithi059 ceph-mon[97450]: pgmap v16239: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:59:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:28.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:28.364865+0000) 2022-09-08T05:59:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:28.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:28.364874+0000) 2022-09-08T05:59:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:29.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:29.365089+0000) 2022-09-08T05:59:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:29.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:29.365020+0000) 2022-09-08T05:59:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:29 smithi055 ceph-mon[97033]: pgmap v16240: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:59:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:29 smithi059 ceph-mon[97450]: pgmap v16240: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:59:30.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:30.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:30.365247+0000) 2022-09-08T05:59:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:30.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:30.365169+0000) 2022-09-08T05:59:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:31.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:31.365478+0000) 2022-09-08T05:59:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:31.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:31.365401+0000) 2022-09-08T05:59:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:31 smithi055 ceph-mon[97033]: pgmap v16241: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:59:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:31 smithi059 ceph-mon[97450]: pgmap v16241: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T05:59:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:32.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:32.365695+0000) 2022-09-08T05:59:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:32.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:32.365738+0000) 2022-09-08T05:59:33.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:33.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:33.365984+0000) 2022-09-08T05:59:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:33.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:33.365905+0000) 2022-09-08T05:59:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:33 smithi055 ceph-mon[97033]: pgmap v16242: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:59:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:33 smithi059 ceph-mon[97450]: pgmap v16242: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T05:59:34.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:59:34.302+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:59:34.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:34.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:34.366259+0000) 2022-09-08T05:59:34.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:34.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:34.366144+0000) 2022-09-08T05:59:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:35.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:35.366380+0000) 2022-09-08T05:59:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:35.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:35.366453+0000) 2022-09-08T05:59:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:35 smithi055 ceph-mon[97033]: pgmap v16243: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:35 smithi059 ceph-mon[97450]: pgmap v16243: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:36.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:36.366627+0000) 2022-09-08T05:59:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:36.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:36.366531+0000) 2022-09-08T05:59:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:37.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:37.366818+0000) 2022-09-08T05:59:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:37.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:37.366752+0000) 2022-09-08T05:59:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:37 smithi055 ceph-mon[97033]: pgmap v16244: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:59:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:37 smithi059 ceph-mon[97450]: pgmap v16244: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:59:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:38.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:38.366984+0000) 2022-09-08T05:59:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:38.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:38.366885+0000) 2022-09-08T05:59:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:39.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:39.367132+0000) 2022-09-08T05:59:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:39.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:39.367025+0000) 2022-09-08T05:59:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:39 smithi055 ceph-mon[97033]: pgmap v16245: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:39 smithi059 ceph-mon[97450]: pgmap v16245: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:40.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:40.367226+0000) 2022-09-08T05:59:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:40.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:40.367307+0000) 2022-09-08T05:59:41.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:41.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:41.367552+0000) 2022-09-08T05:59:41.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:41.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:41.367468+0000) 2022-09-08T05:59:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:41 smithi055 ceph-mon[97033]: pgmap v16246: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:59:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:41 smithi059 ceph-mon[97450]: pgmap v16246: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:59:42.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:42.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:42.367743+0000) 2022-09-08T05:59:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:42.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:42.367654+0000) 2022-09-08T05:59:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:43.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:43.367856+0000) 2022-09-08T05:59:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:43.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:43.367913+0000) 2022-09-08T05:59:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:43 smithi055 ceph-mon[97033]: pgmap v16247: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:43.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:43 smithi059 ceph-mon[97450]: pgmap v16247: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:44.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:44.368151+0000) 2022-09-08T05:59:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:44.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:44.368043+0000) 2022-09-08T05:59:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:45.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:45.368345+0000) 2022-09-08T05:59:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:45.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:45.368243+0000) 2022-09-08T05:59:45.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:45 smithi055 ceph-mon[97033]: pgmap v16248: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:45 smithi059 ceph-mon[97450]: pgmap v16248: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:46.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:46.368477+0000) 2022-09-08T05:59:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:46.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:46.368602+0000) 2022-09-08T05:59:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:47.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:47.368823+0000) 2022-09-08T05:59:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:47.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:47.368740+0000) 2022-09-08T05:59:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:47 smithi055 ceph-mon[97033]: pgmap v16249: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:59:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:47 smithi059 ceph-mon[97450]: pgmap v16249: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:59:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:48.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:48.369037+0000) 2022-09-08T05:59:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:48.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:48.368912+0000) 2022-09-08T05:59:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:49.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:49.369278+0000) 2022-09-08T05:59:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:49.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:49.369129+0000) 2022-09-08T05:59:49.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 05:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T05:59:49.318+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T05:59:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:49 smithi055 ceph-mon[97033]: pgmap v16250: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:49.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:49 smithi059 ceph-mon[97450]: pgmap v16250: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:50.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:50.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:50.369529+0000) 2022-09-08T05:59:50.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:50.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:50.369376+0000) 2022-09-08T05:59:51.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:51.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:51.369577+0000) 2022-09-08T05:59:51.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:51.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:51.369729+0000) 2022-09-08T05:59:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:51 smithi055 ceph-mon[97033]: pgmap v16251: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:59:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:51 smithi059 ceph-mon[97450]: pgmap v16251: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:59:52.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:52.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:52.369751+0000) 2022-09-08T05:59:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:52.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:52.369890+0000) 2022-09-08T05:59:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:53.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:53.370053+0000) 2022-09-08T05:59:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:53.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:53.369969+0000) 2022-09-08T05:59:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:53 smithi055 ceph-mon[97033]: pgmap v16252: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:53 smithi059 ceph-mon[97450]: pgmap v16252: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:54.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:54.370215+0000) 2022-09-08T05:59:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:54.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:54.370130+0000) 2022-09-08T05:59:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:55.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:55.370421+0000) 2022-09-08T05:59:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:55.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:55.370374+0000) 2022-09-08T05:59:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:55 smithi055 ceph-mon[97033]: pgmap v16253: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:55 smithi059 ceph-mon[97450]: pgmap v16253: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:56.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:56.370581+0000) 2022-09-08T05:59:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:56.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:56.370565+0000) 2022-09-08T05:59:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:57.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:57.370760+0000) 2022-09-08T05:59:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:57.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:57.370756+0000) 2022-09-08T05:59:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:57 smithi055 ceph-mon[97033]: pgmap v16254: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:59:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:57 smithi059 ceph-mon[97450]: pgmap v16254: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T05:59:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:58.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:58.370995+0000) 2022-09-08T05:59:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:58.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:58.370996+0000) 2022-09-08T05:59:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 05:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T05:59:59.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:59.371292+0000) 2022-09-08T05:59:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 05:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T05:59:59.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T04:59:59.371282+0000) 2022-09-08T05:59:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 05:59:59 smithi055 ceph-mon[97033]: pgmap v16255: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T05:59:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 05:59:59 smithi059 ceph-mon[97450]: pgmap v16255: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:00.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:00.371590+0000) 2022-09-08T06:00:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:00.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:00.371536+0000) 2022-09-08T06:00:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T06:00:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T06:00:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T06:00:00.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T06:00:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T06:00:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T06:00:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:01.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:01.371884+0000) 2022-09-08T06:00:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:01.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:01.371764+0000) 2022-09-08T06:00:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:01 smithi059 ceph-mon[97450]: pgmap v16256: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:00:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:01 smithi055 ceph-mon[97033]: pgmap v16256: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:00:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:02.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:02.372074+0000) 2022-09-08T06:00:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:02.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:02.371955+0000) 2022-09-08T06:00:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:03.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:03.372260+0000) 2022-09-08T06:00:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:03.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:03.372196+0000) 2022-09-08T06:00:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:03 smithi059 ceph-mon[97450]: pgmap v16257: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:03 smithi055 ceph-mon[97033]: pgmap v16257: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:04.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:00:04.326+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:00:04.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:04.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:04.372438+0000) 2022-09-08T06:00:04.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:04.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:04.372398+0000) 2022-09-08T06:00:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:05.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:05.372680+0000) 2022-09-08T06:00:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:05.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:05.372684+0000) 2022-09-08T06:00:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:05 smithi059 ceph-mon[97450]: pgmap v16258: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:05 smithi055 ceph-mon[97033]: pgmap v16258: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:06.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:06.372883+0000) 2022-09-08T06:00:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:06.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:06.372872+0000) 2022-09-08T06:00:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:00:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:00:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:07.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:07.373178+0000) 2022-09-08T06:00:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:07.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:07.373106+0000) 2022-09-08T06:00:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:07 smithi059 ceph-mon[97450]: pgmap v16259: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:00:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:00:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:07 smithi055 ceph-mon[97033]: pgmap v16259: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:00:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:00:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:08.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:08.373368+0000) 2022-09-08T06:00:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:08.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:08.373363+0000) 2022-09-08T06:00:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:09.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:09.373544+0000) 2022-09-08T06:00:09.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:09.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:09.373543+0000) 2022-09-08T06:00:09.793 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:09 smithi059 ceph-mon[97450]: pgmap v16260: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:00:09.794 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:00:09.795 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:00:09.796 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:00:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:09 smithi055 ceph-mon[97033]: pgmap v16260: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:00:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:00:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:00:09.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:00:10.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:10.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:10.373774+0000) 2022-09-08T06:00:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:10.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:10.373761+0000) 2022-09-08T06:00:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:10 smithi059 ceph-mon[97450]: osdmap e1095: 12 total, 11 up, 11 in 2022-09-08T06:00:10.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:00:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:10 smithi055 ceph-mon[97033]: osdmap e1095: 12 total, 11 up, 11 in 2022-09-08T06:00:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:00:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:11.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:11.373994+0000) 2022-09-08T06:00:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:11.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:11.373989+0000) 2022-09-08T06:00:11.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:11 smithi059 ceph-mon[97450]: pgmap v16262: 129 pgs: 6 active+clean+snaptrim_wait, 123 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:00:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:11 smithi059 ceph-mon[97450]: osdmap e1096: 12 total, 11 up, 11 in 2022-09-08T06:00:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:11 smithi055 ceph-mon[97033]: pgmap v16262: 129 pgs: 6 active+clean+snaptrim_wait, 123 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:00:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:11 smithi055 ceph-mon[97033]: osdmap e1096: 12 total, 11 up, 11 in 2022-09-08T06:00:12.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:12.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:12.374170+0000) 2022-09-08T06:00:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:12.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:12.374170+0000) 2022-09-08T06:00:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:13.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:13.374362+0000) 2022-09-08T06:00:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:13.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:13.374382+0000) 2022-09-08T06:00:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:13 smithi059 ceph-mon[97450]: pgmap v16264: 129 pgs: 6 active+clean+snaptrim_wait, 123 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:00:13.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:13 smithi055 ceph-mon[97033]: pgmap v16264: 129 pgs: 6 active+clean+snaptrim_wait, 123 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:00:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:14.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:14.374660+0000) 2022-09-08T06:00:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:14.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:14.374681+0000) 2022-09-08T06:00:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:15.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:15.374882+0000) 2022-09-08T06:00:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:15.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:15.374889+0000) 2022-09-08T06:00:15.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:15 smithi059 ceph-mon[97450]: pgmap v16265: 129 pgs: 6 active+clean+snaptrim_wait, 123 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:00:15.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:15 smithi055 ceph-mon[97033]: pgmap v16265: 129 pgs: 6 active+clean+snaptrim_wait, 123 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:00:16.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:16.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:16.375131+0000) 2022-09-08T06:00:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:16.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:16.375138+0000) 2022-09-08T06:00:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:17.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:17.375376+0000) 2022-09-08T06:00:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:17.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:17.375365+0000) 2022-09-08T06:00:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:17 smithi059 ceph-mon[97450]: pgmap v16266: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:00:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:17 smithi055 ceph-mon[97033]: pgmap v16266: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:00:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:18.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:18.375550+0000) 2022-09-08T06:00:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:18.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:18.375522+0000) 2022-09-08T06:00:19.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:19.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:19.375732+0000) 2022-09-08T06:00:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:19.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:19.375699+0000) 2022-09-08T06:00:19.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:00:19.339+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:00:19.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:19 smithi055 ceph-mon[97033]: pgmap v16267: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 235 B/s wr, 7 op/s 2022-09-08T06:00:20.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:19 smithi059 ceph-mon[97450]: pgmap v16267: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 235 B/s wr, 7 op/s 2022-09-08T06:00:20.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:20.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:20.375971+0000) 2022-09-08T06:00:20.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:20.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:20.375879+0000) 2022-09-08T06:00:21.662 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:21.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:21.376161+0000) 2022-09-08T06:00:21.663 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:21.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:21.376086+0000) 2022-09-08T06:00:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:21 smithi055 ceph-mon[97033]: pgmap v16268: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:00:21.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:21 smithi055 ceph-mon[97033]: osdmap e1097: 12 total, 11 up, 11 in 2022-09-08T06:00:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:21 smithi059 ceph-mon[97450]: pgmap v16268: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:00:22.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:21 smithi059 ceph-mon[97450]: osdmap e1097: 12 total, 11 up, 11 in 2022-09-08T06:00:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:22.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:22.376411+0000) 2022-09-08T06:00:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:22.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:22.376338+0000) 2022-09-08T06:00:23.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:23.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:23.376589+0000) 2022-09-08T06:00:23.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:23.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:23.376555+0000) 2022-09-08T06:00:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:23 smithi055 ceph-mon[97033]: pgmap v16270: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:00:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:23 smithi059 ceph-mon[97450]: pgmap v16270: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:00:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:24.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:24.376831+0000) 2022-09-08T06:00:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:24.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:24.376830+0000) 2022-09-08T06:00:25.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:25.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:25.377080+0000) 2022-09-08T06:00:25.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:25.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:25.377080+0000) 2022-09-08T06:00:25.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:25 smithi055 ceph-mon[97033]: pgmap v16271: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:00:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:25 smithi059 ceph-mon[97450]: pgmap v16271: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:00:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:26.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:26.377296+0000) 2022-09-08T06:00:26.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:26.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:26.377297+0000) 2022-09-08T06:00:27.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:27.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:27.377553+0000) 2022-09-08T06:00:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:27.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:27.377553+0000) 2022-09-08T06:00:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:27 smithi055 ceph-mon[97033]: pgmap v16272: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:00:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:27 smithi059 ceph-mon[97450]: pgmap v16272: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:00:28.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:28.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:28.377824+0000) 2022-09-08T06:00:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:28.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:28.377831+0000) 2022-09-08T06:00:29.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:29.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:29.378069+0000) 2022-09-08T06:00:29.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:29.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:29.378069+0000) 2022-09-08T06:00:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:29 smithi055 ceph-mon[97033]: pgmap v16273: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:00:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:29 smithi059 ceph-mon[97450]: pgmap v16273: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:00:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:30.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:30.378312+0000) 2022-09-08T06:00:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:30.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:30.378308+0000) 2022-09-08T06:00:31.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:31.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:31.378573+0000) 2022-09-08T06:00:31.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:31.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:31.378572+0000) 2022-09-08T06:00:31.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:31 smithi055 ceph-mon[97033]: pgmap v16274: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:00:32.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:31 smithi059 ceph-mon[97450]: pgmap v16274: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:00:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:32.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:32.378819+0000) 2022-09-08T06:00:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:32.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:32.378818+0000) 2022-09-08T06:00:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:33.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:33.379055+0000) 2022-09-08T06:00:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:33.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:33.379055+0000) 2022-09-08T06:00:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:33 smithi055 ceph-mon[97033]: pgmap v16275: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:00:34.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:33 smithi059 ceph-mon[97450]: pgmap v16275: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:00:34.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:00:34.341+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:00:34.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:34.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:34.379332+0000) 2022-09-08T06:00:34.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:34.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:34.379332+0000) 2022-09-08T06:00:35.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:35.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:35.379514+0000) 2022-09-08T06:00:35.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:35.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:35.379513+0000) 2022-09-08T06:00:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:35 smithi055 ceph-mon[97033]: pgmap v16276: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:35 smithi059 ceph-mon[97450]: pgmap v16276: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:36.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:36.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:36.379758+0000) 2022-09-08T06:00:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:36.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:36.379756+0000) 2022-09-08T06:00:37.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:37.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:37.379953+0000) 2022-09-08T06:00:37.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:37.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:37.379953+0000) 2022-09-08T06:00:37.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:37 smithi055 ceph-mon[97033]: pgmap v16277: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:00:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:37 smithi059 ceph-mon[97450]: pgmap v16277: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:00:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:38.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:38.380233+0000) 2022-09-08T06:00:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:38.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:38.380224+0000) 2022-09-08T06:00:39.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:39.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:39.380445+0000) 2022-09-08T06:00:39.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:39.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:39.380442+0000) 2022-09-08T06:00:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:39 smithi055 ceph-mon[97033]: pgmap v16278: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:39 smithi059 ceph-mon[97450]: pgmap v16278: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:40.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:40.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:40.380624+0000) 2022-09-08T06:00:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:40.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:40.380624+0000) 2022-09-08T06:00:41.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:41.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:41.380850+0000) 2022-09-08T06:00:41.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:41.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:41.380815+0000) 2022-09-08T06:00:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:41 smithi055 ceph-mon[97033]: pgmap v16279: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:00:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:41 smithi059 ceph-mon[97450]: pgmap v16279: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:00:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:42.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:42.381120+0000) 2022-09-08T06:00:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:42.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:42.381120+0000) 2022-09-08T06:00:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:43.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:43.381426+0000) 2022-09-08T06:00:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:43.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:43.381426+0000) 2022-09-08T06:00:43.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:43 smithi055 ceph-mon[97033]: pgmap v16280: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:43 smithi059 ceph-mon[97450]: pgmap v16280: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:44.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:44.381750+0000) 2022-09-08T06:00:44.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:44.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:44.381747+0000) 2022-09-08T06:00:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:45.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:45.381977+0000) 2022-09-08T06:00:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:45.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:45.381978+0000) 2022-09-08T06:00:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:45 smithi055 ceph-mon[97033]: pgmap v16281: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:45 smithi059 ceph-mon[97450]: pgmap v16281: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:46.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:46.382229+0000) 2022-09-08T06:00:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:46.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:46.382228+0000) 2022-09-08T06:00:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:47.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:47.382444+0000) 2022-09-08T06:00:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:47.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:47.382441+0000) 2022-09-08T06:00:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:47 smithi055 ceph-mon[97033]: pgmap v16282: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:00:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:47 smithi059 ceph-mon[97450]: pgmap v16282: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:00:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:48.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:48.382662+0000) 2022-09-08T06:00:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:48.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:48.382660+0000) 2022-09-08T06:00:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:49.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:49.382916+0000) 2022-09-08T06:00:49.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:00:49.349+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:00:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:49.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:49.382916+0000) 2022-09-08T06:00:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:49 smithi055 ceph-mon[97033]: pgmap v16283: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:49 smithi059 ceph-mon[97450]: pgmap v16283: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:50.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:50.383196+0000) 2022-09-08T06:00:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:50.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:50.383196+0000) 2022-09-08T06:00:51.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:51.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:51.383484+0000) 2022-09-08T06:00:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:51.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:51.383480+0000) 2022-09-08T06:00:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:51 smithi055 ceph-mon[97033]: pgmap v16284: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:00:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:51 smithi059 ceph-mon[97450]: pgmap v16284: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:00:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:52.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:52.383764+0000) 2022-09-08T06:00:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:52.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:52.383762+0000) 2022-09-08T06:00:53.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:53.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:53.383986+0000) 2022-09-08T06:00:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:53.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:53.383983+0000) 2022-09-08T06:00:53.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:53 smithi055 ceph-mon[97033]: pgmap v16285: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:54.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:53 smithi059 ceph-mon[97450]: pgmap v16285: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:54.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:54.384232+0000) 2022-09-08T06:00:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:54.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:54.384232+0000) 2022-09-08T06:00:55.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:55.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:55.384436+0000) 2022-09-08T06:00:55.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:55.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:55.384435+0000) 2022-09-08T06:00:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:55 smithi055 ceph-mon[97033]: pgmap v16286: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:55 smithi059 ceph-mon[97450]: pgmap v16286: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:00:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:56.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:56.384743+0000) 2022-09-08T06:00:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:56.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:56.384746+0000) 2022-09-08T06:00:57.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:57.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:57.384908+0000) 2022-09-08T06:00:57.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:57.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:57.384908+0000) 2022-09-08T06:00:57.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:57 smithi055 ceph-mon[97033]: pgmap v16287: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:00:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:57 smithi059 ceph-mon[97450]: pgmap v16287: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:00:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:58.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:58.385150+0000) 2022-09-08T06:00:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:58.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:58.385162+0000) 2022-09-08T06:00:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:00:59.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:59.385389+0000) 2022-09-08T06:00:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:00:59.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:00:59.385389+0000) 2022-09-08T06:00:59.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:00:59 smithi055 ceph-mon[97033]: pgmap v16288: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:00.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:00:59 smithi059 ceph-mon[97450]: pgmap v16288: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:00.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:00.385541+0000) 2022-09-08T06:01:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:00.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:00.385542+0000) 2022-09-08T06:01:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:01.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:01.385769+0000) 2022-09-08T06:01:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:01.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:01.385782+0000) 2022-09-08T06:01:01.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:01 smithi055 ceph-mon[97033]: pgmap v16289: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:01:02.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:01 smithi059 ceph-mon[97450]: pgmap v16289: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:01:02.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:02.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:02.385962+0000) 2022-09-08T06:01:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:02.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:02.385981+0000) 2022-09-08T06:01:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:03.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:03.386256+0000) 2022-09-08T06:01:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:03.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:03.386199+0000) 2022-09-08T06:01:03.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:03 smithi055 ceph-mon[97033]: pgmap v16290: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:04.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:03 smithi059 ceph-mon[97450]: pgmap v16290: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:04.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:01:04.350+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:01:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:04.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:04.386448+0000) 2022-09-08T06:01:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:04.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:04.386552+0000) 2022-09-08T06:01:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:05.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:05.386766+0000) 2022-09-08T06:01:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:05.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:05.386743+0000) 2022-09-08T06:01:05.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:05 smithi055 ceph-mon[97033]: pgmap v16291: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:06.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:05 smithi059 ceph-mon[97450]: pgmap v16291: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:06.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:06.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:06.387010+0000) 2022-09-08T06:01:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:06.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:06.386979+0000) 2022-09-08T06:01:06.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:01:07.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:01:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:07.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:07.387199+0000) 2022-09-08T06:01:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:07.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:07.387157+0000) 2022-09-08T06:01:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:07 smithi055 ceph-mon[97033]: pgmap v16292: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:01:07.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:01:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:07 smithi059 ceph-mon[97450]: pgmap v16292: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:01:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:01:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:08.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:08.387409+0000) 2022-09-08T06:01:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:08.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:08.387444+0000) 2022-09-08T06:01:09.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:09.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:09.387651+0000) 2022-09-08T06:01:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:09.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:09.387625+0000) 2022-09-08T06:01:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:09 smithi055 ceph-mon[97033]: pgmap v16293: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:01:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:09 smithi059 ceph-mon[97450]: pgmap v16293: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:01:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:10.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:10.387856+0000) 2022-09-08T06:01:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:10.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:10.387848+0000) 2022-09-08T06:01:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:10 smithi055 ceph-mon[97033]: osdmap e1098: 12 total, 11 up, 11 in 2022-09-08T06:01:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:01:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:01:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:01:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:01:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:01:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:10 smithi059 ceph-mon[97450]: osdmap e1098: 12 total, 11 up, 11 in 2022-09-08T06:01:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:01:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:01:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:01:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:01:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:01:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:11.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:11.388044+0000) 2022-09-08T06:01:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:11.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:11.388044+0000) 2022-09-08T06:01:11.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:11 smithi055 ceph-mon[97033]: pgmap v16295: 129 pgs: 1 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:01:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:11 smithi059 ceph-mon[97450]: pgmap v16295: 129 pgs: 1 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:01:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:12.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:12.388260+0000) 2022-09-08T06:01:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:12.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:12.388250+0000) 2022-09-08T06:01:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:13.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:13.388461+0000) 2022-09-08T06:01:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:13.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:13.388446+0000) 2022-09-08T06:01:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:13 smithi055 ceph-mon[97033]: pgmap v16296: 129 pgs: 1 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:01:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:01:13.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:01:13.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:01:13.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:01:13.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:13.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:13.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:13.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:13 smithi059 ceph-mon[97450]: pgmap v16296: 129 pgs: 1 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:01:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:01:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:01:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:01:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:01:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:14.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:14.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:14.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:14.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:14.388660+0000) 2022-09-08T06:01:14.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:14.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:14.388642+0000) 2022-09-08T06:01:14.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:14 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T06:01:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:14 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T06:01:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:14.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:14.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:14 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T06:01:14.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:14 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T06:01:14.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:01:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:14 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 197.2M 2022-09-08T06:01:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:14 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 206879232: error parsing value: Value '206879232' is below minimum 939524096 2022-09-08T06:01:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:15.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:15.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:01:15.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:14 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T06:01:15.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:14 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T06:01:15.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:01:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:15.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:15.388814+0000) 2022-09-08T06:01:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:15.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:15.388766+0000) 2022-09-08T06:01:16.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:15 smithi055 ceph-mon[97033]: pgmap v16297: 129 pgs: 1 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:01:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:15 smithi059 ceph-mon[97450]: pgmap v16297: 129 pgs: 1 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:01:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:16.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:16.389006+0000) 2022-09-08T06:01:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:16.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:16.389062+0000) 2022-09-08T06:01:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:17.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:17.389216+0000) 2022-09-08T06:01:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:17.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:17.389322+0000) 2022-09-08T06:01:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:17 smithi055 ceph-mon[97033]: pgmap v16298: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:01:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:17 smithi059 ceph-mon[97450]: pgmap v16298: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:01:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:18.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:18.389445+0000) 2022-09-08T06:01:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:18.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:18.389509+0000) 2022-09-08T06:01:19.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:01:19.356+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:01:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:19.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:19.389649+0000) 2022-09-08T06:01:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:19.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:19.389696+0000) 2022-09-08T06:01:20.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:19 smithi055 ceph-mon[97033]: pgmap v16299: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:01:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:19 smithi059 ceph-mon[97450]: pgmap v16299: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:01:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:20.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:20.389831+0000) 2022-09-08T06:01:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:20.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:20.389881+0000) 2022-09-08T06:01:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:21.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:21.390132+0000) 2022-09-08T06:01:21.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:21.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:21.390070+0000) 2022-09-08T06:01:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:21 smithi055 ceph-mon[97033]: pgmap v16300: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 193 B/s wr, 8 op/s 2022-09-08T06:01:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:21 smithi055 ceph-mon[97033]: osdmap e1099: 12 total, 11 up, 11 in 2022-09-08T06:01:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:21 smithi059 ceph-mon[97450]: pgmap v16300: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 193 B/s wr, 8 op/s 2022-09-08T06:01:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:21 smithi059 ceph-mon[97450]: osdmap e1099: 12 total, 11 up, 11 in 2022-09-08T06:01:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:22.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:22.390336+0000) 2022-09-08T06:01:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:22.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:22.390287+0000) 2022-09-08T06:01:23.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:23.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:23.390543+0000) 2022-09-08T06:01:23.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:23.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:23.390531+0000) 2022-09-08T06:01:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:23 smithi055 ceph-mon[97033]: pgmap v16302: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T06:01:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:23 smithi059 ceph-mon[97450]: pgmap v16302: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T06:01:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:24.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:24.390831+0000) 2022-09-08T06:01:24.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:24.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:24.390792+0000) 2022-09-08T06:01:25.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:25.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:25.391084+0000) 2022-09-08T06:01:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:25.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:25.391028+0000) 2022-09-08T06:01:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:25 smithi055 ceph-mon[97033]: pgmap v16303: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T06:01:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:25 smithi059 ceph-mon[97450]: pgmap v16303: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T06:01:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:26.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:26.391296+0000) 2022-09-08T06:01:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:26.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:26.391280+0000) 2022-09-08T06:01:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:27.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:27.391462+0000) 2022-09-08T06:01:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:27.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:27.391439+0000) 2022-09-08T06:01:28.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:27 smithi055 ceph-mon[97033]: pgmap v16304: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:01:28.294 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:27 smithi059 ceph-mon[97450]: pgmap v16304: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:01:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:28.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:28.391668+0000) 2022-09-08T06:01:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:28.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:28.391678+0000) 2022-09-08T06:01:29.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:29.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:29.391863+0000) 2022-09-08T06:01:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:29.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:29.391837+0000) 2022-09-08T06:01:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:29 smithi055 ceph-mon[97033]: pgmap v16305: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:01:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:29 smithi059 ceph-mon[97450]: pgmap v16305: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:01:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:30.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:30.392071+0000) 2022-09-08T06:01:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:30.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:30.392071+0000) 2022-09-08T06:01:31.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:31.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:31.392347+0000) 2022-09-08T06:01:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:31.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:31.392354+0000) 2022-09-08T06:01:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:31 smithi055 ceph-mon[97033]: pgmap v16306: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:01:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:31 smithi059 ceph-mon[97450]: pgmap v16306: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:01:32.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:32.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:32.392687+0000) 2022-09-08T06:01:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:32.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:32.392687+0000) 2022-09-08T06:01:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:33.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:33.392959+0000) 2022-09-08T06:01:33.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:33.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:33.392962+0000) 2022-09-08T06:01:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:33 smithi055 ceph-mon[97033]: pgmap v16307: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:01:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:33 smithi059 ceph-mon[97450]: pgmap v16307: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:01:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:34.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:34.393132+0000) 2022-09-08T06:01:34.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:34.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:34.393132+0000) 2022-09-08T06:01:34.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:01:34.371+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:01:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:35.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:35.393329+0000) 2022-09-08T06:01:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:35.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:35.393327+0000) 2022-09-08T06:01:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:35 smithi055 ceph-mon[97033]: pgmap v16308: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:35 smithi059 ceph-mon[97450]: pgmap v16308: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:36.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:36.393605+0000) 2022-09-08T06:01:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:36.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:36.393604+0000) 2022-09-08T06:01:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:37.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:37.393813+0000) 2022-09-08T06:01:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:37.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:37.393814+0000) 2022-09-08T06:01:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:37 smithi055 ceph-mon[97033]: pgmap v16309: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:01:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:37 smithi059 ceph-mon[97450]: pgmap v16309: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:01:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:38.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:38.393962+0000) 2022-09-08T06:01:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:38.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:38.393978+0000) 2022-09-08T06:01:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:39.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:39.394124+0000) 2022-09-08T06:01:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:39.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:39.394124+0000) 2022-09-08T06:01:40.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:39 smithi055 ceph-mon[97033]: pgmap v16310: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:39 smithi059 ceph-mon[97450]: pgmap v16310: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:40.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:40.394364+0000) 2022-09-08T06:01:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:40.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:40.394364+0000) 2022-09-08T06:01:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:41.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:41.394666+0000) 2022-09-08T06:01:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:41.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:41.394654+0000) 2022-09-08T06:01:42.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:41 smithi055 ceph-mon[97033]: pgmap v16311: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:01:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:41 smithi059 ceph-mon[97450]: pgmap v16311: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:01:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:42.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:42.394909+0000) 2022-09-08T06:01:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:42.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:42.394909+0000) 2022-09-08T06:01:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:43.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:43.395163+0000) 2022-09-08T06:01:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:43.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:43.395159+0000) 2022-09-08T06:01:44.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:43 smithi055 ceph-mon[97033]: pgmap v16312: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:43 smithi059 ceph-mon[97450]: pgmap v16312: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:44.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:44.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:44.395433+0000) 2022-09-08T06:01:44.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:44.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:44.395458+0000) 2022-09-08T06:01:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:45.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:45.395617+0000) 2022-09-08T06:01:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:45.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:45.395647+0000) 2022-09-08T06:01:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:45 smithi055 ceph-mon[97033]: pgmap v16313: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:45 smithi059 ceph-mon[97450]: pgmap v16313: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:46.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:46.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:46.395844+0000) 2022-09-08T06:01:46.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:46.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:46.395853+0000) 2022-09-08T06:01:47.661 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:47.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:47.396057+0000) 2022-09-08T06:01:47.662 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:47.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:47.396085+0000) 2022-09-08T06:01:48.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:47 smithi055 ceph-mon[97033]: pgmap v16314: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:01:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:47 smithi059 ceph-mon[97450]: pgmap v16314: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:01:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:48.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:48.396251+0000) 2022-09-08T06:01:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:48.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:48.396283+0000) 2022-09-08T06:01:49.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:01:49.381+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:01:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:49.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:49.396375+0000) 2022-09-08T06:01:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:49.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:49.396419+0000) 2022-09-08T06:01:50.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:49 smithi055 ceph-mon[97033]: pgmap v16315: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:49 smithi059 ceph-mon[97450]: pgmap v16315: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:50.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:50.396650+0000) 2022-09-08T06:01:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:50.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:50.396669+0000) 2022-09-08T06:01:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:51.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:51.396881+0000) 2022-09-08T06:01:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:51.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:51.396886+0000) 2022-09-08T06:01:52.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:51 smithi055 ceph-mon[97033]: pgmap v16316: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:01:52.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:51 smithi059 ceph-mon[97450]: pgmap v16316: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:01:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:52.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:52.397086+0000) 2022-09-08T06:01:52.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:52.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:52.397105+0000) 2022-09-08T06:01:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:53.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:53.397243+0000) 2022-09-08T06:01:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:53.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:53.397246+0000) 2022-09-08T06:01:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:53 smithi055 ceph-mon[97033]: pgmap v16317: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:53 smithi059 ceph-mon[97450]: pgmap v16317: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:54.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:54.397416+0000) 2022-09-08T06:01:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:54.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:54.397416+0000) 2022-09-08T06:01:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:55.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:55.397653+0000) 2022-09-08T06:01:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:55.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:55.397653+0000) 2022-09-08T06:01:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:55 smithi055 ceph-mon[97033]: pgmap v16318: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:55 smithi059 ceph-mon[97450]: pgmap v16318: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:01:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:56.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:56.397868+0000) 2022-09-08T06:01:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:56.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:56.397868+0000) 2022-09-08T06:01:57.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:57.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:57.398122+0000) 2022-09-08T06:01:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:57.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:57.398125+0000) 2022-09-08T06:01:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:57 smithi055 ceph-mon[97033]: pgmap v16319: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:01:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:57 smithi059 ceph-mon[97450]: pgmap v16319: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:01:58.652 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:58.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:58.398359+0000) 2022-09-08T06:01:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:58.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:58.398352+0000) 2022-09-08T06:01:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:01:59.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:59.398595+0000) 2022-09-08T06:01:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:01:59.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:01:59.398595+0000) 2022-09-08T06:02:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:01:59 smithi055 ceph-mon[97033]: pgmap v16320: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:01:59 smithi059 ceph-mon[97450]: pgmap v16320: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:00.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:00.398807+0000) 2022-09-08T06:02:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:00.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:00.398807+0000) 2022-09-08T06:02:01.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:01.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:01.398986+0000) 2022-09-08T06:02:01.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:01.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:01.398983+0000) 2022-09-08T06:02:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:01 smithi055 ceph-mon[97033]: pgmap v16321: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:02:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:01 smithi059 ceph-mon[97450]: pgmap v16321: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:02:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:02.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:02.399242+0000) 2022-09-08T06:02:02.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:02.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:02.399239+0000) 2022-09-08T06:02:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:03.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:03.399463+0000) 2022-09-08T06:02:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:03.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:03.399463+0000) 2022-09-08T06:02:04.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:03 smithi055 ceph-mon[97033]: pgmap v16322: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:03 smithi059 ceph-mon[97450]: pgmap v16322: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:04.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:04.399737+0000) 2022-09-08T06:02:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:04.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:04.399725+0000) 2022-09-08T06:02:04.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:02:04.387+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:02:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:05.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:05.399904+0000) 2022-09-08T06:02:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:05.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:05.399909+0000) 2022-09-08T06:02:06.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:05 smithi055 ceph-mon[97033]: pgmap v16323: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:02:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:05 smithi059 ceph-mon[97450]: pgmap v16323: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:02:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:06.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:06.400149+0000) 2022-09-08T06:02:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:06.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:06.400149+0000) 2022-09-08T06:02:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:07 smithi059 ceph-mon[97450]: pgmap v16324: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:02:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:02:07.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:07.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:07.400398+0000) 2022-09-08T06:02:07.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:07.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:07.400398+0000) 2022-09-08T06:02:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:07 smithi055 ceph-mon[97033]: pgmap v16324: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:02:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:02:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:08.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:08.400696+0000) 2022-09-08T06:02:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:08.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:08.400695+0000) 2022-09-08T06:02:09.539 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:09 smithi059 ceph-mon[97450]: pgmap v16325: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:02:09.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:09.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:09.400899+0000) 2022-09-08T06:02:09.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:09.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:09.401006+0000) 2022-09-08T06:02:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:09 smithi055 ceph-mon[97033]: pgmap v16325: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:02:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:10.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:10.401140+0000) 2022-09-08T06:02:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:10.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:10.401245+0000) 2022-09-08T06:02:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:10 smithi055 ceph-mon[97033]: osdmap e1100: 12 total, 11 up, 11 in 2022-09-08T06:02:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:10 smithi059 ceph-mon[97450]: osdmap e1100: 12 total, 11 up, 11 in 2022-09-08T06:02:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:11 smithi055 ceph-mon[97033]: pgmap v16327: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:02:11.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:11 smithi055 ceph-mon[97033]: osdmap e1101: 12 total, 11 up, 11 in 2022-09-08T06:02:11.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:11 smithi059 ceph-mon[97450]: pgmap v16327: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:02:11.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:11 smithi059 ceph-mon[97450]: osdmap e1101: 12 total, 11 up, 11 in 2022-09-08T06:02:11.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:11.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:11.401468+0000) 2022-09-08T06:02:11.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:11.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:11.401334+0000) 2022-09-08T06:02:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:12.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:12.401769+0000) 2022-09-08T06:02:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:12.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:12.401583+0000) 2022-09-08T06:02:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:13 smithi055 ceph-mon[97033]: pgmap v16329: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:02:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:13 smithi059 ceph-mon[97450]: pgmap v16329: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:02:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:13.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:13.401962+0000) 2022-09-08T06:02:13.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:13.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:13.401758+0000) 2022-09-08T06:02:14.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:02:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:02:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:02:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:02:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:02:14.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:02:14.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:14.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:14.402122+0000) 2022-09-08T06:02:14.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:14.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:14.401971+0000) 2022-09-08T06:02:15.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:15 smithi055 ceph-mon[97033]: pgmap v16330: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:02:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:02:15.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:15 smithi059 ceph-mon[97450]: pgmap v16330: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:02:15.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:02:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:15.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:15.402227+0000) 2022-09-08T06:02:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:15.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:15.402328+0000) 2022-09-08T06:02:16.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:16.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:16.402477+0000) 2022-09-08T06:02:16.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:16.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:16.402581+0000) 2022-09-08T06:02:17.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:17 smithi055 ceph-mon[97033]: pgmap v16331: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:02:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:17 smithi059 ceph-mon[97450]: pgmap v16331: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:02:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:17.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:17.402724+0000) 2022-09-08T06:02:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:17.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:17.402765+0000) 2022-09-08T06:02:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:18.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:18.402945+0000) 2022-09-08T06:02:18.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:18.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:18.402974+0000) 2022-09-08T06:02:19.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:02:19.392+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:02:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:19.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:19.403130+0000) 2022-09-08T06:02:19.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:19.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:19.403081+0000) 2022-09-08T06:02:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:19 smithi055 ceph-mon[97033]: pgmap v16332: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 228 B/s wr, 6 op/s 2022-09-08T06:02:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:19 smithi059 ceph-mon[97450]: pgmap v16332: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 228 B/s wr, 6 op/s 2022-09-08T06:02:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:20.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:20.403312+0000) 2022-09-08T06:02:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:20.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:20.403361+0000) 2022-09-08T06:02:21.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:21 smithi055 ceph-mon[97033]: pgmap v16333: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:02:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:21 smithi059 ceph-mon[97450]: pgmap v16333: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:02:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:21.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:21.403570+0000) 2022-09-08T06:02:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:21.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:21.403578+0000) 2022-09-08T06:02:22.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:22.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:22.403779+0000) 2022-09-08T06:02:22.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:22.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:22.403773+0000) 2022-09-08T06:02:23.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:23 smithi055 ceph-mon[97033]: pgmap v16334: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T06:02:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:23 smithi059 ceph-mon[97450]: pgmap v16334: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T06:02:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:23.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:23.404021+0000) 2022-09-08T06:02:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:23.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:23.404006+0000) 2022-09-08T06:02:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:24.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:24.404220+0000) 2022-09-08T06:02:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:24.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:24.404202+0000) 2022-09-08T06:02:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:25 smithi055 ceph-mon[97033]: pgmap v16335: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:02:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:25 smithi059 ceph-mon[97450]: pgmap v16335: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:02:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:25.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:25.404465+0000) 2022-09-08T06:02:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:25.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:25.404445+0000) 2022-09-08T06:02:26.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:26.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:26.404622+0000) 2022-09-08T06:02:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:26.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:26.404669+0000) 2022-09-08T06:02:27.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:27 smithi055 ceph-mon[97033]: pgmap v16336: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:02:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:27 smithi059 ceph-mon[97450]: pgmap v16336: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:02:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:27.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:27.404762+0000) 2022-09-08T06:02:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:27.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:27.404802+0000) 2022-09-08T06:02:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:28.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:28.404944+0000) 2022-09-08T06:02:28.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:28.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:28.404970+0000) 2022-09-08T06:02:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:29 smithi055 ceph-mon[97033]: pgmap v16337: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T06:02:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:29 smithi059 ceph-mon[97450]: pgmap v16337: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T06:02:29.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:29.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:29.405184+0000) 2022-09-08T06:02:29.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:29.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:29.405169+0000) 2022-09-08T06:02:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:30.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:30.405436+0000) 2022-09-08T06:02:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:30.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:30.405432+0000) 2022-09-08T06:02:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:31 smithi055 ceph-mon[97033]: pgmap v16338: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T06:02:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:31 smithi059 ceph-mon[97450]: pgmap v16338: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T06:02:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:31.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:31.405674+0000) 2022-09-08T06:02:31.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:31.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:31.405668+0000) 2022-09-08T06:02:32.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:32.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:32.405892+0000) 2022-09-08T06:02:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:32.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:32.405886+0000) 2022-09-08T06:02:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:33 smithi055 ceph-mon[97033]: pgmap v16339: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:33 smithi059 ceph-mon[97450]: pgmap v16339: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:33.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:33.406088+0000) 2022-09-08T06:02:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:33.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:33.406086+0000) 2022-09-08T06:02:34.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:02:34.408+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:02:34.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:34.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:34.406347+0000) 2022-09-08T06:02:34.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:34.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:34.406326+0000) 2022-09-08T06:02:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:35 smithi055 ceph-mon[97033]: pgmap v16340: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:35.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:35 smithi059 ceph-mon[97450]: pgmap v16340: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:35.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:35.406605+0000) 2022-09-08T06:02:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:35.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:35.406588+0000) 2022-09-08T06:02:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:36.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:36.406914+0000) 2022-09-08T06:02:36.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:36.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:36.406906+0000) 2022-09-08T06:02:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:37 smithi055 ceph-mon[97033]: pgmap v16341: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:02:37.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:37 smithi059 ceph-mon[97450]: pgmap v16341: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:02:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:37.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:37.407186+0000) 2022-09-08T06:02:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:37.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:37.407146+0000) 2022-09-08T06:02:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:38.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:38.407399+0000) 2022-09-08T06:02:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:38.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:38.407460+0000) 2022-09-08T06:02:39.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:39 smithi055 ceph-mon[97033]: pgmap v16342: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:39.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:39 smithi059 ceph-mon[97450]: pgmap v16342: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:39.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:39.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:39.407749+0000) 2022-09-08T06:02:39.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:39.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:39.407726+0000) 2022-09-08T06:02:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:40.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:40.407898+0000) 2022-09-08T06:02:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:40.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:40.407900+0000) 2022-09-08T06:02:41.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:41 smithi055 ceph-mon[97033]: pgmap v16343: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:02:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:41 smithi059 ceph-mon[97450]: pgmap v16343: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:02:41.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:41.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:41.408047+0000) 2022-09-08T06:02:41.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:41.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:41.408050+0000) 2022-09-08T06:02:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:42.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:42.408305+0000) 2022-09-08T06:02:42.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:42.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:42.408303+0000) 2022-09-08T06:02:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:43 smithi055 ceph-mon[97033]: pgmap v16344: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:43 smithi059 ceph-mon[97450]: pgmap v16344: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:43.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:43.408588+0000) 2022-09-08T06:02:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:43.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:43.408606+0000) 2022-09-08T06:02:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:44.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:44.408860+0000) 2022-09-08T06:02:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:44.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:44.408860+0000) 2022-09-08T06:02:45.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:45 smithi055 ceph-mon[97033]: pgmap v16345: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:45 smithi059 ceph-mon[97450]: pgmap v16345: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:45.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:45.409130+0000) 2022-09-08T06:02:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:45.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:45.409128+0000) 2022-09-08T06:02:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:46.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:46.409381+0000) 2022-09-08T06:02:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:46.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:46.409379+0000) 2022-09-08T06:02:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:47 smithi055 ceph-mon[97033]: pgmap v16346: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:02:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:47 smithi059 ceph-mon[97450]: pgmap v16346: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:02:47.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:47.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:47.409580+0000) 2022-09-08T06:02:47.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:47.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:47.409580+0000) 2022-09-08T06:02:48.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:48.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:48.409795+0000) 2022-09-08T06:02:48.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:48.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:48.409795+0000) 2022-09-08T06:02:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:49 smithi055 ceph-mon[97033]: pgmap v16347: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:49 smithi059 ceph-mon[97450]: pgmap v16347: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:49.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:02:49.422+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:02:49.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:49.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:49.410083+0000) 2022-09-08T06:02:49.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:49.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:49.410082+0000) 2022-09-08T06:02:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:50.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:50.410341+0000) 2022-09-08T06:02:50.913 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:50.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:50.410338+0000) 2022-09-08T06:02:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:51 smithi055 ceph-mon[97033]: pgmap v16348: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:02:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:51 smithi059 ceph-mon[97450]: pgmap v16348: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:02:51.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:51.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:51.410559+0000) 2022-09-08T06:02:51.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:51.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:51.410553+0000) 2022-09-08T06:02:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:52.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:52.410729+0000) 2022-09-08T06:02:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:52.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:52.410729+0000) 2022-09-08T06:02:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:53 smithi055 ceph-mon[97033]: pgmap v16349: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:53 smithi059 ceph-mon[97450]: pgmap v16349: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:53.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:53.410951+0000) 2022-09-08T06:02:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:53.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:53.410951+0000) 2022-09-08T06:02:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:54.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:54.411160+0000) 2022-09-08T06:02:54.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:54.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:54.411162+0000) 2022-09-08T06:02:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:55 smithi055 ceph-mon[97033]: pgmap v16350: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:55 smithi059 ceph-mon[97450]: pgmap v16350: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:55.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:55.411370+0000) 2022-09-08T06:02:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:55.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:55.411370+0000) 2022-09-08T06:02:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:56.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:56.411542+0000) 2022-09-08T06:02:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:56.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:56.411540+0000) 2022-09-08T06:02:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:57 smithi059 ceph-mon[97450]: pgmap v16351: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:02:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:57.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:57.411796+0000) 2022-09-08T06:02:57.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:57.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:57.411796+0000) 2022-09-08T06:02:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:57 smithi055 ceph-mon[97033]: pgmap v16351: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:02:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:58.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:58.411981+0000) 2022-09-08T06:02:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:58.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:58.411981+0000) 2022-09-08T06:02:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:02:59 smithi059 ceph-mon[97450]: pgmap v16352: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:02:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:02:59.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:59.412123+0000) 2022-09-08T06:02:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:02:59.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:02:59.412123+0000) 2022-09-08T06:02:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:02:59 smithi055 ceph-mon[97033]: pgmap v16352: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:00.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:00.412372+0000) 2022-09-08T06:03:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:00.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:00.412372+0000) 2022-09-08T06:03:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:01 smithi059 ceph-mon[97450]: pgmap v16353: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:03:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:01.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:01.412688+0000) 2022-09-08T06:03:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:01.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:01.412681+0000) 2022-09-08T06:03:01.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:01 smithi055 ceph-mon[97033]: pgmap v16353: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:03:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:02.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:02.412929+0000) 2022-09-08T06:03:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:02.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:02.412929+0000) 2022-09-08T06:03:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:03 smithi059 ceph-mon[97450]: pgmap v16354: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:03.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:03.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:03.413136+0000) 2022-09-08T06:03:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:03.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:03.413133+0000) 2022-09-08T06:03:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:03 smithi055 ceph-mon[97033]: pgmap v16354: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:04.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:03:04.428+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:03:04.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:04.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:04.413401+0000) 2022-09-08T06:03:04.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:04.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:04.413400+0000) 2022-09-08T06:03:05.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:05 smithi059 ceph-mon[97450]: pgmap v16355: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:05.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:05.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:05.413589+0000) 2022-09-08T06:03:05.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:05.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:05.413589+0000) 2022-09-08T06:03:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:05 smithi055 ceph-mon[97033]: pgmap v16355: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:03:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:06.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:06.413856+0000) 2022-09-08T06:03:06.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:06.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:06.413855+0000) 2022-09-08T06:03:06.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:03:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:07 smithi059 ceph-mon[97450]: pgmap v16356: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:03:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:03:07.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:07.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:07.413994+0000) 2022-09-08T06:03:07.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:07.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:07.413996+0000) 2022-09-08T06:03:07.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:07 smithi055 ceph-mon[97033]: pgmap v16356: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:03:07.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:03:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:08.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:08.414204+0000) 2022-09-08T06:03:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:08.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:08.414204+0000) 2022-09-08T06:03:09.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:09 smithi059 ceph-mon[97450]: pgmap v16357: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:03:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:09.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:09.414411+0000) 2022-09-08T06:03:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:09.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:09.414410+0000) 2022-09-08T06:03:09.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:09 smithi055 ceph-mon[97033]: pgmap v16357: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:03:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:10 smithi059 ceph-mon[97450]: osdmap e1102: 12 total, 11 up, 11 in 2022-09-08T06:03:10.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:10.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:10.414564+0000) 2022-09-08T06:03:10.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:10.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:10.414565+0000) 2022-09-08T06:03:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:10 smithi055 ceph-mon[97033]: osdmap e1102: 12 total, 11 up, 11 in 2022-09-08T06:03:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:11 smithi059 ceph-mon[97450]: pgmap v16359: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:03:11.928 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:11.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:11.414756+0000) 2022-09-08T06:03:11.929 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:11.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:11.414756+0000) 2022-09-08T06:03:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:11 smithi055 ceph-mon[97033]: pgmap v16359: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:03:12.913 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:12.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:12.414978+0000) 2022-09-08T06:03:12.914 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:12.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:12.414970+0000) 2022-09-08T06:03:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:13 smithi059 ceph-mon[97450]: pgmap v16360: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:03:13.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:13.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:13.415258+0000) 2022-09-08T06:03:13.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:13.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:13.415258+0000) 2022-09-08T06:03:13.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:13 smithi055 ceph-mon[97033]: pgmap v16360: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:03:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:03:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:03:14.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:03:14.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:14.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:14.415522+0000) 2022-09-08T06:03:14.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:14.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:14.415522+0000) 2022-09-08T06:03:14.991 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:03:14.992 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:03:14.992 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:03:15.913 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:15.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:15.415771+0000) 2022-09-08T06:03:15.915 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:15.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:15.415772+0000) 2022-09-08T06:03:16.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:15 smithi055 ceph-mon[97033]: pgmap v16361: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:03:16.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:03:16.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:03:16.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:03:16.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:15 smithi059 ceph-mon[97450]: pgmap v16361: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:03:16.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:03:16.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:03:16.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:03:16.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:16.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:16.415975+0000) 2022-09-08T06:03:16.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:16.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:16.415976+0000) 2022-09-08T06:03:17.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:17.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:17.416210+0000) 2022-09-08T06:03:17.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:17.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:17.416209+0000) 2022-09-08T06:03:18.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:17 smithi055 ceph-mon[97033]: pgmap v16362: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:03:18.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:17 smithi059 ceph-mon[97450]: pgmap v16362: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:03:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:18.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:18.416494+0000) 2022-09-08T06:03:18.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:18.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:18.416494+0000) 2022-09-08T06:03:19.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:19.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:19.416776+0000) 2022-09-08T06:03:19.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:19.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:19.416775+0000) 2022-09-08T06:03:19.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:03:19.443+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:03:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:19 smithi055 ceph-mon[97033]: pgmap v16363: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:03:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:19 smithi059 ceph-mon[97450]: pgmap v16363: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:03:20.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:20.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:20.416949+0000) 2022-09-08T06:03:20.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:20.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:20.416949+0000) 2022-09-08T06:03:21.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:21.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:21.417190+0000) 2022-09-08T06:03:21.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:21.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:21.417190+0000) 2022-09-08T06:03:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:21 smithi055 ceph-mon[97033]: pgmap v16364: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 190 B/s wr, 8 op/s 2022-09-08T06:03:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:21 smithi055 ceph-mon[97033]: osdmap e1103: 12 total, 11 up, 11 in 2022-09-08T06:03:22.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:21 smithi059 ceph-mon[97450]: pgmap v16364: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 190 B/s wr, 8 op/s 2022-09-08T06:03:22.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:21 smithi059 ceph-mon[97450]: osdmap e1103: 12 total, 11 up, 11 in 2022-09-08T06:03:22.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:22.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:22.417387+0000) 2022-09-08T06:03:22.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:22.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:22.417397+0000) 2022-09-08T06:03:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:23.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:23.417534+0000) 2022-09-08T06:03:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:23.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:23.417528+0000) 2022-09-08T06:03:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:23 smithi055 ceph-mon[97033]: pgmap v16366: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:03:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:23 smithi059 ceph-mon[97450]: pgmap v16366: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:03:24.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:24.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:24.417757+0000) 2022-09-08T06:03:24.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:24.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:24.417758+0000) 2022-09-08T06:03:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:25.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:25.417900+0000) 2022-09-08T06:03:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:25.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:25.417961+0000) 2022-09-08T06:03:26.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:25 smithi055 ceph-mon[97033]: pgmap v16367: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:03:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:25 smithi059 ceph-mon[97450]: pgmap v16367: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:03:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:26.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:26.418108+0000) 2022-09-08T06:03:26.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:26.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:26.418036+0000) 2022-09-08T06:03:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:27.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:27.418185+0000) 2022-09-08T06:03:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:27.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:27.418234+0000) 2022-09-08T06:03:28.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:27 smithi059 ceph-mon[97450]: pgmap v16368: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:03:28.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:27 smithi055 ceph-mon[97033]: pgmap v16368: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:03:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:28.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:28.418394+0000) 2022-09-08T06:03:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:28.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:28.418303+0000) 2022-09-08T06:03:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:29.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:29.418511+0000) 2022-09-08T06:03:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:29.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:29.418554+0000) 2022-09-08T06:03:30.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:29 smithi055 ceph-mon[97033]: pgmap v16369: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:03:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:29 smithi059 ceph-mon[97450]: pgmap v16369: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:03:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:30.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:30.418771+0000) 2022-09-08T06:03:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:30.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:30.418810+0000) 2022-09-08T06:03:31.915 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:31.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:31.418932+0000) 2022-09-08T06:03:31.916 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:31.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:31.418954+0000) 2022-09-08T06:03:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:31 smithi059 ceph-mon[97450]: pgmap v16370: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:03:32.491 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:31 smithi055 ceph-mon[97033]: pgmap v16370: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:03:32.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:32.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:32.419098+0000) 2022-09-08T06:03:32.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:32.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:32.419125+0000) 2022-09-08T06:03:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:33.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:33.419319+0000) 2022-09-08T06:03:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:33.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:33.419313+0000) 2022-09-08T06:03:34.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:33 smithi059 ceph-mon[97450]: pgmap v16371: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:03:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:33 smithi055 ceph-mon[97033]: pgmap v16371: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:03:34.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:03:34.451+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:03:34.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:34.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:34.419574+0000) 2022-09-08T06:03:34.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:34.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:34.419550+0000) 2022-09-08T06:03:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:35.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:35.419896+0000) 2022-09-08T06:03:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:35.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:35.419800+0000) 2022-09-08T06:03:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:35 smithi059 ceph-mon[97450]: pgmap v16372: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:36.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:35 smithi055 ceph-mon[97033]: pgmap v16372: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:36.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:36.420002+0000) 2022-09-08T06:03:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:36.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:36.420052+0000) 2022-09-08T06:03:37.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:37.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:37.420264+0000) 2022-09-08T06:03:37.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:37.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:37.420237+0000) 2022-09-08T06:03:38.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:37 smithi059 ceph-mon[97450]: pgmap v16373: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:03:38.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:37 smithi055 ceph-mon[97033]: pgmap v16373: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:03:38.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:38.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:38.420461+0000) 2022-09-08T06:03:38.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:38.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:38.420464+0000) 2022-09-08T06:03:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:39.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:39.420730+0000) 2022-09-08T06:03:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:39.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:39.420738+0000) 2022-09-08T06:03:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:39 smithi059 ceph-mon[97450]: pgmap v16374: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:40.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:39 smithi055 ceph-mon[97033]: pgmap v16374: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:40.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:40.420970+0000) 2022-09-08T06:03:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:40.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:40.420933+0000) 2022-09-08T06:03:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:41.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:41.421177+0000) 2022-09-08T06:03:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:41.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:41.421124+0000) 2022-09-08T06:03:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:41 smithi059 ceph-mon[97450]: pgmap v16375: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:03:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:41 smithi055 ceph-mon[97033]: pgmap v16375: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:03:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:42.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:42.421307+0000) 2022-09-08T06:03:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:42.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:42.421281+0000) 2022-09-08T06:03:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:43.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:43.421454+0000) 2022-09-08T06:03:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:43.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:43.421542+0000) 2022-09-08T06:03:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:44 smithi059 ceph-mon[97450]: pgmap v16376: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:44 smithi055 ceph-mon[97033]: pgmap v16376: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:44.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:44.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:44.421804+0000) 2022-09-08T06:03:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:44.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:44.421730+0000) 2022-09-08T06:03:45.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:45 smithi055 ceph-mon[97033]: pgmap v16377: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:45.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:45 smithi059 ceph-mon[97450]: pgmap v16377: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:45.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:45.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:45.422034+0000) 2022-09-08T06:03:45.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:45.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:45.421902+0000) 2022-09-08T06:03:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:46.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:46.422231+0000) 2022-09-08T06:03:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:46.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:46.422151+0000) 2022-09-08T06:03:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:47 smithi055 ceph-mon[97033]: pgmap v16378: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:03:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:47 smithi059 ceph-mon[97450]: pgmap v16378: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:03:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:47.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:47.422380+0000) 2022-09-08T06:03:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:47.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:47.422306+0000) 2022-09-08T06:03:48.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:48.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:48.422517+0000) 2022-09-08T06:03:48.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:48.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:48.422485+0000) 2022-09-08T06:03:49.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:49 smithi055 ceph-mon[97033]: pgmap v16379: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:49 smithi059 ceph-mon[97450]: pgmap v16379: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:49.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:03:49.462+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:03:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:49.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:49.422815+0000) 2022-09-08T06:03:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:49.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:49.422753+0000) 2022-09-08T06:03:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:50.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:50.422951+0000) 2022-09-08T06:03:50.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:50.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:50.422938+0000) 2022-09-08T06:03:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:51 smithi055 ceph-mon[97033]: pgmap v16380: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:03:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:51 smithi059 ceph-mon[97450]: pgmap v16380: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:03:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:51.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:51.423171+0000) 2022-09-08T06:03:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:51.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:51.423151+0000) 2022-09-08T06:03:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:52.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:52.423418+0000) 2022-09-08T06:03:52.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:52.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:52.423389+0000) 2022-09-08T06:03:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:53 smithi055 ceph-mon[97033]: pgmap v16381: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:53 smithi059 ceph-mon[97450]: pgmap v16381: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:53.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:53.423668+0000) 2022-09-08T06:03:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:53.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:53.423623+0000) 2022-09-08T06:03:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:54.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:54.423909+0000) 2022-09-08T06:03:54.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:54.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:54.423892+0000) 2022-09-08T06:03:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:55 smithi055 ceph-mon[97033]: pgmap v16382: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:55 smithi059 ceph-mon[97450]: pgmap v16382: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:55.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:55.424073+0000) 2022-09-08T06:03:55.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:55.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:55.424099+0000) 2022-09-08T06:03:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:56.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:56.424299+0000) 2022-09-08T06:03:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:56.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:56.424300+0000) 2022-09-08T06:03:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:57 smithi055 ceph-mon[97033]: pgmap v16383: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:03:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:57 smithi059 ceph-mon[97450]: pgmap v16383: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:03:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:57.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:57.424487+0000) 2022-09-08T06:03:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:57.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:57.424464+0000) 2022-09-08T06:03:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:58.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:58.424756+0000) 2022-09-08T06:03:58.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:58.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:58.424750+0000) 2022-09-08T06:03:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:03:59 smithi055 ceph-mon[97033]: pgmap v16384: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:03:59 smithi059 ceph-mon[97450]: pgmap v16384: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:03:59.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:03:59.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:59.425020+0000) 2022-09-08T06:03:59.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:03:59.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:03:59.424980+0000) 2022-09-08T06:04:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:00.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:00.425318+0000) 2022-09-08T06:04:00.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:00.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:00.425294+0000) 2022-09-08T06:04:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:01 smithi055 ceph-mon[97033]: pgmap v16385: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:04:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:01 smithi059 ceph-mon[97450]: pgmap v16385: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:04:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:01.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:01.425547+0000) 2022-09-08T06:04:01.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:01.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:01.425546+0000) 2022-09-08T06:04:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:02.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:02.425799+0000) 2022-09-08T06:04:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:02.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:02.425805+0000) 2022-09-08T06:04:03.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:03 smithi055 ceph-mon[97033]: pgmap v16386: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:03 smithi059 ceph-mon[97450]: pgmap v16386: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:03.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:03.425958+0000) 2022-09-08T06:04:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:03.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:03.425981+0000) 2022-09-08T06:04:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:04.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:04.426160+0000) 2022-09-08T06:04:04.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:04.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:04.426193+0000) 2022-09-08T06:04:04.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:04:04.479+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:04:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:05 smithi055 ceph-mon[97033]: pgmap v16387: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:05.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:05 smithi059 ceph-mon[97450]: pgmap v16387: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:05.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:05.426378+0000) 2022-09-08T06:04:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:05.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:05.426406+0000) 2022-09-08T06:04:06.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:04:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:04:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:06.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:06.426579+0000) 2022-09-08T06:04:06.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:06.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:06.426588+0000) 2022-09-08T06:04:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:07 smithi055 ceph-mon[97033]: pgmap v16388: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:04:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:04:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:07 smithi059 ceph-mon[97450]: pgmap v16388: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:04:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:04:07.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:07.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:07.426801+0000) 2022-09-08T06:04:07.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:07.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:07.426806+0000) 2022-09-08T06:04:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:08.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:08.427027+0000) 2022-09-08T06:04:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:08.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:08.427022+0000) 2022-09-08T06:04:09.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:09 smithi055 ceph-mon[97033]: pgmap v16389: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:04:09.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:09 smithi059 ceph-mon[97450]: pgmap v16389: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:04:09.914 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:09.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:09.427195+0000) 2022-09-08T06:04:09.915 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:09.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:09.427175+0000) 2022-09-08T06:04:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:10 smithi055 ceph-mon[97033]: osdmap e1104: 12 total, 11 up, 11 in 2022-09-08T06:04:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:10 smithi059 ceph-mon[97450]: osdmap e1104: 12 total, 11 up, 11 in 2022-09-08T06:04:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:10.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:10.427366+0000) 2022-09-08T06:04:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:10.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:10.427342+0000) 2022-09-08T06:04:11.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:11 smithi055 ceph-mon[97033]: pgmap v16391: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:04:11.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:11 smithi055 ceph-mon[97033]: osdmap e1105: 12 total, 11 up, 11 in 2022-09-08T06:04:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:11 smithi059 ceph-mon[97450]: pgmap v16391: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:04:11.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:11 smithi059 ceph-mon[97450]: osdmap e1105: 12 total, 11 up, 11 in 2022-09-08T06:04:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:11.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:11.427508+0000) 2022-09-08T06:04:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:11.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:11.427509+0000) 2022-09-08T06:04:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:12.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:12.427776+0000) 2022-09-08T06:04:12.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:12.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:12.427778+0000) 2022-09-08T06:04:13.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:13 smithi055 ceph-mon[97033]: pgmap v16393: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T06:04:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:13 smithi059 ceph-mon[97450]: pgmap v16393: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T06:04:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:13.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:13.427959+0000) 2022-09-08T06:04:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:13.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:13.427959+0000) 2022-09-08T06:04:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:14.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:14.428179+0000) 2022-09-08T06:04:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:14.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:14.428179+0000) 2022-09-08T06:04:15.649 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:15 smithi059 ceph-mon[97450]: pgmap v16394: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:04:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:15 smithi055 ceph-mon[97033]: pgmap v16394: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:04:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:15.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:15.428388+0000) 2022-09-08T06:04:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:15.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:15.428388+0000) 2022-09-08T06:04:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:04:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:04:16.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:04:16.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:04:16.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:04:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:04:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:04:16.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:04:16.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:16.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:16.428695+0000) 2022-09-08T06:04:16.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:16.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:16.428698+0000) 2022-09-08T06:04:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:17 smithi055 ceph-mon[97033]: pgmap v16395: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:04:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:17 smithi059 ceph-mon[97450]: pgmap v16395: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:04:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:17.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:17.428930+0000) 2022-09-08T06:04:17.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:17.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:17.428926+0000) 2022-09-08T06:04:18.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:18.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:18.429111+0000) 2022-09-08T06:04:18.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:18.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:18.429114+0000) 2022-09-08T06:04:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:19 smithi055 ceph-mon[97033]: pgmap v16396: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T06:04:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:19 smithi059 ceph-mon[97450]: pgmap v16396: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T06:04:19.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:04:19.491+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:04:19.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:19.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:19.429283+0000) 2022-09-08T06:04:19.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:19.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:19.429283+0000) 2022-09-08T06:04:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:20.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:20.429466+0000) 2022-09-08T06:04:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:20.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:20.429468+0000) 2022-09-08T06:04:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:21 smithi055 ceph-mon[97033]: pgmap v16397: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:04:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:21 smithi059 ceph-mon[97450]: pgmap v16397: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:04:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:21.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:21.429622+0000) 2022-09-08T06:04:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:21.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:21.429622+0000) 2022-09-08T06:04:22.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:22.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:22.429862+0000) 2022-09-08T06:04:22.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:22.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:22.429866+0000) 2022-09-08T06:04:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:23 smithi055 ceph-mon[97033]: pgmap v16398: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T06:04:23.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:23 smithi059 ceph-mon[97450]: pgmap v16398: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T06:04:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:23.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:23.430001+0000) 2022-09-08T06:04:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:23.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:23.430001+0000) 2022-09-08T06:04:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:24.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:24.430156+0000) 2022-09-08T06:04:24.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:24.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:24.430156+0000) 2022-09-08T06:04:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:25 smithi055 ceph-mon[97033]: pgmap v16399: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:04:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:25 smithi059 ceph-mon[97450]: pgmap v16399: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:04:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:25.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:25.430316+0000) 2022-09-08T06:04:25.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:25.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:25.430315+0000) 2022-09-08T06:04:26.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:26.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:26.430446+0000) 2022-09-08T06:04:26.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:26.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:26.430456+0000) 2022-09-08T06:04:27.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:27 smithi055 ceph-mon[97033]: pgmap v16400: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:04:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:27 smithi059 ceph-mon[97450]: pgmap v16400: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:04:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:27.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:27.430583+0000) 2022-09-08T06:04:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:27.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:27.430577+0000) 2022-09-08T06:04:28.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:28.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:28.430739+0000) 2022-09-08T06:04:28.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:28.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:28.430715+0000) 2022-09-08T06:04:29.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:29 smithi055 ceph-mon[97033]: pgmap v16401: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:29 smithi059 ceph-mon[97450]: pgmap v16401: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:29.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:29.430879+0000) 2022-09-08T06:04:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:29.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:29.430842+0000) 2022-09-08T06:04:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:30.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:30.431020+0000) 2022-09-08T06:04:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:30.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:30.430996+0000) 2022-09-08T06:04:31.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:31 smithi055 ceph-mon[97033]: pgmap v16402: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:04:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:31 smithi059 ceph-mon[97450]: pgmap v16402: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:04:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:31.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:31.431159+0000) 2022-09-08T06:04:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:31.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:31.431122+0000) 2022-09-08T06:04:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:32.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:32.431373+0000) 2022-09-08T06:04:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:32.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:32.431380+0000) 2022-09-08T06:04:33.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:33 smithi055 ceph-mon[97033]: pgmap v16403: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:33 smithi059 ceph-mon[97450]: pgmap v16403: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:33.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:33.431558+0000) 2022-09-08T06:04:33.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:33.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:33.431558+0000) 2022-09-08T06:04:34.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:04:34.503+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:04:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:34.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:34.431807+0000) 2022-09-08T06:04:34.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:34.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:34.431785+0000) 2022-09-08T06:04:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:35 smithi055 ceph-mon[97033]: pgmap v16404: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:35 smithi059 ceph-mon[97450]: pgmap v16404: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:35.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:35.431980+0000) 2022-09-08T06:04:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:35.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:35.431926+0000) 2022-09-08T06:04:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:36.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:36.432386+0000) 2022-09-08T06:04:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:36.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:36.432191+0000) 2022-09-08T06:04:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:37 smithi055 ceph-mon[97033]: pgmap v16405: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:04:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:37 smithi059 ceph-mon[97450]: pgmap v16405: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:04:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:37.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:37.432570+0000) 2022-09-08T06:04:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:37.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:37.432414+0000) 2022-09-08T06:04:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:38.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:38.432725+0000) 2022-09-08T06:04:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:38.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:38.432654+0000) 2022-09-08T06:04:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:39 smithi055 ceph-mon[97033]: pgmap v16406: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:39 smithi059 ceph-mon[97450]: pgmap v16406: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:39.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:39.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:39.432858+0000) 2022-09-08T06:04:39.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:39.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:39.432770+0000) 2022-09-08T06:04:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:40.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:40.432965+0000) 2022-09-08T06:04:40.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:40.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:40.432997+0000) 2022-09-08T06:04:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:41 smithi055 ceph-mon[97033]: pgmap v16407: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:04:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:41 smithi059 ceph-mon[97450]: pgmap v16407: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:04:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:41.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:41.433177+0000) 2022-09-08T06:04:41.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:41.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:41.433186+0000) 2022-09-08T06:04:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:42.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:42.433394+0000) 2022-09-08T06:04:42.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:42.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:42.433378+0000) 2022-09-08T06:04:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:43 smithi055 ceph-mon[97033]: pgmap v16408: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:43.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:43 smithi059 ceph-mon[97450]: pgmap v16408: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:43.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:43.433586+0000) 2022-09-08T06:04:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:43.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:43.433629+0000) 2022-09-08T06:04:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:44.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:44.433917+0000) 2022-09-08T06:04:44.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:44.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:44.433786+0000) 2022-09-08T06:04:45.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:45 smithi055 ceph-mon[97033]: pgmap v16409: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:45 smithi059 ceph-mon[97450]: pgmap v16409: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:45.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:45.433965+0000) 2022-09-08T06:04:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:45.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:45.434159+0000) 2022-09-08T06:04:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:46.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:46.434391+0000) 2022-09-08T06:04:46.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:46.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:46.434138+0000) 2022-09-08T06:04:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:47 smithi055 ceph-mon[97033]: pgmap v16410: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:04:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:47 smithi059 ceph-mon[97450]: pgmap v16410: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:04:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:47.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:47.434617+0000) 2022-09-08T06:04:47.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:47.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:47.434354+0000) 2022-09-08T06:04:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:48.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:48.434807+0000) 2022-09-08T06:04:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:48.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:48.434561+0000) 2022-09-08T06:04:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:49 smithi055 ceph-mon[97033]: pgmap v16411: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:49 smithi059 ceph-mon[97450]: pgmap v16411: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:04:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:49.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:49.434965+0000) 2022-09-08T06:04:49.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:04:49.519+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:04:49.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:49.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:49.434765+0000) 2022-09-08T06:04:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:50.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:50.434916+0000) 2022-09-08T06:04:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:50.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:50.435120+0000) 2022-09-08T06:04:51.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:51 smithi055 ceph-mon[97033]: pgmap v16412: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:04:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:51 smithi059 ceph-mon[97450]: pgmap v16412: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:04:51.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:51.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:51.435052+0000) 2022-09-08T06:04:51.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:51.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:51.435280+0000) 2022-09-08T06:04:52.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:52.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:52.435188+0000) 2022-09-08T06:04:52.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:52.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:52.435421+0000) 2022-09-08T06:04:53.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:53 smithi055 ceph-mon[97033]: pgmap v16413: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:04:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:53 smithi059 ceph-mon[97450]: pgmap v16413: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:04:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:53.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:53.435392+0000) 2022-09-08T06:04:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:53.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:53.435670+0000) 2022-09-08T06:04:54.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:54.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:54.435539+0000) 2022-09-08T06:04:54.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:54.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:54.435817+0000) 2022-09-08T06:04:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:55 smithi055 ceph-mon[97033]: pgmap v16414: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:04:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:55 smithi059 ceph-mon[97450]: pgmap v16414: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:04:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:55.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:55.435735+0000) 2022-09-08T06:04:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:55.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:55.435951+0000) 2022-09-08T06:04:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:56.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:56.435955+0000) 2022-09-08T06:04:56.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:56.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:56.436109+0000) 2022-09-08T06:04:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:57 smithi055 ceph-mon[97033]: pgmap v16415: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:04:57.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:57 smithi059 ceph-mon[97450]: pgmap v16415: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:04:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:57.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:57.436302+0000) 2022-09-08T06:04:57.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:57.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:57.436180+0000) 2022-09-08T06:04:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:58.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:58.436482+0000) 2022-09-08T06:04:58.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:58.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:58.436378+0000) 2022-09-08T06:04:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:04:59 smithi055 ceph-mon[97033]: pgmap v16416: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:04:59.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:04:59 smithi059 ceph-mon[97450]: pgmap v16416: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:04:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:04:59.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:59.436550+0000) 2022-09-08T06:04:59.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:04:59.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:04:59.436652+0000) 2022-09-08T06:05:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:00.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:00.436775+0000) 2022-09-08T06:05:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:00.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:00.436697+0000) 2022-09-08T06:05:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:01 smithi055 ceph-mon[97033]: pgmap v16417: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:05:01.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:01 smithi059 ceph-mon[97450]: pgmap v16417: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:05:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:01.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:01.436885+0000) 2022-09-08T06:05:01.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:01.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:01.436973+0000) 2022-09-08T06:05:02.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:02.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:02.437146+0000) 2022-09-08T06:05:02.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:02.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:02.437205+0000) 2022-09-08T06:05:03.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:03 smithi055 ceph-mon[97033]: pgmap v16418: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:03 smithi059 ceph-mon[97450]: pgmap v16418: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:03.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:03.437360+0000) 2022-09-08T06:05:03.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:03.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:03.437466+0000) 2022-09-08T06:05:04.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:05:04.534+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:05:04.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:04.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:04.437689+0000) 2022-09-08T06:05:04.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:04.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:04.437756+0000) 2022-09-08T06:05:05.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:05 smithi055 ceph-mon[97033]: pgmap v16419: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:05 smithi059 ceph-mon[97450]: pgmap v16419: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:05.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:05.437846+0000) 2022-09-08T06:05:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:05.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:05.437944+0000) 2022-09-08T06:05:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:05:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:06.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:06.438045+0000) 2022-09-08T06:05:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:06.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:06.438190+0000) 2022-09-08T06:05:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:05:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:07 smithi059 ceph-mon[97450]: pgmap v16420: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:05:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:05:07.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:07.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:07.438236+0000) 2022-09-08T06:05:07.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:07.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:07.438352+0000) 2022-09-08T06:05:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:07 smithi055 ceph-mon[97033]: pgmap v16420: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:05:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:05:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:08.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:08.438438+0000) 2022-09-08T06:05:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:08.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:08.438562+0000) 2022-09-08T06:05:09.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:09 smithi059 ceph-mon[97450]: pgmap v16421: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:05:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:09.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:09.438727+0000) 2022-09-08T06:05:09.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:09.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:09.438791+0000) 2022-09-08T06:05:09.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:09 smithi055 ceph-mon[97033]: pgmap v16421: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:05:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:10 smithi059 ceph-mon[97450]: osdmap e1106: 12 total, 11 up, 11 in 2022-09-08T06:05:10.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:10.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:10.438978+0000) 2022-09-08T06:05:10.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:10.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:10.439038+0000) 2022-09-08T06:05:10.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:10 smithi055 ceph-mon[97033]: osdmap e1106: 12 total, 11 up, 11 in 2022-09-08T06:05:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:11 smithi059 ceph-mon[97450]: pgmap v16423: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:05:11.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:11.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:11.439180+0000) 2022-09-08T06:05:11.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:11.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:11.439204+0000) 2022-09-08T06:05:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:11 smithi055 ceph-mon[97033]: pgmap v16423: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:05:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:12.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:12.439409+0000) 2022-09-08T06:05:12.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:12.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:12.439395+0000) 2022-09-08T06:05:13.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:13 smithi059 ceph-mon[97450]: pgmap v16424: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:05:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:13.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:13.439615+0000) 2022-09-08T06:05:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:13.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:13.439626+0000) 2022-09-08T06:05:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:13 smithi055 ceph-mon[97033]: pgmap v16424: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:05:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:14.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:14.439900+0000) 2022-09-08T06:05:14.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:14.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:14.439908+0000) 2022-09-08T06:05:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:15 smithi059 ceph-mon[97450]: pgmap v16425: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:05:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:15.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:15.440089+0000) 2022-09-08T06:05:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:15.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:15.440113+0000) 2022-09-08T06:05:15.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:15 smithi055 ceph-mon[97033]: pgmap v16425: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:05:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:05:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:05:16.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:05:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:16.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:16.440274+0000) 2022-09-08T06:05:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:16.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:16.440280+0000) 2022-09-08T06:05:16.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:05:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:05:16.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:05:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:17.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:17.440491+0000) 2022-09-08T06:05:17.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:17.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:17.440494+0000) 2022-09-08T06:05:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:17 smithi055 ceph-mon[97033]: pgmap v16426: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:05:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:05:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:17 smithi059 ceph-mon[97450]: pgmap v16426: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:05:18.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:05:18.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:18.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:18.440713+0000) 2022-09-08T06:05:18.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:18.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:18.440724+0000) 2022-09-08T06:05:19.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:05:19.550+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:05:19.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:19.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:19.440880+0000) 2022-09-08T06:05:19.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:19.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:19.440950+0000) 2022-09-08T06:05:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:19 smithi055 ceph-mon[97033]: pgmap v16427: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:05:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:19 smithi059 ceph-mon[97450]: pgmap v16427: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:05:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:20.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:20.441121+0000) 2022-09-08T06:05:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:20.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:20.441094+0000) 2022-09-08T06:05:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:21.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:21.441305+0000) 2022-09-08T06:05:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:21.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:21.441325+0000) 2022-09-08T06:05:21.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:21 smithi055 ceph-mon[97033]: pgmap v16428: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 189 B/s wr, 8 op/s 2022-09-08T06:05:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:21 smithi055 ceph-mon[97033]: osdmap e1107: 12 total, 11 up, 11 in 2022-09-08T06:05:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:21 smithi059 ceph-mon[97450]: pgmap v16428: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 189 B/s wr, 8 op/s 2022-09-08T06:05:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:21 smithi059 ceph-mon[97450]: osdmap e1107: 12 total, 11 up, 11 in 2022-09-08T06:05:22.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:22.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:22.441551+0000) 2022-09-08T06:05:22.912 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:22.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:22.441541+0000) 2022-09-08T06:05:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:23.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:23.441732+0000) 2022-09-08T06:05:23.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:23.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:23.441731+0000) 2022-09-08T06:05:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:23 smithi055 ceph-mon[97033]: pgmap v16430: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:05:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:23 smithi059 ceph-mon[97450]: pgmap v16430: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:05:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:24.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:24.441925+0000) 2022-09-08T06:05:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:24.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:24.441940+0000) 2022-09-08T06:05:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:25.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:25.442157+0000) 2022-09-08T06:05:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:25.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:25.442151+0000) 2022-09-08T06:05:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:25 smithi055 ceph-mon[97033]: pgmap v16431: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:05:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:25 smithi059 ceph-mon[97450]: pgmap v16431: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:05:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:26.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:26.442406+0000) 2022-09-08T06:05:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:26.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:26.442354+0000) 2022-09-08T06:05:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:27.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:27.442558+0000) 2022-09-08T06:05:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:27.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:27.442582+0000) 2022-09-08T06:05:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:27 smithi055 ceph-mon[97033]: pgmap v16432: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:05:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:27 smithi059 ceph-mon[97450]: pgmap v16432: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:05:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:28.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:28.442791+0000) 2022-09-08T06:05:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:28.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:28.442785+0000) 2022-09-08T06:05:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:29.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:29.443058+0000) 2022-09-08T06:05:29.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:29.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:29.443081+0000) 2022-09-08T06:05:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:29 smithi055 ceph-mon[97033]: pgmap v16433: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:05:30.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:29 smithi059 ceph-mon[97450]: pgmap v16433: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:05:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:30.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:30.443351+0000) 2022-09-08T06:05:30.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:30.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:30.443316+0000) 2022-09-08T06:05:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:31.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:31.443608+0000) 2022-09-08T06:05:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:31.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:31.443569+0000) 2022-09-08T06:05:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:31 smithi055 ceph-mon[97033]: pgmap v16434: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:05:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:31 smithi059 ceph-mon[97450]: pgmap v16434: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:05:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:32.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:32.443821+0000) 2022-09-08T06:05:32.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:32.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:32.443893+0000) 2022-09-08T06:05:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:33.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:33.444062+0000) 2022-09-08T06:05:33.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:33.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:33.444127+0000) 2022-09-08T06:05:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:33 smithi055 ceph-mon[97033]: pgmap v16435: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:05:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:33 smithi059 ceph-mon[97450]: pgmap v16435: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:05:34.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:05:34.561+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:05:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:34.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:34.444308+0000) 2022-09-08T06:05:34.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:34.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:34.444341+0000) 2022-09-08T06:05:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:35.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:35.444510+0000) 2022-09-08T06:05:35.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:35.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:35.444490+0000) 2022-09-08T06:05:35.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:35 smithi055 ceph-mon[97033]: pgmap v16436: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:35 smithi059 ceph-mon[97450]: pgmap v16436: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:36.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:36.444743+0000) 2022-09-08T06:05:36.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:36.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:36.444907+0000) 2022-09-08T06:05:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:37.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:37.444935+0000) 2022-09-08T06:05:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:37.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:37.445055+0000) 2022-09-08T06:05:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:37 smithi055 ceph-mon[97033]: pgmap v16437: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:05:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:37 smithi059 ceph-mon[97450]: pgmap v16437: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:05:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:38.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:38.445250+0000) 2022-09-08T06:05:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:38.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:38.445147+0000) 2022-09-08T06:05:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:39.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:39.445361+0000) 2022-09-08T06:05:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:39.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:39.445422+0000) 2022-09-08T06:05:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:39 smithi055 ceph-mon[97033]: pgmap v16438: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:39 smithi059 ceph-mon[97450]: pgmap v16438: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:40.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:40.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:40.445596+0000) 2022-09-08T06:05:40.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:40.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:40.445708+0000) 2022-09-08T06:05:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:41.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:41.445968+0000) 2022-09-08T06:05:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:41.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:41.445941+0000) 2022-09-08T06:05:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:41 smithi055 ceph-mon[97033]: pgmap v16439: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:05:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:41 smithi059 ceph-mon[97450]: pgmap v16439: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:05:42.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:42.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:42.446151+0000) 2022-09-08T06:05:42.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:42.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:42.446147+0000) 2022-09-08T06:05:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:43.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:43.446388+0000) 2022-09-08T06:05:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:43.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:43.446386+0000) 2022-09-08T06:05:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:43 smithi055 ceph-mon[97033]: pgmap v16440: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:43 smithi059 ceph-mon[97450]: pgmap v16440: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:44.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:44.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:44.446642+0000) 2022-09-08T06:05:44.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:44.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:44.446669+0000) 2022-09-08T06:05:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:45.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:45.446811+0000) 2022-09-08T06:05:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:45.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:45.446908+0000) 2022-09-08T06:05:46.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:45 smithi059 ceph-mon[97450]: pgmap v16441: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:46.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:45 smithi055 ceph-mon[97033]: pgmap v16441: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:46.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:46.447005+0000) 2022-09-08T06:05:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:46.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:46.447108+0000) 2022-09-08T06:05:47.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:47.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:47.447181+0000) 2022-09-08T06:05:47.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:47.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:47.447273+0000) 2022-09-08T06:05:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:47 smithi059 ceph-mon[97450]: pgmap v16442: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:05:48.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:47 smithi055 ceph-mon[97033]: pgmap v16442: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:05:48.913 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:48.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:48.447324+0000) 2022-09-08T06:05:48.914 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:48.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:48.447416+0000) 2022-09-08T06:05:49.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:05:49.568+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:05:49.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:49.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:49.447486+0000) 2022-09-08T06:05:49.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:49.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:49.447590+0000) 2022-09-08T06:05:50.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:49 smithi059 ceph-mon[97450]: pgmap v16443: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:50.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:49 smithi055 ceph-mon[97033]: pgmap v16443: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:50.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:50.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:50.447681+0000) 2022-09-08T06:05:50.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:50.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:50.447765+0000) 2022-09-08T06:05:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:51.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:51.447918+0000) 2022-09-08T06:05:51.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:51.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:51.448006+0000) 2022-09-08T06:05:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:51 smithi059 ceph-mon[97450]: pgmap v16444: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:05:52.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:51 smithi055 ceph-mon[97033]: pgmap v16444: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:05:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:52.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:52.448174+0000) 2022-09-08T06:05:52.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:52.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:52.448234+0000) 2022-09-08T06:05:53.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:53.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:53.448438+0000) 2022-09-08T06:05:53.909 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:53.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:53.448400+0000) 2022-09-08T06:05:54.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:53 smithi059 ceph-mon[97450]: pgmap v16445: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:53 smithi055 ceph-mon[97033]: pgmap v16445: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:54.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:54.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:54.448630+0000) 2022-09-08T06:05:54.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:54.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:54.448666+0000) 2022-09-08T06:05:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:55.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:55.448886+0000) 2022-09-08T06:05:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:55.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:55.448906+0000) 2022-09-08T06:05:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:55 smithi059 ceph-mon[97450]: pgmap v16446: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:55 smithi055 ceph-mon[97033]: pgmap v16446: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:05:56.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:56.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:56.449135+0000) 2022-09-08T06:05:56.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:56.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:56.449129+0000) 2022-09-08T06:05:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:57.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:57.449384+0000) 2022-09-08T06:05:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:57.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:57.449410+0000) 2022-09-08T06:05:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:57 smithi059 ceph-mon[97450]: pgmap v16447: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:05:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:57 smithi055 ceph-mon[97033]: pgmap v16447: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:05:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:58.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:58.449654+0000) 2022-09-08T06:05:58.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:58.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:58.449649+0000) 2022-09-08T06:05:59.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:05:59.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:59.449904+0000) 2022-09-08T06:05:59.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:05:59.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:05:59.449927+0000) 2022-09-08T06:06:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:05:59 smithi059 ceph-mon[97450]: pgmap v16448: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:06:00.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:05:59 smithi055 ceph-mon[97033]: pgmap v16448: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:06:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:00.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:00.450074+0000) 2022-09-08T06:06:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:00.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:00.450065+0000) 2022-09-08T06:06:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:01.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:01.450303+0000) 2022-09-08T06:06:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:01.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:01.450328+0000) 2022-09-08T06:06:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:01 smithi059 ceph-mon[97450]: pgmap v16449: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:06:02.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:01 smithi055 ceph-mon[97033]: pgmap v16449: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:06:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:02.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:02.450551+0000) 2022-09-08T06:06:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:02.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:02.450570+0000) 2022-09-08T06:06:03.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:03.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:03.450787+0000) 2022-09-08T06:06:03.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:03.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:03.450824+0000) 2022-09-08T06:06:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:03 smithi059 ceph-mon[97450]: pgmap v16450: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:06:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:03 smithi055 ceph-mon[97033]: pgmap v16450: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:06:04.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:06:04.581+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:06:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:04.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:04.451087+0000) 2022-09-08T06:06:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:04.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:04.451104+0000) 2022-09-08T06:06:05.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:05.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:05.451377+0000) 2022-09-08T06:06:05.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:05.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:05.451348+0000) 2022-09-08T06:06:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:05 smithi059 ceph-mon[97450]: pgmap v16451: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:06:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:05 smithi055 ceph-mon[97033]: pgmap v16451: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:06:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:06.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:06.451566+0000) 2022-09-08T06:06:06.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:06.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:06.451545+0000) 2022-09-08T06:06:07.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:06:07.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:06:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:06:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:06:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:07.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:07.451839+0000) 2022-09-08T06:06:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:07.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:07.451799+0000) 2022-09-08T06:06:08.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:07 smithi059 ceph-mon[97450]: pgmap v16452: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:06:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:07 smithi055 ceph-mon[97033]: pgmap v16452: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:06:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:08.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:08.452051+0000) 2022-09-08T06:06:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:08.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:08.452055+0000) 2022-09-08T06:06:09.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:09.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:09.452228+0000) 2022-09-08T06:06:09.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:09.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:09.452247+0000) 2022-09-08T06:06:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:09 smithi055 ceph-mon[97033]: pgmap v16453: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:06:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:09 smithi059 ceph-mon[97450]: pgmap v16453: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:06:10.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:10.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:10.452483+0000) 2022-09-08T06:06:10.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:10.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:10.452533+0000) 2022-09-08T06:06:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:10 smithi055 ceph-mon[97033]: osdmap e1108: 12 total, 11 up, 11 in 2022-09-08T06:06:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:10 smithi059 ceph-mon[97450]: osdmap e1108: 12 total, 11 up, 11 in 2022-09-08T06:06:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:11.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:11.452755+0000) 2022-09-08T06:06:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:11.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:11.452768+0000) 2022-09-08T06:06:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:11 smithi055 ceph-mon[97033]: pgmap v16455: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:06:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:11 smithi059 ceph-mon[97450]: pgmap v16455: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:06:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:12.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:12.452898+0000) 2022-09-08T06:06:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:12.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:12.452914+0000) 2022-09-08T06:06:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:13.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:13.453139+0000) 2022-09-08T06:06:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:13.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:13.453154+0000) 2022-09-08T06:06:14.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:13 smithi055 ceph-mon[97033]: pgmap v16456: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:06:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:13 smithi059 ceph-mon[97450]: pgmap v16456: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:06:14.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:14.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:14.453330+0000) 2022-09-08T06:06:14.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:14.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:14.453389+0000) 2022-09-08T06:06:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:15.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:15.453556+0000) 2022-09-08T06:06:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:15.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:15.453584+0000) 2022-09-08T06:06:16.239 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:15 smithi055 ceph-mon[97033]: pgmap v16457: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:06:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:15 smithi059 ceph-mon[97450]: pgmap v16457: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:06:16.815 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:16.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:16.453735+0000) 2022-09-08T06:06:16.816 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:16.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:16.453821+0000) 2022-09-08T06:06:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:17 smithi055 ceph-mon[97033]: pgmap v16458: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:06:17.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:06:17.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:06:17.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:06:17.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:17 smithi059 ceph-mon[97450]: pgmap v16458: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:06:17.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:06:17.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:06:17.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:06:17.913 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:17.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:17.454008+0000) 2022-09-08T06:06:17.915 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:17.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:17.454093+0000) 2022-09-08T06:06:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:18.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:18.454241+0000) 2022-09-08T06:06:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:18.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:18.454322+0000) 2022-09-08T06:06:19.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:19 smithi055 ceph-mon[97033]: pgmap v16459: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:06:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:19 smithi059 ceph-mon[97450]: pgmap v16459: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:06:19.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:06:19.598+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:06:19.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:19.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:19.454456+0000) 2022-09-08T06:06:19.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:19.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:19.454488+0000) 2022-09-08T06:06:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:20.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:20.454731+0000) 2022-09-08T06:06:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:20.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:20.454758+0000) 2022-09-08T06:06:21.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:21 smithi055 ceph-mon[97033]: pgmap v16460: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.5 KiB/s rd, 194 B/s wr, 8 op/s 2022-09-08T06:06:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:21 smithi059 ceph-mon[97450]: pgmap v16460: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.5 KiB/s rd, 194 B/s wr, 8 op/s 2022-09-08T06:06:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:21.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:21.454929+0000) 2022-09-08T06:06:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:21.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:21.454927+0000) 2022-09-08T06:06:22.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:22 smithi055 ceph-mon[97033]: osdmap e1109: 12 total, 11 up, 11 in 2022-09-08T06:06:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:22 smithi059 ceph-mon[97450]: osdmap e1109: 12 total, 11 up, 11 in 2022-09-08T06:06:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:22.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:22.455088+0000) 2022-09-08T06:06:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:22.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:22.455108+0000) 2022-09-08T06:06:23.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:23 smithi055 ceph-mon[97033]: pgmap v16462: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:06:23.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:23 smithi059 ceph-mon[97450]: pgmap v16462: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:06:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:23.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:23.455285+0000) 2022-09-08T06:06:23.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:23.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:23.455291+0000) 2022-09-08T06:06:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:24.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:24.455481+0000) 2022-09-08T06:06:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:24.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:24.455501+0000) 2022-09-08T06:06:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:25 smithi055 ceph-mon[97033]: pgmap v16463: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:06:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:25 smithi059 ceph-mon[97450]: pgmap v16463: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:06:25.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:25.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:25.455657+0000) 2022-09-08T06:06:25.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:25.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:25.455684+0000) 2022-09-08T06:06:26.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:26.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:26.455793+0000) 2022-09-08T06:06:26.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:26.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:26.455857+0000) 2022-09-08T06:06:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:27 smithi055 ceph-mon[97033]: pgmap v16464: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:06:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:27 smithi059 ceph-mon[97450]: pgmap v16464: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:06:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:27.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:27.456060+0000) 2022-09-08T06:06:27.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:27.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:27.456031+0000) 2022-09-08T06:06:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:28.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:28.456220+0000) 2022-09-08T06:06:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:28.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:28.456249+0000) 2022-09-08T06:06:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:29 smithi055 ceph-mon[97033]: pgmap v16465: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:06:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:29 smithi059 ceph-mon[97450]: pgmap v16465: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:06:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:29.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:29.456423+0000) 2022-09-08T06:06:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:29.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:29.456442+0000) 2022-09-08T06:06:30.915 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:30.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:30.456721+0000) 2022-09-08T06:06:30.916 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:30.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:30.456593+0000) 2022-09-08T06:06:31.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:31 smithi055 ceph-mon[97033]: pgmap v16466: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:06:31.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:31 smithi059 ceph-mon[97450]: pgmap v16466: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:06:31.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:31.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:31.456898+0000) 2022-09-08T06:06:31.909 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:31.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:31.456839+0000) 2022-09-08T06:06:32.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:32.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:32.457079+0000) 2022-09-08T06:06:32.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:32.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:32.457119+0000) 2022-09-08T06:06:33.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:33 smithi055 ceph-mon[97033]: pgmap v16467: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:06:33.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:33 smithi059 ceph-mon[97450]: pgmap v16467: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:06:33.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:33.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:33.457345+0000) 2022-09-08T06:06:33.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:33.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:33.457336+0000) 2022-09-08T06:06:34.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:34.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:34.457527+0000) 2022-09-08T06:06:34.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:34.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:34.457553+0000) 2022-09-08T06:06:34.908 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:06:34.613+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:06:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:35 smithi055 ceph-mon[97033]: pgmap v16468: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:06:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:35 smithi059 ceph-mon[97450]: pgmap v16468: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:06:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:35.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:35.457753+0000) 2022-09-08T06:06:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:35.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:35.457763+0000) 2022-09-08T06:06:36.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:36.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:36.457957+0000) 2022-09-08T06:06:36.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:36.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:36.457998+0000) 2022-09-08T06:06:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:37 smithi055 ceph-mon[97033]: pgmap v16469: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:06:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:37 smithi059 ceph-mon[97450]: pgmap v16469: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:06:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:37.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:37.458241+0000) 2022-09-08T06:06:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:37.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:37.458241+0000) 2022-09-08T06:06:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:38.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:38.458429+0000) 2022-09-08T06:06:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:38.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:38.458411+0000) 2022-09-08T06:06:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:39 smithi055 ceph-mon[97033]: pgmap v16470: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:06:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:39 smithi059 ceph-mon[97450]: pgmap v16470: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:06:39.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:39.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:39.458589+0000) 2022-09-08T06:06:39.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:39.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:39.458567+0000) 2022-09-08T06:06:40.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:40.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:40.458815+0000) 2022-09-08T06:06:40.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:40.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:40.458788+0000) 2022-09-08T06:06:41.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:41 smithi055 ceph-mon[97033]: pgmap v16471: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:06:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:41 smithi059 ceph-mon[97450]: pgmap v16471: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:06:41.889 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:41.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:41.459041+0000) 2022-09-08T06:06:41.890 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:41.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:41.459026+0000) 2022-09-08T06:06:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:42.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:42.459265+0000) 2022-09-08T06:06:42.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:42.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:42.459233+0000) 2022-09-08T06:06:43.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:43 smithi055 ceph-mon[97033]: pgmap v16472: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:06:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:43 smithi059 ceph-mon[97450]: pgmap v16472: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:06:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:43.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:43.459367+0000) 2022-09-08T06:06:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:43.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:43.459422+0000) 2022-09-08T06:06:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:44.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:44.459602+0000) 2022-09-08T06:06:44.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:44.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:44.459678+0000) 2022-09-08T06:06:45.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:45 smithi055 ceph-mon[97033]: pgmap v16473: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:06:45.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:45 smithi059 ceph-mon[97450]: pgmap v16473: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:06:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:45.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:45.459933+0000) 2022-09-08T06:06:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:45.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:45.459870+0000) 2022-09-08T06:06:46.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:46.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:46.460093+0000) 2022-09-08T06:06:46.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:46.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:46.460142+0000) 2022-09-08T06:06:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:47.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:47.460339+0000) 2022-09-08T06:06:47.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:47.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:47.460312+0000) 2022-09-08T06:06:48.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:47 smithi055 ceph-mon[97033]: pgmap v16474: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:06:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:47 smithi055 ceph-mon[97033]: pgmap v16475: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:06:48.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:06:48.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:47 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T06:06:48.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:47 smithi059 ceph-mon[97450]: pgmap v16474: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:06:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:47 smithi059 ceph-mon[97450]: pgmap v16475: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:06:48.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:06:48.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:47 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T06:06:48.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:48.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:48.460535+0000) 2022-09-08T06:06:48.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:48.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:48.460554+0000) 2022-09-08T06:06:49.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:06:49.624+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:06:49.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:49.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:49.460744+0000) 2022-09-08T06:06:49.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:49.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:49.460761+0000) 2022-09-08T06:06:50.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:49 smithi055 ceph-mon[97033]: pgmap v16476: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:06:50.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:49 smithi059 ceph-mon[97450]: pgmap v16476: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:06:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:50.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:50.460968+0000) 2022-09-08T06:06:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:50.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:50.460982+0000) 2022-09-08T06:06:51.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:51.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:51.461219+0000) 2022-09-08T06:06:51.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:51.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:51.461212+0000) 2022-09-08T06:06:52.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:51 smithi055 ceph-mon[97033]: pgmap v16477: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:06:52.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:51 smithi059 ceph-mon[97450]: pgmap v16477: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:06:52.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:52.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:52.461373+0000) 2022-09-08T06:06:52.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:52.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:52.461405+0000) 2022-09-08T06:06:53.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:53.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:53.461615+0000) 2022-09-08T06:06:53.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:53.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:53.461667+0000) 2022-09-08T06:06:54.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:53 smithi055 ceph-mon[97033]: pgmap v16478: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:06:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:53 smithi059 ceph-mon[97450]: pgmap v16478: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:06:54.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:54.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:54.461943+0000) 2022-09-08T06:06:54.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:54.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:54.461868+0000) 2022-09-08T06:06:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:55.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:55.462215+0000) 2022-09-08T06:06:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:55.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:55.462122+0000) 2022-09-08T06:06:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:55 smithi055 ceph-mon[97033]: pgmap v16479: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T06:06:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:55 smithi059 ceph-mon[97450]: pgmap v16479: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T06:06:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:56.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:56.462436+0000) 2022-09-08T06:06:56.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:56.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:56.462335+0000) 2022-09-08T06:06:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:06:57 smithi055 ceph-mon[97033]: pgmap v16480: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:06:57.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:06:57 smithi059 ceph-mon[97450]: pgmap v16480: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:06:57.920 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:57.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:57.462608+0000) 2022-09-08T06:06:57.922 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:57.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:57.462562+0000) 2022-09-08T06:06:58.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:58.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:58.462736+0000) 2022-09-08T06:06:58.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:58.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:58.462718+0000) 2022-09-08T06:06:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:06:59.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:59.462919+0000) 2022-09-08T06:06:59.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:06:59.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:06:59.462941+0000) 2022-09-08T06:07:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:00 smithi059 ceph-mon[97450]: pgmap v16481: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:00 smithi055 ceph-mon[97033]: pgmap v16481: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:00.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:00.463139+0000) 2022-09-08T06:07:00.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:00.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:00.463158+0000) 2022-09-08T06:07:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:01.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:01.463351+0000) 2022-09-08T06:07:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:01.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:01.463374+0000) 2022-09-08T06:07:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:02 smithi059 ceph-mon[97450]: pgmap v16482: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:07:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:02 smithi055 ceph-mon[97033]: pgmap v16482: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:07:02.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:02.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:02.463569+0000) 2022-09-08T06:07:02.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:02.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:02.463582+0000) 2022-09-08T06:07:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:03.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:03.463801+0000) 2022-09-08T06:07:03.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:03.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:03.463809+0000) 2022-09-08T06:07:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:04 smithi059 ceph-mon[97450]: pgmap v16483: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:04 smithi055 ceph-mon[97033]: pgmap v16483: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:04.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:07:04.631+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:07:04.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:04.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:04.464076+0000) 2022-09-08T06:07:04.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:04.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:04.464074+0000) 2022-09-08T06:07:05.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:05.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:05.464329+0000) 2022-09-08T06:07:05.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:05.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:05.464344+0000) 2022-09-08T06:07:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:06 smithi059 ceph-mon[97450]: pgmap v16484: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:07:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:07:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:06 smithi055 ceph-mon[97033]: pgmap v16484: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:07:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:07:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:06.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:06.464590+0000) 2022-09-08T06:07:06.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:06.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:06.464599+0000) 2022-09-08T06:07:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:07:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:07 smithi055 ceph-mon[97033]: pgmap v16485: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:07:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:07:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:07 smithi059 ceph-mon[97450]: pgmap v16485: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:07:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:07.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:07.464882+0000) 2022-09-08T06:07:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:07.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:07.464868+0000) 2022-09-08T06:07:08.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:08.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:08.465135+0000) 2022-09-08T06:07:08.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:08.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:08.465135+0000) 2022-09-08T06:07:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:09.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:09.465323+0000) 2022-09-08T06:07:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:09.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:09.465333+0000) 2022-09-08T06:07:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:09 smithi055 ceph-mon[97033]: osdmap e1110: 12 total, 11 up, 11 in 2022-09-08T06:07:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:09 smithi055 ceph-mon[97033]: pgmap v16487: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:07:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:09 smithi059 ceph-mon[97450]: osdmap e1110: 12 total, 11 up, 11 in 2022-09-08T06:07:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:09 smithi059 ceph-mon[97450]: pgmap v16487: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:07:10.919 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:10.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:10.465571+0000) 2022-09-08T06:07:10.921 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:10.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:10.465600+0000) 2022-09-08T06:07:11.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:11.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:11.465828+0000) 2022-09-08T06:07:11.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:11.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:11.465909+0000) 2022-09-08T06:07:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:12 smithi059 ceph-mon[97450]: pgmap v16488: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:07:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:12 smithi055 ceph-mon[97033]: pgmap v16488: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:07:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:12.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:12.466085+0000) 2022-09-08T06:07:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:12.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:12.466203+0000) 2022-09-08T06:07:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:13.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:13.466340+0000) 2022-09-08T06:07:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:13.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:13.466376+0000) 2022-09-08T06:07:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:14 smithi059 ceph-mon[97450]: pgmap v16489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:07:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:14 smithi055 ceph-mon[97033]: pgmap v16489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:07:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:14.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:14.466539+0000) 2022-09-08T06:07:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:14.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:14.466550+0000) 2022-09-08T06:07:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:15.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:15.466777+0000) 2022-09-08T06:07:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:15.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:15.466777+0000) 2022-09-08T06:07:16.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:16 smithi059 ceph-mon[97450]: pgmap v16490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:07:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:16 smithi055 ceph-mon[97033]: pgmap v16490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:07:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:16.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:16.466989+0000) 2022-09-08T06:07:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:16.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:16.467013+0000) 2022-09-08T06:07:17.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:17 smithi055 ceph-mon[97033]: osdmap e1111: 12 total, 11 up, 11 in 2022-09-08T06:07:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:17 smithi055 ceph-mon[97033]: pgmap v16492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-08T06:07:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:17 smithi059 ceph-mon[97450]: osdmap e1111: 12 total, 11 up, 11 in 2022-09-08T06:07:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:17 smithi059 ceph-mon[97450]: pgmap v16492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 254 B/s wr, 7 op/s 2022-09-08T06:07:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:17.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:17.467218+0000) 2022-09-08T06:07:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:17.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:17.467215+0000) 2022-09-08T06:07:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:18.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:18.467441+0000) 2022-09-08T06:07:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:18.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:18.467429+0000) 2022-09-08T06:07:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:19.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:19.467704+0000) 2022-09-08T06:07:19.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:07:19.646+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:07:19.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:19.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:19.467707+0000) 2022-09-08T06:07:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:20 smithi059 ceph-mon[97450]: pgmap v16493: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:07:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:20 smithi055 ceph-mon[97033]: pgmap v16493: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:07:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:20.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:20.467947+0000) 2022-09-08T06:07:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:20.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:20.467947+0000) 2022-09-08T06:07:21.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:21.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:21.468146+0000) 2022-09-08T06:07:21.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:21.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:21.468148+0000) 2022-09-08T06:07:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:22 smithi059 ceph-mon[97450]: pgmap v16494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:07:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:22 smithi055 ceph-mon[97033]: pgmap v16494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:07:22.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:22.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:22.468338+0000) 2022-09-08T06:07:22.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:22.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:22.468333+0000) 2022-09-08T06:07:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:23.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:23.468541+0000) 2022-09-08T06:07:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:23.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:23.468565+0000) 2022-09-08T06:07:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:24 smithi059 ceph-mon[97450]: pgmap v16495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:07:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:24 smithi055 ceph-mon[97033]: pgmap v16495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:07:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:24.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:24.468822+0000) 2022-09-08T06:07:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:24.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:24.468740+0000) 2022-09-08T06:07:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:25.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:25.468986+0000) 2022-09-08T06:07:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:25.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:25.468907+0000) 2022-09-08T06:07:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:26 smithi059 ceph-mon[97450]: pgmap v16496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:07:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:26 smithi055 ceph-mon[97033]: pgmap v16496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:07:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:26.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:26.469158+0000) 2022-09-08T06:07:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:26.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:26.469108+0000) 2022-09-08T06:07:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:27 smithi055 ceph-mon[97033]: pgmap v16497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:07:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:27 smithi059 ceph-mon[97450]: pgmap v16497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:07:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:27.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:27.469331+0000) 2022-09-08T06:07:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:27.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:27.469330+0000) 2022-09-08T06:07:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:28.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:28.469562+0000) 2022-09-08T06:07:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:28.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:28.469567+0000) 2022-09-08T06:07:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:29.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:29.469828+0000) 2022-09-08T06:07:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:29.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:29.469831+0000) 2022-09-08T06:07:30.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:30 smithi059 ceph-mon[97450]: pgmap v16498: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:30 smithi055 ceph-mon[97033]: pgmap v16498: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:30.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:30.470114+0000) 2022-09-08T06:07:30.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:30.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:30.470126+0000) 2022-09-08T06:07:31.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:31.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:31.470362+0000) 2022-09-08T06:07:31.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:31.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:31.470342+0000) 2022-09-08T06:07:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:32 smithi059 ceph-mon[97450]: pgmap v16499: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:07:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:32 smithi055 ceph-mon[97033]: pgmap v16499: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:07:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:32.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:32.470579+0000) 2022-09-08T06:07:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:32.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:32.470554+0000) 2022-09-08T06:07:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:33.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:33.470802+0000) 2022-09-08T06:07:33.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:33.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:33.470778+0000) 2022-09-08T06:07:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:34 smithi059 ceph-mon[97450]: pgmap v16500: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:34.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:34 smithi055 ceph-mon[97033]: pgmap v16500: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:34.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:07:34.661+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:07:34.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:34.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:34.471038+0000) 2022-09-08T06:07:34.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:34.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:34.471059+0000) 2022-09-08T06:07:35.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:35.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:35.471204+0000) 2022-09-08T06:07:35.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:35.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:35.471247+0000) 2022-09-08T06:07:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:36 smithi059 ceph-mon[97450]: pgmap v16501: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:07:36.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:36 smithi055 ceph-mon[97033]: pgmap v16501: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:07:36.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:36.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:36.471485+0000) 2022-09-08T06:07:36.930 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:36.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:36.471434+0000) 2022-09-08T06:07:37.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:37 smithi055 ceph-mon[97033]: pgmap v16502: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:37 smithi059 ceph-mon[97450]: pgmap v16502: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:37.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:37.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:37.471757+0000) 2022-09-08T06:07:37.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:37.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:37.471708+0000) 2022-09-08T06:07:38.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:38.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:38.471913+0000) 2022-09-08T06:07:38.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:38.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:38.471978+0000) 2022-09-08T06:07:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:39.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:39.472124+0000) 2022-09-08T06:07:39.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:39.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:39.472161+0000) 2022-09-08T06:07:40.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:40 smithi059 ceph-mon[97450]: pgmap v16503: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:40 smithi055 ceph-mon[97033]: pgmap v16503: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:40.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:40.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:40.472371+0000) 2022-09-08T06:07:40.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:40.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:40.472481+0000) 2022-09-08T06:07:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:41.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:41.472589+0000) 2022-09-08T06:07:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:41.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:41.472704+0000) 2022-09-08T06:07:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:42 smithi059 ceph-mon[97450]: pgmap v16504: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:07:42.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:42 smithi055 ceph-mon[97033]: pgmap v16504: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:07:42.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:42.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:42.472740+0000) 2022-09-08T06:07:42.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:42.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:42.472858+0000) 2022-09-08T06:07:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:43.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:43.473001+0000) 2022-09-08T06:07:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:43.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:43.472918+0000) 2022-09-08T06:07:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:44 smithi059 ceph-mon[97450]: pgmap v16505: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:44.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:44 smithi055 ceph-mon[97033]: pgmap v16505: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:44.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:44.473172+0000) 2022-09-08T06:07:44.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:44.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:44.473152+0000) 2022-09-08T06:07:45.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:45.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:45.473373+0000) 2022-09-08T06:07:45.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:45.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:45.473354+0000) 2022-09-08T06:07:46.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:46 smithi059 ceph-mon[97450]: pgmap v16506: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:07:46.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:46 smithi055 ceph-mon[97033]: pgmap v16506: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:07:46.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:46.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:46.473629+0000) 2022-09-08T06:07:46.911 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:46.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:46.473592+0000) 2022-09-08T06:07:47.364 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:47 smithi059 ceph-mon[97450]: pgmap v16507: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:47.365 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:07:47.365 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:07:47.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:47 smithi055 ceph-mon[97033]: pgmap v16507: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:47.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:07:47.487 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:07:47.487 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:07:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:07:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:47.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:47.473820+0000) 2022-09-08T06:07:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:47.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:47.473862+0000) 2022-09-08T06:07:48.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:48.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:48.474050+0000) 2022-09-08T06:07:48.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:48.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:48.474087+0000) 2022-09-08T06:07:49.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:07:49.670+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:07:49.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:49.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:49.474282+0000) 2022-09-08T06:07:49.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:49.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:49.474344+0000) 2022-09-08T06:07:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:50 smithi059 ceph-mon[97450]: pgmap v16508: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:50 smithi055 ceph-mon[97033]: pgmap v16508: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:50.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:50.474522+0000) 2022-09-08T06:07:50.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:50.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:50.474671+0000) 2022-09-08T06:07:51.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:51.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:51.474750+0000) 2022-09-08T06:07:51.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:51.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:51.474879+0000) 2022-09-08T06:07:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:52 smithi059 ceph-mon[97450]: pgmap v16509: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:07:52.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:52 smithi055 ceph-mon[97033]: pgmap v16509: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:07:52.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:52.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:52.475036+0000) 2022-09-08T06:07:52.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:52.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:52.474916+0000) 2022-09-08T06:07:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:53.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:53.475205+0000) 2022-09-08T06:07:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:53.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:53.475130+0000) 2022-09-08T06:07:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:54 smithi059 ceph-mon[97450]: pgmap v16510: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:54.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:54 smithi055 ceph-mon[97033]: pgmap v16510: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:54.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:54.475406+0000) 2022-09-08T06:07:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:54.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:54.475387+0000) 2022-09-08T06:07:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:55.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:55.475697+0000) 2022-09-08T06:07:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:55.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:55.475675+0000) 2022-09-08T06:07:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:56 smithi059 ceph-mon[97450]: pgmap v16511: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:07:56.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:56 smithi055 ceph-mon[97033]: pgmap v16511: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:07:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:56.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:56.475889+0000) 2022-09-08T06:07:56.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:56.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:56.475848+0000) 2022-09-08T06:07:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:07:57 smithi055 ceph-mon[97033]: pgmap v16512: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:07:57 smithi059 ceph-mon[97450]: pgmap v16512: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:07:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:57.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:57.475980+0000) 2022-09-08T06:07:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:57.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:57.476028+0000) 2022-09-08T06:07:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:58.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:58.476130+0000) 2022-09-08T06:07:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:58.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:58.476193+0000) 2022-09-08T06:07:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:07:59.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:59.476348+0000) 2022-09-08T06:07:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:07:59.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:07:59.476257+0000) 2022-09-08T06:08:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:00 smithi059 ceph-mon[97450]: pgmap v16513: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:00 smithi055 ceph-mon[97033]: pgmap v16513: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:00.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:00.476559+0000) 2022-09-08T06:08:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:00.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:00.476519+0000) 2022-09-08T06:08:01.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:01.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:01.476787+0000) 2022-09-08T06:08:01.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:01.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:01.476785+0000) 2022-09-08T06:08:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:02 smithi059 ceph-mon[97450]: pgmap v16514: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:08:02.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:02 smithi055 ceph-mon[97033]: pgmap v16514: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:08:02.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:02.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:02.477079+0000) 2022-09-08T06:08:02.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:02.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:02.477076+0000) 2022-09-08T06:08:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:03.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:03.477316+0000) 2022-09-08T06:08:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:03.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:03.477314+0000) 2022-09-08T06:08:04.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:04 smithi059 ceph-mon[97450]: pgmap v16515: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:04 smithi055 ceph-mon[97033]: pgmap v16515: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:04.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:04.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:04.477531+0000) 2022-09-08T06:08:04.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:04.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:04.477531+0000) 2022-09-08T06:08:04.907 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:08:04.676+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:08:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:05.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:05.477836+0000) 2022-09-08T06:08:05.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:05.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:05.477833+0000) 2022-09-08T06:08:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:06 smithi055 ceph-mon[97033]: pgmap v16516: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:08:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:08:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:06 smithi059 ceph-mon[97450]: pgmap v16516: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:08:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:08:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:06.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:06.478131+0000) 2022-09-08T06:08:06.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:06.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:06.478131+0000) 2022-09-08T06:08:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:08:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:07 smithi055 ceph-mon[97033]: pgmap v16517: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:08:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:08:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:07 smithi059 ceph-mon[97450]: pgmap v16517: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:08:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:07.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:07.478363+0000) 2022-09-08T06:08:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:07.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:07.478363+0000) 2022-09-08T06:08:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:08.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:08.478618+0000) 2022-09-08T06:08:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:08.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:08.478616+0000) 2022-09-08T06:08:09.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:09.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:09.478874+0000) 2022-09-08T06:08:09.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:09.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:09.478874+0000) 2022-09-08T06:08:10.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:09 smithi055 ceph-mon[97033]: osdmap e1112: 12 total, 11 up, 11 in 2022-09-08T06:08:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:09 smithi055 ceph-mon[97033]: pgmap v16519: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:08:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:09 smithi059 ceph-mon[97450]: osdmap e1112: 12 total, 11 up, 11 in 2022-09-08T06:08:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:09 smithi059 ceph-mon[97450]: pgmap v16519: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:08:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:10.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:10.479144+0000) 2022-09-08T06:08:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:10.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:10.479143+0000) 2022-09-08T06:08:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:11.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:11.479396+0000) 2022-09-08T06:08:11.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:11.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:11.479396+0000) 2022-09-08T06:08:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:12 smithi059 ceph-mon[97450]: pgmap v16520: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:08:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:12 smithi055 ceph-mon[97033]: pgmap v16520: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:08:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:12.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:12.479759+0000) 2022-09-08T06:08:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:12.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:12.479739+0000) 2022-09-08T06:08:13.912 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:13.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:13.479976+0000) 2022-09-08T06:08:13.914 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:13.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:13.479976+0000) 2022-09-08T06:08:14.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:14 smithi059 ceph-mon[97450]: pgmap v16521: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:08:14.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:14 smithi055 ceph-mon[97033]: pgmap v16521: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:08:14.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:14.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:14.480246+0000) 2022-09-08T06:08:14.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:14.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:14.480246+0000) 2022-09-08T06:08:15.915 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:15.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:15.480472+0000) 2022-09-08T06:08:15.916 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:15.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:15.480470+0000) 2022-09-08T06:08:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:16 smithi055 ceph-mon[97033]: pgmap v16522: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:08:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:16 smithi059 ceph-mon[97450]: pgmap v16522: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:08:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:16.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:16.480769+0000) 2022-09-08T06:08:16.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:16.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:16.480772+0000) 2022-09-08T06:08:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:17 smithi055 ceph-mon[97033]: osdmap e1113: 12 total, 11 up, 11 in 2022-09-08T06:08:17.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:17 smithi055 ceph-mon[97033]: pgmap v16524: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T06:08:17.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:08:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:17 smithi059 ceph-mon[97450]: osdmap e1113: 12 total, 11 up, 11 in 2022-09-08T06:08:17.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:17 smithi059 ceph-mon[97450]: pgmap v16524: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T06:08:17.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:08:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:17.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:17.480967+0000) 2022-09-08T06:08:17.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:17.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:17.480967+0000) 2022-09-08T06:08:18.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:18 smithi055 ceph-mon[97033]: pgmap v16525: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T06:08:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:18 smithi055 ceph-mon[97033]: pgmap v16526: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T06:08:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:18 smithi059 ceph-mon[97450]: pgmap v16525: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 248 B/s wr, 7 op/s 2022-09-08T06:08:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:18 smithi059 ceph-mon[97450]: pgmap v16526: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T06:08:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:18.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:18.481106+0000) 2022-09-08T06:08:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:18.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:18.481117+0000) 2022-09-08T06:08:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:19.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:19.481350+0000) 2022-09-08T06:08:19.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:08:19.687+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:08:19.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:19.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:19.481350+0000) 2022-09-08T06:08:20.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:20 smithi055 ceph-mon[97033]: pgmap v16527: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T06:08:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:20 smithi059 ceph-mon[97450]: pgmap v16527: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T06:08:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:20.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:20.481560+0000) 2022-09-08T06:08:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:20.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:20.481560+0000) 2022-09-08T06:08:21.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:21.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:21.481756+0000) 2022-09-08T06:08:21.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:21.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:21.481766+0000) 2022-09-08T06:08:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:22 smithi055 ceph-mon[97033]: pgmap v16528: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:08:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:22 smithi059 ceph-mon[97450]: pgmap v16528: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:08:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:22.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:22.482002+0000) 2022-09-08T06:08:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:22.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:22.481995+0000) 2022-09-08T06:08:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:23.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:23.482206+0000) 2022-09-08T06:08:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:23.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:23.482206+0000) 2022-09-08T06:08:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:24 smithi055 ceph-mon[97033]: pgmap v16529: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-09-08T06:08:24.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:24 smithi059 ceph-mon[97450]: pgmap v16529: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-09-08T06:08:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:24.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:24.482458+0000) 2022-09-08T06:08:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:24.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:24.482456+0000) 2022-09-08T06:08:25.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:25.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:25.482718+0000) 2022-09-08T06:08:25.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:25.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:25.482723+0000) 2022-09-08T06:08:26.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:26 smithi055 ceph-mon[97033]: pgmap v16530: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:08:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:26 smithi059 ceph-mon[97450]: pgmap v16530: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:08:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:26.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:26.482932+0000) 2022-09-08T06:08:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:26.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:26.482932+0000) 2022-09-08T06:08:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:27.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:27.483135+0000) 2022-09-08T06:08:27.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:27.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:27.483119+0000) 2022-09-08T06:08:28.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:28 smithi055 ceph-mon[97033]: pgmap v16531: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:08:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:28 smithi059 ceph-mon[97450]: pgmap v16531: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:08:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:28.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:28.483324+0000) 2022-09-08T06:08:28.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:28.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:28.483329+0000) 2022-09-08T06:08:29.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:29.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:29.483512+0000) 2022-09-08T06:08:29.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:29.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:29.483515+0000) 2022-09-08T06:08:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:30 smithi055 ceph-mon[97033]: pgmap v16532: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:30 smithi059 ceph-mon[97450]: pgmap v16532: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:30.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:30.483721+0000) 2022-09-08T06:08:30.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:30.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:30.483726+0000) 2022-09-08T06:08:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:31.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:31.483913+0000) 2022-09-08T06:08:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:31.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:31.483924+0000) 2022-09-08T06:08:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:32 smithi055 ceph-mon[97033]: pgmap v16533: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:08:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:32 smithi059 ceph-mon[97450]: pgmap v16533: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:08:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:32.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:32.484106+0000) 2022-09-08T06:08:32.936 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:32.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:32.484101+0000) 2022-09-08T06:08:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:33.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:33.484332+0000) 2022-09-08T06:08:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:33.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:33.484312+0000) 2022-09-08T06:08:34.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:34 smithi055 ceph-mon[97033]: pgmap v16534: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:34 smithi059 ceph-mon[97450]: pgmap v16534: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:34.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:34.484607+0000) 2022-09-08T06:08:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:34.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:34.484684+0000) 2022-09-08T06:08:34.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:08:34.697+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:08:35.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:35.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:35.484891+0000) 2022-09-08T06:08:35.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:35.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:35.484924+0000) 2022-09-08T06:08:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:36 smithi055 ceph-mon[97033]: pgmap v16535: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:08:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:36 smithi059 ceph-mon[97450]: pgmap v16535: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:08:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:36.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:36.485124+0000) 2022-09-08T06:08:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:36.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:36.485129+0000) 2022-09-08T06:08:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:37.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:37.485302+0000) 2022-09-08T06:08:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:37.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:37.485269+0000) 2022-09-08T06:08:38.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:38 smithi055 ceph-mon[97033]: pgmap v16536: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:38 smithi059 ceph-mon[97450]: pgmap v16536: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:38.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:38.485387+0000) 2022-09-08T06:08:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:38.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:38.485452+0000) 2022-09-08T06:08:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:39.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:39.485553+0000) 2022-09-08T06:08:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:39.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:39.485610+0000) 2022-09-08T06:08:40.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:40 smithi055 ceph-mon[97033]: pgmap v16537: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:40 smithi059 ceph-mon[97450]: pgmap v16537: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:40.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:40.485701+0000) 2022-09-08T06:08:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:40.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:40.485786+0000) 2022-09-08T06:08:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:41.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:41.485936+0000) 2022-09-08T06:08:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:41.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:41.486022+0000) 2022-09-08T06:08:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:42 smithi055 ceph-mon[97033]: pgmap v16538: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:08:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:42 smithi059 ceph-mon[97450]: pgmap v16538: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:08:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:42.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:42.486177+0000) 2022-09-08T06:08:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:42.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:42.486263+0000) 2022-09-08T06:08:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:43.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:43.486400+0000) 2022-09-08T06:08:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:43.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:43.486442+0000) 2022-09-08T06:08:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:44 smithi055 ceph-mon[97033]: pgmap v16539: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:44 smithi059 ceph-mon[97450]: pgmap v16539: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:44.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:44.486607+0000) 2022-09-08T06:08:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:44.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:44.486613+0000) 2022-09-08T06:08:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:45.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:45.486772+0000) 2022-09-08T06:08:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:45.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:45.486778+0000) 2022-09-08T06:08:46.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:46 smithi055 ceph-mon[97033]: pgmap v16540: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:08:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:46 smithi059 ceph-mon[97450]: pgmap v16540: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:08:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:46.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:46.487031+0000) 2022-09-08T06:08:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:46.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:46.487021+0000) 2022-09-08T06:08:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:47.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:47.487280+0000) 2022-09-08T06:08:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:47.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:47.487263+0000) 2022-09-08T06:08:48.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:48 smithi055 ceph-mon[97033]: pgmap v16541: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:48 smithi059 ceph-mon[97450]: pgmap v16541: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:48.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:48.487477+0000) 2022-09-08T06:08:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:48.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:48.487488+0000) 2022-09-08T06:08:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:49.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:49.487759+0000) 2022-09-08T06:08:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:49.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:49.487750+0000) 2022-09-08T06:08:49.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:08:49.712+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:08:50.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:50 smithi055 ceph-mon[97033]: pgmap v16542: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:50 smithi059 ceph-mon[97450]: pgmap v16542: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:50.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:50.487945+0000) 2022-09-08T06:08:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:50.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:50.487975+0000) 2022-09-08T06:08:51.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:51.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:51.488176+0000) 2022-09-08T06:08:51.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:51.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:51.488223+0000) 2022-09-08T06:08:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:52 smithi055 ceph-mon[97033]: pgmap v16543: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:08:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:52 smithi059 ceph-mon[97450]: pgmap v16543: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:08:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:52.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:52.488450+0000) 2022-09-08T06:08:52.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:52.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:52.488474+0000) 2022-09-08T06:08:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:53.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:53.488737+0000) 2022-09-08T06:08:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:53.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:53.488752+0000) 2022-09-08T06:08:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:54 smithi055 ceph-mon[97033]: pgmap v16544: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:54.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:54 smithi059 ceph-mon[97450]: pgmap v16544: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:54.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:54.488975+0000) 2022-09-08T06:08:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:54.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:54.488964+0000) 2022-09-08T06:08:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:55.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:55.489229+0000) 2022-09-08T06:08:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:55.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:55.489199+0000) 2022-09-08T06:08:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:56 smithi055 ceph-mon[97033]: pgmap v16545: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:08:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:56 smithi059 ceph-mon[97450]: pgmap v16545: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:08:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:56.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:56.489504+0000) 2022-09-08T06:08:56.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:56.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:56.489483+0000) 2022-09-08T06:08:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:57.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:57.489702+0000) 2022-09-08T06:08:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:57.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:57.489702+0000) 2022-09-08T06:08:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:08:58 smithi055 ceph-mon[97033]: pgmap v16546: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:58.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:08:58 smithi059 ceph-mon[97450]: pgmap v16546: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:08:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:58.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:58.489834+0000) 2022-09-08T06:08:58.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:58.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:58.489836+0000) 2022-09-08T06:08:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:08:59.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:59.490049+0000) 2022-09-08T06:08:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:08:59.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:08:59.490047+0000) 2022-09-08T06:09:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:00 smithi055 ceph-mon[97033]: pgmap v16547: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:00 smithi059 ceph-mon[97450]: pgmap v16547: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:00.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:00.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:00.490199+0000) 2022-09-08T06:09:00.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:00.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:00.490243+0000) 2022-09-08T06:09:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:01.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:01.490406+0000) 2022-09-08T06:09:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:01.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:01.490425+0000) 2022-09-08T06:09:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:02 smithi055 ceph-mon[97033]: pgmap v16548: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:09:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:02 smithi059 ceph-mon[97450]: pgmap v16548: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:09:02.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:02.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:02.490590+0000) 2022-09-08T06:09:02.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:02.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:02.490584+0000) 2022-09-08T06:09:03.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:03.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:03.490795+0000) 2022-09-08T06:09:03.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:03.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:03.490800+0000) 2022-09-08T06:09:04.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:04 smithi055 ceph-mon[97033]: pgmap v16549: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:04 smithi059 ceph-mon[97450]: pgmap v16549: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:04.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:09:04.717+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:09:04.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:04.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:04.491019+0000) 2022-09-08T06:09:04.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:04.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:04.491020+0000) 2022-09-08T06:09:05.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:05.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:05.491247+0000) 2022-09-08T06:09:05.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:05.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:05.491230+0000) 2022-09-08T06:09:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:06 smithi055 ceph-mon[97033]: pgmap v16550: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:09:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:09:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:06 smithi059 ceph-mon[97450]: pgmap v16550: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:09:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:09:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:06.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:06.491479+0000) 2022-09-08T06:09:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:06.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:06.491483+0000) 2022-09-08T06:09:07.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:09:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:09:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:07.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:07.491720+0000) 2022-09-08T06:09:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:07.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:07.491760+0000) 2022-09-08T06:09:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:08 smithi055 ceph-mon[97033]: pgmap v16551: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:09:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:08 smithi059 ceph-mon[97450]: pgmap v16551: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:09:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:08.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:08.491940+0000) 2022-09-08T06:09:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:08.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:08.492011+0000) 2022-09-08T06:09:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:09.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:09.492169+0000) 2022-09-08T06:09:09.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:09.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:09.492229+0000) 2022-09-08T06:09:10.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:10 smithi059 ceph-mon[97450]: pgmap v16552: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:09:10.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:10 smithi059 ceph-mon[97450]: osdmap e1114: 12 total, 11 up, 11 in 2022-09-08T06:09:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:10.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:10.492340+0000) 2022-09-08T06:09:10.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:10.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:10.492403+0000) 2022-09-08T06:09:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:10 smithi055 ceph-mon[97033]: pgmap v16552: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:09:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:10 smithi055 ceph-mon[97033]: osdmap e1114: 12 total, 11 up, 11 in 2022-09-08T06:09:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:11.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:11.492494+0000) 2022-09-08T06:09:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:11.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:11.492560+0000) 2022-09-08T06:09:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:12 smithi059 ceph-mon[97450]: pgmap v16554: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:09:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:12.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:12.492764+0000) 2022-09-08T06:09:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:12.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:12.492775+0000) 2022-09-08T06:09:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:12 smithi055 ceph-mon[97033]: pgmap v16554: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:09:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:13.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:13.492932+0000) 2022-09-08T06:09:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:13.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:13.492935+0000) 2022-09-08T06:09:14.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:14 smithi059 ceph-mon[97450]: pgmap v16555: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:09:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:14.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:14.493147+0000) 2022-09-08T06:09:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:14.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:14.493148+0000) 2022-09-08T06:09:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:14 smithi055 ceph-mon[97033]: pgmap v16555: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:09:15.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:15.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:15.493316+0000) 2022-09-08T06:09:15.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:15.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:15.493333+0000) 2022-09-08T06:09:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:16 smithi059 ceph-mon[97450]: pgmap v16556: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:09:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:16 smithi059 ceph-mon[97450]: osdmap e1115: 12 total, 11 up, 11 in 2022-09-08T06:09:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:16.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:16.493486+0000) 2022-09-08T06:09:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:16.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:16.493486+0000) 2022-09-08T06:09:16.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:16 smithi055 ceph-mon[97033]: pgmap v16556: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:09:16.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:16 smithi055 ceph-mon[97033]: osdmap e1115: 12 total, 11 up, 11 in 2022-09-08T06:09:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:17.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:17.493719+0000) 2022-09-08T06:09:17.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:17.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:17.493719+0000) 2022-09-08T06:09:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:09:17.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:09:17.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:09:17.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:09:17.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:09:17.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:09:18.794 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:18 smithi059 ceph-mon[97450]: pgmap v16558: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:09:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:18.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:18.493961+0000) 2022-09-08T06:09:18.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:18.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:18.493954+0000) 2022-09-08T06:09:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:18 smithi055 ceph-mon[97033]: pgmap v16558: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:09:19.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:09:19.721+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:09:19.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:19.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:19.494198+0000) 2022-09-08T06:09:19.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:19.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:19.494198+0000) 2022-09-08T06:09:20.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:20 smithi059 ceph-mon[97450]: pgmap v16559: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2022-09-08T06:09:20.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:20.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:20.494381+0000) 2022-09-08T06:09:20.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:20.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:20.494383+0000) 2022-09-08T06:09:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:20 smithi055 ceph-mon[97033]: pgmap v16559: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2022-09-08T06:09:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:21.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:21.494633+0000) 2022-09-08T06:09:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:21.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:21.494633+0000) 2022-09-08T06:09:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:22 smithi059 ceph-mon[97450]: pgmap v16560: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:09:22.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:22.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:22.494849+0000) 2022-09-08T06:09:22.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:22.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:22.494849+0000) 2022-09-08T06:09:22.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:22 smithi055 ceph-mon[97033]: pgmap v16560: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:09:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:23.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:23.495103+0000) 2022-09-08T06:09:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:23.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:23.495103+0000) 2022-09-08T06:09:24.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:24 smithi059 ceph-mon[97450]: pgmap v16561: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:09:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:24.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:24.495339+0000) 2022-09-08T06:09:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:24.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:24.495336+0000) 2022-09-08T06:09:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:24 smithi055 ceph-mon[97033]: pgmap v16561: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:09:25.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:25.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:25.495614+0000) 2022-09-08T06:09:25.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:25.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:25.495613+0000) 2022-09-08T06:09:26.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:26 smithi059 ceph-mon[97450]: pgmap v16562: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:09:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:26.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:26.495785+0000) 2022-09-08T06:09:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:26.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:26.495773+0000) 2022-09-08T06:09:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:26 smithi055 ceph-mon[97033]: pgmap v16562: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:09:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:27.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:27.495948+0000) 2022-09-08T06:09:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:27.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:27.495948+0000) 2022-09-08T06:09:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:28 smithi059 ceph-mon[97450]: pgmap v16563: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:09:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:28.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:28.496144+0000) 2022-09-08T06:09:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:28.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:28.496138+0000) 2022-09-08T06:09:28.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:28 smithi055 ceph-mon[97033]: pgmap v16563: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:09:29.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:29.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:29.496369+0000) 2022-09-08T06:09:29.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:29.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:29.496369+0000) 2022-09-08T06:09:30.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:30 smithi059 ceph-mon[97450]: pgmap v16564: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:30.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:30.496596+0000) 2022-09-08T06:09:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:30.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:30.496600+0000) 2022-09-08T06:09:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:30 smithi055 ceph-mon[97033]: pgmap v16564: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:31.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:31.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:31.496824+0000) 2022-09-08T06:09:31.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:31.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:31.496824+0000) 2022-09-08T06:09:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:32 smithi059 ceph-mon[97450]: pgmap v16565: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:09:32.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:32.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:32.497049+0000) 2022-09-08T06:09:32.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:32.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:32.497049+0000) 2022-09-08T06:09:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:32 smithi055 ceph-mon[97033]: pgmap v16565: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:09:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:33.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:33.497302+0000) 2022-09-08T06:09:33.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:33.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:33.497303+0000) 2022-09-08T06:09:34.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:09:34.733+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:09:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:34.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:34.497488+0000) 2022-09-08T06:09:34.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:34.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:34.497492+0000) 2022-09-08T06:09:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:34 smithi055 ceph-mon[97033]: pgmap v16566: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:34 smithi059 ceph-mon[97450]: pgmap v16566: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:35.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:35.497752+0000) 2022-09-08T06:09:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:35.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:35.497756+0000) 2022-09-08T06:09:36.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:36.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:36.497987+0000) 2022-09-08T06:09:36.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:36.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:36.498006+0000) 2022-09-08T06:09:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:36 smithi055 ceph-mon[97033]: pgmap v16567: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:09:37.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:36 smithi059 ceph-mon[97450]: pgmap v16567: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:09:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:37.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:37.498256+0000) 2022-09-08T06:09:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:37.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:37.498255+0000) 2022-09-08T06:09:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:38.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:38.498442+0000) 2022-09-08T06:09:38.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:38.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:38.498445+0000) 2022-09-08T06:09:38.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:38 smithi055 ceph-mon[97033]: pgmap v16568: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:39.043 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:38 smithi059 ceph-mon[97450]: pgmap v16568: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:39.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:39.498705+0000) 2022-09-08T06:09:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:39.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:39.498690+0000) 2022-09-08T06:09:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:40.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:40.498999+0000) 2022-09-08T06:09:40.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:40.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:40.498999+0000) 2022-09-08T06:09:40.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:40 smithi055 ceph-mon[97033]: pgmap v16569: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:40 smithi059 ceph-mon[97450]: pgmap v16569: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:41.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:41.499248+0000) 2022-09-08T06:09:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:41.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:41.499248+0000) 2022-09-08T06:09:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:42.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:42.499437+0000) 2022-09-08T06:09:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:42.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:42.499439+0000) 2022-09-08T06:09:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:42 smithi055 ceph-mon[97033]: pgmap v16570: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:09:43.050 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:42 smithi059 ceph-mon[97450]: pgmap v16570: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:09:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:43.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:43.499621+0000) 2022-09-08T06:09:43.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:43.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:43.499621+0000) 2022-09-08T06:09:44.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:44.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:44.499779+0000) 2022-09-08T06:09:44.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:44.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:44.499788+0000) 2022-09-08T06:09:44.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:44 smithi055 ceph-mon[97033]: pgmap v16571: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:44 smithi059 ceph-mon[97450]: pgmap v16571: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:45.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:45.499994+0000) 2022-09-08T06:09:45.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:45.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:45.499994+0000) 2022-09-08T06:09:46.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:46.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:46.500174+0000) 2022-09-08T06:09:46.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:46.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:46.500172+0000) 2022-09-08T06:09:46.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:46 smithi055 ceph-mon[97033]: pgmap v16572: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:09:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:46 smithi059 ceph-mon[97450]: pgmap v16572: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:09:47.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:47.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:47.500443+0000) 2022-09-08T06:09:47.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:47.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:47.500441+0000) 2022-09-08T06:09:48.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:48 smithi059 ceph-mon[97450]: pgmap v16573: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:48.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:48 smithi059 ceph-mon[97450]: pgmap v16574: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:09:48.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:48 smithi059 ceph-mon[97450]: pgmap v16575: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:09:48.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:09:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:48.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:48.500732+0000) 2022-09-08T06:09:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:48.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:48.500726+0000) 2022-09-08T06:09:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:48 smithi055 ceph-mon[97033]: pgmap v16573: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:09:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:48 smithi055 ceph-mon[97033]: pgmap v16574: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:09:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:48 smithi055 ceph-mon[97033]: pgmap v16575: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:09:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:09:49.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:09:49.748+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:09:49.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:49.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:49.500924+0000) 2022-09-08T06:09:49.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:49.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:49.500923+0000) 2022-09-08T06:09:50.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:50 smithi059 ceph-mon[97450]: pgmap v16576: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:09:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:50.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:50.501160+0000) 2022-09-08T06:09:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:50.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:50.501144+0000) 2022-09-08T06:09:50.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:50 smithi055 ceph-mon[97033]: pgmap v16576: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:09:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:51.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:51.501320+0000) 2022-09-08T06:09:51.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:51.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:51.501350+0000) 2022-09-08T06:09:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:52 smithi055 ceph-mon[97033]: pgmap v16577: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:09:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:52 smithi059 ceph-mon[97450]: pgmap v16577: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:09:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:52.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:52.501564+0000) 2022-09-08T06:09:52.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:52.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:52.501564+0000) 2022-09-08T06:09:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:53.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:53.501862+0000) 2022-09-08T06:09:53.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:53.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:53.501857+0000) 2022-09-08T06:09:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:54.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:54.502168+0000) 2022-09-08T06:09:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:54.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:54.502161+0000) 2022-09-08T06:09:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:54 smithi055 ceph-mon[97033]: pgmap v16578: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:09:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:54 smithi059 ceph-mon[97450]: pgmap v16578: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:09:55.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:55.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:55.502462+0000) 2022-09-08T06:09:55.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:55.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:55.502460+0000) 2022-09-08T06:09:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:56.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:56.502697+0000) 2022-09-08T06:09:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:56.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:56.502711+0000) 2022-09-08T06:09:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:56 smithi055 ceph-mon[97033]: pgmap v16579: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:09:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:56 smithi059 ceph-mon[97450]: pgmap v16579: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:09:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:57.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:57.502928+0000) 2022-09-08T06:09:57.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:57.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:57.502928+0000) 2022-09-08T06:09:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:58.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:58.503180+0000) 2022-09-08T06:09:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:58.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:58.503184+0000) 2022-09-08T06:09:58.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:09:58 smithi055 ceph-mon[97033]: pgmap v16580: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:09:59.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:09:58 smithi059 ceph-mon[97450]: pgmap v16580: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:09:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:09:59.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:59.503394+0000) 2022-09-08T06:09:59.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:09:59.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:09:59.503381+0000) 2022-09-08T06:10:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:00.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:00.503536+0000) 2022-09-08T06:10:00.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:00.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:00.503503+0000) 2022-09-08T06:10:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:00 smithi055 ceph-mon[97033]: pgmap v16581: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T06:10:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T06:10:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T06:10:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T06:10:00.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T06:10:01.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:00 smithi059 ceph-mon[97450]: pgmap v16581: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:01.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T06:10:01.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T06:10:01.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T06:10:01.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T06:10:01.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T06:10:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:01.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:01.503765+0000) 2022-09-08T06:10:01.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:01.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:01.503773+0000) 2022-09-08T06:10:02.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:02 smithi055 ceph-mon[97033]: pgmap v16582: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:10:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:02 smithi059 ceph-mon[97450]: pgmap v16582: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:10:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:02.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:02.503956+0000) 2022-09-08T06:10:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:02.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:02.503941+0000) 2022-09-08T06:10:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:03.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:03.504210+0000) 2022-09-08T06:10:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:03.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:03.504214+0000) 2022-09-08T06:10:04.760 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:04.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:04.504419+0000) 2022-09-08T06:10:04.760 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:04.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:04.504426+0000) 2022-09-08T06:10:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:04 smithi055 ceph-mon[97033]: pgmap v16583: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:04 smithi059 ceph-mon[97450]: pgmap v16583: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:05.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:10:04.757+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:10:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:05.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:05.504612+0000) 2022-09-08T06:10:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:05.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:05.504603+0000) 2022-09-08T06:10:06.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:06 smithi059 ceph-mon[97450]: pgmap v16584: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:10:06.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:10:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:06.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:06.504871+0000) 2022-09-08T06:10:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:06.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:06.504844+0000) 2022-09-08T06:10:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:06 smithi055 ceph-mon[97033]: pgmap v16584: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:10:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:10:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:07.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:07.505092+0000) 2022-09-08T06:10:07.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:07.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:07.505110+0000) 2022-09-08T06:10:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:10:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:10:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:08.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:08.505327+0000) 2022-09-08T06:10:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:08.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:08.505340+0000) 2022-09-08T06:10:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:08 smithi055 ceph-mon[97033]: pgmap v16585: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:10:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:08 smithi059 ceph-mon[97450]: pgmap v16585: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:10:09.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:09.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:09.505570+0000) 2022-09-08T06:10:09.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:09.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:09.505594+0000) 2022-09-08T06:10:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:10.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:10.505840+0000) 2022-09-08T06:10:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:10.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:10.505721+0000) 2022-09-08T06:10:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:10 smithi055 ceph-mon[97033]: pgmap v16586: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:10:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:10 smithi055 ceph-mon[97033]: osdmap e1116: 12 total, 11 up, 11 in 2022-09-08T06:10:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:10 smithi059 ceph-mon[97450]: pgmap v16586: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:10:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:10 smithi059 ceph-mon[97450]: osdmap e1116: 12 total, 11 up, 11 in 2022-09-08T06:10:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:11.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:11.505882+0000) 2022-09-08T06:10:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:11.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:11.505983+0000) 2022-09-08T06:10:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:12 smithi055 ceph-mon[97033]: pgmap v16588: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:10:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:12 smithi059 ceph-mon[97450]: pgmap v16588: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:10:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:12.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:12.506050+0000) 2022-09-08T06:10:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:12.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:12.506211+0000) 2022-09-08T06:10:13.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:13.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:13.506273+0000) 2022-09-08T06:10:13.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:13.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:13.506368+0000) 2022-09-08T06:10:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:14.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:14.506514+0000) 2022-09-08T06:10:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:14.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:14.506470+0000) 2022-09-08T06:10:14.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:14 smithi055 ceph-mon[97033]: pgmap v16589: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:10:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:14 smithi059 ceph-mon[97450]: pgmap v16589: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:10:15.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:15.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:15.506655+0000) 2022-09-08T06:10:15.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:15.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:15.506693+0000) 2022-09-08T06:10:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:16.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:16.506821+0000) 2022-09-08T06:10:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:16.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:16.506843+0000) 2022-09-08T06:10:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:16 smithi055 ceph-mon[97033]: pgmap v16590: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:10:16.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:16 smithi055 ceph-mon[97033]: osdmap e1117: 12 total, 11 up, 11 in 2022-09-08T06:10:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:16 smithi059 ceph-mon[97450]: pgmap v16590: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:10:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:16 smithi059 ceph-mon[97450]: osdmap e1117: 12 total, 11 up, 11 in 2022-09-08T06:10:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:17.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:17.506995+0000) 2022-09-08T06:10:17.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:17.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:17.506995+0000) 2022-09-08T06:10:18.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:18.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:18.507197+0000) 2022-09-08T06:10:18.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:18.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:18.507197+0000) 2022-09-08T06:10:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:18 smithi055 ceph-mon[97033]: pgmap v16592: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:10:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:18 smithi059 ceph-mon[97450]: pgmap v16592: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:10:19.774 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:19.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:19.507442+0000) 2022-09-08T06:10:19.776 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:19.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:19.507442+0000) 2022-09-08T06:10:20.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:10:19.771+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:10:20.927 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:20.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:20.507730+0000) 2022-09-08T06:10:20.928 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:20.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:20.507760+0000) 2022-09-08T06:10:20.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:20 smithi055 ceph-mon[97033]: pgmap v16593: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:10:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:20 smithi059 ceph-mon[97450]: pgmap v16593: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:10:21.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:21.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:21.507895+0000) 2022-09-08T06:10:21.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:21.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:21.507965+0000) 2022-09-08T06:10:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:22 smithi055 ceph-mon[97033]: pgmap v16594: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:10:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:22 smithi059 ceph-mon[97450]: pgmap v16594: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:10:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:22.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:22.508110+0000) 2022-09-08T06:10:22.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:22.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:22.508193+0000) 2022-09-08T06:10:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:23.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:23.508327+0000) 2022-09-08T06:10:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:23.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:23.508367+0000) 2022-09-08T06:10:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:24.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:24.508607+0000) 2022-09-08T06:10:24.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:24.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:24.508556+0000) 2022-09-08T06:10:24.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:24 smithi055 ceph-mon[97033]: pgmap v16595: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:10:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:24 smithi059 ceph-mon[97450]: pgmap v16595: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:10:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:25.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:25.508701+0000) 2022-09-08T06:10:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:25.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:25.508826+0000) 2022-09-08T06:10:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:26.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:26.509008+0000) 2022-09-08T06:10:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:26.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:26.508926+0000) 2022-09-08T06:10:26.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:26 smithi055 ceph-mon[97033]: pgmap v16596: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:10:27.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:26 smithi059 ceph-mon[97450]: pgmap v16596: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:10:27.914 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:27.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:27.509143+0000) 2022-09-08T06:10:27.915 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:27.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:27.509175+0000) 2022-09-08T06:10:28.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:28.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:28.509348+0000) 2022-09-08T06:10:28.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:28.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:28.509365+0000) 2022-09-08T06:10:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:28 smithi055 ceph-mon[97033]: pgmap v16597: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:10:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:28 smithi059 ceph-mon[97450]: pgmap v16597: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:10:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:29.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:29.509600+0000) 2022-09-08T06:10:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:29.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:29.509599+0000) 2022-09-08T06:10:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:30.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:30.509897+0000) 2022-09-08T06:10:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:30.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:30.509900+0000) 2022-09-08T06:10:30.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:30 smithi055 ceph-mon[97033]: pgmap v16598: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:31.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:30 smithi059 ceph-mon[97450]: pgmap v16598: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:31.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:31.510130+0000) 2022-09-08T06:10:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:31.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:31.510124+0000) 2022-09-08T06:10:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:32 smithi055 ceph-mon[97033]: pgmap v16599: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:10:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:32 smithi059 ceph-mon[97450]: pgmap v16599: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:10:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:32.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:32.510363+0000) 2022-09-08T06:10:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:32.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:32.510362+0000) 2022-09-08T06:10:33.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:33.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:33.510665+0000) 2022-09-08T06:10:33.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:33.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:33.510572+0000) 2022-09-08T06:10:34.786 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:34.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:34.510906+0000) 2022-09-08T06:10:34.787 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:34.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:34.510817+0000) 2022-09-08T06:10:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:34 smithi055 ceph-mon[97033]: pgmap v16600: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:35.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:34 smithi059 ceph-mon[97450]: pgmap v16600: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:35.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:10:34.783+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:10:35.911 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:35.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:35.511058+0000) 2022-09-08T06:10:35.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:35.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:35.510957+0000) 2022-09-08T06:10:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:36.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:36.511220+0000) 2022-09-08T06:10:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:36.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:36.511113+0000) 2022-09-08T06:10:36.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:36 smithi055 ceph-mon[97033]: pgmap v16601: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:10:37.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:36 smithi059 ceph-mon[97450]: pgmap v16601: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:10:37.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:37.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:37.511449+0000) 2022-09-08T06:10:37.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:37.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:37.511329+0000) 2022-09-08T06:10:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:38.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:38.511600+0000) 2022-09-08T06:10:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:38.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:38.511503+0000) 2022-09-08T06:10:38.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:38 smithi055 ceph-mon[97033]: pgmap v16602: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:38 smithi059 ceph-mon[97450]: pgmap v16602: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:39.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:39.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:39.511762+0000) 2022-09-08T06:10:39.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:39.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:39.511682+0000) 2022-09-08T06:10:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:40.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:40.511895+0000) 2022-09-08T06:10:40.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:40.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:40.511836+0000) 2022-09-08T06:10:40.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:40 smithi055 ceph-mon[97033]: pgmap v16603: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:40 smithi059 ceph-mon[97450]: pgmap v16603: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:41.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:41.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:41.512097+0000) 2022-09-08T06:10:41.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:41.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:41.512039+0000) 2022-09-08T06:10:42.738 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:42 smithi055 ceph-mon[97033]: pgmap v16604: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:10:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:42 smithi059 ceph-mon[97450]: pgmap v16604: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:10:42.912 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:42.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:42.512270+0000) 2022-09-08T06:10:42.916 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:42.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:42.512316+0000) 2022-09-08T06:10:43.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:43.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:43.512528+0000) 2022-09-08T06:10:43.934 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:43.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:43.512518+0000) 2022-09-08T06:10:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:44.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:44.512770+0000) 2022-09-08T06:10:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:44.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:44.512772+0000) 2022-09-08T06:10:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:44 smithi055 ceph-mon[97033]: pgmap v16605: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:45.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:44 smithi059 ceph-mon[97450]: pgmap v16605: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:45.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:45.513000+0000) 2022-09-08T06:10:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:45.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:45.512999+0000) 2022-09-08T06:10:46.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:46.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:46.513191+0000) 2022-09-08T06:10:46.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:46.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:46.513208+0000) 2022-09-08T06:10:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:46 smithi055 ceph-mon[97033]: pgmap v16606: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:10:47.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:46 smithi059 ceph-mon[97450]: pgmap v16606: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:10:47.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:47.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:47.513435+0000) 2022-09-08T06:10:47.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:47.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:47.513457+0000) 2022-09-08T06:10:48.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:48.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:48.513704+0000) 2022-09-08T06:10:48.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:48.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:48.513711+0000) 2022-09-08T06:10:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:48 smithi055 ceph-mon[97033]: pgmap v16607: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:10:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:10:48.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:10:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:48 smithi059 ceph-mon[97450]: pgmap v16607: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:49.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:10:49.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:10:49.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:10:49.787 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:49.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:49.513914+0000) 2022-09-08T06:10:49.788 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:49.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:49.513917+0000) 2022-09-08T06:10:50.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:10:49.784+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:10:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:50.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:50.514131+0000) 2022-09-08T06:10:50.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:50.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:50.514143+0000) 2022-09-08T06:10:50.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:50 smithi055 ceph-mon[97033]: pgmap v16608: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:50 smithi059 ceph-mon[97450]: pgmap v16608: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:51.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:51.514368+0000) 2022-09-08T06:10:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:51.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:51.514360+0000) 2022-09-08T06:10:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:52 smithi055 ceph-mon[97033]: pgmap v16609: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:10:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:52 smithi059 ceph-mon[97450]: pgmap v16609: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:10:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:52.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:52.514576+0000) 2022-09-08T06:10:52.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:52.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:52.514604+0000) 2022-09-08T06:10:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:53.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:53.514736+0000) 2022-09-08T06:10:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:53.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:53.514754+0000) 2022-09-08T06:10:54.911 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:54.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:54.514961+0000) 2022-09-08T06:10:54.912 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:54.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:54.514974+0000) 2022-09-08T06:10:54.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:54 smithi055 ceph-mon[97033]: pgmap v16610: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:54 smithi059 ceph-mon[97450]: pgmap v16610: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:55.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:55.515234+0000) 2022-09-08T06:10:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:55.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:55.515228+0000) 2022-09-08T06:10:56.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:56.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:56.515480+0000) 2022-09-08T06:10:56.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:56.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:56.515487+0000) 2022-09-08T06:10:56.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:56 smithi055 ceph-mon[97033]: pgmap v16611: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:10:57.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:56 smithi059 ceph-mon[97450]: pgmap v16611: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:10:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:57.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:57.515754+0000) 2022-09-08T06:10:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:57.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:57.515756+0000) 2022-09-08T06:10:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:58.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:58.515957+0000) 2022-09-08T06:10:58.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:58.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:58.515967+0000) 2022-09-08T06:10:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:10:58 smithi055 ceph-mon[97033]: pgmap v16612: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:10:58 smithi059 ceph-mon[97450]: pgmap v16612: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:10:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:10:59.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:59.516124+0000) 2022-09-08T06:10:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:10:59.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:10:59.516137+0000) 2022-09-08T06:11:00.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:00.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:00.516340+0000) 2022-09-08T06:11:00.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:00.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:00.516352+0000) 2022-09-08T06:11:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:00 smithi055 ceph-mon[97033]: pgmap v16613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:00 smithi059 ceph-mon[97450]: pgmap v16613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:01.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:01.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:01.516572+0000) 2022-09-08T06:11:01.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:01.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:01.516572+0000) 2022-09-08T06:11:02.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:02 smithi055 ceph-mon[97033]: pgmap v16614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:11:02.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:02 smithi059 ceph-mon[97450]: pgmap v16614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:11:02.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:02.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:02.516779+0000) 2022-09-08T06:11:02.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:02.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:02.516779+0000) 2022-09-08T06:11:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:03.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:03.516929+0000) 2022-09-08T06:11:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:03.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:03.516931+0000) 2022-09-08T06:11:04.807 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:04.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:04.517105+0000) 2022-09-08T06:11:04.813 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:04.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:04.517105+0000) 2022-09-08T06:11:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:04 smithi055 ceph-mon[97033]: pgmap v16615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:04 smithi059 ceph-mon[97450]: pgmap v16615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:11:04.799+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:11:05.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:05.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:05.517305+0000) 2022-09-08T06:11:05.912 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:05.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:05.517309+0000) 2022-09-08T06:11:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:06.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:06.517575+0000) 2022-09-08T06:11:06.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:06.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:06.517575+0000) 2022-09-08T06:11:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:06 smithi055 ceph-mon[97033]: pgmap v16616: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:11:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:11:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:06 smithi059 ceph-mon[97450]: pgmap v16616: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:11:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:11:07.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:07.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:07.517861+0000) 2022-09-08T06:11:07.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:07.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:07.517854+0000) 2022-09-08T06:11:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:11:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:11:08.915 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:08.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:08.518134+0000) 2022-09-08T06:11:08.916 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:08.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:08.518132+0000) 2022-09-08T06:11:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:08 smithi055 ceph-mon[97033]: pgmap v16617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:11:09.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:08 smithi059 ceph-mon[97450]: pgmap v16617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:11:09.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:09.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:09.518306+0000) 2022-09-08T06:11:09.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:09.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:09.518306+0000) 2022-09-08T06:11:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:10.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:10.518460+0000) 2022-09-08T06:11:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:10.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:10.518461+0000) 2022-09-08T06:11:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:10 smithi055 ceph-mon[97033]: pgmap v16618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:11:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:10 smithi055 ceph-mon[97033]: osdmap e1118: 12 total, 11 up, 11 in 2022-09-08T06:11:11.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:10 smithi059 ceph-mon[97450]: pgmap v16618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:11:11.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:10 smithi059 ceph-mon[97450]: osdmap e1118: 12 total, 11 up, 11 in 2022-09-08T06:11:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:11.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:11.518762+0000) 2022-09-08T06:11:11.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:11.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:11.518762+0000) 2022-09-08T06:11:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:12 smithi055 ceph-mon[97033]: pgmap v16620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:11:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:12 smithi059 ceph-mon[97450]: pgmap v16620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:11:12.788 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:12.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:12.518919+0000) 2022-09-08T06:11:12.790 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:12.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:12.518914+0000) 2022-09-08T06:11:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:13.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:13.519090+0000) 2022-09-08T06:11:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:13.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:13.519089+0000) 2022-09-08T06:11:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:14.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:14.519359+0000) 2022-09-08T06:11:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:14.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:14.519343+0000) 2022-09-08T06:11:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:14 smithi055 ceph-mon[97033]: pgmap v16621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:11:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:14 smithi059 ceph-mon[97450]: pgmap v16621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:11:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:15.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:15.519564+0000) 2022-09-08T06:11:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:15.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:15.519564+0000) 2022-09-08T06:11:16.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:16.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:16.519772+0000) 2022-09-08T06:11:16.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:16.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:16.519771+0000) 2022-09-08T06:11:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:16 smithi055 ceph-mon[97033]: pgmap v16622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:11:16.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:16 smithi055 ceph-mon[97033]: osdmap e1119: 12 total, 11 up, 11 in 2022-09-08T06:11:17.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:16 smithi059 ceph-mon[97450]: pgmap v16622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:11:17.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:16 smithi059 ceph-mon[97450]: osdmap e1119: 12 total, 11 up, 11 in 2022-09-08T06:11:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:17.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:17.519941+0000) 2022-09-08T06:11:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:17.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:17.519941+0000) 2022-09-08T06:11:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:18.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:18.520164+0000) 2022-09-08T06:11:18.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:18.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:18.520164+0000) 2022-09-08T06:11:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:18 smithi059 ceph-mon[97450]: pgmap v16624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:11:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:18 smithi059 ceph-mon[97450]: pgmap v16625: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T06:11:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:18 smithi059 ceph-mon[97450]: pgmap v16626: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T06:11:19.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:11:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:18 smithi055 ceph-mon[97033]: pgmap v16624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:11:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:18 smithi055 ceph-mon[97033]: pgmap v16625: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T06:11:19.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:18 smithi055 ceph-mon[97033]: pgmap v16626: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T06:11:19.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:11:19.814 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:19.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:19.520442+0000) 2022-09-08T06:11:19.815 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:19.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:19.520442+0000) 2022-09-08T06:11:20.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:11:19.806+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:11:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:20.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:20.520750+0000) 2022-09-08T06:11:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:20.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:20.520738+0000) 2022-09-08T06:11:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:20 smithi059 ceph-mon[97450]: pgmap v16627: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T06:11:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:20 smithi055 ceph-mon[97033]: pgmap v16627: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T06:11:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:21.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:21.521013+0000) 2022-09-08T06:11:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:21.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:21.521013+0000) 2022-09-08T06:11:22.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:22 smithi055 ceph-mon[97033]: pgmap v16628: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 4 op/s 2022-09-08T06:11:22.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:22 smithi059 ceph-mon[97450]: pgmap v16628: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 4 op/s 2022-09-08T06:11:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:22.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:22.521301+0000) 2022-09-08T06:11:22.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:22.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:22.521301+0000) 2022-09-08T06:11:23.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:23.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:23.521603+0000) 2022-09-08T06:11:23.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:23.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:23.521605+0000) 2022-09-08T06:11:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:24.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:24.521876+0000) 2022-09-08T06:11:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:24.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:24.521884+0000) 2022-09-08T06:11:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:24 smithi055 ceph-mon[97033]: pgmap v16629: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-09-08T06:11:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:24 smithi059 ceph-mon[97450]: pgmap v16629: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-09-08T06:11:25.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:25.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:25.522092+0000) 2022-09-08T06:11:25.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:25.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:25.522092+0000) 2022-09-08T06:11:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:26.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:26.522358+0000) 2022-09-08T06:11:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:26.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:26.522355+0000) 2022-09-08T06:11:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:26 smithi055 ceph-mon[97033]: pgmap v16630: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T06:11:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:26 smithi059 ceph-mon[97450]: pgmap v16630: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T06:11:27.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:27.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:27.522666+0000) 2022-09-08T06:11:27.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:27.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:27.522631+0000) 2022-09-08T06:11:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:28.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:28.522943+0000) 2022-09-08T06:11:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:28.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:28.522943+0000) 2022-09-08T06:11:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:28 smithi055 ceph-mon[97033]: pgmap v16631: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T06:11:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:28 smithi059 ceph-mon[97450]: pgmap v16631: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T06:11:29.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:29.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:29.523199+0000) 2022-09-08T06:11:29.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:29.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:29.523202+0000) 2022-09-08T06:11:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:30.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:30.523427+0000) 2022-09-08T06:11:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:30.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:30.523427+0000) 2022-09-08T06:11:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:30 smithi055 ceph-mon[97033]: pgmap v16632: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T06:11:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:30 smithi059 ceph-mon[97450]: pgmap v16632: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T06:11:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:31.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:31.523655+0000) 2022-09-08T06:11:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:31.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:31.523649+0000) 2022-09-08T06:11:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:32 smithi055 ceph-mon[97033]: pgmap v16633: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T06:11:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:32 smithi059 ceph-mon[97450]: pgmap v16633: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T06:11:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:32.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:32.523913+0000) 2022-09-08T06:11:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:32.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:32.523914+0000) 2022-09-08T06:11:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:33.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:33.524227+0000) 2022-09-08T06:11:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:33.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:33.524227+0000) 2022-09-08T06:11:34.825 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:34.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:34.524426+0000) 2022-09-08T06:11:34.825 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:34.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:34.524427+0000) 2022-09-08T06:11:35.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:11:34.822+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:11:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:34 smithi055 ceph-mon[97033]: pgmap v16634: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:34 smithi059 ceph-mon[97450]: pgmap v16634: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:35.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:35.524750+0000) 2022-09-08T06:11:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:35.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:35.524747+0000) 2022-09-08T06:11:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:36.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:36.525017+0000) 2022-09-08T06:11:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:36.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:36.525017+0000) 2022-09-08T06:11:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:36 smithi055 ceph-mon[97033]: pgmap v16635: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:11:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:36 smithi059 ceph-mon[97450]: pgmap v16635: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:11:37.794 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:37.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:37.525320+0000) 2022-09-08T06:11:37.796 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:37.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:37.525317+0000) 2022-09-08T06:11:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:38.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:38.525593+0000) 2022-09-08T06:11:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:38.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:38.525603+0000) 2022-09-08T06:11:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:38 smithi055 ceph-mon[97033]: pgmap v16636: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:38 smithi059 ceph-mon[97450]: pgmap v16636: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:39.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:39.525813+0000) 2022-09-08T06:11:39.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:39.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:39.525818+0000) 2022-09-08T06:11:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:40.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:40.526033+0000) 2022-09-08T06:11:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:40.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:40.526033+0000) 2022-09-08T06:11:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:40 smithi055 ceph-mon[97033]: pgmap v16637: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:40 smithi059 ceph-mon[97450]: pgmap v16637: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:41.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:41.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:41.526235+0000) 2022-09-08T06:11:41.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:41.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:41.526235+0000) 2022-09-08T06:11:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:42 smithi055 ceph-mon[97033]: pgmap v16638: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:11:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:42 smithi059 ceph-mon[97450]: pgmap v16638: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:11:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:42.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:42.526507+0000) 2022-09-08T06:11:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:42.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:42.526510+0000) 2022-09-08T06:11:43.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:43.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:43.526784+0000) 2022-09-08T06:11:43.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:43.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:43.526798+0000) 2022-09-08T06:11:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:44.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:44.527008+0000) 2022-09-08T06:11:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:44.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:44.527010+0000) 2022-09-08T06:11:45.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:44 smithi055 ceph-mon[97033]: pgmap v16639: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:44 smithi059 ceph-mon[97450]: pgmap v16639: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:45.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:45.527271+0000) 2022-09-08T06:11:45.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:45.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:45.527272+0000) 2022-09-08T06:11:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:46.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:46.527533+0000) 2022-09-08T06:11:46.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:46.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:46.527534+0000) 2022-09-08T06:11:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:46 smithi055 ceph-mon[97033]: pgmap v16640: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:11:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:46 smithi059 ceph-mon[97450]: pgmap v16640: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:11:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:47.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:47.527765+0000) 2022-09-08T06:11:47.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:47.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:47.527765+0000) 2022-09-08T06:11:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:48.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:48.527938+0000) 2022-09-08T06:11:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:48.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:48.527950+0000) 2022-09-08T06:11:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:48 smithi055 ceph-mon[97033]: pgmap v16641: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:48 smithi059 ceph-mon[97450]: pgmap v16641: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:49.838 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:49.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:49.528124+0000) 2022-09-08T06:11:49.839 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:49.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:49.528124+0000) 2022-09-08T06:11:50.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:11:49.835+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:11:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:50.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:50.528315+0000) 2022-09-08T06:11:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:50.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:50.528312+0000) 2022-09-08T06:11:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:50 smithi055 ceph-mon[97033]: pgmap v16642: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:50 smithi059 ceph-mon[97450]: pgmap v16642: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:51.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:51.528568+0000) 2022-09-08T06:11:51.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:51.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:51.528563+0000) 2022-09-08T06:11:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:52 smithi055 ceph-mon[97033]: pgmap v16643: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:11:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:52 smithi059 ceph-mon[97450]: pgmap v16643: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:11:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:52.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:52.528824+0000) 2022-09-08T06:11:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:52.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:52.528817+0000) 2022-09-08T06:11:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:53.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:53.529098+0000) 2022-09-08T06:11:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:53.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:53.529099+0000) 2022-09-08T06:11:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:54.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:54.529365+0000) 2022-09-08T06:11:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:54.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:54.529369+0000) 2022-09-08T06:11:55.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:54 smithi055 ceph-mon[97033]: pgmap v16644: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:55.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:54 smithi059 ceph-mon[97450]: pgmap v16644: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:55.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:55.529620+0000) 2022-09-08T06:11:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:55.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:55.529619+0000) 2022-09-08T06:11:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:56.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:56.529944+0000) 2022-09-08T06:11:56.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:56.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:56.529944+0000) 2022-09-08T06:11:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:56 smithi055 ceph-mon[97033]: pgmap v16645: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:11:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:56 smithi059 ceph-mon[97450]: pgmap v16645: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:11:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:57.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:57.530132+0000) 2022-09-08T06:11:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:57.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:57.530140+0000) 2022-09-08T06:11:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:58.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:58.530391+0000) 2022-09-08T06:11:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:58.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:58.530391+0000) 2022-09-08T06:11:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:11:58 smithi055 ceph-mon[97033]: pgmap v16646: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:11:58 smithi059 ceph-mon[97450]: pgmap v16646: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:11:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:11:59.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:59.530665+0000) 2022-09-08T06:11:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:11:59.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:11:59.530665+0000) 2022-09-08T06:12:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:00.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:00.530900+0000) 2022-09-08T06:12:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:00.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:00.530900+0000) 2022-09-08T06:12:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:00 smithi055 ceph-mon[97033]: pgmap v16647: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:12:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:00 smithi059 ceph-mon[97450]: pgmap v16647: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:12:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:01.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:01.531152+0000) 2022-09-08T06:12:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:01.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:01.531152+0000) 2022-09-08T06:12:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:02 smithi055 ceph-mon[97033]: pgmap v16648: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:12:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:02 smithi059 ceph-mon[97450]: pgmap v16648: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:12:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:02.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:02.531444+0000) 2022-09-08T06:12:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:02.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:02.531444+0000) 2022-09-08T06:12:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:03.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:03.531715+0000) 2022-09-08T06:12:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:03.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:03.531710+0000) 2022-09-08T06:12:04.852 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:04.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:04.531992+0000) 2022-09-08T06:12:04.853 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:04.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:04.531992+0000) 2022-09-08T06:12:05.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:12:04.848+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:12:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:04 smithi055 ceph-mon[97033]: pgmap v16649: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:12:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:04 smithi059 ceph-mon[97450]: pgmap v16649: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:12:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:05.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:05.532250+0000) 2022-09-08T06:12:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:05.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:05.532222+0000) 2022-09-08T06:12:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:12:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:12:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:06.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:06.532421+0000) 2022-09-08T06:12:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:06.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:06.532393+0000) 2022-09-08T06:12:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:06 smithi055 ceph-mon[97033]: pgmap v16650: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:12:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:12:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:06 smithi059 ceph-mon[97450]: pgmap v16650: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:12:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:12:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:07.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:07.532592+0000) 2022-09-08T06:12:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:07.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:07.532589+0000) 2022-09-08T06:12:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:08.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:08.532790+0000) 2022-09-08T06:12:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:08.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:08.532790+0000) 2022-09-08T06:12:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:08 smithi055 ceph-mon[97033]: pgmap v16651: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:12:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:08 smithi059 ceph-mon[97450]: pgmap v16651: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:12:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:09.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:09.532935+0000) 2022-09-08T06:12:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:09.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:09.532903+0000) 2022-09-08T06:12:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:10.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:10.533170+0000) 2022-09-08T06:12:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:10.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:10.533119+0000) 2022-09-08T06:12:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:10 smithi055 ceph-mon[97033]: pgmap v16652: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T06:12:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:10 smithi055 ceph-mon[97033]: osdmap e1120: 12 total, 11 up, 11 in 2022-09-08T06:12:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:10 smithi059 ceph-mon[97450]: pgmap v16652: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T06:12:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:10 smithi059 ceph-mon[97450]: osdmap e1120: 12 total, 11 up, 11 in 2022-09-08T06:12:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:11.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:11.533305+0000) 2022-09-08T06:12:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:11.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:11.533366+0000) 2022-09-08T06:12:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:12 smithi055 ceph-mon[97033]: pgmap v16654: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:12:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:12 smithi059 ceph-mon[97450]: pgmap v16654: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:12:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:12.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:12.533579+0000) 2022-09-08T06:12:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:12.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:12.533482+0000) 2022-09-08T06:12:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:13.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:13.533747+0000) 2022-09-08T06:12:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:13.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:13.533841+0000) 2022-09-08T06:12:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:14.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:14.533943+0000) 2022-09-08T06:12:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:14.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:14.534065+0000) 2022-09-08T06:12:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:14 smithi055 ceph-mon[97033]: pgmap v16655: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:12:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:14 smithi059 ceph-mon[97450]: pgmap v16655: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:12:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:15.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:15.534237+0000) 2022-09-08T06:12:15.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:15.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:15.534138+0000) 2022-09-08T06:12:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:16.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:16.534295+0000) 2022-09-08T06:12:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:16.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:16.534380+0000) 2022-09-08T06:12:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:16 smithi055 ceph-mon[97033]: pgmap v16656: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:12:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:16 smithi055 ceph-mon[97033]: osdmap e1121: 12 total, 11 up, 11 in 2022-09-08T06:12:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:16 smithi059 ceph-mon[97450]: pgmap v16656: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:12:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:16 smithi059 ceph-mon[97450]: osdmap e1121: 12 total, 11 up, 11 in 2022-09-08T06:12:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:17.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:17.534469+0000) 2022-09-08T06:12:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:17.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:17.534523+0000) 2022-09-08T06:12:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:18.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:18.534777+0000) 2022-09-08T06:12:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:18.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:18.534781+0000) 2022-09-08T06:12:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:18 smithi055 ceph-mon[97033]: pgmap v16658: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-08T06:12:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:12:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:12:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:12:19.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:12:19.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:12:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:18 smithi059 ceph-mon[97450]: pgmap v16658: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-08T06:12:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:12:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:12:19.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:12:19.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:12:19.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:12:19.857 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:19.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:19.534998+0000) 2022-09-08T06:12:19.858 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:19.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:19.534997+0000) 2022-09-08T06:12:20.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:12:19.853+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:12:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:20.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:20.535222+0000) 2022-09-08T06:12:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:20.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:20.535220+0000) 2022-09-08T06:12:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:20 smithi059 ceph-mon[97450]: pgmap v16659: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 103 B/s wr, 5 op/s 2022-09-08T06:12:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:12:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:12:21.158 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:20 smithi055 ceph-mon[97033]: pgmap v16659: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 103 B/s wr, 5 op/s 2022-09-08T06:12:21.158 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:12:21.159 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:12:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:21.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:21.535442+0000) 2022-09-08T06:12:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:21.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:21.535439+0000) 2022-09-08T06:12:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:12:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:12:22.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 196.0M 2022-09-08T06:12:22.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 205537054: error parsing value: Value '205537054' is below minimum 939524096 2022-09-08T06:12:22.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T06:12:22.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T06:12:22.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:22 smithi055 ceph-mon[97033]: pgmap v16660: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:12:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:12:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:12:22.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 196.0M 2022-09-08T06:12:22.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 205537054: error parsing value: Value '205537054' is below minimum 939524096 2022-09-08T06:12:22.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:12:22.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T06:12:22.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T06:12:22.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:22 smithi059 ceph-mon[97450]: pgmap v16660: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:12:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:22.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:22.535724+0000) 2022-09-08T06:12:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:22.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:22.535730+0000) 2022-09-08T06:12:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:23.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:23.535902+0000) 2022-09-08T06:12:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:23.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:23.535902+0000) 2022-09-08T06:12:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:24.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:24.536144+0000) 2022-09-08T06:12:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:24.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:24.536145+0000) 2022-09-08T06:12:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:24 smithi055 ceph-mon[97033]: pgmap v16661: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:12:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:24 smithi059 ceph-mon[97450]: pgmap v16661: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:12:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:25.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:25.536424+0000) 2022-09-08T06:12:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:25.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:25.536368+0000) 2022-09-08T06:12:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:26.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:26.536610+0000) 2022-09-08T06:12:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:26.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:26.536617+0000) 2022-09-08T06:12:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:26 smithi055 ceph-mon[97033]: pgmap v16662: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:12:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:26 smithi059 ceph-mon[97450]: pgmap v16662: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:12:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:27.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:27.536776+0000) 2022-09-08T06:12:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:27.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:27.536777+0000) 2022-09-08T06:12:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:28.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:28.537110+0000) 2022-09-08T06:12:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:28.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:28.537042+0000) 2022-09-08T06:12:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:28 smithi055 ceph-mon[97033]: pgmap v16663: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T06:12:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:28 smithi059 ceph-mon[97450]: pgmap v16663: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T06:12:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:29.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:29.537282+0000) 2022-09-08T06:12:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:29.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:29.537221+0000) 2022-09-08T06:12:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:30.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:30.537452+0000) 2022-09-08T06:12:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:30.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:30.537423+0000) 2022-09-08T06:12:31.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:30 smithi055 ceph-mon[97033]: pgmap v16664: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T06:12:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:30 smithi059 ceph-mon[97450]: pgmap v16664: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T06:12:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:31.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:31.537627+0000) 2022-09-08T06:12:31.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:31.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:31.537720+0000) 2022-09-08T06:12:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:32 smithi055 ceph-mon[97033]: pgmap v16665: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T06:12:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:32 smithi059 ceph-mon[97450]: pgmap v16665: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T06:12:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:32.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:32.537872+0000) 2022-09-08T06:12:32.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:32.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:32.537921+0000) 2022-09-08T06:12:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:33.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:33.538167+0000) 2022-09-08T06:12:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:33.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:33.538110+0000) 2022-09-08T06:12:34.870 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:34.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:34.538362+0000) 2022-09-08T06:12:34.871 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:34.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:34.538347+0000) 2022-09-08T06:12:35.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:12:34.867+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:12:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:34 smithi055 ceph-mon[97033]: pgmap v16666: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:12:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:34 smithi059 ceph-mon[97450]: pgmap v16666: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:12:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:35.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:35.538598+0000) 2022-09-08T06:12:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:35.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:35.538574+0000) 2022-09-08T06:12:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:36.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:36.538916+0000) 2022-09-08T06:12:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:36.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:36.538894+0000) 2022-09-08T06:12:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:36 smithi055 ceph-mon[97033]: pgmap v16667: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:12:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:36 smithi059 ceph-mon[97450]: pgmap v16667: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:12:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:37.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:37.539166+0000) 2022-09-08T06:12:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:37.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:37.539126+0000) 2022-09-08T06:12:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:38.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:38.539411+0000) 2022-09-08T06:12:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:38.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:38.539411+0000) 2022-09-08T06:12:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:38 smithi055 ceph-mon[97033]: pgmap v16668: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:12:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:38 smithi059 ceph-mon[97450]: pgmap v16668: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:12:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:39.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:39.539601+0000) 2022-09-08T06:12:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:39.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:39.539602+0000) 2022-09-08T06:12:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:40.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:40.539856+0000) 2022-09-08T06:12:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:40.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:40.539856+0000) 2022-09-08T06:12:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:40 smithi055 ceph-mon[97033]: pgmap v16669: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T06:12:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:40 smithi059 ceph-mon[97450]: pgmap v16669: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T06:12:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:41.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:41.540045+0000) 2022-09-08T06:12:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:41.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:41.540044+0000) 2022-09-08T06:12:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:42 smithi055 ceph-mon[97033]: pgmap v16670: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T06:12:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:42 smithi059 ceph-mon[97450]: pgmap v16670: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T06:12:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:42.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:42.540292+0000) 2022-09-08T06:12:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:42.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:42.540290+0000) 2022-09-08T06:12:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:43.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:43.540521+0000) 2022-09-08T06:12:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:43.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:43.540520+0000) 2022-09-08T06:12:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:44.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:44.540774+0000) 2022-09-08T06:12:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:44.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:44.540771+0000) 2022-09-08T06:12:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:44 smithi055 ceph-mon[97033]: pgmap v16671: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:12:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:44 smithi059 ceph-mon[97450]: pgmap v16671: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:12:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:45.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:45.541005+0000) 2022-09-08T06:12:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:45.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:45.541024+0000) 2022-09-08T06:12:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:46.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:46.541241+0000) 2022-09-08T06:12:46.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:46.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:46.541242+0000) 2022-09-08T06:12:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:46 smithi055 ceph-mon[97033]: pgmap v16672: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:12:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:46 smithi059 ceph-mon[97450]: pgmap v16672: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:12:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:47.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:47.541460+0000) 2022-09-08T06:12:47.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:47.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:47.541460+0000) 2022-09-08T06:12:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:48.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:48.541689+0000) 2022-09-08T06:12:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:48.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:48.541689+0000) 2022-09-08T06:12:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:49 smithi055 ceph-mon[97033]: pgmap v16673: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:12:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:49 smithi055 ceph-mon[97033]: pgmap v16674: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:12:49.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:49 smithi055 ceph-mon[97033]: pgmap v16675: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-08T06:12:49.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:12:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:49 smithi059 ceph-mon[97450]: pgmap v16673: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:12:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:49 smithi059 ceph-mon[97450]: pgmap v16674: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:12:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:49 smithi059 ceph-mon[97450]: pgmap v16675: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-08T06:12:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:12:49.876 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:49.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:49.541908+0000) 2022-09-08T06:12:49.877 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:49.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:49.541908+0000) 2022-09-08T06:12:50.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:12:49.873+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:12:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:50.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:50.542167+0000) 2022-09-08T06:12:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:50.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:50.542167+0000) 2022-09-08T06:12:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:51 smithi055 ceph-mon[97033]: pgmap v16676: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:12:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:51 smithi059 ceph-mon[97450]: pgmap v16676: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:12:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:51.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:51.542430+0000) 2022-09-08T06:12:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:51.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:51.542430+0000) 2022-09-08T06:12:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:52 smithi055 ceph-mon[97033]: pgmap v16677: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:12:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:52 smithi059 ceph-mon[97450]: pgmap v16677: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:12:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:52.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:52.542632+0000) 2022-09-08T06:12:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:52.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:52.542649+0000) 2022-09-08T06:12:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:53.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:53.542856+0000) 2022-09-08T06:12:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:53.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:53.542856+0000) 2022-09-08T06:12:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:54.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:54.543111+0000) 2022-09-08T06:12:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:54.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:54.543111+0000) 2022-09-08T06:12:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:55 smithi055 ceph-mon[97033]: pgmap v16678: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:12:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:55 smithi059 ceph-mon[97450]: pgmap v16678: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:12:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:55.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:55.543344+0000) 2022-09-08T06:12:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:55.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:55.543344+0000) 2022-09-08T06:12:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:56.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:56.543610+0000) 2022-09-08T06:12:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:56.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:56.543609+0000) 2022-09-08T06:12:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:57 smithi055 ceph-mon[97033]: pgmap v16679: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:12:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:57 smithi059 ceph-mon[97450]: pgmap v16679: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:12:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:57.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:57.543945+0000) 2022-09-08T06:12:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:57.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:57.543949+0000) 2022-09-08T06:12:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:58.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:58.544198+0000) 2022-09-08T06:12:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:58.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:58.544205+0000) 2022-09-08T06:12:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:12:59 smithi055 ceph-mon[97033]: pgmap v16680: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:12:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:12:59 smithi059 ceph-mon[97450]: pgmap v16680: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:12:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:12:59.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:59.544441+0000) 2022-09-08T06:12:59.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:12:59.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:12:59.544441+0000) 2022-09-08T06:13:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:00 smithi055 ceph-mon[97033]: pgmap v16681: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:13:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:00 smithi059 ceph-mon[97450]: pgmap v16681: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:13:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:00.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:00.544756+0000) 2022-09-08T06:13:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:00.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:00.544755+0000) 2022-09-08T06:13:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:01.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:01.544928+0000) 2022-09-08T06:13:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:01.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:01.544930+0000) 2022-09-08T06:13:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:02 smithi055 ceph-mon[97033]: pgmap v16682: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:02 smithi059 ceph-mon[97450]: pgmap v16682: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:02.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:02.545204+0000) 2022-09-08T06:13:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:02.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:02.545204+0000) 2022-09-08T06:13:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:03.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:03.545407+0000) 2022-09-08T06:13:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:03.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:03.545410+0000) 2022-09-08T06:13:04.885 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:04.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:04.545716+0000) 2022-09-08T06:13:04.885 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:04.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:04.545717+0000) 2022-09-08T06:13:05.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:13:04.881+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:13:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:05 smithi055 ceph-mon[97033]: pgmap v16683: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:05.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:05 smithi059 ceph-mon[97450]: pgmap v16683: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:05.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:05.545911+0000) 2022-09-08T06:13:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:05.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:05.545916+0000) 2022-09-08T06:13:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:13:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:13:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:06.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:06.546057+0000) 2022-09-08T06:13:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:06.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:06.546057+0000) 2022-09-08T06:13:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:07 smithi055 ceph-mon[97033]: pgmap v16684: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:13:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:13:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:07 smithi059 ceph-mon[97450]: pgmap v16684: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:13:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:13:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:07.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:07.546327+0000) 2022-09-08T06:13:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:07.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:07.546327+0000) 2022-09-08T06:13:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:08 smithi055 ceph-mon[97033]: pgmap v16685: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:13:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:08 smithi059 ceph-mon[97450]: pgmap v16685: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:13:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:08.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:08.546582+0000) 2022-09-08T06:13:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:08.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:08.546567+0000) 2022-09-08T06:13:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:09.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:09.546849+0000) 2022-09-08T06:13:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:09.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:09.546846+0000) 2022-09-08T06:13:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:10 smithi055 ceph-mon[97033]: osdmap e1122: 12 total, 11 up, 11 in 2022-09-08T06:13:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:10 smithi055 ceph-mon[97033]: pgmap v16687: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:13:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:10 smithi059 ceph-mon[97450]: osdmap e1122: 12 total, 11 up, 11 in 2022-09-08T06:13:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:10 smithi059 ceph-mon[97450]: pgmap v16687: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:13:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:10.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:10.547147+0000) 2022-09-08T06:13:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:10.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:10.547163+0000) 2022-09-08T06:13:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:11.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:11.547399+0000) 2022-09-08T06:13:11.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:11.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:11.547395+0000) 2022-09-08T06:13:12.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:12 smithi055 ceph-mon[97033]: pgmap v16688: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:13:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:12 smithi059 ceph-mon[97450]: pgmap v16688: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:13:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:12.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:12.547702+0000) 2022-09-08T06:13:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:12.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:12.547711+0000) 2022-09-08T06:13:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:13.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:13.548000+0000) 2022-09-08T06:13:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:13.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:13.548003+0000) 2022-09-08T06:13:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:14.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:14.548280+0000) 2022-09-08T06:13:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:14.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:14.548280+0000) 2022-09-08T06:13:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:15 smithi055 ceph-mon[97033]: pgmap v16689: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:13:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:15 smithi059 ceph-mon[97450]: pgmap v16689: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:13:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:15.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:15.548545+0000) 2022-09-08T06:13:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:15.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:15.548545+0000) 2022-09-08T06:13:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:16.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:16.548815+0000) 2022-09-08T06:13:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:16.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:16.548815+0000) 2022-09-08T06:13:17.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:17 smithi055 ceph-mon[97033]: pgmap v16690: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:13:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:17 smithi055 ceph-mon[97033]: osdmap e1123: 12 total, 11 up, 11 in 2022-09-08T06:13:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:17 smithi059 ceph-mon[97450]: pgmap v16690: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:13:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:17 smithi059 ceph-mon[97450]: osdmap e1123: 12 total, 11 up, 11 in 2022-09-08T06:13:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:17.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:17.549080+0000) 2022-09-08T06:13:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:17.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:17.549072+0000) 2022-09-08T06:13:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:18 smithi055 ceph-mon[97033]: pgmap v16692: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T06:13:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:18 smithi059 ceph-mon[97450]: pgmap v16692: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T06:13:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:18.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:18.549306+0000) 2022-09-08T06:13:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:18.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:18.549306+0000) 2022-09-08T06:13:19.899 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:19.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:19.549510+0000) 2022-09-08T06:13:19.900 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:19.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:19.549510+0000) 2022-09-08T06:13:20.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:13:19.896+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:13:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:20.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:20.549814+0000) 2022-09-08T06:13:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:20.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:20.549814+0000) 2022-09-08T06:13:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:21 smithi055 ceph-mon[97033]: pgmap v16693: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:13:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:21 smithi059 ceph-mon[97450]: pgmap v16693: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:13:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:21.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:21.550038+0000) 2022-09-08T06:13:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:21.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:21.550038+0000) 2022-09-08T06:13:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:22 smithi055 ceph-mon[97033]: pgmap v16694: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:13:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:22 smithi059 ceph-mon[97450]: pgmap v16694: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:13:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:22.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:22.550260+0000) 2022-09-08T06:13:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:22.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:22.550260+0000) 2022-09-08T06:13:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:23.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:23.550518+0000) 2022-09-08T06:13:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:23.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:23.550518+0000) 2022-09-08T06:13:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:24.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:24.550772+0000) 2022-09-08T06:13:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:24.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:24.550773+0000) 2022-09-08T06:13:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:25 smithi055 ceph-mon[97033]: pgmap v16695: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:13:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:25 smithi059 ceph-mon[97450]: pgmap v16695: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:13:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:25.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:25.550993+0000) 2022-09-08T06:13:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:25.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:25.550991+0000) 2022-09-08T06:13:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:26 smithi055 ceph-mon[97033]: pgmap v16696: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:13:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:26 smithi059 ceph-mon[97450]: pgmap v16696: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:13:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:26.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:26.551190+0000) 2022-09-08T06:13:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:26.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:26.551190+0000) 2022-09-08T06:13:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:27.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:27.551462+0000) 2022-09-08T06:13:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:27.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:27.551462+0000) 2022-09-08T06:13:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:28.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:28.551772+0000) 2022-09-08T06:13:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:28.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:28.551774+0000) 2022-09-08T06:13:29.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:29 smithi055 ceph-mon[97033]: pgmap v16697: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T06:13:29.539 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:29 smithi059 ceph-mon[97450]: pgmap v16697: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T06:13:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:29.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:29.552002+0000) 2022-09-08T06:13:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:29.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:29.552001+0000) 2022-09-08T06:13:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:30 smithi055 ceph-mon[97033]: pgmap v16698: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:13:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:30 smithi059 ceph-mon[97450]: pgmap v16698: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:13:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:30.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:30.552301+0000) 2022-09-08T06:13:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:30.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:30.552306+0000) 2022-09-08T06:13:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:31.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:31.552581+0000) 2022-09-08T06:13:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:31.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:31.552580+0000) 2022-09-08T06:13:32.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:32 smithi055 ceph-mon[97033]: pgmap v16699: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:32.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:32 smithi059 ceph-mon[97450]: pgmap v16699: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:32.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:32.552846+0000) 2022-09-08T06:13:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:32.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:32.552846+0000) 2022-09-08T06:13:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:33.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:33.553113+0000) 2022-09-08T06:13:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:33.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:33.553126+0000) 2022-09-08T06:13:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:34.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:34.553302+0000) 2022-09-08T06:13:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:34.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:34.553298+0000) 2022-09-08T06:13:35.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:13:34.912+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:13:35.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:35 smithi055 ceph-mon[97033]: pgmap v16700: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:35 smithi059 ceph-mon[97450]: pgmap v16700: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:35.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:35.553519+0000) 2022-09-08T06:13:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:35.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:35.553519+0000) 2022-09-08T06:13:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:36 smithi055 ceph-mon[97033]: pgmap v16701: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:13:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:36 smithi059 ceph-mon[97450]: pgmap v16701: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:13:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:36.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:36.553785+0000) 2022-09-08T06:13:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:36.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:36.553782+0000) 2022-09-08T06:13:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:37.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:37.553982+0000) 2022-09-08T06:13:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:37.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:37.553971+0000) 2022-09-08T06:13:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:38.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:38.554202+0000) 2022-09-08T06:13:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:38.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:38.554203+0000) 2022-09-08T06:13:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:39 smithi055 ceph-mon[97033]: pgmap v16702: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:39 smithi059 ceph-mon[97450]: pgmap v16702: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:39.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:39.554338+0000) 2022-09-08T06:13:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:39.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:39.554337+0000) 2022-09-08T06:13:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:40 smithi055 ceph-mon[97033]: pgmap v16703: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:13:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:40 smithi059 ceph-mon[97450]: pgmap v16703: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:13:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:40.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:40.554612+0000) 2022-09-08T06:13:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:40.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:40.554612+0000) 2022-09-08T06:13:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:41.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:41.554848+0000) 2022-09-08T06:13:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:41.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:41.554854+0000) 2022-09-08T06:13:42.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:42 smithi055 ceph-mon[97033]: pgmap v16704: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:13:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:42 smithi059 ceph-mon[97450]: pgmap v16704: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:13:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:42.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:42.555130+0000) 2022-09-08T06:13:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:42.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:42.555129+0000) 2022-09-08T06:13:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:43.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:43.555343+0000) 2022-09-08T06:13:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:43.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:43.555343+0000) 2022-09-08T06:13:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:44 smithi055 ceph-mon[97033]: pgmap v16705: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:13:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:44 smithi059 ceph-mon[97450]: pgmap v16705: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:13:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:44.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:44.555573+0000) 2022-09-08T06:13:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:44.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:44.555572+0000) 2022-09-08T06:13:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:45.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:45.555758+0000) 2022-09-08T06:13:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:45.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:45.555758+0000) 2022-09-08T06:13:46.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:46 smithi055 ceph-mon[97033]: pgmap v16706: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:13:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:46 smithi059 ceph-mon[97450]: pgmap v16706: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:13:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:46.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:46.555945+0000) 2022-09-08T06:13:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:46.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:46.555950+0000) 2022-09-08T06:13:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:47.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:47.556164+0000) 2022-09-08T06:13:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:47.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:47.556164+0000) 2022-09-08T06:13:48.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:48 smithi055 ceph-mon[97033]: pgmap v16707: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:13:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:48 smithi059 ceph-mon[97450]: pgmap v16707: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:13:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:48.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:48.556417+0000) 2022-09-08T06:13:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:48.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:48.556415+0000) 2022-09-08T06:13:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:13:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:13:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:13:49.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:13:49.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:13:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:13:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:13:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:13:49.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:13:49.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:13:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:49.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:49.556612+0000) 2022-09-08T06:13:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:49.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:49.556612+0000) 2022-09-08T06:13:50.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:13:49.915+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:13:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:50 smithi055 ceph-mon[97033]: pgmap v16708: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:13:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:50 smithi059 ceph-mon[97450]: pgmap v16708: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:13:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:50.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:50.556819+0000) 2022-09-08T06:13:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:50.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:50.556849+0000) 2022-09-08T06:13:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:51.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:51.557016+0000) 2022-09-08T06:13:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:51.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:51.557093+0000) 2022-09-08T06:13:52.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:52 smithi055 ceph-mon[97033]: pgmap v16709: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:52 smithi059 ceph-mon[97450]: pgmap v16709: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:52.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:52.557221+0000) 2022-09-08T06:13:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:52.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:52.557275+0000) 2022-09-08T06:13:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:53.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:53.557426+0000) 2022-09-08T06:13:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:53.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:53.557513+0000) 2022-09-08T06:13:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:54 smithi055 ceph-mon[97033]: pgmap v16710: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:54 smithi059 ceph-mon[97450]: pgmap v16710: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:54.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:54.557767+0000) 2022-09-08T06:13:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:54.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:54.557728+0000) 2022-09-08T06:13:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:55.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:55.557955+0000) 2022-09-08T06:13:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:55.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:55.557963+0000) 2022-09-08T06:13:56.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:56 smithi055 ceph-mon[97033]: pgmap v16711: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:13:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:56 smithi059 ceph-mon[97450]: pgmap v16711: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:13:56.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:56.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:56.558143+0000) 2022-09-08T06:13:56.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:56.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:56.558174+0000) 2022-09-08T06:13:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:57.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:57.558390+0000) 2022-09-08T06:13:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:57.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:57.558454+0000) 2022-09-08T06:13:58.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:13:58 smithi055 ceph-mon[97033]: pgmap v16712: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:13:58 smithi059 ceph-mon[97450]: pgmap v16712: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:13:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:58.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:58.558729+0000) 2022-09-08T06:13:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:58.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:58.558670+0000) 2022-09-08T06:13:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:13:59.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:59.558902+0000) 2022-09-08T06:13:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:13:59.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:13:59.558904+0000) 2022-09-08T06:14:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:00 smithi055 ceph-mon[97033]: pgmap v16713: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:14:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:00 smithi059 ceph-mon[97450]: pgmap v16713: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:14:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:00.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:00.559117+0000) 2022-09-08T06:14:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:00.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:00.559098+0000) 2022-09-08T06:14:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:01.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:01.559368+0000) 2022-09-08T06:14:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:01.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:01.559368+0000) 2022-09-08T06:14:02.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:02 smithi055 ceph-mon[97033]: pgmap v16714: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:02 smithi059 ceph-mon[97450]: pgmap v16714: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:02.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:02.559551+0000) 2022-09-08T06:14:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:02.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:02.559556+0000) 2022-09-08T06:14:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:03.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:03.559758+0000) 2022-09-08T06:14:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:03.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:03.559774+0000) 2022-09-08T06:14:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:04 smithi055 ceph-mon[97033]: pgmap v16715: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:04 smithi059 ceph-mon[97450]: pgmap v16715: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:04.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:04.559942+0000) 2022-09-08T06:14:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:04.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:04.559942+0000) 2022-09-08T06:14:05.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:14:04.918+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:14:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:05.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:05.560151+0000) 2022-09-08T06:14:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:05.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:05.560151+0000) 2022-09-08T06:14:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:14:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:14:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:06.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:06.560385+0000) 2022-09-08T06:14:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:06.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:06.560384+0000) 2022-09-08T06:14:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:06 smithi055 ceph-mon[97033]: pgmap v16716: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:14:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:14:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:06 smithi059 ceph-mon[97450]: pgmap v16716: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:14:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:14:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:07.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:07.560682+0000) 2022-09-08T06:14:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:07.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:07.560680+0000) 2022-09-08T06:14:08.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:08 smithi055 ceph-mon[97033]: pgmap v16717: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:14:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:08 smithi059 ceph-mon[97450]: pgmap v16717: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:14:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:08.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:08.560858+0000) 2022-09-08T06:14:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:08.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:08.560860+0000) 2022-09-08T06:14:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:09.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:09.561089+0000) 2022-09-08T06:14:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:09.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:09.561088+0000) 2022-09-08T06:14:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:10 smithi055 ceph-mon[97033]: osdmap e1124: 12 total, 11 up, 11 in 2022-09-08T06:14:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:10 smithi055 ceph-mon[97033]: pgmap v16719: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:14:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:10 smithi059 ceph-mon[97450]: osdmap e1124: 12 total, 11 up, 11 in 2022-09-08T06:14:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:10 smithi059 ceph-mon[97450]: pgmap v16719: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:14:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:10.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:10.561281+0000) 2022-09-08T06:14:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:10.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:10.561294+0000) 2022-09-08T06:14:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:11.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:11.561504+0000) 2022-09-08T06:14:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:11.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:11.561502+0000) 2022-09-08T06:14:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:12 smithi055 ceph-mon[97033]: osdmap e1125: 12 total, 11 up, 11 in 2022-09-08T06:14:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:12 smithi055 ceph-mon[97033]: pgmap v16721: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:14:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:12 smithi059 ceph-mon[97450]: osdmap e1125: 12 total, 11 up, 11 in 2022-09-08T06:14:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:12 smithi059 ceph-mon[97450]: pgmap v16721: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:14:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:12.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:12.561724+0000) 2022-09-08T06:14:12.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:12.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:12.561722+0000) 2022-09-08T06:14:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:13.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:13.562013+0000) 2022-09-08T06:14:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:13.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:13.561991+0000) 2022-09-08T06:14:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:14 smithi055 ceph-mon[97033]: pgmap v16722: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:14:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:14 smithi059 ceph-mon[97450]: pgmap v16722: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:14:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:14.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:14.562232+0000) 2022-09-08T06:14:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:14.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:14.562240+0000) 2022-09-08T06:14:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:15.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:15.562450+0000) 2022-09-08T06:14:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:15.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:15.562458+0000) 2022-09-08T06:14:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:16 smithi055 ceph-mon[97033]: pgmap v16723: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:14:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:16 smithi059 ceph-mon[97450]: pgmap v16723: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:14:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:16.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:16.562700+0000) 2022-09-08T06:14:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:16.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:16.562706+0000) 2022-09-08T06:14:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:17.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:17.562937+0000) 2022-09-08T06:14:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:17.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:17.562951+0000) 2022-09-08T06:14:18.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:18 smithi055 ceph-mon[97033]: pgmap v16724: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T06:14:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:18 smithi059 ceph-mon[97450]: pgmap v16724: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2022-09-08T06:14:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:18.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:18.563199+0000) 2022-09-08T06:14:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:18.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:18.563151+0000) 2022-09-08T06:14:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:19 smithi055 ceph-mon[97033]: pgmap v16725: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 124 B/s wr, 3 op/s 2022-09-08T06:14:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:19 smithi055 ceph-mon[97033]: pgmap v16726: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 149 B/s wr, 4 op/s 2022-09-08T06:14:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:14:19.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:19 smithi059 ceph-mon[97450]: pgmap v16725: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 124 B/s wr, 3 op/s 2022-09-08T06:14:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:19 smithi059 ceph-mon[97450]: pgmap v16726: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 149 B/s wr, 4 op/s 2022-09-08T06:14:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:14:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:19.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:19.563399+0000) 2022-09-08T06:14:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:19.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:19.563365+0000) 2022-09-08T06:14:20.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:14:19.934+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:14:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:20.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:20.563617+0000) 2022-09-08T06:14:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:20.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:20.563629+0000) 2022-09-08T06:14:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:21 smithi055 ceph-mon[97033]: pgmap v16727: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T06:14:21.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:21 smithi059 ceph-mon[97450]: pgmap v16727: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T06:14:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:21.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:21.563821+0000) 2022-09-08T06:14:21.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:21.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:21.563862+0000) 2022-09-08T06:14:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:22.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:22.563962+0000) 2022-09-08T06:14:22.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:22.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:22.564003+0000) 2022-09-08T06:14:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:23 smithi055 ceph-mon[97033]: pgmap v16728: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T06:14:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:23 smithi059 ceph-mon[97450]: pgmap v16728: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T06:14:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:23.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:23.564163+0000) 2022-09-08T06:14:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:23.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:23.564129+0000) 2022-09-08T06:14:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:24.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:24.564297+0000) 2022-09-08T06:14:24.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:24.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:24.564393+0000) 2022-09-08T06:14:25.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:25 smithi055 ceph-mon[97033]: pgmap v16729: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:14:25.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:25 smithi059 ceph-mon[97450]: pgmap v16729: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:14:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:25.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:25.564605+0000) 2022-09-08T06:14:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:25.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:25.564536+0000) 2022-09-08T06:14:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:26.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:26.564808+0000) 2022-09-08T06:14:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:26.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:26.564772+0000) 2022-09-08T06:14:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:27 smithi055 ceph-mon[97033]: pgmap v16730: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:14:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:27 smithi059 ceph-mon[97450]: pgmap v16730: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:14:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:27.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:27.564987+0000) 2022-09-08T06:14:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:27.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:27.564986+0000) 2022-09-08T06:14:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:28.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:28.565160+0000) 2022-09-08T06:14:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:28.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:28.565140+0000) 2022-09-08T06:14:29.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:29 smithi059 ceph-mon[97450]: pgmap v16731: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:14:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:29.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:29.565375+0000) 2022-09-08T06:14:29.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:29.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:29.565335+0000) 2022-09-08T06:14:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:29 smithi055 ceph-mon[97033]: pgmap v16731: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:14:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:30.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:30.565665+0000) 2022-09-08T06:14:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:30.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:30.565618+0000) 2022-09-08T06:14:31.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:31 smithi059 ceph-mon[97450]: pgmap v16732: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:14:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:31.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:31.565875+0000) 2022-09-08T06:14:31.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:31.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:31.565927+0000) 2022-09-08T06:14:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:31 smithi055 ceph-mon[97033]: pgmap v16732: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:14:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:32.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:32.566118+0000) 2022-09-08T06:14:32.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:32.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:32.566161+0000) 2022-09-08T06:14:33.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:33 smithi059 ceph-mon[97450]: pgmap v16733: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:33.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:33.566359+0000) 2022-09-08T06:14:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:33.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:33.566454+0000) 2022-09-08T06:14:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:33 smithi055 ceph-mon[97033]: pgmap v16733: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:34.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:34.566778+0000) 2022-09-08T06:14:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:34.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:34.566619+0000) 2022-09-08T06:14:35.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:14:34.935+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:14:35.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:35 smithi059 ceph-mon[97450]: pgmap v16734: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:35.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:35.566863+0000) 2022-09-08T06:14:35.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:35.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:35.566959+0000) 2022-09-08T06:14:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:35 smithi055 ceph-mon[97033]: pgmap v16734: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:36.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:36.567016+0000) 2022-09-08T06:14:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:36.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:36.567111+0000) 2022-09-08T06:14:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:37 smithi059 ceph-mon[97450]: pgmap v16735: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:14:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:37.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:37.567353+0000) 2022-09-08T06:14:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:37.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:37.567253+0000) 2022-09-08T06:14:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:37 smithi055 ceph-mon[97033]: pgmap v16735: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:14:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:38.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:38.567588+0000) 2022-09-08T06:14:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:38.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:38.567499+0000) 2022-09-08T06:14:39.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:39 smithi059 ceph-mon[97450]: pgmap v16736: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:39.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:39.567889+0000) 2022-09-08T06:14:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:39.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:39.567758+0000) 2022-09-08T06:14:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:39 smithi055 ceph-mon[97033]: pgmap v16736: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:40.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:40.568154+0000) 2022-09-08T06:14:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:40.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:40.567984+0000) 2022-09-08T06:14:41.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:41 smithi059 ceph-mon[97450]: pgmap v16737: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:14:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:41.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:41.568349+0000) 2022-09-08T06:14:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:41.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:41.568184+0000) 2022-09-08T06:14:41.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:41 smithi055 ceph-mon[97033]: pgmap v16737: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:14:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:42.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:42.568581+0000) 2022-09-08T06:14:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:42.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:42.568420+0000) 2022-09-08T06:14:43.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:43 smithi059 ceph-mon[97450]: pgmap v16738: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:43.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:43.568804+0000) 2022-09-08T06:14:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:43.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:43.568619+0000) 2022-09-08T06:14:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:43 smithi055 ceph-mon[97033]: pgmap v16738: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:44.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:44.569011+0000) 2022-09-08T06:14:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:44.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:44.568834+0000) 2022-09-08T06:14:45.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:45 smithi059 ceph-mon[97450]: pgmap v16739: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:45.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:45.569135+0000) 2022-09-08T06:14:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:45.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:45.569039+0000) 2022-09-08T06:14:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:45 smithi055 ceph-mon[97033]: pgmap v16739: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:46.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:46.569258+0000) 2022-09-08T06:14:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:46.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:46.569167+0000) 2022-09-08T06:14:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:47 smithi059 ceph-mon[97450]: pgmap v16740: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:14:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:47.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:47.569457+0000) 2022-09-08T06:14:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:47.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:47.569436+0000) 2022-09-08T06:14:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:47 smithi055 ceph-mon[97033]: pgmap v16740: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:14:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:48.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:48.569714+0000) 2022-09-08T06:14:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:48.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:48.569730+0000) 2022-09-08T06:14:49.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:49 smithi059 ceph-mon[97450]: pgmap v16741: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:49.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:49.569948+0000) 2022-09-08T06:14:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:49.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:49.569978+0000) 2022-09-08T06:14:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:49 smithi055 ceph-mon[97033]: pgmap v16741: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:50.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:14:49.948+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:14:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:50.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:50.570228+0000) 2022-09-08T06:14:50.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:50.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:50.570228+0000) 2022-09-08T06:14:51.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:51 smithi059 ceph-mon[97450]: pgmap v16742: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:14:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:51.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:51.570432+0000) 2022-09-08T06:14:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:51.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:51.570432+0000) 2022-09-08T06:14:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:51 smithi055 ceph-mon[97033]: pgmap v16742: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:14:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:52.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:52.570650+0000) 2022-09-08T06:14:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:52.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:52.570650+0000) 2022-09-08T06:14:53.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:53 smithi059 ceph-mon[97450]: pgmap v16743: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:53.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:53.570766+0000) 2022-09-08T06:14:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:53.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:53.570801+0000) 2022-09-08T06:14:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:53 smithi055 ceph-mon[97033]: pgmap v16743: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:54.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:54.570978+0000) 2022-09-08T06:14:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:54.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:54.570997+0000) 2022-09-08T06:14:55.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:55 smithi059 ceph-mon[97450]: pgmap v16744: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:55.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:55.571192+0000) 2022-09-08T06:14:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:55.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:55.571192+0000) 2022-09-08T06:14:55.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:55 smithi055 ceph-mon[97033]: pgmap v16744: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:56.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:56.571426+0000) 2022-09-08T06:14:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:56.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:56.571426+0000) 2022-09-08T06:14:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:57 smithi059 ceph-mon[97450]: pgmap v16745: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:14:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:57.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:57.571718+0000) 2022-09-08T06:14:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:57.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:57.571727+0000) 2022-09-08T06:14:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:57 smithi055 ceph-mon[97033]: pgmap v16745: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:14:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:58.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:58.571955+0000) 2022-09-08T06:14:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:58.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:58.571955+0000) 2022-09-08T06:14:59.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:14:59 smithi059 ceph-mon[97450]: pgmap v16746: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:14:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:14:59.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:59.572184+0000) 2022-09-08T06:14:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:14:59.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:14:59.572182+0000) 2022-09-08T06:14:59.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:14:59 smithi055 ceph-mon[97033]: pgmap v16746: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:00.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:00.572378+0000) 2022-09-08T06:15:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:00.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:00.572379+0000) 2022-09-08T06:15:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:01.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:01.572528+0000) 2022-09-08T06:15:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:01.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:01.572530+0000) 2022-09-08T06:15:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:01 smithi055 ceph-mon[97033]: pgmap v16747: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:15:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:01 smithi059 ceph-mon[97450]: pgmap v16747: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:15:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:02.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:02.572719+0000) 2022-09-08T06:15:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:02.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:02.572696+0000) 2022-09-08T06:15:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:03.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:03.572882+0000) 2022-09-08T06:15:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:03.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:03.572859+0000) 2022-09-08T06:15:03.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:03 smithi055 ceph-mon[97033]: pgmap v16748: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:04.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:03 smithi059 ceph-mon[97450]: pgmap v16748: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:04.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:04.573093+0000) 2022-09-08T06:15:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:04.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:04.573067+0000) 2022-09-08T06:15:05.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:15:04.961+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:15:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:05.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:05.573290+0000) 2022-09-08T06:15:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:05.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:05.573306+0000) 2022-09-08T06:15:05.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:05 smithi055 ceph-mon[97033]: pgmap v16749: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:06.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:05 smithi059 ceph-mon[97450]: pgmap v16749: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:06.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:06.573530+0000) 2022-09-08T06:15:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:06.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:06.573534+0000) 2022-09-08T06:15:06.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:15:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:15:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:07.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:07.573768+0000) 2022-09-08T06:15:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:07.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:07.573768+0000) 2022-09-08T06:15:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:07 smithi055 ceph-mon[97033]: pgmap v16750: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:15:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:15:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:07 smithi059 ceph-mon[97450]: pgmap v16750: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:15:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:15:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:08.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:08.574082+0000) 2022-09-08T06:15:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:08.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:08.574082+0000) 2022-09-08T06:15:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:09.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:09.574264+0000) 2022-09-08T06:15:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:09.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:09.574262+0000) 2022-09-08T06:15:09.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:09 smithi055 ceph-mon[97033]: pgmap v16751: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:15:10.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:09 smithi059 ceph-mon[97450]: pgmap v16751: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:15:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:10.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:10.574484+0000) 2022-09-08T06:15:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:10.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:10.574501+0000) 2022-09-08T06:15:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:10 smithi055 ceph-mon[97033]: osdmap e1126: 12 total, 11 up, 11 in 2022-09-08T06:15:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:10 smithi059 ceph-mon[97450]: osdmap e1126: 12 total, 11 up, 11 in 2022-09-08T06:15:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:11.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:11.574751+0000) 2022-09-08T06:15:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:11.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:11.574752+0000) 2022-09-08T06:15:11.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:11 smithi055 ceph-mon[97033]: pgmap v16753: 129 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:15:12.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:11 smithi059 ceph-mon[97450]: pgmap v16753: 129 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:15:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:12.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:12.575048+0000) 2022-09-08T06:15:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:12.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:12.575041+0000) 2022-09-08T06:15:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:13.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:13.575242+0000) 2022-09-08T06:15:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:13.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:13.575242+0000) 2022-09-08T06:15:13.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:13 smithi055 ceph-mon[97033]: pgmap v16754: 129 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:15:14.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:13 smithi059 ceph-mon[97450]: pgmap v16754: 129 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:15:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:14.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:14.575488+0000) 2022-09-08T06:15:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:14.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:14.575519+0000) 2022-09-08T06:15:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:15.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:15.575689+0000) 2022-09-08T06:15:15.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:15.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:15.575777+0000) 2022-09-08T06:15:15.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:15 smithi055 ceph-mon[97033]: pgmap v16755: 129 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:15:16.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:15 smithi059 ceph-mon[97450]: pgmap v16755: 129 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:15:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:16.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:16.575955+0000) 2022-09-08T06:15:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:16.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:16.575834+0000) 2022-09-08T06:15:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:17.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:17.576160+0000) 2022-09-08T06:15:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:17.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:17.576082+0000) 2022-09-08T06:15:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:17 smithi055 ceph-mon[97033]: pgmap v16756: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:15:18.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:17 smithi059 ceph-mon[97450]: pgmap v16756: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:15:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:15:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:15:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:15:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:18.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:18.576402+0000) 2022-09-08T06:15:18.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:18.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:18.576298+0000) 2022-09-08T06:15:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:15:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:15:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:15:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:19.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:19.576683+0000) 2022-09-08T06:15:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:19.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:19.576530+0000) 2022-09-08T06:15:19.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:19 smithi055 ceph-mon[97033]: pgmap v16757: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:15:20.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:19 smithi059 ceph-mon[97450]: pgmap v16757: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:15:20.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:15:19.975+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:15:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:20.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:20.576831+0000) 2022-09-08T06:15:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:20.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:20.576735+0000) 2022-09-08T06:15:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:21.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:21.576981+0000) 2022-09-08T06:15:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:21.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:21.576865+0000) 2022-09-08T06:15:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:21 smithi055 ceph-mon[97033]: pgmap v16758: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 190 B/s wr, 8 op/s 2022-09-08T06:15:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:21 smithi055 ceph-mon[97033]: osdmap e1127: 12 total, 11 up, 11 in 2022-09-08T06:15:22.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:21 smithi059 ceph-mon[97450]: pgmap v16758: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.4 KiB/s rd, 190 B/s wr, 8 op/s 2022-09-08T06:15:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:21 smithi059 ceph-mon[97450]: osdmap e1127: 12 total, 11 up, 11 in 2022-09-08T06:15:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:22.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:22.577193+0000) 2022-09-08T06:15:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:22.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:22.577099+0000) 2022-09-08T06:15:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:23.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:23.577338+0000) 2022-09-08T06:15:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:23.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:23.577265+0000) 2022-09-08T06:15:23.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:23 smithi055 ceph-mon[97033]: pgmap v16760: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:15:24.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:23 smithi059 ceph-mon[97450]: pgmap v16760: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:15:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:24.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:24.577513+0000) 2022-09-08T06:15:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:24.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:24.577397+0000) 2022-09-08T06:15:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:25.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:25.577684+0000) 2022-09-08T06:15:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:25.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:25.577551+0000) 2022-09-08T06:15:25.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:25 smithi055 ceph-mon[97033]: pgmap v16761: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:15:26.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:25 smithi059 ceph-mon[97450]: pgmap v16761: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:15:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:26.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:26.577841+0000) 2022-09-08T06:15:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:26.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:26.577708+0000) 2022-09-08T06:15:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:27.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:27.578072+0000) 2022-09-08T06:15:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:27.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:27.577920+0000) 2022-09-08T06:15:27.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:27 smithi055 ceph-mon[97033]: pgmap v16762: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:15:28.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:27 smithi059 ceph-mon[97450]: pgmap v16762: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:15:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:28.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:28.578226+0000) 2022-09-08T06:15:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:28.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:28.578085+0000) 2022-09-08T06:15:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:29.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:29.578471+0000) 2022-09-08T06:15:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:29.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:29.578296+0000) 2022-09-08T06:15:29.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:29 smithi055 ceph-mon[97033]: pgmap v16763: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:15:30.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:29 smithi059 ceph-mon[97450]: pgmap v16763: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:15:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:30.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:30.578657+0000) 2022-09-08T06:15:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:30.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:30.578477+0000) 2022-09-08T06:15:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:31.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:31.578790+0000) 2022-09-08T06:15:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:31.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:31.578654+0000) 2022-09-08T06:15:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:31 smithi055 ceph-mon[97033]: pgmap v16764: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:15:32.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:31 smithi059 ceph-mon[97450]: pgmap v16764: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:15:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:32.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:32.578943+0000) 2022-09-08T06:15:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:32.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:32.578824+0000) 2022-09-08T06:15:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:33.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:33.579106+0000) 2022-09-08T06:15:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:33.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:33.578955+0000) 2022-09-08T06:15:33.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:33 smithi055 ceph-mon[97033]: pgmap v16765: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:15:34.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:33 smithi059 ceph-mon[97450]: pgmap v16765: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:15:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:34.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:34.579294+0000) 2022-09-08T06:15:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:34.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:34.579081+0000) 2022-09-08T06:15:35.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:15:34.991+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:15:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:35.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:35.579447+0000) 2022-09-08T06:15:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:35.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:35.579214+0000) 2022-09-08T06:15:35.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:35 smithi055 ceph-mon[97033]: pgmap v16766: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:36.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:35 smithi059 ceph-mon[97450]: pgmap v16766: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:36.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:36.579607+0000) 2022-09-08T06:15:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:36.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:36.579408+0000) 2022-09-08T06:15:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:37.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:37.579737+0000) 2022-09-08T06:15:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:37.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:37.579554+0000) 2022-09-08T06:15:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:37 smithi055 ceph-mon[97033]: pgmap v16767: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:15:38.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:37 smithi059 ceph-mon[97450]: pgmap v16767: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:15:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:38.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:38.579897+0000) 2022-09-08T06:15:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:38.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:38.579693+0000) 2022-09-08T06:15:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:39.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:39.580036+0000) 2022-09-08T06:15:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:39.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:39.579836+0000) 2022-09-08T06:15:39.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:39 smithi055 ceph-mon[97033]: pgmap v16768: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:40.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:39 smithi059 ceph-mon[97450]: pgmap v16768: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:40.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:40.580161+0000) 2022-09-08T06:15:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:40.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:40.579981+0000) 2022-09-08T06:15:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:41.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:41.580282+0000) 2022-09-08T06:15:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:41.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:41.580120+0000) 2022-09-08T06:15:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:41 smithi055 ceph-mon[97033]: pgmap v16769: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:15:42.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:41 smithi059 ceph-mon[97450]: pgmap v16769: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:15:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:42.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:42.580407+0000) 2022-09-08T06:15:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:42.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:42.580245+0000) 2022-09-08T06:15:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:43.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:43.580540+0000) 2022-09-08T06:15:43.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:43.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:43.580376+0000) 2022-09-08T06:15:43.960 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:43 smithi055 ceph-mon[97033]: pgmap v16770: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:44.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:43 smithi059 ceph-mon[97450]: pgmap v16770: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:44.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:44.580686+0000) 2022-09-08T06:15:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:44.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:44.580508+0000) 2022-09-08T06:15:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:45.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:45.580815+0000) 2022-09-08T06:15:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:45.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:45.580631+0000) 2022-09-08T06:15:45.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:45 smithi055 ceph-mon[97033]: pgmap v16771: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:46.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:45 smithi059 ceph-mon[97450]: pgmap v16771: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:46.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:46.580940+0000) 2022-09-08T06:15:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:46.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:46.580779+0000) 2022-09-08T06:15:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:47.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:47.581121+0000) 2022-09-08T06:15:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:47.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:47.580940+0000) 2022-09-08T06:15:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:47 smithi055 ceph-mon[97033]: pgmap v16772: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:15:48.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:47 smithi059 ceph-mon[97450]: pgmap v16772: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:15:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:48.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:48.581250+0000) 2022-09-08T06:15:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:48.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:48.581104+0000) 2022-09-08T06:15:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:49.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:49.581389+0000) 2022-09-08T06:15:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:49.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:49.581236+0000) 2022-09-08T06:15:49.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:49 smithi055 ceph-mon[97033]: pgmap v16773: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:50.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:49 smithi059 ceph-mon[97450]: pgmap v16773: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:50.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:15:50.007+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:15:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:50.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:50.581540+0000) 2022-09-08T06:15:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:50.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:50.581373+0000) 2022-09-08T06:15:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:51.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:51.581708+0000) 2022-09-08T06:15:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:51.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:51.581527+0000) 2022-09-08T06:15:51.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:51 smithi055 ceph-mon[97033]: pgmap v16774: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:15:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:51 smithi059 ceph-mon[97450]: pgmap v16774: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:15:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:52.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:52.581841+0000) 2022-09-08T06:15:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:52.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:52.581733+0000) 2022-09-08T06:15:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:53.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:53.582041+0000) 2022-09-08T06:15:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:53.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:53.581945+0000) 2022-09-08T06:15:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:54 smithi055 ceph-mon[97033]: pgmap v16775: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:54.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:15:54.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:15:54.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:54 smithi055 ceph-mon[97033]: pgmap v16776: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:15:54.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:54 smithi055 ceph-mon[97033]: pgmap v16777: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T06:15:54.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:15:54.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:54 smithi059 ceph-mon[97450]: pgmap v16775: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:15:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:15:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:15:54.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:54 smithi059 ceph-mon[97450]: pgmap v16776: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:15:54.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:54 smithi059 ceph-mon[97450]: pgmap v16777: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T06:15:54.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:15:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:54.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:54.582126+0000) 2022-09-08T06:15:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:54.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:54.582209+0000) 2022-09-08T06:15:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:55.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:55.582363+0000) 2022-09-08T06:15:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:55.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:55.582257+0000) 2022-09-08T06:15:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:56 smithi055 ceph-mon[97033]: pgmap v16778: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T06:15:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:56 smithi059 ceph-mon[97450]: pgmap v16778: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T06:15:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:56.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:56.582577+0000) 2022-09-08T06:15:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:56.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:56.582419+0000) 2022-09-08T06:15:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:15:57 smithi059 ceph-mon[97450]: pgmap v16779: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T06:15:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:57.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:57.582604+0000) 2022-09-08T06:15:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:57.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:57.582799+0000) 2022-09-08T06:15:57.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:15:57 smithi055 ceph-mon[97033]: pgmap v16779: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T06:15:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:58.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:58.582783+0000) 2022-09-08T06:15:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:58.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:58.582970+0000) 2022-09-08T06:15:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:15:59.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:59.583097+0000) 2022-09-08T06:15:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:15:59.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:15:59.582910+0000) 2022-09-08T06:16:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:00 smithi055 ceph-mon[97033]: pgmap v16780: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T06:16:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:00 smithi059 ceph-mon[97450]: pgmap v16780: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T06:16:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:00.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:00.583217+0000) 2022-09-08T06:16:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:00.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:00.583040+0000) 2022-09-08T06:16:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:01.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:01.583352+0000) 2022-09-08T06:16:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:01.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:01.583172+0000) 2022-09-08T06:16:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:02 smithi055 ceph-mon[97033]: pgmap v16781: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T06:16:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:02 smithi059 ceph-mon[97450]: pgmap v16781: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T06:16:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:02.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:02.583347+0000) 2022-09-08T06:16:02.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:02.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:02.583548+0000) 2022-09-08T06:16:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:03.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:03.583487+0000) 2022-09-08T06:16:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:03.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:03.583762+0000) 2022-09-08T06:16:04.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:04 smithi055 ceph-mon[97033]: pgmap v16782: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:16:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:04 smithi059 ceph-mon[97450]: pgmap v16782: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:16:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:04.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:04.583891+0000) 2022-09-08T06:16:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:04.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:04.583678+0000) 2022-09-08T06:16:05.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:16:05.022+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:16:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:05.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:05.584021+0000) 2022-09-08T06:16:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:05.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:05.583801+0000) 2022-09-08T06:16:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:06 smithi055 ceph-mon[97033]: pgmap v16783: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:16:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:16:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:06 smithi059 ceph-mon[97450]: pgmap v16783: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:16:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:16:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:06.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:06.584218+0000) 2022-09-08T06:16:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:06.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:06.584005+0000) 2022-09-08T06:16:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:16:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:07 smithi059 ceph-mon[97450]: pgmap v16784: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:16:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:07.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:07.584393+0000) 2022-09-08T06:16:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:07.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:07.584159+0000) 2022-09-08T06:16:07.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:16:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:07 smithi055 ceph-mon[97033]: pgmap v16784: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:16:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:08.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:08.584521+0000) 2022-09-08T06:16:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:08.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:08.584301+0000) 2022-09-08T06:16:09.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:09.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:09.584664+0000) 2022-09-08T06:16:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:09.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:09.584433+0000) 2022-09-08T06:16:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:09 smithi055 ceph-mon[97033]: osdmap e1128: 12 total, 11 up, 11 in 2022-09-08T06:16:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:09 smithi055 ceph-mon[97033]: pgmap v16786: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:16:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:09 smithi059 ceph-mon[97450]: osdmap e1128: 12 total, 11 up, 11 in 2022-09-08T06:16:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:09 smithi059 ceph-mon[97450]: pgmap v16786: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:16:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:10.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:10.584604+0000) 2022-09-08T06:16:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:10.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:10.584855+0000) 2022-09-08T06:16:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:11.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:11.584980+0000) 2022-09-08T06:16:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:11.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:11.584766+0000) 2022-09-08T06:16:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:12 smithi055 ceph-mon[97033]: pgmap v16787: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:16:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:12 smithi055 ceph-mon[97033]: osdmap e1129: 12 total, 11 up, 11 in 2022-09-08T06:16:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:12 smithi059 ceph-mon[97450]: pgmap v16787: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:16:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:12 smithi059 ceph-mon[97450]: osdmap e1129: 12 total, 11 up, 11 in 2022-09-08T06:16:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:12.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:12.585166+0000) 2022-09-08T06:16:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:12.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:12.584944+0000) 2022-09-08T06:16:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:13.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:13.585372+0000) 2022-09-08T06:16:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:13.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:13.585122+0000) 2022-09-08T06:16:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:14 smithi055 ceph-mon[97033]: pgmap v16789: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:16:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:14 smithi059 ceph-mon[97450]: pgmap v16789: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:16:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:14.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:14.585570+0000) 2022-09-08T06:16:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:14.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:14.585271+0000) 2022-09-08T06:16:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:15.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:15.585433+0000) 2022-09-08T06:16:15.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:15.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:15.585699+0000) 2022-09-08T06:16:16.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:16 smithi055 ceph-mon[97033]: pgmap v16790: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:16:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:16 smithi059 ceph-mon[97450]: pgmap v16790: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:16:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:16.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:16.585655+0000) 2022-09-08T06:16:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:16.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:16.585823+0000) 2022-09-08T06:16:17.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:17 smithi059 ceph-mon[97450]: pgmap v16791: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T06:16:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:17.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:17.585971+0000) 2022-09-08T06:16:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:17.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:17.585876+0000) 2022-09-08T06:16:17.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:17 smithi055 ceph-mon[97033]: pgmap v16791: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T06:16:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:18.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:18.586164+0000) 2022-09-08T06:16:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:18.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:18.586154+0000) 2022-09-08T06:16:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:19.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:19.586417+0000) 2022-09-08T06:16:19.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:19.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:19.586417+0000) 2022-09-08T06:16:20.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:16:20.034+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:16:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:20 smithi055 ceph-mon[97033]: pgmap v16792: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:16:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:20 smithi059 ceph-mon[97450]: pgmap v16792: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:16:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:20.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:20.586626+0000) 2022-09-08T06:16:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:20.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:20.586626+0000) 2022-09-08T06:16:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:21.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:21.586801+0000) 2022-09-08T06:16:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:21.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:21.586813+0000) 2022-09-08T06:16:22.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:22 smithi055 ceph-mon[97033]: pgmap v16793: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:16:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:22 smithi059 ceph-mon[97450]: pgmap v16793: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:16:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:22.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:22.587073+0000) 2022-09-08T06:16:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:22.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:22.587071+0000) 2022-09-08T06:16:23.909 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:23.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:23.587301+0000) 2022-09-08T06:16:23.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:23.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:23.587296+0000) 2022-09-08T06:16:24.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:24 smithi055 ceph-mon[97033]: pgmap v16794: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 88 B/s wr, 5 op/s 2022-09-08T06:16:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:24 smithi059 ceph-mon[97450]: pgmap v16794: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 88 B/s wr, 5 op/s 2022-09-08T06:16:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:24.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:24.587521+0000) 2022-09-08T06:16:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:24.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:24.587521+0000) 2022-09-08T06:16:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:25.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:25.587748+0000) 2022-09-08T06:16:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:25.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:25.587762+0000) 2022-09-08T06:16:26.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:26 smithi055 ceph-mon[97033]: pgmap v16795: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:16:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:26 smithi059 ceph-mon[97450]: pgmap v16795: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:16:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:26.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:26.588050+0000) 2022-09-08T06:16:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:26.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:26.588050+0000) 2022-09-08T06:16:27.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:27 smithi059 ceph-mon[97450]: pgmap v16796: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:27.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:27.588332+0000) 2022-09-08T06:16:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:27.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:27.588330+0000) 2022-09-08T06:16:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:27 smithi055 ceph-mon[97033]: pgmap v16796: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:28.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:28.588579+0000) 2022-09-08T06:16:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:28.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:28.588577+0000) 2022-09-08T06:16:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:29.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:29.588739+0000) 2022-09-08T06:16:29.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:29.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:29.588726+0000) 2022-09-08T06:16:30.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:30 smithi055 ceph-mon[97033]: pgmap v16797: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:30 smithi059 ceph-mon[97450]: pgmap v16797: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:30.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:30.588991+0000) 2022-09-08T06:16:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:30.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:30.588992+0000) 2022-09-08T06:16:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:31.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:31.589284+0000) 2022-09-08T06:16:31.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:31.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:31.589284+0000) 2022-09-08T06:16:32.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:32 smithi055 ceph-mon[97033]: pgmap v16798: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:16:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:32 smithi059 ceph-mon[97450]: pgmap v16798: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:16:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:32.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:32.589524+0000) 2022-09-08T06:16:32.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:32.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:32.589524+0000) 2022-09-08T06:16:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:33.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:33.589721+0000) 2022-09-08T06:16:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:33.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:33.589736+0000) 2022-09-08T06:16:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:34 smithi055 ceph-mon[97033]: pgmap v16799: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:34 smithi059 ceph-mon[97450]: pgmap v16799: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:34.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:34.589996+0000) 2022-09-08T06:16:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:34.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:34.589995+0000) 2022-09-08T06:16:35.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:16:35.049+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:16:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:35.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:35.590169+0000) 2022-09-08T06:16:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:35.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:35.590165+0000) 2022-09-08T06:16:36.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:36 smithi055 ceph-mon[97033]: pgmap v16800: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:16:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:36 smithi059 ceph-mon[97450]: pgmap v16800: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:16:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:36.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:36.590416+0000) 2022-09-08T06:16:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:36.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:36.590416+0000) 2022-09-08T06:16:37.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:37 smithi059 ceph-mon[97450]: pgmap v16801: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:37.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:37.590611+0000) 2022-09-08T06:16:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:37.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:37.590607+0000) 2022-09-08T06:16:37.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:37 smithi055 ceph-mon[97033]: pgmap v16801: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:38.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:38.590802+0000) 2022-09-08T06:16:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:38.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:38.590871+0000) 2022-09-08T06:16:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:39.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:39.591077+0000) 2022-09-08T06:16:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:39.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:39.591071+0000) 2022-09-08T06:16:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:40 smithi055 ceph-mon[97033]: pgmap v16802: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:40 smithi059 ceph-mon[97450]: pgmap v16802: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:40.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:40.591339+0000) 2022-09-08T06:16:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:40.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:40.591339+0000) 2022-09-08T06:16:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:41.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:41.591530+0000) 2022-09-08T06:16:41.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:41.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:41.591500+0000) 2022-09-08T06:16:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:42 smithi055 ceph-mon[97033]: pgmap v16803: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:16:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:42 smithi059 ceph-mon[97450]: pgmap v16803: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:16:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:42.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:42.591699+0000) 2022-09-08T06:16:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:42.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:42.591662+0000) 2022-09-08T06:16:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:43.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:43.591936+0000) 2022-09-08T06:16:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:43.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:43.591887+0000) 2022-09-08T06:16:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:44 smithi055 ceph-mon[97033]: pgmap v16804: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:44 smithi059 ceph-mon[97450]: pgmap v16804: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:44.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:44.592167+0000) 2022-09-08T06:16:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:44.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:44.592144+0000) 2022-09-08T06:16:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:45.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:45.592374+0000) 2022-09-08T06:16:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:45.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:45.592377+0000) 2022-09-08T06:16:46.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:46 smithi055 ceph-mon[97033]: pgmap v16805: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:16:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:46 smithi059 ceph-mon[97450]: pgmap v16805: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:16:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:46.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:46.592602+0000) 2022-09-08T06:16:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:46.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:46.592608+0000) 2022-09-08T06:16:47.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:47 smithi059 ceph-mon[97450]: pgmap v16806: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:47.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:47.592863+0000) 2022-09-08T06:16:47.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:47.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:47.592841+0000) 2022-09-08T06:16:47.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:47 smithi055 ceph-mon[97033]: pgmap v16806: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:48.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:48.593130+0000) 2022-09-08T06:16:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:48.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:48.593123+0000) 2022-09-08T06:16:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:49.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:49.593289+0000) 2022-09-08T06:16:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:49.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:49.593316+0000) 2022-09-08T06:16:50.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:16:50.050+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:16:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:50 smithi055 ceph-mon[97033]: pgmap v16807: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:50 smithi059 ceph-mon[97450]: pgmap v16807: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:50.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:50.593545+0000) 2022-09-08T06:16:50.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:50.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:50.593524+0000) 2022-09-08T06:16:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:51.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:51.593803+0000) 2022-09-08T06:16:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:51.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:51.593780+0000) 2022-09-08T06:16:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:52 smithi055 ceph-mon[97033]: pgmap v16808: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:16:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:52 smithi059 ceph-mon[97450]: pgmap v16808: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:16:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:52.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:52.593971+0000) 2022-09-08T06:16:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:52.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:52.593913+0000) 2022-09-08T06:16:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:53.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:53.594113+0000) 2022-09-08T06:16:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:53.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:53.594148+0000) 2022-09-08T06:16:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:54 smithi055 ceph-mon[97033]: pgmap v16809: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:16:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:16:54.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:16:54.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:16:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:54 smithi059 ceph-mon[97450]: pgmap v16809: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:16:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:16:54.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:16:54.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:16:54.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:16:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:54.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:54.594367+0000) 2022-09-08T06:16:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:54.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:54.594350+0000) 2022-09-08T06:16:55.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:55 smithi055 ceph-mon[97033]: pgmap v16810: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:16:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:55 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T06:16:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:55 smithi059 ceph-mon[97450]: pgmap v16810: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:16:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:55 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T06:16:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:55.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:55.594592+0000) 2022-09-08T06:16:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:55.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:55.594608+0000) 2022-09-08T06:16:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:56.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:56.594807+0000) 2022-09-08T06:16:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:56.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:56.594895+0000) 2022-09-08T06:16:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:57 smithi055 ceph-mon[97033]: pgmap v16811: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:16:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:57 smithi059 ceph-mon[97450]: pgmap v16811: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:16:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:57.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:57.595040+0000) 2022-09-08T06:16:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:57.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:57.594996+0000) 2022-09-08T06:16:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:58.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:58.595269+0000) 2022-09-08T06:16:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:58.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:58.595227+0000) 2022-09-08T06:16:59.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:16:59 smithi055 ceph-mon[97033]: pgmap v16812: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:16:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:16:59 smithi059 ceph-mon[97450]: pgmap v16812: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:16:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:16:59.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:59.595431+0000) 2022-09-08T06:16:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:16:59.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:16:59.595437+0000) 2022-09-08T06:17:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:00.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:00.595605+0000) 2022-09-08T06:17:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:00.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:00.595669+0000) 2022-09-08T06:17:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:01 smithi055 ceph-mon[97033]: pgmap v16813: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:17:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:01 smithi059 ceph-mon[97450]: pgmap v16813: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:17:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:01.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:01.595828+0000) 2022-09-08T06:17:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:01.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:01.595922+0000) 2022-09-08T06:17:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:02 smithi059 ceph-mon[97450]: pgmap v16814: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:17:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:02.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:02.595980+0000) 2022-09-08T06:17:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:02.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:02.596104+0000) 2022-09-08T06:17:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:02 smithi055 ceph-mon[97033]: pgmap v16814: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:17:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:03.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:03.596305+0000) 2022-09-08T06:17:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:03.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:03.596187+0000) 2022-09-08T06:17:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:04.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:04.596439+0000) 2022-09-08T06:17:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:04.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:04.596550+0000) 2022-09-08T06:17:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:04 smithi055 ceph-mon[97033]: pgmap v16815: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:17:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:04 smithi059 ceph-mon[97450]: pgmap v16815: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:17:05.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:17:05.061+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:17:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:05.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:05.596617+0000) 2022-09-08T06:17:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:05.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:05.596743+0000) 2022-09-08T06:17:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:06.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:06.596901+0000) 2022-09-08T06:17:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:06.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:06.596980+0000) 2022-09-08T06:17:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:06 smithi055 ceph-mon[97033]: pgmap v16816: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:17:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:17:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:17:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:06 smithi059 ceph-mon[97450]: pgmap v16816: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:17:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:17:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:17:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:07.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:07.597119+0000) 2022-09-08T06:17:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:07.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:07.597220+0000) 2022-09-08T06:17:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:08.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:08.597388+0000) 2022-09-08T06:17:08.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:08.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:08.597406+0000) 2022-09-08T06:17:08.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:08 smithi055 ceph-mon[97033]: pgmap v16817: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:17:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:08 smithi059 ceph-mon[97450]: pgmap v16817: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:17:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:09.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:09.597669+0000) 2022-09-08T06:17:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:09.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:09.597662+0000) 2022-09-08T06:17:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:10.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:10.597967+0000) 2022-09-08T06:17:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:10.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:10.597925+0000) 2022-09-08T06:17:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:10 smithi059 ceph-mon[97450]: pgmap v16818: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:17:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:10 smithi059 ceph-mon[97450]: osdmap e1130: 12 total, 11 up, 11 in 2022-09-08T06:17:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:10 smithi055 ceph-mon[97033]: pgmap v16818: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:17:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:10 smithi055 ceph-mon[97033]: osdmap e1130: 12 total, 11 up, 11 in 2022-09-08T06:17:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:11.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:11.598099+0000) 2022-09-08T06:17:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:11.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:11.598094+0000) 2022-09-08T06:17:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:12 smithi059 ceph-mon[97450]: pgmap v16820: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:17:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:12.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:12.598272+0000) 2022-09-08T06:17:12.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:12.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:12.598269+0000) 2022-09-08T06:17:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:12 smithi055 ceph-mon[97033]: pgmap v16820: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:17:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:13.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:13.598510+0000) 2022-09-08T06:17:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:13.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:13.598523+0000) 2022-09-08T06:17:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:14.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:14.598731+0000) 2022-09-08T06:17:14.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:14.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:14.598741+0000) 2022-09-08T06:17:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:14 smithi055 ceph-mon[97033]: pgmap v16821: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:17:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:14 smithi059 ceph-mon[97450]: pgmap v16821: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:17:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:15.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:15.598973+0000) 2022-09-08T06:17:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:15.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:15.598983+0000) 2022-09-08T06:17:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:16.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:16.599180+0000) 2022-09-08T06:17:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:16.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:16.599167+0000) 2022-09-08T06:17:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:16 smithi055 ceph-mon[97033]: pgmap v16822: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:17:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:16 smithi055 ceph-mon[97033]: osdmap e1131: 12 total, 11 up, 11 in 2022-09-08T06:17:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:16 smithi059 ceph-mon[97450]: pgmap v16822: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:17:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:16 smithi059 ceph-mon[97450]: osdmap e1131: 12 total, 11 up, 11 in 2022-09-08T06:17:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:17.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:17.599411+0000) 2022-09-08T06:17:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:17.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:17.599412+0000) 2022-09-08T06:17:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:18.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:18.599617+0000) 2022-09-08T06:17:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:18.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:18.599601+0000) 2022-09-08T06:17:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:18 smithi055 ceph-mon[97033]: pgmap v16824: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:17:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:18 smithi059 ceph-mon[97450]: pgmap v16824: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:17:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:19.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:19.599864+0000) 2022-09-08T06:17:19.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:19.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:19.599841+0000) 2022-09-08T06:17:20.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:17:20.075+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:17:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:20.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:20.600105+0000) 2022-09-08T06:17:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:20.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:20.600104+0000) 2022-09-08T06:17:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:20 smithi059 ceph-mon[97450]: pgmap v16825: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:17:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:20 smithi055 ceph-mon[97033]: pgmap v16825: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:17:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:21.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:21.600376+0000) 2022-09-08T06:17:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:21.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:21.600349+0000) 2022-09-08T06:17:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:22 smithi059 ceph-mon[97450]: pgmap v16826: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:17:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:22.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:22.600594+0000) 2022-09-08T06:17:22.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:22.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:22.600630+0000) 2022-09-08T06:17:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:22 smithi055 ceph-mon[97033]: pgmap v16826: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:17:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:23.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:23.600892+0000) 2022-09-08T06:17:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:23.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:23.600904+0000) 2022-09-08T06:17:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:24.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:24.601128+0000) 2022-09-08T06:17:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:24.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:24.601161+0000) 2022-09-08T06:17:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:24 smithi055 ceph-mon[97033]: pgmap v16827: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:17:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:24 smithi059 ceph-mon[97450]: pgmap v16827: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:17:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:25.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:25.601388+0000) 2022-09-08T06:17:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:25.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:25.601431+0000) 2022-09-08T06:17:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:26.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:26.601548+0000) 2022-09-08T06:17:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:26.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:26.601610+0000) 2022-09-08T06:17:26.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:26 smithi055 ceph-mon[97033]: pgmap v16828: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:17:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:26 smithi059 ceph-mon[97450]: pgmap v16828: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:17:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:27.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:27.601739+0000) 2022-09-08T06:17:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:27.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:27.601796+0000) 2022-09-08T06:17:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:28.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:28.601990+0000) 2022-09-08T06:17:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:28.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:28.602002+0000) 2022-09-08T06:17:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:28 smithi059 ceph-mon[97450]: pgmap v16829: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:17:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:28 smithi055 ceph-mon[97033]: pgmap v16829: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:17:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:29.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:29.602263+0000) 2022-09-08T06:17:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:29.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:29.602249+0000) 2022-09-08T06:17:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:30.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:30.602511+0000) 2022-09-08T06:17:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:30.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:30.602489+0000) 2022-09-08T06:17:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:30 smithi059 ceph-mon[97450]: pgmap v16830: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:17:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:30 smithi055 ceph-mon[97033]: pgmap v16830: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:17:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:31.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:31.602677+0000) 2022-09-08T06:17:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:31.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:31.602718+0000) 2022-09-08T06:17:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:32.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:32.602905+0000) 2022-09-08T06:17:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:32.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:32.602934+0000) 2022-09-08T06:17:32.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:32 smithi055 ceph-mon[97033]: pgmap v16831: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:17:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:32 smithi059 ceph-mon[97450]: pgmap v16831: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:17:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:33.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:33.603041+0000) 2022-09-08T06:17:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:33.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:33.603126+0000) 2022-09-08T06:17:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:34.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:34.603292+0000) 2022-09-08T06:17:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:34.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:34.603363+0000) 2022-09-08T06:17:34.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:34 smithi055 ceph-mon[97033]: pgmap v16832: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:17:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:34 smithi059 ceph-mon[97450]: pgmap v16832: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:17:35.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:17:35.081+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:17:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:35.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:35.603566+0000) 2022-09-08T06:17:35.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:35.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:35.603515+0000) 2022-09-08T06:17:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:36.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:36.603793+0000) 2022-09-08T06:17:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:36.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:36.603741+0000) 2022-09-08T06:17:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:36 smithi055 ceph-mon[97033]: pgmap v16833: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:17:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:36 smithi059 ceph-mon[97450]: pgmap v16833: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:17:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:37.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:37.604034+0000) 2022-09-08T06:17:37.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:37.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:37.603984+0000) 2022-09-08T06:17:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:38.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:38.604242+0000) 2022-09-08T06:17:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:38.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:38.604312+0000) 2022-09-08T06:17:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:38 smithi055 ceph-mon[97033]: pgmap v16834: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:17:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:38 smithi059 ceph-mon[97450]: pgmap v16834: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:17:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:39.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:39.604495+0000) 2022-09-08T06:17:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:39.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:39.604590+0000) 2022-09-08T06:17:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:40.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:40.604701+0000) 2022-09-08T06:17:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:40.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:40.604898+0000) 2022-09-08T06:17:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:40 smithi055 ceph-mon[97033]: pgmap v16835: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:17:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:40 smithi059 ceph-mon[97450]: pgmap v16835: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:17:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:41.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:41.604913+0000) 2022-09-08T06:17:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:41.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:41.605089+0000) 2022-09-08T06:17:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:42 smithi059 ceph-mon[97450]: pgmap v16836: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:17:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:42.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:42.605156+0000) 2022-09-08T06:17:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:42.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:42.605218+0000) 2022-09-08T06:17:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:42 smithi055 ceph-mon[97033]: pgmap v16836: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:17:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:43.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:43.605408+0000) 2022-09-08T06:17:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:43.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:43.605419+0000) 2022-09-08T06:17:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:44.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:44.605619+0000) 2022-09-08T06:17:44.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:44.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:44.605625+0000) 2022-09-08T06:17:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:44 smithi055 ceph-mon[97033]: pgmap v16837: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:17:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:44 smithi059 ceph-mon[97450]: pgmap v16837: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:17:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:45.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:45.605797+0000) 2022-09-08T06:17:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:45.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:45.605796+0000) 2022-09-08T06:17:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:46.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:46.606014+0000) 2022-09-08T06:17:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:46.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:46.606038+0000) 2022-09-08T06:17:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:46 smithi059 ceph-mon[97450]: pgmap v16838: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:17:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:46 smithi055 ceph-mon[97033]: pgmap v16838: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:17:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:47.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:47.606227+0000) 2022-09-08T06:17:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:47.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:47.606236+0000) 2022-09-08T06:17:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:48.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:48.606388+0000) 2022-09-08T06:17:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:48.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:48.606407+0000) 2022-09-08T06:17:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:48 smithi059 ceph-mon[97450]: pgmap v16839: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:17:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:48 smithi055 ceph-mon[97033]: pgmap v16839: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:17:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:49.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:49.606672+0000) 2022-09-08T06:17:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:49.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:49.606672+0000) 2022-09-08T06:17:50.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:17:50.083+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:17:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:50.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:50.606868+0000) 2022-09-08T06:17:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:50.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:50.606847+0000) 2022-09-08T06:17:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:50 smithi059 ceph-mon[97450]: pgmap v16840: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:17:51.071 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:50 smithi055 ceph-mon[97033]: pgmap v16840: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:17:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:51.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:51.607000+0000) 2022-09-08T06:17:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:51.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:51.607017+0000) 2022-09-08T06:17:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:52.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:52.607178+0000) 2022-09-08T06:17:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:52.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:52.607172+0000) 2022-09-08T06:17:52.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:52 smithi055 ceph-mon[97033]: pgmap v16841: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:17:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:52 smithi059 ceph-mon[97450]: pgmap v16841: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:17:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:53.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:53.607332+0000) 2022-09-08T06:17:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:53.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:53.607328+0000) 2022-09-08T06:17:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:54.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:54.607514+0000) 2022-09-08T06:17:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:54.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:54.607514+0000) 2022-09-08T06:17:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:54 smithi059 ceph-mon[97450]: pgmap v16842: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:17:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:17:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:17:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:17:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:17:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:54 smithi055 ceph-mon[97033]: pgmap v16842: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:17:55.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:17:55.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:17:55.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:17:55.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:17:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:55.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:55.607680+0000) 2022-09-08T06:17:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:55.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:55.607680+0000) 2022-09-08T06:17:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:56.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:56.607846+0000) 2022-09-08T06:17:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:56.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:56.607824+0000) 2022-09-08T06:17:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:56 smithi059 ceph-mon[97450]: pgmap v16843: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:17:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:56 smithi055 ceph-mon[97033]: pgmap v16843: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:17:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:57.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:57.607959+0000) 2022-09-08T06:17:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:57.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:57.607954+0000) 2022-09-08T06:17:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:58.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:58.608096+0000) 2022-09-08T06:17:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:58.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:58.608096+0000) 2022-09-08T06:17:59.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:17:58 smithi059 ceph-mon[97450]: pgmap v16844: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:17:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:17:58 smithi055 ceph-mon[97033]: pgmap v16844: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:17:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:17:59.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:59.608314+0000) 2022-09-08T06:17:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:17:59.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:17:59.608286+0000) 2022-09-08T06:18:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:00.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:00.608465+0000) 2022-09-08T06:18:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:00.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:00.608417+0000) 2022-09-08T06:18:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:00 smithi059 ceph-mon[97450]: pgmap v16845: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:00 smithi055 ceph-mon[97033]: pgmap v16845: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:01.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:01.608564+0000) 2022-09-08T06:18:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:01.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:01.608622+0000) 2022-09-08T06:18:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:02.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:02.608802+0000) 2022-09-08T06:18:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:02.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:02.608828+0000) 2022-09-08T06:18:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:02 smithi055 ceph-mon[97033]: pgmap v16846: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:18:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:02 smithi059 ceph-mon[97450]: pgmap v16846: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:18:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:03.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:03.609001+0000) 2022-09-08T06:18:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:03.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:03.608982+0000) 2022-09-08T06:18:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:04.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:04.609195+0000) 2022-09-08T06:18:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:04.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:04.609228+0000) 2022-09-08T06:18:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:04 smithi059 ceph-mon[97450]: pgmap v16847: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:04 smithi055 ceph-mon[97033]: pgmap v16847: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:05.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:18:05.089+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:18:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:05.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:05.609424+0000) 2022-09-08T06:18:05.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:05.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:05.609389+0000) 2022-09-08T06:18:06.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:06.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:06.609562+0000) 2022-09-08T06:18:06.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:06.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:06.609557+0000) 2022-09-08T06:18:07.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:06 smithi059 ceph-mon[97450]: pgmap v16848: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:18:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:18:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:18:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:06 smithi055 ceph-mon[97033]: pgmap v16848: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:18:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:18:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:18:07.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:07.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:07.609696+0000) 2022-09-08T06:18:07.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:07.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:07.609714+0000) 2022-09-08T06:18:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:08.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:08.609959+0000) 2022-09-08T06:18:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:08.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:08.609879+0000) 2022-09-08T06:18:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:08 smithi059 ceph-mon[97450]: pgmap v16849: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:18:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:08 smithi055 ceph-mon[97033]: pgmap v16849: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:18:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:09.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:09.610078+0000) 2022-09-08T06:18:09.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:09.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:09.610124+0000) 2022-09-08T06:18:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:10.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:10.610263+0000) 2022-09-08T06:18:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:10.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:10.610231+0000) 2022-09-08T06:18:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:10 smithi059 ceph-mon[97450]: pgmap v16850: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:18:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:10 smithi059 ceph-mon[97450]: osdmap e1132: 12 total, 11 up, 11 in 2022-09-08T06:18:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:10 smithi055 ceph-mon[97033]: pgmap v16850: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:18:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:10 smithi055 ceph-mon[97033]: osdmap e1132: 12 total, 11 up, 11 in 2022-09-08T06:18:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:11.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:11.610547+0000) 2022-09-08T06:18:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:11.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:11.610494+0000) 2022-09-08T06:18:12.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:12.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:12.610672+0000) 2022-09-08T06:18:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:12.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:12.610722+0000) 2022-09-08T06:18:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:12 smithi055 ceph-mon[97033]: pgmap v16852: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:18:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:12 smithi059 ceph-mon[97450]: pgmap v16852: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:18:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:13.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:13.610863+0000) 2022-09-08T06:18:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:13.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:13.610829+0000) 2022-09-08T06:18:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:14.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:14.611085+0000) 2022-09-08T06:18:14.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:14.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:14.611021+0000) 2022-09-08T06:18:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:14 smithi059 ceph-mon[97450]: pgmap v16853: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:18:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:14 smithi055 ceph-mon[97033]: pgmap v16853: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:18:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:15.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:15.611235+0000) 2022-09-08T06:18:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:15.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:15.611190+0000) 2022-09-08T06:18:16.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:16.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:16.611386+0000) 2022-09-08T06:18:16.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:16.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:16.611352+0000) 2022-09-08T06:18:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:16 smithi059 ceph-mon[97450]: pgmap v16854: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:18:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:16 smithi059 ceph-mon[97450]: osdmap e1133: 12 total, 11 up, 11 in 2022-09-08T06:18:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:16 smithi055 ceph-mon[97033]: pgmap v16854: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:18:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:16 smithi055 ceph-mon[97033]: osdmap e1133: 12 total, 11 up, 11 in 2022-09-08T06:18:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:17.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:17.611616+0000) 2022-09-08T06:18:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:17.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:17.611548+0000) 2022-09-08T06:18:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:18.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:18.611816+0000) 2022-09-08T06:18:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:18.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:18.611771+0000) 2022-09-08T06:18:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:18 smithi059 ceph-mon[97450]: pgmap v16856: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:18:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:18 smithi055 ceph-mon[97033]: pgmap v16856: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:18:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:19.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:19.611972+0000) 2022-09-08T06:18:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:19.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:19.611915+0000) 2022-09-08T06:18:20.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:18:20.104+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:18:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:20.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:20.612150+0000) 2022-09-08T06:18:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:20.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:20.612104+0000) 2022-09-08T06:18:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:20 smithi059 ceph-mon[97450]: pgmap v16857: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:18:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:20 smithi055 ceph-mon[97033]: pgmap v16857: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:18:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:21.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:21.612373+0000) 2022-09-08T06:18:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:21.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:21.612352+0000) 2022-09-08T06:18:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:22.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:22.612549+0000) 2022-09-08T06:18:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:22.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:22.612530+0000) 2022-09-08T06:18:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:22 smithi055 ceph-mon[97033]: pgmap v16858: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:18:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:22 smithi059 ceph-mon[97450]: pgmap v16858: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:18:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:23.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:23.612723+0000) 2022-09-08T06:18:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:23.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:23.612746+0000) 2022-09-08T06:18:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:24.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:24.612888+0000) 2022-09-08T06:18:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:24.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:24.612920+0000) 2022-09-08T06:18:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:24 smithi059 ceph-mon[97450]: pgmap v16859: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:18:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:24 smithi055 ceph-mon[97033]: pgmap v16859: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:18:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:25.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:25.613112+0000) 2022-09-08T06:18:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:25.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:25.613156+0000) 2022-09-08T06:18:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:26.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:26.613407+0000) 2022-09-08T06:18:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:26.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:26.613361+0000) 2022-09-08T06:18:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:26 smithi059 ceph-mon[97450]: pgmap v16860: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:18:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:26 smithi055 ceph-mon[97033]: pgmap v16860: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:18:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:27.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:27.613593+0000) 2022-09-08T06:18:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:27.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:27.613679+0000) 2022-09-08T06:18:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:28.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:28.613919+0000) 2022-09-08T06:18:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:28.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:28.613872+0000) 2022-09-08T06:18:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:28 smithi059 ceph-mon[97450]: pgmap v16861: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:18:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:28 smithi055 ceph-mon[97033]: pgmap v16861: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:18:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:29.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:29.614138+0000) 2022-09-08T06:18:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:29.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:29.614113+0000) 2022-09-08T06:18:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:30.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:30.614363+0000) 2022-09-08T06:18:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:30.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:30.614315+0000) 2022-09-08T06:18:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:30 smithi059 ceph-mon[97450]: pgmap v16862: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:30 smithi055 ceph-mon[97033]: pgmap v16862: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:31.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:31.614475+0000) 2022-09-08T06:18:31.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:31.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:31.614571+0000) 2022-09-08T06:18:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:32.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:32.614727+0000) 2022-09-08T06:18:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:32.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:32.614617+0000) 2022-09-08T06:18:32.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:32 smithi055 ceph-mon[97033]: pgmap v16863: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:18:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:32 smithi059 ceph-mon[97450]: pgmap v16863: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:18:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:33.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:33.614904+0000) 2022-09-08T06:18:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:33.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:33.614861+0000) 2022-09-08T06:18:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:34.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:34.615106+0000) 2022-09-08T06:18:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:34.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:34.615032+0000) 2022-09-08T06:18:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:34 smithi059 ceph-mon[97450]: pgmap v16864: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:34 smithi055 ceph-mon[97033]: pgmap v16864: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:35.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:18:35.108+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:18:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:35.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:35.615311+0000) 2022-09-08T06:18:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:35.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:35.615227+0000) 2022-09-08T06:18:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:36.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:36.615510+0000) 2022-09-08T06:18:36.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:36.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:36.615483+0000) 2022-09-08T06:18:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:36 smithi059 ceph-mon[97450]: pgmap v16865: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:18:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:36 smithi055 ceph-mon[97033]: pgmap v16865: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:18:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:37.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:37.615759+0000) 2022-09-08T06:18:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:37.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:37.615758+0000) 2022-09-08T06:18:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:38.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:38.615944+0000) 2022-09-08T06:18:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:38.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:38.615940+0000) 2022-09-08T06:18:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:38 smithi059 ceph-mon[97450]: pgmap v16866: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:38 smithi055 ceph-mon[97033]: pgmap v16866: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:39.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:39.616094+0000) 2022-09-08T06:18:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:39.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:39.616091+0000) 2022-09-08T06:18:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:40.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:40.616271+0000) 2022-09-08T06:18:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:40.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:40.616403+0000) 2022-09-08T06:18:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:40 smithi059 ceph-mon[97450]: pgmap v16867: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:40 smithi055 ceph-mon[97033]: pgmap v16867: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:41.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:41.616413+0000) 2022-09-08T06:18:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:41.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:41.616658+0000) 2022-09-08T06:18:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:42.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:42.616625+0000) 2022-09-08T06:18:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:42.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:42.616855+0000) 2022-09-08T06:18:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:42 smithi055 ceph-mon[97033]: pgmap v16868: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:18:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:42 smithi059 ceph-mon[97450]: pgmap v16868: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:18:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:43.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:43.617017+0000) 2022-09-08T06:18:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:43.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:43.616818+0000) 2022-09-08T06:18:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:44.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:44.617060+0000) 2022-09-08T06:18:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:44.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:44.617221+0000) 2022-09-08T06:18:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:44 smithi059 ceph-mon[97450]: pgmap v16869: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:44 smithi055 ceph-mon[97033]: pgmap v16869: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:45.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:45.617304+0000) 2022-09-08T06:18:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:45.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:45.617433+0000) 2022-09-08T06:18:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:46.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:46.617488+0000) 2022-09-08T06:18:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:46.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:46.617630+0000) 2022-09-08T06:18:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:46 smithi059 ceph-mon[97450]: pgmap v16870: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:18:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:46 smithi055 ceph-mon[97033]: pgmap v16870: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:18:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:47.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:47.617721+0000) 2022-09-08T06:18:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:47.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:47.617801+0000) 2022-09-08T06:18:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:48.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:48.617969+0000) 2022-09-08T06:18:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:48.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:48.618013+0000) 2022-09-08T06:18:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:48 smithi059 ceph-mon[97450]: pgmap v16871: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:48 smithi055 ceph-mon[97033]: pgmap v16871: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:49.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:49.618164+0000) 2022-09-08T06:18:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:49.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:49.618186+0000) 2022-09-08T06:18:50.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:18:50.109+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:18:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:50.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:50.618392+0000) 2022-09-08T06:18:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:50.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:50.618430+0000) 2022-09-08T06:18:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:50 smithi059 ceph-mon[97450]: pgmap v16872: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:50 smithi055 ceph-mon[97033]: pgmap v16872: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:51.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:51.618615+0000) 2022-09-08T06:18:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:51.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:51.618614+0000) 2022-09-08T06:18:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:52.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:52.618782+0000) 2022-09-08T06:18:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:52.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:52.618777+0000) 2022-09-08T06:18:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:52 smithi055 ceph-mon[97033]: pgmap v16873: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:18:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:52 smithi059 ceph-mon[97450]: pgmap v16873: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:18:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:53.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:53.618945+0000) 2022-09-08T06:18:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:53.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:53.618925+0000) 2022-09-08T06:18:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:54.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:54.619121+0000) 2022-09-08T06:18:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:54.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:54.619193+0000) 2022-09-08T06:18:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:54 smithi059 ceph-mon[97450]: pgmap v16874: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:18:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:18:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:18:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:54 smithi055 ceph-mon[97033]: pgmap v16874: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:18:55.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:18:55.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:18:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:55.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:55.619394+0000) 2022-09-08T06:18:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:55.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:55.619339+0000) 2022-09-08T06:18:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:18:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:18:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:56.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:56.619472+0000) 2022-09-08T06:18:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:56.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:56.619544+0000) 2022-09-08T06:18:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:56 smithi055 ceph-mon[97033]: pgmap v16875: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:18:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:56 smithi059 ceph-mon[97450]: pgmap v16875: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:18:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:57.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:57.619786+0000) 2022-09-08T06:18:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:57.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:57.619678+0000) 2022-09-08T06:18:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:58.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:58.619850+0000) 2022-09-08T06:18:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:58.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:58.619958+0000) 2022-09-08T06:18:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:18:58 smithi055 ceph-mon[97033]: pgmap v16876: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:18:58 smithi059 ceph-mon[97450]: pgmap v16876: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:18:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:18:59.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:59.620127+0000) 2022-09-08T06:18:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:18:59.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:18:59.620009+0000) 2022-09-08T06:19:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:00.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:00.620308+0000) 2022-09-08T06:19:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:00.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:00.620142+0000) 2022-09-08T06:19:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:00 smithi055 ceph-mon[97033]: pgmap v16877: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:00 smithi059 ceph-mon[97450]: pgmap v16877: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:01.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:01.620334+0000) 2022-09-08T06:19:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:01.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:01.620458+0000) 2022-09-08T06:19:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:02.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:02.620615+0000) 2022-09-08T06:19:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:02.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:02.620476+0000) 2022-09-08T06:19:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:02 smithi055 ceph-mon[97033]: pgmap v16878: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:19:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:02 smithi059 ceph-mon[97450]: pgmap v16878: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:19:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:03.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:03.620783+0000) 2022-09-08T06:19:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:03.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:03.620654+0000) 2022-09-08T06:19:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:04.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:04.620973+0000) 2022-09-08T06:19:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:04.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:04.620815+0000) 2022-09-08T06:19:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:04 smithi059 ceph-mon[97450]: pgmap v16879: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:04 smithi055 ceph-mon[97033]: pgmap v16879: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:05.413 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:19:05.124+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:19:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:05.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:05.621111+0000) 2022-09-08T06:19:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:05.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:05.620957+0000) 2022-09-08T06:19:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:06.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:06.621284+0000) 2022-09-08T06:19:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:06.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:06.621155+0000) 2022-09-08T06:19:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:06 smithi059 ceph-mon[97450]: pgmap v16880: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:19:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:19:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:19:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:06 smithi055 ceph-mon[97033]: pgmap v16880: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:19:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:19:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:19:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:07.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:07.621448+0000) 2022-09-08T06:19:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:07.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:07.621307+0000) 2022-09-08T06:19:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:08.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:08.621687+0000) 2022-09-08T06:19:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:08.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:08.621486+0000) 2022-09-08T06:19:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:08 smithi059 ceph-mon[97450]: pgmap v16881: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:19:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:08 smithi055 ceph-mon[97033]: pgmap v16881: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:19:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:09.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:09.621840+0000) 2022-09-08T06:19:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:09.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:09.621713+0000) 2022-09-08T06:19:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:10.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:10.622019+0000) 2022-09-08T06:19:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:10.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:10.621888+0000) 2022-09-08T06:19:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:10 smithi059 ceph-mon[97450]: pgmap v16882: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:19:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:10 smithi059 ceph-mon[97450]: osdmap e1134: 12 total, 11 up, 11 in 2022-09-08T06:19:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:10 smithi055 ceph-mon[97033]: pgmap v16882: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:19:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:10 smithi055 ceph-mon[97033]: osdmap e1134: 12 total, 11 up, 11 in 2022-09-08T06:19:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:11.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:11.622168+0000) 2022-09-08T06:19:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:11.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:11.622026+0000) 2022-09-08T06:19:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:12.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:12.622382+0000) 2022-09-08T06:19:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:12.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:12.622210+0000) 2022-09-08T06:19:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:12 smithi055 ceph-mon[97033]: pgmap v16884: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:19:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:12 smithi059 ceph-mon[97450]: pgmap v16884: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:19:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:13.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:13.622566+0000) 2022-09-08T06:19:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:13.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:13.622407+0000) 2022-09-08T06:19:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:14.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:14.622792+0000) 2022-09-08T06:19:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:14.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:14.622580+0000) 2022-09-08T06:19:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:14 smithi059 ceph-mon[97450]: pgmap v16885: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:19:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:14 smithi055 ceph-mon[97033]: pgmap v16885: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:19:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:15.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:15.622703+0000) 2022-09-08T06:19:15.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:15.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:15.622923+0000) 2022-09-08T06:19:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:16.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:16.623051+0000) 2022-09-08T06:19:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:16.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:16.622836+0000) 2022-09-08T06:19:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:16 smithi059 ceph-mon[97450]: pgmap v16886: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:19:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:16 smithi059 ceph-mon[97450]: osdmap e1135: 12 total, 11 up, 11 in 2022-09-08T06:19:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:16 smithi055 ceph-mon[97033]: pgmap v16886: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:19:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:16 smithi055 ceph-mon[97033]: osdmap e1135: 12 total, 11 up, 11 in 2022-09-08T06:19:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:17.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:17.623182+0000) 2022-09-08T06:19:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:17.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:17.623028+0000) 2022-09-08T06:19:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:18.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:18.623335+0000) 2022-09-08T06:19:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:18.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:18.623185+0000) 2022-09-08T06:19:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:18 smithi059 ceph-mon[97450]: pgmap v16888: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:19:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:18 smithi055 ceph-mon[97033]: pgmap v16888: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:19:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:19.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:19.623489+0000) 2022-09-08T06:19:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:19.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:19.623365+0000) 2022-09-08T06:19:20.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:19:20.140+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:19:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:20.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:20.623629+0000) 2022-09-08T06:19:20.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:20.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:20.623524+0000) 2022-09-08T06:19:21.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:20 smithi059 ceph-mon[97450]: pgmap v16889: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:19:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:20 smithi055 ceph-mon[97033]: pgmap v16889: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:19:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:21.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:21.623820+0000) 2022-09-08T06:19:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:21.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:21.623791+0000) 2022-09-08T06:19:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:22.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:22.623963+0000) 2022-09-08T06:19:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:22.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:22.623963+0000) 2022-09-08T06:19:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:22 smithi055 ceph-mon[97033]: pgmap v16890: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:19:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:22 smithi059 ceph-mon[97450]: pgmap v16890: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:19:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:23.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:23.624139+0000) 2022-09-08T06:19:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:23.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:23.624154+0000) 2022-09-08T06:19:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:24.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:24.624367+0000) 2022-09-08T06:19:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:24.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:24.624360+0000) 2022-09-08T06:19:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:24 smithi059 ceph-mon[97450]: pgmap v16891: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:19:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:24 smithi055 ceph-mon[97033]: pgmap v16891: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:19:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:25.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:25.624595+0000) 2022-09-08T06:19:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:25.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:25.624572+0000) 2022-09-08T06:19:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:26.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:26.624854+0000) 2022-09-08T06:19:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:26.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:26.624770+0000) 2022-09-08T06:19:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:26 smithi059 ceph-mon[97450]: pgmap v16892: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:19:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:26 smithi055 ceph-mon[97033]: pgmap v16892: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:19:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:27.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:27.624920+0000) 2022-09-08T06:19:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:27.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:27.625014+0000) 2022-09-08T06:19:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:28.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:28.625194+0000) 2022-09-08T06:19:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:28.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:28.625145+0000) 2022-09-08T06:19:29.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:28 smithi059 ceph-mon[97450]: pgmap v16893: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:19:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:28 smithi055 ceph-mon[97033]: pgmap v16893: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:19:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:29.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:29.625282+0000) 2022-09-08T06:19:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:29.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:29.625340+0000) 2022-09-08T06:19:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:30.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:30.625438+0000) 2022-09-08T06:19:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:30.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:30.625523+0000) 2022-09-08T06:19:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:30 smithi059 ceph-mon[97450]: pgmap v16894: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:30 smithi055 ceph-mon[97033]: pgmap v16894: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:31.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:31.625688+0000) 2022-09-08T06:19:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:31.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:31.625748+0000) 2022-09-08T06:19:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:32.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:32.625969+0000) 2022-09-08T06:19:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:32.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:32.625976+0000) 2022-09-08T06:19:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:32 smithi055 ceph-mon[97033]: pgmap v16895: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:19:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:32 smithi059 ceph-mon[97450]: pgmap v16895: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:19:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:33.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:33.626111+0000) 2022-09-08T06:19:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:33.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:33.626134+0000) 2022-09-08T06:19:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:34.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:34.626269+0000) 2022-09-08T06:19:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:34.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:34.626295+0000) 2022-09-08T06:19:35.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:34 smithi059 ceph-mon[97450]: pgmap v16896: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:34 smithi055 ceph-mon[97033]: pgmap v16896: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:35.628 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:19:35.153+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:19:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:35.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:35.626493+0000) 2022-09-08T06:19:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:35.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:35.626482+0000) 2022-09-08T06:19:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:36.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:36.626714+0000) 2022-09-08T06:19:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:36.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:36.626703+0000) 2022-09-08T06:19:37.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:36 smithi059 ceph-mon[97450]: pgmap v16897: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:19:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:36 smithi055 ceph-mon[97033]: pgmap v16897: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:19:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:37.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:37.626863+0000) 2022-09-08T06:19:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:37.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:37.626872+0000) 2022-09-08T06:19:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:38.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:38.627047+0000) 2022-09-08T06:19:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:38.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:38.627028+0000) 2022-09-08T06:19:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:38 smithi059 ceph-mon[97450]: pgmap v16898: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:38 smithi055 ceph-mon[97033]: pgmap v16898: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:39.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:39.627188+0000) 2022-09-08T06:19:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:39.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:39.627165+0000) 2022-09-08T06:19:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:40.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:40.627433+0000) 2022-09-08T06:19:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:40.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:40.627361+0000) 2022-09-08T06:19:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:40 smithi059 ceph-mon[97450]: pgmap v16899: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:40 smithi055 ceph-mon[97033]: pgmap v16899: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:41.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:41.627575+0000) 2022-09-08T06:19:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:41.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:41.627562+0000) 2022-09-08T06:19:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:42.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:42.627814+0000) 2022-09-08T06:19:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:42.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:42.627758+0000) 2022-09-08T06:19:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:42 smithi055 ceph-mon[97033]: pgmap v16900: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:19:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:42 smithi059 ceph-mon[97450]: pgmap v16900: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:19:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:43.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:43.627960+0000) 2022-09-08T06:19:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:43.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:43.627918+0000) 2022-09-08T06:19:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:44.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:44.628128+0000) 2022-09-08T06:19:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:44.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:44.628052+0000) 2022-09-08T06:19:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:44 smithi059 ceph-mon[97450]: pgmap v16901: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:44 smithi055 ceph-mon[97033]: pgmap v16901: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:45.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:45.628343+0000) 2022-09-08T06:19:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:45.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:45.628238+0000) 2022-09-08T06:19:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:46.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:46.628484+0000) 2022-09-08T06:19:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:46.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:46.628395+0000) 2022-09-08T06:19:47.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:46 smithi059 ceph-mon[97450]: pgmap v16902: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:19:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:46 smithi055 ceph-mon[97033]: pgmap v16902: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:19:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:47.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:47.628654+0000) 2022-09-08T06:19:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:47.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:47.628532+0000) 2022-09-08T06:19:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:48.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:48.628804+0000) 2022-09-08T06:19:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:48.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:48.628745+0000) 2022-09-08T06:19:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:48 smithi059 ceph-mon[97450]: pgmap v16903: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:48 smithi055 ceph-mon[97033]: pgmap v16903: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:49.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:49.628975+0000) 2022-09-08T06:19:49.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:49.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:49.629032+0000) 2022-09-08T06:19:50.630 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:19:50.169+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:19:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:50.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:50.629176+0000) 2022-09-08T06:19:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:50.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:50.629130+0000) 2022-09-08T06:19:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:50 smithi059 ceph-mon[97450]: pgmap v16904: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:50 smithi055 ceph-mon[97033]: pgmap v16904: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:51.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:51.629314+0000) 2022-09-08T06:19:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:51.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:51.629276+0000) 2022-09-08T06:19:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:52.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:52.629467+0000) 2022-09-08T06:19:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:52.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:52.629413+0000) 2022-09-08T06:19:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:52 smithi055 ceph-mon[97033]: pgmap v16905: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:19:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:52 smithi059 ceph-mon[97450]: pgmap v16905: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:19:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:53.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:53.629623+0000) 2022-09-08T06:19:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:53.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:53.629581+0000) 2022-09-08T06:19:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:54.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:54.629798+0000) 2022-09-08T06:19:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:54.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:54.629708+0000) 2022-09-08T06:19:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:54 smithi059 ceph-mon[97450]: pgmap v16906: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:55.036 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:54 smithi055 ceph-mon[97033]: pgmap v16906: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:55.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:55.629943+0000) 2022-09-08T06:19:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:55.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:55.629844+0000) 2022-09-08T06:19:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:19:55.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:19:55.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:19:55.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:19:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:19:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:19:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:19:56.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:19:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:56.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:56.630105+0000) 2022-09-08T06:19:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:56.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:56.630063+0000) 2022-09-08T06:19:57.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:56 smithi059 ceph-mon[97450]: pgmap v16907: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:19:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:56 smithi055 ceph-mon[97033]: pgmap v16907: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:19:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:57.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:57.630262+0000) 2022-09-08T06:19:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:57.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:57.630205+0000) 2022-09-08T06:19:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:58.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:58.630502+0000) 2022-09-08T06:19:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:58.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:58.630435+0000) 2022-09-08T06:19:59.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:19:58 smithi059 ceph-mon[97450]: pgmap v16908: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:19:58 smithi055 ceph-mon[97033]: pgmap v16908: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:19:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:19:59.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:59.630714+0000) 2022-09-08T06:19:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:19:59.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:19:59.630690+0000) 2022-09-08T06:20:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:00.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:00.630911+0000) 2022-09-08T06:20:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:00.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:00.631005+0000) 2022-09-08T06:20:01.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:00 smithi059 ceph-mon[97450]: pgmap v16909: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T06:20:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T06:20:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T06:20:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:00 smithi055 ceph-mon[97033]: pgmap v16909: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T06:20:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T06:20:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T06:20:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:01.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:01.631167+0000) 2022-09-08T06:20:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:01.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:01.631236+0000) 2022-09-08T06:20:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:02.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:02.631416+0000) 2022-09-08T06:20:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:02.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:02.631437+0000) 2022-09-08T06:20:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:02 smithi055 ceph-mon[97033]: pgmap v16910: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:20:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:02 smithi059 ceph-mon[97450]: pgmap v16910: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:20:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:03.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:03.631567+0000) 2022-09-08T06:20:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:03.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:03.631577+0000) 2022-09-08T06:20:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:04.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:04.631810+0000) 2022-09-08T06:20:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:04.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:04.631766+0000) 2022-09-08T06:20:04.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:04 smithi055 ceph-mon[97033]: pgmap v16911: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:05.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:04 smithi059 ceph-mon[97450]: pgmap v16911: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:05.633 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:20:05.184+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:20:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:05.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:05.632022+0000) 2022-09-08T06:20:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:05.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:05.631949+0000) 2022-09-08T06:20:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:06.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:06.632176+0000) 2022-09-08T06:20:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:06.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:06.632095+0000) 2022-09-08T06:20:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:06 smithi059 ceph-mon[97450]: pgmap v16912: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:20:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:20:07.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:20:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:06 smithi055 ceph-mon[97033]: pgmap v16912: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:20:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:20:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:20:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:07.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:07.632357+0000) 2022-09-08T06:20:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:07.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:07.632262+0000) 2022-09-08T06:20:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:08.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:08.632513+0000) 2022-09-08T06:20:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:08.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:08.632405+0000) 2022-09-08T06:20:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:08 smithi059 ceph-mon[97450]: pgmap v16913: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:20:09.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:08 smithi055 ceph-mon[97033]: pgmap v16913: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:20:09.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:09.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:09.632731+0000) 2022-09-08T06:20:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:09.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:09.632599+0000) 2022-09-08T06:20:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:10.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:10.632855+0000) 2022-09-08T06:20:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:10.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:10.632767+0000) 2022-09-08T06:20:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:10 smithi055 ceph-mon[97033]: pgmap v16914: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:20:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:10 smithi055 ceph-mon[97033]: osdmap e1136: 12 total, 11 up, 11 in 2022-09-08T06:20:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:10 smithi059 ceph-mon[97450]: pgmap v16914: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:20:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:10 smithi059 ceph-mon[97450]: osdmap e1136: 12 total, 11 up, 11 in 2022-09-08T06:20:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:11.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:11.632980+0000) 2022-09-08T06:20:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:11.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:11.632905+0000) 2022-09-08T06:20:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:12.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:12.633129+0000) 2022-09-08T06:20:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:12.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:12.633062+0000) 2022-09-08T06:20:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:12 smithi055 ceph-mon[97033]: pgmap v16916: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:20:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:12 smithi059 ceph-mon[97450]: pgmap v16916: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:20:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:13.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:13.633261+0000) 2022-09-08T06:20:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:13.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:13.633213+0000) 2022-09-08T06:20:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:14.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:14.633395+0000) 2022-09-08T06:20:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:14.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:14.633376+0000) 2022-09-08T06:20:15.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:14 smithi059 ceph-mon[97450]: pgmap v16917: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:20:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:14 smithi055 ceph-mon[97033]: pgmap v16917: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:20:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:15.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:15.633515+0000) 2022-09-08T06:20:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:15.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:15.633528+0000) 2022-09-08T06:20:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:16.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:16.633775+0000) 2022-09-08T06:20:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:16.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:16.633723+0000) 2022-09-08T06:20:17.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:16 smithi059 ceph-mon[97450]: pgmap v16918: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:20:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:16 smithi059 ceph-mon[97450]: osdmap e1137: 12 total, 11 up, 11 in 2022-09-08T06:20:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:16 smithi055 ceph-mon[97033]: pgmap v16918: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:20:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:16 smithi055 ceph-mon[97033]: osdmap e1137: 12 total, 11 up, 11 in 2022-09-08T06:20:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:17.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:17.633865+0000) 2022-09-08T06:20:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:17.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:17.633954+0000) 2022-09-08T06:20:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:18.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:18.634045+0000) 2022-09-08T06:20:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:18.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:18.634108+0000) 2022-09-08T06:20:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:18 smithi055 ceph-mon[97033]: pgmap v16920: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:20:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:18 smithi059 ceph-mon[97450]: pgmap v16920: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:20:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:19.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:19.634221+0000) 2022-09-08T06:20:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:19.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:19.634233+0000) 2022-09-08T06:20:20.636 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:20:20.189+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:20:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:20.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:20.634376+0000) 2022-09-08T06:20:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:20.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:20.634385+0000) 2022-09-08T06:20:21.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:20 smithi055 ceph-mon[97033]: pgmap v16921: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:20:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:20 smithi059 ceph-mon[97450]: pgmap v16921: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:20:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:21.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:21.634544+0000) 2022-09-08T06:20:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:21.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:21.634561+0000) 2022-09-08T06:20:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:22.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:22.634683+0000) 2022-09-08T06:20:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:22.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:22.634729+0000) 2022-09-08T06:20:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:22 smithi055 ceph-mon[97033]: pgmap v16922: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:20:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:22 smithi059 ceph-mon[97450]: pgmap v16922: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:20:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:23.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:23.634894+0000) 2022-09-08T06:20:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:23.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:23.634947+0000) 2022-09-08T06:20:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:24.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:24.635155+0000) 2022-09-08T06:20:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:24.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:24.635179+0000) 2022-09-08T06:20:25.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:24 smithi059 ceph-mon[97450]: pgmap v16923: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:20:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:24 smithi055 ceph-mon[97033]: pgmap v16923: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:20:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:25.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:25.635334+0000) 2022-09-08T06:20:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:25.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:25.635366+0000) 2022-09-08T06:20:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:26.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:26.635460+0000) 2022-09-08T06:20:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:26.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:26.635520+0000) 2022-09-08T06:20:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:26 smithi055 ceph-mon[97033]: pgmap v16924: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:20:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:26 smithi059 ceph-mon[97450]: pgmap v16924: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:20:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:27.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:27.635662+0000) 2022-09-08T06:20:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:27.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:27.635706+0000) 2022-09-08T06:20:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:28.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:28.635949+0000) 2022-09-08T06:20:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:28.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:28.635941+0000) 2022-09-08T06:20:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:28 smithi055 ceph-mon[97033]: pgmap v16925: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:20:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:28 smithi059 ceph-mon[97450]: pgmap v16925: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:20:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:29.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:29.636126+0000) 2022-09-08T06:20:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:29.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:29.636087+0000) 2022-09-08T06:20:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:30.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:30.636249+0000) 2022-09-08T06:20:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:30.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:30.636301+0000) 2022-09-08T06:20:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:30 smithi055 ceph-mon[97033]: pgmap v16926: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:30 smithi059 ceph-mon[97450]: pgmap v16926: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:31.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:31.636479+0000) 2022-09-08T06:20:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:31.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:31.636502+0000) 2022-09-08T06:20:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:32.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:32.636677+0000) 2022-09-08T06:20:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:32.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:32.636679+0000) 2022-09-08T06:20:32.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:32 smithi055 ceph-mon[97033]: pgmap v16927: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:20:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:32 smithi059 ceph-mon[97450]: pgmap v16927: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:20:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:33.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:33.636862+0000) 2022-09-08T06:20:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:33.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:33.636874+0000) 2022-09-08T06:20:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:34.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:34.637029+0000) 2022-09-08T06:20:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:34.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:34.637013+0000) 2022-09-08T06:20:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:34 smithi055 ceph-mon[97033]: pgmap v16928: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:34 smithi059 ceph-mon[97450]: pgmap v16928: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:35.639 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:20:35.203+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:20:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:35.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:35.637356+0000) 2022-09-08T06:20:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:35.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:35.637211+0000) 2022-09-08T06:20:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:36.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:36.637527+0000) 2022-09-08T06:20:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:36.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:36.637376+0000) 2022-09-08T06:20:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:36 smithi055 ceph-mon[97033]: pgmap v16929: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:20:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:36 smithi059 ceph-mon[97450]: pgmap v16929: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:20:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:37.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:37.637692+0000) 2022-09-08T06:20:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:37.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:37.637516+0000) 2022-09-08T06:20:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:38.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:38.637894+0000) 2022-09-08T06:20:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:38.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:38.637776+0000) 2022-09-08T06:20:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:38 smithi055 ceph-mon[97033]: pgmap v16930: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:38 smithi059 ceph-mon[97450]: pgmap v16930: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:39.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:39.637982+0000) 2022-09-08T06:20:39.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:39.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:39.638118+0000) 2022-09-08T06:20:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:40.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:40.638347+0000) 2022-09-08T06:20:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:40.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:40.638241+0000) 2022-09-08T06:20:41.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:40 smithi055 ceph-mon[97033]: pgmap v16931: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:40 smithi059 ceph-mon[97450]: pgmap v16931: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:41.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:41.638483+0000) 2022-09-08T06:20:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:41.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:41.638448+0000) 2022-09-08T06:20:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:42.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:42.638647+0000) 2022-09-08T06:20:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:42.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:42.638647+0000) 2022-09-08T06:20:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:42 smithi055 ceph-mon[97033]: pgmap v16932: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:20:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:42 smithi059 ceph-mon[97450]: pgmap v16932: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:20:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:43.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:43.638860+0000) 2022-09-08T06:20:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:43.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:43.638829+0000) 2022-09-08T06:20:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:44.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:44.639044+0000) 2022-09-08T06:20:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:44.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:44.639048+0000) 2022-09-08T06:20:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:44 smithi055 ceph-mon[97033]: pgmap v16933: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:44 smithi059 ceph-mon[97450]: pgmap v16933: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:45.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:45.639304+0000) 2022-09-08T06:20:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:45.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:45.639215+0000) 2022-09-08T06:20:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:46.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:46.639366+0000) 2022-09-08T06:20:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:46.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:46.639492+0000) 2022-09-08T06:20:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:46 smithi055 ceph-mon[97033]: pgmap v16934: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:20:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:46 smithi059 ceph-mon[97450]: pgmap v16934: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:20:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:47.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:47.639533+0000) 2022-09-08T06:20:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:47.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:47.639730+0000) 2022-09-08T06:20:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:48.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:48.639687+0000) 2022-09-08T06:20:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:48.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:48.639868+0000) 2022-09-08T06:20:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:48 smithi055 ceph-mon[97033]: pgmap v16935: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:48 smithi059 ceph-mon[97450]: pgmap v16935: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:49.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:49.639897+0000) 2022-09-08T06:20:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:49.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:49.640075+0000) 2022-09-08T06:20:50.642 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:20:50.205+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:20:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:50.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:50.640094+0000) 2022-09-08T06:20:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:50.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:50.640256+0000) 2022-09-08T06:20:51.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:50 smithi055 ceph-mon[97033]: pgmap v16936: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:50 smithi059 ceph-mon[97450]: pgmap v16936: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:51.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:51.640406+0000) 2022-09-08T06:20:51.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:51.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:51.640262+0000) 2022-09-08T06:20:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:52.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:52.640565+0000) 2022-09-08T06:20:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:52.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:52.640401+0000) 2022-09-08T06:20:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:52 smithi055 ceph-mon[97033]: pgmap v16937: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:20:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:52 smithi059 ceph-mon[97450]: pgmap v16937: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:20:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:53.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:53.640562+0000) 2022-09-08T06:20:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:53.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:53.640743+0000) 2022-09-08T06:20:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:54.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:54.640771+0000) 2022-09-08T06:20:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:54.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:54.640913+0000) 2022-09-08T06:20:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:54 smithi055 ceph-mon[97033]: pgmap v16938: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:54 smithi059 ceph-mon[97450]: pgmap v16938: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:55.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:55.640992+0000) 2022-09-08T06:20:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:55.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:55.641128+0000) 2022-09-08T06:20:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:20:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:20:55.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:20:56.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:20:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:20:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:20:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:56.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:56.641157+0000) 2022-09-08T06:20:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:56.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:56.641298+0000) 2022-09-08T06:20:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:57 smithi055 ceph-mon[97033]: pgmap v16939: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:20:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:20:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:57 smithi059 ceph-mon[97450]: pgmap v16939: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:20:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:20:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:57.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:57.641304+0000) 2022-09-08T06:20:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:57.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:57.641436+0000) 2022-09-08T06:20:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:58.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:58.641524+0000) 2022-09-08T06:20:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:58.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:58.641670+0000) 2022-09-08T06:20:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:20:59 smithi055 ceph-mon[97033]: pgmap v16940: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:20:59 smithi059 ceph-mon[97450]: pgmap v16940: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:20:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:20:59.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:59.641795+0000) 2022-09-08T06:20:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:20:59.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:20:59.641845+0000) 2022-09-08T06:21:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:00.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:00.641967+0000) 2022-09-08T06:21:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:00.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:00.642017+0000) 2022-09-08T06:21:01.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:01 smithi055 ceph-mon[97033]: pgmap v16941: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:01 smithi059 ceph-mon[97450]: pgmap v16941: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:01.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:01.642222+0000) 2022-09-08T06:21:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:01.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:01.642186+0000) 2022-09-08T06:21:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:02.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:02.642418+0000) 2022-09-08T06:21:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:02.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:02.642373+0000) 2022-09-08T06:21:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:02 smithi055 ceph-mon[97033]: pgmap v16942: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:21:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:02 smithi059 ceph-mon[97450]: pgmap v16942: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:21:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:03.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:03.642676+0000) 2022-09-08T06:21:03.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:03.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:03.642627+0000) 2022-09-08T06:21:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:04.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:04.642930+0000) 2022-09-08T06:21:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:04.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:04.642900+0000) 2022-09-08T06:21:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:04 smithi055 ceph-mon[97033]: pgmap v16943: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:04 smithi059 ceph-mon[97450]: pgmap v16943: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:05.645 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:21:05.217+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:21:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:05.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:05.643120+0000) 2022-09-08T06:21:05.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:05.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:05.643097+0000) 2022-09-08T06:21:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:06.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:06.643262+0000) 2022-09-08T06:21:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:06.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:06.643248+0000) 2022-09-08T06:21:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:06 smithi055 ceph-mon[97033]: pgmap v16944: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:21:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:21:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:21:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:06 smithi059 ceph-mon[97450]: pgmap v16944: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:21:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:21:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:21:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:07.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:07.643422+0000) 2022-09-08T06:21:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:07.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:07.643381+0000) 2022-09-08T06:21:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:08.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:08.643669+0000) 2022-09-08T06:21:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:08.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:08.643550+0000) 2022-09-08T06:21:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:08 smithi055 ceph-mon[97033]: pgmap v16945: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:21:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:08 smithi059 ceph-mon[97450]: pgmap v16945: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:21:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:09.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:09.643829+0000) 2022-09-08T06:21:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:09.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:09.643709+0000) 2022-09-08T06:21:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:10.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:10.643980+0000) 2022-09-08T06:21:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:10.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:10.644057+0000) 2022-09-08T06:21:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:10 smithi055 ceph-mon[97033]: pgmap v16946: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:21:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:10 smithi055 ceph-mon[97033]: osdmap e1138: 12 total, 11 up, 11 in 2022-09-08T06:21:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:10 smithi059 ceph-mon[97450]: pgmap v16946: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:21:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:10 smithi059 ceph-mon[97450]: osdmap e1138: 12 total, 11 up, 11 in 2022-09-08T06:21:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:11.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:11.644225+0000) 2022-09-08T06:21:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:11.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:11.644140+0000) 2022-09-08T06:21:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:12.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:12.644363+0000) 2022-09-08T06:21:12.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:12.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:12.644299+0000) 2022-09-08T06:21:12.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:12 smithi055 ceph-mon[97033]: pgmap v16948: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:21:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:12 smithi059 ceph-mon[97450]: pgmap v16948: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:21:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:13.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:13.644535+0000) 2022-09-08T06:21:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:13.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:13.644508+0000) 2022-09-08T06:21:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:14.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:14.644723+0000) 2022-09-08T06:21:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:14.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:14.644693+0000) 2022-09-08T06:21:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:14 smithi055 ceph-mon[97033]: pgmap v16949: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:21:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:14 smithi059 ceph-mon[97450]: pgmap v16949: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:21:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:15.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:15.644929+0000) 2022-09-08T06:21:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:15.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:15.644900+0000) 2022-09-08T06:21:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:16.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:16.645178+0000) 2022-09-08T06:21:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:16.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:16.645159+0000) 2022-09-08T06:21:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:16 smithi055 ceph-mon[97033]: pgmap v16950: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:21:17.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:16 smithi055 ceph-mon[97033]: osdmap e1139: 12 total, 11 up, 11 in 2022-09-08T06:21:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:16 smithi059 ceph-mon[97450]: pgmap v16950: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:21:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:16 smithi059 ceph-mon[97450]: osdmap e1139: 12 total, 11 up, 11 in 2022-09-08T06:21:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:17.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:17.645376+0000) 2022-09-08T06:21:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:17.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:17.645341+0000) 2022-09-08T06:21:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:18.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:18.645498+0000) 2022-09-08T06:21:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:18.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:18.645533+0000) 2022-09-08T06:21:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:18 smithi055 ceph-mon[97033]: pgmap v16952: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:21:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:18 smithi059 ceph-mon[97450]: pgmap v16952: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:21:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:19.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:19.645679+0000) 2022-09-08T06:21:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:19.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:19.645649+0000) 2022-09-08T06:21:20.647 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:21:20.232+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:21:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:20.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:20.645820+0000) 2022-09-08T06:21:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:20.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:20.645902+0000) 2022-09-08T06:21:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:20 smithi055 ceph-mon[97033]: pgmap v16953: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:21:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:20 smithi059 ceph-mon[97450]: pgmap v16953: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:21:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:21.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:21.645947+0000) 2022-09-08T06:21:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:21.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:21.646021+0000) 2022-09-08T06:21:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:22.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:22.646069+0000) 2022-09-08T06:21:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:22.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:22.646170+0000) 2022-09-08T06:21:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:22 smithi055 ceph-mon[97033]: pgmap v16954: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:21:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:22 smithi059 ceph-mon[97450]: pgmap v16954: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:21:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:23.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:23.646230+0000) 2022-09-08T06:21:23.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:23.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:23.646329+0000) 2022-09-08T06:21:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:24.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:24.646365+0000) 2022-09-08T06:21:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:24.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:24.646481+0000) 2022-09-08T06:21:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:24 smithi055 ceph-mon[97033]: pgmap v16955: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:21:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:24 smithi059 ceph-mon[97450]: pgmap v16955: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:21:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:25.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:25.646504+0000) 2022-09-08T06:21:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:25.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:25.646655+0000) 2022-09-08T06:21:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:26.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:26.646697+0000) 2022-09-08T06:21:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:26.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:26.646814+0000) 2022-09-08T06:21:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:26 smithi055 ceph-mon[97033]: pgmap v16956: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T06:21:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:26 smithi059 ceph-mon[97450]: pgmap v16956: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T06:21:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:27.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:27.646857+0000) 2022-09-08T06:21:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:27.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:27.646974+0000) 2022-09-08T06:21:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:28.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:28.647069+0000) 2022-09-08T06:21:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:28.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:28.647103+0000) 2022-09-08T06:21:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:28 smithi055 ceph-mon[97033]: pgmap v16957: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:21:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:28 smithi059 ceph-mon[97450]: pgmap v16957: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:21:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:29.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:29.647318+0000) 2022-09-08T06:21:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:29.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:29.647369+0000) 2022-09-08T06:21:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:30.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:30.647509+0000) 2022-09-08T06:21:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:30.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:30.647582+0000) 2022-09-08T06:21:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:30 smithi055 ceph-mon[97033]: pgmap v16958: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T06:21:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:30 smithi059 ceph-mon[97450]: pgmap v16958: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T06:21:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:31.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:31.647724+0000) 2022-09-08T06:21:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:31.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:31.647752+0000) 2022-09-08T06:21:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:32.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:32.647950+0000) 2022-09-08T06:21:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:32.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:32.647941+0000) 2022-09-08T06:21:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:32 smithi055 ceph-mon[97033]: pgmap v16959: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T06:21:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:32 smithi059 ceph-mon[97450]: pgmap v16959: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T06:21:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:33.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:33.648115+0000) 2022-09-08T06:21:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:33.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:33.648109+0000) 2022-09-08T06:21:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:34.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:34.648304+0000) 2022-09-08T06:21:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:34.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:34.648291+0000) 2022-09-08T06:21:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:34 smithi055 ceph-mon[97033]: pgmap v16960: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:34 smithi059 ceph-mon[97450]: pgmap v16960: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:35.650 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:21:35.245+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:21:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:35.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:35.648524+0000) 2022-09-08T06:21:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:35.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:35.648523+0000) 2022-09-08T06:21:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:36.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:36.648709+0000) 2022-09-08T06:21:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:36.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:36.648709+0000) 2022-09-08T06:21:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:36 smithi055 ceph-mon[97033]: pgmap v16961: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:21:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:36 smithi059 ceph-mon[97450]: pgmap v16961: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:21:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:37.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:37.648922+0000) 2022-09-08T06:21:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:37.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:37.648882+0000) 2022-09-08T06:21:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:38.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:38.649155+0000) 2022-09-08T06:21:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:38.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:38.649130+0000) 2022-09-08T06:21:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:38 smithi055 ceph-mon[97033]: pgmap v16962: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:38 smithi059 ceph-mon[97450]: pgmap v16962: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:39.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:39.649415+0000) 2022-09-08T06:21:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:39.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:39.649333+0000) 2022-09-08T06:21:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:40.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:40.649588+0000) 2022-09-08T06:21:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:40.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:40.649541+0000) 2022-09-08T06:21:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:40 smithi055 ceph-mon[97033]: pgmap v16963: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:40 smithi059 ceph-mon[97450]: pgmap v16963: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:41.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:41.649729+0000) 2022-09-08T06:21:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:41.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:41.649702+0000) 2022-09-08T06:21:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:42.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:42.649834+0000) 2022-09-08T06:21:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:42.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:42.649880+0000) 2022-09-08T06:21:42.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:42 smithi055 ceph-mon[97033]: pgmap v16964: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:21:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:42 smithi059 ceph-mon[97450]: pgmap v16964: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:21:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:43.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:43.650064+0000) 2022-09-08T06:21:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:43.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:43.649995+0000) 2022-09-08T06:21:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:44.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:44.650243+0000) 2022-09-08T06:21:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:44.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:44.650184+0000) 2022-09-08T06:21:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:44 smithi055 ceph-mon[97033]: pgmap v16965: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:44 smithi059 ceph-mon[97450]: pgmap v16965: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:45.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:45.650403+0000) 2022-09-08T06:21:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:45.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:45.650319+0000) 2022-09-08T06:21:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:46.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:46.650614+0000) 2022-09-08T06:21:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:46.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:46.650487+0000) 2022-09-08T06:21:47.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:46 smithi055 ceph-mon[97033]: pgmap v16966: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:21:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:46 smithi059 ceph-mon[97450]: pgmap v16966: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:21:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:47.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:47.650844+0000) 2022-09-08T06:21:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:47.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:47.650714+0000) 2022-09-08T06:21:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:48.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:48.650907+0000) 2022-09-08T06:21:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:48.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:48.651035+0000) 2022-09-08T06:21:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:48 smithi055 ceph-mon[97033]: pgmap v16967: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:48 smithi059 ceph-mon[97450]: pgmap v16967: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:49.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:49.651270+0000) 2022-09-08T06:21:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:49.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:49.651112+0000) 2022-09-08T06:21:50.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:21:50.246+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:21:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:50.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:50.651512+0000) 2022-09-08T06:21:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:50.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:50.651294+0000) 2022-09-08T06:21:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:50 smithi055 ceph-mon[97033]: pgmap v16968: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:50 smithi059 ceph-mon[97450]: pgmap v16968: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:51.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:51.651658+0000) 2022-09-08T06:21:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:51.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:51.651428+0000) 2022-09-08T06:21:52.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:52 smithi055 ceph-mon[97033]: pgmap v16969: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:21:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:52 smithi059 ceph-mon[97450]: pgmap v16969: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:21:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:52.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:52.651789+0000) 2022-09-08T06:21:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:52.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:52.651697+0000) 2022-09-08T06:21:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:53.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:53.651951+0000) 2022-09-08T06:21:54.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:53.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:53.651845+0000) 2022-09-08T06:21:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:54.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:54.652110+0000) 2022-09-08T06:21:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:54.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:54.652023+0000) 2022-09-08T06:21:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:54 smithi055 ceph-mon[97033]: pgmap v16970: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:54 smithi059 ceph-mon[97450]: pgmap v16970: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:55.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:55.652314+0000) 2022-09-08T06:21:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:55.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:55.652240+0000) 2022-09-08T06:21:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:56.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:56.652492+0000) 2022-09-08T06:21:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:56.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:56.652395+0000) 2022-09-08T06:21:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:56 smithi055 ceph-mon[97033]: pgmap v16971: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:21:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:21:57.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:21:57.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:21:57.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:21:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:56 smithi059 ceph-mon[97450]: pgmap v16971: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:21:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:21:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:21:57.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:21:57.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:21:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:57.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:57.652693+0000) 2022-09-08T06:21:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:57.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:57.652629+0000) 2022-09-08T06:21:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:58.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:58.652873+0000) 2022-09-08T06:21:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:58.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:58.652851+0000) 2022-09-08T06:21:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:21:58 smithi055 ceph-mon[97033]: pgmap v16972: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:21:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:21:58 smithi059 ceph-mon[97450]: pgmap v16972: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:22:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:21:59.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:59.653073+0000) 2022-09-08T06:22:00.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:21:59.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:21:59.653060+0000) 2022-09-08T06:22:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:00.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:00.653249+0000) 2022-09-08T06:22:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:00.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:00.653268+0000) 2022-09-08T06:22:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:00 smithi055 ceph-mon[97033]: pgmap v16973: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:22:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:00 smithi059 ceph-mon[97450]: pgmap v16973: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:22:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:01.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:01.653412+0000) 2022-09-08T06:22:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:01.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:01.653448+0000) 2022-09-08T06:22:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:02 smithi055 ceph-mon[97033]: pgmap v16974: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:22:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:02 smithi059 ceph-mon[97450]: pgmap v16974: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:22:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:02.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:02.653708+0000) 2022-09-08T06:22:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:02.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:02.653630+0000) 2022-09-08T06:22:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:03.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:03.653854+0000) 2022-09-08T06:22:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:03.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:03.653931+0000) 2022-09-08T06:22:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:04.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:04.654142+0000) 2022-09-08T06:22:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:04.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:04.654214+0000) 2022-09-08T06:22:05.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:04 smithi055 ceph-mon[97033]: pgmap v16975: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:22:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:04 smithi059 ceph-mon[97450]: pgmap v16975: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:22:05.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:22:05.254+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:22:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:05.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:05.654403+0000) 2022-09-08T06:22:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:05.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:05.654381+0000) 2022-09-08T06:22:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:06.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:06.654540+0000) 2022-09-08T06:22:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:06.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:06.654546+0000) 2022-09-08T06:22:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:06 smithi055 ceph-mon[97033]: pgmap v16976: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:22:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:22:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:22:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:06 smithi059 ceph-mon[97450]: pgmap v16976: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:22:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:22:07.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:22:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:07.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:07.654711+0000) 2022-09-08T06:22:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:07.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:07.654696+0000) 2022-09-08T06:22:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:08.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:08.654858+0000) 2022-09-08T06:22:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:08.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:08.654868+0000) 2022-09-08T06:22:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:08 smithi055 ceph-mon[97033]: pgmap v16977: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:22:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:08 smithi059 ceph-mon[97450]: pgmap v16977: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:22:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:09.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:09.654989+0000) 2022-09-08T06:22:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:09.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:09.655051+0000) 2022-09-08T06:22:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:10.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:10.655304+0000) 2022-09-08T06:22:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:10.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:10.655223+0000) 2022-09-08T06:22:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:10 smithi055 ceph-mon[97033]: pgmap v16978: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:22:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:10 smithi055 ceph-mon[97033]: osdmap e1140: 12 total, 11 up, 11 in 2022-09-08T06:22:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:10 smithi059 ceph-mon[97450]: pgmap v16978: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:22:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:10 smithi059 ceph-mon[97450]: osdmap e1140: 12 total, 11 up, 11 in 2022-09-08T06:22:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:11.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:11.655433+0000) 2022-09-08T06:22:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:11.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:11.655446+0000) 2022-09-08T06:22:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:12 smithi055 ceph-mon[97033]: pgmap v16980: 129 pgs: 2 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:22:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:12 smithi059 ceph-mon[97450]: pgmap v16980: 129 pgs: 2 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:22:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:12.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:12.655705+0000) 2022-09-08T06:22:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:12.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:12.655679+0000) 2022-09-08T06:22:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:13.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:13.655857+0000) 2022-09-08T06:22:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:13.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:13.655858+0000) 2022-09-08T06:22:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:14.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:14.656037+0000) 2022-09-08T06:22:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:14.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:14.656022+0000) 2022-09-08T06:22:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:14 smithi055 ceph-mon[97033]: pgmap v16981: 129 pgs: 2 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:22:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:14 smithi059 ceph-mon[97450]: pgmap v16981: 129 pgs: 2 active+clean+snaptrim_wait, 127 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:22:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:15.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:15.656192+0000) 2022-09-08T06:22:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:15.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:15.656195+0000) 2022-09-08T06:22:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:16.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:16.656371+0000) 2022-09-08T06:22:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:16.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:16.656389+0000) 2022-09-08T06:22:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:16 smithi055 ceph-mon[97033]: pgmap v16982: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:22:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:16 smithi055 ceph-mon[97033]: osdmap e1141: 12 total, 11 up, 11 in 2022-09-08T06:22:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:16 smithi059 ceph-mon[97450]: pgmap v16982: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:22:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:16 smithi059 ceph-mon[97450]: osdmap e1141: 12 total, 11 up, 11 in 2022-09-08T06:22:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:17.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:17.656601+0000) 2022-09-08T06:22:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:17.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:17.656622+0000) 2022-09-08T06:22:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:18.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:18.656813+0000) 2022-09-08T06:22:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:18.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:18.656766+0000) 2022-09-08T06:22:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:18 smithi055 ceph-mon[97033]: pgmap v16984: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:22:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:18 smithi059 ceph-mon[97450]: pgmap v16984: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:22:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:19.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:19.656946+0000) 2022-09-08T06:22:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:19.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:19.656931+0000) 2022-09-08T06:22:20.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:22:20.267+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:22:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:20.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:20.657145+0000) 2022-09-08T06:22:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:20.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:20.657164+0000) 2022-09-08T06:22:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:20 smithi055 ceph-mon[97033]: pgmap v16985: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:22:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:20 smithi059 ceph-mon[97450]: pgmap v16985: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:22:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:21.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:21.657346+0000) 2022-09-08T06:22:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:21.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:21.657302+0000) 2022-09-08T06:22:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:22 smithi055 ceph-mon[97033]: pgmap v16986: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:22:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:22 smithi059 ceph-mon[97450]: pgmap v16986: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:22:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:22.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:22.657458+0000) 2022-09-08T06:22:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:22.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:22.657497+0000) 2022-09-08T06:22:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:23.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:23.657662+0000) 2022-09-08T06:22:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:23.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:23.657700+0000) 2022-09-08T06:22:25.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:24.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:24.657844+0000) 2022-09-08T06:22:25.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:24.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:24.657855+0000) 2022-09-08T06:22:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:24 smithi055 ceph-mon[97033]: pgmap v16987: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:22:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:24 smithi059 ceph-mon[97450]: pgmap v16987: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:22:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:25.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:25.657988+0000) 2022-09-08T06:22:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:25.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:25.658031+0000) 2022-09-08T06:22:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:26.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:26.658176+0000) 2022-09-08T06:22:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:26.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:26.658121+0000) 2022-09-08T06:22:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:26 smithi055 ceph-mon[97033]: pgmap v16988: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:22:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:26 smithi059 ceph-mon[97450]: pgmap v16988: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:22:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:27.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:27.658306+0000) 2022-09-08T06:22:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:27.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:27.658387+0000) 2022-09-08T06:22:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:28.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:28.658521+0000) 2022-09-08T06:22:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:28.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:28.658623+0000) 2022-09-08T06:22:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:28 smithi055 ceph-mon[97033]: pgmap v16989: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T06:22:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:28 smithi059 ceph-mon[97450]: pgmap v16989: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T06:22:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:29.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:29.658730+0000) 2022-09-08T06:22:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:29.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:29.658845+0000) 2022-09-08T06:22:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:30.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:30.658872+0000) 2022-09-08T06:22:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:30.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:30.659026+0000) 2022-09-08T06:22:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:30 smithi055 ceph-mon[97033]: pgmap v16990: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:22:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:30 smithi059 ceph-mon[97450]: pgmap v16990: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:22:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:31.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:31.659192+0000) 2022-09-08T06:22:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:31.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:31.659075+0000) 2022-09-08T06:22:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:32 smithi055 ceph-mon[97033]: pgmap v16991: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:22:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:32 smithi059 ceph-mon[97450]: pgmap v16991: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:22:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:32.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:32.659378+0000) 2022-09-08T06:22:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:32.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:32.659332+0000) 2022-09-08T06:22:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:33.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:33.659521+0000) 2022-09-08T06:22:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:33.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:33.659495+0000) 2022-09-08T06:22:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:34.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:34.659742+0000) 2022-09-08T06:22:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:34.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:34.659769+0000) 2022-09-08T06:22:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:34 smithi055 ceph-mon[97033]: pgmap v16992: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:22:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:34 smithi059 ceph-mon[97450]: pgmap v16992: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:22:35.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:22:35.274+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:22:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:35.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:35.659995+0000) 2022-09-08T06:22:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:35.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:35.660000+0000) 2022-09-08T06:22:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:36.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:36.660269+0000) 2022-09-08T06:22:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:36.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:36.660269+0000) 2022-09-08T06:22:37.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:36 smithi055 ceph-mon[97033]: pgmap v16993: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:22:37.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:36 smithi059 ceph-mon[97450]: pgmap v16993: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:22:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:37.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:37.660485+0000) 2022-09-08T06:22:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:37.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:37.660494+0000) 2022-09-08T06:22:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:38.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:38.660696+0000) 2022-09-08T06:22:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:38.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:38.660695+0000) 2022-09-08T06:22:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:38 smithi055 ceph-mon[97033]: pgmap v16994: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:22:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:38 smithi059 ceph-mon[97450]: pgmap v16994: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:22:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:39.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:39.660840+0000) 2022-09-08T06:22:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:39.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:39.660847+0000) 2022-09-08T06:22:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:40.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:40.661001+0000) 2022-09-08T06:22:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:40.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:40.661024+0000) 2022-09-08T06:22:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:40 smithi055 ceph-mon[97033]: pgmap v16995: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:22:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:40 smithi059 ceph-mon[97450]: pgmap v16995: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:22:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:41.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:41.661138+0000) 2022-09-08T06:22:42.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:41.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:41.661172+0000) 2022-09-08T06:22:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:42 smithi055 ceph-mon[97033]: pgmap v16996: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:22:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:42 smithi059 ceph-mon[97450]: pgmap v16996: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:22:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:42.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:42.661312+0000) 2022-09-08T06:22:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:42.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:42.661370+0000) 2022-09-08T06:22:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:43.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:43.661477+0000) 2022-09-08T06:22:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:43.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:43.661493+0000) 2022-09-08T06:22:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:44.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:44.661621+0000) 2022-09-08T06:22:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:44.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:44.661622+0000) 2022-09-08T06:22:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:44 smithi055 ceph-mon[97033]: pgmap v16997: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:22:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:44 smithi059 ceph-mon[97450]: pgmap v16997: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:22:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:45.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:45.661873+0000) 2022-09-08T06:22:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:45.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:45.661862+0000) 2022-09-08T06:22:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:46.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:46.662040+0000) 2022-09-08T06:22:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:46.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:46.662065+0000) 2022-09-08T06:22:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:46 smithi055 ceph-mon[97033]: pgmap v16998: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:22:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:46 smithi059 ceph-mon[97450]: pgmap v16998: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:22:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:47.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:47.662177+0000) 2022-09-08T06:22:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:47.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:47.662227+0000) 2022-09-08T06:22:49.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:48.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:48.662307+0000) 2022-09-08T06:22:49.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:48.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:48.662393+0000) 2022-09-08T06:22:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:48 smithi055 ceph-mon[97033]: pgmap v16999: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:22:49.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:48 smithi059 ceph-mon[97450]: pgmap v16999: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:22:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:49.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:49.662519+0000) 2022-09-08T06:22:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:49.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:49.662656+0000) 2022-09-08T06:22:50.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:22:50.281+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:22:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:50.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:50.662739+0000) 2022-09-08T06:22:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:50.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:50.662825+0000) 2022-09-08T06:22:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:50 smithi055 ceph-mon[97033]: pgmap v17000: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:22:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:50 smithi059 ceph-mon[97450]: pgmap v17000: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:22:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:51.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:51.662965+0000) 2022-09-08T06:22:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:51.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:51.662895+0000) 2022-09-08T06:22:52.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:52 smithi055 ceph-mon[97033]: pgmap v17001: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:22:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:52 smithi059 ceph-mon[97450]: pgmap v17001: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:22:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:52.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:52.663181+0000) 2022-09-08T06:22:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:52.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:52.663086+0000) 2022-09-08T06:22:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:53.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:53.663260+0000) 2022-09-08T06:22:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:53.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:53.663389+0000) 2022-09-08T06:22:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:54.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:54.663458+0000) 2022-09-08T06:22:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:54.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:54.663554+0000) 2022-09-08T06:22:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:54 smithi055 ceph-mon[97033]: pgmap v17002: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:22:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:54 smithi059 ceph-mon[97450]: pgmap v17002: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:22:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:55.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:55.663694+0000) 2022-09-08T06:22:56.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:55.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:55.663734+0000) 2022-09-08T06:22:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:56.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:56.663948+0000) 2022-09-08T06:22:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:56.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:56.663959+0000) 2022-09-08T06:22:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:56 smithi055 ceph-mon[97033]: pgmap v17003: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:22:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:56 smithi059 ceph-mon[97450]: pgmap v17003: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:22:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:57.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:57.664122+0000) 2022-09-08T06:22:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:57.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:57.664122+0000) 2022-09-08T06:22:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:22:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:22:58.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:22:58.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:22:58.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:22:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:22:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:22:58.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:22:58.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:22:58.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:22:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:58.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:58.664289+0000) 2022-09-08T06:22:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:58.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:58.664289+0000) 2022-09-08T06:22:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:22:58 smithi055 ceph-mon[97033]: pgmap v17004: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:22:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:22:58 smithi059 ceph-mon[97450]: pgmap v17004: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:22:59.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:59.664503+0000) 2022-09-08T06:23:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:22:59.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:22:59.664503+0000) 2022-09-08T06:23:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:23:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:23:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: pgmap v17005: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:23:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:23:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:00.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:00.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 186.5M 2022-09-08T06:23:00.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 195604943: error parsing value: Value '195604943' is below minimum 939524096 2022-09-08T06:23:00.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:00.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:00.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:00.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:00.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:23:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:23:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:23:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: pgmap v17005: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:01.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:23:01.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:23:01.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:01.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:01.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:01.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:01.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 186.5M 2022-09-08T06:23:01.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 195604943: error parsing value: Value '195604943' is below minimum 939524096 2022-09-08T06:23:01.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:01.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:01.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:01.041 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:23:01.042 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:23:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:00.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:00.664661+0000) 2022-09-08T06:23:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:00.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:00.664688+0000) 2022-09-08T06:23:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:01 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T06:23:01.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:01 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T06:23:02.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:01 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T06:23:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:01 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T06:23:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:01.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:01.664852+0000) 2022-09-08T06:23:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:01.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:01.664911+0000) 2022-09-08T06:23:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:02 smithi055 ceph-mon[97033]: pgmap v17006: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:23:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:02 smithi059 ceph-mon[97450]: pgmap v17006: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:23:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:02.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:02.665014+0000) 2022-09-08T06:23:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:02.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:02.665092+0000) 2022-09-08T06:23:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:03.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:03.665250+0000) 2022-09-08T06:23:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:03.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:03.665297+0000) 2022-09-08T06:23:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:04.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:04.665440+0000) 2022-09-08T06:23:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:04.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:04.665482+0000) 2022-09-08T06:23:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:04 smithi055 ceph-mon[97033]: pgmap v17007: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:04 smithi059 ceph-mon[97450]: pgmap v17007: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:05.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:23:05.291+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:23:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:05.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:05.665594+0000) 2022-09-08T06:23:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:05.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:05.665653+0000) 2022-09-08T06:23:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:23:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:23:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:06.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:06.665710+0000) 2022-09-08T06:23:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:06.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:06.665816+0000) 2022-09-08T06:23:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:06 smithi055 ceph-mon[97033]: pgmap v17008: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:23:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:23:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:06 smithi059 ceph-mon[97450]: pgmap v17008: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:23:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:23:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:07.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:07.665862+0000) 2022-09-08T06:23:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:07.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:07.665987+0000) 2022-09-08T06:23:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:08.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:08.666015+0000) 2022-09-08T06:23:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:08.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:08.666143+0000) 2022-09-08T06:23:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:08 smithi055 ceph-mon[97033]: pgmap v17009: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:23:09.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:08 smithi059 ceph-mon[97450]: pgmap v17009: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:23:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:09.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:09.666175+0000) 2022-09-08T06:23:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:09.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:09.666302+0000) 2022-09-08T06:23:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:10.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:10.666482+0000) 2022-09-08T06:23:11.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:10.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:10.666344+0000) 2022-09-08T06:23:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:10 smithi055 ceph-mon[97033]: pgmap v17010: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:23:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:10 smithi055 ceph-mon[97033]: osdmap e1142: 12 total, 11 up, 11 in 2022-09-08T06:23:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:10 smithi059 ceph-mon[97450]: pgmap v17010: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:23:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:10 smithi059 ceph-mon[97450]: osdmap e1142: 12 total, 11 up, 11 in 2022-09-08T06:23:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:11.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:11.666701+0000) 2022-09-08T06:23:12.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:11.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:11.666555+0000) 2022-09-08T06:23:12.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:12 smithi055 ceph-mon[97033]: pgmap v17012: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:23:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:12 smithi059 ceph-mon[97450]: pgmap v17012: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:23:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:12.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:12.666806+0000) 2022-09-08T06:23:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:12.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:12.666871+0000) 2022-09-08T06:23:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:13.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:13.666975+0000) 2022-09-08T06:23:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:13.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:13.667013+0000) 2022-09-08T06:23:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:14.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:14.667169+0000) 2022-09-08T06:23:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:14.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:14.667162+0000) 2022-09-08T06:23:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:14 smithi055 ceph-mon[97033]: pgmap v17013: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:23:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:14 smithi059 ceph-mon[97450]: pgmap v17013: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:23:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:15.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:15.667336+0000) 2022-09-08T06:23:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:15.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:15.667338+0000) 2022-09-08T06:23:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:16.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:16.667582+0000) 2022-09-08T06:23:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:16.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:16.667583+0000) 2022-09-08T06:23:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:16 smithi055 ceph-mon[97033]: pgmap v17014: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:23:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:16 smithi055 ceph-mon[97033]: osdmap e1143: 12 total, 11 up, 11 in 2022-09-08T06:23:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:16 smithi059 ceph-mon[97450]: pgmap v17014: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:23:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:16 smithi059 ceph-mon[97450]: osdmap e1143: 12 total, 11 up, 11 in 2022-09-08T06:23:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:17.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:17.667824+0000) 2022-09-08T06:23:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:17.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:17.667769+0000) 2022-09-08T06:23:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:18.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:18.667932+0000) 2022-09-08T06:23:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:18.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:18.667966+0000) 2022-09-08T06:23:19.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:18 smithi055 ceph-mon[97033]: pgmap v17016: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:23:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:18 smithi059 ceph-mon[97450]: pgmap v17016: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:23:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:19.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:19.668114+0000) 2022-09-08T06:23:20.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:19.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:19.668092+0000) 2022-09-08T06:23:20.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:23:20.302+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:23:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:20.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:20.668306+0000) 2022-09-08T06:23:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:20.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:20.668272+0000) 2022-09-08T06:23:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:20 smithi055 ceph-mon[97033]: pgmap v17017: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 103 B/s wr, 5 op/s 2022-09-08T06:23:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:20 smithi059 ceph-mon[97450]: pgmap v17017: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 103 B/s wr, 5 op/s 2022-09-08T06:23:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:21.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:21.668506+0000) 2022-09-08T06:23:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:21.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:21.668499+0000) 2022-09-08T06:23:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:22 smithi055 ceph-mon[97033]: pgmap v17018: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:23:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:22 smithi059 ceph-mon[97450]: pgmap v17018: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:23:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:22.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:22.668692+0000) 2022-09-08T06:23:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:22.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:22.668714+0000) 2022-09-08T06:23:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:23.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:23.668819+0000) 2022-09-08T06:23:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:23.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:23.668861+0000) 2022-09-08T06:23:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:24.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:24.668966+0000) 2022-09-08T06:23:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:24.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:24.669007+0000) 2022-09-08T06:23:25.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:24 smithi055 ceph-mon[97033]: pgmap v17019: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:23:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:24 smithi059 ceph-mon[97450]: pgmap v17019: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:23:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:25.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:25.669091+0000) 2022-09-08T06:23:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:25.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:25.669152+0000) 2022-09-08T06:23:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:26.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:26.669296+0000) 2022-09-08T06:23:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:26.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:26.669225+0000) 2022-09-08T06:23:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:26 smithi055 ceph-mon[97033]: pgmap v17020: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:23:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:26 smithi059 ceph-mon[97450]: pgmap v17020: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:23:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:27.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:27.669524+0000) 2022-09-08T06:23:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:27.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:27.669464+0000) 2022-09-08T06:23:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:28.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:28.669874+0000) 2022-09-08T06:23:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:28.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:28.669730+0000) 2022-09-08T06:23:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:28 smithi055 ceph-mon[97033]: pgmap v17021: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:23:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:28 smithi059 ceph-mon[97450]: pgmap v17021: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:23:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:29.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:29.670025+0000) 2022-09-08T06:23:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:29.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:29.669858+0000) 2022-09-08T06:23:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:30.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:30.670193+0000) 2022-09-08T06:23:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:30.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:30.669993+0000) 2022-09-08T06:23:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:30 smithi055 ceph-mon[97033]: pgmap v17022: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:30 smithi059 ceph-mon[97450]: pgmap v17022: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:31.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:31.670147+0000) 2022-09-08T06:23:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:31.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:31.670335+0000) 2022-09-08T06:23:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:32 smithi055 ceph-mon[97033]: pgmap v17023: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:23:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:32 smithi059 ceph-mon[97450]: pgmap v17023: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:23:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:32.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:32.670326+0000) 2022-09-08T06:23:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:32.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:32.670524+0000) 2022-09-08T06:23:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:33.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:33.670552+0000) 2022-09-08T06:23:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:33.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:33.670689+0000) 2022-09-08T06:23:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:34.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:34.670753+0000) 2022-09-08T06:23:35.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:34.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:34.670846+0000) 2022-09-08T06:23:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:34 smithi055 ceph-mon[97033]: pgmap v17024: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:34 smithi059 ceph-mon[97450]: pgmap v17024: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:35.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:23:35.313+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:23:36.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:35.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:35.670959+0000) 2022-09-08T06:23:36.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:35.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:35.671047+0000) 2022-09-08T06:23:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:36.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:36.671127+0000) 2022-09-08T06:23:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:36.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:36.671239+0000) 2022-09-08T06:23:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:36 smithi055 ceph-mon[97033]: pgmap v17025: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:23:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:36 smithi059 ceph-mon[97450]: pgmap v17025: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:23:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:37.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:37.671330+0000) 2022-09-08T06:23:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:37.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:37.671465+0000) 2022-09-08T06:23:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:38.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:38.671565+0000) 2022-09-08T06:23:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:38.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:38.671656+0000) 2022-09-08T06:23:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:38 smithi055 ceph-mon[97033]: pgmap v17026: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:38 smithi059 ceph-mon[97450]: pgmap v17026: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:39.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:39.671722+0000) 2022-09-08T06:23:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:39.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:39.671813+0000) 2022-09-08T06:23:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:40.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:40.671860+0000) 2022-09-08T06:23:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:40.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:40.671989+0000) 2022-09-08T06:23:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:40 smithi055 ceph-mon[97033]: pgmap v17027: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:40 smithi059 ceph-mon[97450]: pgmap v17027: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:41.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:41.671991+0000) 2022-09-08T06:23:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:41.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:41.672141+0000) 2022-09-08T06:23:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:42 smithi055 ceph-mon[97033]: pgmap v17028: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:23:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:42 smithi059 ceph-mon[97450]: pgmap v17028: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:23:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:42.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:42.672154+0000) 2022-09-08T06:23:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:42.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:42.672292+0000) 2022-09-08T06:23:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:43.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:43.672345+0000) 2022-09-08T06:23:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:43.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:43.672433+0000) 2022-09-08T06:23:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:44.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:44.672527+0000) 2022-09-08T06:23:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:44.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:44.672657+0000) 2022-09-08T06:23:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:44 smithi055 ceph-mon[97033]: pgmap v17029: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:44 smithi059 ceph-mon[97450]: pgmap v17029: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:45.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:45.672721+0000) 2022-09-08T06:23:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:45.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:45.672827+0000) 2022-09-08T06:23:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:46.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:46.672878+0000) 2022-09-08T06:23:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:46.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:46.672980+0000) 2022-09-08T06:23:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:46 smithi055 ceph-mon[97033]: pgmap v17030: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:23:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:46 smithi059 ceph-mon[97450]: pgmap v17030: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:23:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:47.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:47.673026+0000) 2022-09-08T06:23:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:47.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:47.673125+0000) 2022-09-08T06:23:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:48.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:48.673179+0000) 2022-09-08T06:23:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:48.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:48.673285+0000) 2022-09-08T06:23:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:48 smithi055 ceph-mon[97033]: pgmap v17031: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:48 smithi059 ceph-mon[97450]: pgmap v17031: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:49.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:49.673385+0000) 2022-09-08T06:23:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:49.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:49.673513+0000) 2022-09-08T06:23:50.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:23:50.316+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:23:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:50.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:50.673608+0000) 2022-09-08T06:23:51.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:50.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:50.673798+0000) 2022-09-08T06:23:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:50 smithi055 ceph-mon[97033]: pgmap v17032: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:51.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:50 smithi059 ceph-mon[97450]: pgmap v17032: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:51.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:51.673809+0000) 2022-09-08T06:23:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:51.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:51.673986+0000) 2022-09-08T06:23:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:52 smithi055 ceph-mon[97033]: pgmap v17033: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:23:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:52 smithi059 ceph-mon[97450]: pgmap v17033: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:23:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:52.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:52.674133+0000) 2022-09-08T06:23:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:52.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:52.673933+0000) 2022-09-08T06:23:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:53.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:53.674299+0000) 2022-09-08T06:23:54.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:53.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:53.674133+0000) 2022-09-08T06:23:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:54.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:54.674351+0000) 2022-09-08T06:23:55.162 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:54.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:54.674496+0000) 2022-09-08T06:23:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:54 smithi055 ceph-mon[97033]: pgmap v17034: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:54 smithi059 ceph-mon[97450]: pgmap v17034: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:55.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:55.674516+0000) 2022-09-08T06:23:56.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:55.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:55.674668+0000) 2022-09-08T06:23:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:56.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:56.674679+0000) 2022-09-08T06:23:57.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:56.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:56.674809+0000) 2022-09-08T06:23:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:56 smithi055 ceph-mon[97033]: pgmap v17035: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:23:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:56 smithi059 ceph-mon[97450]: pgmap v17035: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:23:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:57.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:57.674903+0000) 2022-09-08T06:23:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:57.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:57.675000+0000) 2022-09-08T06:23:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:58.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:58.675190+0000) 2022-09-08T06:23:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:58.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:58.675173+0000) 2022-09-08T06:23:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:23:58 smithi055 ceph-mon[97033]: pgmap v17036: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:23:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:23:58 smithi059 ceph-mon[97450]: pgmap v17036: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:23:59.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:59.675407+0000) 2022-09-08T06:24:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:23:59.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:23:59.675391+0000) 2022-09-08T06:24:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:00 smithi059 ceph-mon[97450]: pgmap v17037: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:24:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:24:01.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:24:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:00.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:00.675571+0000) 2022-09-08T06:24:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:00.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:00.675548+0000) 2022-09-08T06:24:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:00 smithi055 ceph-mon[97033]: pgmap v17037: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:24:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:24:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:24:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:01.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:01.675757+0000) 2022-09-08T06:24:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:01.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:01.675763+0000) 2022-09-08T06:24:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:24:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:24:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:24:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:24:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:24:02.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:24:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:02.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:02.675892+0000) 2022-09-08T06:24:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:02.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:02.675910+0000) 2022-09-08T06:24:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:03 smithi055 ceph-mon[97033]: pgmap v17038: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:24:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:03 smithi059 ceph-mon[97450]: pgmap v17038: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:24:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:03.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:03.676117+0000) 2022-09-08T06:24:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:03.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:03.676145+0000) 2022-09-08T06:24:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:04.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:04.676428+0000) 2022-09-08T06:24:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:04.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:04.676387+0000) 2022-09-08T06:24:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:05 smithi055 ceph-mon[97033]: pgmap v17039: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:05 smithi059 ceph-mon[97450]: pgmap v17039: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:05.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:24:05.330+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:24:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:05.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:05.676721+0000) 2022-09-08T06:24:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:05.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:05.676684+0000) 2022-09-08T06:24:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:24:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:24:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:06.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:06.676892+0000) 2022-09-08T06:24:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:06.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:06.676835+0000) 2022-09-08T06:24:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:07 smithi055 ceph-mon[97033]: pgmap v17040: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:24:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:24:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:07 smithi059 ceph-mon[97450]: pgmap v17040: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:24:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:24:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:07.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:07.677026+0000) 2022-09-08T06:24:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:07.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:07.677016+0000) 2022-09-08T06:24:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:08.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:08.677205+0000) 2022-09-08T06:24:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:08.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:08.677197+0000) 2022-09-08T06:24:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:09 smithi055 ceph-mon[97033]: pgmap v17041: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:24:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:09 smithi059 ceph-mon[97450]: pgmap v17041: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:24:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:09.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:09.677339+0000) 2022-09-08T06:24:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:09.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:09.677363+0000) 2022-09-08T06:24:10.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:10 smithi055 ceph-mon[97033]: osdmap e1144: 12 total, 11 up, 11 in 2022-09-08T06:24:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:10 smithi059 ceph-mon[97450]: osdmap e1144: 12 total, 11 up, 11 in 2022-09-08T06:24:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:10.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:10.677532+0000) 2022-09-08T06:24:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:10.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:10.677481+0000) 2022-09-08T06:24:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:11 smithi055 ceph-mon[97033]: pgmap v17043: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:24:11.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:11 smithi059 ceph-mon[97450]: pgmap v17043: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:24:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:11.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:11.677741+0000) 2022-09-08T06:24:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:11.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:11.677710+0000) 2022-09-08T06:24:12.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:12 smithi055 ceph-mon[97033]: pgmap v17044: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:24:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:12 smithi059 ceph-mon[97450]: pgmap v17044: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:24:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:12.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:12.677962+0000) 2022-09-08T06:24:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:12.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:12.677933+0000) 2022-09-08T06:24:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:13.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:13.678124+0000) 2022-09-08T06:24:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:13.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:13.678092+0000) 2022-09-08T06:24:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:14.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:14.678308+0000) 2022-09-08T06:24:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:14.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:14.678258+0000) 2022-09-08T06:24:15.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:14 smithi055 ceph-mon[97033]: pgmap v17045: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:24:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:14 smithi059 ceph-mon[97450]: pgmap v17045: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:24:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:15.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:15.678545+0000) 2022-09-08T06:24:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:15.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:15.678486+0000) 2022-09-08T06:24:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:16.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:16.680429+0000) 2022-09-08T06:24:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:16.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:16.678710+0000) 2022-09-08T06:24:17.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:16 smithi055 ceph-mon[97033]: pgmap v17046: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:24:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:16 smithi055 ceph-mon[97033]: osdmap e1145: 12 total, 11 up, 11 in 2022-09-08T06:24:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:16 smithi059 ceph-mon[97450]: pgmap v17046: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:24:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:16 smithi059 ceph-mon[97450]: osdmap e1145: 12 total, 11 up, 11 in 2022-09-08T06:24:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:17.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:17.680648+0000) 2022-09-08T06:24:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:17.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:17.678855+0000) 2022-09-08T06:24:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:18.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:18.680793+0000) 2022-09-08T06:24:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:18.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:18.679005+0000) 2022-09-08T06:24:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:18 smithi055 ceph-mon[97033]: pgmap v17048: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2022-09-08T06:24:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:18 smithi059 ceph-mon[97450]: pgmap v17048: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2022-09-08T06:24:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:19.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:19.680978+0000) 2022-09-08T06:24:20.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:19.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:19.679153+0000) 2022-09-08T06:24:20.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:24:20.336+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:24:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:20.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:20.681156+0000) 2022-09-08T06:24:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:20.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:20.679333+0000) 2022-09-08T06:24:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:20 smithi055 ceph-mon[97033]: pgmap v17049: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:24:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:20 smithi059 ceph-mon[97450]: pgmap v17049: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:24:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:21.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:21.681358+0000) 2022-09-08T06:24:22.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:21.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:21.679490+0000) 2022-09-08T06:24:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:22 smithi055 ceph-mon[97033]: pgmap v17050: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:24:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:22 smithi059 ceph-mon[97450]: pgmap v17050: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:24:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:22.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:22.681588+0000) 2022-09-08T06:24:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:22.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:22.679623+0000) 2022-09-08T06:24:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:23.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:23.681750+0000) 2022-09-08T06:24:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:23.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:23.679838+0000) 2022-09-08T06:24:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:24.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:24.681894+0000) 2022-09-08T06:24:25.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:24.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:24.679990+0000) 2022-09-08T06:24:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:24 smithi055 ceph-mon[97033]: pgmap v17051: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:24:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:24 smithi059 ceph-mon[97450]: pgmap v17051: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:24:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:25.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:25.680192+0000) 2022-09-08T06:24:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:25.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:25.682064+0000) 2022-09-08T06:24:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:26.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:26.682253+0000) 2022-09-08T06:24:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:26.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:26.680337+0000) 2022-09-08T06:24:27.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:26 smithi055 ceph-mon[97033]: pgmap v17052: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:24:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:26 smithi059 ceph-mon[97450]: pgmap v17052: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:24:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:27.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:27.682470+0000) 2022-09-08T06:24:28.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:27.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:27.680471+0000) 2022-09-08T06:24:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:28.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:28.680668+0000) 2022-09-08T06:24:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:28.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:28.682726+0000) 2022-09-08T06:24:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:28 smithi055 ceph-mon[97033]: pgmap v17053: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:24:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:28 smithi059 ceph-mon[97450]: pgmap v17053: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:24:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:29.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:29.680799+0000) 2022-09-08T06:24:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:29.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:29.682912+0000) 2022-09-08T06:24:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:30.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:30.683097+0000) 2022-09-08T06:24:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:30.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:30.680945+0000) 2022-09-08T06:24:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:30 smithi055 ceph-mon[97033]: pgmap v17054: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:30 smithi059 ceph-mon[97450]: pgmap v17054: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:31.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:31.683284+0000) 2022-09-08T06:24:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:31.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:31.681107+0000) 2022-09-08T06:24:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:32 smithi055 ceph-mon[97033]: pgmap v17055: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:24:33.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:32 smithi059 ceph-mon[97450]: pgmap v17055: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:24:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:32.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:32.683544+0000) 2022-09-08T06:24:33.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:32.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:32.681311+0000) 2022-09-08T06:24:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:33.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:33.683735+0000) 2022-09-08T06:24:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:33.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:33.681558+0000) 2022-09-08T06:24:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:34.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:34.683873+0000) 2022-09-08T06:24:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:34.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:34.681795+0000) 2022-09-08T06:24:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:34 smithi055 ceph-mon[97033]: pgmap v17056: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:34 smithi059 ceph-mon[97450]: pgmap v17056: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:35.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:24:35.336+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:24:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:35.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:35.684045+0000) 2022-09-08T06:24:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:35.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:35.681969+0000) 2022-09-08T06:24:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:36.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:36.682116+0000) 2022-09-08T06:24:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:36.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:36.684208+0000) 2022-09-08T06:24:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:36 smithi055 ceph-mon[97033]: pgmap v17057: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:24:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:36 smithi059 ceph-mon[97450]: pgmap v17057: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:24:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:37.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:37.684403+0000) 2022-09-08T06:24:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:37.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:37.682291+0000) 2022-09-08T06:24:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:38.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:38.682442+0000) 2022-09-08T06:24:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:38.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:38.684588+0000) 2022-09-08T06:24:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:38 smithi055 ceph-mon[97033]: pgmap v17058: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:38 smithi059 ceph-mon[97450]: pgmap v17058: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:39.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:39.684723+0000) 2022-09-08T06:24:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:39.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:39.682705+0000) 2022-09-08T06:24:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:40.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:40.684895+0000) 2022-09-08T06:24:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:40.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:40.682877+0000) 2022-09-08T06:24:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:40 smithi055 ceph-mon[97033]: pgmap v17059: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:40 smithi059 ceph-mon[97450]: pgmap v17059: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:41.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:41.685064+0000) 2022-09-08T06:24:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:41.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:41.683022+0000) 2022-09-08T06:24:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:42 smithi055 ceph-mon[97033]: pgmap v17060: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:24:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:42 smithi059 ceph-mon[97450]: pgmap v17060: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:24:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:42.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:42.685291+0000) 2022-09-08T06:24:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:42.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:42.683161+0000) 2022-09-08T06:24:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:43.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:43.685425+0000) 2022-09-08T06:24:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:43.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:43.683342+0000) 2022-09-08T06:24:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:44.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:44.685667+0000) 2022-09-08T06:24:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:44.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:44.683515+0000) 2022-09-08T06:24:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:44 smithi055 ceph-mon[97033]: pgmap v17061: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:44 smithi059 ceph-mon[97450]: pgmap v17061: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:45.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:45.685857+0000) 2022-09-08T06:24:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:45.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:45.683773+0000) 2022-09-08T06:24:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:46.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:46.686004+0000) 2022-09-08T06:24:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:46.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:46.683937+0000) 2022-09-08T06:24:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:46 smithi055 ceph-mon[97033]: pgmap v17062: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:24:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:46 smithi059 ceph-mon[97450]: pgmap v17062: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:24:48.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:47.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:47.686182+0000) 2022-09-08T06:24:48.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:47.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:47.684083+0000) 2022-09-08T06:24:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:48.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:48.686393+0000) 2022-09-08T06:24:49.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:48.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:48.684211+0000) 2022-09-08T06:24:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:48 smithi055 ceph-mon[97033]: pgmap v17063: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:49.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:48 smithi059 ceph-mon[97450]: pgmap v17063: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:50.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:49.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:49.686625+0000) 2022-09-08T06:24:50.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:49.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:49.684341+0000) 2022-09-08T06:24:50.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:24:50.346+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:24:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:50.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:50.686835+0000) 2022-09-08T06:24:51.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:50.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:50.684501+0000) 2022-09-08T06:24:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:50 smithi055 ceph-mon[97033]: pgmap v17064: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:51.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:50 smithi059 ceph-mon[97450]: pgmap v17064: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:51.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:51.684622+0000) 2022-09-08T06:24:52.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:51.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:51.687059+0000) 2022-09-08T06:24:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:52 smithi055 ceph-mon[97033]: pgmap v17065: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:24:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:52 smithi059 ceph-mon[97450]: pgmap v17065: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:24:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:52.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:52.684846+0000) 2022-09-08T06:24:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:52.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:52.687278+0000) 2022-09-08T06:24:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:53.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:53.687500+0000) 2022-09-08T06:24:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:53.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:53.684993+0000) 2022-09-08T06:24:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:54.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:54.687786+0000) 2022-09-08T06:24:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:54.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:54.685214+0000) 2022-09-08T06:24:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:54 smithi055 ceph-mon[97033]: pgmap v17066: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:54 smithi059 ceph-mon[97450]: pgmap v17066: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:55.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:55.687948+0000) 2022-09-08T06:24:56.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:55.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:55.685415+0000) 2022-09-08T06:24:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:56.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:56.688185+0000) 2022-09-08T06:24:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:56.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:56.685540+0000) 2022-09-08T06:24:57.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:56 smithi055 ceph-mon[97033]: pgmap v17067: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:24:57.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:56 smithi059 ceph-mon[97450]: pgmap v17067: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:24:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:57.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:57.688399+0000) 2022-09-08T06:24:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:57.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:57.685752+0000) 2022-09-08T06:24:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:58.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:58.688627+0000) 2022-09-08T06:24:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:58.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:58.685927+0000) 2022-09-08T06:24:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:24:58 smithi055 ceph-mon[97033]: pgmap v17068: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:24:59.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:24:58 smithi059 ceph-mon[97450]: pgmap v17068: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:24:59.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:59.688809+0000) 2022-09-08T06:25:00.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:24:59.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:24:59.686093+0000) 2022-09-08T06:25:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:00.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:00.688942+0000) 2022-09-08T06:25:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:00.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:00.686247+0000) 2022-09-08T06:25:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:00 smithi055 ceph-mon[97033]: pgmap v17069: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:00 smithi059 ceph-mon[97450]: pgmap v17069: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:01.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:01.689139+0000) 2022-09-08T06:25:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:01.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:01.686374+0000) 2022-09-08T06:25:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:25:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:25:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:25:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:02 smithi055 ceph-mon[97033]: pgmap v17070: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:25:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:25:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:25:03.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:25:03.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:02 smithi059 ceph-mon[97450]: pgmap v17070: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:25:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:02.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:02.689374+0000) 2022-09-08T06:25:03.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:02.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:02.686521+0000) 2022-09-08T06:25:03.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:03.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:03.897 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:03.898 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:03.898 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:03.899 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:03.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:03.689587+0000) 2022-09-08T06:25:04.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:03.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:03.686673+0000) 2022-09-08T06:25:04.942 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:04 smithi059 ceph-mon[97450]: pgmap v17071: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:04 smithi055 ceph-mon[97033]: pgmap v17071: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:04.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:04.689766+0000) 2022-09-08T06:25:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:04.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:04.686873+0000) 2022-09-08T06:25:05.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:25:05.351+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:25:06.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:05.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:05.689898+0000) 2022-09-08T06:25:06.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:05.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:05.687046+0000) 2022-09-08T06:25:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:06 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T06:25:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:06 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T06:25:06.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:06.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:06.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:06.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:06.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:06 smithi055 ceph-mon[97033]: pgmap v17072: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:25:06.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:25:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:06 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T06:25:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:06 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T06:25:06.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:06.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:06.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:06.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:25:06.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:06 smithi059 ceph-mon[97450]: pgmap v17072: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:25:06.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:25:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:06.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:06.690023+0000) 2022-09-08T06:25:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:06.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:06.687183+0000) 2022-09-08T06:25:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:25:08.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:25:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:07.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:07.690218+0000) 2022-09-08T06:25:08.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:07.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:07.687334+0000) 2022-09-08T06:25:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:08 smithi055 ceph-mon[97033]: pgmap v17073: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:25:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:08 smithi059 ceph-mon[97450]: pgmap v17073: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:25:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:08.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:08.690476+0000) 2022-09-08T06:25:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:08.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:08.687469+0000) 2022-09-08T06:25:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:09.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:09.690734+0000) 2022-09-08T06:25:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:09.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:09.687611+0000) 2022-09-08T06:25:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:10 smithi055 ceph-mon[97033]: osdmap e1146: 12 total, 11 up, 11 in 2022-09-08T06:25:10.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:10 smithi055 ceph-mon[97033]: pgmap v17075: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:25:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:10 smithi059 ceph-mon[97450]: osdmap e1146: 12 total, 11 up, 11 in 2022-09-08T06:25:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:10 smithi059 ceph-mon[97450]: pgmap v17075: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:25:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:10.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:10.687810+0000) 2022-09-08T06:25:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:10.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:10.690907+0000) 2022-09-08T06:25:12.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:11.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:11.687954+0000) 2022-09-08T06:25:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:11.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:11.691074+0000) 2022-09-08T06:25:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:12 smithi055 ceph-mon[97033]: pgmap v17076: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:25:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:12 smithi059 ceph-mon[97450]: pgmap v17076: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:25:13.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:12.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:12.691320+0000) 2022-09-08T06:25:13.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:12.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:12.688111+0000) 2022-09-08T06:25:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:13.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:13.688254+0000) 2022-09-08T06:25:14.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:13.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:13.691582+0000) 2022-09-08T06:25:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:14.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:14.691794+0000) 2022-09-08T06:25:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:14.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:14.688468+0000) 2022-09-08T06:25:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:14 smithi055 ceph-mon[97033]: pgmap v17077: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:25:15.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:14 smithi059 ceph-mon[97450]: pgmap v17077: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:25:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:15.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:15.691983+0000) 2022-09-08T06:25:16.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:15.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:15.688627+0000) 2022-09-08T06:25:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:16.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:16.692185+0000) 2022-09-08T06:25:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:16.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:16.688776+0000) 2022-09-08T06:25:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:16 smithi055 ceph-mon[97033]: pgmap v17078: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:25:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:16 smithi055 ceph-mon[97033]: osdmap e1147: 12 total, 11 up, 11 in 2022-09-08T06:25:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:16 smithi059 ceph-mon[97450]: pgmap v17078: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:25:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:16 smithi059 ceph-mon[97450]: osdmap e1147: 12 total, 11 up, 11 in 2022-09-08T06:25:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:17.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:17.692408+0000) 2022-09-08T06:25:18.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:17.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:17.688955+0000) 2022-09-08T06:25:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:18.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:18.689106+0000) 2022-09-08T06:25:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:18.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:18.692671+0000) 2022-09-08T06:25:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:18 smithi055 ceph-mon[97033]: pgmap v17080: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T06:25:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:18 smithi059 ceph-mon[97450]: pgmap v17080: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T06:25:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:19.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:19.689334+0000) 2022-09-08T06:25:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:19.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:19.692813+0000) 2022-09-08T06:25:20.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:25:20.361+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:25:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:20.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:20.689512+0000) 2022-09-08T06:25:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:20.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:20.692993+0000) 2022-09-08T06:25:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:20 smithi055 ceph-mon[97033]: pgmap v17081: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:25:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:20 smithi059 ceph-mon[97450]: pgmap v17081: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:25:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:21.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:21.693206+0000) 2022-09-08T06:25:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:21.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:21.689679+0000) 2022-09-08T06:25:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:22 smithi055 ceph-mon[97033]: pgmap v17082: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:25:23.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:22 smithi059 ceph-mon[97450]: pgmap v17082: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:25:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:22.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:22.693441+0000) 2022-09-08T06:25:23.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:22.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:22.689886+0000) 2022-09-08T06:25:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:23.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:23.693714+0000) 2022-09-08T06:25:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:23.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:23.690071+0000) 2022-09-08T06:25:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:24.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:24.693956+0000) 2022-09-08T06:25:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:24.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:24.690225+0000) 2022-09-08T06:25:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:24 smithi055 ceph-mon[97033]: pgmap v17083: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:25:25.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:24 smithi059 ceph-mon[97450]: pgmap v17083: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:25:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:25.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:25.694169+0000) 2022-09-08T06:25:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:25.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:25.690441+0000) 2022-09-08T06:25:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:26.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:26.694430+0000) 2022-09-08T06:25:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:26.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:26.690580+0000) 2022-09-08T06:25:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:26 smithi055 ceph-mon[97033]: pgmap v17084: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:25:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:26 smithi059 ceph-mon[97450]: pgmap v17084: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:25:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:27.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:27.694726+0000) 2022-09-08T06:25:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:27.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:27.690739+0000) 2022-09-08T06:25:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:28.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:28.694900+0000) 2022-09-08T06:25:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:28.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:28.690926+0000) 2022-09-08T06:25:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:28 smithi055 ceph-mon[97033]: pgmap v17085: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:25:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:28 smithi059 ceph-mon[97450]: pgmap v17085: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:25:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:29.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:29.695044+0000) 2022-09-08T06:25:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:29.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:29.691088+0000) 2022-09-08T06:25:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:30.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:30.695251+0000) 2022-09-08T06:25:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:30.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:30.691290+0000) 2022-09-08T06:25:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:30 smithi055 ceph-mon[97033]: pgmap v17086: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:30 smithi059 ceph-mon[97450]: pgmap v17086: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:31.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:31.695463+0000) 2022-09-08T06:25:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:31.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:31.691439+0000) 2022-09-08T06:25:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:32 smithi055 ceph-mon[97033]: pgmap v17087: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:25:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:32 smithi059 ceph-mon[97450]: pgmap v17087: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:25:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:32.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:32.695715+0000) 2022-09-08T06:25:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:32.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:32.691674+0000) 2022-09-08T06:25:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:33.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:33.695837+0000) 2022-09-08T06:25:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:33.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:33.691871+0000) 2022-09-08T06:25:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:34.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:34.696017+0000) 2022-09-08T06:25:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:34.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:34.692084+0000) 2022-09-08T06:25:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:34 smithi055 ceph-mon[97033]: pgmap v17088: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:34 smithi059 ceph-mon[97450]: pgmap v17088: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:35.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:25:35.378+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:25:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:35.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:35.696246+0000) 2022-09-08T06:25:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:35.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:35.692334+0000) 2022-09-08T06:25:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:36.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:36.696524+0000) 2022-09-08T06:25:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:36.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:36.692483+0000) 2022-09-08T06:25:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:36 smithi055 ceph-mon[97033]: pgmap v17089: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:25:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:36 smithi059 ceph-mon[97450]: pgmap v17089: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:25:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:37.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:37.696682+0000) 2022-09-08T06:25:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:37.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:37.692706+0000) 2022-09-08T06:25:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:38.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:38.696799+0000) 2022-09-08T06:25:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:38.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:38.692892+0000) 2022-09-08T06:25:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:38 smithi055 ceph-mon[97033]: pgmap v17090: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:38 smithi059 ceph-mon[97450]: pgmap v17090: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:39.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:39.696930+0000) 2022-09-08T06:25:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:39.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:39.693102+0000) 2022-09-08T06:25:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:40.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:40.697076+0000) 2022-09-08T06:25:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:40.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:40.693273+0000) 2022-09-08T06:25:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:40 smithi055 ceph-mon[97033]: pgmap v17091: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:40 smithi059 ceph-mon[97450]: pgmap v17091: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:41.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:41.697266+0000) 2022-09-08T06:25:42.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:41.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:41.693449+0000) 2022-09-08T06:25:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:42 smithi055 ceph-mon[97033]: pgmap v17092: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:25:43.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:42 smithi059 ceph-mon[97450]: pgmap v17092: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:25:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:42.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:42.697491+0000) 2022-09-08T06:25:43.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:42.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:42.693713+0000) 2022-09-08T06:25:44.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:43.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:43.697702+0000) 2022-09-08T06:25:44.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:43.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:43.693868+0000) 2022-09-08T06:25:45.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:44.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:44.697880+0000) 2022-09-08T06:25:45.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:44.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:44.694072+0000) 2022-09-08T06:25:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:44 smithi055 ceph-mon[97033]: pgmap v17093: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:44 smithi059 ceph-mon[97450]: pgmap v17093: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:45.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:45.698058+0000) 2022-09-08T06:25:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:45.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:45.694231+0000) 2022-09-08T06:25:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:46.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:46.694450+0000) 2022-09-08T06:25:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:46.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:46.698260+0000) 2022-09-08T06:25:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:46 smithi055 ceph-mon[97033]: pgmap v17094: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:25:47.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:46 smithi059 ceph-mon[97450]: pgmap v17094: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:25:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:47.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:47.698451+0000) 2022-09-08T06:25:48.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:47.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:47.694626+0000) 2022-09-08T06:25:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:48.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:48.694780+0000) 2022-09-08T06:25:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:48.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:48.698681+0000) 2022-09-08T06:25:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:48 smithi055 ceph-mon[97033]: pgmap v17095: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:48 smithi059 ceph-mon[97450]: pgmap v17095: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:49.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:49.694913+0000) 2022-09-08T06:25:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:49.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:49.698849+0000) 2022-09-08T06:25:50.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:25:50.388+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:25:51.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:50.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:50.695134+0000) 2022-09-08T06:25:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:50.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:50.698971+0000) 2022-09-08T06:25:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:50 smithi055 ceph-mon[97033]: pgmap v17096: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:50 smithi059 ceph-mon[97450]: pgmap v17096: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:51.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:51.695287+0000) 2022-09-08T06:25:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:51.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:51.699095+0000) 2022-09-08T06:25:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:52 smithi055 ceph-mon[97033]: pgmap v17097: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:25:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:52 smithi059 ceph-mon[97450]: pgmap v17097: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:25:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:52.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:52.699330+0000) 2022-09-08T06:25:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:52.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:52.695414+0000) 2022-09-08T06:25:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:53.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:53.699601+0000) 2022-09-08T06:25:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:53.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:53.695624+0000) 2022-09-08T06:25:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:54.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:54.699877+0000) 2022-09-08T06:25:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:54.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:54.695780+0000) 2022-09-08T06:25:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:54 smithi055 ceph-mon[97033]: pgmap v17098: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:55.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:54 smithi059 ceph-mon[97450]: pgmap v17098: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:25:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:55.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:55.700042+0000) 2022-09-08T06:25:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:55.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:55.695981+0000) 2022-09-08T06:25:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:56.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:56.700234+0000) 2022-09-08T06:25:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:56.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:56.696202+0000) 2022-09-08T06:25:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:56 smithi055 ceph-mon[97033]: pgmap v17099: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:25:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:56 smithi059 ceph-mon[97450]: pgmap v17099: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:25:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:57.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:57.700480+0000) 2022-09-08T06:25:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:57.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:57.696409+0000) 2022-09-08T06:25:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:58.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:58.700732+0000) 2022-09-08T06:25:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:58.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:58.696540+0000) 2022-09-08T06:25:59.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:25:58 smithi055 ceph-mon[97033]: pgmap v17100: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:25:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:25:58 smithi059 ceph-mon[97450]: pgmap v17100: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:26:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:25:59.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:59.700919+0000) 2022-09-08T06:26:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:25:59.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:25:59.696774+0000) 2022-09-08T06:26:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:00.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:00.701091+0000) 2022-09-08T06:26:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:00.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:00.696954+0000) 2022-09-08T06:26:01.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:00 smithi055 ceph-mon[97033]: pgmap v17101: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:26:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:00 smithi059 ceph-mon[97450]: pgmap v17101: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:26:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:01.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:01.701269+0000) 2022-09-08T06:26:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:01.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:01.697103+0000) 2022-09-08T06:26:02.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:02 smithi055 ceph-mon[97033]: pgmap v17102: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:26:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:02 smithi059 ceph-mon[97450]: pgmap v17102: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:26:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:02.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:02.701461+0000) 2022-09-08T06:26:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:02.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:02.697265+0000) 2022-09-08T06:26:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:03.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:03.701727+0000) 2022-09-08T06:26:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:03.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:03.697449+0000) 2022-09-08T06:26:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:04.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:04.701882+0000) 2022-09-08T06:26:05.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:04.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:04.697693+0000) 2022-09-08T06:26:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:04 smithi055 ceph-mon[97033]: pgmap v17103: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:26:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:04 smithi059 ceph-mon[97450]: pgmap v17103: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:26:05.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:26:05.396+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:26:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:05.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:05.702057+0000) 2022-09-08T06:26:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:05.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:05.697832+0000) 2022-09-08T06:26:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:26:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:26:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:26:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:26:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:26:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:26:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:26:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:26:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:06.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:06.702251+0000) 2022-09-08T06:26:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:06.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:06.698007+0000) 2022-09-08T06:26:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:06 smithi055 ceph-mon[97033]: pgmap v17104: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:26:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:26:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:06 smithi059 ceph-mon[97450]: pgmap v17104: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:26:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:26:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:07.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:07.702500+0000) 2022-09-08T06:26:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:07.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:07.698176+0000) 2022-09-08T06:26:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:08.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:08.702730+0000) 2022-09-08T06:26:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:08.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:08.698385+0000) 2022-09-08T06:26:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:08 smithi055 ceph-mon[97033]: pgmap v17105: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:26:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:08 smithi059 ceph-mon[97450]: pgmap v17105: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:26:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:09.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:09.702895+0000) 2022-09-08T06:26:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:09.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:09.698544+0000) 2022-09-08T06:26:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:09 smithi055 ceph-mon[97033]: osdmap e1148: 12 total, 11 up, 11 in 2022-09-08T06:26:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:09 smithi059 ceph-mon[97450]: osdmap e1148: 12 total, 11 up, 11 in 2022-09-08T06:26:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:10.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:10.703069+0000) 2022-09-08T06:26:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:10.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:10.698741+0000) 2022-09-08T06:26:11.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:10 smithi055 ceph-mon[97033]: pgmap v17107: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 25 op/s 2022-09-08T06:26:11.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:10 smithi059 ceph-mon[97450]: pgmap v17107: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 25 op/s 2022-09-08T06:26:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:11.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:11.703276+0000) 2022-09-08T06:26:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:11.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:11.698906+0000) 2022-09-08T06:26:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:12 smithi055 ceph-mon[97033]: pgmap v17108: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:26:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:12 smithi059 ceph-mon[97450]: pgmap v17108: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:26:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:12.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:12.703496+0000) 2022-09-08T06:26:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:12.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:12.699053+0000) 2022-09-08T06:26:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:13.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:13.703771+0000) 2022-09-08T06:26:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:13.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:13.699253+0000) 2022-09-08T06:26:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:14.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:14.703976+0000) 2022-09-08T06:26:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:14.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:14.699431+0000) 2022-09-08T06:26:15.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:14 smithi055 ceph-mon[97033]: pgmap v17109: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:26:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:14 smithi059 ceph-mon[97450]: pgmap v17109: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:26:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:15.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:15.704168+0000) 2022-09-08T06:26:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:15.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:15.699592+0000) 2022-09-08T06:26:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:16.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:16.704407+0000) 2022-09-08T06:26:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:16.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:16.699758+0000) 2022-09-08T06:26:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:16 smithi055 ceph-mon[97033]: pgmap v17110: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:26:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:16 smithi055 ceph-mon[97033]: osdmap e1149: 12 total, 11 up, 11 in 2022-09-08T06:26:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:16 smithi059 ceph-mon[97450]: pgmap v17110: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:26:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:16 smithi059 ceph-mon[97450]: osdmap e1149: 12 total, 11 up, 11 in 2022-09-08T06:26:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:17.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:17.699954+0000) 2022-09-08T06:26:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:17.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:17.704706+0000) 2022-09-08T06:26:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:18.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:18.704900+0000) 2022-09-08T06:26:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:18.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:18.700092+0000) 2022-09-08T06:26:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:18 smithi055 ceph-mon[97033]: pgmap v17112: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T06:26:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:18 smithi059 ceph-mon[97450]: pgmap v17112: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T06:26:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:19.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:19.700247+0000) 2022-09-08T06:26:20.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:19.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:19.705144+0000) 2022-09-08T06:26:20.702 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:26:20.412+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:26:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:20.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:20.705342+0000) 2022-09-08T06:26:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:20.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:20.700419+0000) 2022-09-08T06:26:21.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:20 smithi055 ceph-mon[97033]: pgmap v17113: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:26:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:20 smithi059 ceph-mon[97450]: pgmap v17113: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:26:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:21.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:21.705548+0000) 2022-09-08T06:26:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:21.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:21.700575+0000) 2022-09-08T06:26:22.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:22 smithi055 ceph-mon[97033]: pgmap v17114: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:26:23.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:22 smithi059 ceph-mon[97450]: pgmap v17114: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:26:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:22.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:22.700795+0000) 2022-09-08T06:26:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:22.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:22.705796+0000) 2022-09-08T06:26:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:23.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:23.705927+0000) 2022-09-08T06:26:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:23.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:23.700945+0000) 2022-09-08T06:26:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:24.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:24.706105+0000) 2022-09-08T06:26:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:24.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:24.701160+0000) 2022-09-08T06:26:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:24 smithi055 ceph-mon[97033]: pgmap v17115: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:26:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:24 smithi059 ceph-mon[97450]: pgmap v17115: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:26:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:25.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:25.706275+0000) 2022-09-08T06:26:26.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:25.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:25.701338+0000) 2022-09-08T06:26:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:26.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:26.706505+0000) 2022-09-08T06:26:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:26.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:26.701581+0000) 2022-09-08T06:26:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:26 smithi055 ceph-mon[97033]: pgmap v17116: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:26:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:26 smithi059 ceph-mon[97450]: pgmap v17116: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:26:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:27.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:27.706748+0000) 2022-09-08T06:26:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:27.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:27.701739+0000) 2022-09-08T06:26:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:28.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:28.706896+0000) 2022-09-08T06:26:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:28.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:28.701945+0000) 2022-09-08T06:26:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:28 smithi055 ceph-mon[97033]: pgmap v17117: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:26:29.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:28 smithi059 ceph-mon[97450]: pgmap v17117: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:26:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:29.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:29.707065+0000) 2022-09-08T06:26:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:29.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:29.702120+0000) 2022-09-08T06:26:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:30.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:30.707268+0000) 2022-09-08T06:26:30.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:30.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:30.702266+0000) 2022-09-08T06:26:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:30 smithi055 ceph-mon[97033]: pgmap v17118: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:26:31.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:30 smithi059 ceph-mon[97450]: pgmap v17118: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:26:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:31.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:31.707475+0000) 2022-09-08T06:26:32.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:31.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:31.702455+0000) 2022-09-08T06:26:32.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:32 smithi055 ceph-mon[97033]: pgmap v17119: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:26:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:32 smithi059 ceph-mon[97450]: pgmap v17119: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:26:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:32.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:32.707687+0000) 2022-09-08T06:26:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:32.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:32.702724+0000) 2022-09-08T06:26:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:33.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:33.707860+0000) 2022-09-08T06:26:34.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:33.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:33.702883+0000) 2022-09-08T06:26:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:34.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:34.708025+0000) 2022-09-08T06:26:35.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:34.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:34.703017+0000) 2022-09-08T06:26:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:34 smithi055 ceph-mon[97033]: pgmap v17120: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:26:35.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:34 smithi059 ceph-mon[97450]: pgmap v17120: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:26:35.706 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:26:35.424+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:26:36.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:35.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:35.703214+0000) 2022-09-08T06:26:36.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:35.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:35.708213+0000) 2022-09-08T06:26:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:36 smithi055 ceph-mon[97033]: pgmap v17121: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T06:26:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:26:36.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:36 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T06:26:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:36 smithi059 ceph-mon[97450]: pgmap v17121: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T06:26:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:26:37.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:36 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T06:26:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:36.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:36.708446+0000) 2022-09-08T06:26:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:36.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:36.703354+0000) 2022-09-08T06:26:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:37.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:37.708670+0000) 2022-09-08T06:26:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:37.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:37.703477+0000) 2022-09-08T06:26:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:38 smithi055 ceph-mon[97033]: pgmap v17122: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T06:26:39.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:38 smithi059 ceph-mon[97450]: pgmap v17122: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T06:26:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:38.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:38.708853+0000) 2022-09-08T06:26:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:38.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:38.703732+0000) 2022-09-08T06:26:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:39.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:39.709030+0000) 2022-09-08T06:26:40.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:39.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:39.703894+0000) 2022-09-08T06:26:40.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:40 smithi055 ceph-mon[97033]: pgmap v17123: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T06:26:41.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:40 smithi059 ceph-mon[97450]: pgmap v17123: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T06:26:41.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:40.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:40.709245+0000) 2022-09-08T06:26:41.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:40.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:40.704029+0000) 2022-09-08T06:26:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:41.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:41.709448+0000) 2022-09-08T06:26:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:41.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:41.704204+0000) 2022-09-08T06:26:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:42 smithi055 ceph-mon[97033]: pgmap v17124: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 6 op/s 2022-09-08T06:26:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:42 smithi059 ceph-mon[97450]: pgmap v17124: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 6 op/s 2022-09-08T06:26:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:42.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:42.709583+0000) 2022-09-08T06:26:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:42.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:42.704347+0000) 2022-09-08T06:26:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:43.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:43.709815+0000) 2022-09-08T06:26:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:43.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:43.704539+0000) 2022-09-08T06:26:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:44 smithi055 ceph-mon[97033]: pgmap v17125: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T06:26:45.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:44 smithi059 ceph-mon[97450]: pgmap v17125: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T06:26:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:44.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:44.710036+0000) 2022-09-08T06:26:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:44.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:44.704789+0000) 2022-09-08T06:26:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:45.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:45.710244+0000) 2022-09-08T06:26:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:45.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:45.704953+0000) 2022-09-08T06:26:46.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:46 smithi055 ceph-mon[97033]: pgmap v17126: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T06:26:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:46 smithi059 ceph-mon[97450]: pgmap v17126: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T06:26:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:46.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:46.710475+0000) 2022-09-08T06:26:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:46.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:46.705135+0000) 2022-09-08T06:26:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:47.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:47.710681+0000) 2022-09-08T06:26:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:47.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:47.705279+0000) 2022-09-08T06:26:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:48 smithi055 ceph-mon[97033]: pgmap v17127: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:26:49.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:48 smithi059 ceph-mon[97450]: pgmap v17127: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:26:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:48.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:48.710807+0000) 2022-09-08T06:26:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:48.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:48.705500+0000) 2022-09-08T06:26:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:49.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:49.710955+0000) 2022-09-08T06:26:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:49.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:49.705672+0000) 2022-09-08T06:26:50.707 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:26:50.439+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:26:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:50 smithi055 ceph-mon[97033]: pgmap v17128: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:26:51.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:50 smithi059 ceph-mon[97450]: pgmap v17128: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:26:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:50.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:50.711103+0000) 2022-09-08T06:26:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:50.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:50.705823+0000) 2022-09-08T06:26:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:51.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:51.711311+0000) 2022-09-08T06:26:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:51.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:51.705969+0000) 2022-09-08T06:26:52.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:52 smithi055 ceph-mon[97033]: pgmap v17129: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:26:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:52 smithi059 ceph-mon[97450]: pgmap v17129: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:26:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:52.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:52.706187+0000) 2022-09-08T06:26:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:52.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:52.711484+0000) 2022-09-08T06:26:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:53.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:53.711719+0000) 2022-09-08T06:26:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:53.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:53.706403+0000) 2022-09-08T06:26:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:54 smithi055 ceph-mon[97033]: pgmap v17130: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:26:55.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:54 smithi059 ceph-mon[97450]: pgmap v17130: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:26:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:54.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:54.711859+0000) 2022-09-08T06:26:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:54.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:54.706594+0000) 2022-09-08T06:26:56.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:55.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:55.706745+0000) 2022-09-08T06:26:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:55.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:55.711985+0000) 2022-09-08T06:26:56.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:56 smithi055 ceph-mon[97033]: pgmap v17131: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:26:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:56 smithi059 ceph-mon[97450]: pgmap v17131: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:26:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:56.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:56.712110+0000) 2022-09-08T06:26:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:56.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:56.706892+0000) 2022-09-08T06:26:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:57.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:57.712296+0000) 2022-09-08T06:26:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:57.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:57.707099+0000) 2022-09-08T06:26:58.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:26:58 smithi055 ceph-mon[97033]: pgmap v17132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:26:59.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:26:58 smithi059 ceph-mon[97450]: pgmap v17132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:26:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:58.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:58.712456+0000) 2022-09-08T06:26:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:58.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:58.707285+0000) 2022-09-08T06:27:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:26:59.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:59.707501+0000) 2022-09-08T06:27:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:26:59.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:26:59.712664+0000) 2022-09-08T06:27:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:00 smithi055 ceph-mon[97033]: pgmap v17133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:00 smithi059 ceph-mon[97450]: pgmap v17133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:00.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:00.712797+0000) 2022-09-08T06:27:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:00.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:00.707663+0000) 2022-09-08T06:27:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:01.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:01.712922+0000) 2022-09-08T06:27:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:01.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:01.707811+0000) 2022-09-08T06:27:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:02 smithi055 ceph-mon[97033]: pgmap v17134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:27:03.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:02 smithi059 ceph-mon[97450]: pgmap v17134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:27:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:02.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:02.713051+0000) 2022-09-08T06:27:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:02.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:02.707937+0000) 2022-09-08T06:27:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:03.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:03.713174+0000) 2022-09-08T06:27:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:03.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:03.708157+0000) 2022-09-08T06:27:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:04 smithi055 ceph-mon[97033]: pgmap v17135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:04 smithi059 ceph-mon[97450]: pgmap v17135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:04.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:04.713385+0000) 2022-09-08T06:27:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:04.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:04.708320+0000) 2022-09-08T06:27:05.710 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:27:05.443+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:27:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:05.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:05.713604+0000) 2022-09-08T06:27:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:05.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:05.708513+0000) 2022-09-08T06:27:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:06 smithi055 ceph-mon[97033]: pgmap v17136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:27:06.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:27:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:06 smithi059 ceph-mon[97450]: pgmap v17136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:27:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:27:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:06.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:06.708760+0000) 2022-09-08T06:27:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:06.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:06.713851+0000) 2022-09-08T06:27:08.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:27:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:07.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:07.714094+0000) 2022-09-08T06:27:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:07.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:07.708902+0000) 2022-09-08T06:27:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:27:09.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:08 smithi059 ceph-mon[97450]: pgmap v17137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:27:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:08.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:08.714318+0000) 2022-09-08T06:27:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:08.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:08.709061+0000) 2022-09-08T06:27:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:08 smithi055 ceph-mon[97033]: pgmap v17137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:27:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:09.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:09.714590+0000) 2022-09-08T06:27:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:09.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:09.709223+0000) 2022-09-08T06:27:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:10 smithi059 ceph-mon[97450]: pgmap v17138: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:27:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:10 smithi059 ceph-mon[97450]: osdmap e1150: 12 total, 11 up, 11 in 2022-09-08T06:27:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:10.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:10.714851+0000) 2022-09-08T06:27:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:10.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:10.709505+0000) 2022-09-08T06:27:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:10 smithi055 ceph-mon[97033]: pgmap v17138: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:27:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:10 smithi055 ceph-mon[97033]: osdmap e1150: 12 total, 11 up, 11 in 2022-09-08T06:27:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:11.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:11.715032+0000) 2022-09-08T06:27:12.165 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:11.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:11.709757+0000) 2022-09-08T06:27:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:12 smithi059 ceph-mon[97450]: pgmap v17140: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:27:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:12.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:12.715241+0000) 2022-09-08T06:27:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:12.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:12.709981+0000) 2022-09-08T06:27:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:12 smithi055 ceph-mon[97033]: pgmap v17140: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:27:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:13.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:13.715459+0000) 2022-09-08T06:27:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:13.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:13.710131+0000) 2022-09-08T06:27:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:14 smithi059 ceph-mon[97450]: pgmap v17141: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:27:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:14.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:14.710288+0000) 2022-09-08T06:27:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:14.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:14.715674+0000) 2022-09-08T06:27:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:14 smithi055 ceph-mon[97033]: pgmap v17141: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:27:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:15.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:15.715839+0000) 2022-09-08T06:27:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:15.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:15.710417+0000) 2022-09-08T06:27:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:16 smithi059 ceph-mon[97450]: pgmap v17142: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:27:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:16 smithi059 ceph-mon[97450]: osdmap e1151: 12 total, 11 up, 11 in 2022-09-08T06:27:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:16.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:16.715976+0000) 2022-09-08T06:27:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:16.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:16.710581+0000) 2022-09-08T06:27:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:16 smithi055 ceph-mon[97033]: pgmap v17142: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:27:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:16 smithi055 ceph-mon[97033]: osdmap e1151: 12 total, 11 up, 11 in 2022-09-08T06:27:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:17.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:17.710781+0000) 2022-09-08T06:27:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:17.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:17.716154+0000) 2022-09-08T06:27:19.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:18 smithi059 ceph-mon[97450]: pgmap v17144: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:27:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:18.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:18.716365+0000) 2022-09-08T06:27:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:18.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:18.711012+0000) 2022-09-08T06:27:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:18 smithi055 ceph-mon[97033]: pgmap v17144: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:27:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:19.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:19.716658+0000) 2022-09-08T06:27:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:19.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:19.711166+0000) 2022-09-08T06:27:20.713 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:27:20.459+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:27:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:20 smithi059 ceph-mon[97450]: pgmap v17145: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:27:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:20.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:20.716890+0000) 2022-09-08T06:27:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:20.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:20.711380+0000) 2022-09-08T06:27:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:20 smithi055 ceph-mon[97033]: pgmap v17145: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:27:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:21.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:21.717092+0000) 2022-09-08T06:27:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:21.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:21.711589+0000) 2022-09-08T06:27:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:22 smithi059 ceph-mon[97450]: pgmap v17146: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:27:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:22.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:22.717280+0000) 2022-09-08T06:27:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:22.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:22.711764+0000) 2022-09-08T06:27:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:22 smithi055 ceph-mon[97033]: pgmap v17146: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:27:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:23.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:23.717496+0000) 2022-09-08T06:27:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:23.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:23.711902+0000) 2022-09-08T06:27:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:24 smithi059 ceph-mon[97450]: pgmap v17147: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:27:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:24.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:24.717788+0000) 2022-09-08T06:27:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:24.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:24.712115+0000) 2022-09-08T06:27:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:24 smithi055 ceph-mon[97033]: pgmap v17147: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:27:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:25.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:25.717978+0000) 2022-09-08T06:27:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:25.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:25.712382+0000) 2022-09-08T06:27:27.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:26 smithi059 ceph-mon[97450]: pgmap v17148: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:27:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:26.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:26.718245+0000) 2022-09-08T06:27:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:26.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:26.712553+0000) 2022-09-08T06:27:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:26 smithi055 ceph-mon[97033]: pgmap v17148: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:27:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:27.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:27.718532+0000) 2022-09-08T06:27:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:27.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:27.712698+0000) 2022-09-08T06:27:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:28 smithi059 ceph-mon[97450]: pgmap v17149: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:27:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:28.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:28.718759+0000) 2022-09-08T06:27:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:28.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:28.712869+0000) 2022-09-08T06:27:29.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:28 smithi055 ceph-mon[97033]: pgmap v17149: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:27:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:29.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:29.719012+0000) 2022-09-08T06:27:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:29.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:29.713049+0000) 2022-09-08T06:27:31.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:30 smithi059 ceph-mon[97450]: pgmap v17150: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:30.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:30.719259+0000) 2022-09-08T06:27:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:30.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:30.713220+0000) 2022-09-08T06:27:31.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:30 smithi055 ceph-mon[97033]: pgmap v17150: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:31.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:31.719504+0000) 2022-09-08T06:27:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:31.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:31.713432+0000) 2022-09-08T06:27:33.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:32 smithi059 ceph-mon[97450]: pgmap v17151: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:27:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:32.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:32.719762+0000) 2022-09-08T06:27:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:32.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:32.713578+0000) 2022-09-08T06:27:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:32 smithi055 ceph-mon[97033]: pgmap v17151: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:27:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:33.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:33.719887+0000) 2022-09-08T06:27:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:33.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:33.713760+0000) 2022-09-08T06:27:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:34 smithi059 ceph-mon[97450]: pgmap v17152: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:34.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:34.720098+0000) 2022-09-08T06:27:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:34.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:34.713929+0000) 2022-09-08T06:27:35.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:34 smithi055 ceph-mon[97033]: pgmap v17152: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:35.800 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:27:35.471+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:27:35.802 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:35.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:35.720260+0000) 2022-09-08T06:27:35.802 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:35.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:35.714151+0000) 2022-09-08T06:27:36.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:27:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:27:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:27:36.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:27:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:27:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:27:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:36.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:36.720402+0000) 2022-09-08T06:27:37.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:36.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:36.714366+0000) 2022-09-08T06:27:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:36 smithi055 ceph-mon[97033]: pgmap v17153: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:27:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:36 smithi059 ceph-mon[97450]: pgmap v17153: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:27:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:37.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:37.720511+0000) 2022-09-08T06:27:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:37.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:37.714505+0000) 2022-09-08T06:27:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:38.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:38.720680+0000) 2022-09-08T06:27:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:38.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:38.714718+0000) 2022-09-08T06:27:39.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:38 smithi055 ceph-mon[97033]: pgmap v17154: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:38 smithi059 ceph-mon[97450]: pgmap v17154: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:39.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:39.720799+0000) 2022-09-08T06:27:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:39.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:39.714917+0000) 2022-09-08T06:27:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:40.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:40.720934+0000) 2022-09-08T06:27:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:40.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:40.715137+0000) 2022-09-08T06:27:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:40 smithi055 ceph-mon[97033]: pgmap v17155: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:40 smithi059 ceph-mon[97450]: pgmap v17155: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:41.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:41.721144+0000) 2022-09-08T06:27:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:41.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:41.715372+0000) 2022-09-08T06:27:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:42.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:42.715564+0000) 2022-09-08T06:27:43.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:42.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:42.721382+0000) 2022-09-08T06:27:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:42 smithi055 ceph-mon[97033]: pgmap v17156: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:27:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:42 smithi059 ceph-mon[97450]: pgmap v17156: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:27:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:43.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:43.715780+0000) 2022-09-08T06:27:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:43.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:43.721538+0000) 2022-09-08T06:27:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:44.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:44.715902+0000) 2022-09-08T06:27:45.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:44.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:44.721763+0000) 2022-09-08T06:27:45.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:44 smithi055 ceph-mon[97033]: pgmap v17157: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:44 smithi059 ceph-mon[97450]: pgmap v17157: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:45.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:45.721885+0000) 2022-09-08T06:27:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:45.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:45.716135+0000) 2022-09-08T06:27:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:46.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:46.722000+0000) 2022-09-08T06:27:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:46.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:46.716332+0000) 2022-09-08T06:27:47.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:46 smithi055 ceph-mon[97033]: pgmap v17158: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:27:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:46 smithi059 ceph-mon[97450]: pgmap v17158: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:27:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:47.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:47.722150+0000) 2022-09-08T06:27:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:47.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:47.716490+0000) 2022-09-08T06:27:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:48.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:48.722297+0000) 2022-09-08T06:27:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:48.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:48.716668+0000) 2022-09-08T06:27:49.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:48 smithi055 ceph-mon[97033]: pgmap v17159: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:48 smithi059 ceph-mon[97450]: pgmap v17159: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:49.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:49.722526+0000) 2022-09-08T06:27:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:49.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:49.716894+0000) 2022-09-08T06:27:50.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:27:50.478+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:27:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:50.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:50.722791+0000) 2022-09-08T06:27:50.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:50.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:50.717090+0000) 2022-09-08T06:27:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:50 smithi055 ceph-mon[97033]: pgmap v17160: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:50 smithi059 ceph-mon[97450]: pgmap v17160: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:51.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:51.722931+0000) 2022-09-08T06:27:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:51.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:51.717250+0000) 2022-09-08T06:27:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:52.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:52.723065+0000) 2022-09-08T06:27:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:52.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:52.717505+0000) 2022-09-08T06:27:53.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:52 smithi055 ceph-mon[97033]: pgmap v17161: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:27:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:52 smithi059 ceph-mon[97450]: pgmap v17161: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:27:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:53.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:53.717737+0000) 2022-09-08T06:27:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:53.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:53.723267+0000) 2022-09-08T06:27:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:54.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:54.717923+0000) 2022-09-08T06:27:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:54.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:54.723509+0000) 2022-09-08T06:27:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:54 smithi055 ceph-mon[97033]: pgmap v17162: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:54 smithi059 ceph-mon[97450]: pgmap v17162: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:55.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:55.723787+0000) 2022-09-08T06:27:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:55.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:55.718056+0000) 2022-09-08T06:27:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:56.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:56.723971+0000) 2022-09-08T06:27:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:56.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:56.718242+0000) 2022-09-08T06:27:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:56 smithi055 ceph-mon[97033]: pgmap v17163: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:27:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:56 smithi059 ceph-mon[97450]: pgmap v17163: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:27:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:57.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:57.718486+0000) 2022-09-08T06:27:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:57.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:57.724158+0000) 2022-09-08T06:27:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:58.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:58.724385+0000) 2022-09-08T06:27:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:58.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:58.718751+0000) 2022-09-08T06:27:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:27:58 smithi055 ceph-mon[97033]: pgmap v17164: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:27:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:27:58 smithi059 ceph-mon[97450]: pgmap v17164: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:27:59.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:59.724581+0000) 2022-09-08T06:28:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:27:59.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:27:59.718998+0000) 2022-09-08T06:28:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:00.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:00.719139+0000) 2022-09-08T06:28:01.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:00.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:00.724760+0000) 2022-09-08T06:28:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:00 smithi055 ceph-mon[97033]: pgmap v17165: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:00 smithi059 ceph-mon[97450]: pgmap v17165: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:01.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:01.724933+0000) 2022-09-08T06:28:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:01.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:01.719360+0000) 2022-09-08T06:28:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:02.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:02.719541+0000) 2022-09-08T06:28:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:02.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:02.725157+0000) 2022-09-08T06:28:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:02 smithi055 ceph-mon[97033]: pgmap v17166: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:28:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:02 smithi059 ceph-mon[97450]: pgmap v17166: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:28:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:03.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:03.725425+0000) 2022-09-08T06:28:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:03.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:03.719730+0000) 2022-09-08T06:28:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:04.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:04.725739+0000) 2022-09-08T06:28:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:04.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:04.719975+0000) 2022-09-08T06:28:05.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:04 smithi055 ceph-mon[97033]: pgmap v17167: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:04 smithi059 ceph-mon[97450]: pgmap v17167: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:05.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:28:05.485+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:28:05.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:05.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:05.725942+0000) 2022-09-08T06:28:05.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:05.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:05.720214+0000) 2022-09-08T06:28:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:28:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:28:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:06.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:06.726114+0000) 2022-09-08T06:28:07.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:06.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:06.720352+0000) 2022-09-08T06:28:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:06 smithi055 ceph-mon[97033]: pgmap v17168: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:28:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:06 smithi055 ceph-mon[97033]: pgmap v17169: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T06:28:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:06 smithi055 ceph-mon[97033]: pgmap v17170: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T06:28:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:28:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:28:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:06 smithi059 ceph-mon[97450]: pgmap v17168: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:28:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:06 smithi059 ceph-mon[97450]: pgmap v17169: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T06:28:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:06 smithi059 ceph-mon[97450]: pgmap v17170: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T06:28:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:28:07.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:28:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:07.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:07.726303+0000) 2022-09-08T06:28:08.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:07.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:07.720556+0000) 2022-09-08T06:28:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:08.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:08.726532+0000) 2022-09-08T06:28:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:08.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:08.720779+0000) 2022-09-08T06:28:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:08 smithi055 ceph-mon[97033]: pgmap v17171: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T06:28:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:08 smithi059 ceph-mon[97450]: pgmap v17171: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T06:28:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:09.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:09.720994+0000) 2022-09-08T06:28:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:09.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:09.726796+0000) 2022-09-08T06:28:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:09 smithi055 ceph-mon[97033]: osdmap e1152: 12 total, 11 up, 11 in 2022-09-08T06:28:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:09 smithi059 ceph-mon[97450]: osdmap e1152: 12 total, 11 up, 11 in 2022-09-08T06:28:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:10.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:10.726975+0000) 2022-09-08T06:28:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:10.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:10.721222+0000) 2022-09-08T06:28:11.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:10 smithi055 ceph-mon[97033]: pgmap v17173: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T06:28:11.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:10 smithi059 ceph-mon[97450]: pgmap v17173: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T06:28:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:11.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:11.727164+0000) 2022-09-08T06:28:12.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:11.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:11.721437+0000) 2022-09-08T06:28:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:12 smithi055 ceph-mon[97033]: osdmap e1153: 12 total, 11 up, 11 in 2022-09-08T06:28:12.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:12 smithi055 ceph-mon[97033]: pgmap v17175: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2022-09-08T06:28:13.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:12 smithi059 ceph-mon[97450]: osdmap e1153: 12 total, 11 up, 11 in 2022-09-08T06:28:13.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:12 smithi059 ceph-mon[97450]: pgmap v17175: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2022-09-08T06:28:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:12.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:12.727320+0000) 2022-09-08T06:28:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:12.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:12.721697+0000) 2022-09-08T06:28:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:13.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:13.727533+0000) 2022-09-08T06:28:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:13.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:13.721912+0000) 2022-09-08T06:28:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:14.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:14.727781+0000) 2022-09-08T06:28:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:14.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:14.722170+0000) 2022-09-08T06:28:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:14 smithi059 ceph-mon[97450]: pgmap v17176: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:28:15.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:14 smithi055 ceph-mon[97033]: pgmap v17176: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:28:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:15.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:15.727974+0000) 2022-09-08T06:28:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:15.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:15.722322+0000) 2022-09-08T06:28:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:16.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:16.728202+0000) 2022-09-08T06:28:17.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:16.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:16.722536+0000) 2022-09-08T06:28:17.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:16 smithi059 ceph-mon[97450]: pgmap v17177: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:28:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:16 smithi055 ceph-mon[97033]: pgmap v17177: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:28:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:17.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:17.722750+0000) 2022-09-08T06:28:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:17.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:17.728363+0000) 2022-09-08T06:28:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:18.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:18.728568+0000) 2022-09-08T06:28:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:18.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:18.722905+0000) 2022-09-08T06:28:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:18 smithi059 ceph-mon[97450]: pgmap v17178: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T06:28:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:18 smithi055 ceph-mon[97033]: pgmap v17178: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T06:28:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:19.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:19.723042+0000) 2022-09-08T06:28:20.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:19.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:19.728815+0000) 2022-09-08T06:28:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:20.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:20.728958+0000) 2022-09-08T06:28:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:20.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:20.723213+0000) 2022-09-08T06:28:20.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:28:20.498+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:28:21.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:20 smithi059 ceph-mon[97450]: pgmap v17179: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:28:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:20 smithi055 ceph-mon[97033]: pgmap v17179: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:28:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:21.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:21.729181+0000) 2022-09-08T06:28:22.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:21.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:21.723367+0000) 2022-09-08T06:28:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:22 smithi055 ceph-mon[97033]: pgmap v17180: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T06:28:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:22 smithi059 ceph-mon[97450]: pgmap v17180: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T06:28:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:22.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:22.723504+0000) 2022-09-08T06:28:23.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:22.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:22.729382+0000) 2022-09-08T06:28:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:23.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:23.723627+0000) 2022-09-08T06:28:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:23.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:23.729601+0000) 2022-09-08T06:28:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:24.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:24.729864+0000) 2022-09-08T06:28:25.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:24.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:24.723846+0000) 2022-09-08T06:28:25.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:24 smithi059 ceph-mon[97450]: pgmap v17181: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:28:25.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:24 smithi055 ceph-mon[97033]: pgmap v17181: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:28:26.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:25.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:25.723980+0000) 2022-09-08T06:28:26.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:25.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:25.730057+0000) 2022-09-08T06:28:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:26.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:26.730294+0000) 2022-09-08T06:28:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:26.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:26.724125+0000) 2022-09-08T06:28:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:26 smithi059 ceph-mon[97450]: pgmap v17182: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:28:27.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:26 smithi055 ceph-mon[97033]: pgmap v17182: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:28:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:27.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:27.724246+0000) 2022-09-08T06:28:28.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:27.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:27.730470+0000) 2022-09-08T06:28:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:28.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:28.730663+0000) 2022-09-08T06:28:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:28.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:28.724410+0000) 2022-09-08T06:28:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:28 smithi059 ceph-mon[97450]: pgmap v17183: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:28 smithi055 ceph-mon[97033]: pgmap v17183: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:29.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:29.724549+0000) 2022-09-08T06:28:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:29.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:29.730882+0000) 2022-09-08T06:28:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:30.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:30.724716+0000) 2022-09-08T06:28:31.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:30.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:30.731049+0000) 2022-09-08T06:28:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:30 smithi059 ceph-mon[97450]: pgmap v17184: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:28:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:30 smithi055 ceph-mon[97033]: pgmap v17184: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:28:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:31.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:31.731204+0000) 2022-09-08T06:28:32.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:31.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:31.724880+0000) 2022-09-08T06:28:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:32 smithi055 ceph-mon[97033]: pgmap v17185: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:32 smithi059 ceph-mon[97450]: pgmap v17185: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:33.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:32.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:32.725043+0000) 2022-09-08T06:28:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:32.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:32.731460+0000) 2022-09-08T06:28:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:33.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:33.731701+0000) 2022-09-08T06:28:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:33.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:33.725169+0000) 2022-09-08T06:28:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:34.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:34.725306+0000) 2022-09-08T06:28:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:34.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:34.731873+0000) 2022-09-08T06:28:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:34 smithi059 ceph-mon[97450]: pgmap v17186: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:34 smithi055 ceph-mon[97033]: pgmap v17186: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:35.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:28:35.512+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:28:35.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:35.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:35.732017+0000) 2022-09-08T06:28:35.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:35.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:35.725531+0000) 2022-09-08T06:28:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:36.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:36.725687+0000) 2022-09-08T06:28:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:36.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:36.732189+0000) 2022-09-08T06:28:37.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:36 smithi059 ceph-mon[97450]: pgmap v17187: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:28:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:36 smithi055 ceph-mon[97033]: pgmap v17187: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:28:38.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:37.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:37.725815+0000) 2022-09-08T06:28:38.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:37.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:37.732445+0000) 2022-09-08T06:28:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:38.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:38.725951+0000) 2022-09-08T06:28:39.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:38.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:38.732612+0000) 2022-09-08T06:28:39.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:38 smithi059 ceph-mon[97450]: pgmap v17188: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:38 smithi055 ceph-mon[97033]: pgmap v17188: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:39.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:39.726097+0000) 2022-09-08T06:28:40.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:39.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:39.732858+0000) 2022-09-08T06:28:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:40.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:40.733007+0000) 2022-09-08T06:28:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:40.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:40.726244+0000) 2022-09-08T06:28:41.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:40 smithi059 ceph-mon[97450]: pgmap v17189: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:28:41.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:41 smithi055 ceph-mon[97033]: pgmap v17189: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:28:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:41.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:41.733203+0000) 2022-09-08T06:28:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:41.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:41.726391+0000) 2022-09-08T06:28:42.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:42 smithi055 ceph-mon[97033]: pgmap v17190: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:42 smithi059 ceph-mon[97450]: pgmap v17190: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:42.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:42.733394+0000) 2022-09-08T06:28:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:42.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:42.726565+0000) 2022-09-08T06:28:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:43.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:43.733586+0000) 2022-09-08T06:28:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:43.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:43.726728+0000) 2022-09-08T06:28:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:44.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:44.733737+0000) 2022-09-08T06:28:45.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:44.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:44.726865+0000) 2022-09-08T06:28:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:44 smithi059 ceph-mon[97450]: pgmap v17191: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:44 smithi055 ceph-mon[97033]: pgmap v17191: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:45.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:45.727092+0000) 2022-09-08T06:28:46.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:45.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:45.733874+0000) 2022-09-08T06:28:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:46.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:46.734004+0000) 2022-09-08T06:28:47.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:46.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:46.727234+0000) 2022-09-08T06:28:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:46 smithi059 ceph-mon[97450]: pgmap v17192: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:28:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:47 smithi055 ceph-mon[97033]: pgmap v17192: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:28:48.161 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:47.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:47.734181+0000) 2022-09-08T06:28:48.162 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:47.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:47.727370+0000) 2022-09-08T06:28:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:48.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:48.734329+0000) 2022-09-08T06:28:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:48.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:48.727517+0000) 2022-09-08T06:28:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:49 smithi059 ceph-mon[97450]: pgmap v17193: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:49 smithi055 ceph-mon[97033]: pgmap v17193: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:49.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:49.734473+0000) 2022-09-08T06:28:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:49.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:49.727656+0000) 2022-09-08T06:28:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:50.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:50.734731+0000) 2022-09-08T06:28:50.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:50.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:50.727796+0000) 2022-09-08T06:28:50.906 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:28:50.517+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:28:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:51 smithi059 ceph-mon[97450]: pgmap v17194: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:28:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:51 smithi055 ceph-mon[97033]: pgmap v17194: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:28:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:51.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:51.734940+0000) 2022-09-08T06:28:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:51.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:51.727921+0000) 2022-09-08T06:28:52.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:52 smithi055 ceph-mon[97033]: pgmap v17195: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:52 smithi059 ceph-mon[97450]: pgmap v17195: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:52.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:52.735111+0000) 2022-09-08T06:28:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:52.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:52.728053+0000) 2022-09-08T06:28:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:53.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:53.735276+0000) 2022-09-08T06:28:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:53.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:53.728179+0000) 2022-09-08T06:28:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:54.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:54.735541+0000) 2022-09-08T06:28:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:54.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:54.728374+0000) 2022-09-08T06:28:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:54 smithi059 ceph-mon[97450]: pgmap v17196: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:55 smithi055 ceph-mon[97033]: pgmap v17196: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:55.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:55.735766+0000) 2022-09-08T06:28:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:55.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:55.728523+0000) 2022-09-08T06:28:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:56.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:56.735976+0000) 2022-09-08T06:28:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:56.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:56.728669+0000) 2022-09-08T06:28:57.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:56 smithi059 ceph-mon[97450]: pgmap v17197: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:28:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:57 smithi055 ceph-mon[97033]: pgmap v17197: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:28:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:57.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:57.736203+0000) 2022-09-08T06:28:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:57.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:57.728795+0000) 2022-09-08T06:28:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:58.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:58.736425+0000) 2022-09-08T06:28:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:58.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:58.728916+0000) 2022-09-08T06:28:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:28:59 smithi059 ceph-mon[97450]: pgmap v17198: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:28:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:28:59 smithi055 ceph-mon[97033]: pgmap v17198: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:28:59.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:59.736676+0000) 2022-09-08T06:29:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:28:59.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:28:59.729046+0000) 2022-09-08T06:29:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:00.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:00.736916+0000) 2022-09-08T06:29:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:00.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:00.729186+0000) 2022-09-08T06:29:01.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:01 smithi059 ceph-mon[97450]: pgmap v17199: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:29:01.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:01 smithi055 ceph-mon[97033]: pgmap v17199: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:29:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:01.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:01.737086+0000) 2022-09-08T06:29:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:01.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:01.729307+0000) 2022-09-08T06:29:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:02 smithi055 ceph-mon[97033]: pgmap v17200: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:03.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:02 smithi059 ceph-mon[97450]: pgmap v17200: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:02.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:02.737301+0000) 2022-09-08T06:29:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:02.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:02.729430+0000) 2022-09-08T06:29:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:03.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:03.737513+0000) 2022-09-08T06:29:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:03.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:03.729575+0000) 2022-09-08T06:29:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:04.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:04.737746+0000) 2022-09-08T06:29:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:04.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:04.729745+0000) 2022-09-08T06:29:05.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:04 smithi059 ceph-mon[97450]: pgmap v17201: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:05 smithi055 ceph-mon[97033]: pgmap v17201: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:05.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:29:05.523+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:29:05.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:05.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:05.737914+0000) 2022-09-08T06:29:05.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:05.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:05.729947+0000) 2022-09-08T06:29:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:29:06.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:29:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:06.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:06.738112+0000) 2022-09-08T06:29:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:06.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:06.730074+0000) 2022-09-08T06:29:07.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:07 smithi059 ceph-mon[97450]: pgmap v17202: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:29:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:29:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:29:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:29:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:29:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:07 smithi055 ceph-mon[97033]: pgmap v17202: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:29:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:29:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:29:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:29:07.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:29:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:07.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:07.738326+0000) 2022-09-08T06:29:08.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:07.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:07.730241+0000) 2022-09-08T06:29:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:08.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:08.738510+0000) 2022-09-08T06:29:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:08.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:08.730431+0000) 2022-09-08T06:29:09.289 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:09 smithi059 ceph-mon[97450]: pgmap v17203: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:29:09.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:09 smithi055 ceph-mon[97033]: pgmap v17203: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:29:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:09.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:09.738629+0000) 2022-09-08T06:29:10.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:09.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:09.730586+0000) 2022-09-08T06:29:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:10 smithi059 ceph-mon[97450]: osdmap e1154: 12 total, 11 up, 11 in 2022-09-08T06:29:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:10 smithi055 ceph-mon[97033]: osdmap e1154: 12 total, 11 up, 11 in 2022-09-08T06:29:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:10.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:10.738793+0000) 2022-09-08T06:29:11.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:10.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:10.730710+0000) 2022-09-08T06:29:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:11 smithi059 ceph-mon[97450]: pgmap v17205: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:29:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:11 smithi055 ceph-mon[97033]: pgmap v17205: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:29:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:11.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:11.738997+0000) 2022-09-08T06:29:12.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:11.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:11.730842+0000) 2022-09-08T06:29:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:12 smithi055 ceph-mon[97033]: osdmap e1155: 12 total, 11 up, 11 in 2022-09-08T06:29:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:12 smithi055 ceph-mon[97033]: pgmap v17207: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:29:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:12 smithi059 ceph-mon[97450]: osdmap e1155: 12 total, 11 up, 11 in 2022-09-08T06:29:13.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:12 smithi059 ceph-mon[97450]: pgmap v17207: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:29:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:12.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:12.739172+0000) 2022-09-08T06:29:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:12.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:12.730986+0000) 2022-09-08T06:29:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:13.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:13.739381+0000) 2022-09-08T06:29:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:13.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:13.731148+0000) 2022-09-08T06:29:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:14.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:14.739537+0000) 2022-09-08T06:29:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:14.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:14.731312+0000) 2022-09-08T06:29:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:15 smithi059 ceph-mon[97450]: pgmap v17208: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:29:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:15 smithi055 ceph-mon[97033]: pgmap v17208: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:29:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:15.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:15.739729+0000) 2022-09-08T06:29:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:15.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:15.731502+0000) 2022-09-08T06:29:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:16.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:16.731760+0000) 2022-09-08T06:29:17.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:16.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:16.739963+0000) 2022-09-08T06:29:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:17 smithi059 ceph-mon[97450]: pgmap v17209: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:29:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:17 smithi055 ceph-mon[97033]: pgmap v17209: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:29:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:17.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:17.740202+0000) 2022-09-08T06:29:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:17.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:17.731958+0000) 2022-09-08T06:29:19.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:18.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:18.732122+0000) 2022-09-08T06:29:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:18.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:18.740378+0000) 2022-09-08T06:29:19.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:19 smithi059 ceph-mon[97450]: pgmap v17210: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T06:29:19.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:19 smithi055 ceph-mon[97033]: pgmap v17210: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T06:29:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:19.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:19.740580+0000) 2022-09-08T06:29:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:19.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:19.732272+0000) 2022-09-08T06:29:20.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:29:20.538+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:29:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:20.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:20.740862+0000) 2022-09-08T06:29:20.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:20.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:20.732470+0000) 2022-09-08T06:29:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:21 smithi059 ceph-mon[97450]: pgmap v17211: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:29:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:21 smithi055 ceph-mon[97033]: pgmap v17211: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:29:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:21.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:21.741010+0000) 2022-09-08T06:29:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:21.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:21.732617+0000) 2022-09-08T06:29:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:22 smithi055 ceph-mon[97033]: pgmap v17212: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T06:29:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:22 smithi059 ceph-mon[97450]: pgmap v17212: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T06:29:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:22.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:22.741233+0000) 2022-09-08T06:29:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:22.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:22.732779+0000) 2022-09-08T06:29:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:23.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:23.741446+0000) 2022-09-08T06:29:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:23.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:23.732922+0000) 2022-09-08T06:29:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:24.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:24.733083+0000) 2022-09-08T06:29:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:24.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:24.741627+0000) 2022-09-08T06:29:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:25 smithi059 ceph-mon[97450]: pgmap v17213: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:29:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:25 smithi055 ceph-mon[97033]: pgmap v17213: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:29:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:25.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:25.741847+0000) 2022-09-08T06:29:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:25.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:25.733225+0000) 2022-09-08T06:29:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:26.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:26.742031+0000) 2022-09-08T06:29:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:26.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:26.733367+0000) 2022-09-08T06:29:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:27 smithi059 ceph-mon[97450]: pgmap v17214: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:29:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:27 smithi055 ceph-mon[97033]: pgmap v17214: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:29:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:27.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:27.733538+0000) 2022-09-08T06:29:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:27.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:27.742183+0000) 2022-09-08T06:29:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:28.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:28.742308+0000) 2022-09-08T06:29:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:28.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:28.733779+0000) 2022-09-08T06:29:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:29 smithi059 ceph-mon[97450]: pgmap v17215: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:29 smithi055 ceph-mon[97033]: pgmap v17215: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:29.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:29.742531+0000) 2022-09-08T06:29:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:29.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:29.733967+0000) 2022-09-08T06:29:31.145 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:30.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:30.734217+0000) 2022-09-08T06:29:31.146 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:30.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:30.742762+0000) 2022-09-08T06:29:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:31 smithi059 ceph-mon[97450]: pgmap v17216: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:29:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:31 smithi055 ceph-mon[97033]: pgmap v17216: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:29:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:31.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:31.742922+0000) 2022-09-08T06:29:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:31.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:31.734479+0000) 2022-09-08T06:29:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:32 smithi055 ceph-mon[97033]: pgmap v17217: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:32 smithi059 ceph-mon[97450]: pgmap v17217: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:32.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:32.743084+0000) 2022-09-08T06:29:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:32.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:32.734648+0000) 2022-09-08T06:29:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:33.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:33.743257+0000) 2022-09-08T06:29:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:33.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:33.734817+0000) 2022-09-08T06:29:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:34.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:34.734974+0000) 2022-09-08T06:29:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:34.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:34.743484+0000) 2022-09-08T06:29:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:35 smithi059 ceph-mon[97450]: pgmap v17218: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:35.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:35 smithi055 ceph-mon[97033]: pgmap v17218: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:35.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:29:35.544+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:29:35.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:35.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:35.735100+0000) 2022-09-08T06:29:35.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:35.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:35.743718+0000) 2022-09-08T06:29:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:36.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:36.743883+0000) 2022-09-08T06:29:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:36.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:36.735239+0000) 2022-09-08T06:29:37.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:37 smithi055 ceph-mon[97033]: pgmap v17219: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:29:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:37 smithi055 ceph-mon[97033]: pgmap v17220: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:29:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:37 smithi055 ceph-mon[97033]: pgmap v17221: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:29:37.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:29:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:37 smithi059 ceph-mon[97450]: pgmap v17219: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:29:37.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:37 smithi059 ceph-mon[97450]: pgmap v17220: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:29:37.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:37 smithi059 ceph-mon[97450]: pgmap v17221: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:29:37.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:29:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:37.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:37.744046+0000) 2022-09-08T06:29:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:37.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:37.735401+0000) 2022-09-08T06:29:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:38.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:38.735610+0000) 2022-09-08T06:29:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:38.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:38.744278+0000) 2022-09-08T06:29:39.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:39 smithi055 ceph-mon[97033]: pgmap v17222: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:29:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:39 smithi059 ceph-mon[97450]: pgmap v17222: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:29:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:39.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:39.744493+0000) 2022-09-08T06:29:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:39.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:39.735702+0000) 2022-09-08T06:29:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:40.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:40.744676+0000) 2022-09-08T06:29:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:40.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:40.735907+0000) 2022-09-08T06:29:41.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:41 smithi055 ceph-mon[97033]: pgmap v17223: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:29:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:41 smithi059 ceph-mon[97450]: pgmap v17223: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:29:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:41.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:41.744806+0000) 2022-09-08T06:29:42.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:41.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:41.736117+0000) 2022-09-08T06:29:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:42 smithi059 ceph-mon[97450]: pgmap v17224: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:29:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:42.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:42.744976+0000) 2022-09-08T06:29:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:42.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:42.736260+0000) 2022-09-08T06:29:43.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:42 smithi055 ceph-mon[97033]: pgmap v17224: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:29:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:43.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:43.745198+0000) 2022-09-08T06:29:44.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:43.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:43.736453+0000) 2022-09-08T06:29:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:44.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:44.745415+0000) 2022-09-08T06:29:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:44.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:44.736671+0000) 2022-09-08T06:29:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:45 smithi059 ceph-mon[97450]: pgmap v17225: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:29:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:45 smithi055 ceph-mon[97033]: pgmap v17225: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:29:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:45.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:45.745586+0000) 2022-09-08T06:29:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:45.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:45.736812+0000) 2022-09-08T06:29:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:46.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:46.745810+0000) 2022-09-08T06:29:47.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:46.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:46.736960+0000) 2022-09-08T06:29:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:47 smithi059 ceph-mon[97450]: pgmap v17226: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:29:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:47 smithi055 ceph-mon[97033]: pgmap v17226: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:29:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:47.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:47.745983+0000) 2022-09-08T06:29:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:47.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:47.737128+0000) 2022-09-08T06:29:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:48.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:48.746099+0000) 2022-09-08T06:29:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:48.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:48.737305+0000) 2022-09-08T06:29:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:49 smithi059 ceph-mon[97450]: pgmap v17227: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:49.740 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:49 smithi055 ceph-mon[97033]: pgmap v17227: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:49.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:49.746221+0000) 2022-09-08T06:29:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:49.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:49.737516+0000) 2022-09-08T06:29:50.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:29:50.557+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:29:50.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:50.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:50.746422+0000) 2022-09-08T06:29:50.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:50.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:50.737788+0000) 2022-09-08T06:29:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:51 smithi059 ceph-mon[97450]: pgmap v17228: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:29:51.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:51 smithi055 ceph-mon[97033]: pgmap v17228: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:29:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:51.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:51.746697+0000) 2022-09-08T06:29:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:51.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:51.737921+0000) 2022-09-08T06:29:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:52 smithi059 ceph-mon[97450]: pgmap v17229: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:53.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:52.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:52.746824+0000) 2022-09-08T06:29:53.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:52.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:52.738051+0000) 2022-09-08T06:29:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:52 smithi055 ceph-mon[97033]: pgmap v17229: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:53.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:53.738274+0000) 2022-09-08T06:29:54.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:53.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:53.746962+0000) 2022-09-08T06:29:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:54.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:54.738480+0000) 2022-09-08T06:29:55.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:54.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:54.747112+0000) 2022-09-08T06:29:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:55 smithi059 ceph-mon[97450]: pgmap v17230: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:55 smithi055 ceph-mon[97033]: pgmap v17230: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:55.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:55.747322+0000) 2022-09-08T06:29:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:55.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:55.738717+0000) 2022-09-08T06:29:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:56.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:56.747498+0000) 2022-09-08T06:29:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:56.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:56.738948+0000) 2022-09-08T06:29:57.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:57 smithi059 ceph-mon[97450]: pgmap v17231: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:29:57.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:57 smithi055 ceph-mon[97033]: pgmap v17231: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:29:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:57.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:57.747653+0000) 2022-09-08T06:29:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:57.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:57.739135+0000) 2022-09-08T06:29:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:58.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:58.747876+0000) 2022-09-08T06:29:59.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:58.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:58.739366+0000) 2022-09-08T06:29:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:29:59 smithi059 ceph-mon[97450]: pgmap v17232: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:29:59.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:29:59 smithi055 ceph-mon[97033]: pgmap v17232: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:00.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:29:59.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:59.748052+0000) 2022-09-08T06:30:00.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:29:59.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:29:59.739501+0000) 2022-09-08T06:30:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T06:30:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T06:30:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T06:30:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T06:30:00.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T06:30:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T06:30:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T06:30:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T06:30:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T06:30:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T06:30:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:00.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:00.748266+0000) 2022-09-08T06:30:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:00.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:00.739654+0000) 2022-09-08T06:30:01.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:01 smithi059 ceph-mon[97450]: pgmap v17233: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:30:01.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:01 smithi055 ceph-mon[97033]: pgmap v17233: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:30:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:01.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:01.748534+0000) 2022-09-08T06:30:02.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:01.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:01.739779+0000) 2022-09-08T06:30:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:02 smithi059 ceph-mon[97450]: pgmap v17234: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:03.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:02.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:02.748808+0000) 2022-09-08T06:30:03.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:02.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:02.739941+0000) 2022-09-08T06:30:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:02 smithi055 ceph-mon[97033]: pgmap v17234: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:03.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:03.749064+0000) 2022-09-08T06:30:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:03.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:03.740158+0000) 2022-09-08T06:30:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:04.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:04.749274+0000) 2022-09-08T06:30:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:04.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:04.740386+0000) 2022-09-08T06:30:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:05 smithi059 ceph-mon[97450]: pgmap v17235: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:05 smithi055 ceph-mon[97033]: pgmap v17235: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:05.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:30:05.572+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:30:05.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:05.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:05.749518+0000) 2022-09-08T06:30:05.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:05.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:05.740601+0000) 2022-09-08T06:30:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:30:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:30:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:06.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:06.749804+0000) 2022-09-08T06:30:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:06.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:06.740711+0000) 2022-09-08T06:30:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:07 smithi059 ceph-mon[97450]: pgmap v17236: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:30:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:30:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:07 smithi055 ceph-mon[97033]: pgmap v17236: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:30:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:30:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:07.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:07.749927+0000) 2022-09-08T06:30:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:07.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:07.740946+0000) 2022-09-08T06:30:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:08.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:08.741119+0000) 2022-09-08T06:30:09.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:08.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:08.750136+0000) 2022-09-08T06:30:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:09 smithi055 ceph-mon[97033]: pgmap v17237: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:30:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:09 smithi059 ceph-mon[97450]: pgmap v17237: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:30:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:09.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:09.750407+0000) 2022-09-08T06:30:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:09.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:09.741308+0000) 2022-09-08T06:30:10.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:10 smithi055 ceph-mon[97033]: osdmap e1156: 12 total, 11 up, 11 in 2022-09-08T06:30:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:10 smithi055 ceph-mon[97033]: pgmap v17239: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:30:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:10 smithi059 ceph-mon[97450]: osdmap e1156: 12 total, 11 up, 11 in 2022-09-08T06:30:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:10 smithi059 ceph-mon[97450]: pgmap v17239: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:30:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:10.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:10.750700+0000) 2022-09-08T06:30:11.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:10.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:10.741529+0000) 2022-09-08T06:30:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:11.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:11.750964+0000) 2022-09-08T06:30:12.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:11.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:11.741747+0000) 2022-09-08T06:30:13.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:12 smithi059 ceph-mon[97450]: osdmap e1157: 12 total, 11 up, 11 in 2022-09-08T06:30:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:12 smithi059 ceph-mon[97450]: pgmap v17241: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:30:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:12.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:12.751104+0000) 2022-09-08T06:30:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:12.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:12.741900+0000) 2022-09-08T06:30:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:12 smithi055 ceph-mon[97033]: osdmap e1157: 12 total, 11 up, 11 in 2022-09-08T06:30:13.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:12 smithi055 ceph-mon[97033]: pgmap v17241: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:30:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:13.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:13.751323+0000) 2022-09-08T06:30:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:13.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:13.742048+0000) 2022-09-08T06:30:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:14.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:14.751577+0000) 2022-09-08T06:30:15.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:14.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:14.742230+0000) 2022-09-08T06:30:15.539 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:15 smithi059 ceph-mon[97450]: pgmap v17242: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:30:15.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:15 smithi055 ceph-mon[97033]: pgmap v17242: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:30:16.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:15.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:15.751785+0000) 2022-09-08T06:30:16.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:15.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:15.742438+0000) 2022-09-08T06:30:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:16.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:16.751995+0000) 2022-09-08T06:30:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:16.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:16.742619+0000) 2022-09-08T06:30:17.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:17 smithi059 ceph-mon[97450]: pgmap v17243: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:30:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:17 smithi055 ceph-mon[97033]: pgmap v17243: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:30:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:17.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:17.752139+0000) 2022-09-08T06:30:18.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:17.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:17.742801+0000) 2022-09-08T06:30:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:18.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:18.752369+0000) 2022-09-08T06:30:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:18.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:18.743038+0000) 2022-09-08T06:30:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:19 smithi059 ceph-mon[97450]: pgmap v17244: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T06:30:19.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:19 smithi055 ceph-mon[97033]: pgmap v17244: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T06:30:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:19.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:19.752608+0000) 2022-09-08T06:30:20.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:19.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:19.743245+0000) 2022-09-08T06:30:20.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:30:20.587+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:30:20.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:20.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:20.752821+0000) 2022-09-08T06:30:20.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:20.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:20.743504+0000) 2022-09-08T06:30:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:21 smithi059 ceph-mon[97450]: pgmap v17245: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:30:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:21 smithi055 ceph-mon[97033]: pgmap v17245: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:30:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:21.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:21.752963+0000) 2022-09-08T06:30:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:21.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:21.743691+0000) 2022-09-08T06:30:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:22 smithi059 ceph-mon[97450]: pgmap v17246: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2022-09-08T06:30:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:22.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:22.753172+0000) 2022-09-08T06:30:23.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:22.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:22.743882+0000) 2022-09-08T06:30:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:22 smithi055 ceph-mon[97033]: pgmap v17246: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2022-09-08T06:30:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:23.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:23.753388+0000) 2022-09-08T06:30:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:23.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:23.744054+0000) 2022-09-08T06:30:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:24.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:24.744227+0000) 2022-09-08T06:30:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:24.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:24.753580+0000) 2022-09-08T06:30:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:25 smithi059 ceph-mon[97450]: pgmap v17247: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:30:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:25 smithi055 ceph-mon[97033]: pgmap v17247: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:30:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:25.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:25.753810+0000) 2022-09-08T06:30:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:25.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:25.744384+0000) 2022-09-08T06:30:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:26.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:26.754033+0000) 2022-09-08T06:30:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:26.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:26.744576+0000) 2022-09-08T06:30:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:27 smithi059 ceph-mon[97450]: pgmap v17248: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:30:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:27 smithi055 ceph-mon[97033]: pgmap v17248: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:30:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:27.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:27.754212+0000) 2022-09-08T06:30:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:27.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:27.744735+0000) 2022-09-08T06:30:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:28.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:28.754365+0000) 2022-09-08T06:30:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:28.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:28.744953+0000) 2022-09-08T06:30:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:29 smithi059 ceph-mon[97450]: pgmap v17249: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:30:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:29 smithi055 ceph-mon[97033]: pgmap v17249: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:30:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:29.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:29.754548+0000) 2022-09-08T06:30:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:29.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:29.745107+0000) 2022-09-08T06:30:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:30.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:30.754747+0000) 2022-09-08T06:30:31.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:30.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:30.745245+0000) 2022-09-08T06:30:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:31 smithi059 ceph-mon[97450]: pgmap v17250: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:30:31.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:31 smithi055 ceph-mon[97033]: pgmap v17250: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:30:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:31.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:31.754862+0000) 2022-09-08T06:30:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:31.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:31.745486+0000) 2022-09-08T06:30:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:32 smithi059 ceph-mon[97450]: pgmap v17251: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T06:30:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:32.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:32.755071+0000) 2022-09-08T06:30:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:32.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:32.745740+0000) 2022-09-08T06:30:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:32 smithi055 ceph-mon[97033]: pgmap v17251: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T06:30:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:33.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:33.755239+0000) 2022-09-08T06:30:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:33.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:33.745936+0000) 2022-09-08T06:30:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:34.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:34.755413+0000) 2022-09-08T06:30:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:34.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:34.746097+0000) 2022-09-08T06:30:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:35 smithi059 ceph-mon[97450]: pgmap v17252: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T06:30:35.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:35 smithi055 ceph-mon[97033]: pgmap v17252: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T06:30:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:35.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:35.755629+0000) 2022-09-08T06:30:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:35.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:35.746241+0000) 2022-09-08T06:30:35.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:30:35.591+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:30:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:36.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:36.755779+0000) 2022-09-08T06:30:37.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:36.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:36.746486+0000) 2022-09-08T06:30:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:37 smithi059 ceph-mon[97450]: pgmap v17253: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:30:37.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:30:37.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:30:37.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:30:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:37 smithi055 ceph-mon[97033]: pgmap v17253: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:30:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:30:37.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:30:37.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:30:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:37.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:37.755933+0000) 2022-09-08T06:30:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:37.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:37.746701+0000) 2022-09-08T06:30:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:38.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:38.756119+0000) 2022-09-08T06:30:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:38.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:38.746907+0000) 2022-09-08T06:30:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:39 smithi059 ceph-mon[97450]: pgmap v17254: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:30:39.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:39 smithi055 ceph-mon[97033]: pgmap v17254: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:30:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:39.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:39.756327+0000) 2022-09-08T06:30:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:39.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:39.747065+0000) 2022-09-08T06:30:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:40.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:40.756581+0000) 2022-09-08T06:30:41.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:40.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:40.747311+0000) 2022-09-08T06:30:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:41 smithi055 ceph-mon[97033]: pgmap v17255: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:30:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:41 smithi059 ceph-mon[97450]: pgmap v17255: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:30:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:41.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:41.756694+0000) 2022-09-08T06:30:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:41.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:41.747549+0000) 2022-09-08T06:30:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:42 smithi059 ceph-mon[97450]: pgmap v17256: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:43.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:42.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:42.747797+0000) 2022-09-08T06:30:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:42.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:42.756934+0000) 2022-09-08T06:30:43.241 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:42 smithi055 ceph-mon[97033]: pgmap v17256: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:43.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:43.757097+0000) 2022-09-08T06:30:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:43.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:43.748034+0000) 2022-09-08T06:30:45.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:44.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:44.748248+0000) 2022-09-08T06:30:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:44.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:44.757226+0000) 2022-09-08T06:30:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:45 smithi059 ceph-mon[97450]: pgmap v17257: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:45 smithi055 ceph-mon[97033]: pgmap v17257: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:45.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:45.757471+0000) 2022-09-08T06:30:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:45.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:45.748474+0000) 2022-09-08T06:30:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:46.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:46.757691+0000) 2022-09-08T06:30:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:46.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:46.748739+0000) 2022-09-08T06:30:47.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:47 smithi059 ceph-mon[97450]: pgmap v17258: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:30:47.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:47 smithi055 ceph-mon[97033]: pgmap v17258: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:30:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:47.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:47.757920+0000) 2022-09-08T06:30:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:47.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:47.748964+0000) 2022-09-08T06:30:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:48.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:48.758112+0000) 2022-09-08T06:30:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:48.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:48.749132+0000) 2022-09-08T06:30:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:49 smithi055 ceph-mon[97033]: pgmap v17259: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:49.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:49 smithi059 ceph-mon[97450]: pgmap v17259: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:49.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:49.758325+0000) 2022-09-08T06:30:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:49.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:49.749350+0000) 2022-09-08T06:30:50.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:30:50.596+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:30:50.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:50.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:50.758562+0000) 2022-09-08T06:30:50.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:50.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:50.749600+0000) 2022-09-08T06:30:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:51 smithi055 ceph-mon[97033]: pgmap v17260: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:30:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:51 smithi059 ceph-mon[97450]: pgmap v17260: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:30:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:51.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:51.758742+0000) 2022-09-08T06:30:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:51.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:51.749856+0000) 2022-09-08T06:30:53.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:52 smithi059 ceph-mon[97450]: pgmap v17261: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:52.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:52.758911+0000) 2022-09-08T06:30:53.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:52.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:52.750016+0000) 2022-09-08T06:30:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:52 smithi055 ceph-mon[97033]: pgmap v17261: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:53.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:53.750233+0000) 2022-09-08T06:30:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:53.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:53.759123+0000) 2022-09-08T06:30:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:54.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:54.750405+0000) 2022-09-08T06:30:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:54.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:54.759379+0000) 2022-09-08T06:30:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:55 smithi059 ceph-mon[97450]: pgmap v17262: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:55.729 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:55 smithi055 ceph-mon[97033]: pgmap v17262: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:55.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:55.759604+0000) 2022-09-08T06:30:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:55.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:55.750580+0000) 2022-09-08T06:30:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:56.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:56.759817+0000) 2022-09-08T06:30:57.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:56.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:56.750707+0000) 2022-09-08T06:30:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:57 smithi055 ceph-mon[97033]: pgmap v17263: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:30:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:57 smithi059 ceph-mon[97450]: pgmap v17263: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:30:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:57.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:57.760018+0000) 2022-09-08T06:30:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:57.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:57.750843+0000) 2022-09-08T06:30:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:58.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:58.760191+0000) 2022-09-08T06:30:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:58.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:58.750979+0000) 2022-09-08T06:30:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:30:59 smithi055 ceph-mon[97033]: pgmap v17264: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:30:59.790 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:30:59 smithi059 ceph-mon[97450]: pgmap v17264: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:00.168 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:30:59.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:59.760414+0000) 2022-09-08T06:31:00.170 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:30:59.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:30:59.751201+0000) 2022-09-08T06:31:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:00 smithi055 ceph-mon[97033]: pgmap v17265: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:31:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:00 smithi059 ceph-mon[97450]: pgmap v17265: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:31:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:00.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:00.760572+0000) 2022-09-08T06:31:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:00.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:00.751352+0000) 2022-09-08T06:31:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:01.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:01.760772+0000) 2022-09-08T06:31:02.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:01.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:01.751530+0000) 2022-09-08T06:31:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:02 smithi059 ceph-mon[97450]: pgmap v17266: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:02.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:02.760972+0000) 2022-09-08T06:31:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:02.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:02.751670+0000) 2022-09-08T06:31:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:02 smithi055 ceph-mon[97033]: pgmap v17266: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:03.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:03.761181+0000) 2022-09-08T06:31:04.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:03.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:03.751840+0000) 2022-09-08T06:31:05.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:04.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:04.761432+0000) 2022-09-08T06:31:05.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:04.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:04.752059+0000) 2022-09-08T06:31:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:05 smithi059 ceph-mon[97450]: pgmap v17267: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:05 smithi055 ceph-mon[97033]: pgmap v17267: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:05.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:31:05.606+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:31:05.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:05.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:05.761741+0000) 2022-09-08T06:31:05.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:05.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:05.752310+0000) 2022-09-08T06:31:06.554 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:31:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:31:07.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:06.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:06.752461+0000) 2022-09-08T06:31:07.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:06.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:06.761883+0000) 2022-09-08T06:31:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:07 smithi055 ceph-mon[97033]: pgmap v17268: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:31:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:07 smithi055 ceph-mon[97033]: pgmap v17269: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T06:31:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:31:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:31:07.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:07 smithi059 ceph-mon[97450]: pgmap v17268: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:31:07.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:07 smithi059 ceph-mon[97450]: pgmap v17269: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T06:31:07.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:31:07.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:31:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:07.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:07.752607+0000) 2022-09-08T06:31:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:07.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:07.762083+0000) 2022-09-08T06:31:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:08.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:08.752791+0000) 2022-09-08T06:31:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:08.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:08.762272+0000) 2022-09-08T06:31:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:09 smithi059 ceph-mon[97450]: pgmap v17270: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T06:31:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:09 smithi055 ceph-mon[97033]: pgmap v17270: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T06:31:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:09.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:09.762508+0000) 2022-09-08T06:31:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:09.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:09.752945+0000) 2022-09-08T06:31:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:10 smithi059 ceph-mon[97450]: osdmap e1158: 12 total, 11 up, 11 in 2022-09-08T06:31:10.979 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:10 smithi055 ceph-mon[97033]: osdmap e1158: 12 total, 11 up, 11 in 2022-09-08T06:31:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:10.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:10.762804+0000) 2022-09-08T06:31:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:10.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:10.753098+0000) 2022-09-08T06:31:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:11 smithi055 ceph-mon[97033]: pgmap v17272: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T06:31:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:11 smithi059 ceph-mon[97450]: pgmap v17272: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T06:31:12.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:11.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:11.762978+0000) 2022-09-08T06:31:12.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:11.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:11.753229+0000) 2022-09-08T06:31:13.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:12.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:12.763128+0000) 2022-09-08T06:31:13.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:12.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:12.753412+0000) 2022-09-08T06:31:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:13 smithi059 ceph-mon[97450]: pgmap v17273: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T06:31:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:13.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:13.763260+0000) 2022-09-08T06:31:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:13.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:13.753556+0000) 2022-09-08T06:31:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:13 smithi055 ceph-mon[97033]: pgmap v17273: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 65 KiB/s wr, 29 op/s 2022-09-08T06:31:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:14.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:14.763475+0000) 2022-09-08T06:31:15.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:14.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:14.753779+0000) 2022-09-08T06:31:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:15 smithi059 ceph-mon[97450]: pgmap v17274: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 124 B/s wr, 3 op/s 2022-09-08T06:31:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:15.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:15.763755+0000) 2022-09-08T06:31:16.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:15.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:15.754017+0000) 2022-09-08T06:31:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:15 smithi055 ceph-mon[97033]: pgmap v17274: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 124 B/s wr, 3 op/s 2022-09-08T06:31:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:16.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:16.754249+0000) 2022-09-08T06:31:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:16.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:16.763955+0000) 2022-09-08T06:31:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:17 smithi059 ceph-mon[97450]: pgmap v17275: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:31:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:17.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:17.754491+0000) 2022-09-08T06:31:18.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:17.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:17.764118+0000) 2022-09-08T06:31:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:17 smithi055 ceph-mon[97033]: pgmap v17275: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:31:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:18.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:18.764309+0000) 2022-09-08T06:31:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:18.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:18.754749+0000) 2022-09-08T06:31:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:19 smithi059 ceph-mon[97450]: pgmap v17276: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:31:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:19.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:19.764522+0000) 2022-09-08T06:31:20.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:19.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:19.754948+0000) 2022-09-08T06:31:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:19 smithi055 ceph-mon[97033]: pgmap v17276: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:31:20.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:31:20.612+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:31:20.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:20.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:20.755135+0000) 2022-09-08T06:31:20.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:20.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:20.764769+0000) 2022-09-08T06:31:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:21 smithi059 ceph-mon[97450]: pgmap v17277: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T06:31:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:21 smithi059 ceph-mon[97450]: osdmap e1159: 12 total, 11 up, 11 in 2022-09-08T06:31:22.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:21.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:21.755279+0000) 2022-09-08T06:31:22.160 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:21.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:21.764919+0000) 2022-09-08T06:31:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:21 smithi055 ceph-mon[97033]: pgmap v17277: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.3 KiB/s rd, 187 B/s wr, 8 op/s 2022-09-08T06:31:22.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:21 smithi055 ceph-mon[97033]: osdmap e1159: 12 total, 11 up, 11 in 2022-09-08T06:31:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:22.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:22.755491+0000) 2022-09-08T06:31:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:22.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:22.765090+0000) 2022-09-08T06:31:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:23 smithi059 ceph-mon[97450]: pgmap v17279: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:31:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:23.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:23.755737+0000) 2022-09-08T06:31:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:23.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:23.765318+0000) 2022-09-08T06:31:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:23 smithi055 ceph-mon[97033]: pgmap v17279: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:31:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:24.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:24.765552+0000) 2022-09-08T06:31:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:24.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:24.755929+0000) 2022-09-08T06:31:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:25 smithi059 ceph-mon[97450]: pgmap v17280: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:31:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:25.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:25.765717+0000) 2022-09-08T06:31:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:25.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:25.756100+0000) 2022-09-08T06:31:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:25 smithi055 ceph-mon[97033]: pgmap v17280: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:31:26.767 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:26.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:26.756293+0000) 2022-09-08T06:31:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:26.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:26.765854+0000) 2022-09-08T06:31:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:27.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:27.766022+0000) 2022-09-08T06:31:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:27.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:27.756502+0000) 2022-09-08T06:31:28.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:27 smithi055 ceph-mon[97033]: pgmap v17281: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:31:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:27 smithi059 ceph-mon[97450]: pgmap v17281: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:31:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:28.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:28.766248+0000) 2022-09-08T06:31:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:28.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:28.756734+0000) 2022-09-08T06:31:30.164 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:29.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:29.766459+0000) 2022-09-08T06:31:30.164 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:29.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:29.756902+0000) 2022-09-08T06:31:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:29 smithi055 ceph-mon[97033]: pgmap v17282: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:31:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:29 smithi059 ceph-mon[97450]: pgmap v17282: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:31:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:30.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:30.766705+0000) 2022-09-08T06:31:31.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:30.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:30.757087+0000) 2022-09-08T06:31:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:31.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:31.766867+0000) 2022-09-08T06:31:32.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:31.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:31.757248+0000) 2022-09-08T06:31:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:31 smithi055 ceph-mon[97033]: pgmap v17283: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:31:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:31 smithi059 ceph-mon[97450]: pgmap v17283: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:31:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:32.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:32.767028+0000) 2022-09-08T06:31:33.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:32.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:32.757423+0000) 2022-09-08T06:31:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:33.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:33.767141+0000) 2022-09-08T06:31:34.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:33.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:33.757666+0000) 2022-09-08T06:31:34.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:33 smithi055 ceph-mon[97033]: pgmap v17284: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:31:34.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:33 smithi059 ceph-mon[97450]: pgmap v17284: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:31:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:34.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:34.767405+0000) 2022-09-08T06:31:35.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:34.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:34.757937+0000) 2022-09-08T06:31:35.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:31:35.614+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:31:35.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:35.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:35.767593+0000) 2022-09-08T06:31:35.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:35.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:35.758095+0000) 2022-09-08T06:31:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:35 smithi055 ceph-mon[97033]: pgmap v17285: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:35 smithi059 ceph-mon[97450]: pgmap v17285: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:36.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:36.767749+0000) 2022-09-08T06:31:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:36.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:36.758240+0000) 2022-09-08T06:31:38.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:37.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:37.767932+0000) 2022-09-08T06:31:38.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:37.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:37.758392+0000) 2022-09-08T06:31:38.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:37 smithi055 ceph-mon[97033]: pgmap v17286: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:31:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:37 smithi059 ceph-mon[97450]: pgmap v17286: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:31:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:38.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:38.768151+0000) 2022-09-08T06:31:39.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:38.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:38.758542+0000) 2022-09-08T06:31:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:39.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:39.768338+0000) 2022-09-08T06:31:40.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:39.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:39.758698+0000) 2022-09-08T06:31:40.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:39 smithi055 ceph-mon[97033]: pgmap v17287: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:40.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:39 smithi059 ceph-mon[97450]: pgmap v17287: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:41.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:40.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:40.768596+0000) 2022-09-08T06:31:41.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:40.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:40.758845+0000) 2022-09-08T06:31:42.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:41.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:41.768809+0000) 2022-09-08T06:31:42.159 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:41.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:41.759006+0000) 2022-09-08T06:31:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:41 smithi055 ceph-mon[97033]: pgmap v17288: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:31:42.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:41 smithi059 ceph-mon[97450]: pgmap v17288: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:31:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:42.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:42.768977+0000) 2022-09-08T06:31:43.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:42.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:42.759230+0000) 2022-09-08T06:31:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:43.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:43.769119+0000) 2022-09-08T06:31:44.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:43.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:43.759389+0000) 2022-09-08T06:31:44.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:43 smithi055 ceph-mon[97033]: pgmap v17289: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:44.290 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:43 smithi059 ceph-mon[97450]: pgmap v17289: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:45.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:44.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:44.769316+0000) 2022-09-08T06:31:45.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:44.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:44.759540+0000) 2022-09-08T06:31:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:45.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:45.769521+0000) 2022-09-08T06:31:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:45.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:45.759761+0000) 2022-09-08T06:31:46.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:45 smithi055 ceph-mon[97033]: pgmap v17290: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:45 smithi059 ceph-mon[97450]: pgmap v17290: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:47.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:46.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:46.759906+0000) 2022-09-08T06:31:47.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:46.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:46.769735+0000) 2022-09-08T06:31:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:47.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:47.769871+0000) 2022-09-08T06:31:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:47.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:47.760079+0000) 2022-09-08T06:31:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:47 smithi055 ceph-mon[97033]: pgmap v17291: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:31:48.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:47 smithi059 ceph-mon[97450]: pgmap v17291: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:31:49.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:48.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:48.770019+0000) 2022-09-08T06:31:49.169 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:48.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:48.760285+0000) 2022-09-08T06:31:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:49.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:49.760496+0000) 2022-09-08T06:31:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:49.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:49.770183+0000) 2022-09-08T06:31:50.239 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:49 smithi055 ceph-mon[97033]: pgmap v17292: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:49 smithi059 ceph-mon[97450]: pgmap v17292: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:50.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:31:50.630+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:31:50.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:50.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:50.770414+0000) 2022-09-08T06:31:50.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:50.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:50.760769+0000) 2022-09-08T06:31:52.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:51.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:51.770680+0000) 2022-09-08T06:31:52.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:51.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:51.760977+0000) 2022-09-08T06:31:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:51 smithi055 ceph-mon[97033]: pgmap v17293: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:31:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:51 smithi059 ceph-mon[97450]: pgmap v17293: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:31:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:52.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:52.761156+0000) 2022-09-08T06:31:53.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:52.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:52.770862+0000) 2022-09-08T06:31:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:53.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:53.771062+0000) 2022-09-08T06:31:54.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:53.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:53.761323+0000) 2022-09-08T06:31:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:53 smithi055 ceph-mon[97033]: pgmap v17294: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:53 smithi059 ceph-mon[97450]: pgmap v17294: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:54.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:54.771308+0000) 2022-09-08T06:31:55.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:54.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:54.761554+0000) 2022-09-08T06:31:56.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:55.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:55.771535+0000) 2022-09-08T06:31:56.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:55.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:55.761715+0000) 2022-09-08T06:31:56.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:55 smithi055 ceph-mon[97033]: pgmap v17295: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:56.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:55 smithi059 ceph-mon[97450]: pgmap v17295: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:31:57.164 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:56.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:56.771785+0000) 2022-09-08T06:31:57.166 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:56.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:56.761878+0000) 2022-09-08T06:31:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:57.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:57.771945+0000) 2022-09-08T06:31:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:57.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:57.762038+0000) 2022-09-08T06:31:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:57 smithi055 ceph-mon[97033]: pgmap v17296: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:31:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:57 smithi059 ceph-mon[97450]: pgmap v17296: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:31:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:58.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:58.762197+0000) 2022-09-08T06:31:59.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:58.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:58.772098+0000) 2022-09-08T06:32:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:31:59.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:59.772310+0000) 2022-09-08T06:32:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:31:59.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:31:59.762372+0000) 2022-09-08T06:32:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:31:59 smithi055 ceph-mon[97033]: pgmap v17297: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:32:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:31:59 smithi059 ceph-mon[97450]: pgmap v17297: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:32:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:00.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:00.762517+0000) 2022-09-08T06:32:01.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:00.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:00.772533+0000) 2022-09-08T06:32:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:01.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:01.772821+0000) 2022-09-08T06:32:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:01.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:01.762658+0000) 2022-09-08T06:32:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:01 smithi055 ceph-mon[97033]: pgmap v17298: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:32:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:01 smithi059 ceph-mon[97450]: pgmap v17298: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:32:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:02.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:02.773057+0000) 2022-09-08T06:32:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:02.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:02.762797+0000) 2022-09-08T06:32:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:03.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:03.773237+0000) 2022-09-08T06:32:04.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:03.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:03.762918+0000) 2022-09-08T06:32:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:03 smithi055 ceph-mon[97033]: pgmap v17299: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:32:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:03 smithi059 ceph-mon[97450]: pgmap v17299: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:32:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:04.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:04.773466+0000) 2022-09-08T06:32:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:04.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:04.763056+0000) 2022-09-08T06:32:05.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:32:05.642+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:32:05.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:05.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:05.763208+0000) 2022-09-08T06:32:05.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:05.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:05.773728+0000) 2022-09-08T06:32:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:05 smithi055 ceph-mon[97033]: pgmap v17300: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:32:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:32:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:05 smithi059 ceph-mon[97450]: pgmap v17300: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:32:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:32:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:06.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:06.773947+0000) 2022-09-08T06:32:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:06.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:06.763330+0000) 2022-09-08T06:32:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:32:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:32:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:32:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:32:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:32:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:32:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:32:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:32:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:07.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:07.763498+0000) 2022-09-08T06:32:08.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:07.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:07.774194+0000) 2022-09-08T06:32:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:07 smithi055 ceph-mon[97033]: pgmap v17301: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:32:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:07 smithi059 ceph-mon[97450]: pgmap v17301: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:32:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:08.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:08.763762+0000) 2022-09-08T06:32:09.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:08.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:08.774333+0000) 2022-09-08T06:32:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:09.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:09.774541+0000) 2022-09-08T06:32:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:09.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:09.763997+0000) 2022-09-08T06:32:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:09 smithi055 ceph-mon[97033]: pgmap v17302: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:32:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:09 smithi059 ceph-mon[97450]: pgmap v17302: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:32:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:10.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:10.774745+0000) 2022-09-08T06:32:11.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:10.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:10.764219+0000) 2022-09-08T06:32:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:10 smithi055 ceph-mon[97033]: osdmap e1160: 12 total, 11 up, 11 in 2022-09-08T06:32:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:10 smithi059 ceph-mon[97450]: osdmap e1160: 12 total, 11 up, 11 in 2022-09-08T06:32:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:11.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:11.774954+0000) 2022-09-08T06:32:12.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:11.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:11.764428+0000) 2022-09-08T06:32:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:11 smithi055 ceph-mon[97033]: pgmap v17304: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:32:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:11 smithi059 ceph-mon[97450]: pgmap v17304: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:32:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:12.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:12.775195+0000) 2022-09-08T06:32:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:12.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:12.764609+0000) 2022-09-08T06:32:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:13.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:13.775376+0000) 2022-09-08T06:32:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:13.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:13.764778+0000) 2022-09-08T06:32:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:13 smithi055 ceph-mon[97033]: pgmap v17305: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:32:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:13 smithi059 ceph-mon[97450]: pgmap v17305: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:32:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:14.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:14.775583+0000) 2022-09-08T06:32:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:14.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:14.765005+0000) 2022-09-08T06:32:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:15.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:15.775776+0000) 2022-09-08T06:32:16.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:15.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:15.765150+0000) 2022-09-08T06:32:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:15 smithi055 ceph-mon[97033]: pgmap v17306: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:32:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:15 smithi059 ceph-mon[97450]: pgmap v17306: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:32:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:16.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:16.775981+0000) 2022-09-08T06:32:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:16.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:16.765278+0000) 2022-09-08T06:32:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:17.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:17.776185+0000) 2022-09-08T06:32:18.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:17.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:17.765489+0000) 2022-09-08T06:32:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:17 smithi055 ceph-mon[97033]: pgmap v17307: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:32:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:17 smithi059 ceph-mon[97450]: pgmap v17307: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:32:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:18.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:18.776422+0000) 2022-09-08T06:32:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:18.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:18.765691+0000) 2022-09-08T06:32:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:19.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:19.776573+0000) 2022-09-08T06:32:20.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:19.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:19.765927+0000) 2022-09-08T06:32:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:19 smithi055 ceph-mon[97033]: pgmap v17308: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:32:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:19 smithi059 ceph-mon[97450]: pgmap v17308: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:32:21.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:32:20.652+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:32:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:20.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:20.776810+0000) 2022-09-08T06:32:21.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:20.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:20.766206+0000) 2022-09-08T06:32:22.166 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:21.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:21.776950+0000) 2022-09-08T06:32:22.167 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:21.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:21.766447+0000) 2022-09-08T06:32:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:21 smithi055 ceph-mon[97033]: pgmap v17309: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 193 B/s wr, 7 op/s 2022-09-08T06:32:22.570 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:21 smithi055 ceph-mon[97033]: osdmap e1161: 12 total, 11 up, 11 in 2022-09-08T06:32:22.571 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:21 smithi059 ceph-mon[97450]: pgmap v17309: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 193 B/s wr, 7 op/s 2022-09-08T06:32:22.572 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:21 smithi059 ceph-mon[97450]: osdmap e1161: 12 total, 11 up, 11 in 2022-09-08T06:32:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:22.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:22.777091+0000) 2022-09-08T06:32:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:22.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:22.766598+0000) 2022-09-08T06:32:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:23.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:23.777229+0000) 2022-09-08T06:32:24.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:23.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:23.766812+0000) 2022-09-08T06:32:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:23 smithi055 ceph-mon[97033]: pgmap v17311: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:32:24.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:23 smithi059 ceph-mon[97450]: pgmap v17311: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:32:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:24.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:24.777467+0000) 2022-09-08T06:32:25.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:24.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:24.766962+0000) 2022-09-08T06:32:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:25.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:25.777744+0000) 2022-09-08T06:32:26.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:25.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:25.767118+0000) 2022-09-08T06:32:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:25 smithi055 ceph-mon[97033]: pgmap v17312: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:32:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:25 smithi059 ceph-mon[97450]: pgmap v17312: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:32:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:26.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:26.778010+0000) 2022-09-08T06:32:27.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:26.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:26.767288+0000) 2022-09-08T06:32:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:27.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:27.778234+0000) 2022-09-08T06:32:28.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:27.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:27.767454+0000) 2022-09-08T06:32:28.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:27 smithi055 ceph-mon[97033]: pgmap v17313: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:32:28.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:27 smithi059 ceph-mon[97450]: pgmap v17313: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:32:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:28.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:28.778469+0000) 2022-09-08T06:32:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:28.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:28.767696+0000) 2022-09-08T06:32:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:29.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:29.767926+0000) 2022-09-08T06:32:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:29.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:29.778769+0000) 2022-09-08T06:32:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:29 smithi055 ceph-mon[97033]: pgmap v17314: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:32:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:29 smithi059 ceph-mon[97450]: pgmap v17314: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:32:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:30.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:30.768131+0000) 2022-09-08T06:32:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:30.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:30.779058+0000) 2022-09-08T06:32:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:31.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:31.779225+0000) 2022-09-08T06:32:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:31.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:31.768285+0000) 2022-09-08T06:32:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:31 smithi055 ceph-mon[97033]: pgmap v17315: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:32:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:31 smithi059 ceph-mon[97450]: pgmap v17315: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:32:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:32.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:32.779417+0000) 2022-09-08T06:32:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:32.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:32.768496+0000) 2022-09-08T06:32:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:33.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:33.779660+0000) 2022-09-08T06:32:34.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:33.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:33.768667+0000) 2022-09-08T06:32:34.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:33 smithi055 ceph-mon[97033]: pgmap v17316: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:32:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:33 smithi059 ceph-mon[97450]: pgmap v17316: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:32:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:34.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:34.779819+0000) 2022-09-08T06:32:35.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:34.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:34.768879+0000) 2022-09-08T06:32:36.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:32:35.667+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:32:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:35.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:35.780049+0000) 2022-09-08T06:32:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:35.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:35.769069+0000) 2022-09-08T06:32:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:35 smithi055 ceph-mon[97033]: pgmap v17317: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:32:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:35 smithi059 ceph-mon[97450]: pgmap v17317: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:32:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:36.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:36.780228+0000) 2022-09-08T06:32:37.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:36.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:36.769296+0000) 2022-09-08T06:32:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:37 smithi059 ceph-mon[97450]: pgmap v17318: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:32:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:37 smithi059 ceph-mon[97450]: pgmap v17319: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:32:38.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:37 smithi059 ceph-mon[97450]: pgmap v17320: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:32:38.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:32:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:37.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:37.780393+0000) 2022-09-08T06:32:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:37.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:37.769598+0000) 2022-09-08T06:32:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:37 smithi055 ceph-mon[97033]: pgmap v17318: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:32:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:37 smithi055 ceph-mon[97033]: pgmap v17319: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:32:38.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:37 smithi055 ceph-mon[97033]: pgmap v17320: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:32:38.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:32:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:38.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:38.780580+0000) 2022-09-08T06:32:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:38.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:38.769771+0000) 2022-09-08T06:32:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:39 smithi059 ceph-mon[97450]: pgmap v17321: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:32:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:39.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:39.780820+0000) 2022-09-08T06:32:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:39.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:39.769970+0000) 2022-09-08T06:32:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:39 smithi055 ceph-mon[97033]: pgmap v17321: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:32:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:40.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:40.781038+0000) 2022-09-08T06:32:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:40.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:40.770132+0000) 2022-09-08T06:32:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:41 smithi059 ceph-mon[97450]: pgmap v17322: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:32:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:41.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:41.781273+0000) 2022-09-08T06:32:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:41.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:41.770311+0000) 2022-09-08T06:32:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:41 smithi055 ceph-mon[97033]: pgmap v17322: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:32:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:42.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:42.781498+0000) 2022-09-08T06:32:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:42.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:42.770445+0000) 2022-09-08T06:32:44.030 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:43 smithi059 ceph-mon[97450]: pgmap v17323: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:32:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:43.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:43.781736+0000) 2022-09-08T06:32:44.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:43.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:43.770682+0000) 2022-09-08T06:32:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:43 smithi055 ceph-mon[97033]: pgmap v17323: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:32:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:44.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:44.781917+0000) 2022-09-08T06:32:45.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:44.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:44.770876+0000) 2022-09-08T06:32:46.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:45 smithi059 ceph-mon[97450]: pgmap v17324: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:32:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:45.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:45.782107+0000) 2022-09-08T06:32:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:45.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:45.771065+0000) 2022-09-08T06:32:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:45 smithi055 ceph-mon[97033]: pgmap v17324: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:32:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:46.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:46.782273+0000) 2022-09-08T06:32:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:46.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:46.771219+0000) 2022-09-08T06:32:48.113 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:47 smithi059 ceph-mon[97450]: pgmap v17325: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:32:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:47.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:47.782485+0000) 2022-09-08T06:32:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:47.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:47.771361+0000) 2022-09-08T06:32:48.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:47 smithi055 ceph-mon[97033]: pgmap v17325: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:32:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:48.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:48.782734+0000) 2022-09-08T06:32:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:48.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:48.771490+0000) 2022-09-08T06:32:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:49.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:49.771654+0000) 2022-09-08T06:32:50.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:49.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:49.782871+0000) 2022-09-08T06:32:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:49 smithi055 ceph-mon[97033]: pgmap v17326: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:32:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:49 smithi059 ceph-mon[97450]: pgmap v17326: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:32:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:50.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:50.771803+0000) 2022-09-08T06:32:51.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:32:50.683+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:32:51.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:50.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:50.783075+0000) 2022-09-08T06:32:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:51.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:51.771994+0000) 2022-09-08T06:32:52.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:51.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:51.783311+0000) 2022-09-08T06:32:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:51 smithi055 ceph-mon[97033]: pgmap v17327: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:32:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:51 smithi059 ceph-mon[97450]: pgmap v17327: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:32:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:52.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:52.772143+0000) 2022-09-08T06:32:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:52.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:52.783542+0000) 2022-09-08T06:32:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:53.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:53.783774+0000) 2022-09-08T06:32:54.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:53.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:53.772351+0000) 2022-09-08T06:32:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:53 smithi055 ceph-mon[97033]: pgmap v17328: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:32:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:53 smithi059 ceph-mon[97450]: pgmap v17328: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:32:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:54.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:54.783991+0000) 2022-09-08T06:32:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:54.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:54.772526+0000) 2022-09-08T06:32:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:55.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:55.784191+0000) 2022-09-08T06:32:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:55.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:55.772726+0000) 2022-09-08T06:32:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:55 smithi055 ceph-mon[97033]: pgmap v17329: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:32:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:55 smithi059 ceph-mon[97450]: pgmap v17329: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:32:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:56.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:56.784380+0000) 2022-09-08T06:32:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:56.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:56.772870+0000) 2022-09-08T06:32:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:57.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:57.784514+0000) 2022-09-08T06:32:58.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:57.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:57.773033+0000) 2022-09-08T06:32:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:57 smithi055 ceph-mon[97033]: pgmap v17330: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:32:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:57 smithi059 ceph-mon[97450]: pgmap v17330: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:32:59.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:58.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:58.784706+0000) 2022-09-08T06:32:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:58.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:58.773265+0000) 2022-09-08T06:33:00.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:32:59.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:59.784873+0000) 2022-09-08T06:33:00.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:32:59.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:32:59.773481+0000) 2022-09-08T06:33:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:32:59 smithi055 ceph-mon[97033]: pgmap v17331: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:32:59 smithi059 ceph-mon[97450]: pgmap v17331: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:00.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:00.785074+0000) 2022-09-08T06:33:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:00.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:00.773699+0000) 2022-09-08T06:33:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:01.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:01.785301+0000) 2022-09-08T06:33:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:01.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:01.773839+0000) 2022-09-08T06:33:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:01 smithi055 ceph-mon[97033]: pgmap v17332: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:33:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:01 smithi059 ceph-mon[97450]: pgmap v17332: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:33:03.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:02.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:02.785528+0000) 2022-09-08T06:33:03.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:02.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:02.773987+0000) 2022-09-08T06:33:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:03.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:03.785763+0000) 2022-09-08T06:33:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:03.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:03.774122+0000) 2022-09-08T06:33:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:03 smithi055 ceph-mon[97033]: pgmap v17333: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:03 smithi059 ceph-mon[97450]: pgmap v17333: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:04.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:04.785968+0000) 2022-09-08T06:33:05.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:04.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:04.774273+0000) 2022-09-08T06:33:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:05.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:05.786158+0000) 2022-09-08T06:33:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:05.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:05.774506+0000) 2022-09-08T06:33:06.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:33:05.684+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:33:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:05 smithi055 ceph-mon[97033]: pgmap v17334: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:05 smithi059 ceph-mon[97450]: pgmap v17334: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:06.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:06.786372+0000) 2022-09-08T06:33:07.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:06.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:06.774758+0000) 2022-09-08T06:33:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:33:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:33:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:33:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:33:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:07.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:07.786514+0000) 2022-09-08T06:33:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:07.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:07.774995+0000) 2022-09-08T06:33:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:07 smithi055 ceph-mon[97033]: pgmap v17335: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:33:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:07 smithi059 ceph-mon[97450]: pgmap v17335: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:33:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:08.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:08.786709+0000) 2022-09-08T06:33:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:08.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:08.775167+0000) 2022-09-08T06:33:10.160 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:09.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:09.786841+0000) 2022-09-08T06:33:10.162 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:09.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:09.775305+0000) 2022-09-08T06:33:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:09 smithi055 ceph-mon[97033]: pgmap v17336: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:33:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:09 smithi059 ceph-mon[97450]: pgmap v17336: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:33:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:10.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:10.775538+0000) 2022-09-08T06:33:11.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:10.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:10.787081+0000) 2022-09-08T06:33:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:10 smithi055 ceph-mon[97033]: osdmap e1162: 12 total, 11 up, 11 in 2022-09-08T06:33:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:10 smithi059 ceph-mon[97450]: osdmap e1162: 12 total, 11 up, 11 in 2022-09-08T06:33:12.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:11.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:11.787336+0000) 2022-09-08T06:33:12.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:11.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:11.775709+0000) 2022-09-08T06:33:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:11 smithi055 ceph-mon[97033]: pgmap v17338: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:33:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:11 smithi059 ceph-mon[97450]: pgmap v17338: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:33:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:12.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:12.775865+0000) 2022-09-08T06:33:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:12.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:12.787567+0000) 2022-09-08T06:33:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:13.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:13.775995+0000) 2022-09-08T06:33:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:13.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:13.787761+0000) 2022-09-08T06:33:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:13 smithi055 ceph-mon[97033]: pgmap v17339: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:33:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:13 smithi059 ceph-mon[97450]: pgmap v17339: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:33:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:14.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:14.787991+0000) 2022-09-08T06:33:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:14.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:14.776338+0000) 2022-09-08T06:33:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:15.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:15.788178+0000) 2022-09-08T06:33:16.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:15.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:15.776478+0000) 2022-09-08T06:33:16.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:15 smithi055 ceph-mon[97033]: pgmap v17340: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:33:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:15 smithi059 ceph-mon[97450]: pgmap v17340: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:33:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:16.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:16.788320+0000) 2022-09-08T06:33:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:16.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:16.776657+0000) 2022-09-08T06:33:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:17.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:17.788515+0000) 2022-09-08T06:33:18.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:17.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:17.776836+0000) 2022-09-08T06:33:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:17 smithi055 ceph-mon[97033]: pgmap v17341: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:33:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:17 smithi059 ceph-mon[97450]: pgmap v17341: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:33:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:18.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:18.788743+0000) 2022-09-08T06:33:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:18.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:18.777011+0000) 2022-09-08T06:33:20.160 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:19.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:19.788882+0000) 2022-09-08T06:33:20.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:19.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:19.777222+0000) 2022-09-08T06:33:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:19 smithi055 ceph-mon[97033]: pgmap v17342: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:33:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:19 smithi059 ceph-mon[97450]: pgmap v17342: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:33:21.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:33:20.694+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:33:21.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:20.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:20.789007+0000) 2022-09-08T06:33:21.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:20.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:20.777377+0000) 2022-09-08T06:33:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:21.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:21.789171+0000) 2022-09-08T06:33:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:21.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:21.777579+0000) 2022-09-08T06:33:22.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:21 smithi055 ceph-mon[97033]: pgmap v17343: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 187 B/s wr, 7 op/s 2022-09-08T06:33:22.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:21 smithi055 ceph-mon[97033]: osdmap e1163: 12 total, 11 up, 11 in 2022-09-08T06:33:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:21 smithi059 ceph-mon[97450]: pgmap v17343: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 187 B/s wr, 7 op/s 2022-09-08T06:33:22.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:21 smithi059 ceph-mon[97450]: osdmap e1163: 12 total, 11 up, 11 in 2022-09-08T06:33:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:22.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:22.789422+0000) 2022-09-08T06:33:23.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:22.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:22.777792+0000) 2022-09-08T06:33:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:23.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:23.789690+0000) 2022-09-08T06:33:24.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:23.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:23.777949+0000) 2022-09-08T06:33:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:23 smithi055 ceph-mon[97033]: pgmap v17345: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:33:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:23 smithi059 ceph-mon[97450]: pgmap v17345: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:33:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:24.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:24.789878+0000) 2022-09-08T06:33:25.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:24.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:24.778134+0000) 2022-09-08T06:33:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:25.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:25.790132+0000) 2022-09-08T06:33:26.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:25.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:25.778273+0000) 2022-09-08T06:33:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:25 smithi055 ceph-mon[97033]: pgmap v17346: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:33:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:25 smithi059 ceph-mon[97450]: pgmap v17346: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:33:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:26.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:26.790332+0000) 2022-09-08T06:33:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:26.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:26.778399+0000) 2022-09-08T06:33:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:27.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:27.790558+0000) 2022-09-08T06:33:28.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:27.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:27.778624+0000) 2022-09-08T06:33:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:27 smithi055 ceph-mon[97033]: pgmap v17347: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:33:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:27 smithi059 ceph-mon[97450]: pgmap v17347: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:33:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:28.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:28.790694+0000) 2022-09-08T06:33:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:28.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:28.778848+0000) 2022-09-08T06:33:30.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:29.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:29.779065+0000) 2022-09-08T06:33:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:29.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:29.790831+0000) 2022-09-08T06:33:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:29 smithi055 ceph-mon[97033]: pgmap v17348: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:33:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:29 smithi059 ceph-mon[97450]: pgmap v17348: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:33:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:30.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:30.791066+0000) 2022-09-08T06:33:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:30.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:30.779229+0000) 2022-09-08T06:33:32.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:31.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:31.779368+0000) 2022-09-08T06:33:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:31.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:31.791230+0000) 2022-09-08T06:33:32.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:31 smithi055 ceph-mon[97033]: pgmap v17349: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:33:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:31 smithi059 ceph-mon[97450]: pgmap v17349: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:33:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:32.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:32.779598+0000) 2022-09-08T06:33:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:32.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:32.791409+0000) 2022-09-08T06:33:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:33.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:33.791627+0000) 2022-09-08T06:33:34.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:33.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:33.779797+0000) 2022-09-08T06:33:34.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:33 smithi055 ceph-mon[97033]: pgmap v17350: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:33:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:33 smithi059 ceph-mon[97450]: pgmap v17350: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:33:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:34.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:34.791884+0000) 2022-09-08T06:33:35.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:34.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:34.780059+0000) 2022-09-08T06:33:36.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:33:35.708+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:33:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:35.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:35.792118+0000) 2022-09-08T06:33:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:35.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:35.780203+0000) 2022-09-08T06:33:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:35 smithi055 ceph-mon[97033]: pgmap v17351: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:35 smithi059 ceph-mon[97450]: pgmap v17351: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:36.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:36.792242+0000) 2022-09-08T06:33:37.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:36.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:36.780403+0000) 2022-09-08T06:33:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:37.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:37.792369+0000) 2022-09-08T06:33:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:37.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:37.780665+0000) 2022-09-08T06:33:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:37 smithi055 ceph-mon[97033]: pgmap v17352: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:33:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:33:38.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:33:38.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:33:38.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:33:38.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:33:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:37 smithi059 ceph-mon[97450]: pgmap v17352: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:33:38.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:33:38.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:33:38.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:33:38.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:33:38.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:33:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:38.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:38.792589+0000) 2022-09-08T06:33:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:38.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:38.780850+0000) 2022-09-08T06:33:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:39 smithi059 ceph-mon[97450]: pgmap v17353: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:33:40.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:33:40.140 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:39 smithi055 ceph-mon[97033]: pgmap v17353: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:40.141 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:33:40.141 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:33:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:39.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:39.792772+0000) 2022-09-08T06:33:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:39.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:39.781026+0000) 2022-09-08T06:33:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:40.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:40.792967+0000) 2022-09-08T06:33:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:40.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:40.781167+0000) 2022-09-08T06:33:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:33:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:33:41.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:41 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 177.3M 2022-09-08T06:33:41.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:41 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 185941266: error parsing value: Value '185941266' is below minimum 939524096 2022-09-08T06:33:41.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:33:41.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:33:41.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:41 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 177.3M 2022-09-08T06:33:41.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:41 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 185941266: error parsing value: Value '185941266' is below minimum 939524096 2022-09-08T06:33:41.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:41.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:33:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:41.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:41.793200+0000) 2022-09-08T06:33:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:41.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:41.781299+0000) 2022-09-08T06:33:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:42 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T06:33:42.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:42 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T06:33:42.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:42 smithi055 ceph-mon[97033]: pgmap v17354: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:33:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:42 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T06:33:42.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:42 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T06:33:42.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:42 smithi059 ceph-mon[97450]: pgmap v17354: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:33:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:42.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:42.793436+0000) 2022-09-08T06:33:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:42.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:42.781436+0000) 2022-09-08T06:33:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:43.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:43.781567+0000) 2022-09-08T06:33:44.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:43.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:43.793687+0000) 2022-09-08T06:33:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:44 smithi055 ceph-mon[97033]: pgmap v17355: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:44 smithi059 ceph-mon[97450]: pgmap v17355: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:44.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:44.793920+0000) 2022-09-08T06:33:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:44.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:44.781709+0000) 2022-09-08T06:33:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:45.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:45.781933+0000) 2022-09-08T06:33:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:45.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:45.794127+0000) 2022-09-08T06:33:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:46 smithi055 ceph-mon[97033]: pgmap v17356: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:46 smithi059 ceph-mon[97450]: pgmap v17356: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:46.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:46.794353+0000) 2022-09-08T06:33:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:46.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:46.782177+0000) 2022-09-08T06:33:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:47.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:47.794572+0000) 2022-09-08T06:33:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:47.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:47.782449+0000) 2022-09-08T06:33:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:48 smithi055 ceph-mon[97033]: pgmap v17357: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:33:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:48 smithi059 ceph-mon[97450]: pgmap v17357: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:33:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:48.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:48.794788+0000) 2022-09-08T06:33:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:48.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:48.782689+0000) 2022-09-08T06:33:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:49.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:49.795015+0000) 2022-09-08T06:33:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:49.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:49.782835+0000) 2022-09-08T06:33:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:50 smithi055 ceph-mon[97033]: pgmap v17358: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:50 smithi059 ceph-mon[97450]: pgmap v17358: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:51.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:33:50.712+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:33:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:50.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:50.795228+0000) 2022-09-08T06:33:51.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:50.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:50.783036+0000) 2022-09-08T06:33:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:51.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:51.795453+0000) 2022-09-08T06:33:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:51.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:51.783201+0000) 2022-09-08T06:33:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:52 smithi055 ceph-mon[97033]: pgmap v17359: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:33:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:52 smithi059 ceph-mon[97450]: pgmap v17359: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:33:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:52.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:52.795655+0000) 2022-09-08T06:33:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:52.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:52.783381+0000) 2022-09-08T06:33:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:53.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:53.795886+0000) 2022-09-08T06:33:54.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:53.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:53.783577+0000) 2022-09-08T06:33:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:54 smithi055 ceph-mon[97033]: pgmap v17360: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:54 smithi059 ceph-mon[97450]: pgmap v17360: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:54.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:54.796042+0000) 2022-09-08T06:33:55.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:54.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:54.783760+0000) 2022-09-08T06:33:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:55.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:55.796291+0000) 2022-09-08T06:33:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:55.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:55.783934+0000) 2022-09-08T06:33:56.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:56 smithi055 ceph-mon[97033]: pgmap v17361: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:56 smithi059 ceph-mon[97450]: pgmap v17361: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:33:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:56.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:56.784116+0000) 2022-09-08T06:33:57.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:56.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:56.796495+0000) 2022-09-08T06:33:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:57.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:57.784360+0000) 2022-09-08T06:33:58.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:57.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:57.796724+0000) 2022-09-08T06:33:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:33:58 smithi055 ceph-mon[97033]: pgmap v17362: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:33:58.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:33:58 smithi059 ceph-mon[97450]: pgmap v17362: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:33:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:58.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:58.796853+0000) 2022-09-08T06:33:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:58.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:58.784559+0000) 2022-09-08T06:34:00.161 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:33:59.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:59.797057+0000) 2022-09-08T06:34:00.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:33:59.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:33:59.784794+0000) 2022-09-08T06:34:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:00 smithi055 ceph-mon[97033]: pgmap v17363: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:00 smithi059 ceph-mon[97450]: pgmap v17363: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:00.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:00.797294+0000) 2022-09-08T06:34:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:00.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:00.784955+0000) 2022-09-08T06:34:02.040 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:01.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:01.797509+0000) 2022-09-08T06:34:02.041 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:01.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:01.785134+0000) 2022-09-08T06:34:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:02 smithi055 ceph-mon[97033]: pgmap v17364: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:34:02.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:02 smithi059 ceph-mon[97450]: pgmap v17364: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:34:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:02.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:02.797717+0000) 2022-09-08T06:34:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:02.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:02.785264+0000) 2022-09-08T06:34:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:03.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:03.785456+0000) 2022-09-08T06:34:04.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:03.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:03.797940+0000) 2022-09-08T06:34:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:04 smithi055 ceph-mon[97033]: pgmap v17365: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:04 smithi059 ceph-mon[97450]: pgmap v17365: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:04.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:04.798158+0000) 2022-09-08T06:34:05.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:04.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:04.785648+0000) 2022-09-08T06:34:06.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:34:05.713+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:34:06.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:05.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:05.798424+0000) 2022-09-08T06:34:06.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:05.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:05.785852+0000) 2022-09-08T06:34:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:06 smithi055 ceph-mon[97033]: pgmap v17366: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:34:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:06 smithi059 ceph-mon[97450]: pgmap v17366: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:34:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:06.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:06.798623+0000) 2022-09-08T06:34:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:06.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:06.786016+0000) 2022-09-08T06:34:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:34:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:34:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:34:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:34:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:07.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:07.798921+0000) 2022-09-08T06:34:08.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:07.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:07.786282+0000) 2022-09-08T06:34:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:08 smithi055 ceph-mon[97033]: pgmap v17367: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:34:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:08 smithi055 ceph-mon[97033]: pgmap v17368: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T06:34:08.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:08 smithi055 ceph-mon[97033]: pgmap v17369: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T06:34:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:08 smithi059 ceph-mon[97450]: pgmap v17367: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:34:08.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:08 smithi059 ceph-mon[97450]: pgmap v17368: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T06:34:08.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:08 smithi059 ceph-mon[97450]: pgmap v17369: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T06:34:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:08.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:08.799176+0000) 2022-09-08T06:34:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:08.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:08.786471+0000) 2022-09-08T06:34:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:09.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:09.786630+0000) 2022-09-08T06:34:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:09.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:09.799382+0000) 2022-09-08T06:34:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:10 smithi055 ceph-mon[97033]: pgmap v17370: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T06:34:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:10 smithi055 ceph-mon[97033]: osdmap e1164: 12 total, 11 up, 11 in 2022-09-08T06:34:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:10 smithi059 ceph-mon[97450]: pgmap v17370: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T06:34:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:10 smithi059 ceph-mon[97450]: osdmap e1164: 12 total, 11 up, 11 in 2022-09-08T06:34:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:10.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:10.786962+0000) 2022-09-08T06:34:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:10.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:10.799596+0000) 2022-09-08T06:34:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:11.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:11.799729+0000) 2022-09-08T06:34:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:11.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:11.787219+0000) 2022-09-08T06:34:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:12 smithi055 ceph-mon[97033]: pgmap v17372: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T06:34:12.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:12 smithi055 ceph-mon[97033]: osdmap e1165: 12 total, 11 up, 11 in 2022-09-08T06:34:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:12 smithi059 ceph-mon[97450]: pgmap v17372: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T06:34:12.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:12 smithi059 ceph-mon[97450]: osdmap e1165: 12 total, 11 up, 11 in 2022-09-08T06:34:13.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:12.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:12.787447+0000) 2022-09-08T06:34:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:12.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:12.799956+0000) 2022-09-08T06:34:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:13.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:13.800236+0000) 2022-09-08T06:34:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:13.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:13.787613+0000) 2022-09-08T06:34:14.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:14 smithi055 ceph-mon[97033]: pgmap v17374: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2022-09-08T06:34:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:14 smithi059 ceph-mon[97450]: pgmap v17374: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2022-09-08T06:34:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:14.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:14.800494+0000) 2022-09-08T06:34:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:14.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:14.787797+0000) 2022-09-08T06:34:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:15.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:15.800686+0000) 2022-09-08T06:34:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:15.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:15.787977+0000) 2022-09-08T06:34:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:16 smithi055 ceph-mon[97033]: pgmap v17375: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:34:16.541 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:16 smithi059 ceph-mon[97450]: pgmap v17375: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:34:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:16.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:16.800909+0000) 2022-09-08T06:34:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:16.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:16.788128+0000) 2022-09-08T06:34:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:17.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:17.801118+0000) 2022-09-08T06:34:18.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:17.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:17.788282+0000) 2022-09-08T06:34:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:17 smithi055 ceph-mon[97033]: pgmap v17376: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:34:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:17 smithi059 ceph-mon[97450]: pgmap v17376: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:34:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:18.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:18.801355+0000) 2022-09-08T06:34:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:18.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:18.788400+0000) 2022-09-08T06:34:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:19.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:19.801491+0000) 2022-09-08T06:34:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:19.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:19.788575+0000) 2022-09-08T06:34:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:20 smithi055 ceph-mon[97033]: pgmap v17377: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 208 B/s wr, 6 op/s 2022-09-08T06:34:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:20 smithi059 ceph-mon[97450]: pgmap v17377: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 208 B/s wr, 6 op/s 2022-09-08T06:34:21.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:34:20.724+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:34:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:20.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:20.801723+0000) 2022-09-08T06:34:21.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:20.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:20.788759+0000) 2022-09-08T06:34:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:21.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:21.801930+0000) 2022-09-08T06:34:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:21.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:21.788924+0000) 2022-09-08T06:34:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:22 smithi055 ceph-mon[97033]: pgmap v17378: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:34:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:22 smithi059 ceph-mon[97450]: pgmap v17378: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:34:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:22.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:22.802145+0000) 2022-09-08T06:34:23.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:22.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:22.789158+0000) 2022-09-08T06:34:24.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:23.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:23.802341+0000) 2022-09-08T06:34:24.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:23.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:23.789387+0000) 2022-09-08T06:34:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:24 smithi055 ceph-mon[97033]: pgmap v17379: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T06:34:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:24 smithi059 ceph-mon[97450]: pgmap v17379: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T06:34:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:24.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:24.802616+0000) 2022-09-08T06:34:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:24.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:24.789553+0000) 2022-09-08T06:34:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:25.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:25.802775+0000) 2022-09-08T06:34:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:25.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:25.789704+0000) 2022-09-08T06:34:26.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:26 smithi055 ceph-mon[97033]: pgmap v17380: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:34:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:26 smithi059 ceph-mon[97450]: pgmap v17380: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:34:27.058 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:26.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:26.789848+0000) 2022-09-08T06:34:27.059 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:26.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:26.802898+0000) 2022-09-08T06:34:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:27.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:27.803073+0000) 2022-09-08T06:34:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:27.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:27.789995+0000) 2022-09-08T06:34:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:27 smithi055 ceph-mon[97033]: pgmap v17381: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:27 smithi059 ceph-mon[97450]: pgmap v17381: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:28.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:28.803313+0000) 2022-09-08T06:34:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:28.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:28.790219+0000) 2022-09-08T06:34:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:29.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:29.803516+0000) 2022-09-08T06:34:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:29.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:29.790389+0000) 2022-09-08T06:34:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:30 smithi055 ceph-mon[97033]: pgmap v17382: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:30 smithi059 ceph-mon[97450]: pgmap v17382: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:30.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:30.803752+0000) 2022-09-08T06:34:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:30.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:30.790566+0000) 2022-09-08T06:34:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:31.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:31.803990+0000) 2022-09-08T06:34:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:31.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:31.790719+0000) 2022-09-08T06:34:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:32 smithi055 ceph-mon[97033]: pgmap v17383: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:34:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:32 smithi059 ceph-mon[97450]: pgmap v17383: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:34:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:32.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:32.804230+0000) 2022-09-08T06:34:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:32.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:32.790882+0000) 2022-09-08T06:34:34.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:33.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:33.804473+0000) 2022-09-08T06:34:34.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:33.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:33.791093+0000) 2022-09-08T06:34:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:34 smithi055 ceph-mon[97033]: pgmap v17384: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:34 smithi059 ceph-mon[97450]: pgmap v17384: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:34.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:34.804748+0000) 2022-09-08T06:34:35.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:34.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:34.791281+0000) 2022-09-08T06:34:36.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:34:35.732+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:34:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:35.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:35.804971+0000) 2022-09-08T06:34:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:35.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:35.791447+0000) 2022-09-08T06:34:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:36 smithi055 ceph-mon[97033]: pgmap v17385: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:34:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:36 smithi059 ceph-mon[97450]: pgmap v17385: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:34:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:36.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:36.805231+0000) 2022-09-08T06:34:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:36.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:36.791702+0000) 2022-09-08T06:34:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:37.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:37.805511+0000) 2022-09-08T06:34:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:37.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:37.791917+0000) 2022-09-08T06:34:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:37 smithi055 ceph-mon[97033]: pgmap v17386: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:37 smithi059 ceph-mon[97450]: pgmap v17386: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:38.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:38.805774+0000) 2022-09-08T06:34:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:38.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:38.792089+0000) 2022-09-08T06:34:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:39.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:39.806017+0000) 2022-09-08T06:34:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:39.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:39.792266+0000) 2022-09-08T06:34:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:40 smithi055 ceph-mon[97033]: pgmap v17387: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:40 smithi059 ceph-mon[97450]: pgmap v17387: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:41.162 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:40.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:40.806174+0000) 2022-09-08T06:34:41.163 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:40.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:40.792460+0000) 2022-09-08T06:34:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:41.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:41.806417+0000) 2022-09-08T06:34:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:41.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:41.792594+0000) 2022-09-08T06:34:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:42 smithi055 ceph-mon[97033]: pgmap v17388: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:34:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:42 smithi059 ceph-mon[97450]: pgmap v17388: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:34:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:42.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:42.806609+0000) 2022-09-08T06:34:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:42.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:42.792797+0000) 2022-09-08T06:34:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:43.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:43.806834+0000) 2022-09-08T06:34:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:43.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:43.792966+0000) 2022-09-08T06:34:44.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:44 smithi055 ceph-mon[97033]: pgmap v17389: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:44 smithi059 ceph-mon[97450]: pgmap v17389: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:44.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:44.807038+0000) 2022-09-08T06:34:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:44.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:44.793112+0000) 2022-09-08T06:34:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:45.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:45.807224+0000) 2022-09-08T06:34:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:45.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:45.793312+0000) 2022-09-08T06:34:46.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:46 smithi055 ceph-mon[97033]: pgmap v17390: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:34:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:46 smithi059 ceph-mon[97450]: pgmap v17390: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:34:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:46.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:46.793477+0000) 2022-09-08T06:34:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:46.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:46.807433+0000) 2022-09-08T06:34:48.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:47.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:47.793616+0000) 2022-09-08T06:34:48.161 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:47.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:47.807694+0000) 2022-09-08T06:34:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:47 smithi055 ceph-mon[97033]: pgmap v17391: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:47 smithi059 ceph-mon[97450]: pgmap v17391: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:48.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:48.807928+0000) 2022-09-08T06:34:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:48.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:48.793740+0000) 2022-09-08T06:34:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:49.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:49.808144+0000) 2022-09-08T06:34:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:49.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:49.793934+0000) 2022-09-08T06:34:50.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:50 smithi055 ceph-mon[97033]: pgmap v17392: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:50 smithi059 ceph-mon[97450]: pgmap v17392: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:50.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:50.808352+0000) 2022-09-08T06:34:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:50.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:50.794138+0000) 2022-09-08T06:34:51.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:34:50.742+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:34:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:51.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:51.808585+0000) 2022-09-08T06:34:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:51.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:51.794350+0000) 2022-09-08T06:34:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:52 smithi055 ceph-mon[97033]: pgmap v17393: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:34:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:52 smithi059 ceph-mon[97450]: pgmap v17393: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:34:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:52.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:52.808702+0000) 2022-09-08T06:34:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:52.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:52.794578+0000) 2022-09-08T06:34:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:53.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:53.808880+0000) 2022-09-08T06:34:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:53.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:53.794785+0000) 2022-09-08T06:34:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:54 smithi055 ceph-mon[97033]: pgmap v17394: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:54 smithi059 ceph-mon[97450]: pgmap v17394: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:54.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:54.795006+0000) 2022-09-08T06:34:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:54.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:54.809113+0000) 2022-09-08T06:34:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:55.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:55.809296+0000) 2022-09-08T06:34:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:55.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:55.795175+0000) 2022-09-08T06:34:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:56 smithi055 ceph-mon[97033]: pgmap v17395: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:34:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:56 smithi059 ceph-mon[97450]: pgmap v17395: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:34:57.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:56.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:56.795383+0000) 2022-09-08T06:34:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:56.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:56.809509+0000) 2022-09-08T06:34:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:57.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:57.809739+0000) 2022-09-08T06:34:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:57.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:57.795580+0000) 2022-09-08T06:34:58.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:34:57 smithi055 ceph-mon[97033]: pgmap v17396: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:34:57 smithi059 ceph-mon[97450]: pgmap v17396: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:34:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:58.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:58.809949+0000) 2022-09-08T06:34:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:58.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:58.795775+0000) 2022-09-08T06:35:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:34:59.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:59.795955+0000) 2022-09-08T06:35:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:34:59.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:34:59.810164+0000) 2022-09-08T06:35:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:00 smithi055 ceph-mon[97033]: pgmap v17397: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:00 smithi059 ceph-mon[97450]: pgmap v17397: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:00.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:00.796080+0000) 2022-09-08T06:35:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:00.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:00.810364+0000) 2022-09-08T06:35:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:01.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:01.796254+0000) 2022-09-08T06:35:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:01.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:01.810518+0000) 2022-09-08T06:35:02.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:02 smithi055 ceph-mon[97033]: pgmap v17398: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:35:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:02 smithi059 ceph-mon[97450]: pgmap v17398: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:35:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:02.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:02.796468+0000) 2022-09-08T06:35:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:02.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:02.810737+0000) 2022-09-08T06:35:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:03.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:03.810969+0000) 2022-09-08T06:35:04.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:03.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:03.796657+0000) 2022-09-08T06:35:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:04 smithi055 ceph-mon[97033]: pgmap v17399: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:04 smithi059 ceph-mon[97450]: pgmap v17399: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:04.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:04.811255+0000) 2022-09-08T06:35:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:04.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:04.796889+0000) 2022-09-08T06:35:06.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:35:05.754+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:35:06.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:05.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:05.811520+0000) 2022-09-08T06:35:06.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:05.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:05.797021+0000) 2022-09-08T06:35:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:06 smithi055 ceph-mon[97033]: pgmap v17400: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:35:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:35:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:06 smithi059 ceph-mon[97450]: pgmap v17400: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:35:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:35:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:06.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:06.811735+0000) 2022-09-08T06:35:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:06.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:06.797187+0000) 2022-09-08T06:35:07.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:35:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:35:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:07.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:07.811935+0000) 2022-09-08T06:35:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:07.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:07.797415+0000) 2022-09-08T06:35:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:08 smithi055 ceph-mon[97033]: pgmap v17401: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:35:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:35:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:35:08.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:35:08.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:35:08.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:35:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:08 smithi059 ceph-mon[97450]: pgmap v17401: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:35:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:35:08.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:35:08.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:35:08.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:35:08.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:35:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:08.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:08.812176+0000) 2022-09-08T06:35:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:08.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:08.797550+0000) 2022-09-08T06:35:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:09.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:09.797758+0000) 2022-09-08T06:35:10.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:09.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:09.812427+0000) 2022-09-08T06:35:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:10 smithi055 ceph-mon[97033]: pgmap v17402: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:35:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:10 smithi055 ceph-mon[97033]: osdmap e1166: 12 total, 11 up, 11 in 2022-09-08T06:35:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:10 smithi059 ceph-mon[97450]: pgmap v17402: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:35:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:10 smithi059 ceph-mon[97450]: osdmap e1166: 12 total, 11 up, 11 in 2022-09-08T06:35:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:10.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:10.812731+0000) 2022-09-08T06:35:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:10.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:10.797928+0000) 2022-09-08T06:35:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:11.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:11.812932+0000) 2022-09-08T06:35:12.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:11.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:11.798096+0000) 2022-09-08T06:35:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:12 smithi055 ceph-mon[97033]: pgmap v17404: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:35:12.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:12 smithi055 ceph-mon[97033]: osdmap e1167: 12 total, 11 up, 11 in 2022-09-08T06:35:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:12 smithi059 ceph-mon[97450]: pgmap v17404: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:35:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:12 smithi059 ceph-mon[97450]: osdmap e1167: 12 total, 11 up, 11 in 2022-09-08T06:35:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:12.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:12.813194+0000) 2022-09-08T06:35:13.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:12.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:12.798324+0000) 2022-09-08T06:35:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:13.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:13.813458+0000) 2022-09-08T06:35:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:13.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:13.798468+0000) 2022-09-08T06:35:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:14 smithi055 ceph-mon[97033]: pgmap v17406: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:35:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:14 smithi059 ceph-mon[97450]: pgmap v17406: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:35:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:14.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:14.813649+0000) 2022-09-08T06:35:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:14.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:14.798716+0000) 2022-09-08T06:35:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:15.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:15.813886+0000) 2022-09-08T06:35:16.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:15.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:15.798870+0000) 2022-09-08T06:35:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:16 smithi055 ceph-mon[97033]: pgmap v17407: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:35:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:16 smithi059 ceph-mon[97450]: pgmap v17407: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:35:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:16.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:16.814119+0000) 2022-09-08T06:35:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:16.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:16.799052+0000) 2022-09-08T06:35:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:17.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:17.799288+0000) 2022-09-08T06:35:18.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:17.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:17.814308+0000) 2022-09-08T06:35:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:17 smithi055 ceph-mon[97033]: pgmap v17408: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:35:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:17 smithi059 ceph-mon[97450]: pgmap v17408: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:35:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:18.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:18.814508+0000) 2022-09-08T06:35:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:18.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:18.799551+0000) 2022-09-08T06:35:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:19.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:19.799764+0000) 2022-09-08T06:35:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:19.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:19.814756+0000) 2022-09-08T06:35:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:20 smithi055 ceph-mon[97033]: pgmap v17409: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:35:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:20 smithi059 ceph-mon[97450]: pgmap v17409: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:35:21.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:35:20.763+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:35:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:20.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:20.814940+0000) 2022-09-08T06:35:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:20.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:20.799976+0000) 2022-09-08T06:35:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:21.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:21.815169+0000) 2022-09-08T06:35:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:21.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:21.800147+0000) 2022-09-08T06:35:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:22 smithi055 ceph-mon[97033]: pgmap v17410: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:35:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:22 smithi059 ceph-mon[97450]: pgmap v17410: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:35:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:22.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:22.815429+0000) 2022-09-08T06:35:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:22.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:22.800412+0000) 2022-09-08T06:35:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:23.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:23.815666+0000) 2022-09-08T06:35:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:23.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:23.800585+0000) 2022-09-08T06:35:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:24 smithi055 ceph-mon[97033]: pgmap v17411: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T06:35:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:24 smithi059 ceph-mon[97450]: pgmap v17411: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 91 B/s wr, 5 op/s 2022-09-08T06:35:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:24.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:24.815900+0000) 2022-09-08T06:35:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:24.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:24.800788+0000) 2022-09-08T06:35:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:25.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:25.816102+0000) 2022-09-08T06:35:26.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:25.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:25.800979+0000) 2022-09-08T06:35:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:26 smithi055 ceph-mon[97033]: pgmap v17412: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:35:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:26 smithi059 ceph-mon[97450]: pgmap v17412: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:35:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:26.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:26.816354+0000) 2022-09-08T06:35:27.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:26.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:26.801211+0000) 2022-09-08T06:35:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:27.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:27.816555+0000) 2022-09-08T06:35:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:27.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:27.801365+0000) 2022-09-08T06:35:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:27 smithi055 ceph-mon[97033]: pgmap v17413: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:27 smithi059 ceph-mon[97450]: pgmap v17413: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:28.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:28.801500+0000) 2022-09-08T06:35:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:28.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:28.816784+0000) 2022-09-08T06:35:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:29.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:29.817019+0000) 2022-09-08T06:35:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:29.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:29.801655+0000) 2022-09-08T06:35:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:30 smithi055 ceph-mon[97033]: pgmap v17414: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:30 smithi059 ceph-mon[97450]: pgmap v17414: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:30.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:30.817277+0000) 2022-09-08T06:35:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:30.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:30.801834+0000) 2022-09-08T06:35:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:31.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:31.817520+0000) 2022-09-08T06:35:32.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:31.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:31.801976+0000) 2022-09-08T06:35:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:32 smithi055 ceph-mon[97033]: pgmap v17415: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:35:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:32 smithi059 ceph-mon[97450]: pgmap v17415: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:35:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:32.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:32.817718+0000) 2022-09-08T06:35:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:32.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:32.802191+0000) 2022-09-08T06:35:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:33.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:33.802445+0000) 2022-09-08T06:35:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:33.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:33.817898+0000) 2022-09-08T06:35:34.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:34 smithi055 ceph-mon[97033]: pgmap v17416: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:34 smithi059 ceph-mon[97450]: pgmap v17416: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:34.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:34.802611+0000) 2022-09-08T06:35:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:34.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:34.818129+0000) 2022-09-08T06:35:36.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:35:35.776+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:35:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:35.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:35.818388+0000) 2022-09-08T06:35:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:35.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:35.802773+0000) 2022-09-08T06:35:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:36 smithi055 ceph-mon[97033]: pgmap v17417: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:35:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:36 smithi059 ceph-mon[97450]: pgmap v17417: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:35:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:36.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:36.818696+0000) 2022-09-08T06:35:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:36.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:36.802900+0000) 2022-09-08T06:35:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:37.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:37.818909+0000) 2022-09-08T06:35:38.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:37.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:37.803057+0000) 2022-09-08T06:35:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:37 smithi055 ceph-mon[97033]: pgmap v17418: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:38.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:37 smithi059 ceph-mon[97450]: pgmap v17418: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:38.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:38.819149+0000) 2022-09-08T06:35:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:38.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:38.803258+0000) 2022-09-08T06:35:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:39.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:39.819389+0000) 2022-09-08T06:35:40.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:39.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:39.803491+0000) 2022-09-08T06:35:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:40 smithi055 ceph-mon[97033]: pgmap v17419: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:40 smithi059 ceph-mon[97450]: pgmap v17419: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:40.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:40.819667+0000) 2022-09-08T06:35:41.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:40.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:40.803670+0000) 2022-09-08T06:35:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:41.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:41.819900+0000) 2022-09-08T06:35:42.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:41.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:41.803858+0000) 2022-09-08T06:35:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:42 smithi059 ceph-mon[97450]: pgmap v17420: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:35:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:35:42.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:35:42.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:35:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:42 smithi055 ceph-mon[97033]: pgmap v17420: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:35:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:35:42.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:35:42.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:35:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:42.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:42.820099+0000) 2022-09-08T06:35:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:42.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:42.804048+0000) 2022-09-08T06:35:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:43 smithi055 ceph-mon[97033]: pgmap v17421: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:35:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:43 smithi055 ceph-mon[97033]: pgmap v17422: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T06:35:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:43 smithi059 ceph-mon[97450]: pgmap v17421: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:35:43.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:43 smithi059 ceph-mon[97450]: pgmap v17422: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T06:35:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:43.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:43.820299+0000) 2022-09-08T06:35:44.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:43.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:43.804247+0000) 2022-09-08T06:35:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:44.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:44.820552+0000) 2022-09-08T06:35:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:44.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:44.804407+0000) 2022-09-08T06:35:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:45 smithi055 ceph-mon[97033]: pgmap v17423: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T06:35:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:45 smithi059 ceph-mon[97450]: pgmap v17423: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T06:35:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:45.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:45.820802+0000) 2022-09-08T06:35:46.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:45.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:45.804597+0000) 2022-09-08T06:35:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:46.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:46.821016+0000) 2022-09-08T06:35:47.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:46.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:46.804749+0000) 2022-09-08T06:35:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:47 smithi055 ceph-mon[97033]: pgmap v17424: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T06:35:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:47 smithi059 ceph-mon[97450]: pgmap v17424: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T06:35:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:47.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:47.821168+0000) 2022-09-08T06:35:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:47.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:47.804888+0000) 2022-09-08T06:35:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:48.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:48.821377+0000) 2022-09-08T06:35:49.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:48.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:48.805103+0000) 2022-09-08T06:35:49.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:49 smithi055 ceph-mon[97033]: pgmap v17425: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T06:35:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:49 smithi059 ceph-mon[97450]: pgmap v17425: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T06:35:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:49.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:49.821603+0000) 2022-09-08T06:35:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:49.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:49.805248+0000) 2022-09-08T06:35:51.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:35:50.791+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:35:51.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:50.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:50.821808+0000) 2022-09-08T06:35:51.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:50.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:50.805466+0000) 2022-09-08T06:35:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:51 smithi055 ceph-mon[97033]: pgmap v17426: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T06:35:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:51 smithi059 ceph-mon[97450]: pgmap v17426: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T06:35:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:51.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:51.822004+0000) 2022-09-08T06:35:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:51.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:51.805654+0000) 2022-09-08T06:35:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:52.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:52.822222+0000) 2022-09-08T06:35:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:52.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:52.805873+0000) 2022-09-08T06:35:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:53 smithi055 ceph-mon[97033]: pgmap v17427: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:35:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:53 smithi059 ceph-mon[97450]: pgmap v17427: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:35:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:53.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:53.822482+0000) 2022-09-08T06:35:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:53.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:53.806060+0000) 2022-09-08T06:35:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:54.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:54.806218+0000) 2022-09-08T06:35:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:54.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:54.822777+0000) 2022-09-08T06:35:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:55 smithi055 ceph-mon[97033]: pgmap v17428: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:55 smithi059 ceph-mon[97450]: pgmap v17428: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:55.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:55.823033+0000) 2022-09-08T06:35:56.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:55.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:55.806377+0000) 2022-09-08T06:35:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:56.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:56.823251+0000) 2022-09-08T06:35:57.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:56.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:56.806538+0000) 2022-09-08T06:35:57.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:57 smithi055 ceph-mon[97033]: pgmap v17429: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:35:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:57 smithi059 ceph-mon[97450]: pgmap v17429: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:35:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:57.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:57.823483+0000) 2022-09-08T06:35:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:57.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:57.806757+0000) 2022-09-08T06:35:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:58.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:58.823743+0000) 2022-09-08T06:35:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:58.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:58.806917+0000) 2022-09-08T06:35:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:35:59 smithi055 ceph-mon[97033]: pgmap v17430: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:35:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:35:59 smithi059 ceph-mon[97450]: pgmap v17430: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:35:59.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:59.823962+0000) 2022-09-08T06:36:00.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:35:59.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:35:59.807157+0000) 2022-09-08T06:36:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:00.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:00.824114+0000) 2022-09-08T06:36:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:00.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:00.807342+0000) 2022-09-08T06:36:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:01 smithi055 ceph-mon[97033]: pgmap v17431: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:01 smithi059 ceph-mon[97450]: pgmap v17431: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:01.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:01.807488+0000) 2022-09-08T06:36:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:01.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:01.824369+0000) 2022-09-08T06:36:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:02.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:02.824683+0000) 2022-09-08T06:36:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:02.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:02.807732+0000) 2022-09-08T06:36:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:03 smithi055 ceph-mon[97033]: pgmap v17432: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:36:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:03 smithi059 ceph-mon[97450]: pgmap v17432: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:36:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:03.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:03.824907+0000) 2022-09-08T06:36:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:03.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:03.807858+0000) 2022-09-08T06:36:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:04.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:04.825127+0000) 2022-09-08T06:36:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:04.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:04.807998+0000) 2022-09-08T06:36:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:05 smithi055 ceph-mon[97033]: pgmap v17433: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:05 smithi059 ceph-mon[97450]: pgmap v17433: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:06.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:05.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:05.808139+0000) 2022-09-08T06:36:06.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:36:05.806+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:36:06.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:05.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:05.825376+0000) 2022-09-08T06:36:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:36:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:36:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:06.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:06.825546+0000) 2022-09-08T06:36:07.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:06.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:06.808312+0000) 2022-09-08T06:36:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:07 smithi055 ceph-mon[97033]: pgmap v17434: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:36:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:36:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:07 smithi059 ceph-mon[97450]: pgmap v17434: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:36:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:36:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:07.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:07.808462+0000) 2022-09-08T06:36:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:07.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:07.825752+0000) 2022-09-08T06:36:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:08.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:08.825882+0000) 2022-09-08T06:36:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:08.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:08.808689+0000) 2022-09-08T06:36:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:09 smithi055 ceph-mon[97033]: pgmap v17435: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:36:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:09 smithi059 ceph-mon[97450]: pgmap v17435: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:36:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:09.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:09.826133+0000) 2022-09-08T06:36:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:09.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:09.808887+0000) 2022-09-08T06:36:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:10 smithi055 ceph-mon[97033]: osdmap e1168: 12 total, 11 up, 11 in 2022-09-08T06:36:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:10 smithi059 ceph-mon[97450]: osdmap e1168: 12 total, 11 up, 11 in 2022-09-08T06:36:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:10.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:10.826381+0000) 2022-09-08T06:36:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:10.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:10.809051+0000) 2022-09-08T06:36:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:11 smithi055 ceph-mon[97033]: pgmap v17437: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:36:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:11 smithi059 ceph-mon[97450]: pgmap v17437: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:36:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:11.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:11.826587+0000) 2022-09-08T06:36:12.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:11.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:11.809200+0000) 2022-09-08T06:36:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:12.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:12.826805+0000) 2022-09-08T06:36:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:12.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:12.809404+0000) 2022-09-08T06:36:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:13 smithi055 ceph-mon[97033]: pgmap v17438: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:36:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:13 smithi059 ceph-mon[97450]: pgmap v17438: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:36:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:13.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:13.826971+0000) 2022-09-08T06:36:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:13.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:13.809599+0000) 2022-09-08T06:36:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:14.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:14.827234+0000) 2022-09-08T06:36:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:14.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:14.809767+0000) 2022-09-08T06:36:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:15 smithi055 ceph-mon[97033]: pgmap v17439: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:36:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:15 smithi059 ceph-mon[97450]: pgmap v17439: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:36:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:15.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:15.827474+0000) 2022-09-08T06:36:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:15.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:15.809918+0000) 2022-09-08T06:36:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:16.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:16.827746+0000) 2022-09-08T06:36:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:16.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:16.810087+0000) 2022-09-08T06:36:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:17 smithi055 ceph-mon[97033]: pgmap v17440: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:36:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:17 smithi055 ceph-mon[97033]: osdmap e1169: 12 total, 11 up, 11 in 2022-09-08T06:36:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:17 smithi059 ceph-mon[97450]: pgmap v17440: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:36:17.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:17 smithi059 ceph-mon[97450]: osdmap e1169: 12 total, 11 up, 11 in 2022-09-08T06:36:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:17.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:17.827896+0000) 2022-09-08T06:36:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:17.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:17.810312+0000) 2022-09-08T06:36:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:18.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:18.828017+0000) 2022-09-08T06:36:19.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:18.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:18.810544+0000) 2022-09-08T06:36:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:19 smithi055 ceph-mon[97033]: pgmap v17442: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2022-09-08T06:36:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:19 smithi059 ceph-mon[97450]: pgmap v17442: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2022-09-08T06:36:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:19.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:19.828235+0000) 2022-09-08T06:36:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:19.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:19.810763+0000) 2022-09-08T06:36:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:20.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:20.828458+0000) 2022-09-08T06:36:21.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:36:20.817+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:36:21.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:20.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:20.810959+0000) 2022-09-08T06:36:21.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:21 smithi055 ceph-mon[97033]: pgmap v17443: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:36:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:21 smithi059 ceph-mon[97450]: pgmap v17443: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:36:22.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:21.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:21.828603+0000) 2022-09-08T06:36:22.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:21.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:21.811158+0000) 2022-09-08T06:36:23.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:22.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:22.828821+0000) 2022-09-08T06:36:23.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:22.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:22.811367+0000) 2022-09-08T06:36:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:23 smithi055 ceph-mon[97033]: pgmap v17444: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:36:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:23 smithi059 ceph-mon[97450]: pgmap v17444: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:36:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:23.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:23.829106+0000) 2022-09-08T06:36:24.162 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:23.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:23.811537+0000) 2022-09-08T06:36:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:24.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:24.829428+0000) 2022-09-08T06:36:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:24.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:24.811721+0000) 2022-09-08T06:36:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:25 smithi055 ceph-mon[97033]: pgmap v17445: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:36:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:25 smithi059 ceph-mon[97450]: pgmap v17445: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:36:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:25.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:25.829751+0000) 2022-09-08T06:36:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:25.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:25.811967+0000) 2022-09-08T06:36:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:26.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:26.829918+0000) 2022-09-08T06:36:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:26.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:26.812156+0000) 2022-09-08T06:36:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:27 smithi055 ceph-mon[97033]: pgmap v17446: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:36:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:27 smithi059 ceph-mon[97450]: pgmap v17446: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:36:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:27.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:27.830130+0000) 2022-09-08T06:36:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:27.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:27.812305+0000) 2022-09-08T06:36:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:28.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:28.830352+0000) 2022-09-08T06:36:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:28.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:28.812478+0000) 2022-09-08T06:36:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:29 smithi055 ceph-mon[97033]: pgmap v17447: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:36:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:29 smithi059 ceph-mon[97450]: pgmap v17447: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:36:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:29.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:29.830688+0000) 2022-09-08T06:36:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:29.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:29.812737+0000) 2022-09-08T06:36:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:30.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:30.830933+0000) 2022-09-08T06:36:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:30.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:30.812921+0000) 2022-09-08T06:36:31.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:31 smithi055 ceph-mon[97033]: pgmap v17448: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:31 smithi059 ceph-mon[97450]: pgmap v17448: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:31.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:31.831187+0000) 2022-09-08T06:36:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:31.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:31.813055+0000) 2022-09-08T06:36:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:32.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:32.831452+0000) 2022-09-08T06:36:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:32.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:32.813191+0000) 2022-09-08T06:36:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:33 smithi055 ceph-mon[97033]: pgmap v17449: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:36:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:33 smithi059 ceph-mon[97450]: pgmap v17449: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:36:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:33.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:33.831650+0000) 2022-09-08T06:36:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:33.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:33.813405+0000) 2022-09-08T06:36:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:34.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:34.831887+0000) 2022-09-08T06:36:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:34.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:34.813614+0000) 2022-09-08T06:36:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:35 smithi055 ceph-mon[97033]: pgmap v17450: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:35 smithi059 ceph-mon[97450]: pgmap v17450: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:35.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:35.832057+0000) 2022-09-08T06:36:36.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:36:35.829+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:36:36.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:35.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:35.813703+0000) 2022-09-08T06:36:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:36.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:36.832246+0000) 2022-09-08T06:36:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:36.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:36.813901+0000) 2022-09-08T06:36:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:37 smithi055 ceph-mon[97033]: pgmap v17451: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:36:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:37 smithi059 ceph-mon[97450]: pgmap v17451: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:36:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:37.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:37.832440+0000) 2022-09-08T06:36:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:37.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:37.814047+0000) 2022-09-08T06:36:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:38.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:38.832601+0000) 2022-09-08T06:36:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:38.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:38.814263+0000) 2022-09-08T06:36:39.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:39 smithi055 ceph-mon[97033]: pgmap v17452: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:39 smithi059 ceph-mon[97450]: pgmap v17452: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:39.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:39.832699+0000) 2022-09-08T06:36:40.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:39.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:39.814407+0000) 2022-09-08T06:36:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:40.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:40.832924+0000) 2022-09-08T06:36:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:40.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:40.814601+0000) 2022-09-08T06:36:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:41 smithi055 ceph-mon[97033]: pgmap v17453: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:41 smithi059 ceph-mon[97450]: pgmap v17453: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:41.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:41.833136+0000) 2022-09-08T06:36:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:41.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:41.814807+0000) 2022-09-08T06:36:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:42.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:42.833325+0000) 2022-09-08T06:36:43.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:42.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:42.815020+0000) 2022-09-08T06:36:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:42 smithi055 ceph-mon[97033]: pgmap v17454: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:36:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:36:43.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:36:43.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:36:43.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:36:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:42 smithi059 ceph-mon[97450]: pgmap v17454: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:36:43.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:36:43.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:36:43.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:36:43.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:36:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:43.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:43.833565+0000) 2022-09-08T06:36:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:43.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:43.815198+0000) 2022-09-08T06:36:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:43 smithi055 ceph-mon[97033]: pgmap v17455: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:36:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:43 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T06:36:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:43 smithi059 ceph-mon[97450]: pgmap v17455: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:36:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:43 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T06:36:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:44.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:44.833840+0000) 2022-09-08T06:36:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:44.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:44.815379+0000) 2022-09-08T06:36:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:45.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:45.834038+0000) 2022-09-08T06:36:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:45.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:45.815566+0000) 2022-09-08T06:36:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:45 smithi055 ceph-mon[97033]: pgmap v17456: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:36:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:45 smithi059 ceph-mon[97450]: pgmap v17456: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:36:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:46.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:46.834170+0000) 2022-09-08T06:36:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:46.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:46.815716+0000) 2022-09-08T06:36:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:47.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:47.834364+0000) 2022-09-08T06:36:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:47.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:47.815904+0000) 2022-09-08T06:36:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:47 smithi055 ceph-mon[97033]: pgmap v17457: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:36:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:47 smithi059 ceph-mon[97450]: pgmap v17457: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:36:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:48.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:48.834597+0000) 2022-09-08T06:36:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:48.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:48.816103+0000) 2022-09-08T06:36:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:49.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:49.834838+0000) 2022-09-08T06:36:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:49.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:49.816276+0000) 2022-09-08T06:36:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:49 smithi055 ceph-mon[97033]: pgmap v17458: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:36:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:49 smithi059 ceph-mon[97450]: pgmap v17458: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:36:51.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:36:50.843+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:36:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:50.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:50.835001+0000) 2022-09-08T06:36:51.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:50.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:50.816469+0000) 2022-09-08T06:36:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:51.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:51.835233+0000) 2022-09-08T06:36:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:51.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:51.816652+0000) 2022-09-08T06:36:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:51 smithi055 ceph-mon[97033]: pgmap v17459: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T06:36:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:51 smithi059 ceph-mon[97450]: pgmap v17459: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T06:36:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:52.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:52.835474+0000) 2022-09-08T06:36:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:52.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:52.816779+0000) 2022-09-08T06:36:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:53.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:53.835732+0000) 2022-09-08T06:36:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:53.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:53.816986+0000) 2022-09-08T06:36:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:53 smithi055 ceph-mon[97033]: pgmap v17460: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:36:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:53 smithi059 ceph-mon[97450]: pgmap v17460: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:36:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:54.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:54.835891+0000) 2022-09-08T06:36:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:54.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:54.817220+0000) 2022-09-08T06:36:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:55.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:55.836117+0000) 2022-09-08T06:36:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:55.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:55.817398+0000) 2022-09-08T06:36:56.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:55 smithi055 ceph-mon[97033]: pgmap v17461: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:55 smithi059 ceph-mon[97450]: pgmap v17461: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:36:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:56.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:56.836351+0000) 2022-09-08T06:36:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:56.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:56.817657+0000) 2022-09-08T06:36:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:57.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:57.836579+0000) 2022-09-08T06:36:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:57.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:57.817846+0000) 2022-09-08T06:36:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:57 smithi055 ceph-mon[97033]: pgmap v17462: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:36:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:57 smithi059 ceph-mon[97450]: pgmap v17462: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:36:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:58.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:58.836785+0000) 2022-09-08T06:36:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:58.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:58.818053+0000) 2022-09-08T06:37:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:36:59.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:59.836975+0000) 2022-09-08T06:37:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:36:59.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:36:59.818262+0000) 2022-09-08T06:37:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:36:59 smithi055 ceph-mon[97033]: pgmap v17463: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:36:59 smithi059 ceph-mon[97450]: pgmap v17463: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:00.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:00.837204+0000) 2022-09-08T06:37:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:00.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:00.818438+0000) 2022-09-08T06:37:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:01.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:01.837393+0000) 2022-09-08T06:37:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:01.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:01.818602+0000) 2022-09-08T06:37:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:01 smithi055 ceph-mon[97033]: pgmap v17464: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:37:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:01 smithi059 ceph-mon[97450]: pgmap v17464: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:37:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:02.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:02.837614+0000) 2022-09-08T06:37:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:02.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:02.818793+0000) 2022-09-08T06:37:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:03.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:03.837863+0000) 2022-09-08T06:37:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:03.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:03.819022+0000) 2022-09-08T06:37:04.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:03 smithi055 ceph-mon[97033]: pgmap v17465: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:03 smithi059 ceph-mon[97450]: pgmap v17465: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:04.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:04.838099+0000) 2022-09-08T06:37:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:04.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:04.819237+0000) 2022-09-08T06:37:06.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:37:05.858+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:37:06.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:05.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:05.838267+0000) 2022-09-08T06:37:06.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:05.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:05.819493+0000) 2022-09-08T06:37:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:05 smithi055 ceph-mon[97033]: pgmap v17466: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:37:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:05 smithi059 ceph-mon[97450]: pgmap v17466: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:37:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:06.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:06.838417+0000) 2022-09-08T06:37:07.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:06.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:06.819707+0000) 2022-09-08T06:37:07.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:37:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:37:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:07.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:07.838609+0000) 2022-09-08T06:37:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:07.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:07.819843+0000) 2022-09-08T06:37:08.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:07 smithi055 ceph-mon[97033]: pgmap v17467: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:37:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:07 smithi059 ceph-mon[97450]: pgmap v17467: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:37:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:08.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:08.838844+0000) 2022-09-08T06:37:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:08.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:08.820066+0000) 2022-09-08T06:37:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:09.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:09.839078+0000) 2022-09-08T06:37:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:09.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:09.820243+0000) 2022-09-08T06:37:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:09 smithi055 ceph-mon[97033]: pgmap v17468: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:37:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:09 smithi055 ceph-mon[97033]: osdmap e1170: 12 total, 11 up, 11 in 2022-09-08T06:37:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:09 smithi059 ceph-mon[97450]: pgmap v17468: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:37:10.310 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:09 smithi059 ceph-mon[97450]: osdmap e1170: 12 total, 11 up, 11 in 2022-09-08T06:37:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:10.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:10.839326+0000) 2022-09-08T06:37:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:10.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:10.820499+0000) 2022-09-08T06:37:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:11.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:11.839585+0000) 2022-09-08T06:37:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:11.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:11.820675+0000) 2022-09-08T06:37:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:11 smithi055 ceph-mon[97033]: pgmap v17470: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:37:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:11 smithi055 ceph-mon[97033]: osdmap e1171: 12 total, 11 up, 11 in 2022-09-08T06:37:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:11 smithi059 ceph-mon[97450]: pgmap v17470: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:37:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:11 smithi059 ceph-mon[97450]: osdmap e1171: 12 total, 11 up, 11 in 2022-09-08T06:37:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:12.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:12.839881+0000) 2022-09-08T06:37:13.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:12.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:12.820939+0000) 2022-09-08T06:37:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:13.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:13.840120+0000) 2022-09-08T06:37:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:13.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:13.821131+0000) 2022-09-08T06:37:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:13 smithi055 ceph-mon[97033]: pgmap v17472: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:37:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:13 smithi059 ceph-mon[97450]: pgmap v17472: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:37:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:14.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:14.840375+0000) 2022-09-08T06:37:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:14.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:14.821282+0000) 2022-09-08T06:37:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:15.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:15.840601+0000) 2022-09-08T06:37:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:15.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:15.821443+0000) 2022-09-08T06:37:16.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:15 smithi055 ceph-mon[97033]: pgmap v17473: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:37:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:15 smithi059 ceph-mon[97450]: pgmap v17473: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:37:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:16.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:16.840775+0000) 2022-09-08T06:37:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:16.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:16.821620+0000) 2022-09-08T06:37:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:17.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:17.841008+0000) 2022-09-08T06:37:18.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:17.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:17.821912+0000) 2022-09-08T06:37:18.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:17 smithi055 ceph-mon[97033]: pgmap v17474: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:37:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:17 smithi059 ceph-mon[97450]: pgmap v17474: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:37:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:18.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:18.841248+0000) 2022-09-08T06:37:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:18.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:18.822058+0000) 2022-09-08T06:37:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:19.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:19.841483+0000) 2022-09-08T06:37:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:19.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:19.822208+0000) 2022-09-08T06:37:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:19 smithi055 ceph-mon[97033]: pgmap v17475: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2022-09-08T06:37:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:19 smithi059 ceph-mon[97450]: pgmap v17475: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2022-09-08T06:37:21.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:37:20.874+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:37:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:20.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:20.841728+0000) 2022-09-08T06:37:21.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:20.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:20.822389+0000) 2022-09-08T06:37:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:21.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:21.841957+0000) 2022-09-08T06:37:22.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:21.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:21.822609+0000) 2022-09-08T06:37:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:21 smithi055 ceph-mon[97033]: pgmap v17476: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:37:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:21 smithi059 ceph-mon[97450]: pgmap v17476: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:37:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:22.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:22.842164+0000) 2022-09-08T06:37:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:22.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:22.822909+0000) 2022-09-08T06:37:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:23.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:23.842371+0000) 2022-09-08T06:37:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:23.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:23.823117+0000) 2022-09-08T06:37:24.229 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:23 smithi055 ceph-mon[97033]: pgmap v17477: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:37:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:23 smithi059 ceph-mon[97450]: pgmap v17477: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:37:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:24.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:24.842533+0000) 2022-09-08T06:37:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:24.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:24.823301+0000) 2022-09-08T06:37:26.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:25.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:25.842692+0000) 2022-09-08T06:37:26.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:25.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:25.823476+0000) 2022-09-08T06:37:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:25 smithi055 ceph-mon[97033]: pgmap v17478: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:37:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:25 smithi059 ceph-mon[97450]: pgmap v17478: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:37:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:26.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:26.842831+0000) 2022-09-08T06:37:27.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:26.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:26.823694+0000) 2022-09-08T06:37:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:27.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:27.843020+0000) 2022-09-08T06:37:28.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:27.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:27.823868+0000) 2022-09-08T06:37:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:27 smithi055 ceph-mon[97033]: pgmap v17479: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:37:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:27 smithi059 ceph-mon[97450]: pgmap v17479: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:37:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:28.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:28.843245+0000) 2022-09-08T06:37:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:28.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:28.824093+0000) 2022-09-08T06:37:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:29.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:29.843462+0000) 2022-09-08T06:37:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:29.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:29.824305+0000) 2022-09-08T06:37:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:29 smithi055 ceph-mon[97033]: pgmap v17480: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:30.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:29 smithi059 ceph-mon[97450]: pgmap v17480: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:30.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:30.843727+0000) 2022-09-08T06:37:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:30.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:30.824572+0000) 2022-09-08T06:37:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:31.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:31.843853+0000) 2022-09-08T06:37:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:31.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:31.824721+0000) 2022-09-08T06:37:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:31 smithi055 ceph-mon[97033]: pgmap v17481: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:37:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:31 smithi059 ceph-mon[97450]: pgmap v17481: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:37:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:32.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:32.844037+0000) 2022-09-08T06:37:33.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:32.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:32.824937+0000) 2022-09-08T06:37:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:33.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:33.844245+0000) 2022-09-08T06:37:34.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:33.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:33.825064+0000) 2022-09-08T06:37:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:33 smithi055 ceph-mon[97033]: pgmap v17482: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:33 smithi059 ceph-mon[97450]: pgmap v17482: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:34.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:34.844473+0000) 2022-09-08T06:37:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:34.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:34.825191+0000) 2022-09-08T06:37:36.161 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:37:35.883+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:37:36.162 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:35.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:35.844748+0000) 2022-09-08T06:37:36.162 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:35.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:35.825398+0000) 2022-09-08T06:37:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:35 smithi055 ceph-mon[97033]: pgmap v17483: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:35 smithi059 ceph-mon[97450]: pgmap v17483: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:36.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:36.844898+0000) 2022-09-08T06:37:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:36.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:36.825609+0000) 2022-09-08T06:37:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:37.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:37.845044+0000) 2022-09-08T06:37:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:37.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:37.825867+0000) 2022-09-08T06:37:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:37 smithi055 ceph-mon[97033]: pgmap v17484: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:37:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:37 smithi059 ceph-mon[97450]: pgmap v17484: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:37:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:38.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:38.845240+0000) 2022-09-08T06:37:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:38.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:38.826078+0000) 2022-09-08T06:37:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:39.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:39.845370+0000) 2022-09-08T06:37:40.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:39.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:39.826314+0000) 2022-09-08T06:37:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:39 smithi055 ceph-mon[97033]: pgmap v17485: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:39 smithi059 ceph-mon[97450]: pgmap v17485: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:40.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:40.845571+0000) 2022-09-08T06:37:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:40.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:40.826567+0000) 2022-09-08T06:37:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:41.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:41.845832+0000) 2022-09-08T06:37:42.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:41.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:41.826743+0000) 2022-09-08T06:37:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:41 smithi055 ceph-mon[97033]: pgmap v17486: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:37:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:41 smithi059 ceph-mon[97450]: pgmap v17486: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:37:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:42.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:42.846045+0000) 2022-09-08T06:37:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:42.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:42.826929+0000) 2022-09-08T06:37:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:37:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:37:43.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:37:43.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:37:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:37:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:37:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:43.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:43.846271+0000) 2022-09-08T06:37:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:43.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:43.827075+0000) 2022-09-08T06:37:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:44 smithi059 ceph-mon[97450]: pgmap v17487: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:37:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:44 smithi055 ceph-mon[97033]: pgmap v17487: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:37:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:44.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:44.846474+0000) 2022-09-08T06:37:45.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:44.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:44.827288+0000) 2022-09-08T06:37:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:45.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:45.846734+0000) 2022-09-08T06:37:46.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:45.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:45.827526+0000) 2022-09-08T06:37:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:46 smithi059 ceph-mon[97450]: pgmap v17488: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:46 smithi055 ceph-mon[97033]: pgmap v17488: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:46.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:46.846950+0000) 2022-09-08T06:37:47.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:46.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:46.827771+0000) 2022-09-08T06:37:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:47.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:47.847189+0000) 2022-09-08T06:37:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:47.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:47.827938+0000) 2022-09-08T06:37:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:48 smithi055 ceph-mon[97033]: pgmap v17489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:37:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:48 smithi059 ceph-mon[97450]: pgmap v17489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:37:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:48.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:48.847400+0000) 2022-09-08T06:37:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:48.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:48.828101+0000) 2022-09-08T06:37:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:49.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:49.847588+0000) 2022-09-08T06:37:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:49.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:49.828243+0000) 2022-09-08T06:37:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:50 smithi055 ceph-mon[97033]: pgmap v17490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:50 smithi059 ceph-mon[97450]: pgmap v17490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:51.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:37:50.895+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:37:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:50.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:50.847710+0000) 2022-09-08T06:37:51.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:50.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:50.828475+0000) 2022-09-08T06:37:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:51.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:51.847912+0000) 2022-09-08T06:37:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:51.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:51.828687+0000) 2022-09-08T06:37:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:52 smithi055 ceph-mon[97033]: pgmap v17491: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:37:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:52 smithi059 ceph-mon[97450]: pgmap v17491: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:37:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:52.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:52.848062+0000) 2022-09-08T06:37:53.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:52.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:52.828890+0000) 2022-09-08T06:37:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:53.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:53.848267+0000) 2022-09-08T06:37:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:53.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:53.829120+0000) 2022-09-08T06:37:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:54 smithi055 ceph-mon[97033]: pgmap v17492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:54 smithi059 ceph-mon[97450]: pgmap v17492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:54.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:54.848419+0000) 2022-09-08T06:37:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:54.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:54.829321+0000) 2022-09-08T06:37:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:55.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:55.848604+0000) 2022-09-08T06:37:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:55.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:55.829459+0000) 2022-09-08T06:37:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:56 smithi055 ceph-mon[97033]: pgmap v17493: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:56 smithi059 ceph-mon[97450]: pgmap v17493: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:37:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:56.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:56.848818+0000) 2022-09-08T06:37:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:56.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:56.829662+0000) 2022-09-08T06:37:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:57.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:57.849046+0000) 2022-09-08T06:37:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:57.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:57.829909+0000) 2022-09-08T06:37:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:37:58 smithi055 ceph-mon[97033]: pgmap v17494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:37:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:37:58 smithi059 ceph-mon[97450]: pgmap v17494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:37:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:58.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:58.849296+0000) 2022-09-08T06:37:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:58.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:58.830101+0000) 2022-09-08T06:38:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:37:59.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:59.849564+0000) 2022-09-08T06:38:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:37:59.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:37:59.830291+0000) 2022-09-08T06:38:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:00 smithi055 ceph-mon[97033]: pgmap v17495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:00 smithi059 ceph-mon[97450]: pgmap v17495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:00.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:00.849743+0000) 2022-09-08T06:38:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:00.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:00.830549+0000) 2022-09-08T06:38:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:01.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:01.849973+0000) 2022-09-08T06:38:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:01.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:01.830711+0000) 2022-09-08T06:38:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:02 smithi055 ceph-mon[97033]: pgmap v17496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:38:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:02 smithi059 ceph-mon[97450]: pgmap v17496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:38:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:02.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:02.830908+0000) 2022-09-08T06:38:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:02.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:02.850191+0000) 2022-09-08T06:38:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:03.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:03.831083+0000) 2022-09-08T06:38:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:03.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:03.850426+0000) 2022-09-08T06:38:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:04 smithi055 ceph-mon[97033]: pgmap v17497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:04 smithi059 ceph-mon[97450]: pgmap v17497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:04.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:04.831248+0000) 2022-09-08T06:38:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:04.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:04.850708+0000) 2022-09-08T06:38:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:05.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:05.850866+0000) 2022-09-08T06:38:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:05.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:05.831438+0000) 2022-09-08T06:38:06.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:38:05.905+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:38:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:06 smithi055 ceph-mon[97033]: pgmap v17498: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:38:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:06 smithi059 ceph-mon[97450]: pgmap v17498: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:38:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:06.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:06.851050+0000) 2022-09-08T06:38:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:06.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:06.831630+0000) 2022-09-08T06:38:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:38:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:38:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:07.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:07.851226+0000) 2022-09-08T06:38:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:07.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:07.831898+0000) 2022-09-08T06:38:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:08 smithi055 ceph-mon[97033]: pgmap v17499: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:38:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:08 smithi059 ceph-mon[97450]: pgmap v17499: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:38:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:08.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:08.851452+0000) 2022-09-08T06:38:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:08.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:08.832069+0000) 2022-09-08T06:38:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:09.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:09.851664+0000) 2022-09-08T06:38:10.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:09.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:09.832274+0000) 2022-09-08T06:38:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:10 smithi055 ceph-mon[97033]: pgmap v17500: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:38:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:10 smithi055 ceph-mon[97033]: osdmap e1172: 12 total, 11 up, 11 in 2022-09-08T06:38:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:10 smithi059 ceph-mon[97450]: pgmap v17500: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:38:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:10 smithi059 ceph-mon[97450]: osdmap e1172: 12 total, 11 up, 11 in 2022-09-08T06:38:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:10.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:10.851853+0000) 2022-09-08T06:38:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:10.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:10.832521+0000) 2022-09-08T06:38:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:11.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:11.852114+0000) 2022-09-08T06:38:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:11.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:11.832742+0000) 2022-09-08T06:38:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:12 smithi055 ceph-mon[97033]: pgmap v17502: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:38:12.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:12 smithi055 ceph-mon[97033]: osdmap e1173: 12 total, 11 up, 11 in 2022-09-08T06:38:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:12 smithi059 ceph-mon[97450]: pgmap v17502: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:38:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:12 smithi059 ceph-mon[97450]: osdmap e1173: 12 total, 11 up, 11 in 2022-09-08T06:38:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:12.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:12.852351+0000) 2022-09-08T06:38:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:12.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:12.832875+0000) 2022-09-08T06:38:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:13.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:13.852606+0000) 2022-09-08T06:38:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:13.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:13.833051+0000) 2022-09-08T06:38:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:14 smithi055 ceph-mon[97033]: pgmap v17504: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:38:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:14 smithi059 ceph-mon[97450]: pgmap v17504: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:38:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:14.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:14.833208+0000) 2022-09-08T06:38:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:14.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:14.852826+0000) 2022-09-08T06:38:16.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:15.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:15.833374+0000) 2022-09-08T06:38:16.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:15.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:15.853073+0000) 2022-09-08T06:38:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:16 smithi055 ceph-mon[97033]: pgmap v17505: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:38:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:16 smithi059 ceph-mon[97450]: pgmap v17505: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:38:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:16.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:16.853299+0000) 2022-09-08T06:38:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:16.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:16.833513+0000) 2022-09-08T06:38:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:17.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:17.853493+0000) 2022-09-08T06:38:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:17.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:17.833680+0000) 2022-09-08T06:38:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:18 smithi055 ceph-mon[97033]: pgmap v17506: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:38:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:18 smithi059 ceph-mon[97450]: pgmap v17506: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:38:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:18.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:18.853721+0000) 2022-09-08T06:38:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:18.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:18.833854+0000) 2022-09-08T06:38:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:19.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:19.853897+0000) 2022-09-08T06:38:20.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:19.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:19.834065+0000) 2022-09-08T06:38:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:20 smithi055 ceph-mon[97033]: pgmap v17507: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 218 B/s wr, 6 op/s 2022-09-08T06:38:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:20 smithi059 ceph-mon[97450]: pgmap v17507: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 218 B/s wr, 6 op/s 2022-09-08T06:38:21.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:38:20.918+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:38:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:20.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:20.834271+0000) 2022-09-08T06:38:21.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:20.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:20.854095+0000) 2022-09-08T06:38:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:21.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:21.834490+0000) 2022-09-08T06:38:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:21.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:21.854300+0000) 2022-09-08T06:38:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:22 smithi055 ceph-mon[97033]: pgmap v17508: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:38:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:22 smithi059 ceph-mon[97450]: pgmap v17508: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:38:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:22.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:22.854459+0000) 2022-09-08T06:38:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:22.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:22.834712+0000) 2022-09-08T06:38:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:23.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:23.854705+0000) 2022-09-08T06:38:24.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:23.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:23.834920+0000) 2022-09-08T06:38:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:24 smithi055 ceph-mon[97033]: pgmap v17509: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:38:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:24 smithi059 ceph-mon[97450]: pgmap v17509: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:38:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:24.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:24.854909+0000) 2022-09-08T06:38:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:24.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:24.835119+0000) 2022-09-08T06:38:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:25.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:25.855149+0000) 2022-09-08T06:38:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:25.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:25.835313+0000) 2022-09-08T06:38:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:26 smithi055 ceph-mon[97033]: pgmap v17510: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:38:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:26 smithi059 ceph-mon[97450]: pgmap v17510: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:38:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:26.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:26.855383+0000) 2022-09-08T06:38:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:26.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:26.835526+0000) 2022-09-08T06:38:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:27.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:27.835703+0000) 2022-09-08T06:38:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:27.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:27.855558+0000) 2022-09-08T06:38:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:28 smithi055 ceph-mon[97033]: pgmap v17511: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:38:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:28 smithi059 ceph-mon[97450]: pgmap v17511: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:38:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:28.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:28.855704+0000) 2022-09-08T06:38:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:28.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:28.835860+0000) 2022-09-08T06:38:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:29.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:29.855928+0000) 2022-09-08T06:38:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:29.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:29.836065+0000) 2022-09-08T06:38:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:30 smithi055 ceph-mon[97033]: pgmap v17512: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:30 smithi059 ceph-mon[97450]: pgmap v17512: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:30.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:30.836228+0000) 2022-09-08T06:38:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:30.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:30.856171+0000) 2022-09-08T06:38:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:31.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:31.856435+0000) 2022-09-08T06:38:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:31.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:31.836404+0000) 2022-09-08T06:38:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:32 smithi055 ceph-mon[97033]: pgmap v17513: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:38:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:32 smithi059 ceph-mon[97450]: pgmap v17513: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:38:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:32.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:32.856723+0000) 2022-09-08T06:38:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:32.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:32.836536+0000) 2022-09-08T06:38:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:33.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:33.856950+0000) 2022-09-08T06:38:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:33.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:33.836711+0000) 2022-09-08T06:38:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:34 smithi055 ceph-mon[97033]: pgmap v17514: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:34 smithi059 ceph-mon[97450]: pgmap v17514: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:34.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:34.857142+0000) 2022-09-08T06:38:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:34.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:34.836878+0000) 2022-09-08T06:38:36.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:38:35.934+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:38:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:35.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:35.857321+0000) 2022-09-08T06:38:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:35.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:35.837014+0000) 2022-09-08T06:38:36.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:36 smithi055 ceph-mon[97033]: pgmap v17515: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:36 smithi059 ceph-mon[97450]: pgmap v17515: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:36.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:36.857538+0000) 2022-09-08T06:38:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:36.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:36.837221+0000) 2022-09-08T06:38:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:37.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:37.857752+0000) 2022-09-08T06:38:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:37.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:37.837406+0000) 2022-09-08T06:38:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:38 smithi055 ceph-mon[97033]: pgmap v17516: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:38:38.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:38 smithi059 ceph-mon[97450]: pgmap v17516: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:38:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:38.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:38.857931+0000) 2022-09-08T06:38:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:38.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:38.837592+0000) 2022-09-08T06:38:40.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:39.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:39.837814+0000) 2022-09-08T06:38:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:39.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:39.858146+0000) 2022-09-08T06:38:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:40 smithi055 ceph-mon[97033]: pgmap v17517: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:40 smithi059 ceph-mon[97450]: pgmap v17517: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:40.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:40.858408+0000) 2022-09-08T06:38:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:40.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:40.838043+0000) 2022-09-08T06:38:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:41.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:41.858651+0000) 2022-09-08T06:38:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:41.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:41.838192+0000) 2022-09-08T06:38:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:42 smithi055 ceph-mon[97033]: pgmap v17518: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:38:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:42 smithi059 ceph-mon[97450]: pgmap v17518: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:38:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:42.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:42.858867+0000) 2022-09-08T06:38:43.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:42.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:42.838371+0000) 2022-09-08T06:38:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:38:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:38:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:38:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:38:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:38:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:38:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:43.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:43.859120+0000) 2022-09-08T06:38:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:43.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:43.838611+0000) 2022-09-08T06:38:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:44 smithi055 ceph-mon[97033]: pgmap v17519: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:38:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:44 smithi059 ceph-mon[97450]: pgmap v17519: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:38:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:44.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:44.859356+0000) 2022-09-08T06:38:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:44.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:44.838787+0000) 2022-09-08T06:38:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:45.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:45.859665+0000) 2022-09-08T06:38:46.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:45.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:45.838919+0000) 2022-09-08T06:38:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:46 smithi055 ceph-mon[97033]: pgmap v17520: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:46 smithi059 ceph-mon[97450]: pgmap v17520: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:46.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:46.839078+0000) 2022-09-08T06:38:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:46.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:46.859892+0000) 2022-09-08T06:38:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:47.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:47.860122+0000) 2022-09-08T06:38:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:47.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:47.839286+0000) 2022-09-08T06:38:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:48 smithi055 ceph-mon[97033]: pgmap v17521: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:38:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:48 smithi059 ceph-mon[97450]: pgmap v17521: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:38:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:48.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:48.860371+0000) 2022-09-08T06:38:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:48.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:48.839552+0000) 2022-09-08T06:38:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:49.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:49.860555+0000) 2022-09-08T06:38:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:49.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:49.839756+0000) 2022-09-08T06:38:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:50 smithi055 ceph-mon[97033]: pgmap v17522: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:50 smithi059 ceph-mon[97450]: pgmap v17522: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:51.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:38:50.946+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:38:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:50.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:50.860768+0000) 2022-09-08T06:38:51.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:50.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:50.839954+0000) 2022-09-08T06:38:52.160 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:51.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:51.860947+0000) 2022-09-08T06:38:52.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:51.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:51.840104+0000) 2022-09-08T06:38:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:52 smithi055 ceph-mon[97033]: pgmap v17523: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:38:52.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:52 smithi059 ceph-mon[97450]: pgmap v17523: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:38:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:52.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:52.861188+0000) 2022-09-08T06:38:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:52.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:52.840280+0000) 2022-09-08T06:38:54.160 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:53.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:53.861429+0000) 2022-09-08T06:38:54.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:53.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:53.840449+0000) 2022-09-08T06:38:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:54 smithi055 ceph-mon[97033]: pgmap v17524: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:54 smithi059 ceph-mon[97450]: pgmap v17524: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:54.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:54.861626+0000) 2022-09-08T06:38:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:54.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:54.840624+0000) 2022-09-08T06:38:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:55.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:55.861778+0000) 2022-09-08T06:38:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:55.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:55.840864+0000) 2022-09-08T06:38:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:56 smithi055 ceph-mon[97033]: pgmap v17525: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:56 smithi059 ceph-mon[97450]: pgmap v17525: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:38:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:56.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:56.861963+0000) 2022-09-08T06:38:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:56.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:56.841048+0000) 2022-09-08T06:38:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:57.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:57.862181+0000) 2022-09-08T06:38:58.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:57.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:57.841260+0000) 2022-09-08T06:38:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:38:58 smithi055 ceph-mon[97033]: pgmap v17526: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:38:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:38:58 smithi059 ceph-mon[97450]: pgmap v17526: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:38:59.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:58.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:58.841472+0000) 2022-09-08T06:38:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:58.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:58.862404+0000) 2022-09-08T06:39:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:38:59.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:59.862654+0000) 2022-09-08T06:39:00.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:38:59.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:38:59.841622+0000) 2022-09-08T06:39:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:00 smithi055 ceph-mon[97033]: pgmap v17527: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:00 smithi059 ceph-mon[97450]: pgmap v17527: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:00.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:00.862854+0000) 2022-09-08T06:39:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:00.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:00.841787+0000) 2022-09-08T06:39:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:01.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:01.863021+0000) 2022-09-08T06:39:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:01.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:01.841939+0000) 2022-09-08T06:39:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:02 smithi055 ceph-mon[97033]: pgmap v17528: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:39:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:02 smithi059 ceph-mon[97450]: pgmap v17528: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:39:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:02.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:02.863239+0000) 2022-09-08T06:39:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:02.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:02.842103+0000) 2022-09-08T06:39:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:03.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:03.863422+0000) 2022-09-08T06:39:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:03.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:03.842246+0000) 2022-09-08T06:39:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:04 smithi055 ceph-mon[97033]: pgmap v17529: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:04 smithi059 ceph-mon[97450]: pgmap v17529: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:04.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:04.863621+0000) 2022-09-08T06:39:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:04.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:04.842393+0000) 2022-09-08T06:39:06.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:39:05.958+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:39:06.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:05.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:05.863799+0000) 2022-09-08T06:39:06.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:05.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:05.842701+0000) 2022-09-08T06:39:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:06 smithi055 ceph-mon[97033]: pgmap v17530: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:39:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:06 smithi059 ceph-mon[97450]: pgmap v17530: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:39:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:06.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:06.863938+0000) 2022-09-08T06:39:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:06.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:06.842888+0000) 2022-09-08T06:39:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:39:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:39:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:07.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:07.864106+0000) 2022-09-08T06:39:08.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:07.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:07.843017+0000) 2022-09-08T06:39:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:08 smithi055 ceph-mon[97033]: pgmap v17531: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:39:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:08 smithi059 ceph-mon[97450]: pgmap v17531: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:39:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:08.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:08.843151+0000) 2022-09-08T06:39:09.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:08.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:08.864333+0000) 2022-09-08T06:39:10.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:09.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:09.843310+0000) 2022-09-08T06:39:10.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:09.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:09.864575+0000) 2022-09-08T06:39:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:10 smithi055 ceph-mon[97033]: pgmap v17532: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:39:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:10 smithi055 ceph-mon[97033]: osdmap e1174: 12 total, 11 up, 11 in 2022-09-08T06:39:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:10 smithi059 ceph-mon[97450]: pgmap v17532: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:39:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:10 smithi059 ceph-mon[97450]: osdmap e1174: 12 total, 11 up, 11 in 2022-09-08T06:39:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:10.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:10.843475+0000) 2022-09-08T06:39:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:10.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:10.864804+0000) 2022-09-08T06:39:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:11.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:11.864925+0000) 2022-09-08T06:39:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:11.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:11.843738+0000) 2022-09-08T06:39:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:12 smithi055 ceph-mon[97033]: pgmap v17534: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:39:12.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:12 smithi055 ceph-mon[97033]: osdmap e1175: 12 total, 11 up, 11 in 2022-09-08T06:39:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:12 smithi059 ceph-mon[97450]: pgmap v17534: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:39:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:12 smithi059 ceph-mon[97450]: osdmap e1175: 12 total, 11 up, 11 in 2022-09-08T06:39:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:12.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:12.865101+0000) 2022-09-08T06:39:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:12.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:12.843913+0000) 2022-09-08T06:39:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:13.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:13.865313+0000) 2022-09-08T06:39:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:13.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:13.844057+0000) 2022-09-08T06:39:14.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:14 smithi059 ceph-mon[97450]: pgmap v17536: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:39:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:14 smithi055 ceph-mon[97033]: pgmap v17536: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:39:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:14.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:14.865562+0000) 2022-09-08T06:39:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:14.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:14.844257+0000) 2022-09-08T06:39:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:15.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:15.865835+0000) 2022-09-08T06:39:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:15.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:15.844455+0000) 2022-09-08T06:39:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:16 smithi059 ceph-mon[97450]: pgmap v17537: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:39:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:16 smithi055 ceph-mon[97033]: pgmap v17537: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:39:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:16.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:16.866058+0000) 2022-09-08T06:39:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:16.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:16.844685+0000) 2022-09-08T06:39:18.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:17.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:17.866341+0000) 2022-09-08T06:39:18.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:17.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:17.844892+0000) 2022-09-08T06:39:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:18 smithi059 ceph-mon[97450]: pgmap v17538: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:39:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:18 smithi055 ceph-mon[97033]: pgmap v17538: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:39:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:18.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:18.866666+0000) 2022-09-08T06:39:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:18.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:18.845057+0000) 2022-09-08T06:39:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:19.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:19.866933+0000) 2022-09-08T06:39:20.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:19.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:19.845264+0000) 2022-09-08T06:39:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:20 smithi059 ceph-mon[97450]: pgmap v17539: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s 2022-09-08T06:39:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:20 smithi055 ceph-mon[97033]: pgmap v17539: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s 2022-09-08T06:39:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:20.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:20.867172+0000) 2022-09-08T06:39:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:20.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:20.845448+0000) 2022-09-08T06:39:21.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:39:20.963+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:39:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:21.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:21.867392+0000) 2022-09-08T06:39:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:21.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:21.845655+0000) 2022-09-08T06:39:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:22 smithi059 ceph-mon[97450]: pgmap v17540: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:39:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:22 smithi055 ceph-mon[97033]: pgmap v17540: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:39:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:22.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:22.867603+0000) 2022-09-08T06:39:23.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:22.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:22.845813+0000) 2022-09-08T06:39:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:23.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:23.846006+0000) 2022-09-08T06:39:24.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:23.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:23.867815+0000) 2022-09-08T06:39:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:24 smithi059 ceph-mon[97450]: pgmap v17541: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:39:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:24 smithi055 ceph-mon[97033]: pgmap v17541: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:39:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:24.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:24.868040+0000) 2022-09-08T06:39:25.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:24.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:24.846165+0000) 2022-09-08T06:39:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:25.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:25.868219+0000) 2022-09-08T06:39:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:25.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:25.846403+0000) 2022-09-08T06:39:26.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:26 smithi059 ceph-mon[97450]: pgmap v17542: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:39:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:26 smithi055 ceph-mon[97033]: pgmap v17542: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:39:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:26.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:26.868404+0000) 2022-09-08T06:39:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:26.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:26.846570+0000) 2022-09-08T06:39:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:27.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:27.868578+0000) 2022-09-08T06:39:28.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:27.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:27.846792+0000) 2022-09-08T06:39:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:28 smithi059 ceph-mon[97450]: pgmap v17543: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:39:28.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:28 smithi055 ceph-mon[97033]: pgmap v17543: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:39:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:28.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:28.868884+0000) 2022-09-08T06:39:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:28.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:28.846987+0000) 2022-09-08T06:39:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:29.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:29.869078+0000) 2022-09-08T06:39:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:29.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:29.847123+0000) 2022-09-08T06:39:30.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:30 smithi059 ceph-mon[97450]: pgmap v17544: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:30 smithi055 ceph-mon[97033]: pgmap v17544: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:31.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:30.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:30.869222+0000) 2022-09-08T06:39:31.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:30.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:30.847332+0000) 2022-09-08T06:39:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:31.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:31.869415+0000) 2022-09-08T06:39:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:31.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:31.847488+0000) 2022-09-08T06:39:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:32 smithi059 ceph-mon[97450]: pgmap v17545: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:39:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:32 smithi055 ceph-mon[97033]: pgmap v17545: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:39:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:32.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:32.869690+0000) 2022-09-08T06:39:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:32.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:32.847689+0000) 2022-09-08T06:39:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:33.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:33.869892+0000) 2022-09-08T06:39:34.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:33.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:33.847854+0000) 2022-09-08T06:39:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:34 smithi059 ceph-mon[97450]: pgmap v17546: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:34 smithi055 ceph-mon[97033]: pgmap v17546: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:34.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:34.870103+0000) 2022-09-08T06:39:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:34.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:34.848008+0000) 2022-09-08T06:39:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:35.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:35.870311+0000) 2022-09-08T06:39:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:35.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:35.848167+0000) 2022-09-08T06:39:36.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:39:35.974+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:39:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:36 smithi059 ceph-mon[97450]: pgmap v17547: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:36.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:36 smithi055 ceph-mon[97033]: pgmap v17547: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:36.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:36.870515+0000) 2022-09-08T06:39:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:36.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:36.848326+0000) 2022-09-08T06:39:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:37.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:37.870755+0000) 2022-09-08T06:39:38.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:37.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:37.848600+0000) 2022-09-08T06:39:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:38 smithi059 ceph-mon[97450]: pgmap v17548: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:39:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:38 smithi055 ceph-mon[97033]: pgmap v17548: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:39:39.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:38.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:38.848810+0000) 2022-09-08T06:39:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:38.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:38.870911+0000) 2022-09-08T06:39:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:39.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:39.849006+0000) 2022-09-08T06:39:40.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:39.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:39.871136+0000) 2022-09-08T06:39:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:40 smithi059 ceph-mon[97450]: pgmap v17549: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:40 smithi055 ceph-mon[97033]: pgmap v17549: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:40.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:40.871372+0000) 2022-09-08T06:39:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:40.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:40.849253+0000) 2022-09-08T06:39:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:41.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:41.871632+0000) 2022-09-08T06:39:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:41.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:41.849471+0000) 2022-09-08T06:39:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:42 smithi059 ceph-mon[97450]: pgmap v17550: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:39:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:42 smithi055 ceph-mon[97033]: pgmap v17550: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:39:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:42.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:42.849712+0000) 2022-09-08T06:39:43.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:42.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:42.871772+0000) 2022-09-08T06:39:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:43.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:43.871975+0000) 2022-09-08T06:39:44.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:43.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:43.849937+0000) 2022-09-08T06:39:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:44 smithi059 ceph-mon[97450]: pgmap v17551: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:39:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:39:44.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:39:44.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:39:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:44 smithi055 ceph-mon[97033]: pgmap v17551: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:39:44.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:39:44.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:39:44.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:39:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:44.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:44.872168+0000) 2022-09-08T06:39:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:44.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:44.850125+0000) 2022-09-08T06:39:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:45.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:45.850278+0000) 2022-09-08T06:39:46.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:45.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:45.872384+0000) 2022-09-08T06:39:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:46 smithi059 ceph-mon[97450]: pgmap v17552: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:46 smithi055 ceph-mon[97033]: pgmap v17552: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:46.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:46.872620+0000) 2022-09-08T06:39:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:46.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:46.850508+0000) 2022-09-08T06:39:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:47.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:47.872881+0000) 2022-09-08T06:39:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:47.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:47.850686+0000) 2022-09-08T06:39:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:48 smithi055 ceph-mon[97033]: pgmap v17553: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:39:48.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:48 smithi059 ceph-mon[97450]: pgmap v17553: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:39:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:48.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:48.873084+0000) 2022-09-08T06:39:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:48.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:48.850826+0000) 2022-09-08T06:39:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:49.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:49.850985+0000) 2022-09-08T06:39:50.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:49.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:49.873257+0000) 2022-09-08T06:39:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:50 smithi055 ceph-mon[97033]: pgmap v17554: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:50 smithi059 ceph-mon[97450]: pgmap v17554: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:50.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:50.873490+0000) 2022-09-08T06:39:51.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:50.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:50.851133+0000) 2022-09-08T06:39:51.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:39:50.975+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:39:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:51.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:51.873712+0000) 2022-09-08T06:39:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:51.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:51.851238+0000) 2022-09-08T06:39:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:52 smithi055 ceph-mon[97033]: pgmap v17555: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:39:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:52 smithi059 ceph-mon[97450]: pgmap v17555: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:39:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:52.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:52.851381+0000) 2022-09-08T06:39:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:52.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:52.873915+0000) 2022-09-08T06:39:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:53.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:53.874096+0000) 2022-09-08T06:39:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:53.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:53.851543+0000) 2022-09-08T06:39:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:54 smithi055 ceph-mon[97033]: pgmap v17556: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:54 smithi059 ceph-mon[97450]: pgmap v17556: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:54.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:54.851699+0000) 2022-09-08T06:39:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:54.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:54.874309+0000) 2022-09-08T06:39:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:55.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:55.874603+0000) 2022-09-08T06:39:56.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:55.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:55.851867+0000) 2022-09-08T06:39:56.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:56 smithi055 ceph-mon[97033]: pgmap v17557: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:56 smithi059 ceph-mon[97450]: pgmap v17557: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:39:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:56.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:56.874783+0000) 2022-09-08T06:39:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:56.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:56.852032+0000) 2022-09-08T06:39:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:57.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:57.875028+0000) 2022-09-08T06:39:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:57.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:57.852242+0000) 2022-09-08T06:39:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:39:58 smithi055 ceph-mon[97033]: pgmap v17558: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:39:58.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:39:58 smithi059 ceph-mon[97450]: pgmap v17558: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:39:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:58.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:58.875277+0000) 2022-09-08T06:39:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:58.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:58.852457+0000) 2022-09-08T06:40:00.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:39:59.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:59.852684+0000) 2022-09-08T06:40:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:39:59.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:39:59.875554+0000) 2022-09-08T06:40:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:00 smithi055 ceph-mon[97033]: pgmap v17559: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:40:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T06:40:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T06:40:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T06:40:00.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:00 smithi059 ceph-mon[97450]: pgmap v17559: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:40:00.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T06:40:00.788 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T06:40:00.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T06:40:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:00.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:00.875697+0000) 2022-09-08T06:40:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:00.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:00.852811+0000) 2022-09-08T06:40:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:01.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:01.875863+0000) 2022-09-08T06:40:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:01.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:01.852968+0000) 2022-09-08T06:40:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:02 smithi055 ceph-mon[97033]: pgmap v17560: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:40:02.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:02 smithi059 ceph-mon[97450]: pgmap v17560: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:40:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:02.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:02.853152+0000) 2022-09-08T06:40:03.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:02.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:02.876028+0000) 2022-09-08T06:40:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:03.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:03.853297+0000) 2022-09-08T06:40:04.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:03.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:03.876256+0000) 2022-09-08T06:40:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:04 smithi055 ceph-mon[97033]: pgmap v17561: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:40:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:04 smithi059 ceph-mon[97450]: pgmap v17561: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:40:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:04.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:04.876478+0000) 2022-09-08T06:40:05.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:04.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:04.853480+0000) 2022-09-08T06:40:06.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:40:05.988+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:40:06.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:05.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:05.876733+0000) 2022-09-08T06:40:06.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:05.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:05.853619+0000) 2022-09-08T06:40:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:06 smithi055 ceph-mon[97033]: pgmap v17562: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:40:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:40:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:06 smithi059 ceph-mon[97450]: pgmap v17562: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:40:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:40:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:06.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:06.876951+0000) 2022-09-08T06:40:07.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:06.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:06.853779+0000) 2022-09-08T06:40:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:40:07.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:40:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:07.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:07.877182+0000) 2022-09-08T06:40:08.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:07.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:07.853996+0000) 2022-09-08T06:40:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:08 smithi055 ceph-mon[97033]: pgmap v17563: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:40:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:08 smithi059 ceph-mon[97450]: pgmap v17563: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:40:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:08.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:08.854220+0000) 2022-09-08T06:40:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:08.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:08.877326+0000) 2022-09-08T06:40:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:09.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:09.877486+0000) 2022-09-08T06:40:10.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:09.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:09.854435+0000) 2022-09-08T06:40:10.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:10 smithi055 ceph-mon[97033]: pgmap v17564: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:40:10.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:10 smithi055 ceph-mon[97033]: osdmap e1176: 12 total, 11 up, 11 in 2022-09-08T06:40:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:10 smithi059 ceph-mon[97450]: pgmap v17564: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:40:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:10 smithi059 ceph-mon[97450]: osdmap e1176: 12 total, 11 up, 11 in 2022-09-08T06:40:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:10.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:10.854656+0000) 2022-09-08T06:40:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:10.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:10.877727+0000) 2022-09-08T06:40:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:11.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:11.854790+0000) 2022-09-08T06:40:12.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:11.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:11.877869+0000) 2022-09-08T06:40:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:12 smithi055 ceph-mon[97033]: pgmap v17566: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:40:12.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:12 smithi055 ceph-mon[97033]: osdmap e1177: 12 total, 11 up, 11 in 2022-09-08T06:40:12.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:12 smithi059 ceph-mon[97450]: pgmap v17566: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:40:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:12 smithi059 ceph-mon[97450]: osdmap e1177: 12 total, 11 up, 11 in 2022-09-08T06:40:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:12.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:12.878099+0000) 2022-09-08T06:40:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:12.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:12.854983+0000) 2022-09-08T06:40:14.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:13.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:13.855216+0000) 2022-09-08T06:40:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:13.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:13.878294+0000) 2022-09-08T06:40:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:14 smithi055 ceph-mon[97033]: pgmap v17568: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T06:40:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:14 smithi059 ceph-mon[97450]: pgmap v17568: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 30 op/s 2022-09-08T06:40:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:14.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:14.878470+0000) 2022-09-08T06:40:15.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:14.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:14.855430+0000) 2022-09-08T06:40:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:15.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:15.878700+0000) 2022-09-08T06:40:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:15.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:15.855567+0000) 2022-09-08T06:40:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:16 smithi055 ceph-mon[97033]: pgmap v17569: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:40:16.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:16 smithi059 ceph-mon[97450]: pgmap v17569: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:40:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:16.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:16.878921+0000) 2022-09-08T06:40:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:16.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:16.855699+0000) 2022-09-08T06:40:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:17.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:17.879119+0000) 2022-09-08T06:40:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:17.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:17.855920+0000) 2022-09-08T06:40:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:18 smithi055 ceph-mon[97033]: pgmap v17570: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:40:18.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:18 smithi059 ceph-mon[97450]: pgmap v17570: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:40:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:18.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:18.879243+0000) 2022-09-08T06:40:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:18.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:18.856113+0000) 2022-09-08T06:40:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:19.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:19.879446+0000) 2022-09-08T06:40:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:19.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:19.856232+0000) 2022-09-08T06:40:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:20 smithi055 ceph-mon[97033]: pgmap v17571: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2022-09-08T06:40:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:20 smithi059 ceph-mon[97450]: pgmap v17571: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2022-09-08T06:40:21.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:40:20.990+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:40:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:20.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:20.879682+0000) 2022-09-08T06:40:21.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:20.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:20.856388+0000) 2022-09-08T06:40:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:21.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:21.879894+0000) 2022-09-08T06:40:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:21.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:21.856539+0000) 2022-09-08T06:40:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:22 smithi055 ceph-mon[97033]: pgmap v17572: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:40:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:22 smithi059 ceph-mon[97450]: pgmap v17572: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:40:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:22.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:22.880122+0000) 2022-09-08T06:40:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:22.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:22.856681+0000) 2022-09-08T06:40:24.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:23.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:23.880330+0000) 2022-09-08T06:40:24.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:23.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:23.856835+0000) 2022-09-08T06:40:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:24 smithi055 ceph-mon[97033]: pgmap v17573: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:40:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:24 smithi059 ceph-mon[97450]: pgmap v17573: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:40:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:24.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:24.880569+0000) 2022-09-08T06:40:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:24.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:24.856995+0000) 2022-09-08T06:40:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:25.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:25.880792+0000) 2022-09-08T06:40:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:25.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:25.857159+0000) 2022-09-08T06:40:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:26 smithi055 ceph-mon[97033]: pgmap v17574: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:40:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:26 smithi059 ceph-mon[97450]: pgmap v17574: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:40:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:26.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:26.880971+0000) 2022-09-08T06:40:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:26.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:26.857318+0000) 2022-09-08T06:40:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:27.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:27.881152+0000) 2022-09-08T06:40:28.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:27.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:27.857458+0000) 2022-09-08T06:40:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:28 smithi055 ceph-mon[97033]: pgmap v17575: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:40:28.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:28 smithi059 ceph-mon[97450]: pgmap v17575: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:40:29.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:28.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:28.881340+0000) 2022-09-08T06:40:29.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:28.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:28.857661+0000) 2022-09-08T06:40:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:29.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:29.881490+0000) 2022-09-08T06:40:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:29.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:29.857795+0000) 2022-09-08T06:40:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:30 smithi055 ceph-mon[97033]: pgmap v17576: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:40:30.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:30 smithi059 ceph-mon[97450]: pgmap v17576: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:40:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:30.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:30.881693+0000) 2022-09-08T06:40:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:30.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:30.858022+0000) 2022-09-08T06:40:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:31.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:31.881915+0000) 2022-09-08T06:40:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:31.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:31.858180+0000) 2022-09-08T06:40:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:32 smithi055 ceph-mon[97033]: pgmap v17577: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:40:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:32 smithi059 ceph-mon[97450]: pgmap v17577: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:40:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:32.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:32.882058+0000) 2022-09-08T06:40:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:32.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:32.858335+0000) 2022-09-08T06:40:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:33.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:33.882290+0000) 2022-09-08T06:40:34.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:33.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:33.858575+0000) 2022-09-08T06:40:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:34 smithi055 ceph-mon[97033]: pgmap v17578: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:40:34.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:34 smithi059 ceph-mon[97450]: pgmap v17578: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:40:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:34.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:34.882503+0000) 2022-09-08T06:40:35.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:34.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:34.858766+0000) 2022-09-08T06:40:36.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:40:35.999+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:40:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:35.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:35.882685+0000) 2022-09-08T06:40:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:35.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:35.858942+0000) 2022-09-08T06:40:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:36 smithi055 ceph-mon[97033]: pgmap v17579: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:40:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:36 smithi059 ceph-mon[97450]: pgmap v17579: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:40:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:36.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:36.882891+0000) 2022-09-08T06:40:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:36.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:36.859129+0000) 2022-09-08T06:40:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:37.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:37.883129+0000) 2022-09-08T06:40:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:37.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:37.859315+0000) 2022-09-08T06:40:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:38 smithi055 ceph-mon[97033]: pgmap v17580: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:40:38.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:38 smithi059 ceph-mon[97450]: pgmap v17580: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:40:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:38.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:38.883265+0000) 2022-09-08T06:40:39.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:38.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:38.859491+0000) 2022-09-08T06:40:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:39.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:39.883389+0000) 2022-09-08T06:40:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:39.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:39.859766+0000) 2022-09-08T06:40:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:40 smithi055 ceph-mon[97033]: pgmap v17581: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:40:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:40 smithi059 ceph-mon[97450]: pgmap v17581: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:40:41.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:40.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:40.883550+0000) 2022-09-08T06:40:41.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:40.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:40.860011+0000) 2022-09-08T06:40:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:41.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:41.883804+0000) 2022-09-08T06:40:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:41.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:41.860161+0000) 2022-09-08T06:40:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:42 smithi055 ceph-mon[97033]: pgmap v17582: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:40:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:42 smithi059 ceph-mon[97450]: pgmap v17582: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:40:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:42.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:42.883966+0000) 2022-09-08T06:40:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:42.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:42.860306+0000) 2022-09-08T06:40:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:43.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:43.884163+0000) 2022-09-08T06:40:44.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:43.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:43.860473+0000) 2022-09-08T06:40:44.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:44 smithi055 ceph-mon[97033]: pgmap v17583: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:40:44.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:40:44.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:40:44.735 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:40:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:44 smithi059 ceph-mon[97450]: pgmap v17583: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:40:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:40:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:40:44.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:40:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:44.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:44.884343+0000) 2022-09-08T06:40:45.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:44.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:44.860649+0000) 2022-09-08T06:40:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:45.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:45.884580+0000) 2022-09-08T06:40:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:45.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:45.860818+0000) 2022-09-08T06:40:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:45 smithi055 ceph-mon[97033]: pgmap v17584: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:40:46.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:40:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:45 smithi059 ceph-mon[97450]: pgmap v17584: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:40:46.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:40:47.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:46.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:46.884850+0000) 2022-09-08T06:40:47.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:46.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:46.860970+0000) 2022-09-08T06:40:48.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:47.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:47.861139+0000) 2022-09-08T06:40:48.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:47.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:47.885002+0000) 2022-09-08T06:40:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:47 smithi055 ceph-mon[97033]: pgmap v17585: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:40:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:47 smithi059 ceph-mon[97450]: pgmap v17585: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:40:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:48.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:48.885182+0000) 2022-09-08T06:40:49.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:48.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:48.861275+0000) 2022-09-08T06:40:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:49.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:49.885396+0000) 2022-09-08T06:40:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:49.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:49.861425+0000) 2022-09-08T06:40:50.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:49 smithi055 ceph-mon[97033]: pgmap v17586: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:40:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:49 smithi059 ceph-mon[97450]: pgmap v17586: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:40:51.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:40:51.003+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:40:51.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:50.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:50.885516+0000) 2022-09-08T06:40:51.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:50.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:50.861710+0000) 2022-09-08T06:40:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:51.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:51.861874+0000) 2022-09-08T06:40:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:51.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:51.885729+0000) 2022-09-08T06:40:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:51 smithi055 ceph-mon[97033]: pgmap v17587: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:40:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:51 smithi059 ceph-mon[97450]: pgmap v17587: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:40:53.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:52.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:52.862068+0000) 2022-09-08T06:40:53.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:52.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:52.885956+0000) 2022-09-08T06:40:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:53.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:53.862212+0000) 2022-09-08T06:40:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:53.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:53.886184+0000) 2022-09-08T06:40:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:53 smithi055 ceph-mon[97033]: pgmap v17588: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:40:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:53 smithi059 ceph-mon[97450]: pgmap v17588: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:40:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:54.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:54.886375+0000) 2022-09-08T06:40:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:54.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:54.862361+0000) 2022-09-08T06:40:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:55.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:55.886575+0000) 2022-09-08T06:40:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:55.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:55.862514+0000) 2022-09-08T06:40:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:55 smithi055 ceph-mon[97033]: pgmap v17589: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:40:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:55 smithi059 ceph-mon[97450]: pgmap v17589: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:40:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:56.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:56.886782+0000) 2022-09-08T06:40:57.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:56.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:56.862694+0000) 2022-09-08T06:40:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:57.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:57.887047+0000) 2022-09-08T06:40:58.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:57.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:57.862884+0000) 2022-09-08T06:40:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:57 smithi055 ceph-mon[97033]: pgmap v17590: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:40:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:57 smithi059 ceph-mon[97450]: pgmap v17590: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:40:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:58.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:58.887339+0000) 2022-09-08T06:40:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:58.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:58.863055+0000) 2022-09-08T06:41:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:40:59.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:59.887554+0000) 2022-09-08T06:41:00.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:40:59.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:40:59.863299+0000) 2022-09-08T06:41:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:40:59 smithi055 ceph-mon[97033]: pgmap v17591: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:40:59 smithi059 ceph-mon[97450]: pgmap v17591: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:00.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:00.887728+0000) 2022-09-08T06:41:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:00.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:00.863540+0000) 2022-09-08T06:41:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:01.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:01.887938+0000) 2022-09-08T06:41:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:01.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:01.863679+0000) 2022-09-08T06:41:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:01 smithi055 ceph-mon[97033]: pgmap v17592: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:41:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:01 smithi059 ceph-mon[97450]: pgmap v17592: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:41:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:02.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:02.888135+0000) 2022-09-08T06:41:03.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:02.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:02.863851+0000) 2022-09-08T06:41:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:03.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:03.888373+0000) 2022-09-08T06:41:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:03.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:03.864039+0000) 2022-09-08T06:41:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:03 smithi055 ceph-mon[97033]: pgmap v17593: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:03 smithi059 ceph-mon[97450]: pgmap v17593: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:04.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:04.888569+0000) 2022-09-08T06:41:05.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:04.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:04.864211+0000) 2022-09-08T06:41:06.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:41:06.014+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:41:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:05.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:05.888736+0000) 2022-09-08T06:41:06.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:05.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:05.864383+0000) 2022-09-08T06:41:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:05 smithi055 ceph-mon[97033]: pgmap v17594: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:05 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:41:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:05 smithi059 ceph-mon[97450]: pgmap v17594: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:41:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:06.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:06.888984+0000) 2022-09-08T06:41:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:06.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:06.864526+0000) 2022-09-08T06:41:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:41:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:41:08.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:07.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:07.864690+0000) 2022-09-08T06:41:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:07.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:07.889179+0000) 2022-09-08T06:41:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:07 smithi055 ceph-mon[97033]: pgmap v17595: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:41:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:07 smithi059 ceph-mon[97450]: pgmap v17595: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:41:09.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:08.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:08.864905+0000) 2022-09-08T06:41:09.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:08.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:08.889396+0000) 2022-09-08T06:41:10.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:09.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:09.865073+0000) 2022-09-08T06:41:10.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:09.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:09.889673+0000) 2022-09-08T06:41:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:09 smithi055 ceph-mon[97033]: pgmap v17596: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:41:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:09 smithi055 ceph-mon[97033]: osdmap e1178: 12 total, 11 up, 11 in 2022-09-08T06:41:10.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:09 smithi059 ceph-mon[97450]: pgmap v17596: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:41:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:09 smithi059 ceph-mon[97450]: osdmap e1178: 12 total, 11 up, 11 in 2022-09-08T06:41:11.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:10.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:10.865245+0000) 2022-09-08T06:41:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:10.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:10.889806+0000) 2022-09-08T06:41:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:11.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:11.890031+0000) 2022-09-08T06:41:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:11.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:11.865383+0000) 2022-09-08T06:41:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:11 smithi055 ceph-mon[97033]: pgmap v17598: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:41:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:11 smithi055 ceph-mon[97033]: osdmap e1179: 12 total, 11 up, 11 in 2022-09-08T06:41:12.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:11 smithi059 ceph-mon[97450]: pgmap v17598: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:41:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:11 smithi059 ceph-mon[97450]: osdmap e1179: 12 total, 11 up, 11 in 2022-09-08T06:41:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:12.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:12.865558+0000) 2022-09-08T06:41:13.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:12.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:12.890267+0000) 2022-09-08T06:41:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:13.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:13.890483+0000) 2022-09-08T06:41:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:13.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:13.865785+0000) 2022-09-08T06:41:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:13 smithi055 ceph-mon[97033]: pgmap v17600: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:41:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:13 smithi059 ceph-mon[97450]: pgmap v17600: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:41:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:14.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:14.890732+0000) 2022-09-08T06:41:15.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:14.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:14.865967+0000) 2022-09-08T06:41:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:15.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:15.890944+0000) 2022-09-08T06:41:16.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:15.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:15.866128+0000) 2022-09-08T06:41:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:15 smithi055 ceph-mon[97033]: pgmap v17601: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:41:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:15 smithi059 ceph-mon[97450]: pgmap v17601: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:41:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:16.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:16.891182+0000) 2022-09-08T06:41:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:16.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:16.866322+0000) 2022-09-08T06:41:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:17.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:17.891417+0000) 2022-09-08T06:41:18.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:17.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:17.866473+0000) 2022-09-08T06:41:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:17 smithi055 ceph-mon[97033]: pgmap v17602: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:41:18.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:17 smithi059 ceph-mon[97450]: pgmap v17602: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:41:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:18.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:18.891695+0000) 2022-09-08T06:41:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:18.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:18.866715+0000) 2022-09-08T06:41:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:19.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:19.891958+0000) 2022-09-08T06:41:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:19.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:19.866998+0000) 2022-09-08T06:41:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:19 smithi055 ceph-mon[97033]: pgmap v17603: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T06:41:20.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:19 smithi059 ceph-mon[97450]: pgmap v17603: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T06:41:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:20.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:20.892197+0000) 2022-09-08T06:41:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:20.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:20.867150+0000) 2022-09-08T06:41:21.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:41:21.020+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:41:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:21.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:21.892465+0000) 2022-09-08T06:41:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:21.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:21.867298+0000) 2022-09-08T06:41:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:21 smithi055 ceph-mon[97033]: pgmap v17604: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:41:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:21 smithi059 ceph-mon[97450]: pgmap v17604: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:41:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:22.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:22.892694+0000) 2022-09-08T06:41:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:22.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:22.867442+0000) 2022-09-08T06:41:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:23.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:23.892891+0000) 2022-09-08T06:41:24.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:23.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:23.867581+0000) 2022-09-08T06:41:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:23 smithi055 ceph-mon[97033]: pgmap v17605: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:41:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:23 smithi059 ceph-mon[97450]: pgmap v17605: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:41:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:24.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:24.867706+0000) 2022-09-08T06:41:25.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:24.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:24.893215+0000) 2022-09-08T06:41:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:25.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:25.893500+0000) 2022-09-08T06:41:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:25.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:25.867912+0000) 2022-09-08T06:41:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:25 smithi059 ceph-mon[97450]: pgmap v17606: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:41:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:25 smithi055 ceph-mon[97033]: pgmap v17606: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:41:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:26.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:26.893699+0000) 2022-09-08T06:41:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:26.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:26.868055+0000) 2022-09-08T06:41:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:27.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:27.893938+0000) 2022-09-08T06:41:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:27.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:27.868222+0000) 2022-09-08T06:41:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:27 smithi059 ceph-mon[97450]: pgmap v17607: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:41:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:27 smithi055 ceph-mon[97033]: pgmap v17607: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:41:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:28.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:28.894106+0000) 2022-09-08T06:41:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:28.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:28.868359+0000) 2022-09-08T06:41:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:29.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:29.894256+0000) 2022-09-08T06:41:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:29.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:29.868512+0000) 2022-09-08T06:41:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:29 smithi059 ceph-mon[97450]: pgmap v17608: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:29 smithi055 ceph-mon[97033]: pgmap v17608: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:30.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:30.894456+0000) 2022-09-08T06:41:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:30.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:30.868689+0000) 2022-09-08T06:41:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:31.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:31.894697+0000) 2022-09-08T06:41:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:31.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:31.868814+0000) 2022-09-08T06:41:32.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:31 smithi059 ceph-mon[97450]: pgmap v17609: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:41:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:31 smithi055 ceph-mon[97033]: pgmap v17609: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:41:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:32.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:32.894943+0000) 2022-09-08T06:41:33.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:32.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:32.869025+0000) 2022-09-08T06:41:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:33.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:33.895094+0000) 2022-09-08T06:41:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:33.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:33.869197+0000) 2022-09-08T06:41:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:33 smithi059 ceph-mon[97450]: pgmap v17610: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:33 smithi055 ceph-mon[97033]: pgmap v17610: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:34.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:34.895312+0000) 2022-09-08T06:41:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:34.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:34.869358+0000) 2022-09-08T06:41:36.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:41:36.032+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:41:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:35.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:35.895483+0000) 2022-09-08T06:41:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:35.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:35.869606+0000) 2022-09-08T06:41:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:35 smithi059 ceph-mon[97450]: pgmap v17611: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:36 smithi055 ceph-mon[97033]: pgmap v17611: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:36.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:36.895731+0000) 2022-09-08T06:41:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:36.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:36.869773+0000) 2022-09-08T06:41:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:37.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:37.895961+0000) 2022-09-08T06:41:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:37.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:37.869949+0000) 2022-09-08T06:41:38.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:38 smithi059 ceph-mon[97450]: pgmap v17612: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:41:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:38 smithi055 ceph-mon[97033]: pgmap v17612: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:41:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:38.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:38.896184+0000) 2022-09-08T06:41:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:38.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:38.870123+0000) 2022-09-08T06:41:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:39.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:39.896390+0000) 2022-09-08T06:41:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:39.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:39.870290+0000) 2022-09-08T06:41:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:40 smithi059 ceph-mon[97450]: pgmap v17613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:40 smithi055 ceph-mon[97033]: pgmap v17613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:40.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:40.870484+0000) 2022-09-08T06:41:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:40.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:40.896566+0000) 2022-09-08T06:41:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:41.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:41.896773+0000) 2022-09-08T06:41:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:41.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:41.870629+0000) 2022-09-08T06:41:42.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:42 smithi059 ceph-mon[97450]: pgmap v17614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:41:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:42 smithi055 ceph-mon[97033]: pgmap v17614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:41:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:42.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:42.897009+0000) 2022-09-08T06:41:43.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:42.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:42.870896+0000) 2022-09-08T06:41:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:43.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:43.897160+0000) 2022-09-08T06:41:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:43.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:43.871039+0000) 2022-09-08T06:41:44.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:44 smithi059 ceph-mon[97450]: pgmap v17615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:44.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:44 smithi055 ceph-mon[97033]: pgmap v17615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:44.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:44.897352+0000) 2022-09-08T06:41:45.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:44.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:44.871207+0000) 2022-09-08T06:41:45.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:41:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:41:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:41:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:41:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:41:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:41:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:45.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:45.897575+0000) 2022-09-08T06:41:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:45.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:45.871377+0000) 2022-09-08T06:41:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:46 smithi059 ceph-mon[97450]: pgmap v17616: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:41:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:46 smithi055 ceph-mon[97033]: pgmap v17616: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:46.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:41:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:46.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:46.897788+0000) 2022-09-08T06:41:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:46.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:46.871528+0000) 2022-09-08T06:41:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:47.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:47.898007+0000) 2022-09-08T06:41:48.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:47.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:47.871751+0000) 2022-09-08T06:41:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:48 smithi059 ceph-mon[97450]: pgmap v17617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:41:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:48 smithi055 ceph-mon[97033]: pgmap v17617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:41:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:48.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:48.898190+0000) 2022-09-08T06:41:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:48.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:48.871892+0000) 2022-09-08T06:41:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:49.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:49.898319+0000) 2022-09-08T06:41:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:49.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:49.872089+0000) 2022-09-08T06:41:50.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:50 smithi059 ceph-mon[97450]: pgmap v17618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:50 smithi055 ceph-mon[97033]: pgmap v17618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:50.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:50.898538+0000) 2022-09-08T06:41:51.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:41:51.049+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:41:51.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:50.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:50.872296+0000) 2022-09-08T06:41:52.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:51.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:51.898711+0000) 2022-09-08T06:41:52.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:51.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:51.872506+0000) 2022-09-08T06:41:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:52 smithi055 ceph-mon[97033]: pgmap v17619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:41:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:52 smithi059 ceph-mon[97450]: pgmap v17619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:41:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:52.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:52.898935+0000) 2022-09-08T06:41:53.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:52.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:52.872756+0000) 2022-09-08T06:41:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:53.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:53.899090+0000) 2022-09-08T06:41:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:53.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:53.872901+0000) 2022-09-08T06:41:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:54 smithi055 ceph-mon[97033]: pgmap v17620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:54 smithi059 ceph-mon[97450]: pgmap v17620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:54.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:54.899218+0000) 2022-09-08T06:41:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:54.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:54.873080+0000) 2022-09-08T06:41:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:55.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:55.899351+0000) 2022-09-08T06:41:56.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:55.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:55.873314+0000) 2022-09-08T06:41:56.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:56 smithi055 ceph-mon[97033]: pgmap v17621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:56.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:56 smithi059 ceph-mon[97450]: pgmap v17621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:41:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:56.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:56.899543+0000) 2022-09-08T06:41:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:56.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:56.873532+0000) 2022-09-08T06:41:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:57.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:57.899769+0000) 2022-09-08T06:41:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:57.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:57.873714+0000) 2022-09-08T06:41:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:41:58 smithi055 ceph-mon[97033]: pgmap v17622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:41:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:41:58 smithi059 ceph-mon[97450]: pgmap v17622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:41:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:58.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:58.899998+0000) 2022-09-08T06:41:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:58.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:58.873912+0000) 2022-09-08T06:42:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:41:59.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:59.900187+0000) 2022-09-08T06:42:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:41:59.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:41:59.874128+0000) 2022-09-08T06:42:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:00 smithi055 ceph-mon[97033]: pgmap v17623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:00 smithi059 ceph-mon[97450]: pgmap v17623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:00.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:00.900383+0000) 2022-09-08T06:42:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:00.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:00.874298+0000) 2022-09-08T06:42:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:01.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:01.900617+0000) 2022-09-08T06:42:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:01.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:01.874421+0000) 2022-09-08T06:42:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:02 smithi055 ceph-mon[97033]: pgmap v17624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:42:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:02 smithi059 ceph-mon[97450]: pgmap v17624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:42:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:02.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:02.874583+0000) 2022-09-08T06:42:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:02.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:02.900836+0000) 2022-09-08T06:42:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:03.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:03.901010+0000) 2022-09-08T06:42:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:03.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:03.874755+0000) 2022-09-08T06:42:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:04 smithi055 ceph-mon[97033]: pgmap v17625: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:04 smithi059 ceph-mon[97450]: pgmap v17625: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:04.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:04.901237+0000) 2022-09-08T06:42:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:04.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:04.874938+0000) 2022-09-08T06:42:06.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:42:06.064+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:42:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:05.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:05.901450+0000) 2022-09-08T06:42:06.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:05.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:05.875153+0000) 2022-09-08T06:42:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:06 smithi055 ceph-mon[97033]: pgmap v17626: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:42:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:06 smithi059 ceph-mon[97450]: pgmap v17626: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:42:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:06.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:06.901590+0000) 2022-09-08T06:42:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:06.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:06.875321+0000) 2022-09-08T06:42:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:42:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:42:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:07.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:07.901831+0000) 2022-09-08T06:42:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:07.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:07.875496+0000) 2022-09-08T06:42:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:08 smithi055 ceph-mon[97033]: pgmap v17627: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:42:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:08 smithi059 ceph-mon[97450]: pgmap v17627: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:42:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:08.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:08.902034+0000) 2022-09-08T06:42:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:08.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:08.875768+0000) 2022-09-08T06:42:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:09.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:09.902234+0000) 2022-09-08T06:42:10.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:09.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:09.876010+0000) 2022-09-08T06:42:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:10 smithi055 ceph-mon[97033]: pgmap v17628: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:42:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:10 smithi055 ceph-mon[97033]: osdmap e1180: 12 total, 11 up, 11 in 2022-09-08T06:42:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:10 smithi059 ceph-mon[97450]: pgmap v17628: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:42:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:10 smithi059 ceph-mon[97450]: osdmap e1180: 12 total, 11 up, 11 in 2022-09-08T06:42:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:10.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:10.902418+0000) 2022-09-08T06:42:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:10.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:10.876171+0000) 2022-09-08T06:42:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:11.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:11.902674+0000) 2022-09-08T06:42:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:11.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:11.876338+0000) 2022-09-08T06:42:12.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:12 smithi055 ceph-mon[97033]: pgmap v17630: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:42:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:12 smithi055 ceph-mon[97033]: osdmap e1181: 12 total, 11 up, 11 in 2022-09-08T06:42:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:12 smithi059 ceph-mon[97450]: pgmap v17630: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:42:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:12 smithi059 ceph-mon[97450]: osdmap e1181: 12 total, 11 up, 11 in 2022-09-08T06:42:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:12.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:12.902906+0000) 2022-09-08T06:42:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:12.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:12.876584+0000) 2022-09-08T06:42:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:13.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:13.903088+0000) 2022-09-08T06:42:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:13.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:13.876829+0000) 2022-09-08T06:42:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:14 smithi055 ceph-mon[97033]: pgmap v17632: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:42:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:14 smithi059 ceph-mon[97450]: pgmap v17632: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:42:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:14.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:14.903331+0000) 2022-09-08T06:42:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:14.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:14.876981+0000) 2022-09-08T06:42:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:15.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:15.903502+0000) 2022-09-08T06:42:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:15.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:15.877219+0000) 2022-09-08T06:42:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:16 smithi055 ceph-mon[97033]: pgmap v17633: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:42:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:16 smithi059 ceph-mon[97450]: pgmap v17633: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:42:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:16.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:16.903736+0000) 2022-09-08T06:42:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:16.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:16.877457+0000) 2022-09-08T06:42:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:17.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:17.903987+0000) 2022-09-08T06:42:18.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:17.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:17.877651+0000) 2022-09-08T06:42:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:18 smithi055 ceph-mon[97033]: pgmap v17634: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:42:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:18 smithi059 ceph-mon[97450]: pgmap v17634: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:42:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:18.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:18.904141+0000) 2022-09-08T06:42:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:18.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:18.877890+0000) 2022-09-08T06:42:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:19.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:19.878060+0000) 2022-09-08T06:42:20.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:19.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:19.904292+0000) 2022-09-08T06:42:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:20 smithi055 ceph-mon[97033]: pgmap v17635: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 216 B/s wr, 6 op/s 2022-09-08T06:42:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:20 smithi059 ceph-mon[97450]: pgmap v17635: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 216 B/s wr, 6 op/s 2022-09-08T06:42:21.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:42:21.072+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:42:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:20.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:20.878240+0000) 2022-09-08T06:42:21.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:20.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:20.904528+0000) 2022-09-08T06:42:22.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:21.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:21.878470+0000) 2022-09-08T06:42:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:21.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:21.904785+0000) 2022-09-08T06:42:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:22 smithi055 ceph-mon[97033]: pgmap v17636: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:42:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:22 smithi059 ceph-mon[97450]: pgmap v17636: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:42:23.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:22.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:22.878667+0000) 2022-09-08T06:42:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:22.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:22.905029+0000) 2022-09-08T06:42:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:23.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:23.878801+0000) 2022-09-08T06:42:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:23.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:23.905190+0000) 2022-09-08T06:42:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:24 smithi055 ceph-mon[97033]: pgmap v17637: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:42:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:24 smithi059 ceph-mon[97450]: pgmap v17637: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:42:25.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:24.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:24.878937+0000) 2022-09-08T06:42:25.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:24.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:24.905366+0000) 2022-09-08T06:42:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:25.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:25.879152+0000) 2022-09-08T06:42:26.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:25.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:25.905574+0000) 2022-09-08T06:42:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:26 smithi055 ceph-mon[97033]: pgmap v17638: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:42:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:26 smithi059 ceph-mon[97450]: pgmap v17638: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:42:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:26.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:26.879360+0000) 2022-09-08T06:42:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:26.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:26.905734+0000) 2022-09-08T06:42:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:27.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:27.905994+0000) 2022-09-08T06:42:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:27.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:27.879569+0000) 2022-09-08T06:42:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:28 smithi055 ceph-mon[97033]: pgmap v17639: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:42:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:28 smithi059 ceph-mon[97450]: pgmap v17639: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:42:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:28.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:28.906164+0000) 2022-09-08T06:42:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:28.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:28.879731+0000) 2022-09-08T06:42:30.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:29.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:29.906409+0000) 2022-09-08T06:42:30.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:29.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:29.879879+0000) 2022-09-08T06:42:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:30 smithi055 ceph-mon[97033]: pgmap v17640: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:30 smithi059 ceph-mon[97450]: pgmap v17640: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:30.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:30.906567+0000) 2022-09-08T06:42:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:30.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:30.880057+0000) 2022-09-08T06:42:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:31.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:31.906731+0000) 2022-09-08T06:42:32.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:31.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:31.880227+0000) 2022-09-08T06:42:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:32 smithi055 ceph-mon[97033]: pgmap v17641: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:42:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:32 smithi059 ceph-mon[97450]: pgmap v17641: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:42:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:32.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:32.906959+0000) 2022-09-08T06:42:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:32.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:32.880417+0000) 2022-09-08T06:42:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:33.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:33.907102+0000) 2022-09-08T06:42:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:33.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:33.880675+0000) 2022-09-08T06:42:34.488 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:34 smithi055 ceph-mon[97033]: pgmap v17642: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:34 smithi059 ceph-mon[97450]: pgmap v17642: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:34.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:34.907329+0000) 2022-09-08T06:42:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:34.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:34.880897+0000) 2022-09-08T06:42:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:35.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:35.907518+0000) 2022-09-08T06:42:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:35.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:35.881104+0000) 2022-09-08T06:42:36.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:42:36.077+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:42:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:36 smithi055 ceph-mon[97033]: pgmap v17643: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:36 smithi059 ceph-mon[97450]: pgmap v17643: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:36.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:36.907757+0000) 2022-09-08T06:42:37.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:36.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:36.881322+0000) 2022-09-08T06:42:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:37.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:37.907975+0000) 2022-09-08T06:42:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:37.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:37.881518+0000) 2022-09-08T06:42:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:38 smithi055 ceph-mon[97033]: pgmap v17644: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:42:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:38 smithi059 ceph-mon[97450]: pgmap v17644: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:42:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:38.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:38.908148+0000) 2022-09-08T06:42:39.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:38.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:38.881735+0000) 2022-09-08T06:42:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:39.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:39.881891+0000) 2022-09-08T06:42:40.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:39.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:39.908314+0000) 2022-09-08T06:42:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:40 smithi055 ceph-mon[97033]: pgmap v17645: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:40 smithi059 ceph-mon[97450]: pgmap v17645: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:40.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:40.882064+0000) 2022-09-08T06:42:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:40.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:40.908523+0000) 2022-09-08T06:42:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:41.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:41.882251+0000) 2022-09-08T06:42:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:41.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:41.908714+0000) 2022-09-08T06:42:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:42 smithi055 ceph-mon[97033]: pgmap v17646: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:42:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:42 smithi059 ceph-mon[97450]: pgmap v17646: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:42:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:42.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:42.908859+0000) 2022-09-08T06:42:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:42.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:42.882455+0000) 2022-09-08T06:42:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:43.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:43.909012+0000) 2022-09-08T06:42:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:43.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:43.882695+0000) 2022-09-08T06:42:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:44 smithi055 ceph-mon[97033]: pgmap v17647: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:44 smithi059 ceph-mon[97450]: pgmap v17647: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:44.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:44.909172+0000) 2022-09-08T06:42:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:44.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:44.882849+0000) 2022-09-08T06:42:46.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:45.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:45.883008+0000) 2022-09-08T06:42:46.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:45.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:45.909325+0000) 2022-09-08T06:42:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:46 smithi055 ceph-mon[97033]: pgmap v17648: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:46.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:42:46.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:42:46.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:42:46.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:42:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:46 smithi059 ceph-mon[97450]: pgmap v17648: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:42:46.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:42:46.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:42:46.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:42:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:46.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:46.883137+0000) 2022-09-08T06:42:47.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:46.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:46.909522+0000) 2022-09-08T06:42:48.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:47.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:47.883279+0000) 2022-09-08T06:42:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:47.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:47.909771+0000) 2022-09-08T06:42:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:48 smithi055 ceph-mon[97033]: pgmap v17649: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:42:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:48 smithi059 ceph-mon[97450]: pgmap v17649: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:42:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:48.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:48.909988+0000) 2022-09-08T06:42:49.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:48.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:48.883409+0000) 2022-09-08T06:42:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:49.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:49.910191+0000) 2022-09-08T06:42:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:49.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:49.883537+0000) 2022-09-08T06:42:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:50 smithi055 ceph-mon[97033]: pgmap v17650: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:50 smithi059 ceph-mon[97450]: pgmap v17650: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:51.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:42:51.080+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:42:51.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:50.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:50.910374+0000) 2022-09-08T06:42:51.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:50.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:50.883699+0000) 2022-09-08T06:42:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:51.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:51.910541+0000) 2022-09-08T06:42:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:51.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:51.883909+0000) 2022-09-08T06:42:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:52 smithi055 ceph-mon[97033]: pgmap v17651: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:42:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:52 smithi059 ceph-mon[97450]: pgmap v17651: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:42:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:52.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:52.910711+0000) 2022-09-08T06:42:53.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:52.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:52.884094+0000) 2022-09-08T06:42:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:53.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:53.910831+0000) 2022-09-08T06:42:54.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:53.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:53.884229+0000) 2022-09-08T06:42:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:54 smithi055 ceph-mon[97033]: pgmap v17652: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:54 smithi059 ceph-mon[97450]: pgmap v17652: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:54.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:54.911079+0000) 2022-09-08T06:42:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:54.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:54.884447+0000) 2022-09-08T06:42:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:55.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:55.911305+0000) 2022-09-08T06:42:56.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:55.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:55.884659+0000) 2022-09-08T06:42:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:56 smithi055 ceph-mon[97033]: pgmap v17653: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:56 smithi059 ceph-mon[97450]: pgmap v17653: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:42:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:56.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:56.911465+0000) 2022-09-08T06:42:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:56.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:56.884835+0000) 2022-09-08T06:42:58.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:57.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:57.911629+0000) 2022-09-08T06:42:58.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:57.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:57.885060+0000) 2022-09-08T06:42:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:42:58 smithi055 ceph-mon[97033]: pgmap v17654: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:42:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:42:58 smithi059 ceph-mon[97450]: pgmap v17654: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:42:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:58.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:58.911776+0000) 2022-09-08T06:42:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:58.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:58.885331+0000) 2022-09-08T06:43:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:42:59.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:59.911980+0000) 2022-09-08T06:43:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:42:59.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:42:59.885500+0000) 2022-09-08T06:43:00.479 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:00 smithi055 ceph-mon[97033]: pgmap v17655: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:00 smithi059 ceph-mon[97450]: pgmap v17655: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:00.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:00.912172+0000) 2022-09-08T06:43:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:00.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:00.885725+0000) 2022-09-08T06:43:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:01.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:01.912381+0000) 2022-09-08T06:43:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:01.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:01.885864+0000) 2022-09-08T06:43:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:02 smithi055 ceph-mon[97033]: pgmap v17656: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:43:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:02 smithi059 ceph-mon[97450]: pgmap v17656: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:43:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:02.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:02.912550+0000) 2022-09-08T06:43:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:02.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:02.886039+0000) 2022-09-08T06:43:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:03.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:03.912761+0000) 2022-09-08T06:43:04.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:03.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:03.886258+0000) 2022-09-08T06:43:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:04 smithi055 ceph-mon[97033]: pgmap v17657: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:04.541 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:04 smithi059 ceph-mon[97450]: pgmap v17657: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:04.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:04.912913+0000) 2022-09-08T06:43:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:04.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:04.886505+0000) 2022-09-08T06:43:06.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:43:06.095+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:43:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:05.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:05.913137+0000) 2022-09-08T06:43:06.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:05.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:05.886754+0000) 2022-09-08T06:43:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:06 smithi055 ceph-mon[97033]: pgmap v17658: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:43:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:06 smithi059 ceph-mon[97450]: pgmap v17658: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:43:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:06.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:06.913311+0000) 2022-09-08T06:43:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:06.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:06.886971+0000) 2022-09-08T06:43:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:43:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:43:07.915 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:07.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:07.887197+0000) 2022-09-08T06:43:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:07.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:07.913484+0000) 2022-09-08T06:43:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:08 smithi055 ceph-mon[97033]: pgmap v17659: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:43:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:08 smithi059 ceph-mon[97450]: pgmap v17659: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:43:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:08.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:08.913717+0000) 2022-09-08T06:43:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:08.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:08.887413+0000) 2022-09-08T06:43:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:09.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:09.913950+0000) 2022-09-08T06:43:10.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:09.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:09.887579+0000) 2022-09-08T06:43:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:10 smithi055 ceph-mon[97033]: pgmap v17660: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:43:10.523 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:10 smithi055 ceph-mon[97033]: osdmap e1182: 12 total, 11 up, 11 in 2022-09-08T06:43:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:10 smithi059 ceph-mon[97450]: pgmap v17660: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:43:10.979 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:10 smithi059 ceph-mon[97450]: osdmap e1182: 12 total, 11 up, 11 in 2022-09-08T06:43:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:10.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:10.914209+0000) 2022-09-08T06:43:11.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:10.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:10.887760+0000) 2022-09-08T06:43:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:11.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:11.914397+0000) 2022-09-08T06:43:12.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:11.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:11.887905+0000) 2022-09-08T06:43:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:12 smithi055 ceph-mon[97033]: pgmap v17662: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:43:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:12 smithi059 ceph-mon[97450]: pgmap v17662: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:43:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:12.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:12.914589+0000) 2022-09-08T06:43:13.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:12.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:12.888136+0000) 2022-09-08T06:43:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:13.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:13.914786+0000) 2022-09-08T06:43:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:13.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:13.888341+0000) 2022-09-08T06:43:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:14 smithi055 ceph-mon[97033]: pgmap v17663: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:43:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:14 smithi059 ceph-mon[97450]: pgmap v17663: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:43:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:14.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:14.888473+0000) 2022-09-08T06:43:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:14.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:14.914915+0000) 2022-09-08T06:43:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:15.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:15.915031+0000) 2022-09-08T06:43:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:15.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:15.888607+0000) 2022-09-08T06:43:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:16 smithi055 ceph-mon[97033]: pgmap v17664: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:43:16.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:16 smithi059 ceph-mon[97450]: pgmap v17664: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:43:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:16.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:16.888768+0000) 2022-09-08T06:43:17.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:16.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:16.915259+0000) 2022-09-08T06:43:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:17.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:17.915438+0000) 2022-09-08T06:43:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:17.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:17.888993+0000) 2022-09-08T06:43:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:18 smithi055 ceph-mon[97033]: pgmap v17665: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:43:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:18 smithi059 ceph-mon[97450]: pgmap v17665: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:43:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:18.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:18.915609+0000) 2022-09-08T06:43:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:18.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:18.889245+0000) 2022-09-08T06:43:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:19.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:19.889534+0000) 2022-09-08T06:43:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:19.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:19.915753+0000) 2022-09-08T06:43:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:20 smithi055 ceph-mon[97033]: pgmap v17666: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:43:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:20 smithi059 ceph-mon[97450]: pgmap v17666: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:43:21.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:43:21.106+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:43:21.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:20.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:20.889703+0000) 2022-09-08T06:43:21.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:20.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:20.915961+0000) 2022-09-08T06:43:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:21.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:21.916144+0000) 2022-09-08T06:43:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:21.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:21.889918+0000) 2022-09-08T06:43:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:22 smithi055 ceph-mon[97033]: pgmap v17667: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 180 B/s wr, 8 op/s 2022-09-08T06:43:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:22 smithi055 ceph-mon[97033]: osdmap e1183: 12 total, 11 up, 11 in 2022-09-08T06:43:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:22 smithi059 ceph-mon[97450]: pgmap v17667: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 180 B/s wr, 8 op/s 2022-09-08T06:43:22.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:22 smithi059 ceph-mon[97450]: osdmap e1183: 12 total, 11 up, 11 in 2022-09-08T06:43:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:22.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:22.916316+0000) 2022-09-08T06:43:23.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:22.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:22.890082+0000) 2022-09-08T06:43:24.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:23.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:23.890224+0000) 2022-09-08T06:43:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:23.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:23.916488+0000) 2022-09-08T06:43:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:24 smithi055 ceph-mon[97033]: pgmap v17669: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:43:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:24 smithi059 ceph-mon[97450]: pgmap v17669: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:43:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:24.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:24.916703+0000) 2022-09-08T06:43:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:24.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:24.890386+0000) 2022-09-08T06:43:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:25.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:25.916856+0000) 2022-09-08T06:43:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:25.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:25.890598+0000) 2022-09-08T06:43:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:26 smithi055 ceph-mon[97033]: pgmap v17670: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:43:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:26 smithi059 ceph-mon[97450]: pgmap v17670: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:43:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:26.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:26.917087+0000) 2022-09-08T06:43:27.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:26.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:26.890817+0000) 2022-09-08T06:43:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:27.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:27.917335+0000) 2022-09-08T06:43:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:27.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:27.890967+0000) 2022-09-08T06:43:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:28 smithi055 ceph-mon[97033]: pgmap v17671: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:43:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:28 smithi059 ceph-mon[97450]: pgmap v17671: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:43:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:28.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:28.917564+0000) 2022-09-08T06:43:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:28.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:28.891109+0000) 2022-09-08T06:43:29.923 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:29.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:29.917786+0000) 2022-09-08T06:43:29.924 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:29.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:29.891318+0000) 2022-09-08T06:43:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:30 smithi055 ceph-mon[97033]: pgmap v17672: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:43:30.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:30 smithi059 ceph-mon[97450]: pgmap v17672: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:43:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:30.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:30.917974+0000) 2022-09-08T06:43:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:30.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:30.891458+0000) 2022-09-08T06:43:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:31.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:31.918243+0000) 2022-09-08T06:43:32.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:31.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:31.891708+0000) 2022-09-08T06:43:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:32 smithi055 ceph-mon[97033]: pgmap v17673: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:43:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:32 smithi059 ceph-mon[97450]: pgmap v17673: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:43:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:32.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:32.918438+0000) 2022-09-08T06:43:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:32.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:32.891901+0000) 2022-09-08T06:43:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:33.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:33.918712+0000) 2022-09-08T06:43:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:33.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:33.892088+0000) 2022-09-08T06:43:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:34 smithi055 ceph-mon[97033]: pgmap v17674: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:43:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:34 smithi059 ceph-mon[97450]: pgmap v17674: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:43:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:34.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:34.918968+0000) 2022-09-08T06:43:35.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:34.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:34.892285+0000) 2022-09-08T06:43:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:35.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:35.919197+0000) 2022-09-08T06:43:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:35.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:35.892425+0000) 2022-09-08T06:43:36.156 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:43:36.121+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:43:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:36 smithi055 ceph-mon[97033]: pgmap v17675: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:36 smithi059 ceph-mon[97450]: pgmap v17675: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:37.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:36.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:36.892697+0000) 2022-09-08T06:43:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:36.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:36.919428+0000) 2022-09-08T06:43:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:37.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:37.919601+0000) 2022-09-08T06:43:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:37.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:37.892865+0000) 2022-09-08T06:43:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:38 smithi055 ceph-mon[97033]: pgmap v17676: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:43:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:38 smithi059 ceph-mon[97450]: pgmap v17676: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:43:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:38.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:38.919719+0000) 2022-09-08T06:43:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:38.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:38.893046+0000) 2022-09-08T06:43:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:39.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:39.919851+0000) 2022-09-08T06:43:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:39.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:39.893183+0000) 2022-09-08T06:43:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:40 smithi059 ceph-mon[97450]: pgmap v17677: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:40 smithi055 ceph-mon[97033]: pgmap v17677: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:40.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:40.920037+0000) 2022-09-08T06:43:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:40.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:40.893337+0000) 2022-09-08T06:43:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:41.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:41.920273+0000) 2022-09-08T06:43:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:41.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:41.893477+0000) 2022-09-08T06:43:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:42 smithi059 ceph-mon[97450]: pgmap v17678: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:43:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:42 smithi055 ceph-mon[97033]: pgmap v17678: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:43:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:42.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:42.920507+0000) 2022-09-08T06:43:43.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:42.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:42.893623+0000) 2022-09-08T06:43:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:43.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:43.920742+0000) 2022-09-08T06:43:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:43.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:43.893785+0000) 2022-09-08T06:43:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:44 smithi059 ceph-mon[97450]: pgmap v17679: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:44 smithi055 ceph-mon[97033]: pgmap v17679: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:44.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:44.920924+0000) 2022-09-08T06:43:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:44.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:44.893939+0000) 2022-09-08T06:43:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:45.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:45.921144+0000) 2022-09-08T06:43:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:45.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:45.894186+0000) 2022-09-08T06:43:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:46 smithi059 ceph-mon[97450]: pgmap v17680: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:43:46.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:43:46.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:43:46.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:46 smithi055 ceph-mon[97033]: pgmap v17680: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:46.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:43:46.717 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:43:46.718 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:43:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:46.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:46.921295+0000) 2022-09-08T06:43:47.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:46.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:46.894348+0000) 2022-09-08T06:43:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:47 smithi055 ceph-mon[97033]: pgmap v17681: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:43:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:43:47.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:43:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:47 smithi059 ceph-mon[97450]: pgmap v17681: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:43:48.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:43:48.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:43:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:47.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:47.921507+0000) 2022-09-08T06:43:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:47.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:47.894554+0000) 2022-09-08T06:43:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:48.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:48.921783+0000) 2022-09-08T06:43:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:48.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:48.894799+0000) 2022-09-08T06:43:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:49.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:49.921969+0000) 2022-09-08T06:43:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:49.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:49.895046+0000) 2022-09-08T06:43:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:49 smithi055 ceph-mon[97033]: pgmap v17682: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:43:50.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:43:50.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:43:50.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:43:50.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:43:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:49 smithi059 ceph-mon[97450]: pgmap v17682: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:43:50.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:43:50.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:43:50.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:43:50.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:43:50.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:43:51.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:43:51.125+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:43:51.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:50.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:50.922218+0000) 2022-09-08T06:43:51.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:50.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:50.895209+0000) 2022-09-08T06:43:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:50 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 168.1M 2022-09-08T06:43:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:50 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 176277590: error parsing value: Value '176277590' is below minimum 939524096 2022-09-08T06:43:51.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:50 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T06:43:51.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:50 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T06:43:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:50 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 168.1M 2022-09-08T06:43:51.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:50 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 176277590: error parsing value: Value '176277590' is below minimum 939524096 2022-09-08T06:43:51.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:50 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T06:43:51.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:50 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T06:43:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:51.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:51.922414+0000) 2022-09-08T06:43:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:51.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:51.895437+0000) 2022-09-08T06:43:52.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:52 smithi059 ceph-mon[97450]: pgmap v17683: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:43:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:52 smithi055 ceph-mon[97033]: pgmap v17683: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:43:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:52.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:52.922650+0000) 2022-09-08T06:43:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:52.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:52.895665+0000) 2022-09-08T06:43:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:53.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:53.922827+0000) 2022-09-08T06:43:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:53.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:53.895827+0000) 2022-09-08T06:43:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:54 smithi059 ceph-mon[97450]: pgmap v17684: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:54 smithi055 ceph-mon[97033]: pgmap v17684: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:54.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:54.923004+0000) 2022-09-08T06:43:55.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:54.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:54.895963+0000) 2022-09-08T06:43:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:55.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:55.923180+0000) 2022-09-08T06:43:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:55.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:55.896091+0000) 2022-09-08T06:43:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:56 smithi059 ceph-mon[97450]: pgmap v17685: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:56 smithi055 ceph-mon[97033]: pgmap v17685: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:43:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:56.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:56.923360+0000) 2022-09-08T06:43:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:56.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:56.896231+0000) 2022-09-08T06:43:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:57.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:57.923631+0000) 2022-09-08T06:43:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:57.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:57.896489+0000) 2022-09-08T06:43:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:43:58 smithi059 ceph-mon[97450]: pgmap v17686: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:43:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:43:58 smithi055 ceph-mon[97033]: pgmap v17686: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:43:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:58.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:58.923842+0000) 2022-09-08T06:43:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:58.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:58.896667+0000) 2022-09-08T06:44:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:43:59.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:59.923959+0000) 2022-09-08T06:44:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:43:59.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:43:59.896802+0000) 2022-09-08T06:44:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:00 smithi059 ceph-mon[97450]: pgmap v17687: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:00 smithi055 ceph-mon[97033]: pgmap v17687: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:00.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:00.924147+0000) 2022-09-08T06:44:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:00.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:00.896938+0000) 2022-09-08T06:44:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:01.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:01.897090+0000) 2022-09-08T06:44:02.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:01.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:01.924388+0000) 2022-09-08T06:44:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:02 smithi059 ceph-mon[97450]: pgmap v17688: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:44:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:02 smithi055 ceph-mon[97033]: pgmap v17688: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:44:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:02.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:02.924617+0000) 2022-09-08T06:44:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:02.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:02.897267+0000) 2022-09-08T06:44:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:03.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:03.924867+0000) 2022-09-08T06:44:04.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:03.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:03.897508+0000) 2022-09-08T06:44:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:04 smithi059 ceph-mon[97450]: pgmap v17689: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:04 smithi055 ceph-mon[97033]: pgmap v17689: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:04.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:04.925088+0000) 2022-09-08T06:44:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:04.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:04.897697+0000) 2022-09-08T06:44:06.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:44:06.134+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:44:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:05.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:05.897864+0000) 2022-09-08T06:44:06.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:05.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:05.925312+0000) 2022-09-08T06:44:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:06 smithi059 ceph-mon[97450]: pgmap v17690: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:44:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:06 smithi055 ceph-mon[97033]: pgmap v17690: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:44:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:06.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:06.925582+0000) 2022-09-08T06:44:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:06.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:06.898096+0000) 2022-09-08T06:44:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:44:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:44:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:07.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:07.925799+0000) 2022-09-08T06:44:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:07.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:07.898247+0000) 2022-09-08T06:44:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:08 smithi059 ceph-mon[97450]: pgmap v17691: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:44:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:08 smithi055 ceph-mon[97033]: pgmap v17691: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:44:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:08.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:08.925971+0000) 2022-09-08T06:44:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:08.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:08.898384+0000) 2022-09-08T06:44:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:09.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:09.926184+0000) 2022-09-08T06:44:10.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:09.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:09.898711+0000) 2022-09-08T06:44:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:10 smithi055 ceph-mon[97033]: pgmap v17692: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:44:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:10 smithi055 ceph-mon[97033]: osdmap e1184: 12 total, 11 up, 11 in 2022-09-08T06:44:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:10 smithi059 ceph-mon[97450]: pgmap v17692: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:44:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:10 smithi059 ceph-mon[97450]: osdmap e1184: 12 total, 11 up, 11 in 2022-09-08T06:44:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:10.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:10.926342+0000) 2022-09-08T06:44:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:10.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:10.898879+0000) 2022-09-08T06:44:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:11.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:11.899042+0000) 2022-09-08T06:44:12.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:11.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:11.926562+0000) 2022-09-08T06:44:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:12 smithi055 ceph-mon[97033]: pgmap v17694: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:44:12.503 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:12 smithi055 ceph-mon[97033]: osdmap e1185: 12 total, 11 up, 11 in 2022-09-08T06:44:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:12 smithi059 ceph-mon[97450]: pgmap v17694: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:44:12.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:12 smithi059 ceph-mon[97450]: osdmap e1185: 12 total, 11 up, 11 in 2022-09-08T06:44:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:12.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:12.926753+0000) 2022-09-08T06:44:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:12.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:12.899191+0000) 2022-09-08T06:44:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:13.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:13.926951+0000) 2022-09-08T06:44:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:13.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:13.899412+0000) 2022-09-08T06:44:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:14 smithi055 ceph-mon[97033]: pgmap v17696: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:44:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:14 smithi059 ceph-mon[97450]: pgmap v17696: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:44:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:14.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:14.899599+0000) 2022-09-08T06:44:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:14.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:14.927112+0000) 2022-09-08T06:44:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:15.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:15.927264+0000) 2022-09-08T06:44:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:15.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:15.899811+0000) 2022-09-08T06:44:16.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:16 smithi055 ceph-mon[97033]: pgmap v17697: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:44:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:16 smithi059 ceph-mon[97450]: pgmap v17697: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:44:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:16.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:16.927491+0000) 2022-09-08T06:44:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:16.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:16.900037+0000) 2022-09-08T06:44:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:17.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:17.927710+0000) 2022-09-08T06:44:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:17.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:17.900180+0000) 2022-09-08T06:44:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:18 smithi055 ceph-mon[97033]: pgmap v17698: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:44:18.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:18 smithi059 ceph-mon[97450]: pgmap v17698: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:44:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:18.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:18.927883+0000) 2022-09-08T06:44:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:18.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:18.900333+0000) 2022-09-08T06:44:20.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:19.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:19.900484+0000) 2022-09-08T06:44:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:19.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:19.928131+0000) 2022-09-08T06:44:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:20 smithi055 ceph-mon[97033]: pgmap v17699: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2022-09-08T06:44:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:20 smithi059 ceph-mon[97450]: pgmap v17699: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2022-09-08T06:44:21.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:44:21.142+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:44:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:20.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:20.928340+0000) 2022-09-08T06:44:21.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:20.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:20.900629+0000) 2022-09-08T06:44:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:21.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:21.928587+0000) 2022-09-08T06:44:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:21.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:21.900785+0000) 2022-09-08T06:44:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:22 smithi055 ceph-mon[97033]: pgmap v17700: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:44:22.538 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:22 smithi059 ceph-mon[97450]: pgmap v17700: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:44:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:22.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:22.928748+0000) 2022-09-08T06:44:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:22.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:22.900928+0000) 2022-09-08T06:44:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:23.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:23.928976+0000) 2022-09-08T06:44:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:23.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:23.901068+0000) 2022-09-08T06:44:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:24 smithi055 ceph-mon[97033]: pgmap v17701: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:44:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:24 smithi059 ceph-mon[97450]: pgmap v17701: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:44:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:24.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:24.929181+0000) 2022-09-08T06:44:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:24.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:24.901245+0000) 2022-09-08T06:44:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:25.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:25.929343+0000) 2022-09-08T06:44:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:25.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:25.901395+0000) 2022-09-08T06:44:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:26 smithi055 ceph-mon[97033]: pgmap v17702: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:44:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:26 smithi059 ceph-mon[97450]: pgmap v17702: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:44:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:26.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:26.901612+0000) 2022-09-08T06:44:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:26.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:26.929569+0000) 2022-09-08T06:44:28.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:27.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:27.929786+0000) 2022-09-08T06:44:28.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:27.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:27.901739+0000) 2022-09-08T06:44:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:28 smithi055 ceph-mon[97033]: pgmap v17703: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:44:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:28 smithi059 ceph-mon[97450]: pgmap v17703: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:44:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:28.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:28.929933+0000) 2022-09-08T06:44:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:28.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:28.901868+0000) 2022-09-08T06:44:30.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:29.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:29.930122+0000) 2022-09-08T06:44:30.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:29.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:29.902012+0000) 2022-09-08T06:44:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:30 smithi055 ceph-mon[97033]: pgmap v17704: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:30 smithi059 ceph-mon[97450]: pgmap v17704: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:30.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:30.930270+0000) 2022-09-08T06:44:31.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:30.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:30.902155+0000) 2022-09-08T06:44:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:31.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:31.902334+0000) 2022-09-08T06:44:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:31.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:31.930479+0000) 2022-09-08T06:44:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:32 smithi055 ceph-mon[97033]: pgmap v17705: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:44:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:32 smithi059 ceph-mon[97450]: pgmap v17705: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:44:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:32.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:32.930746+0000) 2022-09-08T06:44:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:32.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:32.902501+0000) 2022-09-08T06:44:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:33.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:33.930988+0000) 2022-09-08T06:44:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:33.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:33.902693+0000) 2022-09-08T06:44:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:34 smithi055 ceph-mon[97033]: pgmap v17706: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:34 smithi059 ceph-mon[97450]: pgmap v17706: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:34.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:34.931252+0000) 2022-09-08T06:44:35.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:34.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:34.902838+0000) 2022-09-08T06:44:36.158 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:35.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:35.931476+0000) 2022-09-08T06:44:36.161 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:35.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:35.903042+0000) 2022-09-08T06:44:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:36 smithi055 ceph-mon[97033]: pgmap v17707: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:36 smithi059 ceph-mon[97450]: pgmap v17707: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:36.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:44:36.155+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:44:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:36.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:36.931691+0000) 2022-09-08T06:44:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:36.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:36.903192+0000) 2022-09-08T06:44:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:37.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:37.931892+0000) 2022-09-08T06:44:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:37.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:37.903312+0000) 2022-09-08T06:44:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:38 smithi055 ceph-mon[97033]: pgmap v17708: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:44:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:38 smithi059 ceph-mon[97450]: pgmap v17708: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:44:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:38.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:38.932106+0000) 2022-09-08T06:44:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:38.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:38.903521+0000) 2022-09-08T06:44:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:39.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:39.932317+0000) 2022-09-08T06:44:40.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:39.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:39.903686+0000) 2022-09-08T06:44:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:40 smithi055 ceph-mon[97033]: pgmap v17709: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:40 smithi059 ceph-mon[97450]: pgmap v17709: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:40.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:40.932573+0000) 2022-09-08T06:44:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:40.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:40.903828+0000) 2022-09-08T06:44:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:41.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:41.904061+0000) 2022-09-08T06:44:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:41.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:41.932795+0000) 2022-09-08T06:44:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:42 smithi055 ceph-mon[97033]: pgmap v17710: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:44:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:42 smithi059 ceph-mon[97450]: pgmap v17710: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:44:43.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:42.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:42.932956+0000) 2022-09-08T06:44:43.438 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:42.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:42.904254+0000) 2022-09-08T06:44:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:43.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:43.933183+0000) 2022-09-08T06:44:44.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:43.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:43.904476+0000) 2022-09-08T06:44:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:44 smithi055 ceph-mon[97033]: pgmap v17711: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:44 smithi059 ceph-mon[97450]: pgmap v17711: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:44.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:44.933359+0000) 2022-09-08T06:44:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:44.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:44.904693+0000) 2022-09-08T06:44:46.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:45.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:45.933531+0000) 2022-09-08T06:44:46.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:45.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:45.904881+0000) 2022-09-08T06:44:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:46 smithi055 ceph-mon[97033]: pgmap v17712: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:46 smithi059 ceph-mon[97450]: pgmap v17712: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:46.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:46.933763+0000) 2022-09-08T06:44:47.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:46.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:46.905054+0000) 2022-09-08T06:44:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:47.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:47.934002+0000) 2022-09-08T06:44:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:47.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:47.905205+0000) 2022-09-08T06:44:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:48 smithi055 ceph-mon[97033]: pgmap v17713: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:44:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:48 smithi059 ceph-mon[97450]: pgmap v17713: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:44:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:48.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:48.905341+0000) 2022-09-08T06:44:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:48.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:48.934246+0000) 2022-09-08T06:44:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:49.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:49.934400+0000) 2022-09-08T06:44:50.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:49.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:49.905478+0000) 2022-09-08T06:44:50.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:50 smithi055 ceph-mon[97033]: pgmap v17714: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:50.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:44:50.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:44:50.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:44:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:50 smithi059 ceph-mon[97450]: pgmap v17714: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:44:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:44:50.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:44:51.173 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:50.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:50.934663+0000) 2022-09-08T06:44:51.174 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:50.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:50.905648+0000) 2022-09-08T06:44:51.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:44:51.170+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:44:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:44:51.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:51 smithi055 ceph-mon[97033]: pgmap v17715: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:44:52.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:44:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:51 smithi059 ceph-mon[97450]: pgmap v17715: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:44:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:51.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:51.934933+0000) 2022-09-08T06:44:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:51.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:51.905841+0000) 2022-09-08T06:44:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:52.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:52.935119+0000) 2022-09-08T06:44:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:52.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:52.906052+0000) 2022-09-08T06:44:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:53 smithi059 ceph-mon[97450]: pgmap v17716: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:53.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:53.935350+0000) 2022-09-08T06:44:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:53.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:53.906282+0000) 2022-09-08T06:44:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:53 smithi055 ceph-mon[97033]: pgmap v17716: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:54.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:54.935667+0000) 2022-09-08T06:44:55.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:54.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:54.906418+0000) 2022-09-08T06:44:56.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:55 smithi059 ceph-mon[97450]: pgmap v17717: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:55.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:55.906568+0000) 2022-09-08T06:44:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:55.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:55.935944+0000) 2022-09-08T06:44:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:55 smithi055 ceph-mon[97033]: pgmap v17717: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:44:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:56.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:56.936233+0000) 2022-09-08T06:44:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:56.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:56.906732+0000) 2022-09-08T06:44:58.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:57 smithi059 ceph-mon[97450]: pgmap v17718: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:44:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:57.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:57.936476+0000) 2022-09-08T06:44:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:57.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:57.906934+0000) 2022-09-08T06:44:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:57 smithi055 ceph-mon[97033]: pgmap v17718: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:44:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:58.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:58.936745+0000) 2022-09-08T06:44:59.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:58.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:58.907109+0000) 2022-09-08T06:45:00.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:44:59 smithi059 ceph-mon[97450]: pgmap v17719: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:44:59.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:59.936994+0000) 2022-09-08T06:45:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:44:59.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:44:59.907301+0000) 2022-09-08T06:45:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:44:59 smithi055 ceph-mon[97033]: pgmap v17719: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:00.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:00.937171+0000) 2022-09-08T06:45:01.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:00.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:00.907460+0000) 2022-09-08T06:45:02.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:01 smithi059 ceph-mon[97450]: pgmap v17720: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:45:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:01.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:01.937364+0000) 2022-09-08T06:45:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:01.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:01.907589+0000) 2022-09-08T06:45:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:01 smithi055 ceph-mon[97033]: pgmap v17720: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:45:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:02.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:02.937566+0000) 2022-09-08T06:45:03.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:02.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:02.907806+0000) 2022-09-08T06:45:04.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:04 smithi059 ceph-mon[97450]: pgmap v17721: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:03.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:03.937745+0000) 2022-09-08T06:45:04.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:03.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:03.907958+0000) 2022-09-08T06:45:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:04 smithi055 ceph-mon[97033]: pgmap v17721: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:04.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:04.937971+0000) 2022-09-08T06:45:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:04.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:04.908170+0000) 2022-09-08T06:45:06.189 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:05.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:05.938141+0000) 2022-09-08T06:45:06.190 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:05.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:05.908318+0000) 2022-09-08T06:45:06.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:06 smithi059 ceph-mon[97450]: pgmap v17722: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:45:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:06 smithi055 ceph-mon[97033]: pgmap v17722: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:45:06.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:45:06.186+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:45:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:45:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:06.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:06.938278+0000) 2022-09-08T06:45:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:06.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:06.908459+0000) 2022-09-08T06:45:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:45:08.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:08 smithi059 ceph-mon[97450]: pgmap v17723: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:45:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:07.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:07.938498+0000) 2022-09-08T06:45:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:07.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:07.908695+0000) 2022-09-08T06:45:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:08 smithi055 ceph-mon[97033]: pgmap v17723: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:45:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:08.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:08.938760+0000) 2022-09-08T06:45:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:08.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:08.908902+0000) 2022-09-08T06:45:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:10 smithi059 ceph-mon[97450]: pgmap v17724: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:45:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:10 smithi059 ceph-mon[97450]: osdmap e1186: 12 total, 11 up, 11 in 2022-09-08T06:45:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:09.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:09.938987+0000) 2022-09-08T06:45:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:09.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:09.909053+0000) 2022-09-08T06:45:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:10 smithi055 ceph-mon[97033]: pgmap v17724: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:45:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:10 smithi055 ceph-mon[97033]: osdmap e1186: 12 total, 11 up, 11 in 2022-09-08T06:45:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:10.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:10.939172+0000) 2022-09-08T06:45:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:10.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:10.909202+0000) 2022-09-08T06:45:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:12 smithi059 ceph-mon[97450]: pgmap v17726: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:45:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:12 smithi059 ceph-mon[97450]: osdmap e1187: 12 total, 11 up, 11 in 2022-09-08T06:45:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:11.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:11.939405+0000) 2022-09-08T06:45:12.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:11.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:11.909347+0000) 2022-09-08T06:45:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:12 smithi055 ceph-mon[97033]: pgmap v17726: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:45:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:12 smithi055 ceph-mon[97033]: osdmap e1187: 12 total, 11 up, 11 in 2022-09-08T06:45:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:12.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:12.939686+0000) 2022-09-08T06:45:13.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:12.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:12.909490+0000) 2022-09-08T06:45:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:14 smithi059 ceph-mon[97450]: pgmap v17728: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:45:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:13.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:13.939858+0000) 2022-09-08T06:45:14.432 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:13.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:13.909645+0000) 2022-09-08T06:45:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:14 smithi055 ceph-mon[97033]: pgmap v17728: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:45:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:14.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:14.940137+0000) 2022-09-08T06:45:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:14.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:14.909874+0000) 2022-09-08T06:45:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:15.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:15.940333+0000) 2022-09-08T06:45:16.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:15.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:15.910026+0000) 2022-09-08T06:45:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:16 smithi055 ceph-mon[97033]: pgmap v17729: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:45:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:16 smithi059 ceph-mon[97450]: pgmap v17729: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:45:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:16.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:16.940574+0000) 2022-09-08T06:45:17.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:16.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:16.910187+0000) 2022-09-08T06:45:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:17.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:17.940763+0000) 2022-09-08T06:45:18.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:17.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:17.910312+0000) 2022-09-08T06:45:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:18 smithi055 ceph-mon[97033]: pgmap v17730: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:45:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:18 smithi059 ceph-mon[97450]: pgmap v17730: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:45:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:18.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:18.941002+0000) 2022-09-08T06:45:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:18.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:18.910444+0000) 2022-09-08T06:45:20.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:19.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:19.941218+0000) 2022-09-08T06:45:20.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:19.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:19.910595+0000) 2022-09-08T06:45:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:20 smithi055 ceph-mon[97033]: pgmap v17731: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T06:45:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:20 smithi059 ceph-mon[97450]: pgmap v17731: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T06:45:21.196 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:20.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:20.941454+0000) 2022-09-08T06:45:21.198 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:20.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:20.910842+0000) 2022-09-08T06:45:21.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:45:21.192+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:45:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:21.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:21.941721+0000) 2022-09-08T06:45:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:21.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:21.911014+0000) 2022-09-08T06:45:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:22 smithi055 ceph-mon[97033]: pgmap v17732: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:45:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:22 smithi059 ceph-mon[97450]: pgmap v17732: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:45:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:22.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:22.941927+0000) 2022-09-08T06:45:23.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:22.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:22.911245+0000) 2022-09-08T06:45:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:23.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:23.942152+0000) 2022-09-08T06:45:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:23.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:23.911487+0000) 2022-09-08T06:45:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:24 smithi055 ceph-mon[97033]: pgmap v17733: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:45:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:24 smithi059 ceph-mon[97450]: pgmap v17733: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T06:45:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:24.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:24.942352+0000) 2022-09-08T06:45:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:24.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:24.911710+0000) 2022-09-08T06:45:26.412 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:25.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:25.942594+0000) 2022-09-08T06:45:26.413 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:25.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:25.911849+0000) 2022-09-08T06:45:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:26 smithi055 ceph-mon[97033]: pgmap v17734: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:45:26.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:26 smithi059 ceph-mon[97450]: pgmap v17734: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T06:45:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:26.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:26.942865+0000) 2022-09-08T06:45:27.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:26.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:26.912051+0000) 2022-09-08T06:45:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:27.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:27.943049+0000) 2022-09-08T06:45:28.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:27.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:27.912208+0000) 2022-09-08T06:45:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:28 smithi055 ceph-mon[97033]: pgmap v17735: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:45:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:28 smithi059 ceph-mon[97450]: pgmap v17735: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:45:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:28.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:28.943229+0000) 2022-09-08T06:45:29.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:28.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:28.912382+0000) 2022-09-08T06:45:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:29.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:29.943426+0000) 2022-09-08T06:45:30.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:29.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:29.912540+0000) 2022-09-08T06:45:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:30 smithi055 ceph-mon[97033]: pgmap v17736: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:30 smithi059 ceph-mon[97450]: pgmap v17736: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:30.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:30.943698+0000) 2022-09-08T06:45:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:30.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:30.912706+0000) 2022-09-08T06:45:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:31.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:31.943926+0000) 2022-09-08T06:45:32.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:31.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:31.912862+0000) 2022-09-08T06:45:32.487 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:32 smithi055 ceph-mon[97033]: pgmap v17737: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:45:32.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:32 smithi059 ceph-mon[97450]: pgmap v17737: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:45:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:32.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:32.944151+0000) 2022-09-08T06:45:33.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:32.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:32.912988+0000) 2022-09-08T06:45:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:33.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:33.944375+0000) 2022-09-08T06:45:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:33.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:33.913198+0000) 2022-09-08T06:45:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:34 smithi055 ceph-mon[97033]: pgmap v17738: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:34 smithi059 ceph-mon[97450]: pgmap v17738: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:34.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:34.944530+0000) 2022-09-08T06:45:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:34.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:34.913353+0000) 2022-09-08T06:45:36.212 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:35.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:35.944758+0000) 2022-09-08T06:45:36.213 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:35.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:35.913551+0000) 2022-09-08T06:45:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:36 smithi055 ceph-mon[97033]: pgmap v17739: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:36 smithi059 ceph-mon[97450]: pgmap v17739: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:36.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:45:36.208+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:45:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:36.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:36.944982+0000) 2022-09-08T06:45:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:36.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:36.913711+0000) 2022-09-08T06:45:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:37.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:37.913936+0000) 2022-09-08T06:45:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:37.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:37.945182+0000) 2022-09-08T06:45:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:38 smithi055 ceph-mon[97033]: pgmap v17740: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:45:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:38 smithi059 ceph-mon[97450]: pgmap v17740: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:45:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:38.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:38.914159+0000) 2022-09-08T06:45:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:38.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:38.945415+0000) 2022-09-08T06:45:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:39.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:39.914334+0000) 2022-09-08T06:45:40.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:39.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:39.945627+0000) 2022-09-08T06:45:40.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:40 smithi055 ceph-mon[97033]: pgmap v17741: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:40 smithi059 ceph-mon[97450]: pgmap v17741: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:40.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:40.914536+0000) 2022-09-08T06:45:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:40.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:40.945847+0000) 2022-09-08T06:45:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:41.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:41.946068+0000) 2022-09-08T06:45:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:41.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:41.914718+0000) 2022-09-08T06:45:42.489 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:42 smithi055 ceph-mon[97033]: pgmap v17742: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:45:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:42 smithi059 ceph-mon[97450]: pgmap v17742: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:45:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:42.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:42.946277+0000) 2022-09-08T06:45:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:42.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:42.914875+0000) 2022-09-08T06:45:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:43.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:43.946515+0000) 2022-09-08T06:45:44.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:43.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:43.915046+0000) 2022-09-08T06:45:44.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:44 smithi055 ceph-mon[97033]: pgmap v17743: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:44 smithi059 ceph-mon[97450]: pgmap v17743: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:44.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:44.946753+0000) 2022-09-08T06:45:45.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:44.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:44.915230+0000) 2022-09-08T06:45:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:45.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:45.946998+0000) 2022-09-08T06:45:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:45.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:45.915465+0000) 2022-09-08T06:45:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:46 smithi055 ceph-mon[97033]: pgmap v17744: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:46 smithi059 ceph-mon[97450]: pgmap v17744: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:46.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:46.947217+0000) 2022-09-08T06:45:47.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:46.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:46.915627+0000) 2022-09-08T06:45:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:47.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:47.947448+0000) 2022-09-08T06:45:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:47.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:47.915771+0000) 2022-09-08T06:45:48.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:48 smithi055 ceph-mon[97033]: pgmap v17745: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:45:48.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:48 smithi059 ceph-mon[97450]: pgmap v17745: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:45:49.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:48.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:48.947732+0000) 2022-09-08T06:45:49.412 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:48.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:48.915909+0000) 2022-09-08T06:45:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:49.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:49.947987+0000) 2022-09-08T06:45:50.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:49.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:49.916050+0000) 2022-09-08T06:45:50.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:50 smithi055 ceph-mon[97033]: pgmap v17746: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:50 smithi059 ceph-mon[97450]: pgmap v17746: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:50.951 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:50.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:50.916298+0000) 2022-09-08T06:45:51.227 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:50.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:50.948189+0000) 2022-09-08T06:45:51.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:45:51.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:45:51.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:45:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:45:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:45:51.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:45:51.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:45:51.224+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:45:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:51.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:51.948414+0000) 2022-09-08T06:45:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:51.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:51.916525+0000) 2022-09-08T06:45:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:52 smithi055 ceph-mon[97033]: pgmap v17747: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:45:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:45:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:45:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:52 smithi059 ceph-mon[97450]: pgmap v17747: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:45:52.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:45:52.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:45:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:52.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:52.948632+0000) 2022-09-08T06:45:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:52.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:52.916730+0000) 2022-09-08T06:45:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:53.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:53.948826+0000) 2022-09-08T06:45:54.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:53.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:53.916897+0000) 2022-09-08T06:45:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:54 smithi055 ceph-mon[97033]: pgmap v17748: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:54 smithi059 ceph-mon[97450]: pgmap v17748: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:55.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:54.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:54.948967+0000) 2022-09-08T06:45:55.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:54.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:54.917120+0000) 2022-09-08T06:45:56.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:55.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:55.949182+0000) 2022-09-08T06:45:56.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:55.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:55.917258+0000) 2022-09-08T06:45:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:56 smithi055 ceph-mon[97033]: pgmap v17749: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:56 smithi059 ceph-mon[97450]: pgmap v17749: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:45:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:56.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:56.949427+0000) 2022-09-08T06:45:57.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:56.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:56.917454+0000) 2022-09-08T06:45:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:57.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:57.949702+0000) 2022-09-08T06:45:58.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:57.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:57.917714+0000) 2022-09-08T06:45:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:45:58 smithi055 ceph-mon[97033]: pgmap v17750: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:45:58.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:45:58 smithi059 ceph-mon[97450]: pgmap v17750: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:45:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:58.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:58.917971+0000) 2022-09-08T06:45:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:58.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:58.949842+0000) 2022-09-08T06:46:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:45:59.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:59.918222+0000) 2022-09-08T06:46:00.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:45:59.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:45:59.950073+0000) 2022-09-08T06:46:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:00 smithi055 ceph-mon[97033]: pgmap v17751: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:00 smithi059 ceph-mon[97450]: pgmap v17751: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:00.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:00.950300+0000) 2022-09-08T06:46:01.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:00.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:00.918453+0000) 2022-09-08T06:46:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:01.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:01.950544+0000) 2022-09-08T06:46:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:01.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:01.918755+0000) 2022-09-08T06:46:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:02 smithi055 ceph-mon[97033]: pgmap v17752: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:46:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:02 smithi059 ceph-mon[97450]: pgmap v17752: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:46:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:02.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:02.950737+0000) 2022-09-08T06:46:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:02.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:02.918977+0000) 2022-09-08T06:46:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:03.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:03.950883+0000) 2022-09-08T06:46:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:03.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:03.919203+0000) 2022-09-08T06:46:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:04 smithi055 ceph-mon[97033]: pgmap v17753: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:04 smithi059 ceph-mon[97450]: pgmap v17753: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:04.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:04.951030+0000) 2022-09-08T06:46:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:04.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:04.919362+0000) 2022-09-08T06:46:06.239 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:05.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:05.951265+0000) 2022-09-08T06:46:06.240 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:05.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:05.919560+0000) 2022-09-08T06:46:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:06 smithi055 ceph-mon[97033]: pgmap v17754: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:46:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:06 smithi059 ceph-mon[97450]: pgmap v17754: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:46:06.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:46:06.235+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:46:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:06.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:06.919766+0000) 2022-09-08T06:46:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:06.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:06.951483+0000) 2022-09-08T06:46:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:46:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:46:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:07.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:07.951738+0000) 2022-09-08T06:46:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:07.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:07.919989+0000) 2022-09-08T06:46:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:08 smithi055 ceph-mon[97033]: pgmap v17755: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:46:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:08 smithi059 ceph-mon[97450]: pgmap v17755: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:46:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:08.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:08.952003+0000) 2022-09-08T06:46:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:08.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:08.920155+0000) 2022-09-08T06:46:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:09.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:09.952233+0000) 2022-09-08T06:46:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:09.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:09.920354+0000) 2022-09-08T06:46:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:10 smithi055 ceph-mon[97033]: pgmap v17756: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:46:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:10 smithi055 ceph-mon[97033]: osdmap e1188: 12 total, 11 up, 11 in 2022-09-08T06:46:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:10 smithi059 ceph-mon[97450]: pgmap v17756: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:46:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:10 smithi059 ceph-mon[97450]: osdmap e1188: 12 total, 11 up, 11 in 2022-09-08T06:46:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:10.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:10.952481+0000) 2022-09-08T06:46:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:10.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:10.920515+0000) 2022-09-08T06:46:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:11.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:11.952666+0000) 2022-09-08T06:46:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:11.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:11.920780+0000) 2022-09-08T06:46:12.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:12 smithi055 ceph-mon[97033]: pgmap v17758: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:46:12.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:12 smithi055 ceph-mon[97033]: osdmap e1189: 12 total, 11 up, 11 in 2022-09-08T06:46:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:12 smithi059 ceph-mon[97450]: pgmap v17758: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:46:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:12 smithi059 ceph-mon[97450]: osdmap e1189: 12 total, 11 up, 11 in 2022-09-08T06:46:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:12.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:12.952901+0000) 2022-09-08T06:46:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:12.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:12.921018+0000) 2022-09-08T06:46:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:13.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:13.953153+0000) 2022-09-08T06:46:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:13.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:13.921191+0000) 2022-09-08T06:46:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:14 smithi055 ceph-mon[97033]: pgmap v17760: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:46:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:14 smithi059 ceph-mon[97450]: pgmap v17760: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2022-09-08T06:46:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:14.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:14.953358+0000) 2022-09-08T06:46:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:14.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:14.921426+0000) 2022-09-08T06:46:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:15.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:15.953589+0000) 2022-09-08T06:46:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:15.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:15.921766+0000) 2022-09-08T06:46:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:16 smithi055 ceph-mon[97033]: pgmap v17761: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:46:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:16 smithi059 ceph-mon[97450]: pgmap v17761: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:46:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:16.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:16.921921+0000) 2022-09-08T06:46:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:16.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:16.953828+0000) 2022-09-08T06:46:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:17.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:17.922078+0000) 2022-09-08T06:46:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:17.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:17.954071+0000) 2022-09-08T06:46:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:18 smithi055 ceph-mon[97033]: pgmap v17762: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:46:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:18 smithi059 ceph-mon[97450]: pgmap v17762: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:46:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:18.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:18.922232+0000) 2022-09-08T06:46:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:18.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:18.954235+0000) 2022-09-08T06:46:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:19.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:19.954450+0000) 2022-09-08T06:46:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:19.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:19.922480+0000) 2022-09-08T06:46:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:20 smithi055 ceph-mon[97033]: pgmap v17763: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 217 B/s wr, 6 op/s 2022-09-08T06:46:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:20 smithi059 ceph-mon[97450]: pgmap v17763: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 217 B/s wr, 6 op/s 2022-09-08T06:46:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:20.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:20.954714+0000) 2022-09-08T06:46:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:20.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:20.922732+0000) 2022-09-08T06:46:21.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:46:21.239+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:46:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:21 smithi055 ceph-mon[97033]: pgmap v17764: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:46:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:21 smithi055 ceph-mon[97033]: pgmap v17765: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 116 B/s wr, 7 op/s 2022-09-08T06:46:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:46:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:21 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T06:46:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:21 smithi059 ceph-mon[97450]: pgmap v17764: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:46:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:21 smithi059 ceph-mon[97450]: pgmap v17765: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 116 B/s wr, 7 op/s 2022-09-08T06:46:22.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:46:22.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:21 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T06:46:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:21.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:21.954905+0000) 2022-09-08T06:46:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:21.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:21.922957+0000) 2022-09-08T06:46:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:22.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:22.955158+0000) 2022-09-08T06:46:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:22.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:22.923098+0000) 2022-09-08T06:46:24.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:24 smithi059 ceph-mon[97450]: pgmap v17766: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T06:46:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:23.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:23.955338+0000) 2022-09-08T06:46:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:23.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:23.923298+0000) 2022-09-08T06:46:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:24 smithi055 ceph-mon[97033]: pgmap v17766: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T06:46:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:24.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:24.923510+0000) 2022-09-08T06:46:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:24.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:24.955543+0000) 2022-09-08T06:46:26.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:26 smithi059 ceph-mon[97450]: pgmap v17767: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T06:46:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:25.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:25.955765+0000) 2022-09-08T06:46:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:25.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:25.923683+0000) 2022-09-08T06:46:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:26 smithi055 ceph-mon[97033]: pgmap v17767: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T06:46:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:26.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:26.923829+0000) 2022-09-08T06:46:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:26.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:26.955951+0000) 2022-09-08T06:46:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:28 smithi059 ceph-mon[97450]: pgmap v17768: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:46:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:27.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:27.956120+0000) 2022-09-08T06:46:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:27.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:27.923973+0000) 2022-09-08T06:46:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:28 smithi055 ceph-mon[97033]: pgmap v17768: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:46:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:28.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:28.956342+0000) 2022-09-08T06:46:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:28.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:28.924107+0000) 2022-09-08T06:46:30.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:30 smithi059 ceph-mon[97450]: pgmap v17769: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:46:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:29.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:29.956583+0000) 2022-09-08T06:46:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:29.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:29.924276+0000) 2022-09-08T06:46:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:30 smithi055 ceph-mon[97033]: pgmap v17769: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:46:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:30.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:30.956747+0000) 2022-09-08T06:46:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:30.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:30.924455+0000) 2022-09-08T06:46:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:32 smithi059 ceph-mon[97450]: pgmap v17770: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:46:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:31.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:31.956943+0000) 2022-09-08T06:46:32.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:31.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:31.924618+0000) 2022-09-08T06:46:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:32 smithi055 ceph-mon[97033]: pgmap v17770: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:46:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:32.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:32.957178+0000) 2022-09-08T06:46:33.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:32.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:32.924822+0000) 2022-09-08T06:46:34.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:34 smithi059 ceph-mon[97450]: pgmap v17771: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:33.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:33.957338+0000) 2022-09-08T06:46:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:33.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:33.925014+0000) 2022-09-08T06:46:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:34 smithi055 ceph-mon[97033]: pgmap v17771: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:34.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:34.957553+0000) 2022-09-08T06:46:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:34.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:34.925211+0000) 2022-09-08T06:46:36.247 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:35.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:35.957794+0000) 2022-09-08T06:46:36.248 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:35.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:35.925365+0000) 2022-09-08T06:46:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:36 smithi059 ceph-mon[97450]: pgmap v17772: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:46:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:36 smithi055 ceph-mon[97033]: pgmap v17772: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:46:36.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:46:36.244+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:46:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:36.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:36.925557+0000) 2022-09-08T06:46:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:36.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:36.958011+0000) 2022-09-08T06:46:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:37.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:37.958222+0000) 2022-09-08T06:46:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:37.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:37.925707+0000) 2022-09-08T06:46:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:38 smithi055 ceph-mon[97033]: pgmap v17773: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:46:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:38 smithi059 ceph-mon[97450]: pgmap v17773: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:46:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:38.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:38.958463+0000) 2022-09-08T06:46:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:38.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:38.925887+0000) 2022-09-08T06:46:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:39.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:39.958743+0000) 2022-09-08T06:46:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:39.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:39.926108+0000) 2022-09-08T06:46:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:40 smithi055 ceph-mon[97033]: pgmap v17774: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:40 smithi059 ceph-mon[97450]: pgmap v17774: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:40.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:40.958941+0000) 2022-09-08T06:46:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:40.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:40.926341+0000) 2022-09-08T06:46:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:41.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:41.959214+0000) 2022-09-08T06:46:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:41.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:41.926471+0000) 2022-09-08T06:46:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:42 smithi055 ceph-mon[97033]: pgmap v17775: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:46:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:42 smithi059 ceph-mon[97450]: pgmap v17775: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:46:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:42.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:42.959454+0000) 2022-09-08T06:46:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:42.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:42.926721+0000) 2022-09-08T06:46:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:43.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:43.959725+0000) 2022-09-08T06:46:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:43.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:43.926900+0000) 2022-09-08T06:46:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:44 smithi055 ceph-mon[97033]: pgmap v17776: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:44 smithi059 ceph-mon[97450]: pgmap v17776: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:44.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:44.959940+0000) 2022-09-08T06:46:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:44.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:44.927032+0000) 2022-09-08T06:46:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:45.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:45.960172+0000) 2022-09-08T06:46:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:45.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:45.927160+0000) 2022-09-08T06:46:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:46 smithi055 ceph-mon[97033]: pgmap v17777: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:46:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:46 smithi059 ceph-mon[97450]: pgmap v17777: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:46:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:46.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:46.960414+0000) 2022-09-08T06:46:47.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:46.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:46.927358+0000) 2022-09-08T06:46:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:47.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:47.960664+0000) 2022-09-08T06:46:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:47.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:47.927524+0000) 2022-09-08T06:46:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:48 smithi055 ceph-mon[97033]: pgmap v17778: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:46:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:48 smithi059 ceph-mon[97450]: pgmap v17778: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:46:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:48.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:48.960916+0000) 2022-09-08T06:46:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:48.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:48.927693+0000) 2022-09-08T06:46:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:49.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:49.961091+0000) 2022-09-08T06:46:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:49.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:49.927851+0000) 2022-09-08T06:46:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:50 smithi055 ceph-mon[97033]: pgmap v17779: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:50 smithi059 ceph-mon[97450]: pgmap v17779: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:51.259 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:50.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:50.927986+0000) 2022-09-08T06:46:51.259 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:50.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:50.961298+0000) 2022-09-08T06:46:51.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:46:51.256+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:46:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:51.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:51.961520+0000) 2022-09-08T06:46:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:51.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:51.928147+0000) 2022-09-08T06:46:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:52 smithi055 ceph-mon[97033]: pgmap v17780: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:46:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:52 smithi059 ceph-mon[97450]: pgmap v17780: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:46:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:52.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:52.961745+0000) 2022-09-08T06:46:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:52.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:52.928382+0000) 2022-09-08T06:46:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:53.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:53.961989+0000) 2022-09-08T06:46:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:53.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:53.928656+0000) 2022-09-08T06:46:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:54 smithi055 ceph-mon[97033]: pgmap v17781: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:54 smithi059 ceph-mon[97450]: pgmap v17781: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:46:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:54.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:54.962154+0000) 2022-09-08T06:46:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:54.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:54.928819+0000) 2022-09-08T06:46:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:55.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:55.962371+0000) 2022-09-08T06:46:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:55.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:55.928944+0000) 2022-09-08T06:46:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:56 smithi055 ceph-mon[97033]: pgmap v17782: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:46:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:56 smithi059 ceph-mon[97450]: pgmap v17782: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:46:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:56.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:56.962618+0000) 2022-09-08T06:46:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:56.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:56.929085+0000) 2022-09-08T06:46:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:57.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:57.929269+0000) 2022-09-08T06:46:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:57.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:57.962831+0000) 2022-09-08T06:46:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:46:58 smithi055 ceph-mon[97033]: pgmap v17783: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:46:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:46:58 smithi059 ceph-mon[97450]: pgmap v17783: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:46:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:58.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:58.962984+0000) 2022-09-08T06:46:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:58.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:58.929436+0000) 2022-09-08T06:47:00.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:46:59.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:59.963189+0000) 2022-09-08T06:47:00.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:46:59.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:46:59.929612+0000) 2022-09-08T06:47:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:00 smithi055 ceph-mon[97033]: pgmap v17784: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:47:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:00 smithi059 ceph-mon[97450]: pgmap v17784: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:47:01.409 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:00.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:00.929825+0000) 2022-09-08T06:47:01.410 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:00.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:00.963341+0000) 2022-09-08T06:47:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:01.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:01.963586+0000) 2022-09-08T06:47:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:01.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:01.929953+0000) 2022-09-08T06:47:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:02 smithi055 ceph-mon[97033]: pgmap v17785: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:47:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:02 smithi059 ceph-mon[97450]: pgmap v17785: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:47:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:02.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:02.963925+0000) 2022-09-08T06:47:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:02.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:02.930057+0000) 2022-09-08T06:47:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:03.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:03.964222+0000) 2022-09-08T06:47:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:03.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:03.930268+0000) 2022-09-08T06:47:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:04 smithi055 ceph-mon[97033]: pgmap v17786: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:47:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:04 smithi059 ceph-mon[97450]: pgmap v17786: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:47:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:04.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:04.964463+0000) 2022-09-08T06:47:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:04.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:04.930525+0000) 2022-09-08T06:47:06.265 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:05.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:05.964724+0000) 2022-09-08T06:47:06.266 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:05.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:05.930755+0000) 2022-09-08T06:47:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:06 smithi055 ceph-mon[97033]: pgmap v17787: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 22 KiB/s wr, 18 op/s 2022-09-08T06:47:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:47:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:06 smithi059 ceph-mon[97450]: pgmap v17787: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 22 KiB/s wr, 18 op/s 2022-09-08T06:47:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:47:06.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:47:06.261+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:47:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:06.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:06.964959+0000) 2022-09-08T06:47:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:06.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:06.930887+0000) 2022-09-08T06:47:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:47:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:47:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:07.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:07.965207+0000) 2022-09-08T06:47:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:07.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:07.931070+0000) 2022-09-08T06:47:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:08 smithi055 ceph-mon[97033]: pgmap v17788: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T06:47:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:08 smithi059 ceph-mon[97450]: pgmap v17788: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T06:47:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:08.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:08.965416+0000) 2022-09-08T06:47:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:08.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:08.931193+0000) 2022-09-08T06:47:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:09.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:09.965674+0000) 2022-09-08T06:47:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:09.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:09.931405+0000) 2022-09-08T06:47:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:10 smithi055 ceph-mon[97033]: pgmap v17789: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:47:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:10 smithi055 ceph-mon[97033]: osdmap e1190: 12 total, 11 up, 11 in 2022-09-08T06:47:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:10 smithi059 ceph-mon[97450]: pgmap v17789: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:47:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:10 smithi059 ceph-mon[97450]: osdmap e1190: 12 total, 11 up, 11 in 2022-09-08T06:47:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:10.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:10.965912+0000) 2022-09-08T06:47:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:10.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:10.931661+0000) 2022-09-08T06:47:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:11.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:11.931886+0000) 2022-09-08T06:47:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:11.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:11.966110+0000) 2022-09-08T06:47:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:12 smithi055 ceph-mon[97033]: pgmap v17791: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:47:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:12 smithi055 ceph-mon[97033]: osdmap e1191: 12 total, 11 up, 11 in 2022-09-08T06:47:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:12 smithi059 ceph-mon[97450]: pgmap v17791: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:47:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:12 smithi059 ceph-mon[97450]: osdmap e1191: 12 total, 11 up, 11 in 2022-09-08T06:47:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:12.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:12.966319+0000) 2022-09-08T06:47:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:12.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:12.932045+0000) 2022-09-08T06:47:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:13.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:13.966541+0000) 2022-09-08T06:47:14.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:13.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:13.932225+0000) 2022-09-08T06:47:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:14 smithi055 ceph-mon[97033]: pgmap v17793: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.5 KiB/s rd, 34 KiB/s wr, 9 op/s 2022-09-08T06:47:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:14 smithi059 ceph-mon[97450]: pgmap v17793: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.5 KiB/s rd, 34 KiB/s wr, 9 op/s 2022-09-08T06:47:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:14.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:14.966743+0000) 2022-09-08T06:47:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:14.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:14.932370+0000) 2022-09-08T06:47:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:15.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:15.966911+0000) 2022-09-08T06:47:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:15.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:15.932555+0000) 2022-09-08T06:47:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:16 smithi055 ceph-mon[97033]: pgmap v17794: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-08T06:47:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:16 smithi059 ceph-mon[97450]: pgmap v17794: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-08T06:47:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:16.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:16.967119+0000) 2022-09-08T06:47:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:16.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:16.932743+0000) 2022-09-08T06:47:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:17.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:17.967284+0000) 2022-09-08T06:47:18.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:17.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:17.932867+0000) 2022-09-08T06:47:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:18 smithi055 ceph-mon[97033]: pgmap v17795: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:47:18.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:18 smithi059 ceph-mon[97450]: pgmap v17795: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:47:19.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:18.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:18.967464+0000) 2022-09-08T06:47:19.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:18.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:18.933059+0000) 2022-09-08T06:47:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:19.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:19.967740+0000) 2022-09-08T06:47:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:19.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:19.933223+0000) 2022-09-08T06:47:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:20 smithi055 ceph-mon[97033]: pgmap v17796: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 210 B/s wr, 6 op/s 2022-09-08T06:47:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:20 smithi059 ceph-mon[97450]: pgmap v17796: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 210 B/s wr, 6 op/s 2022-09-08T06:47:21.268 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:20.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:20.967891+0000) 2022-09-08T06:47:21.269 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:20.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:20.933393+0000) 2022-09-08T06:47:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:47:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:47:21.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:47:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:47:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:47:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:47:21.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:47:21.264+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:47:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:21.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:21.968055+0000) 2022-09-08T06:47:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:21.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:21.933533+0000) 2022-09-08T06:47:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:22 smithi055 ceph-mon[97033]: pgmap v17797: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:47:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:22 smithi059 ceph-mon[97450]: pgmap v17797: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:47:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:22.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:22.968277+0000) 2022-09-08T06:47:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:22.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:22.933751+0000) 2022-09-08T06:47:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:23.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:23.933954+0000) 2022-09-08T06:47:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:23.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:23.968509+0000) 2022-09-08T06:47:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:24 smithi055 ceph-mon[97033]: pgmap v17798: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 94 B/s wr, 5 op/s 2022-09-08T06:47:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:24 smithi059 ceph-mon[97450]: pgmap v17798: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 94 B/s wr, 5 op/s 2022-09-08T06:47:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:24.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:24.968680+0000) 2022-09-08T06:47:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:24.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:24.934133+0000) 2022-09-08T06:47:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:25.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:25.968847+0000) 2022-09-08T06:47:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:25.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:25.934274+0000) 2022-09-08T06:47:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:26 smithi055 ceph-mon[97033]: pgmap v17799: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2022-09-08T06:47:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:26 smithi059 ceph-mon[97450]: pgmap v17799: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2022-09-08T06:47:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:26.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:26.969005+0000) 2022-09-08T06:47:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:26.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:26.934401+0000) 2022-09-08T06:47:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:27.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:27.934518+0000) 2022-09-08T06:47:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:27.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:27.969234+0000) 2022-09-08T06:47:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:28 smithi055 ceph-mon[97033]: pgmap v17800: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:47:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:28 smithi059 ceph-mon[97450]: pgmap v17800: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:47:29.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:28.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:28.934691+0000) 2022-09-08T06:47:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:28.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:28.969493+0000) 2022-09-08T06:47:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:29.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:29.969738+0000) 2022-09-08T06:47:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:29.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:29.934924+0000) 2022-09-08T06:47:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:30 smithi055 ceph-mon[97033]: pgmap v17801: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:47:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:30 smithi059 ceph-mon[97450]: pgmap v17801: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:47:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:30.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:30.969995+0000) 2022-09-08T06:47:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:30.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:30.935094+0000) 2022-09-08T06:47:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:31.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:31.970229+0000) 2022-09-08T06:47:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:31.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:31.935259+0000) 2022-09-08T06:47:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:32 smithi055 ceph-mon[97033]: pgmap v17802: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:47:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:32 smithi059 ceph-mon[97450]: pgmap v17802: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:47:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:32.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:32.935469+0000) 2022-09-08T06:47:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:32.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:32.970448+0000) 2022-09-08T06:47:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:33.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:33.935681+0000) 2022-09-08T06:47:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:33.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:33.970704+0000) 2022-09-08T06:47:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:34 smithi055 ceph-mon[97033]: pgmap v17803: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:47:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:34 smithi059 ceph-mon[97450]: pgmap v17803: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:47:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:34.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:34.935818+0000) 2022-09-08T06:47:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:34.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:34.970964+0000) 2022-09-08T06:47:36.273 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:35.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:35.971189+0000) 2022-09-08T06:47:36.273 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:35.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:35.936035+0000) 2022-09-08T06:47:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:36 smithi055 ceph-mon[97033]: pgmap v17804: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:47:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:36 smithi059 ceph-mon[97450]: pgmap v17804: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:47:36.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:47:36.270+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:47:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:36.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:36.971403+0000) 2022-09-08T06:47:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:36.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:36.936208+0000) 2022-09-08T06:47:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:37.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:37.936354+0000) 2022-09-08T06:47:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:37.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:37.971625+0000) 2022-09-08T06:47:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:38 smithi055 ceph-mon[97033]: pgmap v17805: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:47:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:38 smithi059 ceph-mon[97450]: pgmap v17805: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:47:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:38.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:38.971883+0000) 2022-09-08T06:47:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:38.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:38.936552+0000) 2022-09-08T06:47:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:39.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:39.972112+0000) 2022-09-08T06:47:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:39.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:39.936712+0000) 2022-09-08T06:47:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:40 smithi055 ceph-mon[97033]: pgmap v17806: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:47:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:40 smithi059 ceph-mon[97450]: pgmap v17806: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:47:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:40.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:40.972352+0000) 2022-09-08T06:47:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:40.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:40.936862+0000) 2022-09-08T06:47:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:41.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:41.972632+0000) 2022-09-08T06:47:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:41.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:41.937019+0000) 2022-09-08T06:47:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:42 smithi055 ceph-mon[97033]: pgmap v17807: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:47:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:42 smithi059 ceph-mon[97450]: pgmap v17807: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:47:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:42.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:42.972933+0000) 2022-09-08T06:47:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:42.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:42.937167+0000) 2022-09-08T06:47:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:43.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:43.973139+0000) 2022-09-08T06:47:44.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:43.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:43.937328+0000) 2022-09-08T06:47:44.490 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:44 smithi055 ceph-mon[97033]: pgmap v17808: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:47:44.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:44 smithi059 ceph-mon[97450]: pgmap v17808: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:47:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:44.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:44.937480+0000) 2022-09-08T06:47:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:44.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:44.973317+0000) 2022-09-08T06:47:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:45.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:45.973552+0000) 2022-09-08T06:47:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:45.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:45.937652+0000) 2022-09-08T06:47:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:46 smithi055 ceph-mon[97033]: pgmap v17809: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:47:46.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:46 smithi059 ceph-mon[97450]: pgmap v17809: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:47:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:46.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:46.973838+0000) 2022-09-08T06:47:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:46.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:46.937820+0000) 2022-09-08T06:47:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:47.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:47.974101+0000) 2022-09-08T06:47:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:47.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:47.937969+0000) 2022-09-08T06:47:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:48 smithi055 ceph-mon[97033]: pgmap v17810: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:47:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:48 smithi059 ceph-mon[97450]: pgmap v17810: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:47:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:48.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:48.974324+0000) 2022-09-08T06:47:49.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:48.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:48.938164+0000) 2022-09-08T06:47:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:49.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:49.974554+0000) 2022-09-08T06:47:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:49.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:49.938375+0000) 2022-09-08T06:47:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:50 smithi055 ceph-mon[97033]: pgmap v17811: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:47:50.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:50 smithi059 ceph-mon[97450]: pgmap v17811: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:47:51.277 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:50.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:50.974832+0000) 2022-09-08T06:47:51.278 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:50.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:50.938549+0000) 2022-09-08T06:47:51.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:47:51.273+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:47:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:51.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:51.975128+0000) 2022-09-08T06:47:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:51.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:51.938762+0000) 2022-09-08T06:47:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:52 smithi055 ceph-mon[97033]: pgmap v17812: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:47:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:52 smithi055 ceph-mon[97033]: pgmap v17813: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:47:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:52 smithi055 ceph-mon[97033]: pgmap v17814: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:47:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:47:52.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:52 smithi059 ceph-mon[97450]: pgmap v17812: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:47:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:52 smithi059 ceph-mon[97450]: pgmap v17813: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:47:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:52 smithi059 ceph-mon[97450]: pgmap v17814: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:47:52.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:47:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:52.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:52.938909+0000) 2022-09-08T06:47:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:52.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:52.975367+0000) 2022-09-08T06:47:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:53.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:53.975570+0000) 2022-09-08T06:47:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:53.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:53.939067+0000) 2022-09-08T06:47:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:54 smithi055 ceph-mon[97033]: pgmap v17815: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:47:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:54 smithi059 ceph-mon[97450]: pgmap v17815: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:47:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:54.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:54.975832+0000) 2022-09-08T06:47:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:54.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:54.939211+0000) 2022-09-08T06:47:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:55.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:55.975969+0000) 2022-09-08T06:47:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:55.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:55.939337+0000) 2022-09-08T06:47:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:56 smithi055 ceph-mon[97033]: pgmap v17816: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:47:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:56 smithi059 ceph-mon[97450]: pgmap v17816: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:47:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:56.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:56.976207+0000) 2022-09-08T06:47:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:56.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:56.939556+0000) 2022-09-08T06:47:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:57.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:57.976436+0000) 2022-09-08T06:47:58.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:57.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:57.939722+0000) 2022-09-08T06:47:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:47:58 smithi055 ceph-mon[97033]: pgmap v17817: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:47:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:47:58 smithi059 ceph-mon[97450]: pgmap v17817: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:47:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:58.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:58.976600+0000) 2022-09-08T06:47:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:58.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:58.939861+0000) 2022-09-08T06:48:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:47:59.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:59.976832+0000) 2022-09-08T06:48:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:47:59.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:47:59.940023+0000) 2022-09-08T06:48:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:00 smithi055 ceph-mon[97033]: pgmap v17818: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:48:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:00 smithi059 ceph-mon[97450]: pgmap v17818: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:48:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:00.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:00.977048+0000) 2022-09-08T06:48:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:00.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:00.940272+0000) 2022-09-08T06:48:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:01.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:01.940515+0000) 2022-09-08T06:48:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:01.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:01.977295+0000) 2022-09-08T06:48:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:02 smithi055 ceph-mon[97033]: pgmap v17819: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:48:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:02 smithi059 ceph-mon[97450]: pgmap v17819: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:48:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:02.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:02.940688+0000) 2022-09-08T06:48:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:02.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:02.977525+0000) 2022-09-08T06:48:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:03.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:03.940867+0000) 2022-09-08T06:48:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:03.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:03.977758+0000) 2022-09-08T06:48:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:04 smithi055 ceph-mon[97033]: pgmap v17820: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:04.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:04 smithi059 ceph-mon[97450]: pgmap v17820: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:04.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:04.978031+0000) 2022-09-08T06:48:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:04.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:04.941069+0000) 2022-09-08T06:48:06.289 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:05.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:05.978240+0000) 2022-09-08T06:48:06.290 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:05.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:05.941285+0000) 2022-09-08T06:48:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:06 smithi055 ceph-mon[97033]: pgmap v17821: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:48:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:48:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:06 smithi059 ceph-mon[97450]: pgmap v17821: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:48:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:48:06.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:48:06.286+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:48:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:06.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:06.978460+0000) 2022-09-08T06:48:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:06.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:06.941452+0000) 2022-09-08T06:48:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:48:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:48:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:07.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:07.978621+0000) 2022-09-08T06:48:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:07.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:07.941684+0000) 2022-09-08T06:48:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:08 smithi055 ceph-mon[97033]: pgmap v17822: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:48:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:08 smithi059 ceph-mon[97450]: pgmap v17822: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:48:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:08.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:08.978817+0000) 2022-09-08T06:48:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:08.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:08.941849+0000) 2022-09-08T06:48:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:09.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:09.978963+0000) 2022-09-08T06:48:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:09.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:09.941980+0000) 2022-09-08T06:48:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:10 smithi055 ceph-mon[97033]: pgmap v17823: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:48:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:10 smithi055 ceph-mon[97033]: osdmap e1192: 12 total, 11 up, 11 in 2022-09-08T06:48:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:10 smithi059 ceph-mon[97450]: pgmap v17823: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:48:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:10 smithi059 ceph-mon[97450]: osdmap e1192: 12 total, 11 up, 11 in 2022-09-08T06:48:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:10.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:10.979189+0000) 2022-09-08T06:48:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:10.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:10.942193+0000) 2022-09-08T06:48:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:11.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:11.979430+0000) 2022-09-08T06:48:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:11.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:11.942429+0000) 2022-09-08T06:48:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:12 smithi055 ceph-mon[97033]: pgmap v17825: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:48:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:12 smithi055 ceph-mon[97033]: osdmap e1193: 12 total, 11 up, 11 in 2022-09-08T06:48:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:12 smithi059 ceph-mon[97450]: pgmap v17825: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:48:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:12 smithi059 ceph-mon[97450]: osdmap e1193: 12 total, 11 up, 11 in 2022-09-08T06:48:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:12.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:12.979699+0000) 2022-09-08T06:48:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:12.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:12.942612+0000) 2022-09-08T06:48:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:13.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:13.979909+0000) 2022-09-08T06:48:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:13.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:13.942797+0000) 2022-09-08T06:48:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:14 smithi055 ceph-mon[97033]: pgmap v17827: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:48:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:14 smithi059 ceph-mon[97450]: pgmap v17827: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:48:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:14.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:14.980148+0000) 2022-09-08T06:48:15.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:14.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:14.942954+0000) 2022-09-08T06:48:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:15.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:15.980374+0000) 2022-09-08T06:48:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:15.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:15.943103+0000) 2022-09-08T06:48:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:16 smithi055 ceph-mon[97033]: pgmap v17828: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:48:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:16 smithi059 ceph-mon[97450]: pgmap v17828: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:48:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:16.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:16.980613+0000) 2022-09-08T06:48:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:16.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:16.943250+0000) 2022-09-08T06:48:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:17.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:17.980836+0000) 2022-09-08T06:48:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:17.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:17.943419+0000) 2022-09-08T06:48:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:18 smithi055 ceph-mon[97033]: pgmap v17829: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:48:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:18 smithi059 ceph-mon[97450]: pgmap v17829: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:48:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:18.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:18.981028+0000) 2022-09-08T06:48:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:18.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:18.943562+0000) 2022-09-08T06:48:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:19.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:19.981241+0000) 2022-09-08T06:48:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:19.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:19.943787+0000) 2022-09-08T06:48:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:20 smithi055 ceph-mon[97033]: pgmap v17830: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:48:20.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:20 smithi059 ceph-mon[97450]: pgmap v17830: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:48:21.291 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:20.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:20.981415+0000) 2022-09-08T06:48:21.291 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:20.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:20.944022+0000) 2022-09-08T06:48:21.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:48:21.288+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:48:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:21.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:21.981565+0000) 2022-09-08T06:48:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:21.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:21.944197+0000) 2022-09-08T06:48:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:22 smithi055 ceph-mon[97033]: pgmap v17831: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:48:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:22 smithi059 ceph-mon[97450]: pgmap v17831: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:48:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:22.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:22.981772+0000) 2022-09-08T06:48:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:22.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:22.944352+0000) 2022-09-08T06:48:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:23.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:23.982024+0000) 2022-09-08T06:48:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:23.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:23.944565+0000) 2022-09-08T06:48:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:24 smithi055 ceph-mon[97033]: pgmap v17832: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T06:48:24.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:24 smithi059 ceph-mon[97450]: pgmap v17832: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 92 B/s wr, 5 op/s 2022-09-08T06:48:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:24.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:24.982238+0000) 2022-09-08T06:48:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:24.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:24.944748+0000) 2022-09-08T06:48:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:25.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:25.982449+0000) 2022-09-08T06:48:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:25.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:25.944901+0000) 2022-09-08T06:48:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:26 smithi055 ceph-mon[97033]: pgmap v17833: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:48:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:26 smithi059 ceph-mon[97450]: pgmap v17833: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:48:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:26.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:26.982694+0000) 2022-09-08T06:48:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:26.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:26.945132+0000) 2022-09-08T06:48:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:27.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:27.982919+0000) 2022-09-08T06:48:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:27.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:27.945337+0000) 2022-09-08T06:48:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:28 smithi055 ceph-mon[97033]: pgmap v17834: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:28 smithi059 ceph-mon[97450]: pgmap v17834: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:28.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:28.983167+0000) 2022-09-08T06:48:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:28.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:28.945472+0000) 2022-09-08T06:48:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:29.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:29.983392+0000) 2022-09-08T06:48:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:29.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:29.945600+0000) 2022-09-08T06:48:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:30 smithi055 ceph-mon[97033]: pgmap v17835: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:30 smithi059 ceph-mon[97450]: pgmap v17835: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:30.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:30.945823+0000) 2022-09-08T06:48:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:30.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:30.983675+0000) 2022-09-08T06:48:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:31.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:31.983865+0000) 2022-09-08T06:48:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:31.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:31.945986+0000) 2022-09-08T06:48:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:32 smithi055 ceph-mon[97033]: pgmap v17836: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:48:32.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:32 smithi059 ceph-mon[97450]: pgmap v17836: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:48:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:32.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:32.984117+0000) 2022-09-08T06:48:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:32.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:32.946197+0000) 2022-09-08T06:48:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:33.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:33.946387+0000) 2022-09-08T06:48:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:33.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:33.984390+0000) 2022-09-08T06:48:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:34 smithi055 ceph-mon[97033]: pgmap v17837: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:34.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:34 smithi059 ceph-mon[97450]: pgmap v17837: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:34.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:34.984568+0000) 2022-09-08T06:48:35.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:34.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:34.946540+0000) 2022-09-08T06:48:36.295 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:35.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:35.984778+0000) 2022-09-08T06:48:36.296 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:35.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:35.946701+0000) 2022-09-08T06:48:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:36 smithi055 ceph-mon[97033]: pgmap v17838: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:48:36.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:36 smithi059 ceph-mon[97450]: pgmap v17838: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:48:36.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:48:36.291+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:48:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:36.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:36.984955+0000) 2022-09-08T06:48:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:36.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:36.946862+0000) 2022-09-08T06:48:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:37.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:37.985183+0000) 2022-09-08T06:48:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:37.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:37.947029+0000) 2022-09-08T06:48:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:38 smithi055 ceph-mon[97033]: pgmap v17839: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:38 smithi059 ceph-mon[97450]: pgmap v17839: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:38.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:38.985399+0000) 2022-09-08T06:48:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:38.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:38.947176+0000) 2022-09-08T06:48:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:39.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:39.985707+0000) 2022-09-08T06:48:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:39.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:39.947339+0000) 2022-09-08T06:48:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:40 smithi055 ceph-mon[97033]: pgmap v17840: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:40.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:40 smithi059 ceph-mon[97450]: pgmap v17840: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:41.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:40.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:40.947524+0000) 2022-09-08T06:48:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:40.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:40.985923+0000) 2022-09-08T06:48:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:41.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:41.986047+0000) 2022-09-08T06:48:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:41.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:41.947711+0000) 2022-09-08T06:48:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:42 smithi055 ceph-mon[97033]: pgmap v17841: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:48:42.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:42 smithi059 ceph-mon[97450]: pgmap v17841: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:48:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:42.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:42.947910+0000) 2022-09-08T06:48:43.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:42.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:42.986246+0000) 2022-09-08T06:48:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:43.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:43.986407+0000) 2022-09-08T06:48:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:43.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:43.948084+0000) 2022-09-08T06:48:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:44 smithi055 ceph-mon[97033]: pgmap v17842: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:44 smithi059 ceph-mon[97450]: pgmap v17842: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:44.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:44.986673+0000) 2022-09-08T06:48:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:44.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:44.948303+0000) 2022-09-08T06:48:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:45.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:45.986830+0000) 2022-09-08T06:48:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:45.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:45.948504+0000) 2022-09-08T06:48:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:46 smithi055 ceph-mon[97033]: pgmap v17843: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:48:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:46 smithi059 ceph-mon[97450]: pgmap v17843: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:48:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:46.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:46.986958+0000) 2022-09-08T06:48:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:46.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:46.948628+0000) 2022-09-08T06:48:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:47.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:47.987108+0000) 2022-09-08T06:48:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:47.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:47.948887+0000) 2022-09-08T06:48:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:48 smithi055 ceph-mon[97033]: pgmap v17844: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:48 smithi059 ceph-mon[97450]: pgmap v17844: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:48.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:48.987317+0000) 2022-09-08T06:48:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:48.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:48.949095+0000) 2022-09-08T06:48:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:49.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:49.987480+0000) 2022-09-08T06:48:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:49.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:49.949305+0000) 2022-09-08T06:48:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:50 smithi055 ceph-mon[97033]: pgmap v17845: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:50.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:50 smithi059 ceph-mon[97450]: pgmap v17845: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:51.243 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:50.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:50.987716+0000) 2022-09-08T06:48:51.244 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:50.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:50.949437+0000) 2022-09-08T06:48:51.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:48:51.301+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:48:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:51.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:51.987996+0000) 2022-09-08T06:48:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:51.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:51.949577+0000) 2022-09-08T06:48:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:52 smithi055 ceph-mon[97033]: pgmap v17846: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:48:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:48:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:48:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:48:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:52 smithi059 ceph-mon[97450]: pgmap v17846: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:48:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:48:52.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:48:52.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:48:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:52.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:52.988163+0000) 2022-09-08T06:48:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:52.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:52.949775+0000) 2022-09-08T06:48:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:53.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:53.988395+0000) 2022-09-08T06:48:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:53.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:53.949920+0000) 2022-09-08T06:48:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:54 smithi055 ceph-mon[97033]: pgmap v17847: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:54.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:54 smithi059 ceph-mon[97450]: pgmap v17847: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:54.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:54.988584+0000) 2022-09-08T06:48:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:54.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:54.950114+0000) 2022-09-08T06:48:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:55.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:55.950282+0000) 2022-09-08T06:48:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:55.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:55.988816+0000) 2022-09-08T06:48:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:56 smithi055 ceph-mon[97033]: pgmap v17848: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:48:56.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:56 smithi059 ceph-mon[97450]: pgmap v17848: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:48:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:56.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:56.950445+0000) 2022-09-08T06:48:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:56.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:56.989073+0000) 2022-09-08T06:48:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:57.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:57.989353+0000) 2022-09-08T06:48:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:57.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:57.950675+0000) 2022-09-08T06:48:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:48:58 smithi055 ceph-mon[97033]: pgmap v17849: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:48:58 smithi059 ceph-mon[97450]: pgmap v17849: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:48:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:58.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:58.989604+0000) 2022-09-08T06:48:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:58.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:58.950905+0000) 2022-09-08T06:49:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:48:59.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:59.989771+0000) 2022-09-08T06:49:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:48:59.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:48:59.951071+0000) 2022-09-08T06:49:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:00 smithi055 ceph-mon[97033]: pgmap v17850: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:49:00.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:00 smithi059 ceph-mon[97450]: pgmap v17850: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:49:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:00.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:00.951202+0000) 2022-09-08T06:49:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:00.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:00.989892+0000) 2022-09-08T06:49:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:01.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:01.990062+0000) 2022-09-08T06:49:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:01.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:01.951361+0000) 2022-09-08T06:49:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:02 smithi055 ceph-mon[97033]: pgmap v17851: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:49:02.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:02 smithi059 ceph-mon[97450]: pgmap v17851: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:49:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:02.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:02.990282+0000) 2022-09-08T06:49:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:02.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:02.951589+0000) 2022-09-08T06:49:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:03.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:03.990505+0000) 2022-09-08T06:49:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:03.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:03.951764+0000) 2022-09-08T06:49:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:04 smithi055 ceph-mon[97033]: pgmap v17852: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:49:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:04 smithi059 ceph-mon[97450]: pgmap v17852: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:49:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:04.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:04.990735+0000) 2022-09-08T06:49:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:04.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:04.951921+0000) 2022-09-08T06:49:06.320 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:05.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:05.990963+0000) 2022-09-08T06:49:06.320 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:05.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:05.952148+0000) 2022-09-08T06:49:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:06 smithi055 ceph-mon[97033]: pgmap v17853: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:49:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:49:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:06 smithi059 ceph-mon[97450]: pgmap v17853: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:49:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:49:06.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:49:06.316+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:49:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:06.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:06.991157+0000) 2022-09-08T06:49:07.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:06.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:06.952332+0000) 2022-09-08T06:49:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:49:07.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:49:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:07.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:07.991354+0000) 2022-09-08T06:49:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:07.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:07.952570+0000) 2022-09-08T06:49:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:08 smithi055 ceph-mon[97033]: pgmap v17854: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:49:08.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:08 smithi059 ceph-mon[97450]: pgmap v17854: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:49:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:08.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:08.991583+0000) 2022-09-08T06:49:09.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:08.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:08.952760+0000) 2022-09-08T06:49:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:09.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:09.991756+0000) 2022-09-08T06:49:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:09.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:09.952923+0000) 2022-09-08T06:49:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:10 smithi055 ceph-mon[97033]: pgmap v17855: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:49:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:10 smithi055 ceph-mon[97033]: osdmap e1194: 12 total, 11 up, 11 in 2022-09-08T06:49:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:10 smithi059 ceph-mon[97450]: pgmap v17855: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:49:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:10 smithi059 ceph-mon[97450]: osdmap e1194: 12 total, 11 up, 11 in 2022-09-08T06:49:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:10.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:10.991914+0000) 2022-09-08T06:49:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:10.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:10.953135+0000) 2022-09-08T06:49:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:11.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:11.992085+0000) 2022-09-08T06:49:12.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:11.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:11.953320+0000) 2022-09-08T06:49:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:12 smithi055 ceph-mon[97033]: pgmap v17857: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:49:12.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:12 smithi059 ceph-mon[97450]: pgmap v17857: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:49:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:12.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:12.992271+0000) 2022-09-08T06:49:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:12.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:12.953543+0000) 2022-09-08T06:49:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:13.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:13.992474+0000) 2022-09-08T06:49:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:13.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:13.953791+0000) 2022-09-08T06:49:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:14 smithi055 ceph-mon[97033]: pgmap v17858: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:49:14.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:14 smithi059 ceph-mon[97450]: pgmap v17858: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:49:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:14.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:14.992696+0000) 2022-09-08T06:49:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:14.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:14.953954+0000) 2022-09-08T06:49:16.314 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:15.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:15.992922+0000) 2022-09-08T06:49:16.315 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:15.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:15.954171+0000) 2022-09-08T06:49:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:16 smithi055 ceph-mon[97033]: pgmap v17859: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:49:16.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:16 smithi059 ceph-mon[97450]: pgmap v17859: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:49:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:16.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:16.993079+0000) 2022-09-08T06:49:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:16.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:16.954390+0000) 2022-09-08T06:49:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:17.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:17.954551+0000) 2022-09-08T06:49:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:17.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:17.993317+0000) 2022-09-08T06:49:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:18 smithi055 ceph-mon[97033]: osdmap e1195: 12 total, 11 up, 11 in 2022-09-08T06:49:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:18 smithi055 ceph-mon[97033]: pgmap v17861: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:49:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:18 smithi059 ceph-mon[97450]: osdmap e1195: 12 total, 11 up, 11 in 2022-09-08T06:49:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:18 smithi059 ceph-mon[97450]: pgmap v17861: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:49:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:18.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:18.993554+0000) 2022-09-08T06:49:19.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:18.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:18.954768+0000) 2022-09-08T06:49:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:19.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:19.954943+0000) 2022-09-08T06:49:20.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:19.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:19.993753+0000) 2022-09-08T06:49:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:20 smithi055 ceph-mon[97033]: pgmap v17862: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:49:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:20 smithi059 ceph-mon[97450]: pgmap v17862: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:49:21.333 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:20.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:20.994023+0000) 2022-09-08T06:49:21.334 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:20.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:20.955155+0000) 2022-09-08T06:49:21.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:49:21.329+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:49:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:21.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:21.994284+0000) 2022-09-08T06:49:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:21.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:21.955373+0000) 2022-09-08T06:49:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:22 smithi055 ceph-mon[97033]: pgmap v17863: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T06:49:22.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:22 smithi055 ceph-mon[97033]: pgmap v17864: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T06:49:22.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:22 smithi055 ceph-mon[97033]: pgmap v17865: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:49:22.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:49:22.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:22 smithi059 ceph-mon[97450]: pgmap v17863: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T06:49:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:22 smithi059 ceph-mon[97450]: pgmap v17864: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T06:49:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:22 smithi059 ceph-mon[97450]: pgmap v17865: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:49:22.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:49:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:22.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:22.994534+0000) 2022-09-08T06:49:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:22.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:22.955603+0000) 2022-09-08T06:49:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:23.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:23.994760+0000) 2022-09-08T06:49:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:23.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:23.955823+0000) 2022-09-08T06:49:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:24 smithi055 ceph-mon[97033]: pgmap v17866: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:49:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:24 smithi059 ceph-mon[97450]: pgmap v17866: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:49:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:24.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:24.995010+0000) 2022-09-08T06:49:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:24.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:24.956078+0000) 2022-09-08T06:49:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:25.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:25.995250+0000) 2022-09-08T06:49:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:25.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:25.956217+0000) 2022-09-08T06:49:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:26 smithi055 ceph-mon[97033]: pgmap v17867: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:49:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:26 smithi059 ceph-mon[97450]: pgmap v17867: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:49:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:26.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:26.956454+0000) 2022-09-08T06:49:27.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:26.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:26.995429+0000) 2022-09-08T06:49:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:27.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:27.995619+0000) 2022-09-08T06:49:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:27.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:27.956722+0000) 2022-09-08T06:49:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:28 smithi055 ceph-mon[97033]: pgmap v17868: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:49:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:28 smithi059 ceph-mon[97450]: pgmap v17868: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:49:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:28.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:28.995783+0000) 2022-09-08T06:49:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:28.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:28.956942+0000) 2022-09-08T06:49:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:29.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:29.996043+0000) 2022-09-08T06:49:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:29.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:29.957107+0000) 2022-09-08T06:49:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:30 smithi055 ceph-mon[97033]: pgmap v17869: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:49:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:30 smithi059 ceph-mon[97450]: pgmap v17869: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:49:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:30.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:30.996257+0000) 2022-09-08T06:49:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:30.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:30.957314+0000) 2022-09-08T06:49:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:31.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:31.996484+0000) 2022-09-08T06:49:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:31.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:31.957441+0000) 2022-09-08T06:49:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:32 smithi055 ceph-mon[97033]: pgmap v17870: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:49:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:32 smithi059 ceph-mon[97450]: pgmap v17870: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:49:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:32.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:32.996745+0000) 2022-09-08T06:49:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:32.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:32.957571+0000) 2022-09-08T06:49:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:33.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:33.997000+0000) 2022-09-08T06:49:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:33.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:33.957732+0000) 2022-09-08T06:49:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:34 smithi055 ceph-mon[97033]: pgmap v17871: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:49:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:34 smithi059 ceph-mon[97450]: pgmap v17871: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:49:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:34.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:34.997246+0000) 2022-09-08T06:49:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:34.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:34.957901+0000) 2022-09-08T06:49:36.349 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:35.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:35.997444+0000) 2022-09-08T06:49:36.350 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:35.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:35.958033+0000) 2022-09-08T06:49:36.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:49:36.346+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:49:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:36 smithi055 ceph-mon[97033]: pgmap v17872: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:49:36.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:36 smithi059 ceph-mon[97450]: pgmap v17872: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:49:37.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:36.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:36.997669+0000) 2022-09-08T06:49:37.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:36.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:36.958232+0000) 2022-09-08T06:49:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:37.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:37.997961+0000) 2022-09-08T06:49:38.652 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:37.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:37.958407+0000) 2022-09-08T06:49:38.653 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:38 smithi059 ceph-mon[97450]: pgmap v17873: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:49:38.654 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:38 smithi055 ceph-mon[97033]: pgmap v17873: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:49:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:38.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:38.998155+0000) 2022-09-08T06:49:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:38.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:38.958598+0000) 2022-09-08T06:49:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:39.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:39.998298+0000) 2022-09-08T06:49:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:39.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:39.958758+0000) 2022-09-08T06:49:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:40 smithi055 ceph-mon[97033]: pgmap v17874: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:49:40.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:40 smithi059 ceph-mon[97450]: pgmap v17874: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:49:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:40.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:40.998503+0000) 2022-09-08T06:49:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:40.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:40.958919+0000) 2022-09-08T06:49:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:41.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:41.998753+0000) 2022-09-08T06:49:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:41.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:41.959066+0000) 2022-09-08T06:49:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:42 smithi055 ceph-mon[97033]: pgmap v17875: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:49:42.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:42 smithi059 ceph-mon[97450]: pgmap v17875: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:49:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:42.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:42.998960+0000) 2022-09-08T06:49:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:42.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:42.959193+0000) 2022-09-08T06:49:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:43.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:43.999189+0000) 2022-09-08T06:49:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:43.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:43.959382+0000) 2022-09-08T06:49:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:44 smithi055 ceph-mon[97033]: pgmap v17876: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:49:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:44 smithi059 ceph-mon[97450]: pgmap v17876: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:49:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:44.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:44.999426+0000) 2022-09-08T06:49:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:44.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:44.959562+0000) 2022-09-08T06:49:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:45.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:45.999709+0000) 2022-09-08T06:49:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:45.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:45.959832+0000) 2022-09-08T06:49:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:46 smithi055 ceph-mon[97033]: pgmap v17877: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:49:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:46 smithi059 ceph-mon[97450]: pgmap v17877: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:49:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:46.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:46.999859+0000) 2022-09-08T06:49:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:46.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:46.960049+0000) 2022-09-08T06:49:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:47.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:48.000029+0000) 2022-09-08T06:49:48.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:47.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:47.960297+0000) 2022-09-08T06:49:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:48 smithi055 ceph-mon[97033]: pgmap v17878: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:49:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:48 smithi059 ceph-mon[97450]: pgmap v17878: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:49:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:48.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:49.000229+0000) 2022-09-08T06:49:49.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:48.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:48.960537+0000) 2022-09-08T06:49:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:50.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:50.000443+0000) 2022-09-08T06:49:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:49.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:49.960794+0000) 2022-09-08T06:49:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:50 smithi055 ceph-mon[97033]: pgmap v17879: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:49:50.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:50 smithi059 ceph-mon[97450]: pgmap v17879: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:49:51.352 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:51.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:51.000683+0000) 2022-09-08T06:49:51.353 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:50.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:50.961033+0000) 2022-09-08T06:49:51.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:49:51.349+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:49:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:52.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:52.000908+0000) 2022-09-08T06:49:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:51.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:51.961162+0000) 2022-09-08T06:49:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:52 smithi055 ceph-mon[97033]: pgmap v17880: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:49:52.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:52 smithi059 ceph-mon[97450]: pgmap v17880: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:49:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:53.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:53.001136+0000) 2022-09-08T06:49:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:52.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:52.961395+0000) 2022-09-08T06:49:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:54.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:54.001353+0000) 2022-09-08T06:49:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:53.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:53.961547+0000) 2022-09-08T06:49:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:54 smithi055 ceph-mon[97033]: pgmap v17881: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:49:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:54 smithi059 ceph-mon[97450]: pgmap v17881: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:49:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:55.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:55.001584+0000) 2022-09-08T06:49:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:54.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:54.961763+0000) 2022-09-08T06:49:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:56.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:56.001860+0000) 2022-09-08T06:49:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:55.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:55.961967+0000) 2022-09-08T06:49:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:56 smithi055 ceph-mon[97033]: pgmap v17882: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:49:56.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:56 smithi059 ceph-mon[97450]: pgmap v17882: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:49:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:57.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:57.002083+0000) 2022-09-08T06:49:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:56.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:56.962222+0000) 2022-09-08T06:49:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:58.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:58.002232+0000) 2022-09-08T06:49:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:57.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:57.962455+0000) 2022-09-08T06:49:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:49:58 smithi055 ceph-mon[97033]: pgmap v17883: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:49:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:49:58 smithi059 ceph-mon[97450]: pgmap v17883: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:49:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:49:59.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:59.002368+0000) 2022-09-08T06:49:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:58.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:58.962614+0000) 2022-09-08T06:50:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:00.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:00.002597+0000) 2022-09-08T06:50:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:49:59.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:49:59.962823+0000) 2022-09-08T06:50:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:00 smithi055 ceph-mon[97033]: pgmap v17884: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T06:50:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T06:50:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T06:50:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T06:50:00.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T06:50:00.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:00 smithi059 ceph-mon[97450]: pgmap v17884: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T06:50:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T06:50:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T06:50:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T06:50:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T06:50:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:01.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:01.002854+0000) 2022-09-08T06:50:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:00.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:00.963042+0000) 2022-09-08T06:50:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:02.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:02.003028+0000) 2022-09-08T06:50:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:01.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:01.963280+0000) 2022-09-08T06:50:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:02 smithi055 ceph-mon[97033]: pgmap v17885: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:50:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:02 smithi059 ceph-mon[97450]: pgmap v17885: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:50:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:02.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:02.963501+0000) 2022-09-08T06:50:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:03.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:03.003228+0000) 2022-09-08T06:50:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:03.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:03.963771+0000) 2022-09-08T06:50:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:04.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:04.003432+0000) 2022-09-08T06:50:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:04 smithi055 ceph-mon[97033]: pgmap v17886: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:04.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:04 smithi059 ceph-mon[97450]: pgmap v17886: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:04.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:04.964004+0000) 2022-09-08T06:50:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:05.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:05.003709+0000) 2022-09-08T06:50:06.368 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:06.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:06.003934+0000) 2022-09-08T06:50:06.369 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:05.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:05.964254+0000) 2022-09-08T06:50:06.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:50:06.364+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:50:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:06 smithi055 ceph-mon[97033]: pgmap v17887: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:50:06.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:50:06.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:06 smithi059 ceph-mon[97450]: pgmap v17887: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:50:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:50:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:06.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:06.964414+0000) 2022-09-08T06:50:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:07.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:07.004162+0000) 2022-09-08T06:50:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:50:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:50:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:08.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:08.004374+0000) 2022-09-08T06:50:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:07.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:07.964610+0000) 2022-09-08T06:50:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:08 smithi055 ceph-mon[97033]: pgmap v17888: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:50:08.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:08 smithi059 ceph-mon[97450]: pgmap v17888: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:50:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:09.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:09.004541+0000) 2022-09-08T06:50:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:08.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:08.964807+0000) 2022-09-08T06:50:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:10.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:10.004740+0000) 2022-09-08T06:50:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:09.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:09.965003+0000) 2022-09-08T06:50:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:10 smithi055 ceph-mon[97033]: pgmap v17889: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:50:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:10 smithi055 ceph-mon[97033]: osdmap e1196: 12 total, 11 up, 11 in 2022-09-08T06:50:10.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:10 smithi059 ceph-mon[97450]: pgmap v17889: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:50:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:10 smithi059 ceph-mon[97450]: osdmap e1196: 12 total, 11 up, 11 in 2022-09-08T06:50:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:11.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:11.004926+0000) 2022-09-08T06:50:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:10.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:10.965220+0000) 2022-09-08T06:50:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:11.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:11.965434+0000) 2022-09-08T06:50:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:12.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:12.005157+0000) 2022-09-08T06:50:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:12 smithi055 ceph-mon[97033]: pgmap v17891: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:50:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:12 smithi059 ceph-mon[97450]: pgmap v17891: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:50:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:13.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:13.005393+0000) 2022-09-08T06:50:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:12.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:12.965708+0000) 2022-09-08T06:50:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:14.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:14.005667+0000) 2022-09-08T06:50:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:13.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:13.965936+0000) 2022-09-08T06:50:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:14 smithi055 ceph-mon[97033]: pgmap v17892: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:50:14.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:14 smithi059 ceph-mon[97450]: pgmap v17892: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:50:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:15.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:15.005936+0000) 2022-09-08T06:50:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:14.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:14.966109+0000) 2022-09-08T06:50:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:16.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:16.006129+0000) 2022-09-08T06:50:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:15.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:15.966347+0000) 2022-09-08T06:50:16.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:16 smithi055 ceph-mon[97033]: pgmap v17893: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:50:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:16 smithi059 ceph-mon[97450]: pgmap v17893: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:50:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:17.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:17.006255+0000) 2022-09-08T06:50:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:16.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:16.966497+0000) 2022-09-08T06:50:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:18.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:18.006480+0000) 2022-09-08T06:50:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:17.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:17.966679+0000) 2022-09-08T06:50:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:18 smithi055 ceph-mon[97033]: osdmap e1197: 12 total, 11 up, 11 in 2022-09-08T06:50:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:18 smithi055 ceph-mon[97033]: pgmap v17895: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:50:18.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:18 smithi059 ceph-mon[97450]: osdmap e1197: 12 total, 11 up, 11 in 2022-09-08T06:50:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:18 smithi059 ceph-mon[97450]: pgmap v17895: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:50:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:19.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:19.006745+0000) 2022-09-08T06:50:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:18.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:18.966824+0000) 2022-09-08T06:50:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:20.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:20.006991+0000) 2022-09-08T06:50:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:19.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:19.966989+0000) 2022-09-08T06:50:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:20 smithi055 ceph-mon[97033]: pgmap v17896: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:50:20.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:20 smithi059 ceph-mon[97450]: pgmap v17896: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T06:50:21.369 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:21.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:21.007215+0000) 2022-09-08T06:50:21.370 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:20.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:20.967211+0000) 2022-09-08T06:50:21.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:50:21.366+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:50:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:21.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:21.967373+0000) 2022-09-08T06:50:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:22.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:22.007359+0000) 2022-09-08T06:50:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:22 smithi055 ceph-mon[97033]: pgmap v17897: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:50:22.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:50:22.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:50:22.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:50:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:22 smithi059 ceph-mon[97450]: pgmap v17897: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:50:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:50:22.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:50:22.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:50:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:23.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:23.007574+0000) 2022-09-08T06:50:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:22.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:22.967523+0000) 2022-09-08T06:50:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:24.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:24.007825+0000) 2022-09-08T06:50:24.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:23.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:23.967684+0000) 2022-09-08T06:50:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:24 smithi055 ceph-mon[97033]: pgmap v17898: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:50:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:24 smithi059 ceph-mon[97450]: pgmap v17898: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:50:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:25.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:25.008059+0000) 2022-09-08T06:50:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:24.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:24.967867+0000) 2022-09-08T06:50:26.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:25.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:25.968047+0000) 2022-09-08T06:50:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:26.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:26.008279+0000) 2022-09-08T06:50:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:26 smithi055 ceph-mon[97033]: pgmap v17899: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:50:26.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:26 smithi059 ceph-mon[97450]: pgmap v17899: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:50:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:27.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:27.008516+0000) 2022-09-08T06:50:27.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:26.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:26.968206+0000) 2022-09-08T06:50:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:28.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:28.008745+0000) 2022-09-08T06:50:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:27.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:27.968358+0000) 2022-09-08T06:50:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:28 smithi055 ceph-mon[97033]: pgmap v17900: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:50:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:28 smithi059 ceph-mon[97450]: pgmap v17900: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:50:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:29.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:29.008942+0000) 2022-09-08T06:50:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:28.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:28.968497+0000) 2022-09-08T06:50:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:30.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:30.009187+0000) 2022-09-08T06:50:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:29.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:29.968685+0000) 2022-09-08T06:50:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:30 smithi055 ceph-mon[97033]: pgmap v17901: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:30 smithi059 ceph-mon[97450]: pgmap v17901: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:30.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:30.968826+0000) 2022-09-08T06:50:31.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:31.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:31.009327+0000) 2022-09-08T06:50:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:31.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:31.968951+0000) 2022-09-08T06:50:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:32.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:32.009495+0000) 2022-09-08T06:50:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:32 smithi055 ceph-mon[97033]: pgmap v17902: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:50:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:32 smithi059 ceph-mon[97450]: pgmap v17902: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:50:33.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:32.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:32.969094+0000) 2022-09-08T06:50:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:33.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:33.009776+0000) 2022-09-08T06:50:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:33.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:33.969248+0000) 2022-09-08T06:50:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:34.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:34.009933+0000) 2022-09-08T06:50:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:34 smithi055 ceph-mon[97033]: pgmap v17903: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:34 smithi059 ceph-mon[97450]: pgmap v17903: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:35.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:35.010126+0000) 2022-09-08T06:50:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:34.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:34.969389+0000) 2022-09-08T06:50:36.384 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:36.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:36.010317+0000) 2022-09-08T06:50:36.384 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:35.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:35.969547+0000) 2022-09-08T06:50:36.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:50:36.381+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:50:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:36 smithi055 ceph-mon[97033]: pgmap v17904: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:50:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:36 smithi059 ceph-mon[97450]: pgmap v17904: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:50:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:37.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:37.010517+0000) 2022-09-08T06:50:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:36.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:36.969711+0000) 2022-09-08T06:50:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:37.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:37.969866+0000) 2022-09-08T06:50:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:38.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:38.010717+0000) 2022-09-08T06:50:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:38 smithi055 ceph-mon[97033]: pgmap v17905: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:38 smithi059 ceph-mon[97450]: pgmap v17905: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:39.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:39.010887+0000) 2022-09-08T06:50:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:38.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:38.970014+0000) 2022-09-08T06:50:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:40.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:40.011170+0000) 2022-09-08T06:50:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:39.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:39.970192+0000) 2022-09-08T06:50:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:40 smithi055 ceph-mon[97033]: pgmap v17906: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:40 smithi059 ceph-mon[97450]: pgmap v17906: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:41.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:41.011426+0000) 2022-09-08T06:50:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:40.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:40.970433+0000) 2022-09-08T06:50:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:41.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:41.970610+0000) 2022-09-08T06:50:42.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:42.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:42.011591+0000) 2022-09-08T06:50:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:42 smithi055 ceph-mon[97033]: pgmap v17907: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:50:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:42 smithi059 ceph-mon[97450]: pgmap v17907: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:50:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:42.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:42.970806+0000) 2022-09-08T06:50:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:43.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:43.011805+0000) 2022-09-08T06:50:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:43.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:43.970988+0000) 2022-09-08T06:50:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:44.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:44.011960+0000) 2022-09-08T06:50:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:44 smithi055 ceph-mon[97033]: pgmap v17908: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:44.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:44 smithi059 ceph-mon[97450]: pgmap v17908: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:44.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:44.971163+0000) 2022-09-08T06:50:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:45.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:45.012195+0000) 2022-09-08T06:50:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:46.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:46.012423+0000) 2022-09-08T06:50:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:45.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:45.971380+0000) 2022-09-08T06:50:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:46 smithi055 ceph-mon[97033]: pgmap v17909: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:50:46.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:46 smithi059 ceph-mon[97450]: pgmap v17909: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:50:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:46.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:46.971536+0000) 2022-09-08T06:50:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:47.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:47.012628+0000) 2022-09-08T06:50:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:48.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:48.012845+0000) 2022-09-08T06:50:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:47.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:47.971748+0000) 2022-09-08T06:50:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:48 smithi055 ceph-mon[97033]: pgmap v17910: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:48.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:48 smithi059 ceph-mon[97450]: pgmap v17910: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:48.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:48.971971+0000) 2022-09-08T06:50:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:49.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:49.013038+0000) 2022-09-08T06:50:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:50.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:50.013269+0000) 2022-09-08T06:50:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:49.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:49.972148+0000) 2022-09-08T06:50:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:50 smithi055 ceph-mon[97033]: pgmap v17911: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:50.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:50 smithi059 ceph-mon[97450]: pgmap v17911: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:50:51.386 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:51.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:51.013521+0000) 2022-09-08T06:50:51.387 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:50.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:50.972299+0000) 2022-09-08T06:50:51.638 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:50:51.382+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:50:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:52.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:52.013788+0000) 2022-09-08T06:50:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:51.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:51.972435+0000) 2022-09-08T06:50:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:52 smithi055 ceph-mon[97033]: pgmap v17912: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:50:52.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:50:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:52 smithi059 ceph-mon[97450]: pgmap v17912: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:50:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:50:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:53.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:53.013968+0000) 2022-09-08T06:50:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:52.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:52.972564+0000) 2022-09-08T06:50:53.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:53 smithi055 ceph-mon[97033]: pgmap v17913: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:50:53.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:53 smithi055 ceph-mon[97033]: pgmap v17914: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:50:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:53 smithi059 ceph-mon[97450]: pgmap v17913: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:50:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:53 smithi059 ceph-mon[97450]: pgmap v17914: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:50:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:54.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:54.014166+0000) 2022-09-08T06:50:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:53.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:53.972771+0000) 2022-09-08T06:50:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:55.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:55.014418+0000) 2022-09-08T06:50:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:54.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:54.972992+0000) 2022-09-08T06:50:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:55 smithi055 ceph-mon[97033]: pgmap v17915: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:50:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:55 smithi059 ceph-mon[97450]: pgmap v17915: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:50:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:56.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:56.014625+0000) 2022-09-08T06:50:56.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:55.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:55.973204+0000) 2022-09-08T06:50:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:57.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:57.014881+0000) 2022-09-08T06:50:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:56.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:56.973370+0000) 2022-09-08T06:50:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:57 smithi055 ceph-mon[97033]: pgmap v17916: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:50:57.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:57 smithi059 ceph-mon[97450]: pgmap v17916: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:50:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:58.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:58.015104+0000) 2022-09-08T06:50:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:57.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:57.973577+0000) 2022-09-08T06:50:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:50:59.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:59.015336+0000) 2022-09-08T06:50:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:58.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:58.973780+0000) 2022-09-08T06:50:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:50:59 smithi055 ceph-mon[97033]: pgmap v17917: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:50:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:50:59 smithi059 ceph-mon[97450]: pgmap v17917: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:51:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:00.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:00.015559+0000) 2022-09-08T06:51:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:50:59.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:50:59.973997+0000) 2022-09-08T06:51:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:00.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:00.974130+0000) 2022-09-08T06:51:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:01.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:01.015707+0000) 2022-09-08T06:51:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:01 smithi055 ceph-mon[97033]: pgmap v17918: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:51:01.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:01 smithi059 ceph-mon[97450]: pgmap v17918: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:51:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:02.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:02.015846+0000) 2022-09-08T06:51:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:01.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:01.974264+0000) 2022-09-08T06:51:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:02.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:02.974407+0000) 2022-09-08T06:51:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:03.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:03.016178+0000) 2022-09-08T06:51:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:03 smithi059 ceph-mon[97450]: pgmap v17919: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:51:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:03 smithi055 ceph-mon[97033]: pgmap v17919: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:51:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:04.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:04.016421+0000) 2022-09-08T06:51:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:03.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:03.974630+0000) 2022-09-08T06:51:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:04.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:04.974845+0000) 2022-09-08T06:51:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:05.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:05.016705+0000) 2022-09-08T06:51:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:05 smithi059 ceph-mon[97450]: pgmap v17920: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:05 smithi055 ceph-mon[97033]: pgmap v17920: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:06.389 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:06.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:06.016929+0000) 2022-09-08T06:51:06.390 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:05.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:05.975014+0000) 2022-09-08T06:51:06.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:51:06.387+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:51:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:51:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:51:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:07.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:07.017156+0000) 2022-09-08T06:51:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:06.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:06.975134+0000) 2022-09-08T06:51:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:07 smithi059 ceph-mon[97450]: pgmap v17921: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:51:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:51:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:07 smithi055 ceph-mon[97033]: pgmap v17921: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:51:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:51:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:08.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:08.017345+0000) 2022-09-08T06:51:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:07.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:07.975294+0000) 2022-09-08T06:51:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:08 smithi055 ceph-mon[97033]: pgmap v17922: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:51:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:08 smithi059 ceph-mon[97450]: pgmap v17922: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:51:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:09.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:09.017535+0000) 2022-09-08T06:51:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:08.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:08.975420+0000) 2022-09-08T06:51:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:10.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:10.017682+0000) 2022-09-08T06:51:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:09.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:09.975882+0000) 2022-09-08T06:51:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:10 smithi055 ceph-mon[97033]: osdmap e1198: 12 total, 11 up, 11 in 2022-09-08T06:51:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:10 smithi055 ceph-mon[97033]: pgmap v17924: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:51:11.031 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:10 smithi059 ceph-mon[97450]: osdmap e1198: 12 total, 11 up, 11 in 2022-09-08T06:51:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:10 smithi059 ceph-mon[97450]: pgmap v17924: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:51:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:11.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:11.017862+0000) 2022-09-08T06:51:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:10.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:10.976088+0000) 2022-09-08T06:51:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:12.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:12.017997+0000) 2022-09-08T06:51:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:11.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:11.976349+0000) 2022-09-08T06:51:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:13.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:13.018206+0000) 2022-09-08T06:51:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:12.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:12.976533+0000) 2022-09-08T06:51:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:13 smithi055 ceph-mon[97033]: pgmap v17925: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:51:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:13 smithi059 ceph-mon[97450]: pgmap v17925: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:51:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:14.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:14.018385+0000) 2022-09-08T06:51:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:13.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:13.976787+0000) 2022-09-08T06:51:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:14.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:14.976925+0000) 2022-09-08T06:51:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:15.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:15.018600+0000) 2022-09-08T06:51:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:15 smithi055 ceph-mon[97033]: pgmap v17926: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:51:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:15 smithi059 ceph-mon[97450]: pgmap v17926: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:51:16.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:15.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:15.977133+0000) 2022-09-08T06:51:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:16.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:16.018853+0000) 2022-09-08T06:51:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:17.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:17.019010+0000) 2022-09-08T06:51:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:16.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:16.977383+0000) 2022-09-08T06:51:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:17 smithi055 ceph-mon[97033]: pgmap v17927: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:51:17.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:17 smithi055 ceph-mon[97033]: osdmap e1199: 12 total, 11 up, 11 in 2022-09-08T06:51:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:17 smithi059 ceph-mon[97450]: pgmap v17927: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:51:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:17 smithi059 ceph-mon[97450]: osdmap e1199: 12 total, 11 up, 11 in 2022-09-08T06:51:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:17.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:17.977632+0000) 2022-09-08T06:51:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:18.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:18.019230+0000) 2022-09-08T06:51:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:18.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:18.977794+0000) 2022-09-08T06:51:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:19.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:19.019474+0000) 2022-09-08T06:51:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:19 smithi055 ceph-mon[97033]: pgmap v17929: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T06:51:19.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:19 smithi059 ceph-mon[97450]: pgmap v17929: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T06:51:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:20.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:20.019594+0000) 2022-09-08T06:51:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:19.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:19.977955+0000) 2022-09-08T06:51:21.395 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:21.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:21.019830+0000) 2022-09-08T06:51:21.395 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:20.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:20.978144+0000) 2022-09-08T06:51:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:21 smithi055 ceph-mon[97033]: pgmap v17930: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:51:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:21 smithi059 ceph-mon[97450]: pgmap v17930: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:51:21.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:51:21.391+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:51:22.409 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:22.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:22.020055+0000) 2022-09-08T06:51:22.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:21.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:21.978371+0000) 2022-09-08T06:51:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:23.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:23.020317+0000) 2022-09-08T06:51:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:22.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:22.978604+0000) 2022-09-08T06:51:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:23 smithi055 ceph-mon[97033]: pgmap v17931: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:51:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:23 smithi059 ceph-mon[97450]: pgmap v17931: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:51:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:24.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:24.020549+0000) 2022-09-08T06:51:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:23.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:23.978855+0000) 2022-09-08T06:51:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:25.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:25.020788+0000) 2022-09-08T06:51:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:24.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:24.979005+0000) 2022-09-08T06:51:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:25 smithi055 ceph-mon[97033]: pgmap v17932: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:51:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:25 smithi059 ceph-mon[97450]: pgmap v17932: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:51:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:26.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:26.020995+0000) 2022-09-08T06:51:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:25.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:25.979236+0000) 2022-09-08T06:51:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:27.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:27.021199+0000) 2022-09-08T06:51:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:26.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:26.979448+0000) 2022-09-08T06:51:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:27 smithi055 ceph-mon[97033]: pgmap v17933: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:51:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:27 smithi059 ceph-mon[97450]: pgmap v17933: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:51:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:27.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:27.979581+0000) 2022-09-08T06:51:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:28.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:28.021367+0000) 2022-09-08T06:51:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:29.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:29.021553+0000) 2022-09-08T06:51:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:28.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:28.979741+0000) 2022-09-08T06:51:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:29 smithi055 ceph-mon[97033]: pgmap v17934: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:51:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:29 smithi059 ceph-mon[97450]: pgmap v17934: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:51:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:30.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:30.021776+0000) 2022-09-08T06:51:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:29.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:29.979929+0000) 2022-09-08T06:51:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:31.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:31.021986+0000) 2022-09-08T06:51:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:30.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:30.980076+0000) 2022-09-08T06:51:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:31 smithi055 ceph-mon[97033]: pgmap v17935: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:31 smithi059 ceph-mon[97450]: pgmap v17935: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:32.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:32.022192+0000) 2022-09-08T06:51:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:31.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:31.980270+0000) 2022-09-08T06:51:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:33.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:33.022424+0000) 2022-09-08T06:51:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:32.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:32.980484+0000) 2022-09-08T06:51:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:33 smithi055 ceph-mon[97033]: pgmap v17936: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:51:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:33 smithi059 ceph-mon[97450]: pgmap v17936: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:51:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:34.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:34.022762+0000) 2022-09-08T06:51:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:33.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:33.980631+0000) 2022-09-08T06:51:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:34.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:34.980771+0000) 2022-09-08T06:51:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:35.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:35.022970+0000) 2022-09-08T06:51:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:35 smithi055 ceph-mon[97033]: pgmap v17937: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:35 smithi059 ceph-mon[97450]: pgmap v17937: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:35.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:35.980968+0000) 2022-09-08T06:51:36.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:36.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:36.023138+0000) 2022-09-08T06:51:36.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:51:36.406+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:51:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:37.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:37.023315+0000) 2022-09-08T06:51:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:36.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:36.981095+0000) 2022-09-08T06:51:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:37 smithi055 ceph-mon[97033]: pgmap v17938: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:51:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:37 smithi059 ceph-mon[97450]: pgmap v17938: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:51:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:38.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:38.023558+0000) 2022-09-08T06:51:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:37.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:37.981231+0000) 2022-09-08T06:51:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:38.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:38.981417+0000) 2022-09-08T06:51:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:39.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:39.023708+0000) 2022-09-08T06:51:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:39 smithi055 ceph-mon[97033]: pgmap v17939: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:39.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:39 smithi059 ceph-mon[97450]: pgmap v17939: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:40.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:40.023943+0000) 2022-09-08T06:51:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:39.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:39.981594+0000) 2022-09-08T06:51:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:41.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:41.024198+0000) 2022-09-08T06:51:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:40.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:40.981816+0000) 2022-09-08T06:51:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:41 smithi055 ceph-mon[97033]: pgmap v17940: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:41 smithi059 ceph-mon[97450]: pgmap v17940: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:42.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:42.024432+0000) 2022-09-08T06:51:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:41.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:41.982051+0000) 2022-09-08T06:51:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:43.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:43.024550+0000) 2022-09-08T06:51:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:42.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:42.982242+0000) 2022-09-08T06:51:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:43 smithi055 ceph-mon[97033]: pgmap v17941: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:51:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:43 smithi059 ceph-mon[97450]: pgmap v17941: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:51:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:44.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:44.024688+0000) 2022-09-08T06:51:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:43.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:43.982390+0000) 2022-09-08T06:51:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:45.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:45.024918+0000) 2022-09-08T06:51:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:44.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:44.982625+0000) 2022-09-08T06:51:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:45 smithi055 ceph-mon[97033]: pgmap v17942: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:45 smithi059 ceph-mon[97450]: pgmap v17942: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:46.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:46.025125+0000) 2022-09-08T06:51:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:45.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:45.982813+0000) 2022-09-08T06:51:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:47.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:47.025352+0000) 2022-09-08T06:51:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:46.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:46.982964+0000) 2022-09-08T06:51:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:47 smithi055 ceph-mon[97033]: pgmap v17943: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:51:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:47 smithi059 ceph-mon[97450]: pgmap v17943: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:51:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:48.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:48.025582+0000) 2022-09-08T06:51:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:47.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:47.983137+0000) 2022-09-08T06:51:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:49.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:49.025747+0000) 2022-09-08T06:51:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:48.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:48.983362+0000) 2022-09-08T06:51:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:49 smithi055 ceph-mon[97033]: pgmap v17944: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:49 smithi059 ceph-mon[97450]: pgmap v17944: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:50.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:50.025864+0000) 2022-09-08T06:51:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:49.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:49.983605+0000) 2022-09-08T06:51:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:51.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:51.026099+0000) 2022-09-08T06:51:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:50.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:50.983846+0000) 2022-09-08T06:51:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:51 smithi055 ceph-mon[97033]: pgmap v17945: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:51 smithi059 ceph-mon[97450]: pgmap v17945: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:51.800 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:51:51.411+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:51:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:52.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:52.026372+0000) 2022-09-08T06:51:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:51.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:51.984001+0000) 2022-09-08T06:51:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:51:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:51:52.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:51:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:51:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:51:52.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:51:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:53.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:53.026665+0000) 2022-09-08T06:51:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:52.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:52.984143+0000) 2022-09-08T06:51:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:53 smithi055 ceph-mon[97033]: pgmap v17946: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:51:53.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:53 smithi059 ceph-mon[97450]: pgmap v17946: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:51:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:54.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:54.026901+0000) 2022-09-08T06:51:54.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:53.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:53.984355+0000) 2022-09-08T06:51:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:55.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:55.027087+0000) 2022-09-08T06:51:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:54.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:54.984545+0000) 2022-09-08T06:51:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:55 smithi055 ceph-mon[97033]: pgmap v17947: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:55.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:55 smithi059 ceph-mon[97450]: pgmap v17947: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:56.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:56.027276+0000) 2022-09-08T06:51:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:55.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:55.984755+0000) 2022-09-08T06:51:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:57.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:57.027488+0000) 2022-09-08T06:51:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:56.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:56.984936+0000) 2022-09-08T06:51:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:57 smithi055 ceph-mon[97033]: pgmap v17948: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:51:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:57 smithi059 ceph-mon[97450]: pgmap v17948: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:51:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:58.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:58.027683+0000) 2022-09-08T06:51:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:57.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:57.985089+0000) 2022-09-08T06:51:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:51:59.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:59.027865+0000) 2022-09-08T06:51:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:58.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:58.985283+0000) 2022-09-08T06:51:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:51:59 smithi055 ceph-mon[97033]: pgmap v17949: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:51:59.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:51:59 smithi059 ceph-mon[97450]: pgmap v17949: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:00.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:00.028127+0000) 2022-09-08T06:52:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:51:59.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:51:59.985499+0000) 2022-09-08T06:52:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:01.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:01.028350+0000) 2022-09-08T06:52:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:00.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:00.985675+0000) 2022-09-08T06:52:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:01 smithi055 ceph-mon[97033]: pgmap v17950: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:01 smithi059 ceph-mon[97450]: pgmap v17950: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:02.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:02.028624+0000) 2022-09-08T06:52:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:01.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:01.985815+0000) 2022-09-08T06:52:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:03.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:03.028904+0000) 2022-09-08T06:52:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:02.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:02.986021+0000) 2022-09-08T06:52:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:03 smithi055 ceph-mon[97033]: pgmap v17951: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:52:03.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:03 smithi059 ceph-mon[97450]: pgmap v17951: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:52:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:04.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:04.029146+0000) 2022-09-08T06:52:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:03.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:03.986195+0000) 2022-09-08T06:52:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:05.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:05.029371+0000) 2022-09-08T06:52:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:04.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:04.986354+0000) 2022-09-08T06:52:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:05 smithi055 ceph-mon[97033]: pgmap v17952: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:05 smithi059 ceph-mon[97450]: pgmap v17952: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:05.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:05.986611+0000) 2022-09-08T06:52:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:06.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:06.029508+0000) 2022-09-08T06:52:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:52:06.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:52:06.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:52:06.423+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:52:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:07.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:07.029730+0000) 2022-09-08T06:52:07.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:06.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:06.986872+0000) 2022-09-08T06:52:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:07 smithi055 ceph-mon[97033]: pgmap v17953: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:52:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:52:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:07 smithi059 ceph-mon[97450]: pgmap v17953: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:52:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:52:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:08.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:08.029972+0000) 2022-09-08T06:52:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:07.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:07.987045+0000) 2022-09-08T06:52:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:09.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:09.030212+0000) 2022-09-08T06:52:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:08.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:08.987282+0000) 2022-09-08T06:52:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:09 smithi055 ceph-mon[97033]: pgmap v17954: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:52:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:09 smithi059 ceph-mon[97450]: pgmap v17954: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:52:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:10.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:10.030421+0000) 2022-09-08T06:52:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:09.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:09.987529+0000) 2022-09-08T06:52:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:10 smithi059 ceph-mon[97450]: osdmap e1200: 12 total, 11 up, 11 in 2022-09-08T06:52:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:10 smithi055 ceph-mon[97033]: osdmap e1200: 12 total, 11 up, 11 in 2022-09-08T06:52:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:11.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:11.030555+0000) 2022-09-08T06:52:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:10.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:10.987751+0000) 2022-09-08T06:52:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:11 smithi059 ceph-mon[97450]: pgmap v17956: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:52:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:11 smithi055 ceph-mon[97033]: pgmap v17956: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:52:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:12.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:12.030757+0000) 2022-09-08T06:52:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:11.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:11.987961+0000) 2022-09-08T06:52:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:13.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:13.031013+0000) 2022-09-08T06:52:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:12.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:12.988161+0000) 2022-09-08T06:52:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:13 smithi059 ceph-mon[97450]: pgmap v17957: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:52:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:13 smithi055 ceph-mon[97033]: pgmap v17957: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:52:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:14.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:14.031240+0000) 2022-09-08T06:52:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:13.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:13.988340+0000) 2022-09-08T06:52:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:15.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:15.031438+0000) 2022-09-08T06:52:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:14.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:14.988494+0000) 2022-09-08T06:52:15.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:15 smithi059 ceph-mon[97450]: pgmap v17958: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:52:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:15 smithi055 ceph-mon[97033]: pgmap v17958: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:52:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:16.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:16.031701+0000) 2022-09-08T06:52:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:15.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:15.988698+0000) 2022-09-08T06:52:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:17.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:17.031980+0000) 2022-09-08T06:52:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:16.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:16.988857+0000) 2022-09-08T06:52:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:17 smithi059 ceph-mon[97450]: pgmap v17959: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:52:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:17 smithi059 ceph-mon[97450]: osdmap e1201: 12 total, 11 up, 11 in 2022-09-08T06:52:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:17 smithi055 ceph-mon[97033]: pgmap v17959: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:52:17.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:17 smithi055 ceph-mon[97033]: osdmap e1201: 12 total, 11 up, 11 in 2022-09-08T06:52:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:17.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:17.989013+0000) 2022-09-08T06:52:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:18.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:18.032228+0000) 2022-09-08T06:52:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:19.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:19.032447+0000) 2022-09-08T06:52:19.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:18.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:18.989153+0000) 2022-09-08T06:52:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:19 smithi059 ceph-mon[97450]: pgmap v17961: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-08T06:52:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:19 smithi055 ceph-mon[97033]: pgmap v17961: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-08T06:52:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:20.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:20.032713+0000) 2022-09-08T06:52:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:19.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:19.989341+0000) 2022-09-08T06:52:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:21.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:21.032942+0000) 2022-09-08T06:52:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:20.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:20.989579+0000) 2022-09-08T06:52:21.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:21 smithi059 ceph-mon[97450]: pgmap v17962: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:52:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:21 smithi055 ceph-mon[97033]: pgmap v17962: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:52:21.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:52:21.438+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:52:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:22.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:22.033180+0000) 2022-09-08T06:52:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:21.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:21.989772+0000) 2022-09-08T06:52:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:22 smithi055 ceph-mon[97033]: pgmap v17963: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:52:23.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:22 smithi055 ceph-mon[97033]: pgmap v17964: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T06:52:23.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:22 smithi055 ceph-mon[97033]: pgmap v17965: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:52:23.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:52:23.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:22 smithi059 ceph-mon[97450]: pgmap v17963: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:52:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:22 smithi059 ceph-mon[97450]: pgmap v17964: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T06:52:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:22 smithi059 ceph-mon[97450]: pgmap v17965: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:52:23.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:52:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:23.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:23.033389+0000) 2022-09-08T06:52:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:22.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:22.990042+0000) 2022-09-08T06:52:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:24.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:24.033592+0000) 2022-09-08T06:52:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:23.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:23.990202+0000) 2022-09-08T06:52:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:24 smithi055 ceph-mon[97033]: pgmap v17966: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-09-08T06:52:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:24 smithi059 ceph-mon[97450]: pgmap v17966: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-09-08T06:52:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:25.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:25.033839+0000) 2022-09-08T06:52:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:24.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:24.990360+0000) 2022-09-08T06:52:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:26.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:26.034023+0000) 2022-09-08T06:52:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:25.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:25.990503+0000) 2022-09-08T06:52:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:26 smithi055 ceph-mon[97033]: pgmap v17967: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:52:27.281 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:26 smithi059 ceph-mon[97450]: pgmap v17967: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:52:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:27.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:27.034246+0000) 2022-09-08T06:52:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:26.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:26.990654+0000) 2022-09-08T06:52:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:28.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:28.034495+0000) 2022-09-08T06:52:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:27.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:27.990885+0000) 2022-09-08T06:52:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:29.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:29.034752+0000) 2022-09-08T06:52:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:28.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:28.991063+0000) 2022-09-08T06:52:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:29 smithi055 ceph-mon[97033]: pgmap v17968: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:52:29.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:29 smithi059 ceph-mon[97450]: pgmap v17968: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:52:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:30.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:30.034916+0000) 2022-09-08T06:52:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:29.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:29.991227+0000) 2022-09-08T06:52:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:31.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:31.035116+0000) 2022-09-08T06:52:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:30.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:30.991383+0000) 2022-09-08T06:52:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:31 smithi055 ceph-mon[97033]: pgmap v17969: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-08T06:52:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:31 smithi059 ceph-mon[97450]: pgmap v17969: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-08T06:52:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:31.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:31.991521+0000) 2022-09-08T06:52:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:32.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:32.035341+0000) 2022-09-08T06:52:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:33.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:33.035535+0000) 2022-09-08T06:52:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:32.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:32.991702+0000) 2022-09-08T06:52:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:33 smithi055 ceph-mon[97033]: pgmap v17970: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:52:33.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:33 smithi059 ceph-mon[97450]: pgmap v17970: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:52:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:34.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:34.035679+0000) 2022-09-08T06:52:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:33.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:33.991844+0000) 2022-09-08T06:52:35.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:34.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:34.992001+0000) 2022-09-08T06:52:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:35.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:35.035911+0000) 2022-09-08T06:52:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:35 smithi055 ceph-mon[97033]: pgmap v17971: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:35.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:35 smithi059 ceph-mon[97450]: pgmap v17971: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:35.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:35.992239+0000) 2022-09-08T06:52:36.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:36.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:36.036092+0000) 2022-09-08T06:52:36.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:52:36.440+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:52:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:37.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:37.036301+0000) 2022-09-08T06:52:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:36.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:36.992418+0000) 2022-09-08T06:52:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:37 smithi055 ceph-mon[97033]: pgmap v17972: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:52:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:37 smithi059 ceph-mon[97450]: pgmap v17972: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:52:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:38.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:38.036487+0000) 2022-09-08T06:52:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:37.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:37.992575+0000) 2022-09-08T06:52:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:39.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:39.036708+0000) 2022-09-08T06:52:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:38.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:38.992754+0000) 2022-09-08T06:52:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:39 smithi055 ceph-mon[97033]: pgmap v17973: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:39 smithi059 ceph-mon[97450]: pgmap v17973: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:40.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:40.036972+0000) 2022-09-08T06:52:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:39.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:39.992901+0000) 2022-09-08T06:52:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:41.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:41.037212+0000) 2022-09-08T06:52:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:40.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:40.993120+0000) 2022-09-08T06:52:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:41 smithi055 ceph-mon[97033]: pgmap v17974: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:52:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:41 smithi059 ceph-mon[97450]: pgmap v17974: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:52:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:42.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:42.037468+0000) 2022-09-08T06:52:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:41.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:41.993344+0000) 2022-09-08T06:52:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:43.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:43.037744+0000) 2022-09-08T06:52:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:42.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:42.993536+0000) 2022-09-08T06:52:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:43 smithi055 ceph-mon[97033]: pgmap v17975: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:52:43.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:43 smithi059 ceph-mon[97450]: pgmap v17975: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:52:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:44.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:44.037999+0000) 2022-09-08T06:52:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:43.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:43.993736+0000) 2022-09-08T06:52:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:45.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:45.038287+0000) 2022-09-08T06:52:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:44.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:44.993939+0000) 2022-09-08T06:52:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:45 smithi055 ceph-mon[97033]: pgmap v17976: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:45 smithi059 ceph-mon[97450]: pgmap v17976: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:46.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:46.038518+0000) 2022-09-08T06:52:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:45.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:45.994079+0000) 2022-09-08T06:52:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:47.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:47.038665+0000) 2022-09-08T06:52:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:46.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:46.994210+0000) 2022-09-08T06:52:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:47 smithi055 ceph-mon[97033]: pgmap v17977: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:52:47.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:47 smithi059 ceph-mon[97450]: pgmap v17977: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:52:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:47.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:47.994350+0000) 2022-09-08T06:52:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:48.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:48.038871+0000) 2022-09-08T06:52:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:49.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:49.039123+0000) 2022-09-08T06:52:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:48.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:48.994556+0000) 2022-09-08T06:52:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:49 smithi055 ceph-mon[97033]: pgmap v17978: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:49.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:49 smithi059 ceph-mon[97450]: pgmap v17978: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:50.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:50.039388+0000) 2022-09-08T06:52:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:49.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:49.994759+0000) 2022-09-08T06:52:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:51.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:51.039578+0000) 2022-09-08T06:52:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:50.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:50.994892+0000) 2022-09-08T06:52:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:51 smithi055 ceph-mon[97033]: pgmap v17979: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:52:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:51 smithi059 ceph-mon[97450]: pgmap v17979: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:52:51.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:52:51.446+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:52:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:52.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:52.039784+0000) 2022-09-08T06:52:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:51.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:51.995078+0000) 2022-09-08T06:52:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:53.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:53.039992+0000) 2022-09-08T06:52:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:52.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:52.995286+0000) 2022-09-08T06:52:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:53 smithi055 ceph-mon[97033]: pgmap v17980: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:52:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:53 smithi059 ceph-mon[97450]: pgmap v17980: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:52:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:54.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:54.040229+0000) 2022-09-08T06:52:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:53.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:53.995518+0000) 2022-09-08T06:52:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:55.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:55.040407+0000) 2022-09-08T06:52:55.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:54.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:54.995689+0000) 2022-09-08T06:52:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:55 smithi055 ceph-mon[97033]: pgmap v17981: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:55 smithi059 ceph-mon[97450]: pgmap v17981: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:56.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:56.040620+0000) 2022-09-08T06:52:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:55.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:55.995888+0000) 2022-09-08T06:52:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:57.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:57.040858+0000) 2022-09-08T06:52:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:56.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:56.996020+0000) 2022-09-08T06:52:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:57 smithi055 ceph-mon[97033]: pgmap v17982: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:52:57.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:57 smithi059 ceph-mon[97450]: pgmap v17982: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:52:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:58.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:58.041069+0000) 2022-09-08T06:52:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:57.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:57.996155+0000) 2022-09-08T06:52:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:52:59.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:59.041246+0000) 2022-09-08T06:52:59.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:58.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:58.996294+0000) 2022-09-08T06:52:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:52:59 smithi055 ceph-mon[97033]: pgmap v17983: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:52:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:52:59 smithi059 ceph-mon[97450]: pgmap v17983: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:53:00.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:52:59.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:52:59.996448+0000) 2022-09-08T06:53:00.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:00.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:00.041435+0000) 2022-09-08T06:53:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:01.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:01.041577+0000) 2022-09-08T06:53:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:00.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:00.996578+0000) 2022-09-08T06:53:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:01 smithi055 ceph-mon[97033]: pgmap v17984: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:53:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:01 smithi059 ceph-mon[97450]: pgmap v17984: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:53:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:02.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:02.041802+0000) 2022-09-08T06:53:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:01.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:01.996788+0000) 2022-09-08T06:53:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:02.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:02.997029+0000) 2022-09-08T06:53:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:03.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:03.042035+0000) 2022-09-08T06:53:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:03 smithi055 ceph-mon[97033]: pgmap v17985: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:53:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:03 smithi059 ceph-mon[97450]: pgmap v17985: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:53:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:04.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:04.042269+0000) 2022-09-08T06:53:04.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:03.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:03.997195+0000) 2022-09-08T06:53:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:05.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:05.042492+0000) 2022-09-08T06:53:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:04.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:04.997357+0000) 2022-09-08T06:53:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:05 smithi055 ceph-mon[97033]: pgmap v17986: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:53:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:05 smithi059 ceph-mon[97450]: pgmap v17986: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:53:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:05.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:05.997538+0000) 2022-09-08T06:53:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:06.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:06.042665+0000) 2022-09-08T06:53:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:53:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:53:06.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:53:06.458+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:53:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:07.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:07.042829+0000) 2022-09-08T06:53:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:06.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:06.997734+0000) 2022-09-08T06:53:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:07 smithi055 ceph-mon[97033]: pgmap v17987: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:53:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:53:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:07 smithi059 ceph-mon[97450]: pgmap v17987: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:53:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:53:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:08.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:08.043048+0000) 2022-09-08T06:53:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:07.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:07.997928+0000) 2022-09-08T06:53:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:09.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:09.043282+0000) 2022-09-08T06:53:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:08.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:08.998236+0000) 2022-09-08T06:53:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:09 smithi055 ceph-mon[97033]: pgmap v17988: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:53:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:09 smithi059 ceph-mon[97450]: pgmap v17988: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:53:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:10.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:10.043523+0000) 2022-09-08T06:53:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:09.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:09.998491+0000) 2022-09-08T06:53:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:10 smithi055 ceph-mon[97033]: osdmap e1202: 12 total, 11 up, 11 in 2022-09-08T06:53:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:10 smithi059 ceph-mon[97450]: osdmap e1202: 12 total, 11 up, 11 in 2022-09-08T06:53:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:11.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:11.043764+0000) 2022-09-08T06:53:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:10.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:10.998657+0000) 2022-09-08T06:53:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:11 smithi055 ceph-mon[97033]: pgmap v17990: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:53:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:11 smithi059 ceph-mon[97450]: pgmap v17990: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:53:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:12.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:12.043992+0000) 2022-09-08T06:53:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:11.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:11.998749+0000) 2022-09-08T06:53:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:13.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:13.044236+0000) 2022-09-08T06:53:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:12.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:12.998895+0000) 2022-09-08T06:53:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:13 smithi055 ceph-mon[97033]: pgmap v17991: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:53:13.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:13 smithi059 ceph-mon[97450]: pgmap v17991: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:53:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:14.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:14.044450+0000) 2022-09-08T06:53:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:13.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:13.999109+0000) 2022-09-08T06:53:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:15.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:15.044650+0000) 2022-09-08T06:53:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:14.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:14.999299+0000) 2022-09-08T06:53:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:15 smithi055 ceph-mon[97033]: pgmap v17992: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:53:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:15 smithi059 ceph-mon[97450]: pgmap v17992: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:53:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:16.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:16.044879+0000) 2022-09-08T06:53:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:15.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:15.999527+0000) 2022-09-08T06:53:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:17.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:17.045052+0000) 2022-09-08T06:53:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:16.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:16.999765+0000) 2022-09-08T06:53:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:17 smithi055 ceph-mon[97033]: pgmap v17993: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:53:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:17 smithi055 ceph-mon[97033]: osdmap e1203: 12 total, 11 up, 11 in 2022-09-08T06:53:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:17 smithi059 ceph-mon[97450]: pgmap v17993: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:53:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:17 smithi059 ceph-mon[97450]: osdmap e1203: 12 total, 11 up, 11 in 2022-09-08T06:53:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:18.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:18.045283+0000) 2022-09-08T06:53:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:17.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:18.000017+0000) 2022-09-08T06:53:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:19.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:19.045529+0000) 2022-09-08T06:53:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:18.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:19.000236+0000) 2022-09-08T06:53:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:19 smithi055 ceph-mon[97033]: pgmap v17995: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-08T06:53:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:19 smithi059 ceph-mon[97450]: pgmap v17995: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-08T06:53:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:19.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:20.000470+0000) 2022-09-08T06:53:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:20.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:20.045770+0000) 2022-09-08T06:53:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:21.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:21.046004+0000) 2022-09-08T06:53:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:20.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:21.000716+0000) 2022-09-08T06:53:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:21 smithi055 ceph-mon[97033]: pgmap v17996: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:53:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:21 smithi059 ceph-mon[97450]: pgmap v17996: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:53:21.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:53:21.471+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:53:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:22.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:22.046166+0000) 2022-09-08T06:53:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:22.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:22.000936+0000) 2022-09-08T06:53:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:53:22.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:53:22.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:53:22.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:53:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:53:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:53:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:23.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:23.046329+0000) 2022-09-08T06:53:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:22.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:23.001117+0000) 2022-09-08T06:53:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:23 smithi059 ceph-mon[97450]: pgmap v17997: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:53:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:23 smithi055 ceph-mon[97033]: pgmap v17997: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:53:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:24.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:24.046545+0000) 2022-09-08T06:53:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:24.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:24.001314+0000) 2022-09-08T06:53:25.409 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:25.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:25.046825+0000) 2022-09-08T06:53:25.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:25.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:25.001483+0000) 2022-09-08T06:53:25.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:25 smithi059 ceph-mon[97450]: pgmap v17998: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:53:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:25 smithi055 ceph-mon[97033]: pgmap v17998: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:53:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:26.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:26.047030+0000) 2022-09-08T06:53:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:26.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:26.001615+0000) 2022-09-08T06:53:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:27.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:27.047178+0000) 2022-09-08T06:53:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:27.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:27.001797+0000) 2022-09-08T06:53:27.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:27 smithi059 ceph-mon[97450]: pgmap v17999: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:53:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:27 smithi055 ceph-mon[97033]: pgmap v17999: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:53:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:28.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:28.047382+0000) 2022-09-08T06:53:28.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:28.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:28.002019+0000) 2022-09-08T06:53:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:29.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:29.047511+0000) 2022-09-08T06:53:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:29.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:29.002317+0000) 2022-09-08T06:53:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:29 smithi059 ceph-mon[97450]: pgmap v18000: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:53:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:29 smithi055 ceph-mon[97033]: pgmap v18000: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:53:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:30.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:30.047710+0000) 2022-09-08T06:53:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:30.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:30.002532+0000) 2022-09-08T06:53:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:31.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:31.002767+0000) 2022-09-08T06:53:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:31.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:31.047920+0000) 2022-09-08T06:53:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:31 smithi059 ceph-mon[97450]: pgmap v18001: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:53:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:31 smithi055 ceph-mon[97033]: pgmap v18001: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:53:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:32.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:32.003005+0000) 2022-09-08T06:53:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:32.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:32.048151+0000) 2022-09-08T06:53:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:33.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:33.048379+0000) 2022-09-08T06:53:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:33.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:33.003241+0000) 2022-09-08T06:53:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:33 smithi059 ceph-mon[97450]: pgmap v18002: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:53:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:33 smithi055 ceph-mon[97033]: pgmap v18002: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:53:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:34.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:34.048579+0000) 2022-09-08T06:53:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:34.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:34.003432+0000) 2022-09-08T06:53:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:35.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:35.048835+0000) 2022-09-08T06:53:35.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:35.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:35.003569+0000) 2022-09-08T06:53:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:35 smithi059 ceph-mon[97450]: pgmap v18003: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:53:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:35 smithi055 ceph-mon[97033]: pgmap v18003: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:53:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:36.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:36.003726+0000) 2022-09-08T06:53:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:36.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:36.049056+0000) 2022-09-08T06:53:36.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:53:36.484+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:53:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:37.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:37.049267+0000) 2022-09-08T06:53:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:37.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:37.003958+0000) 2022-09-08T06:53:37.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:37 smithi059 ceph-mon[97450]: pgmap v18004: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:53:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:37 smithi055 ceph-mon[97033]: pgmap v18004: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:53:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:38.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:38.049502+0000) 2022-09-08T06:53:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:38.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:38.004228+0000) 2022-09-08T06:53:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:39.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:39.049729+0000) 2022-09-08T06:53:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:39.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:39.004371+0000) 2022-09-08T06:53:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:39 smithi059 ceph-mon[97450]: pgmap v18005: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:53:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:39 smithi055 ceph-mon[97033]: pgmap v18005: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:53:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:40.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:40.049967+0000) 2022-09-08T06:53:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:40.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:40.004524+0000) 2022-09-08T06:53:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:41.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:41.050127+0000) 2022-09-08T06:53:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:41.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:41.004700+0000) 2022-09-08T06:53:41.538 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:41 smithi059 ceph-mon[97450]: pgmap v18006: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:53:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:41 smithi055 ceph-mon[97033]: pgmap v18006: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:53:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:42.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:42.050362+0000) 2022-09-08T06:53:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:42.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:42.004906+0000) 2022-09-08T06:53:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:43.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:43.050580+0000) 2022-09-08T06:53:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:43.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:43.005111+0000) 2022-09-08T06:53:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:43 smithi059 ceph-mon[97450]: pgmap v18007: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:53:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:43 smithi055 ceph-mon[97033]: pgmap v18007: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:53:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:44.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:44.050826+0000) 2022-09-08T06:53:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:44.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:44.005293+0000) 2022-09-08T06:53:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:45.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:45.051081+0000) 2022-09-08T06:53:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:45.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:45.005434+0000) 2022-09-08T06:53:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:45 smithi059 ceph-mon[97450]: pgmap v18008: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:53:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:45 smithi055 ceph-mon[97033]: pgmap v18008: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:53:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:46.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:46.051235+0000) 2022-09-08T06:53:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:46.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:46.005652+0000) 2022-09-08T06:53:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:47.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:47.051420+0000) 2022-09-08T06:53:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:47.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:47.005780+0000) 2022-09-08T06:53:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:47 smithi059 ceph-mon[97450]: pgmap v18009: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:53:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:47 smithi055 ceph-mon[97033]: pgmap v18009: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:53:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:48.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:48.051691+0000) 2022-09-08T06:53:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:48.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:48.005935+0000) 2022-09-08T06:53:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:49.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:49.051923+0000) 2022-09-08T06:53:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:49.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:49.006104+0000) 2022-09-08T06:53:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:49 smithi059 ceph-mon[97450]: pgmap v18010: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:53:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:49 smithi055 ceph-mon[97033]: pgmap v18010: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:53:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:50.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:50.052096+0000) 2022-09-08T06:53:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:50.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:50.006279+0000) 2022-09-08T06:53:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:51.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:51.052275+0000) 2022-09-08T06:53:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:51.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:51.006424+0000) 2022-09-08T06:53:51.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:51 smithi059 ceph-mon[97450]: pgmap v18011: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:53:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:51 smithi055 ceph-mon[97033]: pgmap v18011: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T06:53:51.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:53:51.497+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:53:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:52.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:52.052467+0000) 2022-09-08T06:53:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:52.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:52.006655+0000) 2022-09-08T06:53:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:53.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:53.052660+0000) 2022-09-08T06:53:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:53.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:53.006854+0000) 2022-09-08T06:53:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:53 smithi055 ceph-mon[97033]: pgmap v18012: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:53:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:53 smithi055 ceph-mon[97033]: pgmap v18013: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:53:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:53 smithi055 ceph-mon[97033]: pgmap v18014: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:53:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:53:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:53 smithi059 ceph-mon[97450]: pgmap v18012: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T06:53:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:53 smithi059 ceph-mon[97450]: pgmap v18013: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:53:53.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:53 smithi059 ceph-mon[97450]: pgmap v18014: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:53:53.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:53:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:54.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:54.052832+0000) 2022-09-08T06:53:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:54.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:54.007016+0000) 2022-09-08T06:53:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:55.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:55.053075+0000) 2022-09-08T06:53:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:55.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:55.007175+0000) 2022-09-08T06:53:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:55 smithi055 ceph-mon[97033]: pgmap v18015: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:53:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:55 smithi059 ceph-mon[97450]: pgmap v18015: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T06:53:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:56.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:56.053314+0000) 2022-09-08T06:53:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:56.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:56.007343+0000) 2022-09-08T06:53:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:57.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:57.053567+0000) 2022-09-08T06:53:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:57.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:57.007509+0000) 2022-09-08T06:53:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:57 smithi055 ceph-mon[97033]: pgmap v18016: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:53:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:57 smithi059 ceph-mon[97450]: pgmap v18016: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:53:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:58.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:58.053774+0000) 2022-09-08T06:53:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:58.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:58.007749+0000) 2022-09-08T06:53:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:53:59.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:59.053917+0000) 2022-09-08T06:53:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:53:59.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:53:59.007922+0000) 2022-09-08T06:53:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:53:59 smithi055 ceph-mon[97033]: pgmap v18017: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:53:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:53:59 smithi059 ceph-mon[97450]: pgmap v18017: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:54:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:00.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:00.008127+0000) 2022-09-08T06:54:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:00.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:00.054055+0000) 2022-09-08T06:54:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:01.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:01.054249+0000) 2022-09-08T06:54:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:01.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:01.008387+0000) 2022-09-08T06:54:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:01 smithi055 ceph-mon[97033]: pgmap v18018: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:54:01.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:01 smithi059 ceph-mon[97450]: pgmap v18018: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:54:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:02.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:02.008604+0000) 2022-09-08T06:54:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:02.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:02.054468+0000) 2022-09-08T06:54:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:03.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:03.054721+0000) 2022-09-08T06:54:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:03.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:03.008846+0000) 2022-09-08T06:54:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:03 smithi055 ceph-mon[97033]: pgmap v18019: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:54:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:03 smithi059 ceph-mon[97450]: pgmap v18019: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:54:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:04.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:04.054971+0000) 2022-09-08T06:54:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:04.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:04.009097+0000) 2022-09-08T06:54:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:05.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:05.055195+0000) 2022-09-08T06:54:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:05.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:05.009320+0000) 2022-09-08T06:54:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:05 smithi059 ceph-mon[97450]: pgmap v18020: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:05 smithi055 ceph-mon[97033]: pgmap v18020: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:06.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:06.055430+0000) 2022-09-08T06:54:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:06.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:06.009518+0000) 2022-09-08T06:54:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:54:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:54:06.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:54:06.507+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:54:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:07.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:07.055663+0000) 2022-09-08T06:54:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:07.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:07.009734+0000) 2022-09-08T06:54:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:07 smithi059 ceph-mon[97450]: pgmap v18021: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:54:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:54:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:07 smithi055 ceph-mon[97033]: pgmap v18021: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:54:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:54:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:08.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:08.055854+0000) 2022-09-08T06:54:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:08.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:08.009961+0000) 2022-09-08T06:54:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:09.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:09.010125+0000) 2022-09-08T06:54:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:09.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:09.056041+0000) 2022-09-08T06:54:09.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:09 smithi059 ceph-mon[97450]: pgmap v18022: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:54:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:09 smithi055 ceph-mon[97033]: pgmap v18022: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:54:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:10.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:10.056185+0000) 2022-09-08T06:54:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:10.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:10.010310+0000) 2022-09-08T06:54:10.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:10 smithi059 ceph-mon[97450]: osdmap e1204: 12 total, 11 up, 11 in 2022-09-08T06:54:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:10 smithi055 ceph-mon[97033]: osdmap e1204: 12 total, 11 up, 11 in 2022-09-08T06:54:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:11.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:11.056391+0000) 2022-09-08T06:54:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:11.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:11.010487+0000) 2022-09-08T06:54:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:11 smithi059 ceph-mon[97450]: pgmap v18024: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:54:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:11 smithi055 ceph-mon[97033]: pgmap v18024: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:54:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:12.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:12.010692+0000) 2022-09-08T06:54:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:12.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:12.056587+0000) 2022-09-08T06:54:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:13.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:13.056883+0000) 2022-09-08T06:54:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:13.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:13.010859+0000) 2022-09-08T06:54:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:13 smithi055 ceph-mon[97033]: pgmap v18025: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:54:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:13 smithi059 ceph-mon[97450]: pgmap v18025: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:54:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:14.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:14.057119+0000) 2022-09-08T06:54:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:14.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:14.011010+0000) 2022-09-08T06:54:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:15.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:15.057340+0000) 2022-09-08T06:54:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:15.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:15.011221+0000) 2022-09-08T06:54:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:15 smithi059 ceph-mon[97450]: pgmap v18026: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:54:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:15 smithi055 ceph-mon[97033]: pgmap v18026: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:54:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:16.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:16.057598+0000) 2022-09-08T06:54:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:16.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:16.011410+0000) 2022-09-08T06:54:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:17.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:17.011610+0000) 2022-09-08T06:54:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:17.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:17.057745+0000) 2022-09-08T06:54:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:17 smithi059 ceph-mon[97450]: pgmap v18027: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:54:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:17 smithi059 ceph-mon[97450]: osdmap e1205: 12 total, 11 up, 11 in 2022-09-08T06:54:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:17 smithi055 ceph-mon[97033]: pgmap v18027: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:54:17.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:17 smithi055 ceph-mon[97033]: osdmap e1205: 12 total, 11 up, 11 in 2022-09-08T06:54:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:18.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:18.057925+0000) 2022-09-08T06:54:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:18.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:18.011728+0000) 2022-09-08T06:54:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:19.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:19.058086+0000) 2022-09-08T06:54:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:19.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:19.011877+0000) 2022-09-08T06:54:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:19 smithi059 ceph-mon[97450]: pgmap v18029: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T06:54:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:19 smithi055 ceph-mon[97033]: pgmap v18029: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T06:54:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:20.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:20.058301+0000) 2022-09-08T06:54:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:20.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:20.012094+0000) 2022-09-08T06:54:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:21.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:21.058540+0000) 2022-09-08T06:54:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:21.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:21.012317+0000) 2022-09-08T06:54:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:21 smithi059 ceph-mon[97450]: pgmap v18030: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:54:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:21 smithi055 ceph-mon[97033]: pgmap v18030: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:54:21.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:54:21.524+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:54:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:22.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:22.058795+0000) 2022-09-08T06:54:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:22.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:22.012462+0000) 2022-09-08T06:54:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:23.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:23.058979+0000) 2022-09-08T06:54:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:23.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:23.012593+0000) 2022-09-08T06:54:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:23 smithi055 ceph-mon[97033]: pgmap v18031: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:54:23.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:23 smithi059 ceph-mon[97450]: pgmap v18031: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:54:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:24.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:24.012792+0000) 2022-09-08T06:54:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:24.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:24.059211+0000) 2022-09-08T06:54:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:25.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:25.059402+0000) 2022-09-08T06:54:25.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:25.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:25.012945+0000) 2022-09-08T06:54:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:25 smithi059 ceph-mon[97450]: pgmap v18032: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:54:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:25 smithi055 ceph-mon[97033]: pgmap v18032: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:54:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:26.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:26.059674+0000) 2022-09-08T06:54:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:26.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:26.013080+0000) 2022-09-08T06:54:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:27.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:27.059917+0000) 2022-09-08T06:54:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:27.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:27.013271+0000) 2022-09-08T06:54:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:27 smithi059 ceph-mon[97450]: pgmap v18033: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:54:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:27 smithi055 ceph-mon[97033]: pgmap v18033: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:54:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:28.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:28.060128+0000) 2022-09-08T06:54:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:28.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:28.013501+0000) 2022-09-08T06:54:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:29.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:29.060317+0000) 2022-09-08T06:54:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:29.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:29.013694+0000) 2022-09-08T06:54:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:29 smithi059 ceph-mon[97450]: pgmap v18034: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:54:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:29 smithi055 ceph-mon[97033]: pgmap v18034: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T06:54:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:30.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:30.060540+0000) 2022-09-08T06:54:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:30.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:30.013873+0000) 2022-09-08T06:54:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:31.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:31.060766+0000) 2022-09-08T06:54:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:31.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:31.014039+0000) 2022-09-08T06:54:31.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:31 smithi059 ceph-mon[97450]: pgmap v18035: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:54:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:31 smithi055 ceph-mon[97033]: pgmap v18035: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:54:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:32.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:32.061024+0000) 2022-09-08T06:54:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:32.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:32.014186+0000) 2022-09-08T06:54:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:33.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:33.061232+0000) 2022-09-08T06:54:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:33.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:33.014333+0000) 2022-09-08T06:54:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:33 smithi055 ceph-mon[97033]: pgmap v18036: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:33 smithi059 ceph-mon[97450]: pgmap v18036: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:34.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:34.061520+0000) 2022-09-08T06:54:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:34.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:34.014530+0000) 2022-09-08T06:54:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:35.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:35.061726+0000) 2022-09-08T06:54:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:35.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:35.014740+0000) 2022-09-08T06:54:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:35 smithi059 ceph-mon[97450]: pgmap v18037: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:35 smithi055 ceph-mon[97033]: pgmap v18037: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:36.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:36.061947+0000) 2022-09-08T06:54:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:36.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:36.014954+0000) 2022-09-08T06:54:36.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:54:36.539+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:54:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:37.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:37.062146+0000) 2022-09-08T06:54:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:37.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:37.015193+0000) 2022-09-08T06:54:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:37 smithi059 ceph-mon[97450]: pgmap v18038: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:54:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:37 smithi055 ceph-mon[97033]: pgmap v18038: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:54:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:38.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:38.062391+0000) 2022-09-08T06:54:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:38.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:38.015332+0000) 2022-09-08T06:54:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:39.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:39.062676+0000) 2022-09-08T06:54:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:39.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:39.015504+0000) 2022-09-08T06:54:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:39 smithi059 ceph-mon[97450]: pgmap v18039: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:39 smithi055 ceph-mon[97033]: pgmap v18039: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:40.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:40.062946+0000) 2022-09-08T06:54:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:40.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:40.015704+0000) 2022-09-08T06:54:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:41.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:41.063191+0000) 2022-09-08T06:54:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:41.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:41.015882+0000) 2022-09-08T06:54:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:41 smithi059 ceph-mon[97450]: pgmap v18040: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:54:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:41 smithi055 ceph-mon[97033]: pgmap v18040: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:54:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:42.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:42.063445+0000) 2022-09-08T06:54:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:42.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:42.016046+0000) 2022-09-08T06:54:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:43.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:43.063699+0000) 2022-09-08T06:54:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:43.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:43.016255+0000) 2022-09-08T06:54:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:43 smithi055 ceph-mon[97033]: pgmap v18041: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:43 smithi059 ceph-mon[97450]: pgmap v18041: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:44.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:44.063912+0000) 2022-09-08T06:54:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:44.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:44.016454+0000) 2022-09-08T06:54:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:45.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:45.064103+0000) 2022-09-08T06:54:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:45.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:45.016630+0000) 2022-09-08T06:54:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:45 smithi059 ceph-mon[97450]: pgmap v18042: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:45 smithi055 ceph-mon[97033]: pgmap v18042: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:46.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:46.064350+0000) 2022-09-08T06:54:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:46.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:46.016814+0000) 2022-09-08T06:54:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:47.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:47.064575+0000) 2022-09-08T06:54:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:47.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:47.017013+0000) 2022-09-08T06:54:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:47 smithi059 ceph-mon[97450]: pgmap v18043: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:54:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:47 smithi055 ceph-mon[97033]: pgmap v18043: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:54:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:48.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:48.064817+0000) 2022-09-08T06:54:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:48.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:48.017248+0000) 2022-09-08T06:54:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:49.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:49.064998+0000) 2022-09-08T06:54:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:49.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:49.017489+0000) 2022-09-08T06:54:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:49 smithi059 ceph-mon[97450]: pgmap v18044: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:49 smithi055 ceph-mon[97033]: pgmap v18044: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:50.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:50.065211+0000) 2022-09-08T06:54:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:50.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:50.017691+0000) 2022-09-08T06:54:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:51.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:51.065451+0000) 2022-09-08T06:54:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:51.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:51.017847+0000) 2022-09-08T06:54:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:51 smithi059 ceph-mon[97450]: pgmap v18045: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:54:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:51 smithi055 ceph-mon[97033]: pgmap v18045: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:54:51.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:54:51.548+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:54:52.347 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:52.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:52.065683+0000) 2022-09-08T06:54:52.348 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:52.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:52.018033+0000) 2022-09-08T06:54:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:53.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:53.065890+0000) 2022-09-08T06:54:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:53.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:53.018234+0000) 2022-09-08T06:54:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:53 smithi055 ceph-mon[97033]: pgmap v18046: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:54:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:54:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:54:53.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:54:53.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:54:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:53 smithi059 ceph-mon[97450]: pgmap v18046: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:53.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:54:53.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:54:53.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:54:53.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:54:53.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:54:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:54.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:54.066151+0000) 2022-09-08T06:54:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:54.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:54.018399+0000) 2022-09-08T06:54:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:55.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:55.066382+0000) 2022-09-08T06:54:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:55.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:55.018530+0000) 2022-09-08T06:54:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:55 smithi059 ceph-mon[97450]: pgmap v18047: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:54:55.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:54:55.592 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:55 smithi055 ceph-mon[97033]: pgmap v18047: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:55.592 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:54:55.593 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:54:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:56.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:56.066534+0000) 2022-09-08T06:54:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:56.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:56.018687+0000) 2022-09-08T06:54:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:54:56.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:54:56.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:56.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:56.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:56.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:56.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 158.3M 2022-09-08T06:54:56.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 166077042: error parsing value: Value '166077042' is below minimum 939524096 2022-09-08T06:54:56.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:56.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:56.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:56.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:56.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T06:54:56.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T06:54:56.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:56 smithi055 ceph-mon[97033]: pgmap v18048: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:54:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:54:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:54:57.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:57.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:57.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:57.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:57.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 158.3M 2022-09-08T06:54:57.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 166077042: error parsing value: Value '166077042' is below minimum 939524096 2022-09-08T06:54:57.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:57.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:57.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:57.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T06:54:57.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T06:54:57.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T06:54:57.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:56 smithi059 ceph-mon[97450]: pgmap v18048: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:54:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:57.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:57.066741+0000) 2022-09-08T06:54:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:57.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:57.018874+0000) 2022-09-08T06:54:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:58.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:58.066880+0000) 2022-09-08T06:54:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:58.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:58.019080+0000) 2022-09-08T06:54:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:54:59.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:59.067103+0000) 2022-09-08T06:54:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:54:59.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:54:59.019240+0000) 2022-09-08T06:54:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:54:59 smithi059 ceph-mon[97450]: pgmap v18049: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:54:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:54:59 smithi055 ceph-mon[97033]: pgmap v18049: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:00.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:00.067317+0000) 2022-09-08T06:55:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:00.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:00.019457+0000) 2022-09-08T06:55:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:01.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:01.067548+0000) 2022-09-08T06:55:01.436 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:01.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:01.019688+0000) 2022-09-08T06:55:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:01 smithi059 ceph-mon[97450]: pgmap v18050: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:55:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:01 smithi055 ceph-mon[97033]: pgmap v18050: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:55:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:02.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:02.067717+0000) 2022-09-08T06:55:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:02.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:02.019865+0000) 2022-09-08T06:55:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:03.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:03.067900+0000) 2022-09-08T06:55:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:03.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:03.019998+0000) 2022-09-08T06:55:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:03 smithi055 ceph-mon[97033]: pgmap v18051: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:03 smithi059 ceph-mon[97450]: pgmap v18051: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:04.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:04.068116+0000) 2022-09-08T06:55:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:04.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:04.020239+0000) 2022-09-08T06:55:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:05.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:05.068396+0000) 2022-09-08T06:55:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:05.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:05.020464+0000) 2022-09-08T06:55:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:05 smithi059 ceph-mon[97450]: pgmap v18052: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:05 smithi055 ceph-mon[97033]: pgmap v18052: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:06.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:06.068686+0000) 2022-09-08T06:55:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:06.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:06.020660+0000) 2022-09-08T06:55:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:55:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:55:06.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:55:06.563+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:55:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:07.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:07.068923+0000) 2022-09-08T06:55:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:07.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:07.020889+0000) 2022-09-08T06:55:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:07 smithi059 ceph-mon[97450]: pgmap v18053: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:55:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:55:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:07 smithi055 ceph-mon[97033]: pgmap v18053: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:55:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:55:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:08.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:08.021047+0000) 2022-09-08T06:55:08.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:08.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:08.069115+0000) 2022-09-08T06:55:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:09.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:09.069333+0000) 2022-09-08T06:55:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:09.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:09.021268+0000) 2022-09-08T06:55:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:09 smithi055 ceph-mon[97033]: pgmap v18054: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:55:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:09 smithi059 ceph-mon[97450]: pgmap v18054: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:55:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:10.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:10.069533+0000) 2022-09-08T06:55:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:10.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:10.021471+0000) 2022-09-08T06:55:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:10 smithi055 ceph-mon[97033]: osdmap e1206: 12 total, 11 up, 11 in 2022-09-08T06:55:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:10 smithi059 ceph-mon[97450]: osdmap e1206: 12 total, 11 up, 11 in 2022-09-08T06:55:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:11.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:11.069773+0000) 2022-09-08T06:55:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:11.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:11.021594+0000) 2022-09-08T06:55:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:11 smithi055 ceph-mon[97033]: pgmap v18056: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:55:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:11 smithi059 ceph-mon[97450]: pgmap v18056: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:55:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:12.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:12.069982+0000) 2022-09-08T06:55:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:12.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:12.021749+0000) 2022-09-08T06:55:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:13.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:13.070114+0000) 2022-09-08T06:55:13.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:13.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:13.021936+0000) 2022-09-08T06:55:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:13 smithi055 ceph-mon[97033]: pgmap v18057: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:55:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:13 smithi059 ceph-mon[97450]: pgmap v18057: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:55:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:14.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:14.070289+0000) 2022-09-08T06:55:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:14.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:14.022155+0000) 2022-09-08T06:55:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:15.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:15.070527+0000) 2022-09-08T06:55:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:15.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:15.022312+0000) 2022-09-08T06:55:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:15 smithi059 ceph-mon[97450]: pgmap v18058: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:55:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:15 smithi055 ceph-mon[97033]: pgmap v18058: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:55:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:16.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:16.070750+0000) 2022-09-08T06:55:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:16.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:16.022544+0000) 2022-09-08T06:55:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:17.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:17.070989+0000) 2022-09-08T06:55:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:17.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:17.022726+0000) 2022-09-08T06:55:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:17 smithi059 ceph-mon[97450]: pgmap v18059: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:55:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:17 smithi059 ceph-mon[97450]: osdmap e1207: 12 total, 11 up, 11 in 2022-09-08T06:55:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:17 smithi055 ceph-mon[97033]: pgmap v18059: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:55:17.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:17 smithi055 ceph-mon[97033]: osdmap e1207: 12 total, 11 up, 11 in 2022-09-08T06:55:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:18.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:18.071237+0000) 2022-09-08T06:55:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:18.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:18.022925+0000) 2022-09-08T06:55:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:19.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:19.071412+0000) 2022-09-08T06:55:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:19.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:19.023061+0000) 2022-09-08T06:55:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:19 smithi059 ceph-mon[97450]: pgmap v18061: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T06:55:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:19 smithi055 ceph-mon[97033]: pgmap v18061: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T06:55:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:20.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:20.071594+0000) 2022-09-08T06:55:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:20.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:20.023216+0000) 2022-09-08T06:55:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:21.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:21.071818+0000) 2022-09-08T06:55:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:21.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:21.023463+0000) 2022-09-08T06:55:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:21 smithi055 ceph-mon[97033]: pgmap v18062: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:55:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:21 smithi059 ceph-mon[97450]: pgmap v18062: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:55:21.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:55:21.573+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:55:22.037 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:22.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:22.023627+0000) 2022-09-08T06:55:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:22.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:22.071979+0000) 2022-09-08T06:55:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:23.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:23.072203+0000) 2022-09-08T06:55:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:23.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:23.023796+0000) 2022-09-08T06:55:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:23 smithi055 ceph-mon[97033]: pgmap v18063: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:55:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:23 smithi059 ceph-mon[97450]: pgmap v18063: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:55:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:24.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:24.023937+0000) 2022-09-08T06:55:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:24.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:24.072436+0000) 2022-09-08T06:55:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:25.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:25.024115+0000) 2022-09-08T06:55:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:25.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:25.072706+0000) 2022-09-08T06:55:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:25 smithi059 ceph-mon[97450]: pgmap v18064: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:55:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:25 smithi055 ceph-mon[97033]: pgmap v18064: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:55:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:26.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:26.072838+0000) 2022-09-08T06:55:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:26.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:26.024274+0000) 2022-09-08T06:55:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:27.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:27.072982+0000) 2022-09-08T06:55:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:27.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:27.024461+0000) 2022-09-08T06:55:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:27 smithi055 ceph-mon[97033]: pgmap v18065: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:55:27.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:55:27.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:55:27.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:55:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:27 smithi059 ceph-mon[97450]: pgmap v18065: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:55:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:55:27.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:55:27.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:55:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:28.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:28.073186+0000) 2022-09-08T06:55:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:28.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:28.024729+0000) 2022-09-08T06:55:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:28 smithi059 ceph-mon[97450]: pgmap v18066: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 6 op/s 2022-09-08T06:55:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:28 smithi055 ceph-mon[97033]: pgmap v18066: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 6 op/s 2022-09-08T06:55:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:29.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:29.073426+0000) 2022-09-08T06:55:29.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:29.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:29.024920+0000) 2022-09-08T06:55:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:30.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:30.073698+0000) 2022-09-08T06:55:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:30.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:30.025091+0000) 2022-09-08T06:55:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:30 smithi055 ceph-mon[97033]: pgmap v18067: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:55:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:30 smithi059 ceph-mon[97450]: pgmap v18067: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:55:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:31.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:31.073838+0000) 2022-09-08T06:55:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:31.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:31.025265+0000) 2022-09-08T06:55:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:32.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:32.074033+0000) 2022-09-08T06:55:32.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:32.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:32.025441+0000) 2022-09-08T06:55:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:32 smithi055 ceph-mon[97033]: pgmap v18068: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:55:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:32 smithi059 ceph-mon[97450]: pgmap v18068: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:55:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:33.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:33.074250+0000) 2022-09-08T06:55:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:33.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:33.025679+0000) 2022-09-08T06:55:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:34.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:34.074446+0000) 2022-09-08T06:55:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:34.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:34.025871+0000) 2022-09-08T06:55:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:34 smithi055 ceph-mon[97033]: pgmap v18069: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:55:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:34 smithi059 ceph-mon[97450]: pgmap v18069: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:55:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:35.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:35.074705+0000) 2022-09-08T06:55:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:35.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:35.026083+0000) 2022-09-08T06:55:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:36.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:36.074926+0000) 2022-09-08T06:55:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:36.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:36.026252+0000) 2022-09-08T06:55:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:36 smithi055 ceph-mon[97033]: pgmap v18070: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T06:55:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:36 smithi059 ceph-mon[97450]: pgmap v18070: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T06:55:36.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:55:36.588+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:55:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:37.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:37.026447+0000) 2022-09-08T06:55:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:37.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:37.075147+0000) 2022-09-08T06:55:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:38.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:38.075387+0000) 2022-09-08T06:55:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:38.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:38.026608+0000) 2022-09-08T06:55:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:38 smithi055 ceph-mon[97033]: pgmap v18071: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:55:38.781 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:38 smithi059 ceph-mon[97450]: pgmap v18071: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T06:55:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:39.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:39.026791+0000) 2022-09-08T06:55:39.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:39.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:39.075692+0000) 2022-09-08T06:55:40.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:40.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:40.026952+0000) 2022-09-08T06:55:40.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:40.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:40.075865+0000) 2022-09-08T06:55:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:40 smithi055 ceph-mon[97033]: pgmap v18072: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:40 smithi059 ceph-mon[97450]: pgmap v18072: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:41.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:41.076084+0000) 2022-09-08T06:55:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:41.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:41.027181+0000) 2022-09-08T06:55:42.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:42.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:42.027408+0000) 2022-09-08T06:55:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:42.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:42.076259+0000) 2022-09-08T06:55:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:42 smithi055 ceph-mon[97033]: pgmap v18073: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:55:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:42 smithi059 ceph-mon[97450]: pgmap v18073: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:55:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:43.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:43.076519+0000) 2022-09-08T06:55:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:43.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:43.027615+0000) 2022-09-08T06:55:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:44.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:44.076745+0000) 2022-09-08T06:55:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:44.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:44.027903+0000) 2022-09-08T06:55:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:44 smithi055 ceph-mon[97033]: pgmap v18074: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:44 smithi059 ceph-mon[97450]: pgmap v18074: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:45.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:45.076972+0000) 2022-09-08T06:55:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:45.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:45.028105+0000) 2022-09-08T06:55:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:46.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:46.077178+0000) 2022-09-08T06:55:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:46.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:46.028332+0000) 2022-09-08T06:55:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:46 smithi055 ceph-mon[97033]: pgmap v18075: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:55:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:46 smithi059 ceph-mon[97450]: pgmap v18075: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:55:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:47.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:47.077400+0000) 2022-09-08T06:55:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:47.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:47.028543+0000) 2022-09-08T06:55:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:48.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:48.077659+0000) 2022-09-08T06:55:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:48.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:48.028749+0000) 2022-09-08T06:55:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:48 smithi055 ceph-mon[97033]: pgmap v18076: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:48 smithi059 ceph-mon[97450]: pgmap v18076: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:49.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:49.028977+0000) 2022-09-08T06:55:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:49.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:49.077869+0000) 2022-09-08T06:55:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:50.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:50.078106+0000) 2022-09-08T06:55:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:50.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:50.029228+0000) 2022-09-08T06:55:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:50 smithi055 ceph-mon[97033]: pgmap v18077: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:50.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:50 smithi059 ceph-mon[97450]: pgmap v18077: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:51.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:51.078335+0000) 2022-09-08T06:55:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:51.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:51.029467+0000) 2022-09-08T06:55:51.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:55:51.604+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:55:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:52.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:52.029700+0000) 2022-09-08T06:55:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:52.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:52.078575+0000) 2022-09-08T06:55:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:52 smithi055 ceph-mon[97033]: pgmap v18078: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:55:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:52 smithi059 ceph-mon[97450]: pgmap v18078: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:55:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:53.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:53.078715+0000) 2022-09-08T06:55:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:53.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:53.029906+0000) 2022-09-08T06:55:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:54.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:54.030083+0000) 2022-09-08T06:55:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:54.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:54.078954+0000) 2022-09-08T06:55:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:54 smithi055 ceph-mon[97033]: pgmap v18079: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:54 smithi059 ceph-mon[97450]: pgmap v18079: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:55.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:55.079132+0000) 2022-09-08T06:55:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:55.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:55.030313+0000) 2022-09-08T06:55:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:56.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:56.079319+0000) 2022-09-08T06:55:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:56.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:56.030554+0000) 2022-09-08T06:55:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:56 smithi055 ceph-mon[97033]: pgmap v18080: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:55:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:56 smithi059 ceph-mon[97450]: pgmap v18080: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:55:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:57.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:57.030795+0000) 2022-09-08T06:55:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:57.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:57.079542+0000) 2022-09-08T06:55:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:58.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:58.079764+0000) 2022-09-08T06:55:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:58.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:58.030982+0000) 2022-09-08T06:55:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:55:58 smithi055 ceph-mon[97033]: pgmap v18081: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:55:58 smithi059 ceph-mon[97450]: pgmap v18081: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:55:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:55:59.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:59.080010+0000) 2022-09-08T06:55:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:55:59.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:55:59.031249+0000) 2022-09-08T06:56:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:00.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:00.080145+0000) 2022-09-08T06:56:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:00.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:00.031463+0000) 2022-09-08T06:56:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:00 smithi055 ceph-mon[97033]: pgmap v18082: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:56:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:00 smithi059 ceph-mon[97450]: pgmap v18082: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:56:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:01.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:01.031620+0000) 2022-09-08T06:56:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:01.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:01.080357+0000) 2022-09-08T06:56:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:02.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:02.080559+0000) 2022-09-08T06:56:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:02.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:02.031738+0000) 2022-09-08T06:56:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:02 smithi055 ceph-mon[97033]: pgmap v18083: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:56:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:02 smithi059 ceph-mon[97450]: pgmap v18083: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:56:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:03.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:03.080790+0000) 2022-09-08T06:56:03.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:03.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:03.031953+0000) 2022-09-08T06:56:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:04.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:04.080941+0000) 2022-09-08T06:56:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:04.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:04.032169+0000) 2022-09-08T06:56:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:04 smithi055 ceph-mon[97033]: pgmap v18084: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:56:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:04 smithi059 ceph-mon[97450]: pgmap v18084: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:56:05.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:05.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:05.081138+0000) 2022-09-08T06:56:05.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:05.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:05.032402+0000) 2022-09-08T06:56:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:06.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:06.081360+0000) 2022-09-08T06:56:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:06.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:06.032630+0000) 2022-09-08T06:56:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:06 smithi055 ceph-mon[97033]: pgmap v18085: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:56:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:56:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:06 smithi059 ceph-mon[97450]: pgmap v18085: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:56:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:56:06.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:56:06.617+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:56:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:07.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:07.032849+0000) 2022-09-08T06:56:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:07.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:07.081604+0000) 2022-09-08T06:56:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:56:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:56:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:08.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:08.081782+0000) 2022-09-08T06:56:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:08.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:08.033047+0000) 2022-09-08T06:56:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:08 smithi055 ceph-mon[97033]: pgmap v18086: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:56:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:08 smithi059 ceph-mon[97450]: pgmap v18086: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:56:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:09.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:09.082000+0000) 2022-09-08T06:56:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:09.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:09.033266+0000) 2022-09-08T06:56:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:10.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:10.082210+0000) 2022-09-08T06:56:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:10.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:10.033513+0000) 2022-09-08T06:56:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:10 smithi055 ceph-mon[97033]: pgmap v18087: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:56:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:10 smithi055 ceph-mon[97033]: osdmap e1208: 12 total, 11 up, 11 in 2022-09-08T06:56:10.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:10 smithi059 ceph-mon[97450]: pgmap v18087: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:56:10.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:10 smithi059 ceph-mon[97450]: osdmap e1208: 12 total, 11 up, 11 in 2022-09-08T06:56:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:11.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:11.082434+0000) 2022-09-08T06:56:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:11.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:11.033715+0000) 2022-09-08T06:56:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:12.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:12.082717+0000) 2022-09-08T06:56:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:12.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:12.033961+0000) 2022-09-08T06:56:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:12 smithi055 ceph-mon[97033]: pgmap v18089: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:56:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:12 smithi059 ceph-mon[97450]: pgmap v18089: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:56:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:13.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:13.082917+0000) 2022-09-08T06:56:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:13.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:13.034204+0000) 2022-09-08T06:56:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:14.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:14.083138+0000) 2022-09-08T06:56:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:14.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:14.034378+0000) 2022-09-08T06:56:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:14 smithi055 ceph-mon[97033]: pgmap v18090: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:56:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:14 smithi059 ceph-mon[97450]: pgmap v18090: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T06:56:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:15.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:15.083355+0000) 2022-09-08T06:56:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:15.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:15.034511+0000) 2022-09-08T06:56:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:16.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:16.083562+0000) 2022-09-08T06:56:16.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:16.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:16.034728+0000) 2022-09-08T06:56:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:16 smithi055 ceph-mon[97033]: pgmap v18091: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:56:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:16 smithi059 ceph-mon[97450]: pgmap v18091: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:56:17.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:17.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:17.034924+0000) 2022-09-08T06:56:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:17.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:17.083792+0000) 2022-09-08T06:56:18.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:18.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:18.035079+0000) 2022-09-08T06:56:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:18.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:18.084013+0000) 2022-09-08T06:56:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:18 smithi055 ceph-mon[97033]: pgmap v18092: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:56:18.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:18 smithi055 ceph-mon[97033]: osdmap e1209: 12 total, 11 up, 11 in 2022-09-08T06:56:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:18 smithi059 ceph-mon[97450]: pgmap v18092: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:56:18.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:18 smithi059 ceph-mon[97450]: osdmap e1209: 12 total, 11 up, 11 in 2022-09-08T06:56:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:19.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:19.084275+0000) 2022-09-08T06:56:19.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:19.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:19.035253+0000) 2022-09-08T06:56:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:20.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:20.084458+0000) 2022-09-08T06:56:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:20.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:20.035403+0000) 2022-09-08T06:56:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:20 smithi055 ceph-mon[97033]: pgmap v18094: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 211 B/s wr, 6 op/s 2022-09-08T06:56:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:20 smithi059 ceph-mon[97450]: pgmap v18094: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 211 B/s wr, 6 op/s 2022-09-08T06:56:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:21.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:21.084665+0000) 2022-09-08T06:56:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:21.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:21.035555+0000) 2022-09-08T06:56:21.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:56:21.632+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:56:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:22.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:22.084915+0000) 2022-09-08T06:56:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:22.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:22.035789+0000) 2022-09-08T06:56:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:22 smithi055 ceph-mon[97033]: pgmap v18095: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:56:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:22 smithi059 ceph-mon[97450]: pgmap v18095: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:56:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:23.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:23.085109+0000) 2022-09-08T06:56:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:23.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:23.035960+0000) 2022-09-08T06:56:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:24.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:24.085347+0000) 2022-09-08T06:56:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:24.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:24.036116+0000) 2022-09-08T06:56:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:24 smithi055 ceph-mon[97033]: pgmap v18096: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:56:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:24 smithi059 ceph-mon[97450]: pgmap v18096: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:56:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:25.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:25.085548+0000) 2022-09-08T06:56:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:25.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:25.036244+0000) 2022-09-08T06:56:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:26.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:26.085772+0000) 2022-09-08T06:56:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:26.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:26.036398+0000) 2022-09-08T06:56:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:26 smithi055 ceph-mon[97033]: pgmap v18097: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:56:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:26 smithi059 ceph-mon[97450]: pgmap v18097: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:56:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:27.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:27.085997+0000) 2022-09-08T06:56:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:27.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:27.036607+0000) 2022-09-08T06:56:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:56:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:56:27.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:56:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:56:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:56:27.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:56:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:28.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:28.086220+0000) 2022-09-08T06:56:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:28.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:28.036752+0000) 2022-09-08T06:56:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:28 smithi055 ceph-mon[97033]: pgmap v18098: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:56:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:28.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:28.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:28.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:28.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:28 smithi059 ceph-mon[97450]: pgmap v18098: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:56:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:28.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:28.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:29.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:29.086465+0000) 2022-09-08T06:56:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:29.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:29.036989+0000) 2022-09-08T06:56:29.701 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:29.702 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:29.728 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:29.729 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:30.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:30.086651+0000) 2022-09-08T06:56:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:30.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:30.037163+0000) 2022-09-08T06:56:30.653 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:30 smithi055 ceph-mon[97033]: pgmap v18099: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T06:56:30.970 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:30 smithi059 ceph-mon[97450]: pgmap v18099: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T06:56:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:31.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:31.086823+0000) 2022-09-08T06:56:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:31.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:31.037353+0000) 2022-09-08T06:56:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:31 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T06:56:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:31.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:31.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:31 smithi059 ceph-mon[97450]: pgmap v18100: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:56:31.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:31 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T06:56:31.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:31.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:31.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:31 smithi059 ceph-mon[97450]: pgmap v18101: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:56:31.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:31 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T06:56:31.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:31.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:31.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:31 smithi055 ceph-mon[97033]: pgmap v18100: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:56:31.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:31 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T06:56:31.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:31.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:31.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:31 smithi055 ceph-mon[97033]: pgmap v18101: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:56:31.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:56:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:32.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:32.087066+0000) 2022-09-08T06:56:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:32.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:32.037611+0000) 2022-09-08T06:56:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:32 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T06:56:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:32 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T06:56:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:33.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:33.087259+0000) 2022-09-08T06:56:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:33.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:33.037864+0000) 2022-09-08T06:56:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:33 smithi055 ceph-mon[97033]: pgmap v18102: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:56:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:33 smithi059 ceph-mon[97450]: pgmap v18102: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:56:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:34.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:34.087446+0000) 2022-09-08T06:56:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:34.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:34.038098+0000) 2022-09-08T06:56:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:35.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:35.087582+0000) 2022-09-08T06:56:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:35.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:35.038298+0000) 2022-09-08T06:56:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:35 smithi055 ceph-mon[97033]: pgmap v18103: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:56:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:35 smithi059 ceph-mon[97450]: pgmap v18103: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:56:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:36.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:36.087739+0000) 2022-09-08T06:56:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:36.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:36.038531+0000) 2022-09-08T06:56:36.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:56:36.639+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:56:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:37.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:37.087932+0000) 2022-09-08T06:56:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:37.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:37.038773+0000) 2022-09-08T06:56:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:38.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:38.088223+0000) 2022-09-08T06:56:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:38.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:38.038988+0000) 2022-09-08T06:56:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:38 smithi055 ceph-mon[97033]: pgmap v18104: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:56:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:38 smithi059 ceph-mon[97450]: pgmap v18104: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:56:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:39.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:39.039177+0000) 2022-09-08T06:56:39.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:39.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:39.088505+0000) 2022-09-08T06:56:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:39 smithi055 ceph-mon[97033]: pgmap v18105: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:56:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:39 smithi059 ceph-mon[97450]: pgmap v18105: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:56:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:40.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:40.088788+0000) 2022-09-08T06:56:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:40.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:40.039408+0000) 2022-09-08T06:56:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:41.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:41.088960+0000) 2022-09-08T06:56:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:41.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:41.039676+0000) 2022-09-08T06:56:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:41 smithi055 ceph-mon[97033]: pgmap v18106: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:56:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:41 smithi059 ceph-mon[97450]: pgmap v18106: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:56:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:42.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:42.089236+0000) 2022-09-08T06:56:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:42.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:42.039814+0000) 2022-09-08T06:56:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:43.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:43.089523+0000) 2022-09-08T06:56:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:43.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:43.040036+0000) 2022-09-08T06:56:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:43 smithi055 ceph-mon[97033]: pgmap v18107: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:56:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:43 smithi059 ceph-mon[97450]: pgmap v18107: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:56:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:44.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:44.089749+0000) 2022-09-08T06:56:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:44.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:44.040300+0000) 2022-09-08T06:56:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:45.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:45.090002+0000) 2022-09-08T06:56:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:45.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:45.040547+0000) 2022-09-08T06:56:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:45 smithi055 ceph-mon[97033]: pgmap v18108: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:56:46.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:45 smithi059 ceph-mon[97450]: pgmap v18108: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:56:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:46.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:46.090190+0000) 2022-09-08T06:56:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:46.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:46.040798+0000) 2022-09-08T06:56:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:47.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:47.090379+0000) 2022-09-08T06:56:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:47.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:47.040956+0000) 2022-09-08T06:56:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:48.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:48.090587+0000) 2022-09-08T06:56:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:48.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:48.041154+0000) 2022-09-08T06:56:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:48 smithi055 ceph-mon[97033]: pgmap v18109: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:56:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:48 smithi059 ceph-mon[97450]: pgmap v18109: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:56:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:49.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:49.090809+0000) 2022-09-08T06:56:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:49.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:49.041355+0000) 2022-09-08T06:56:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:49 smithi055 ceph-mon[97033]: pgmap v18110: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:56:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:49 smithi059 ceph-mon[97450]: pgmap v18110: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:56:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:50.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:50.091020+0000) 2022-09-08T06:56:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:50.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:50.041617+0000) 2022-09-08T06:56:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:51.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:51.091209+0000) 2022-09-08T06:56:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:51.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:51.041862+0000) 2022-09-08T06:56:51.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:56:51.649+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:56:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:51 smithi055 ceph-mon[97033]: pgmap v18111: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:56:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:51 smithi059 ceph-mon[97450]: pgmap v18111: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:56:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:52.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:52.091455+0000) 2022-09-08T06:56:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:52.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:52.042046+0000) 2022-09-08T06:56:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:53.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:53.091647+0000) 2022-09-08T06:56:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:53.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:53.042245+0000) 2022-09-08T06:56:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:53 smithi055 ceph-mon[97033]: pgmap v18112: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:56:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:53 smithi059 ceph-mon[97450]: pgmap v18112: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:56:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:54.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:54.091870+0000) 2022-09-08T06:56:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:54.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:54.042399+0000) 2022-09-08T06:56:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:55.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:55.092123+0000) 2022-09-08T06:56:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:55.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:55.042576+0000) 2022-09-08T06:56:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:55 smithi055 ceph-mon[97033]: pgmap v18113: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:56:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:55 smithi059 ceph-mon[97450]: pgmap v18113: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:56:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:56.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:56.092344+0000) 2022-09-08T06:56:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:56.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:56.042727+0000) 2022-09-08T06:56:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:57.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:57.092587+0000) 2022-09-08T06:56:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:57.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:57.042963+0000) 2022-09-08T06:56:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:58.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:58.092809+0000) 2022-09-08T06:56:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:58.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:58.043182+0000) 2022-09-08T06:56:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:58 smithi055 ceph-mon[97033]: pgmap v18114: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:56:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:58 smithi059 ceph-mon[97450]: pgmap v18114: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:56:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:56:59.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:59.093056+0000) 2022-09-08T06:56:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:56:59.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:56:59.043425+0000) 2022-09-08T06:56:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:56:59 smithi055 ceph-mon[97033]: pgmap v18115: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:57:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:56:59 smithi059 ceph-mon[97450]: pgmap v18115: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:57:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:00.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:00.093211+0000) 2022-09-08T06:57:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:00.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:00.043699+0000) 2022-09-08T06:57:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:01.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:01.093351+0000) 2022-09-08T06:57:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:01.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:01.043866+0000) 2022-09-08T06:57:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:01 smithi055 ceph-mon[97033]: pgmap v18116: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:57:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:01 smithi059 ceph-mon[97450]: pgmap v18116: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:57:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:02.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:02.093577+0000) 2022-09-08T06:57:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:02.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:02.044084+0000) 2022-09-08T06:57:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:03.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:03.093810+0000) 2022-09-08T06:57:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:03.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:03.044305+0000) 2022-09-08T06:57:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:03 smithi055 ceph-mon[97033]: pgmap v18117: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:57:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:03 smithi059 ceph-mon[97450]: pgmap v18117: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:57:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:04.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:04.094000+0000) 2022-09-08T06:57:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:04.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:04.044556+0000) 2022-09-08T06:57:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:05.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:05.044720+0000) 2022-09-08T06:57:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:05.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:05.094154+0000) 2022-09-08T06:57:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:05 smithi055 ceph-mon[97033]: pgmap v18118: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:57:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:05 smithi059 ceph-mon[97450]: pgmap v18118: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:57:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:06.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:06.094384+0000) 2022-09-08T06:57:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:06.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:06.044862+0000) 2022-09-08T06:57:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:57:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:57:07.047 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:57:06.657+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:57:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:07.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:07.045079+0000) 2022-09-08T06:57:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:07.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:07.094604+0000) 2022-09-08T06:57:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:57:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:07 smithi055 ceph-mon[97033]: pgmap v18119: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:57:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:57:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:07 smithi059 ceph-mon[97450]: pgmap v18119: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:57:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:08.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:08.094782+0000) 2022-09-08T06:57:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:08.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:08.045241+0000) 2022-09-08T06:57:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:09.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:09.094981+0000) 2022-09-08T06:57:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:09.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:09.045413+0000) 2022-09-08T06:57:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:10.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:10.095143+0000) 2022-09-08T06:57:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:10.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:10.045589+0000) 2022-09-08T06:57:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:10 smithi055 ceph-mon[97033]: osdmap e1210: 12 total, 11 up, 11 in 2022-09-08T06:57:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:10 smithi055 ceph-mon[97033]: pgmap v18121: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:57:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:10 smithi059 ceph-mon[97450]: osdmap e1210: 12 total, 11 up, 11 in 2022-09-08T06:57:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:10 smithi059 ceph-mon[97450]: pgmap v18121: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:57:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:11.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:11.045750+0000) 2022-09-08T06:57:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:11.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:11.095318+0000) 2022-09-08T06:57:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:11 smithi055 ceph-mon[97033]: pgmap v18122: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:57:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:11 smithi059 ceph-mon[97450]: pgmap v18122: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:57:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:12.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:12.095576+0000) 2022-09-08T06:57:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:12.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:12.045939+0000) 2022-09-08T06:57:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:13.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:13.095783+0000) 2022-09-08T06:57:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:13.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:13.046162+0000) 2022-09-08T06:57:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:13 smithi055 ceph-mon[97033]: osdmap e1211: 12 total, 11 up, 11 in 2022-09-08T06:57:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:13 smithi059 ceph-mon[97450]: osdmap e1211: 12 total, 11 up, 11 in 2022-09-08T06:57:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:14.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:14.046334+0000) 2022-09-08T06:57:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:14.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:14.096041+0000) 2022-09-08T06:57:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:14 smithi055 ceph-mon[97033]: pgmap v18124: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:57:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:14 smithi059 ceph-mon[97450]: pgmap v18124: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T06:57:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:15.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:15.046488+0000) 2022-09-08T06:57:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:15.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:15.096247+0000) 2022-09-08T06:57:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:15 smithi055 ceph-mon[97033]: pgmap v18125: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:57:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:15 smithi059 ceph-mon[97450]: pgmap v18125: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T06:57:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:16.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:16.096363+0000) 2022-09-08T06:57:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:16.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:16.046690+0000) 2022-09-08T06:57:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:17.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:17.096599+0000) 2022-09-08T06:57:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:17.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:17.046886+0000) 2022-09-08T06:57:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:18.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:18.096842+0000) 2022-09-08T06:57:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:18.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:18.047033+0000) 2022-09-08T06:57:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:18 smithi055 ceph-mon[97033]: pgmap v18126: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T06:57:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:18 smithi059 ceph-mon[97450]: pgmap v18126: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T06:57:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:19.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:19.097016+0000) 2022-09-08T06:57:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:19.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:19.047196+0000) 2022-09-08T06:57:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:19 smithi055 ceph-mon[97033]: pgmap v18127: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:57:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:19 smithi059 ceph-mon[97450]: pgmap v18127: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:57:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:20.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:20.097193+0000) 2022-09-08T06:57:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:20.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:20.047457+0000) 2022-09-08T06:57:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:21.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:21.097416+0000) 2022-09-08T06:57:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:21.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:21.047694+0000) 2022-09-08T06:57:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:21 smithi055 ceph-mon[97033]: pgmap v18128: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:57:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:21 smithi059 ceph-mon[97450]: pgmap v18128: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:57:22.050 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:57:21.666+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:57:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:22.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:22.097713+0000) 2022-09-08T06:57:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:22.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:22.047937+0000) 2022-09-08T06:57:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:23.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:23.097914+0000) 2022-09-08T06:57:23.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:23.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:23.048138+0000) 2022-09-08T06:57:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:23 smithi055 ceph-mon[97033]: pgmap v18129: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-08T06:57:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:23 smithi059 ceph-mon[97450]: pgmap v18129: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-08T06:57:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:24.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:24.098110+0000) 2022-09-08T06:57:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:24.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:24.048384+0000) 2022-09-08T06:57:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:25.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:25.098322+0000) 2022-09-08T06:57:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:25.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:25.048666+0000) 2022-09-08T06:57:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:25 smithi055 ceph-mon[97033]: pgmap v18130: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:57:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:25 smithi059 ceph-mon[97450]: pgmap v18130: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T06:57:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:26.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:26.098514+0000) 2022-09-08T06:57:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:26.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:26.048926+0000) 2022-09-08T06:57:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:27.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:27.098693+0000) 2022-09-08T06:57:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:27.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:27.049162+0000) 2022-09-08T06:57:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:28.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:28.098863+0000) 2022-09-08T06:57:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:28.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:28.049416+0000) 2022-09-08T06:57:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:28 smithi055 ceph-mon[97033]: pgmap v18131: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:57:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:28 smithi059 ceph-mon[97450]: pgmap v18131: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:57:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:29.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:29.099059+0000) 2022-09-08T06:57:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:29.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:29.049692+0000) 2022-09-08T06:57:29.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:29 smithi055 ceph-mon[97033]: pgmap v18132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:57:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:29 smithi059 ceph-mon[97450]: pgmap v18132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:57:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:30.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:30.099281+0000) 2022-09-08T06:57:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:30.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:30.049882+0000) 2022-09-08T06:57:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:31.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:31.099535+0000) 2022-09-08T06:57:31.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:31.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:31.050025+0000) 2022-09-08T06:57:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:31 smithi055 ceph-mon[97033]: pgmap v18133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:57:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:31 smithi059 ceph-mon[97450]: pgmap v18133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:57:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:32.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:32.099766+0000) 2022-09-08T06:57:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:32.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:32.050278+0000) 2022-09-08T06:57:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:57:32.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:57:32.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:57:32.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:57:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:57:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:57:33.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:57:33.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:57:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:33.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:33.099953+0000) 2022-09-08T06:57:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:33.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:33.050511+0000) 2022-09-08T06:57:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:33 smithi055 ceph-mon[97033]: pgmap v18134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:57:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:33 smithi059 ceph-mon[97450]: pgmap v18134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:57:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:34.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:34.100175+0000) 2022-09-08T06:57:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:34.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:34.050749+0000) 2022-09-08T06:57:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:35.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:35.100316+0000) 2022-09-08T06:57:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:35.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:35.050933+0000) 2022-09-08T06:57:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:35 smithi055 ceph-mon[97033]: pgmap v18135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:57:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:35 smithi059 ceph-mon[97450]: pgmap v18135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:57:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:36.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:36.100557+0000) 2022-09-08T06:57:36.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:36.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:36.051097+0000) 2022-09-08T06:57:37.053 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:57:36.680+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:57:37.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:37.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:37.051320+0000) 2022-09-08T06:57:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:37.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:37.100787+0000) 2022-09-08T06:57:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:38.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:38.101017+0000) 2022-09-08T06:57:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:38.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:38.051524+0000) 2022-09-08T06:57:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:38 smithi055 ceph-mon[97033]: pgmap v18136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:57:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:38 smithi059 ceph-mon[97450]: pgmap v18136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:57:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:39.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:39.101254+0000) 2022-09-08T06:57:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:39.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:39.051696+0000) 2022-09-08T06:57:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:39 smithi055 ceph-mon[97033]: pgmap v18137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:57:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:39 smithi059 ceph-mon[97450]: pgmap v18137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:57:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:40.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:40.101482+0000) 2022-09-08T06:57:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:40.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:40.051936+0000) 2022-09-08T06:57:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:41.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:41.101675+0000) 2022-09-08T06:57:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:41.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:41.052111+0000) 2022-09-08T06:57:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:41 smithi055 ceph-mon[97033]: pgmap v18138: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:57:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:41 smithi059 ceph-mon[97450]: pgmap v18138: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:57:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:42.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:42.101886+0000) 2022-09-08T06:57:42.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:42.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:42.052280+0000) 2022-09-08T06:57:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:43.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:43.102057+0000) 2022-09-08T06:57:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:43.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:43.052513+0000) 2022-09-08T06:57:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:43 smithi055 ceph-mon[97033]: pgmap v18139: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:57:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:43 smithi059 ceph-mon[97450]: pgmap v18139: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:57:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:44.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:44.102209+0000) 2022-09-08T06:57:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:44.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:44.052682+0000) 2022-09-08T06:57:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:45.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:45.102456+0000) 2022-09-08T06:57:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:45.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:45.052920+0000) 2022-09-08T06:57:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:45 smithi055 ceph-mon[97033]: pgmap v18140: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:57:46.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:45 smithi059 ceph-mon[97450]: pgmap v18140: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:57:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:46.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:46.102746+0000) 2022-09-08T06:57:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:46.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:46.053212+0000) 2022-09-08T06:57:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:47.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:47.102983+0000) 2022-09-08T06:57:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:47.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:47.053437+0000) 2022-09-08T06:57:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:48.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:48.053600+0000) 2022-09-08T06:57:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:48.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:48.103180+0000) 2022-09-08T06:57:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:48 smithi055 ceph-mon[97033]: pgmap v18141: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:57:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:48 smithi059 ceph-mon[97450]: pgmap v18141: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:57:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:49.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:49.103424+0000) 2022-09-08T06:57:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:49.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:49.053714+0000) 2022-09-08T06:57:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:49 smithi055 ceph-mon[97033]: pgmap v18142: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:57:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:49 smithi059 ceph-mon[97450]: pgmap v18142: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:57:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:50.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:50.103681+0000) 2022-09-08T06:57:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:50.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:50.053854+0000) 2022-09-08T06:57:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:51.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:51.103898+0000) 2022-09-08T06:57:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:51.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:51.053998+0000) 2022-09-08T06:57:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:51 smithi055 ceph-mon[97033]: pgmap v18143: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:57:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:51 smithi059 ceph-mon[97450]: pgmap v18143: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:57:52.056 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:57:51.687+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:57:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:52.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:52.104133+0000) 2022-09-08T06:57:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:52.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:52.054132+0000) 2022-09-08T06:57:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:53.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:53.104271+0000) 2022-09-08T06:57:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:53.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:53.054316+0000) 2022-09-08T06:57:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:53 smithi055 ceph-mon[97033]: pgmap v18144: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:57:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:53 smithi059 ceph-mon[97450]: pgmap v18144: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:57:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:54.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:54.104461+0000) 2022-09-08T06:57:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:54.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:54.054493+0000) 2022-09-08T06:57:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:55.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:55.054667+0000) 2022-09-08T06:57:55.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:55.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:55.104696+0000) 2022-09-08T06:57:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:55 smithi055 ceph-mon[97033]: pgmap v18145: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:57:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:55 smithi059 ceph-mon[97450]: pgmap v18145: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T06:57:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:56.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:56.104932+0000) 2022-09-08T06:57:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:56.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:56.054870+0000) 2022-09-08T06:57:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:57.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:57.105137+0000) 2022-09-08T06:57:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:57.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:57.055012+0000) 2022-09-08T06:57:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:58.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:58.105292+0000) 2022-09-08T06:57:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:58.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:58.055214+0000) 2022-09-08T06:57:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:58 smithi055 ceph-mon[97033]: pgmap v18146: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:57:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:58 smithi059 ceph-mon[97450]: pgmap v18146: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:57:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:57:59.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:59.105528+0000) 2022-09-08T06:57:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:57:59.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:57:59.055437+0000) 2022-09-08T06:57:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:57:59 smithi055 ceph-mon[97033]: pgmap v18147: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:58:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:57:59 smithi059 ceph-mon[97450]: pgmap v18147: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:58:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:00.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:00.105721+0000) 2022-09-08T06:58:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:00.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:00.055603+0000) 2022-09-08T06:58:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:01.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:01.105903+0000) 2022-09-08T06:58:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:01.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:01.055784+0000) 2022-09-08T06:58:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:01 smithi055 ceph-mon[97033]: pgmap v18148: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:58:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:01 smithi059 ceph-mon[97450]: pgmap v18148: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:58:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:02.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:02.106118+0000) 2022-09-08T06:58:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:02.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:02.055922+0000) 2022-09-08T06:58:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:03.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:03.106289+0000) 2022-09-08T06:58:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:03.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:03.056100+0000) 2022-09-08T06:58:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:03 smithi055 ceph-mon[97033]: pgmap v18149: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:58:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:03 smithi059 ceph-mon[97450]: pgmap v18149: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T06:58:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:04.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:04.106512+0000) 2022-09-08T06:58:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:04.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:04.056329+0000) 2022-09-08T06:58:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:05.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:05.106771+0000) 2022-09-08T06:58:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:05.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:05.056549+0000) 2022-09-08T06:58:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:05 smithi055 ceph-mon[97033]: pgmap v18150: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:58:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:05 smithi059 ceph-mon[97450]: pgmap v18150: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:58:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:06.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:06.056716+0000) 2022-09-08T06:58:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:06.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:06.106982+0000) 2022-09-08T06:58:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:58:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:58:07.059 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:58:06.698+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:58:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:07.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:07.107206+0000) 2022-09-08T06:58:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:07.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:07.056948+0000) 2022-09-08T06:58:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:58:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:07 smithi055 ceph-mon[97033]: pgmap v18151: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:58:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:58:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:07 smithi059 ceph-mon[97450]: pgmap v18151: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:58:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:08.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:08.107380+0000) 2022-09-08T06:58:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:08.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:08.057115+0000) 2022-09-08T06:58:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:09.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:09.107608+0000) 2022-09-08T06:58:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:09.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:09.057333+0000) 2022-09-08T06:58:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:10.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:10.107848+0000) 2022-09-08T06:58:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:10.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:10.057508+0000) 2022-09-08T06:58:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:10 smithi055 ceph-mon[97033]: osdmap e1212: 12 total, 11 up, 11 in 2022-09-08T06:58:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:10 smithi055 ceph-mon[97033]: pgmap v18153: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:58:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:10 smithi059 ceph-mon[97450]: osdmap e1212: 12 total, 11 up, 11 in 2022-09-08T06:58:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:10 smithi059 ceph-mon[97450]: pgmap v18153: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:58:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:11.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:11.108024+0000) 2022-09-08T06:58:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:11.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:11.057668+0000) 2022-09-08T06:58:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:11 smithi055 ceph-mon[97033]: pgmap v18154: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:58:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:11 smithi059 ceph-mon[97450]: pgmap v18154: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:58:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:12.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:12.108203+0000) 2022-09-08T06:58:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:12.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:12.057924+0000) 2022-09-08T06:58:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:13.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:13.108386+0000) 2022-09-08T06:58:13.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:13.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:13.058104+0000) 2022-09-08T06:58:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:13 smithi055 ceph-mon[97033]: pgmap v18155: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:58:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:13 smithi059 ceph-mon[97450]: pgmap v18155: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:58:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:14.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:14.108535+0000) 2022-09-08T06:58:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:14.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:14.058358+0000) 2022-09-08T06:58:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:15.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:15.108769+0000) 2022-09-08T06:58:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:15.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:15.058599+0000) 2022-09-08T06:58:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:15 smithi055 ceph-mon[97033]: pgmap v18156: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:58:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:15 smithi059 ceph-mon[97450]: pgmap v18156: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:58:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:16.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:16.108998+0000) 2022-09-08T06:58:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:16.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:16.058809+0000) 2022-09-08T06:58:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:17.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:17.109193+0000) 2022-09-08T06:58:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:17.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:17.058970+0000) 2022-09-08T06:58:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:18.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:18.109406+0000) 2022-09-08T06:58:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:18.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:18.059164+0000) 2022-09-08T06:58:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:18 smithi055 ceph-mon[97033]: osdmap e1213: 12 total, 11 up, 11 in 2022-09-08T06:58:18.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:18 smithi055 ceph-mon[97033]: pgmap v18158: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T06:58:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:18 smithi059 ceph-mon[97450]: osdmap e1213: 12 total, 11 up, 11 in 2022-09-08T06:58:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:18 smithi059 ceph-mon[97450]: pgmap v18158: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T06:58:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:19.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:19.109672+0000) 2022-09-08T06:58:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:19.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:19.059332+0000) 2022-09-08T06:58:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:19 smithi055 ceph-mon[97033]: pgmap v18159: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:58:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:19 smithi059 ceph-mon[97450]: pgmap v18159: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:58:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:20.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:20.109811+0000) 2022-09-08T06:58:20.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:20.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:20.059615+0000) 2022-09-08T06:58:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:21.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:21.059797+0000) 2022-09-08T06:58:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:21.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:21.110025+0000) 2022-09-08T06:58:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:21 smithi055 ceph-mon[97033]: pgmap v18160: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:58:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:21 smithi059 ceph-mon[97450]: pgmap v18160: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:58:22.062 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:58:21.711+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:58:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:22.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:22.110277+0000) 2022-09-08T06:58:22.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:22.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:22.059971+0000) 2022-09-08T06:58:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:23.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:23.110527+0000) 2022-09-08T06:58:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:23.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:23.060114+0000) 2022-09-08T06:58:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:23 smithi055 ceph-mon[97033]: pgmap v18161: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:58:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:23 smithi059 ceph-mon[97450]: pgmap v18161: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:58:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:24.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:24.060274+0000) 2022-09-08T06:58:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:24.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:24.110752+0000) 2022-09-08T06:58:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:25.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:25.110905+0000) 2022-09-08T06:58:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:25.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:25.060486+0000) 2022-09-08T06:58:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:25 smithi055 ceph-mon[97033]: pgmap v18162: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:58:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:25 smithi059 ceph-mon[97450]: pgmap v18162: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:58:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:26.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:26.111082+0000) 2022-09-08T06:58:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:26.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:26.060683+0000) 2022-09-08T06:58:27.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:27.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:27.060920+0000) 2022-09-08T06:58:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:27.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:27.111319+0000) 2022-09-08T06:58:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:28.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:28.111544+0000) 2022-09-08T06:58:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:28.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:28.061165+0000) 2022-09-08T06:58:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:28 smithi055 ceph-mon[97033]: pgmap v18163: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:58:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:28 smithi059 ceph-mon[97450]: pgmap v18163: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:58:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:29.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:29.111780+0000) 2022-09-08T06:58:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:29.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:29.061404+0000) 2022-09-08T06:58:29.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:29 smithi055 ceph-mon[97033]: pgmap v18164: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:29 smithi059 ceph-mon[97450]: pgmap v18164: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:30.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:30.111910+0000) 2022-09-08T06:58:30.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:30.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:30.061570+0000) 2022-09-08T06:58:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:31.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:31.112114+0000) 2022-09-08T06:58:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:31.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:31.061752+0000) 2022-09-08T06:58:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:31 smithi055 ceph-mon[97033]: pgmap v18165: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:58:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:31 smithi059 ceph-mon[97450]: pgmap v18165: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:58:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:32.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:32.112244+0000) 2022-09-08T06:58:32.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:32.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:32.061896+0000) 2022-09-08T06:58:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:33.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:33.112462+0000) 2022-09-08T06:58:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:33.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:33.062110+0000) 2022-09-08T06:58:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:58:33.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:58:33.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:58:33.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:58:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:58:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:58:33.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:58:33.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:58:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:34.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:34.112732+0000) 2022-09-08T06:58:34.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:34.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:34.062318+0000) 2022-09-08T06:58:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:34 smithi055 ceph-mon[97033]: pgmap v18166: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:34 smithi059 ceph-mon[97450]: pgmap v18166: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:35.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:35.112960+0000) 2022-09-08T06:58:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:35.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:35.062458+0000) 2022-09-08T06:58:35.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:35 smithi055 ceph-mon[97033]: pgmap v18167: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:58:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:35 smithi059 ceph-mon[97450]: pgmap v18167: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:58:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:36.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:36.113170+0000) 2022-09-08T06:58:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:36.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:36.062679+0000) 2022-09-08T06:58:37.065 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:58:36.720+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:58:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:37.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:37.113372+0000) 2022-09-08T06:58:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:37.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:37.062826+0000) 2022-09-08T06:58:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:38.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:38.113590+0000) 2022-09-08T06:58:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:38.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:38.063017+0000) 2022-09-08T06:58:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:38 smithi055 ceph-mon[97033]: pgmap v18168: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:38.531 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:38 smithi059 ceph-mon[97450]: pgmap v18168: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:39.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:39.113791+0000) 2022-09-08T06:58:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:39.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:39.063210+0000) 2022-09-08T06:58:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:39 smithi055 ceph-mon[97033]: pgmap v18169: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:39 smithi059 ceph-mon[97450]: pgmap v18169: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:40.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:40.114016+0000) 2022-09-08T06:58:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:40.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:40.063416+0000) 2022-09-08T06:58:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:41.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:41.114191+0000) 2022-09-08T06:58:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:41.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:41.063580+0000) 2022-09-08T06:58:41.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:41 smithi055 ceph-mon[97033]: pgmap v18170: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:58:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:41 smithi059 ceph-mon[97450]: pgmap v18170: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:58:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:42.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:42.114440+0000) 2022-09-08T06:58:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:42.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:42.063732+0000) 2022-09-08T06:58:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:43.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:43.063943+0000) 2022-09-08T06:58:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:43.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:43.114559+0000) 2022-09-08T06:58:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:43 smithi055 ceph-mon[97033]: pgmap v18171: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:43 smithi059 ceph-mon[97450]: pgmap v18171: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:44.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:44.114744+0000) 2022-09-08T06:58:44.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:44.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:44.064133+0000) 2022-09-08T06:58:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:45.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:45.114994+0000) 2022-09-08T06:58:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:45.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:45.064368+0000) 2022-09-08T06:58:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:45 smithi055 ceph-mon[97033]: pgmap v18172: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:58:46.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:45 smithi059 ceph-mon[97450]: pgmap v18172: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:58:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:46.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:46.115221+0000) 2022-09-08T06:58:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:46.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:46.064589+0000) 2022-09-08T06:58:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:47.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:47.115450+0000) 2022-09-08T06:58:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:47.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:47.064786+0000) 2022-09-08T06:58:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:48.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:48.115681+0000) 2022-09-08T06:58:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:48.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:48.065021+0000) 2022-09-08T06:58:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:48 smithi055 ceph-mon[97033]: pgmap v18173: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:48 smithi059 ceph-mon[97450]: pgmap v18173: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:49.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:49.115921+0000) 2022-09-08T06:58:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:49.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:49.065256+0000) 2022-09-08T06:58:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:49 smithi055 ceph-mon[97033]: pgmap v18174: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:49 smithi059 ceph-mon[97450]: pgmap v18174: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:50.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:50.116113+0000) 2022-09-08T06:58:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:50.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:50.065533+0000) 2022-09-08T06:58:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:51.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:51.116287+0000) 2022-09-08T06:58:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:51.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:51.065758+0000) 2022-09-08T06:58:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:51 smithi055 ceph-mon[97033]: pgmap v18175: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:58:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:51 smithi059 ceph-mon[97450]: pgmap v18175: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:58:52.068 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:58:51.729+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:58:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:52.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:52.116461+0000) 2022-09-08T06:58:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:52.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:52.065960+0000) 2022-09-08T06:58:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:53.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:53.116710+0000) 2022-09-08T06:58:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:53.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:53.066195+0000) 2022-09-08T06:58:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:53 smithi055 ceph-mon[97033]: pgmap v18176: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:53 smithi059 ceph-mon[97450]: pgmap v18176: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:54.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:54.116897+0000) 2022-09-08T06:58:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:54.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:54.066435+0000) 2022-09-08T06:58:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:55.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:55.117141+0000) 2022-09-08T06:58:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:55.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:55.066716+0000) 2022-09-08T06:58:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:55 smithi055 ceph-mon[97033]: pgmap v18177: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:58:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:55 smithi059 ceph-mon[97450]: pgmap v18177: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:58:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:56.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:56.117379+0000) 2022-09-08T06:58:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:56.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:56.066925+0000) 2022-09-08T06:58:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:57.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:57.117566+0000) 2022-09-08T06:58:57.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:57.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:57.067080+0000) 2022-09-08T06:58:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:58.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:58.067301+0000) 2022-09-08T06:58:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:58.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:58.117794+0000) 2022-09-08T06:58:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:58 smithi055 ceph-mon[97033]: pgmap v18178: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:58 smithi059 ceph-mon[97450]: pgmap v18178: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:58:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:58:59.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:59.118025+0000) 2022-09-08T06:58:59.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:58:59.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:58:59.067468+0000) 2022-09-08T06:58:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:58:59 smithi055 ceph-mon[97033]: pgmap v18179: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:58:59 smithi059 ceph-mon[97450]: pgmap v18179: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:00.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:00.118270+0000) 2022-09-08T06:59:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:00.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:00.067731+0000) 2022-09-08T06:59:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:01.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:01.118477+0000) 2022-09-08T06:59:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:01.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:01.067983+0000) 2022-09-08T06:59:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:01 smithi055 ceph-mon[97033]: pgmap v18180: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:59:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:01 smithi059 ceph-mon[97450]: pgmap v18180: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:59:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:02.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:02.118730+0000) 2022-09-08T06:59:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:02.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:02.068228+0000) 2022-09-08T06:59:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:03.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:03.118922+0000) 2022-09-08T06:59:03.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:03.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:03.068470+0000) 2022-09-08T06:59:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:03 smithi055 ceph-mon[97033]: pgmap v18181: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:03 smithi059 ceph-mon[97450]: pgmap v18181: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:04.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:04.119152+0000) 2022-09-08T06:59:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:04.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:04.068700+0000) 2022-09-08T06:59:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:05.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:05.119386+0000) 2022-09-08T06:59:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:05.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:05.068853+0000) 2022-09-08T06:59:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:05 smithi055 ceph-mon[97033]: pgmap v18182: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:59:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:05 smithi059 ceph-mon[97450]: pgmap v18182: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T06:59:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:06.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:06.119628+0000) 2022-09-08T06:59:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:06.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:06.068998+0000) 2022-09-08T06:59:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:59:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T06:59:07.071 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:59:06.732+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:59:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:07.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:07.119870+0000) 2022-09-08T06:59:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:07.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:07.069218+0000) 2022-09-08T06:59:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:59:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:07 smithi055 ceph-mon[97033]: pgmap v18183: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:59:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T06:59:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:07 smithi059 ceph-mon[97450]: pgmap v18183: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T06:59:08.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:08.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:08.069456+0000) 2022-09-08T06:59:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:08.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:08.120013+0000) 2022-09-08T06:59:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:09.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:09.120166+0000) 2022-09-08T06:59:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:09.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:09.069732+0000) 2022-09-08T06:59:10.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:10.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:10.069902+0000) 2022-09-08T06:59:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:10.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:10.120411+0000) 2022-09-08T06:59:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:10 smithi055 ceph-mon[97033]: osdmap e1214: 12 total, 11 up, 11 in 2022-09-08T06:59:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:10 smithi055 ceph-mon[97033]: pgmap v18185: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:59:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:10 smithi059 ceph-mon[97450]: osdmap e1214: 12 total, 11 up, 11 in 2022-09-08T06:59:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:10 smithi059 ceph-mon[97450]: pgmap v18185: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:59:11.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:11.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:11.070109+0000) 2022-09-08T06:59:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:11.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:11.120681+0000) 2022-09-08T06:59:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:11 smithi055 ceph-mon[97033]: pgmap v18186: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:59:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:11 smithi059 ceph-mon[97450]: pgmap v18186: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:59:12.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:12.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:12.070287+0000) 2022-09-08T06:59:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:12.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:12.120864+0000) 2022-09-08T06:59:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:13.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:13.121058+0000) 2022-09-08T06:59:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:13.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:13.070515+0000) 2022-09-08T06:59:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:13 smithi055 ceph-mon[97033]: pgmap v18187: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:59:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:13 smithi059 ceph-mon[97450]: pgmap v18187: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T06:59:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:14.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:14.070704+0000) 2022-09-08T06:59:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:14.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:14.121247+0000) 2022-09-08T06:59:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:15.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:15.070928+0000) 2022-09-08T06:59:15.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:15.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:15.121472+0000) 2022-09-08T06:59:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:15 smithi055 ceph-mon[97033]: pgmap v18188: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:59:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:15 smithi059 ceph-mon[97450]: pgmap v18188: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:59:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:16.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:16.121743+0000) 2022-09-08T06:59:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:16.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:16.071188+0000) 2022-09-08T06:59:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:17.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:17.121983+0000) 2022-09-08T06:59:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:17.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:17.071412+0000) 2022-09-08T06:59:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:18.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:18.122240+0000) 2022-09-08T06:59:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:18.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:18.071563+0000) 2022-09-08T06:59:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:18 smithi055 ceph-mon[97033]: osdmap e1215: 12 total, 11 up, 11 in 2022-09-08T06:59:18.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:18 smithi055 ceph-mon[97033]: pgmap v18190: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 244 B/s wr, 7 op/s 2022-09-08T06:59:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:18 smithi059 ceph-mon[97450]: osdmap e1215: 12 total, 11 up, 11 in 2022-09-08T06:59:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:18 smithi059 ceph-mon[97450]: pgmap v18190: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 244 B/s wr, 7 op/s 2022-09-08T06:59:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:19.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:19.071707+0000) 2022-09-08T06:59:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:19.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:19.122379+0000) 2022-09-08T06:59:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:19 smithi055 ceph-mon[97033]: pgmap v18191: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:59:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:19 smithi059 ceph-mon[97450]: pgmap v18191: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T06:59:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:20.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:20.071860+0000) 2022-09-08T06:59:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:20.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:20.122610+0000) 2022-09-08T06:59:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:21.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:21.122926+0000) 2022-09-08T06:59:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:21.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:21.072021+0000) 2022-09-08T06:59:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:21 smithi055 ceph-mon[97033]: pgmap v18192: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:59:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:21 smithi059 ceph-mon[97450]: pgmap v18192: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:59:22.074 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:59:21.735+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:59:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:22.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:22.072168+0000) 2022-09-08T06:59:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:22.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:22.123073+0000) 2022-09-08T06:59:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:23.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:23.123285+0000) 2022-09-08T06:59:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:23.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:23.072357+0000) 2022-09-08T06:59:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:23 smithi055 ceph-mon[97033]: pgmap v18193: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:59:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:23 smithi059 ceph-mon[97450]: pgmap v18193: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T06:59:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:24.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:24.123524+0000) 2022-09-08T06:59:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:24.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:24.072510+0000) 2022-09-08T06:59:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:25.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:25.123758+0000) 2022-09-08T06:59:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:25.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:25.072676+0000) 2022-09-08T06:59:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:25 smithi055 ceph-mon[97033]: pgmap v18194: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:59:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:25 smithi059 ceph-mon[97450]: pgmap v18194: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T06:59:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:26.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:26.123904+0000) 2022-09-08T06:59:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:26.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:26.072852+0000) 2022-09-08T06:59:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:27.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:27.124128+0000) 2022-09-08T06:59:27.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:27.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:27.073018+0000) 2022-09-08T06:59:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:28.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:28.124262+0000) 2022-09-08T06:59:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:28.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:28.073173+0000) 2022-09-08T06:59:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:29.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:29.124482+0000) 2022-09-08T06:59:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:29.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:29.073329+0000) 2022-09-08T06:59:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:29 smithi055 ceph-mon[97033]: pgmap v18195: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:59:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:29 smithi059 ceph-mon[97450]: pgmap v18195: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T06:59:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:30.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:30.124757+0000) 2022-09-08T06:59:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:30.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:30.073481+0000) 2022-09-08T06:59:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:31.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:31.124988+0000) 2022-09-08T06:59:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:31.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:31.073765+0000) 2022-09-08T06:59:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:31 smithi059 ceph-mon[97450]: pgmap v18196: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:31 smithi055 ceph-mon[97033]: pgmap v18196: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:32.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:32.125211+0000) 2022-09-08T06:59:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:32.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:32.073931+0000) 2022-09-08T06:59:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:33.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:33.125436+0000) 2022-09-08T06:59:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:33.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:33.074196+0000) 2022-09-08T06:59:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:33 smithi059 ceph-mon[97450]: pgmap v18197: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:59:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:59:33.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:59:33.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:59:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:33 smithi055 ceph-mon[97033]: pgmap v18197: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:59:33.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T06:59:33.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T06:59:33.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T06:59:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:34.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:34.125714+0000) 2022-09-08T06:59:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:34.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:34.074381+0000) 2022-09-08T06:59:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:59:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T06:59:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:35.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:35.125911+0000) 2022-09-08T06:59:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:35.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:35.074536+0000) 2022-09-08T06:59:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:35 smithi055 ceph-mon[97033]: pgmap v18198: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:35 smithi059 ceph-mon[97450]: pgmap v18198: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:36.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:36.126157+0000) 2022-09-08T06:59:36.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:36.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:36.074714+0000) 2022-09-08T06:59:37.077 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:59:36.751+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:59:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:37.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:37.126418+0000) 2022-09-08T06:59:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:37.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:37.074881+0000) 2022-09-08T06:59:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:37 smithi055 ceph-mon[97033]: pgmap v18199: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:59:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:37 smithi059 ceph-mon[97450]: pgmap v18199: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:59:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:38.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:38.075060+0000) 2022-09-08T06:59:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:38.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:38.126670+0000) 2022-09-08T06:59:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:39.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:39.075282+0000) 2022-09-08T06:59:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:39.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:39.126913+0000) 2022-09-08T06:59:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:39 smithi055 ceph-mon[97033]: pgmap v18200: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:39 smithi059 ceph-mon[97450]: pgmap v18200: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:40.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:40.127168+0000) 2022-09-08T06:59:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:40.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:40.075430+0000) 2022-09-08T06:59:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:41.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:41.127412+0000) 2022-09-08T06:59:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:41.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:41.075679+0000) 2022-09-08T06:59:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:41 smithi055 ceph-mon[97033]: pgmap v18201: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:41 smithi059 ceph-mon[97450]: pgmap v18201: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:42.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:42.127623+0000) 2022-09-08T06:59:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:42.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:42.075864+0000) 2022-09-08T06:59:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:43.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:43.127895+0000) 2022-09-08T06:59:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:43.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:43.076008+0000) 2022-09-08T06:59:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:43 smithi055 ceph-mon[97033]: pgmap v18202: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:59:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:43 smithi059 ceph-mon[97450]: pgmap v18202: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:59:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:44.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:44.128139+0000) 2022-09-08T06:59:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:44.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:44.076166+0000) 2022-09-08T06:59:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:45.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:45.128365+0000) 2022-09-08T06:59:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:45.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:45.076406+0000) 2022-09-08T06:59:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:45 smithi055 ceph-mon[97033]: pgmap v18203: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:45 smithi059 ceph-mon[97450]: pgmap v18203: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:46.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:46.128575+0000) 2022-09-08T06:59:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:46.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:46.076562+0000) 2022-09-08T06:59:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:47.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:47.076698+0000) 2022-09-08T06:59:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:47.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:47.128806+0000) 2022-09-08T06:59:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:47 smithi055 ceph-mon[97033]: pgmap v18204: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:59:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:47 smithi059 ceph-mon[97450]: pgmap v18204: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:59:48.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:48.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:48.076923+0000) 2022-09-08T06:59:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:48.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:48.129028+0000) 2022-09-08T06:59:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:49.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:49.077073+0000) 2022-09-08T06:59:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:49.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:49.129199+0000) 2022-09-08T06:59:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:49 smithi055 ceph-mon[97033]: pgmap v18205: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:49 smithi059 ceph-mon[97450]: pgmap v18205: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:50.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:50.129349+0000) 2022-09-08T06:59:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:50.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:50.077233+0000) 2022-09-08T06:59:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:51.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:51.129529+0000) 2022-09-08T06:59:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:51.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:51.077378+0000) 2022-09-08T06:59:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:51 smithi055 ceph-mon[97033]: pgmap v18206: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:51 smithi059 ceph-mon[97450]: pgmap v18206: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:52.079 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 06:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T06:59:51.753+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T06:59:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:52.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:52.129735+0000) 2022-09-08T06:59:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:52.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:52.077566+0000) 2022-09-08T06:59:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:53.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:53.129906+0000) 2022-09-08T06:59:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:53.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:53.077778+0000) 2022-09-08T06:59:53.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:53 smithi055 ceph-mon[97033]: pgmap v18207: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:59:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:53 smithi059 ceph-mon[97450]: pgmap v18207: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:59:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:54.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:54.130148+0000) 2022-09-08T06:59:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:54.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:54.077965+0000) 2022-09-08T06:59:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:55.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:55.130331+0000) 2022-09-08T06:59:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:55.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:55.078156+0000) 2022-09-08T06:59:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:55 smithi055 ceph-mon[97033]: pgmap v18208: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:55 smithi059 ceph-mon[97450]: pgmap v18208: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:56.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:56.130554+0000) 2022-09-08T06:59:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:56.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:56.078328+0000) 2022-09-08T06:59:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:57.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:57.078513+0000) 2022-09-08T06:59:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:57.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:57.130745+0000) 2022-09-08T06:59:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:57 smithi055 ceph-mon[97033]: pgmap v18209: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:59:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:57 smithi059 ceph-mon[97450]: pgmap v18209: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T06:59:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:58.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:58.130906+0000) 2022-09-08T06:59:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:58.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:58.078723+0000) 2022-09-08T06:59:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 06:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T06:59:59.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:59.131158+0000) 2022-09-08T06:59:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 06:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T06:59:59.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T05:59:59.078898+0000) 2022-09-08T06:59:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 06:59:59 smithi055 ceph-mon[97033]: pgmap v18210: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T06:59:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 06:59:59 smithi059 ceph-mon[97450]: pgmap v18210: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:00.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:00.131378+0000) 2022-09-08T07:00:00.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:00.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:00.079120+0000) 2022-09-08T07:00:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T07:00:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T07:00:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T07:00:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T07:00:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T07:00:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T07:00:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:01.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:01.131599+0000) 2022-09-08T07:00:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:01.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:01.079287+0000) 2022-09-08T07:00:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:01 smithi055 ceph-mon[97033]: pgmap v18211: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:01 smithi059 ceph-mon[97450]: pgmap v18211: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:02.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:02.131745+0000) 2022-09-08T07:00:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:02.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:02.079495+0000) 2022-09-08T07:00:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:03.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:03.131950+0000) 2022-09-08T07:00:03.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:03.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:03.079680+0000) 2022-09-08T07:00:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:03 smithi055 ceph-mon[97033]: pgmap v18212: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:00:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:03 smithi059 ceph-mon[97450]: pgmap v18212: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:00:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:04.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:04.132199+0000) 2022-09-08T07:00:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:04.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:04.079898+0000) 2022-09-08T07:00:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:05.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:05.132361+0000) 2022-09-08T07:00:05.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:05.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:05.080128+0000) 2022-09-08T07:00:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:05 smithi055 ceph-mon[97033]: pgmap v18213: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:05 smithi059 ceph-mon[97450]: pgmap v18213: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:06.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:06.132579+0000) 2022-09-08T07:00:06.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:06.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:06.080357+0000) 2022-09-08T07:00:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:00:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:00:07.082 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:00:06.762+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:00:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:07.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:07.132742+0000) 2022-09-08T07:00:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:07.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:07.080511+0000) 2022-09-08T07:00:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:07 smithi059 ceph-mon[97450]: pgmap v18214: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:00:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:00:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:07 smithi055 ceph-mon[97033]: pgmap v18214: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:00:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:00:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:08.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:08.132984+0000) 2022-09-08T07:00:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:08.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:08.080689+0000) 2022-09-08T07:00:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:09.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:09.133188+0000) 2022-09-08T07:00:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:09.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:09.080907+0000) 2022-09-08T07:00:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:09 smithi059 ceph-mon[97450]: pgmap v18215: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:00:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:09 smithi055 ceph-mon[97033]: pgmap v18215: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:00:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:10.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:10.133333+0000) 2022-09-08T07:00:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:10.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:10.081072+0000) 2022-09-08T07:00:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:10 smithi059 ceph-mon[97450]: osdmap e1216: 12 total, 11 up, 11 in 2022-09-08T07:00:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:10 smithi055 ceph-mon[97033]: osdmap e1216: 12 total, 11 up, 11 in 2022-09-08T07:00:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:11.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:11.133582+0000) 2022-09-08T07:00:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:11.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:11.081220+0000) 2022-09-08T07:00:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:11 smithi059 ceph-mon[97450]: pgmap v18217: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:00:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:11 smithi055 ceph-mon[97033]: pgmap v18217: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:00:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:12.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:12.133739+0000) 2022-09-08T07:00:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:12.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:12.081373+0000) 2022-09-08T07:00:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:13.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:13.133951+0000) 2022-09-08T07:00:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:13.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:13.081594+0000) 2022-09-08T07:00:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:13 smithi059 ceph-mon[97450]: pgmap v18218: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:00:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:13 smithi055 ceph-mon[97033]: pgmap v18218: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:00:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:14.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:14.134190+0000) 2022-09-08T07:00:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:14.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:14.081730+0000) 2022-09-08T07:00:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:15.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:15.134424+0000) 2022-09-08T07:00:15.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:15.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:15.081950+0000) 2022-09-08T07:00:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:15 smithi059 ceph-mon[97450]: pgmap v18219: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:00:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:15 smithi055 ceph-mon[97033]: pgmap v18219: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:00:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:16.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:16.134642+0000) 2022-09-08T07:00:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:16.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:16.082136+0000) 2022-09-08T07:00:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:17.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:17.134813+0000) 2022-09-08T07:00:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:17.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:17.082338+0000) 2022-09-08T07:00:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:17 smithi059 ceph-mon[97450]: pgmap v18220: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:00:17.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:17 smithi059 ceph-mon[97450]: osdmap e1217: 12 total, 11 up, 11 in 2022-09-08T07:00:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:17 smithi055 ceph-mon[97033]: pgmap v18220: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:00:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:17 smithi055 ceph-mon[97033]: osdmap e1217: 12 total, 11 up, 11 in 2022-09-08T07:00:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:18.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:18.135004+0000) 2022-09-08T07:00:18.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:18.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:18.082528+0000) 2022-09-08T07:00:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:19.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:19.135212+0000) 2022-09-08T07:00:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:19.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:19.082733+0000) 2022-09-08T07:00:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:19 smithi059 ceph-mon[97450]: pgmap v18222: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T07:00:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:19 smithi055 ceph-mon[97033]: pgmap v18222: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T07:00:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:20.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:20.135446+0000) 2022-09-08T07:00:20.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:20.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:20.082899+0000) 2022-09-08T07:00:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:21.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:21.135726+0000) 2022-09-08T07:00:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:21.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:21.083104+0000) 2022-09-08T07:00:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:21 smithi059 ceph-mon[97450]: pgmap v18223: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:00:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:21 smithi055 ceph-mon[97033]: pgmap v18223: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:00:22.085 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:00:21.771+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:00:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:22.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:22.083285+0000) 2022-09-08T07:00:22.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:22.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:22.135970+0000) 2022-09-08T07:00:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:23.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:23.136231+0000) 2022-09-08T07:00:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:23.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:23.083449+0000) 2022-09-08T07:00:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:23 smithi059 ceph-mon[97450]: pgmap v18224: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:00:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:23 smithi055 ceph-mon[97033]: pgmap v18224: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:00:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:24.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:24.083699+0000) 2022-09-08T07:00:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:24.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:24.136442+0000) 2022-09-08T07:00:25.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:25.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:25.083884+0000) 2022-09-08T07:00:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:25.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:25.136657+0000) 2022-09-08T07:00:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:25 smithi059 ceph-mon[97450]: pgmap v18225: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:00:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:25 smithi055 ceph-mon[97033]: pgmap v18225: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:00:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:26.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:26.136819+0000) 2022-09-08T07:00:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:26.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:26.084027+0000) 2022-09-08T07:00:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:27.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:27.136966+0000) 2022-09-08T07:00:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:27.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:27.084275+0000) 2022-09-08T07:00:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:27 smithi059 ceph-mon[97450]: pgmap v18226: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:00:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:27 smithi055 ceph-mon[97033]: pgmap v18226: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:00:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:28.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:28.084507+0000) 2022-09-08T07:00:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:28.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:28.137107+0000) 2022-09-08T07:00:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:29.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:29.137308+0000) 2022-09-08T07:00:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:29.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:29.084660+0000) 2022-09-08T07:00:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:29 smithi059 ceph-mon[97450]: pgmap v18227: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:00:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:29 smithi055 ceph-mon[97033]: pgmap v18227: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:00:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:30.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:30.137495+0000) 2022-09-08T07:00:30.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:30.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:30.084864+0000) 2022-09-08T07:00:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:31.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:31.137741+0000) 2022-09-08T07:00:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:31.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:31.085036+0000) 2022-09-08T07:00:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:31 smithi055 ceph-mon[97033]: pgmap v18228: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:31 smithi059 ceph-mon[97450]: pgmap v18228: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:32.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:32.137912+0000) 2022-09-08T07:00:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:32.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:32.085189+0000) 2022-09-08T07:00:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:33.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:33.138126+0000) 2022-09-08T07:00:33.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:33.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:33.085348+0000) 2022-09-08T07:00:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:33 smithi055 ceph-mon[97033]: pgmap v18229: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:00:33.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:00:33.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:00:33.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:00:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:33 smithi059 ceph-mon[97450]: pgmap v18229: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:00:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:00:34.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:00:34.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:00:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:34.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:34.138287+0000) 2022-09-08T07:00:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:34.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:34.085560+0000) 2022-09-08T07:00:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:35.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:35.138494+0000) 2022-09-08T07:00:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:35.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:35.085750+0000) 2022-09-08T07:00:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:35 smithi055 ceph-mon[97033]: pgmap v18230: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:35.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:00:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:35 smithi059 ceph-mon[97450]: pgmap v18230: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:00:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:36.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:36.138706+0000) 2022-09-08T07:00:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:36.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:36.085944+0000) 2022-09-08T07:00:37.088 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:00:36.787+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:00:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:37.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:37.138859+0000) 2022-09-08T07:00:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:37.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:37.086098+0000) 2022-09-08T07:00:37.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:37 smithi055 ceph-mon[97033]: pgmap v18231: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:00:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:37 smithi059 ceph-mon[97450]: pgmap v18231: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:00:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:38.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:38.139082+0000) 2022-09-08T07:00:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:38.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:38.086241+0000) 2022-09-08T07:00:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:39.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:39.139304+0000) 2022-09-08T07:00:39.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:39.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:39.086397+0000) 2022-09-08T07:00:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:39 smithi059 ceph-mon[97450]: pgmap v18232: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:39 smithi055 ceph-mon[97033]: pgmap v18232: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:40.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:40.139554+0000) 2022-09-08T07:00:40.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:40.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:40.086607+0000) 2022-09-08T07:00:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:41.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:41.139800+0000) 2022-09-08T07:00:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:41.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:41.086749+0000) 2022-09-08T07:00:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:41 smithi059 ceph-mon[97450]: pgmap v18233: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:41 smithi055 ceph-mon[97033]: pgmap v18233: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:42.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:42.140036+0000) 2022-09-08T07:00:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:42.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:42.086958+0000) 2022-09-08T07:00:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:43.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:43.140258+0000) 2022-09-08T07:00:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:43.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:43.087140+0000) 2022-09-08T07:00:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:43 smithi059 ceph-mon[97450]: pgmap v18234: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:00:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:43 smithi055 ceph-mon[97033]: pgmap v18234: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:00:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:44.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:44.140480+0000) 2022-09-08T07:00:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:44.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:44.087296+0000) 2022-09-08T07:00:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:45.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:45.140743+0000) 2022-09-08T07:00:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:45.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:45.087425+0000) 2022-09-08T07:00:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:45 smithi059 ceph-mon[97450]: pgmap v18235: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:45 smithi055 ceph-mon[97033]: pgmap v18235: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:46.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:46.140926+0000) 2022-09-08T07:00:46.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:46.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:46.087602+0000) 2022-09-08T07:00:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:47.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:47.141152+0000) 2022-09-08T07:00:47.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:47.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:47.087829+0000) 2022-09-08T07:00:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:47 smithi059 ceph-mon[97450]: pgmap v18236: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:00:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:47 smithi055 ceph-mon[97033]: pgmap v18236: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:00:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:48.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:48.141393+0000) 2022-09-08T07:00:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:48.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:48.087975+0000) 2022-09-08T07:00:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:49.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:49.141666+0000) 2022-09-08T07:00:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:49.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:49.088103+0000) 2022-09-08T07:00:49.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:49 smithi059 ceph-mon[97450]: pgmap v18237: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:49 smithi055 ceph-mon[97033]: pgmap v18237: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:50.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:50.141927+0000) 2022-09-08T07:00:50.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:50.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:50.088242+0000) 2022-09-08T07:00:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:51.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:51.088447+0000) 2022-09-08T07:00:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:51.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:51.142177+0000) 2022-09-08T07:00:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:51 smithi059 ceph-mon[97450]: pgmap v18238: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:51 smithi055 ceph-mon[97033]: pgmap v18238: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:52.090 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:00:51.797+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:00:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:52.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:52.142404+0000) 2022-09-08T07:00:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:52.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:52.088599+0000) 2022-09-08T07:00:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:53.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:53.142676+0000) 2022-09-08T07:00:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:53.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:53.088795+0000) 2022-09-08T07:00:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:53 smithi059 ceph-mon[97450]: pgmap v18239: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:00:53.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:53 smithi055 ceph-mon[97033]: pgmap v18239: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:00:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:54.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:54.142829+0000) 2022-09-08T07:00:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:54.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:54.089011+0000) 2022-09-08T07:00:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:55.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:55.143088+0000) 2022-09-08T07:00:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:55.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:55.089221+0000) 2022-09-08T07:00:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:55 smithi059 ceph-mon[97450]: pgmap v18240: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:55 smithi055 ceph-mon[97033]: pgmap v18240: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:56.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:56.143346+0000) 2022-09-08T07:00:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:56.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:56.089406+0000) 2022-09-08T07:00:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:57.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:57.143672+0000) 2022-09-08T07:00:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:57.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:57.089543+0000) 2022-09-08T07:00:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:57 smithi055 ceph-mon[97033]: pgmap v18241: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:00:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:57 smithi059 ceph-mon[97450]: pgmap v18241: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:00:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:58.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:58.143913+0000) 2022-09-08T07:00:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:58.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:58.089775+0000) 2022-09-08T07:00:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:00:59.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:59.144141+0000) 2022-09-08T07:00:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:00:59.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:00:59.089998+0000) 2022-09-08T07:00:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:00:59 smithi055 ceph-mon[97033]: pgmap v18242: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:00:59.789 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:00:59 smithi059 ceph-mon[97450]: pgmap v18242: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:00.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:00.144344+0000) 2022-09-08T07:01:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:00.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:00.090241+0000) 2022-09-08T07:01:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:01.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:01.144503+0000) 2022-09-08T07:01:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:01.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:01.090416+0000) 2022-09-08T07:01:01.647 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:01 smithi055 ceph-mon[97033]: pgmap v18243: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:01 smithi059 ceph-mon[97450]: pgmap v18243: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:02.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:02.144720+0000) 2022-09-08T07:01:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:02.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:02.090552+0000) 2022-09-08T07:01:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:03.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:03.144901+0000) 2022-09-08T07:01:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:03.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:03.090774+0000) 2022-09-08T07:01:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:03 smithi055 ceph-mon[97033]: pgmap v18244: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:01:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:03 smithi059 ceph-mon[97450]: pgmap v18244: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:01:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:04.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:04.145036+0000) 2022-09-08T07:01:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:04.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:04.091023+0000) 2022-09-08T07:01:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:05.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:05.145215+0000) 2022-09-08T07:01:05.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:05.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:05.091161+0000) 2022-09-08T07:01:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:05 smithi055 ceph-mon[97033]: pgmap v18245: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:05 smithi059 ceph-mon[97450]: pgmap v18245: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:06.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:06.145386+0000) 2022-09-08T07:01:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:06.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:06.091362+0000) 2022-09-08T07:01:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:01:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:01:07.093 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:01:06.799+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:01:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:07.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:07.091533+0000) 2022-09-08T07:01:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:07.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:07.145502+0000) 2022-09-08T07:01:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:07 smithi055 ceph-mon[97033]: pgmap v18246: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:01:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:01:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:07 smithi059 ceph-mon[97450]: pgmap v18246: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:01:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:01:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:08.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:08.145724+0000) 2022-09-08T07:01:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:08.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:08.091732+0000) 2022-09-08T07:01:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:09.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:09.145927+0000) 2022-09-08T07:01:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:09.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:09.091990+0000) 2022-09-08T07:01:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:09 smithi055 ceph-mon[97033]: pgmap v18247: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:01:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:09 smithi059 ceph-mon[97450]: pgmap v18247: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:01:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:10.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:10.146095+0000) 2022-09-08T07:01:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:10.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:10.092159+0000) 2022-09-08T07:01:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:10 smithi055 ceph-mon[97033]: osdmap e1218: 12 total, 11 up, 11 in 2022-09-08T07:01:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:10 smithi059 ceph-mon[97450]: osdmap e1218: 12 total, 11 up, 11 in 2022-09-08T07:01:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:11.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:11.146243+0000) 2022-09-08T07:01:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:11.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:11.092286+0000) 2022-09-08T07:01:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:11 smithi055 ceph-mon[97033]: pgmap v18249: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:01:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:11 smithi059 ceph-mon[97450]: pgmap v18249: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:01:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:12.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:12.146437+0000) 2022-09-08T07:01:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:12.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:12.092469+0000) 2022-09-08T07:01:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:13.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:13.146674+0000) 2022-09-08T07:01:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:13.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:13.092658+0000) 2022-09-08T07:01:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:13 smithi055 ceph-mon[97033]: pgmap v18250: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:01:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:13 smithi059 ceph-mon[97450]: pgmap v18250: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:01:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:14.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:14.146916+0000) 2022-09-08T07:01:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:14.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:14.092902+0000) 2022-09-08T07:01:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:15.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:15.147097+0000) 2022-09-08T07:01:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:15.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:15.093073+0000) 2022-09-08T07:01:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:15 smithi055 ceph-mon[97033]: pgmap v18251: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:01:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:15 smithi059 ceph-mon[97450]: pgmap v18251: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:01:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:16.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:16.147302+0000) 2022-09-08T07:01:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:16.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:16.093195+0000) 2022-09-08T07:01:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:17.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:17.147553+0000) 2022-09-08T07:01:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:17.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:17.093372+0000) 2022-09-08T07:01:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:17 smithi055 ceph-mon[97033]: pgmap v18252: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:01:17.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:17 smithi055 ceph-mon[97033]: osdmap e1219: 12 total, 11 up, 11 in 2022-09-08T07:01:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:17 smithi059 ceph-mon[97450]: pgmap v18252: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:01:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:17 smithi059 ceph-mon[97450]: osdmap e1219: 12 total, 11 up, 11 in 2022-09-08T07:01:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:18.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:18.147695+0000) 2022-09-08T07:01:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:18.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:18.093601+0000) 2022-09-08T07:01:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:19.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:19.093820+0000) 2022-09-08T07:01:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:19.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:19.147812+0000) 2022-09-08T07:01:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:19 smithi055 ceph-mon[97033]: pgmap v18254: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 246 B/s wr, 7 op/s 2022-09-08T07:01:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:19 smithi059 ceph-mon[97450]: pgmap v18254: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 246 B/s wr, 7 op/s 2022-09-08T07:01:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:20.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:20.148040+0000) 2022-09-08T07:01:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:20.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:20.094070+0000) 2022-09-08T07:01:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:21.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:21.148177+0000) 2022-09-08T07:01:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:21.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:21.094265+0000) 2022-09-08T07:01:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:21 smithi055 ceph-mon[97033]: pgmap v18255: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:01:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:21 smithi059 ceph-mon[97450]: pgmap v18255: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:01:22.096 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:01:21.815+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:01:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:22.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:22.148464+0000) 2022-09-08T07:01:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:22.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:22.094410+0000) 2022-09-08T07:01:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:23.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:23.148683+0000) 2022-09-08T07:01:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:23.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:23.094588+0000) 2022-09-08T07:01:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:23 smithi055 ceph-mon[97033]: pgmap v18256: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:01:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:23 smithi059 ceph-mon[97450]: pgmap v18256: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:01:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:24.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:24.148876+0000) 2022-09-08T07:01:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:24.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:24.094785+0000) 2022-09-08T07:01:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:25.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:25.149074+0000) 2022-09-08T07:01:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:25.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:25.094949+0000) 2022-09-08T07:01:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:25 smithi055 ceph-mon[97033]: pgmap v18257: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:01:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:25 smithi059 ceph-mon[97450]: pgmap v18257: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:01:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:26.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:26.149276+0000) 2022-09-08T07:01:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:26.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:26.095151+0000) 2022-09-08T07:01:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:27.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:27.149518+0000) 2022-09-08T07:01:27.440 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:27.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:27.095327+0000) 2022-09-08T07:01:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:27 smithi055 ceph-mon[97033]: pgmap v18258: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:01:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:27 smithi059 ceph-mon[97450]: pgmap v18258: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:01:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:28.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:28.149750+0000) 2022-09-08T07:01:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:28.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:28.095492+0000) 2022-09-08T07:01:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:29.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:29.149979+0000) 2022-09-08T07:01:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:29.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:29.095659+0000) 2022-09-08T07:01:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:29 smithi055 ceph-mon[97033]: pgmap v18259: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:01:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:29 smithi059 ceph-mon[97450]: pgmap v18259: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:01:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:30.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:30.150212+0000) 2022-09-08T07:01:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:30.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:30.095865+0000) 2022-09-08T07:01:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:31.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:31.150442+0000) 2022-09-08T07:01:31.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:31.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:31.096057+0000) 2022-09-08T07:01:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:31 smithi055 ceph-mon[97033]: pgmap v18260: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:31 smithi059 ceph-mon[97450]: pgmap v18260: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:32.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:32.150681+0000) 2022-09-08T07:01:32.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:32.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:32.096265+0000) 2022-09-08T07:01:33.410 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:33.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:33.150908+0000) 2022-09-08T07:01:33.410 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:33.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:33.096499+0000) 2022-09-08T07:01:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:33 smithi055 ceph-mon[97033]: pgmap v18261: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:01:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:33 smithi059 ceph-mon[97450]: pgmap v18261: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:01:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:34.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:34.151117+0000) 2022-09-08T07:01:34.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:34.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:34.096675+0000) 2022-09-08T07:01:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:01:34.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:01:34.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:34 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:01:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:01:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:01:34.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:34 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:01:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:35.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:35.151284+0000) 2022-09-08T07:01:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:35.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:35.096804+0000) 2022-09-08T07:01:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:35 smithi055 ceph-mon[97033]: pgmap v18262: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:35.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:01:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:35 smithi059 ceph-mon[97450]: pgmap v18262: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:01:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:36.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:36.151522+0000) 2022-09-08T07:01:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:36.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:36.096935+0000) 2022-09-08T07:01:37.099 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:01:36.823+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:01:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:37.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:37.151751+0000) 2022-09-08T07:01:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:37.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:37.097128+0000) 2022-09-08T07:01:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:37 smithi055 ceph-mon[97033]: pgmap v18263: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:01:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:37 smithi059 ceph-mon[97450]: pgmap v18263: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:01:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:38.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:38.151933+0000) 2022-09-08T07:01:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:38.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:38.097274+0000) 2022-09-08T07:01:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:39.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:39.152137+0000) 2022-09-08T07:01:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:39.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:39.097405+0000) 2022-09-08T07:01:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:39 smithi055 ceph-mon[97033]: pgmap v18264: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:39 smithi059 ceph-mon[97450]: pgmap v18264: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:40.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:40.152353+0000) 2022-09-08T07:01:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:40.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:40.097575+0000) 2022-09-08T07:01:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:41.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:41.152563+0000) 2022-09-08T07:01:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:41.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:41.097760+0000) 2022-09-08T07:01:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:41 smithi055 ceph-mon[97033]: pgmap v18265: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:41 smithi059 ceph-mon[97450]: pgmap v18265: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:42.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:42.152806+0000) 2022-09-08T07:01:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:42.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:42.097984+0000) 2022-09-08T07:01:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:43.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:43.153041+0000) 2022-09-08T07:01:43.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:43.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:43.098144+0000) 2022-09-08T07:01:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:43 smithi055 ceph-mon[97033]: pgmap v18266: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:01:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:43 smithi059 ceph-mon[97450]: pgmap v18266: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:01:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:44.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:44.153307+0000) 2022-09-08T07:01:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:44.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:44.098343+0000) 2022-09-08T07:01:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:45.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:45.153582+0000) 2022-09-08T07:01:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:45.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:45.098500+0000) 2022-09-08T07:01:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:45 smithi055 ceph-mon[97033]: pgmap v18267: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:45 smithi059 ceph-mon[97450]: pgmap v18267: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:46.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:46.153764+0000) 2022-09-08T07:01:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:46.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:46.098734+0000) 2022-09-08T07:01:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:47.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:47.153980+0000) 2022-09-08T07:01:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:47.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:47.098905+0000) 2022-09-08T07:01:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:47 smithi055 ceph-mon[97033]: pgmap v18268: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:01:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:47 smithi059 ceph-mon[97450]: pgmap v18268: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:01:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:48.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:48.154217+0000) 2022-09-08T07:01:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:48.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:48.099122+0000) 2022-09-08T07:01:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:49.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:49.154441+0000) 2022-09-08T07:01:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:49.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:49.099336+0000) 2022-09-08T07:01:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:49 smithi055 ceph-mon[97033]: pgmap v18269: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:49 smithi059 ceph-mon[97450]: pgmap v18269: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:50.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:50.154734+0000) 2022-09-08T07:01:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:50.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:50.099508+0000) 2022-09-08T07:01:51.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:51.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:51.099671+0000) 2022-09-08T07:01:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:51.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:51.154897+0000) 2022-09-08T07:01:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:51 smithi055 ceph-mon[97033]: pgmap v18270: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:51 smithi059 ceph-mon[97450]: pgmap v18270: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:52.102 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:01:51.838+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:01:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:52.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:52.155077+0000) 2022-09-08T07:01:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:52.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:52.099888+0000) 2022-09-08T07:01:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:53.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:53.155314+0000) 2022-09-08T07:01:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:53.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:53.100059+0000) 2022-09-08T07:01:53.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:53 smithi055 ceph-mon[97033]: pgmap v18271: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:01:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:53 smithi059 ceph-mon[97450]: pgmap v18271: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:01:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:54.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:54.155536+0000) 2022-09-08T07:01:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:54.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:54.100267+0000) 2022-09-08T07:01:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:55.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:55.155785+0000) 2022-09-08T07:01:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:55.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:55.100481+0000) 2022-09-08T07:01:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:55 smithi055 ceph-mon[97033]: pgmap v18272: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:55 smithi059 ceph-mon[97450]: pgmap v18272: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:56.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:56.155985+0000) 2022-09-08T07:01:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:56.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:56.100715+0000) 2022-09-08T07:01:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:57.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:57.156214+0000) 2022-09-08T07:01:57.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:57.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:57.100878+0000) 2022-09-08T07:01:57.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:57 smithi055 ceph-mon[97033]: pgmap v18273: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:01:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:57 smithi059 ceph-mon[97450]: pgmap v18273: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:01:58.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:58.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:58.101099+0000) 2022-09-08T07:01:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:58.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:58.156383+0000) 2022-09-08T07:01:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:01:59.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:59.156583+0000) 2022-09-08T07:01:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:01:59.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:01:59.101299+0000) 2022-09-08T07:01:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:01:59 smithi055 ceph-mon[97033]: pgmap v18274: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:01:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:01:59 smithi059 ceph-mon[97450]: pgmap v18274: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:00.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:00.156766+0000) 2022-09-08T07:02:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:00.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:00.101533+0000) 2022-09-08T07:02:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:01.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:01.156922+0000) 2022-09-08T07:02:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:01.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:01.101714+0000) 2022-09-08T07:02:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:01 smithi055 ceph-mon[97033]: pgmap v18275: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:01 smithi059 ceph-mon[97450]: pgmap v18275: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:02.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:02.157104+0000) 2022-09-08T07:02:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:02.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:02.101973+0000) 2022-09-08T07:02:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:03.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:03.157248+0000) 2022-09-08T07:02:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:03.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:03.102219+0000) 2022-09-08T07:02:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:03 smithi055 ceph-mon[97033]: pgmap v18276: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:02:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:03 smithi059 ceph-mon[97450]: pgmap v18276: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:02:04.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:04.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:04.102368+0000) 2022-09-08T07:02:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:04.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:04.157406+0000) 2022-09-08T07:02:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:05.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:05.157620+0000) 2022-09-08T07:02:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:05.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:05.102567+0000) 2022-09-08T07:02:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:05 smithi055 ceph-mon[97033]: pgmap v18277: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:05 smithi059 ceph-mon[97450]: pgmap v18277: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:06.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:06.157882+0000) 2022-09-08T07:02:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:06.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:06.102799+0000) 2022-09-08T07:02:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:02:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:02:07.105 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:02:06.848+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:02:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:07.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:07.158128+0000) 2022-09-08T07:02:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:07.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:07.102938+0000) 2022-09-08T07:02:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:07 smithi055 ceph-mon[97033]: pgmap v18278: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:02:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:02:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:07 smithi059 ceph-mon[97450]: pgmap v18278: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:02:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:02:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:08.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:08.158290+0000) 2022-09-08T07:02:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:08.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:08.103082+0000) 2022-09-08T07:02:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:09.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:09.158475+0000) 2022-09-08T07:02:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:09.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:09.103262+0000) 2022-09-08T07:02:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:09 smithi055 ceph-mon[97033]: pgmap v18279: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:02:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:09 smithi059 ceph-mon[97450]: pgmap v18279: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:02:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:10.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:10.158740+0000) 2022-09-08T07:02:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:10.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:10.103475+0000) 2022-09-08T07:02:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:10 smithi055 ceph-mon[97033]: osdmap e1220: 12 total, 11 up, 11 in 2022-09-08T07:02:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:10 smithi059 ceph-mon[97450]: osdmap e1220: 12 total, 11 up, 11 in 2022-09-08T07:02:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:11.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:11.158922+0000) 2022-09-08T07:02:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:11.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:11.103716+0000) 2022-09-08T07:02:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:11 smithi055 ceph-mon[97033]: pgmap v18281: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:02:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:11 smithi059 ceph-mon[97450]: pgmap v18281: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:02:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:12.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:12.159114+0000) 2022-09-08T07:02:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:12.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:12.103937+0000) 2022-09-08T07:02:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:13.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:13.159355+0000) 2022-09-08T07:02:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:13.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:13.104134+0000) 2022-09-08T07:02:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:13 smithi055 ceph-mon[97033]: pgmap v18282: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:02:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:13 smithi059 ceph-mon[97450]: pgmap v18282: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:02:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:14.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:14.159577+0000) 2022-09-08T07:02:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:14.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:14.104355+0000) 2022-09-08T07:02:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:15.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:15.104560+0000) 2022-09-08T07:02:15.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:15.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:15.159823+0000) 2022-09-08T07:02:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:15 smithi055 ceph-mon[97033]: pgmap v18283: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:02:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:15 smithi059 ceph-mon[97450]: pgmap v18283: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:02:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:16.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:16.160071+0000) 2022-09-08T07:02:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:16.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:16.104728+0000) 2022-09-08T07:02:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:17.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:17.160303+0000) 2022-09-08T07:02:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:17.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:17.104894+0000) 2022-09-08T07:02:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:17 smithi055 ceph-mon[97033]: pgmap v18284: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:02:17.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:17 smithi055 ceph-mon[97033]: osdmap e1221: 12 total, 11 up, 11 in 2022-09-08T07:02:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:17 smithi059 ceph-mon[97450]: pgmap v18284: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:02:17.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:17 smithi059 ceph-mon[97450]: osdmap e1221: 12 total, 11 up, 11 in 2022-09-08T07:02:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:18.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:18.160520+0000) 2022-09-08T07:02:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:18.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:18.105140+0000) 2022-09-08T07:02:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:19.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:19.160745+0000) 2022-09-08T07:02:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:19.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:19.105318+0000) 2022-09-08T07:02:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:19 smithi055 ceph-mon[97033]: pgmap v18286: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 249 B/s wr, 7 op/s 2022-09-08T07:02:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:19 smithi059 ceph-mon[97450]: pgmap v18286: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 249 B/s wr, 7 op/s 2022-09-08T07:02:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:20.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:20.160932+0000) 2022-09-08T07:02:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:20.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:20.105474+0000) 2022-09-08T07:02:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:21.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:21.161195+0000) 2022-09-08T07:02:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:21.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:21.105621+0000) 2022-09-08T07:02:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:21 smithi055 ceph-mon[97033]: pgmap v18287: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:02:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:21 smithi059 ceph-mon[97450]: pgmap v18287: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:02:22.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:02:21.862+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:02:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:22.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:22.105855+0000) 2022-09-08T07:02:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:22.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:22.161436+0000) 2022-09-08T07:02:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:23.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:23.161721+0000) 2022-09-08T07:02:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:23.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:23.106083+0000) 2022-09-08T07:02:23.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:23 smithi055 ceph-mon[97033]: pgmap v18288: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:02:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:23 smithi059 ceph-mon[97450]: pgmap v18288: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:02:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:24.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:24.161964+0000) 2022-09-08T07:02:24.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:24.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:24.106310+0000) 2022-09-08T07:02:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:25.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:25.162234+0000) 2022-09-08T07:02:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:25.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:25.106505+0000) 2022-09-08T07:02:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:25 smithi055 ceph-mon[97033]: pgmap v18289: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:02:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:25 smithi059 ceph-mon[97450]: pgmap v18289: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:02:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:26.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:26.162392+0000) 2022-09-08T07:02:26.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:26.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:26.106739+0000) 2022-09-08T07:02:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:27.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:27.162599+0000) 2022-09-08T07:02:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:27.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:27.106937+0000) 2022-09-08T07:02:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:27 smithi059 ceph-mon[97450]: pgmap v18290: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:02:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:27 smithi055 ceph-mon[97033]: pgmap v18290: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:02:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:28.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:28.162778+0000) 2022-09-08T07:02:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:28.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:28.107135+0000) 2022-09-08T07:02:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:29.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:29.162979+0000) 2022-09-08T07:02:29.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:29.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:29.107272+0000) 2022-09-08T07:02:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:29 smithi059 ceph-mon[97450]: pgmap v18291: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:02:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:29 smithi055 ceph-mon[97033]: pgmap v18291: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:02:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:30.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:30.163182+0000) 2022-09-08T07:02:30.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:30.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:30.107419+0000) 2022-09-08T07:02:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:31.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:31.163404+0000) 2022-09-08T07:02:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:31.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:31.107551+0000) 2022-09-08T07:02:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:31 smithi059 ceph-mon[97450]: pgmap v18292: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:31 smithi055 ceph-mon[97033]: pgmap v18292: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:32.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:32.163559+0000) 2022-09-08T07:02:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:32.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:32.107733+0000) 2022-09-08T07:02:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:33.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:33.163750+0000) 2022-09-08T07:02:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:33.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:33.107964+0000) 2022-09-08T07:02:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:33 smithi059 ceph-mon[97450]: pgmap v18293: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:02:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:33 smithi055 ceph-mon[97033]: pgmap v18293: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:02:34.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:34.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:34.108225+0000) 2022-09-08T07:02:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:34.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:34.163936+0000) 2022-09-08T07:02:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:35.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:35.164087+0000) 2022-09-08T07:02:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:35.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:35.108477+0000) 2022-09-08T07:02:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:35 smithi055 ceph-mon[97033]: pgmap v18294: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:35.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:02:35.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:02:35.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:02:35.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:02:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:35 smithi059 ceph-mon[97450]: pgmap v18294: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:35.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:02:35.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:02:35.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:02:35.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:02:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:36.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:36.164296+0000) 2022-09-08T07:02:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:36.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:36.108684+0000) 2022-09-08T07:02:37.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:02:36.866+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:02:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:37.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:37.108813+0000) 2022-09-08T07:02:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:37.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:37.164520+0000) 2022-09-08T07:02:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:37 smithi059 ceph-mon[97450]: pgmap v18295: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:02:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:37 smithi055 ceph-mon[97033]: pgmap v18295: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:02:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:38.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:38.109010+0000) 2022-09-08T07:02:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:38.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:38.164757+0000) 2022-09-08T07:02:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:39.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:39.164979+0000) 2022-09-08T07:02:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:39.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:39.109200+0000) 2022-09-08T07:02:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:39 smithi059 ceph-mon[97450]: pgmap v18296: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:39 smithi055 ceph-mon[97033]: pgmap v18296: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:40.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:40.165157+0000) 2022-09-08T07:02:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:40.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:40.109431+0000) 2022-09-08T07:02:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:41.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:41.165385+0000) 2022-09-08T07:02:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:41.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:41.109667+0000) 2022-09-08T07:02:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:41 smithi059 ceph-mon[97450]: pgmap v18297: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:41.988 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:41 smithi055 ceph-mon[97033]: pgmap v18297: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:42.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:42.165674+0000) 2022-09-08T07:02:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:42.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:42.109829+0000) 2022-09-08T07:02:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:43.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:43.165923+0000) 2022-09-08T07:02:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:43.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:43.110017+0000) 2022-09-08T07:02:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:43 smithi059 ceph-mon[97450]: pgmap v18298: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:02:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:43 smithi055 ceph-mon[97033]: pgmap v18298: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:02:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:44.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:44.166166+0000) 2022-09-08T07:02:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:44.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:44.110259+0000) 2022-09-08T07:02:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:45.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:45.166395+0000) 2022-09-08T07:02:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:45.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:45.110431+0000) 2022-09-08T07:02:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:45 smithi059 ceph-mon[97450]: pgmap v18299: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:45.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:45 smithi055 ceph-mon[97033]: pgmap v18299: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:46.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:46.166586+0000) 2022-09-08T07:02:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:46.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:46.110626+0000) 2022-09-08T07:02:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:47.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:47.166810+0000) 2022-09-08T07:02:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:47.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:47.110826+0000) 2022-09-08T07:02:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:47 smithi059 ceph-mon[97450]: pgmap v18300: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:02:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:47 smithi055 ceph-mon[97033]: pgmap v18300: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:02:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:48.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:48.167053+0000) 2022-09-08T07:02:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:48.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:48.111073+0000) 2022-09-08T07:02:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:49.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:49.167228+0000) 2022-09-08T07:02:49.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:49.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:49.111276+0000) 2022-09-08T07:02:49.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:49 smithi059 ceph-mon[97450]: pgmap v18301: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:49.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:49 smithi055 ceph-mon[97033]: pgmap v18301: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:50.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:50.167436+0000) 2022-09-08T07:02:50.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:50.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:50.111473+0000) 2022-09-08T07:02:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:51.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:51.167664+0000) 2022-09-08T07:02:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:51.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:51.111745+0000) 2022-09-08T07:02:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:51 smithi059 ceph-mon[97450]: pgmap v18302: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:51 smithi055 ceph-mon[97033]: pgmap v18302: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:52.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:52.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:52.112002+0000) 2022-09-08T07:02:52.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:02:51.882+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:02:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:52.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:52.167855+0000) 2022-09-08T07:02:53.170 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:53.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:53.112212+0000) 2022-09-08T07:02:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:53.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:53.168086+0000) 2022-09-08T07:02:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:53 smithi059 ceph-mon[97450]: pgmap v18303: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:02:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:53 smithi055 ceph-mon[97033]: pgmap v18303: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:02:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:54.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:54.168322+0000) 2022-09-08T07:02:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:54.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:54.112416+0000) 2022-09-08T07:02:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:55.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:55.168537+0000) 2022-09-08T07:02:55.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:55.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:55.112588+0000) 2022-09-08T07:02:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:55 smithi059 ceph-mon[97450]: pgmap v18304: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:55 smithi055 ceph-mon[97033]: pgmap v18304: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:02:56.115 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:56.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:56.112703+0000) 2022-09-08T07:02:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:56.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:56.168689+0000) 2022-09-08T07:02:57.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:57.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:57.112840+0000) 2022-09-08T07:02:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:57.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:57.168829+0000) 2022-09-08T07:02:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:57 smithi055 ceph-mon[97033]: pgmap v18305: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:02:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:57 smithi059 ceph-mon[97450]: pgmap v18305: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:02:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:58.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:58.169063+0000) 2022-09-08T07:02:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:58.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:58.113061+0000) 2022-09-08T07:02:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:02:59.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:59.169289+0000) 2022-09-08T07:02:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:02:59.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:02:59.113260+0000) 2022-09-08T07:02:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:02:59 smithi055 ceph-mon[97033]: pgmap v18306: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:02:59 smithi059 ceph-mon[97450]: pgmap v18306: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:00.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:00.169518+0000) 2022-09-08T07:03:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:00.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:00.113464+0000) 2022-09-08T07:03:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:01.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:01.169710+0000) 2022-09-08T07:03:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:01.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:01.113708+0000) 2022-09-08T07:03:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:01 smithi055 ceph-mon[97033]: pgmap v18307: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:01 smithi059 ceph-mon[97450]: pgmap v18307: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:02.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:02.169931+0000) 2022-09-08T07:03:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:02.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:02.113942+0000) 2022-09-08T07:03:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:03.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:03.170160+0000) 2022-09-08T07:03:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:03.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:03.114077+0000) 2022-09-08T07:03:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:03 smithi055 ceph-mon[97033]: pgmap v18308: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:03:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:03 smithi059 ceph-mon[97450]: pgmap v18308: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:03:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:04.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:04.170370+0000) 2022-09-08T07:03:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:04.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:04.114322+0000) 2022-09-08T07:03:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:05.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:05.170560+0000) 2022-09-08T07:03:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:05.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:05.114488+0000) 2022-09-08T07:03:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:05 smithi055 ceph-mon[97033]: pgmap v18309: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:05 smithi059 ceph-mon[97450]: pgmap v18309: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:06.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:06.170811+0000) 2022-09-08T07:03:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:06.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:06.114679+0000) 2022-09-08T07:03:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:03:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:03:07.152 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:03:06.897+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:03:07.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:07.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:07.114950+0000) 2022-09-08T07:03:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:07.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:07.171005+0000) 2022-09-08T07:03:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:07 smithi055 ceph-mon[97033]: pgmap v18310: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:03:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:03:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:07 smithi059 ceph-mon[97450]: pgmap v18310: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:03:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:03:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:08.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:08.171167+0000) 2022-09-08T07:03:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:08.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:08.115153+0000) 2022-09-08T07:03:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:09.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:09.171377+0000) 2022-09-08T07:03:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:09.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:09.115369+0000) 2022-09-08T07:03:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:09 smithi055 ceph-mon[97033]: pgmap v18311: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:03:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:09 smithi059 ceph-mon[97450]: pgmap v18311: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:03:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:10.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:10.171605+0000) 2022-09-08T07:03:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:10.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:10.115595+0000) 2022-09-08T07:03:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:10 smithi055 ceph-mon[97033]: osdmap e1222: 12 total, 11 up, 11 in 2022-09-08T07:03:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:10 smithi059 ceph-mon[97450]: osdmap e1222: 12 total, 11 up, 11 in 2022-09-08T07:03:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:11.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:11.171823+0000) 2022-09-08T07:03:11.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:11.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:11.115830+0000) 2022-09-08T07:03:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:11 smithi055 ceph-mon[97033]: pgmap v18313: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:03:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:11 smithi059 ceph-mon[97450]: pgmap v18313: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:03:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:12.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:12.172095+0000) 2022-09-08T07:03:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:12.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:12.116064+0000) 2022-09-08T07:03:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:13.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:13.172351+0000) 2022-09-08T07:03:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:13.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:13.116211+0000) 2022-09-08T07:03:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:13 smithi055 ceph-mon[97033]: pgmap v18314: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:03:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:13 smithi059 ceph-mon[97450]: pgmap v18314: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:03:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:14.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:14.172593+0000) 2022-09-08T07:03:14.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:14.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:14.116377+0000) 2022-09-08T07:03:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:15.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:15.172771+0000) 2022-09-08T07:03:15.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:15.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:15.116505+0000) 2022-09-08T07:03:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:15 smithi055 ceph-mon[97033]: pgmap v18315: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:03:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:15 smithi059 ceph-mon[97450]: pgmap v18315: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:03:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:16.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:16.173017+0000) 2022-09-08T07:03:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:16.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:16.116741+0000) 2022-09-08T07:03:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:17.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:17.173244+0000) 2022-09-08T07:03:17.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:17.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:17.116963+0000) 2022-09-08T07:03:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:17 smithi055 ceph-mon[97033]: pgmap v18316: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:03:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:17 smithi055 ceph-mon[97033]: osdmap e1223: 12 total, 11 up, 11 in 2022-09-08T07:03:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:17 smithi059 ceph-mon[97450]: pgmap v18316: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:03:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:17 smithi059 ceph-mon[97450]: osdmap e1223: 12 total, 11 up, 11 in 2022-09-08T07:03:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:18.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:18.173475+0000) 2022-09-08T07:03:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:18.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:18.117143+0000) 2022-09-08T07:03:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:19.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:19.173610+0000) 2022-09-08T07:03:19.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:19.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:19.117291+0000) 2022-09-08T07:03:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:19 smithi055 ceph-mon[97033]: pgmap v18318: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T07:03:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:19 smithi059 ceph-mon[97450]: pgmap v18318: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 253 B/s wr, 7 op/s 2022-09-08T07:03:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:20.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:20.173747+0000) 2022-09-08T07:03:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:20.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:20.117441+0000) 2022-09-08T07:03:21.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:21.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:21.117597+0000) 2022-09-08T07:03:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:21.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:21.173867+0000) 2022-09-08T07:03:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:21 smithi055 ceph-mon[97033]: pgmap v18319: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:03:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:21 smithi059 ceph-mon[97450]: pgmap v18319: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:03:22.176 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:03:21.912+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:03:22.177 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:22.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:22.117720+0000) 2022-09-08T07:03:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:22.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:22.174109+0000) 2022-09-08T07:03:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:23.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:23.174347+0000) 2022-09-08T07:03:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:23.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:23.117932+0000) 2022-09-08T07:03:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:23 smithi055 ceph-mon[97033]: pgmap v18320: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:03:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:23 smithi059 ceph-mon[97450]: pgmap v18320: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:03:24.176 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:24.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:24.118114+0000) 2022-09-08T07:03:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:24.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:24.174559+0000) 2022-09-08T07:03:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:25.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:25.174797+0000) 2022-09-08T07:03:25.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:25.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:25.118253+0000) 2022-09-08T07:03:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:25 smithi055 ceph-mon[97033]: pgmap v18321: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:03:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:25 smithi059 ceph-mon[97450]: pgmap v18321: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:03:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:26.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:26.174985+0000) 2022-09-08T07:03:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:26.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:26.118427+0000) 2022-09-08T07:03:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:27.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:27.175123+0000) 2022-09-08T07:03:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:27.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:27.118582+0000) 2022-09-08T07:03:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:27 smithi055 ceph-mon[97033]: pgmap v18322: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:03:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:27 smithi059 ceph-mon[97450]: pgmap v18322: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:03:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:28.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:28.175359+0000) 2022-09-08T07:03:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:28.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:28.118730+0000) 2022-09-08T07:03:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:29.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:29.175576+0000) 2022-09-08T07:03:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:29.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:29.118866+0000) 2022-09-08T07:03:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:29 smithi055 ceph-mon[97033]: pgmap v18323: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:03:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:29 smithi059 ceph-mon[97450]: pgmap v18323: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:03:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:30.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:30.175805+0000) 2022-09-08T07:03:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:30.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:30.119030+0000) 2022-09-08T07:03:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:31.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:31.176023+0000) 2022-09-08T07:03:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:31.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:31.119243+0000) 2022-09-08T07:03:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:31 smithi055 ceph-mon[97033]: pgmap v18324: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:31 smithi059 ceph-mon[97450]: pgmap v18324: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:32.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:32.176306+0000) 2022-09-08T07:03:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:32.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:32.119440+0000) 2022-09-08T07:03:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:33.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:33.176530+0000) 2022-09-08T07:03:33.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:33.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:33.119695+0000) 2022-09-08T07:03:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:33 smithi055 ceph-mon[97033]: pgmap v18325: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:03:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:33 smithi059 ceph-mon[97450]: pgmap v18325: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:03:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:34.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:34.176752+0000) 2022-09-08T07:03:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:34.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:34.119931+0000) 2022-09-08T07:03:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:35.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:35.176950+0000) 2022-09-08T07:03:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:35.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:35.120077+0000) 2022-09-08T07:03:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:35 smithi055 ceph-mon[97033]: pgmap v18326: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:35.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:03:35.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:03:35.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:03:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:35 smithi059 ceph-mon[97450]: pgmap v18326: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:03:35.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:03:35.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:03:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:36.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:36.177171+0000) 2022-09-08T07:03:36.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:36.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:36.120238+0000) 2022-09-08T07:03:37.180 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:03:36.914+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:03:37.181 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:37.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:37.120436+0000) 2022-09-08T07:03:37.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:36 smithi055 ceph-mon[97033]: pgmap v18327: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:03:37.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:03:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:36 smithi059 ceph-mon[97450]: pgmap v18327: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:03:37.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:03:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:37.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:37.177418+0000) 2022-09-08T07:03:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:38.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:38.177615+0000) 2022-09-08T07:03:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:38.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:38.120745+0000) 2022-09-08T07:03:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:39.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:39.177793+0000) 2022-09-08T07:03:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:39.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:39.120913+0000) 2022-09-08T07:03:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:39 smithi055 ceph-mon[97033]: pgmap v18328: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:39 smithi059 ceph-mon[97450]: pgmap v18328: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:40.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:40.178038+0000) 2022-09-08T07:03:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:40.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:40.121112+0000) 2022-09-08T07:03:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:41.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:41.178262+0000) 2022-09-08T07:03:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:41.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:41.121300+0000) 2022-09-08T07:03:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:41 smithi055 ceph-mon[97033]: pgmap v18329: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:41 smithi059 ceph-mon[97450]: pgmap v18329: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:42.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:42.178508+0000) 2022-09-08T07:03:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:42.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:42.121480+0000) 2022-09-08T07:03:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:43.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:43.178734+0000) 2022-09-08T07:03:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:43.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:43.121610+0000) 2022-09-08T07:03:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:43 smithi055 ceph-mon[97033]: pgmap v18330: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:03:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:43 smithi059 ceph-mon[97450]: pgmap v18330: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:03:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:44.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:44.121740+0000) 2022-09-08T07:03:44.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:44.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:44.178867+0000) 2022-09-08T07:03:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:45.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:45.121886+0000) 2022-09-08T07:03:45.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:45.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:45.179002+0000) 2022-09-08T07:03:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:45 smithi055 ceph-mon[97033]: pgmap v18331: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:45 smithi059 ceph-mon[97450]: pgmap v18331: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:46.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:46.179155+0000) 2022-09-08T07:03:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:46.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:46.122033+0000) 2022-09-08T07:03:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:47.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:47.179305+0000) 2022-09-08T07:03:47.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:47.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:47.122198+0000) 2022-09-08T07:03:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:47 smithi055 ceph-mon[97033]: pgmap v18332: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:03:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:47 smithi059 ceph-mon[97450]: pgmap v18332: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:03:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:48.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:48.179533+0000) 2022-09-08T07:03:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:48.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:48.122361+0000) 2022-09-08T07:03:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:49.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:49.179758+0000) 2022-09-08T07:03:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:49.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:49.122523+0000) 2022-09-08T07:03:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:49 smithi055 ceph-mon[97033]: pgmap v18333: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:49 smithi059 ceph-mon[97450]: pgmap v18333: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:50.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:50.180004+0000) 2022-09-08T07:03:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:50.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:50.122683+0000) 2022-09-08T07:03:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:51.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:51.180212+0000) 2022-09-08T07:03:51.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:51.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:51.122810+0000) 2022-09-08T07:03:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:51 smithi055 ceph-mon[97033]: pgmap v18334: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:51 smithi059 ceph-mon[97450]: pgmap v18334: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:52.182 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:03:51.917+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:03:52.183 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:52.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:52.122968+0000) 2022-09-08T07:03:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:52.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:52.180476+0000) 2022-09-08T07:03:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:53.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:53.180762+0000) 2022-09-08T07:03:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:53.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:53.123146+0000) 2022-09-08T07:03:53.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:53 smithi055 ceph-mon[97033]: pgmap v18335: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:03:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:53 smithi059 ceph-mon[97450]: pgmap v18335: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:03:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:54.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:54.181028+0000) 2022-09-08T07:03:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:54.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:54.123294+0000) 2022-09-08T07:03:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:55.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:55.181269+0000) 2022-09-08T07:03:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:55.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:55.123427+0000) 2022-09-08T07:03:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:55 smithi055 ceph-mon[97033]: pgmap v18336: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:55 smithi059 ceph-mon[97450]: pgmap v18336: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:56.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:56.181503+0000) 2022-09-08T07:03:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:56.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:56.123557+0000) 2022-09-08T07:03:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:57.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:57.181778+0000) 2022-09-08T07:03:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:57.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:57.123711+0000) 2022-09-08T07:03:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:57 smithi055 ceph-mon[97033]: pgmap v18337: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:03:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:57 smithi059 ceph-mon[97450]: pgmap v18337: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:03:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:58.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:58.182057+0000) 2022-09-08T07:03:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:58.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:58.123850+0000) 2022-09-08T07:03:59.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:03:59.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:59.123988+0000) 2022-09-08T07:03:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:03:59.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:03:59.182209+0000) 2022-09-08T07:03:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:03:59 smithi055 ceph-mon[97033]: pgmap v18338: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:03:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:03:59 smithi059 ceph-mon[97450]: pgmap v18338: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:00.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:00.182432+0000) 2022-09-08T07:04:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:00.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:00.124155+0000) 2022-09-08T07:04:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:01.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:01.182587+0000) 2022-09-08T07:04:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:01.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:01.124309+0000) 2022-09-08T07:04:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:01 smithi055 ceph-mon[97033]: pgmap v18339: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:01 smithi059 ceph-mon[97450]: pgmap v18339: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:02.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:02.182738+0000) 2022-09-08T07:04:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:02.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:02.124434+0000) 2022-09-08T07:04:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:03.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:03.182880+0000) 2022-09-08T07:04:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:03.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:03.124558+0000) 2022-09-08T07:04:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:03 smithi055 ceph-mon[97033]: pgmap v18340: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:04:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:03 smithi059 ceph-mon[97450]: pgmap v18340: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:04:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:04.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:04.183120+0000) 2022-09-08T07:04:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:04.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:04.124734+0000) 2022-09-08T07:04:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:05.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:05.183382+0000) 2022-09-08T07:04:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:05.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:05.124966+0000) 2022-09-08T07:04:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:05 smithi055 ceph-mon[97033]: pgmap v18341: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:05 smithi059 ceph-mon[97450]: pgmap v18341: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:06.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:06.183625+0000) 2022-09-08T07:04:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:06.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:06.125223+0000) 2022-09-08T07:04:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:04:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:04:07.186 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:04:06.920+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:04:07.187 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:07.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:07.125403+0000) 2022-09-08T07:04:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:07.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:07.183928+0000) 2022-09-08T07:04:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:07 smithi055 ceph-mon[97033]: pgmap v18342: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:04:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:04:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:07 smithi059 ceph-mon[97450]: pgmap v18342: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:04:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:04:08.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:08.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:08.184197+0000) 2022-09-08T07:04:08.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:08.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:08.125557+0000) 2022-09-08T07:04:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:09.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:09.184422+0000) 2022-09-08T07:04:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:09.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:09.125718+0000) 2022-09-08T07:04:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:09 smithi055 ceph-mon[97033]: pgmap v18343: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:04:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:09 smithi059 ceph-mon[97450]: pgmap v18343: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:04:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:10.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:10.184609+0000) 2022-09-08T07:04:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:10.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:10.125962+0000) 2022-09-08T07:04:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:10 smithi055 ceph-mon[97033]: osdmap e1224: 12 total, 11 up, 11 in 2022-09-08T07:04:10.790 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:10 smithi059 ceph-mon[97450]: osdmap e1224: 12 total, 11 up, 11 in 2022-09-08T07:04:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:11.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:11.184822+0000) 2022-09-08T07:04:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:11.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:11.126128+0000) 2022-09-08T07:04:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:11 smithi055 ceph-mon[97033]: pgmap v18345: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:04:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:11 smithi059 ceph-mon[97450]: pgmap v18345: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:04:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:12.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:12.185069+0000) 2022-09-08T07:04:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:12.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:12.126287+0000) 2022-09-08T07:04:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:13.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:13.185232+0000) 2022-09-08T07:04:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:13.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:13.126455+0000) 2022-09-08T07:04:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:13 smithi055 ceph-mon[97033]: pgmap v18346: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:04:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:13 smithi059 ceph-mon[97450]: pgmap v18346: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:04:14.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:14.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:14.126652+0000) 2022-09-08T07:04:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:14.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:14.185397+0000) 2022-09-08T07:04:15.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:15.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:15.126754+0000) 2022-09-08T07:04:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:15.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:15.185592+0000) 2022-09-08T07:04:15.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:15 smithi055 ceph-mon[97033]: pgmap v18347: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:04:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:15 smithi059 ceph-mon[97450]: pgmap v18347: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:04:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:16.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:16.185813+0000) 2022-09-08T07:04:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:16.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:16.126965+0000) 2022-09-08T07:04:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:17.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:17.186052+0000) 2022-09-08T07:04:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:17.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:17.127114+0000) 2022-09-08T07:04:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:17 smithi055 ceph-mon[97033]: pgmap v18348: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:04:17.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:17 smithi055 ceph-mon[97033]: osdmap e1225: 12 total, 11 up, 11 in 2022-09-08T07:04:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:17 smithi059 ceph-mon[97450]: pgmap v18348: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:04:17.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:17 smithi059 ceph-mon[97450]: osdmap e1225: 12 total, 11 up, 11 in 2022-09-08T07:04:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:18.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:18.186205+0000) 2022-09-08T07:04:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:18.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:18.127268+0000) 2022-09-08T07:04:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:19.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:19.127463+0000) 2022-09-08T07:04:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:19.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:19.186417+0000) 2022-09-08T07:04:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:19 smithi055 ceph-mon[97033]: pgmap v18350: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 246 B/s wr, 7 op/s 2022-09-08T07:04:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:19 smithi059 ceph-mon[97450]: pgmap v18350: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 246 B/s wr, 7 op/s 2022-09-08T07:04:20.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:20.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:20.127627+0000) 2022-09-08T07:04:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:20.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:20.186692+0000) 2022-09-08T07:04:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:21.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:21.186929+0000) 2022-09-08T07:04:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:21.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:21.127862+0000) 2022-09-08T07:04:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:21 smithi055 ceph-mon[97033]: pgmap v18351: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:04:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:21 smithi059 ceph-mon[97450]: pgmap v18351: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:04:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:22.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:22.187122+0000) 2022-09-08T07:04:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:22.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:22.128039+0000) 2022-09-08T07:04:22.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:04:21.937+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:04:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:23.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:23.187334+0000) 2022-09-08T07:04:23.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:23.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:23.128210+0000) 2022-09-08T07:04:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:23 smithi055 ceph-mon[97033]: pgmap v18352: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:04:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:23 smithi059 ceph-mon[97450]: pgmap v18352: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:04:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:24.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:24.187551+0000) 2022-09-08T07:04:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:24.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:24.128459+0000) 2022-09-08T07:04:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:25.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:25.187797+0000) 2022-09-08T07:04:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:25.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:25.128686+0000) 2022-09-08T07:04:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:25 smithi055 ceph-mon[97033]: pgmap v18353: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:04:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:25 smithi059 ceph-mon[97450]: pgmap v18353: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:04:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:26.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:26.188032+0000) 2022-09-08T07:04:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:26.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:26.128902+0000) 2022-09-08T07:04:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:27.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:27.188232+0000) 2022-09-08T07:04:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:27.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:27.129067+0000) 2022-09-08T07:04:27.742 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:27 smithi055 ceph-mon[97033]: pgmap v18354: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:04:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:27 smithi059 ceph-mon[97450]: pgmap v18354: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:04:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:28.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:28.188509+0000) 2022-09-08T07:04:28.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:28.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:28.129250+0000) 2022-09-08T07:04:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:29.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:29.188721+0000) 2022-09-08T07:04:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:29.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:29.129415+0000) 2022-09-08T07:04:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:29 smithi055 ceph-mon[97033]: pgmap v18355: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:04:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:29 smithi059 ceph-mon[97450]: pgmap v18355: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:04:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:30.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:30.188923+0000) 2022-09-08T07:04:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:30.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:30.129655+0000) 2022-09-08T07:04:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:31.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:31.189120+0000) 2022-09-08T07:04:31.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:31.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:31.129792+0000) 2022-09-08T07:04:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:31 smithi055 ceph-mon[97033]: pgmap v18356: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:31 smithi059 ceph-mon[97450]: pgmap v18356: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:32.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:32.189373+0000) 2022-09-08T07:04:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:32.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:32.129969+0000) 2022-09-08T07:04:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:33.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:33.130122+0000) 2022-09-08T07:04:33.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:33.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:33.189666+0000) 2022-09-08T07:04:33.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:33 smithi055 ceph-mon[97033]: pgmap v18357: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:04:33.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:33 smithi059 ceph-mon[97450]: pgmap v18357: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:04:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:34.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:34.130274+0000) 2022-09-08T07:04:34.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:34.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:34.189839+0000) 2022-09-08T07:04:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:35.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:35.130412+0000) 2022-09-08T07:04:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:35.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:35.190071+0000) 2022-09-08T07:04:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:35 smithi055 ceph-mon[97033]: pgmap v18358: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:35 smithi059 ceph-mon[97450]: pgmap v18358: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:36.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:36.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:36.130632+0000) 2022-09-08T07:04:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:36.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:36.190240+0000) 2022-09-08T07:04:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:04:36.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:04:36.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:04:36.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:04:36.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:04:36.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:04:37.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:04:36.941+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:04:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:37.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:37.190438+0000) 2022-09-08T07:04:37.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:37.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:37.130829+0000) 2022-09-08T07:04:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:37 smithi059 ceph-mon[97450]: pgmap v18359: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:04:37.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:04:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:37 smithi055 ceph-mon[97033]: pgmap v18359: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:04:37.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:04:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:38.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:38.190709+0000) 2022-09-08T07:04:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:38.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:38.131056+0000) 2022-09-08T07:04:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:39.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:39.131243+0000) 2022-09-08T07:04:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:39.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:39.190875+0000) 2022-09-08T07:04:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:39 smithi059 ceph-mon[97450]: pgmap v18360: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:39 smithi055 ceph-mon[97033]: pgmap v18360: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:40.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:40.191103+0000) 2022-09-08T07:04:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:40.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:40.131435+0000) 2022-09-08T07:04:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:41.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:41.191324+0000) 2022-09-08T07:04:41.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:41.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:41.131600+0000) 2022-09-08T07:04:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:41 smithi059 ceph-mon[97450]: pgmap v18361: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:41 smithi055 ceph-mon[97033]: pgmap v18361: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:42.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:42.191554+0000) 2022-09-08T07:04:42.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:42.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:42.131823+0000) 2022-09-08T07:04:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:43.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:43.191758+0000) 2022-09-08T07:04:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:43.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:43.132086+0000) 2022-09-08T07:04:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:43 smithi059 ceph-mon[97450]: pgmap v18362: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:04:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:43 smithi055 ceph-mon[97033]: pgmap v18362: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:04:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:44.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:44.191949+0000) 2022-09-08T07:04:44.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:44.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:44.132267+0000) 2022-09-08T07:04:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:45.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:45.192137+0000) 2022-09-08T07:04:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:45.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:45.132403+0000) 2022-09-08T07:04:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:45 smithi059 ceph-mon[97450]: pgmap v18363: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:45 smithi055 ceph-mon[97033]: pgmap v18363: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:46.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:46.192347+0000) 2022-09-08T07:04:46.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:46.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:46.132552+0000) 2022-09-08T07:04:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:47.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:47.132767+0000) 2022-09-08T07:04:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:47.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:47.192571+0000) 2022-09-08T07:04:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:47 smithi059 ceph-mon[97450]: pgmap v18364: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:04:47.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:47 smithi055 ceph-mon[97033]: pgmap v18364: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:04:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:48.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:48.192757+0000) 2022-09-08T07:04:48.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:48.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:48.132952+0000) 2022-09-08T07:04:49.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:49.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:49.133103+0000) 2022-09-08T07:04:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:49.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:49.192937+0000) 2022-09-08T07:04:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:49 smithi059 ceph-mon[97450]: pgmap v18365: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:49 smithi055 ceph-mon[97033]: pgmap v18365: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:50.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:50.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:50.133302+0000) 2022-09-08T07:04:50.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:50.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:50.193091+0000) 2022-09-08T07:04:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:51.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:51.193301+0000) 2022-09-08T07:04:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:51.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:51.133505+0000) 2022-09-08T07:04:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:51 smithi059 ceph-mon[97450]: pgmap v18366: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:51 smithi055 ceph-mon[97033]: pgmap v18366: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:52.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:04:51.956+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:04:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:52.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:52.193532+0000) 2022-09-08T07:04:52.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:52.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:52.133694+0000) 2022-09-08T07:04:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:53.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:53.193717+0000) 2022-09-08T07:04:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:53.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:53.133901+0000) 2022-09-08T07:04:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:53 smithi055 ceph-mon[97033]: pgmap v18367: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:04:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:53 smithi059 ceph-mon[97450]: pgmap v18367: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:04:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:54.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:54.134108+0000) 2022-09-08T07:04:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:54.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:54.193900+0000) 2022-09-08T07:04:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:55.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:55.194175+0000) 2022-09-08T07:04:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:55.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:55.134301+0000) 2022-09-08T07:04:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:55 smithi055 ceph-mon[97033]: pgmap v18368: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:55 smithi059 ceph-mon[97450]: pgmap v18368: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:04:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:56.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:56.194424+0000) 2022-09-08T07:04:56.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:56.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:56.134447+0000) 2022-09-08T07:04:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:57.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:57.194668+0000) 2022-09-08T07:04:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:57.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:57.134675+0000) 2022-09-08T07:04:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:57 smithi055 ceph-mon[97033]: pgmap v18369: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:04:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:57 smithi059 ceph-mon[97450]: pgmap v18369: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:04:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:58.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:58.194881+0000) 2022-09-08T07:04:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:58.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:58.134891+0000) 2022-09-08T07:04:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:04:59.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:59.195052+0000) 2022-09-08T07:04:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:04:59.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:04:59.135120+0000) 2022-09-08T07:04:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:04:59 smithi055 ceph-mon[97033]: pgmap v18370: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:04:59 smithi059 ceph-mon[97450]: pgmap v18370: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:00.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:00.135309+0000) 2022-09-08T07:05:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:00.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:00.195284+0000) 2022-09-08T07:05:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:01.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:01.135447+0000) 2022-09-08T07:05:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:01.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:01.195546+0000) 2022-09-08T07:05:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:01 smithi055 ceph-mon[97033]: pgmap v18371: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:01 smithi059 ceph-mon[97450]: pgmap v18371: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:02.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:02.195804+0000) 2022-09-08T07:05:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:02.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:02.135679+0000) 2022-09-08T07:05:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:03.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:03.195976+0000) 2022-09-08T07:05:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:03.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:03.135902+0000) 2022-09-08T07:05:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:03 smithi055 ceph-mon[97033]: pgmap v18372: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:05:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:03 smithi059 ceph-mon[97450]: pgmap v18372: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:05:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:04.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:04.196143+0000) 2022-09-08T07:05:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:04.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:04.136136+0000) 2022-09-08T07:05:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:05.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:05.196324+0000) 2022-09-08T07:05:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:05.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:05.136307+0000) 2022-09-08T07:05:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:05 smithi055 ceph-mon[97033]: pgmap v18373: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:05 smithi059 ceph-mon[97450]: pgmap v18373: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:06.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:06.196487+0000) 2022-09-08T07:05:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:06.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:06.136534+0000) 2022-09-08T07:05:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:05:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:05:07.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:05:06.972+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:05:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:07.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:07.196764+0000) 2022-09-08T07:05:07.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:07.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:07.136758+0000) 2022-09-08T07:05:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:07 smithi055 ceph-mon[97033]: pgmap v18374: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 47 KiB/s wr, 22 op/s 2022-09-08T07:05:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:05:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:07 smithi059 ceph-mon[97450]: pgmap v18374: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 47 KiB/s wr, 22 op/s 2022-09-08T07:05:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:05:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:08.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:08.196982+0000) 2022-09-08T07:05:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:08.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:08.136955+0000) 2022-09-08T07:05:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:09.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:09.137148+0000) 2022-09-08T07:05:09.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:09.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:09.197194+0000) 2022-09-08T07:05:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:09 smithi055 ceph-mon[97033]: pgmap v18375: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 47 KiB/s wr, 19 op/s 2022-09-08T07:05:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:09 smithi059 ceph-mon[97450]: pgmap v18375: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 47 KiB/s wr, 19 op/s 2022-09-08T07:05:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:10.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:10.197435+0000) 2022-09-08T07:05:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:10.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:10.137255+0000) 2022-09-08T07:05:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:10 smithi055 ceph-mon[97033]: osdmap e1226: 12 total, 11 up, 11 in 2022-09-08T07:05:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:10 smithi059 ceph-mon[97450]: osdmap e1226: 12 total, 11 up, 11 in 2022-09-08T07:05:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:11.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:11.197649+0000) 2022-09-08T07:05:11.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:11.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:11.137454+0000) 2022-09-08T07:05:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:11 smithi055 ceph-mon[97033]: pgmap v18377: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 56 KiB/s wr, 23 op/s 2022-09-08T07:05:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:11 smithi059 ceph-mon[97450]: pgmap v18377: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 56 KiB/s wr, 23 op/s 2022-09-08T07:05:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:12.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:12.197840+0000) 2022-09-08T07:05:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:12.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:12.137657+0000) 2022-09-08T07:05:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:13.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:13.198060+0000) 2022-09-08T07:05:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:13.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:13.137841+0000) 2022-09-08T07:05:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:13 smithi055 ceph-mon[97033]: pgmap v18378: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-08T07:05:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:13 smithi059 ceph-mon[97450]: pgmap v18378: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-08T07:05:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:14.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:14.198253+0000) 2022-09-08T07:05:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:14.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:14.138010+0000) 2022-09-08T07:05:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:15.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:15.198515+0000) 2022-09-08T07:05:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:15.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:15.138211+0000) 2022-09-08T07:05:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:15 smithi055 ceph-mon[97033]: pgmap v18379: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-08T07:05:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:15 smithi059 ceph-mon[97450]: pgmap v18379: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 17 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-08T07:05:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:16.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:16.198734+0000) 2022-09-08T07:05:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:16.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:16.138373+0000) 2022-09-08T07:05:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:17.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:17.198964+0000) 2022-09-08T07:05:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:17.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:17.138518+0000) 2022-09-08T07:05:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:17 smithi055 ceph-mon[97033]: pgmap v18380: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:05:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:17 smithi055 ceph-mon[97033]: osdmap e1227: 12 total, 11 up, 11 in 2022-09-08T07:05:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:17 smithi059 ceph-mon[97450]: pgmap v18380: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:05:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:17 smithi059 ceph-mon[97450]: osdmap e1227: 12 total, 11 up, 11 in 2022-09-08T07:05:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:18.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:18.199202+0000) 2022-09-08T07:05:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:18.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:18.138697+0000) 2022-09-08T07:05:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:19.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:19.199446+0000) 2022-09-08T07:05:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:19.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:19.138830+0000) 2022-09-08T07:05:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:19 smithi055 ceph-mon[97033]: pgmap v18382: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T07:05:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:19 smithi059 ceph-mon[97450]: pgmap v18382: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T07:05:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:20.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:20.199574+0000) 2022-09-08T07:05:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:20.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:20.138982+0000) 2022-09-08T07:05:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:21.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:21.199767+0000) 2022-09-08T07:05:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:21.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:21.139138+0000) 2022-09-08T07:05:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:21 smithi055 ceph-mon[97033]: pgmap v18383: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:05:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:21 smithi059 ceph-mon[97450]: pgmap v18383: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:05:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:22.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:22.199967+0000) 2022-09-08T07:05:22.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:05:21.975+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:05:22.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:22.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:22.139344+0000) 2022-09-08T07:05:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:23.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:23.200195+0000) 2022-09-08T07:05:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:23.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:23.139523+0000) 2022-09-08T07:05:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:23 smithi055 ceph-mon[97033]: pgmap v18384: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:05:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:23 smithi059 ceph-mon[97450]: pgmap v18384: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:05:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:24.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:24.200430+0000) 2022-09-08T07:05:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:24.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:24.139697+0000) 2022-09-08T07:05:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:25.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:25.200698+0000) 2022-09-08T07:05:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:25.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:25.139898+0000) 2022-09-08T07:05:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:25 smithi055 ceph-mon[97033]: pgmap v18385: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:05:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:25 smithi059 ceph-mon[97450]: pgmap v18385: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:05:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:26.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:26.200922+0000) 2022-09-08T07:05:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:26.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:26.140050+0000) 2022-09-08T07:05:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:27.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:27.201186+0000) 2022-09-08T07:05:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:27.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:27.140188+0000) 2022-09-08T07:05:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:27 smithi055 ceph-mon[97033]: pgmap v18386: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:05:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:27 smithi059 ceph-mon[97450]: pgmap v18386: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:05:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:28.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:28.201406+0000) 2022-09-08T07:05:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:28.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:28.140368+0000) 2022-09-08T07:05:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:29.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:29.201648+0000) 2022-09-08T07:05:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:29.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:29.140523+0000) 2022-09-08T07:05:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:29 smithi055 ceph-mon[97033]: pgmap v18387: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:05:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:29 smithi059 ceph-mon[97450]: pgmap v18387: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:05:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:30.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:30.201848+0000) 2022-09-08T07:05:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:30.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:30.140696+0000) 2022-09-08T07:05:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:31.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:31.202093+0000) 2022-09-08T07:05:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:31.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:31.140919+0000) 2022-09-08T07:05:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:31 smithi055 ceph-mon[97033]: pgmap v18388: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:31 smithi059 ceph-mon[97450]: pgmap v18388: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:32.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:32.141163+0000) 2022-09-08T07:05:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:32.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:32.202265+0000) 2022-09-08T07:05:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:33.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:33.202488+0000) 2022-09-08T07:05:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:33.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:33.141376+0000) 2022-09-08T07:05:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:33 smithi055 ceph-mon[97033]: pgmap v18389: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:05:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:33 smithi059 ceph-mon[97450]: pgmap v18389: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:05:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:34.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:34.202747+0000) 2022-09-08T07:05:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:34.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:34.141561+0000) 2022-09-08T07:05:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:35.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:35.203071+0000) 2022-09-08T07:05:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:35.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:35.141709+0000) 2022-09-08T07:05:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:35 smithi055 ceph-mon[97033]: pgmap v18390: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:35 smithi059 ceph-mon[97450]: pgmap v18390: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:36.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:36.203306+0000) 2022-09-08T07:05:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:36.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:36.141887+0000) 2022-09-08T07:05:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:05:36.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:05:36.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:05:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:05:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:05:37.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:05:37.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:05:36.977+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:05:37.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:37.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:37.203452+0000) 2022-09-08T07:05:37.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:37.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:37.142062+0000) 2022-09-08T07:05:37.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:37 smithi055 ceph-mon[97033]: pgmap v18391: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:05:37.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:05:37.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:05:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:37 smithi059 ceph-mon[97450]: pgmap v18391: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:05:38.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:05:38.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:05:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:38.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:38.203676+0000) 2022-09-08T07:05:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:38.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:38.142228+0000) 2022-09-08T07:05:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:39.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:39.203882+0000) 2022-09-08T07:05:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:39.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:39.142382+0000) 2022-09-08T07:05:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:39 smithi059 ceph-mon[97450]: pgmap v18392: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:39.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:05:39.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:05:39.884 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:39 smithi055 ceph-mon[97033]: pgmap v18392: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:39.885 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:05:39.885 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:05:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:40.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:40.204077+0000) 2022-09-08T07:05:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:40.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:40.142629+0000) 2022-09-08T07:05:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: pgmap v18393: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:41.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:05:41.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:05:41.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 148.9M 2022-09-08T07:05:41.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 156144931: error parsing value: Value '156144931' is below minimum 939524096 2022-09-08T07:05:41.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T07:05:41.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:40 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T07:05:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: pgmap v18393: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:41.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:05:41.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:05:41.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 148.9M 2022-09-08T07:05:41.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 156144931: error parsing value: Value '156144931' is below minimum 939524096 2022-09-08T07:05:41.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:05:41.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T07:05:41.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:40 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T07:05:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:41.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:41.204339+0000) 2022-09-08T07:05:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:41.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:41.142874+0000) 2022-09-08T07:05:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:42.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:42.204531+0000) 2022-09-08T07:05:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:42.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:42.143095+0000) 2022-09-08T07:05:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:43.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:43.204700+0000) 2022-09-08T07:05:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:43.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:43.143268+0000) 2022-09-08T07:05:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:43 smithi055 ceph-mon[97033]: pgmap v18394: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:05:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:43 smithi059 ceph-mon[97450]: pgmap v18394: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:05:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:44.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:44.204889+0000) 2022-09-08T07:05:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:44.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:44.143418+0000) 2022-09-08T07:05:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:45.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:45.205142+0000) 2022-09-08T07:05:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:45.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:45.143560+0000) 2022-09-08T07:05:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:45 smithi055 ceph-mon[97033]: pgmap v18395: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:45 smithi059 ceph-mon[97450]: pgmap v18395: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:46.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:46.205361+0000) 2022-09-08T07:05:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:46.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:46.143759+0000) 2022-09-08T07:05:47.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:47.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:47.143977+0000) 2022-09-08T07:05:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:47.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:47.205603+0000) 2022-09-08T07:05:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:47 smithi055 ceph-mon[97033]: pgmap v18396: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:05:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:47 smithi059 ceph-mon[97450]: pgmap v18396: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:05:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:48.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:48.205870+0000) 2022-09-08T07:05:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:48.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:48.144187+0000) 2022-09-08T07:05:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:49.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:49.206093+0000) 2022-09-08T07:05:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:49.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:49.144395+0000) 2022-09-08T07:05:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:49 smithi055 ceph-mon[97033]: pgmap v18397: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:49 smithi059 ceph-mon[97450]: pgmap v18397: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:50.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:50.206339+0000) 2022-09-08T07:05:50.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:50.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:50.144581+0000) 2022-09-08T07:05:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:51.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:51.206561+0000) 2022-09-08T07:05:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:51.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:51.144816+0000) 2022-09-08T07:05:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:51 smithi055 ceph-mon[97033]: pgmap v18398: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:51.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:51 smithi059 ceph-mon[97450]: pgmap v18398: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:52.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:05:51.979+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:05:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:52.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:52.206751+0000) 2022-09-08T07:05:52.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:52.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:52.144998+0000) 2022-09-08T07:05:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:53.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:53.206888+0000) 2022-09-08T07:05:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:53.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:53.145228+0000) 2022-09-08T07:05:53.739 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:53 smithi055 ceph-mon[97033]: pgmap v18399: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:05:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:53 smithi059 ceph-mon[97450]: pgmap v18399: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:05:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:54.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:54.207129+0000) 2022-09-08T07:05:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:54.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:54.145401+0000) 2022-09-08T07:05:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:55.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:55.207381+0000) 2022-09-08T07:05:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:55.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:55.145563+0000) 2022-09-08T07:05:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:55 smithi055 ceph-mon[97033]: pgmap v18400: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:55.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:55 smithi059 ceph-mon[97450]: pgmap v18400: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:56.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:56.207597+0000) 2022-09-08T07:05:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:56.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:56.145740+0000) 2022-09-08T07:05:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:57.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:57.207917+0000) 2022-09-08T07:05:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:57.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:57.145978+0000) 2022-09-08T07:05:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:57 smithi055 ceph-mon[97033]: pgmap v18401: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:05:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:57 smithi059 ceph-mon[97450]: pgmap v18401: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:05:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:58.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:58.208111+0000) 2022-09-08T07:05:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:58.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:58.146192+0000) 2022-09-08T07:05:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:05:59.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:59.146326+0000) 2022-09-08T07:05:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:05:59.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:05:59.208244+0000) 2022-09-08T07:05:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:05:59 smithi055 ceph-mon[97033]: pgmap v18402: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:05:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:05:59 smithi059 ceph-mon[97450]: pgmap v18402: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:00.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:00.146473+0000) 2022-09-08T07:06:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:00.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:00.208491+0000) 2022-09-08T07:06:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:01.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:01.208664+0000) 2022-09-08T07:06:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:01.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:01.146605+0000) 2022-09-08T07:06:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:01 smithi055 ceph-mon[97033]: pgmap v18403: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:01 smithi059 ceph-mon[97450]: pgmap v18403: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:02.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:02.146783+0000) 2022-09-08T07:06:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:02.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:02.208841+0000) 2022-09-08T07:06:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:03.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:03.209000+0000) 2022-09-08T07:06:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:03.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:03.146983+0000) 2022-09-08T07:06:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:03 smithi055 ceph-mon[97033]: pgmap v18404: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:06:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:03 smithi059 ceph-mon[97450]: pgmap v18404: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:06:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:04.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:04.209222+0000) 2022-09-08T07:06:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:04.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:04.147212+0000) 2022-09-08T07:06:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:05.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:05.209483+0000) 2022-09-08T07:06:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:05.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:05.147384+0000) 2022-09-08T07:06:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:05 smithi055 ceph-mon[97033]: pgmap v18405: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:05 smithi059 ceph-mon[97450]: pgmap v18405: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:06.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:06.209704+0000) 2022-09-08T07:06:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:06.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:06.147533+0000) 2022-09-08T07:06:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:06:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:06:07.149 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:06:06.989+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:06:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:07.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:07.209921+0000) 2022-09-08T07:06:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:07.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:07.147696+0000) 2022-09-08T07:06:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:07 smithi055 ceph-mon[97033]: pgmap v18406: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:06:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:06:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:06:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:07 smithi059 ceph-mon[97450]: pgmap v18406: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:06:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:06:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:06:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:08.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:08.210140+0000) 2022-09-08T07:06:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:08.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:08.147887+0000) 2022-09-08T07:06:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:08 smithi055 ceph-mon[97033]: pgmap v18407: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 48 KiB/s wr, 21 op/s 2022-09-08T07:06:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:08 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T07:06:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:08 smithi059 ceph-mon[97450]: pgmap v18407: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 48 KiB/s wr, 21 op/s 2022-09-08T07:06:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:08 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T07:06:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:09.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:09.210401+0000) 2022-09-08T07:06:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:09.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:09.148044+0000) 2022-09-08T07:06:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:10.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:10.210608+0000) 2022-09-08T07:06:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:10.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:10.148186+0000) 2022-09-08T07:06:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:10 smithi055 ceph-mon[97033]: pgmap v18408: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 48 KiB/s wr, 21 op/s 2022-09-08T07:06:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:10 smithi055 ceph-mon[97033]: osdmap e1228: 12 total, 11 up, 11 in 2022-09-08T07:06:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:10 smithi059 ceph-mon[97450]: pgmap v18408: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 48 KiB/s wr, 21 op/s 2022-09-08T07:06:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:10 smithi059 ceph-mon[97450]: osdmap e1228: 12 total, 11 up, 11 in 2022-09-08T07:06:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:11.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:11.210814+0000) 2022-09-08T07:06:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:11.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:11.148333+0000) 2022-09-08T07:06:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:12.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:12.211032+0000) 2022-09-08T07:06:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:12.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:12.148574+0000) 2022-09-08T07:06:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:12 smithi055 ceph-mon[97033]: pgmap v18410: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 59 KiB/s wr, 26 op/s 2022-09-08T07:06:12.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:12 smithi055 ceph-mon[97033]: osdmap e1229: 12 total, 11 up, 11 in 2022-09-08T07:06:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:12 smithi059 ceph-mon[97450]: pgmap v18410: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 59 KiB/s wr, 26 op/s 2022-09-08T07:06:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:12 smithi059 ceph-mon[97450]: osdmap e1229: 12 total, 11 up, 11 in 2022-09-08T07:06:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:13.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:13.211235+0000) 2022-09-08T07:06:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:13.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:13.148765+0000) 2022-09-08T07:06:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:14.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:14.211432+0000) 2022-09-08T07:06:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:14.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:14.149021+0000) 2022-09-08T07:06:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:14 smithi055 ceph-mon[97033]: pgmap v18412: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.1 KiB/s rd, 145 B/s wr, 4 op/s 2022-09-08T07:06:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:14 smithi059 ceph-mon[97450]: pgmap v18412: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.1 KiB/s rd, 145 B/s wr, 4 op/s 2022-09-08T07:06:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:15.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:15.211762+0000) 2022-09-08T07:06:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:15.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:15.149228+0000) 2022-09-08T07:06:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:16.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:16.212009+0000) 2022-09-08T07:06:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:16.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:16.149401+0000) 2022-09-08T07:06:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:16 smithi055 ceph-mon[97033]: pgmap v18413: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:06:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:16 smithi059 ceph-mon[97450]: pgmap v18413: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:06:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:17.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:17.212246+0000) 2022-09-08T07:06:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:17.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:17.149559+0000) 2022-09-08T07:06:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:18.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:18.149720+0000) 2022-09-08T07:06:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:18.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:18.212447+0000) 2022-09-08T07:06:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:18 smithi055 ceph-mon[97033]: pgmap v18414: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:06:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:18 smithi059 ceph-mon[97450]: pgmap v18414: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:06:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:19.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:19.212750+0000) 2022-09-08T07:06:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:19.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:19.149924+0000) 2022-09-08T07:06:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:20.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:20.212931+0000) 2022-09-08T07:06:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:20.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:20.150061+0000) 2022-09-08T07:06:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:20 smithi055 ceph-mon[97033]: pgmap v18415: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 218 B/s wr, 6 op/s 2022-09-08T07:06:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:20 smithi059 ceph-mon[97450]: pgmap v18415: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 218 B/s wr, 6 op/s 2022-09-08T07:06:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:21.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:21.213085+0000) 2022-09-08T07:06:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:21.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:21.150283+0000) 2022-09-08T07:06:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:22.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:22.213306+0000) 2022-09-08T07:06:22.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:06:22.003+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:06:22.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:22.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:22.150514+0000) 2022-09-08T07:06:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:22 smithi055 ceph-mon[97033]: pgmap v18416: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:06:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:22 smithi059 ceph-mon[97450]: pgmap v18416: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:06:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:23.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:23.150751+0000) 2022-09-08T07:06:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:23.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:23.213460+0000) 2022-09-08T07:06:24.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:24.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:24.150912+0000) 2022-09-08T07:06:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:24.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:24.213700+0000) 2022-09-08T07:06:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:24 smithi055 ceph-mon[97033]: pgmap v18417: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T07:06:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:24 smithi059 ceph-mon[97450]: pgmap v18417: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 98 B/s wr, 5 op/s 2022-09-08T07:06:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:25.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:25.213874+0000) 2022-09-08T07:06:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:25.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:25.151057+0000) 2022-09-08T07:06:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:26.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:26.214114+0000) 2022-09-08T07:06:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:26.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:26.151264+0000) 2022-09-08T07:06:26.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:26 smithi055 ceph-mon[97033]: pgmap v18418: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T07:06:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:26 smithi059 ceph-mon[97450]: pgmap v18418: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2022-09-08T07:06:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:27.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:27.214362+0000) 2022-09-08T07:06:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:27.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:27.151402+0000) 2022-09-08T07:06:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:28.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:28.214521+0000) 2022-09-08T07:06:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:28.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:28.151623+0000) 2022-09-08T07:06:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:28 smithi055 ceph-mon[97033]: pgmap v18419: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:06:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:28 smithi059 ceph-mon[97450]: pgmap v18419: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:06:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:29.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:29.214743+0000) 2022-09-08T07:06:29.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:29.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:29.151829+0000) 2022-09-08T07:06:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:30.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:30.151967+0000) 2022-09-08T07:06:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:30.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:30.214985+0000) 2022-09-08T07:06:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:30 smithi055 ceph-mon[97033]: pgmap v18420: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:30 smithi059 ceph-mon[97450]: pgmap v18420: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:31.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:31.215140+0000) 2022-09-08T07:06:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:31.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:31.152132+0000) 2022-09-08T07:06:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:32.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:32.215331+0000) 2022-09-08T07:06:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:32.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:32.152305+0000) 2022-09-08T07:06:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:32 smithi055 ceph-mon[97033]: pgmap v18421: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:06:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:32 smithi059 ceph-mon[97450]: pgmap v18421: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:06:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:33.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:33.215524+0000) 2022-09-08T07:06:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:33.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:33.152470+0000) 2022-09-08T07:06:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:34.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:34.152611+0000) 2022-09-08T07:06:34.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:34.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:34.215725+0000) 2022-09-08T07:06:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:34 smithi055 ceph-mon[97033]: pgmap v18422: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:34 smithi059 ceph-mon[97450]: pgmap v18422: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:35.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:35.215929+0000) 2022-09-08T07:06:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:35.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:35.152816+0000) 2022-09-08T07:06:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:36.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:36.153019+0000) 2022-09-08T07:06:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:36.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:36.216150+0000) 2022-09-08T07:06:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:36 smithi055 ceph-mon[97033]: pgmap v18423: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:36 smithi059 ceph-mon[97450]: pgmap v18423: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:37.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:06:37.018+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:06:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:37.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:37.216378+0000) 2022-09-08T07:06:37.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:37.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:37.153215+0000) 2022-09-08T07:06:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:38.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:38.216655+0000) 2022-09-08T07:06:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:38.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:38.153410+0000) 2022-09-08T07:06:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:38 smithi055 ceph-mon[97033]: pgmap v18424: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:06:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:38 smithi059 ceph-mon[97450]: pgmap v18424: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:06:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:39.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:39.216885+0000) 2022-09-08T07:06:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:39.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:39.153584+0000) 2022-09-08T07:06:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:40.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:40.217146+0000) 2022-09-08T07:06:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:40.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:40.153855+0000) 2022-09-08T07:06:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:40 smithi055 ceph-mon[97033]: pgmap v18425: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:40 smithi059 ceph-mon[97450]: pgmap v18425: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:41.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:41.217348+0000) 2022-09-08T07:06:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:41.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:41.154134+0000) 2022-09-08T07:06:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:42.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:42.217593+0000) 2022-09-08T07:06:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:42.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:42.154313+0000) 2022-09-08T07:06:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:42 smithi055 ceph-mon[97033]: pgmap v18426: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:06:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:42 smithi059 ceph-mon[97450]: pgmap v18426: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:06:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:43.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:43.217758+0000) 2022-09-08T07:06:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:43.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:43.154501+0000) 2022-09-08T07:06:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:44.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:44.217904+0000) 2022-09-08T07:06:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:44.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:44.154722+0000) 2022-09-08T07:06:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:44 smithi055 ceph-mon[97033]: pgmap v18427: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:44 smithi059 ceph-mon[97450]: pgmap v18427: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:45.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:45.218193+0000) 2022-09-08T07:06:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:45.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:45.154911+0000) 2022-09-08T07:06:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:46.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:46.218449+0000) 2022-09-08T07:06:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:46.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:46.155045+0000) 2022-09-08T07:06:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:46 smithi055 ceph-mon[97033]: pgmap v18428: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:46 smithi059 ceph-mon[97450]: pgmap v18428: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:47.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:47.218724+0000) 2022-09-08T07:06:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:47.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:47.155187+0000) 2022-09-08T07:06:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:48.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:48.218946+0000) 2022-09-08T07:06:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:48.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:48.155370+0000) 2022-09-08T07:06:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:48 smithi055 ceph-mon[97033]: pgmap v18429: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:06:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:48 smithi059 ceph-mon[97450]: pgmap v18429: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:06:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:49.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:49.219209+0000) 2022-09-08T07:06:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:49.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:49.155503+0000) 2022-09-08T07:06:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:50.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:50.219430+0000) 2022-09-08T07:06:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:50.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:50.155750+0000) 2022-09-08T07:06:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:50 smithi055 ceph-mon[97033]: pgmap v18430: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:50.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:50 smithi059 ceph-mon[97450]: pgmap v18430: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:51.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:51.219658+0000) 2022-09-08T07:06:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:51.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:51.155953+0000) 2022-09-08T07:06:52.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:06:52.026+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:06:52.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:52.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:52.219861+0000) 2022-09-08T07:06:52.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:52.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:52.156154+0000) 2022-09-08T07:06:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:52 smithi055 ceph-mon[97033]: pgmap v18431: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:06:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:52 smithi059 ceph-mon[97450]: pgmap v18431: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:06:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:53.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:53.220121+0000) 2022-09-08T07:06:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:53.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:53.156313+0000) 2022-09-08T07:06:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:54.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:54.156467+0000) 2022-09-08T07:06:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:54.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:54.220405+0000) 2022-09-08T07:06:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:54 smithi059 ceph-mon[97450]: pgmap v18432: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:54 smithi055 ceph-mon[97033]: pgmap v18432: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:55.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:55.220628+0000) 2022-09-08T07:06:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:55.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:55.156612+0000) 2022-09-08T07:06:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:56.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:56.220796+0000) 2022-09-08T07:06:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:56.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:56.156773+0000) 2022-09-08T07:06:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:56 smithi059 ceph-mon[97450]: pgmap v18433: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:56 smithi055 ceph-mon[97033]: pgmap v18433: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:06:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:57.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:57.156969+0000) 2022-09-08T07:06:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:57.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:57.221055+0000) 2022-09-08T07:06:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:58.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:58.221342+0000) 2022-09-08T07:06:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:58.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:58.157147+0000) 2022-09-08T07:06:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:06:58 smithi059 ceph-mon[97450]: pgmap v18434: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:06:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:06:58 smithi055 ceph-mon[97033]: pgmap v18434: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:06:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:06:59.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:59.221595+0000) 2022-09-08T07:06:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:06:59.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:06:59.157308+0000) 2022-09-08T07:07:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:00.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:00.221840+0000) 2022-09-08T07:07:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:00.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:00.157465+0000) 2022-09-08T07:07:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:00 smithi059 ceph-mon[97450]: pgmap v18435: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:07:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:00 smithi055 ceph-mon[97033]: pgmap v18435: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:07:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:01.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:01.222020+0000) 2022-09-08T07:07:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:01.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:01.157642+0000) 2022-09-08T07:07:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:02.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:02.222231+0000) 2022-09-08T07:07:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:02.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:02.157773+0000) 2022-09-08T07:07:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:02 smithi059 ceph-mon[97450]: pgmap v18436: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:07:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:02 smithi055 ceph-mon[97033]: pgmap v18436: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:07:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:03.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:03.222474+0000) 2022-09-08T07:07:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:03.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:03.157942+0000) 2022-09-08T07:07:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:04.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:04.222688+0000) 2022-09-08T07:07:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:04.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:04.158159+0000) 2022-09-08T07:07:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:04 smithi059 ceph-mon[97450]: pgmap v18437: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:07:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:04 smithi055 ceph-mon[97033]: pgmap v18437: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:07:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:05.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:05.222858+0000) 2022-09-08T07:07:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:05.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:05.158387+0000) 2022-09-08T07:07:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:06.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:06.223046+0000) 2022-09-08T07:07:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:06.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:06.158662+0000) 2022-09-08T07:07:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:06 smithi059 ceph-mon[97450]: pgmap v18438: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:07:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:07:06.940 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:06 smithi055 ceph-mon[97033]: pgmap v18438: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:07:06.940 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:07:07.161 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:07:07.027+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:07:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:07.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:07.223216+0000) 2022-09-08T07:07:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:07.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:07.158889+0000) 2022-09-08T07:07:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:07:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:07:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:07:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:07:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:07:07.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:07:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:07:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:07:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:07:07.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:07:07.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:07:07.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:07:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:08.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:08.223401+0000) 2022-09-08T07:07:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:08.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:08.159060+0000) 2022-09-08T07:07:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:08 smithi059 ceph-mon[97450]: pgmap v18439: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:07:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:08 smithi055 ceph-mon[97033]: pgmap v18439: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:07:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:09.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:09.223663+0000) 2022-09-08T07:07:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:09.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:09.159277+0000) 2022-09-08T07:07:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:10.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:10.223802+0000) 2022-09-08T07:07:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:10.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:10.159560+0000) 2022-09-08T07:07:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:10 smithi059 ceph-mon[97450]: pgmap v18440: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:07:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:10 smithi059 ceph-mon[97450]: osdmap e1230: 12 total, 11 up, 11 in 2022-09-08T07:07:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:10 smithi055 ceph-mon[97033]: pgmap v18440: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:07:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:10 smithi055 ceph-mon[97033]: osdmap e1230: 12 total, 11 up, 11 in 2022-09-08T07:07:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:11.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:11.159759+0000) 2022-09-08T07:07:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:11.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:11.223993+0000) 2022-09-08T07:07:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:12.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:12.224154+0000) 2022-09-08T07:07:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:12.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:12.159967+0000) 2022-09-08T07:07:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:12 smithi059 ceph-mon[97450]: pgmap v18442: 129 pgs: 1 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 112 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:07:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:12 smithi055 ceph-mon[97033]: pgmap v18442: 129 pgs: 1 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 112 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:07:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:13.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:13.224303+0000) 2022-09-08T07:07:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:13.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:13.160209+0000) 2022-09-08T07:07:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:14.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:14.224513+0000) 2022-09-08T07:07:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:14.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:14.160474+0000) 2022-09-08T07:07:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:14 smithi059 ceph-mon[97450]: pgmap v18443: 129 pgs: 1 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 112 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:07:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:14 smithi055 ceph-mon[97033]: pgmap v18443: 129 pgs: 1 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 112 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:07:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:15.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:15.224737+0000) 2022-09-08T07:07:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:15.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:15.160596+0000) 2022-09-08T07:07:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:16.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:16.160750+0000) 2022-09-08T07:07:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:16.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:16.224934+0000) 2022-09-08T07:07:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:16 smithi059 ceph-mon[97450]: pgmap v18444: 129 pgs: 1 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 112 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:07:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:16 smithi055 ceph-mon[97033]: pgmap v18444: 129 pgs: 1 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 112 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:07:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:17.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:17.225168+0000) 2022-09-08T07:07:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:17.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:17.160873+0000) 2022-09-08T07:07:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:18.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:18.225408+0000) 2022-09-08T07:07:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:18.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:18.161100+0000) 2022-09-08T07:07:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:18 smithi055 ceph-mon[97033]: pgmap v18445: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:07:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:18 smithi059 ceph-mon[97450]: pgmap v18445: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:07:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:19.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:19.225647+0000) 2022-09-08T07:07:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:19.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:19.161313+0000) 2022-09-08T07:07:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:20.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:20.225849+0000) 2022-09-08T07:07:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:20.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:20.161506+0000) 2022-09-08T07:07:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:20 smithi055 ceph-mon[97033]: pgmap v18446: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:07:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:20 smithi059 ceph-mon[97450]: pgmap v18446: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:07:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:21.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:21.226033+0000) 2022-09-08T07:07:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:21.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:21.161737+0000) 2022-09-08T07:07:22.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:07:22.034+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:07:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:22.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:22.226231+0000) 2022-09-08T07:07:22.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:22.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:22.161921+0000) 2022-09-08T07:07:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:22 smithi055 ceph-mon[97033]: pgmap v18447: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 181 B/s wr, 8 op/s 2022-09-08T07:07:22.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:22 smithi055 ceph-mon[97033]: osdmap e1231: 12 total, 11 up, 11 in 2022-09-08T07:07:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:22 smithi059 ceph-mon[97450]: pgmap v18447: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 181 B/s wr, 8 op/s 2022-09-08T07:07:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:22 smithi059 ceph-mon[97450]: osdmap e1231: 12 total, 11 up, 11 in 2022-09-08T07:07:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:23.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:23.226398+0000) 2022-09-08T07:07:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:23.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:23.162124+0000) 2022-09-08T07:07:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:24.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:24.226630+0000) 2022-09-08T07:07:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:24.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:24.162368+0000) 2022-09-08T07:07:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:24 smithi055 ceph-mon[97033]: pgmap v18449: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:07:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:24 smithi059 ceph-mon[97450]: pgmap v18449: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:07:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:25.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:25.226837+0000) 2022-09-08T07:07:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:25.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:25.162623+0000) 2022-09-08T07:07:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:26.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:26.227053+0000) 2022-09-08T07:07:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:26.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:26.162838+0000) 2022-09-08T07:07:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:26 smithi055 ceph-mon[97033]: pgmap v18450: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:07:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:26 smithi059 ceph-mon[97450]: pgmap v18450: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:07:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:27.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:27.227324+0000) 2022-09-08T07:07:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:27.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:27.162986+0000) 2022-09-08T07:07:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:28.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:28.227553+0000) 2022-09-08T07:07:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:28.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:28.163141+0000) 2022-09-08T07:07:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:28 smithi055 ceph-mon[97033]: pgmap v18451: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:07:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:28 smithi059 ceph-mon[97450]: pgmap v18451: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:07:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:29.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:29.227782+0000) 2022-09-08T07:07:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:29.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:29.163355+0000) 2022-09-08T07:07:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:30.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:30.228022+0000) 2022-09-08T07:07:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:30.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:30.163561+0000) 2022-09-08T07:07:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:30 smithi055 ceph-mon[97033]: pgmap v18452: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:07:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:30 smithi059 ceph-mon[97450]: pgmap v18452: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:07:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:31.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:31.228245+0000) 2022-09-08T07:07:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:31.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:31.163724+0000) 2022-09-08T07:07:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:32.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:32.228494+0000) 2022-09-08T07:07:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:32.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:32.163882+0000) 2022-09-08T07:07:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:32 smithi055 ceph-mon[97033]: pgmap v18453: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:07:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:32 smithi059 ceph-mon[97450]: pgmap v18453: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:07:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:33.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:33.228742+0000) 2022-09-08T07:07:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:33.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:33.164025+0000) 2022-09-08T07:07:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:34.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:34.228944+0000) 2022-09-08T07:07:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:34.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:34.164151+0000) 2022-09-08T07:07:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:34 smithi055 ceph-mon[97033]: pgmap v18454: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:07:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:34 smithi059 ceph-mon[97450]: pgmap v18454: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:07:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:35.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:35.229224+0000) 2022-09-08T07:07:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:35.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:35.164356+0000) 2022-09-08T07:07:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:36.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:36.164597+0000) 2022-09-08T07:07:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:36.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:36.229461+0000) 2022-09-08T07:07:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:36 smithi055 ceph-mon[97033]: pgmap v18455: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:07:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:36 smithi059 ceph-mon[97450]: pgmap v18455: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:07:37.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:07:37.049+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:07:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:37.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:37.164877+0000) 2022-09-08T07:07:37.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:37.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:37.229677+0000) 2022-09-08T07:07:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:38 smithi055 ceph-mon[97033]: pgmap v18456: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:07:38.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:38 smithi055 ceph-mon[97033]: pgmap v18457: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:07:38.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:38 smithi055 ceph-mon[97033]: pgmap v18458: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-09-08T07:07:38.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:07:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:38 smithi059 ceph-mon[97450]: pgmap v18456: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:07:38.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:38 smithi059 ceph-mon[97450]: pgmap v18457: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:07:38.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:38 smithi059 ceph-mon[97450]: pgmap v18458: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-09-08T07:07:38.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:07:38.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:38.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:38.229886+0000) 2022-09-08T07:07:38.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:38.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:38.165104+0000) 2022-09-08T07:07:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:39.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:39.230058+0000) 2022-09-08T07:07:39.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:39.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:39.165287+0000) 2022-09-08T07:07:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:40 smithi055 ceph-mon[97033]: pgmap v18459: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T07:07:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:40 smithi059 ceph-mon[97450]: pgmap v18459: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T07:07:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:40.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:40.230241+0000) 2022-09-08T07:07:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:40.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:40.165499+0000) 2022-09-08T07:07:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:41.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:41.230459+0000) 2022-09-08T07:07:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:41.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:41.165716+0000) 2022-09-08T07:07:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:42 smithi055 ceph-mon[97033]: pgmap v18460: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-09-08T07:07:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:42 smithi059 ceph-mon[97450]: pgmap v18460: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-09-08T07:07:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:42.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:42.230671+0000) 2022-09-08T07:07:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:42.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:42.165875+0000) 2022-09-08T07:07:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:43.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:43.230893+0000) 2022-09-08T07:07:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:43.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:43.166001+0000) 2022-09-08T07:07:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:44.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:44.231054+0000) 2022-09-08T07:07:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:44.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:44.166141+0000) 2022-09-08T07:07:44.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:44 smithi055 ceph-mon[97033]: pgmap v18461: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-09-08T07:07:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:44 smithi059 ceph-mon[97450]: pgmap v18461: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-09-08T07:07:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:45.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:45.166273+0000) 2022-09-08T07:07:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:45.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:45.231248+0000) 2022-09-08T07:07:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:46.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:46.231434+0000) 2022-09-08T07:07:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:46.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:46.166452+0000) 2022-09-08T07:07:46.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:46 smithi055 ceph-mon[97033]: pgmap v18462: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:07:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:46 smithi059 ceph-mon[97450]: pgmap v18462: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:07:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:47.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:47.231694+0000) 2022-09-08T07:07:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:47.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:47.166696+0000) 2022-09-08T07:07:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:48.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:48.231953+0000) 2022-09-08T07:07:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:48.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:48.166864+0000) 2022-09-08T07:07:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:48 smithi055 ceph-mon[97033]: pgmap v18463: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:07:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:48 smithi059 ceph-mon[97450]: pgmap v18463: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:07:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:49.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:49.232134+0000) 2022-09-08T07:07:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:49.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:49.167038+0000) 2022-09-08T07:07:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:50.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:50.232353+0000) 2022-09-08T07:07:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:50.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:50.167278+0000) 2022-09-08T07:07:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:50 smithi055 ceph-mon[97033]: pgmap v18464: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:07:50.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:50 smithi059 ceph-mon[97450]: pgmap v18464: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:07:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:51.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:51.232545+0000) 2022-09-08T07:07:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:51.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:51.167452+0000) 2022-09-08T07:07:52.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:07:52.056+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:07:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:52.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:52.232782+0000) 2022-09-08T07:07:52.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:52.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:52.167699+0000) 2022-09-08T07:07:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:52 smithi055 ceph-mon[97033]: pgmap v18465: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:07:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:52 smithi059 ceph-mon[97450]: pgmap v18465: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:07:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:53.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:53.233016+0000) 2022-09-08T07:07:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:53.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:53.167899+0000) 2022-09-08T07:07:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:54.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:54.233195+0000) 2022-09-08T07:07:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:54.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:54.168088+0000) 2022-09-08T07:07:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:54 smithi055 ceph-mon[97033]: pgmap v18466: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:07:54.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:54 smithi059 ceph-mon[97450]: pgmap v18466: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:07:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:55.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:55.233368+0000) 2022-09-08T07:07:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:55.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:55.168299+0000) 2022-09-08T07:07:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:56.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:56.233589+0000) 2022-09-08T07:07:56.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:56.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:56.168522+0000) 2022-09-08T07:07:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:56 smithi055 ceph-mon[97033]: pgmap v18467: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:07:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:56 smithi059 ceph-mon[97450]: pgmap v18467: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:07:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:57.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:57.233805+0000) 2022-09-08T07:07:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:57.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:57.168745+0000) 2022-09-08T07:07:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:58.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:58.233929+0000) 2022-09-08T07:07:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:58.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:58.168901+0000) 2022-09-08T07:07:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:07:58 smithi055 ceph-mon[97033]: pgmap v18468: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:07:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:07:58 smithi059 ceph-mon[97450]: pgmap v18468: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:07:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:07:59.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:59.234157+0000) 2022-09-08T07:07:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:07:59.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:07:59.169024+0000) 2022-09-08T07:08:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:00.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:00.234307+0000) 2022-09-08T07:08:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:00.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:00.169254+0000) 2022-09-08T07:08:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:00 smithi055 ceph-mon[97033]: pgmap v18469: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:00 smithi059 ceph-mon[97450]: pgmap v18469: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:01.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:01.234513+0000) 2022-09-08T07:08:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:01.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:01.169445+0000) 2022-09-08T07:08:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:02.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:02.234734+0000) 2022-09-08T07:08:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:02.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:02.169688+0000) 2022-09-08T07:08:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:02 smithi055 ceph-mon[97033]: pgmap v18470: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:08:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:02 smithi059 ceph-mon[97450]: pgmap v18470: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:08:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:03.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:03.234981+0000) 2022-09-08T07:08:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:03.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:03.169915+0000) 2022-09-08T07:08:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:04.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:04.235144+0000) 2022-09-08T07:08:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:04.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:04.170065+0000) 2022-09-08T07:08:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:04 smithi055 ceph-mon[97033]: pgmap v18471: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:04 smithi059 ceph-mon[97450]: pgmap v18471: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:05.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:05.235320+0000) 2022-09-08T07:08:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:05.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:05.170193+0000) 2022-09-08T07:08:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:06.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:06.235455+0000) 2022-09-08T07:08:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:06.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:06.170366+0000) 2022-09-08T07:08:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:06 smithi055 ceph-mon[97033]: pgmap v18472: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:08:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:08:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:06 smithi059 ceph-mon[97450]: pgmap v18472: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:08:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:08:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:07.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:07.235717+0000) 2022-09-08T07:08:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:07.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:07.170519+0000) 2022-09-08T07:08:07.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:08:07.057+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:08:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:08:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:08:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:08.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:08.235875+0000) 2022-09-08T07:08:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:08.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:08.170723+0000) 2022-09-08T07:08:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:08 smithi055 ceph-mon[97033]: pgmap v18473: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:08:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:08 smithi059 ceph-mon[97450]: pgmap v18473: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:08:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:09.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:09.235992+0000) 2022-09-08T07:08:09.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:09.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:09.170886+0000) 2022-09-08T07:08:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:10.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:10.171017+0000) 2022-09-08T07:08:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:10.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:10.236184+0000) 2022-09-08T07:08:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:10 smithi055 ceph-mon[97033]: pgmap v18474: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:08:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:10 smithi055 ceph-mon[97033]: osdmap e1232: 12 total, 11 up, 11 in 2022-09-08T07:08:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:10 smithi059 ceph-mon[97450]: pgmap v18474: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:08:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:10 smithi059 ceph-mon[97450]: osdmap e1232: 12 total, 11 up, 11 in 2022-09-08T07:08:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:11.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:11.171194+0000) 2022-09-08T07:08:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:11.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:11.236372+0000) 2022-09-08T07:08:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:12.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:12.236578+0000) 2022-09-08T07:08:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:12.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:12.171345+0000) 2022-09-08T07:08:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:12 smithi055 ceph-mon[97033]: pgmap v18476: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:08:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:12 smithi059 ceph-mon[97450]: pgmap v18476: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:08:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:13.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:13.236802+0000) 2022-09-08T07:08:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:13.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:13.171503+0000) 2022-09-08T07:08:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:14.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:14.237005+0000) 2022-09-08T07:08:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:14.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:14.171756+0000) 2022-09-08T07:08:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:14 smithi055 ceph-mon[97033]: pgmap v18477: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:08:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:14 smithi059 ceph-mon[97450]: pgmap v18477: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:08:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:15.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:15.237241+0000) 2022-09-08T07:08:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:15.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:15.171937+0000) 2022-09-08T07:08:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:16.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:16.172166+0000) 2022-09-08T07:08:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:16.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:16.237417+0000) 2022-09-08T07:08:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:16 smithi055 ceph-mon[97033]: pgmap v18478: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:08:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:16 smithi059 ceph-mon[97450]: pgmap v18478: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:08:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:17.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:17.237675+0000) 2022-09-08T07:08:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:17.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:17.172377+0000) 2022-09-08T07:08:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:18.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:18.237854+0000) 2022-09-08T07:08:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:18.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:18.172597+0000) 2022-09-08T07:08:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:18 smithi055 ceph-mon[97033]: pgmap v18479: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:08:18.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:18 smithi055 ceph-mon[97033]: osdmap e1233: 12 total, 11 up, 11 in 2022-09-08T07:08:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:18 smithi059 ceph-mon[97450]: pgmap v18479: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:08:18.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:18 smithi059 ceph-mon[97450]: osdmap e1233: 12 total, 11 up, 11 in 2022-09-08T07:08:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:19.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:19.238084+0000) 2022-09-08T07:08:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:19.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:19.172779+0000) 2022-09-08T07:08:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:20.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:20.238336+0000) 2022-09-08T07:08:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:20.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:20.172930+0000) 2022-09-08T07:08:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:20 smithi055 ceph-mon[97033]: pgmap v18481: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T07:08:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:20 smithi059 ceph-mon[97450]: pgmap v18481: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T07:08:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:21.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:21.173173+0000) 2022-09-08T07:08:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:21.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:21.238485+0000) 2022-09-08T07:08:22.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:08:22.066+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:08:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:22.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:22.173342+0000) 2022-09-08T07:08:22.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:22.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:22.238740+0000) 2022-09-08T07:08:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:22 smithi055 ceph-mon[97033]: pgmap v18482: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:08:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:22 smithi059 ceph-mon[97450]: pgmap v18482: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:08:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:23.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:23.238970+0000) 2022-09-08T07:08:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:23.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:23.173511+0000) 2022-09-08T07:08:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:24.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:24.239181+0000) 2022-09-08T07:08:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:24.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:24.173689+0000) 2022-09-08T07:08:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:24 smithi055 ceph-mon[97033]: pgmap v18483: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:08:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:24 smithi059 ceph-mon[97450]: pgmap v18483: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:08:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:25.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:25.239415+0000) 2022-09-08T07:08:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:25.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:25.173905+0000) 2022-09-08T07:08:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:26.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:26.239629+0000) 2022-09-08T07:08:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:26.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:26.174095+0000) 2022-09-08T07:08:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:26 smithi055 ceph-mon[97033]: pgmap v18484: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:08:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:26 smithi059 ceph-mon[97450]: pgmap v18484: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:08:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:27.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:27.239866+0000) 2022-09-08T07:08:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:27.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:27.174305+0000) 2022-09-08T07:08:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:28.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:28.240045+0000) 2022-09-08T07:08:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:28.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:28.174568+0000) 2022-09-08T07:08:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:28 smithi055 ceph-mon[97033]: pgmap v18485: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:08:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:28 smithi059 ceph-mon[97450]: pgmap v18485: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:08:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:29.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:29.240282+0000) 2022-09-08T07:08:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:29.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:29.174732+0000) 2022-09-08T07:08:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:30.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:30.240497+0000) 2022-09-08T07:08:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:30.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:30.174953+0000) 2022-09-08T07:08:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:30 smithi055 ceph-mon[97033]: pgmap v18486: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:08:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:30 smithi059 ceph-mon[97450]: pgmap v18486: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:08:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:31.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:31.240758+0000) 2022-09-08T07:08:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:31.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:31.175104+0000) 2022-09-08T07:08:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:32.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:32.241015+0000) 2022-09-08T07:08:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:32.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:32.175249+0000) 2022-09-08T07:08:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:32 smithi055 ceph-mon[97033]: pgmap v18487: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:08:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:32 smithi059 ceph-mon[97450]: pgmap v18487: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:08:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:33.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:33.241272+0000) 2022-09-08T07:08:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:33.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:33.175416+0000) 2022-09-08T07:08:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:34.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:34.241427+0000) 2022-09-08T07:08:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:34.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:34.175580+0000) 2022-09-08T07:08:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:34 smithi055 ceph-mon[97033]: pgmap v18488: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:34 smithi059 ceph-mon[97450]: pgmap v18488: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:35.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:35.241661+0000) 2022-09-08T07:08:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:35.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:35.175725+0000) 2022-09-08T07:08:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:36.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:36.241908+0000) 2022-09-08T07:08:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:36.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:36.175861+0000) 2022-09-08T07:08:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:36 smithi055 ceph-mon[97033]: pgmap v18489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:08:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:36 smithi059 ceph-mon[97450]: pgmap v18489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:08:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:37.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:37.242046+0000) 2022-09-08T07:08:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:37.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:37.176011+0000) 2022-09-08T07:08:37.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:08:37.081+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:08:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:08:37.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:08:37.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:08:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:08:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:08:37.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:08:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:38.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:38.242269+0000) 2022-09-08T07:08:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:38.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:38.176269+0000) 2022-09-08T07:08:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:38 smithi055 ceph-mon[97033]: pgmap v18490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:38 smithi059 ceph-mon[97450]: pgmap v18490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:39.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:39.242514+0000) 2022-09-08T07:08:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:39.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:39.176540+0000) 2022-09-08T07:08:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:40.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:40.242726+0000) 2022-09-08T07:08:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:40.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:40.176727+0000) 2022-09-08T07:08:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:40 smithi055 ceph-mon[97033]: pgmap v18491: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:40 smithi059 ceph-mon[97450]: pgmap v18491: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:41.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:41.242930+0000) 2022-09-08T07:08:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:41.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:41.176872+0000) 2022-09-08T07:08:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:42.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:42.243138+0000) 2022-09-08T07:08:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:42.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:42.177028+0000) 2022-09-08T07:08:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:42 smithi055 ceph-mon[97033]: pgmap v18492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:08:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:42 smithi059 ceph-mon[97450]: pgmap v18492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:08:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:43.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:43.243342+0000) 2022-09-08T07:08:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:43.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:43.177151+0000) 2022-09-08T07:08:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:44.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:44.243554+0000) 2022-09-08T07:08:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:44.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:44.177317+0000) 2022-09-08T07:08:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:44 smithi055 ceph-mon[97033]: pgmap v18493: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:44 smithi059 ceph-mon[97450]: pgmap v18493: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:45.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:45.243792+0000) 2022-09-08T07:08:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:45.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:45.177465+0000) 2022-09-08T07:08:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:46.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:46.244022+0000) 2022-09-08T07:08:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:46.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:46.177621+0000) 2022-09-08T07:08:46.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:46 smithi055 ceph-mon[97033]: pgmap v18494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:08:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:46 smithi059 ceph-mon[97450]: pgmap v18494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:08:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:47.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:47.244211+0000) 2022-09-08T07:08:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:47.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:47.177815+0000) 2022-09-08T07:08:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:48.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:48.244442+0000) 2022-09-08T07:08:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:48.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:48.177990+0000) 2022-09-08T07:08:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:48 smithi055 ceph-mon[97033]: pgmap v18495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:48 smithi059 ceph-mon[97450]: pgmap v18495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:49.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:49.178153+0000) 2022-09-08T07:08:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:49.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:49.244699+0000) 2022-09-08T07:08:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:50.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:50.244908+0000) 2022-09-08T07:08:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:50.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:50.178325+0000) 2022-09-08T07:08:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:50 smithi055 ceph-mon[97033]: pgmap v18496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:50.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:50 smithi059 ceph-mon[97450]: pgmap v18496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:51.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:51.178503+0000) 2022-09-08T07:08:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:51.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:51.245127+0000) 2022-09-08T07:08:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:52.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:52.178748+0000) 2022-09-08T07:08:52.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:08:52.092+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:08:52.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:52.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:52.245355+0000) 2022-09-08T07:08:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:52 smithi055 ceph-mon[97033]: pgmap v18497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:08:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:52 smithi059 ceph-mon[97450]: pgmap v18497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:08:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:53.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:53.245600+0000) 2022-09-08T07:08:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:53.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:53.178925+0000) 2022-09-08T07:08:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:54.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:54.245796+0000) 2022-09-08T07:08:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:54.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:54.179081+0000) 2022-09-08T07:08:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:54 smithi055 ceph-mon[97033]: pgmap v18498: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:54 smithi059 ceph-mon[97450]: pgmap v18498: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:55.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:55.246004+0000) 2022-09-08T07:08:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:55.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:55.179277+0000) 2022-09-08T07:08:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:56.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:56.246210+0000) 2022-09-08T07:08:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:56.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:56.179514+0000) 2022-09-08T07:08:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:56 smithi055 ceph-mon[97033]: pgmap v18499: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:08:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:56 smithi059 ceph-mon[97450]: pgmap v18499: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:08:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:57.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:57.246412+0000) 2022-09-08T07:08:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:57.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:57.179686+0000) 2022-09-08T07:08:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:58.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:58.246657+0000) 2022-09-08T07:08:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:58.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:58.179820+0000) 2022-09-08T07:08:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:08:58 smithi055 ceph-mon[97033]: pgmap v18500: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:08:58 smithi059 ceph-mon[97450]: pgmap v18500: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:08:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:08:59.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:59.246869+0000) 2022-09-08T07:08:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:08:59.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:08:59.179954+0000) 2022-09-08T07:09:00.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:00.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:00.180109+0000) 2022-09-08T07:09:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:00.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:00.247075+0000) 2022-09-08T07:09:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:00 smithi055 ceph-mon[97033]: pgmap v18501: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:00 smithi059 ceph-mon[97450]: pgmap v18501: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:01.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:01.180329+0000) 2022-09-08T07:09:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:01.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:01.247299+0000) 2022-09-08T07:09:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:02.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:02.247520+0000) 2022-09-08T07:09:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:02.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:02.180573+0000) 2022-09-08T07:09:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:02 smithi055 ceph-mon[97033]: pgmap v18502: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:09:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:02 smithi059 ceph-mon[97450]: pgmap v18502: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:09:03.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:03.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:03.180708+0000) 2022-09-08T07:09:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:03.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:03.247771+0000) 2022-09-08T07:09:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:04.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:04.180952+0000) 2022-09-08T07:09:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:04.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:04.247941+0000) 2022-09-08T07:09:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:04 smithi055 ceph-mon[97033]: pgmap v18503: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:04 smithi059 ceph-mon[97450]: pgmap v18503: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:05.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:05.181169+0000) 2022-09-08T07:09:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:05.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:05.248166+0000) 2022-09-08T07:09:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:06.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:06.181306+0000) 2022-09-08T07:09:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:06.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:06.248345+0000) 2022-09-08T07:09:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:06 smithi055 ceph-mon[97033]: pgmap v18504: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:09:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:09:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:06 smithi059 ceph-mon[97450]: pgmap v18504: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:09:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:09:07.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:09:07.107+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:09:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:07.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:07.248583+0000) 2022-09-08T07:09:07.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:07.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:07.181461+0000) 2022-09-08T07:09:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:09:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:09:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:09:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:09:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:08.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:08.248844+0000) 2022-09-08T07:09:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:08.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:08.181600+0000) 2022-09-08T07:09:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:08 smithi059 ceph-mon[97450]: pgmap v18505: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:09:08.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:08 smithi059 ceph-mon[97450]: pgmap v18506: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T07:09:08.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:08 smithi059 ceph-mon[97450]: pgmap v18507: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T07:09:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:08 smithi055 ceph-mon[97033]: pgmap v18505: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:09:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:08 smithi055 ceph-mon[97033]: pgmap v18506: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T07:09:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:08 smithi055 ceph-mon[97033]: pgmap v18507: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T07:09:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:09.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:09.249070+0000) 2022-09-08T07:09:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:09.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:09.181743+0000) 2022-09-08T07:09:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:10.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:10.249234+0000) 2022-09-08T07:09:10.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:10.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:10.181977+0000) 2022-09-08T07:09:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:10 smithi059 ceph-mon[97450]: pgmap v18508: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T07:09:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:10 smithi059 ceph-mon[97450]: osdmap e1234: 12 total, 11 up, 11 in 2022-09-08T07:09:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:10 smithi055 ceph-mon[97033]: pgmap v18508: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T07:09:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:10 smithi055 ceph-mon[97033]: osdmap e1234: 12 total, 11 up, 11 in 2022-09-08T07:09:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:11.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:11.249419+0000) 2022-09-08T07:09:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:11.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:11.182146+0000) 2022-09-08T07:09:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:12.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:12.249662+0000) 2022-09-08T07:09:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:12.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:12.182386+0000) 2022-09-08T07:09:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:12 smithi059 ceph-mon[97450]: pgmap v18510: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T07:09:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:12 smithi055 ceph-mon[97033]: pgmap v18510: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T07:09:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:13.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:13.249892+0000) 2022-09-08T07:09:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:13.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:13.182585+0000) 2022-09-08T07:09:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:14.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:14.250072+0000) 2022-09-08T07:09:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:14.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:14.182769+0000) 2022-09-08T07:09:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:14 smithi059 ceph-mon[97450]: pgmap v18511: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T07:09:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:14 smithi055 ceph-mon[97033]: pgmap v18511: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T07:09:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:15.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:15.250303+0000) 2022-09-08T07:09:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:15.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:15.182993+0000) 2022-09-08T07:09:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:16.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:16.250582+0000) 2022-09-08T07:09:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:16.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:16.183262+0000) 2022-09-08T07:09:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:16 smithi059 ceph-mon[97450]: pgmap v18512: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:09:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:16 smithi055 ceph-mon[97033]: pgmap v18512: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:09:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:17.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:17.183444+0000) 2022-09-08T07:09:17.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:17.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:17.250801+0000) 2022-09-08T07:09:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:18.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:18.251080+0000) 2022-09-08T07:09:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:18.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:18.183716+0000) 2022-09-08T07:09:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:18 smithi055 ceph-mon[97033]: pgmap v18513: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:09:18.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:18 smithi055 ceph-mon[97033]: osdmap e1235: 12 total, 11 up, 11 in 2022-09-08T07:09:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:18 smithi059 ceph-mon[97450]: pgmap v18513: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:09:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:18 smithi059 ceph-mon[97450]: osdmap e1235: 12 total, 11 up, 11 in 2022-09-08T07:09:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:19.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:19.251327+0000) 2022-09-08T07:09:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:19.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:19.183876+0000) 2022-09-08T07:09:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:20.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:20.251570+0000) 2022-09-08T07:09:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:20.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:20.184020+0000) 2022-09-08T07:09:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:20 smithi055 ceph-mon[97033]: pgmap v18515: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2022-09-08T07:09:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:20 smithi059 ceph-mon[97450]: pgmap v18515: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2022-09-08T07:09:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:21.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:21.251734+0000) 2022-09-08T07:09:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:21.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:21.184180+0000) 2022-09-08T07:09:22.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:09:22.111+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:09:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:22.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:22.251958+0000) 2022-09-08T07:09:22.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:22.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:22.184306+0000) 2022-09-08T07:09:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:22 smithi055 ceph-mon[97033]: pgmap v18516: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:09:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:22 smithi059 ceph-mon[97450]: pgmap v18516: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:09:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:23.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:23.184534+0000) 2022-09-08T07:09:23.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:23.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:23.252222+0000) 2022-09-08T07:09:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:24.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:24.252489+0000) 2022-09-08T07:09:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:24.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:24.184766+0000) 2022-09-08T07:09:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:24 smithi055 ceph-mon[97033]: pgmap v18517: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:09:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:24 smithi059 ceph-mon[97450]: pgmap v18517: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:09:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:25.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:25.252764+0000) 2022-09-08T07:09:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:25.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:25.185005+0000) 2022-09-08T07:09:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:26.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:26.252930+0000) 2022-09-08T07:09:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:26.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:26.185243+0000) 2022-09-08T07:09:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:26 smithi055 ceph-mon[97033]: pgmap v18518: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:09:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:26 smithi059 ceph-mon[97450]: pgmap v18518: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:09:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:27.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:27.253159+0000) 2022-09-08T07:09:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:27.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:27.185424+0000) 2022-09-08T07:09:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:28.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:28.253383+0000) 2022-09-08T07:09:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:28.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:28.185700+0000) 2022-09-08T07:09:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:28 smithi055 ceph-mon[97033]: pgmap v18519: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:09:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:28 smithi059 ceph-mon[97450]: pgmap v18519: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:09:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:29.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:29.253686+0000) 2022-09-08T07:09:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:29.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:29.185902+0000) 2022-09-08T07:09:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:30.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:30.253948+0000) 2022-09-08T07:09:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:30.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:30.186070+0000) 2022-09-08T07:09:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:30 smithi055 ceph-mon[97033]: pgmap v18520: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T07:09:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:30 smithi059 ceph-mon[97450]: pgmap v18520: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T07:09:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:31.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:31.186250+0000) 2022-09-08T07:09:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:31.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:31.254227+0000) 2022-09-08T07:09:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:32.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:32.254445+0000) 2022-09-08T07:09:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:32.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:32.186459+0000) 2022-09-08T07:09:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:32 smithi055 ceph-mon[97033]: pgmap v18521: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:09:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:32 smithi059 ceph-mon[97450]: pgmap v18521: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:09:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:33.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:33.186729+0000) 2022-09-08T07:09:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:33.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:33.254656+0000) 2022-09-08T07:09:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:34.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:34.254837+0000) 2022-09-08T07:09:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:34.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:34.186954+0000) 2022-09-08T07:09:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:34 smithi055 ceph-mon[97033]: pgmap v18522: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:34 smithi059 ceph-mon[97450]: pgmap v18522: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:35.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:35.255017+0000) 2022-09-08T07:09:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:35.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:35.187136+0000) 2022-09-08T07:09:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:36.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:36.187302+0000) 2022-09-08T07:09:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:36.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:36.255229+0000) 2022-09-08T07:09:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:36 smithi055 ceph-mon[97033]: pgmap v18523: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:09:36.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:36 smithi059 ceph-mon[97450]: pgmap v18523: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:09:37.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:09:37.119+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:09:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:37.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:37.255467+0000) 2022-09-08T07:09:37.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:37.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:37.187430+0000) 2022-09-08T07:09:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:38.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:38.255741+0000) 2022-09-08T07:09:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:38.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:38.187598+0000) 2022-09-08T07:09:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:38 smithi055 ceph-mon[97033]: pgmap v18524: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:38 smithi059 ceph-mon[97450]: pgmap v18524: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:39.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:39.255963+0000) 2022-09-08T07:09:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:39.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:39.187801+0000) 2022-09-08T07:09:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:40.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:40.256189+0000) 2022-09-08T07:09:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:40.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:40.187972+0000) 2022-09-08T07:09:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:40 smithi055 ceph-mon[97033]: pgmap v18525: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:40 smithi059 ceph-mon[97450]: pgmap v18525: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:41.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:41.256371+0000) 2022-09-08T07:09:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:41.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:41.188163+0000) 2022-09-08T07:09:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:42.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:42.256600+0000) 2022-09-08T07:09:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:42.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:42.188328+0000) 2022-09-08T07:09:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:42 smithi055 ceph-mon[97033]: pgmap v18526: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:09:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:42 smithi059 ceph-mon[97450]: pgmap v18526: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:09:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:43.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:43.256798+0000) 2022-09-08T07:09:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:43.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:43.188499+0000) 2022-09-08T07:09:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:44.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:44.257016+0000) 2022-09-08T07:09:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:44.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:44.188761+0000) 2022-09-08T07:09:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:44 smithi055 ceph-mon[97033]: pgmap v18527: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:44 smithi059 ceph-mon[97450]: pgmap v18527: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:45.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:45.257301+0000) 2022-09-08T07:09:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:45.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:45.189027+0000) 2022-09-08T07:09:46.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:46.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:46.189203+0000) 2022-09-08T07:09:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:46.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:46.257552+0000) 2022-09-08T07:09:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:46 smithi055 ceph-mon[97033]: pgmap v18528: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:09:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:46 smithi059 ceph-mon[97450]: pgmap v18528: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:09:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:47.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:47.257743+0000) 2022-09-08T07:09:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:47.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:47.189410+0000) 2022-09-08T07:09:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:48.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:48.257891+0000) 2022-09-08T07:09:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:48.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:48.189590+0000) 2022-09-08T07:09:48.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:48 smithi055 ceph-mon[97033]: pgmap v18529: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:48 smithi059 ceph-mon[97450]: pgmap v18529: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:49.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:49.258128+0000) 2022-09-08T07:09:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:49.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:49.189814+0000) 2022-09-08T07:09:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:50.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:50.258309+0000) 2022-09-08T07:09:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:50.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:50.189977+0000) 2022-09-08T07:09:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:50 smithi055 ceph-mon[97033]: pgmap v18530: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:50.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:50 smithi059 ceph-mon[97450]: pgmap v18530: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:51.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:51.258490+0000) 2022-09-08T07:09:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:51.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:51.190146+0000) 2022-09-08T07:09:52.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:09:52.134+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:09:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:52.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:52.190301+0000) 2022-09-08T07:09:52.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:52.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:52.258731+0000) 2022-09-08T07:09:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:52 smithi055 ceph-mon[97033]: pgmap v18531: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:09:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:52 smithi059 ceph-mon[97450]: pgmap v18531: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:09:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:53.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:53.258979+0000) 2022-09-08T07:09:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:53.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:53.190485+0000) 2022-09-08T07:09:54.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:54.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:54.190663+0000) 2022-09-08T07:09:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:54.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:54.259201+0000) 2022-09-08T07:09:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:54 smithi055 ceph-mon[97033]: pgmap v18532: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:54 smithi059 ceph-mon[97450]: pgmap v18532: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:55.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:55.259490+0000) 2022-09-08T07:09:55.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:55.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:55.190884+0000) 2022-09-08T07:09:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:56.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:56.259778+0000) 2022-09-08T07:09:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:56.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:56.191102+0000) 2022-09-08T07:09:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:56 smithi055 ceph-mon[97033]: pgmap v18533: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:09:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:56 smithi059 ceph-mon[97450]: pgmap v18533: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:09:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:57.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:57.260032+0000) 2022-09-08T07:09:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:57.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:57.191295+0000) 2022-09-08T07:09:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:58.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:58.260268+0000) 2022-09-08T07:09:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:58.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:58.191462+0000) 2022-09-08T07:09:58.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:09:58 smithi055 ceph-mon[97033]: pgmap v18534: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:09:58 smithi059 ceph-mon[97450]: pgmap v18534: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:09:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:09:59.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:59.260535+0000) 2022-09-08T07:09:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:09:59.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:09:59.191611+0000) 2022-09-08T07:10:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:00.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:00.260778+0000) 2022-09-08T07:10:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:00.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:00.191795+0000) 2022-09-08T07:10:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:00 smithi055 ceph-mon[97033]: pgmap v18535: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T07:10:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T07:10:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T07:10:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T07:10:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T07:10:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:00 smithi059 ceph-mon[97450]: pgmap v18535: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T07:10:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T07:10:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T07:10:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T07:10:00.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T07:10:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:01.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:01.260995+0000) 2022-09-08T07:10:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:01.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:01.191936+0000) 2022-09-08T07:10:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:02.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:02.261202+0000) 2022-09-08T07:10:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:02.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:02.192094+0000) 2022-09-08T07:10:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:02 smithi055 ceph-mon[97033]: pgmap v18536: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:10:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:02 smithi059 ceph-mon[97450]: pgmap v18536: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:10:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:03.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:03.261423+0000) 2022-09-08T07:10:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:03.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:03.192250+0000) 2022-09-08T07:10:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:04.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:04.261566+0000) 2022-09-08T07:10:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:04.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:04.192467+0000) 2022-09-08T07:10:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:04 smithi055 ceph-mon[97033]: pgmap v18537: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:04 smithi059 ceph-mon[97450]: pgmap v18537: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:05.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:05.261780+0000) 2022-09-08T07:10:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:05.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:05.192718+0000) 2022-09-08T07:10:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:06.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:06.261964+0000) 2022-09-08T07:10:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:06.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:06.192951+0000) 2022-09-08T07:10:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:06 smithi055 ceph-mon[97033]: pgmap v18538: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:10:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:10:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:06 smithi059 ceph-mon[97450]: pgmap v18538: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:10:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:10:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:07.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:07.262121+0000) 2022-09-08T07:10:07.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:10:07.148+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:10:07.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:07.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:07.193110+0000) 2022-09-08T07:10:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:10:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:10:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:10:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:10:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:10:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:10:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:10:07.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:10:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:08.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:08.262380+0000) 2022-09-08T07:10:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:08.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:08.193336+0000) 2022-09-08T07:10:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:08 smithi059 ceph-mon[97450]: pgmap v18539: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:10:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:08 smithi055 ceph-mon[97033]: pgmap v18539: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:10:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:09.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:09.262568+0000) 2022-09-08T07:10:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:09.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:09.193485+0000) 2022-09-08T07:10:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:10.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:10.262766+0000) 2022-09-08T07:10:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:10.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:10.193660+0000) 2022-09-08T07:10:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:10 smithi059 ceph-mon[97450]: pgmap v18540: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:10:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:10 smithi059 ceph-mon[97450]: osdmap e1236: 12 total, 11 up, 11 in 2022-09-08T07:10:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:10 smithi055 ceph-mon[97033]: pgmap v18540: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:10:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:10 smithi055 ceph-mon[97033]: osdmap e1236: 12 total, 11 up, 11 in 2022-09-08T07:10:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:11.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:11.262961+0000) 2022-09-08T07:10:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:11.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:11.193848+0000) 2022-09-08T07:10:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:12.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:12.263140+0000) 2022-09-08T07:10:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:12.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:12.194105+0000) 2022-09-08T07:10:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:12 smithi059 ceph-mon[97450]: pgmap v18542: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:10:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:12 smithi059 ceph-mon[97450]: osdmap e1237: 12 total, 11 up, 11 in 2022-09-08T07:10:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:12 smithi055 ceph-mon[97033]: pgmap v18542: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:10:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:12 smithi055 ceph-mon[97033]: osdmap e1237: 12 total, 11 up, 11 in 2022-09-08T07:10:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:13.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:13.263359+0000) 2022-09-08T07:10:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:13.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:13.194271+0000) 2022-09-08T07:10:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:14.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:14.194432+0000) 2022-09-08T07:10:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:14.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:14.263601+0000) 2022-09-08T07:10:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:14 smithi059 ceph-mon[97450]: pgmap v18544: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:10:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:14 smithi055 ceph-mon[97033]: pgmap v18544: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:10:15.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:15.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:15.194692+0000) 2022-09-08T07:10:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:15.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:15.263806+0000) 2022-09-08T07:10:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:16.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:16.194858+0000) 2022-09-08T07:10:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:16.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:16.263924+0000) 2022-09-08T07:10:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:16 smithi059 ceph-mon[97450]: pgmap v18545: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:10:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:16 smithi055 ceph-mon[97033]: pgmap v18545: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:10:17.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:17.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:17.194992+0000) 2022-09-08T07:10:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:17.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:17.264109+0000) 2022-09-08T07:10:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:18.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:18.264292+0000) 2022-09-08T07:10:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:18.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:18.195203+0000) 2022-09-08T07:10:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:18 smithi059 ceph-mon[97450]: pgmap v18546: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:10:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:18 smithi055 ceph-mon[97033]: pgmap v18546: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:10:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:19.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:19.195365+0000) 2022-09-08T07:10:19.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:19.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:19.264486+0000) 2022-09-08T07:10:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:20.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:20.195532+0000) 2022-09-08T07:10:20.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:20.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:20.264774+0000) 2022-09-08T07:10:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:20 smithi059 ceph-mon[97450]: pgmap v18547: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T07:10:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:20 smithi055 ceph-mon[97033]: pgmap v18547: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T07:10:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:21.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:21.264895+0000) 2022-09-08T07:10:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:21.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:21.195792+0000) 2022-09-08T07:10:22.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:10:22.163+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:10:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:22.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:22.265041+0000) 2022-09-08T07:10:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:22.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:22.195954+0000) 2022-09-08T07:10:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:22 smithi059 ceph-mon[97450]: pgmap v18548: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:10:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:22 smithi055 ceph-mon[97033]: pgmap v18548: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:10:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:23.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:23.196097+0000) 2022-09-08T07:10:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:23.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:23.265265+0000) 2022-09-08T07:10:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:24.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:24.265495+0000) 2022-09-08T07:10:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:24.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:24.196328+0000) 2022-09-08T07:10:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:24 smithi059 ceph-mon[97450]: pgmap v18549: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T07:10:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:24 smithi055 ceph-mon[97033]: pgmap v18549: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 93 B/s wr, 5 op/s 2022-09-08T07:10:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:25.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:25.265745+0000) 2022-09-08T07:10:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:25.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:25.196534+0000) 2022-09-08T07:10:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:26.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:26.196733+0000) 2022-09-08T07:10:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:26.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:26.265977+0000) 2022-09-08T07:10:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:26 smithi059 ceph-mon[97450]: pgmap v18550: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:10:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:26 smithi055 ceph-mon[97033]: pgmap v18550: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:10:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:27.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:27.266186+0000) 2022-09-08T07:10:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:27.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:27.196866+0000) 2022-09-08T07:10:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:28.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:28.266393+0000) 2022-09-08T07:10:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:28.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:28.197041+0000) 2022-09-08T07:10:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:28 smithi059 ceph-mon[97450]: pgmap v18551: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:28 smithi055 ceph-mon[97033]: pgmap v18551: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:29.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:29.266654+0000) 2022-09-08T07:10:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:29.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:29.197196+0000) 2022-09-08T07:10:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:30.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:30.266806+0000) 2022-09-08T07:10:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:30.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:30.197373+0000) 2022-09-08T07:10:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:30 smithi059 ceph-mon[97450]: pgmap v18552: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:30 smithi055 ceph-mon[97033]: pgmap v18552: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:31.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:31.267003+0000) 2022-09-08T07:10:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:31.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:31.197512+0000) 2022-09-08T07:10:32.542 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:32.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:32.267142+0000) 2022-09-08T07:10:32.543 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:32.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:32.197749+0000) 2022-09-08T07:10:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:32 smithi059 ceph-mon[97450]: pgmap v18553: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:10:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:32 smithi055 ceph-mon[97033]: pgmap v18553: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:10:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:33.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:33.267313+0000) 2022-09-08T07:10:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:33.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:33.197909+0000) 2022-09-08T07:10:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:34.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:34.267469+0000) 2022-09-08T07:10:34.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:34.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:34.198076+0000) 2022-09-08T07:10:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:34 smithi055 ceph-mon[97033]: pgmap v18554: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:34 smithi059 ceph-mon[97450]: pgmap v18554: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:35.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:35.267693+0000) 2022-09-08T07:10:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:35.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:35.198216+0000) 2022-09-08T07:10:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:36.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:36.267874+0000) 2022-09-08T07:10:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:36.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:36.198395+0000) 2022-09-08T07:10:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:36 smithi055 ceph-mon[97033]: pgmap v18555: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:10:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:36 smithi059 ceph-mon[97450]: pgmap v18555: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:10:37.595 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:37.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:37.267997+0000) 2022-09-08T07:10:37.596 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:37.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:37.198540+0000) 2022-09-08T07:10:37.596 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:10:37.178+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:10:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:38.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:38.198717+0000) 2022-09-08T07:10:38.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:38.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:38.268132+0000) 2022-09-08T07:10:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:38 smithi055 ceph-mon[97033]: pgmap v18556: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:38.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:38 smithi055 ceph-mon[97033]: pgmap v18557: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:10:38.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:38 smithi055 ceph-mon[97033]: pgmap v18558: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:10:38.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:10:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:38 smithi059 ceph-mon[97450]: pgmap v18556: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:38 smithi059 ceph-mon[97450]: pgmap v18557: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:10:39.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:38 smithi059 ceph-mon[97450]: pgmap v18558: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:10:39.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:10:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:39.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:39.198916+0000) 2022-09-08T07:10:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:39.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:39.268339+0000) 2022-09-08T07:10:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:40.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:40.199087+0000) 2022-09-08T07:10:40.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:40.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:40.268553+0000) 2022-09-08T07:10:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:40 smithi055 ceph-mon[97033]: pgmap v18559: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:10:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:40 smithi059 ceph-mon[97450]: pgmap v18559: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:10:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:41.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:41.268812+0000) 2022-09-08T07:10:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:41.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:41.199332+0000) 2022-09-08T07:10:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:42.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:42.268991+0000) 2022-09-08T07:10:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:42.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:42.199471+0000) 2022-09-08T07:10:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:42 smithi055 ceph-mon[97033]: pgmap v18560: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:10:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:42 smithi059 ceph-mon[97450]: pgmap v18560: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:10:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:43.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:43.269237+0000) 2022-09-08T07:10:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:43.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:43.199617+0000) 2022-09-08T07:10:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:44.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:44.269449+0000) 2022-09-08T07:10:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:44.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:44.199805+0000) 2022-09-08T07:10:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:44 smithi055 ceph-mon[97033]: pgmap v18561: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:10:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:44 smithi059 ceph-mon[97450]: pgmap v18561: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:10:45.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:45.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:45.200045+0000) 2022-09-08T07:10:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:45.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:45.269621+0000) 2022-09-08T07:10:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:46.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:46.269880+0000) 2022-09-08T07:10:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:46.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:46.200193+0000) 2022-09-08T07:10:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:46 smithi055 ceph-mon[97033]: pgmap v18562: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:10:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:46 smithi059 ceph-mon[97450]: pgmap v18562: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:10:47.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:47.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:47.200381+0000) 2022-09-08T07:10:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:47.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:47.270084+0000) 2022-09-08T07:10:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:48.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:48.270318+0000) 2022-09-08T07:10:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:48.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:48.200618+0000) 2022-09-08T07:10:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:48 smithi055 ceph-mon[97033]: pgmap v18563: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:10:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:48 smithi059 ceph-mon[97450]: pgmap v18563: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:10:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:49.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:49.270533+0000) 2022-09-08T07:10:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:49.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:49.200775+0000) 2022-09-08T07:10:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:50.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:50.270761+0000) 2022-09-08T07:10:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:50.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:50.200994+0000) 2022-09-08T07:10:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:50 smithi055 ceph-mon[97033]: pgmap v18564: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:50 smithi059 ceph-mon[97450]: pgmap v18564: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:51.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:51.270940+0000) 2022-09-08T07:10:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:51.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:51.201232+0000) 2022-09-08T07:10:52.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:10:52.191+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:10:52.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:52.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:52.271196+0000) 2022-09-08T07:10:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:52.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:52.201402+0000) 2022-09-08T07:10:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:52 smithi055 ceph-mon[97033]: pgmap v18565: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:10:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:52 smithi059 ceph-mon[97450]: pgmap v18565: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:10:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:53.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:53.201613+0000) 2022-09-08T07:10:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:53.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:53.271391+0000) 2022-09-08T07:10:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:54.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:54.271627+0000) 2022-09-08T07:10:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:54.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:54.201805+0000) 2022-09-08T07:10:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:54 smithi055 ceph-mon[97033]: pgmap v18566: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:54 smithi059 ceph-mon[97450]: pgmap v18566: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:55.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:55.271923+0000) 2022-09-08T07:10:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:55.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:55.202011+0000) 2022-09-08T07:10:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:56.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:56.272082+0000) 2022-09-08T07:10:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:56.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:56.202262+0000) 2022-09-08T07:10:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:56 smithi055 ceph-mon[97033]: pgmap v18567: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:10:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:56 smithi059 ceph-mon[97450]: pgmap v18567: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:10:57.649 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:57.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:57.272322+0000) 2022-09-08T07:10:57.649 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:57.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:57.202455+0000) 2022-09-08T07:10:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:58.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:58.202736+0000) 2022-09-08T07:10:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:58.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:58.272503+0000) 2022-09-08T07:10:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:10:58 smithi055 ceph-mon[97033]: pgmap v18568: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:10:58 smithi059 ceph-mon[97450]: pgmap v18568: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:10:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:10:59.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:59.272737+0000) 2022-09-08T07:10:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:10:59.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:10:59.202970+0000) 2022-09-08T07:11:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:00.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:00.272899+0000) 2022-09-08T07:11:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:00.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:00.203142+0000) 2022-09-08T07:11:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:00 smithi055 ceph-mon[97033]: pgmap v18569: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:00 smithi059 ceph-mon[97450]: pgmap v18569: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:01.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:01.273126+0000) 2022-09-08T07:11:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:01.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:01.203353+0000) 2022-09-08T07:11:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:02.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:02.273346+0000) 2022-09-08T07:11:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:02.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:02.203587+0000) 2022-09-08T07:11:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:02 smithi055 ceph-mon[97033]: pgmap v18570: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:11:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:02 smithi059 ceph-mon[97450]: pgmap v18570: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:11:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:03.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:03.273567+0000) 2022-09-08T07:11:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:03.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:03.203801+0000) 2022-09-08T07:11:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:04.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:04.203950+0000) 2022-09-08T07:11:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:04.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:04.273807+0000) 2022-09-08T07:11:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:04 smithi055 ceph-mon[97033]: pgmap v18571: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:04 smithi059 ceph-mon[97450]: pgmap v18571: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:05.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:05.204127+0000) 2022-09-08T07:11:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:05.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:05.274022+0000) 2022-09-08T07:11:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:06.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:06.274219+0000) 2022-09-08T07:11:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:06.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:06.204366+0000) 2022-09-08T07:11:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:06 smithi055 ceph-mon[97033]: pgmap v18572: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:11:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:11:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:06 smithi059 ceph-mon[97450]: pgmap v18572: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:11:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:11:07.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:11:07.206+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:11:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:07.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:07.274440+0000) 2022-09-08T07:11:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:07.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:07.204550+0000) 2022-09-08T07:11:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:11:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:11:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:08.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:08.274727+0000) 2022-09-08T07:11:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:08.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:08.204762+0000) 2022-09-08T07:11:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:08 smithi055 ceph-mon[97033]: pgmap v18573: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:11:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:08 smithi059 ceph-mon[97450]: pgmap v18573: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:11:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:09.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:09.274893+0000) 2022-09-08T07:11:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:09.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:09.204906+0000) 2022-09-08T07:11:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:10.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:10.275085+0000) 2022-09-08T07:11:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:10.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:10.205120+0000) 2022-09-08T07:11:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:10 smithi055 ceph-mon[97033]: pgmap v18574: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:11:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:10 smithi055 ceph-mon[97033]: osdmap e1238: 12 total, 11 up, 11 in 2022-09-08T07:11:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:10 smithi059 ceph-mon[97450]: pgmap v18574: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:11:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:10 smithi059 ceph-mon[97450]: osdmap e1238: 12 total, 11 up, 11 in 2022-09-08T07:11:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:11.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:11.275300+0000) 2022-09-08T07:11:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:11.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:11.205289+0000) 2022-09-08T07:11:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:12.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:12.275455+0000) 2022-09-08T07:11:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:12.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:12.205495+0000) 2022-09-08T07:11:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:12 smithi055 ceph-mon[97033]: pgmap v18576: 129 pgs: 1 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:11:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:12 smithi055 ceph-mon[97033]: osdmap e1239: 12 total, 11 up, 11 in 2022-09-08T07:11:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:12 smithi059 ceph-mon[97450]: pgmap v18576: 129 pgs: 1 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:11:13.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:12 smithi059 ceph-mon[97450]: osdmap e1239: 12 total, 11 up, 11 in 2022-09-08T07:11:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:13.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:13.275719+0000) 2022-09-08T07:11:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:13.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:13.205692+0000) 2022-09-08T07:11:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:14.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:14.275928+0000) 2022-09-08T07:11:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:14.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:14.205836+0000) 2022-09-08T07:11:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:14 smithi055 ceph-mon[97033]: pgmap v18578: 129 pgs: 1 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:11:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:14 smithi059 ceph-mon[97450]: pgmap v18578: 129 pgs: 1 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:11:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:15.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:15.276126+0000) 2022-09-08T07:11:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:15.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:15.206021+0000) 2022-09-08T07:11:16.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:16.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:16.206206+0000) 2022-09-08T07:11:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:16.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:16.276381+0000) 2022-09-08T07:11:16.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:16 smithi055 ceph-mon[97033]: pgmap v18579: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:11:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:16 smithi059 ceph-mon[97450]: pgmap v18579: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:11:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:17.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:17.276546+0000) 2022-09-08T07:11:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:17.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:17.206444+0000) 2022-09-08T07:11:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:18.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:18.276729+0000) 2022-09-08T07:11:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:18.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:18.206720+0000) 2022-09-08T07:11:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:18 smithi055 ceph-mon[97033]: osdmap e1240: 12 total, 11 up, 11 in 2022-09-08T07:11:18.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:18 smithi055 ceph-mon[97033]: pgmap v18581: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 258 B/s wr, 8 op/s 2022-09-08T07:11:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:18 smithi059 ceph-mon[97450]: osdmap e1240: 12 total, 11 up, 11 in 2022-09-08T07:11:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:18 smithi059 ceph-mon[97450]: pgmap v18581: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 258 B/s wr, 8 op/s 2022-09-08T07:11:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:19.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:19.276848+0000) 2022-09-08T07:11:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:19.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:19.206956+0000) 2022-09-08T07:11:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:20.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:20.277071+0000) 2022-09-08T07:11:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:20.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:20.207172+0000) 2022-09-08T07:11:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:20 smithi055 ceph-mon[97033]: pgmap v18582: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:11:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:20 smithi059 ceph-mon[97450]: pgmap v18582: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:11:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:21.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:21.277261+0000) 2022-09-08T07:11:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:21.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:21.207380+0000) 2022-09-08T07:11:22.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:11:22.218+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:11:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:22.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:22.277450+0000) 2022-09-08T07:11:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:22.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:22.207676+0000) 2022-09-08T07:11:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:22 smithi055 ceph-mon[97033]: pgmap v18583: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 111 B/s wr, 6 op/s 2022-09-08T07:11:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:22 smithi059 ceph-mon[97450]: pgmap v18583: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 111 B/s wr, 6 op/s 2022-09-08T07:11:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:23.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:23.277708+0000) 2022-09-08T07:11:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:23.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:23.207917+0000) 2022-09-08T07:11:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:24.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:24.277922+0000) 2022-09-08T07:11:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:24.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:24.208108+0000) 2022-09-08T07:11:24.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:24 smithi055 ceph-mon[97033]: pgmap v18584: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:11:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:24 smithi059 ceph-mon[97450]: pgmap v18584: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:11:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:25.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:25.278120+0000) 2022-09-08T07:11:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:25.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:25.208370+0000) 2022-09-08T07:11:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:26.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:26.278347+0000) 2022-09-08T07:11:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:26.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:26.208597+0000) 2022-09-08T07:11:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:26 smithi055 ceph-mon[97033]: pgmap v18585: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:11:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:26 smithi059 ceph-mon[97450]: pgmap v18585: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:11:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:27.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:27.278584+0000) 2022-09-08T07:11:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:27.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:27.208840+0000) 2022-09-08T07:11:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:28.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:28.278823+0000) 2022-09-08T07:11:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:28.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:28.209026+0000) 2022-09-08T07:11:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:28 smithi055 ceph-mon[97033]: pgmap v18586: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:11:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:28 smithi059 ceph-mon[97450]: pgmap v18586: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:11:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:29.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:29.279055+0000) 2022-09-08T07:11:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:29.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:29.209169+0000) 2022-09-08T07:11:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:30.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:30.279203+0000) 2022-09-08T07:11:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:30.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:30.209314+0000) 2022-09-08T07:11:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:30 smithi055 ceph-mon[97033]: pgmap v18587: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:30 smithi059 ceph-mon[97450]: pgmap v18587: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:31.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:31.279366+0000) 2022-09-08T07:11:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:31.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:31.209502+0000) 2022-09-08T07:11:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:32.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:32.279560+0000) 2022-09-08T07:11:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:32.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:32.209765+0000) 2022-09-08T07:11:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:32 smithi055 ceph-mon[97033]: pgmap v18588: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:11:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:32 smithi059 ceph-mon[97450]: pgmap v18588: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:11:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:33.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:33.279730+0000) 2022-09-08T07:11:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:33.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:33.210025+0000) 2022-09-08T07:11:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:34.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:34.279859+0000) 2022-09-08T07:11:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:34.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:34.210247+0000) 2022-09-08T07:11:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:34 smithi055 ceph-mon[97033]: pgmap v18589: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:34 smithi059 ceph-mon[97450]: pgmap v18589: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:35.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:35.280106+0000) 2022-09-08T07:11:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:35.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:35.210416+0000) 2022-09-08T07:11:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:36.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:36.280340+0000) 2022-09-08T07:11:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:36.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:36.210562+0000) 2022-09-08T07:11:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:36 smithi055 ceph-mon[97033]: pgmap v18590: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:11:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:36 smithi059 ceph-mon[97450]: pgmap v18590: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:11:37.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:11:37.231+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:11:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:37.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:37.280570+0000) 2022-09-08T07:11:37.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:37.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:37.210747+0000) 2022-09-08T07:11:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:38.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:38.210912+0000) 2022-09-08T07:11:38.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:38.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:38.280742+0000) 2022-09-08T07:11:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:38 smithi055 ceph-mon[97033]: pgmap v18591: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:11:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:11:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:11:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:38 smithi059 ceph-mon[97450]: pgmap v18591: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:11:38.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:11:38.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:11:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:39.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:39.211064+0000) 2022-09-08T07:11:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:39.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:39.280866+0000) 2022-09-08T07:11:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:40.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:40.281071+0000) 2022-09-08T07:11:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:40.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:40.211220+0000) 2022-09-08T07:11:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:40 smithi055 ceph-mon[97033]: pgmap v18592: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:40 smithi059 ceph-mon[97450]: pgmap v18592: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:41.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:41.211392+0000) 2022-09-08T07:11:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:41.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:41.281263+0000) 2022-09-08T07:11:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:42.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:42.281510+0000) 2022-09-08T07:11:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:42.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:42.211615+0000) 2022-09-08T07:11:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:42 smithi055 ceph-mon[97033]: pgmap v18593: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:11:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:42 smithi059 ceph-mon[97450]: pgmap v18593: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:11:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:43.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:43.281772+0000) 2022-09-08T07:11:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:43.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:43.211872+0000) 2022-09-08T07:11:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:44.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:44.281958+0000) 2022-09-08T07:11:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:44.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:44.212085+0000) 2022-09-08T07:11:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:44 smithi055 ceph-mon[97033]: pgmap v18594: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:44 smithi059 ceph-mon[97450]: pgmap v18594: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:45.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:45.282234+0000) 2022-09-08T07:11:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:45.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:45.212237+0000) 2022-09-08T07:11:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:46.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:46.282507+0000) 2022-09-08T07:11:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:46.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:46.212393+0000) 2022-09-08T07:11:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:46 smithi055 ceph-mon[97033]: pgmap v18595: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:11:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:46 smithi059 ceph-mon[97450]: pgmap v18595: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:11:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:47.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:47.282786+0000) 2022-09-08T07:11:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:47.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:47.212551+0000) 2022-09-08T07:11:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:48.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:48.283076+0000) 2022-09-08T07:11:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:48.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:48.212768+0000) 2022-09-08T07:11:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:48 smithi055 ceph-mon[97033]: pgmap v18596: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:48 smithi059 ceph-mon[97450]: pgmap v18596: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:49.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:49.283306+0000) 2022-09-08T07:11:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:49.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:49.212939+0000) 2022-09-08T07:11:50.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:50.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:50.283523+0000) 2022-09-08T07:11:50.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:50.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:50.213198+0000) 2022-09-08T07:11:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:50 smithi055 ceph-mon[97033]: pgmap v18597: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:50 smithi059 ceph-mon[97450]: pgmap v18597: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:51.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:51.283689+0000) 2022-09-08T07:11:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:51.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:51.213454+0000) 2022-09-08T07:11:52.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:11:52.241+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:11:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:52.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:52.283830+0000) 2022-09-08T07:11:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:52.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:52.213664+0000) 2022-09-08T07:11:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:52 smithi055 ceph-mon[97033]: pgmap v18598: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:11:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:52 smithi059 ceph-mon[97450]: pgmap v18598: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:11:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:53.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:53.284053+0000) 2022-09-08T07:11:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:53.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:53.213811+0000) 2022-09-08T07:11:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:54.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:54.284279+0000) 2022-09-08T07:11:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:54.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:54.214021+0000) 2022-09-08T07:11:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:54 smithi055 ceph-mon[97033]: pgmap v18599: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:54 smithi059 ceph-mon[97450]: pgmap v18599: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:55.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:55.284504+0000) 2022-09-08T07:11:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:55.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:55.214231+0000) 2022-09-08T07:11:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:56.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:56.284767+0000) 2022-09-08T07:11:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:56.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:56.214431+0000) 2022-09-08T07:11:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:56 smithi055 ceph-mon[97033]: pgmap v18600: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:11:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:56 smithi059 ceph-mon[97450]: pgmap v18600: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:11:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:57.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:57.284959+0000) 2022-09-08T07:11:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:57.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:57.214589+0000) 2022-09-08T07:11:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:58.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:58.285153+0000) 2022-09-08T07:11:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:58.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:58.214739+0000) 2022-09-08T07:11:58.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:11:58 smithi055 ceph-mon[97033]: pgmap v18601: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:11:58 smithi059 ceph-mon[97450]: pgmap v18601: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:11:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:11:59.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:59.285334+0000) 2022-09-08T07:11:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:11:59.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:11:59.214960+0000) 2022-09-08T07:12:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:00.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:00.215101+0000) 2022-09-08T07:12:00.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:00.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:00.285464+0000) 2022-09-08T07:12:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:00 smithi055 ceph-mon[97033]: pgmap v18602: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:00 smithi059 ceph-mon[97450]: pgmap v18602: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:01.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:01.285730+0000) 2022-09-08T07:12:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:01.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:01.215327+0000) 2022-09-08T07:12:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:02.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:02.285961+0000) 2022-09-08T07:12:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:02.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:02.215495+0000) 2022-09-08T07:12:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:02 smithi055 ceph-mon[97033]: pgmap v18603: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:12:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:02 smithi059 ceph-mon[97450]: pgmap v18603: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:12:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:03.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:03.286143+0000) 2022-09-08T07:12:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:03.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:03.215729+0000) 2022-09-08T07:12:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:04.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:04.215944+0000) 2022-09-08T07:12:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:04.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:04.286353+0000) 2022-09-08T07:12:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:04 smithi055 ceph-mon[97033]: pgmap v18604: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:04 smithi059 ceph-mon[97450]: pgmap v18604: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:05.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:05.286618+0000) 2022-09-08T07:12:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:05.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:05.216152+0000) 2022-09-08T07:12:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:06.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:06.286834+0000) 2022-09-08T07:12:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:06.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:06.216303+0000) 2022-09-08T07:12:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:06 smithi055 ceph-mon[97033]: pgmap v18605: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:12:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:12:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:06 smithi059 ceph-mon[97450]: pgmap v18605: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:12:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:12:07.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:12:07.254+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:12:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:07.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:07.216452+0000) 2022-09-08T07:12:07.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:07.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:07.287059+0000) 2022-09-08T07:12:07.963 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:12:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:12:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:08.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:08.287260+0000) 2022-09-08T07:12:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:08.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:08.216625+0000) 2022-09-08T07:12:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:08 smithi055 ceph-mon[97033]: pgmap v18606: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:12:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:12:09.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:08 smithi059 ceph-mon[97450]: pgmap v18606: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:12:09.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:12:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:09.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:09.287482+0000) 2022-09-08T07:12:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:09.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:09.216906+0000) 2022-09-08T07:12:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:09 smithi055 ceph-mon[97033]: pgmap v18607: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T07:12:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:09 smithi055 ceph-mon[97033]: pgmap v18608: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T07:12:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:09 smithi059 ceph-mon[97450]: pgmap v18607: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T07:12:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:09 smithi059 ceph-mon[97450]: pgmap v18608: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T07:12:10.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:10.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:10.217106+0000) 2022-09-08T07:12:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:10.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:10.287687+0000) 2022-09-08T07:12:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:10 smithi055 ceph-mon[97033]: osdmap e1241: 12 total, 11 up, 11 in 2022-09-08T07:12:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:10 smithi059 ceph-mon[97450]: osdmap e1241: 12 total, 11 up, 11 in 2022-09-08T07:12:11.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:11.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:11.287813+0000) 2022-09-08T07:12:11.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:11.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:11.217245+0000) 2022-09-08T07:12:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:11 smithi055 ceph-mon[97033]: pgmap v18610: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T07:12:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:11 smithi059 ceph-mon[97450]: pgmap v18610: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T07:12:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:12.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:12.217411+0000) 2022-09-08T07:12:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:12.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:12.288019+0000) 2022-09-08T07:12:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:13.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:13.288244+0000) 2022-09-08T07:12:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:13.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:13.217670+0000) 2022-09-08T07:12:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:13 smithi055 ceph-mon[97033]: pgmap v18611: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T07:12:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:13 smithi059 ceph-mon[97450]: pgmap v18611: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T07:12:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:14.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:14.288437+0000) 2022-09-08T07:12:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:14.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:14.217911+0000) 2022-09-08T07:12:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:15.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:15.288623+0000) 2022-09-08T07:12:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:15.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:15.218044+0000) 2022-09-08T07:12:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:15 smithi055 ceph-mon[97033]: pgmap v18612: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T07:12:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:15 smithi059 ceph-mon[97450]: pgmap v18612: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 164 B/s wr, 5 op/s 2022-09-08T07:12:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:16.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:16.288890+0000) 2022-09-08T07:12:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:16.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:16.218166+0000) 2022-09-08T07:12:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:17.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:17.289115+0000) 2022-09-08T07:12:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:17.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:17.218315+0000) 2022-09-08T07:12:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:17 smithi055 ceph-mon[97033]: pgmap v18613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:12:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:17 smithi055 ceph-mon[97033]: osdmap e1242: 12 total, 11 up, 11 in 2022-09-08T07:12:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:17 smithi059 ceph-mon[97450]: pgmap v18613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:12:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:17 smithi059 ceph-mon[97450]: osdmap e1242: 12 total, 11 up, 11 in 2022-09-08T07:12:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:18.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:18.289329+0000) 2022-09-08T07:12:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:18.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:18.218467+0000) 2022-09-08T07:12:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:19.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:19.218592+0000) 2022-09-08T07:12:19.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:19.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:19.289529+0000) 2022-09-08T07:12:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:19 smithi055 ceph-mon[97033]: pgmap v18615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T07:12:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:19 smithi059 ceph-mon[97450]: pgmap v18615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 250 B/s wr, 7 op/s 2022-09-08T07:12:20.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:20.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:20.218718+0000) 2022-09-08T07:12:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:20.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:20.289780+0000) 2022-09-08T07:12:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:21.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:21.289920+0000) 2022-09-08T07:12:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:21.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:21.218963+0000) 2022-09-08T07:12:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:21 smithi055 ceph-mon[97033]: pgmap v18616: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:12:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:21 smithi059 ceph-mon[97450]: pgmap v18616: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:12:22.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:12:22.267+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:12:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:22.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:22.219138+0000) 2022-09-08T07:12:22.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:22.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:22.290078+0000) 2022-09-08T07:12:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:23.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:23.290247+0000) 2022-09-08T07:12:23.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:23.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:23.219271+0000) 2022-09-08T07:12:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:23 smithi055 ceph-mon[97033]: pgmap v18617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:12:23.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:23 smithi059 ceph-mon[97450]: pgmap v18617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:12:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:24.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:24.290491+0000) 2022-09-08T07:12:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:24.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:24.219405+0000) 2022-09-08T07:12:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:25.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:25.290661+0000) 2022-09-08T07:12:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:25.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:25.219558+0000) 2022-09-08T07:12:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:25 smithi055 ceph-mon[97033]: pgmap v18618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:12:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:25 smithi059 ceph-mon[97450]: pgmap v18618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:12:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:26.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:26.290832+0000) 2022-09-08T07:12:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:26.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:26.219711+0000) 2022-09-08T07:12:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:27.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:27.291065+0000) 2022-09-08T07:12:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:27.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:27.219828+0000) 2022-09-08T07:12:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:27 smithi055 ceph-mon[97033]: pgmap v18619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:12:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:27 smithi059 ceph-mon[97450]: pgmap v18619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:12:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:28.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:28.291305+0000) 2022-09-08T07:12:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:28.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:28.219957+0000) 2022-09-08T07:12:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:29.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:29.291529+0000) 2022-09-08T07:12:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:29.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:29.220156+0000) 2022-09-08T07:12:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:29 smithi055 ceph-mon[97033]: pgmap v18620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:12:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:29 smithi059 ceph-mon[97450]: pgmap v18620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:12:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:30.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:30.291780+0000) 2022-09-08T07:12:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:30.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:30.220311+0000) 2022-09-08T07:12:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:31.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:31.292016+0000) 2022-09-08T07:12:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:31.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:31.220449+0000) 2022-09-08T07:12:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:31 smithi055 ceph-mon[97033]: pgmap v18621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:12:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:31 smithi059 ceph-mon[97450]: pgmap v18621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:12:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:32.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:32.292228+0000) 2022-09-08T07:12:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:32.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:32.220617+0000) 2022-09-08T07:12:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:33.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:33.292445+0000) 2022-09-08T07:12:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:33.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:33.220811+0000) 2022-09-08T07:12:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:33 smithi055 ceph-mon[97033]: pgmap v18622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:33 smithi059 ceph-mon[97450]: pgmap v18622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:34.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:34.292677+0000) 2022-09-08T07:12:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:34.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:34.220948+0000) 2022-09-08T07:12:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:35.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:35.292865+0000) 2022-09-08T07:12:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:35.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:35.221149+0000) 2022-09-08T07:12:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:35 smithi055 ceph-mon[97033]: pgmap v18623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:35 smithi059 ceph-mon[97450]: pgmap v18623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:36.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:36.293084+0000) 2022-09-08T07:12:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:36.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:36.221314+0000) 2022-09-08T07:12:37.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:12:37.276+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:12:37.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:37.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:37.293308+0000) 2022-09-08T07:12:37.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:37.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:37.221444+0000) 2022-09-08T07:12:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:37 smithi055 ceph-mon[97033]: pgmap v18624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:12:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:37 smithi059 ceph-mon[97450]: pgmap v18624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:12:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:38.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:38.293528+0000) 2022-09-08T07:12:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:38.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:38.221570+0000) 2022-09-08T07:12:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:39.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:39.293761+0000) 2022-09-08T07:12:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:39.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:39.221776+0000) 2022-09-08T07:12:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:39 smithi055 ceph-mon[97033]: pgmap v18625: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:39 smithi059 ceph-mon[97450]: pgmap v18625: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:40.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:40.293981+0000) 2022-09-08T07:12:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:40.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:40.221944+0000) 2022-09-08T07:12:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:41.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:41.294244+0000) 2022-09-08T07:12:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:41.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:41.222165+0000) 2022-09-08T07:12:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:41 smithi055 ceph-mon[97033]: pgmap v18626: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:12:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:41 smithi059 ceph-mon[97450]: pgmap v18626: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:12:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:42.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:42.294497+0000) 2022-09-08T07:12:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:42.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:42.222393+0000) 2022-09-08T07:12:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:43.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:43.294725+0000) 2022-09-08T07:12:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:43.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:43.222603+0000) 2022-09-08T07:12:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:43 smithi055 ceph-mon[97033]: pgmap v18627: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:43 smithi059 ceph-mon[97450]: pgmap v18627: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:44.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:44.294942+0000) 2022-09-08T07:12:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:44.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:44.222800+0000) 2022-09-08T07:12:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:45.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:45.295159+0000) 2022-09-08T07:12:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:45.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:45.222993+0000) 2022-09-08T07:12:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:45 smithi055 ceph-mon[97033]: pgmap v18628: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:45 smithi059 ceph-mon[97450]: pgmap v18628: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:46.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:46.295322+0000) 2022-09-08T07:12:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:46.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:46.223150+0000) 2022-09-08T07:12:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:47.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:47.295490+0000) 2022-09-08T07:12:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:47.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:47.223344+0000) 2022-09-08T07:12:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:47 smithi059 ceph-mon[97450]: pgmap v18629: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:12:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:47 smithi055 ceph-mon[97033]: pgmap v18629: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:12:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:48.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:48.295718+0000) 2022-09-08T07:12:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:48.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:48.223566+0000) 2022-09-08T07:12:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:49.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:49.295957+0000) 2022-09-08T07:12:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:49.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:49.223781+0000) 2022-09-08T07:12:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:49 smithi059 ceph-mon[97450]: pgmap v18630: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:49 smithi055 ceph-mon[97033]: pgmap v18630: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:50.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:50.296211+0000) 2022-09-08T07:12:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:50.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:50.224010+0000) 2022-09-08T07:12:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:51.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:51.296419+0000) 2022-09-08T07:12:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:51.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:51.224180+0000) 2022-09-08T07:12:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:51 smithi059 ceph-mon[97450]: pgmap v18631: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:12:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:51 smithi055 ceph-mon[97033]: pgmap v18631: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:12:52.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:12:52.293+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:12:52.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:52.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:52.296610+0000) 2022-09-08T07:12:52.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:52.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:52.224335+0000) 2022-09-08T07:12:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:53.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:53.224549+0000) 2022-09-08T07:12:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:53.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:53.296797+0000) 2022-09-08T07:12:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:53 smithi059 ceph-mon[97450]: pgmap v18632: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:53 smithi055 ceph-mon[97033]: pgmap v18632: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:54.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:54.297017+0000) 2022-09-08T07:12:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:54.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:54.224705+0000) 2022-09-08T07:12:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:55.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:55.297271+0000) 2022-09-08T07:12:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:55.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:55.224916+0000) 2022-09-08T07:12:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:55 smithi059 ceph-mon[97450]: pgmap v18633: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:55.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:55 smithi055 ceph-mon[97033]: pgmap v18633: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:56.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:56.297475+0000) 2022-09-08T07:12:56.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:56.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:56.225046+0000) 2022-09-08T07:12:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:57.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:57.297663+0000) 2022-09-08T07:12:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:57.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:57.225187+0000) 2022-09-08T07:12:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:57 smithi059 ceph-mon[97450]: pgmap v18634: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:12:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:57 smithi055 ceph-mon[97033]: pgmap v18634: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:12:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:58.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:58.297893+0000) 2022-09-08T07:12:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:58.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:58.225398+0000) 2022-09-08T07:12:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:12:59.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:59.298113+0000) 2022-09-08T07:12:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:12:59.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:12:59.225625+0000) 2022-09-08T07:12:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:12:59 smithi059 ceph-mon[97450]: pgmap v18635: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:12:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:12:59 smithi055 ceph-mon[97033]: pgmap v18635: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:00.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:00.298374+0000) 2022-09-08T07:13:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:00.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:00.225868+0000) 2022-09-08T07:13:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:01.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:01.298626+0000) 2022-09-08T07:13:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:01.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:01.226014+0000) 2022-09-08T07:13:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:01 smithi059 ceph-mon[97450]: pgmap v18636: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:13:01.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:01 smithi055 ceph-mon[97033]: pgmap v18636: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:13:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:02.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:02.298910+0000) 2022-09-08T07:13:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:02.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:02.226157+0000) 2022-09-08T07:13:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:03.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:03.299105+0000) 2022-09-08T07:13:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:03.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:03.226286+0000) 2022-09-08T07:13:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:03 smithi059 ceph-mon[97450]: pgmap v18637: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:03 smithi055 ceph-mon[97033]: pgmap v18637: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:04.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:04.299310+0000) 2022-09-08T07:13:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:04.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:04.226506+0000) 2022-09-08T07:13:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:05.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:05.299476+0000) 2022-09-08T07:13:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:05.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:05.226743+0000) 2022-09-08T07:13:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:05 smithi059 ceph-mon[97450]: pgmap v18638: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:05.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:05 smithi055 ceph-mon[97033]: pgmap v18638: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:06.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:06.299709+0000) 2022-09-08T07:13:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:06.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:06.226985+0000) 2022-09-08T07:13:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:13:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:13:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:07.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:07.299952+0000) 2022-09-08T07:13:07.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:13:07.308+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:13:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:07.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:07.227216+0000) 2022-09-08T07:13:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:07 smithi059 ceph-mon[97450]: pgmap v18639: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:13:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:13:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:07 smithi055 ceph-mon[97033]: pgmap v18639: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:13:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:13:08.302 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:08.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:08.227793+0000) 2022-09-08T07:13:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:08.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:08.300200+0000) 2022-09-08T07:13:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:13:08.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:13:08.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:13:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:13:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:13:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:13:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:09.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:09.300445+0000) 2022-09-08T07:13:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:09.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:09.227971+0000) 2022-09-08T07:13:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:09 smithi059 ceph-mon[97450]: pgmap v18640: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:13:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:09 smithi055 ceph-mon[97033]: pgmap v18640: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:13:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:10.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:10.300737+0000) 2022-09-08T07:13:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:10.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:10.228152+0000) 2022-09-08T07:13:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:10 smithi055 ceph-mon[97033]: osdmap e1243: 12 total, 11 up, 11 in 2022-09-08T07:13:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:10 smithi059 ceph-mon[97450]: osdmap e1243: 12 total, 11 up, 11 in 2022-09-08T07:13:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:11.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:11.300981+0000) 2022-09-08T07:13:11.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:11.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:11.228304+0000) 2022-09-08T07:13:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:11 smithi055 ceph-mon[97033]: pgmap v18642: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:13:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:11 smithi059 ceph-mon[97450]: pgmap v18642: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:13:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:12.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:12.301211+0000) 2022-09-08T07:13:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:12.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:12.228544+0000) 2022-09-08T07:13:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:13.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:13.301457+0000) 2022-09-08T07:13:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:13.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:13.228744+0000) 2022-09-08T07:13:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:13 smithi055 ceph-mon[97033]: pgmap v18643: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:13:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:13 smithi059 ceph-mon[97450]: pgmap v18643: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:13:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:14.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:14.301661+0000) 2022-09-08T07:13:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:14.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:14.228991+0000) 2022-09-08T07:13:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:15.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:15.301901+0000) 2022-09-08T07:13:15.694 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:15.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:15.229233+0000) 2022-09-08T07:13:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:15 smithi055 ceph-mon[97033]: pgmap v18644: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:13:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:15 smithi059 ceph-mon[97450]: pgmap v18644: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:13:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:16.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:16.302062+0000) 2022-09-08T07:13:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:16.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:16.229409+0000) 2022-09-08T07:13:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:17.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:17.302267+0000) 2022-09-08T07:13:17.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:17.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:17.229693+0000) 2022-09-08T07:13:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:17 smithi055 ceph-mon[97033]: pgmap v18645: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:13:17.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:17 smithi055 ceph-mon[97033]: osdmap e1244: 12 total, 11 up, 11 in 2022-09-08T07:13:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:17 smithi059 ceph-mon[97450]: pgmap v18645: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:13:18.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:17 smithi059 ceph-mon[97450]: osdmap e1244: 12 total, 11 up, 11 in 2022-09-08T07:13:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:18.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:18.302430+0000) 2022-09-08T07:13:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:18.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:18.229927+0000) 2022-09-08T07:13:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:19.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:19.230164+0000) 2022-09-08T07:13:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:19.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:19.302611+0000) 2022-09-08T07:13:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:19 smithi055 ceph-mon[97033]: pgmap v18647: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T07:13:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:19 smithi059 ceph-mon[97450]: pgmap v18647: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2022-09-08T07:13:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:20.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:20.302816+0000) 2022-09-08T07:13:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:20.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:20.230382+0000) 2022-09-08T07:13:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:21.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:21.303051+0000) 2022-09-08T07:13:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:21.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:21.230625+0000) 2022-09-08T07:13:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:21 smithi055 ceph-mon[97033]: pgmap v18648: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:13:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:21 smithi059 ceph-mon[97450]: pgmap v18648: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:13:22.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:13:22.317+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:13:22.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:22.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:22.303238+0000) 2022-09-08T07:13:22.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:22.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:22.230836+0000) 2022-09-08T07:13:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:23.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:23.303443+0000) 2022-09-08T07:13:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:23.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:23.231010+0000) 2022-09-08T07:13:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:23 smithi055 ceph-mon[97033]: pgmap v18649: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:13:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:23 smithi059 ceph-mon[97450]: pgmap v18649: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:13:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:24.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:24.303691+0000) 2022-09-08T07:13:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:24.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:24.231157+0000) 2022-09-08T07:13:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:25.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:25.303926+0000) 2022-09-08T07:13:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:25.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:25.231373+0000) 2022-09-08T07:13:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:25 smithi055 ceph-mon[97033]: pgmap v18650: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:13:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:25 smithi059 ceph-mon[97450]: pgmap v18650: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:13:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:26.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:26.304103+0000) 2022-09-08T07:13:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:26.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:26.231542+0000) 2022-09-08T07:13:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:27.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:27.231692+0000) 2022-09-08T07:13:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:27.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:27.304305+0000) 2022-09-08T07:13:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:27 smithi055 ceph-mon[97033]: pgmap v18651: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:13:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:27 smithi059 ceph-mon[97450]: pgmap v18651: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:13:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:28.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:28.304461+0000) 2022-09-08T07:13:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:28.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:28.231884+0000) 2022-09-08T07:13:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:29.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:29.304663+0000) 2022-09-08T07:13:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:29.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:29.232081+0000) 2022-09-08T07:13:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:29 smithi055 ceph-mon[97033]: pgmap v18652: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:13:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:29 smithi059 ceph-mon[97450]: pgmap v18652: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:13:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:30.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:30.232298+0000) 2022-09-08T07:13:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:30.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:30.304829+0000) 2022-09-08T07:13:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:31.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:31.232540+0000) 2022-09-08T07:13:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:31.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:31.305022+0000) 2022-09-08T07:13:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:31 smithi055 ceph-mon[97033]: pgmap v18653: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:13:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:31 smithi059 ceph-mon[97450]: pgmap v18653: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:13:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:32.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:32.305229+0000) 2022-09-08T07:13:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:32.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:32.232793+0000) 2022-09-08T07:13:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:33.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:33.305480+0000) 2022-09-08T07:13:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:33.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:33.232961+0000) 2022-09-08T07:13:33.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:33 smithi055 ceph-mon[97033]: pgmap v18654: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:34.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:33 smithi059 ceph-mon[97450]: pgmap v18654: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:34.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:34.233179+0000) 2022-09-08T07:13:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:34.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:34.305687+0000) 2022-09-08T07:13:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:35.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:35.233413+0000) 2022-09-08T07:13:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:35.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:35.305923+0000) 2022-09-08T07:13:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:35 smithi055 ceph-mon[97033]: pgmap v18655: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:36.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:35 smithi059 ceph-mon[97450]: pgmap v18655: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:36.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:36.233579+0000) 2022-09-08T07:13:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:36.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:36.306172+0000) 2022-09-08T07:13:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:37.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:37.233734+0000) 2022-09-08T07:13:37.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:13:37.333+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:13:37.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:37.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:37.306397+0000) 2022-09-08T07:13:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:37 smithi055 ceph-mon[97033]: pgmap v18656: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:13:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:37 smithi059 ceph-mon[97450]: pgmap v18656: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:13:38.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:38.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:38.306671+0000) 2022-09-08T07:13:38.407 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:38.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:38.233883+0000) 2022-09-08T07:13:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:39 smithi055 ceph-mon[97033]: pgmap v18657: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:39.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:39 smithi055 ceph-mon[97033]: pgmap v18658: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:13:39.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:39 smithi055 ceph-mon[97033]: pgmap v18659: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:13:39.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:13:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:39 smithi059 ceph-mon[97450]: pgmap v18657: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:39.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:39 smithi059 ceph-mon[97450]: pgmap v18658: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:13:39.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:39 smithi059 ceph-mon[97450]: pgmap v18659: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:13:39.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:13:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:39.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:39.306910+0000) 2022-09-08T07:13:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:39.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:39.234111+0000) 2022-09-08T07:13:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:40.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:40.234350+0000) 2022-09-08T07:13:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:40.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:40.307071+0000) 2022-09-08T07:13:41.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:41 smithi055 ceph-mon[97033]: pgmap v18660: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:13:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:41 smithi059 ceph-mon[97450]: pgmap v18660: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:13:41.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:41.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:41.234526+0000) 2022-09-08T07:13:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:41.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:41.307239+0000) 2022-09-08T07:13:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:42.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:42.307472+0000) 2022-09-08T07:13:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:42.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:42.234779+0000) 2022-09-08T07:13:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:43.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:43.307685+0000) 2022-09-08T07:13:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:43.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:43.234964+0000) 2022-09-08T07:13:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:43 smithi055 ceph-mon[97033]: pgmap v18661: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:13:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:43 smithi059 ceph-mon[97450]: pgmap v18661: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:13:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:44.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:44.307813+0000) 2022-09-08T07:13:44.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:44.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:44.235159+0000) 2022-09-08T07:13:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:45.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:45.308045+0000) 2022-09-08T07:13:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:45.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:45.235408+0000) 2022-09-08T07:13:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:45 smithi055 ceph-mon[97033]: pgmap v18662: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:13:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:45 smithi059 ceph-mon[97450]: pgmap v18662: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:13:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:46.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:46.308291+0000) 2022-09-08T07:13:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:46.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:46.235628+0000) 2022-09-08T07:13:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:47.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:47.308481+0000) 2022-09-08T07:13:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:47.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:47.235895+0000) 2022-09-08T07:13:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:47 smithi055 ceph-mon[97033]: pgmap v18663: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:13:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:47 smithi059 ceph-mon[97450]: pgmap v18663: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:13:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:48.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:48.308695+0000) 2022-09-08T07:13:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:48.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:48.236047+0000) 2022-09-08T07:13:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:49.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:49.308895+0000) 2022-09-08T07:13:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:49.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:49.236269+0000) 2022-09-08T07:13:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:49 smithi055 ceph-mon[97033]: pgmap v18664: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:13:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:49 smithi059 ceph-mon[97450]: pgmap v18664: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:13:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:50.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:50.309057+0000) 2022-09-08T07:13:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:50.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:50.236513+0000) 2022-09-08T07:13:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:51.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:51.309264+0000) 2022-09-08T07:13:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:51.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:51.236760+0000) 2022-09-08T07:13:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:51 smithi059 ceph-mon[97450]: pgmap v18665: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:13:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:51 smithi055 ceph-mon[97033]: pgmap v18665: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:13:52.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:13:52.349+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:13:52.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:52.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:52.309435+0000) 2022-09-08T07:13:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:52.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:52.236999+0000) 2022-09-08T07:13:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:53.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:53.309665+0000) 2022-09-08T07:13:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:53.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:53.237250+0000) 2022-09-08T07:13:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:53 smithi059 ceph-mon[97450]: pgmap v18666: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:53 smithi055 ceph-mon[97033]: pgmap v18666: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:54.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:54.237470+0000) 2022-09-08T07:13:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:54.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:54.309896+0000) 2022-09-08T07:13:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:55.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:55.310126+0000) 2022-09-08T07:13:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:55.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:55.237742+0000) 2022-09-08T07:13:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:55 smithi059 ceph-mon[97450]: pgmap v18667: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:55 smithi055 ceph-mon[97033]: pgmap v18667: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:56.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:56.237968+0000) 2022-09-08T07:13:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:56.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:56.310357+0000) 2022-09-08T07:13:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:57.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:57.238166+0000) 2022-09-08T07:13:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:57.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:57.310564+0000) 2022-09-08T07:13:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:57 smithi059 ceph-mon[97450]: pgmap v18668: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:13:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:57 smithi055 ceph-mon[97033]: pgmap v18668: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:13:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:58.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:58.238390+0000) 2022-09-08T07:13:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:58.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:58.310786+0000) 2022-09-08T07:13:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:13:59.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:59.310991+0000) 2022-09-08T07:13:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:13:59.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:13:59.238565+0000) 2022-09-08T07:13:59.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:13:59 smithi059 ceph-mon[97450]: pgmap v18669: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:13:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:13:59 smithi055 ceph-mon[97033]: pgmap v18669: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:00.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:00.311202+0000) 2022-09-08T07:14:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:00.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:00.238777+0000) 2022-09-08T07:14:01.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:01.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:01.238973+0000) 2022-09-08T07:14:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:01.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:01.311369+0000) 2022-09-08T07:14:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:01 smithi059 ceph-mon[97450]: pgmap v18670: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:14:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:01 smithi055 ceph-mon[97033]: pgmap v18670: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:14:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:02.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:02.311510+0000) 2022-09-08T07:14:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:02.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:02.239154+0000) 2022-09-08T07:14:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:03.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:03.311746+0000) 2022-09-08T07:14:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:03.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:03.239374+0000) 2022-09-08T07:14:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:03 smithi059 ceph-mon[97450]: pgmap v18671: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:14:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:03 smithi055 ceph-mon[97033]: pgmap v18671: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:14:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:04.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:04.311991+0000) 2022-09-08T07:14:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:04.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:04.239559+0000) 2022-09-08T07:14:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:05.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:05.239815+0000) 2022-09-08T07:14:05.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:05.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:05.312250+0000) 2022-09-08T07:14:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:05 smithi059 ceph-mon[97450]: pgmap v18672: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:14:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:05 smithi055 ceph-mon[97033]: pgmap v18672: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:14:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:06.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:06.312463+0000) 2022-09-08T07:14:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:06.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:06.239956+0000) 2022-09-08T07:14:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:14:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:14:07.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:14:07.364+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:14:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:07.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:07.312598+0000) 2022-09-08T07:14:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:07.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:07.240097+0000) 2022-09-08T07:14:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:07 smithi059 ceph-mon[97450]: pgmap v18673: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:14:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:14:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:07 smithi055 ceph-mon[97033]: pgmap v18673: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:14:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:14:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:08.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:08.312822+0000) 2022-09-08T07:14:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:08.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:08.240350+0000) 2022-09-08T07:14:09.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:09.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:09.313001+0000) 2022-09-08T07:14:09.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:09.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:09.240515+0000) 2022-09-08T07:14:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:09 smithi059 ceph-mon[97450]: pgmap v18674: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:14:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:09 smithi055 ceph-mon[97033]: pgmap v18674: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:14:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:10.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:10.313220+0000) 2022-09-08T07:14:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:10.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:10.240726+0000) 2022-09-08T07:14:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:10 smithi059 ceph-mon[97450]: osdmap e1245: 12 total, 11 up, 11 in 2022-09-08T07:14:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:10 smithi055 ceph-mon[97033]: osdmap e1245: 12 total, 11 up, 11 in 2022-09-08T07:14:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:11.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:11.313474+0000) 2022-09-08T07:14:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:11.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:11.240883+0000) 2022-09-08T07:14:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:11 smithi059 ceph-mon[97450]: pgmap v18676: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:14:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:11 smithi055 ceph-mon[97033]: pgmap v18676: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:14:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:12.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:12.313723+0000) 2022-09-08T07:14:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:12.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:12.241042+0000) 2022-09-08T07:14:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:13.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:13.313958+0000) 2022-09-08T07:14:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:13.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:13.241209+0000) 2022-09-08T07:14:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:13 smithi055 ceph-mon[97033]: pgmap v18677: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:14:13.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:13 smithi055 ceph-mon[97033]: osdmap e1246: 12 total, 11 up, 11 in 2022-09-08T07:14:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:13 smithi059 ceph-mon[97450]: pgmap v18677: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:14:13.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:13 smithi059 ceph-mon[97450]: osdmap e1246: 12 total, 11 up, 11 in 2022-09-08T07:14:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:14.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:14.314174+0000) 2022-09-08T07:14:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:14.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:14.241447+0000) 2022-09-08T07:14:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:15.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:15.314327+0000) 2022-09-08T07:14:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:15.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:15.241603+0000) 2022-09-08T07:14:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:15 smithi055 ceph-mon[97033]: pgmap v18679: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:14:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:15 smithi059 ceph-mon[97450]: pgmap v18679: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:14:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:16.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:16.314599+0000) 2022-09-08T07:14:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:16.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:16.241791+0000) 2022-09-08T07:14:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:17.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:17.314734+0000) 2022-09-08T07:14:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:17.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:17.241978+0000) 2022-09-08T07:14:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:17 smithi059 ceph-mon[97450]: pgmap v18680: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:14:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:17 smithi055 ceph-mon[97033]: pgmap v18680: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:14:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:18.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:18.314904+0000) 2022-09-08T07:14:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:18.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:18.242189+0000) 2022-09-08T07:14:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:19.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:19.315153+0000) 2022-09-08T07:14:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:19.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:19.242347+0000) 2022-09-08T07:14:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:19 smithi059 ceph-mon[97450]: pgmap v18681: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 236 B/s wr, 7 op/s 2022-09-08T07:14:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:19 smithi055 ceph-mon[97033]: pgmap v18681: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 236 B/s wr, 7 op/s 2022-09-08T07:14:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:20.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:20.315402+0000) 2022-09-08T07:14:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:20.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:20.242537+0000) 2022-09-08T07:14:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:21.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:21.315632+0000) 2022-09-08T07:14:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:21.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:21.242735+0000) 2022-09-08T07:14:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:21 smithi059 ceph-mon[97450]: pgmap v18682: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:14:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:21 smithi055 ceph-mon[97033]: pgmap v18682: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:14:22.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:14:22.374+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:14:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:22.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:22.315859+0000) 2022-09-08T07:14:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:22.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:22.242953+0000) 2022-09-08T07:14:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:23.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:23.243109+0000) 2022-09-08T07:14:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:23.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:23.316033+0000) 2022-09-08T07:14:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:23 smithi059 ceph-mon[97450]: pgmap v18683: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:14:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:23 smithi055 ceph-mon[97033]: pgmap v18683: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:14:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:24.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:24.316263+0000) 2022-09-08T07:14:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:24.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:24.243280+0000) 2022-09-08T07:14:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:25.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:25.316455+0000) 2022-09-08T07:14:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:25.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:25.243480+0000) 2022-09-08T07:14:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:25 smithi059 ceph-mon[97450]: pgmap v18684: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 87 B/s wr, 5 op/s 2022-09-08T07:14:25.990 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:25 smithi055 ceph-mon[97033]: pgmap v18684: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 87 B/s wr, 5 op/s 2022-09-08T07:14:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:26.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:26.316651+0000) 2022-09-08T07:14:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:26.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:26.243665+0000) 2022-09-08T07:14:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:27.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:27.316779+0000) 2022-09-08T07:14:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:27.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:27.243844+0000) 2022-09-08T07:14:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:27 smithi059 ceph-mon[97450]: pgmap v18685: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:14:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:27 smithi055 ceph-mon[97033]: pgmap v18685: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:14:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:28.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:28.317001+0000) 2022-09-08T07:14:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:28.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:28.243991+0000) 2022-09-08T07:14:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:29.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:29.244143+0000) 2022-09-08T07:14:29.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:29.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:29.317201+0000) 2022-09-08T07:14:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:29 smithi059 ceph-mon[97450]: pgmap v18686: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:14:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:29 smithi055 ceph-mon[97033]: pgmap v18686: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:14:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:30.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:30.317417+0000) 2022-09-08T07:14:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:30.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:30.244389+0000) 2022-09-08T07:14:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:31.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:31.244619+0000) 2022-09-08T07:14:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:31.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:31.317569+0000) 2022-09-08T07:14:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:31 smithi059 ceph-mon[97450]: pgmap v18687: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:14:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:31 smithi055 ceph-mon[97033]: pgmap v18687: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:14:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:32.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:32.317794+0000) 2022-09-08T07:14:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:32.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:32.244815+0000) 2022-09-08T07:14:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:33.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:33.317978+0000) 2022-09-08T07:14:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:33.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:33.244975+0000) 2022-09-08T07:14:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:33 smithi059 ceph-mon[97450]: pgmap v18688: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:33 smithi055 ceph-mon[97033]: pgmap v18688: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:34.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:34.318178+0000) 2022-09-08T07:14:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:34.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:34.245113+0000) 2022-09-08T07:14:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:35.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:35.318403+0000) 2022-09-08T07:14:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:35.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:35.245319+0000) 2022-09-08T07:14:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:35 smithi059 ceph-mon[97450]: pgmap v18689: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:35 smithi055 ceph-mon[97033]: pgmap v18689: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:36.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:36.318579+0000) 2022-09-08T07:14:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:36.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:36.245532+0000) 2022-09-08T07:14:37.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:14:37.379+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:14:37.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:37.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:37.245716+0000) 2022-09-08T07:14:37.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:37.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:37.318777+0000) 2022-09-08T07:14:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:37 smithi059 ceph-mon[97450]: pgmap v18690: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:14:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:37 smithi055 ceph-mon[97033]: pgmap v18690: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:14:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:38.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:38.318920+0000) 2022-09-08T07:14:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:38.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:38.245915+0000) 2022-09-08T07:14:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:14:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:14:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:14:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:14:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:14:38.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:14:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:39.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:39.319150+0000) 2022-09-08T07:14:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:39.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:39.246055+0000) 2022-09-08T07:14:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:39 smithi059 ceph-mon[97450]: pgmap v18691: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:39 smithi055 ceph-mon[97033]: pgmap v18691: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:40.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:40.319388+0000) 2022-09-08T07:14:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:40.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:40.246219+0000) 2022-09-08T07:14:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:41.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:41.246389+0000) 2022-09-08T07:14:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:41.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:41.319599+0000) 2022-09-08T07:14:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:41 smithi059 ceph-mon[97450]: pgmap v18692: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:14:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:41 smithi055 ceph-mon[97033]: pgmap v18692: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:14:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:42.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:42.319830+0000) 2022-09-08T07:14:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:42.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:42.246629+0000) 2022-09-08T07:14:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:43.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:43.320095+0000) 2022-09-08T07:14:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:43.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:43.246778+0000) 2022-09-08T07:14:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:43 smithi055 ceph-mon[97033]: pgmap v18693: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:43 smithi059 ceph-mon[97450]: pgmap v18693: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:44.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:44.320358+0000) 2022-09-08T07:14:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:44.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:44.247028+0000) 2022-09-08T07:14:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:45.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:45.320581+0000) 2022-09-08T07:14:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:45.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:45.247195+0000) 2022-09-08T07:14:45.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:45 smithi055 ceph-mon[97033]: pgmap v18694: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:46.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:45 smithi059 ceph-mon[97450]: pgmap v18694: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:46.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:46.247374+0000) 2022-09-08T07:14:46.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:46.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:46.320842+0000) 2022-09-08T07:14:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:47.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:47.321081+0000) 2022-09-08T07:14:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:47.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:47.247507+0000) 2022-09-08T07:14:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:47 smithi055 ceph-mon[97033]: pgmap v18695: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:14:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:47 smithi059 ceph-mon[97450]: pgmap v18695: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:14:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:48.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:48.321218+0000) 2022-09-08T07:14:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:48.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:48.247753+0000) 2022-09-08T07:14:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:49.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:49.321348+0000) 2022-09-08T07:14:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:49.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:49.247979+0000) 2022-09-08T07:14:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:49 smithi055 ceph-mon[97033]: pgmap v18696: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:49 smithi059 ceph-mon[97450]: pgmap v18696: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:50.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:50.321602+0000) 2022-09-08T07:14:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:50.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:50.248154+0000) 2022-09-08T07:14:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:51.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:51.248343+0000) 2022-09-08T07:14:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:51.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:51.321801+0000) 2022-09-08T07:14:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:51 smithi055 ceph-mon[97033]: pgmap v18697: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:14:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:51 smithi059 ceph-mon[97450]: pgmap v18697: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:14:52.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:14:52.394+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:14:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:52.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:52.322007+0000) 2022-09-08T07:14:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:52.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:52.248549+0000) 2022-09-08T07:14:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:53.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:53.322249+0000) 2022-09-08T07:14:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:53.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:53.248767+0000) 2022-09-08T07:14:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:53 smithi055 ceph-mon[97033]: pgmap v18698: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:53 smithi059 ceph-mon[97450]: pgmap v18698: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:54.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:54.322539+0000) 2022-09-08T07:14:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:54.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:54.248999+0000) 2022-09-08T07:14:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:55.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:55.322828+0000) 2022-09-08T07:14:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:55.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:55.249167+0000) 2022-09-08T07:14:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:55 smithi055 ceph-mon[97033]: pgmap v18699: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:55 smithi059 ceph-mon[97450]: pgmap v18699: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:14:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:56.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:56.249366+0000) 2022-09-08T07:14:56.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:56.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:56.323062+0000) 2022-09-08T07:14:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:57.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:57.249586+0000) 2022-09-08T07:14:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:57.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:57.323278+0000) 2022-09-08T07:14:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:57 smithi055 ceph-mon[97033]: pgmap v18700: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:14:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:57 smithi059 ceph-mon[97450]: pgmap v18700: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:14:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:58.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:58.249778+0000) 2022-09-08T07:14:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:58.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:58.323470+0000) 2022-09-08T07:14:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:14:59.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:59.323626+0000) 2022-09-08T07:14:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:14:59.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:14:59.249951+0000) 2022-09-08T07:14:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:14:59 smithi055 ceph-mon[97033]: pgmap v18701: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:14:59 smithi059 ceph-mon[97450]: pgmap v18701: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:00.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:00.323771+0000) 2022-09-08T07:15:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:00.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:00.250090+0000) 2022-09-08T07:15:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:01.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:01.323894+0000) 2022-09-08T07:15:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:01.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:01.250249+0000) 2022-09-08T07:15:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:01 smithi055 ceph-mon[97033]: pgmap v18702: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:15:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:01 smithi059 ceph-mon[97450]: pgmap v18702: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:15:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:02.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:02.324050+0000) 2022-09-08T07:15:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:02.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:02.250415+0000) 2022-09-08T07:15:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:03.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:03.324255+0000) 2022-09-08T07:15:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:03.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:03.250586+0000) 2022-09-08T07:15:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:03 smithi055 ceph-mon[97033]: pgmap v18703: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:03 smithi059 ceph-mon[97450]: pgmap v18703: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:04.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:04.250827+0000) 2022-09-08T07:15:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:04.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:04.324491+0000) 2022-09-08T07:15:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:05.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:05.251099+0000) 2022-09-08T07:15:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:05.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:05.324675+0000) 2022-09-08T07:15:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:05 smithi055 ceph-mon[97033]: pgmap v18704: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:05 smithi059 ceph-mon[97450]: pgmap v18704: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:06.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:06.324793+0000) 2022-09-08T07:15:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:06.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:06.251263+0000) 2022-09-08T07:15:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:15:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:15:07.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:15:07.409+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:15:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:07.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:07.325017+0000) 2022-09-08T07:15:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:07.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:07.251416+0000) 2022-09-08T07:15:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:07 smithi055 ceph-mon[97033]: pgmap v18705: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:15:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:15:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:07 smithi059 ceph-mon[97450]: pgmap v18705: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:15:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:15:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:08.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:08.325230+0000) 2022-09-08T07:15:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:08.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:08.251562+0000) 2022-09-08T07:15:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:09.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:09.325472+0000) 2022-09-08T07:15:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:09.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:09.251719+0000) 2022-09-08T07:15:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:09 smithi055 ceph-mon[97033]: pgmap v18706: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:15:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:09 smithi059 ceph-mon[97450]: pgmap v18706: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:15:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:10.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:10.325624+0000) 2022-09-08T07:15:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:10.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:10.251930+0000) 2022-09-08T07:15:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:10 smithi055 ceph-mon[97033]: osdmap e1247: 12 total, 11 up, 11 in 2022-09-08T07:15:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:10 smithi059 ceph-mon[97450]: osdmap e1247: 12 total, 11 up, 11 in 2022-09-08T07:15:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:11.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:11.325905+0000) 2022-09-08T07:15:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:11.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:11.252108+0000) 2022-09-08T07:15:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:11 smithi055 ceph-mon[97033]: pgmap v18708: 129 pgs: 4 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:15:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:11 smithi059 ceph-mon[97450]: pgmap v18708: 129 pgs: 4 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:15:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:12.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:12.326070+0000) 2022-09-08T07:15:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:12.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:12.252264+0000) 2022-09-08T07:15:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:13.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:13.326274+0000) 2022-09-08T07:15:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:13.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:13.252400+0000) 2022-09-08T07:15:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:13 smithi055 ceph-mon[97033]: pgmap v18709: 129 pgs: 4 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:15:13.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:15:13.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:15:13.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:15:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:13 smithi059 ceph-mon[97450]: pgmap v18709: 129 pgs: 4 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:15:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:15:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:15:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:15:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:14.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:14.252561+0000) 2022-09-08T07:15:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:14.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:14.326443+0000) 2022-09-08T07:15:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:14 smithi055 ceph-mon[97033]: pgmap v18710: 129 pgs: 4 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 60 KiB/s wr, 27 op/s 2022-09-08T07:15:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:14 smithi059 ceph-mon[97450]: pgmap v18710: 129 pgs: 4 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 60 KiB/s wr, 27 op/s 2022-09-08T07:15:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:15.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:15.326686+0000) 2022-09-08T07:15:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:15.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:15.252742+0000) 2022-09-08T07:15:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:16.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:16.326919+0000) 2022-09-08T07:15:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:16.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:16.252964+0000) 2022-09-08T07:15:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:16 smithi055 ceph-mon[97033]: pgmap v18711: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-08T07:15:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:16 smithi059 ceph-mon[97450]: pgmap v18711: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-08T07:15:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:17.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:17.327126+0000) 2022-09-08T07:15:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:17.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:17.253179+0000) 2022-09-08T07:15:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:18.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:18.327364+0000) 2022-09-08T07:15:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:18.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:18.253397+0000) 2022-09-08T07:15:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:18 smithi059 ceph-mon[97450]: pgmap v18712: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-08T07:15:18.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:18 smithi059 ceph-mon[97450]: osdmap e1248: 12 total, 11 up, 11 in 2022-09-08T07:15:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:18 smithi055 ceph-mon[97033]: pgmap v18712: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2022-09-08T07:15:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:18 smithi055 ceph-mon[97033]: osdmap e1248: 12 total, 11 up, 11 in 2022-09-08T07:15:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:19.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:19.327531+0000) 2022-09-08T07:15:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:19.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:19.253556+0000) 2022-09-08T07:15:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:20.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:20.327734+0000) 2022-09-08T07:15:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:20.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:20.253771+0000) 2022-09-08T07:15:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:20 smithi059 ceph-mon[97450]: pgmap v18714: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 115 B/s wr, 3 op/s 2022-09-08T07:15:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:20 smithi055 ceph-mon[97033]: pgmap v18714: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 115 B/s wr, 3 op/s 2022-09-08T07:15:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:21.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:21.327890+0000) 2022-09-08T07:15:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:21.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:21.254004+0000) 2022-09-08T07:15:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:22.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:22.328068+0000) 2022-09-08T07:15:22.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:15:22.416+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:15:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:22.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:22.254165+0000) 2022-09-08T07:15:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:22 smithi059 ceph-mon[97450]: pgmap v18715: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 115 B/s wr, 7 op/s 2022-09-08T07:15:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:22 smithi055 ceph-mon[97033]: pgmap v18715: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 115 B/s wr, 7 op/s 2022-09-08T07:15:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:23.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:23.328324+0000) 2022-09-08T07:15:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:23.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:23.254387+0000) 2022-09-08T07:15:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:24.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:24.328534+0000) 2022-09-08T07:15:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:24.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:24.254593+0000) 2022-09-08T07:15:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:24 smithi059 ceph-mon[97450]: pgmap v18716: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:15:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:24 smithi055 ceph-mon[97033]: pgmap v18716: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:15:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:25.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:25.328774+0000) 2022-09-08T07:15:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:25.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:25.254792+0000) 2022-09-08T07:15:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:26.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:26.328890+0000) 2022-09-08T07:15:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:26.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:26.254928+0000) 2022-09-08T07:15:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:26 smithi059 ceph-mon[97450]: pgmap v18717: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:15:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:26 smithi055 ceph-mon[97033]: pgmap v18717: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:15:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:27.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:27.329028+0000) 2022-09-08T07:15:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:27.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:27.255144+0000) 2022-09-08T07:15:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:28.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:28.329166+0000) 2022-09-08T07:15:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:28.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:28.255392+0000) 2022-09-08T07:15:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:28 smithi059 ceph-mon[97450]: pgmap v18718: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:15:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:28 smithi055 ceph-mon[97033]: pgmap v18718: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:15:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:29.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:29.329384+0000) 2022-09-08T07:15:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:29.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:29.255596+0000) 2022-09-08T07:15:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:30.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:30.329611+0000) 2022-09-08T07:15:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:30.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:30.255766+0000) 2022-09-08T07:15:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:30 smithi059 ceph-mon[97450]: pgmap v18719: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:15:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:30 smithi055 ceph-mon[97033]: pgmap v18719: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:15:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:31.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:31.329872+0000) 2022-09-08T07:15:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:31.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:31.255982+0000) 2022-09-08T07:15:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:32.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:32.330009+0000) 2022-09-08T07:15:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:32.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:32.256219+0000) 2022-09-08T07:15:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:32 smithi059 ceph-mon[97450]: pgmap v18720: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:15:32.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:32 smithi055 ceph-mon[97033]: pgmap v18720: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:15:33.332 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:33.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:33.256425+0000) 2022-09-08T07:15:33.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:33.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:33.330219+0000) 2022-09-08T07:15:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:34.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:34.330456+0000) 2022-09-08T07:15:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:34.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:34.256601+0000) 2022-09-08T07:15:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:34 smithi059 ceph-mon[97450]: pgmap v18721: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:34.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:34 smithi055 ceph-mon[97033]: pgmap v18721: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:35.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:35.330679+0000) 2022-09-08T07:15:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:35.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:35.256697+0000) 2022-09-08T07:15:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:36.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:36.330909+0000) 2022-09-08T07:15:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:36.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:36.256905+0000) 2022-09-08T07:15:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:36 smithi059 ceph-mon[97450]: pgmap v18722: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:15:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:36 smithi055 ceph-mon[97033]: pgmap v18722: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:15:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:37.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:37.331088+0000) 2022-09-08T07:15:37.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:37.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:37.257097+0000) 2022-09-08T07:15:37.657 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:15:37.429+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:15:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:38.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:38.331307+0000) 2022-09-08T07:15:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:38.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:38.257289+0000) 2022-09-08T07:15:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:38 smithi059 ceph-mon[97450]: pgmap v18723: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:38 smithi055 ceph-mon[97033]: pgmap v18723: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:39.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:39.331537+0000) 2022-09-08T07:15:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:39.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:39.257459+0000) 2022-09-08T07:15:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:40.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:40.331791+0000) 2022-09-08T07:15:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:40.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:40.257699+0000) 2022-09-08T07:15:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:40 smithi059 ceph-mon[97450]: pgmap v18724: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:40 smithi055 ceph-mon[97033]: pgmap v18724: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:41.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:41.332001+0000) 2022-09-08T07:15:41.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:41.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:41.257983+0000) 2022-09-08T07:15:42.662 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:42.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:42.332213+0000) 2022-09-08T07:15:42.663 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:42.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:42.258189+0000) 2022-09-08T07:15:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:42 smithi059 ceph-mon[97450]: pgmap v18725: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:15:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:42 smithi055 ceph-mon[97033]: pgmap v18725: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:15:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:43.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:43.332442+0000) 2022-09-08T07:15:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:43.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:43.258393+0000) 2022-09-08T07:15:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:44.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:44.332657+0000) 2022-09-08T07:15:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:44.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:44.258579+0000) 2022-09-08T07:15:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:44 smithi055 ceph-mon[97033]: pgmap v18726: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:15:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:44 smithi059 ceph-mon[97450]: pgmap v18726: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:15:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:45.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:45.332881+0000) 2022-09-08T07:15:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:45.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:45.258759+0000) 2022-09-08T07:15:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:46.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:46.333110+0000) 2022-09-08T07:15:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:46.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:46.258969+0000) 2022-09-08T07:15:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:46 smithi055 ceph-mon[97033]: pgmap v18727: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:15:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:46 smithi059 ceph-mon[97450]: pgmap v18727: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:15:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:47.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:47.333285+0000) 2022-09-08T07:15:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:47.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:47.259114+0000) 2022-09-08T07:15:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:48.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:48.333504+0000) 2022-09-08T07:15:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:48.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:48.259258+0000) 2022-09-08T07:15:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:48 smithi055 ceph-mon[97033]: pgmap v18728: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:15:49.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:48 smithi059 ceph-mon[97450]: pgmap v18728: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:15:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:49.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:49.333755+0000) 2022-09-08T07:15:49.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:49.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:49.259471+0000) 2022-09-08T07:15:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:50.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:50.333965+0000) 2022-09-08T07:15:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:50.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:50.259749+0000) 2022-09-08T07:15:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:50 smithi055 ceph-mon[97033]: pgmap v18729: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:15:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:50 smithi059 ceph-mon[97450]: pgmap v18729: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:15:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:51.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:51.334179+0000) 2022-09-08T07:15:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:51.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:51.259925+0000) 2022-09-08T07:15:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:52.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:52.334383+0000) 2022-09-08T07:15:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:52.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:52.260069+0000) 2022-09-08T07:15:52.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:15:52.436+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:15:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:52 smithi055 ceph-mon[97033]: pgmap v18730: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:15:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:52 smithi059 ceph-mon[97450]: pgmap v18730: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:15:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:53.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:53.334573+0000) 2022-09-08T07:15:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:53.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:53.260213+0000) 2022-09-08T07:15:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:54.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:54.334804+0000) 2022-09-08T07:15:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:54.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:54.260374+0000) 2022-09-08T07:15:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:54 smithi055 ceph-mon[97033]: pgmap v18731: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:54 smithi059 ceph-mon[97450]: pgmap v18731: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:55.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:55.334971+0000) 2022-09-08T07:15:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:55.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:55.260530+0000) 2022-09-08T07:15:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:56.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:56.335184+0000) 2022-09-08T07:15:56.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:56.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:56.260718+0000) 2022-09-08T07:15:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:56 smithi055 ceph-mon[97033]: pgmap v18732: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:15:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:56 smithi059 ceph-mon[97450]: pgmap v18732: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:15:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:57.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:57.335421+0000) 2022-09-08T07:15:57.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:57.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:57.260939+0000) 2022-09-08T07:15:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:58.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:58.335720+0000) 2022-09-08T07:15:58.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:58.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:58.261144+0000) 2022-09-08T07:15:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:15:58 smithi055 ceph-mon[97033]: pgmap v18733: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:15:58 smithi059 ceph-mon[97450]: pgmap v18733: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:15:59.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:15:59.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:59.335912+0000) 2022-09-08T07:15:59.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:15:59.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:15:59.261352+0000) 2022-09-08T07:16:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:00.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:00.336138+0000) 2022-09-08T07:16:00.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:00.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:00.261568+0000) 2022-09-08T07:16:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:00 smithi055 ceph-mon[97033]: pgmap v18734: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:00 smithi059 ceph-mon[97450]: pgmap v18734: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:01.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:01.336362+0000) 2022-09-08T07:16:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:01.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:01.261745+0000) 2022-09-08T07:16:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:02.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:02.336610+0000) 2022-09-08T07:16:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:02.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:02.261962+0000) 2022-09-08T07:16:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:02 smithi055 ceph-mon[97033]: pgmap v18735: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:16:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:02 smithi059 ceph-mon[97450]: pgmap v18735: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:16:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:03.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:03.336856+0000) 2022-09-08T07:16:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:03.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:03.262189+0000) 2022-09-08T07:16:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:04.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:04.262423+0000) 2022-09-08T07:16:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:04.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:04.337101+0000) 2022-09-08T07:16:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:04 smithi055 ceph-mon[97033]: pgmap v18736: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:04 smithi059 ceph-mon[97450]: pgmap v18736: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:05.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:05.262654+0000) 2022-09-08T07:16:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:05.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:05.337334+0000) 2022-09-08T07:16:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:06.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:06.337556+0000) 2022-09-08T07:16:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:06.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:06.262885+0000) 2022-09-08T07:16:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:06 smithi055 ceph-mon[97033]: pgmap v18737: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:16:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:16:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:06 smithi059 ceph-mon[97450]: pgmap v18737: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:16:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:16:07.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:16:07.445+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:16:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:07.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:07.337815+0000) 2022-09-08T07:16:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:07.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:07.263020+0000) 2022-09-08T07:16:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:16:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:16:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:08.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:08.338065+0000) 2022-09-08T07:16:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:08.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:08.263155+0000) 2022-09-08T07:16:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:08 smithi055 ceph-mon[97033]: pgmap v18738: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:16:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:08 smithi059 ceph-mon[97450]: pgmap v18738: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:16:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:09.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:09.338281+0000) 2022-09-08T07:16:09.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:09.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:09.263354+0000) 2022-09-08T07:16:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:10.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:10.338586+0000) 2022-09-08T07:16:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:10.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:10.263609+0000) 2022-09-08T07:16:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:10 smithi055 ceph-mon[97033]: pgmap v18739: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:16:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:10 smithi055 ceph-mon[97033]: osdmap e1249: 12 total, 11 up, 11 in 2022-09-08T07:16:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:10 smithi059 ceph-mon[97450]: pgmap v18739: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:16:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:10 smithi059 ceph-mon[97450]: osdmap e1249: 12 total, 11 up, 11 in 2022-09-08T07:16:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:11.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:11.338863+0000) 2022-09-08T07:16:11.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:11.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:11.263797+0000) 2022-09-08T07:16:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:12.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:12.339144+0000) 2022-09-08T07:16:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:12.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:12.263977+0000) 2022-09-08T07:16:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:12 smithi055 ceph-mon[97033]: pgmap v18741: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:16:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:12 smithi059 ceph-mon[97450]: pgmap v18741: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:16:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:13.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:13.339384+0000) 2022-09-08T07:16:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:13.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:13.264204+0000) 2022-09-08T07:16:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:16:13.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:16:13.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:16:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:16:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:16:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:16:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:14.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:14.339573+0000) 2022-09-08T07:16:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:14.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:14.264400+0000) 2022-09-08T07:16:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:14 smithi055 ceph-mon[97033]: pgmap v18742: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:16:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:16:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:16:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:14 smithi059 ceph-mon[97450]: pgmap v18742: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:16:15.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:16:15.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:16:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:15.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:15.339798+0000) 2022-09-08T07:16:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:15.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:15.264556+0000) 2022-09-08T07:16:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:16.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:16.340007+0000) 2022-09-08T07:16:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:16.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:16.264720+0000) 2022-09-08T07:16:16.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:16 smithi055 ceph-mon[97033]: pgmap v18743: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:16:16.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:16:16.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:16:16.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:16:16.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:16:16.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:16.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:16.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:16.734 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:16.976 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:16 smithi059 ceph-mon[97450]: pgmap v18743: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:16:16.977 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:16:16.977 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:16:16.977 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:16:16.978 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:16:16.978 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:16.979 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:16.979 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:16.979 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:17.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:17.340202+0000) 2022-09-08T07:16:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:17.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:17.264918+0000) 2022-09-08T07:16:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:17 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 139.1M 2022-09-08T07:16:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:17 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 145944383: error parsing value: Value '145944383' is below minimum 939524096 2022-09-08T07:16:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:17.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:17.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:17.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:17.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:16:17.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:17 smithi055 ceph-mon[97033]: osdmap e1250: 12 total, 11 up, 11 in 2022-09-08T07:16:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:17 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 139.1M 2022-09-08T07:16:18.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:17 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 145944383: error parsing value: Value '145944383' is below minimum 939524096 2022-09-08T07:16:18.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:18.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:18.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:18.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:16:18.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:16:18.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:17 smithi059 ceph-mon[97450]: osdmap e1250: 12 total, 11 up, 11 in 2022-09-08T07:16:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:18.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:18.340447+0000) 2022-09-08T07:16:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:18.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:18.265108+0000) 2022-09-08T07:16:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:18 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T07:16:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:18 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T07:16:18.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:18 smithi055 ceph-mon[97033]: pgmap v18744: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T07:16:18.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:18 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T07:16:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:18 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T07:16:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:18 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T07:16:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:18 smithi059 ceph-mon[97450]: pgmap v18744: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T07:16:19.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:18 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T07:16:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:19.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:19.340688+0000) 2022-09-08T07:16:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:19.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:19.265290+0000) 2022-09-08T07:16:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:20.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:20.340809+0000) 2022-09-08T07:16:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:20.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:20.265516+0000) 2022-09-08T07:16:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:20 smithi055 ceph-mon[97033]: pgmap v18746: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-08T07:16:21.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:20 smithi059 ceph-mon[97450]: pgmap v18746: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-08T07:16:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:21.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:21.341035+0000) 2022-09-08T07:16:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:21.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:21.265758+0000) 2022-09-08T07:16:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:22.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:22.341277+0000) 2022-09-08T07:16:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:22.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:22.265920+0000) 2022-09-08T07:16:22.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:16:22.450+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:16:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:22 smithi055 ceph-mon[97033]: pgmap v18747: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 103 B/s wr, 6 op/s 2022-09-08T07:16:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:22 smithi059 ceph-mon[97450]: pgmap v18747: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 103 B/s wr, 6 op/s 2022-09-08T07:16:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:23.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:23.266079+0000) 2022-09-08T07:16:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:23.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:23.341529+0000) 2022-09-08T07:16:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:24.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:24.341747+0000) 2022-09-08T07:16:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:24.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:24.266297+0000) 2022-09-08T07:16:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:24 smithi055 ceph-mon[97033]: pgmap v18748: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 103 B/s wr, 6 op/s 2022-09-08T07:16:25.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:24 smithi059 ceph-mon[97450]: pgmap v18748: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 103 B/s wr, 6 op/s 2022-09-08T07:16:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:25.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:25.341974+0000) 2022-09-08T07:16:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:25.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:25.266465+0000) 2022-09-08T07:16:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:26.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:26.342194+0000) 2022-09-08T07:16:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:26.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:26.266653+0000) 2022-09-08T07:16:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:26 smithi055 ceph-mon[97033]: pgmap v18749: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T07:16:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:26 smithi059 ceph-mon[97450]: pgmap v18749: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T07:16:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:27.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:27.342433+0000) 2022-09-08T07:16:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:27.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:27.266868+0000) 2022-09-08T07:16:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:28.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:28.267027+0000) 2022-09-08T07:16:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:28.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:28.342687+0000) 2022-09-08T07:16:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:28 smithi055 ceph-mon[97033]: pgmap v18750: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T07:16:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:28 smithi059 ceph-mon[97450]: pgmap v18750: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T07:16:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:29.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:29.267238+0000) 2022-09-08T07:16:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:29.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:29.342838+0000) 2022-09-08T07:16:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:30.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:30.343092+0000) 2022-09-08T07:16:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:30.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:30.267385+0000) 2022-09-08T07:16:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:30 smithi055 ceph-mon[97033]: pgmap v18751: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:16:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:30 smithi059 ceph-mon[97450]: pgmap v18751: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:16:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:31.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:31.343310+0000) 2022-09-08T07:16:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:31.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:31.267564+0000) 2022-09-08T07:16:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:32.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:32.343578+0000) 2022-09-08T07:16:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:32.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:32.267793+0000) 2022-09-08T07:16:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:32 smithi055 ceph-mon[97033]: pgmap v18752: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T07:16:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:32 smithi059 ceph-mon[97450]: pgmap v18752: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T07:16:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:33.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:33.343831+0000) 2022-09-08T07:16:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:33.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:33.267977+0000) 2022-09-08T07:16:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:34.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:34.343991+0000) 2022-09-08T07:16:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:34.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:34.268218+0000) 2022-09-08T07:16:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:34 smithi055 ceph-mon[97033]: pgmap v18753: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:34 smithi059 ceph-mon[97450]: pgmap v18753: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:35.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:35.344254+0000) 2022-09-08T07:16:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:35.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:35.268462+0000) 2022-09-08T07:16:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:36.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:36.344481+0000) 2022-09-08T07:16:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:36.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:36.268761+0000) 2022-09-08T07:16:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:36 smithi055 ceph-mon[97033]: pgmap v18754: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:16:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:36 smithi059 ceph-mon[97450]: pgmap v18754: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:16:37.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:16:37.459+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:16:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:37.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:37.344752+0000) 2022-09-08T07:16:37.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:37.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:37.269013+0000) 2022-09-08T07:16:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:38.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:38.344879+0000) 2022-09-08T07:16:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:38.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:38.269210+0000) 2022-09-08T07:16:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:38 smithi055 ceph-mon[97033]: pgmap v18755: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:38 smithi059 ceph-mon[97450]: pgmap v18755: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:39.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:39.269434+0000) 2022-09-08T07:16:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:39.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:39.345086+0000) 2022-09-08T07:16:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:40.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:40.345316+0000) 2022-09-08T07:16:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:40.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:40.269594+0000) 2022-09-08T07:16:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:40 smithi055 ceph-mon[97033]: pgmap v18756: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:40 smithi059 ceph-mon[97450]: pgmap v18756: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:41.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:41.345530+0000) 2022-09-08T07:16:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:41.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:41.269757+0000) 2022-09-08T07:16:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:42.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:42.345758+0000) 2022-09-08T07:16:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:42.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:42.269938+0000) 2022-09-08T07:16:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:42 smithi055 ceph-mon[97033]: pgmap v18757: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:16:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:42 smithi059 ceph-mon[97450]: pgmap v18757: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:16:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:43.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:43.270137+0000) 2022-09-08T07:16:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:43.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:43.346016+0000) 2022-09-08T07:16:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:44.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:44.346247+0000) 2022-09-08T07:16:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:44.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:44.270337+0000) 2022-09-08T07:16:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:44 smithi055 ceph-mon[97033]: pgmap v18758: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:44 smithi059 ceph-mon[97450]: pgmap v18758: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:45.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:45.346488+0000) 2022-09-08T07:16:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:45.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:45.270561+0000) 2022-09-08T07:16:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:46.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:46.346751+0000) 2022-09-08T07:16:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:46.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:46.270755+0000) 2022-09-08T07:16:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:46 smithi055 ceph-mon[97033]: pgmap v18759: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:16:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:46 smithi059 ceph-mon[97450]: pgmap v18759: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:16:47.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:47.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:47.347017+0000) 2022-09-08T07:16:47.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:47.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:47.270917+0000) 2022-09-08T07:16:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:48.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:48.347287+0000) 2022-09-08T07:16:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:48.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:48.271135+0000) 2022-09-08T07:16:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:48 smithi055 ceph-mon[97033]: pgmap v18760: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:48 smithi059 ceph-mon[97450]: pgmap v18760: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:49.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:49.347513+0000) 2022-09-08T07:16:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:49.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:49.271357+0000) 2022-09-08T07:16:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:50.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:50.347699+0000) 2022-09-08T07:16:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:50.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:50.271541+0000) 2022-09-08T07:16:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:50 smithi055 ceph-mon[97033]: pgmap v18761: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:50 smithi059 ceph-mon[97450]: pgmap v18761: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:51.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:51.347833+0000) 2022-09-08T07:16:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:51.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:51.271744+0000) 2022-09-08T07:16:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:52.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:52.348057+0000) 2022-09-08T07:16:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:52.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:52.271958+0000) 2022-09-08T07:16:52.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:16:52.472+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:16:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:52 smithi055 ceph-mon[97033]: pgmap v18762: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:16:53.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:52 smithi059 ceph-mon[97450]: pgmap v18762: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:16:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:53.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:53.348289+0000) 2022-09-08T07:16:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:53.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:53.272182+0000) 2022-09-08T07:16:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:54.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:54.348443+0000) 2022-09-08T07:16:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:54.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:54.272389+0000) 2022-09-08T07:16:54.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:54 smithi055 ceph-mon[97033]: pgmap v18763: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:54 smithi059 ceph-mon[97450]: pgmap v18763: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:55.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:55.272604+0000) 2022-09-08T07:16:55.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:55.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:55.348708+0000) 2022-09-08T07:16:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:56.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:56.272851+0000) 2022-09-08T07:16:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:56.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:56.348884+0000) 2022-09-08T07:16:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:56 smithi055 ceph-mon[97033]: pgmap v18764: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:16:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:56 smithi059 ceph-mon[97450]: pgmap v18764: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:16:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:57.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:57.349105+0000) 2022-09-08T07:16:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:57.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:57.273080+0000) 2022-09-08T07:16:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:58.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:58.349312+0000) 2022-09-08T07:16:58.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:58.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:58.273223+0000) 2022-09-08T07:16:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:16:58 smithi055 ceph-mon[97033]: pgmap v18765: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:16:58 smithi059 ceph-mon[97450]: pgmap v18765: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:16:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:16:59.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:59.273424+0000) 2022-09-08T07:16:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:16:59.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:16:59.349488+0000) 2022-09-08T07:17:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:00.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:00.349737+0000) 2022-09-08T07:17:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:00.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:00.273623+0000) 2022-09-08T07:17:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:00 smithi055 ceph-mon[97033]: pgmap v18766: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:00 smithi059 ceph-mon[97450]: pgmap v18766: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:01.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:01.349949+0000) 2022-09-08T07:17:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:01.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:01.273829+0000) 2022-09-08T07:17:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:02.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:02.350175+0000) 2022-09-08T07:17:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:02.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:02.273994+0000) 2022-09-08T07:17:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:02 smithi055 ceph-mon[97033]: pgmap v18767: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:17:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:02 smithi059 ceph-mon[97450]: pgmap v18767: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:17:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:03.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:03.350398+0000) 2022-09-08T07:17:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:03.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:03.274196+0000) 2022-09-08T07:17:04.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:04.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:04.274398+0000) 2022-09-08T07:17:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:04.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:04.350611+0000) 2022-09-08T07:17:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:04 smithi055 ceph-mon[97033]: pgmap v18768: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:04 smithi059 ceph-mon[97450]: pgmap v18768: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:05.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:05.274562+0000) 2022-09-08T07:17:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:05.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:05.350838+0000) 2022-09-08T07:17:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:06.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:06.274721+0000) 2022-09-08T07:17:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:06.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:06.351030+0000) 2022-09-08T07:17:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:06 smithi055 ceph-mon[97033]: pgmap v18769: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-08T07:17:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:17:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:06 smithi059 ceph-mon[97450]: pgmap v18769: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-08T07:17:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:17:07.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:17:07.486+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:17:07.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:07.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:07.351210+0000) 2022-09-08T07:17:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:07.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:07.274950+0000) 2022-09-08T07:17:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:17:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:17:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:08.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:08.351435+0000) 2022-09-08T07:17:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:08.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:08.275118+0000) 2022-09-08T07:17:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:08 smithi055 ceph-mon[97033]: pgmap v18770: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:17:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:08 smithi059 ceph-mon[97450]: pgmap v18770: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:17:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:09.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:09.351660+0000) 2022-09-08T07:17:09.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:09.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:09.275290+0000) 2022-09-08T07:17:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:10.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:10.351789+0000) 2022-09-08T07:17:10.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:10.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:10.275461+0000) 2022-09-08T07:17:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:10 smithi059 ceph-mon[97450]: pgmap v18771: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:17:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:10 smithi059 ceph-mon[97450]: osdmap e1251: 12 total, 11 up, 11 in 2022-09-08T07:17:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:10 smithi055 ceph-mon[97033]: pgmap v18771: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:17:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:10 smithi055 ceph-mon[97033]: osdmap e1251: 12 total, 11 up, 11 in 2022-09-08T07:17:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:11.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:11.351986+0000) 2022-09-08T07:17:11.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:11.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:11.275767+0000) 2022-09-08T07:17:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:12.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:12.352267+0000) 2022-09-08T07:17:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:12.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:12.276014+0000) 2022-09-08T07:17:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:12 smithi059 ceph-mon[97450]: pgmap v18773: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:17:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:12 smithi055 ceph-mon[97033]: pgmap v18773: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:17:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:13.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:13.352529+0000) 2022-09-08T07:17:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:13.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:13.276201+0000) 2022-09-08T07:17:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:14.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:14.352782+0000) 2022-09-08T07:17:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:14.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:14.276363+0000) 2022-09-08T07:17:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:14 smithi059 ceph-mon[97450]: pgmap v18774: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:17:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:14 smithi055 ceph-mon[97033]: pgmap v18774: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:17:15.355 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:15.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:15.276558+0000) 2022-09-08T07:17:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:15.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:15.352976+0000) 2022-09-08T07:17:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:16.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:16.353216+0000) 2022-09-08T07:17:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:16.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:16.276727+0000) 2022-09-08T07:17:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:16 smithi059 ceph-mon[97450]: pgmap v18775: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2022-09-08T07:17:17.043 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:16 smithi055 ceph-mon[97033]: pgmap v18775: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2022-09-08T07:17:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:17.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:17.353440+0000) 2022-09-08T07:17:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:17.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:17.276861+0000) 2022-09-08T07:17:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:17:17.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:17:17.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:17:17.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:17:17.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:17:18.010 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:17:18.010 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:17:18.011 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:17:18.011 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:17:18.011 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:17:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:18.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:18.353708+0000) 2022-09-08T07:17:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:18.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:18.277030+0000) 2022-09-08T07:17:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:18 smithi059 ceph-mon[97450]: pgmap v18776: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:17:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:17:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:18 smithi055 ceph-mon[97033]: pgmap v18776: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:17:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:17:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:19.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:19.353917+0000) 2022-09-08T07:17:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:19.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:19.277207+0000) 2022-09-08T07:17:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:20.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:20.354075+0000) 2022-09-08T07:17:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:20.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:20.277486+0000) 2022-09-08T07:17:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:20 smithi059 ceph-mon[97450]: pgmap v18777: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:17:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:20 smithi055 ceph-mon[97033]: pgmap v18777: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:17:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:21.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:21.354323+0000) 2022-09-08T07:17:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:21.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:21.277707+0000) 2022-09-08T07:17:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:22.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:22.354584+0000) 2022-09-08T07:17:22.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:17:22.495+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:17:22.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:22.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:22.277833+0000) 2022-09-08T07:17:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:22 smithi059 ceph-mon[97450]: pgmap v18778: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 183 B/s wr, 8 op/s 2022-09-08T07:17:23.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:22 smithi059 ceph-mon[97450]: osdmap e1252: 12 total, 11 up, 11 in 2022-09-08T07:17:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:22 smithi055 ceph-mon[97033]: pgmap v18778: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 183 B/s wr, 8 op/s 2022-09-08T07:17:23.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:22 smithi055 ceph-mon[97033]: osdmap e1252: 12 total, 11 up, 11 in 2022-09-08T07:17:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:23.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:23.354818+0000) 2022-09-08T07:17:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:23.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:23.277964+0000) 2022-09-08T07:17:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:24.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:24.355058+0000) 2022-09-08T07:17:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:24.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:24.278130+0000) 2022-09-08T07:17:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:24 smithi059 ceph-mon[97450]: pgmap v18780: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:17:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:24 smithi055 ceph-mon[97033]: pgmap v18780: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:17:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:25.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:25.355273+0000) 2022-09-08T07:17:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:25.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:25.278379+0000) 2022-09-08T07:17:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:26.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:26.355486+0000) 2022-09-08T07:17:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:26.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:26.278547+0000) 2022-09-08T07:17:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:26 smithi059 ceph-mon[97450]: pgmap v18781: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:17:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:26 smithi055 ceph-mon[97033]: pgmap v18781: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:17:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:27.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:27.355697+0000) 2022-09-08T07:17:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:27.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:27.278754+0000) 2022-09-08T07:17:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:28.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:28.355913+0000) 2022-09-08T07:17:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:28.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:28.278969+0000) 2022-09-08T07:17:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:28 smithi059 ceph-mon[97450]: pgmap v18782: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:17:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:28 smithi055 ceph-mon[97033]: pgmap v18782: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:17:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:29.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:29.356153+0000) 2022-09-08T07:17:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:29.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:29.279158+0000) 2022-09-08T07:17:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:30.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:30.356386+0000) 2022-09-08T07:17:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:30.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:30.279386+0000) 2022-09-08T07:17:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:30 smithi059 ceph-mon[97450]: pgmap v18783: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:17:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:30 smithi055 ceph-mon[97033]: pgmap v18783: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:17:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:31.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:31.356567+0000) 2022-09-08T07:17:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:31.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:31.279549+0000) 2022-09-08T07:17:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:32.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:32.356760+0000) 2022-09-08T07:17:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:32.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:32.279692+0000) 2022-09-08T07:17:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:32 smithi059 ceph-mon[97450]: pgmap v18784: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:17:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:32 smithi055 ceph-mon[97033]: pgmap v18784: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:17:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:33.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:33.356960+0000) 2022-09-08T07:17:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:33.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:33.279822+0000) 2022-09-08T07:17:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:34.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:34.357170+0000) 2022-09-08T07:17:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:34.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:34.280031+0000) 2022-09-08T07:17:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:34 smithi059 ceph-mon[97450]: pgmap v18785: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:17:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:34 smithi055 ceph-mon[97033]: pgmap v18785: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:17:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:35.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:35.357408+0000) 2022-09-08T07:17:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:35.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:35.280240+0000) 2022-09-08T07:17:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:35 smithi055 ceph-mon[97033]: pgmap v18786: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:17:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:35 smithi059 ceph-mon[97450]: pgmap v18786: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:17:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:36.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:36.357632+0000) 2022-09-08T07:17:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:36.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:36.280511+0000) 2022-09-08T07:17:37.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:17:37.496+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:17:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:37.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:37.357887+0000) 2022-09-08T07:17:37.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:37.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:37.280763+0000) 2022-09-08T07:17:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:38.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:38.358106+0000) 2022-09-08T07:17:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:38.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:38.280961+0000) 2022-09-08T07:17:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:38 smithi055 ceph-mon[97033]: pgmap v18787: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:38 smithi059 ceph-mon[97450]: pgmap v18787: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:39.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:39.358342+0000) 2022-09-08T07:17:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:39.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:39.281207+0000) 2022-09-08T07:17:40.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:40.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:40.358564+0000) 2022-09-08T07:17:40.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:40.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:40.281444+0000) 2022-09-08T07:17:40.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:40 smithi055 ceph-mon[97033]: pgmap v18788: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:40 smithi059 ceph-mon[97450]: pgmap v18788: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:41.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:41.358731+0000) 2022-09-08T07:17:41.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:41.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:41.281680+0000) 2022-09-08T07:17:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:42.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:42.358924+0000) 2022-09-08T07:17:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:42.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:42.281820+0000) 2022-09-08T07:17:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:42 smithi055 ceph-mon[97033]: pgmap v18789: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:17:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:42 smithi059 ceph-mon[97450]: pgmap v18789: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:17:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:43.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:43.359078+0000) 2022-09-08T07:17:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:43.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:43.282015+0000) 2022-09-08T07:17:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:44.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:44.359319+0000) 2022-09-08T07:17:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:44.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:44.282169+0000) 2022-09-08T07:17:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:44 smithi055 ceph-mon[97033]: pgmap v18790: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:44 smithi059 ceph-mon[97450]: pgmap v18790: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:45.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:45.359572+0000) 2022-09-08T07:17:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:45.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:45.282307+0000) 2022-09-08T07:17:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:46.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:46.359804+0000) 2022-09-08T07:17:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:46.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:46.282501+0000) 2022-09-08T07:17:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:46 smithi055 ceph-mon[97033]: pgmap v18791: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:17:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:46 smithi059 ceph-mon[97450]: pgmap v18791: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:17:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:47.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:47.360052+0000) 2022-09-08T07:17:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:47.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:47.282756+0000) 2022-09-08T07:17:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:48.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:48.360320+0000) 2022-09-08T07:17:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:48.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:48.282961+0000) 2022-09-08T07:17:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:48 smithi055 ceph-mon[97033]: pgmap v18792: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:48 smithi059 ceph-mon[97450]: pgmap v18792: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:49.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:49.360585+0000) 2022-09-08T07:17:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:49.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:49.283185+0000) 2022-09-08T07:17:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:50.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:50.360770+0000) 2022-09-08T07:17:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:50.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:50.283338+0000) 2022-09-08T07:17:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:50 smithi055 ceph-mon[97033]: pgmap v18793: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:50 smithi059 ceph-mon[97450]: pgmap v18793: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:51.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:51.360944+0000) 2022-09-08T07:17:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:51.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:51.283493+0000) 2022-09-08T07:17:52.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:17:52.511+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:17:52.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:52.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:52.361140+0000) 2022-09-08T07:17:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:52.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:52.283729+0000) 2022-09-08T07:17:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:52 smithi055 ceph-mon[97033]: pgmap v18794: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:17:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:52 smithi059 ceph-mon[97450]: pgmap v18794: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:17:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:53.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:53.361361+0000) 2022-09-08T07:17:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:53.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:53.284004+0000) 2022-09-08T07:17:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:54.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:54.361615+0000) 2022-09-08T07:17:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:54.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:54.284220+0000) 2022-09-08T07:17:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:54 smithi055 ceph-mon[97033]: pgmap v18795: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:54 smithi059 ceph-mon[97450]: pgmap v18795: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:55.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:55.361896+0000) 2022-09-08T07:17:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:55.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:55.284412+0000) 2022-09-08T07:17:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:56.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:56.362125+0000) 2022-09-08T07:17:56.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:56.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:56.284684+0000) 2022-09-08T07:17:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:56 smithi055 ceph-mon[97033]: pgmap v18796: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:17:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:56 smithi059 ceph-mon[97450]: pgmap v18796: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:17:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:57.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:57.362322+0000) 2022-09-08T07:17:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:57.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:57.284884+0000) 2022-09-08T07:17:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:58.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:58.362567+0000) 2022-09-08T07:17:58.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:58.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:58.285109+0000) 2022-09-08T07:17:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:17:58 smithi055 ceph-mon[97033]: pgmap v18797: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:17:58 smithi059 ceph-mon[97450]: pgmap v18797: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:17:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:17:59.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:59.362759+0000) 2022-09-08T07:17:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:17:59.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:17:59.285253+0000) 2022-09-08T07:18:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:00.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:00.363001+0000) 2022-09-08T07:18:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:00.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:00.285417+0000) 2022-09-08T07:18:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:00 smithi055 ceph-mon[97033]: pgmap v18798: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:00 smithi059 ceph-mon[97450]: pgmap v18798: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:01.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:01.363191+0000) 2022-09-08T07:18:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:01.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:01.285566+0000) 2022-09-08T07:18:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:02.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:02.363421+0000) 2022-09-08T07:18:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:02.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:02.285717+0000) 2022-09-08T07:18:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:02 smithi055 ceph-mon[97033]: pgmap v18799: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:18:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:02 smithi059 ceph-mon[97450]: pgmap v18799: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:18:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:03.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:03.363610+0000) 2022-09-08T07:18:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:03.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:03.285863+0000) 2022-09-08T07:18:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:04.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:04.363833+0000) 2022-09-08T07:18:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:04.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:04.286052+0000) 2022-09-08T07:18:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:04 smithi055 ceph-mon[97033]: pgmap v18800: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:04 smithi059 ceph-mon[97450]: pgmap v18800: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:05.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:05.364043+0000) 2022-09-08T07:18:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:05.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:05.286266+0000) 2022-09-08T07:18:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:06.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:06.364305+0000) 2022-09-08T07:18:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:06.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:06.286429+0000) 2022-09-08T07:18:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:06 smithi055 ceph-mon[97033]: pgmap v18801: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-08T07:18:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:18:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:06 smithi059 ceph-mon[97450]: pgmap v18801: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-08T07:18:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:18:07.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:18:07.527+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:18:07.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:07.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:07.364569+0000) 2022-09-08T07:18:07.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:07.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:07.286666+0000) 2022-09-08T07:18:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:18:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:18:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:08.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:08.364858+0000) 2022-09-08T07:18:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:08.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:08.286888+0000) 2022-09-08T07:18:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:08 smithi055 ceph-mon[97033]: pgmap v18802: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:18:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:08 smithi059 ceph-mon[97450]: pgmap v18802: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:18:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:09.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:09.365128+0000) 2022-09-08T07:18:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:09.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:09.287041+0000) 2022-09-08T07:18:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:10.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:10.365354+0000) 2022-09-08T07:18:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:10.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:10.287203+0000) 2022-09-08T07:18:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:10 smithi055 ceph-mon[97033]: pgmap v18803: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:18:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:10 smithi055 ceph-mon[97033]: osdmap e1253: 12 total, 11 up, 11 in 2022-09-08T07:18:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:10 smithi059 ceph-mon[97450]: pgmap v18803: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:18:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:10 smithi059 ceph-mon[97450]: osdmap e1253: 12 total, 11 up, 11 in 2022-09-08T07:18:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:11.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:11.365559+0000) 2022-09-08T07:18:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:11.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:11.287371+0000) 2022-09-08T07:18:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:12.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:12.365751+0000) 2022-09-08T07:18:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:12.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:12.287495+0000) 2022-09-08T07:18:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:12 smithi055 ceph-mon[97033]: pgmap v18805: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:18:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:12 smithi059 ceph-mon[97450]: pgmap v18805: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:18:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:13.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:13.365904+0000) 2022-09-08T07:18:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:13.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:13.287721+0000) 2022-09-08T07:18:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:14.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:14.366093+0000) 2022-09-08T07:18:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:14.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:14.287871+0000) 2022-09-08T07:18:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:14 smithi055 ceph-mon[97033]: pgmap v18806: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:18:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:14 smithi059 ceph-mon[97450]: pgmap v18806: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:18:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:15.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:15.366293+0000) 2022-09-08T07:18:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:15.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:15.288008+0000) 2022-09-08T07:18:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:16.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:16.288258+0000) 2022-09-08T07:18:16.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:16.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:16.366525+0000) 2022-09-08T07:18:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:16 smithi055 ceph-mon[97033]: pgmap v18807: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2022-09-08T07:18:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:16 smithi059 ceph-mon[97450]: pgmap v18807: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2022-09-08T07:18:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:17.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:17.366786+0000) 2022-09-08T07:18:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:17.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:17.288454+0000) 2022-09-08T07:18:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:18.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:18.366957+0000) 2022-09-08T07:18:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:18.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:18.288689+0000) 2022-09-08T07:18:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:18 smithi055 ceph-mon[97033]: pgmap v18808: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:18:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:18:18.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:18:18.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:18:18.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:18:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:18 smithi059 ceph-mon[97450]: pgmap v18808: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:18:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:18:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:18:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:18:19.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:18:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:19.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:19.367119+0000) 2022-09-08T07:18:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:19.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:19.288834+0000) 2022-09-08T07:18:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:20.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:20.367334+0000) 2022-09-08T07:18:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:20.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:20.289053+0000) 2022-09-08T07:18:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:20 smithi055 ceph-mon[97033]: pgmap v18809: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:18:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:20 smithi059 ceph-mon[97450]: pgmap v18809: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:18:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:21.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:21.367573+0000) 2022-09-08T07:18:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:21.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:21.289240+0000) 2022-09-08T07:18:22.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:18:22.534+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:18:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:22.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:22.367800+0000) 2022-09-08T07:18:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:22.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:22.289373+0000) 2022-09-08T07:18:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:22 smithi055 ceph-mon[97033]: pgmap v18810: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 180 B/s wr, 8 op/s 2022-09-08T07:18:22.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:22 smithi055 ceph-mon[97033]: osdmap e1254: 12 total, 11 up, 11 in 2022-09-08T07:18:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:22 smithi059 ceph-mon[97450]: pgmap v18810: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 180 B/s wr, 8 op/s 2022-09-08T07:18:23.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:22 smithi059 ceph-mon[97450]: osdmap e1254: 12 total, 11 up, 11 in 2022-09-08T07:18:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:23.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:23.367963+0000) 2022-09-08T07:18:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:23.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:23.289519+0000) 2022-09-08T07:18:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:24.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:24.368147+0000) 2022-09-08T07:18:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:24.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:24.289674+0000) 2022-09-08T07:18:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:24 smithi055 ceph-mon[97033]: pgmap v18812: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:18:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:24 smithi059 ceph-mon[97450]: pgmap v18812: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:18:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:25.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:25.368346+0000) 2022-09-08T07:18:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:25.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:25.289813+0000) 2022-09-08T07:18:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:26.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:26.368567+0000) 2022-09-08T07:18:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:26.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:26.289964+0000) 2022-09-08T07:18:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:26 smithi055 ceph-mon[97033]: pgmap v18813: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:18:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:26 smithi059 ceph-mon[97450]: pgmap v18813: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:18:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:27.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:27.368790+0000) 2022-09-08T07:18:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:27.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:27.290100+0000) 2022-09-08T07:18:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:28.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:28.290249+0000) 2022-09-08T07:18:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:28.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:28.369031+0000) 2022-09-08T07:18:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:28 smithi055 ceph-mon[97033]: pgmap v18814: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:18:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:28 smithi059 ceph-mon[97450]: pgmap v18814: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:18:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:29.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:29.369206+0000) 2022-09-08T07:18:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:29.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:29.290430+0000) 2022-09-08T07:18:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:30.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:30.369409+0000) 2022-09-08T07:18:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:30.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:30.290607+0000) 2022-09-08T07:18:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:30 smithi055 ceph-mon[97033]: pgmap v18815: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:18:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:30 smithi059 ceph-mon[97450]: pgmap v18815: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:18:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:31.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:31.369652+0000) 2022-09-08T07:18:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:31.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:31.290824+0000) 2022-09-08T07:18:32.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:32.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:32.369904+0000) 2022-09-08T07:18:32.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:32.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:32.291073+0000) 2022-09-08T07:18:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:32 smithi055 ceph-mon[97033]: pgmap v18816: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:18:33.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:32 smithi059 ceph-mon[97450]: pgmap v18816: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:18:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:33.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:33.370142+0000) 2022-09-08T07:18:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:33.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:33.291252+0000) 2022-09-08T07:18:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:34.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:34.370353+0000) 2022-09-08T07:18:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:34.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:34.291402+0000) 2022-09-08T07:18:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:34 smithi055 ceph-mon[97033]: pgmap v18817: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:18:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:34 smithi059 ceph-mon[97450]: pgmap v18817: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:18:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:35.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:35.370550+0000) 2022-09-08T07:18:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:35.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:35.291575+0000) 2022-09-08T07:18:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:36.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:36.370738+0000) 2022-09-08T07:18:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:36.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:36.291707+0000) 2022-09-08T07:18:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:36 smithi055 ceph-mon[97033]: pgmap v18818: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:18:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:36 smithi059 ceph-mon[97450]: pgmap v18818: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:18:37.551 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:37.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:37.370994+0000) 2022-09-08T07:18:37.551 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:37.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:37.291913+0000) 2022-09-08T07:18:37.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:18:37.546+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:18:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:38.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:38.371226+0000) 2022-09-08T07:18:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:38.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:38.292142+0000) 2022-09-08T07:18:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:38 smithi055 ceph-mon[97033]: pgmap v18819: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:38 smithi059 ceph-mon[97450]: pgmap v18819: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:39.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:39.371400+0000) 2022-09-08T07:18:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:39.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:39.292279+0000) 2022-09-08T07:18:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:40.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:40.371675+0000) 2022-09-08T07:18:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:40.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:40.292435+0000) 2022-09-08T07:18:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:40 smithi055 ceph-mon[97033]: pgmap v18820: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:40 smithi059 ceph-mon[97450]: pgmap v18820: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:41.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:41.292599+0000) 2022-09-08T07:18:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:41.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:41.371930+0000) 2022-09-08T07:18:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:42.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:42.292750+0000) 2022-09-08T07:18:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:42.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:42.372166+0000) 2022-09-08T07:18:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:42 smithi055 ceph-mon[97033]: pgmap v18821: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:18:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:42 smithi059 ceph-mon[97450]: pgmap v18821: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:18:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:43.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:43.292913+0000) 2022-09-08T07:18:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:43.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:43.372376+0000) 2022-09-08T07:18:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:44.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:44.372593+0000) 2022-09-08T07:18:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:44.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:44.293111+0000) 2022-09-08T07:18:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:44 smithi055 ceph-mon[97033]: pgmap v18822: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:44 smithi059 ceph-mon[97450]: pgmap v18822: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:45.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:45.293283+0000) 2022-09-08T07:18:45.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:45.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:45.372850+0000) 2022-09-08T07:18:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:46.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:46.293433+0000) 2022-09-08T07:18:46.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:46.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:46.372966+0000) 2022-09-08T07:18:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:46 smithi055 ceph-mon[97033]: pgmap v18823: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:18:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:46 smithi059 ceph-mon[97450]: pgmap v18823: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:18:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:47.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:47.373084+0000) 2022-09-08T07:18:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:47.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:47.293613+0000) 2022-09-08T07:18:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:48.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:48.373255+0000) 2022-09-08T07:18:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:48.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:48.293835+0000) 2022-09-08T07:18:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:48 smithi055 ceph-mon[97033]: pgmap v18824: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:48 smithi059 ceph-mon[97450]: pgmap v18824: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:49.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:49.373396+0000) 2022-09-08T07:18:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:49.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:49.293972+0000) 2022-09-08T07:18:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:50.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:50.373631+0000) 2022-09-08T07:18:50.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:50.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:50.294166+0000) 2022-09-08T07:18:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:50 smithi055 ceph-mon[97033]: pgmap v18825: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:50 smithi059 ceph-mon[97450]: pgmap v18825: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:51.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:51.373913+0000) 2022-09-08T07:18:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:51.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:51.294306+0000) 2022-09-08T07:18:52.556 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:52.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:52.374147+0000) 2022-09-08T07:18:52.557 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:52.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:52.294524+0000) 2022-09-08T07:18:52.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:18:52.552+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:18:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:52 smithi055 ceph-mon[97033]: pgmap v18826: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:18:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:52 smithi059 ceph-mon[97450]: pgmap v18826: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:18:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:53.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:53.374315+0000) 2022-09-08T07:18:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:53.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:53.294729+0000) 2022-09-08T07:18:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:54.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:54.374463+0000) 2022-09-08T07:18:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:54.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:54.294926+0000) 2022-09-08T07:18:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:54 smithi055 ceph-mon[97033]: pgmap v18827: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:54 smithi059 ceph-mon[97450]: pgmap v18827: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:55.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:55.374720+0000) 2022-09-08T07:18:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:55.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:55.295078+0000) 2022-09-08T07:18:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:56.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:56.374940+0000) 2022-09-08T07:18:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:56.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:56.295273+0000) 2022-09-08T07:18:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:56 smithi055 ceph-mon[97033]: pgmap v18828: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:18:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:56 smithi059 ceph-mon[97450]: pgmap v18828: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:18:57.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:57.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:57.375122+0000) 2022-09-08T07:18:57.664 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:57.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:57.295484+0000) 2022-09-08T07:18:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:58.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:58.375346+0000) 2022-09-08T07:18:58.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:58.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:58.295632+0000) 2022-09-08T07:18:58.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:18:58 smithi055 ceph-mon[97033]: pgmap v18829: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:59.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:18:58 smithi059 ceph-mon[97450]: pgmap v18829: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:18:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:18:59.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:59.375603+0000) 2022-09-08T07:18:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:18:59.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:18:59.295836+0000) 2022-09-08T07:19:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:00.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:00.375793+0000) 2022-09-08T07:19:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:00.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:00.295966+0000) 2022-09-08T07:19:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:00 smithi055 ceph-mon[97033]: pgmap v18830: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:00 smithi059 ceph-mon[97450]: pgmap v18830: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:01.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:01.375915+0000) 2022-09-08T07:19:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:01.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:01.296193+0000) 2022-09-08T07:19:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:02.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:02.376077+0000) 2022-09-08T07:19:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:02.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:02.296445+0000) 2022-09-08T07:19:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:02 smithi055 ceph-mon[97033]: pgmap v18831: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:19:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:02 smithi059 ceph-mon[97450]: pgmap v18831: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:19:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:03.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:03.376285+0000) 2022-09-08T07:19:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:03.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:03.296573+0000) 2022-09-08T07:19:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:04.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:04.296783+0000) 2022-09-08T07:19:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:04.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:04.376517+0000) 2022-09-08T07:19:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:04 smithi055 ceph-mon[97033]: pgmap v18832: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:04 smithi059 ceph-mon[97450]: pgmap v18832: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:05.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:05.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:05.296954+0000) 2022-09-08T07:19:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:05.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:05.376717+0000) 2022-09-08T07:19:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:06.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:06.376963+0000) 2022-09-08T07:19:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:06.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:06.297107+0000) 2022-09-08T07:19:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:06 smithi055 ceph-mon[97033]: pgmap v18833: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-08T07:19:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:19:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:06 smithi059 ceph-mon[97450]: pgmap v18833: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-08T07:19:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:19:07.564 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:07.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:07.377191+0000) 2022-09-08T07:19:07.565 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:07.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:07.297263+0000) 2022-09-08T07:19:07.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:19:07.561+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:19:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:19:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:19:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:08.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:08.377428+0000) 2022-09-08T07:19:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:08.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:08.297433+0000) 2022-09-08T07:19:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:08 smithi055 ceph-mon[97033]: pgmap v18834: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:19:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:08 smithi059 ceph-mon[97450]: pgmap v18834: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:19:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:09.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:09.377692+0000) 2022-09-08T07:19:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:09.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:09.297632+0000) 2022-09-08T07:19:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:10.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:10.377879+0000) 2022-09-08T07:19:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:10.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:10.297833+0000) 2022-09-08T07:19:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:10 smithi059 ceph-mon[97450]: pgmap v18835: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:19:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:10 smithi059 ceph-mon[97450]: osdmap e1255: 12 total, 11 up, 11 in 2022-09-08T07:19:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:10 smithi055 ceph-mon[97033]: pgmap v18835: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:19:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:10 smithi055 ceph-mon[97033]: osdmap e1255: 12 total, 11 up, 11 in 2022-09-08T07:19:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:11.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:11.378104+0000) 2022-09-08T07:19:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:11.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:11.298098+0000) 2022-09-08T07:19:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:12.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:12.378319+0000) 2022-09-08T07:19:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:12.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:12.298253+0000) 2022-09-08T07:19:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:12 smithi059 ceph-mon[97450]: pgmap v18837: 129 pgs: 3 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 100 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:19:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:12 smithi055 ceph-mon[97033]: pgmap v18837: 129 pgs: 3 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 100 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:19:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:13.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:13.378525+0000) 2022-09-08T07:19:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:13.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:13.298490+0000) 2022-09-08T07:19:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:14.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:14.378751+0000) 2022-09-08T07:19:14.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:14.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:14.298672+0000) 2022-09-08T07:19:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:14 smithi059 ceph-mon[97450]: pgmap v18838: 129 pgs: 3 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 100 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:19:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:14 smithi055 ceph-mon[97033]: pgmap v18838: 129 pgs: 3 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 100 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:19:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:15.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:15.378954+0000) 2022-09-08T07:19:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:15.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:15.298870+0000) 2022-09-08T07:19:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:16.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:16.379174+0000) 2022-09-08T07:19:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:16.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:16.299094+0000) 2022-09-08T07:19:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:16 smithi059 ceph-mon[97450]: pgmap v18839: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2022-09-08T07:19:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:16 smithi055 ceph-mon[97033]: pgmap v18839: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2022-09-08T07:19:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:17.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:17.379370+0000) 2022-09-08T07:19:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:17.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:17.299324+0000) 2022-09-08T07:19:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:18.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:18.379612+0000) 2022-09-08T07:19:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:18.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:18.299527+0000) 2022-09-08T07:19:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:18 smithi055 ceph-mon[97033]: pgmap v18840: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:19:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:18 smithi055 ceph-mon[97033]: osdmap e1256: 12 total, 11 up, 11 in 2022-09-08T07:19:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:18 smithi059 ceph-mon[97450]: pgmap v18840: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:19:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:18 smithi059 ceph-mon[97450]: osdmap e1256: 12 total, 11 up, 11 in 2022-09-08T07:19:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:19.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:19.379823+0000) 2022-09-08T07:19:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:19.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:19.299686+0000) 2022-09-08T07:19:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:19:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:19:19.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:19:19.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:19:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:19:20.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:19:20.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:19:20.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:19:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:20.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:20.379954+0000) 2022-09-08T07:19:20.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:20.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:20.299820+0000) 2022-09-08T07:19:20.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:20 smithi055 ceph-mon[97033]: pgmap v18842: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2022-09-08T07:19:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:20 smithi059 ceph-mon[97450]: pgmap v18842: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2022-09-08T07:19:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:21.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:21.380180+0000) 2022-09-08T07:19:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:21.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:21.300006+0000) 2022-09-08T07:19:22.570 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:22.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:22.380401+0000) 2022-09-08T07:19:22.571 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:22.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:22.300175+0000) 2022-09-08T07:19:22.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:19:22.567+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:19:22.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:22 smithi055 ceph-mon[97033]: pgmap v18843: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:19:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:22 smithi059 ceph-mon[97450]: pgmap v18843: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:19:23.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:23.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:23.380625+0000) 2022-09-08T07:19:23.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:23.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:23.300361+0000) 2022-09-08T07:19:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:24.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:24.380878+0000) 2022-09-08T07:19:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:24.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:24.300571+0000) 2022-09-08T07:19:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:24 smithi055 ceph-mon[97033]: pgmap v18844: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:19:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:24 smithi059 ceph-mon[97450]: pgmap v18844: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:19:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:25.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:25.381115+0000) 2022-09-08T07:19:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:25.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:25.300769+0000) 2022-09-08T07:19:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:26.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:26.381338+0000) 2022-09-08T07:19:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:26.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:26.300921+0000) 2022-09-08T07:19:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:26 smithi055 ceph-mon[97033]: pgmap v18845: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:19:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:26 smithi059 ceph-mon[97450]: pgmap v18845: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:19:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:27.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:27.381574+0000) 2022-09-08T07:19:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:27.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:27.301134+0000) 2022-09-08T07:19:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:28.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:28.381836+0000) 2022-09-08T07:19:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:28.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:28.301310+0000) 2022-09-08T07:19:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:28 smithi055 ceph-mon[97033]: pgmap v18846: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:19:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:28 smithi059 ceph-mon[97450]: pgmap v18846: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:19:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:29.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:29.382095+0000) 2022-09-08T07:19:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:29.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:29.301524+0000) 2022-09-08T07:19:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:30.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:30.382246+0000) 2022-09-08T07:19:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:30.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:30.301698+0000) 2022-09-08T07:19:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:30 smithi055 ceph-mon[97033]: pgmap v18847: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:19:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:30 smithi059 ceph-mon[97450]: pgmap v18847: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:19:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:31.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:31.382474+0000) 2022-09-08T07:19:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:31.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:31.301848+0000) 2022-09-08T07:19:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:32.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:32.301986+0000) 2022-09-08T07:19:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:32.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:32.382732+0000) 2022-09-08T07:19:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:32 smithi055 ceph-mon[97033]: pgmap v18848: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:19:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:32 smithi059 ceph-mon[97450]: pgmap v18848: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:19:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:33.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:33.382921+0000) 2022-09-08T07:19:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:33.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:33.302159+0000) 2022-09-08T07:19:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:34.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:34.383150+0000) 2022-09-08T07:19:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:34.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:34.302283+0000) 2022-09-08T07:19:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:34 smithi055 ceph-mon[97033]: pgmap v18849: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:34 smithi059 ceph-mon[97450]: pgmap v18849: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:35.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:35.383354+0000) 2022-09-08T07:19:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:35.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:35.302491+0000) 2022-09-08T07:19:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:36.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:36.302666+0000) 2022-09-08T07:19:36.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:36.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:36.383528+0000) 2022-09-08T07:19:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:36 smithi055 ceph-mon[97033]: pgmap v18850: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:19:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:36 smithi059 ceph-mon[97450]: pgmap v18850: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:19:37.585 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:37.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:37.302793+0000) 2022-09-08T07:19:37.586 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:37.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:37.383766+0000) 2022-09-08T07:19:37.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:19:37.581+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:19:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:38.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:38.384006+0000) 2022-09-08T07:19:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:38.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:38.302922+0000) 2022-09-08T07:19:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:38 smithi055 ceph-mon[97033]: pgmap v18851: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:38 smithi059 ceph-mon[97450]: pgmap v18851: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:39.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:39.384229+0000) 2022-09-08T07:19:39.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:39.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:39.303055+0000) 2022-09-08T07:19:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:40.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:40.384482+0000) 2022-09-08T07:19:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:40.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:40.303220+0000) 2022-09-08T07:19:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:40 smithi055 ceph-mon[97033]: pgmap v18852: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:40 smithi059 ceph-mon[97450]: pgmap v18852: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:41.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:41.384686+0000) 2022-09-08T07:19:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:41.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:41.303437+0000) 2022-09-08T07:19:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:42.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:42.384912+0000) 2022-09-08T07:19:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:42.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:42.303748+0000) 2022-09-08T07:19:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:42 smithi055 ceph-mon[97033]: pgmap v18853: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:19:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:42 smithi059 ceph-mon[97450]: pgmap v18853: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:19:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:43.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:43.385087+0000) 2022-09-08T07:19:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:43.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:43.303918+0000) 2022-09-08T07:19:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:44.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:44.385290+0000) 2022-09-08T07:19:44.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:44.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:44.304159+0000) 2022-09-08T07:19:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:44 smithi055 ceph-mon[97033]: pgmap v18854: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:44 smithi059 ceph-mon[97450]: pgmap v18854: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:45.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:45.385516+0000) 2022-09-08T07:19:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:45.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:45.304326+0000) 2022-09-08T07:19:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:46.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:46.385774+0000) 2022-09-08T07:19:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:46.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:46.304536+0000) 2022-09-08T07:19:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:46 smithi055 ceph-mon[97033]: pgmap v18855: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:19:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:46 smithi059 ceph-mon[97450]: pgmap v18855: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:19:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:47.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:47.385975+0000) 2022-09-08T07:19:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:47.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:47.304742+0000) 2022-09-08T07:19:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:48.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:48.386193+0000) 2022-09-08T07:19:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:48.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:48.304937+0000) 2022-09-08T07:19:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:48 smithi055 ceph-mon[97033]: pgmap v18856: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:48 smithi059 ceph-mon[97450]: pgmap v18856: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:49.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:49.305093+0000) 2022-09-08T07:19:49.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:49.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:49.386418+0000) 2022-09-08T07:19:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:50.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:50.305333+0000) 2022-09-08T07:19:50.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:50.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:50.386611+0000) 2022-09-08T07:19:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:50 smithi055 ceph-mon[97033]: pgmap v18857: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:50 smithi059 ceph-mon[97450]: pgmap v18857: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:51.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:51.305480+0000) 2022-09-08T07:19:51.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:51.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:51.386875+0000) 2022-09-08T07:19:52.601 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:52.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:52.387052+0000) 2022-09-08T07:19:52.602 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:52.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:52.305614+0000) 2022-09-08T07:19:52.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:19:52.598+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:19:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:52 smithi055 ceph-mon[97033]: pgmap v18858: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:19:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:52 smithi059 ceph-mon[97450]: pgmap v18858: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:19:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:53.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:53.387230+0000) 2022-09-08T07:19:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:53.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:53.305805+0000) 2022-09-08T07:19:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:54.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:54.305954+0000) 2022-09-08T07:19:54.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:54.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:54.387429+0000) 2022-09-08T07:19:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:54 smithi055 ceph-mon[97033]: pgmap v18859: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:54 smithi059 ceph-mon[97450]: pgmap v18859: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:55.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:55.387722+0000) 2022-09-08T07:19:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:55.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:55.306126+0000) 2022-09-08T07:19:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:56.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:56.387940+0000) 2022-09-08T07:19:56.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:56.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:56.306335+0000) 2022-09-08T07:19:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:56 smithi055 ceph-mon[97033]: pgmap v18860: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:19:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:56 smithi059 ceph-mon[97450]: pgmap v18860: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:19:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:57.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:57.388163+0000) 2022-09-08T07:19:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:57.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:57.306486+0000) 2022-09-08T07:19:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:58.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:58.388392+0000) 2022-09-08T07:19:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:58.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:58.306652+0000) 2022-09-08T07:19:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:19:58 smithi055 ceph-mon[97033]: pgmap v18861: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:19:58 smithi059 ceph-mon[97450]: pgmap v18861: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:19:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:19:59.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:59.388658+0000) 2022-09-08T07:19:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:19:59.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:19:59.306806+0000) 2022-09-08T07:20:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:00.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:00.388890+0000) 2022-09-08T07:20:00.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:00.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:00.306950+0000) 2022-09-08T07:20:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:00 smithi055 ceph-mon[97033]: pgmap v18862: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T07:20:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T07:20:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T07:20:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:00 smithi059 ceph-mon[97450]: pgmap v18862: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T07:20:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T07:20:01.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T07:20:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:01.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:01.389108+0000) 2022-09-08T07:20:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:01.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:01.307096+0000) 2022-09-08T07:20:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:02.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:02.389273+0000) 2022-09-08T07:20:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:02.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:02.307219+0000) 2022-09-08T07:20:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:02 smithi055 ceph-mon[97033]: pgmap v18863: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:20:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:02 smithi059 ceph-mon[97450]: pgmap v18863: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:20:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:03.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:03.389493+0000) 2022-09-08T07:20:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:03.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:03.307341+0000) 2022-09-08T07:20:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:04.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:04.389741+0000) 2022-09-08T07:20:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:04.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:04.307470+0000) 2022-09-08T07:20:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:04 smithi055 ceph-mon[97033]: pgmap v18864: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:04 smithi059 ceph-mon[97450]: pgmap v18864: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:05.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:05.390003+0000) 2022-09-08T07:20:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:05.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:05.307741+0000) 2022-09-08T07:20:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:06.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:06.390196+0000) 2022-09-08T07:20:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:06.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:06.307969+0000) 2022-09-08T07:20:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:06 smithi055 ceph-mon[97033]: pgmap v18865: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:20:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:20:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:06 smithi059 ceph-mon[97450]: pgmap v18865: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:20:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:20:07.608 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:07.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:07.390372+0000) 2022-09-08T07:20:07.609 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:07.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:07.308134+0000) 2022-09-08T07:20:07.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:20:07.605+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:20:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:20:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:20:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:08.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:08.308262+0000) 2022-09-08T07:20:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:08.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:08.390574+0000) 2022-09-08T07:20:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:08 smithi055 ceph-mon[97033]: pgmap v18866: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:20:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:08 smithi059 ceph-mon[97450]: pgmap v18866: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:20:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:09.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:09.308441+0000) 2022-09-08T07:20:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:09.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:09.390800+0000) 2022-09-08T07:20:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:10.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:10.308598+0000) 2022-09-08T07:20:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:10.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:10.391050+0000) 2022-09-08T07:20:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:10 smithi055 ceph-mon[97033]: pgmap v18867: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:20:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:10 smithi055 ceph-mon[97033]: osdmap e1257: 12 total, 11 up, 11 in 2022-09-08T07:20:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:10 smithi059 ceph-mon[97450]: pgmap v18867: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:20:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:10 smithi059 ceph-mon[97450]: osdmap e1257: 12 total, 11 up, 11 in 2022-09-08T07:20:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:11.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:11.391263+0000) 2022-09-08T07:20:11.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:11.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:11.308765+0000) 2022-09-08T07:20:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:12.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:12.308886+0000) 2022-09-08T07:20:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:12.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:12.391503+0000) 2022-09-08T07:20:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:12 smithi055 ceph-mon[97033]: pgmap v18869: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:20:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:12 smithi059 ceph-mon[97450]: pgmap v18869: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:20:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:13.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:13.391745+0000) 2022-09-08T07:20:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:13.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:13.309115+0000) 2022-09-08T07:20:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:14.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:14.391973+0000) 2022-09-08T07:20:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:14.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:14.309306+0000) 2022-09-08T07:20:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:14 smithi055 ceph-mon[97033]: pgmap v18870: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:20:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:14 smithi059 ceph-mon[97450]: pgmap v18870: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:20:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:15.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:15.309465+0000) 2022-09-08T07:20:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:15.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:15.392197+0000) 2022-09-08T07:20:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:16.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:16.392412+0000) 2022-09-08T07:20:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:16.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:16.309697+0000) 2022-09-08T07:20:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:16 smithi055 ceph-mon[97033]: pgmap v18871: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:20:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:16 smithi059 ceph-mon[97450]: pgmap v18871: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:20:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:17.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:17.392605+0000) 2022-09-08T07:20:17.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:17.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:17.309854+0000) 2022-09-08T07:20:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:18.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:18.392829+0000) 2022-09-08T07:20:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:18.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:18.310026+0000) 2022-09-08T07:20:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:18 smithi055 ceph-mon[97033]: pgmap v18872: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:20:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:18 smithi055 ceph-mon[97033]: osdmap e1258: 12 total, 11 up, 11 in 2022-09-08T07:20:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:18 smithi059 ceph-mon[97450]: pgmap v18872: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:20:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:18 smithi059 ceph-mon[97450]: osdmap e1258: 12 total, 11 up, 11 in 2022-09-08T07:20:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:19.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:19.310194+0000) 2022-09-08T07:20:19.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:19.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:19.393081+0000) 2022-09-08T07:20:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:20:19.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:20:19.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:20:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:20:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:20:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:20:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:20.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:20.393293+0000) 2022-09-08T07:20:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:20.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:20.310382+0000) 2022-09-08T07:20:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:20 smithi055 ceph-mon[97033]: pgmap v18874: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-08T07:20:20.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:20:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:20 smithi059 ceph-mon[97450]: pgmap v18874: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-08T07:20:21.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:20:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:21.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:21.393488+0000) 2022-09-08T07:20:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:21.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:21.310602+0000) 2022-09-08T07:20:22.609 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:22.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:22.310817+0000) 2022-09-08T07:20:22.610 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:22.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:22.393732+0000) 2022-09-08T07:20:22.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:20:22.605+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:20:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:22 smithi055 ceph-mon[97033]: pgmap v18875: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:20:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:22 smithi059 ceph-mon[97450]: pgmap v18875: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:20:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:23.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:23.393986+0000) 2022-09-08T07:20:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:23.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:23.310977+0000) 2022-09-08T07:20:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:24.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:24.394187+0000) 2022-09-08T07:20:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:24.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:24.311126+0000) 2022-09-08T07:20:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:24 smithi055 ceph-mon[97033]: pgmap v18876: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:20:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:24 smithi059 ceph-mon[97450]: pgmap v18876: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:20:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:25.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:25.394464+0000) 2022-09-08T07:20:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:25.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:25.311362+0000) 2022-09-08T07:20:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:26.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:26.394688+0000) 2022-09-08T07:20:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:26.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:26.311647+0000) 2022-09-08T07:20:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:26 smithi055 ceph-mon[97033]: pgmap v18877: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:20:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:26 smithi059 ceph-mon[97450]: pgmap v18877: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:20:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:27.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:27.394882+0000) 2022-09-08T07:20:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:27.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:27.311789+0000) 2022-09-08T07:20:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:28.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:28.395099+0000) 2022-09-08T07:20:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:28.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:28.311970+0000) 2022-09-08T07:20:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:28 smithi055 ceph-mon[97033]: pgmap v18878: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:20:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:28 smithi059 ceph-mon[97450]: pgmap v18878: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:20:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:29.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:29.312176+0000) 2022-09-08T07:20:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:29.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:29.395343+0000) 2022-09-08T07:20:30.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:30.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:30.312415+0000) 2022-09-08T07:20:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:30.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:30.395678+0000) 2022-09-08T07:20:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:30 smithi055 ceph-mon[97033]: pgmap v18879: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:20:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:30 smithi059 ceph-mon[97450]: pgmap v18879: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:20:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:31.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:31.395894+0000) 2022-09-08T07:20:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:31.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:31.312567+0000) 2022-09-08T07:20:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:32.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:32.312785+0000) 2022-09-08T07:20:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:32.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:32.396098+0000) 2022-09-08T07:20:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:32 smithi055 ceph-mon[97033]: pgmap v18880: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:20:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:32 smithi059 ceph-mon[97450]: pgmap v18880: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:20:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:33.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:33.396340+0000) 2022-09-08T07:20:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:33.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:33.312928+0000) 2022-09-08T07:20:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:34.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:34.396582+0000) 2022-09-08T07:20:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:34.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:34.313130+0000) 2022-09-08T07:20:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:34 smithi055 ceph-mon[97033]: pgmap v18881: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:34 smithi059 ceph-mon[97450]: pgmap v18881: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:35.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:35.396807+0000) 2022-09-08T07:20:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:35.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:35.313314+0000) 2022-09-08T07:20:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:36.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:36.396993+0000) 2022-09-08T07:20:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:36.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:36.313469+0000) 2022-09-08T07:20:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:36 smithi055 ceph-mon[97033]: pgmap v18882: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:20:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:36 smithi059 ceph-mon[97450]: pgmap v18882: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:20:37.616 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:37.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:37.397220+0000) 2022-09-08T07:20:37.617 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:37.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:37.313616+0000) 2022-09-08T07:20:37.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:20:37.613+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:20:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:38.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:38.397453+0000) 2022-09-08T07:20:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:38.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:38.313767+0000) 2022-09-08T07:20:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:38 smithi055 ceph-mon[97033]: pgmap v18883: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:38 smithi059 ceph-mon[97450]: pgmap v18883: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:39.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:39.397649+0000) 2022-09-08T07:20:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:39.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:39.313986+0000) 2022-09-08T07:20:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:40.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:40.397846+0000) 2022-09-08T07:20:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:40.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:40.314161+0000) 2022-09-08T07:20:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:40 smithi055 ceph-mon[97033]: pgmap v18884: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:40 smithi059 ceph-mon[97450]: pgmap v18884: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:41.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:41.398061+0000) 2022-09-08T07:20:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:41.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:41.314404+0000) 2022-09-08T07:20:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:42.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:42.398291+0000) 2022-09-08T07:20:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:42.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:42.314577+0000) 2022-09-08T07:20:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:42 smithi055 ceph-mon[97033]: pgmap v18885: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:20:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:42 smithi059 ceph-mon[97450]: pgmap v18885: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:20:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:43.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:43.398524+0000) 2022-09-08T07:20:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:43.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:43.314711+0000) 2022-09-08T07:20:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:44.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:44.398734+0000) 2022-09-08T07:20:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:44.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:44.314846+0000) 2022-09-08T07:20:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:44 smithi055 ceph-mon[97033]: pgmap v18886: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:44 smithi059 ceph-mon[97450]: pgmap v18886: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:45.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:45.398942+0000) 2022-09-08T07:20:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:45.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:45.315084+0000) 2022-09-08T07:20:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:46.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:46.399128+0000) 2022-09-08T07:20:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:46.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:46.315295+0000) 2022-09-08T07:20:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:46 smithi055 ceph-mon[97033]: pgmap v18887: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:20:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:46 smithi059 ceph-mon[97450]: pgmap v18887: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:20:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:47.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:47.399300+0000) 2022-09-08T07:20:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:47.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:47.315424+0000) 2022-09-08T07:20:48.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:48.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:48.399522+0000) 2022-09-08T07:20:48.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:48.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:48.315581+0000) 2022-09-08T07:20:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:48 smithi055 ceph-mon[97033]: pgmap v18888: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:48 smithi059 ceph-mon[97450]: pgmap v18888: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:49.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:49.399733+0000) 2022-09-08T07:20:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:49.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:49.315820+0000) 2022-09-08T07:20:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:50.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:50.399965+0000) 2022-09-08T07:20:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:50.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:50.315975+0000) 2022-09-08T07:20:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:50 smithi055 ceph-mon[97033]: pgmap v18889: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:50 smithi059 ceph-mon[97450]: pgmap v18889: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:51.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:51.400131+0000) 2022-09-08T07:20:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:51.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:51.316183+0000) 2022-09-08T07:20:52.632 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:52.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:52.400337+0000) 2022-09-08T07:20:52.633 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:52.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:52.316351+0000) 2022-09-08T07:20:52.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:20:52.628+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:20:52.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:52 smithi055 ceph-mon[97033]: pgmap v18890: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:20:53.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:52 smithi059 ceph-mon[97450]: pgmap v18890: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:20:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:53.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:53.316570+0000) 2022-09-08T07:20:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:53.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:53.400568+0000) 2022-09-08T07:20:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:54.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:54.400790+0000) 2022-09-08T07:20:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:54.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:54.316742+0000) 2022-09-08T07:20:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:54 smithi055 ceph-mon[97033]: pgmap v18891: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:54 smithi059 ceph-mon[97450]: pgmap v18891: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:55.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:55.400984+0000) 2022-09-08T07:20:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:55.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:55.316950+0000) 2022-09-08T07:20:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:56.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:56.401229+0000) 2022-09-08T07:20:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:56.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:56.317170+0000) 2022-09-08T07:20:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:56 smithi055 ceph-mon[97033]: pgmap v18892: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:20:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:56 smithi059 ceph-mon[97450]: pgmap v18892: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:20:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:57.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:57.401440+0000) 2022-09-08T07:20:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:57.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:57.317337+0000) 2022-09-08T07:20:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:58.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:58.401629+0000) 2022-09-08T07:20:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:58.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:58.317548+0000) 2022-09-08T07:20:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:20:58 smithi055 ceph-mon[97033]: pgmap v18893: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:20:58 smithi059 ceph-mon[97450]: pgmap v18893: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:20:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:20:59.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:59.401869+0000) 2022-09-08T07:20:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:20:59.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:20:59.317697+0000) 2022-09-08T07:21:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:00.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:00.402031+0000) 2022-09-08T07:21:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:00.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:00.317847+0000) 2022-09-08T07:21:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:00 smithi055 ceph-mon[97033]: pgmap v18894: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:00 smithi059 ceph-mon[97450]: pgmap v18894: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:01.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:01.402217+0000) 2022-09-08T07:21:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:01.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:01.318072+0000) 2022-09-08T07:21:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:02.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:02.402407+0000) 2022-09-08T07:21:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:02.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:02.318230+0000) 2022-09-08T07:21:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:02 smithi055 ceph-mon[97033]: pgmap v18895: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:21:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:02 smithi059 ceph-mon[97450]: pgmap v18895: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:21:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:03.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:03.318460+0000) 2022-09-08T07:21:03.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:03.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:03.402560+0000) 2022-09-08T07:21:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:04.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:04.402771+0000) 2022-09-08T07:21:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:04.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:04.318725+0000) 2022-09-08T07:21:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:04 smithi055 ceph-mon[97033]: pgmap v18896: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:04 smithi059 ceph-mon[97450]: pgmap v18896: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:05.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:05.402905+0000) 2022-09-08T07:21:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:05.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:05.318926+0000) 2022-09-08T07:21:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:06.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:06.403139+0000) 2022-09-08T07:21:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:06.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:06.319115+0000) 2022-09-08T07:21:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:06 smithi055 ceph-mon[97033]: pgmap v18897: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:21:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:21:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:06 smithi059 ceph-mon[97450]: pgmap v18897: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:21:07.068 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:21:07.647 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:07.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:07.403348+0000) 2022-09-08T07:21:07.648 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:07.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:07.319249+0000) 2022-09-08T07:21:07.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:21:07.643+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:21:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:21:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:21:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:08.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:08.403577+0000) 2022-09-08T07:21:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:08.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:08.319385+0000) 2022-09-08T07:21:08.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:08 smithi055 ceph-mon[97033]: pgmap v18898: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:21:09.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:08 smithi059 ceph-mon[97450]: pgmap v18898: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:21:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:09.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:09.403749+0000) 2022-09-08T07:21:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:09.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:09.319529+0000) 2022-09-08T07:21:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:10.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:10.403970+0000) 2022-09-08T07:21:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:10.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:10.319737+0000) 2022-09-08T07:21:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:10 smithi055 ceph-mon[97033]: pgmap v18899: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:21:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:10 smithi055 ceph-mon[97033]: osdmap e1259: 12 total, 11 up, 11 in 2022-09-08T07:21:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:10 smithi059 ceph-mon[97450]: pgmap v18899: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:21:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:10 smithi059 ceph-mon[97450]: osdmap e1259: 12 total, 11 up, 11 in 2022-09-08T07:21:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:11.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:11.404188+0000) 2022-09-08T07:21:11.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:11.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:11.319894+0000) 2022-09-08T07:21:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:12.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:12.404429+0000) 2022-09-08T07:21:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:12.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:12.320054+0000) 2022-09-08T07:21:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:12 smithi055 ceph-mon[97033]: pgmap v18901: 129 pgs: 11 active+clean+snaptrim_wait, 118 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:21:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:12 smithi059 ceph-mon[97450]: pgmap v18901: 129 pgs: 11 active+clean+snaptrim_wait, 118 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:21:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:13.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:13.404627+0000) 2022-09-08T07:21:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:13.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:13.320271+0000) 2022-09-08T07:21:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:14.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:14.404814+0000) 2022-09-08T07:21:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:14.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:14.320438+0000) 2022-09-08T07:21:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:14 smithi055 ceph-mon[97033]: pgmap v18902: 129 pgs: 11 active+clean+snaptrim_wait, 118 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:21:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:14 smithi059 ceph-mon[97450]: pgmap v18902: 129 pgs: 11 active+clean+snaptrim_wait, 118 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:21:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:15.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:15.405038+0000) 2022-09-08T07:21:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:15.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:15.320596+0000) 2022-09-08T07:21:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:16.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:16.405187+0000) 2022-09-08T07:21:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:16.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:16.320749+0000) 2022-09-08T07:21:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:16 smithi055 ceph-mon[97033]: pgmap v18903: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:21:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:16 smithi059 ceph-mon[97450]: pgmap v18903: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:21:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:17.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:17.405333+0000) 2022-09-08T07:21:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:17.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:17.320918+0000) 2022-09-08T07:21:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:18.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:18.405562+0000) 2022-09-08T07:21:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:18.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:18.321076+0000) 2022-09-08T07:21:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:18 smithi055 ceph-mon[97033]: pgmap v18904: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:21:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:18 smithi055 ceph-mon[97033]: osdmap e1260: 12 total, 11 up, 11 in 2022-09-08T07:21:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:18 smithi059 ceph-mon[97450]: pgmap v18904: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:21:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:18 smithi059 ceph-mon[97450]: osdmap e1260: 12 total, 11 up, 11 in 2022-09-08T07:21:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:19.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:19.405791+0000) 2022-09-08T07:21:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:19.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:19.321223+0000) 2022-09-08T07:21:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:20.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:20.405963+0000) 2022-09-08T07:21:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:20.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:20.321386+0000) 2022-09-08T07:21:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:20 smithi055 ceph-mon[97033]: pgmap v18906: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-08T07:21:20.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:21:20.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:21:20.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:21:20.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:21:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:20 smithi059 ceph-mon[97450]: pgmap v18906: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-08T07:21:21.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:21:21.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:21:21.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:21:21.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:21:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:21.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:21.406172+0000) 2022-09-08T07:21:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:21.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:21.321593+0000) 2022-09-08T07:21:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:22.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:22.406413+0000) 2022-09-08T07:21:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:22.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:22.321811+0000) 2022-09-08T07:21:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:22 smithi055 ceph-mon[97033]: pgmap v18907: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:21:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:22 smithi059 ceph-mon[97450]: pgmap v18907: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:21:23.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:21:22.659+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:21:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:23.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:23.406580+0000) 2022-09-08T07:21:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:23.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:23.321938+0000) 2022-09-08T07:21:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:24.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:24.406765+0000) 2022-09-08T07:21:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:24.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:24.322066+0000) 2022-09-08T07:21:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:24 smithi055 ceph-mon[97033]: pgmap v18908: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:21:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:24 smithi059 ceph-mon[97450]: pgmap v18908: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:21:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:25.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:25.406981+0000) 2022-09-08T07:21:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:25.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:25.322206+0000) 2022-09-08T07:21:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:26.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:26.322352+0000) 2022-09-08T07:21:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:26.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:26.407160+0000) 2022-09-08T07:21:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:26 smithi055 ceph-mon[97033]: pgmap v18909: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:21:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:26 smithi059 ceph-mon[97450]: pgmap v18909: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:21:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:27.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:27.322557+0000) 2022-09-08T07:21:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:27.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:27.407325+0000) 2022-09-08T07:21:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:28.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:28.322758+0000) 2022-09-08T07:21:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:28.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:28.407518+0000) 2022-09-08T07:21:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:28 smithi055 ceph-mon[97033]: pgmap v18910: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:21:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:28 smithi059 ceph-mon[97450]: pgmap v18910: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:21:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:29.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:29.322916+0000) 2022-09-08T07:21:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:29.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:29.407751+0000) 2022-09-08T07:21:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:30.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:30.407915+0000) 2022-09-08T07:21:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:30.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:30.323138+0000) 2022-09-08T07:21:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:30 smithi055 ceph-mon[97033]: pgmap v18911: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:21:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:30 smithi059 ceph-mon[97450]: pgmap v18911: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:21:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:31.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:31.323408+0000) 2022-09-08T07:21:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:31.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:31.408093+0000) 2022-09-08T07:21:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:32.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:32.408315+0000) 2022-09-08T07:21:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:32.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:32.323580+0000) 2022-09-08T07:21:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:32 smithi055 ceph-mon[97033]: pgmap v18912: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:21:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:32 smithi059 ceph-mon[97450]: pgmap v18912: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:21:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:33.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:33.408541+0000) 2022-09-08T07:21:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:33.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:33.323720+0000) 2022-09-08T07:21:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:34.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:34.408775+0000) 2022-09-08T07:21:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:34.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:34.323885+0000) 2022-09-08T07:21:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:34 smithi055 ceph-mon[97033]: pgmap v18913: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:35.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:34 smithi059 ceph-mon[97450]: pgmap v18913: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:35.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:35.324078+0000) 2022-09-08T07:21:35.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:35.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:35.409009+0000) 2022-09-08T07:21:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:36.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:36.409176+0000) 2022-09-08T07:21:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:36.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:36.324208+0000) 2022-09-08T07:21:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:36 smithi055 ceph-mon[97033]: pgmap v18914: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:21:37.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:36 smithi059 ceph-mon[97450]: pgmap v18914: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:21:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:37.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:37.409416+0000) 2022-09-08T07:21:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:37.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:37.324420+0000) 2022-09-08T07:21:38.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:21:37.666+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:21:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:38.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:38.409665+0000) 2022-09-08T07:21:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:38.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:38.324587+0000) 2022-09-08T07:21:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:38 smithi055 ceph-mon[97033]: pgmap v18915: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:38 smithi059 ceph-mon[97450]: pgmap v18915: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:39.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:39.324770+0000) 2022-09-08T07:21:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:39.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:39.409913+0000) 2022-09-08T07:21:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:40.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:40.410076+0000) 2022-09-08T07:21:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:40.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:40.324951+0000) 2022-09-08T07:21:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:40 smithi055 ceph-mon[97033]: pgmap v18916: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:40 smithi059 ceph-mon[97450]: pgmap v18916: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:41.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:41.410303+0000) 2022-09-08T07:21:41.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:41.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:41.325100+0000) 2022-09-08T07:21:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:42.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:42.410578+0000) 2022-09-08T07:21:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:42.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:42.325327+0000) 2022-09-08T07:21:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:42 smithi055 ceph-mon[97033]: pgmap v18917: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:21:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:42 smithi059 ceph-mon[97450]: pgmap v18917: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:21:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:43.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:43.410718+0000) 2022-09-08T07:21:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:43.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:43.325558+0000) 2022-09-08T07:21:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:44.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:44.410940+0000) 2022-09-08T07:21:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:44.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:44.325710+0000) 2022-09-08T07:21:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:44 smithi055 ceph-mon[97033]: pgmap v18918: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:44 smithi059 ceph-mon[97450]: pgmap v18918: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:45.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:45.411171+0000) 2022-09-08T07:21:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:45.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:45.325836+0000) 2022-09-08T07:21:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:46.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:46.411356+0000) 2022-09-08T07:21:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:46.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:46.325962+0000) 2022-09-08T07:21:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:46 smithi055 ceph-mon[97033]: pgmap v18919: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:21:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:46 smithi059 ceph-mon[97450]: pgmap v18919: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:21:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:47.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:47.411564+0000) 2022-09-08T07:21:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:47.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:47.326132+0000) 2022-09-08T07:21:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:48.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:48.411808+0000) 2022-09-08T07:21:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:48.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:48.326304+0000) 2022-09-08T07:21:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:48 smithi055 ceph-mon[97033]: pgmap v18920: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:48 smithi059 ceph-mon[97450]: pgmap v18920: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:49.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:49.412058+0000) 2022-09-08T07:21:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:49.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:49.326537+0000) 2022-09-08T07:21:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:50.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:50.412255+0000) 2022-09-08T07:21:50.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:50.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:50.326716+0000) 2022-09-08T07:21:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:50 smithi055 ceph-mon[97033]: pgmap v18921: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:50 smithi059 ceph-mon[97450]: pgmap v18921: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:51.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:51.412469+0000) 2022-09-08T07:21:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:51.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:51.326930+0000) 2022-09-08T07:21:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:52.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:52.412728+0000) 2022-09-08T07:21:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:52.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:52.327113+0000) 2022-09-08T07:21:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:52 smithi055 ceph-mon[97033]: pgmap v18922: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:21:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:52 smithi059 ceph-mon[97450]: pgmap v18922: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:21:53.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:21:52.677+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:21:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:53.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:53.412908+0000) 2022-09-08T07:21:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:53.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:53.327244+0000) 2022-09-08T07:21:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:54.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:54.413136+0000) 2022-09-08T07:21:54.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:54.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:54.327440+0000) 2022-09-08T07:21:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:54 smithi055 ceph-mon[97033]: pgmap v18923: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:54 smithi059 ceph-mon[97450]: pgmap v18923: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:55.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:55.413345+0000) 2022-09-08T07:21:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:55.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:55.327700+0000) 2022-09-08T07:21:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:56.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:56.413544+0000) 2022-09-08T07:21:56.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:56.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:56.327853+0000) 2022-09-08T07:21:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:56 smithi055 ceph-mon[97033]: pgmap v18924: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:21:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:56 smithi059 ceph-mon[97450]: pgmap v18924: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:21:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:57.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:57.413793+0000) 2022-09-08T07:21:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:57.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:57.328070+0000) 2022-09-08T07:21:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:58.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:58.414002+0000) 2022-09-08T07:21:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:58.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:58.328222+0000) 2022-09-08T07:21:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:21:58 smithi055 ceph-mon[97033]: pgmap v18925: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:21:58 smithi059 ceph-mon[97450]: pgmap v18925: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:21:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:21:59.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:59.414138+0000) 2022-09-08T07:21:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:21:59.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:21:59.328368+0000) 2022-09-08T07:22:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:00.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:00.414362+0000) 2022-09-08T07:22:00.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:00.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:00.328559+0000) 2022-09-08T07:22:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:00 smithi055 ceph-mon[97033]: pgmap v18926: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:22:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:00 smithi059 ceph-mon[97450]: pgmap v18926: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:22:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:01.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:01.414585+0000) 2022-09-08T07:22:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:01.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:01.328744+0000) 2022-09-08T07:22:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:02.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:02.414780+0000) 2022-09-08T07:22:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:02.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:02.329018+0000) 2022-09-08T07:22:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:02 smithi055 ceph-mon[97033]: pgmap v18927: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:22:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:02 smithi059 ceph-mon[97450]: pgmap v18927: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:22:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:03.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:03.414985+0000) 2022-09-08T07:22:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:03.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:03.329334+0000) 2022-09-08T07:22:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:04.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:04.415220+0000) 2022-09-08T07:22:04.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:04.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:04.329508+0000) 2022-09-08T07:22:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:04 smithi055 ceph-mon[97033]: pgmap v18928: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:22:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:04 smithi059 ceph-mon[97450]: pgmap v18928: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:22:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:05.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:05.415468+0000) 2022-09-08T07:22:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:05.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:05.329756+0000) 2022-09-08T07:22:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:06.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:06.415731+0000) 2022-09-08T07:22:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:06.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:06.329898+0000) 2022-09-08T07:22:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:06 smithi055 ceph-mon[97033]: pgmap v18929: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:22:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:22:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:06 smithi059 ceph-mon[97450]: pgmap v18929: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:22:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:22:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:07.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:07.415882+0000) 2022-09-08T07:22:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:07.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:07.330054+0000) 2022-09-08T07:22:07.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:22:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:22:08.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:22:07.693+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:22:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:08.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:08.330214+0000) 2022-09-08T07:22:08.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:08.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:08.416054+0000) 2022-09-08T07:22:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:08 smithi055 ceph-mon[97033]: pgmap v18930: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:22:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:08 smithi059 ceph-mon[97450]: pgmap v18930: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:22:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:09.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:09.416252+0000) 2022-09-08T07:22:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:09.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:09.330450+0000) 2022-09-08T07:22:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:10.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:10.416409+0000) 2022-09-08T07:22:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:10.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:10.330629+0000) 2022-09-08T07:22:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:10 smithi055 ceph-mon[97033]: pgmap v18931: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:22:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:10 smithi055 ceph-mon[97033]: osdmap e1261: 12 total, 11 up, 11 in 2022-09-08T07:22:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:10 smithi059 ceph-mon[97450]: pgmap v18931: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:22:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:10 smithi059 ceph-mon[97450]: osdmap e1261: 12 total, 11 up, 11 in 2022-09-08T07:22:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:11.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:11.416540+0000) 2022-09-08T07:22:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:11.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:11.330794+0000) 2022-09-08T07:22:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:12.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:12.416701+0000) 2022-09-08T07:22:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:12.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:12.331007+0000) 2022-09-08T07:22:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:12 smithi055 ceph-mon[97033]: pgmap v18933: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:22:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:12 smithi059 ceph-mon[97450]: pgmap v18933: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:22:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:13.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:13.331246+0000) 2022-09-08T07:22:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:13.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:13.416942+0000) 2022-09-08T07:22:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:14.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:14.417150+0000) 2022-09-08T07:22:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:14.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:14.331491+0000) 2022-09-08T07:22:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:14 smithi055 ceph-mon[97033]: pgmap v18934: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:22:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:14 smithi059 ceph-mon[97450]: pgmap v18934: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:22:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:15.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:15.417388+0000) 2022-09-08T07:22:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:15.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:15.331743+0000) 2022-09-08T07:22:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:16.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:16.417608+0000) 2022-09-08T07:22:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:16.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:16.331895+0000) 2022-09-08T07:22:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:16 smithi055 ceph-mon[97033]: pgmap v18935: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:22:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:16 smithi059 ceph-mon[97450]: pgmap v18935: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:22:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:17.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:17.417857+0000) 2022-09-08T07:22:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:17.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:17.332076+0000) 2022-09-08T07:22:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:18.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:18.418005+0000) 2022-09-08T07:22:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:18.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:18.332240+0000) 2022-09-08T07:22:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:18 smithi055 ceph-mon[97033]: pgmap v18936: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:22:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:18 smithi055 ceph-mon[97033]: osdmap e1262: 12 total, 11 up, 11 in 2022-09-08T07:22:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:18 smithi059 ceph-mon[97450]: pgmap v18936: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:22:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:18 smithi059 ceph-mon[97450]: osdmap e1262: 12 total, 11 up, 11 in 2022-09-08T07:22:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:19.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:19.418219+0000) 2022-09-08T07:22:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:19.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:19.332468+0000) 2022-09-08T07:22:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:20.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:20.418468+0000) 2022-09-08T07:22:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:20.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:20.332754+0000) 2022-09-08T07:22:20.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:20 smithi055 ceph-mon[97033]: pgmap v18938: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-08T07:22:20.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:22:20.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:22:20.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:22:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:20 smithi059 ceph-mon[97450]: pgmap v18938: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-08T07:22:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:22:21.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:22:21.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:22:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:21.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:21.418754+0000) 2022-09-08T07:22:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:21.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:21.332973+0000) 2022-09-08T07:22:22.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:22 smithi055 ceph-mon[97033]: pgmap v18939: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:22:22.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:22:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:22 smithi059 ceph-mon[97450]: pgmap v18939: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:22:22.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:22:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:22.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:22.333123+0000) 2022-09-08T07:22:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:22.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:22.418983+0000) 2022-09-08T07:22:23.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:22:22.697+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:22:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:23.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:23.419179+0000) 2022-09-08T07:22:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:23.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:23.333254+0000) 2022-09-08T07:22:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:24.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:24.333467+0000) 2022-09-08T07:22:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:24.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:24.419412+0000) 2022-09-08T07:22:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:24 smithi055 ceph-mon[97033]: pgmap v18940: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:22:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:24 smithi059 ceph-mon[97450]: pgmap v18940: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:22:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:25.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:25.333743+0000) 2022-09-08T07:22:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:25.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:25.419613+0000) 2022-09-08T07:22:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:26.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:26.333958+0000) 2022-09-08T07:22:26.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:26.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:26.419785+0000) 2022-09-08T07:22:26.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:26 smithi055 ceph-mon[97033]: pgmap v18941: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:22:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:26 smithi059 ceph-mon[97450]: pgmap v18941: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:22:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:27.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:27.334129+0000) 2022-09-08T07:22:27.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:27.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:27.419965+0000) 2022-09-08T07:22:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:28.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:28.420192+0000) 2022-09-08T07:22:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:28.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:28.334282+0000) 2022-09-08T07:22:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:28 smithi055 ceph-mon[97033]: pgmap v18942: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:22:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:28 smithi059 ceph-mon[97450]: pgmap v18942: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:22:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:29.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:29.420388+0000) 2022-09-08T07:22:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:29.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:29.334495+0000) 2022-09-08T07:22:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:30.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:30.420631+0000) 2022-09-08T07:22:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:30.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:30.334661+0000) 2022-09-08T07:22:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:30 smithi055 ceph-mon[97033]: pgmap v18943: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:22:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:30 smithi059 ceph-mon[97450]: pgmap v18943: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:22:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:31.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:31.420904+0000) 2022-09-08T07:22:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:31.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:31.334809+0000) 2022-09-08T07:22:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:32.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:32.421052+0000) 2022-09-08T07:22:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:32.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:32.334983+0000) 2022-09-08T07:22:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:32 smithi055 ceph-mon[97033]: pgmap v18944: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:22:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:32 smithi059 ceph-mon[97450]: pgmap v18944: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:22:33.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:33.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:33.335209+0000) 2022-09-08T07:22:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:33.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:33.421262+0000) 2022-09-08T07:22:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:34.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:34.421484+0000) 2022-09-08T07:22:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:34.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:34.335424+0000) 2022-09-08T07:22:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:34 smithi055 ceph-mon[97033]: pgmap v18945: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:22:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:34 smithi059 ceph-mon[97450]: pgmap v18945: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:22:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:35.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:35.421711+0000) 2022-09-08T07:22:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:35.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:35.335658+0000) 2022-09-08T07:22:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:36.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:36.335858+0000) 2022-09-08T07:22:36.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:36.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:36.421913+0000) 2022-09-08T07:22:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:36 smithi055 ceph-mon[97033]: pgmap v18946: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:22:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:36 smithi059 ceph-mon[97450]: pgmap v18946: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:22:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:37.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:37.422036+0000) 2022-09-08T07:22:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:37.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:37.336065+0000) 2022-09-08T07:22:38.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:22:37.707+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:22:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:38.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:38.422261+0000) 2022-09-08T07:22:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:38.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:38.336297+0000) 2022-09-08T07:22:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:38 smithi055 ceph-mon[97033]: pgmap v18947: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:22:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:38 smithi059 ceph-mon[97450]: pgmap v18947: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:22:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:39.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:39.422531+0000) 2022-09-08T07:22:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:39.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:39.336463+0000) 2022-09-08T07:22:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:40.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:40.422792+0000) 2022-09-08T07:22:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:40.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:40.336633+0000) 2022-09-08T07:22:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:40 smithi055 ceph-mon[97033]: pgmap v18948: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:22:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:40 smithi059 ceph-mon[97450]: pgmap v18948: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:22:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:41.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:41.422983+0000) 2022-09-08T07:22:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:41.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:41.336877+0000) 2022-09-08T07:22:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:42.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:42.423217+0000) 2022-09-08T07:22:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:42.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:42.337046+0000) 2022-09-08T07:22:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:42 smithi055 ceph-mon[97033]: pgmap v18949: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:22:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:42 smithi059 ceph-mon[97450]: pgmap v18949: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:22:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:43.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:43.423446+0000) 2022-09-08T07:22:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:43.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:43.337245+0000) 2022-09-08T07:22:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:44.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:44.337371+0000) 2022-09-08T07:22:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:44.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:44.423601+0000) 2022-09-08T07:22:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:44 smithi055 ceph-mon[97033]: pgmap v18950: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:22:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:44 smithi059 ceph-mon[97450]: pgmap v18950: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:22:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:45.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:45.423811+0000) 2022-09-08T07:22:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:45.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:45.337534+0000) 2022-09-08T07:22:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:46.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:46.337696+0000) 2022-09-08T07:22:46.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:46.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:46.423975+0000) 2022-09-08T07:22:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:46 smithi055 ceph-mon[97033]: pgmap v18951: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:22:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:46 smithi059 ceph-mon[97450]: pgmap v18951: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:22:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:47.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:47.424211+0000) 2022-09-08T07:22:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:47.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:47.337813+0000) 2022-09-08T07:22:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:48.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:48.424420+0000) 2022-09-08T07:22:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:48.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:48.337989+0000) 2022-09-08T07:22:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:48 smithi055 ceph-mon[97033]: pgmap v18952: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:22:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:48 smithi059 ceph-mon[97450]: pgmap v18952: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:22:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:49.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:49.424621+0000) 2022-09-08T07:22:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:49.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:49.338153+0000) 2022-09-08T07:22:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:50.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:50.424875+0000) 2022-09-08T07:22:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:50.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:50.338388+0000) 2022-09-08T07:22:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:50 smithi055 ceph-mon[97033]: pgmap v18953: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:22:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:50 smithi059 ceph-mon[97450]: pgmap v18953: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:22:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:51.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:51.425055+0000) 2022-09-08T07:22:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:51.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:51.338550+0000) 2022-09-08T07:22:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:52.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:52.338774+0000) 2022-09-08T07:22:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:52.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:52.425231+0000) 2022-09-08T07:22:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:52 smithi055 ceph-mon[97033]: pgmap v18954: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:22:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:52 smithi059 ceph-mon[97450]: pgmap v18954: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:22:53.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:22:52.717+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:22:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:53.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:53.425438+0000) 2022-09-08T07:22:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:53.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:53.338974+0000) 2022-09-08T07:22:54.661 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:54.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:54.425674+0000) 2022-09-08T07:22:54.662 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:54.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:54.339108+0000) 2022-09-08T07:22:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:54 smithi055 ceph-mon[97033]: pgmap v18955: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:22:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:54 smithi059 ceph-mon[97450]: pgmap v18955: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:22:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:55.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:55.339327+0000) 2022-09-08T07:22:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:55.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:55.425919+0000) 2022-09-08T07:22:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:56.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:56.426141+0000) 2022-09-08T07:22:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:56.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:56.339566+0000) 2022-09-08T07:22:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:56 smithi055 ceph-mon[97033]: pgmap v18956: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:22:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:56 smithi059 ceph-mon[97450]: pgmap v18956: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:22:57.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:57.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:57.426383+0000) 2022-09-08T07:22:57.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:57.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:57.339798+0000) 2022-09-08T07:22:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:58.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:58.426605+0000) 2022-09-08T07:22:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:58.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:58.340041+0000) 2022-09-08T07:22:58.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:22:58 smithi055 ceph-mon[97033]: pgmap v18957: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:22:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:22:58 smithi059 ceph-mon[97450]: pgmap v18957: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:22:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:22:59.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:59.426883+0000) 2022-09-08T07:22:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:22:59.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:22:59.340258+0000) 2022-09-08T07:23:00.351 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:00.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:00.340513+0000) 2022-09-08T07:23:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:00.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:00.427058+0000) 2022-09-08T07:23:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:00 smithi055 ceph-mon[97033]: pgmap v18958: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:23:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:00 smithi059 ceph-mon[97450]: pgmap v18958: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:23:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:01.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:01.340695+0000) 2022-09-08T07:23:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:01.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:01.427261+0000) 2022-09-08T07:23:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:02.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:02.427476+0000) 2022-09-08T07:23:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:02.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:02.340966+0000) 2022-09-08T07:23:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:02 smithi055 ceph-mon[97033]: pgmap v18959: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:23:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:02 smithi059 ceph-mon[97450]: pgmap v18959: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:23:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:03.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:03.427734+0000) 2022-09-08T07:23:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:03.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:03.341207+0000) 2022-09-08T07:23:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:04.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:04.427982+0000) 2022-09-08T07:23:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:04.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:04.341424+0000) 2022-09-08T07:23:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:04 smithi055 ceph-mon[97033]: pgmap v18960: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:23:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:04 smithi059 ceph-mon[97450]: pgmap v18960: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:23:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:05.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:05.428237+0000) 2022-09-08T07:23:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:05.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:05.341620+0000) 2022-09-08T07:23:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:06.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:06.428466+0000) 2022-09-08T07:23:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:06.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:06.341766+0000) 2022-09-08T07:23:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:06 smithi055 ceph-mon[97033]: pgmap v18961: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:23:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:23:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:06 smithi059 ceph-mon[97450]: pgmap v18961: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:23:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:23:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:07.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:07.428732+0000) 2022-09-08T07:23:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:07.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:07.341991+0000) 2022-09-08T07:23:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:23:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:23:08.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:23:07.731+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:23:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:08.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:08.428953+0000) 2022-09-08T07:23:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:08.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:08.342163+0000) 2022-09-08T07:23:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:08 smithi055 ceph-mon[97033]: pgmap v18962: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:23:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:08 smithi059 ceph-mon[97450]: pgmap v18962: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:23:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:09.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:09.342291+0000) 2022-09-08T07:23:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:09.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:09.429141+0000) 2022-09-08T07:23:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:10.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:10.429295+0000) 2022-09-08T07:23:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:10.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:10.342462+0000) 2022-09-08T07:23:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:10 smithi059 ceph-mon[97450]: pgmap v18963: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:23:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:10 smithi059 ceph-mon[97450]: osdmap e1263: 12 total, 11 up, 11 in 2022-09-08T07:23:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:10 smithi055 ceph-mon[97033]: pgmap v18963: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:23:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:10 smithi055 ceph-mon[97033]: osdmap e1263: 12 total, 11 up, 11 in 2022-09-08T07:23:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:11.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:11.429524+0000) 2022-09-08T07:23:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:11.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:11.342728+0000) 2022-09-08T07:23:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:12.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:12.429773+0000) 2022-09-08T07:23:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:12.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:12.342977+0000) 2022-09-08T07:23:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:12 smithi059 ceph-mon[97450]: pgmap v18965: 129 pgs: 22 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 104 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:23:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:12 smithi055 ceph-mon[97033]: pgmap v18965: 129 pgs: 22 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 104 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:23:13.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:13.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:13.343123+0000) 2022-09-08T07:23:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:13.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:13.430051+0000) 2022-09-08T07:23:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:14.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:14.430316+0000) 2022-09-08T07:23:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:14.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:14.343328+0000) 2022-09-08T07:23:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:14 smithi059 ceph-mon[97450]: pgmap v18966: 129 pgs: 22 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 104 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:23:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:14 smithi055 ceph-mon[97033]: pgmap v18966: 129 pgs: 22 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 104 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:23:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:15.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:15.430549+0000) 2022-09-08T07:23:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:15.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:15.343549+0000) 2022-09-08T07:23:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:16.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:16.430786+0000) 2022-09-08T07:23:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:16.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:16.343726+0000) 2022-09-08T07:23:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:16 smithi059 ceph-mon[97450]: pgmap v18967: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:23:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:16 smithi055 ceph-mon[97033]: pgmap v18967: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:23:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:17.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:17.431085+0000) 2022-09-08T07:23:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:17.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:17.343881+0000) 2022-09-08T07:23:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:18.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:18.431282+0000) 2022-09-08T07:23:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:18.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:18.344065+0000) 2022-09-08T07:23:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:18 smithi055 ceph-mon[97033]: pgmap v18968: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:23:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:18 smithi055 ceph-mon[97033]: osdmap e1264: 12 total, 11 up, 11 in 2022-09-08T07:23:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:18 smithi059 ceph-mon[97450]: pgmap v18968: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:23:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:18 smithi059 ceph-mon[97450]: osdmap e1264: 12 total, 11 up, 11 in 2022-09-08T07:23:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:19.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:19.431501+0000) 2022-09-08T07:23:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:19.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:19.344210+0000) 2022-09-08T07:23:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:20.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:20.431739+0000) 2022-09-08T07:23:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:20.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:20.344452+0000) 2022-09-08T07:23:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:20 smithi055 ceph-mon[97033]: pgmap v18970: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-08T07:23:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:20 smithi059 ceph-mon[97450]: pgmap v18970: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-08T07:23:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:21.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:21.431989+0000) 2022-09-08T07:23:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:21.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:21.344761+0000) 2022-09-08T07:23:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:23:21.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:23:21.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:23:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:23:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:23:22.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:23:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:22.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:22.432230+0000) 2022-09-08T07:23:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:22.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:22.344961+0000) 2022-09-08T07:23:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:22 smithi055 ceph-mon[97033]: pgmap v18971: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:23:22.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:23:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:22 smithi059 ceph-mon[97450]: pgmap v18971: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:23:23.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:23:23.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:23:22.740+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:23:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:23.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:23.432426+0000) 2022-09-08T07:23:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:23.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:23.345121+0000) 2022-09-08T07:23:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:24.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:24.432574+0000) 2022-09-08T07:23:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:24.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:24.345278+0000) 2022-09-08T07:23:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:24 smithi055 ceph-mon[97033]: pgmap v18972: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:23:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:24 smithi059 ceph-mon[97450]: pgmap v18972: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:23:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:25.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:25.432846+0000) 2022-09-08T07:23:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:25.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:25.345494+0000) 2022-09-08T07:23:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:26.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:26.433062+0000) 2022-09-08T07:23:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:26.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:26.345705+0000) 2022-09-08T07:23:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:26 smithi055 ceph-mon[97033]: pgmap v18973: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:23:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:26 smithi059 ceph-mon[97450]: pgmap v18973: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:23:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:27.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:27.433299+0000) 2022-09-08T07:23:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:27.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:27.345854+0000) 2022-09-08T07:23:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:28.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:28.433534+0000) 2022-09-08T07:23:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:28.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:28.346067+0000) 2022-09-08T07:23:28.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:28 smithi055 ceph-mon[97033]: pgmap v18974: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:23:29.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:28 smithi059 ceph-mon[97450]: pgmap v18974: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:23:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:29.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:29.433751+0000) 2022-09-08T07:23:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:29.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:29.346268+0000) 2022-09-08T07:23:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:30.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:30.433918+0000) 2022-09-08T07:23:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:30.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:30.346410+0000) 2022-09-08T07:23:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:30 smithi055 ceph-mon[97033]: pgmap v18975: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:23:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:30 smithi059 ceph-mon[97450]: pgmap v18975: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:23:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:31.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:31.434150+0000) 2022-09-08T07:23:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:31.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:31.346568+0000) 2022-09-08T07:23:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:32.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:32.434410+0000) 2022-09-08T07:23:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:32.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:32.346724+0000) 2022-09-08T07:23:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:32 smithi055 ceph-mon[97033]: pgmap v18976: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:23:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:32 smithi059 ceph-mon[97450]: pgmap v18976: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:23:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:33.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:33.434674+0000) 2022-09-08T07:23:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:33.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:33.346876+0000) 2022-09-08T07:23:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:34.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:34.434854+0000) 2022-09-08T07:23:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:34.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:34.347052+0000) 2022-09-08T07:23:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:34 smithi055 ceph-mon[97033]: pgmap v18977: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:23:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:34 smithi059 ceph-mon[97450]: pgmap v18977: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:23:35.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:35.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:35.347196+0000) 2022-09-08T07:23:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:35.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:35.435112+0000) 2022-09-08T07:23:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:36.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:36.435327+0000) 2022-09-08T07:23:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:36.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:36.347369+0000) 2022-09-08T07:23:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:36 smithi055 ceph-mon[97033]: pgmap v18978: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:23:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:36 smithi059 ceph-mon[97450]: pgmap v18978: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:23:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:37.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:37.435531+0000) 2022-09-08T07:23:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:37.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:37.347509+0000) 2022-09-08T07:23:38.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:23:37.745+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:23:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:38.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:38.435769+0000) 2022-09-08T07:23:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:38.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:38.347749+0000) 2022-09-08T07:23:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:38 smithi055 ceph-mon[97033]: pgmap v18979: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:23:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:38 smithi059 ceph-mon[97450]: pgmap v18979: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:23:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:39.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:39.435890+0000) 2022-09-08T07:23:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:39.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:39.347983+0000) 2022-09-08T07:23:40.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:40.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:40.348182+0000) 2022-09-08T07:23:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:40.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:40.436118+0000) 2022-09-08T07:23:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:40 smithi055 ceph-mon[97033]: pgmap v18980: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:23:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:40 smithi059 ceph-mon[97450]: pgmap v18980: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:23:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:41.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:41.436308+0000) 2022-09-08T07:23:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:41.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:41.348352+0000) 2022-09-08T07:23:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:42.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:42.348611+0000) 2022-09-08T07:23:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:42.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:42.436531+0000) 2022-09-08T07:23:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:42 smithi055 ceph-mon[97033]: pgmap v18981: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:23:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:42 smithi059 ceph-mon[97450]: pgmap v18981: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:23:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:43.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:43.436798+0000) 2022-09-08T07:23:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:43.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:43.348748+0000) 2022-09-08T07:23:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:44.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:44.436995+0000) 2022-09-08T07:23:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:44.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:44.348906+0000) 2022-09-08T07:23:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:44 smithi055 ceph-mon[97033]: pgmap v18982: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:23:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:44 smithi059 ceph-mon[97450]: pgmap v18982: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:23:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:45.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:45.437205+0000) 2022-09-08T07:23:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:45.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:45.349081+0000) 2022-09-08T07:23:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:46.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:46.437457+0000) 2022-09-08T07:23:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:46.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:46.349231+0000) 2022-09-08T07:23:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:46 smithi059 ceph-mon[97450]: pgmap v18983: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:23:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:46 smithi055 ceph-mon[97033]: pgmap v18983: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:23:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:47.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:47.437606+0000) 2022-09-08T07:23:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:47.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:47.349396+0000) 2022-09-08T07:23:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:48.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:48.437778+0000) 2022-09-08T07:23:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:48.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:48.349615+0000) 2022-09-08T07:23:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:48 smithi059 ceph-mon[97450]: pgmap v18984: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:23:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:48 smithi055 ceph-mon[97033]: pgmap v18984: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:23:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:49.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:49.349839+0000) 2022-09-08T07:23:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:49.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:49.437979+0000) 2022-09-08T07:23:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:50.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:50.438194+0000) 2022-09-08T07:23:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:50.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:50.350017+0000) 2022-09-08T07:23:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:50 smithi059 ceph-mon[97450]: pgmap v18985: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:23:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:50 smithi055 ceph-mon[97033]: pgmap v18985: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:23:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:51.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:51.438416+0000) 2022-09-08T07:23:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:51.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:51.350191+0000) 2022-09-08T07:23:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:52.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:52.438678+0000) 2022-09-08T07:23:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:52.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:52.350454+0000) 2022-09-08T07:23:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:52 smithi059 ceph-mon[97450]: pgmap v18986: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:23:53.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:23:52.746+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:23:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:52 smithi055 ceph-mon[97033]: pgmap v18986: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:23:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:53.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:53.438867+0000) 2022-09-08T07:23:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:53.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:53.350607+0000) 2022-09-08T07:23:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:54.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:54.439074+0000) 2022-09-08T07:23:54.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:54.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:54.350784+0000) 2022-09-08T07:23:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:54 smithi059 ceph-mon[97450]: pgmap v18987: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:23:55.239 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:54 smithi055 ceph-mon[97033]: pgmap v18987: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:23:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:55.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:55.439278+0000) 2022-09-08T07:23:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:55.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:55.351017+0000) 2022-09-08T07:23:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:56.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:56.439504+0000) 2022-09-08T07:23:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:56.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:56.351170+0000) 2022-09-08T07:23:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:56 smithi059 ceph-mon[97450]: pgmap v18988: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:23:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:56 smithi055 ceph-mon[97033]: pgmap v18988: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:23:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:57.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:57.439735+0000) 2022-09-08T07:23:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:57.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:57.351317+0000) 2022-09-08T07:23:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:58.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:58.439907+0000) 2022-09-08T07:23:58.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:58.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:58.351458+0000) 2022-09-08T07:23:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:23:58 smithi059 ceph-mon[97450]: pgmap v18989: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:23:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:23:58 smithi055 ceph-mon[97033]: pgmap v18989: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:23:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:23:59.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:59.440188+0000) 2022-09-08T07:23:59.699 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:23:59.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:23:59.351716+0000) 2022-09-08T07:24:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:00.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:00.440454+0000) 2022-09-08T07:24:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:00.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:00.351882+0000) 2022-09-08T07:24:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:00 smithi059 ceph-mon[97450]: pgmap v18990: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:00 smithi055 ceph-mon[97033]: pgmap v18990: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:01.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:01.352133+0000) 2022-09-08T07:24:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:01.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:01.440679+0000) 2022-09-08T07:24:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:02.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:02.440870+0000) 2022-09-08T07:24:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:02.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:02.352303+0000) 2022-09-08T07:24:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:02 smithi059 ceph-mon[97450]: pgmap v18991: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:24:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:02 smithi055 ceph-mon[97033]: pgmap v18991: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:24:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:03.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:03.441000+0000) 2022-09-08T07:24:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:03.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:03.352499+0000) 2022-09-08T07:24:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:04.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:04.441216+0000) 2022-09-08T07:24:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:04.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:04.352758+0000) 2022-09-08T07:24:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:04 smithi059 ceph-mon[97450]: pgmap v18992: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:04 smithi055 ceph-mon[97033]: pgmap v18992: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:05.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:05.441429+0000) 2022-09-08T07:24:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:05.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:05.352922+0000) 2022-09-08T07:24:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:06.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:06.353131+0000) 2022-09-08T07:24:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:06.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:06.441697+0000) 2022-09-08T07:24:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:06 smithi059 ceph-mon[97450]: pgmap v18993: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:24:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:24:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:06 smithi055 ceph-mon[97033]: pgmap v18993: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:24:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:24:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:07.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:07.441891+0000) 2022-09-08T07:24:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:07.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:07.353294+0000) 2022-09-08T07:24:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:24:08.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:24:07.752+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:24:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:24:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:08.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:08.353456+0000) 2022-09-08T07:24:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:08.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:08.442008+0000) 2022-09-08T07:24:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:08 smithi059 ceph-mon[97450]: pgmap v18994: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:24:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:08 smithi055 ceph-mon[97033]: pgmap v18994: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:24:09.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:09.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:09.353663+0000) 2022-09-08T07:24:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:09.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:09.442223+0000) 2022-09-08T07:24:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:10.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:10.353928+0000) 2022-09-08T07:24:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:10.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:10.442476+0000) 2022-09-08T07:24:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:10 smithi055 ceph-mon[97033]: pgmap v18995: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:24:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:10 smithi055 ceph-mon[97033]: osdmap e1265: 12 total, 11 up, 11 in 2022-09-08T07:24:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:10 smithi059 ceph-mon[97450]: pgmap v18995: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:24:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:10 smithi059 ceph-mon[97450]: osdmap e1265: 12 total, 11 up, 11 in 2022-09-08T07:24:11.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:11.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:11.354130+0000) 2022-09-08T07:24:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:11.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:11.442677+0000) 2022-09-08T07:24:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:12.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:12.442875+0000) 2022-09-08T07:24:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:12.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:12.354376+0000) 2022-09-08T07:24:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:12 smithi055 ceph-mon[97033]: pgmap v18997: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:24:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:12 smithi059 ceph-mon[97450]: pgmap v18997: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:24:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:13.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:13.443109+0000) 2022-09-08T07:24:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:13.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:13.354524+0000) 2022-09-08T07:24:14.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:14.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:14.354702+0000) 2022-09-08T07:24:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:14.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:14.443331+0000) 2022-09-08T07:24:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:14 smithi055 ceph-mon[97033]: pgmap v18998: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:24:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:14 smithi059 ceph-mon[97450]: pgmap v18998: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:24:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:15.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:15.443564+0000) 2022-09-08T07:24:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:15.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:15.354896+0000) 2022-09-08T07:24:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:16.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:16.443825+0000) 2022-09-08T07:24:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:16.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:16.355077+0000) 2022-09-08T07:24:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:16 smithi055 ceph-mon[97033]: pgmap v18999: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:24:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:16 smithi059 ceph-mon[97450]: pgmap v18999: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:24:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:17.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:17.444076+0000) 2022-09-08T07:24:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:17.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:17.355225+0000) 2022-09-08T07:24:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:18.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:18.444262+0000) 2022-09-08T07:24:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:18.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:18.355473+0000) 2022-09-08T07:24:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:18 smithi055 ceph-mon[97033]: pgmap v19000: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:24:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:18 smithi055 ceph-mon[97033]: osdmap e1266: 12 total, 11 up, 11 in 2022-09-08T07:24:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:18 smithi059 ceph-mon[97450]: pgmap v19000: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:24:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:18 smithi059 ceph-mon[97450]: osdmap e1266: 12 total, 11 up, 11 in 2022-09-08T07:24:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:19.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:19.444432+0000) 2022-09-08T07:24:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:19.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:19.355648+0000) 2022-09-08T07:24:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:20.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:20.355859+0000) 2022-09-08T07:24:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:20.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:20.444713+0000) 2022-09-08T07:24:20.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:20 smithi055 ceph-mon[97033]: pgmap v19002: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s 2022-09-08T07:24:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:20 smithi059 ceph-mon[97450]: pgmap v19002: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s 2022-09-08T07:24:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:21.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:21.444980+0000) 2022-09-08T07:24:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:21.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:21.356065+0000) 2022-09-08T07:24:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:22.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:22.445268+0000) 2022-09-08T07:24:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:22.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:22.356200+0000) 2022-09-08T07:24:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:22 smithi055 ceph-mon[97033]: pgmap v19003: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:24:22.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:24:22.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:24:22.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:24:22.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:24:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:22 smithi059 ceph-mon[97450]: pgmap v19003: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:24:23.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:24:23.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:24:23.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:24:23.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:24:23.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:24:22.768+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:24:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:23.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:23.445563+0000) 2022-09-08T07:24:23.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:23.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:23.356351+0000) 2022-09-08T07:24:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:24.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:24.445820+0000) 2022-09-08T07:24:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:24.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:24.356587+0000) 2022-09-08T07:24:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:24 smithi055 ceph-mon[97033]: pgmap v19004: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:24:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:24 smithi059 ceph-mon[97450]: pgmap v19004: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:24:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:25.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:25.446074+0000) 2022-09-08T07:24:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:25.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:25.356701+0000) 2022-09-08T07:24:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:26.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:26.446296+0000) 2022-09-08T07:24:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:26.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:26.356855+0000) 2022-09-08T07:24:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:26 smithi055 ceph-mon[97033]: pgmap v19005: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:24:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:26 smithi059 ceph-mon[97450]: pgmap v19005: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:24:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:27.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:27.446442+0000) 2022-09-08T07:24:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:27.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:27.357003+0000) 2022-09-08T07:24:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:28.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:28.446666+0000) 2022-09-08T07:24:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:28.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:28.357247+0000) 2022-09-08T07:24:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:28 smithi055 ceph-mon[97033]: pgmap v19006: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:24:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:28 smithi059 ceph-mon[97450]: pgmap v19006: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:24:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:29.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:29.446910+0000) 2022-09-08T07:24:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:29.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:29.357483+0000) 2022-09-08T07:24:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:30.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:30.447131+0000) 2022-09-08T07:24:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:30.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:30.357677+0000) 2022-09-08T07:24:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:30 smithi055 ceph-mon[97033]: pgmap v19007: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:24:31.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:30 smithi059 ceph-mon[97450]: pgmap v19007: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:24:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:31.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:31.447296+0000) 2022-09-08T07:24:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:31.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:31.357830+0000) 2022-09-08T07:24:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:32.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:32.447544+0000) 2022-09-08T07:24:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:32.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:32.357987+0000) 2022-09-08T07:24:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:32 smithi055 ceph-mon[97033]: pgmap v19008: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:24:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:32 smithi059 ceph-mon[97450]: pgmap v19008: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:24:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:33.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:33.447768+0000) 2022-09-08T07:24:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:33.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:33.358198+0000) 2022-09-08T07:24:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:34.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:34.448003+0000) 2022-09-08T07:24:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:34.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:34.358422+0000) 2022-09-08T07:24:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:34 smithi055 ceph-mon[97033]: pgmap v19009: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:34 smithi059 ceph-mon[97450]: pgmap v19009: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:35.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:35.358583+0000) 2022-09-08T07:24:35.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:35.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:35.448182+0000) 2022-09-08T07:24:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:36.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:36.358714+0000) 2022-09-08T07:24:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:36.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:36.448362+0000) 2022-09-08T07:24:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:36 smithi055 ceph-mon[97033]: pgmap v19010: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:24:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:36 smithi059 ceph-mon[97450]: pgmap v19010: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:24:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:37.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:37.448570+0000) 2022-09-08T07:24:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:37.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:37.358864+0000) 2022-09-08T07:24:38.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:24:37.784+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:24:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:38.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:38.448821+0000) 2022-09-08T07:24:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:38.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:38.359020+0000) 2022-09-08T07:24:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:38 smithi055 ceph-mon[97033]: pgmap v19011: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:38 smithi059 ceph-mon[97450]: pgmap v19011: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:39.451 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:39.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:39.359193+0000) 2022-09-08T07:24:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:39.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:39.449068+0000) 2022-09-08T07:24:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:40.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:40.449230+0000) 2022-09-08T07:24:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:40.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:40.359371+0000) 2022-09-08T07:24:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:40 smithi055 ceph-mon[97033]: pgmap v19012: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:40 smithi059 ceph-mon[97450]: pgmap v19012: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:41.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:41.449396+0000) 2022-09-08T07:24:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:41.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:41.359543+0000) 2022-09-08T07:24:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:42.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:42.449654+0000) 2022-09-08T07:24:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:42.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:42.359745+0000) 2022-09-08T07:24:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:42 smithi055 ceph-mon[97033]: pgmap v19013: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:24:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:42 smithi059 ceph-mon[97450]: pgmap v19013: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:24:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:43.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:43.359945+0000) 2022-09-08T07:24:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:43.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:43.449866+0000) 2022-09-08T07:24:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:44.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:44.450098+0000) 2022-09-08T07:24:44.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:44.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:44.360131+0000) 2022-09-08T07:24:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:44 smithi055 ceph-mon[97033]: pgmap v19014: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:44 smithi059 ceph-mon[97450]: pgmap v19014: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:45.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:45.450267+0000) 2022-09-08T07:24:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:45.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:45.360301+0000) 2022-09-08T07:24:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:46.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:46.450467+0000) 2022-09-08T07:24:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:46.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:46.360454+0000) 2022-09-08T07:24:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:46 smithi055 ceph-mon[97033]: pgmap v19015: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:24:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:46 smithi059 ceph-mon[97450]: pgmap v19015: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:24:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:47.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:47.450674+0000) 2022-09-08T07:24:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:47.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:47.360629+0000) 2022-09-08T07:24:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:48.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:48.450845+0000) 2022-09-08T07:24:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:48.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:48.360848+0000) 2022-09-08T07:24:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:48 smithi055 ceph-mon[97033]: pgmap v19016: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:48 smithi059 ceph-mon[97450]: pgmap v19016: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:49.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:49.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:49.361006+0000) 2022-09-08T07:24:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:49.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:49.451057+0000) 2022-09-08T07:24:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:50.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:50.451268+0000) 2022-09-08T07:24:50.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:50.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:50.361147+0000) 2022-09-08T07:24:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:50 smithi055 ceph-mon[97033]: pgmap v19017: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:50 smithi059 ceph-mon[97450]: pgmap v19017: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:51.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:51.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:51.451471+0000) 2022-09-08T07:24:51.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:51.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:51.361285+0000) 2022-09-08T07:24:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:52.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:52.451730+0000) 2022-09-08T07:24:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:52.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:52.361423+0000) 2022-09-08T07:24:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:52 smithi055 ceph-mon[97033]: pgmap v19018: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:24:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:52 smithi059 ceph-mon[97450]: pgmap v19018: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:24:53.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:24:52.798+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:24:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:53.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:53.451962+0000) 2022-09-08T07:24:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:53.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:53.361613+0000) 2022-09-08T07:24:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:54.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:54.452195+0000) 2022-09-08T07:24:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:54.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:54.361784+0000) 2022-09-08T07:24:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:54 smithi055 ceph-mon[97033]: pgmap v19019: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:54 smithi059 ceph-mon[97450]: pgmap v19019: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:55.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:55.452425+0000) 2022-09-08T07:24:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:55.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:55.361917+0000) 2022-09-08T07:24:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:56.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:56.452699+0000) 2022-09-08T07:24:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:56.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:56.362074+0000) 2022-09-08T07:24:56.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:56 smithi055 ceph-mon[97033]: pgmap v19020: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:24:57.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:56 smithi059 ceph-mon[97450]: pgmap v19020: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:24:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:57.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:57.362292+0000) 2022-09-08T07:24:57.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:57.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:57.452929+0000) 2022-09-08T07:24:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:58.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:58.453151+0000) 2022-09-08T07:24:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:58.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:58.362551+0000) 2022-09-08T07:24:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:24:58 smithi055 ceph-mon[97033]: pgmap v19021: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:24:58 smithi059 ceph-mon[97450]: pgmap v19021: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:24:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:24:59.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:59.453303+0000) 2022-09-08T07:24:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:24:59.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:24:59.362696+0000) 2022-09-08T07:25:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:00.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:00.453551+0000) 2022-09-08T07:25:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:00.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:00.362880+0000) 2022-09-08T07:25:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:00 smithi055 ceph-mon[97033]: pgmap v19022: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:25:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:00 smithi059 ceph-mon[97450]: pgmap v19022: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:25:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:01.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:01.453840+0000) 2022-09-08T07:25:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:01.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:01.363103+0000) 2022-09-08T07:25:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:02.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:02.454065+0000) 2022-09-08T07:25:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:02.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:02.363329+0000) 2022-09-08T07:25:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:02 smithi055 ceph-mon[97033]: pgmap v19023: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:25:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:02 smithi059 ceph-mon[97450]: pgmap v19023: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:25:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:03.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:03.454238+0000) 2022-09-08T07:25:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:03.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:03.363464+0000) 2022-09-08T07:25:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:04.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:04.454470+0000) 2022-09-08T07:25:04.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:04.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:04.363628+0000) 2022-09-08T07:25:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:04 smithi059 ceph-mon[97450]: pgmap v19024: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:25:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:04 smithi055 ceph-mon[97033]: pgmap v19024: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:25:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:05.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:05.454726+0000) 2022-09-08T07:25:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:05.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:05.363809+0000) 2022-09-08T07:25:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:06.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:06.454947+0000) 2022-09-08T07:25:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:06.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:06.363960+0000) 2022-09-08T07:25:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:06 smithi059 ceph-mon[97450]: pgmap v19025: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:25:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:25:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:06 smithi055 ceph-mon[97033]: pgmap v19025: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:25:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:25:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:07.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:07.455091+0000) 2022-09-08T07:25:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:07.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:07.364110+0000) 2022-09-08T07:25:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:25:08.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:25:07.813+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:25:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:25:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:08.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:08.455297+0000) 2022-09-08T07:25:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:08.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:08.364302+0000) 2022-09-08T07:25:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:08 smithi059 ceph-mon[97450]: pgmap v19026: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:25:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:08 smithi055 ceph-mon[97033]: pgmap v19026: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:25:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:09.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:09.455508+0000) 2022-09-08T07:25:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:09.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:09.364492+0000) 2022-09-08T07:25:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:10.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:10.455760+0000) 2022-09-08T07:25:10.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:10.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:10.364658+0000) 2022-09-08T07:25:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:10 smithi059 ceph-mon[97450]: pgmap v19027: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:25:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:10 smithi059 ceph-mon[97450]: osdmap e1267: 12 total, 11 up, 11 in 2022-09-08T07:25:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:10 smithi055 ceph-mon[97033]: pgmap v19027: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:25:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:10 smithi055 ceph-mon[97033]: osdmap e1267: 12 total, 11 up, 11 in 2022-09-08T07:25:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:11.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:11.456015+0000) 2022-09-08T07:25:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:11.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:11.364885+0000) 2022-09-08T07:25:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:12.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:12.365048+0000) 2022-09-08T07:25:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:12.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:12.456268+0000) 2022-09-08T07:25:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:12 smithi059 ceph-mon[97450]: pgmap v19029: 129 pgs: 8 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 120 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:25:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:12 smithi055 ceph-mon[97033]: pgmap v19029: 129 pgs: 8 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 120 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:25:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:13.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:13.456494+0000) 2022-09-08T07:25:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:13.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:13.365208+0000) 2022-09-08T07:25:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:14.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:14.456744+0000) 2022-09-08T07:25:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:14.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:14.365374+0000) 2022-09-08T07:25:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:14 smithi059 ceph-mon[97450]: pgmap v19030: 129 pgs: 8 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 120 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:25:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:14 smithi055 ceph-mon[97033]: pgmap v19030: 129 pgs: 8 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 120 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:25:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:15.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:15.456943+0000) 2022-09-08T07:25:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:15.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:15.365535+0000) 2022-09-08T07:25:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:16.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:16.457117+0000) 2022-09-08T07:25:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:16.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:16.365709+0000) 2022-09-08T07:25:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:16 smithi059 ceph-mon[97450]: pgmap v19031: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:25:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:16 smithi055 ceph-mon[97033]: pgmap v19031: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:25:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:17.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:17.457326+0000) 2022-09-08T07:25:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:17.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:17.365868+0000) 2022-09-08T07:25:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:18.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:18.366047+0000) 2022-09-08T07:25:18.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:18.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:18.457600+0000) 2022-09-08T07:25:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:18 smithi055 ceph-mon[97033]: pgmap v19032: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:25:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:18 smithi055 ceph-mon[97033]: osdmap e1268: 12 total, 11 up, 11 in 2022-09-08T07:25:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:18 smithi059 ceph-mon[97450]: pgmap v19032: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:25:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:18 smithi059 ceph-mon[97450]: osdmap e1268: 12 total, 11 up, 11 in 2022-09-08T07:25:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:19.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:19.457894+0000) 2022-09-08T07:25:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:19.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:19.366223+0000) 2022-09-08T07:25:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:20.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:20.458130+0000) 2022-09-08T07:25:20.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:20.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:20.366444+0000) 2022-09-08T07:25:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:20 smithi055 ceph-mon[97033]: pgmap v19034: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-08T07:25:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:20 smithi059 ceph-mon[97450]: pgmap v19034: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 219 B/s wr, 6 op/s 2022-09-08T07:25:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:21.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:21.458327+0000) 2022-09-08T07:25:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:21.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:21.366711+0000) 2022-09-08T07:25:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:22.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:22.458465+0000) 2022-09-08T07:25:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:22.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:22.366960+0000) 2022-09-08T07:25:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:22 smithi055 ceph-mon[97033]: pgmap v19035: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:25:22.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:25:22.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:25:22.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:25:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:22 smithi059 ceph-mon[97450]: pgmap v19035: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:25:22.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:25:22.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:25:22.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:25:23.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:25:22.830+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:25:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:23.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:23.458757+0000) 2022-09-08T07:25:23.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:23.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:23.367099+0000) 2022-09-08T07:25:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:24.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:24.458969+0000) 2022-09-08T07:25:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:24.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:24.367248+0000) 2022-09-08T07:25:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:24 smithi055 ceph-mon[97033]: pgmap v19036: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:25:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:24 smithi059 ceph-mon[97450]: pgmap v19036: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:25:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:25.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:25.459220+0000) 2022-09-08T07:25:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:25.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:25.367469+0000) 2022-09-08T07:25:26.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:26.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:26.459434+0000) 2022-09-08T07:25:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:26.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:26.367662+0000) 2022-09-08T07:25:26.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:26 smithi055 ceph-mon[97033]: pgmap v19037: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:25:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:26 smithi059 ceph-mon[97450]: pgmap v19037: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:25:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:27.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:27.459654+0000) 2022-09-08T07:25:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:27.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:27.367900+0000) 2022-09-08T07:25:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:28.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:28.459856+0000) 2022-09-08T07:25:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:28.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:28.368121+0000) 2022-09-08T07:25:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:28 smithi055 ceph-mon[97033]: pgmap v19038: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:25:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:28 smithi059 ceph-mon[97450]: pgmap v19038: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:25:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:29.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:29.460091+0000) 2022-09-08T07:25:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:29.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:29.368270+0000) 2022-09-08T07:25:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:30.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:30.460299+0000) 2022-09-08T07:25:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:30.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:30.368404+0000) 2022-09-08T07:25:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:30 smithi055 ceph-mon[97033]: pgmap v19039: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:25:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:30 smithi059 ceph-mon[97450]: pgmap v19039: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:25:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:31.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:31.460476+0000) 2022-09-08T07:25:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:31.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:31.368663+0000) 2022-09-08T07:25:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:32.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:32.460697+0000) 2022-09-08T07:25:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:32.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:32.368813+0000) 2022-09-08T07:25:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:32 smithi055 ceph-mon[97033]: pgmap v19040: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:25:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:32 smithi059 ceph-mon[97450]: pgmap v19040: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:25:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:33.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:33.460919+0000) 2022-09-08T07:25:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:33.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:33.368955+0000) 2022-09-08T07:25:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:34.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:34.461182+0000) 2022-09-08T07:25:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:34.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:34.369120+0000) 2022-09-08T07:25:34.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:34 smithi055 ceph-mon[97033]: pgmap v19041: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:25:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:34 smithi059 ceph-mon[97450]: pgmap v19041: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:25:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:35.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:35.461393+0000) 2022-09-08T07:25:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:35.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:35.369334+0000) 2022-09-08T07:25:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:36.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:36.461516+0000) 2022-09-08T07:25:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:36.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:36.369574+0000) 2022-09-08T07:25:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:36 smithi055 ceph-mon[97033]: pgmap v19042: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:25:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:36 smithi059 ceph-mon[97450]: pgmap v19042: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:25:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:37.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:37.461760+0000) 2022-09-08T07:25:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:37.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:37.369752+0000) 2022-09-08T07:25:38.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:25:37.831+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:25:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:38.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:38.369907+0000) 2022-09-08T07:25:38.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:38.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:38.461964+0000) 2022-09-08T07:25:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:38 smithi055 ceph-mon[97033]: pgmap v19043: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:25:39.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:38 smithi059 ceph-mon[97450]: pgmap v19043: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:25:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:39.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:39.462203+0000) 2022-09-08T07:25:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:39.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:39.370093+0000) 2022-09-08T07:25:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:40.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:40.462431+0000) 2022-09-08T07:25:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:40.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:40.370277+0000) 2022-09-08T07:25:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:40 smithi055 ceph-mon[97033]: pgmap v19044: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:25:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:40 smithi059 ceph-mon[97450]: pgmap v19044: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:25:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:41.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:41.462712+0000) 2022-09-08T07:25:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:41.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:41.370505+0000) 2022-09-08T07:25:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:42.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:42.462901+0000) 2022-09-08T07:25:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:42.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:42.370752+0000) 2022-09-08T07:25:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:42 smithi055 ceph-mon[97033]: pgmap v19045: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:25:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:42 smithi059 ceph-mon[97450]: pgmap v19045: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:25:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:43.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:43.463062+0000) 2022-09-08T07:25:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:43.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:43.370978+0000) 2022-09-08T07:25:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:44.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:44.463281+0000) 2022-09-08T07:25:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:44.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:44.371202+0000) 2022-09-08T07:25:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:44 smithi055 ceph-mon[97033]: pgmap v19046: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:25:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:44 smithi059 ceph-mon[97450]: pgmap v19046: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:25:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:45.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:45.463511+0000) 2022-09-08T07:25:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:45.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:45.371416+0000) 2022-09-08T07:25:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:46.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:46.463738+0000) 2022-09-08T07:25:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:46.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:46.371596+0000) 2022-09-08T07:25:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:46 smithi055 ceph-mon[97033]: pgmap v19047: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:25:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:46 smithi059 ceph-mon[97450]: pgmap v19047: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:25:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:47.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:47.463963+0000) 2022-09-08T07:25:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:47.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:47.371758+0000) 2022-09-08T07:25:48.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:48.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:48.371935+0000) 2022-09-08T07:25:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:48.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:48.464189+0000) 2022-09-08T07:25:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:48 smithi055 ceph-mon[97033]: pgmap v19048: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:25:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:48 smithi059 ceph-mon[97450]: pgmap v19048: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:25:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:49.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:49.464423+0000) 2022-09-08T07:25:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:49.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:49.372100+0000) 2022-09-08T07:25:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:50.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:50.464597+0000) 2022-09-08T07:25:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:50.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:50.372260+0000) 2022-09-08T07:25:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:50 smithi055 ceph-mon[97033]: pgmap v19049: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:25:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:50 smithi059 ceph-mon[97450]: pgmap v19049: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:25:51.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:51.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:51.372494+0000) 2022-09-08T07:25:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:51.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:51.464842+0000) 2022-09-08T07:25:52.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:52.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:52.372693+0000) 2022-09-08T07:25:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:52.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:52.465037+0000) 2022-09-08T07:25:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:52 smithi055 ceph-mon[97033]: pgmap v19050: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:25:52.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:25:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:52 smithi059 ceph-mon[97450]: pgmap v19050: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:25:53.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:25:53.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:25:52.832+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:25:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:53.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:53.465234+0000) 2022-09-08T07:25:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:53.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:53.372872+0000) 2022-09-08T07:25:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:53 smithi059 ceph-mon[97450]: pgmap v19051: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:25:54.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:53 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T07:25:54.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:53 smithi055 ceph-mon[97033]: pgmap v19051: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:25:54.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:53 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T07:25:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:54.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:54.465416+0000) 2022-09-08T07:25:54.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:54.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:54.373038+0000) 2022-09-08T07:25:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:55.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:55.465672+0000) 2022-09-08T07:25:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:55.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:55.373256+0000) 2022-09-08T07:25:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:55 smithi059 ceph-mon[97450]: pgmap v19052: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:25:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:55 smithi055 ceph-mon[97033]: pgmap v19052: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:25:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:56.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:56.465897+0000) 2022-09-08T07:25:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:56.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:56.373430+0000) 2022-09-08T07:25:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:57.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:57.466105+0000) 2022-09-08T07:25:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:57.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:57.373669+0000) 2022-09-08T07:25:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:57 smithi059 ceph-mon[97450]: pgmap v19053: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:25:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:57 smithi055 ceph-mon[97033]: pgmap v19053: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:25:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:58.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:58.466318+0000) 2022-09-08T07:25:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:58.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:58.373847+0000) 2022-09-08T07:25:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:25:59.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:59.466560+0000) 2022-09-08T07:25:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:25:59.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:25:59.374031+0000) 2022-09-08T07:26:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:25:59 smithi059 ceph-mon[97450]: pgmap v19054: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:26:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:25:59 smithi055 ceph-mon[97033]: pgmap v19054: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:26:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:00.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:00.374377+0000) 2022-09-08T07:26:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:00.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:00.466817+0000) 2022-09-08T07:26:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:01.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:01.466987+0000) 2022-09-08T07:26:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:01.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:01.374552+0000) 2022-09-08T07:26:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:01 smithi059 ceph-mon[97450]: pgmap v19055: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T07:26:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:01 smithi055 ceph-mon[97033]: pgmap v19055: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T07:26:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:02.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:02.467165+0000) 2022-09-08T07:26:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:02.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:02.374792+0000) 2022-09-08T07:26:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:03.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:03.467331+0000) 2022-09-08T07:26:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:03.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:03.374960+0000) 2022-09-08T07:26:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:03 smithi059 ceph-mon[97450]: pgmap v19056: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:26:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:03 smithi055 ceph-mon[97033]: pgmap v19056: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:26:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:04.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:04.467587+0000) 2022-09-08T07:26:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:04.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:04.375114+0000) 2022-09-08T07:26:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:05.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:05.467800+0000) 2022-09-08T07:26:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:05.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:05.375259+0000) 2022-09-08T07:26:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:05 smithi059 ceph-mon[97450]: pgmap v19057: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:05 smithi055 ceph-mon[97033]: pgmap v19057: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:06.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:06.468053+0000) 2022-09-08T07:26:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:06.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:06.375459+0000) 2022-09-08T07:26:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:26:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:26:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:07.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:07.468310+0000) 2022-09-08T07:26:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:07.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:07.375687+0000) 2022-09-08T07:26:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:07 smithi059 ceph-mon[97450]: pgmap v19058: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:26:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:26:08.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:26:07.847+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:26:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:07 smithi055 ceph-mon[97033]: pgmap v19058: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:26:08.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:26:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:08.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:08.468546+0000) 2022-09-08T07:26:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:08.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:08.375889+0000) 2022-09-08T07:26:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:09.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:09.468697+0000) 2022-09-08T07:26:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:09.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:09.376072+0000) 2022-09-08T07:26:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:09 smithi059 ceph-mon[97450]: pgmap v19059: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:26:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:09 smithi055 ceph-mon[97033]: pgmap v19059: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:26:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:10.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:10.468860+0000) 2022-09-08T07:26:10.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:10.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:10.376228+0000) 2022-09-08T07:26:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:10 smithi055 ceph-mon[97033]: osdmap e1269: 12 total, 11 up, 11 in 2022-09-08T07:26:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:10 smithi059 ceph-mon[97450]: osdmap e1269: 12 total, 11 up, 11 in 2022-09-08T07:26:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:11.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:11.469090+0000) 2022-09-08T07:26:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:11.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:11.376442+0000) 2022-09-08T07:26:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:11 smithi055 ceph-mon[97033]: pgmap v19061: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:26:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:11 smithi059 ceph-mon[97450]: pgmap v19061: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:26:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:12.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:12.469337+0000) 2022-09-08T07:26:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:12.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:12.376570+0000) 2022-09-08T07:26:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:13.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:13.469543+0000) 2022-09-08T07:26:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:13.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:13.376739+0000) 2022-09-08T07:26:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:13 smithi055 ceph-mon[97033]: pgmap v19062: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:26:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:13 smithi059 ceph-mon[97450]: pgmap v19062: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:26:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:14.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:14.469716+0000) 2022-09-08T07:26:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:14.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:14.376984+0000) 2022-09-08T07:26:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:15.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:15.469948+0000) 2022-09-08T07:26:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:15.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:15.377231+0000) 2022-09-08T07:26:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:15 smithi055 ceph-mon[97033]: pgmap v19063: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:26:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:15 smithi059 ceph-mon[97450]: pgmap v19063: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:26:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:16.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:16.470202+0000) 2022-09-08T07:26:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:16.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:16.377404+0000) 2022-09-08T07:26:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:17.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:17.470480+0000) 2022-09-08T07:26:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:17.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:17.377568+0000) 2022-09-08T07:26:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:17 smithi055 ceph-mon[97033]: pgmap v19064: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:26:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:17 smithi055 ceph-mon[97033]: osdmap e1270: 12 total, 11 up, 11 in 2022-09-08T07:26:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:17 smithi059 ceph-mon[97450]: pgmap v19064: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:26:18.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:17 smithi059 ceph-mon[97450]: osdmap e1270: 12 total, 11 up, 11 in 2022-09-08T07:26:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:18.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:18.470710+0000) 2022-09-08T07:26:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:18.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:18.377803+0000) 2022-09-08T07:26:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:19.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:19.470852+0000) 2022-09-08T07:26:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:19.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:19.377954+0000) 2022-09-08T07:26:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:19 smithi055 ceph-mon[97033]: pgmap v19066: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 233 B/s wr, 5 op/s 2022-09-08T07:26:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:19 smithi059 ceph-mon[97450]: pgmap v19066: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 233 B/s wr, 5 op/s 2022-09-08T07:26:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:20.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:20.471059+0000) 2022-09-08T07:26:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:20.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:20.378105+0000) 2022-09-08T07:26:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:21.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:21.471290+0000) 2022-09-08T07:26:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:21.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:21.378259+0000) 2022-09-08T07:26:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:21 smithi055 ceph-mon[97033]: pgmap v19067: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:26:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:21 smithi059 ceph-mon[97450]: pgmap v19067: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:26:22.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:22.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:22.378495+0000) 2022-09-08T07:26:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:22.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:22.471536+0000) 2022-09-08T07:26:23.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:26:22.863+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:26:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:23.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:23.378687+0000) 2022-09-08T07:26:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:23.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:23.471796+0000) 2022-09-08T07:26:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:23 smithi055 ceph-mon[97033]: pgmap v19068: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:26:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:23 smithi059 ceph-mon[97450]: pgmap v19068: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:26:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:24.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:24.471951+0000) 2022-09-08T07:26:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:24.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:24.378863+0000) 2022-09-08T07:26:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:25.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:25.472177+0000) 2022-09-08T07:26:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:25.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:25.379039+0000) 2022-09-08T07:26:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:25 smithi055 ceph-mon[97033]: pgmap v19069: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:26:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:25 smithi059 ceph-mon[97450]: pgmap v19069: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:26:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:26.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:26.472384+0000) 2022-09-08T07:26:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:26.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:26.379250+0000) 2022-09-08T07:26:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:27.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:27.472616+0000) 2022-09-08T07:26:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:27.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:27.379411+0000) 2022-09-08T07:26:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:27 smithi055 ceph-mon[97033]: pgmap v19070: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:26:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:27 smithi059 ceph-mon[97450]: pgmap v19070: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:26:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:28.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:28.379616+0000) 2022-09-08T07:26:28.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:28.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:28.472859+0000) 2022-09-08T07:26:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:29.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:29.473108+0000) 2022-09-08T07:26:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:29.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:29.379909+0000) 2022-09-08T07:26:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:29 smithi055 ceph-mon[97033]: pgmap v19071: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:26:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:29 smithi059 ceph-mon[97450]: pgmap v19071: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:26:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:30.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:30.473348+0000) 2022-09-08T07:26:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:30.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:30.380088+0000) 2022-09-08T07:26:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:31.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:31.473588+0000) 2022-09-08T07:26:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:31.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:31.380256+0000) 2022-09-08T07:26:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:31 smithi055 ceph-mon[97033]: pgmap v19072: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:26:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:31 smithi059 ceph-mon[97450]: pgmap v19072: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:26:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:32.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:32.473809+0000) 2022-09-08T07:26:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:32.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:32.380417+0000) 2022-09-08T07:26:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:33.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:33.474069+0000) 2022-09-08T07:26:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:33.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:33.380701+0000) 2022-09-08T07:26:34.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:33 smithi055 ceph-mon[97033]: pgmap v19073: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:33 smithi059 ceph-mon[97450]: pgmap v19073: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:34.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:34.474330+0000) 2022-09-08T07:26:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:34.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:34.380901+0000) 2022-09-08T07:26:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:35.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:35.474616+0000) 2022-09-08T07:26:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:35.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:35.381084+0000) 2022-09-08T07:26:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:35 smithi055 ceph-mon[97033]: pgmap v19074: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:35 smithi059 ceph-mon[97450]: pgmap v19074: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:36.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:36.474836+0000) 2022-09-08T07:26:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:36.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:36.381236+0000) 2022-09-08T07:26:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:37.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:37.475056+0000) 2022-09-08T07:26:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:37.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:37.381381+0000) 2022-09-08T07:26:38.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:26:37.873+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:26:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:37 smithi055 ceph-mon[97033]: pgmap v19075: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:26:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:37 smithi059 ceph-mon[97450]: pgmap v19075: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:26:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:38.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:38.381524+0000) 2022-09-08T07:26:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:38.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:38.475256+0000) 2022-09-08T07:26:39.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:39.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:39.381671+0000) 2022-09-08T07:26:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:39.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:39.475377+0000) 2022-09-08T07:26:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:39 smithi055 ceph-mon[97033]: pgmap v19076: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:39 smithi059 ceph-mon[97450]: pgmap v19076: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:40.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:40.475610+0000) 2022-09-08T07:26:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:40.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:40.381869+0000) 2022-09-08T07:26:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:41.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:41.475820+0000) 2022-09-08T07:26:41.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:41.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:41.382127+0000) 2022-09-08T07:26:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:41 smithi055 ceph-mon[97033]: pgmap v19077: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:26:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:41 smithi059 ceph-mon[97450]: pgmap v19077: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:26:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:42.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:42.476067+0000) 2022-09-08T07:26:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:42.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:42.382285+0000) 2022-09-08T07:26:43.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:43.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:43.382507+0000) 2022-09-08T07:26:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:43.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:43.476363+0000) 2022-09-08T07:26:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:43 smithi055 ceph-mon[97033]: pgmap v19078: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:43 smithi059 ceph-mon[97450]: pgmap v19078: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:44.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:44.382778+0000) 2022-09-08T07:26:44.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:44.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:44.476620+0000) 2022-09-08T07:26:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:45.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:45.382920+0000) 2022-09-08T07:26:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:45.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:45.476905+0000) 2022-09-08T07:26:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:45 smithi055 ceph-mon[97033]: pgmap v19079: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:45 smithi059 ceph-mon[97450]: pgmap v19079: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:46.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:46.477156+0000) 2022-09-08T07:26:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:46.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:46.383087+0000) 2022-09-08T07:26:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:47.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:47.477385+0000) 2022-09-08T07:26:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:47.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:47.383280+0000) 2022-09-08T07:26:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:47 smithi055 ceph-mon[97033]: pgmap v19080: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:26:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:47 smithi059 ceph-mon[97450]: pgmap v19080: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:26:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:48.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:48.477620+0000) 2022-09-08T07:26:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:48.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:48.383453+0000) 2022-09-08T07:26:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:49.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:49.477808+0000) 2022-09-08T07:26:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:49.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:49.383683+0000) 2022-09-08T07:26:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:49 smithi055 ceph-mon[97033]: pgmap v19081: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:49 smithi059 ceph-mon[97450]: pgmap v19081: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:50.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:50.478015+0000) 2022-09-08T07:26:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:50.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:50.383821+0000) 2022-09-08T07:26:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:51.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:51.478195+0000) 2022-09-08T07:26:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:51.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:51.383968+0000) 2022-09-08T07:26:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:51 smithi055 ceph-mon[97033]: pgmap v19082: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:26:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:51 smithi059 ceph-mon[97450]: pgmap v19082: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:26:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:52.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:52.384189+0000) 2022-09-08T07:26:52.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:52.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:52.478379+0000) 2022-09-08T07:26:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:26:52.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:26:52.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:52 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:26:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:26:53.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:26:53.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:52 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:26:53.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:26:52.885+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:26:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:53.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:53.478607+0000) 2022-09-08T07:26:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:53.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:53.384457+0000) 2022-09-08T07:26:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:53 smithi055 ceph-mon[97033]: pgmap v19083: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:54.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:54.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:53 smithi059 ceph-mon[97450]: pgmap v19083: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:54.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:54.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:54.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:54.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:54.478877+0000) 2022-09-08T07:26:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:54.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:54.384662+0000) 2022-09-08T07:26:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:55.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:55.479022+0000) 2022-09-08T07:26:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:55.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:55.384807+0000) 2022-09-08T07:26:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:55 smithi055 ceph-mon[97033]: pgmap v19084: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:56.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:56.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:55 smithi059 ceph-mon[97450]: pgmap v19084: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:56.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:56.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:56.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:56.479246+0000) 2022-09-08T07:26:56.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:56.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:56.384943+0000) 2022-09-08T07:26:57.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:57.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:57.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:57.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:57.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:56 smithi055 ceph-mon[97033]: pgmap v19085: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:26:57.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:57.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:57.256 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:57.256 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:57.257 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:57.257 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:57.258 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:56 smithi059 ceph-mon[97450]: pgmap v19085: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:26:57.258 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:57.258 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:57.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:57.479516+0000) 2022-09-08T07:26:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:57.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:57.385178+0000) 2022-09-08T07:26:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:58.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:58.479767+0000) 2022-09-08T07:26:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:58.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:58.385413+0000) 2022-09-08T07:26:59.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:26:59.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:59.480006+0000) 2022-09-08T07:26:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:26:59.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:26:59.385559+0000) 2022-09-08T07:26:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:59 smithi055 ceph-mon[97033]: pgmap v19086: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:26:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:26:59.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:26:59.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:26:59.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:26:59.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:26:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:27:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:59 smithi059 ceph-mon[97450]: pgmap v19086: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:27:00.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:27:00.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:27:00.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:27:00.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:27:00.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:27:00.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:26:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:27:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:00.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:00.480245+0000) 2022-09-08T07:27:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:00.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:00.385689+0000) 2022-09-08T07:27:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:00 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T07:27:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:00 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 129.4M 2022-09-08T07:27:01.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:00 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 135743836: error parsing value: Value '135743836' is below minimum 939524096 2022-09-08T07:27:01.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:00 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T07:27:01.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:27:01.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:27:01.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:27:01.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:27:01.234 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:27:01.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:27:01.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:00 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T07:27:01.236 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:00 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T07:27:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:00 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T07:27:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:00 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 129.4M 2022-09-08T07:27:01.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:00 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 135743836: error parsing value: Value '135743836' is below minimum 939524096 2022-09-08T07:27:01.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:00 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T07:27:01.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:27:01.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:27:01.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:27:01.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:27:01.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:27:01.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:27:01.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:00 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T07:27:01.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:00 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T07:27:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:01.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:01.480477+0000) 2022-09-08T07:27:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:01.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:01.385820+0000) 2022-09-08T07:27:02.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:01 smithi055 ceph-mon[97033]: pgmap v19087: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:27:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:01 smithi059 ceph-mon[97450]: pgmap v19087: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:27:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:02.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:02.480731+0000) 2022-09-08T07:27:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:02.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:02.385955+0000) 2022-09-08T07:27:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:03.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:03.480920+0000) 2022-09-08T07:27:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:03.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:03.386107+0000) 2022-09-08T07:27:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:03 smithi055 ceph-mon[97033]: pgmap v19088: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:27:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:03 smithi059 ceph-mon[97450]: pgmap v19088: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:27:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:04.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:04.481158+0000) 2022-09-08T07:27:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:04.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:04.386333+0000) 2022-09-08T07:27:05.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:05.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:05.481402+0000) 2022-09-08T07:27:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:05.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:05.386487+0000) 2022-09-08T07:27:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:05 smithi055 ceph-mon[97033]: pgmap v19089: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:27:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:05 smithi059 ceph-mon[97450]: pgmap v19089: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:27:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:06.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:06.481570+0000) 2022-09-08T07:27:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:06.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:06.386654+0000) 2022-09-08T07:27:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:27:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:27:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:27:07.307 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:27:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:07.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:07.386895+0000) 2022-09-08T07:27:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:07.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:07.481767+0000) 2022-09-08T07:27:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:07 smithi055 ceph-mon[97033]: pgmap v19090: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:27:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:07 smithi059 ceph-mon[97450]: pgmap v19090: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:27:08.389 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:27:07.901+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:27:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:08.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:08.481975+0000) 2022-09-08T07:27:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:08.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:08.387060+0000) 2022-09-08T07:27:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:08 smithi055 ceph-mon[97033]: pgmap v19091: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:27:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:08 smithi059 ceph-mon[97450]: pgmap v19091: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:27:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:09.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:09.482244+0000) 2022-09-08T07:27:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:09.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:09.387284+0000) 2022-09-08T07:27:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:10.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:10.482494+0000) 2022-09-08T07:27:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:10.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:10.387507+0000) 2022-09-08T07:27:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:10 smithi055 ceph-mon[97033]: osdmap e1271: 12 total, 11 up, 11 in 2022-09-08T07:27:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:10 smithi055 ceph-mon[97033]: pgmap v19093: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:27:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:10 smithi059 ceph-mon[97450]: osdmap e1271: 12 total, 11 up, 11 in 2022-09-08T07:27:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:10 smithi059 ceph-mon[97450]: pgmap v19093: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:27:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:11.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:11.482755+0000) 2022-09-08T07:27:11.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:11.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:11.387770+0000) 2022-09-08T07:27:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:12.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:12.483006+0000) 2022-09-08T07:27:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:12.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:12.387997+0000) 2022-09-08T07:27:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:13.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:13.483166+0000) 2022-09-08T07:27:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:13.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:13.388196+0000) 2022-09-08T07:27:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:13 smithi055 ceph-mon[97033]: pgmap v19094: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:27:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:13 smithi059 ceph-mon[97450]: pgmap v19094: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:27:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:14.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:14.483367+0000) 2022-09-08T07:27:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:14.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:14.388390+0000) 2022-09-08T07:27:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:15.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:15.483510+0000) 2022-09-08T07:27:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:15.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:15.388576+0000) 2022-09-08T07:27:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:15 smithi055 ceph-mon[97033]: pgmap v19095: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:27:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:15 smithi059 ceph-mon[97450]: pgmap v19095: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:27:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:16.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:16.483750+0000) 2022-09-08T07:27:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:16.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:16.388698+0000) 2022-09-08T07:27:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:17.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:17.483969+0000) 2022-09-08T07:27:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:17.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:17.388855+0000) 2022-09-08T07:27:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:17 smithi055 ceph-mon[97033]: pgmap v19096: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:27:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:17 smithi059 ceph-mon[97450]: pgmap v19096: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:27:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:18.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:18.484136+0000) 2022-09-08T07:27:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:18.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:18.388992+0000) 2022-09-08T07:27:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:18 smithi055 ceph-mon[97033]: osdmap e1272: 12 total, 11 up, 11 in 2022-09-08T07:27:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:18 smithi059 ceph-mon[97450]: osdmap e1272: 12 total, 11 up, 11 in 2022-09-08T07:27:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:19.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:19.484343+0000) 2022-09-08T07:27:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:19.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:19.389148+0000) 2022-09-08T07:27:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:19 smithi055 ceph-mon[97033]: pgmap v19098: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 118 B/s wr, 3 op/s 2022-09-08T07:27:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:19 smithi059 ceph-mon[97450]: pgmap v19098: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 118 B/s wr, 3 op/s 2022-09-08T07:27:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:20.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:20.484589+0000) 2022-09-08T07:27:20.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:20.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:20.389366+0000) 2022-09-08T07:27:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:21.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:21.484797+0000) 2022-09-08T07:27:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:21.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:21.389592+0000) 2022-09-08T07:27:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:21 smithi059 ceph-mon[97450]: pgmap v19099: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:27:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:21 smithi055 ceph-mon[97033]: pgmap v19099: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:27:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:22.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:22.484949+0000) 2022-09-08T07:27:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:22.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:22.389743+0000) 2022-09-08T07:27:23.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:27:22.910+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:27:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:23.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:23.389894+0000) 2022-09-08T07:27:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:23.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:23.485142+0000) 2022-09-08T07:27:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:23 smithi055 ceph-mon[97033]: pgmap v19100: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:27:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:23 smithi055 ceph-mon[97033]: pgmap v19101: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T07:27:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:23 smithi055 ceph-mon[97033]: pgmap v19102: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:27:24.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:27:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:23 smithi059 ceph-mon[97450]: pgmap v19100: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:27:24.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:23 smithi059 ceph-mon[97450]: pgmap v19101: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T07:27:24.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:23 smithi059 ceph-mon[97450]: pgmap v19102: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:27:24.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:27:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:24.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:24.485368+0000) 2022-09-08T07:27:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:24.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:24.390037+0000) 2022-09-08T07:27:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:25.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:25.485602+0000) 2022-09-08T07:27:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:25.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:25.390201+0000) 2022-09-08T07:27:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:25 smithi055 ceph-mon[97033]: pgmap v19103: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T07:27:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:25 smithi059 ceph-mon[97450]: pgmap v19103: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T07:27:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:26.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:26.485811+0000) 2022-09-08T07:27:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:26.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:26.390332+0000) 2022-09-08T07:27:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:27.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:27.485938+0000) 2022-09-08T07:27:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:27.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:27.390547+0000) 2022-09-08T07:27:28.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:27 smithi055 ceph-mon[97033]: pgmap v19104: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:27:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:27 smithi059 ceph-mon[97450]: pgmap v19104: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:27:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:28.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:28.486077+0000) 2022-09-08T07:27:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:28.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:28.390771+0000) 2022-09-08T07:27:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:29.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:29.390927+0000) 2022-09-08T07:27:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:29.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:29.486295+0000) 2022-09-08T07:27:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:29 smithi055 ceph-mon[97033]: pgmap v19105: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:27:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:29 smithi059 ceph-mon[97450]: pgmap v19105: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:27:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:30.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:30.486520+0000) 2022-09-08T07:27:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:30.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:30.391134+0000) 2022-09-08T07:27:31.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:31.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:31.391320+0000) 2022-09-08T07:27:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:31.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:31.486728+0000) 2022-09-08T07:27:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:31 smithi055 ceph-mon[97033]: pgmap v19106: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:27:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:31 smithi059 ceph-mon[97450]: pgmap v19106: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:27:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:32.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:32.486862+0000) 2022-09-08T07:27:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:32.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:32.391480+0000) 2022-09-08T07:27:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:33.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:33.487032+0000) 2022-09-08T07:27:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:33.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:33.391693+0000) 2022-09-08T07:27:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:33 smithi055 ceph-mon[97033]: pgmap v19107: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:27:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:33 smithi059 ceph-mon[97450]: pgmap v19107: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:27:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:34.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:34.487164+0000) 2022-09-08T07:27:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:34.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:34.391849+0000) 2022-09-08T07:27:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:35.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:35.487407+0000) 2022-09-08T07:27:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:35.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:35.391999+0000) 2022-09-08T07:27:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:35 smithi055 ceph-mon[97033]: pgmap v19108: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:27:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:35 smithi059 ceph-mon[97450]: pgmap v19108: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:27:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:36.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:36.487667+0000) 2022-09-08T07:27:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:36.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:36.392129+0000) 2022-09-08T07:27:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:37.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:37.487826+0000) 2022-09-08T07:27:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:37.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:37.392348+0000) 2022-09-08T07:27:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:37 smithi055 ceph-mon[97033]: pgmap v19109: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T07:27:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:37 smithi059 ceph-mon[97450]: pgmap v19109: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T07:27:38.395 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:27:37.912+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:27:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:38.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:38.487979+0000) 2022-09-08T07:27:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:38.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:38.392570+0000) 2022-09-08T07:27:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:38 smithi055 ceph-mon[97033]: pgmap v19110: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:27:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:38 smithi059 ceph-mon[97450]: pgmap v19110: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:27:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:39.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:39.488148+0000) 2022-09-08T07:27:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:39.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:39.392765+0000) 2022-09-08T07:27:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:40.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:40.488318+0000) 2022-09-08T07:27:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:40.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:40.392921+0000) 2022-09-08T07:27:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:41.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:41.488468+0000) 2022-09-08T07:27:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:41.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:41.393098+0000) 2022-09-08T07:27:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:41 smithi055 ceph-mon[97033]: pgmap v19111: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:27:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:41 smithi059 ceph-mon[97450]: pgmap v19111: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:27:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:42.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:42.488677+0000) 2022-09-08T07:27:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:42.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:42.393322+0000) 2022-09-08T07:27:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:43.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:43.488896+0000) 2022-09-08T07:27:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:43.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:43.393537+0000) 2022-09-08T07:27:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:43 smithi055 ceph-mon[97033]: pgmap v19112: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:27:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:43 smithi059 ceph-mon[97450]: pgmap v19112: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:27:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:44.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:44.489091+0000) 2022-09-08T07:27:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:44.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:44.393787+0000) 2022-09-08T07:27:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:45.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:45.489263+0000) 2022-09-08T07:27:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:45.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:45.394000+0000) 2022-09-08T07:27:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:45 smithi055 ceph-mon[97033]: pgmap v19113: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:27:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:45 smithi059 ceph-mon[97450]: pgmap v19113: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:27:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:46.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:46.489510+0000) 2022-09-08T07:27:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:46.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:46.394173+0000) 2022-09-08T07:27:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:46 smithi055 ceph-mon[97033]: pgmap v19114: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T07:27:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:46 smithi059 ceph-mon[97450]: pgmap v19114: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T07:27:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:47.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:47.489707+0000) 2022-09-08T07:27:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:47.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:47.394375+0000) 2022-09-08T07:27:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:48.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:48.489866+0000) 2022-09-08T07:27:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:48.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:48.394610+0000) 2022-09-08T07:27:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:49.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:49.490097+0000) 2022-09-08T07:27:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:49.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:49.394804+0000) 2022-09-08T07:27:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:49 smithi055 ceph-mon[97033]: pgmap v19115: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:27:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:49 smithi059 ceph-mon[97450]: pgmap v19115: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:27:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:50.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:50.490268+0000) 2022-09-08T07:27:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:50.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:50.394933+0000) 2022-09-08T07:27:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:50 smithi055 ceph-mon[97033]: pgmap v19116: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:27:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:50 smithi059 ceph-mon[97450]: pgmap v19116: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:27:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:51.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:51.490498+0000) 2022-09-08T07:27:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:51.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:51.395100+0000) 2022-09-08T07:27:52.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:52.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:52.490704+0000) 2022-09-08T07:27:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:52.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:52.395236+0000) 2022-09-08T07:27:53.397 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:27:52.924+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:27:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:53.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:53.490902+0000) 2022-09-08T07:27:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:53.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:53.395413+0000) 2022-09-08T07:27:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:53 smithi055 ceph-mon[97033]: pgmap v19117: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:27:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:53 smithi059 ceph-mon[97450]: pgmap v19117: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:27:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:54.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:54.491094+0000) 2022-09-08T07:27:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:54.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:54.395630+0000) 2022-09-08T07:27:55.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:55.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:55.395818+0000) 2022-09-08T07:27:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:55.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:55.491220+0000) 2022-09-08T07:27:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:55 smithi055 ceph-mon[97033]: pgmap v19118: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:27:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:55 smithi059 ceph-mon[97450]: pgmap v19118: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:27:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:56.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:56.491415+0000) 2022-09-08T07:27:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:56.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:56.395992+0000) 2022-09-08T07:27:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:56 smithi055 ceph-mon[97033]: pgmap v19119: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T07:27:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:56 smithi059 ceph-mon[97450]: pgmap v19119: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T07:27:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:57.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:57.491574+0000) 2022-09-08T07:27:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:57.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:57.396123+0000) 2022-09-08T07:27:58.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:58.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:58.491794+0000) 2022-09-08T07:27:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:58.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:58.396311+0000) 2022-09-08T07:27:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:27:59.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:59.492052+0000) 2022-09-08T07:27:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:27:59.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:27:59.396555+0000) 2022-09-08T07:28:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:27:59 smithi055 ceph-mon[97033]: pgmap v19120: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:28:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:27:59 smithi059 ceph-mon[97450]: pgmap v19120: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:28:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:00.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:00.492205+0000) 2022-09-08T07:28:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:00.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:00.396712+0000) 2022-09-08T07:28:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:00 smithi055 ceph-mon[97033]: pgmap v19121: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:28:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:00 smithi059 ceph-mon[97450]: pgmap v19121: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:28:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:01.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:01.492328+0000) 2022-09-08T07:28:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:01.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:01.396851+0000) 2022-09-08T07:28:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:02.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:02.492544+0000) 2022-09-08T07:28:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:02.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:02.397029+0000) 2022-09-08T07:28:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:03.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:03.492787+0000) 2022-09-08T07:28:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:03.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:03.397196+0000) 2022-09-08T07:28:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:03 smithi055 ceph-mon[97033]: pgmap v19122: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:28:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:03 smithi059 ceph-mon[97450]: pgmap v19122: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:28:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:04.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:04.492974+0000) 2022-09-08T07:28:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:04.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:04.397443+0000) 2022-09-08T07:28:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:04 smithi055 ceph-mon[97033]: pgmap v19123: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 8 op/s 2022-09-08T07:28:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:04 smithi059 ceph-mon[97450]: pgmap v19123: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 8 op/s 2022-09-08T07:28:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:05.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:05.493138+0000) 2022-09-08T07:28:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:05.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:05.397591+0000) 2022-09-08T07:28:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:28:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:28:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:06.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:06.493312+0000) 2022-09-08T07:28:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:06.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:06.397813+0000) 2022-09-08T07:28:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:07 smithi055 ceph-mon[97033]: pgmap v19124: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T07:28:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:28:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:07 smithi059 ceph-mon[97450]: pgmap v19124: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T07:28:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:28:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:07.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:07.493534+0000) 2022-09-08T07:28:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:07.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:07.398057+0000) 2022-09-08T07:28:08.400 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:28:07.927+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:28:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:08.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:08.398303+0000) 2022-09-08T07:28:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:08.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:08.493700+0000) 2022-09-08T07:28:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:08 smithi055 ceph-mon[97033]: pgmap v19125: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:28:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:08 smithi059 ceph-mon[97450]: pgmap v19125: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:28:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:09.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:09.493927+0000) 2022-09-08T07:28:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:09.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:09.398472+0000) 2022-09-08T07:28:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:10.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:10.494094+0000) 2022-09-08T07:28:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:10.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:10.398652+0000) 2022-09-08T07:28:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:10 smithi055 ceph-mon[97033]: osdmap e1273: 12 total, 11 up, 11 in 2022-09-08T07:28:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:10 smithi055 ceph-mon[97033]: pgmap v19127: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:28:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:10 smithi059 ceph-mon[97450]: osdmap e1273: 12 total, 11 up, 11 in 2022-09-08T07:28:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:10 smithi059 ceph-mon[97450]: pgmap v19127: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:28:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:11.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:11.494283+0000) 2022-09-08T07:28:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:11.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:11.398797+0000) 2022-09-08T07:28:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:12.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:12.494458+0000) 2022-09-08T07:28:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:12.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:12.398929+0000) 2022-09-08T07:28:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:13.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:13.494712+0000) 2022-09-08T07:28:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:13.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:13.399095+0000) 2022-09-08T07:28:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:13 smithi055 ceph-mon[97033]: pgmap v19128: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:28:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:13 smithi059 ceph-mon[97450]: pgmap v19128: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:28:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:14.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:14.494897+0000) 2022-09-08T07:28:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:14.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:14.399316+0000) 2022-09-08T07:28:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:14 smithi055 ceph-mon[97033]: pgmap v19129: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-08T07:28:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:14 smithi059 ceph-mon[97450]: pgmap v19129: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-08T07:28:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:15.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:15.495086+0000) 2022-09-08T07:28:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:15.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:15.399431+0000) 2022-09-08T07:28:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:16.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:16.495237+0000) 2022-09-08T07:28:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:16.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:16.399674+0000) 2022-09-08T07:28:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:16 smithi055 ceph-mon[97033]: pgmap v19130: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:28:17.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:16 smithi059 ceph-mon[97450]: pgmap v19130: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:28:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:17.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:17.495486+0000) 2022-09-08T07:28:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:17.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:17.399841+0000) 2022-09-08T07:28:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:18.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:18.495748+0000) 2022-09-08T07:28:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:18.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:18.400004+0000) 2022-09-08T07:28:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:18 smithi055 ceph-mon[97033]: osdmap e1274: 12 total, 11 up, 11 in 2022-09-08T07:28:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:18 smithi059 ceph-mon[97450]: osdmap e1274: 12 total, 11 up, 11 in 2022-09-08T07:28:19.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:19.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:19.400168+0000) 2022-09-08T07:28:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:19.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:19.495886+0000) 2022-09-08T07:28:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:19 smithi055 ceph-mon[97033]: pgmap v19132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 115 B/s wr, 3 op/s 2022-09-08T07:28:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:19 smithi059 ceph-mon[97450]: pgmap v19132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 115 B/s wr, 3 op/s 2022-09-08T07:28:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:20.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:20.400296+0000) 2022-09-08T07:28:20.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:20.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:20.496018+0000) 2022-09-08T07:28:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:20 smithi055 ceph-mon[97033]: pgmap v19133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:28:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:20 smithi059 ceph-mon[97450]: pgmap v19133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:28:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:21.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:21.496253+0000) 2022-09-08T07:28:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:21.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:21.400529+0000) 2022-09-08T07:28:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:22.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:22.496478+0000) 2022-09-08T07:28:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:22.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:22.400725+0000) 2022-09-08T07:28:23.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:28:22.943+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:28:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:23.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:23.400890+0000) 2022-09-08T07:28:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:23.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:23.496736+0000) 2022-09-08T07:28:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:23 smithi059 ceph-mon[97450]: pgmap v19134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:28:23.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:28:23.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:28:23.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:28:23.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:28:23.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:28:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:23 smithi055 ceph-mon[97033]: pgmap v19134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:28:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:28:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:28:24.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:28:24.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:28:24.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:28:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:24.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:24.496960+0000) 2022-09-08T07:28:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:24.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:24.401058+0000) 2022-09-08T07:28:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:24 smithi055 ceph-mon[97033]: pgmap v19135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:28:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:24 smithi059 ceph-mon[97450]: pgmap v19135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:28:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:25.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:25.497240+0000) 2022-09-08T07:28:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:25.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:25.401199+0000) 2022-09-08T07:28:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:26.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:26.497453+0000) 2022-09-08T07:28:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:26.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:26.401378+0000) 2022-09-08T07:28:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:26 smithi055 ceph-mon[97033]: pgmap v19136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:28:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:26 smithi059 ceph-mon[97450]: pgmap v19136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:28:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:27.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:27.497707+0000) 2022-09-08T07:28:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:27.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:27.401528+0000) 2022-09-08T07:28:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:28.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:28.497892+0000) 2022-09-08T07:28:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:28.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:28.401763+0000) 2022-09-08T07:28:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:28 smithi055 ceph-mon[97033]: pgmap v19137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:28:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:28 smithi059 ceph-mon[97450]: pgmap v19137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:28:29.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:29.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:29.498112+0000) 2022-09-08T07:28:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:29.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:29.401895+0000) 2022-09-08T07:28:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:30.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:30.498311+0000) 2022-09-08T07:28:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:30.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:30.402105+0000) 2022-09-08T07:28:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:30 smithi055 ceph-mon[97033]: pgmap v19138: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:28:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:30 smithi059 ceph-mon[97450]: pgmap v19138: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:28:31.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:31.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:31.498534+0000) 2022-09-08T07:28:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:31.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:31.402258+0000) 2022-09-08T07:28:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:32.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:32.498798+0000) 2022-09-08T07:28:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:32.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:32.402385+0000) 2022-09-08T07:28:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:33.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:33.499042+0000) 2022-09-08T07:28:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:33.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:33.402584+0000) 2022-09-08T07:28:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:33 smithi055 ceph-mon[97033]: pgmap v19139: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:28:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:33 smithi059 ceph-mon[97450]: pgmap v19139: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:28:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:34.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:34.499281+0000) 2022-09-08T07:28:34.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:34.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:34.402715+0000) 2022-09-08T07:28:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:34 smithi055 ceph-mon[97033]: pgmap v19140: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:28:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:34 smithi059 ceph-mon[97450]: pgmap v19140: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:28:35.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:35.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:35.499485+0000) 2022-09-08T07:28:35.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:35.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:35.402928+0000) 2022-09-08T07:28:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:36.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:36.499726+0000) 2022-09-08T07:28:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:36.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:36.403144+0000) 2022-09-08T07:28:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:36 smithi055 ceph-mon[97033]: pgmap v19141: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T07:28:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:36 smithi059 ceph-mon[97450]: pgmap v19141: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T07:28:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:37.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:37.499983+0000) 2022-09-08T07:28:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:37.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:37.403408+0000) 2022-09-08T07:28:38.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:28:37.951+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:28:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:38.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:38.500229+0000) 2022-09-08T07:28:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:38.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:38.403564+0000) 2022-09-08T07:28:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:38 smithi055 ceph-mon[97033]: pgmap v19142: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:28:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:38 smithi059 ceph-mon[97450]: pgmap v19142: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:28:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:39.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:39.500465+0000) 2022-09-08T07:28:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:39.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:39.403782+0000) 2022-09-08T07:28:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:40.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:40.500686+0000) 2022-09-08T07:28:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:40.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:40.403951+0000) 2022-09-08T07:28:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:40 smithi055 ceph-mon[97033]: pgmap v19143: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:28:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:40 smithi059 ceph-mon[97450]: pgmap v19143: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:28:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:41.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:41.500929+0000) 2022-09-08T07:28:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:41.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:41.404180+0000) 2022-09-08T07:28:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:42.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:42.501120+0000) 2022-09-08T07:28:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:42.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:42.404387+0000) 2022-09-08T07:28:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:43.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:43.404546+0000) 2022-09-08T07:28:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:43.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:43.501345+0000) 2022-09-08T07:28:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:43 smithi059 ceph-mon[97450]: pgmap v19144: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:28:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:43 smithi055 ceph-mon[97033]: pgmap v19144: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:28:44.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:44.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:44.404758+0000) 2022-09-08T07:28:44.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:44.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:44.501566+0000) 2022-09-08T07:28:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:44 smithi055 ceph-mon[97033]: pgmap v19145: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:28:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:44 smithi059 ceph-mon[97450]: pgmap v19145: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:28:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:45.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:45.501808+0000) 2022-09-08T07:28:45.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:45.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:45.405000+0000) 2022-09-08T07:28:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:46.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:46.502018+0000) 2022-09-08T07:28:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:46.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:46.405218+0000) 2022-09-08T07:28:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:46 smithi055 ceph-mon[97033]: pgmap v19146: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T07:28:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:46 smithi059 ceph-mon[97450]: pgmap v19146: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T07:28:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:47.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:47.502203+0000) 2022-09-08T07:28:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:47.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:47.405356+0000) 2022-09-08T07:28:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:48.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:48.502456+0000) 2022-09-08T07:28:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:48.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:48.405568+0000) 2022-09-08T07:28:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:48 smithi055 ceph-mon[97033]: pgmap v19147: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:28:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:48 smithi059 ceph-mon[97450]: pgmap v19147: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:28:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:49.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:49.502754+0000) 2022-09-08T07:28:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:49.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:49.405720+0000) 2022-09-08T07:28:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:50.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:50.502902+0000) 2022-09-08T07:28:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:50.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:50.405935+0000) 2022-09-08T07:28:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:50 smithi055 ceph-mon[97033]: pgmap v19148: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:28:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:50 smithi059 ceph-mon[97450]: pgmap v19148: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:28:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:51.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:51.503051+0000) 2022-09-08T07:28:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:51.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:51.406115+0000) 2022-09-08T07:28:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:52.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:52.503256+0000) 2022-09-08T07:28:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:52.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:52.406358+0000) 2022-09-08T07:28:53.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:28:52.962+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:28:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:53.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:53.503423+0000) 2022-09-08T07:28:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:53.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:53.406527+0000) 2022-09-08T07:28:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:53 smithi059 ceph-mon[97450]: pgmap v19149: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:28:54.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:28:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:53 smithi055 ceph-mon[97033]: pgmap v19149: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:28:54.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:28:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:54.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:54.503670+0000) 2022-09-08T07:28:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:54.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:54.406725+0000) 2022-09-08T07:28:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:54 smithi059 ceph-mon[97450]: pgmap v19150: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:28:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:54 smithi059 ceph-mon[97450]: pgmap v19151: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-08T07:28:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:54 smithi055 ceph-mon[97033]: pgmap v19150: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:28:55.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:54 smithi055 ceph-mon[97033]: pgmap v19151: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-08T07:28:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:55.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:55.503851+0000) 2022-09-08T07:28:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:55.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:55.406942+0000) 2022-09-08T07:28:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:56.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:56.504024+0000) 2022-09-08T07:28:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:56.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:56.407149+0000) 2022-09-08T07:28:57.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:56 smithi059 ceph-mon[97450]: pgmap v19152: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:28:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:56 smithi055 ceph-mon[97033]: pgmap v19152: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:28:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:57.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:57.504238+0000) 2022-09-08T07:28:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:57.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:57.407296+0000) 2022-09-08T07:28:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:58.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:58.504459+0000) 2022-09-08T07:28:58.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:58.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:58.407487+0000) 2022-09-08T07:28:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:28:58 smithi059 ceph-mon[97450]: pgmap v19153: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:28:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:28:58 smithi055 ceph-mon[97033]: pgmap v19153: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:28:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:28:59.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:59.504720+0000) 2022-09-08T07:28:59.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:28:59.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:28:59.407722+0000) 2022-09-08T07:29:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:00.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:00.504915+0000) 2022-09-08T07:29:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:00.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:00.407928+0000) 2022-09-08T07:29:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:00 smithi059 ceph-mon[97450]: pgmap v19154: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:29:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:00 smithi055 ceph-mon[97033]: pgmap v19154: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:29:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:01.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:01.505163+0000) 2022-09-08T07:29:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:01.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:01.408112+0000) 2022-09-08T07:29:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:02.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:02.505377+0000) 2022-09-08T07:29:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:02.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:02.408290+0000) 2022-09-08T07:29:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:02 smithi059 ceph-mon[97450]: pgmap v19155: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:29:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:02 smithi055 ceph-mon[97033]: pgmap v19155: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:29:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:03.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:03.505667+0000) 2022-09-08T07:29:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:03.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:03.408491+0000) 2022-09-08T07:29:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:04.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:04.408748+0000) 2022-09-08T07:29:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:04.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:04.505825+0000) 2022-09-08T07:29:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:04 smithi059 ceph-mon[97450]: pgmap v19156: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:29:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:04 smithi055 ceph-mon[97033]: pgmap v19156: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:29:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:05.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:05.505967+0000) 2022-09-08T07:29:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:05.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:05.408950+0000) 2022-09-08T07:29:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:06.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:06.506169+0000) 2022-09-08T07:29:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:06.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:06.409106+0000) 2022-09-08T07:29:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:06 smithi059 ceph-mon[97450]: pgmap v19157: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:29:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:29:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:06 smithi055 ceph-mon[97033]: pgmap v19157: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:29:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:29:07.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:07.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:07.506307+0000) 2022-09-08T07:29:07.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:07.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:07.409272+0000) 2022-09-08T07:29:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:29:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:29:08.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:29:07.979+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:29:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:08.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:08.506458+0000) 2022-09-08T07:29:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:08.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:08.409434+0000) 2022-09-08T07:29:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:08 smithi059 ceph-mon[97450]: pgmap v19158: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:29:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:08 smithi055 ceph-mon[97033]: pgmap v19158: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:29:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:09.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:09.506743+0000) 2022-09-08T07:29:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:09.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:09.409603+0000) 2022-09-08T07:29:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:10.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:10.506970+0000) 2022-09-08T07:29:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:10.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:10.409765+0000) 2022-09-08T07:29:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:10 smithi059 ceph-mon[97450]: pgmap v19159: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:29:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:10 smithi059 ceph-mon[97450]: osdmap e1275: 12 total, 11 up, 11 in 2022-09-08T07:29:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:10 smithi055 ceph-mon[97033]: pgmap v19159: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:29:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:10 smithi055 ceph-mon[97033]: osdmap e1275: 12 total, 11 up, 11 in 2022-09-08T07:29:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:11.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:11.507158+0000) 2022-09-08T07:29:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:11.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:11.409935+0000) 2022-09-08T07:29:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:12.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:12.507368+0000) 2022-09-08T07:29:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:12.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:12.410083+0000) 2022-09-08T07:29:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:12 smithi055 ceph-mon[97033]: pgmap v19161: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:29:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:12 smithi059 ceph-mon[97450]: pgmap v19161: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:29:13.510 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:13.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:13.410324+0000) 2022-09-08T07:29:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:13.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:13.507573+0000) 2022-09-08T07:29:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:14.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:14.507810+0000) 2022-09-08T07:29:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:14.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:14.410519+0000) 2022-09-08T07:29:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:14 smithi055 ceph-mon[97033]: pgmap v19162: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:29:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:14 smithi059 ceph-mon[97450]: pgmap v19162: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:29:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:15.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:15.410696+0000) 2022-09-08T07:29:15.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:15.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:15.508038+0000) 2022-09-08T07:29:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:16.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:16.508234+0000) 2022-09-08T07:29:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:16.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:16.410900+0000) 2022-09-08T07:29:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:16 smithi055 ceph-mon[97033]: pgmap v19163: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:29:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:16 smithi059 ceph-mon[97450]: pgmap v19163: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:29:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:17.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:17.411057+0000) 2022-09-08T07:29:17.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:17.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:17.508468+0000) 2022-09-08T07:29:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:18.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:18.508708+0000) 2022-09-08T07:29:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:18.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:18.411230+0000) 2022-09-08T07:29:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:18 smithi059 ceph-mon[97450]: pgmap v19164: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:29:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:18 smithi059 ceph-mon[97450]: osdmap e1276: 12 total, 11 up, 11 in 2022-09-08T07:29:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:18 smithi055 ceph-mon[97033]: pgmap v19164: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:29:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:18 smithi055 ceph-mon[97033]: osdmap e1276: 12 total, 11 up, 11 in 2022-09-08T07:29:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:19.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:19.508959+0000) 2022-09-08T07:29:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:19.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:19.411389+0000) 2022-09-08T07:29:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:20.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:20.411608+0000) 2022-09-08T07:29:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:20.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:20.509218+0000) 2022-09-08T07:29:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:20 smithi059 ceph-mon[97450]: pgmap v19166: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 218 B/s wr, 6 op/s 2022-09-08T07:29:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:20 smithi055 ceph-mon[97033]: pgmap v19166: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 218 B/s wr, 6 op/s 2022-09-08T07:29:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:21.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:21.509454+0000) 2022-09-08T07:29:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:21.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:21.411867+0000) 2022-09-08T07:29:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:22.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:22.509690+0000) 2022-09-08T07:29:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:22.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:22.412019+0000) 2022-09-08T07:29:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:22 smithi059 ceph-mon[97450]: pgmap v19167: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:29:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:22 smithi055 ceph-mon[97033]: pgmap v19167: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:29:23.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:29:22.990+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:29:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:23.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:23.509931+0000) 2022-09-08T07:29:23.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:23.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:23.412173+0000) 2022-09-08T07:29:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:24.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:24.510158+0000) 2022-09-08T07:29:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:24.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:24.412379+0000) 2022-09-08T07:29:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:24 smithi059 ceph-mon[97450]: pgmap v19168: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:29:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:24 smithi055 ceph-mon[97033]: pgmap v19168: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:29:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:25.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:25.510377+0000) 2022-09-08T07:29:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:25.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:25.412525+0000) 2022-09-08T07:29:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:26.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:26.510609+0000) 2022-09-08T07:29:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:26.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:26.412683+0000) 2022-09-08T07:29:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:26 smithi059 ceph-mon[97450]: pgmap v19169: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:29:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:26 smithi055 ceph-mon[97033]: pgmap v19169: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:29:27.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:27.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:27.510847+0000) 2022-09-08T07:29:27.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:27.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:27.412875+0000) 2022-09-08T07:29:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:28.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:28.511023+0000) 2022-09-08T07:29:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:28.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:28.413032+0000) 2022-09-08T07:29:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:28 smithi059 ceph-mon[97450]: pgmap v19170: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:29:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:28 smithi055 ceph-mon[97033]: pgmap v19170: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:29:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:29.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:29.511206+0000) 2022-09-08T07:29:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:29.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:29.413193+0000) 2022-09-08T07:29:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:30.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:30.511450+0000) 2022-09-08T07:29:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:30.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:30.413448+0000) 2022-09-08T07:29:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:30 smithi059 ceph-mon[97450]: pgmap v19171: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:29:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:30 smithi055 ceph-mon[97033]: pgmap v19171: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:29:31.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:31.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:31.511718+0000) 2022-09-08T07:29:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:31.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:31.413686+0000) 2022-09-08T07:29:32.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:32.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:32.511919+0000) 2022-09-08T07:29:32.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:32.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:32.413837+0000) 2022-09-08T07:29:33.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:32 smithi059 ceph-mon[97450]: pgmap v19172: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:29:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:32 smithi055 ceph-mon[97033]: pgmap v19172: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:29:33.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:33.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:33.512152+0000) 2022-09-08T07:29:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:33.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:33.413986+0000) 2022-09-08T07:29:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:34.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:34.512344+0000) 2022-09-08T07:29:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:34.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:34.414218+0000) 2022-09-08T07:29:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:34 smithi059 ceph-mon[97450]: pgmap v19173: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:34 smithi055 ceph-mon[97033]: pgmap v19173: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:35.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:35.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:35.512534+0000) 2022-09-08T07:29:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:35.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:35.414406+0000) 2022-09-08T07:29:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:36.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:36.512694+0000) 2022-09-08T07:29:36.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:36.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:36.414619+0000) 2022-09-08T07:29:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:36 smithi059 ceph-mon[97450]: pgmap v19174: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:29:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:36 smithi055 ceph-mon[97033]: pgmap v19174: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:29:37.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:37.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:37.512921+0000) 2022-09-08T07:29:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:37.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:37.414809+0000) 2022-09-08T07:29:38.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:29:38.001+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:29:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:38.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:38.513169+0000) 2022-09-08T07:29:38.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:38.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:38.415036+0000) 2022-09-08T07:29:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:38 smithi059 ceph-mon[97450]: pgmap v19175: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:38 smithi055 ceph-mon[97033]: pgmap v19175: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:39.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:39.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:39.513365+0000) 2022-09-08T07:29:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:39.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:39.415163+0000) 2022-09-08T07:29:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:40.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:40.513569+0000) 2022-09-08T07:29:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:40.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:40.415418+0000) 2022-09-08T07:29:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:40 smithi059 ceph-mon[97450]: pgmap v19176: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:40 smithi055 ceph-mon[97033]: pgmap v19176: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:41.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:41.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:41.513814+0000) 2022-09-08T07:29:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:41.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:41.415622+0000) 2022-09-08T07:29:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:42.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:42.513985+0000) 2022-09-08T07:29:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:42.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:42.415798+0000) 2022-09-08T07:29:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:42 smithi059 ceph-mon[97450]: pgmap v19177: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:29:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:42 smithi055 ceph-mon[97033]: pgmap v19177: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:29:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:43.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:43.415967+0000) 2022-09-08T07:29:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:43.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:43.514214+0000) 2022-09-08T07:29:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:44.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:44.416163+0000) 2022-09-08T07:29:44.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:44.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:44.514466+0000) 2022-09-08T07:29:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:44 smithi059 ceph-mon[97450]: pgmap v19178: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:44 smithi055 ceph-mon[97033]: pgmap v19178: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:45.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:45.514693+0000) 2022-09-08T07:29:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:45.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:45.416371+0000) 2022-09-08T07:29:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:46.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:46.514894+0000) 2022-09-08T07:29:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:46.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:46.416610+0000) 2022-09-08T07:29:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:46 smithi055 ceph-mon[97033]: pgmap v19179: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:29:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:46 smithi059 ceph-mon[97450]: pgmap v19179: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:29:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:47.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:47.515117+0000) 2022-09-08T07:29:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:47.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:47.416805+0000) 2022-09-08T07:29:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:48.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:48.515353+0000) 2022-09-08T07:29:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:48.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:48.417011+0000) 2022-09-08T07:29:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:48 smithi055 ceph-mon[97033]: pgmap v19180: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:48 smithi059 ceph-mon[97450]: pgmap v19180: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:49.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:49.515589+0000) 2022-09-08T07:29:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:49.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:49.417218+0000) 2022-09-08T07:29:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:50.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:50.515785+0000) 2022-09-08T07:29:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:50.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:50.417446+0000) 2022-09-08T07:29:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:50 smithi055 ceph-mon[97033]: pgmap v19181: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:50 smithi059 ceph-mon[97450]: pgmap v19181: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:51.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:51.515964+0000) 2022-09-08T07:29:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:51.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:51.417599+0000) 2022-09-08T07:29:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:52.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:52.417824+0000) 2022-09-08T07:29:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:52.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:52.516158+0000) 2022-09-08T07:29:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:52 smithi055 ceph-mon[97033]: pgmap v19182: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:29:53.273 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:29:53.018+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:29:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:52 smithi059 ceph-mon[97450]: pgmap v19182: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:29:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:53.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:53.417962+0000) 2022-09-08T07:29:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:53.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:53.516449+0000) 2022-09-08T07:29:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:29:54.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:29:54.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:29:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:29:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:29:54.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:29:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:54.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:54.516718+0000) 2022-09-08T07:29:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:54.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:54.418163+0000) 2022-09-08T07:29:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:54 smithi055 ceph-mon[97033]: pgmap v19183: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:54 smithi059 ceph-mon[97450]: pgmap v19183: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:55.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:55.516954+0000) 2022-09-08T07:29:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:55.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:55.418282+0000) 2022-09-08T07:29:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:56.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:56.517186+0000) 2022-09-08T07:29:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:56.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:56.418510+0000) 2022-09-08T07:29:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:56 smithi055 ceph-mon[97033]: pgmap v19184: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:29:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:56 smithi059 ceph-mon[97450]: pgmap v19184: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:29:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:57.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:57.418784+0000) 2022-09-08T07:29:57.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:57.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:57.517448+0000) 2022-09-08T07:29:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:58.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:58.418981+0000) 2022-09-08T07:29:58.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:58.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:58.517657+0000) 2022-09-08T07:29:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:29:58 smithi055 ceph-mon[97033]: pgmap v19185: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:29:58 smithi059 ceph-mon[97450]: pgmap v19185: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:29:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:29:59.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:59.517858+0000) 2022-09-08T07:29:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:29:59.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:29:59.419204+0000) 2022-09-08T07:30:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:00.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:00.518070+0000) 2022-09-08T07:30:00.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:00.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:00.419401+0000) 2022-09-08T07:30:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:00 smithi055 ceph-mon[97033]: pgmap v19186: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T07:30:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T07:30:01.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T07:30:01.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T07:30:01.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T07:30:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:00 smithi059 ceph-mon[97450]: pgmap v19186: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T07:30:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T07:30:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T07:30:01.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T07:30:01.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T07:30:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:01.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:01.518261+0000) 2022-09-08T07:30:01.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:01.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:01.419536+0000) 2022-09-08T07:30:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:02.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:02.518484+0000) 2022-09-08T07:30:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:02.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:02.419696+0000) 2022-09-08T07:30:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:02 smithi055 ceph-mon[97033]: pgmap v19187: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:30:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:02 smithi059 ceph-mon[97450]: pgmap v19187: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:30:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:03.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:03.518730+0000) 2022-09-08T07:30:03.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:03.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:03.419836+0000) 2022-09-08T07:30:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:04.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:04.518965+0000) 2022-09-08T07:30:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:04.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:04.419994+0000) 2022-09-08T07:30:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:04 smithi055 ceph-mon[97033]: pgmap v19188: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:04 smithi059 ceph-mon[97450]: pgmap v19188: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:05.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:05.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:05.519221+0000) 2022-09-08T07:30:05.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:05.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:05.420168+0000) 2022-09-08T07:30:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:06.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:06.519478+0000) 2022-09-08T07:30:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:06.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:06.420288+0000) 2022-09-08T07:30:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:06 smithi055 ceph-mon[97033]: pgmap v19189: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:30:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:30:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:06 smithi059 ceph-mon[97450]: pgmap v19189: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:30:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:30:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:07.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:07.519720+0000) 2022-09-08T07:30:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:07.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:07.420418+0000) 2022-09-08T07:30:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:30:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:30:08.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:30:08.030+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:30:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:08.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:08.519950+0000) 2022-09-08T07:30:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:08.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:08.420683+0000) 2022-09-08T07:30:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:08 smithi055 ceph-mon[97033]: pgmap v19190: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:30:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:08 smithi059 ceph-mon[97450]: pgmap v19190: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:30:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:09.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:09.520197+0000) 2022-09-08T07:30:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:09.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:09.420862+0000) 2022-09-08T07:30:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:10.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:10.420986+0000) 2022-09-08T07:30:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:10.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:10.520447+0000) 2022-09-08T07:30:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:10 smithi055 ceph-mon[97033]: pgmap v19191: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:30:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:10 smithi055 ceph-mon[97033]: osdmap e1277: 12 total, 11 up, 11 in 2022-09-08T07:30:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:10 smithi059 ceph-mon[97450]: pgmap v19191: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:30:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:10 smithi059 ceph-mon[97450]: osdmap e1277: 12 total, 11 up, 11 in 2022-09-08T07:30:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:11.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:11.520725+0000) 2022-09-08T07:30:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:11.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:11.421116+0000) 2022-09-08T07:30:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:12.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:12.421308+0000) 2022-09-08T07:30:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:12.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:12.520900+0000) 2022-09-08T07:30:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:12 smithi055 ceph-mon[97033]: pgmap v19193: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:30:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:12 smithi059 ceph-mon[97450]: pgmap v19193: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:30:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:13.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:13.521049+0000) 2022-09-08T07:30:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:13.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:13.421579+0000) 2022-09-08T07:30:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:14.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:14.521284+0000) 2022-09-08T07:30:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:14.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:14.421732+0000) 2022-09-08T07:30:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:14 smithi055 ceph-mon[97033]: pgmap v19194: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:30:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:14 smithi059 ceph-mon[97450]: pgmap v19194: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:30:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:15.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:15.521452+0000) 2022-09-08T07:30:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:15.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:15.421948+0000) 2022-09-08T07:30:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:16.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:16.521581+0000) 2022-09-08T07:30:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:16.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:16.422092+0000) 2022-09-08T07:30:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:16 smithi055 ceph-mon[97033]: pgmap v19195: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:30:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:16 smithi059 ceph-mon[97450]: pgmap v19195: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:30:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:17.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:17.521788+0000) 2022-09-08T07:30:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:17.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:17.422245+0000) 2022-09-08T07:30:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:18.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:18.522034+0000) 2022-09-08T07:30:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:18.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:18.422449+0000) 2022-09-08T07:30:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:18 smithi059 ceph-mon[97450]: pgmap v19196: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:30:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:18 smithi059 ceph-mon[97450]: osdmap e1278: 12 total, 11 up, 11 in 2022-09-08T07:30:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:18 smithi055 ceph-mon[97033]: pgmap v19196: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:30:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:18 smithi055 ceph-mon[97033]: osdmap e1278: 12 total, 11 up, 11 in 2022-09-08T07:30:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:19.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:19.522274+0000) 2022-09-08T07:30:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:19.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:19.422621+0000) 2022-09-08T07:30:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:20.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:20.522523+0000) 2022-09-08T07:30:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:20.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:20.422786+0000) 2022-09-08T07:30:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:20 smithi059 ceph-mon[97450]: pgmap v19198: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 219 B/s wr, 5 op/s 2022-09-08T07:30:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:20 smithi055 ceph-mon[97033]: pgmap v19198: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 219 B/s wr, 5 op/s 2022-09-08T07:30:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:21.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:21.522727+0000) 2022-09-08T07:30:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:21.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:21.423034+0000) 2022-09-08T07:30:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:22.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:22.522907+0000) 2022-09-08T07:30:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:22.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:22.423207+0000) 2022-09-08T07:30:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:22 smithi059 ceph-mon[97450]: pgmap v19199: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:30:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:22 smithi055 ceph-mon[97033]: pgmap v19199: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:30:23.380 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:30:23.042+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:30:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:23.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:23.523086+0000) 2022-09-08T07:30:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:23.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:23.423389+0000) 2022-09-08T07:30:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:24 smithi055 ceph-mon[97033]: pgmap v19200: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:30:24.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:24 smithi055 ceph-mon[97033]: pgmap v19201: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:30:24.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:24 smithi055 ceph-mon[97033]: pgmap v19202: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:30:24.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:30:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:24 smithi059 ceph-mon[97450]: pgmap v19200: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:30:24.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:24 smithi059 ceph-mon[97450]: pgmap v19201: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:30:24.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:24 smithi059 ceph-mon[97450]: pgmap v19202: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:30:24.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:30:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:24.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:24.423561+0000) 2022-09-08T07:30:24.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:24.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:24.523323+0000) 2022-09-08T07:30:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:25.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:25.523600+0000) 2022-09-08T07:30:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:25.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:25.423817+0000) 2022-09-08T07:30:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:26 smithi055 ceph-mon[97033]: pgmap v19203: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T07:30:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:26 smithi059 ceph-mon[97450]: pgmap v19203: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T07:30:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:26.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:26.523843+0000) 2022-09-08T07:30:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:26.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:26.423980+0000) 2022-09-08T07:30:27.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:27.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:27.524071+0000) 2022-09-08T07:30:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:27.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:27.424209+0000) 2022-09-08T07:30:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:28.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:28.524317+0000) 2022-09-08T07:30:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:28.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:28.424365+0000) 2022-09-08T07:30:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:28 smithi059 ceph-mon[97450]: pgmap v19204: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:30:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:28 smithi055 ceph-mon[97033]: pgmap v19204: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:30:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:29.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:29.524557+0000) 2022-09-08T07:30:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:29.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:29.424515+0000) 2022-09-08T07:30:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:30.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:30.524786+0000) 2022-09-08T07:30:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:30.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:30.424707+0000) 2022-09-08T07:30:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:30 smithi059 ceph-mon[97450]: pgmap v19205: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:30:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:30 smithi055 ceph-mon[97033]: pgmap v19205: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:30:31.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:31.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:31.525019+0000) 2022-09-08T07:30:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:31.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:31.424912+0000) 2022-09-08T07:30:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:32.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:32.525240+0000) 2022-09-08T07:30:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:32.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:32.425135+0000) 2022-09-08T07:30:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:32 smithi059 ceph-mon[97450]: pgmap v19206: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:30:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:32 smithi055 ceph-mon[97033]: pgmap v19206: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:30:33.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:33.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:33.525466+0000) 2022-09-08T07:30:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:33.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:33.425303+0000) 2022-09-08T07:30:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:34.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:34.525712+0000) 2022-09-08T07:30:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:34.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:34.425446+0000) 2022-09-08T07:30:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:34 smithi059 ceph-mon[97450]: pgmap v19207: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:30:35.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:34 smithi055 ceph-mon[97033]: pgmap v19207: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:30:35.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:35.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:35.525962+0000) 2022-09-08T07:30:35.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:35.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:35.425591+0000) 2022-09-08T07:30:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:36.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:36.526137+0000) 2022-09-08T07:30:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:36.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:36.425738+0000) 2022-09-08T07:30:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:36 smithi059 ceph-mon[97450]: pgmap v19208: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:30:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:36 smithi055 ceph-mon[97033]: pgmap v19208: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:30:37.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:37.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:37.526344+0000) 2022-09-08T07:30:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:37.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:37.425961+0000) 2022-09-08T07:30:38.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:30:38.046+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:30:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:38.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:38.526599+0000) 2022-09-08T07:30:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:38.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:38.426140+0000) 2022-09-08T07:30:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:38 smithi059 ceph-mon[97450]: pgmap v19209: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:38 smithi055 ceph-mon[97033]: pgmap v19209: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:39.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:39.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:39.526822+0000) 2022-09-08T07:30:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:39.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:39.426305+0000) 2022-09-08T07:30:40.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:40.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:40.527035+0000) 2022-09-08T07:30:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:40.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:40.426544+0000) 2022-09-08T07:30:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:40 smithi059 ceph-mon[97450]: pgmap v19210: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:40 smithi055 ceph-mon[97033]: pgmap v19210: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:41.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:41.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:41.527242+0000) 2022-09-08T07:30:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:41.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:41.426730+0000) 2022-09-08T07:30:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:42.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:42.426941+0000) 2022-09-08T07:30:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:42.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:42.527459+0000) 2022-09-08T07:30:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:42 smithi059 ceph-mon[97450]: pgmap v19211: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:30:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:42 smithi055 ceph-mon[97033]: pgmap v19211: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:30:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:43.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:43.527675+0000) 2022-09-08T07:30:43.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:43.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:43.427085+0000) 2022-09-08T07:30:44.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:44.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:44.527842+0000) 2022-09-08T07:30:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:44.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:44.427220+0000) 2022-09-08T07:30:45.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:44 smithi055 ceph-mon[97033]: pgmap v19212: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:44 smithi059 ceph-mon[97450]: pgmap v19212: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:45.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:45.528025+0000) 2022-09-08T07:30:45.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:45.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:45.427456+0000) 2022-09-08T07:30:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:46.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:46.528200+0000) 2022-09-08T07:30:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:46.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:46.427689+0000) 2022-09-08T07:30:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:46 smithi055 ceph-mon[97033]: pgmap v19213: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:30:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:46 smithi059 ceph-mon[97450]: pgmap v19213: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:30:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:47.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:47.528376+0000) 2022-09-08T07:30:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:47.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:47.427832+0000) 2022-09-08T07:30:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:48.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:48.528554+0000) 2022-09-08T07:30:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:48.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:48.428028+0000) 2022-09-08T07:30:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:48 smithi055 ceph-mon[97033]: pgmap v19214: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:48 smithi059 ceph-mon[97450]: pgmap v19214: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:49.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:49.528722+0000) 2022-09-08T07:30:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:49.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:49.428213+0000) 2022-09-08T07:30:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:50.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:50.528890+0000) 2022-09-08T07:30:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:50.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:50.428431+0000) 2022-09-08T07:30:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:50 smithi055 ceph-mon[97033]: pgmap v19215: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:50 smithi059 ceph-mon[97450]: pgmap v19215: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:51.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:51.529119+0000) 2022-09-08T07:30:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:51.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:51.428684+0000) 2022-09-08T07:30:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:52.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:52.529359+0000) 2022-09-08T07:30:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:52.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:52.428880+0000) 2022-09-08T07:30:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:52 smithi055 ceph-mon[97033]: pgmap v19216: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:30:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:52 smithi059 ceph-mon[97450]: pgmap v19216: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:30:53.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:30:53.057+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:30:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:53.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:53.529591+0000) 2022-09-08T07:30:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:53.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:53.429032+0000) 2022-09-08T07:30:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:54.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:54.529862+0000) 2022-09-08T07:30:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:54.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:54.429217+0000) 2022-09-08T07:30:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:54 smithi055 ceph-mon[97033]: pgmap v19217: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:54 smithi059 ceph-mon[97450]: pgmap v19217: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:55.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:55.530077+0000) 2022-09-08T07:30:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:55.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:55.429441+0000) 2022-09-08T07:30:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:56.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:56.530272+0000) 2022-09-08T07:30:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:56.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:56.429675+0000) 2022-09-08T07:30:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:56 smithi055 ceph-mon[97033]: pgmap v19218: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:30:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:56 smithi059 ceph-mon[97450]: pgmap v19218: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:30:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:57.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:57.530455+0000) 2022-09-08T07:30:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:57.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:57.429886+0000) 2022-09-08T07:30:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:58.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:58.530739+0000) 2022-09-08T07:30:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:58.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:58.430078+0000) 2022-09-08T07:30:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:30:58 smithi055 ceph-mon[97033]: pgmap v19219: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:30:58 smithi059 ceph-mon[97450]: pgmap v19219: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:30:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:30:59.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:59.530937+0000) 2022-09-08T07:30:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:30:59.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:30:59.430281+0000) 2022-09-08T07:31:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:00.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:00.531161+0000) 2022-09-08T07:31:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:00.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:00.430419+0000) 2022-09-08T07:31:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:00 smithi055 ceph-mon[97033]: pgmap v19220: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:31:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:00 smithi059 ceph-mon[97450]: pgmap v19220: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:31:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:01.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:01.531376+0000) 2022-09-08T07:31:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:01.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:01.430631+0000) 2022-09-08T07:31:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:02.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:02.531612+0000) 2022-09-08T07:31:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:02.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:02.430898+0000) 2022-09-08T07:31:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:02 smithi055 ceph-mon[97033]: pgmap v19221: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:31:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:02 smithi059 ceph-mon[97450]: pgmap v19221: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:31:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:03.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:03.531782+0000) 2022-09-08T07:31:03.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:03.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:03.431087+0000) 2022-09-08T07:31:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:04.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:04.431228+0000) 2022-09-08T07:31:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:04.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:04.532044+0000) 2022-09-08T07:31:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:04 smithi055 ceph-mon[97033]: pgmap v19222: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:31:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:04 smithi059 ceph-mon[97450]: pgmap v19222: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:31:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:05.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:05.431374+0000) 2022-09-08T07:31:05.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:05.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:05.532298+0000) 2022-09-08T07:31:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:06.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:06.532458+0000) 2022-09-08T07:31:06.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:06.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:06.431587+0000) 2022-09-08T07:31:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:06 smithi055 ceph-mon[97033]: pgmap v19223: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:31:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:31:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:06 smithi059 ceph-mon[97450]: pgmap v19223: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:31:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:31:07.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:07.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:07.532679+0000) 2022-09-08T07:31:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:07.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:07.431861+0000) 2022-09-08T07:31:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:31:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:31:08.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:31:08.066+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:31:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:08.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:08.532821+0000) 2022-09-08T07:31:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:08.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:08.432024+0000) 2022-09-08T07:31:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:08 smithi055 ceph-mon[97033]: pgmap v19224: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:31:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:08 smithi059 ceph-mon[97450]: pgmap v19224: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:31:09.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:09.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:09.432152+0000) 2022-09-08T07:31:09.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:09.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:09.533037+0000) 2022-09-08T07:31:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:10.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:10.533224+0000) 2022-09-08T07:31:10.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:10.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:10.432354+0000) 2022-09-08T07:31:11.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:10 smithi055 ceph-mon[97033]: pgmap v19225: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:31:11.239 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:10 smithi055 ceph-mon[97033]: osdmap e1279: 12 total, 11 up, 11 in 2022-09-08T07:31:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:10 smithi059 ceph-mon[97450]: pgmap v19225: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:31:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:10 smithi059 ceph-mon[97450]: osdmap e1279: 12 total, 11 up, 11 in 2022-09-08T07:31:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:11.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:11.533437+0000) 2022-09-08T07:31:11.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:11.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:11.432491+0000) 2022-09-08T07:31:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:12.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:12.533719+0000) 2022-09-08T07:31:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:12.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:12.432671+0000) 2022-09-08T07:31:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:12 smithi055 ceph-mon[97033]: pgmap v19227: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:31:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:12 smithi059 ceph-mon[97450]: pgmap v19227: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:31:13.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:13.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:13.533973+0000) 2022-09-08T07:31:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:13.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:13.432845+0000) 2022-09-08T07:31:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:14.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:14.534154+0000) 2022-09-08T07:31:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:14.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:14.432986+0000) 2022-09-08T07:31:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:14 smithi055 ceph-mon[97033]: pgmap v19228: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:31:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:14 smithi059 ceph-mon[97450]: pgmap v19228: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:31:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:15.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:15.534374+0000) 2022-09-08T07:31:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:15.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:15.433192+0000) 2022-09-08T07:31:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:16.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:16.433377+0000) 2022-09-08T07:31:16.918 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:16.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:16.534561+0000) 2022-09-08T07:31:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:16 smithi055 ceph-mon[97033]: pgmap v19229: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:31:17.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:16 smithi059 ceph-mon[97450]: pgmap v19229: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:31:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:17.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:17.534759+0000) 2022-09-08T07:31:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:17.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:17.433518+0000) 2022-09-08T07:31:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:18.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:18.534909+0000) 2022-09-08T07:31:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:18.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:18.433712+0000) 2022-09-08T07:31:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:18 smithi059 ceph-mon[97450]: pgmap v19230: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:31:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:18 smithi059 ceph-mon[97450]: osdmap e1280: 12 total, 11 up, 11 in 2022-09-08T07:31:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:18 smithi055 ceph-mon[97033]: pgmap v19230: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:31:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:18 smithi055 ceph-mon[97033]: osdmap e1280: 12 total, 11 up, 11 in 2022-09-08T07:31:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:19.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:19.535037+0000) 2022-09-08T07:31:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:19.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:19.433926+0000) 2022-09-08T07:31:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:20.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:20.434154+0000) 2022-09-08T07:31:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:20.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:20.535273+0000) 2022-09-08T07:31:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:20 smithi059 ceph-mon[97450]: pgmap v19232: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 214 B/s wr, 5 op/s 2022-09-08T07:31:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:20 smithi055 ceph-mon[97033]: pgmap v19232: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 214 B/s wr, 5 op/s 2022-09-08T07:31:21.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:21.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:21.535565+0000) 2022-09-08T07:31:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:21.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:21.434384+0000) 2022-09-08T07:31:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:22.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:22.535800+0000) 2022-09-08T07:31:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:22.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:22.434584+0000) 2022-09-08T07:31:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:22 smithi059 ceph-mon[97450]: pgmap v19233: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:31:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:22 smithi055 ceph-mon[97033]: pgmap v19233: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:31:23.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:31:23.081+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:31:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:23.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:23.535974+0000) 2022-09-08T07:31:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:23.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:23.434722+0000) 2022-09-08T07:31:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:31:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:31:23.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:31:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:31:24.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:31:24.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:31:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:24.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:24.536198+0000) 2022-09-08T07:31:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:24.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:24.434922+0000) 2022-09-08T07:31:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:24 smithi055 ceph-mon[97033]: pgmap v19234: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:31:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:24 smithi059 ceph-mon[97450]: pgmap v19234: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:31:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:25.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:25.536366+0000) 2022-09-08T07:31:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:25.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:25.435122+0000) 2022-09-08T07:31:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:26.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:26.536517+0000) 2022-09-08T07:31:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:26.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:26.435269+0000) 2022-09-08T07:31:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:26 smithi055 ceph-mon[97033]: pgmap v19235: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:31:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:26 smithi059 ceph-mon[97450]: pgmap v19235: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:31:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:27.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:27.536660+0000) 2022-09-08T07:31:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:27.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:27.435468+0000) 2022-09-08T07:31:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:28.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:28.536865+0000) 2022-09-08T07:31:28.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:28.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:28.435713+0000) 2022-09-08T07:31:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:28 smithi055 ceph-mon[97033]: pgmap v19236: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:31:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:28 smithi059 ceph-mon[97450]: pgmap v19236: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:31:29.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:29.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:29.537037+0000) 2022-09-08T07:31:29.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:29.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:29.435915+0000) 2022-09-08T07:31:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:30.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:30.537269+0000) 2022-09-08T07:31:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:30.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:30.436080+0000) 2022-09-08T07:31:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:30 smithi055 ceph-mon[97033]: pgmap v19237: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:31:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:30 smithi059 ceph-mon[97450]: pgmap v19237: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:31:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:31.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:31.537528+0000) 2022-09-08T07:31:31.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:31.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:31.436228+0000) 2022-09-08T07:31:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:32.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:32.436431+0000) 2022-09-08T07:31:32.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:32.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:32.537778+0000) 2022-09-08T07:31:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:32 smithi055 ceph-mon[97033]: pgmap v19238: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:31:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:32 smithi059 ceph-mon[97450]: pgmap v19238: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:31:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:33.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:33.537945+0000) 2022-09-08T07:31:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:33.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:33.436622+0000) 2022-09-08T07:31:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:34.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:34.538185+0000) 2022-09-08T07:31:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:34.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:34.436791+0000) 2022-09-08T07:31:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:34 smithi055 ceph-mon[97033]: pgmap v19239: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:31:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:34 smithi059 ceph-mon[97450]: pgmap v19239: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:31:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:35.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:35.538394+0000) 2022-09-08T07:31:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:35.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:35.436964+0000) 2022-09-08T07:31:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:36.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:36.538621+0000) 2022-09-08T07:31:36.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:36.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:36.437182+0000) 2022-09-08T07:31:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:36 smithi055 ceph-mon[97033]: pgmap v19240: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:31:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:36 smithi059 ceph-mon[97450]: pgmap v19240: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:31:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:37.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:37.538897+0000) 2022-09-08T07:31:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:37.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:37.437400+0000) 2022-09-08T07:31:38.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:31:38.097+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:31:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:38.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:38.539117+0000) 2022-09-08T07:31:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:38.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:38.437541+0000) 2022-09-08T07:31:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:38 smithi055 ceph-mon[97033]: pgmap v19241: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:31:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:38 smithi059 ceph-mon[97450]: pgmap v19241: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:31:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:39.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:39.539344+0000) 2022-09-08T07:31:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:39.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:39.437684+0000) 2022-09-08T07:31:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:40.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:40.539590+0000) 2022-09-08T07:31:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:40.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:40.437823+0000) 2022-09-08T07:31:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:40 smithi055 ceph-mon[97033]: pgmap v19242: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:31:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:40 smithi059 ceph-mon[97450]: pgmap v19242: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:31:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:41.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:41.539826+0000) 2022-09-08T07:31:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:41.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:41.438040+0000) 2022-09-08T07:31:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:42.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:42.540079+0000) 2022-09-08T07:31:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:42.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:42.438198+0000) 2022-09-08T07:31:43.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:42 smithi055 ceph-mon[97033]: pgmap v19243: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:31:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:42 smithi059 ceph-mon[97450]: pgmap v19243: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:31:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:43.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:43.540303+0000) 2022-09-08T07:31:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:43.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:43.438357+0000) 2022-09-08T07:31:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:44.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:44.540541+0000) 2022-09-08T07:31:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:44.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:44.438492+0000) 2022-09-08T07:31:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:44 smithi055 ceph-mon[97033]: pgmap v19244: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:31:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:45.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:45.540793+0000) 2022-09-08T07:31:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:45.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:45.438749+0000) 2022-09-08T07:31:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:46.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:46.438908+0000) 2022-09-08T07:31:46.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:46.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:46.540987+0000) 2022-09-08T07:31:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:46 smithi055 ceph-mon[97033]: pgmap v19245: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:31:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:44 smithi059 ceph-mon[97450]: pgmap v19244: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:31:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:46 smithi059 ceph-mon[97450]: pgmap v19245: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:31:47.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:47.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:47.439118+0000) 2022-09-08T07:31:47.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:47.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:47.541236+0000) 2022-09-08T07:31:48.704 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:48.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:48.541404+0000) 2022-09-08T07:31:48.705 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:48.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:48.439338+0000) 2022-09-08T07:31:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:48 smithi055 ceph-mon[97033]: pgmap v19246: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:31:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:48 smithi059 ceph-mon[97450]: pgmap v19246: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:31:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:49.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:49.541623+0000) 2022-09-08T07:31:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:49.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:49.439552+0000) 2022-09-08T07:31:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:50.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:50.541865+0000) 2022-09-08T07:31:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:50.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:50.439782+0000) 2022-09-08T07:31:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:50 smithi055 ceph-mon[97033]: pgmap v19247: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:31:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:50 smithi059 ceph-mon[97450]: pgmap v19247: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:31:51.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:51.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:51.542037+0000) 2022-09-08T07:31:51.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:51.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:51.439925+0000) 2022-09-08T07:31:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:52.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:52.542246+0000) 2022-09-08T07:31:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:52.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:52.440103+0000) 2022-09-08T07:31:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:52 smithi055 ceph-mon[97033]: pgmap v19248: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:31:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:52 smithi059 ceph-mon[97450]: pgmap v19248: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:31:53.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:31:53.112+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:31:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:53.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:53.542419+0000) 2022-09-08T07:31:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:53.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:53.440310+0000) 2022-09-08T07:31:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:54.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:54.542663+0000) 2022-09-08T07:31:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:54.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:54.440578+0000) 2022-09-08T07:31:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:54 smithi055 ceph-mon[97033]: pgmap v19249: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:31:54.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:54 smithi055 ceph-mon[97033]: pgmap v19250: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:31:54.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:54 smithi055 ceph-mon[97033]: pgmap v19251: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:31:54.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:31:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:54 smithi059 ceph-mon[97450]: pgmap v19249: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:31:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:54 smithi059 ceph-mon[97450]: pgmap v19250: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:31:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:54 smithi059 ceph-mon[97450]: pgmap v19251: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:31:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:31:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:55.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:55.542937+0000) 2022-09-08T07:31:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:55.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:55.440824+0000) 2022-09-08T07:31:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:56.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:56.543061+0000) 2022-09-08T07:31:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:56.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:56.440962+0000) 2022-09-08T07:31:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:56 smithi055 ceph-mon[97033]: pgmap v19252: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:31:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:56 smithi059 ceph-mon[97450]: pgmap v19252: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:31:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:57.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:57.543247+0000) 2022-09-08T07:31:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:57.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:57.441122+0000) 2022-09-08T07:31:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:58.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:58.543456+0000) 2022-09-08T07:31:58.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:58.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:58.441273+0000) 2022-09-08T07:31:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:31:58 smithi059 ceph-mon[97450]: pgmap v19253: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:31:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:31:58 smithi055 ceph-mon[97033]: pgmap v19253: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:31:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:31:59.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:59.543710+0000) 2022-09-08T07:31:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:31:59.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:31:59.441418+0000) 2022-09-08T07:32:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:00.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:00.543892+0000) 2022-09-08T07:32:00.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:00.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:00.441581+0000) 2022-09-08T07:32:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:00 smithi055 ceph-mon[97033]: pgmap v19254: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:32:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:00 smithi059 ceph-mon[97450]: pgmap v19254: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:32:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:01.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:01.544091+0000) 2022-09-08T07:32:01.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:01.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:01.441740+0000) 2022-09-08T07:32:02.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:02.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:02.544289+0000) 2022-09-08T07:32:02.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:02.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:02.441884+0000) 2022-09-08T07:32:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:02 smithi055 ceph-mon[97033]: pgmap v19255: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:32:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:02 smithi059 ceph-mon[97450]: pgmap v19255: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:32:03.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:03.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:03.442092+0000) 2022-09-08T07:32:03.911 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:03.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:03.544497+0000) 2022-09-08T07:32:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:04.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:04.544725+0000) 2022-09-08T07:32:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:04.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:04.442309+0000) 2022-09-08T07:32:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:04 smithi055 ceph-mon[97033]: pgmap v19256: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:32:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:04 smithi059 ceph-mon[97450]: pgmap v19256: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:32:05.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:05.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:05.545004+0000) 2022-09-08T07:32:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:05.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:05.442533+0000) 2022-09-08T07:32:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:06.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:06.545266+0000) 2022-09-08T07:32:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:06.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:06.442774+0000) 2022-09-08T07:32:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:06 smithi055 ceph-mon[97033]: pgmap v19257: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:32:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:32:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:06 smithi059 ceph-mon[97450]: pgmap v19257: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:32:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:32:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:07.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:07.545481+0000) 2022-09-08T07:32:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:07.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:07.442936+0000) 2022-09-08T07:32:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:32:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:32:08.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:32:08.127+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:32:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:08.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:08.545739+0000) 2022-09-08T07:32:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:08.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:08.443148+0000) 2022-09-08T07:32:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:08 smithi055 ceph-mon[97033]: pgmap v19258: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:32:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:08 smithi059 ceph-mon[97450]: pgmap v19258: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:32:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:09.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:09.545943+0000) 2022-09-08T07:32:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:09.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:09.443289+0000) 2022-09-08T07:32:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:10.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:10.546088+0000) 2022-09-08T07:32:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:10.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:10.443551+0000) 2022-09-08T07:32:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:10 smithi055 ceph-mon[97033]: pgmap v19259: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:32:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:10 smithi055 ceph-mon[97033]: osdmap e1281: 12 total, 11 up, 11 in 2022-09-08T07:32:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:10 smithi059 ceph-mon[97450]: pgmap v19259: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:32:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:10 smithi059 ceph-mon[97450]: osdmap e1281: 12 total, 11 up, 11 in 2022-09-08T07:32:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:11.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:11.546322+0000) 2022-09-08T07:32:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:11.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:11.443778+0000) 2022-09-08T07:32:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:12.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:12.546506+0000) 2022-09-08T07:32:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:12.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:12.444023+0000) 2022-09-08T07:32:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:12 smithi055 ceph-mon[97033]: pgmap v19261: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:32:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:12 smithi059 ceph-mon[97450]: pgmap v19261: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:32:13.720 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:13.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:13.444174+0000) 2022-09-08T07:32:13.721 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:13.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:13.546742+0000) 2022-09-08T07:32:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:14.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:14.444315+0000) 2022-09-08T07:32:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:14.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:14.546882+0000) 2022-09-08T07:32:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:14 smithi055 ceph-mon[97033]: pgmap v19262: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:32:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:14 smithi059 ceph-mon[97450]: pgmap v19262: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:32:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:15.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:15.547060+0000) 2022-09-08T07:32:15.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:15.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:15.444517+0000) 2022-09-08T07:32:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:16.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:16.547283+0000) 2022-09-08T07:32:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:16.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:16.444753+0000) 2022-09-08T07:32:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:16 smithi055 ceph-mon[97033]: pgmap v19263: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:32:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:16 smithi059 ceph-mon[97450]: pgmap v19263: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:32:17.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:17.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:17.547528+0000) 2022-09-08T07:32:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:17.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:17.444991+0000) 2022-09-08T07:32:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:18.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:18.547761+0000) 2022-09-08T07:32:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:18.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:18.445259+0000) 2022-09-08T07:32:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:18 smithi055 ceph-mon[97033]: pgmap v19264: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:32:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:18 smithi055 ceph-mon[97033]: osdmap e1282: 12 total, 11 up, 11 in 2022-09-08T07:32:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:18 smithi059 ceph-mon[97450]: pgmap v19264: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:32:19.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:18 smithi059 ceph-mon[97450]: osdmap e1282: 12 total, 11 up, 11 in 2022-09-08T07:32:19.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:19.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:19.547903+0000) 2022-09-08T07:32:19.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:19.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:19.445390+0000) 2022-09-08T07:32:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:20.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:20.548112+0000) 2022-09-08T07:32:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:20.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:20.445648+0000) 2022-09-08T07:32:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:20 smithi055 ceph-mon[97033]: pgmap v19266: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T07:32:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:20 smithi059 ceph-mon[97450]: pgmap v19266: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2022-09-08T07:32:21.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:21.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:21.548259+0000) 2022-09-08T07:32:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:21.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:21.445885+0000) 2022-09-08T07:32:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:22.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:22.548502+0000) 2022-09-08T07:32:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:22.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:22.446069+0000) 2022-09-08T07:32:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:22 smithi055 ceph-mon[97033]: pgmap v19267: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:32:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:22 smithi059 ceph-mon[97450]: pgmap v19267: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:32:23.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:32:23.143+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:32:23.910 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:23.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:23.548685+0000) 2022-09-08T07:32:23.911 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:23.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:23.446256+0000) 2022-09-08T07:32:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:24.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:24.548954+0000) 2022-09-08T07:32:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:24.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:24.446415+0000) 2022-09-08T07:32:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:24 smithi055 ceph-mon[97033]: pgmap v19268: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:32:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:24 smithi059 ceph-mon[97450]: pgmap v19268: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:32:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:25.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:25.549190+0000) 2022-09-08T07:32:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:25.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:25.446565+0000) 2022-09-08T07:32:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:26.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:26.549437+0000) 2022-09-08T07:32:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:26.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:26.446818+0000) 2022-09-08T07:32:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:26 smithi055 ceph-mon[97033]: pgmap v19269: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:32:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:26 smithi059 ceph-mon[97450]: pgmap v19269: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:32:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:27.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:27.549761+0000) 2022-09-08T07:32:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:27.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:27.447061+0000) 2022-09-08T07:32:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:28.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:28.550048+0000) 2022-09-08T07:32:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:28.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:28.447294+0000) 2022-09-08T07:32:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:28 smithi055 ceph-mon[97033]: pgmap v19270: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:32:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:28 smithi059 ceph-mon[97450]: pgmap v19270: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:32:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:29.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:29.550260+0000) 2022-09-08T07:32:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:29.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:29.447444+0000) 2022-09-08T07:32:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:30.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:30.550471+0000) 2022-09-08T07:32:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:30.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:30.447596+0000) 2022-09-08T07:32:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:30 smithi055 ceph-mon[97033]: pgmap v19271: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T07:32:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:30 smithi059 ceph-mon[97450]: pgmap v19271: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T07:32:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:31.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:31.550658+0000) 2022-09-08T07:32:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:31.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:31.447798+0000) 2022-09-08T07:32:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:32.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:32.550890+0000) 2022-09-08T07:32:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:32.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:32.448061+0000) 2022-09-08T07:32:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:32 smithi055 ceph-mon[97033]: pgmap v19272: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:32:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:32 smithi059 ceph-mon[97450]: pgmap v19272: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:32:33.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:33.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:33.448311+0000) 2022-09-08T07:32:33.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:33.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:33.551044+0000) 2022-09-08T07:32:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:34.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:34.448457+0000) 2022-09-08T07:32:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:34.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:34.551171+0000) 2022-09-08T07:32:35.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:34 smithi055 ceph-mon[97033]: pgmap v19273: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:34 smithi059 ceph-mon[97450]: pgmap v19273: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:35.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:35.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:35.551417+0000) 2022-09-08T07:32:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:35.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:35.448713+0000) 2022-09-08T07:32:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:36.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:36.551653+0000) 2022-09-08T07:32:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:36.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:36.448847+0000) 2022-09-08T07:32:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:36 smithi055 ceph-mon[97033]: pgmap v19274: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:32:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:36 smithi059 ceph-mon[97450]: pgmap v19274: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:32:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:37.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:37.551837+0000) 2022-09-08T07:32:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:37.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:37.449055+0000) 2022-09-08T07:32:38.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:32:38.148+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:32:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:38.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:38.552086+0000) 2022-09-08T07:32:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:38.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:38.449228+0000) 2022-09-08T07:32:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:38 smithi055 ceph-mon[97033]: pgmap v19275: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:38 smithi059 ceph-mon[97450]: pgmap v19275: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:39.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:39.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:39.552211+0000) 2022-09-08T07:32:39.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:39.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:39.449436+0000) 2022-09-08T07:32:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:40.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:40.552438+0000) 2022-09-08T07:32:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:40.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:40.449729+0000) 2022-09-08T07:32:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:40 smithi055 ceph-mon[97033]: pgmap v19276: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:40 smithi059 ceph-mon[97450]: pgmap v19276: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:41.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:41.552673+0000) 2022-09-08T07:32:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:41.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:41.449909+0000) 2022-09-08T07:32:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:42.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:42.450120+0000) 2022-09-08T07:32:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:42.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:42.552902+0000) 2022-09-08T07:32:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:42 smithi055 ceph-mon[97033]: pgmap v19277: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:32:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:42 smithi059 ceph-mon[97450]: pgmap v19277: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:32:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:43.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:43.553072+0000) 2022-09-08T07:32:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:43.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:43.450340+0000) 2022-09-08T07:32:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:44.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:44.553213+0000) 2022-09-08T07:32:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:44.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:44.450516+0000) 2022-09-08T07:32:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:44 smithi055 ceph-mon[97033]: pgmap v19278: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:44 smithi059 ceph-mon[97450]: pgmap v19278: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:45.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:45.553426+0000) 2022-09-08T07:32:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:45.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:45.450747+0000) 2022-09-08T07:32:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:46.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:46.553665+0000) 2022-09-08T07:32:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:46.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:46.450935+0000) 2022-09-08T07:32:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:46 smithi055 ceph-mon[97033]: pgmap v19279: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:32:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:46 smithi059 ceph-mon[97450]: pgmap v19279: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:32:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:47.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:47.553821+0000) 2022-09-08T07:32:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:47.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:47.451139+0000) 2022-09-08T07:32:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:48.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:48.553972+0000) 2022-09-08T07:32:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:48.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:48.451291+0000) 2022-09-08T07:32:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:48 smithi055 ceph-mon[97033]: pgmap v19280: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:48 smithi059 ceph-mon[97450]: pgmap v19280: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:49.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:49.554165+0000) 2022-09-08T07:32:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:49.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:49.451524+0000) 2022-09-08T07:32:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:50.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:50.554389+0000) 2022-09-08T07:32:50.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:50.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:50.451788+0000) 2022-09-08T07:32:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:50 smithi055 ceph-mon[97033]: pgmap v19281: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:50 smithi059 ceph-mon[97450]: pgmap v19281: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:51.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:51.554614+0000) 2022-09-08T07:32:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:51.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:51.451915+0000) 2022-09-08T07:32:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:52.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:52.554874+0000) 2022-09-08T07:32:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:52.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:52.452112+0000) 2022-09-08T07:32:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:52 smithi055 ceph-mon[97033]: pgmap v19282: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:32:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:52 smithi059 ceph-mon[97450]: pgmap v19282: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:32:53.454 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:32:53.162+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:32:53.813 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:53.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:53.555101+0000) 2022-09-08T07:32:53.814 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:53.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:53.452302+0000) 2022-09-08T07:32:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:32:54.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:32:54.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:53 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:32:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:32:54.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:32:54.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:53 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:32:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:54.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:54.555338+0000) 2022-09-08T07:32:54.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:54.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:54.452532+0000) 2022-09-08T07:32:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:54 smithi055 ceph-mon[97033]: pgmap v19283: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:54 smithi059 ceph-mon[97450]: pgmap v19283: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:55.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:55.555517+0000) 2022-09-08T07:32:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:55.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:55.452783+0000) 2022-09-08T07:32:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:56.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:56.555704+0000) 2022-09-08T07:32:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:56.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:56.452997+0000) 2022-09-08T07:32:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:56 smithi055 ceph-mon[97033]: pgmap v19284: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:32:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:56 smithi059 ceph-mon[97450]: pgmap v19284: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:32:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:57.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:57.555954+0000) 2022-09-08T07:32:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:57.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:57.453179+0000) 2022-09-08T07:32:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:58.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:58.556219+0000) 2022-09-08T07:32:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:58.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:58.453376+0000) 2022-09-08T07:32:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:32:58 smithi055 ceph-mon[97033]: pgmap v19285: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:32:58 smithi059 ceph-mon[97450]: pgmap v19285: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:32:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:32:59.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:59.556367+0000) 2022-09-08T07:32:59.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:32:59.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:32:59.453517+0000) 2022-09-08T07:33:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:00.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:00.556598+0000) 2022-09-08T07:33:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:00.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:00.453671+0000) 2022-09-08T07:33:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:00 smithi055 ceph-mon[97033]: pgmap v19286: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:33:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:00 smithi059 ceph-mon[97450]: pgmap v19286: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:33:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:01.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:01.556799+0000) 2022-09-08T07:33:01.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:01.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:01.453801+0000) 2022-09-08T07:33:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:02.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:02.557006+0000) 2022-09-08T07:33:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:02.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:02.453954+0000) 2022-09-08T07:33:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:02 smithi055 ceph-mon[97033]: pgmap v19287: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:33:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:02 smithi059 ceph-mon[97450]: pgmap v19287: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:33:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:03.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:03.557271+0000) 2022-09-08T07:33:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:03.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:03.454191+0000) 2022-09-08T07:33:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:04.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:04.557458+0000) 2022-09-08T07:33:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:04.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:04.454356+0000) 2022-09-08T07:33:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:04 smithi055 ceph-mon[97033]: pgmap v19288: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:33:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:04 smithi059 ceph-mon[97450]: pgmap v19288: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:33:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:05.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:05.454518+0000) 2022-09-08T07:33:05.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:05.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:05.557709+0000) 2022-09-08T07:33:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:06.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:06.557928+0000) 2022-09-08T07:33:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:06.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:06.454716+0000) 2022-09-08T07:33:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:06 smithi055 ceph-mon[97033]: pgmap v19289: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:33:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:33:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:33:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:06 smithi059 ceph-mon[97450]: pgmap v19289: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:33:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:33:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:33:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:07.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:07.558118+0000) 2022-09-08T07:33:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:07.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:07.454876+0000) 2022-09-08T07:33:08.457 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:33:08.165+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:33:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:08.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:08.558291+0000) 2022-09-08T07:33:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:08.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:08.455023+0000) 2022-09-08T07:33:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:08 smithi055 ceph-mon[97033]: pgmap v19290: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:33:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:08 smithi059 ceph-mon[97450]: pgmap v19290: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:33:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:09.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:09.558522+0000) 2022-09-08T07:33:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:09.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:09.455214+0000) 2022-09-08T07:33:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:10.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:10.558721+0000) 2022-09-08T07:33:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:10.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:10.455403+0000) 2022-09-08T07:33:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:10 smithi055 ceph-mon[97033]: pgmap v19291: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:33:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:10 smithi055 ceph-mon[97033]: osdmap e1283: 12 total, 11 up, 11 in 2022-09-08T07:33:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:10 smithi059 ceph-mon[97450]: pgmap v19291: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:33:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:10 smithi059 ceph-mon[97450]: osdmap e1283: 12 total, 11 up, 11 in 2022-09-08T07:33:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:11.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:11.455540+0000) 2022-09-08T07:33:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:11.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:11.558938+0000) 2022-09-08T07:33:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:12.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:12.559193+0000) 2022-09-08T07:33:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:12.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:12.455777+0000) 2022-09-08T07:33:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:12 smithi055 ceph-mon[97033]: pgmap v19293: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:33:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:12 smithi059 ceph-mon[97450]: pgmap v19293: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:33:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:13.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:13.559443+0000) 2022-09-08T07:33:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:13.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:13.455941+0000) 2022-09-08T07:33:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:14.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:14.559622+0000) 2022-09-08T07:33:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:14.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:14.456129+0000) 2022-09-08T07:33:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:14 smithi055 ceph-mon[97033]: pgmap v19294: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:33:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:14 smithi059 ceph-mon[97450]: pgmap v19294: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:33:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:15.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:15.456319+0000) 2022-09-08T07:33:15.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:15.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:15.559831+0000) 2022-09-08T07:33:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:16.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:16.560005+0000) 2022-09-08T07:33:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:16.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:16.456466+0000) 2022-09-08T07:33:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:16 smithi055 ceph-mon[97033]: pgmap v19295: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:33:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:16 smithi059 ceph-mon[97450]: pgmap v19295: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:33:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:17.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:17.560216+0000) 2022-09-08T07:33:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:17.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:17.456703+0000) 2022-09-08T07:33:18.875 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:18.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:18.560357+0000) 2022-09-08T07:33:18.876 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:18.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:18.456975+0000) 2022-09-08T07:33:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:18 smithi055 ceph-mon[97033]: pgmap v19296: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:33:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:18 smithi055 ceph-mon[97033]: osdmap e1284: 12 total, 11 up, 11 in 2022-09-08T07:33:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:18 smithi059 ceph-mon[97450]: pgmap v19296: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:33:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:18 smithi059 ceph-mon[97450]: osdmap e1284: 12 total, 11 up, 11 in 2022-09-08T07:33:19.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:19.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:19.560590+0000) 2022-09-08T07:33:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:19.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:19.457161+0000) 2022-09-08T07:33:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:20.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:20.457307+0000) 2022-09-08T07:33:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:20.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:20.560878+0000) 2022-09-08T07:33:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:20 smithi055 ceph-mon[97033]: pgmap v19298: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 209 B/s wr, 5 op/s 2022-09-08T07:33:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:20 smithi059 ceph-mon[97450]: pgmap v19298: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 209 B/s wr, 5 op/s 2022-09-08T07:33:21.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:21.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:21.561091+0000) 2022-09-08T07:33:21.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:21.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:21.457517+0000) 2022-09-08T07:33:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:22.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:22.457703+0000) 2022-09-08T07:33:22.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:22.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:22.561340+0000) 2022-09-08T07:33:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:22 smithi055 ceph-mon[97033]: pgmap v19299: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:33:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:22 smithi059 ceph-mon[97450]: pgmap v19299: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:33:23.460 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:33:23.169+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:33:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:23.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:23.457909+0000) 2022-09-08T07:33:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:23.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:23.561532+0000) 2022-09-08T07:33:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:24.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:24.458134+0000) 2022-09-08T07:33:24.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:24.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:24.561693+0000) 2022-09-08T07:33:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:24 smithi055 ceph-mon[97033]: pgmap v19300: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:33:25.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:24 smithi055 ceph-mon[97033]: pgmap v19301: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:33:25.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:24 smithi055 ceph-mon[97033]: pgmap v19302: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:33:25.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:33:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:24 smithi059 ceph-mon[97450]: pgmap v19300: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:33:25.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:24 smithi059 ceph-mon[97450]: pgmap v19301: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:33:25.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:24 smithi059 ceph-mon[97450]: pgmap v19302: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:33:25.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:33:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:25.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:25.561935+0000) 2022-09-08T07:33:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:25.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:25.458323+0000) 2022-09-08T07:33:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:26.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:26.562118+0000) 2022-09-08T07:33:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:26.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:26.458511+0000) 2022-09-08T07:33:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:26 smithi055 ceph-mon[97033]: pgmap v19303: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:33:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:26 smithi059 ceph-mon[97450]: pgmap v19303: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:33:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:27.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:27.458752+0000) 2022-09-08T07:33:27.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:27.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:27.562374+0000) 2022-09-08T07:33:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:28.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:28.562590+0000) 2022-09-08T07:33:28.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:28.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:28.458915+0000) 2022-09-08T07:33:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:28 smithi055 ceph-mon[97033]: pgmap v19304: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:33:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:28 smithi059 ceph-mon[97450]: pgmap v19304: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:33:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:29.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:29.459117+0000) 2022-09-08T07:33:29.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:29.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:29.562807+0000) 2022-09-08T07:33:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:29 smithi059 ceph-mon[97450]: pgmap v19305: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:33:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:29 smithi055 ceph-mon[97033]: pgmap v19305: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:33:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:30.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:30.563064+0000) 2022-09-08T07:33:30.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:30.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:30.459332+0000) 2022-09-08T07:33:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:31.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:31.459549+0000) 2022-09-08T07:33:31.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:31.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:31.563305+0000) 2022-09-08T07:33:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:31 smithi059 ceph-mon[97450]: pgmap v19306: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:33:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:31 smithi055 ceph-mon[97033]: pgmap v19306: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:33:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:32.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:32.563508+0000) 2022-09-08T07:33:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:32.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:32.459792+0000) 2022-09-08T07:33:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:33.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:33.563648+0000) 2022-09-08T07:33:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:33.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:33.459976+0000) 2022-09-08T07:33:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:33 smithi059 ceph-mon[97450]: pgmap v19307: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:33:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:33 smithi055 ceph-mon[97033]: pgmap v19307: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:33:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:34.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:34.563891+0000) 2022-09-08T07:33:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:34.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:34.460235+0000) 2022-09-08T07:33:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:35.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:35.564164+0000) 2022-09-08T07:33:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:35.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:35.460475+0000) 2022-09-08T07:33:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:35 smithi059 ceph-mon[97450]: pgmap v19308: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:33:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:35 smithi055 ceph-mon[97033]: pgmap v19308: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:33:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:36.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:36.564360+0000) 2022-09-08T07:33:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:36.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:36.460746+0000) 2022-09-08T07:33:37.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:37.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:37.564508+0000) 2022-09-08T07:33:37.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:37.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:37.460913+0000) 2022-09-08T07:33:38.463 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:33:38.179+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:33:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:38.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:38.564705+0000) 2022-09-08T07:33:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:38.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:38.461065+0000) 2022-09-08T07:33:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:38 smithi055 ceph-mon[97033]: pgmap v19309: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:33:39.241 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:38 smithi059 ceph-mon[97450]: pgmap v19309: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:33:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:39.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:39.564899+0000) 2022-09-08T07:33:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:39.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:39.461192+0000) 2022-09-08T07:33:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:39 smithi059 ceph-mon[97450]: pgmap v19310: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:33:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:39 smithi055 ceph-mon[97033]: pgmap v19310: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:33:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:40.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:40.565153+0000) 2022-09-08T07:33:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:40.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:40.461440+0000) 2022-09-08T07:33:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:41.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:41.565304+0000) 2022-09-08T07:33:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:41.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:41.461728+0000) 2022-09-08T07:33:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:41 smithi059 ceph-mon[97450]: pgmap v19311: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T07:33:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:41 smithi055 ceph-mon[97033]: pgmap v19311: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T07:33:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:42.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:42.565521+0000) 2022-09-08T07:33:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:42.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:42.461966+0000) 2022-09-08T07:33:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:43.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:43.565705+0000) 2022-09-08T07:33:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:43.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:43.462133+0000) 2022-09-08T07:33:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:43 smithi059 ceph-mon[97450]: pgmap v19312: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:33:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:43 smithi055 ceph-mon[97033]: pgmap v19312: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:33:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:44.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:44.565859+0000) 2022-09-08T07:33:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:44.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:44.462374+0000) 2022-09-08T07:33:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:45.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:45.566033+0000) 2022-09-08T07:33:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:45.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:45.462630+0000) 2022-09-08T07:33:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:45 smithi059 ceph-mon[97450]: pgmap v19313: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:33:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:45 smithi055 ceph-mon[97033]: pgmap v19313: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:33:46.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:46.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:46.566206+0000) 2022-09-08T07:33:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:46.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:46.462896+0000) 2022-09-08T07:33:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:47.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:47.566416+0000) 2022-09-08T07:33:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:47.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:47.463128+0000) 2022-09-08T07:33:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:48.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:48.566596+0000) 2022-09-08T07:33:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:48.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:48.463309+0000) 2022-09-08T07:33:49.125 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:48 smithi059 ceph-mon[97450]: pgmap v19314: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:33:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:48 smithi055 ceph-mon[97033]: pgmap v19314: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:33:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:49.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:49.566772+0000) 2022-09-08T07:33:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:49.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:49.463541+0000) 2022-09-08T07:33:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:49 smithi059 ceph-mon[97450]: pgmap v19315: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:33:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:49 smithi055 ceph-mon[97033]: pgmap v19315: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:33:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:50.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:50.566938+0000) 2022-09-08T07:33:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:50.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:50.463737+0000) 2022-09-08T07:33:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:51.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:51.567074+0000) 2022-09-08T07:33:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:51.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:51.463903+0000) 2022-09-08T07:33:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:51 smithi059 ceph-mon[97450]: pgmap v19316: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:33:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:51 smithi055 ceph-mon[97033]: pgmap v19316: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:33:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:52.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:52.567210+0000) 2022-09-08T07:33:52.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:52.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:52.464064+0000) 2022-09-08T07:33:53.466 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:33:53.192+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:33:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:53.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:53.567416+0000) 2022-09-08T07:33:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:53.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:53.464306+0000) 2022-09-08T07:33:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:53 smithi059 ceph-mon[97450]: pgmap v19317: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:33:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:53 smithi055 ceph-mon[97033]: pgmap v19317: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:33:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:54.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:54.567731+0000) 2022-09-08T07:33:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:54.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:54.464550+0000) 2022-09-08T07:33:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:55.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:55.567972+0000) 2022-09-08T07:33:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:55.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:55.464773+0000) 2022-09-08T07:33:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:55 smithi059 ceph-mon[97450]: pgmap v19318: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:33:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:55 smithi055 ceph-mon[97033]: pgmap v19318: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:33:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:56.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:56.568170+0000) 2022-09-08T07:33:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:56.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:56.465010+0000) 2022-09-08T07:33:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:57.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:57.568432+0000) 2022-09-08T07:33:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:57.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:57.465285+0000) 2022-09-08T07:33:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:58.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:58.568673+0000) 2022-09-08T07:33:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:58.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:58.465491+0000) 2022-09-08T07:33:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:33:58 smithi055 ceph-mon[97033]: pgmap v19319: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:33:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:58 smithi059 ceph-mon[97450]: pgmap v19319: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:33:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:33:59.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:59.465683+0000) 2022-09-08T07:33:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:33:59.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:33:59.568892+0000) 2022-09-08T07:34:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:33:59 smithi059 ceph-mon[97450]: pgmap v19320: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:34:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:00 smithi055 ceph-mon[97033]: pgmap v19320: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:34:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:00.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:00.569132+0000) 2022-09-08T07:34:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:00.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:00.465844+0000) 2022-09-08T07:34:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:01.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:01.569329+0000) 2022-09-08T07:34:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:01.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:01.466060+0000) 2022-09-08T07:34:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:02 smithi059 ceph-mon[97450]: pgmap v19321: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:34:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:02 smithi055 ceph-mon[97033]: pgmap v19321: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:34:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:02.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:02.569560+0000) 2022-09-08T07:34:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:02.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:02.466320+0000) 2022-09-08T07:34:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:03.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:03.569792+0000) 2022-09-08T07:34:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:03.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:03.466450+0000) 2022-09-08T07:34:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:04 smithi059 ceph-mon[97450]: pgmap v19322: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:34:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:04 smithi055 ceph-mon[97033]: pgmap v19322: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:34:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:04.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:04.569995+0000) 2022-09-08T07:34:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:04.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:04.466609+0000) 2022-09-08T07:34:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:05.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:05.570260+0000) 2022-09-08T07:34:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:05.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:05.466764+0000) 2022-09-08T07:34:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:05 smithi059 ceph-mon[97450]: pgmap v19323: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:34:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:05 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:34:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:06 smithi055 ceph-mon[97033]: pgmap v19323: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:34:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:34:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:06.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:06.570479+0000) 2022-09-08T07:34:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:06.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:06.466958+0000) 2022-09-08T07:34:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:34:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:34:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:07.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:07.570737+0000) 2022-09-08T07:34:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:07.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:07.467122+0000) 2022-09-08T07:34:08.469 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:34:08.199+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:34:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:08.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:08.570982+0000) 2022-09-08T07:34:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:08.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:08.467273+0000) 2022-09-08T07:34:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:08 smithi055 ceph-mon[97033]: pgmap v19324: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:34:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:08 smithi059 ceph-mon[97450]: pgmap v19324: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:34:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:09.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:09.571202+0000) 2022-09-08T07:34:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:09.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:09.467413+0000) 2022-09-08T07:34:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:10.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:10.467632+0000) 2022-09-08T07:34:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:10.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:10.571403+0000) 2022-09-08T07:34:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:10 smithi055 ceph-mon[97033]: osdmap e1285: 12 total, 11 up, 11 in 2022-09-08T07:34:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:10 smithi055 ceph-mon[97033]: pgmap v19326: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:34:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:10 smithi059 ceph-mon[97450]: osdmap e1285: 12 total, 11 up, 11 in 2022-09-08T07:34:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:10 smithi059 ceph-mon[97450]: pgmap v19326: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:34:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:11.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:11.571690+0000) 2022-09-08T07:34:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:11.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:11.467868+0000) 2022-09-08T07:34:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:12 smithi059 ceph-mon[97450]: pgmap v19327: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:34:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:12 smithi055 ceph-mon[97033]: pgmap v19327: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:34:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:12.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:12.571917+0000) 2022-09-08T07:34:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:12.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:12.468054+0000) 2022-09-08T07:34:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:13.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:13.572132+0000) 2022-09-08T07:34:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:13.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:13.468261+0000) 2022-09-08T07:34:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:14 smithi059 ceph-mon[97450]: pgmap v19328: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:34:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:14 smithi055 ceph-mon[97033]: pgmap v19328: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:34:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:14.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:14.572297+0000) 2022-09-08T07:34:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:14.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:14.468461+0000) 2022-09-08T07:34:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:15.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:15.572546+0000) 2022-09-08T07:34:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:15.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:15.468652+0000) 2022-09-08T07:34:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:16 smithi059 ceph-mon[97450]: pgmap v19329: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:34:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:16 smithi055 ceph-mon[97033]: pgmap v19329: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:34:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:16.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:16.468840+0000) 2022-09-08T07:34:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:16.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:16.572755+0000) 2022-09-08T07:34:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:17.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:17.469040+0000) 2022-09-08T07:34:17.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:17.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:17.572990+0000) 2022-09-08T07:34:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:18.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:18.573186+0000) 2022-09-08T07:34:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:18.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:18.469258+0000) 2022-09-08T07:34:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:18 smithi055 ceph-mon[97033]: osdmap e1286: 12 total, 11 up, 11 in 2022-09-08T07:34:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:18 smithi055 ceph-mon[97033]: pgmap v19331: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T07:34:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:18 smithi059 ceph-mon[97450]: osdmap e1286: 12 total, 11 up, 11 in 2022-09-08T07:34:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:18 smithi059 ceph-mon[97450]: pgmap v19331: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 252 B/s wr, 7 op/s 2022-09-08T07:34:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:19.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:19.573371+0000) 2022-09-08T07:34:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:19.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:19.469433+0000) 2022-09-08T07:34:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:20 smithi059 ceph-mon[97450]: pgmap v19332: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:34:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:20 smithi055 ceph-mon[97033]: pgmap v19332: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:34:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:20.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:20.573592+0000) 2022-09-08T07:34:20.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:20.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:20.469590+0000) 2022-09-08T07:34:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:21.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:21.573792+0000) 2022-09-08T07:34:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:21.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:21.469805+0000) 2022-09-08T07:34:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:22 smithi059 ceph-mon[97450]: pgmap v19333: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:34:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:22 smithi055 ceph-mon[97033]: pgmap v19333: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:34:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:22.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:22.574025+0000) 2022-09-08T07:34:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:22.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:22.469945+0000) 2022-09-08T07:34:23.472 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:34:23.213+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:34:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:23.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:23.574166+0000) 2022-09-08T07:34:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:23.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:23.470084+0000) 2022-09-08T07:34:24.011 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:24 smithi055 ceph-mon[97033]: pgmap v19334: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:34:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:24 smithi059 ceph-mon[97450]: pgmap v19334: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:34:24.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:24.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:24.574324+0000) 2022-09-08T07:34:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:24.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:24.470273+0000) 2022-09-08T07:34:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:34:25.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:34:25.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:34:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:34:25.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:34:25.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:34:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:25.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:25.574524+0000) 2022-09-08T07:34:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:25.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:25.470499+0000) 2022-09-08T07:34:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:26 smithi059 ceph-mon[97450]: pgmap v19335: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:34:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:26 smithi055 ceph-mon[97033]: pgmap v19335: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:34:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:26.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:26.574714+0000) 2022-09-08T07:34:26.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:26.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:26.470651+0000) 2022-09-08T07:34:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:27.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:27.574863+0000) 2022-09-08T07:34:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:27.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:27.470783+0000) 2022-09-08T07:34:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:28.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:28.470929+0000) 2022-09-08T07:34:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:28.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:28.575077+0000) 2022-09-08T07:34:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:28 smithi055 ceph-mon[97033]: pgmap v19336: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:34:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:28 smithi059 ceph-mon[97450]: pgmap v19336: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:34:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:29.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:29.575240+0000) 2022-09-08T07:34:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:29.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:29.471091+0000) 2022-09-08T07:34:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:30 smithi059 ceph-mon[97450]: pgmap v19337: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-08T07:34:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:30 smithi055 ceph-mon[97033]: pgmap v19337: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-08T07:34:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:30.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:30.575472+0000) 2022-09-08T07:34:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:30.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:30.471299+0000) 2022-09-08T07:34:31.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:31.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:31.575653+0000) 2022-09-08T07:34:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:31.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:31.471525+0000) 2022-09-08T07:34:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:32 smithi059 ceph-mon[97450]: pgmap v19338: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:34:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:32 smithi055 ceph-mon[97033]: pgmap v19338: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:34:32.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:32.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:32.471757+0000) 2022-09-08T07:34:32.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:32.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:32.575891+0000) 2022-09-08T07:34:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:33.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:33.576157+0000) 2022-09-08T07:34:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:33.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:33.471939+0000) 2022-09-08T07:34:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:34.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:34.576353+0000) 2022-09-08T07:34:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:34.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:34.472124+0000) 2022-09-08T07:34:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:35 smithi059 ceph-mon[97450]: pgmap v19339: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:34:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:35 smithi055 ceph-mon[97033]: pgmap v19339: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:34:35.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:35.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:35.472297+0000) 2022-09-08T07:34:35.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:35.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:35.576599+0000) 2022-09-08T07:34:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:36.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:36.472448+0000) 2022-09-08T07:34:36.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:36.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:36.576793+0000) 2022-09-08T07:34:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:37 smithi059 ceph-mon[97450]: pgmap v19340: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:34:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:37 smithi055 ceph-mon[97033]: pgmap v19340: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:34:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:37.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:37.577052+0000) 2022-09-08T07:34:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:37.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:37.472601+0000) 2022-09-08T07:34:38.475 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:34:38.218+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:34:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:38.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:38.577287+0000) 2022-09-08T07:34:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:38.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:38.472753+0000) 2022-09-08T07:34:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:39.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:39.577524+0000) 2022-09-08T07:34:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:39.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:39.472905+0000) 2022-09-08T07:34:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:39 smithi055 ceph-mon[97033]: pgmap v19341: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:34:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:39 smithi059 ceph-mon[97450]: pgmap v19341: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:34:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:40.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:40.577747+0000) 2022-09-08T07:34:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:40.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:40.473116+0000) 2022-09-08T07:34:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:41.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:41.577963+0000) 2022-09-08T07:34:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:41.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:41.473305+0000) 2022-09-08T07:34:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:41 smithi055 ceph-mon[97033]: pgmap v19342: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:34:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:41 smithi059 ceph-mon[97450]: pgmap v19342: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:34:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:42.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:42.473510+0000) 2022-09-08T07:34:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:42.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:42.578157+0000) 2022-09-08T07:34:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:43.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:43.578305+0000) 2022-09-08T07:34:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:43.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:43.473630+0000) 2022-09-08T07:34:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:43 smithi055 ceph-mon[97033]: pgmap v19343: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:34:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:43 smithi059 ceph-mon[97450]: pgmap v19343: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:34:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:44.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:44.578496+0000) 2022-09-08T07:34:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:44.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:44.473806+0000) 2022-09-08T07:34:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:45.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:45.578743+0000) 2022-09-08T07:34:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:45.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:45.473941+0000) 2022-09-08T07:34:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:45 smithi055 ceph-mon[97033]: pgmap v19344: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:34:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:45 smithi059 ceph-mon[97450]: pgmap v19344: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:34:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:46.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:46.474146+0000) 2022-09-08T07:34:46.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:46.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:46.578959+0000) 2022-09-08T07:34:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:47.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:47.579146+0000) 2022-09-08T07:34:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:47.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:47.474374+0000) 2022-09-08T07:34:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:47 smithi055 ceph-mon[97033]: pgmap v19345: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:34:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:47 smithi059 ceph-mon[97450]: pgmap v19345: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:34:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:48.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:48.474625+0000) 2022-09-08T07:34:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:48.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:48.579420+0000) 2022-09-08T07:34:49.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:49.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:49.474934+0000) 2022-09-08T07:34:49.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:49.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:49.579658+0000) 2022-09-08T07:34:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:49 smithi055 ceph-mon[97033]: pgmap v19346: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:34:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:49 smithi059 ceph-mon[97450]: pgmap v19346: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:34:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:50.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:50.579833+0000) 2022-09-08T07:34:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:50.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:50.475196+0000) 2022-09-08T07:34:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:51.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:51.580061+0000) 2022-09-08T07:34:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:51.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:51.475435+0000) 2022-09-08T07:34:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:51 smithi055 ceph-mon[97033]: pgmap v19347: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:34:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:51 smithi059 ceph-mon[97450]: pgmap v19347: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T07:34:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:52.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:52.580287+0000) 2022-09-08T07:34:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:52.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:52.475578+0000) 2022-09-08T07:34:53.478 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:34:53.221+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:34:53.774 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:53.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:53.580477+0000) 2022-09-08T07:34:53.775 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:53.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:53.475712+0000) 2022-09-08T07:34:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:53 smithi055 ceph-mon[97033]: pgmap v19348: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:34:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:53 smithi059 ceph-mon[97450]: pgmap v19348: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:34:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:54.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:54.580674+0000) 2022-09-08T07:34:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:54.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:54.475899+0000) 2022-09-08T07:34:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:55.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:55.580840+0000) 2022-09-08T07:34:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:55.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:55.476030+0000) 2022-09-08T07:34:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:55 smithi055 ceph-mon[97033]: pgmap v19349: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:34:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:55 smithi059 ceph-mon[97450]: pgmap v19349: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:34:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:56.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:56.581036+0000) 2022-09-08T07:34:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:56.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:56.476243+0000) 2022-09-08T07:34:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:57.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:57.581244+0000) 2022-09-08T07:34:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:57.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:57.476416+0000) 2022-09-08T07:34:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:57 smithi055 ceph-mon[97033]: pgmap v19350: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:34:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:57 smithi059 ceph-mon[97450]: pgmap v19350: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:34:58.752 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:58.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:58.581477+0000) 2022-09-08T07:34:58.753 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:58.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:58.476599+0000) 2022-09-08T07:34:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:34:59.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:59.581698+0000) 2022-09-08T07:34:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:34:59.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:34:59.476778+0000) 2022-09-08T07:35:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:59 smithi059 ceph-mon[97450]: pgmap v19351: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:00.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:35:00.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:35:00.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:59 smithi059 ceph-mon[97450]: pgmap v19352: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:35:00.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:59 smithi059 ceph-mon[97450]: pgmap v19353: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T07:35:00.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:34:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:35:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:59 smithi055 ceph-mon[97033]: pgmap v19351: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:35:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:35:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:59 smithi055 ceph-mon[97033]: pgmap v19352: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:35:00.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:59 smithi055 ceph-mon[97033]: pgmap v19353: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T07:35:00.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:34:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:35:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:00.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:00.581829+0000) 2022-09-08T07:35:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:00.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:00.477006+0000) 2022-09-08T07:35:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:01.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:01.582029+0000) 2022-09-08T07:35:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:01.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:01.477178+0000) 2022-09-08T07:35:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:01 smithi055 ceph-mon[97033]: pgmap v19354: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T07:35:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:01 smithi059 ceph-mon[97450]: pgmap v19354: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T07:35:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:02.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:02.582317+0000) 2022-09-08T07:35:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:02.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:02.477321+0000) 2022-09-08T07:35:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:03.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:03.582586+0000) 2022-09-08T07:35:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:03.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:03.477529+0000) 2022-09-08T07:35:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:03 smithi055 ceph-mon[97033]: pgmap v19355: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T07:35:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:03 smithi059 ceph-mon[97450]: pgmap v19355: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-09-08T07:35:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:04.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:04.582786+0000) 2022-09-08T07:35:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:04.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:04.477773+0000) 2022-09-08T07:35:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:05.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:05.582954+0000) 2022-09-08T07:35:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:05.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:05.478016+0000) 2022-09-08T07:35:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:05 smithi055 ceph-mon[97033]: pgmap v19356: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T07:35:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:05 smithi059 ceph-mon[97450]: pgmap v19356: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T07:35:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:06.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:06.583153+0000) 2022-09-08T07:35:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:06.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:06.478264+0000) 2022-09-08T07:35:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:35:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:35:07.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:07.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:07.583402+0000) 2022-09-08T07:35:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:07.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:07.478448+0000) 2022-09-08T07:35:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:07 smithi055 ceph-mon[97033]: pgmap v19357: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 60 KiB/s wr, 27 op/s 2022-09-08T07:35:08.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:35:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:07 smithi059 ceph-mon[97450]: pgmap v19357: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 60 KiB/s wr, 27 op/s 2022-09-08T07:35:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:35:08.480 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:35:08.226+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:35:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:08.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:08.583689+0000) 2022-09-08T07:35:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:08.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:08.478628+0000) 2022-09-08T07:35:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:09.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:09.583935+0000) 2022-09-08T07:35:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:09.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:09.478781+0000) 2022-09-08T07:35:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:09 smithi055 ceph-mon[97033]: pgmap v19358: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:35:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:09 smithi059 ceph-mon[97450]: pgmap v19358: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:35:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:10.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:10.584142+0000) 2022-09-08T07:35:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:10.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:10.478935+0000) 2022-09-08T07:35:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:10 smithi055 ceph-mon[97033]: osdmap e1287: 12 total, 11 up, 11 in 2022-09-08T07:35:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:10 smithi059 ceph-mon[97450]: osdmap e1287: 12 total, 11 up, 11 in 2022-09-08T07:35:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:11.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:11.584356+0000) 2022-09-08T07:35:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:11.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:11.479111+0000) 2022-09-08T07:35:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:11 smithi055 ceph-mon[97033]: pgmap v19360: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:35:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:11 smithi059 ceph-mon[97450]: pgmap v19360: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:35:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:12.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:12.584588+0000) 2022-09-08T07:35:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:12.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:12.479331+0000) 2022-09-08T07:35:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:13.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:13.584794+0000) 2022-09-08T07:35:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:13.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:13.479533+0000) 2022-09-08T07:35:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:13 smithi055 ceph-mon[97033]: pgmap v19361: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:35:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:13 smithi059 ceph-mon[97450]: pgmap v19361: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:35:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:14.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:14.585022+0000) 2022-09-08T07:35:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:14.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:14.479735+0000) 2022-09-08T07:35:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:15.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:15.479890+0000) 2022-09-08T07:35:15.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:15.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:15.585257+0000) 2022-09-08T07:35:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:15 smithi055 ceph-mon[97033]: pgmap v19362: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:35:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:15 smithi059 ceph-mon[97450]: pgmap v19362: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:35:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:16.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:16.585399+0000) 2022-09-08T07:35:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:16.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:16.480109+0000) 2022-09-08T07:35:17.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:17.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:17.585632+0000) 2022-09-08T07:35:17.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:17.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:17.480313+0000) 2022-09-08T07:35:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:17 smithi055 ceph-mon[97033]: pgmap v19363: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:35:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:17 smithi055 ceph-mon[97033]: osdmap e1288: 12 total, 11 up, 11 in 2022-09-08T07:35:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:17 smithi059 ceph-mon[97450]: pgmap v19363: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:35:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:17 smithi059 ceph-mon[97450]: osdmap e1288: 12 total, 11 up, 11 in 2022-09-08T07:35:18.800 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:18.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:18.585901+0000) 2022-09-08T07:35:18.801 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:18.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:18.480532+0000) 2022-09-08T07:35:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:19.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:19.480708+0000) 2022-09-08T07:35:19.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:19.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:19.586128+0000) 2022-09-08T07:35:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:19 smithi055 ceph-mon[97033]: pgmap v19365: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 229 B/s wr, 5 op/s 2022-09-08T07:35:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:19 smithi059 ceph-mon[97450]: pgmap v19365: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 229 B/s wr, 5 op/s 2022-09-08T07:35:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:20.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:20.480857+0000) 2022-09-08T07:35:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:20.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:20.586334+0000) 2022-09-08T07:35:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:21.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:21.586466+0000) 2022-09-08T07:35:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:21.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:21.481064+0000) 2022-09-08T07:35:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:21 smithi055 ceph-mon[97033]: pgmap v19366: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:35:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:21 smithi059 ceph-mon[97450]: pgmap v19366: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:35:22.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:22.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:22.586650+0000) 2022-09-08T07:35:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:22.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:22.481278+0000) 2022-09-08T07:35:23.589 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:35:23.242+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:35:23.590 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:23.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:23.481446+0000) 2022-09-08T07:35:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:23.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:23.586892+0000) 2022-09-08T07:35:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:23 smithi055 ceph-mon[97033]: pgmap v19367: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:35:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:23 smithi059 ceph-mon[97450]: pgmap v19367: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:35:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:24.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:24.587157+0000) 2022-09-08T07:35:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:24.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:24.481697+0000) 2022-09-08T07:35:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:25.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:25.587416+0000) 2022-09-08T07:35:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:25.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:25.481862+0000) 2022-09-08T07:35:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:25 smithi055 ceph-mon[97033]: pgmap v19368: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:35:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:25 smithi059 ceph-mon[97450]: pgmap v19368: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:35:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:26.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:26.482014+0000) 2022-09-08T07:35:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:26.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:26.587627+0000) 2022-09-08T07:35:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:27.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:27.587874+0000) 2022-09-08T07:35:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:27.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:27.482154+0000) 2022-09-08T07:35:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:27 smithi055 ceph-mon[97033]: pgmap v19369: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:35:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:27 smithi059 ceph-mon[97450]: pgmap v19369: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:35:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:28.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:28.588007+0000) 2022-09-08T07:35:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:28.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:28.482306+0000) 2022-09-08T07:35:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:29.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:29.588212+0000) 2022-09-08T07:35:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:29.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:29.482472+0000) 2022-09-08T07:35:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:29 smithi055 ceph-mon[97033]: pgmap v19370: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:35:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:29 smithi059 ceph-mon[97450]: pgmap v19370: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:35:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:30.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:30.588412+0000) 2022-09-08T07:35:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:30.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:30.482673+0000) 2022-09-08T07:35:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:31.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:31.588677+0000) 2022-09-08T07:35:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:31.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:31.482905+0000) 2022-09-08T07:35:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:31 smithi055 ceph-mon[97033]: pgmap v19371: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:35:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:31 smithi059 ceph-mon[97450]: pgmap v19371: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:35:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:32.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:32.588897+0000) 2022-09-08T07:35:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:32.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:32.483069+0000) 2022-09-08T07:35:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:33.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:33.589167+0000) 2022-09-08T07:35:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:33.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:33.483292+0000) 2022-09-08T07:35:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:33 smithi055 ceph-mon[97033]: pgmap v19372: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:33 smithi059 ceph-mon[97450]: pgmap v19372: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:34.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:34.589303+0000) 2022-09-08T07:35:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:34.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:34.483531+0000) 2022-09-08T07:35:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:35.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:35.589516+0000) 2022-09-08T07:35:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:35.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:35.483800+0000) 2022-09-08T07:35:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:35 smithi055 ceph-mon[97033]: pgmap v19373: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:35 smithi059 ceph-mon[97450]: pgmap v19373: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:36.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:36.589754+0000) 2022-09-08T07:35:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:36.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:36.483979+0000) 2022-09-08T07:35:37.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:37.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:37.589984+0000) 2022-09-08T07:35:37.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:37.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:37.484120+0000) 2022-09-08T07:35:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:37 smithi055 ceph-mon[97033]: pgmap v19374: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:35:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:37 smithi059 ceph-mon[97450]: pgmap v19374: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:35:38.592 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:38.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:38.484263+0000) 2022-09-08T07:35:38.593 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:35:38.244+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:35:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:38.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:38.590121+0000) 2022-09-08T07:35:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:39.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:39.590390+0000) 2022-09-08T07:35:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:39.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:39.484502+0000) 2022-09-08T07:35:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:39 smithi055 ceph-mon[97033]: pgmap v19375: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:39 smithi059 ceph-mon[97450]: pgmap v19375: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:40.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:40.590612+0000) 2022-09-08T07:35:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:40.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:40.484693+0000) 2022-09-08T07:35:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:41.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:41.590829+0000) 2022-09-08T07:35:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:41.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:41.484953+0000) 2022-09-08T07:35:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:41 smithi055 ceph-mon[97033]: pgmap v19376: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:35:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:41 smithi059 ceph-mon[97450]: pgmap v19376: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:35:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:42.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:42.591052+0000) 2022-09-08T07:35:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:42.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:42.485224+0000) 2022-09-08T07:35:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:43.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:43.591287+0000) 2022-09-08T07:35:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:43.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:43.485417+0000) 2022-09-08T07:35:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:43 smithi055 ceph-mon[97033]: pgmap v19377: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:43 smithi059 ceph-mon[97450]: pgmap v19377: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:44.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:44.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:44.591559+0000) 2022-09-08T07:35:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:44.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:44.485564+0000) 2022-09-08T07:35:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:45.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:45.591862+0000) 2022-09-08T07:35:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:45.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:45.485768+0000) 2022-09-08T07:35:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:45 smithi055 ceph-mon[97033]: pgmap v19378: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:45 smithi059 ceph-mon[97450]: pgmap v19378: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:46.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:46.592024+0000) 2022-09-08T07:35:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:46.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:46.485901+0000) 2022-09-08T07:35:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:47.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:47.592144+0000) 2022-09-08T07:35:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:47.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:47.486092+0000) 2022-09-08T07:35:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:47 smithi055 ceph-mon[97033]: pgmap v19379: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:35:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:47 smithi059 ceph-mon[97450]: pgmap v19379: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:35:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:48.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:48.592340+0000) 2022-09-08T07:35:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:48.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:48.486230+0000) 2022-09-08T07:35:49.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:49.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:49.592510+0000) 2022-09-08T07:35:49.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:49.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:49.486424+0000) 2022-09-08T07:35:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:49 smithi055 ceph-mon[97033]: pgmap v19380: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:49 smithi059 ceph-mon[97450]: pgmap v19380: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:50.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:50.592734+0000) 2022-09-08T07:35:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:50.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:50.486580+0000) 2022-09-08T07:35:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:51.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:51.486801+0000) 2022-09-08T07:35:51.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:51.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:51.592893+0000) 2022-09-08T07:35:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:51 smithi055 ceph-mon[97033]: pgmap v19381: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:35:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:51 smithi059 ceph-mon[97450]: pgmap v19381: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:35:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:52.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:52.593084+0000) 2022-09-08T07:35:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:52.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:52.486943+0000) 2022-09-08T07:35:53.595 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:35:53.246+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:35:53.596 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:53.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:53.487093+0000) 2022-09-08T07:35:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:53.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:53.593305+0000) 2022-09-08T07:35:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:53 smithi055 ceph-mon[97033]: pgmap v19382: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:53 smithi059 ceph-mon[97450]: pgmap v19382: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:54.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:54.593478+0000) 2022-09-08T07:35:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:54.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:54.487243+0000) 2022-09-08T07:35:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:55.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:55.593694+0000) 2022-09-08T07:35:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:55.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:55.487386+0000) 2022-09-08T07:35:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:55 smithi055 ceph-mon[97033]: pgmap v19383: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:55 smithi059 ceph-mon[97450]: pgmap v19383: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:35:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:56.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:56.593923+0000) 2022-09-08T07:35:56.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:56.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:56.487587+0000) 2022-09-08T07:35:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:57.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:57.594078+0000) 2022-09-08T07:35:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:57.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:57.487789+0000) 2022-09-08T07:35:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:57 smithi055 ceph-mon[97033]: pgmap v19384: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:35:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:57 smithi059 ceph-mon[97450]: pgmap v19384: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:35:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:58.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:58.594262+0000) 2022-09-08T07:35:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:58.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:58.488018+0000) 2022-09-08T07:35:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:35:59.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:35:59.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:35:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:35:59.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:35:59.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:35:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:35:59.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:59.594477+0000) 2022-09-08T07:35:59.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:35:59.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:35:59.488188+0000) 2022-09-08T07:36:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:59 smithi059 ceph-mon[97450]: pgmap v19385: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:59 smithi059 ceph-mon[97450]: pgmap v19386: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:36:00.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:35:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:36:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:59 smithi055 ceph-mon[97033]: pgmap v19385: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:59 smithi055 ceph-mon[97033]: pgmap v19386: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:36:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:35:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:36:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:00.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:00.594633+0000) 2022-09-08T07:36:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:00.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:00.488331+0000) 2022-09-08T07:36:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:00 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T07:36:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:00 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T07:36:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:01.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:01.594761+0000) 2022-09-08T07:36:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:01.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:01.488529+0000) 2022-09-08T07:36:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:01 smithi059 ceph-mon[97450]: pgmap v19387: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:36:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:01 smithi055 ceph-mon[97033]: pgmap v19387: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:36:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:02.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:02.594964+0000) 2022-09-08T07:36:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:02.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:02.488740+0000) 2022-09-08T07:36:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:03.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:03.488906+0000) 2022-09-08T07:36:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:03.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:03.595173+0000) 2022-09-08T07:36:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:04.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:04.595382+0000) 2022-09-08T07:36:04.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:04.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:04.489122+0000) 2022-09-08T07:36:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:04 smithi055 ceph-mon[97033]: pgmap v19388: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:36:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:04 smithi059 ceph-mon[97450]: pgmap v19388: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:36:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:05.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:05.595522+0000) 2022-09-08T07:36:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:05.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:05.489310+0000) 2022-09-08T07:36:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:06.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:06.595759+0000) 2022-09-08T07:36:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:06.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:06.489527+0000) 2022-09-08T07:36:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:06 smithi055 ceph-mon[97033]: pgmap v19389: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 50 KiB/s wr, 25 op/s 2022-09-08T07:36:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:36:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:06 smithi059 ceph-mon[97450]: pgmap v19389: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 50 KiB/s wr, 25 op/s 2022-09-08T07:36:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:36:07.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:07.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:07.596007+0000) 2022-09-08T07:36:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:07.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:07.489756+0000) 2022-09-08T07:36:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:36:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:36:08.598 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:36:08.262+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:36:08.600 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:08.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:08.489979+0000) 2022-09-08T07:36:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:08.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:08.596263+0000) 2022-09-08T07:36:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:08 smithi055 ceph-mon[97033]: pgmap v19390: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T07:36:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:08 smithi059 ceph-mon[97450]: pgmap v19390: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T07:36:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:09.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:09.596509+0000) 2022-09-08T07:36:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:09.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:09.490183+0000) 2022-09-08T07:36:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:10.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:10.596772+0000) 2022-09-08T07:36:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:10.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:10.490408+0000) 2022-09-08T07:36:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:10 smithi055 ceph-mon[97033]: pgmap v19391: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T07:36:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:10 smithi055 ceph-mon[97033]: osdmap e1289: 12 total, 11 up, 11 in 2022-09-08T07:36:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:10 smithi059 ceph-mon[97450]: pgmap v19391: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T07:36:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:10 smithi059 ceph-mon[97450]: osdmap e1289: 12 total, 11 up, 11 in 2022-09-08T07:36:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:11.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:11.597030+0000) 2022-09-08T07:36:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:11.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:11.490678+0000) 2022-09-08T07:36:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:12.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:12.597250+0000) 2022-09-08T07:36:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:12.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:12.490817+0000) 2022-09-08T07:36:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:12 smithi055 ceph-mon[97033]: pgmap v19393: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:36:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:12 smithi059 ceph-mon[97450]: pgmap v19393: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:36:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:13.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:13.597503+0000) 2022-09-08T07:36:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:13.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:13.490994+0000) 2022-09-08T07:36:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:14.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:14.597728+0000) 2022-09-08T07:36:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:14.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:14.491142+0000) 2022-09-08T07:36:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:14 smithi055 ceph-mon[97033]: pgmap v19394: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:36:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:14 smithi059 ceph-mon[97450]: pgmap v19394: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:36:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:15.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:15.597982+0000) 2022-09-08T07:36:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:15.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:15.491285+0000) 2022-09-08T07:36:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:16.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:16.598146+0000) 2022-09-08T07:36:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:16.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:16.491502+0000) 2022-09-08T07:36:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:16 smithi055 ceph-mon[97033]: pgmap v19395: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:36:17.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:16 smithi059 ceph-mon[97450]: pgmap v19395: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:36:17.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:17.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:17.598288+0000) 2022-09-08T07:36:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:17.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:17.491700+0000) 2022-09-08T07:36:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:18.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:18.598478+0000) 2022-09-08T07:36:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:18.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:18.491853+0000) 2022-09-08T07:36:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:18 smithi055 ceph-mon[97033]: pgmap v19396: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:36:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:18 smithi055 ceph-mon[97033]: osdmap e1290: 12 total, 11 up, 11 in 2022-09-08T07:36:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:18 smithi059 ceph-mon[97450]: pgmap v19396: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:36:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:18 smithi059 ceph-mon[97450]: osdmap e1290: 12 total, 11 up, 11 in 2022-09-08T07:36:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:19.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:19.598680+0000) 2022-09-08T07:36:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:19.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:19.491977+0000) 2022-09-08T07:36:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:20.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:20.598928+0000) 2022-09-08T07:36:20.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:20.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:20.492196+0000) 2022-09-08T07:36:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:20 smithi055 ceph-mon[97033]: pgmap v19398: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 214 B/s wr, 5 op/s 2022-09-08T07:36:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:20 smithi059 ceph-mon[97450]: pgmap v19398: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 214 B/s wr, 5 op/s 2022-09-08T07:36:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:21.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:21.492355+0000) 2022-09-08T07:36:21.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:21.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:21.599118+0000) 2022-09-08T07:36:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:22.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:22.599338+0000) 2022-09-08T07:36:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:22.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:22.492539+0000) 2022-09-08T07:36:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:22 smithi055 ceph-mon[97033]: pgmap v19399: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:36:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:22 smithi059 ceph-mon[97450]: pgmap v19399: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:36:23.601 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:36:23.278+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:36:23.602 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:23.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:23.492732+0000) 2022-09-08T07:36:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:23.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:23.599568+0000) 2022-09-08T07:36:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:24.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:24.599761+0000) 2022-09-08T07:36:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:24.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:24.492940+0000) 2022-09-08T07:36:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:24 smithi055 ceph-mon[97033]: pgmap v19400: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:36:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:24 smithi059 ceph-mon[97450]: pgmap v19400: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:36:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:25.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:25.599968+0000) 2022-09-08T07:36:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:25.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:25.493085+0000) 2022-09-08T07:36:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:26.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:26.600211+0000) 2022-09-08T07:36:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:26.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:26.493232+0000) 2022-09-08T07:36:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:26 smithi055 ceph-mon[97033]: pgmap v19401: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:36:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:26 smithi059 ceph-mon[97450]: pgmap v19401: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:36:27.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:27.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:27.600379+0000) 2022-09-08T07:36:27.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:27.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:27.493434+0000) 2022-09-08T07:36:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:28.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:28.493618+0000) 2022-09-08T07:36:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:28.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:28.600604+0000) 2022-09-08T07:36:29.238 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:28 smithi055 ceph-mon[97033]: pgmap v19402: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:36:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:28 smithi059 ceph-mon[97450]: pgmap v19402: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:36:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:29.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:29.600825+0000) 2022-09-08T07:36:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:29.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:29.493842+0000) 2022-09-08T07:36:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:30.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:30.601019+0000) 2022-09-08T07:36:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:30.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:30.494079+0000) 2022-09-08T07:36:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:30 smithi055 ceph-mon[97033]: pgmap v19403: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:36:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:30 smithi059 ceph-mon[97450]: pgmap v19403: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:36:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:31.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:31.601234+0000) 2022-09-08T07:36:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:31.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:31.494324+0000) 2022-09-08T07:36:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:32.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:32.601453+0000) 2022-09-08T07:36:32.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:32.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:32.494550+0000) 2022-09-08T07:36:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:32 smithi055 ceph-mon[97033]: pgmap v19404: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:36:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:32 smithi059 ceph-mon[97450]: pgmap v19404: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:36:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:33.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:33.601677+0000) 2022-09-08T07:36:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:33.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:33.494790+0000) 2022-09-08T07:36:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:34.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:34.601946+0000) 2022-09-08T07:36:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:34.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:34.495039+0000) 2022-09-08T07:36:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:34 smithi055 ceph-mon[97033]: pgmap v19405: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:34 smithi059 ceph-mon[97450]: pgmap v19405: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:35.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:35.602201+0000) 2022-09-08T07:36:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:35.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:35.495217+0000) 2022-09-08T07:36:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:36.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:36.495461+0000) 2022-09-08T07:36:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:36.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:36.602424+0000) 2022-09-08T07:36:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:36 smithi055 ceph-mon[97033]: pgmap v19406: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:36:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:36 smithi059 ceph-mon[97450]: pgmap v19406: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:36:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:37.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:37.495621+0000) 2022-09-08T07:36:37.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:37.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:37.602666+0000) 2022-09-08T07:36:38.605 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:36:38.281+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:36:38.606 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:38.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:38.495817+0000) 2022-09-08T07:36:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:38.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:38.602842+0000) 2022-09-08T07:36:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:38 smithi055 ceph-mon[97033]: pgmap v19407: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:38 smithi059 ceph-mon[97450]: pgmap v19407: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:39.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:39.603047+0000) 2022-09-08T07:36:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:39.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:39.495986+0000) 2022-09-08T07:36:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:40.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:40.496204+0000) 2022-09-08T07:36:40.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:40.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:40.603305+0000) 2022-09-08T07:36:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:40 smithi055 ceph-mon[97033]: pgmap v19408: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:40 smithi059 ceph-mon[97450]: pgmap v19408: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:41.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:41.603496+0000) 2022-09-08T07:36:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:41.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:41.496396+0000) 2022-09-08T07:36:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:42.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:42.603692+0000) 2022-09-08T07:36:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:42.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:42.496620+0000) 2022-09-08T07:36:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:42 smithi055 ceph-mon[97033]: pgmap v19409: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:36:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:42 smithi059 ceph-mon[97450]: pgmap v19409: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:36:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:43.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:43.603907+0000) 2022-09-08T07:36:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:43.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:43.496870+0000) 2022-09-08T07:36:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:44.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:44.604173+0000) 2022-09-08T07:36:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:44.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:44.497058+0000) 2022-09-08T07:36:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:44 smithi055 ceph-mon[97033]: pgmap v19410: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:44 smithi059 ceph-mon[97450]: pgmap v19410: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:45.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:45.604422+0000) 2022-09-08T07:36:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:45.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:45.497263+0000) 2022-09-08T07:36:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:46.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:46.604716+0000) 2022-09-08T07:36:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:46.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:46.497492+0000) 2022-09-08T07:36:47.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:46 smithi055 ceph-mon[97033]: pgmap v19411: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:36:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:46 smithi059 ceph-mon[97450]: pgmap v19411: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:36:47.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:47.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:47.604872+0000) 2022-09-08T07:36:47.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:47.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:47.497666+0000) 2022-09-08T07:36:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:48.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:48.605074+0000) 2022-09-08T07:36:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:48.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:48.497803+0000) 2022-09-08T07:36:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:48 smithi055 ceph-mon[97033]: pgmap v19412: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:48 smithi059 ceph-mon[97450]: pgmap v19412: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:49.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:49.605232+0000) 2022-09-08T07:36:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:49.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:49.497945+0000) 2022-09-08T07:36:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:50.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:50.605454+0000) 2022-09-08T07:36:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:50.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:50.498112+0000) 2022-09-08T07:36:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:50 smithi055 ceph-mon[97033]: pgmap v19413: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:50 smithi059 ceph-mon[97450]: pgmap v19413: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:51.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:51.605682+0000) 2022-09-08T07:36:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:51.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:51.498335+0000) 2022-09-08T07:36:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:52.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:52.605922+0000) 2022-09-08T07:36:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:52.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:52.498486+0000) 2022-09-08T07:36:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:52 smithi055 ceph-mon[97033]: pgmap v19414: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:36:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:52 smithi059 ceph-mon[97450]: pgmap v19414: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:36:53.608 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:36:53.296+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:36:53.609 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:53.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:53.498722+0000) 2022-09-08T07:36:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:53.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:53.606099+0000) 2022-09-08T07:36:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:54.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:54.606340+0000) 2022-09-08T07:36:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:54.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:54.498877+0000) 2022-09-08T07:36:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:54 smithi055 ceph-mon[97033]: pgmap v19415: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:54 smithi059 ceph-mon[97450]: pgmap v19415: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:55.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:55.606588+0000) 2022-09-08T07:36:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:55.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:55.499001+0000) 2022-09-08T07:36:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:56.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:56.606805+0000) 2022-09-08T07:36:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:56.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:56.499148+0000) 2022-09-08T07:36:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:56 smithi055 ceph-mon[97033]: pgmap v19416: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:36:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:56 smithi059 ceph-mon[97450]: pgmap v19416: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:36:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:57.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:57.606934+0000) 2022-09-08T07:36:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:57.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:57.499353+0000) 2022-09-08T07:36:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:58.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:58.607086+0000) 2022-09-08T07:36:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:58.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:58.499569+0000) 2022-09-08T07:36:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:58 smithi055 ceph-mon[97033]: pgmap v19417: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:58 smithi059 ceph-mon[97450]: pgmap v19417: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:36:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:36:59.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:59.499716+0000) 2022-09-08T07:36:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:36:59.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:36:59.607338+0000) 2022-09-08T07:37:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:37:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:37:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:36:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:37:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:37:00.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:37:00.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:36:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:37:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:00.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:00.607518+0000) 2022-09-08T07:37:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:00.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:00.499865+0000) 2022-09-08T07:37:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:00 smithi055 ceph-mon[97033]: pgmap v19418: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:37:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:37:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:00 smithi059 ceph-mon[97450]: pgmap v19418: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:37:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:37:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:01.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:01.607736+0000) 2022-09-08T07:37:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:01.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:01.500038+0000) 2022-09-08T07:37:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:02.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:02.607934+0000) 2022-09-08T07:37:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:02.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:02.500257+0000) 2022-09-08T07:37:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:02 smithi055 ceph-mon[97033]: pgmap v19419: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:37:03.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:37:03.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:37:03.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:03.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:03.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:03.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:03.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:37:03.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:02 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:37:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:02 smithi059 ceph-mon[97450]: pgmap v19419: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:37:03.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:37:03.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:37:03.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:03.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:03.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:03.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:03.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:37:03.288 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:02 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:37:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:03.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:03.608124+0000) 2022-09-08T07:37:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:03.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:03.500471+0000) 2022-09-08T07:37:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:03 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 120.4M 2022-09-08T07:37:04.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:03 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 126348595: error parsing value: Value '126348595' is below minimum 939524096 2022-09-08T07:37:04.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:04.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:04.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:04.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:04.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:37:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:03 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 120.4M 2022-09-08T07:37:04.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:03 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 126348595: error parsing value: Value '126348595' is below minimum 939524096 2022-09-08T07:37:04.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:04.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:04.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:04.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:37:04.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:37:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:04.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:04.608371+0000) 2022-09-08T07:37:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:04.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:04.500626+0000) 2022-09-08T07:37:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:04 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T07:37:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:04 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T07:37:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:04 smithi059 ceph-mon[97450]: pgmap v19420: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:04 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T07:37:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:04 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T07:37:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:04 smithi055 ceph-mon[97033]: pgmap v19420: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:05.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:05.608569+0000) 2022-09-08T07:37:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:05.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:05.500861+0000) 2022-09-08T07:37:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:06.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:06.608758+0000) 2022-09-08T07:37:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:06.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:06.500983+0000) 2022-09-08T07:37:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:06 smithi059 ceph-mon[97450]: pgmap v19421: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:37:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:37:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:37:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:06 smithi055 ceph-mon[97033]: pgmap v19421: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:37:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:37:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:37:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:07.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:07.608920+0000) 2022-09-08T07:37:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:07.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:07.501167+0000) 2022-09-08T07:37:08.611 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:37:08.312+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:37:08.612 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:08.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:08.501340+0000) 2022-09-08T07:37:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:08.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:08.609168+0000) 2022-09-08T07:37:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:08 smithi055 ceph-mon[97033]: pgmap v19422: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:37:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:08 smithi059 ceph-mon[97450]: pgmap v19422: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:37:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:09.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:09.609459+0000) 2022-09-08T07:37:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:09.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:09.501504+0000) 2022-09-08T07:37:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:10.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:10.609674+0000) 2022-09-08T07:37:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:10.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:10.501778+0000) 2022-09-08T07:37:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:10 smithi055 ceph-mon[97033]: pgmap v19423: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:37:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:10 smithi055 ceph-mon[97033]: osdmap e1291: 12 total, 11 up, 11 in 2022-09-08T07:37:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:10 smithi059 ceph-mon[97450]: pgmap v19423: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:37:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:10 smithi059 ceph-mon[97450]: osdmap e1291: 12 total, 11 up, 11 in 2022-09-08T07:37:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:11.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:11.609905+0000) 2022-09-08T07:37:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:11.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:11.501956+0000) 2022-09-08T07:37:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:12.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:12.502199+0000) 2022-09-08T07:37:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:12.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:12.610082+0000) 2022-09-08T07:37:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:12 smithi055 ceph-mon[97033]: pgmap v19425: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:37:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:12 smithi059 ceph-mon[97450]: pgmap v19425: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:37:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:13.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:13.610273+0000) 2022-09-08T07:37:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:13.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:13.502378+0000) 2022-09-08T07:37:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:14.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:14.610494+0000) 2022-09-08T07:37:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:14.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:14.502632+0000) 2022-09-08T07:37:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:14 smithi055 ceph-mon[97033]: pgmap v19426: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:37:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:14 smithi059 ceph-mon[97450]: pgmap v19426: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:37:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:15.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:15.610691+0000) 2022-09-08T07:37:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:15.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:15.502891+0000) 2022-09-08T07:37:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:16.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:16.610917+0000) 2022-09-08T07:37:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:16.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:16.503086+0000) 2022-09-08T07:37:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:16 smithi055 ceph-mon[97033]: pgmap v19427: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:37:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:16 smithi059 ceph-mon[97450]: pgmap v19427: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:37:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:17.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:17.611058+0000) 2022-09-08T07:37:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:17.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:17.503276+0000) 2022-09-08T07:37:18.907 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:18.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:18.611178+0000) 2022-09-08T07:37:18.908 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:18.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:18.503498+0000) 2022-09-08T07:37:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:18 smithi055 ceph-mon[97033]: pgmap v19428: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:37:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:18 smithi055 ceph-mon[97033]: osdmap e1292: 12 total, 11 up, 11 in 2022-09-08T07:37:19.290 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:18 smithi059 ceph-mon[97450]: pgmap v19428: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:37:19.291 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:18 smithi059 ceph-mon[97450]: osdmap e1292: 12 total, 11 up, 11 in 2022-09-08T07:37:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:19.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:19.611366+0000) 2022-09-08T07:37:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:19.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:19.503763+0000) 2022-09-08T07:37:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:20.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:20.611597+0000) 2022-09-08T07:37:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:20.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:20.504014+0000) 2022-09-08T07:37:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:20 smithi055 ceph-mon[97033]: pgmap v19430: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 214 B/s wr, 5 op/s 2022-09-08T07:37:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:20 smithi059 ceph-mon[97450]: pgmap v19430: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 214 B/s wr, 5 op/s 2022-09-08T07:37:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:21.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:21.611828+0000) 2022-09-08T07:37:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:21.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:21.504303+0000) 2022-09-08T07:37:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:22.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:22.612029+0000) 2022-09-08T07:37:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:22.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:22.504464+0000) 2022-09-08T07:37:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:22 smithi055 ceph-mon[97033]: pgmap v19431: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:37:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:22 smithi059 ceph-mon[97450]: pgmap v19431: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:37:23.614 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:37:23.328+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:37:23.615 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:23.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:23.504732+0000) 2022-09-08T07:37:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:23.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:23.612258+0000) 2022-09-08T07:37:24.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:24.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:24.612497+0000) 2022-09-08T07:37:24.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:24.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:24.504888+0000) 2022-09-08T07:37:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:24 smithi055 ceph-mon[97033]: pgmap v19432: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:37:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:24 smithi059 ceph-mon[97450]: pgmap v19432: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:37:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:25.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:25.612789+0000) 2022-09-08T07:37:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:25.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:25.505096+0000) 2022-09-08T07:37:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:26.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:26.613062+0000) 2022-09-08T07:37:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:26.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:26.505359+0000) 2022-09-08T07:37:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:26 smithi055 ceph-mon[97033]: pgmap v19433: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:37:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:26 smithi059 ceph-mon[97450]: pgmap v19433: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:37:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:27.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:27.613298+0000) 2022-09-08T07:37:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:27.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:27.505568+0000) 2022-09-08T07:37:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:28.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:28.613488+0000) 2022-09-08T07:37:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:28.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:28.505790+0000) 2022-09-08T07:37:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:28 smithi055 ceph-mon[97033]: pgmap v19434: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:37:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:28 smithi059 ceph-mon[97450]: pgmap v19434: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:37:29.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:29.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:29.613754+0000) 2022-09-08T07:37:29.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:29.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:29.506019+0000) 2022-09-08T07:37:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:30.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:30.614032+0000) 2022-09-08T07:37:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:30.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:30.506233+0000) 2022-09-08T07:37:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:30 smithi055 ceph-mon[97033]: pgmap v19435: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:37:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:30 smithi059 ceph-mon[97450]: pgmap v19435: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:37:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:31.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:31.614171+0000) 2022-09-08T07:37:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:31.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:31.506396+0000) 2022-09-08T07:37:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:32.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:32.614405+0000) 2022-09-08T07:37:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:32.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:32.506546+0000) 2022-09-08T07:37:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:32 smithi055 ceph-mon[97033]: pgmap v19436: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:37:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:32 smithi059 ceph-mon[97450]: pgmap v19436: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:37:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:33.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:33.614530+0000) 2022-09-08T07:37:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:33.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:33.506714+0000) 2022-09-08T07:37:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:34.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:34.506960+0000) 2022-09-08T07:37:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:34.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:34.614779+0000) 2022-09-08T07:37:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:34 smithi055 ceph-mon[97033]: pgmap v19437: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:34 smithi059 ceph-mon[97450]: pgmap v19437: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:35.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:35.614953+0000) 2022-09-08T07:37:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:35.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:35.507171+0000) 2022-09-08T07:37:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:36.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:36.615110+0000) 2022-09-08T07:37:36.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:36.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:36.507383+0000) 2022-09-08T07:37:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:36 smithi055 ceph-mon[97033]: pgmap v19438: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:37:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:36 smithi059 ceph-mon[97450]: pgmap v19438: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:37:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:37.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:37.615334+0000) 2022-09-08T07:37:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:37.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:37.507576+0000) 2022-09-08T07:37:38.617 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:37:38.331+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:37:38.618 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:38.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:38.507736+0000) 2022-09-08T07:37:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:38.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:38.615483+0000) 2022-09-08T07:37:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:38 smithi055 ceph-mon[97033]: pgmap v19439: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:38 smithi059 ceph-mon[97450]: pgmap v19439: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:39.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:39.615748+0000) 2022-09-08T07:37:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:39.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:39.508005+0000) 2022-09-08T07:37:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:40.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:40.615996+0000) 2022-09-08T07:37:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:40.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:40.508232+0000) 2022-09-08T07:37:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:40 smithi055 ceph-mon[97033]: pgmap v19440: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:40 smithi059 ceph-mon[97450]: pgmap v19440: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:41.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:41.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:41.616201+0000) 2022-09-08T07:37:41.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:41.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:41.508481+0000) 2022-09-08T07:37:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:42.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:42.616440+0000) 2022-09-08T07:37:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:42.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:42.508678+0000) 2022-09-08T07:37:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:42 smithi055 ceph-mon[97033]: pgmap v19441: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:37:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:42 smithi059 ceph-mon[97450]: pgmap v19441: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:37:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:43.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:43.616697+0000) 2022-09-08T07:37:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:43.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:43.508827+0000) 2022-09-08T07:37:44.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:44.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:44.616836+0000) 2022-09-08T07:37:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:44.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:44.509039+0000) 2022-09-08T07:37:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:44 smithi055 ceph-mon[97033]: pgmap v19442: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:44 smithi059 ceph-mon[97450]: pgmap v19442: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:45.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:45.509291+0000) 2022-09-08T07:37:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:45.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:45.617006+0000) 2022-09-08T07:37:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:46.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:46.617219+0000) 2022-09-08T07:37:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:46.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:46.509543+0000) 2022-09-08T07:37:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:46 smithi055 ceph-mon[97033]: pgmap v19443: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:37:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:46 smithi059 ceph-mon[97450]: pgmap v19443: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:37:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:47.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:47.617391+0000) 2022-09-08T07:37:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:47.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:47.509756+0000) 2022-09-08T07:37:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:48.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:48.617521+0000) 2022-09-08T07:37:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:48.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:48.510021+0000) 2022-09-08T07:37:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:48 smithi055 ceph-mon[97033]: pgmap v19444: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:48 smithi059 ceph-mon[97450]: pgmap v19444: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:49.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:49.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:49.617760+0000) 2022-09-08T07:37:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:49.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:49.510173+0000) 2022-09-08T07:37:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:50.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:50.617991+0000) 2022-09-08T07:37:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:50.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:50.510345+0000) 2022-09-08T07:37:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:50 smithi055 ceph-mon[97033]: pgmap v19445: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:50 smithi059 ceph-mon[97450]: pgmap v19445: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:51.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:51.618192+0000) 2022-09-08T07:37:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:51.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:51.510496+0000) 2022-09-08T07:37:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:52.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:52.618399+0000) 2022-09-08T07:37:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:52.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:52.510756+0000) 2022-09-08T07:37:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:52 smithi055 ceph-mon[97033]: pgmap v19446: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:37:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:52 smithi059 ceph-mon[97450]: pgmap v19446: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:37:53.620 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:37:53.348+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:37:53.621 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:53.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:53.510960+0000) 2022-09-08T07:37:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:53.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:53.618604+0000) 2022-09-08T07:37:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:54.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:54.511202+0000) 2022-09-08T07:37:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:54.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:54.618852+0000) 2022-09-08T07:37:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:54 smithi055 ceph-mon[97033]: pgmap v19447: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:54 smithi059 ceph-mon[97450]: pgmap v19447: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:55.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:55.619104+0000) 2022-09-08T07:37:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:55.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:55.511369+0000) 2022-09-08T07:37:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:56.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:56.619359+0000) 2022-09-08T07:37:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:56.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:56.511514+0000) 2022-09-08T07:37:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:56 smithi055 ceph-mon[97033]: pgmap v19448: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:37:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:56 smithi059 ceph-mon[97450]: pgmap v19448: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:37:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:57.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:57.619615+0000) 2022-09-08T07:37:57.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:57.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:57.511746+0000) 2022-09-08T07:37:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:58.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:58.619896+0000) 2022-09-08T07:37:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:58.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:58.512006+0000) 2022-09-08T07:37:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:37:58 smithi055 ceph-mon[97033]: pgmap v19449: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:37:58 smithi059 ceph-mon[97450]: pgmap v19449: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:37:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:37:59.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:59.620050+0000) 2022-09-08T07:37:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:37:59.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:37:59.512181+0000) 2022-09-08T07:38:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:00.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:00.620297+0000) 2022-09-08T07:38:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:00.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:00.512322+0000) 2022-09-08T07:38:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:00 smithi055 ceph-mon[97033]: pgmap v19450: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:00 smithi059 ceph-mon[97450]: pgmap v19450: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:01.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:01.620494+0000) 2022-09-08T07:38:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:01.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:01.512532+0000) 2022-09-08T07:38:02.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:02.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:02.620713+0000) 2022-09-08T07:38:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:02.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:02.512758+0000) 2022-09-08T07:38:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:02 smithi055 ceph-mon[97033]: pgmap v19451: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:38:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:02 smithi059 ceph-mon[97450]: pgmap v19451: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:38:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:03.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:03.620886+0000) 2022-09-08T07:38:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:03.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:03.512906+0000) 2022-09-08T07:38:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:38:04.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:38:04.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:03 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:38:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:38:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:38:04.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:03 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:38:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:04.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:04.621113+0000) 2022-09-08T07:38:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:04.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:04.513096+0000) 2022-09-08T07:38:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:04 smithi055 ceph-mon[97033]: pgmap v19452: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:05.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:38:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:04 smithi059 ceph-mon[97450]: pgmap v19452: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:38:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:05.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:05.621320+0000) 2022-09-08T07:38:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:05.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:05.513301+0000) 2022-09-08T07:38:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:06.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:06.621518+0000) 2022-09-08T07:38:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:06.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:06.513515+0000) 2022-09-08T07:38:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:06 smithi055 ceph-mon[97033]: pgmap v19453: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:38:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:38:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:38:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:06 smithi059 ceph-mon[97450]: pgmap v19453: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:38:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:38:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:38:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:07.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:07.621727+0000) 2022-09-08T07:38:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:07.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:07.513697+0000) 2022-09-08T07:38:08.624 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:38:08.353+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:38:08.625 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:08.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:08.513829+0000) 2022-09-08T07:38:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:08.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:08.621867+0000) 2022-09-08T07:38:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:08 smithi055 ceph-mon[97033]: pgmap v19454: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:38:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:08 smithi059 ceph-mon[97450]: pgmap v19454: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:38:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:09.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:09.622123+0000) 2022-09-08T07:38:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:09.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:09.514054+0000) 2022-09-08T07:38:10.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:10.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:10.514318+0000) 2022-09-08T07:38:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:10.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:10.622372+0000) 2022-09-08T07:38:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:10 smithi055 ceph-mon[97033]: pgmap v19455: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:38:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:10 smithi055 ceph-mon[97033]: osdmap e1293: 12 total, 11 up, 11 in 2022-09-08T07:38:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:10 smithi059 ceph-mon[97450]: pgmap v19455: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:38:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:10 smithi059 ceph-mon[97450]: osdmap e1293: 12 total, 11 up, 11 in 2022-09-08T07:38:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:11.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:11.622569+0000) 2022-09-08T07:38:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:11.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:11.514597+0000) 2022-09-08T07:38:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:12.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:12.622844+0000) 2022-09-08T07:38:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:12.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:12.514721+0000) 2022-09-08T07:38:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:12 smithi055 ceph-mon[97033]: pgmap v19457: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:38:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:12 smithi059 ceph-mon[97450]: pgmap v19457: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:38:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:13.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:13.623025+0000) 2022-09-08T07:38:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:13.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:13.514937+0000) 2022-09-08T07:38:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:14.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:14.623221+0000) 2022-09-08T07:38:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:14.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:14.515100+0000) 2022-09-08T07:38:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:14 smithi055 ceph-mon[97033]: pgmap v19458: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:38:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:14 smithi059 ceph-mon[97450]: pgmap v19458: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:38:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:15.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:15.623441+0000) 2022-09-08T07:38:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:15.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:15.515243+0000) 2022-09-08T07:38:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:16.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:16.623695+0000) 2022-09-08T07:38:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:16.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:16.515486+0000) 2022-09-08T07:38:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:16 smithi059 ceph-mon[97450]: pgmap v19459: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:38:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:16 smithi055 ceph-mon[97033]: pgmap v19459: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:38:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:17.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:17.623936+0000) 2022-09-08T07:38:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:17.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:17.515733+0000) 2022-09-08T07:38:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:18.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:18.624144+0000) 2022-09-08T07:38:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:18.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:18.515911+0000) 2022-09-08T07:38:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:18 smithi055 ceph-mon[97033]: pgmap v19460: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:38:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:18 smithi055 ceph-mon[97033]: osdmap e1294: 12 total, 11 up, 11 in 2022-09-08T07:38:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:18 smithi059 ceph-mon[97450]: pgmap v19460: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:38:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:18 smithi059 ceph-mon[97450]: osdmap e1294: 12 total, 11 up, 11 in 2022-09-08T07:38:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:19.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:19.624307+0000) 2022-09-08T07:38:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:19.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:19.516060+0000) 2022-09-08T07:38:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:20.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:20.624477+0000) 2022-09-08T07:38:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:20.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:20.516307+0000) 2022-09-08T07:38:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:20 smithi055 ceph-mon[97033]: pgmap v19462: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 108 B/s wr, 4 op/s 2022-09-08T07:38:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:20 smithi059 ceph-mon[97450]: pgmap v19462: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.2 KiB/s rd, 108 B/s wr, 4 op/s 2022-09-08T07:38:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:21.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:21.624718+0000) 2022-09-08T07:38:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:21.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:21.516516+0000) 2022-09-08T07:38:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:22.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:22.624949+0000) 2022-09-08T07:38:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:22.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:22.516687+0000) 2022-09-08T07:38:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:22 smithi055 ceph-mon[97033]: pgmap v19463: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:38:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:22 smithi059 ceph-mon[97450]: pgmap v19463: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:38:23.627 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:38:23.367+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:38:23.628 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:23.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:23.516831+0000) 2022-09-08T07:38:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:23.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:23.625160+0000) 2022-09-08T07:38:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:24.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:24.625344+0000) 2022-09-08T07:38:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:24.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:24.516996+0000) 2022-09-08T07:38:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:24 smithi055 ceph-mon[97033]: pgmap v19464: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:38:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:24 smithi059 ceph-mon[97450]: pgmap v19464: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:38:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:25.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:25.517176+0000) 2022-09-08T07:38:25.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:25.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:25.625603+0000) 2022-09-08T07:38:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:26.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:26.625908+0000) 2022-09-08T07:38:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:26.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:26.517336+0000) 2022-09-08T07:38:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:26 smithi055 ceph-mon[97033]: pgmap v19465: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:38:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:26 smithi059 ceph-mon[97450]: pgmap v19465: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:38:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:27.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:27.626101+0000) 2022-09-08T07:38:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:27.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:27.517529+0000) 2022-09-08T07:38:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:28.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:28.626308+0000) 2022-09-08T07:38:28.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:28.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:28.517757+0000) 2022-09-08T07:38:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:28 smithi055 ceph-mon[97033]: pgmap v19466: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:38:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:28 smithi059 ceph-mon[97450]: pgmap v19466: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:38:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:29.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:29.626451+0000) 2022-09-08T07:38:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:29.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:29.517999+0000) 2022-09-08T07:38:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:30.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:30.626704+0000) 2022-09-08T07:38:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:30.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:30.518176+0000) 2022-09-08T07:38:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:30 smithi055 ceph-mon[97033]: pgmap v19467: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:38:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:30 smithi059 ceph-mon[97450]: pgmap v19467: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:38:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:31.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:31.626843+0000) 2022-09-08T07:38:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:31.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:31.518409+0000) 2022-09-08T07:38:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:32.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:32.627081+0000) 2022-09-08T07:38:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:32.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:32.518579+0000) 2022-09-08T07:38:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:32 smithi055 ceph-mon[97033]: pgmap v19468: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:38:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:32 smithi059 ceph-mon[97450]: pgmap v19468: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:38:33.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:33.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:33.627279+0000) 2022-09-08T07:38:33.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:33.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:33.518729+0000) 2022-09-08T07:38:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:34.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:34.518877+0000) 2022-09-08T07:38:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:34.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:34.627553+0000) 2022-09-08T07:38:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:34 smithi055 ceph-mon[97033]: pgmap v19469: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:34 smithi059 ceph-mon[97450]: pgmap v19469: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:35.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:35.627826+0000) 2022-09-08T07:38:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:35.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:35.519054+0000) 2022-09-08T07:38:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:36.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:36.519197+0000) 2022-09-08T07:38:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:36.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:36.628099+0000) 2022-09-08T07:38:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:36 smithi055 ceph-mon[97033]: pgmap v19470: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:38:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:36 smithi059 ceph-mon[97450]: pgmap v19470: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:38:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:37.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:37.628357+0000) 2022-09-08T07:38:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:37.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:37.519377+0000) 2022-09-08T07:38:38.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:38.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:38.519507+0000) 2022-09-08T07:38:38.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:38:38.384+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:38:38.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:38.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:38.628555+0000) 2022-09-08T07:38:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:38 smithi055 ceph-mon[97033]: pgmap v19471: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:38 smithi059 ceph-mon[97450]: pgmap v19471: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:39.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:39.628798+0000) 2022-09-08T07:38:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:39.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:39.519688+0000) 2022-09-08T07:38:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:40.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:40.629033+0000) 2022-09-08T07:38:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:40.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:40.519922+0000) 2022-09-08T07:38:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:40 smithi055 ceph-mon[97033]: pgmap v19472: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:40 smithi059 ceph-mon[97450]: pgmap v19472: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:41.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:41.629276+0000) 2022-09-08T07:38:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:41.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:41.520144+0000) 2022-09-08T07:38:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:42.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:42.629504+0000) 2022-09-08T07:38:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:42.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:42.520395+0000) 2022-09-08T07:38:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:42 smithi055 ceph-mon[97033]: pgmap v19473: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:38:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:42 smithi059 ceph-mon[97450]: pgmap v19473: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:38:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:43.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:43.629711+0000) 2022-09-08T07:38:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:43.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:43.520547+0000) 2022-09-08T07:38:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:44.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:44.629871+0000) 2022-09-08T07:38:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:44.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:44.520722+0000) 2022-09-08T07:38:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:44 smithi055 ceph-mon[97033]: pgmap v19474: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:44 smithi059 ceph-mon[97450]: pgmap v19474: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:45.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:45.630017+0000) 2022-09-08T07:38:45.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:45.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:45.520888+0000) 2022-09-08T07:38:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:46.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:46.630248+0000) 2022-09-08T07:38:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:46.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:46.521043+0000) 2022-09-08T07:38:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:46 smithi055 ceph-mon[97033]: pgmap v19475: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:38:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:46 smithi059 ceph-mon[97450]: pgmap v19475: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:38:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:47.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:47.630472+0000) 2022-09-08T07:38:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:47.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:47.521249+0000) 2022-09-08T07:38:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:48.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:48.521405+0000) 2022-09-08T07:38:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:48.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:48.630699+0000) 2022-09-08T07:38:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:48 smithi055 ceph-mon[97033]: pgmap v19476: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:48 smithi059 ceph-mon[97450]: pgmap v19476: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:49.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:49.630830+0000) 2022-09-08T07:38:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:49.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:49.521575+0000) 2022-09-08T07:38:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:50.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:50.631071+0000) 2022-09-08T07:38:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:50.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:50.521783+0000) 2022-09-08T07:38:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:50 smithi055 ceph-mon[97033]: pgmap v19477: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:50 smithi059 ceph-mon[97450]: pgmap v19477: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:51.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:51.631308+0000) 2022-09-08T07:38:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:51.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:51.521990+0000) 2022-09-08T07:38:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:52.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:52.522139+0000) 2022-09-08T07:38:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:52.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:52.631586+0000) 2022-09-08T07:38:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:52 smithi055 ceph-mon[97033]: pgmap v19478: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:38:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:52 smithi059 ceph-mon[97450]: pgmap v19478: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:38:53.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:38:53.399+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:38:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:53.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:53.522344+0000) 2022-09-08T07:38:53.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:53.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:53.631877+0000) 2022-09-08T07:38:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:54.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:54.632099+0000) 2022-09-08T07:38:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:54.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:54.522550+0000) 2022-09-08T07:38:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:54 smithi055 ceph-mon[97033]: pgmap v19479: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:54 smithi059 ceph-mon[97450]: pgmap v19479: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:55.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:55.632343+0000) 2022-09-08T07:38:55.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:55.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:55.522776+0000) 2022-09-08T07:38:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:56.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:56.632518+0000) 2022-09-08T07:38:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:56.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:56.523018+0000) 2022-09-08T07:38:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:56 smithi055 ceph-mon[97033]: pgmap v19480: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:38:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:56 smithi059 ceph-mon[97450]: pgmap v19480: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:38:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:57.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:57.632746+0000) 2022-09-08T07:38:57.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:57.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:57.523187+0000) 2022-09-08T07:38:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:58.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:58.632974+0000) 2022-09-08T07:38:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:58.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:58.523342+0000) 2022-09-08T07:38:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:38:58 smithi055 ceph-mon[97033]: pgmap v19481: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:38:58 smithi059 ceph-mon[97450]: pgmap v19481: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:38:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:38:59.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:59.523463+0000) 2022-09-08T07:38:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:38:59.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:38:59.633189+0000) 2022-09-08T07:39:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:00.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:00.523630+0000) 2022-09-08T07:39:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:00.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:00.633314+0000) 2022-09-08T07:39:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:00 smithi055 ceph-mon[97033]: pgmap v19482: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:00 smithi059 ceph-mon[97450]: pgmap v19482: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:01.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:01.633537+0000) 2022-09-08T07:39:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:01.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:01.523829+0000) 2022-09-08T07:39:02.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:02.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:02.633751+0000) 2022-09-08T07:39:02.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:02.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:02.523966+0000) 2022-09-08T07:39:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:02 smithi055 ceph-mon[97033]: pgmap v19483: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:39:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:02 smithi059 ceph-mon[97450]: pgmap v19483: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:39:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:03.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:03.633986+0000) 2022-09-08T07:39:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:03.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:03.524120+0000) 2022-09-08T07:39:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:04.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:04.634236+0000) 2022-09-08T07:39:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:04.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:04.524295+0000) 2022-09-08T07:39:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:04 smithi059 ceph-mon[97450]: pgmap v19484: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:39:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:39:05.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:39:05.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:39:05.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:39:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:04 smithi055 ceph-mon[97033]: pgmap v19484: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:39:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:39:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:39:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:39:05.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:39:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:05.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:05.634457+0000) 2022-09-08T07:39:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:05.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:05.524475+0000) 2022-09-08T07:39:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:39:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:06 smithi055 ceph-mon[97033]: pgmap v19485: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:39:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:39:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:39:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:06 smithi059 ceph-mon[97450]: pgmap v19485: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:39:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:39:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:06.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:06.634697+0000) 2022-09-08T07:39:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:06.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:06.524607+0000) 2022-09-08T07:39:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:39:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:39:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:07.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:07.634929+0000) 2022-09-08T07:39:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:07.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:07.524796+0000) 2022-09-08T07:39:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:08.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:08.635197+0000) 2022-09-08T07:39:08.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:39:08.409+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:39:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:08.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:08.524956+0000) 2022-09-08T07:39:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:08 smithi055 ceph-mon[97033]: pgmap v19486: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:39:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:08 smithi059 ceph-mon[97450]: pgmap v19486: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:39:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:09.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:09.635377+0000) 2022-09-08T07:39:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:09.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:09.525153+0000) 2022-09-08T07:39:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:10.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:10.635632+0000) 2022-09-08T07:39:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:10.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:10.525304+0000) 2022-09-08T07:39:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:10 smithi055 ceph-mon[97033]: pgmap v19487: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:39:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:10 smithi055 ceph-mon[97033]: osdmap e1295: 12 total, 11 up, 11 in 2022-09-08T07:39:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:10 smithi059 ceph-mon[97450]: pgmap v19487: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:39:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:10 smithi059 ceph-mon[97450]: osdmap e1295: 12 total, 11 up, 11 in 2022-09-08T07:39:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:11.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:11.525461+0000) 2022-09-08T07:39:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:11.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:11.635877+0000) 2022-09-08T07:39:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:12.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:12.636084+0000) 2022-09-08T07:39:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:12.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:12.525603+0000) 2022-09-08T07:39:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:12 smithi055 ceph-mon[97033]: pgmap v19489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:39:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:12 smithi059 ceph-mon[97450]: pgmap v19489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:39:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:13.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:13.636294+0000) 2022-09-08T07:39:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:13.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:13.525782+0000) 2022-09-08T07:39:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:14.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:14.636419+0000) 2022-09-08T07:39:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:14.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:14.526036+0000) 2022-09-08T07:39:15.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:14 smithi055 ceph-mon[97033]: pgmap v19490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:39:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:14 smithi059 ceph-mon[97450]: pgmap v19490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:39:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:15.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:15.636694+0000) 2022-09-08T07:39:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:15.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:15.526282+0000) 2022-09-08T07:39:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:16.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:16.636972+0000) 2022-09-08T07:39:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:16.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:16.526534+0000) 2022-09-08T07:39:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:16 smithi055 ceph-mon[97033]: pgmap v19491: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:39:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:16 smithi059 ceph-mon[97450]: pgmap v19491: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:39:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:17.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:17.637181+0000) 2022-09-08T07:39:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:17.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:17.526765+0000) 2022-09-08T07:39:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:18.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:18.637342+0000) 2022-09-08T07:39:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:18.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:18.526994+0000) 2022-09-08T07:39:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:18 smithi055 ceph-mon[97033]: pgmap v19492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:39:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:18 smithi055 ceph-mon[97033]: osdmap e1296: 12 total, 11 up, 11 in 2022-09-08T07:39:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:18 smithi059 ceph-mon[97450]: pgmap v19492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:39:19.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:18 smithi059 ceph-mon[97450]: osdmap e1296: 12 total, 11 up, 11 in 2022-09-08T07:39:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:19.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:19.637548+0000) 2022-09-08T07:39:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:19.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:19.527184+0000) 2022-09-08T07:39:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:20.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:20.637784+0000) 2022-09-08T07:39:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:20.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:20.527358+0000) 2022-09-08T07:39:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:20 smithi055 ceph-mon[97033]: pgmap v19494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 214 B/s wr, 5 op/s 2022-09-08T07:39:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:20 smithi059 ceph-mon[97450]: pgmap v19494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.3 KiB/s rd, 214 B/s wr, 5 op/s 2022-09-08T07:39:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:21.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:21.638005+0000) 2022-09-08T07:39:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:21.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:21.527581+0000) 2022-09-08T07:39:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:22.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:22.638237+0000) 2022-09-08T07:39:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:22.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:22.527786+0000) 2022-09-08T07:39:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:22 smithi055 ceph-mon[97033]: pgmap v19495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:39:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:22 smithi059 ceph-mon[97450]: pgmap v19495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:39:23.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:39:23.418+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:39:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:23.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:23.638485+0000) 2022-09-08T07:39:23.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:23.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:23.527993+0000) 2022-09-08T07:39:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:24.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:24.638732+0000) 2022-09-08T07:39:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:24.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:24.528140+0000) 2022-09-08T07:39:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:24 smithi055 ceph-mon[97033]: pgmap v19496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:39:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:24 smithi059 ceph-mon[97450]: pgmap v19496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:39:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:25.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:25.638931+0000) 2022-09-08T07:39:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:25.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:25.528316+0000) 2022-09-08T07:39:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:26.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:26.639046+0000) 2022-09-08T07:39:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:26.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:26.528463+0000) 2022-09-08T07:39:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:26 smithi055 ceph-mon[97033]: pgmap v19497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:39:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:26 smithi059 ceph-mon[97450]: pgmap v19497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:39:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:27.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:27.639248+0000) 2022-09-08T07:39:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:27.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:27.528604+0000) 2022-09-08T07:39:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:28.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:28.639414+0000) 2022-09-08T07:39:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:28.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:28.528789+0000) 2022-09-08T07:39:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:28 smithi055 ceph-mon[97033]: pgmap v19498: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:39:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:28 smithi059 ceph-mon[97450]: pgmap v19498: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:39:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:29.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:29.639605+0000) 2022-09-08T07:39:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:29.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:29.528958+0000) 2022-09-08T07:39:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:30.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:30.639831+0000) 2022-09-08T07:39:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:30.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:30.529112+0000) 2022-09-08T07:39:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:30 smithi055 ceph-mon[97033]: pgmap v19499: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:39:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:30 smithi059 ceph-mon[97450]: pgmap v19499: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:39:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:31.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:31.640064+0000) 2022-09-08T07:39:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:31.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:31.529275+0000) 2022-09-08T07:39:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:32.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:32.640296+0000) 2022-09-08T07:39:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:32.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:32.529459+0000) 2022-09-08T07:39:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:32 smithi055 ceph-mon[97033]: pgmap v19500: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:39:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:32 smithi059 ceph-mon[97450]: pgmap v19500: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:39:33.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:33.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:33.640442+0000) 2022-09-08T07:39:33.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:33.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:33.529704+0000) 2022-09-08T07:39:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:34.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:34.640689+0000) 2022-09-08T07:39:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:34.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:34.529851+0000) 2022-09-08T07:39:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:34 smithi055 ceph-mon[97033]: pgmap v19501: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:34 smithi059 ceph-mon[97450]: pgmap v19501: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:35.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:35.640954+0000) 2022-09-08T07:39:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:35.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:35.530040+0000) 2022-09-08T07:39:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:36.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:36.641173+0000) 2022-09-08T07:39:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:36.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:36.530271+0000) 2022-09-08T07:39:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:36 smithi055 ceph-mon[97033]: pgmap v19502: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:39:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:36 smithi059 ceph-mon[97450]: pgmap v19502: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:39:37.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:37.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:37.641392+0000) 2022-09-08T07:39:37.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:37.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:37.530497+0000) 2022-09-08T07:39:38.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:39:38.421+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:39:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:38.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:38.641610+0000) 2022-09-08T07:39:38.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:38.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:38.530705+0000) 2022-09-08T07:39:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:38 smithi055 ceph-mon[97033]: pgmap v19503: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:38 smithi059 ceph-mon[97450]: pgmap v19503: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:39.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:39.641785+0000) 2022-09-08T07:39:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:39.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:39.530835+0000) 2022-09-08T07:39:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:40.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:40.641902+0000) 2022-09-08T07:39:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:40.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:40.531022+0000) 2022-09-08T07:39:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:40 smithi055 ceph-mon[97033]: pgmap v19504: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:40 smithi059 ceph-mon[97450]: pgmap v19504: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:41.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:41.642119+0000) 2022-09-08T07:39:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:41.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:41.531176+0000) 2022-09-08T07:39:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:42.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:42.642335+0000) 2022-09-08T07:39:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:42.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:42.531363+0000) 2022-09-08T07:39:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:42 smithi055 ceph-mon[97033]: pgmap v19505: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:39:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:42 smithi059 ceph-mon[97450]: pgmap v19505: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:39:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:43.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:43.642554+0000) 2022-09-08T07:39:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:43.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:43.531583+0000) 2022-09-08T07:39:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:44.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:44.642766+0000) 2022-09-08T07:39:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:44.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:44.531741+0000) 2022-09-08T07:39:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:44 smithi055 ceph-mon[97033]: pgmap v19506: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:44 smithi059 ceph-mon[97450]: pgmap v19506: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:45.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:45.642932+0000) 2022-09-08T07:39:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:45.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:45.531921+0000) 2022-09-08T07:39:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:46.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:46.643135+0000) 2022-09-08T07:39:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:46.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:46.532096+0000) 2022-09-08T07:39:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:46 smithi055 ceph-mon[97033]: pgmap v19507: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:39:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:46 smithi059 ceph-mon[97450]: pgmap v19507: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:39:47.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:47.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:47.643303+0000) 2022-09-08T07:39:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:47.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:47.532297+0000) 2022-09-08T07:39:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:48.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:48.643521+0000) 2022-09-08T07:39:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:48.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:48.532497+0000) 2022-09-08T07:39:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:48 smithi055 ceph-mon[97033]: pgmap v19508: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:48 smithi059 ceph-mon[97450]: pgmap v19508: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:49.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:49.643723+0000) 2022-09-08T07:39:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:49.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:49.532689+0000) 2022-09-08T07:39:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:50.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:50.643867+0000) 2022-09-08T07:39:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:50.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:50.532908+0000) 2022-09-08T07:39:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:50 smithi055 ceph-mon[97033]: pgmap v19509: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:50 smithi059 ceph-mon[97450]: pgmap v19509: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:51.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:51.644113+0000) 2022-09-08T07:39:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:51.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:51.533096+0000) 2022-09-08T07:39:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:52.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:52.644307+0000) 2022-09-08T07:39:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:52.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:52.533218+0000) 2022-09-08T07:39:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:52 smithi055 ceph-mon[97033]: pgmap v19510: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:39:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:52 smithi059 ceph-mon[97450]: pgmap v19510: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:39:53.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:39:53.432+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:39:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:53.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:53.533372+0000) 2022-09-08T07:39:53.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:53.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:53.644427+0000) 2022-09-08T07:39:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:54.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:54.644689+0000) 2022-09-08T07:39:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:54.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:54.533513+0000) 2022-09-08T07:39:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:54 smithi059 ceph-mon[97450]: pgmap v19511: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:54 smithi055 ceph-mon[97033]: pgmap v19511: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:55.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:55.644926+0000) 2022-09-08T07:39:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:55.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:55.533681+0000) 2022-09-08T07:39:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:56.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:56.645133+0000) 2022-09-08T07:39:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:56.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:56.533835+0000) 2022-09-08T07:39:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:56 smithi059 ceph-mon[97450]: pgmap v19512: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:39:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:56 smithi055 ceph-mon[97033]: pgmap v19512: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:39:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:57.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:57.645325+0000) 2022-09-08T07:39:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:57.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:57.533993+0000) 2022-09-08T07:39:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:58.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:58.645559+0000) 2022-09-08T07:39:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:58.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:58.534150+0000) 2022-09-08T07:39:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:39:58 smithi059 ceph-mon[97450]: pgmap v19513: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:59.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:39:58 smithi055 ceph-mon[97033]: pgmap v19513: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:39:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:39:59.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:59.534371+0000) 2022-09-08T07:39:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:39:59.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:39:59.645753+0000) 2022-09-08T07:40:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:00.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:00.534544+0000) 2022-09-08T07:40:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:00.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:00.645881+0000) 2022-09-08T07:40:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:00 smithi059 ceph-mon[97450]: pgmap v19514: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:40:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T07:40:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T07:40:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T07:40:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:00 smithi055 ceph-mon[97033]: pgmap v19514: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:40:01.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T07:40:01.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T07:40:01.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T07:40:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:01.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:01.534777+0000) 2022-09-08T07:40:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:01.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:01.646118+0000) 2022-09-08T07:40:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:02.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:02.646329+0000) 2022-09-08T07:40:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:02.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:02.534931+0000) 2022-09-08T07:40:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:02 smithi059 ceph-mon[97450]: pgmap v19515: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:40:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:02 smithi055 ceph-mon[97033]: pgmap v19515: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:40:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:03.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:03.646540+0000) 2022-09-08T07:40:03.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:03.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:03.535056+0000) 2022-09-08T07:40:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:04.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:04.646765+0000) 2022-09-08T07:40:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:04.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:04.535199+0000) 2022-09-08T07:40:05.265 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:04 smithi059 ceph-mon[97450]: pgmap v19516: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:40:05.268 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:05 smithi055 ceph-mon[97033]: pgmap v19516: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:40:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:05.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:05.647025+0000) 2022-09-08T07:40:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:05.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:05.535394+0000) 2022-09-08T07:40:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:40:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:40:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:40:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:40:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:40:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:40:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:40:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:40:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:06.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:06.647266+0000) 2022-09-08T07:40:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:06.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:06.535563+0000) 2022-09-08T07:40:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:07 smithi059 ceph-mon[97450]: pgmap v19517: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:40:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:40:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:40:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:07 smithi055 ceph-mon[97033]: pgmap v19517: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:40:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:40:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:40:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:07.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:07.647531+0000) 2022-09-08T07:40:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:07.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:07.535736+0000) 2022-09-08T07:40:08.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:40:08.449+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:40:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:08.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:08.647768+0000) 2022-09-08T07:40:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:08.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:08.535949+0000) 2022-09-08T07:40:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:09 smithi059 ceph-mon[97450]: pgmap v19518: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:40:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:09 smithi055 ceph-mon[97033]: pgmap v19518: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:40:09.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:09.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:09.648011+0000) 2022-09-08T07:40:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:09.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:09.536112+0000) 2022-09-08T07:40:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:10 smithi059 ceph-mon[97450]: osdmap e1297: 12 total, 11 up, 11 in 2022-09-08T07:40:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:10 smithi059 ceph-mon[97450]: pgmap v19520: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:40:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:10 smithi055 ceph-mon[97033]: osdmap e1297: 12 total, 11 up, 11 in 2022-09-08T07:40:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:10 smithi055 ceph-mon[97033]: pgmap v19520: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:40:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:10.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:10.648248+0000) 2022-09-08T07:40:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:10.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:10.536272+0000) 2022-09-08T07:40:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:11.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:11.648483+0000) 2022-09-08T07:40:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:11.535+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:11.536452+0000) 2022-09-08T07:40:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:12.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:12.648729+0000) 2022-09-08T07:40:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:12.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:12.536625+0000) 2022-09-08T07:40:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:12 smithi055 ceph-mon[97033]: pgmap v19521: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:40:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:12 smithi059 ceph-mon[97450]: pgmap v19521: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:40:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:13.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:13.648893+0000) 2022-09-08T07:40:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:13.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:13.536808+0000) 2022-09-08T07:40:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:13 smithi055 ceph-mon[97033]: osdmap e1298: 12 total, 11 up, 11 in 2022-09-08T07:40:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:13 smithi059 ceph-mon[97450]: osdmap e1298: 12 total, 11 up, 11 in 2022-09-08T07:40:14.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:14.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:14.536998+0000) 2022-09-08T07:40:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:14.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:14.649112+0000) 2022-09-08T07:40:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:14 smithi055 ceph-mon[97033]: pgmap v19523: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:40:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:14 smithi059 ceph-mon[97450]: pgmap v19523: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:40:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:15.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:15.649341+0000) 2022-09-08T07:40:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:15.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:15.537183+0000) 2022-09-08T07:40:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:16.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:16.649518+0000) 2022-09-08T07:40:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:16.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:16.537337+0000) 2022-09-08T07:40:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:16 smithi055 ceph-mon[97033]: pgmap v19524: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:40:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:16 smithi059 ceph-mon[97450]: pgmap v19524: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:40:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:17.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:17.649741+0000) 2022-09-08T07:40:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:17.536+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:17.537548+0000) 2022-09-08T07:40:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:18.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:18.649988+0000) 2022-09-08T07:40:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:18.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:18.537779+0000) 2022-09-08T07:40:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:18 smithi055 ceph-mon[97033]: pgmap v19525: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 241 B/s wr, 7 op/s 2022-09-08T07:40:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:18 smithi059 ceph-mon[97450]: pgmap v19525: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 241 B/s wr, 7 op/s 2022-09-08T07:40:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:19.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:19.650185+0000) 2022-09-08T07:40:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:19.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:19.538039+0000) 2022-09-08T07:40:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:20.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:20.650311+0000) 2022-09-08T07:40:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:20.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:20.538205+0000) 2022-09-08T07:40:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:20 smithi055 ceph-mon[97033]: pgmap v19526: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:40:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:20 smithi059 ceph-mon[97450]: pgmap v19526: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:40:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:21.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:21.650535+0000) 2022-09-08T07:40:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:21.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:21.538349+0000) 2022-09-08T07:40:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:22.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:22.650717+0000) 2022-09-08T07:40:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:22.537+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:22.538484+0000) 2022-09-08T07:40:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:22 smithi055 ceph-mon[97033]: pgmap v19527: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:40:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:22 smithi059 ceph-mon[97450]: pgmap v19527: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:40:23.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:40:23.463+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:40:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:23.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:23.650947+0000) 2022-09-08T07:40:23.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:23.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:23.538664+0000) 2022-09-08T07:40:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:24.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:24.651174+0000) 2022-09-08T07:40:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:24.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:24.538903+0000) 2022-09-08T07:40:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:24 smithi055 ceph-mon[97033]: pgmap v19528: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T07:40:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:24 smithi059 ceph-mon[97450]: pgmap v19528: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T07:40:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:25.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:25.651354+0000) 2022-09-08T07:40:25.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:25.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:25.539086+0000) 2022-09-08T07:40:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:26.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:26.651564+0000) 2022-09-08T07:40:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:26.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:26.539288+0000) 2022-09-08T07:40:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:26 smithi055 ceph-mon[97033]: pgmap v19529: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:40:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:26 smithi059 ceph-mon[97450]: pgmap v19529: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:40:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:27.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:27.651824+0000) 2022-09-08T07:40:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:27.538+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:27.539434+0000) 2022-09-08T07:40:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:28.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:28.652052+0000) 2022-09-08T07:40:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:28.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:28.539667+0000) 2022-09-08T07:40:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:28 smithi055 ceph-mon[97033]: pgmap v19530: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:40:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:28 smithi059 ceph-mon[97450]: pgmap v19530: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:40:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:29.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:29.652261+0000) 2022-09-08T07:40:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:29.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:29.539845+0000) 2022-09-08T07:40:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:30.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:30.652452+0000) 2022-09-08T07:40:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:30.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:30.539998+0000) 2022-09-08T07:40:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:30 smithi055 ceph-mon[97033]: pgmap v19531: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:40:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:30 smithi059 ceph-mon[97450]: pgmap v19531: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:40:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:31.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:31.652653+0000) 2022-09-08T07:40:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:31.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:31.540203+0000) 2022-09-08T07:40:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:32.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:32.652823+0000) 2022-09-08T07:40:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:32.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:32.540356+0000) 2022-09-08T07:40:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:32 smithi055 ceph-mon[97033]: pgmap v19532: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:40:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:32 smithi059 ceph-mon[97450]: pgmap v19532: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:40:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:33.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:33.653028+0000) 2022-09-08T07:40:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:33.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:33.540554+0000) 2022-09-08T07:40:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:34.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:34.653168+0000) 2022-09-08T07:40:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:34.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:34.540805+0000) 2022-09-08T07:40:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:34 smithi055 ceph-mon[97033]: pgmap v19533: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:40:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:34 smithi059 ceph-mon[97450]: pgmap v19533: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:40:35.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:35.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:35.540970+0000) 2022-09-08T07:40:35.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:35.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:35.653377+0000) 2022-09-08T07:40:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:36.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:36.653598+0000) 2022-09-08T07:40:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:36.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:36.541119+0000) 2022-09-08T07:40:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:36 smithi055 ceph-mon[97033]: pgmap v19534: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:40:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:36 smithi059 ceph-mon[97450]: pgmap v19534: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:40:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:37.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:37.653748+0000) 2022-09-08T07:40:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:37.539+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:37.541265+0000) 2022-09-08T07:40:38.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:40:38.478+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:40:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:38.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:38.653918+0000) 2022-09-08T07:40:38.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:38.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:38.541401+0000) 2022-09-08T07:40:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:38 smithi055 ceph-mon[97033]: pgmap v19535: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:40:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:38 smithi059 ceph-mon[97450]: pgmap v19535: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:40:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:39.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:39.654143+0000) 2022-09-08T07:40:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:39.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:39.541618+0000) 2022-09-08T07:40:40.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:40.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:40.654381+0000) 2022-09-08T07:40:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:40.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:40.541793+0000) 2022-09-08T07:40:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:40 smithi059 ceph-mon[97450]: pgmap v19536: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:40:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:40 smithi055 ceph-mon[97033]: pgmap v19536: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:40:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:41.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:41.654650+0000) 2022-09-08T07:40:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:41.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:41.541929+0000) 2022-09-08T07:40:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:42.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:42.542100+0000) 2022-09-08T07:40:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:42.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:42.654825+0000) 2022-09-08T07:40:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:42 smithi059 ceph-mon[97450]: pgmap v19537: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:40:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:42 smithi055 ceph-mon[97033]: pgmap v19537: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:40:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:43.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:43.654969+0000) 2022-09-08T07:40:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:43.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:43.542300+0000) 2022-09-08T07:40:44.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:44.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:44.655190+0000) 2022-09-08T07:40:44.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:44.540+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:44.542434+0000) 2022-09-08T07:40:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:44 smithi059 ceph-mon[97450]: pgmap v19538: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:40:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:44 smithi055 ceph-mon[97033]: pgmap v19538: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:40:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:45.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:45.655371+0000) 2022-09-08T07:40:45.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:45.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:45.542594+0000) 2022-09-08T07:40:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:46.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:46.655595+0000) 2022-09-08T07:40:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:46.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:46.542812+0000) 2022-09-08T07:40:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:46 smithi059 ceph-mon[97450]: pgmap v19539: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:40:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:46 smithi055 ceph-mon[97033]: pgmap v19539: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:40:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:47.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:47.655813+0000) 2022-09-08T07:40:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:47.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:47.543060+0000) 2022-09-08T07:40:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:48.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:48.655966+0000) 2022-09-08T07:40:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:48.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:48.543228+0000) 2022-09-08T07:40:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:48 smithi059 ceph-mon[97450]: pgmap v19540: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:40:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:48 smithi055 ceph-mon[97033]: pgmap v19540: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:40:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:49.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:49.656216+0000) 2022-09-08T07:40:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:49.541+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:49.543402+0000) 2022-09-08T07:40:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:50.542+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:50.543542+0000) 2022-09-08T07:40:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:50.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:50.656469+0000) 2022-09-08T07:40:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:50 smithi059 ceph-mon[97450]: pgmap v19541: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:40:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:50 smithi055 ceph-mon[97033]: pgmap v19541: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:40:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:51.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:51.543688+0000) 2022-09-08T07:40:51.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:51.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:51.656740+0000) 2022-09-08T07:40:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:52.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:52.543940+0000) 2022-09-08T07:40:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:52.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:52.656945+0000) 2022-09-08T07:40:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:52 smithi059 ceph-mon[97450]: pgmap v19542: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:40:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:53 smithi055 ceph-mon[97033]: pgmap v19542: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:40:53.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:40:53.490+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:40:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:53.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:53.544142+0000) 2022-09-08T07:40:53.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:53.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:53.657161+0000) 2022-09-08T07:40:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:54.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:54.544340+0000) 2022-09-08T07:40:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:54.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:54.657390+0000) 2022-09-08T07:40:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:55 smithi059 ceph-mon[97450]: pgmap v19543: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:40:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:55 smithi055 ceph-mon[97033]: pgmap v19543: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:40:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:55.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:55.657679+0000) 2022-09-08T07:40:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:55.543+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:55.544545+0000) 2022-09-08T07:40:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:56.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:56.657863+0000) 2022-09-08T07:40:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:56.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:56.544755+0000) 2022-09-08T07:40:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:57 smithi059 ceph-mon[97450]: pgmap v19544: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:40:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:57 smithi055 ceph-mon[97033]: pgmap v19544: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:40:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:57.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:57.544921+0000) 2022-09-08T07:40:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:57.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:57.658102+0000) 2022-09-08T07:40:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:58.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:58.658344+0000) 2022-09-08T07:40:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:58.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:58.545188+0000) 2022-09-08T07:40:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:40:59 smithi059 ceph-mon[97450]: pgmap v19545: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:40:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:40:59 smithi055 ceph-mon[97033]: pgmap v19545: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:40:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:40:59.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:59.658573+0000) 2022-09-08T07:40:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:40:59.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:40:59.545374+0000) 2022-09-08T07:41:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:00.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:00.658702+0000) 2022-09-08T07:41:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:00.544+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:00.545519+0000) 2022-09-08T07:41:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:01 smithi059 ceph-mon[97450]: pgmap v19546: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:41:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:01 smithi055 ceph-mon[97033]: pgmap v19546: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:41:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:01.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:01.658974+0000) 2022-09-08T07:41:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:01.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:01.545714+0000) 2022-09-08T07:41:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:02.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:02.659110+0000) 2022-09-08T07:41:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:02.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:02.545876+0000) 2022-09-08T07:41:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:03 smithi059 ceph-mon[97450]: pgmap v19547: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:41:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:03 smithi055 ceph-mon[97033]: pgmap v19547: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:41:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:03.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:03.659305+0000) 2022-09-08T07:41:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:03.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:03.546100+0000) 2022-09-08T07:41:04.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:04.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:04.546296+0000) 2022-09-08T07:41:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:04.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:04.659521+0000) 2022-09-08T07:41:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:05 smithi059 ceph-mon[97450]: pgmap v19548: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:05 smithi055 ceph-mon[97033]: pgmap v19548: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:05.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:05.659801+0000) 2022-09-08T07:41:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:05.545+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:05.546478+0000) 2022-09-08T07:41:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:41:06.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:41:06.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:41:06.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:41:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:41:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:41:06.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:41:06.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:41:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:06.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:06.659968+0000) 2022-09-08T07:41:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:06.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:06.546626+0000) 2022-09-08T07:41:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:07 smithi059 ceph-mon[97450]: pgmap v19549: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:41:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:41:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:41:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:07 smithi055 ceph-mon[97033]: pgmap v19549: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:41:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:41:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:41:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:07.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:07.660195+0000) 2022-09-08T07:41:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:07.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:07.546787+0000) 2022-09-08T07:41:08.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:41:08.503+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:41:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:08.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:08.546923+0000) 2022-09-08T07:41:08.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:08.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:08.660433+0000) 2022-09-08T07:41:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:08 smithi055 ceph-mon[97033]: pgmap v19550: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:41:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:08 smithi059 ceph-mon[97450]: pgmap v19550: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:41:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:09.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:09.660605+0000) 2022-09-08T07:41:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:09.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:09.547065+0000) 2022-09-08T07:41:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:10.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:10.660730+0000) 2022-09-08T07:41:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:10.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:10.547280+0000) 2022-09-08T07:41:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:10 smithi055 ceph-mon[97033]: pgmap v19551: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:41:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:10 smithi055 ceph-mon[97033]: osdmap e1299: 12 total, 11 up, 11 in 2022-09-08T07:41:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:10 smithi059 ceph-mon[97450]: pgmap v19551: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:41:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:10 smithi059 ceph-mon[97450]: osdmap e1299: 12 total, 11 up, 11 in 2022-09-08T07:41:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:11.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:11.660913+0000) 2022-09-08T07:41:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:11.546+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:11.547508+0000) 2022-09-08T07:41:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:12.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:12.661195+0000) 2022-09-08T07:41:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:12.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:12.547729+0000) 2022-09-08T07:41:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:12 smithi055 ceph-mon[97033]: pgmap v19553: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:41:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:12 smithi059 ceph-mon[97450]: pgmap v19553: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:41:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:13.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:13.661409+0000) 2022-09-08T07:41:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:13.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:13.547889+0000) 2022-09-08T07:41:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:14.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:14.661595+0000) 2022-09-08T07:41:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:14.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:14.548035+0000) 2022-09-08T07:41:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:14 smithi055 ceph-mon[97033]: pgmap v19554: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:41:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:14 smithi059 ceph-mon[97450]: pgmap v19554: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:41:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:15.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:15.661784+0000) 2022-09-08T07:41:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:15.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:15.548183+0000) 2022-09-08T07:41:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:16.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:16.661981+0000) 2022-09-08T07:41:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:16.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:16.548381+0000) 2022-09-08T07:41:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:16 smithi055 ceph-mon[97033]: pgmap v19555: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:41:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:16 smithi059 ceph-mon[97450]: pgmap v19555: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:41:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:17.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:17.662217+0000) 2022-09-08T07:41:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:17.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:17.548552+0000) 2022-09-08T07:41:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:18.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:18.662438+0000) 2022-09-08T07:41:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:18.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:18.548796+0000) 2022-09-08T07:41:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:18 smithi055 ceph-mon[97033]: pgmap v19556: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:41:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:18 smithi055 ceph-mon[97033]: osdmap e1300: 12 total, 11 up, 11 in 2022-09-08T07:41:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:18 smithi059 ceph-mon[97450]: pgmap v19556: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:41:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:18 smithi059 ceph-mon[97450]: osdmap e1300: 12 total, 11 up, 11 in 2022-09-08T07:41:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:19.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:19.662602+0000) 2022-09-08T07:41:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:19.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:19.549065+0000) 2022-09-08T07:41:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:20.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:20.662820+0000) 2022-09-08T07:41:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:20.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:20.549309+0000) 2022-09-08T07:41:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:20 smithi055 ceph-mon[97033]: pgmap v19558: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 107 B/s wr, 3 op/s 2022-09-08T07:41:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:20 smithi059 ceph-mon[97450]: pgmap v19558: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 107 B/s wr, 3 op/s 2022-09-08T07:41:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:21.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:21.663015+0000) 2022-09-08T07:41:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:21.547+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:21.549536+0000) 2022-09-08T07:41:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:22.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:22.663225+0000) 2022-09-08T07:41:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:22.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:22.549710+0000) 2022-09-08T07:41:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:22 smithi055 ceph-mon[97033]: pgmap v19559: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:41:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:22 smithi059 ceph-mon[97450]: pgmap v19559: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:41:23.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:41:23.511+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:41:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:23.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:23.663371+0000) 2022-09-08T07:41:23.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:23.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:23.549839+0000) 2022-09-08T07:41:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:24.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:24.663577+0000) 2022-09-08T07:41:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:24.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:24.550048+0000) 2022-09-08T07:41:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:24 smithi055 ceph-mon[97033]: pgmap v19560: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:41:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:24 smithi059 ceph-mon[97450]: pgmap v19560: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:41:25.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:25.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:25.663819+0000) 2022-09-08T07:41:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:25.548+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:25.550273+0000) 2022-09-08T07:41:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:26.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:26.664004+0000) 2022-09-08T07:41:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:26.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:26.550497+0000) 2022-09-08T07:41:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:26 smithi055 ceph-mon[97033]: pgmap v19561: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:41:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:26 smithi059 ceph-mon[97450]: pgmap v19561: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:41:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:27.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:27.664153+0000) 2022-09-08T07:41:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:27.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:27.550667+0000) 2022-09-08T07:41:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:28.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:28.550851+0000) 2022-09-08T07:41:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:28.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:28.664311+0000) 2022-09-08T07:41:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:28 smithi055 ceph-mon[97033]: pgmap v19562: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:41:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:28 smithi059 ceph-mon[97450]: pgmap v19562: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:41:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:29.549+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:29.551010+0000) 2022-09-08T07:41:29.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:29.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:29.664539+0000) 2022-09-08T07:41:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:30.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:30.551193+0000) 2022-09-08T07:41:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:30.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:30.664751+0000) 2022-09-08T07:41:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:30 smithi055 ceph-mon[97033]: pgmap v19563: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:41:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:30 smithi059 ceph-mon[97450]: pgmap v19563: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:41:31.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:31.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:31.551422+0000) 2022-09-08T07:41:31.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:31.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:31.664971+0000) 2022-09-08T07:41:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:32.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:32.665209+0000) 2022-09-08T07:41:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:32.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:32.551610+0000) 2022-09-08T07:41:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:32 smithi055 ceph-mon[97033]: pgmap v19564: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:41:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:32 smithi059 ceph-mon[97450]: pgmap v19564: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:41:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:33.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:33.665444+0000) 2022-09-08T07:41:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:33.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:33.551815+0000) 2022-09-08T07:41:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:34.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:34.665676+0000) 2022-09-08T07:41:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:34.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:34.552027+0000) 2022-09-08T07:41:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:34 smithi055 ceph-mon[97033]: pgmap v19565: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:34 smithi059 ceph-mon[97450]: pgmap v19565: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:35.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:35.665906+0000) 2022-09-08T07:41:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:35.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:35.552237+0000) 2022-09-08T07:41:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:36.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:36.666105+0000) 2022-09-08T07:41:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:36.550+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:36.552455+0000) 2022-09-08T07:41:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:36 smithi055 ceph-mon[97033]: pgmap v19566: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:41:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:36 smithi059 ceph-mon[97450]: pgmap v19566: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:41:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:37.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:37.666345+0000) 2022-09-08T07:41:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:37.551+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:37.552660+0000) 2022-09-08T07:41:38.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:41:38.520+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:41:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:38.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:38.666610+0000) 2022-09-08T07:41:38.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:38.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:38.552856+0000) 2022-09-08T07:41:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:38 smithi055 ceph-mon[97033]: pgmap v19567: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:38 smithi059 ceph-mon[97450]: pgmap v19567: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:39.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:39.666837+0000) 2022-09-08T07:41:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:39.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:39.553062+0000) 2022-09-08T07:41:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:40.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:40.667047+0000) 2022-09-08T07:41:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:40.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:40.553216+0000) 2022-09-08T07:41:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:40 smithi055 ceph-mon[97033]: pgmap v19568: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:40 smithi059 ceph-mon[97450]: pgmap v19568: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:41.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:41.667306+0000) 2022-09-08T07:41:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:41.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:41.553420+0000) 2022-09-08T07:41:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:42.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:42.667565+0000) 2022-09-08T07:41:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:42.552+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:42.553579+0000) 2022-09-08T07:41:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:42 smithi059 ceph-mon[97450]: pgmap v19569: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:41:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:42 smithi055 ceph-mon[97033]: pgmap v19569: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:41:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:43.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:43.667822+0000) 2022-09-08T07:41:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:43.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:43.553777+0000) 2022-09-08T07:41:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:44.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:44.668015+0000) 2022-09-08T07:41:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:44.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:44.554020+0000) 2022-09-08T07:41:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:44 smithi059 ceph-mon[97450]: pgmap v19570: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:44 smithi055 ceph-mon[97033]: pgmap v19570: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:45.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:45.668268+0000) 2022-09-08T07:41:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:45.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:45.554268+0000) 2022-09-08T07:41:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:46.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:46.668476+0000) 2022-09-08T07:41:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:46.553+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:46.554453+0000) 2022-09-08T07:41:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:46 smithi059 ceph-mon[97450]: pgmap v19571: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:41:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:46 smithi055 ceph-mon[97033]: pgmap v19571: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:41:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:47.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:47.668719+0000) 2022-09-08T07:41:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:47.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:47.554713+0000) 2022-09-08T07:41:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:48.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:48.668923+0000) 2022-09-08T07:41:48.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:48.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:48.554951+0000) 2022-09-08T07:41:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:48 smithi059 ceph-mon[97450]: pgmap v19572: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:48 smithi055 ceph-mon[97033]: pgmap v19572: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:49.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:49.669119+0000) 2022-09-08T07:41:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:49.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:49.555159+0000) 2022-09-08T07:41:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:50.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:50.669283+0000) 2022-09-08T07:41:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:50.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:50.555384+0000) 2022-09-08T07:41:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:50 smithi059 ceph-mon[97450]: pgmap v19573: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:50 smithi055 ceph-mon[97033]: pgmap v19573: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:51.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:51.669499+0000) 2022-09-08T07:41:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:51.554+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:51.555608+0000) 2022-09-08T07:41:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:52.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:52.669748+0000) 2022-09-08T07:41:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:52.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:52.555877+0000) 2022-09-08T07:41:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:52 smithi059 ceph-mon[97450]: pgmap v19574: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:41:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:52 smithi055 ceph-mon[97033]: pgmap v19574: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:41:53.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:41:53.534+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:41:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:53.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:53.670000+0000) 2022-09-08T07:41:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:53.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:53.556116+0000) 2022-09-08T07:41:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:54.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:54.670192+0000) 2022-09-08T07:41:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:54.555+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:54.556355+0000) 2022-09-08T07:41:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:54 smithi059 ceph-mon[97450]: pgmap v19575: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:55 smithi055 ceph-mon[97033]: pgmap v19575: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:55.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:55.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:55.670352+0000) 2022-09-08T07:41:55.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:55.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:55.556553+0000) 2022-09-08T07:41:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:56.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:56.670561+0000) 2022-09-08T07:41:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:56.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:56.556701+0000) 2022-09-08T07:41:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:57 smithi059 ceph-mon[97450]: pgmap v19576: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:41:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:57 smithi055 ceph-mon[97033]: pgmap v19576: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:41:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:57.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:57.670736+0000) 2022-09-08T07:41:57.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:57.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:57.556847+0000) 2022-09-08T07:41:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:58.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:58.670965+0000) 2022-09-08T07:41:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:58.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:58.556956+0000) 2022-09-08T07:41:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:41:59 smithi059 ceph-mon[97450]: pgmap v19577: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:41:59 smithi055 ceph-mon[97033]: pgmap v19577: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:41:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:41:59.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:59.671207+0000) 2022-09-08T07:41:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:41:59.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:41:59.557192+0000) 2022-09-08T07:42:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:00.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:00.671348+0000) 2022-09-08T07:42:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:00.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:00.557413+0000) 2022-09-08T07:42:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:01 smithi059 ceph-mon[97450]: pgmap v19578: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:01 smithi055 ceph-mon[97033]: pgmap v19578: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:01.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:01.671546+0000) 2022-09-08T07:42:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:01.556+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:01.557563+0000) 2022-09-08T07:42:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:02.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:02.671751+0000) 2022-09-08T07:42:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:02.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:02.557716+0000) 2022-09-08T07:42:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:03 smithi059 ceph-mon[97450]: pgmap v19579: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:42:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:03 smithi055 ceph-mon[97033]: pgmap v19579: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:42:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:03.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:03.557882+0000) 2022-09-08T07:42:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:03.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:03.671983+0000) 2022-09-08T07:42:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:04.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:04.672285+0000) 2022-09-08T07:42:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:04.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:04.558051+0000) 2022-09-08T07:42:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:05 smithi059 ceph-mon[97450]: pgmap v19580: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:05 smithi055 ceph-mon[97033]: pgmap v19580: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:05.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:05.672548+0000) 2022-09-08T07:42:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:05.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:05.558267+0000) 2022-09-08T07:42:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:42:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:42:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:06.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:06.672779+0000) 2022-09-08T07:42:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:06.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:06.558450+0000) 2022-09-08T07:42:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:07 smithi059 ceph-mon[97450]: pgmap v19581: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:42:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:42:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:42:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:42:07.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:42:07.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:42:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:07 smithi055 ceph-mon[97033]: pgmap v19581: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:42:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:42:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:42:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:42:07.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:42:07.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:42:07.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:07.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:07.672999+0000) 2022-09-08T07:42:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:07.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:07.558724+0000) 2022-09-08T07:42:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:08.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:08.673200+0000) 2022-09-08T07:42:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:08.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:08.558905+0000) 2022-09-08T07:42:08.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:42:08.537+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:42:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:09 smithi055 ceph-mon[97033]: pgmap v19582: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:42:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:09 smithi059 ceph-mon[97450]: pgmap v19582: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:42:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:09.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:09.673455+0000) 2022-09-08T07:42:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:09.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:09.559071+0000) 2022-09-08T07:42:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:10 smithi055 ceph-mon[97033]: osdmap e1301: 12 total, 11 up, 11 in 2022-09-08T07:42:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:10 smithi055 ceph-mon[97033]: pgmap v19584: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:42:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:10 smithi059 ceph-mon[97450]: osdmap e1301: 12 total, 11 up, 11 in 2022-09-08T07:42:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:10 smithi059 ceph-mon[97450]: pgmap v19584: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:42:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:10.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:10.673771+0000) 2022-09-08T07:42:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:10.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:10.559209+0000) 2022-09-08T07:42:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:11.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:11.559368+0000) 2022-09-08T07:42:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:11.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:11.674033+0000) 2022-09-08T07:42:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:12.557+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:12.559552+0000) 2022-09-08T07:42:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:12.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:12.674226+0000) 2022-09-08T07:42:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:12 smithi055 ceph-mon[97033]: pgmap v19585: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:42:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:12 smithi059 ceph-mon[97450]: pgmap v19585: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:42:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:13.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:13.674438+0000) 2022-09-08T07:42:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:13.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:13.559799+0000) 2022-09-08T07:42:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:13 smithi055 ceph-mon[97033]: osdmap e1302: 12 total, 11 up, 11 in 2022-09-08T07:42:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:13 smithi059 ceph-mon[97450]: osdmap e1302: 12 total, 11 up, 11 in 2022-09-08T07:42:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:14.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:14.674720+0000) 2022-09-08T07:42:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:14.558+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:14.560037+0000) 2022-09-08T07:42:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:14 smithi055 ceph-mon[97033]: pgmap v19587: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:42:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:14 smithi059 ceph-mon[97450]: pgmap v19587: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:42:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:15.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:15.674994+0000) 2022-09-08T07:42:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:15.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:15.560259+0000) 2022-09-08T07:42:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:16.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:16.675226+0000) 2022-09-08T07:42:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:16.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:16.560498+0000) 2022-09-08T07:42:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:16 smithi055 ceph-mon[97033]: pgmap v19588: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:42:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:16 smithi059 ceph-mon[97450]: pgmap v19588: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:42:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:17.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:17.675449+0000) 2022-09-08T07:42:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:17.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:17.560702+0000) 2022-09-08T07:42:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:18.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:18.675686+0000) 2022-09-08T07:42:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:18.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:18.560858+0000) 2022-09-08T07:42:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:18 smithi055 ceph-mon[97033]: pgmap v19589: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 241 B/s wr, 7 op/s 2022-09-08T07:42:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:18 smithi059 ceph-mon[97450]: pgmap v19589: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 241 B/s wr, 7 op/s 2022-09-08T07:42:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:19.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:19.675839+0000) 2022-09-08T07:42:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:19.559+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:19.560998+0000) 2022-09-08T07:42:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:20.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:20.676045+0000) 2022-09-08T07:42:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:20.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:20.561137+0000) 2022-09-08T07:42:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:20 smithi055 ceph-mon[97033]: pgmap v19590: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:42:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:20 smithi059 ceph-mon[97450]: pgmap v19590: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:42:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:21.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:21.676275+0000) 2022-09-08T07:42:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:21.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:21.561375+0000) 2022-09-08T07:42:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:22.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:22.676474+0000) 2022-09-08T07:42:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:22.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:22.561583+0000) 2022-09-08T07:42:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:22 smithi059 ceph-mon[97450]: pgmap v19591: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:42:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:22 smithi055 ceph-mon[97033]: pgmap v19591: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:42:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:23.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:23.676722+0000) 2022-09-08T07:42:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:23.560+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:23.561784+0000) 2022-09-08T07:42:23.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:42:23.549+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:42:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:24.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:24.676860+0000) 2022-09-08T07:42:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:24.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:24.561967+0000) 2022-09-08T07:42:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:24 smithi059 ceph-mon[97450]: pgmap v19592: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T07:42:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:24 smithi055 ceph-mon[97033]: pgmap v19592: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T07:42:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:25.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:25.677107+0000) 2022-09-08T07:42:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:25.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:25.562199+0000) 2022-09-08T07:42:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:26.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:26.677276+0000) 2022-09-08T07:42:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:26.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:26.562357+0000) 2022-09-08T07:42:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:26 smithi059 ceph-mon[97450]: pgmap v19593: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:42:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:26 smithi055 ceph-mon[97033]: pgmap v19593: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:42:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:27.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:27.677510+0000) 2022-09-08T07:42:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:27.561+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:27.562512+0000) 2022-09-08T07:42:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:28.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:28.677717+0000) 2022-09-08T07:42:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:28.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:28.562781+0000) 2022-09-08T07:42:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:28 smithi059 ceph-mon[97450]: pgmap v19594: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:28 smithi055 ceph-mon[97033]: pgmap v19594: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:29.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:29.677964+0000) 2022-09-08T07:42:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:29.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:29.562945+0000) 2022-09-08T07:42:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:30.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:30.678175+0000) 2022-09-08T07:42:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:30.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:30.563084+0000) 2022-09-08T07:42:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:30 smithi059 ceph-mon[97450]: pgmap v19595: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:30 smithi055 ceph-mon[97033]: pgmap v19595: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:31.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:31.678331+0000) 2022-09-08T07:42:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:31.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:31.563266+0000) 2022-09-08T07:42:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:32.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:32.678532+0000) 2022-09-08T07:42:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:32.562+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:32.563503+0000) 2022-09-08T07:42:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:32 smithi059 ceph-mon[97450]: pgmap v19596: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:42:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:32 smithi055 ceph-mon[97033]: pgmap v19596: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:42:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:33.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:33.678758+0000) 2022-09-08T07:42:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:33.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:33.563701+0000) 2022-09-08T07:42:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:34.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:34.679005+0000) 2022-09-08T07:42:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:34.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:34.563945+0000) 2022-09-08T07:42:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:34 smithi059 ceph-mon[97450]: pgmap v19597: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:34 smithi055 ceph-mon[97033]: pgmap v19597: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:35.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:35.679233+0000) 2022-09-08T07:42:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:35.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:35.564121+0000) 2022-09-08T07:42:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:36.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:36.679424+0000) 2022-09-08T07:42:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:36.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:36.564284+0000) 2022-09-08T07:42:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:36 smithi059 ceph-mon[97450]: pgmap v19598: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:42:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:37 smithi055 ceph-mon[97033]: pgmap v19598: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:42:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:37.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:37.679592+0000) 2022-09-08T07:42:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:37.563+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:37.564502+0000) 2022-09-08T07:42:38.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:42:38.564+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:42:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:38.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:38.564662+0000) 2022-09-08T07:42:38.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:38.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:38.679813+0000) 2022-09-08T07:42:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:39 smithi059 ceph-mon[97450]: pgmap v19599: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:39 smithi055 ceph-mon[97033]: pgmap v19599: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:39.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:39.680045+0000) 2022-09-08T07:42:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:39.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:39.564817+0000) 2022-09-08T07:42:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:40.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:40.680266+0000) 2022-09-08T07:42:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:40.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:40.565033+0000) 2022-09-08T07:42:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:41 smithi059 ceph-mon[97450]: pgmap v19600: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:41 smithi055 ceph-mon[97033]: pgmap v19600: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:41.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:41.680501+0000) 2022-09-08T07:42:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:41.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:41.565217+0000) 2022-09-08T07:42:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:42.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:42.680784+0000) 2022-09-08T07:42:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:42.564+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:42.565427+0000) 2022-09-08T07:42:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:43 smithi059 ceph-mon[97450]: pgmap v19601: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:42:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:43 smithi055 ceph-mon[97033]: pgmap v19601: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:42:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:43.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:43.681024+0000) 2022-09-08T07:42:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:43.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:43.565625+0000) 2022-09-08T07:42:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:44.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:44.681303+0000) 2022-09-08T07:42:44.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:44.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:44.565811+0000) 2022-09-08T07:42:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:45 smithi059 ceph-mon[97450]: pgmap v19602: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:45 smithi055 ceph-mon[97033]: pgmap v19602: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:45.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:45.681550+0000) 2022-09-08T07:42:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:45.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:45.565959+0000) 2022-09-08T07:42:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:46.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:46.681740+0000) 2022-09-08T07:42:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:46.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:46.566166+0000) 2022-09-08T07:42:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:47 smithi059 ceph-mon[97450]: pgmap v19603: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:42:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:47 smithi055 ceph-mon[97033]: pgmap v19603: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:42:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:47.565+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:47.566321+0000) 2022-09-08T07:42:47.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:47.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:47.681965+0000) 2022-09-08T07:42:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:48.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:48.682184+0000) 2022-09-08T07:42:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:48.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:48.566576+0000) 2022-09-08T07:42:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:49 smithi059 ceph-mon[97450]: pgmap v19604: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:49 smithi055 ceph-mon[97033]: pgmap v19604: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:49.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:49.682434+0000) 2022-09-08T07:42:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:49.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:49.566774+0000) 2022-09-08T07:42:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:50.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:50.682661+0000) 2022-09-08T07:42:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:50.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:50.566986+0000) 2022-09-08T07:42:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:51 smithi059 ceph-mon[97450]: pgmap v19605: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:51 smithi055 ceph-mon[97033]: pgmap v19605: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:51.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:51.682838+0000) 2022-09-08T07:42:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:51.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:51.567211+0000) 2022-09-08T07:42:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:52.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:52.683032+0000) 2022-09-08T07:42:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:52.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:52.567361+0000) 2022-09-08T07:42:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:53 smithi059 ceph-mon[97450]: pgmap v19606: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:42:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:53 smithi055 ceph-mon[97033]: pgmap v19606: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:42:53.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:42:53.576+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:42:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:53.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:53.683260+0000) 2022-09-08T07:42:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:53.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:53.567559+0000) 2022-09-08T07:42:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:54.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:54.683398+0000) 2022-09-08T07:42:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:54.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:54.567759+0000) 2022-09-08T07:42:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:55 smithi059 ceph-mon[97450]: pgmap v19607: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:55 smithi055 ceph-mon[97033]: pgmap v19607: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:55.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:55.683667+0000) 2022-09-08T07:42:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:55.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:55.567930+0000) 2022-09-08T07:42:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:56.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:56.683928+0000) 2022-09-08T07:42:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:56.566+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:56.568091+0000) 2022-09-08T07:42:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:57 smithi055 ceph-mon[97033]: pgmap v19608: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:42:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:57 smithi059 ceph-mon[97450]: pgmap v19608: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:42:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:57.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:57.684121+0000) 2022-09-08T07:42:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:57.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:57.568265+0000) 2022-09-08T07:42:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:58.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:58.684311+0000) 2022-09-08T07:42:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:58.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:58.568409+0000) 2022-09-08T07:42:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:42:59 smithi055 ceph-mon[97033]: pgmap v19609: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:42:59 smithi059 ceph-mon[97450]: pgmap v19609: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:42:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:42:59.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:59.684488+0000) 2022-09-08T07:42:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:42:59.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:42:59.568571+0000) 2022-09-08T07:43:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:00.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:00.684666+0000) 2022-09-08T07:43:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:00.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:00.568775+0000) 2022-09-08T07:43:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:01 smithi055 ceph-mon[97033]: pgmap v19610: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:01 smithi059 ceph-mon[97450]: pgmap v19610: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:01.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:01.684882+0000) 2022-09-08T07:43:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:01.567+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:01.568997+0000) 2022-09-08T07:43:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:02 smithi055 ceph-mon[97033]: pgmap v19611: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:43:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:02 smithi059 ceph-mon[97450]: pgmap v19611: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:43:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:02.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:02.685139+0000) 2022-09-08T07:43:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:02.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:02.569200+0000) 2022-09-08T07:43:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:03.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:03.685365+0000) 2022-09-08T07:43:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:03.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:03.569360+0000) 2022-09-08T07:43:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:04.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:04.685593+0000) 2022-09-08T07:43:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:04.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:04.569530+0000) 2022-09-08T07:43:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:04 smithi055 ceph-mon[97033]: pgmap v19612: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:04 smithi059 ceph-mon[97450]: pgmap v19612: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:05.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:05.685855+0000) 2022-09-08T07:43:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:05.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:05.569714+0000) 2022-09-08T07:43:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:06.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:06.686052+0000) 2022-09-08T07:43:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:06.568+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:06.569913+0000) 2022-09-08T07:43:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:06 smithi055 ceph-mon[97033]: pgmap v19613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:43:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:43:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:43:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:06 smithi059 ceph-mon[97450]: pgmap v19613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:43:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:43:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:43:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:07.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:07.686210+0000) 2022-09-08T07:43:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:07.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:07.570083+0000) 2022-09-08T07:43:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:43:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:43:08.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:43:08.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:43:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:43:08.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:43:08.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:43:08.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:43:08.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:43:08.583+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:43:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:08.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:08.686421+0000) 2022-09-08T07:43:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:08.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:08.570262+0000) 2022-09-08T07:43:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:08 smithi059 ceph-mon[97450]: pgmap v19614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:43:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:08 smithi055 ceph-mon[97033]: pgmap v19614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:43:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:09.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:09.686684+0000) 2022-09-08T07:43:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:09.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:09.570434+0000) 2022-09-08T07:43:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:10.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:10.686878+0000) 2022-09-08T07:43:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:10.569+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:10.570736+0000) 2022-09-08T07:43:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:10 smithi059 ceph-mon[97450]: pgmap v19615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:43:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:10 smithi059 ceph-mon[97450]: osdmap e1303: 12 total, 11 up, 11 in 2022-09-08T07:43:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:10 smithi055 ceph-mon[97033]: pgmap v19615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:43:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:10 smithi055 ceph-mon[97033]: osdmap e1303: 12 total, 11 up, 11 in 2022-09-08T07:43:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:11.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:11.687089+0000) 2022-09-08T07:43:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:11.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:11.570953+0000) 2022-09-08T07:43:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:12.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:12.687336+0000) 2022-09-08T07:43:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:12.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:12.571150+0000) 2022-09-08T07:43:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:12 smithi059 ceph-mon[97450]: pgmap v19617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:43:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:12 smithi055 ceph-mon[97033]: pgmap v19617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:43:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:13.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:13.687548+0000) 2022-09-08T07:43:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:13.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:13.571299+0000) 2022-09-08T07:43:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:14.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:14.687757+0000) 2022-09-08T07:43:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:14.570+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:14.571508+0000) 2022-09-08T07:43:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:14 smithi059 ceph-mon[97450]: pgmap v19618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:43:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:14 smithi055 ceph-mon[97033]: pgmap v19618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:43:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:15.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:15.687939+0000) 2022-09-08T07:43:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:15.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:15.571756+0000) 2022-09-08T07:43:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:16.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:16.688069+0000) 2022-09-08T07:43:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:16.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:16.572017+0000) 2022-09-08T07:43:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:16 smithi059 ceph-mon[97450]: pgmap v19619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:43:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:17 smithi055 ceph-mon[97033]: pgmap v19619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:43:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:17.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:17.688294+0000) 2022-09-08T07:43:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:17.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:17.572174+0000) 2022-09-08T07:43:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:18.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:18.688536+0000) 2022-09-08T07:43:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:18.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:18.572349+0000) 2022-09-08T07:43:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:18 smithi055 ceph-mon[97033]: pgmap v19620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:43:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:18 smithi055 ceph-mon[97033]: osdmap e1304: 12 total, 11 up, 11 in 2022-09-08T07:43:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:18 smithi059 ceph-mon[97450]: pgmap v19620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:43:19.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:18 smithi059 ceph-mon[97450]: osdmap e1304: 12 total, 11 up, 11 in 2022-09-08T07:43:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:19.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:19.688750+0000) 2022-09-08T07:43:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:19.571+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:19.572508+0000) 2022-09-08T07:43:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:20.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:20.688888+0000) 2022-09-08T07:43:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:20.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:20.572691+0000) 2022-09-08T07:43:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:20 smithi055 ceph-mon[97033]: pgmap v19622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 107 B/s wr, 3 op/s 2022-09-08T07:43:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:20 smithi059 ceph-mon[97450]: pgmap v19622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.7 KiB/s rd, 107 B/s wr, 3 op/s 2022-09-08T07:43:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:21.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:21.689033+0000) 2022-09-08T07:43:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:21.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:21.572850+0000) 2022-09-08T07:43:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:22.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:22.689230+0000) 2022-09-08T07:43:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:22.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:22.573015+0000) 2022-09-08T07:43:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:22 smithi059 ceph-mon[97450]: pgmap v19623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:43:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:22 smithi055 ceph-mon[97033]: pgmap v19623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:43:23.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:23.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:23.573255+0000) 2022-09-08T07:43:23.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:43:23.597+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:43:23.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:23.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:23.689432+0000) 2022-09-08T07:43:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:24.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:24.573421+0000) 2022-09-08T07:43:24.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:24.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:24.689667+0000) 2022-09-08T07:43:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:24 smithi059 ceph-mon[97450]: pgmap v19624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:43:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:24 smithi055 ceph-mon[97033]: pgmap v19624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:43:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:25.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:25.689851+0000) 2022-09-08T07:43:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:25.572+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:25.573572+0000) 2022-09-08T07:43:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:26.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:26.573806+0000) 2022-09-08T07:43:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:26.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:26.690052+0000) 2022-09-08T07:43:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:26 smithi059 ceph-mon[97450]: pgmap v19625: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T07:43:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:26 smithi055 ceph-mon[97033]: pgmap v19625: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T07:43:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:27.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:27.690266+0000) 2022-09-08T07:43:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:27.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:27.574048+0000) 2022-09-08T07:43:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:28.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:28.690430+0000) 2022-09-08T07:43:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:28.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:28.574220+0000) 2022-09-08T07:43:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:28 smithi059 ceph-mon[97450]: pgmap v19626: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T07:43:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:28 smithi055 ceph-mon[97033]: pgmap v19626: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T07:43:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:29.573+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:29.574472+0000) 2022-09-08T07:43:29.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:29.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:29.690581+0000) 2022-09-08T07:43:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:30.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:30.690754+0000) 2022-09-08T07:43:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:30.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:30.574696+0000) 2022-09-08T07:43:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:30 smithi059 ceph-mon[97450]: pgmap v19627: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:43:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:30 smithi055 ceph-mon[97033]: pgmap v19627: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:43:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:31.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:31.691001+0000) 2022-09-08T07:43:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:31.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:31.574825+0000) 2022-09-08T07:43:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:32.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:32.691233+0000) 2022-09-08T07:43:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:32.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:32.574976+0000) 2022-09-08T07:43:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:32 smithi059 ceph-mon[97450]: pgmap v19628: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T07:43:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:32 smithi055 ceph-mon[97033]: pgmap v19628: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T07:43:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:33.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:33.691427+0000) 2022-09-08T07:43:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:33.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:33.575189+0000) 2022-09-08T07:43:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:34.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:34.691719+0000) 2022-09-08T07:43:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:34.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:34.575374+0000) 2022-09-08T07:43:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:35 smithi059 ceph-mon[97450]: pgmap v19629: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:35 smithi055 ceph-mon[97033]: pgmap v19629: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:35.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:35.692048+0000) 2022-09-08T07:43:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:35.574+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:35.575516+0000) 2022-09-08T07:43:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:36.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:36.692294+0000) 2022-09-08T07:43:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:36.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:36.575710+0000) 2022-09-08T07:43:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:37 smithi059 ceph-mon[97450]: pgmap v19630: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:43:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:37 smithi055 ceph-mon[97033]: pgmap v19630: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:43:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:37.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:37.692520+0000) 2022-09-08T07:43:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:37.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:37.575840+0000) 2022-09-08T07:43:38.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:43:38.607+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:43:38.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:38.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:38.692743+0000) 2022-09-08T07:43:38.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:38.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:38.575998+0000) 2022-09-08T07:43:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:39 smithi059 ceph-mon[97450]: pgmap v19631: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:39 smithi055 ceph-mon[97033]: pgmap v19631: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:39.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:39.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:39.576152+0000) 2022-09-08T07:43:39.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:39.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:39.692936+0000) 2022-09-08T07:43:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:40.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:40.693078+0000) 2022-09-08T07:43:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:40.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:40.576341+0000) 2022-09-08T07:43:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:41 smithi059 ceph-mon[97450]: pgmap v19632: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:41 smithi055 ceph-mon[97033]: pgmap v19632: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:41.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:41.693301+0000) 2022-09-08T07:43:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:41.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:41.576492+0000) 2022-09-08T07:43:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:42.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:42.693536+0000) 2022-09-08T07:43:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:42.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:42.576663+0000) 2022-09-08T07:43:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:43 smithi059 ceph-mon[97450]: pgmap v19633: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:43:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:43 smithi055 ceph-mon[97033]: pgmap v19633: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:43:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:43.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:43.693747+0000) 2022-09-08T07:43:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:43.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:43.576833+0000) 2022-09-08T07:43:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:44.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:44.694020+0000) 2022-09-08T07:43:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:44.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:44.577009+0000) 2022-09-08T07:43:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:45 smithi059 ceph-mon[97450]: pgmap v19634: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:45 smithi055 ceph-mon[97033]: pgmap v19634: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:45.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:45.694285+0000) 2022-09-08T07:43:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:45.575+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:45.577147+0000) 2022-09-08T07:43:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:46.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:46.694535+0000) 2022-09-08T07:43:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:46.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:46.577336+0000) 2022-09-08T07:43:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:47 smithi059 ceph-mon[97450]: pgmap v19635: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:43:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:47 smithi055 ceph-mon[97033]: pgmap v19635: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:43:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:47.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:47.694759+0000) 2022-09-08T07:43:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:47.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:47.577553+0000) 2022-09-08T07:43:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:48.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:48.695001+0000) 2022-09-08T07:43:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:48.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:48.577725+0000) 2022-09-08T07:43:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:49 smithi059 ceph-mon[97450]: pgmap v19636: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:49 smithi055 ceph-mon[97033]: pgmap v19636: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:49.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:49.695188+0000) 2022-09-08T07:43:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:49.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:49.577998+0000) 2022-09-08T07:43:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:50.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:50.695414+0000) 2022-09-08T07:43:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:50.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:50.578218+0000) 2022-09-08T07:43:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:51 smithi059 ceph-mon[97450]: pgmap v19637: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:51 smithi055 ceph-mon[97033]: pgmap v19637: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:51.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:51.695616+0000) 2022-09-08T07:43:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:51.576+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:51.578389+0000) 2022-09-08T07:43:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:52.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:52.578668+0000) 2022-09-08T07:43:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:52.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:52.695897+0000) 2022-09-08T07:43:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:53 smithi055 ceph-mon[97033]: pgmap v19638: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:43:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:53 smithi059 ceph-mon[97450]: pgmap v19638: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:43:53.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:43:53.622+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:43:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:53.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:53.696064+0000) 2022-09-08T07:43:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:53.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:53.578856+0000) 2022-09-08T07:43:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:54.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:54.696285+0000) 2022-09-08T07:43:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:54.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:54.579068+0000) 2022-09-08T07:43:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:55 smithi055 ceph-mon[97033]: pgmap v19639: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:55 smithi059 ceph-mon[97450]: pgmap v19639: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:55.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:55.696497+0000) 2022-09-08T07:43:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:55.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:55.579263+0000) 2022-09-08T07:43:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:56.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:56.696741+0000) 2022-09-08T07:43:56.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:56.577+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:56.579495+0000) 2022-09-08T07:43:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:57 smithi055 ceph-mon[97033]: pgmap v19640: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:43:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:57 smithi059 ceph-mon[97450]: pgmap v19640: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:43:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:57.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:57.579625+0000) 2022-09-08T07:43:57.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:57.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:57.696918+0000) 2022-09-08T07:43:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:58.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:58.697107+0000) 2022-09-08T07:43:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:58.578+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:58.579764+0000) 2022-09-08T07:43:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:43:59 smithi055 ceph-mon[97033]: pgmap v19641: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:43:59 smithi059 ceph-mon[97450]: pgmap v19641: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:43:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:43:59.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:59.697349+0000) 2022-09-08T07:43:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:43:59.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:43:59.579891+0000) 2022-09-08T07:44:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:00.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:00.697535+0000) 2022-09-08T07:44:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:00.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:00.580095+0000) 2022-09-08T07:44:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:01 smithi055 ceph-mon[97033]: pgmap v19642: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:01 smithi059 ceph-mon[97450]: pgmap v19642: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:01.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:01.697684+0000) 2022-09-08T07:44:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:01.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:01.580268+0000) 2022-09-08T07:44:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:02.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:02.697891+0000) 2022-09-08T07:44:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:02.579+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:02.580498+0000) 2022-09-08T07:44:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:03 smithi055 ceph-mon[97033]: pgmap v19643: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:44:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:03 smithi059 ceph-mon[97450]: pgmap v19643: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:44:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:03.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:03.698029+0000) 2022-09-08T07:44:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:03.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:03.580628+0000) 2022-09-08T07:44:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:04 smithi055 ceph-mon[97033]: pgmap v19644: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:04 smithi059 ceph-mon[97450]: pgmap v19644: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:04.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:04.698162+0000) 2022-09-08T07:44:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:04.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:04.580776+0000) 2022-09-08T07:44:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:05.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:05.698362+0000) 2022-09-08T07:44:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:05.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:05.580922+0000) 2022-09-08T07:44:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:06.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:06.698552+0000) 2022-09-08T07:44:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:06.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:06.581144+0000) 2022-09-08T07:44:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:06 smithi055 ceph-mon[97033]: pgmap v19645: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:44:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:44:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:06 smithi059 ceph-mon[97450]: pgmap v19645: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:44:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:44:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:07.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:07.698694+0000) 2022-09-08T07:44:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:07.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:07.581314+0000) 2022-09-08T07:44:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:44:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:44:08.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:44:08.638+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:44:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:08.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:08.698867+0000) 2022-09-08T07:44:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:08.580+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:08.581504+0000) 2022-09-08T07:44:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:08 smithi055 ceph-mon[97033]: pgmap v19646: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:44:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:44:08.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:44:08.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:44:08.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:44:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:08 smithi059 ceph-mon[97450]: pgmap v19646: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:44:09.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:44:09.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:44:09.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:44:09.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:44:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:09.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:09.581714+0000) 2022-09-08T07:44:09.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:09.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:09.699028+0000) 2022-09-08T07:44:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:10.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:10.581845+0000) 2022-09-08T07:44:10.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:10.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:10.699236+0000) 2022-09-08T07:44:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:10 smithi055 ceph-mon[97033]: pgmap v19647: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:44:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:10 smithi055 ceph-mon[97033]: osdmap e1305: 12 total, 11 up, 11 in 2022-09-08T07:44:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:10 smithi059 ceph-mon[97450]: pgmap v19647: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:44:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:10 smithi059 ceph-mon[97450]: osdmap e1305: 12 total, 11 up, 11 in 2022-09-08T07:44:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:11.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:11.581979+0000) 2022-09-08T07:44:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:11.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:11.699469+0000) 2022-09-08T07:44:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:12.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:12.699693+0000) 2022-09-08T07:44:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:12.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:12.582123+0000) 2022-09-08T07:44:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:12 smithi055 ceph-mon[97033]: pgmap v19649: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:44:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:12 smithi059 ceph-mon[97450]: pgmap v19649: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:44:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:13.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:13.699866+0000) 2022-09-08T07:44:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:13.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:13.582381+0000) 2022-09-08T07:44:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:14.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:14.700051+0000) 2022-09-08T07:44:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:14.581+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:14.582599+0000) 2022-09-08T07:44:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:14 smithi059 ceph-mon[97450]: pgmap v19650: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:44:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:14 smithi055 ceph-mon[97033]: pgmap v19650: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:44:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:15.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:15.700266+0000) 2022-09-08T07:44:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:15.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:15.582703+0000) 2022-09-08T07:44:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:16.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:16.582940+0000) 2022-09-08T07:44:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:16.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:16.700514+0000) 2022-09-08T07:44:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:16 smithi059 ceph-mon[97450]: pgmap v19651: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:44:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:16 smithi055 ceph-mon[97033]: pgmap v19651: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:44:17.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:17.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:17.700777+0000) 2022-09-08T07:44:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:17.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:17.583098+0000) 2022-09-08T07:44:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:18.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:18.583394+0000) 2022-09-08T07:44:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:18.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:18.700947+0000) 2022-09-08T07:44:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:18 smithi059 ceph-mon[97450]: pgmap v19652: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:44:19.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:18 smithi059 ceph-mon[97450]: osdmap e1306: 12 total, 11 up, 11 in 2022-09-08T07:44:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:18 smithi055 ceph-mon[97033]: pgmap v19652: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:44:19.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:18 smithi055 ceph-mon[97033]: osdmap e1306: 12 total, 11 up, 11 in 2022-09-08T07:44:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:19.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:19.701186+0000) 2022-09-08T07:44:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:19.582+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:19.583570+0000) 2022-09-08T07:44:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:20.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:20.701426+0000) 2022-09-08T07:44:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:20.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:20.583709+0000) 2022-09-08T07:44:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:20 smithi059 ceph-mon[97450]: pgmap v19654: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T07:44:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:20 smithi055 ceph-mon[97033]: pgmap v19654: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T07:44:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:21.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:21.701659+0000) 2022-09-08T07:44:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:21.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:21.583845+0000) 2022-09-08T07:44:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:22.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:22.701764+0000) 2022-09-08T07:44:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:22.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:22.583990+0000) 2022-09-08T07:44:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:22 smithi059 ceph-mon[97450]: pgmap v19655: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:44:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:22 smithi055 ceph-mon[97033]: pgmap v19655: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:44:23.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:44:23.651+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:44:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:23.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:23.701987+0000) 2022-09-08T07:44:23.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:23.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:23.584157+0000) 2022-09-08T07:44:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:24.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:24.702229+0000) 2022-09-08T07:44:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:24.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:24.584315+0000) 2022-09-08T07:44:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:24 smithi059 ceph-mon[97450]: pgmap v19656: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:44:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:24 smithi055 ceph-mon[97033]: pgmap v19656: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:44:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:25.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:25.702472+0000) 2022-09-08T07:44:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:25.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:25.584470+0000) 2022-09-08T07:44:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:26.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:26.702591+0000) 2022-09-08T07:44:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:26.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:26.584706+0000) 2022-09-08T07:44:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:27 smithi059 ceph-mon[97450]: pgmap v19657: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T07:44:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:27 smithi055 ceph-mon[97033]: pgmap v19657: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T07:44:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:27.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:27.702780+0000) 2022-09-08T07:44:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:27.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:27.584898+0000) 2022-09-08T07:44:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:28.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:28.702980+0000) 2022-09-08T07:44:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:28.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:28.585031+0000) 2022-09-08T07:44:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:29 smithi059 ceph-mon[97450]: pgmap v19658: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T07:44:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:29 smithi055 ceph-mon[97033]: pgmap v19658: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T07:44:29.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:29.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:29.703227+0000) 2022-09-08T07:44:29.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:29.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:29.585270+0000) 2022-09-08T07:44:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:30.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:30.703430+0000) 2022-09-08T07:44:30.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:30.583+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:30.585444+0000) 2022-09-08T07:44:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:31 smithi059 ceph-mon[97450]: pgmap v19659: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:44:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:31 smithi055 ceph-mon[97033]: pgmap v19659: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:44:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:31.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:31.703678+0000) 2022-09-08T07:44:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:31.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:31.585685+0000) 2022-09-08T07:44:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:32.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:32.703842+0000) 2022-09-08T07:44:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:32.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:32.585854+0000) 2022-09-08T07:44:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:33 smithi059 ceph-mon[97450]: pgmap v19660: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T07:44:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:33 smithi055 ceph-mon[97033]: pgmap v19660: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T07:44:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:33.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:33.704036+0000) 2022-09-08T07:44:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:33.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:33.586057+0000) 2022-09-08T07:44:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:34.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:34.586220+0000) 2022-09-08T07:44:34.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:34.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:34.704290+0000) 2022-09-08T07:44:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:35 smithi059 ceph-mon[97450]: pgmap v19661: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:35 smithi055 ceph-mon[97033]: pgmap v19661: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:35.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:35.704465+0000) 2022-09-08T07:44:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:35.584+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:35.586365+0000) 2022-09-08T07:44:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:36.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:36.704700+0000) 2022-09-08T07:44:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:36.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:36.586525+0000) 2022-09-08T07:44:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:37 smithi059 ceph-mon[97450]: pgmap v19662: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:44:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:37 smithi055 ceph-mon[97033]: pgmap v19662: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:44:37.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:37.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:37.704959+0000) 2022-09-08T07:44:37.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:37.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:37.586711+0000) 2022-09-08T07:44:38.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:44:38.657+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:44:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:38.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:38.705222+0000) 2022-09-08T07:44:38.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:38.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:38.586911+0000) 2022-09-08T07:44:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:39 smithi059 ceph-mon[97450]: pgmap v19663: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:39 smithi055 ceph-mon[97033]: pgmap v19663: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:39.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:39.705419+0000) 2022-09-08T07:44:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:39.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:39.587075+0000) 2022-09-08T07:44:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:40.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:40.705706+0000) 2022-09-08T07:44:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:40.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:40.587297+0000) 2022-09-08T07:44:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:41 smithi059 ceph-mon[97450]: pgmap v19664: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:41 smithi055 ceph-mon[97033]: pgmap v19664: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:41.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:41.705929+0000) 2022-09-08T07:44:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:41.585+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:41.587470+0000) 2022-09-08T07:44:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:42.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:42.706123+0000) 2022-09-08T07:44:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:42.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:42.587684+0000) 2022-09-08T07:44:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:43 smithi055 ceph-mon[97033]: pgmap v19665: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:44:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:43 smithi059 ceph-mon[97450]: pgmap v19665: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:44:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:43.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:43.706343+0000) 2022-09-08T07:44:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:43.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:43.587820+0000) 2022-09-08T07:44:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:44.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:44.706525+0000) 2022-09-08T07:44:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:44.586+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:44.587987+0000) 2022-09-08T07:44:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:45 smithi055 ceph-mon[97033]: pgmap v19666: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:45 smithi059 ceph-mon[97450]: pgmap v19666: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:45.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:45.706755+0000) 2022-09-08T07:44:45.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:45.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:45.588139+0000) 2022-09-08T07:44:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:46.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:46.706988+0000) 2022-09-08T07:44:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:46.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:46.588342+0000) 2022-09-08T07:44:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:47 smithi055 ceph-mon[97033]: pgmap v19667: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:44:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:47 smithi059 ceph-mon[97450]: pgmap v19667: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:44:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:47.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:47.707214+0000) 2022-09-08T07:44:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:47.587+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:47.588523+0000) 2022-09-08T07:44:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:48.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:48.707450+0000) 2022-09-08T07:44:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:48.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:48.588733+0000) 2022-09-08T07:44:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:49 smithi055 ceph-mon[97033]: pgmap v19668: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:49 smithi059 ceph-mon[97450]: pgmap v19668: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:49.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:49.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:49.707731+0000) 2022-09-08T07:44:49.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:49.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:49.588928+0000) 2022-09-08T07:44:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:50.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:50.707873+0000) 2022-09-08T07:44:50.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:50.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:50.589113+0000) 2022-09-08T07:44:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:51 smithi055 ceph-mon[97033]: pgmap v19669: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:51 smithi059 ceph-mon[97450]: pgmap v19669: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:51.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:51.708015+0000) 2022-09-08T07:44:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:51.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:51.589311+0000) 2022-09-08T07:44:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:52.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:52.708130+0000) 2022-09-08T07:44:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:52.588+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:52.589533+0000) 2022-09-08T07:44:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:53 smithi055 ceph-mon[97033]: pgmap v19670: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:44:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:53 smithi059 ceph-mon[97450]: pgmap v19670: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:44:53.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:44:53.674+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:44:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:53.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:53.708347+0000) 2022-09-08T07:44:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:53.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:53.589703+0000) 2022-09-08T07:44:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:54.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:54.708491+0000) 2022-09-08T07:44:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:54.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:54.589846+0000) 2022-09-08T07:44:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:55 smithi055 ceph-mon[97033]: pgmap v19671: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:55 smithi059 ceph-mon[97450]: pgmap v19671: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:55.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:55.589964+0000) 2022-09-08T07:44:55.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:55.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:55.708730+0000) 2022-09-08T07:44:56.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:56.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:56.590102+0000) 2022-09-08T07:44:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:56.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:56.708843+0000) 2022-09-08T07:44:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:57 smithi055 ceph-mon[97033]: pgmap v19672: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:44:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:57 smithi059 ceph-mon[97450]: pgmap v19672: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:44:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:57.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:57.590281+0000) 2022-09-08T07:44:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:57.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:57.709083+0000) 2022-09-08T07:44:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:58.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:58.709301+0000) 2022-09-08T07:44:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:58.589+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:58.590413+0000) 2022-09-08T07:44:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:44:59 smithi055 ceph-mon[97033]: pgmap v19673: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:44:59 smithi059 ceph-mon[97450]: pgmap v19673: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:44:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:44:59.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:59.709542+0000) 2022-09-08T07:44:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:44:59.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:44:59.590668+0000) 2022-09-08T07:45:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:00 smithi055 ceph-mon[97033]: pgmap v19674: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:00 smithi059 ceph-mon[97450]: pgmap v19674: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:00.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:00.590899+0000) 2022-09-08T07:45:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:00.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:00.709689+0000) 2022-09-08T07:45:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:01.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:01.591104+0000) 2022-09-08T07:45:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:01.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:01.709920+0000) 2022-09-08T07:45:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:02.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:02.710159+0000) 2022-09-08T07:45:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:02.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:02.591251+0000) 2022-09-08T07:45:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:02 smithi055 ceph-mon[97033]: pgmap v19675: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:45:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:02 smithi059 ceph-mon[97450]: pgmap v19675: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:45:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:03.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:03.710380+0000) 2022-09-08T07:45:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:03.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:03.591387+0000) 2022-09-08T07:45:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:04.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:04.710626+0000) 2022-09-08T07:45:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:04.590+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:04.591538+0000) 2022-09-08T07:45:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:05 smithi059 ceph-mon[97450]: pgmap v19676: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:05 smithi055 ceph-mon[97033]: pgmap v19676: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:05.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:05.710841+0000) 2022-09-08T07:45:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:05.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:05.591711+0000) 2022-09-08T07:45:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:06.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:06.711074+0000) 2022-09-08T07:45:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:06.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:06.591882+0000) 2022-09-08T07:45:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:07 smithi059 ceph-mon[97450]: pgmap v19677: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:45:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:45:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:45:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:07 smithi055 ceph-mon[97033]: pgmap v19677: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:45:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:45:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:45:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:07.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:07.711363+0000) 2022-09-08T07:45:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:07.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:07.592058+0000) 2022-09-08T07:45:08.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:08.591+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:08.592244+0000) 2022-09-08T07:45:09.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:45:08.689+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:45:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:08.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:08.711674+0000) 2022-09-08T07:45:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:09 smithi059 ceph-mon[97450]: pgmap v19678: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:45:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:45:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:45:09.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:45:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:09 smithi055 ceph-mon[97033]: pgmap v19678: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:45:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:45:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:45:09.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:45:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:09.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:09.711835+0000) 2022-09-08T07:45:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:09.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:09.592479+0000) 2022-09-08T07:45:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:10 smithi059 ceph-mon[97450]: osdmap e1307: 12 total, 11 up, 11 in 2022-09-08T07:45:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:10 smithi055 ceph-mon[97033]: osdmap e1307: 12 total, 11 up, 11 in 2022-09-08T07:45:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:10.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:10.712006+0000) 2022-09-08T07:45:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:10.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:10.592661+0000) 2022-09-08T07:45:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:11 smithi059 ceph-mon[97450]: pgmap v19680: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:45:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:11 smithi055 ceph-mon[97033]: pgmap v19680: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:45:11.881 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:11.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:11.712233+0000) 2022-09-08T07:45:11.881 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:11.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:11.592798+0000) 2022-09-08T07:45:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:12.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:12.712454+0000) 2022-09-08T07:45:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:12.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:12.593041+0000) 2022-09-08T07:45:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:13 smithi059 ceph-mon[97450]: pgmap v19681: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:45:13.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:13 smithi059 ceph-mon[97450]: osdmap e1308: 12 total, 11 up, 11 in 2022-09-08T07:45:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:13 smithi055 ceph-mon[97033]: pgmap v19681: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:45:13.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:13 smithi055 ceph-mon[97033]: osdmap e1308: 12 total, 11 up, 11 in 2022-09-08T07:45:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:13.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:13.712685+0000) 2022-09-08T07:45:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:13.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:13.593199+0000) 2022-09-08T07:45:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:14.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:14.712908+0000) 2022-09-08T07:45:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:14.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:14.593359+0000) 2022-09-08T07:45:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:15 smithi059 ceph-mon[97450]: pgmap v19683: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.1 KiB/s rd, 127 B/s wr, 4 op/s 2022-09-08T07:45:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:15 smithi055 ceph-mon[97033]: pgmap v19683: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.1 KiB/s rd, 127 B/s wr, 4 op/s 2022-09-08T07:45:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:15.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:15.713160+0000) 2022-09-08T07:45:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:15.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:15.593561+0000) 2022-09-08T07:45:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:16.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:16.713413+0000) 2022-09-08T07:45:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:16.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:16.593752+0000) 2022-09-08T07:45:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:17 smithi059 ceph-mon[97450]: pgmap v19684: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2022-09-08T07:45:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:17 smithi055 ceph-mon[97033]: pgmap v19684: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2022-09-08T07:45:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:17.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:17.713658+0000) 2022-09-08T07:45:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:17.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:17.593900+0000) 2022-09-08T07:45:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:18.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:18.713875+0000) 2022-09-08T07:45:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:18.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:18.594047+0000) 2022-09-08T07:45:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:19 smithi055 ceph-mon[97033]: pgmap v19685: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 238 B/s wr, 7 op/s 2022-09-08T07:45:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:19 smithi059 ceph-mon[97450]: pgmap v19685: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 238 B/s wr, 7 op/s 2022-09-08T07:45:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:19.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:19.714108+0000) 2022-09-08T07:45:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:19.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:19.594216+0000) 2022-09-08T07:45:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:20.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:20.714276+0000) 2022-09-08T07:45:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:20.592+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:20.594389+0000) 2022-09-08T07:45:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:21 smithi055 ceph-mon[97033]: pgmap v19686: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:45:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:21 smithi059 ceph-mon[97450]: pgmap v19686: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:45:21.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:21.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:21.594607+0000) 2022-09-08T07:45:21.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:21.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:21.714503+0000) 2022-09-08T07:45:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:22.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:22.714753+0000) 2022-09-08T07:45:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:22.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:22.594883+0000) 2022-09-08T07:45:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:23 smithi055 ceph-mon[97033]: pgmap v19687: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:45:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:23 smithi059 ceph-mon[97450]: pgmap v19687: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:45:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:23.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:23.714898+0000) 2022-09-08T07:45:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:23.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:23.595112+0000) 2022-09-08T07:45:23.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:45:23.701+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:45:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:24.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:24.715130+0000) 2022-09-08T07:45:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:24.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:24.595259+0000) 2022-09-08T07:45:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:25 smithi055 ceph-mon[97033]: pgmap v19688: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T07:45:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:25 smithi059 ceph-mon[97450]: pgmap v19688: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 96 B/s wr, 5 op/s 2022-09-08T07:45:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:25.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:25.715334+0000) 2022-09-08T07:45:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:25.593+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:25.595413+0000) 2022-09-08T07:45:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:26.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:26.715559+0000) 2022-09-08T07:45:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:26.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:26.595580+0000) 2022-09-08T07:45:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:27 smithi055 ceph-mon[97033]: pgmap v19689: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:45:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:27 smithi059 ceph-mon[97450]: pgmap v19689: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:45:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:27.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:27.715707+0000) 2022-09-08T07:45:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:27.594+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:27.595786+0000) 2022-09-08T07:45:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:28.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:28.715879+0000) 2022-09-08T07:45:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:28.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:28.595996+0000) 2022-09-08T07:45:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:29 smithi055 ceph-mon[97033]: pgmap v19690: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:29 smithi059 ceph-mon[97450]: pgmap v19690: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:29.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:29.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:29.716111+0000) 2022-09-08T07:45:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:29.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:29.596209+0000) 2022-09-08T07:45:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:30.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:30.716343+0000) 2022-09-08T07:45:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:30.595+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:30.596443+0000) 2022-09-08T07:45:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:31 smithi055 ceph-mon[97033]: pgmap v19691: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:31 smithi059 ceph-mon[97450]: pgmap v19691: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:31.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:31.716584+0000) 2022-09-08T07:45:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:31.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:31.596704+0000) 2022-09-08T07:45:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:32.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:32.716829+0000) 2022-09-08T07:45:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:32.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:32.596923+0000) 2022-09-08T07:45:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:33 smithi055 ceph-mon[97033]: pgmap v19692: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:45:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:33 smithi059 ceph-mon[97450]: pgmap v19692: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:45:33.719 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:33.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:33.597119+0000) 2022-09-08T07:45:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:33.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:33.717028+0000) 2022-09-08T07:45:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:34.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:34.717218+0000) 2022-09-08T07:45:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:34.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:34.597316+0000) 2022-09-08T07:45:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:35 smithi055 ceph-mon[97033]: pgmap v19693: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:35 smithi059 ceph-mon[97450]: pgmap v19693: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:35.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:35.717461+0000) 2022-09-08T07:45:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:35.596+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:35.597485+0000) 2022-09-08T07:45:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:36.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:36.717725+0000) 2022-09-08T07:45:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:36.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:36.597736+0000) 2022-09-08T07:45:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:37 smithi055 ceph-mon[97033]: pgmap v19694: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:45:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:37 smithi059 ceph-mon[97450]: pgmap v19694: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:45:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:37.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:37.717971+0000) 2022-09-08T07:45:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:37.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:37.597909+0000) 2022-09-08T07:45:38.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:45:38.714+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:45:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:38.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:38.718149+0000) 2022-09-08T07:45:38.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:38.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:38.598052+0000) 2022-09-08T07:45:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:39 smithi055 ceph-mon[97033]: pgmap v19695: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:39.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:45:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:39 smithi059 ceph-mon[97450]: pgmap v19695: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:39.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:45:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:39.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:39.718363+0000) 2022-09-08T07:45:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:39.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:39.598187+0000) 2022-09-08T07:45:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:40 smithi055 ceph-mon[97033]: pgmap v19696: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:45:40.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:40 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T07:45:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:40 smithi059 ceph-mon[97450]: pgmap v19696: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:45:40.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:40 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T07:45:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:40.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:40.718583+0000) 2022-09-08T07:45:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:40.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:40.598340+0000) 2022-09-08T07:45:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:41 smithi055 ceph-mon[97033]: pgmap v19697: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T07:45:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:41 smithi059 ceph-mon[97450]: pgmap v19697: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T07:45:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:41.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:41.718751+0000) 2022-09-08T07:45:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:41.597+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:41.598549+0000) 2022-09-08T07:45:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:42.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:42.718973+0000) 2022-09-08T07:45:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:42.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:42.598754+0000) 2022-09-08T07:45:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:43.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:43.719161+0000) 2022-09-08T07:45:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:43.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:43.598945+0000) 2022-09-08T07:45:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:44 smithi059 ceph-mon[97450]: pgmap v19698: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:45:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:44 smithi055 ceph-mon[97033]: pgmap v19698: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:45:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:44.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:44.719375+0000) 2022-09-08T07:45:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:44.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:44.599139+0000) 2022-09-08T07:45:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:45.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:45.719628+0000) 2022-09-08T07:45:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:45.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:45.599278+0000) 2022-09-08T07:45:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:46 smithi059 ceph-mon[97450]: pgmap v19699: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:45:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:46 smithi055 ceph-mon[97033]: pgmap v19699: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:45:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:46.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:46.719884+0000) 2022-09-08T07:45:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:46.598+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:46.599410+0000) 2022-09-08T07:45:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:47.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:47.720072+0000) 2022-09-08T07:45:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:47.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:47.599610+0000) 2022-09-08T07:45:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:48 smithi059 ceph-mon[97450]: pgmap v19700: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:45:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:48 smithi055 ceph-mon[97033]: pgmap v19700: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:45:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:48.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:48.720231+0000) 2022-09-08T07:45:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:48.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:48.599809+0000) 2022-09-08T07:45:49.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:49.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:49.720416+0000) 2022-09-08T07:45:49.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:49.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:49.599967+0000) 2022-09-08T07:45:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:50 smithi059 ceph-mon[97450]: pgmap v19701: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:45:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:50 smithi055 ceph-mon[97033]: pgmap v19701: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:45:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:50.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:50.720603+0000) 2022-09-08T07:45:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:50.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:50.600150+0000) 2022-09-08T07:45:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:51.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:51.720874+0000) 2022-09-08T07:45:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:51.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:51.600334+0000) 2022-09-08T07:45:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:52 smithi055 ceph-mon[97033]: pgmap v19702: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:45:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:52 smithi059 ceph-mon[97450]: pgmap v19702: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:45:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:52.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:52.721154+0000) 2022-09-08T07:45:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:52.599+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:52.600499+0000) 2022-09-08T07:45:53.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:45:53.730+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:45:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:53.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:53.721377+0000) 2022-09-08T07:45:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:53.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:53.600718+0000) 2022-09-08T07:45:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:54 smithi055 ceph-mon[97033]: pgmap v19703: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:54 smithi059 ceph-mon[97450]: pgmap v19703: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:54.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:54.721548+0000) 2022-09-08T07:45:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:54.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:54.600879+0000) 2022-09-08T07:45:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:55.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:55.721751+0000) 2022-09-08T07:45:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:55.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:55.601030+0000) 2022-09-08T07:45:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:56 smithi055 ceph-mon[97033]: pgmap v19704: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:56 smithi059 ceph-mon[97450]: pgmap v19704: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:45:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:56.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:56.721961+0000) 2022-09-08T07:45:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:56.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:56.601168+0000) 2022-09-08T07:45:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:57.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:57.722200+0000) 2022-09-08T07:45:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:57.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:57.601325+0000) 2022-09-08T07:45:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:45:58 smithi055 ceph-mon[97033]: pgmap v19705: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:45:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:45:58 smithi059 ceph-mon[97450]: pgmap v19705: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:45:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:58.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:58.601491+0000) 2022-09-08T07:45:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:58.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:58.722462+0000) 2022-09-08T07:45:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:45:59.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:59.722713+0000) 2022-09-08T07:45:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:45:59.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:45:59.601660+0000) 2022-09-08T07:46:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:00 smithi055 ceph-mon[97033]: pgmap v19706: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:00 smithi059 ceph-mon[97450]: pgmap v19706: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:00.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:00.722895+0000) 2022-09-08T07:46:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:00.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:00.601822+0000) 2022-09-08T07:46:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:01.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:01.723028+0000) 2022-09-08T07:46:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:01.600+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:01.602067+0000) 2022-09-08T07:46:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:02 smithi055 ceph-mon[97033]: pgmap v19707: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:46:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:02 smithi059 ceph-mon[97450]: pgmap v19707: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:46:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:02.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:02.723233+0000) 2022-09-08T07:46:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:02.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:02.602320+0000) 2022-09-08T07:46:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:03.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:03.723402+0000) 2022-09-08T07:46:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:03.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:03.602468+0000) 2022-09-08T07:46:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:04 smithi055 ceph-mon[97033]: pgmap v19708: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:04 smithi059 ceph-mon[97450]: pgmap v19708: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:04.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:04.723671+0000) 2022-09-08T07:46:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:04.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:04.602625+0000) 2022-09-08T07:46:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:05.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:05.723936+0000) 2022-09-08T07:46:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:05.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:05.602790+0000) 2022-09-08T07:46:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:06 smithi055 ceph-mon[97033]: pgmap v19709: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:46:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:06 smithi059 ceph-mon[97450]: pgmap v19709: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:46:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:06.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:06.724135+0000) 2022-09-08T07:46:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:06.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:06.602943+0000) 2022-09-08T07:46:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:46:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:46:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:07.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:07.724378+0000) 2022-09-08T07:46:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:07.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:07.603123+0000) 2022-09-08T07:46:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:08 smithi055 ceph-mon[97033]: pgmap v19710: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:46:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:08 smithi059 ceph-mon[97450]: pgmap v19710: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:46:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:08.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:08.724691+0000) 2022-09-08T07:46:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:08.601+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:08.603312+0000) 2022-09-08T07:46:08.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:46:08.742+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:46:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:09.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:09.724939+0000) 2022-09-08T07:46:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:09.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:09.603457+0000) 2022-09-08T07:46:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:10 smithi055 ceph-mon[97033]: pgmap v19711: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:46:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:10 smithi055 ceph-mon[97033]: osdmap e1309: 12 total, 11 up, 11 in 2022-09-08T07:46:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:10 smithi059 ceph-mon[97450]: pgmap v19711: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:46:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:10 smithi059 ceph-mon[97450]: osdmap e1309: 12 total, 11 up, 11 in 2022-09-08T07:46:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:10.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:10.725207+0000) 2022-09-08T07:46:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:10.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:10.603657+0000) 2022-09-08T07:46:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:11.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:11.725373+0000) 2022-09-08T07:46:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:11.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:11.603845+0000) 2022-09-08T07:46:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:12 smithi055 ceph-mon[97033]: pgmap v19713: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:46:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:12 smithi059 ceph-mon[97450]: pgmap v19713: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:46:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:12.723+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:12.725522+0000) 2022-09-08T07:46:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:12.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:12.604080+0000) 2022-09-08T07:46:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:13.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:13.725704+0000) 2022-09-08T07:46:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:13.602+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:13.604244+0000) 2022-09-08T07:46:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:14 smithi059 ceph-mon[97450]: pgmap v19714: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:46:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:14 smithi059 ceph-mon[97450]: osdmap e1310: 12 total, 11 up, 11 in 2022-09-08T07:46:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:14 smithi055 ceph-mon[97033]: pgmap v19714: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:46:14.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:14 smithi055 ceph-mon[97033]: osdmap e1310: 12 total, 11 up, 11 in 2022-09-08T07:46:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:14.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:14.725921+0000) 2022-09-08T07:46:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:14.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:14.604467+0000) 2022-09-08T07:46:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:15.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:15.726209+0000) 2022-09-08T07:46:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:15.603+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:15.604738+0000) 2022-09-08T07:46:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:16 smithi059 ceph-mon[97450]: pgmap v19716: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:46:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:16 smithi055 ceph-mon[97033]: pgmap v19716: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:46:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:16.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:16.726388+0000) 2022-09-08T07:46:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:16.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:16.604956+0000) 2022-09-08T07:46:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:17.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:17.726548+0000) 2022-09-08T07:46:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:17.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:17.605087+0000) 2022-09-08T07:46:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:18 smithi059 ceph-mon[97450]: pgmap v19717: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:46:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:18 smithi055 ceph-mon[97033]: pgmap v19717: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:46:18.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:18.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:18.605244+0000) 2022-09-08T07:46:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:18.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:18.726767+0000) 2022-09-08T07:46:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:19.604+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:19.605485+0000) 2022-09-08T07:46:19.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:19.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:19.726967+0000) 2022-09-08T07:46:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:20 smithi055 ceph-mon[97033]: pgmap v19718: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 217 B/s wr, 6 op/s 2022-09-08T07:46:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:20 smithi059 ceph-mon[97450]: pgmap v19718: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 217 B/s wr, 6 op/s 2022-09-08T07:46:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:20.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:20.727163+0000) 2022-09-08T07:46:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:20.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:20.605685+0000) 2022-09-08T07:46:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:21.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:21.727384+0000) 2022-09-08T07:46:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:21.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:21.605914+0000) 2022-09-08T07:46:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:22 smithi055 ceph-mon[97033]: pgmap v19719: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:46:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:22 smithi059 ceph-mon[97450]: pgmap v19719: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:46:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:22.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:22.727603+0000) 2022-09-08T07:46:22.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:22.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:22.606078+0000) 2022-09-08T07:46:23.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:46:23.752+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:46:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:23.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:23.727788+0000) 2022-09-08T07:46:23.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:23.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:23.606236+0000) 2022-09-08T07:46:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:24 smithi055 ceph-mon[97033]: pgmap v19720: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:46:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:24 smithi059 ceph-mon[97450]: pgmap v19720: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:46:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:24.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:24.728010+0000) 2022-09-08T07:46:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:24.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:24.606389+0000) 2022-09-08T07:46:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:25.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:25.728240+0000) 2022-09-08T07:46:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:25.605+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:25.606554+0000) 2022-09-08T07:46:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:26 smithi055 ceph-mon[97033]: pgmap v19721: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T07:46:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:26 smithi059 ceph-mon[97450]: pgmap v19721: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2022-09-08T07:46:26.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:26.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:26.606746+0000) 2022-09-08T07:46:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:26.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:26.728469+0000) 2022-09-08T07:46:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:27.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:27.728728+0000) 2022-09-08T07:46:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:27.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:27.606918+0000) 2022-09-08T07:46:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:28 smithi055 ceph-mon[97033]: pgmap v19722: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:46:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:28 smithi059 ceph-mon[97450]: pgmap v19722: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:46:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:28.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:28.728927+0000) 2022-09-08T07:46:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:28.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:28.607057+0000) 2022-09-08T07:46:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:29.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:29.729074+0000) 2022-09-08T07:46:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:29.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:29.607194+0000) 2022-09-08T07:46:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:30 smithi055 ceph-mon[97033]: pgmap v19723: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:30 smithi059 ceph-mon[97450]: pgmap v19723: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:30.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:30.729288+0000) 2022-09-08T07:46:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:30.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:30.607355+0000) 2022-09-08T07:46:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:31.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:31.729519+0000) 2022-09-08T07:46:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:31.606+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:31.607507+0000) 2022-09-08T07:46:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:32 smithi055 ceph-mon[97033]: pgmap v19724: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:46:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:32 smithi059 ceph-mon[97450]: pgmap v19724: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:46:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:32.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:32.729805+0000) 2022-09-08T07:46:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:32.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:32.607664+0000) 2022-09-08T07:46:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:33.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:33.729995+0000) 2022-09-08T07:46:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:33.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:33.607849+0000) 2022-09-08T07:46:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:34 smithi055 ceph-mon[97033]: pgmap v19725: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:34 smithi059 ceph-mon[97450]: pgmap v19725: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:34.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:34.730157+0000) 2022-09-08T07:46:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:34.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:34.608035+0000) 2022-09-08T07:46:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:35.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:35.730376+0000) 2022-09-08T07:46:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:35.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:35.608268+0000) 2022-09-08T07:46:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:36 smithi055 ceph-mon[97033]: pgmap v19726: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:36 smithi059 ceph-mon[97450]: pgmap v19726: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:36.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:36.730622+0000) 2022-09-08T07:46:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:36.607+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:36.608519+0000) 2022-09-08T07:46:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:37.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:37.730865+0000) 2022-09-08T07:46:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:37.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:37.608685+0000) 2022-09-08T07:46:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:38 smithi055 ceph-mon[97033]: pgmap v19727: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:46:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:38 smithi059 ceph-mon[97450]: pgmap v19727: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:46:38.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:46:38.766+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:46:38.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:38.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:38.731044+0000) 2022-09-08T07:46:38.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:38.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:38.608832+0000) 2022-09-08T07:46:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:46:39.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:46:39.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:39 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:46:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:46:39.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:46:39.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:39 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:46:39.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:39.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:39.731333+0000) 2022-09-08T07:46:39.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:39.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:39.608976+0000) 2022-09-08T07:46:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:40 smithi055 ceph-mon[97033]: pgmap v19728: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:40 smithi059 ceph-mon[97450]: pgmap v19728: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:40.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:40.731595+0000) 2022-09-08T07:46:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:40.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:40.609162+0000) 2022-09-08T07:46:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:41.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:41.731832+0000) 2022-09-08T07:46:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:41.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:41.609371+0000) 2022-09-08T07:46:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:42 smithi055 ceph-mon[97033]: pgmap v19729: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:46:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:42 smithi059 ceph-mon[97450]: pgmap v19729: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:46:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:42.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:42.732110+0000) 2022-09-08T07:46:42.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:42.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:42.609578+0000) 2022-09-08T07:46:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:43.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:43.732315+0000) 2022-09-08T07:46:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:43.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:43.609711+0000) 2022-09-08T07:46:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:44 smithi055 ceph-mon[97033]: pgmap v19730: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:44 smithi059 ceph-mon[97450]: pgmap v19730: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:44.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:44.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:44.732523+0000) 2022-09-08T07:46:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:44.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:44.609927+0000) 2022-09-08T07:46:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:45 smithi055 ceph-mon[97033]: pgmap v19731: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:45 smithi059 ceph-mon[97450]: pgmap v19731: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:45.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:45.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:45.610109+0000) 2022-09-08T07:46:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:45.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:45.732749+0000) 2022-09-08T07:46:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:46.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:46.732947+0000) 2022-09-08T07:46:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:46.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:46.610342+0000) 2022-09-08T07:46:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:47.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:47.733171+0000) 2022-09-08T07:46:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:47.608+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:47.610485+0000) 2022-09-08T07:46:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:47 smithi055 ceph-mon[97033]: pgmap v19732: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:46:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:47 smithi059 ceph-mon[97450]: pgmap v19732: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:46:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:48.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:48.733353+0000) 2022-09-08T07:46:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:48.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:48.610672+0000) 2022-09-08T07:46:49.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:49.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:49.733537+0000) 2022-09-08T07:46:49.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:49.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:49.610896+0000) 2022-09-08T07:46:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:50 smithi055 ceph-mon[97033]: pgmap v19733: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:50 smithi059 ceph-mon[97450]: pgmap v19733: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:50.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:50.733766+0000) 2022-09-08T07:46:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:50.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:50.611097+0000) 2022-09-08T07:46:51.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:51.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:51.611295+0000) 2022-09-08T07:46:51.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:51.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:51.733983+0000) 2022-09-08T07:46:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:52 smithi055 ceph-mon[97033]: pgmap v19734: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:46:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:52 smithi059 ceph-mon[97450]: pgmap v19734: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:46:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:52.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:52.734198+0000) 2022-09-08T07:46:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:52.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:52.611448+0000) 2022-09-08T07:46:53.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:46:53.766+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:46:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:53.732+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:53.734390+0000) 2022-09-08T07:46:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:53.609+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:53.611595+0000) 2022-09-08T07:46:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:54 smithi055 ceph-mon[97033]: pgmap v19735: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:54 smithi059 ceph-mon[97450]: pgmap v19735: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:54.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:54.734621+0000) 2022-09-08T07:46:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:54.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:54.611785+0000) 2022-09-08T07:46:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:55.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:55.734860+0000) 2022-09-08T07:46:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:55.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:55.612008+0000) 2022-09-08T07:46:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:56 smithi055 ceph-mon[97033]: pgmap v19736: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:56 smithi059 ceph-mon[97450]: pgmap v19736: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:46:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:56.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:56.735061+0000) 2022-09-08T07:46:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:56.610+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:56.612156+0000) 2022-09-08T07:46:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:57.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:57.735224+0000) 2022-09-08T07:46:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:57.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:57.612302+0000) 2022-09-08T07:46:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:46:58 smithi055 ceph-mon[97033]: pgmap v19737: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:46:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:46:58 smithi059 ceph-mon[97450]: pgmap v19737: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:46:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:58.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:58.612501+0000) 2022-09-08T07:46:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:58.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:58.735445+0000) 2022-09-08T07:46:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:46:59.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:59.612688+0000) 2022-09-08T07:46:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:46:59.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:46:59.735707+0000) 2022-09-08T07:47:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:00 smithi055 ceph-mon[97033]: pgmap v19738: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:00 smithi059 ceph-mon[97450]: pgmap v19738: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:00.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:00.735895+0000) 2022-09-08T07:47:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:00.611+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:00.612907+0000) 2022-09-08T07:47:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:01.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:01.736099+0000) 2022-09-08T07:47:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:01.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:01.613129+0000) 2022-09-08T07:47:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:02 smithi055 ceph-mon[97033]: pgmap v19739: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:47:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:02 smithi059 ceph-mon[97450]: pgmap v19739: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:47:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:02.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:02.736264+0000) 2022-09-08T07:47:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:02.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:02.613330+0000) 2022-09-08T07:47:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:03.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:03.736507+0000) 2022-09-08T07:47:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:03.612+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:03.613495+0000) 2022-09-08T07:47:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:04 smithi055 ceph-mon[97033]: pgmap v19740: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:04 smithi059 ceph-mon[97450]: pgmap v19740: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:04.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:04.736760+0000) 2022-09-08T07:47:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:04.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:04.613677+0000) 2022-09-08T07:47:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:05.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:05.736928+0000) 2022-09-08T07:47:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:05.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:05.613891+0000) 2022-09-08T07:47:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:06 smithi055 ceph-mon[97033]: pgmap v19741: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:47:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:06 smithi059 ceph-mon[97450]: pgmap v19741: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:47:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:06.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:06.737205+0000) 2022-09-08T07:47:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:06.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:06.614038+0000) 2022-09-08T07:47:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:47:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:47:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:07.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:07.737471+0000) 2022-09-08T07:47:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:07.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:07.614264+0000) 2022-09-08T07:47:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:08 smithi055 ceph-mon[97033]: pgmap v19742: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:47:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:08 smithi059 ceph-mon[97450]: pgmap v19742: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:47:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:08.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:08.737703+0000) 2022-09-08T07:47:08.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:47:08.782+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:47:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:08.613+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:08.614481+0000) 2022-09-08T07:47:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:09.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:09.737918+0000) 2022-09-08T07:47:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:09.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:09.614710+0000) 2022-09-08T07:47:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:09 smithi059 ceph-mon[97450]: pgmap v19743: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:47:10.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:09 smithi059 ceph-mon[97450]: pgmap v19744: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T07:47:10.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:09 smithi059 ceph-mon[97450]: pgmap v19745: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T07:47:10.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:47:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:09 smithi055 ceph-mon[97033]: pgmap v19743: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:47:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:09 smithi055 ceph-mon[97033]: pgmap v19744: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 52 KiB/s wr, 23 op/s 2022-09-08T07:47:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:09 smithi055 ceph-mon[97033]: pgmap v19745: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T07:47:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:47:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:10.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:10.738104+0000) 2022-09-08T07:47:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:10.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:10.614873+0000) 2022-09-08T07:47:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:10 smithi055 ceph-mon[97033]: osdmap e1311: 12 total, 11 up, 11 in 2022-09-08T07:47:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:10 smithi059 ceph-mon[97450]: osdmap e1311: 12 total, 11 up, 11 in 2022-09-08T07:47:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:11.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:11.738371+0000) 2022-09-08T07:47:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:11.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:11.615034+0000) 2022-09-08T07:47:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:11 smithi055 ceph-mon[97033]: pgmap v19747: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T07:47:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:11 smithi059 ceph-mon[97450]: pgmap v19747: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T07:47:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:12.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:12.738615+0000) 2022-09-08T07:47:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:12.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:12.615171+0000) 2022-09-08T07:47:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:13.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:13.738811+0000) 2022-09-08T07:47:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:13.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:13.615321+0000) 2022-09-08T07:47:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:14 smithi055 ceph-mon[97033]: pgmap v19748: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T07:47:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:14 smithi059 ceph-mon[97450]: pgmap v19748: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T07:47:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:14.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:14.739064+0000) 2022-09-08T07:47:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:14.614+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:14.615459+0000) 2022-09-08T07:47:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:15.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:15.739307+0000) 2022-09-08T07:47:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:15.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:15.615601+0000) 2022-09-08T07:47:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:16 smithi055 ceph-mon[97033]: pgmap v19749: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T07:47:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:16 smithi059 ceph-mon[97450]: pgmap v19749: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T07:47:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:16.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:16.739519+0000) 2022-09-08T07:47:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:16.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:16.615766+0000) 2022-09-08T07:47:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:17.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:17.739752+0000) 2022-09-08T07:47:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:17.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:17.615973+0000) 2022-09-08T07:47:18.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:18 smithi055 ceph-mon[97033]: pgmap v19750: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:47:18.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:18 smithi055 ceph-mon[97033]: osdmap e1312: 12 total, 11 up, 11 in 2022-09-08T07:47:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:18 smithi059 ceph-mon[97450]: pgmap v19750: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:47:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:18 smithi059 ceph-mon[97450]: osdmap e1312: 12 total, 11 up, 11 in 2022-09-08T07:47:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:18.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:18.739992+0000) 2022-09-08T07:47:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:18.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:18.616168+0000) 2022-09-08T07:47:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:19.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:19.740158+0000) 2022-09-08T07:47:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:19.615+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:19.616348+0000) 2022-09-08T07:47:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:20 smithi055 ceph-mon[97033]: pgmap v19752: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T07:47:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:20 smithi059 ceph-mon[97450]: pgmap v19752: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T07:47:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:20.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:20.740386+0000) 2022-09-08T07:47:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:20.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:20.616573+0000) 2022-09-08T07:47:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:21.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:21.740664+0000) 2022-09-08T07:47:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:21.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:21.616815+0000) 2022-09-08T07:47:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:22 smithi055 ceph-mon[97033]: pgmap v19753: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:47:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:22 smithi059 ceph-mon[97450]: pgmap v19753: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:47:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:22.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:22.740884+0000) 2022-09-08T07:47:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:22.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:22.616980+0000) 2022-09-08T07:47:23.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:47:23.793+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:47:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:23.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:23.617139+0000) 2022-09-08T07:47:23.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:23.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:23.741040+0000) 2022-09-08T07:47:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:24 smithi055 ceph-mon[97033]: pgmap v19754: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:47:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:24 smithi059 ceph-mon[97450]: pgmap v19754: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:47:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:24.616+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:24.617281+0000) 2022-09-08T07:47:24.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:24.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:24.741240+0000) 2022-09-08T07:47:25.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:25.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:25.617415+0000) 2022-09-08T07:47:25.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:25.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:25.741496+0000) 2022-09-08T07:47:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:26 smithi055 ceph-mon[97033]: pgmap v19755: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:47:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:26 smithi059 ceph-mon[97450]: pgmap v19755: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:47:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:26.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:26.741769+0000) 2022-09-08T07:47:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:26.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:26.617554+0000) 2022-09-08T07:47:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:27.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:27.741941+0000) 2022-09-08T07:47:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:27.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:27.617684+0000) 2022-09-08T07:47:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:28 smithi055 ceph-mon[97033]: pgmap v19756: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:47:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:28 smithi059 ceph-mon[97450]: pgmap v19756: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:47:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:28.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:28.742162+0000) 2022-09-08T07:47:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:28.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:28.617897+0000) 2022-09-08T07:47:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:29.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:29.742417+0000) 2022-09-08T07:47:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:29.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:29.618110+0000) 2022-09-08T07:47:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:30 smithi055 ceph-mon[97033]: pgmap v19757: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:47:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:30 smithi059 ceph-mon[97450]: pgmap v19757: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:47:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:30.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:30.742668+0000) 2022-09-08T07:47:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:30.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:30.618261+0000) 2022-09-08T07:47:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:31.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:31.742885+0000) 2022-09-08T07:47:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:31.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:31.618420+0000) 2022-09-08T07:47:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:32 smithi055 ceph-mon[97033]: pgmap v19758: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:47:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:32 smithi059 ceph-mon[97450]: pgmap v19758: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:47:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:32.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:32.743103+0000) 2022-09-08T07:47:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:32.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:32.618593+0000) 2022-09-08T07:47:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:33.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:33.743330+0000) 2022-09-08T07:47:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:33.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:33.618842+0000) 2022-09-08T07:47:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:34 smithi055 ceph-mon[97033]: pgmap v19759: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:34 smithi059 ceph-mon[97450]: pgmap v19759: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:34.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:34.743583+0000) 2022-09-08T07:47:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:34.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:34.618969+0000) 2022-09-08T07:47:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:35 smithi055 ceph-mon[97033]: pgmap v19760: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:35 smithi059 ceph-mon[97450]: pgmap v19760: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:35.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:35.743783+0000) 2022-09-08T07:47:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:35.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:35.619143+0000) 2022-09-08T07:47:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:36.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:36.619394+0000) 2022-09-08T07:47:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:36.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:36.743973+0000) 2022-09-08T07:47:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:37.617+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:37.619521+0000) 2022-09-08T07:47:37.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:37.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:37.744185+0000) 2022-09-08T07:47:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:37 smithi055 ceph-mon[97033]: pgmap v19761: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:47:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:37 smithi059 ceph-mon[97450]: pgmap v19761: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:47:38.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:47:38.808+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:47:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:38.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:38.744425+0000) 2022-09-08T07:47:38.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:38.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:38.619665+0000) 2022-09-08T07:47:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:39.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:39.744629+0000) 2022-09-08T07:47:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:39.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:39.619879+0000) 2022-09-08T07:47:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:40 smithi055 ceph-mon[97033]: pgmap v19762: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:40 smithi059 ceph-mon[97450]: pgmap v19762: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:40.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:40.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:40.744911+0000) 2022-09-08T07:47:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:40.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:40.620105+0000) 2022-09-08T07:47:41.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:41.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:41.745159+0000) 2022-09-08T07:47:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:41.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:41.620287+0000) 2022-09-08T07:47:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:42 smithi055 ceph-mon[97033]: pgmap v19763: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:47:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:42 smithi059 ceph-mon[97450]: pgmap v19763: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:47:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:42.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:42.745389+0000) 2022-09-08T07:47:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:42.618+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:42.620459+0000) 2022-09-08T07:47:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:43.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:43.745616+0000) 2022-09-08T07:47:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:43.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:43.620708+0000) 2022-09-08T07:47:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:44 smithi055 ceph-mon[97033]: pgmap v19764: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:44 smithi059 ceph-mon[97450]: pgmap v19764: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:44.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:44.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:44.745806+0000) 2022-09-08T07:47:44.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:44.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:44.620845+0000) 2022-09-08T07:47:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:45.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:45.745987+0000) 2022-09-08T07:47:45.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:45.619+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:45.620981+0000) 2022-09-08T07:47:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:46 smithi055 ceph-mon[97033]: pgmap v19765: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:46 smithi059 ceph-mon[97450]: pgmap v19765: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:46.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:46.746225+0000) 2022-09-08T07:47:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:46.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:46.621123+0000) 2022-09-08T07:47:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:47.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:47.746473+0000) 2022-09-08T07:47:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:47.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:47.621258+0000) 2022-09-08T07:47:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:48 smithi055 ceph-mon[97033]: pgmap v19766: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:47:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:48 smithi059 ceph-mon[97450]: pgmap v19766: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:47:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:48.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:48.746682+0000) 2022-09-08T07:47:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:48.620+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:48.621478+0000) 2022-09-08T07:47:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:49.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:49.746864+0000) 2022-09-08T07:47:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:49.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:49.621716+0000) 2022-09-08T07:47:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:50 smithi055 ceph-mon[97033]: pgmap v19767: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:50 smithi059 ceph-mon[97450]: pgmap v19767: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:50.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:50.747096+0000) 2022-09-08T07:47:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:50.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:50.621884+0000) 2022-09-08T07:47:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:51.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:51.747350+0000) 2022-09-08T07:47:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:51.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:51.622076+0000) 2022-09-08T07:47:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:52 smithi055 ceph-mon[97033]: pgmap v19768: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:47:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:52 smithi059 ceph-mon[97450]: pgmap v19768: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:47:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:52.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:52.747561+0000) 2022-09-08T07:47:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:52.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:52.622213+0000) 2022-09-08T07:47:53.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:47:53.824+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:47:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:53.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:53.747770+0000) 2022-09-08T07:47:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:53.621+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:53.622434+0000) 2022-09-08T07:47:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:54 smithi055 ceph-mon[97033]: pgmap v19769: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:54 smithi059 ceph-mon[97450]: pgmap v19769: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:54.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:54.622625+0000) 2022-09-08T07:47:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:54.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:54.747966+0000) 2022-09-08T07:47:55.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:55.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:55.622838+0000) 2022-09-08T07:47:55.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:55.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:55.748175+0000) 2022-09-08T07:47:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:56 smithi055 ceph-mon[97033]: pgmap v19770: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:56 smithi059 ceph-mon[97450]: pgmap v19770: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:47:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:56.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:56.748412+0000) 2022-09-08T07:47:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:56.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:56.623018+0000) 2022-09-08T07:47:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:57.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:57.748651+0000) 2022-09-08T07:47:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:57.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:57.623162+0000) 2022-09-08T07:47:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:47:58 smithi055 ceph-mon[97033]: pgmap v19771: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:47:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:47:58 smithi059 ceph-mon[97450]: pgmap v19771: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:47:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:58.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:58.748873+0000) 2022-09-08T07:47:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:58.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:58.623298+0000) 2022-09-08T07:47:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:47:59.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:59.749087+0000) 2022-09-08T07:47:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:47:59.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:47:59.623419+0000) 2022-09-08T07:48:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:00 smithi055 ceph-mon[97033]: pgmap v19772: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:48:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:00 smithi059 ceph-mon[97450]: pgmap v19772: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:48:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:00.622+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:00.623534+0000) 2022-09-08T07:48:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:00.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:00.749291+0000) 2022-09-08T07:48:01.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:01.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:01.623736+0000) 2022-09-08T07:48:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:01.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:01.749531+0000) 2022-09-08T07:48:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:02 smithi055 ceph-mon[97033]: pgmap v19773: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:48:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:02 smithi059 ceph-mon[97450]: pgmap v19773: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:48:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:02.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:02.749745+0000) 2022-09-08T07:48:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:02.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:02.623878+0000) 2022-09-08T07:48:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:03.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:03.624000+0000) 2022-09-08T07:48:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:03.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:03.749942+0000) 2022-09-08T07:48:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:04 smithi055 ceph-mon[97033]: pgmap v19774: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:48:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:04 smithi059 ceph-mon[97450]: pgmap v19774: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:48:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:04.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:04.750056+0000) 2022-09-08T07:48:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:04.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:04.624172+0000) 2022-09-08T07:48:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:05.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:05.624355+0000) 2022-09-08T07:48:05.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:05.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:05.750242+0000) 2022-09-08T07:48:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:06 smithi055 ceph-mon[97033]: pgmap v19775: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:48:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:48:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:06 smithi059 ceph-mon[97450]: pgmap v19775: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:48:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:48:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:06.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:06.750441+0000) 2022-09-08T07:48:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:06.623+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:06.624516+0000) 2022-09-08T07:48:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:07 smithi055 ceph-mon[97033]: pgmap v19776: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:48:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:48:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:07 smithi059 ceph-mon[97450]: pgmap v19776: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:48:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:48:07.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:07.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:07.750700+0000) 2022-09-08T07:48:07.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:07.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:07.624694+0000) 2022-09-08T07:48:08.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:48:08.833+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:48:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:08.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:08.750940+0000) 2022-09-08T07:48:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:08.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:08.624859+0000) 2022-09-08T07:48:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:48:09.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:48:09.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:48:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:48:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:48:09.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:48:09.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:09.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:09.751166+0000) 2022-09-08T07:48:09.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:09.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:09.625018+0000) 2022-09-08T07:48:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:10 smithi055 ceph-mon[97033]: pgmap v19777: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:48:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:10 smithi055 ceph-mon[97033]: osdmap e1313: 12 total, 11 up, 11 in 2022-09-08T07:48:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:48:10.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:48:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:10 smithi059 ceph-mon[97450]: pgmap v19777: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:48:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:10 smithi059 ceph-mon[97450]: osdmap e1313: 12 total, 11 up, 11 in 2022-09-08T07:48:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:48:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:48:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:10.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:10.751313+0000) 2022-09-08T07:48:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:10.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:10.625177+0000) 2022-09-08T07:48:11.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:11.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:11.625334+0000) 2022-09-08T07:48:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:11.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:11.751475+0000) 2022-09-08T07:48:12.398 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:12 smithi055 ceph-mon[97033]: pgmap v19779: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:48:12.398 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:48:12.399 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:12 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:48:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:12 smithi059 ceph-mon[97450]: pgmap v19779: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:48:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:48:12.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:12 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:48:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:12.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:12.751699+0000) 2022-09-08T07:48:12.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:12.624+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:12.625490+0000) 2022-09-08T07:48:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:48:13.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:48:13.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:13 smithi055 ceph-mon[97033]: osdmap e1314: 12 total, 11 up, 11 in 2022-09-08T07:48:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:48:13.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:48:13.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:48:13.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:13 smithi059 ceph-mon[97450]: osdmap e1314: 12 total, 11 up, 11 in 2022-09-08T07:48:13.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:13.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:13.751829+0000) 2022-09-08T07:48:13.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:13.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:13.625694+0000) 2022-09-08T07:48:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:14 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 110.7M 2022-09-08T07:48:14.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:14 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 116148048: error parsing value: Value '116148048' is below minimum 939524096 2022-09-08T07:48:14.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:14 smithi055 ceph-mon[97033]: pgmap v19780: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:48:14.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:14 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T07:48:14.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:14 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T07:48:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:14 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 110.7M 2022-09-08T07:48:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:14 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 116148048: error parsing value: Value '116148048' is below minimum 939524096 2022-09-08T07:48:14.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:14 smithi059 ceph-mon[97450]: pgmap v19780: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:48:14.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:14 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T07:48:14.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:14 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T07:48:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:14.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:14.752001+0000) 2022-09-08T07:48:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:14.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:14.625929+0000) 2022-09-08T07:48:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:15.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:15.752287+0000) 2022-09-08T07:48:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:15.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:15.626090+0000) 2022-09-08T07:48:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:16 smithi055 ceph-mon[97033]: pgmap v19782: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:48:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:16 smithi059 ceph-mon[97450]: pgmap v19782: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:48:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:16.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:16.752536+0000) 2022-09-08T07:48:16.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:16.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:16.626311+0000) 2022-09-08T07:48:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:17.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:17.752757+0000) 2022-09-08T07:48:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:17.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:17.626464+0000) 2022-09-08T07:48:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:18 smithi055 ceph-mon[97033]: pgmap v19783: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:48:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:18 smithi059 ceph-mon[97450]: pgmap v19783: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:48:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:18.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:18.752979+0000) 2022-09-08T07:48:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:18.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:18.626724+0000) 2022-09-08T07:48:19.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:19.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:19.626961+0000) 2022-09-08T07:48:19.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:19.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:19.753182+0000) 2022-09-08T07:48:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:20 smithi055 ceph-mon[97033]: pgmap v19784: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 210 B/s wr, 6 op/s 2022-09-08T07:48:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:20 smithi059 ceph-mon[97450]: pgmap v19784: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 210 B/s wr, 6 op/s 2022-09-08T07:48:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:20.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:20.753400+0000) 2022-09-08T07:48:20.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:20.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:20.627144+0000) 2022-09-08T07:48:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:21.625+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:21.627340+0000) 2022-09-08T07:48:21.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:21.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:21.753657+0000) 2022-09-08T07:48:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:22 smithi055 ceph-mon[97033]: pgmap v19785: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:48:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:22 smithi059 ceph-mon[97450]: pgmap v19785: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:48:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:22.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:22.627471+0000) 2022-09-08T07:48:22.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:22.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:22.753903+0000) 2022-09-08T07:48:23.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:48:23.841+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:48:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:23.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:23.627654+0000) 2022-09-08T07:48:23.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:23.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:23.754116+0000) 2022-09-08T07:48:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:24 smithi055 ceph-mon[97033]: pgmap v19786: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:48:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:24 smithi059 ceph-mon[97450]: pgmap v19786: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:48:24.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:24.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:24.627812+0000) 2022-09-08T07:48:24.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:24.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:24.754308+0000) 2022-09-08T07:48:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:25.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:25.754523+0000) 2022-09-08T07:48:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:25.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:25.627973+0000) 2022-09-08T07:48:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:26 smithi055 ceph-mon[97033]: pgmap v19787: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 87 B/s wr, 5 op/s 2022-09-08T07:48:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:26 smithi059 ceph-mon[97450]: pgmap v19787: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 87 B/s wr, 5 op/s 2022-09-08T07:48:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:26.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:26.754780+0000) 2022-09-08T07:48:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:26.626+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:26.628179+0000) 2022-09-08T07:48:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:27 smithi055 ceph-mon[97033]: pgmap v19788: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:48:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:27 smithi059 ceph-mon[97450]: pgmap v19788: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:48:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:27.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:27.755013+0000) 2022-09-08T07:48:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:27.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:27.628389+0000) 2022-09-08T07:48:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:28.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:28.755256+0000) 2022-09-08T07:48:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:28.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:28.628605+0000) 2022-09-08T07:48:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:29.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:29.755477+0000) 2022-09-08T07:48:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:29.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:29.628853+0000) 2022-09-08T07:48:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:30 smithi055 ceph-mon[97033]: pgmap v19789: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:48:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:30 smithi059 ceph-mon[97450]: pgmap v19789: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:48:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:30.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:30.755740+0000) 2022-09-08T07:48:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:30.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:30.629006+0000) 2022-09-08T07:48:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:31.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:31.755927+0000) 2022-09-08T07:48:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:31.627+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:31.629177+0000) 2022-09-08T07:48:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:32 smithi055 ceph-mon[97033]: pgmap v19790: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:48:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:32 smithi059 ceph-mon[97450]: pgmap v19790: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:48:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:32.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:32.756148+0000) 2022-09-08T07:48:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:32.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:32.629369+0000) 2022-09-08T07:48:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:33.628+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:33.629551+0000) 2022-09-08T07:48:33.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:33.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:33.756377+0000) 2022-09-08T07:48:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:34 smithi055 ceph-mon[97033]: pgmap v19791: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:48:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:34 smithi059 ceph-mon[97450]: pgmap v19791: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:48:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:34.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:34.756509+0000) 2022-09-08T07:48:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:34.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:34.629773+0000) 2022-09-08T07:48:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:35.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:35.756766+0000) 2022-09-08T07:48:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:35.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:35.629940+0000) 2022-09-08T07:48:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:36 smithi055 ceph-mon[97033]: pgmap v19792: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:48:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:36 smithi059 ceph-mon[97450]: pgmap v19792: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:48:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:36.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:36.757027+0000) 2022-09-08T07:48:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:36.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:36.630099+0000) 2022-09-08T07:48:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:37.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:37.757189+0000) 2022-09-08T07:48:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:37.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:37.630312+0000) 2022-09-08T07:48:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:38 smithi055 ceph-mon[97033]: pgmap v19793: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:48:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:38 smithi059 ceph-mon[97450]: pgmap v19793: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:48:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:38.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:38.757413+0000) 2022-09-08T07:48:38.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:38.629+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:38.630507+0000) 2022-09-08T07:48:38.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:48:38.846+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:48:39.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:39.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:39.757669+0000) 2022-09-08T07:48:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:39.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:39.630687+0000) 2022-09-08T07:48:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:40 smithi055 ceph-mon[97033]: pgmap v19794: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:48:40.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:40 smithi055 ceph-mon[97033]: pgmap v19795: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:48:40.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:40 smithi055 ceph-mon[97033]: pgmap v19796: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:48:40.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:48:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:40 smithi059 ceph-mon[97450]: pgmap v19794: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:48:40.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:40 smithi059 ceph-mon[97450]: pgmap v19795: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:48:40.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:40 smithi059 ceph-mon[97450]: pgmap v19796: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:48:40.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:48:40.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:40.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:40.757912+0000) 2022-09-08T07:48:40.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:40.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:40.630823+0000) 2022-09-08T07:48:41.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:41.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:41.758075+0000) 2022-09-08T07:48:41.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:41.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:41.631037+0000) 2022-09-08T07:48:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:42 smithi055 ceph-mon[97033]: pgmap v19797: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:48:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:42 smithi059 ceph-mon[97450]: pgmap v19797: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:48:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:42.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:42.758291+0000) 2022-09-08T07:48:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:42.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:42.631222+0000) 2022-09-08T07:48:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:43.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:43.758506+0000) 2022-09-08T07:48:43.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:43.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:43.631423+0000) 2022-09-08T07:48:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:44 smithi055 ceph-mon[97033]: pgmap v19798: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:48:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:44 smithi059 ceph-mon[97450]: pgmap v19798: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:48:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:44.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:44.758691+0000) 2022-09-08T07:48:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:44.630+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:44.631599+0000) 2022-09-08T07:48:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:45.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:45.758873+0000) 2022-09-08T07:48:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:45.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:45.631747+0000) 2022-09-08T07:48:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:46 smithi055 ceph-mon[97033]: pgmap v19799: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:48:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:46 smithi059 ceph-mon[97450]: pgmap v19799: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:48:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:46.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:46.759094+0000) 2022-09-08T07:48:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:46.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:46.631951+0000) 2022-09-08T07:48:47.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:47.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:47.759298+0000) 2022-09-08T07:48:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:47.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:47.632121+0000) 2022-09-08T07:48:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:48 smithi055 ceph-mon[97033]: pgmap v19800: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:48:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:48 smithi059 ceph-mon[97450]: pgmap v19800: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:48:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:48.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:48.632345+0000) 2022-09-08T07:48:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:48.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:48.759543+0000) 2022-09-08T07:48:49.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:49.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:49.759793+0000) 2022-09-08T07:48:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:49.631+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:49.632570+0000) 2022-09-08T07:48:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:50 smithi055 ceph-mon[97033]: pgmap v19801: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:48:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:50 smithi059 ceph-mon[97450]: pgmap v19801: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:48:50.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:50.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:50.632864+0000) 2022-09-08T07:48:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:50.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:50.759962+0000) 2022-09-08T07:48:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:51.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:51.760147+0000) 2022-09-08T07:48:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:51.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:51.633057+0000) 2022-09-08T07:48:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:52 smithi055 ceph-mon[97033]: pgmap v19802: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:48:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:52 smithi059 ceph-mon[97450]: pgmap v19802: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:48:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:52.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:52.760367+0000) 2022-09-08T07:48:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:52.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:52.633261+0000) 2022-09-08T07:48:53.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:48:53.861+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:48:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:53.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:53.760675+0000) 2022-09-08T07:48:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:53.632+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:53.633487+0000) 2022-09-08T07:48:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:54 smithi055 ceph-mon[97033]: pgmap v19803: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:48:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:54 smithi059 ceph-mon[97450]: pgmap v19803: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:48:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:54.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:54.633715+0000) 2022-09-08T07:48:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:54.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:54.760894+0000) 2022-09-08T07:48:55.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:55.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:55.761196+0000) 2022-09-08T07:48:55.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:55.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:55.633939+0000) 2022-09-08T07:48:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:56 smithi055 ceph-mon[97033]: pgmap v19804: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:48:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:56 smithi059 ceph-mon[97450]: pgmap v19804: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:48:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:56.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:56.761462+0000) 2022-09-08T07:48:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:56.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:56.634139+0000) 2022-09-08T07:48:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:57.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:57.761681+0000) 2022-09-08T07:48:57.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:57.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:57.634345+0000) 2022-09-08T07:48:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:48:58 smithi055 ceph-mon[97033]: pgmap v19805: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:48:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:48:58 smithi059 ceph-mon[97450]: pgmap v19805: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:48:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:58.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:58.761898+0000) 2022-09-08T07:48:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:58.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:58.634556+0000) 2022-09-08T07:48:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:48:59.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:59.762027+0000) 2022-09-08T07:48:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:48:59.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:48:59.634717+0000) 2022-09-08T07:49:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:00 smithi055 ceph-mon[97033]: pgmap v19806: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:49:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:00 smithi059 ceph-mon[97450]: pgmap v19806: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:49:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:00.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:00.762226+0000) 2022-09-08T07:49:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:00.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:00.634893+0000) 2022-09-08T07:49:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:01.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:01.762462+0000) 2022-09-08T07:49:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:01.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:01.635128+0000) 2022-09-08T07:49:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:02 smithi055 ceph-mon[97033]: pgmap v19807: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:49:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:02 smithi059 ceph-mon[97450]: pgmap v19807: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:49:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:02.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:02.762712+0000) 2022-09-08T07:49:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:02.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:02.635265+0000) 2022-09-08T07:49:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:03.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:03.762953+0000) 2022-09-08T07:49:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:03.633+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:03.635404+0000) 2022-09-08T07:49:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:04 smithi055 ceph-mon[97033]: pgmap v19808: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:04 smithi059 ceph-mon[97450]: pgmap v19808: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:04.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:04.763120+0000) 2022-09-08T07:49:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:04.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:04.635629+0000) 2022-09-08T07:49:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:05.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:05.635831+0000) 2022-09-08T07:49:05.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:05.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:05.763290+0000) 2022-09-08T07:49:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:06 smithi055 ceph-mon[97033]: pgmap v19809: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:49:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:49:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:06 smithi059 ceph-mon[97450]: pgmap v19809: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:49:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:49:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:06.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:06.763547+0000) 2022-09-08T07:49:06.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:06.634+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:06.636004+0000) 2022-09-08T07:49:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:49:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:49:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:07.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:07.763828+0000) 2022-09-08T07:49:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:07.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:07.636236+0000) 2022-09-08T07:49:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:08 smithi055 ceph-mon[97033]: pgmap v19810: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:49:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:08 smithi059 ceph-mon[97450]: pgmap v19810: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:49:08.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:49:08.874+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:49:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:08.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:08.764022+0000) 2022-09-08T07:49:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:08.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:08.636485+0000) 2022-09-08T07:49:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:09 smithi055 ceph-mon[97033]: pgmap v19811: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:49:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:09 smithi059 ceph-mon[97450]: pgmap v19811: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:49:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:09.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:09.764188+0000) 2022-09-08T07:49:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:09.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:09.636652+0000) 2022-09-08T07:49:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:10 smithi055 ceph-mon[97033]: osdmap e1315: 12 total, 11 up, 11 in 2022-09-08T07:49:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:10 smithi059 ceph-mon[97450]: osdmap e1315: 12 total, 11 up, 11 in 2022-09-08T07:49:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:10.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:10.764429+0000) 2022-09-08T07:49:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:10.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:10.636831+0000) 2022-09-08T07:49:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:11 smithi055 ceph-mon[97033]: pgmap v19813: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:49:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:11 smithi059 ceph-mon[97450]: pgmap v19813: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:49:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:11.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:11.764633+0000) 2022-09-08T07:49:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:11.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:11.636973+0000) 2022-09-08T07:49:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:12.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:12.764936+0000) 2022-09-08T07:49:12.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:12.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:12.637140+0000) 2022-09-08T07:49:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:13.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:13.765126+0000) 2022-09-08T07:49:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:13.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:13.637286+0000) 2022-09-08T07:49:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:14 smithi055 ceph-mon[97033]: pgmap v19814: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:49:14.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:14 smithi055 ceph-mon[97033]: osdmap e1316: 12 total, 11 up, 11 in 2022-09-08T07:49:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:14 smithi059 ceph-mon[97450]: pgmap v19814: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:49:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:14 smithi059 ceph-mon[97450]: osdmap e1316: 12 total, 11 up, 11 in 2022-09-08T07:49:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:14.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:14.765357+0000) 2022-09-08T07:49:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:14.635+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:14.637431+0000) 2022-09-08T07:49:15.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:15.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:15.637574+0000) 2022-09-08T07:49:15.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:15.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:15.765765+0000) 2022-09-08T07:49:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:16 smithi055 ceph-mon[97033]: pgmap v19816: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:49:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:16 smithi059 ceph-mon[97450]: pgmap v19816: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:49:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:16.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:16.766007+0000) 2022-09-08T07:49:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:16.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:16.637718+0000) 2022-09-08T07:49:17.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:17.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:17.766246+0000) 2022-09-08T07:49:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:17.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:17.637849+0000) 2022-09-08T07:49:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:18 smithi055 ceph-mon[97033]: pgmap v19817: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:49:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:18 smithi059 ceph-mon[97450]: pgmap v19817: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:49:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:18.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:18.766489+0000) 2022-09-08T07:49:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:18.636+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:18.638021+0000) 2022-09-08T07:49:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:19.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:19.766716+0000) 2022-09-08T07:49:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:19.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:19.638143+0000) 2022-09-08T07:49:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:20 smithi055 ceph-mon[97033]: pgmap v19818: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T07:49:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:20 smithi059 ceph-mon[97450]: pgmap v19818: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T07:49:20.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:20.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:20.638287+0000) 2022-09-08T07:49:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:20.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:20.766868+0000) 2022-09-08T07:49:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:21.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:21.767099+0000) 2022-09-08T07:49:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:21.637+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:21.638483+0000) 2022-09-08T07:49:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:22 smithi055 ceph-mon[97033]: pgmap v19819: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:49:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:22 smithi059 ceph-mon[97450]: pgmap v19819: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:49:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:22.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:22.767329+0000) 2022-09-08T07:49:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:22.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:22.638682+0000) 2022-09-08T07:49:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:23.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:23.767504+0000) 2022-09-08T07:49:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:23.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:23.638901+0000) 2022-09-08T07:49:23.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:49:23.883+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:49:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:24 smithi055 ceph-mon[97033]: pgmap v19820: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:49:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:24 smithi059 ceph-mon[97450]: pgmap v19820: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:49:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:24.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:24.767736+0000) 2022-09-08T07:49:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:24.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:24.639062+0000) 2022-09-08T07:49:25.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:25.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:25.768022+0000) 2022-09-08T07:49:25.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:25.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:25.639226+0000) 2022-09-08T07:49:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:26 smithi055 ceph-mon[97033]: pgmap v19821: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:49:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:26 smithi059 ceph-mon[97450]: pgmap v19821: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:49:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:26.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:26.768316+0000) 2022-09-08T07:49:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:26.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:26.639389+0000) 2022-09-08T07:49:27.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:27.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:27.768505+0000) 2022-09-08T07:49:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:27.638+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:27.639542+0000) 2022-09-08T07:49:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:28 smithi055 ceph-mon[97033]: pgmap v19822: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:28 smithi059 ceph-mon[97450]: pgmap v19822: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:28.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:28.768698+0000) 2022-09-08T07:49:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:28.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:28.639710+0000) 2022-09-08T07:49:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:29.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:29.768918+0000) 2022-09-08T07:49:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:29.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:29.639908+0000) 2022-09-08T07:49:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:30 smithi055 ceph-mon[97033]: pgmap v19823: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:30 smithi059 ceph-mon[97450]: pgmap v19823: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:30.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:30.769166+0000) 2022-09-08T07:49:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:30.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:30.640091+0000) 2022-09-08T07:49:31.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:31.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:31.769361+0000) 2022-09-08T07:49:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:31.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:31.640277+0000) 2022-09-08T07:49:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:32 smithi055 ceph-mon[97033]: pgmap v19824: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:49:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:32 smithi059 ceph-mon[97450]: pgmap v19824: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:49:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:32.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:32.769567+0000) 2022-09-08T07:49:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:32.639+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:32.640445+0000) 2022-09-08T07:49:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:33.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:33.769870+0000) 2022-09-08T07:49:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:33.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:33.640671+0000) 2022-09-08T07:49:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:34 smithi055 ceph-mon[97033]: pgmap v19825: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:34 smithi059 ceph-mon[97450]: pgmap v19825: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:34.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:34.770028+0000) 2022-09-08T07:49:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:34.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:34.640943+0000) 2022-09-08T07:49:35.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:35.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:35.770274+0000) 2022-09-08T07:49:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:35.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:35.641195+0000) 2022-09-08T07:49:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:36 smithi055 ceph-mon[97033]: pgmap v19826: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:49:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:36 smithi059 ceph-mon[97450]: pgmap v19826: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:49:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:36.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:36.770423+0000) 2022-09-08T07:49:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:36.640+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:36.641359+0000) 2022-09-08T07:49:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:37 smithi055 ceph-mon[97033]: pgmap v19827: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:37 smithi059 ceph-mon[97450]: pgmap v19827: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:37.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:37.641581+0000) 2022-09-08T07:49:37.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:37.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:37.770745+0000) 2022-09-08T07:49:38.902 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:38.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:38.770909+0000) 2022-09-08T07:49:38.902 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:38.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:38.641802+0000) 2022-09-08T07:49:39.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:49:38.898+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:49:39.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:39.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:39.771140+0000) 2022-09-08T07:49:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:39.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:39.642019+0000) 2022-09-08T07:49:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:40 smithi055 ceph-mon[97033]: pgmap v19828: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:40.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:49:40.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:49:40.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:49:40.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:49:40.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:49:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:40 smithi059 ceph-mon[97450]: pgmap v19828: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:40.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:49:40.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:49:40.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:49:40.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:49:40.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:49:40.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:40.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:40.771327+0000) 2022-09-08T07:49:40.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:40.641+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:40.642243+0000) 2022-09-08T07:49:41.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:41.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:41.642484+0000) 2022-09-08T07:49:41.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:41.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:41.771557+0000) 2022-09-08T07:49:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:42 smithi055 ceph-mon[97033]: pgmap v19829: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:49:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:42 smithi059 ceph-mon[97450]: pgmap v19829: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:49:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:42.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:42.642685+0000) 2022-09-08T07:49:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:42.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:42.771728+0000) 2022-09-08T07:49:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:43.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:43.771864+0000) 2022-09-08T07:49:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:43.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:43.642901+0000) 2022-09-08T07:49:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:44 smithi055 ceph-mon[97033]: pgmap v19830: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:44 smithi059 ceph-mon[97450]: pgmap v19830: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:44.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:44.772024+0000) 2022-09-08T07:49:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:44.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:44.643065+0000) 2022-09-08T07:49:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:45.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:45.772252+0000) 2022-09-08T07:49:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:45.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:45.643225+0000) 2022-09-08T07:49:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:46 smithi055 ceph-mon[97033]: pgmap v19831: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:49:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:46 smithi059 ceph-mon[97450]: pgmap v19831: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:49:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:46.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:46.772476+0000) 2022-09-08T07:49:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:46.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:46.643383+0000) 2022-09-08T07:49:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:47.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:47.772666+0000) 2022-09-08T07:49:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:47.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:47.643547+0000) 2022-09-08T07:49:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:48 smithi055 ceph-mon[97033]: pgmap v19832: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:48 smithi059 ceph-mon[97450]: pgmap v19832: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:48.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:48.772908+0000) 2022-09-08T07:49:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:48.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:48.643767+0000) 2022-09-08T07:49:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:49.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:49.773136+0000) 2022-09-08T07:49:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:49.642+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:49.643936+0000) 2022-09-08T07:49:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:50 smithi055 ceph-mon[97033]: pgmap v19833: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:50 smithi059 ceph-mon[97450]: pgmap v19833: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:50.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:50.773336+0000) 2022-09-08T07:49:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:50.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:50.644183+0000) 2022-09-08T07:49:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:51.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:51.773512+0000) 2022-09-08T07:49:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:51.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:51.644383+0000) 2022-09-08T07:49:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:52 smithi055 ceph-mon[97033]: pgmap v19834: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:49:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:52 smithi059 ceph-mon[97450]: pgmap v19834: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:49:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:52.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:52.644564+0000) 2022-09-08T07:49:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:52.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:52.773702+0000) 2022-09-08T07:49:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:53.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:53.773862+0000) 2022-09-08T07:49:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:53.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:53.644761+0000) 2022-09-08T07:49:54.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:49:53.905+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:49:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:54 smithi055 ceph-mon[97033]: pgmap v19835: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:54 smithi059 ceph-mon[97450]: pgmap v19835: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:54.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:54.774103+0000) 2022-09-08T07:49:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:54.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:54.644969+0000) 2022-09-08T07:49:55.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:55.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:55.774368+0000) 2022-09-08T07:49:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:55.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:55.645238+0000) 2022-09-08T07:49:56.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:56 smithi055 ceph-mon[97033]: pgmap v19836: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:49:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:56 smithi059 ceph-mon[97450]: pgmap v19836: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:49:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:56.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:56.774596+0000) 2022-09-08T07:49:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:56.643+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:56.645404+0000) 2022-09-08T07:49:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:49:57 smithi055 ceph-mon[97033]: pgmap v19837: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:49:57 smithi059 ceph-mon[97450]: pgmap v19837: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:49:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:57.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:57.774780+0000) 2022-09-08T07:49:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:57.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:57.645667+0000) 2022-09-08T07:49:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:58.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:58.774961+0000) 2022-09-08T07:49:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:58.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:58.645881+0000) 2022-09-08T07:49:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:49:59.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:59.775179+0000) 2022-09-08T07:49:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:49:59.644+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:49:59.646116+0000) 2022-09-08T07:50:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:00 smithi055 ceph-mon[97033]: pgmap v19838: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T07:50:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T07:50:00.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T07:50:00.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T07:50:00.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T07:50:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:00 smithi059 ceph-mon[97450]: pgmap v19838: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T07:50:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T07:50:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T07:50:00.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T07:50:00.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T07:50:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:00.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:00.775392+0000) 2022-09-08T07:50:00.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:00.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:00.646317+0000) 2022-09-08T07:50:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:01.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:01.775559+0000) 2022-09-08T07:50:01.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:01.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:01.646464+0000) 2022-09-08T07:50:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:02 smithi055 ceph-mon[97033]: pgmap v19839: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:50:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:02 smithi059 ceph-mon[97450]: pgmap v19839: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:50:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:02.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:02.775763+0000) 2022-09-08T07:50:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:02.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:02.646715+0000) 2022-09-08T07:50:03.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:03.645+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:03.646904+0000) 2022-09-08T07:50:03.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:03.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:03.775915+0000) 2022-09-08T07:50:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:04 smithi055 ceph-mon[97033]: pgmap v19840: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:04 smithi059 ceph-mon[97450]: pgmap v19840: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:04.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:04.776050+0000) 2022-09-08T07:50:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:04.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:04.647144+0000) 2022-09-08T07:50:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:05.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:05.776321+0000) 2022-09-08T07:50:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:05.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:05.647283+0000) 2022-09-08T07:50:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:06 smithi055 ceph-mon[97033]: pgmap v19841: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:50:06.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:50:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:06 smithi059 ceph-mon[97450]: pgmap v19841: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:50:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:50:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:06.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:06.776559+0000) 2022-09-08T07:50:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:06.646+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:06.647535+0000) 2022-09-08T07:50:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:50:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:50:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:07.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:07.776743+0000) 2022-09-08T07:50:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:07.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:07.647722+0000) 2022-09-08T07:50:08.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:08 smithi055 ceph-mon[97033]: pgmap v19842: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:50:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:08 smithi059 ceph-mon[97450]: pgmap v19842: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:50:08.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:08.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:08.776974+0000) 2022-09-08T07:50:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:08.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:08.647901+0000) 2022-09-08T07:50:09.325 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:50:08.920+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:50:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:09.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:09.777127+0000) 2022-09-08T07:50:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:09.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:09.648039+0000) 2022-09-08T07:50:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:10 smithi055 ceph-mon[97033]: pgmap v19843: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:50:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:10 smithi055 ceph-mon[97033]: osdmap e1317: 12 total, 11 up, 11 in 2022-09-08T07:50:10.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:50:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:10 smithi059 ceph-mon[97450]: pgmap v19843: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:50:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:10 smithi059 ceph-mon[97450]: osdmap e1317: 12 total, 11 up, 11 in 2022-09-08T07:50:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:50:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:10.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:10.777359+0000) 2022-09-08T07:50:10.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:10.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:10.648178+0000) 2022-09-08T07:50:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:11 smithi055 ceph-mon[97033]: pgmap v19845: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T07:50:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:11 smithi055 ceph-mon[97033]: pgmap v19846: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 19 KiB/s rd, 85 KiB/s wr, 33 op/s 2022-09-08T07:50:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:11 smithi059 ceph-mon[97450]: pgmap v19845: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15 KiB/s rd, 64 KiB/s wr, 25 op/s 2022-09-08T07:50:11.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:11 smithi059 ceph-mon[97450]: pgmap v19846: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 19 KiB/s rd, 85 KiB/s wr, 33 op/s 2022-09-08T07:50:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:11.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:11.777563+0000) 2022-09-08T07:50:11.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:11.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:11.648413+0000) 2022-09-08T07:50:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:12 smithi055 ceph-mon[97033]: pgmap v19847: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T07:50:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:12 smithi059 ceph-mon[97450]: pgmap v19847: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T07:50:12.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:12.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:12.777836+0000) 2022-09-08T07:50:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:12.647+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:12.648581+0000) 2022-09-08T07:50:13.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:13.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:13.648827+0000) 2022-09-08T07:50:13.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:13.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:13.778028+0000) 2022-09-08T07:50:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:14.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:14.778228+0000) 2022-09-08T07:50:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:14.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:14.649025+0000) 2022-09-08T07:50:15.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:15 smithi055 ceph-mon[97033]: pgmap v19848: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T07:50:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:15 smithi059 ceph-mon[97450]: pgmap v19848: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T07:50:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:15.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:15.778446+0000) 2022-09-08T07:50:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:15.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:15.649194+0000) 2022-09-08T07:50:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:16.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:16.778726+0000) 2022-09-08T07:50:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:16.648+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:16.649435+0000) 2022-09-08T07:50:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:17 smithi055 ceph-mon[97033]: pgmap v19849: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 5.1 KiB/s rd, 327 B/s wr, 10 op/s 2022-09-08T07:50:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:17 smithi059 ceph-mon[97450]: pgmap v19849: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 5.1 KiB/s rd, 327 B/s wr, 10 op/s 2022-09-08T07:50:17.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:17.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:17.778929+0000) 2022-09-08T07:50:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:17.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:17.649746+0000) 2022-09-08T07:50:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:18.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:18.779127+0000) 2022-09-08T07:50:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:18.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:18.649917+0000) 2022-09-08T07:50:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:19 smithi055 ceph-mon[97033]: pgmap v19850: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 249 B/s wr, 7 op/s 2022-09-08T07:50:19.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:19 smithi055 ceph-mon[97033]: osdmap e1318: 12 total, 11 up, 11 in 2022-09-08T07:50:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:19 smithi059 ceph-mon[97450]: pgmap v19850: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 249 B/s wr, 7 op/s 2022-09-08T07:50:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:19 smithi059 ceph-mon[97450]: osdmap e1318: 12 total, 11 up, 11 in 2022-09-08T07:50:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:19.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:19.779390+0000) 2022-09-08T07:50:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:19.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:19.650049+0000) 2022-09-08T07:50:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:20.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:20.779713+0000) 2022-09-08T07:50:20.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:20.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:20.650286+0000) 2022-09-08T07:50:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:21 smithi055 ceph-mon[97033]: pgmap v19852: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:50:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:21 smithi059 ceph-mon[97450]: pgmap v19852: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:50:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:21.649+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:21.650520+0000) 2022-09-08T07:50:21.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:21.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:21.779984+0000) 2022-09-08T07:50:22.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:22.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:22.650729+0000) 2022-09-08T07:50:22.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:22.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:22.780184+0000) 2022-09-08T07:50:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:23 smithi055 ceph-mon[97033]: pgmap v19853: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:50:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:23 smithi059 ceph-mon[97450]: pgmap v19853: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:50:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:23.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:23.780351+0000) 2022-09-08T07:50:23.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:23.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:23.650936+0000) 2022-09-08T07:50:24.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:50:23.934+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:50:24.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:24.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:24.780519+0000) 2022-09-08T07:50:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:24.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:24.651143+0000) 2022-09-08T07:50:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:25 smithi055 ceph-mon[97033]: pgmap v19854: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:50:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:25 smithi059 ceph-mon[97450]: pgmap v19854: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:50:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:25.650+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:25.651325+0000) 2022-09-08T07:50:26.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:25.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:25.780706+0000) 2022-09-08T07:50:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:26.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:26.780936+0000) 2022-09-08T07:50:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:26.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:26.651535+0000) 2022-09-08T07:50:27.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:27 smithi055 ceph-mon[97033]: pgmap v19855: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:50:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:27 smithi059 ceph-mon[97450]: pgmap v19855: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:50:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:27.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:27.781146+0000) 2022-09-08T07:50:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:27.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:27.651687+0000) 2022-09-08T07:50:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:28.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:28.781344+0000) 2022-09-08T07:50:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:28.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:28.651815+0000) 2022-09-08T07:50:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:29 smithi055 ceph-mon[97033]: pgmap v19856: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:50:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:29 smithi059 ceph-mon[97450]: pgmap v19856: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:50:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:29.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:29.781543+0000) 2022-09-08T07:50:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:29.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:29.651984+0000) 2022-09-08T07:50:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:30.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:30.781708+0000) 2022-09-08T07:50:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:30.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:30.652199+0000) 2022-09-08T07:50:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:31 smithi055 ceph-mon[97033]: pgmap v19857: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:50:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:31 smithi059 ceph-mon[97450]: pgmap v19857: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:50:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:31.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:31.781960+0000) 2022-09-08T07:50:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:31.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:31.652422+0000) 2022-09-08T07:50:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:32.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:32.782133+0000) 2022-09-08T07:50:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:32.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:32.652612+0000) 2022-09-08T07:50:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:33 smithi055 ceph-mon[97033]: pgmap v19858: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:50:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:33 smithi059 ceph-mon[97450]: pgmap v19858: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:50:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:33.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:33.782327+0000) 2022-09-08T07:50:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:33.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:33.652855+0000) 2022-09-08T07:50:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:34.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:34.653069+0000) 2022-09-08T07:50:35.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:34.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:34.782594+0000) 2022-09-08T07:50:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:35 smithi055 ceph-mon[97033]: pgmap v19859: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:35 smithi059 ceph-mon[97450]: pgmap v19859: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:35.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:35.782845+0000) 2022-09-08T07:50:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:35.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:35.653260+0000) 2022-09-08T07:50:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:36 smithi055 ceph-mon[97033]: pgmap v19860: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:50:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:36 smithi059 ceph-mon[97450]: pgmap v19860: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:50:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:36.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:36.783079+0000) 2022-09-08T07:50:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:36.651+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:36.653413+0000) 2022-09-08T07:50:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:37.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:37.783319+0000) 2022-09-08T07:50:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:37.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:37.653664+0000) 2022-09-08T07:50:38.943 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:38.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:38.783584+0000) 2022-09-08T07:50:38.943 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:38.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:38.653874+0000) 2022-09-08T07:50:39.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:50:38.938+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:50:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:39 smithi055 ceph-mon[97033]: pgmap v19861: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:39 smithi059 ceph-mon[97450]: pgmap v19861: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:39.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:39.783845+0000) 2022-09-08T07:50:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:39.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:39.654104+0000) 2022-09-08T07:50:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:40.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:40.784034+0000) 2022-09-08T07:50:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:40.652+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:40.654249+0000) 2022-09-08T07:50:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:41 smithi055 ceph-mon[97033]: pgmap v19862: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:41 smithi059 ceph-mon[97450]: pgmap v19862: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:41.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:41.654497+0000) 2022-09-08T07:50:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:41.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:41.784252+0000) 2022-09-08T07:50:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:42.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:42.784480+0000) 2022-09-08T07:50:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:42.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:42.654706+0000) 2022-09-08T07:50:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:43 smithi055 ceph-mon[97033]: pgmap v19863: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:50:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:43 smithi059 ceph-mon[97450]: pgmap v19863: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:50:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:43.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:43.784699+0000) 2022-09-08T07:50:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:43.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:43.654902+0000) 2022-09-08T07:50:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:44.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:44.784909+0000) 2022-09-08T07:50:45.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:44.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:44.655084+0000) 2022-09-08T07:50:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:45 smithi055 ceph-mon[97033]: pgmap v19864: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:45 smithi059 ceph-mon[97450]: pgmap v19864: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:45.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:45.785152+0000) 2022-09-08T07:50:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:45.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:45.655215+0000) 2022-09-08T07:50:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:46.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:46.785370+0000) 2022-09-08T07:50:47.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:46.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:46.655342+0000) 2022-09-08T07:50:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:47 smithi055 ceph-mon[97033]: pgmap v19865: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:50:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:47 smithi059 ceph-mon[97450]: pgmap v19865: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:50:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:47.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:47.785586+0000) 2022-09-08T07:50:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:47.653+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:47.655490+0000) 2022-09-08T07:50:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:48.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:48.785842+0000) 2022-09-08T07:50:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:48.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:48.655755+0000) 2022-09-08T07:50:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:49 smithi055 ceph-mon[97033]: pgmap v19866: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:49 smithi059 ceph-mon[97450]: pgmap v19866: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:49.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:49.786068+0000) 2022-09-08T07:50:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:49.654+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:49.655904+0000) 2022-09-08T07:50:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:50.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:50.786319+0000) 2022-09-08T07:50:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:50.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:50.656139+0000) 2022-09-08T07:50:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:51 smithi055 ceph-mon[97033]: pgmap v19867: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:51 smithi059 ceph-mon[97450]: pgmap v19867: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:51.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:51.786591+0000) 2022-09-08T07:50:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:51.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:51.656357+0000) 2022-09-08T07:50:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:52.655+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:52.656523+0000) 2022-09-08T07:50:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:52.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:52.786744+0000) 2022-09-08T07:50:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:53 smithi055 ceph-mon[97033]: pgmap v19868: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:50:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:53 smithi059 ceph-mon[97450]: pgmap v19868: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:50:53.946 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:53.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:53.786994+0000) 2022-09-08T07:50:53.947 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:53.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:53.656704+0000) 2022-09-08T07:50:54.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:50:53.942+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:50:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:54.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:54.787249+0000) 2022-09-08T07:50:55.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:54.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:54.656845+0000) 2022-09-08T07:50:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:55 smithi055 ceph-mon[97033]: pgmap v19869: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:55 smithi059 ceph-mon[97450]: pgmap v19869: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:55.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:55.787464+0000) 2022-09-08T07:50:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:55.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:55.657017+0000) 2022-09-08T07:50:56.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:56 smithi055 ceph-mon[97033]: pgmap v19870: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:50:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:56 smithi059 ceph-mon[97450]: pgmap v19870: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:50:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:56.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:56.787718+0000) 2022-09-08T07:50:57.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:56.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:56.657253+0000) 2022-09-08T07:50:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:57.656+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:57.657513+0000) 2022-09-08T07:50:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:57.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:57.787880+0000) 2022-09-08T07:50:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:58.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:58.657717+0000) 2022-09-08T07:50:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:58.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:58.788118+0000) 2022-09-08T07:50:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:50:59 smithi055 ceph-mon[97033]: pgmap v19871: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:50:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:50:59 smithi059 ceph-mon[97450]: pgmap v19871: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:00.158 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:50:59.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:59.657913+0000) 2022-09-08T07:51:00.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:50:59.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:50:59.788282+0000) 2022-09-08T07:51:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:00.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:00.658075+0000) 2022-09-08T07:51:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:00.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:00.788485+0000) 2022-09-08T07:51:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:01 smithi055 ceph-mon[97033]: pgmap v19872: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:01 smithi059 ceph-mon[97450]: pgmap v19872: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:01.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:01.658312+0000) 2022-09-08T07:51:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:01.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:01.788710+0000) 2022-09-08T07:51:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:02.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:02.788922+0000) 2022-09-08T07:51:03.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:02.657+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:02.658471+0000) 2022-09-08T07:51:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:03 smithi055 ceph-mon[97033]: pgmap v19873: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:51:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:03 smithi059 ceph-mon[97450]: pgmap v19873: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:51:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:03.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:03.789097+0000) 2022-09-08T07:51:04.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:03.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:03.658728+0000) 2022-09-08T07:51:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:04.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:04.789268+0000) 2022-09-08T07:51:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:04.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:04.658910+0000) 2022-09-08T07:51:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:05 smithi055 ceph-mon[97033]: pgmap v19874: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:05 smithi059 ceph-mon[97450]: pgmap v19874: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:05.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:05.789549+0000) 2022-09-08T07:51:06.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:05.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:05.659044+0000) 2022-09-08T07:51:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:51:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:51:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:06.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:06.789755+0000) 2022-09-08T07:51:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:06.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:06.659194+0000) 2022-09-08T07:51:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:07 smithi055 ceph-mon[97033]: pgmap v19875: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:51:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:51:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:07 smithi059 ceph-mon[97450]: pgmap v19875: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:51:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:51:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:07.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:07.789958+0000) 2022-09-08T07:51:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:07.658+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:07.659424+0000) 2022-09-08T07:51:08.952 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:08.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:08.790185+0000) 2022-09-08T07:51:08.952 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:08.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:08.659600+0000) 2022-09-08T07:51:09.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:51:08.948+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:51:09.478 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:09 smithi055 ceph-mon[97033]: pgmap v19876: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:51:09.480 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:09 smithi059 ceph-mon[97450]: pgmap v19876: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:51:09.792 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:09.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:09.659766+0000) 2022-09-08T07:51:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:09.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:09.790421+0000) 2022-09-08T07:51:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:10 smithi055 ceph-mon[97033]: osdmap e1319: 12 total, 11 up, 11 in 2022-09-08T07:51:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:10 smithi055 ceph-mon[97033]: pgmap v19878: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:51:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:51:10.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:51:10.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:51:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:10 smithi059 ceph-mon[97450]: osdmap e1319: 12 total, 11 up, 11 in 2022-09-08T07:51:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:10 smithi059 ceph-mon[97450]: pgmap v19878: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:51:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:51:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:51:10.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:51:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:10.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:10.790628+0000) 2022-09-08T07:51:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:10.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:10.659978+0000) 2022-09-08T07:51:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:11.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:11.790884+0000) 2022-09-08T07:51:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:11.659+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:11.660212+0000) 2022-09-08T07:51:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:12 smithi055 ceph-mon[97033]: pgmap v19879: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:51:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:12 smithi059 ceph-mon[97450]: pgmap v19879: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:51:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:12.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:12.791099+0000) 2022-09-08T07:51:13.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:12.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:12.660417+0000) 2022-09-08T07:51:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:13.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:13.791336+0000) 2022-09-08T07:51:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:13.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:13.660652+0000) 2022-09-08T07:51:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:14 smithi055 ceph-mon[97033]: osdmap e1320: 12 total, 11 up, 11 in 2022-09-08T07:51:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:14 smithi059 ceph-mon[97450]: osdmap e1320: 12 total, 11 up, 11 in 2022-09-08T07:51:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:14.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:14.791585+0000) 2022-09-08T07:51:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:14.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:14.660866+0000) 2022-09-08T07:51:15.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:15 smithi055 ceph-mon[97033]: pgmap v19881: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:51:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:15 smithi059 ceph-mon[97450]: pgmap v19881: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:51:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:15.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:15.791811+0000) 2022-09-08T07:51:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:15.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:15.661112+0000) 2022-09-08T07:51:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:16.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:16.792022+0000) 2022-09-08T07:51:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:16.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:16.661326+0000) 2022-09-08T07:51:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:17 smithi055 ceph-mon[97033]: pgmap v19882: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:51:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:17 smithi059 ceph-mon[97450]: pgmap v19882: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:51:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:17.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:17.792267+0000) 2022-09-08T07:51:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:17.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:17.661542+0000) 2022-09-08T07:51:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:18.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:18.792506+0000) 2022-09-08T07:51:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:18.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:18.661795+0000) 2022-09-08T07:51:19.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:19 smithi055 ceph-mon[97033]: pgmap v19883: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 249 B/s wr, 7 op/s 2022-09-08T07:51:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:19 smithi059 ceph-mon[97450]: pgmap v19883: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 249 B/s wr, 7 op/s 2022-09-08T07:51:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:19.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:19.792662+0000) 2022-09-08T07:51:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:19.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:19.662030+0000) 2022-09-08T07:51:20.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:20 smithi055 ceph-mon[97033]: pgmap v19884: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:51:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:20 smithi059 ceph-mon[97450]: pgmap v19884: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:51:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:20.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:20.792841+0000) 2022-09-08T07:51:21.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:20.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:20.662259+0000) 2022-09-08T07:51:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:21.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:21.793057+0000) 2022-09-08T07:51:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:21.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:21.662455+0000) 2022-09-08T07:51:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:22 smithi055 ceph-mon[97033]: pgmap v19885: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:51:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:22 smithi059 ceph-mon[97450]: pgmap v19885: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:51:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:22.791+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:22.793290+0000) 2022-09-08T07:51:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:22.660+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:22.662612+0000) 2022-09-08T07:51:23.963 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:23.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:23.793529+0000) 2022-09-08T07:51:23.964 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:23.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:23.662763+0000) 2022-09-08T07:51:24.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:51:23.959+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:51:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:24.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:24.793769+0000) 2022-09-08T07:51:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:24.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:24.662932+0000) 2022-09-08T07:51:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:25 smithi055 ceph-mon[97033]: pgmap v19886: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-08T07:51:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:25 smithi059 ceph-mon[97450]: pgmap v19886: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-08T07:51:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:25.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:25.794061+0000) 2022-09-08T07:51:26.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:25.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:25.663084+0000) 2022-09-08T07:51:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:26.792+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:26.794282+0000) 2022-09-08T07:51:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:26.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:26.663249+0000) 2022-09-08T07:51:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:27 smithi055 ceph-mon[97033]: pgmap v19887: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:51:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:27 smithi059 ceph-mon[97450]: pgmap v19887: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:51:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:27.661+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:27.663374+0000) 2022-09-08T07:51:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:27.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:27.794500+0000) 2022-09-08T07:51:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:28.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:28.794727+0000) 2022-09-08T07:51:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:28.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:28.663596+0000) 2022-09-08T07:51:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:29 smithi055 ceph-mon[97033]: pgmap v19888: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:29 smithi059 ceph-mon[97450]: pgmap v19888: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:29.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:29.794861+0000) 2022-09-08T07:51:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:29.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:29.663791+0000) 2022-09-08T07:51:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:30.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:30.795012+0000) 2022-09-08T07:51:31.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:30.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:30.664009+0000) 2022-09-08T07:51:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:31 smithi055 ceph-mon[97033]: pgmap v19889: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:31 smithi059 ceph-mon[97450]: pgmap v19889: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:31.794+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:31.795229+0000) 2022-09-08T07:51:32.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:31.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:31.664186+0000) 2022-09-08T07:51:33.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:32.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:32.795437+0000) 2022-09-08T07:51:33.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:32.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:32.664334+0000) 2022-09-08T07:51:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:33 smithi055 ceph-mon[97033]: pgmap v19890: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:51:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:33 smithi059 ceph-mon[97450]: pgmap v19890: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:51:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:33.793+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:33.795618+0000) 2022-09-08T07:51:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:33.662+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:33.664531+0000) 2022-09-08T07:51:34.798 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:34.663+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:34.664794+0000) 2022-09-08T07:51:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:34.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:34.795870+0000) 2022-09-08T07:51:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:35 smithi055 ceph-mon[97033]: pgmap v19891: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:35 smithi059 ceph-mon[97450]: pgmap v19891: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:35.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:35.796038+0000) 2022-09-08T07:51:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:35.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:35.665028+0000) 2022-09-08T07:51:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:36.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:36.796255+0000) 2022-09-08T07:51:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:36.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:36.665203+0000) 2022-09-08T07:51:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:37 smithi055 ceph-mon[97033]: pgmap v19892: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:51:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:37 smithi059 ceph-mon[97450]: pgmap v19892: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:51:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:37.795+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:37.796472+0000) 2022-09-08T07:51:38.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:37.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:37.665411+0000) 2022-09-08T07:51:38.977 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:38.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:38.796764+0000) 2022-09-08T07:51:38.978 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:38.664+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:38.665554+0000) 2022-09-08T07:51:39.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:51:38.974+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:51:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:39 smithi055 ceph-mon[97033]: pgmap v19893: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:39 smithi059 ceph-mon[97450]: pgmap v19893: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:39.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:39.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:39.796943+0000) 2022-09-08T07:51:39.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:39.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:39.665714+0000) 2022-09-08T07:51:40.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:40 smithi055 ceph-mon[97033]: pgmap v19894: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:40.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:40 smithi055 ceph-mon[97033]: pgmap v19895: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:51:40.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:40 smithi055 ceph-mon[97033]: pgmap v19896: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:51:40.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:51:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:40 smithi059 ceph-mon[97450]: pgmap v19894: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:41.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:40 smithi059 ceph-mon[97450]: pgmap v19895: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T07:51:41.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:40 smithi059 ceph-mon[97450]: pgmap v19896: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:51:41.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:51:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:40.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:40.797165+0000) 2022-09-08T07:51:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:40.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:40.665927+0000) 2022-09-08T07:51:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:41.796+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:41.797389+0000) 2022-09-08T07:51:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:41.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:41.666194+0000) 2022-09-08T07:51:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:42 smithi055 ceph-mon[97033]: pgmap v19897: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:51:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:42 smithi059 ceph-mon[97450]: pgmap v19897: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:51:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:42.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:42.797580+0000) 2022-09-08T07:51:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:42.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:42.666407+0000) 2022-09-08T07:51:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:43.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:43.797805+0000) 2022-09-08T07:51:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:43.665+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:43.666532+0000) 2022-09-08T07:51:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:44.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:44.798039+0000) 2022-09-08T07:51:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:44.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:44.666736+0000) 2022-09-08T07:51:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:45 smithi055 ceph-mon[97033]: pgmap v19898: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:51:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:45 smithi059 ceph-mon[97450]: pgmap v19898: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T07:51:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:45.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:45.798221+0000) 2022-09-08T07:51:46.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:45.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:45.666974+0000) 2022-09-08T07:51:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:46.797+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:46.798456+0000) 2022-09-08T07:51:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:46.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:46.667195+0000) 2022-09-08T07:51:47.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:47 smithi055 ceph-mon[97033]: pgmap v19899: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:51:47.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:47 smithi059 ceph-mon[97450]: pgmap v19899: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:51:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:47.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:47.798680+0000) 2022-09-08T07:51:48.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:47.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:47.667351+0000) 2022-09-08T07:51:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:48.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:48.798898+0000) 2022-09-08T07:51:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:48.666+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:48.667587+0000) 2022-09-08T07:51:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:49 smithi055 ceph-mon[97033]: pgmap v19900: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:51:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:49 smithi059 ceph-mon[97450]: pgmap v19900: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T07:51:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:49.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:49.799072+0000) 2022-09-08T07:51:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:49.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:49.667853+0000) 2022-09-08T07:51:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:50.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:50.668043+0000) 2022-09-08T07:51:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:50.798+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:50.799345+0000) 2022-09-08T07:51:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:51 smithi055 ceph-mon[97033]: pgmap v19901: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:51:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:51 smithi059 ceph-mon[97450]: pgmap v19901: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:51:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:51.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:51.799624+0000) 2022-09-08T07:51:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:51.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:51.668265+0000) 2022-09-08T07:51:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:52.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:52.799827+0000) 2022-09-08T07:51:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:52.667+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:52.668490+0000) 2022-09-08T07:51:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:53 smithi055 ceph-mon[97033]: pgmap v19902: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:51:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:53 smithi059 ceph-mon[97450]: pgmap v19902: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:51:53.985 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:53.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:53.800038+0000) 2022-09-08T07:51:53.986 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:53.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:53.668650+0000) 2022-09-08T07:51:54.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:51:53.982+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:51:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:54.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:54.800214+0000) 2022-09-08T07:51:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:54.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:54.668812+0000) 2022-09-08T07:51:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:55 smithi055 ceph-mon[97033]: pgmap v19903: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:55 smithi059 ceph-mon[97450]: pgmap v19903: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:55.799+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:55.800443+0000) 2022-09-08T07:51:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:55.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:55.668979+0000) 2022-09-08T07:51:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:56.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:56.669110+0000) 2022-09-08T07:51:57.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:56.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:56.800681+0000) 2022-09-08T07:51:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:57 smithi055 ceph-mon[97033]: pgmap v19904: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:51:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:57 smithi059 ceph-mon[97450]: pgmap v19904: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:51:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:57.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:57.800911+0000) 2022-09-08T07:51:58.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:57.668+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:57.669325+0000) 2022-09-08T07:51:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:58.800+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:58.801152+0000) 2022-09-08T07:51:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:58.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:58.669498+0000) 2022-09-08T07:51:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:51:59 smithi055 ceph-mon[97033]: pgmap v19905: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:51:59 smithi059 ceph-mon[97450]: pgmap v19905: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:51:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:51:59.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:59.801394+0000) 2022-09-08T07:51:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:51:59.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:51:59.669671+0000) 2022-09-08T07:52:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:00.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:00.801544+0000) 2022-09-08T07:52:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:00.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:00.669855+0000) 2022-09-08T07:52:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:01 smithi055 ceph-mon[97033]: pgmap v19906: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:01 smithi059 ceph-mon[97450]: pgmap v19906: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:01.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:01.801774+0000) 2022-09-08T07:52:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:01.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:01.670127+0000) 2022-09-08T07:52:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:02.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:02.801982+0000) 2022-09-08T07:52:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:02.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:02.670345+0000) 2022-09-08T07:52:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:03 smithi055 ceph-mon[97033]: pgmap v19907: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:52:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:03 smithi059 ceph-mon[97450]: pgmap v19907: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:52:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:03.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:03.802252+0000) 2022-09-08T07:52:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:03.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:03.670509+0000) 2022-09-08T07:52:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:04.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:04.802493+0000) 2022-09-08T07:52:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:04.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:04.670760+0000) 2022-09-08T07:52:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:05 smithi055 ceph-mon[97033]: pgmap v19908: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:05 smithi059 ceph-mon[97450]: pgmap v19908: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:05.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:05.802738+0000) 2022-09-08T07:52:06.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:05.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:05.670998+0000) 2022-09-08T07:52:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:52:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:52:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:06.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:06.803014+0000) 2022-09-08T07:52:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:06.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:06.671177+0000) 2022-09-08T07:52:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:07 smithi055 ceph-mon[97033]: pgmap v19909: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:52:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:52:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:07 smithi059 ceph-mon[97450]: pgmap v19909: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:52:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:52:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:07.801+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:07.803201+0000) 2022-09-08T07:52:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:07.669+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:07.671391+0000) 2022-09-08T07:52:08.989 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:08.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:08.803504+0000) 2022-09-08T07:52:08.989 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:08.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:08.671613+0000) 2022-09-08T07:52:09.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:52:08.985+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:52:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:09 smithi055 ceph-mon[97033]: pgmap v19910: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:52:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:09 smithi059 ceph-mon[97450]: pgmap v19910: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:52:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:09.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:09.803758+0000) 2022-09-08T07:52:10.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:09.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:09.671766+0000) 2022-09-08T07:52:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:10 smithi055 ceph-mon[97033]: osdmap e1321: 12 total, 11 up, 11 in 2022-09-08T07:52:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:10 smithi055 ceph-mon[97033]: pgmap v19912: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:52:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:10 smithi059 ceph-mon[97450]: osdmap e1321: 12 total, 11 up, 11 in 2022-09-08T07:52:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:10 smithi059 ceph-mon[97450]: pgmap v19912: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:52:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:10.802+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:10.803919+0000) 2022-09-08T07:52:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:10.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:10.671919+0000) 2022-09-08T07:52:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:11.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:11.804178+0000) 2022-09-08T07:52:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:11.670+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:11.672156+0000) 2022-09-08T07:52:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:12 smithi055 ceph-mon[97033]: pgmap v19913: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:52:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:12 smithi059 ceph-mon[97450]: pgmap v19913: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:52:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:12.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:12.804420+0000) 2022-09-08T07:52:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:12.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:12.672404+0000) 2022-09-08T07:52:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:13.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:13.804663+0000) 2022-09-08T07:52:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:13.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:13.672580+0000) 2022-09-08T07:52:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:14 smithi055 ceph-mon[97033]: osdmap e1322: 12 total, 11 up, 11 in 2022-09-08T07:52:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:14 smithi059 ceph-mon[97450]: osdmap e1322: 12 total, 11 up, 11 in 2022-09-08T07:52:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:14.803+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:14.804808+0000) 2022-09-08T07:52:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:14.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:14.672775+0000) 2022-09-08T07:52:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:15 smithi055 ceph-mon[97033]: pgmap v19915: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:52:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:15 smithi059 ceph-mon[97450]: pgmap v19915: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:52:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:15.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:15.805043+0000) 2022-09-08T07:52:16.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:15.671+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:15.672893+0000) 2022-09-08T07:52:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:16.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:16.805231+0000) 2022-09-08T07:52:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:16.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:16.673107+0000) 2022-09-08T07:52:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:17 smithi055 ceph-mon[97033]: pgmap v19916: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:52:17.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:17 smithi059 ceph-mon[97450]: pgmap v19916: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:52:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:17.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:17.805402+0000) 2022-09-08T07:52:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:17.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:17.673289+0000) 2022-09-08T07:52:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:18.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:18.673486+0000) 2022-09-08T07:52:19.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:18.804+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:18.805627+0000) 2022-09-08T07:52:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:19 smithi055 ceph-mon[97033]: pgmap v19917: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-08T07:52:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:19 smithi059 ceph-mon[97450]: pgmap v19917: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 242 B/s wr, 7 op/s 2022-09-08T07:52:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:19.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:19.805879+0000) 2022-09-08T07:52:20.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:19.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:19.673649+0000) 2022-09-08T07:52:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:20.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:20.806100+0000) 2022-09-08T07:52:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:20.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:20.673906+0000) 2022-09-08T07:52:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:21 smithi055 ceph-mon[97033]: pgmap v19918: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:52:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:21 smithi059 ceph-mon[97450]: pgmap v19918: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:52:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:21.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:21.806322+0000) 2022-09-08T07:52:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:21.672+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:21.674074+0000) 2022-09-08T07:52:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:22.805+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:22.806527+0000) 2022-09-08T07:52:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:22.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:22.674241+0000) 2022-09-08T07:52:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:23 smithi055 ceph-mon[97033]: pgmap v19919: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:52:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:23 smithi059 ceph-mon[97450]: pgmap v19919: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:52:24.002 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:23.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:23.806699+0000) 2022-09-08T07:52:24.003 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:23.673+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:23.674377+0000) 2022-09-08T07:52:24.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:52:23.999+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:52:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:24.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:24.806831+0000) 2022-09-08T07:52:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:24.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:24.674601+0000) 2022-09-08T07:52:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:25 smithi055 ceph-mon[97033]: pgmap v19920: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2022-09-08T07:52:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:25 smithi059 ceph-mon[97450]: pgmap v19920: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2022-09-08T07:52:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:25.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:25.807056+0000) 2022-09-08T07:52:26.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:25.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:25.674821+0000) 2022-09-08T07:52:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:26.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:26.807285+0000) 2022-09-08T07:52:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:26.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:26.674990+0000) 2022-09-08T07:52:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:27 smithi055 ceph-mon[97033]: pgmap v19921: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:52:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:27 smithi059 ceph-mon[97450]: pgmap v19921: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:52:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:27.806+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:27.807532+0000) 2022-09-08T07:52:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:27.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:27.675214+0000) 2022-09-08T07:52:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:28.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:28.807766+0000) 2022-09-08T07:52:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:28.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:28.675426+0000) 2022-09-08T07:52:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:29 smithi055 ceph-mon[97033]: pgmap v19922: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:29 smithi059 ceph-mon[97450]: pgmap v19922: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:29.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:29.807973+0000) 2022-09-08T07:52:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:29.674+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:29.675612+0000) 2022-09-08T07:52:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:30.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:30.808112+0000) 2022-09-08T07:52:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:30.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:30.675851+0000) 2022-09-08T07:52:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:31 smithi055 ceph-mon[97033]: pgmap v19923: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:31 smithi059 ceph-mon[97450]: pgmap v19923: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:31.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:31.808324+0000) 2022-09-08T07:52:32.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:31.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:31.676068+0000) 2022-09-08T07:52:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:32.807+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:32.808529+0000) 2022-09-08T07:52:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:32.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:32.676240+0000) 2022-09-08T07:52:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:33 smithi055 ceph-mon[97033]: pgmap v19924: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:52:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:33 smithi059 ceph-mon[97450]: pgmap v19924: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:52:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:33.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:33.808763+0000) 2022-09-08T07:52:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:33.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:33.676380+0000) 2022-09-08T07:52:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:34.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:34.808924+0000) 2022-09-08T07:52:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:34.675+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:34.676615+0000) 2022-09-08T07:52:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:35 smithi055 ceph-mon[97033]: pgmap v19925: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:35 smithi059 ceph-mon[97450]: pgmap v19925: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:35.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:35.809109+0000) 2022-09-08T07:52:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:35.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:35.676727+0000) 2022-09-08T07:52:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:36.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:36.809357+0000) 2022-09-08T07:52:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:36.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:36.676874+0000) 2022-09-08T07:52:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:37 smithi055 ceph-mon[97033]: pgmap v19926: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:52:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:37 smithi059 ceph-mon[97450]: pgmap v19926: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:52:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:37.808+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:37.809497+0000) 2022-09-08T07:52:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:37.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:37.677045+0000) 2022-09-08T07:52:39.006 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:38.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:38.809745+0000) 2022-09-08T07:52:39.007 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:38.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:38.677211+0000) 2022-09-08T07:52:39.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:52:39.003+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:52:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:39 smithi055 ceph-mon[97033]: pgmap v19927: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:39 smithi059 ceph-mon[97450]: pgmap v19927: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:39.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:39.809935+0000) 2022-09-08T07:52:40.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:39.676+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:39.677374+0000) 2022-09-08T07:52:40.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:52:40.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:52:40.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:52:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:52:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:52:40.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:52:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:40.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:40.810153+0000) 2022-09-08T07:52:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:40.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:40.677606+0000) 2022-09-08T07:52:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:41 smithi055 ceph-mon[97033]: pgmap v19928: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:41 smithi059 ceph-mon[97450]: pgmap v19928: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:41.809+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:41.810343+0000) 2022-09-08T07:52:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:41.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:41.677857+0000) 2022-09-08T07:52:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:42.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:42.810548+0000) 2022-09-08T07:52:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:42.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:42.678047+0000) 2022-09-08T07:52:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:43 smithi055 ceph-mon[97033]: pgmap v19929: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:52:43.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:43 smithi059 ceph-mon[97450]: pgmap v19929: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:52:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:43.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:43.810748+0000) 2022-09-08T07:52:44.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:43.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:43.678177+0000) 2022-09-08T07:52:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:44.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:44.810916+0000) 2022-09-08T07:52:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:44.677+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:44.678359+0000) 2022-09-08T07:52:45.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:45 smithi055 ceph-mon[97033]: pgmap v19930: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:45 smithi059 ceph-mon[97450]: pgmap v19930: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:45.810+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:45.811131+0000) 2022-09-08T07:52:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:45.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:45.678511+0000) 2022-09-08T07:52:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:46.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:46.811403+0000) 2022-09-08T07:52:47.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:46.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:46.678687+0000) 2022-09-08T07:52:47.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:47 smithi055 ceph-mon[97033]: pgmap v19931: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:52:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:47 smithi059 ceph-mon[97450]: pgmap v19931: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:52:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:47.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:47.811667+0000) 2022-09-08T07:52:48.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:47.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:47.678837+0000) 2022-09-08T07:52:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:48.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:48.811895+0000) 2022-09-08T07:52:49.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:48.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:48.678989+0000) 2022-09-08T07:52:49.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:49 smithi055 ceph-mon[97033]: pgmap v19932: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:49 smithi059 ceph-mon[97450]: pgmap v19932: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:49.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:49.812041+0000) 2022-09-08T07:52:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:49.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:49.679133+0000) 2022-09-08T07:52:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:50.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:50.812226+0000) 2022-09-08T07:52:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:50.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:50.679340+0000) 2022-09-08T07:52:51.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:51 smithi055 ceph-mon[97033]: pgmap v19933: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:51 smithi059 ceph-mon[97450]: pgmap v19933: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:52:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:51.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:51.679533+0000) 2022-09-08T07:52:52.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:51.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:51.812441+0000) 2022-09-08T07:52:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:52.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:52.812686+0000) 2022-09-08T07:52:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:52.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:52.679691+0000) 2022-09-08T07:52:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:53 smithi055 ceph-mon[97033]: pgmap v19934: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:52:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:53 smithi059 ceph-mon[97450]: pgmap v19934: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:52:54.022 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:53.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:53.812909+0000) 2022-09-08T07:52:54.022 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:53.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:53.679831+0000) 2022-09-08T07:52:54.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:52:54.017+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:52:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:54.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:54.680009+0000) 2022-09-08T07:52:55.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:54.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:54.813092+0000) 2022-09-08T07:52:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:55 smithi055 ceph-mon[97033]: pgmap v19935: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:52:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:55 smithi059 ceph-mon[97450]: pgmap v19935: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:52:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:55.811+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:55.813298+0000) 2022-09-08T07:52:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:55.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:55.680180+0000) 2022-09-08T07:52:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:56 smithi055 ceph-mon[97033]: pgmap v19936: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:52:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:56 smithi059 ceph-mon[97450]: pgmap v19936: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:52:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:56.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:56.813440+0000) 2022-09-08T07:52:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:56.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:56.680310+0000) 2022-09-08T07:52:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:57.678+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:57.680439+0000) 2022-09-08T07:52:58.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:57.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:57.813708+0000) 2022-09-08T07:52:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:58.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:58.813841+0000) 2022-09-08T07:52:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:58.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:58.680565+0000) 2022-09-08T07:52:59.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:52:59 smithi055 ceph-mon[97033]: pgmap v19937: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:52:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:52:59 smithi059 ceph-mon[97450]: pgmap v19937: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:53:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:52:59.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:59.813969+0000) 2022-09-08T07:53:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:52:59.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:52:59.680786+0000) 2022-09-08T07:53:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:00.812+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:00.814185+0000) 2022-09-08T07:53:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:00.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:00.680922+0000) 2022-09-08T07:53:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:01 smithi055 ceph-mon[97033]: pgmap v19938: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:53:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:01 smithi059 ceph-mon[97450]: pgmap v19938: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:53:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:01.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:01.814437+0000) 2022-09-08T07:53:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:01.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:01.681078+0000) 2022-09-08T07:53:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:02.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:02.814692+0000) 2022-09-08T07:53:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:02.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:02.681232+0000) 2022-09-08T07:53:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:03 smithi055 ceph-mon[97033]: pgmap v19939: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:53:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:03 smithi059 ceph-mon[97450]: pgmap v19939: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:53:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:03.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:03.681351+0000) 2022-09-08T07:53:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:03.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:03.814899+0000) 2022-09-08T07:53:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:04.679+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:04.681507+0000) 2022-09-08T07:53:05.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:04.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:04.815040+0000) 2022-09-08T07:53:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:05 smithi055 ceph-mon[97033]: pgmap v19940: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:05 smithi059 ceph-mon[97450]: pgmap v19940: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:05.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:05.815200+0000) 2022-09-08T07:53:06.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:05.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:05.681719+0000) 2022-09-08T07:53:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:53:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:53:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:06.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:06.815349+0000) 2022-09-08T07:53:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:06.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:06.681859+0000) 2022-09-08T07:53:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:07 smithi055 ceph-mon[97033]: pgmap v19941: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:53:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:53:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:07 smithi059 ceph-mon[97450]: pgmap v19941: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:53:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:53:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:07.813+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:07.815508+0000) 2022-09-08T07:53:08.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:07.680+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:07.681987+0000) 2022-09-08T07:53:09.028 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:08.814+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:08.815756+0000) 2022-09-08T07:53:09.029 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:08.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:08.682112+0000) 2022-09-08T07:53:09.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:53:09.024+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:53:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:09 smithi055 ceph-mon[97033]: pgmap v19942: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:53:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:09 smithi059 ceph-mon[97450]: pgmap v19942: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:53:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:09.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:09.815955+0000) 2022-09-08T07:53:10.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:09.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:09.682255+0000) 2022-09-08T07:53:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:10 smithi059 ceph-mon[97450]: osdmap e1323: 12 total, 11 up, 11 in 2022-09-08T07:53:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:10 smithi059 ceph-mon[97450]: pgmap v19944: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:53:10.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:10 smithi059 ceph-mon[97450]: pgmap v19945: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T07:53:10.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:10 smithi059 ceph-mon[97450]: pgmap v19946: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T07:53:10.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:53:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:10 smithi055 ceph-mon[97033]: osdmap e1323: 12 total, 11 up, 11 in 2022-09-08T07:53:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:10 smithi055 ceph-mon[97033]: pgmap v19944: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:53:10.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:10 smithi055 ceph-mon[97033]: pgmap v19945: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T07:53:10.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:10 smithi055 ceph-mon[97033]: pgmap v19946: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 22 KiB/s rd, 85 KiB/s wr, 38 op/s 2022-09-08T07:53:10.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:53:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:10.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:10.816171+0000) 2022-09-08T07:53:11.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:10.681+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:10.682463+0000) 2022-09-08T07:53:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:11.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:11.816347+0000) 2022-09-08T07:53:12.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:11.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:11.682735+0000) 2022-09-08T07:53:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:12.815+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:12.816513+0000) 2022-09-08T07:53:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:12.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:12.682985+0000) 2022-09-08T07:53:13.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:12 smithi055 ceph-mon[97033]: pgmap v19947: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T07:53:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:12 smithi059 ceph-mon[97450]: pgmap v19947: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T07:53:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:13.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:13.816752+0000) 2022-09-08T07:53:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:13.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:13.683149+0000) 2022-09-08T07:53:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:14 smithi055 ceph-mon[97033]: osdmap e1324: 12 total, 11 up, 11 in 2022-09-08T07:53:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:14 smithi059 ceph-mon[97450]: osdmap e1324: 12 total, 11 up, 11 in 2022-09-08T07:53:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:14.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:14.816996+0000) 2022-09-08T07:53:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:14.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:14.683289+0000) 2022-09-08T07:53:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:15 smithi055 ceph-mon[97033]: pgmap v19949: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 217 B/s wr, 6 op/s 2022-09-08T07:53:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:15 smithi059 ceph-mon[97450]: pgmap v19949: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 217 B/s wr, 6 op/s 2022-09-08T07:53:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:15.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:15.817227+0000) 2022-09-08T07:53:16.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:15.682+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:15.683459+0000) 2022-09-08T07:53:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:16.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:16.817395+0000) 2022-09-08T07:53:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:16.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:16.683732+0000) 2022-09-08T07:53:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:17 smithi055 ceph-mon[97033]: pgmap v19950: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 5.1 KiB/s rd, 326 B/s wr, 10 op/s 2022-09-08T07:53:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:17 smithi059 ceph-mon[97450]: pgmap v19950: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 5.1 KiB/s rd, 326 B/s wr, 10 op/s 2022-09-08T07:53:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:17.816+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:17.817571+0000) 2022-09-08T07:53:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:17.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:17.683930+0000) 2022-09-08T07:53:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:18.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:18.817791+0000) 2022-09-08T07:53:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:18.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:18.684098+0000) 2022-09-08T07:53:19.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:19 smithi055 ceph-mon[97033]: pgmap v19951: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:53:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:19 smithi059 ceph-mon[97450]: pgmap v19951: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:53:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:19.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:19.817950+0000) 2022-09-08T07:53:20.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:19.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:19.684289+0000) 2022-09-08T07:53:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:20 smithi055 ceph-mon[97033]: pgmap v19952: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:53:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:20 smithi059 ceph-mon[97450]: pgmap v19952: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:53:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:20.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:20.818180+0000) 2022-09-08T07:53:21.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:20.683+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:20.684467+0000) 2022-09-08T07:53:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:21.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:21.684762+0000) 2022-09-08T07:53:22.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:21.817+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:21.818460+0000) 2022-09-08T07:53:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:22.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:22.818712+0000) 2022-09-08T07:53:23.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:22.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:22.684936+0000) 2022-09-08T07:53:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:22 smithi055 ceph-mon[97033]: pgmap v19953: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:53:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:22 smithi059 ceph-mon[97450]: pgmap v19953: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:53:24.031 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:23.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:23.818920+0000) 2022-09-08T07:53:24.032 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:23.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:23.685180+0000) 2022-09-08T07:53:24.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:53:24.027+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:53:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:24.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:24.819102+0000) 2022-09-08T07:53:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:24.684+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:24.685417+0000) 2022-09-08T07:53:25.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:25 smithi055 ceph-mon[97033]: pgmap v19954: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 95 B/s wr, 5 op/s 2022-09-08T07:53:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:25 smithi059 ceph-mon[97450]: pgmap v19954: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 95 B/s wr, 5 op/s 2022-09-08T07:53:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:25.818+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:25.819322+0000) 2022-09-08T07:53:26.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:25.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:25.685574+0000) 2022-09-08T07:53:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:26.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:26.819569+0000) 2022-09-08T07:53:27.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:26.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:26.685825+0000) 2022-09-08T07:53:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:27 smithi055 ceph-mon[97033]: pgmap v19955: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:53:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:27 smithi059 ceph-mon[97450]: pgmap v19955: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:53:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:27.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:27.819795+0000) 2022-09-08T07:53:28.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:27.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:27.686091+0000) 2022-09-08T07:53:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:28.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:28.819917+0000) 2022-09-08T07:53:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:28.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:28.686223+0000) 2022-09-08T07:53:29.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:29 smithi055 ceph-mon[97033]: pgmap v19956: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:53:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:29 smithi059 ceph-mon[97450]: pgmap v19956: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:53:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:29.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:29.820116+0000) 2022-09-08T07:53:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:29.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:29.686388+0000) 2022-09-08T07:53:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:30 smithi055 ceph-mon[97033]: pgmap v19957: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T07:53:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:30 smithi059 ceph-mon[97450]: pgmap v19957: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T07:53:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:30.685+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:30.686554+0000) 2022-09-08T07:53:31.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:30.819+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:30.820371+0000) 2022-09-08T07:53:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:31.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:31.820601+0000) 2022-09-08T07:53:32.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:31.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:31.686777+0000) 2022-09-08T07:53:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:32.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:32.820800+0000) 2022-09-08T07:53:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:32.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:32.686965+0000) 2022-09-08T07:53:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:32 smithi055 ceph-mon[97033]: pgmap v19958: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:32 smithi059 ceph-mon[97450]: pgmap v19958: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:33.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:33.820914+0000) 2022-09-08T07:53:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:33.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:33.687110+0000) 2022-09-08T07:53:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:34.820+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:34.821064+0000) 2022-09-08T07:53:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:34.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:34.687322+0000) 2022-09-08T07:53:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:35 smithi055 ceph-mon[97033]: pgmap v19959: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:35 smithi059 ceph-mon[97450]: pgmap v19959: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:35.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:35.821278+0000) 2022-09-08T07:53:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:35.686+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:35.687504+0000) 2022-09-08T07:53:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:36.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:36.821484+0000) 2022-09-08T07:53:37.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:36.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:36.687760+0000) 2022-09-08T07:53:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:37 smithi055 ceph-mon[97033]: pgmap v19960: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:53:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:37 smithi059 ceph-mon[97450]: pgmap v19960: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:53:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:37.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:37.687946+0000) 2022-09-08T07:53:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:37.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:37.821763+0000) 2022-09-08T07:53:39.040 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:38.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:38.821938+0000) 2022-09-08T07:53:39.041 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:38.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:38.688107+0000) 2022-09-08T07:53:39.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:53:39.037+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:53:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:39 smithi055 ceph-mon[97033]: pgmap v19961: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:39 smithi059 ceph-mon[97450]: pgmap v19961: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:39.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:39.822176+0000) 2022-09-08T07:53:40.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:39.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:39.688287+0000) 2022-09-08T07:53:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:40 smithi055 ceph-mon[97033]: pgmap v19962: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:53:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:40 smithi059 ceph-mon[97450]: pgmap v19962: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:53:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:40.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:40.822404+0000) 2022-09-08T07:53:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:40.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:40.688466+0000) 2022-09-08T07:53:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:41.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:41.822699+0000) 2022-09-08T07:53:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:41.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:41.688700+0000) 2022-09-08T07:53:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:42.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:42.688871+0000) 2022-09-08T07:53:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:42.821+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:42.822955+0000) 2022-09-08T07:53:43.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:42 smithi055 ceph-mon[97033]: pgmap v19963: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:42 smithi059 ceph-mon[97450]: pgmap v19963: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:43.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:43.823186+0000) 2022-09-08T07:53:44.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:43.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:43.689068+0000) 2022-09-08T07:53:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:44.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:44.823397+0000) 2022-09-08T07:53:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:44.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:44.689244+0000) 2022-09-08T07:53:45.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:45 smithi055 ceph-mon[97033]: pgmap v19964: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:45 smithi059 ceph-mon[97450]: pgmap v19964: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:45.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:45.823668+0000) 2022-09-08T07:53:46.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:45.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:45.689401+0000) 2022-09-08T07:53:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:46.822+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:46.823923+0000) 2022-09-08T07:53:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:46.687+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:46.689560+0000) 2022-09-08T07:53:47.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:47 smithi055 ceph-mon[97033]: pgmap v19965: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:53:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:47 smithi059 ceph-mon[97450]: pgmap v19965: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:53:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:47.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:47.689722+0000) 2022-09-08T07:53:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:47.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:47.824049+0000) 2022-09-08T07:53:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:48.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:48.824251+0000) 2022-09-08T07:53:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:48.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:48.689927+0000) 2022-09-08T07:53:49.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:49 smithi055 ceph-mon[97033]: pgmap v19966: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:49 smithi059 ceph-mon[97450]: pgmap v19966: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:49.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:49.824489+0000) 2022-09-08T07:53:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:49.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:49.690129+0000) 2022-09-08T07:53:50.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:50 smithi055 ceph-mon[97033]: pgmap v19967: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:53:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:50 smithi059 ceph-mon[97450]: pgmap v19967: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:53:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:50.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:50.824719+0000) 2022-09-08T07:53:51.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:50.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:50.690260+0000) 2022-09-08T07:53:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:51.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:51.824945+0000) 2022-09-08T07:53:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:51.688+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:51.690487+0000) 2022-09-08T07:53:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:52.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:52.690741+0000) 2022-09-08T07:53:53.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:52.823+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:52.825125+0000) 2022-09-08T07:53:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:52 smithi055 ceph-mon[97033]: pgmap v19968: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:52 smithi059 ceph-mon[97450]: pgmap v19968: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:54.043 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:53.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:53.825319+0000) 2022-09-08T07:53:54.044 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:53.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:53.690986+0000) 2022-09-08T07:53:54.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:53:54.039+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:53:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:54.824+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:54.825555+0000) 2022-09-08T07:53:55.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:54.689+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:54.691215+0000) 2022-09-08T07:53:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:55 smithi055 ceph-mon[97033]: pgmap v19969: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:55 smithi059 ceph-mon[97450]: pgmap v19969: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:55.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:55.825760+0000) 2022-09-08T07:53:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:55.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:55.691370+0000) 2022-09-08T07:53:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:56 smithi055 ceph-mon[97033]: pgmap v19970: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:53:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:56 smithi059 ceph-mon[97450]: pgmap v19970: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:53:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:56.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:56.825907+0000) 2022-09-08T07:53:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:56.690+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:56.691531+0000) 2022-09-08T07:53:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:57.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:57.826073+0000) 2022-09-08T07:53:58.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:57.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:57.691697+0000) 2022-09-08T07:53:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:58.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:58.826260+0000) 2022-09-08T07:53:59.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:58.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:58.691832+0000) 2022-09-08T07:53:59.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:53:59 smithi055 ceph-mon[97033]: pgmap v19971: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:53:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:53:59 smithi059 ceph-mon[97450]: pgmap v19971: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:53:59.825+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:59.826499+0000) 2022-09-08T07:54:00.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:53:59.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:53:59.691960+0000) 2022-09-08T07:54:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:00.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:00.826743+0000) 2022-09-08T07:54:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:00.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:00.692094+0000) 2022-09-08T07:54:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:01 smithi055 ceph-mon[97033]: pgmap v19972: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:54:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:01 smithi059 ceph-mon[97450]: pgmap v19972: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:54:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:01.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:01.826942+0000) 2022-09-08T07:54:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:01.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:01.692285+0000) 2022-09-08T07:54:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:02.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:02.827099+0000) 2022-09-08T07:54:03.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:02.691+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:02.692530+0000) 2022-09-08T07:54:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:03 smithi055 ceph-mon[97033]: pgmap v19973: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:03 smithi059 ceph-mon[97450]: pgmap v19973: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:03.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:03.827323+0000) 2022-09-08T07:54:04.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:03.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:03.692778+0000) 2022-09-08T07:54:04.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:04 smithi055 ceph-mon[97033]: pgmap v19974: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:04 smithi059 ceph-mon[97450]: pgmap v19974: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:04.826+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:04.827572+0000) 2022-09-08T07:54:05.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:04.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:04.692995+0000) 2022-09-08T07:54:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:05.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:05.827750+0000) 2022-09-08T07:54:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:05.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:05.693143+0000) 2022-09-08T07:54:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:06.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:06.827967+0000) 2022-09-08T07:54:07.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:06.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:06.693359+0000) 2022-09-08T07:54:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:06 smithi055 ceph-mon[97033]: pgmap v19975: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:54:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:54:07.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:54:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:06 smithi059 ceph-mon[97450]: pgmap v19975: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:54:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:54:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:54:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:07.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:07.828202+0000) 2022-09-08T07:54:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:07.692+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:07.693595+0000) 2022-09-08T07:54:09.057 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:08.827+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:08.828449+0000) 2022-09-08T07:54:09.058 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:08.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:08.693795+0000) 2022-09-08T07:54:09.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:54:09.054+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:54:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:09 smithi055 ceph-mon[97033]: pgmap v19976: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:54:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:09 smithi059 ceph-mon[97450]: pgmap v19976: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:54:10.030 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:09.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:09.828665+0000) 2022-09-08T07:54:10.031 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:09.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:09.693945+0000) 2022-09-08T07:54:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:10 smithi055 ceph-mon[97033]: osdmap e1325: 12 total, 11 up, 11 in 2022-09-08T07:54:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:10 smithi055 ceph-mon[97033]: pgmap v19978: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:54:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:54:10.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:54:10.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:54:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:10 smithi059 ceph-mon[97450]: osdmap e1325: 12 total, 11 up, 11 in 2022-09-08T07:54:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:10 smithi059 ceph-mon[97450]: pgmap v19978: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:54:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:54:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:54:10.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:54:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:10.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:10.828816+0000) 2022-09-08T07:54:11.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:10.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:10.694092+0000) 2022-09-08T07:54:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:11.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:11.829024+0000) 2022-09-08T07:54:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:11.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:11.694324+0000) 2022-09-08T07:54:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:12.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:12.829263+0000) 2022-09-08T07:54:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:12.693+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:12.694548+0000) 2022-09-08T07:54:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:12 smithi055 ceph-mon[97033]: pgmap v19979: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:54:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:12 smithi059 ceph-mon[97450]: pgmap v19979: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:54:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:13.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:13.694790+0000) 2022-09-08T07:54:14.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:13.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:13.829441+0000) 2022-09-08T07:54:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:14 smithi055 ceph-mon[97033]: osdmap e1326: 12 total, 11 up, 11 in 2022-09-08T07:54:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:14 smithi059 ceph-mon[97450]: osdmap e1326: 12 total, 11 up, 11 in 2022-09-08T07:54:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:14.828+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:14.829574+0000) 2022-09-08T07:54:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:14.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:14.694959+0000) 2022-09-08T07:54:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:15 smithi055 ceph-mon[97033]: pgmap v19981: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:54:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:15 smithi059 ceph-mon[97450]: pgmap v19981: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:54:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:15.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:15.829761+0000) 2022-09-08T07:54:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:15.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:15.695092+0000) 2022-09-08T07:54:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:16.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:16.829892+0000) 2022-09-08T07:54:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:16.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:16.695247+0000) 2022-09-08T07:54:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:17 smithi055 ceph-mon[97033]: pgmap v19982: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:54:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:17 smithi059 ceph-mon[97450]: pgmap v19982: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:54:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:17.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:17.830039+0000) 2022-09-08T07:54:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:17.694+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:17.695487+0000) 2022-09-08T07:54:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:18.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:18.830231+0000) 2022-09-08T07:54:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:18.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:18.695742+0000) 2022-09-08T07:54:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:19 smithi055 ceph-mon[97033]: pgmap v19983: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-08T07:54:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:19 smithi059 ceph-mon[97450]: pgmap v19983: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2022-09-08T07:54:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:19.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:19.830364+0000) 2022-09-08T07:54:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:19.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:19.695880+0000) 2022-09-08T07:54:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:20.829+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:20.830571+0000) 2022-09-08T07:54:21.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:20.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:20.696090+0000) 2022-09-08T07:54:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:21 smithi055 ceph-mon[97033]: pgmap v19984: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:54:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:21 smithi059 ceph-mon[97450]: pgmap v19984: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:54:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:21.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:21.830711+0000) 2022-09-08T07:54:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:21.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:21.696246+0000) 2022-09-08T07:54:22.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:22 smithi055 ceph-mon[97033]: pgmap v19985: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:54:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:22 smithi059 ceph-mon[97450]: pgmap v19985: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:54:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:22.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:22.830848+0000) 2022-09-08T07:54:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:22.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:22.696458+0000) 2022-09-08T07:54:24.069 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:23.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:23.696668+0000) 2022-09-08T07:54:24.070 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:23.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:23.831027+0000) 2022-09-08T07:54:24.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:54:24.066+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:54:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:24.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:24.831162+0000) 2022-09-08T07:54:25.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:24.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:24.696811+0000) 2022-09-08T07:54:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:25 smithi055 ceph-mon[97033]: pgmap v19986: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 95 B/s wr, 5 op/s 2022-09-08T07:54:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:25 smithi059 ceph-mon[97450]: pgmap v19986: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 95 B/s wr, 5 op/s 2022-09-08T07:54:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:25.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:25.831335+0000) 2022-09-08T07:54:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:25.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:25.696960+0000) 2022-09-08T07:54:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:26 smithi055 ceph-mon[97033]: pgmap v19987: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:54:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:26 smithi059 ceph-mon[97450]: pgmap v19987: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:54:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:26.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:26.831538+0000) 2022-09-08T07:54:27.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:26.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:26.697154+0000) 2022-09-08T07:54:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:27.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:27.831751+0000) 2022-09-08T07:54:28.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:27.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:27.697371+0000) 2022-09-08T07:54:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:28.695+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:28.697595+0000) 2022-09-08T07:54:29.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:28.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:28.831910+0000) 2022-09-08T07:54:29.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:29 smithi055 ceph-mon[97033]: pgmap v19988: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:54:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:29 smithi059 ceph-mon[97450]: pgmap v19988: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T07:54:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:29.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:29.832088+0000) 2022-09-08T07:54:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:29.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:29.697834+0000) 2022-09-08T07:54:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:30.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:30.832293+0000) 2022-09-08T07:54:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:30.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:30.697976+0000) 2022-09-08T07:54:31.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:31 smithi055 ceph-mon[97033]: pgmap v19989: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T07:54:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:31 smithi059 ceph-mon[97450]: pgmap v19989: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T07:54:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:31.830+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:31.832555+0000) 2022-09-08T07:54:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:31.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:31.698130+0000) 2022-09-08T07:54:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:32 smithi055 ceph-mon[97033]: pgmap v19990: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:32 smithi059 ceph-mon[97450]: pgmap v19990: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:32.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:32.832812+0000) 2022-09-08T07:54:33.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:32.696+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:32.698255+0000) 2022-09-08T07:54:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:33.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:33.832980+0000) 2022-09-08T07:54:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:33.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:33.698448+0000) 2022-09-08T07:54:35.098 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:34.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:34.833236+0000) 2022-09-08T07:54:35.099 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:34.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:34.698629+0000) 2022-09-08T07:54:35.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:35 smithi055 ceph-mon[97033]: pgmap v19991: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:35 smithi059 ceph-mon[97450]: pgmap v19991: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:35.831+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:35.833513+0000) 2022-09-08T07:54:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:35.697+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:35.698783+0000) 2022-09-08T07:54:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:36 smithi055 ceph-mon[97033]: pgmap v19992: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:54:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:36 smithi059 ceph-mon[97450]: pgmap v19992: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:54:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:36.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:36.833766+0000) 2022-09-08T07:54:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:36.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:36.699002+0000) 2022-09-08T07:54:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:37.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:37.833916+0000) 2022-09-08T07:54:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:37.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:37.699240+0000) 2022-09-08T07:54:39.080 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:38.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:38.834090+0000) 2022-09-08T07:54:39.081 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:38.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:38.699506+0000) 2022-09-08T07:54:39.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:54:39.076+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:54:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:39 smithi055 ceph-mon[97033]: pgmap v19993: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:39 smithi059 ceph-mon[97450]: pgmap v19993: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:39.832+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:39.834263+0000) 2022-09-08T07:54:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:39.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:39.699686+0000) 2022-09-08T07:54:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:40 smithi055 ceph-mon[97033]: pgmap v19994: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:54:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:40 smithi059 ceph-mon[97450]: pgmap v19994: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:54:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:40.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:40.834473+0000) 2022-09-08T07:54:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:40.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:40.699841+0000) 2022-09-08T07:54:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:41.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:41.834720+0000) 2022-09-08T07:54:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:41.698+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:41.700016+0000) 2022-09-08T07:54:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:42.833+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:42.834922+0000) 2022-09-08T07:54:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:42.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:42.700161+0000) 2022-09-08T07:54:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:42 smithi055 ceph-mon[97033]: pgmap v19995: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:42 smithi059 ceph-mon[97450]: pgmap v19995: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:43.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:43.835062+0000) 2022-09-08T07:54:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:43.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:43.700327+0000) 2022-09-08T07:54:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:44.699+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:44.700484+0000) 2022-09-08T07:54:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:44.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:44.835263+0000) 2022-09-08T07:54:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:45 smithi055 ceph-mon[97033]: pgmap v19996: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:45.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:54:45.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:54:45.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:54:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:45 smithi059 ceph-mon[97450]: pgmap v19996: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:45.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:54:45.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:54:45.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:54:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:45.834+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:45.835485+0000) 2022-09-08T07:54:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:45.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:45.700743+0000) 2022-09-08T07:54:46.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:46 smithi055 ceph-mon[97033]: pgmap v19997: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:54:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:46 smithi059 ceph-mon[97450]: pgmap v19997: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:54:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:46.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:46.835709+0000) 2022-09-08T07:54:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:46.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:46.700944+0000) 2022-09-08T07:54:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:47 smithi055 ceph-mon[97033]: pgmap v19998: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:54:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:47 smithi059 ceph-mon[97450]: pgmap v19998: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:54:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:47.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:47.835943+0000) 2022-09-08T07:54:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:47.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:47.701108+0000) 2022-09-08T07:54:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:48.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:48.836175+0000) 2022-09-08T07:54:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:48.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:48.701310+0000) 2022-09-08T07:54:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:49 smithi055 ceph-mon[97033]: pgmap v19999: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:54:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:49 smithi059 ceph-mon[97450]: pgmap v19999: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:54:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:49.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:49.836353+0000) 2022-09-08T07:54:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:49.700+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:49.701473+0000) 2022-09-08T07:54:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:50.835+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:50.836567+0000) 2022-09-08T07:54:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:50.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:50.701676+0000) 2022-09-08T07:54:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:51.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:51.836825+0000) 2022-09-08T07:54:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:51.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:51.701867+0000) 2022-09-08T07:54:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:51 smithi055 ceph-mon[97033]: pgmap v20000: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:54:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:51 smithi059 ceph-mon[97450]: pgmap v20000: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:54:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:52.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:52.836980+0000) 2022-09-08T07:54:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:52.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:52.702055+0000) 2022-09-08T07:54:54.084 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:53.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:53.837208+0000) 2022-09-08T07:54:54.085 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:53.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:53.702221+0000) 2022-09-08T07:54:54.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:54:54.081+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:54:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:54 smithi055 ceph-mon[97033]: pgmap v20001: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:54:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:54 smithi059 ceph-mon[97450]: pgmap v20001: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:54:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:54.836+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:54.837428+0000) 2022-09-08T07:54:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:54.701+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:54.702448+0000) 2022-09-08T07:54:55.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:55 smithi055 ceph-mon[97033]: pgmap v20002: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:54:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:55 smithi059 ceph-mon[97450]: pgmap v20002: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:54:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:55.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:55.837679+0000) 2022-09-08T07:54:56.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:55.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:55.702703+0000) 2022-09-08T07:54:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:56.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:56.702881+0000) 2022-09-08T07:54:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:56.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:56.837932+0000) 2022-09-08T07:54:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:57.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:57.838171+0000) 2022-09-08T07:54:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:57.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:57.703051+0000) 2022-09-08T07:54:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:57 smithi055 ceph-mon[97033]: pgmap v20003: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:54:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:57 smithi059 ceph-mon[97450]: pgmap v20003: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:54:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:58.837+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:58.838392+0000) 2022-09-08T07:54:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:58.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:58.703253+0000) 2022-09-08T07:54:59.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:54:59 smithi055 ceph-mon[97033]: pgmap v20004: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:54:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:54:59 smithi059 ceph-mon[97450]: pgmap v20004: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:55:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:54:59.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:59.838611+0000) 2022-09-08T07:55:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:54:59.702+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:54:59.703432+0000) 2022-09-08T07:55:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:00.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:00.838798+0000) 2022-09-08T07:55:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:00.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:00.703707+0000) 2022-09-08T07:55:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:01.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:01.838987+0000) 2022-09-08T07:55:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:01.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:01.703880+0000) 2022-09-08T07:55:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:01 smithi055 ceph-mon[97033]: pgmap v20005: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:55:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:01 smithi059 ceph-mon[97450]: pgmap v20005: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:55:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:02.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:02.839146+0000) 2022-09-08T07:55:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:02.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:02.704002+0000) 2022-09-08T07:55:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:03.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:03.839290+0000) 2022-09-08T07:55:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:03.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:03.704196+0000) 2022-09-08T07:55:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:03 smithi055 ceph-mon[97033]: pgmap v20006: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:55:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:03 smithi059 ceph-mon[97450]: pgmap v20006: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:55:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:04.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:04.839483+0000) 2022-09-08T07:55:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:04.703+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:04.704412+0000) 2022-09-08T07:55:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:05.838+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:05.839619+0000) 2022-09-08T07:55:06.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:05.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:05.704674+0000) 2022-09-08T07:55:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:05 smithi055 ceph-mon[97033]: pgmap v20007: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:55:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:05 smithi059 ceph-mon[97450]: pgmap v20007: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:55:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:06.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:06.839844+0000) 2022-09-08T07:55:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:06.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:06.704823+0000) 2022-09-08T07:55:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:55:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:55:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:07.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:07.840016+0000) 2022-09-08T07:55:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:07.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:07.704957+0000) 2022-09-08T07:55:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:07 smithi055 ceph-mon[97033]: pgmap v20008: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:55:08.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:55:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:07 smithi059 ceph-mon[97450]: pgmap v20008: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:55:08.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:55:09.099 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:08.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:08.840192+0000) 2022-09-08T07:55:09.100 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:08.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:08.705163+0000) 2022-09-08T07:55:09.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:55:09.096+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:55:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:09 smithi055 ceph-mon[97033]: pgmap v20009: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:55:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:09 smithi059 ceph-mon[97450]: pgmap v20009: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:55:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:09.839+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:09.840551+0000) 2022-09-08T07:55:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:09.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:09.705337+0000) 2022-09-08T07:55:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:10 smithi055 ceph-mon[97033]: osdmap e1327: 12 total, 11 up, 11 in 2022-09-08T07:55:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:10 smithi059 ceph-mon[97450]: osdmap e1327: 12 total, 11 up, 11 in 2022-09-08T07:55:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:10.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:10.840759+0000) 2022-09-08T07:55:11.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:10.704+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:10.705478+0000) 2022-09-08T07:55:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:11 smithi055 ceph-mon[97033]: pgmap v20011: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:55:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:11 smithi059 ceph-mon[97450]: pgmap v20011: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:55:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:11.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:11.840919+0000) 2022-09-08T07:55:12.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:11.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:11.705699+0000) 2022-09-08T07:55:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:12.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:12.705844+0000) 2022-09-08T07:55:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:12.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:12.841100+0000) 2022-09-08T07:55:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:13.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:13.841280+0000) 2022-09-08T07:55:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:13.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:13.706061+0000) 2022-09-08T07:55:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:13 smithi055 ceph-mon[97033]: pgmap v20012: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:55:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:13 smithi059 ceph-mon[97450]: pgmap v20012: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:55:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:14.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:14.841424+0000) 2022-09-08T07:55:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:14.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:14.706220+0000) 2022-09-08T07:55:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:15.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:15.841598+0000) 2022-09-08T07:55:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:15.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:15.706365+0000) 2022-09-08T07:55:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:15 smithi055 ceph-mon[97033]: pgmap v20013: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:55:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:15 smithi059 ceph-mon[97450]: pgmap v20013: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T07:55:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:16.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:16.841806+0000) 2022-09-08T07:55:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:16.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:16.706521+0000) 2022-09-08T07:55:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:17.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:17.842010+0000) 2022-09-08T07:55:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:17.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:17.706684+0000) 2022-09-08T07:55:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:17 smithi055 ceph-mon[97033]: pgmap v20014: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:55:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:17 smithi059 ceph-mon[97450]: pgmap v20014: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:55:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:18.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:18.706822+0000) 2022-09-08T07:55:19.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:18.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:18.842203+0000) 2022-09-08T07:55:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:19 smithi055 ceph-mon[97033]: osdmap e1328: 12 total, 11 up, 11 in 2022-09-08T07:55:19.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:19 smithi055 ceph-mon[97033]: pgmap v20016: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2022-09-08T07:55:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:19 smithi059 ceph-mon[97450]: osdmap e1328: 12 total, 11 up, 11 in 2022-09-08T07:55:19.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:19 smithi059 ceph-mon[97450]: pgmap v20016: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2022-09-08T07:55:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:19.840+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:19.842432+0000) 2022-09-08T07:55:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:19.705+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:19.707042+0000) 2022-09-08T07:55:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:20.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:20.842674+0000) 2022-09-08T07:55:21.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:20.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:20.707323+0000) 2022-09-08T07:55:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:21.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:21.842916+0000) 2022-09-08T07:55:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:21.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:21.707579+0000) 2022-09-08T07:55:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:21 smithi055 ceph-mon[97033]: pgmap v20017: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:55:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:21 smithi059 ceph-mon[97450]: pgmap v20017: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:55:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:22.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:22.843140+0000) 2022-09-08T07:55:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:22.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:22.707754+0000) 2022-09-08T07:55:24.110 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:23.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:23.843344+0000) 2022-09-08T07:55:24.111 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:23.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:23.707980+0000) 2022-09-08T07:55:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:23 smithi055 ceph-mon[97033]: pgmap v20018: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:55:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:23 smithi059 ceph-mon[97450]: pgmap v20018: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:55:24.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:55:24.106+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:55:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:24.841+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:24.843607+0000) 2022-09-08T07:55:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:24.706+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:24.708197+0000) 2022-09-08T07:55:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:25.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:25.843820+0000) 2022-09-08T07:55:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:25.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:25.708396+0000) 2022-09-08T07:55:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:25 smithi055 ceph-mon[97033]: pgmap v20019: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:55:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:25 smithi059 ceph-mon[97450]: pgmap v20019: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:55:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:26.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:26.708552+0000) 2022-09-08T07:55:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:26.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:26.844005+0000) 2022-09-08T07:55:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:27.842+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:27.844163+0000) 2022-09-08T07:55:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:27.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:27.708704+0000) 2022-09-08T07:55:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:27 smithi055 ceph-mon[97033]: pgmap v20020: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:55:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:27 smithi059 ceph-mon[97450]: pgmap v20020: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:55:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:28.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:28.844423+0000) 2022-09-08T07:55:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:28.707+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:28.708837+0000) 2022-09-08T07:55:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:29 smithi055 ceph-mon[97033]: pgmap v20021: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:55:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:29 smithi059 ceph-mon[97450]: pgmap v20021: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:55:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:29.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:29.844738+0000) 2022-09-08T07:55:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:29.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:29.709047+0000) 2022-09-08T07:55:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:30.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:30.709274+0000) 2022-09-08T07:55:31.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:30.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:30.844950+0000) 2022-09-08T07:55:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:31.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:31.709502+0000) 2022-09-08T07:55:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:31.844+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:31.845210+0000) 2022-09-08T07:55:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:31 smithi055 ceph-mon[97033]: pgmap v20022: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:55:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:31 smithi059 ceph-mon[97450]: pgmap v20022: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:55:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:32.843+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:32.845452+0000) 2022-09-08T07:55:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:32.708+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:32.709690+0000) 2022-09-08T07:55:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:33.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:33.845721+0000) 2022-09-08T07:55:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:33.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:33.709916+0000) 2022-09-08T07:55:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:33 smithi055 ceph-mon[97033]: pgmap v20023: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:55:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:33 smithi059 ceph-mon[97450]: pgmap v20023: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:55:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:34.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:34.845970+0000) 2022-09-08T07:55:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:34.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:34.710121+0000) 2022-09-08T07:55:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:35.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:35.846225+0000) 2022-09-08T07:55:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:35.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:35.710267+0000) 2022-09-08T07:55:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:35 smithi055 ceph-mon[97033]: pgmap v20024: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:55:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:35 smithi059 ceph-mon[97450]: pgmap v20024: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:55:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:36.845+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:36.846488+0000) 2022-09-08T07:55:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:36.709+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:36.710442+0000) 2022-09-08T07:55:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:37.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:37.846719+0000) 2022-09-08T07:55:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:37.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:37.710704+0000) 2022-09-08T07:55:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:37 smithi055 ceph-mon[97033]: pgmap v20025: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:55:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:37 smithi059 ceph-mon[97450]: pgmap v20025: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:55:39.121 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:38.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:38.846956+0000) 2022-09-08T07:55:39.122 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:38.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:38.710870+0000) 2022-09-08T07:55:39.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:55:39.118+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:55:39.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:39 smithi055 ceph-mon[97033]: pgmap v20026: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:55:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:39 smithi059 ceph-mon[97450]: pgmap v20026: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:55:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:39.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:39.847211+0000) 2022-09-08T07:55:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:39.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:39.711027+0000) 2022-09-08T07:55:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:40.846+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:40.847375+0000) 2022-09-08T07:55:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:40.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:40.711208+0000) 2022-09-08T07:55:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:41.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:41.847616+0000) 2022-09-08T07:55:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:41.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:41.711370+0000) 2022-09-08T07:55:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:41 smithi055 ceph-mon[97033]: pgmap v20027: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:55:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:41 smithi059 ceph-mon[97450]: pgmap v20027: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:55:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:42.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:42.847874+0000) 2022-09-08T07:55:43.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:42.710+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:42.711526+0000) 2022-09-08T07:55:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:43.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:43.848094+0000) 2022-09-08T07:55:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:43.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:43.711742+0000) 2022-09-08T07:55:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:43 smithi055 ceph-mon[97033]: pgmap v20028: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:55:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:43 smithi059 ceph-mon[97450]: pgmap v20028: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:55:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:44.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:44.848274+0000) 2022-09-08T07:55:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:44.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:44.711929+0000) 2022-09-08T07:55:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:45.847+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:45.848498+0000) 2022-09-08T07:55:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:45.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:45.712114+0000) 2022-09-08T07:55:46.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:45 smithi055 ceph-mon[97033]: pgmap v20029: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:55:46.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:55:46.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:55:46.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:55:46.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:55:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:45 smithi059 ceph-mon[97450]: pgmap v20029: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:55:46.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:55:46.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:55:46.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:55:46.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:55:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:46.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:46.848658+0000) 2022-09-08T07:55:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:46.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:46.712340+0000) 2022-09-08T07:55:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:46 smithi055 ceph-mon[97033]: pgmap v20030: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T07:55:47.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:46 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T07:55:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:46 smithi059 ceph-mon[97450]: pgmap v20030: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T07:55:47.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:46 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T07:55:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:47.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:47.848849+0000) 2022-09-08T07:55:48.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:47.711+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:47.712511+0000) 2022-09-08T07:55:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:48.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:48.848978+0000) 2022-09-08T07:55:49.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:48.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:48.712702+0000) 2022-09-08T07:55:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:48 smithi055 ceph-mon[97033]: pgmap v20031: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:55:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:48 smithi059 ceph-mon[97450]: pgmap v20031: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:55:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:49.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:49.712852+0000) 2022-09-08T07:55:50.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:49.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:49.849175+0000) 2022-09-08T07:55:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:50.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:50.713004+0000) 2022-09-08T07:55:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:50.848+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:50.849341+0000) 2022-09-08T07:55:51.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:50 smithi055 ceph-mon[97033]: pgmap v20032: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:55:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:50 smithi059 ceph-mon[97450]: pgmap v20032: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:55:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:51.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:51.713262+0000) 2022-09-08T07:55:52.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:51.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:51.849551+0000) 2022-09-08T07:55:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:52.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:52.713428+0000) 2022-09-08T07:55:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:52.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:52.849795+0000) 2022-09-08T07:55:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:52 smithi055 ceph-mon[97033]: pgmap v20033: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:55:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:52 smithi059 ceph-mon[97450]: pgmap v20033: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:55:54.134 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:53.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:53.849956+0000) 2022-09-08T07:55:54.135 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:53.712+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:53.713574+0000) 2022-09-08T07:55:54.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:55:54.131+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:55:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:54 smithi055 ceph-mon[97033]: pgmap v20034: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:55:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:54 smithi059 ceph-mon[97450]: pgmap v20034: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T07:55:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:54.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:54.850144+0000) 2022-09-08T07:55:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:54.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:54.713742+0000) 2022-09-08T07:55:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:55.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:55.850286+0000) 2022-09-08T07:55:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:55.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:55.713948+0000) 2022-09-08T07:55:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:56 smithi055 ceph-mon[97033]: pgmap v20035: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T07:55:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:56 smithi059 ceph-mon[97450]: pgmap v20035: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T07:55:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:56.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:56.714202+0000) 2022-09-08T07:55:57.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:56.849+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:56.850441+0000) 2022-09-08T07:55:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:57.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:57.714350+0000) 2022-09-08T07:55:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:57.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:57.850680+0000) 2022-09-08T07:55:58.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:55:58 smithi055 ceph-mon[97033]: pgmap v20036: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:55:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:55:58 smithi059 ceph-mon[97450]: pgmap v20036: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:55:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:58.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:58.850848+0000) 2022-09-08T07:55:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:58.713+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:58.714504+0000) 2022-09-08T07:56:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:55:59.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:59.851059+0000) 2022-09-08T07:56:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:55:59.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:55:59.714752+0000) 2022-09-08T07:56:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:00 smithi055 ceph-mon[97033]: pgmap v20037: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:00 smithi059 ceph-mon[97450]: pgmap v20037: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:00.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:00.851267+0000) 2022-09-08T07:56:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:00.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:00.714929+0000) 2022-09-08T07:56:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:01.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:01.851502+0000) 2022-09-08T07:56:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:01.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:01.715134+0000) 2022-09-08T07:56:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:02 smithi055 ceph-mon[97033]: pgmap v20038: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:56:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:02 smithi059 ceph-mon[97450]: pgmap v20038: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:56:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:02.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:02.851739+0000) 2022-09-08T07:56:03.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:02.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:02.715359+0000) 2022-09-08T07:56:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:03.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:03.851978+0000) 2022-09-08T07:56:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:03.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:03.715587+0000) 2022-09-08T07:56:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:04 smithi055 ceph-mon[97033]: pgmap v20039: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:04 smithi059 ceph-mon[97450]: pgmap v20039: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:04.850+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:04.852152+0000) 2022-09-08T07:56:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:04.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:04.715790+0000) 2022-09-08T07:56:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:05.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:05.852333+0000) 2022-09-08T07:56:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:05.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:05.716005+0000) 2022-09-08T07:56:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:06 smithi055 ceph-mon[97033]: pgmap v20040: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:56:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:56:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:06 smithi059 ceph-mon[97450]: pgmap v20040: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:56:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:56:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:06.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:06.852551+0000) 2022-09-08T07:56:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:06.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:06.716145+0000) 2022-09-08T07:56:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:56:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:56:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:07.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:07.852807+0000) 2022-09-08T07:56:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:07.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:07.716353+0000) 2022-09-08T07:56:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:08 smithi055 ceph-mon[97033]: pgmap v20041: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:56:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:08 smithi059 ceph-mon[97450]: pgmap v20041: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:56:09.146 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:08.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:08.853032+0000) 2022-09-08T07:56:09.147 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:08.714+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:08.716576+0000) 2022-09-08T07:56:09.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:56:09.142+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:56:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:09.851+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:09.853229+0000) 2022-09-08T07:56:10.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:09.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:09.716775+0000) 2022-09-08T07:56:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:10 smithi055 ceph-mon[97033]: pgmap v20042: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:56:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:10 smithi055 ceph-mon[97033]: osdmap e1329: 12 total, 11 up, 11 in 2022-09-08T07:56:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:10 smithi059 ceph-mon[97450]: pgmap v20042: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:56:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:10 smithi059 ceph-mon[97450]: osdmap e1329: 12 total, 11 up, 11 in 2022-09-08T07:56:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:10.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:10.853474+0000) 2022-09-08T07:56:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:10.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:10.716922+0000) 2022-09-08T07:56:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:11.852+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:11.853719+0000) 2022-09-08T07:56:12.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:11.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:11.717109+0000) 2022-09-08T07:56:12.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:12 smithi055 ceph-mon[97033]: pgmap v20044: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:56:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:12 smithi059 ceph-mon[97450]: pgmap v20044: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:56:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:12.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:12.853957+0000) 2022-09-08T07:56:13.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:12.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:12.717310+0000) 2022-09-08T07:56:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:13.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:13.854214+0000) 2022-09-08T07:56:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:13.715+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:13.717516+0000) 2022-09-08T07:56:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:14 smithi055 ceph-mon[97033]: osdmap e1330: 12 total, 11 up, 11 in 2022-09-08T07:56:14.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:14 smithi055 ceph-mon[97033]: pgmap v20046: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:56:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:14 smithi059 ceph-mon[97450]: osdmap e1330: 12 total, 11 up, 11 in 2022-09-08T07:56:14.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:14 smithi059 ceph-mon[97450]: pgmap v20046: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:56:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:14.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:14.854487+0000) 2022-09-08T07:56:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:14.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:14.717691+0000) 2022-09-08T07:56:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:15.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:15.717863+0000) 2022-09-08T07:56:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:15.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:15.854725+0000) 2022-09-08T07:56:16.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:16 smithi055 ceph-mon[97033]: pgmap v20047: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:56:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:16 smithi059 ceph-mon[97450]: pgmap v20047: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:56:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:16.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:16.854953+0000) 2022-09-08T07:56:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:16.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:16.718010+0000) 2022-09-08T07:56:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:17.853+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:17.855204+0000) 2022-09-08T07:56:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:17.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:17.718133+0000) 2022-09-08T07:56:18.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:18 smithi055 ceph-mon[97033]: pgmap v20048: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:56:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:18 smithi059 ceph-mon[97450]: pgmap v20048: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:56:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:18.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:18.855445+0000) 2022-09-08T07:56:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:18.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:18.718304+0000) 2022-09-08T07:56:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:19.854+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:19.855715+0000) 2022-09-08T07:56:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:19.716+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:19.718476+0000) 2022-09-08T07:56:20.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:20 smithi055 ceph-mon[97033]: pgmap v20049: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T07:56:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:20 smithi059 ceph-mon[97450]: pgmap v20049: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T07:56:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:20.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:20.855991+0000) 2022-09-08T07:56:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:20.717+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:20.718648+0000) 2022-09-08T07:56:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:21.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:21.856171+0000) 2022-09-08T07:56:22.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:21.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:21.718842+0000) 2022-09-08T07:56:22.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:22 smithi055 ceph-mon[97033]: pgmap v20050: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:56:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:22 smithi059 ceph-mon[97450]: pgmap v20050: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:56:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:22.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:22.856344+0000) 2022-09-08T07:56:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:22.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:22.719048+0000) 2022-09-08T07:56:24.147 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:23.855+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:23.856558+0000) 2022-09-08T07:56:24.148 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:23.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:23.719204+0000) 2022-09-08T07:56:24.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:56:24.144+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:56:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:24 smithi055 ceph-mon[97033]: pgmap v20051: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T07:56:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:24 smithi059 ceph-mon[97450]: pgmap v20051: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T07:56:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:24.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:24.856806+0000) 2022-09-08T07:56:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:24.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:24.719422+0000) 2022-09-08T07:56:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:25.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:25.856950+0000) 2022-09-08T07:56:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:25.718+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:25.719605+0000) 2022-09-08T07:56:26.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:26 smithi055 ceph-mon[97033]: pgmap v20052: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:56:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:26 smithi059 ceph-mon[97450]: pgmap v20052: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:56:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:26.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:26.857198+0000) 2022-09-08T07:56:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:26.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:26.719755+0000) 2022-09-08T07:56:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:27.856+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:27.857448+0000) 2022-09-08T07:56:28.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:27.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:27.719974+0000) 2022-09-08T07:56:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:28 smithi055 ceph-mon[97033]: pgmap v20053: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:28 smithi059 ceph-mon[97450]: pgmap v20053: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:28.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:28.720167+0000) 2022-09-08T07:56:29.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:28.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:28.857764+0000) 2022-09-08T07:56:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:29.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:29.858010+0000) 2022-09-08T07:56:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:29.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:29.720337+0000) 2022-09-08T07:56:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:30 smithi055 ceph-mon[97033]: pgmap v20054: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:30 smithi059 ceph-mon[97450]: pgmap v20054: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:30.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:30.858236+0000) 2022-09-08T07:56:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:30.719+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:30.720544+0000) 2022-09-08T07:56:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:31.857+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:31.858468+0000) 2022-09-08T07:56:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:31.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:31.720721+0000) 2022-09-08T07:56:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:32 smithi059 ceph-mon[97450]: pgmap v20055: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:56:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:32 smithi055 ceph-mon[97033]: pgmap v20055: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:56:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:32.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:32.858735+0000) 2022-09-08T07:56:33.177 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:32.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:32.720849+0000) 2022-09-08T07:56:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:33.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:33.858907+0000) 2022-09-08T07:56:34.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:33.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:33.720980+0000) 2022-09-08T07:56:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:34 smithi055 ceph-mon[97033]: pgmap v20056: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:34 smithi059 ceph-mon[97450]: pgmap v20056: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:34.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:34.859161+0000) 2022-09-08T07:56:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:34.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:34.721165+0000) 2022-09-08T07:56:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:35.858+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:35.859428+0000) 2022-09-08T07:56:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:35.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:35.721379+0000) 2022-09-08T07:56:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:36 smithi055 ceph-mon[97033]: pgmap v20057: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:56:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:36 smithi059 ceph-mon[97450]: pgmap v20057: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:56:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:36.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:36.859629+0000) 2022-09-08T07:56:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:36.720+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:36.721553+0000) 2022-09-08T07:56:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:37.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:37.859857+0000) 2022-09-08T07:56:38.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:37.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:37.721704+0000) 2022-09-08T07:56:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:38 smithi055 ceph-mon[97033]: pgmap v20058: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:38 smithi059 ceph-mon[97450]: pgmap v20058: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:38.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:38.860087+0000) 2022-09-08T07:56:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:38.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:38.721910+0000) 2022-09-08T07:56:39.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:56:39.160+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:56:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:39.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:39.860300+0000) 2022-09-08T07:56:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:39.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:39.722134+0000) 2022-09-08T07:56:40.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:40 smithi055 ceph-mon[97033]: pgmap v20059: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:40 smithi059 ceph-mon[97450]: pgmap v20059: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:40.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:40.860464+0000) 2022-09-08T07:56:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:40.721+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:40.722332+0000) 2022-09-08T07:56:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:41.859+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:41.860619+0000) 2022-09-08T07:56:42.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:41.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:41.722534+0000) 2022-09-08T07:56:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:42 smithi055 ceph-mon[97033]: pgmap v20060: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:56:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:42 smithi059 ceph-mon[97450]: pgmap v20060: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:56:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:42.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:42.860882+0000) 2022-09-08T07:56:43.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:42.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:42.722702+0000) 2022-09-08T07:56:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:43.860+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:43.861147+0000) 2022-09-08T07:56:44.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:43.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:43.722886+0000) 2022-09-08T07:56:44.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:44 smithi055 ceph-mon[97033]: pgmap v20061: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:44 smithi059 ceph-mon[97450]: pgmap v20061: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:44.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:44.861427+0000) 2022-09-08T07:56:45.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:44.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:44.723047+0000) 2022-09-08T07:56:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:45.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:45.861592+0000) 2022-09-08T07:56:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:45.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:45.723282+0000) 2022-09-08T07:56:46.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:46 smithi055 ceph-mon[97033]: pgmap v20062: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:56:46.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:56:46.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:56:46.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:56:46.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:56:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:46 smithi059 ceph-mon[97450]: pgmap v20062: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:56:46.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:56:46.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:56:46.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:56:46.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:56:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:46.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:46.861801+0000) 2022-09-08T07:56:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:46.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:46.723473+0000) 2022-09-08T07:56:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:47.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:47.862056+0000) 2022-09-08T07:56:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:47.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:47.723608+0000) 2022-09-08T07:56:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:48 smithi055 ceph-mon[97033]: pgmap v20063: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:48 smithi059 ceph-mon[97450]: pgmap v20063: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:48.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:48.862215+0000) 2022-09-08T07:56:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:48.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:48.723858+0000) 2022-09-08T07:56:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:49.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:49.862399+0000) 2022-09-08T07:56:50.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:49.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:49.724106+0000) 2022-09-08T07:56:50.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:50 smithi059 ceph-mon[97450]: pgmap v20064: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:50 smithi055 ceph-mon[97033]: pgmap v20064: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:56:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:50.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:50.862609+0000) 2022-09-08T07:56:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:50.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:50.724255+0000) 2022-09-08T07:56:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:51.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:51.862855+0000) 2022-09-08T07:56:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:51.722+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:51.724379+0000) 2022-09-08T07:56:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:52 smithi059 ceph-mon[97450]: pgmap v20065: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:56:52.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:52 smithi055 ceph-mon[97033]: pgmap v20065: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:56:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:52.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:52.863076+0000) 2022-09-08T07:56:53.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:52.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:52.724604+0000) 2022-09-08T07:56:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:53.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:53.863278+0000) 2022-09-08T07:56:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:53.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:53.724762+0000) 2022-09-08T07:56:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:54 smithi055 ceph-mon[97033]: pgmap v20066: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:56:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:54 smithi059 ceph-mon[97450]: pgmap v20066: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:56:54.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:56:54.166+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:56:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:54.861+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:54.863559+0000) 2022-09-08T07:56:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:54.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:54.724924+0000) 2022-09-08T07:56:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:55.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:55.725149+0000) 2022-09-08T07:56:56.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:55.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:55.863780+0000) 2022-09-08T07:56:56.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:56 smithi055 ceph-mon[97033]: pgmap v20067: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:56:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:56 smithi059 ceph-mon[97450]: pgmap v20067: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:56:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:56.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:56.725294+0000) 2022-09-08T07:56:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:56.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:56.864032+0000) 2022-09-08T07:56:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:57.723+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:57.725439+0000) 2022-09-08T07:56:58.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:57.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:57.864174+0000) 2022-09-08T07:56:58.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:56:58 smithi055 ceph-mon[97033]: pgmap v20068: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:56:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:56:58 smithi059 ceph-mon[97450]: pgmap v20068: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:56:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:58.862+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:58.864339+0000) 2022-09-08T07:56:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:58.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:58.725700+0000) 2022-09-08T07:57:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:56:59.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:59.864571+0000) 2022-09-08T07:57:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:56:59.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:56:59.725851+0000) 2022-09-08T07:57:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:00 smithi059 ceph-mon[97450]: pgmap v20069: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:57:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:00 smithi055 ceph-mon[97033]: pgmap v20069: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T07:57:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:00.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:00.864690+0000) 2022-09-08T07:57:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:00.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:00.725987+0000) 2022-09-08T07:57:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:01.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:01.864835+0000) 2022-09-08T07:57:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:01.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:01.726129+0000) 2022-09-08T07:57:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:02 smithi059 ceph-mon[97450]: pgmap v20070: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:57:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:02 smithi055 ceph-mon[97033]: pgmap v20070: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:57:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:02.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:02.865002+0000) 2022-09-08T07:57:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:02.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:02.726273+0000) 2022-09-08T07:57:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:03.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:03.865241+0000) 2022-09-08T07:57:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:03.724+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:03.726411+0000) 2022-09-08T07:57:04.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:04 smithi055 ceph-mon[97033]: pgmap v20071: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:04 smithi059 ceph-mon[97450]: pgmap v20071: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:04.863+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:04.865407+0000) 2022-09-08T07:57:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:04.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:04.726600+0000) 2022-09-08T07:57:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:05.864+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:05.865704+0000) 2022-09-08T07:57:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:05.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:05.726709+0000) 2022-09-08T07:57:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:06 smithi055 ceph-mon[97033]: pgmap v20072: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:57:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:57:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:06 smithi059 ceph-mon[97450]: pgmap v20072: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:57:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:57:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:06.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:06.865918+0000) 2022-09-08T07:57:07.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:06.725+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:06.726846+0000) 2022-09-08T07:57:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:57:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:57:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:07.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:07.866138+0000) 2022-09-08T07:57:08.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:07.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:07.726997+0000) 2022-09-08T07:57:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:08 smithi059 ceph-mon[97450]: pgmap v20073: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:57:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:08 smithi055 ceph-mon[97033]: pgmap v20073: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:57:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:08.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:08.866355+0000) 2022-09-08T07:57:09.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:08.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:08.727158+0000) 2022-09-08T07:57:09.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:57:09.176+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:57:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:09.865+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:09.866503+0000) 2022-09-08T07:57:10.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:09.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:09.727392+0000) 2022-09-08T07:57:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:10 smithi059 ceph-mon[97450]: pgmap v20074: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:57:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:10 smithi059 ceph-mon[97450]: osdmap e1331: 12 total, 11 up, 11 in 2022-09-08T07:57:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:10 smithi055 ceph-mon[97033]: pgmap v20074: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:57:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:10 smithi055 ceph-mon[97033]: osdmap e1331: 12 total, 11 up, 11 in 2022-09-08T07:57:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:10.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:10.866747+0000) 2022-09-08T07:57:11.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:10.726+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:10.727596+0000) 2022-09-08T07:57:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:11.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:11.866986+0000) 2022-09-08T07:57:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:11.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:11.727830+0000) 2022-09-08T07:57:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:12 smithi059 ceph-mon[97450]: pgmap v20076: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:57:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:12 smithi055 ceph-mon[97033]: pgmap v20076: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:57:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:12.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:12.728000+0000) 2022-09-08T07:57:13.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:12.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:12.867190+0000) 2022-09-08T07:57:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:13.866+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:13.867434+0000) 2022-09-08T07:57:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:13.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:13.728164+0000) 2022-09-08T07:57:14.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:14 smithi055 ceph-mon[97033]: pgmap v20077: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:57:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:14 smithi059 ceph-mon[97450]: pgmap v20077: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:57:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:14.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:14.728343+0000) 2022-09-08T07:57:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:14.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:14.867703+0000) 2022-09-08T07:57:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:15.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:15.867962+0000) 2022-09-08T07:57:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:15.727+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:15.728493+0000) 2022-09-08T07:57:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:16 smithi059 ceph-mon[97450]: pgmap v20078: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:57:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:16 smithi055 ceph-mon[97033]: pgmap v20078: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:57:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:16.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:16.728666+0000) 2022-09-08T07:57:17.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:16.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:16.868163+0000) 2022-09-08T07:57:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:17.867+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:17.868394+0000) 2022-09-08T07:57:18.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:17.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:17.728850+0000) 2022-09-08T07:57:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:18 smithi059 ceph-mon[97450]: pgmap v20079: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:57:18.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:18 smithi059 ceph-mon[97450]: osdmap e1332: 12 total, 11 up, 11 in 2022-09-08T07:57:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:18 smithi055 ceph-mon[97033]: pgmap v20079: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:57:18.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:18 smithi055 ceph-mon[97033]: osdmap e1332: 12 total, 11 up, 11 in 2022-09-08T07:57:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:18.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:18.868615+0000) 2022-09-08T07:57:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:18.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:18.729086+0000) 2022-09-08T07:57:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:19.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:19.868845+0000) 2022-09-08T07:57:20.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:19.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:19.729301+0000) 2022-09-08T07:57:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:20 smithi059 ceph-mon[97450]: pgmap v20081: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T07:57:20.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:20 smithi055 ceph-mon[97033]: pgmap v20081: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T07:57:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:20.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:20.868964+0000) 2022-09-08T07:57:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:20.728+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:20.729510+0000) 2022-09-08T07:57:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:21.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:21.869181+0000) 2022-09-08T07:57:22.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:21.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:21.729718+0000) 2022-09-08T07:57:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:22 smithi059 ceph-mon[97450]: pgmap v20082: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:57:22.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:22 smithi055 ceph-mon[97033]: pgmap v20082: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:57:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:22.868+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:22.869408+0000) 2022-09-08T07:57:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:22.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:22.729967+0000) 2022-09-08T07:57:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:23.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:23.869672+0000) 2022-09-08T07:57:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:23.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:23.730216+0000) 2022-09-08T07:57:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:24 smithi055 ceph-mon[97033]: pgmap v20083: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:57:24.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:24 smithi059 ceph-mon[97450]: pgmap v20083: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:57:24.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:57:24.178+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:57:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:24.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:24.869820+0000) 2022-09-08T07:57:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:24.729+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:24.730453+0000) 2022-09-08T07:57:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:25.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:25.869996+0000) 2022-09-08T07:57:26.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:25.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:25.730710+0000) 2022-09-08T07:57:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:26 smithi059 ceph-mon[97450]: pgmap v20084: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:57:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:26 smithi055 ceph-mon[97033]: pgmap v20084: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:57:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:26.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:26.870215+0000) 2022-09-08T07:57:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:26.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:26.730853+0000) 2022-09-08T07:57:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:27.869+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:27.870461+0000) 2022-09-08T07:57:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:27.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:27.731032+0000) 2022-09-08T07:57:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:28 smithi059 ceph-mon[97450]: pgmap v20085: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:57:28.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:28 smithi055 ceph-mon[97033]: pgmap v20085: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:57:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:28.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:28.870629+0000) 2022-09-08T07:57:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:28.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:28.731182+0000) 2022-09-08T07:57:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:29.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:29.870796+0000) 2022-09-08T07:57:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:29.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:29.731328+0000) 2022-09-08T07:57:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:30 smithi059 ceph-mon[97450]: pgmap v20086: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:57:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:30 smithi055 ceph-mon[97033]: pgmap v20086: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:57:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:30.730+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:30.731541+0000) 2022-09-08T07:57:31.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:30.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:30.871034+0000) 2022-09-08T07:57:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:31.870+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:31.871167+0000) 2022-09-08T07:57:32.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:31.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:31.731704+0000) 2022-09-08T07:57:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:32 smithi059 ceph-mon[97450]: pgmap v20087: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:57:32.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:32 smithi055 ceph-mon[97033]: pgmap v20087: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:57:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:32.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:32.871441+0000) 2022-09-08T07:57:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:32.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:32.731924+0000) 2022-09-08T07:57:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:33.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:33.871672+0000) 2022-09-08T07:57:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:33.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:33.732097+0000) 2022-09-08T07:57:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:34 smithi055 ceph-mon[97033]: pgmap v20088: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:34 smithi059 ceph-mon[97450]: pgmap v20088: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:34.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:34.871906+0000) 2022-09-08T07:57:35.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:34.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:34.732252+0000) 2022-09-08T07:57:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:35.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:35.872188+0000) 2022-09-08T07:57:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:35.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:35.732404+0000) 2022-09-08T07:57:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:36 smithi059 ceph-mon[97450]: pgmap v20089: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:57:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:36 smithi055 ceph-mon[97033]: pgmap v20089: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:57:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:36.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:36.732535+0000) 2022-09-08T07:57:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:36.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:36.872377+0000) 2022-09-08T07:57:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:37.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:37.732736+0000) 2022-09-08T07:57:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:37.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:37.872608+0000) 2022-09-08T07:57:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:38 smithi059 ceph-mon[97450]: pgmap v20090: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:38 smithi055 ceph-mon[97033]: pgmap v20090: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:38.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:38.872843+0000) 2022-09-08T07:57:39.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:38.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:38.732887+0000) 2022-09-08T07:57:39.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:57:39.192+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:57:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:39.871+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:39.873054+0000) 2022-09-08T07:57:40.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:39.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:39.733051+0000) 2022-09-08T07:57:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:40 smithi059 ceph-mon[97450]: pgmap v20091: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:40.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:40 smithi055 ceph-mon[97033]: pgmap v20091: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:40.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:40.873273+0000) 2022-09-08T07:57:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:40.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:40.733206+0000) 2022-09-08T07:57:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:41.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:41.873503+0000) 2022-09-08T07:57:42.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:41.731+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:41.733369+0000) 2022-09-08T07:57:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:42 smithi059 ceph-mon[97450]: pgmap v20092: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:57:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:42 smithi055 ceph-mon[97033]: pgmap v20092: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:57:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:42.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:42.873751+0000) 2022-09-08T07:57:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:42.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:42.733589+0000) 2022-09-08T07:57:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:43.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:43.733800+0000) 2022-09-08T07:57:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:43.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:43.873978+0000) 2022-09-08T07:57:44.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:44 smithi055 ceph-mon[97033]: pgmap v20093: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:44 smithi059 ceph-mon[97450]: pgmap v20093: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:44.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:44.733964+0000) 2022-09-08T07:57:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:44.872+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:44.874164+0000) 2022-09-08T07:57:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:45.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:45.874355+0000) 2022-09-08T07:57:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:45.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:45.734127+0000) 2022-09-08T07:57:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:46 smithi059 ceph-mon[97450]: pgmap v20094: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:57:46.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:57:46.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:57:46.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:57:46.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:46 smithi055 ceph-mon[97033]: pgmap v20094: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:57:46.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:57:46.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:57:46.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:57:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:46.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:46.874555+0000) 2022-09-08T07:57:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:46.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:46.734284+0000) 2022-09-08T07:57:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:47.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:47.734418+0000) 2022-09-08T07:57:48.161 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:47.873+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:47.874802+0000) 2022-09-08T07:57:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:48.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:48.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:48.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:48 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:48.385 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:48.386 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:48.386 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:48.386 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:48 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:48.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:48.875021+0000) 2022-09-08T07:57:49.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:48.732+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:48.734564+0000) 2022-09-08T07:57:49.316 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:49 smithi055 ceph-mon[97033]: pgmap v20095: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:49.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:49 smithi059 ceph-mon[97450]: pgmap v20095: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:49.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:49.875266+0000) 2022-09-08T07:57:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:49.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:49.734790+0000) 2022-09-08T07:57:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:50.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:50.875486+0000) 2022-09-08T07:57:51.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:50.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:50.734992+0000) 2022-09-08T07:57:51.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:51 smithi055 ceph-mon[97033]: pgmap v20096: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:51.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:51.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:51.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:51.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:51.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:51 smithi059 ceph-mon[97450]: pgmap v20096: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:51.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:51.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:51.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:51.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:51.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:57:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:51.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:51.875730+0000) 2022-09-08T07:57:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:51.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:51.735235+0000) 2022-09-08T07:57:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:52 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T07:57:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:52 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T07:57:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:52 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T07:57:52.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:52 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T07:57:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:52.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:52.875852+0000) 2022-09-08T07:57:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:52.733+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:52.735409+0000) 2022-09-08T07:57:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:53 smithi055 ceph-mon[97033]: pgmap v20097: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:57:53.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:53 smithi059 ceph-mon[97450]: pgmap v20097: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:57:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:53.874+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:53.876001+0000) 2022-09-08T07:57:54.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:53.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:53.735577+0000) 2022-09-08T07:57:54.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:54 smithi055 ceph-mon[97033]: pgmap v20098: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:54 smithi059 ceph-mon[97450]: pgmap v20098: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:54.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:57:54.193+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:57:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:54.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:54.876234+0000) 2022-09-08T07:57:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:54.734+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:54.735809+0000) 2022-09-08T07:57:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:55.875+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:55.876470+0000) 2022-09-08T07:57:56.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:55.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:55.735996+0000) 2022-09-08T07:57:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:56 smithi059 ceph-mon[97450]: pgmap v20099: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:57:56.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:56 smithi055 ceph-mon[97033]: pgmap v20099: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:57:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:56.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:56.876706+0000) 2022-09-08T07:57:57.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:56.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:56.736136+0000) 2022-09-08T07:57:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:57.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:57.876943+0000) 2022-09-08T07:57:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:57.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:57.736313+0000) 2022-09-08T07:57:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:57:58 smithi059 ceph-mon[97450]: pgmap v20100: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:57:58 smithi055 ceph-mon[97033]: pgmap v20100: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:57:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:58.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:58.877169+0000) 2022-09-08T07:57:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:58.735+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:58.736482+0000) 2022-09-08T07:58:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:57:59.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:59.877403+0000) 2022-09-08T07:58:00.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:57:59.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:57:59.736657+0000) 2022-09-08T07:58:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:00 smithi059 ceph-mon[97450]: pgmap v20101: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:00 smithi055 ceph-mon[97033]: pgmap v20101: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:00.876+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:00.877576+0000) 2022-09-08T07:58:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:00.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:00.736831+0000) 2022-09-08T07:58:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:01.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:01.877806+0000) 2022-09-08T07:58:02.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:01.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:01.737111+0000) 2022-09-08T07:58:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:02 smithi059 ceph-mon[97450]: pgmap v20102: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:58:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:02 smithi055 ceph-mon[97033]: pgmap v20102: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:58:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:02.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:02.878023+0000) 2022-09-08T07:58:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:02.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:02.737364+0000) 2022-09-08T07:58:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:03.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:03.878203+0000) 2022-09-08T07:58:04.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:03.736+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:03.737592+0000) 2022-09-08T07:58:04.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:04 smithi055 ceph-mon[97033]: pgmap v20103: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:04 smithi059 ceph-mon[97450]: pgmap v20103: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:04.877+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:04.878418+0000) 2022-09-08T07:58:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:04.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:04.737765+0000) 2022-09-08T07:58:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:05.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:05.878664+0000) 2022-09-08T07:58:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:05.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:05.737915+0000) 2022-09-08T07:58:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:06 smithi059 ceph-mon[97450]: pgmap v20104: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:58:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:58:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:06 smithi055 ceph-mon[97033]: pgmap v20104: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:58:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:58:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:06.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:06.878888+0000) 2022-09-08T07:58:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:06.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:06.738042+0000) 2022-09-08T07:58:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:58:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:58:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:07.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:07.879129+0000) 2022-09-08T07:58:08.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:07.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:07.738235+0000) 2022-09-08T07:58:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:08 smithi059 ceph-mon[97450]: pgmap v20105: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:58:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:08 smithi055 ceph-mon[97033]: pgmap v20105: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:58:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:08.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:08.879352+0000) 2022-09-08T07:58:09.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:08.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:08.738393+0000) 2022-09-08T07:58:09.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:58:09.202+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:58:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:09.878+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:09.879589+0000) 2022-09-08T07:58:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:09.737+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:09.738524+0000) 2022-09-08T07:58:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:10 smithi059 ceph-mon[97450]: pgmap v20106: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:58:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:10 smithi059 ceph-mon[97450]: osdmap e1333: 12 total, 11 up, 11 in 2022-09-08T07:58:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:10 smithi055 ceph-mon[97033]: pgmap v20106: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:58:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:10 smithi055 ceph-mon[97033]: osdmap e1333: 12 total, 11 up, 11 in 2022-09-08T07:58:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:10.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:10.879831+0000) 2022-09-08T07:58:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:10.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:10.738748+0000) 2022-09-08T07:58:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:11.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:11.880053+0000) 2022-09-08T07:58:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:11.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:11.738943+0000) 2022-09-08T07:58:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:12 smithi059 ceph-mon[97450]: pgmap v20108: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:58:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:12 smithi055 ceph-mon[97033]: pgmap v20108: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:58:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:12.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:12.880272+0000) 2022-09-08T07:58:13.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:12.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:12.739094+0000) 2022-09-08T07:58:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:13.879+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:13.880485+0000) 2022-09-08T07:58:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:13.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:13.739259+0000) 2022-09-08T07:58:14.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:14 smithi055 ceph-mon[97033]: osdmap e1334: 12 total, 11 up, 11 in 2022-09-08T07:58:14.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:14 smithi055 ceph-mon[97033]: pgmap v20110: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:58:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:14 smithi059 ceph-mon[97450]: osdmap e1334: 12 total, 11 up, 11 in 2022-09-08T07:58:14.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:14 smithi059 ceph-mon[97450]: pgmap v20110: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T07:58:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:14.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:14.880652+0000) 2022-09-08T07:58:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:14.738+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:14.739523+0000) 2022-09-08T07:58:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:15.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:15.880860+0000) 2022-09-08T07:58:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:15.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:15.739693+0000) 2022-09-08T07:58:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:16 smithi059 ceph-mon[97450]: pgmap v20111: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:58:16.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:16 smithi055 ceph-mon[97033]: pgmap v20111: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:58:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:16.880+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:16.881064+0000) 2022-09-08T07:58:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:16.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:16.739874+0000) 2022-09-08T07:58:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:17.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:17.881296+0000) 2022-09-08T07:58:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:17.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:17.740128+0000) 2022-09-08T07:58:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:18 smithi059 ceph-mon[97450]: pgmap v20112: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:58:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:18 smithi055 ceph-mon[97033]: pgmap v20112: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T07:58:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:18.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:18.881487+0000) 2022-09-08T07:58:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:18.739+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:18.740280+0000) 2022-09-08T07:58:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:19.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:19.881626+0000) 2022-09-08T07:58:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:19.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:19.740493+0000) 2022-09-08T07:58:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:20 smithi059 ceph-mon[97450]: pgmap v20113: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T07:58:20.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:20 smithi055 ceph-mon[97033]: pgmap v20113: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T07:58:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:20.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:20.881795+0000) 2022-09-08T07:58:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:20.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:20.740622+0000) 2022-09-08T07:58:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:21.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:21.881943+0000) 2022-09-08T07:58:22.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:21.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:21.740782+0000) 2022-09-08T07:58:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:22 smithi059 ceph-mon[97450]: pgmap v20114: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:58:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:22 smithi055 ceph-mon[97033]: pgmap v20114: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:58:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:22.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:22.882140+0000) 2022-09-08T07:58:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:22.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:22.741074+0000) 2022-09-08T07:58:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:23.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:23.882350+0000) 2022-09-08T07:58:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:23.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:23.741342+0000) 2022-09-08T07:58:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:24 smithi059 ceph-mon[97450]: pgmap v20115: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T07:58:24.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:58:24.203+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:58:24.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:24 smithi055 ceph-mon[97033]: pgmap v20115: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 100 B/s wr, 6 op/s 2022-09-08T07:58:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:24.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:24.882543+0000) 2022-09-08T07:58:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:24.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:24.741547+0000) 2022-09-08T07:58:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:25.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:25.882783+0000) 2022-09-08T07:58:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:25.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:25.741795+0000) 2022-09-08T07:58:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:26 smithi059 ceph-mon[97450]: pgmap v20116: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:58:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:26 smithi055 ceph-mon[97033]: pgmap v20116: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T07:58:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:26.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:26.883023+0000) 2022-09-08T07:58:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:26.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:26.742017+0000) 2022-09-08T07:58:28.159 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:27.881+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:27.883254+0000) 2022-09-08T07:58:28.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:27.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:27.742200+0000) 2022-09-08T07:58:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:28 smithi059 ceph-mon[97450]: pgmap v20117: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:28 smithi055 ceph-mon[97033]: pgmap v20117: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:28.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:28.883439+0000) 2022-09-08T07:58:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:28.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:28.742366+0000) 2022-09-08T07:58:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:29.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:29.883685+0000) 2022-09-08T07:58:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:29.740+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:29.742567+0000) 2022-09-08T07:58:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:30 smithi059 ceph-mon[97450]: pgmap v20118: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:30 smithi055 ceph-mon[97033]: pgmap v20118: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:30.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:30.883892+0000) 2022-09-08T07:58:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:30.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:30.742789+0000) 2022-09-08T07:58:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:31.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:31.884061+0000) 2022-09-08T07:58:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:31.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:31.742929+0000) 2022-09-08T07:58:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:32 smithi059 ceph-mon[97450]: pgmap v20119: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:58:32.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:32 smithi055 ceph-mon[97033]: pgmap v20119: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:58:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:32.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:32.884293+0000) 2022-09-08T07:58:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:32.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:32.743092+0000) 2022-09-08T07:58:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:33.882+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:33.884518+0000) 2022-09-08T07:58:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:33.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:33.743248+0000) 2022-09-08T07:58:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:34 smithi059 ceph-mon[97450]: pgmap v20120: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:34 smithi055 ceph-mon[97033]: pgmap v20120: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:34.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:34.884744+0000) 2022-09-08T07:58:35.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:34.741+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:34.743425+0000) 2022-09-08T07:58:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:35.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:35.884982+0000) 2022-09-08T07:58:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:35.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:35.743581+0000) 2022-09-08T07:58:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:36 smithi059 ceph-mon[97450]: pgmap v20121: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:58:36.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:36 smithi055 ceph-mon[97033]: pgmap v20121: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:58:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:36.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:36.885232+0000) 2022-09-08T07:58:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:36.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:36.743790+0000) 2022-09-08T07:58:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:37.883+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:37.885414+0000) 2022-09-08T07:58:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:37.742+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:37.743973+0000) 2022-09-08T07:58:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:38 smithi059 ceph-mon[97450]: pgmap v20122: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:38 smithi055 ceph-mon[97033]: pgmap v20122: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:38.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:38.885659+0000) 2022-09-08T07:58:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:38.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:38.744123+0000) 2022-09-08T07:58:39.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:58:39.204+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:58:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:39.884+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:39.885832+0000) 2022-09-08T07:58:40.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:39.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:39.744252+0000) 2022-09-08T07:58:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:40 smithi059 ceph-mon[97450]: pgmap v20123: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:40 smithi055 ceph-mon[97033]: pgmap v20123: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:40.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:40.744421+0000) 2022-09-08T07:58:41.175 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:40.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:40.886038+0000) 2022-09-08T07:58:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:41.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:41.744630+0000) 2022-09-08T07:58:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:41.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:41.886237+0000) 2022-09-08T07:58:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:42 smithi059 ceph-mon[97450]: pgmap v20124: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:58:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:42 smithi055 ceph-mon[97033]: pgmap v20124: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:58:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:42.743+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:42.744857+0000) 2022-09-08T07:58:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:42.885+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:42.886482+0000) 2022-09-08T07:58:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:43.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:43.745081+0000) 2022-09-08T07:58:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:43.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:43.886736+0000) 2022-09-08T07:58:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:44 smithi059 ceph-mon[97450]: pgmap v20125: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:44.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:44 smithi055 ceph-mon[97033]: pgmap v20125: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:44.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:44.745255+0000) 2022-09-08T07:58:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:44.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:44.886873+0000) 2022-09-08T07:58:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:45.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:45.887040+0000) 2022-09-08T07:58:46.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:45.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:45.745425+0000) 2022-09-08T07:58:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:46 smithi059 ceph-mon[97450]: pgmap v20126: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:58:46.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:46 smithi055 ceph-mon[97033]: pgmap v20126: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:58:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:46.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:46.887256+0000) 2022-09-08T07:58:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:46.744+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:46.745563+0000) 2022-09-08T07:58:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:47.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:47.745699+0000) 2022-09-08T07:58:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:47.886+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:47.887467+0000) 2022-09-08T07:58:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:48 smithi059 ceph-mon[97450]: pgmap v20127: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:48 smithi055 ceph-mon[97033]: pgmap v20127: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:48.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:48.887657+0000) 2022-09-08T07:58:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:48.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:48.745841+0000) 2022-09-08T07:58:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:49.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:49.887892+0000) 2022-09-08T07:58:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:49.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:49.745970+0000) 2022-09-08T07:58:50.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:50 smithi059 ceph-mon[97450]: pgmap v20128: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:50.829 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:50 smithi055 ceph-mon[97033]: pgmap v20128: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:50.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:50.888165+0000) 2022-09-08T07:58:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:50.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:50.746125+0000) 2022-09-08T07:58:51.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:58:51.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:58:51.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:58:51.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:58:51.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:51 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:58:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:58:51.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:58:51.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:58:51.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:58:51.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:51 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:58:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:51.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:51.888292+0000) 2022-09-08T07:58:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:51.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:51.746281+0000) 2022-09-08T07:58:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:52 smithi059 ceph-mon[97450]: pgmap v20129: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:58:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:52 smithi055 ceph-mon[97033]: pgmap v20129: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:58:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:52.887+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:52.888481+0000) 2022-09-08T07:58:53.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:52.745+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:52.746455+0000) 2022-09-08T07:58:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:53.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:53.888706+0000) 2022-09-08T07:58:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:53.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:53.746670+0000) 2022-09-08T07:58:54.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:58:54.220+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:58:54.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: pgmap v20130: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:54.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:58:54.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:58:54.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:58:54.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:58:54.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:54.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:54.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:54.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:54.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 101.5M 2022-09-08T07:58:54.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 106484371: error parsing value: Value '106484371' is below minimum 939524096 2022-09-08T07:58:54.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:54.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:54.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:54.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:54.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:54 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:58:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: pgmap v20130: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:58:55.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:58:55.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:58:55.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:58:55.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:55.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:55.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:55.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:55.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 101.5M 2022-09-08T07:58:55.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 106484371: error parsing value: Value '106484371' is below minimum 939524096 2022-09-08T07:58:55.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:55.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:55.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:55.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T07:58:55.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:54 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:58:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:54.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:54.888953+0000) 2022-09-08T07:58:55.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:54.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:54.746929+0000) 2022-09-08T07:58:55.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:55 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T07:58:55.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:55 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T07:58:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:55 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T07:58:56.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:55 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T07:58:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:55.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:55.747143+0000) 2022-09-08T07:58:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:55.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:55.889153+0000) 2022-09-08T07:58:56.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:56 smithi055 ceph-mon[97033]: pgmap v20131: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:58:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:56 smithi059 ceph-mon[97450]: pgmap v20131: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:58:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:56.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:56.747269+0000) 2022-09-08T07:58:57.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:56.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:56.889311+0000) 2022-09-08T07:58:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:57.888+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:57.889530+0000) 2022-09-08T07:58:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:57.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:57.747403+0000) 2022-09-08T07:58:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:58:58 smithi055 ceph-mon[97033]: pgmap v20132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:58:58 smithi059 ceph-mon[97450]: pgmap v20132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:58:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:58.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:58.889754+0000) 2022-09-08T07:58:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:58.746+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:58.747588+0000) 2022-09-08T07:59:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:58:59.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:59.889966+0000) 2022-09-08T07:59:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:58:59.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:58:59.747810+0000) 2022-09-08T07:59:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:00 smithi055 ceph-mon[97033]: pgmap v20133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:00 smithi059 ceph-mon[97450]: pgmap v20133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:00.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:00.890189+0000) 2022-09-08T07:59:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:00.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:00.747971+0000) 2022-09-08T07:59:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:01.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:01.748150+0000) 2022-09-08T07:59:02.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:01.889+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:01.890327+0000) 2022-09-08T07:59:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:02 smithi055 ceph-mon[97033]: pgmap v20134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:59:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:02 smithi059 ceph-mon[97450]: pgmap v20134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:59:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:02.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:02.890538+0000) 2022-09-08T07:59:03.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:02.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:02.748347+0000) 2022-09-08T07:59:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:03.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:03.890795+0000) 2022-09-08T07:59:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:03.747+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:03.748496+0000) 2022-09-08T07:59:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:04 smithi059 ceph-mon[97450]: pgmap v20135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:04 smithi055 ceph-mon[97033]: pgmap v20135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:04.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:04.891067+0000) 2022-09-08T07:59:05.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:04.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:04.748715+0000) 2022-09-08T07:59:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:05.890+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:05.891303+0000) 2022-09-08T07:59:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:05.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:05.748946+0000) 2022-09-08T07:59:06.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:06 smithi059 ceph-mon[97450]: pgmap v20136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:59:06.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:59:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:06 smithi055 ceph-mon[97033]: pgmap v20136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T07:59:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T07:59:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:06.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:06.891539+0000) 2022-09-08T07:59:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:06.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:06.749126+0000) 2022-09-08T07:59:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:59:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T07:59:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:07.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:07.891759+0000) 2022-09-08T07:59:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:07.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:07.749289+0000) 2022-09-08T07:59:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:08 smithi059 ceph-mon[97450]: pgmap v20137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:59:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:08 smithi055 ceph-mon[97033]: pgmap v20137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:59:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:08.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:08.891995+0000) 2022-09-08T07:59:09.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:08.748+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:08.749505+0000) 2022-09-08T07:59:09.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:59:09.236+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:59:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:09.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:09.892144+0000) 2022-09-08T07:59:10.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:09.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:09.749787+0000) 2022-09-08T07:59:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:10 smithi055 ceph-mon[97033]: pgmap v20138: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:59:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:10 smithi055 ceph-mon[97033]: osdmap e1335: 12 total, 11 up, 11 in 2022-09-08T07:59:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:10 smithi059 ceph-mon[97450]: pgmap v20138: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T07:59:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:10 smithi059 ceph-mon[97450]: osdmap e1335: 12 total, 11 up, 11 in 2022-09-08T07:59:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:10.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:10.892369+0000) 2022-09-08T07:59:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:10.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:10.749962+0000) 2022-09-08T07:59:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:11.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:11.892601+0000) 2022-09-08T07:59:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:11.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:11.750099+0000) 2022-09-08T07:59:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:12 smithi055 ceph-mon[97033]: pgmap v20140: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:59:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:12 smithi059 ceph-mon[97450]: pgmap v20140: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:59:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:12.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:12.892799+0000) 2022-09-08T07:59:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:12.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:12.750319+0000) 2022-09-08T07:59:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:13.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:13.892966+0000) 2022-09-08T07:59:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:13.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:13.750540+0000) 2022-09-08T07:59:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:14 smithi059 ceph-mon[97450]: pgmap v20141: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:59:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:14 smithi055 ceph-mon[97033]: pgmap v20141: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T07:59:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:14.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:14.893194+0000) 2022-09-08T07:59:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:14.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:14.750755+0000) 2022-09-08T07:59:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:15.891+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:15.893407+0000) 2022-09-08T07:59:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:15.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:15.750916+0000) 2022-09-08T07:59:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:16 smithi059 ceph-mon[97450]: pgmap v20142: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:59:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:16 smithi055 ceph-mon[97033]: pgmap v20142: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:59:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:16.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:16.893756+0000) 2022-09-08T07:59:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:16.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:16.751063+0000) 2022-09-08T07:59:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:17.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:17.893926+0000) 2022-09-08T07:59:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:17.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:17.751276+0000) 2022-09-08T07:59:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:18 smithi059 ceph-mon[97450]: pgmap v20143: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:59:18.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:18 smithi059 ceph-mon[97450]: osdmap e1336: 12 total, 11 up, 11 in 2022-09-08T07:59:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:18 smithi055 ceph-mon[97033]: pgmap v20143: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T07:59:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:18 smithi055 ceph-mon[97033]: osdmap e1336: 12 total, 11 up, 11 in 2022-09-08T07:59:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:18.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:18.894154+0000) 2022-09-08T07:59:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:18.749+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:18.751535+0000) 2022-09-08T07:59:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:19.892+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:19.894380+0000) 2022-09-08T07:59:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:19.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:19.751747+0000) 2022-09-08T07:59:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:20 smithi059 ceph-mon[97450]: pgmap v20145: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T07:59:20.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:20 smithi055 ceph-mon[97033]: pgmap v20145: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T07:59:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:20.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:20.894625+0000) 2022-09-08T07:59:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:20.750+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:20.751953+0000) 2022-09-08T07:59:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:21.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:21.894807+0000) 2022-09-08T07:59:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:21.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:21.752199+0000) 2022-09-08T07:59:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:22 smithi059 ceph-mon[97450]: pgmap v20146: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:59:22.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:22 smithi055 ceph-mon[97033]: pgmap v20146: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:59:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:22.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:22.895014+0000) 2022-09-08T07:59:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:22.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:22.752386+0000) 2022-09-08T07:59:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:23.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:23.895194+0000) 2022-09-08T07:59:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:23.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:23.752667+0000) 2022-09-08T07:59:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:24 smithi059 ceph-mon[97450]: pgmap v20147: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:59:24.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:59:24.246+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:59:24.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:24 smithi055 ceph-mon[97033]: pgmap v20147: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T07:59:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:24.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:24.895369+0000) 2022-09-08T07:59:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:24.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:24.752870+0000) 2022-09-08T07:59:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:25.893+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:25.895499+0000) 2022-09-08T07:59:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:25.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:25.753106+0000) 2022-09-08T07:59:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:26 smithi059 ceph-mon[97450]: pgmap v20148: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:59:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:26 smithi055 ceph-mon[97033]: pgmap v20148: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:59:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:26.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:26.895750+0000) 2022-09-08T07:59:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:26.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:26.753362+0000) 2022-09-08T07:59:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:27.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:27.895919+0000) 2022-09-08T07:59:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:27.751+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:27.753573+0000) 2022-09-08T07:59:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:28 smithi059 ceph-mon[97450]: pgmap v20149: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:59:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:28 smithi055 ceph-mon[97033]: pgmap v20149: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T07:59:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:28.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:28.753736+0000) 2022-09-08T07:59:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:28.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:28.896136+0000) 2022-09-08T07:59:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:29.894+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:29.896352+0000) 2022-09-08T07:59:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:29.752+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:29.753879+0000) 2022-09-08T07:59:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:30 smithi059 ceph-mon[97450]: pgmap v20150: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:59:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:30 smithi055 ceph-mon[97033]: pgmap v20150: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T07:59:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:30.895+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:30.896597+0000) 2022-09-08T07:59:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:30.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:30.754024+0000) 2022-09-08T07:59:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:31.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:31.896848+0000) 2022-09-08T07:59:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:31.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:31.754249+0000) 2022-09-08T07:59:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:32 smithi059 ceph-mon[97450]: pgmap v20151: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:59:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:32 smithi055 ceph-mon[97033]: pgmap v20151: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:59:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:32.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:32.897026+0000) 2022-09-08T07:59:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:32.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:32.754417+0000) 2022-09-08T07:59:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:33.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:33.897216+0000) 2022-09-08T07:59:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:33.753+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:33.754564+0000) 2022-09-08T07:59:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:34 smithi059 ceph-mon[97450]: pgmap v20152: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:34 smithi055 ceph-mon[97033]: pgmap v20152: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:34.896+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:34.897445+0000) 2022-09-08T07:59:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:34.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:34.754716+0000) 2022-09-08T07:59:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:35.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:35.897744+0000) 2022-09-08T07:59:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:35.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:35.754888+0000) 2022-09-08T07:59:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:36 smithi059 ceph-mon[97450]: pgmap v20153: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:59:36.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:36 smithi055 ceph-mon[97033]: pgmap v20153: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:59:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:36.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:36.898009+0000) 2022-09-08T07:59:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:36.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:36.755043+0000) 2022-09-08T07:59:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:37.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:37.898262+0000) 2022-09-08T07:59:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:37.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:37.755265+0000) 2022-09-08T07:59:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:38 smithi059 ceph-mon[97450]: pgmap v20154: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:38 smithi055 ceph-mon[97033]: pgmap v20154: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:38.897+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:38.898500+0000) 2022-09-08T07:59:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:38.754+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:38.755458+0000) 2022-09-08T07:59:39.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:59:39.263+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:59:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:39.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:39.755664+0000) 2022-09-08T07:59:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:39.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:39.898724+0000) 2022-09-08T07:59:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:40 smithi059 ceph-mon[97450]: pgmap v20155: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:40.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:40 smithi055 ceph-mon[97033]: pgmap v20155: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:40.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:40.755809+0000) 2022-09-08T07:59:41.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:40.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:40.898958+0000) 2022-09-08T07:59:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:41.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:41.899138+0000) 2022-09-08T07:59:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:41.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:41.755956+0000) 2022-09-08T07:59:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:42 smithi055 ceph-mon[97033]: pgmap v20156: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:59:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:42 smithi059 ceph-mon[97450]: pgmap v20156: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:59:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:42.898+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:42.899314+0000) 2022-09-08T07:59:43.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:42.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:42.756116+0000) 2022-09-08T07:59:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:43.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:43.899546+0000) 2022-09-08T07:59:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:43.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:43.756276+0000) 2022-09-08T07:59:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:44 smithi059 ceph-mon[97450]: pgmap v20157: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:44 smithi055 ceph-mon[97033]: pgmap v20157: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:44.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:44.756414+0000) 2022-09-08T07:59:45.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:44.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:44.899792+0000) 2022-09-08T07:59:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:45.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:45.899992+0000) 2022-09-08T07:59:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:45.755+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:45.756574+0000) 2022-09-08T07:59:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:46 smithi059 ceph-mon[97450]: pgmap v20158: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:59:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:46 smithi055 ceph-mon[97033]: pgmap v20158: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:59:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:46.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:46.900168+0000) 2022-09-08T07:59:47.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:46.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:46.756771+0000) 2022-09-08T07:59:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:47.899+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:47.900376+0000) 2022-09-08T07:59:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:47.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:47.756911+0000) 2022-09-08T07:59:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:48 smithi059 ceph-mon[97450]: pgmap v20159: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:48 smithi055 ceph-mon[97033]: pgmap v20159: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:48.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:48.900620+0000) 2022-09-08T07:59:49.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:48.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:48.757084+0000) 2022-09-08T07:59:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:49.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:49.900890+0000) 2022-09-08T07:59:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:49.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:49.757232+0000) 2022-09-08T07:59:50.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:50 smithi055 ceph-mon[97033]: pgmap v20160: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:50 smithi059 ceph-mon[97450]: pgmap v20160: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:50.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:50.901083+0000) 2022-09-08T07:59:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:50.756+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:50.757455+0000) 2022-09-08T07:59:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:51.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:51.901293+0000) 2022-09-08T07:59:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:51.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:51.757688+0000) 2022-09-08T07:59:52.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:52 smithi055 ceph-mon[97033]: pgmap v20161: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:59:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:52 smithi059 ceph-mon[97450]: pgmap v20161: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:59:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:52.900+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:52.901523+0000) 2022-09-08T07:59:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:52.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:52.757843+0000) 2022-09-08T07:59:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:53.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:53.901773+0000) 2022-09-08T07:59:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:53.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:53.758007+0000) 2022-09-08T07:59:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:54 smithi059 ceph-mon[97450]: pgmap v20162: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:54.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 07:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T07:59:54.277+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T07:59:54.682 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:54 smithi055 ceph-mon[97033]: pgmap v20162: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:54.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:54.901997+0000) 2022-09-08T07:59:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:54.757+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:54.758241+0000) 2022-09-08T07:59:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:59:55.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:59:55.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:59:55.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:59:55.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:55 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:59:55.643 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T07:59:55.644 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T07:59:55.644 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T07:59:55.644 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:59:55.645 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:55 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:59:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:55.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:55.902182+0000) 2022-09-08T07:59:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:55.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:55.758485+0000) 2022-09-08T07:59:56.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:56 smithi055 ceph-mon[97033]: pgmap v20163: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:59:56.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:59:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:56 smithi059 ceph-mon[97450]: pgmap v20163: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T07:59:57.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T07:59:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:56.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:56.902323+0000) 2022-09-08T07:59:57.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:56.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:56.758700+0000) 2022-09-08T07:59:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:57.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:57.902531+0000) 2022-09-08T07:59:58.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:57.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:57.758847+0000) 2022-09-08T07:59:58.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 07:59:58 smithi055 ceph-mon[97033]: pgmap v20164: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 07:59:58 smithi059 ceph-mon[97450]: pgmap v20164: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T07:59:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:58.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:58.902759+0000) 2022-09-08T07:59:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:58.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:58.759043+0000) 2022-09-08T08:00:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 07:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T07:59:59.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:59.902900+0000) 2022-09-08T08:00:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 07:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T07:59:59.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T06:59:59.759204+0000) 2022-09-08T08:00:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:00 smithi055 ceph-mon[97033]: pgmap v20165: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T08:00:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T08:00:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T08:00:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:00 smithi059 ceph-mon[97450]: pgmap v20165: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T08:00:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T08:00:01.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T08:00:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:00.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:00.903132+0000) 2022-09-08T08:00:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:00.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:00.759415+0000) 2022-09-08T08:00:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:01.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:01.903307+0000) 2022-09-08T08:00:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:01.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:01.759598+0000) 2022-09-08T08:00:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:02 smithi055 ceph-mon[97033]: pgmap v20166: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:00:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:02 smithi059 ceph-mon[97450]: pgmap v20166: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:00:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:02.901+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:02.903548+0000) 2022-09-08T08:00:03.168 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:02.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:02.759805+0000) 2022-09-08T08:00:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:03.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:03.903775+0000) 2022-09-08T08:00:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:03.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:03.759994+0000) 2022-09-08T08:00:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:04 smithi059 ceph-mon[97450]: pgmap v20167: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:04.547 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:04 smithi055 ceph-mon[97033]: pgmap v20167: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:04.902+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:04.903902+0000) 2022-09-08T08:00:05.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:04.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:04.760154+0000) 2022-09-08T08:00:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:05.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:05.904160+0000) 2022-09-08T08:00:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:05.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:05.760320+0000) 2022-09-08T08:00:06.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:06 smithi059 ceph-mon[97450]: pgmap v20168: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:00:06.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:00:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:06 smithi055 ceph-mon[97033]: pgmap v20168: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:00:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:00:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:06.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:06.904408+0000) 2022-09-08T08:00:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:06.758+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:06.760479+0000) 2022-09-08T08:00:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:00:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:00:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:07.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:07.904663+0000) 2022-09-08T08:00:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:07.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:07.760728+0000) 2022-09-08T08:00:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:08 smithi055 ceph-mon[97033]: pgmap v20169: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:00:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:08 smithi059 ceph-mon[97450]: pgmap v20169: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:00:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:08.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:08.904925+0000) 2022-09-08T08:00:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:08.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:08.760925+0000) 2022-09-08T08:00:09.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:00:09.283+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:00:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:09.903+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:09.905058+0000) 2022-09-08T08:00:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:09.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:09.761103+0000) 2022-09-08T08:00:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:10 smithi055 ceph-mon[97033]: pgmap v20170: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:00:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:10 smithi055 ceph-mon[97033]: osdmap e1337: 12 total, 11 up, 11 in 2022-09-08T08:00:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:10 smithi059 ceph-mon[97450]: pgmap v20170: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:00:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:10 smithi059 ceph-mon[97450]: osdmap e1337: 12 total, 11 up, 11 in 2022-09-08T08:00:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:10.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:10.905290+0000) 2022-09-08T08:00:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:10.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:10.761255+0000) 2022-09-08T08:00:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:11.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:11.905501+0000) 2022-09-08T08:00:12.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:11.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:11.761458+0000) 2022-09-08T08:00:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:12 smithi055 ceph-mon[97033]: pgmap v20172: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:00:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:12 smithi059 ceph-mon[97450]: pgmap v20172: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:00:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:12.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:12.905733+0000) 2022-09-08T08:00:13.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:12.759+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:12.761614+0000) 2022-09-08T08:00:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:13.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:13.761793+0000) 2022-09-08T08:00:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:13.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:13.905966+0000) 2022-09-08T08:00:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:14 smithi059 ceph-mon[97450]: pgmap v20173: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:00:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:14 smithi055 ceph-mon[97033]: pgmap v20173: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:00:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:14.904+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:14.906123+0000) 2022-09-08T08:00:15.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:14.760+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:14.761970+0000) 2022-09-08T08:00:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:15.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:15.906357+0000) 2022-09-08T08:00:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:15.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:15.762178+0000) 2022-09-08T08:00:16.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:16 smithi055 ceph-mon[97033]: pgmap v20174: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:00:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:16 smithi059 ceph-mon[97450]: pgmap v20174: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:00:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:16.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:16.762320+0000) 2022-09-08T08:00:17.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:16.905+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:16.906562+0000) 2022-09-08T08:00:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:17.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:17.906805+0000) 2022-09-08T08:00:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:17.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:17.762475+0000) 2022-09-08T08:00:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:18 smithi055 ceph-mon[97033]: pgmap v20175: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:00:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:18 smithi055 ceph-mon[97033]: osdmap e1338: 12 total, 11 up, 11 in 2022-09-08T08:00:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:18 smithi059 ceph-mon[97450]: pgmap v20175: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:00:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:18 smithi059 ceph-mon[97450]: osdmap e1338: 12 total, 11 up, 11 in 2022-09-08T08:00:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:18.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:18.906944+0000) 2022-09-08T08:00:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:18.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:18.762682+0000) 2022-09-08T08:00:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:19.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:19.907137+0000) 2022-09-08T08:00:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:19.761+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:19.762826+0000) 2022-09-08T08:00:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:20 smithi055 ceph-mon[97033]: pgmap v20177: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T08:00:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:20 smithi059 ceph-mon[97450]: pgmap v20177: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T08:00:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:20.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:20.907346+0000) 2022-09-08T08:00:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:20.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:20.762976+0000) 2022-09-08T08:00:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:21.906+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:21.907560+0000) 2022-09-08T08:00:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:21.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:21.763189+0000) 2022-09-08T08:00:22.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:22 smithi055 ceph-mon[97033]: pgmap v20178: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:00:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:22 smithi059 ceph-mon[97450]: pgmap v20178: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:00:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:22.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:22.907802+0000) 2022-09-08T08:00:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:22.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:22.763331+0000) 2022-09-08T08:00:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:23.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:23.908028+0000) 2022-09-08T08:00:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:23.762+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:23.763552+0000) 2022-09-08T08:00:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:24 smithi059 ceph-mon[97450]: pgmap v20179: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:00:24.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:00:24.287+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:00:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:24 smithi055 ceph-mon[97033]: pgmap v20179: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:00:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:24.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:24.908240+0000) 2022-09-08T08:00:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:24.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:24.763798+0000) 2022-09-08T08:00:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:25.907+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:25.908432+0000) 2022-09-08T08:00:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:25.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:25.764044+0000) 2022-09-08T08:00:26.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:26 smithi055 ceph-mon[97033]: pgmap v20180: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:00:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:26 smithi059 ceph-mon[97450]: pgmap v20180: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:00:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:26.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:26.908678+0000) 2022-09-08T08:00:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:26.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:26.764230+0000) 2022-09-08T08:00:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:27.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:27.908916+0000) 2022-09-08T08:00:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:27.763+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:27.764461+0000) 2022-09-08T08:00:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:28 smithi055 ceph-mon[97033]: pgmap v20181: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:00:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:28 smithi059 ceph-mon[97450]: pgmap v20181: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:00:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:28.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:28.909154+0000) 2022-09-08T08:00:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:28.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:28.764629+0000) 2022-09-08T08:00:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:29.908+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:29.909402+0000) 2022-09-08T08:00:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:29.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:29.764818+0000) 2022-09-08T08:00:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:30 smithi055 ceph-mon[97033]: pgmap v20182: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T08:00:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:30 smithi059 ceph-mon[97450]: pgmap v20182: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T08:00:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:30.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:30.909676+0000) 2022-09-08T08:00:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:30.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:30.765088+0000) 2022-09-08T08:00:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:31.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:31.909897+0000) 2022-09-08T08:00:32.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:31.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:31.765251+0000) 2022-09-08T08:00:32.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:32 smithi055 ceph-mon[97033]: pgmap v20183: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:00:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:32 smithi059 ceph-mon[97450]: pgmap v20183: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:00:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:32.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:32.910136+0000) 2022-09-08T08:00:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:32.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:32.765484+0000) 2022-09-08T08:00:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:33.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:33.910328+0000) 2022-09-08T08:00:34.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:33.764+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:33.765612+0000) 2022-09-08T08:00:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:34 smithi059 ceph-mon[97450]: pgmap v20184: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:34 smithi055 ceph-mon[97033]: pgmap v20184: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:34.909+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:34.910559+0000) 2022-09-08T08:00:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:34.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:34.765778+0000) 2022-09-08T08:00:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:35.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:35.910810+0000) 2022-09-08T08:00:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:35.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:35.765993+0000) 2022-09-08T08:00:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:36 smithi055 ceph-mon[97033]: pgmap v20185: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:00:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:36 smithi059 ceph-mon[97450]: pgmap v20185: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:00:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:36.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:36.910954+0000) 2022-09-08T08:00:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:36.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:36.766221+0000) 2022-09-08T08:00:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:37.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:37.911087+0000) 2022-09-08T08:00:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:37.765+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:37.766426+0000) 2022-09-08T08:00:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:38 smithi055 ceph-mon[97033]: pgmap v20186: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:38 smithi059 ceph-mon[97450]: pgmap v20186: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:38.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:38.766605+0000) 2022-09-08T08:00:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:38.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:38.911294+0000) 2022-09-08T08:00:39.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:00:39.290+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:00:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:39.910+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:39.911415+0000) 2022-09-08T08:00:40.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:39.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:39.766732+0000) 2022-09-08T08:00:40.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:40 smithi055 ceph-mon[97033]: pgmap v20187: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:40 smithi059 ceph-mon[97450]: pgmap v20187: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:40.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:40.911629+0000) 2022-09-08T08:00:41.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:40.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:40.766873+0000) 2022-09-08T08:00:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:41.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:41.911882+0000) 2022-09-08T08:00:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:41.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:41.767123+0000) 2022-09-08T08:00:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:42 smithi055 ceph-mon[97033]: pgmap v20188: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:00:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:42 smithi059 ceph-mon[97450]: pgmap v20188: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:00:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:42.911+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:42.912112+0000) 2022-09-08T08:00:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:42.766+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:42.767309+0000) 2022-09-08T08:00:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:43.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:43.912326+0000) 2022-09-08T08:00:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:43.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:43.767512+0000) 2022-09-08T08:00:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:44 smithi059 ceph-mon[97450]: pgmap v20189: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:44 smithi055 ceph-mon[97033]: pgmap v20189: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:44.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:44.912540+0000) 2022-09-08T08:00:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:44.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:44.767660+0000) 2022-09-08T08:00:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:45.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:45.767833+0000) 2022-09-08T08:00:46.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:45.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:45.912768+0000) 2022-09-08T08:00:46.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:46 smithi055 ceph-mon[97033]: pgmap v20190: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:00:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:46 smithi059 ceph-mon[97450]: pgmap v20190: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:00:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:46.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:46.913003+0000) 2022-09-08T08:00:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:46.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:46.768005+0000) 2022-09-08T08:00:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:47.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:47.913171+0000) 2022-09-08T08:00:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:47.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:47.768176+0000) 2022-09-08T08:00:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:48 smithi055 ceph-mon[97033]: pgmap v20191: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:48 smithi059 ceph-mon[97450]: pgmap v20191: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:48.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:48.913400+0000) 2022-09-08T08:00:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:48.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:48.768415+0000) 2022-09-08T08:00:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:49.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:49.913629+0000) 2022-09-08T08:00:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:49.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:49.768697+0000) 2022-09-08T08:00:50.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:50 smithi055 ceph-mon[97033]: pgmap v20192: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:50 smithi059 ceph-mon[97450]: pgmap v20192: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:50.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:50.913881+0000) 2022-09-08T08:00:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:50.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:50.768851+0000) 2022-09-08T08:00:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:51.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:51.914120+0000) 2022-09-08T08:00:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:51.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:51.769000+0000) 2022-09-08T08:00:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:52 smithi055 ceph-mon[97033]: pgmap v20193: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:00:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:52 smithi059 ceph-mon[97450]: pgmap v20193: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:00:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:52.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:52.914347+0000) 2022-09-08T08:00:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:52.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:52.769156+0000) 2022-09-08T08:00:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:53.912+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:53.914587+0000) 2022-09-08T08:00:54.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:53.767+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:53.769346+0000) 2022-09-08T08:00:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:54 smithi059 ceph-mon[97450]: pgmap v20194: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:54.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:00:54.301+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:00:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:54 smithi055 ceph-mon[97033]: pgmap v20194: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:54.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:54.914831+0000) 2022-09-08T08:00:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:54.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:54.769548+0000) 2022-09-08T08:00:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:55.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:55.914989+0000) 2022-09-08T08:00:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:55.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:55.769742+0000) 2022-09-08T08:00:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:56 smithi059 ceph-mon[97450]: pgmap v20195: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:00:56.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:00:56.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:00:56.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:00:56.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:00:56.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:56 smithi055 ceph-mon[97033]: pgmap v20195: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:00:56.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:00:56.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:00:56.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:00:56.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:00:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:56.913+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:56.915178+0000) 2022-09-08T08:00:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:56.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:56.769917+0000) 2022-09-08T08:00:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:57.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:57.915323+0000) 2022-09-08T08:00:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:57.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:57.770072+0000) 2022-09-08T08:00:58.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:00:58 smithi055 ceph-mon[97033]: pgmap v20196: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:00:58 smithi059 ceph-mon[97450]: pgmap v20196: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:00:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:58.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:58.915519+0000) 2022-09-08T08:00:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:58.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:58.770253+0000) 2022-09-08T08:01:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:00:59.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:59.915705+0000) 2022-09-08T08:01:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:00:59.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:00:59.770390+0000) 2022-09-08T08:01:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:00 smithi055 ceph-mon[97033]: pgmap v20197: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:01:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:00 smithi059 ceph-mon[97450]: pgmap v20197: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:01:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:00.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:00.915849+0000) 2022-09-08T08:01:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:00.768+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:00.770535+0000) 2022-09-08T08:01:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:01.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:01.916018+0000) 2022-09-08T08:01:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:01.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:01.770709+0000) 2022-09-08T08:01:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:02 smithi055 ceph-mon[97033]: pgmap v20198: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:01:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:02 smithi059 ceph-mon[97450]: pgmap v20198: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:01:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:02.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:02.770870+0000) 2022-09-08T08:01:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:02.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:02.916206+0000) 2022-09-08T08:01:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:03.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:03.916392+0000) 2022-09-08T08:01:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:03.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:03.770991+0000) 2022-09-08T08:01:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:04 smithi059 ceph-mon[97450]: pgmap v20199: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:01:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:04 smithi055 ceph-mon[97033]: pgmap v20199: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:01:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:04.914+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:04.916545+0000) 2022-09-08T08:01:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:04.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:04.771128+0000) 2022-09-08T08:01:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:05.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:05.916784+0000) 2022-09-08T08:01:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:05.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:05.771278+0000) 2022-09-08T08:01:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:06 smithi055 ceph-mon[97033]: pgmap v20200: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:01:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:01:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:06 smithi059 ceph-mon[97450]: pgmap v20200: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:01:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:01:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:06.915+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:06.916921+0000) 2022-09-08T08:01:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:06.769+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:06.771444+0000) 2022-09-08T08:01:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:01:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:01:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:07.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:07.917138+0000) 2022-09-08T08:01:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:07.770+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:07.771608+0000) 2022-09-08T08:01:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:08 smithi055 ceph-mon[97033]: pgmap v20201: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:01:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:08 smithi059 ceph-mon[97450]: pgmap v20201: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:01:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:08.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:08.917352+0000) 2022-09-08T08:01:09.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:08.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:08.771781+0000) 2022-09-08T08:01:09.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:01:09.317+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:01:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:09.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:09.771996+0000) 2022-09-08T08:01:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:09.916+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:09.917527+0000) 2022-09-08T08:01:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:10 smithi055 ceph-mon[97033]: pgmap v20202: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:01:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:10 smithi055 ceph-mon[97033]: osdmap e1339: 12 total, 11 up, 11 in 2022-09-08T08:01:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:10 smithi059 ceph-mon[97450]: pgmap v20202: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:01:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:10 smithi059 ceph-mon[97450]: osdmap e1339: 12 total, 11 up, 11 in 2022-09-08T08:01:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:10.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:10.917760+0000) 2022-09-08T08:01:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:10.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:10.772146+0000) 2022-09-08T08:01:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:11.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:11.918009+0000) 2022-09-08T08:01:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:11.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:11.772313+0000) 2022-09-08T08:01:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:12 smithi055 ceph-mon[97033]: pgmap v20204: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:01:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:12 smithi059 ceph-mon[97450]: pgmap v20204: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:01:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:12.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:12.918273+0000) 2022-09-08T08:01:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:12.771+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:12.772466+0000) 2022-09-08T08:01:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:13.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:13.772653+0000) 2022-09-08T08:01:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:13.917+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:13.918542+0000) 2022-09-08T08:01:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:14 smithi059 ceph-mon[97450]: pgmap v20205: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:01:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:14 smithi055 ceph-mon[97033]: pgmap v20205: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:01:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:14.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:14.772852+0000) 2022-09-08T08:01:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:14.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:14.918734+0000) 2022-09-08T08:01:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:15.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:15.918997+0000) 2022-09-08T08:01:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:15.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:15.773026+0000) 2022-09-08T08:01:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:16 smithi055 ceph-mon[97033]: pgmap v20206: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:01:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:16 smithi059 ceph-mon[97450]: pgmap v20206: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:01:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:16.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:16.919211+0000) 2022-09-08T08:01:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:16.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:16.773238+0000) 2022-09-08T08:01:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:17.918+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:17.919445+0000) 2022-09-08T08:01:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:17.772+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:17.773435+0000) 2022-09-08T08:01:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:18 smithi055 ceph-mon[97033]: pgmap v20207: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:01:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:18 smithi055 ceph-mon[97033]: osdmap e1340: 12 total, 11 up, 11 in 2022-09-08T08:01:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:18 smithi059 ceph-mon[97450]: pgmap v20207: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:01:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:18 smithi059 ceph-mon[97450]: osdmap e1340: 12 total, 11 up, 11 in 2022-09-08T08:01:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:18.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:18.919702+0000) 2022-09-08T08:01:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:18.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:18.773672+0000) 2022-09-08T08:01:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:19.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:19.919845+0000) 2022-09-08T08:01:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:19.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:19.773857+0000) 2022-09-08T08:01:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:20 smithi055 ceph-mon[97033]: pgmap v20209: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T08:01:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:20 smithi059 ceph-mon[97450]: pgmap v20209: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T08:01:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:20.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:20.920035+0000) 2022-09-08T08:01:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:20.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:20.774091+0000) 2022-09-08T08:01:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:21.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:21.774261+0000) 2022-09-08T08:01:22.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:21.919+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:21.920258+0000) 2022-09-08T08:01:22.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:22 smithi055 ceph-mon[97033]: pgmap v20210: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:01:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:22 smithi059 ceph-mon[97450]: pgmap v20210: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:01:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:22.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:22.920490+0000) 2022-09-08T08:01:23.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:22.773+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:22.774491+0000) 2022-09-08T08:01:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:23.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:23.920756+0000) 2022-09-08T08:01:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:23.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:23.774719+0000) 2022-09-08T08:01:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:24 smithi059 ceph-mon[97450]: pgmap v20211: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:01:24.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:01:24.333+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:01:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:24 smithi055 ceph-mon[97033]: pgmap v20211: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:01:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:24.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:24.920991+0000) 2022-09-08T08:01:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:24.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:24.774871+0000) 2022-09-08T08:01:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:25.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:25.921149+0000) 2022-09-08T08:01:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:25.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:25.775003+0000) 2022-09-08T08:01:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:26 smithi055 ceph-mon[97033]: pgmap v20212: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:01:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:26 smithi059 ceph-mon[97450]: pgmap v20212: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:01:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:26.920+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:26.921343+0000) 2022-09-08T08:01:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:26.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:26.775142+0000) 2022-09-08T08:01:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:27.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:27.921555+0000) 2022-09-08T08:01:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:27.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:27.775326+0000) 2022-09-08T08:01:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:28 smithi055 ceph-mon[97033]: pgmap v20213: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:01:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:28 smithi059 ceph-mon[97450]: pgmap v20213: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:01:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:28.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:28.921786+0000) 2022-09-08T08:01:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:28.774+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:28.775571+0000) 2022-09-08T08:01:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:29.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:29.921960+0000) 2022-09-08T08:01:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:29.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:29.775727+0000) 2022-09-08T08:01:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:30 smithi055 ceph-mon[97033]: pgmap v20214: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T08:01:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:30 smithi059 ceph-mon[97450]: pgmap v20214: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T08:01:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:30.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:30.922183+0000) 2022-09-08T08:01:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:30.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:30.775887+0000) 2022-09-08T08:01:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:31.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:31.922402+0000) 2022-09-08T08:01:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:31.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:31.776037+0000) 2022-09-08T08:01:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:32 smithi055 ceph-mon[97033]: pgmap v20215: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:01:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:32 smithi059 ceph-mon[97450]: pgmap v20215: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:01:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:32.921+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:32.922552+0000) 2022-09-08T08:01:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:32.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:32.776205+0000) 2022-09-08T08:01:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:33.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:33.776423+0000) 2022-09-08T08:01:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:33.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:33.922777+0000) 2022-09-08T08:01:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:34 smithi059 ceph-mon[97450]: pgmap v20216: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:01:34.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:34 smithi055 ceph-mon[97033]: pgmap v20216: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:01:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:34.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:34.922975+0000) 2022-09-08T08:01:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:34.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:34.776614+0000) 2022-09-08T08:01:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:35.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:35.776851+0000) 2022-09-08T08:01:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:35.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:35.923152+0000) 2022-09-08T08:01:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:36 smithi055 ceph-mon[97033]: pgmap v20217: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:01:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:36 smithi059 ceph-mon[97450]: pgmap v20217: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:01:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:36.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:36.923359+0000) 2022-09-08T08:01:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:36.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:36.777089+0000) 2022-09-08T08:01:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:37.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:37.923598+0000) 2022-09-08T08:01:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:37.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:37.777340+0000) 2022-09-08T08:01:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:38 smithi055 ceph-mon[97033]: pgmap v20218: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:01:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:38 smithi059 ceph-mon[97450]: pgmap v20218: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:01:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:38.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:38.923906+0000) 2022-09-08T08:01:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:38.775+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:38.777537+0000) 2022-09-08T08:01:39.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:01:39.340+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:01:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:39.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:39.924165+0000) 2022-09-08T08:01:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:39.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:39.777695+0000) 2022-09-08T08:01:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:40 smithi055 ceph-mon[97033]: pgmap v20219: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:01:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:40 smithi059 ceph-mon[97450]: pgmap v20219: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:01:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:40.922+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:40.924354+0000) 2022-09-08T08:01:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:40.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:40.777897+0000) 2022-09-08T08:01:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:41.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:41.924571+0000) 2022-09-08T08:01:42.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:41.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:41.778048+0000) 2022-09-08T08:01:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:42 smithi055 ceph-mon[97033]: pgmap v20220: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:01:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:42 smithi059 ceph-mon[97450]: pgmap v20220: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:01:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:42.923+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:42.924785+0000) 2022-09-08T08:01:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:42.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:42.778272+0000) 2022-09-08T08:01:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:43.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:43.925027+0000) 2022-09-08T08:01:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:43.776+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:43.778399+0000) 2022-09-08T08:01:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:44 smithi055 ceph-mon[97033]: pgmap v20221: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:01:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:44 smithi059 ceph-mon[97450]: pgmap v20221: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:01:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:44.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:44.925289+0000) 2022-09-08T08:01:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:44.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:44.778558+0000) 2022-09-08T08:01:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:45.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:45.778764+0000) 2022-09-08T08:01:46.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:45.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:45.925566+0000) 2022-09-08T08:01:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:46 smithi055 ceph-mon[97033]: pgmap v20222: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:01:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:46 smithi059 ceph-mon[97450]: pgmap v20222: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:01:47.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:46.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:46.778961+0000) 2022-09-08T08:01:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:46.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:46.925856+0000) 2022-09-08T08:01:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:47.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:47.926066+0000) 2022-09-08T08:01:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:47.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:47.779191+0000) 2022-09-08T08:01:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:48 smithi055 ceph-mon[97033]: pgmap v20223: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:01:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:48 smithi059 ceph-mon[97450]: pgmap v20223: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:01:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:48.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:48.926303+0000) 2022-09-08T08:01:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:48.777+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:48.779358+0000) 2022-09-08T08:01:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:49.924+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:49.926540+0000) 2022-09-08T08:01:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:49.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:49.779599+0000) 2022-09-08T08:01:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:50 smithi055 ceph-mon[97033]: pgmap v20224: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:01:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:50 smithi059 ceph-mon[97450]: pgmap v20224: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:01:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:50.925+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:50.926696+0000) 2022-09-08T08:01:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:50.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:50.779789+0000) 2022-09-08T08:01:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:51.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:51.926937+0000) 2022-09-08T08:01:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:51.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:51.779926+0000) 2022-09-08T08:01:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:52 smithi055 ceph-mon[97033]: pgmap v20225: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:01:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:52 smithi059 ceph-mon[97450]: pgmap v20225: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:01:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:52.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:52.927198+0000) 2022-09-08T08:01:53.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:52.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:52.780098+0000) 2022-09-08T08:01:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:53.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:53.927453+0000) 2022-09-08T08:01:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:53.778+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:53.780257+0000) 2022-09-08T08:01:54.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:01:54.347+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:01:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:54 smithi055 ceph-mon[97033]: pgmap v20226: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:01:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:54 smithi059 ceph-mon[97450]: pgmap v20226: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:01:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:54.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:54.780444+0000) 2022-09-08T08:01:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:54.926+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:54.927612+0000) 2022-09-08T08:01:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:55.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:55.927800+0000) 2022-09-08T08:01:56.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:55.779+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:55.780627+0000) 2022-09-08T08:01:56.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:56 smithi055 ceph-mon[97033]: pgmap v20227: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:01:56.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:01:56.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:01:56.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:56 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:01:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:56 smithi059 ceph-mon[97450]: pgmap v20227: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:01:56.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:01:56.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:01:56.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:56 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:01:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:56.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:56.927976+0000) 2022-09-08T08:01:57.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:56.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:56.780783+0000) 2022-09-08T08:01:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:57.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:57.928150+0000) 2022-09-08T08:01:58.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:57.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:57.780928+0000) 2022-09-08T08:01:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:01:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:01:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:58.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:58.928368+0000) 2022-09-08T08:01:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:58.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:58.781103+0000) 2022-09-08T08:01:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:01:58 smithi055 ceph-mon[97033]: pgmap v20228: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:01:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:01:58 smithi059 ceph-mon[97450]: pgmap v20228: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:01:59.927+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:59.928529+0000) 2022-09-08T08:02:00.164 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:01:59.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:01:59.781318+0000) 2022-09-08T08:02:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:00.780+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:00.781521+0000) 2022-09-08T08:02:01.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:00.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:00.928665+0000) 2022-09-08T08:02:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:00 smithi055 ceph-mon[97033]: pgmap v20229: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:00 smithi059 ceph-mon[97450]: pgmap v20229: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:01.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:01.928841+0000) 2022-09-08T08:02:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:01.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:01.781668+0000) 2022-09-08T08:02:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:02.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:02.929069+0000) 2022-09-08T08:02:03.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:02.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:02.781865+0000) 2022-09-08T08:02:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:02 smithi055 ceph-mon[97033]: pgmap v20230: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:02:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:02 smithi059 ceph-mon[97450]: pgmap v20230: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:02:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:03.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:03.781994+0000) 2022-09-08T08:02:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:03.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:03.929287+0000) 2022-09-08T08:02:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:04 smithi055 ceph-mon[97033]: pgmap v20231: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:04 smithi059 ceph-mon[97450]: pgmap v20231: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:04.928+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:04.929473+0000) 2022-09-08T08:02:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:04.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:04.782143+0000) 2022-09-08T08:02:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:05.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:05.929671+0000) 2022-09-08T08:02:06.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:05.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:05.782321+0000) 2022-09-08T08:02:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:06 smithi055 ceph-mon[97033]: pgmap v20232: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:02:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:02:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:06 smithi059 ceph-mon[97450]: pgmap v20232: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:02:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:02:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:06.781+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:06.782562+0000) 2022-09-08T08:02:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:06.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:06.929883+0000) 2022-09-08T08:02:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:02:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:02:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:07.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:07.782776+0000) 2022-09-08T08:02:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:07.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:07.930088+0000) 2022-09-08T08:02:08.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:08 smithi055 ceph-mon[97033]: pgmap v20233: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:02:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:08 smithi059 ceph-mon[97450]: pgmap v20233: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:02:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:08.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:08.930269+0000) 2022-09-08T08:02:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:08.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:08.782939+0000) 2022-09-08T08:02:09.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:02:09.361+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:02:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:09.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:09.783109+0000) 2022-09-08T08:02:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:09.929+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:09.930440+0000) 2022-09-08T08:02:10.933 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:10.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:10.783282+0000) 2022-09-08T08:02:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:10 smithi055 ceph-mon[97033]: pgmap v20234: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:02:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:10 smithi055 ceph-mon[97033]: osdmap e1341: 12 total, 11 up, 11 in 2022-09-08T08:02:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:10 smithi059 ceph-mon[97450]: pgmap v20234: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:02:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:10 smithi059 ceph-mon[97450]: osdmap e1341: 12 total, 11 up, 11 in 2022-09-08T08:02:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:10.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:10.930728+0000) 2022-09-08T08:02:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:11.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:11.930865+0000) 2022-09-08T08:02:12.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:11.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:11.783452+0000) 2022-09-08T08:02:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:12 smithi055 ceph-mon[97033]: pgmap v20236: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:02:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:12 smithi059 ceph-mon[97450]: pgmap v20236: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:02:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:12.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:12.931034+0000) 2022-09-08T08:02:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:12.782+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:12.783600+0000) 2022-09-08T08:02:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:13.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:13.931188+0000) 2022-09-08T08:02:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:13.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:13.783750+0000) 2022-09-08T08:02:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:14 smithi055 ceph-mon[97033]: pgmap v20237: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:02:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:14 smithi059 ceph-mon[97450]: pgmap v20237: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:02:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:14.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:14.931369+0000) 2022-09-08T08:02:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:14.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:14.783964+0000) 2022-09-08T08:02:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:15.930+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:15.931585+0000) 2022-09-08T08:02:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:15.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:15.784116+0000) 2022-09-08T08:02:16.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:16 smithi055 ceph-mon[97033]: pgmap v20238: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:02:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:16 smithi059 ceph-mon[97450]: pgmap v20238: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:02:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:16.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:16.931790+0000) 2022-09-08T08:02:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:16.783+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:16.784347+0000) 2022-09-08T08:02:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:17.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:17.931984+0000) 2022-09-08T08:02:18.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:17.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:17.784597+0000) 2022-09-08T08:02:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:18 smithi055 ceph-mon[97033]: pgmap v20239: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:02:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:18 smithi055 ceph-mon[97033]: osdmap e1342: 12 total, 11 up, 11 in 2022-09-08T08:02:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:18 smithi059 ceph-mon[97450]: pgmap v20239: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:02:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:18 smithi059 ceph-mon[97450]: osdmap e1342: 12 total, 11 up, 11 in 2022-09-08T08:02:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:18.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:18.932145+0000) 2022-09-08T08:02:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:18.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:18.784746+0000) 2022-09-08T08:02:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:19.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:19.932302+0000) 2022-09-08T08:02:20.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:19.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:19.784973+0000) 2022-09-08T08:02:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:20 smithi055 ceph-mon[97033]: pgmap v20241: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T08:02:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:20 smithi059 ceph-mon[97450]: pgmap v20241: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T08:02:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:20.931+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:20.932513+0000) 2022-09-08T08:02:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:20.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:20.785213+0000) 2022-09-08T08:02:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:21.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:21.932751+0000) 2022-09-08T08:02:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:21.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:21.785479+0000) 2022-09-08T08:02:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:22 smithi055 ceph-mon[97033]: pgmap v20242: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:02:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:22 smithi059 ceph-mon[97450]: pgmap v20242: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:02:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:22.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:22.933002+0000) 2022-09-08T08:02:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:22.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:22.785694+0000) 2022-09-08T08:02:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:23.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:23.933264+0000) 2022-09-08T08:02:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:23.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:23.785817+0000) 2022-09-08T08:02:24.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:02:24.367+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:02:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:24 smithi055 ceph-mon[97033]: pgmap v20243: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:02:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:24 smithi059 ceph-mon[97450]: pgmap v20243: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:02:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:24.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:24.933485+0000) 2022-09-08T08:02:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:24.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:24.785971+0000) 2022-09-08T08:02:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:25.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:25.933668+0000) 2022-09-08T08:02:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:25.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:25.786137+0000) 2022-09-08T08:02:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:26 smithi055 ceph-mon[97033]: pgmap v20244: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:02:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:26 smithi059 ceph-mon[97450]: pgmap v20244: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:02:27.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:26.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:26.786272+0000) 2022-09-08T08:02:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:26.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:26.933910+0000) 2022-09-08T08:02:28.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:27.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:27.786413+0000) 2022-09-08T08:02:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:27.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:27.934094+0000) 2022-09-08T08:02:28.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:28 smithi055 ceph-mon[97033]: pgmap v20245: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:02:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:28 smithi059 ceph-mon[97450]: pgmap v20245: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:02:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:28.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:28.934329+0000) 2022-09-08T08:02:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:28.784+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:28.786555+0000) 2022-09-08T08:02:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:29.932+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:29.934581+0000) 2022-09-08T08:02:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:29.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:29.786709+0000) 2022-09-08T08:02:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:30 smithi055 ceph-mon[97033]: pgmap v20246: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:02:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:30 smithi059 ceph-mon[97450]: pgmap v20246: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:02:31.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:30.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:30.786877+0000) 2022-09-08T08:02:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:30.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:30.934857+0000) 2022-09-08T08:02:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:31.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:31.935108+0000) 2022-09-08T08:02:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:31.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:31.787092+0000) 2022-09-08T08:02:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:32 smithi055 ceph-mon[97033]: pgmap v20247: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:02:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:32 smithi059 ceph-mon[97450]: pgmap v20247: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:02:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:32.933+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:32.935242+0000) 2022-09-08T08:02:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:32.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:32.787330+0000) 2022-09-08T08:02:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:33.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:33.935472+0000) 2022-09-08T08:02:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:33.785+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:33.787521+0000) 2022-09-08T08:02:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:34 smithi055 ceph-mon[97033]: pgmap v20248: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:34 smithi059 ceph-mon[97450]: pgmap v20248: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:34.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:34.787683+0000) 2022-09-08T08:02:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:34.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:34.935690+0000) 2022-09-08T08:02:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:35.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:35.935920+0000) 2022-09-08T08:02:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:35.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:35.787890+0000) 2022-09-08T08:02:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:36 smithi055 ceph-mon[97033]: pgmap v20249: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:02:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:36 smithi059 ceph-mon[97450]: pgmap v20249: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:02:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:36.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:36.936157+0000) 2022-09-08T08:02:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:36.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:36.788118+0000) 2022-09-08T08:02:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:37.934+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:37.936350+0000) 2022-09-08T08:02:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:37.786+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:37.788331+0000) 2022-09-08T08:02:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:38 smithi055 ceph-mon[97033]: pgmap v20250: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:38 smithi059 ceph-mon[97450]: pgmap v20250: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:38.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:38.936522+0000) 2022-09-08T08:02:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:38.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:38.788478+0000) 2022-09-08T08:02:39.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:02:39.367+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:02:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:39.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:39.936749+0000) 2022-09-08T08:02:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:39.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:39.788706+0000) 2022-09-08T08:02:40.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:40 smithi055 ceph-mon[97033]: pgmap v20251: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:40 smithi059 ceph-mon[97450]: pgmap v20251: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:40.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:40.936912+0000) 2022-09-08T08:02:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:40.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:40.788888+0000) 2022-09-08T08:02:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:41.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:41.937135+0000) 2022-09-08T08:02:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:41.787+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:41.789058+0000) 2022-09-08T08:02:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:42 smithi055 ceph-mon[97033]: pgmap v20252: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:02:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:42 smithi059 ceph-mon[97450]: pgmap v20252: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:02:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:42.935+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:42.937362+0000) 2022-09-08T08:02:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:42.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:42.789241+0000) 2022-09-08T08:02:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:43.936+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:43.937537+0000) 2022-09-08T08:02:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:43.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:43.789431+0000) 2022-09-08T08:02:44.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:44 smithi055 ceph-mon[97033]: pgmap v20253: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:44 smithi059 ceph-mon[97450]: pgmap v20253: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:44.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:44.937683+0000) 2022-09-08T08:02:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:44.788+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:44.789601+0000) 2022-09-08T08:02:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:45.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:45.937853+0000) 2022-09-08T08:02:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:45.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:45.789803+0000) 2022-09-08T08:02:46.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:46 smithi055 ceph-mon[97033]: pgmap v20254: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:02:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:46 smithi059 ceph-mon[97450]: pgmap v20254: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:02:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:46.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:46.938073+0000) 2022-09-08T08:02:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:46.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:46.790023+0000) 2022-09-08T08:02:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:47.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:47.790263+0000) 2022-09-08T08:02:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:47.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:47.938277+0000) 2022-09-08T08:02:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:48 smithi055 ceph-mon[97033]: pgmap v20255: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:48 smithi059 ceph-mon[97450]: pgmap v20255: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:48.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:48.938396+0000) 2022-09-08T08:02:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:48.789+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:48.790490+0000) 2022-09-08T08:02:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:49.937+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:49.938548+0000) 2022-09-08T08:02:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:49.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:49.790670+0000) 2022-09-08T08:02:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:50 smithi055 ceph-mon[97033]: pgmap v20256: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:50 smithi059 ceph-mon[97450]: pgmap v20256: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:50.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:50.938753+0000) 2022-09-08T08:02:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:50.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:50.790855+0000) 2022-09-08T08:02:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:51.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:51.938982+0000) 2022-09-08T08:02:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:51.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:51.791101+0000) 2022-09-08T08:02:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:52 smithi055 ceph-mon[97033]: pgmap v20257: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:02:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:52 smithi059 ceph-mon[97450]: pgmap v20257: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:02:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:52.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:52.939182+0000) 2022-09-08T08:02:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:52.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:52.791281+0000) 2022-09-08T08:02:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:53.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:53.791446+0000) 2022-09-08T08:02:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:53.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:53.939402+0000) 2022-09-08T08:02:54.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:02:54.384+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:02:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:54 smithi055 ceph-mon[97033]: pgmap v20258: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:54 smithi059 ceph-mon[97450]: pgmap v20258: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:54.938+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:54.939625+0000) 2022-09-08T08:02:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:54.790+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:54.791594+0000) 2022-09-08T08:02:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:55.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:55.939879+0000) 2022-09-08T08:02:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:55.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:55.791736+0000) 2022-09-08T08:02:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:56 smithi055 ceph-mon[97033]: pgmap v20259: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:02:57.021 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:56 smithi059 ceph-mon[97450]: pgmap v20259: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:02:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:56.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:56.940119+0000) 2022-09-08T08:02:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:56.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:56.791968+0000) 2022-09-08T08:02:57.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:02:57.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:02:57.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:02:57.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:02:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:02:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:02:58.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:02:58.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:02:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:57.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:57.940371+0000) 2022-09-08T08:02:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:57.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:57.792161+0000) 2022-09-08T08:02:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:02:58 smithi055 ceph-mon[97033]: pgmap v20260: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:02:58 smithi059 ceph-mon[97450]: pgmap v20260: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:02:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:58.939+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:58.940600+0000) 2022-09-08T08:02:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:58.791+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:58.792409+0000) 2022-09-08T08:03:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:02:59.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:59.940822+0000) 2022-09-08T08:03:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:02:59.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:02:59.792717+0000) 2022-09-08T08:03:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:00 smithi055 ceph-mon[97033]: pgmap v20261: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:00 smithi059 ceph-mon[97450]: pgmap v20261: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:00.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:00.940986+0000) 2022-09-08T08:03:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:00.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:00.792893+0000) 2022-09-08T08:03:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:01.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:01.941166+0000) 2022-09-08T08:03:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:01.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:01.793040+0000) 2022-09-08T08:03:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:02 smithi055 ceph-mon[97033]: pgmap v20262: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:03:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:02 smithi059 ceph-mon[97450]: pgmap v20262: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:03:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:02.940+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:02.941406+0000) 2022-09-08T08:03:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:02.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:02.793167+0000) 2022-09-08T08:03:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:03.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:03.941717+0000) 2022-09-08T08:03:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:03.792+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:03.793332+0000) 2022-09-08T08:03:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:04 smithi055 ceph-mon[97033]: pgmap v20263: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:04 smithi059 ceph-mon[97450]: pgmap v20263: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:04.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:04.793541+0000) 2022-09-08T08:03:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:04.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:04.941904+0000) 2022-09-08T08:03:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:05.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:05.793784+0000) 2022-09-08T08:03:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:05.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:05.942160+0000) 2022-09-08T08:03:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:06 smithi055 ceph-mon[97033]: pgmap v20264: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:03:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:03:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:06 smithi059 ceph-mon[97450]: pgmap v20264: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:03:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:03:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:06.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:06.793993+0000) 2022-09-08T08:03:07.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:06.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:06.942389+0000) 2022-09-08T08:03:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:03:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:03:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:07.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:07.794141+0000) 2022-09-08T08:03:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:07.941+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:07.942600+0000) 2022-09-08T08:03:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:08 smithi055 ceph-mon[97033]: pgmap v20265: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:03:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:08 smithi059 ceph-mon[97450]: pgmap v20265: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:03:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:08.793+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:08.794358+0000) 2022-09-08T08:03:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:08.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:08.942860+0000) 2022-09-08T08:03:09.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:03:09.397+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:03:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:09.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:09.943093+0000) 2022-09-08T08:03:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:09.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:09.794557+0000) 2022-09-08T08:03:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:10 smithi055 ceph-mon[97033]: pgmap v20266: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:03:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:10 smithi055 ceph-mon[97033]: osdmap e1343: 12 total, 11 up, 11 in 2022-09-08T08:03:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:10 smithi059 ceph-mon[97450]: pgmap v20266: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:03:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:10 smithi059 ceph-mon[97450]: osdmap e1343: 12 total, 11 up, 11 in 2022-09-08T08:03:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:10.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:10.943306+0000) 2022-09-08T08:03:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:10.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:10.794741+0000) 2022-09-08T08:03:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:11.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:11.943455+0000) 2022-09-08T08:03:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:11.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:11.794914+0000) 2022-09-08T08:03:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:12 smithi055 ceph-mon[97033]: pgmap v20268: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:03:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:12 smithi059 ceph-mon[97450]: pgmap v20268: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:03:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:12.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:12.943613+0000) 2022-09-08T08:03:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:12.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:12.795108+0000) 2022-09-08T08:03:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:13.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:13.943820+0000) 2022-09-08T08:03:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:13.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:13.795284+0000) 2022-09-08T08:03:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:14 smithi055 ceph-mon[97033]: pgmap v20269: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:03:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:14 smithi059 ceph-mon[97450]: pgmap v20269: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:03:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:14.942+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:14.943989+0000) 2022-09-08T08:03:15.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:14.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:14.795537+0000) 2022-09-08T08:03:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:15.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:15.944217+0000) 2022-09-08T08:03:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:15.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:15.795720+0000) 2022-09-08T08:03:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:16 smithi055 ceph-mon[97033]: pgmap v20270: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:03:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:16 smithi059 ceph-mon[97450]: pgmap v20270: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:03:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:16.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:16.944422+0000) 2022-09-08T08:03:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:16.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:16.795882+0000) 2022-09-08T08:03:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:17.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:17.944689+0000) 2022-09-08T08:03:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:17.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:17.796098+0000) 2022-09-08T08:03:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:18 smithi055 ceph-mon[97033]: pgmap v20271: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:03:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:18 smithi055 ceph-mon[97033]: osdmap e1344: 12 total, 11 up, 11 in 2022-09-08T08:03:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:18 smithi059 ceph-mon[97450]: pgmap v20271: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:03:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:18 smithi059 ceph-mon[97450]: osdmap e1344: 12 total, 11 up, 11 in 2022-09-08T08:03:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:18.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:18.944928+0000) 2022-09-08T08:03:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:18.794+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:18.796321+0000) 2022-09-08T08:03:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:19.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:19.945095+0000) 2022-09-08T08:03:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:19.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:19.796560+0000) 2022-09-08T08:03:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:20 smithi055 ceph-mon[97033]: pgmap v20273: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T08:03:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:20 smithi059 ceph-mon[97450]: pgmap v20273: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T08:03:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:20.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:20.945248+0000) 2022-09-08T08:03:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:20.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:20.796766+0000) 2022-09-08T08:03:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:21.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:21.945442+0000) 2022-09-08T08:03:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:21.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:21.796939+0000) 2022-09-08T08:03:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:22 smithi055 ceph-mon[97033]: pgmap v20274: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:03:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:22 smithi059 ceph-mon[97450]: pgmap v20274: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:03:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:22.943+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:22.945615+0000) 2022-09-08T08:03:23.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:22.795+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:22.797207+0000) 2022-09-08T08:03:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:23.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:23.945843+0000) 2022-09-08T08:03:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:23.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:23.797435+0000) 2022-09-08T08:03:24.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:03:24.398+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:03:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:24 smithi055 ceph-mon[97033]: pgmap v20275: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:03:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:24 smithi059 ceph-mon[97450]: pgmap v20275: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:03:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:24.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:24.946005+0000) 2022-09-08T08:03:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:24.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:24.797649+0000) 2022-09-08T08:03:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:25.944+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:25.946177+0000) 2022-09-08T08:03:26.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:25.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:25.797785+0000) 2022-09-08T08:03:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:26 smithi055 ceph-mon[97033]: pgmap v20276: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:03:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:26 smithi059 ceph-mon[97450]: pgmap v20276: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:03:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:26.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:26.946353+0000) 2022-09-08T08:03:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:26.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:26.797935+0000) 2022-09-08T08:03:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:27.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:27.946588+0000) 2022-09-08T08:03:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:27.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:27.798059+0000) 2022-09-08T08:03:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:28 smithi055 ceph-mon[97033]: pgmap v20277: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:03:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:28 smithi059 ceph-mon[97450]: pgmap v20277: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:03:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:28.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:28.946808+0000) 2022-09-08T08:03:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:28.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:28.798192+0000) 2022-09-08T08:03:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:29.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:29.946966+0000) 2022-09-08T08:03:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:29.796+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:29.798332+0000) 2022-09-08T08:03:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:30 smithi055 ceph-mon[97033]: pgmap v20278: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:03:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:30 smithi059 ceph-mon[97450]: pgmap v20278: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:03:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:30.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:30.947160+0000) 2022-09-08T08:03:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:30.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:30.798479+0000) 2022-09-08T08:03:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:31.945+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:31.947329+0000) 2022-09-08T08:03:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:31.797+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:31.798682+0000) 2022-09-08T08:03:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:32 smithi055 ceph-mon[97033]: pgmap v20279: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:03:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:32 smithi059 ceph-mon[97450]: pgmap v20279: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:03:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:32.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:32.798806+0000) 2022-09-08T08:03:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:32.946+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:32.947548+0000) 2022-09-08T08:03:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:33.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:33.947774+0000) 2022-09-08T08:03:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:33.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:33.798963+0000) 2022-09-08T08:03:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:34 smithi055 ceph-mon[97033]: pgmap v20280: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:34 smithi059 ceph-mon[97450]: pgmap v20280: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:34.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:34.947913+0000) 2022-09-08T08:03:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:34.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:34.799111+0000) 2022-09-08T08:03:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:35.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:35.948052+0000) 2022-09-08T08:03:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:35.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:35.799267+0000) 2022-09-08T08:03:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:36 smithi055 ceph-mon[97033]: pgmap v20281: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:03:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:36 smithi059 ceph-mon[97450]: pgmap v20281: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:03:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:36.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:36.948293+0000) 2022-09-08T08:03:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:36.798+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:36.799466+0000) 2022-09-08T08:03:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:37.947+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:37.948534+0000) 2022-09-08T08:03:38.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:37.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:37.799700+0000) 2022-09-08T08:03:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:38 smithi055 ceph-mon[97033]: pgmap v20282: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:38 smithi059 ceph-mon[97450]: pgmap v20282: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:38.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:38.799900+0000) 2022-09-08T08:03:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:38.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:38.948724+0000) 2022-09-08T08:03:39.802 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:03:39.405+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:03:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:39.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:39.948848+0000) 2022-09-08T08:03:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:39.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:39.800080+0000) 2022-09-08T08:03:40.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:40 smithi055 ceph-mon[97033]: pgmap v20283: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:40 smithi059 ceph-mon[97450]: pgmap v20283: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:40.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:40.949076+0000) 2022-09-08T08:03:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:40.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:40.800234+0000) 2022-09-08T08:03:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:41.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:41.949284+0000) 2022-09-08T08:03:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:41.799+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:41.800407+0000) 2022-09-08T08:03:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:42 smithi055 ceph-mon[97033]: pgmap v20284: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:03:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:42 smithi059 ceph-mon[97450]: pgmap v20284: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:03:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:42.948+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:42.949519+0000) 2022-09-08T08:03:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:42.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:42.800618+0000) 2022-09-08T08:03:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:43.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:43.949753+0000) 2022-09-08T08:03:44.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:43.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:43.800820+0000) 2022-09-08T08:03:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:44 smithi055 ceph-mon[97033]: pgmap v20285: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:44 smithi059 ceph-mon[97450]: pgmap v20285: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:44.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:44.949940+0000) 2022-09-08T08:03:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:44.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:44.800980+0000) 2022-09-08T08:03:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:45.949+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:45.950193+0000) 2022-09-08T08:03:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:45.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:45.801135+0000) 2022-09-08T08:03:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:46 smithi055 ceph-mon[97033]: pgmap v20286: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:03:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:46 smithi059 ceph-mon[97450]: pgmap v20286: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:03:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:46.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:46.950400+0000) 2022-09-08T08:03:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:46.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:46.801287+0000) 2022-09-08T08:03:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:47.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:47.950665+0000) 2022-09-08T08:03:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:47.800+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:47.801493+0000) 2022-09-08T08:03:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:48 smithi055 ceph-mon[97033]: pgmap v20287: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:48 smithi059 ceph-mon[97450]: pgmap v20287: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:48.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:48.950904+0000) 2022-09-08T08:03:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:48.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:48.801710+0000) 2022-09-08T08:03:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:49.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:49.951103+0000) 2022-09-08T08:03:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:49.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:49.802006+0000) 2022-09-08T08:03:50.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:50 smithi055 ceph-mon[97033]: pgmap v20288: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:50 smithi059 ceph-mon[97450]: pgmap v20288: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:50.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:50.951334+0000) 2022-09-08T08:03:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:50.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:50.802246+0000) 2022-09-08T08:03:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:51.950+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:51.951550+0000) 2022-09-08T08:03:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:51.801+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:51.802453+0000) 2022-09-08T08:03:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:52 smithi055 ceph-mon[97033]: pgmap v20289: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:03:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:52 smithi059 ceph-mon[97450]: pgmap v20289: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:03:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:52.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:52.951755+0000) 2022-09-08T08:03:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:52.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:52.802716+0000) 2022-09-08T08:03:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:53.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:53.952006+0000) 2022-09-08T08:03:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:53.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:53.802869+0000) 2022-09-08T08:03:54.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:54 smithi055 ceph-mon[97033]: pgmap v20290: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:54 smithi059 ceph-mon[97450]: pgmap v20290: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:54.805 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:03:54.421+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:03:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:54.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:54.803039+0000) 2022-09-08T08:03:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:54.951+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:54.952202+0000) 2022-09-08T08:03:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:55.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:55.952446+0000) 2022-09-08T08:03:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:55.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:55.803178+0000) 2022-09-08T08:03:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:56 smithi055 ceph-mon[97033]: pgmap v20291: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:03:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:56 smithi059 ceph-mon[97450]: pgmap v20291: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:03:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:56.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:56.952721+0000) 2022-09-08T08:03:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:56.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:56.803337+0000) 2022-09-08T08:03:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:03:57.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:03:57.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:57 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:03:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:03:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:03:58.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:57 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:03:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:57.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:57.952952+0000) 2022-09-08T08:03:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:57.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:57.803493+0000) 2022-09-08T08:03:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:58.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:58.953137+0000) 2022-09-08T08:03:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:58.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:58.803665+0000) 2022-09-08T08:03:59.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:59 smithi055 ceph-mon[97033]: pgmap v20292: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:59.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:03:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:03:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:59 smithi059 ceph-mon[97450]: pgmap v20292: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:03:59.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:03:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:04:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:03:59.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:59.953312+0000) 2022-09-08T08:04:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:03:59.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:03:59.803795+0000) 2022-09-08T08:04:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:00.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:00.953527+0000) 2022-09-08T08:04:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:00.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:00.803920+0000) 2022-09-08T08:04:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:01 smithi055 ceph-mon[97033]: pgmap v20293: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:01 smithi059 ceph-mon[97450]: pgmap v20293: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:01.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:01.953772+0000) 2022-09-08T08:04:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:01.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:01.804067+0000) 2022-09-08T08:04:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:02.952+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:02.953996+0000) 2022-09-08T08:04:03.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:02.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:02.804239+0000) 2022-09-08T08:04:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:03 smithi055 ceph-mon[97033]: pgmap v20294: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:04:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:03 smithi059 ceph-mon[97450]: pgmap v20294: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:04:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:03.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:03.954229+0000) 2022-09-08T08:04:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:03.802+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:03.804427+0000) 2022-09-08T08:04:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:04 smithi055 ceph-mon[97033]: pgmap v20295: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:04 smithi059 ceph-mon[97450]: pgmap v20295: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:04.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:04.954456+0000) 2022-09-08T08:04:05.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:04.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:04.804619+0000) 2022-09-08T08:04:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:05.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:05.954649+0000) 2022-09-08T08:04:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:05.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:05.804806+0000) 2022-09-08T08:04:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:06 smithi055 ceph-mon[97033]: pgmap v20296: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:04:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:04:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:06 smithi059 ceph-mon[97450]: pgmap v20296: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:04:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:04:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:06.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:06.954774+0000) 2022-09-08T08:04:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:06.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:06.804950+0000) 2022-09-08T08:04:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:04:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:04:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:07.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:07.954971+0000) 2022-09-08T08:04:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:07.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:07.805082+0000) 2022-09-08T08:04:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:08 smithi055 ceph-mon[97033]: pgmap v20297: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:04:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:08 smithi059 ceph-mon[97450]: pgmap v20297: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:04:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:08.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:08.955208+0000) 2022-09-08T08:04:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:08.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:08.805289+0000) 2022-09-08T08:04:09.808 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:04:09.435+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:04:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:09.953+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:09.955453+0000) 2022-09-08T08:04:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:09.803+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:09.805443+0000) 2022-09-08T08:04:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:10 smithi055 ceph-mon[97033]: pgmap v20298: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:04:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:10 smithi055 ceph-mon[97033]: osdmap e1345: 12 total, 11 up, 11 in 2022-09-08T08:04:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:10 smithi059 ceph-mon[97450]: pgmap v20298: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:04:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:10 smithi059 ceph-mon[97450]: osdmap e1345: 12 total, 11 up, 11 in 2022-09-08T08:04:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:10.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:10.955707+0000) 2022-09-08T08:04:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:10.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:10.805577+0000) 2022-09-08T08:04:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:11.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:11.805721+0000) 2022-09-08T08:04:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:11.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:11.955875+0000) 2022-09-08T08:04:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:12 smithi055 ceph-mon[97033]: pgmap v20300: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:04:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:12 smithi059 ceph-mon[97450]: pgmap v20300: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:04:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:12.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:12.956055+0000) 2022-09-08T08:04:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:12.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:12.805915+0000) 2022-09-08T08:04:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:13.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:13.956244+0000) 2022-09-08T08:04:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:13.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:13.806058+0000) 2022-09-08T08:04:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:14 smithi055 ceph-mon[97033]: osdmap e1346: 12 total, 11 up, 11 in 2022-09-08T08:04:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:14 smithi055 ceph-mon[97033]: pgmap v20302: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:04:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:14 smithi059 ceph-mon[97450]: osdmap e1346: 12 total, 11 up, 11 in 2022-09-08T08:04:14.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:14 smithi059 ceph-mon[97450]: pgmap v20302: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:04:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:14.954+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:14.956442+0000) 2022-09-08T08:04:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:14.804+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:14.806200+0000) 2022-09-08T08:04:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:15.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:15.956627+0000) 2022-09-08T08:04:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:15.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:15.806328+0000) 2022-09-08T08:04:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:16 smithi055 ceph-mon[97033]: pgmap v20303: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:04:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:16 smithi059 ceph-mon[97450]: pgmap v20303: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:04:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:16.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:16.956875+0000) 2022-09-08T08:04:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:16.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:16.806469+0000) 2022-09-08T08:04:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:17.955+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:17.957023+0000) 2022-09-08T08:04:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:17.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:17.806608+0000) 2022-09-08T08:04:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:18 smithi055 ceph-mon[97033]: pgmap v20304: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:04:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:18 smithi059 ceph-mon[97450]: pgmap v20304: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:04:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:18.805+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:18.806759+0000) 2022-09-08T08:04:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:18.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:18.957243+0000) 2022-09-08T08:04:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:19.956+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:19.957470+0000) 2022-09-08T08:04:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:19.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:19.806952+0000) 2022-09-08T08:04:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:20 smithi055 ceph-mon[97033]: pgmap v20305: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T08:04:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:20 smithi059 ceph-mon[97450]: pgmap v20305: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T08:04:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:20.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:20.957694+0000) 2022-09-08T08:04:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:20.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:20.807147+0000) 2022-09-08T08:04:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:21.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:21.807351+0000) 2022-09-08T08:04:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:21.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:21.957838+0000) 2022-09-08T08:04:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:22 smithi055 ceph-mon[97033]: pgmap v20306: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:04:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:22 smithi059 ceph-mon[97450]: pgmap v20306: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:04:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:22.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:22.958077+0000) 2022-09-08T08:04:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:22.806+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:22.807537+0000) 2022-09-08T08:04:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:23.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:23.958325+0000) 2022-09-08T08:04:24.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:23.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:23.807715+0000) 2022-09-08T08:04:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:24 smithi055 ceph-mon[97033]: pgmap v20307: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 100 B/s wr, 5 op/s 2022-09-08T08:04:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:24 smithi059 ceph-mon[97450]: pgmap v20307: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 100 B/s wr, 5 op/s 2022-09-08T08:04:24.810 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:04:24.443+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:04:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:24.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:24.807919+0000) 2022-09-08T08:04:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:24.957+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:24.958607+0000) 2022-09-08T08:04:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:25.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:25.958881+0000) 2022-09-08T08:04:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:25.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:25.808157+0000) 2022-09-08T08:04:26.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:26 smithi055 ceph-mon[97033]: pgmap v20308: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T08:04:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:26 smithi059 ceph-mon[97450]: pgmap v20308: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T08:04:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:26.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:26.959128+0000) 2022-09-08T08:04:27.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:26.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:26.808299+0000) 2022-09-08T08:04:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:27.958+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:27.959386+0000) 2022-09-08T08:04:28.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:27.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:27.808435+0000) 2022-09-08T08:04:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:28 smithi055 ceph-mon[97033]: pgmap v20309: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:28 smithi059 ceph-mon[97450]: pgmap v20309: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:28.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:28.959632+0000) 2022-09-08T08:04:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:28.807+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:28.808595+0000) 2022-09-08T08:04:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:29.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:29.959838+0000) 2022-09-08T08:04:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:29.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:29.808709+0000) 2022-09-08T08:04:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:30 smithi055 ceph-mon[97033]: pgmap v20310: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:30 smithi059 ceph-mon[97450]: pgmap v20310: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:30.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:30.960078+0000) 2022-09-08T08:04:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:30.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:30.808895+0000) 2022-09-08T08:04:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:31.959+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:31.960290+0000) 2022-09-08T08:04:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:31.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:31.809075+0000) 2022-09-08T08:04:32.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:32 smithi055 ceph-mon[97033]: pgmap v20311: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:04:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:32 smithi059 ceph-mon[97450]: pgmap v20311: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:04:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:32.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:32.960494+0000) 2022-09-08T08:04:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:32.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:32.809227+0000) 2022-09-08T08:04:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:33.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:33.960632+0000) 2022-09-08T08:04:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:33.808+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:33.809440+0000) 2022-09-08T08:04:34.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:34 smithi055 ceph-mon[97033]: pgmap v20312: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:34 smithi059 ceph-mon[97450]: pgmap v20312: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:34.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:34.809680+0000) 2022-09-08T08:04:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:34.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:34.960838+0000) 2022-09-08T08:04:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:35.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:35.961089+0000) 2022-09-08T08:04:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:35.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:35.809891+0000) 2022-09-08T08:04:36.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:36 smithi055 ceph-mon[97033]: pgmap v20313: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:04:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:36 smithi059 ceph-mon[97450]: pgmap v20313: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:04:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:36.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:36.961318+0000) 2022-09-08T08:04:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:36.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:36.810085+0000) 2022-09-08T08:04:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:37.960+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:37.961533+0000) 2022-09-08T08:04:38.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:37.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:37.810305+0000) 2022-09-08T08:04:38.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:38 smithi055 ceph-mon[97033]: pgmap v20314: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:38 smithi059 ceph-mon[97450]: pgmap v20314: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:38.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:38.961709+0000) 2022-09-08T08:04:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:38.809+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:38.810482+0000) 2022-09-08T08:04:39.813 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:04:39.459+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:04:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:39.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:39.961926+0000) 2022-09-08T08:04:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:39.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:39.810711+0000) 2022-09-08T08:04:40.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:40 smithi055 ceph-mon[97033]: pgmap v20315: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:40 smithi059 ceph-mon[97450]: pgmap v20315: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:40.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:40.962078+0000) 2022-09-08T08:04:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:40.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:40.810884+0000) 2022-09-08T08:04:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:41.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:41.962321+0000) 2022-09-08T08:04:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:41.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:41.811095+0000) 2022-09-08T08:04:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:42 smithi055 ceph-mon[97033]: pgmap v20316: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:04:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:42 smithi059 ceph-mon[97450]: pgmap v20316: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:04:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:42.961+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:42.962526+0000) 2022-09-08T08:04:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:42.810+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:42.811310+0000) 2022-09-08T08:04:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:43.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:43.962730+0000) 2022-09-08T08:04:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:43.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:43.811523+0000) 2022-09-08T08:04:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:44 smithi055 ceph-mon[97033]: pgmap v20317: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:44 smithi059 ceph-mon[97450]: pgmap v20317: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:44.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:44.962851+0000) 2022-09-08T08:04:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:44.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:44.811705+0000) 2022-09-08T08:04:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:45.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:45.963072+0000) 2022-09-08T08:04:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:45.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:45.811934+0000) 2022-09-08T08:04:46.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:46 smithi055 ceph-mon[97033]: pgmap v20318: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:04:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:46 smithi059 ceph-mon[97450]: pgmap v20318: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:04:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:46.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:46.963310+0000) 2022-09-08T08:04:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:46.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:46.812150+0000) 2022-09-08T08:04:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:47.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:47.963508+0000) 2022-09-08T08:04:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:47.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:47.812323+0000) 2022-09-08T08:04:48.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:48 smithi055 ceph-mon[97033]: pgmap v20319: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:48 smithi059 ceph-mon[97450]: pgmap v20319: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:48.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:48.963726+0000) 2022-09-08T08:04:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:48.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:48.812462+0000) 2022-09-08T08:04:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:49.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:49.963898+0000) 2022-09-08T08:04:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:49.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:49.812707+0000) 2022-09-08T08:04:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:50 smithi055 ceph-mon[97033]: pgmap v20320: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:50 smithi059 ceph-mon[97450]: pgmap v20320: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:50.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:50.812889+0000) 2022-09-08T08:04:51.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:50.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:50.964042+0000) 2022-09-08T08:04:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:51.962+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:51.964208+0000) 2022-09-08T08:04:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:51.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:51.813092+0000) 2022-09-08T08:04:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:52 smithi055 ceph-mon[97033]: pgmap v20321: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:04:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:52 smithi059 ceph-mon[97450]: pgmap v20321: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:04:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:52.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:52.964444+0000) 2022-09-08T08:04:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:52.811+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:52.813245+0000) 2022-09-08T08:04:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:53.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:53.964722+0000) 2022-09-08T08:04:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:53.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:53.813434+0000) 2022-09-08T08:04:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:54 smithi055 ceph-mon[97033]: pgmap v20322: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:54 smithi059 ceph-mon[97450]: pgmap v20322: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:54.816 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:04:54.462+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:04:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:54.963+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:54.964950+0000) 2022-09-08T08:04:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:54.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:54.813696+0000) 2022-09-08T08:04:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:55.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:55.965200+0000) 2022-09-08T08:04:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:55.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:55.813854+0000) 2022-09-08T08:04:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:56 smithi055 ceph-mon[97033]: pgmap v20323: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:04:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:56 smithi059 ceph-mon[97450]: pgmap v20323: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:04:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:56.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:56.814092+0000) 2022-09-08T08:04:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:56.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:56.965440+0000) 2022-09-08T08:04:58.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:57.812+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:57.814282+0000) 2022-09-08T08:04:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:57.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:57.965631+0000) 2022-09-08T08:04:58.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:58 smithi055 ceph-mon[97033]: pgmap v20324: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:58.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:04:58.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:04:58.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:04:58 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:04:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:58 smithi059 ceph-mon[97450]: pgmap v20324: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:04:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:04:59.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:04:59.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:04:58 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:04:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:58.964+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:58.965802+0000) 2022-09-08T08:04:59.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:58.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:58.814433+0000) 2022-09-08T08:05:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:04:59.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:59.966030+0000) 2022-09-08T08:05:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:04:59.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:04:59.814609+0000) 2022-09-08T08:05:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:00 smithi055 ceph-mon[97033]: pgmap v20325: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:05:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:00 smithi059 ceph-mon[97450]: pgmap v20325: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:05:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:00.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:00.814764+0000) 2022-09-08T08:05:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:00.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:00.966217+0000) 2022-09-08T08:05:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:01.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:01.966427+0000) 2022-09-08T08:05:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:01.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:01.814925+0000) 2022-09-08T08:05:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:02 smithi055 ceph-mon[97033]: pgmap v20326: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:05:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:02 smithi059 ceph-mon[97450]: pgmap v20326: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:05:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:02.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:02.966586+0000) 2022-09-08T08:05:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:02.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:02.815104+0000) 2022-09-08T08:05:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:03.965+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:03.966750+0000) 2022-09-08T08:05:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:03.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:03.815317+0000) 2022-09-08T08:05:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:04 smithi055 ceph-mon[97033]: pgmap v20327: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:05:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:04 smithi059 ceph-mon[97450]: pgmap v20327: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:05:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:04.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:04.967018+0000) 2022-09-08T08:05:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:04.813+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:04.815511+0000) 2022-09-08T08:05:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:05.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:05.815694+0000) 2022-09-08T08:05:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:05.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:05.967278+0000) 2022-09-08T08:05:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:06 smithi055 ceph-mon[97033]: pgmap v20328: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:05:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:05:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:06 smithi059 ceph-mon[97450]: pgmap v20328: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:05:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:05:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:06.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:06.967481+0000) 2022-09-08T08:05:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:06.814+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:06.815851+0000) 2022-09-08T08:05:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:05:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:05:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:07.966+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:07.967772+0000) 2022-09-08T08:05:08.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:07.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:07.816014+0000) 2022-09-08T08:05:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:08 smithi055 ceph-mon[97033]: pgmap v20329: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:05:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:08 smithi059 ceph-mon[97450]: pgmap v20329: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:05:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:08.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:08.816175+0000) 2022-09-08T08:05:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:08.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:08.968000+0000) 2022-09-08T08:05:09.818 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:05:09.473+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:05:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:09.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:09.968229+0000) 2022-09-08T08:05:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:09.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:09.816352+0000) 2022-09-08T08:05:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:10 smithi055 ceph-mon[97033]: pgmap v20330: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:05:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:10 smithi055 ceph-mon[97033]: osdmap e1347: 12 total, 11 up, 11 in 2022-09-08T08:05:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:10 smithi059 ceph-mon[97450]: pgmap v20330: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:05:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:10 smithi059 ceph-mon[97450]: osdmap e1347: 12 total, 11 up, 11 in 2022-09-08T08:05:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:10.967+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:10.968447+0000) 2022-09-08T08:05:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:10.815+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:10.816513+0000) 2022-09-08T08:05:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:11.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:11.968721+0000) 2022-09-08T08:05:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:11.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:11.816685+0000) 2022-09-08T08:05:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:12 smithi055 ceph-mon[97033]: pgmap v20332: 129 pgs: 11 active+clean+snaptrim_wait, 118 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:05:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:12 smithi059 ceph-mon[97450]: pgmap v20332: 129 pgs: 11 active+clean+snaptrim_wait, 118 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:05:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:12.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:12.968901+0000) 2022-09-08T08:05:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:12.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:12.816822+0000) 2022-09-08T08:05:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:13.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:13.969156+0000) 2022-09-08T08:05:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:13.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:13.817006+0000) 2022-09-08T08:05:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:14 smithi055 ceph-mon[97033]: osdmap e1348: 12 total, 11 up, 11 in 2022-09-08T08:05:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:14 smithi055 ceph-mon[97033]: pgmap v20334: 129 pgs: 11 active+clean+snaptrim_wait, 118 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:05:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:14 smithi059 ceph-mon[97450]: osdmap e1348: 12 total, 11 up, 11 in 2022-09-08T08:05:14.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:14 smithi059 ceph-mon[97450]: pgmap v20334: 129 pgs: 11 active+clean+snaptrim_wait, 118 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:05:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:14.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:14.817229+0000) 2022-09-08T08:05:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:14.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:14.969368+0000) 2022-09-08T08:05:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:15.968+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:15.969607+0000) 2022-09-08T08:05:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:15.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:15.817389+0000) 2022-09-08T08:05:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:16 smithi055 ceph-mon[97033]: pgmap v20335: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:05:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:16 smithi059 ceph-mon[97450]: pgmap v20335: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:05:17.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:16.816+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:16.817532+0000) 2022-09-08T08:05:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:16.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:16.969892+0000) 2022-09-08T08:05:18.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:17.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:17.817685+0000) 2022-09-08T08:05:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:17.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:17.970114+0000) 2022-09-08T08:05:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:18 smithi055 ceph-mon[97033]: pgmap v20336: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:05:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:18 smithi055 ceph-mon[97033]: osdmap e1349: 12 total, 11 up, 11 in 2022-09-08T08:05:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:18 smithi059 ceph-mon[97450]: pgmap v20336: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:05:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:18 smithi059 ceph-mon[97450]: osdmap e1349: 12 total, 11 up, 11 in 2022-09-08T08:05:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:18.969+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:18.970313+0000) 2022-09-08T08:05:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:18.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:18.817817+0000) 2022-09-08T08:05:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:19.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:19.970578+0000) 2022-09-08T08:05:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:19.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:19.817953+0000) 2022-09-08T08:05:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:20 smithi055 ceph-mon[97033]: pgmap v20338: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:05:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:20 smithi059 ceph-mon[97450]: pgmap v20338: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:05:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:20.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:20.818174+0000) 2022-09-08T08:05:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:20.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:20.970804+0000) 2022-09-08T08:05:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:21.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:21.971055+0000) 2022-09-08T08:05:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:21.817+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:21.818426+0000) 2022-09-08T08:05:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:22 smithi055 ceph-mon[97033]: pgmap v20339: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T08:05:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:22 smithi059 ceph-mon[97450]: pgmap v20339: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 124 B/s wr, 7 op/s 2022-09-08T08:05:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:22.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:22.971316+0000) 2022-09-08T08:05:23.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:22.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:22.818652+0000) 2022-09-08T08:05:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:23.970+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:23.971566+0000) 2022-09-08T08:05:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:23.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:23.818856+0000) 2022-09-08T08:05:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:24 smithi055 ceph-mon[97033]: pgmap v20340: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:05:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:24 smithi059 ceph-mon[97450]: pgmap v20340: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:05:24.821 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:05:24.489+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:05:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:24.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:24.971823+0000) 2022-09-08T08:05:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:24.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:24.819029+0000) 2022-09-08T08:05:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:25.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:25.971992+0000) 2022-09-08T08:05:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:25.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:25.819244+0000) 2022-09-08T08:05:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:26 smithi055 ceph-mon[97033]: pgmap v20341: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:05:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:26 smithi059 ceph-mon[97450]: pgmap v20341: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:05:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:26.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:26.972201+0000) 2022-09-08T08:05:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:26.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:26.819403+0000) 2022-09-08T08:05:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:27.971+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:27.972430+0000) 2022-09-08T08:05:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:27.818+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:27.819532+0000) 2022-09-08T08:05:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:28 smithi055 ceph-mon[97033]: pgmap v20342: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:05:28.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:05:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:28 smithi059 ceph-mon[97450]: pgmap v20342: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:05:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:05:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:28.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:28.972676+0000) 2022-09-08T08:05:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:28.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:28.819704+0000) 2022-09-08T08:05:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:29 smithi055 ceph-mon[97033]: pgmap v20343: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:05:29.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:29 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T08:05:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:29 smithi059 ceph-mon[97450]: pgmap v20343: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:05:30.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:29 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T08:05:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:29.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:29.972938+0000) 2022-09-08T08:05:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:29.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:29.819888+0000) 2022-09-08T08:05:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:30.972+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:30.973233+0000) 2022-09-08T08:05:31.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:30.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:30.820106+0000) 2022-09-08T08:05:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:31 smithi055 ceph-mon[97033]: pgmap v20344: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T08:05:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:31 smithi059 ceph-mon[97450]: pgmap v20344: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T08:05:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:31.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:31.973525+0000) 2022-09-08T08:05:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:31.819+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:31.820286+0000) 2022-09-08T08:05:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:32.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:32.973736+0000) 2022-09-08T08:05:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:32.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:32.820495+0000) 2022-09-08T08:05:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:33 smithi055 ceph-mon[97033]: pgmap v20345: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:05:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:33 smithi059 ceph-mon[97450]: pgmap v20345: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:05:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:33.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:33.973978+0000) 2022-09-08T08:05:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:33.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:33.820718+0000) 2022-09-08T08:05:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:34.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:34.820988+0000) 2022-09-08T08:05:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:34.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:34.974205+0000) 2022-09-08T08:05:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:35 smithi055 ceph-mon[97033]: pgmap v20346: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:05:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:35 smithi059 ceph-mon[97450]: pgmap v20346: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:05:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:35.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:35.974471+0000) 2022-09-08T08:05:36.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:35.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:35.821218+0000) 2022-09-08T08:05:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:36.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:36.974672+0000) 2022-09-08T08:05:37.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:36.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:36.821393+0000) 2022-09-08T08:05:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:37 smithi055 ceph-mon[97033]: pgmap v20347: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:05:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:37 smithi059 ceph-mon[97450]: pgmap v20347: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:05:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:37.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:37.974911+0000) 2022-09-08T08:05:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:37.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:37.821559+0000) 2022-09-08T08:05:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:38.973+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:38.975106+0000) 2022-09-08T08:05:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:38.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:38.821779+0000) 2022-09-08T08:05:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:39 smithi055 ceph-mon[97033]: pgmap v20348: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:05:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:39 smithi059 ceph-mon[97450]: pgmap v20348: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:05:39.824 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:05:39.504+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:05:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:39.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:39.975350+0000) 2022-09-08T08:05:40.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:39.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:39.821939+0000) 2022-09-08T08:05:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:40.820+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:40.822144+0000) 2022-09-08T08:05:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:40.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:40.975598+0000) 2022-09-08T08:05:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:41 smithi055 ceph-mon[97033]: pgmap v20349: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:05:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:41 smithi059 ceph-mon[97450]: pgmap v20349: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:05:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:41.974+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:41.975841+0000) 2022-09-08T08:05:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:41.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:41.822309+0000) 2022-09-08T08:05:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:42.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:42.976064+0000) 2022-09-08T08:05:43.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:42.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:42.822473+0000) 2022-09-08T08:05:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:43 smithi055 ceph-mon[97033]: pgmap v20350: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:05:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:43 smithi059 ceph-mon[97450]: pgmap v20350: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:05:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:43.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:43.976307+0000) 2022-09-08T08:05:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:43.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:43.822651+0000) 2022-09-08T08:05:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:44.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:44.976567+0000) 2022-09-08T08:05:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:44.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:44.822831+0000) 2022-09-08T08:05:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:45 smithi055 ceph-mon[97033]: pgmap v20351: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:05:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:45 smithi059 ceph-mon[97450]: pgmap v20351: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:05:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:45.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:45.976880+0000) 2022-09-08T08:05:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:45.821+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:45.823066+0000) 2022-09-08T08:05:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:46.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:46.977035+0000) 2022-09-08T08:05:47.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:46.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:46.823331+0000) 2022-09-08T08:05:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:47 smithi055 ceph-mon[97033]: pgmap v20352: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:05:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:47 smithi059 ceph-mon[97450]: pgmap v20352: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:05:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:47.975+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:47.977262+0000) 2022-09-08T08:05:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:47.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:47.823497+0000) 2022-09-08T08:05:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:48.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:48.977510+0000) 2022-09-08T08:05:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:48.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:48.823738+0000) 2022-09-08T08:05:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:49 smithi055 ceph-mon[97033]: pgmap v20353: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:05:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:49 smithi059 ceph-mon[97450]: pgmap v20353: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:05:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:49.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:49.977743+0000) 2022-09-08T08:05:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:49.822+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:49.823904+0000) 2022-09-08T08:05:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:50.976+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:50.977881+0000) 2022-09-08T08:05:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:50.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:50.824061+0000) 2022-09-08T08:05:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:51 smithi055 ceph-mon[97033]: pgmap v20354: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:05:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:51 smithi059 ceph-mon[97450]: pgmap v20354: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:05:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:51.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:51.978112+0000) 2022-09-08T08:05:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:51.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:51.824257+0000) 2022-09-08T08:05:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:52.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:52.978334+0000) 2022-09-08T08:05:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:52.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:52.824397+0000) 2022-09-08T08:05:53.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:53 smithi055 ceph-mon[97033]: pgmap v20355: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:05:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:53 smithi059 ceph-mon[97450]: pgmap v20355: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:05:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:53.977+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:53.978582+0000) 2022-09-08T08:05:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:53.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:53.824532+0000) 2022-09-08T08:05:54.827 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:05:54.510+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:05:55.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:54.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:54.824760+0000) 2022-09-08T08:05:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:54.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:54.978777+0000) 2022-09-08T08:05:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:55 smithi055 ceph-mon[97033]: pgmap v20356: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:05:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:55 smithi059 ceph-mon[97450]: pgmap v20356: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:05:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:55.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:55.979032+0000) 2022-09-08T08:05:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:55.823+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:55.824947+0000) 2022-09-08T08:05:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:56.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:56.979185+0000) 2022-09-08T08:05:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:56.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:56.825152+0000) 2022-09-08T08:05:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:57 smithi055 ceph-mon[97033]: pgmap v20357: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:05:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:57 smithi059 ceph-mon[97450]: pgmap v20357: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:05:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:57.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:57.979388+0000) 2022-09-08T08:05:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:57.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:57.825353+0000) 2022-09-08T08:05:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:58.978+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:58.979603+0000) 2022-09-08T08:05:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:58.824+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:58.825574+0000) 2022-09-08T08:05:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:05:59 smithi055 ceph-mon[97033]: pgmap v20358: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:05:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:05:59 smithi059 ceph-mon[97450]: pgmap v20358: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:06:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:05:59.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:59.979783+0000) 2022-09-08T08:06:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:05:59.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:05:59.825733+0000) 2022-09-08T08:06:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:00.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:00.979922+0000) 2022-09-08T08:06:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:00.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:00.825879+0000) 2022-09-08T08:06:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:01 smithi055 ceph-mon[97033]: pgmap v20359: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:06:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:01 smithi059 ceph-mon[97450]: pgmap v20359: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:06:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:01.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:01.980041+0000) 2022-09-08T08:06:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:01.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:01.826111+0000) 2022-09-08T08:06:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:02.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:02.980263+0000) 2022-09-08T08:06:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:02.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:02.826252+0000) 2022-09-08T08:06:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:03 smithi055 ceph-mon[97033]: pgmap v20360: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:03 smithi059 ceph-mon[97450]: pgmap v20360: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:03.979+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:03.980513+0000) 2022-09-08T08:06:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:03.825+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:03.826465+0000) 2022-09-08T08:06:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:04.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:04.980774+0000) 2022-09-08T08:06:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:04.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:04.826716+0000) 2022-09-08T08:06:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:05 smithi055 ceph-mon[97033]: pgmap v20361: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:05 smithi059 ceph-mon[97450]: pgmap v20361: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:05.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:05.981007+0000) 2022-09-08T08:06:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:05.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:05.826892+0000) 2022-09-08T08:06:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:06:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:06:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:06.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:06.981243+0000) 2022-09-08T08:06:07.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:06.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:06.827080+0000) 2022-09-08T08:06:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:07 smithi055 ceph-mon[97033]: pgmap v20362: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:06:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:06:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:07 smithi059 ceph-mon[97450]: pgmap v20362: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:06:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:06:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:07.980+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:07.981520+0000) 2022-09-08T08:06:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:07.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:07.827289+0000) 2022-09-08T08:06:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:08.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:08.981721+0000) 2022-09-08T08:06:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:08.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:08.827458+0000) 2022-09-08T08:06:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:09 smithi055 ceph-mon[97033]: pgmap v20363: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:06:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:09 smithi059 ceph-mon[97450]: pgmap v20363: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:06:09.830 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:06:09.518+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:06:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:09.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:09.981894+0000) 2022-09-08T08:06:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:09.826+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:09.827601+0000) 2022-09-08T08:06:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:10 smithi055 ceph-mon[97033]: osdmap e1350: 12 total, 11 up, 11 in 2022-09-08T08:06:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:10 smithi059 ceph-mon[97450]: osdmap e1350: 12 total, 11 up, 11 in 2022-09-08T08:06:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:10.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:10.982150+0000) 2022-09-08T08:06:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:10.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:10.827753+0000) 2022-09-08T08:06:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:11 smithi055 ceph-mon[97033]: pgmap v20365: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:06:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:11 smithi059 ceph-mon[97450]: pgmap v20365: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:06:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:11.981+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:11.982362+0000) 2022-09-08T08:06:12.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:11.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:11.827947+0000) 2022-09-08T08:06:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:12.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:12.982561+0000) 2022-09-08T08:06:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:12.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:12.828115+0000) 2022-09-08T08:06:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:13 smithi055 ceph-mon[97033]: pgmap v20366: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:06:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:13 smithi059 ceph-mon[97450]: pgmap v20366: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:06:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:13.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:13.982780+0000) 2022-09-08T08:06:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:13.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:13.828298+0000) 2022-09-08T08:06:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:14.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:14.983009+0000) 2022-09-08T08:06:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:14.827+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:14.828454+0000) 2022-09-08T08:06:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:15 smithi055 ceph-mon[97033]: pgmap v20367: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:06:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:15 smithi059 ceph-mon[97450]: pgmap v20367: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:06:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:15.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:15.983224+0000) 2022-09-08T08:06:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:15.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:15.828691+0000) 2022-09-08T08:06:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:16.982+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:16.983408+0000) 2022-09-08T08:06:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:16.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:16.828893+0000) 2022-09-08T08:06:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:17 smithi055 ceph-mon[97033]: pgmap v20368: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:06:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:17 smithi059 ceph-mon[97450]: pgmap v20368: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:06:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:17.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:17.983608+0000) 2022-09-08T08:06:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:17.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:17.829069+0000) 2022-09-08T08:06:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:18.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:18.829206+0000) 2022-09-08T08:06:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:18.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:18.983801+0000) 2022-09-08T08:06:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:19 smithi055 ceph-mon[97033]: osdmap e1351: 12 total, 11 up, 11 in 2022-09-08T08:06:19.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:19 smithi055 ceph-mon[97033]: pgmap v20370: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 226 B/s wr, 7 op/s 2022-09-08T08:06:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:19 smithi059 ceph-mon[97450]: osdmap e1351: 12 total, 11 up, 11 in 2022-09-08T08:06:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:19 smithi059 ceph-mon[97450]: pgmap v20370: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 226 B/s wr, 7 op/s 2022-09-08T08:06:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:19.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:19.984018+0000) 2022-09-08T08:06:20.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:19.828+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:19.829351+0000) 2022-09-08T08:06:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:20.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:20.984247+0000) 2022-09-08T08:06:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:20.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:20.829568+0000) 2022-09-08T08:06:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:21 smithi055 ceph-mon[97033]: pgmap v20371: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:06:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:21 smithi059 ceph-mon[97450]: pgmap v20371: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:06:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:21.983+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:21.984473+0000) 2022-09-08T08:06:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:21.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:21.829726+0000) 2022-09-08T08:06:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:22.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:22.984681+0000) 2022-09-08T08:06:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:22.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:22.829848+0000) 2022-09-08T08:06:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:23 smithi055 ceph-mon[97033]: pgmap v20372: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:06:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:23 smithi059 ceph-mon[97450]: pgmap v20372: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:06:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:23.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:23.984891+0000) 2022-09-08T08:06:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:23.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:23.830003+0000) 2022-09-08T08:06:24.832 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:06:24.522+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:06:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:24.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:24.985103+0000) 2022-09-08T08:06:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:24.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:24.830230+0000) 2022-09-08T08:06:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:25 smithi055 ceph-mon[97033]: pgmap v20373: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:06:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:25 smithi059 ceph-mon[97450]: pgmap v20373: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:06:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:25.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:25.830440+0000) 2022-09-08T08:06:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:25.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:25.985349+0000) 2022-09-08T08:06:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:26.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:26.830603+0000) 2022-09-08T08:06:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:26.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:26.985592+0000) 2022-09-08T08:06:27.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:27 smithi055 ceph-mon[97033]: pgmap v20374: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:06:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:27 smithi059 ceph-mon[97450]: pgmap v20374: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:06:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:27.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:27.830808+0000) 2022-09-08T08:06:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:27.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:27.985807+0000) 2022-09-08T08:06:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:28.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:28.985994+0000) 2022-09-08T08:06:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:28.829+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:28.831013+0000) 2022-09-08T08:06:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:29 smithi055 ceph-mon[97033]: pgmap v20375: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:06:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:06:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:06:29.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:06:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:29 smithi059 ceph-mon[97450]: pgmap v20375: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:06:29.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:06:29.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:06:29.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:06:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:29.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:29.986227+0000) 2022-09-08T08:06:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:29.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:29.831177+0000) 2022-09-08T08:06:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:30.984+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:30.986375+0000) 2022-09-08T08:06:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:30.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:30.831394+0000) 2022-09-08T08:06:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:31 smithi055 ceph-mon[97033]: pgmap v20376: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:06:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:31 smithi059 ceph-mon[97450]: pgmap v20376: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:06:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:31.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:31.986608+0000) 2022-09-08T08:06:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:31.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:31.831575+0000) 2022-09-08T08:06:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:32.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:32.986835+0000) 2022-09-08T08:06:33.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:32.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:32.831700+0000) 2022-09-08T08:06:33.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:33 smithi055 ceph-mon[97033]: pgmap v20377: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:33 smithi059 ceph-mon[97450]: pgmap v20377: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:33.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:33.987074+0000) 2022-09-08T08:06:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:33.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:33.831922+0000) 2022-09-08T08:06:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:34.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:34.987279+0000) 2022-09-08T08:06:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:34.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:34.832153+0000) 2022-09-08T08:06:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:35 smithi055 ceph-mon[97033]: pgmap v20378: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:35 smithi059 ceph-mon[97450]: pgmap v20378: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:35.985+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:35.987486+0000) 2022-09-08T08:06:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:35.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:35.832342+0000) 2022-09-08T08:06:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:36.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:36.987739+0000) 2022-09-08T08:06:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:36.830+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:36.832497+0000) 2022-09-08T08:06:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:37 smithi055 ceph-mon[97033]: pgmap v20379: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:06:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:37 smithi059 ceph-mon[97450]: pgmap v20379: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:06:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:37.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:37.987993+0000) 2022-09-08T08:06:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:37.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:37.832737+0000) 2022-09-08T08:06:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:38.986+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:38.988233+0000) 2022-09-08T08:06:39.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:38.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:38.832899+0000) 2022-09-08T08:06:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:39 smithi055 ceph-mon[97033]: pgmap v20380: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:39 smithi059 ceph-mon[97450]: pgmap v20380: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:39.835 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:06:39.528+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:06:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:39.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:39.988397+0000) 2022-09-08T08:06:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:39.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:39.833024+0000) 2022-09-08T08:06:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:40.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:40.833219+0000) 2022-09-08T08:06:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:40.987+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:40.988584+0000) 2022-09-08T08:06:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:41 smithi055 ceph-mon[97033]: pgmap v20381: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:06:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:41 smithi059 ceph-mon[97450]: pgmap v20381: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:06:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:41.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:41.988842+0000) 2022-09-08T08:06:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:41.831+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:41.833399+0000) 2022-09-08T08:06:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:42.832+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:42.833678+0000) 2022-09-08T08:06:43.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:42.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:42.989097+0000) 2022-09-08T08:06:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:43 smithi055 ceph-mon[97033]: pgmap v20382: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:43 smithi059 ceph-mon[97450]: pgmap v20382: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:43.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:43.989330+0000) 2022-09-08T08:06:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:43.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:43.833923+0000) 2022-09-08T08:06:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:44.988+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:44.989580+0000) 2022-09-08T08:06:45.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:44.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:44.834123+0000) 2022-09-08T08:06:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:45 smithi055 ceph-mon[97033]: pgmap v20383: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:45 smithi059 ceph-mon[97450]: pgmap v20383: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:45.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:45.989807+0000) 2022-09-08T08:06:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:45.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:45.834378+0000) 2022-09-08T08:06:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:46.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:46.990074+0000) 2022-09-08T08:06:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:46.833+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:46.834562+0000) 2022-09-08T08:06:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:47 smithi055 ceph-mon[97033]: pgmap v20384: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:06:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:47 smithi059 ceph-mon[97450]: pgmap v20384: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:06:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:47.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:47.990292+0000) 2022-09-08T08:06:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:47.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:47.834786+0000) 2022-09-08T08:06:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:48.989+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:48.990513+0000) 2022-09-08T08:06:49.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:48.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:48.835041+0000) 2022-09-08T08:06:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:49 smithi055 ceph-mon[97033]: pgmap v20385: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:49 smithi059 ceph-mon[97450]: pgmap v20385: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:49.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:49.990732+0000) 2022-09-08T08:06:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:49.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:49.835210+0000) 2022-09-08T08:06:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:50.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:50.990949+0000) 2022-09-08T08:06:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:50.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:50.835356+0000) 2022-09-08T08:06:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:51 smithi055 ceph-mon[97033]: pgmap v20386: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:06:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:51 smithi059 ceph-mon[97450]: pgmap v20386: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:06:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:51.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:51.991126+0000) 2022-09-08T08:06:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:51.834+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:51.835506+0000) 2022-09-08T08:06:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:52.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:52.991351+0000) 2022-09-08T08:06:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:52.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:52.835750+0000) 2022-09-08T08:06:53.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:53 smithi055 ceph-mon[97033]: pgmap v20387: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:53 smithi059 ceph-mon[97450]: pgmap v20387: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:53.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:53.835973+0000) 2022-09-08T08:06:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:53.990+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:53.991552+0000) 2022-09-08T08:06:54.838 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:06:54.534+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:06:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:54.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:54.991710+0000) 2022-09-08T08:06:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:54.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:54.836105+0000) 2022-09-08T08:06:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:55 smithi055 ceph-mon[97033]: pgmap v20388: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:55 smithi059 ceph-mon[97450]: pgmap v20388: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:55.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:55.991904+0000) 2022-09-08T08:06:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:55.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:55.836233+0000) 2022-09-08T08:06:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:56.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:56.992073+0000) 2022-09-08T08:06:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:56.835+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:56.836396+0000) 2022-09-08T08:06:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:57 smithi055 ceph-mon[97033]: pgmap v20389: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:06:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:57 smithi059 ceph-mon[97450]: pgmap v20389: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:06:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:57.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:57.992344+0000) 2022-09-08T08:06:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:57.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:57.836616+0000) 2022-09-08T08:06:58.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:58.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:58.836893+0000) 2022-09-08T08:06:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:58.991+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:58.992541+0000) 2022-09-08T08:06:59.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:59 smithi055 ceph-mon[97033]: pgmap v20390: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:06:59.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:59 smithi055 ceph-mon[97033]: pgmap v20391: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:06:59.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:59 smithi055 ceph-mon[97033]: pgmap v20392: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:06:59.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:06:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:07:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:59 smithi059 ceph-mon[97450]: pgmap v20390: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:59 smithi059 ceph-mon[97450]: pgmap v20391: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:07:00.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:59 smithi059 ceph-mon[97450]: pgmap v20392: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:07:00.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:06:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:07:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:06:59.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:59.992764+0000) 2022-09-08T08:07:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:06:59.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:06:59.837046+0000) 2022-09-08T08:07:01.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:00.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:00.837236+0000) 2022-09-08T08:07:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:00.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:00.992932+0000) 2022-09-08T08:07:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:01 smithi055 ceph-mon[97033]: pgmap v20393: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:07:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:01 smithi059 ceph-mon[97450]: pgmap v20393: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:07:02.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:01.836+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:01.837465+0000) 2022-09-08T08:07:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:01.992+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:01.993157+0000) 2022-09-08T08:07:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:02.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:02.993374+0000) 2022-09-08T08:07:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:02.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:02.837709+0000) 2022-09-08T08:07:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:03 smithi055 ceph-mon[97033]: pgmap v20394: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:07:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:03 smithi059 ceph-mon[97450]: pgmap v20394: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:07:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:03.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:03.993623+0000) 2022-09-08T08:07:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:03.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:03.837850+0000) 2022-09-08T08:07:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:04.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:04.993878+0000) 2022-09-08T08:07:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:04.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:04.838091+0000) 2022-09-08T08:07:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:05 smithi055 ceph-mon[97033]: pgmap v20395: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:07:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:05 smithi059 ceph-mon[97450]: pgmap v20395: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:07:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:05.993+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:05.994112+0000) 2022-09-08T08:07:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:05.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:05.838265+0000) 2022-09-08T08:07:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:07:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:07:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:06.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:06.994362+0000) 2022-09-08T08:07:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:06.837+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:06.838410+0000) 2022-09-08T08:07:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:07 smithi055 ceph-mon[97033]: pgmap v20396: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T08:07:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:07:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:07 smithi059 ceph-mon[97450]: pgmap v20396: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 64 KiB/s wr, 28 op/s 2022-09-08T08:07:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:07:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:07.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:07.994607+0000) 2022-09-08T08:07:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:07.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:07.838578+0000) 2022-09-08T08:07:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:08.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:08.994885+0000) 2022-09-08T08:07:09.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:08.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:08.838819+0000) 2022-09-08T08:07:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:09 smithi055 ceph-mon[97033]: pgmap v20397: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:07:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:09 smithi059 ceph-mon[97450]: pgmap v20397: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:07:09.841 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:07:09.539+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:07:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:09.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:09.995125+0000) 2022-09-08T08:07:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:09.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:09.839025+0000) 2022-09-08T08:07:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:10 smithi055 ceph-mon[97033]: osdmap e1352: 12 total, 11 up, 11 in 2022-09-08T08:07:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:10 smithi059 ceph-mon[97450]: osdmap e1352: 12 total, 11 up, 11 in 2022-09-08T08:07:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:10.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:10.995310+0000) 2022-09-08T08:07:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:10.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:10.839206+0000) 2022-09-08T08:07:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:11 smithi055 ceph-mon[97033]: pgmap v20399: 129 pgs: 1 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 110 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:07:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:11 smithi059 ceph-mon[97450]: pgmap v20399: 129 pgs: 1 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 110 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:07:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:11.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:11.995483+0000) 2022-09-08T08:07:12.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:11.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:11.839465+0000) 2022-09-08T08:07:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:12.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:12.995758+0000) 2022-09-08T08:07:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:12.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:12.839721+0000) 2022-09-08T08:07:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:13 smithi055 ceph-mon[97033]: pgmap v20400: 129 pgs: 1 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 110 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:07:13.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:13 smithi055 ceph-mon[97033]: osdmap e1353: 12 total, 11 up, 11 in 2022-09-08T08:07:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:13 smithi059 ceph-mon[97450]: pgmap v20400: 129 pgs: 1 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 110 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:07:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:13 smithi059 ceph-mon[97450]: osdmap e1353: 12 total, 11 up, 11 in 2022-09-08T08:07:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:13.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:13.995932+0000) 2022-09-08T08:07:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:13.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:13.839953+0000) 2022-09-08T08:07:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:14.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:14.996112+0000) 2022-09-08T08:07:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:14.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:14.840119+0000) 2022-09-08T08:07:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:15 smithi055 ceph-mon[97033]: pgmap v20402: 129 pgs: 1 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 110 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:07:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:15 smithi059 ceph-mon[97450]: pgmap v20402: 129 pgs: 1 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 110 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:07:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:15.994+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:15.996283+0000) 2022-09-08T08:07:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:15.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:15.840256+0000) 2022-09-08T08:07:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:16.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:16.996493+0000) 2022-09-08T08:07:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:16.838+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:16.840401+0000) 2022-09-08T08:07:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:17 smithi055 ceph-mon[97033]: pgmap v20403: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:07:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:17 smithi059 ceph-mon[97450]: pgmap v20403: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:07:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:17.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:17.996726+0000) 2022-09-08T08:07:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:17.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:17.840593+0000) 2022-09-08T08:07:18.999 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:18.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:18.840759+0000) 2022-09-08T08:07:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:18.995+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:18.996940+0000) 2022-09-08T08:07:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:19 smithi055 ceph-mon[97033]: osdmap e1354: 12 total, 11 up, 11 in 2022-09-08T08:07:19.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:19 smithi055 ceph-mon[97033]: pgmap v20405: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:07:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:19 smithi059 ceph-mon[97450]: osdmap e1354: 12 total, 11 up, 11 in 2022-09-08T08:07:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:19 smithi059 ceph-mon[97450]: pgmap v20405: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:07:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:19.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:19.997189+0000) 2022-09-08T08:07:20.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:19.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:19.840966+0000) 2022-09-08T08:07:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:20.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:20.997427+0000) 2022-09-08T08:07:21.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:20.839+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:20.841120+0000) 2022-09-08T08:07:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:21 smithi055 ceph-mon[97033]: pgmap v20406: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T08:07:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:21 smithi059 ceph-mon[97450]: pgmap v20406: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T08:07:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:21.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:21.997689+0000) 2022-09-08T08:07:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:21.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:21.841344+0000) 2022-09-08T08:07:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:22.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:22.841521+0000) 2022-09-08T08:07:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:22.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:22.997956+0000) 2022-09-08T08:07:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:23 smithi055 ceph-mon[97033]: pgmap v20407: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 110 B/s wr, 6 op/s 2022-09-08T08:07:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:23 smithi059 ceph-mon[97450]: pgmap v20407: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 110 B/s wr, 6 op/s 2022-09-08T08:07:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:23.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:23.841689+0000) 2022-09-08T08:07:24.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:23.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:23.998165+0000) 2022-09-08T08:07:24.844 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:07:24.553+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:07:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:24.996+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:24.998394+0000) 2022-09-08T08:07:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:24.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:24.841902+0000) 2022-09-08T08:07:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:25 smithi055 ceph-mon[97033]: pgmap v20408: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:07:26.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:25 smithi059 ceph-mon[97450]: pgmap v20408: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:07:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:25.840+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:25.842133+0000) 2022-09-08T08:07:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:25.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:25.998722+0000) 2022-09-08T08:07:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:26.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:26.842371+0000) 2022-09-08T08:07:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:26.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:26.998966+0000) 2022-09-08T08:07:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:27 smithi055 ceph-mon[97033]: pgmap v20409: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:07:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:27 smithi059 ceph-mon[97450]: pgmap v20409: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:07:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:27.997+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:27.999206+0000) 2022-09-08T08:07:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:27.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:27.842514+0000) 2022-09-08T08:07:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:28.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:28.999438+0000) 2022-09-08T08:07:29.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:28.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:28.842702+0000) 2022-09-08T08:07:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:29 smithi055 ceph-mon[97033]: pgmap v20410: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:07:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:29 smithi059 ceph-mon[97450]: pgmap v20410: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:07:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:29.998+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:29.999716+0000) 2022-09-08T08:07:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:29.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:29.842900+0000) 2022-09-08T08:07:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:30.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:30.999859+0000) 2022-09-08T08:07:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:30.841+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:30.843049+0000) 2022-09-08T08:07:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:31 smithi055 ceph-mon[97033]: pgmap v20411: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:07:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:31 smithi059 ceph-mon[97450]: pgmap v20411: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:07:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:31.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:32.000093+0000) 2022-09-08T08:07:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:31.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:31.843227+0000) 2022-09-08T08:07:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:32.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:33.000337+0000) 2022-09-08T08:07:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:32.842+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:32.843390+0000) 2022-09-08T08:07:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:33 smithi055 ceph-mon[97033]: pgmap v20412: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:33 smithi059 ceph-mon[97450]: pgmap v20412: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:33.999+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:34.000565+0000) 2022-09-08T08:07:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:33.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:33.843627+0000) 2022-09-08T08:07:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:35.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:35.000720+0000) 2022-09-08T08:07:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:34.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:34.843838+0000) 2022-09-08T08:07:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:35 smithi059 ceph-mon[97450]: pgmap v20413: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:36.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:36.000914+0000) 2022-09-08T08:07:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:35.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:35.843961+0000) 2022-09-08T08:07:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:35 smithi055 ceph-mon[97033]: pgmap v20413: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:37.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:37.001105+0000) 2022-09-08T08:07:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:36.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:36.844185+0000) 2022-09-08T08:07:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:37 smithi059 ceph-mon[97450]: pgmap v20414: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:07:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:38.000+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:38.001320+0000) 2022-09-08T08:07:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:37.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:37.844323+0000) 2022-09-08T08:07:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:37 smithi055 ceph-mon[97033]: pgmap v20414: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:07:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:39.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:39.001617+0000) 2022-09-08T08:07:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:38.843+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:38.844469+0000) 2022-09-08T08:07:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:39 smithi055 ceph-mon[97033]: pgmap v20415: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:39 smithi059 ceph-mon[97450]: pgmap v20415: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:39.847 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:07:39.554+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:07:40.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:40.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:40.001903+0000) 2022-09-08T08:07:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:39.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:39.844752+0000) 2022-09-08T08:07:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:41.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:41.002111+0000) 2022-09-08T08:07:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:40.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:40.844991+0000) 2022-09-08T08:07:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:41 smithi055 ceph-mon[97033]: pgmap v20416: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:07:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:41 smithi059 ceph-mon[97450]: pgmap v20416: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:07:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:42.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:42.002344+0000) 2022-09-08T08:07:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:41.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:41.845171+0000) 2022-09-08T08:07:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:43.001+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:43.002570+0000) 2022-09-08T08:07:43.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:42.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:42.845362+0000) 2022-09-08T08:07:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:43 smithi055 ceph-mon[97033]: pgmap v20417: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:43 smithi059 ceph-mon[97450]: pgmap v20417: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:44.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:44.002780+0000) 2022-09-08T08:07:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:43.844+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:43.845573+0000) 2022-09-08T08:07:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:45.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:45.002956+0000) 2022-09-08T08:07:45.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:44.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:44.845748+0000) 2022-09-08T08:07:46.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:45 smithi059 ceph-mon[97450]: pgmap v20418: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:46.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:46.003155+0000) 2022-09-08T08:07:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:45.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:45.845906+0000) 2022-09-08T08:07:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:45 smithi055 ceph-mon[97033]: pgmap v20418: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:47.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:47.003368+0000) 2022-09-08T08:07:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:46.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:46.846090+0000) 2022-09-08T08:07:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:47 smithi059 ceph-mon[97450]: pgmap v20419: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:07:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:48.002+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:48.003593+0000) 2022-09-08T08:07:48.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:47.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:47.846231+0000) 2022-09-08T08:07:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:47 smithi055 ceph-mon[97033]: pgmap v20419: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:07:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:49.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:49.003821+0000) 2022-09-08T08:07:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:48.845+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:48.846459+0000) 2022-09-08T08:07:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:49 smithi055 ceph-mon[97033]: pgmap v20420: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:49 smithi059 ceph-mon[97450]: pgmap v20420: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:50.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:50.004055+0000) 2022-09-08T08:07:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:49.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:49.846673+0000) 2022-09-08T08:07:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:51.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:51.004346+0000) 2022-09-08T08:07:51.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:50.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:50.846843+0000) 2022-09-08T08:07:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:51 smithi055 ceph-mon[97033]: pgmap v20421: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:07:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:51 smithi059 ceph-mon[97450]: pgmap v20421: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:07:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:52.003+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:52.004590+0000) 2022-09-08T08:07:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:51.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:51.847003+0000) 2022-09-08T08:07:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:53.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:53.004839+0000) 2022-09-08T08:07:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:52.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:52.847146+0000) 2022-09-08T08:07:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:53 smithi059 ceph-mon[97450]: pgmap v20422: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:54.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:53.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:53.847308+0000) 2022-09-08T08:07:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:54.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:54.005018+0000) 2022-09-08T08:07:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:53 smithi055 ceph-mon[97033]: pgmap v20422: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:54.849 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:07:54.567+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:07:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:55.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:55.005261+0000) 2022-09-08T08:07:55.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:54.846+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:54.847502+0000) 2022-09-08T08:07:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:55 smithi059 ceph-mon[97450]: pgmap v20423: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:56.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:56.005542+0000) 2022-09-08T08:07:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:55.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:55.847744+0000) 2022-09-08T08:07:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:55 smithi055 ceph-mon[97033]: pgmap v20423: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:57.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:57.005769+0000) 2022-09-08T08:07:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:56.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:56.847916+0000) 2022-09-08T08:07:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:57 smithi059 ceph-mon[97450]: pgmap v20424: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:07:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:58.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:58.005913+0000) 2022-09-08T08:07:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:57.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:57.848085+0000) 2022-09-08T08:07:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:57 smithi055 ceph-mon[97033]: pgmap v20424: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:07:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:07:59.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:59.006084+0000) 2022-09-08T08:07:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:58.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:58.848225+0000) 2022-09-08T08:07:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:59 smithi055 ceph-mon[97033]: pgmap v20425: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:07:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:07:59.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:07:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:07:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:59 smithi059 ceph-mon[97450]: pgmap v20425: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:07:59.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:07:59.786 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:07:59.787 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:07:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:08:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:00.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:00.006249+0000) 2022-09-08T08:08:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:07:59.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:07:59.848423+0000) 2022-09-08T08:08:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:01.004+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:01.006475+0000) 2022-09-08T08:08:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:00.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:00.848682+0000) 2022-09-08T08:08:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:01 smithi055 ceph-mon[97033]: pgmap v20426: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:08:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:01 smithi059 ceph-mon[97450]: pgmap v20426: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:08:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:02.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:02.006690+0000) 2022-09-08T08:08:02.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:01.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:01.848897+0000) 2022-09-08T08:08:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:03.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:03.006878+0000) 2022-09-08T08:08:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:02.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:02.849049+0000) 2022-09-08T08:08:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:03 smithi059 ceph-mon[97450]: pgmap v20427: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:08:04.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:03.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:03.849188+0000) 2022-09-08T08:08:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:04.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:04.007114+0000) 2022-09-08T08:08:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:03 smithi055 ceph-mon[97033]: pgmap v20427: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:08:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:05.005+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:05.007357+0000) 2022-09-08T08:08:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:04.847+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:04.849346+0000) 2022-09-08T08:08:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:05 smithi059 ceph-mon[97450]: pgmap v20428: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:08:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:06.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:06.007576+0000) 2022-09-08T08:08:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:05.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:05.849512+0000) 2022-09-08T08:08:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:05 smithi055 ceph-mon[97033]: pgmap v20428: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:08:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:08:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:07.006+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:07.007795+0000) 2022-09-08T08:08:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:06.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:06.849684+0000) 2022-09-08T08:08:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:08:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:07 smithi059 ceph-mon[97450]: pgmap v20429: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:08:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:08:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:08.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:08.008068+0000) 2022-09-08T08:08:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:07.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:07.849856+0000) 2022-09-08T08:08:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:07 smithi055 ceph-mon[97033]: pgmap v20429: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:08:08.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:08:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:09.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:09.008318+0000) 2022-09-08T08:08:09.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:08.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:08.850050+0000) 2022-09-08T08:08:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:09 smithi055 ceph-mon[97033]: pgmap v20430: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:08:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:09 smithi059 ceph-mon[97450]: pgmap v20430: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:08:09.852 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:08:09.573+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:08:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:10.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:10.008570+0000) 2022-09-08T08:08:10.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:09.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:09.850262+0000) 2022-09-08T08:08:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:10 smithi055 ceph-mon[97033]: osdmap e1355: 12 total, 11 up, 11 in 2022-09-08T08:08:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:10 smithi059 ceph-mon[97450]: osdmap e1355: 12 total, 11 up, 11 in 2022-09-08T08:08:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:11.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:11.008777+0000) 2022-09-08T08:08:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:10.848+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:10.850424+0000) 2022-09-08T08:08:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:11 smithi055 ceph-mon[97033]: pgmap v20432: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:08:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:11 smithi059 ceph-mon[97450]: pgmap v20432: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:08:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:11.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:11.850671+0000) 2022-09-08T08:08:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:12.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:12.009001+0000) 2022-09-08T08:08:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:13.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:13.009194+0000) 2022-09-08T08:08:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:12.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:12.850829+0000) 2022-09-08T08:08:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:13 smithi059 ceph-mon[97450]: pgmap v20433: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:08:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:13 smithi059 ceph-mon[97450]: osdmap e1356: 12 total, 11 up, 11 in 2022-09-08T08:08:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:13.849+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:13.850998+0000) 2022-09-08T08:08:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:14.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:14.009360+0000) 2022-09-08T08:08:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:13 smithi055 ceph-mon[97033]: pgmap v20433: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:08:14.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:13 smithi055 ceph-mon[97033]: osdmap e1356: 12 total, 11 up, 11 in 2022-09-08T08:08:15.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:14.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:14.851201+0000) 2022-09-08T08:08:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:15.007+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:15.009567+0000) 2022-09-08T08:08:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:15 smithi059 ceph-mon[97450]: pgmap v20435: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:08:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:16.008+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:16.009798+0000) 2022-09-08T08:08:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:15.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:15.851420+0000) 2022-09-08T08:08:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:15 smithi055 ceph-mon[97033]: pgmap v20435: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:08:17.156 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:16.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:16.851599+0000) 2022-09-08T08:08:17.157 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:17.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:17.010030+0000) 2022-09-08T08:08:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:17 smithi059 ceph-mon[97450]: pgmap v20436: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:08:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:18.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:18.010254+0000) 2022-09-08T08:08:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:17.850+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:17.851769+0000) 2022-09-08T08:08:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:17 smithi055 ceph-mon[97033]: pgmap v20436: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:08:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:19.009+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:19.010540+0000) 2022-09-08T08:08:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:18.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:18.851910+0000) 2022-09-08T08:08:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:19 smithi055 ceph-mon[97033]: pgmap v20437: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-08T08:08:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:19 smithi059 ceph-mon[97450]: pgmap v20437: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-08T08:08:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:20.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:20.010831+0000) 2022-09-08T08:08:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:19.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:19.852122+0000) 2022-09-08T08:08:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:21.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:21.011098+0000) 2022-09-08T08:08:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:20.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:20.852312+0000) 2022-09-08T08:08:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:21 smithi059 ceph-mon[97450]: pgmap v20438: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:08:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:22.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:22.011274+0000) 2022-09-08T08:08:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:21.851+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:21.852572+0000) 2022-09-08T08:08:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:21 smithi055 ceph-mon[97033]: pgmap v20438: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:08:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:23.010+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:23.011455+0000) 2022-09-08T08:08:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:22.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:22.852722+0000) 2022-09-08T08:08:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:23 smithi059 ceph-mon[97450]: pgmap v20439: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:08:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:24.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:24.011979+0000) 2022-09-08T08:08:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:23.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:23.852851+0000) 2022-09-08T08:08:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:23 smithi055 ceph-mon[97033]: pgmap v20439: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:08:24.855 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:08:24.587+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:08:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:25.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:25.012257+0000) 2022-09-08T08:08:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:24.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:24.853069+0000) 2022-09-08T08:08:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:25 smithi059 ceph-mon[97450]: pgmap v20440: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-08T08:08:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:26.011+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:26.012460+0000) 2022-09-08T08:08:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:25.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:25.853204+0000) 2022-09-08T08:08:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:25 smithi055 ceph-mon[97033]: pgmap v20440: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2022-09-08T08:08:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:27.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:27.012712+0000) 2022-09-08T08:08:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:26.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:26.853422+0000) 2022-09-08T08:08:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:27 smithi059 ceph-mon[97450]: pgmap v20441: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T08:08:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:28.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:28.012874+0000) 2022-09-08T08:08:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:27.852+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:27.853590+0000) 2022-09-08T08:08:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:27 smithi055 ceph-mon[97033]: pgmap v20441: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T08:08:28.941 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:28.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:28.853719+0000) 2022-09-08T08:08:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:29.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:29.013119+0000) 2022-09-08T08:08:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:30.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:30.013348+0000) 2022-09-08T08:08:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:29.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:29.853951+0000) 2022-09-08T08:08:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:29 smithi055 ceph-mon[97033]: pgmap v20442: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:08:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:29 smithi055 ceph-mon[97033]: pgmap v20443: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:08:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:29 smithi055 ceph-mon[97033]: pgmap v20444: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:08:30.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:08:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:29 smithi059 ceph-mon[97450]: pgmap v20442: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:08:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:29 smithi059 ceph-mon[97450]: pgmap v20443: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:08:30.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:29 smithi059 ceph-mon[97450]: pgmap v20444: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:08:30.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:08:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:31.012+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:31.013595+0000) 2022-09-08T08:08:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:30.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:30.854162+0000) 2022-09-08T08:08:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:32.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:32.013833+0000) 2022-09-08T08:08:32.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:31.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:31.854320+0000) 2022-09-08T08:08:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:31 smithi055 ceph-mon[97033]: pgmap v20445: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:08:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:31 smithi059 ceph-mon[97450]: pgmap v20445: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:08:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:33.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:33.014061+0000) 2022-09-08T08:08:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:32.853+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:32.854486+0000) 2022-09-08T08:08:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:34.013+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:34.014307+0000) 2022-09-08T08:08:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:33.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:33.854770+0000) 2022-09-08T08:08:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:33 smithi055 ceph-mon[97033]: pgmap v20446: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:08:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:33 smithi059 ceph-mon[97450]: pgmap v20446: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:08:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:35.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:35.014537+0000) 2022-09-08T08:08:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:34.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:34.854980+0000) 2022-09-08T08:08:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:36.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:36.014732+0000) 2022-09-08T08:08:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:35.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:35.855200+0000) 2022-09-08T08:08:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:35 smithi055 ceph-mon[97033]: pgmap v20447: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 6 op/s 2022-09-08T08:08:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:35 smithi059 ceph-mon[97450]: pgmap v20447: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 6 op/s 2022-09-08T08:08:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:37.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:37.014881+0000) 2022-09-08T08:08:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:36.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:36.855400+0000) 2022-09-08T08:08:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:38.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:38.015080+0000) 2022-09-08T08:08:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:37.854+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:37.855553+0000) 2022-09-08T08:08:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:37 smithi055 ceph-mon[97033]: pgmap v20448: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:08:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:37 smithi059 ceph-mon[97450]: pgmap v20448: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:08:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:39.014+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:39.015277+0000) 2022-09-08T08:08:39.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:38.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:38.855762+0000) 2022-09-08T08:08:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:39 smithi055 ceph-mon[97033]: pgmap v20449: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:08:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:39 smithi059 ceph-mon[97450]: pgmap v20449: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:08:39.858 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:08:39.599+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:08:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:40.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:40.015477+0000) 2022-09-08T08:08:40.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:39.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:39.855924+0000) 2022-09-08T08:08:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:41.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:41.015713+0000) 2022-09-08T08:08:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:40.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:40.856137+0000) 2022-09-08T08:08:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:42.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:42.015963+0000) 2022-09-08T08:08:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:41.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:41.856358+0000) 2022-09-08T08:08:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:41 smithi059 ceph-mon[97450]: pgmap v20450: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:08:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:42 smithi055 ceph-mon[97033]: pgmap v20450: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:08:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:43.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:43.016183+0000) 2022-09-08T08:08:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:42.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:42.856527+0000) 2022-09-08T08:08:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:44.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:44.016363+0000) 2022-09-08T08:08:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:43.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:43.856739+0000) 2022-09-08T08:08:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:44 smithi059 ceph-mon[97450]: pgmap v20451: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:08:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:44 smithi055 ceph-mon[97033]: pgmap v20451: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:08:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:45.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:45.016580+0000) 2022-09-08T08:08:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:44.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:44.856895+0000) 2022-09-08T08:08:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:46.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:46.016820+0000) 2022-09-08T08:08:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:45.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:45.857050+0000) 2022-09-08T08:08:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:46 smithi059 ceph-mon[97450]: pgmap v20452: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:08:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:46 smithi055 ceph-mon[97033]: pgmap v20452: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:08:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:47.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:47.017095+0000) 2022-09-08T08:08:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:46.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:46.857203+0000) 2022-09-08T08:08:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:48.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:48.017310+0000) 2022-09-08T08:08:48.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:47.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:47.857333+0000) 2022-09-08T08:08:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:48 smithi059 ceph-mon[97450]: pgmap v20453: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:08:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:48 smithi055 ceph-mon[97033]: pgmap v20453: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:08:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:48.855+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:48.857467+0000) 2022-09-08T08:08:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:49.015+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:49.017552+0000) 2022-09-08T08:08:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:49 smithi055 ceph-mon[97033]: pgmap v20454: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:08:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:49 smithi059 ceph-mon[97450]: pgmap v20454: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:08:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:50.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:50.017778+0000) 2022-09-08T08:08:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:49.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:49.857625+0000) 2022-09-08T08:08:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:50.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:50.857874+0000) 2022-09-08T08:08:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:51.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:51.017957+0000) 2022-09-08T08:08:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:52.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:52.018173+0000) 2022-09-08T08:08:52.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:51.856+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:51.858086+0000) 2022-09-08T08:08:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:52 smithi059 ceph-mon[97450]: pgmap v20455: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:08:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:52 smithi055 ceph-mon[97033]: pgmap v20455: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:08:53.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:53.016+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:53.018336+0000) 2022-09-08T08:08:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:52.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:52.858221+0000) 2022-09-08T08:08:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:53.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:53.858367+0000) 2022-09-08T08:08:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:54.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:54.018566+0000) 2022-09-08T08:08:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:54 smithi059 ceph-mon[97450]: pgmap v20456: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:08:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:54 smithi055 ceph-mon[97033]: pgmap v20456: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:08:54.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:54.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:54.858604+0000) 2022-09-08T08:08:54.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:08:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:08:54.614+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:08:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:55.017+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:55.018774+0000) 2022-09-08T08:08:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:56.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:56.018988+0000) 2022-09-08T08:08:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:55.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:55.858856+0000) 2022-09-08T08:08:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:56 smithi059 ceph-mon[97450]: pgmap v20457: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:08:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:56 smithi055 ceph-mon[97033]: pgmap v20457: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:08:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:57.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:57.019229+0000) 2022-09-08T08:08:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:56.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:56.859075+0000) 2022-09-08T08:08:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:58.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:58.019453+0000) 2022-09-08T08:08:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:57.857+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:57.859238+0000) 2022-09-08T08:08:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:58 smithi059 ceph-mon[97450]: pgmap v20458: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:08:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:58 smithi055 ceph-mon[97033]: pgmap v20458: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:08:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:08:59.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:59.019647+0000) 2022-09-08T08:08:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:58.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:58.859389+0000) 2022-09-08T08:08:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:08:59 smithi055 ceph-mon[97033]: pgmap v20459: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:08:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:08:59 smithi059 ceph-mon[97450]: pgmap v20459: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:00.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:00.019833+0000) 2022-09-08T08:09:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:08:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:08:59.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:08:59.859619+0000) 2022-09-08T08:09:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:01.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:01.019965+0000) 2022-09-08T08:09:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:00.858+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:00.859754+0000) 2022-09-08T08:09:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:02.018+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:02.020162+0000) 2022-09-08T08:09:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:01.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:01.860001+0000) 2022-09-08T08:09:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:02 smithi059 ceph-mon[97450]: pgmap v20460: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:09:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:02 smithi055 ceph-mon[97033]: pgmap v20460: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:09:03.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:02.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:02.860190+0000) 2022-09-08T08:09:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:03.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:03.020324+0000) 2022-09-08T08:09:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:04.019+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:04.020543+0000) 2022-09-08T08:09:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:03.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:03.860398+0000) 2022-09-08T08:09:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:04 smithi059 ceph-mon[97450]: pgmap v20461: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:04 smithi055 ceph-mon[97033]: pgmap v20461: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:05.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:05.020761+0000) 2022-09-08T08:09:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:04.859+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:04.860573+0000) 2022-09-08T08:09:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:06.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:06.020987+0000) 2022-09-08T08:09:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:05.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:05.860813+0000) 2022-09-08T08:09:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:06 smithi059 ceph-mon[97450]: pgmap v20462: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 22 KiB/s wr, 19 op/s 2022-09-08T08:09:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:06 smithi055 ceph-mon[97033]: pgmap v20462: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 22 KiB/s wr, 19 op/s 2022-09-08T08:09:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:07.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:07.021234+0000) 2022-09-08T08:09:07.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:06.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:06.860944+0000) 2022-09-08T08:09:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:09:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:09:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:09:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:09:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:08.020+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:08.021445+0000) 2022-09-08T08:09:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:07.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:07.861153+0000) 2022-09-08T08:09:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:08 smithi059 ceph-mon[97450]: pgmap v20463: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T08:09:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:08 smithi055 ceph-mon[97033]: pgmap v20463: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T08:09:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:09.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:09.021724+0000) 2022-09-08T08:09:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:08.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:08.861362+0000) 2022-09-08T08:09:09.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:09 smithi055 ceph-mon[97033]: pgmap v20464: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:09:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:09 smithi059 ceph-mon[97450]: pgmap v20464: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:09:09.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:09:09.620+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:09:09.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:09.860+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:09.861516+0000) 2022-09-08T08:09:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:10.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:10.021953+0000) 2022-09-08T08:09:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:10 smithi055 ceph-mon[97033]: osdmap e1357: 12 total, 11 up, 11 in 2022-09-08T08:09:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:10 smithi059 ceph-mon[97450]: osdmap e1357: 12 total, 11 up, 11 in 2022-09-08T08:09:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:10.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:10.861746+0000) 2022-09-08T08:09:11.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:11.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:11.022138+0000) 2022-09-08T08:09:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:11 smithi055 ceph-mon[97033]: pgmap v20466: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:09:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:11 smithi059 ceph-mon[97450]: pgmap v20466: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:09:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:12.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:12.022320+0000) 2022-09-08T08:09:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:11.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:11.861895+0000) 2022-09-08T08:09:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:13.021+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:13.022541+0000) 2022-09-08T08:09:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:12.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:12.862106+0000) 2022-09-08T08:09:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:14.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:14.022800+0000) 2022-09-08T08:09:14.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:13.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:13.862261+0000) 2022-09-08T08:09:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:14 smithi059 ceph-mon[97450]: pgmap v20467: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:09:14.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:14 smithi059 ceph-mon[97450]: osdmap e1358: 12 total, 11 up, 11 in 2022-09-08T08:09:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:14 smithi055 ceph-mon[97033]: pgmap v20467: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:09:14.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:14 smithi055 ceph-mon[97033]: osdmap e1358: 12 total, 11 up, 11 in 2022-09-08T08:09:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:15.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:15.023042+0000) 2022-09-08T08:09:15.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:14.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:14.862415+0000) 2022-09-08T08:09:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:16.022+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:16.023251+0000) 2022-09-08T08:09:16.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:15.861+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:15.862570+0000) 2022-09-08T08:09:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:16 smithi059 ceph-mon[97450]: pgmap v20469: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-08T08:09:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:16 smithi055 ceph-mon[97033]: pgmap v20469: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2022-09-08T08:09:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:17.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:17.023491+0000) 2022-09-08T08:09:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:16.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:16.862813+0000) 2022-09-08T08:09:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:18.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:18.023751+0000) 2022-09-08T08:09:18.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:17.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:17.862973+0000) 2022-09-08T08:09:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:18 smithi059 ceph-mon[97450]: pgmap v20470: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:09:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:18 smithi055 ceph-mon[97033]: pgmap v20470: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:09:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:19.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:19.023935+0000) 2022-09-08T08:09:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:18.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:18.863107+0000) 2022-09-08T08:09:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:19 smithi055 ceph-mon[97033]: pgmap v20471: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T08:09:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:19 smithi059 ceph-mon[97450]: pgmap v20471: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2022-09-08T08:09:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:20.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:20.024163+0000) 2022-09-08T08:09:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:19.862+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:19.863335+0000) 2022-09-08T08:09:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:21.023+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:21.024316+0000) 2022-09-08T08:09:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:20.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:20.863478+0000) 2022-09-08T08:09:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:22.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:22.024542+0000) 2022-09-08T08:09:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:21.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:21.863780+0000) 2022-09-08T08:09:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:22 smithi059 ceph-mon[97450]: pgmap v20472: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:09:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:22 smithi055 ceph-mon[97033]: pgmap v20472: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:09:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:23.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:23.024756+0000) 2022-09-08T08:09:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:22.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:22.863911+0000) 2022-09-08T08:09:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:24.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:24.024988+0000) 2022-09-08T08:09:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:23.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:23.864032+0000) 2022-09-08T08:09:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:24 smithi059 ceph-mon[97450]: pgmap v20473: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:09:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:24 smithi055 ceph-mon[97033]: pgmap v20473: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:09:24.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:09:24.626+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:09:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:24.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:24.864187+0000) 2022-09-08T08:09:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:25.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:25.025228+0000) 2022-09-08T08:09:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:26.024+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:26.025484+0000) 2022-09-08T08:09:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:25.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:25.864379+0000) 2022-09-08T08:09:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:26 smithi059 ceph-mon[97450]: pgmap v20474: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 88 B/s wr, 7 op/s 2022-09-08T08:09:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:26 smithi055 ceph-mon[97033]: pgmap v20474: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.5 KiB/s rd, 88 B/s wr, 7 op/s 2022-09-08T08:09:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:27.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:27.025700+0000) 2022-09-08T08:09:27.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:26.863+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:26.864544+0000) 2022-09-08T08:09:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:28.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:28.025906+0000) 2022-09-08T08:09:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:27.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:27.864708+0000) 2022-09-08T08:09:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:28 smithi059 ceph-mon[97450]: pgmap v20475: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:09:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:28 smithi055 ceph-mon[97033]: pgmap v20475: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:09:29.135 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:29.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:29.026122+0000) 2022-09-08T08:09:29.136 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:28.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:28.864852+0000) 2022-09-08T08:09:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:29 smithi055 ceph-mon[97033]: pgmap v20476: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:09:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:09:29.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:09:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:29 smithi059 ceph-mon[97450]: pgmap v20476: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:09:29.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:09:29.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:09:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:30.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:30.026328+0000) 2022-09-08T08:09:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:29.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:29.865065+0000) 2022-09-08T08:09:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:09:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:09:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:09:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:09:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:31.025+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:31.026476+0000) 2022-09-08T08:09:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:30.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:30.865290+0000) 2022-09-08T08:09:31.925 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:31 smithi059 ceph-mon[97450]: pgmap v20477: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:09:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:31 smithi055 ceph-mon[97033]: pgmap v20477: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:09:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:32.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:32.026736+0000) 2022-09-08T08:09:32.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:31.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:31.865546+0000) 2022-09-08T08:09:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:32.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:32.865715+0000) 2022-09-08T08:09:33.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:33.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:33.026990+0000) 2022-09-08T08:09:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:09:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:09:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:33.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:33.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:33.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:09:33.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:09:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:09:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:09:33.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:33.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:33.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:33.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:33.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:09:33.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:09:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:34.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:34.027198+0000) 2022-09-08T08:09:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:33.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:33.865911+0000) 2022-09-08T08:09:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:33 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 94551k 2022-09-08T08:09:34.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:33 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 96820695: error parsing value: Value '96820695' is below minimum 939524096 2022-09-08T08:09:34.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:34.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:34.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:34.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:33 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:34.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:33 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T08:09:34.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:33 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T08:09:34.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:33 smithi055 ceph-mon[97033]: pgmap v20478: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:33 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 94551k 2022-09-08T08:09:34.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:33 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 96820695: error parsing value: Value '96820695' is below minimum 939524096 2022-09-08T08:09:34.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:34.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:34.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:34.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:33 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:09:34.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:33 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T08:09:34.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:33 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T08:09:34.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:33 smithi059 ceph-mon[97450]: pgmap v20478: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:35.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:35.027403+0000) 2022-09-08T08:09:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:34.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:34.866105+0000) 2022-09-08T08:09:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:36.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:36.027674+0000) 2022-09-08T08:09:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:35.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:35.866334+0000) 2022-09-08T08:09:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:36 smithi059 ceph-mon[97450]: pgmap v20479: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:09:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:36 smithi055 ceph-mon[97033]: pgmap v20479: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:09:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:36.864+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:36.866494+0000) 2022-09-08T08:09:37.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:37.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:37.027812+0000) 2022-09-08T08:09:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:37.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:37.866712+0000) 2022-09-08T08:09:38.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:38.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:38.027996+0000) 2022-09-08T08:09:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:38 smithi059 ceph-mon[97450]: pgmap v20480: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:09:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:38 smithi055 ceph-mon[97033]: pgmap v20480: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:09:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:39.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:39.028172+0000) 2022-09-08T08:09:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:38.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:38.866843+0000) 2022-09-08T08:09:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:39 smithi055 ceph-mon[97033]: pgmap v20481: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:39 smithi059 ceph-mon[97450]: pgmap v20481: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:39.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:09:39.627+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:09:39.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:39.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:39.867060+0000) 2022-09-08T08:09:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:40.026+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:40.028363+0000) 2022-09-08T08:09:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:41.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:41.028675+0000) 2022-09-08T08:09:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:40.865+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:40.867204+0000) 2022-09-08T08:09:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:42.027+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:42.028913+0000) 2022-09-08T08:09:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:41.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:41.867355+0000) 2022-09-08T08:09:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:42 smithi059 ceph-mon[97450]: pgmap v20482: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:09:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:42 smithi055 ceph-mon[97033]: pgmap v20482: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:09:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:42.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:42.867502+0000) 2022-09-08T08:09:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:43.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:43.029123+0000) 2022-09-08T08:09:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:43.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:43.867665+0000) 2022-09-08T08:09:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:44.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:44.029290+0000) 2022-09-08T08:09:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:44 smithi059 ceph-mon[97450]: pgmap v20483: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:44 smithi055 ceph-mon[97033]: pgmap v20483: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:45.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:45.029523+0000) 2022-09-08T08:09:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:44.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:44.867834+0000) 2022-09-08T08:09:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:46.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:46.029754+0000) 2022-09-08T08:09:46.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:45.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:45.867997+0000) 2022-09-08T08:09:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:46 smithi059 ceph-mon[97450]: pgmap v20484: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:09:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:46 smithi055 ceph-mon[97033]: pgmap v20484: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:09:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:47.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:47.029981+0000) 2022-09-08T08:09:47.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:46.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:46.868149+0000) 2022-09-08T08:09:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:48.028+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:48.030184+0000) 2022-09-08T08:09:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:47.866+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:47.868295+0000) 2022-09-08T08:09:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:48 smithi059 ceph-mon[97450]: pgmap v20485: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:09:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:48 smithi055 ceph-mon[97033]: pgmap v20485: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:09:49.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:48.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:48.868444+0000) 2022-09-08T08:09:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:49.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:49.030409+0000) 2022-09-08T08:09:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:49 smithi055 ceph-mon[97033]: pgmap v20486: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:49 smithi059 ceph-mon[97450]: pgmap v20486: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:50.029+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:50.030609+0000) 2022-09-08T08:09:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:49.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:49.868631+0000) 2022-09-08T08:09:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:51.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:51.030750+0000) 2022-09-08T08:09:51.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:50.867+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:50.868798+0000) 2022-09-08T08:09:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:52.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:52.030992+0000) 2022-09-08T08:09:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:51.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:51.868940+0000) 2022-09-08T08:09:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:52 smithi059 ceph-mon[97450]: pgmap v20487: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:09:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:52 smithi055 ceph-mon[97033]: pgmap v20487: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:09:53.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:53.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:53.031164+0000) 2022-09-08T08:09:53.157 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:52.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:52.869088+0000) 2022-09-08T08:09:54.149 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:54.030+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:54.031401+0000) 2022-09-08T08:09:54.149 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:53.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:53.869236+0000) 2022-09-08T08:09:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:54 smithi059 ceph-mon[97450]: pgmap v20488: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:54 smithi055 ceph-mon[97033]: pgmap v20488: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:09:54.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:09:54.643+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:09:54.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:54.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:54.869459+0000) 2022-09-08T08:09:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:55.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:55.031703+0000) 2022-09-08T08:09:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:56.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:56.031845+0000) 2022-09-08T08:09:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:55.868+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:55.869624+0000) 2022-09-08T08:09:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:56 smithi059 ceph-mon[97450]: pgmap v20489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:09:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:56 smithi055 ceph-mon[97033]: pgmap v20489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:09:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:57.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:57.032020+0000) 2022-09-08T08:09:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:56.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:56.869795+0000) 2022-09-08T08:09:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:58.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:58.032236+0000) 2022-09-08T08:09:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:57.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:57.869924+0000) 2022-09-08T08:09:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:09:58 smithi055 ceph-mon[97033]: pgmap v20490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:09:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:09:58 smithi059 ceph-mon[97450]: pgmap v20490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:09:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:09:59.031+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:59.032482+0000) 2022-09-08T08:09:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:58.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:58.870039+0000) 2022-09-08T08:10:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:00.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:00.032748+0000) 2022-09-08T08:10:00.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:09:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:09:59.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:09:59.870276+0000) 2022-09-08T08:10:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:00 smithi055 ceph-mon[97033]: pgmap v20491: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:00 smithi055 ceph-mon[97033]: pgmap v20492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:10:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:00 smithi055 ceph-mon[97033]: pgmap v20493: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:10:00.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:10:00.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T08:10:00.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T08:10:00.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T08:10:00.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T08:10:00.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T08:10:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:00 smithi059 ceph-mon[97450]: pgmap v20491: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:00 smithi059 ceph-mon[97450]: pgmap v20492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:10:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:00 smithi059 ceph-mon[97450]: pgmap v20493: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:10:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:10:00.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T08:10:00.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T08:10:00.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T08:10:00.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T08:10:00.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T08:10:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:01.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:01.032900+0000) 2022-09-08T08:10:01.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:00.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:00.870453+0000) 2022-09-08T08:10:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:02.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:02.033034+0000) 2022-09-08T08:10:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:01.869+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:01.870612+0000) 2022-09-08T08:10:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:02 smithi055 ceph-mon[97033]: pgmap v20494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:10:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:02 smithi059 ceph-mon[97450]: pgmap v20494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:10:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:03.032+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:03.033238+0000) 2022-09-08T08:10:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:02.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:02.870796+0000) 2022-09-08T08:10:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:04.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:04.033496+0000) 2022-09-08T08:10:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:03.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:03.871051+0000) 2022-09-08T08:10:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:04 smithi055 ceph-mon[97033]: pgmap v20495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:10:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:04 smithi059 ceph-mon[97450]: pgmap v20495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:10:05.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:04.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:04.871272+0000) 2022-09-08T08:10:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:05.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:05.033737+0000) 2022-09-08T08:10:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:06.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:06.033928+0000) 2022-09-08T08:10:06.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:05.870+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:05.871535+0000) 2022-09-08T08:10:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:06 smithi055 ceph-mon[97033]: pgmap v20496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T08:10:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:10:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:06 smithi059 ceph-mon[97450]: pgmap v20496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T08:10:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:10:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:06.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:06.871768+0000) 2022-09-08T08:10:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:07.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:07.034172+0000) 2022-09-08T08:10:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:10:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:10:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:08.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:08.034427+0000) 2022-09-08T08:10:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:07.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:07.871959+0000) 2022-09-08T08:10:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:08 smithi059 ceph-mon[97450]: pgmap v20497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T08:10:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:08 smithi055 ceph-mon[97033]: pgmap v20497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T08:10:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:08.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:08.872120+0000) 2022-09-08T08:10:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:09.033+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:09.034557+0000) 2022-09-08T08:10:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:09 smithi055 ceph-mon[97033]: pgmap v20498: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:10:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:09 smithi059 ceph-mon[97450]: pgmap v20498: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:10:09.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:10:09.648+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:10:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:09.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:09.872273+0000) 2022-09-08T08:10:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:10.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:10.034764+0000) 2022-09-08T08:10:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:10 smithi055 ceph-mon[97033]: osdmap e1359: 12 total, 11 up, 11 in 2022-09-08T08:10:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:10 smithi059 ceph-mon[97450]: osdmap e1359: 12 total, 11 up, 11 in 2022-09-08T08:10:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:11.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:11.035001+0000) 2022-09-08T08:10:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:10.871+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:10.872447+0000) 2022-09-08T08:10:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:11 smithi055 ceph-mon[97033]: pgmap v20500: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:10:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:11 smithi059 ceph-mon[97450]: pgmap v20500: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:10:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:12.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:12.035218+0000) 2022-09-08T08:10:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:11.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:11.872621+0000) 2022-09-08T08:10:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:13.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:13.035466+0000) 2022-09-08T08:10:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:12.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:12.872817+0000) 2022-09-08T08:10:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:14.034+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:14.035604+0000) 2022-09-08T08:10:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:13.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:13.873064+0000) 2022-09-08T08:10:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:14 smithi059 ceph-mon[97450]: pgmap v20501: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:10:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:14 smithi055 ceph-mon[97033]: pgmap v20501: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:10:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:15.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:15.035821+0000) 2022-09-08T08:10:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:14.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:14.873227+0000) 2022-09-08T08:10:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:16.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:16.036020+0000) 2022-09-08T08:10:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:15.872+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:15.873360+0000) 2022-09-08T08:10:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:16 smithi059 ceph-mon[97450]: pgmap v20502: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:10:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:16 smithi055 ceph-mon[97033]: pgmap v20502: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:10:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:17.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:17.036231+0000) 2022-09-08T08:10:17.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:16.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:16.873520+0000) 2022-09-08T08:10:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:18.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:18.036457+0000) 2022-09-08T08:10:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:17.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:17.873778+0000) 2022-09-08T08:10:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:18 smithi059 ceph-mon[97450]: pgmap v20503: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:10:18.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:18 smithi055 ceph-mon[97033]: pgmap v20503: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:10:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:19.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:19.036615+0000) 2022-09-08T08:10:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:18.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:18.873951+0000) 2022-09-08T08:10:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:19 smithi055 ceph-mon[97033]: osdmap e1360: 12 total, 11 up, 11 in 2022-09-08T08:10:19.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:19 smithi055 ceph-mon[97033]: pgmap v20505: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2022-09-08T08:10:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:19 smithi059 ceph-mon[97450]: osdmap e1360: 12 total, 11 up, 11 in 2022-09-08T08:10:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:19 smithi059 ceph-mon[97450]: pgmap v20505: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2022-09-08T08:10:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:20.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:20.036852+0000) 2022-09-08T08:10:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:19.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:19.874104+0000) 2022-09-08T08:10:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:21.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:21.037029+0000) 2022-09-08T08:10:21.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:20.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:20.874328+0000) 2022-09-08T08:10:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:22.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:22.037237+0000) 2022-09-08T08:10:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:21.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:21.874536+0000) 2022-09-08T08:10:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:22 smithi059 ceph-mon[97450]: pgmap v20506: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:10:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:22 smithi055 ceph-mon[97033]: pgmap v20506: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:10:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:23.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:23.037457+0000) 2022-09-08T08:10:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:22.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:22.874699+0000) 2022-09-08T08:10:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:24.035+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:24.037599+0000) 2022-09-08T08:10:24.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:23.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:23.874907+0000) 2022-09-08T08:10:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:24 smithi059 ceph-mon[97450]: pgmap v20507: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:10:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:24 smithi055 ceph-mon[97033]: pgmap v20507: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:10:25.040 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:10:24.663+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:10:25.041 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:24.873+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:24.875055+0000) 2022-09-08T08:10:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:25.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:25.037821+0000) 2022-09-08T08:10:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:26.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:26.038128+0000) 2022-09-08T08:10:26.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:25.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:25.875267+0000) 2022-09-08T08:10:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:26 smithi059 ceph-mon[97450]: pgmap v20508: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:10:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:26 smithi055 ceph-mon[97033]: pgmap v20508: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:10:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:27.036+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:27.038325+0000) 2022-09-08T08:10:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:26.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:26.875433+0000) 2022-09-08T08:10:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:28.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:28.038547+0000) 2022-09-08T08:10:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:27.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:27.875701+0000) 2022-09-08T08:10:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:28 smithi059 ceph-mon[97450]: pgmap v20509: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:10:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:28 smithi055 ceph-mon[97033]: pgmap v20509: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:10:29.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:28.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:28.875929+0000) 2022-09-08T08:10:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:29.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:29.038751+0000) 2022-09-08T08:10:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:29 smithi055 ceph-mon[97033]: pgmap v20510: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:10:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:29 smithi059 ceph-mon[97450]: pgmap v20510: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:10:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:30.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:30.038970+0000) 2022-09-08T08:10:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:29.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:29.876106+0000) 2022-09-08T08:10:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:31.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:31.039200+0000) 2022-09-08T08:10:31.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:30.874+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:30.876307+0000) 2022-09-08T08:10:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:32.037+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:32.039357+0000) 2022-09-08T08:10:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:31.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:31.876533+0000) 2022-09-08T08:10:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:32 smithi059 ceph-mon[97450]: pgmap v20511: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:10:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:32 smithi055 ceph-mon[97033]: pgmap v20511: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:10:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:33.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:33.039560+0000) 2022-09-08T08:10:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:32.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:32.876715+0000) 2022-09-08T08:10:34.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:33.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:33.876928+0000) 2022-09-08T08:10:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:34.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:34.039750+0000) 2022-09-08T08:10:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:34 smithi059 ceph-mon[97450]: pgmap v20512: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:34 smithi055 ceph-mon[97033]: pgmap v20512: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:35.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:34.875+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:34.877076+0000) 2022-09-08T08:10:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:35.038+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:35.039884+0000) 2022-09-08T08:10:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:36.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:36.040036+0000) 2022-09-08T08:10:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:35.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:35.877250+0000) 2022-09-08T08:10:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:36 smithi059 ceph-mon[97450]: pgmap v20513: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:10:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:36 smithi055 ceph-mon[97033]: pgmap v20513: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:10:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:37.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:37.040251+0000) 2022-09-08T08:10:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:36.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:36.877423+0000) 2022-09-08T08:10:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:38.039+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:38.040494+0000) 2022-09-08T08:10:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:37.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:37.877674+0000) 2022-09-08T08:10:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:38 smithi059 ceph-mon[97450]: pgmap v20514: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:38 smithi055 ceph-mon[97033]: pgmap v20514: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:39.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:39.040724+0000) 2022-09-08T08:10:39.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:38.876+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:38.877842+0000) 2022-09-08T08:10:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:39 smithi055 ceph-mon[97033]: pgmap v20515: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:39 smithi059 ceph-mon[97450]: pgmap v20515: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:40.043 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:10:39.678+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:10:40.044 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:39.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:39.877979+0000) 2022-09-08T08:10:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:40.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:40.040938+0000) 2022-09-08T08:10:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:40.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:40.878143+0000) 2022-09-08T08:10:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:41.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:41.041129+0000) 2022-09-08T08:10:42.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:41.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:41.878305+0000) 2022-09-08T08:10:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:42.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:42.041322+0000) 2022-09-08T08:10:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:42 smithi059 ceph-mon[97450]: pgmap v20516: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:10:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:42 smithi055 ceph-mon[97033]: pgmap v20516: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:10:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:43.040+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:43.041532+0000) 2022-09-08T08:10:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:42.877+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:42.878440+0000) 2022-09-08T08:10:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:44.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:44.041752+0000) 2022-09-08T08:10:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:43.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:43.878703+0000) 2022-09-08T08:10:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:44 smithi055 ceph-mon[97033]: pgmap v20517: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:44 smithi059 ceph-mon[97450]: pgmap v20517: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:45.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:45.041992+0000) 2022-09-08T08:10:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:44.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:44.878877+0000) 2022-09-08T08:10:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:46.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:46.042242+0000) 2022-09-08T08:10:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:45.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:45.879030+0000) 2022-09-08T08:10:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:46 smithi055 ceph-mon[97033]: pgmap v20518: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:10:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:46 smithi059 ceph-mon[97450]: pgmap v20518: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:10:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:47.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:47.042402+0000) 2022-09-08T08:10:47.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:46.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:46.879200+0000) 2022-09-08T08:10:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:48.041+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:48.042596+0000) 2022-09-08T08:10:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:47.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:47.879407+0000) 2022-09-08T08:10:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:48 smithi055 ceph-mon[97033]: pgmap v20519: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:48 smithi059 ceph-mon[97450]: pgmap v20519: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:49.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:49.042814+0000) 2022-09-08T08:10:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:48.878+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:48.879541+0000) 2022-09-08T08:10:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:49 smithi055 ceph-mon[97033]: pgmap v20520: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:49 smithi059 ceph-mon[97450]: pgmap v20520: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:50.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:49.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:49.879731+0000) 2022-09-08T08:10:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:50.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:50.043015+0000) 2022-09-08T08:10:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:51.042+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:51.043244+0000) 2022-09-08T08:10:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:50.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:50.879952+0000) 2022-09-08T08:10:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:52.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:52.043471+0000) 2022-09-08T08:10:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:51.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:51.880088+0000) 2022-09-08T08:10:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:52 smithi059 ceph-mon[97450]: pgmap v20521: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:10:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:52 smithi055 ceph-mon[97033]: pgmap v20521: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:10:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:53.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:53.043748+0000) 2022-09-08T08:10:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:52.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:52.880291+0000) 2022-09-08T08:10:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:54.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:54.043933+0000) 2022-09-08T08:10:54.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:53.879+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:53.880496+0000) 2022-09-08T08:10:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:54 smithi059 ceph-mon[97450]: pgmap v20522: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:54 smithi055 ceph-mon[97033]: pgmap v20522: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:55.046 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:10:54.693+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:10:55.047 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:54.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:54.880689+0000) 2022-09-08T08:10:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:55.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:55.044161+0000) 2022-09-08T08:10:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:56.043+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:56.044396+0000) 2022-09-08T08:10:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:55.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:55.880854+0000) 2022-09-08T08:10:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:56 smithi055 ceph-mon[97033]: pgmap v20523: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:10:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:56 smithi059 ceph-mon[97450]: pgmap v20523: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:10:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:57.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:57.044609+0000) 2022-09-08T08:10:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:56.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:56.880982+0000) 2022-09-08T08:10:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:58.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:58.044813+0000) 2022-09-08T08:10:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:57.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:57.881143+0000) 2022-09-08T08:10:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:58 smithi055 ceph-mon[97033]: pgmap v20524: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:58 smithi059 ceph-mon[97450]: pgmap v20524: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:10:59.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:59.045048+0000) 2022-09-08T08:10:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:58.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:58.881283+0000) 2022-09-08T08:10:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:59 smithi055 ceph-mon[97033]: pgmap v20525: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:10:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:10:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:10:59 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:10:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:59 smithi059 ceph-mon[97450]: pgmap v20525: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:10:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:10:59.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:10:59.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:10:59 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:11:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:00.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:00.045321+0000) 2022-09-08T08:11:00.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:10:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:10:59.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:10:59.881434+0000) 2022-09-08T08:11:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:01.044+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:01.045473+0000) 2022-09-08T08:11:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:00.880+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:00.881587+0000) 2022-09-08T08:11:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:11:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:11:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:11:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:11:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:02.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:02.045721+0000) 2022-09-08T08:11:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:01.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:01.881765+0000) 2022-09-08T08:11:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:01 smithi055 ceph-mon[97033]: pgmap v20526: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:11:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:01 smithi059 ceph-mon[97450]: pgmap v20526: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:11:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:03.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:03.045977+0000) 2022-09-08T08:11:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:02.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:02.881899+0000) 2022-09-08T08:11:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:04.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:04.046142+0000) 2022-09-08T08:11:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:03.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:03.882076+0000) 2022-09-08T08:11:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:04 smithi055 ceph-mon[97033]: pgmap v20527: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:11:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:04 smithi059 ceph-mon[97450]: pgmap v20527: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:11:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:05.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:05.046319+0000) 2022-09-08T08:11:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:04.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:04.882282+0000) 2022-09-08T08:11:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:06.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:06.046505+0000) 2022-09-08T08:11:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:05.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:05.882474+0000) 2022-09-08T08:11:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:06 smithi055 ceph-mon[97033]: pgmap v20528: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:11:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:11:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:06 smithi059 ceph-mon[97450]: pgmap v20528: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:11:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:11:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:07.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:07.046751+0000) 2022-09-08T08:11:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:06.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:06.882752+0000) 2022-09-08T08:11:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:11:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:11:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:08.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:08.046949+0000) 2022-09-08T08:11:08.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:07.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:07.882911+0000) 2022-09-08T08:11:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:08 smithi055 ceph-mon[97033]: pgmap v20529: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:11:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:08 smithi059 ceph-mon[97450]: pgmap v20529: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:11:09.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:09.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:09.047154+0000) 2022-09-08T08:11:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:08.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:08.883110+0000) 2022-09-08T08:11:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:09 smithi055 ceph-mon[97033]: pgmap v20530: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:11:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:09 smithi059 ceph-mon[97450]: pgmap v20530: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:11:10.050 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:11:09.708+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:11:10.050 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:09.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:09.883340+0000) 2022-09-08T08:11:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:10.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:10.047362+0000) 2022-09-08T08:11:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:10 smithi055 ceph-mon[97033]: osdmap e1361: 12 total, 11 up, 11 in 2022-09-08T08:11:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:10 smithi059 ceph-mon[97450]: osdmap e1361: 12 total, 11 up, 11 in 2022-09-08T08:11:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:11.045+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:11.047540+0000) 2022-09-08T08:11:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:10.881+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:10.883502+0000) 2022-09-08T08:11:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:11 smithi055 ceph-mon[97033]: pgmap v20532: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:11:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:11 smithi059 ceph-mon[97450]: pgmap v20532: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:11:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:12.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:12.047741+0000) 2022-09-08T08:11:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:11.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:11.883677+0000) 2022-09-08T08:11:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:13.046+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:13.047948+0000) 2022-09-08T08:11:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:12.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:12.883849+0000) 2022-09-08T08:11:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:14.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:14.048124+0000) 2022-09-08T08:11:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:13.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:13.883995+0000) 2022-09-08T08:11:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:14 smithi055 ceph-mon[97033]: pgmap v20533: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:11:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:14 smithi059 ceph-mon[97450]: pgmap v20533: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:11:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:15.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:15.048349+0000) 2022-09-08T08:11:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:14.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:14.884169+0000) 2022-09-08T08:11:16.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:16.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:16.048667+0000) 2022-09-08T08:11:16.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:15.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:15.884385+0000) 2022-09-08T08:11:16.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:16 smithi055 ceph-mon[97033]: pgmap v20534: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:11:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:16 smithi059 ceph-mon[97450]: pgmap v20534: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:11:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:17.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:17.048879+0000) 2022-09-08T08:11:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:16.882+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:16.884571+0000) 2022-09-08T08:11:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:18.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:18.049100+0000) 2022-09-08T08:11:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:17.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:17.884764+0000) 2022-09-08T08:11:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:18 smithi055 ceph-mon[97033]: pgmap v20535: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:11:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:18 smithi059 ceph-mon[97450]: pgmap v20535: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:11:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:19.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:19.049299+0000) 2022-09-08T08:11:19.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:18.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:18.884914+0000) 2022-09-08T08:11:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:19 smithi055 ceph-mon[97033]: osdmap e1362: 12 total, 11 up, 11 in 2022-09-08T08:11:19.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:19 smithi055 ceph-mon[97033]: pgmap v20537: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2022-09-08T08:11:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:19 smithi059 ceph-mon[97450]: osdmap e1362: 12 total, 11 up, 11 in 2022-09-08T08:11:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:19 smithi059 ceph-mon[97450]: pgmap v20537: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2022-09-08T08:11:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:20.047+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:20.049535+0000) 2022-09-08T08:11:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:19.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:19.885107+0000) 2022-09-08T08:11:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:21.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:21.049767+0000) 2022-09-08T08:11:21.160 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:20.883+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:20.885319+0000) 2022-09-08T08:11:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:22.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:22.049994+0000) 2022-09-08T08:11:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:21.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:21.885524+0000) 2022-09-08T08:11:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:22 smithi055 ceph-mon[97033]: pgmap v20538: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:11:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:22 smithi059 ceph-mon[97450]: pgmap v20538: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:11:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:23.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:23.050172+0000) 2022-09-08T08:11:23.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:22.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:22.885701+0000) 2022-09-08T08:11:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:24.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:24.050327+0000) 2022-09-08T08:11:24.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:23.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:23.885839+0000) 2022-09-08T08:11:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:24 smithi055 ceph-mon[97033]: pgmap v20539: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:11:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:24 smithi059 ceph-mon[97450]: pgmap v20539: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:11:25.052 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:11:24.724+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:11:25.053 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:24.884+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:24.886008+0000) 2022-09-08T08:11:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:25.048+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:25.050529+0000) 2022-09-08T08:11:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:26.049+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:26.050691+0000) 2022-09-08T08:11:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:25.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:25.886147+0000) 2022-09-08T08:11:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:26 smithi055 ceph-mon[97033]: pgmap v20540: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:11:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:26 smithi059 ceph-mon[97450]: pgmap v20540: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:11:27.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:27.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:27.050925+0000) 2022-09-08T08:11:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:26.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:26.886322+0000) 2022-09-08T08:11:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:28.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:28.051138+0000) 2022-09-08T08:11:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:27.885+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:27.886510+0000) 2022-09-08T08:11:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:28 smithi055 ceph-mon[97033]: pgmap v20541: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:11:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:28 smithi059 ceph-mon[97450]: pgmap v20541: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:11:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:29.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:29.051382+0000) 2022-09-08T08:11:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:28.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:28.886700+0000) 2022-09-08T08:11:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:29 smithi055 ceph-mon[97033]: pgmap v20542: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:11:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:29 smithi059 ceph-mon[97450]: pgmap v20542: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:11:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:30.050+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:30.051614+0000) 2022-09-08T08:11:30.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:29.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:29.886905+0000) 2022-09-08T08:11:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:30 smithi059 ceph-mon[97450]: pgmap v20543: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:11:30.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:30 smithi059 ceph-mon[97450]: pgmap v20544: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:11:30.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:11:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:30 smithi055 ceph-mon[97033]: pgmap v20543: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:11:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:30 smithi055 ceph-mon[97033]: pgmap v20544: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:11:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:11:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:31.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:31.051850+0000) 2022-09-08T08:11:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:30.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:30.887084+0000) 2022-09-08T08:11:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:32.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:32.052088+0000) 2022-09-08T08:11:32.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:31.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:31.887221+0000) 2022-09-08T08:11:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:32 smithi059 ceph-mon[97450]: pgmap v20545: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:11:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:32 smithi055 ceph-mon[97033]: pgmap v20545: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:11:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:33.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:33.052318+0000) 2022-09-08T08:11:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:32.886+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:32.887400+0000) 2022-09-08T08:11:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:34.051+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:34.052476+0000) 2022-09-08T08:11:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:33.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:33.887686+0000) 2022-09-08T08:11:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:34 smithi055 ceph-mon[97033]: pgmap v20546: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:11:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:34 smithi059 ceph-mon[97450]: pgmap v20546: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:11:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:35.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:35.052646+0000) 2022-09-08T08:11:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:34.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:34.887857+0000) 2022-09-08T08:11:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:36.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:36.052857+0000) 2022-09-08T08:11:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:35.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:35.887988+0000) 2022-09-08T08:11:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:36 smithi055 ceph-mon[97033]: pgmap v20547: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-09-08T08:11:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:36 smithi059 ceph-mon[97450]: pgmap v20547: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-09-08T08:11:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:37.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:37.053020+0000) 2022-09-08T08:11:37.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:36.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:36.888214+0000) 2022-09-08T08:11:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:38.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:38.053272+0000) 2022-09-08T08:11:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:37.887+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:37.888421+0000) 2022-09-08T08:11:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:38 smithi055 ceph-mon[97033]: pgmap v20548: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-09-08T08:11:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:38 smithi059 ceph-mon[97450]: pgmap v20548: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-09-08T08:11:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:39.052+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:39.053508+0000) 2022-09-08T08:11:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:38.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:38.888617+0000) 2022-09-08T08:11:40.056 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:11:39.738+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:11:40.056 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:39.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:39.888780+0000) 2022-09-08T08:11:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:40.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:40.053725+0000) 2022-09-08T08:11:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:40 smithi055 ceph-mon[97033]: pgmap v20549: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:11:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:40 smithi059 ceph-mon[97450]: pgmap v20549: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:11:41.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:40.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:40.888963+0000) 2022-09-08T08:11:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:41.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:41.053946+0000) 2022-09-08T08:11:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:41.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:41.889160+0000) 2022-09-08T08:11:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:42.053+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:42.054177+0000) 2022-09-08T08:11:42.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:42 smithi055 ceph-mon[97033]: pgmap v20550: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:11:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:42 smithi059 ceph-mon[97450]: pgmap v20550: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:11:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:43.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:43.054429+0000) 2022-09-08T08:11:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:42.888+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:42.889357+0000) 2022-09-08T08:11:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:44.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:44.054663+0000) 2022-09-08T08:11:44.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:43.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:43.889570+0000) 2022-09-08T08:11:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:44 smithi055 ceph-mon[97033]: pgmap v20551: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:11:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:44 smithi059 ceph-mon[97450]: pgmap v20551: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:11:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:45.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:45.054873+0000) 2022-09-08T08:11:45.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:44.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:44.889798+0000) 2022-09-08T08:11:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:46.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:46.055142+0000) 2022-09-08T08:11:46.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:45.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:45.889992+0000) 2022-09-08T08:11:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:46 smithi055 ceph-mon[97033]: pgmap v20552: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:11:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:46 smithi059 ceph-mon[97450]: pgmap v20552: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:11:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:47.054+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:47.055351+0000) 2022-09-08T08:11:47.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:46.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:46.890154+0000) 2022-09-08T08:11:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:48.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:48.055607+0000) 2022-09-08T08:11:48.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:47.889+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:47.890278+0000) 2022-09-08T08:11:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:48 smithi055 ceph-mon[97033]: pgmap v20553: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:11:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:48 smithi059 ceph-mon[97450]: pgmap v20553: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:11:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:49.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:49.055927+0000) 2022-09-08T08:11:49.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:48.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:48.890448+0000) 2022-09-08T08:11:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:50.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:50.056154+0000) 2022-09-08T08:11:50.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:49.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:49.890703+0000) 2022-09-08T08:11:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:50 smithi055 ceph-mon[97033]: pgmap v20554: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:11:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:50 smithi059 ceph-mon[97450]: pgmap v20554: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:11:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:51.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:51.056328+0000) 2022-09-08T08:11:51.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:50.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:50.890843+0000) 2022-09-08T08:11:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:52.055+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:52.056578+0000) 2022-09-08T08:11:52.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:51.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:51.891051+0000) 2022-09-08T08:11:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:52 smithi055 ceph-mon[97033]: pgmap v20555: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:11:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:52 smithi059 ceph-mon[97450]: pgmap v20555: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:11:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:53.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:53.056851+0000) 2022-09-08T08:11:53.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:52.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:52.891205+0000) 2022-09-08T08:11:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:54.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:54.057088+0000) 2022-09-08T08:11:54.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:53.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:53.891390+0000) 2022-09-08T08:11:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:54 smithi055 ceph-mon[97033]: pgmap v20556: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:11:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:54 smithi059 ceph-mon[97450]: pgmap v20556: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:11:55.059 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:11:54.754+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:11:55.060 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:54.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:54.891614+0000) 2022-09-08T08:11:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:55.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:55.057285+0000) 2022-09-08T08:11:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:56.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:56.057496+0000) 2022-09-08T08:11:56.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:55.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:55.891780+0000) 2022-09-08T08:11:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:56 smithi055 ceph-mon[97033]: pgmap v20557: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:11:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:56 smithi059 ceph-mon[97450]: pgmap v20557: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:11:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:57.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:57.057667+0000) 2022-09-08T08:11:57.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:56.890+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:56.892015+0000) 2022-09-08T08:11:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:58.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:58.057785+0000) 2022-09-08T08:11:58.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:57.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:57.892208+0000) 2022-09-08T08:11:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:11:58 smithi055 ceph-mon[97033]: pgmap v20558: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:11:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:11:58 smithi059 ceph-mon[97450]: pgmap v20558: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:11:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:11:59.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:59.057970+0000) 2022-09-08T08:11:59.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:58.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:58.892375+0000) 2022-09-08T08:12:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:00.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:00.058174+0000) 2022-09-08T08:12:00.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:11:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:11:59.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:11:59.892544+0000) 2022-09-08T08:12:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:00 smithi055 ceph-mon[97033]: pgmap v20559: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:00 smithi059 ceph-mon[97450]: pgmap v20559: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:01.056+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:01.058395+0000) 2022-09-08T08:12:01.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:00.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:00.892791+0000) 2022-09-08T08:12:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:02.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:02.058598+0000) 2022-09-08T08:12:02.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:01.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:01.893000+0000) 2022-09-08T08:12:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:02 smithi055 ceph-mon[97033]: pgmap v20560: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:12:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:02 smithi059 ceph-mon[97450]: pgmap v20560: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:12:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:03.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:03.058809+0000) 2022-09-08T08:12:03.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:02.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:02.893215+0000) 2022-09-08T08:12:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:04.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:04.059011+0000) 2022-09-08T08:12:04.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:03.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:03.893396+0000) 2022-09-08T08:12:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:04 smithi055 ceph-mon[97033]: pgmap v20561: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:04 smithi059 ceph-mon[97450]: pgmap v20561: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:05.057+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:05.059202+0000) 2022-09-08T08:12:05.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:04.891+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:04.893551+0000) 2022-09-08T08:12:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:06.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:06.059423+0000) 2022-09-08T08:12:06.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:05.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:05.893784+0000) 2022-09-08T08:12:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:06 smithi055 ceph-mon[97033]: pgmap v20562: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:12:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:12:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:06 smithi059 ceph-mon[97450]: pgmap v20562: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:12:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:12:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:07.058+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:07.059661+0000) 2022-09-08T08:12:07.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:06.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:06.893964+0000) 2022-09-08T08:12:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:12:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:12:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:08.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:08.059875+0000) 2022-09-08T08:12:08.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:07.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:07.894131+0000) 2022-09-08T08:12:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:08 smithi055 ceph-mon[97033]: pgmap v20563: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:12:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:08 smithi059 ceph-mon[97450]: pgmap v20563: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:12:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:09.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:09.060083+0000) 2022-09-08T08:12:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:08.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:08.894303+0000) 2022-09-08T08:12:10.062 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:12:09.768+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:12:10.063 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:09.892+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:09.894552+0000) 2022-09-08T08:12:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:10.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:10.060226+0000) 2022-09-08T08:12:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:10 smithi055 ceph-mon[97033]: pgmap v20564: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:12:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:10 smithi055 ceph-mon[97033]: osdmap e1363: 12 total, 11 up, 11 in 2022-09-08T08:12:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:10 smithi059 ceph-mon[97450]: pgmap v20564: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:12:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:10 smithi059 ceph-mon[97450]: osdmap e1363: 12 total, 11 up, 11 in 2022-09-08T08:12:11.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:10.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:10.894766+0000) 2022-09-08T08:12:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:11.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:11.060430+0000) 2022-09-08T08:12:12.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:11.893+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:11.894930+0000) 2022-09-08T08:12:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:12.059+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:12.060657+0000) 2022-09-08T08:12:12.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:12 smithi055 ceph-mon[97033]: pgmap v20566: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:12:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:12 smithi059 ceph-mon[97450]: pgmap v20566: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:12:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:13.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:13.060892+0000) 2022-09-08T08:12:13.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:12.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:12.895174+0000) 2022-09-08T08:12:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:14.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:14.061104+0000) 2022-09-08T08:12:14.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:13.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:13.895320+0000) 2022-09-08T08:12:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:14 smithi055 ceph-mon[97033]: pgmap v20567: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:12:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:14 smithi059 ceph-mon[97450]: pgmap v20567: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:12:15.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:14.894+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:14.895505+0000) 2022-09-08T08:12:15.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:15.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:15.061283+0000) 2022-09-08T08:12:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:16.060+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:16.061538+0000) 2022-09-08T08:12:16.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:15.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:15.895705+0000) 2022-09-08T08:12:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:16 smithi055 ceph-mon[97033]: pgmap v20568: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:12:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:16 smithi059 ceph-mon[97450]: pgmap v20568: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:12:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:17.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:17.061805+0000) 2022-09-08T08:12:17.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:16.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:16.895922+0000) 2022-09-08T08:12:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:18.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:18.062082+0000) 2022-09-08T08:12:18.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:17.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:17.896137+0000) 2022-09-08T08:12:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:18 smithi055 ceph-mon[97033]: pgmap v20569: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:12:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:18 smithi055 ceph-mon[97033]: osdmap e1364: 12 total, 11 up, 11 in 2022-09-08T08:12:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:18 smithi059 ceph-mon[97450]: pgmap v20569: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:12:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:18 smithi059 ceph-mon[97450]: osdmap e1364: 12 total, 11 up, 11 in 2022-09-08T08:12:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:19.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:19.062339+0000) 2022-09-08T08:12:19.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:18.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:18.896315+0000) 2022-09-08T08:12:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:20.061+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:20.062576+0000) 2022-09-08T08:12:20.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:19.895+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:19.896516+0000) 2022-09-08T08:12:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:20 smithi055 ceph-mon[97033]: pgmap v20571: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T08:12:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:20 smithi059 ceph-mon[97450]: pgmap v20571: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2022-09-08T08:12:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:21.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:21.062843+0000) 2022-09-08T08:12:21.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:20.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:20.896704+0000) 2022-09-08T08:12:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:22.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:22.063027+0000) 2022-09-08T08:12:22.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:21.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:21.896969+0000) 2022-09-08T08:12:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:22 smithi055 ceph-mon[97033]: pgmap v20572: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:12:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:22 smithi059 ceph-mon[97450]: pgmap v20572: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:12:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:23.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:23.063216+0000) 2022-09-08T08:12:23.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:22.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:22.897167+0000) 2022-09-08T08:12:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:24.062+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:24.063403+0000) 2022-09-08T08:12:24.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:23.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:23.897366+0000) 2022-09-08T08:12:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:24 smithi055 ceph-mon[97033]: pgmap v20573: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:12:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:24 smithi059 ceph-mon[97450]: pgmap v20573: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:12:25.066 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:12:24.784+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:12:25.067 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:24.896+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:24.897530+0000) 2022-09-08T08:12:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:25.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:25.063706+0000) 2022-09-08T08:12:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:26.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:26.063963+0000) 2022-09-08T08:12:26.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:25.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:25.897672+0000) 2022-09-08T08:12:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:26 smithi055 ceph-mon[97033]: pgmap v20574: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:12:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:26 smithi059 ceph-mon[97450]: pgmap v20574: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:12:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:27.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:27.064118+0000) 2022-09-08T08:12:27.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:26.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:26.897823+0000) 2022-09-08T08:12:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:28.063+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:28.064326+0000) 2022-09-08T08:12:28.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:27.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:27.897948+0000) 2022-09-08T08:12:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:28 smithi055 ceph-mon[97033]: pgmap v20575: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:12:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:28 smithi059 ceph-mon[97450]: pgmap v20575: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:12:29.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:28.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:28.898108+0000) 2022-09-08T08:12:29.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:29.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:29.064529+0000) 2022-09-08T08:12:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:30.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:30.064755+0000) 2022-09-08T08:12:30.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:29.897+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:29.898347+0000) 2022-09-08T08:12:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:30 smithi055 ceph-mon[97033]: pgmap v20576: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:12:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:12:30.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:12:30.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:12:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:30 smithi059 ceph-mon[97450]: pgmap v20576: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:12:31.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:12:31.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:12:31.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:12:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:31.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:31.065003+0000) 2022-09-08T08:12:31.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:30.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:30.898628+0000) 2022-09-08T08:12:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:32.064+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:32.065233+0000) 2022-09-08T08:12:32.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:31.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:31.898823+0000) 2022-09-08T08:12:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:32 smithi055 ceph-mon[97033]: pgmap v20577: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:12:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:32 smithi059 ceph-mon[97450]: pgmap v20577: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:12:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:33.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:33.065504+0000) 2022-09-08T08:12:33.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:32.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:32.898997+0000) 2022-09-08T08:12:34.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:33.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:33.899190+0000) 2022-09-08T08:12:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:34.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:34.065767+0000) 2022-09-08T08:12:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:34 smithi055 ceph-mon[97033]: pgmap v20578: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:34 smithi059 ceph-mon[97450]: pgmap v20578: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:35.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:35.066017+0000) 2022-09-08T08:12:35.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:34.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:34.899337+0000) 2022-09-08T08:12:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:36.065+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:36.066212+0000) 2022-09-08T08:12:36.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:35.898+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:35.899519+0000) 2022-09-08T08:12:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:36 smithi055 ceph-mon[97033]: pgmap v20579: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:12:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:36 smithi059 ceph-mon[97450]: pgmap v20579: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:12:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:37.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:37.066455+0000) 2022-09-08T08:12:37.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:36.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:36.899767+0000) 2022-09-08T08:12:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:38.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:38.066753+0000) 2022-09-08T08:12:38.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:37.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:37.899990+0000) 2022-09-08T08:12:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:38 smithi055 ceph-mon[97033]: pgmap v20580: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:38 smithi059 ceph-mon[97450]: pgmap v20580: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:39.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:39.066965+0000) 2022-09-08T08:12:39.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:38.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:38.900199+0000) 2022-09-08T08:12:40.069 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:12:39.787+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:12:40.070 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:39.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:39.900393+0000) 2022-09-08T08:12:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:40.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:40.067131+0000) 2022-09-08T08:12:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:40 smithi055 ceph-mon[97033]: pgmap v20581: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:40 smithi059 ceph-mon[97450]: pgmap v20581: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:41.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:41.067312+0000) 2022-09-08T08:12:41.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:40.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:40.900621+0000) 2022-09-08T08:12:42.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:42.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:42.067499+0000) 2022-09-08T08:12:42.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:41.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:41.900819+0000) 2022-09-08T08:12:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:42 smithi055 ceph-mon[97033]: pgmap v20582: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:12:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:42 smithi059 ceph-mon[97450]: pgmap v20582: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:12:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:43.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:43.067733+0000) 2022-09-08T08:12:43.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:42.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:42.901060+0000) 2022-09-08T08:12:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:44.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:44.067862+0000) 2022-09-08T08:12:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:43.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:43.901303+0000) 2022-09-08T08:12:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:44 smithi059 ceph-mon[97450]: pgmap v20583: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:44 smithi055 ceph-mon[97033]: pgmap v20583: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:45.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:45.068083+0000) 2022-09-08T08:12:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:44.899+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:44.901505+0000) 2022-09-08T08:12:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:46.066+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:46.068346+0000) 2022-09-08T08:12:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:45.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:45.901662+0000) 2022-09-08T08:12:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:46 smithi055 ceph-mon[97033]: pgmap v20584: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:12:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:46 smithi059 ceph-mon[97450]: pgmap v20584: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:12:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:47.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:47.068615+0000) 2022-09-08T08:12:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:46.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:46.901875+0000) 2022-09-08T08:12:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:48.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:48.068877+0000) 2022-09-08T08:12:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:47.900+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:47.902028+0000) 2022-09-08T08:12:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:48 smithi055 ceph-mon[97033]: pgmap v20585: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:48 smithi059 ceph-mon[97450]: pgmap v20585: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:49.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:49.069060+0000) 2022-09-08T08:12:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:48.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:48.902191+0000) 2022-09-08T08:12:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:50.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:50.069309+0000) 2022-09-08T08:12:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:49.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:49.902335+0000) 2022-09-08T08:12:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:50 smithi055 ceph-mon[97033]: pgmap v20586: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:50 smithi059 ceph-mon[97450]: pgmap v20586: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:51.067+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:51.069502+0000) 2022-09-08T08:12:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:50.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:50.902507+0000) 2022-09-08T08:12:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:52.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:52.069744+0000) 2022-09-08T08:12:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:51.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:51.902712+0000) 2022-09-08T08:12:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:52 smithi055 ceph-mon[97033]: pgmap v20587: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:12:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:52 smithi059 ceph-mon[97450]: pgmap v20587: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:12:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:53.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:53.069928+0000) 2022-09-08T08:12:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:52.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:52.902864+0000) 2022-09-08T08:12:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:54.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:54.070093+0000) 2022-09-08T08:12:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:53.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:53.903096+0000) 2022-09-08T08:12:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:54 smithi055 ceph-mon[97033]: pgmap v20588: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:54 smithi059 ceph-mon[97450]: pgmap v20588: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:55.073 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:12:54.798+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:12:55.073 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:54.901+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:54.903386+0000) 2022-09-08T08:12:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:55.068+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:55.070376+0000) 2022-09-08T08:12:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:56.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:56.070647+0000) 2022-09-08T08:12:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:55.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:55.903543+0000) 2022-09-08T08:12:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:56 smithi055 ceph-mon[97033]: pgmap v20589: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:12:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:56 smithi059 ceph-mon[97450]: pgmap v20589: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:12:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:57.069+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:57.070783+0000) 2022-09-08T08:12:57.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:56.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:56.903740+0000) 2022-09-08T08:12:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:58.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:58.071030+0000) 2022-09-08T08:12:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:57.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:57.903886+0000) 2022-09-08T08:12:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:12:58 smithi055 ceph-mon[97033]: pgmap v20590: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:12:58 smithi059 ceph-mon[97450]: pgmap v20590: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:12:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:12:59.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:59.071212+0000) 2022-09-08T08:12:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:58.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:58.904022+0000) 2022-09-08T08:13:00.073 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:12:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:12:59.902+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:12:59.904256+0000) 2022-09-08T08:13:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:00.070+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:00.071453+0000) 2022-09-08T08:13:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:00 smithi059 ceph-mon[97450]: pgmap v20591: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:00 smithi059 ceph-mon[97450]: pgmap v20592: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:13:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:00 smithi059 ceph-mon[97450]: pgmap v20593: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:13:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:13:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:00 smithi055 ceph-mon[97033]: pgmap v20591: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:00 smithi055 ceph-mon[97033]: pgmap v20592: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:13:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:00 smithi055 ceph-mon[97033]: pgmap v20593: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:13:01.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:13:01.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:00.903+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:00.904454+0000) 2022-09-08T08:13:01.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:01.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:01.071712+0000) 2022-09-08T08:13:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:01.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:01.904667+0000) 2022-09-08T08:13:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:02.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:02.071933+0000) 2022-09-08T08:13:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:02 smithi059 ceph-mon[97450]: pgmap v20594: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:13:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:02 smithi055 ceph-mon[97033]: pgmap v20594: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:13:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:03.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:03.072144+0000) 2022-09-08T08:13:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:02.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:02.904806+0000) 2022-09-08T08:13:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:04.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:04.072341+0000) 2022-09-08T08:13:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:03.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:03.904934+0000) 2022-09-08T08:13:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:04 smithi055 ceph-mon[97033]: pgmap v20595: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:13:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:04 smithi059 ceph-mon[97450]: pgmap v20595: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:13:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:04.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:04.905095+0000) 2022-09-08T08:13:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:05.071+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:05.072607+0000) 2022-09-08T08:13:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:06.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:06.072776+0000) 2022-09-08T08:13:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:05.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:05.905274+0000) 2022-09-08T08:13:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:06 smithi055 ceph-mon[97033]: pgmap v20596: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T08:13:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:13:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:06 smithi059 ceph-mon[97450]: pgmap v20596: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T08:13:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:13:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:07.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:07.072913+0000) 2022-09-08T08:13:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:06.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:06.905441+0000) 2022-09-08T08:13:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:13:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:13:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:07.904+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:07.905578+0000) 2022-09-08T08:13:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:08.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:08.073062+0000) 2022-09-08T08:13:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:08 smithi055 ceph-mon[97033]: pgmap v20597: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T08:13:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:08 smithi059 ceph-mon[97450]: pgmap v20597: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T08:13:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:09.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:09.073259+0000) 2022-09-08T08:13:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:08.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:08.905700+0000) 2022-09-08T08:13:10.076 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:13:09.815+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:13:10.076 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:09.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:09.905832+0000) 2022-09-08T08:13:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:10.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:10.073414+0000) 2022-09-08T08:13:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:10 smithi055 ceph-mon[97033]: pgmap v20598: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:13:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:10 smithi055 ceph-mon[97033]: osdmap e1365: 12 total, 11 up, 11 in 2022-09-08T08:13:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:10 smithi059 ceph-mon[97450]: pgmap v20598: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:13:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:10 smithi059 ceph-mon[97450]: osdmap e1365: 12 total, 11 up, 11 in 2022-09-08T08:13:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:11.072+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:11.073570+0000) 2022-09-08T08:13:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:10.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:10.905975+0000) 2022-09-08T08:13:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:12.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:12.073702+0000) 2022-09-08T08:13:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:11.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:11.906129+0000) 2022-09-08T08:13:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:12 smithi055 ceph-mon[97033]: pgmap v20600: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:13:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:12 smithi059 ceph-mon[97450]: pgmap v20600: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:13:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:13.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:13.073844+0000) 2022-09-08T08:13:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:12.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:12.906269+0000) 2022-09-08T08:13:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:14.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:14.074015+0000) 2022-09-08T08:13:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:13.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:13.906404+0000) 2022-09-08T08:13:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:14 smithi059 ceph-mon[97450]: pgmap v20601: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:13:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:14 smithi055 ceph-mon[97033]: pgmap v20601: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:13:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:15.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:15.074176+0000) 2022-09-08T08:13:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:14.905+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:14.906543+0000) 2022-09-08T08:13:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:16.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:16.074328+0000) 2022-09-08T08:13:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:15.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:15.906744+0000) 2022-09-08T08:13:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:16 smithi055 ceph-mon[97033]: pgmap v20602: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:13:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:16 smithi059 ceph-mon[97450]: pgmap v20602: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:13:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:17.073+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:17.074468+0000) 2022-09-08T08:13:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:16.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:16.906945+0000) 2022-09-08T08:13:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:18.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:18.074727+0000) 2022-09-08T08:13:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:17.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:17.907124+0000) 2022-09-08T08:13:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:18 smithi055 ceph-mon[97033]: pgmap v20603: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:13:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:18 smithi055 ceph-mon[97033]: osdmap e1366: 12 total, 11 up, 11 in 2022-09-08T08:13:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:18 smithi059 ceph-mon[97450]: pgmap v20603: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:13:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:18 smithi059 ceph-mon[97450]: osdmap e1366: 12 total, 11 up, 11 in 2022-09-08T08:13:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:19.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:19.074911+0000) 2022-09-08T08:13:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:18.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:18.907354+0000) 2022-09-08T08:13:20.077 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:19.906+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:19.907541+0000) 2022-09-08T08:13:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:20.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:20.075137+0000) 2022-09-08T08:13:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:20 smithi055 ceph-mon[97033]: pgmap v20605: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.6 KiB/s rd, 103 B/s wr, 4 op/s 2022-09-08T08:13:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:20 smithi059 ceph-mon[97450]: pgmap v20605: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.6 KiB/s rd, 103 B/s wr, 4 op/s 2022-09-08T08:13:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:21.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:21.075349+0000) 2022-09-08T08:13:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:20.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:20.907705+0000) 2022-09-08T08:13:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:22.074+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:22.075566+0000) 2022-09-08T08:13:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:21.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:21.907894+0000) 2022-09-08T08:13:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:22 smithi055 ceph-mon[97033]: pgmap v20606: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:13:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:22 smithi059 ceph-mon[97450]: pgmap v20606: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:13:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:23.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:23.075765+0000) 2022-09-08T08:13:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:22.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:22.908059+0000) 2022-09-08T08:13:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:24.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:24.075981+0000) 2022-09-08T08:13:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:23.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:23.908288+0000) 2022-09-08T08:13:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:24 smithi059 ceph-mon[97450]: pgmap v20607: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:13:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:24 smithi055 ceph-mon[97033]: pgmap v20607: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:13:25.078 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:13:24.821+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:13:25.079 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:24.907+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:24.908456+0000) 2022-09-08T08:13:25.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:25.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:25.076229+0000) 2022-09-08T08:13:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:26.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:26.076450+0000) 2022-09-08T08:13:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:25.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:25.908696+0000) 2022-09-08T08:13:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:26 smithi055 ceph-mon[97033]: pgmap v20608: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:13:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:26 smithi059 ceph-mon[97450]: pgmap v20608: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:13:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:27.075+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:27.076624+0000) 2022-09-08T08:13:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:26.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:26.908905+0000) 2022-09-08T08:13:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:28.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:28.076805+0000) 2022-09-08T08:13:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:27.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:27.909091+0000) 2022-09-08T08:13:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:28 smithi055 ceph-mon[97033]: pgmap v20609: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:13:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:28 smithi059 ceph-mon[97450]: pgmap v20609: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:13:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:29.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:29.076924+0000) 2022-09-08T08:13:29.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:28.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:28.909313+0000) 2022-09-08T08:13:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:30.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:30.077157+0000) 2022-09-08T08:13:30.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:29.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:29.909541+0000) 2022-09-08T08:13:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:30 smithi055 ceph-mon[97033]: pgmap v20610: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T08:13:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:30 smithi059 ceph-mon[97450]: pgmap v20610: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T08:13:31.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:30.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:30.909764+0000) 2022-09-08T08:13:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:31.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:31.077375+0000) 2022-09-08T08:13:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:32.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:32.077579+0000) 2022-09-08T08:13:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:31.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:31.909942+0000) 2022-09-08T08:13:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:32 smithi055 ceph-mon[97033]: pgmap v20611: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:13:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:32 smithi059 ceph-mon[97450]: pgmap v20611: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:13:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:33.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:33.077815+0000) 2022-09-08T08:13:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:32.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:32.910144+0000) 2022-09-08T08:13:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:34.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:34.078016+0000) 2022-09-08T08:13:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:33.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:33.910370+0000) 2022-09-08T08:13:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:34 smithi059 ceph-mon[97450]: pgmap v20612: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:34 smithi055 ceph-mon[97033]: pgmap v20612: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:35.076+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:35.078203+0000) 2022-09-08T08:13:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:34.908+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:34.910578+0000) 2022-09-08T08:13:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:36.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:36.078435+0000) 2022-09-08T08:13:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:35.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:35.910723+0000) 2022-09-08T08:13:37.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:36 smithi055 ceph-mon[97033]: pgmap v20613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:13:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:36 smithi059 ceph-mon[97450]: pgmap v20613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:13:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:37.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:37.078627+0000) 2022-09-08T08:13:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:36.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:36.910912+0000) 2022-09-08T08:13:38.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:37.909+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:37.911055+0000) 2022-09-08T08:13:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:38.077+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:38.078804+0000) 2022-09-08T08:13:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:38 smithi055 ceph-mon[97033]: pgmap v20614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:38 smithi059 ceph-mon[97450]: pgmap v20614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:39.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:39.079046+0000) 2022-09-08T08:13:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:38.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:38.911191+0000) 2022-09-08T08:13:40.081 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:13:39.823+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:13:40.082 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:39.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:39.911375+0000) 2022-09-08T08:13:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:40.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:40.079292+0000) 2022-09-08T08:13:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:40 smithi055 ceph-mon[97033]: pgmap v20615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:40 smithi059 ceph-mon[97450]: pgmap v20615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:41.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:41.079543+0000) 2022-09-08T08:13:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:40.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:40.911516+0000) 2022-09-08T08:13:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:42.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:42.079833+0000) 2022-09-08T08:13:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:41.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:41.911764+0000) 2022-09-08T08:13:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:42 smithi055 ceph-mon[97033]: pgmap v20616: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:13:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:42 smithi059 ceph-mon[97450]: pgmap v20616: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:13:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:43.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:43.079999+0000) 2022-09-08T08:13:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:42.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:42.912017+0000) 2022-09-08T08:13:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:44.078+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:44.080177+0000) 2022-09-08T08:13:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:43.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:43.912266+0000) 2022-09-08T08:13:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:44 smithi059 ceph-mon[97450]: pgmap v20617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:44 smithi055 ceph-mon[97033]: pgmap v20617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:45.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:45.080384+0000) 2022-09-08T08:13:45.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:44.910+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:44.912516+0000) 2022-09-08T08:13:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:46.079+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:46.080613+0000) 2022-09-08T08:13:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:45.911+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:45.912709+0000) 2022-09-08T08:13:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:46 smithi055 ceph-mon[97033]: pgmap v20618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:13:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:46 smithi059 ceph-mon[97450]: pgmap v20618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:13:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:47.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:47.080880+0000) 2022-09-08T08:13:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:46.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:46.912954+0000) 2022-09-08T08:13:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:48.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:48.081108+0000) 2022-09-08T08:13:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:47.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:47.913156+0000) 2022-09-08T08:13:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:48 smithi055 ceph-mon[97033]: pgmap v20619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:48 smithi059 ceph-mon[97450]: pgmap v20619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:49.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:49.081278+0000) 2022-09-08T08:13:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:48.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:48.913324+0000) 2022-09-08T08:13:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:50.080+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:50.081497+0000) 2022-09-08T08:13:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:49.912+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:49.913525+0000) 2022-09-08T08:13:51.235 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:50 smithi055 ceph-mon[97033]: pgmap v20620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:50 smithi059 ceph-mon[97450]: pgmap v20620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:51.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:51.081765+0000) 2022-09-08T08:13:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:50.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:50.913748+0000) 2022-09-08T08:13:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:52.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:52.082052+0000) 2022-09-08T08:13:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:51.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:51.913892+0000) 2022-09-08T08:13:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:52 smithi055 ceph-mon[97033]: pgmap v20621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:13:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:52 smithi059 ceph-mon[97450]: pgmap v20621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:13:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:53.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:53.082320+0000) 2022-09-08T08:13:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:52.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:52.914107+0000) 2022-09-08T08:13:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:53.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:53.914331+0000) 2022-09-08T08:13:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:54.081+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:54.082552+0000) 2022-09-08T08:13:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:54 smithi059 ceph-mon[97450]: pgmap v20622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:54 smithi055 ceph-mon[97033]: pgmap v20622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:55.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:13:54.839+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:13:55.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:55.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:55.082789+0000) 2022-09-08T08:13:55.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:54.913+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:54.914551+0000) 2022-09-08T08:13:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:56.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:56.083041+0000) 2022-09-08T08:13:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:55.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:55.914745+0000) 2022-09-08T08:13:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:56 smithi055 ceph-mon[97033]: pgmap v20623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:13:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:56 smithi059 ceph-mon[97450]: pgmap v20623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:13:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:57.082+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:57.083300+0000) 2022-09-08T08:13:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:56.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:56.914959+0000) 2022-09-08T08:13:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:57.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:57.915250+0000) 2022-09-08T08:13:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:58.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:58.083520+0000) 2022-09-08T08:13:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:13:58 smithi055 ceph-mon[97033]: pgmap v20624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:13:58 smithi059 ceph-mon[97450]: pgmap v20624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:13:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:13:59.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:59.083709+0000) 2022-09-08T08:13:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:58.914+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:58.915465+0000) 2022-09-08T08:14:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:00.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:00.083906+0000) 2022-09-08T08:14:00.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:13:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:13:59.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:13:59.915632+0000) 2022-09-08T08:14:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:00 smithi055 ceph-mon[97033]: pgmap v20625: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:14:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:14:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:14:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:14:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:00 smithi059 ceph-mon[97450]: pgmap v20625: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:14:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:14:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:14:01.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:14:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:01.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:01.084043+0000) 2022-09-08T08:14:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:00.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:00.915792+0000) 2022-09-08T08:14:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:02.083+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:02.084245+0000) 2022-09-08T08:14:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:01.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:01.915997+0000) 2022-09-08T08:14:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:02 smithi055 ceph-mon[97033]: pgmap v20626: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:14:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:02 smithi059 ceph-mon[97450]: pgmap v20626: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:14:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:03.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:03.084493+0000) 2022-09-08T08:14:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:02.915+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:02.916261+0000) 2022-09-08T08:14:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:04.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:04.084766+0000) 2022-09-08T08:14:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:03.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:03.916509+0000) 2022-09-08T08:14:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:04 smithi059 ceph-mon[97450]: pgmap v20627: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:14:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:04 smithi055 ceph-mon[97033]: pgmap v20627: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:14:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:05.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:05.085056+0000) 2022-09-08T08:14:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:04.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:04.916768+0000) 2022-09-08T08:14:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:06.084+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:06.085322+0000) 2022-09-08T08:14:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:05.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:05.916994+0000) 2022-09-08T08:14:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:06 smithi055 ceph-mon[97033]: pgmap v20628: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:14:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:14:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:06 smithi059 ceph-mon[97450]: pgmap v20628: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:14:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:14:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:07.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:07.085563+0000) 2022-09-08T08:14:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:06.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:06.917221+0000) 2022-09-08T08:14:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:14:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:14:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:08.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:08.085731+0000) 2022-09-08T08:14:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:07.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:07.917379+0000) 2022-09-08T08:14:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:08 smithi055 ceph-mon[97033]: pgmap v20629: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:14:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:08 smithi059 ceph-mon[97450]: pgmap v20629: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:14:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:09.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:09.085912+0000) 2022-09-08T08:14:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:08.916+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:08.917596+0000) 2022-09-08T08:14:10.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:14:09.841+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:14:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:10.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:10.086169+0000) 2022-09-08T08:14:10.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:09.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:09.917858+0000) 2022-09-08T08:14:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:10 smithi055 ceph-mon[97033]: pgmap v20630: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:14:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:10 smithi055 ceph-mon[97033]: osdmap e1367: 12 total, 11 up, 11 in 2022-09-08T08:14:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:10 smithi059 ceph-mon[97450]: pgmap v20630: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:14:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:10 smithi059 ceph-mon[97450]: osdmap e1367: 12 total, 11 up, 11 in 2022-09-08T08:14:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:11.085+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:11.086403+0000) 2022-09-08T08:14:11.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:10.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:10.918125+0000) 2022-09-08T08:14:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:11.917+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:11.918320+0000) 2022-09-08T08:14:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:12.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:12.086675+0000) 2022-09-08T08:14:13.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:12 smithi055 ceph-mon[97033]: pgmap v20632: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:14:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:12 smithi059 ceph-mon[97450]: pgmap v20632: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:14:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:13.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:13.086818+0000) 2022-09-08T08:14:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:12.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:12.918526+0000) 2022-09-08T08:14:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:13.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:13.918683+0000) 2022-09-08T08:14:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:14.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:14.086990+0000) 2022-09-08T08:14:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:14 smithi059 ceph-mon[97450]: pgmap v20633: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:14:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:14 smithi055 ceph-mon[97033]: pgmap v20633: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:14:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:15.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:15.087230+0000) 2022-09-08T08:14:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:14.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:14.918903+0000) 2022-09-08T08:14:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:16.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:16.087483+0000) 2022-09-08T08:14:16.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:15.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:15.919054+0000) 2022-09-08T08:14:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:16 smithi055 ceph-mon[97033]: pgmap v20634: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:14:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:16 smithi059 ceph-mon[97450]: pgmap v20634: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:14:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:17.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:17.087728+0000) 2022-09-08T08:14:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:16.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:16.919275+0000) 2022-09-08T08:14:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:18.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:18.087958+0000) 2022-09-08T08:14:18.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:17.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:17.919448+0000) 2022-09-08T08:14:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:18 smithi055 ceph-mon[97033]: pgmap v20635: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:14:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:18 smithi055 ceph-mon[97033]: osdmap e1368: 12 total, 11 up, 11 in 2022-09-08T08:14:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:18 smithi059 ceph-mon[97450]: pgmap v20635: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:14:19.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:18 smithi059 ceph-mon[97450]: osdmap e1368: 12 total, 11 up, 11 in 2022-09-08T08:14:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:19.086+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:19.088095+0000) 2022-09-08T08:14:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:18.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:18.919596+0000) 2022-09-08T08:14:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:20.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:20.088298+0000) 2022-09-08T08:14:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:19.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:19.919753+0000) 2022-09-08T08:14:21.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:20 smithi055 ceph-mon[97033]: pgmap v20637: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.6 KiB/s rd, 103 B/s wr, 4 op/s 2022-09-08T08:14:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:20 smithi059 ceph-mon[97450]: pgmap v20637: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.6 KiB/s rd, 103 B/s wr, 4 op/s 2022-09-08T08:14:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:20.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:20.919898+0000) 2022-09-08T08:14:21.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:21.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:21.088546+0000) 2022-09-08T08:14:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:22.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:22.088781+0000) 2022-09-08T08:14:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:21.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:21.920110+0000) 2022-09-08T08:14:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:22 smithi055 ceph-mon[97033]: pgmap v20638: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:14:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:22 smithi059 ceph-mon[97450]: pgmap v20638: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:14:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:23.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:23.088992+0000) 2022-09-08T08:14:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:22.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:22.920332+0000) 2022-09-08T08:14:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:24.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:24.089175+0000) 2022-09-08T08:14:24.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:23.918+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:23.920525+0000) 2022-09-08T08:14:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:24 smithi059 ceph-mon[97450]: pgmap v20639: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:14:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:24 smithi055 ceph-mon[97033]: pgmap v20639: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:14:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:25.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:25.089361+0000) 2022-09-08T08:14:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:24.919+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:24.920776+0000) 2022-09-08T08:14:25.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:14:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:14:24.853+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:14:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:26.087+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:26.089613+0000) 2022-09-08T08:14:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:25.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:25.921006+0000) 2022-09-08T08:14:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:26 smithi055 ceph-mon[97033]: pgmap v20640: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:14:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:26 smithi059 ceph-mon[97450]: pgmap v20640: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:14:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:27.088+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:27.089839+0000) 2022-09-08T08:14:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:26.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:26.921233+0000) 2022-09-08T08:14:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:28.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:28.090073+0000) 2022-09-08T08:14:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:27.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:27.921403+0000) 2022-09-08T08:14:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:28 smithi055 ceph-mon[97033]: pgmap v20641: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:14:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:28 smithi059 ceph-mon[97450]: pgmap v20641: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:14:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:29.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:29.090297+0000) 2022-09-08T08:14:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:28.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:28.921688+0000) 2022-09-08T08:14:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:30.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:30.090547+0000) 2022-09-08T08:14:30.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:29.920+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:29.921864+0000) 2022-09-08T08:14:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:30 smithi055 ceph-mon[97033]: pgmap v20642: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T08:14:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:30 smithi059 ceph-mon[97450]: pgmap v20642: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T08:14:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:30.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:30.922018+0000) 2022-09-08T08:14:31.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:31.089+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:31.090800+0000) 2022-09-08T08:14:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:32.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:32.090996+0000) 2022-09-08T08:14:32.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:31.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:31.922228+0000) 2022-09-08T08:14:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:32 smithi055 ceph-mon[97033]: pgmap v20643: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:14:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:32 smithi059 ceph-mon[97450]: pgmap v20643: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:14:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:33.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:33.091251+0000) 2022-09-08T08:14:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:32.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:32.922425+0000) 2022-09-08T08:14:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:34.090+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:34.091455+0000) 2022-09-08T08:14:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:33.921+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:33.922622+0000) 2022-09-08T08:14:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:34 smithi059 ceph-mon[97450]: pgmap v20644: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:14:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:34 smithi055 ceph-mon[97033]: pgmap v20644: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:14:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:35.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:35.091715+0000) 2022-09-08T08:14:35.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:34.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:34.922828+0000) 2022-09-08T08:14:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:14:35.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:14:35.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:35 smithi059 ceph-mon[97450]: pgmap v20645: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:14:35.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:35 smithi059 ceph-mon[97450]: pgmap v20646: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T08:14:35.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:14:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:14:35.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:14:35.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:35 smithi055 ceph-mon[97033]: pgmap v20645: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:14:35.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:35 smithi055 ceph-mon[97033]: pgmap v20646: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T08:14:35.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:14:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:36.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:36.091870+0000) 2022-09-08T08:14:36.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:35.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:35.922966+0000) 2022-09-08T08:14:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:37.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:37.091999+0000) 2022-09-08T08:14:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:36.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:36.923174+0000) 2022-09-08T08:14:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:37 smithi059 ceph-mon[97450]: pgmap v20647: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:14:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:37 smithi055 ceph-mon[97033]: pgmap v20647: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:14:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:38.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:38.092205+0000) 2022-09-08T08:14:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:37.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:37.923342+0000) 2022-09-08T08:14:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:39.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:39.092404+0000) 2022-09-08T08:14:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:38.922+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:38.923597+0000) 2022-09-08T08:14:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:39 smithi059 ceph-mon[97450]: pgmap v20648: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:14:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:39 smithi055 ceph-mon[97033]: pgmap v20648: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:14:40.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:14:39.854+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:14:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:40.091+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:40.092570+0000) 2022-09-08T08:14:40.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:39.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:39.923807+0000) 2022-09-08T08:14:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:41.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:41.092714+0000) 2022-09-08T08:14:41.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:40.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:40.924010+0000) 2022-09-08T08:14:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:41 smithi055 ceph-mon[97033]: pgmap v20649: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:14:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:41 smithi059 ceph-mon[97450]: pgmap v20649: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:14:42.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:42.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:42.092890+0000) 2022-09-08T08:14:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:41.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:41.924220+0000) 2022-09-08T08:14:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:43.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:43.093094+0000) 2022-09-08T08:14:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:42.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:42.924388+0000) 2022-09-08T08:14:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:43 smithi055 ceph-mon[97033]: pgmap v20650: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:14:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:43 smithi059 ceph-mon[97450]: pgmap v20650: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:14:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:44.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:44.093270+0000) 2022-09-08T08:14:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:43.923+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:43.924536+0000) 2022-09-08T08:14:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:45.092+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:45.093485+0000) 2022-09-08T08:14:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:44.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:44.924732+0000) 2022-09-08T08:14:45.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:45 smithi055 ceph-mon[97033]: pgmap v20651: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:14:46.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:45 smithi059 ceph-mon[97450]: pgmap v20651: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:14:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:46.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:46.093737+0000) 2022-09-08T08:14:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:45.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:45.924866+0000) 2022-09-08T08:14:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:47.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:47.093855+0000) 2022-09-08T08:14:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:46.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:46.925091+0000) 2022-09-08T08:14:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:47 smithi055 ceph-mon[97033]: pgmap v20652: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:14:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:47 smithi059 ceph-mon[97450]: pgmap v20652: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:14:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:48.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:48.094052+0000) 2022-09-08T08:14:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:47.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:47.925222+0000) 2022-09-08T08:14:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:49.093+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:49.094275+0000) 2022-09-08T08:14:49.434 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:48.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:48.925346+0000) 2022-09-08T08:14:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:49 smithi059 ceph-mon[97450]: pgmap v20653: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:14:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:49 smithi055 ceph-mon[97033]: pgmap v20653: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:14:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:50.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:50.094525+0000) 2022-09-08T08:14:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:49.924+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:49.925550+0000) 2022-09-08T08:14:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:51.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:51.094735+0000) 2022-09-08T08:14:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:50.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:50.925748+0000) 2022-09-08T08:14:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:51 smithi055 ceph-mon[97033]: pgmap v20654: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:14:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:51 smithi059 ceph-mon[97450]: pgmap v20654: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:14:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:52.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:52.094999+0000) 2022-09-08T08:14:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:51.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:51.925972+0000) 2022-09-08T08:14:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:53.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:53.095230+0000) 2022-09-08T08:14:53.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:52.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:52.926221+0000) 2022-09-08T08:14:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:53 smithi055 ceph-mon[97033]: pgmap v20655: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:14:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:53 smithi059 ceph-mon[97450]: pgmap v20655: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:14:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:54.094+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:54.095393+0000) 2022-09-08T08:14:54.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:53.925+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:53.926397+0000) 2022-09-08T08:14:55.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:14:54.869+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:14:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:55.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:55.095621+0000) 2022-09-08T08:14:55.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:54.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:54.926550+0000) 2022-09-08T08:14:55.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:55 smithi055 ceph-mon[97033]: pgmap v20656: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:14:56.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:55 smithi059 ceph-mon[97450]: pgmap v20656: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:14:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:56.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:56.095852+0000) 2022-09-08T08:14:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:55.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:55.926712+0000) 2022-09-08T08:14:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:57.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:57.096058+0000) 2022-09-08T08:14:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:56.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:56.926870+0000) 2022-09-08T08:14:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:57 smithi055 ceph-mon[97033]: pgmap v20657: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:14:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:57 smithi059 ceph-mon[97450]: pgmap v20657: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:14:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:58.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:58.096217+0000) 2022-09-08T08:14:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:57.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:57.927020+0000) 2022-09-08T08:14:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:14:59.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:59.096455+0000) 2022-09-08T08:14:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:58.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:58.927214+0000) 2022-09-08T08:14:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:14:59 smithi059 ceph-mon[97450]: pgmap v20658: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:14:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:14:59 smithi055 ceph-mon[97033]: pgmap v20658: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:00.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:00.096686+0000) 2022-09-08T08:15:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:14:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:14:59.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:14:59.927375+0000) 2022-09-08T08:15:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:01.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:01.096815+0000) 2022-09-08T08:15:01.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:00.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:00.927529+0000) 2022-09-08T08:15:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:01 smithi055 ceph-mon[97033]: pgmap v20659: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:15:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:01 smithi059 ceph-mon[97450]: pgmap v20659: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:15:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:02.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:02.096957+0000) 2022-09-08T08:15:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:01.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:01.927684+0000) 2022-09-08T08:15:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:03.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:03.097190+0000) 2022-09-08T08:15:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:02.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:02.927840+0000) 2022-09-08T08:15:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:03 smithi055 ceph-mon[97033]: pgmap v20660: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:03 smithi059 ceph-mon[97450]: pgmap v20660: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:04.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:04.097418+0000) 2022-09-08T08:15:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:03.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:03.927987+0000) 2022-09-08T08:15:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:05.095+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:05.097597+0000) 2022-09-08T08:15:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:04.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:04.928119+0000) 2022-09-08T08:15:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:05 smithi055 ceph-mon[97033]: pgmap v20661: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:05 smithi059 ceph-mon[97450]: pgmap v20661: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:06.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:06.097833+0000) 2022-09-08T08:15:06.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:05.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:05.928286+0000) 2022-09-08T08:15:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:15:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:15:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:07.096+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:07.098019+0000) 2022-09-08T08:15:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:06.926+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:06.928425+0000) 2022-09-08T08:15:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:07 smithi055 ceph-mon[97033]: pgmap v20662: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:15:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:15:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:07 smithi059 ceph-mon[97450]: pgmap v20662: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:15:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:15:08.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:08.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:08.098258+0000) 2022-09-08T08:15:08.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:07.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:07.928677+0000) 2022-09-08T08:15:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:09.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:09.098529+0000) 2022-09-08T08:15:09.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:08.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:08.928870+0000) 2022-09-08T08:15:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:09 smithi059 ceph-mon[97450]: pgmap v20663: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:15:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:09 smithi055 ceph-mon[97033]: pgmap v20663: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:15:10.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:15:09.883+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:15:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:10.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:10.098796+0000) 2022-09-08T08:15:10.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:09.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:09.929002+0000) 2022-09-08T08:15:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:10 smithi059 ceph-mon[97450]: osdmap e1369: 12 total, 11 up, 11 in 2022-09-08T08:15:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:10 smithi055 ceph-mon[97033]: osdmap e1369: 12 total, 11 up, 11 in 2022-09-08T08:15:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:11.097+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:11.099000+0000) 2022-09-08T08:15:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:10.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:10.929157+0000) 2022-09-08T08:15:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:11 smithi059 ceph-mon[97450]: pgmap v20665: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:15:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:11 smithi055 ceph-mon[97033]: pgmap v20665: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:15:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:12.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:12.099196+0000) 2022-09-08T08:15:12.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:11.927+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:11.929383+0000) 2022-09-08T08:15:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:13.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:13.099407+0000) 2022-09-08T08:15:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:12.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:12.929653+0000) 2022-09-08T08:15:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:13 smithi055 ceph-mon[97033]: pgmap v20666: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:15:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:13 smithi059 ceph-mon[97450]: pgmap v20666: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:15:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:14.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:14.099630+0000) 2022-09-08T08:15:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:13.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:13.929836+0000) 2022-09-08T08:15:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:15.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:15.099908+0000) 2022-09-08T08:15:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:14.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:14.930084+0000) 2022-09-08T08:15:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:15 smithi055 ceph-mon[97033]: pgmap v20667: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:15:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:15 smithi059 ceph-mon[97450]: pgmap v20667: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:15:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:16.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:16.100147+0000) 2022-09-08T08:15:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:15.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:15.930245+0000) 2022-09-08T08:15:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:17.099+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:17.100362+0000) 2022-09-08T08:15:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:16.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:16.930406+0000) 2022-09-08T08:15:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:17 smithi055 ceph-mon[97033]: pgmap v20668: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:15:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:17 smithi059 ceph-mon[97450]: pgmap v20668: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:15:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:18.098+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:18.100596+0000) 2022-09-08T08:15:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:17.928+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:17.930541+0000) 2022-09-08T08:15:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:19.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:19.100869+0000) 2022-09-08T08:15:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:18.929+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:18.930699+0000) 2022-09-08T08:15:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:19 smithi059 ceph-mon[97450]: osdmap e1370: 12 total, 11 up, 11 in 2022-09-08T08:15:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:19 smithi059 ceph-mon[97450]: pgmap v20670: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T08:15:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:19 smithi055 ceph-mon[97033]: osdmap e1370: 12 total, 11 up, 11 in 2022-09-08T08:15:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:19 smithi055 ceph-mon[97033]: pgmap v20670: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T08:15:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:20.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:20.101051+0000) 2022-09-08T08:15:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:19.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:19.930877+0000) 2022-09-08T08:15:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:21.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:21.101292+0000) 2022-09-08T08:15:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:20.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:20.931026+0000) 2022-09-08T08:15:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:21 smithi055 ceph-mon[97033]: pgmap v20671: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:15:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:21 smithi059 ceph-mon[97450]: pgmap v20671: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:15:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:22.100+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:22.101535+0000) 2022-09-08T08:15:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:21.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:21.931203+0000) 2022-09-08T08:15:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:23.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:23.101792+0000) 2022-09-08T08:15:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:22.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:22.931342+0000) 2022-09-08T08:15:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:23 smithi055 ceph-mon[97033]: pgmap v20672: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:15:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:23 smithi059 ceph-mon[97450]: pgmap v20672: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:15:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:24.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:24.102030+0000) 2022-09-08T08:15:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:23.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:23.931478+0000) 2022-09-08T08:15:25.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:15:24.888+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:15:25.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:25.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:25.102220+0000) 2022-09-08T08:15:25.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:24.930+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:24.931606+0000) 2022-09-08T08:15:25.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:25 smithi055 ceph-mon[97033]: pgmap v20673: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:15:26.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:25 smithi059 ceph-mon[97450]: pgmap v20673: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:15:26.410 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:26.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:26.102402+0000) 2022-09-08T08:15:26.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:25.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:25.931790+0000) 2022-09-08T08:15:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:27.101+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:27.102566+0000) 2022-09-08T08:15:27.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:26.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:26.931980+0000) 2022-09-08T08:15:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:27 smithi055 ceph-mon[97033]: pgmap v20674: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:15:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:27 smithi059 ceph-mon[97450]: pgmap v20674: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:15:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:27.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:27.932188+0000) 2022-09-08T08:15:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:28.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:28.102768+0000) 2022-09-08T08:15:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:29.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:29.102978+0000) 2022-09-08T08:15:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:28.931+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:28.932453+0000) 2022-09-08T08:15:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:29 smithi059 ceph-mon[97450]: pgmap v20675: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:15:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:29 smithi055 ceph-mon[97033]: pgmap v20675: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:15:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:30.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:30.103124+0000) 2022-09-08T08:15:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:29.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:29.932730+0000) 2022-09-08T08:15:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:31.102+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:31.103306+0000) 2022-09-08T08:15:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:30.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:30.932940+0000) 2022-09-08T08:15:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:31 smithi055 ceph-mon[97033]: pgmap v20676: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:15:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:31 smithi059 ceph-mon[97450]: pgmap v20676: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:15:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:32.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:32.103544+0000) 2022-09-08T08:15:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:31.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:31.933184+0000) 2022-09-08T08:15:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:33.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:33.103779+0000) 2022-09-08T08:15:33.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:32.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:32.933376+0000) 2022-09-08T08:15:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:33 smithi055 ceph-mon[97033]: pgmap v20677: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:33 smithi059 ceph-mon[97450]: pgmap v20677: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:34.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:34.103947+0000) 2022-09-08T08:15:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:33.932+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:33.933606+0000) 2022-09-08T08:15:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:35.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:35.104129+0000) 2022-09-08T08:15:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:34.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:34.933804+0000) 2022-09-08T08:15:35.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:35 smithi055 ceph-mon[97033]: pgmap v20678: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:35.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:15:35.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:15:35.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:15:35.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:35 smithi055 ceph-mon[97033]: pgmap v20679: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T08:15:35.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:35 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:15:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:35 smithi059 ceph-mon[97450]: pgmap v20678: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:36.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:15:36.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:15:36.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:15:36.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:35 smithi059 ceph-mon[97450]: pgmap v20679: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T08:15:36.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:35 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:15:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:36.103+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:36.104381+0000) 2022-09-08T08:15:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:35.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:35.934010+0000) 2022-09-08T08:15:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:36 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T08:15:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:36 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T08:15:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:37.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:37.104610+0000) 2022-09-08T08:15:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:36.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:36.934251+0000) 2022-09-08T08:15:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:37 smithi055 ceph-mon[97033]: pgmap v20680: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:15:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:37 smithi059 ceph-mon[97450]: pgmap v20680: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:15:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:38.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:38.104911+0000) 2022-09-08T08:15:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:37.933+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:37.934396+0000) 2022-09-08T08:15:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:39.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:39.105086+0000) 2022-09-08T08:15:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:38.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:38.934599+0000) 2022-09-08T08:15:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:39 smithi059 ceph-mon[97450]: pgmap v20681: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:15:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:39 smithi055 ceph-mon[97033]: pgmap v20681: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:15:40.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:15:39.896+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:15:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:40.104+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:40.105301+0000) 2022-09-08T08:15:40.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:39.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:39.934713+0000) 2022-09-08T08:15:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:41.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:41.105537+0000) 2022-09-08T08:15:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:40.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:40.934896+0000) 2022-09-08T08:15:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:41.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:41.935112+0000) 2022-09-08T08:15:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:42.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:42.105772+0000) 2022-09-08T08:15:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:42 smithi055 ceph-mon[97033]: pgmap v20682: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:15:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:42 smithi059 ceph-mon[97450]: pgmap v20682: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:15:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:43.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:43.105999+0000) 2022-09-08T08:15:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:42.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:42.935317+0000) 2022-09-08T08:15:44.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:43.934+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:43.935541+0000) 2022-09-08T08:15:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:44.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:44.106242+0000) 2022-09-08T08:15:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:44 smithi055 ceph-mon[97033]: pgmap v20683: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:15:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:44 smithi059 ceph-mon[97450]: pgmap v20683: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:15:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:45.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:45.106432+0000) 2022-09-08T08:15:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:44.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:44.935775+0000) 2022-09-08T08:15:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:46.105+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:46.106558+0000) 2022-09-08T08:15:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:45.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:45.936007+0000) 2022-09-08T08:15:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:46 smithi055 ceph-mon[97033]: pgmap v20684: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T08:15:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:46 smithi059 ceph-mon[97450]: pgmap v20684: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T08:15:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:47.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:47.106783+0000) 2022-09-08T08:15:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:46.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:46.936176+0000) 2022-09-08T08:15:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:48.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:48.106973+0000) 2022-09-08T08:15:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:47.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:47.936335+0000) 2022-09-08T08:15:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:48 smithi055 ceph-mon[97033]: pgmap v20685: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:48 smithi059 ceph-mon[97450]: pgmap v20685: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:48.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:48.936474+0000) 2022-09-08T08:15:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:49.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:49.107194+0000) 2022-09-08T08:15:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:49 smithi059 ceph-mon[97450]: pgmap v20686: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:49 smithi055 ceph-mon[97033]: pgmap v20686: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:50.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:50.107361+0000) 2022-09-08T08:15:50.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:49.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:49.936755+0000) 2022-09-08T08:15:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:51.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:51.107527+0000) 2022-09-08T08:15:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:50.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:50.936903+0000) 2022-09-08T08:15:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:52.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:52.107783+0000) 2022-09-08T08:15:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:51.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:51.937081+0000) 2022-09-08T08:15:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:52 smithi055 ceph-mon[97033]: pgmap v20687: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:15:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:52 smithi059 ceph-mon[97450]: pgmap v20687: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:15:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:53.106+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:53.107995+0000) 2022-09-08T08:15:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:52.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:52.937286+0000) 2022-09-08T08:15:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:54.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:54.108246+0000) 2022-09-08T08:15:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:53.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:53.937433+0000) 2022-09-08T08:15:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:54 smithi055 ceph-mon[97033]: pgmap v20688: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:54 smithi059 ceph-mon[97450]: pgmap v20688: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:55.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:55.108517+0000) 2022-09-08T08:15:55.155 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:54.935+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:54.937582+0000) 2022-09-08T08:15:55.155 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:15:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:15:54.897+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:15:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:56.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:56.108798+0000) 2022-09-08T08:15:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:55.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:55.937789+0000) 2022-09-08T08:15:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:56 smithi055 ceph-mon[97033]: pgmap v20689: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:15:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:56 smithi059 ceph-mon[97450]: pgmap v20689: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:15:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:57.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:57.109047+0000) 2022-09-08T08:15:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:56.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:56.938000+0000) 2022-09-08T08:15:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:58.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:58.109271+0000) 2022-09-08T08:15:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:57.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:57.938171+0000) 2022-09-08T08:15:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:58 smithi055 ceph-mon[97033]: pgmap v20690: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:58 smithi059 ceph-mon[97450]: pgmap v20690: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:15:59.107+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:59.109494+0000) 2022-09-08T08:15:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:58.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:58.938318+0000) 2022-09-08T08:15:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:15:59 smithi059 ceph-mon[97450]: pgmap v20691: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:15:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:15:59 smithi055 ceph-mon[97033]: pgmap v20691: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:00.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:00.109697+0000) 2022-09-08T08:16:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:15:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:15:59.936+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:15:59.938526+0000) 2022-09-08T08:16:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:01.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:01.109936+0000) 2022-09-08T08:16:01.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:00.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:00.938746+0000) 2022-09-08T08:16:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:02.108+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:02.110171+0000) 2022-09-08T08:16:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:01.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:01.938901+0000) 2022-09-08T08:16:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:02 smithi055 ceph-mon[97033]: pgmap v20692: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:16:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:02 smithi059 ceph-mon[97450]: pgmap v20692: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:16:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:02.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:02.939084+0000) 2022-09-08T08:16:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:03.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:03.110382+0000) 2022-09-08T08:16:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:04.109+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:04.110595+0000) 2022-09-08T08:16:04.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:03.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:03.939301+0000) 2022-09-08T08:16:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:04 smithi055 ceph-mon[97033]: pgmap v20693: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:04 smithi059 ceph-mon[97450]: pgmap v20693: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:05.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:05.110894+0000) 2022-09-08T08:16:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:04.937+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:04.939455+0000) 2022-09-08T08:16:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:06.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:06.111176+0000) 2022-09-08T08:16:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:05.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:05.939698+0000) 2022-09-08T08:16:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:06 smithi059 ceph-mon[97450]: pgmap v20694: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:16:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:16:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:06 smithi055 ceph-mon[97033]: pgmap v20694: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:16:06.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:16:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:07.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:07.111428+0000) 2022-09-08T08:16:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:06.938+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:06.939930+0000) 2022-09-08T08:16:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:16:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:16:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:08.110+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:08.111603+0000) 2022-09-08T08:16:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:07.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:07.940072+0000) 2022-09-08T08:16:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:08 smithi059 ceph-mon[97450]: pgmap v20695: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:16:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:08 smithi055 ceph-mon[97033]: pgmap v20695: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:16:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:09.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:09.111906+0000) 2022-09-08T08:16:09.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:08.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:08.940220+0000) 2022-09-08T08:16:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:09 smithi055 ceph-mon[97033]: pgmap v20696: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:16:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:09 smithi059 ceph-mon[97450]: pgmap v20696: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:16:10.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:16:09.913+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:16:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:10.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:10.112094+0000) 2022-09-08T08:16:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:09.939+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:09.940424+0000) 2022-09-08T08:16:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:10 smithi055 ceph-mon[97033]: osdmap e1371: 12 total, 11 up, 11 in 2022-09-08T08:16:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:10 smithi059 ceph-mon[97450]: osdmap e1371: 12 total, 11 up, 11 in 2022-09-08T08:16:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:11.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:11.112285+0000) 2022-09-08T08:16:11.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:10.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:10.940702+0000) 2022-09-08T08:16:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:11 smithi055 ceph-mon[97033]: pgmap v20698: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:16:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:11 smithi059 ceph-mon[97450]: pgmap v20698: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:16:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:12.111+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:12.112514+0000) 2022-09-08T08:16:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:11.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:11.940901+0000) 2022-09-08T08:16:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:13.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:13.112691+0000) 2022-09-08T08:16:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:12.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:12.941048+0000) 2022-09-08T08:16:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:14.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:14.112825+0000) 2022-09-08T08:16:14.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:13.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:13.941225+0000) 2022-09-08T08:16:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:14 smithi055 ceph-mon[97033]: pgmap v20699: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:16:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:14 smithi059 ceph-mon[97450]: pgmap v20699: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:16:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:15.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:15.113051+0000) 2022-09-08T08:16:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:14.940+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:14.941381+0000) 2022-09-08T08:16:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:16.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:16.113304+0000) 2022-09-08T08:16:16.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:15.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:15.941632+0000) 2022-09-08T08:16:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:16 smithi055 ceph-mon[97033]: pgmap v20700: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:16:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:16 smithi059 ceph-mon[97450]: pgmap v20700: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:16:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:17.112+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:17.113542+0000) 2022-09-08T08:16:17.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:16.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:16.941856+0000) 2022-09-08T08:16:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:18.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:18.113787+0000) 2022-09-08T08:16:18.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:17.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:17.942031+0000) 2022-09-08T08:16:18.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:18 smithi059 ceph-mon[97450]: pgmap v20701: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:16:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:18 smithi055 ceph-mon[97033]: pgmap v20701: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:16:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:19.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:19.113983+0000) 2022-09-08T08:16:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:18.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:18.942213+0000) 2022-09-08T08:16:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:19 smithi059 ceph-mon[97450]: osdmap e1372: 12 total, 11 up, 11 in 2022-09-08T08:16:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:19 smithi059 ceph-mon[97450]: pgmap v20703: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T08:16:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:19 smithi055 ceph-mon[97033]: osdmap e1372: 12 total, 11 up, 11 in 2022-09-08T08:16:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:19 smithi055 ceph-mon[97033]: pgmap v20703: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T08:16:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:20.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:20.114186+0000) 2022-09-08T08:16:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:19.941+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:19.942378+0000) 2022-09-08T08:16:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:21.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:21.114394+0000) 2022-09-08T08:16:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:20.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:20.942668+0000) 2022-09-08T08:16:22.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:21.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:21.942912+0000) 2022-09-08T08:16:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:22.113+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:22.114598+0000) 2022-09-08T08:16:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:22 smithi055 ceph-mon[97033]: pgmap v20704: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:16:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:22 smithi059 ceph-mon[97450]: pgmap v20704: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:16:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:23.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:23.114814+0000) 2022-09-08T08:16:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:22.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:22.943053+0000) 2022-09-08T08:16:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:24.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:24.115038+0000) 2022-09-08T08:16:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:23.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:23.943191+0000) 2022-09-08T08:16:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:24 smithi059 ceph-mon[97450]: pgmap v20705: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:16:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:24 smithi055 ceph-mon[97033]: pgmap v20705: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:16:25.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:16:24.924+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:16:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:25.114+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:25.115220+0000) 2022-09-08T08:16:25.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:24.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:24.943346+0000) 2022-09-08T08:16:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:26.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:26.115435+0000) 2022-09-08T08:16:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:25.942+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:25.943480+0000) 2022-09-08T08:16:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:26 smithi059 ceph-mon[97450]: pgmap v20706: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:16:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:26 smithi055 ceph-mon[97033]: pgmap v20706: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:16:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:27.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:27.115737+0000) 2022-09-08T08:16:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:26.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:26.943714+0000) 2022-09-08T08:16:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:28.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:28.115969+0000) 2022-09-08T08:16:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:27.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:27.943906+0000) 2022-09-08T08:16:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:28 smithi059 ceph-mon[97450]: pgmap v20707: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:16:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:28 smithi055 ceph-mon[97033]: pgmap v20707: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:16:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:29.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:29.116113+0000) 2022-09-08T08:16:29.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:28.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:28.944071+0000) 2022-09-08T08:16:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:29 smithi055 ceph-mon[97033]: pgmap v20708: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:16:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:29 smithi059 ceph-mon[97450]: pgmap v20708: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:16:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:29.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:29.944218+0000) 2022-09-08T08:16:30.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:30.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:30.116336+0000) 2022-09-08T08:16:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:31.115+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:31.116525+0000) 2022-09-08T08:16:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:30.943+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:30.944391+0000) 2022-09-08T08:16:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:32.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:32.116750+0000) 2022-09-08T08:16:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:31.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:31.944555+0000) 2022-09-08T08:16:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:32 smithi059 ceph-mon[97450]: pgmap v20709: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:16:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:32 smithi055 ceph-mon[97033]: pgmap v20709: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:16:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:33.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:33.116961+0000) 2022-09-08T08:16:33.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:32.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:32.944769+0000) 2022-09-08T08:16:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:34.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:34.117192+0000) 2022-09-08T08:16:34.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:33.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:33.944922+0000) 2022-09-08T08:16:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:34 smithi059 ceph-mon[97450]: pgmap v20710: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:34 smithi055 ceph-mon[97033]: pgmap v20710: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:35.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:35.117468+0000) 2022-09-08T08:16:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:34.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:34.945070+0000) 2022-09-08T08:16:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:35.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:35.945324+0000) 2022-09-08T08:16:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:36.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:36.117680+0000) 2022-09-08T08:16:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:36 smithi059 ceph-mon[97450]: pgmap v20711: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:16:36.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:16:36.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:16:36.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:16:36.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:16:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:36 smithi055 ceph-mon[97033]: pgmap v20711: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:16:36.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:16:36.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:16:36.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:16:36.733 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:16:37.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:37.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:37.117841+0000) 2022-09-08T08:16:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:36.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:36.945531+0000) 2022-09-08T08:16:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:38.116+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:38.118008+0000) 2022-09-08T08:16:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:37.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:37.945696+0000) 2022-09-08T08:16:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:38 smithi059 ceph-mon[97450]: pgmap v20712: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:38 smithi055 ceph-mon[97033]: pgmap v20712: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:39.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:38.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:38.945840+0000) 2022-09-08T08:16:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:39.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:39.118261+0000) 2022-09-08T08:16:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:39 smithi055 ceph-mon[97033]: pgmap v20713: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:39 smithi059 ceph-mon[97450]: pgmap v20713: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:40.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:16:39.930+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:16:40.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:40.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:40.118544+0000) 2022-09-08T08:16:40.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:39.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:39.946039+0000) 2022-09-08T08:16:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:41.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:41.118753+0000) 2022-09-08T08:16:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:40.944+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:40.946202+0000) 2022-09-08T08:16:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:42.117+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:42.118915+0000) 2022-09-08T08:16:42.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:41.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:41.946461+0000) 2022-09-08T08:16:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:42 smithi059 ceph-mon[97450]: pgmap v20714: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:16:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:42 smithi055 ceph-mon[97033]: pgmap v20714: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:16:43.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:42.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:42.946724+0000) 2022-09-08T08:16:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:43.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:43.119147+0000) 2022-09-08T08:16:44.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:44.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:44.119393+0000) 2022-09-08T08:16:44.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:43.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:43.946877+0000) 2022-09-08T08:16:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:44 smithi059 ceph-mon[97450]: pgmap v20715: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:44 smithi055 ceph-mon[97033]: pgmap v20715: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:45.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:45.119603+0000) 2022-09-08T08:16:45.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:44.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:44.947032+0000) 2022-09-08T08:16:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:46.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:46.119841+0000) 2022-09-08T08:16:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:45.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:45.947191+0000) 2022-09-08T08:16:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:46 smithi059 ceph-mon[97450]: pgmap v20716: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:16:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:46 smithi055 ceph-mon[97033]: pgmap v20716: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:16:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:47.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:47.120029+0000) 2022-09-08T08:16:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:46.945+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:46.947357+0000) 2022-09-08T08:16:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:48.118+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:48.120146+0000) 2022-09-08T08:16:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:47.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:47.947487+0000) 2022-09-08T08:16:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:48 smithi059 ceph-mon[97450]: pgmap v20717: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:48 smithi055 ceph-mon[97033]: pgmap v20717: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:49.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:49.120377+0000) 2022-09-08T08:16:49.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:48.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:48.947649+0000) 2022-09-08T08:16:49.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:49 smithi055 ceph-mon[97033]: pgmap v20718: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:49 smithi059 ceph-mon[97450]: pgmap v20718: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:50.119+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:50.120665+0000) 2022-09-08T08:16:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:49.946+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:49.947827+0000) 2022-09-08T08:16:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:51.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:51.120925+0000) 2022-09-08T08:16:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:50.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:50.948006+0000) 2022-09-08T08:16:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:52.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:52.121152+0000) 2022-09-08T08:16:52.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:51.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:51.948166+0000) 2022-09-08T08:16:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:52 smithi059 ceph-mon[97450]: pgmap v20719: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:16:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:52 smithi055 ceph-mon[97033]: pgmap v20719: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:16:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:53.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:53.121340+0000) 2022-09-08T08:16:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:52.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:52.948382+0000) 2022-09-08T08:16:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:53.947+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:53.948678+0000) 2022-09-08T08:16:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:54.120+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:54.121514+0000) 2022-09-08T08:16:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:54 smithi059 ceph-mon[97450]: pgmap v20720: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:54 smithi055 ceph-mon[97033]: pgmap v20720: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:55.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:55.121721+0000) 2022-09-08T08:16:55.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:54.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:54.948849+0000) 2022-09-08T08:16:55.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:16:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:16:54.938+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:16:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:56.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:56.121969+0000) 2022-09-08T08:16:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:55.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:55.949040+0000) 2022-09-08T08:16:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:56 smithi059 ceph-mon[97450]: pgmap v20721: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:16:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:56 smithi055 ceph-mon[97033]: pgmap v20721: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:16:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:56.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:56.949244+0000) 2022-09-08T08:16:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:57.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:57.122226+0000) 2022-09-08T08:16:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:58.121+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:58.122445+0000) 2022-09-08T08:16:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:57.948+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:57.949421+0000) 2022-09-08T08:16:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:58 smithi059 ceph-mon[97450]: pgmap v20722: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:58 smithi055 ceph-mon[97033]: pgmap v20722: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:16:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:16:59.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:59.122725+0000) 2022-09-08T08:16:59.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:58.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:58.949676+0000) 2022-09-08T08:16:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:16:59 smithi055 ceph-mon[97033]: pgmap v20723: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:16:59 smithi059 ceph-mon[97450]: pgmap v20723: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:00.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:00.122955+0000) 2022-09-08T08:17:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:16:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:16:59.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:16:59.949846+0000) 2022-09-08T08:17:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:01.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:01.123195+0000) 2022-09-08T08:17:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:00.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:00.950058+0000) 2022-09-08T08:17:02.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:01.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:01.950326+0000) 2022-09-08T08:17:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:02.122+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:02.123380+0000) 2022-09-08T08:17:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:02 smithi059 ceph-mon[97450]: pgmap v20724: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:17:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:02 smithi055 ceph-mon[97033]: pgmap v20724: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:17:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:03.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:03.123620+0000) 2022-09-08T08:17:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:02.949+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:02.950555+0000) 2022-09-08T08:17:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:04.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:04.123875+0000) 2022-09-08T08:17:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:03.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:03.950762+0000) 2022-09-08T08:17:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:04 smithi059 ceph-mon[97450]: pgmap v20725: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:04 smithi055 ceph-mon[97033]: pgmap v20725: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:05.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:05.124106+0000) 2022-09-08T08:17:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:04.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:04.950957+0000) 2022-09-08T08:17:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:06.123+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:06.124395+0000) 2022-09-08T08:17:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:05.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:05.951155+0000) 2022-09-08T08:17:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:06 smithi059 ceph-mon[97450]: pgmap v20726: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:17:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:17:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:06 smithi055 ceph-mon[97033]: pgmap v20726: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:17:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:17:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:07.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:07.124684+0000) 2022-09-08T08:17:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:06.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:06.951378+0000) 2022-09-08T08:17:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:17:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:17:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:08.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:08.124900+0000) 2022-09-08T08:17:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:07.950+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:07.951606+0000) 2022-09-08T08:17:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:08 smithi059 ceph-mon[97450]: pgmap v20727: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:17:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:08 smithi055 ceph-mon[97033]: pgmap v20727: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:17:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:08.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:08.951800+0000) 2022-09-08T08:17:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:09.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:09.125062+0000) 2022-09-08T08:17:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:09 smithi055 ceph-mon[97033]: pgmap v20728: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:17:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:09 smithi059 ceph-mon[97450]: pgmap v20728: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:17:10.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:17:09.953+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:17:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:10.124+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:10.125257+0000) 2022-09-08T08:17:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:09.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:09.951950+0000) 2022-09-08T08:17:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:10 smithi055 ceph-mon[97033]: osdmap e1373: 12 total, 11 up, 11 in 2022-09-08T08:17:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:10 smithi059 ceph-mon[97450]: osdmap e1373: 12 total, 11 up, 11 in 2022-09-08T08:17:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:11.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:11.125525+0000) 2022-09-08T08:17:11.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:10.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:10.952113+0000) 2022-09-08T08:17:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:11 smithi055 ceph-mon[97033]: pgmap v20730: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:17:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:11 smithi059 ceph-mon[97450]: pgmap v20730: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:17:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:12.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:12.125755+0000) 2022-09-08T08:17:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:11.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:11.952290+0000) 2022-09-08T08:17:13.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:13.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:13.125980+0000) 2022-09-08T08:17:13.412 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:12.951+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:12.952468+0000) 2022-09-08T08:17:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:14.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:14.126138+0000) 2022-09-08T08:17:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:13.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:13.952719+0000) 2022-09-08T08:17:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:14 smithi059 ceph-mon[97450]: pgmap v20731: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:17:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:14 smithi059 ceph-mon[97450]: osdmap e1374: 12 total, 11 up, 11 in 2022-09-08T08:17:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:14 smithi055 ceph-mon[97033]: pgmap v20731: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:17:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:14 smithi055 ceph-mon[97033]: osdmap e1374: 12 total, 11 up, 11 in 2022-09-08T08:17:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:15.125+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:15.126308+0000) 2022-09-08T08:17:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:14.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:14.952874+0000) 2022-09-08T08:17:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:16.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:16.126520+0000) 2022-09-08T08:17:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:15.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:15.953102+0000) 2022-09-08T08:17:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:16 smithi059 ceph-mon[97450]: pgmap v20733: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:17:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:16 smithi055 ceph-mon[97033]: pgmap v20733: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:17:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:17.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:17.126694+0000) 2022-09-08T08:17:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:16.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:16.953216+0000) 2022-09-08T08:17:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:17.952+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:17.953411+0000) 2022-09-08T08:17:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:18.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:18.126895+0000) 2022-09-08T08:17:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:18 smithi059 ceph-mon[97450]: pgmap v20734: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:17:18.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:18 smithi055 ceph-mon[97033]: pgmap v20734: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:17:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:19.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:19.127074+0000) 2022-09-08T08:17:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:18.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:18.953539+0000) 2022-09-08T08:17:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:19 smithi055 ceph-mon[97033]: pgmap v20735: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T08:17:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:19 smithi059 ceph-mon[97450]: pgmap v20735: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T08:17:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:20.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:20.127311+0000) 2022-09-08T08:17:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:19.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:19.953698+0000) 2022-09-08T08:17:21.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:21.126+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:21.127501+0000) 2022-09-08T08:17:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:20.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:20.953874+0000) 2022-09-08T08:17:22.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:22.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:22.127717+0000) 2022-09-08T08:17:22.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:21.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:21.954124+0000) 2022-09-08T08:17:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:22 smithi059 ceph-mon[97450]: pgmap v20736: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:17:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:22 smithi055 ceph-mon[97033]: pgmap v20736: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:17:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:23.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:23.127946+0000) 2022-09-08T08:17:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:22.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:22.954352+0000) 2022-09-08T08:17:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:24.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:24.128139+0000) 2022-09-08T08:17:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:23.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:23.954548+0000) 2022-09-08T08:17:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:24 smithi059 ceph-mon[97450]: pgmap v20737: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:17:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:24 smithi055 ceph-mon[97033]: pgmap v20737: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:17:25.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:17:24.968+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:17:25.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:24.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:24.954744+0000) 2022-09-08T08:17:25.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:25.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:25.128368+0000) 2022-09-08T08:17:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:25.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:25.954924+0000) 2022-09-08T08:17:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:26.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:26.128586+0000) 2022-09-08T08:17:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:26 smithi059 ceph-mon[97450]: pgmap v20738: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-08T08:17:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:26 smithi055 ceph-mon[97033]: pgmap v20738: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-08T08:17:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:27.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:27.128782+0000) 2022-09-08T08:17:27.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:26.953+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:26.955137+0000) 2022-09-08T08:17:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:28.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:28.129007+0000) 2022-09-08T08:17:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:27.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:27.955323+0000) 2022-09-08T08:17:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:28 smithi059 ceph-mon[97450]: pgmap v20739: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:28 smithi055 ceph-mon[97033]: pgmap v20739: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:29.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:29.129173+0000) 2022-09-08T08:17:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:28.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:28.955494+0000) 2022-09-08T08:17:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:29 smithi055 ceph-mon[97033]: pgmap v20740: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:29 smithi059 ceph-mon[97450]: pgmap v20740: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:30.127+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:30.129311+0000) 2022-09-08T08:17:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:29.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:29.955672+0000) 2022-09-08T08:17:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:31.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:31.129524+0000) 2022-09-08T08:17:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:30.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:30.955827+0000) 2022-09-08T08:17:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:32.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:32.129795+0000) 2022-09-08T08:17:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:31.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:31.956000+0000) 2022-09-08T08:17:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:32 smithi059 ceph-mon[97450]: pgmap v20741: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:17:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:32 smithi055 ceph-mon[97033]: pgmap v20741: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:17:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:33.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:33.130024+0000) 2022-09-08T08:17:33.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:32.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:32.956225+0000) 2022-09-08T08:17:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:34.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:34.130250+0000) 2022-09-08T08:17:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:33.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:33.956459+0000) 2022-09-08T08:17:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:34 smithi059 ceph-mon[97450]: pgmap v20742: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:34 smithi055 ceph-mon[97033]: pgmap v20742: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:35.128+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:35.130476+0000) 2022-09-08T08:17:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:34.954+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:34.956591+0000) 2022-09-08T08:17:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:36.129+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:36.130716+0000) 2022-09-08T08:17:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:35.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:35.956759+0000) 2022-09-08T08:17:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:36 smithi059 ceph-mon[97450]: pgmap v20743: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:17:36.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:17:36.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:17:36.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:36 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:17:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:36 smithi055 ceph-mon[97033]: pgmap v20743: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:17:36.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:17:36.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:17:36.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:36 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:17:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:37.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:37.130914+0000) 2022-09-08T08:17:37.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:36.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:36.956974+0000) 2022-09-08T08:17:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:17:37.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:37 smithi055 ceph-mon[97033]: pgmap v20744: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:17:37.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:37 smithi059 ceph-mon[97450]: pgmap v20744: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:38.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:38.131126+0000) 2022-09-08T08:17:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:37.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:37.957181+0000) 2022-09-08T08:17:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:39.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:39.131308+0000) 2022-09-08T08:17:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:38.955+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:38.957394+0000) 2022-09-08T08:17:39.845 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:39 smithi055 ceph-mon[97033]: pgmap v20745: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:39 smithi059 ceph-mon[97450]: pgmap v20745: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:40.130+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:40.131553+0000) 2022-09-08T08:17:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:39.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:39.957658+0000) 2022-09-08T08:17:40.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:17:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:17:39.979+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:17:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:41.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:41.131782+0000) 2022-09-08T08:17:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:40.956+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:40.957828+0000) 2022-09-08T08:17:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:41.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:41.957982+0000) 2022-09-08T08:17:42.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:42.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:42.131906+0000) 2022-09-08T08:17:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:42 smithi059 ceph-mon[97450]: pgmap v20746: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:17:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:42 smithi055 ceph-mon[97033]: pgmap v20746: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:17:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:42.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:42.958133+0000) 2022-09-08T08:17:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:43.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:43.132112+0000) 2022-09-08T08:17:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:44.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:44.132307+0000) 2022-09-08T08:17:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:43.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:43.958299+0000) 2022-09-08T08:17:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:44 smithi059 ceph-mon[97450]: pgmap v20747: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:44 smithi055 ceph-mon[97033]: pgmap v20747: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:45.131+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:45.132520+0000) 2022-09-08T08:17:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:44.957+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:44.958517+0000) 2022-09-08T08:17:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:46.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:46.132761+0000) 2022-09-08T08:17:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:45.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:45.958693+0000) 2022-09-08T08:17:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:46 smithi059 ceph-mon[97450]: pgmap v20748: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:17:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:46 smithi055 ceph-mon[97033]: pgmap v20748: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:17:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:47.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:47.132964+0000) 2022-09-08T08:17:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:46.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:46.958831+0000) 2022-09-08T08:17:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:48.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:48.133223+0000) 2022-09-08T08:17:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:47.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:47.959020+0000) 2022-09-08T08:17:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:48 smithi059 ceph-mon[97450]: pgmap v20749: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:48.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:48 smithi055 ceph-mon[97033]: pgmap v20749: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:49.132+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:49.133433+0000) 2022-09-08T08:17:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:48.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:48.959247+0000) 2022-09-08T08:17:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:49 smithi055 ceph-mon[97033]: pgmap v20750: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:49 smithi059 ceph-mon[97450]: pgmap v20750: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:50.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:50.133719+0000) 2022-09-08T08:17:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:49.958+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:49.959445+0000) 2022-09-08T08:17:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:50.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:50.959736+0000) 2022-09-08T08:17:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:51.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:51.133950+0000) 2022-09-08T08:17:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:51.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:51.959996+0000) 2022-09-08T08:17:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:52.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:52.134128+0000) 2022-09-08T08:17:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:52 smithi059 ceph-mon[97450]: pgmap v20751: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:17:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:52 smithi055 ceph-mon[97033]: pgmap v20751: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:17:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:53.133+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:53.134360+0000) 2022-09-08T08:17:53.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:52.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:52.960247+0000) 2022-09-08T08:17:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:54.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:54.134592+0000) 2022-09-08T08:17:54.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:53.959+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:53.960488+0000) 2022-09-08T08:17:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:54 smithi059 ceph-mon[97450]: pgmap v20752: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:54 smithi055 ceph-mon[97033]: pgmap v20752: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:55.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:17:54.990+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:17:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:55.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:55.134778+0000) 2022-09-08T08:17:55.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:54.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:54.960748+0000) 2022-09-08T08:17:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:56.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:56.135013+0000) 2022-09-08T08:17:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:55.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:55.961001+0000) 2022-09-08T08:17:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:56 smithi059 ceph-mon[97450]: pgmap v20753: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:17:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:56 smithi055 ceph-mon[97033]: pgmap v20753: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:17:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:56.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:56.961208+0000) 2022-09-08T08:17:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:57.134+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:57.135212+0000) 2022-09-08T08:17:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:58.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:58.135421+0000) 2022-09-08T08:17:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:57.960+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:57.961435+0000) 2022-09-08T08:17:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:58 smithi059 ceph-mon[97450]: pgmap v20754: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:58 smithi055 ceph-mon[97033]: pgmap v20754: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:17:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:17:59.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:59.135663+0000) 2022-09-08T08:17:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:58.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:58.961693+0000) 2022-09-08T08:17:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:17:59 smithi055 ceph-mon[97033]: pgmap v20755: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:17:59 smithi059 ceph-mon[97450]: pgmap v20755: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:00.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:00.135910+0000) 2022-09-08T08:18:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:17:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:17:59.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:17:59.961864+0000) 2022-09-08T08:18:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:01.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:01.136030+0000) 2022-09-08T08:18:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:00.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:00.961979+0000) 2022-09-08T08:18:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:02.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:02.136273+0000) 2022-09-08T08:18:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:01.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:01.962115+0000) 2022-09-08T08:18:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:02 smithi059 ceph-mon[97450]: pgmap v20756: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:18:02.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:02 smithi055 ceph-mon[97033]: pgmap v20756: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:18:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:02.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:02.962330+0000) 2022-09-08T08:18:03.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:03.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:03.136487+0000) 2022-09-08T08:18:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:04.135+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:04.136611+0000) 2022-09-08T08:18:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:03.961+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:03.962562+0000) 2022-09-08T08:18:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:04 smithi059 ceph-mon[97450]: pgmap v20757: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:04 smithi055 ceph-mon[97033]: pgmap v20757: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:05.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:05.136796+0000) 2022-09-08T08:18:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:04.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:04.962795+0000) 2022-09-08T08:18:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:06.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:06.137015+0000) 2022-09-08T08:18:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:05.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:05.963030+0000) 2022-09-08T08:18:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:06 smithi059 ceph-mon[97450]: pgmap v20758: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:18:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:18:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:06 smithi055 ceph-mon[97033]: pgmap v20758: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:18:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:18:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:07.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:07.137179+0000) 2022-09-08T08:18:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:06.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:06.963243+0000) 2022-09-08T08:18:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:18:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:18:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:08.136+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:08.137360+0000) 2022-09-08T08:18:08.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:07.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:07.963376+0000) 2022-09-08T08:18:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:08 smithi059 ceph-mon[97450]: pgmap v20759: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:18:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:08 smithi055 ceph-mon[97033]: pgmap v20759: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:18:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:09.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:09.137601+0000) 2022-09-08T08:18:09.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:08.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:08.963536+0000) 2022-09-08T08:18:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:09 smithi055 ceph-mon[97033]: pgmap v20760: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:18:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:09 smithi059 ceph-mon[97450]: pgmap v20760: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:18:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:10.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:10.137834+0000) 2022-09-08T08:18:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:09.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:09.963718+0000) 2022-09-08T08:18:10.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:18:10.005+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:18:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:10 smithi055 ceph-mon[97033]: osdmap e1375: 12 total, 11 up, 11 in 2022-09-08T08:18:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:10 smithi059 ceph-mon[97450]: osdmap e1375: 12 total, 11 up, 11 in 2022-09-08T08:18:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:11.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:11.138065+0000) 2022-09-08T08:18:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:10.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:10.963924+0000) 2022-09-08T08:18:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:11 smithi055 ceph-mon[97033]: pgmap v20762: 129 pgs: 3 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 114 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:18:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:11 smithi059 ceph-mon[97450]: pgmap v20762: 129 pgs: 3 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 114 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:18:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:12.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:12.138271+0000) 2022-09-08T08:18:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:11.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:11.964163+0000) 2022-09-08T08:18:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:13.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:13.138480+0000) 2022-09-08T08:18:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:12.962+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:12.964368+0000) 2022-09-08T08:18:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:14.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:14.138738+0000) 2022-09-08T08:18:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:13.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:13.964618+0000) 2022-09-08T08:18:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:14 smithi059 ceph-mon[97450]: pgmap v20763: 129 pgs: 3 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 114 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:18:14.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:14 smithi059 ceph-mon[97450]: osdmap e1376: 12 total, 11 up, 11 in 2022-09-08T08:18:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:14 smithi055 ceph-mon[97033]: pgmap v20763: 129 pgs: 3 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 114 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:18:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:14 smithi055 ceph-mon[97033]: osdmap e1376: 12 total, 11 up, 11 in 2022-09-08T08:18:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:15.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:15.138890+0000) 2022-09-08T08:18:15.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:14.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:14.964914+0000) 2022-09-08T08:18:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:16.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:16.139074+0000) 2022-09-08T08:18:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:15.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:15.965069+0000) 2022-09-08T08:18:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:16 smithi059 ceph-mon[97450]: pgmap v20765: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:18:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:16 smithi055 ceph-mon[97033]: pgmap v20765: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:18:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:17.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:17.139286+0000) 2022-09-08T08:18:17.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:16.963+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:16.965226+0000) 2022-09-08T08:18:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:18.137+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:18.139528+0000) 2022-09-08T08:18:18.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:17.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:17.965399+0000) 2022-09-08T08:18:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:18 smithi059 ceph-mon[97450]: pgmap v20766: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:18:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:18 smithi055 ceph-mon[97033]: pgmap v20766: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:18:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:19.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:19.139716+0000) 2022-09-08T08:18:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:18.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:18.965568+0000) 2022-09-08T08:18:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:19 smithi055 ceph-mon[97033]: osdmap e1377: 12 total, 11 up, 11 in 2022-09-08T08:18:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:19 smithi055 ceph-mon[97033]: pgmap v20768: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:18:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:19 smithi059 ceph-mon[97450]: osdmap e1377: 12 total, 11 up, 11 in 2022-09-08T08:18:20.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:19 smithi059 ceph-mon[97450]: pgmap v20768: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:18:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:20.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:20.139921+0000) 2022-09-08T08:18:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:19.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:19.965761+0000) 2022-09-08T08:18:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:21.138+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:21.140061+0000) 2022-09-08T08:18:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:20.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:20.965973+0000) 2022-09-08T08:18:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:22.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:22.140217+0000) 2022-09-08T08:18:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:21.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:21.966188+0000) 2022-09-08T08:18:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:22 smithi059 ceph-mon[97450]: pgmap v20769: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T08:18:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:22 smithi055 ceph-mon[97033]: pgmap v20769: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T08:18:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:23.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:23.140485+0000) 2022-09-08T08:18:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:22.964+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:22.966400+0000) 2022-09-08T08:18:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:24.139+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:24.140808+0000) 2022-09-08T08:18:24.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:23.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:23.966664+0000) 2022-09-08T08:18:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:24 smithi059 ceph-mon[97450]: pgmap v20770: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 106 B/s wr, 6 op/s 2022-09-08T08:18:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:24 smithi055 ceph-mon[97033]: pgmap v20770: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 106 B/s wr, 6 op/s 2022-09-08T08:18:25.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:18:25.020+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:18:25.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:25.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:25.141042+0000) 2022-09-08T08:18:25.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:24.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:24.966841+0000) 2022-09-08T08:18:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:25.965+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:25.966985+0000) 2022-09-08T08:18:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:26.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:26.141294+0000) 2022-09-08T08:18:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:26 smithi059 ceph-mon[97450]: pgmap v20771: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:18:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:26 smithi055 ceph-mon[97033]: pgmap v20771: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:18:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:27.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:27.141539+0000) 2022-09-08T08:18:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:26.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:26.967125+0000) 2022-09-08T08:18:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:28.140+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:28.141722+0000) 2022-09-08T08:18:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:27.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:27.967309+0000) 2022-09-08T08:18:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:28 smithi059 ceph-mon[97450]: pgmap v20772: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:18:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:28 smithi055 ceph-mon[97033]: pgmap v20772: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:18:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:29.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:29.141984+0000) 2022-09-08T08:18:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:28.966+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:28.967533+0000) 2022-09-08T08:18:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:29 smithi055 ceph-mon[97033]: pgmap v20773: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:18:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:29 smithi059 ceph-mon[97450]: pgmap v20773: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:18:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:29.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:29.967696+0000) 2022-09-08T08:18:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:30.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:30.142225+0000) 2022-09-08T08:18:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:30.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:30.967912+0000) 2022-09-08T08:18:31.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:31.141+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:31.142462+0000) 2022-09-08T08:18:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:32.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:32.142728+0000) 2022-09-08T08:18:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:31.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:31.968111+0000) 2022-09-08T08:18:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:32 smithi059 ceph-mon[97450]: pgmap v20774: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:18:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:32 smithi055 ceph-mon[97033]: pgmap v20774: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:18:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:33.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:33.142949+0000) 2022-09-08T08:18:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:32.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:32.968355+0000) 2022-09-08T08:18:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:34.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:34.143169+0000) 2022-09-08T08:18:34.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:33.967+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:33.968535+0000) 2022-09-08T08:18:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:34 smithi059 ceph-mon[97450]: pgmap v20775: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:34 smithi055 ceph-mon[97033]: pgmap v20775: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:34.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:34.968701+0000) 2022-09-08T08:18:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:35.142+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:35.143378+0000) 2022-09-08T08:18:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:35.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:35.968843+0000) 2022-09-08T08:18:36.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:36.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:36.143621+0000) 2022-09-08T08:18:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:36 smithi059 ceph-mon[97450]: pgmap v20776: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:18:36.541 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:36 smithi055 ceph-mon[97033]: pgmap v20776: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:18:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:37.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:37.143912+0000) 2022-09-08T08:18:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:36.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:36.969034+0000) 2022-09-08T08:18:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:18:37.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:18:37.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:18:37.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:18:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:18:37.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:18:37.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:18:37.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:18:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:38.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:38.144084+0000) 2022-09-08T08:18:38.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:37.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:37.969265+0000) 2022-09-08T08:18:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:38 smithi059 ceph-mon[97450]: pgmap v20777: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:38 smithi055 ceph-mon[97033]: pgmap v20777: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:39.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:39.144328+0000) 2022-09-08T08:18:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:38.968+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:38.969501+0000) 2022-09-08T08:18:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:39 smithi055 ceph-mon[97033]: pgmap v20778: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:39 smithi059 ceph-mon[97450]: pgmap v20778: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:40.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:18:40.035+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:18:40.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:40.143+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:40.144520+0000) 2022-09-08T08:18:40.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:39.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:39.969753+0000) 2022-09-08T08:18:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:40.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:40.969902+0000) 2022-09-08T08:18:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:41.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:41.144684+0000) 2022-09-08T08:18:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:42.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:42.144901+0000) 2022-09-08T08:18:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:41.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:41.970040+0000) 2022-09-08T08:18:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:42 smithi059 ceph-mon[97450]: pgmap v20779: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:18:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:42 smithi055 ceph-mon[97033]: pgmap v20779: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:18:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:43.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:43.145132+0000) 2022-09-08T08:18:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:42.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:42.970194+0000) 2022-09-08T08:18:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:43.969+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:43.970407+0000) 2022-09-08T08:18:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:44.144+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:44.145344+0000) 2022-09-08T08:18:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:44 smithi059 ceph-mon[97450]: pgmap v20780: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:44 smithi055 ceph-mon[97033]: pgmap v20780: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:45.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:45.145669+0000) 2022-09-08T08:18:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:44.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:44.970612+0000) 2022-09-08T08:18:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:45.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:45.970813+0000) 2022-09-08T08:18:46.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:46.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:46.145868+0000) 2022-09-08T08:18:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:46 smithi059 ceph-mon[97450]: pgmap v20781: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:18:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:46 smithi055 ceph-mon[97033]: pgmap v20781: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:18:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:46.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:46.971053+0000) 2022-09-08T08:18:47.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:47.145+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:47.146122+0000) 2022-09-08T08:18:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:48.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:48.146409+0000) 2022-09-08T08:18:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:47.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:47.971242+0000) 2022-09-08T08:18:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:48 smithi059 ceph-mon[97450]: pgmap v20782: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:48 smithi055 ceph-mon[97033]: pgmap v20782: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:49.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:49.146711+0000) 2022-09-08T08:18:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:48.970+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:48.971469+0000) 2022-09-08T08:18:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:49 smithi055 ceph-mon[97033]: pgmap v20783: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:49 smithi059 ceph-mon[97450]: pgmap v20783: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:50.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:50.146931+0000) 2022-09-08T08:18:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:49.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:49.971689+0000) 2022-09-08T08:18:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:51.146+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:51.147100+0000) 2022-09-08T08:18:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:50.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:50.971833+0000) 2022-09-08T08:18:52.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:51.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:51.972037+0000) 2022-09-08T08:18:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:52.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:52.147335+0000) 2022-09-08T08:18:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:52 smithi059 ceph-mon[97450]: pgmap v20784: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:18:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:52 smithi055 ceph-mon[97033]: pgmap v20784: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:18:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:53.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:53.147572+0000) 2022-09-08T08:18:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:52.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:52.972217+0000) 2022-09-08T08:18:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:54.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:54.147810+0000) 2022-09-08T08:18:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:53.971+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:53.972415+0000) 2022-09-08T08:18:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:54 smithi059 ceph-mon[97450]: pgmap v20785: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:54 smithi055 ceph-mon[97033]: pgmap v20785: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:18:55.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:18:55.046+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:18:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:54.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:54.972600+0000) 2022-09-08T08:18:55.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:55.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:55.148032+0000) 2022-09-08T08:18:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:56.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:56.148249+0000) 2022-09-08T08:18:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:55.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:55.972778+0000) 2022-09-08T08:18:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:56 smithi059 ceph-mon[97450]: pgmap v20786: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:18:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:56 smithi055 ceph-mon[97033]: pgmap v20786: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:18:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:57.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:57.148461+0000) 2022-09-08T08:18:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:56.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:56.973043+0000) 2022-09-08T08:18:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:58.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:58.148722+0000) 2022-09-08T08:18:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:57.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:57.973184+0000) 2022-09-08T08:18:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:58 smithi055 ceph-mon[97033]: pgmap v20787: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:18:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:58 smithi059 ceph-mon[97450]: pgmap v20787: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:18:59.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:18:59.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:59.148951+0000) 2022-09-08T08:18:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:58.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:58.973410+0000) 2022-09-08T08:18:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:18:59 smithi055 ceph-mon[97033]: pgmap v20788: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:19:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:18:59 smithi059 ceph-mon[97450]: pgmap v20788: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:19:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:00.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:00.149167+0000) 2022-09-08T08:19:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:18:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:18:59.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:18:59.973630+0000) 2022-09-08T08:19:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:01.147+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:01.149286+0000) 2022-09-08T08:19:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:00.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:00.973868+0000) 2022-09-08T08:19:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:02.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:02.149492+0000) 2022-09-08T08:19:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:01.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:01.974075+0000) 2022-09-08T08:19:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:02 smithi059 ceph-mon[97450]: pgmap v20789: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:19:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:02 smithi055 ceph-mon[97033]: pgmap v20789: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:19:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:03.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:03.149729+0000) 2022-09-08T08:19:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:02.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:02.974250+0000) 2022-09-08T08:19:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:04.148+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:04.149926+0000) 2022-09-08T08:19:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:03.972+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:03.974381+0000) 2022-09-08T08:19:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:04 smithi059 ceph-mon[97450]: pgmap v20790: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:19:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:04 smithi055 ceph-mon[97033]: pgmap v20790: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:19:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:05.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:05.150144+0000) 2022-09-08T08:19:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:04.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:04.974630+0000) 2022-09-08T08:19:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:06.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:06.150391+0000) 2022-09-08T08:19:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:05.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:05.974898+0000) 2022-09-08T08:19:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:06 smithi059 ceph-mon[97450]: pgmap v20791: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:19:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:19:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:06 smithi055 ceph-mon[97033]: pgmap v20791: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:19:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:19:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:07.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:07.150619+0000) 2022-09-08T08:19:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:06.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:06.975071+0000) 2022-09-08T08:19:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:19:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:19:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:08.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:08.150895+0000) 2022-09-08T08:19:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:07.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:07.975263+0000) 2022-09-08T08:19:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:08 smithi055 ceph-mon[97033]: pgmap v20792: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:19:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:08 smithi059 ceph-mon[97450]: pgmap v20792: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:19:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:09.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:09.151045+0000) 2022-09-08T08:19:09.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:08.973+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:08.975471+0000) 2022-09-08T08:19:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:09 smithi055 ceph-mon[97033]: pgmap v20793: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:19:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:09 smithi059 ceph-mon[97450]: pgmap v20793: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:19:10.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:19:10.049+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:19:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:10.149+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:10.151211+0000) 2022-09-08T08:19:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:09.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:09.975726+0000) 2022-09-08T08:19:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:10 smithi055 ceph-mon[97033]: osdmap e1378: 12 total, 11 up, 11 in 2022-09-08T08:19:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:10 smithi059 ceph-mon[97450]: osdmap e1378: 12 total, 11 up, 11 in 2022-09-08T08:19:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:11.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:11.151448+0000) 2022-09-08T08:19:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:10.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:10.975937+0000) 2022-09-08T08:19:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:11 smithi055 ceph-mon[97033]: pgmap v20795: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:19:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:11 smithi059 ceph-mon[97450]: pgmap v20795: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:19:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:12.150+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:12.151677+0000) 2022-09-08T08:19:12.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:11.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:11.976148+0000) 2022-09-08T08:19:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:13.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:13.151917+0000) 2022-09-08T08:19:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:12.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:12.976370+0000) 2022-09-08T08:19:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:14.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:14.152168+0000) 2022-09-08T08:19:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:13.974+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:13.976548+0000) 2022-09-08T08:19:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:14 smithi059 ceph-mon[97450]: pgmap v20796: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:19:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:14 smithi055 ceph-mon[97033]: pgmap v20796: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:19:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:15.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:15.152436+0000) 2022-09-08T08:19:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:14.975+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:14.976769+0000) 2022-09-08T08:19:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:15.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:15.976994+0000) 2022-09-08T08:19:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:16.151+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:16.152674+0000) 2022-09-08T08:19:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:16 smithi055 ceph-mon[97033]: pgmap v20797: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:19:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:16 smithi059 ceph-mon[97450]: pgmap v20797: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:19:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:17.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:17.152897+0000) 2022-09-08T08:19:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:16.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:16.977196+0000) 2022-09-08T08:19:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:18.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:18.153152+0000) 2022-09-08T08:19:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:17.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:17.977331+0000) 2022-09-08T08:19:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:18 smithi055 ceph-mon[97033]: pgmap v20798: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:19:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:18 smithi059 ceph-mon[97450]: pgmap v20798: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:19:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:19.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:19.153379+0000) 2022-09-08T08:19:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:18.976+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:18.977473+0000) 2022-09-08T08:19:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:19 smithi055 ceph-mon[97033]: osdmap e1379: 12 total, 11 up, 11 in 2022-09-08T08:19:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:19 smithi055 ceph-mon[97033]: pgmap v20800: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T08:19:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:19 smithi059 ceph-mon[97450]: osdmap e1379: 12 total, 11 up, 11 in 2022-09-08T08:19:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:19 smithi059 ceph-mon[97450]: pgmap v20800: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T08:19:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:20.152+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:20.153567+0000) 2022-09-08T08:19:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:19.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:19.977738+0000) 2022-09-08T08:19:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:21.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:21.153793+0000) 2022-09-08T08:19:21.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:20.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:20.977912+0000) 2022-09-08T08:19:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:22.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:22.154038+0000) 2022-09-08T08:19:22.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:21.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:21.978155+0000) 2022-09-08T08:19:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:22 smithi055 ceph-mon[97033]: pgmap v20801: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:19:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:22 smithi059 ceph-mon[97450]: pgmap v20801: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:19:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:23.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:23.154214+0000) 2022-09-08T08:19:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:22.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:22.978314+0000) 2022-09-08T08:19:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:24.153+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:24.154489+0000) 2022-09-08T08:19:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:23.977+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:23.978461+0000) 2022-09-08T08:19:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:24 smithi055 ceph-mon[97033]: pgmap v20802: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:19:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:24 smithi059 ceph-mon[97450]: pgmap v20802: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:19:25.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:19:25.054+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:19:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:25.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:25.154765+0000) 2022-09-08T08:19:25.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:24.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:24.978745+0000) 2022-09-08T08:19:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:26.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:26.155002+0000) 2022-09-08T08:19:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:25.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:25.978913+0000) 2022-09-08T08:19:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:26 smithi055 ceph-mon[97033]: pgmap v20803: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:19:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:26 smithi059 ceph-mon[97450]: pgmap v20803: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:19:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:26.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:26.979049+0000) 2022-09-08T08:19:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:27.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:27.155145+0000) 2022-09-08T08:19:28.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:27.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:27.979180+0000) 2022-09-08T08:19:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:28.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:28.155351+0000) 2022-09-08T08:19:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:28 smithi055 ceph-mon[97033]: pgmap v20804: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:19:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:28 smithi059 ceph-mon[97450]: pgmap v20804: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:19:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:29.154+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:29.155531+0000) 2022-09-08T08:19:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:28.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:28.979342+0000) 2022-09-08T08:19:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:29 smithi055 ceph-mon[97033]: pgmap v20805: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:19:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:29 smithi059 ceph-mon[97450]: pgmap v20805: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:19:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:30.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:30.155735+0000) 2022-09-08T08:19:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:29.978+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:29.979561+0000) 2022-09-08T08:19:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:31.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:31.155927+0000) 2022-09-08T08:19:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:30.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:30.979757+0000) 2022-09-08T08:19:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:32.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:32.156125+0000) 2022-09-08T08:19:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:31.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:31.979953+0000) 2022-09-08T08:19:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:32 smithi055 ceph-mon[97033]: pgmap v20806: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:19:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:32 smithi059 ceph-mon[97450]: pgmap v20806: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:19:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:33.155+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:33.156326+0000) 2022-09-08T08:19:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:32.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:32.980133+0000) 2022-09-08T08:19:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:34.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:34.156509+0000) 2022-09-08T08:19:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:33.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:33.980370+0000) 2022-09-08T08:19:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:34 smithi055 ceph-mon[97033]: pgmap v20807: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:34 smithi059 ceph-mon[97450]: pgmap v20807: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:35.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:35.156770+0000) 2022-09-08T08:19:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:34.979+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:34.980601+0000) 2022-09-08T08:19:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:36.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:36.156955+0000) 2022-09-08T08:19:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:35.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:35.980836+0000) 2022-09-08T08:19:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:36 smithi055 ceph-mon[97033]: pgmap v20808: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:19:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:36 smithi059 ceph-mon[97450]: pgmap v20808: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:19:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:37.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:37.157123+0000) 2022-09-08T08:19:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:36.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:36.981040+0000) 2022-09-08T08:19:37.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:19:37.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:19:37.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:37 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:19:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:19:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:19:37.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:37 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:19:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:38.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:38.157320+0000) 2022-09-08T08:19:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:37.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:37.981208+0000) 2022-09-08T08:19:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:38 smithi055 ceph-mon[97033]: pgmap v20809: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:19:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:38 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:19:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:38 smithi059 ceph-mon[97450]: pgmap v20809: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:19:38.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:38 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:19:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:39.156+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:39.157508+0000) 2022-09-08T08:19:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:38.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:38.981353+0000) 2022-09-08T08:19:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:39 smithi055 ceph-mon[97033]: pgmap v20810: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:39 smithi059 ceph-mon[97450]: pgmap v20810: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:40.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:40.157720+0000) 2022-09-08T08:19:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:39.980+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:39.981556+0000) 2022-09-08T08:19:40.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:19:40.059+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:19:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:19:41.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:19:41.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:19:41.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:19:41.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.285 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.286 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:41 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:19:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:41.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:41.157884+0000) 2022-09-08T08:19:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:40.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:40.981802+0000) 2022-09-08T08:19:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:19:41.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:19:41.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:19:41.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:19:41.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:19:41.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:41 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:19:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:42.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:42.158111+0000) 2022-09-08T08:19:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:41.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:41.981946+0000) 2022-09-08T08:19:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:42 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 92192k 2022-09-08T08:19:42.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:42 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 94404776: error parsing value: Value '94404776' is below minimum 939524096 2022-09-08T08:19:42.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:42 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T08:19:42.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:42 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T08:19:42.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:42 smithi055 ceph-mon[97033]: pgmap v20811: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:19:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:42 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 92192k 2022-09-08T08:19:42.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:42 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 94404776: error parsing value: Value '94404776' is below minimum 939524096 2022-09-08T08:19:42.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:42 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T08:19:42.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:42 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T08:19:42.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:42 smithi059 ceph-mon[97450]: pgmap v20811: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:19:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:43.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:43.158332+0000) 2022-09-08T08:19:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:42.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:42.982097+0000) 2022-09-08T08:19:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:44.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:44.158542+0000) 2022-09-08T08:19:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:43.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:43.982317+0000) 2022-09-08T08:19:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:44 smithi055 ceph-mon[97033]: pgmap v20812: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:44 smithi059 ceph-mon[97450]: pgmap v20812: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:45.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:45.158732+0000) 2022-09-08T08:19:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:44.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:44.982507+0000) 2022-09-08T08:19:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:46.157+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:46.158875+0000) 2022-09-08T08:19:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:45.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:45.982711+0000) 2022-09-08T08:19:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:46 smithi055 ceph-mon[97033]: pgmap v20813: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:19:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:46 smithi059 ceph-mon[97450]: pgmap v20813: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:19:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:47.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:47.159073+0000) 2022-09-08T08:19:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:46.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:46.982853+0000) 2022-09-08T08:19:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:48.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:48.159363+0000) 2022-09-08T08:19:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:47.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:47.983073+0000) 2022-09-08T08:19:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:48 smithi055 ceph-mon[97033]: pgmap v20814: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:48 smithi059 ceph-mon[97450]: pgmap v20814: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:49.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:49.159625+0000) 2022-09-08T08:19:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:48.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:48.983231+0000) 2022-09-08T08:19:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:49 smithi055 ceph-mon[97033]: pgmap v20815: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:49 smithi059 ceph-mon[97450]: pgmap v20815: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:50.158+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:50.159945+0000) 2022-09-08T08:19:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:49.981+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:49.983441+0000) 2022-09-08T08:19:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:51.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:51.160186+0000) 2022-09-08T08:19:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:50.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:50.983590+0000) 2022-09-08T08:19:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:52.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:52.160376+0000) 2022-09-08T08:19:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:51.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:51.983794+0000) 2022-09-08T08:19:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:52 smithi055 ceph-mon[97033]: pgmap v20816: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:19:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:52 smithi059 ceph-mon[97450]: pgmap v20816: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:19:53.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:52.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:52.983968+0000) 2022-09-08T08:19:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:53.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:53.160613+0000) 2022-09-08T08:19:54.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:54.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:54.160820+0000) 2022-09-08T08:19:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:53.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:53.984122+0000) 2022-09-08T08:19:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:54 smithi055 ceph-mon[97033]: pgmap v20817: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:54 smithi059 ceph-mon[97450]: pgmap v20817: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:55.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:19:55.074+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:19:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:55.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:55.160950+0000) 2022-09-08T08:19:55.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:54.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:54.984252+0000) 2022-09-08T08:19:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:56.159+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:56.161111+0000) 2022-09-08T08:19:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:55.982+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:55.984398+0000) 2022-09-08T08:19:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:56 smithi055 ceph-mon[97033]: pgmap v20818: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:19:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:56 smithi059 ceph-mon[97450]: pgmap v20818: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:19:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:57.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:57.161341+0000) 2022-09-08T08:19:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:56.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:56.984649+0000) 2022-09-08T08:19:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:58.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:58.161585+0000) 2022-09-08T08:19:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:57.983+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:57.984804+0000) 2022-09-08T08:19:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:58 smithi055 ceph-mon[97033]: pgmap v20819: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:58 smithi059 ceph-mon[97450]: pgmap v20819: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:19:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:58.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:58.984958+0000) 2022-09-08T08:19:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:19:59.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:59.161873+0000) 2022-09-08T08:19:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:19:59 smithi055 ceph-mon[97033]: pgmap v20820: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:19:59 smithi059 ceph-mon[97450]: pgmap v20820: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:19:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:19:59.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:19:59.985172+0000) 2022-09-08T08:20:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:00.160+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:00.162142+0000) 2022-09-08T08:20:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T08:20:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T08:20:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T08:20:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T08:20:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T08:20:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T08:20:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:01.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:01.162410+0000) 2022-09-08T08:20:01.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:00.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:00.985318+0000) 2022-09-08T08:20:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:01 smithi055 ceph-mon[97033]: pgmap v20821: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:20:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:01 smithi059 ceph-mon[97450]: pgmap v20821: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:20:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:02.161+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:02.162721+0000) 2022-09-08T08:20:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:01.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:01.985544+0000) 2022-09-08T08:20:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:03.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:03.162984+0000) 2022-09-08T08:20:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:02.984+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:02.985793+0000) 2022-09-08T08:20:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:04.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:04.163256+0000) 2022-09-08T08:20:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:03.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:03.985960+0000) 2022-09-08T08:20:04.547 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:04 smithi055 ceph-mon[97033]: pgmap v20822: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:04 smithi059 ceph-mon[97450]: pgmap v20822: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:05.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:04.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:04.986100+0000) 2022-09-08T08:20:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:05.162+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:05.163446+0000) 2022-09-08T08:20:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:06.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:06.163726+0000) 2022-09-08T08:20:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:05.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:05.986302+0000) 2022-09-08T08:20:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:06 smithi055 ceph-mon[97033]: pgmap v20823: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:20:06.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:20:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:06 smithi059 ceph-mon[97450]: pgmap v20823: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:20:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:20:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:07.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:07.163930+0000) 2022-09-08T08:20:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:06.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:06.986433+0000) 2022-09-08T08:20:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:20:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:20:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:08.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:08.164173+0000) 2022-09-08T08:20:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:07.985+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:07.986573+0000) 2022-09-08T08:20:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:08 smithi055 ceph-mon[97033]: pgmap v20824: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:20:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:08 smithi059 ceph-mon[97450]: pgmap v20824: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:20:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:09.163+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:09.164419+0000) 2022-09-08T08:20:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:08.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:08.986710+0000) 2022-09-08T08:20:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:09 smithi055 ceph-mon[97033]: pgmap v20825: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:20:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:09 smithi059 ceph-mon[97450]: pgmap v20825: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:20:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:10.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:10.164686+0000) 2022-09-08T08:20:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:09.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:09.986845+0000) 2022-09-08T08:20:10.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:20:10.090+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:20:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:10 smithi055 ceph-mon[97033]: osdmap e1380: 12 total, 11 up, 11 in 2022-09-08T08:20:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:10 smithi059 ceph-mon[97450]: osdmap e1380: 12 total, 11 up, 11 in 2022-09-08T08:20:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:11.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:11.164830+0000) 2022-09-08T08:20:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:10.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:10.987030+0000) 2022-09-08T08:20:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:11 smithi055 ceph-mon[97033]: pgmap v20827: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:20:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:11 smithi059 ceph-mon[97450]: pgmap v20827: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:20:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:11.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:11.987247+0000) 2022-09-08T08:20:12.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:12.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:12.165059+0000) 2022-09-08T08:20:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:12.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:12.987397+0000) 2022-09-08T08:20:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:13.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:13.165244+0000) 2022-09-08T08:20:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:14.164+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:14.165476+0000) 2022-09-08T08:20:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:13.986+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:13.987622+0000) 2022-09-08T08:20:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:14 smithi055 ceph-mon[97033]: pgmap v20828: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:20:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:14 smithi059 ceph-mon[97450]: pgmap v20828: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:20:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:15.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:15.165707+0000) 2022-09-08T08:20:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:14.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:14.987792+0000) 2022-09-08T08:20:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:16.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:16.165947+0000) 2022-09-08T08:20:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:15.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:15.987946+0000) 2022-09-08T08:20:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:16 smithi055 ceph-mon[97033]: pgmap v20829: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:20:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:16 smithi059 ceph-mon[97450]: pgmap v20829: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:20:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:17.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:17.166155+0000) 2022-09-08T08:20:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:16.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:16.988084+0000) 2022-09-08T08:20:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:18.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:18.166373+0000) 2022-09-08T08:20:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:17.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:17.988335+0000) 2022-09-08T08:20:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:18 smithi055 ceph-mon[97033]: pgmap v20830: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:20:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:18 smithi059 ceph-mon[97450]: pgmap v20830: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:20:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:19.165+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:19.166613+0000) 2022-09-08T08:20:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:18.987+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:18.988504+0000) 2022-09-08T08:20:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:19 smithi055 ceph-mon[97033]: osdmap e1381: 12 total, 11 up, 11 in 2022-09-08T08:20:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:19 smithi055 ceph-mon[97033]: pgmap v20832: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T08:20:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:19 smithi059 ceph-mon[97450]: osdmap e1381: 12 total, 11 up, 11 in 2022-09-08T08:20:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:19 smithi059 ceph-mon[97450]: pgmap v20832: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T08:20:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:20.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:20.166941+0000) 2022-09-08T08:20:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:19.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:19.988753+0000) 2022-09-08T08:20:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:21.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:21.167220+0000) 2022-09-08T08:20:21.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:20.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:20.988892+0000) 2022-09-08T08:20:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:22.166+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:22.167474+0000) 2022-09-08T08:20:22.434 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:21.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:21.989045+0000) 2022-09-08T08:20:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:22 smithi055 ceph-mon[97033]: pgmap v20833: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:20:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:22 smithi059 ceph-mon[97450]: pgmap v20833: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:20:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:23.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:23.167740+0000) 2022-09-08T08:20:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:22.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:22.989187+0000) 2022-09-08T08:20:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:24.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:24.167961+0000) 2022-09-08T08:20:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:23.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:23.989406+0000) 2022-09-08T08:20:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:24 smithi055 ceph-mon[97033]: pgmap v20834: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:20:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:24 smithi059 ceph-mon[97450]: pgmap v20834: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:20:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:24.988+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:24.989577+0000) 2022-09-08T08:20:25.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:20:25.106+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:20:25.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:25.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:25.168167+0000) 2022-09-08T08:20:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:25.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:25.989740+0000) 2022-09-08T08:20:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:26.167+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:26.168340+0000) 2022-09-08T08:20:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:26 smithi055 ceph-mon[97033]: pgmap v20835: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:20:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:26 smithi059 ceph-mon[97450]: pgmap v20835: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:20:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:27.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:27.168485+0000) 2022-09-08T08:20:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:26.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:26.989875+0000) 2022-09-08T08:20:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:28.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:28.168733+0000) 2022-09-08T08:20:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:27.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:27.990055+0000) 2022-09-08T08:20:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:28 smithi055 ceph-mon[97033]: pgmap v20836: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:20:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:28 smithi059 ceph-mon[97450]: pgmap v20836: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:20:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:29.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:29.168970+0000) 2022-09-08T08:20:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:28.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:28.990214+0000) 2022-09-08T08:20:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:29 smithi055 ceph-mon[97033]: pgmap v20837: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:20:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:29 smithi059 ceph-mon[97450]: pgmap v20837: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:20:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:30.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:30.169133+0000) 2022-09-08T08:20:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:29.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:29.990383+0000) 2022-09-08T08:20:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:31.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:31.169318+0000) 2022-09-08T08:20:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:30.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:30.990545+0000) 2022-09-08T08:20:32.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:31.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:31.990709+0000) 2022-09-08T08:20:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:32.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:32.169524+0000) 2022-09-08T08:20:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:32 smithi055 ceph-mon[97033]: pgmap v20838: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:20:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:32 smithi059 ceph-mon[97450]: pgmap v20838: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:20:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:33.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:33.169731+0000) 2022-09-08T08:20:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:32.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:32.990842+0000) 2022-09-08T08:20:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:34.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:34.169923+0000) 2022-09-08T08:20:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:33.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:33.991049+0000) 2022-09-08T08:20:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:34 smithi055 ceph-mon[97033]: pgmap v20839: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:34 smithi059 ceph-mon[97450]: pgmap v20839: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:35.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:35.170129+0000) 2022-09-08T08:20:35.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:34.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:34.991234+0000) 2022-09-08T08:20:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:36.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:36.170310+0000) 2022-09-08T08:20:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:35.989+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:35.991422+0000) 2022-09-08T08:20:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:36 smithi055 ceph-mon[97033]: pgmap v20840: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:20:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:36 smithi059 ceph-mon[97450]: pgmap v20840: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:20:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:37.168+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:37.170488+0000) 2022-09-08T08:20:37.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:36.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:36.991615+0000) 2022-09-08T08:20:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:38.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:38.170726+0000) 2022-09-08T08:20:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:37.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:37.991799+0000) 2022-09-08T08:20:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:38 smithi055 ceph-mon[97033]: pgmap v20841: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:38 smithi059 ceph-mon[97450]: pgmap v20841: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:39.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:39.170905+0000) 2022-09-08T08:20:39.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:38.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:38.991950+0000) 2022-09-08T08:20:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:39 smithi055 ceph-mon[97033]: pgmap v20842: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:39 smithi059 ceph-mon[97450]: pgmap v20842: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:40.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:20:40.120+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:20:40.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:40.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:40.171132+0000) 2022-09-08T08:20:40.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:39.990+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:39.992100+0000) 2022-09-08T08:20:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:20:41.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:20:41.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:40 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:20:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:20:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:20:41.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:40 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:20:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:41.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:41.171299+0000) 2022-09-08T08:20:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:40.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:40.992320+0000) 2022-09-08T08:20:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:42.169+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:42.171469+0000) 2022-09-08T08:20:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:41.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:41.992541+0000) 2022-09-08T08:20:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:42 smithi055 ceph-mon[97033]: pgmap v20843: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:20:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:20:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:42 smithi059 ceph-mon[97450]: pgmap v20843: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:20:42.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:20:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:43.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:43.171660+0000) 2022-09-08T08:20:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:42.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:42.992779+0000) 2022-09-08T08:20:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:44.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:44.171781+0000) 2022-09-08T08:20:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:43.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:43.993053+0000) 2022-09-08T08:20:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:44 smithi055 ceph-mon[97033]: pgmap v20844: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:44 smithi059 ceph-mon[97450]: pgmap v20844: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:45.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:45.171969+0000) 2022-09-08T08:20:45.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:44.991+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:44.993206+0000) 2022-09-08T08:20:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:46.170+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:46.172132+0000) 2022-09-08T08:20:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:45.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:45.993335+0000) 2022-09-08T08:20:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:46 smithi055 ceph-mon[97033]: pgmap v20845: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:20:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:46 smithi059 ceph-mon[97450]: pgmap v20845: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:20:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:47.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:47.172363+0000) 2022-09-08T08:20:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:46.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:46.993506+0000) 2022-09-08T08:20:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:48.171+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:48.172605+0000) 2022-09-08T08:20:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:47.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:47.993685+0000) 2022-09-08T08:20:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:48 smithi055 ceph-mon[97033]: pgmap v20846: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:48 smithi059 ceph-mon[97450]: pgmap v20846: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:49.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:49.172870+0000) 2022-09-08T08:20:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:48.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:48.993816+0000) 2022-09-08T08:20:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:49 smithi055 ceph-mon[97033]: pgmap v20847: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:49 smithi059 ceph-mon[97450]: pgmap v20847: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:50.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:50.173075+0000) 2022-09-08T08:20:50.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:49.992+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:49.993949+0000) 2022-09-08T08:20:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:51.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:51.173243+0000) 2022-09-08T08:20:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:50.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:50.994123+0000) 2022-09-08T08:20:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:52.172+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:52.173454+0000) 2022-09-08T08:20:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:51.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:51.994284+0000) 2022-09-08T08:20:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:52 smithi055 ceph-mon[97033]: pgmap v20848: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:20:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:52 smithi059 ceph-mon[97450]: pgmap v20848: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:20:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:53.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:53.173725+0000) 2022-09-08T08:20:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:52.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:52.994437+0000) 2022-09-08T08:20:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:54.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:54.173948+0000) 2022-09-08T08:20:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:53.993+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:53.994597+0000) 2022-09-08T08:20:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:54 smithi055 ceph-mon[97033]: pgmap v20849: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:54 smithi059 ceph-mon[97450]: pgmap v20849: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:55.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:20:55.137+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:20:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:55.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:55.174148+0000) 2022-09-08T08:20:55.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:54.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:54.994745+0000) 2022-09-08T08:20:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:56.173+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:56.174377+0000) 2022-09-08T08:20:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:55.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:55.994869+0000) 2022-09-08T08:20:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:56 smithi055 ceph-mon[97033]: pgmap v20850: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:20:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:56 smithi059 ceph-mon[97450]: pgmap v20850: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:20:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:57.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:57.174671+0000) 2022-09-08T08:20:57.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:56.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:56.994999+0000) 2022-09-08T08:20:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:58.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:58.174845+0000) 2022-09-08T08:20:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:57.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:57.995215+0000) 2022-09-08T08:20:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:58 smithi055 ceph-mon[97033]: pgmap v20851: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:58 smithi059 ceph-mon[97450]: pgmap v20851: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:20:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:20:59.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:59.175087+0000) 2022-09-08T08:20:59.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:58.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:58.995479+0000) 2022-09-08T08:20:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:20:59 smithi055 ceph-mon[97033]: pgmap v20852: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:20:59 smithi059 ceph-mon[97450]: pgmap v20852: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:00.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:00.175292+0000) 2022-09-08T08:21:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:20:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:20:59.994+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:20:59.995610+0000) 2022-09-08T08:21:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:00.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:00.995742+0000) 2022-09-08T08:21:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:01.174+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:01.175464+0000) 2022-09-08T08:21:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:02.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:02.175708+0000) 2022-09-08T08:21:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:01.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:01.995892+0000) 2022-09-08T08:21:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:02 smithi055 ceph-mon[97033]: pgmap v20853: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:21:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:02 smithi059 ceph-mon[97450]: pgmap v20853: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:21:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:03.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:03.175953+0000) 2022-09-08T08:21:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:02.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:02.996057+0000) 2022-09-08T08:21:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:04.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:04.176165+0000) 2022-09-08T08:21:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:03.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:03.996211+0000) 2022-09-08T08:21:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:04 smithi055 ceph-mon[97033]: pgmap v20854: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:04 smithi059 ceph-mon[97450]: pgmap v20854: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:05.175+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:05.176389+0000) 2022-09-08T08:21:05.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:04.995+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:04.996384+0000) 2022-09-08T08:21:06.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:06.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:06.176630+0000) 2022-09-08T08:21:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:05.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:05.996609+0000) 2022-09-08T08:21:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:06 smithi055 ceph-mon[97033]: pgmap v20855: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:21:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:21:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:06 smithi059 ceph-mon[97450]: pgmap v20855: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:21:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:21:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:07.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:07.176915+0000) 2022-09-08T08:21:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:06.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:06.996789+0000) 2022-09-08T08:21:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:21:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:21:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:08.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:08.177205+0000) 2022-09-08T08:21:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:07.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:07.996994+0000) 2022-09-08T08:21:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:08 smithi055 ceph-mon[97033]: pgmap v20856: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:21:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:08 smithi059 ceph-mon[97450]: pgmap v20856: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:21:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:09.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:09.177360+0000) 2022-09-08T08:21:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:08.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:08.997126+0000) 2022-09-08T08:21:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:09 smithi055 ceph-mon[97033]: pgmap v20857: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:21:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:09 smithi059 ceph-mon[97450]: pgmap v20857: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:21:10.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:21:10.152+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:21:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:10.176+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:10.177570+0000) 2022-09-08T08:21:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:09.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:09.997268+0000) 2022-09-08T08:21:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:10 smithi055 ceph-mon[97033]: osdmap e1382: 12 total, 11 up, 11 in 2022-09-08T08:21:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:10 smithi059 ceph-mon[97450]: osdmap e1382: 12 total, 11 up, 11 in 2022-09-08T08:21:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:11.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:11.177729+0000) 2022-09-08T08:21:11.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:10.996+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:10.997448+0000) 2022-09-08T08:21:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:11 smithi055 ceph-mon[97033]: pgmap v20859: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:21:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:11 smithi059 ceph-mon[97450]: pgmap v20859: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:21:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:12.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:12.177877+0000) 2022-09-08T08:21:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:11.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:11.997732+0000) 2022-09-08T08:21:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:13.177+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:13.178096+0000) 2022-09-08T08:21:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:12.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:12.997918+0000) 2022-09-08T08:21:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:14.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:14.178355+0000) 2022-09-08T08:21:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:13.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:13.998062+0000) 2022-09-08T08:21:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:14 smithi055 ceph-mon[97033]: pgmap v20860: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:21:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:14 smithi059 ceph-mon[97450]: pgmap v20860: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:21:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:15.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:15.178590+0000) 2022-09-08T08:21:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:14.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:14.998270+0000) 2022-09-08T08:21:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:15.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:15.998461+0000) 2022-09-08T08:21:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:16.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:16.178810+0000) 2022-09-08T08:21:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:16 smithi055 ceph-mon[97033]: pgmap v20861: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:21:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:16 smithi059 ceph-mon[97450]: pgmap v20861: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:21:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:17.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:17.179043+0000) 2022-09-08T08:21:17.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:16.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:16.998620+0000) 2022-09-08T08:21:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:17.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:17.998794+0000) 2022-09-08T08:21:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:18.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:18.179241+0000) 2022-09-08T08:21:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:18 smithi055 ceph-mon[97033]: pgmap v20862: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:21:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:18 smithi059 ceph-mon[97450]: pgmap v20862: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:21:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:19.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:19.179466+0000) 2022-09-08T08:21:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:18.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:18.998935+0000) 2022-09-08T08:21:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:19 smithi055 ceph-mon[97033]: osdmap e1383: 12 total, 11 up, 11 in 2022-09-08T08:21:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:19 smithi055 ceph-mon[97033]: pgmap v20864: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T08:21:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:19 smithi059 ceph-mon[97450]: osdmap e1383: 12 total, 11 up, 11 in 2022-09-08T08:21:20.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:19 smithi059 ceph-mon[97450]: pgmap v20864: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T08:21:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:20.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:20.179676+0000) 2022-09-08T08:21:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:19.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:19.999092+0000) 2022-09-08T08:21:21.407 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:21.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:21.179898+0000) 2022-09-08T08:21:21.408 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:20.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:20.999305+0000) 2022-09-08T08:21:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:22.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:22.180038+0000) 2022-09-08T08:21:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:21.997+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:21.999491+0000) 2022-09-08T08:21:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:22 smithi055 ceph-mon[97033]: pgmap v20865: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:21:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:22 smithi059 ceph-mon[97450]: pgmap v20865: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:21:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:23.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:23.180278+0000) 2022-09-08T08:21:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:22.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:22.999671+0000) 2022-09-08T08:21:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:24.178+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:24.180439+0000) 2022-09-08T08:21:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:23.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:23.999842+0000) 2022-09-08T08:21:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:24 smithi055 ceph-mon[97033]: pgmap v20866: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:21:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:24 smithi059 ceph-mon[97450]: pgmap v20866: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:21:25.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:21:25.162+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:21:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:25.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:25.180675+0000) 2022-09-08T08:21:25.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:24.998+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:25.000072+0000) 2022-09-08T08:21:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:26.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:26.180848+0000) 2022-09-08T08:21:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:25.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:26.000230+0000) 2022-09-08T08:21:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:26 smithi055 ceph-mon[97033]: pgmap v20867: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:21:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:26 smithi059 ceph-mon[97450]: pgmap v20867: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:21:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:27.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:27.181071+0000) 2022-09-08T08:21:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:26.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:27.000423+0000) 2022-09-08T08:21:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:27.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:28.000624+0000) 2022-09-08T08:21:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:28.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:28.181299+0000) 2022-09-08T08:21:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:28 smithi055 ceph-mon[97033]: pgmap v20868: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:21:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:28 smithi059 ceph-mon[97450]: pgmap v20868: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:21:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:29.179+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:29.181534+0000) 2022-09-08T08:21:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:28.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:29.000876+0000) 2022-09-08T08:21:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:29 smithi055 ceph-mon[97033]: pgmap v20869: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:21:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:29 smithi059 ceph-mon[97450]: pgmap v20869: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:21:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:30.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:30.181792+0000) 2022-09-08T08:21:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:29.999+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:30.001068+0000) 2022-09-08T08:21:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:31.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:31.182009+0000) 2022-09-08T08:21:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:31.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:31.001258+0000) 2022-09-08T08:21:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:32.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:32.182225+0000) 2022-09-08T08:21:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:32.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:32.001448+0000) 2022-09-08T08:21:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:32 smithi055 ceph-mon[97033]: pgmap v20870: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:21:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:32 smithi059 ceph-mon[97450]: pgmap v20870: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:21:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:33.180+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:33.182424+0000) 2022-09-08T08:21:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:33.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:33.001610+0000) 2022-09-08T08:21:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:34.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:34.182668+0000) 2022-09-08T08:21:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:34.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:34.001709+0000) 2022-09-08T08:21:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:34 smithi055 ceph-mon[97033]: pgmap v20871: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:34 smithi059 ceph-mon[97450]: pgmap v20871: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:35.181+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:35.182874+0000) 2022-09-08T08:21:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:35.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:35.001921+0000) 2022-09-08T08:21:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:36.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:36.183062+0000) 2022-09-08T08:21:36.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:36.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:36.002093+0000) 2022-09-08T08:21:36.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:36 smithi055 ceph-mon[97033]: pgmap v20872: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:21:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:36 smithi059 ceph-mon[97450]: pgmap v20872: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:21:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:37.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:37.183245+0000) 2022-09-08T08:21:37.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:37.000+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:37.002315+0000) 2022-09-08T08:21:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:38.182+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:38.183488+0000) 2022-09-08T08:21:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:38.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:38.002459+0000) 2022-09-08T08:21:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:38 smithi055 ceph-mon[97033]: pgmap v20873: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:38 smithi059 ceph-mon[97450]: pgmap v20873: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:39.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:39.183737+0000) 2022-09-08T08:21:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:39.001+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:39.002593+0000) 2022-09-08T08:21:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:39 smithi055 ceph-mon[97033]: pgmap v20874: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:39 smithi059 ceph-mon[97450]: pgmap v20874: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:40.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:21:40.178+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:21:40.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:40.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:40.183928+0000) 2022-09-08T08:21:40.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:40.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:40.002833+0000) 2022-09-08T08:21:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:41.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:41.003043+0000) 2022-09-08T08:21:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:41.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:41.184106+0000) 2022-09-08T08:21:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:42.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:42.184335+0000) 2022-09-08T08:21:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:42.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:42.003240+0000) 2022-09-08T08:21:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:42 smithi055 ceph-mon[97033]: pgmap v20875: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:21:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:21:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:21:42.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:21:42.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:21:42.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:42 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:21:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:42 smithi059 ceph-mon[97450]: pgmap v20875: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:21:42.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:21:42.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:21:42.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:21:42.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:21:42.785 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:42 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:21:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:43.183+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:43.184579+0000) 2022-09-08T08:21:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:43.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:43.003393+0000) 2022-09-08T08:21:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:21:43.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:43 smithi055 ceph-mon[97033]: pgmap v20876: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:21:43.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:43 smithi059 ceph-mon[97450]: pgmap v20876: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:44.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:44.184749+0000) 2022-09-08T08:21:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:44.002+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:44.003519+0000) 2022-09-08T08:21:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:45.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:45.184961+0000) 2022-09-08T08:21:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:45.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:45.003743+0000) 2022-09-08T08:21:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:46.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:46.003931+0000) 2022-09-08T08:21:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:46.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:46.185195+0000) 2022-09-08T08:21:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:46 smithi055 ceph-mon[97033]: pgmap v20877: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:21:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:46 smithi059 ceph-mon[97450]: pgmap v20877: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:21:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:47.184+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:47.185396+0000) 2022-09-08T08:21:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:47.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:47.004061+0000) 2022-09-08T08:21:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:48.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:48.185659+0000) 2022-09-08T08:21:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:48.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:48.004233+0000) 2022-09-08T08:21:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:48 smithi055 ceph-mon[97033]: pgmap v20878: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:48 smithi059 ceph-mon[97450]: pgmap v20878: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:49.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:49.185815+0000) 2022-09-08T08:21:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:49.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:49.004412+0000) 2022-09-08T08:21:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:49 smithi055 ceph-mon[97033]: pgmap v20879: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:49 smithi059 ceph-mon[97450]: pgmap v20879: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:50.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:50.185961+0000) 2022-09-08T08:21:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:50.003+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:50.004587+0000) 2022-09-08T08:21:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:51.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:51.186153+0000) 2022-09-08T08:21:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:51.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:51.004722+0000) 2022-09-08T08:21:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:52.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:52.186311+0000) 2022-09-08T08:21:52.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:52.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:52.004924+0000) 2022-09-08T08:21:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:52 smithi055 ceph-mon[97033]: pgmap v20880: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:21:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:52 smithi059 ceph-mon[97450]: pgmap v20880: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:21:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:53.185+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:53.186523+0000) 2022-09-08T08:21:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:53.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:53.005131+0000) 2022-09-08T08:21:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:54.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:54.186764+0000) 2022-09-08T08:21:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:54.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:54.005283+0000) 2022-09-08T08:21:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:54 smithi055 ceph-mon[97033]: pgmap v20881: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:54 smithi059 ceph-mon[97450]: pgmap v20881: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:55.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:21:55.185+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:21:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:55.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:55.186987+0000) 2022-09-08T08:21:55.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:55.004+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:55.005510+0000) 2022-09-08T08:21:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:56.186+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:56.187241+0000) 2022-09-08T08:21:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:56.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:56.005754+0000) 2022-09-08T08:21:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:56 smithi055 ceph-mon[97033]: pgmap v20882: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:21:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:56 smithi059 ceph-mon[97450]: pgmap v20882: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:21:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:57.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:57.187476+0000) 2022-09-08T08:21:57.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:57.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:57.005921+0000) 2022-09-08T08:21:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:58.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:58.187757+0000) 2022-09-08T08:21:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:58.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:58.006161+0000) 2022-09-08T08:21:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:58 smithi055 ceph-mon[97033]: pgmap v20883: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:58 smithi059 ceph-mon[97450]: pgmap v20883: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:21:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:21:59.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:59.187998+0000) 2022-09-08T08:21:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:21:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:21:59.005+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:21:59.006420+0000) 2022-09-08T08:21:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:21:59 smithi055 ceph-mon[97033]: pgmap v20884: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:21:59 smithi059 ceph-mon[97450]: pgmap v20884: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:00.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:00.188197+0000) 2022-09-08T08:22:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:00.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:00.006632+0000) 2022-09-08T08:22:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:01.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:01.188402+0000) 2022-09-08T08:22:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:01.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:01.006793+0000) 2022-09-08T08:22:02.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:02.187+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:02.188593+0000) 2022-09-08T08:22:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:02.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:02.006935+0000) 2022-09-08T08:22:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:02 smithi055 ceph-mon[97033]: pgmap v20885: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:22:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:02 smithi059 ceph-mon[97450]: pgmap v20885: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:22:03.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:03.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:03.188769+0000) 2022-09-08T08:22:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:03.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:03.007129+0000) 2022-09-08T08:22:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:04.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:04.007359+0000) 2022-09-08T08:22:04.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:04.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:04.188934+0000) 2022-09-08T08:22:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:04 smithi055 ceph-mon[97033]: pgmap v20886: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:04 smithi059 ceph-mon[97450]: pgmap v20886: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:05.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:05.189211+0000) 2022-09-08T08:22:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:05.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:05.007560+0000) 2022-09-08T08:22:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:06.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:06.189504+0000) 2022-09-08T08:22:06.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:06.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:06.007692+0000) 2022-09-08T08:22:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:06 smithi055 ceph-mon[97033]: pgmap v20887: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:22:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:22:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:06 smithi059 ceph-mon[97450]: pgmap v20887: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:22:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:22:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:07.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:07.189722+0000) 2022-09-08T08:22:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:07.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:07.007889+0000) 2022-09-08T08:22:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:22:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:22:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:08.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:08.189865+0000) 2022-09-08T08:22:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:08.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:08.008107+0000) 2022-09-08T08:22:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:08 smithi055 ceph-mon[97033]: pgmap v20888: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:22:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:08 smithi059 ceph-mon[97450]: pgmap v20888: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:22:09.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:09.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:09.008337+0000) 2022-09-08T08:22:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:09.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:09.190096+0000) 2022-09-08T08:22:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:09 smithi055 ceph-mon[97033]: pgmap v20889: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:22:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:09 smithi059 ceph-mon[97450]: pgmap v20889: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:22:10.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:22:10.200+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:22:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:10.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:10.190318+0000) 2022-09-08T08:22:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:10.006+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:10.008507+0000) 2022-09-08T08:22:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:10 smithi055 ceph-mon[97033]: osdmap e1384: 12 total, 11 up, 11 in 2022-09-08T08:22:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:10 smithi059 ceph-mon[97450]: osdmap e1384: 12 total, 11 up, 11 in 2022-09-08T08:22:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:11.188+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:11.190538+0000) 2022-09-08T08:22:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:11.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:11.008664+0000) 2022-09-08T08:22:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:11 smithi055 ceph-mon[97033]: pgmap v20891: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:22:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:11 smithi059 ceph-mon[97450]: pgmap v20891: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:22:12.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:12.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:12.190768+0000) 2022-09-08T08:22:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:12.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:12.008913+0000) 2022-09-08T08:22:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:13.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:13.191008+0000) 2022-09-08T08:22:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:13.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:13.009061+0000) 2022-09-08T08:22:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:14.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:14.191242+0000) 2022-09-08T08:22:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:14.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:14.009205+0000) 2022-09-08T08:22:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:14 smithi055 ceph-mon[97033]: pgmap v20892: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:22:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:14 smithi055 ceph-mon[97033]: osdmap e1385: 12 total, 11 up, 11 in 2022-09-08T08:22:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:14 smithi059 ceph-mon[97450]: pgmap v20892: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:22:14.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:14 smithi059 ceph-mon[97450]: osdmap e1385: 12 total, 11 up, 11 in 2022-09-08T08:22:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:15.189+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:15.191437+0000) 2022-09-08T08:22:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:15.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:15.009343+0000) 2022-09-08T08:22:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:16.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:16.191619+0000) 2022-09-08T08:22:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:16.007+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:16.009537+0000) 2022-09-08T08:22:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:16 smithi055 ceph-mon[97033]: pgmap v20894: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2022-09-08T08:22:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:16 smithi059 ceph-mon[97450]: pgmap v20894: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2022-09-08T08:22:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:17.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:17.191784+0000) 2022-09-08T08:22:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:17.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:17.009775+0000) 2022-09-08T08:22:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:18.190+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:18.191963+0000) 2022-09-08T08:22:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:18.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:18.009948+0000) 2022-09-08T08:22:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:18 smithi055 ceph-mon[97033]: pgmap v20895: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2022-09-08T08:22:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:18 smithi059 ceph-mon[97450]: pgmap v20895: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2022-09-08T08:22:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:19.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:19.192182+0000) 2022-09-08T08:22:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:19.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:19.010077+0000) 2022-09-08T08:22:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:19 smithi055 ceph-mon[97033]: pgmap v20896: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.4 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T08:22:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:19 smithi059 ceph-mon[97450]: pgmap v20896: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.4 KiB/s rd, 212 B/s wr, 6 op/s 2022-09-08T08:22:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:20.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:20.192429+0000) 2022-09-08T08:22:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:20.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:20.010221+0000) 2022-09-08T08:22:21.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:21.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:21.010368+0000) 2022-09-08T08:22:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:21.191+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:21.192677+0000) 2022-09-08T08:22:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:22.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:22.192862+0000) 2022-09-08T08:22:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:22.008+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:22.010523+0000) 2022-09-08T08:22:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:22 smithi055 ceph-mon[97033]: pgmap v20897: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:22:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:22 smithi059 ceph-mon[97450]: pgmap v20897: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:22:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:23.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:23.193042+0000) 2022-09-08T08:22:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:23.009+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:23.010768+0000) 2022-09-08T08:22:24.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:24.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:24.193247+0000) 2022-09-08T08:22:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:24.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:24.010925+0000) 2022-09-08T08:22:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:24 smithi055 ceph-mon[97033]: pgmap v20898: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:22:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:24 smithi059 ceph-mon[97450]: pgmap v20898: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:22:25.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:22:25.211+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:22:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:25.192+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:25.193509+0000) 2022-09-08T08:22:25.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:25.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:25.011145+0000) 2022-09-08T08:22:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:26.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:26.011303+0000) 2022-09-08T08:22:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:26.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:26.193789+0000) 2022-09-08T08:22:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:26 smithi055 ceph-mon[97033]: pgmap v20899: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-08T08:22:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:26 smithi059 ceph-mon[97450]: pgmap v20899: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-08T08:22:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:27.010+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:27.011531+0000) 2022-09-08T08:22:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:27.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:27.194027+0000) 2022-09-08T08:22:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:28.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:28.194237+0000) 2022-09-08T08:22:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:28.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:28.011793+0000) 2022-09-08T08:22:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:28 smithi055 ceph-mon[97033]: pgmap v20900: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:28 smithi059 ceph-mon[97450]: pgmap v20900: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:29.193+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:29.194459+0000) 2022-09-08T08:22:29.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:29.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:29.011960+0000) 2022-09-08T08:22:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:29 smithi055 ceph-mon[97033]: pgmap v20901: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:29 smithi059 ceph-mon[97450]: pgmap v20901: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:30.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:30.194721+0000) 2022-09-08T08:22:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:30.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:30.012167+0000) 2022-09-08T08:22:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:31.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:31.194856+0000) 2022-09-08T08:22:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:31.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:31.012321+0000) 2022-09-08T08:22:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:32.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:32.195026+0000) 2022-09-08T08:22:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:32.011+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:32.012576+0000) 2022-09-08T08:22:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:32 smithi055 ceph-mon[97033]: pgmap v20902: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:22:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:32 smithi059 ceph-mon[97450]: pgmap v20902: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:22:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:33.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:33.195162+0000) 2022-09-08T08:22:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:33.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:33.012750+0000) 2022-09-08T08:22:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:34.194+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:34.195315+0000) 2022-09-08T08:22:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:34.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:34.012949+0000) 2022-09-08T08:22:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:34 smithi055 ceph-mon[97033]: pgmap v20903: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:34 smithi059 ceph-mon[97450]: pgmap v20903: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:35.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:35.195542+0000) 2022-09-08T08:22:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:35.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:35.013206+0000) 2022-09-08T08:22:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:36.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:36.195749+0000) 2022-09-08T08:22:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:36.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:36.013375+0000) 2022-09-08T08:22:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:36 smithi055 ceph-mon[97033]: pgmap v20904: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:22:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:36 smithi059 ceph-mon[97450]: pgmap v20904: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:22:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:37.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:37.195995+0000) 2022-09-08T08:22:37.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:37.012+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:37.013608+0000) 2022-09-08T08:22:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:38.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:38.196175+0000) 2022-09-08T08:22:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:38.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:38.013828+0000) 2022-09-08T08:22:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:38 smithi055 ceph-mon[97033]: pgmap v20905: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:38 smithi059 ceph-mon[97450]: pgmap v20905: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:39.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:39.195+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:39.196398+0000) 2022-09-08T08:22:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:39.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:39.014004+0000) 2022-09-08T08:22:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:39 smithi055 ceph-mon[97033]: pgmap v20906: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:39 smithi059 ceph-mon[97450]: pgmap v20906: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:40.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:40.014150+0000) 2022-09-08T08:22:40.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:22:40.227+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:22:40.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:40.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:40.196650+0000) 2022-09-08T08:22:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:41.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:41.196770+0000) 2022-09-08T08:22:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:41.013+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:41.014381+0000) 2022-09-08T08:22:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:42.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:42.196954+0000) 2022-09-08T08:22:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:42.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:42.014632+0000) 2022-09-08T08:22:42.627 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:42 smithi059 ceph-mon[97450]: pgmap v20907: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:22:42.637 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:42 smithi055 ceph-mon[97033]: pgmap v20907: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:22:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:43.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:43.197190+0000) 2022-09-08T08:22:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:43.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:43.014914+0000) 2022-09-08T08:22:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:22:43.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:22:43.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:22:43.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:22:43.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:22:43.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:22:43.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:22:43.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:22:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:44.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:44.197425+0000) 2022-09-08T08:22:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:44.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:44.015125+0000) 2022-09-08T08:22:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:44 smithi055 ceph-mon[97033]: pgmap v20908: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:44 smithi059 ceph-mon[97450]: pgmap v20908: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:45.196+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:45.197608+0000) 2022-09-08T08:22:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:45.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:45.015326+0000) 2022-09-08T08:22:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:46.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:46.197865+0000) 2022-09-08T08:22:46.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:46.014+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:46.015560+0000) 2022-09-08T08:22:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:46 smithi055 ceph-mon[97033]: pgmap v20909: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:22:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:46 smithi059 ceph-mon[97450]: pgmap v20909: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:22:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:47.197+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:47.198065+0000) 2022-09-08T08:22:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:47.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:47.015736+0000) 2022-09-08T08:22:48.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:48.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:48.198313+0000) 2022-09-08T08:22:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:48.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:48.015976+0000) 2022-09-08T08:22:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:48 smithi055 ceph-mon[97033]: pgmap v20910: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:48 smithi059 ceph-mon[97450]: pgmap v20910: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:49.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:49.198531+0000) 2022-09-08T08:22:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:49.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:49.016143+0000) 2022-09-08T08:22:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:49 smithi055 ceph-mon[97033]: pgmap v20911: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:49 smithi059 ceph-mon[97450]: pgmap v20911: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:50.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:50.198774+0000) 2022-09-08T08:22:50.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:50.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:50.016386+0000) 2022-09-08T08:22:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:51.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:51.016543+0000) 2022-09-08T08:22:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:51.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:51.198925+0000) 2022-09-08T08:22:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:52.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:52.199152+0000) 2022-09-08T08:22:52.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:52.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:52.016797+0000) 2022-09-08T08:22:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:52 smithi055 ceph-mon[97033]: pgmap v20912: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:22:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:52 smithi059 ceph-mon[97450]: pgmap v20912: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:22:53.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:53.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:53.199389+0000) 2022-09-08T08:22:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:53.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:53.016951+0000) 2022-09-08T08:22:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:54.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:54.199612+0000) 2022-09-08T08:22:54.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:54.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:54.017102+0000) 2022-09-08T08:22:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:54 smithi055 ceph-mon[97033]: pgmap v20913: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:54 smithi059 ceph-mon[97450]: pgmap v20913: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:55.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:22:55.241+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:22:55.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:55.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:55.199846+0000) 2022-09-08T08:22:55.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:55.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:55.017331+0000) 2022-09-08T08:22:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:56.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:56.199977+0000) 2022-09-08T08:22:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:56.015+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:56.017539+0000) 2022-09-08T08:22:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:56 smithi055 ceph-mon[97033]: pgmap v20914: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:22:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:56 smithi059 ceph-mon[97450]: pgmap v20914: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:22:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:57.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:57.200148+0000) 2022-09-08T08:22:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:57.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:57.017765+0000) 2022-09-08T08:22:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:58.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:58.200322+0000) 2022-09-08T08:22:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:58.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:58.018003+0000) 2022-09-08T08:22:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:58 smithi055 ceph-mon[97033]: pgmap v20915: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:58 smithi059 ceph-mon[97450]: pgmap v20915: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:22:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:22:59.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:59.018145+0000) 2022-09-08T08:22:59.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:22:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:22:59.198+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:22:59.200543+0000) 2022-09-08T08:22:59.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:22:59 smithi055 ceph-mon[97033]: pgmap v20916: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:22:59 smithi059 ceph-mon[97450]: pgmap v20916: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:00.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:00.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:00.018392+0000) 2022-09-08T08:23:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:00.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:00.200753+0000) 2022-09-08T08:23:01.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:01.016+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:01.018572+0000) 2022-09-08T08:23:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:01.199+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:01.200935+0000) 2022-09-08T08:23:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:02.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:02.201136+0000) 2022-09-08T08:23:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:02.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:02.018798+0000) 2022-09-08T08:23:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:02 smithi055 ceph-mon[97033]: pgmap v20917: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:23:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:02 smithi059 ceph-mon[97450]: pgmap v20917: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:23:03.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:03.017+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:03.018977+0000) 2022-09-08T08:23:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:03.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:03.201357+0000) 2022-09-08T08:23:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:04.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:04.201592+0000) 2022-09-08T08:23:04.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:04.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:04.019161+0000) 2022-09-08T08:23:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:04 smithi055 ceph-mon[97033]: pgmap v20918: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:04 smithi059 ceph-mon[97450]: pgmap v20918: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:05.200+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:05.201850+0000) 2022-09-08T08:23:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:05.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:05.019408+0000) 2022-09-08T08:23:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:06.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:06.019614+0000) 2022-09-08T08:23:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:06.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:06.202133+0000) 2022-09-08T08:23:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:06 smithi055 ceph-mon[97033]: pgmap v20919: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:23:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:23:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:06 smithi059 ceph-mon[97450]: pgmap v20919: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:23:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:23:07.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:07.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:07.019867+0000) 2022-09-08T08:23:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:07.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:07.202353+0000) 2022-09-08T08:23:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:23:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:23:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:08.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:08.020047+0000) 2022-09-08T08:23:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:08.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:08.202473+0000) 2022-09-08T08:23:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:08 smithi055 ceph-mon[97033]: pgmap v20920: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:23:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:08 smithi059 ceph-mon[97450]: pgmap v20920: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:23:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:09.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:09.020182+0000) 2022-09-08T08:23:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:09.201+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:09.202728+0000) 2022-09-08T08:23:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:09 smithi055 ceph-mon[97033]: pgmap v20921: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:23:10.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:09 smithi059 ceph-mon[97450]: pgmap v20921: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:23:10.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:23:10.257+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:23:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:10.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:10.202952+0000) 2022-09-08T08:23:10.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:10.019+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:10.020332+0000) 2022-09-08T08:23:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:10 smithi055 ceph-mon[97033]: osdmap e1386: 12 total, 11 up, 11 in 2022-09-08T08:23:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:10 smithi059 ceph-mon[97450]: osdmap e1386: 12 total, 11 up, 11 in 2022-09-08T08:23:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:11.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:11.203151+0000) 2022-09-08T08:23:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:11.018+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:11.020576+0000) 2022-09-08T08:23:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:11 smithi055 ceph-mon[97033]: pgmap v20923: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:23:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:11 smithi059 ceph-mon[97450]: pgmap v20923: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:23:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:12.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:12.203348+0000) 2022-09-08T08:23:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:12.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:12.020762+0000) 2022-09-08T08:23:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:13.202+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:13.203560+0000) 2022-09-08T08:23:13.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:13.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:13.020919+0000) 2022-09-08T08:23:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:14.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:14.203805+0000) 2022-09-08T08:23:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:14.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:14.021166+0000) 2022-09-08T08:23:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:14 smithi055 ceph-mon[97033]: pgmap v20924: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:23:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:14 smithi059 ceph-mon[97450]: pgmap v20924: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:23:15.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:15.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:15.204045+0000) 2022-09-08T08:23:15.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:15.020+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:15.021401+0000) 2022-09-08T08:23:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:16.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:16.204235+0000) 2022-09-08T08:23:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:16.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:16.021696+0000) 2022-09-08T08:23:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:16 smithi055 ceph-mon[97033]: pgmap v20925: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:23:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:16 smithi059 ceph-mon[97450]: pgmap v20925: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:23:17.206 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:17.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:17.021870+0000) 2022-09-08T08:23:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:17.203+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:17.204413+0000) 2022-09-08T08:23:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:18.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:18.204626+0000) 2022-09-08T08:23:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:18.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:18.022032+0000) 2022-09-08T08:23:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:18 smithi055 ceph-mon[97033]: pgmap v20926: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:23:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:18 smithi059 ceph-mon[97450]: pgmap v20926: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:23:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:19.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:19.204864+0000) 2022-09-08T08:23:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:19.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:19.022232+0000) 2022-09-08T08:23:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:19 smithi055 ceph-mon[97033]: osdmap e1387: 12 total, 11 up, 11 in 2022-09-08T08:23:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:19 smithi055 ceph-mon[97033]: pgmap v20928: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 211 B/s wr, 6 op/s 2022-09-08T08:23:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:19 smithi059 ceph-mon[97450]: osdmap e1387: 12 total, 11 up, 11 in 2022-09-08T08:23:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:19 smithi059 ceph-mon[97450]: pgmap v20928: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 211 B/s wr, 6 op/s 2022-09-08T08:23:20.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:20.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:20.205055+0000) 2022-09-08T08:23:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:20.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:20.022455+0000) 2022-09-08T08:23:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:21.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:21.205233+0000) 2022-09-08T08:23:21.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:21.021+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:21.022614+0000) 2022-09-08T08:23:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:22.204+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:22.205425+0000) 2022-09-08T08:23:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:22.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:22.022811+0000) 2022-09-08T08:23:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:22 smithi055 ceph-mon[97033]: pgmap v20929: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:23:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:22 smithi059 ceph-mon[97450]: pgmap v20929: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:23:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:23.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:23.205666+0000) 2022-09-08T08:23:23.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:23.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:23.023049+0000) 2022-09-08T08:23:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:24.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:24.205903+0000) 2022-09-08T08:23:24.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:24.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:24.023189+0000) 2022-09-08T08:23:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:24 smithi055 ceph-mon[97033]: pgmap v20930: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:23:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:24 smithi059 ceph-mon[97450]: pgmap v20930: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:23:25.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:23:25.258+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:23:25.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:25.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:25.206112+0000) 2022-09-08T08:23:25.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:25.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:25.023349+0000) 2022-09-08T08:23:26.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:26.205+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:26.206316+0000) 2022-09-08T08:23:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:26.022+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:26.023496+0000) 2022-09-08T08:23:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:26 smithi055 ceph-mon[97033]: pgmap v20931: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:23:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:26 smithi059 ceph-mon[97450]: pgmap v20931: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:23:27.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:27.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:27.206543+0000) 2022-09-08T08:23:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:27.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:27.023754+0000) 2022-09-08T08:23:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:28.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:28.206779+0000) 2022-09-08T08:23:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:28.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:28.023987+0000) 2022-09-08T08:23:28.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:28 smithi055 ceph-mon[97033]: pgmap v20932: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:23:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:28 smithi059 ceph-mon[97450]: pgmap v20932: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:23:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:29.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:29.206937+0000) 2022-09-08T08:23:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:29.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:29.024151+0000) 2022-09-08T08:23:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:29 smithi055 ceph-mon[97033]: pgmap v20933: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T08:23:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:29 smithi059 ceph-mon[97450]: pgmap v20933: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.7 KiB/s rd, 5 op/s 2022-09-08T08:23:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:30.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:30.207113+0000) 2022-09-08T08:23:30.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:30.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:30.024287+0000) 2022-09-08T08:23:31.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:31.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:31.207234+0000) 2022-09-08T08:23:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:31.023+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:31.024445+0000) 2022-09-08T08:23:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:32.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:32.207380+0000) 2022-09-08T08:23:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:32.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:32.024738+0000) 2022-09-08T08:23:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:32 smithi055 ceph-mon[97033]: pgmap v20934: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:23:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:32 smithi059 ceph-mon[97450]: pgmap v20934: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:23:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:33.206+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:33.207598+0000) 2022-09-08T08:23:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:33.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:33.024934+0000) 2022-09-08T08:23:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:34.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:34.207782+0000) 2022-09-08T08:23:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:34.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:34.025065+0000) 2022-09-08T08:23:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:34 smithi055 ceph-mon[97033]: pgmap v20935: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:23:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:34 smithi059 ceph-mon[97450]: pgmap v20935: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:23:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:35.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:35.025292+0000) 2022-09-08T08:23:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:35.207+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:35.208061+0000) 2022-09-08T08:23:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:36.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:36.208328+0000) 2022-09-08T08:23:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:36.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:36.025487+0000) 2022-09-08T08:23:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:36 smithi055 ceph-mon[97033]: pgmap v20936: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:23:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:36 smithi059 ceph-mon[97450]: pgmap v20936: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:23:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:37.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:37.208589+0000) 2022-09-08T08:23:37.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:37.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:37.025675+0000) 2022-09-08T08:23:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:38.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:38.208860+0000) 2022-09-08T08:23:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:38.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:38.025842+0000) 2022-09-08T08:23:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:38 smithi055 ceph-mon[97033]: pgmap v20937: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:38 smithi059 ceph-mon[97450]: pgmap v20937: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:39.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:39.208994+0000) 2022-09-08T08:23:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:39.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:39.025995+0000) 2022-09-08T08:23:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:39 smithi055 ceph-mon[97033]: pgmap v20938: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:39 smithi059 ceph-mon[97450]: pgmap v20938: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:40.403 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:23:40.274+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:23:40.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:40.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:40.209149+0000) 2022-09-08T08:23:40.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:40.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:40.026215+0000) 2022-09-08T08:23:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:41.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:41.209364+0000) 2022-09-08T08:23:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:41.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:41.026435+0000) 2022-09-08T08:23:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:42.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:42.209612+0000) 2022-09-08T08:23:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:42.024+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:42.026594+0000) 2022-09-08T08:23:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:42 smithi055 ceph-mon[97033]: pgmap v20939: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:23:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:42 smithi059 ceph-mon[97450]: pgmap v20939: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:23:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:43.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:43.209789+0000) 2022-09-08T08:23:43.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:43.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:43.026754+0000) 2022-09-08T08:23:43.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:23:43.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:23:43.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:43 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:23:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:23:43.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:23:43.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:43 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:23:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:44.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:44.209947+0000) 2022-09-08T08:23:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:44.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:44.026918+0000) 2022-09-08T08:23:44.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:44 smithi055 ceph-mon[97033]: pgmap v20940: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:44.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:23:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:44 smithi059 ceph-mon[97450]: pgmap v20940: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:44.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:23:45.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:45.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:45.027138+0000) 2022-09-08T08:23:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:45.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:45.210159+0000) 2022-09-08T08:23:46.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:46.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:46.027295+0000) 2022-09-08T08:23:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:46.208+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:46.210396+0000) 2022-09-08T08:23:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:46 smithi055 ceph-mon[97033]: pgmap v20941: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:23:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:46 smithi059 ceph-mon[97450]: pgmap v20941: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:23:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:47.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:47.210688+0000) 2022-09-08T08:23:47.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:47.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:47.027464+0000) 2022-09-08T08:23:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:48.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:48.210908+0000) 2022-09-08T08:23:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:48.025+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:48.027602+0000) 2022-09-08T08:23:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:48 smithi055 ceph-mon[97033]: pgmap v20942: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:48 smithi059 ceph-mon[97450]: pgmap v20942: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:49.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:49.209+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:49.211053+0000) 2022-09-08T08:23:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:49.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:49.027755+0000) 2022-09-08T08:23:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:49 smithi055 ceph-mon[97033]: pgmap v20943: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:49 smithi059 ceph-mon[97450]: pgmap v20943: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:50.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:50.211230+0000) 2022-09-08T08:23:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:50.026+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:50.027917+0000) 2022-09-08T08:23:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:51.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:51.211465+0000) 2022-09-08T08:23:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:51.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:51.028052+0000) 2022-09-08T08:23:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:52.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:52.211668+0000) 2022-09-08T08:23:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:52.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:52.028295+0000) 2022-09-08T08:23:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:52 smithi055 ceph-mon[97033]: pgmap v20944: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:23:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:52 smithi059 ceph-mon[97450]: pgmap v20944: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:23:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:53.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:53.211893+0000) 2022-09-08T08:23:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:53.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:53.028430+0000) 2022-09-08T08:23:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:54.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:54.028588+0000) 2022-09-08T08:23:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:54.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:54.212140+0000) 2022-09-08T08:23:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:54 smithi055 ceph-mon[97033]: pgmap v20945: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:54 smithi059 ceph-mon[97450]: pgmap v20945: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:55.292 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:55.210+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:55.212347+0000) 2022-09-08T08:23:55.293 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:55.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:55.028736+0000) 2022-09-08T08:23:55.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:23:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:23:55.289+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:23:56.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:56.027+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:56.028902+0000) 2022-09-08T08:23:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:56.211+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:56.212578+0000) 2022-09-08T08:23:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:56 smithi055 ceph-mon[97033]: pgmap v20946: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:23:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:56 smithi059 ceph-mon[97450]: pgmap v20946: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:23:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:57.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:57.212772+0000) 2022-09-08T08:23:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:57.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:57.029130+0000) 2022-09-08T08:23:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:58.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:58.213002+0000) 2022-09-08T08:23:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:58.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:58.029380+0000) 2022-09-08T08:23:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:58 smithi055 ceph-mon[97033]: pgmap v20947: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:58 smithi059 ceph-mon[97450]: pgmap v20947: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:23:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:23:59.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:59.213136+0000) 2022-09-08T08:23:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:23:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:23:59.028+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:23:59.029597+0000) 2022-09-08T08:23:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:23:59 smithi055 ceph-mon[97033]: pgmap v20948: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:23:59 smithi059 ceph-mon[97450]: pgmap v20948: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:00.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:00.213288+0000) 2022-09-08T08:24:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:00.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:00.029824+0000) 2022-09-08T08:24:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:01.212+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:01.213531+0000) 2022-09-08T08:24:01.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:01.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:01.030019+0000) 2022-09-08T08:24:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:02.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:02.213761+0000) 2022-09-08T08:24:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:02.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:02.030251+0000) 2022-09-08T08:24:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:02 smithi055 ceph-mon[97033]: pgmap v20949: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:24:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:02 smithi059 ceph-mon[97450]: pgmap v20949: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:24:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:03.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:03.213925+0000) 2022-09-08T08:24:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:03.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:03.030421+0000) 2022-09-08T08:24:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:04.029+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:04.030582+0000) 2022-09-08T08:24:04.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:04.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:04.214067+0000) 2022-09-08T08:24:04.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:04 smithi055 ceph-mon[97033]: pgmap v20950: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:04 smithi059 ceph-mon[97450]: pgmap v20950: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:05.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:05.214301+0000) 2022-09-08T08:24:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:05.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:05.030723+0000) 2022-09-08T08:24:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:06.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:06.030910+0000) 2022-09-08T08:24:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:06.213+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:06.214498+0000) 2022-09-08T08:24:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:06 smithi055 ceph-mon[97033]: pgmap v20951: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:24:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:24:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:06 smithi059 ceph-mon[97450]: pgmap v20951: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:24:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:24:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:07.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:07.031058+0000) 2022-09-08T08:24:07.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:07.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:07.214745+0000) 2022-09-08T08:24:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:24:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:24:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:08.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:08.031191+0000) 2022-09-08T08:24:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:08.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:08.214978+0000) 2022-09-08T08:24:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:08 smithi055 ceph-mon[97033]: pgmap v20952: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:24:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:08 smithi059 ceph-mon[97450]: pgmap v20952: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:24:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:09.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:09.215134+0000) 2022-09-08T08:24:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:09.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:09.031321+0000) 2022-09-08T08:24:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:09 smithi055 ceph-mon[97033]: pgmap v20953: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:24:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:09 smithi059 ceph-mon[97450]: pgmap v20953: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:24:10.306 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:10.214+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:10.215363+0000) 2022-09-08T08:24:10.306 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:10.030+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:10.031484+0000) 2022-09-08T08:24:10.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:24:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:24:10.303+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:24:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:10 smithi055 ceph-mon[97033]: osdmap e1388: 12 total, 11 up, 11 in 2022-09-08T08:24:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:10 smithi059 ceph-mon[97450]: osdmap e1388: 12 total, 11 up, 11 in 2022-09-08T08:24:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:11.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:11.215607+0000) 2022-09-08T08:24:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:11.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:11.031727+0000) 2022-09-08T08:24:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:11 smithi055 ceph-mon[97033]: pgmap v20955: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:24:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:11 smithi059 ceph-mon[97450]: pgmap v20955: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:24:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:12.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:12.215896+0000) 2022-09-08T08:24:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:12.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:12.031944+0000) 2022-09-08T08:24:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:13.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:13.216142+0000) 2022-09-08T08:24:13.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:13.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:13.032184+0000) 2022-09-08T08:24:14.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:14.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:14.216331+0000) 2022-09-08T08:24:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:14.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:14.032410+0000) 2022-09-08T08:24:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:14 smithi055 ceph-mon[97033]: pgmap v20956: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:24:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:14 smithi055 ceph-mon[97033]: osdmap e1389: 12 total, 11 up, 11 in 2022-09-08T08:24:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:14 smithi059 ceph-mon[97450]: pgmap v20956: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:24:14.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:14 smithi059 ceph-mon[97450]: osdmap e1389: 12 total, 11 up, 11 in 2022-09-08T08:24:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:15.215+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:15.216472+0000) 2022-09-08T08:24:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:15.031+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:15.032564+0000) 2022-09-08T08:24:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:16.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:16.216762+0000) 2022-09-08T08:24:16.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:16.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:16.032684+0000) 2022-09-08T08:24:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:16 smithi055 ceph-mon[97033]: pgmap v20958: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:24:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:16 smithi059 ceph-mon[97450]: pgmap v20958: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:24:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:17.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:17.217012+0000) 2022-09-08T08:24:17.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:17.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:17.032836+0000) 2022-09-08T08:24:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:18.216+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:18.217181+0000) 2022-09-08T08:24:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:18.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:18.033062+0000) 2022-09-08T08:24:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:18 smithi055 ceph-mon[97033]: pgmap v20959: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:24:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:18 smithi059 ceph-mon[97450]: pgmap v20959: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:24:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:19.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:19.217419+0000) 2022-09-08T08:24:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:19.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:19.033218+0000) 2022-09-08T08:24:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:19 smithi055 ceph-mon[97033]: pgmap v20960: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 211 B/s wr, 6 op/s 2022-09-08T08:24:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:19 smithi059 ceph-mon[97450]: pgmap v20960: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 211 B/s wr, 6 op/s 2022-09-08T08:24:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:20.032+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:20.033406+0000) 2022-09-08T08:24:20.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:20.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:20.217702+0000) 2022-09-08T08:24:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:21.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:21.033578+0000) 2022-09-08T08:24:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:21.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:21.217987+0000) 2022-09-08T08:24:22.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:22.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:22.218154+0000) 2022-09-08T08:24:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:22.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:22.033796+0000) 2022-09-08T08:24:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:22 smithi055 ceph-mon[97033]: pgmap v20961: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:24:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:22 smithi059 ceph-mon[97450]: pgmap v20961: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:24:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:23.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:23.218404+0000) 2022-09-08T08:24:23.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:23.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:23.033984+0000) 2022-09-08T08:24:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:24.217+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:24.218618+0000) 2022-09-08T08:24:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:24.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:24.034134+0000) 2022-09-08T08:24:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:24 smithi055 ceph-mon[97033]: pgmap v20962: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:24:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:24 smithi059 ceph-mon[97450]: pgmap v20962: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:24:25.320 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:25.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:25.218839+0000) 2022-09-08T08:24:25.321 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:25.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:25.034278+0000) 2022-09-08T08:24:25.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:24:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:24:25.317+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:24:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:26.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:26.219116+0000) 2022-09-08T08:24:26.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:26.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:26.034441+0000) 2022-09-08T08:24:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:26 smithi055 ceph-mon[97033]: pgmap v20963: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-08T08:24:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:26 smithi059 ceph-mon[97450]: pgmap v20963: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2022-09-08T08:24:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:27.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:27.219325+0000) 2022-09-08T08:24:27.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:27.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:27.034600+0000) 2022-09-08T08:24:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:28.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:28.219509+0000) 2022-09-08T08:24:28.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:28.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:28.034843+0000) 2022-09-08T08:24:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:28 smithi055 ceph-mon[97033]: pgmap v20964: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:28 smithi059 ceph-mon[97450]: pgmap v20964: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:29.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:29.219757+0000) 2022-09-08T08:24:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:29.033+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:29.035059+0000) 2022-09-08T08:24:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:29 smithi055 ceph-mon[97033]: pgmap v20965: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:29 smithi059 ceph-mon[97450]: pgmap v20965: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:30.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:30.220029+0000) 2022-09-08T08:24:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:30.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:30.035330+0000) 2022-09-08T08:24:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:31.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:31.220300+0000) 2022-09-08T08:24:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:31.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:31.035546+0000) 2022-09-08T08:24:32.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:32.218+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:32.220527+0000) 2022-09-08T08:24:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:32.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:32.035732+0000) 2022-09-08T08:24:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:32 smithi055 ceph-mon[97033]: pgmap v20966: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:24:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:32 smithi059 ceph-mon[97450]: pgmap v20966: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:24:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:33.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:33.220788+0000) 2022-09-08T08:24:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:33.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:33.035882+0000) 2022-09-08T08:24:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:34.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:34.221072+0000) 2022-09-08T08:24:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:34.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:34.036046+0000) 2022-09-08T08:24:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:34 smithi055 ceph-mon[97033]: pgmap v20967: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:34 smithi059 ceph-mon[97450]: pgmap v20967: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:35.219+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:35.221349+0000) 2022-09-08T08:24:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:35.034+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:35.036201+0000) 2022-09-08T08:24:36.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:36.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:36.221620+0000) 2022-09-08T08:24:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:36.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:36.036338+0000) 2022-09-08T08:24:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:36 smithi055 ceph-mon[97033]: pgmap v20968: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:24:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:36 smithi059 ceph-mon[97450]: pgmap v20968: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:24:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:37.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:37.221903+0000) 2022-09-08T08:24:37.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:37.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:37.036505+0000) 2022-09-08T08:24:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:38.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:38.222068+0000) 2022-09-08T08:24:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:38.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:38.036768+0000) 2022-09-08T08:24:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:38 smithi055 ceph-mon[97033]: pgmap v20969: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:38 smithi059 ceph-mon[97450]: pgmap v20969: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:39.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:39.222287+0000) 2022-09-08T08:24:39.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:39.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:39.036933+0000) 2022-09-08T08:24:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:39 smithi055 ceph-mon[97033]: pgmap v20970: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:39 smithi059 ceph-mon[97450]: pgmap v20970: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:40.326 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:40.220+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:40.222520+0000) 2022-09-08T08:24:40.327 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:40.035+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:40.037078+0000) 2022-09-08T08:24:40.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:24:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:24:40.322+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:24:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:41.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:41.222759+0000) 2022-09-08T08:24:41.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:41.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:41.037265+0000) 2022-09-08T08:24:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:42.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:42.222984+0000) 2022-09-08T08:24:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:42.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:42.037496+0000) 2022-09-08T08:24:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:42 smithi055 ceph-mon[97033]: pgmap v20971: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:24:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:42 smithi059 ceph-mon[97450]: pgmap v20971: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:24:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:43.222+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:43.223237+0000) 2022-09-08T08:24:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:43.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:43.037656+0000) 2022-09-08T08:24:44.153 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:44.036+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:44.037820+0000) 2022-09-08T08:24:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:44.221+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:44.223496+0000) 2022-09-08T08:24:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:44 smithi055 ceph-mon[97033]: pgmap v20972: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:44.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:24:44.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:24:44.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:24:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:44 smithi059 ceph-mon[97450]: pgmap v20972: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:44.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:24:44.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:24:44.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:24:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:45.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:45.223718+0000) 2022-09-08T08:24:45.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:45.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:45.037974+0000) 2022-09-08T08:24:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:46.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:46.223920+0000) 2022-09-08T08:24:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:46.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:46.038105+0000) 2022-09-08T08:24:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:46 smithi055 ceph-mon[97033]: pgmap v20973: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:24:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:46 smithi059 ceph-mon[97450]: pgmap v20973: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:24:47.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:47.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:47.224152+0000) 2022-09-08T08:24:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:47.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:47.038246+0000) 2022-09-08T08:24:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:48.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:48.224385+0000) 2022-09-08T08:24:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:48.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:48.038382+0000) 2022-09-08T08:24:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:48 smithi055 ceph-mon[97033]: pgmap v20974: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:48 smithi059 ceph-mon[97450]: pgmap v20974: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:49.223+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:49.224566+0000) 2022-09-08T08:24:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:49.037+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:49.038591+0000) 2022-09-08T08:24:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:49 smithi055 ceph-mon[97033]: pgmap v20975: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:49 smithi059 ceph-mon[97450]: pgmap v20975: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:50.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:50.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:50.224791+0000) 2022-09-08T08:24:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:50.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:50.038790+0000) 2022-09-08T08:24:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:51.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:51.225010+0000) 2022-09-08T08:24:51.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:51.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:51.038992+0000) 2022-09-08T08:24:52.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:52.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:52.225236+0000) 2022-09-08T08:24:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:52.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:52.039203+0000) 2022-09-08T08:24:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:52 smithi055 ceph-mon[97033]: pgmap v20976: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:24:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:52 smithi059 ceph-mon[97450]: pgmap v20976: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:24:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:53.038+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:53.039434+0000) 2022-09-08T08:24:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:53.224+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:53.225484+0000) 2022-09-08T08:24:54.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:54.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:54.039601+0000) 2022-09-08T08:24:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:54.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:54.225677+0000) 2022-09-08T08:24:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:54 smithi055 ceph-mon[97033]: pgmap v20977: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:54 smithi059 ceph-mon[97450]: pgmap v20977: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:55.326 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:55.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:55.225890+0000) 2022-09-08T08:24:55.327 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:55.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:55.039743+0000) 2022-09-08T08:24:55.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:24:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:24:55.323+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:24:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:56.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:56.226152+0000) 2022-09-08T08:24:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:56.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:56.039907+0000) 2022-09-08T08:24:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:56 smithi055 ceph-mon[97033]: pgmap v20978: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:24:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:56 smithi059 ceph-mon[97450]: pgmap v20978: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:24:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:57.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:57.226299+0000) 2022-09-08T08:24:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:57.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:57.040141+0000) 2022-09-08T08:24:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:58.225+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:58.226519+0000) 2022-09-08T08:24:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:58.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:58.040329+0000) 2022-09-08T08:24:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:58 smithi055 ceph-mon[97033]: pgmap v20979: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:58 smithi059 ceph-mon[97450]: pgmap v20979: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:24:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:24:59.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:59.226680+0000) 2022-09-08T08:24:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:24:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:24:59.039+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:24:59.040508+0000) 2022-09-08T08:24:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:24:59 smithi055 ceph-mon[97033]: pgmap v20980: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:24:59 smithi059 ceph-mon[97450]: pgmap v20980: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:00.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:00.226845+0000) 2022-09-08T08:25:00.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:00.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:00.040700+0000) 2022-09-08T08:25:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:01.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:01.227075+0000) 2022-09-08T08:25:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:01.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:01.040878+0000) 2022-09-08T08:25:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:02.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:02.227271+0000) 2022-09-08T08:25:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:02.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:02.041114+0000) 2022-09-08T08:25:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:02 smithi055 ceph-mon[97033]: pgmap v20981: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:25:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:02 smithi059 ceph-mon[97450]: pgmap v20981: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:25:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:03.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:03.041281+0000) 2022-09-08T08:25:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:03.226+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:03.227491+0000) 2022-09-08T08:25:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:04.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:04.227694+0000) 2022-09-08T08:25:04.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:04.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:04.041405+0000) 2022-09-08T08:25:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:04 smithi055 ceph-mon[97033]: pgmap v20982: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:04 smithi059 ceph-mon[97450]: pgmap v20982: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:05.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:05.227921+0000) 2022-09-08T08:25:05.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:05.040+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:05.041546+0000) 2022-09-08T08:25:06.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:06.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:06.041755+0000) 2022-09-08T08:25:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:06.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:06.228171+0000) 2022-09-08T08:25:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:06 smithi055 ceph-mon[97033]: pgmap v20983: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:25:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:25:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:06 smithi059 ceph-mon[97450]: pgmap v20983: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:25:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:25:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:07.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:07.228376+0000) 2022-09-08T08:25:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:07.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:07.041932+0000) 2022-09-08T08:25:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:25:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:25:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:08.227+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:08.228590+0000) 2022-09-08T08:25:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:08.041+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:08.042079+0000) 2022-09-08T08:25:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:08 smithi055 ceph-mon[97033]: pgmap v20984: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:25:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:08 smithi059 ceph-mon[97450]: pgmap v20984: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:25:09.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:09.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:09.042247+0000) 2022-09-08T08:25:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:09.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:09.228765+0000) 2022-09-08T08:25:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:09 smithi055 ceph-mon[97033]: pgmap v20985: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:25:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:09 smithi059 ceph-mon[97450]: pgmap v20985: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:25:10.342 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:10.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:10.042482+0000) 2022-09-08T08:25:10.343 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:10.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:10.228948+0000) 2022-09-08T08:25:10.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:25:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:25:10.338+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:25:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:10 smithi055 ceph-mon[97033]: osdmap e1390: 12 total, 11 up, 11 in 2022-09-08T08:25:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:10 smithi059 ceph-mon[97450]: osdmap e1390: 12 total, 11 up, 11 in 2022-09-08T08:25:11.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:11.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:11.229172+0000) 2022-09-08T08:25:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:11.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:11.042735+0000) 2022-09-08T08:25:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:11 smithi055 ceph-mon[97033]: pgmap v20987: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:25:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:11 smithi059 ceph-mon[97450]: pgmap v20987: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:25:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:12.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:12.229412+0000) 2022-09-08T08:25:12.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:12.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:12.042957+0000) 2022-09-08T08:25:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:13.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:13.229558+0000) 2022-09-08T08:25:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:13.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:13.043113+0000) 2022-09-08T08:25:14.232 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:14.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:14.043253+0000) 2022-09-08T08:25:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:14.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:14.229778+0000) 2022-09-08T08:25:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:14 smithi055 ceph-mon[97033]: pgmap v20988: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:25:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:14 smithi055 ceph-mon[97033]: osdmap e1391: 12 total, 11 up, 11 in 2022-09-08T08:25:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:25:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:14 smithi059 ceph-mon[97450]: pgmap v20988: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:25:14.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:14 smithi059 ceph-mon[97450]: osdmap e1391: 12 total, 11 up, 11 in 2022-09-08T08:25:14.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:25:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:15.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:15.229979+0000) 2022-09-08T08:25:15.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:15.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:15.043465+0000) 2022-09-08T08:25:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:15 smithi055 ceph-mon[97033]: pgmap v20990: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 155 B/s wr, 4 op/s 2022-09-08T08:25:15.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:15 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T08:25:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:15 smithi059 ceph-mon[97450]: pgmap v20990: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 155 B/s wr, 4 op/s 2022-09-08T08:25:15.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:15 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T08:25:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:16.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:16.043652+0000) 2022-09-08T08:25:16.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:16.228+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:16.230213+0000) 2022-09-08T08:25:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:17.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:17.043851+0000) 2022-09-08T08:25:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:17.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:17.230447+0000) 2022-09-08T08:25:17.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:17 smithi055 ceph-mon[97033]: pgmap v20991: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.9 KiB/s rd, 310 B/s wr, 9 op/s 2022-09-08T08:25:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:17 smithi059 ceph-mon[97450]: pgmap v20991: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.9 KiB/s rd, 310 B/s wr, 9 op/s 2022-09-08T08:25:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:18.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:18.230673+0000) 2022-09-08T08:25:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:18.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:18.044013+0000) 2022-09-08T08:25:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:19.042+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:19.044186+0000) 2022-09-08T08:25:19.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:19.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:19.230867+0000) 2022-09-08T08:25:19.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:19 smithi055 ceph-mon[97033]: pgmap v20992: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 247 B/s wr, 7 op/s 2022-09-08T08:25:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:19 smithi059 ceph-mon[97450]: pgmap v20992: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 247 B/s wr, 7 op/s 2022-09-08T08:25:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:20.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:20.044420+0000) 2022-09-08T08:25:20.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:20.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:20.231083+0000) 2022-09-08T08:25:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:21.229+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:21.231266+0000) 2022-09-08T08:25:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:21.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:21.044648+0000) 2022-09-08T08:25:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:21 smithi055 ceph-mon[97033]: pgmap v20993: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 119 B/s wr, 5 op/s 2022-09-08T08:25:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:21 smithi059 ceph-mon[97450]: pgmap v20993: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 119 B/s wr, 5 op/s 2022-09-08T08:25:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:22.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:22.231494+0000) 2022-09-08T08:25:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:22.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:22.044796+0000) 2022-09-08T08:25:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:23.230+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:23.231726+0000) 2022-09-08T08:25:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:23.043+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:23.044945+0000) 2022-09-08T08:25:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:23 smithi055 ceph-mon[97033]: pgmap v20994: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 119 B/s wr, 7 op/s 2022-09-08T08:25:23.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:23 smithi059 ceph-mon[97450]: pgmap v20994: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 119 B/s wr, 7 op/s 2022-09-08T08:25:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:24.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:24.231916+0000) 2022-09-08T08:25:24.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:24.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:24.045168+0000) 2022-09-08T08:25:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:24 smithi055 ceph-mon[97033]: pgmap v20995: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-08T08:25:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:24 smithi059 ceph-mon[97450]: pgmap v20995: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-08T08:25:25.354 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:25.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:25.232128+0000) 2022-09-08T08:25:25.355 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:25.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:25.045384+0000) 2022-09-08T08:25:25.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:25:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:25:25.350+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:25:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:26.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:26.232308+0000) 2022-09-08T08:25:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:26.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:26.045538+0000) 2022-09-08T08:25:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:26 smithi059 ceph-mon[97450]: pgmap v20996: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T08:25:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:27.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:27.232502+0000) 2022-09-08T08:25:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:27.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:27.045700+0000) 2022-09-08T08:25:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:26 smithi055 ceph-mon[97033]: pgmap v20996: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T08:25:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:28.231+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:28.232745+0000) 2022-09-08T08:25:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:28.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:28.045867+0000) 2022-09-08T08:25:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:28 smithi059 ceph-mon[97450]: pgmap v20997: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:29.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:29.232988+0000) 2022-09-08T08:25:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:29.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:29.046083+0000) 2022-09-08T08:25:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:29 smithi055 ceph-mon[97033]: pgmap v20997: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:30.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:30.233139+0000) 2022-09-08T08:25:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:30.044+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:30.046251+0000) 2022-09-08T08:25:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:30 smithi059 ceph-mon[97450]: pgmap v20998: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T08:25:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:31.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:31.233435+0000) 2022-09-08T08:25:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:31.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:31.046453+0000) 2022-09-08T08:25:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:30 smithi055 ceph-mon[97033]: pgmap v20998: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T08:25:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:32.232+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:32.233714+0000) 2022-09-08T08:25:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:32.045+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:32.046748+0000) 2022-09-08T08:25:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:33 smithi059 ceph-mon[97450]: pgmap v20999: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:25:33.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:33.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:33.233944+0000) 2022-09-08T08:25:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:33.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:33.046947+0000) 2022-09-08T08:25:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:33 smithi055 ceph-mon[97033]: pgmap v20999: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:25:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:34.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:34.234184+0000) 2022-09-08T08:25:34.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:34.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:34.047172+0000) 2022-09-08T08:25:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:34 smithi055 ceph-mon[97033]: pgmap v21000: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:34 smithi059 ceph-mon[97450]: pgmap v21000: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:35.233+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:35.234417+0000) 2022-09-08T08:25:35.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:35.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:35.047403+0000) 2022-09-08T08:25:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:36.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:36.234730+0000) 2022-09-08T08:25:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:36.046+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:36.047555+0000) 2022-09-08T08:25:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:36 smithi059 ceph-mon[97450]: pgmap v21001: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:25:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:37.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:37.234975+0000) 2022-09-08T08:25:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:37.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:37.047701+0000) 2022-09-08T08:25:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:37 smithi055 ceph-mon[97033]: pgmap v21001: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:25:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:38.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:38.235220+0000) 2022-09-08T08:25:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:38.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:38.047886+0000) 2022-09-08T08:25:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:39 smithi059 ceph-mon[97450]: pgmap v21002: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:39.234+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:39.235491+0000) 2022-09-08T08:25:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:39.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:39.048055+0000) 2022-09-08T08:25:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:39 smithi055 ceph-mon[97033]: pgmap v21002: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:40.364 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:40.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:40.235700+0000) 2022-09-08T08:25:40.365 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:40.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:40.048317+0000) 2022-09-08T08:25:40.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:25:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:25:40.361+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:25:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:41 smithi059 ceph-mon[97450]: pgmap v21003: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T08:25:41.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:41.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:41.235921+0000) 2022-09-08T08:25:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:41.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:41.048463+0000) 2022-09-08T08:25:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:41 smithi055 ceph-mon[97033]: pgmap v21003: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T08:25:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:42.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:42.236071+0000) 2022-09-08T08:25:42.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:42.047+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:42.048615+0000) 2022-09-08T08:25:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:43 smithi059 ceph-mon[97450]: pgmap v21004: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:25:43.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:43.235+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:43.236295+0000) 2022-09-08T08:25:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:43.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:43.048824+0000) 2022-09-08T08:25:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:43 smithi055 ceph-mon[97033]: pgmap v21004: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:25:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:44.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:44.236518+0000) 2022-09-08T08:25:44.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:44.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:44.048974+0000) 2022-09-08T08:25:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:44 smithi055 ceph-mon[97033]: pgmap v21005: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:44 smithi059 ceph-mon[97450]: pgmap v21005: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:45.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:45.236763+0000) 2022-09-08T08:25:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:45.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:45.049197+0000) 2022-09-08T08:25:46.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:46.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:46.236938+0000) 2022-09-08T08:25:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:46.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:46.049347+0000) 2022-09-08T08:25:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:46 smithi059 ceph-mon[97450]: pgmap v21006: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:25:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:47.048+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:47.049478+0000) 2022-09-08T08:25:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:47.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:47.237120+0000) 2022-09-08T08:25:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:46 smithi055 ceph-mon[97033]: pgmap v21006: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:25:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:48.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:48.049673+0000) 2022-09-08T08:25:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:48.236+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:48.237283+0000) 2022-09-08T08:25:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:49 smithi059 ceph-mon[97450]: pgmap v21007: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:49.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:49.237516+0000) 2022-09-08T08:25:49.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:49.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:49.049810+0000) 2022-09-08T08:25:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:49 smithi055 ceph-mon[97033]: pgmap v21007: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:50.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:50.237798+0000) 2022-09-08T08:25:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:50.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:50.049979+0000) 2022-09-08T08:25:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:51 smithi059 ceph-mon[97450]: pgmap v21008: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T08:25:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:51.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:51.238061+0000) 2022-09-08T08:25:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:51.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:51.050131+0000) 2022-09-08T08:25:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:51 smithi055 ceph-mon[97033]: pgmap v21008: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T08:25:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:52.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:52.238286+0000) 2022-09-08T08:25:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:52.049+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:52.050303+0000) 2022-09-08T08:25:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:53 smithi059 ceph-mon[97450]: pgmap v21009: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:25:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:53.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:53.050509+0000) 2022-09-08T08:25:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:53.237+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:53.238486+0000) 2022-09-08T08:25:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:53 smithi055 ceph-mon[97033]: pgmap v21009: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:25:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:54.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:54.238771+0000) 2022-09-08T08:25:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:54.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:54.050714+0000) 2022-09-08T08:25:54.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:54 smithi055 ceph-mon[97033]: pgmap v21010: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:54 smithi059 ceph-mon[97450]: pgmap v21010: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:55.371 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:55.238+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:55.239025+0000) 2022-09-08T08:25:55.372 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:55.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:55.050939+0000) 2022-09-08T08:25:55.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:25:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:25:55.368+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:25:56.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:56.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:56.239209+0000) 2022-09-08T08:25:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:56.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:56.051142+0000) 2022-09-08T08:25:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:57 smithi059 ceph-mon[97450]: pgmap v21011: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:25:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:57.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:57.239500+0000) 2022-09-08T08:25:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:57.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:57.051395+0000) 2022-09-08T08:25:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:57 smithi055 ceph-mon[97033]: pgmap v21011: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:25:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:58.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:58.239719+0000) 2022-09-08T08:25:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:58.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:58.051590+0000) 2022-09-08T08:25:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:25:59 smithi059 ceph-mon[97450]: pgmap v21012: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:25:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:25:59.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:59.239963+0000) 2022-09-08T08:25:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:25:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:25:59.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:25:59.051803+0000) 2022-09-08T08:25:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:25:59 smithi055 ceph-mon[97033]: pgmap v21012: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:26:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:00.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:00.240178+0000) 2022-09-08T08:26:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:00.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:00.051953+0000) 2022-09-08T08:26:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:01 smithi059 ceph-mon[97450]: pgmap v21013: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T08:26:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:01.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:01.052106+0000) 2022-09-08T08:26:01.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:01.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:01.240407+0000) 2022-09-08T08:26:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:01 smithi055 ceph-mon[97033]: pgmap v21013: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T08:26:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:02.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:02.240623+0000) 2022-09-08T08:26:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:02.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:02.052268+0000) 2022-09-08T08:26:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:03 smithi059 ceph-mon[97450]: pgmap v21014: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:26:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:03.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:03.240918+0000) 2022-09-08T08:26:03.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:03.050+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:03.052427+0000) 2022-09-08T08:26:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:03 smithi055 ceph-mon[97033]: pgmap v21014: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:26:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:04.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:04.241149+0000) 2022-09-08T08:26:04.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:04.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:04.052686+0000) 2022-09-08T08:26:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:04 smithi055 ceph-mon[97033]: pgmap v21015: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:26:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:04 smithi059 ceph-mon[97450]: pgmap v21015: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:26:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:05.239+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:05.241380+0000) 2022-09-08T08:26:05.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:05.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:05.052881+0000) 2022-09-08T08:26:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:06.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:06.241581+0000) 2022-09-08T08:26:06.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:06.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:06.053016+0000) 2022-09-08T08:26:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:07 smithi059 ceph-mon[97450]: pgmap v21016: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:26:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:26:07.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:26:07.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:07.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:07.241840+0000) 2022-09-08T08:26:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:07.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:07.053223+0000) 2022-09-08T08:26:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:07 smithi055 ceph-mon[97033]: pgmap v21016: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:26:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:26:07.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:26:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:08.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:08.242071+0000) 2022-09-08T08:26:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:08.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:08.053411+0000) 2022-09-08T08:26:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:09 smithi059 ceph-mon[97450]: pgmap v21017: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:26:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:09.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:09.242302+0000) 2022-09-08T08:26:09.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:09.051+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:09.053570+0000) 2022-09-08T08:26:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:09 smithi055 ceph-mon[97033]: pgmap v21017: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:26:10.376 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:10.240+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:10.242511+0000) 2022-09-08T08:26:10.377 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:10.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:10.053765+0000) 2022-09-08T08:26:10.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:26:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:26:10.372+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:26:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:11 smithi059 ceph-mon[97450]: pgmap v21018: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:26:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:11 smithi059 ceph-mon[97450]: osdmap e1392: 12 total, 11 up, 11 in 2022-09-08T08:26:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:11.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:11.242731+0000) 2022-09-08T08:26:11.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:11.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:11.053931+0000) 2022-09-08T08:26:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:11 smithi055 ceph-mon[97033]: pgmap v21018: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:26:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:11 smithi055 ceph-mon[97033]: osdmap e1392: 12 total, 11 up, 11 in 2022-09-08T08:26:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:12.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:12.054074+0000) 2022-09-08T08:26:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:12.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:12.242973+0000) 2022-09-08T08:26:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:13 smithi059 ceph-mon[97450]: pgmap v21020: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:26:13.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:13.242+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:13.243184+0000) 2022-09-08T08:26:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:13.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:13.054249+0000) 2022-09-08T08:26:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:13 smithi055 ceph-mon[97033]: pgmap v21020: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:26:14.246 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:14.052+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:14.054402+0000) 2022-09-08T08:26:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:14.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:14.243403+0000) 2022-09-08T08:26:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:14 smithi055 ceph-mon[97033]: pgmap v21021: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:26:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:26:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:26:14.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:26:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:14 smithi059 ceph-mon[97450]: pgmap v21021: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:26:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:26:15.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:26:15.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:26:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:15.241+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:15.243605+0000) 2022-09-08T08:26:15.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:15.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:15.054572+0000) 2022-09-08T08:26:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:16.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:16.243820+0000) 2022-09-08T08:26:16.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:16.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:16.054783+0000) 2022-09-08T08:26:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:17 smithi059 ceph-mon[97450]: pgmap v21022: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:26:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:17.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:17.244055+0000) 2022-09-08T08:26:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:17.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:17.054964+0000) 2022-09-08T08:26:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:17 smithi055 ceph-mon[97033]: pgmap v21022: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:26:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:18.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:18.244326+0000) 2022-09-08T08:26:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:18.053+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:18.055197+0000) 2022-09-08T08:26:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:19 smithi059 ceph-mon[97450]: pgmap v21023: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:26:19.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:19 smithi059 ceph-mon[97450]: osdmap e1393: 12 total, 11 up, 11 in 2022-09-08T08:26:19.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:19.243+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:19.244575+0000) 2022-09-08T08:26:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:19.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:19.055371+0000) 2022-09-08T08:26:19.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:19 smithi055 ceph-mon[97033]: pgmap v21023: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:26:19.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:19 smithi055 ceph-mon[97033]: osdmap e1393: 12 total, 11 up, 11 in 2022-09-08T08:26:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:20.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:20.244818+0000) 2022-09-08T08:26:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:20.054+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:20.055543+0000) 2022-09-08T08:26:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:21 smithi059 ceph-mon[97450]: pgmap v21025: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 103 B/s wr, 6 op/s 2022-09-08T08:26:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:21.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:21.055725+0000) 2022-09-08T08:26:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:21.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:21.245095+0000) 2022-09-08T08:26:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:21 smithi055 ceph-mon[97033]: pgmap v21025: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 103 B/s wr, 6 op/s 2022-09-08T08:26:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:22.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:22.245321+0000) 2022-09-08T08:26:22.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:22.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:22.055946+0000) 2022-09-08T08:26:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:23 smithi059 ceph-mon[97450]: pgmap v21026: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:26:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:23.244+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:23.245519+0000) 2022-09-08T08:26:23.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:23.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:23.056104+0000) 2022-09-08T08:26:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:23 smithi055 ceph-mon[97033]: pgmap v21026: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:26:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:24.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:24.245727+0000) 2022-09-08T08:26:24.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:24.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:24.056276+0000) 2022-09-08T08:26:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:24 smithi055 ceph-mon[97033]: pgmap v21027: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:26:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:24 smithi059 ceph-mon[97450]: pgmap v21027: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:26:25.392 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:25.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:25.245891+0000) 2022-09-08T08:26:25.393 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:25.055+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:25.056441+0000) 2022-09-08T08:26:25.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:26:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:26:25.389+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:26:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:26.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:26.056701+0000) 2022-09-08T08:26:26.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:26.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:26.246065+0000) 2022-09-08T08:26:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:27 smithi059 ceph-mon[97450]: pgmap v21028: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:26:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:27.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:27.246187+0000) 2022-09-08T08:26:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:27.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:27.056930+0000) 2022-09-08T08:26:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:27 smithi055 ceph-mon[97033]: pgmap v21028: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:26:28.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:28.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:28.246396+0000) 2022-09-08T08:26:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:28.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:28.057121+0000) 2022-09-08T08:26:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:29 smithi059 ceph-mon[97450]: pgmap v21029: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:26:29.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:29.245+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:29.246535+0000) 2022-09-08T08:26:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:29.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:29.057297+0000) 2022-09-08T08:26:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:29 smithi055 ceph-mon[97033]: pgmap v21029: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:26:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:30.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:30.246751+0000) 2022-09-08T08:26:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:30.056+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:30.057447+0000) 2022-09-08T08:26:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:31 smithi059 ceph-mon[97450]: pgmap v21030: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T08:26:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:31.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:31.246992+0000) 2022-09-08T08:26:31.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:31.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:31.057680+0000) 2022-09-08T08:26:31.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:31 smithi055 ceph-mon[97033]: pgmap v21030: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T08:26:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:32.246+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:32.247242+0000) 2022-09-08T08:26:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:32.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:32.057861+0000) 2022-09-08T08:26:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:33 smithi059 ceph-mon[97450]: pgmap v21031: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:26:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:33.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:33.247472+0000) 2022-09-08T08:26:33.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:33.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:33.057995+0000) 2022-09-08T08:26:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:33 smithi055 ceph-mon[97033]: pgmap v21031: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:26:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:34.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:34.247720+0000) 2022-09-08T08:26:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:34.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:34.058113+0000) 2022-09-08T08:26:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:34 smithi055 ceph-mon[97033]: pgmap v21032: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:26:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:34 smithi059 ceph-mon[97450]: pgmap v21032: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:26:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:35.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:35.247942+0000) 2022-09-08T08:26:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:35.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:35.058299+0000) 2022-09-08T08:26:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:36.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:36.248109+0000) 2022-09-08T08:26:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:36.057+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:36.058526+0000) 2022-09-08T08:26:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:37 smithi059 ceph-mon[97450]: pgmap v21033: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:26:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:37.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:37.248252+0000) 2022-09-08T08:26:37.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:37.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:37.058703+0000) 2022-09-08T08:26:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:37 smithi055 ceph-mon[97033]: pgmap v21033: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:26:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:38.247+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:38.248421+0000) 2022-09-08T08:26:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:38.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:38.058932+0000) 2022-09-08T08:26:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:39 smithi059 ceph-mon[97450]: pgmap v21034: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:26:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:39.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:39.248660+0000) 2022-09-08T08:26:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:39.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:39.059143+0000) 2022-09-08T08:26:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:39 smithi055 ceph-mon[97033]: pgmap v21034: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:26:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:40.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:40.248918+0000) 2022-09-08T08:26:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:40.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:40.059334+0000) 2022-09-08T08:26:40.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:26:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:26:40.400+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:26:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:41 smithi059 ceph-mon[97450]: pgmap v21035: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:26:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:41.248+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:41.249158+0000) 2022-09-08T08:26:41.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:41.058+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:41.059550+0000) 2022-09-08T08:26:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:41 smithi055 ceph-mon[97033]: pgmap v21035: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:26:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:42.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:42.249401+0000) 2022-09-08T08:26:42.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:42.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:42.059731+0000) 2022-09-08T08:26:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:43 smithi059 ceph-mon[97450]: pgmap v21036: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:26:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:43.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:43.249614+0000) 2022-09-08T08:26:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:43.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:43.059953+0000) 2022-09-08T08:26:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:43 smithi055 ceph-mon[97033]: pgmap v21036: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:26:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:44.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:44.249746+0000) 2022-09-08T08:26:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:44.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:44.060103+0000) 2022-09-08T08:26:45.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:45.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:45.249937+0000) 2022-09-08T08:26:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:45.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:45.060311+0000) 2022-09-08T08:26:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:45 smithi055 ceph-mon[97033]: pgmap v21037: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:26:45.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:45 smithi055 ceph-mon[97033]: pgmap v21038: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:26:45.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:45 smithi055 ceph-mon[97033]: pgmap v21039: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:26:45.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:26:45.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:45 smithi059 ceph-mon[97450]: pgmap v21037: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:26:45.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:45 smithi059 ceph-mon[97450]: pgmap v21038: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:26:45.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:45 smithi059 ceph-mon[97450]: pgmap v21039: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:26:45.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:26:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:46.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:46.250111+0000) 2022-09-08T08:26:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:46.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:46.060485+0000) 2022-09-08T08:26:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:47.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:47.250271+0000) 2022-09-08T08:26:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:47.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:47.060711+0000) 2022-09-08T08:26:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:47 smithi055 ceph-mon[97033]: pgmap v21040: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:26:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:47 smithi059 ceph-mon[97450]: pgmap v21040: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:26:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:48.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:48.250446+0000) 2022-09-08T08:26:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:48.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:48.060920+0000) 2022-09-08T08:26:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:49.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:49.250664+0000) 2022-09-08T08:26:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:49.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:49.061055+0000) 2022-09-08T08:26:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:49 smithi055 ceph-mon[97033]: pgmap v21041: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:26:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:49 smithi059 ceph-mon[97450]: pgmap v21041: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:26:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:50.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:50.250884+0000) 2022-09-08T08:26:50.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:50.059+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:50.061198+0000) 2022-09-08T08:26:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:51.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:51.251094+0000) 2022-09-08T08:26:51.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:51.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:51.061396+0000) 2022-09-08T08:26:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:51 smithi055 ceph-mon[97033]: pgmap v21042: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:26:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:51 smithi059 ceph-mon[97450]: pgmap v21042: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:26:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:52.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:52.251334+0000) 2022-09-08T08:26:52.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:52.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:52.061560+0000) 2022-09-08T08:26:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:53.249+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:53.251563+0000) 2022-09-08T08:26:53.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:53.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:53.061708+0000) 2022-09-08T08:26:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:53 smithi055 ceph-mon[97033]: pgmap v21043: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:26:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:53 smithi059 ceph-mon[97450]: pgmap v21043: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:26:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:54.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:54.251792+0000) 2022-09-08T08:26:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:54.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:54.061851+0000) 2022-09-08T08:26:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:54 smithi055 ceph-mon[97033]: pgmap v21044: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:26:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:54 smithi059 ceph-mon[97450]: pgmap v21044: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:26:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:55.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:55.252018+0000) 2022-09-08T08:26:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:55.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:55.062067+0000) 2022-09-08T08:26:55.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:26:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:26:55.414+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:26:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:56.060+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:56.062217+0000) 2022-09-08T08:26:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:56.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:56.252178+0000) 2022-09-08T08:26:57.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:57.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:57.252349+0000) 2022-09-08T08:26:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:57.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:57.062359+0000) 2022-09-08T08:26:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:57 smithi059 ceph-mon[97450]: pgmap v21045: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:26:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:57 smithi055 ceph-mon[97033]: pgmap v21045: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:26:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:58.250+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:58.252541+0000) 2022-09-08T08:26:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:58.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:58.062508+0000) 2022-09-08T08:26:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:26:59.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:59.252745+0000) 2022-09-08T08:26:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:26:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:26:59.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:26:59.062673+0000) 2022-09-08T08:26:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:26:59 smithi059 ceph-mon[97450]: pgmap v21046: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:26:59.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:26:59 smithi055 ceph-mon[97033]: pgmap v21046: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:00.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:00.252931+0000) 2022-09-08T08:27:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:00.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:00.062861+0000) 2022-09-08T08:27:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:01.251+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:01.253066+0000) 2022-09-08T08:27:01.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:01.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:01.063027+0000) 2022-09-08T08:27:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:01 smithi059 ceph-mon[97450]: pgmap v21047: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:27:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:01 smithi055 ceph-mon[97033]: pgmap v21047: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:27:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:02.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:02.253310+0000) 2022-09-08T08:27:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:02.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:02.063184+0000) 2022-09-08T08:27:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:03.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:03.253566+0000) 2022-09-08T08:27:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:03.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:03.063354+0000) 2022-09-08T08:27:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:03 smithi059 ceph-mon[97450]: pgmap v21048: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:03.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:03 smithi055 ceph-mon[97033]: pgmap v21048: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:04.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:04.252+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:04.253794+0000) 2022-09-08T08:27:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:04.061+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:04.063497+0000) 2022-09-08T08:27:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:04 smithi055 ceph-mon[97033]: pgmap v21049: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:04 smithi059 ceph-mon[97450]: pgmap v21049: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:05.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:05.253930+0000) 2022-09-08T08:27:05.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:05.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:05.063705+0000) 2022-09-08T08:27:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:06.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:06.254157+0000) 2022-09-08T08:27:06.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:06.062+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:06.063905+0000) 2022-09-08T08:27:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:27:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:27:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:07.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:07.064046+0000) 2022-09-08T08:27:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:07.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:07.254337+0000) 2022-09-08T08:27:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:07 smithi055 ceph-mon[97033]: pgmap v21050: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:27:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:27:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:07 smithi059 ceph-mon[97450]: pgmap v21050: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:27:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:27:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:08.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:08.064199+0000) 2022-09-08T08:27:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:08.253+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:08.254562+0000) 2022-09-08T08:27:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:09.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:09.254795+0000) 2022-09-08T08:27:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:09.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:09.064400+0000) 2022-09-08T08:27:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:09 smithi059 ceph-mon[97450]: pgmap v21051: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:27:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:09 smithi055 ceph-mon[97033]: pgmap v21051: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:27:10.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:10.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:10.255008+0000) 2022-09-08T08:27:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:10.063+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:10.064624+0000) 2022-09-08T08:27:10.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:27:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:27:10.425+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:27:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:10 smithi055 ceph-mon[97033]: osdmap e1394: 12 total, 11 up, 11 in 2022-09-08T08:27:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:10 smithi055 ceph-mon[97033]: pgmap v21053: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:27:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:10 smithi059 ceph-mon[97450]: osdmap e1394: 12 total, 11 up, 11 in 2022-09-08T08:27:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:10 smithi059 ceph-mon[97450]: pgmap v21053: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:27:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:11.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:11.255254+0000) 2022-09-08T08:27:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:11.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:11.064790+0000) 2022-09-08T08:27:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:12.254+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:12.255472+0000) 2022-09-08T08:27:12.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:12.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:12.064994+0000) 2022-09-08T08:27:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:13.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:13.255739+0000) 2022-09-08T08:27:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:13.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:13.065192+0000) 2022-09-08T08:27:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:13 smithi059 ceph-mon[97450]: pgmap v21054: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:27:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:13 smithi055 ceph-mon[97033]: pgmap v21054: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:27:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:14.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:14.255978+0000) 2022-09-08T08:27:14.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:14.064+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:14.065400+0000) 2022-09-08T08:27:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:14 smithi055 ceph-mon[97033]: pgmap v21055: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:27:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:14 smithi059 ceph-mon[97450]: pgmap v21055: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:27:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:15.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:15.256234+0000) 2022-09-08T08:27:15.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:15.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:15.065668+0000) 2022-09-08T08:27:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:16.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:16.256463+0000) 2022-09-08T08:27:16.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:16.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:16.065812+0000) 2022-09-08T08:27:17.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:17.255+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:17.256601+0000) 2022-09-08T08:27:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:17.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:17.066015+0000) 2022-09-08T08:27:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:17 smithi059 ceph-mon[97450]: pgmap v21056: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:27:17.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:17 smithi055 ceph-mon[97033]: pgmap v21056: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:27:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:18.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:18.256852+0000) 2022-09-08T08:27:18.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:18.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:18.066202+0000) 2022-09-08T08:27:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:19.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:19.066350+0000) 2022-09-08T08:27:19.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:19.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:19.257000+0000) 2022-09-08T08:27:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:19 smithi059 ceph-mon[97450]: pgmap v21057: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:27:19.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:19 smithi059 ceph-mon[97450]: osdmap e1395: 12 total, 11 up, 11 in 2022-09-08T08:27:19.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:19 smithi055 ceph-mon[97033]: pgmap v21057: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:27:19.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:19 smithi055 ceph-mon[97033]: osdmap e1395: 12 total, 11 up, 11 in 2022-09-08T08:27:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:20.065+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:20.066488+0000) 2022-09-08T08:27:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:20.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:20.257214+0000) 2022-09-08T08:27:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:21.256+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:21.257441+0000) 2022-09-08T08:27:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:21.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:21.066705+0000) 2022-09-08T08:27:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:21 smithi059 ceph-mon[97450]: pgmap v21059: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:27:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:21 smithi055 ceph-mon[97033]: pgmap v21059: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:27:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:22.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:22.257686+0000) 2022-09-08T08:27:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:22.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:22.066875+0000) 2022-09-08T08:27:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:23.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:23.257935+0000) 2022-09-08T08:27:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:23.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:23.067087+0000) 2022-09-08T08:27:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:23 smithi059 ceph-mon[97450]: pgmap v21060: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:27:23.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:23 smithi055 ceph-mon[97033]: pgmap v21060: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:27:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:24.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:24.258124+0000) 2022-09-08T08:27:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:24.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:24.067307+0000) 2022-09-08T08:27:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:24 smithi055 ceph-mon[97033]: pgmap v21061: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:27:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:24 smithi059 ceph-mon[97450]: pgmap v21061: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:27:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:25.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:25.258378+0000) 2022-09-08T08:27:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:25.066+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:25.067529+0000) 2022-09-08T08:27:25.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:27:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:27:25.427+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:27:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:26.257+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:26.258596+0000) 2022-09-08T08:27:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:26.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:26.067770+0000) 2022-09-08T08:27:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:27.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:27.258876+0000) 2022-09-08T08:27:27.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:27.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:27.067938+0000) 2022-09-08T08:27:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:27 smithi059 ceph-mon[97450]: pgmap v21062: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:27:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:27 smithi055 ceph-mon[97033]: pgmap v21062: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:27:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:28.258+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:28.259138+0000) 2022-09-08T08:27:28.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:28.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:28.068098+0000) 2022-09-08T08:27:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:29.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:29.259377+0000) 2022-09-08T08:27:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:29.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:29.068262+0000) 2022-09-08T08:27:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:29 smithi059 ceph-mon[97450]: pgmap v21063: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:27:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:29 smithi055 ceph-mon[97033]: pgmap v21063: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:27:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:30.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:30.259583+0000) 2022-09-08T08:27:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:30.067+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:30.068480+0000) 2022-09-08T08:27:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:31.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:31.068754+0000) 2022-09-08T08:27:31.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:31.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:31.259813+0000) 2022-09-08T08:27:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:31 smithi059 ceph-mon[97450]: pgmap v21064: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T08:27:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:31 smithi055 ceph-mon[97033]: pgmap v21064: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T08:27:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:32.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:32.260057+0000) 2022-09-08T08:27:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:32.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:32.068921+0000) 2022-09-08T08:27:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:33.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:33.260309+0000) 2022-09-08T08:27:33.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:33.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:33.069058+0000) 2022-09-08T08:27:33.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:33 smithi055 ceph-mon[97033]: pgmap v21065: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:33 smithi059 ceph-mon[97450]: pgmap v21065: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:34.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:34.260492+0000) 2022-09-08T08:27:34.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:34.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:34.069196+0000) 2022-09-08T08:27:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:34 smithi055 ceph-mon[97033]: pgmap v21066: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:34 smithi059 ceph-mon[97450]: pgmap v21066: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:35.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:35.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:35.260698+0000) 2022-09-08T08:27:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:35.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:35.069358+0000) 2022-09-08T08:27:36.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:36.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:36.069571+0000) 2022-09-08T08:27:36.406 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:36.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:36.260889+0000) 2022-09-08T08:27:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:37.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:37.069752+0000) 2022-09-08T08:27:37.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:37.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:37.261057+0000) 2022-09-08T08:27:37.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:37 smithi059 ceph-mon[97450]: pgmap v21067: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:27:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:37 smithi055 ceph-mon[97033]: pgmap v21067: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:27:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:38.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:38.069916+0000) 2022-09-08T08:27:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:38.259+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:38.261272+0000) 2022-09-08T08:27:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:39.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:39.261459+0000) 2022-09-08T08:27:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:39.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:39.070069+0000) 2022-09-08T08:27:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:39 smithi059 ceph-mon[97450]: pgmap v21068: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:39 smithi055 ceph-mon[97033]: pgmap v21068: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:40.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:40.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:40.261709+0000) 2022-09-08T08:27:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:40.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:40.070280+0000) 2022-09-08T08:27:40.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:27:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:27:40.436+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:27:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:41.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:41.261944+0000) 2022-09-08T08:27:41.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:41.068+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:41.070540+0000) 2022-09-08T08:27:41.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:41 smithi055 ceph-mon[97033]: pgmap v21069: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:27:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:41 smithi059 ceph-mon[97450]: pgmap v21069: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:27:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:42.260+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:42.262113+0000) 2022-09-08T08:27:42.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:42.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:42.070746+0000) 2022-09-08T08:27:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:43.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:43.262357+0000) 2022-09-08T08:27:43.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:43.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:43.070888+0000) 2022-09-08T08:27:43.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:43 smithi055 ceph-mon[97033]: pgmap v21070: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:43 smithi059 ceph-mon[97450]: pgmap v21070: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:44.356 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:44.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:44.071094+0000) 2022-09-08T08:27:44.357 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:44.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:44.262587+0000) 2022-09-08T08:27:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:44 smithi055 ceph-mon[97033]: pgmap v21071: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:44.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:27:44.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:27:44.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:44 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:27:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:44 smithi059 ceph-mon[97450]: pgmap v21071: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:45.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:27:45.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:27:45.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:44 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:27:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:45.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:45.262781+0000) 2022-09-08T08:27:45.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:45.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:45.071228+0000) 2022-09-08T08:27:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:46.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:46.263006+0000) 2022-09-08T08:27:46.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:46.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:46.071358+0000) 2022-09-08T08:27:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:47.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:47.263221+0000) 2022-09-08T08:27:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:47.069+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:47.071509+0000) 2022-09-08T08:27:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:47 smithi059 ceph-mon[97450]: pgmap v21072: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:27:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:47 smithi055 ceph-mon[97033]: pgmap v21072: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:27:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:48.261+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:48.263418+0000) 2022-09-08T08:27:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:48.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:48.071740+0000) 2022-09-08T08:27:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:49.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:49.071926+0000) 2022-09-08T08:27:49.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:49.262+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:49.263627+0000) 2022-09-08T08:27:49.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:49 smithi055 ceph-mon[97033]: pgmap v21073: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:27:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:49 smithi059 ceph-mon[97450]: pgmap v21073: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:27:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:50.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:50.072121+0000) 2022-09-08T08:27:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:50.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:50.263894+0000) 2022-09-08T08:27:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:51.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:51.264134+0000) 2022-09-08T08:27:51.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:51.070+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:51.072274+0000) 2022-09-08T08:27:51.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:51 smithi055 ceph-mon[97033]: pgmap v21074: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:27:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:51 smithi059 ceph-mon[97450]: pgmap v21074: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:27:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:52.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:52.072486+0000) 2022-09-08T08:27:52.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:52.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:52.264377+0000) 2022-09-08T08:27:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:53.263+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:53.264563+0000) 2022-09-08T08:27:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:53.071+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:53.072739+0000) 2022-09-08T08:27:53.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:53 smithi055 ceph-mon[97033]: pgmap v21075: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:27:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:53 smithi059 ceph-mon[97450]: pgmap v21075: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:27:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:54.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:54.264788+0000) 2022-09-08T08:27:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:54.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:54.072943+0000) 2022-09-08T08:27:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:54 smithi055 ceph-mon[97033]: pgmap v21076: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:27:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:54 smithi059 ceph-mon[97450]: pgmap v21076: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:27:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:55.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:55.264926+0000) 2022-09-08T08:27:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:55.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:55.073087+0000) 2022-09-08T08:27:55.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:27:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:27:55.440+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:27:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:56.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:56.265143+0000) 2022-09-08T08:27:56.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:56.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:56.073231+0000) 2022-09-08T08:27:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:57.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:57.073364+0000) 2022-09-08T08:27:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:57.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:57.265371+0000) 2022-09-08T08:27:57.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:57 smithi055 ceph-mon[97033]: pgmap v21077: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:27:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:57 smithi059 ceph-mon[97450]: pgmap v21077: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:27:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:58.072+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:58.073511+0000) 2022-09-08T08:27:58.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:58.264+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:58.265573+0000) 2022-09-08T08:27:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:27:59.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:59.265810+0000) 2022-09-08T08:27:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:27:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:27:59.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:27:59.073711+0000) 2022-09-08T08:27:59.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:27:59 smithi055 ceph-mon[97033]: pgmap v21078: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:27:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:27:59 smithi059 ceph-mon[97450]: pgmap v21078: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:00.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:00.265953+0000) 2022-09-08T08:28:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:00.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:00.073936+0000) 2022-09-08T08:28:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:01.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:01.266145+0000) 2022-09-08T08:28:01.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:01.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:01.074150+0000) 2022-09-08T08:28:01.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:01 smithi055 ceph-mon[97033]: pgmap v21079: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:28:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:01 smithi059 ceph-mon[97450]: pgmap v21079: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:28:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:02.265+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:02.266352+0000) 2022-09-08T08:28:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:02.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:02.074391+0000) 2022-09-08T08:28:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:03.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:03.266604+0000) 2022-09-08T08:28:03.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:03.073+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:03.074604+0000) 2022-09-08T08:28:03.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:03 smithi055 ceph-mon[97033]: pgmap v21080: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:03 smithi059 ceph-mon[97450]: pgmap v21080: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:04.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:04.266812+0000) 2022-09-08T08:28:04.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:04.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:04.074833+0000) 2022-09-08T08:28:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:04 smithi055 ceph-mon[97033]: pgmap v21081: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:04 smithi059 ceph-mon[97450]: pgmap v21081: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:05.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:05.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:05.267053+0000) 2022-09-08T08:28:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:05.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:05.075077+0000) 2022-09-08T08:28:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:06.266+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:06.267291+0000) 2022-09-08T08:28:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:06.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:06.075271+0000) 2022-09-08T08:28:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:28:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:28:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:07.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:07.267530+0000) 2022-09-08T08:28:07.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:07.074+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:07.075515+0000) 2022-09-08T08:28:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:07 smithi055 ceph-mon[97033]: pgmap v21082: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:28:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:28:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:07 smithi059 ceph-mon[97450]: pgmap v21082: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:28:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:28:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:08.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:08.267778+0000) 2022-09-08T08:28:08.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:08.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:08.075761+0000) 2022-09-08T08:28:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:09.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:09.268024+0000) 2022-09-08T08:28:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:09.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:09.075908+0000) 2022-09-08T08:28:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:09 smithi055 ceph-mon[97033]: pgmap v21083: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:28:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:09 smithi059 ceph-mon[97450]: pgmap v21083: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:28:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:10.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:10.268190+0000) 2022-09-08T08:28:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:10.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:10.076109+0000) 2022-09-08T08:28:10.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:28:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:28:10.452+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:28:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:10 smithi055 ceph-mon[97033]: osdmap e1396: 12 total, 11 up, 11 in 2022-09-08T08:28:10.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:10 smithi055 ceph-mon[97033]: pgmap v21085: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:28:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:10 smithi059 ceph-mon[97450]: osdmap e1396: 12 total, 11 up, 11 in 2022-09-08T08:28:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:10 smithi059 ceph-mon[97450]: pgmap v21085: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:28:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:11.267+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:11.268412+0000) 2022-09-08T08:28:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:11.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:11.076259+0000) 2022-09-08T08:28:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:12.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:12.268595+0000) 2022-09-08T08:28:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:12.075+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:12.076461+0000) 2022-09-08T08:28:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:13.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:13.268742+0000) 2022-09-08T08:28:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:13.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:13.076741+0000) 2022-09-08T08:28:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:13 smithi055 ceph-mon[97033]: pgmap v21086: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:28:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:13 smithi059 ceph-mon[97450]: pgmap v21086: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:28:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:14.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:14.268977+0000) 2022-09-08T08:28:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:14.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:14.076968+0000) 2022-09-08T08:28:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:15.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:15.269229+0000) 2022-09-08T08:28:15.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:15.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:15.077152+0000) 2022-09-08T08:28:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:15 smithi059 ceph-mon[97450]: pgmap v21087: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:28:15.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:15 smithi059 ceph-mon[97450]: pgmap v21088: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.9 KiB/s rd, 124 B/s wr, 3 op/s 2022-09-08T08:28:15.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:15 smithi059 ceph-mon[97450]: pgmap v21089: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T08:28:15.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:15 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:28:15.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:15 smithi055 ceph-mon[97033]: pgmap v21087: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:28:15.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:15 smithi055 ceph-mon[97033]: pgmap v21088: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.9 KiB/s rd, 124 B/s wr, 3 op/s 2022-09-08T08:28:15.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:15 smithi055 ceph-mon[97033]: pgmap v21089: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T08:28:15.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:15 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:28:16.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:16.268+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:16.269445+0000) 2022-09-08T08:28:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:16.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:16.077295+0000) 2022-09-08T08:28:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:17.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:17.269739+0000) 2022-09-08T08:28:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:17.076+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:17.077467+0000) 2022-09-08T08:28:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:17 smithi059 ceph-mon[97450]: pgmap v21090: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.7 KiB/s rd, 302 B/s wr, 9 op/s 2022-09-08T08:28:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:17 smithi055 ceph-mon[97033]: pgmap v21090: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.7 KiB/s rd, 302 B/s wr, 9 op/s 2022-09-08T08:28:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:18.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:18.269984+0000) 2022-09-08T08:28:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:18.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:18.077758+0000) 2022-09-08T08:28:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:19.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:19.270126+0000) 2022-09-08T08:28:19.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:19.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:19.077947+0000) 2022-09-08T08:28:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:19 smithi059 ceph-mon[97450]: pgmap v21091: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.9 KiB/s rd, 124 B/s wr, 3 op/s 2022-09-08T08:28:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:19 smithi059 ceph-mon[97450]: osdmap e1397: 12 total, 11 up, 11 in 2022-09-08T08:28:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:19 smithi055 ceph-mon[97033]: pgmap v21091: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.9 KiB/s rd, 124 B/s wr, 3 op/s 2022-09-08T08:28:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:19 smithi055 ceph-mon[97033]: osdmap e1397: 12 total, 11 up, 11 in 2022-09-08T08:28:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:20.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:20.270335+0000) 2022-09-08T08:28:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:20.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:20.078121+0000) 2022-09-08T08:28:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:21.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:21.270573+0000) 2022-09-08T08:28:21.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:21.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:21.078292+0000) 2022-09-08T08:28:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:21 smithi059 ceph-mon[97450]: pgmap v21093: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 5.0 KiB/s rd, 163 B/s wr, 9 op/s 2022-09-08T08:28:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:21 smithi055 ceph-mon[97033]: pgmap v21093: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 5.0 KiB/s rd, 163 B/s wr, 9 op/s 2022-09-08T08:28:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:22.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:22.270797+0000) 2022-09-08T08:28:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:22.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:22.078475+0000) 2022-09-08T08:28:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:23.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:23.270998+0000) 2022-09-08T08:28:23.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:23.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:23.078701+0000) 2022-09-08T08:28:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:23 smithi059 ceph-mon[97450]: pgmap v21094: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T08:28:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:23 smithi055 ceph-mon[97033]: pgmap v21094: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T08:28:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:24.269+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:24.271202+0000) 2022-09-08T08:28:24.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:24.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:24.078858+0000) 2022-09-08T08:28:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:24 smithi059 ceph-mon[97450]: pgmap v21095: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:28:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:24 smithi055 ceph-mon[97033]: pgmap v21095: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:28:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:25.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:25.271420+0000) 2022-09-08T08:28:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:25.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:25.079015+0000) 2022-09-08T08:28:25.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:28:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:28:25.462+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:28:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:26.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:26.079216+0000) 2022-09-08T08:28:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:26.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:26.271664+0000) 2022-09-08T08:28:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:26 smithi055 ceph-mon[97033]: pgmap v21096: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:28:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:26 smithi059 ceph-mon[97450]: pgmap v21096: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:28:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:27.077+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:27.079466+0000) 2022-09-08T08:28:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:27.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:27.271843+0000) 2022-09-08T08:28:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:28.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:28.272057+0000) 2022-09-08T08:28:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:28.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:28.079713+0000) 2022-09-08T08:28:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:28 smithi055 ceph-mon[97033]: pgmap v21097: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:28:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:28 smithi059 ceph-mon[97450]: pgmap v21097: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:28:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:29.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:29.272321+0000) 2022-09-08T08:28:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:29.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:29.079957+0000) 2022-09-08T08:28:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:30.270+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:30.272545+0000) 2022-09-08T08:28:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:30.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:30.080111+0000) 2022-09-08T08:28:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:30 smithi055 ceph-mon[97033]: pgmap v21098: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:28:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:30 smithi059 ceph-mon[97450]: pgmap v21098: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:28:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:31.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:31.272786+0000) 2022-09-08T08:28:31.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:31.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:31.080280+0000) 2022-09-08T08:28:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:32.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:32.273016+0000) 2022-09-08T08:28:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:32.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:32.080426+0000) 2022-09-08T08:28:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:32 smithi055 ceph-mon[97033]: pgmap v21099: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:32 smithi059 ceph-mon[97450]: pgmap v21099: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:33.078+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:33.080561+0000) 2022-09-08T08:28:33.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:33.271+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:33.273186+0000) 2022-09-08T08:28:34.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:34.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:34.273359+0000) 2022-09-08T08:28:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:34.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:34.080762+0000) 2022-09-08T08:28:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:34 smithi059 ceph-mon[97450]: pgmap v21100: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:34 smithi055 ceph-mon[97033]: pgmap v21100: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:35.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:35.273607+0000) 2022-09-08T08:28:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:35.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:35.080976+0000) 2022-09-08T08:28:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:36.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:36.273926+0000) 2022-09-08T08:28:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:36.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:36.081202+0000) 2022-09-08T08:28:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:36 smithi055 ceph-mon[97033]: pgmap v21101: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:28:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:36 smithi059 ceph-mon[97450]: pgmap v21101: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:28:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:37.272+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:37.274086+0000) 2022-09-08T08:28:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:37.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:37.081357+0000) 2022-09-08T08:28:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:38.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:38.274293+0000) 2022-09-08T08:28:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:38.079+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:38.081509+0000) 2022-09-08T08:28:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:38 smithi055 ceph-mon[97033]: pgmap v21102: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:38 smithi059 ceph-mon[97450]: pgmap v21102: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:39.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:39.274504+0000) 2022-09-08T08:28:39.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:39.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:39.081676+0000) 2022-09-08T08:28:40.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:40.273+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:40.274734+0000) 2022-09-08T08:28:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:40.080+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:40.081807+0000) 2022-09-08T08:28:40.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:28:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:28:40.478+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:28:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:40 smithi055 ceph-mon[97033]: pgmap v21103: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:28:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:40 smithi059 ceph-mon[97450]: pgmap v21103: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:28:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:41.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:41.274900+0000) 2022-09-08T08:28:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:41.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:41.081973+0000) 2022-09-08T08:28:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:42.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:42.275146+0000) 2022-09-08T08:28:42.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:42.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:42.082127+0000) 2022-09-08T08:28:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:42 smithi055 ceph-mon[97033]: pgmap v21104: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:42 smithi059 ceph-mon[97450]: pgmap v21104: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:43.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:43.275316+0000) 2022-09-08T08:28:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:43.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:43.082334+0000) 2022-09-08T08:28:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:44.274+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:44.275505+0000) 2022-09-08T08:28:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:44.081+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:44.082504+0000) 2022-09-08T08:28:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:44 smithi059 ceph-mon[97450]: pgmap v21105: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:44 smithi055 ceph-mon[97033]: pgmap v21105: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:45.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:45.275730+0000) 2022-09-08T08:28:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:45.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:45.082698+0000) 2022-09-08T08:28:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:46.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:46.275885+0000) 2022-09-08T08:28:46.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:46.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:46.082912+0000) 2022-09-08T08:28:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:46 smithi055 ceph-mon[97033]: pgmap v21106: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:28:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:46 smithi059 ceph-mon[97450]: pgmap v21106: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:28:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:47.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:47.276071+0000) 2022-09-08T08:28:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:47.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:47.083166+0000) 2022-09-08T08:28:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:48.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:48.276303+0000) 2022-09-08T08:28:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:48.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:48.083349+0000) 2022-09-08T08:28:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:48 smithi055 ceph-mon[97033]: pgmap v21107: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:48 smithi059 ceph-mon[97450]: pgmap v21107: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:49.275+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:49.276545+0000) 2022-09-08T08:28:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:49.082+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:49.083577+0000) 2022-09-08T08:28:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:50.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:50.276743+0000) 2022-09-08T08:28:50.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:50.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:50.083789+0000) 2022-09-08T08:28:50.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:50 smithi055 ceph-mon[97033]: pgmap v21108: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:28:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:50 smithi059 ceph-mon[97450]: pgmap v21108: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:28:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:51.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:51.276897+0000) 2022-09-08T08:28:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:51.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:51.083975+0000) 2022-09-08T08:28:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:52.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:52.277084+0000) 2022-09-08T08:28:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:52.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:52.084117+0000) 2022-09-08T08:28:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:52 smithi055 ceph-mon[97033]: pgmap v21109: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:52 smithi059 ceph-mon[97450]: pgmap v21109: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:53.276+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:53.277279+0000) 2022-09-08T08:28:53.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:53.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:53.084361+0000) 2022-09-08T08:28:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:54.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:54.277481+0000) 2022-09-08T08:28:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:54.083+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:54.084610+0000) 2022-09-08T08:28:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:54 smithi059 ceph-mon[97450]: pgmap v21110: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:55.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:54 smithi055 ceph-mon[97033]: pgmap v21110: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:55.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:55.277750+0000) 2022-09-08T08:28:55.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:55.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:55.084721+0000) 2022-09-08T08:28:55.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:28:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:28:55.481+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:28:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:56.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:56.277925+0000) 2022-09-08T08:28:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:56.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:56.084891+0000) 2022-09-08T08:28:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:56 smithi055 ceph-mon[97033]: pgmap v21111: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:28:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:56 smithi059 ceph-mon[97450]: pgmap v21111: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:28:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:57.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:57.278116+0000) 2022-09-08T08:28:57.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:57.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:57.085128+0000) 2022-09-08T08:28:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:58.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:58.278328+0000) 2022-09-08T08:28:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:58.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:58.085397+0000) 2022-09-08T08:28:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:28:58 smithi055 ceph-mon[97033]: pgmap v21112: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:28:58 smithi059 ceph-mon[97450]: pgmap v21112: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:28:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:28:59.277+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:59.278529+0000) 2022-09-08T08:28:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:28:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:28:59.084+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:28:59.085544+0000) 2022-09-08T08:29:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:00.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:00.278760+0000) 2022-09-08T08:29:00.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:00.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:00.085709+0000) 2022-09-08T08:29:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:00 smithi055 ceph-mon[97033]: pgmap v21113: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:29:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:00 smithi059 ceph-mon[97450]: pgmap v21113: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:29:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:01.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:01.278932+0000) 2022-09-08T08:29:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:01.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:01.085905+0000) 2022-09-08T08:29:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:02.278+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:02.279144+0000) 2022-09-08T08:29:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:02.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:02.086105+0000) 2022-09-08T08:29:02.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:02 smithi055 ceph-mon[97033]: pgmap v21114: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:29:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:02 smithi059 ceph-mon[97450]: pgmap v21114: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:29:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:03.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:03.279391+0000) 2022-09-08T08:29:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:03.085+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:03.086262+0000) 2022-09-08T08:29:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:04.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:04.279682+0000) 2022-09-08T08:29:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:04.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:04.086411+0000) 2022-09-08T08:29:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:04 smithi059 ceph-mon[97450]: pgmap v21115: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:29:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:04 smithi055 ceph-mon[97033]: pgmap v21115: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:29:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:05.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:05.279843+0000) 2022-09-08T08:29:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:05.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:05.086632+0000) 2022-09-08T08:29:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:06.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:06.086825+0000) 2022-09-08T08:29:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:06.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:06.280024+0000) 2022-09-08T08:29:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:29:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:29:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:07.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:07.086981+0000) 2022-09-08T08:29:07.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:07.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:07.280225+0000) 2022-09-08T08:29:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:07 smithi055 ceph-mon[97033]: pgmap v21116: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:29:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:29:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:07 smithi059 ceph-mon[97450]: pgmap v21116: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:29:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:29:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:08.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:08.280462+0000) 2022-09-08T08:29:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:08.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:08.087141+0000) 2022-09-08T08:29:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:08 smithi055 ceph-mon[97033]: pgmap v21117: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:29:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:08 smithi059 ceph-mon[97450]: pgmap v21117: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:29:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:09.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:09.280699+0000) 2022-09-08T08:29:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:09.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:09.087332+0000) 2022-09-08T08:29:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:10.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:10.280837+0000) 2022-09-08T08:29:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:10.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:10.087520+0000) 2022-09-08T08:29:10.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:29:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:29:10.486+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:29:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:10 smithi059 ceph-mon[97450]: osdmap e1398: 12 total, 11 up, 11 in 2022-09-08T08:29:11.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:10 smithi059 ceph-mon[97450]: pgmap v21119: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:29:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:10 smithi055 ceph-mon[97033]: osdmap e1398: 12 total, 11 up, 11 in 2022-09-08T08:29:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:10 smithi055 ceph-mon[97033]: pgmap v21119: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:29:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:11.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:11.280988+0000) 2022-09-08T08:29:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:11.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:11.087709+0000) 2022-09-08T08:29:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:12.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:12.281217+0000) 2022-09-08T08:29:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:12.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:12.087932+0000) 2022-09-08T08:29:12.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:12 smithi055 ceph-mon[97033]: pgmap v21120: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:29:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:12 smithi059 ceph-mon[97450]: pgmap v21120: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:29:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:13.279+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:13.281403+0000) 2022-09-08T08:29:13.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:13.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:13.088167+0000) 2022-09-08T08:29:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:14.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:14.281623+0000) 2022-09-08T08:29:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:14.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:14.088420+0000) 2022-09-08T08:29:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:14 smithi055 ceph-mon[97033]: pgmap v21121: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:29:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:29:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:29:14.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:29:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:14 smithi059 ceph-mon[97450]: pgmap v21121: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:29:15.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:29:15.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:29:15.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:29:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:15.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:15.281771+0000) 2022-09-08T08:29:15.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:15.086+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:15.088615+0000) 2022-09-08T08:29:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:16.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:16.282006+0000) 2022-09-08T08:29:16.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:16.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:16.088801+0000) 2022-09-08T08:29:16.685 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:16.686 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:16.686 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:16.687 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:16.687 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:16 smithi059 ceph-mon[97450]: pgmap v21122: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:29:16.724 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:16.724 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:16.725 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:16.725 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:16.726 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:16 smithi055 ceph-mon[97033]: pgmap v21122: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:29:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:17.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:17.282233+0000) 2022-09-08T08:29:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:17.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:17.088945+0000) 2022-09-08T08:29:18.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:18.280+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:18.282437+0000) 2022-09-08T08:29:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:18.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:18.089094+0000) 2022-09-08T08:29:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:18 smithi055 ceph-mon[97033]: pgmap v21123: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:29:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:18 smithi059 ceph-mon[97450]: pgmap v21123: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:29:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:19.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:19.282687+0000) 2022-09-08T08:29:19.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:19.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:19.089251+0000) 2022-09-08T08:29:19.575 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:19 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T08:29:19.576 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:19 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T08:29:19.577 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:19.578 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:19 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T08:29:19.579 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:19 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T08:29:19.579 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:19 smithi055 ceph-mon[97033]: osdmap e1399: 12 total, 11 up, 11 in 2022-09-08T08:29:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:19 smithi059 ceph-mon[97450]: osdmap e1399: 12 total, 11 up, 11 in 2022-09-08T08:29:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:20.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:20.282869+0000) 2022-09-08T08:29:20.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:20.087+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:20.089491+0000) 2022-09-08T08:29:20.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:20 smithi055 ceph-mon[97033]: pgmap v21125: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:29:21.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:20 smithi059 ceph-mon[97450]: pgmap v21125: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:29:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:21.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:21.283010+0000) 2022-09-08T08:29:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:21.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:21.089693+0000) 2022-09-08T08:29:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:22.281+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:22.283159+0000) 2022-09-08T08:29:22.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:22.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:22.089886+0000) 2022-09-08T08:29:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:22 smithi055 ceph-mon[97033]: pgmap v21126: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:29:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:22 smithi059 ceph-mon[97450]: pgmap v21126: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:29:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:23.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:23.283413+0000) 2022-09-08T08:29:23.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:23.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:23.090056+0000) 2022-09-08T08:29:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:24.282+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:24.283615+0000) 2022-09-08T08:29:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:24.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:24.090269+0000) 2022-09-08T08:29:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:24 smithi059 ceph-mon[97450]: pgmap v21127: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:29:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:24 smithi055 ceph-mon[97033]: pgmap v21127: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:29:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:25.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:25.283892+0000) 2022-09-08T08:29:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:25.088+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:25.090488+0000) 2022-09-08T08:29:25.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:29:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:29:25.492+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:29:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:26.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:26.284185+0000) 2022-09-08T08:29:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:26.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:26.090754+0000) 2022-09-08T08:29:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:26 smithi055 ceph-mon[97033]: pgmap v21128: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:29:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:26 smithi059 ceph-mon[97450]: pgmap v21128: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:29:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:27.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:27.284372+0000) 2022-09-08T08:29:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:27.089+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:27.090920+0000) 2022-09-08T08:29:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:28.283+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:28.284613+0000) 2022-09-08T08:29:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:28.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:28.091077+0000) 2022-09-08T08:29:28.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:28 smithi055 ceph-mon[97033]: pgmap v21129: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:29:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:28 smithi059 ceph-mon[97450]: pgmap v21129: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:29:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:29.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:29.284829+0000) 2022-09-08T08:29:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:29.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:29.091227+0000) 2022-09-08T08:29:30.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:30.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:30.285113+0000) 2022-09-08T08:29:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:30.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:30.091393+0000) 2022-09-08T08:29:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:30 smithi055 ceph-mon[97033]: pgmap v21130: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:29:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:30 smithi059 ceph-mon[97450]: pgmap v21130: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:29:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:31.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:31.285275+0000) 2022-09-08T08:29:31.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:31.090+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:31.091572+0000) 2022-09-08T08:29:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:32.284+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:32.285515+0000) 2022-09-08T08:29:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:32.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:32.091748+0000) 2022-09-08T08:29:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:32 smithi055 ceph-mon[97033]: pgmap v21131: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:29:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:32 smithi059 ceph-mon[97450]: pgmap v21131: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:29:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:33.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:33.285813+0000) 2022-09-08T08:29:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:33.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:33.091969+0000) 2022-09-08T08:29:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:34.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:34.286066+0000) 2022-09-08T08:29:34.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:34.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:34.092158+0000) 2022-09-08T08:29:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:34 smithi059 ceph-mon[97450]: pgmap v21132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:29:35.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:34 smithi055 ceph-mon[97033]: pgmap v21132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:29:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:35.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:35.286285+0000) 2022-09-08T08:29:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:35.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:35.092377+0000) 2022-09-08T08:29:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:36.285+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:36.286500+0000) 2022-09-08T08:29:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:36.091+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:36.092535+0000) 2022-09-08T08:29:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:36 smithi055 ceph-mon[97033]: pgmap v21133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:29:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:36 smithi059 ceph-mon[97450]: pgmap v21133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:29:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:37.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:37.286739+0000) 2022-09-08T08:29:37.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:37.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:37.092735+0000) 2022-09-08T08:29:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:38.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:38.286954+0000) 2022-09-08T08:29:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:38.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:38.092881+0000) 2022-09-08T08:29:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:38 smithi055 ceph-mon[97033]: pgmap v21134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:29:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:38 smithi059 ceph-mon[97450]: pgmap v21134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:29:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:39.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:39.287083+0000) 2022-09-08T08:29:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:39.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:39.093036+0000) 2022-09-08T08:29:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:40.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:40.093244+0000) 2022-09-08T08:29:40.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:40.286+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:40.287264+0000) 2022-09-08T08:29:40.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:29:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:29:40.496+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:29:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:40 smithi055 ceph-mon[97033]: pgmap v21135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:29:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:40 smithi059 ceph-mon[97450]: pgmap v21135: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:29:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:41.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:41.093404+0000) 2022-09-08T08:29:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:41.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:41.287458+0000) 2022-09-08T08:29:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:42.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:42.287624+0000) 2022-09-08T08:29:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:42.092+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:42.093561+0000) 2022-09-08T08:29:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:42 smithi055 ceph-mon[97033]: pgmap v21136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:29:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:42 smithi059 ceph-mon[97450]: pgmap v21136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:29:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:43.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:43.287808+0000) 2022-09-08T08:29:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:43.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:43.093786+0000) 2022-09-08T08:29:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:44.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:44.287937+0000) 2022-09-08T08:29:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:44.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:44.093911+0000) 2022-09-08T08:29:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:44 smithi059 ceph-mon[97450]: pgmap v21137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:29:45.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:44 smithi055 ceph-mon[97033]: pgmap v21137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:29:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:45.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:45.288088+0000) 2022-09-08T08:29:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:45.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:45.094086+0000) 2022-09-08T08:29:46.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:45 smithi059 ceph-mon[97450]: pgmap v21138: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:29:46.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:45 smithi059 ceph-mon[97450]: pgmap v21139: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:29:46.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:45 smithi055 ceph-mon[97033]: pgmap v21138: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:29:46.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:45 smithi055 ceph-mon[97033]: pgmap v21139: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:29:46.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:29:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:46.287+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:46.288317+0000) 2022-09-08T08:29:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:46.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:46.094253+0000) 2022-09-08T08:29:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:47.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:47.288522+0000) 2022-09-08T08:29:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:47.093+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:47.094430+0000) 2022-09-08T08:29:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:47 smithi059 ceph-mon[97450]: pgmap v21140: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:29:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:47 smithi055 ceph-mon[97033]: pgmap v21140: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:29:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:48.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:48.288773+0000) 2022-09-08T08:29:48.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:48.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:48.094693+0000) 2022-09-08T08:29:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:49.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:49.288925+0000) 2022-09-08T08:29:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:49.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:49.094934+0000) 2022-09-08T08:29:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:49 smithi059 ceph-mon[97450]: pgmap v21141: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:29:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:49 smithi055 ceph-mon[97033]: pgmap v21141: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:29:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:50.288+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:50.289162+0000) 2022-09-08T08:29:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:50.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:50.095114+0000) 2022-09-08T08:29:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:51.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:51.289421+0000) 2022-09-08T08:29:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:51.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:51.095261+0000) 2022-09-08T08:29:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:51 smithi055 ceph-mon[97033]: pgmap v21142: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:29:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:51 smithi059 ceph-mon[97450]: pgmap v21142: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:29:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:52.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:52.289706+0000) 2022-09-08T08:29:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:52.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:52.095404+0000) 2022-09-08T08:29:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:53.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:53.289903+0000) 2022-09-08T08:29:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:53.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:53.095561+0000) 2022-09-08T08:29:54.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:53 smithi055 ceph-mon[97033]: pgmap v21143: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:29:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:53 smithi059 ceph-mon[97450]: pgmap v21143: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:29:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:54.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:54.095729+0000) 2022-09-08T08:29:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:54.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:54.290058+0000) 2022-09-08T08:29:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:55.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:55.290243+0000) 2022-09-08T08:29:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:55.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:55.095874+0000) 2022-09-08T08:29:55.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:29:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:29:55.505+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:29:56.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:55 smithi055 ceph-mon[97033]: pgmap v21144: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:29:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:55 smithi059 ceph-mon[97450]: pgmap v21144: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:29:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:56.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:56.290451+0000) 2022-09-08T08:29:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:56.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:56.096069+0000) 2022-09-08T08:29:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:57.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:57.290758+0000) 2022-09-08T08:29:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:57.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:57.096229+0000) 2022-09-08T08:29:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:57 smithi055 ceph-mon[97033]: pgmap v21145: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:29:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:57 smithi059 ceph-mon[97450]: pgmap v21145: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:29:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:58.289+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:58.290948+0000) 2022-09-08T08:29:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:58.094+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:58.096470+0000) 2022-09-08T08:29:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:29:59.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:59.291152+0000) 2022-09-08T08:29:59.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:29:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:29:59.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:29:59.096684+0000) 2022-09-08T08:30:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:29:59 smithi059 ceph-mon[97450]: pgmap v21146: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:29:59 smithi055 ceph-mon[97033]: pgmap v21146: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:00.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:00.291383+0000) 2022-09-08T08:30:00.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:00.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:00.096876+0000) 2022-09-08T08:30:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T08:30:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T08:30:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T08:30:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T08:30:01.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T08:30:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T08:30:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T08:30:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T08:30:01.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T08:30:01.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T08:30:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:01.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:01.291603+0000) 2022-09-08T08:30:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:01.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:01.097025+0000) 2022-09-08T08:30:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:01 smithi059 ceph-mon[97450]: pgmap v21147: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:30:02.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:01 smithi055 ceph-mon[97033]: pgmap v21147: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:30:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:02.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:02.291816+0000) 2022-09-08T08:30:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:02.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:02.097256+0000) 2022-09-08T08:30:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:03.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:03.291952+0000) 2022-09-08T08:30:03.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:03.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:03.097430+0000) 2022-09-08T08:30:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:03 smithi055 ceph-mon[97033]: pgmap v21148: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:03 smithi059 ceph-mon[97450]: pgmap v21148: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:04.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:04.292124+0000) 2022-09-08T08:30:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:04.095+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:04.097573+0000) 2022-09-08T08:30:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:05.290+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:05.292324+0000) 2022-09-08T08:30:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:05.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:05.097710+0000) 2022-09-08T08:30:06.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:05 smithi055 ceph-mon[97033]: pgmap v21149: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:05 smithi059 ceph-mon[97450]: pgmap v21149: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:06.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:06.292577+0000) 2022-09-08T08:30:06.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:06.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:06.097924+0000) 2022-09-08T08:30:07.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:30:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:30:07.412 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:07.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:07.292813+0000) 2022-09-08T08:30:07.412 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:07.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:07.098116+0000) 2022-09-08T08:30:08.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:07 smithi055 ceph-mon[97033]: pgmap v21150: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:30:08.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:30:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:07 smithi059 ceph-mon[97450]: pgmap v21150: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:30:08.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:30:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:08.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:08.293037+0000) 2022-09-08T08:30:08.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:08.096+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:08.098299+0000) 2022-09-08T08:30:09.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:09.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:09.293275+0000) 2022-09-08T08:30:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:09.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:09.098454+0000) 2022-09-08T08:30:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:09 smithi059 ceph-mon[97450]: pgmap v21151: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:30:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:09 smithi055 ceph-mon[97033]: pgmap v21151: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:30:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:10.291+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:10.293435+0000) 2022-09-08T08:30:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:10.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:10.098697+0000) 2022-09-08T08:30:10.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:30:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:30:10.514+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:30:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:10 smithi059 ceph-mon[97450]: osdmap e1400: 12 total, 11 up, 11 in 2022-09-08T08:30:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:10 smithi055 ceph-mon[97033]: osdmap e1400: 12 total, 11 up, 11 in 2022-09-08T08:30:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:11.292+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:11.293701+0000) 2022-09-08T08:30:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:11.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:11.098878+0000) 2022-09-08T08:30:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:11 smithi059 ceph-mon[97450]: pgmap v21153: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:30:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:11 smithi055 ceph-mon[97033]: pgmap v21153: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:30:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:12.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:12.293957+0000) 2022-09-08T08:30:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:12.097+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:12.099017+0000) 2022-09-08T08:30:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:13.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:13.294188+0000) 2022-09-08T08:30:13.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:13.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:13.099232+0000) 2022-09-08T08:30:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:13 smithi059 ceph-mon[97450]: pgmap v21154: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:30:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:13 smithi055 ceph-mon[97033]: pgmap v21154: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:30:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:14.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:14.294371+0000) 2022-09-08T08:30:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:14.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:14.099481+0000) 2022-09-08T08:30:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:15.293+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:15.294582+0000) 2022-09-08T08:30:15.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:15.098+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:15.099763+0000) 2022-09-08T08:30:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:15 smithi059 ceph-mon[97450]: pgmap v21155: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:30:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:15 smithi055 ceph-mon[97033]: pgmap v21155: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:30:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:16.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:16.294754+0000) 2022-09-08T08:30:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:16.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:16.100001+0000) 2022-09-08T08:30:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:17.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:17.100259+0000) 2022-09-08T08:30:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:17.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:17.294962+0000) 2022-09-08T08:30:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:17 smithi059 ceph-mon[97450]: pgmap v21156: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:30:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:17 smithi055 ceph-mon[97033]: pgmap v21156: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:30:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:18.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:18.295163+0000) 2022-09-08T08:30:18.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:18.099+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:18.100487+0000) 2022-09-08T08:30:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:19.294+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:19.295433+0000) 2022-09-08T08:30:19.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:19.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:19.100767+0000) 2022-09-08T08:30:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:19 smithi059 ceph-mon[97450]: osdmap e1401: 12 total, 11 up, 11 in 2022-09-08T08:30:20.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:19 smithi059 ceph-mon[97450]: pgmap v21158: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.4 KiB/s rd, 227 B/s wr, 6 op/s 2022-09-08T08:30:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:19 smithi055 ceph-mon[97033]: osdmap e1401: 12 total, 11 up, 11 in 2022-09-08T08:30:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:19 smithi055 ceph-mon[97033]: pgmap v21158: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.4 KiB/s rd, 227 B/s wr, 6 op/s 2022-09-08T08:30:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:20.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:20.295668+0000) 2022-09-08T08:30:20.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:20.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:20.101023+0000) 2022-09-08T08:30:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:21.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:21.295824+0000) 2022-09-08T08:30:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:21.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:21.101211+0000) 2022-09-08T08:30:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:21 smithi055 ceph-mon[97033]: pgmap v21159: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:30:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:21 smithi059 ceph-mon[97450]: pgmap v21159: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:30:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:22.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:22.295994+0000) 2022-09-08T08:30:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:22.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:22.101367+0000) 2022-09-08T08:30:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:23.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:23.296155+0000) 2022-09-08T08:30:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:23.100+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:23.101602+0000) 2022-09-08T08:30:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:23 smithi055 ceph-mon[97033]: pgmap v21160: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:30:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:23 smithi059 ceph-mon[97450]: pgmap v21160: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:30:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:24.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:24.296297+0000) 2022-09-08T08:30:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:24.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:24.101864+0000) 2022-09-08T08:30:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:25.295+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:25.296426+0000) 2022-09-08T08:30:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:25.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:25.102103+0000) 2022-09-08T08:30:25.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:30:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:30:25.529+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:30:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:25 smithi055 ceph-mon[97033]: pgmap v21161: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:30:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:25 smithi059 ceph-mon[97450]: pgmap v21161: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:30:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:26.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:26.296678+0000) 2022-09-08T08:30:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:26.101+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:26.102353+0000) 2022-09-08T08:30:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:27.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:27.296856+0000) 2022-09-08T08:30:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:27.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:27.102524+0000) 2022-09-08T08:30:28.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:27 smithi055 ceph-mon[97033]: pgmap v21162: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:30:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:27 smithi059 ceph-mon[97450]: pgmap v21162: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:30:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:28.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:28.102709+0000) 2022-09-08T08:30:28.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:28.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:28.297064+0000) 2022-09-08T08:30:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:29.296+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:29.297315+0000) 2022-09-08T08:30:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:29.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:29.102858+0000) 2022-09-08T08:30:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:29 smithi059 ceph-mon[97450]: pgmap v21163: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:30:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:29 smithi055 ceph-mon[97033]: pgmap v21163: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:30:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:30.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:30.297602+0000) 2022-09-08T08:30:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:30.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:30.103073+0000) 2022-09-08T08:30:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:31.102+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:31.103253+0000) 2022-09-08T08:30:31.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:31.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:31.297840+0000) 2022-09-08T08:30:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:31 smithi055 ceph-mon[97033]: pgmap v21164: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:30:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:31 smithi059 ceph-mon[97450]: pgmap v21164: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:30:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:32.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:32.297992+0000) 2022-09-08T08:30:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:32.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:32.103491+0000) 2022-09-08T08:30:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:33.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:33.298238+0000) 2022-09-08T08:30:33.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:33.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:33.103704+0000) 2022-09-08T08:30:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:33 smithi055 ceph-mon[97033]: pgmap v21165: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:30:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:33 smithi059 ceph-mon[97450]: pgmap v21165: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:30:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:34.297+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:34.298402+0000) 2022-09-08T08:30:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:34.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:34.103923+0000) 2022-09-08T08:30:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:35.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:35.298569+0000) 2022-09-08T08:30:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:35.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:35.104102+0000) 2022-09-08T08:30:36.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:35 smithi055 ceph-mon[97033]: pgmap v21166: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:30:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:35 smithi059 ceph-mon[97450]: pgmap v21166: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:30:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:36.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:36.298804+0000) 2022-09-08T08:30:36.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:36.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:36.104302+0000) 2022-09-08T08:30:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:37.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:37.299024+0000) 2022-09-08T08:30:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:37.103+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:37.104549+0000) 2022-09-08T08:30:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:37 smithi055 ceph-mon[97033]: pgmap v21167: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:30:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:37 smithi059 ceph-mon[97450]: pgmap v21167: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:30:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:38.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:38.104795+0000) 2022-09-08T08:30:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:38.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:38.299243+0000) 2022-09-08T08:30:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:39.298+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:39.299421+0000) 2022-09-08T08:30:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:39.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:39.105023+0000) 2022-09-08T08:30:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:39 smithi059 ceph-mon[97450]: pgmap v21168: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:30:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:39 smithi055 ceph-mon[97033]: pgmap v21168: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:30:40.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:40.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:40.299624+0000) 2022-09-08T08:30:40.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:40.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:40.105277+0000) 2022-09-08T08:30:40.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:30:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:30:40.537+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:30:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:41.104+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:41.105460+0000) 2022-09-08T08:30:41.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:41.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:41.299820+0000) 2022-09-08T08:30:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:41 smithi055 ceph-mon[97033]: pgmap v21169: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:30:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:41 smithi059 ceph-mon[97450]: pgmap v21169: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:30:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:42.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:42.105734+0000) 2022-09-08T08:30:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:42.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:42.300054+0000) 2022-09-08T08:30:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:43.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:43.300294+0000) 2022-09-08T08:30:43.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:43.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:43.105972+0000) 2022-09-08T08:30:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:43 smithi055 ceph-mon[97033]: pgmap v21170: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:43 smithi059 ceph-mon[97450]: pgmap v21170: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:44.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:44.300522+0000) 2022-09-08T08:30:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:44.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:44.106224+0000) 2022-09-08T08:30:45.303 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:45.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:45.106418+0000) 2022-09-08T08:30:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:45.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:45.300719+0000) 2022-09-08T08:30:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:45 smithi055 ceph-mon[97033]: pgmap v21171: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:46.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:30:46.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:30:46.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:30:46.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:30:46.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:45 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:30:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:45 smithi059 ceph-mon[97450]: pgmap v21171: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:46.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:30:46.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:30:46.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:30:46.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:30:46.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:45 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:30:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:46.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:46.300853+0000) 2022-09-08T08:30:46.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:46.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:46.106599+0000) 2022-09-08T08:30:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:47.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:47.300994+0000) 2022-09-08T08:30:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:47.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:47.106775+0000) 2022-09-08T08:30:48.208 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:47 smithi055 ceph-mon[97033]: pgmap v21172: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:30:48.209 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:30:48.210 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:47 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:30:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:47 smithi059 ceph-mon[97450]: pgmap v21172: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:30:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:30:48.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:47 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:30:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:48.299+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:48.301170+0000) 2022-09-08T08:30:48.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:48.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:48.107015+0000) 2022-09-08T08:30:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:49.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:49.301326+0000) 2022-09-08T08:30:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:49.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:49.107274+0000) 2022-09-08T08:30:49.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:30:49.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:30:49.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:49 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 89832k 2022-09-08T08:30:49.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:49 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 91988857: error parsing value: Value '91988857' is below minimum 939524096 2022-09-08T08:30:49.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:30:49.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:30:49.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:49 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 89832k 2022-09-08T08:30:49.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:49 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 91988857: error parsing value: Value '91988857' is below minimum 939524096 2022-09-08T08:30:49.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:49.537 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:30:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:50.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:50.301551+0000) 2022-09-08T08:30:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:50.105+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:50.107453+0000) 2022-09-08T08:30:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:50 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T08:30:50.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:50 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T08:30:50.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:50 smithi055 ceph-mon[97033]: pgmap v21173: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:50 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T08:30:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:50 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T08:30:50.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:50 smithi059 ceph-mon[97450]: pgmap v21173: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:51.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:51.301722+0000) 2022-09-08T08:30:51.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:51.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:51.107680+0000) 2022-09-08T08:30:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:52.300+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:52.301892+0000) 2022-09-08T08:30:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:52.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:52.107870+0000) 2022-09-08T08:30:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:52 smithi055 ceph-mon[97033]: pgmap v21174: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:30:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:52 smithi059 ceph-mon[97450]: pgmap v21174: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:30:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:53.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:53.302133+0000) 2022-09-08T08:30:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:53.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:53.108049+0000) 2022-09-08T08:30:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:54.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:54.302368+0000) 2022-09-08T08:30:54.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:54.106+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:54.108242+0000) 2022-09-08T08:30:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:54 smithi055 ceph-mon[97033]: pgmap v21175: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:54 smithi059 ceph-mon[97450]: pgmap v21175: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:55.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:55.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:55.302571+0000) 2022-09-08T08:30:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:55.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:55.108415+0000) 2022-09-08T08:30:55.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:30:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:30:55.545+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:30:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:56.301+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:56.302775+0000) 2022-09-08T08:30:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:56.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:56.108691+0000) 2022-09-08T08:30:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:56 smithi055 ceph-mon[97033]: pgmap v21176: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:56 smithi059 ceph-mon[97450]: pgmap v21176: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:30:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:57.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:57.302988+0000) 2022-09-08T08:30:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:57.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:57.108892+0000) 2022-09-08T08:30:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:58.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:58.303208+0000) 2022-09-08T08:30:58.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:58.107+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:58.109080+0000) 2022-09-08T08:30:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:58 smithi055 ceph-mon[97033]: pgmap v21177: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:30:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:58 smithi059 ceph-mon[97450]: pgmap v21177: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:30:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:30:59.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:59.303409+0000) 2022-09-08T08:30:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:30:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:30:59.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:30:59.109316+0000) 2022-09-08T08:31:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:30:59 smithi059 ceph-mon[97450]: pgmap v21178: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:30:59 smithi055 ceph-mon[97033]: pgmap v21178: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:00.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:00.303600+0000) 2022-09-08T08:31:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:00.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:00.109481+0000) 2022-09-08T08:31:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:01.302+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:01.303742+0000) 2022-09-08T08:31:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:01.108+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:01.109684+0000) 2022-09-08T08:31:02.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:01 smithi055 ceph-mon[97033]: pgmap v21179: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:31:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:01 smithi059 ceph-mon[97450]: pgmap v21179: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:31:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:02.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:02.303967+0000) 2022-09-08T08:31:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:02.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:02.109932+0000) 2022-09-08T08:31:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:03.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:03.304187+0000) 2022-09-08T08:31:03.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:03.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:03.110172+0000) 2022-09-08T08:31:04.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:03 smithi055 ceph-mon[97033]: pgmap v21180: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:03 smithi059 ceph-mon[97450]: pgmap v21180: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:04.303+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:04.304399+0000) 2022-09-08T08:31:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:04.109+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:04.110414+0000) 2022-09-08T08:31:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:05.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:05.304667+0000) 2022-09-08T08:31:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:05.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:05.110684+0000) 2022-09-08T08:31:06.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:05 smithi055 ceph-mon[97033]: pgmap v21181: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:05 smithi059 ceph-mon[97450]: pgmap v21181: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:06.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:06.304906+0000) 2022-09-08T08:31:06.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:06.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:06.110962+0000) 2022-09-08T08:31:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:31:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:31:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:07.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:07.305120+0000) 2022-09-08T08:31:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:07.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:07.111130+0000) 2022-09-08T08:31:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:07 smithi055 ceph-mon[97033]: pgmap v21182: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:31:08.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:31:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:07 smithi059 ceph-mon[97450]: pgmap v21182: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:31:08.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:31:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:08.304+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:08.305368+0000) 2022-09-08T08:31:08.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:08.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:08.111302+0000) 2022-09-08T08:31:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:09.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:09.305602+0000) 2022-09-08T08:31:09.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:09.110+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:09.111493+0000) 2022-09-08T08:31:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:09 smithi059 ceph-mon[97450]: pgmap v21183: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:31:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:09 smithi055 ceph-mon[97033]: pgmap v21183: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:31:10.308 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:10.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:10.111676+0000) 2022-09-08T08:31:10.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:31:10.554+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:31:10.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:10.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:10.305836+0000) 2022-09-08T08:31:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:10 smithi059 ceph-mon[97450]: osdmap e1402: 12 total, 11 up, 11 in 2022-09-08T08:31:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:10 smithi055 ceph-mon[97033]: osdmap e1402: 12 total, 11 up, 11 in 2022-09-08T08:31:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:11.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:11.306038+0000) 2022-09-08T08:31:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:11.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:11.111836+0000) 2022-09-08T08:31:12.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:11 smithi055 ceph-mon[97033]: pgmap v21185: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:31:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:11 smithi059 ceph-mon[97450]: pgmap v21185: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:31:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:12.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:12.306277+0000) 2022-09-08T08:31:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:12.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:12.111962+0000) 2022-09-08T08:31:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:13.305+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:13.306423+0000) 2022-09-08T08:31:13.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:13.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:13.112133+0000) 2022-09-08T08:31:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:13 smithi055 ceph-mon[97033]: pgmap v21186: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:31:14.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:13 smithi059 ceph-mon[97450]: pgmap v21186: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:31:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:14.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:14.112271+0000) 2022-09-08T08:31:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:14.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:14.306629+0000) 2022-09-08T08:31:15.309 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:15.111+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:15.112485+0000) 2022-09-08T08:31:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:15.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:15.306860+0000) 2022-09-08T08:31:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:16.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:16.307069+0000) 2022-09-08T08:31:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:16.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:16.112755+0000) 2022-09-08T08:31:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:16 smithi055 ceph-mon[97033]: pgmap v21187: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:31:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:16 smithi055 ceph-mon[97033]: pgmap v21188: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 247 B/s wr, 7 op/s 2022-09-08T08:31:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:16 smithi055 ceph-mon[97033]: pgmap v21189: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 5.1 KiB/s rd, 326 B/s wr, 10 op/s 2022-09-08T08:31:16.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:31:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:16 smithi059 ceph-mon[97450]: pgmap v21187: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:31:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:16 smithi059 ceph-mon[97450]: pgmap v21188: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 247 B/s wr, 7 op/s 2022-09-08T08:31:16.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:16 smithi059 ceph-mon[97450]: pgmap v21189: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 5.1 KiB/s rd, 326 B/s wr, 10 op/s 2022-09-08T08:31:16.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:31:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:17.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:17.307310+0000) 2022-09-08T08:31:17.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:17.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:17.112981+0000) 2022-09-08T08:31:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:18.306+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:18.307556+0000) 2022-09-08T08:31:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:18.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:18.113156+0000) 2022-09-08T08:31:18.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:18 smithi055 ceph-mon[97033]: pgmap v21190: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.4 KiB/s rd, 281 B/s wr, 8 op/s 2022-09-08T08:31:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:18 smithi059 ceph-mon[97450]: pgmap v21190: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.4 KiB/s rd, 281 B/s wr, 8 op/s 2022-09-08T08:31:19.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:19.112+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:19.113312+0000) 2022-09-08T08:31:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:19.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:19.307820+0000) 2022-09-08T08:31:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:19 smithi059 ceph-mon[97450]: osdmap e1403: 12 total, 11 up, 11 in 2022-09-08T08:31:20.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:19 smithi059 ceph-mon[97450]: pgmap v21192: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T08:31:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:19 smithi055 ceph-mon[97033]: osdmap e1403: 12 total, 11 up, 11 in 2022-09-08T08:31:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:19 smithi055 ceph-mon[97033]: pgmap v21192: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.6 KiB/s rd, 163 B/s wr, 5 op/s 2022-09-08T08:31:20.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:20.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:20.113517+0000) 2022-09-08T08:31:20.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:20.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:20.308035+0000) 2022-09-08T08:31:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:21.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:21.308288+0000) 2022-09-08T08:31:21.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:21.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:21.113779+0000) 2022-09-08T08:31:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:22.307+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:22.308439+0000) 2022-09-08T08:31:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:22.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:22.114018+0000) 2022-09-08T08:31:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:22 smithi055 ceph-mon[97033]: pgmap v21193: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 5.1 KiB/s rd, 163 B/s wr, 10 op/s 2022-09-08T08:31:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:22 smithi059 ceph-mon[97450]: pgmap v21193: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 5.1 KiB/s rd, 163 B/s wr, 10 op/s 2022-09-08T08:31:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:23.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:23.308722+0000) 2022-09-08T08:31:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:23.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:23.114261+0000) 2022-09-08T08:31:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:24.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:24.308942+0000) 2022-09-08T08:31:24.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:24.113+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:24.114432+0000) 2022-09-08T08:31:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:24 smithi055 ceph-mon[97033]: pgmap v21194: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.0 KiB/s rd, 3 op/s 2022-09-08T08:31:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:24 smithi059 ceph-mon[97450]: pgmap v21194: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.0 KiB/s rd, 3 op/s 2022-09-08T08:31:25.154 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:25.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:25.114678+0000) 2022-09-08T08:31:25.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:31:25.555+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:31:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:25.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:25.309121+0000) 2022-09-08T08:31:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:26.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:26.309348+0000) 2022-09-08T08:31:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:26.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:26.114906+0000) 2022-09-08T08:31:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:26 smithi055 ceph-mon[97033]: pgmap v21195: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:31:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:26 smithi059 ceph-mon[97450]: pgmap v21195: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:31:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:27.308+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:27.309610+0000) 2022-09-08T08:31:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:27.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:27.115077+0000) 2022-09-08T08:31:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:28.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:28.309873+0000) 2022-09-08T08:31:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:28.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:28.115199+0000) 2022-09-08T08:31:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:28 smithi055 ceph-mon[97033]: pgmap v21196: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:31:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:28 smithi059 ceph-mon[97450]: pgmap v21196: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:31:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:29.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:29.310058+0000) 2022-09-08T08:31:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:29.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:29.115354+0000) 2022-09-08T08:31:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:29 smithi055 ceph-mon[97033]: pgmap v21197: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:31:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:29 smithi059 ceph-mon[97450]: pgmap v21197: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:31:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:30.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:30.310255+0000) 2022-09-08T08:31:30.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:30.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:30.115580+0000) 2022-09-08T08:31:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:31.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:31.310452+0000) 2022-09-08T08:31:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:31.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:31.115728+0000) 2022-09-08T08:31:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:32.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:32.310657+0000) 2022-09-08T08:31:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:32.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:32.115930+0000) 2022-09-08T08:31:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:32 smithi055 ceph-mon[97033]: pgmap v21198: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:31:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:32 smithi059 ceph-mon[97450]: pgmap v21198: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:31:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:33.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:33.310857+0000) 2022-09-08T08:31:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:33.114+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:33.116071+0000) 2022-09-08T08:31:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:34.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:34.311102+0000) 2022-09-08T08:31:34.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:34.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:34.116302+0000) 2022-09-08T08:31:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:34 smithi055 ceph-mon[97033]: pgmap v21199: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:34 smithi059 ceph-mon[97450]: pgmap v21199: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:35.309+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:35.311288+0000) 2022-09-08T08:31:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:35.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:35.116536+0000) 2022-09-08T08:31:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:36.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:36.311521+0000) 2022-09-08T08:31:36.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:36.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:36.116781+0000) 2022-09-08T08:31:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:36 smithi055 ceph-mon[97033]: pgmap v21200: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:31:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:36 smithi059 ceph-mon[97450]: pgmap v21200: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:31:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:37.310+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:37.311785+0000) 2022-09-08T08:31:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:37.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:37.117008+0000) 2022-09-08T08:31:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:38.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:38.312044+0000) 2022-09-08T08:31:38.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:38.115+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:38.117164+0000) 2022-09-08T08:31:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:38 smithi055 ceph-mon[97033]: pgmap v21201: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:38 smithi059 ceph-mon[97450]: pgmap v21201: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:39.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:39.312276+0000) 2022-09-08T08:31:39.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:39.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:39.117394+0000) 2022-09-08T08:31:40.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:39 smithi055 ceph-mon[97033]: pgmap v21202: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:39 smithi059 ceph-mon[97450]: pgmap v21202: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:40.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:40.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:40.312530+0000) 2022-09-08T08:31:40.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:40.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:40.117626+0000) 2022-09-08T08:31:40.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:31:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:31:40.570+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:31:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:41.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:41.117809+0000) 2022-09-08T08:31:41.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:41.311+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:41.312786+0000) 2022-09-08T08:31:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:42.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:42.313052+0000) 2022-09-08T08:31:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:42.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:42.118031+0000) 2022-09-08T08:31:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:42 smithi055 ceph-mon[97033]: pgmap v21203: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:31:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:42 smithi059 ceph-mon[97450]: pgmap v21203: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:31:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:43.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:43.313291+0000) 2022-09-08T08:31:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:43.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:43.118173+0000) 2022-09-08T08:31:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:44.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:44.313540+0000) 2022-09-08T08:31:44.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:44.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:44.118378+0000) 2022-09-08T08:31:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:44 smithi055 ceph-mon[97033]: pgmap v21204: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:44 smithi059 ceph-mon[97450]: pgmap v21204: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:45.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:45.116+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:45.118595+0000) 2022-09-08T08:31:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:45.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:45.313750+0000) 2022-09-08T08:31:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:46.312+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:46.313938+0000) 2022-09-08T08:31:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:46.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:46.118786+0000) 2022-09-08T08:31:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:46 smithi055 ceph-mon[97033]: pgmap v21205: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:31:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:46 smithi059 ceph-mon[97450]: pgmap v21205: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:31:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:47.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:47.314137+0000) 2022-09-08T08:31:47.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:47.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:47.118916+0000) 2022-09-08T08:31:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:48.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:48.314385+0000) 2022-09-08T08:31:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:48.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:48.119089+0000) 2022-09-08T08:31:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:48 smithi055 ceph-mon[97033]: pgmap v21206: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:48 smithi059 ceph-mon[97450]: pgmap v21206: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:49.313+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:49.314632+0000) 2022-09-08T08:31:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:49.117+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:49.119350+0000) 2022-09-08T08:31:50.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:49 smithi055 ceph-mon[97033]: pgmap v21207: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:49 smithi059 ceph-mon[97450]: pgmap v21207: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:50.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:50.314889+0000) 2022-09-08T08:31:50.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:50.118+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:50.119561+0000) 2022-09-08T08:31:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:51.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:51.315056+0000) 2022-09-08T08:31:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:51.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:51.119775+0000) 2022-09-08T08:31:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:52.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:52.315281+0000) 2022-09-08T08:31:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:52.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:52.119963+0000) 2022-09-08T08:31:52.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:52 smithi055 ceph-mon[97033]: pgmap v21208: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:31:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:52 smithi059 ceph-mon[97450]: pgmap v21208: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:31:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:53.314+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:53.315535+0000) 2022-09-08T08:31:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:53.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:53.120106+0000) 2022-09-08T08:31:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:54.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:54.315752+0000) 2022-09-08T08:31:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:54.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:54.120320+0000) 2022-09-08T08:31:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:54 smithi055 ceph-mon[97033]: pgmap v21209: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:54 smithi059 ceph-mon[97450]: pgmap v21209: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:55.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:55.315960+0000) 2022-09-08T08:31:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:55.119+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:55.120476+0000) 2022-09-08T08:31:55.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:31:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:31:55.582+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:31:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:56.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:56.316194+0000) 2022-09-08T08:31:56.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:56.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:56.120626+0000) 2022-09-08T08:31:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:56 smithi055 ceph-mon[97033]: pgmap v21210: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:31:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:56 smithi059 ceph-mon[97450]: pgmap v21210: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:31:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:57.315+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:57.316401+0000) 2022-09-08T08:31:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:57.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:57.120840+0000) 2022-09-08T08:31:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:58.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:58.316611+0000) 2022-09-08T08:31:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:58.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:58.121089+0000) 2022-09-08T08:31:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:58 smithi055 ceph-mon[97033]: pgmap v21211: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:58 smithi059 ceph-mon[97450]: pgmap v21211: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:31:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:31:59.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:59.316877+0000) 2022-09-08T08:31:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:31:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:31:59.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:31:59.121304+0000) 2022-09-08T08:32:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:31:59 smithi055 ceph-mon[97033]: pgmap v21212: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:32:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:31:59 smithi059 ceph-mon[97450]: pgmap v21212: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:32:00.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:00.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:00.317059+0000) 2022-09-08T08:32:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:00.120+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:00.121553+0000) 2022-09-08T08:32:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:01.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:01.317310+0000) 2022-09-08T08:32:01.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:01.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:01.121752+0000) 2022-09-08T08:32:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:02.316+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:02.317494+0000) 2022-09-08T08:32:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:02.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:02.121994+0000) 2022-09-08T08:32:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:02 smithi055 ceph-mon[97033]: pgmap v21213: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:32:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:02 smithi059 ceph-mon[97450]: pgmap v21213: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:32:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:03.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:03.317715+0000) 2022-09-08T08:32:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:03.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:03.122170+0000) 2022-09-08T08:32:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:04.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:04.317931+0000) 2022-09-08T08:32:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:04.121+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:04.122350+0000) 2022-09-08T08:32:04.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:04 smithi055 ceph-mon[97033]: pgmap v21214: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:32:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:04 smithi059 ceph-mon[97450]: pgmap v21214: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:32:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:05.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:05.318070+0000) 2022-09-08T08:32:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:05.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:05.122541+0000) 2022-09-08T08:32:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:06.317+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:06.318299+0000) 2022-09-08T08:32:06.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:06.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:06.122725+0000) 2022-09-08T08:32:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:06 smithi055 ceph-mon[97033]: pgmap v21215: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:32:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:32:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:06 smithi059 ceph-mon[97450]: pgmap v21215: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:32:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:32:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:07.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:07.318542+0000) 2022-09-08T08:32:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:07.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:07.122894+0000) 2022-09-08T08:32:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:32:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:32:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:08.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:08.318803+0000) 2022-09-08T08:32:08.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:08.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:08.123028+0000) 2022-09-08T08:32:08.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:08 smithi055 ceph-mon[97033]: pgmap v21216: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:32:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:08 smithi059 ceph-mon[97450]: pgmap v21216: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:32:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:09.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:09.319062+0000) 2022-09-08T08:32:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:09.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:09.123195+0000) 2022-09-08T08:32:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:09 smithi055 ceph-mon[97033]: pgmap v21217: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:32:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:09 smithi059 ceph-mon[97450]: pgmap v21217: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:32:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:10.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:10.319272+0000) 2022-09-08T08:32:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:10.122+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:10.123336+0000) 2022-09-08T08:32:10.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:32:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:32:10.595+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:32:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:10 smithi055 ceph-mon[97033]: osdmap e1404: 12 total, 11 up, 11 in 2022-09-08T08:32:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:10 smithi059 ceph-mon[97450]: osdmap e1404: 12 total, 11 up, 11 in 2022-09-08T08:32:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:11.318+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:11.319469+0000) 2022-09-08T08:32:11.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:11.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:11.123573+0000) 2022-09-08T08:32:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:11 smithi055 ceph-mon[97033]: pgmap v21219: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:32:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:11 smithi059 ceph-mon[97450]: pgmap v21219: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:32:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:12.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:12.319708+0000) 2022-09-08T08:32:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:12.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:12.123713+0000) 2022-09-08T08:32:13.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:13.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:13.123855+0000) 2022-09-08T08:32:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:13.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:13.319932+0000) 2022-09-08T08:32:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:14.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:14.320212+0000) 2022-09-08T08:32:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:14.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:14.124002+0000) 2022-09-08T08:32:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:14 smithi055 ceph-mon[97033]: pgmap v21220: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:32:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:14 smithi059 ceph-mon[97450]: pgmap v21220: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:32:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:15.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:15.320468+0000) 2022-09-08T08:32:15.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:15.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:15.124219+0000) 2022-09-08T08:32:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:16.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:16.320699+0000) 2022-09-08T08:32:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:16.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:16.124483+0000) 2022-09-08T08:32:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:16 smithi055 ceph-mon[97033]: pgmap v21221: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:32:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:32:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:32:16.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:32:16.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:32:16.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:16 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:32:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:16 smithi059 ceph-mon[97450]: pgmap v21221: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:32:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:32:16.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:32:16.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:32:16.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:32:16.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:16 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:32:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:17.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:17.320870+0000) 2022-09-08T08:32:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:17.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:17.124682+0000) 2022-09-08T08:32:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:18.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:18.321065+0000) 2022-09-08T08:32:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:18.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:18.124899+0000) 2022-09-08T08:32:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:18 smithi055 ceph-mon[97033]: pgmap v21222: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:32:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:18 smithi059 ceph-mon[97450]: pgmap v21222: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:32:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:19.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:19.321242+0000) 2022-09-08T08:32:19.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:19.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:19.125104+0000) 2022-09-08T08:32:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:19 smithi055 ceph-mon[97033]: osdmap e1405: 12 total, 11 up, 11 in 2022-09-08T08:32:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:19 smithi055 ceph-mon[97033]: pgmap v21224: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-08T08:32:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:19 smithi059 ceph-mon[97450]: osdmap e1405: 12 total, 11 up, 11 in 2022-09-08T08:32:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:19 smithi059 ceph-mon[97450]: pgmap v21224: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2022-09-08T08:32:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:20.319+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:20.321406+0000) 2022-09-08T08:32:20.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:20.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:20.125261+0000) 2022-09-08T08:32:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:21.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:21.321629+0000) 2022-09-08T08:32:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:21.123+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:21.125418+0000) 2022-09-08T08:32:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:22.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:22.125585+0000) 2022-09-08T08:32:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:22.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:22.321798+0000) 2022-09-08T08:32:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:22 smithi055 ceph-mon[97033]: pgmap v21225: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:32:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:22 smithi059 ceph-mon[97450]: pgmap v21225: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:32:23.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:23.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:23.321932+0000) 2022-09-08T08:32:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:23.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:23.125760+0000) 2022-09-08T08:32:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:24.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:24.322140+0000) 2022-09-08T08:32:24.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:24.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:24.125933+0000) 2022-09-08T08:32:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:24 smithi055 ceph-mon[97033]: pgmap v21226: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:32:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:24 smithi059 ceph-mon[97450]: pgmap v21226: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:32:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:25.320+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:25.322321+0000) 2022-09-08T08:32:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:25.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:25.126093+0000) 2022-09-08T08:32:25.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:32:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:32:25.599+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:32:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:26.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:26.322548+0000) 2022-09-08T08:32:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:26.124+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:26.126244+0000) 2022-09-08T08:32:26.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:26 smithi055 ceph-mon[97033]: pgmap v21227: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:32:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:26 smithi059 ceph-mon[97450]: pgmap v21227: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:32:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:27.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:27.322736+0000) 2022-09-08T08:32:27.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:27.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:27.126471+0000) 2022-09-08T08:32:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:28.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:28.322955+0000) 2022-09-08T08:32:28.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:28.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:28.126620+0000) 2022-09-08T08:32:28.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:28 smithi055 ceph-mon[97033]: pgmap v21228: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:32:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:28 smithi059 ceph-mon[97450]: pgmap v21228: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:32:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:29.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:29.126802+0000) 2022-09-08T08:32:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:29.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:29.323171+0000) 2022-09-08T08:32:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:29 smithi055 ceph-mon[97033]: pgmap v21229: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:32:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:29 smithi059 ceph-mon[97450]: pgmap v21229: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:32:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:30.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:30.323379+0000) 2022-09-08T08:32:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:30.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:30.126946+0000) 2022-09-08T08:32:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:31.321+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:31.323581+0000) 2022-09-08T08:32:31.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:31.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:31.127166+0000) 2022-09-08T08:32:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:32.322+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:32.323825+0000) 2022-09-08T08:32:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:32.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:32.127329+0000) 2022-09-08T08:32:32.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:32 smithi055 ceph-mon[97033]: pgmap v21230: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:32:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:32 smithi059 ceph-mon[97450]: pgmap v21230: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:32:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:33.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:33.324086+0000) 2022-09-08T08:32:33.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:33.125+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:33.127473+0000) 2022-09-08T08:32:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:34.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:34.324373+0000) 2022-09-08T08:32:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:34.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:34.127615+0000) 2022-09-08T08:32:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:34 smithi055 ceph-mon[97033]: pgmap v21231: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:32:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:34 smithi059 ceph-mon[97450]: pgmap v21231: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:32:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:35.323+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:35.324624+0000) 2022-09-08T08:32:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:35.126+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:35.127801+0000) 2022-09-08T08:32:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:36.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:36.324925+0000) 2022-09-08T08:32:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:36.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:36.127994+0000) 2022-09-08T08:32:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:36 smithi055 ceph-mon[97033]: pgmap v21232: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:32:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:36 smithi059 ceph-mon[97450]: pgmap v21232: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:32:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:37.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:37.325144+0000) 2022-09-08T08:32:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:37.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:37.128193+0000) 2022-09-08T08:32:38.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:38.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:38.325383+0000) 2022-09-08T08:32:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:38.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:38.128364+0000) 2022-09-08T08:32:38.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:38 smithi055 ceph-mon[97033]: pgmap v21233: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:32:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:38 smithi059 ceph-mon[97450]: pgmap v21233: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:32:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:39.324+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:39.325615+0000) 2022-09-08T08:32:39.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:39.127+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:39.128530+0000) 2022-09-08T08:32:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:39 smithi055 ceph-mon[97033]: pgmap v21234: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:32:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:39 smithi059 ceph-mon[97450]: pgmap v21234: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:32:40.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:40.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:40.325803+0000) 2022-09-08T08:32:40.406 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:40.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:40.128756+0000) 2022-09-08T08:32:40.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:32:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:32:40.606+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:32:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:41.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:41.325919+0000) 2022-09-08T08:32:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:41.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:41.128947+0000) 2022-09-08T08:32:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:42.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:42.326109+0000) 2022-09-08T08:32:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:42.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:42.129124+0000) 2022-09-08T08:32:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:42 smithi055 ceph-mon[97033]: pgmap v21235: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:32:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:42 smithi059 ceph-mon[97450]: pgmap v21235: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:32:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:43.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:43.326331+0000) 2022-09-08T08:32:43.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:43.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:43.129350+0000) 2022-09-08T08:32:44.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:44.325+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:44.326552+0000) 2022-09-08T08:32:44.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:44.128+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:44.129561+0000) 2022-09-08T08:32:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:44 smithi055 ceph-mon[97033]: pgmap v21236: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:32:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:44 smithi059 ceph-mon[97450]: pgmap v21236: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:32:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:45.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:45.326728+0000) 2022-09-08T08:32:45.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:45.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:45.129746+0000) 2022-09-08T08:32:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:46.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:46.326862+0000) 2022-09-08T08:32:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:46.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:46.129977+0000) 2022-09-08T08:32:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:46 smithi059 ceph-mon[97450]: pgmap v21237: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:32:46.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:46 smithi059 ceph-mon[97450]: pgmap v21238: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:32:46.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:46 smithi059 ceph-mon[97450]: pgmap v21239: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:32:46.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:32:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:46 smithi055 ceph-mon[97033]: pgmap v21237: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:32:46.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:46 smithi055 ceph-mon[97033]: pgmap v21238: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:32:46.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:46 smithi055 ceph-mon[97033]: pgmap v21239: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:32:46.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:32:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:47.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:47.327061+0000) 2022-09-08T08:32:47.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:47.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:47.130148+0000) 2022-09-08T08:32:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:48.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:48.130288+0000) 2022-09-08T08:32:48.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:48.326+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:48.327319+0000) 2022-09-08T08:32:48.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:48 smithi055 ceph-mon[97033]: pgmap v21240: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:32:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:48 smithi059 ceph-mon[97450]: pgmap v21240: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:32:49.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:49.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:49.327552+0000) 2022-09-08T08:32:49.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:49.129+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:49.130450+0000) 2022-09-08T08:32:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:49 smithi055 ceph-mon[97033]: pgmap v21241: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:32:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:49 smithi059 ceph-mon[97450]: pgmap v21241: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:32:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:50.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:50.327829+0000) 2022-09-08T08:32:50.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:50.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:50.130675+0000) 2022-09-08T08:32:51.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:51.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:51.327990+0000) 2022-09-08T08:32:51.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:51.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:51.130859+0000) 2022-09-08T08:32:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:52.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:52.328248+0000) 2022-09-08T08:32:52.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:52.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:52.131016+0000) 2022-09-08T08:32:52.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:52 smithi055 ceph-mon[97033]: pgmap v21242: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:32:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:52 smithi059 ceph-mon[97450]: pgmap v21242: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:32:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:53.327+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:53.328460+0000) 2022-09-08T08:32:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:53.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:53.131172+0000) 2022-09-08T08:32:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:54.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:54.328672+0000) 2022-09-08T08:32:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:54.130+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:54.131429+0000) 2022-09-08T08:32:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:54 smithi055 ceph-mon[97033]: pgmap v21243: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:32:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:54 smithi059 ceph-mon[97450]: pgmap v21243: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:32:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:55.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:55.328894+0000) 2022-09-08T08:32:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:55.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:55.131695+0000) 2022-09-08T08:32:55.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:32:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:32:55.609+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:32:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:56.328+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:56.329131+0000) 2022-09-08T08:32:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:56.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:56.131858+0000) 2022-09-08T08:32:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:56 smithi055 ceph-mon[97033]: pgmap v21244: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:32:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:56 smithi059 ceph-mon[97450]: pgmap v21244: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:32:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:57.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:57.329405+0000) 2022-09-08T08:32:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:57.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:57.132039+0000) 2022-09-08T08:32:58.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:58.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:58.329625+0000) 2022-09-08T08:32:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:58.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:58.132276+0000) 2022-09-08T08:32:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:58 smithi055 ceph-mon[97033]: pgmap v21245: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:32:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:58 smithi059 ceph-mon[97450]: pgmap v21245: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:32:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:32:59.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:59.329950+0000) 2022-09-08T08:32:59.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:32:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:32:59.131+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:32:59.132427+0000) 2022-09-08T08:33:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:32:59 smithi055 ceph-mon[97033]: pgmap v21246: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:32:59 smithi059 ceph-mon[97450]: pgmap v21246: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:00.329+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:00.330154+0000) 2022-09-08T08:33:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:00.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:00.132630+0000) 2022-09-08T08:33:01.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:01.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:01.330395+0000) 2022-09-08T08:33:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:01.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:01.132796+0000) 2022-09-08T08:33:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:02.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:02.330620+0000) 2022-09-08T08:33:02.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:02.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:02.132954+0000) 2022-09-08T08:33:02.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:02 smithi055 ceph-mon[97033]: pgmap v21247: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:33:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:02 smithi059 ceph-mon[97450]: pgmap v21247: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:33:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:03.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:03.330903+0000) 2022-09-08T08:33:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:03.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:03.133108+0000) 2022-09-08T08:33:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:04.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:04.331134+0000) 2022-09-08T08:33:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:04.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:04.133334+0000) 2022-09-08T08:33:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:04 smithi055 ceph-mon[97033]: pgmap v21248: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:04 smithi059 ceph-mon[97450]: pgmap v21248: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:05.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:05.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:05.331358+0000) 2022-09-08T08:33:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:05.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:05.133474+0000) 2022-09-08T08:33:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:06.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:06.331534+0000) 2022-09-08T08:33:06.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:06.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:06.133665+0000) 2022-09-08T08:33:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:06 smithi055 ceph-mon[97033]: pgmap v21249: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:33:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:33:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:06 smithi059 ceph-mon[97450]: pgmap v21249: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:33:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:33:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:07.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:07.331773+0000) 2022-09-08T08:33:07.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:07.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:07.133840+0000) 2022-09-08T08:33:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:33:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:33:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:08.330+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:08.331996+0000) 2022-09-08T08:33:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:08.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:08.133981+0000) 2022-09-08T08:33:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:08 smithi055 ceph-mon[97033]: pgmap v21250: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:33:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:08 smithi059 ceph-mon[97450]: pgmap v21250: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:33:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:09.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:09.332234+0000) 2022-09-08T08:33:09.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:09.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:09.134181+0000) 2022-09-08T08:33:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:09 smithi055 ceph-mon[97033]: pgmap v21251: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:33:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:09 smithi059 ceph-mon[97450]: pgmap v21251: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:33:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:10.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:10.332459+0000) 2022-09-08T08:33:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:10.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:10.134333+0000) 2022-09-08T08:33:10.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:33:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:33:10.614+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:33:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:10 smithi055 ceph-mon[97033]: osdmap e1406: 12 total, 11 up, 11 in 2022-09-08T08:33:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:10 smithi059 ceph-mon[97450]: osdmap e1406: 12 total, 11 up, 11 in 2022-09-08T08:33:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:11.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:11.332702+0000) 2022-09-08T08:33:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:11.132+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:11.134472+0000) 2022-09-08T08:33:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:11 smithi055 ceph-mon[97033]: pgmap v21253: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:33:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:11 smithi059 ceph-mon[97450]: pgmap v21253: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:33:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:12.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:12.332856+0000) 2022-09-08T08:33:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:12.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:12.134747+0000) 2022-09-08T08:33:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:13.331+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:13.333101+0000) 2022-09-08T08:33:13.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:13.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:13.134941+0000) 2022-09-08T08:33:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:14.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:14.333338+0000) 2022-09-08T08:33:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:14.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:14.135127+0000) 2022-09-08T08:33:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:14 smithi055 ceph-mon[97033]: pgmap v21254: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:33:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:14 smithi059 ceph-mon[97450]: pgmap v21254: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:33:15.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:15.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:15.333547+0000) 2022-09-08T08:33:15.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:15.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:15.135258+0000) 2022-09-08T08:33:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:16.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:16.333755+0000) 2022-09-08T08:33:16.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:16.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:16.135378+0000) 2022-09-08T08:33:16.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:16 smithi055 ceph-mon[97033]: pgmap v21255: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:33:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:16 smithi059 ceph-mon[97450]: pgmap v21255: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:33:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:17.133+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:17.135550+0000) 2022-09-08T08:33:17.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:17.332+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:17.333917+0000) 2022-09-08T08:33:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:18.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:18.334166+0000) 2022-09-08T08:33:18.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:18.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:18.135794+0000) 2022-09-08T08:33:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:18 smithi055 ceph-mon[97033]: pgmap v21256: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:33:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:18 smithi059 ceph-mon[97450]: pgmap v21256: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:33:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:19.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:19.334418+0000) 2022-09-08T08:33:19.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:19.134+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:19.136048+0000) 2022-09-08T08:33:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:19 smithi055 ceph-mon[97033]: osdmap e1407: 12 total, 11 up, 11 in 2022-09-08T08:33:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:19 smithi055 ceph-mon[97033]: pgmap v21258: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2022-09-08T08:33:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:19 smithi059 ceph-mon[97450]: osdmap e1407: 12 total, 11 up, 11 in 2022-09-08T08:33:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:19 smithi059 ceph-mon[97450]: pgmap v21258: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2022-09-08T08:33:20.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:20.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:20.334632+0000) 2022-09-08T08:33:20.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:20.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:20.136249+0000) 2022-09-08T08:33:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:21.333+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:21.334780+0000) 2022-09-08T08:33:21.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:21.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:21.136370+0000) 2022-09-08T08:33:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:22.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:22.334945+0000) 2022-09-08T08:33:22.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:22.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:22.136518+0000) 2022-09-08T08:33:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:22 smithi055 ceph-mon[97033]: pgmap v21259: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:33:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:22 smithi059 ceph-mon[97450]: pgmap v21259: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:33:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:23.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:23.335090+0000) 2022-09-08T08:33:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:23.135+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:23.136755+0000) 2022-09-08T08:33:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:24.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:24.335318+0000) 2022-09-08T08:33:24.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:24.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:24.137021+0000) 2022-09-08T08:33:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:24 smithi055 ceph-mon[97033]: pgmap v21260: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:33:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:24 smithi059 ceph-mon[97450]: pgmap v21260: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:33:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:25.334+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:25.335595+0000) 2022-09-08T08:33:25.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:25.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:25.137204+0000) 2022-09-08T08:33:25.903 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:33:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:33:25.631+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:33:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:26.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:26.335836+0000) 2022-09-08T08:33:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:26.136+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:26.137421+0000) 2022-09-08T08:33:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:26 smithi055 ceph-mon[97033]: pgmap v21261: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:33:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:26 smithi059 ceph-mon[97450]: pgmap v21261: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:33:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:27.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:27.336034+0000) 2022-09-08T08:33:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:27.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:27.137623+0000) 2022-09-08T08:33:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:28.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:28.336225+0000) 2022-09-08T08:33:28.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:28.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:28.137805+0000) 2022-09-08T08:33:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:28 smithi055 ceph-mon[97033]: pgmap v21262: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:33:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:28 smithi059 ceph-mon[97450]: pgmap v21262: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:33:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:29.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:29.336376+0000) 2022-09-08T08:33:29.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:29.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:29.137939+0000) 2022-09-08T08:33:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:29 smithi055 ceph-mon[97033]: pgmap v21263: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:33:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:29 smithi059 ceph-mon[97450]: pgmap v21263: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:33:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:30.335+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:30.336514+0000) 2022-09-08T08:33:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:30.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:30.138072+0000) 2022-09-08T08:33:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:31.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:31.138260+0000) 2022-09-08T08:33:31.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:31.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:31.336698+0000) 2022-09-08T08:33:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:32.137+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:32.138452+0000) 2022-09-08T08:33:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:32.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:32.336884+0000) 2022-09-08T08:33:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:32 smithi055 ceph-mon[97033]: pgmap v21264: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:33:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:32 smithi059 ceph-mon[97450]: pgmap v21264: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:33:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:33.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:33.337100+0000) 2022-09-08T08:33:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:33.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:33.138629+0000) 2022-09-08T08:33:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:34.336+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:34.337372+0000) 2022-09-08T08:33:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:34.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:34.138864+0000) 2022-09-08T08:33:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:34 smithi055 ceph-mon[97033]: pgmap v21265: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:34 smithi059 ceph-mon[97450]: pgmap v21265: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:35.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:35.337626+0000) 2022-09-08T08:33:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:35.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:35.139048+0000) 2022-09-08T08:33:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:36.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:36.337841+0000) 2022-09-08T08:33:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:36.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:36.139248+0000) 2022-09-08T08:33:36.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:36 smithi055 ceph-mon[97033]: pgmap v21266: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:33:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:36 smithi059 ceph-mon[97450]: pgmap v21266: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:33:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:37.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:37.337975+0000) 2022-09-08T08:33:37.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:37.138+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:37.139424+0000) 2022-09-08T08:33:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:38.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:38.338221+0000) 2022-09-08T08:33:38.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:38.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:38.139622+0000) 2022-09-08T08:33:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:38 smithi055 ceph-mon[97033]: pgmap v21267: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:38 smithi059 ceph-mon[97450]: pgmap v21267: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:39.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:39.337+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:39.338496+0000) 2022-09-08T08:33:39.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:39.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:39.139802+0000) 2022-09-08T08:33:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:39 smithi055 ceph-mon[97033]: pgmap v21268: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:39 smithi059 ceph-mon[97450]: pgmap v21268: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:40.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:40.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:40.338725+0000) 2022-09-08T08:33:40.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:40.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:40.140010+0000) 2022-09-08T08:33:40.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:33:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:33:40.646+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:33:41.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:41.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:41.338966+0000) 2022-09-08T08:33:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:41.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:41.140253+0000) 2022-09-08T08:33:42.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:42.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:42.339183+0000) 2022-09-08T08:33:42.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:42.139+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:42.140509+0000) 2022-09-08T08:33:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:42 smithi055 ceph-mon[97033]: pgmap v21269: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:33:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:42 smithi059 ceph-mon[97450]: pgmap v21269: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:33:43.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:43.338+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:43.339413+0000) 2022-09-08T08:33:43.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:43.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:43.140672+0000) 2022-09-08T08:33:44.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:44.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:44.339662+0000) 2022-09-08T08:33:44.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:44.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:44.140880+0000) 2022-09-08T08:33:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:44 smithi055 ceph-mon[97033]: pgmap v21270: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:44 smithi059 ceph-mon[97450]: pgmap v21270: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:45.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:45.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:45.339822+0000) 2022-09-08T08:33:45.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:45.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:45.141063+0000) 2022-09-08T08:33:46.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:46.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:46.340055+0000) 2022-09-08T08:33:46.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:46.140+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:46.141301+0000) 2022-09-08T08:33:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:46 smithi055 ceph-mon[97033]: pgmap v21271: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:33:46.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:33:46.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:33:46.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:46 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:33:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:46 smithi059 ceph-mon[97450]: pgmap v21271: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:33:46.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:33:46.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:33:46.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:46 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:33:47.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:47.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:47.340264+0000) 2022-09-08T08:33:47.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:47.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:47.141564+0000) 2022-09-08T08:33:48.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:48.339+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:48.340484+0000) 2022-09-08T08:33:48.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:48.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:48.141788+0000) 2022-09-08T08:33:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:48 smithi055 ceph-mon[97033]: pgmap v21272: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:48 smithi059 ceph-mon[97450]: pgmap v21272: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:49.411 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:49.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:49.340739+0000) 2022-09-08T08:33:49.411 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:49.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:49.141933+0000) 2022-09-08T08:33:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:49 smithi055 ceph-mon[97033]: pgmap v21273: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:49 smithi059 ceph-mon[97450]: pgmap v21273: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:50.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:50.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:50.340893+0000) 2022-09-08T08:33:50.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:50.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:50.142088+0000) 2022-09-08T08:33:51.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:51.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:51.341018+0000) 2022-09-08T08:33:51.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:51.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:51.142281+0000) 2022-09-08T08:33:52.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:52.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:52.142475+0000) 2022-09-08T08:33:52.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:52.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:52.341235+0000) 2022-09-08T08:33:52.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:52 smithi055 ceph-mon[97033]: pgmap v21274: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:33:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:52 smithi059 ceph-mon[97450]: pgmap v21274: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:33:53.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:53.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:53.341443+0000) 2022-09-08T08:33:53.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:53.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:53.142727+0000) 2022-09-08T08:33:54.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:54.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:54.143002+0000) 2022-09-08T08:33:54.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:54.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:54.341683+0000) 2022-09-08T08:33:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:54 smithi055 ceph-mon[97033]: pgmap v21275: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:54 smithi059 ceph-mon[97450]: pgmap v21275: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:55.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:55.340+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:55.341924+0000) 2022-09-08T08:33:55.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:55.141+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:55.143275+0000) 2022-09-08T08:33:56.146 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:33:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:33:55.661+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:33:56.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:56.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:56.342142+0000) 2022-09-08T08:33:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:56.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:56.143417+0000) 2022-09-08T08:33:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:56 smithi055 ceph-mon[97033]: pgmap v21276: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:33:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:56 smithi059 ceph-mon[97450]: pgmap v21276: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:33:57.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:57.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:57.342373+0000) 2022-09-08T08:33:57.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:57.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:57.143621+0000) 2022-09-08T08:33:58.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:58.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:58.342661+0000) 2022-09-08T08:33:58.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:58.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:58.143835+0000) 2022-09-08T08:33:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:58 smithi055 ceph-mon[97033]: pgmap v21277: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:58 smithi059 ceph-mon[97450]: pgmap v21277: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:33:59.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:33:59.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:59.342870+0000) 2022-09-08T08:33:59.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:33:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:33:59.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:33:59.144047+0000) 2022-09-08T08:34:00.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:33:59 smithi055 ceph-mon[97033]: pgmap v21278: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:33:59 smithi059 ceph-mon[97450]: pgmap v21278: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:00.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:00.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:00.343058+0000) 2022-09-08T08:34:00.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:00.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:00.144201+0000) 2022-09-08T08:34:01.403 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:01.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:01.343274+0000) 2022-09-08T08:34:01.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:01.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:01.144389+0000) 2022-09-08T08:34:02.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:02.341+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:02.343539+0000) 2022-09-08T08:34:02.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:02.142+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:02.144608+0000) 2022-09-08T08:34:02.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:02 smithi055 ceph-mon[97033]: pgmap v21279: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:34:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:02 smithi059 ceph-mon[97450]: pgmap v21279: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:34:03.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:03.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:03.343788+0000) 2022-09-08T08:34:03.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:03.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:03.144821+0000) 2022-09-08T08:34:04.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:04.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:04.343984+0000) 2022-09-08T08:34:04.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:04.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:04.145037+0000) 2022-09-08T08:34:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:04 smithi055 ceph-mon[97033]: pgmap v21280: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:04 smithi059 ceph-mon[97450]: pgmap v21280: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:05.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:05.143+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:05.145179+0000) 2022-09-08T08:34:05.405 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:05.342+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:05.344192+0000) 2022-09-08T08:34:06.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:06.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:06.344445+0000) 2022-09-08T08:34:06.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:06.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:06.145317+0000) 2022-09-08T08:34:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:06 smithi055 ceph-mon[97033]: pgmap v21281: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:34:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:34:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:06 smithi059 ceph-mon[97450]: pgmap v21281: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:34:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:34:07.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:07.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:07.344675+0000) 2022-09-08T08:34:07.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:07.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:07.145546+0000) 2022-09-08T08:34:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:34:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:34:08.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:08.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:08.344897+0000) 2022-09-08T08:34:08.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:08.144+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:08.145728+0000) 2022-09-08T08:34:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:08 smithi055 ceph-mon[97033]: pgmap v21282: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:34:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:08 smithi059 ceph-mon[97450]: pgmap v21282: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:34:09.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:09.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:09.345107+0000) 2022-09-08T08:34:09.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:09.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:09.145938+0000) 2022-09-08T08:34:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:09 smithi055 ceph-mon[97033]: pgmap v21283: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:34:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:09 smithi059 ceph-mon[97450]: pgmap v21283: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:34:10.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:10.343+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:10.345296+0000) 2022-09-08T08:34:10.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:10.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:10.146090+0000) 2022-09-08T08:34:10.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:34:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:34:10.667+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:34:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:10 smithi055 ceph-mon[97033]: osdmap e1408: 12 total, 11 up, 11 in 2022-09-08T08:34:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:10 smithi059 ceph-mon[97450]: osdmap e1408: 12 total, 11 up, 11 in 2022-09-08T08:34:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:11.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:11.146220+0000) 2022-09-08T08:34:11.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:11.344+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:11.345469+0000) 2022-09-08T08:34:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:11 smithi055 ceph-mon[97033]: pgmap v21285: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:34:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:11 smithi059 ceph-mon[97450]: pgmap v21285: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:34:12.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:12.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:12.345763+0000) 2022-09-08T08:34:12.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:12.145+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:12.146448+0000) 2022-09-08T08:34:13.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:13.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:13.345974+0000) 2022-09-08T08:34:13.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:13.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:13.146688+0000) 2022-09-08T08:34:14.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:14.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:14.346225+0000) 2022-09-08T08:34:14.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:14.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:14.146837+0000) 2022-09-08T08:34:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:14 smithi055 ceph-mon[97033]: pgmap v21286: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:34:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:14 smithi059 ceph-mon[97450]: pgmap v21286: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:34:15.349 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:15.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:15.147055+0000) 2022-09-08T08:34:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:15.345+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:15.346482+0000) 2022-09-08T08:34:16.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:16.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:16.346732+0000) 2022-09-08T08:34:16.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:16.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:16.147263+0000) 2022-09-08T08:34:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:16 smithi055 ceph-mon[97033]: pgmap v21287: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:34:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:16 smithi059 ceph-mon[97450]: pgmap v21287: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:34:17.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:17.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:17.346875+0000) 2022-09-08T08:34:17.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:17.146+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:17.147502+0000) 2022-09-08T08:34:18.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:18.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:18.347071+0000) 2022-09-08T08:34:18.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:18.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:18.147730+0000) 2022-09-08T08:34:18.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:18 smithi055 ceph-mon[97033]: pgmap v21288: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:34:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:18 smithi059 ceph-mon[97450]: pgmap v21288: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:34:19.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:19.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:19.347223+0000) 2022-09-08T08:34:19.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:19.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:19.147883+0000) 2022-09-08T08:34:20.112 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:19 smithi059 ceph-mon[97450]: osdmap e1409: 12 total, 11 up, 11 in 2022-09-08T08:34:20.113 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:19 smithi059 ceph-mon[97450]: pgmap v21290: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2022-09-08T08:34:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:19 smithi055 ceph-mon[97033]: osdmap e1409: 12 total, 11 up, 11 in 2022-09-08T08:34:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:19 smithi055 ceph-mon[97033]: pgmap v21290: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2022-09-08T08:34:20.350 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:20.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:20.148088+0000) 2022-09-08T08:34:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:20.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:20.347439+0000) 2022-09-08T08:34:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:34:21.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:34:21.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:21 smithi059 ceph-mon[97450]: pgmap v21291: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T08:34:21.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:34:21.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:21.346+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:21.347593+0000) 2022-09-08T08:34:21.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:21.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:21.148256+0000) 2022-09-08T08:34:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:34:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:34:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:21 smithi055 ceph-mon[97033]: pgmap v21291: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T08:34:21.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:34:22.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:22.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:22.347844+0000) 2022-09-08T08:34:22.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:22.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:22.148456+0000) 2022-09-08T08:34:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:23 smithi059 ceph-mon[97450]: pgmap v21292: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T08:34:23.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:23.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:23.348053+0000) 2022-09-08T08:34:23.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:23.147+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:23.148620+0000) 2022-09-08T08:34:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:23 smithi055 ceph-mon[97033]: pgmap v21292: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T08:34:24.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:24.347+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:24.348279+0000) 2022-09-08T08:34:24.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:24.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:24.148833+0000) 2022-09-08T08:34:25.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:24 smithi055 ceph-mon[97033]: pgmap v21293: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T08:34:25.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:24 smithi059 ceph-mon[97450]: pgmap v21293: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T08:34:25.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:25.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:25.348526+0000) 2022-09-08T08:34:25.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:25.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:25.148997+0000) 2022-09-08T08:34:26.151 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:34:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:34:25.680+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:34:26.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:26.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:26.348803+0000) 2022-09-08T08:34:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:26.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:26.149153+0000) 2022-09-08T08:34:27.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:27.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:27.348989+0000) 2022-09-08T08:34:27.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:27.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:27.149292+0000) 2022-09-08T08:34:27.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:27 smithi055 ceph-mon[97033]: pgmap v21294: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T08:34:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:27 smithi059 ceph-mon[97450]: pgmap v21294: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T08:34:28.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:28.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:28.349196+0000) 2022-09-08T08:34:28.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:28.148+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:28.149422+0000) 2022-09-08T08:34:29.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:29.348+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:29.349446+0000) 2022-09-08T08:34:29.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:29.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:29.149580+0000) 2022-09-08T08:34:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:29 smithi055 ceph-mon[97033]: pgmap v21295: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:34:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:29 smithi059 ceph-mon[97450]: pgmap v21295: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:34:30.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:30.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:30.349763+0000) 2022-09-08T08:34:30.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:30.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:30.149782+0000) 2022-09-08T08:34:31.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:31.349+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:31.350059+0000) 2022-09-08T08:34:31.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:31.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:31.149990+0000) 2022-09-08T08:34:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:31 smithi055 ceph-mon[97033]: pgmap v21296: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T08:34:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:31 smithi059 ceph-mon[97450]: pgmap v21296: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-08T08:34:32.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:32.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:32.350290+0000) 2022-09-08T08:34:32.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:32.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:32.150184+0000) 2022-09-08T08:34:33.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:33.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:33.350518+0000) 2022-09-08T08:34:33.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:33.149+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:33.150427+0000) 2022-09-08T08:34:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:33 smithi055 ceph-mon[97033]: pgmap v21297: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:33 smithi059 ceph-mon[97450]: pgmap v21297: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:34.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:34.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:34.350767+0000) 2022-09-08T08:34:34.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:34.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:34.150631+0000) 2022-09-08T08:34:35.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:34 smithi055 ceph-mon[97033]: pgmap v21298: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:35.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:34 smithi059 ceph-mon[97450]: pgmap v21298: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:35.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:35.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:35.350984+0000) 2022-09-08T08:34:35.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:35.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:35.150794+0000) 2022-09-08T08:34:36.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:36.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:36.351197+0000) 2022-09-08T08:34:36.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:36.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:36.150929+0000) 2022-09-08T08:34:37.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:37.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:37.351396+0000) 2022-09-08T08:34:37.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:37.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:37.151071+0000) 2022-09-08T08:34:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:37 smithi055 ceph-mon[97033]: pgmap v21299: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:34:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:37 smithi059 ceph-mon[97450]: pgmap v21299: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:34:38.404 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:38.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:38.351556+0000) 2022-09-08T08:34:38.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:38.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:38.151202+0000) 2022-09-08T08:34:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:39 smithi055 ceph-mon[97033]: pgmap v21300: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:39 smithi059 ceph-mon[97450]: pgmap v21300: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:39.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:39.351712+0000) 2022-09-08T08:34:39.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:39.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:39.151339+0000) 2022-09-08T08:34:40.354 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:40.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:40.151535+0000) 2022-09-08T08:34:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:40.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:40.351848+0000) 2022-09-08T08:34:41.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:34:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:34:40.689+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:34:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:41 smithi055 ceph-mon[97033]: pgmap v21301: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:34:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:41 smithi059 ceph-mon[97450]: pgmap v21301: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:34:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:41.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:41.351999+0000) 2022-09-08T08:34:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:41.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:41.151712+0000) 2022-09-08T08:34:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:42.350+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:42.352162+0000) 2022-09-08T08:34:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:42.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:42.151916+0000) 2022-09-08T08:34:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:43 smithi055 ceph-mon[97033]: pgmap v21302: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:43 smithi059 ceph-mon[97450]: pgmap v21302: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:43.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:43.352363+0000) 2022-09-08T08:34:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:43.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:43.152043+0000) 2022-09-08T08:34:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:44.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:44.352568+0000) 2022-09-08T08:34:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:44.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:44.152183+0000) 2022-09-08T08:34:45.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:44 smithi055 ceph-mon[97033]: pgmap v21303: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:45.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:44 smithi059 ceph-mon[97450]: pgmap v21303: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:45.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:45.352699+0000) 2022-09-08T08:34:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:45.150+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:45.152366+0000) 2022-09-08T08:34:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:46.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:46.352903+0000) 2022-09-08T08:34:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:46.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:46.152553+0000) 2022-09-08T08:34:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:47 smithi055 ceph-mon[97033]: pgmap v21304: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:34:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:47 smithi059 ceph-mon[97450]: pgmap v21304: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:34:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:47.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:47.353124+0000) 2022-09-08T08:34:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:47.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:47.152700+0000) 2022-09-08T08:34:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:48.351+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:48.353347+0000) 2022-09-08T08:34:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:48.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:48.152882+0000) 2022-09-08T08:34:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:49 smithi055 ceph-mon[97033]: pgmap v21305: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:49 smithi059 ceph-mon[97450]: pgmap v21305: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:49.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:49.353524+0000) 2022-09-08T08:34:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:49.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:49.153041+0000) 2022-09-08T08:34:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:50.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:50.153196+0000) 2022-09-08T08:34:50.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:50.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:50.353770+0000) 2022-09-08T08:34:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:51 smithi055 ceph-mon[97033]: pgmap v21306: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:34:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:51 smithi059 ceph-mon[97450]: pgmap v21306: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:34:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:51.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:51.353901+0000) 2022-09-08T08:34:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:51.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:51.153344+0000) 2022-09-08T08:34:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:52.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:52.354090+0000) 2022-09-08T08:34:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:52.151+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:52.153490+0000) 2022-09-08T08:34:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:53 smithi055 ceph-mon[97033]: pgmap v21307: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:53 smithi059 ceph-mon[97450]: pgmap v21307: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:53.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:53.354322+0000) 2022-09-08T08:34:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:53.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:53.153623+0000) 2022-09-08T08:34:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:54.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:54.153829+0000) 2022-09-08T08:34:54.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:54.352+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:54.354520+0000) 2022-09-08T08:34:55.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:54 smithi055 ceph-mon[97033]: pgmap v21308: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:55.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:54 smithi059 ceph-mon[97450]: pgmap v21308: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:55.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:55.153990+0000) 2022-09-08T08:34:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:55.353+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:55.354778+0000) 2022-09-08T08:34:56.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:34:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:34:55.701+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:34:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:56.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:56.355052+0000) 2022-09-08T08:34:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:56.152+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:56.154126+0000) 2022-09-08T08:34:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:57 smithi055 ceph-mon[97033]: pgmap v21309: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:34:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:57 smithi059 ceph-mon[97450]: pgmap v21309: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:34:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:57.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:57.154261+0000) 2022-09-08T08:34:57.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:57.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:57.355237+0000) 2022-09-08T08:34:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:58.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:58.154391+0000) 2022-09-08T08:34:58.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:58.354+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:58.355433+0000) 2022-09-08T08:34:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:34:59 smithi055 ceph-mon[97033]: pgmap v21310: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:34:59 smithi059 ceph-mon[97450]: pgmap v21310: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:34:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:34:59.153+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:59.154609+0000) 2022-09-08T08:34:59.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:34:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:34:59.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:34:59.355689+0000) 2022-09-08T08:35:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:00.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:00.355916+0000) 2022-09-08T08:35:00.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:00.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:00.154791+0000) 2022-09-08T08:35:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:01 smithi055 ceph-mon[97033]: pgmap v21311: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:35:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:01 smithi059 ceph-mon[97450]: pgmap v21311: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:35:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:01.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:01.356110+0000) 2022-09-08T08:35:01.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:01.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:01.154920+0000) 2022-09-08T08:35:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:02.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:02.356223+0000) 2022-09-08T08:35:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:02.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:02.155112+0000) 2022-09-08T08:35:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:03 smithi055 ceph-mon[97033]: pgmap v21312: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:03 smithi059 ceph-mon[97450]: pgmap v21312: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:03.355+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:03.356429+0000) 2022-09-08T08:35:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:03.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:03.155274+0000) 2022-09-08T08:35:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:04.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:04.356727+0000) 2022-09-08T08:35:04.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:04.154+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:04.155413+0000) 2022-09-08T08:35:05.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:04 smithi055 ceph-mon[97033]: pgmap v21313: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:05.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:04 smithi059 ceph-mon[97450]: pgmap v21313: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:05.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:05.356920+0000) 2022-09-08T08:35:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:05.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:05.155673+0000) 2022-09-08T08:35:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:06.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:06.357094+0000) 2022-09-08T08:35:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:06.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:06.155903+0000) 2022-09-08T08:35:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:07 smithi055 ceph-mon[97033]: pgmap v21314: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:35:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:35:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:35:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:07 smithi059 ceph-mon[97450]: pgmap v21314: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:35:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:35:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:35:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:07.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:07.357288+0000) 2022-09-08T08:35:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:07.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:07.156040+0000) 2022-09-08T08:35:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:08.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:08.156262+0000) 2022-09-08T08:35:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:08.356+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:08.357516+0000) 2022-09-08T08:35:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:09 smithi055 ceph-mon[97033]: pgmap v21315: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:35:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:09 smithi059 ceph-mon[97450]: pgmap v21315: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:35:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:09.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:09.357790+0000) 2022-09-08T08:35:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:09.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:09.156424+0000) 2022-09-08T08:35:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:10.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:10.357995+0000) 2022-09-08T08:35:10.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:10.155+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:10.156547+0000) 2022-09-08T08:35:11.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:35:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:35:10.708+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:35:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:10 smithi055 ceph-mon[97033]: osdmap e1410: 12 total, 11 up, 11 in 2022-09-08T08:35:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:10 smithi055 ceph-mon[97033]: pgmap v21317: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:35:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:10 smithi059 ceph-mon[97450]: osdmap e1410: 12 total, 11 up, 11 in 2022-09-08T08:35:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:10 smithi059 ceph-mon[97450]: pgmap v21317: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:35:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:11.357+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:11.358203+0000) 2022-09-08T08:35:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:11.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:11.156712+0000) 2022-09-08T08:35:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:12.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:12.358443+0000) 2022-09-08T08:35:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:12.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:12.156867+0000) 2022-09-08T08:35:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:13 smithi055 ceph-mon[97033]: pgmap v21318: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:35:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:13 smithi059 ceph-mon[97450]: pgmap v21318: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:35:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:13.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:13.358706+0000) 2022-09-08T08:35:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:13.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:13.157030+0000) 2022-09-08T08:35:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:14.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:14.358832+0000) 2022-09-08T08:35:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:14.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:14.157181+0000) 2022-09-08T08:35:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:14 smithi055 ceph-mon[97033]: pgmap v21319: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:35:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:14 smithi059 ceph-mon[97450]: pgmap v21319: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:35:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:15.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:15.358953+0000) 2022-09-08T08:35:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:15.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:15.157311+0000) 2022-09-08T08:35:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:16.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:16.359162+0000) 2022-09-08T08:35:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:16.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:16.157454+0000) 2022-09-08T08:35:17.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:17 smithi055 ceph-mon[97033]: pgmap v21320: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:35:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:17 smithi059 ceph-mon[97450]: pgmap v21320: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:35:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:17.156+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:17.157594+0000) 2022-09-08T08:35:17.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:17.358+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:17.359384+0000) 2022-09-08T08:35:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:18.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:18.359617+0000) 2022-09-08T08:35:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:18.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:18.157757+0000) 2022-09-08T08:35:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:19 smithi055 ceph-mon[97033]: pgmap v21321: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:35:19.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:19 smithi055 ceph-mon[97033]: osdmap e1411: 12 total, 11 up, 11 in 2022-09-08T08:35:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:19 smithi059 ceph-mon[97450]: pgmap v21321: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:35:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:19 smithi059 ceph-mon[97450]: osdmap e1411: 12 total, 11 up, 11 in 2022-09-08T08:35:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:19.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:19.157964+0000) 2022-09-08T08:35:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:19.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:19.359876+0000) 2022-09-08T08:35:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:20.359+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:20.360090+0000) 2022-09-08T08:35:20.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:20.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:20.158137+0000) 2022-09-08T08:35:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:21 smithi055 ceph-mon[97033]: pgmap v21323: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:35:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:35:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:35:21.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:35:21.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:35:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:21 smithi059 ceph-mon[97450]: pgmap v21323: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:35:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:35:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:35:21.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:35:21.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:35:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:21.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:21.360350+0000) 2022-09-08T08:35:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:21.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:21.158288+0000) 2022-09-08T08:35:22.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:22 smithi055 ceph-mon[97033]: pgmap v21324: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T08:35:22.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:22 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T08:35:22.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:22 smithi059 ceph-mon[97450]: pgmap v21324: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T08:35:22.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:22 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T08:35:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:22.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:22.360602+0000) 2022-09-08T08:35:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:22.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:22.158441+0000) 2022-09-08T08:35:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:23.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:23.360814+0000) 2022-09-08T08:35:23.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:23.157+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:23.158579+0000) 2022-09-08T08:35:24.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:24 smithi055 ceph-mon[97033]: pgmap v21325: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T08:35:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:24 smithi059 ceph-mon[97450]: pgmap v21325: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 118 B/s wr, 7 op/s 2022-09-08T08:35:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:24.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:24.361052+0000) 2022-09-08T08:35:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:24.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:24.158740+0000) 2022-09-08T08:35:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:25.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:25.361320+0000) 2022-09-08T08:35:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:25.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:25.158954+0000) 2022-09-08T08:35:26.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:35:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:35:25.724+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:35:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:26 smithi055 ceph-mon[97033]: pgmap v21326: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T08:35:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:26 smithi059 ceph-mon[97450]: pgmap v21326: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-09-08T08:35:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:26.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:26.361576+0000) 2022-09-08T08:35:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:26.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:26.159130+0000) 2022-09-08T08:35:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:27.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:27.361806+0000) 2022-09-08T08:35:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:27.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:27.159273+0000) 2022-09-08T08:35:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:28 smithi055 ceph-mon[97033]: pgmap v21327: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T08:35:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:28 smithi059 ceph-mon[97450]: pgmap v21327: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-09-08T08:35:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:28.360+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:28.361998+0000) 2022-09-08T08:35:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:28.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:28.159407+0000) 2022-09-08T08:35:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:29.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:29.362211+0000) 2022-09-08T08:35:29.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:29.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:29.159643+0000) 2022-09-08T08:35:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:30 smithi055 ceph-mon[97033]: pgmap v21328: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:35:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:30 smithi059 ceph-mon[97450]: pgmap v21328: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:35:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:30.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:30.362373+0000) 2022-09-08T08:35:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:30.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:30.159793+0000) 2022-09-08T08:35:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:31.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:31.362603+0000) 2022-09-08T08:35:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:31.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:31.159987+0000) 2022-09-08T08:35:32.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:32 smithi055 ceph-mon[97033]: pgmap v21329: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:35:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:32 smithi059 ceph-mon[97450]: pgmap v21329: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:35:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:32.361+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:32.362871+0000) 2022-09-08T08:35:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:32.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:32.160152+0000) 2022-09-08T08:35:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:33.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:33.363108+0000) 2022-09-08T08:35:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:33.158+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:33.160357+0000) 2022-09-08T08:35:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:34 smithi055 ceph-mon[97033]: pgmap v21330: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:34 smithi059 ceph-mon[97450]: pgmap v21330: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:34.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:34.363370+0000) 2022-09-08T08:35:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:34.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:34.160524+0000) 2022-09-08T08:35:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:35.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:35.160691+0000) 2022-09-08T08:35:35.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:35.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:35.363555+0000) 2022-09-08T08:35:36.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:36 smithi055 ceph-mon[97033]: pgmap v21331: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:36 smithi059 ceph-mon[97450]: pgmap v21331: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:36.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:36.363717+0000) 2022-09-08T08:35:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:36.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:36.160836+0000) 2022-09-08T08:35:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:37.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:37.363962+0000) 2022-09-08T08:35:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:37.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:37.160967+0000) 2022-09-08T08:35:38.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:38 smithi055 ceph-mon[97033]: pgmap v21332: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:35:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:38 smithi059 ceph-mon[97450]: pgmap v21332: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:35:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:38.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:38.364181+0000) 2022-09-08T08:35:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:38.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:38.161113+0000) 2022-09-08T08:35:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:39.362+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:39.364424+0000) 2022-09-08T08:35:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:39.159+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:39.161268+0000) 2022-09-08T08:35:40.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:40 smithi055 ceph-mon[97033]: pgmap v21333: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:40 smithi059 ceph-mon[97450]: pgmap v21333: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:40.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:40.364723+0000) 2022-09-08T08:35:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:40.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:40.161488+0000) 2022-09-08T08:35:41.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:35:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:35:40.732+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:35:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:41.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:41.364975+0000) 2022-09-08T08:35:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:41.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:41.161676+0000) 2022-09-08T08:35:42.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:42 smithi055 ceph-mon[97033]: pgmap v21334: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:35:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:42 smithi059 ceph-mon[97450]: pgmap v21334: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:35:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:42.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:42.365180+0000) 2022-09-08T08:35:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:42.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:42.161819+0000) 2022-09-08T08:35:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:43.363+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:43.365391+0000) 2022-09-08T08:35:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:43.160+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:43.161944+0000) 2022-09-08T08:35:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:44 smithi055 ceph-mon[97033]: pgmap v21335: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:44 smithi059 ceph-mon[97450]: pgmap v21335: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:44.364+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:44.365621+0000) 2022-09-08T08:35:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:44.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:44.162071+0000) 2022-09-08T08:35:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:45.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:45.365848+0000) 2022-09-08T08:35:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:45.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:45.162279+0000) 2022-09-08T08:35:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:46 smithi055 ceph-mon[97033]: pgmap v21336: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:46 smithi059 ceph-mon[97450]: pgmap v21336: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:46.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:46.366054+0000) 2022-09-08T08:35:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:46.161+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:46.162426+0000) 2022-09-08T08:35:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:47.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:47.366288+0000) 2022-09-08T08:35:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:47.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:47.162625+0000) 2022-09-08T08:35:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:48 smithi055 ceph-mon[97033]: pgmap v21337: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:35:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:48 smithi059 ceph-mon[97450]: pgmap v21337: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:35:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:48.365+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:48.366482+0000) 2022-09-08T08:35:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:48.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:48.162842+0000) 2022-09-08T08:35:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:49.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:49.366737+0000) 2022-09-08T08:35:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:49.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:49.163030+0000) 2022-09-08T08:35:50.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:50 smithi055 ceph-mon[97033]: pgmap v21338: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:50 smithi059 ceph-mon[97450]: pgmap v21338: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:50.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:50.366927+0000) 2022-09-08T08:35:50.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:50.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:50.163170+0000) 2022-09-08T08:35:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:51.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:51.367154+0000) 2022-09-08T08:35:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:51.162+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:51.163422+0000) 2022-09-08T08:35:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:52 smithi055 ceph-mon[97033]: pgmap v21339: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:35:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:52 smithi059 ceph-mon[97450]: pgmap v21339: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:35:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:52.366+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:52.367378+0000) 2022-09-08T08:35:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:52.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:52.163696+0000) 2022-09-08T08:35:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:53.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:53.163863+0000) 2022-09-08T08:35:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:53.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:53.367622+0000) 2022-09-08T08:35:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:54 smithi055 ceph-mon[97033]: pgmap v21340: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:54 smithi059 ceph-mon[97450]: pgmap v21340: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:54.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:54.367815+0000) 2022-09-08T08:35:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:54.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:54.164110+0000) 2022-09-08T08:35:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:55.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:55.368020+0000) 2022-09-08T08:35:55.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:55.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:55.164345+0000) 2022-09-08T08:35:56.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:35:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:35:55.744+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:35:56.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:56 smithi055 ceph-mon[97033]: pgmap v21341: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:56 smithi059 ceph-mon[97450]: pgmap v21341: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:35:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:56.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:56.368296+0000) 2022-09-08T08:35:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:56.163+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:56.164578+0000) 2022-09-08T08:35:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:57.367+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:57.368513+0000) 2022-09-08T08:35:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:57.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:57.164763+0000) 2022-09-08T08:35:58.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:35:58 smithi055 ceph-mon[97033]: pgmap v21342: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:35:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:35:58 smithi059 ceph-mon[97450]: pgmap v21342: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:35:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:58.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:58.368725+0000) 2022-09-08T08:35:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:58.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:58.164997+0000) 2022-09-08T08:35:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:35:59.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:59.368865+0000) 2022-09-08T08:35:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:35:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:35:59.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:35:59.165188+0000) 2022-09-08T08:36:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:00 smithi055 ceph-mon[97033]: pgmap v21343: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:36:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:00 smithi059 ceph-mon[97450]: pgmap v21343: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:36:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:00.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:00.369000+0000) 2022-09-08T08:36:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:00.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:00.165335+0000) 2022-09-08T08:36:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:01.368+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:01.369230+0000) 2022-09-08T08:36:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:01.164+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:01.165539+0000) 2022-09-08T08:36:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:02 smithi055 ceph-mon[97033]: pgmap v21344: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:36:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:02 smithi059 ceph-mon[97450]: pgmap v21344: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:36:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:02.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:02.369487+0000) 2022-09-08T08:36:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:02.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:02.165782+0000) 2022-09-08T08:36:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:03.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:03.369694+0000) 2022-09-08T08:36:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:03.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:03.166004+0000) 2022-09-08T08:36:04.485 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:04 smithi055 ceph-mon[97033]: pgmap v21345: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:36:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:04 smithi059 ceph-mon[97450]: pgmap v21345: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:36:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:04.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:04.166177+0000) 2022-09-08T08:36:04.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:04.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:04.369931+0000) 2022-09-08T08:36:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:05.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:05.370189+0000) 2022-09-08T08:36:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:05.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:05.166362+0000) 2022-09-08T08:36:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:06 smithi055 ceph-mon[97033]: pgmap v21346: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:36:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:36:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:06 smithi059 ceph-mon[97450]: pgmap v21346: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:36:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:36:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:06.369+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:06.370428+0000) 2022-09-08T08:36:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:06.165+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:06.166605+0000) 2022-09-08T08:36:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:36:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:36:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:07.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:07.370730+0000) 2022-09-08T08:36:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:07.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:07.166871+0000) 2022-09-08T08:36:08.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:08 smithi055 ceph-mon[97033]: pgmap v21347: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:36:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:08 smithi059 ceph-mon[97450]: pgmap v21347: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:36:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:08.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:08.370893+0000) 2022-09-08T08:36:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:08.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:08.167021+0000) 2022-09-08T08:36:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:09.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:09.371120+0000) 2022-09-08T08:36:09.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:09.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:09.167243+0000) 2022-09-08T08:36:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:10 smithi055 ceph-mon[97033]: pgmap v21348: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:36:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:10 smithi055 ceph-mon[97033]: osdmap e1412: 12 total, 11 up, 11 in 2022-09-08T08:36:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:10 smithi059 ceph-mon[97450]: pgmap v21348: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:36:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:10 smithi059 ceph-mon[97450]: osdmap e1412: 12 total, 11 up, 11 in 2022-09-08T08:36:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:10.166+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:10.167424+0000) 2022-09-08T08:36:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:10.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:10.371349+0000) 2022-09-08T08:36:11.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:36:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:36:10.752+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:36:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:11.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:11.371568+0000) 2022-09-08T08:36:11.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:11.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:11.167590+0000) 2022-09-08T08:36:12.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:12 smithi055 ceph-mon[97033]: pgmap v21350: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:36:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:12 smithi059 ceph-mon[97450]: pgmap v21350: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:36:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:12.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:12.371797+0000) 2022-09-08T08:36:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:12.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:12.167841+0000) 2022-09-08T08:36:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:13.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:13.371915+0000) 2022-09-08T08:36:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:13.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:13.168085+0000) 2022-09-08T08:36:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:14 smithi055 ceph-mon[97033]: pgmap v21351: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:36:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:14 smithi059 ceph-mon[97450]: pgmap v21351: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:36:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:14.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:14.168246+0000) 2022-09-08T08:36:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:14.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:14.372092+0000) 2022-09-08T08:36:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:15.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:15.372320+0000) 2022-09-08T08:36:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:15.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:15.168388+0000) 2022-09-08T08:36:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:16 smithi055 ceph-mon[97033]: pgmap v21352: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:36:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:16 smithi059 ceph-mon[97450]: pgmap v21352: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:36:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:16.370+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:16.372559+0000) 2022-09-08T08:36:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:16.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:16.168613+0000) 2022-09-08T08:36:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:17.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:17.372810+0000) 2022-09-08T08:36:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:17.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:17.168781+0000) 2022-09-08T08:36:18.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:18 smithi055 ceph-mon[97033]: pgmap v21353: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:36:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:18 smithi059 ceph-mon[97450]: pgmap v21353: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:36:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:18.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:18.372925+0000) 2022-09-08T08:36:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:18.167+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:18.168923+0000) 2022-09-08T08:36:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:19.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:19.373131+0000) 2022-09-08T08:36:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:19.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:19.169134+0000) 2022-09-08T08:36:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:19 smithi055 ceph-mon[97033]: pgmap v21354: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:36:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:19 smithi055 ceph-mon[97033]: osdmap e1413: 12 total, 11 up, 11 in 2022-09-08T08:36:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:19 smithi059 ceph-mon[97450]: pgmap v21354: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:36:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:19 smithi059 ceph-mon[97450]: osdmap e1413: 12 total, 11 up, 11 in 2022-09-08T08:36:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:20.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:20.373344+0000) 2022-09-08T08:36:20.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:20.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:20.169377+0000) 2022-09-08T08:36:21.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:36:21.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:36:21.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:20 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:36:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:36:21.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:36:21.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:20 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:36:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:21.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:21.169578+0000) 2022-09-08T08:36:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:21.371+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:21.373501+0000) 2022-09-08T08:36:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:22.372+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:22.373751+0000) 2022-09-08T08:36:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:22.168+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:22.169787+0000) 2022-09-08T08:36:22.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:22 smithi055 ceph-mon[97033]: pgmap v21356: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:36:22.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:36:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:22 smithi059 ceph-mon[97450]: pgmap v21356: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:36:22.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:36:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:23.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:23.373985+0000) 2022-09-08T08:36:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:23.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:23.169989+0000) 2022-09-08T08:36:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:24.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:24.374229+0000) 2022-09-08T08:36:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:24.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:24.170227+0000) 2022-09-08T08:36:24.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:24 smithi055 ceph-mon[97033]: pgmap v21357: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:36:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:24 smithi059 ceph-mon[97450]: pgmap v21357: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:36:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:25.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:25.374442+0000) 2022-09-08T08:36:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:25.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:25.170440+0000) 2022-09-08T08:36:26.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:36:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:36:25.766+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:36:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:26.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:26.374719+0000) 2022-09-08T08:36:26.659 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:26.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:26.170615+0000) 2022-09-08T08:36:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:26 smithi055 ceph-mon[97033]: pgmap v21358: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:36:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:26 smithi059 ceph-mon[97450]: pgmap v21358: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:36:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:27.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:27.170789+0000) 2022-09-08T08:36:27.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:27.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:27.374954+0000) 2022-09-08T08:36:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:28.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:28.170960+0000) 2022-09-08T08:36:28.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:28.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:28.375166+0000) 2022-09-08T08:36:28.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:28 smithi055 ceph-mon[97033]: pgmap v21359: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:36:28.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:28 smithi059 ceph-mon[97450]: pgmap v21359: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:36:29.378 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:29.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:29.171155+0000) 2022-09-08T08:36:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:29.373+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:29.375400+0000) 2022-09-08T08:36:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:30.374+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:30.375693+0000) 2022-09-08T08:36:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:30.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:30.171329+0000) 2022-09-08T08:36:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:30 smithi055 ceph-mon[97033]: pgmap v21360: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:36:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:30 smithi059 ceph-mon[97450]: pgmap v21360: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:36:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:31.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:31.375848+0000) 2022-09-08T08:36:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:31.169+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:31.171546+0000) 2022-09-08T08:36:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:32.170+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:32.171810+0000) 2022-09-08T08:36:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:32.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:32.376011+0000) 2022-09-08T08:36:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:32 smithi055 ceph-mon[97033]: pgmap v21361: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:36:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:32 smithi059 ceph-mon[97450]: pgmap v21361: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:36:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:33.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:33.376268+0000) 2022-09-08T08:36:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:33.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:33.172031+0000) 2022-09-08T08:36:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:34.375+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:34.376519+0000) 2022-09-08T08:36:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:34.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:34.172174+0000) 2022-09-08T08:36:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:34 smithi055 ceph-mon[97033]: pgmap v21362: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:34 smithi059 ceph-mon[97450]: pgmap v21362: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:35.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:35.376809+0000) 2022-09-08T08:36:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:35.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:35.172395+0000) 2022-09-08T08:36:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:36.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:36.377091+0000) 2022-09-08T08:36:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:36.171+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:36.172596+0000) 2022-09-08T08:36:36.736 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:36 smithi055 ceph-mon[97033]: pgmap v21363: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:36 smithi059 ceph-mon[97450]: pgmap v21363: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:37.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:37.377340+0000) 2022-09-08T08:36:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:37.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:37.172788+0000) 2022-09-08T08:36:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:38.376+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:38.377500+0000) 2022-09-08T08:36:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:38.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:38.172932+0000) 2022-09-08T08:36:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:38 smithi055 ceph-mon[97033]: pgmap v21364: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:36:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:38 smithi059 ceph-mon[97450]: pgmap v21364: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:36:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:39.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:39.377739+0000) 2022-09-08T08:36:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:39.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:39.173110+0000) 2022-09-08T08:36:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:40.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:40.377881+0000) 2022-09-08T08:36:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:40.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:40.173267+0000) 2022-09-08T08:36:40.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:40 smithi055 ceph-mon[97033]: pgmap v21365: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:40 smithi059 ceph-mon[97450]: pgmap v21365: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:41.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:36:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:36:40.770+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:36:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:41.172+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:41.173445+0000) 2022-09-08T08:36:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:41.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:41.378081+0000) 2022-09-08T08:36:42.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:42.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:42.173707+0000) 2022-09-08T08:36:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:42.377+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:42.378303+0000) 2022-09-08T08:36:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:42 smithi055 ceph-mon[97033]: pgmap v21366: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:36:42.784 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:42 smithi059 ceph-mon[97450]: pgmap v21366: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:36:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:43.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:43.173910+0000) 2022-09-08T08:36:43.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:43.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:43.378553+0000) 2022-09-08T08:36:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:44.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:44.174122+0000) 2022-09-08T08:36:44.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:44.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:44.378807+0000) 2022-09-08T08:36:44.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:44 smithi055 ceph-mon[97033]: pgmap v21367: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:44 smithi059 ceph-mon[97450]: pgmap v21367: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:45.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:45.378965+0000) 2022-09-08T08:36:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:45.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:45.174322+0000) 2022-09-08T08:36:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:46.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:46.379221+0000) 2022-09-08T08:36:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:46.173+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:46.174511+0000) 2022-09-08T08:36:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:46 smithi055 ceph-mon[97033]: pgmap v21368: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:46 smithi059 ceph-mon[97450]: pgmap v21368: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:47.378+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:47.379435+0000) 2022-09-08T08:36:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:47.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:47.174710+0000) 2022-09-08T08:36:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:48.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:48.379663+0000) 2022-09-08T08:36:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:48.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:48.174843+0000) 2022-09-08T08:36:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:48 smithi055 ceph-mon[97033]: pgmap v21369: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:36:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:48 smithi059 ceph-mon[97450]: pgmap v21369: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:36:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:49.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:49.175022+0000) 2022-09-08T08:36:49.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:49.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:49.379818+0000) 2022-09-08T08:36:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:50.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:50.379964+0000) 2022-09-08T08:36:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:50.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:50.175241+0000) 2022-09-08T08:36:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:50 smithi055 ceph-mon[97033]: pgmap v21370: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:50.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:50 smithi059 ceph-mon[97450]: pgmap v21370: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:51.379+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:51.380118+0000) 2022-09-08T08:36:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:51.174+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:51.175410+0000) 2022-09-08T08:36:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:52.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:52.380323+0000) 2022-09-08T08:36:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:52.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:52.175585+0000) 2022-09-08T08:36:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:52 smithi055 ceph-mon[97033]: pgmap v21371: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:36:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:52 smithi059 ceph-mon[97450]: pgmap v21371: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:36:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:53.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:53.175802+0000) 2022-09-08T08:36:53.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:53.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:53.380486+0000) 2022-09-08T08:36:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:54.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:54.380727+0000) 2022-09-08T08:36:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:54.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:54.176056+0000) 2022-09-08T08:36:54.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:54 smithi055 ceph-mon[97033]: pgmap v21372: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:54 smithi059 ceph-mon[97450]: pgmap v21372: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:55.175+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:55.176259+0000) 2022-09-08T08:36:55.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:55.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:55.381002+0000) 2022-09-08T08:36:56.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:36:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:36:55.784+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:36:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:56.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:56.381274+0000) 2022-09-08T08:36:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:56.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:56.176409+0000) 2022-09-08T08:36:56.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:56 smithi055 ceph-mon[97033]: pgmap v21373: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:56 smithi059 ceph-mon[97450]: pgmap v21373: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:36:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:57.380+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:57.381493+0000) 2022-09-08T08:36:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:57.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:57.176608+0000) 2022-09-08T08:36:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:58.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:58.176922+0000) 2022-09-08T08:36:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:58.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:58.381742+0000) 2022-09-08T08:36:58.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:36:58 smithi055 ceph-mon[97033]: pgmap v21374: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:36:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:36:58 smithi059 ceph-mon[97450]: pgmap v21374: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:36:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:36:59.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:59.177102+0000) 2022-09-08T08:36:59.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:36:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:36:59.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:36:59.381906+0000) 2022-09-08T08:37:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:00.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:00.382048+0000) 2022-09-08T08:37:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:00.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:00.177304+0000) 2022-09-08T08:37:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:00 smithi055 ceph-mon[97033]: pgmap v21375: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:00 smithi059 ceph-mon[97450]: pgmap v21375: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:01.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:01.382273+0000) 2022-09-08T08:37:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:01.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:01.177461+0000) 2022-09-08T08:37:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:02.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:02.382516+0000) 2022-09-08T08:37:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:02.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:02.177736+0000) 2022-09-08T08:37:02.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:02 smithi055 ceph-mon[97033]: pgmap v21376: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:37:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:02 smithi059 ceph-mon[97450]: pgmap v21376: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:37:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:03.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:03.382792+0000) 2022-09-08T08:37:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:03.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:03.177935+0000) 2022-09-08T08:37:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:04.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:04.382952+0000) 2022-09-08T08:37:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:04.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:04.178099+0000) 2022-09-08T08:37:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:04 smithi055 ceph-mon[97033]: pgmap v21377: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:04 smithi059 ceph-mon[97450]: pgmap v21377: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:05.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:05.383100+0000) 2022-09-08T08:37:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:05.176+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:05.178248+0000) 2022-09-08T08:37:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:06.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:06.383352+0000) 2022-09-08T08:37:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:06.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:06.178411+0000) 2022-09-08T08:37:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:06 smithi055 ceph-mon[97033]: pgmap v21378: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:37:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:06 smithi059 ceph-mon[97450]: pgmap v21378: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:37:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:07.381+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:07.383575+0000) 2022-09-08T08:37:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:07.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:07.178587+0000) 2022-09-08T08:37:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:37:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:37:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:08.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:08.178724+0000) 2022-09-08T08:37:08.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:08.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:08.383794+0000) 2022-09-08T08:37:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:08 smithi055 ceph-mon[97033]: pgmap v21379: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:37:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:08 smithi059 ceph-mon[97450]: pgmap v21379: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:37:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:09.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:09.384002+0000) 2022-09-08T08:37:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:09.177+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:09.178896+0000) 2022-09-08T08:37:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:10.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:10.384265+0000) 2022-09-08T08:37:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:10.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:10.179125+0000) 2022-09-08T08:37:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:10 smithi055 ceph-mon[97033]: pgmap v21380: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:37:10.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:10 smithi055 ceph-mon[97033]: osdmap e1414: 12 total, 11 up, 11 in 2022-09-08T08:37:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:10 smithi059 ceph-mon[97450]: pgmap v21380: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:37:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:10 smithi059 ceph-mon[97450]: osdmap e1414: 12 total, 11 up, 11 in 2022-09-08T08:37:11.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:37:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:37:10.787+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:37:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:11.382+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:11.384449+0000) 2022-09-08T08:37:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:11.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:11.179377+0000) 2022-09-08T08:37:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:12.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:12.179573+0000) 2022-09-08T08:37:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:12.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:12.384719+0000) 2022-09-08T08:37:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:12 smithi055 ceph-mon[97033]: pgmap v21382: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:37:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:12 smithi059 ceph-mon[97450]: pgmap v21382: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:37:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:13.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:13.384941+0000) 2022-09-08T08:37:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:13.178+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:13.179793+0000) 2022-09-08T08:37:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:14.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:14.385178+0000) 2022-09-08T08:37:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:14.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:14.180040+0000) 2022-09-08T08:37:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:14 smithi055 ceph-mon[97033]: pgmap v21383: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:37:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:14 smithi059 ceph-mon[97450]: pgmap v21383: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:37:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:15.383+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:15.385404+0000) 2022-09-08T08:37:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:15.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:15.180239+0000) 2022-09-08T08:37:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:16.384+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:16.385677+0000) 2022-09-08T08:37:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:16.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:16.180407+0000) 2022-09-08T08:37:16.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:16 smithi055 ceph-mon[97033]: pgmap v21384: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:37:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:16 smithi059 ceph-mon[97450]: pgmap v21384: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:37:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:17.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:17.385898+0000) 2022-09-08T08:37:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:17.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:17.180557+0000) 2022-09-08T08:37:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:18.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:18.386127+0000) 2022-09-08T08:37:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:18.179+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:18.180700+0000) 2022-09-08T08:37:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:18 smithi055 ceph-mon[97033]: pgmap v21385: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:37:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:18 smithi059 ceph-mon[97450]: pgmap v21385: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:37:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:19.385+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:19.386401+0000) 2022-09-08T08:37:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:19.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:19.180869+0000) 2022-09-08T08:37:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:19 smithi055 ceph-mon[97033]: pgmap v21386: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:37:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:19 smithi055 ceph-mon[97033]: osdmap e1415: 12 total, 11 up, 11 in 2022-09-08T08:37:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:19 smithi059 ceph-mon[97450]: pgmap v21386: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:37:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:19 smithi059 ceph-mon[97450]: osdmap e1415: 12 total, 11 up, 11 in 2022-09-08T08:37:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:20.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:20.386616+0000) 2022-09-08T08:37:20.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:20.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:20.181037+0000) 2022-09-08T08:37:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:21.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:21.386821+0000) 2022-09-08T08:37:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:21.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:21.181260+0000) 2022-09-08T08:37:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:21 smithi055 ceph-mon[97033]: pgmap v21388: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:37:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:37:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:37:22.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:21 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:37:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:21 smithi059 ceph-mon[97450]: pgmap v21388: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:37:22.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:37:22.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:37:22.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:21 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:37:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:22.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:22.387041+0000) 2022-09-08T08:37:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:22.180+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:22.181468+0000) 2022-09-08T08:37:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:37:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:37:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:23.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:23.387293+0000) 2022-09-08T08:37:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:23.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:23.181709+0000) 2022-09-08T08:37:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:23 smithi055 ceph-mon[97033]: pgmap v21389: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:37:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:23 smithi059 ceph-mon[97450]: pgmap v21389: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:37:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:24.386+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:24.387533+0000) 2022-09-08T08:37:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:24.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:24.181891+0000) 2022-09-08T08:37:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:25.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:25.387777+0000) 2022-09-08T08:37:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:25.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:25.182101+0000) 2022-09-08T08:37:26.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:37:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:37:25.793+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:37:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:25 smithi055 ceph-mon[97033]: pgmap v21390: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:37:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:25 smithi059 ceph-mon[97450]: pgmap v21390: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:37:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:26.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:26.388026+0000) 2022-09-08T08:37:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:26.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:26.182238+0000) 2022-09-08T08:37:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:27.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:27.388273+0000) 2022-09-08T08:37:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:27.181+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:27.182430+0000) 2022-09-08T08:37:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:27 smithi055 ceph-mon[97033]: pgmap v21391: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:37:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:27 smithi059 ceph-mon[97450]: pgmap v21391: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:37:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:28.387+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:28.388517+0000) 2022-09-08T08:37:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:28.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:28.182667+0000) 2022-09-08T08:37:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:29.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:29.388756+0000) 2022-09-08T08:37:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:29.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:29.182813+0000) 2022-09-08T08:37:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:29 smithi055 ceph-mon[97033]: pgmap v21392: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:37:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:29 smithi059 ceph-mon[97450]: pgmap v21392: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:37:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:30.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:30.388991+0000) 2022-09-08T08:37:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:30.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:30.182974+0000) 2022-09-08T08:37:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:31.388+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:31.389240+0000) 2022-09-08T08:37:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:31.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:31.183114+0000) 2022-09-08T08:37:32.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:31 smithi055 ceph-mon[97033]: pgmap v21393: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:37:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:31 smithi059 ceph-mon[97450]: pgmap v21393: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:37:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:32.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:32.389497+0000) 2022-09-08T08:37:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:32.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:32.183256+0000) 2022-09-08T08:37:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:33.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:33.183384+0000) 2022-09-08T08:37:33.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:33.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:33.389745+0000) 2022-09-08T08:37:34.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:33 smithi055 ceph-mon[97033]: pgmap v21394: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:33 smithi059 ceph-mon[97450]: pgmap v21394: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:34.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:34.389892+0000) 2022-09-08T08:37:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:34.182+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:34.183561+0000) 2022-09-08T08:37:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:35.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:35.390129+0000) 2022-09-08T08:37:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:35.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:35.183763+0000) 2022-09-08T08:37:36.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:35 smithi055 ceph-mon[97033]: pgmap v21395: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:35 smithi059 ceph-mon[97450]: pgmap v21395: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:36.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:36.390426+0000) 2022-09-08T08:37:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:36.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:36.183902+0000) 2022-09-08T08:37:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:37.389+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:37.390596+0000) 2022-09-08T08:37:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:37.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:37.184074+0000) 2022-09-08T08:37:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:37 smithi059 ceph-mon[97450]: pgmap v21396: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:37:38.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:37 smithi055 ceph-mon[97033]: pgmap v21396: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:37:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:38.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:38.390783+0000) 2022-09-08T08:37:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:38.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:38.184246+0000) 2022-09-08T08:37:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:39.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:39.390991+0000) 2022-09-08T08:37:39.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:39.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:39.184384+0000) 2022-09-08T08:37:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:39 smithi059 ceph-mon[97450]: pgmap v21397: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:39 smithi055 ceph-mon[97033]: pgmap v21397: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:40.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:40.391232+0000) 2022-09-08T08:37:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:40.183+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:40.184517+0000) 2022-09-08T08:37:41.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:37:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:37:40.798+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:37:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:41.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:41.391402+0000) 2022-09-08T08:37:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:41.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:41.184663+0000) 2022-09-08T08:37:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:41 smithi059 ceph-mon[97450]: pgmap v21398: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:37:42.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:41 smithi055 ceph-mon[97033]: pgmap v21398: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:37:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:42.390+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:42.391553+0000) 2022-09-08T08:37:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:42.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:42.184823+0000) 2022-09-08T08:37:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:43.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:43.391745+0000) 2022-09-08T08:37:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:43.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:43.184964+0000) 2022-09-08T08:37:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:43 smithi059 ceph-mon[97450]: pgmap v21399: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:44.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:43 smithi055 ceph-mon[97033]: pgmap v21399: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:44.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:44.391959+0000) 2022-09-08T08:37:44.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:44.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:44.185140+0000) 2022-09-08T08:37:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:45.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:45.392157+0000) 2022-09-08T08:37:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:45.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:45.185344+0000) 2022-09-08T08:37:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:45 smithi059 ceph-mon[97450]: pgmap v21400: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:46.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:45 smithi055 ceph-mon[97033]: pgmap v21400: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:46.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:46.392360+0000) 2022-09-08T08:37:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:46.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:46.185526+0000) 2022-09-08T08:37:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:47.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:47.392583+0000) 2022-09-08T08:37:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:47.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:47.185701+0000) 2022-09-08T08:37:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:47 smithi059 ceph-mon[97450]: pgmap v21401: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:37:48.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:47 smithi055 ceph-mon[97033]: pgmap v21401: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:37:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:48.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:48.392806+0000) 2022-09-08T08:37:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:48.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:48.185839+0000) 2022-09-08T08:37:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:49.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:49.393024+0000) 2022-09-08T08:37:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:49.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:49.185975+0000) 2022-09-08T08:37:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:49 smithi059 ceph-mon[97450]: pgmap v21402: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:50.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:50 smithi055 ceph-mon[97033]: pgmap v21402: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:50.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:50.393211+0000) 2022-09-08T08:37:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:50.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:50.186111+0000) 2022-09-08T08:37:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:51.391+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:51.393336+0000) 2022-09-08T08:37:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:51.184+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:51.186348+0000) 2022-09-08T08:37:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:52 smithi059 ceph-mon[97450]: pgmap v21403: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:37:52.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:52 smithi055 ceph-mon[97033]: pgmap v21403: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:37:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:52.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:52.186549+0000) 2022-09-08T08:37:52.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:52.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:52.393558+0000) 2022-09-08T08:37:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:53.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:53.393774+0000) 2022-09-08T08:37:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:53.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:53.186719+0000) 2022-09-08T08:37:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:54 smithi059 ceph-mon[97450]: pgmap v21404: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:54.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:54 smithi055 ceph-mon[97033]: pgmap v21404: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:54.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:54.393993+0000) 2022-09-08T08:37:54.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:54.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:54.186929+0000) 2022-09-08T08:37:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:55.392+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:55.394209+0000) 2022-09-08T08:37:55.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:55.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:55.187144+0000) 2022-09-08T08:37:56.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:37:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:37:55.813+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:37:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:56 smithi059 ceph-mon[97450]: pgmap v21405: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:56 smithi055 ceph-mon[97033]: pgmap v21405: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:37:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:56.185+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:56.187327+0000) 2022-09-08T08:37:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:56.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:56.394510+0000) 2022-09-08T08:37:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:57.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:57.394752+0000) 2022-09-08T08:37:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:57.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:57.187684+0000) 2022-09-08T08:37:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:37:58 smithi059 ceph-mon[97450]: pgmap v21406: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:37:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:37:58 smithi055 ceph-mon[97033]: pgmap v21406: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:37:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:58.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:58.187874+0000) 2022-09-08T08:37:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:58.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:58.394981+0000) 2022-09-08T08:37:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:37:59.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:59.395221+0000) 2022-09-08T08:37:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:37:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:37:59.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:37:59.188027+0000) 2022-09-08T08:38:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:00 smithi059 ceph-mon[97450]: pgmap v21407: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:00 smithi055 ceph-mon[97033]: pgmap v21407: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:00.393+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:00.395471+0000) 2022-09-08T08:38:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:00.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:00.188190+0000) 2022-09-08T08:38:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:01.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:01.395662+0000) 2022-09-08T08:38:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:01.186+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:01.188371+0000) 2022-09-08T08:38:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:02 smithi059 ceph-mon[97450]: pgmap v21408: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:38:02.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:02 smithi055 ceph-mon[97033]: pgmap v21408: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:38:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:02.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:02.395872+0000) 2022-09-08T08:38:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:02.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:02.188515+0000) 2022-09-08T08:38:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:03.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:03.396073+0000) 2022-09-08T08:38:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:03.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:03.188726+0000) 2022-09-08T08:38:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:04 smithi059 ceph-mon[97450]: pgmap v21409: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:04 smithi055 ceph-mon[97033]: pgmap v21409: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:04.394+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:04.396228+0000) 2022-09-08T08:38:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:04.187+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:04.188859+0000) 2022-09-08T08:38:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:05.395+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:05.396442+0000) 2022-09-08T08:38:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:05.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:05.188990+0000) 2022-09-08T08:38:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:06 smithi059 ceph-mon[97450]: pgmap v21410: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:06 smithi055 ceph-mon[97033]: pgmap v21410: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:06.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:06.396706+0000) 2022-09-08T08:38:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:06.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:06.189127+0000) 2022-09-08T08:38:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:38:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:38:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:38:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:38:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:07.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:07.396931+0000) 2022-09-08T08:38:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:07.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:07.189338+0000) 2022-09-08T08:38:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:08 smithi055 ceph-mon[97033]: pgmap v21411: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:38:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:08 smithi059 ceph-mon[97450]: pgmap v21411: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:38:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:08.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:08.397130+0000) 2022-09-08T08:38:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:08.188+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:08.189595+0000) 2022-09-08T08:38:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:09.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:09.397375+0000) 2022-09-08T08:38:09.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:09.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:09.189763+0000) 2022-09-08T08:38:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:10 smithi055 ceph-mon[97033]: pgmap v21412: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:38:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:10 smithi059 ceph-mon[97450]: pgmap v21412: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:38:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:10.396+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:10.397598+0000) 2022-09-08T08:38:10.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:10.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:10.189953+0000) 2022-09-08T08:38:11.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:38:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:38:10.826+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:38:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:11 smithi055 ceph-mon[97033]: osdmap e1416: 12 total, 11 up, 11 in 2022-09-08T08:38:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:11 smithi059 ceph-mon[97450]: osdmap e1416: 12 total, 11 up, 11 in 2022-09-08T08:38:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:11.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:11.397808+0000) 2022-09-08T08:38:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:11.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:11.190104+0000) 2022-09-08T08:38:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:12 smithi055 ceph-mon[97033]: pgmap v21414: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:38:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:12 smithi059 ceph-mon[97450]: pgmap v21414: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:38:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:12.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:12.398062+0000) 2022-09-08T08:38:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:12.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:12.190253+0000) 2022-09-08T08:38:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:13.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:13.398290+0000) 2022-09-08T08:38:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:13.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:13.190409+0000) 2022-09-08T08:38:14.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:14 smithi055 ceph-mon[97033]: pgmap v21415: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:38:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:14 smithi059 ceph-mon[97450]: pgmap v21415: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:38:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:14.397+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:14.398514+0000) 2022-09-08T08:38:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:14.189+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:14.190612+0000) 2022-09-08T08:38:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:15.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:15.398760+0000) 2022-09-08T08:38:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:15.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:15.190829+0000) 2022-09-08T08:38:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:16 smithi055 ceph-mon[97033]: pgmap v21416: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:38:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:16 smithi059 ceph-mon[97450]: pgmap v21416: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:38:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:16.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:16.398987+0000) 2022-09-08T08:38:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:16.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:16.191128+0000) 2022-09-08T08:38:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:17.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:17.399167+0000) 2022-09-08T08:38:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:17.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:17.191263+0000) 2022-09-08T08:38:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:18 smithi055 ceph-mon[97033]: pgmap v21417: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:38:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:18 smithi059 ceph-mon[97450]: pgmap v21417: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:38:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:18.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:18.191417+0000) 2022-09-08T08:38:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:18.398+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:18.399409+0000) 2022-09-08T08:38:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:19.190+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:19.191547+0000) 2022-09-08T08:38:19.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:19.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:19.399694+0000) 2022-09-08T08:38:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:19 smithi055 ceph-mon[97033]: pgmap v21418: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:38:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:19 smithi055 ceph-mon[97033]: osdmap e1417: 12 total, 11 up, 11 in 2022-09-08T08:38:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:19 smithi059 ceph-mon[97450]: pgmap v21418: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:38:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:19 smithi059 ceph-mon[97450]: osdmap e1417: 12 total, 11 up, 11 in 2022-09-08T08:38:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:20.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:20.191707+0000) 2022-09-08T08:38:20.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:20.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:20.399837+0000) 2022-09-08T08:38:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:21.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:21.400097+0000) 2022-09-08T08:38:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:21.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:21.191854+0000) 2022-09-08T08:38:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:21 smithi055 ceph-mon[97033]: pgmap v21420: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:38:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:21 smithi059 ceph-mon[97450]: pgmap v21420: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:38:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:22.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:22.400327+0000) 2022-09-08T08:38:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:22.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:22.191990+0000) 2022-09-08T08:38:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:38:23.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:38:23.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:38:23.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:38:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:38:23.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:38:23.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:38:23.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:38:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:23.399+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:23.400521+0000) 2022-09-08T08:38:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:23.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:23.192241+0000) 2022-09-08T08:38:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:24.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:24.400752+0000) 2022-09-08T08:38:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:24.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:24.192400+0000) 2022-09-08T08:38:24.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:24 smithi055 ceph-mon[97033]: pgmap v21421: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:38:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:24 smithi059 ceph-mon[97450]: pgmap v21421: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:38:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:25.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:25.400903+0000) 2022-09-08T08:38:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:25.191+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:25.192548+0000) 2022-09-08T08:38:26.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:38:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:38:25.842+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:38:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:26.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:26.401141+0000) 2022-09-08T08:38:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:26.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:26.192732+0000) 2022-09-08T08:38:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:26 smithi055 ceph-mon[97033]: pgmap v21422: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:38:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:26 smithi059 ceph-mon[97450]: pgmap v21422: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:38:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:27.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:27.192910+0000) 2022-09-08T08:38:27.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:27.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:27.401372+0000) 2022-09-08T08:38:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:28.400+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:28.401604+0000) 2022-09-08T08:38:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:28.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:28.193116+0000) 2022-09-08T08:38:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:28 smithi055 ceph-mon[97033]: pgmap v21423: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:38:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:28 smithi059 ceph-mon[97450]: pgmap v21423: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:38:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:29.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:29.401786+0000) 2022-09-08T08:38:29.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:29.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:29.193353+0000) 2022-09-08T08:38:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:30.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:30.402007+0000) 2022-09-08T08:38:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:30.192+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:30.193532+0000) 2022-09-08T08:38:30.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:30 smithi055 ceph-mon[97033]: pgmap v21424: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:38:31.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:30 smithi059 ceph-mon[97450]: pgmap v21424: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:38:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:31.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:31.402261+0000) 2022-09-08T08:38:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:31.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:31.193717+0000) 2022-09-08T08:38:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:32.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:32.402449+0000) 2022-09-08T08:38:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:32.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:32.193974+0000) 2022-09-08T08:38:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:32 smithi055 ceph-mon[97033]: pgmap v21425: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:38:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:32 smithi059 ceph-mon[97450]: pgmap v21425: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:38:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:33.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:33.402718+0000) 2022-09-08T08:38:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:33.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:33.194206+0000) 2022-09-08T08:38:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:34.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:34.402956+0000) 2022-09-08T08:38:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:34.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:34.194335+0000) 2022-09-08T08:38:34.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:34 smithi055 ceph-mon[97033]: pgmap v21426: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:34 smithi059 ceph-mon[97450]: pgmap v21426: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:35.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:35.403174+0000) 2022-09-08T08:38:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:35.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:35.194497+0000) 2022-09-08T08:38:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:36.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:36.403382+0000) 2022-09-08T08:38:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:36.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:36.194676+0000) 2022-09-08T08:38:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:36 smithi055 ceph-mon[97033]: pgmap v21427: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:36 smithi059 ceph-mon[97450]: pgmap v21427: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:37.401+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:37.403502+0000) 2022-09-08T08:38:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:37.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:37.194827+0000) 2022-09-08T08:38:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:38.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:38.403729+0000) 2022-09-08T08:38:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:38.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:38.195066+0000) 2022-09-08T08:38:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:38 smithi055 ceph-mon[97033]: pgmap v21428: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:38:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:38 smithi059 ceph-mon[97450]: pgmap v21428: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:38:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:39.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:39.403877+0000) 2022-09-08T08:38:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:39.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:39.195217+0000) 2022-09-08T08:38:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:40.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:40.404098+0000) 2022-09-08T08:38:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:40.193+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:40.195429+0000) 2022-09-08T08:38:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:40 smithi055 ceph-mon[97033]: pgmap v21429: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:40 smithi059 ceph-mon[97450]: pgmap v21429: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:41.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:38:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:38:40.850+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:38:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:41.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:41.404334+0000) 2022-09-08T08:38:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:41.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:41.195690+0000) 2022-09-08T08:38:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:42.402+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:42.404564+0000) 2022-09-08T08:38:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:42.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:42.195874+0000) 2022-09-08T08:38:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:42 smithi055 ceph-mon[97033]: pgmap v21430: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:38:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:42 smithi059 ceph-mon[97450]: pgmap v21430: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:38:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:43.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:43.196017+0000) 2022-09-08T08:38:43.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:43.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:43.404753+0000) 2022-09-08T08:38:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:44.403+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:44.404975+0000) 2022-09-08T08:38:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:44.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:44.196215+0000) 2022-09-08T08:38:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:44 smithi055 ceph-mon[97033]: pgmap v21431: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:44 smithi059 ceph-mon[97450]: pgmap v21431: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:45.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:45.405214+0000) 2022-09-08T08:38:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:45.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:45.196372+0000) 2022-09-08T08:38:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:46.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:46.405466+0000) 2022-09-08T08:38:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:46.194+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:46.196584+0000) 2022-09-08T08:38:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:46 smithi055 ceph-mon[97033]: pgmap v21432: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:46 smithi059 ceph-mon[97450]: pgmap v21432: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:47.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:47.405751+0000) 2022-09-08T08:38:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:47.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:47.196760+0000) 2022-09-08T08:38:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:48.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:48.405950+0000) 2022-09-08T08:38:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:48.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:48.197002+0000) 2022-09-08T08:38:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:48 smithi055 ceph-mon[97033]: pgmap v21433: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:38:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:48 smithi059 ceph-mon[97450]: pgmap v21433: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:38:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:49.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:49.406202+0000) 2022-09-08T08:38:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:49.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:49.197165+0000) 2022-09-08T08:38:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:50.195+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:50.197400+0000) 2022-09-08T08:38:50.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:50.404+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:50.406430+0000) 2022-09-08T08:38:50.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:50 smithi055 ceph-mon[97033]: pgmap v21434: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:50 smithi059 ceph-mon[97450]: pgmap v21434: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:51.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:51.406631+0000) 2022-09-08T08:38:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:51.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:51.197693+0000) 2022-09-08T08:38:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:52.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:52.406894+0000) 2022-09-08T08:38:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:52.196+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:52.197876+0000) 2022-09-08T08:38:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:52 smithi055 ceph-mon[97033]: pgmap v21435: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:38:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:52 smithi059 ceph-mon[97450]: pgmap v21435: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:38:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:53.405+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:53.407074+0000) 2022-09-08T08:38:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:53.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:53.198013+0000) 2022-09-08T08:38:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:54.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:54.407316+0000) 2022-09-08T08:38:54.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:54.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:54.198191+0000) 2022-09-08T08:38:54.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:54 smithi055 ceph-mon[97033]: pgmap v21436: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:54 smithi059 ceph-mon[97450]: pgmap v21436: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:55.406+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:55.407495+0000) 2022-09-08T08:38:55.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:55.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:55.198396+0000) 2022-09-08T08:38:56.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:38:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:38:55.860+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:38:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:56.197+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:56.198610+0000) 2022-09-08T08:38:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:56.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:56.407732+0000) 2022-09-08T08:38:56.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:56 smithi055 ceph-mon[97033]: pgmap v21437: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:56 smithi059 ceph-mon[97450]: pgmap v21437: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:38:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:57.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:57.407938+0000) 2022-09-08T08:38:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:57.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:57.198803+0000) 2022-09-08T08:38:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:58.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:58.408119+0000) 2022-09-08T08:38:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:58.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:58.199007+0000) 2022-09-08T08:38:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:38:58 smithi055 ceph-mon[97033]: pgmap v21438: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:38:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:38:58 smithi059 ceph-mon[97450]: pgmap v21438: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:38:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:38:59.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:59.408332+0000) 2022-09-08T08:38:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:38:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:38:59.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:38:59.199217+0000) 2022-09-08T08:39:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:00.407+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:00.408526+0000) 2022-09-08T08:39:00.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:00.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:00.199383+0000) 2022-09-08T08:39:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:00 smithi055 ceph-mon[97033]: pgmap v21439: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:00 smithi059 ceph-mon[97450]: pgmap v21439: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:01.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:01.408738+0000) 2022-09-08T08:39:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:01.198+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:01.199590+0000) 2022-09-08T08:39:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:02.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:02.408934+0000) 2022-09-08T08:39:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:02.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:02.199768+0000) 2022-09-08T08:39:02.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:02 smithi055 ceph-mon[97033]: pgmap v21440: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:39:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:02 smithi059 ceph-mon[97450]: pgmap v21440: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:39:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:03.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:03.409095+0000) 2022-09-08T08:39:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:03.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:03.200022+0000) 2022-09-08T08:39:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:04.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:04.409300+0000) 2022-09-08T08:39:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:04.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:04.200193+0000) 2022-09-08T08:39:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:04 smithi055 ceph-mon[97033]: pgmap v21441: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:04 smithi059 ceph-mon[97450]: pgmap v21441: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:05.408+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:05.409559+0000) 2022-09-08T08:39:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:05.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:05.200346+0000) 2022-09-08T08:39:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:05 smithi055 ceph-mon[97033]: pgmap v21442: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:05 smithi059 ceph-mon[97450]: pgmap v21442: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:06.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:06.200494+0000) 2022-09-08T08:39:06.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:06.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:06.409756+0000) 2022-09-08T08:39:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:39:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:39:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:07.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:07.409974+0000) 2022-09-08T08:39:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:07.199+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:07.200625+0000) 2022-09-08T08:39:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:07 smithi055 ceph-mon[97033]: pgmap v21443: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:39:07.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:39:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:07 smithi059 ceph-mon[97450]: pgmap v21443: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:39:08.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:39:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:08.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:08.200800+0000) 2022-09-08T08:39:08.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:08.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:08.410169+0000) 2022-09-08T08:39:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:09.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:09.200981+0000) 2022-09-08T08:39:09.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:09.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:09.410364+0000) 2022-09-08T08:39:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:09 smithi055 ceph-mon[97033]: pgmap v21444: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:39:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:09 smithi059 ceph-mon[97450]: pgmap v21444: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:39:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:10.409+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:10.410536+0000) 2022-09-08T08:39:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:10.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:10.201124+0000) 2022-09-08T08:39:11.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:39:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:39:10.862+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:39:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:10 smithi055 ceph-mon[97033]: osdmap e1418: 12 total, 11 up, 11 in 2022-09-08T08:39:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:10 smithi059 ceph-mon[97450]: osdmap e1418: 12 total, 11 up, 11 in 2022-09-08T08:39:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:11.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:11.410795+0000) 2022-09-08T08:39:11.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:11.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:11.201338+0000) 2022-09-08T08:39:12.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:11 smithi055 ceph-mon[97033]: pgmap v21446: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:39:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:11 smithi059 ceph-mon[97450]: pgmap v21446: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:39:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:12.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:12.410944+0000) 2022-09-08T08:39:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:12.200+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:12.201545+0000) 2022-09-08T08:39:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:13.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:13.201702+0000) 2022-09-08T08:39:13.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:13.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:13.411126+0000) 2022-09-08T08:39:14.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:13 smithi055 ceph-mon[97033]: pgmap v21447: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:39:14.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:13 smithi059 ceph-mon[97450]: pgmap v21447: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:39:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:14.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:14.411273+0000) 2022-09-08T08:39:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:14.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:14.201857+0000) 2022-09-08T08:39:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:15.410+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:15.411490+0000) 2022-09-08T08:39:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:15.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:15.202073+0000) 2022-09-08T08:39:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:15 smithi055 ceph-mon[97033]: pgmap v21448: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:39:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:15 smithi059 ceph-mon[97450]: pgmap v21448: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:39:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:16.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:16.411757+0000) 2022-09-08T08:39:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:16.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:16.202323+0000) 2022-09-08T08:39:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:17.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:17.411965+0000) 2022-09-08T08:39:17.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:17.201+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:17.202500+0000) 2022-09-08T08:39:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:17 smithi055 ceph-mon[97033]: pgmap v21449: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:39:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:17 smithi059 ceph-mon[97450]: pgmap v21449: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:39:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:18.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:18.412198+0000) 2022-09-08T08:39:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:18.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:18.202668+0000) 2022-09-08T08:39:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:19.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:19.202859+0000) 2022-09-08T08:39:19.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:19.411+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:19.412442+0000) 2022-09-08T08:39:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:19 smithi055 ceph-mon[97033]: pgmap v21450: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:39:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:19 smithi055 ceph-mon[97033]: osdmap e1419: 12 total, 11 up, 11 in 2022-09-08T08:39:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:19 smithi059 ceph-mon[97450]: pgmap v21450: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:39:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:19 smithi059 ceph-mon[97450]: osdmap e1419: 12 total, 11 up, 11 in 2022-09-08T08:39:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:20.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:20.412734+0000) 2022-09-08T08:39:20.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:20.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:20.203056+0000) 2022-09-08T08:39:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:21.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:21.413002+0000) 2022-09-08T08:39:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:21.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:21.203214+0000) 2022-09-08T08:39:22.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:21 smithi055 ceph-mon[97033]: pgmap v21452: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T08:39:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:21 smithi059 ceph-mon[97450]: pgmap v21452: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T08:39:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:22.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:22.413238+0000) 2022-09-08T08:39:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:22.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:22.203435+0000) 2022-09-08T08:39:22.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:39:22.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:39:22.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:22 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:39:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:39:23.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:39:23.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:22 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:39:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:23.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:23.413469+0000) 2022-09-08T08:39:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:23.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:23.203684+0000) 2022-09-08T08:39:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:24 smithi055 ceph-mon[97033]: pgmap v21453: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T08:39:24.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:24 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:39:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:24 smithi059 ceph-mon[97450]: pgmap v21453: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T08:39:24.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:24 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:39:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:24.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:24.413681+0000) 2022-09-08T08:39:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:24.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:24.203883+0000) 2022-09-08T08:39:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:25.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:25.413871+0000) 2022-09-08T08:39:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:25.202+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:25.204049+0000) 2022-09-08T08:39:26.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:39:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:39:25.877+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:39:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:26 smithi055 ceph-mon[97033]: pgmap v21454: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T08:39:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:26 smithi059 ceph-mon[97450]: pgmap v21454: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2022-09-08T08:39:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:26.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:26.414165+0000) 2022-09-08T08:39:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:26.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:26.204183+0000) 2022-09-08T08:39:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:27.412+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:27.414355+0000) 2022-09-08T08:39:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:27.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:27.204334+0000) 2022-09-08T08:39:28.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:28 smithi055 ceph-mon[97033]: pgmap v21455: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:39:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:28 smithi059 ceph-mon[97450]: pgmap v21455: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:39:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:28.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:28.414546+0000) 2022-09-08T08:39:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:28.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:28.204501+0000) 2022-09-08T08:39:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:29.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:29.414778+0000) 2022-09-08T08:39:29.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:29.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:29.204775+0000) 2022-09-08T08:39:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:30 smithi055 ceph-mon[97033]: pgmap v21456: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:39:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:30 smithi059 ceph-mon[97450]: pgmap v21456: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:39:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:30.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:30.415030+0000) 2022-09-08T08:39:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:30.203+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:30.204962+0000) 2022-09-08T08:39:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:31.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:31.415265+0000) 2022-09-08T08:39:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:31.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:31.205103+0000) 2022-09-08T08:39:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:32 smithi055 ceph-mon[97033]: pgmap v21457: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:39:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:32 smithi059 ceph-mon[97450]: pgmap v21457: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:39:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:32.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:32.205324+0000) 2022-09-08T08:39:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:32.413+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:32.415463+0000) 2022-09-08T08:39:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:33.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:33.205530+0000) 2022-09-08T08:39:33.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:33.414+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:33.415732+0000) 2022-09-08T08:39:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:34 smithi055 ceph-mon[97033]: pgmap v21458: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:34 smithi059 ceph-mon[97450]: pgmap v21458: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:34.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:34.415942+0000) 2022-09-08T08:39:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:34.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:34.205704+0000) 2022-09-08T08:39:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:35.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:35.416186+0000) 2022-09-08T08:39:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:35.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:35.205854+0000) 2022-09-08T08:39:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:36 smithi055 ceph-mon[97033]: pgmap v21459: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:36 smithi059 ceph-mon[97450]: pgmap v21459: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:36.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:36.416367+0000) 2022-09-08T08:39:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:36.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:36.205997+0000) 2022-09-08T08:39:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:37.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:37.416608+0000) 2022-09-08T08:39:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:37.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:37.206147+0000) 2022-09-08T08:39:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:38 smithi055 ceph-mon[97033]: pgmap v21460: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:39:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:38 smithi059 ceph-mon[97450]: pgmap v21460: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:39:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:38.415+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:38.416848+0000) 2022-09-08T08:39:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:38.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:38.206304+0000) 2022-09-08T08:39:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:39.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:39.417104+0000) 2022-09-08T08:39:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:39.204+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:39.206530+0000) 2022-09-08T08:39:40.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:40 smithi055 ceph-mon[97033]: pgmap v21461: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:40 smithi059 ceph-mon[97450]: pgmap v21461: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:40.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:40.417345+0000) 2022-09-08T08:39:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:40.205+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:40.206708+0000) 2022-09-08T08:39:41.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:39:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:39:40.886+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:39:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:41.416+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:41.417513+0000) 2022-09-08T08:39:41.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:41.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:41.206910+0000) 2022-09-08T08:39:42.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:42 smithi055 ceph-mon[97033]: pgmap v21462: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:39:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:42 smithi059 ceph-mon[97450]: pgmap v21462: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:39:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:42.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:42.417739+0000) 2022-09-08T08:39:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:42.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:42.207101+0000) 2022-09-08T08:39:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:43.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:43.207257+0000) 2022-09-08T08:39:43.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:43.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:43.417944+0000) 2022-09-08T08:39:44.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:44 smithi055 ceph-mon[97033]: pgmap v21463: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:44 smithi059 ceph-mon[97450]: pgmap v21463: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:44.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:44.207388+0000) 2022-09-08T08:39:44.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:44.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:44.418164+0000) 2022-09-08T08:39:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:45.417+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:45.418435+0000) 2022-09-08T08:39:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:45.206+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:45.207537+0000) 2022-09-08T08:39:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:46 smithi055 ceph-mon[97033]: pgmap v21464: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:46 smithi059 ceph-mon[97450]: pgmap v21464: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:46.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:46.207688+0000) 2022-09-08T08:39:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:46.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:46.418682+0000) 2022-09-08T08:39:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:47.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:47.418920+0000) 2022-09-08T08:39:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:47.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:47.207893+0000) 2022-09-08T08:39:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:48 smithi055 ceph-mon[97033]: pgmap v21465: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:39:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:48 smithi059 ceph-mon[97450]: pgmap v21465: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:39:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:48.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:48.419167+0000) 2022-09-08T08:39:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:48.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:48.208127+0000) 2022-09-08T08:39:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:49.418+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:49.419424+0000) 2022-09-08T08:39:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:49.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:49.208345+0000) 2022-09-08T08:39:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:50 smithi055 ceph-mon[97033]: pgmap v21466: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:50 smithi059 ceph-mon[97450]: pgmap v21466: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:50.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:50.419736+0000) 2022-09-08T08:39:50.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:50.207+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:50.208542+0000) 2022-09-08T08:39:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:51.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:51.419953+0000) 2022-09-08T08:39:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:51.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:51.208703+0000) 2022-09-08T08:39:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:52 smithi055 ceph-mon[97033]: pgmap v21467: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:39:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:52 smithi059 ceph-mon[97450]: pgmap v21467: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:39:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:52.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:52.420141+0000) 2022-09-08T08:39:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:52.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:52.208892+0000) 2022-09-08T08:39:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:53.419+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:53.420398+0000) 2022-09-08T08:39:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:53.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:53.209061+0000) 2022-09-08T08:39:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:54 smithi055 ceph-mon[97033]: pgmap v21468: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:54 smithi059 ceph-mon[97450]: pgmap v21468: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:54.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:54.420659+0000) 2022-09-08T08:39:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:54.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:54.209224+0000) 2022-09-08T08:39:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:55.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:55.420874+0000) 2022-09-08T08:39:55.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:55.208+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:55.209404+0000) 2022-09-08T08:39:56.212 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:39:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:39:55.902+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:39:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:56 smithi059 ceph-mon[97450]: pgmap v21469: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:56.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:56.421097+0000) 2022-09-08T08:39:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:56.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:56.209625+0000) 2022-09-08T08:39:56.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:56 smithi055 ceph-mon[97033]: pgmap v21469: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:39:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:57.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:57.421341+0000) 2022-09-08T08:39:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:57.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:57.209870+0000) 2022-09-08T08:39:58.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:39:58 smithi055 ceph-mon[97033]: pgmap v21470: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:39:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:39:58 smithi059 ceph-mon[97450]: pgmap v21470: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:39:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:58.420+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:58.421553+0000) 2022-09-08T08:39:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:58.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:58.210085+0000) 2022-09-08T08:39:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:39:59.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:59.421780+0000) 2022-09-08T08:39:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:39:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:39:59.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:39:59.210299+0000) 2022-09-08T08:40:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:00 smithi059 ceph-mon[97450]: pgmap v21471: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T08:40:00.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T08:40:00.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T08:40:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:00.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:00.421933+0000) 2022-09-08T08:40:00.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:00.209+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:00.210431+0000) 2022-09-08T08:40:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:00 smithi055 ceph-mon[97033]: pgmap v21471: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T08:40:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T08:40:00.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T08:40:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:01.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:01.422151+0000) 2022-09-08T08:40:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:01.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:01.210617+0000) 2022-09-08T08:40:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:02 smithi059 ceph-mon[97450]: pgmap v21472: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:40:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:02.421+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:02.422298+0000) 2022-09-08T08:40:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:02.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:02.210882+0000) 2022-09-08T08:40:02.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:02 smithi055 ceph-mon[97033]: pgmap v21472: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:40:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:03.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:03.422490+0000) 2022-09-08T08:40:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:03.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:03.211035+0000) 2022-09-08T08:40:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:04 smithi059 ceph-mon[97450]: pgmap v21473: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:04.548 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:04 smithi055 ceph-mon[97033]: pgmap v21473: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:04.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:04.422742+0000) 2022-09-08T08:40:04.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:04.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:04.211176+0000) 2022-09-08T08:40:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:05.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:05.422907+0000) 2022-09-08T08:40:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:05.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:05.211375+0000) 2022-09-08T08:40:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:06 smithi059 ceph-mon[97450]: pgmap v21474: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:06.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:40:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:06.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:06.423104+0000) 2022-09-08T08:40:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:06.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:06.211572+0000) 2022-09-08T08:40:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:06 smithi055 ceph-mon[97033]: pgmap v21474: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:40:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:40:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:07.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:07.423280+0000) 2022-09-08T08:40:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:07.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:07.211726+0000) 2022-09-08T08:40:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:40:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:08 smithi059 ceph-mon[97450]: pgmap v21475: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:40:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:08.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:08.423457+0000) 2022-09-08T08:40:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:08.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:08.211872+0000) 2022-09-08T08:40:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:08 smithi055 ceph-mon[97033]: pgmap v21475: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:40:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:09.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:09.423703+0000) 2022-09-08T08:40:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:09.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:09.212105+0000) 2022-09-08T08:40:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:10 smithi059 ceph-mon[97450]: pgmap v21476: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:40:10.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:10 smithi059 ceph-mon[97450]: osdmap e1420: 12 total, 11 up, 11 in 2022-09-08T08:40:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:10.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:10.423837+0000) 2022-09-08T08:40:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:10.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:10.212274+0000) 2022-09-08T08:40:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:10 smithi055 ceph-mon[97033]: pgmap v21476: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:40:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:10 smithi055 ceph-mon[97033]: osdmap e1420: 12 total, 11 up, 11 in 2022-09-08T08:40:11.215 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:40:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:40:10.913+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:40:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:11.422+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:11.424086+0000) 2022-09-08T08:40:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:11.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:11.212427+0000) 2022-09-08T08:40:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:12 smithi059 ceph-mon[97450]: pgmap v21478: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:40:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:12.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:12.424299+0000) 2022-09-08T08:40:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:12.210+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:12.212607+0000) 2022-09-08T08:40:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:12 smithi055 ceph-mon[97033]: pgmap v21478: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:40:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:13.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:13.424543+0000) 2022-09-08T08:40:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:13.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:13.212816+0000) 2022-09-08T08:40:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:14 smithi059 ceph-mon[97450]: pgmap v21479: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:40:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:14.423+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:14.424818+0000) 2022-09-08T08:40:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:14.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:14.212968+0000) 2022-09-08T08:40:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:14 smithi055 ceph-mon[97033]: pgmap v21479: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:40:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:15.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:15.425081+0000) 2022-09-08T08:40:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:15.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:15.213139+0000) 2022-09-08T08:40:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:16 smithi059 ceph-mon[97450]: pgmap v21480: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:40:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:16.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:16.425281+0000) 2022-09-08T08:40:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:16.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:16.213322+0000) 2022-09-08T08:40:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:16 smithi055 ceph-mon[97033]: pgmap v21480: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:40:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:17.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:17.425476+0000) 2022-09-08T08:40:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:17.211+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:17.213554+0000) 2022-09-08T08:40:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:18 smithi059 ceph-mon[97450]: pgmap v21481: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:40:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:18.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:18.213722+0000) 2022-09-08T08:40:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:18.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:18.425744+0000) 2022-09-08T08:40:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:18 smithi055 ceph-mon[97033]: pgmap v21481: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:40:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:19.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:19.425989+0000) 2022-09-08T08:40:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:19.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:19.213897+0000) 2022-09-08T08:40:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:19 smithi055 ceph-mon[97033]: pgmap v21482: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:40:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:19 smithi055 ceph-mon[97033]: osdmap e1421: 12 total, 11 up, 11 in 2022-09-08T08:40:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:19 smithi059 ceph-mon[97450]: pgmap v21482: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:40:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:19 smithi059 ceph-mon[97450]: osdmap e1421: 12 total, 11 up, 11 in 2022-09-08T08:40:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:20.424+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:20.426214+0000) 2022-09-08T08:40:20.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:20.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:20.214075+0000) 2022-09-08T08:40:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:21.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:21.426455+0000) 2022-09-08T08:40:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:21.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:21.214253+0000) 2022-09-08T08:40:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:21 smithi055 ceph-mon[97033]: pgmap v21484: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:40:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:21 smithi059 ceph-mon[97450]: pgmap v21484: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:40:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:22.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:22.426668+0000) 2022-09-08T08:40:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:22.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:22.214389+0000) 2022-09-08T08:40:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:23.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:23.426878+0000) 2022-09-08T08:40:23.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:23.212+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:23.214540+0000) 2022-09-08T08:40:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:23 smithi055 ceph-mon[97033]: pgmap v21485: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:40:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:40:24.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:40:24.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:40:24.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:40:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:23 smithi059 ceph-mon[97450]: pgmap v21485: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:40:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:40:24.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:40:24.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:40:24.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:40:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:24.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:24.427115+0000) 2022-09-08T08:40:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:24.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:24.214739+0000) 2022-09-08T08:40:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:25.425+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:25.427334+0000) 2022-09-08T08:40:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:25.213+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:25.214872+0000) 2022-09-08T08:40:26.217 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:40:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:40:25.922+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:40:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:25 smithi055 ceph-mon[97033]: pgmap v21486: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:40:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:25 smithi059 ceph-mon[97450]: pgmap v21486: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:40:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:26.426+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:26.427623+0000) 2022-09-08T08:40:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:26.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:26.215009+0000) 2022-09-08T08:40:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:27.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:27.427940+0000) 2022-09-08T08:40:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:27.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:27.215137+0000) 2022-09-08T08:40:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:27 smithi055 ceph-mon[97033]: pgmap v21487: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:40:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:27 smithi059 ceph-mon[97450]: pgmap v21487: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:40:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:28.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:28.428158+0000) 2022-09-08T08:40:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:28.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:28.215323+0000) 2022-09-08T08:40:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:29.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:29.428379+0000) 2022-09-08T08:40:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:29.214+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:29.215511+0000) 2022-09-08T08:40:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:29 smithi055 ceph-mon[97033]: pgmap v21488: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:40:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:29 smithi059 ceph-mon[97450]: pgmap v21488: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:40:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:30.427+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:30.428613+0000) 2022-09-08T08:40:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:30.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:30.215696+0000) 2022-09-08T08:40:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:31.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:31.428895+0000) 2022-09-08T08:40:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:31.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:31.215835+0000) 2022-09-08T08:40:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:31 smithi055 ceph-mon[97033]: pgmap v21489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:40:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:31 smithi059 ceph-mon[97450]: pgmap v21489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:40:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:32.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:32.429140+0000) 2022-09-08T08:40:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:32.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:32.215968+0000) 2022-09-08T08:40:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:33.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:33.429363+0000) 2022-09-08T08:40:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:33.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:33.216102+0000) 2022-09-08T08:40:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:33 smithi055 ceph-mon[97033]: pgmap v21490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:33 smithi059 ceph-mon[97450]: pgmap v21490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:34.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:34.428+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:34.429581+0000) 2022-09-08T08:40:34.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:34.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:34.216310+0000) 2022-09-08T08:40:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:35.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:35.429856+0000) 2022-09-08T08:40:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:35.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:35.216459+0000) 2022-09-08T08:40:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:35 smithi055 ceph-mon[97033]: pgmap v21491: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:35 smithi059 ceph-mon[97450]: pgmap v21491: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:36.215+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:36.216591+0000) 2022-09-08T08:40:36.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:36.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:36.430114+0000) 2022-09-08T08:40:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:37.429+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:37.430348+0000) 2022-09-08T08:40:37.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:37.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:37.216787+0000) 2022-09-08T08:40:38.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:37 smithi055 ceph-mon[97033]: pgmap v21492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:40:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:37 smithi059 ceph-mon[97450]: pgmap v21492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:40:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:38.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:38.430537+0000) 2022-09-08T08:40:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:38.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:38.216927+0000) 2022-09-08T08:40:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:39.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:39.430719+0000) 2022-09-08T08:40:39.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:39.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:39.217102+0000) 2022-09-08T08:40:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:39 smithi055 ceph-mon[97033]: pgmap v21493: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:39 smithi059 ceph-mon[97450]: pgmap v21493: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:40.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:40.430946+0000) 2022-09-08T08:40:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:40.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:40.217335+0000) 2022-09-08T08:40:41.220 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:40:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:40:40.928+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:40:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:41.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:41.431115+0000) 2022-09-08T08:40:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:41.216+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:41.217541+0000) 2022-09-08T08:40:42.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:41 smithi055 ceph-mon[97033]: pgmap v21494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:40:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:41 smithi059 ceph-mon[97450]: pgmap v21494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:40:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:42.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:42.217787+0000) 2022-09-08T08:40:42.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:42.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:42.431334+0000) 2022-09-08T08:40:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:43.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:43.217974+0000) 2022-09-08T08:40:43.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:43.430+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:43.431580+0000) 2022-09-08T08:40:44.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:43 smithi055 ceph-mon[97033]: pgmap v21495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:43 smithi059 ceph-mon[97450]: pgmap v21495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:44.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:44.431759+0000) 2022-09-08T08:40:44.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:44.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:44.218185+0000) 2022-09-08T08:40:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:45.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:45.431996+0000) 2022-09-08T08:40:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:45.217+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:45.218387+0000) 2022-09-08T08:40:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:45 smithi055 ceph-mon[97033]: pgmap v21496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:45 smithi059 ceph-mon[97450]: pgmap v21496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:46.431+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:46.432256+0000) 2022-09-08T08:40:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:46.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:46.218584+0000) 2022-09-08T08:40:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:47.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:47.432487+0000) 2022-09-08T08:40:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:47.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:47.218817+0000) 2022-09-08T08:40:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:47 smithi055 ceph-mon[97033]: pgmap v21497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:40:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:47 smithi059 ceph-mon[97450]: pgmap v21497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:40:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:48.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:48.432724+0000) 2022-09-08T08:40:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:48.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:48.219015+0000) 2022-09-08T08:40:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:49.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:49.432973+0000) 2022-09-08T08:40:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:49.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:49.219153+0000) 2022-09-08T08:40:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:49 smithi055 ceph-mon[97033]: pgmap v21498: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:49 smithi059 ceph-mon[97450]: pgmap v21498: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:50.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:50.219313+0000) 2022-09-08T08:40:50.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:50.432+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:50.433238+0000) 2022-09-08T08:40:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:51.218+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:51.219460+0000) 2022-09-08T08:40:51.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:51.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:51.433478+0000) 2022-09-08T08:40:52.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:51 smithi055 ceph-mon[97033]: pgmap v21499: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:40:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:51 smithi059 ceph-mon[97450]: pgmap v21499: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:40:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:52.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:52.433678+0000) 2022-09-08T08:40:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:52.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:52.219688+0000) 2022-09-08T08:40:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:53.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:53.433905+0000) 2022-09-08T08:40:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:53.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:53.219946+0000) 2022-09-08T08:40:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:53 smithi059 ceph-mon[97450]: pgmap v21500: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:54.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:53 smithi055 ceph-mon[97033]: pgmap v21500: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:54.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:54.220150+0000) 2022-09-08T08:40:54.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:54.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:54.434110+0000) 2022-09-08T08:40:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:55.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:55.434301+0000) 2022-09-08T08:40:55.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:55.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:55.220293+0000) 2022-09-08T08:40:56.223 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:40:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:40:55.944+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:40:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:55 smithi059 ceph-mon[97450]: pgmap v21501: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:56.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:55 smithi055 ceph-mon[97033]: pgmap v21501: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:40:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:56.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:56.434514+0000) 2022-09-08T08:40:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:56.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:56.220444+0000) 2022-09-08T08:40:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:57.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:57.434713+0000) 2022-09-08T08:40:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:57.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:57.220689+0000) 2022-09-08T08:40:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:57 smithi059 ceph-mon[97450]: pgmap v21502: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:40:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:57 smithi055 ceph-mon[97033]: pgmap v21502: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:40:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:58.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:58.434945+0000) 2022-09-08T08:40:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:58.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:58.220879+0000) 2022-09-08T08:40:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:40:59.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:59.435132+0000) 2022-09-08T08:40:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:40:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:40:59.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:40:59.221040+0000) 2022-09-08T08:41:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:40:59 smithi059 ceph-mon[97450]: pgmap v21503: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:41:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:40:59 smithi055 ceph-mon[97033]: pgmap v21503: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:41:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:00.433+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:00.435287+0000) 2022-09-08T08:41:00.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:00.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:00.221258+0000) 2022-09-08T08:41:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:01.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:01.435477+0000) 2022-09-08T08:41:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:01.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:01.221409+0000) 2022-09-08T08:41:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:01 smithi059 ceph-mon[97450]: pgmap v21504: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:41:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:01 smithi055 ceph-mon[97033]: pgmap v21504: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:41:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:02.219+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:02.221588+0000) 2022-09-08T08:41:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:02.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:02.435678+0000) 2022-09-08T08:41:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:03.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:03.221816+0000) 2022-09-08T08:41:03.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:03.434+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:03.435815+0000) 2022-09-08T08:41:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:03 smithi059 ceph-mon[97450]: pgmap v21505: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:41:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:03 smithi055 ceph-mon[97033]: pgmap v21505: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:41:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:04.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:04.435995+0000) 2022-09-08T08:41:04.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:04.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:04.221955+0000) 2022-09-08T08:41:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:05.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:05.436238+0000) 2022-09-08T08:41:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:05.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:05.222116+0000) 2022-09-08T08:41:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:05 smithi059 ceph-mon[97450]: pgmap v21506: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-08T08:41:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:05 smithi055 ceph-mon[97033]: pgmap v21506: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 22 KiB/s wr, 7 op/s 2022-09-08T08:41:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:06.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:06.436497+0000) 2022-09-08T08:41:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:06.220+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:06.222309+0000) 2022-09-08T08:41:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:41:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:41:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:07.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:07.436758+0000) 2022-09-08T08:41:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:07.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:07.222459+0000) 2022-09-08T08:41:08.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:08 smithi059 ceph-mon[97450]: pgmap v21507: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:41:08.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:41:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:08 smithi055 ceph-mon[97033]: pgmap v21507: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:41:08.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:41:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:08.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:08.436949+0000) 2022-09-08T08:41:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:08.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:08.222708+0000) 2022-09-08T08:41:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:09.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:09.437158+0000) 2022-09-08T08:41:09.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:09.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:09.222920+0000) 2022-09-08T08:41:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:10 smithi059 ceph-mon[97450]: pgmap v21508: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:41:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:10 smithi055 ceph-mon[97033]: pgmap v21508: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:41:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:10.435+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:10.437393+0000) 2022-09-08T08:41:10.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:10.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:10.223061+0000) 2022-09-08T08:41:11.226 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:41:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:41:10.950+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:41:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:11 smithi059 ceph-mon[97450]: osdmap e1422: 12 total, 11 up, 11 in 2022-09-08T08:41:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:11 smithi055 ceph-mon[97033]: osdmap e1422: 12 total, 11 up, 11 in 2022-09-08T08:41:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:11.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:11.437605+0000) 2022-09-08T08:41:11.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:11.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:11.223224+0000) 2022-09-08T08:41:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:12 smithi059 ceph-mon[97450]: pgmap v21510: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:41:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:12 smithi055 ceph-mon[97033]: pgmap v21510: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:41:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:12.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:12.437849+0000) 2022-09-08T08:41:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:12.221+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:12.223405+0000) 2022-09-08T08:41:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:13.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:13.438056+0000) 2022-09-08T08:41:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:13.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:13.223789+0000) 2022-09-08T08:41:14.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:14 smithi055 ceph-mon[97033]: pgmap v21511: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:41:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:14 smithi059 ceph-mon[97450]: pgmap v21511: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:41:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:14.437+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:14.438306+0000) 2022-09-08T08:41:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:14.222+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:14.223917+0000) 2022-09-08T08:41:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:15.436+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:15.438561+0000) 2022-09-08T08:41:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:15.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:15.224093+0000) 2022-09-08T08:41:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:16 smithi055 ceph-mon[97033]: pgmap v21512: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-08T08:41:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:16 smithi059 ceph-mon[97450]: pgmap v21512: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-08T08:41:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:16.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:16.438802+0000) 2022-09-08T08:41:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:16.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:16.224257+0000) 2022-09-08T08:41:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:17.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:17.224401+0000) 2022-09-08T08:41:17.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:17.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:17.438996+0000) 2022-09-08T08:41:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:18 smithi055 ceph-mon[97033]: pgmap v21513: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:41:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:18 smithi059 ceph-mon[97450]: pgmap v21513: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:41:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:18.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:18.439252+0000) 2022-09-08T08:41:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:18.223+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:18.224541+0000) 2022-09-08T08:41:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:19.438+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:19.439549+0000) 2022-09-08T08:41:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:19.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:19.224700+0000) 2022-09-08T08:41:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:19 smithi055 ceph-mon[97033]: pgmap v21514: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:41:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:19 smithi055 ceph-mon[97033]: osdmap e1423: 12 total, 11 up, 11 in 2022-09-08T08:41:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:19 smithi059 ceph-mon[97450]: pgmap v21514: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:41:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:19 smithi059 ceph-mon[97450]: osdmap e1423: 12 total, 11 up, 11 in 2022-09-08T08:41:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:20.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:20.439759+0000) 2022-09-08T08:41:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:20.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:20.224855+0000) 2022-09-08T08:41:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:21.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:21.439971+0000) 2022-09-08T08:41:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:21.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:21.225006+0000) 2022-09-08T08:41:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:21 smithi055 ceph-mon[97033]: pgmap v21516: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:41:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:21 smithi059 ceph-mon[97450]: pgmap v21516: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:41:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:22.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:22.440163+0000) 2022-09-08T08:41:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:22.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:22.225158+0000) 2022-09-08T08:41:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:23.439+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:23.440389+0000) 2022-09-08T08:41:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:23.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:23.225367+0000) 2022-09-08T08:41:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:23 smithi055 ceph-mon[97033]: pgmap v21517: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:41:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:41:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:41:24.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:23 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:41:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:23 smithi059 ceph-mon[97450]: pgmap v21517: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:41:24.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:41:24.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:41:24.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:23 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:41:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:24.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:24.440628+0000) 2022-09-08T08:41:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:24.224+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:24.225533+0000) 2022-09-08T08:41:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:25.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:25.440923+0000) 2022-09-08T08:41:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:25.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:25.225767+0000) 2022-09-08T08:41:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:41:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:25 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:41:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:41:25.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:25 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:41:26.228 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:41:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:41:25.960+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:41:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:26.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:26.441128+0000) 2022-09-08T08:41:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:26.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:26.225944+0000) 2022-09-08T08:41:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:26 smithi055 ceph-mon[97033]: pgmap v21518: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.2 KiB/s rd, 102 B/s wr, 7 op/s 2022-09-08T08:41:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:26 smithi059 ceph-mon[97450]: pgmap v21518: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.2 KiB/s rd, 102 B/s wr, 7 op/s 2022-09-08T08:41:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:27.440+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:27.441400+0000) 2022-09-08T08:41:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:27.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:27.226099+0000) 2022-09-08T08:41:27.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: pgmap v21519: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:41:27.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:41:27.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:41:27.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:41:27.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:41:27.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:27.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:27.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:27.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:27.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 89046k 2022-09-08T08:41:27.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 91183550: error parsing value: Value '91183550' is below minimum 939524096 2022-09-08T08:41:27.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:27.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:27.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:27.986 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:27.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T08:41:27.987 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:41:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: pgmap v21519: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:41:28.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:41:28.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:41:28.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:41:28.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:41:28.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:28.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:28.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:28.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:28.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 89046k 2022-09-08T08:41:28.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 91183550: error parsing value: Value '91183550' is below minimum 939524096 2022-09-08T08:41:28.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:28.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:28.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:28.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:41:28.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T08:41:28.038 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:41:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:28.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:28.441607+0000) 2022-09-08T08:41:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:28.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:28.226274+0000) 2022-09-08T08:41:29.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:28 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T08:41:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:28 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T08:41:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:29.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:29.441896+0000) 2022-09-08T08:41:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:29.225+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:29.226459+0000) 2022-09-08T08:41:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:29 smithi055 ceph-mon[97033]: pgmap v21520: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:41:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:29 smithi059 ceph-mon[97450]: pgmap v21520: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:41:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:30.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:30.442139+0000) 2022-09-08T08:41:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:30.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:30.226665+0000) 2022-09-08T08:41:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:31.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:31.226805+0000) 2022-09-08T08:41:31.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:31.441+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:31.442274+0000) 2022-09-08T08:41:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:31 smithi055 ceph-mon[97033]: pgmap v21521: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:41:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:31 smithi059 ceph-mon[97450]: pgmap v21521: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:41:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:32.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:32.442472+0000) 2022-09-08T08:41:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:32.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:32.226955+0000) 2022-09-08T08:41:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:33.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:33.227162+0000) 2022-09-08T08:41:33.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:33.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:33.442728+0000) 2022-09-08T08:41:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:33 smithi055 ceph-mon[97033]: pgmap v21522: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:41:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:33 smithi059 ceph-mon[97450]: pgmap v21522: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:41:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:34.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:34.442953+0000) 2022-09-08T08:41:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:34.226+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:34.227381+0000) 2022-09-08T08:41:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:35.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:35.443132+0000) 2022-09-08T08:41:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:35.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:35.227632+0000) 2022-09-08T08:41:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:35 smithi055 ceph-mon[97033]: pgmap v21523: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:41:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:35 smithi059 ceph-mon[97450]: pgmap v21523: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T08:41:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:36.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:36.443374+0000) 2022-09-08T08:41:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:36.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:36.227825+0000) 2022-09-08T08:41:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:37.442+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:37.443550+0000) 2022-09-08T08:41:37.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:37.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:37.227988+0000) 2022-09-08T08:41:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:37 smithi055 ceph-mon[97033]: pgmap v21524: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T08:41:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:37 smithi059 ceph-mon[97450]: pgmap v21524: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T08:41:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:38.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:38.443782+0000) 2022-09-08T08:41:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:38.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:38.228185+0000) 2022-09-08T08:41:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:39.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:39.444037+0000) 2022-09-08T08:41:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:39.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:39.228395+0000) 2022-09-08T08:41:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:39 smithi055 ceph-mon[97033]: pgmap v21525: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:41:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:39 smithi059 ceph-mon[97450]: pgmap v21525: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:41:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:40.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:40.444259+0000) 2022-09-08T08:41:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:40.227+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:40.228544+0000) 2022-09-08T08:41:41.231 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:41:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:41:40.976+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:41:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:41.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:41.444530+0000) 2022-09-08T08:41:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:41.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:41.228687+0000) 2022-09-08T08:41:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:41 smithi055 ceph-mon[97033]: pgmap v21526: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:41:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:41 smithi059 ceph-mon[97450]: pgmap v21526: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:41:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:42.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:42.444754+0000) 2022-09-08T08:41:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:42.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:42.228865+0000) 2022-09-08T08:41:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:43.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:43.444896+0000) 2022-09-08T08:41:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:43.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:43.229072+0000) 2022-09-08T08:41:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:43 smithi055 ceph-mon[97033]: pgmap v21527: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:41:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:43 smithi059 ceph-mon[97450]: pgmap v21527: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:41:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:44.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:44.445061+0000) 2022-09-08T08:41:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:44.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:44.229272+0000) 2022-09-08T08:41:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:45.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:45.445339+0000) 2022-09-08T08:41:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:45.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:45.229432+0000) 2022-09-08T08:41:46.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:45 smithi055 ceph-mon[97033]: pgmap v21528: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:41:46.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:45 smithi059 ceph-mon[97450]: pgmap v21528: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:41:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:46.443+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:46.445562+0000) 2022-09-08T08:41:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:46.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:46.229576+0000) 2022-09-08T08:41:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:47.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:47.445781+0000) 2022-09-08T08:41:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:47.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:47.229704+0000) 2022-09-08T08:41:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:47 smithi055 ceph-mon[97033]: pgmap v21529: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T08:41:48.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:47 smithi059 ceph-mon[97450]: pgmap v21529: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T08:41:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:48.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:48.445903+0000) 2022-09-08T08:41:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:48.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:48.229913+0000) 2022-09-08T08:41:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:49.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:49.446125+0000) 2022-09-08T08:41:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:49.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:49.230064+0000) 2022-09-08T08:41:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:49 smithi059 ceph-mon[97450]: pgmap v21530: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:41:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:49 smithi055 ceph-mon[97033]: pgmap v21530: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:41:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:50.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:50.446364+0000) 2022-09-08T08:41:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:50.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:50.230215+0000) 2022-09-08T08:41:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:51.444+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:51.446587+0000) 2022-09-08T08:41:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:51.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:51.230419+0000) 2022-09-08T08:41:52.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:51 smithi059 ceph-mon[97450]: pgmap v21531: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:41:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:51 smithi055 ceph-mon[97033]: pgmap v21531: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:41:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:52.445+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:52.446812+0000) 2022-09-08T08:41:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:52.228+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:52.230592+0000) 2022-09-08T08:41:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:53.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:53.447056+0000) 2022-09-08T08:41:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:53.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:53.230762+0000) 2022-09-08T08:41:54.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:53 smithi059 ceph-mon[97450]: pgmap v21532: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:41:54.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:53 smithi055 ceph-mon[97033]: pgmap v21532: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:41:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:54.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:54.447279+0000) 2022-09-08T08:41:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:54.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:54.230932+0000) 2022-09-08T08:41:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:55.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:55.447513+0000) 2022-09-08T08:41:55.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:55.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:55.231106+0000) 2022-09-08T08:41:56.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:55 smithi059 ceph-mon[97450]: pgmap v21533: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:41:56.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:41:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:41:55.990+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:41:56.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:56.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:56.231299+0000) 2022-09-08T08:41:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:55 smithi055 ceph-mon[97033]: pgmap v21533: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:41:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:56.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:56.447743+0000) 2022-09-08T08:41:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:57.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:57.447992+0000) 2022-09-08T08:41:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:57.229+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:57.231538+0000) 2022-09-08T08:41:58.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:57 smithi059 ceph-mon[97450]: pgmap v21534: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:41:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:41:57 smithi055 ceph-mon[97033]: pgmap v21534: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:41:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:58.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:58.448160+0000) 2022-09-08T08:41:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:58.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:58.231739+0000) 2022-09-08T08:41:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:41:59.447+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:59.448319+0000) 2022-09-08T08:41:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:41:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:41:59.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:41:59.231916+0000) 2022-09-08T08:42:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:41:59 smithi059 ceph-mon[97450]: pgmap v21535: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:42:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:00 smithi055 ceph-mon[97033]: pgmap v21535: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:42:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:00.446+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:00.448577+0000) 2022-09-08T08:42:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:00.230+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:00.232058+0000) 2022-09-08T08:42:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:01.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:01.448783+0000) 2022-09-08T08:42:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:01.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:01.232284+0000) 2022-09-08T08:42:02.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:02 smithi059 ceph-mon[97450]: pgmap v21536: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:42:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:02 smithi055 ceph-mon[97033]: pgmap v21536: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:42:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:02.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:02.448985+0000) 2022-09-08T08:42:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:02.231+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:02.232498+0000) 2022-09-08T08:42:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:03.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:03.449238+0000) 2022-09-08T08:42:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:03.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:03.232755+0000) 2022-09-08T08:42:04.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:04 smithi059 ceph-mon[97450]: pgmap v21537: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:42:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:04 smithi055 ceph-mon[97033]: pgmap v21537: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:42:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:04.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:04.449377+0000) 2022-09-08T08:42:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:04.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:04.232963+0000) 2022-09-08T08:42:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:05.448+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:05.449586+0000) 2022-09-08T08:42:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:05.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:05.233196+0000) 2022-09-08T08:42:06.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:06 smithi059 ceph-mon[97450]: pgmap v21538: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T08:42:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:06 smithi055 ceph-mon[97033]: pgmap v21538: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T08:42:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:06.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:06.449855+0000) 2022-09-08T08:42:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:06.232+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:06.233469+0000) 2022-09-08T08:42:07.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:42:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:42:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:07.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:07.450107+0000) 2022-09-08T08:42:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:07.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:07.233750+0000) 2022-09-08T08:42:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:08 smithi059 ceph-mon[97450]: pgmap v21539: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T08:42:08.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:42:08.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:08 smithi055 ceph-mon[97033]: pgmap v21539: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T08:42:08.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:42:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:08.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:08.450371+0000) 2022-09-08T08:42:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:08.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:08.234003+0000) 2022-09-08T08:42:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:09.449+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:09.450615+0000) 2022-09-08T08:42:09.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:09.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:09.234199+0000) 2022-09-08T08:42:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:10 smithi055 ceph-mon[97033]: pgmap v21540: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:42:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:10 smithi059 ceph-mon[97450]: pgmap v21540: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:42:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:10.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:10.450833+0000) 2022-09-08T08:42:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:10.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:10.234357+0000) 2022-09-08T08:42:11.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:42:11.007+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:42:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:11.233+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:11.234543+0000) 2022-09-08T08:42:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:11 smithi055 ceph-mon[97033]: osdmap e1424: 12 total, 11 up, 11 in 2022-09-08T08:42:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:11 smithi059 ceph-mon[97450]: osdmap e1424: 12 total, 11 up, 11 in 2022-09-08T08:42:11.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:11.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:11.451065+0000) 2022-09-08T08:42:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:12 smithi055 ceph-mon[97033]: pgmap v21542: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:42:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:12 smithi059 ceph-mon[97450]: pgmap v21542: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:42:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:12.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:12.451208+0000) 2022-09-08T08:42:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:12.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:12.234805+0000) 2022-09-08T08:42:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:13.450+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:13.451445+0000) 2022-09-08T08:42:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:13.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:13.235003+0000) 2022-09-08T08:42:14.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:14 smithi055 ceph-mon[97033]: pgmap v21543: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:42:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:14 smithi059 ceph-mon[97450]: pgmap v21543: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:42:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:14.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:14.451734+0000) 2022-09-08T08:42:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:14.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:14.235176+0000) 2022-09-08T08:42:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:15.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:15.452001+0000) 2022-09-08T08:42:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:15.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:15.235339+0000) 2022-09-08T08:42:16.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:16 smithi055 ceph-mon[97033]: pgmap v21544: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:42:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:16 smithi059 ceph-mon[97450]: pgmap v21544: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:42:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:16.451+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:16.452248+0000) 2022-09-08T08:42:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:16.234+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:16.235513+0000) 2022-09-08T08:42:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:17.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:17.452481+0000) 2022-09-08T08:42:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:17.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:17.235683+0000) 2022-09-08T08:42:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:18 smithi055 ceph-mon[97033]: pgmap v21545: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:42:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:18 smithi059 ceph-mon[97450]: pgmap v21545: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:42:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:18.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:18.452769+0000) 2022-09-08T08:42:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:18.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:18.235877+0000) 2022-09-08T08:42:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:19.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:19.236028+0000) 2022-09-08T08:42:19.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:19.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:19.453038+0000) 2022-09-08T08:42:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:19 smithi055 ceph-mon[97033]: pgmap v21546: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:42:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:19 smithi055 ceph-mon[97033]: osdmap e1425: 12 total, 11 up, 11 in 2022-09-08T08:42:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:19 smithi059 ceph-mon[97450]: pgmap v21546: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:42:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:19 smithi059 ceph-mon[97450]: osdmap e1425: 12 total, 11 up, 11 in 2022-09-08T08:42:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:20.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:20.236236+0000) 2022-09-08T08:42:20.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:20.452+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:20.453237+0000) 2022-09-08T08:42:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:21.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:21.453481+0000) 2022-09-08T08:42:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:21.235+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:21.236452+0000) 2022-09-08T08:42:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:21 smithi055 ceph-mon[97033]: pgmap v21548: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:42:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:21 smithi059 ceph-mon[97450]: pgmap v21548: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:42:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:22.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:22.453676+0000) 2022-09-08T08:42:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:22.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:22.236705+0000) 2022-09-08T08:42:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:23.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:23.453885+0000) 2022-09-08T08:42:23.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:23.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:23.236857+0000) 2022-09-08T08:42:24.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:23 smithi055 ceph-mon[97033]: pgmap v21549: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:42:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:23 smithi059 ceph-mon[97450]: pgmap v21549: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:42:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:24.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:24.454128+0000) 2022-09-08T08:42:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:24.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:24.237067+0000) 2022-09-08T08:42:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:25.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:25.454360+0000) 2022-09-08T08:42:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:25.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:25.237200+0000) 2022-09-08T08:42:26.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:25 smithi055 ceph-mon[97033]: pgmap v21550: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:42:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:25 smithi059 ceph-mon[97450]: pgmap v21550: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:42:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:26.236+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:26.237368+0000) 2022-09-08T08:42:26.405 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:42:26.020+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:42:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:26.453+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:26.454582+0000) 2022-09-08T08:42:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:27.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:27.454872+0000) 2022-09-08T08:42:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:27.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:27.237516+0000) 2022-09-08T08:42:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:27 smithi055 ceph-mon[97033]: pgmap v21551: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:42:28.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:42:28.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:42:28.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:27 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:42:28.259 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:27 smithi059 ceph-mon[97450]: pgmap v21551: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:42:28.259 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:42:28.260 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:42:28.260 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:27 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:42:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:28.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:28.455058+0000) 2022-09-08T08:42:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:28.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:28.237691+0000) 2022-09-08T08:42:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:42:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:42:29.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:29 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:42:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:29.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:29.455242+0000) 2022-09-08T08:42:29.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:29.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:29.237878+0000) 2022-09-08T08:42:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:42:29.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:42:29.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:29 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:42:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:30 smithi059 ceph-mon[97450]: pgmap v21552: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:42:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:30.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:30.455446+0000) 2022-09-08T08:42:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:30.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:30.238027+0000) 2022-09-08T08:42:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:30 smithi055 ceph-mon[97033]: pgmap v21552: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:42:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:31.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:31.455586+0000) 2022-09-08T08:42:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:31.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:31.238196+0000) 2022-09-08T08:42:32.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:32 smithi059 ceph-mon[97450]: pgmap v21553: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:42:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:32.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:32.455776+0000) 2022-09-08T08:42:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:32.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:32.238397+0000) 2022-09-08T08:42:32.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:32 smithi055 ceph-mon[97033]: pgmap v21553: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:42:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:33.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:33.456036+0000) 2022-09-08T08:42:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:33.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:33.238534+0000) 2022-09-08T08:42:34.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:34 smithi059 ceph-mon[97450]: pgmap v21554: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:42:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:34.454+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:34.456248+0000) 2022-09-08T08:42:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:34.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:34.238706+0000) 2022-09-08T08:42:34.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:34 smithi055 ceph-mon[97033]: pgmap v21554: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:42:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:35.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:35.456462+0000) 2022-09-08T08:42:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:35.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:35.238906+0000) 2022-09-08T08:42:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:36 smithi059 ceph-mon[97450]: pgmap v21555: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T08:42:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:36.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:36.456652+0000) 2022-09-08T08:42:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:36.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:36.239123+0000) 2022-09-08T08:42:36.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:36 smithi055 ceph-mon[97033]: pgmap v21555: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T08:42:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:37.455+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:37.456796+0000) 2022-09-08T08:42:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:37.237+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:37.239303+0000) 2022-09-08T08:42:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:38 smithi059 ceph-mon[97450]: pgmap v21556: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-08T08:42:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:38.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:38.457057+0000) 2022-09-08T08:42:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:38.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:38.239520+0000) 2022-09-08T08:42:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:38 smithi055 ceph-mon[97033]: pgmap v21556: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-08T08:42:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:39.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:39.457280+0000) 2022-09-08T08:42:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:39.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:39.239698+0000) 2022-09-08T08:42:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:40 smithi059 ceph-mon[97450]: pgmap v21557: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:42:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:40.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:40.457525+0000) 2022-09-08T08:42:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:40.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:40.239942+0000) 2022-09-08T08:42:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:40 smithi055 ceph-mon[97033]: pgmap v21557: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:42:41.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:42:41.025+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:42:41.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:41.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:41.240114+0000) 2022-09-08T08:42:41.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:41.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:41.457726+0000) 2022-09-08T08:42:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:42 smithi059 ceph-mon[97450]: pgmap v21558: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:42:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:42.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:42.457990+0000) 2022-09-08T08:42:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:42.238+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:42.240341+0000) 2022-09-08T08:42:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:42 smithi055 ceph-mon[97033]: pgmap v21558: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:42:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:43.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:43.458271+0000) 2022-09-08T08:42:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:43.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:43.240517+0000) 2022-09-08T08:42:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:44 smithi059 ceph-mon[97450]: pgmap v21559: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:42:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:44.456+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:44.458473+0000) 2022-09-08T08:42:44.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:44.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:44.240688+0000) 2022-09-08T08:42:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:44 smithi055 ceph-mon[97033]: pgmap v21559: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:42:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:45.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:45.458739+0000) 2022-09-08T08:42:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:45.239+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:45.240873+0000) 2022-09-08T08:42:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:46 smithi059 ceph-mon[97450]: pgmap v21560: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T08:42:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:46.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:46.458924+0000) 2022-09-08T08:42:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:46.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:46.241048+0000) 2022-09-08T08:42:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:46 smithi055 ceph-mon[97033]: pgmap v21560: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T08:42:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:47.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:47.459145+0000) 2022-09-08T08:42:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:47.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:47.241202+0000) 2022-09-08T08:42:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:48 smithi059 ceph-mon[97450]: pgmap v21561: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:42:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:48.457+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:48.459355+0000) 2022-09-08T08:42:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:48.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:48.241355+0000) 2022-09-08T08:42:48.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:48 smithi055 ceph-mon[97033]: pgmap v21561: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:42:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:49.458+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:49.459601+0000) 2022-09-08T08:42:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:49.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:49.241574+0000) 2022-09-08T08:42:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:50 smithi059 ceph-mon[97450]: pgmap v21562: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:42:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:50.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:50.459853+0000) 2022-09-08T08:42:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:50.240+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:50.241762+0000) 2022-09-08T08:42:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:50 smithi055 ceph-mon[97033]: pgmap v21562: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:42:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:51.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:51.460012+0000) 2022-09-08T08:42:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:51.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:51.241978+0000) 2022-09-08T08:42:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:52 smithi059 ceph-mon[97450]: pgmap v21563: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:42:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:52.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:52.460204+0000) 2022-09-08T08:42:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:52.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:52.242155+0000) 2022-09-08T08:42:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:52 smithi055 ceph-mon[97033]: pgmap v21563: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:42:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:53.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:53.242384+0000) 2022-09-08T08:42:53.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:53.459+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:53.460399+0000) 2022-09-08T08:42:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:54 smithi059 ceph-mon[97450]: pgmap v21564: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:42:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:54.241+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:54.242575+0000) 2022-09-08T08:42:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:54.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:54.460628+0000) 2022-09-08T08:42:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:54 smithi055 ceph-mon[97033]: pgmap v21564: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:42:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:55.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:55.242740+0000) 2022-09-08T08:42:55.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:55.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:55.460810+0000) 2022-09-08T08:42:56.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:42:56.027+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:42:56.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:56.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:56.242949+0000) 2022-09-08T08:42:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:56 smithi055 ceph-mon[97033]: pgmap v21565: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:42:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:56 smithi059 ceph-mon[97450]: pgmap v21565: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:42:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:56.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:56.460962+0000) 2022-09-08T08:42:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:57.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:57.461196+0000) 2022-09-08T08:42:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:57.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:57.243196+0000) 2022-09-08T08:42:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:58.460+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:58.461428+0000) 2022-09-08T08:42:58.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:58.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:58.243378+0000) 2022-09-08T08:42:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:42:58 smithi055 ceph-mon[97033]: pgmap v21566: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:42:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:42:58 smithi059 ceph-mon[97450]: pgmap v21566: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:42:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:42:59.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:59.461696+0000) 2022-09-08T08:42:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:42:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:42:59.242+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:42:59.243532+0000) 2022-09-08T08:43:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:00.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:00.461920+0000) 2022-09-08T08:43:00.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:00.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:00.243673+0000) 2022-09-08T08:43:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:00 smithi055 ceph-mon[97033]: pgmap v21567: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:43:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:00 smithi059 ceph-mon[97450]: pgmap v21567: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:43:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:01.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:01.462176+0000) 2022-09-08T08:43:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:01.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:01.243823+0000) 2022-09-08T08:43:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:02.461+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:02.462400+0000) 2022-09-08T08:43:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:02.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:02.244030+0000) 2022-09-08T08:43:02.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:02 smithi055 ceph-mon[97033]: pgmap v21568: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:43:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:02 smithi059 ceph-mon[97450]: pgmap v21568: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:43:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:03.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:03.462632+0000) 2022-09-08T08:43:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:03.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:03.244192+0000) 2022-09-08T08:43:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:04.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:04.462890+0000) 2022-09-08T08:43:04.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:04.243+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:04.244406+0000) 2022-09-08T08:43:04.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:04 smithi055 ceph-mon[97033]: pgmap v21569: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:43:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:04 smithi059 ceph-mon[97450]: pgmap v21569: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:43:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:05.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:05.463110+0000) 2022-09-08T08:43:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:05.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:05.244698+0000) 2022-09-08T08:43:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:06.462+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:06.463348+0000) 2022-09-08T08:43:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:06.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:06.244862+0000) 2022-09-08T08:43:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:06 smithi055 ceph-mon[97033]: pgmap v21570: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T08:43:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:43:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:06 smithi059 ceph-mon[97450]: pgmap v21570: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T08:43:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:43:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:07.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:07.463591+0000) 2022-09-08T08:43:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:07.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:07.245026+0000) 2022-09-08T08:43:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:43:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:43:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:08.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:08.463813+0000) 2022-09-08T08:43:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:08.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:08.245199+0000) 2022-09-08T08:43:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:08 smithi055 ceph-mon[97033]: pgmap v21571: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T08:43:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:08 smithi059 ceph-mon[97450]: pgmap v21571: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T08:43:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:09.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:09.463986+0000) 2022-09-08T08:43:09.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:09.244+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:09.245433+0000) 2022-09-08T08:43:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:10.463+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:10.464118+0000) 2022-09-08T08:43:10.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:10.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:10.245607+0000) 2022-09-08T08:43:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:10 smithi055 ceph-mon[97033]: pgmap v21572: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:43:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:10 smithi055 ceph-mon[97033]: osdmap e1426: 12 total, 11 up, 11 in 2022-09-08T08:43:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:10 smithi059 ceph-mon[97450]: pgmap v21572: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:43:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:10 smithi059 ceph-mon[97450]: osdmap e1426: 12 total, 11 up, 11 in 2022-09-08T08:43:11.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:43:11.033+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:43:11.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:11.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:11.245722+0000) 2022-09-08T08:43:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:11.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:11.464280+0000) 2022-09-08T08:43:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:12.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:12.464522+0000) 2022-09-08T08:43:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:12.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:12.245883+0000) 2022-09-08T08:43:12.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:12 smithi055 ceph-mon[97033]: pgmap v21574: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:43:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:12 smithi059 ceph-mon[97450]: pgmap v21574: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:43:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:13.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:13.464761+0000) 2022-09-08T08:43:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:13.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:13.246040+0000) 2022-09-08T08:43:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:14.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:14.464960+0000) 2022-09-08T08:43:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:14.245+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:14.246272+0000) 2022-09-08T08:43:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:14 smithi055 ceph-mon[97033]: pgmap v21575: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:43:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:14 smithi059 ceph-mon[97450]: pgmap v21575: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:43:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:15.464+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:15.465195+0000) 2022-09-08T08:43:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:15.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:15.246494+0000) 2022-09-08T08:43:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:16.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:16.465409+0000) 2022-09-08T08:43:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:16.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:16.246768+0000) 2022-09-08T08:43:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:16 smithi055 ceph-mon[97033]: pgmap v21576: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:43:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:16 smithi059 ceph-mon[97450]: pgmap v21576: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:43:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:17.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:17.465612+0000) 2022-09-08T08:43:17.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:17.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:17.246954+0000) 2022-09-08T08:43:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:18.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:18.465854+0000) 2022-09-08T08:43:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:18.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:18.247089+0000) 2022-09-08T08:43:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:18 smithi055 ceph-mon[97033]: pgmap v21577: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:43:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:18 smithi059 ceph-mon[97450]: pgmap v21577: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:43:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:19.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:19.466081+0000) 2022-09-08T08:43:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:19.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:19.247315+0000) 2022-09-08T08:43:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:19 smithi055 ceph-mon[97033]: pgmap v21578: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:43:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:19 smithi055 ceph-mon[97033]: osdmap e1427: 12 total, 11 up, 11 in 2022-09-08T08:43:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:19 smithi059 ceph-mon[97450]: pgmap v21578: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:43:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:19 smithi059 ceph-mon[97450]: osdmap e1427: 12 total, 11 up, 11 in 2022-09-08T08:43:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:20.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:20.466306+0000) 2022-09-08T08:43:20.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:20.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:20.247468+0000) 2022-09-08T08:43:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:21.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:21.466492+0000) 2022-09-08T08:43:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:21.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:21.247713+0000) 2022-09-08T08:43:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:21 smithi059 ceph-mon[97450]: pgmap v21580: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:43:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:21 smithi055 ceph-mon[97033]: pgmap v21580: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:43:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:22.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:22.247871+0000) 2022-09-08T08:43:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:22.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:22.466731+0000) 2022-09-08T08:43:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:23.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:23.466963+0000) 2022-09-08T08:43:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:23.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:23.248027+0000) 2022-09-08T08:43:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:23 smithi059 ceph-mon[97450]: pgmap v21581: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:43:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:23 smithi055 ceph-mon[97033]: pgmap v21581: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:43:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:24.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:24.467209+0000) 2022-09-08T08:43:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:24.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:24.248250+0000) 2022-09-08T08:43:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:25.465+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:25.467398+0000) 2022-09-08T08:43:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:25.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:25.248429+0000) 2022-09-08T08:43:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:25 smithi059 ceph-mon[97450]: pgmap v21582: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:43:26.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:43:26.044+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:43:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:26.246+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:26.248563+0000) 2022-09-08T08:43:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:25 smithi055 ceph-mon[97033]: pgmap v21582: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:43:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:26.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:26.467622+0000) 2022-09-08T08:43:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:27.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:27.467914+0000) 2022-09-08T08:43:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:27.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:27.248738+0000) 2022-09-08T08:43:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:27 smithi059 ceph-mon[97450]: pgmap v21583: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:43:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:27 smithi055 ceph-mon[97033]: pgmap v21583: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:43:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:28.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:28.468100+0000) 2022-09-08T08:43:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:28.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:28.248902+0000) 2022-09-08T08:43:29.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:43:29.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:43:29.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:28 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:43:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:43:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:43:29.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:28 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:43:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:29.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:29.249123+0000) 2022-09-08T08:43:29.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:29.466+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:29.468272+0000) 2022-09-08T08:43:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:30.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:30.468499+0000) 2022-09-08T08:43:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:30.247+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:30.249371+0000) 2022-09-08T08:43:30.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:30 smithi055 ceph-mon[97033]: pgmap v21584: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:43:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:43:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:30 smithi059 ceph-mon[97450]: pgmap v21584: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:43:30.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:43:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:31.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:31.468746+0000) 2022-09-08T08:43:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:31.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:31.249518+0000) 2022-09-08T08:43:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:32.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:32.468988+0000) 2022-09-08T08:43:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:32.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:32.249695+0000) 2022-09-08T08:43:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:32 smithi055 ceph-mon[97033]: pgmap v21585: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:43:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:32 smithi059 ceph-mon[97450]: pgmap v21585: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:43:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:33.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:33.469168+0000) 2022-09-08T08:43:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:33.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:33.249903+0000) 2022-09-08T08:43:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:34.467+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:34.469344+0000) 2022-09-08T08:43:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:34.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:34.250122+0000) 2022-09-08T08:43:34.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:34 smithi055 ceph-mon[97033]: pgmap v21586: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:43:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:34 smithi059 ceph-mon[97450]: pgmap v21586: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:43:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:35.468+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:35.469550+0000) 2022-09-08T08:43:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:35.248+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:35.250322+0000) 2022-09-08T08:43:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:36.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:36.469788+0000) 2022-09-08T08:43:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:36.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:36.250550+0000) 2022-09-08T08:43:36.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:36 smithi055 ceph-mon[97033]: pgmap v21587: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T08:43:36.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:36 smithi059 ceph-mon[97450]: pgmap v21587: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-09-08T08:43:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:37.249+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:37.250774+0000) 2022-09-08T08:43:37.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:37.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:37.470040+0000) 2022-09-08T08:43:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:38.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:38.250947+0000) 2022-09-08T08:43:38.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:38.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:38.470174+0000) 2022-09-08T08:43:38.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:38 smithi055 ceph-mon[97033]: pgmap v21588: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:43:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:38 smithi059 ceph-mon[97450]: pgmap v21588: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:43:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:39.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:39.470290+0000) 2022-09-08T08:43:39.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:39.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:39.251148+0000) 2022-09-08T08:43:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:40.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:40.251356+0000) 2022-09-08T08:43:40.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:40.469+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:40.470424+0000) 2022-09-08T08:43:40.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:40 smithi055 ceph-mon[97033]: pgmap v21589: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:43:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:40 smithi059 ceph-mon[97450]: pgmap v21589: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:43:41.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:43:41.050+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:43:41.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:41.250+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:41.251561+0000) 2022-09-08T08:43:41.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:41.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:41.470691+0000) 2022-09-08T08:43:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:42.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:42.470901+0000) 2022-09-08T08:43:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:42.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:42.251717+0000) 2022-09-08T08:43:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:42 smithi055 ceph-mon[97033]: pgmap v21590: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:43:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:42 smithi059 ceph-mon[97450]: pgmap v21590: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:43:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:43.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:43.251933+0000) 2022-09-08T08:43:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:43.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:43.471099+0000) 2022-09-08T08:43:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:44.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:44.471302+0000) 2022-09-08T08:43:44.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:44.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:44.252170+0000) 2022-09-08T08:43:44.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:44 smithi055 ceph-mon[97033]: pgmap v21591: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:43:44.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:44 smithi059 ceph-mon[97450]: pgmap v21591: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:43:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:45.470+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:45.471579+0000) 2022-09-08T08:43:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:45.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:45.252434+0000) 2022-09-08T08:43:46.474 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:46.251+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:46.252619+0000) 2022-09-08T08:43:46.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:46 smithi055 ceph-mon[97033]: pgmap v21592: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:43:46.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:46 smithi059 ceph-mon[97450]: pgmap v21592: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:43:46.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:46.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:46.471763+0000) 2022-09-08T08:43:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:47.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:47.471997+0000) 2022-09-08T08:43:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:47.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:47.252824+0000) 2022-09-08T08:43:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:48.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:48.472131+0000) 2022-09-08T08:43:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:48.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:48.253010+0000) 2022-09-08T08:43:48.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:48 smithi055 ceph-mon[97033]: pgmap v21593: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:43:48.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:48 smithi059 ceph-mon[97450]: pgmap v21593: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:43:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:49.471+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:49.472316+0000) 2022-09-08T08:43:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:49.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:49.253202+0000) 2022-09-08T08:43:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:50.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:50.472519+0000) 2022-09-08T08:43:50.689 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:50.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:50.253377+0000) 2022-09-08T08:43:50.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:50 smithi055 ceph-mon[97033]: pgmap v21594: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:43:50.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:50 smithi059 ceph-mon[97450]: pgmap v21594: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:43:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:51.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:51.472720+0000) 2022-09-08T08:43:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:51.252+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:51.253529+0000) 2022-09-08T08:43:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:52.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:52.472885+0000) 2022-09-08T08:43:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:52.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:52.253710+0000) 2022-09-08T08:43:52.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:52 smithi055 ceph-mon[97033]: pgmap v21595: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:43:52.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:52 smithi059 ceph-mon[97450]: pgmap v21595: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:43:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:53.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:53.473039+0000) 2022-09-08T08:43:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:53.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:53.253890+0000) 2022-09-08T08:43:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:54.472+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:54.473264+0000) 2022-09-08T08:43:54.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:54.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:54.254085+0000) 2022-09-08T08:43:54.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:54 smithi055 ceph-mon[97033]: pgmap v21596: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:43:54.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:54 smithi059 ceph-mon[97450]: pgmap v21596: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:43:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:55.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:55.473498+0000) 2022-09-08T08:43:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:55.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:55.254297+0000) 2022-09-08T08:43:56.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:43:56.063+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:43:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:56.253+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:56.254520+0000) 2022-09-08T08:43:56.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:56 smithi055 ceph-mon[97033]: pgmap v21597: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:43:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:56 smithi059 ceph-mon[97450]: pgmap v21597: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:43:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:56.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:56.473691+0000) 2022-09-08T08:43:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:57.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:57.473911+0000) 2022-09-08T08:43:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:57.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:57.254743+0000) 2022-09-08T08:43:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:58.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:58.474121+0000) 2022-09-08T08:43:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:58.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:58.254992+0000) 2022-09-08T08:43:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:43:58 smithi055 ceph-mon[97033]: pgmap v21598: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:43:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:43:58 smithi059 ceph-mon[97450]: pgmap v21598: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:43:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:43:59.473+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:59.474266+0000) 2022-09-08T08:43:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:43:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:43:59.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:43:59.255148+0000) 2022-09-08T08:44:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:00.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:00.474439+0000) 2022-09-08T08:44:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:00.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:00.255273+0000) 2022-09-08T08:44:00.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:00 smithi055 ceph-mon[97033]: pgmap v21599: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:44:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:00 smithi059 ceph-mon[97450]: pgmap v21599: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:44:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:01.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:01.474657+0000) 2022-09-08T08:44:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:01.254+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:01.255423+0000) 2022-09-08T08:44:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:02.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:02.474867+0000) 2022-09-08T08:44:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:02.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:02.255711+0000) 2022-09-08T08:44:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:02 smithi055 ceph-mon[97033]: pgmap v21600: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:44:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:02 smithi059 ceph-mon[97450]: pgmap v21600: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:44:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:03.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:03.475111+0000) 2022-09-08T08:44:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:03.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:03.255865+0000) 2022-09-08T08:44:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:04.474+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:04.475377+0000) 2022-09-08T08:44:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:04.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:04.256015+0000) 2022-09-08T08:44:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:04 smithi055 ceph-mon[97033]: pgmap v21601: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:44:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:04 smithi059 ceph-mon[97450]: pgmap v21601: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:44:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:05.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:05.475698+0000) 2022-09-08T08:44:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:05.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:05.256170+0000) 2022-09-08T08:44:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:06.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:06.475966+0000) 2022-09-08T08:44:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:06.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:06.256369+0000) 2022-09-08T08:44:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:06 smithi055 ceph-mon[97033]: pgmap v21602: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T08:44:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:44:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:06 smithi059 ceph-mon[97450]: pgmap v21602: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T08:44:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:44:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:07.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:07.476177+0000) 2022-09-08T08:44:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:07.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:07.256561+0000) 2022-09-08T08:44:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:44:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:44:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:08.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:08.476398+0000) 2022-09-08T08:44:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:08.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:08.256791+0000) 2022-09-08T08:44:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:08 smithi055 ceph-mon[97033]: pgmap v21603: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T08:44:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:08 smithi059 ceph-mon[97450]: pgmap v21603: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T08:44:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:09.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:09.476658+0000) 2022-09-08T08:44:09.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:09.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:09.257011+0000) 2022-09-08T08:44:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:10.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:10.476806+0000) 2022-09-08T08:44:10.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:10.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:10.257173+0000) 2022-09-08T08:44:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:10 smithi055 ceph-mon[97033]: pgmap v21604: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:44:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:10 smithi055 ceph-mon[97033]: osdmap e1428: 12 total, 11 up, 11 in 2022-09-08T08:44:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:10 smithi059 ceph-mon[97450]: pgmap v21604: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:44:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:10 smithi059 ceph-mon[97450]: osdmap e1428: 12 total, 11 up, 11 in 2022-09-08T08:44:11.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:44:11.072+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:44:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:11.255+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:11.257340+0000) 2022-09-08T08:44:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:11.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:11.476935+0000) 2022-09-08T08:44:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:12.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:12.477096+0000) 2022-09-08T08:44:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:12.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:12.257515+0000) 2022-09-08T08:44:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:12 smithi055 ceph-mon[97033]: pgmap v21606: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-08T08:44:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:12 smithi059 ceph-mon[97450]: pgmap v21606: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-08T08:44:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:13.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:13.477270+0000) 2022-09-08T08:44:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:13.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:13.257663+0000) 2022-09-08T08:44:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:14.475+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:14.477471+0000) 2022-09-08T08:44:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:14.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:14.257832+0000) 2022-09-08T08:44:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:14 smithi055 ceph-mon[97033]: pgmap v21607: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-08T08:44:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:14 smithi059 ceph-mon[97450]: pgmap v21607: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 56 KiB/s wr, 24 op/s 2022-09-08T08:44:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:15.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:15.257985+0000) 2022-09-08T08:44:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:15.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:15.477690+0000) 2022-09-08T08:44:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:16.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:16.477959+0000) 2022-09-08T08:44:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:16.256+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:16.258155+0000) 2022-09-08T08:44:16.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:16 smithi055 ceph-mon[97033]: pgmap v21608: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2022-09-08T08:44:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:16 smithi059 ceph-mon[97450]: pgmap v21608: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2022-09-08T08:44:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:17.476+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:17.478184+0000) 2022-09-08T08:44:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:17.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:17.258369+0000) 2022-09-08T08:44:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:18.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:18.478474+0000) 2022-09-08T08:44:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:18.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:18.258541+0000) 2022-09-08T08:44:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:18 smithi055 ceph-mon[97033]: pgmap v21609: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2022-09-08T08:44:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:18 smithi059 ceph-mon[97450]: pgmap v21609: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2022-09-08T08:44:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:19.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:19.478703+0000) 2022-09-08T08:44:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:19.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:19.258737+0000) 2022-09-08T08:44:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:19 smithi059 ceph-mon[97450]: pgmap v21610: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2022-09-08T08:44:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:19 smithi059 ceph-mon[97450]: osdmap e1429: 12 total, 11 up, 11 in 2022-09-08T08:44:20.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:19 smithi055 ceph-mon[97033]: pgmap v21610: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2022-09-08T08:44:20.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:19 smithi055 ceph-mon[97033]: osdmap e1429: 12 total, 11 up, 11 in 2022-09-08T08:44:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:20.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:20.478822+0000) 2022-09-08T08:44:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:20.257+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:20.258920+0000) 2022-09-08T08:44:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:21.477+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:21.479023+0000) 2022-09-08T08:44:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:21.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:21.259159+0000) 2022-09-08T08:44:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:21 smithi059 ceph-mon[97450]: pgmap v21612: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:44:22.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:21 smithi055 ceph-mon[97033]: pgmap v21612: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:44:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:22.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:22.259348+0000) 2022-09-08T08:44:22.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:22.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:22.479265+0000) 2022-09-08T08:44:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:23.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:23.479433+0000) 2022-09-08T08:44:23.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:23.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:23.259479+0000) 2022-09-08T08:44:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:23 smithi059 ceph-mon[97450]: pgmap v21613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:44:24.486 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:23 smithi055 ceph-mon[97033]: pgmap v21613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:44:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:24.478+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:24.479680+0000) 2022-09-08T08:44:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:24.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:24.259668+0000) 2022-09-08T08:44:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:25.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:25.479818+0000) 2022-09-08T08:44:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:25.258+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:25.259808+0000) 2022-09-08T08:44:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:25 smithi059 ceph-mon[97450]: pgmap v21614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:44:26.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:44:26.089+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:44:26.405 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:26.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:26.259991+0000) 2022-09-08T08:44:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:25 smithi055 ceph-mon[97033]: pgmap v21614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:44:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:26.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:26.480050+0000) 2022-09-08T08:44:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:27.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:27.480301+0000) 2022-09-08T08:44:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:27.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:27.260125+0000) 2022-09-08T08:44:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:28 smithi059 ceph-mon[97450]: pgmap v21615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:44:28.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:28 smithi055 ceph-mon[97033]: pgmap v21615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:44:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:28.479+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:28.480526+0000) 2022-09-08T08:44:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:28.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:28.260257+0000) 2022-09-08T08:44:29.532 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:29.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:29.480676+0000) 2022-09-08T08:44:29.533 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:29.259+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:29.260432+0000) 2022-09-08T08:44:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:30 smithi059 ceph-mon[97450]: pgmap v21616: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:44:30.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:44:30.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:44:30.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:44:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:30 smithi055 ceph-mon[97033]: pgmap v21616: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:44:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:44:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:44:30.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:44:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:30.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:30.480834+0000) 2022-09-08T08:44:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:30.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:30.260696+0000) 2022-09-08T08:44:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:31.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:31.480993+0000) 2022-09-08T08:44:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:31.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:31.260846+0000) 2022-09-08T08:44:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:32 smithi059 ceph-mon[97450]: pgmap v21617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:44:32.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:32 smithi055 ceph-mon[97033]: pgmap v21617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T08:44:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:32.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:32.481140+0000) 2022-09-08T08:44:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:32.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:32.261020+0000) 2022-09-08T08:44:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:33.480+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:33.481356+0000) 2022-09-08T08:44:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:33.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:33.261149+0000) 2022-09-08T08:44:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:34 smithi059 ceph-mon[97450]: pgmap v21618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:44:34.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:34 smithi055 ceph-mon[97033]: pgmap v21618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:44:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:34.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:34.481556+0000) 2022-09-08T08:44:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:34.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:34.261309+0000) 2022-09-08T08:44:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:35.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:35.481782+0000) 2022-09-08T08:44:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:35.260+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:35.261454+0000) 2022-09-08T08:44:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:36 smithi059 ceph-mon[97450]: pgmap v21619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:44:36.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:36 smithi055 ceph-mon[97033]: pgmap v21619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:44:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:36.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:36.482036+0000) 2022-09-08T08:44:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:36.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:36.261628+0000) 2022-09-08T08:44:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:37.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:37.482276+0000) 2022-09-08T08:44:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:37.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:37.261877+0000) 2022-09-08T08:44:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:38 smithi059 ceph-mon[97450]: pgmap v21620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:44:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:38 smithi055 ceph-mon[97033]: pgmap v21620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:44:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:38.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:38.482454+0000) 2022-09-08T08:44:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:38.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:38.262062+0000) 2022-09-08T08:44:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:39.481+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:39.482605+0000) 2022-09-08T08:44:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:39.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:39.262245+0000) 2022-09-08T08:44:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:40 smithi059 ceph-mon[97450]: pgmap v21621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:44:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:40 smithi055 ceph-mon[97033]: pgmap v21621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:44:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:40.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:40.482778+0000) 2022-09-08T08:44:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:40.261+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:40.262408+0000) 2022-09-08T08:44:41.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:44:41.092+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:44:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:41.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:41.262560+0000) 2022-09-08T08:44:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:41.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:41.483010+0000) 2022-09-08T08:44:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:42 smithi059 ceph-mon[97450]: pgmap v21622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:44:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:42 smithi055 ceph-mon[97033]: pgmap v21622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:44:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:42.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:42.483257+0000) 2022-09-08T08:44:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:42.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:42.262754+0000) 2022-09-08T08:44:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:43.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:43.483414+0000) 2022-09-08T08:44:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:43.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:43.262995+0000) 2022-09-08T08:44:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:44 smithi055 ceph-mon[97033]: pgmap v21623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:44:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:44 smithi059 ceph-mon[97450]: pgmap v21623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:44:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:44.482+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:44.483612+0000) 2022-09-08T08:44:44.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:44.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:44.263161+0000) 2022-09-08T08:44:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:45.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:45.483794+0000) 2022-09-08T08:44:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:45.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:45.263392+0000) 2022-09-08T08:44:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:46 smithi055 ceph-mon[97033]: pgmap v21624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:44:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:46 smithi059 ceph-mon[97450]: pgmap v21624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:44:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:46.262+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:46.263563+0000) 2022-09-08T08:44:46.663 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:46.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:46.484071+0000) 2022-09-08T08:44:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:47.483+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:47.484268+0000) 2022-09-08T08:44:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:47.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:47.263766+0000) 2022-09-08T08:44:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:48 smithi055 ceph-mon[97033]: pgmap v21625: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:44:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:48 smithi059 ceph-mon[97450]: pgmap v21625: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:44:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:48.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:48.484510+0000) 2022-09-08T08:44:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:48.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:48.263906+0000) 2022-09-08T08:44:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:49.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:49.484755+0000) 2022-09-08T08:44:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:49.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:49.264124+0000) 2022-09-08T08:44:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:50 smithi055 ceph-mon[97033]: pgmap v21626: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:44:50.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:50 smithi059 ceph-mon[97450]: pgmap v21626: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:44:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:50.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:50.485017+0000) 2022-09-08T08:44:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:50.263+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:50.264335+0000) 2022-09-08T08:44:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:51.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:51.485247+0000) 2022-09-08T08:44:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:51.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:51.264559+0000) 2022-09-08T08:44:52.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:52 smithi055 ceph-mon[97033]: pgmap v21627: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:44:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:52 smithi059 ceph-mon[97450]: pgmap v21627: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:44:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:52.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:52.485467+0000) 2022-09-08T08:44:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:52.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:52.264821+0000) 2022-09-08T08:44:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:53.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:53.485656+0000) 2022-09-08T08:44:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:53.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:53.264956+0000) 2022-09-08T08:44:54.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:54 smithi055 ceph-mon[97033]: pgmap v21628: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:44:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:54 smithi059 ceph-mon[97450]: pgmap v21628: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:44:54.545 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:54.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:54.485861+0000) 2022-09-08T08:44:54.546 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:54.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:54.265139+0000) 2022-09-08T08:44:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:55.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:55.486057+0000) 2022-09-08T08:44:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:55.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:55.265298+0000) 2022-09-08T08:44:56.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:44:56.107+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:44:56.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:56.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:56.265524+0000) 2022-09-08T08:44:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:56 smithi055 ceph-mon[97033]: pgmap v21629: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:44:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:56 smithi059 ceph-mon[97450]: pgmap v21629: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:44:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:56.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:56.486255+0000) 2022-09-08T08:44:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:57.484+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:57.486422+0000) 2022-09-08T08:44:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:57.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:57.265700+0000) 2022-09-08T08:44:58.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:44:58 smithi055 ceph-mon[97033]: pgmap v21630: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:44:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:44:58 smithi059 ceph-mon[97450]: pgmap v21630: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:44:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:58.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:58.486633+0000) 2022-09-08T08:44:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:58.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:58.265872+0000) 2022-09-08T08:44:59.600 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:44:59.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:59.486856+0000) 2022-09-08T08:44:59.601 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:44:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:44:59.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:44:59.266080+0000) 2022-09-08T08:45:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:00.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:00.487011+0000) 2022-09-08T08:45:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:00.264+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:00.266254+0000) 2022-09-08T08:45:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:00 smithi055 ceph-mon[97033]: pgmap v21631: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:00 smithi055 ceph-mon[97033]: pgmap v21632: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:45:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:45:00.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:00 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T08:45:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:00 smithi059 ceph-mon[97450]: pgmap v21631: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:00 smithi059 ceph-mon[97450]: pgmap v21632: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:45:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:45:01.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:00 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T08:45:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:01.485+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:01.487215+0000) 2022-09-08T08:45:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:01.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:01.266401+0000) 2022-09-08T08:45:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:02.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:02.487473+0000) 2022-09-08T08:45:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:02.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:02.266575+0000) 2022-09-08T08:45:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:02 smithi055 ceph-mon[97033]: pgmap v21633: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:45:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:02 smithi059 ceph-mon[97450]: pgmap v21633: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:45:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:03.486+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:03.487752+0000) 2022-09-08T08:45:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:03.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:03.266783+0000) 2022-09-08T08:45:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:04.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:04.487987+0000) 2022-09-08T08:45:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:04.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:04.266957+0000) 2022-09-08T08:45:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:04 smithi055 ceph-mon[97033]: pgmap v21634: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:45:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:04 smithi059 ceph-mon[97450]: pgmap v21634: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:45:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:05.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:05.488199+0000) 2022-09-08T08:45:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:05.265+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:05.267164+0000) 2022-09-08T08:45:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:06.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:06.488443+0000) 2022-09-08T08:45:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:06.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:06.267413+0000) 2022-09-08T08:45:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:06 smithi055 ceph-mon[97033]: pgmap v21635: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 49 KiB/s wr, 22 op/s 2022-09-08T08:45:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:45:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:06 smithi059 ceph-mon[97450]: pgmap v21635: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 49 KiB/s wr, 22 op/s 2022-09-08T08:45:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:45:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:07.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:07.488586+0000) 2022-09-08T08:45:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:07.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:07.267680+0000) 2022-09-08T08:45:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:45:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:45:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:08.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:08.488827+0000) 2022-09-08T08:45:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:08.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:08.267884+0000) 2022-09-08T08:45:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:08 smithi055 ceph-mon[97033]: pgmap v21636: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 49 KiB/s wr, 22 op/s 2022-09-08T08:45:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:08 smithi059 ceph-mon[97450]: pgmap v21636: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 49 KiB/s wr, 22 op/s 2022-09-08T08:45:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:09.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:09.488977+0000) 2022-09-08T08:45:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:09.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:09.268093+0000) 2022-09-08T08:45:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:10 smithi059 ceph-mon[97450]: pgmap v21637: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 49 KiB/s wr, 22 op/s 2022-09-08T08:45:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:10 smithi055 ceph-mon[97033]: pgmap v21637: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 49 KiB/s wr, 22 op/s 2022-09-08T08:45:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:10.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:10.489114+0000) 2022-09-08T08:45:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:10.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:10.268251+0000) 2022-09-08T08:45:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:11 smithi059 ceph-mon[97450]: osdmap e1430: 12 total, 11 up, 11 in 2022-09-08T08:45:11.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:45:11.122+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:45:11.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:11.266+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:11.268416+0000) 2022-09-08T08:45:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:11 smithi055 ceph-mon[97033]: osdmap e1430: 12 total, 11 up, 11 in 2022-09-08T08:45:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:11.487+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:11.489268+0000) 2022-09-08T08:45:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:12 smithi059 ceph-mon[97450]: pgmap v21639: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:45:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:12 smithi055 ceph-mon[97033]: pgmap v21639: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:45:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:12.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:12.489443+0000) 2022-09-08T08:45:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:12.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:12.268625+0000) 2022-09-08T08:45:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:13.488+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:13.489698+0000) 2022-09-08T08:45:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:13.267+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:13.268854+0000) 2022-09-08T08:45:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:14.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:14.489895+0000) 2022-09-08T08:45:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:14.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:14.269024+0000) 2022-09-08T08:45:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:14 smithi055 ceph-mon[97033]: pgmap v21640: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:45:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:14 smithi059 ceph-mon[97450]: pgmap v21640: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:45:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:15.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:15.490142+0000) 2022-09-08T08:45:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:15.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:15.269230+0000) 2022-09-08T08:45:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:16.489+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:16.490362+0000) 2022-09-08T08:45:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:16.268+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:16.269457+0000) 2022-09-08T08:45:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:16 smithi055 ceph-mon[97033]: pgmap v21641: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:45:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:16 smithi059 ceph-mon[97450]: pgmap v21641: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:45:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:17.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:17.490670+0000) 2022-09-08T08:45:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:17.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:17.269716+0000) 2022-09-08T08:45:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:18.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:18.490923+0000) 2022-09-08T08:45:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:18.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:18.269888+0000) 2022-09-08T08:45:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:18 smithi055 ceph-mon[97033]: pgmap v21642: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:45:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:18 smithi059 ceph-mon[97450]: pgmap v21642: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:45:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:19.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:19.491164+0000) 2022-09-08T08:45:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:19.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:19.270059+0000) 2022-09-08T08:45:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:20 smithi059 ceph-mon[97450]: osdmap e1431: 12 total, 11 up, 11 in 2022-09-08T08:45:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:20 smithi059 ceph-mon[97450]: pgmap v21644: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.7 KiB/s rd, 106 B/s wr, 3 op/s 2022-09-08T08:45:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:20 smithi055 ceph-mon[97033]: osdmap e1431: 12 total, 11 up, 11 in 2022-09-08T08:45:20.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:20 smithi055 ceph-mon[97033]: pgmap v21644: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.7 KiB/s rd, 106 B/s wr, 3 op/s 2022-09-08T08:45:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:20.490+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:20.491402+0000) 2022-09-08T08:45:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:20.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:20.270272+0000) 2022-09-08T08:45:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:21.269+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:21.270454+0000) 2022-09-08T08:45:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:21.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:21.491607+0000) 2022-09-08T08:45:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:22.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:22.270665+0000) 2022-09-08T08:45:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:22.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:22.491833+0000) 2022-09-08T08:45:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:22 smithi055 ceph-mon[97033]: pgmap v21645: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:45:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:22 smithi059 ceph-mon[97450]: pgmap v21645: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:45:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:23.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:23.270877+0000) 2022-09-08T08:45:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:23.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:23.492052+0000) 2022-09-08T08:45:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:24.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:24.492256+0000) 2022-09-08T08:45:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:24.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:24.271020+0000) 2022-09-08T08:45:24.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:24 smithi055 ceph-mon[97033]: pgmap v21646: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:45:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:24 smithi059 ceph-mon[97450]: pgmap v21646: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:45:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:25.491+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:25.492413+0000) 2022-09-08T08:45:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:25.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:25.271152+0000) 2022-09-08T08:45:26.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:45:26.138+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:45:26.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:26.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:26.271288+0000) 2022-09-08T08:45:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:26.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:26.492688+0000) 2022-09-08T08:45:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:26 smithi055 ceph-mon[97033]: pgmap v21647: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:45:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:26 smithi059 ceph-mon[97450]: pgmap v21647: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:45:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:27.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:27.492890+0000) 2022-09-08T08:45:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:27.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:27.271422+0000) 2022-09-08T08:45:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:28.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:28.493033+0000) 2022-09-08T08:45:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:28.270+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:28.271587+0000) 2022-09-08T08:45:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:28 smithi055 ceph-mon[97033]: pgmap v21648: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:45:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:28 smithi059 ceph-mon[97450]: pgmap v21648: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:45:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:29.492+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:29.493222+0000) 2022-09-08T08:45:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:29.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:29.271804+0000) 2022-09-08T08:45:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:30 smithi059 ceph-mon[97450]: pgmap v21649: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:45:30.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:30 smithi055 ceph-mon[97033]: pgmap v21649: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:45:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:30.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:30.493449+0000) 2022-09-08T08:45:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:30.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:30.272044+0000) 2022-09-08T08:45:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:31.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:31.493648+0000) 2022-09-08T08:45:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:31.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:31.272221+0000) 2022-09-08T08:45:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:32.271+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:32.272401+0000) 2022-09-08T08:45:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:32.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:32.493900+0000) 2022-09-08T08:45:32.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:32 smithi055 ceph-mon[97033]: pgmap v21650: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:45:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:32 smithi059 ceph-mon[97450]: pgmap v21650: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:45:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:33.493+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:33.494076+0000) 2022-09-08T08:45:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:33.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:33.272617+0000) 2022-09-08T08:45:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:34.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:34.494327+0000) 2022-09-08T08:45:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:34.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:34.272836+0000) 2022-09-08T08:45:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:34 smithi055 ceph-mon[97033]: pgmap v21651: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:34 smithi059 ceph-mon[97450]: pgmap v21651: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:35.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:35.494567+0000) 2022-09-08T08:45:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:35.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:35.272982+0000) 2022-09-08T08:45:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:36.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:36.494747+0000) 2022-09-08T08:45:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:36.272+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:36.273224+0000) 2022-09-08T08:45:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:36 smithi055 ceph-mon[97033]: pgmap v21652: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:45:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:36 smithi059 ceph-mon[97450]: pgmap v21652: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:45:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:37.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:37.494980+0000) 2022-09-08T08:45:37.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:37.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:37.273494+0000) 2022-09-08T08:45:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:38.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:38.495119+0000) 2022-09-08T08:45:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:38.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:38.273758+0000) 2022-09-08T08:45:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:38 smithi055 ceph-mon[97033]: pgmap v21653: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:38 smithi059 ceph-mon[97450]: pgmap v21653: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:39.494+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:39.495330+0000) 2022-09-08T08:45:39.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:39.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:39.273896+0000) 2022-09-08T08:45:40.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:40 smithi059 ceph-mon[97450]: pgmap v21654: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:40 smithi055 ceph-mon[97033]: pgmap v21654: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:40.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:40.495549+0000) 2022-09-08T08:45:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:40.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:40.274042+0000) 2022-09-08T08:45:41.404 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:41.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:41.274264+0000) 2022-09-08T08:45:41.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:45:41.140+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:45:41.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:41.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:41.495711+0000) 2022-09-08T08:45:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:42.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:42.495935+0000) 2022-09-08T08:45:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:42.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:42.274495+0000) 2022-09-08T08:45:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:42 smithi055 ceph-mon[97033]: pgmap v21655: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:45:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:42 smithi059 ceph-mon[97450]: pgmap v21655: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:45:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:43.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:43.496164+0000) 2022-09-08T08:45:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:43.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:43.274670+0000) 2022-09-08T08:45:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:44.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:44.496397+0000) 2022-09-08T08:45:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:44.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:44.274808+0000) 2022-09-08T08:45:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:44 smithi055 ceph-mon[97033]: pgmap v21656: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:44 smithi059 ceph-mon[97450]: pgmap v21656: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:45.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:45.496679+0000) 2022-09-08T08:45:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:45.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:45.274971+0000) 2022-09-08T08:45:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:46.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:46.496912+0000) 2022-09-08T08:45:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:46.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:46.275114+0000) 2022-09-08T08:45:46.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:46 smithi055 ceph-mon[97033]: pgmap v21657: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:45:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:46 smithi059 ceph-mon[97450]: pgmap v21657: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:45:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:47.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:47.275258+0000) 2022-09-08T08:45:47.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:47.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:47.497112+0000) 2022-09-08T08:45:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:48.495+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:48.497334+0000) 2022-09-08T08:45:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:48.273+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:48.275410+0000) 2022-09-08T08:45:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:48 smithi055 ceph-mon[97033]: pgmap v21658: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:48 smithi059 ceph-mon[97450]: pgmap v21658: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:49.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:49.497587+0000) 2022-09-08T08:45:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:49.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:49.275658+0000) 2022-09-08T08:45:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:50 smithi059 ceph-mon[97450]: pgmap v21659: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:50 smithi055 ceph-mon[97033]: pgmap v21659: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:50.496+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:50.497804+0000) 2022-09-08T08:45:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:50.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:50.275891+0000) 2022-09-08T08:45:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:51.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:51.498066+0000) 2022-09-08T08:45:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:51.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:51.276033+0000) 2022-09-08T08:45:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:52.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:52.498339+0000) 2022-09-08T08:45:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:52.274+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:52.276183+0000) 2022-09-08T08:45:52.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:52 smithi055 ceph-mon[97033]: pgmap v21660: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:45:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:52 smithi059 ceph-mon[97450]: pgmap v21660: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:45:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:53.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:53.498607+0000) 2022-09-08T08:45:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:53.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:53.276334+0000) 2022-09-08T08:45:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:54.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:54.498869+0000) 2022-09-08T08:45:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:54.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:54.276477+0000) 2022-09-08T08:45:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:54 smithi055 ceph-mon[97033]: pgmap v21661: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:54 smithi059 ceph-mon[97450]: pgmap v21661: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:55.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:55.498992+0000) 2022-09-08T08:45:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:55.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:55.276615+0000) 2022-09-08T08:45:56.501 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:45:56.155+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:45:56.502 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:56.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:56.276723+0000) 2022-09-08T08:45:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:56.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:56.499179+0000) 2022-09-08T08:45:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:56 smithi055 ceph-mon[97033]: pgmap v21662: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:45:57.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:56 smithi059 ceph-mon[97450]: pgmap v21662: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:45:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:57.497+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:57.499374+0000) 2022-09-08T08:45:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:57.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:57.276877+0000) 2022-09-08T08:45:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:58.275+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:58.277032+0000) 2022-09-08T08:45:58.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:58.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:58.499541+0000) 2022-09-08T08:45:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:45:58 smithi055 ceph-mon[97033]: pgmap v21663: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:45:58 smithi059 ceph-mon[97450]: pgmap v21663: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:45:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:45:59.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:59.277187+0000) 2022-09-08T08:45:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:45:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:45:59.498+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:45:59.499760+0000) 2022-09-08T08:46:00.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:00 smithi059 ceph-mon[97450]: pgmap v21664: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:46:00.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:46:00.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:46:00.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:00 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:46:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:00 smithi055 ceph-mon[97033]: pgmap v21664: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:46:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:46:00.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:46:00.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:00 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:46:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:00.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:00.499974+0000) 2022-09-08T08:46:00.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:00.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:00.277335+0000) 2022-09-08T08:46:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:01.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:01.500145+0000) 2022-09-08T08:46:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:01.276+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:01.277489+0000) 2022-09-08T08:46:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:02.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:02.277664+0000) 2022-09-08T08:46:02.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:02.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:02.500375+0000) 2022-09-08T08:46:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:02 smithi055 ceph-mon[97033]: pgmap v21665: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:46:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:02 smithi059 ceph-mon[97450]: pgmap v21665: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:46:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:03.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:03.277822+0000) 2022-09-08T08:46:03.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:03.499+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:03.500615+0000) 2022-09-08T08:46:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:04.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:04.500860+0000) 2022-09-08T08:46:04.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:04.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:04.277982+0000) 2022-09-08T08:46:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:04 smithi055 ceph-mon[97033]: pgmap v21666: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:46:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:04 smithi059 ceph-mon[97450]: pgmap v21666: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:46:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:05.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:05.501082+0000) 2022-09-08T08:46:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:05.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:05.278161+0000) 2022-09-08T08:46:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:06.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:06.501345+0000) 2022-09-08T08:46:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:06.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:06.278338+0000) 2022-09-08T08:46:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:06 smithi055 ceph-mon[97033]: pgmap v21667: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:46:06.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:46:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:06 smithi059 ceph-mon[97450]: pgmap v21667: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:46:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:46:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:07.500+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:07.501598+0000) 2022-09-08T08:46:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:07.277+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:07.278544+0000) 2022-09-08T08:46:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:46:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:46:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:08.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:08.278769+0000) 2022-09-08T08:46:08.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:08.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:08.501749+0000) 2022-09-08T08:46:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:08 smithi055 ceph-mon[97033]: pgmap v21668: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:46:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:08 smithi059 ceph-mon[97450]: pgmap v21668: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:46:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:09.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:09.278951+0000) 2022-09-08T08:46:09.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:09.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:09.501923+0000) 2022-09-08T08:46:10.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:10 smithi059 ceph-mon[97450]: pgmap v21669: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:46:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:10 smithi055 ceph-mon[97033]: pgmap v21669: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:46:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:10.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:10.502167+0000) 2022-09-08T08:46:10.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:10.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:10.279200+0000) 2022-09-08T08:46:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:11 smithi059 ceph-mon[97450]: osdmap e1432: 12 total, 11 up, 11 in 2022-09-08T08:46:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:11 smithi055 ceph-mon[97033]: osdmap e1432: 12 total, 11 up, 11 in 2022-09-08T08:46:11.505 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:46:11.171+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:46:11.506 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:11.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:11.279456+0000) 2022-09-08T08:46:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:11.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:11.502412+0000) 2022-09-08T08:46:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:12 smithi055 ceph-mon[97033]: pgmap v21671: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:46:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:12 smithi059 ceph-mon[97450]: pgmap v21671: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:46:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:12.501+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:12.502557+0000) 2022-09-08T08:46:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:12.278+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:12.279598+0000) 2022-09-08T08:46:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:13.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:13.502765+0000) 2022-09-08T08:46:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:13.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:13.279704+0000) 2022-09-08T08:46:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:14.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:14.502935+0000) 2022-09-08T08:46:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:14.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:14.279860+0000) 2022-09-08T08:46:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:14 smithi055 ceph-mon[97033]: pgmap v21672: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:46:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:14 smithi059 ceph-mon[97450]: pgmap v21672: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:46:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:15.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:15.503181+0000) 2022-09-08T08:46:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:15.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:15.279994+0000) 2022-09-08T08:46:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:16.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:16.503359+0000) 2022-09-08T08:46:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:16.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:16.280144+0000) 2022-09-08T08:46:16.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:16 smithi055 ceph-mon[97033]: pgmap v21673: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:46:17.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:16 smithi059 ceph-mon[97450]: pgmap v21673: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:46:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:17.502+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:17.503575+0000) 2022-09-08T08:46:17.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:17.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:17.280359+0000) 2022-09-08T08:46:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:18.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:18.503842+0000) 2022-09-08T08:46:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:18.279+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:18.280526+0000) 2022-09-08T08:46:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:18 smithi055 ceph-mon[97033]: pgmap v21674: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:46:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:18 smithi059 ceph-mon[97450]: pgmap v21674: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:46:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:19.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:19.504086+0000) 2022-09-08T08:46:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:19.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:19.280703+0000) 2022-09-08T08:46:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:20 smithi059 ceph-mon[97450]: osdmap e1433: 12 total, 11 up, 11 in 2022-09-08T08:46:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:20 smithi059 ceph-mon[97450]: pgmap v21676: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.7 KiB/s rd, 106 B/s wr, 3 op/s 2022-09-08T08:46:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:20 smithi055 ceph-mon[97033]: osdmap e1433: 12 total, 11 up, 11 in 2022-09-08T08:46:20.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:20 smithi055 ceph-mon[97033]: pgmap v21676: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.7 KiB/s rd, 106 B/s wr, 3 op/s 2022-09-08T08:46:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:20.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:20.504305+0000) 2022-09-08T08:46:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:20.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:20.280848+0000) 2022-09-08T08:46:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:21.503+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:21.504527+0000) 2022-09-08T08:46:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:21.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:21.280996+0000) 2022-09-08T08:46:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:22.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:22.504748+0000) 2022-09-08T08:46:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:22.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:22.281163+0000) 2022-09-08T08:46:22.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:22 smithi055 ceph-mon[97033]: pgmap v21677: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:46:23.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:22 smithi059 ceph-mon[97450]: pgmap v21677: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:46:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:23.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:23.504879+0000) 2022-09-08T08:46:23.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:23.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:23.281314+0000) 2022-09-08T08:46:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:24.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:24.505087+0000) 2022-09-08T08:46:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:24.280+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:24.281539+0000) 2022-09-08T08:46:24.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:24 smithi055 ceph-mon[97033]: pgmap v21678: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:46:25.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:24 smithi059 ceph-mon[97450]: pgmap v21678: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:46:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:25.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:25.281733+0000) 2022-09-08T08:46:25.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:25.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:25.505302+0000) 2022-09-08T08:46:26.508 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:46:26.187+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:46:26.509 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:26.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:26.281911+0000) 2022-09-08T08:46:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:26.504+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:26.505499+0000) 2022-09-08T08:46:26.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:26 smithi055 ceph-mon[97033]: pgmap v21679: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:46:27.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:26 smithi059 ceph-mon[97450]: pgmap v21679: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:46:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:27.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:27.505733+0000) 2022-09-08T08:46:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:27.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:27.282063+0000) 2022-09-08T08:46:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:28.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:28.505873+0000) 2022-09-08T08:46:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:28.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:28.282288+0000) 2022-09-08T08:46:28.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:28 smithi055 ceph-mon[97033]: pgmap v21680: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:46:29.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:28 smithi059 ceph-mon[97450]: pgmap v21680: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:46:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:29.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:29.506098+0000) 2022-09-08T08:46:29.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:29.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:29.282497+0000) 2022-09-08T08:46:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:30 smithi055 ceph-mon[97033]: pgmap v21681: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:46:30.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:46:30.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:30 smithi059 ceph-mon[97450]: pgmap v21681: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:46:30.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:46:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:30.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:30.506263+0000) 2022-09-08T08:46:30.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:30.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:30.282717+0000) 2022-09-08T08:46:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:31 smithi055 ceph-mon[97033]: pgmap v21682: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.4 KiB/s rd, 8 op/s 2022-09-08T08:46:31.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:31 smithi055 ceph-mon[97033]: pgmap v21683: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-08T08:46:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:31 smithi059 ceph-mon[97450]: pgmap v21682: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.4 KiB/s rd, 8 op/s 2022-09-08T08:46:31.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:31 smithi059 ceph-mon[97450]: pgmap v21683: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-08T08:46:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:31.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:31.282931+0000) 2022-09-08T08:46:31.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:31.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:31.506507+0000) 2022-09-08T08:46:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:32.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:32.506720+0000) 2022-09-08T08:46:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:32.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:32.283124+0000) 2022-09-08T08:46:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:33 smithi055 ceph-mon[97033]: pgmap v21684: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:46:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:33 smithi059 ceph-mon[97450]: pgmap v21684: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:46:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:33.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:33.506954+0000) 2022-09-08T08:46:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:33.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:33.283291+0000) 2022-09-08T08:46:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:34.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:34.507225+0000) 2022-09-08T08:46:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:34.281+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:34.283471+0000) 2022-09-08T08:46:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:35 smithi055 ceph-mon[97033]: pgmap v21685: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:46:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:35 smithi059 ceph-mon[97450]: pgmap v21685: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:46:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:35.505+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:35.507440+0000) 2022-09-08T08:46:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:35.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:35.283678+0000) 2022-09-08T08:46:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:36.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:36.507609+0000) 2022-09-08T08:46:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:36.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:36.283817+0000) 2022-09-08T08:46:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:37 smithi055 ceph-mon[97033]: pgmap v21686: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:46:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:37 smithi059 ceph-mon[97450]: pgmap v21686: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:46:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:37.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:37.284049+0000) 2022-09-08T08:46:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:37.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:37.507793+0000) 2022-09-08T08:46:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:38.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:38.508048+0000) 2022-09-08T08:46:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:38.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:38.284213+0000) 2022-09-08T08:46:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:39 smithi055 ceph-mon[97033]: pgmap v21687: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:46:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:39 smithi059 ceph-mon[97450]: pgmap v21687: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:46:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:39.282+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:39.284360+0000) 2022-09-08T08:46:39.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:39.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:39.508191+0000) 2022-09-08T08:46:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:40.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:40.508417+0000) 2022-09-08T08:46:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:40.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:40.284607+0000) 2022-09-08T08:46:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:41 smithi055 ceph-mon[97033]: pgmap v21688: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:46:41.511 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:46:41.202+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:46:41.512 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:41.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:41.284889+0000) 2022-09-08T08:46:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:41 smithi059 ceph-mon[97450]: pgmap v21688: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:46:41.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:41.506+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:41.508613+0000) 2022-09-08T08:46:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:42.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:42.508797+0000) 2022-09-08T08:46:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:42.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:42.285132+0000) 2022-09-08T08:46:43.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:43 smithi055 ceph-mon[97033]: pgmap v21689: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:46:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:43 smithi059 ceph-mon[97450]: pgmap v21689: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:46:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:43.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:43.509047+0000) 2022-09-08T08:46:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:43.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:43.285349+0000) 2022-09-08T08:46:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:44.507+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:44.509175+0000) 2022-09-08T08:46:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:44.283+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:44.285510+0000) 2022-09-08T08:46:45.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:45 smithi055 ceph-mon[97033]: pgmap v21690: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:46:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:45 smithi059 ceph-mon[97450]: pgmap v21690: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:46:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:45.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:45.285712+0000) 2022-09-08T08:46:45.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:45.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:45.509364+0000) 2022-09-08T08:46:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:46.284+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:46.285850+0000) 2022-09-08T08:46:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:46.508+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:46.509582+0000) 2022-09-08T08:46:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:47 smithi055 ceph-mon[97033]: pgmap v21691: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:46:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:47 smithi059 ceph-mon[97450]: pgmap v21691: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:46:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:47.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:47.509801+0000) 2022-09-08T08:46:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:47.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:47.286071+0000) 2022-09-08T08:46:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:48.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:48.510028+0000) 2022-09-08T08:46:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:48.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:48.286260+0000) 2022-09-08T08:46:49.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:49 smithi055 ceph-mon[97033]: pgmap v21692: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:46:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:49 smithi059 ceph-mon[97450]: pgmap v21692: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:46:49.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:49.285+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:49.286464+0000) 2022-09-08T08:46:49.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:49.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:49.510230+0000) 2022-09-08T08:46:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:50.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:50.510493+0000) 2022-09-08T08:46:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:50.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:50.286683+0000) 2022-09-08T08:46:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:51 smithi055 ceph-mon[97033]: pgmap v21693: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:46:51.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:51 smithi059 ceph-mon[97450]: pgmap v21693: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:46:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:51.509+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:51.510623+0000) 2022-09-08T08:46:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:51.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:51.286827+0000) 2022-09-08T08:46:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:52.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:52.510795+0000) 2022-09-08T08:46:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:52.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:52.287041+0000) 2022-09-08T08:46:53.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:53 smithi055 ceph-mon[97033]: pgmap v21694: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:46:53.536 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:53 smithi059 ceph-mon[97450]: pgmap v21694: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:46:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:53.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:53.511017+0000) 2022-09-08T08:46:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:53.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:53.287196+0000) 2022-09-08T08:46:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:54.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:54.511193+0000) 2022-09-08T08:46:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:54.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:54.287410+0000) 2022-09-08T08:46:55.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:55 smithi055 ceph-mon[97033]: pgmap v21695: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:46:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:55 smithi059 ceph-mon[97450]: pgmap v21695: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:46:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:55.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:55.511408+0000) 2022-09-08T08:46:55.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:55.286+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:55.287589+0000) 2022-09-08T08:46:56.514 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:46:56.218+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:46:56.515 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:56.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:56.287782+0000) 2022-09-08T08:46:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:56.510+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:56.511614+0000) 2022-09-08T08:46:57.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:57 smithi055 ceph-mon[97033]: pgmap v21696: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:46:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:57 smithi059 ceph-mon[97450]: pgmap v21696: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:46:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:57.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:57.511851+0000) 2022-09-08T08:46:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:57.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:57.287937+0000) 2022-09-08T08:46:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:58.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:58.288145+0000) 2022-09-08T08:46:58.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:58.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:58.511982+0000) 2022-09-08T08:46:59.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:46:59 smithi055 ceph-mon[97033]: pgmap v21697: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:46:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:46:59 smithi059 ceph-mon[97450]: pgmap v21697: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:46:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:46:59.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:59.288282+0000) 2022-09-08T08:46:59.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:46:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:46:59.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:46:59.512202+0000) 2022-09-08T08:47:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:00.511+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:00.512404+0000) 2022-09-08T08:47:00.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:00.287+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:00.288532+0000) 2022-09-08T08:47:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:01 smithi055 ceph-mon[97033]: pgmap v21698: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:47:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:01 smithi059 ceph-mon[97450]: pgmap v21698: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:47:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:01.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:01.512672+0000) 2022-09-08T08:47:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:01.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:01.288683+0000) 2022-09-08T08:47:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:02.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:02.512932+0000) 2022-09-08T08:47:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:02.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:02.288822+0000) 2022-09-08T08:47:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:03 smithi055 ceph-mon[97033]: pgmap v21699: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:47:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:03 smithi059 ceph-mon[97450]: pgmap v21699: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:47:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:03.512+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:03.513185+0000) 2022-09-08T08:47:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:03.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:03.288960+0000) 2022-09-08T08:47:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:04.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:04.513428+0000) 2022-09-08T08:47:04.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:04.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:04.289129+0000) 2022-09-08T08:47:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:05 smithi055 ceph-mon[97033]: pgmap v21700: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:47:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:05 smithi059 ceph-mon[97450]: pgmap v21700: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:47:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:05.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:05.513713+0000) 2022-09-08T08:47:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:05.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:05.289254+0000) 2022-09-08T08:47:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:06.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:06.513977+0000) 2022-09-08T08:47:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:06.288+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:06.289396+0000) 2022-09-08T08:47:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:07 smithi055 ceph-mon[97033]: pgmap v21701: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:47:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:47:07.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:47:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:07 smithi059 ceph-mon[97450]: pgmap v21701: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:47:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:47:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:47:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:07.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:07.514240+0000) 2022-09-08T08:47:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:07.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:07.289557+0000) 2022-09-08T08:47:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:08.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:08.514460+0000) 2022-09-08T08:47:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:08.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:08.289787+0000) 2022-09-08T08:47:09.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:09 smithi055 ceph-mon[97033]: pgmap v21702: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:47:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:09 smithi059 ceph-mon[97450]: pgmap v21702: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:47:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:09.513+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:09.514592+0000) 2022-09-08T08:47:09.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:09.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:09.289973+0000) 2022-09-08T08:47:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:10.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:10.290104+0000) 2022-09-08T08:47:10.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:10.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:10.514832+0000) 2022-09-08T08:47:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:11 smithi055 ceph-mon[97033]: pgmap v21703: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:47:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:11 smithi055 ceph-mon[97033]: osdmap e1434: 12 total, 11 up, 11 in 2022-09-08T08:47:11.517 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:47:11.234+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:47:11.518 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:11.289+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:11.290296+0000) 2022-09-08T08:47:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:11 smithi059 ceph-mon[97450]: pgmap v21703: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:47:11.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:11 smithi059 ceph-mon[97450]: osdmap e1434: 12 total, 11 up, 11 in 2022-09-08T08:47:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:11.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:11.515059+0000) 2022-09-08T08:47:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:12.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:12.515297+0000) 2022-09-08T08:47:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:12.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:12.290458+0000) 2022-09-08T08:47:13.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:13 smithi055 ceph-mon[97033]: pgmap v21705: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:47:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:13 smithi059 ceph-mon[97450]: pgmap v21705: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:47:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:13.514+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:13.515515+0000) 2022-09-08T08:47:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:13.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:13.290733+0000) 2022-09-08T08:47:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:14.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:14.515774+0000) 2022-09-08T08:47:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:14.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:14.290964+0000) 2022-09-08T08:47:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:15 smithi055 ceph-mon[97033]: pgmap v21706: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:47:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:15 smithi059 ceph-mon[97450]: pgmap v21706: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:47:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:15.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:15.291126+0000) 2022-09-08T08:47:15.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:15.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:15.515964+0000) 2022-09-08T08:47:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:16.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:16.516139+0000) 2022-09-08T08:47:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:16.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:16.291269+0000) 2022-09-08T08:47:17.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:17 smithi055 ceph-mon[97033]: pgmap v21707: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:47:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:17 smithi059 ceph-mon[97450]: pgmap v21707: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:47:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:17.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:17.291409+0000) 2022-09-08T08:47:17.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:17.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:17.516282+0000) 2022-09-08T08:47:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:18.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:18.291590+0000) 2022-09-08T08:47:18.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:18.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:18.516455+0000) 2022-09-08T08:47:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:19 smithi055 ceph-mon[97033]: pgmap v21708: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:47:19.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:19 smithi055 ceph-mon[97033]: osdmap e1435: 12 total, 11 up, 11 in 2022-09-08T08:47:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:19 smithi059 ceph-mon[97450]: pgmap v21708: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:47:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:19 smithi059 ceph-mon[97450]: osdmap e1435: 12 total, 11 up, 11 in 2022-09-08T08:47:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:19.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:19.516705+0000) 2022-09-08T08:47:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:19.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:19.291735+0000) 2022-09-08T08:47:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:20.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:20.516870+0000) 2022-09-08T08:47:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:20.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:20.291892+0000) 2022-09-08T08:47:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:21 smithi055 ceph-mon[97033]: pgmap v21710: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 103 B/s wr, 5 op/s 2022-09-08T08:47:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:21 smithi059 ceph-mon[97450]: pgmap v21710: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 103 B/s wr, 5 op/s 2022-09-08T08:47:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:21.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:21.517081+0000) 2022-09-08T08:47:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:21.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:21.292055+0000) 2022-09-08T08:47:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:22.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:22.517254+0000) 2022-09-08T08:47:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:22.290+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:22.292262+0000) 2022-09-08T08:47:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:23 smithi055 ceph-mon[97033]: pgmap v21711: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:47:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:23 smithi059 ceph-mon[97450]: pgmap v21711: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:47:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:23.515+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:23.517466+0000) 2022-09-08T08:47:23.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:23.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:23.292421+0000) 2022-09-08T08:47:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:24.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:24.517693+0000) 2022-09-08T08:47:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:24.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:24.292675+0000) 2022-09-08T08:47:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:25 smithi055 ceph-mon[97033]: pgmap v21712: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:47:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:25 smithi059 ceph-mon[97450]: pgmap v21712: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:47:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:25.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:25.517847+0000) 2022-09-08T08:47:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:25.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:25.292891+0000) 2022-09-08T08:47:26.520 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:47:26.248+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:47:26.521 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:26.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:26.293082+0000) 2022-09-08T08:47:26.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:26.516+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:26.517991+0000) 2022-09-08T08:47:27.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:27 smithi055 ceph-mon[97033]: pgmap v21713: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:47:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:27 smithi059 ceph-mon[97450]: pgmap v21713: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:47:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:27.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:27.518176+0000) 2022-09-08T08:47:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:27.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:27.293271+0000) 2022-09-08T08:47:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:28.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:28.518402+0000) 2022-09-08T08:47:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:28.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:28.293445+0000) 2022-09-08T08:47:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:29 smithi055 ceph-mon[97033]: pgmap v21714: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:47:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:29 smithi059 ceph-mon[97450]: pgmap v21714: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:47:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:29.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:29.518620+0000) 2022-09-08T08:47:29.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:29.291+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:29.293605+0000) 2022-09-08T08:47:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:47:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:47:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:30 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:47:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:47:30.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:47:30.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:30 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:47:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:30.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:30.518872+0000) 2022-09-08T08:47:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:30.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:30.293803+0000) 2022-09-08T08:47:31.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:31 smithi055 ceph-mon[97033]: pgmap v21715: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 7 op/s 2022-09-08T08:47:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:31 smithi059 ceph-mon[97450]: pgmap v21715: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 7 op/s 2022-09-08T08:47:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:31.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:31.519027+0000) 2022-09-08T08:47:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:31.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:31.293925+0000) 2022-09-08T08:47:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:32.517+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:32.519229+0000) 2022-09-08T08:47:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:32.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:32.294104+0000) 2022-09-08T08:47:33.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:33 smithi055 ceph-mon[97033]: pgmap v21716: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:47:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:33 smithi059 ceph-mon[97450]: pgmap v21716: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:47:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:33.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:33.519420+0000) 2022-09-08T08:47:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:33.292+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:33.294275+0000) 2022-09-08T08:47:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:34.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:34.294427+0000) 2022-09-08T08:47:34.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:34.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:34.519608+0000) 2022-09-08T08:47:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:35 smithi055 ceph-mon[97033]: pgmap v21717: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:47:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:35 smithi059 ceph-mon[97450]: pgmap v21717: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:47:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:35.518+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:35.519849+0000) 2022-09-08T08:47:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:35.293+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:35.294586+0000) 2022-09-08T08:47:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:36.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:36.520005+0000) 2022-09-08T08:47:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:36.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:36.294745+0000) 2022-09-08T08:47:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:37 smithi055 ceph-mon[97033]: pgmap v21718: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:47:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:37 smithi059 ceph-mon[97450]: pgmap v21718: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:47:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:37.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:37.520179+0000) 2022-09-08T08:47:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:37.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:37.294915+0000) 2022-09-08T08:47:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:38.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:38.520325+0000) 2022-09-08T08:47:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:38.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:38.295048+0000) 2022-09-08T08:47:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:39 smithi055 ceph-mon[97033]: pgmap v21719: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:47:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:39 smithi059 ceph-mon[97450]: pgmap v21719: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:47:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:39.519+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:39.520468+0000) 2022-09-08T08:47:39.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:39.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:39.295263+0000) 2022-09-08T08:47:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:40.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:40.520736+0000) 2022-09-08T08:47:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:40.294+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:40.295498+0000) 2022-09-08T08:47:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:41 smithi055 ceph-mon[97033]: pgmap v21720: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-08T08:47:41.524 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:47:41.264+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:47:41.525 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:41.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:41.295631+0000) 2022-09-08T08:47:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:41 smithi059 ceph-mon[97450]: pgmap v21720: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-08T08:47:41.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:41.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:41.520997+0000) 2022-09-08T08:47:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:42.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:42.521276+0000) 2022-09-08T08:47:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:42.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:42.295801+0000) 2022-09-08T08:47:43.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:43 smithi055 ceph-mon[97033]: pgmap v21721: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.3 KiB/s rd, 5 op/s 2022-09-08T08:47:43.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:43 smithi059 ceph-mon[97450]: pgmap v21721: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.3 KiB/s rd, 5 op/s 2022-09-08T08:47:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:43.520+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:43.521561+0000) 2022-09-08T08:47:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:43.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:43.295952+0000) 2022-09-08T08:47:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:44.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:44.521823+0000) 2022-09-08T08:47:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:44.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:44.296147+0000) 2022-09-08T08:47:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:45 smithi055 ceph-mon[97033]: pgmap v21722: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T08:47:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:45 smithi059 ceph-mon[97450]: pgmap v21722: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T08:47:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:45.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:45.522064+0000) 2022-09-08T08:47:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:45.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:45.296399+0000) 2022-09-08T08:47:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:46.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:46.522287+0000) 2022-09-08T08:47:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:46.295+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:46.296613+0000) 2022-09-08T08:47:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:47 smithi055 ceph-mon[97033]: pgmap v21723: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:47:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:47 smithi059 ceph-mon[97450]: pgmap v21723: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:47:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:47.521+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:47.522492+0000) 2022-09-08T08:47:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:47.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:47.296794+0000) 2022-09-08T08:47:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:48.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:48.522749+0000) 2022-09-08T08:47:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:48.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:48.296922+0000) 2022-09-08T08:47:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:49 smithi055 ceph-mon[97033]: pgmap v21724: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T08:47:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:49 smithi059 ceph-mon[97450]: pgmap v21724: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-09-08T08:47:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:49.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:49.522974+0000) 2022-09-08T08:47:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:49.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:49.297075+0000) 2022-09-08T08:47:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:50.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:50.297289+0000) 2022-09-08T08:47:50.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:50.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:50.523224+0000) 2022-09-08T08:47:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:51 smithi055 ceph-mon[97033]: pgmap v21725: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-08T08:47:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:51 smithi059 ceph-mon[97450]: pgmap v21725: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-08T08:47:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:51.522+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:51.523400+0000) 2022-09-08T08:47:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:51.296+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:51.297437+0000) 2022-09-08T08:47:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:52.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:52.523619+0000) 2022-09-08T08:47:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:52.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:52.297688+0000) 2022-09-08T08:47:53.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:53 smithi055 ceph-mon[97033]: pgmap v21726: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:47:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:53 smithi059 ceph-mon[97450]: pgmap v21726: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:47:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:53.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:53.523886+0000) 2022-09-08T08:47:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:53.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:53.297932+0000) 2022-09-08T08:47:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:54.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:54.524052+0000) 2022-09-08T08:47:54.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:54.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:54.298166+0000) 2022-09-08T08:47:55.403 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:55.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:55.298321+0000) 2022-09-08T08:47:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:55 smithi055 ceph-mon[97033]: pgmap v21727: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:47:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:55 smithi059 ceph-mon[97450]: pgmap v21727: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:47:55.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:55.523+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:55.524204+0000) 2022-09-08T08:47:56.527 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:47:56.273+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:47:56.528 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:56.297+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:56.298521+0000) 2022-09-08T08:47:56.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:56.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:56.524472+0000) 2022-09-08T08:47:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:57 smithi055 ceph-mon[97033]: pgmap v21728: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:47:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:57 smithi059 ceph-mon[97450]: pgmap v21728: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:47:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:57.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:57.524692+0000) 2022-09-08T08:47:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:57.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:57.298759+0000) 2022-09-08T08:47:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:58.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:58.524896+0000) 2022-09-08T08:47:58.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:58.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:58.298924+0000) 2022-09-08T08:47:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:47:59 smithi055 ceph-mon[97033]: pgmap v21729: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:47:59.541 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:47:59 smithi059 ceph-mon[97450]: pgmap v21729: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:47:59.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:47:59.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:59.525122+0000) 2022-09-08T08:47:59.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:47:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:47:59.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:47:59.299080+0000) 2022-09-08T08:48:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:00.524+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:00.525299+0000) 2022-09-08T08:48:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:00.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:00.299219+0000) 2022-09-08T08:48:01.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:01 smithi055 ceph-mon[97033]: pgmap v21730: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:48:01.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:01 smithi055 ceph-mon[97033]: pgmap v21731: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T08:48:01.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:01 smithi055 ceph-mon[97033]: pgmap v21732: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:48:01.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:48:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:01 smithi059 ceph-mon[97450]: pgmap v21730: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:48:01.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:01 smithi059 ceph-mon[97450]: pgmap v21731: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T08:48:01.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:01 smithi059 ceph-mon[97450]: pgmap v21732: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:48:01.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:48:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:01.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:01.525532+0000) 2022-09-08T08:48:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:01.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:01.299440+0000) 2022-09-08T08:48:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:02.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:02.525806+0000) 2022-09-08T08:48:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:02.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:02.299630+0000) 2022-09-08T08:48:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:03 smithi055 ceph-mon[97033]: pgmap v21733: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:48:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:03 smithi059 ceph-mon[97450]: pgmap v21733: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:48:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:03.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:03.525982+0000) 2022-09-08T08:48:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:03.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:03.299830+0000) 2022-09-08T08:48:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:04.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:04.526237+0000) 2022-09-08T08:48:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:04.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:04.299963+0000) 2022-09-08T08:48:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:05 smithi055 ceph-mon[97033]: pgmap v21734: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:48:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:05 smithi059 ceph-mon[97450]: pgmap v21734: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:48:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:05.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:05.526489+0000) 2022-09-08T08:48:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:05.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:05.300100+0000) 2022-09-08T08:48:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:48:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:48:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:06.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:06.300308+0000) 2022-09-08T08:48:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:06.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:06.526735+0000) 2022-09-08T08:48:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:07 smithi055 ceph-mon[97033]: pgmap v21735: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T08:48:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:48:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:07 smithi059 ceph-mon[97450]: pgmap v21735: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T08:48:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:48:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:07.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:07.526895+0000) 2022-09-08T08:48:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:07.298+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:07.300477+0000) 2022-09-08T08:48:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:08.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:08.527086+0000) 2022-09-08T08:48:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:08.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:08.300730+0000) 2022-09-08T08:48:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:09 smithi055 ceph-mon[97033]: pgmap v21736: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 65 KiB/s wr, 25 op/s 2022-09-08T08:48:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:09 smithi059 ceph-mon[97450]: pgmap v21736: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 65 KiB/s wr, 25 op/s 2022-09-08T08:48:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:09.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:09.300889+0000) 2022-09-08T08:48:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:09.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:09.527233+0000) 2022-09-08T08:48:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:10.525+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:10.527486+0000) 2022-09-08T08:48:10.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:10.299+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:10.301068+0000) 2022-09-08T08:48:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:11 smithi055 ceph-mon[97033]: osdmap e1436: 12 total, 11 up, 11 in 2022-09-08T08:48:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:11 smithi055 ceph-mon[97033]: pgmap v21738: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:48:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:11 smithi059 ceph-mon[97450]: osdmap e1436: 12 total, 11 up, 11 in 2022-09-08T08:48:11.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:11 smithi059 ceph-mon[97450]: pgmap v21738: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:48:11.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:48:11.288+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:48:11.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:11.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:11.301289+0000) 2022-09-08T08:48:11.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:11.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:11.527691+0000) 2022-09-08T08:48:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:12.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:12.527907+0000) 2022-09-08T08:48:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:12.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:12.301490+0000) 2022-09-08T08:48:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:13 smithi055 ceph-mon[97033]: pgmap v21739: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:48:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:13 smithi059 ceph-mon[97450]: pgmap v21739: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:48:13.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:13.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:13.528155+0000) 2022-09-08T08:48:13.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:13.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:13.301692+0000) 2022-09-08T08:48:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:14.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:14.528417+0000) 2022-09-08T08:48:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:14.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:14.301937+0000) 2022-09-08T08:48:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:15 smithi055 ceph-mon[97033]: pgmap v21740: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:48:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:15 smithi059 ceph-mon[97450]: pgmap v21740: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:48:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:15.526+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:15.528584+0000) 2022-09-08T08:48:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:15.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:15.302101+0000) 2022-09-08T08:48:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:16.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:16.528795+0000) 2022-09-08T08:48:16.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:16.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:16.302263+0000) 2022-09-08T08:48:17.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:17 smithi055 ceph-mon[97033]: pgmap v21741: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:48:17.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:17 smithi059 ceph-mon[97450]: pgmap v21741: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:48:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:17.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:17.529005+0000) 2022-09-08T08:48:17.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:17.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:17.302412+0000) 2022-09-08T08:48:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:18.300+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:18.302611+0000) 2022-09-08T08:48:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:18.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:18.529267+0000) 2022-09-08T08:48:19.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:19 smithi055 ceph-mon[97033]: pgmap v21742: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:48:19.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:19 smithi055 ceph-mon[97033]: osdmap e1437: 12 total, 11 up, 11 in 2022-09-08T08:48:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:19 smithi059 ceph-mon[97450]: pgmap v21742: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:48:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:19 smithi059 ceph-mon[97450]: osdmap e1437: 12 total, 11 up, 11 in 2022-09-08T08:48:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:19.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:19.302777+0000) 2022-09-08T08:48:19.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:19.527+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:19.529531+0000) 2022-09-08T08:48:20.532 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:20.301+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:20.302937+0000) 2022-09-08T08:48:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:20.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:20.529790+0000) 2022-09-08T08:48:21.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:21 smithi055 ceph-mon[97033]: pgmap v21744: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:48:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:21 smithi059 ceph-mon[97450]: pgmap v21744: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:48:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:21.528+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:21.530032+0000) 2022-09-08T08:48:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:21.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:21.303110+0000) 2022-09-08T08:48:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:22.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:22.530230+0000) 2022-09-08T08:48:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:22.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:22.303318+0000) 2022-09-08T08:48:23.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:23 smithi055 ceph-mon[97033]: pgmap v21745: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:48:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:23 smithi059 ceph-mon[97450]: pgmap v21745: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:48:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:23.529+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:23.530411+0000) 2022-09-08T08:48:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:23.302+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:23.303519+0000) 2022-09-08T08:48:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:24.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:24.530673+0000) 2022-09-08T08:48:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:24.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:24.303776+0000) 2022-09-08T08:48:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:25 smithi055 ceph-mon[97033]: pgmap v21746: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:48:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:25 smithi059 ceph-mon[97450]: pgmap v21746: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:48:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:25.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:25.530937+0000) 2022-09-08T08:48:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:25.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:25.303953+0000) 2022-09-08T08:48:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:26.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:26.531218+0000) 2022-09-08T08:48:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:26.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:26.304092+0000) 2022-09-08T08:48:26.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:48:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:48:26.302+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:48:27.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:27 smithi055 ceph-mon[97033]: pgmap v21747: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:48:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:27 smithi059 ceph-mon[97450]: pgmap v21747: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:48:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:27.530+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:27.531459+0000) 2022-09-08T08:48:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:27.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:27.304246+0000) 2022-09-08T08:48:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:28.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:28.531718+0000) 2022-09-08T08:48:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:28.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:28.304419+0000) 2022-09-08T08:48:29.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:29 smithi055 ceph-mon[97033]: pgmap v21748: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:48:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:29 smithi059 ceph-mon[97450]: pgmap v21748: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:48:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:29.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:29.531952+0000) 2022-09-08T08:48:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:29.303+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:29.304571+0000) 2022-09-08T08:48:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:30.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:30.532222+0000) 2022-09-08T08:48:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:30.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:30.304764+0000) 2022-09-08T08:48:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:31 smithi055 ceph-mon[97033]: pgmap v21749: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T08:48:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:31 smithi059 ceph-mon[97450]: pgmap v21749: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T08:48:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:31.531+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:31.532517+0000) 2022-09-08T08:48:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:31.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:31.304968+0000) 2022-09-08T08:48:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:32.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:32.532704+0000) 2022-09-08T08:48:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:32.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:32.305149+0000) 2022-09-08T08:48:33.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:33 smithi055 ceph-mon[97033]: pgmap v21750: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:33.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:33 smithi059 ceph-mon[97450]: pgmap v21750: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:33.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:33.532937+0000) 2022-09-08T08:48:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:33.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:33.305382+0000) 2022-09-08T08:48:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:34.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:34.533190+0000) 2022-09-08T08:48:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:34.304+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:34.305562+0000) 2022-09-08T08:48:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:35 smithi055 ceph-mon[97033]: pgmap v21751: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:35 smithi059 ceph-mon[97450]: pgmap v21751: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:35.532+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:35.533485+0000) 2022-09-08T08:48:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:35.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:35.305760+0000) 2022-09-08T08:48:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:36.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:36.533772+0000) 2022-09-08T08:48:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:36.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:36.306004+0000) 2022-09-08T08:48:37.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:37 smithi055 ceph-mon[97033]: pgmap v21752: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:48:37.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:37 smithi059 ceph-mon[97450]: pgmap v21752: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:48:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:37.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:37.534056+0000) 2022-09-08T08:48:37.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:37.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:37.306184+0000) 2022-09-08T08:48:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:38.533+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:38.534344+0000) 2022-09-08T08:48:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:38.305+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:38.306453+0000) 2022-09-08T08:48:39.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:39 smithi055 ceph-mon[97033]: pgmap v21753: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:39.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:39 smithi059 ceph-mon[97450]: pgmap v21753: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:39.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:39.534547+0000) 2022-09-08T08:48:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:39.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:39.306616+0000) 2022-09-08T08:48:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:40.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:40.534685+0000) 2022-09-08T08:48:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:40.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:40.306825+0000) 2022-09-08T08:48:41.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:41 smithi055 ceph-mon[97033]: pgmap v21754: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:48:41.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:41 smithi059 ceph-mon[97450]: pgmap v21754: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:48:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:41.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:41.534828+0000) 2022-09-08T08:48:41.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:41.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:41.307000+0000) 2022-09-08T08:48:41.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:48:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:48:41.310+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:48:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:42.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:42.535014+0000) 2022-09-08T08:48:42.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:42.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:42.307161+0000) 2022-09-08T08:48:43.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:43 smithi055 ceph-mon[97033]: pgmap v21755: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:43.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:43 smithi059 ceph-mon[97450]: pgmap v21755: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:43.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:43.535183+0000) 2022-09-08T08:48:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:43.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:43.307368+0000) 2022-09-08T08:48:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:44.306+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:44.307544+0000) 2022-09-08T08:48:44.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:44.534+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:44.535396+0000) 2022-09-08T08:48:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:45 smithi055 ceph-mon[97033]: pgmap v21756: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:45 smithi059 ceph-mon[97450]: pgmap v21756: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:45.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:45.307700+0000) 2022-09-08T08:48:45.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:45.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:45.535660+0000) 2022-09-08T08:48:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:46.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:46.535796+0000) 2022-09-08T08:48:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:46.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:46.307830+0000) 2022-09-08T08:48:47.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:47 smithi055 ceph-mon[97033]: pgmap v21757: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:48:47.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:47 smithi059 ceph-mon[97450]: pgmap v21757: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:48:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:47.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:47.535971+0000) 2022-09-08T08:48:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:47.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:47.308015+0000) 2022-09-08T08:48:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:48.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:48.536199+0000) 2022-09-08T08:48:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:48.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:48.308165+0000) 2022-09-08T08:48:49.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:49 smithi055 ceph-mon[97033]: pgmap v21758: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:49.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:49 smithi059 ceph-mon[97450]: pgmap v21758: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:49.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:49.536410+0000) 2022-09-08T08:48:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:49.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:49.308402+0000) 2022-09-08T08:48:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:50.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:50.536665+0000) 2022-09-08T08:48:50.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:50.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:50.308688+0000) 2022-09-08T08:48:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:51 smithi055 ceph-mon[97033]: pgmap v21759: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:48:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:51 smithi059 ceph-mon[97450]: pgmap v21759: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:48:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:51.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:51.536891+0000) 2022-09-08T08:48:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:51.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:51.308848+0000) 2022-09-08T08:48:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:52.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:52.537064+0000) 2022-09-08T08:48:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:52.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:52.309014+0000) 2022-09-08T08:48:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:53 smithi055 ceph-mon[97033]: pgmap v21760: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:53 smithi059 ceph-mon[97450]: pgmap v21760: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:53.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:53.309237+0000) 2022-09-08T08:48:53.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:53.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:53.537220+0000) 2022-09-08T08:48:54.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:54.535+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:54.537441+0000) 2022-09-08T08:48:54.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:54.307+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:54.309509+0000) 2022-09-08T08:48:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:55 smithi055 ceph-mon[97033]: pgmap v21761: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:55 smithi059 ceph-mon[97450]: pgmap v21761: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:55.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:55.309753+0000) 2022-09-08T08:48:55.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:55.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:55.537691+0000) 2022-09-08T08:48:56.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:48:56.319+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:48:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:56.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:56.537921+0000) 2022-09-08T08:48:56.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:56.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:56.309913+0000) 2022-09-08T08:48:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:57 smithi055 ceph-mon[97033]: pgmap v21762: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:48:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:57 smithi059 ceph-mon[97450]: pgmap v21762: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:48:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:57.536+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:57.538106+0000) 2022-09-08T08:48:57.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:57.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:57.310126+0000) 2022-09-08T08:48:58.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:58.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:58.538327+0000) 2022-09-08T08:48:58.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:58.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:58.310341+0000) 2022-09-08T08:48:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:48:59 smithi055 ceph-mon[97033]: pgmap v21763: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:48:59 smithi059 ceph-mon[97450]: pgmap v21763: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:48:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:48:59.308+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:59.310578+0000) 2022-09-08T08:48:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:48:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:48:59.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:48:59.538568+0000) 2022-09-08T08:49:00.542 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:00.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:00.310713+0000) 2022-09-08T08:49:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:00.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:00.538712+0000) 2022-09-08T08:49:01.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:01 smithi055 ceph-mon[97033]: pgmap v21764: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:49:01.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:49:01.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:49:01.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:49:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:01 smithi059 ceph-mon[97450]: pgmap v21764: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:49:01.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:49:01.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:49:01.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:49:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:01.537+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:01.538924+0000) 2022-09-08T08:49:01.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:01.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:01.310886+0000) 2022-09-08T08:49:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:02.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:02.539121+0000) 2022-09-08T08:49:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:02.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:02.311059+0000) 2022-09-08T08:49:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:03 smithi059 ceph-mon[97450]: pgmap v21765: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:03.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:03.539316+0000) 2022-09-08T08:49:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:03.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:03.311211+0000) 2022-09-08T08:49:03.740 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:03 smithi055 ceph-mon[97033]: pgmap v21765: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:04.662 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:04.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:04.539510+0000) 2022-09-08T08:49:04.663 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:04.309+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:04.311457+0000) 2022-09-08T08:49:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:05 smithi055 ceph-mon[97033]: pgmap v21766: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:05 smithi059 ceph-mon[97450]: pgmap v21766: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:05.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:05.311657+0000) 2022-09-08T08:49:05.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:05.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:05.539734+0000) 2022-09-08T08:49:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:49:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:49:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:06.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:06.539983+0000) 2022-09-08T08:49:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:06.310+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:06.311855+0000) 2022-09-08T08:49:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:07 smithi055 ceph-mon[97033]: pgmap v21767: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:49:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:49:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:07 smithi059 ceph-mon[97450]: pgmap v21767: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:49:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:49:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:07.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:07.540218+0000) 2022-09-08T08:49:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:07.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:07.312066+0000) 2022-09-08T08:49:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:08.538+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:08.540461+0000) 2022-09-08T08:49:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:08.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:08.312307+0000) 2022-09-08T08:49:09.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:09 smithi055 ceph-mon[97033]: pgmap v21768: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:49:09.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:09 smithi059 ceph-mon[97450]: pgmap v21768: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:49:09.786 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:09.539+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:09.540684+0000) 2022-09-08T08:49:09.787 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:09.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:09.312452+0000) 2022-09-08T08:49:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:10.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:10.540931+0000) 2022-09-08T08:49:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:10.311+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:10.312696+0000) 2022-09-08T08:49:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:11 smithi055 ceph-mon[97033]: osdmap e1438: 12 total, 11 up, 11 in 2022-09-08T08:49:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:11 smithi055 ceph-mon[97033]: pgmap v21770: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:49:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:11 smithi059 ceph-mon[97450]: osdmap e1438: 12 total, 11 up, 11 in 2022-09-08T08:49:11.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:11 smithi059 ceph-mon[97450]: pgmap v21770: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:49:11.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:11.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:11.541168+0000) 2022-09-08T08:49:11.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:11.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:11.312881+0000) 2022-09-08T08:49:11.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:49:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:49:11.335+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:49:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:12.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:12.541334+0000) 2022-09-08T08:49:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:12.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:12.313031+0000) 2022-09-08T08:49:13.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:13 smithi055 ceph-mon[97033]: pgmap v21771: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:49:13.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:13 smithi059 ceph-mon[97450]: pgmap v21771: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:49:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:13.540+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:13.541531+0000) 2022-09-08T08:49:13.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:13.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:13.313259+0000) 2022-09-08T08:49:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:14.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:14.541759+0000) 2022-09-08T08:49:14.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:14.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:14.313465+0000) 2022-09-08T08:49:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:15 smithi055 ceph-mon[97033]: pgmap v21772: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:49:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:15 smithi059 ceph-mon[97450]: pgmap v21772: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:49:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:15.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:15.541899+0000) 2022-09-08T08:49:15.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:15.312+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:15.313612+0000) 2022-09-08T08:49:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:16.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:16.542108+0000) 2022-09-08T08:49:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:16.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:16.313727+0000) 2022-09-08T08:49:17.484 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:17 smithi055 ceph-mon[97033]: pgmap v21773: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:49:17.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:17 smithi059 ceph-mon[97450]: pgmap v21773: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:49:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:17.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:17.542241+0000) 2022-09-08T08:49:17.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:17.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:17.313886+0000) 2022-09-08T08:49:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:18.541+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:18.542453+0000) 2022-09-08T08:49:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:18.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:18.314040+0000) 2022-09-08T08:49:19.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:19 smithi055 ceph-mon[97033]: pgmap v21774: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:49:19.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:19 smithi055 ceph-mon[97033]: osdmap e1439: 12 total, 11 up, 11 in 2022-09-08T08:49:19.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:19 smithi059 ceph-mon[97450]: pgmap v21774: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:49:19.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:19 smithi059 ceph-mon[97450]: osdmap e1439: 12 total, 11 up, 11 in 2022-09-08T08:49:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:19.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:19.542725+0000) 2022-09-08T08:49:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:19.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:19.314236+0000) 2022-09-08T08:49:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:20.313+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:20.314481+0000) 2022-09-08T08:49:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:20.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:20.542970+0000) 2022-09-08T08:49:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:21 smithi059 ceph-mon[97450]: pgmap v21776: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:49:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:21.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:21.314665+0000) 2022-09-08T08:49:21.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:21.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:21.543191+0000) 2022-09-08T08:49:21.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:21 smithi055 ceph-mon[97033]: pgmap v21776: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:49:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:22.542+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:22.543416+0000) 2022-09-08T08:49:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:22.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:22.314873+0000) 2022-09-08T08:49:23.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:23 smithi059 ceph-mon[97450]: pgmap v21777: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:49:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:23.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:23.543661+0000) 2022-09-08T08:49:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:23.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:23.315016+0000) 2022-09-08T08:49:23.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:23 smithi055 ceph-mon[97033]: pgmap v21777: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:49:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:24.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:24.543831+0000) 2022-09-08T08:49:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:24.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:24.315188+0000) 2022-09-08T08:49:25.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:25 smithi055 ceph-mon[97033]: pgmap v21778: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:49:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:25 smithi059 ceph-mon[97450]: pgmap v21778: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:49:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:25.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:25.543995+0000) 2022-09-08T08:49:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:25.314+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:25.315432+0000) 2022-09-08T08:49:26.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:49:26.337+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:49:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:26.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:26.544232+0000) 2022-09-08T08:49:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:26.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:26.315679+0000) 2022-09-08T08:49:27.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:27 smithi059 ceph-mon[97450]: pgmap v21779: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:49:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:27.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:27.544368+0000) 2022-09-08T08:49:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:27.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:27.315861+0000) 2022-09-08T08:49:27.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:27 smithi055 ceph-mon[97033]: pgmap v21779: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:49:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:28.543+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:28.544576+0000) 2022-09-08T08:49:28.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:28.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:28.316024+0000) 2022-09-08T08:49:29.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:29 smithi059 ceph-mon[97450]: pgmap v21780: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:49:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:29.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:29.544825+0000) 2022-09-08T08:49:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:29.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:29.316195+0000) 2022-09-08T08:49:29.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:29 smithi055 ceph-mon[97033]: pgmap v21780: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:49:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:30.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:30.545036+0000) 2022-09-08T08:49:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:30.315+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:30.316409+0000) 2022-09-08T08:49:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:31.544+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:31.545233+0000) 2022-09-08T08:49:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:31.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:31.316673+0000) 2022-09-08T08:49:31.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:31 smithi055 ceph-mon[97033]: pgmap v21781: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T08:49:31.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:31 smithi055 ceph-mon[97033]: pgmap v21782: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:49:31.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:31 smithi055 ceph-mon[97033]: pgmap v21783: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:49:31.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:49:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:31 smithi059 ceph-mon[97450]: pgmap v21781: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T08:49:31.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:31 smithi059 ceph-mon[97450]: pgmap v21782: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:49:31.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:31 smithi059 ceph-mon[97450]: pgmap v21783: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:49:31.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:49:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:32.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:32.545451+0000) 2022-09-08T08:49:32.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:32.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:32.316835+0000) 2022-09-08T08:49:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:33.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:33.545676+0000) 2022-09-08T08:49:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:33.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:33.317029+0000) 2022-09-08T08:49:33.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:33 smithi055 ceph-mon[97033]: pgmap v21784: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:49:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:33 smithi059 ceph-mon[97450]: pgmap v21784: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:49:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:34.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:34.545799+0000) 2022-09-08T08:49:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:34.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:34.317175+0000) 2022-09-08T08:49:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:35 smithi055 ceph-mon[97033]: pgmap v21785: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:49:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:35 smithi059 ceph-mon[97450]: pgmap v21785: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:49:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:35.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:35.545951+0000) 2022-09-08T08:49:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:35.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:35.317332+0000) 2022-09-08T08:49:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:36.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:36.546097+0000) 2022-09-08T08:49:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:36.316+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:36.317550+0000) 2022-09-08T08:49:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:37.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:37.546310+0000) 2022-09-08T08:49:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:37.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:37.317719+0000) 2022-09-08T08:49:37.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:37 smithi055 ceph-mon[97033]: pgmap v21786: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:49:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:37 smithi059 ceph-mon[97450]: pgmap v21786: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:49:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:38.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:38.546554+0000) 2022-09-08T08:49:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:38.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:38.317885+0000) 2022-09-08T08:49:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:39.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:39.546783+0000) 2022-09-08T08:49:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:39.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:39.318022+0000) 2022-09-08T08:49:39.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:39 smithi055 ceph-mon[97033]: pgmap v21787: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:49:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:39 smithi059 ceph-mon[97450]: pgmap v21787: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:49:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:40.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:40.547010+0000) 2022-09-08T08:49:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:40.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:40.318226+0000) 2022-09-08T08:49:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:41.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:41.547230+0000) 2022-09-08T08:49:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:41.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:41.318416+0000) 2022-09-08T08:49:41.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:49:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:49:41.337+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:49:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:41 smithi059 ceph-mon[97450]: pgmap v21788: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:49:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:41 smithi055 ceph-mon[97033]: pgmap v21788: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:49:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:42.545+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:42.547416+0000) 2022-09-08T08:49:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:42.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:42.318602+0000) 2022-09-08T08:49:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:43.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:43.547697+0000) 2022-09-08T08:49:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:43.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:43.318799+0000) 2022-09-08T08:49:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:43 smithi059 ceph-mon[97450]: pgmap v21789: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:43 smithi055 ceph-mon[97033]: pgmap v21789: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:44.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:44.547962+0000) 2022-09-08T08:49:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:44.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:44.318988+0000) 2022-09-08T08:49:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:45 smithi055 ceph-mon[97033]: pgmap v21790: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:45 smithi059 ceph-mon[97450]: pgmap v21790: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:45.546+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:45.548215+0000) 2022-09-08T08:49:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:45.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:45.319156+0000) 2022-09-08T08:49:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:46.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:46.548473+0000) 2022-09-08T08:49:46.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:46.317+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:46.319362+0000) 2022-09-08T08:49:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:47.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:47.548679+0000) 2022-09-08T08:49:47.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:47.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:47.319521+0000) 2022-09-08T08:49:47.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:47 smithi055 ceph-mon[97033]: pgmap v21791: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:49:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:47 smithi059 ceph-mon[97450]: pgmap v21791: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:49:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:48.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:48.548857+0000) 2022-09-08T08:49:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:48.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:48.319696+0000) 2022-09-08T08:49:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:49.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:49.549028+0000) 2022-09-08T08:49:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:49.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:49.319835+0000) 2022-09-08T08:49:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:49 smithi059 ceph-mon[97450]: pgmap v21792: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:49 smithi055 ceph-mon[97033]: pgmap v21792: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:50.547+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:50.549215+0000) 2022-09-08T08:49:50.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:50.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:50.319991+0000) 2022-09-08T08:49:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:51.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:51.549415+0000) 2022-09-08T08:49:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:51.318+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:51.320122+0000) 2022-09-08T08:49:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:51 smithi059 ceph-mon[97450]: pgmap v21793: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:49:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:51 smithi055 ceph-mon[97033]: pgmap v21793: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:49:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:52.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:52.549573+0000) 2022-09-08T08:49:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:52.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:52.320303+0000) 2022-09-08T08:49:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:53.548+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:53.549790+0000) 2022-09-08T08:49:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:53.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:53.320502+0000) 2022-09-08T08:49:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:53 smithi059 ceph-mon[97450]: pgmap v21794: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:53 smithi055 ceph-mon[97033]: pgmap v21794: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:54.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:54.550039+0000) 2022-09-08T08:49:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:54.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:54.320652+0000) 2022-09-08T08:49:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:55 smithi055 ceph-mon[97033]: pgmap v21795: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:55 smithi059 ceph-mon[97450]: pgmap v21795: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:55.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:55.550220+0000) 2022-09-08T08:49:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:55.319+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:55.320776+0000) 2022-09-08T08:49:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:56.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:56.320926+0000) 2022-09-08T08:49:56.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:49:56.353+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:49:56.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:56.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:56.550424+0000) 2022-09-08T08:49:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:57.549+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:57.550716+0000) 2022-09-08T08:49:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:57.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:57.321159+0000) 2022-09-08T08:49:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:57 smithi059 ceph-mon[97450]: pgmap v21796: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:49:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:57 smithi055 ceph-mon[97033]: pgmap v21796: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:49:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:58.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:58.550912+0000) 2022-09-08T08:49:58.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:58.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:58.321358+0000) 2022-09-08T08:49:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:49:59.320+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:59.321515+0000) 2022-09-08T08:49:59.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:49:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:49:59.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:49:59.551087+0000) 2022-09-08T08:49:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:49:59 smithi059 ceph-mon[97450]: pgmap v21797: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:49:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:49:59 smithi055 ceph-mon[97033]: pgmap v21797: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:00.553 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:00.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:00.321767+0000) 2022-09-08T08:50:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T08:50:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:00 smithi059 ceph-mon[97450]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T08:50:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:00 smithi059 ceph-mon[97450]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T08:50:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T08:50:00.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T08:50:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:00.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:00.551261+0000) 2022-09-08T08:50:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum c,b 2022-09-08T08:50:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:00 smithi055 ceph-mon[97033]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2022-09-08T08:50:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:00 smithi055 ceph-mon[97033]: host smithi047 (172.21.15.47) failed check: Unable to reach remote host smithi047. 2022-09-08T08:50:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T08:50:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T08:50:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:01.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:01.321955+0000) 2022-09-08T08:50:01.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:01.550+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:01.551469+0000) 2022-09-08T08:50:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:01 smithi059 ceph-mon[97450]: pgmap v21798: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:50:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:01 smithi055 ceph-mon[97033]: pgmap v21798: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:50:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:02.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:02.551690+0000) 2022-09-08T08:50:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:02.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:02.322125+0000) 2022-09-08T08:50:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:03.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:03.551834+0000) 2022-09-08T08:50:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:03.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:03.322313+0000) 2022-09-08T08:50:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:03 smithi059 ceph-mon[97450]: pgmap v21799: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:03 smithi055 ceph-mon[97033]: pgmap v21799: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:04.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:04.551956+0000) 2022-09-08T08:50:04.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:04.321+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:04.322516+0000) 2022-09-08T08:50:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:05 smithi055 ceph-mon[97033]: pgmap v21800: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:05 smithi059 ceph-mon[97450]: pgmap v21800: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:05.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:05.552150+0000) 2022-09-08T08:50:05.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:05.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:05.322665+0000) 2022-09-08T08:50:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:50:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:50:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:06.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:06.552380+0000) 2022-09-08T08:50:06.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:06.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:06.322886+0000) 2022-09-08T08:50:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:07 smithi055 ceph-mon[97033]: pgmap v21801: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:50:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:50:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:07 smithi059 ceph-mon[97450]: pgmap v21801: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:50:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:50:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:07.551+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:07.552618+0000) 2022-09-08T08:50:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:07.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:07.323102+0000) 2022-09-08T08:50:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:08.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:08.552812+0000) 2022-09-08T08:50:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:08.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:08.323299+0000) 2022-09-08T08:50:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:09.552+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:09.553063+0000) 2022-09-08T08:50:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:09.322+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:09.323464+0000) 2022-09-08T08:50:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:09 smithi059 ceph-mon[97450]: pgmap v21802: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:50:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:09 smithi055 ceph-mon[97033]: pgmap v21802: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:50:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:10.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:10.553810+0000) 2022-09-08T08:50:10.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:10.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:10.323653+0000) 2022-09-08T08:50:11.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:11 smithi055 ceph-mon[97033]: osdmap e1440: 12 total, 11 up, 11 in 2022-09-08T08:50:11.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:11 smithi055 ceph-mon[97033]: pgmap v21804: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:50:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:11 smithi059 ceph-mon[97450]: osdmap e1440: 12 total, 11 up, 11 in 2022-09-08T08:50:11.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:11 smithi059 ceph-mon[97450]: pgmap v21804: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:50:11.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:50:11.362+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:50:11.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:11.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:11.553980+0000) 2022-09-08T08:50:11.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:11.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:11.323809+0000) 2022-09-08T08:50:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:12.553+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:12.554160+0000) 2022-09-08T08:50:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:12.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:12.324072+0000) 2022-09-08T08:50:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:13.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:13.554438+0000) 2022-09-08T08:50:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:13.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:13.324280+0000) 2022-09-08T08:50:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:13 smithi059 ceph-mon[97450]: pgmap v21805: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:50:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:13 smithi055 ceph-mon[97033]: pgmap v21805: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:50:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:14.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:14.554718+0000) 2022-09-08T08:50:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:14.323+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:14.324447+0000) 2022-09-08T08:50:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:15 smithi055 ceph-mon[97033]: pgmap v21806: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:50:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:15 smithi059 ceph-mon[97450]: pgmap v21806: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:50:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:15.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:15.554946+0000) 2022-09-08T08:50:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:15.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:15.324590+0000) 2022-09-08T08:50:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:16.554+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:16.555195+0000) 2022-09-08T08:50:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:16.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:16.324740+0000) 2022-09-08T08:50:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:17.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:17.555441+0000) 2022-09-08T08:50:17.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:17.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:17.324893+0000) 2022-09-08T08:50:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:17 smithi059 ceph-mon[97450]: pgmap v21807: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:50:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:17 smithi055 ceph-mon[97033]: pgmap v21807: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:50:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:18.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:18.555732+0000) 2022-09-08T08:50:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:18.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:18.325069+0000) 2022-09-08T08:50:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:19.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:19.555971+0000) 2022-09-08T08:50:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:19.324+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:19.325287+0000) 2022-09-08T08:50:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:19 smithi059 ceph-mon[97450]: pgmap v21808: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:50:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:19 smithi059 ceph-mon[97450]: osdmap e1441: 12 total, 11 up, 11 in 2022-09-08T08:50:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:19 smithi055 ceph-mon[97033]: pgmap v21808: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:50:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:19 smithi055 ceph-mon[97033]: osdmap e1441: 12 total, 11 up, 11 in 2022-09-08T08:50:20.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:20.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:20.556201+0000) 2022-09-08T08:50:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:20.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:20.325437+0000) 2022-09-08T08:50:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:21.555+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:21.556459+0000) 2022-09-08T08:50:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:21.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:21.325717+0000) 2022-09-08T08:50:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:21 smithi059 ceph-mon[97450]: pgmap v21810: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:50:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:21 smithi055 ceph-mon[97033]: pgmap v21810: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:50:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:22.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:22.556742+0000) 2022-09-08T08:50:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:22.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:22.326002+0000) 2022-09-08T08:50:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:23.556+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:23.556971+0000) 2022-09-08T08:50:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:23.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:23.326160+0000) 2022-09-08T08:50:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:23 smithi059 ceph-mon[97450]: pgmap v21811: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:50:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:23 smithi055 ceph-mon[97033]: pgmap v21811: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:50:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:24.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:24.557231+0000) 2022-09-08T08:50:24.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:24.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:24.326296+0000) 2022-09-08T08:50:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:25 smithi055 ceph-mon[97033]: pgmap v21812: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:50:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:25 smithi059 ceph-mon[97450]: pgmap v21812: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:50:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:25.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:25.557419+0000) 2022-09-08T08:50:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:25.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:25.326428+0000) 2022-09-08T08:50:26.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:50:26.376+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:50:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:26.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:26.557683+0000) 2022-09-08T08:50:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:26.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:26.326587+0000) 2022-09-08T08:50:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:27.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:27.557926+0000) 2022-09-08T08:50:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:27.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:27.326716+0000) 2022-09-08T08:50:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:27 smithi059 ceph-mon[97450]: pgmap v21813: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:50:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:27 smithi055 ceph-mon[97033]: pgmap v21813: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:50:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:28.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:28.558194+0000) 2022-09-08T08:50:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:28.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:28.326931+0000) 2022-09-08T08:50:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:29.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:29.558461+0000) 2022-09-08T08:50:29.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:29.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:29.327065+0000) 2022-09-08T08:50:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:29 smithi059 ceph-mon[97450]: pgmap v21814: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:50:29.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:29 smithi055 ceph-mon[97033]: pgmap v21814: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:50:30.614 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:30.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:30.558666+0000) 2022-09-08T08:50:30.615 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:30.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:30.327289+0000) 2022-09-08T08:50:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:31.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:31.558854+0000) 2022-09-08T08:50:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:31.325+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:31.327522+0000) 2022-09-08T08:50:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:31 smithi059 ceph-mon[97450]: pgmap v21815: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T08:50:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:50:31.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:50:31.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:50:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:31 smithi055 ceph-mon[97033]: pgmap v21815: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T08:50:31.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:50:31.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:50:31.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:50:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:32.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:32.559121+0000) 2022-09-08T08:50:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:32.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:32.327690+0000) 2022-09-08T08:50:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:33.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:33.559267+0000) 2022-09-08T08:50:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:33.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:33.327830+0000) 2022-09-08T08:50:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:33 smithi059 ceph-mon[97450]: pgmap v21816: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:33 smithi055 ceph-mon[97033]: pgmap v21816: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:34.557+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:34.559492+0000) 2022-09-08T08:50:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:34.326+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:34.328066+0000) 2022-09-08T08:50:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:35 smithi055 ceph-mon[97033]: pgmap v21817: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:35 smithi059 ceph-mon[97450]: pgmap v21817: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:35.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:35.559758+0000) 2022-09-08T08:50:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:35.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:35.328289+0000) 2022-09-08T08:50:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:36.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:36.559935+0000) 2022-09-08T08:50:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:36.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:36.328515+0000) 2022-09-08T08:50:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:37.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:37.328699+0000) 2022-09-08T08:50:37.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:37.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:37.560174+0000) 2022-09-08T08:50:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:37 smithi059 ceph-mon[97450]: pgmap v21818: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:50:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:37 smithi055 ceph-mon[97033]: pgmap v21818: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:50:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:38.558+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:38.560413+0000) 2022-09-08T08:50:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:38.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:38.328904+0000) 2022-09-08T08:50:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:39.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:39.560659+0000) 2022-09-08T08:50:39.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:39.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:39.329129+0000) 2022-09-08T08:50:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:39 smithi059 ceph-mon[97450]: pgmap v21819: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:39 smithi055 ceph-mon[97033]: pgmap v21819: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:40.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:40.560860+0000) 2022-09-08T08:50:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:40.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:40.329299+0000) 2022-09-08T08:50:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:41.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:41.561116+0000) 2022-09-08T08:50:41.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:41.327+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:41.329493+0000) 2022-09-08T08:50:41.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:50:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:50:41.390+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:50:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:41 smithi059 ceph-mon[97450]: pgmap v21820: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:50:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:41 smithi055 ceph-mon[97033]: pgmap v21820: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:50:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:42.559+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:42.561396+0000) 2022-09-08T08:50:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:42.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:42.329700+0000) 2022-09-08T08:50:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:43.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:43.561594+0000) 2022-09-08T08:50:43.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:43.328+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:43.329885+0000) 2022-09-08T08:50:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:43 smithi059 ceph-mon[97450]: pgmap v21821: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:43 smithi055 ceph-mon[97033]: pgmap v21821: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:44.560+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:44.561727+0000) 2022-09-08T08:50:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:44.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:44.330120+0000) 2022-09-08T08:50:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:45 smithi055 ceph-mon[97033]: pgmap v21822: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:45 smithi059 ceph-mon[97450]: pgmap v21822: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:45.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:45.561883+0000) 2022-09-08T08:50:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:45.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:45.330364+0000) 2022-09-08T08:50:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:46.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:46.562086+0000) 2022-09-08T08:50:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:46.329+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:46.330625+0000) 2022-09-08T08:50:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:47.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:47.562368+0000) 2022-09-08T08:50:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:47.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:47.330869+0000) 2022-09-08T08:50:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:47 smithi059 ceph-mon[97450]: pgmap v21823: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:50:47.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:47 smithi055 ceph-mon[97033]: pgmap v21823: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:50:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:48.561+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:48.562608+0000) 2022-09-08T08:50:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:48.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:48.331089+0000) 2022-09-08T08:50:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:49.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:49.562782+0000) 2022-09-08T08:50:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:49.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:49.331268+0000) 2022-09-08T08:50:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:49 smithi059 ceph-mon[97450]: pgmap v21824: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:49 smithi055 ceph-mon[97033]: pgmap v21824: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:50:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:50.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:50.563016+0000) 2022-09-08T08:50:50.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:50.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:50.331400+0000) 2022-09-08T08:50:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:51.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:51.563229+0000) 2022-09-08T08:50:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:51.330+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:51.331526+0000) 2022-09-08T08:50:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:51 smithi059 ceph-mon[97450]: pgmap v21825: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:50:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:51 smithi055 ceph-mon[97033]: pgmap v21825: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:50:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:52.562+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:52.563439+0000) 2022-09-08T08:50:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:52.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:52.331770+0000) 2022-09-08T08:50:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:53.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:53.563690+0000) 2022-09-08T08:50:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:53.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:53.331965+0000) 2022-09-08T08:50:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:53 smithi059 ceph-mon[97450]: pgmap v21826: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:50:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:53 smithi055 ceph-mon[97033]: pgmap v21826: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:50:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:54.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:54.563910+0000) 2022-09-08T08:50:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:54.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:54.332192+0000) 2022-09-08T08:50:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:55 smithi055 ceph-mon[97033]: pgmap v21827: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:50:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:55 smithi059 ceph-mon[97450]: pgmap v21827: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:50:55.621 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:55.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:55.564130+0000) 2022-09-08T08:50:55.622 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:55.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:55.332362+0000) 2022-09-08T08:50:56.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:50:56.400+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:50:56.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:56.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:56.564393+0000) 2022-09-08T08:50:56.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:56.331+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:56.332525+0000) 2022-09-08T08:50:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:57.563+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:57.564616+0000) 2022-09-08T08:50:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:57.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:57.332741+0000) 2022-09-08T08:50:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:57 smithi059 ceph-mon[97450]: pgmap v21828: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:50:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:57 smithi055 ceph-mon[97033]: pgmap v21828: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:50:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:58.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:58.564875+0000) 2022-09-08T08:50:58.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:58.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:58.332883+0000) 2022-09-08T08:50:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:50:59.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:59.565038+0000) 2022-09-08T08:50:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:50:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:50:59.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:50:59.333058+0000) 2022-09-08T08:50:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:50:59 smithi059 ceph-mon[97450]: pgmap v21829: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:50:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:50:59 smithi055 ceph-mon[97033]: pgmap v21829: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T08:51:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:00.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:00.565302+0000) 2022-09-08T08:51:00.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:00.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:00.333256+0000) 2022-09-08T08:51:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:01.564+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:01.565518+0000) 2022-09-08T08:51:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:01.332+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:01.333418+0000) 2022-09-08T08:51:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:01 smithi055 ceph-mon[97033]: pgmap v21830: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:51:01.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:01 smithi055 ceph-mon[97033]: pgmap v21831: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:51:01.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:01 smithi055 ceph-mon[97033]: pgmap v21832: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:51:01.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:51:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:01 smithi059 ceph-mon[97450]: pgmap v21830: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:51:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:01 smithi059 ceph-mon[97450]: pgmap v21831: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:51:02.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:01 smithi059 ceph-mon[97450]: pgmap v21832: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:51:02.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:51:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:02.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:02.565692+0000) 2022-09-08T08:51:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:02.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:02.333633+0000) 2022-09-08T08:51:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:03.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:03.565932+0000) 2022-09-08T08:51:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:03.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:03.333814+0000) 2022-09-08T08:51:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:03 smithi055 ceph-mon[97033]: pgmap v21833: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:51:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:03 smithi059 ceph-mon[97450]: pgmap v21833: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:51:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:04.565+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:04.566134+0000) 2022-09-08T08:51:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:04.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:04.334015+0000) 2022-09-08T08:51:05.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:05 smithi055 ceph-mon[97033]: pgmap v21834: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:51:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:05 smithi059 ceph-mon[97450]: pgmap v21834: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:51:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:05.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:05.566377+0000) 2022-09-08T08:51:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:05.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:05.334232+0000) 2022-09-08T08:51:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:51:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:51:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:06.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:06.566565+0000) 2022-09-08T08:51:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:06.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:06.334358+0000) 2022-09-08T08:51:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:07 smithi055 ceph-mon[97033]: pgmap v21835: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T08:51:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:51:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:07 smithi059 ceph-mon[97450]: pgmap v21835: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 65 KiB/s wr, 28 op/s 2022-09-08T08:51:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:51:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:07.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:07.566815+0000) 2022-09-08T08:51:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:07.333+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:07.334542+0000) 2022-09-08T08:51:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:08.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:08.566963+0000) 2022-09-08T08:51:08.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:08.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:08.334795+0000) 2022-09-08T08:51:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:09.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:09.567181+0000) 2022-09-08T08:51:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:09.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:09.334974+0000) 2022-09-08T08:51:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:09 smithi059 ceph-mon[97450]: pgmap v21836: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 15 KiB/s rd, 65 KiB/s wr, 25 op/s 2022-09-08T08:51:10.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:09 smithi055 ceph-mon[97033]: pgmap v21836: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 15 KiB/s rd, 65 KiB/s wr, 25 op/s 2022-09-08T08:51:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:10.566+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:10.567446+0000) 2022-09-08T08:51:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:10.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:10.335217+0000) 2022-09-08T08:51:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:10 smithi059 ceph-mon[97450]: osdmap e1442: 12 total, 11 up, 11 in 2022-09-08T08:51:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:10 smithi055 ceph-mon[97033]: osdmap e1442: 12 total, 11 up, 11 in 2022-09-08T08:51:11.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:51:11.416+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:51:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:11.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:11.567733+0000) 2022-09-08T08:51:11.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:11.334+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:11.335343+0000) 2022-09-08T08:51:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:11 smithi059 ceph-mon[97450]: pgmap v21838: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:51:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:11 smithi055 ceph-mon[97033]: pgmap v21838: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:51:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:12.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:12.567889+0000) 2022-09-08T08:51:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:12.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:12.335516+0000) 2022-09-08T08:51:13.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:12 smithi059 ceph-mon[97450]: pgmap v21839: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:51:13.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:12 smithi055 ceph-mon[97033]: pgmap v21839: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:51:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:13.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:13.568125+0000) 2022-09-08T08:51:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:13.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:13.335734+0000) 2022-09-08T08:51:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:14.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:14.568319+0000) 2022-09-08T08:51:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:14.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:14.335958+0000) 2022-09-08T08:51:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:15 smithi055 ceph-mon[97033]: pgmap v21840: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:51:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:15 smithi059 ceph-mon[97450]: pgmap v21840: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:51:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:15.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:15.568545+0000) 2022-09-08T08:51:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:15.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:15.336204+0000) 2022-09-08T08:51:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:16.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:16.568720+0000) 2022-09-08T08:51:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:16.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:16.336375+0000) 2022-09-08T08:51:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:17.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:17.568943+0000) 2022-09-08T08:51:17.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:17.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:17.336602+0000) 2022-09-08T08:51:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:17 smithi059 ceph-mon[97450]: pgmap v21841: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:51:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:17 smithi055 ceph-mon[97033]: pgmap v21841: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:51:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:18.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:18.569188+0000) 2022-09-08T08:51:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:18.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:18.336856+0000) 2022-09-08T08:51:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:19.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:19.569425+0000) 2022-09-08T08:51:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:19.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:19.336987+0000) 2022-09-08T08:51:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:19 smithi059 ceph-mon[97450]: pgmap v21842: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:51:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:19 smithi059 ceph-mon[97450]: osdmap e1443: 12 total, 11 up, 11 in 2022-09-08T08:51:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:19 smithi055 ceph-mon[97033]: pgmap v21842: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:51:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:19 smithi055 ceph-mon[97033]: osdmap e1443: 12 total, 11 up, 11 in 2022-09-08T08:51:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:20.335+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:20.337206+0000) 2022-09-08T08:51:20.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:20.567+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:20.569592+0000) 2022-09-08T08:51:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:21.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:21.337390+0000) 2022-09-08T08:51:21.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:21.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:21.569789+0000) 2022-09-08T08:51:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:21 smithi059 ceph-mon[97450]: pgmap v21844: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:51:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:21 smithi055 ceph-mon[97033]: pgmap v21844: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:51:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:22.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:22.570025+0000) 2022-09-08T08:51:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:22.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:22.337630+0000) 2022-09-08T08:51:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:23.568+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:23.570209+0000) 2022-09-08T08:51:23.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:23.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:23.337855+0000) 2022-09-08T08:51:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:23 smithi059 ceph-mon[97450]: pgmap v21845: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:51:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:23 smithi055 ceph-mon[97033]: pgmap v21845: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:51:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:24.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:24.570399+0000) 2022-09-08T08:51:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:24.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:24.338020+0000) 2022-09-08T08:51:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:25 smithi055 ceph-mon[97033]: pgmap v21846: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:51:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:25 smithi059 ceph-mon[97450]: pgmap v21846: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:51:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:25.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:25.570622+0000) 2022-09-08T08:51:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:25.336+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:25.338175+0000) 2022-09-08T08:51:26.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:51:26.424+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:51:26.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:26.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:26.570881+0000) 2022-09-08T08:51:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:26.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:26.338403+0000) 2022-09-08T08:51:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:27.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:27.571056+0000) 2022-09-08T08:51:27.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:27.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:27.338581+0000) 2022-09-08T08:51:28.037 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:27 smithi059 ceph-mon[97450]: pgmap v21847: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:51:28.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:27 smithi055 ceph-mon[97033]: pgmap v21847: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:51:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:28.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:28.571286+0000) 2022-09-08T08:51:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:28.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:28.338782+0000) 2022-09-08T08:51:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:29.569+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:29.571461+0000) 2022-09-08T08:51:29.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:29.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:29.338941+0000) 2022-09-08T08:51:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:29 smithi059 ceph-mon[97450]: pgmap v21848: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:51:30.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:29 smithi055 ceph-mon[97033]: pgmap v21848: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:51:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:30.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:30.571676+0000) 2022-09-08T08:51:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:30.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:30.339084+0000) 2022-09-08T08:51:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:31.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:31.571873+0000) 2022-09-08T08:51:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:31.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:31.339229+0000) 2022-09-08T08:51:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:31 smithi059 ceph-mon[97450]: pgmap v21849: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T08:51:32.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:31 smithi055 ceph-mon[97033]: pgmap v21849: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T08:51:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:32.570+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:32.572030+0000) 2022-09-08T08:51:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:32.337+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:32.339432+0000) 2022-09-08T08:51:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:33.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:33.572223+0000) 2022-09-08T08:51:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:33.338+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:33.339676+0000) 2022-09-08T08:51:34.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:33 smithi059 ceph-mon[97450]: pgmap v21850: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:33 smithi055 ceph-mon[97033]: pgmap v21850: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:34.571+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:34.572413+0000) 2022-09-08T08:51:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:34.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:34.339896+0000) 2022-09-08T08:51:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:35 smithi055 ceph-mon[97033]: pgmap v21851: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:35 smithi059 ceph-mon[97450]: pgmap v21851: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:35.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:35.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:35.572674+0000) 2022-09-08T08:51:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:35.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:35.340094+0000) 2022-09-08T08:51:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:36.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:36.572849+0000) 2022-09-08T08:51:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:36.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:36.340256+0000) 2022-09-08T08:51:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:37.339+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:37.340434+0000) 2022-09-08T08:51:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:37.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:37.573050+0000) 2022-09-08T08:51:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:37 smithi059 ceph-mon[97450]: pgmap v21852: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:51:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:37 smithi055 ceph-mon[97033]: pgmap v21852: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:51:38.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:38.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:38.573226+0000) 2022-09-08T08:51:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:38.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:38.340631+0000) 2022-09-08T08:51:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:39.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:39.573395+0000) 2022-09-08T08:51:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:39.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:39.340869+0000) 2022-09-08T08:51:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:39 smithi059 ceph-mon[97450]: pgmap v21853: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:40.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:39 smithi055 ceph-mon[97033]: pgmap v21853: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:40.572+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:40.573571+0000) 2022-09-08T08:51:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:40.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:40.341123+0000) 2022-09-08T08:51:41.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:51:41.432+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:51:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:41.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:41.573693+0000) 2022-09-08T08:51:41.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:41.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:41.341321+0000) 2022-09-08T08:51:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:41 smithi059 ceph-mon[97450]: pgmap v21854: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:51:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:41 smithi055 ceph-mon[97033]: pgmap v21854: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:51:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:42.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:42.573946+0000) 2022-09-08T08:51:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:42.340+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:42.341470+0000) 2022-09-08T08:51:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:42 smithi055 ceph-mon[97033]: pgmap v21855: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:42 smithi059 ceph-mon[97450]: pgmap v21855: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:43.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:43.574188+0000) 2022-09-08T08:51:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:43.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:43.341735+0000) 2022-09-08T08:51:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:44.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:44.574334+0000) 2022-09-08T08:51:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:44.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:44.341904+0000) 2022-09-08T08:51:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:45 smithi055 ceph-mon[97033]: pgmap v21856: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:45 smithi059 ceph-mon[97450]: pgmap v21856: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:45.573+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:45.574487+0000) 2022-09-08T08:51:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:45.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:45.342055+0000) 2022-09-08T08:51:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:46.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:46.574737+0000) 2022-09-08T08:51:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:46.341+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:46.342295+0000) 2022-09-08T08:51:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:47.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:47.574961+0000) 2022-09-08T08:51:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:47.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:47.342528+0000) 2022-09-08T08:51:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:47 smithi059 ceph-mon[97450]: pgmap v21857: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:51:48.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:47 smithi055 ceph-mon[97033]: pgmap v21857: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:51:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:48.574+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:48.575189+0000) 2022-09-08T08:51:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:48.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:48.342794+0000) 2022-09-08T08:51:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:49.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:49.343053+0000) 2022-09-08T08:51:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:49.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:49.575448+0000) 2022-09-08T08:51:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:49 smithi059 ceph-mon[97450]: pgmap v21858: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:50.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:49 smithi055 ceph-mon[97033]: pgmap v21858: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:50.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:50.343206+0000) 2022-09-08T08:51:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:50.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:50.575706+0000) 2022-09-08T08:51:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:51.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:51.575976+0000) 2022-09-08T08:51:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:51.342+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:51.343416+0000) 2022-09-08T08:51:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:51 smithi059 ceph-mon[97450]: pgmap v21859: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:51:52.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:51 smithi055 ceph-mon[97033]: pgmap v21859: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:51:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:52.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:52.343662+0000) 2022-09-08T08:51:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:52.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:52.576215+0000) 2022-09-08T08:51:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:52 smithi059 ceph-mon[97450]: pgmap v21860: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:52 smithi055 ceph-mon[97033]: pgmap v21860: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:53.575+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:53.576451+0000) 2022-09-08T08:51:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:53.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:53.343830+0000) 2022-09-08T08:51:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:54.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:54.576655+0000) 2022-09-08T08:51:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:54.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:54.344062+0000) 2022-09-08T08:51:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:55 smithi055 ceph-mon[97033]: pgmap v21861: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:55 smithi059 ceph-mon[97450]: pgmap v21861: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:51:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:55.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:55.576798+0000) 2022-09-08T08:51:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:55.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:55.344297+0000) 2022-09-08T08:51:56.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:51:56.448+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:51:56.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:56.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:56.577051+0000) 2022-09-08T08:51:56.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:56.343+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:56.344459+0000) 2022-09-08T08:51:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:57.576+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:57.577196+0000) 2022-09-08T08:51:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:57.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:57.344677+0000) 2022-09-08T08:51:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:57 smithi059 ceph-mon[97450]: pgmap v21862: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:51:58.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:57 smithi055 ceph-mon[97033]: pgmap v21862: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:51:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:58.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:58.577402+0000) 2022-09-08T08:51:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:58.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:58.344887+0000) 2022-09-08T08:51:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:51:59.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:59.577676+0000) 2022-09-08T08:51:59.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:51:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:51:59.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:51:59.345070+0000) 2022-09-08T08:52:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:51:59 smithi059 ceph-mon[97450]: pgmap v21863: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:52:00.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:51:59 smithi055 ceph-mon[97033]: pgmap v21863: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:52:00.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:00.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:00.577875+0000) 2022-09-08T08:52:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:00.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:00.345201+0000) 2022-09-08T08:52:01.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:00 smithi055 ceph-mon[97033]: pgmap v21864: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:52:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:00 smithi059 ceph-mon[97450]: pgmap v21864: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:52:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:01.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:01.345326+0000) 2022-09-08T08:52:01.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:01.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:01.578119+0000) 2022-09-08T08:52:02.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:52:02.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:52:02.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:52:02.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:52:02.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:01 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:52:02.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:52:02.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:52:02.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:52:02.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:52:02.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:01 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:52:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:02.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:02.578325+0000) 2022-09-08T08:52:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:02.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:02.345533+0000) 2022-09-08T08:52:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:02 smithi055 ceph-mon[97033]: pgmap v21865: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:52:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:02 smithi059 ceph-mon[97450]: pgmap v21865: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:52:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:03.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:03.578466+0000) 2022-09-08T08:52:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:03.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:03.345682+0000) 2022-09-08T08:52:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:04.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:04.578681+0000) 2022-09-08T08:52:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:04.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:04.345833+0000) 2022-09-08T08:52:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:52:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:52:04.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:52:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:52:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:04.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:04.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:04.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:04.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:04.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:04.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:04.985 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:04 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:52:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:52:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:52:05.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:52:05.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:05.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:05.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:05.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:05.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:05.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:05.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:05.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:04 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T08:52:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:05.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:05.578913+0000) 2022-09-08T08:52:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:05.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:05.345957+0000) 2022-09-08T08:52:05.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:05 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 87473k 2022-09-08T08:52:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:05 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 89572937: error parsing value: Value '89572937' is below minimum 939524096 2022-09-08T08:52:05.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:05 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T08:52:05.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:05 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T08:52:05.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:05 smithi055 ceph-mon[97033]: pgmap v21866: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:52:06.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:05 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 87473k 2022-09-08T08:52:06.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:05 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 89572937: error parsing value: Value '89572937' is below minimum 939524096 2022-09-08T08:52:06.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:05 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T08:52:06.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:05 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T08:52:06.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:05 smithi059 ceph-mon[97450]: pgmap v21866: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:52:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:06.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:06.579212+0000) 2022-09-08T08:52:06.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:06.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:06.346116+0000) 2022-09-08T08:52:06.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:52:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:52:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:07.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:07.579360+0000) 2022-09-08T08:52:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:07.344+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:07.346303+0000) 2022-09-08T08:52:07.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:07 smithi055 ceph-mon[97033]: pgmap v21867: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:52:07.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:52:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:07 smithi059 ceph-mon[97450]: pgmap v21867: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:52:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:52:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:08.577+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:08.579556+0000) 2022-09-08T08:52:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:08.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:08.346547+0000) 2022-09-08T08:52:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:09.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:09.579795+0000) 2022-09-08T08:52:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:09.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:09.346781+0000) 2022-09-08T08:52:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:09 smithi059 ceph-mon[97450]: pgmap v21868: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:52:10.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:09 smithi055 ceph-mon[97033]: pgmap v21868: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:52:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:10.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:10.580033+0000) 2022-09-08T08:52:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:10.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:10.346919+0000) 2022-09-08T08:52:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:10 smithi059 ceph-mon[97450]: osdmap e1444: 12 total, 11 up, 11 in 2022-09-08T08:52:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:10 smithi059 ceph-mon[97450]: pgmap v21870: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:52:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:10 smithi055 ceph-mon[97033]: osdmap e1444: 12 total, 11 up, 11 in 2022-09-08T08:52:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:10 smithi055 ceph-mon[97033]: pgmap v21870: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:52:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:11.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:11.580254+0000) 2022-09-08T08:52:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:11.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:11.347125+0000) 2022-09-08T08:52:11.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:52:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:52:11.455+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:52:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:12.578+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:12.580479+0000) 2022-09-08T08:52:12.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:12.345+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:12.347355+0000) 2022-09-08T08:52:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:13.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:13.580740+0000) 2022-09-08T08:52:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:13.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:13.347500+0000) 2022-09-08T08:52:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:13 smithi059 ceph-mon[97450]: pgmap v21871: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:52:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:13 smithi055 ceph-mon[97033]: pgmap v21871: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:52:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:14.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:14.580890+0000) 2022-09-08T08:52:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:14.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:14.347693+0000) 2022-09-08T08:52:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:15 smithi055 ceph-mon[97033]: pgmap v21872: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:52:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:15 smithi059 ceph-mon[97450]: pgmap v21872: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:52:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:15.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:15.581016+0000) 2022-09-08T08:52:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:15.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:15.347839+0000) 2022-09-08T08:52:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:16.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:16.581267+0000) 2022-09-08T08:52:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:16.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:16.348043+0000) 2022-09-08T08:52:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:17.580+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:17.581432+0000) 2022-09-08T08:52:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:17.346+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:17.348216+0000) 2022-09-08T08:52:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:17 smithi059 ceph-mon[97450]: pgmap v21873: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:52:18.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:17 smithi055 ceph-mon[97033]: pgmap v21873: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:52:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:18.579+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:18.581593+0000) 2022-09-08T08:52:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:18.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:18.348387+0000) 2022-09-08T08:52:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:18 smithi059 ceph-mon[97450]: pgmap v21874: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:52:19.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:18 smithi055 ceph-mon[97033]: pgmap v21874: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:52:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:19.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:19.581790+0000) 2022-09-08T08:52:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:19.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:19.348532+0000) 2022-09-08T08:52:20.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:20 smithi055 ceph-mon[97033]: osdmap e1445: 12 total, 11 up, 11 in 2022-09-08T08:52:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:20 smithi059 ceph-mon[97450]: osdmap e1445: 12 total, 11 up, 11 in 2022-09-08T08:52:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:20.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:20.348688+0000) 2022-09-08T08:52:20.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:20.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:20.582014+0000) 2022-09-08T08:52:21.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:21 smithi055 ceph-mon[97033]: pgmap v21876: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:52:21.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:21 smithi059 ceph-mon[97450]: pgmap v21876: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:52:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:21.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:21.582263+0000) 2022-09-08T08:52:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:21.347+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:21.348820+0000) 2022-09-08T08:52:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:22.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:22.348971+0000) 2022-09-08T08:52:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:22.581+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:22.582466+0000) 2022-09-08T08:52:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:23.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:23.349101+0000) 2022-09-08T08:52:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:23.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:23.582696+0000) 2022-09-08T08:52:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:23 smithi059 ceph-mon[97450]: pgmap v21877: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:52:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:23 smithi055 ceph-mon[97033]: pgmap v21877: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:52:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:24.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:24.349316+0000) 2022-09-08T08:52:24.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:24.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:24.582871+0000) 2022-09-08T08:52:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:25 smithi055 ceph-mon[97033]: pgmap v21878: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:52:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:25 smithi059 ceph-mon[97450]: pgmap v21878: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:52:25.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:25.348+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:25.349544+0000) 2022-09-08T08:52:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:25.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:25.583059+0000) 2022-09-08T08:52:26.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:52:26.462+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:52:26.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:26.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:26.583288+0000) 2022-09-08T08:52:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:26.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:26.349744+0000) 2022-09-08T08:52:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:26 smithi055 ceph-mon[97033]: pgmap v21879: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:52:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:26 smithi059 ceph-mon[97450]: pgmap v21879: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:52:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:27.582+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:27.583462+0000) 2022-09-08T08:52:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:27.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:27.349931+0000) 2022-09-08T08:52:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:28.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:28.583687+0000) 2022-09-08T08:52:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:28.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:28.350159+0000) 2022-09-08T08:52:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:28 smithi055 ceph-mon[97033]: pgmap v21880: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:52:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:28 smithi059 ceph-mon[97450]: pgmap v21880: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:52:29.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:29.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:29.350344+0000) 2022-09-08T08:52:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:29.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:29.583808+0000) 2022-09-08T08:52:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:30.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:30.584023+0000) 2022-09-08T08:52:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:30.349+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:30.350504+0000) 2022-09-08T08:52:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:30 smithi055 ceph-mon[97033]: pgmap v21881: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T08:52:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:30 smithi059 ceph-mon[97450]: pgmap v21881: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T08:52:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:31.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:31.584260+0000) 2022-09-08T08:52:31.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:31.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:31.350743+0000) 2022-09-08T08:52:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:31 smithi055 ceph-mon[97033]: pgmap v21882: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:52:32.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:31 smithi055 ceph-mon[97033]: pgmap v21883: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:52:32.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:52:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:31 smithi059 ceph-mon[97450]: pgmap v21882: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:52:32.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:31 smithi059 ceph-mon[97450]: pgmap v21883: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:52:32.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:52:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:32.583+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:32.584503+0000) 2022-09-08T08:52:32.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:32.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:32.351002+0000) 2022-09-08T08:52:33.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:33.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:33.584757+0000) 2022-09-08T08:52:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:33.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:33.351191+0000) 2022-09-08T08:52:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:33 smithi055 ceph-mon[97033]: pgmap v21884: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:52:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:33 smithi059 ceph-mon[97450]: pgmap v21884: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:52:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:34.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:34.585006+0000) 2022-09-08T08:52:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:34.350+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:34.351401+0000) 2022-09-08T08:52:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:35.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:35.585238+0000) 2022-09-08T08:52:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:35.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:35.351602+0000) 2022-09-08T08:52:36.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:36 smithi055 ceph-mon[97033]: pgmap v21885: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-08T08:52:36.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:36 smithi059 ceph-mon[97450]: pgmap v21885: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 6 op/s 2022-09-08T08:52:36.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:36.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:36.351795+0000) 2022-09-08T08:52:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:36.584+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:36.585487+0000) 2022-09-08T08:52:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:37.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:37.585706+0000) 2022-09-08T08:52:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:37.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:37.351937+0000) 2022-09-08T08:52:38.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:38 smithi055 ceph-mon[97033]: pgmap v21886: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:52:38.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:38 smithi059 ceph-mon[97450]: pgmap v21886: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T08:52:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:38.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:38.585934+0000) 2022-09-08T08:52:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:38.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:38.352086+0000) 2022-09-08T08:52:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:39.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:39.586197+0000) 2022-09-08T08:52:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:39.351+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:39.352274+0000) 2022-09-08T08:52:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:40 smithi055 ceph-mon[97033]: pgmap v21887: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:52:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:40 smithi059 ceph-mon[97450]: pgmap v21887: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T08:52:40.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:40.585+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:40.586458+0000) 2022-09-08T08:52:40.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:40.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:40.352489+0000) 2022-09-08T08:52:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:41.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:41.586678+0000) 2022-09-08T08:52:41.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:41.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:41.352769+0000) 2022-09-08T08:52:41.656 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:52:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:52:41.476+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:52:42.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:42 smithi055 ceph-mon[97033]: pgmap v21888: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:52:42.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:42 smithi059 ceph-mon[97450]: pgmap v21888: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:52:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:42.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:42.586839+0000) 2022-09-08T08:52:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:42.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:42.352949+0000) 2022-09-08T08:52:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:43.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:43.587054+0000) 2022-09-08T08:52:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:43.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:43.353149+0000) 2022-09-08T08:52:44.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:44 smithi055 ceph-mon[97033]: pgmap v21889: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:52:44.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:44 smithi059 ceph-mon[97450]: pgmap v21889: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:52:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:44.586+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:44.587267+0000) 2022-09-08T08:52:44.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:44.352+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:44.353387+0000) 2022-09-08T08:52:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:45.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:45.587543+0000) 2022-09-08T08:52:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:45.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:45.353562+0000) 2022-09-08T08:52:46.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:46 smithi055 ceph-mon[97033]: pgmap v21890: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:52:46.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:46 smithi059 ceph-mon[97450]: pgmap v21890: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:52:46.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:46.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:46.587799+0000) 2022-09-08T08:52:46.666 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:46.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:46.353814+0000) 2022-09-08T08:52:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:47.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:47.588049+0000) 2022-09-08T08:52:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:47.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:47.353970+0000) 2022-09-08T08:52:48.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:48 smithi055 ceph-mon[97033]: pgmap v21891: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:52:48.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:48 smithi059 ceph-mon[97450]: pgmap v21891: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:52:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:48.587+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:48.588280+0000) 2022-09-08T08:52:48.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:48.353+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:48.354203+0000) 2022-09-08T08:52:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:49.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:49.588506+0000) 2022-09-08T08:52:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:49.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:49.354377+0000) 2022-09-08T08:52:50.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:50 smithi055 ceph-mon[97033]: pgmap v21892: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:52:50.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:50 smithi059 ceph-mon[97450]: pgmap v21892: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:52:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:50.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:50.588733+0000) 2022-09-08T08:52:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:50.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:50.354585+0000) 2022-09-08T08:52:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:51.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:51.588968+0000) 2022-09-08T08:52:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:51.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:51.354822+0000) 2022-09-08T08:52:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:52 smithi055 ceph-mon[97033]: pgmap v21893: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:52:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:52 smithi059 ceph-mon[97450]: pgmap v21893: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:52:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:52.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:52.589139+0000) 2022-09-08T08:52:52.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:52.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:52.355023+0000) 2022-09-08T08:52:53.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:53.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:53.589298+0000) 2022-09-08T08:52:53.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:53.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:53.355185+0000) 2022-09-08T08:52:54.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:54 smithi055 ceph-mon[97033]: pgmap v21894: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:52:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:54 smithi059 ceph-mon[97450]: pgmap v21894: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:52:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:54.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:54.589481+0000) 2022-09-08T08:52:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:54.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:54.355354+0000) 2022-09-08T08:52:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:55.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:55.355509+0000) 2022-09-08T08:52:55.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:55.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:55.589737+0000) 2022-09-08T08:52:56.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:56 smithi055 ceph-mon[97033]: pgmap v21895: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:52:56.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:56 smithi059 ceph-mon[97450]: pgmap v21895: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:52:56.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:52:56.478+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:52:56.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:56.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:56.355672+0000) 2022-09-08T08:52:56.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:56.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:56.589972+0000) 2022-09-08T08:52:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:57.588+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:57.590188+0000) 2022-09-08T08:52:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:57.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:57.355812+0000) 2022-09-08T08:52:58.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:52:58 smithi055 ceph-mon[97033]: pgmap v21896: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:52:58.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:52:58 smithi059 ceph-mon[97450]: pgmap v21896: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:52:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:58.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:58.590428+0000) 2022-09-08T08:52:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:58.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:58.355977+0000) 2022-09-08T08:52:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:52:59.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:59.590671+0000) 2022-09-08T08:52:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:52:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:52:59.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:52:59.356110+0000) 2022-09-08T08:53:00.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:00 smithi055 ceph-mon[97033]: pgmap v21897: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:00.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:00 smithi059 ceph-mon[97450]: pgmap v21897: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:00.354+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:00.356252+0000) 2022-09-08T08:53:00.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:00.589+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:00.590824+0000) 2022-09-08T08:53:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:01.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:01.356434+0000) 2022-09-08T08:53:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:01.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:01.591048+0000) 2022-09-08T08:53:02.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:02 smithi055 ceph-mon[97033]: pgmap v21898: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:53:02.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:02 smithi059 ceph-mon[97450]: pgmap v21898: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:53:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:02.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:02.356630+0000) 2022-09-08T08:53:02.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:02.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:02.591270+0000) 2022-09-08T08:53:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:03.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:03.591540+0000) 2022-09-08T08:53:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:03.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:03.356832+0000) 2022-09-08T08:53:04.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:04 smithi055 ceph-mon[97033]: pgmap v21899: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:04.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:04 smithi059 ceph-mon[97450]: pgmap v21899: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:04.590+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:04.591731+0000) 2022-09-08T08:53:04.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:04.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:04.356990+0000) 2022-09-08T08:53:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:05.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:05.591934+0000) 2022-09-08T08:53:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:05.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:05.357121+0000) 2022-09-08T08:53:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:06 smithi055 ceph-mon[97033]: pgmap v21900: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T08:53:06.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:53:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:06 smithi059 ceph-mon[97450]: pgmap v21900: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T08:53:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:53:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:06.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:06.592178+0000) 2022-09-08T08:53:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:06.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:06.357295+0000) 2022-09-08T08:53:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:53:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:53:07.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:07.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:07.592409+0000) 2022-09-08T08:53:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:07.355+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:07.357478+0000) 2022-09-08T08:53:08.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:08 smithi055 ceph-mon[97033]: pgmap v21901: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T08:53:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:08 smithi059 ceph-mon[97450]: pgmap v21901: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 20 op/s 2022-09-08T08:53:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:08.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:08.592576+0000) 2022-09-08T08:53:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:08.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:08.357665+0000) 2022-09-08T08:53:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:09.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:09.357828+0000) 2022-09-08T08:53:09.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:09.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:09.592744+0000) 2022-09-08T08:53:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:10 smithi055 ceph-mon[97033]: pgmap v21902: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:53:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:10 smithi055 ceph-mon[97033]: osdmap e1446: 12 total, 11 up, 11 in 2022-09-08T08:53:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:10 smithi059 ceph-mon[97450]: pgmap v21902: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:53:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:10 smithi059 ceph-mon[97450]: osdmap e1446: 12 total, 11 up, 11 in 2022-09-08T08:53:10.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:10.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:10.592976+0000) 2022-09-08T08:53:10.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:10.356+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:10.357977+0000) 2022-09-08T08:53:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:11.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:11.358104+0000) 2022-09-08T08:53:11.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:53:11.494+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:53:11.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:11.591+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:11.593107+0000) 2022-09-08T08:53:12.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:12 smithi055 ceph-mon[97033]: pgmap v21904: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:53:12.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:12 smithi059 ceph-mon[97450]: pgmap v21904: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:53:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:12.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:12.593338+0000) 2022-09-08T08:53:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:12.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:12.358238+0000) 2022-09-08T08:53:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:13.592+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:13.593565+0000) 2022-09-08T08:53:13.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:13.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:13.358398+0000) 2022-09-08T08:53:14.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:14 smithi055 ceph-mon[97033]: pgmap v21905: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:53:14.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:14 smithi055 ceph-mon[97033]: osdmap e1447: 12 total, 11 up, 11 in 2022-09-08T08:53:14.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:14 smithi059 ceph-mon[97450]: pgmap v21905: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T08:53:14.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:14 smithi059 ceph-mon[97450]: osdmap e1447: 12 total, 11 up, 11 in 2022-09-08T08:53:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:14.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:14.593697+0000) 2022-09-08T08:53:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:14.357+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:14.358595+0000) 2022-09-08T08:53:15.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:15.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:15.593894+0000) 2022-09-08T08:53:15.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:15.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:15.358715+0000) 2022-09-08T08:53:16.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:16 smithi055 ceph-mon[97033]: pgmap v21907: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:53:16.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:16 smithi059 ceph-mon[97450]: pgmap v21907: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:53:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:16.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:16.594115+0000) 2022-09-08T08:53:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:16.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:16.358838+0000) 2022-09-08T08:53:17.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:17.593+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:17.594362+0000) 2022-09-08T08:53:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:17.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:17.358986+0000) 2022-09-08T08:53:18.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:18 smithi055 ceph-mon[97033]: pgmap v21908: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:53:18.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:18 smithi059 ceph-mon[97450]: pgmap v21908: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:53:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:18.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:18.594617+0000) 2022-09-08T08:53:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:18.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:18.359126+0000) 2022-09-08T08:53:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:19.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:19.594782+0000) 2022-09-08T08:53:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:19.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:19.359318+0000) 2022-09-08T08:53:20.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:20 smithi055 ceph-mon[97033]: pgmap v21909: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2022-09-08T08:53:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:20 smithi059 ceph-mon[97450]: pgmap v21909: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2022-09-08T08:53:20.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:20.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:20.595013+0000) 2022-09-08T08:53:20.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:20.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:20.359468+0000) 2022-09-08T08:53:21.660 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:21.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:21.595261+0000) 2022-09-08T08:53:21.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:21.358+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:21.359600+0000) 2022-09-08T08:53:22.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:22 smithi055 ceph-mon[97033]: pgmap v21910: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:53:22.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:22 smithi059 ceph-mon[97450]: pgmap v21910: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:53:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:22.594+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:22.595499+0000) 2022-09-08T08:53:22.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:22.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:22.359783+0000) 2022-09-08T08:53:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:23.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:23.359932+0000) 2022-09-08T08:53:23.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:23.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:23.595734+0000) 2022-09-08T08:53:24.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:24 smithi055 ceph-mon[97033]: pgmap v21911: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:53:24.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:24 smithi059 ceph-mon[97450]: pgmap v21911: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:53:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:24.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:24.595952+0000) 2022-09-08T08:53:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:24.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:24.360066+0000) 2022-09-08T08:53:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:25.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:25.360235+0000) 2022-09-08T08:53:25.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:25.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:25.596141+0000) 2022-09-08T08:53:26.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:26 smithi055 ceph-mon[97033]: pgmap v21912: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.2 KiB/s rd, 94 B/s wr, 8 op/s 2022-09-08T08:53:26.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:26 smithi059 ceph-mon[97450]: pgmap v21912: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.2 KiB/s rd, 94 B/s wr, 8 op/s 2022-09-08T08:53:26.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:53:26.504+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:53:26.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:26.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:26.360389+0000) 2022-09-08T08:53:26.657 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:26.595+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:26.596269+0000) 2022-09-08T08:53:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:27.359+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:27.360506+0000) 2022-09-08T08:53:27.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:27.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:27.596477+0000) 2022-09-08T08:53:28.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:28 smithi055 ceph-mon[97033]: pgmap v21913: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:53:28.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:28 smithi059 ceph-mon[97450]: pgmap v21913: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:53:28.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:28.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:28.360645+0000) 2022-09-08T08:53:28.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:28.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:28.596701+0000) 2022-09-08T08:53:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:29.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:29.596894+0000) 2022-09-08T08:53:29.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:29.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:29.360815+0000) 2022-09-08T08:53:30.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:30 smithi055 ceph-mon[97033]: pgmap v21914: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:30 smithi059 ceph-mon[97450]: pgmap v21914: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:30.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:30.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:30.597105+0000) 2022-09-08T08:53:30.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:30.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:30.360956+0000) 2022-09-08T08:53:31.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:53:31.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:53:31.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:31 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:53:31.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:53:31.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:53:31.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:31 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:53:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:31.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:31.361085+0000) 2022-09-08T08:53:31.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:31.596+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:31.597289+0000) 2022-09-08T08:53:32.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:32.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:32.597544+0000) 2022-09-08T08:53:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:32.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:32.361251+0000) 2022-09-08T08:53:32.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:32 smithi055 ceph-mon[97033]: pgmap v21915: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:53:32.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:53:32.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:32 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:53:33.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:32 smithi059 ceph-mon[97450]: pgmap v21915: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:53:33.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:53:33.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:32 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:53:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:33.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:33.597819+0000) 2022-09-08T08:53:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:33.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:33.361399+0000) 2022-09-08T08:53:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:34.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:34.598083+0000) 2022-09-08T08:53:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:34.360+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:34.361551+0000) 2022-09-08T08:53:34.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:34 smithi055 ceph-mon[97033]: pgmap v21916: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:35.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:34 smithi059 ceph-mon[97450]: pgmap v21916: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:35.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:35.598357+0000) 2022-09-08T08:53:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:35.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:35.361724+0000) 2022-09-08T08:53:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:36.597+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:36.598555+0000) 2022-09-08T08:53:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:36.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:36.361857+0000) 2022-09-08T08:53:36.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:36 smithi055 ceph-mon[97033]: pgmap v21917: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:53:37.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:36 smithi059 ceph-mon[97450]: pgmap v21917: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:53:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:37.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:37.362000+0000) 2022-09-08T08:53:37.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:37.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:37.598771+0000) 2022-09-08T08:53:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:38.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:38.598955+0000) 2022-09-08T08:53:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:38.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:38.362172+0000) 2022-09-08T08:53:38.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:38 smithi055 ceph-mon[97033]: pgmap v21918: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:53:39.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:38 smithi059 ceph-mon[97450]: pgmap v21918: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:53:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:39.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:39.599197+0000) 2022-09-08T08:53:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:39.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:39.362323+0000) 2022-09-08T08:53:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:40.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:40.599351+0000) 2022-09-08T08:53:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:40.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:40.362475+0000) 2022-09-08T08:53:40.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:40 smithi055 ceph-mon[97033]: pgmap v21919: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:41.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:40 smithi059 ceph-mon[97450]: pgmap v21919: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:41.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:41.599590+0000) 2022-09-08T08:53:41.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:53:41.520+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:53:41.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:41.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:41.362654+0000) 2022-09-08T08:53:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:42.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:42.599785+0000) 2022-09-08T08:53:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:42.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:42.362813+0000) 2022-09-08T08:53:42.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:42 smithi055 ceph-mon[97033]: pgmap v21920: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:53:43.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:42 smithi059 ceph-mon[97450]: pgmap v21920: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:53:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:43.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:43.600029+0000) 2022-09-08T08:53:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:43.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:43.362932+0000) 2022-09-08T08:53:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:44.598+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:44.600230+0000) 2022-09-08T08:53:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:44.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:44.363078+0000) 2022-09-08T08:53:44.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:44 smithi055 ceph-mon[97033]: pgmap v21921: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:45.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:44 smithi059 ceph-mon[97450]: pgmap v21921: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:45.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:45.600444+0000) 2022-09-08T08:53:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:45.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:45.363246+0000) 2022-09-08T08:53:46.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:46.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:46.600715+0000) 2022-09-08T08:53:46.661 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:46.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:46.363399+0000) 2022-09-08T08:53:46.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:46 smithi055 ceph-mon[97033]: pgmap v21922: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:53:47.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:46 smithi059 ceph-mon[97450]: pgmap v21922: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:53:47.656 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:47.599+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:47.600979+0000) 2022-09-08T08:53:47.657 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:47.361+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:47.363554+0000) 2022-09-08T08:53:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:48.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:48.601197+0000) 2022-09-08T08:53:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:48.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:48.363794+0000) 2022-09-08T08:53:48.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:48 smithi055 ceph-mon[97033]: pgmap v21923: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:53:49.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:48 smithi059 ceph-mon[97450]: pgmap v21923: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:53:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:49.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:49.601392+0000) 2022-09-08T08:53:49.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:49.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:49.364035+0000) 2022-09-08T08:53:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:50.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:50.601608+0000) 2022-09-08T08:53:50.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:50.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:50.364197+0000) 2022-09-08T08:53:50.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:50 smithi055 ceph-mon[97033]: pgmap v21924: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:51.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:50 smithi059 ceph-mon[97450]: pgmap v21924: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:51.658 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:51.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:51.601829+0000) 2022-09-08T08:53:51.658 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:51.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:51.364401+0000) 2022-09-08T08:53:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:52.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:52.602044+0000) 2022-09-08T08:53:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:52.362+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:52.364573+0000) 2022-09-08T08:53:52.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:52 smithi055 ceph-mon[97033]: pgmap v21925: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:53:53.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:52 smithi059 ceph-mon[97450]: pgmap v21925: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:53:53.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:53.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:53.602276+0000) 2022-09-08T08:53:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:53.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:53.364747+0000) 2022-09-08T08:53:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:54.600+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:54.602461+0000) 2022-09-08T08:53:54.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:54.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:54.364890+0000) 2022-09-08T08:53:54.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:54 smithi055 ceph-mon[97033]: pgmap v21926: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:55.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:54 smithi059 ceph-mon[97450]: pgmap v21926: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:53:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:55.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:55.602701+0000) 2022-09-08T08:53:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:55.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:55.365026+0000) 2022-09-08T08:53:56.529 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:56.363+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:56.365181+0000) 2022-09-08T08:53:56.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:53:56.525+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:53:56.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:56.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:56.602936+0000) 2022-09-08T08:53:56.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:56 smithi055 ceph-mon[97033]: pgmap v21927: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:53:57.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:56 smithi059 ceph-mon[97450]: pgmap v21927: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:53:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:57.601+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:57.603177+0000) 2022-09-08T08:53:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:57.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:57.365421+0000) 2022-09-08T08:53:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:58.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:58.603413+0000) 2022-09-08T08:53:58.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:58.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:58.365654+0000) 2022-09-08T08:53:58.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:53:58 smithi055 ceph-mon[97033]: pgmap v21928: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:53:59.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:53:58 smithi059 ceph-mon[97450]: pgmap v21928: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T08:53:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:53:59.602+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:59.603619+0000) 2022-09-08T08:53:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:53:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:53:59.364+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:53:59.365840+0000) 2022-09-08T08:54:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:00.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:00.603798+0000) 2022-09-08T08:54:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:00.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:00.366010+0000) 2022-09-08T08:54:00.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:00 smithi055 ceph-mon[97033]: pgmap v21929: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:01.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:00 smithi059 ceph-mon[97450]: pgmap v21929: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:01.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:01.604061+0000) 2022-09-08T08:54:01.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:01.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:01.366145+0000) 2022-09-08T08:54:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:02.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:02.604258+0000) 2022-09-08T08:54:02.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:02.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:02.366305+0000) 2022-09-08T08:54:02.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:02 smithi055 ceph-mon[97033]: pgmap v21930: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:54:03.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:02 smithi059 ceph-mon[97450]: pgmap v21930: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:54:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:03.603+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:03.604484+0000) 2022-09-08T08:54:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:03.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:03.366448+0000) 2022-09-08T08:54:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:04.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:04.604655+0000) 2022-09-08T08:54:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:04.365+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:04.366597+0000) 2022-09-08T08:54:04.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:04 smithi055 ceph-mon[97033]: pgmap v21931: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:05.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:04 smithi059 ceph-mon[97450]: pgmap v21931: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:05.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:05.604835+0000) 2022-09-08T08:54:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:05.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:05.366740+0000) 2022-09-08T08:54:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:06.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:06.366990+0000) 2022-09-08T08:54:06.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:06.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:06.605003+0000) 2022-09-08T08:54:07.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:06 smithi059 ceph-mon[97450]: pgmap v21932: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:54:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:54:07.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:54:07.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:06 smithi059 ceph-mon[97450]: pgmap v21933: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 49 KiB/s wr, 22 op/s 2022-09-08T08:54:07.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:06 smithi059 ceph-mon[97450]: pgmap v21934: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 15 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T08:54:07.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:54:07.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:54:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:06 smithi055 ceph-mon[97033]: pgmap v21932: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:54:07.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:54:07.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:54:07.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:06 smithi055 ceph-mon[97033]: pgmap v21933: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 49 KiB/s wr, 22 op/s 2022-09-08T08:54:07.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:06 smithi055 ceph-mon[97033]: pgmap v21934: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 15 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T08:54:07.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:54:07.233 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:54:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:07.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:07.605179+0000) 2022-09-08T08:54:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:07.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:07.367147+0000) 2022-09-08T08:54:08.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:54:08.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:54:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:08.604+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:08.605395+0000) 2022-09-08T08:54:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:08.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:08.367292+0000) 2022-09-08T08:54:09.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:08 smithi055 ceph-mon[97033]: pgmap v21935: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 15 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T08:54:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:08 smithi059 ceph-mon[97450]: pgmap v21935: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 15 KiB/s rd, 61 KiB/s wr, 27 op/s 2022-09-08T08:54:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:09.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:09.605574+0000) 2022-09-08T08:54:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:09.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:09.367433+0000) 2022-09-08T08:54:10.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:10 smithi055 ceph-mon[97033]: pgmap v21936: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 61 KiB/s wr, 23 op/s 2022-09-08T08:54:10.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:10 smithi055 ceph-mon[97033]: osdmap e1448: 12 total, 11 up, 11 in 2022-09-08T08:54:10.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:10 smithi059 ceph-mon[97450]: pgmap v21936: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 61 KiB/s wr, 23 op/s 2022-09-08T08:54:10.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:10 smithi059 ceph-mon[97450]: osdmap e1448: 12 total, 11 up, 11 in 2022-09-08T08:54:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:10.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:10.605792+0000) 2022-09-08T08:54:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:10.366+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:10.367595+0000) 2022-09-08T08:54:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:11.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:11.606002+0000) 2022-09-08T08:54:11.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:54:11.541+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:54:11.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:11.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:11.367694+0000) 2022-09-08T08:54:12.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:11 smithi055 ceph-mon[97033]: pgmap v21938: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2022-09-08T08:54:12.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:11 smithi059 ceph-mon[97450]: pgmap v21938: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 151 B/s wr, 4 op/s 2022-09-08T08:54:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:12.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:12.606199+0000) 2022-09-08T08:54:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:12.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:12.367839+0000) 2022-09-08T08:54:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:13.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:13.606354+0000) 2022-09-08T08:54:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:13.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:13.368042+0000) 2022-09-08T08:54:14.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:13 smithi055 ceph-mon[97033]: pgmap v21939: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:54:14.287 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:13 smithi059 ceph-mon[97450]: pgmap v21939: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:54:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:14.605+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:14.606569+0000) 2022-09-08T08:54:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:14.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:14.368206+0000) 2022-09-08T08:54:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:15.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:15.606772+0000) 2022-09-08T08:54:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:15.367+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:15.368350+0000) 2022-09-08T08:54:16.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:15 smithi055 ceph-mon[97033]: pgmap v21940: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:54:16.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:15 smithi059 ceph-mon[97450]: pgmap v21940: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:54:16.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:16.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:16.606924+0000) 2022-09-08T08:54:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:16.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:16.368555+0000) 2022-09-08T08:54:17.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:17.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:17.607124+0000) 2022-09-08T08:54:17.656 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:17.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:17.368797+0000) 2022-09-08T08:54:18.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:17 smithi055 ceph-mon[97033]: pgmap v21941: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:54:18.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:17 smithi059 ceph-mon[97450]: pgmap v21941: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:54:18.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:18.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:18.607269+0000) 2022-09-08T08:54:18.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:18.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:18.369003+0000) 2022-09-08T08:54:19.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:19.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:19.607410+0000) 2022-09-08T08:54:19.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:19.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:19.369232+0000) 2022-09-08T08:54:20.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:20 smithi055 ceph-mon[97033]: osdmap e1449: 12 total, 11 up, 11 in 2022-09-08T08:54:20.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:20 smithi055 ceph-mon[97033]: pgmap v21943: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T08:54:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:20 smithi059 ceph-mon[97450]: osdmap e1449: 12 total, 11 up, 11 in 2022-09-08T08:54:20.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:20 smithi059 ceph-mon[97450]: pgmap v21943: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 205 B/s wr, 6 op/s 2022-09-08T08:54:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:20.606+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:20.607610+0000) 2022-09-08T08:54:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:20.368+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:20.369432+0000) 2022-09-08T08:54:21.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:21.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:21.607774+0000) 2022-09-08T08:54:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:21.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:21.369585+0000) 2022-09-08T08:54:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:21 smithi055 ceph-mon[97033]: pgmap v21944: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:54:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:21 smithi059 ceph-mon[97450]: pgmap v21944: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:54:22.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:22.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:22.608027+0000) 2022-09-08T08:54:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:22.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:22.369774+0000) 2022-09-08T08:54:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:23.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:23.608263+0000) 2022-09-08T08:54:23.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:23.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:23.369975+0000) 2022-09-08T08:54:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:23 smithi055 ceph-mon[97033]: pgmap v21945: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:54:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:23 smithi059 ceph-mon[97450]: pgmap v21945: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:54:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:24.607+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:24.608514+0000) 2022-09-08T08:54:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:24.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:24.370161+0000) 2022-09-08T08:54:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:25.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:25.608703+0000) 2022-09-08T08:54:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:25.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:25.370344+0000) 2022-09-08T08:54:26.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:25 smithi055 ceph-mon[97033]: pgmap v21946: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:54:26.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:25 smithi059 ceph-mon[97450]: pgmap v21946: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:54:26.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:54:26.542+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:54:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:26.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:26.608928+0000) 2022-09-08T08:54:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:26.369+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:26.370544+0000) 2022-09-08T08:54:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:27.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:27.609158+0000) 2022-09-08T08:54:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:27.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:27.370676+0000) 2022-09-08T08:54:28.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:27 smithi055 ceph-mon[97033]: pgmap v21947: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:54:28.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:27 smithi059 ceph-mon[97450]: pgmap v21947: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:54:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:28.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:28.370807+0000) 2022-09-08T08:54:28.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:28.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:28.609350+0000) 2022-09-08T08:54:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:29.608+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:29.609555+0000) 2022-09-08T08:54:29.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:29.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:29.370941+0000) 2022-09-08T08:54:30.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:30 smithi055 ceph-mon[97033]: pgmap v21948: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:54:30.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:30 smithi059 ceph-mon[97450]: pgmap v21948: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 5 op/s 2022-09-08T08:54:30.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:30.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:30.609779+0000) 2022-09-08T08:54:30.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:30.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:30.371147+0000) 2022-09-08T08:54:31.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:31.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:31.609964+0000) 2022-09-08T08:54:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:31.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:31.371356+0000) 2022-09-08T08:54:32.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:31 smithi055 ceph-mon[97033]: pgmap v21949: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:54:32.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:31 smithi059 ceph-mon[97450]: pgmap v21949: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:54:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:32.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:32.610216+0000) 2022-09-08T08:54:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:32.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:32.371534+0000) 2022-09-08T08:54:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:33.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:33.610379+0000) 2022-09-08T08:54:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:33.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:33.371771+0000) 2022-09-08T08:54:34.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:33 smithi055 ceph-mon[97033]: pgmap v21950: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:34.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:33 smithi059 ceph-mon[97450]: pgmap v21950: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:34.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:34.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:34.610608+0000) 2022-09-08T08:54:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:34.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:34.372012+0000) 2022-09-08T08:54:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:35.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:35.610824+0000) 2022-09-08T08:54:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:35.370+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:35.372157+0000) 2022-09-08T08:54:36.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:35 smithi055 ceph-mon[97033]: pgmap v21951: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:54:36.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:35 smithi059 ceph-mon[97450]: pgmap v21951: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:54:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:36.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:36.611010+0000) 2022-09-08T08:54:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:36.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:36.372406+0000) 2022-09-08T08:54:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:37.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:37.611184+0000) 2022-09-08T08:54:37.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:37.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:37.372687+0000) 2022-09-08T08:54:38.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:37 smithi055 ceph-mon[97033]: pgmap v21952: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:38.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:37 smithi059 ceph-mon[97450]: pgmap v21952: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:38.609+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:38.611426+0000) 2022-09-08T08:54:38.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:38.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:38.372949+0000) 2022-09-08T08:54:39.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:39.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:39.611774+0000) 2022-09-08T08:54:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:39.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:39.373092+0000) 2022-09-08T08:54:40.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:40 smithi055 ceph-mon[97033]: pgmap v21953: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:40.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:40 smithi059 ceph-mon[97450]: pgmap v21953: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:40.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:40.612046+0000) 2022-09-08T08:54:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:40.371+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:40.373296+0000) 2022-09-08T08:54:41.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:41.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:41.612231+0000) 2022-09-08T08:54:41.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:41.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:41.373485+0000) 2022-09-08T08:54:41.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:54:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:54:41.544+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:54:42.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:41 smithi055 ceph-mon[97033]: pgmap v21954: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:54:42.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:41 smithi059 ceph-mon[97450]: pgmap v21954: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:54:42.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:42.610+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:42.612453+0000) 2022-09-08T08:54:42.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:42.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:42.373682+0000) 2022-09-08T08:54:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:43.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:43.612671+0000) 2022-09-08T08:54:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:43.372+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:43.373871+0000) 2022-09-08T08:54:44.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:43 smithi055 ceph-mon[97033]: pgmap v21955: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:44.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:43 smithi059 ceph-mon[97450]: pgmap v21955: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:44.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:44.612803+0000) 2022-09-08T08:54:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:44.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:44.374134+0000) 2022-09-08T08:54:45.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:45.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:45.612932+0000) 2022-09-08T08:54:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:45.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:45.374402+0000) 2022-09-08T08:54:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:46.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:46.613082+0000) 2022-09-08T08:54:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:46.373+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:46.374684+0000) 2022-09-08T08:54:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:46 smithi055 ceph-mon[97033]: pgmap v21956: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:54:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:46 smithi059 ceph-mon[97450]: pgmap v21956: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:54:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:47.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:47.374895+0000) 2022-09-08T08:54:47.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:47.611+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:47.613231+0000) 2022-09-08T08:54:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:48.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:48.375053+0000) 2022-09-08T08:54:48.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:48.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:48.613441+0000) 2022-09-08T08:54:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:48 smithi055 ceph-mon[97033]: pgmap v21957: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:48 smithi059 ceph-mon[97450]: pgmap v21957: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:49.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:49.612+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:49.613656+0000) 2022-09-08T08:54:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:49.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:49.375195+0000) 2022-09-08T08:54:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:50.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:50.613797+0000) 2022-09-08T08:54:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:50.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:50.375404+0000) 2022-09-08T08:54:51.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:51 smithi055 ceph-mon[97033]: pgmap v21958: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:51.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:51 smithi059 ceph-mon[97450]: pgmap v21958: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:51.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:51.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:51.614022+0000) 2022-09-08T08:54:51.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:51.374+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:51.375600+0000) 2022-09-08T08:54:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:52.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:52.614237+0000) 2022-09-08T08:54:52.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:52.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:52.375870+0000) 2022-09-08T08:54:53.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:53 smithi055 ceph-mon[97033]: pgmap v21959: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:54:53.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:53 smithi059 ceph-mon[97450]: pgmap v21959: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:54:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:53.613+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:53.614470+0000) 2022-09-08T08:54:53.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:53.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:53.376048+0000) 2022-09-08T08:54:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:54.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:54.614708+0000) 2022-09-08T08:54:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:54.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:54.376292+0000) 2022-09-08T08:54:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:55 smithi055 ceph-mon[97033]: pgmap v21960: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:55 smithi059 ceph-mon[97450]: pgmap v21960: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:55.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:55.614837+0000) 2022-09-08T08:54:55.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:55.375+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:55.376556+0000) 2022-09-08T08:54:56.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:56.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:56.615042+0000) 2022-09-08T08:54:56.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:56.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:56.376793+0000) 2022-09-08T08:54:56.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:54:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:54:56.558+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:54:57.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:57 smithi055 ceph-mon[97033]: pgmap v21961: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:54:57.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:57 smithi059 ceph-mon[97450]: pgmap v21961: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:54:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:57.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:57.615282+0000) 2022-09-08T08:54:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:57.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:57.376933+0000) 2022-09-08T08:54:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:58.614+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:58.615523+0000) 2022-09-08T08:54:58.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:58.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:58.377106+0000) 2022-09-08T08:54:59.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:54:59 smithi055 ceph-mon[97033]: pgmap v21962: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:59.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:54:59 smithi059 ceph-mon[97450]: pgmap v21962: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:54:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:54:59.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:59.615779+0000) 2022-09-08T08:54:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:54:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:54:59.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:54:59.377218+0000) 2022-09-08T08:55:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:00.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:00.616048+0000) 2022-09-08T08:55:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:00.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:00.377367+0000) 2022-09-08T08:55:01.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:01 smithi055 ceph-mon[97033]: pgmap v21963: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:01.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:01 smithi059 ceph-mon[97450]: pgmap v21963: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:01.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:01.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:01.616297+0000) 2022-09-08T08:55:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:01.376+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:01.377563+0000) 2022-09-08T08:55:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:02.615+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:02.616511+0000) 2022-09-08T08:55:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:02.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:02.377734+0000) 2022-09-08T08:55:03.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:03 smithi055 ceph-mon[97033]: pgmap v21964: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:55:03.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:03 smithi059 ceph-mon[97450]: pgmap v21964: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:55:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:03.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:03.616659+0000) 2022-09-08T08:55:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:03.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:03.377973+0000) 2022-09-08T08:55:04.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:04.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:04.616893+0000) 2022-09-08T08:55:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:04.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:04.378132+0000) 2022-09-08T08:55:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:05 smithi055 ceph-mon[97033]: pgmap v21965: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:05 smithi059 ceph-mon[97450]: pgmap v21965: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:05.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:05.617086+0000) 2022-09-08T08:55:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:05.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:05.378338+0000) 2022-09-08T08:55:06.450 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:55:06.451 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:55:06.451 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:55:06.451 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:55:06.495 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:55:06.495 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:55:06.496 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:55:06.496 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:55:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:06.377+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:06.378564+0000) 2022-09-08T08:55:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:06.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:06.617325+0000) 2022-09-08T08:55:07.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:07.616+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:07.617529+0000) 2022-09-08T08:55:07.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:07.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:07.378779+0000) 2022-09-08T08:55:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:07 smithi055 ceph-mon[97033]: pgmap v21966: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:55:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:07 smithi055 ceph-mon[97033]: pgmap v21967: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T08:55:07.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:55:07.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:55:07.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:07 smithi055 ceph-mon[97033]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T08:55:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:07 smithi059 ceph-mon[97450]: pgmap v21966: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:55:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:07 smithi059 ceph-mon[97450]: pgmap v21967: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T08:55:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:55:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:55:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:07 smithi059 ceph-mon[97450]: Health check cleared: CEPHADM_HOST_CHECK_FAILED (was: 1 hosts fail cephadm check) 2022-09-08T08:55:08.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:08.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:08.617771+0000) 2022-09-08T08:55:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:08.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:08.378930+0000) 2022-09-08T08:55:09.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:09.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:09.617900+0000) 2022-09-08T08:55:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:09.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:09.379139+0000) 2022-09-08T08:55:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:09 smithi055 ceph-mon[97033]: pgmap v21968: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T08:55:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:09 smithi059 ceph-mon[97450]: pgmap v21968: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2022-09-08T08:55:10.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:10.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:10.618051+0000) 2022-09-08T08:55:10.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:10.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:10.379344+0000) 2022-09-08T08:55:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:10 smithi055 ceph-mon[97033]: osdmap e1450: 12 total, 11 up, 11 in 2022-09-08T08:55:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:10 smithi059 ceph-mon[97450]: osdmap e1450: 12 total, 11 up, 11 in 2022-09-08T08:55:11.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:11.617+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:11.618197+0000) 2022-09-08T08:55:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:11.378+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:11.379586+0000) 2022-09-08T08:55:11.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:55:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:55:11.565+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:55:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:11 smithi055 ceph-mon[97033]: pgmap v21970: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2022-09-08T08:55:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:11 smithi059 ceph-mon[97450]: pgmap v21970: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2022-09-08T08:55:12.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:12.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:12.379761+0000) 2022-09-08T08:55:12.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:12.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:12.618355+0000) 2022-09-08T08:55:13.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:13.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:13.618551+0000) 2022-09-08T08:55:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:13.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:13.380007+0000) 2022-09-08T08:55:13.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:13 smithi055 ceph-mon[97033]: pgmap v21971: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2022-09-08T08:55:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:13 smithi059 ceph-mon[97450]: pgmap v21971: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2022-09-08T08:55:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:14.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:14.380285+0000) 2022-09-08T08:55:14.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:14.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:14.618722+0000) 2022-09-08T08:55:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:15 smithi055 ceph-mon[97033]: osdmap e1451: 12 total, 11 up, 11 in 2022-09-08T08:55:15.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:15 smithi055 ceph-mon[97033]: pgmap v21973: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:55:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:15 smithi059 ceph-mon[97450]: osdmap e1451: 12 total, 11 up, 11 in 2022-09-08T08:55:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:15 smithi059 ceph-mon[97450]: pgmap v21973: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T08:55:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:15.379+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:15.380454+0000) 2022-09-08T08:55:15.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:15.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:15.618943+0000) 2022-09-08T08:55:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:16.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:16.619131+0000) 2022-09-08T08:55:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:16.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:16.380656+0000) 2022-09-08T08:55:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:17.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:17.619288+0000) 2022-09-08T08:55:17.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:17.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:17.380823+0000) 2022-09-08T08:55:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:17 smithi059 ceph-mon[97450]: pgmap v21974: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:55:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:17 smithi055 ceph-mon[97033]: pgmap v21974: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T08:55:18.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:18.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:18.381072+0000) 2022-09-08T08:55:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:18.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:18.619507+0000) 2022-09-08T08:55:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:19.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:19.619747+0000) 2022-09-08T08:55:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:19.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:19.381294+0000) 2022-09-08T08:55:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:19 smithi059 ceph-mon[97450]: pgmap v21975: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T08:55:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:19 smithi055 ceph-mon[97033]: pgmap v21975: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 227 B/s wr, 7 op/s 2022-09-08T08:55:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:20.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:20.619956+0000) 2022-09-08T08:55:20.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:20.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:20.381431+0000) 2022-09-08T08:55:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:21.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:21.620146+0000) 2022-09-08T08:55:21.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:21.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:21.381702+0000) 2022-09-08T08:55:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:21 smithi059 ceph-mon[97450]: pgmap v21976: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:55:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:21 smithi055 ceph-mon[97033]: pgmap v21976: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:55:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:22.618+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:22.620398+0000) 2022-09-08T08:55:22.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:22.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:22.381915+0000) 2022-09-08T08:55:23.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:23.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:23.382073+0000) 2022-09-08T08:55:23.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:23.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:23.620617+0000) 2022-09-08T08:55:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:23 smithi059 ceph-mon[97450]: pgmap v21977: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:55:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:23 smithi055 ceph-mon[97033]: pgmap v21977: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:55:24.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:24.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:24.620823+0000) 2022-09-08T08:55:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:24.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:24.382244+0000) 2022-09-08T08:55:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:25 smithi055 ceph-mon[97033]: pgmap v21978: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-08T08:55:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:25 smithi059 ceph-mon[97450]: pgmap v21978: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 99 B/s wr, 6 op/s 2022-09-08T08:55:25.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:25.619+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:25.621032+0000) 2022-09-08T08:55:25.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:25.380+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:25.382396+0000) 2022-09-08T08:55:26.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:55:26.565+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:55:26.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:26.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:26.621193+0000) 2022-09-08T08:55:26.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:26.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:26.382623+0000) 2022-09-08T08:55:27.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:27.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:27.382779+0000) 2022-09-08T08:55:27.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:27.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:27.621466+0000) 2022-09-08T08:55:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:27 smithi059 ceph-mon[97450]: pgmap v21979: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T08:55:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:27 smithi055 ceph-mon[97033]: pgmap v21979: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2022-09-08T08:55:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:28.620+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:28.621745+0000) 2022-09-08T08:55:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:28.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:28.382957+0000) 2022-09-08T08:55:29.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:29.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:29.621989+0000) 2022-09-08T08:55:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:29.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:29.383160+0000) 2022-09-08T08:55:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:29 smithi059 ceph-mon[97450]: pgmap v21980: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:29 smithi055 ceph-mon[97033]: pgmap v21980: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:30.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:30.622235+0000) 2022-09-08T08:55:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:30.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:30.383454+0000) 2022-09-08T08:55:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:31.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:31.622461+0000) 2022-09-08T08:55:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:31.381+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:31.383599+0000) 2022-09-08T08:55:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:31 smithi059 ceph-mon[97450]: pgmap v21981: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:55:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:31 smithi055 ceph-mon[97033]: pgmap v21981: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:55:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:32.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:32.622699+0000) 2022-09-08T08:55:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:32.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:32.383760+0000) 2022-09-08T08:55:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:33.621+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:33.622850+0000) 2022-09-08T08:55:33.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:33.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:33.383908+0000) 2022-09-08T08:55:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:33 smithi059 ceph-mon[97450]: pgmap v21982: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:33 smithi055 ceph-mon[97033]: pgmap v21982: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:34.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:34.623042+0000) 2022-09-08T08:55:34.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:34.382+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:34.384051+0000) 2022-09-08T08:55:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:35 smithi055 ceph-mon[97033]: pgmap v21983: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:35 smithi059 ceph-mon[97450]: pgmap v21983: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:35.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:35.623274+0000) 2022-09-08T08:55:35.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:35.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:35.384200+0000) 2022-09-08T08:55:36.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:36.622+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:36.623530+0000) 2022-09-08T08:55:36.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:36.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:36.384351+0000) 2022-09-08T08:55:37.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:37.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:37.623759+0000) 2022-09-08T08:55:37.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:37.383+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:37.384501+0000) 2022-09-08T08:55:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:37 smithi059 ceph-mon[97450]: pgmap v21984: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:55:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:37 smithi055 ceph-mon[97033]: pgmap v21984: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:55:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:38.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:38.623947+0000) 2022-09-08T08:55:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:38.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:38.384772+0000) 2022-09-08T08:55:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:39.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:39.624134+0000) 2022-09-08T08:55:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:39.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:39.384943+0000) 2022-09-08T08:55:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:39 smithi059 ceph-mon[97450]: pgmap v21985: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:39 smithi055 ceph-mon[97033]: pgmap v21985: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:40.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:40.623+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:40.624333+0000) 2022-09-08T08:55:40.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:40.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:40.385106+0000) 2022-09-08T08:55:41.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:55:41.577+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:55:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:41.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:41.624675+0000) 2022-09-08T08:55:41.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:41.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:41.385250+0000) 2022-09-08T08:55:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:41 smithi059 ceph-mon[97450]: pgmap v21986: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:55:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:41 smithi055 ceph-mon[97033]: pgmap v21986: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:55:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:42.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:42.624879+0000) 2022-09-08T08:55:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:42.384+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:42.385498+0000) 2022-09-08T08:55:43.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:43.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:43.625065+0000) 2022-09-08T08:55:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:43.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:43.385661+0000) 2022-09-08T08:55:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:43 smithi059 ceph-mon[97450]: pgmap v21987: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:43 smithi055 ceph-mon[97033]: pgmap v21987: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:44.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:44.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:44.625301+0000) 2022-09-08T08:55:44.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:44.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:44.385857+0000) 2022-09-08T08:55:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:45 smithi055 ceph-mon[97033]: pgmap v21988: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:45 smithi059 ceph-mon[97450]: pgmap v21988: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:45.624+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:45.625557+0000) 2022-09-08T08:55:45.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:45.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:45.386030+0000) 2022-09-08T08:55:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:46.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:46.625816+0000) 2022-09-08T08:55:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:46.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:46.386166+0000) 2022-09-08T08:55:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:47.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:47.626029+0000) 2022-09-08T08:55:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:47.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:47.386308+0000) 2022-09-08T08:55:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:47 smithi059 ceph-mon[97450]: pgmap v21989: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:55:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:47 smithi055 ceph-mon[97033]: pgmap v21989: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:55:48.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:48.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:48.626217+0000) 2022-09-08T08:55:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:48.385+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:48.386479+0000) 2022-09-08T08:55:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:49.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:49.386738+0000) 2022-09-08T08:55:49.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:49.625+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:49.626405+0000) 2022-09-08T08:55:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:49 smithi059 ceph-mon[97450]: pgmap v21990: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:49 smithi055 ceph-mon[97033]: pgmap v21990: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:50.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:50.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:50.626692+0000) 2022-09-08T08:55:50.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:50.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:50.387005+0000) 2022-09-08T08:55:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:51.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:51.626929+0000) 2022-09-08T08:55:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:51.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:51.387147+0000) 2022-09-08T08:55:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:51 smithi059 ceph-mon[97450]: pgmap v21991: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:55:51.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:51 smithi055 ceph-mon[97033]: pgmap v21991: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:55:52.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:52.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:52.627145+0000) 2022-09-08T08:55:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:52.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:52.387353+0000) 2022-09-08T08:55:53.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:53.386+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:53.387609+0000) 2022-09-08T08:55:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:53.626+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:53.627384+0000) 2022-09-08T08:55:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:53 smithi059 ceph-mon[97450]: pgmap v21992: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:53 smithi055 ceph-mon[97033]: pgmap v21992: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:54.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:54.627630+0000) 2022-09-08T08:55:54.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:54.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:54.387802+0000) 2022-09-08T08:55:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:55 smithi055 ceph-mon[97033]: pgmap v21993: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:55 smithi059 ceph-mon[97450]: pgmap v21993: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:55.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:55.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:55.627868+0000) 2022-09-08T08:55:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:55.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:55.387993+0000) 2022-09-08T08:55:56.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:55:56.581+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:55:56.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:56.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:56.628071+0000) 2022-09-08T08:55:56.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:56.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:56.388139+0000) 2022-09-08T08:55:57.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:57.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:57.388306+0000) 2022-09-08T08:55:57.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:57.627+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:57.628307+0000) 2022-09-08T08:55:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:57 smithi059 ceph-mon[97450]: pgmap v21994: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:55:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:57 smithi055 ceph-mon[97033]: pgmap v21994: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:55:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:58.387+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:58.388448+0000) 2022-09-08T08:55:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:58.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:58.628547+0000) 2022-09-08T08:55:59.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:55:59.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:59.628759+0000) 2022-09-08T08:55:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:55:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:55:59.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:55:59.388567+0000) 2022-09-08T08:55:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:55:59 smithi059 ceph-mon[97450]: pgmap v21995: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:55:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:55:59 smithi055 ceph-mon[97033]: pgmap v21995: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:00.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:00.628928+0000) 2022-09-08T08:56:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:00.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:00.388691+0000) 2022-09-08T08:56:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:01.628+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:01.629136+0000) 2022-09-08T08:56:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:01.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:01.388826+0000) 2022-09-08T08:56:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:01 smithi059 ceph-mon[97450]: pgmap v21996: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:56:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:01 smithi055 ceph-mon[97033]: pgmap v21996: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:56:02.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:02.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:02.629372+0000) 2022-09-08T08:56:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:02.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:02.388989+0000) 2022-09-08T08:56:03.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:03.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:03.629633+0000) 2022-09-08T08:56:03.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:03.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:03.389155+0000) 2022-09-08T08:56:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:03 smithi059 ceph-mon[97450]: pgmap v21997: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:03 smithi055 ceph-mon[97033]: pgmap v21997: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:04.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:04.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:04.389327+0000) 2022-09-08T08:56:04.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:04.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:04.629861+0000) 2022-09-08T08:56:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:05 smithi055 ceph-mon[97033]: pgmap v21998: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:05 smithi059 ceph-mon[97450]: pgmap v21998: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:05.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:05.630000+0000) 2022-09-08T08:56:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:05.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:05.389485+0000) 2022-09-08T08:56:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:56:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:56:06.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:06.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:06.630212+0000) 2022-09-08T08:56:06.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:06.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:06.389687+0000) 2022-09-08T08:56:07.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:07 smithi055 ceph-mon[97033]: pgmap v21999: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:56:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:56:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:56:07.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:56:07.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:56:07.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:56:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:07 smithi059 ceph-mon[97450]: pgmap v21999: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:56:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:56:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:56:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:56:07.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:56:07.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:56:07.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:07.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:07.630424+0000) 2022-09-08T08:56:07.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:07.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:07.389845+0000) 2022-09-08T08:56:08.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:08.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:08.630698+0000) 2022-09-08T08:56:08.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:08.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:08.390049+0000) 2022-09-08T08:56:09.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:09.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:09.630969+0000) 2022-09-08T08:56:09.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:09.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:09.390255+0000) 2022-09-08T08:56:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:09 smithi059 ceph-mon[97450]: pgmap v22000: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:56:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:09 smithi055 ceph-mon[97033]: pgmap v22000: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:56:10.659 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:10.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:10.631183+0000) 2022-09-08T08:56:10.660 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:10.388+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:10.390458+0000) 2022-09-08T08:56:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:10 smithi059 ceph-mon[97450]: osdmap e1452: 12 total, 11 up, 11 in 2022-09-08T08:56:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:10 smithi055 ceph-mon[97033]: osdmap e1452: 12 total, 11 up, 11 in 2022-09-08T08:56:11.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:11.629+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:11.631390+0000) 2022-09-08T08:56:11.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:11.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:11.390659+0000) 2022-09-08T08:56:11.655 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:56:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:56:11.595+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:56:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:11 smithi059 ceph-mon[97450]: pgmap v22002: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:56:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:11 smithi055 ceph-mon[97033]: pgmap v22002: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:56:12.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:12.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:12.631617+0000) 2022-09-08T08:56:12.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:12.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:12.390837+0000) 2022-09-08T08:56:13.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:13.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:13.631825+0000) 2022-09-08T08:56:13.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:13.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:13.391031+0000) 2022-09-08T08:56:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:13 smithi059 ceph-mon[97450]: pgmap v22003: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:56:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:13 smithi055 ceph-mon[97033]: pgmap v22003: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:56:14.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:14.630+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:14.631944+0000) 2022-09-08T08:56:14.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:14.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:14.391248+0000) 2022-09-08T08:56:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:15 smithi055 ceph-mon[97033]: pgmap v22004: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:56:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:15 smithi059 ceph-mon[97450]: pgmap v22004: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:56:15.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:15.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:15.632171+0000) 2022-09-08T08:56:15.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:15.389+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:15.391434+0000) 2022-09-08T08:56:16.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:16.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:16.632356+0000) 2022-09-08T08:56:16.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:16.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:16.391683+0000) 2022-09-08T08:56:17.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:17.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:17.632598+0000) 2022-09-08T08:56:17.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:17.390+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:17.391854+0000) 2022-09-08T08:56:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:17 smithi059 ceph-mon[97450]: pgmap v22005: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:56:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:17 smithi055 ceph-mon[97033]: pgmap v22005: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:56:18.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:18.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:18.632874+0000) 2022-09-08T08:56:18.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:18.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:18.392048+0000) 2022-09-08T08:56:19.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:19.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:19.633091+0000) 2022-09-08T08:56:19.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:19.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:19.392282+0000) 2022-09-08T08:56:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:19 smithi059 ceph-mon[97450]: pgmap v22006: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:56:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:19 smithi059 ceph-mon[97450]: osdmap e1453: 12 total, 11 up, 11 in 2022-09-08T08:56:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:19 smithi055 ceph-mon[97033]: pgmap v22006: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:56:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:19 smithi055 ceph-mon[97033]: osdmap e1453: 12 total, 11 up, 11 in 2022-09-08T08:56:20.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:20.632+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:20.633258+0000) 2022-09-08T08:56:20.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:20.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:20.392539+0000) 2022-09-08T08:56:21.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:21.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:21.392786+0000) 2022-09-08T08:56:21.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:21.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:21.633464+0000) 2022-09-08T08:56:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:21 smithi059 ceph-mon[97450]: pgmap v22008: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:56:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:21 smithi055 ceph-mon[97033]: pgmap v22008: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:56:22.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:22.631+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:22.633622+0000) 2022-09-08T08:56:22.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:22.391+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:22.392964+0000) 2022-09-08T08:56:23.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:23.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:23.633817+0000) 2022-09-08T08:56:23.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:23.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:23.393107+0000) 2022-09-08T08:56:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:23 smithi059 ceph-mon[97450]: pgmap v22009: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:56:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:23 smithi055 ceph-mon[97033]: pgmap v22009: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:56:24.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:24.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:24.634022+0000) 2022-09-08T08:56:24.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:24.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:24.393311+0000) 2022-09-08T08:56:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:25 smithi055 ceph-mon[97033]: pgmap v22010: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:56:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:25 smithi059 ceph-mon[97450]: pgmap v22010: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:56:25.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:25.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:25.634281+0000) 2022-09-08T08:56:25.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:25.392+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:25.393515+0000) 2022-09-08T08:56:26.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:26.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:26.393714+0000) 2022-09-08T08:56:26.654 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:56:26.602+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:56:26.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:26.633+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:26.634545+0000) 2022-09-08T08:56:27.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:27.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:27.634774+0000) 2022-09-08T08:56:27.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:27.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:27.393945+0000) 2022-09-08T08:56:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:27 smithi059 ceph-mon[97450]: pgmap v22011: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:56:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:27 smithi055 ceph-mon[97033]: pgmap v22011: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:56:28.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:28.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:28.635021+0000) 2022-09-08T08:56:28.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:28.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:28.394102+0000) 2022-09-08T08:56:29.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:29.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:29.635266+0000) 2022-09-08T08:56:29.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:29.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:29.394238+0000) 2022-09-08T08:56:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:29 smithi059 ceph-mon[97450]: pgmap v22012: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:56:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:29 smithi055 ceph-mon[97033]: pgmap v22012: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T08:56:30.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:30.634+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:30.635524+0000) 2022-09-08T08:56:30.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:30.393+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:30.394469+0000) 2022-09-08T08:56:31.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:31.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:31.635776+0000) 2022-09-08T08:56:31.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:31.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:31.394771+0000) 2022-09-08T08:56:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:31 smithi059 ceph-mon[97450]: pgmap v22013: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T08:56:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:31 smithi055 ceph-mon[97033]: pgmap v22013: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T08:56:32.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:32.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:32.636040+0000) 2022-09-08T08:56:32.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:32.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:32.394949+0000) 2022-09-08T08:56:33.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:33.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:33.636284+0000) 2022-09-08T08:56:33.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:33.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:33.395108+0000) 2022-09-08T08:56:33.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:33 smithi059 ceph-mon[97450]: pgmap v22014: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:33 smithi055 ceph-mon[97033]: pgmap v22014: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:34.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:34.635+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:34.636528+0000) 2022-09-08T08:56:34.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:34.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:34.395238+0000) 2022-09-08T08:56:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:35 smithi055 ceph-mon[97033]: pgmap v22015: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:35 smithi059 ceph-mon[97450]: pgmap v22015: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:35.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:35.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:35.636766+0000) 2022-09-08T08:56:35.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:35.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:35.395380+0000) 2022-09-08T08:56:36.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:36.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:36.636938+0000) 2022-09-08T08:56:36.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:36.394+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:36.395615+0000) 2022-09-08T08:56:37.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:37.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:37.395830+0000) 2022-09-08T08:56:37.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:37.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:37.637171+0000) 2022-09-08T08:56:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:37 smithi059 ceph-mon[97450]: pgmap v22016: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:56:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:37 smithi055 ceph-mon[97033]: pgmap v22016: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:56:38.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:38.636+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:38.637394+0000) 2022-09-08T08:56:38.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:38.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:38.396061+0000) 2022-09-08T08:56:39.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:39.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:39.637665+0000) 2022-09-08T08:56:39.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:39.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:39.396206+0000) 2022-09-08T08:56:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:39 smithi059 ceph-mon[97450]: pgmap v22017: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:39 smithi055 ceph-mon[97033]: pgmap v22017: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:40.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:40.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:40.637898+0000) 2022-09-08T08:56:40.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:40.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:40.396343+0000) 2022-09-08T08:56:41.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:56:41.617+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:56:41.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:41.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:41.638057+0000) 2022-09-08T08:56:41.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:41.395+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:41.396479+0000) 2022-09-08T08:56:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:41 smithi059 ceph-mon[97450]: pgmap v22018: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:56:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:41 smithi055 ceph-mon[97033]: pgmap v22018: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:56:42.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:42.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:42.638266+0000) 2022-09-08T08:56:42.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:42.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:42.396681+0000) 2022-09-08T08:56:43.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:43.637+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:43.638491+0000) 2022-09-08T08:56:43.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:43.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:43.396857+0000) 2022-09-08T08:56:43.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:43 smithi059 ceph-mon[97450]: pgmap v22019: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:43 smithi055 ceph-mon[97033]: pgmap v22019: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:44.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:44.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:44.638711+0000) 2022-09-08T08:56:44.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:44.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:44.397085+0000) 2022-09-08T08:56:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:45 smithi055 ceph-mon[97033]: pgmap v22020: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:45 smithi059 ceph-mon[97450]: pgmap v22020: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:45.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:45.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:45.638946+0000) 2022-09-08T08:56:45.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:45.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:45.397325+0000) 2022-09-08T08:56:46.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:46.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:46.639225+0000) 2022-09-08T08:56:46.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:46.396+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:46.397499+0000) 2022-09-08T08:56:47.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:47.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:47.639389+0000) 2022-09-08T08:56:47.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:47.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:47.397650+0000) 2022-09-08T08:56:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:47 smithi059 ceph-mon[97450]: pgmap v22021: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:56:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:47 smithi055 ceph-mon[97033]: pgmap v22021: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:56:48.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:48.638+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:48.639568+0000) 2022-09-08T08:56:48.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:48.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:48.397876+0000) 2022-09-08T08:56:49.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:49.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:49.639716+0000) 2022-09-08T08:56:49.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:49.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:49.398107+0000) 2022-09-08T08:56:49.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:49 smithi059 ceph-mon[97450]: pgmap v22022: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:49 smithi055 ceph-mon[97033]: pgmap v22022: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:50.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:50.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:50.639868+0000) 2022-09-08T08:56:50.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:50.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:50.398278+0000) 2022-09-08T08:56:51.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:51.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:51.640047+0000) 2022-09-08T08:56:51.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:51.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:51.398475+0000) 2022-09-08T08:56:51.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:51 smithi059 ceph-mon[97450]: pgmap v22023: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:56:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:51 smithi055 ceph-mon[97033]: pgmap v22023: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:56:52.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:52.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:52.398665+0000) 2022-09-08T08:56:52.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:52.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:52.640241+0000) 2022-09-08T08:56:53.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:53.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:53.640464+0000) 2022-09-08T08:56:53.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:53.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:53.398860+0000) 2022-09-08T08:56:53.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:53 smithi059 ceph-mon[97450]: pgmap v22024: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:53 smithi055 ceph-mon[97033]: pgmap v22024: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:54.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:54.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:54.640725+0000) 2022-09-08T08:56:54.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:54.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:54.399006+0000) 2022-09-08T08:56:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:55 smithi055 ceph-mon[97033]: pgmap v22025: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:55 smithi059 ceph-mon[97450]: pgmap v22025: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:55.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:55.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:55.640892+0000) 2022-09-08T08:56:55.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:55.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:55.399135+0000) 2022-09-08T08:56:56.653 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:56:56.624+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:56:56.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:56.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:56.641040+0000) 2022-09-08T08:56:56.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:56.397+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:56.399322+0000) 2022-09-08T08:56:57.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:57.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:57.641277+0000) 2022-09-08T08:56:57.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:57.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:57.399558+0000) 2022-09-08T08:56:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:57 smithi059 ceph-mon[97450]: pgmap v22026: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:56:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:57 smithi055 ceph-mon[97033]: pgmap v22026: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:56:58.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:58.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:58.399815+0000) 2022-09-08T08:56:58.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:58.639+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:58.641477+0000) 2022-09-08T08:56:59.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:56:59.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:59.399975+0000) 2022-09-08T08:56:59.655 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:56:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:56:59.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:56:59.641777+0000) 2022-09-08T08:56:59.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:56:59 smithi059 ceph-mon[97450]: pgmap v22027: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:56:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:56:59 smithi055 ceph-mon[97033]: pgmap v22027: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:00.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:00.640+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:00.641944+0000) 2022-09-08T08:57:00.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:00.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:00.400106+0000) 2022-09-08T08:57:01.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:01.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:01.642151+0000) 2022-09-08T08:57:01.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:01.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:01.400245+0000) 2022-09-08T08:57:01.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:01 smithi059 ceph-mon[97450]: pgmap v22028: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:57:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:01 smithi055 ceph-mon[97033]: pgmap v22028: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:57:02.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:02.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:02.642380+0000) 2022-09-08T08:57:02.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:02.398+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:02.400389+0000) 2022-09-08T08:57:03.653 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:03.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:03.642595+0000) 2022-09-08T08:57:03.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:03.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:03.400607+0000) 2022-09-08T08:57:03.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:03 smithi059 ceph-mon[97450]: pgmap v22029: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:03 smithi055 ceph-mon[97033]: pgmap v22029: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:04.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:04.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:04.642791+0000) 2022-09-08T08:57:04.655 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:04.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:04.400844+0000) 2022-09-08T08:57:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:05 smithi055 ceph-mon[97033]: pgmap v22030: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:05 smithi059 ceph-mon[97450]: pgmap v22030: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:05.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:05.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:05.642970+0000) 2022-09-08T08:57:05.654 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:05.399+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:05.401048+0000) 2022-09-08T08:57:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:57:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:57:06.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:06.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:06.401281+0000) 2022-09-08T08:57:06.654 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:06.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:06.643227+0000) 2022-09-08T08:57:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:07 smithi055 ceph-mon[97033]: pgmap v22031: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:57:07.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:57:07.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:57:07.482 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:57:07.483 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:57:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:07 smithi059 ceph-mon[97450]: pgmap v22031: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:57:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:57:07.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:57:07.534 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:57:07.535 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:57:07.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:07.641+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:07.643390+0000) 2022-09-08T08:57:07.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:07.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:07.401433+0000) 2022-09-08T08:57:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:08.642+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:08.643598+0000) 2022-09-08T08:57:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:08.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:08.401591+0000) 2022-09-08T08:57:08.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:57:09.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:57:09.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:09.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:09.643780+0000) 2022-09-08T08:57:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:09.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:09.401710+0000) 2022-09-08T08:57:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:09 smithi055 ceph-mon[97033]: pgmap v22032: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:57:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:09 smithi059 ceph-mon[97450]: pgmap v22032: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:57:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:10.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:10.644011+0000) 2022-09-08T08:57:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:10.400+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:10.401927+0000) 2022-09-08T08:57:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:10 smithi055 ceph-mon[97033]: osdmap e1454: 12 total, 11 up, 11 in 2022-09-08T08:57:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:10 smithi059 ceph-mon[97450]: osdmap e1454: 12 total, 11 up, 11 in 2022-09-08T08:57:11.904 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:57:11.632+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:57:11.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:11.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:11.644232+0000) 2022-09-08T08:57:11.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:11.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:11.402137+0000) 2022-09-08T08:57:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:11 smithi055 ceph-mon[97033]: pgmap v22034: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:57:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:11 smithi059 ceph-mon[97450]: pgmap v22034: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:57:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:12.643+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:12.644469+0000) 2022-09-08T08:57:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:12.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:12.402328+0000) 2022-09-08T08:57:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:13.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:13.644736+0000) 2022-09-08T08:57:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:13.401+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:13.402543+0000) 2022-09-08T08:57:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:13 smithi055 ceph-mon[97033]: pgmap v22035: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:57:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:13 smithi059 ceph-mon[97450]: pgmap v22035: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:57:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:14.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:14.644980+0000) 2022-09-08T08:57:14.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:14.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:14.402715+0000) 2022-09-08T08:57:15.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:15 smithi055 ceph-mon[97033]: pgmap v22036: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:57:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:15 smithi059 ceph-mon[97450]: pgmap v22036: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:57:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:15.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:15.645155+0000) 2022-09-08T08:57:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:15.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:15.402911+0000) 2022-09-08T08:57:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:16.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:16.645366+0000) 2022-09-08T08:57:16.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:16.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:16.403153+0000) 2022-09-08T08:57:17.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:17 smithi059 ceph-mon[97450]: pgmap v22037: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:57:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:17.644+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:17.645584+0000) 2022-09-08T08:57:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:17.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:17.403305+0000) 2022-09-08T08:57:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:17 smithi055 ceph-mon[97033]: pgmap v22037: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:57:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:18.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:18.645905+0000) 2022-09-08T08:57:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:18.402+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:18.403536+0000) 2022-09-08T08:57:19.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:19 smithi059 ceph-mon[97450]: pgmap v22038: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:57:19.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:19 smithi059 ceph-mon[97450]: osdmap e1455: 12 total, 11 up, 11 in 2022-09-08T08:57:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:19.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:19.646123+0000) 2022-09-08T08:57:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:19.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:19.403714+0000) 2022-09-08T08:57:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:19 smithi055 ceph-mon[97033]: pgmap v22038: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:57:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:19 smithi055 ceph-mon[97033]: osdmap e1455: 12 total, 11 up, 11 in 2022-09-08T08:57:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:20.645+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:20.646382+0000) 2022-09-08T08:57:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:20.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:20.403874+0000) 2022-09-08T08:57:21.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:21 smithi059 ceph-mon[97450]: pgmap v22040: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:57:21.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:21.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:21.646625+0000) 2022-09-08T08:57:21.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:21.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:21.404114+0000) 2022-09-08T08:57:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:21 smithi055 ceph-mon[97033]: pgmap v22040: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:57:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:22.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:22.646867+0000) 2022-09-08T08:57:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:22.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:22.404339+0000) 2022-09-08T08:57:23.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:23 smithi059 ceph-mon[97450]: pgmap v22041: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:57:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:23.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:23.647093+0000) 2022-09-08T08:57:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:23.403+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:23.404524+0000) 2022-09-08T08:57:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:23 smithi055 ceph-mon[97033]: pgmap v22041: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:57:24.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:24.646+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:24.647346+0000) 2022-09-08T08:57:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:24.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:24.404742+0000) 2022-09-08T08:57:25.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:25 smithi055 ceph-mon[97033]: pgmap v22042: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:57:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:25 smithi059 ceph-mon[97450]: pgmap v22042: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:57:25.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:25.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:25.647553+0000) 2022-09-08T08:57:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:25.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:25.404914+0000) 2022-09-08T08:57:26.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:26.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:26.647748+0000) 2022-09-08T08:57:26.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:26.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:26.405101+0000) 2022-09-08T08:57:26.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:57:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:57:26.640+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:57:27.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:27 smithi059 ceph-mon[97450]: pgmap v22043: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:57:27.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:27.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:27.405244+0000) 2022-09-08T08:57:27.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:27.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:27.647928+0000) 2022-09-08T08:57:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:27 smithi055 ceph-mon[97033]: pgmap v22043: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:57:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:28.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:28.648131+0000) 2022-09-08T08:57:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:28.404+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:28.405458+0000) 2022-09-08T08:57:29.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:29 smithi059 ceph-mon[97450]: pgmap v22044: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:57:29.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:29.647+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:29.648337+0000) 2022-09-08T08:57:29.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:29.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:29.405745+0000) 2022-09-08T08:57:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:29 smithi055 ceph-mon[97033]: pgmap v22044: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T08:57:30.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:30.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:30.648572+0000) 2022-09-08T08:57:30.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:30.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:30.405884+0000) 2022-09-08T08:57:31.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:31 smithi059 ceph-mon[97450]: pgmap v22045: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T08:57:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:31.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:31.648767+0000) 2022-09-08T08:57:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:31.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:31.406020+0000) 2022-09-08T08:57:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:31 smithi055 ceph-mon[97033]: pgmap v22045: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T08:57:32.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:32.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:32.648989+0000) 2022-09-08T08:57:32.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:32.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:32.406242+0000) 2022-09-08T08:57:33.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:33.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:33.649206+0000) 2022-09-08T08:57:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:33.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:33.406431+0000) 2022-09-08T08:57:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:33 smithi055 ceph-mon[97033]: pgmap v22046: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:33 smithi059 ceph-mon[97450]: pgmap v22046: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:34.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:34.649393+0000) 2022-09-08T08:57:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:34.405+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:34.406585+0000) 2022-09-08T08:57:35.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:35 smithi055 ceph-mon[97033]: pgmap v22047: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:35 smithi059 ceph-mon[97450]: pgmap v22047: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:35.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:35.648+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:35.649614+0000) 2022-09-08T08:57:35.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:35.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:35.406833+0000) 2022-09-08T08:57:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:36.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:36.649849+0000) 2022-09-08T08:57:36.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:36.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:36.407050+0000) 2022-09-08T08:57:37.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:37 smithi059 ceph-mon[97450]: pgmap v22048: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:57:37.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:37.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:37.650075+0000) 2022-09-08T08:57:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:37.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:37.407276+0000) 2022-09-08T08:57:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:37 smithi055 ceph-mon[97033]: pgmap v22048: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:57:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:38.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:38.650300+0000) 2022-09-08T08:57:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:38.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:38.407409+0000) 2022-09-08T08:57:39.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:39 smithi059 ceph-mon[97450]: pgmap v22049: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:39.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:39.649+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:39.650540+0000) 2022-09-08T08:57:39.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:39.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:39.407555+0000) 2022-09-08T08:57:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:39 smithi055 ceph-mon[97033]: pgmap v22049: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:40.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:40.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:40.650783+0000) 2022-09-08T08:57:40.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:40.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:40.407770+0000) 2022-09-08T08:57:41.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:41 smithi059 ceph-mon[97450]: pgmap v22050: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:57:41.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:41.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:41.650981+0000) 2022-09-08T08:57:41.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:57:41.655+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:57:41.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:41.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:41.407979+0000) 2022-09-08T08:57:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:41 smithi055 ceph-mon[97033]: pgmap v22050: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:57:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:42.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:42.408161+0000) 2022-09-08T08:57:42.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:42.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:42.651210+0000) 2022-09-08T08:57:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:43.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:43.651454+0000) 2022-09-08T08:57:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:43.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:43.408315+0000) 2022-09-08T08:57:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:43 smithi055 ceph-mon[97033]: pgmap v22051: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:43 smithi059 ceph-mon[97450]: pgmap v22051: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:44.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:44.651650+0000) 2022-09-08T08:57:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:44.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:44.408479+0000) 2022-09-08T08:57:45.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:45 smithi055 ceph-mon[97033]: pgmap v22052: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:45 smithi059 ceph-mon[97450]: pgmap v22052: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:45.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:45.651808+0000) 2022-09-08T08:57:45.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:45.406+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:45.408622+0000) 2022-09-08T08:57:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:46.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:46.651989+0000) 2022-09-08T08:57:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:46.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:46.408782+0000) 2022-09-08T08:57:47.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:47 smithi059 ceph-mon[97450]: pgmap v22053: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:57:47.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:47.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:47.652178+0000) 2022-09-08T08:57:47.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:47.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:47.408913+0000) 2022-09-08T08:57:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:47 smithi055 ceph-mon[97033]: pgmap v22053: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:57:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:48.650+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:48.652366+0000) 2022-09-08T08:57:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:48.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:48.409055+0000) 2022-09-08T08:57:49.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:49.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:49.652581+0000) 2022-09-08T08:57:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:49.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:49.409260+0000) 2022-09-08T08:57:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:49 smithi055 ceph-mon[97033]: pgmap v22054: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:49 smithi059 ceph-mon[97450]: pgmap v22054: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:50.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:50.652828+0000) 2022-09-08T08:57:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:50.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:50.409474+0000) 2022-09-08T08:57:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:51.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:51.652965+0000) 2022-09-08T08:57:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:51.407+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:51.409604+0000) 2022-09-08T08:57:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:51 smithi055 ceph-mon[97033]: pgmap v22055: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:57:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:51 smithi059 ceph-mon[97450]: pgmap v22055: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:57:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:52.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:52.653191+0000) 2022-09-08T08:57:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:52.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:52.409772+0000) 2022-09-08T08:57:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:53.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:53.653391+0000) 2022-09-08T08:57:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:53.408+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:53.409916+0000) 2022-09-08T08:57:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:53 smithi055 ceph-mon[97033]: pgmap v22056: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:53 smithi059 ceph-mon[97450]: pgmap v22056: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:54.651+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:54.653587+0000) 2022-09-08T08:57:54.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:54.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:54.410061+0000) 2022-09-08T08:57:55.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:55 smithi055 ceph-mon[97033]: pgmap v22057: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:55.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:55 smithi059 ceph-mon[97450]: pgmap v22057: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:57:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:55.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:55.653829+0000) 2022-09-08T08:57:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:55.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:55.410196+0000) 2022-09-08T08:57:56.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:56.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:56.654009+0000) 2022-09-08T08:57:56.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:56.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:56.410415+0000) 2022-09-08T08:57:56.905 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:57:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:57:56.657+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:57:57.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:57 smithi059 ceph-mon[97450]: pgmap v22058: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:57:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:57.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:57.654249+0000) 2022-09-08T08:57:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:57.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:57.410587+0000) 2022-09-08T08:57:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:57 smithi055 ceph-mon[97033]: pgmap v22058: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:57:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:58.653+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:58.654392+0000) 2022-09-08T08:57:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:58.409+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:58.410764+0000) 2022-09-08T08:57:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:57:59.652+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:59.654552+0000) 2022-09-08T08:57:59.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:57:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:57:59.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:57:59.410960+0000) 2022-09-08T08:57:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:57:59 smithi055 ceph-mon[97033]: pgmap v22059: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:57:59 smithi059 ceph-mon[97450]: pgmap v22059: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:00.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:00.654796+0000) 2022-09-08T08:58:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:00.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:00.411142+0000) 2022-09-08T08:58:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:01.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:01.655022+0000) 2022-09-08T08:58:01.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:01.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:01.411307+0000) 2022-09-08T08:58:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:01 smithi055 ceph-mon[97033]: pgmap v22060: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:58:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:01 smithi059 ceph-mon[97450]: pgmap v22060: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:58:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:02.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:02.655281+0000) 2022-09-08T08:58:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:02.410+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:02.411472+0000) 2022-09-08T08:58:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:03.654+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:03.655451+0000) 2022-09-08T08:58:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:03.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:03.411727+0000) 2022-09-08T08:58:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:03 smithi055 ceph-mon[97033]: pgmap v22061: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:03 smithi059 ceph-mon[97450]: pgmap v22061: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:04.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:04.655654+0000) 2022-09-08T08:58:04.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:04.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:04.411974+0000) 2022-09-08T08:58:05.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:05 smithi055 ceph-mon[97033]: pgmap v22062: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:05 smithi059 ceph-mon[97450]: pgmap v22062: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:05.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:05.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:05.655880+0000) 2022-09-08T08:58:05.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:05.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:05.412136+0000) 2022-09-08T08:58:06.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:58:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:58:06.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:06.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:06.656031+0000) 2022-09-08T08:58:06.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:06.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:06.412300+0000) 2022-09-08T08:58:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:07 smithi059 ceph-mon[97450]: pgmap v22063: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:58:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:58:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:07 smithi055 ceph-mon[97033]: pgmap v22063: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:58:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:58:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:07.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:07.656231+0000) 2022-09-08T08:58:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:07.411+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:07.412493+0000) 2022-09-08T08:58:08.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:58:08.560 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:58:08.561 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:58:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:58:08.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:58:08.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:58:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:08.655+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:08.656395+0000) 2022-09-08T08:58:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:08.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:08.412699+0000) 2022-09-08T08:58:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:58:09.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:09 smithi055 ceph-mon[97033]: pgmap v22064: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:58:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:58:09.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:09 smithi059 ceph-mon[97450]: pgmap v22064: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:58:09.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:09.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:09.656666+0000) 2022-09-08T08:58:09.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:09.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:09.412858+0000) 2022-09-08T08:58:10.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:10 smithi055 ceph-mon[97033]: osdmap e1456: 12 total, 11 up, 11 in 2022-09-08T08:58:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:10 smithi059 ceph-mon[97450]: osdmap e1456: 12 total, 11 up, 11 in 2022-09-08T08:58:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:10.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:10.656832+0000) 2022-09-08T08:58:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:10.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:10.413068+0000) 2022-09-08T08:58:11.674 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:11.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:11.657082+0000) 2022-09-08T08:58:11.675 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:11.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:11.413291+0000) 2022-09-08T08:58:11.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:11 smithi055 ceph-mon[97033]: pgmap v22066: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:58:11.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:11 smithi059 ceph-mon[97450]: pgmap v22066: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:58:12.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:58:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:58:11.671+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:58:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:12.656+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:12.657292+0000) 2022-09-08T08:58:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:12.412+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:12.413495+0000) 2022-09-08T08:58:13.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:13.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:13.657513+0000) 2022-09-08T08:58:13.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:13.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:13.413683+0000) 2022-09-08T08:58:13.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:13 smithi055 ceph-mon[97033]: pgmap v22067: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:58:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:13 smithi059 ceph-mon[97450]: pgmap v22067: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:58:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:14.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:14.657673+0000) 2022-09-08T08:58:14.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:14.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:14.413831+0000) 2022-09-08T08:58:15.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:15 smithi055 ceph-mon[97033]: pgmap v22068: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:58:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:15 smithi059 ceph-mon[97450]: pgmap v22068: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:58:15.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:15.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:15.657936+0000) 2022-09-08T08:58:15.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:15.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:15.413973+0000) 2022-09-08T08:58:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:16.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:16.658189+0000) 2022-09-08T08:58:16.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:16.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:16.414152+0000) 2022-09-08T08:58:17.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:17.657+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:17.658380+0000) 2022-09-08T08:58:17.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:17.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:17.414306+0000) 2022-09-08T08:58:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:17 smithi055 ceph-mon[97033]: pgmap v22069: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:58:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:17 smithi059 ceph-mon[97450]: pgmap v22069: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:58:18.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:18.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:18.658605+0000) 2022-09-08T08:58:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:18.413+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:18.414542+0000) 2022-09-08T08:58:19.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:19.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:19.658861+0000) 2022-09-08T08:58:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:19.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:19.414731+0000) 2022-09-08T08:58:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:19 smithi055 ceph-mon[97033]: pgmap v22070: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:58:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:19 smithi055 ceph-mon[97033]: osdmap e1457: 12 total, 11 up, 11 in 2022-09-08T08:58:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:19 smithi059 ceph-mon[97450]: pgmap v22070: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:58:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:19 smithi059 ceph-mon[97450]: osdmap e1457: 12 total, 11 up, 11 in 2022-09-08T08:58:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:20.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:20.659069+0000) 2022-09-08T08:58:20.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:20.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:20.414889+0000) 2022-09-08T08:58:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:21.658+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:21.659294+0000) 2022-09-08T08:58:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:21.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:21.415108+0000) 2022-09-08T08:58:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:21 smithi055 ceph-mon[97033]: pgmap v22072: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:58:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:21 smithi059 ceph-mon[97450]: pgmap v22072: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:58:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:22.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:22.659498+0000) 2022-09-08T08:58:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:22.414+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:22.415356+0000) 2022-09-08T08:58:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:23.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:23.659733+0000) 2022-09-08T08:58:23.906 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:23.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:23.415627+0000) 2022-09-08T08:58:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:23 smithi055 ceph-mon[97033]: pgmap v22073: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:58:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:23 smithi059 ceph-mon[97450]: pgmap v22073: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:58:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:24.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:24.659979+0000) 2022-09-08T08:58:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:24.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:24.415873+0000) 2022-09-08T08:58:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:25 smithi059 ceph-mon[97450]: pgmap v22074: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:58:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:25 smithi055 ceph-mon[97033]: pgmap v22074: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:58:25.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:25.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:25.660119+0000) 2022-09-08T08:58:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:25.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:25.416023+0000) 2022-09-08T08:58:26.678 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:26.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:26.660347+0000) 2022-09-08T08:58:26.679 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:26.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:26.416214+0000) 2022-09-08T08:58:27.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:58:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:58:26.675+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:58:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:27.659+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:27.660568+0000) 2022-09-08T08:58:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:27.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:27.416397+0000) 2022-09-08T08:58:27.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:27 smithi055 ceph-mon[97033]: pgmap v22075: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:58:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:27 smithi059 ceph-mon[97450]: pgmap v22075: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:58:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:28.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:28.660842+0000) 2022-09-08T08:58:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:28.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:28.416574+0000) 2022-09-08T08:58:29.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:29.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:29.661087+0000) 2022-09-08T08:58:29.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:29.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:29.416732+0000) 2022-09-08T08:58:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:29 smithi055 ceph-mon[97033]: pgmap v22076: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:58:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:29 smithi059 ceph-mon[97450]: pgmap v22076: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:58:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:30.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:30.661342+0000) 2022-09-08T08:58:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:30.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:30.416867+0000) 2022-09-08T08:58:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:31.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:31.661517+0000) 2022-09-08T08:58:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:31.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:31.417064+0000) 2022-09-08T08:58:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:31 smithi055 ceph-mon[97033]: pgmap v22077: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T08:58:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:31 smithi059 ceph-mon[97450]: pgmap v22077: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T08:58:32.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:32.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:32.661737+0000) 2022-09-08T08:58:32.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:32.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:32.417291+0000) 2022-09-08T08:58:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:33.660+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:33.661908+0000) 2022-09-08T08:58:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:33.415+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:33.417516+0000) 2022-09-08T08:58:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:33 smithi055 ceph-mon[97033]: pgmap v22078: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:33 smithi059 ceph-mon[97450]: pgmap v22078: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:34.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:34.662112+0000) 2022-09-08T08:58:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:34.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:34.417768+0000) 2022-09-08T08:58:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:35 smithi059 ceph-mon[97450]: pgmap v22079: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:35 smithi055 ceph-mon[97033]: pgmap v22079: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:35.909 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:35.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:35.662358+0000) 2022-09-08T08:58:35.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:35.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:35.417915+0000) 2022-09-08T08:58:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:36.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:36.662697+0000) 2022-09-08T08:58:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:36.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:36.418064+0000) 2022-09-08T08:58:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:37.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:37.662942+0000) 2022-09-08T08:58:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:37.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:37.418271+0000) 2022-09-08T08:58:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:37 smithi055 ceph-mon[97033]: pgmap v22080: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:58:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:37 smithi059 ceph-mon[97450]: pgmap v22080: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:58:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:38.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:38.663093+0000) 2022-09-08T08:58:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:38.416+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:38.418479+0000) 2022-09-08T08:58:39.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:39.661+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:39.663344+0000) 2022-09-08T08:58:39.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:39.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:39.418740+0000) 2022-09-08T08:58:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:39 smithi055 ceph-mon[97033]: pgmap v22081: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:39 smithi059 ceph-mon[97450]: pgmap v22081: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:40.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:40.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:40.663663+0000) 2022-09-08T08:58:40.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:40.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:40.418898+0000) 2022-09-08T08:58:41.683 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:41.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:41.663825+0000) 2022-09-08T08:58:41.684 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:41.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:41.419054+0000) 2022-09-08T08:58:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:41 smithi055 ceph-mon[97033]: pgmap v22082: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:58:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:41 smithi059 ceph-mon[97450]: pgmap v22082: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:58:42.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:58:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:58:41.679+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:58:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:42.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:42.664059+0000) 2022-09-08T08:58:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:42.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:42.419239+0000) 2022-09-08T08:58:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:43.663+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:43.664341+0000) 2022-09-08T08:58:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:43.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:43.419407+0000) 2022-09-08T08:58:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:43 smithi055 ceph-mon[97033]: pgmap v22083: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:43 smithi059 ceph-mon[97450]: pgmap v22083: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:44.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:44.662+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:44.664616+0000) 2022-09-08T08:58:44.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:44.417+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:44.419601+0000) 2022-09-08T08:58:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:45 smithi059 ceph-mon[97450]: pgmap v22084: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:45 smithi055 ceph-mon[97033]: pgmap v22084: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:45.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:45.664905+0000) 2022-09-08T08:58:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:45.418+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:45.419831+0000) 2022-09-08T08:58:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:46.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:46.665134+0000) 2022-09-08T08:58:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:46.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:46.420006+0000) 2022-09-08T08:58:47.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:47.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:47.665355+0000) 2022-09-08T08:58:47.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:47.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:47.420168+0000) 2022-09-08T08:58:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:47 smithi055 ceph-mon[97033]: pgmap v22085: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:58:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:47 smithi059 ceph-mon[97450]: pgmap v22085: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:58:48.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:48.664+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:48.665565+0000) 2022-09-08T08:58:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:48.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:48.420309+0000) 2022-09-08T08:58:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:49.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:49.665765+0000) 2022-09-08T08:58:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:49.419+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:49.420450+0000) 2022-09-08T08:58:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:49 smithi055 ceph-mon[97033]: pgmap v22086: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:49 smithi059 ceph-mon[97450]: pgmap v22086: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:50.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:50.665975+0000) 2022-09-08T08:58:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:50.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:50.420659+0000) 2022-09-08T08:58:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:51.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:51.666171+0000) 2022-09-08T08:58:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:51.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:51.420810+0000) 2022-09-08T08:58:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:51 smithi055 ceph-mon[97033]: pgmap v22087: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:58:52.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:51 smithi059 ceph-mon[97450]: pgmap v22087: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:58:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:52.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:52.666450+0000) 2022-09-08T08:58:52.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:52.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:52.421041+0000) 2022-09-08T08:58:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:53.665+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:53.666591+0000) 2022-09-08T08:58:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:53.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:53.421215+0000) 2022-09-08T08:58:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:53 smithi055 ceph-mon[97033]: pgmap v22088: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:53 smithi059 ceph-mon[97450]: pgmap v22088: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:54.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:54.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:54.666831+0000) 2022-09-08T08:58:54.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:54.420+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:54.421383+0000) 2022-09-08T08:58:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:55 smithi059 ceph-mon[97450]: pgmap v22089: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:55.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:55 smithi055 ceph-mon[97033]: pgmap v22089: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:58:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:55.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:55.667017+0000) 2022-09-08T08:58:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:55.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:55.421675+0000) 2022-09-08T08:58:56.690 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:56.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:56.667187+0000) 2022-09-08T08:58:56.690 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:56.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:56.421964+0000) 2022-09-08T08:58:57.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:58:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:58:56.687+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:58:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:57.666+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:57.667395+0000) 2022-09-08T08:58:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:57.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:57.422121+0000) 2022-09-08T08:58:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:57 smithi055 ceph-mon[97033]: pgmap v22090: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:58:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:57 smithi059 ceph-mon[97450]: pgmap v22090: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:58:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:58.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:58.667628+0000) 2022-09-08T08:58:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:58.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:58.422277+0000) 2022-09-08T08:58:59.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:58:59.421+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:59.422439+0000) 2022-09-08T08:58:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:58:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:58:59.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:58:59.667818+0000) 2022-09-08T08:58:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:58:59 smithi055 ceph-mon[97033]: pgmap v22091: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:58:59 smithi059 ceph-mon[97450]: pgmap v22091: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:00.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:00.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:00.422658+0000) 2022-09-08T08:59:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:00.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:00.668073+0000) 2022-09-08T08:59:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:01.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:01.668290+0000) 2022-09-08T08:59:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:01.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:01.422795+0000) 2022-09-08T08:59:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:01 smithi055 ceph-mon[97033]: pgmap v22092: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:59:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:01 smithi059 ceph-mon[97450]: pgmap v22092: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:59:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:02.667+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:02.668494+0000) 2022-09-08T08:59:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:02.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:02.422950+0000) 2022-09-08T08:59:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:03.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:03.668762+0000) 2022-09-08T08:59:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:03.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:03.423169+0000) 2022-09-08T08:59:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:03 smithi055 ceph-mon[97033]: pgmap v22093: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:03 smithi059 ceph-mon[97450]: pgmap v22093: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:04.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:04.668966+0000) 2022-09-08T08:59:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:04.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:04.423348+0000) 2022-09-08T08:59:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:05 smithi059 ceph-mon[97450]: pgmap v22094: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:05 smithi055 ceph-mon[97033]: pgmap v22094: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:05.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:05.669135+0000) 2022-09-08T08:59:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:05.422+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:05.423488+0000) 2022-09-08T08:59:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:59:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T08:59:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:06.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:06.669358+0000) 2022-09-08T08:59:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:06.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:06.423664+0000) 2022-09-08T08:59:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:07 smithi059 ceph-mon[97450]: pgmap v22095: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:59:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:59:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:07 smithi055 ceph-mon[97033]: pgmap v22095: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T08:59:07.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T08:59:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:07.668+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:07.669569+0000) 2022-09-08T08:59:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:07.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:07.423882+0000) 2022-09-08T08:59:08.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:59:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:59:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:08 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:59:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T08:59:08.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T08:59:08.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:08 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T08:59:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:08.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:08.669788+0000) 2022-09-08T08:59:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:08.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:08.424053+0000) 2022-09-08T08:59:09.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:09 smithi055 ceph-mon[97033]: pgmap v22096: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:59:09.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:59:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:09 smithi059 ceph-mon[97450]: pgmap v22096: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T08:59:09.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T08:59:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:09.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:09.670010+0000) 2022-09-08T08:59:09.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:09.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:09.424286+0000) 2022-09-08T08:59:10.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:10.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:10.670192+0000) 2022-09-08T08:59:10.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:10.423+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:10.424452+0000) 2022-09-08T08:59:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:10 smithi055 ceph-mon[97033]: osdmap e1458: 12 total, 11 up, 11 in 2022-09-08T08:59:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:10 smithi055 ceph-mon[97033]: pgmap v22098: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:59:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:10 smithi059 ceph-mon[97450]: osdmap e1458: 12 total, 11 up, 11 in 2022-09-08T08:59:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:10 smithi059 ceph-mon[97450]: pgmap v22098: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:59:11.697 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:11.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:11.670403+0000) 2022-09-08T08:59:11.697 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:11.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:11.424698+0000) 2022-09-08T08:59:12.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:59:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:59:11.694+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:59:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:12.669+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:12.670564+0000) 2022-09-08T08:59:12.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:12.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:12.424937+0000) 2022-09-08T08:59:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:13.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:13.670798+0000) 2022-09-08T08:59:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:13.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:13.425097+0000) 2022-09-08T08:59:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:13 smithi055 ceph-mon[97033]: pgmap v22099: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:59:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:13 smithi059 ceph-mon[97450]: pgmap v22099: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:59:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:14.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:14.670932+0000) 2022-09-08T08:59:14.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:14.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:14.425256+0000) 2022-09-08T08:59:15.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:15 smithi059 ceph-mon[97450]: pgmap v22100: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:59:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:15 smithi055 ceph-mon[97033]: pgmap v22100: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T08:59:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:15.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:15.671190+0000) 2022-09-08T08:59:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:15.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:15.425384+0000) 2022-09-08T08:59:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:16.670+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:16.671445+0000) 2022-09-08T08:59:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:16.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:16.425633+0000) 2022-09-08T08:59:17.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:17.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:17.671670+0000) 2022-09-08T08:59:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:17.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:17.425817+0000) 2022-09-08T08:59:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:17 smithi055 ceph-mon[97033]: pgmap v22101: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:59:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:17 smithi059 ceph-mon[97450]: pgmap v22101: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:59:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:18.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:18.671934+0000) 2022-09-08T08:59:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:18.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:18.426065+0000) 2022-09-08T08:59:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:19.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:19.672154+0000) 2022-09-08T08:59:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:19.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:19.426308+0000) 2022-09-08T08:59:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:19 smithi055 ceph-mon[97033]: pgmap v22102: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:59:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:19 smithi055 ceph-mon[97033]: osdmap e1459: 12 total, 11 up, 11 in 2022-09-08T08:59:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:19 smithi059 ceph-mon[97450]: pgmap v22102: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T08:59:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:19 smithi059 ceph-mon[97450]: osdmap e1459: 12 total, 11 up, 11 in 2022-09-08T08:59:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:20.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:20.672394+0000) 2022-09-08T08:59:20.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:20.424+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:20.426555+0000) 2022-09-08T08:59:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:21.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:21.672671+0000) 2022-09-08T08:59:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:21.425+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:21.426822+0000) 2022-09-08T08:59:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:21 smithi055 ceph-mon[97033]: pgmap v22104: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:59:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:21 smithi059 ceph-mon[97450]: pgmap v22104: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:59:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:22.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:22.672818+0000) 2022-09-08T08:59:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:22.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:22.427084+0000) 2022-09-08T08:59:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:23.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:23.672944+0000) 2022-09-08T08:59:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:23.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:23.427259+0000) 2022-09-08T08:59:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:23 smithi055 ceph-mon[97033]: pgmap v22105: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:59:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:23 smithi059 ceph-mon[97450]: pgmap v22105: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:59:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:24.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:24.673085+0000) 2022-09-08T08:59:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:24.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:24.427464+0000) 2022-09-08T08:59:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:25 smithi059 ceph-mon[97450]: pgmap v22106: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:59:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:25 smithi055 ceph-mon[97033]: pgmap v22106: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T08:59:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:25.671+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:25.673269+0000) 2022-09-08T08:59:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:25.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:25.427671+0000) 2022-09-08T08:59:26.703 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:26.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:26.673460+0000) 2022-09-08T08:59:26.704 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:26.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:26.427971+0000) 2022-09-08T08:59:27.157 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:59:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:59:26.699+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:59:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:27.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:27.673728+0000) 2022-09-08T08:59:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:27.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:27.428178+0000) 2022-09-08T08:59:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:27 smithi055 ceph-mon[97033]: pgmap v22107: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:59:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:27 smithi059 ceph-mon[97450]: pgmap v22107: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:59:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:28.672+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:28.673878+0000) 2022-09-08T08:59:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:28.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:28.428305+0000) 2022-09-08T08:59:29.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:29.426+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:29.428480+0000) 2022-09-08T08:59:29.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:29.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:29.674020+0000) 2022-09-08T08:59:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:29 smithi055 ceph-mon[97033]: pgmap v22108: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:59:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:29 smithi059 ceph-mon[97450]: pgmap v22108: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 6 op/s 2022-09-08T08:59:30.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:30.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:30.428666+0000) 2022-09-08T08:59:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:30.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:30.674210+0000) 2022-09-08T08:59:31.676 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:31.427+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:31.428849+0000) 2022-09-08T08:59:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:31 smithi055 ceph-mon[97033]: pgmap v22109: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T08:59:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:31 smithi059 ceph-mon[97450]: pgmap v22109: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-08T08:59:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:31.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:31.674386+0000) 2022-09-08T08:59:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:32.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:32.674600+0000) 2022-09-08T08:59:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:32.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:32.429018+0000) 2022-09-08T08:59:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:33.673+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:33.674808+0000) 2022-09-08T08:59:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:33.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:33.429171+0000) 2022-09-08T08:59:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:33 smithi055 ceph-mon[97033]: pgmap v22110: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:33 smithi059 ceph-mon[97450]: pgmap v22110: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:34.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:34.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:34.429429+0000) 2022-09-08T08:59:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:34.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:34.675039+0000) 2022-09-08T08:59:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:35 smithi059 ceph-mon[97450]: pgmap v22111: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:35 smithi055 ceph-mon[97033]: pgmap v22111: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:35.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:35.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:35.675260+0000) 2022-09-08T08:59:35.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:35.428+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:35.429586+0000) 2022-09-08T08:59:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:36.674+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:36.675509+0000) 2022-09-08T08:59:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:36.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:36.429782+0000) 2022-09-08T08:59:37.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:37.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:37.429989+0000) 2022-09-08T08:59:37.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:37.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:37.675742+0000) 2022-09-08T08:59:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:37 smithi055 ceph-mon[97033]: pgmap v22112: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:59:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:37 smithi059 ceph-mon[97450]: pgmap v22112: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:59:38.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:38.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:38.430205+0000) 2022-09-08T08:59:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:38.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:38.675948+0000) 2022-09-08T08:59:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:39.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:39.676142+0000) 2022-09-08T08:59:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:39.429+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:39.430373+0000) 2022-09-08T08:59:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:39 smithi055 ceph-mon[97033]: pgmap v22113: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:39 smithi059 ceph-mon[97450]: pgmap v22113: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:40.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:40.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:40.430615+0000) 2022-09-08T08:59:40.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:40.675+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:40.676363+0000) 2022-09-08T08:59:41.715 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:41.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:41.676620+0000) 2022-09-08T08:59:41.716 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:41.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:41.430870+0000) 2022-09-08T08:59:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:41 smithi055 ceph-mon[97033]: pgmap v22114: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:59:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:41 smithi059 ceph-mon[97450]: pgmap v22114: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:59:42.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:59:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:59:41.712+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:59:42.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:42.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:42.431035+0000) 2022-09-08T08:59:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:42.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:42.676910+0000) 2022-09-08T08:59:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:43.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:43.677123+0000) 2022-09-08T08:59:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:43.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:43.431152+0000) 2022-09-08T08:59:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:43 smithi055 ceph-mon[97033]: pgmap v22115: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:43 smithi059 ceph-mon[97450]: pgmap v22115: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:44.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:44.677363+0000) 2022-09-08T08:59:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:44.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:44.431295+0000) 2022-09-08T08:59:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:45 smithi059 ceph-mon[97450]: pgmap v22116: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:45 smithi055 ceph-mon[97033]: pgmap v22116: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:45.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:45.676+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:45.677605+0000) 2022-09-08T08:59:45.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:45.430+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:45.431516+0000) 2022-09-08T08:59:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:46.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:46.677885+0000) 2022-09-08T08:59:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:46.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:46.431781+0000) 2022-09-08T08:59:47.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:47.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:47.678022+0000) 2022-09-08T08:59:47.910 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:47.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:47.431993+0000) 2022-09-08T08:59:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:47 smithi055 ceph-mon[97033]: pgmap v22117: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:59:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:47 smithi059 ceph-mon[97450]: pgmap v22117: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:59:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:48.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:48.678258+0000) 2022-09-08T08:59:48.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:48.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:48.432153+0000) 2022-09-08T08:59:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:49.677+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:49.678476+0000) 2022-09-08T08:59:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:49.431+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:49.432376+0000) 2022-09-08T08:59:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:49 smithi055 ceph-mon[97033]: pgmap v22118: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:49 smithi059 ceph-mon[97450]: pgmap v22118: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:50.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:50.678750+0000) 2022-09-08T08:59:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:50.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:50.432557+0000) 2022-09-08T08:59:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:51.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:51.678909+0000) 2022-09-08T08:59:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:51.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:51.432800+0000) 2022-09-08T08:59:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:51 smithi055 ceph-mon[97033]: pgmap v22119: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:59:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:51 smithi059 ceph-mon[97450]: pgmap v22119: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:59:52.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:52.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:52.679214+0000) 2022-09-08T08:59:52.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:52.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:52.433023+0000) 2022-09-08T08:59:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:53.678+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:53.679423+0000) 2022-09-08T08:59:53.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:53.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:53.433247+0000) 2022-09-08T08:59:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:53 smithi055 ceph-mon[97033]: pgmap v22120: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:53 smithi059 ceph-mon[97450]: pgmap v22120: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:54.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:54.679625+0000) 2022-09-08T08:59:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:54.432+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:54.433520+0000) 2022-09-08T08:59:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:55 smithi059 ceph-mon[97450]: pgmap v22121: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:55 smithi055 ceph-mon[97033]: pgmap v22121: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T08:59:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:55.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:55.679854+0000) 2022-09-08T08:59:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:55.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:55.433690+0000) 2022-09-08T08:59:56.730 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:56.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:56.680005+0000) 2022-09-08T08:59:56.731 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:56.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:56.433828+0000) 2022-09-08T08:59:57.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 08:59:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T08:59:56.727+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T08:59:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:57.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:57.680196+0000) 2022-09-08T08:59:57.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:57.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:57.434032+0000) 2022-09-08T08:59:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:57 smithi055 ceph-mon[97033]: pgmap v22122: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:59:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:57 smithi059 ceph-mon[97450]: pgmap v22122: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T08:59:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:58.679+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:58.680426+0000) 2022-09-08T08:59:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:58.433+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:58.434268+0000) 2022-09-08T08:59:59.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 08:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T08:59:59.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:59.680662+0000) 2022-09-08T08:59:59.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 08:59:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T08:59:59.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T07:59:59.434525+0000) 2022-09-08T08:59:59.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 08:59:59 smithi055 ceph-mon[97033]: pgmap v22123: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 08:59:59 smithi059 ceph-mon[97450]: pgmap v22123: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:00.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:00.680818+0000) 2022-09-08T09:00:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:00.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:00.434773+0000) 2022-09-08T09:00:00.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:00 smithi055 ceph-mon[97033]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T09:00:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:00 smithi055 ceph-mon[97033]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T09:00:00.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:00 smithi055 ceph-mon[97033]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T09:00:01.039 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:00 smithi059 ceph-mon[97450]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-08T09:00:01.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:00 smithi059 ceph-mon[97450]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-08T09:00:01.040 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:00 smithi059 ceph-mon[97450]: mon.a (rank 0) addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] is down (out of quorum) 2022-09-08T09:00:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:01.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:01.681030+0000) 2022-09-08T09:00:01.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:01.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:01.435038+0000) 2022-09-08T09:00:01.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:01 smithi055 ceph-mon[97033]: pgmap v22124: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:00:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:01 smithi059 ceph-mon[97450]: pgmap v22124: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:00:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:02.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:02.681292+0000) 2022-09-08T09:00:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:02.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:02.435223+0000) 2022-09-08T09:00:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:03.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:03.681523+0000) 2022-09-08T09:00:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:03.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:03.435448+0000) 2022-09-08T09:00:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:03 smithi055 ceph-mon[97033]: pgmap v22125: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:03 smithi059 ceph-mon[97450]: pgmap v22125: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:04.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:04.681707+0000) 2022-09-08T09:00:04.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:04.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:04.435718+0000) 2022-09-08T09:00:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:05 smithi059 ceph-mon[97450]: pgmap v22126: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:05 smithi055 ceph-mon[97033]: pgmap v22126: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:05.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:05.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:05.681917+0000) 2022-09-08T09:00:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:05.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:05.435927+0000) 2022-09-08T09:00:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:00:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:00:06.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:06.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:06.682154+0000) 2022-09-08T09:00:06.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:06.434+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:06.436119+0000) 2022-09-08T09:00:07.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:07 smithi059 ceph-mon[97450]: pgmap v22127: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:00:07.533 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:00:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:07 smithi055 ceph-mon[97033]: pgmap v22127: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:00:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:00:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:07.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:07.436252+0000) 2022-09-08T09:00:07.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:07.680+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:07.682370+0000) 2022-09-08T09:00:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:08.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:08.682589+0000) 2022-09-08T09:00:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:08.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:08.436490+0000) 2022-09-08T09:00:09.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:09.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:09.682818+0000) 2022-09-08T09:00:09.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:09.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:09.436701+0000) 2022-09-08T09:00:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:09 smithi055 ceph-mon[97033]: pgmap v22128: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:00:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T09:00:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T09:00:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:09 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T09:00:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:09 smithi059 ceph-mon[97450]: pgmap v22128: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:00:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T09:00:10.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T09:00:10.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:09 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T09:00:10.898 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:10 smithi055 ceph-mon[97033]: osdmap e1460: 12 total, 11 up, 11 in 2022-09-08T09:00:10.899 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:10.899 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:10.900 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:10.900 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:10 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:10.681+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:10.682967+0000) 2022-09-08T09:00:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:10.435+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:10.436879+0000) 2022-09-08T09:00:10.948 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:10 smithi059 ceph-mon[97450]: osdmap e1460: 12 total, 11 up, 11 in 2022-09-08T09:00:10.949 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:10.949 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:10.950 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:10.951 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:10 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:11.738 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:11.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:11.437106+0000) 2022-09-08T09:00:11.739 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:11.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:11.683163+0000) 2022-09-08T09:00:11.857 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:11 smithi055 ceph-mon[97033]: pgmap v22130: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T09:00:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:11 smithi059 ceph-mon[97450]: pgmap v22130: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T09:00:12.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:00:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:00:11.734+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:00:12.914 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:12.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:12.437312+0000) 2022-09-08T09:00:12.915 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:12.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:12.683418+0000) 2022-09-08T09:00:13.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:13.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:13.683686+0000) 2022-09-08T09:00:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:13.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:13.437565+0000) 2022-09-08T09:00:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:13 smithi055 ceph-mon[97033]: pgmap v22131: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T09:00:13.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:13.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:13.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:13.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:13.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:13 smithi059 ceph-mon[97450]: pgmap v22131: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T09:00:14.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:14.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:14.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:14.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:14.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:00:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:14.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:14.683886+0000) 2022-09-08T09:00:14.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:14.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:14.437771+0000) 2022-09-08T09:00:14.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:14 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi059 2022-09-08T09:00:14.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:14 smithi055 ceph-mon[97033]: Detected new or changed devices on smithi055 2022-09-08T09:00:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:14 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi059 2022-09-08T09:00:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:14 smithi059 ceph-mon[97450]: Detected new or changed devices on smithi055 2022-09-08T09:00:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:15.682+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:15.684091+0000) 2022-09-08T09:00:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:15.436+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:15.437932+0000) 2022-09-08T09:00:15.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:15 smithi055 ceph-mon[97033]: pgmap v22132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T09:00:16.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:15 smithi059 ceph-mon[97450]: pgmap v22132: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T09:00:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:16.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:16.684345+0000) 2022-09-08T09:00:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:16.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:16.438149+0000) 2022-09-08T09:00:17.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:17.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:17.684570+0000) 2022-09-08T09:00:17.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:17.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:17.438391+0000) 2022-09-08T09:00:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:17 smithi055 ceph-mon[97033]: pgmap v22133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:00:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:17 smithi059 ceph-mon[97450]: pgmap v22133: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:00:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:18.683+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:18.684805+0000) 2022-09-08T09:00:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:18.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:18.438598+0000) 2022-09-08T09:00:19.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:19.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:19.685050+0000) 2022-09-08T09:00:19.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:19.437+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:19.438778+0000) 2022-09-08T09:00:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:19 smithi055 ceph-mon[97033]: pgmap v22134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:00:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:19 smithi055 ceph-mon[97033]: osdmap e1461: 12 total, 11 up, 11 in 2022-09-08T09:00:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:19 smithi059 ceph-mon[97450]: pgmap v22134: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:00:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:19 smithi059 ceph-mon[97450]: osdmap e1461: 12 total, 11 up, 11 in 2022-09-08T09:00:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:20.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:20.685222+0000) 2022-09-08T09:00:20.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:20.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:20.438929+0000) 2022-09-08T09:00:21.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:21.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:21.685372+0000) 2022-09-08T09:00:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:21.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:21.439141+0000) 2022-09-08T09:00:21.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:21 smithi055 ceph-mon[97033]: pgmap v22136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:00:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:21 smithi059 ceph-mon[97450]: pgmap v22136: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:00:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:22.684+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:22.685576+0000) 2022-09-08T09:00:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:22.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:22.439315+0000) 2022-09-08T09:00:23.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:23.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:23.685856+0000) 2022-09-08T09:00:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:23.438+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:23.439465+0000) 2022-09-08T09:00:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:23 smithi055 ceph-mon[97033]: pgmap v22137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:00:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:23 smithi059 ceph-mon[97450]: pgmap v22137: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:00:24.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:24.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:24.685986+0000) 2022-09-08T09:00:24.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:24.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:24.439730+0000) 2022-09-08T09:00:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:25 smithi059 ceph-mon[97450]: pgmap v22138: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:00:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:25 smithi055 ceph-mon[97033]: pgmap v22138: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:00:25.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:25.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:25.686175+0000) 2022-09-08T09:00:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:25.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:25.439909+0000) 2022-09-08T09:00:26.754 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:26.685+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:26.686399+0000) 2022-09-08T09:00:26.755 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:26.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:26.440055+0000) 2022-09-08T09:00:27.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:00:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:00:26.751+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:00:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:27.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:27.686654+0000) 2022-09-08T09:00:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:27.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:27.440254+0000) 2022-09-08T09:00:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:27 smithi055 ceph-mon[97033]: pgmap v22139: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:00:28.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:27 smithi059 ceph-mon[97450]: pgmap v22139: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:00:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:28.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:28.440377+0000) 2022-09-08T09:00:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:28.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:28.686852+0000) 2022-09-08T09:00:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:29.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:29.687012+0000) 2022-09-08T09:00:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:29.439+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:29.440607+0000) 2022-09-08T09:00:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:29 smithi055 ceph-mon[97033]: pgmap v22140: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:00:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:29 smithi059 ceph-mon[97450]: pgmap v22140: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:00:30.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:30.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:30.687169+0000) 2022-09-08T09:00:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:30.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:30.440782+0000) 2022-09-08T09:00:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:31.686+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:31.687365+0000) 2022-09-08T09:00:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:31.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:31.440957+0000) 2022-09-08T09:00:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:31 smithi055 ceph-mon[97033]: pgmap v22141: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T09:00:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:31 smithi059 ceph-mon[97450]: pgmap v22141: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T09:00:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:32.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:32.687620+0000) 2022-09-08T09:00:32.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:32.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:32.441108+0000) 2022-09-08T09:00:33.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:33.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:33.687898+0000) 2022-09-08T09:00:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:33.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:33.441251+0000) 2022-09-08T09:00:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:33 smithi055 ceph-mon[97033]: pgmap v22142: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:33 smithi059 ceph-mon[97450]: pgmap v22142: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:34.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:34.688142+0000) 2022-09-08T09:00:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:34.440+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:34.441423+0000) 2022-09-08T09:00:35.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:35 smithi059 ceph-mon[97450]: pgmap v22143: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:35 smithi055 ceph-mon[97033]: pgmap v22143: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:35.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:35.688373+0000) 2022-09-08T09:00:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:35.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:35.441631+0000) 2022-09-08T09:00:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:36.687+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:36.688616+0000) 2022-09-08T09:00:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:36.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:36.441814+0000) 2022-09-08T09:00:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:37.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:37.688908+0000) 2022-09-08T09:00:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:37.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:37.441948+0000) 2022-09-08T09:00:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:37 smithi055 ceph-mon[97033]: pgmap v22144: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:00:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:37 smithi059 ceph-mon[97450]: pgmap v22144: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:00:38.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:38.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:38.689109+0000) 2022-09-08T09:00:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:38.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:38.442131+0000) 2022-09-08T09:00:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:39.688+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:39.689320+0000) 2022-09-08T09:00:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:39.441+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:39.442258+0000) 2022-09-08T09:00:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:39 smithi055 ceph-mon[97033]: pgmap v22145: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:39 smithi059 ceph-mon[97450]: pgmap v22145: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:40.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:40.689548+0000) 2022-09-08T09:00:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:40.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:40.442424+0000) 2022-09-08T09:00:41.770 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:41.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:41.689741+0000) 2022-09-08T09:00:41.770 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:41.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:41.442668+0000) 2022-09-08T09:00:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:41 smithi055 ceph-mon[97033]: pgmap v22146: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:00:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:41 smithi059 ceph-mon[97450]: pgmap v22146: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:00:42.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:00:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:00:41.767+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:00:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:42.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:42.689926+0000) 2022-09-08T09:00:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:42.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:42.442898+0000) 2022-09-08T09:00:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:43.689+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:43.690143+0000) 2022-09-08T09:00:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:43.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:43.443078+0000) 2022-09-08T09:00:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:43 smithi055 ceph-mon[97033]: pgmap v22147: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:43 smithi059 ceph-mon[97450]: pgmap v22147: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:44.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:44.690393+0000) 2022-09-08T09:00:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:44.442+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:44.443251+0000) 2022-09-08T09:00:45.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:45 smithi059 ceph-mon[97450]: pgmap v22148: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:45 smithi055 ceph-mon[97033]: pgmap v22148: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:45.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:45.690661+0000) 2022-09-08T09:00:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:45.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:45.443428+0000) 2022-09-08T09:00:46.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:46.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:46.690885+0000) 2022-09-08T09:00:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:46.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:46.443722+0000) 2022-09-08T09:00:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:47.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:47.691096+0000) 2022-09-08T09:00:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:47.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:47.443846+0000) 2022-09-08T09:00:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:47 smithi055 ceph-mon[97033]: pgmap v22149: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:00:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:47 smithi059 ceph-mon[97450]: pgmap v22149: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:00:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:48.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:48.691336+0000) 2022-09-08T09:00:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:48.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:48.444043+0000) 2022-09-08T09:00:49.906 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:49.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:49.691546+0000) 2022-09-08T09:00:49.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:49.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:49.444255+0000) 2022-09-08T09:00:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:49 smithi055 ceph-mon[97033]: pgmap v22150: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:50.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:49 smithi059 ceph-mon[97450]: pgmap v22150: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:50.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:50.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:50.691772+0000) 2022-09-08T09:00:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:50.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:50.444503+0000) 2022-09-08T09:00:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:51.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:51.691929+0000) 2022-09-08T09:00:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:51.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:51.444759+0000) 2022-09-08T09:00:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:51 smithi055 ceph-mon[97033]: pgmap v22151: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:00:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:51 smithi059 ceph-mon[97450]: pgmap v22151: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:00:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:52.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:52.692153+0000) 2022-09-08T09:00:52.907 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:52.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:52.444928+0000) 2022-09-08T09:00:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:53.690+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:53.692364+0000) 2022-09-08T09:00:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:53.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:53.445108+0000) 2022-09-08T09:00:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:53 smithi055 ceph-mon[97033]: pgmap v22152: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:53 smithi059 ceph-mon[97450]: pgmap v22152: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:54.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:54.692537+0000) 2022-09-08T09:00:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:54.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:54.445312+0000) 2022-09-08T09:00:55.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:55 smithi059 ceph-mon[97450]: pgmap v22153: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:55 smithi055 ceph-mon[97033]: pgmap v22153: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:00:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:55.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:55.692782+0000) 2022-09-08T09:00:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:55.443+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:55.445501+0000) 2022-09-08T09:00:56.785 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:56.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:56.693001+0000) 2022-09-08T09:00:56.786 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:56.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:56.445691+0000) 2022-09-08T09:00:57.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:00:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:00:56.781+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:00:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:57.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:57.445869+0000) 2022-09-08T09:00:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:57.691+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:57.693188+0000) 2022-09-08T09:00:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:57 smithi055 ceph-mon[97033]: pgmap v22154: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:00:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:57 smithi059 ceph-mon[97450]: pgmap v22154: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:00:58.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:58.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:58.693349+0000) 2022-09-08T09:00:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:58.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:58.446017+0000) 2022-09-08T09:00:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:00:59.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:59.693567+0000) 2022-09-08T09:00:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:00:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:00:59.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:00:59.446189+0000) 2022-09-08T09:00:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:00:59 smithi055 ceph-mon[97033]: pgmap v22155: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:00:59 smithi059 ceph-mon[97450]: pgmap v22155: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:00.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:00.692+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:00.693790+0000) 2022-09-08T09:01:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:00.444+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:00.446380+0000) 2022-09-08T09:01:01.886 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:01 smithi059 ceph-mon[97450]: pgmap v22156: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:01:01.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:01.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:01.694056+0000) 2022-09-08T09:01:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:01.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:01.446595+0000) 2022-09-08T09:01:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:01 smithi055 ceph-mon[97033]: pgmap v22156: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:01:02.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:02.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:02.694280+0000) 2022-09-08T09:01:02.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:02.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:02.446887+0000) 2022-09-08T09:01:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:03.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:03.694503+0000) 2022-09-08T09:01:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:03.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:03.447055+0000) 2022-09-08T09:01:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:03 smithi055 ceph-mon[97033]: pgmap v22157: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:03 smithi059 ceph-mon[97450]: pgmap v22157: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:04.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:04.693+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:04.694734+0000) 2022-09-08T09:01:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:04.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:04.447236+0000) 2022-09-08T09:01:05.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:05 smithi059 ceph-mon[97450]: pgmap v22158: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:05 smithi055 ceph-mon[97033]: pgmap v22158: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:05.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:05.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:05.694954+0000) 2022-09-08T09:01:05.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:05.445+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:05.447413+0000) 2022-09-08T09:01:06.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:01:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:01:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:06.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:06.695225+0000) 2022-09-08T09:01:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:06.446+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:06.447719+0000) 2022-09-08T09:01:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:07 smithi055 ceph-mon[97033]: pgmap v22159: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:01:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:01:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:07 smithi059 ceph-mon[97450]: pgmap v22159: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:01:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:01:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:07.694+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:07.695438+0000) 2022-09-08T09:01:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:07.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:07.447863+0000) 2022-09-08T09:01:08.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:08.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:08.448108+0000) 2022-09-08T09:01:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:08.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:08.695700+0000) 2022-09-08T09:01:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:09.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:09.695857+0000) 2022-09-08T09:01:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:09.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:09.448290+0000) 2022-09-08T09:01:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:09 smithi055 ceph-mon[97033]: pgmap v22160: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:01:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:09 smithi059 ceph-mon[97450]: pgmap v22160: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:01:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:10.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:10.696097+0000) 2022-09-08T09:01:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:10.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:10.448430+0000) 2022-09-08T09:01:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:10 smithi055 ceph-mon[97033]: osdmap e1462: 12 total, 11 up, 11 in 2022-09-08T09:01:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:10 smithi059 ceph-mon[97450]: osdmap e1462: 12 total, 11 up, 11 in 2022-09-08T09:01:11.801 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:11.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:11.696367+0000) 2022-09-08T09:01:11.801 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:11.447+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:11.448559+0000) 2022-09-08T09:01:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:11 smithi055 ceph-mon[97033]: pgmap v22162: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:01:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:11 smithi059 ceph-mon[97450]: pgmap v22162: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:01:12.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:01:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:01:11.797+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:01:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:12.695+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:12.696595+0000) 2022-09-08T09:01:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:12.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:12.448751+0000) 2022-09-08T09:01:13.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:13 smithi059 ceph-mon[97450]: pgmap v22163: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:01:13.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T09:01:13.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T09:01:13.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:13 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T09:01:13.884 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:13 smithi055 ceph-mon[97033]: pgmap v22163: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:01:13.885 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T09:01:13.885 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T09:01:13.885 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:13 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T09:01:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:13.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:13.696722+0000) 2022-09-08T09:01:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:13.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:13.448923+0000) 2022-09-08T09:01:14.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:14.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:14.696950+0000) 2022-09-08T09:01:14.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:14.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:14.449172+0000) 2022-09-08T09:01:15.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:01:15.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:14 smithi055 ceph-mon[97033]: pgmap v22164: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:01:15.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:01:15.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:14 smithi059 ceph-mon[97450]: pgmap v22164: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:01:15.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:15.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:15.697155+0000) 2022-09-08T09:01:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:15.448+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:15.449446+0000) 2022-09-08T09:01:16.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:16.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:16.449775+0000) 2022-09-08T09:01:16.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:16.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:16.697365+0000) 2022-09-08T09:01:17.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:17.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:17.449956+0000) 2022-09-08T09:01:17.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:17.696+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:17.697584+0000) 2022-09-08T09:01:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:17 smithi055 ceph-mon[97033]: pgmap v22165: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:01:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:17 smithi059 ceph-mon[97450]: pgmap v22165: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:01:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:18.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:18.450099+0000) 2022-09-08T09:01:18.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:18.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:18.697753+0000) 2022-09-08T09:01:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:19.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:19.698036+0000) 2022-09-08T09:01:19.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:19.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:19.450332+0000) 2022-09-08T09:01:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:19 smithi055 ceph-mon[97033]: pgmap v22166: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:01:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:19 smithi055 ceph-mon[97033]: osdmap e1463: 12 total, 11 up, 11 in 2022-09-08T09:01:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:19 smithi059 ceph-mon[97450]: pgmap v22166: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:01:20.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:19 smithi059 ceph-mon[97450]: osdmap e1463: 12 total, 11 up, 11 in 2022-09-08T09:01:20.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:20.697+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:20.698232+0000) 2022-09-08T09:01:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:20.449+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:20.450503+0000) 2022-09-08T09:01:21.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:21.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:21.698453+0000) 2022-09-08T09:01:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:21.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:21.450675+0000) 2022-09-08T09:01:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:21 smithi055 ceph-mon[97033]: pgmap v22168: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:01:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:21 smithi059 ceph-mon[97450]: pgmap v22168: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:01:22.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:22.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:22.698668+0000) 2022-09-08T09:01:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:22.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:22.450872+0000) 2022-09-08T09:01:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:23.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:23.698800+0000) 2022-09-08T09:01:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:23.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:23.451015+0000) 2022-09-08T09:01:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:23 smithi055 ceph-mon[97033]: pgmap v22169: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:01:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:23 smithi059 ceph-mon[97450]: pgmap v22169: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:01:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:24.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:24.698966+0000) 2022-09-08T09:01:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:24.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:24.451221+0000) 2022-09-08T09:01:25.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:25 smithi059 ceph-mon[97450]: pgmap v22170: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:01:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:25 smithi055 ceph-mon[97033]: pgmap v22170: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:01:25.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:25.698+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:25.699233+0000) 2022-09-08T09:01:25.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:25.450+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:25.451410+0000) 2022-09-08T09:01:26.808 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:26.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:26.699540+0000) 2022-09-08T09:01:26.809 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:26.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:26.451665+0000) 2022-09-08T09:01:27.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:01:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:01:26.805+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:01:27.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:27.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:27.699771+0000) 2022-09-08T09:01:27.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:27.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:27.451805+0000) 2022-09-08T09:01:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:27 smithi055 ceph-mon[97033]: pgmap v22171: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:01:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:27 smithi059 ceph-mon[97450]: pgmap v22171: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:01:28.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:28.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:28.699974+0000) 2022-09-08T09:01:28.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:28.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:28.451984+0000) 2022-09-08T09:01:29.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:29.699+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:29.700168+0000) 2022-09-08T09:01:29.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:29.451+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:29.452191+0000) 2022-09-08T09:01:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:29 smithi055 ceph-mon[97033]: pgmap v22172: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:01:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:29 smithi059 ceph-mon[97450]: pgmap v22172: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:01:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:30.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:30.700391+0000) 2022-09-08T09:01:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:30.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:30.452408+0000) 2022-09-08T09:01:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:31.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:31.700905+0000) 2022-09-08T09:01:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:31.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:31.452655+0000) 2022-09-08T09:01:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:31 smithi055 ceph-mon[97033]: pgmap v22173: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T09:01:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:31 smithi059 ceph-mon[97450]: pgmap v22173: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T09:01:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:32.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:32.701135+0000) 2022-09-08T09:01:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:32.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:32.452898+0000) 2022-09-08T09:01:33.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:33.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:33.701342+0000) 2022-09-08T09:01:33.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:33.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:33.453087+0000) 2022-09-08T09:01:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:33 smithi055 ceph-mon[97033]: pgmap v22174: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:33 smithi059 ceph-mon[97450]: pgmap v22174: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:34.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:34.700+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:34.701570+0000) 2022-09-08T09:01:34.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:34.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:34.453320+0000) 2022-09-08T09:01:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:35 smithi055 ceph-mon[97033]: pgmap v22175: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:35 smithi059 ceph-mon[97450]: pgmap v22175: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:35.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:35.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:35.701841+0000) 2022-09-08T09:01:35.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:35.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:35.453585+0000) 2022-09-08T09:01:36.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:36.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:36.702092+0000) 2022-09-08T09:01:36.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:36.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:36.453700+0000) 2022-09-08T09:01:37.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:37.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:37.702355+0000) 2022-09-08T09:01:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:37.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:37.453873+0000) 2022-09-08T09:01:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:37 smithi055 ceph-mon[97033]: pgmap v22176: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:01:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:37 smithi059 ceph-mon[97450]: pgmap v22176: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:01:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:38.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:38.702609+0000) 2022-09-08T09:01:38.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:38.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:38.454105+0000) 2022-09-08T09:01:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:39.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:39.702904+0000) 2022-09-08T09:01:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:39.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:39.454328+0000) 2022-09-08T09:01:39.989 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:39 smithi055 ceph-mon[97033]: pgmap v22177: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:39 smithi059 ceph-mon[97450]: pgmap v22177: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:40.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:40.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:40.703173+0000) 2022-09-08T09:01:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:40.452+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:40.454474+0000) 2022-09-08T09:01:41.815 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:41.701+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:41.703404+0000) 2022-09-08T09:01:41.815 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:41.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:41.454735+0000) 2022-09-08T09:01:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:41 smithi055 ceph-mon[97033]: pgmap v22178: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:01:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:41 smithi059 ceph-mon[97450]: pgmap v22178: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:01:42.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:01:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:01:41.811+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:01:42.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:42.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:42.703692+0000) 2022-09-08T09:01:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:42.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:42.454943+0000) 2022-09-08T09:01:43.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:43.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:43.455095+0000) 2022-09-08T09:01:43.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:43.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:43.703939+0000) 2022-09-08T09:01:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:43 smithi055 ceph-mon[97033]: pgmap v22179: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:43 smithi059 ceph-mon[97450]: pgmap v22179: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:44.702+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:44.704094+0000) 2022-09-08T09:01:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:44.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:44.455242+0000) 2022-09-08T09:01:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:45 smithi055 ceph-mon[97033]: pgmap v22180: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:45 smithi059 ceph-mon[97450]: pgmap v22180: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:45.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:45.704302+0000) 2022-09-08T09:01:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:45.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:45.455415+0000) 2022-09-08T09:01:46.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:46.453+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:46.455577+0000) 2022-09-08T09:01:46.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:46.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:46.704543+0000) 2022-09-08T09:01:47.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:47.703+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:47.704770+0000) 2022-09-08T09:01:47.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:47.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:47.455746+0000) 2022-09-08T09:01:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:47 smithi055 ceph-mon[97033]: pgmap v22181: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:01:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:47 smithi059 ceph-mon[97450]: pgmap v22181: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:01:48.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:48.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:48.455971+0000) 2022-09-08T09:01:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:48.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:48.704995+0000) 2022-09-08T09:01:49.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:49.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:49.705240+0000) 2022-09-08T09:01:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:49.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:49.456201+0000) 2022-09-08T09:01:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:49 smithi055 ceph-mon[97033]: pgmap v22182: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:49 smithi059 ceph-mon[97450]: pgmap v22182: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:50.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:50.705497+0000) 2022-09-08T09:01:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:50.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:50.456396+0000) 2022-09-08T09:01:51.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:51.704+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:51.705707+0000) 2022-09-08T09:01:51.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:51.454+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:51.456549+0000) 2022-09-08T09:01:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:51 smithi055 ceph-mon[97033]: pgmap v22183: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:01:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:51 smithi059 ceph-mon[97450]: pgmap v22183: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:01:52.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:52.455+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:52.456755+0000) 2022-09-08T09:01:52.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:52.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:52.705928+0000) 2022-09-08T09:01:53.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:53.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:53.706163+0000) 2022-09-08T09:01:53.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:53.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:53.456903+0000) 2022-09-08T09:01:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:53 smithi055 ceph-mon[97033]: pgmap v22184: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:53 smithi059 ceph-mon[97450]: pgmap v22184: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:54.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:54.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:54.706335+0000) 2022-09-08T09:01:54.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:54.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:54.457078+0000) 2022-09-08T09:01:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:55 smithi055 ceph-mon[97033]: pgmap v22185: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:55 smithi059 ceph-mon[97450]: pgmap v22185: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:01:55.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:55.705+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:55.706519+0000) 2022-09-08T09:01:55.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:55.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:55.457234+0000) 2022-09-08T09:01:56.818 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:56.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:56.706737+0000) 2022-09-08T09:01:56.818 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:56.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:56.457396+0000) 2022-09-08T09:01:57.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:01:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:01:56.815+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:01:57.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:57.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:57.706919+0000) 2022-09-08T09:01:57.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:57.456+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:57.457605+0000) 2022-09-08T09:01:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:57 smithi055 ceph-mon[97033]: pgmap v22186: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:01:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:57 smithi059 ceph-mon[97450]: pgmap v22186: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:01:58.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:58.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:58.707057+0000) 2022-09-08T09:01:58.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:58.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:58.457818+0000) 2022-09-08T09:01:59.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:01:59.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:59.707262+0000) 2022-09-08T09:01:59.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:01:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:01:59.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:01:59.457988+0000) 2022-09-08T09:01:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:01:59 smithi055 ceph-mon[97033]: pgmap v22187: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:01:59 smithi059 ceph-mon[97450]: pgmap v22187: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:00.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:00.706+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:00.707477+0000) 2022-09-08T09:02:00.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:00.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:00.458137+0000) 2022-09-08T09:02:01.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:01.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:01.707745+0000) 2022-09-08T09:02:01.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:01.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:01.458353+0000) 2022-09-08T09:02:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:01 smithi055 ceph-mon[97033]: pgmap v22188: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:02:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:01 smithi059 ceph-mon[97450]: pgmap v22188: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:02:02.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:02.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:02.707917+0000) 2022-09-08T09:02:02.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:02.457+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:02.458586+0000) 2022-09-08T09:02:03.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:03.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:03.708142+0000) 2022-09-08T09:02:03.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:03.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:03.458773+0000) 2022-09-08T09:02:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:03 smithi055 ceph-mon[97033]: pgmap v22189: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:03 smithi059 ceph-mon[97450]: pgmap v22189: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:04.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:04.707+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:04.708377+0000) 2022-09-08T09:02:04.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:04.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:04.458915+0000) 2022-09-08T09:02:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:05 smithi055 ceph-mon[97033]: pgmap v22190: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:05 smithi059 ceph-mon[97450]: pgmap v22190: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:05.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:05.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:05.459075+0000) 2022-09-08T09:02:05.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:05.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:05.708630+0000) 2022-09-08T09:02:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:02:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:02:06.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:06.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:06.708834+0000) 2022-09-08T09:02:06.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:06.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:06.459282+0000) 2022-09-08T09:02:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:07 smithi055 ceph-mon[97033]: pgmap v22191: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:02:07.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:02:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:07 smithi059 ceph-mon[97450]: pgmap v22191: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:02:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:02:07.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:07.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:07.708960+0000) 2022-09-08T09:02:07.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:07.458+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:07.459485+0000) 2022-09-08T09:02:08.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:08.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:08.709187+0000) 2022-09-08T09:02:08.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:08.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:08.459697+0000) 2022-09-08T09:02:09.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:09.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:09.709440+0000) 2022-09-08T09:02:09.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:09.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:09.459910+0000) 2022-09-08T09:02:09.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:09 smithi055 ceph-mon[97033]: pgmap v22192: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:02:10.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:09 smithi059 ceph-mon[97450]: pgmap v22192: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:02:10.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:10.708+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:10.709623+0000) 2022-09-08T09:02:10.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:10.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:10.460140+0000) 2022-09-08T09:02:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:10 smithi055 ceph-mon[97033]: osdmap e1464: 12 total, 11 up, 11 in 2022-09-08T09:02:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:10 smithi059 ceph-mon[97450]: osdmap e1464: 12 total, 11 up, 11 in 2022-09-08T09:02:11.832 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:11.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:11.709819+0000) 2022-09-08T09:02:11.833 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:11.459+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:11.460294+0000) 2022-09-08T09:02:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:11 smithi055 ceph-mon[97033]: pgmap v22194: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:02:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:11 smithi059 ceph-mon[97450]: pgmap v22194: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:02:12.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:02:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:02:11.829+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:02:12.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:12.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:12.710039+0000) 2022-09-08T09:02:12.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:12.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:12.460474+0000) 2022-09-08T09:02:13.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:13.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:13.710274+0000) 2022-09-08T09:02:13.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:13.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:13.460673+0000) 2022-09-08T09:02:13.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:13 smithi055 ceph-mon[97033]: pgmap v22195: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:02:13.995 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:13 smithi059 ceph-mon[97450]: pgmap v22195: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:02:14.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T09:02:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T09:02:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T09:02:14.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:02:14.732 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:14 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:02:14.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:14.709+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:14.710508+0000) 2022-09-08T09:02:14.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:14.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:14.460851+0000) 2022-09-08T09:02:15.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T09:02:15.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T09:02:15.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T09:02:15.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:02:15.042 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:14 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:02:15.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:15.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:15.461024+0000) 2022-09-08T09:02:15.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:15.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:15.710669+0000) 2022-09-08T09:02:15.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:15 smithi055 ceph-mon[97033]: pgmap v22196: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:02:16.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:15 smithi059 ceph-mon[97450]: pgmap v22196: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:02:16.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:16.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:16.710854+0000) 2022-09-08T09:02:16.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:16.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:16.461265+0000) 2022-09-08T09:02:17.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:17.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:17.711001+0000) 2022-09-08T09:02:17.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:17.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:17.461456+0000) 2022-09-08T09:02:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:17 smithi055 ceph-mon[97033]: pgmap v22197: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:02:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:02:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:02:17.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:02:17.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:02:17.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:17.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:17.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:17.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:17 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:17 smithi059 ceph-mon[97450]: pgmap v22197: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:02:18.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:02:18.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:02:18.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:02:18.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:02:18.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:18.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:18.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:18.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:17 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:18.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:18.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:18.711194+0000) 2022-09-08T09:02:18.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:18.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:18.461602+0000) 2022-09-08T09:02:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:18 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi059 to 86162k 2022-09-08T09:02:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:18 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi059 to 88230760: error parsing value: Value '88230760' is below minimum 939524096 2022-09-08T09:02:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:18.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:18.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:18.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:18.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:18 smithi055 ceph-mon[97033]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T09:02:18.983 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:18 smithi055 ceph-mon[97033]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T09:02:18.984 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:02:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:18 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi059 to 86162k 2022-09-08T09:02:19.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:18 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi059 to 88230760: error parsing value: Value '88230760' is below minimum 939524096 2022-09-08T09:02:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:19.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:19.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:19.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-08T09:02:19.034 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:18 smithi059 ceph-mon[97450]: Adjusting osd_memory_target on smithi055 to 197.2M 2022-09-08T09:02:19.035 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:18 smithi059 ceph-mon[97450]: Unable to set osd_memory_target on smithi055 to 206878515: error parsing value: Value '206878515' is below minimum 939524096 2022-09-08T09:02:19.036 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:02:19.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:19.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:19.711407+0000) 2022-09-08T09:02:19.937 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:19.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:19.461755+0000) 2022-09-08T09:02:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:19 smithi055 ceph-mon[97033]: pgmap v22198: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:02:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:19 smithi055 ceph-mon[97033]: osdmap e1465: 12 total, 11 up, 11 in 2022-09-08T09:02:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:19 smithi059 ceph-mon[97450]: pgmap v22198: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:02:20.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:19 smithi059 ceph-mon[97450]: osdmap e1465: 12 total, 11 up, 11 in 2022-09-08T09:02:20.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:20.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:20.711674+0000) 2022-09-08T09:02:20.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:20.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:20.461897+0000) 2022-09-08T09:02:21.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:21.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:21.711883+0000) 2022-09-08T09:02:21.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:21.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:21.462099+0000) 2022-09-08T09:02:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:21 smithi059 ceph-mon[97450]: pgmap v22200: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:02:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:21 smithi055 ceph-mon[97033]: pgmap v22200: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:02:22.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:22.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:22.712034+0000) 2022-09-08T09:02:22.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:22.460+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:22.462331+0000) 2022-09-08T09:02:23.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:23.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:23.712195+0000) 2022-09-08T09:02:23.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:23.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:23.462530+0000) 2022-09-08T09:02:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:23 smithi059 ceph-mon[97450]: pgmap v22201: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:02:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:23 smithi055 ceph-mon[97033]: pgmap v22201: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:02:24.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:24.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:24.712397+0000) 2022-09-08T09:02:24.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:24.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:24.462714+0000) 2022-09-08T09:02:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:25 smithi055 ceph-mon[97033]: pgmap v22202: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:02:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:25 smithi059 ceph-mon[97450]: pgmap v22202: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:02:25.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:25.710+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:25.712600+0000) 2022-09-08T09:02:25.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:25.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:25.462882+0000) 2022-09-08T09:02:26.840 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:26.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:26.712810+0000) 2022-09-08T09:02:26.841 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:26.461+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:26.463044+0000) 2022-09-08T09:02:27.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:02:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:02:26.836+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:02:27.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:27.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:27.712958+0000) 2022-09-08T09:02:27.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:27.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:27.463226+0000) 2022-09-08T09:02:27.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:27 smithi055 ceph-mon[97033]: pgmap v22203: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:02:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:27 smithi059 ceph-mon[97450]: pgmap v22203: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:02:28.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:28.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:28.463377+0000) 2022-09-08T09:02:28.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:28.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:28.713157+0000) 2022-09-08T09:02:29.905 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:29.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:29.713353+0000) 2022-09-08T09:02:29.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:29.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:29.463623+0000) 2022-09-08T09:02:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:29 smithi055 ceph-mon[97033]: pgmap v22204: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:02:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:29 smithi059 ceph-mon[97450]: pgmap v22204: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:02:30.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:30.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:30.463839+0000) 2022-09-08T09:02:30.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:30.711+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:30.713547+0000) 2022-09-08T09:02:31.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:31.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:31.713782+0000) 2022-09-08T09:02:31.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:31.462+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:31.463979+0000) 2022-09-08T09:02:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:31 smithi055 ceph-mon[97033]: pgmap v22205: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T09:02:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:31 smithi059 ceph-mon[97450]: pgmap v22205: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T09:02:32.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:32.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:32.713996+0000) 2022-09-08T09:02:32.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:32.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:32.464212+0000) 2022-09-08T09:02:33.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:33.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:33.714180+0000) 2022-09-08T09:02:33.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:33.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:33.464398+0000) 2022-09-08T09:02:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:33 smithi055 ceph-mon[97033]: pgmap v22206: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:33 smithi059 ceph-mon[97450]: pgmap v22206: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:34.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:34.712+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:34.714404+0000) 2022-09-08T09:02:34.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:34.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:34.464600+0000) 2022-09-08T09:02:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:35 smithi055 ceph-mon[97033]: pgmap v22207: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:35 smithi059 ceph-mon[97450]: pgmap v22207: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:35.908 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:35.713+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:35.714687+0000) 2022-09-08T09:02:35.909 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:35.463+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:35.464793+0000) 2022-09-08T09:02:36.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:36.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:36.464951+0000) 2022-09-08T09:02:36.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:36.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:36.714825+0000) 2022-09-08T09:02:37.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:37.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:37.465176+0000) 2022-09-08T09:02:37.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:37.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:37.715006+0000) 2022-09-08T09:02:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:37 smithi055 ceph-mon[97033]: pgmap v22208: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:02:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:37 smithi059 ceph-mon[97450]: pgmap v22208: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:02:38.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:38.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:38.715160+0000) 2022-09-08T09:02:38.905 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:38.464+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:38.465380+0000) 2022-09-08T09:02:39.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:39.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:39.715346+0000) 2022-09-08T09:02:39.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:39.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:39.465621+0000) 2022-09-08T09:02:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:39 smithi055 ceph-mon[97033]: pgmap v22209: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:39 smithi059 ceph-mon[97450]: pgmap v22209: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:40.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:40.714+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:40.715552+0000) 2022-09-08T09:02:40.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:40.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:40.465909+0000) 2022-09-08T09:02:41.840 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:41.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:41.715736+0000) 2022-09-08T09:02:41.841 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:41.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:41.466155+0000) 2022-09-08T09:02:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:41 smithi055 ceph-mon[97033]: pgmap v22210: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:02:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:41 smithi059 ceph-mon[97450]: pgmap v22210: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:02:42.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:02:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:02:41.837+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:02:42.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:42.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:42.715966+0000) 2022-09-08T09:02:42.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:42.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:42.466358+0000) 2022-09-08T09:02:43.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:43.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:43.716202+0000) 2022-09-08T09:02:43.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:43.465+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:43.466519+0000) 2022-09-08T09:02:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:43 smithi055 ceph-mon[97033]: pgmap v22211: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:43 smithi059 ceph-mon[97450]: pgmap v22211: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:44.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:44.715+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:44.716449+0000) 2022-09-08T09:02:44.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:44.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:44.466691+0000) 2022-09-08T09:02:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:45 smithi055 ceph-mon[97033]: pgmap v22212: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:45 smithi059 ceph-mon[97450]: pgmap v22212: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:45.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:45.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:45.716689+0000) 2022-09-08T09:02:45.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:45.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:45.466944+0000) 2022-09-08T09:02:46.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:46.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:46.716916+0000) 2022-09-08T09:02:46.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:46.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:46.467190+0000) 2022-09-08T09:02:47.903 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:47.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:47.717089+0000) 2022-09-08T09:02:47.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:47.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:47.467337+0000) 2022-09-08T09:02:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:47 smithi055 ceph-mon[97033]: pgmap v22213: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:02:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:47 smithi059 ceph-mon[97450]: pgmap v22213: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:02:48.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:48.466+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:48.467486+0000) 2022-09-08T09:02:48.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:48.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:48.717310+0000) 2022-09-08T09:02:49.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:49.716+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:49.717518+0000) 2022-09-08T09:02:49.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:49.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:49.467736+0000) 2022-09-08T09:02:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:49 smithi055 ceph-mon[97033]: pgmap v22214: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:49 smithi059 ceph-mon[97450]: pgmap v22214: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:50.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:50.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:50.717747+0000) 2022-09-08T09:02:50.904 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:50.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:50.467884+0000) 2022-09-08T09:02:51.720 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:51.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:51.468035+0000) 2022-09-08T09:02:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:51 smithi055 ceph-mon[97033]: pgmap v22215: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:02:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:51 smithi059 ceph-mon[97450]: pgmap v22215: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:02:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:51.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:51.717957+0000) 2022-09-08T09:02:52.720 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:52.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:52.468198+0000) 2022-09-08T09:02:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:52.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:52.718182+0000) 2022-09-08T09:02:53.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:53.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:53.468402+0000) 2022-09-08T09:02:53.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:53.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:53.718363+0000) 2022-09-08T09:02:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:53 smithi055 ceph-mon[97033]: pgmap v22216: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:53 smithi059 ceph-mon[97450]: pgmap v22216: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:54.721 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:54.467+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:54.468551+0000) 2022-09-08T09:02:55.070 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:54.717+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:54.718489+0000) 2022-09-08T09:02:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:55 smithi055 ceph-mon[97033]: pgmap v22217: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:55 smithi059 ceph-mon[97450]: pgmap v22217: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:02:55.790 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:55.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:55.718645+0000) 2022-09-08T09:02:55.791 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:55.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:55.468697+0000) 2022-09-08T09:02:56.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:56.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:56.468836+0000) 2022-09-08T09:02:57.010 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:02:56.853+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:02:57.010 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:56.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:56.718773+0000) 2022-09-08T09:02:57.903 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:57.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:57.469035+0000) 2022-09-08T09:02:57.904 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:57.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:57.718987+0000) 2022-09-08T09:02:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:57 smithi055 ceph-mon[97033]: pgmap v22218: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:02:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:57 smithi059 ceph-mon[97450]: pgmap v22218: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:02:58.829 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:58.718+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:58.719172+0000) 2022-09-08T09:02:58.830 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:58.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:58.469201+0000) 2022-09-08T09:02:59.721 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:02:59.468+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:59.469385+0000) 2022-09-08T09:02:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:02:59 smithi055 ceph-mon[97033]: pgmap v22219: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:02:59 smithi059 ceph-mon[97450]: pgmap v22219: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:02:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:02:59.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:02:59.719427+0000) 2022-09-08T09:03:00.722 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:00.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:00.469610+0000) 2022-09-08T09:03:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:00.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:00.719712+0000) 2022-09-08T09:03:01.722 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:01.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:01.469789+0000) 2022-09-08T09:03:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:01 smithi055 ceph-mon[97033]: pgmap v22220: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:03:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:01 smithi059 ceph-mon[97450]: pgmap v22220: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:03:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:01.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:01.719961+0000) 2022-09-08T09:03:02.722 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:02.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:02.469952+0000) 2022-09-08T09:03:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:02.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:02.720166+0000) 2022-09-08T09:03:03.722 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:03.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:03.470158+0000) 2022-09-08T09:03:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:03 smithi055 ceph-mon[97033]: pgmap v22221: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:03 smithi059 ceph-mon[97450]: pgmap v22221: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:03.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:03.720388+0000) 2022-09-08T09:03:04.723 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:04.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:04.470351+0000) 2022-09-08T09:03:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:04.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:04.720566+0000) 2022-09-08T09:03:05.723 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:05.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:05.470577+0000) 2022-09-08T09:03:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:05 smithi055 ceph-mon[97033]: pgmap v22222: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:05 smithi059 ceph-mon[97450]: pgmap v22222: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:05.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:05.720718+0000) 2022-09-08T09:03:06.723 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:06.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:06.470806+0000) 2022-09-08T09:03:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:03:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:03:07.090 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:06.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:06.720984+0000) 2022-09-08T09:03:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:07 smithi055 ceph-mon[97033]: pgmap v22223: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:03:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:03:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:07 smithi059 ceph-mon[97450]: pgmap v22223: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:03:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:03:07.838 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:07.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:07.470970+0000) 2022-09-08T09:03:07.838 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:07.719+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:07.721229+0000) 2022-09-08T09:03:08.723 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:08.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:08.471106+0000) 2022-09-08T09:03:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:08.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:08.721483+0000) 2022-09-08T09:03:09.724 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:09.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:09.471311+0000) 2022-09-08T09:03:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:09 smithi055 ceph-mon[97033]: pgmap v22224: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:03:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:09 smithi059 ceph-mon[97450]: pgmap v22224: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:03:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:09.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:09.721738+0000) 2022-09-08T09:03:10.724 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:10.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:10.471461+0000) 2022-09-08T09:03:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:10 smithi055 ceph-mon[97033]: osdmap e1466: 12 total, 11 up, 11 in 2022-09-08T09:03:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:10 smithi059 ceph-mon[97450]: osdmap e1466: 12 total, 11 up, 11 in 2022-09-08T09:03:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:10.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:10.721981+0000) 2022-09-08T09:03:11.724 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:11.469+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:11.471618+0000) 2022-09-08T09:03:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:11 smithi055 ceph-mon[97033]: pgmap v22226: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:03:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:11 smithi059 ceph-mon[97450]: pgmap v22226: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:03:12.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:11.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:11.722170+0000) 2022-09-08T09:03:12.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:03:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:03:11.861+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:03:12.725 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:12.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:12.471762+0000) 2022-09-08T09:03:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:12.720+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:12.722382+0000) 2022-09-08T09:03:13.725 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:13.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:13.471986+0000) 2022-09-08T09:03:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:13 smithi055 ceph-mon[97033]: pgmap v22227: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:03:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:13 smithi059 ceph-mon[97450]: pgmap v22227: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:03:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:13.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:13.722613+0000) 2022-09-08T09:03:14.725 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:14.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:14.472235+0000) 2022-09-08T09:03:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:14.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:14.722888+0000) 2022-09-08T09:03:15.725 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:15.470+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:15.472430+0000) 2022-09-08T09:03:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:15 smithi055 ceph-mon[97033]: pgmap v22228: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:03:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:15 smithi059 ceph-mon[97450]: pgmap v22228: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:03:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:15.721+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:15.723053+0000) 2022-09-08T09:03:16.725 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:16.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:16.472672+0000) 2022-09-08T09:03:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:16.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:16.723273+0000) 2022-09-08T09:03:17.726 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:17.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:17.472848+0000) 2022-09-08T09:03:17.907 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:17 smithi055 ceph-mon[97033]: pgmap v22229: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:03:17.913 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:17 smithi059 ceph-mon[97450]: pgmap v22229: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:03:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:17.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:17.723486+0000) 2022-09-08T09:03:18.726 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:18.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:18.473095+0000) 2022-09-08T09:03:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T09:03:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T09:03:18.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T09:03:18.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:03:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T09:03:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T09:03:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T09:03:18.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:03:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:18.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:18.723730+0000) 2022-09-08T09:03:19.726 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:19.472+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:19.473276+0000) 2022-09-08T09:03:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:19 smithi055 ceph-mon[97033]: pgmap v22230: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:03:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:19 smithi055 ceph-mon[97033]: osdmap e1467: 12 total, 11 up, 11 in 2022-09-08T09:03:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:19 smithi059 ceph-mon[97450]: pgmap v22230: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:03:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:19 smithi059 ceph-mon[97450]: osdmap e1467: 12 total, 11 up, 11 in 2022-09-08T09:03:20.134 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:19.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:19.724168+0000) 2022-09-08T09:03:20.726 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:20.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:20.473408+0000) 2022-09-08T09:03:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:20.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:20.724335+0000) 2022-09-08T09:03:21.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:21.471+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:21.473532+0000) 2022-09-08T09:03:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:21 smithi055 ceph-mon[97033]: pgmap v22232: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:03:22.003 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:21.722+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:21.724478+0000) 2022-09-08T09:03:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:21 smithi059 ceph-mon[97450]: pgmap v22232: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:03:22.727 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:22.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:22.473749+0000) 2022-09-08T09:03:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:22.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:22.724728+0000) 2022-09-08T09:03:23.727 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:23.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:23.473965+0000) 2022-09-08T09:03:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:23 smithi055 ceph-mon[97033]: pgmap v22233: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:03:24.028 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:23.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:23.724872+0000) 2022-09-08T09:03:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:23 smithi059 ceph-mon[97450]: pgmap v22233: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:03:24.727 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:24.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:24.474150+0000) 2022-09-08T09:03:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:24.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:24.725098+0000) 2022-09-08T09:03:25.727 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:25.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:25.474287+0000) 2022-09-08T09:03:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:25 smithi055 ceph-mon[97033]: pgmap v22234: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:03:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:25 smithi059 ceph-mon[97450]: pgmap v22234: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:03:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:25.724+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:25.725390+0000) 2022-09-08T09:03:26.728 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:26.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:26.474418+0000) 2022-09-08T09:03:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:26.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:26.725666+0000) 2022-09-08T09:03:27.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:03:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:03:26.863+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:03:27.728 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:27.473+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:27.474617+0000) 2022-09-08T09:03:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:27 smithi055 ceph-mon[97033]: pgmap v22235: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:03:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:27 smithi059 ceph-mon[97450]: pgmap v22235: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:03:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:27.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:27.725818+0000) 2022-09-08T09:03:28.728 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:28.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:28.474839+0000) 2022-09-08T09:03:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:28.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:28.726019+0000) 2022-09-08T09:03:29.728 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:29.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:29.474998+0000) 2022-09-08T09:03:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:29 smithi055 ceph-mon[97033]: pgmap v22236: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:03:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:29 smithi059 ceph-mon[97450]: pgmap v22236: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:03:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:29.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:29.726245+0000) 2022-09-08T09:03:30.729 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:30.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:30.475158+0000) 2022-09-08T09:03:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:30.725+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:30.726415+0000) 2022-09-08T09:03:31.729 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:31.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:31.475350+0000) 2022-09-08T09:03:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:31 smithi055 ceph-mon[97033]: pgmap v22237: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T09:03:32.030 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:31.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:31.726624+0000) 2022-09-08T09:03:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:31 smithi059 ceph-mon[97450]: pgmap v22237: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T09:03:32.653 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:32.474+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:32.475506+0000) 2022-09-08T09:03:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:32.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:32.726757+0000) 2022-09-08T09:03:33.729 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:33.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:33.475778+0000) 2022-09-08T09:03:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:33 smithi055 ceph-mon[97033]: pgmap v22238: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:33 smithi059 ceph-mon[97450]: pgmap v22238: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:33.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:33.726875+0000) 2022-09-08T09:03:34.729 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:34.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:34.475984+0000) 2022-09-08T09:03:35.009 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:34.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:34.726998+0000) 2022-09-08T09:03:35.729 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:35.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:35.476197+0000) 2022-09-08T09:03:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:35 smithi055 ceph-mon[97033]: pgmap v22239: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:35 smithi059 ceph-mon[97450]: pgmap v22239: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:35.726+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:35.727229+0000) 2022-09-08T09:03:36.729 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:36.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:36.476433+0000) 2022-09-08T09:03:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:36.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:36.727487+0000) 2022-09-08T09:03:37.730 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:37.475+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:37.476599+0000) 2022-09-08T09:03:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:37 smithi055 ceph-mon[97033]: pgmap v22240: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:03:38.013 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:37.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:37.727736+0000) 2022-09-08T09:03:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:37 smithi059 ceph-mon[97450]: pgmap v22240: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:03:38.730 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:38.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:38.476796+0000) 2022-09-08T09:03:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:38.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:38.727969+0000) 2022-09-08T09:03:39.730 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:39.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:39.476981+0000) 2022-09-08T09:03:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:39 smithi055 ceph-mon[97033]: pgmap v22241: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:39 smithi059 ceph-mon[97450]: pgmap v22241: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:39.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:39.728165+0000) 2022-09-08T09:03:40.730 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:40.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:40.477189+0000) 2022-09-08T09:03:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:40.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:40.728298+0000) 2022-09-08T09:03:41.731 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:41.476+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:41.477406+0000) 2022-09-08T09:03:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:41 smithi055 ceph-mon[97033]: pgmap v22242: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:03:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:41 smithi059 ceph-mon[97450]: pgmap v22242: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:03:42.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:03:41.870+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:03:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:41.727+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:41.728544+0000) 2022-09-08T09:03:42.731 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:42.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:42.477595+0000) 2022-09-08T09:03:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:42.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:42.728799+0000) 2022-09-08T09:03:43.731 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:43.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:43.477817+0000) 2022-09-08T09:03:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:43 smithi055 ceph-mon[97033]: pgmap v22243: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:43 smithi059 ceph-mon[97450]: pgmap v22243: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:44.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:43.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:43.729002+0000) 2022-09-08T09:03:44.731 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:44.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:44.477985+0000) 2022-09-08T09:03:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:44.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:44.729168+0000) 2022-09-08T09:03:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:45 smithi055 ceph-mon[97033]: pgmap v22244: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:45.731 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:45.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:45.478129+0000) 2022-09-08T09:03:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:45 smithi059 ceph-mon[97450]: pgmap v22244: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:45.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:45.729340+0000) 2022-09-08T09:03:46.731 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:46.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:46.478271+0000) 2022-09-08T09:03:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:46.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:46.729524+0000) 2022-09-08T09:03:47.732 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:47.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:47.478388+0000) 2022-09-08T09:03:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:47 smithi055 ceph-mon[97033]: pgmap v22245: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:03:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:47 smithi059 ceph-mon[97450]: pgmap v22245: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:03:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:47.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:47.729742+0000) 2022-09-08T09:03:48.732 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:48.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:48.478585+0000) 2022-09-08T09:03:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:48.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:48.729925+0000) 2022-09-08T09:03:49.732 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:49.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:49.478807+0000) 2022-09-08T09:03:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:49 smithi055 ceph-mon[97033]: pgmap v22246: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:50.023 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:49.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:49.730143+0000) 2022-09-08T09:03:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:49 smithi059 ceph-mon[97450]: pgmap v22246: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:50.732 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:50.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:50.479029+0000) 2022-09-08T09:03:51.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:50.728+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:50.730296+0000) 2022-09-08T09:03:51.733 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:51.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:51.479236+0000) 2022-09-08T09:03:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:51 smithi055 ceph-mon[97033]: pgmap v22247: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:03:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:51 smithi059 ceph-mon[97450]: pgmap v22247: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:03:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:51.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:51.730519+0000) 2022-09-08T09:03:52.733 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:52.477+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:52.479523+0000) 2022-09-08T09:03:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:52.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:52.730755+0000) 2022-09-08T09:03:53.733 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:53.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:53.479702+0000) 2022-09-08T09:03:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:53 smithi055 ceph-mon[97033]: pgmap v22248: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:53 smithi059 ceph-mon[97450]: pgmap v22248: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:54.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:53.729+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:53.730899+0000) 2022-09-08T09:03:54.733 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:54.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:54.479833+0000) 2022-09-08T09:03:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:54.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:54.731110+0000) 2022-09-08T09:03:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:55 smithi055 ceph-mon[97033]: pgmap v22249: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:55.733 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:55.478+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:55.480018+0000) 2022-09-08T09:03:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:55 smithi059 ceph-mon[97450]: pgmap v22249: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:03:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:55.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:55.731315+0000) 2022-09-08T09:03:56.734 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:56.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:56.480236+0000) 2022-09-08T09:03:57.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:56.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:56.731545+0000) 2022-09-08T09:03:57.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:03:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:03:56.871+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:03:57.734 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:57.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:57.480388+0000) 2022-09-08T09:03:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:57 smithi055 ceph-mon[97033]: pgmap v22250: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:03:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:57 smithi059 ceph-mon[97450]: pgmap v22250: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:03:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:57.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:57.731783+0000) 2022-09-08T09:03:58.734 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:58.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:58.480522+0000) 2022-09-08T09:03:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:58.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:58.732007+0000) 2022-09-08T09:03:59.734 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:03:59.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:59.480690+0000) 2022-09-08T09:03:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:03:59 smithi055 ceph-mon[97033]: pgmap v22251: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:03:59 smithi059 ceph-mon[97450]: pgmap v22251: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:00.156 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:03:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:03:59.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:03:59.732243+0000) 2022-09-08T09:04:00.735 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:00.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:00.480821+0000) 2022-09-08T09:04:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:00.730+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:00.732433+0000) 2022-09-08T09:04:01.735 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:01.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:01.480937+0000) 2022-09-08T09:04:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:01 smithi055 ceph-mon[97033]: pgmap v22252: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:04:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:01 smithi059 ceph-mon[97450]: pgmap v22252: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:04:02.035 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:01.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:01.732677+0000) 2022-09-08T09:04:02.735 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:02.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:02.481110+0000) 2022-09-08T09:04:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:02.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:02.732919+0000) 2022-09-08T09:04:03.735 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:03.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:03.481318+0000) 2022-09-08T09:04:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:03 smithi055 ceph-mon[97033]: pgmap v22253: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:03 smithi059 ceph-mon[97450]: pgmap v22253: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:03.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:03.733102+0000) 2022-09-08T09:04:04.736 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:04.479+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:04.481504+0000) 2022-09-08T09:04:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:04.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:04.733337+0000) 2022-09-08T09:04:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:05 smithi055 ceph-mon[97033]: pgmap v22254: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:05.735 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:05.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:05.481723+0000) 2022-09-08T09:04:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:05 smithi059 ceph-mon[97450]: pgmap v22254: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:05.731+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:05.733584+0000) 2022-09-08T09:04:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:04:06.736 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:06.480+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:06.481894+0000) 2022-09-08T09:04:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:04:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:06.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:06.733866+0000) 2022-09-08T09:04:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:07 smithi055 ceph-mon[97033]: pgmap v22255: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:04:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:04:07.736 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:07.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:07.482083+0000) 2022-09-08T09:04:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:07 smithi059 ceph-mon[97450]: pgmap v22255: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:04:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:04:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:07.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:07.734056+0000) 2022-09-08T09:04:08.736 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:08.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:08.482210+0000) 2022-09-08T09:04:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:08.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:08.734180+0000) 2022-09-08T09:04:09.736 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:09.481+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:09.482416+0000) 2022-09-08T09:04:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:09 smithi055 ceph-mon[97033]: pgmap v22256: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:04:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:09 smithi059 ceph-mon[97450]: pgmap v22256: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:04:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:09.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:09.734361+0000) 2022-09-08T09:04:10.737 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:10.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:10.482689+0000) 2022-09-08T09:04:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:10 smithi055 ceph-mon[97033]: osdmap e1468: 12 total, 11 up, 11 in 2022-09-08T09:04:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:10 smithi059 ceph-mon[97450]: osdmap e1468: 12 total, 11 up, 11 in 2022-09-08T09:04:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:10.733+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:10.734559+0000) 2022-09-08T09:04:11.737 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:11.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:11.482841+0000) 2022-09-08T09:04:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:11 smithi055 ceph-mon[97033]: pgmap v22258: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:04:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:11 smithi059 ceph-mon[97450]: pgmap v22258: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:04:12.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:04:11.888+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:04:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:11.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:11.734727+0000) 2022-09-08T09:04:12.737 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:12.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:12.482982+0000) 2022-09-08T09:04:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:12.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:12.734883+0000) 2022-09-08T09:04:13.737 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:13.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:13.483117+0000) 2022-09-08T09:04:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:13 smithi055 ceph-mon[97033]: pgmap v22259: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:04:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:13 smithi059 ceph-mon[97450]: pgmap v22259: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:04:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:13.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:13.735064+0000) 2022-09-08T09:04:14.737 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:14.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:14.483330+0000) 2022-09-08T09:04:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:14.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:14.735296+0000) 2022-09-08T09:04:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:15 smithi055 ceph-mon[97033]: pgmap v22260: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:04:15.737 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:15.482+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:15.483472+0000) 2022-09-08T09:04:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:15 smithi059 ceph-mon[97450]: pgmap v22260: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:04:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:15.734+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:15.735506+0000) 2022-09-08T09:04:16.738 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:16.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:16.483688+0000) 2022-09-08T09:04:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:16.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:16.735736+0000) 2022-09-08T09:04:17.738 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:17.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:17.483913+0000) 2022-09-08T09:04:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:17 smithi055 ceph-mon[97033]: pgmap v22261: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:04:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:17 smithi059 ceph-mon[97450]: pgmap v22261: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:04:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:17.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:17.735866+0000) 2022-09-08T09:04:18.739 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:18.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:18.484051+0000) 2022-09-08T09:04:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T09:04:18.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T09:04:18.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:18 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T09:04:18.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T09:04:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T09:04:18.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:18 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T09:04:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:18.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:18.736082+0000) 2022-09-08T09:04:19.738 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:19.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:19.484218+0000) 2022-09-08T09:04:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:19 smithi055 ceph-mon[97033]: pgmap v22262: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:04:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:04:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:04:19.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:19 smithi055 ceph-mon[97033]: osdmap e1469: 12 total, 11 up, 11 in 2022-09-08T09:04:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:19 smithi059 ceph-mon[97450]: pgmap v22262: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:04:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:04:20.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:04:20.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:19 smithi059 ceph-mon[97450]: osdmap e1469: 12 total, 11 up, 11 in 2022-09-08T09:04:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:19.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:19.736327+0000) 2022-09-08T09:04:20.740 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:20.483+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:20.484389+0000) 2022-09-08T09:04:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:20.735+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:20.736454+0000) 2022-09-08T09:04:21.739 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:21.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:21.484537+0000) 2022-09-08T09:04:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:21 smithi055 ceph-mon[97033]: pgmap v22264: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:04:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:21 smithi059 ceph-mon[97450]: pgmap v22264: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:04:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:21.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:21.736680+0000) 2022-09-08T09:04:22.739 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:22.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:22.484771+0000) 2022-09-08T09:04:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:22.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:22.736837+0000) 2022-09-08T09:04:23.739 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:23.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:23.484966+0000) 2022-09-08T09:04:23.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:23 smithi055 ceph-mon[97033]: pgmap v22265: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:04:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:23 smithi059 ceph-mon[97450]: pgmap v22265: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:04:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:23.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:23.737016+0000) 2022-09-08T09:04:24.739 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:24.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:24.485238+0000) 2022-09-08T09:04:25.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:24.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:24.737246+0000) 2022-09-08T09:04:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:25 smithi055 ceph-mon[97033]: pgmap v22266: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:04:25.739 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:25.484+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:25.485442+0000) 2022-09-08T09:04:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:25 smithi059 ceph-mon[97450]: pgmap v22266: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:04:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:25.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:25.737384+0000) 2022-09-08T09:04:26.740 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:26.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:26.485649+0000) 2022-09-08T09:04:27.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:04:26.890+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:04:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:26.736+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:26.737579+0000) 2022-09-08T09:04:27.740 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:27.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:27.485807+0000) 2022-09-08T09:04:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:27 smithi055 ceph-mon[97033]: pgmap v22267: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:04:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:27 smithi059 ceph-mon[97450]: pgmap v22267: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:04:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:27.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:27.737693+0000) 2022-09-08T09:04:28.740 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:28.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:28.486006+0000) 2022-09-08T09:04:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:28.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:28.737892+0000) 2022-09-08T09:04:29.740 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:29.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:29.486231+0000) 2022-09-08T09:04:29.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:29 smithi055 ceph-mon[97033]: pgmap v22268: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:04:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:29 smithi059 ceph-mon[97450]: pgmap v22268: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:04:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:29.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:29.738089+0000) 2022-09-08T09:04:30.740 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:30.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:30.486422+0000) 2022-09-08T09:04:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:30.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:30.738261+0000) 2022-09-08T09:04:31.741 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:31.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:31.486619+0000) 2022-09-08T09:04:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:31 smithi055 ceph-mon[97033]: pgmap v22269: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T09:04:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:31 smithi059 ceph-mon[97450]: pgmap v22269: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T09:04:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:31.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:31.738487+0000) 2022-09-08T09:04:32.741 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:32.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:32.486883+0000) 2022-09-08T09:04:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:32.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:32.738609+0000) 2022-09-08T09:04:33.741 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:33.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:33.487051+0000) 2022-09-08T09:04:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:33 smithi055 ceph-mon[97033]: pgmap v22270: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:33 smithi059 ceph-mon[97450]: pgmap v22270: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:34.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:33.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:33.738775+0000) 2022-09-08T09:04:34.741 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:34.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:34.487177+0000) 2022-09-08T09:04:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:34.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:34.738964+0000) 2022-09-08T09:04:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:35 smithi055 ceph-mon[97033]: pgmap v22271: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:35.741 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:35.485+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:35.487301+0000) 2022-09-08T09:04:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:35 smithi059 ceph-mon[97450]: pgmap v22271: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:35.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:35.739180+0000) 2022-09-08T09:04:36.741 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:36.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:36.487547+0000) 2022-09-08T09:04:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:36.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:36.739321+0000) 2022-09-08T09:04:37.742 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:37.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:37.487819+0000) 2022-09-08T09:04:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:37 smithi055 ceph-mon[97033]: pgmap v22272: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:04:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:37 smithi059 ceph-mon[97450]: pgmap v22272: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:04:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:37.737+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:37.739466+0000) 2022-09-08T09:04:38.742 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:38.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:38.488000+0000) 2022-09-08T09:04:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:38.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:38.739676+0000) 2022-09-08T09:04:39.742 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:39.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:39.488140+0000) 2022-09-08T09:04:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:39 smithi055 ceph-mon[97033]: pgmap v22273: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:39 smithi059 ceph-mon[97450]: pgmap v22273: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:39.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:39.739849+0000) 2022-09-08T09:04:40.742 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:40.486+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:40.488295+0000) 2022-09-08T09:04:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:40.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:40.739990+0000) 2022-09-08T09:04:41.742 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:41.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:41.488470+0000) 2022-09-08T09:04:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:41 smithi055 ceph-mon[97033]: pgmap v22274: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:04:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:41 smithi059 ceph-mon[97450]: pgmap v22274: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:04:42.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:04:41.904+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:04:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:41.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:41.740207+0000) 2022-09-08T09:04:42.743 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:42.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:42.488674+0000) 2022-09-08T09:04:43.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:42.738+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:42.740402+0000) 2022-09-08T09:04:43.743 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:43.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:43.488883+0000) 2022-09-08T09:04:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:43 smithi055 ceph-mon[97033]: pgmap v22275: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:43 smithi059 ceph-mon[97450]: pgmap v22275: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:43.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:43.740675+0000) 2022-09-08T09:04:44.743 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:44.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:44.489130+0000) 2022-09-08T09:04:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:44.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:44.740930+0000) 2022-09-08T09:04:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:45 smithi055 ceph-mon[97033]: pgmap v22276: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:45.743 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:45.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:45.489286+0000) 2022-09-08T09:04:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:45 smithi059 ceph-mon[97450]: pgmap v22276: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:04:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:45.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:45.741094+0000) 2022-09-08T09:04:46.743 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:46.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:46.489468+0000) 2022-09-08T09:04:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:46.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:46.741297+0000) 2022-09-08T09:04:47.744 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:47.487+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:47.489606+0000) 2022-09-08T09:04:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:47 smithi055 ceph-mon[97033]: pgmap v22277: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:04:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:47 smithi059 ceph-mon[97450]: pgmap v22277: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:04:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:47.739+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:47.741427+0000) 2022-09-08T09:04:48.744 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:48.488+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:48.489754+0000) 2022-09-08T09:04:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:48.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:48.741708+0000) 2022-09-08T09:04:49.744 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:49.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:49.490000+0000) 2022-09-08T09:04:49.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:49 smithi055 ceph-mon[97033]: pgmap v22278: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T09:04:49.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:04:49.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:49 smithi055 ceph-mon[97033]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T09:04:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:49 smithi059 ceph-mon[97450]: pgmap v22278: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T09:04:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:04:50.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:49 smithi059 ceph-mon[97450]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2022-09-08T09:04:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:49.740+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:49.741883+0000) 2022-09-08T09:04:50.744 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:50.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:50.490237+0000) 2022-09-08T09:04:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:50.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:50.742118+0000) 2022-09-08T09:04:51.744 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:51.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:51.490397+0000) 2022-09-08T09:04:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:51 smithi055 ceph-mon[97033]: pgmap v22279: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T09:04:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:51 smithi059 ceph-mon[97450]: pgmap v22279: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T09:04:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:51.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:51.742320+0000) 2022-09-08T09:04:52.745 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:52.489+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:52.490594+0000) 2022-09-08T09:04:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:52.741+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:52.742554+0000) 2022-09-08T09:04:53.745 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:53.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:53.490768+0000) 2022-09-08T09:04:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:53 smithi055 ceph-mon[97033]: pgmap v22280: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T09:04:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:53 smithi059 ceph-mon[97450]: pgmap v22280: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T09:04:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:53.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:53.742735+0000) 2022-09-08T09:04:54.745 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:54.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:54.490896+0000) 2022-09-08T09:04:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:54.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:54.742961+0000) 2022-09-08T09:04:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:55 smithi055 ceph-mon[97033]: pgmap v22281: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T09:04:55.745 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:55.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:55.491091+0000) 2022-09-08T09:04:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:55 smithi059 ceph-mon[97450]: pgmap v22281: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T09:04:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:55.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:55.743158+0000) 2022-09-08T09:04:56.745 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:56.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:56.491287+0000) 2022-09-08T09:04:57.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:04:56.916+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:04:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:56.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:56.743360+0000) 2022-09-08T09:04:57.746 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:57.490+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:57.491414+0000) 2022-09-08T09:04:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:57 smithi055 ceph-mon[97033]: pgmap v22282: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T09:04:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:57 smithi059 ceph-mon[97450]: pgmap v22282: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T09:04:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:57.742+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:57.743603+0000) 2022-09-08T09:04:58.746 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:58.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:58.491632+0000) 2022-09-08T09:04:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:58.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:58.743891+0000) 2022-09-08T09:04:59.746 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:04:59.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:59.491792+0000) 2022-09-08T09:04:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:04:59 smithi055 ceph-mon[97033]: pgmap v22283: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T09:05:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:04:59 smithi059 ceph-mon[97450]: pgmap v22283: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-09-08T09:05:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:04:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:04:59.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:04:59.744096+0000) 2022-09-08T09:05:00.746 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:00.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:00.491916+0000) 2022-09-08T09:05:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:00.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:00.744277+0000) 2022-09-08T09:05:01.746 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:01.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:01.492092+0000) 2022-09-08T09:05:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:01 smithi055 ceph-mon[97033]: pgmap v22284: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:05:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:01 smithi059 ceph-mon[97450]: pgmap v22284: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:05:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:01.743+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:01.744489+0000) 2022-09-08T09:05:02.747 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:02.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:02.492296+0000) 2022-09-08T09:05:03.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:02.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:02.744745+0000) 2022-09-08T09:05:03.747 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:03.491+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:03.492453+0000) 2022-09-08T09:05:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:03 smithi055 ceph-mon[97033]: pgmap v22285: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:03 smithi059 ceph-mon[97450]: pgmap v22285: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:03.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:03.745002+0000) 2022-09-08T09:05:04.747 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:04.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:04.492672+0000) 2022-09-08T09:05:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:04.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:04.745216+0000) 2022-09-08T09:05:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:05 smithi055 ceph-mon[97033]: pgmap v22286: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:05.747 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:05.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:05.492799+0000) 2022-09-08T09:05:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:05 smithi059 ceph-mon[97450]: pgmap v22286: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:05.744+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:05.745367+0000) 2022-09-08T09:05:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:05:06.747 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:06.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:06.492999+0000) 2022-09-08T09:05:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:05:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:06.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:06.745565+0000) 2022-09-08T09:05:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:07 smithi055 ceph-mon[97033]: pgmap v22287: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:05:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:05:07.748 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:07.492+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:07.493218+0000) 2022-09-08T09:05:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:07 smithi059 ceph-mon[97450]: pgmap v22287: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:05:07.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:05:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:07.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:07.745820+0000) 2022-09-08T09:05:08.748 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:08.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:08.493437+0000) 2022-09-08T09:05:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:08.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:08.746071+0000) 2022-09-08T09:05:09.748 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:09.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:09.493648+0000) 2022-09-08T09:05:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:09 smithi055 ceph-mon[97033]: pgmap v22288: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:05:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:09 smithi059 ceph-mon[97450]: pgmap v22288: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:05:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:09.745+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:09.746294+0000) 2022-09-08T09:05:10.749 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:10.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:10.493792+0000) 2022-09-08T09:05:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:10 smithi055 ceph-mon[97033]: osdmap e1470: 12 total, 11 up, 11 in 2022-09-08T09:05:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:10 smithi059 ceph-mon[97450]: osdmap e1470: 12 total, 11 up, 11 in 2022-09-08T09:05:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:10.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:10.746486+0000) 2022-09-08T09:05:11.749 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:11.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:11.493934+0000) 2022-09-08T09:05:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:11 smithi055 ceph-mon[97033]: pgmap v22290: 129 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:05:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:11 smithi059 ceph-mon[97450]: pgmap v22290: 129 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:05:12.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:05:11.922+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:05:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:11.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:11.746650+0000) 2022-09-08T09:05:12.749 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:12.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:12.494173+0000) 2022-09-08T09:05:13.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:12.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:12.746775+0000) 2022-09-08T09:05:13.749 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:13.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:13.494383+0000) 2022-09-08T09:05:13.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:13 smithi055 ceph-mon[97033]: pgmap v22291: 129 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:05:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:13 smithi059 ceph-mon[97450]: pgmap v22291: 129 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:05:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:13.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:13.746900+0000) 2022-09-08T09:05:14.749 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:14.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:14.494580+0000) 2022-09-08T09:05:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:14.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:14.747050+0000) 2022-09-08T09:05:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:15 smithi055 ceph-mon[97033]: osdmap e1471: 12 total, 11 up, 11 in 2022-09-08T09:05:15.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:15 smithi055 ceph-mon[97033]: pgmap v22293: 129 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T09:05:15.749 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:15.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:15.494852+0000) 2022-09-08T09:05:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:15 smithi059 ceph-mon[97450]: osdmap e1471: 12 total, 11 up, 11 in 2022-09-08T09:05:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:15 smithi059 ceph-mon[97450]: pgmap v22293: 129 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 122 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T09:05:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:15.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:15.747268+0000) 2022-09-08T09:05:16.750 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:16.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:16.495018+0000) 2022-09-08T09:05:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:16.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:16.747480+0000) 2022-09-08T09:05:17.750 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:17.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:17.495157+0000) 2022-09-08T09:05:17.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:17 smithi055 ceph-mon[97033]: pgmap v22294: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T09:05:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:17 smithi059 ceph-mon[97450]: pgmap v22294: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T09:05:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:17.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:17.747616+0000) 2022-09-08T09:05:18.750 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:18.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:18.495290+0000) 2022-09-08T09:05:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:18.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:18.747782+0000) 2022-09-08T09:05:19.750 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:19.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:19.495405+0000) 2022-09-08T09:05:19.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:19 smithi055 ceph-mon[97033]: pgmap v22295: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.5 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T09:05:19.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:19 smithi055 ceph-mon[97033]: osdmap e1472: 12 total, 11 up, 11 in 2022-09-08T09:05:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:19 smithi059 ceph-mon[97450]: pgmap v22295: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.5 KiB/s rd, 229 B/s wr, 7 op/s 2022-09-08T09:05:20.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:19 smithi059 ceph-mon[97450]: osdmap e1472: 12 total, 11 up, 11 in 2022-09-08T09:05:20.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:19.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:19.747933+0000) 2022-09-08T09:05:20.750 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:20.493+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:20.495536+0000) 2022-09-08T09:05:21.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:20.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:20.748081+0000) 2022-09-08T09:05:21.750 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:21.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:21.495682+0000) 2022-09-08T09:05:21.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:21 smithi055 ceph-mon[97033]: pgmap v22297: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T09:05:22.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:21 smithi059 ceph-mon[97450]: pgmap v22297: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-08T09:05:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:21.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:21.748294+0000) 2022-09-08T09:05:22.751 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:22.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:22.495828+0000) 2022-09-08T09:05:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:22.746+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:22.748550+0000) 2022-09-08T09:05:23.751 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:23.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:23.495950+0000) 2022-09-08T09:05:23.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:23 smithi055 ceph-mon[97033]: pgmap v22298: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 123 B/s wr, 7 op/s 2022-09-08T09:05:24.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:23 smithi059 ceph-mon[97450]: pgmap v22298: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 123 B/s wr, 7 op/s 2022-09-08T09:05:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:23.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:23.748757+0000) 2022-09-08T09:05:24.751 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:24.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:24.496161+0000) 2022-09-08T09:05:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:24.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:24.748952+0000) 2022-09-08T09:05:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:25 smithi055 ceph-mon[97033]: pgmap v22299: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:05:25.751 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:25.494+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:25.496345+0000) 2022-09-08T09:05:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:25 smithi059 ceph-mon[97450]: pgmap v22299: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:05:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:25.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:25.749229+0000) 2022-09-08T09:05:26.752 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:26.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:26.496485+0000) 2022-09-08T09:05:27.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:05:26.922+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:05:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:26.747+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:26.749469+0000) 2022-09-08T09:05:27.752 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:27.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:27.496696+0000) 2022-09-08T09:05:27.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:27 smithi055 ceph-mon[97033]: pgmap v22300: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:05:28.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:27 smithi059 ceph-mon[97450]: pgmap v22300: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:05:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:27.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:27.749730+0000) 2022-09-08T09:05:28.752 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:28.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:28.496850+0000) 2022-09-08T09:05:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:28.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:28.749893+0000) 2022-09-08T09:05:29.753 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:29.495+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:29.496983+0000) 2022-09-08T09:05:29.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:29 smithi055 ceph-mon[97033]: pgmap v22301: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:05:30.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:29 smithi059 ceph-mon[97450]: pgmap v22301: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:05:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:29.748+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:29.750058+0000) 2022-09-08T09:05:30.753 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:30.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:30.497165+0000) 2022-09-08T09:05:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:30.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:30.750328+0000) 2022-09-08T09:05:31.753 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:31.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:31.497350+0000) 2022-09-08T09:05:31.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:31 smithi055 ceph-mon[97033]: pgmap v22302: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T09:05:32.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:31 smithi059 ceph-mon[97450]: pgmap v22302: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T09:05:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:31.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:31.750546+0000) 2022-09-08T09:05:32.753 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:32.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:32.497547+0000) 2022-09-08T09:05:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:32.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:32.750813+0000) 2022-09-08T09:05:33.753 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:33.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:33.497752+0000) 2022-09-08T09:05:33.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:33 smithi055 ceph-mon[97033]: pgmap v22303: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:34.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:33 smithi059 ceph-mon[97450]: pgmap v22303: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:33.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:33.751051+0000) 2022-09-08T09:05:34.753 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:34.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:34.497899+0000) 2022-09-08T09:05:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:34.749+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:34.751264+0000) 2022-09-08T09:05:35.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:35 smithi055 ceph-mon[97033]: pgmap v22304: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:35.754 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:35.496+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:35.498074+0000) 2022-09-08T09:05:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:35 smithi059 ceph-mon[97450]: pgmap v22304: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:36.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:35.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:35.751477+0000) 2022-09-08T09:05:36.754 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:36.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:36.498216+0000) 2022-09-08T09:05:37.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:36.750+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:36.751730+0000) 2022-09-08T09:05:37.754 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:37.497+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:37.498421+0000) 2022-09-08T09:05:37.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:37 smithi055 ceph-mon[97033]: pgmap v22305: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:05:38.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:37 smithi059 ceph-mon[97450]: pgmap v22305: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:05:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:37.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:37.751912+0000) 2022-09-08T09:05:38.754 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:38.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:38.498662+0000) 2022-09-08T09:05:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:38.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:38.752126+0000) 2022-09-08T09:05:39.754 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:39.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:39.498788+0000) 2022-09-08T09:05:39.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:39 smithi055 ceph-mon[97033]: pgmap v22306: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:40.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:39 smithi059 ceph-mon[97450]: pgmap v22306: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:39.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:39.752342+0000) 2022-09-08T09:05:40.755 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:40.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:40.498960+0000) 2022-09-08T09:05:41.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:40.751+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:40.752564+0000) 2022-09-08T09:05:41.755 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:41.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:41.499135+0000) 2022-09-08T09:05:41.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:41 smithi055 ceph-mon[97033]: pgmap v22307: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:05:42.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:41 smithi059 ceph-mon[97450]: pgmap v22307: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:05:42.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:05:41.929+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:05:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:41.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:41.752805+0000) 2022-09-08T09:05:42.755 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:42.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:42.499308+0000) 2022-09-08T09:05:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:42.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:42.753021+0000) 2022-09-08T09:05:43.755 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:43.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:43.499428+0000) 2022-09-08T09:05:43.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:43 smithi055 ceph-mon[97033]: pgmap v22308: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:44.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:43 smithi059 ceph-mon[97450]: pgmap v22308: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:43.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:43.753144+0000) 2022-09-08T09:05:44.756 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:44.498+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:44.499580+0000) 2022-09-08T09:05:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:44.752+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:44.753378+0000) 2022-09-08T09:05:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:45 smithi055 ceph-mon[97033]: pgmap v22309: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:45.756 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:45.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:45.499752+0000) 2022-09-08T09:05:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:45 smithi059 ceph-mon[97450]: pgmap v22309: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:45.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:45.753617+0000) 2022-09-08T09:05:46.756 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:46.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:46.499883+0000) 2022-09-08T09:05:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:46.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:46.753903+0000) 2022-09-08T09:05:47.756 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:47.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:47.500037+0000) 2022-09-08T09:05:47.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:47 smithi055 ceph-mon[97033]: pgmap v22310: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:05:48.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:47 smithi059 ceph-mon[97450]: pgmap v22310: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:05:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:47.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:47.754083+0000) 2022-09-08T09:05:48.756 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:48.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:48.500314+0000) 2022-09-08T09:05:49.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:48.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:48.754255+0000) 2022-09-08T09:05:49.757 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:49.499+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:49.500563+0000) 2022-09-08T09:05:49.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:49 smithi055 ceph-mon[97033]: pgmap v22311: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:49.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T09:05:49.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T09:05:49.982 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:49 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T09:05:50.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:49 smithi059 ceph-mon[97450]: pgmap v22311: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:50.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T09:05:50.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T09:05:50.033 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:49 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T09:05:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:49.753+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:49.754417+0000) 2022-09-08T09:05:50.757 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:50.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:50.500750+0000) 2022-09-08T09:05:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:50.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:50.754606+0000) 2022-09-08T09:05:51.757 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:51.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:51.500872+0000) 2022-09-08T09:05:51.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:51 smithi055 ceph-mon[97033]: pgmap v22312: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:05:52.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:51 smithi059 ceph-mon[97450]: pgmap v22312: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:05:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:51.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:51.754799+0000) 2022-09-08T09:05:52.757 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:52.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:52.501007+0000) 2022-09-08T09:05:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:52.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:52.754970+0000) 2022-09-08T09:05:53.757 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:53.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:53.501218+0000) 2022-09-08T09:05:53.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:53 smithi055 ceph-mon[97033]: pgmap v22313: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:54.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:53 smithi059 ceph-mon[97450]: pgmap v22313: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:53.754+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:53.755178+0000) 2022-09-08T09:05:54.758 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:54.500+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:54.501425+0000) 2022-09-08T09:05:55.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:54.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:54.755403+0000) 2022-09-08T09:05:55.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:55 smithi055 ceph-mon[97033]: pgmap v22314: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:55.758 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:55.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:55.501588+0000) 2022-09-08T09:05:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:55 smithi059 ceph-mon[97450]: pgmap v22314: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:05:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:55.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:55.755676+0000) 2022-09-08T09:05:56.758 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:56.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:56.501771+0000) 2022-09-08T09:05:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:56.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:56.755942+0000) 2022-09-08T09:05:57.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:05:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:05:56.942+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:05:57.758 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:57.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:57.501902+0000) 2022-09-08T09:05:57.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:57 smithi055 ceph-mon[97033]: pgmap v22315: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:05:58.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:57 smithi059 ceph-mon[97450]: pgmap v22315: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:05:58.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:57.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:57.756190+0000) 2022-09-08T09:05:58.759 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:58.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:58.502109+0000) 2022-09-08T09:05:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:58.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:58.756446+0000) 2022-09-08T09:05:59.759 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:05:59.501+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:59.502299+0000) 2022-09-08T09:05:59.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:05:59 smithi055 ceph-mon[97033]: pgmap v22316: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:00.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:05:59 smithi059 ceph-mon[97450]: pgmap v22316: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:05:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:05:59.755+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:05:59.756602+0000) 2022-09-08T09:06:00.759 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:00.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:00.502544+0000) 2022-09-08T09:06:01.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:00.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:00.756729+0000) 2022-09-08T09:06:01.759 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:01.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:01.502791+0000) 2022-09-08T09:06:01.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:01 smithi055 ceph-mon[97033]: pgmap v22317: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:06:02.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:01 smithi059 ceph-mon[97450]: pgmap v22317: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:06:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:01.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:01.756869+0000) 2022-09-08T09:06:02.759 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:02.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:02.502940+0000) 2022-09-08T09:06:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:02.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:02.757087+0000) 2022-09-08T09:06:03.759 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:03.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:03.503114+0000) 2022-09-08T09:06:03.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:03 smithi055 ceph-mon[97033]: pgmap v22318: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:04.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:03 smithi059 ceph-mon[97450]: pgmap v22318: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:03.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:03.757307+0000) 2022-09-08T09:06:04.760 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:04.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:04.503346+0000) 2022-09-08T09:06:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:04.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:04.757469+0000) 2022-09-08T09:06:05.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:05 smithi055 ceph-mon[97033]: pgmap v22319: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:05.760 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:05.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:05.503564+0000) 2022-09-08T09:06:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:05 smithi059 ceph-mon[97450]: pgmap v22319: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:05.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:05.757608+0000) 2022-09-08T09:06:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:06:06.760 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:06.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:06.503785+0000) 2022-09-08T09:06:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:06:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:06.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:06.757761+0000) 2022-09-08T09:06:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:07 smithi055 ceph-mon[97033]: pgmap v22320: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:06:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:06:07.760 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:07.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:07.504005+0000) 2022-09-08T09:06:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:07 smithi059 ceph-mon[97450]: pgmap v22320: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:06:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:06:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:07.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:07.757997+0000) 2022-09-08T09:06:08.760 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:08.502+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:08.504193+0000) 2022-09-08T09:06:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:08.756+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:08.758241+0000) 2022-09-08T09:06:09.761 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:09.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:09.504368+0000) 2022-09-08T09:06:09.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:09 smithi055 ceph-mon[97033]: pgmap v22321: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:06:10.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:09 smithi059 ceph-mon[97450]: pgmap v22321: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:06:10.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:09.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:09.758450+0000) 2022-09-08T09:06:10.761 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:10.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:10.504552+0000) 2022-09-08T09:06:10.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:10 smithi055 ceph-mon[97033]: osdmap e1473: 12 total, 11 up, 11 in 2022-09-08T09:06:11.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:10 smithi059 ceph-mon[97450]: osdmap e1473: 12 total, 11 up, 11 in 2022-09-08T09:06:11.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:10.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:10.758740+0000) 2022-09-08T09:06:11.761 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:11.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:11.504715+0000) 2022-09-08T09:06:11.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:11 smithi055 ceph-mon[97033]: pgmap v22323: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:06:12.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:11 smithi059 ceph-mon[97450]: pgmap v22323: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:06:12.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:06:11.955+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:06:12.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:11.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:11.758983+0000) 2022-09-08T09:06:12.761 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:12.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:12.504883+0000) 2022-09-08T09:06:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:12.757+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:12.759229+0000) 2022-09-08T09:06:13.762 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:13.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:13.505115+0000) 2022-09-08T09:06:13.980 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:13 smithi055 ceph-mon[97033]: pgmap v22324: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:06:14.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:13 smithi059 ceph-mon[97450]: pgmap v22324: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:06:14.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:13.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:13.759463+0000) 2022-09-08T09:06:14.762 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:14.503+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:14.505264+0000) 2022-09-08T09:06:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:14.758+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:14.759719+0000) 2022-09-08T09:06:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:15 smithi055 ceph-mon[97033]: osdmap e1474: 12 total, 11 up, 11 in 2022-09-08T09:06:15.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:15 smithi055 ceph-mon[97033]: pgmap v22326: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T09:06:15.762 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:15.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:15.505471+0000) 2022-09-08T09:06:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:15 smithi059 ceph-mon[97450]: osdmap e1474: 12 total, 11 up, 11 in 2022-09-08T09:06:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:15 smithi059 ceph-mon[97450]: pgmap v22326: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2022-09-08T09:06:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:15.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:15.759961+0000) 2022-09-08T09:06:16.762 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:16.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:16.505685+0000) 2022-09-08T09:06:17.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:16.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:16.760163+0000) 2022-09-08T09:06:17.763 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:17.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:17.505909+0000) 2022-09-08T09:06:17.981 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:17 smithi055 ceph-mon[97033]: pgmap v22327: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T09:06:18.032 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:17 smithi059 ceph-mon[97450]: pgmap v22327: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T09:06:18.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:17.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:17.760409+0000) 2022-09-08T09:06:18.763 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:18.505+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:18.506138+0000) 2022-09-08T09:06:19.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:18.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:18.760738+0000) 2022-09-08T09:06:19.763 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:19.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:19.506273+0000) 2022-09-08T09:06:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:19.759+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:19.760943+0000) 2022-09-08T09:06:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:19 smithi055 ceph-mon[97033]: pgmap v22328: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.5 KiB/s rd, 228 B/s wr, 6 op/s 2022-09-08T09:06:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:19 smithi055 ceph-mon[97033]: pgmap v22329: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 124 B/s wr, 3 op/s 2022-09-08T09:06:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:19 smithi055 ceph-mon[97033]: pgmap v22330: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 163 B/s wr, 4 op/s 2022-09-08T09:06:20.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:06:20.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:19 smithi055 ceph-mon[97033]: osdmap e1475: 12 total, 11 up, 11 in 2022-09-08T09:06:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:19 smithi059 ceph-mon[97450]: pgmap v22328: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.5 KiB/s rd, 228 B/s wr, 6 op/s 2022-09-08T09:06:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:19 smithi059 ceph-mon[97450]: pgmap v22329: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 124 B/s wr, 3 op/s 2022-09-08T09:06:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:19 smithi059 ceph-mon[97450]: pgmap v22330: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 163 B/s wr, 4 op/s 2022-09-08T09:06:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:06:20.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:19 smithi059 ceph-mon[97450]: osdmap e1475: 12 total, 11 up, 11 in 2022-09-08T09:06:20.763 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:20.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:20.506425+0000) 2022-09-08T09:06:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:20.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:20.761104+0000) 2022-09-08T09:06:21.763 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:21.504+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:21.506571+0000) 2022-09-08T09:06:22.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:21.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:21.761349+0000) 2022-09-08T09:06:22.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:21 smithi055 ceph-mon[97033]: pgmap v22332: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.8 KiB/s rd, 163 B/s wr, 9 op/s 2022-09-08T09:06:22.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:21 smithi059 ceph-mon[97450]: pgmap v22332: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.8 KiB/s rd, 163 B/s wr, 9 op/s 2022-09-08T09:06:22.764 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:22.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:22.506784+0000) 2022-09-08T09:06:23.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:22.760+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:22.761572+0000) 2022-09-08T09:06:23.764 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:23.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:23.506937+0000) 2022-09-08T09:06:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:23.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:23.761791+0000) 2022-09-08T09:06:24.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:23 smithi055 ceph-mon[97033]: pgmap v22333: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T09:06:24.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:23 smithi059 ceph-mon[97450]: pgmap v22333: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T09:06:24.764 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:24.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:24.507065+0000) 2022-09-08T09:06:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:24.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:24.761929+0000) 2022-09-08T09:06:25.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:25 smithi055 ceph-mon[97033]: pgmap v22334: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T09:06:25.764 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:25.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:25.507264+0000) 2022-09-08T09:06:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:25 smithi059 ceph-mon[97450]: pgmap v22334: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-08T09:06:26.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:25.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:25.762142+0000) 2022-09-08T09:06:26.764 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:26.506+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:26.507469+0000) 2022-09-08T09:06:27.153 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:06:26.956+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:06:27.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:26.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:26.762342+0000) 2022-09-08T09:06:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:26 smithi055 ceph-mon[97033]: pgmap v22335: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:06:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:26 smithi059 ceph-mon[97450]: pgmap v22335: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:06:27.765 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:27.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:27.507659+0000) 2022-09-08T09:06:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:27.761+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:27.762556+0000) 2022-09-08T09:06:28.765 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:28.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:28.507859+0000) 2022-09-08T09:06:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:28.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:28.762767+0000) 2022-09-08T09:06:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:28 smithi055 ceph-mon[97033]: pgmap v22336: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:06:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:28 smithi059 ceph-mon[97450]: pgmap v22336: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:06:29.765 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:29.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:29.508048+0000) 2022-09-08T09:06:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:29.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:29.762902+0000) 2022-09-08T09:06:30.765 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:30.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:30.508301+0000) 2022-09-08T09:06:31.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:30.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:30.763098+0000) 2022-09-08T09:06:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:30 smithi055 ceph-mon[97033]: pgmap v22337: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T09:06:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:30 smithi059 ceph-mon[97450]: pgmap v22337: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T09:06:31.765 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:31.507+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:31.508466+0000) 2022-09-08T09:06:32.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:31.762+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:31.763297+0000) 2022-09-08T09:06:32.765 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:32.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:32.508628+0000) 2022-09-08T09:06:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:32.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:32.763471+0000) 2022-09-08T09:06:33.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:32 smithi055 ceph-mon[97033]: pgmap v22338: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:32 smithi059 ceph-mon[97450]: pgmap v22338: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:33.766 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:33.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:33.508824+0000) 2022-09-08T09:06:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:33.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:33.763734+0000) 2022-09-08T09:06:34.766 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:34.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:34.509023+0000) 2022-09-08T09:06:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:34.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:34.763908+0000) 2022-09-08T09:06:35.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:35 smithi055 ceph-mon[97033]: pgmap v22339: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T09:06:35.766 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:35.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:35.509244+0000) 2022-09-08T09:06:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:35 smithi059 ceph-mon[97450]: pgmap v22339: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T09:06:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:35.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:35.764141+0000) 2022-09-08T09:06:36.767 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:36.508+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:36.509487+0000) 2022-09-08T09:06:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:36.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:36.764389+0000) 2022-09-08T09:06:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:36 smithi055 ceph-mon[97033]: pgmap v22340: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T09:06:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:36 smithi059 ceph-mon[97450]: pgmap v22340: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T09:06:37.767 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:37.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:37.509693+0000) 2022-09-08T09:06:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:37.763+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:37.764607+0000) 2022-09-08T09:06:38.767 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:38.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:38.509832+0000) 2022-09-08T09:06:39.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:38.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:38.764859+0000) 2022-09-08T09:06:39.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:38 smithi055 ceph-mon[97033]: pgmap v22341: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:38 smithi059 ceph-mon[97450]: pgmap v22341: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:39.767 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:39.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:39.509967+0000) 2022-09-08T09:06:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:39.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:39.765128+0000) 2022-09-08T09:06:40.767 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:40.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:40.510104+0000) 2022-09-08T09:06:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:40.764+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:40.765318+0000) 2022-09-08T09:06:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:40 smithi055 ceph-mon[97033]: pgmap v22342: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:06:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:40 smithi059 ceph-mon[97450]: pgmap v22342: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:06:41.768 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:41.509+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:41.510331+0000) 2022-09-08T09:06:42.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:06:41.967+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:06:42.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:41.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:41.765551+0000) 2022-09-08T09:06:42.768 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:42.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:42.510566+0000) 2022-09-08T09:06:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:42.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:42.765822+0000) 2022-09-08T09:06:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:42 smithi055 ceph-mon[97033]: pgmap v22343: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:42 smithi059 ceph-mon[97450]: pgmap v22343: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:43.768 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:43.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:43.510783+0000) 2022-09-08T09:06:44.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:43.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:43.766026+0000) 2022-09-08T09:06:44.768 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:44.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:44.510972+0000) 2022-09-08T09:06:45.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:44.765+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:44.766183+0000) 2022-09-08T09:06:45.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:45 smithi055 ceph-mon[97033]: pgmap v22344: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T09:06:45.768 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:45.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:45.511163+0000) 2022-09-08T09:06:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:45 smithi059 ceph-mon[97450]: pgmap v22344: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T09:06:46.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:45.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:45.766407+0000) 2022-09-08T09:06:46.769 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:46.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:46.511379+0000) 2022-09-08T09:06:47.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:46.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:46.766555+0000) 2022-09-08T09:06:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:46 smithi055 ceph-mon[97033]: pgmap v22345: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T09:06:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:46 smithi059 ceph-mon[97450]: pgmap v22345: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T09:06:47.769 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:47.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:47.511531+0000) 2022-09-08T09:06:48.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:47.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:47.766760+0000) 2022-09-08T09:06:48.769 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:48.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:48.511703+0000) 2022-09-08T09:06:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:48.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:48.766974+0000) 2022-09-08T09:06:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:48 smithi055 ceph-mon[97033]: pgmap v22346: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:49.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:48 smithi059 ceph-mon[97450]: pgmap v22346: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:49.769 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:49.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:49.511920+0000) 2022-09-08T09:06:50.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:49.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:49.767149+0000) 2022-09-08T09:06:50.770 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:50.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:50.512177+0000) 2022-09-08T09:06:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:50.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:50.767375+0000) 2022-09-08T09:06:51.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:50 smithi055 ceph-mon[97033]: pgmap v22347: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:06:51.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:50 smithi059 ceph-mon[97450]: pgmap v22347: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:06:51.770 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:51.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:51.512358+0000) 2022-09-08T09:06:52.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:51.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:51.767619+0000) 2022-09-08T09:06:52.770 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:52.510+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:52.512604+0000) 2022-09-08T09:06:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:52.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:52.767804+0000) 2022-09-08T09:06:53.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:52 smithi055 ceph-mon[97033]: pgmap v22348: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:53.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:52 smithi059 ceph-mon[97450]: pgmap v22348: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:53.770 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:53.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:53.512855+0000) 2022-09-08T09:06:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:53.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:53.767945+0000) 2022-09-08T09:06:54.770 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:54.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:54.513083+0000) 2022-09-08T09:06:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:54.766+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:54.768097+0000) 2022-09-08T09:06:55.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:55 smithi055 ceph-mon[97033]: pgmap v22349: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T09:06:55.771 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:55.511+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:55.513208+0000) 2022-09-08T09:06:55.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:55 smithi059 ceph-mon[97450]: pgmap v22349: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T09:06:56.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:55.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:55.768324+0000) 2022-09-08T09:06:56.771 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:56.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:56.513437+0000) 2022-09-08T09:06:57.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:06:56.982+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:06:57.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:56.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:56.768506+0000) 2022-09-08T09:06:57.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:56 smithi055 ceph-mon[97033]: pgmap v22350: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T09:06:57.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:56 smithi059 ceph-mon[97450]: pgmap v22350: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T09:06:57.771 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:57.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:57.513628+0000) 2022-09-08T09:06:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:57.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:57.768692+0000) 2022-09-08T09:06:58.771 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:58.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:58.513881+0000) 2022-09-08T09:06:59.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:58.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:58.768811+0000) 2022-09-08T09:06:59.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:06:58 smithi055 ceph-mon[97033]: pgmap v22351: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:59.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:06:58 smithi059 ceph-mon[97450]: pgmap v22351: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:06:59.771 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:06:59.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:59.514100+0000) 2022-09-08T09:07:00.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:06:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:06:59.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:06:59.768996+0000) 2022-09-08T09:07:00.771 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:00.512+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:00.514294+0000) 2022-09-08T09:07:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:00.767+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:00.769126+0000) 2022-09-08T09:07:01.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:00 smithi055 ceph-mon[97033]: pgmap v22352: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:07:01.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:00 smithi059 ceph-mon[97450]: pgmap v22352: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:07:01.771 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:01.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:01.514524+0000) 2022-09-08T09:07:02.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:01.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:01.769325+0000) 2022-09-08T09:07:02.772 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:02.513+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:02.514786+0000) 2022-09-08T09:07:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:02.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:02.769539+0000) 2022-09-08T09:07:03.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:02 smithi055 ceph-mon[97033]: pgmap v22353: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:07:03.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:02 smithi059 ceph-mon[97450]: pgmap v22353: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:07:03.772 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:03.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:03.515025+0000) 2022-09-08T09:07:04.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:03.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:03.769735+0000) 2022-09-08T09:07:04.772 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:04.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:04.515260+0000) 2022-09-08T09:07:05.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:04.768+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:04.769895+0000) 2022-09-08T09:07:05.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:05 smithi055 ceph-mon[97033]: pgmap v22354: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 8 op/s 2022-09-08T09:07:05.772 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:05.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:05.515448+0000) 2022-09-08T09:07:05.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:05 smithi059 ceph-mon[97450]: pgmap v22354: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 8 op/s 2022-09-08T09:07:06.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:05.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:05.770105+0000) 2022-09-08T09:07:06.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:07:06.773 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:06.514+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:06.515681+0000) 2022-09-08T09:07:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:07:07.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:06.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:06.770357+0000) 2022-09-08T09:07:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:07 smithi055 ceph-mon[97033]: pgmap v22355: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T09:07:07.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:07:07.773 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:07.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:07.515878+0000) 2022-09-08T09:07:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:07 smithi059 ceph-mon[97450]: pgmap v22355: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 21 op/s 2022-09-08T09:07:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:07:08.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:07.769+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:07.770596+0000) 2022-09-08T09:07:08.773 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:08.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:08.516036+0000) 2022-09-08T09:07:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:08.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:08.770864+0000) 2022-09-08T09:07:09.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:08 smithi055 ceph-mon[97033]: pgmap v22356: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:07:09.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:08 smithi059 ceph-mon[97450]: pgmap v22356: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:07:09.773 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:09.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:09.516249+0000) 2022-09-08T09:07:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:09.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:09.771127+0000) 2022-09-08T09:07:10.774 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:10.515+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:10.516459+0000) 2022-09-08T09:07:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:10.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:10.771295+0000) 2022-09-08T09:07:11.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:10 smithi055 ceph-mon[97033]: osdmap e1476: 12 total, 11 up, 11 in 2022-09-08T09:07:11.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:10 smithi055 ceph-mon[97033]: pgmap v22358: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T09:07:11.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:10 smithi059 ceph-mon[97450]: osdmap e1476: 12 total, 11 up, 11 in 2022-09-08T09:07:11.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:10 smithi059 ceph-mon[97450]: pgmap v22358: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T09:07:11.774 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:11.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:11.516722+0000) 2022-09-08T09:07:12.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:07:11.999+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:07:12.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:11.770+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:11.771542+0000) 2022-09-08T09:07:12.774 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:12.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:12.516903+0000) 2022-09-08T09:07:13.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:12.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:12.771799+0000) 2022-09-08T09:07:13.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:12 smithi055 ceph-mon[97033]: pgmap v22359: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T09:07:13.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:12 smithi059 ceph-mon[97450]: pgmap v22359: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 14 KiB/s rd, 53 KiB/s wr, 24 op/s 2022-09-08T09:07:13.774 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:13.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:13.517159+0000) 2022-09-08T09:07:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:13.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:13.771937+0000) 2022-09-08T09:07:14.774 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:14.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:14.517373+0000) 2022-09-08T09:07:15.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:14.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:14.772167+0000) 2022-09-08T09:07:15.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:15 smithi055 ceph-mon[97033]: pgmap v22360: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-08T09:07:15.774 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:15.516+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:15.517581+0000) 2022-09-08T09:07:15.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:15 smithi059 ceph-mon[97450]: pgmap v22360: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2022-09-08T09:07:16.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:15.771+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:15.772328+0000) 2022-09-08T09:07:16.775 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:16.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:16.517753+0000) 2022-09-08T09:07:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:16.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:16.772570+0000) 2022-09-08T09:07:17.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:16 smithi055 ceph-mon[97033]: pgmap v22361: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:07:17.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:16 smithi059 ceph-mon[97450]: pgmap v22361: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:07:17.775 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:17.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:17.517879+0000) 2022-09-08T09:07:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:17.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:17.772841+0000) 2022-09-08T09:07:18.775 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:18.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:18.518121+0000) 2022-09-08T09:07:19.035 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:18.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:18.773083+0000) 2022-09-08T09:07:19.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:18 smithi055 ceph-mon[97033]: pgmap v22362: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:07:19.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:18 smithi059 ceph-mon[97450]: pgmap v22362: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2022-09-08T09:07:19.776 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:19.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:19.518330+0000) 2022-09-08T09:07:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:19.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:19.773341+0000) 2022-09-08T09:07:20.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T09:07:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T09:07:20.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:19 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T09:07:20.232 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:19 smithi055 ceph-mon[97033]: osdmap e1477: 12 total, 11 up, 11 in 2022-09-08T09:07:20.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-08T09:07:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-08T09:07:20.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:19 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-08T09:07:20.284 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:19 smithi059 ceph-mon[97450]: osdmap e1477: 12 total, 11 up, 11 in 2022-09-08T09:07:20.776 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:20.517+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:20.518458+0000) 2022-09-08T09:07:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:20.772+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:20.773525+0000) 2022-09-08T09:07:21.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:20 smithi055 ceph-mon[97033]: pgmap v22364: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:07:21.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:20 smithi059 ceph-mon[97450]: pgmap v22364: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:07:21.776 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:21.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:21.518713+0000) 2022-09-08T09:07:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:21.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:21.773763+0000) 2022-09-08T09:07:22.776 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:22.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:22.518890+0000) 2022-09-08T09:07:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:22.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:22.773941+0000) 2022-09-08T09:07:23.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:22 smithi055 ceph-mon[97033]: pgmap v22365: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:07:23.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:22 smithi059 ceph-mon[97450]: pgmap v22365: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:07:23.776 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:23.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:23.519032+0000) 2022-09-08T09:07:24.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:23.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:23.774193+0000) 2022-09-08T09:07:24.777 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:24.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:24.519230+0000) 2022-09-08T09:07:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:24.773+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:24.774465+0000) 2022-09-08T09:07:25.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:25 smithi055 ceph-mon[97033]: pgmap v22366: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:07:25.777 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:25.518+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:25.519434+0000) 2022-09-08T09:07:25.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:25 smithi059 ceph-mon[97450]: pgmap v22366: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:07:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:25.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:25.774706+0000) 2022-09-08T09:07:26.777 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:26.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:26.519667+0000) 2022-09-08T09:07:27.154 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:07:27.014+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:07:27.155 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:26.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:26.774924+0000) 2022-09-08T09:07:27.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:26 smithi055 ceph-mon[97033]: pgmap v22367: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:07:27.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:26 smithi059 ceph-mon[97450]: pgmap v22367: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:07:27.777 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:27.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:27.519857+0000) 2022-09-08T09:07:28.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:27.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:27.775172+0000) 2022-09-08T09:07:28.778 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:28.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:28.519988+0000) 2022-09-08T09:07:29.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:28.774+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:28.775367+0000) 2022-09-08T09:07:29.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:28 smithi055 ceph-mon[97033]: pgmap v22368: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:07:29.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:28 smithi059 ceph-mon[97450]: pgmap v22368: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:07:29.778 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:29.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:29.520138+0000) 2022-09-08T09:07:30.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:29.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:29.775615+0000) 2022-09-08T09:07:30.778 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:30.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:30.520373+0000) 2022-09-08T09:07:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:30.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:30.775913+0000) 2022-09-08T09:07:31.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:30 smithi055 ceph-mon[97033]: pgmap v22369: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T09:07:31.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:30 smithi059 ceph-mon[97450]: pgmap v22369: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-08T09:07:31.778 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:31.519+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:31.520562+0000) 2022-09-08T09:07:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:31.775+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:31.776245+0000) 2022-09-08T09:07:32.779 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:32.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:32.520800+0000) 2022-09-08T09:07:33.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:32.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:32.776460+0000) 2022-09-08T09:07:33.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:32 smithi055 ceph-mon[97033]: pgmap v22370: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T09:07:33.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:32 smithi059 ceph-mon[97450]: pgmap v22370: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-09-08T09:07:33.779 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:33.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:33.520949+0000) 2022-09-08T09:07:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:33.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:33.776735+0000) 2022-09-08T09:07:34.779 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:34.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:34.521156+0000) 2022-09-08T09:07:35.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:34.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:34.776986+0000) 2022-09-08T09:07:35.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:35 smithi055 ceph-mon[97033]: pgmap v22371: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-08T09:07:35.779 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:35.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:35.521418+0000) 2022-09-08T09:07:35.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:35 smithi059 ceph-mon[97450]: pgmap v22371: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 6 op/s 2022-09-08T09:07:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:35.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:35.777183+0000) 2022-09-08T09:07:36.780 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:36.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:36.521588+0000) 2022-09-08T09:07:37.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:36.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:36.777398+0000) 2022-09-08T09:07:37.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:36 smithi055 ceph-mon[97033]: pgmap v22372: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 6 op/s 2022-09-08T09:07:37.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:36 smithi059 ceph-mon[97450]: pgmap v22372: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.8 KiB/s rd, 6 op/s 2022-09-08T09:07:37.780 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:37.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:37.521772+0000) 2022-09-08T09:07:38.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:37.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:37.777657+0000) 2022-09-08T09:07:38.780 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:38.520+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:38.521934+0000) 2022-09-08T09:07:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:38.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:38.777799+0000) 2022-09-08T09:07:39.231 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:38 smithi055 ceph-mon[97033]: pgmap v22373: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:07:39.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:38 smithi059 ceph-mon[97450]: pgmap v22373: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:07:39.780 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:39.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:39.522136+0000) 2022-09-08T09:07:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:39.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:39.778008+0000) 2022-09-08T09:07:40.780 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:40.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:40.522321+0000) 2022-09-08T09:07:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:40.776+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:40.778225+0000) 2022-09-08T09:07:41.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:40 smithi055 ceph-mon[97033]: pgmap v22374: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:07:41.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:40 smithi059 ceph-mon[97450]: pgmap v22374: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:07:41.781 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:41.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:41.522534+0000) 2022-09-08T09:07:42.033 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:41.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:41.778420+0000) 2022-09-08T09:07:42.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:07:42.029+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:07:42.781 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:42.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:42.522713+0000) 2022-09-08T09:07:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:42.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:42.778587+0000) 2022-09-08T09:07:43.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:42 smithi055 ceph-mon[97033]: pgmap v22375: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:07:43.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:42 smithi059 ceph-mon[97450]: pgmap v22375: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:07:43.781 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:43.521+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:43.522879+0000) 2022-09-08T09:07:44.099 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:43.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:43.778818+0000) 2022-09-08T09:07:44.782 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:44.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:44.523059+0000) 2022-09-08T09:07:45.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:44 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:44.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:44.779025+0000) 2022-09-08T09:07:45.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:45 smithi055 ceph-mon[97033]: pgmap v22376: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T09:07:45.781 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:45.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:45.523240+0000) 2022-09-08T09:07:45.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:45 smithi059 ceph-mon[97450]: pgmap v22376: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 6 op/s 2022-09-08T09:07:46.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:45 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:45.777+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:45.779160+0000) 2022-09-08T09:07:46.781 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:46.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:46.523430+0000) 2022-09-08T09:07:47.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:46 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:46.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:46.779323+0000) 2022-09-08T09:07:47.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:46 smithi055 ceph-mon[97033]: pgmap v22377: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T09:07:47.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:46 smithi059 ceph-mon[97450]: pgmap v22377: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 6 op/s 2022-09-08T09:07:47.782 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:47.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:47.523605+0000) 2022-09-08T09:07:48.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:47 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:47.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:47.779555+0000) 2022-09-08T09:07:48.782 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:48.522+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:48.523824+0000) 2022-09-08T09:07:49.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:48 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:48.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:48.779750+0000) 2022-09-08T09:07:49.230 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:48 smithi055 ceph-mon[97033]: pgmap v22378: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:07:49.243 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:48 smithi059 ceph-mon[97450]: pgmap v22378: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:07:49.782 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:49.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:49.524038+0000) 2022-09-08T09:07:50.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:49 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:49.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:49.779950+0000) 2022-09-08T09:07:50.282 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:50 smithi059 ceph-mon[97450]: pgmap v22379: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T09:07:50.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:50 smithi059 ceph-mon[97450]: pgmap v22380: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T09:07:50.283 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:50 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:07:50.480 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:50 smithi055 ceph-mon[97033]: pgmap v22379: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-09-08T09:07:50.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:50 smithi055 ceph-mon[97033]: pgmap v22380: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.8 KiB/s rd, 3 op/s 2022-09-08T09:07:50.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:50 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' 2022-09-08T09:07:50.782 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:50.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:50.524261+0000) 2022-09-08T09:07:51.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:50 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:50.778+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:50.780084+0000) 2022-09-08T09:07:51.782 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:51.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:51.524479+0000) 2022-09-08T09:07:52.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:51 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:51.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:51.780243+0000) 2022-09-08T09:07:52.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:52 smithi055 ceph-mon[97033]: pgmap v22381: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T09:07:52.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:52 smithi059 ceph-mon[97450]: pgmap v22381: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T09:07:52.783 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:52.523+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:52.524656+0000) 2022-09-08T09:07:53.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:52 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:52.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:52.780459+0000) 2022-09-08T09:07:53.783 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:53.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:53.524897+0000) 2022-09-08T09:07:54.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:53 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:53.779+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:53.780617+0000) 2022-09-08T09:07:54.481 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:54 smithi055 ceph-mon[97033]: pgmap v22382: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-08T09:07:54.532 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:54 smithi059 ceph-mon[97450]: pgmap v22382: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.4 KiB/s rd, 5 op/s 2022-09-08T09:07:54.783 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:54.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:54.525072+0000) 2022-09-08T09:07:55.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:54 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:54.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:54.780856+0000) 2022-09-08T09:07:55.783 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:55.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:55.525262+0000) 2022-09-08T09:07:56.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:55 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:55.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:55.781104+0000) 2022-09-08T09:07:56.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:56 smithi055 ceph-mon[97033]: pgmap v22383: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T09:07:56.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:56 smithi059 ceph-mon[97450]: pgmap v22383: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T09:07:56.783 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:56.524+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:56.525419+0000) 2022-09-08T09:07:57.046 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:56 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:56.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:56.781326+0000) 2022-09-08T09:07:57.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:07:57.043+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:07:57.784 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:57.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:57.525646+0000) 2022-09-08T09:07:58.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:57 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:57.780+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:57.781549+0000) 2022-09-08T09:07:58.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:07:58 smithi055 ceph-mon[97033]: pgmap v22384: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T09:07:58.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:07:58 smithi059 ceph-mon[97450]: pgmap v22384: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-09-08T09:07:58.784 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:58.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:58.525784+0000) 2022-09-08T09:07:59.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:58 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:58.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:58.781792+0000) 2022-09-08T09:07:59.784 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:07:59.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:59.525914+0000) 2022-09-08T09:08:00.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:07:59 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:07:59.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:07:59.782061+0000) 2022-09-08T09:08:00.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:00 smithi055 ceph-mon[97033]: pgmap v22385: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:08:00.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:00 smithi059 ceph-mon[97450]: pgmap v22385: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-09-08T09:08:00.784 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:00.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:00.526048+0000) 2022-09-08T09:08:01.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:00 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:00.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:00.782294+0000) 2022-09-08T09:08:01.785 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:01.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:01.526275+0000) 2022-09-08T09:08:02.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:01 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:01.781+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:01.782541+0000) 2022-09-08T09:08:02.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:02 smithi055 ceph-mon[97033]: pgmap v22386: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:08:02.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:02 smithi059 ceph-mon[97450]: pgmap v22386: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:08:02.785 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:02.525+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:02.526475+0000) 2022-09-08T09:08:03.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:02 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:02.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:02.782838+0000) 2022-09-08T09:08:03.785 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:03.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:03.526727+0000) 2022-09-08T09:08:04.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:03 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:03.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:03.783102+0000) 2022-09-08T09:08:04.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:04 smithi055 ceph-mon[97033]: pgmap v22387: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:08:04.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:04 smithi059 ceph-mon[97450]: pgmap v22387: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:08:04.785 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:04.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:04.526919+0000) 2022-09-08T09:08:05.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:04 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:04.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:04.783262+0000) 2022-09-08T09:08:05.786 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:05.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:05.527101+0000) 2022-09-08T09:08:06.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:05 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:05.782+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:05.783487+0000) 2022-09-08T09:08:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:06 smithi055 ceph-mon[97033]: pgmap v22388: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:08:06.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:06 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:08:06.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:06 smithi059 ceph-mon[97450]: pgmap v22388: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 12 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-08T09:08:06.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:06 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-08T09:08:06.786 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:06.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:06.527298+0000) 2022-09-08T09:08:07.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:06 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:06.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:06.783734+0000) 2022-09-08T09:08:07.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:07 smithi055 ceph-mon[97033]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:08:07.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:07 smithi059 ceph-mon[97450]: from='mgr.25354 172.21.15.47:0/3214539250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-08T09:08:07.786 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:07.526+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:07.527501+0000) 2022-09-08T09:08:08.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:07 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:07.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:07.783948+0000) 2022-09-08T09:08:08.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:08 smithi055 ceph-mon[97033]: pgmap v22389: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:08:08.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:08 smithi059 ceph-mon[97450]: pgmap v22389: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:08:08.786 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:08.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:08.527718+0000) 2022-09-08T09:08:09.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:08 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:08.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:08.784144+0000) 2022-09-08T09:08:09.787 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:09.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:09.527902+0000) 2022-09-08T09:08:10.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:09 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:09.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:09.784300+0000) 2022-09-08T09:08:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:10 smithi055 ceph-mon[97033]: pgmap v22390: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:08:10.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:10 smithi055 ceph-mon[97033]: osdmap e1478: 12 total, 11 up, 11 in 2022-09-08T09:08:10.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:10 smithi059 ceph-mon[97450]: pgmap v22390: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2022-09-08T09:08:10.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:10 smithi059 ceph-mon[97450]: osdmap e1478: 12 total, 11 up, 11 in 2022-09-08T09:08:10.787 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:10.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:10.528116+0000) 2022-09-08T09:08:11.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:10 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:10.783+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:10.784516+0000) 2022-09-08T09:08:11.787 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:11.527+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:11.528278+0000) 2022-09-08T09:08:12.052 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:11 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:11.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:11.784730+0000) 2022-09-08T09:08:12.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:08:12.049+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:08:12.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:12 smithi055 ceph-mon[97033]: pgmap v22392: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:08:12.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:12 smithi059 ceph-mon[97450]: pgmap v22392: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:08:12.787 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:12.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:12.528434+0000) 2022-09-08T09:08:13.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:12 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:12.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:12.784924+0000) 2022-09-08T09:08:13.787 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:13.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:13.528690+0000) 2022-09-08T09:08:14.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:13 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:13.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:13.785122+0000) 2022-09-08T09:08:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:14 smithi055 ceph-mon[97033]: pgmap v22393: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:08:14.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:14 smithi055 ceph-mon[97033]: osdmap e1479: 12 total, 11 up, 11 in 2022-09-08T09:08:14.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:14 smithi059 ceph-mon[97450]: pgmap v22393: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 13 KiB/s rd, 53 KiB/s wr, 23 op/s 2022-09-08T09:08:14.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:14 smithi059 ceph-mon[97450]: osdmap e1479: 12 total, 11 up, 11 in 2022-09-08T09:08:14.787 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:14.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:14.528922+0000) 2022-09-08T09:08:15.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:14 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:14.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:14.785326+0000) 2022-09-08T09:08:15.788 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:15.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:15.529086+0000) 2022-09-08T09:08:16.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:15 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:15.784+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:15.785502+0000) 2022-09-08T09:08:16.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:16 smithi055 ceph-mon[97033]: pgmap v22395: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T09:08:16.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:16 smithi059 ceph-mon[97450]: pgmap v22395: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T09:08:16.788 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:16.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:16.529252+0000) 2022-09-08T09:08:17.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:16 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:16.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:16.785744+0000) 2022-09-08T09:08:17.788 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:17.528+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:17.529482+0000) 2022-09-08T09:08:18.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:17 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:17.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:17.785967+0000) 2022-09-08T09:08:18.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:18 smithi055 ceph-mon[97033]: pgmap v22396: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T09:08:18.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:18 smithi059 ceph-mon[97450]: pgmap v22396: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-08T09:08:18.788 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:18.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:18.529764+0000) 2022-09-08T09:08:19.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:18 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:18.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:18.786112+0000) 2022-09-08T09:08:19.789 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:19.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:19.529938+0000) 2022-09-08T09:08:20.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:19 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:19.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:19.786352+0000) 2022-09-08T09:08:20.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:20 smithi055 ceph-mon[97033]: pgmap v22397: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 210 B/s wr, 6 op/s 2022-09-08T09:08:20.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:20 smithi059 ceph-mon[97450]: pgmap v22397: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.3 KiB/s rd, 210 B/s wr, 6 op/s 2022-09-08T09:08:20.789 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:20.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:20.530122+0000) 2022-09-08T09:08:21.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:20 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:20.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:20.786592+0000) 2022-09-08T09:08:21.789 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:21.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:21.530288+0000) 2022-09-08T09:08:22.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:21 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:21.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:21.786836+0000) 2022-09-08T09:08:22.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:22 smithi055 ceph-mon[97033]: pgmap v22398: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:08:22.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:22 smithi059 ceph-mon[97450]: pgmap v22398: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:08:22.789 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:22.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:22.530480+0000) 2022-09-08T09:08:23.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:22 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:22.785+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:22.787062+0000) 2022-09-08T09:08:23.789 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:23.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:23.530752+0000) 2022-09-08T09:08:24.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:23 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:23.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:23.787311+0000) 2022-09-08T09:08:24.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:24 smithi055 ceph-mon[97033]: pgmap v22399: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:08:24.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:24 smithi059 ceph-mon[97450]: pgmap v22399: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2022-09-08T09:08:24.790 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:24.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:24.530900+0000) 2022-09-08T09:08:25.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:24 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:24.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:24.787599+0000) 2022-09-08T09:08:25.790 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:25.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:25.531119+0000) 2022-09-08T09:08:26.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:25 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:25.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:25.787774+0000) 2022-09-08T09:08:26.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:26 smithi055 ceph-mon[97033]: pgmap v22400: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.4 KiB/s rd, 95 B/s wr, 8 op/s 2022-09-08T09:08:26.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:26 smithi059 ceph-mon[97450]: pgmap v22400: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 4.4 KiB/s rd, 95 B/s wr, 8 op/s 2022-09-08T09:08:26.790 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:26.529+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:26.531353+0000) 2022-09-08T09:08:27.056 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:26 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:26.786+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:26.787933+0000) 2022-09-08T09:08:27.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:08:27.052+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:08:27.790 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:27.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:27.531507+0000) 2022-09-08T09:08:28.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:27 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:27.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:27.788145+0000) 2022-09-08T09:08:28.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:28 smithi055 ceph-mon[97033]: pgmap v22401: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T09:08:28.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:28 smithi059 ceph-mon[97450]: pgmap v22401: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T09:08:28.791 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:28.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:28.531686+0000) 2022-09-08T09:08:29.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:28 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:28.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:28.788391+0000) 2022-09-08T09:08:29.791 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:29.530+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:29.531901+0000) 2022-09-08T09:08:30.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:29 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:29.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:29.788601+0000) 2022-09-08T09:08:30.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:30 smithi055 ceph-mon[97033]: pgmap v22402: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T09:08:30.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:30 smithi059 ceph-mon[97450]: pgmap v22402: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.6 KiB/s rd, 5 op/s 2022-09-08T09:08:30.791 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:30.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:30.532149+0000) 2022-09-08T09:08:31.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:30 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:30.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:30.788851+0000) 2022-09-08T09:08:31.791 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:31.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:31.532302+0000) 2022-09-08T09:08:32.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:31 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:31.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:31.789050+0000) 2022-09-08T09:08:32.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:32 smithi055 ceph-mon[97033]: pgmap v22403: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T09:08:32.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:32 smithi059 ceph-mon[97450]: pgmap v22403: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.8 KiB/s rd, 7 op/s 2022-09-08T09:08:32.791 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:32.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:32.532537+0000) 2022-09-08T09:08:33.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:32 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:32.787+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:32.789299+0000) 2022-09-08T09:08:33.792 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:33.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:33.532774+0000) 2022-09-08T09:08:34.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:33 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:33.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:33.789542+0000) 2022-09-08T09:08:34.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:34 smithi055 ceph-mon[97033]: pgmap v22404: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:08:34.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:34 smithi059 ceph-mon[97450]: pgmap v22404: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:08:34.792 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:34.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:34.532920+0000) 2022-09-08T09:08:35.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:34 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:34.788+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:34.789709+0000) 2022-09-08T09:08:35.792 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:35.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:35.533134+0000) 2022-09-08T09:08:36.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:35 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:35.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:35.789940+0000) 2022-09-08T09:08:36.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:36 smithi055 ceph-mon[97033]: pgmap v22405: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:08:36.783 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:36 smithi059 ceph-mon[97450]: pgmap v22405: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:08:36.792 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:36.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:36.533350+0000) 2022-09-08T09:08:37.161 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:36 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:36.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:36.790175+0000) 2022-09-08T09:08:37.792 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:37.531+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:37.533610+0000) 2022-09-08T09:08:38.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:37 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:37.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:37.790401+0000) 2022-09-08T09:08:38.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:38 smithi055 ceph-mon[97033]: pgmap v22406: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:08:38.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:38 smithi059 ceph-mon[97450]: pgmap v22406: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:08:38.797 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:38.532+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:38.533823+0000) 2022-09-08T09:08:39.153 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:38 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:38.789+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:38.790559+0000) 2022-09-08T09:08:39.793 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:39.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:39.534059+0000) 2022-09-08T09:08:40.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:39 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:39.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:39.790789+0000) 2022-09-08T09:08:40.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:40 smithi055 ceph-mon[97033]: pgmap v22407: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:08:40.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:40 smithi059 ceph-mon[97450]: pgmap v22407: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-09-08T09:08:40.793 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:40.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:40.534222+0000) 2022-09-08T09:08:41.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:40 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:40.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:40.790987+0000) 2022-09-08T09:08:41.793 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:41.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:41.534378+0000) 2022-09-08T09:08:42.070 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:41 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:41.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:41.791171+0000) 2022-09-08T09:08:42.404 INFO:journalctl@ceph.mon.a.smithi047.stdout:Sep 08 09:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mon-a[90063]: 2022-09-08T09:08:42.066+0000 7f834d15a700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-08T09:08:42.737 INFO:journalctl@ceph.mon.b.smithi055.stdout:Sep 08 09:08:42 smithi055 ceph-mon[97033]: pgmap v22408: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:08:42.782 INFO:journalctl@ceph.mon.c.smithi059.stdout:Sep 08 09:08:42 smithi059 ceph-mon[97450]: pgmap v22408: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-09-08T09:08:42.794 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:42.533+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:42.534539+0000) 2022-09-08T09:08:43.154 INFO:journalctl@ceph.osd.0.smithi047.stdout:Sep 08 09:08:42 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-osd-0[102320]: 2022-09-08T09:08:42.790+0000 7f7539751700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:42.791376+0000) 2022-09-08T09:08:43.794 INFO:journalctl@ceph.mds.j.smithi047.stdout:Sep 08 09:08:43 smithi047 ceph-01b8396e-2ef4-11ed-8431-001a4aab830c-mds-j[124055]: 2022-09-08T09:08:43.534+0000 7fdbc7bd0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-08T08:08:43.534721+0000) 2022-09-08T09:08:43.959 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-09-08T09:08:43.961 DEBUG:teuthology.task.console_log:Killing console logger for smithi047 2022-09-08T09:08:43.963 DEBUG:teuthology.task.console_log:Killing console logger for smithi055 2022-09-08T09:08:43.963 DEBUG:teuthology.task.console_log:Killing console logger for smithi059 2022-09-08T09:08:43.964 DEBUG:teuthology.exit:Finished running handlers